2023-10-16T09:31:03.114 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T09:31:03.115 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --name mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi --archive /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966 --owner mchangir --verbose -- /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/orig.config.yaml 2023-10-16T09:31:03.159 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T09:31:03.240 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966 branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: mchangir@redhat.com first_in_suite: false job_id: '7428966' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: c7471068426370a3754c2250e7729280233d552b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7471068426370a3754c2250e7729280233d552b kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: mgr-snap_schedule-support-subvol-and-group-arguments sha1: 7b3d12460841f3cbe9400ba1b68e3e20957ff531 owner: mchangir priority: 25 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 127 sha1: c7471068426370a3754c2250e7729280233d552b sleep_before_teardown: 0 suite: fs suite_branch: mgr-snap_schedule-support-subvol-and-group-arguments suite_path: /home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa suite_relpath: qa suite_repo: https://github.com/mchangir/ceph.git suite_sha1: 7b3d12460841f3cbe9400ba1b68e3e20957ff531 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_09:20:41 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-16T09:31:03.241 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa; will attempt to use it 2023-10-16T09:31:03.241 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks 2023-10-16T09:31:03.242 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-16T09:31:03.243 INFO:teuthology.task.internal:Checking packages... 2023-10-16T09:31:03.266 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'c7471068426370a3754c2250e7729280233d552b' 2023-10-16T09:31:03.266 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-16T09:31:03.266 INFO:teuthology.packaging:ref: None 2023-10-16T09:31:03.267 INFO:teuthology.packaging:tag: None 2023-10-16T09:31:03.267 INFO:teuthology.packaging:branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-16T09:31:03.267 INFO:teuthology.packaging:sha1: c7471068426370a3754c2250e7729280233d552b 2023-10-16T09:31:03.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-16T09:31:03.527 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6675.gc7471068 2023-10-16T09:31:03.527 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-16T09:31:03.533 INFO:teuthology.task.internal:no buildpackages task found 2023-10-16T09:31:03.534 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-16T09:31:03.552 INFO:teuthology.task.internal:Saving configuration 2023-10-16T09:31:03.568 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-16T09:31:03.574 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T09:31:03.599 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 09:25:22.473922', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T09:31:03.619 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 09:25:22.472359', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T09:31:03.620 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-16T09:31:03.626 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-16T09:31:03.626 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-16T09:31:03.626 INFO:teuthology.run_tasks:Running task console_log... 2023-10-16T09:31:03.697 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbdec8b69d0>, signals=[15]) 2023-10-16T09:31:03.698 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-16T09:31:03.704 INFO:teuthology.task.internal:Opening connections... 2023-10-16T09:31:03.704 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:31:03.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T09:31:03.773 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:31:03.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T09:31:03.842 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-16T09:31:03.848 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2023-10-16T09:31:03.868 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2023-10-16T09:31:03.869 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2023-10-16T09:31:03.927 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="Red Hat Enterprise Linux" 2023-10-16T09:31:03.928 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="8.6 (Ootpa)" 2023-10-16T09:31:03.928 INFO:teuthology.orchestra.run.smithi158.stdout:ID="rhel" 2023-10-16T09:31:03.928 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="fedora" 2023-10-16T09:31:03.928 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="8.6" 2023-10-16T09:31:03.928 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el8" 2023-10-16T09:31:03.928 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-16T09:31:03.929 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2023-10-16T09:31:03.929 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-16T09:31:03.929 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://www.redhat.com/" 2023-10-16T09:31:03.929 INFO:teuthology.orchestra.run.smithi158.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-16T09:31:03.929 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T09:31:03.930 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:31:03.930 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-16T09:31:03.930 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-16T09:31:03.930 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-16T09:31:03.931 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-16T09:31:03.931 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2023-10-16T09:31:03.948 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-10-16T09:31:03.969 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-10-16T09:31:03.969 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-10-16T09:31:04.029 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Red Hat Enterprise Linux" 2023-10-16T09:31:04.029 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8.6 (Ootpa)" 2023-10-16T09:31:04.029 INFO:teuthology.orchestra.run.smithi172.stdout:ID="rhel" 2023-10-16T09:31:04.030 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="fedora" 2023-10-16T09:31:04.030 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8.6" 2023-10-16T09:31:04.030 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2023-10-16T09:31:04.030 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-16T09:31:04.030 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2023-10-16T09:31:04.031 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-16T09:31:04.031 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.redhat.com/" 2023-10-16T09:31:04.031 INFO:teuthology.orchestra.run.smithi172.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-16T09:31:04.031 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T09:31:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:31:04.032 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-16T09:31:04.032 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-16T09:31:04.032 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-16T09:31:04.032 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-16T09:31:04.033 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-10-16T09:31:04.057 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-16T09:31:04.065 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-16T09:31:04.070 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-16T09:31:04.071 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2023-10-16T09:31:04.073 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-10-16T09:31:04.092 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-16T09:31:04.098 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-16T09:31:04.098 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2023-10-16T09:31:04.133 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-10-16T09:31:04.167 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-16T09:31:04.220 INFO:teuthology.run_tasks:Running task kernel... 2023-10-16T09:31:04.236 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-10-16T09:31:04.237 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-10-16T09:31:04.237 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-16T09:31:04.238 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T09:31:04.239 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T09:31:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T09:31:04.258 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-16T09:31:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T09:31:04.260 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2023-10-16T09:31:04.317 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:04.317 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:04.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2023-10-16T09:31:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:04.320 INFO:teuthology.task.kernel:Running kernel on smithi158: 4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:04.320 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum install -y kernel 2023-10-16T09:31:05.551 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-16T09:31:05.551 INFO:teuthology.orchestra.run.smithi158.stdout:Unable to read consumer identity 2023-10-16T09:31:05.562 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-16T09:31:05.562 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-16T09:31:07.144 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 9.9 kB/s | 3.5 kB 00:00 2023-10-16T09:31:07.190 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2023-10-16T09:31:08.778 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 16 MB 00:01 2023-10-16T09:31:09.233 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 16 MB 00:02 2023-10-16T09:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 296 kB/s | 24 kB 00:00 2023-10-16T09:31:12.975 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 90 kB/s | 24 kB 00:00 2023-10-16T09:31:13.064 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-16T09:31:13.119 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-16T09:31:13.120 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-16T09:31:13.121 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-16T09:31:13.255 DEBUG:teuthology.orchestra.run.smithi172:> echo no | sudo yum reinstall kernel || true 2023-10-16T09:31:13.585 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-16T09:31:13.585 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-16T09:31:13.669 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-16T09:31:13.728 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-16T09:31:13.729 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-16T09:31:13.730 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-16T09:31:13.847 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:01 ago on Mon 16 Oct 2023 09:31:12 AM UTC. 2023-10-16T09:31:13.880 DEBUG:teuthology.orchestra.run.smithi158:> echo no | sudo yum reinstall kernel || true 2023-10-16T09:31:13.888 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T09:31:13.929 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-10-16T09:31:13.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum reinstall -y kernel || true 2023-10-16T09:31:14.213 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-16T09:31:14.214 INFO:teuthology.orchestra.run.smithi158.stdout:Unable to read consumer identity 2023-10-16T09:31:14.305 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-16T09:31:14.305 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-16T09:31:14.477 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:02 ago on Mon 16 Oct 2023 09:31:12 AM UTC. 2023-10-16T09:31:14.520 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T09:31:14.557 INFO:teuthology.orchestra.run.smithi158.stderr:Error: No packages marked for reinstall. 2023-10-16T09:31:14.561 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:02 ago on Mon 16 Oct 2023 09:31:12 AM UTC. 2023-10-16T09:31:14.601 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T09:31:14.603 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum reinstall -y kernel || true 2023-10-16T09:31:14.654 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-10-16T09:31:14.701 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-10-16T09:31:14.783 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:14.783 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:14.783 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-16T09:31:14.784 DEBUG:teuthology.parallel:result is None 2023-10-16T09:31:14.937 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-16T09:31:14.937 INFO:teuthology.orchestra.run.smithi158.stdout:Unable to read consumer identity 2023-10-16T09:31:15.191 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:03 ago on Mon 16 Oct 2023 09:31:12 AM UTC. 2023-10-16T09:31:15.234 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T09:31:15.263 INFO:teuthology.orchestra.run.smithi158.stderr:Error: No packages marked for reinstall. 2023-10-16T09:31:15.311 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2023-10-16T09:31:15.391 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:15.392 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-16T09:31:15.392 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-16T09:31:15.392 DEBUG:teuthology.parallel:result is None 2023-10-16T09:31:15.393 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-16T09:31:15.401 INFO:teuthology.task.internal:Creating test directory... 2023-10-16T09:31:15.401 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T09:31:15.404 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T09:31:15.422 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-16T09:31:15.428 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-16T09:31:15.435 INFO:teuthology.task.internal:Creating archive directory... 2023-10-16T09:31:15.435 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T09:31:15.460 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T09:31:15.489 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-16T09:31:15.496 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-16T09:31:15.496 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T09:31:15.526 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T09:31:15.565 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T09:31:15.571 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T09:31:15.576 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T09:31:15.590 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T09:31:15.591 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-16T09:31:15.599 INFO:teuthology.task.internal:Configuring sudo... 2023-10-16T09:31:15.599 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T09:31:15.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T09:31:15.649 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-16T09:31:15.657 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-16T09:31:15.657 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T09:31:15.688 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T09:31:15.706 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T09:31:15.784 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T09:31:15.833 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T09:31:15.917 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T09:31:15.958 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:31:15.958 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T09:31:16.025 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T09:31:16.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T09:31:16.124 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T09:31:16.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T09:31:16.250 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:31:16.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T09:31:16.317 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2023-10-16T09:31:16.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-10-16T09:31:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T09:31:16.418 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T09:31:16.761 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-16T09:31:16.769 INFO:teuthology.task.internal:Starting timer... 2023-10-16T09:31:16.770 INFO:teuthology.run_tasks:Running task pcp... 2023-10-16T09:31:16.813 INFO:teuthology.run_tasks:Running task selinux... 2023-10-16T09:31:16.821 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-16T09:31:16.822 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2023-10-16T09:31:16.869 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: 2023-10-16T09:31:16.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2023-10-16T09:31:16.943 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2023-10-16T09:31:16.945 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-16T09:31:16.945 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2023-10-16T09:31:16.963 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2023-10-16T09:31:16.964 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2023-10-16T09:31:16.984 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2023-10-16T09:31:16.984 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi158.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2023-10-16T09:31:16.985 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T09:31:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T09:31:17.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T09:31:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T09:31:17.061 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-16T09:31:17.062 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2023-10-16T09:31:17.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2023-10-16T09:31:17.130 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-16T09:31:17.140 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-16T09:31:17.146 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-16T09:31:17.315 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-16T09:31:17.332 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-16T09:31:17.333 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi158.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-16T09:40:08.615 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi158.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2023-10-16T09:40:08.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-16T09:40:08.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T09:40:08.686 DEBUG:teuthology.orchestra.run.smithi158:> true 2023-10-16T09:40:08.751 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-16T09:40:08.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-16T09:40:08.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T09:40:08.821 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-16T09:40:08.891 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-16T09:40:08.892 INFO:teuthology.run_tasks:Running task clock... 2023-10-16T09:40:08.905 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-16T09:40:08.905 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T09:40:08.906 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T09:40:08.908 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T09:40:08.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T09:40:08.936 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T09:40:08.950 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T09:40:08.961 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T09:40:08.975 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T09:40:09.023 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: ntpd: command not found 2023-10-16T09:40:09.035 INFO:teuthology.orchestra.run.smithi158.stdout:506 Cannot talk to daemon 2023-10-16T09:40:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T09:40:09.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2023-10-16T09:40:09.063 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2023-10-16T09:40:09.064 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T09:40:09.078 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T09:40:09.092 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T09:40:09.127 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2023-10-16T09:40:09.131 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T09:40:09.131 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2023-10-16T09:40:09.131 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.131 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.131 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.132 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.162 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-10-16T09:40:09.165 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T09:40:09.166 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-10-16T09:40:09.166 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.166 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.166 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.167 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T09:40:09.168 INFO:teuthology.run_tasks:Running task install... 2023-10-16T09:40:09.176 DEBUG:teuthology.task.install:project ceph 2023-10-16T09:40:09.176 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}} 2023-10-16T09:40:09.177 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'} 2023-10-16T09:40:09.177 INFO:teuthology.task.install:Using flavor: default 2023-10-16T09:40:09.183 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-16T09:40:09.183 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-16T09:40:09.184 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-16T09:40:09.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T09:40:09.186 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-16T09:40:09.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T09:40:09.439 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/ 2023-10-16T09:40:09.439 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-16T09:40:09.456 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/ 2023-10-16T09:40:09.456 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-16T09:40:09.649 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T09:40:09.649 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:40:09.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T09:40:09.654 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T09:40:09.654 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:40:09.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T09:40:09.678 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T09:40:09.678 DEBUG:teuthology.orchestra.run.smithi158:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T09:40:09.684 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T09:40:09.684 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T09:40:09.746 DEBUG:teuthology.orchestra.run.smithi158:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T09:40:09.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T09:40:09.823 DEBUG:teuthology.orchestra.run.smithi158:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T09:40:09.832 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T09:40:09.888 INFO:teuthology.orchestra.run.smithi158.stdout:check_obsoletes = 1 2023-10-16T09:40:09.890 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum clean all 2023-10-16T09:40:09.899 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2023-10-16T09:40:09.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2023-10-16T09:40:10.261 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-16T09:40:10.270 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-16T09:40:10.495 INFO:teuthology.orchestra.run.smithi158.stdout:44 files removed 2023-10-16T09:40:10.512 INFO:teuthology.orchestra.run.smithi172.stdout:44 files removed 2023-10-16T09:40:10.542 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T09:40:10.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T09:40:10.871 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-16T09:40:10.891 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-16T09:40:11.584 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 229 kB/s | 81 kB 00:00 2023-10-16T09:40:11.686 INFO:teuthology.orchestra.run.smithi158.stdout:ceph packages for x86_64 171 kB/s | 81 kB 00:00 2023-10-16T09:40:11.943 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 29 kB/s | 9.4 kB 00:00 2023-10-16T09:40:12.012 INFO:teuthology.orchestra.run.smithi158.stdout:ceph noarch packages 32 kB/s | 9.4 kB 00:00 2023-10-16T09:40:12.186 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 6.7 kB/s | 1.5 kB 00:00 2023-10-16T09:40:12.537 INFO:teuthology.orchestra.run.smithi158.stdout:ceph source packages 3.0 kB/s | 1.5 kB 00:00 2023-10-16T09:40:12.545 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-10-16T09:40:12.705 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-10-16T09:40:13.733 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2023-10-16T09:40:14.648 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 16 MB 00:02 2023-10-16T09:40:17.248 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 406 kB/s | 24 kB 00:00 2023-10-16T09:40:18.091 INFO:teuthology.orchestra.run.smithi158.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-10-16T09:40:18.138 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 396 kB/s | 24 kB 00:00 2023-10-16T09:40:19.023 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 47 MB 00:00 2023-10-16T09:40:25.917 INFO:teuthology.orchestra.run.smithi158.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 53 MB 00:00 2023-10-16T09:40:26.907 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 53 MB 00:00 2023-10-16T09:40:37.023 INFO:teuthology.orchestra.run.smithi158.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T09:40:37.024 INFO:teuthology.orchestra.run.smithi158.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T09:40:37.039 INFO:teuthology.orchestra.run.smithi158.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-16T09:40:37.040 INFO:teuthology.orchestra.run.smithi158.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-16T09:40:37.041 INFO:teuthology.orchestra.run.smithi158.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-16T09:40:37.042 INFO:teuthology.orchestra.run.smithi158.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-16T09:40:37.044 INFO:teuthology.orchestra.run.smithi158.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-16T09:40:37.047 INFO:teuthology.orchestra.run.smithi158.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-16T09:40:37.048 INFO:teuthology.orchestra.run.smithi158.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-16T09:40:37.181 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-16T09:40:37.193 INFO:teuthology.orchestra.run.smithi158.stdout:=========================================================================================================================== 2023-10-16T09:40:37.193 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-16T09:40:37.193 INFO:teuthology.orchestra.run.smithi158.stdout:=========================================================================================================================== 2023-10-16T09:40:37.193 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-16T09:40:37.194 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-16T09:40:37.194 INFO:teuthology.orchestra.run.smithi158.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-16T09:40:37.194 INFO:teuthology.orchestra.run.smithi158.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.4 k 2023-10-16T09:40:37.194 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el8 ceph 5.4 M 2023-10-16T09:40:37.194 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 919 k 2023-10-16T09:40:37.195 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el8 ceph 167 k 2023-10-16T09:40:37.195 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el8 ceph 1.6 M 2023-10-16T09:40:37.195 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 132 k 2023-10-16T09:40:37.195 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 1.8 M 2023-10-16T09:40:37.195 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 7.4 M 2023-10-16T09:40:37.196 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 47 k 2023-10-16T09:40:37.196 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 14 M 2023-10-16T09:40:37.196 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el8 ceph 49 M 2023-10-16T09:40:37.196 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 261 k 2023-10-16T09:40:37.196 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 192 k 2023-10-16T09:40:37.197 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 261 k 2023-10-16T09:40:37.197 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 18 k 2023-10-16T09:40:37.197 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-16T09:40:37.197 INFO:teuthology.orchestra.run.smithi158.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-16T09:40:37.197 INFO:teuthology.orchestra.run.smithi158.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-16T09:40:37.197 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 31 k 2023-10-16T09:40:37.198 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 839 k 2023-10-16T09:40:37.198 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 139 k 2023-10-16T09:40:37.198 INFO:teuthology.orchestra.run.smithi158.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-16T09:40:37.198 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-16T09:40:37.199 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-16T09:40:37.199 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-16T09:40:37.199 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-16T09:40:37.199 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-16T09:40:37.199 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el8 ceph 197 k 2023-10-16T09:40:37.200 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el8 ceph 371 k 2023-10-16T09:40:37.200 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 364 k 2023-10-16T09:40:37.200 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 107 k 2023-10-16T09:40:37.200 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 94 k 2023-10-16T09:40:37.200 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.6 M 2023-10-16T09:40:37.200 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 180 k 2023-10-16T09:40:37.201 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-16T09:40:37.201 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrading: 2023-10-16T09:40:37.201 INFO:teuthology.orchestra.run.smithi158.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-16T09:40:37.201 INFO:teuthology.orchestra.run.smithi158.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T09:40:37.201 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T09:40:37.202 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-16T09:40:37.202 INFO:teuthology.orchestra.run.smithi158.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-16T09:40:37.202 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 23 M 2023-10-16T09:40:37.202 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 23 k 2023-10-16T09:40:37.202 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el8 ceph 2.3 M 2023-10-16T09:40:37.203 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 243 k 2023-10-16T09:40:37.203 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el8 ceph 4.6 M 2023-10-16T09:40:37.203 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 18 M 2023-10-16T09:40:37.203 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 15 k 2023-10-16T09:40:37.203 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el8 ceph 24 k 2023-10-16T09:40:37.204 INFO:teuthology.orchestra.run.smithi158.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-16T09:40:37.204 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-16T09:40:37.204 INFO:teuthology.orchestra.run.smithi158.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-16T09:40:37.204 INFO:teuthology.orchestra.run.smithi158.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-16T09:40:37.204 INFO:teuthology.orchestra.run.smithi158.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-16T09:40:37.205 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el8 ceph 184 k 2023-10-16T09:40:37.205 INFO:teuthology.orchestra.run.smithi158.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-16T09:40:37.205 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-16T09:40:37.205 INFO:teuthology.orchestra.run.smithi158.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-16T09:40:37.206 INFO:teuthology.orchestra.run.smithi158.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-16T09:40:37.206 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-16T09:40:37.206 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-16T09:40:37.206 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 518 k 2023-10-16T09:40:37.206 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-16T09:40:37.207 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.9 M 2023-10-16T09:40:37.207 INFO:teuthology.orchestra.run.smithi158.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-16T09:40:37.207 INFO:teuthology.orchestra.run.smithi158.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-16T09:40:37.207 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-16T09:40:37.207 INFO:teuthology.orchestra.run.smithi158.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-16T09:40:37.208 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-16T09:40:37.208 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-16T09:40:37.208 INFO:teuthology.orchestra.run.smithi158.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-16T09:40:37.208 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-16T09:40:37.208 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-16T09:40:37.209 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-16T09:40:37.209 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-16T09:40:37.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-16T09:40:37.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-16T09:40:37.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-16T09:40:37.210 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-16T09:40:37.210 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-16T09:40:37.210 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 45 k 2023-10-16T09:40:37.210 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 126 k 2023-10-16T09:40:37.210 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-16T09:40:37.211 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-16T09:40:37.211 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-16T09:40:37.211 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-16T09:40:37.211 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-16T09:40:37.211 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-16T09:40:37.212 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-16T09:40:37.212 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-16T09:40:37.212 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-16T09:40:37.212 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-16T09:40:37.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-16T09:40:37.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T09:40:37.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-16T09:40:37.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-16T09:40:37.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-16T09:40:37.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-16T09:40:37.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-16T09:40:37.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-16T09:40:37.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-16T09:40:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-16T09:40:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-16T09:40:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-16T09:40:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-16T09:40:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-16T09:40:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-16T09:40:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-16T09:40:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-16T09:40:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-16T09:40:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-16T09:40:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-16T09:40:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T09:40:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-16T09:40:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-16T09:40:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-16T09:40:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-16T09:40:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-16T09:40:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-16T09:40:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-16T09:40:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-16T09:40:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-16T09:40:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-16T09:40:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-16T09:40:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-16T09:40:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-16T09:40:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-16T09:40:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-16T09:40:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-16T09:40:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-16T09:40:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-16T09:40:37.221 INFO:teuthology.orchestra.run.smithi158.stdout:Enabling module streams: 2023-10-16T09:40:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql 10 2023-10-16T09:40:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:40:37.221 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-16T09:40:37.221 INFO:teuthology.orchestra.run.smithi158.stdout:=========================================================================================================================== 2023-10-16T09:40:37.222 INFO:teuthology.orchestra.run.smithi158.stdout:Install 128 Packages 2023-10-16T09:40:37.222 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrade 3 Packages 2023-10-16T09:40:37.222 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:40:37.223 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 209 M 2023-10-16T09:40:37.223 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-16T09:40:37.434 INFO:teuthology.orchestra.run.smithi158.stdout:(1/131): ceph-18.0.0-6675.gc7471068.el8.x86_64. 51 kB/s | 6.4 kB 00:00 2023-10-16T09:40:37.768 INFO:teuthology.orchestra.run.smithi158.stdout:(2/131): ceph-fuse-18.0.0-6675.gc7471068.el8.x8 2.7 MB/s | 919 kB 00:00 2023-10-16T09:40:37.843 INFO:teuthology.orchestra.run.smithi158.stdout:(3/131): ceph-base-18.0.0-6675.gc7471068.el8.x8 10 MB/s | 5.4 MB 00:00 2023-10-16T09:40:37.868 INFO:teuthology.orchestra.run.smithi158.stdout:(4/131): ceph-immutable-object-cache-18.0.0-667 1.6 MB/s | 167 kB 00:00 2023-10-16T09:40:37.996 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T09:40:37.997 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T09:40:38.012 INFO:teuthology.orchestra.run.smithi172.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-16T09:40:38.013 INFO:teuthology.orchestra.run.smithi172.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-16T09:40:38.014 INFO:teuthology.orchestra.run.smithi172.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-16T09:40:38.015 INFO:teuthology.orchestra.run.smithi172.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-16T09:40:38.018 INFO:teuthology.orchestra.run.smithi172.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-16T09:40:38.021 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-16T09:40:38.022 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-16T09:40:38.077 INFO:teuthology.orchestra.run.smithi158.stdout:(5/131): ceph-mds-18.0.0-6675.gc7471068.el8.x86 9.9 MB/s | 2.3 MB 00:00 2023-10-16T09:40:38.154 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-16T09:40:38.166 INFO:teuthology.orchestra.run.smithi172.stdout:=========================================================================================================================== 2023-10-16T09:40:38.166 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-16T09:40:38.167 INFO:teuthology.orchestra.run.smithi172.stdout:=========================================================================================================================== 2023-10-16T09:40:38.167 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-16T09:40:38.167 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-16T09:40:38.167 INFO:teuthology.orchestra.run.smithi172.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-16T09:40:38.167 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.4 k 2023-10-16T09:40:38.168 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el8 ceph 5.4 M 2023-10-16T09:40:38.168 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 919 k 2023-10-16T09:40:38.168 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el8 ceph 167 k 2023-10-16T09:40:38.168 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el8 ceph 1.6 M 2023-10-16T09:40:38.168 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 132 k 2023-10-16T09:40:38.169 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 1.8 M 2023-10-16T09:40:38.169 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 7.4 M 2023-10-16T09:40:38.169 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 47 k 2023-10-16T09:40:38.169 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 14 M 2023-10-16T09:40:38.169 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el8 ceph 49 M 2023-10-16T09:40:38.170 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 261 k 2023-10-16T09:40:38.170 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 192 k 2023-10-16T09:40:38.170 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 261 k 2023-10-16T09:40:38.170 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 18 k 2023-10-16T09:40:38.171 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-16T09:40:38.171 INFO:teuthology.orchestra.run.smithi172.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-16T09:40:38.171 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-16T09:40:38.171 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 31 k 2023-10-16T09:40:38.171 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 839 k 2023-10-16T09:40:38.172 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 139 k 2023-10-16T09:40:38.172 INFO:teuthology.orchestra.run.smithi172.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-16T09:40:38.172 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-16T09:40:38.172 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-16T09:40:38.172 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-16T09:40:38.173 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-16T09:40:38.173 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-16T09:40:38.173 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el8 ceph 197 k 2023-10-16T09:40:38.173 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el8 ceph 371 k 2023-10-16T09:40:38.173 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 364 k 2023-10-16T09:40:38.174 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 107 k 2023-10-16T09:40:38.174 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 94 k 2023-10-16T09:40:38.174 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.6 M 2023-10-16T09:40:38.174 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 180 k 2023-10-16T09:40:38.174 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-16T09:40:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2023-10-16T09:40:38.175 INFO:teuthology.orchestra.run.smithi172.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-16T09:40:38.175 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T09:40:38.175 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T09:40:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-16T09:40:38.176 INFO:teuthology.orchestra.run.smithi172.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-16T09:40:38.176 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 23 M 2023-10-16T09:40:38.176 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 23 k 2023-10-16T09:40:38.176 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el8 ceph 2.3 M 2023-10-16T09:40:38.176 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 243 k 2023-10-16T09:40:38.177 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el8 ceph 4.6 M 2023-10-16T09:40:38.177 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 18 M 2023-10-16T09:40:38.177 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 15 k 2023-10-16T09:40:38.177 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el8 ceph 24 k 2023-10-16T09:40:38.177 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-16T09:40:38.178 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-16T09:40:38.178 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-16T09:40:38.178 INFO:teuthology.orchestra.run.smithi172.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-16T09:40:38.178 INFO:teuthology.orchestra.run.smithi172.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-16T09:40:38.178 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el8 ceph 184 k 2023-10-16T09:40:38.179 INFO:teuthology.orchestra.run.smithi172.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-16T09:40:38.179 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-16T09:40:38.179 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-16T09:40:38.179 INFO:teuthology.orchestra.run.smithi172.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-16T09:40:38.179 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-16T09:40:38.180 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-16T09:40:38.180 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 518 k 2023-10-16T09:40:38.180 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-16T09:40:38.180 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.9 M 2023-10-16T09:40:38.180 INFO:teuthology.orchestra.run.smithi172.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-16T09:40:38.181 INFO:teuthology.orchestra.run.smithi172.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-16T09:40:38.181 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-16T09:40:38.181 INFO:teuthology.orchestra.run.smithi172.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-16T09:40:38.181 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-16T09:40:38.181 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-16T09:40:38.182 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-16T09:40:38.182 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-16T09:40:38.182 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-16T09:40:38.182 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-16T09:40:38.182 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-16T09:40:38.183 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-16T09:40:38.183 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-16T09:40:38.183 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-16T09:40:38.183 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-16T09:40:38.183 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-16T09:40:38.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 45 k 2023-10-16T09:40:38.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 126 k 2023-10-16T09:40:38.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-16T09:40:38.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-16T09:40:38.185 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-16T09:40:38.185 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-16T09:40:38.185 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-16T09:40:38.185 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-16T09:40:38.185 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-16T09:40:38.185 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-16T09:40:38.186 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-16T09:40:38.186 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-16T09:40:38.186 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-16T09:40:38.186 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T09:40:38.187 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-16T09:40:38.187 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-16T09:40:38.187 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-16T09:40:38.187 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-16T09:40:38.187 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-16T09:40:38.187 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-16T09:40:38.188 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-16T09:40:38.188 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-16T09:40:38.188 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-16T09:40:38.188 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-16T09:40:38.188 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-16T09:40:38.189 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-16T09:40:38.189 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-16T09:40:38.189 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-16T09:40:38.189 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-16T09:40:38.189 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-16T09:40:38.189 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-16T09:40:38.190 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-16T09:40:38.190 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T09:40:38.190 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-16T09:40:38.190 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-16T09:40:38.190 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-16T09:40:38.191 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-16T09:40:38.191 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-16T09:40:38.191 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-16T09:40:38.191 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-16T09:40:38.191 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-16T09:40:38.191 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-16T09:40:38.192 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-16T09:40:38.192 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-16T09:40:38.192 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-16T09:40:38.192 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-16T09:40:38.192 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-16T09:40:38.193 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-16T09:40:38.193 INFO:teuthology.orchestra.run.smithi172.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-16T09:40:38.193 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-16T09:40:38.193 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-16T09:40:38.193 INFO:teuthology.orchestra.run.smithi172.stdout:Enabling module streams: 2023-10-16T09:40:38.193 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql 10 2023-10-16T09:40:38.194 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:40:38.194 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-16T09:40:38.194 INFO:teuthology.orchestra.run.smithi172.stdout:=========================================================================================================================== 2023-10-16T09:40:38.194 INFO:teuthology.orchestra.run.smithi172.stdout:Install 128 Packages 2023-10-16T09:40:38.194 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 3 Packages 2023-10-16T09:40:38.195 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:40:38.195 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 209 M 2023-10-16T09:40:38.196 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-16T09:40:38.403 INFO:teuthology.orchestra.run.smithi158.stdout:(6/131): ceph-mgr-18.0.0-6675.gc7471068.el8.x86 3.0 MB/s | 1.6 MB 00:00 2023-10-16T09:40:38.469 INFO:teuthology.orchestra.run.smithi158.stdout:(7/131): ceph-mon-18.0.0-6675.gc7471068.el8.x86 12 MB/s | 4.6 MB 00:00 2023-10-16T09:40:38.490 INFO:teuthology.orchestra.run.smithi172.stdout:(1/131): ceph-18.0.0-6675.gc7471068.el8.x86_64. 41 kB/s | 6.4 kB 00:00 2023-10-16T09:40:39.524 INFO:teuthology.orchestra.run.smithi172.stdout:(2/131): ceph-fuse-18.0.0-6675.gc7471068.el8.x8 889 kB/s | 919 kB 00:01 2023-10-16T09:40:39.595 INFO:teuthology.orchestra.run.smithi158.stdout:(8/131): ceph-radosgw-18.0.0-6675.gc7471068.el8 12 MB/s | 14 MB 00:01 2023-10-16T09:40:39.616 INFO:teuthology.orchestra.run.smithi172.stdout:(3/131): ceph-immutable-object-cache-18.0.0-667 1.8 MB/s | 167 kB 00:00 2023-10-16T09:40:39.837 INFO:teuthology.orchestra.run.smithi158.stdout:(9/131): ceph-common-18.0.0-6675.gc7471068.el8. 8.9 MB/s | 23 MB 00:02 2023-10-16T09:40:39.862 INFO:teuthology.orchestra.run.smithi158.stdout:(10/131): ceph-selinux-18.0.0-6675.gc7471068.el 91 kB/s | 24 kB 00:00 2023-10-16T09:40:39.929 INFO:teuthology.orchestra.run.smithi158.stdout:(11/131): cephfs-mirror-18.0.0-6675.gc7471068.e 3.8 MB/s | 261 kB 00:00 2023-10-16T09:40:39.987 INFO:teuthology.orchestra.run.smithi158.stdout:(12/131): libcephfs-devel-18.0.0-6675.gc7471068 532 kB/s | 31 kB 00:00 2023-10-16T09:40:40.113 INFO:teuthology.orchestra.run.smithi158.stdout:(13/131): libcephfs2-18.0.0-6675.gc7471068.el8. 6.6 MB/s | 839 kB 00:00 2023-10-16T09:40:40.180 INFO:teuthology.orchestra.run.smithi158.stdout:(14/131): libcephsqlite-18.0.0-6675.gc7471068.e 2.7 MB/s | 184 kB 00:00 2023-10-16T09:40:40.247 INFO:teuthology.orchestra.run.smithi158.stdout:(15/131): librados-devel-18.0.0-6675.gc7471068. 2.0 MB/s | 139 kB 00:00 2023-10-16T09:40:40.330 INFO:teuthology.orchestra.run.smithi158.stdout:(16/131): libradosstriper1-18.0.0-6675.gc747106 6.1 MB/s | 518 kB 00:00 2023-10-16T09:40:40.783 INFO:teuthology.orchestra.run.smithi172.stdout:(4/131): ceph-mds-18.0.0-6675.gc7471068.el8.x86 2.0 MB/s | 2.3 MB 00:01 2023-10-16T09:40:40.981 INFO:teuthology.orchestra.run.smithi158.stdout:(17/131): librgw2-18.0.0-6675.gc7471068.el8.x86 11 MB/s | 6.9 MB 00:00 2023-10-16T09:40:41.048 INFO:teuthology.orchestra.run.smithi158.stdout:(18/131): python3-ceph-argparse-18.0.0-6675.gc7 671 kB/s | 45 kB 00:00 2023-10-16T09:40:41.114 INFO:teuthology.orchestra.run.smithi158.stdout:(19/131): python3-ceph-common-18.0.0-6675.gc747 1.8 MB/s | 126 kB 00:00 2023-10-16T09:40:41.181 INFO:teuthology.orchestra.run.smithi158.stdout:(20/131): python3-cephfs-18.0.0-6675.gc7471068. 2.9 MB/s | 197 kB 00:00 2023-10-16T09:40:41.257 INFO:teuthology.orchestra.run.smithi158.stdout:(21/131): python3-rados-18.0.0-6675.gc7471068.e 4.8 MB/s | 371 kB 00:00 2023-10-16T09:40:41.349 INFO:teuthology.orchestra.run.smithi158.stdout:(22/131): python3-rbd-18.0.0-6675.gc7471068.el8 3.9 MB/s | 364 kB 00:00 2023-10-16T09:40:41.424 INFO:teuthology.orchestra.run.smithi158.stdout:(23/131): python3-rgw-18.0.0-6675.gc7471068.el8 1.4 MB/s | 107 kB 00:00 2023-10-16T09:40:41.466 INFO:teuthology.orchestra.run.smithi172.stdout:(5/131): ceph-mgr-18.0.0-6675.gc7471068.el8.x86 2.4 MB/s | 1.6 MB 00:00 2023-10-16T09:40:41.491 INFO:teuthology.orchestra.run.smithi158.stdout:(24/131): rbd-fuse-18.0.0-6675.gc7471068.el8.x8 1.4 MB/s | 94 kB 00:00 2023-10-16T09:40:41.858 INFO:teuthology.orchestra.run.smithi172.stdout:(6/131): ceph-base-18.0.0-6675.gc7471068.el8.x8 1.5 MB/s | 5.4 MB 00:03 2023-10-16T09:40:41.866 INFO:teuthology.orchestra.run.smithi158.stdout:(25/131): rbd-mirror-18.0.0-6675.gc7471068.el8. 9.6 MB/s | 3.6 MB 00:00 2023-10-16T09:40:42.075 INFO:teuthology.orchestra.run.smithi158.stdout:(26/131): ceph-osd-18.0.0-6675.gc7471068.el8.x8 4.9 MB/s | 18 MB 00:03 2023-10-16T09:40:42.100 INFO:teuthology.orchestra.run.smithi158.stdout:(27/131): rbd-nbd-18.0.0-6675.gc7471068.el8.x86 769 kB/s | 180 kB 00:00 2023-10-16T09:40:42.134 INFO:teuthology.orchestra.run.smithi158.stdout:(28/131): ceph-grafana-dashboards-18.0.0-6675.g 395 kB/s | 23 kB 00:00 2023-10-16T09:40:42.159 INFO:teuthology.orchestra.run.smithi158.stdout:(29/131): ceph-mgr-cephadm-18.0.0-6675.gc747106 2.2 MB/s | 132 kB 00:00 2023-10-16T09:40:42.568 INFO:teuthology.orchestra.run.smithi158.stdout:(30/131): ceph-mgr-dashboard-18.0.0-6675.gc7471 4.1 MB/s | 1.8 MB 00:00 2023-10-16T09:40:42.668 INFO:teuthology.orchestra.run.smithi158.stdout:(31/131): ceph-mgr-modules-core-18.0.0-6675.gc7 2.4 MB/s | 243 kB 00:00 2023-10-16T09:40:42.760 INFO:teuthology.orchestra.run.smithi158.stdout:(32/131): ceph-mgr-rook-18.0.0-6675.gc7471068.e 518 kB/s | 47 kB 00:00 2023-10-16T09:40:42.844 INFO:teuthology.orchestra.run.smithi158.stdout:(33/131): ceph-mgr-diskprediction-local-18.0.0- 11 MB/s | 7.4 MB 00:00 2023-10-16T09:40:42.869 INFO:teuthology.orchestra.run.smithi158.stdout:(34/131): ceph-prometheus-alerts-18.0.0-6675.gc 134 kB/s | 15 kB 00:00 2023-10-16T09:40:42.911 INFO:teuthology.orchestra.run.smithi158.stdout:(35/131): ceph-volume-18.0.0-6675.gc7471068.el8 3.8 MB/s | 261 kB 00:00 2023-10-16T09:40:42.945 INFO:teuthology.orchestra.run.smithi158.stdout:(36/131): cephadm-18.0.0-6675.gc7471068.el8.noa 2.5 MB/s | 192 kB 00:00 2023-10-16T09:40:42.970 INFO:teuthology.orchestra.run.smithi158.stdout:(37/131): cephfs-top-18.0.0-6675.gc7471068.el8. 307 kB/s | 18 kB 00:00 2023-10-16T09:40:43.187 INFO:teuthology.orchestra.run.smithi158.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 271 kB/s | 59 kB 00:00 2023-10-16T09:40:43.200 INFO:teuthology.orchestra.run.smithi172.stdout:(7/131): ceph-mon-18.0.0-6675.gc7471068.el8.x86 2.7 MB/s | 4.6 MB 00:01 2023-10-16T09:40:43.288 INFO:teuthology.orchestra.run.smithi158.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 752 kB/s | 75 kB 00:00 2023-10-16T09:40:43.338 INFO:teuthology.orchestra.run.smithi158.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 780 kB/s | 306 kB 00:00 2023-10-16T09:40:43.405 INFO:teuthology.orchestra.run.smithi158.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 654 kB/s | 44 kB 00:00 2023-10-16T09:40:43.547 INFO:teuthology.orchestra.run.smithi158.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.8 MB/s | 476 kB 00:00 2023-10-16T09:40:43.597 INFO:teuthology.orchestra.run.smithi158.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 964 kB/s | 185 kB 00:00 2023-10-16T09:40:43.622 INFO:teuthology.orchestra.run.smithi158.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 437 kB/s | 33 kB 00:00 2023-10-16T09:40:43.656 INFO:teuthology.orchestra.run.smithi158.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 278 kB/s | 16 kB 00:00 2023-10-16T09:40:44.131 INFO:teuthology.orchestra.run.smithi158.stdout:(46/131): ceph-test-18.0.0-6675.gc7471068.el8.x 11 MB/s | 49 MB 00:04 2023-10-16T09:40:44.156 INFO:teuthology.orchestra.run.smithi158.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 323 kB/s | 173 kB 00:00 2023-10-16T09:40:44.182 INFO:teuthology.orchestra.run.smithi158.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 731 kB/s | 384 kB 00:00 2023-10-16T09:40:44.207 INFO:teuthology.orchestra.run.smithi158.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 219 kB/s | 11 kB 00:00 2023-10-16T09:40:44.241 INFO:teuthology.orchestra.run.smithi158.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 324 kB/s | 18 kB 00:00 2023-10-16T09:40:44.341 INFO:teuthology.orchestra.run.smithi158.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 491 kB/s | 49 kB 00:00 2023-10-16T09:40:44.458 INFO:teuthology.orchestra.run.smithi158.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 502 kB/s | 59 kB 00:00 2023-10-16T09:40:44.484 INFO:teuthology.orchestra.run.smithi158.stdout:(53/131): python3-google-auth-1.1.1-10.el8.noar 231 kB/s | 81 kB 00:00 2023-10-16T09:40:44.526 INFO:teuthology.orchestra.run.smithi158.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 901 kB/s | 60 kB 00:00 2023-10-16T09:40:44.593 INFO:teuthology.orchestra.run.smithi158.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 234 kB/s | 16 kB 00:00 2023-10-16T09:40:44.618 INFO:teuthology.orchestra.run.smithi158.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2023-10-16T09:40:44.660 INFO:teuthology.orchestra.run.smithi158.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 502 kB/s | 34 kB 00:00 2023-10-16T09:40:44.735 INFO:teuthology.orchestra.run.smithi158.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.6 MB/s | 196 kB 00:00 2023-10-16T09:40:44.760 INFO:teuthology.orchestra.run.smithi158.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 641 kB/s | 64 kB 00:00 2023-10-16T09:40:44.794 INFO:teuthology.orchestra.run.smithi158.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 319 kB/s | 19 kB 00:00 2023-10-16T09:40:44.820 INFO:teuthology.orchestra.run.smithi158.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 408 kB/s | 24 kB 00:00 2023-10-16T09:40:44.861 INFO:teuthology.orchestra.run.smithi158.stdout:(62/131): python3-kubernetes-11.0.0-6.el8.noarc 1.8 MB/s | 1.2 MB 00:00 2023-10-16T09:40:44.887 INFO:teuthology.orchestra.run.smithi158.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 317 kB/s | 29 kB 00:00 2023-10-16T09:40:44.912 INFO:teuthology.orchestra.run.smithi158.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2023-10-16T09:40:45.012 INFO:teuthology.orchestra.run.smithi158.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 613 kB/s | 61 kB 00:00 2023-10-16T09:40:45.046 INFO:teuthology.orchestra.run.smithi158.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 1.4 MB/s | 255 kB 00:00 2023-10-16T09:40:45.071 INFO:teuthology.orchestra.run.smithi158.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2023-10-16T09:40:45.138 INFO:teuthology.orchestra.run.smithi158.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 249 kB/s | 23 kB 00:00 2023-10-16T09:40:45.164 INFO:teuthology.orchestra.run.smithi158.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-10-16T09:40:45.189 INFO:teuthology.orchestra.run.smithi158.stdout:(70/131): python3-webtest-2.0.33-1.el8.noarch.r 484 kB/s | 85 kB 00:00 2023-10-16T09:40:45.256 INFO:teuthology.orchestra.run.smithi158.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.6 MB/s | 110 kB 00:00 2023-10-16T09:40:45.298 INFO:teuthology.orchestra.run.smithi158.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-10-16T09:40:45.325 INFO:teuthology.orchestra.run.smithi172.stdout:(8/131): ceph-radosgw-18.0.0-6675.gc7471068.el8 6.5 MB/s | 14 MB 00:02 2023-10-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi158.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-10-16T09:40:45.382 INFO:teuthology.orchestra.run.smithi158.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-10-16T09:40:45.432 INFO:teuthology.orchestra.run.smithi158.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 772 kB/s | 39 kB 00:00 2023-10-16T09:40:45.483 INFO:teuthology.orchestra.run.smithi158.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-10-16T09:40:45.500 INFO:teuthology.orchestra.run.smithi172.stdout:(9/131): ceph-osd-18.0.0-6675.gc7471068.el8.x86 4.9 MB/s | 18 MB 00:03 2023-10-16T09:40:45.525 INFO:teuthology.orchestra.run.smithi172.stdout:(10/131): ceph-selinux-18.0.0-6675.gc7471068.el 122 kB/s | 24 kB 00:00 2023-10-16T09:40:45.541 INFO:teuthology.orchestra.run.smithi158.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 9.0 MB/s | 538 kB 00:00 2023-10-16T09:40:45.583 INFO:teuthology.orchestra.run.smithi158.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-10-16T09:40:45.592 INFO:teuthology.orchestra.run.smithi172.stdout:(11/131): cephfs-mirror-18.0.0-6675.gc7471068.e 3.8 MB/s | 261 kB 00:00 2023-10-16T09:40:45.634 INFO:teuthology.orchestra.run.smithi172.stdout:(12/131): libcephfs-devel-18.0.0-6675.gc7471068 747 kB/s | 31 kB 00:00 2023-10-16T09:40:45.725 INFO:teuthology.orchestra.run.smithi158.stdout:(79/131): python3-scipy-1.0.0-21.module+el8.5.0 24 MB/s | 14 MB 00:00 2023-10-16T09:40:45.767 INFO:teuthology.orchestra.run.smithi158.stdout:(80/131): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2023-10-16T09:40:45.776 INFO:teuthology.orchestra.run.smithi172.stdout:(13/131): libcephfs2-18.0.0-6675.gc7471068.el8. 5.8 MB/s | 839 kB 00:00 2023-10-16T09:40:45.817 INFO:teuthology.orchestra.run.smithi158.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 801 kB/s | 187 kB 00:00 2023-10-16T09:40:45.826 INFO:teuthology.orchestra.run.smithi172.stdout:(14/131): libcephsqlite-18.0.0-6675.gc7471068.e 3.6 MB/s | 184 kB 00:00 2023-10-16T09:40:45.859 INFO:teuthology.orchestra.run.smithi158.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-10-16T09:40:45.876 INFO:teuthology.orchestra.run.smithi158.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 499 kB/s | 27 kB 00:00 2023-10-16T09:40:45.877 INFO:teuthology.orchestra.run.smithi172.stdout:(15/131): librados-devel-18.0.0-6675.gc7471068. 2.7 MB/s | 139 kB 00:00 2023-10-16T09:40:45.910 INFO:teuthology.orchestra.run.smithi158.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2023-10-16T09:40:45.927 INFO:teuthology.orchestra.run.smithi158.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 875 kB/s | 44 kB 00:00 2023-10-16T09:40:45.977 INFO:teuthology.orchestra.run.smithi172.stdout:(16/131): libradosstriper1-18.0.0-6675.gc747106 5.1 MB/s | 518 kB 00:00 2023-10-16T09:40:45.978 INFO:teuthology.orchestra.run.smithi158.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 8.9 MB/s | 457 kB 00:00 2023-10-16T09:40:46.019 INFO:teuthology.orchestra.run.smithi158.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-10-16T09:40:46.053 INFO:teuthology.orchestra.run.smithi158.stdout:(88/131): uuid-1.6.2-43.el8.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2023-10-16T09:40:46.112 INFO:teuthology.orchestra.run.smithi158.stdout:(89/131): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2023-10-16T09:40:46.137 INFO:teuthology.orchestra.run.smithi158.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2023-10-16T09:40:46.162 INFO:teuthology.orchestra.run.smithi158.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 399 kB/s | 20 kB 00:00 2023-10-16T09:40:46.187 INFO:teuthology.orchestra.run.smithi158.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-10-16T09:40:46.262 INFO:teuthology.orchestra.run.smithi158.stdout:(93/131): python3-babel-2.5.1-7.el8.noarch.rpm 9.7 MB/s | 4.8 MB 00:00 2023-10-16T09:40:46.296 INFO:teuthology.orchestra.run.smithi158.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 5.2 MB/s | 709 kB 00:00 2023-10-16T09:40:46.321 INFO:teuthology.orchestra.run.smithi158.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2023-10-16T09:40:46.363 INFO:teuthology.orchestra.run.smithi158.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-10-16T09:40:46.405 INFO:teuthology.orchestra.run.smithi158.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-10-16T09:40:46.497 INFO:teuthology.orchestra.run.smithi158.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 16 MB/s | 4.9 MB 00:00 2023-10-16T09:40:46.565 INFO:teuthology.orchestra.run.smithi158.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-10-16T09:40:46.606 INFO:teuthology.orchestra.run.smithi158.stdout:(100/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-10-16T09:40:46.640 INFO:teuthology.orchestra.run.smithi158.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-10-16T09:40:46.682 INFO:teuthology.orchestra.run.smithi158.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-10-16T09:40:46.707 INFO:teuthology.orchestra.run.smithi158.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-10-16T09:40:46.732 INFO:teuthology.orchestra.run.smithi158.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 847 kB/s | 43 kB 00:00 2023-10-16T09:40:46.758 INFO:teuthology.orchestra.run.smithi158.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-10-16T09:40:46.769 INFO:teuthology.orchestra.run.smithi172.stdout:(17/131): librgw2-18.0.0-6675.gc7471068.el8.x86 8.8 MB/s | 6.9 MB 00:00 2023-10-16T09:40:46.783 INFO:teuthology.orchestra.run.smithi158.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 955 kB/s | 48 kB 00:00 2023-10-16T09:40:46.809 INFO:teuthology.orchestra.run.smithi158.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-10-16T09:40:46.811 INFO:teuthology.orchestra.run.smithi172.stdout:(18/131): python3-ceph-argparse-18.0.0-6675.gc7 1.1 MB/s | 45 kB 00:00 2023-10-16T09:40:46.834 INFO:teuthology.orchestra.run.smithi158.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 846 kB/s | 43 kB 00:00 2023-10-16T09:40:46.859 INFO:teuthology.orchestra.run.smithi158.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-10-16T09:40:46.878 INFO:teuthology.orchestra.run.smithi172.stdout:(19/131): python3-ceph-common-18.0.0-6675.gc747 1.8 MB/s | 126 kB 00:00 2023-10-16T09:40:46.885 INFO:teuthology.orchestra.run.smithi158.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-10-16T09:40:46.928 INFO:teuthology.orchestra.run.smithi172.stdout:(20/131): python3-cephfs-18.0.0-6675.gc7471068. 3.9 MB/s | 197 kB 00:00 2023-10-16T09:40:46.977 INFO:teuthology.orchestra.run.smithi158.stdout:(111/131): postgresql-server-10.21-2.module+el8 8.9 MB/s | 5.1 MB 00:00 2023-10-16T09:40:46.993 INFO:teuthology.orchestra.run.smithi158.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 172 kB/s | 19 kB 00:00 2023-10-16T09:40:47.003 INFO:teuthology.orchestra.run.smithi172.stdout:(21/131): python3-rados-18.0.0-6675.gc7471068.e 4.8 MB/s | 371 kB 00:00 2023-10-16T09:40:47.019 INFO:teuthology.orchestra.run.smithi158.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.6 MB/s | 155 kB 00:00 2023-10-16T09:40:47.044 INFO:teuthology.orchestra.run.smithi158.stdout:(114/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.0 MB/s | 560 kB 00:00 2023-10-16T09:40:47.069 INFO:teuthology.orchestra.run.smithi158.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 355 kB/s | 18 kB 00:00 2023-10-16T09:40:47.079 INFO:teuthology.orchestra.run.smithi172.stdout:(22/131): python3-rbd-18.0.0-6675.gc7471068.el8 4.7 MB/s | 364 kB 00:00 2023-10-16T09:40:47.103 INFO:teuthology.orchestra.run.smithi158.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 5.0 MB/s | 559 kB 00:00 2023-10-16T09:40:47.128 INFO:teuthology.orchestra.run.smithi158.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 7.0 MB/s | 605 kB 00:00 2023-10-16T09:40:47.129 INFO:teuthology.orchestra.run.smithi172.stdout:(23/131): python3-rgw-18.0.0-6675.gc7471068.el8 2.1 MB/s | 107 kB 00:00 2023-10-16T09:40:47.154 INFO:teuthology.orchestra.run.smithi158.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2023-10-16T09:40:47.171 INFO:teuthology.orchestra.run.smithi172.stdout:(24/131): rbd-fuse-18.0.0-6675.gc7471068.el8.x8 2.2 MB/s | 94 kB 00:00 2023-10-16T09:40:47.179 INFO:teuthology.orchestra.run.smithi158.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.4 MB/s | 72 kB 00:00 2023-10-16T09:40:47.204 INFO:teuthology.orchestra.run.smithi158.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-10-16T09:40:47.229 INFO:teuthology.orchestra.run.smithi158.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-10-16T09:40:47.255 INFO:teuthology.orchestra.run.smithi158.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-10-16T09:40:47.280 INFO:teuthology.orchestra.run.smithi158.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-10-16T09:40:47.305 INFO:teuthology.orchestra.run.smithi158.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 770 kB/s | 39 kB 00:00 2023-10-16T09:40:47.347 INFO:teuthology.orchestra.run.smithi158.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.4 MB/s | 644 kB 00:00 2023-10-16T09:40:47.373 INFO:teuthology.orchestra.run.smithi158.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.5 MB/s | 171 kB 00:00 2023-10-16T09:40:47.398 INFO:teuthology.orchestra.run.smithi158.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-10-16T09:40:47.457 INFO:teuthology.orchestra.run.smithi158.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.6 MB/s | 2.3 MB 00:00 2023-10-16T09:40:47.507 INFO:teuthology.orchestra.run.smithi158.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-10-16T09:40:47.572 INFO:teuthology.orchestra.run.smithi172.stdout:(25/131): rbd-mirror-18.0.0-6675.gc7471068.el8. 9.0 MB/s | 3.6 MB 00:00 2023-10-16T09:40:47.655 INFO:teuthology.orchestra.run.smithi172.stdout:(26/131): rbd-nbd-18.0.0-6675.gc7471068.el8.x86 2.1 MB/s | 180 kB 00:00 2023-10-16T09:40:47.747 INFO:teuthology.orchestra.run.smithi172.stdout:(27/131): ceph-grafana-dashboards-18.0.0-6675.g 251 kB/s | 23 kB 00:00 2023-10-16T09:40:47.798 INFO:teuthology.orchestra.run.smithi172.stdout:(28/131): ceph-mgr-cephadm-18.0.0-6675.gc747106 2.6 MB/s | 132 kB 00:00 2023-10-16T09:40:47.816 INFO:teuthology.orchestra.run.smithi158.stdout:(130/131): librbd1-18.0.0-6675.gc7471068.el8.x8 8.9 MB/s | 3.7 MB 00:00 2023-10-16T09:40:48.024 INFO:teuthology.orchestra.run.smithi158.stdout:(131/131): librados2-18.0.0-6675.gc7471068.el8. 5.7 MB/s | 3.7 MB 00:00 2023-10-16T09:40:48.029 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-16T09:40:48.030 INFO:teuthology.orchestra.run.smithi158.stdout:Total 19 MB/s | 209 MB 00:10 2023-10-16T09:40:48.064 INFO:teuthology.orchestra.run.smithi172.stdout:(29/131): ceph-mgr-dashboard-18.0.0-6675.gc7471 6.6 MB/s | 1.8 MB 00:00 2023-10-16T09:40:48.798 INFO:teuthology.orchestra.run.smithi172.stdout:(30/131): ceph-mgr-diskprediction-local-18.0.0- 10 MB/s | 7.4 MB 00:00 2023-10-16T09:40:48.848 INFO:teuthology.orchestra.run.smithi172.stdout:(31/131): ceph-mgr-modules-core-18.0.0-6675.gc7 4.8 MB/s | 243 kB 00:00 2023-10-16T09:40:48.874 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-16T09:40:48.898 INFO:teuthology.orchestra.run.smithi172.stdout:(32/131): ceph-mgr-rook-18.0.0-6675.gc7471068.e 952 kB/s | 47 kB 00:00 2023-10-16T09:40:48.949 INFO:teuthology.orchestra.run.smithi172.stdout:(33/131): ceph-prometheus-alerts-18.0.0-6675.gc 290 kB/s | 15 kB 00:00 2023-10-16T09:40:48.993 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-16T09:40:48.993 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-16T09:40:49.016 INFO:teuthology.orchestra.run.smithi172.stdout:(34/131): ceph-volume-18.0.0-6675.gc7471068.el8 3.8 MB/s | 261 kB 00:00 2023-10-16T09:40:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:(35/131): cephadm-18.0.0-6675.gc7471068.el8.noa 3.7 MB/s | 192 kB 00:00 2023-10-16T09:40:49.116 INFO:teuthology.orchestra.run.smithi172.stdout:(36/131): cephfs-top-18.0.0-6675.gc7471068.el8. 364 kB/s | 18 kB 00:00 2023-10-16T09:40:49.509 INFO:teuthology.orchestra.run.smithi172.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 782 kB/s | 306 kB 00:00 2023-10-16T09:40:49.567 INFO:teuthology.orchestra.run.smithi172.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-10-16T09:40:49.642 INFO:teuthology.orchestra.run.smithi172.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-10-16T09:40:49.935 INFO:teuthology.orchestra.run.smithi172.stdout:(40/131): ceph-common-18.0.0-6675.gc7471068.el8 1.9 MB/s | 23 MB 00:11 2023-10-16T09:40:49.976 INFO:teuthology.orchestra.run.smithi172.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.4 MB/s | 476 kB 00:00 2023-10-16T09:40:50.069 INFO:teuthology.orchestra.run.smithi172.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 327 kB/s | 44 kB 00:00 2023-10-16T09:40:50.094 INFO:teuthology.orchestra.run.smithi172.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.5 MB/s | 185 kB 00:00 2023-10-16T09:40:50.114 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-16T09:40:50.118 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-16T09:40:50.136 INFO:teuthology.orchestra.run.smithi172.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 493 kB/s | 33 kB 00:00 2023-10-16T09:40:50.161 INFO:teuthology.orchestra.run.smithi172.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 243 kB/s | 16 kB 00:00 2023-10-16T09:40:50.261 INFO:teuthology.orchestra.run.smithi172.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-10-16T09:40:50.328 INFO:teuthology.orchestra.run.smithi172.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.3 MB/s | 384 kB 00:00 2023-10-16T09:40:50.354 INFO:teuthology.orchestra.run.smithi172.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 884 kB/s | 81 kB 00:00 2023-10-16T09:40:50.370 INFO:teuthology.orchestra.run.smithi172.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 265 kB/s | 11 kB 00:00 2023-10-16T09:40:50.396 INFO:teuthology.orchestra.run.smithi172.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 420 kB/s | 18 kB 00:00 2023-10-16T09:40:50.446 INFO:teuthology.orchestra.run.smithi172.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 972 kB/s | 49 kB 00:00 2023-10-16T09:40:50.922 INFO:teuthology.orchestra.run.smithi172.stdout:(52/131): ceph-test-18.0.0-6675.gc7471068.el8.x 9.0 MB/s | 49 MB 00:05 2023-10-16T09:40:50.955 INFO:teuthology.orchestra.run.smithi172.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 115 kB/s | 59 kB 00:00 2023-10-16T09:40:50.989 INFO:teuthology.orchestra.run.smithi172.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 1.9 MB/s | 1.2 MB 00:00 2023-10-16T09:40:51.047 INFO:teuthology.orchestra.run.smithi172.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2023-10-16T09:40:51.073 INFO:teuthology.orchestra.run.smithi172.stdout:(56/131): python3-natsort-7.1.1-2.el8.noarch.rp 400 kB/s | 60 kB 00:00 2023-10-16T09:40:51.106 INFO:teuthology.orchestra.run.smithi172.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 573 kB/s | 34 kB 00:00 2023-10-16T09:40:51.173 INFO:teuthology.orchestra.run.smithi172.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 964 kB/s | 64 kB 00:00 2023-10-16T09:40:51.232 INFO:teuthology.orchestra.run.smithi172.stdout:(59/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.0 MB/s | 283 kB 00:00 2023-10-16T09:40:51.257 INFO:teuthology.orchestra.run.smithi172.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2023-10-16T09:40:51.283 INFO:teuthology.orchestra.run.smithi172.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2023-10-16T09:40:51.308 INFO:teuthology.orchestra.run.smithi172.stdout:(62/131): python3-routes-2.4.1-12.el8.noarch.rp 837 kB/s | 196 kB 00:00 2023-10-16T09:40:51.333 INFO:teuthology.orchestra.run.smithi172.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2023-10-16T09:40:51.358 INFO:teuthology.orchestra.run.smithi172.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 365 kB/s | 27 kB 00:00 2023-10-16T09:40:51.425 INFO:teuthology.orchestra.run.smithi172.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 922 kB/s | 61 kB 00:00 2023-10-16T09:40:51.467 INFO:teuthology.orchestra.run.smithi172.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-10-16T09:40:51.509 INFO:teuthology.orchestra.run.smithi172.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 550 kB/s | 23 kB 00:00 2023-10-16T09:40:51.534 INFO:teuthology.orchestra.run.smithi172.stdout:(68/131): python3-waitress-1.4.3-1.el8.noarch.r 1.1 MB/s | 255 kB 00:00 2023-10-16T09:40:51.559 INFO:teuthology.orchestra.run.smithi172.stdout:(69/131): python3-webtest-2.0.33-1.el8.noarch.r 638 kB/s | 85 kB 00:00 2023-10-16T09:40:51.585 INFO:teuthology.orchestra.run.smithi172.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-10-16T09:40:51.652 INFO:teuthology.orchestra.run.smithi172.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.8 MB/s | 110 kB 00:00 2023-10-16T09:40:51.694 INFO:teuthology.orchestra.run.smithi172.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-10-16T09:40:51.744 INFO:teuthology.orchestra.run.smithi172.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-10-16T09:40:51.794 INFO:teuthology.orchestra.run.smithi172.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-10-16T09:40:51.836 INFO:teuthology.orchestra.run.smithi172.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 936 kB/s | 39 kB 00:00 2023-10-16T09:40:51.886 INFO:teuthology.orchestra.run.smithi172.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-10-16T09:40:51.928 INFO:teuthology.orchestra.run.smithi172.stdout:(77/131): thrift-0.13.0-2.el8.x86_64.rpm 4.1 MB/s | 1.7 MB 00:00 2023-10-16T09:40:51.962 INFO:teuthology.orchestra.run.smithi172.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.0 MB/s | 538 kB 00:00 2023-10-16T09:40:52.034 INFO:teuthology.orchestra.run.smithi172.stdout:(79/131): oniguruma-6.8.2-2.el8.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2023-10-16T09:40:52.054 INFO:teuthology.orchestra.run.smithi172.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-10-16T09:40:52.213 INFO:teuthology.orchestra.run.smithi172.stdout:(81/131): python3-scipy-1.0.0-21.module+el8.5.0 21 MB/s | 14 MB 00:00 2023-10-16T09:40:52.247 INFO:teuthology.orchestra.run.smithi172.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 3.3 MB/s | 713 kB 00:00 2023-10-16T09:40:52.264 INFO:teuthology.orchestra.run.smithi172.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 529 kB/s | 27 kB 00:00 2023-10-16T09:40:52.297 INFO:teuthology.orchestra.run.smithi172.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2023-10-16T09:40:52.314 INFO:teuthology.orchestra.run.smithi172.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 874 kB/s | 44 kB 00:00 2023-10-16T09:40:52.364 INFO:teuthology.orchestra.run.smithi172.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 8.9 MB/s | 457 kB 00:00 2023-10-16T09:40:52.415 INFO:teuthology.orchestra.run.smithi172.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 4.4 MB/s | 225 kB 00:00 2023-10-16T09:40:52.482 INFO:teuthology.orchestra.run.smithi172.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2023-10-16T09:40:52.549 INFO:teuthology.orchestra.run.smithi172.stdout:(89/131): python3-babel-2.5.1-7.el8.noarch.rpm 9.7 MB/s | 4.8 MB 00:00 2023-10-16T09:40:52.574 INFO:teuthology.orchestra.run.smithi172.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 403 kB/s | 64 kB 00:00 2023-10-16T09:40:52.624 INFO:teuthology.orchestra.run.smithi172.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 266 kB/s | 20 kB 00:00 2023-10-16T09:40:52.649 INFO:teuthology.orchestra.run.smithi172.stdout:(92/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 756 kB/s | 126 kB 00:00 2023-10-16T09:40:52.675 INFO:teuthology.orchestra.run.smithi172.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.0 MB/s | 103 kB 00:00 2023-10-16T09:40:52.708 INFO:teuthology.orchestra.run.smithi172.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 8.2 MB/s | 709 kB 00:00 2023-10-16T09:40:52.734 INFO:teuthology.orchestra.run.smithi172.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2023-10-16T09:40:52.775 INFO:teuthology.orchestra.run.smithi172.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-10-16T09:40:52.826 INFO:teuthology.orchestra.run.smithi172.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-10-16T09:40:52.910 INFO:teuthology.orchestra.run.smithi172.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 19 MB/s | 4.9 MB 00:00 2023-10-16T09:40:52.985 INFO:teuthology.orchestra.run.smithi172.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-10-16T09:40:53.018 INFO:teuthology.orchestra.run.smithi172.stdout:(100/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-10-16T09:40:53.052 INFO:teuthology.orchestra.run.smithi172.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-10-16T09:40:53.094 INFO:teuthology.orchestra.run.smithi172.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 14 MB/s | 1.5 MB 00:00 2023-10-16T09:40:53.119 INFO:teuthology.orchestra.run.smithi172.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-10-16T09:40:53.136 INFO:teuthology.orchestra.run.smithi172.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-10-16T09:40:53.180 INFO:teuthology.orchestra.run.smithi172.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.5 MB/s | 220 kB 00:00 2023-10-16T09:40:53.203 INFO:teuthology.orchestra.run.smithi172.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 718 kB/s | 48 kB 00:00 2023-10-16T09:40:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.8 MB/s | 87 kB 00:00 2023-10-16T09:40:53.254 INFO:teuthology.orchestra.run.smithi172.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 849 kB/s | 43 kB 00:00 2023-10-16T09:40:53.279 INFO:teuthology.orchestra.run.smithi172.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-10-16T09:40:53.296 INFO:teuthology.orchestra.run.smithi172.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.3 MB/s | 229 kB 00:00 2023-10-16T09:40:53.339 INFO:teuthology.orchestra.run.smithi172.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-10-16T09:40:53.364 INFO:teuthology.orchestra.run.smithi172.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 278 kB/s | 19 kB 00:00 2023-10-16T09:40:53.389 INFO:teuthology.orchestra.run.smithi172.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-10-16T09:40:53.464 INFO:teuthology.orchestra.run.smithi172.stdout:(114/131): postgresql-server-10.21-2.module+el8 7.9 MB/s | 5.1 MB 00:00 2023-10-16T09:40:53.490 INFO:teuthology.orchestra.run.smithi172.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 178 kB/s | 18 kB 00:00 2023-10-16T09:40:53.515 INFO:teuthology.orchestra.run.smithi172.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 3.6 MB/s | 559 kB 00:00 2023-10-16T09:40:53.540 INFO:teuthology.orchestra.run.smithi172.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.6 MB/s | 238 kB 00:00 2023-10-16T09:40:53.566 INFO:teuthology.orchestra.run.smithi172.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.9 MB/s | 605 kB 00:00 2023-10-16T09:40:53.591 INFO:teuthology.orchestra.run.smithi172.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.4 MB/s | 72 kB 00:00 2023-10-16T09:40:53.616 INFO:teuthology.orchestra.run.smithi172.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-10-16T09:40:53.641 INFO:teuthology.orchestra.run.smithi172.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-10-16T09:40:53.667 INFO:teuthology.orchestra.run.smithi172.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-10-16T09:40:53.692 INFO:teuthology.orchestra.run.smithi172.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-10-16T09:40:53.717 INFO:teuthology.orchestra.run.smithi172.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 767 kB/s | 39 kB 00:00 2023-10-16T09:40:53.776 INFO:teuthology.orchestra.run.smithi172.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 7.5 MB/s | 644 kB 00:00 2023-10-16T09:40:53.801 INFO:teuthology.orchestra.run.smithi172.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.0 MB/s | 171 kB 00:00 2023-10-16T09:40:53.827 INFO:teuthology.orchestra.run.smithi172.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-10-16T09:40:53.885 INFO:teuthology.orchestra.run.smithi172.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.3 MB/s | 2.3 MB 00:00 2023-10-16T09:40:53.936 INFO:teuthology.orchestra.run.smithi172.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-10-16T09:40:54.227 INFO:teuthology.orchestra.run.smithi172.stdout:(130/131): librbd1-18.0.0-6675.gc7471068.el8.x8 9.3 MB/s | 3.7 MB 00:00 2023-10-16T09:40:54.343 INFO:teuthology.orchestra.run.smithi172.stdout:(131/131): librados2-18.0.0-6675.gc7471068.el8. 6.9 MB/s | 3.7 MB 00:00 2023-10-16T09:40:54.346 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-16T09:40:54.346 INFO:teuthology.orchestra.run.smithi172.stdout:Total 13 MB/s | 209 MB 00:16 2023-10-16T09:40:54.958 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-16T09:40:55.190 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-16T09:40:55.310 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-16T09:40:55.311 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-16T09:40:55.467 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-16T09:40:55.649 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-16T09:40:55.851 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-16T09:40:56.043 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-16T09:40:56.169 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-16T09:40:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-16T09:40:56.432 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-16T09:40:56.435 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-16T09:40:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-16T09:40:57.010 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T09:40:57.422 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T09:40:57.618 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T09:40:57.739 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T09:40:57.926 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T09:40:58.022 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T09:40:58.227 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 9/134 2023-10-16T09:40:58.463 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-16T09:40:58.662 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-16T09:40:59.081 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-16T09:40:59.358 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-16T09:40:59.550 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T09:40:59.642 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T09:40:59.809 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T09:40:59.874 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T09:41:00.000 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-16T09:41:00.119 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-16T09:41:00.284 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-16T09:41:00.903 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-16T09:41:01.075 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-16T09:41:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-16T09:41:01.507 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-16T09:41:01.672 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-16T09:41:01.679 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-16T09:41:01.883 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-16T09:41:01.900 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-16T09:41:02.043 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-16T09:41:02.178 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-16T09:41:02.368 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-16T09:41:02.581 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-16T09:41:02.784 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-16T09:41:02.836 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-16T09:41:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T09:41:03.068 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-16T09:41:03.295 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-16T09:41:03.435 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T09:41:03.481 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-16T09:41:03.643 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T09:41:03.760 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T09:41:03.765 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-16T09:41:03.943 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T09:41:03.977 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-16T09:41:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T09:41:04.164 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 26/134 2023-10-16T09:41:04.277 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 9/134 2023-10-16T09:41:04.311 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 27/134 2023-10-16T09:41:04.530 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-16T09:41:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-16T09:41:05.186 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-16T09:41:05.241 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 28/134 2023-10-16T09:41:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-16T09:41:05.548 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-16T09:41:05.784 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T09:41:05.794 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-16T09:41:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T09:41:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T09:41:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T09:41:06.308 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-16T09:41:06.466 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-16T09:41:06.591 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-16T09:41:06.600 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-16T09:41:06.791 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-16T09:41:07.225 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-16T09:41:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-16T09:41:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-16T09:41:08.192 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-16T09:41:08.581 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-16T09:41:08.892 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-16T09:41:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-16T09:41:09.103 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-16T09:41:09.166 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-16T09:41:09.342 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-16T09:41:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-16T09:41:09.504 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 36/134 2023-10-16T09:41:09.620 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-16T09:41:09.636 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 37/134 2023-10-16T09:41:09.789 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-16T09:41:09.791 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-16T09:41:09.958 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-16T09:41:10.014 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-16T09:41:10.135 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-16T09:41:10.227 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-16T09:41:10.277 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-16T09:41:10.398 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 26/134 2023-10-16T09:41:10.524 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-16T09:41:10.552 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 27/134 2023-10-16T09:41:10.738 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-16T09:41:11.109 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-16T09:41:11.368 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-16T09:41:11.462 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 28/134 2023-10-16T09:41:11.608 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-16T09:41:11.800 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-16T09:41:11.821 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-16T09:41:12.045 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-16T09:41:12.061 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-16T09:41:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-16T09:41:12.365 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-16T09:41:12.541 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-16T09:41:12.623 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-16T09:41:12.800 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-16T09:41:13.005 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-16T09:41:13.173 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-16T09:41:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-16T09:41:13.653 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-16T09:41:13.946 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-16T09:41:14.170 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-16T09:41:14.324 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-16T09:41:14.405 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-16T09:41:14.589 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-16T09:41:14.634 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-16T09:41:14.757 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-16T09:41:14.854 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-16T09:41:14.989 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-16T09:41:15.017 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-16T09:41:15.065 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-16T09:41:15.229 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 36/134 2023-10-16T09:41:15.301 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-16T09:41:15.361 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 37/134 2023-10-16T09:41:15.501 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-16T09:41:15.526 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-16T09:41:15.681 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-16T09:41:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-16T09:41:15.744 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-16T09:41:15.893 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-16T09:41:15.952 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-16T09:41:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-16T09:41:16.165 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-16T09:41:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-16T09:41:16.339 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-16T09:41:16.429 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-16T09:41:16.533 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-16T09:41:16.841 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-16T09:41:17.042 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-16T09:41:17.054 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-16T09:41:17.237 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-16T09:41:17.293 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-16T09:41:17.407 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-16T09:41:17.492 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-16T09:41:17.605 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-16T09:41:17.720 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-16T09:41:17.816 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-16T09:41:17.936 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-16T09:41:18.007 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-16T09:41:18.167 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-16T09:41:18.284 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-16T09:41:18.345 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-16T09:41:18.536 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-16T09:41:18.573 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-16T09:41:18.760 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-16T09:41:18.779 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-16T09:41:18.953 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-16T09:41:18.970 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-16T09:41:19.187 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-16T09:41:19.203 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-16T09:41:19.343 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-16T09:41:19.439 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-16T09:41:19.618 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-16T09:41:19.630 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-16T09:41:19.875 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-16T09:41:19.906 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-16T09:41:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-16T09:41:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-16T09:41:20.530 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-16T09:41:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-16T09:41:20.864 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-16T09:41:20.922 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-16T09:41:20.962 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-16T09:41:21.155 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-16T09:41:21.209 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-16T09:41:21.351 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-16T09:41:21.447 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-16T09:41:21.531 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-16T09:41:21.600 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-16T09:41:21.773 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-16T09:41:21.819 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-16T09:41:22.017 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-16T09:41:22.024 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-16T09:41:22.197 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-16T09:41:22.205 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-16T09:41:22.385 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-16T09:41:22.694 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-16T09:41:22.905 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-16T09:41:23.025 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-16T09:41:23.044 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-16T09:41:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-16T09:41:23.221 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T09:41:23.282 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T09:41:23.408 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-16T09:41:23.463 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 93/134 2023-10-16T09:41:23.611 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-16T09:41:23.827 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-16T09:41:23.976 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-16T09:41:24.121 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-16T09:41:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-16T09:41:24.535 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-16T09:41:24.695 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-16T09:41:24.931 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-16T09:41:25.172 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-16T09:41:25.331 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-16T09:41:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-16T09:41:26.684 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-16T09:41:26.951 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-16T09:41:27.149 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T09:41:27.229 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-16T09:41:27.362 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T09:41:27.525 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-16T09:41:27.808 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-16T09:41:27.965 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T09:41:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-16T09:41:28.270 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T09:41:28.352 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T09:41:28.513 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T09:41:28.781 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-16T09:41:28.988 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T09:41:29.060 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T09:41:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 93/134 2023-10-16T09:41:32.926 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T09:41:33.138 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T09:41:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T09:41:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T09:41:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T09:41:34.254 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T09:41:47.378 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T09:41:47.378 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /sys 2023-10-16T09:41:47.378 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /proc 2023-10-16T09:41:47.378 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2023-10-16T09:41:47.379 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /run 2023-10-16T09:41:47.379 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /mnt 2023-10-16T09:41:47.379 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /var/tmp 2023-10-16T09:41:47.379 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /home 2023-10-16T09:41:47.379 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /tmp 2023-10-16T09:41:47.379 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2023-10-16T09:41:47.380 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:41:47.701 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T09:41:49.290 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T09:41:49.466 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T09:41:49.687 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T09:41:49.876 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T09:41:50.330 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T09:41:50.517 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T09:41:50.534 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T09:41:52.242 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T09:41:52.427 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T09:41:52.548 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 102/134 2023-10-16T09:41:52.724 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 103/134 2023-10-16T09:41:52.891 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T09:41:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T09:41:52.942 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2023-10-16T09:41:52.943 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2023-10-16T09:41:52.943 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-10-16T09:41:52.943 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /run 2023-10-16T09:41:52.943 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2023-10-16T09:41:52.943 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2023-10-16T09:41:52.943 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2023-10-16T09:41:52.944 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2023-10-16T09:41:52.944 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-10-16T09:41:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:41:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T09:41:53.318 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T09:41:53.399 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T09:41:54.364 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T09:41:54.508 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T09:41:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T09:41:54.983 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 107/134 2023-10-16T09:41:55.077 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T09:41:55.166 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T09:41:55.201 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T09:41:55.297 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T09:41:55.391 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T09:41:55.403 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T09:41:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T09:41:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T09:41:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T09:41:56.730 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 110/134 2023-10-16T09:41:56.952 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T09:41:56.986 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T09:41:57.132 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T09:41:57.175 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T09:41:57.316 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T09:41:57.676 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T09:41:57.825 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T09:41:57.833 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T09:41:58.004 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T09:41:58.123 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 102/134 2023-10-16T09:41:58.332 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 103/134 2023-10-16T09:41:58.508 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T09:41:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T09:41:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T09:42:00.024 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T09:42:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T09:42:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T09:42:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 107/134 2023-10-16T09:42:00.818 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T09:42:00.852 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T09:42:01.008 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T09:42:01.019 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T09:42:02.353 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 110/134 2023-10-16T09:42:02.509 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T09:42:02.591 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T09:42:02.625 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T09:42:02.782 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T09:42:02.801 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 115/134 2023-10-16T09:42:02.824 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T09:42:02.948 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-16T09:42:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T09:42:03.140 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-16T09:42:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T09:42:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T09:42:04.046 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-16T09:42:04.385 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-16T09:42:04.577 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 119/134 2023-10-16T09:42:04.777 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-16T09:42:04.952 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 121/134 2023-10-16T09:42:05.099 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-16T09:42:07.328 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T09:42:07.622 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T09:42:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T09:42:07.798 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 124/134 2023-10-16T09:42:08.038 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 125/134 2023-10-16T09:42:08.252 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T09:42:08.309 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 126/134 2023-10-16T09:42:08.548 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 115/134 2023-10-16T09:42:08.641 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-16T09:42:08.722 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-16T09:42:08.789 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T09:42:08.797 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T09:42:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-16T09:42:08.972 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-16T09:42:09.146 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-16T09:42:09.228 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-16T09:42:09.430 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-16T09:42:09.443 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-16T09:42:09.443 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T09:42:09.658 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T09:42:09.659 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T09:42:09.826 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-16T09:42:09.910 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T09:42:10.104 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T09:42:10.152 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-16T09:42:10.315 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 119/134 2023-10-16T09:42:10.535 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-16T09:42:10.616 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T09:42:10.616 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 1/134 2023-10-16T09:42:10.617 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2/134 2023-10-16T09:42:10.617 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 3/134 2023-10-16T09:42:10.617 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 4/134 2023-10-16T09:42:10.617 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/134 2023-10-16T09:42:10.617 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T09:42:10.617 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T09:42:10.618 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T09:42:10.618 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 9/134 2023-10-16T09:42:10.618 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 10/134 2023-10-16T09:42:10.618 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 11/134 2023-10-16T09:42:10.619 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 12/134 2023-10-16T09:42:10.619 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 13/134 2023-10-16T09:42:10.619 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 14/134 2023-10-16T09:42:10.619 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 15/134 2023-10-16T09:42:10.619 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 16/134 2023-10-16T09:42:10.619 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 17/134 2023-10-16T09:42:10.620 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 18/134 2023-10-16T09:42:10.620 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 19/134 2023-10-16T09:42:10.620 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/134 2023-10-16T09:42:10.620 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 21/134 2023-10-16T09:42:10.620 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 22/134 2023-10-16T09:42:10.621 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 23/134 2023-10-16T09:42:10.621 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 24/134 2023-10-16T09:42:10.621 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 25/134 2023-10-16T09:42:10.621 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 26/134 2023-10-16T09:42:10.621 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 27/134 2023-10-16T09:42:10.622 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 28/134 2023-10-16T09:42:10.622 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/134 2023-10-16T09:42:10.622 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 30/134 2023-10-16T09:42:10.622 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 31/134 2023-10-16T09:42:10.622 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/134 2023-10-16T09:42:10.623 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/134 2023-10-16T09:42:10.623 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 34/134 2023-10-16T09:42:10.623 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/134 2023-10-16T09:42:10.623 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 36/134 2023-10-16T09:42:10.623 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 37/134 2023-10-16T09:42:10.624 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 38/134 2023-10-16T09:42:10.624 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-16T09:42:10.624 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-16T09:42:10.624 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-16T09:42:10.624 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-16T09:42:10.624 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-16T09:42:10.625 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-16T09:42:10.625 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-16T09:42:10.625 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-16T09:42:10.625 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-16T09:42:10.626 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-16T09:42:10.626 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-16T09:42:10.626 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-16T09:42:10.626 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-16T09:42:10.626 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-16T09:42:10.627 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-16T09:42:10.627 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-16T09:42:10.627 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-16T09:42:10.627 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-16T09:42:10.627 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-16T09:42:10.628 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-16T09:42:10.628 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-16T09:42:10.628 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-16T09:42:10.628 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-16T09:42:10.628 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-16T09:42:10.628 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-16T09:42:10.629 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-16T09:42:10.629 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-16T09:42:10.629 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-16T09:42:10.629 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-16T09:42:10.629 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-16T09:42:10.630 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-16T09:42:10.630 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-16T09:42:10.630 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-16T09:42:10.630 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-16T09:42:10.630 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-16T09:42:10.631 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-16T09:42:10.631 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-16T09:42:10.631 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-16T09:42:10.631 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-16T09:42:10.632 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-16T09:42:10.632 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-16T09:42:10.632 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-16T09:42:10.632 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-16T09:42:10.632 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-16T09:42:10.633 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-16T09:42:10.633 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-16T09:42:10.633 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-16T09:42:10.633 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-16T09:42:10.633 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-16T09:42:10.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-16T09:42:10.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-16T09:42:10.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-16T09:42:10.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-16T09:42:10.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-16T09:42:10.635 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-16T09:42:10.635 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-16T09:42:10.635 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-16T09:42:10.635 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-16T09:42:10.635 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-16T09:42:10.636 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-16T09:42:10.636 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-16T09:42:10.636 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-16T09:42:10.636 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-16T09:42:10.636 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-16T09:42:10.637 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-16T09:42:10.637 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-16T09:42:10.637 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-16T09:42:10.637 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-16T09:42:10.637 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-16T09:42:10.638 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-16T09:42:10.638 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-16T09:42:10.638 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-16T09:42:10.638 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-16T09:42:10.639 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-16T09:42:10.639 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-16T09:42:10.639 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-16T09:42:10.639 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-16T09:42:10.639 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-16T09:42:10.639 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-16T09:42:10.640 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-16T09:42:10.640 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-16T09:42:10.640 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-16T09:42:10.640 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-16T09:42:10.640 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-16T09:42:10.641 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-16T09:42:10.641 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-16T09:42:10.641 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-16T09:42:10.641 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-16T09:42:10.641 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-16T09:42:10.642 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T09:42:10.642 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 129/134 2023-10-16T09:42:10.642 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-16T09:42:10.642 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 131/134 2023-10-16T09:42:10.642 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T09:42:10.642 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-16T09:42:10.701 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 121/134 2023-10-16T09:42:10.840 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-16T09:42:13.049 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T09:42:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T09:42:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T09:42:13.372 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T09:42:13.372 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Oct 2023 09:40:25 AM UTC. 2023-10-16T09:42:13.489 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 124/134 2023-10-16T09:42:13.780 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 125/134 2023-10-16T09:42:14.081 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 126/134 2023-10-16T09:42:14.449 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-16T09:42:14.597 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T09:42:14.606 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T09:42:14.776 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-16T09:42:14.938 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-16T09:42:15.029 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-16T09:42:15.213 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-16T09:42:15.226 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-16T09:42:15.227 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T09:42:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T09:42:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T09:42:15.660 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T09:42:15.837 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T09:42:16.129 INFO:teuthology.orchestra.run.smithi158.stdout:Installed products updated. 2023-10-16T09:42:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T09:42:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 1/134 2023-10-16T09:42:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2/134 2023-10-16T09:42:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 3/134 2023-10-16T09:42:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 4/134 2023-10-16T09:42:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/134 2023-10-16T09:42:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T09:42:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T09:42:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T09:42:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 9/134 2023-10-16T09:42:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 10/134 2023-10-16T09:42:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 11/134 2023-10-16T09:42:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 12/134 2023-10-16T09:42:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 13/134 2023-10-16T09:42:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 14/134 2023-10-16T09:42:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 15/134 2023-10-16T09:42:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 16/134 2023-10-16T09:42:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 17/134 2023-10-16T09:42:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 18/134 2023-10-16T09:42:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 19/134 2023-10-16T09:42:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/134 2023-10-16T09:42:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 21/134 2023-10-16T09:42:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 22/134 2023-10-16T09:42:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 23/134 2023-10-16T09:42:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 24/134 2023-10-16T09:42:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 25/134 2023-10-16T09:42:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 26/134 2023-10-16T09:42:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 27/134 2023-10-16T09:42:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 28/134 2023-10-16T09:42:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/134 2023-10-16T09:42:16.358 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:16.358 INFO:teuthology.orchestra.run.smithi158.stdout:Upgraded: 2023-10-16T09:42:16.358 INFO:teuthology.orchestra.run.smithi158.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-16T09:42:16.358 INFO:teuthology.orchestra.run.smithi158.stdout: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.359 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.359 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-16T09:42:16.359 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-16T09:42:16.359 INFO:teuthology.orchestra.run.smithi158.stdout: automake-1.16.1-7.el8.noarch 2023-10-16T09:42:16.360 INFO:teuthology.orchestra.run.smithi158.stdout: bison-3.0.4-10.el8.x86_64 2023-10-16T09:42:16.360 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.360 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.361 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.361 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.361 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.361 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.361 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.362 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.362 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.362 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.362 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.362 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.363 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.363 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.363 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.363 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.364 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.364 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.364 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.364 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.364 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.365 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.365 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:16.365 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-16T09:42:16.365 INFO:teuthology.orchestra.run.smithi158.stdout: flex-2.6.1-9.el8.x86_64 2023-10-16T09:42:16.365 INFO:teuthology.orchestra.run.smithi158.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-16T09:42:16.366 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-16T09:42:16.366 INFO:teuthology.orchestra.run.smithi158.stdout: jq-1.6-3.el8.x86_64 2023-10-16T09:42:16.366 INFO:teuthology.orchestra.run.smithi158.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-16T09:42:16.366 INFO:teuthology.orchestra.run.smithi158.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-16T09:42:16.366 INFO:teuthology.orchestra.run.smithi158.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-16T09:42:16.367 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.367 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.367 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.367 INFO:teuthology.orchestra.run.smithi158.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-16T09:42:16.368 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-16T09:42:16.368 INFO:teuthology.orchestra.run.smithi158.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-16T09:42:16.368 INFO:teuthology.orchestra.run.smithi158.stdout: libpq-13.5-1.el8.x86_64 2023-10-16T09:42:16.368 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-16T09:42:16.368 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-16T09:42:16.368 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.369 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.369 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-16T09:42:16.369 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.369 INFO:teuthology.orchestra.run.smithi158.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-16T09:42:16.370 INFO:teuthology.orchestra.run.smithi158.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-16T09:42:16.370 INFO:teuthology.orchestra.run.smithi158.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-16T09:42:16.370 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-16T09:42:16.370 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-16T09:42:16.371 INFO:teuthology.orchestra.run.smithi158.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-16T09:42:16.371 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-16T09:42:16.371 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-16T09:42:16.371 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-16T09:42:16.371 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-16T09:42:16.372 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-16T09:42:16.372 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-16T09:42:16.372 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-16T09:42:16.372 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-16T09:42:16.373 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T09:42:16.373 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T09:42:16.373 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T09:42:16.373 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-16T09:42:16.373 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-16T09:42:16.374 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-16T09:42:16.374 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-16T09:42:16.374 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-16T09:42:16.374 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.374 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.375 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.375 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-16T09:42:16.375 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-16T09:42:16.375 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-16T09:42:16.376 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-16T09:42:16.376 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-16T09:42:16.376 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-16T09:42:16.376 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-16T09:42:16.377 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-16T09:42:16.377 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-16T09:42:16.377 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-16T09:42:16.377 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-16T09:42:16.377 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-16T09:42:16.378 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-16T09:42:16.378 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-16T09:42:16.378 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-16T09:42:16.378 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-16T09:42:16.378 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-16T09:42:16.379 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-16T09:42:16.379 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-16T09:42:16.379 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-16T09:42:16.379 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-16T09:42:16.379 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-16T09:42:16.380 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend-2.6-1.el8.noarch 2023-10-16T09:42:16.380 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-16T09:42:16.380 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-16T09:42:16.380 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-16T09:42:16.380 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-16T09:42:16.381 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-16T09:42:16.381 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-16T09:42:16.381 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.381 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.381 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-16T09:42:16.382 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-16T09:42:16.382 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.382 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-16T09:42:16.383 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-16T09:42:16.383 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-16T09:42:16.383 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-16T09:42:16.383 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-16T09:42:16.383 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-16T09:42:16.384 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-16T09:42:16.384 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-16T09:42:16.384 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-16T09:42:16.384 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-16T09:42:16.384 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-16T09:42:16.385 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-16T09:42:16.385 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-16T09:42:16.385 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.385 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.385 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:16.385 INFO:teuthology.orchestra.run.smithi158.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-16T09:42:16.386 INFO:teuthology.orchestra.run.smithi158.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-16T09:42:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-16T09:42:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-16T09:42:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-16T09:42:16.391 INFO:teuthology.orchestra.run.smithi158.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-16T09:42:16.391 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:16.391 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-16T09:42:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 30/134 2023-10-16T09:42:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 31/134 2023-10-16T09:42:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/134 2023-10-16T09:42:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/134 2023-10-16T09:42:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 34/134 2023-10-16T09:42:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/134 2023-10-16T09:42:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 36/134 2023-10-16T09:42:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 37/134 2023-10-16T09:42:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 38/134 2023-10-16T09:42:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-16T09:42:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-16T09:42:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-16T09:42:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-16T09:42:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-16T09:42:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-16T09:42:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-16T09:42:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-16T09:42:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-16T09:42:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-16T09:42:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-16T09:42:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-16T09:42:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-16T09:42:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-16T09:42:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-16T09:42:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-16T09:42:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-16T09:42:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-16T09:42:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-16T09:42:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-16T09:42:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-16T09:42:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-16T09:42:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-16T09:42:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-16T09:42:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-16T09:42:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-16T09:42:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-16T09:42:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-16T09:42:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-16T09:42:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-16T09:42:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-16T09:42:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-16T09:42:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-16T09:42:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-16T09:42:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-16T09:42:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-16T09:42:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-16T09:42:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-16T09:42:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-16T09:42:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-16T09:42:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-16T09:42:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-16T09:42:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-16T09:42:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-16T09:42:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-16T09:42:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-16T09:42:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-16T09:42:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-16T09:42:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-16T09:42:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-16T09:42:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-16T09:42:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-16T09:42:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-16T09:42:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-16T09:42:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-16T09:42:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-16T09:42:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-16T09:42:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-16T09:42:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-16T09:42:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-16T09:42:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-16T09:42:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-16T09:42:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-16T09:42:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-16T09:42:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-16T09:42:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-16T09:42:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-16T09:42:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-16T09:42:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-16T09:42:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-16T09:42:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-16T09:42:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-16T09:42:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-16T09:42:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-16T09:42:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-16T09:42:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-16T09:42:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-16T09:42:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-16T09:42:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-16T09:42:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-16T09:42:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-16T09:42:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-16T09:42:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-16T09:42:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-16T09:42:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-16T09:42:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-16T09:42:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-16T09:42:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-16T09:42:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-16T09:42:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T09:42:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 129/134 2023-10-16T09:42:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-16T09:42:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 131/134 2023-10-16T09:42:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T09:42:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-16T09:42:16.582 DEBUG:teuthology.parallel:result is None 2023-10-16T09:42:19.095 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T09:42:19.095 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Oct 2023 09:40:26 AM UTC. 2023-10-16T09:42:21.745 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-16T09:42:21.968 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:21.969 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2023-10-16T09:42:21.969 INFO:teuthology.orchestra.run.smithi172.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-16T09:42:21.969 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.969 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.970 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-16T09:42:21.970 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-16T09:42:21.970 INFO:teuthology.orchestra.run.smithi172.stdout: automake-1.16.1-7.el8.noarch 2023-10-16T09:42:21.970 INFO:teuthology.orchestra.run.smithi172.stdout: bison-3.0.4-10.el8.x86_64 2023-10-16T09:42:21.970 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.971 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.971 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.971 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.971 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.971 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.972 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.972 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.972 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.972 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.972 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.973 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.973 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.973 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.973 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.973 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.974 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.974 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.974 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.974 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.974 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.974 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.975 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T09:42:21.975 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-16T09:42:21.975 INFO:teuthology.orchestra.run.smithi172.stdout: flex-2.6.1-9.el8.x86_64 2023-10-16T09:42:21.975 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-16T09:42:21.975 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-16T09:42:21.976 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-3.el8.x86_64 2023-10-16T09:42:21.976 INFO:teuthology.orchestra.run.smithi172.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-16T09:42:21.976 INFO:teuthology.orchestra.run.smithi172.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-16T09:42:21.976 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-16T09:42:21.977 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.977 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.977 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.977 INFO:teuthology.orchestra.run.smithi172.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-16T09:42:21.977 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-16T09:42:21.978 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-16T09:42:21.978 INFO:teuthology.orchestra.run.smithi172.stdout: libpq-13.5-1.el8.x86_64 2023-10-16T09:42:21.978 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-16T09:42:21.978 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-16T09:42:21.979 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.979 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.979 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-16T09:42:21.979 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.979 INFO:teuthology.orchestra.run.smithi172.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-16T09:42:21.980 INFO:teuthology.orchestra.run.smithi172.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-16T09:42:21.980 INFO:teuthology.orchestra.run.smithi172.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-16T09:42:21.980 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-16T09:42:21.980 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-16T09:42:21.981 INFO:teuthology.orchestra.run.smithi172.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-16T09:42:21.981 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-16T09:42:21.981 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-16T09:42:21.981 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-16T09:42:21.981 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-16T09:42:21.981 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-16T09:42:21.982 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-16T09:42:21.982 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-16T09:42:21.982 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-16T09:42:21.982 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T09:42:21.982 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T09:42:21.983 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T09:42:21.983 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-16T09:42:21.983 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-16T09:42:21.983 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-16T09:42:21.983 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-16T09:42:21.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-16T09:42:21.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-16T09:42:21.985 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-16T09:42:21.985 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-16T09:42:21.985 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-16T09:42:21.985 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-16T09:42:21.985 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-16T09:42:21.986 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-16T09:42:21.986 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-16T09:42:21.986 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-16T09:42:21.986 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-16T09:42:21.986 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-16T09:42:21.987 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-16T09:42:21.987 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-16T09:42:21.987 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-16T09:42:21.987 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-16T09:42:21.987 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-16T09:42:21.987 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-16T09:42:21.988 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-16T09:42:21.988 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-16T09:42:21.988 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-16T09:42:21.988 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-16T09:42:21.988 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-16T09:42:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-2.6-1.el8.noarch 2023-10-16T09:42:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-16T09:42:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-16T09:42:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-16T09:42:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-16T09:42:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-16T09:42:21.990 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-16T09:42:21.990 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.990 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.990 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-16T09:42:21.990 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-16T09:42:21.991 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.991 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-16T09:42:21.991 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-16T09:42:21.991 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-16T09:42:21.991 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-16T09:42:21.992 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-16T09:42:21.992 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-16T09:42:21.992 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-16T09:42:21.992 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-16T09:42:21.992 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-16T09:42:21.993 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-16T09:42:21.993 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-16T09:42:21.993 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-16T09:42:21.993 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-16T09:42:21.993 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.994 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.994 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T09:42:21.994 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-16T09:42:21.994 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-16T09:42:21.994 INFO:teuthology.orchestra.run.smithi172.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-16T09:42:21.995 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-16T09:42:21.995 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-16T09:42:21.995 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-16T09:42:21.996 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:21.996 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-16T09:42:22.167 DEBUG:teuthology.parallel:result is None 2023-10-16T09:42:22.168 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T09:42:22.309 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T09:42:22.338 INFO:teuthology.orchestra.run.smithi158.stdout:18.0.0-6675.gc7471068.el8 2023-10-16T09:42:22.339 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el8 2023-10-16T09:42:22.339 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-16T09:42:22.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T09:42:22.481 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T09:42:22.504 INFO:teuthology.orchestra.run.smithi172.stdout:18.0.0-6675.gc7471068.el8 2023-10-16T09:42:22.505 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el8 2023-10-16T09:42:22.505 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-16T09:42:22.506 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-16T09:42:22.506 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:22.507 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T09:42:22.556 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:22.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T09:42:22.597 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-16T09:42:22.598 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:22.598 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T09:42:22.628 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T09:42:22.698 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:22.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T09:42:22.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T09:42:22.792 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-16T09:42:22.792 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:22.793 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T09:42:22.821 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T09:42:22.888 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:22.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T09:42:22.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T09:42:22.982 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-16T09:42:22.983 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:22.983 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T09:42:23.010 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T09:42:23.077 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:23.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T09:42:23.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T09:42:23.170 INFO:teuthology.run_tasks:Running task ceph... 2023-10-16T09:42:23.235 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-16T09:42:23.236 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /var/log/ceph 2023-10-16T09:42:23.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2023-10-16T09:42:23.265 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-16T09:42:23.265 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T09:42:23.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T09:42:23.330 INFO:tasks.ceph:Creating extra log directories... 2023-10-16T09:42:23.331 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T09:42:23.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T09:42:23.404 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-16T09:42:23.404 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-16T09:42:23.405 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-16T09:42:23.405 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:23.405 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T09:42:23.453 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T09:42:23.578 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T09:42:23.605 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:23.605 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T09:42:23.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T09:42:23.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T09:42:23.801 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-16T09:42:23.802 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-16T09:42:23.802 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966', 'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7428966', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'mgr-snap_schedule-support-subvol-and-group-arguments', 'sha1': '7b3d12460841f3cbe9400ba1b68e3e20957ff531'}}, 'owner': 'mchangir', 'priority': 25, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 127, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'mgr-snap_schedule-support-subvol-and-group-arguments', 'suite_path': '/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mchangir/ceph.git', 'suite_sha1': '7b3d12460841f3cbe9400ba1b68e3e20957ff531', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi158.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_09:20:41', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-16T09:42:23.803 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T09:42:23.806 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T09:42:23.853 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T09:42:23.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T09:42:23.920 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:23.920 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T09:42:23.944 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T09:42:23.944 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_1 2023-10-16T09:42:24.002 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T09:42:24.002 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:24.002 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 390775 Links: 1 2023-10-16T09:42:24.003 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:24.004 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:24.004 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-16 09:42:10.307681121 +0000 2023-10-16T09:42:24.004 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-16 09:39:39.778263534 +0000 2023-10-16T09:42:24.004 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-16 09:39:39.778263534 +0000 2023-10-16T09:42:24.005 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-16T09:42:24.005 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T09:42:24.067 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-16T09:42:24.068 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-16T09:42:24.068 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.00013606 s, 3.8 MB/s 2023-10-16T09:42:24.069 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T09:42:24.126 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_2 2023-10-16T09:42:24.183 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T09:42:24.184 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:24.184 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 393411 Links: 1 2023-10-16T09:42:24.184 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:24.184 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:24.185 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-16 09:42:10.307681121 +0000 2023-10-16T09:42:24.185 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-16 09:39:40.274258302 +0000 2023-10-16T09:42:24.185 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-16 09:39:40.274258302 +0000 2023-10-16T09:42:24.185 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-16T09:42:24.186 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T09:42:24.248 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-16T09:42:24.249 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-16T09:42:24.249 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.00012704 s, 4.0 MB/s 2023-10-16T09:42:24.250 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T09:42:24.307 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_3 2023-10-16T09:42:24.364 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T09:42:24.364 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:24.364 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 374701 Links: 1 2023-10-16T09:42:24.365 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:24.365 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:24.365 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-16 09:42:10.307681121 +0000 2023-10-16T09:42:24.365 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-16 09:39:40.787252889 +0000 2023-10-16T09:42:24.365 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-16 09:39:40.787252889 +0000 2023-10-16T09:42:24.366 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-16T09:42:24.366 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T09:42:24.429 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-16T09:42:24.429 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-16T09:42:24.430 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 9.4497e-05 s, 5.4 MB/s 2023-10-16T09:42:24.431 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T09:42:24.488 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_4 2023-10-16T09:42:24.545 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T09:42:24.546 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:24.546 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 393193 Links: 1 2023-10-16T09:42:24.546 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:24.546 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:24.547 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-16 09:42:10.307681121 +0000 2023-10-16T09:42:24.547 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-16 09:39:41.285247635 +0000 2023-10-16T09:42:24.547 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-16 09:39:41.285247635 +0000 2023-10-16T09:42:24.547 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-16T09:42:24.548 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T09:42:24.610 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-16T09:42:24.610 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-16T09:42:24.610 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000107533 s, 4.8 MB/s 2023-10-16T09:42:24.612 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T09:42:24.669 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:24.670 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:24.670 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T09:42:24.689 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T09:42:24.689 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2023-10-16T09:42:24.748 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T09:42:24.748 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:24.748 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 391322 Links: 1 2023-10-16T09:42:24.749 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:24.749 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:24.749 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-16 09:42:16.039969706 +0000 2023-10-16T09:42:24.749 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-16 09:39:39.776710701 +0000 2023-10-16T09:42:24.750 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-16 09:39:39.776710701 +0000 2023-10-16T09:42:24.750 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-16T09:42:24.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T09:42:24.816 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-16T09:42:24.816 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-16T09:42:24.817 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000259594 s, 2.0 MB/s 2023-10-16T09:42:24.818 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T09:42:24.876 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2023-10-16T09:42:24.934 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T09:42:24.934 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:24.935 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 393759 Links: 1 2023-10-16T09:42:24.935 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:24.935 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:24.935 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-16 09:42:16.039969706 +0000 2023-10-16T09:42:24.935 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-16 09:39:40.288698403 +0000 2023-10-16T09:42:24.936 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-16 09:39:40.288698403 +0000 2023-10-16T09:42:24.936 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-16T09:42:24.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T09:42:25.001 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-16T09:42:25.001 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-16T09:42:25.001 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000256153 s, 2.0 MB/s 2023-10-16T09:42:25.002 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T09:42:25.060 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2023-10-16T09:42:25.117 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T09:42:25.118 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:25.118 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 393965 Links: 1 2023-10-16T09:42:25.118 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:25.119 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:25.120 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-16 09:42:16.039969706 +0000 2023-10-16T09:42:25.120 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-16 09:39:40.792686298 +0000 2023-10-16T09:42:25.120 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-16 09:39:40.792686298 +0000 2023-10-16T09:42:25.120 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-16T09:42:25.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T09:42:25.182 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-16T09:42:25.183 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-16T09:42:25.183 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000238179 s, 2.1 MB/s 2023-10-16T09:42:25.184 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T09:42:25.242 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2023-10-16T09:42:25.298 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T09:42:25.298 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T09:42:25.298 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 395556 Links: 1 2023-10-16T09:42:25.299 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T09:42:25.299 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T09:42:25.299 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-16 09:42:16.039969706 +0000 2023-10-16T09:42:25.299 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-16 09:39:41.298674146 +0000 2023-10-16T09:42:25.299 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-16 09:39:41.298674146 +0000 2023-10-16T09:42:25.300 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-16T09:42:25.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T09:42:25.364 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-16T09:42:25.364 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-16T09:42:25.365 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000206801 s, 2.5 MB/s 2023-10-16T09:42:25.366 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T09:42:25.423 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:25.423 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T09:42:25.423 INFO:tasks.ceph:Generating config... 2023-10-16T09:42:25.425 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-16T09:42:25.425 INFO:tasks.ceph:[client] debug client = 20 2023-10-16T09:42:25.425 INFO:tasks.ceph:[client] debug ms = 1 2023-10-16T09:42:25.426 INFO:tasks.ceph:[client] pid file = 2023-10-16T09:42:25.426 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-16T09:42:25.426 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-16T09:42:25.426 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-16T09:42:25.427 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-16T09:42:25.427 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-16T09:42:25.427 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-16T09:42:25.427 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-16T09:42:25.428 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-16T09:42:25.428 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-16T09:42:25.428 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-16T09:42:25.428 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-16T09:42:25.429 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-16T09:42:25.429 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-16T09:42:25.429 INFO:tasks.ceph:[mgr] debug client = 20 2023-10-16T09:42:25.430 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-10-16T09:42:25.430 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-16T09:42:25.430 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-16T09:42:25.430 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-16T09:42:25.431 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-16T09:42:25.431 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-16T09:42:25.431 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-16T09:42:25.432 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-16T09:42:25.432 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-16T09:42:25.432 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-16T09:42:25.432 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-16T09:42:25.433 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-16T09:42:25.433 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-16T09:42:25.433 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-16T09:42:25.433 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-16T09:42:25.434 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-16T09:42:25.434 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-16T09:42:25.434 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-16T09:42:25.435 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-16T09:42:25.435 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-16T09:42:25.435 INFO:tasks.ceph:Setting up mon.a... 2023-10-16T09:42:25.436 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-16T09:42:25.491 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/ceph.keyring 2023-10-16T09:42:25.494 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-16T09:42:25.542 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T09:42:25.569 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.158'), ('mon.b', '172.21.15.172'), ('mon.c', '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]')] 2023-10-16T09:42:25.570 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.158,172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-16T09:42:25.571 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:25.571 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T09:42:25.626 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.158 --add b 172.21.15.172 --addv c '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:25.715 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:25.716 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: generated fsid 7cf363df-9988-44b1-be05-df2a6bf24f2d 2023-10-16T09:42:25.716 INFO:teuthology.orchestra.run.smithi158.stdout:setting min_mon_release = pacific 2023-10-16T09:42:25.716 INFO:teuthology.orchestra.run.smithi158.stdout:epoch 0 2023-10-16T09:42:25.716 INFO:teuthology.orchestra.run.smithi158.stdout:fsid 7cf363df-9988-44b1-be05-df2a6bf24f2d 2023-10-16T09:42:25.716 INFO:teuthology.orchestra.run.smithi158.stdout:last_changed 2023-10-16T09:42:25.715000+0000 2023-10-16T09:42:25.717 INFO:teuthology.orchestra.run.smithi158.stdout:created 2023-10-16T09:42:25.715000+0000 2023-10-16T09:42:25.717 INFO:teuthology.orchestra.run.smithi158.stdout:min_mon_release 16 (pacific) 2023-10-16T09:42:25.717 INFO:teuthology.orchestra.run.smithi158.stdout:election_strategy: 1 2023-10-16T09:42:25.717 INFO:teuthology.orchestra.run.smithi158.stdout:0: [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon.a 2023-10-16T09:42:25.717 INFO:teuthology.orchestra.run.smithi158.stdout:1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2023-10-16T09:42:25.717 INFO:teuthology.orchestra.run.smithi158.stdout:2: [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] mon.c 2023-10-16T09:42:25.718 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-16T09:42:25.719 DEBUG:teuthology.orchestra.run.smithi158:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T09:42:25.735 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7cf363df-9988-44b1-be05-df2a6bf24f2d... 2023-10-16T09:42:25.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T09:42:25.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T09:42:25.820 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2023-10-16T09:42:25.820 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2023-10-16T09:42:25.820 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T09:42:25.821 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2023-10-16T09:42:25.821 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.821 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2023-10-16T09:42:25.821 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.821 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2023-10-16T09:42:25.822 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.822 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2023-10-16T09:42:25.822 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2023-10-16T09:42:25.822 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.822 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2023-10-16T09:42:25.823 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2023-10-16T09:42:25.823 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.823 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T09:42:25.823 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2023-10-16T09:42:25.823 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.823 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T09:42:25.824 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T09:42:25.824 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.824 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2023-10-16T09:42:25.824 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.824 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2023-10-16T09:42:25.825 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2023-10-16T09:42:25.825 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2023-10-16T09:42:25.825 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2023-10-16T09:42:25.825 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2023-10-16T09:42:25.825 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2023-10-16T09:42:25.825 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2023-10-16T09:42:25.826 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T09:42:25.826 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T09:42:25.826 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2023-10-16T09:42:25.826 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.826 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T09:42:25.827 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.827 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2023-10-16T09:42:25.827 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.827 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2023-10-16T09:42:25.827 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.828 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2023-10-16T09:42:25.829 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2023-10-16T09:42:25.829 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2023-10-16T09:42:25.830 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.158,172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790]" 2023-10-16T09:42:25.830 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on skipped message = False 2023-10-16T09:42:25.830 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = 7cf363df-9988-44b1-be05-df2a6bf24f2d 2023-10-16T09:42:25.830 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.830 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2023-10-16T09:42:25.831 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2023-10-16T09:42:25.831 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.831 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2023-10-16T09:42:25.831 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2023-10-16T09:42:25.831 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2023-10-16T09:42:25.832 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.832 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2023-10-16T09:42:25.832 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2023-10-16T09:42:25.832 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.832 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2023-10-16T09:42:25.832 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2023-10-16T09:42:25.833 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2023-10-16T09:42:25.833 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.833 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2023-10-16T09:42:25.833 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2023-10-16T09:42:25.833 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.834 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2023-10-16T09:42:25.834 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.834 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2023-10-16T09:42:25.834 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.834 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2023-10-16T09:42:25.835 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.835 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2023-10-16T09:42:25.835 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2023-10-16T09:42:25.835 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.835 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2023-10-16T09:42:25.836 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2023-10-16T09:42:25.836 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2023-10-16T09:42:25.836 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2023-10-16T09:42:25.836 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 20 2023-10-16T09:42:25.836 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 20 2023-10-16T09:42:25.837 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-16T09:42:25.837 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2023-10-16T09:42:25.837 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 10 2023-10-16T09:42:25.837 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T09:42:25.837 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2023-10-16T09:42:25.837 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2023-10-16T09:42:25.838 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2023-10-16T09:42:25.838 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2023-10-16T09:42:25.838 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2023-10-16T09:42:25.838 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.838 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2023-10-16T09:42:25.839 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-16T09:42:25.839 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2023-10-16T09:42:25.839 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-10-16T09:42:25.839 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-10-16T09:42:25.839 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-10-16T09:42:25.840 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-10-16T09:42:25.840 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2023-10-16T09:42:25.840 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2023-10-16T09:42:25.840 INFO:teuthology.orchestra.run.smithi158.stdout: debug finisher = 20 2023-10-16T09:42:25.840 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.840 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2023-10-16T09:42:25.841 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-16T09:42:25.841 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-10-16T09:42:25.841 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2023-10-16T09:42:25.841 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-10-16T09:42:25.841 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2023-10-16T09:42:25.841 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2023-10-16T09:42:25.842 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-10-16T09:42:25.842 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2023-10-16T09:42:25.842 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2023-10-16T09:42:25.842 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-10-16T09:42:25.842 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.843 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T09:42:25.843 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T09:42:25.843 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T09:42:25.843 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.843 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T09:42:25.844 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T09:42:25.844 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T09:42:25.844 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.844 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2023-10-16T09:42:25.844 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2023-10-16T09:42:25.844 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.845 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2023-10-16T09:42:25.845 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2023-10-16T09:42:25.845 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:42:25.845 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2023-10-16T09:42:25.845 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2023-10-16T09:42:25.846 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2023-10-16T09:42:25.846 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2023-10-16T09:42:25.846 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T09:42:25.846 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T09:42:25.846 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2023-10-16T09:42:25.847 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2023-10-16T09:42:25.847 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-16T09:42:25.847 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2023-10-16T09:42:25.847 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2023-10-16T09:42:25.847 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2023-10-16T09:42:25.847 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2023-10-16T09:42:25.848 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2023-10-16T09:42:25.848 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2023-10-16T09:42:25.848 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2023-10-16T09:42:25.848 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2023-10-16T09:42:25.848 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2023-10-16T09:42:25.849 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2023-10-16T09:42:25.849 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2023-10-16T09:42:25.849 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2023-10-16T09:42:25.849 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2023-10-16T09:42:25.849 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-16T09:42:25.850 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2023-10-16T09:42:25.850 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2023-10-16T09:42:25.850 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2023-10-16T09:42:25.850 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2023-10-16T09:42:25.850 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2023-10-16T09:42:25.850 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2023-10-16T09:42:25.851 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2023-10-16T09:42:25.852 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-10-16T09:42:25.853 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2023-10-16T09:42:25.853 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T09:42:25.853 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2023-10-16T09:42:25.853 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.853 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2023-10-16T09:42:25.854 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.854 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2023-10-16T09:42:25.854 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.854 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2023-10-16T09:42:25.854 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2023-10-16T09:42:25.855 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.855 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2023-10-16T09:42:25.855 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2023-10-16T09:42:25.855 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.855 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T09:42:25.856 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2023-10-16T09:42:25.856 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.856 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T09:42:25.856 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T09:42:25.856 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.857 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2023-10-16T09:42:25.857 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.857 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2023-10-16T09:42:25.857 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2023-10-16T09:42:25.857 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2023-10-16T09:42:25.857 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2023-10-16T09:42:25.858 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2023-10-16T09:42:25.858 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2023-10-16T09:42:25.858 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2023-10-16T09:42:25.858 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T09:42:25.858 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T09:42:25.859 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2023-10-16T09:42:25.859 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.859 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T09:42:25.859 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.859 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2023-10-16T09:42:25.860 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.860 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2023-10-16T09:42:25.860 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.860 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2023-10-16T09:42:25.860 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2023-10-16T09:42:25.860 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2023-10-16T09:42:25.861 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "172.21.15.158,172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790]" 2023-10-16T09:42:25.861 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on skipped message = False 2023-10-16T09:42:25.861 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 7cf363df-9988-44b1-be05-df2a6bf24f2d 2023-10-16T09:42:25.861 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.861 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2023-10-16T09:42:25.862 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2023-10-16T09:42:25.862 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.862 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2023-10-16T09:42:25.862 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2023-10-16T09:42:25.862 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2023-10-16T09:42:25.863 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.863 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2023-10-16T09:42:25.863 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2023-10-16T09:42:25.863 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.863 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2023-10-16T09:42:25.863 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2023-10-16T09:42:25.864 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2023-10-16T09:42:25.864 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.864 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2023-10-16T09:42:25.864 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2023-10-16T09:42:25.864 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.865 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2023-10-16T09:42:25.865 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.865 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2023-10-16T09:42:25.865 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.865 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2023-10-16T09:42:25.866 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.866 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2023-10-16T09:42:25.866 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2023-10-16T09:42:25.866 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.866 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2023-10-16T09:42:25.867 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2023-10-16T09:42:25.867 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2023-10-16T09:42:25.867 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2023-10-16T09:42:25.867 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2023-10-16T09:42:25.867 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2023-10-16T09:42:25.867 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-16T09:42:25.868 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2023-10-16T09:42:25.868 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2023-10-16T09:42:25.868 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T09:42:25.868 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2023-10-16T09:42:25.868 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2023-10-16T09:42:25.869 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2023-10-16T09:42:25.869 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2023-10-16T09:42:25.869 INFO:teuthology.orchestra.run.smithi172.stdout: osd op complaint time = 180 2023-10-16T09:42:25.869 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.869 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2023-10-16T09:42:25.870 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-16T09:42:25.870 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2023-10-16T09:42:25.870 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-16T09:42:25.870 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-16T09:42:25.870 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-16T09:42:25.871 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-16T09:42:25.871 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2023-10-16T09:42:25.871 INFO:teuthology.orchestra.run.smithi172.stdout: debug client = 20 2023-10-16T09:42:25.871 INFO:teuthology.orchestra.run.smithi172.stdout: debug finisher = 20 2023-10-16T09:42:25.871 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.871 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2023-10-16T09:42:25.872 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-16T09:42:25.872 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-16T09:42:25.872 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2023-10-16T09:42:25.872 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-16T09:42:25.872 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2023-10-16T09:42:25.873 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2023-10-16T09:42:25.873 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-16T09:42:25.873 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2023-10-16T09:42:25.873 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2023-10-16T09:42:25.873 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-16T09:42:25.874 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.874 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T09:42:25.874 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T09:42:25.874 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T09:42:25.874 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.875 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T09:42:25.875 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T09:42:25.875 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T09:42:25.875 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.875 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2023-10-16T09:42:25.876 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2023-10-16T09:42:25.876 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.876 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2023-10-16T09:42:25.876 INFO:teuthology.orchestra.run.smithi172.stdout: mon op complaint time = 120 2023-10-16T09:42:25.876 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-16T09:42:25.876 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2023-10-16T09:42:25.877 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2023-10-16T09:42:25.877 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2023-10-16T09:42:25.877 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2023-10-16T09:42:25.877 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T09:42:25.877 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T09:42:25.878 INFO:teuthology.orchestra.run.smithi172.stdout: client mount timeout = 600 2023-10-16T09:42:25.878 INFO:teuthology.orchestra.run.smithi172.stdout: debug client = 20 2023-10-16T09:42:25.878 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-16T09:42:25.878 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = "" 2023-10-16T09:42:25.878 INFO:teuthology.orchestra.run.smithi172.stdout: rados mon op timeout = 900 2023-10-16T09:42:25.879 INFO:teuthology.orchestra.run.smithi172.stdout: rados osd op timeout = 900 2023-10-16T09:42:25.879 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2023-10-16T09:42:25.879 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2023-10-16T09:42:25.879 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2023-10-16T09:42:25.879 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.a] 2023-10-16T09:42:25.880 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.c] 2023-10-16T09:42:25.880 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.b] 2023-10-16T09:42:25.880 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.d] 2023-10-16T09:42:25.880 INFO:teuthology.orchestra.run.smithi172.stdout:[mds] 2023-10-16T09:42:25.880 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds = 20 2023-10-16T09:42:25.880 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds balancer = 20 2023-10-16T09:42:25.881 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-16T09:42:25.881 INFO:teuthology.orchestra.run.smithi172.stdout: mds debug frag = True 2023-10-16T09:42:25.881 INFO:teuthology.orchestra.run.smithi172.stdout: mds debug scatterstat = True 2023-10-16T09:42:25.881 INFO:teuthology.orchestra.run.smithi172.stdout: mds op complaint time = 180 2023-10-16T09:42:25.881 INFO:teuthology.orchestra.run.smithi172.stdout: mds verify scatter = True 2023-10-16T09:42:25.882 INFO:teuthology.orchestra.run.smithi172.stdout: osd op complaint time = 180 2023-10-16T09:42:25.882 INFO:teuthology.orchestra.run.smithi172.stdout: rados mon op timeout = 900 2023-10-16T09:42:25.882 INFO:teuthology.orchestra.run.smithi172.stdout: rados osd op timeout = 900 2023-10-16T09:42:25.882 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-16T09:42:25.883 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-16T09:42:25.949 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-16T09:42:25.949 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:25.950 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-16T09:42:25.966 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:25.967 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-16T09:42:26.023 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:42:26.023 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:26.023 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T09:42:26.024 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T09:42:26.100 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:26.100 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:26.157 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:42:26.157 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:26.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T09:42:26.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T09:42:26.195 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:26.195 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:26.254 INFO:tasks.ceph:Setting up mon nodes... 2023-10-16T09:42:26.255 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-16T09:42:26.255 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T09:42:26.308 INFO:teuthology.orchestra.run.smithi158.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T09:42:26.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T09:42:26.381 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T09:42:26.383 INFO:tasks.ceph:Setting up mds nodes... 2023-10-16T09:42:26.384 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T09:42:26.438 INFO:teuthology.orchestra.run.smithi158.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T09:42:26.442 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-16T09:42:26.469 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi158.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T09:42:26.565 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-16T09:42:26.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T09:42:26.645 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T09:42:26.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-16T09:42:26.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T09:42:26.768 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T09:42:26.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-16T09:42:26.796 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-16T09:42:26.797 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-16T09:42:26.841 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-16T09:42:26.854 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-16T09:42:26.897 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-16T09:42:26.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-16T09:42:26.953 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-16T09:42:26.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-16T09:42:27.048 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-16T09:42:27.062 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-16T09:42:27.063 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T09:42:27.063 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-16T09:42:27.090 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:27.090 INFO:tasks.ceph:role: osd.0 2023-10-16T09:42:27.090 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:42:27.091 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T09:42:27.156 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:27.157 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:27.157 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:27.157 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:27.157 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:27.158 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:27.158 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:27.158 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:27.158 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:27.158 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:27.179 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2023-10-16T09:42:27.180 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-10-16T09:42:27.180 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-16T09:42:27.260 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-16T09:42:27.332 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-16T09:42:27.399 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:27.400 INFO:tasks.ceph:role: osd.1 2023-10-16T09:42:27.400 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:42:27.401 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T09:42:27.466 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:27.470 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:27.470 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:27.470 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:27.470 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:27.471 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:27.471 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:27.471 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:27.471 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:27.471 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:27.489 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2023-10-16T09:42:27.490 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-10-16T09:42:27.491 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-16T09:42:27.571 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-16T09:42:27.641 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-16T09:42:27.707 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:27.707 INFO:tasks.ceph:role: osd.2 2023-10-16T09:42:27.708 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:42:27.708 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T09:42:27.773 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:27.773 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:27.774 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:27.774 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:27.774 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:27.774 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:27.774 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:27.775 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:27.775 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:27.775 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:27.796 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2023-10-16T09:42:27.798 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-10-16T09:42:27.798 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-16T09:42:27.877 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-16T09:42:27.947 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-16T09:42:28.014 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:28.014 INFO:tasks.ceph:role: osd.3 2023-10-16T09:42:28.014 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:42:28.015 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T09:42:28.080 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:28.080 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:28.081 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:28.081 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:28.081 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:28.081 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:28.082 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:28.082 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:28.082 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:28.082 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:28.102 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2023-10-16T09:42:28.104 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-10-16T09:42:28.104 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-16T09:42:28.183 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-16T09:42:28.254 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:28.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:28.595+0000 7fef15ed9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-16T09:42:28.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:28.595+0000 7fef15ed9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-16T09:42:28.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:28.595+0000 7fef15ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T09:42:28.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:28.595+0000 7fef15ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T09:42:28.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:28.595+0000 7fef15ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T09:42:28.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:28.595+0000 7fef15ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-16T09:42:31.458 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-16T09:42:31.489 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:31.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:31.578+0000 7fcb9ae9b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-16T09:42:31.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:31.578+0000 7fcb9ae9b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-16T09:42:31.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:31.579+0000 7fcb9ae9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T09:42:31.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:31.579+0000 7fcb9ae9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T09:42:31.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:31.579+0000 7fcb9ae9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T09:42:31.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:31.579+0000 7fcb9ae9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-16T09:42:34.449 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-16T09:42:34.480 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:34.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:34.568+0000 7f3b6abe5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-16T09:42:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:34.568+0000 7f3b6abe5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-16T09:42:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:34.568+0000 7f3b6abe5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T09:42:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:34.568+0000 7f3b6abe5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T09:42:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:34.568+0000 7f3b6abe5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T09:42:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:34.569+0000 7f3b6abe5640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-16T09:42:37.428 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-16T09:42:37.459 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:37.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:37.548+0000 7f7867a19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-16T09:42:37.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:37.548+0000 7f7867a19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-16T09:42:37.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:37.548+0000 7f7867a19640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T09:42:37.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:37.548+0000 7f7867a19640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T09:42:37.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:37.548+0000 7f7867a19640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T09:42:37.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:37.548+0000 7f7867a19640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-16T09:42:40.420 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-16T09:42:40.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-16T09:42:40.485 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:40.486 INFO:tasks.ceph:role: osd.4 2023-10-16T09:42:40.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:42:40.486 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T09:42:40.563 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:40.564 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:40.564 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:40.564 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:40.564 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:40.564 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:40.565 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:40.565 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:40.565 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:40.565 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:40.987 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-16T09:42:40.989 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-16T09:42:40.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-16T09:42:41.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-16T09:42:41.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-16T09:42:41.266 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:41.266 INFO:tasks.ceph:role: osd.5 2023-10-16T09:42:41.267 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:42:41.267 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T09:42:41.342 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:41.343 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:41.344 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:41.344 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:41.344 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:41.771 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-16T09:42:41.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-16T09:42:41.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-16T09:42:41.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-16T09:42:41.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-16T09:42:41.986 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:41.986 INFO:tasks.ceph:role: osd.6 2023-10-16T09:42:41.986 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:42:41.987 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T09:42:42.062 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:42.063 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:42.063 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:42.064 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:42.064 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:42.064 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:42.511 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-16T09:42:42.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-16T09:42:42.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-16T09:42:42.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-16T09:42:42.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-16T09:42:42.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T09:42:42.808 INFO:tasks.ceph:role: osd.7 2023-10-16T09:42:42.808 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:42:42.808 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T09:42:42.884 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T09:42:42.885 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T09:42:42.885 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T09:42:42.885 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T09:42:42.885 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T09:42:42.886 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-16T09:42:42.886 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T09:42:42.886 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T09:42:42.886 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T09:42:42.886 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T09:42:43.328 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-16T09:42:43.330 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-16T09:42:43.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-16T09:42:43.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-16T09:42:43.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:43.635 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:43.633+0000 7fe47eb46640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-16T09:42:43.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:43.633+0000 7fe47eb46640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-16T09:42:43.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:43.633+0000 7fe47eb46640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T09:42:43.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:43.633+0000 7fe47eb46640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T09:42:43.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:43.633+0000 7fe47eb46640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T09:42:43.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:43.634+0000 7fe47eb46640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-16T09:42:46.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-16T09:42:46.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:46.627 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:46.624+0000 7ff6e32c2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-16T09:42:46.627 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:46.625+0000 7ff6e32c2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-16T09:42:46.627 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:46.625+0000 7ff6e32c2640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T09:42:46.627 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:46.625+0000 7ff6e32c2640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T09:42:46.628 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:46.625+0000 7ff6e32c2640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T09:42:46.628 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:46.625+0000 7ff6e32c2640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-16T09:42:49.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-16T09:42:49.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:49.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:49.610+0000 7feba384f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-16T09:42:49.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:49.611+0000 7feba384f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-16T09:42:49.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:49.611+0000 7feba384f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T09:42:49.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:49.611+0000 7feba384f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T09:42:49.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:49.611+0000 7feba384f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T09:42:49.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:49.611+0000 7feba384f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-16T09:42:52.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-16T09:42:52.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:52.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:52.607+0000 7f58016ab640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-16T09:42:52.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:52.608+0000 7f58016ab640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-16T09:42:52.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:52.608+0000 7f58016ab640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T09:42:52.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:52.608+0000 7f58016ab640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T09:42:52.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:52.608+0000 7f58016ab640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T09:42:52.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:42:52.608+0000 7f58016ab640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-16T09:42:53.805 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:42:53.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:42:55.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-16T09:42:55.525 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-16T09:42:55.526 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:55.526 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-16T09:42:55.558 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:55.558 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-16T09:42:55.628 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:55.628 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-16T09:42:55.697 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:55.697 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-16T09:42:55.765 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:55.766 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-16T09:42:55.834 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:55.834 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-16T09:42:55.901 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:55.902 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-16T09:42:55.968 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:55.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-16T09:42:56.000 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-16T09:42:56.071 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-16T09:42:56.142 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-16T09:42:56.211 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-16T09:42:56.280 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-16T09:42:56.348 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-16T09:42:56.416 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:56.416 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-16T09:42:56.436 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:42:56.437 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-16T09:42:56.496 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.496 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-16T09:42:56.513 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:42:56.514 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-16T09:42:56.570 INFO:tasks.ceph:Adding keys to all mons... 2023-10-16T09:42:56.571 DEBUG:teuthology.orchestra.run.smithi158:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T09:42:56.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T09:42:56.614 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr.y] 2023-10-16T09:42:56.614 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA== 2023-10-16T09:42:56.614 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2023-10-16T09:42:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w== 2023-10-16T09:42:56.615 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2023-10-16T09:42:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A== 2023-10-16T09:42:56.615 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.0] 2023-10-16T09:42:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ== 2023-10-16T09:42:56.616 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.1] 2023-10-16T09:42:56.616 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew== 2023-10-16T09:42:56.616 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.2] 2023-10-16T09:42:56.616 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw== 2023-10-16T09:42:56.616 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.3] 2023-10-16T09:42:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w== 2023-10-16T09:42:56.617 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr.x] 2023-10-16T09:42:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA== 2023-10-16T09:42:56.617 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2023-10-16T09:42:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw== 2023-10-16T09:42:56.618 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2023-10-16T09:42:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A== 2023-10-16T09:42:56.618 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.4] 2023-10-16T09:42:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ== 2023-10-16T09:42:56.618 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.5] 2023-10-16T09:42:56.619 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA== 2023-10-16T09:42:56.619 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.6] 2023-10-16T09:42:56.619 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg== 2023-10-16T09:42:56.619 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.7] 2023-10-16T09:42:56.619 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw== 2023-10-16T09:42:56.620 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T09:42:56.620 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T09:42:56.620 INFO:teuthology.orchestra.run.smithi158.stdout:[client.2] 2023-10-16T09:42:56.620 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ== 2023-10-16T09:42:56.620 INFO:teuthology.orchestra.run.smithi158.stdout:[client.1] 2023-10-16T09:42:56.621 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ== 2023-10-16T09:42:56.621 INFO:teuthology.orchestra.run.smithi158.stdout:[client.3] 2023-10-16T09:42:56.621 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A== 2023-10-16T09:42:56.638 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2023-10-16T09:42:56.638 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA== 2023-10-16T09:42:56.638 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.a] 2023-10-16T09:42:56.638 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w== 2023-10-16T09:42:56.639 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.c] 2023-10-16T09:42:56.639 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A== 2023-10-16T09:42:56.639 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-10-16T09:42:56.639 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ== 2023-10-16T09:42:56.639 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-10-16T09:42:56.640 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew== 2023-10-16T09:42:56.640 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-10-16T09:42:56.640 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw== 2023-10-16T09:42:56.640 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2023-10-16T09:42:56.640 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w== 2023-10-16T09:42:56.641 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2023-10-16T09:42:56.641 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA== 2023-10-16T09:42:56.641 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.b] 2023-10-16T09:42:56.641 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw== 2023-10-16T09:42:56.641 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.d] 2023-10-16T09:42:56.642 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A== 2023-10-16T09:42:56.642 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2023-10-16T09:42:56.642 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ== 2023-10-16T09:42:56.642 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2023-10-16T09:42:56.642 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA== 2023-10-16T09:42:56.643 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2023-10-16T09:42:56.643 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg== 2023-10-16T09:42:56.643 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2023-10-16T09:42:56.643 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw== 2023-10-16T09:42:56.643 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-10-16T09:42:56.644 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T09:42:56.644 INFO:teuthology.orchestra.run.smithi172.stdout:[client.2] 2023-10-16T09:42:56.644 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ== 2023-10-16T09:42:56.644 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2023-10-16T09:42:56.644 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ== 2023-10-16T09:42:56.644 INFO:teuthology.orchestra.run.smithi172.stdout:[client.3] 2023-10-16T09:42:56.645 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A== 2023-10-16T09:42:56.646 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T09:42:56.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T09:42:56.738 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:56.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:56.798 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:56.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:56.856 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:56.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:56.912 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:56.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:56.967 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:56.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.022 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.079 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T09:42:57.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T09:42:57.134 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:57.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:57.192 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:57.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T09:42:57.250 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.306 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.361 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.418 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T09:42:57.474 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.532 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.591 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T09:42:57.708 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-16T09:42:57.709 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-16T09:42:57.735 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T09:42:57.833 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-16T09:42:57.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-16T09:42:57.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T09:42:57.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-16T09:42:58.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-16T09:42:58.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T09:42:58.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-16T09:42:58.202 DEBUG:teuthology.orchestra.run.smithi158:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:58.205 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T09:42:58.259 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-16T09:42:58.259 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-16T09:42:58.260 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-16T09:42:58.262 INFO:tasks.ceph.mon.a:Started 2023-10-16T09:42:58.262 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-16T09:42:58.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-16T09:42:58.301 INFO:tasks.ceph.mon.b:Started 2023-10-16T09:42:58.301 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-16T09:42:58.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-16T09:42:58.304 INFO:tasks.ceph.mon.c:Started 2023-10-16T09:42:58.304 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-16T09:42:58.304 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-16T09:42:58.305 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-16T09:42:58.307 INFO:tasks.ceph.mgr.y:Started 2023-10-16T09:42:58.307 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T09:42:58.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T09:42:58.310 INFO:tasks.ceph.mgr.x:Started 2023-10-16T09:42:58.310 INFO:tasks.ceph:Setting crush tunables to default 2023-10-16T09:42:58.311 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd crush tunables default 2023-10-16T09:42:58.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.432+0000 7fc5f395e700 1 Processor -- start 2023-10-16T09:42:58.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.432+0000 7fc5f395e700 1 -- start start 2023-10-16T09:42:58.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.433+0000 7fc5f395e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:42:58.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.433+0000 7fc5f395e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0c52b0 con 0x7fc5ec160d40 2023-10-16T09:42:58.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.433+0000 7fc5f395e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0ce110 con 0x7fc5ec0c49b0 2023-10-16T09:42:58.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.433+0000 7fc5f395e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec0c61e0 con 0x7fc5ec0c4e60 2023-10-16T09:42:58.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.433+0000 7fc5f16fa700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc5ec160d40 0x7fc5ec0ceee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45422/0 (socket says 172.21.15.158:45422) 2023-10-16T09:42:58.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.433+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 learned_addr learned my addr 172.21.15.158:0/2556830240 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:42:58.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.433+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:42:58.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.434+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T09:42:58.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.434+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:42:58.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.434+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:42:58.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.434+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:42:58.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.635+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:42:58.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.635+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T09:42:58.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.635+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:42:58.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.635+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:42:58.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:58.635+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:42:59.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.036+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:42:59.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.036+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T09:42:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.036+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:42:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.036+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:42:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.036+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T09:42:59.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.837+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:42:59.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.837+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T09:42:59.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.837+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:42:59.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.837+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:42:59.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:42:59.837+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T09:43:01.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 msgr2=0x7fc5ec06b2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:01.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5ec0c4e60 0x7fc5ec06b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:01.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 >> v1:172.21.15.172:6789/0 conn(0x7fc5ec0c49b0 legacy=0x7fc5ec06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:01.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 >> v1:172.21.15.158:6789/0 conn(0x7fc5ec160d40 legacy=0x7fc5ec0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:01.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:01.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:01.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d8008dc0 con 0x7fc5d8006280 2023-10-16T09:43:01.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8008f30 con 0x7fc5d80037c0 2023-10-16T09:43:01.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:01.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:01.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T09:43:01.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:01.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:01.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:01.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T09:43:01.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:01.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:01.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.434+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:01.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:01.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:01.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T09:43:01.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:01.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:43:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T09:43:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:01.635+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:43:02.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:02.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T09:43:02.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:02.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:02.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T09:43:02.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:02.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T09:43:02.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:02.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:02.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.036+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T09:43:02.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:02.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T09:43:02.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f0ef9700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:02.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:02.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T09:43:02.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:02.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T09:43:02.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:02.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:02.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:02.837+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T09:43:03.744 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:03.743+0000 7fe6de52c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:43:03.810 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:03.808+0000 7fe6de52c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:43:03.886 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:03.885+0000 7fe6de52c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:43:03.965 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:03.963+0000 7fe6de52c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:43:04.105 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:04.104+0000 7fe6de52c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:43:04.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.433+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 msgr2=0x7fc5d8005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:04.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.433+0000 7fc5eaffd700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d80037c0 0x7fc5d8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.433+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 msgr2=0x7fc5d8008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:04.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.433+0000 7fc5eaffd700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8006280 0x7fc5d8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.433+0000 7fc5eaffd700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8009330 0x7fc5d800b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:04.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.433+0000 7fc5f1efb700 1 -- 172.21.15.158:0/2556830240 reap_dead start 2023-10-16T09:43:04.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5eaffd700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5d8006760 0x7fc5d8003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:04.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0ce3f0 con 0x7fc5d8004150 2023-10-16T09:43:04.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d8008dc0 con 0x7fc5d8009330 2023-10-16T09:43:04.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5eaffd700 1 -- 172.21.15.158:0/2556830240 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d8008f30 con 0x7fc5d8006760 2023-10-16T09:43:04.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.158:3300/0 conn(0x7fc5d8009330 0x7fc5d800b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5d8006760 0x7fc5d8003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5d8006760 msgr2=0x7fc5d8003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5d8006760 0x7fc5d8003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 >> v1:172.21.15.172:6789/0 conn(0x7fc5d8004150 legacy=0x7fc5d8004600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:04.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5f16fa700 1 -- 172.21.15.158:0/2556830240 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8005f90 con 0x7fc5d8009330 2023-10-16T09:43:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8009330 0x7fc5d800b720 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8006730 tx=0x7fc5e4007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f224d8c8d9bbef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5ebfff700 1 -- 172.21.15.158:0/2556830240 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4007910 con 0x7fc5d8009330 2023-10-16T09:43:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5ebfff700 1 -- 172.21.15.158:0/2556830240 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e4007a70 con 0x7fc5d8009330 2023-10-16T09:43:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.434+0000 7fc5ebfff700 1 -- 172.21.15.158:0/2556830240 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40176a0 con 0x7fc5d8009330 2023-10-16T09:43:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 -- 172.21.15.158:0/2556830240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8009330 msgr2=0x7fc5d800b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8009330 0x7fc5d800b720 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8006730 tx=0x7fc5e4007330 comp rx=0 tx=0).stop 2023-10-16T09:43:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 -- 172.21.15.158:0/2556830240 shutdown_connections 2023-10-16T09:43:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 --2- 172.21.15.158:0/2556830240 >> v2:172.21.15.172:3300/0 conn(0x7fc5d8006760 0x7fc5d8003be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 --2- 172.21.15.158:0/2556830240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8009330 0x7fc5d800b720 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 -- 172.21.15.158:0/2556830240 >> 172.21.15.158:0/2556830240 conn(0x7fc5ec0cc5b0 msgr2=0x7fc5ec0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 -- 172.21.15.158:0/2556830240 shutdown_connections 2023-10-16T09:43:04.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.435+0000 7fc5f395e700 1 -- 172.21.15.158:0/2556830240 wait complete. 2023-10-16T09:43:04.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.436+0000 7fc5f395e700 1 Processor -- start 2023-10-16T09:43:04.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.436+0000 7fc5f395e700 1 -- start start 2023-10-16T09:43:04.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f395e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8004150 0x7fc5ec0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:04.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f395e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d8006760 0x7fc5ec0acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:04.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f395e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 0x7fc5ec155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:04.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f395e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0c83f0 con 0x7fc5d8004150 2023-10-16T09:43:04.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f395e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0ad490 con 0x7fc5d8006760 2023-10-16T09:43:04.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f395e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0ae350 con 0x7fc5d8009330 2023-10-16T09:43:04.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f16fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8004150 0x7fc5ec0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:04.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f16fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8004150 0x7fc5ec0c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45316/0 (socket says 172.21.15.158:45316) 2023-10-16T09:43:04.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f16fa700 1 -- 172.21.15.158:0/3730503184 learned_addr learned my addr 172.21.15.158:0/3730503184 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:04.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f0ef9700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d8006760 0x7fc5ec0acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:04.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 0x7fc5ec155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:04.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f1efb700 1 -- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 msgr2=0x7fc5ec155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-16T09:43:04.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.437+0000 7fc5f1efb700 1 -- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 msgr2=0x7fc5ec155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:04.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 0x7fc5ec155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:04.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f1efb700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 0x7fc5ec155eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:04.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f16fa700 1 -- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 msgr2=0x7fc5ec155eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:04.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 0x7fc5ec155eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f16fa700 1 -- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d8006760 msgr2=0x7fc5ec0acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:04.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d8006760 0x7fc5ec0acf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f16fa700 1 -- 172.21.15.158:0/3730503184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e400f040 con 0x7fc5d8004150 2023-10-16T09:43:04.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f16fa700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8004150 0x7fc5ec0c98a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4017e40 tx=0x7fc5e40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:04.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5e9ffb700 1 -- 172.21.15.158:0/3730503184 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4027070 con 0x7fc5d8004150 2023-10-16T09:43:04.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5e9ffb700 1 -- 172.21.15.158:0/3730503184 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e402d430 con 0x7fc5d8004150 2023-10-16T09:43:04.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec1563f0 con 0x7fc5d8004150 2023-10-16T09:43:04.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5e9ffb700 1 -- 172.21.15.158:0/3730503184 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4007910 con 0x7fc5d8004150 2023-10-16T09:43:04.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec0ada20 con 0x7fc5d8004150 2023-10-16T09:43:04.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5e9ffb700 1 -- 172.21.15.158:0/3730503184 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc5e4020420 con 0x7fc5d8004150 2023-10-16T09:43:04.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.438+0000 7fc5e9ffb700 1 -- 172.21.15.158:0/3730503184 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc5e40363f0 con 0x7fc5d8004150 2023-10-16T09:43:04.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.439+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec0ae630 con 0x7fc5d8004150 2023-10-16T09:43:04.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.442+0000 7fc5e9ffb700 1 -- 172.21.15.158:0/3730503184 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc5d8005f90 con 0x7fc5d8004150 2023-10-16T09:43:04.491 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:04.490+0000 7fe6de52c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:43:04.517 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:04.515+0000 7fceb733e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:43:04.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.516+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc5ec066db0 con 0x7fc5d8004150 2023-10-16T09:43:04.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.518+0000 7fc5e9ffb700 1 -- 172.21.15.158:0/3730503184 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc5e401b020 con 0x7fc5d8004150 2023-10-16T09:43:04.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8004150 msgr2=0x7fc5ec0c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:04.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8004150 0x7fc5ec0c98a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4017e40 tx=0x7fc5e40014a0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 shutdown_connections 2023-10-16T09:43:04.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d8009330 0x7fc5ec155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d8004150 0x7fc5ec0c98a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 --2- 172.21.15.158:0/3730503184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d8006760 0x7fc5ec0acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:04.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 >> 172.21.15.158:0/3730503184 conn(0x7fc5ec0cc5b0 msgr2=0x7fc5ec0cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:04.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 shutdown_connections 2023-10-16T09:43:04.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:04.519+0000 7fc5f395e700 1 -- 172.21.15.158:0/3730503184 wait complete. 2023-10-16T09:43:04.523 INFO:teuthology.orchestra.run.smithi158.stderr:adjusted tunables profile to default 2023-10-16T09:43:04.533 INFO:tasks.ceph:check_enable_crimson: False 2023-10-16T09:43:04.533 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-16T09:43:04.534 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:43:04.534 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-16T09:43:04.561 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:43:04.562 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-16T09:43:04.583 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:04.581+0000 7fceb733e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:43:04.614 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:04.613+0000 7fe6de52c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:43:04.628 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:43:04.628 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-16T09:43:04.665 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:04.663+0000 7fceb733e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:43:04.694 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:43:04.694 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-16T09:43:04.707 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:04.705+0000 7fe6de52c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:43:04.732 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:43:04.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-16T09:43:04.744 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:04.743+0000 7fceb733e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:43:04.759 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:43:04.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-16T09:43:04.803 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:04.802+0000 7fe6de52c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:43:04.825 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:43:04.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-16T09:43:04.893 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:43:04.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-16T09:43:04.898 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:04.896+0000 7fceb733e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:43:04.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new cca4abed-4408-40e4-951f-5821a22d0170 0 2023-10-16T09:43:05.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.082+0000 7fe11878c700 1 Processor -- start 2023-10-16T09:43:05.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.082+0000 7fe11878c700 1 -- start start 2023-10-16T09:43:05.093 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe11878c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe11006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.093 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe11878c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe110159660 con 0x7fe1100cbef0 2023-10-16T09:43:05.094 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe11878c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11006b120 con 0x7fe1100d0e10 2023-10-16T09:43:05.094 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe11878c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1100cb710 con 0x7fe110159280 2023-10-16T09:43:05.094 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe116528700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe11006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.094 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe116528700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe11006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46390/0 (socket says 172.21.15.172:46390) 2023-10-16T09:43:05.095 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe115d27700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe1100d0e10 0x7fe11015c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:33870/0 (socket says 172.21.15.172:33870) 2023-10-16T09:43:05.095 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe116528700 1 -- 172.21.15.172:0/2148803450 learned_addr learned my addr 172.21.15.172:0/2148803450 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:05.095 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe116528700 1 -- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 msgr2=0x7fe11006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T09:43:05.095 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe116528700 1 -- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 msgr2=0x7fe11006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:05.095 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe116528700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe11006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:05.096 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe116528700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe11006a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:05.096 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373414045 0 0) 0x7fe11006b120 con 0x7fe1100d0e10 2023-10-16T09:43:05.096 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0fc0035b0 con 0x7fe1100d0e10 2023-10-16T09:43:05.096 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.083+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472809968 0 0) 0x7fe0fc0035b0 con 0x7fe1100d0e10 2023-10-16T09:43:05.097 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 msgr2=0x7fe11006a8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:05.097 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe11006a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.097 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 >> v1:172.21.15.158:6789/0 conn(0x7fe1100cbef0 legacy=0x7fe1100cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.097 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11006b400 con 0x7fe1100d0e10 2023-10-16T09:43:05.098 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe100001670 con 0x7fe1100d0e10 2023-10-16T09:43:05.098 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 >> v1:172.21.15.172:6789/0 conn(0x7fe1100d0e10 legacy=0x7fe11015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.098 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc003a00 0x7fe0fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.098 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe0fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.099 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0fc004d90 0x7fe0fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.099 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc00bac0 con 0x7fe0fc003a00 2023-10-16T09:43:05.099 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc00bc00 con 0x7fe0fc004d90 2023-10-16T09:43:05.099 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc00bdc0 con 0x7fe110159280 2023-10-16T09:43:05.099 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115d27700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe0fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.100 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116d29700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0fc004d90 0x7fe0fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.100 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116528700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc003a00 0x7fe0fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.100 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116d29700 1 -- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 msgr2=0x7fe0fc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:05.100 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116d29700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe110159280 0x7fe0fc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.100 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116d29700 1 -- 172.21.15.172:0/2148803450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc003a00 msgr2=0x7fe0fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:05.101 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116d29700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc003a00 0x7fe0fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.101 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116d29700 1 -- 172.21.15.172:0/2148803450 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc0035b0 con 0x7fe0fc004d90 2023-10-16T09:43:05.101 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116528700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc003a00 0x7fe0fc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:05.101 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe116d29700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0fc004d90 0x7fe0fc005210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe108010290 tx=0x7fe0fc005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bd49657946444b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.101 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe108001e70 con 0x7fe0fc004d90 2023-10-16T09:43:05.102 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe108010e50 con 0x7fe0fc004d90 2023-10-16T09:43:05.102 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.084+0000 7fe115526700 1 -- 172.21.15.172:0/2148803450 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe108016660 con 0x7fe0fc004d90 2023-10-16T09:43:05.102 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.085+0000 7fe11878c700 1 -- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0fc004d90 msgr2=0x7fe0fc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.102 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.085+0000 7fe11878c700 1 --2- 172.21.15.172:0/2148803450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0fc004d90 0x7fe0fc005210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe108010290 tx=0x7fe0fc005880 comp rx=0 tx=0).stop 2023-10-16T09:43:05.102 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.085+0000 7fe116d29700 1 -- 172.21.15.172:0/2148803450 reap_dead start 2023-10-16T09:43:05.103 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.085+0000 7fe11878c700 1 -- 172.21.15.172:0/2148803450 shutdown_connections 2023-10-16T09:43:05.103 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.085+0000 7fe11878c700 1 -- 172.21.15.172:0/2148803450 >> 172.21.15.172:0/2148803450 conn(0x7fe110099a10 msgr2=0x7fe1100d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.103 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.085+0000 7fe11878c700 1 -- 172.21.15.172:0/2148803450 shutdown_connections 2023-10-16T09:43:05.103 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.085+0000 7fe11878c700 1 -- 172.21.15.172:0/2148803450 wait complete. 2023-10-16T09:43:05.103 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.086+0000 7fe11878c700 1 Processor -- start 2023-10-16T09:43:05.104 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.086+0000 7fe11878c700 1 -- start start 2023-10-16T09:43:05.104 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe11878c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc004d90 0x7fe1100ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.104 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe11878c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe110150a00 0x7fe1100adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.104 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe11878c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 0x7fe1100c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.105 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe11878c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe110155c70 con 0x7fe0fc004d90 2023-10-16T09:43:05.105 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe11878c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1100cede0 con 0x7fe110150a00 2023-10-16T09:43:05.105 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe11878c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1100cf0c0 con 0x7fe1100c74a0 2023-10-16T09:43:05.105 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe110150a00 0x7fe1100adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.105 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe110150a00 0x7fe1100adc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38262/0 (socket says 172.21.15.172:38262) 2023-10-16T09:43:05.106 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 -- 172.21.15.172:0/1320693328 learned_addr learned my addr 172.21.15.172:0/1320693328 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:05.106 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe116d29700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 0x7fe1100c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.106 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe116d29700 1 -- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 msgr2=0x7fe1100c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-16T09:43:05.106 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe116d29700 1 -- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 msgr2=0x7fe1100c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:05.106 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe116d29700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 0x7fe1100c7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:05.107 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe116d29700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 0x7fe1100c7920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:05.107 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe116528700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc004d90 0x7fe1100ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.107 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 -- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 msgr2=0x7fe1100c7920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:05.107 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 0x7fe1100c7920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.108 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 -- 172.21.15.172:0/1320693328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc004d90 msgr2=0x7fe1100ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.108 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc004d90 0x7fe1100ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.108 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.087+0000 7fe115d27700 1 -- 172.21.15.172:0/1320693328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110155900 con 0x7fe110150a00 2023-10-16T09:43:05.108 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe115d27700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe110150a00 0x7fe1100adc30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe1000072c0 tx=0x7fe1000075d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.109 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe106ffd700 1 -- 172.21.15.172:0/1320693328 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000f8d0 con 0x7fe110150a00 2023-10-16T09:43:05.109 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe106ffd700 1 -- 172.21.15.172:0/1320693328 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10000fa70 con 0x7fe110150a00 2023-10-16T09:43:05.109 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe110150e80 con 0x7fe110150a00 2023-10-16T09:43:05.109 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe106ffd700 1 -- 172.21.15.172:0/1320693328 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000fe20 con 0x7fe110150a00 2023-10-16T09:43:05.109 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe110156100 con 0x7fe110150a00 2023-10-16T09:43:05.110 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe106ffd700 1 -- 172.21.15.172:0/1320693328 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe10001d760 con 0x7fe110150a00 2023-10-16T09:43:05.110 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.088+0000 7fe106ffd700 1 -- 172.21.15.172:0/1320693328 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe1000136f0 con 0x7fe110150a00 2023-10-16T09:43:05.110 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.089+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1100681b0 con 0x7fe110150a00 2023-10-16T09:43:05.110 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.091+0000 7fe106ffd700 1 -- 172.21.15.172:0/1320693328 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fe100013900 con 0x7fe110150a00 2023-10-16T09:43:05.175 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.173+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cca4abed-4408-40e4-951f-5821a22d0170", "id": 0} v 0) v1 -- 0x7fe1100ccdc0 con 0x7fe110150a00 2023-10-16T09:43:05.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.177+0000 7fe106ffd700 1 -- 172.21.15.172:0/1320693328 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cca4abed-4408-40e4-951f-5821a22d0170", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe10002c0b0 con 0x7fe110150a00 2023-10-16T09:43:05.179 INFO:teuthology.orchestra.run.smithi172.stdout:0 2023-10-16T09:43:05.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe110150a00 msgr2=0x7fe1100adc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe110150a00 0x7fe1100adc30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe1000072c0 tx=0x7fe1000075d0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 shutdown_connections 2023-10-16T09:43:05.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1100c74a0 0x7fe1100c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0fc004d90 0x7fe1100ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 --2- 172.21.15.172:0/1320693328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe110150a00 0x7fe1100adc30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 >> 172.21.15.172:0/1320693328 conn(0x7fe110099a10 msgr2=0x7fe11015b080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 shutdown_connections 2023-10-16T09:43:05.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.178+0000 7fe11878c700 1 -- 172.21.15.172:0/1320693328 wait complete. 2023-10-16T09:43:05.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 009ee1a1-f593-4806-835d-78efa006069f 1 2023-10-16T09:43:05.240 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:05.239+0000 7fe6de52c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:43:05.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.312+0000 7fb83b9b2700 1 Processor -- start 2023-10-16T09:43:05.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.312+0000 7fb83b9b2700 1 -- start start 2023-10-16T09:43:05.317 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83b9b2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 0x7fb8340665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.317 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83b9b2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8340cf190 0x7fb83406d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.317 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83b9b2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8340cf930 con 0x7fb83406d950 2023-10-16T09:43:05.317 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83b9b2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8340cf650 con 0x7fb8340cf190 2023-10-16T09:43:05.318 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83b9b2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340c51b0 con 0x7fb834151220 2023-10-16T09:43:05.318 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb838f4d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8340cf190 0x7fb83406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.318 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83974e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 0x7fb8340665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.318 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb839f4f700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb83406d950 0x7fb8340c5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:33888/0 (socket says 172.21.15.172:33888) 2023-10-16T09:43:05.319 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb839f4f700 1 -- 172.21.15.172:0/3563984930 learned_addr learned my addr 172.21.15.172:0/3563984930 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-10-16T09:43:05.319 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb838f4d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8340cf190 0x7fb83406d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38266/0 (socket says 172.21.15.172:38266) 2023-10-16T09:43:05.319 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83974e700 1 -- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 msgr2=0x7fb8340665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T09:43:05.319 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83974e700 1 -- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 msgr2=0x7fb8340665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:05.319 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83974e700 1 --2- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 0x7fb8340665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:05.320 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb83974e700 1 --2- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 0x7fb8340665a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:05.320 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb838f4d700 1 -- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 msgr2=0x7fb8340665a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:05.320 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb838f4d700 1 --2- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 0x7fb8340665a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.320 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb838f4d700 1 -- 172.21.15.172:0/3563984930 >> v1:172.21.15.172:6789/0 conn(0x7fb83406d950 legacy=0x7fb8340c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.320 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.313+0000 7fb82bfff700 1 -- 172.21.15.172:0/3563984930 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187012153 0 0) 0x7fb8340cf930 con 0x7fb83406d950 2023-10-16T09:43:05.321 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb838f4d700 1 -- 172.21.15.172:0/3563984930 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8340c5490 con 0x7fb8340cf190 2023-10-16T09:43:05.321 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb838f4d700 1 --2- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8340cf190 0x7fb83406d410 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb82400bae0 tx=0x7fb82400bdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbaf8df53666c93d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.321 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb82bfff700 1 -- 172.21.15.172:0/3563984930 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400dd30 con 0x7fb8340cf190 2023-10-16T09:43:05.321 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb82bfff700 1 -- 172.21.15.172:0/3563984930 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82400de90 con 0x7fb8340cf190 2023-10-16T09:43:05.321 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb82bfff700 1 -- 172.21.15.172:0/3563984930 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824010690 con 0x7fb8340cf190 2023-10-16T09:43:05.322 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb83b9b2700 1 -- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8340cf190 msgr2=0x7fb83406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.322 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb83b9b2700 1 --2- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8340cf190 0x7fb83406d410 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb82400bae0 tx=0x7fb82400bdf0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.322 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb83b9b2700 1 -- 172.21.15.172:0/3563984930 shutdown_connections 2023-10-16T09:43:05.322 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb83b9b2700 1 --2- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb834151220 0x7fb8340665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.322 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb83b9b2700 1 --2- 172.21.15.172:0/3563984930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8340cf190 0x7fb83406d410 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.323 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.314+0000 7fb83b9b2700 1 -- 172.21.15.172:0/3563984930 >> 172.21.15.172:0/3563984930 conn(0x7fb8340cc5b0 msgr2=0x7fb8340cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.323 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.315+0000 7fb83b9b2700 1 -- 172.21.15.172:0/3563984930 shutdown_connections 2023-10-16T09:43:05.323 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.315+0000 7fb83b9b2700 1 -- 172.21.15.172:0/3563984930 wait complete. 2023-10-16T09:43:05.327 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 Processor -- start 2023-10-16T09:43:05.327 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 -- start start 2023-10-16T09:43:05.328 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb83406d950 0x7fb8340c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.328 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 0x7fb83415d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.328 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb834151220 0x7fb8341542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.328 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340c7300 con 0x7fb83406d950 2023-10-16T09:43:05.329 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83415d9c0 con 0x7fb834151220 2023-10-16T09:43:05.329 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.316+0000 7fb83b9b2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83415e490 con 0x7fb8340cf190 2023-10-16T09:43:05.329 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb838f4d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 0x7fb83415d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.329 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb834151220 0x7fb8341542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.330 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb834151220 0x7fb8341542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38274/0 (socket says 172.21.15.172:38274) 2023-10-16T09:43:05.330 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 -- 172.21.15.172:0/2884488754 learned_addr learned my addr 172.21.15.172:0/2884488754 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:05.330 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb838f4d700 1 -- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 msgr2=0x7fb83415d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-16T09:43:05.331 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb838f4d700 1 -- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 msgr2=0x7fb83415d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:05.331 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb83974e700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb83406d950 0x7fb8340c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.331 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb838f4d700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 0x7fb83415d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:05.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb838f4d700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 0x7fb83415d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:05.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 -- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 msgr2=0x7fb83415d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:05.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 0x7fb83415d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 -- 172.21.15.172:0/2884488754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb83406d950 msgr2=0x7fb8340c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb83406d950 0x7fb8340c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 -- 172.21.15.172:0/2884488754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82400b740 con 0x7fb834151220 2023-10-16T09:43:05.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb83974e700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb83406d950 0x7fb8340c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:05.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb839f4f700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb834151220 0x7fb8341542b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00cc70 tx=0x7fb82c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb829ffb700 1 -- 172.21.15.172:0/2884488754 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c00ad80 con 0x7fb834151220 2023-10-16T09:43:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb829ffb700 1 -- 172.21.15.172:0/2884488754 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c00ce70 con 0x7fb834151220 2023-10-16T09:43:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb829ffb700 1 -- 172.21.15.172:0/2884488754 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c0072f0 con 0x7fb834151220 2023-10-16T09:43:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83415dbf0 con 0x7fb834151220 2023-10-16T09:43:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.317+0000 7fb829ffb700 1 -- 172.21.15.172:0/2884488754 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb82c007750 con 0x7fb834151220 2023-10-16T09:43:05.335 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.318+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83415e1b0 con 0x7fb834151220 2023-10-16T09:43:05.335 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.318+0000 7fb829ffb700 1 -- 172.21.15.172:0/2884488754 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fb82c012400 con 0x7fb834151220 2023-10-16T09:43:05.335 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.318+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8340681b0 con 0x7fb834151220 2023-10-16T09:43:05.335 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:05.319+0000 7fceb733e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:43:05.336 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.321+0000 7fb829ffb700 1 -- 172.21.15.172:0/2884488754 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb82c018070 con 0x7fb834151220 2023-10-16T09:43:05.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.404+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "009ee1a1-f593-4806-835d-78efa006069f", "id": 1} v 0) v1 -- 0x7fb8340723f0 con 0x7fb834151220 2023-10-16T09:43:05.410 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.408+0000 7fb829ffb700 1 -- 172.21.15.172:0/2884488754 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "009ee1a1-f593-4806-835d-78efa006069f", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb82c0125c0 con 0x7fb834151220 2023-10-16T09:43:05.411 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-10-16T09:43:05.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.409+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb834151220 msgr2=0x7fb8341542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.409+0000 7fb83b9b2700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb834151220 0x7fb8341542b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00cc70 tx=0x7fb82c00a340 comp rx=0 tx=0).stop 2023-10-16T09:43:05.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.409+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 shutdown_connections 2023-10-16T09:43:05.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.409+0000 7fb83b9b2700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb834151220 0x7fb8341542b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.410+0000 7fb83b9b2700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb83406d950 0x7fb8340c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.410+0000 7fb83b9b2700 1 --2- 172.21.15.172:0/2884488754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8340cf190 0x7fb83415d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.410+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 >> 172.21.15.172:0/2884488754 conn(0x7fb8340cc5b0 msgr2=0x7fb834071930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.410+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 shutdown_connections 2023-10-16T09:43:05.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.410+0000 7fb83b9b2700 1 -- 172.21.15.172:0/2884488754 wait complete. 2023-10-16T09:43:05.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new ccc4ad87-c517-4106-9f8d-0da0c62b8e30 2 2023-10-16T09:43:05.446 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:05.444+0000 7fceb733e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:43:05.545 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.543+0000 7fa829696700 1 Processor -- start 2023-10-16T09:43:05.546 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa829696700 1 -- start start 2023-10-16T09:43:05.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa829696700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa82415dad0 0x7fa8240cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa829696700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8240ccc00 0x7fa8240cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa829696700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8240d0ae0 con 0x7fa82415deb0 2023-10-16T09:43:05.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa829696700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8240d12d0 con 0x7fa8240ccc00 2023-10-16T09:43:05.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa829696700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8240cbdf0 con 0x7fa82415dad0 2023-10-16T09:43:05.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa822ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa82415dad0 0x7fa8240cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa8227fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa82415deb0 0x7fa8240ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:33898/0 (socket says 172.21.15.172:33898) 2023-10-16T09:43:05.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.544+0000 7fa8227fc700 1 -- 172.21.15.172:0/2553465133 learned_addr learned my addr 172.21.15.172:0/2553465133 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-10-16T09:43:05.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa8237fe700 1 --2- 172.21.15.172:0/2553465133 >> v2:172.21.15.158:3300/0 conn(0x7fa8240ccc00 0x7fa8240cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa822ffd700 1 -- 172.21.15.172:0/2553465133 >> v2:172.21.15.158:3300/0 conn(0x7fa8240ccc00 msgr2=0x7fa8240cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:05.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa822ffd700 1 --2- 172.21.15.172:0/2553465133 >> v2:172.21.15.158:3300/0 conn(0x7fa8240ccc00 0x7fa8240cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa822ffd700 1 -- 172.21.15.172:0/2553465133 >> v1:172.21.15.172:6789/0 conn(0x7fa82415deb0 legacy=0x7fa8240ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa822ffd700 1 -- 172.21.15.172:0/2553465133 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8240cc280 con 0x7fa82415dad0 2023-10-16T09:43:05.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa821ffb700 1 -- 172.21.15.172:0/2553465133 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459742552 0 0) 0x7fa8240d0ae0 con 0x7fa82415deb0 2023-10-16T09:43:05.552 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa8237fe700 1 --2- 172.21.15.172:0/2553465133 >> v2:172.21.15.158:3300/0 conn(0x7fa8240ccc00 0x7fa8240cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:05.552 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa822ffd700 1 --2- 172.21.15.172:0/2553465133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8240cdfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa814009840 tx=0x7fa814009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=132f04d61b113d30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.552 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa821ffb700 1 -- 172.21.15.172:0/2553465133 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814011830 con 0x7fa82415dad0 2023-10-16T09:43:05.552 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa821ffb700 1 -- 172.21.15.172:0/2553465133 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa814011990 con 0x7fa82415dad0 2023-10-16T09:43:05.553 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.545+0000 7fa821ffb700 1 -- 172.21.15.172:0/2553465133 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814011d20 con 0x7fa82415dad0 2023-10-16T09:43:05.553 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 -- 172.21.15.172:0/2553465133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 msgr2=0x7fa8240cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.553 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 --2- 172.21.15.172:0/2553465133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8240cdfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa814009840 tx=0x7fa814009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:05.553 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 -- 172.21.15.172:0/2553465133 shutdown_connections 2023-10-16T09:43:05.553 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 --2- 172.21.15.172:0/2553465133 >> v2:172.21.15.158:3300/0 conn(0x7fa8240ccc00 0x7fa8240cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.554 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 --2- 172.21.15.172:0/2553465133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8240cdfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.554 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 -- 172.21.15.172:0/2553465133 >> 172.21.15.172:0/2553465133 conn(0x7fa824099a10 msgr2=0x7fa8240d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.554 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 -- 172.21.15.172:0/2553465133 shutdown_connections 2023-10-16T09:43:05.554 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.546+0000 7fa829696700 1 -- 172.21.15.172:0/2553465133 wait complete. 2023-10-16T09:43:05.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.547+0000 7fa829696700 1 Processor -- start 2023-10-16T09:43:05.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.547+0000 7fa829696700 1 -- start start 2023-10-16T09:43:05.557 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8240ccc00 0x7fa8241f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.558 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8241430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.558 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 0x7fa824143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.558 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8241f7210 con 0x7fa8240ccc00 2023-10-16T09:43:05.558 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240b4ff0 con 0x7fa82415dad0 2023-10-16T09:43:05.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240b52d0 con 0x7fa82415deb0 2023-10-16T09:43:05.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8241430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8241430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38284/0 (socket says 172.21.15.172:38284) 2023-10-16T09:43:05.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8237fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 0x7fa824143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 -- 172.21.15.172:0/2673641019 learned_addr learned my addr 172.21.15.172:0/2673641019 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:05.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa822ffd700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8240ccc00 0x7fa8241f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8237fe700 1 -- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 msgr2=0x7fa824143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-16T09:43:05.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8237fe700 1 -- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 msgr2=0x7fa824143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:05.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8237fe700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 0x7fa824143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:05.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8237fe700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 0x7fa824143600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:05.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 -- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 msgr2=0x7fa824143600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:05.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 0x7fa824143600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 -- 172.21.15.172:0/2673641019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8240ccc00 msgr2=0x7fa8241f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8240ccc00 0x7fa8241f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 -- 172.21.15.172:0/2673641019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8140094a0 con 0x7fa82415dad0 2023-10-16T09:43:05.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa8227fc700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8241430c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c550 tx=0x7fa81800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa80b7fe700 1 -- 172.21.15.172:0/2673641019 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818004c80 con 0x7fa82415dad0 2023-10-16T09:43:05.563 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa80b7fe700 1 -- 172.21.15.172:0/2673641019 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818004e00 con 0x7fa82415dad0 2023-10-16T09:43:05.563 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8240b55b0 con 0x7fa82415dad0 2023-10-16T09:43:05.563 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa80b7fe700 1 -- 172.21.15.172:0/2673641019 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818007310 con 0x7fa82415dad0 2023-10-16T09:43:05.563 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.548+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8240b5b70 con 0x7fa82415dad0 2023-10-16T09:43:05.563 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.549+0000 7fa80b7fe700 1 -- 172.21.15.172:0/2673641019 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa81801d3f0 con 0x7fa82415dad0 2023-10-16T09:43:05.564 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.549+0000 7fa80b7fe700 1 -- 172.21.15.172:0/2673641019 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fa8180137e0 con 0x7fa82415dad0 2023-10-16T09:43:05.564 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.549+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8240b5d90 con 0x7fa82415dad0 2023-10-16T09:43:05.564 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:05.549+0000 7fceb733e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:43:05.565 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.552+0000 7fa80b7fe700 1 -- 172.21.15.172:0/2673641019 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa8180133c0 con 0x7fa82415dad0 2023-10-16T09:43:05.637 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:05.635+0000 7fceb733e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:43:05.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.635+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ccc4ad87-c517-4106-9f8d-0da0c62b8e30", "id": 2} v 0) v1 -- 0x7fa8240681b0 con 0x7fa82415dad0 2023-10-16T09:43:05.641 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.639+0000 7fa80b7fe700 1 -- 172.21.15.172:0/2673641019 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ccc4ad87-c517-4106-9f8d-0da0c62b8e30", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa818013a40 con 0x7fa82415dad0 2023-10-16T09:43:05.641 INFO:teuthology.orchestra.run.smithi172.stdout:2 2023-10-16T09:43:05.643 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.640+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 msgr2=0x7fa8241430c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.643 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.640+0000 7fa829696700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8241430c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c550 tx=0x7fa81800c860 comp rx=0 tx=0).stop 2023-10-16T09:43:05.643 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.640+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 shutdown_connections 2023-10-16T09:43:05.643 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.640+0000 7fa829696700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa82415deb0 0x7fa824143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.640+0000 7fa829696700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8240ccc00 0x7fa8241f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.640+0000 7fa829696700 1 --2- 172.21.15.172:0/2673641019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82415dad0 0x7fa8241430c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.640+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 >> 172.21.15.172:0/2673641019 conn(0x7fa824099a10 msgr2=0x7fa82415cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.641+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 shutdown_connections 2023-10-16T09:43:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.641+0000 7fa829696700 1 -- 172.21.15.172:0/2673641019 wait complete. 2023-10-16T09:43:05.654 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new f4d08d1c-e139-471c-ace8-55a28a0c0b52 3 2023-10-16T09:43:05.819 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.818+0000 7f1c2a98d700 1 Processor -- start 2023-10-16T09:43:05.820 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.818+0000 7f1c2a98d700 1 -- start start 2023-10-16T09:43:05.823 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.818+0000 7f1c2a98d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c2406afe0 0x7f1c240caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.823 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.818+0000 7f1c2a98d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 0x7f1c240afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.824 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.818+0000 7f1c2a98d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240cbd30 con 0x7f1c2415da40 2023-10-16T09:43:05.824 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.818+0000 7f1c2a98d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c240cc1c0 con 0x7f1c2406afe0 2023-10-16T09:43:05.824 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.818+0000 7f1c2a98d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415de20 con 0x7f1c240cc590 2023-10-16T09:43:05.824 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c2406afe0 0x7f1c240caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.824 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c28f2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 0x7f1c240afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c2406afe0 0x7f1c240caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38290/0 (socket says 172.21.15.172:38290) 2023-10-16T09:43:05.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c28f2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 0x7f1c240afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46432/0 (socket says 172.21.15.172:46432) 2023-10-16T09:43:05.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 -- 172.21.15.172:0/1565928394 learned_addr learned my addr 172.21.15.172:0/1565928394 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:05.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c28f2a700 1 -- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 msgr2=0x7f1c240afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T09:43:05.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c28f2a700 1 -- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 msgr2=0x7f1c240afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:05.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c28f2a700 1 --2- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 0x7f1c240afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:05.826 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c28f2a700 1 --2- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 0x7f1c240afa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:05.826 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 -- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 msgr2=0x7f1c240afa40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:05.826 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 --2- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 0x7f1c240afa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.826 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 -- 172.21.15.172:0/1565928394 >> v1:172.21.15.158:6789/0 conn(0x7f1c2415da40 legacy=0x7f1c2406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.827 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 -- 172.21.15.172:0/1565928394 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240ce260 con 0x7f1c2406afe0 2023-10-16T09:43:05.827 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c237fe700 1 --2- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2406afe0 0x7f1c240caa20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c180098f0 tx=0x7f1c18009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f47d9b8ef3054de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.827 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c22ffd700 1 -- 172.21.15.172:0/1565928394 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1800b8a0 con 0x7f1c2406afe0 2023-10-16T09:43:05.827 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c22ffd700 1 -- 172.21.15.172:0/1565928394 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1800ba00 con 0x7f1c2406afe0 2023-10-16T09:43:05.827 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.819+0000 7f1c22ffd700 1 -- 172.21.15.172:0/1565928394 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1800bd90 con 0x7f1c2406afe0 2023-10-16T09:43:05.828 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 -- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2406afe0 msgr2=0x7f1c240caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.828 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 --2- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2406afe0 0x7f1c240caa20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c180098f0 tx=0x7f1c18009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:05.828 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 -- 172.21.15.172:0/1565928394 shutdown_connections 2023-10-16T09:43:05.828 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 --2- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c240cc590 0x7f1c240afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.829 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 --2- 172.21.15.172:0/1565928394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2406afe0 0x7f1c240caa20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.829 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 -- 172.21.15.172:0/1565928394 >> 172.21.15.172:0/1565928394 conn(0x7f1c24099a10 msgr2=0x7f1c240d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.829 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 -- 172.21.15.172:0/1565928394 shutdown_connections 2023-10-16T09:43:05.829 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.820+0000 7f1c2a98d700 1 -- 172.21.15.172:0/1565928394 wait complete. 2023-10-16T09:43:05.829 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.821+0000 7f1c2a98d700 1 Processor -- start 2023-10-16T09:43:05.830 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.821+0000 7f1c2a98d700 1 -- start start 2023-10-16T09:43:05.832 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c2a98d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c2406afe0 0x7f1c240cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.832 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c2a98d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c240cc590 0x7f1c2415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.832 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c2a98d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 0x7f1c240c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c2a98d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415c7f0 con 0x7f1c2406afe0 2023-10-16T09:43:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c2a98d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240c9840 con 0x7f1c240cc590 2023-10-16T09:43:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c2a98d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240c89e0 con 0x7f1c2415da40 2023-10-16T09:43:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c28f2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 0x7f1c240c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c240cc590 0x7f1c2415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c240cc590 0x7f1c2415c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38296/0 (socket says 172.21.15.172:38296) 2023-10-16T09:43:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c28f2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 0x7f1c240c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46438/0 (socket says 172.21.15.172:46438) 2023-10-16T09:43:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 -- 172.21.15.172:0/272016854 learned_addr learned my addr 172.21.15.172:0/272016854 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c28f2a700 1 -- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 msgr2=0x7f1c240c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-16T09:43:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c28f2a700 1 -- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 msgr2=0x7f1c240c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c28f2a700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 0x7f1c240c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c23fff700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c2406afe0 0x7f1c240cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c28f2a700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 0x7f1c240c84a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 -- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 msgr2=0x7f1c240c84a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 0x7f1c240c84a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 -- 172.21.15.172:0/272016854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c2406afe0 msgr2=0x7f1c240cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c2406afe0 0x7f1c240cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.822+0000 7f1c237fe700 1 -- 172.21.15.172:0/272016854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c18009550 con 0x7f1c240cc590 2023-10-16T09:43:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c237fe700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c240cc590 0x7f1c2415c2b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c18014040 tx=0x7f1c18004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c20ff9700 1 -- 172.21.15.172:0/272016854 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18004590 con 0x7f1c240cc590 2023-10-16T09:43:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c20ff9700 1 -- 172.21.15.172:0/272016854 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1801e070 con 0x7f1c240cc590 2023-10-16T09:43:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c20ff9700 1 -- 172.21.15.172:0/272016854 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180038e0 con 0x7f1c240cc590 2023-10-16T09:43:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2415cc70 con 0x7f1c240cc590 2023-10-16T09:43:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c240c8f10 con 0x7f1c240cc590 2023-10-16T09:43:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c20ff9700 1 -- 172.21.15.172:0/272016854 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1c18003d40 con 0x7f1c240cc590 2023-10-16T09:43:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.823+0000 7f1c20ff9700 1 -- 172.21.15.172:0/272016854 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f1c18020070 con 0x7f1c240cc590 2023-10-16T09:43:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.824+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c240681b0 con 0x7f1c240cc590 2023-10-16T09:43:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.826+0000 7f1c20ff9700 1 -- 172.21.15.172:0/272016854 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f1c1802a020 con 0x7f1c240cc590 2023-10-16T09:43:05.876 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:05.875+0000 7fe6de52c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:43:05.911 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.909+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f4d08d1c-e139-471c-ace8-55a28a0c0b52", "id": 3} v 0) v1 -- 0x7f1c240c9980 con 0x7f1c240cc590 2023-10-16T09:43:05.914 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.912+0000 7f1c20ff9700 1 -- 172.21.15.172:0/272016854 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4d08d1c-e139-471c-ace8-55a28a0c0b52", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1c18015020 con 0x7f1c240cc590 2023-10-16T09:43:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:3 2023-10-16T09:43:05.915 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c240cc590 msgr2=0x7f1c2415c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:05.915 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c240cc590 0x7f1c2415c2b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c18014040 tx=0x7f1c18004750 comp rx=0 tx=0).stop 2023-10-16T09:43:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 shutdown_connections 2023-10-16T09:43:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c2415da40 0x7f1c240c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c2406afe0 0x7f1c240cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 --2- 172.21.15.172:0/272016854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c240cc590 0x7f1c2415c2b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 >> 172.21.15.172:0/272016854 conn(0x7f1c24099a10 msgr2=0x7f1c240ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 shutdown_connections 2023-10-16T09:43:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:05.913+0000 7f1c2a98d700 1 -- 172.21.15.172:0/272016854 wait complete. 2023-10-16T09:43:05.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 0d0a0844-bf1a-41e7-ae47-58602557ea7b 4 2023-10-16T09:43:05.944 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:05.943+0000 7fe6de52c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:43:06.047 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.046+0000 7f738de58700 1 Processor -- start 2023-10-16T09:43:06.048 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.046+0000 7f738de58700 1 -- start start 2023-10-16T09:43:06.054 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.046+0000 7f738de58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f73880cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.054 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.046+0000 7f738de58700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73880d1ee0 con 0x7f738815daa0 2023-10-16T09:43:06.055 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.046+0000 7f738de58700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73880d12d0 con 0x7f73880cf360 2023-10-16T09:43:06.055 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.046+0000 7f738de58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73880d40a0 con 0x7f738815d6c0 2023-10-16T09:43:06.055 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73877fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f73880cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.055 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f7387fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f73880cf360 0x7f738806b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:33924/0 (socket says 172.21.15.172:33924) 2023-10-16T09:43:06.055 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f7387fff700 1 -- 172.21.15.172:0/893748171 learned_addr learned my addr 172.21.15.172:0/893748171 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-10-16T09:43:06.056 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73877fe700 1 -- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 msgr2=0x7f73880cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T09:43:06.056 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73877fe700 1 -- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 msgr2=0x7f73880cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:06.056 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73877fe700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f73880cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.056 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73877fe700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f73880cff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.057 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715562320 0 0) 0x7f73880d12d0 con 0x7f73880cf360 2023-10-16T09:43:06.057 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73700035b0 con 0x7f73880cf360 2023-10-16T09:43:06.057 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669893258 0 0) 0x7f73880d1ee0 con 0x7f738815daa0 2023-10-16T09:43:06.057 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73880d12d0 con 0x7f738815daa0 2023-10-16T09:43:06.057 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2507759293 0 0) 0x7f73700035b0 con 0x7f73880cf360 2023-10-16T09:43:06.057 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 msgr2=0x7f73880cff30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:06.058 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f73880cff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.058 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 >> v1:172.21.15.158:6789/0 conn(0x7f738815daa0 legacy=0x7f73880cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.058 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880d15b0 con 0x7f73880cf360 2023-10-16T09:43:06.058 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f73800019f0 con 0x7f73880cf360 2023-10-16T09:43:06.058 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 >> v1:172.21.15.172:6789/0 conn(0x7f73880cf360 legacy=0x7f738806b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.059 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7370003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.059 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73700040f0 0x7f7370004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.059 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7370004d90 0x7f7370005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.059 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737000bbd0 con 0x7f7370004d90 2023-10-16T09:43:06.059 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737000bd40 con 0x7f73700040f0 2023-10-16T09:43:06.060 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737000bf00 con 0x7f738815d6c0 2023-10-16T09:43:06.060 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f7386ffd700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73700040f0 0x7f7370004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.060 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f73877fe700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7370003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.060 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.047+0000 7f7387fff700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7370004d90 0x7f7370005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.060 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73877fe700 1 -- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73700040f0 msgr2=0x7f7370004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:06.060 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73877fe700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73700040f0 0x7f7370004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.061 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73877fe700 1 -- 172.21.15.172:0/893748171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7370004d90 msgr2=0x7f7370005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:06.061 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73877fe700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7370004d90 0x7f7370005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.061 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73877fe700 1 -- 172.21.15.172:0/893748171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73700035b0 con 0x7f738815d6c0 2023-10-16T09:43:06.061 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f7386ffd700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73700040f0 0x7f7370004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:06.061 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73877fe700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7370003b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7378009890 tx=0x7f7370005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68a596047ba35e63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.062 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737800c070 con 0x7f738815d6c0 2023-10-16T09:43:06.062 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7378004020 con 0x7f738815d6c0 2023-10-16T09:43:06.062 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f73867fc700 1 -- 172.21.15.172:0/893748171 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73780043b0 con 0x7f738815d6c0 2023-10-16T09:43:06.062 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f738de58700 1 -- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 msgr2=0x7f7370003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.062 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f738de58700 1 --2- 172.21.15.172:0/893748171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7370003b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7378009890 tx=0x7f7370005880 comp rx=0 tx=0).stop 2023-10-16T09:43:06.063 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.048+0000 7f7387fff700 1 -- 172.21.15.172:0/893748171 reap_dead start 2023-10-16T09:43:06.063 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.049+0000 7f738de58700 1 -- 172.21.15.172:0/893748171 shutdown_connections 2023-10-16T09:43:06.063 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.049+0000 7f738de58700 1 -- 172.21.15.172:0/893748171 >> 172.21.15.172:0/893748171 conn(0x7f738809c940 msgr2=0x7f73880d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.063 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.049+0000 7f738de58700 1 -- 172.21.15.172:0/893748171 shutdown_connections 2023-10-16T09:43:06.063 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.049+0000 7f738de58700 1 -- 172.21.15.172:0/893748171 wait complete. 2023-10-16T09:43:06.064 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.050+0000 7f738de58700 1 Processor -- start 2023-10-16T09:43:06.064 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.050+0000 7f738de58700 1 -- start start 2023-10-16T09:43:06.064 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f738de58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7388151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.064 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f738de58700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f738806cc00 0x7f738806d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.064 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f738de58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f738815fa10 0x7f738815fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.065 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f738de58700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738806d5c0 con 0x7f738806cc00 2023-10-16T09:43:06.065 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f738de58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7388152300 con 0x7f738815fa10 2023-10-16T09:43:06.065 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f738de58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7388160f20 con 0x7f738815d6c0 2023-10-16T09:43:06.065 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f73877fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7388151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.065 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7387fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f738815fa10 0x7f738815fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.066 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f73877fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7388151ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46452/0 (socket says 172.21.15.172:46452) 2023-10-16T09:43:06.066 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f73877fe700 1 -- 172.21.15.172:0/4148186878 learned_addr learned my addr 172.21.15.172:0/4148186878 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:06.066 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7386ffd700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f738806cc00 0x7f738806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.066 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f73877fe700 1 -- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 msgr2=0x7f7388151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-16T09:43:06.066 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f73877fe700 1 -- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 msgr2=0x7f7388151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:06.067 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f73877fe700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7388151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.067 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f73877fe700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7388151ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.067 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7387fff700 1 -- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 msgr2=0x7f7388151ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:06.067 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7387fff700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7388151ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.067 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7387fff700 1 -- 172.21.15.172:0/4148186878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f738806cc00 msgr2=0x7f738806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.068 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7387fff700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f738806cc00 0x7f738806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.068 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7387fff700 1 -- 172.21.15.172:0/4148186878 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7388152020 con 0x7f738815fa10 2023-10-16T09:43:06.068 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f7387fff700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f738815fa10 0x7f738815fec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7388162470 tx=0x7f738000a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.068 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f736ffff700 1 -- 172.21.15.172:0/4148186878 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738000aca0 con 0x7f738815fa10 2023-10-16T09:43:06.068 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f736ffff700 1 -- 172.21.15.172:0/4148186878 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738000ae20 con 0x7f738815fa10 2023-10-16T09:43:06.068 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7388160400 con 0x7f738815fa10 2023-10-16T09:43:06.069 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.051+0000 7f736ffff700 1 -- 172.21.15.172:0/4148186878 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380013630 con 0x7f738815fa10 2023-10-16T09:43:06.069 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.052+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7388160960 con 0x7f738815fa10 2023-10-16T09:43:06.069 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.052+0000 7f736ffff700 1 -- 172.21.15.172:0/4148186878 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7380013790 con 0x7f738815fa10 2023-10-16T09:43:06.069 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.052+0000 7f736ffff700 1 -- 172.21.15.172:0/4148186878 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f738001b6d0 con 0x7f738815fa10 2023-10-16T09:43:06.069 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.053+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73880cbf90 con 0x7f738815fa10 2023-10-16T09:43:06.070 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.055+0000 7f736ffff700 1 -- 172.21.15.172:0/4148186878 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f738003bb40 con 0x7f738815fa10 2023-10-16T09:43:06.104 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:06.102+0000 7fceb733e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:43:06.139 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.138+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0d0a0844-bf1a-41e7-ae47-58602557ea7b", "id": 4} v 0) v1 -- 0x7f73880681b0 con 0x7f738815fa10 2023-10-16T09:43:06.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.141+0000 7f736ffff700 1 -- 172.21.15.172:0/4148186878 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0d0a0844-bf1a-41e7-ae47-58602557ea7b", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f738002a030 con 0x7f738815fa10 2023-10-16T09:43:06.143 INFO:teuthology.orchestra.run.smithi172.stdout:4 2023-10-16T09:43:06.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f738815fa10 msgr2=0x7f738815fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f738815fa10 0x7f738815fec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7388162470 tx=0x7f738000a5b0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 shutdown_connections 2023-10-16T09:43:06.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f738815fa10 0x7f738815fec0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f738815d6c0 0x7f7388151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 --2- 172.21.15.172:0/4148186878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f738806cc00 0x7f738806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 >> 172.21.15.172:0/4148186878 conn(0x7f738809c940 msgr2=0x7f73880b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 shutdown_connections 2023-10-16T09:43:06.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.142+0000 7f738de58700 1 -- 172.21.15.172:0/4148186878 wait complete. 2023-10-16T09:43:06.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 886a6d94-630e-4241-945b-5cc0d4c8f848 5 2023-10-16T09:43:06.278 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.277+0000 7f83238d3700 1 Processor -- start 2023-10-16T09:43:06.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.277+0000 7f83238d3700 1 -- start start 2023-10-16T09:43:06.281 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f83238d3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f831c15b6b0 0x7f831c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.282 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f83238d3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 0x7f831c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.282 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f83238d3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c0b4420 con 0x7f831c0b3460 2023-10-16T09:43:06.282 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f83238d3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f831c15ba90 con 0x7f831c15b6b0 2023-10-16T09:43:06.282 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f83238d3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c0b38f0 con 0x7f831c0d0c60 2023-10-16T09:43:06.282 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8321e70700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f831c0b3460 0x7f831c070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:33942/0 (socket says 172.21.15.172:33942) 2023-10-16T09:43:06.283 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8321e70700 1 -- 172.21.15.172:0/2838827072 learned_addr learned my addr 172.21.15.172:0/2838827072 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-10-16T09:43:06.283 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8320e6e700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 0x7f831c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.283 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f832166f700 1 --2- 172.21.15.172:0/2838827072 >> v2:172.21.15.158:3300/0 conn(0x7f831c15b6b0 0x7f831c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.283 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8320e6e700 1 -- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 msgr2=0x7f831c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T09:43:06.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8320e6e700 1 -- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 msgr2=0x7f831c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:06.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8320e6e700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 0x7f831c0b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8320e6e700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 0x7f831c0b2f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8313fff700 1 -- 172.21.15.172:0/2838827072 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065655055 0 0) 0x7f831c0b4420 con 0x7f831c0b3460 2023-10-16T09:43:06.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f8313fff700 1 -- 172.21.15.172:0/2838827072 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83080035b0 con 0x7f831c0b3460 2023-10-16T09:43:06.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f832166f700 1 -- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 msgr2=0x7f831c0b2f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:06.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f832166f700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 0x7f831c0b2f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f832166f700 1 -- 172.21.15.172:0/2838827072 >> v1:172.21.15.172:6789/0 conn(0x7f831c0b3460 legacy=0x7f831c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f832166f700 1 -- 172.21.15.172:0/2838827072 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c0cbc20 con 0x7f831c15b6b0 2023-10-16T09:43:06.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.278+0000 7f832166f700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c15b6b0 0x7f831c0cfdb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8318009840 tx=0x7f8318009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4050348e2215239d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f8313fff700 1 -- 172.21.15.172:0/2838827072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318011830 con 0x7f831c15b6b0 2023-10-16T09:43:06.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f8313fff700 1 -- 172.21.15.172:0/2838827072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8318011990 con 0x7f831c15b6b0 2023-10-16T09:43:06.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f8313fff700 1 -- 172.21.15.172:0/2838827072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318011d20 con 0x7f831c15b6b0 2023-10-16T09:43:06.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 -- 172.21.15.172:0/2838827072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c15b6b0 msgr2=0x7f831c0cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c15b6b0 0x7f831c0cfdb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8318009840 tx=0x7f8318009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:06.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 -- 172.21.15.172:0/2838827072 shutdown_connections 2023-10-16T09:43:06.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c15b6b0 0x7f831c0cfdb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 --2- 172.21.15.172:0/2838827072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c0d0c60 0x7f831c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 -- 172.21.15.172:0/2838827072 >> 172.21.15.172:0/2838827072 conn(0x7f831c0cdf20 msgr2=0x7f831c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 -- 172.21.15.172:0/2838827072 shutdown_connections 2023-10-16T09:43:06.288 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.279+0000 7f83238d3700 1 -- 172.21.15.172:0/2838827072 wait complete. 2023-10-16T09:43:06.290 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 Processor -- start 2023-10-16T09:43:06.290 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 -- start start 2023-10-16T09:43:06.291 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c0b3460 0x7f831c1502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.291 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f831c0d0c60 0x7f831c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.291 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 0x7f831c1f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.291 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c153860 con 0x7f831c0b3460 2023-10-16T09:43:06.291 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c1f0720 con 0x7f831c0d0c60 2023-10-16T09:43:06.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f83238d3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c1517c0 con 0x7f831c15b6b0 2023-10-16T09:43:06.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f8320e6e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f831c0d0c60 0x7f831c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f8321e70700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 0x7f831c1f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f832166f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c0b3460 0x7f831c1502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.293 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f8320e6e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f831c0d0c60 0x7f831c0cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38326/0 (socket says 172.21.15.172:38326) 2023-10-16T09:43:06.293 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.281+0000 7f8320e6e700 1 -- 172.21.15.172:0/2277326871 learned_addr learned my addr 172.21.15.172:0/2277326871 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:06.293 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8321e70700 1 -- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 msgr2=0x7f831c1f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-16T09:43:06.293 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8321e70700 1 -- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 msgr2=0x7f831c1f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:06.293 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8321e70700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 0x7f831c1f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.294 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8321e70700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 0x7f831c1f18e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.294 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8320e6e700 1 -- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 msgr2=0x7f831c1f18e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:06.294 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8320e6e700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 0x7f831c1f18e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.294 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8320e6e700 1 -- 172.21.15.172:0/2277326871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c0b3460 msgr2=0x7f831c1502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.294 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8320e6e700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c0b3460 0x7f831c1502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.295 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8320e6e700 1 -- 172.21.15.172:0/2277326871 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83180094a0 con 0x7f831c0d0c60 2023-10-16T09:43:06.295 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f832166f700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c0b3460 0x7f831c1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:06.295 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8320e6e700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f831c0d0c60 0x7f831c0cad70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f830c00e730 tx=0x7f830c00ea40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.295 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8311ffb700 1 -- 172.21.15.172:0/2277326871 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c013610 con 0x7f831c0d0c60 2023-10-16T09:43:06.295 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8311ffb700 1 -- 172.21.15.172:0/2277326871 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c013790 con 0x7f831c0d0c60 2023-10-16T09:43:06.295 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c151aa0 con 0x7f831c0d0c60 2023-10-16T09:43:06.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8311ffb700 1 -- 172.21.15.172:0/2277326871 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c013b40 con 0x7f831c0d0c60 2023-10-16T09:43:06.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831c152060 con 0x7f831c0d0c60 2023-10-16T09:43:06.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.282+0000 7f8311ffb700 1 -- 172.21.15.172:0/2277326871 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f830c01c450 con 0x7f831c0d0c60 2023-10-16T09:43:06.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.283+0000 7f8311ffb700 1 -- 172.21.15.172:0/2277326871 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f830c01cc80 con 0x7f831c0d0c60 2023-10-16T09:43:06.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.283+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831c1f08f0 con 0x7f831c0d0c60 2023-10-16T09:43:06.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.285+0000 7f8311ffb700 1 -- 172.21.15.172:0/2277326871 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f830c021060 con 0x7f831c0d0c60 2023-10-16T09:43:06.370 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.368+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "886a6d94-630e-4241-945b-5cc0d4c8f848", "id": 5} v 0) v1 -- 0x7f831c0681b0 con 0x7f831c0d0c60 2023-10-16T09:43:06.375 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.374+0000 7f8311ffb700 1 -- 172.21.15.172:0/2277326871 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "886a6d94-630e-4241-945b-5cc0d4c8f848", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f830c017070 con 0x7f831c0d0c60 2023-10-16T09:43:06.376 INFO:teuthology.orchestra.run.smithi172.stdout:5 2023-10-16T09:43:06.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f831c0d0c60 msgr2=0x7f831c0cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f831c0d0c60 0x7f831c0cad70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f830c00e730 tx=0x7f830c00ea40 comp rx=0 tx=0).stop 2023-10-16T09:43:06.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 shutdown_connections 2023-10-16T09:43:06.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f831c15b6b0 0x7f831c1f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f831c0b3460 0x7f831c1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 --2- 172.21.15.172:0/2277326871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f831c0d0c60 0x7f831c0cad70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 >> 172.21.15.172:0/2277326871 conn(0x7f831c0cdf20 msgr2=0x7f831c0ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 shutdown_connections 2023-10-16T09:43:06.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.375+0000 7f83238d3700 1 -- 172.21.15.172:0/2277326871 wait complete. 2023-10-16T09:43:06.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 20128664-438a-4be4-9ced-2375eb76d829 6 2023-10-16T09:43:06.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.516+0000 7ff5dc7ad700 1 Processor -- start 2023-10-16T09:43:06.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.516+0000 7ff5dc7ad700 1 -- start start 2023-10-16T09:43:06.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.517+0000 7ff5dc7ad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 0x7ff5d406eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.517+0000 7ff5dc7ad700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d416aef0 con 0x7ff5d40c9950 2023-10-16T09:43:06.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.517+0000 7ff5dc7ad700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d416ef90 con 0x7ff5d40c9dd0 2023-10-16T09:43:06.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.517+0000 7ff5dc7ad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d416bf70 con 0x7ff5d408a910 2023-10-16T09:43:06.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5da549700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 0x7ff5d406eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5da549700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 0x7ff5d406eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46470/0 (socket says 172.21.15.172:46470) 2023-10-16T09:43:06.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5da549700 1 -- 172.21.15.172:0/3293751784 learned_addr learned my addr 172.21.15.172:0/3293751784 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:06.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5da549700 1 -- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 msgr2=0x7ff5d406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2023-10-16T09:43:06.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5da549700 1 -- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 msgr2=0x7ff5d406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:06.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5da549700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 0x7ff5d406eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5da549700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 0x7ff5d406eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528445089 0 0) 0x7ff5d0001110 con 0x7ff5d40c9950 2023-10-16T09:43:06.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c00035b0 con 0x7ff5d40c9950 2023-10-16T09:43:06.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675537292 0 0) 0x7ff5d416ef90 con 0x7ff5d40c9dd0 2023-10-16T09:43:06.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5d0001110 con 0x7ff5d40c9dd0 2023-10-16T09:43:06.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2743945537 0 0) 0x7ff5c00035b0 con 0x7ff5d40c9950 2023-10-16T09:43:06.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 msgr2=0x7ff5d406eba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:06.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d408a910 0x7ff5d406eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 >> v1:172.21.15.172:6789/0 conn(0x7ff5d40c9dd0 legacy=0x7ff5d416e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d40ca260 con 0x7ff5d40c9950 2023-10-16T09:43:06.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff5d0001800 con 0x7ff5d40c9950 2023-10-16T09:43:06.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 >> v1:172.21.15.158:6789/0 conn(0x7ff5d40c9950 legacy=0x7ff5d416e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c0003bc0 0x7ff5c0003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5c0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.526 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c0005210 0x7ff5c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.526 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c000c050 con 0x7ff5c0005210 2023-10-16T09:43:06.526 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c000c1c0 con 0x7ff5c0004850 2023-10-16T09:43:06.526 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c000c380 con 0x7ff5c0003bc0 2023-10-16T09:43:06.526 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.518+0000 7ff5d9d48700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5c0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.527 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5da549700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c0003bc0 0x7ff5c0003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.527 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5dad4a700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c0005210 0x7ff5c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.527 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5d9d48700 1 -- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c0003bc0 msgr2=0x7ff5c0003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:06.527 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5d9d48700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c0003bc0 0x7ff5c0003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.529 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5d9d48700 1 -- 172.21.15.172:0/3293751784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c0005210 msgr2=0x7ff5c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:06.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5d9d48700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c0005210 0x7ff5c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5d9d48700 1 -- 172.21.15.172:0/3293751784 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c00035b0 con 0x7ff5c0004850 2023-10-16T09:43:06.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5dad4a700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c0005210 0x7ff5c000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:06.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5da549700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c0003bc0 0x7ff5c0003fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:06.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.519+0000 7ff5d9d48700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5c0004cd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007f50 tx=0x7ff5c0005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9517679c12f788c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.522+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0006860 con 0x7ff5c0004850 2023-10-16T09:43:06.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.522+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d00069c0 con 0x7ff5c0004850 2023-10-16T09:43:06.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.522+0000 7ff5d9547700 1 -- 172.21.15.172:0/3293751784 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0006d30 con 0x7ff5c0004850 2023-10-16T09:43:06.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.523+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 msgr2=0x7ff5c0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.523+0000 7ff5dc7ad700 1 --2- 172.21.15.172:0/3293751784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5c0004cd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007f50 tx=0x7ff5c0005880 comp rx=0 tx=0).stop 2023-10-16T09:43:06.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.523+0000 7ff5dad4a700 1 -- 172.21.15.172:0/3293751784 reap_dead start 2023-10-16T09:43:06.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.523+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/3293751784 shutdown_connections 2023-10-16T09:43:06.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.523+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/3293751784 >> 172.21.15.172:0/3293751784 conn(0x7ff5d40b0c70 msgr2=0x7ff5d40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.523+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/3293751784 shutdown_connections 2023-10-16T09:43:06.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.523+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/3293751784 wait complete. 2023-10-16T09:43:06.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.525+0000 7ff5dc7ad700 1 Processor -- start 2023-10-16T09:43:06.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.525+0000 7ff5dc7ad700 1 -- start start 2023-10-16T09:43:06.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5dc7ad700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5d4089d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5dc7ad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 0x7ff5d40b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5dc7ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5d40cd940 0x7ff5d40cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5dc7ad700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40ce300 con 0x7ff5d40cd940 2023-10-16T09:43:06.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5dc7ad700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40b1fb0 con 0x7ff5c0004850 2023-10-16T09:43:06.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5dc7ad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40b2290 con 0x7ff5d40b3df0 2023-10-16T09:43:06.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5d9d48700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 0x7ff5d40b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5d9d48700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 0x7ff5d40b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46482/0 (socket says 172.21.15.172:46482) 2023-10-16T09:43:06.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5d9d48700 1 -- 172.21.15.172:0/4058959571 learned_addr learned my addr 172.21.15.172:0/4058959571 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:06.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5dad4a700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5d40cd940 0x7ff5d40cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5d9d48700 1 -- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 msgr2=0x7ff5d40b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-16T09:43:06.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5d9d48700 1 -- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 msgr2=0x7ff5d40b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:06.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5d9d48700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 0x7ff5d40b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5d9d48700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 0x7ff5d40b4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.536 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.526+0000 7ff5da549700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5d4089d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.536 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dad4a700 1 -- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 msgr2=0x7ff5d40b4270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:06.536 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dad4a700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 0x7ff5d40b4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.536 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dad4a700 1 -- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 msgr2=0x7ff5d4089d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.536 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dad4a700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5d4089d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.537 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dad4a700 1 -- 172.21.15.172:0/4058959571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d40b2570 con 0x7ff5d40cd940 2023-10-16T09:43:06.537 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dad4a700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5d40cd940 0x7ff5d40cddc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4067ea0 tx=0x7ff5c400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.537 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5caffd700 1 -- 172.21.15.172:0/4058959571 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c401a070 con 0x7ff5d40cd940 2023-10-16T09:43:06.537 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5caffd700 1 -- 172.21.15.172:0/4058959571 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c400cd30 con 0x7ff5d40cd940 2023-10-16T09:43:06.537 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5caffd700 1 -- 172.21.15.172:0/4058959571 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400cea0 con 0x7ff5d40cd940 2023-10-16T09:43:06.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d40b2850 con 0x7ff5d40cd940 2023-10-16T09:43:06.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.527+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d40b3050 con 0x7ff5d40cd940 2023-10-16T09:43:06.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.528+0000 7ff5caffd700 1 -- 172.21.15.172:0/4058959571 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff5c4005310 con 0x7ff5d40cd940 2023-10-16T09:43:06.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.528+0000 7ff5caffd700 1 -- 172.21.15.172:0/4058959571 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff5c401fae0 con 0x7ff5d40cd940 2023-10-16T09:43:06.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.528+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d40681b0 con 0x7ff5d40cd940 2023-10-16T09:43:06.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.531+0000 7ff5caffd700 1 -- 172.21.15.172:0/4058959571 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff5cc007a50 con 0x7ff5d40cd940 2023-10-16T09:43:06.632 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.630+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd new", "uuid": "20128664-438a-4be4-9ced-2375eb76d829", "id": 6} v 0) v1 -- 0x7ff5d40b5100 con 0x7ff5d40cd940 2023-10-16T09:43:06.633 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.632+0000 7ff5caffd700 1 -- 172.21.15.172:0/4058959571 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "20128664-438a-4be4-9ced-2375eb76d829", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff5c4001580 con 0x7ff5d40cd940 2023-10-16T09:43:06.634 INFO:teuthology.orchestra.run.smithi172.stdout:6 2023-10-16T09:43:06.635 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5d40cd940 msgr2=0x7ff5d40cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.635 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5d40cd940 0x7ff5d40cddc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4067ea0 tx=0x7ff5c400c6a0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.635 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 shutdown_connections 2023-10-16T09:43:06.635 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5d40cd940 0x7ff5d40cddc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.635 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c0004850 0x7ff5d4089d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 --2- 172.21.15.172:0/4058959571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5d40b3df0 0x7ff5d40b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 >> 172.21.15.172:0/4058959571 conn(0x7ff5d40b0c70 msgr2=0x7ff5d406dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 shutdown_connections 2023-10-16T09:43:06.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.633+0000 7ff5dc7ad700 1 -- 172.21.15.172:0/4058959571 wait complete. 2023-10-16T09:43:06.637 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:06.636+0000 7fe6de52c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:43:06.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new b551f3a3-652d-48ad-ba98-dc3894a4bd38 7 2023-10-16T09:43:06.713 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:06.712+0000 7fe6de52c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:43:06.771 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:06.769+0000 7fceb733e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:43:06.774 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.770+0000 7f70a701f700 1 Processor -- start 2023-10-16T09:43:06.774 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.770+0000 7f70a701f700 1 -- start start 2023-10-16T09:43:06.774 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.770+0000 7f70a701f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70a015da40 0x7f70a00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.775 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.770+0000 7f70a701f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 0x7f70a006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.775 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a701f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a00cbd30 con 0x7f70a006ae10 2023-10-16T09:43:06.775 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a701f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a006b2c0 con 0x7f70a015da40 2023-10-16T09:43:06.775 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a701f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a00cc1c0 con 0x7f70a00cc590 2023-10-16T09:43:06.775 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70a015da40 0x7f70a00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.776 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 0x7f70a006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.776 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70a015da40 0x7f70a00caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38346/0 (socket says 172.21.15.172:38346) 2023-10-16T09:43:06.776 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 0x7f70a006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46486/0 (socket says 172.21.15.172:46486) 2023-10-16T09:43:06.776 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 -- 172.21.15.172:0/2586415072 learned_addr learned my addr 172.21.15.172:0/2586415072 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:06.776 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709ffff700 1 -- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 msgr2=0x7f70a006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T09:43:06.777 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709ffff700 1 -- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 msgr2=0x7f70a006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:06.777 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709ffff700 1 --2- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 0x7f70a006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.777 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709ffff700 1 --2- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 0x7f70a006a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.777 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 -- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 msgr2=0x7f70a006a8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:06.777 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 --2- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 0x7f70a006a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.778 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 -- 172.21.15.172:0/2586415072 >> v1:172.21.15.158:6789/0 conn(0x7f70a006ae10 legacy=0x7f70a00ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.778 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 -- 172.21.15.172:0/2586415072 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a015de20 con 0x7f70a015da40 2023-10-16T09:43:06.778 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f70a4dbb700 1 --2- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00caa20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f70940098f0 tx=0x7f7094009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dedc93494b30b25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.778 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709f7fe700 1 -- 172.21.15.172:0/2586415072 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709400b8a0 con 0x7f70a015da40 2023-10-16T09:43:06.778 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709f7fe700 1 -- 172.21.15.172:0/2586415072 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709400ba00 con 0x7f70a015da40 2023-10-16T09:43:06.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.771+0000 7f709f7fe700 1 -- 172.21.15.172:0/2586415072 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709400bd90 con 0x7f70a015da40 2023-10-16T09:43:06.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 -- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 msgr2=0x7f70a00caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 --2- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00caa20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f70940098f0 tx=0x7f7094009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:06.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 -- 172.21.15.172:0/2586415072 shutdown_connections 2023-10-16T09:43:06.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 --2- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00caa20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 --2- 172.21.15.172:0/2586415072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a00cc590 0x7f70a006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 -- 172.21.15.172:0/2586415072 >> 172.21.15.172:0/2586415072 conn(0x7f70a0099a10 msgr2=0x7f70a00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 -- 172.21.15.172:0/2586415072 shutdown_connections 2023-10-16T09:43:06.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.772+0000 7f70a701f700 1 -- 172.21.15.172:0/2586415072 wait complete. 2023-10-16T09:43:06.782 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.773+0000 7f70a701f700 1 Processor -- start 2023-10-16T09:43:06.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.773+0000 7f70a701f700 1 -- start start 2023-10-16T09:43:06.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a701f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 0x7f70a0147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a701f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70a00cc590 0x7f70a0147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a701f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:06.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a701f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a00ca650 con 0x7f70a00cc590 2023-10-16T09:43:06.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a701f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a00cf7b0 con 0x7f70a015da40 2023-10-16T09:43:06.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a701f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a00cfa90 con 0x7f70a006ae10 2023-10-16T09:43:06.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a4dbb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 0x7f70a0147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a55bc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a4dbb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 0x7f70a0147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:46490/0 (socket says 172.21.15.172:46490) 2023-10-16T09:43:06.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a55bc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38354/0 (socket says 172.21.15.172:38354) 2023-10-16T09:43:06.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a4dbb700 1 -- 172.21.15.172:0/638746300 learned_addr learned my addr 172.21.15.172:0/638746300 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-10-16T09:43:06.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f709ffff700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70a00cc590 0x7f70a0147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:06.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a4dbb700 1 -- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 msgr2=0x7f70a0147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-16T09:43:06.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a4dbb700 1 -- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 msgr2=0x7f70a0147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:06.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a4dbb700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 0x7f70a0147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:06.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a4dbb700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 0x7f70a0147a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:06.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a55bc700 1 -- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 msgr2=0x7f70a0147a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:06.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.774+0000 7f70a55bc700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 0x7f70a0147a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f70a55bc700 1 -- 172.21.15.172:0/638746300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70a00cc590 msgr2=0x7f70a0147120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f70a55bc700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70a00cc590 0x7f70a0147120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f70a55bc700 1 -- 172.21.15.172:0/638746300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7094009550 con 0x7f70a015da40 2023-10-16T09:43:06.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f709ffff700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70a00cc590 0x7f70a0147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:06.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f70a55bc700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00c74a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f709800cb90 tx=0x7f709800cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:06.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f709d7fa700 1 -- 172.21.15.172:0/638746300 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098003c00 con 0x7f70a015da40 2023-10-16T09:43:06.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f709d7fa700 1 -- 172.21.15.172:0/638746300 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098003d60 con 0x7f70a015da40 2023-10-16T09:43:06.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a00cfd70 con 0x7f70a015da40 2023-10-16T09:43:06.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f709d7fa700 1 -- 172.21.15.172:0/638746300 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709800aaa0 con 0x7f70a015da40 2023-10-16T09:43:06.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a0147f80 con 0x7f70a015da40 2023-10-16T09:43:06.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f709d7fa700 1 -- 172.21.15.172:0/638746300 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f70980050d0 con 0x7f70a015da40 2023-10-16T09:43:06.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.775+0000 7f709d7fa700 1 -- 172.21.15.172:0/638746300 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7098012c10 con 0x7f70a015da40 2023-10-16T09:43:06.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.776+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a00d0290 con 0x7f70a015da40 2023-10-16T09:43:06.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.778+0000 7f709d7fa700 1 -- 172.21.15.172:0/638746300 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7098012e20 con 0x7f70a015da40 2023-10-16T09:43:06.844 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:06.842+0000 7fceb733e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:43:06.854 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:06.852+0000 7fe6de52c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:43:06.862 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.861+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b551f3a3-652d-48ad-ba98-dc3894a4bd38", "id": 7} v 0) v1 -- 0x7f70a00681b0 con 0x7f70a015da40 2023-10-16T09:43:06.870 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.868+0000 7f709d7fa700 1 -- 172.21.15.172:0/638746300 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b551f3a3-652d-48ad-ba98-dc3894a4bd38", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f70980055e0 con 0x7f70a015da40 2023-10-16T09:43:06.870 INFO:teuthology.orchestra.run.smithi172.stdout:7 2023-10-16T09:43:06.871 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.869+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 msgr2=0x7f70a00c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:06.872 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.869+0000 7f70a701f700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00c74a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f709800cb90 tx=0x7f709800cea0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.872 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.870+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 shutdown_connections 2023-10-16T09:43:06.872 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.870+0000 7f70a701f700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a015da40 0x7f70a00c74a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.872 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.870+0000 7f70a701f700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a006ae10 0x7f70a0147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.872 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.870+0000 7f70a701f700 1 --2- 172.21.15.172:0/638746300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70a00cc590 0x7f70a0147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:06.873 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.870+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 >> 172.21.15.172:0/638746300 conn(0x7f70a0099a10 msgr2=0x7f70a00b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:06.873 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.870+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 shutdown_connections 2023-10-16T09:43:06.873 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-16T09:43:06.870+0000 7f70a701f700 1 -- 172.21.15.172:0/638746300 wait complete. 2023-10-16T09:43:06.883 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-16T09:43:06.883 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-16T09:43:06.886 INFO:tasks.ceph.osd.0:Started 2023-10-16T09:43:06.886 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-16T09:43:06.886 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-16T09:43:06.888 INFO:tasks.ceph.osd.1:Started 2023-10-16T09:43:06.889 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-16T09:43:06.889 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-16T09:43:06.891 INFO:tasks.ceph.osd.2:Started 2023-10-16T09:43:06.892 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-16T09:43:06.893 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-16T09:43:06.895 INFO:tasks.ceph.osd.3:Started 2023-10-16T09:43:06.895 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-16T09:43:06.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-16T09:43:06.898 INFO:tasks.ceph.osd.4:Started 2023-10-16T09:43:06.898 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-16T09:43:06.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-16T09:43:06.901 INFO:tasks.ceph.osd.5:Started 2023-10-16T09:43:06.901 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-16T09:43:06.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-16T09:43:06.903 INFO:tasks.ceph.osd.6:Started 2023-10-16T09:43:06.904 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-16T09:43:06.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-16T09:43:06.906 INFO:tasks.ceph.osd.7:Started 2023-10-16T09:43:06.907 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:07.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.058+0000 7f0e3c00c700 1 Processor -- start 2023-10-16T09:43:07.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.058+0000 7f0e3c00c700 1 -- start start 2023-10-16T09:43:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e3c00c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0e340cc5d0 0x7f0e3406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e3c00c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0e3406f340 0x7f0e3406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e3c00c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3406e770 con 0x7f0e3415dfb0 2023-10-16T09:43:07.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e3c00c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e340cf270 con 0x7f0e340cc5d0 2023-10-16T09:43:07.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e3c00c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3406fd10 con 0x7f0e3406f340 2023-10-16T09:43:07.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e39da8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0e340cc5d0 0x7f0e3406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:07.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e39da8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0e340cc5d0 0x7f0e3406e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45352/0 (socket says 172.21.15.158:45352) 2023-10-16T09:43:07.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e39da8700 1 -- 172.21.15.158:0/3478387048 learned_addr learned my addr 172.21.15.158:0/3478387048 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:07.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e3a5a9700 1 --2- 172.21.15.158:0/3478387048 >> v2:172.21.15.172:3300/0 conn(0x7f0e3406f340 0x7f0e3406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:07.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e39da8700 1 -- 172.21.15.158:0/3478387048 >> v2:172.21.15.172:3300/0 conn(0x7f0e3406f340 msgr2=0x7f0e3406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:07.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e39da8700 1 --2- 172.21.15.158:0/3478387048 >> v2:172.21.15.172:3300/0 conn(0x7f0e3406f340 0x7f0e3406f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e39da8700 1 -- 172.21.15.158:0/3478387048 >> v1:172.21.15.172:6789/0 conn(0x7f0e3415dfb0 legacy=0x7f0e340cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:07.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.059+0000 7f0e39da8700 1 -- 172.21.15.158:0/3478387048 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e340cbe00 con 0x7f0e340cc5d0 2023-10-16T09:43:07.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e39da8700 1 --2- 172.21.15.158:0/3478387048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e340cc5d0 0x7f0e3406e230 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0e340b61e0 tx=0x7f0e3000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999378475b2fc43e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:07.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e38da6700 1 -- 172.21.15.158:0/3478387048 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3000dc70 con 0x7f0e340cc5d0 2023-10-16T09:43:07.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e38da6700 1 -- 172.21.15.158:0/3478387048 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3000ddd0 con 0x7f0e340cc5d0 2023-10-16T09:43:07.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e38da6700 1 -- 172.21.15.158:0/3478387048 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e30010690 con 0x7f0e340cc5d0 2023-10-16T09:43:07.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 -- 172.21.15.158:0/3478387048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e340cc5d0 msgr2=0x7f0e3406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:07.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 --2- 172.21.15.158:0/3478387048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e340cc5d0 0x7f0e3406e230 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0e340b61e0 tx=0x7f0e3000be70 comp rx=0 tx=0).stop 2023-10-16T09:43:07.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 -- 172.21.15.158:0/3478387048 shutdown_connections 2023-10-16T09:43:07.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 --2- 172.21.15.158:0/3478387048 >> v2:172.21.15.172:3300/0 conn(0x7f0e3406f340 0x7f0e3406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 --2- 172.21.15.158:0/3478387048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e340cc5d0 0x7f0e3406e230 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 -- 172.21.15.158:0/3478387048 >> 172.21.15.158:0/3478387048 conn(0x7f0e340d02b0 msgr2=0x7f0e340d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 -- 172.21.15.158:0/3478387048 shutdown_connections 2023-10-16T09:43:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.060+0000 7f0e3c00c700 1 -- 172.21.15.158:0/3478387048 wait complete. 2023-10-16T09:43:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.061+0000 7f0e3c00c700 1 Processor -- start 2023-10-16T09:43:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3c00c700 1 -- start start 2023-10-16T09:43:07.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3c00c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 0x7f0e340b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:07.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3c00c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e340cc5d0 0x7f0e340c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3c00c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e3415dfb0 0x7f0e340c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3c00c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e340c5820 con 0x7f0e3415dfb0 2023-10-16T09:43:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3c00c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e340c5b00 con 0x7f0e340cc5d0 2023-10-16T09:43:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3c00c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e340c5de0 con 0x7f0e3406f340 2023-10-16T09:43:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3a5a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e3415dfb0 0x7f0e340c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3a5a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e3415dfb0 0x7f0e340c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45362/0 (socket says 172.21.15.158:45362) 2023-10-16T09:43:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e3a5a9700 1 -- 172.21.15.158:0/1479518871 learned_addr learned my addr 172.21.15.158:0/1479518871 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e39da8700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 0x7f0e340b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.062+0000 7f0e395a7700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e340cc5d0 0x7f0e340c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e39da8700 1 -- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 msgr2=0x7f0e340b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-16T09:43:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e39da8700 1 -- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 msgr2=0x7f0e340b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e39da8700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 0x7f0e340b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e39da8700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 0x7f0e340b56e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3a5a9700 1 -- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 msgr2=0x7f0e340b56e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3a5a9700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 0x7f0e340b56e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3a5a9700 1 -- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e340cc5d0 msgr2=0x7f0e340c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3a5a9700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e340cc5d0 0x7f0e340c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3a5a9700 1 -- 172.21.15.158:0/1479518871 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3000b7f0 con 0x7f0e3415dfb0 2023-10-16T09:43:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3a5a9700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e3415dfb0 0x7f0e340c9980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00efc0 tx=0x7f0e2c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e2a7fc700 1 -- 172.21.15.158:0/1479518871 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00cb80 con 0x7f0e3415dfb0 2023-10-16T09:43:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e2a7fc700 1 -- 172.21.15.158:0/1479518871 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c00f040 con 0x7f0e3415dfb0 2023-10-16T09:43:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e340c60c0 con 0x7f0e3415dfb0 2023-10-16T09:43:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e2a7fc700 1 -- 172.21.15.158:0/1479518871 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c014480 con 0x7f0e3415dfb0 2023-10-16T09:43:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e340b5c20 con 0x7f0e3415dfb0 2023-10-16T09:43:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e2a7fc700 1 -- 172.21.15.158:0/1479518871 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0e2c0148e0 con 0x7f0e3415dfb0 2023-10-16T09:43:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.063+0000 7f0e2a7fc700 1 -- 172.21.15.158:0/1479518871 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0e2c013ab0 con 0x7f0e3415dfb0 2023-10-16T09:43:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.064+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e340681b0 con 0x7f0e3415dfb0 2023-10-16T09:43:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.066+0000 7f0e2a7fc700 1 -- 172.21.15.158:0/1479518871 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0e2c018020 con 0x7f0e3415dfb0 2023-10-16T09:43:07.134 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:07.132+0000 7fe6de52c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:43:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.136+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e340b1510 con 0x7f0e3415dfb0 2023-10-16T09:43:07.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e2a7fc700 1 -- 172.21.15.158:0/1479518871 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0e2c014dc0 con 0x7f0e3415dfb0 2023-10-16T09:43:07.139 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:07.139 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":10,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:06.863253+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:07.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e3415dfb0 msgr2=0x7f0e340c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:07.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e3c00c700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e3415dfb0 0x7f0e340c9980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00efc0 tx=0x7f0e2c00c0c0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 shutdown_connections 2023-10-16T09:43:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e3c00c700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e3415dfb0 0x7f0e340c9980 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e3c00c700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e3406f340 0x7f0e340b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e3c00c700 1 --2- 172.21.15.158:0/1479518871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e340cc5d0 0x7f0e340c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.137+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 >> 172.21.15.158:0/1479518871 conn(0x7f0e340d02b0 msgr2=0x7f0e340d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:07.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.138+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 shutdown_connections 2023-10-16T09:43:07.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:07.138+0000 7f0e3c00c700 1 -- 172.21.15.158:0/1479518871 wait complete. 2023-10-16T09:43:07.151 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-16T09:43:07.151 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-16T09:43:07.251 INFO:tasks.ceph.osd.1.smithi158.stderr:2023-10-16T09:43:07.249+0000 7f7c6beb6640 -1 Falling back to public interface 2023-10-16T09:43:07.354 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:07.353+0000 7fe6de52c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:43:07.485 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:07.483+0000 7fe6de52c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:43:07.517 INFO:tasks.ceph.osd.0.smithi158.stderr:2023-10-16T09:43:07.515+0000 7feff57b9640 -1 Falling back to public interface 2023-10-16T09:43:07.517 INFO:tasks.ceph.osd.2.smithi158.stderr:2023-10-16T09:43:07.515+0000 7f3db0324640 -1 Falling back to public interface 2023-10-16T09:43:07.518 INFO:tasks.ceph.osd.0.smithi158.stderr:2023-10-16T09:43:07.516+0000 7fefecb77700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2023-10-16T09:43:07.524 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-10-16T09:43:07.522+0000 7fe14d76c640 -1 Falling back to public interface 2023-10-16T09:43:07.527 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-10-16T09:43:07.525+0000 7f793a440640 -1 Falling back to public interface 2023-10-16T09:43:07.532 INFO:tasks.ceph.osd.3.smithi158.stderr:2023-10-16T09:43:07.530+0000 7f2b6fb2b640 -1 Falling back to public interface 2023-10-16T09:43:07.533 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-10-16T09:43:07.531+0000 7f5f0fd2d640 -1 Falling back to public interface 2023-10-16T09:43:07.537 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-10-16T09:43:07.535+0000 7f1451685640 -1 Falling back to public interface 2023-10-16T09:43:07.579 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:07.577+0000 7fceb733e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:43:07.657 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:07.655+0000 7fceb733e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:43:07.708 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:07.706+0000 7fe6de52c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:43:07.800 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:07.798+0000 7fceb733e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:43:07.866 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:07.864+0000 7fe6de52c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:43:07.957 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:07.982 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:07.980+0000 7fe6de52c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:43:08.040 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:08.038+0000 7fceb733e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:43:08.071 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.069+0000 7fb7a17b1700 1 Processor -- start 2023-10-16T09:43:08.072 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.069+0000 7fb7a17b1700 1 -- start start 2023-10-16T09:43:08.078 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb7a17b1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb79c15b660 0x7fb79c0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:08.078 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb7a17b1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb79c0cd9e0 0x7fb79c0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:08.079 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb7a17b1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 0x7fb79c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:08.079 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb7a17b1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79c0cc450 con 0x7fb79c0cd9e0 2023-10-16T09:43:08.079 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb7a17b1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79c0ce530 con 0x7fb79c15b660 2023-10-16T09:43:08.079 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb7a17b1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c0ce810 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.079 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb79a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb79c0cd9e0 0x7fb79c0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:08.080 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb79a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb79c0cd9e0 0x7fb79c0cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45366/0 (socket says 172.21.15.158:45366) 2023-10-16T09:43:08.080 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.070+0000 7fb79a7fc700 1 -- 172.21.15.158:0/3692970960 learned_addr learned my addr 172.21.15.158:0/3692970960 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:08.080 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79b7fe700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 0x7fb79c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:08.081 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79affd700 1 --2- 172.21.15.158:0/3692970960 >> v2:172.21.15.172:3300/0 conn(0x7fb79c15b660 0x7fb79c0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:08.081 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79b7fe700 1 -- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 msgr2=0x7fb79c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T09:43:08.081 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79b7fe700 1 -- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 msgr2=0x7fb79c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T09:43:08.081 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79b7fe700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 0x7fb79c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:08.081 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79b7fe700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 0x7fb79c0cbf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:08.082 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79a7fc700 1 -- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 msgr2=0x7fb79c0cbf10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:08.082 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79a7fc700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 0x7fb79c0cbf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.082 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79a7fc700 1 -- 172.21.15.158:0/3692970960 >> v2:172.21.15.172:3300/0 conn(0x7fb79c15b660 msgr2=0x7fb79c0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:08.082 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79a7fc700 1 --2- 172.21.15.158:0/3692970960 >> v2:172.21.15.172:3300/0 conn(0x7fb79c15b660 0x7fb79c0c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.082 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79a7fc700 1 -- 172.21.15.158:0/3692970960 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c0ceaf0 con 0x7fb79c0cd9e0 2023-10-16T09:43:08.083 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb79a7fc700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cd9e0 0x7fb79c0cde40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0cef70 tx=0x7fb790009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13009a3b4d195cd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:08.083 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb799ffb700 1 -- 172.21.15.158:0/3692970960 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790011830 con 0x7fb79c0cd9e0 2023-10-16T09:43:08.083 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb799ffb700 1 -- 172.21.15.158:0/3692970960 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb790011990 con 0x7fb79c0cd9e0 2023-10-16T09:43:08.083 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.071+0000 7fb799ffb700 1 -- 172.21.15.158:0/3692970960 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790011d20 con 0x7fb79c0cd9e0 2023-10-16T09:43:08.083 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3692970960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cd9e0 msgr2=0x7fb79c0cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:08.084 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cd9e0 0x7fb79c0cde40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0cef70 tx=0x7fb790009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.084 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3692970960 shutdown_connections 2023-10-16T09:43:08.084 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cbaa0 0x7fb79c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.084 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3692970960 >> v2:172.21.15.172:3300/0 conn(0x7fb79c15b660 0x7fb79c0c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.084 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3692970960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cd9e0 0x7fb79c0cde40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.085 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3692970960 >> 172.21.15.158:0/3692970960 conn(0x7fb79c0705f0 msgr2=0x7fb79c0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:08.085 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3692970960 shutdown_connections 2023-10-16T09:43:08.085 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.072+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3692970960 wait complete. 2023-10-16T09:43:08.085 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.073+0000 7fb7a17b1700 1 Processor -- start 2023-10-16T09:43:08.086 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.073+0000 7fb7a17b1700 1 -- start start 2023-10-16T09:43:08.086 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb7a17b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cbaa0 0x7fb79c15d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:08.086 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb7a17b1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 0x7fb79c0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:08.087 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb7a17b1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb79c15b660 0x7fb79c158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:08.087 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb7a17b1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c06b9e0 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.087 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb7a17b1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c0cab90 con 0x7fb79c15b660 2023-10-16T09:43:08.087 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb7a17b1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c06bc80 con 0x7fb79c0cd9e0 2023-10-16T09:43:08.087 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb79affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cbaa0 0x7fb79c15d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:08.088 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb79affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cbaa0 0x7fb79c15d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45370/0 (socket says 172.21.15.158:45370) 2023-10-16T09:43:08.088 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb79affd700 1 -- 172.21.15.158:0/3385487617 learned_addr learned my addr 172.21.15.158:0/3385487617 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:08.088 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb79a7fc700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 0x7fb79c0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:08.088 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb79b7fe700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb79c15b660 0x7fb79c158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:08.088 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.074+0000 7fb79a7fc700 1 -- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 msgr2=0x7fb79c0ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-16T09:43:08.089 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79a7fc700 1 -- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 msgr2=0x7fb79c0ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T09:43:08.089 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79a7fc700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 0x7fb79c0ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T09:43:08.089 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79a7fc700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 0x7fb79c0ca0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:43:08.089 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79affd700 1 -- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 msgr2=0x7fb79c0ca0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:08.089 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79affd700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 0x7fb79c0ca0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.090 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79affd700 1 -- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb79c15b660 msgr2=0x7fb79c158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:08.090 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79affd700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb79c15b660 0x7fb79c158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.090 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79affd700 1 -- 172.21.15.158:0/3385487617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb790009550 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.090 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79b7fe700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb79c15b660 0x7fb79c158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:08.090 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb79affd700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cbaa0 0x7fb79c15d9c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00e870 tx=0x7fb78c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:08.091 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb7837fe700 1 -- 172.21.15.158:0/3385487617 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c009590 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.091 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb7837fe700 1 -- 172.21.15.158:0/3385487617 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c013680 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.091 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb7837fe700 1 -- 172.21.15.158:0/3385487617 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c0139f0 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.091 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79c06bf60 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.091 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79c06d140 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.091 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb7837fe700 1 -- 172.21.15.158:0/3385487617 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb78c005a20 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.092 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.075+0000 7fb7837fe700 1 -- 172.21.15.158:0/3385487617 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb78c006660 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.092 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.076+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79c04ec20 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.092 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.078+0000 7fb7837fe700 1 -- 172.21.15.158:0/3385487617 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb78c021060 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.151 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.150+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb79c0710e0 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.152 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.150+0000 7fb7837fe700 1 -- 172.21.15.158:0/3385487617 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb78c017070 con 0x7fb79c0cbaa0 2023-10-16T09:43:08.152 INFO:teuthology.misc.health.smithi158.stdout: 2023-10-16T09:43:08.152 INFO:teuthology.misc.health.smithi158.stdout:{"epoch":10,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:06.863253+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:08.154 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cbaa0 msgr2=0x7fb79c15d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:08.154 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cbaa0 0x7fb79c15d9c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00e870 tx=0x7fb78c00eb80 comp rx=0 tx=0).stop 2023-10-16T09:43:08.154 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 shutdown_connections 2023-10-16T09:43:08.154 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb79c15b660 0x7fb79c158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.155 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb79c0cbaa0 0x7fb79c15d9c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.155 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 --2- 172.21.15.158:0/3385487617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb79c0cd9e0 0x7fb79c0ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:08.155 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 >> 172.21.15.158:0/3385487617 conn(0x7fb79c0705f0 msgr2=0x7fb79c0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:08.155 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 shutdown_connections 2023-10-16T09:43:08.156 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:08.151+0000 7fb7a17b1700 1 -- 172.21.15.158:0/3385487617 wait complete. 2023-10-16T09:43:08.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-16T09:43:08.267 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:08.265+0000 7fceb733e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:43:08.318 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:08.316+0000 7fe6de52c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:43:08.403 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:08.401+0000 7fceb733e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:43:08.419 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:08.418+0000 7fe6de52c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:43:08.605 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:08.603+0000 7fe6de52c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:43:08.625 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:08.623+0000 7fceb733e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:43:08.782 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:08.781+0000 7fceb733e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:43:08.898 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:08.897+0000 7fceb733e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:43:09.198 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:09.197+0000 7fe6de52c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:43:09.229 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:09.227+0000 7fceb733e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:43:09.332 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:09.331+0000 7fceb733e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:43:09.521 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:09.519+0000 7fceb733e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:43:10.129 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:43:10.128+0000 7fceb733e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:43:10.221 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:10.220+0000 7fe696490700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-16T09:43:10.224 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:43:10.223+0000 7fe696490700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-16T09:43:10.788 INFO:tasks.ceph.osd.1.smithi158.stderr:2023-10-16T09:43:10.787+0000 7f7c6beb6640 -1 osd.1 0 log_to_monitors true 2023-10-16T09:43:11.046 INFO:tasks.ceph.osd.2.smithi158.stderr:2023-10-16T09:43:11.045+0000 7f3db0324640 -1 osd.2 0 log_to_monitors true 2023-10-16T09:43:11.060 INFO:tasks.ceph.osd.3.smithi158.stderr:2023-10-16T09:43:11.058+0000 7f2b6fb2b640 -1 osd.3 0 log_to_monitors true 2023-10-16T09:43:11.068 INFO:tasks.ceph.osd.0.smithi158.stderr:2023-10-16T09:43:11.067+0000 7feff57b9640 -1 osd.0 0 log_to_monitors true 2023-10-16T09:43:11.081 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-10-16T09:43:11.079+0000 7f793a440640 -1 osd.7 0 log_to_monitors true 2023-10-16T09:43:11.101 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-10-16T09:43:11.099+0000 7f5f0fd2d640 -1 osd.5 0 log_to_monitors true 2023-10-16T09:43:11.109 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-10-16T09:43:11.107+0000 7f1451685640 -1 osd.4 0 log_to_monitors true 2023-10-16T09:43:11.110 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-10-16T09:43:11.108+0000 7fe14d76c640 -1 osd.6 0 log_to_monitors true 2023-10-16T09:43:13.882 INFO:tasks.ceph.osd.0.smithi158.stderr:2023-10-16T09:43:13.878+0000 7fefe9b71700 -1 osd.0 0 waiting for initial osdmap 2023-10-16T09:43:13.887 INFO:tasks.ceph.osd.1.smithi158.stderr:2023-10-16T09:43:13.886+0000 7f7c6026e700 -1 osd.1 0 waiting for initial osdmap 2023-10-16T09:43:13.897 INFO:tasks.ceph.osd.0.smithi158.stderr:2023-10-16T09:43:13.896+0000 7fefe174b700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:13.899 INFO:tasks.ceph.osd.2.smithi158.stderr:2023-10-16T09:43:13.897+0000 7f3da46dc700 -1 osd.2 0 waiting for initial osdmap 2023-10-16T09:43:13.901 INFO:tasks.ceph.osd.1.smithi158.stderr:2023-10-16T09:43:13.900+0000 7f7c57e48700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:13.909 INFO:tasks.ceph.osd.2.smithi158.stderr:2023-10-16T09:43:13.908+0000 7f3d9c2b6700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:13.918 INFO:tasks.ceph.osd.3.smithi158.stderr:2023-10-16T09:43:13.917+0000 7f2b63ee3700 -1 osd.3 0 waiting for initial osdmap 2023-10-16T09:43:13.979 INFO:tasks.ceph.osd.3.smithi158.stderr:2023-10-16T09:43:13.978+0000 7f2b5babd700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:14.582 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-10-16T09:43:14.580+0000 7f792e7f8700 -1 osd.7 0 waiting for initial osdmap 2023-10-16T09:43:14.583 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-10-16T09:43:14.581+0000 7f1445a3d700 -1 osd.4 0 waiting for initial osdmap 2023-10-16T09:43:14.598 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-10-16T09:43:14.597+0000 7f143d617700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:14.603 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-10-16T09:43:14.602+0000 7f79263d2700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:14.629 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-10-16T09:43:14.628+0000 7f5f040e5700 -1 osd.5 0 waiting for initial osdmap 2023-10-16T09:43:14.638 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-10-16T09:43:14.637+0000 7fe141b24700 -1 osd.6 0 waiting for initial osdmap 2023-10-16T09:43:14.639 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-10-16T09:43:14.638+0000 7f5efbcbf700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:14.652 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-10-16T09:43:14.651+0000 7fe1396fe700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T09:43:14.971 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:15.095 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.092+0000 7fceb77b7700 1 Processor -- start 2023-10-16T09:43:15.095 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.092+0000 7fceb77b7700 1 -- start start 2023-10-16T09:43:15.099 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.093+0000 7fceb77b7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 0x7fceb015e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.100 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.093+0000 7fceb77b7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fceb0072720 0x7fceb006c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.100 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.093+0000 7fceb77b7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb00710e0 con 0x7fceb006cdb0 2023-10-16T09:43:15.100 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.093+0000 7fceb77b7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb006d240 con 0x7fceb0072720 2023-10-16T09:43:15.100 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.093+0000 7fceb77b7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb015e610 con 0x7fceb0159a50 2023-10-16T09:43:15.100 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.093+0000 7fceb5d54700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fceb006cdb0 0x7fceb00c9840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45772/0 (socket says 172.21.15.158:45772) 2023-10-16T09:43:15.101 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.093+0000 7fceb5d54700 1 -- 172.21.15.158:0/1538240371 learned_addr learned my addr 172.21.15.158:0/1538240371 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:15.101 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5553700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 0x7fceb015e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.101 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb4d52700 1 --2- 172.21.15.158:0/1538240371 >> v2:172.21.15.172:3300/0 conn(0x7fceb0072720 0x7fceb006c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.101 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695884845 0 0) 0x7fceb00710e0 con 0x7fceb006cdb0 2023-10-16T09:43:15.101 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce9c0035b0 con 0x7fceb006cdb0 2023-10-16T09:43:15.102 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776182132 0 0) 0x7fce9c0035b0 con 0x7fceb006cdb0 2023-10-16T09:43:15.102 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 msgr2=0x7fceb015e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:15.102 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 0x7fceb015e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.102 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 >> v2:172.21.15.172:3300/0 conn(0x7fceb0072720 msgr2=0x7fceb006c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:15.102 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 --2- 172.21.15.158:0/1538240371 >> v2:172.21.15.172:3300/0 conn(0x7fceb0072720 0x7fceb006c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.103 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb0071380 con 0x7fceb006cdb0 2023-10-16T09:43:15.103 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcea8001a40 con 0x7fceb006cdb0 2023-10-16T09:43:15.103 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 >> v1:172.21.15.158:6789/0 conn(0x7fceb006cdb0 legacy=0x7fceb00c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.103 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce9c003a00 0x7fce9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.103 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 0x7fce9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.103 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce9c004d90 0x7fce9c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.104 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb006d240 con 0x7fce9c003a00 2023-10-16T09:43:15.104 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb015e610 con 0x7fce9c004d90 2023-10-16T09:43:15.104 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c00ba10 con 0x7fceb0159a50 2023-10-16T09:43:15.104 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5553700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce9c003a00 0x7fce9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.104 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb4d52700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 0x7fce9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.105 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5d54700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce9c004d90 0x7fce9c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.105 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5553700 1 -- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 msgr2=0x7fce9c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:15.105 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5553700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 0x7fce9c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.105 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5553700 1 -- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce9c004d90 msgr2=0x7fce9c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:15.105 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5553700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce9c004d90 0x7fce9c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.106 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb5553700 1 -- 172.21.15.158:0/1538240371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c0035b0 con 0x7fce9c003a00 2023-10-16T09:43:15.106 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.094+0000 7fceb4d52700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fceb0159a50 0x7fce9c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:15.106 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb5553700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce9c003a00 0x7fce9c003e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009b60 tx=0x7fce9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f24401b4548ba651 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.106 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea000e6c0 con 0x7fce9c003a00 2023-10-16T09:43:15.106 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea000e820 con 0x7fce9c003a00 2023-10-16T09:43:15.107 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fcea7fff700 1 -- 172.21.15.158:0/1538240371 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea000eb90 con 0x7fce9c003a00 2023-10-16T09:43:15.107 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb77b7700 1 -- 172.21.15.158:0/1538240371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce9c003a00 msgr2=0x7fce9c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.107 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb77b7700 1 --2- 172.21.15.158:0/1538240371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce9c003a00 0x7fce9c003e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009b60 tx=0x7fce9c005880 comp rx=0 tx=0).stop 2023-10-16T09:43:15.107 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb5d54700 1 -- 172.21.15.158:0/1538240371 reap_dead start 2023-10-16T09:43:15.107 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb77b7700 1 -- 172.21.15.158:0/1538240371 shutdown_connections 2023-10-16T09:43:15.108 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb77b7700 1 -- 172.21.15.158:0/1538240371 >> 172.21.15.158:0/1538240371 conn(0x7fceb00b0de0 msgr2=0x7fceb00b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:15.108 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb77b7700 1 -- 172.21.15.158:0/1538240371 shutdown_connections 2023-10-16T09:43:15.108 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.095+0000 7fceb77b7700 1 -- 172.21.15.158:0/1538240371 wait complete. 2023-10-16T09:43:15.108 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.097+0000 7fceb77b7700 1 Processor -- start 2023-10-16T09:43:15.108 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.097+0000 7fceb77b7700 1 -- start start 2023-10-16T09:43:15.111 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb77b7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce9c003a00 0x7fceb014a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.111 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb77b7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fceb0148280 0x7fceb0148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.111 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb77b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fceb00cdde0 0x7fceb00ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.112 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb77b7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb014aa40 con 0x7fceb00cdde0 2023-10-16T09:43:15.112 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb77b7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00ccd60 con 0x7fceb0148280 2023-10-16T09:43:15.112 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb77b7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00cd020 con 0x7fce9c003a00 2023-10-16T09:43:15.112 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fceb00cdde0 0x7fceb00ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.112 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fceb00cdde0 0x7fceb00ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45608/0 (socket says 172.21.15.158:45608) 2023-10-16T09:43:15.113 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 -- 172.21.15.158:0/3980629787 learned_addr learned my addr 172.21.15.158:0/3980629787 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:15.113 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5553700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce9c003a00 0x7fceb014a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.113 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb4d52700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fceb0148280 0x7fceb0148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.113 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 -- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce9c003a00 msgr2=0x7fceb014a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.113 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce9c003a00 0x7fceb014a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.114 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 -- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fceb0148280 msgr2=0x7fceb0148700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.114 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fceb0148280 0x7fceb0148700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.114 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.098+0000 7fceb5d54700 1 -- 172.21.15.158:0/3980629787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00cd300 con 0x7fceb00cdde0 2023-10-16T09:43:15.114 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.099+0000 7fceb5d54700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fceb00cdde0 0x7fceb00ce260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcea8001380 tx=0x7fcea800a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.114 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.099+0000 7fcea5ffb700 1 -- 172.21.15.158:0/3980629787 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800ae40 con 0x7fceb00cdde0 2023-10-16T09:43:15.115 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.099+0000 7fcea5ffb700 1 -- 172.21.15.158:0/3980629787 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea80133f0 con 0x7fceb00cdde0 2023-10-16T09:43:15.115 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.099+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb00cd5e0 con 0x7fceb00cdde0 2023-10-16T09:43:15.115 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.099+0000 7fcea5ffb700 1 -- 172.21.15.158:0/3980629787 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea80137a0 con 0x7fceb00cdde0 2023-10-16T09:43:15.115 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.099+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb015a1d0 con 0x7fceb00cdde0 2023-10-16T09:43:15.116 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.100+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb00cc720 con 0x7fceb00cdde0 2023-10-16T09:43:15.116 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.100+0000 7fcea5ffb700 1 -- 172.21.15.158:0/3980629787 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fcea8007d50 con 0x7fceb00cdde0 2023-10-16T09:43:15.116 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.100+0000 7fcea5ffb700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fce9406ae70 0x7fce9406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.116 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.101+0000 7fcea5ffb700 1 -- 172.21.15.158:0/3980629787 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fcea80896b0 con 0x7fceb00cdde0 2023-10-16T09:43:15.116 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.101+0000 7fceb5553700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fce9406ae70 0x7fce9406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.116 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.101+0000 7fceb5553700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fce9406ae70 0x7fce9406d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcea00090b0 tx=0x7fcea0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.117 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.105+0000 7fcea5ffb700 1 -- 172.21.15.158:0/3980629787 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcea8054da0 con 0x7fceb00cdde0 2023-10-16T09:43:15.306 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.304+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fceb004ec20 con 0x7fceb00cdde0 2023-10-16T09:43:15.307 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.305+0000 7fcea5ffb700 1 -- 172.21.15.158:0/3980629787 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10920 (secure 0 0 0) 0x7fcea8017020 con 0x7fceb00cdde0 2023-10-16T09:43:15.307 INFO:teuthology.misc.health.smithi158.stdout: 2023-10-16T09:43:15.308 INFO:teuthology.misc.health.smithi158.stdout:{"epoch":13,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:14.866254+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:15.310 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fce9406ae70 msgr2=0x7fce9406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.310 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fce9406ae70 0x7fce9406d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcea00090b0 tx=0x7fcea0008030 comp rx=0 tx=0).stop 2023-10-16T09:43:15.310 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fceb00cdde0 msgr2=0x7fceb00ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.310 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fceb00cdde0 0x7fceb00ce260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcea8001380 tx=0x7fcea800a5d0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.311 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 shutdown_connections 2023-10-16T09:43:15.311 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fceb00cdde0 0x7fceb00ce260 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.311 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fce9406ae70 0x7fce9406d330 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.311 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce9c003a00 0x7fceb014a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.311 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 --2- 172.21.15.158:0/3980629787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fceb0148280 0x7fceb0148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.311 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 >> 172.21.15.158:0/3980629787 conn(0x7fceb00b0de0 msgr2=0x7fceb0073d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:15.312 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 shutdown_connections 2023-10-16T09:43:15.312 INFO:teuthology.misc.health.smithi158.stderr:2023-10-16T09:43:15.307+0000 7fceb77b7700 1 -- 172.21.15.158:0/3980629787 wait complete. 2023-10-16T09:43:15.321 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-16T09:43:15.321 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-16T09:43:15.322 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T09:43:15.322 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T09:43:15.325 INFO:tasks.ceph.mds.a:Started 2023-10-16T09:43:15.325 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T09:43:15.325 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T09:43:15.327 INFO:tasks.ceph.mds.c:Started 2023-10-16T09:43:15.328 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T09:43:15.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T09:43:15.330 INFO:tasks.ceph.mds.b:Started 2023-10-16T09:43:15.331 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T09:43:15.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T09:43:15.333 INFO:tasks.ceph.mds.d:Started 2023-10-16T09:43:15.333 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-16T09:43:15.334 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:15.405 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T09:43:15.405 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T09:43:15.412 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:43:15.413 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T09:43:15.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.460+0000 7f34fa1a7700 1 Processor -- start 2023-10-16T09:43:15.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.460+0000 7f34fa1a7700 1 -- start start 2023-10-16T09:43:15.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.460+0000 7f34fa1a7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f34f406e5d0 0x7f34f406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.460+0000 7f34fa1a7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4161270 con 0x7f34f406ab40 2023-10-16T09:43:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.460+0000 7f34fa1a7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4161510 con 0x7f34f40729b0 2023-10-16T09:43:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.460+0000 7f34fa1a7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f41617f0 con 0x7f34f406e5d0 2023-10-16T09:43:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f37fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f34f406ab40 0x7f34f40722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45820/0 (socket says 172.21.15.158:45820) 2023-10-16T09:43:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f37fe700 1 -- 172.21.15.158:0/3196198287 learned_addr learned my addr 172.21.15.158:0/3196198287 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f2ffd700 1 --2- 172.21.15.158:0/3196198287 >> v2:172.21.15.172:3300/0 conn(0x7f34f406e5d0 0x7f34f406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905328468 0 0) 0x7f34f4161270 con 0x7f34f406ab40 2023-10-16T09:43:15.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34dc0035b0 con 0x7f34f406ab40 2023-10-16T09:43:15.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212005459 0 0) 0x7f34f4161510 con 0x7f34f40729b0 2023-10-16T09:43:15.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4161270 con 0x7f34f40729b0 2023-10-16T09:43:15.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f2ffd700 1 -- 172.21.15.158:0/3196198287 >> v1:172.21.15.172:6789/0 conn(0x7f34f40729b0 legacy=0x7f34f415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048498349 0 0) 0x7f34dc0035b0 con 0x7f34f406ab40 2023-10-16T09:43:15.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f2ffd700 1 -- 172.21.15.158:0/3196198287 >> v1:172.21.15.158:6789/0 conn(0x7f34f406ab40 legacy=0x7f34f40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f2ffd700 1 -- 172.21.15.158:0/3196198287 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f4161ad0 con 0x7f34f406e5d0 2023-10-16T09:43:15.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f2ffd700 1 --2- 172.21.15.158:0/3196198287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406e5d0 0x7f34f406d820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009840 tx=0x7f34e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c754a989f14ee9b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4001670 con 0x7f34f406e5d0 2023-10-16T09:43:15.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e8004030 con 0x7f34f406e5d0 2023-10-16T09:43:15.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.461+0000 7f34f27fc700 1 -- 172.21.15.158:0/3196198287 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e80043c0 con 0x7f34f406e5d0 2023-10-16T09:43:15.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.462+0000 7f34fa1a7700 1 -- 172.21.15.158:0/3196198287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406e5d0 msgr2=0x7f34f406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.462+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/3196198287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406e5d0 0x7f34f406d820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009840 tx=0x7f34e8009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:15.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.462+0000 7f34fa1a7700 1 -- 172.21.15.158:0/3196198287 shutdown_connections 2023-10-16T09:43:15.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.462+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/3196198287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406e5d0 0x7f34f406d820 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.462+0000 7f34fa1a7700 1 -- 172.21.15.158:0/3196198287 >> 172.21.15.158:0/3196198287 conn(0x7f34f40d02b0 msgr2=0x7f34f40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:15.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.462+0000 7f34fa1a7700 1 -- 172.21.15.158:0/3196198287 shutdown_connections 2023-10-16T09:43:15.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.462+0000 7f34fa1a7700 1 -- 172.21.15.158:0/3196198287 wait complete. 2023-10-16T09:43:15.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.463+0000 7f34fa1a7700 1 Processor -- start 2023-10-16T09:43:15.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.463+0000 7f34fa1a7700 1 -- start start 2023-10-16T09:43:15.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34fa1a7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406ab40 0x7f34f40c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34fa1a7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34f406e5d0 0x7f34f40b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34fa1a7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34f40729b0 0x7f34f414a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34fa1a7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f40b1270 con 0x7f34f40729b0 2023-10-16T09:43:15.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34fa1a7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f40b1550 con 0x7f34f406ab40 2023-10-16T09:43:15.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34fa1a7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f40b1830 con 0x7f34f406e5d0 2023-10-16T09:43:15.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34f40729b0 0x7f34f414a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34f40729b0 0x7f34f414a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45652/0 (socket says 172.21.15.158:45652) 2023-10-16T09:43:15.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 -- 172.21.15.158:0/4014449612 learned_addr learned my addr 172.21.15.158:0/4014449612 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:15.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f2ffd700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34f406e5d0 0x7f34f40b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f37fe700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406ab40 0x7f34f40c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 -- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34f406e5d0 msgr2=0x7f34f40b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34f406e5d0 0x7f34f40b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 -- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406ab40 msgr2=0x7f34f40c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406ab40 0x7f34f40c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.464+0000 7f34f3fff700 1 -- 172.21.15.158:0/4014449612 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80094a0 con 0x7f34f40729b0 2023-10-16T09:43:15.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.465+0000 7f34f3fff700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34f40729b0 0x7f34f414a5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00cb50 tx=0x7f34ec00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.465+0000 7f34dbfff700 1 -- 172.21.15.158:0/4014449612 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec003c00 con 0x7f34f40729b0 2023-10-16T09:43:15.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.465+0000 7f34dbfff700 1 -- 172.21.15.158:0/4014449612 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec003d60 con 0x7f34f40729b0 2023-10-16T09:43:15.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.465+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f40b1b10 con 0x7f34f40729b0 2023-10-16T09:43:15.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.465+0000 7f34dbfff700 1 -- 172.21.15.158:0/4014449612 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00aa20 con 0x7f34f40729b0 2023-10-16T09:43:15.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.465+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f40c4f10 con 0x7f34f40729b0 2023-10-16T09:43:15.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.466+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f40b2030 con 0x7f34f40729b0 2023-10-16T09:43:15.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.466+0000 7f34dbfff700 1 -- 172.21.15.158:0/4014449612 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f34ec00ab80 con 0x7f34f40729b0 2023-10-16T09:43:15.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.467+0000 7f34dbfff700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34e006ae70 0x7f34e006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.467+0000 7f34f37fe700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34e006ae70 0x7f34e006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.467+0000 7f34dbfff700 1 -- 172.21.15.158:0/4014449612 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f34ec089010 con 0x7f34f40729b0 2023-10-16T09:43:15.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.467+0000 7f34f37fe700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34e006ae70 0x7f34e006d330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34e40011e0 tx=0x7f34e4000ed0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.471+0000 7f34dbfff700 1 -- 172.21.15.158:0/4014449612 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34f40b2030 con 0x7f34f40729b0 2023-10-16T09:43:15.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.671+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34f40b2030 con 0x7f34f40729b0 2023-10-16T09:43:15.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.672+0000 7f34dbfff700 1 -- 172.21.15.158:0/4014449612 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10920 (secure 0 0 0) 0x7f34f40b2030 con 0x7f34f40729b0 2023-10-16T09:43:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:15.675 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":13,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:14.866254+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:15.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34e006ae70 msgr2=0x7f34e006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34e006ae70 0x7f34e006d330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34e40011e0 tx=0x7f34e4000ed0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34f40729b0 msgr2=0x7f34f414a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34f40729b0 0x7f34f414a5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00cb50 tx=0x7f34ec00ce60 comp rx=0 tx=0).stop 2023-10-16T09:43:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 shutdown_connections 2023-10-16T09:43:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34f40729b0 0x7f34f414a5c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34f406ab40 0x7f34f40c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34e006ae70 0x7f34e006d330 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 --2- 172.21.15.158:0/4014449612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34f406e5d0 0x7f34f40b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 >> 172.21.15.158:0/4014449612 conn(0x7f34f40d02b0 msgr2=0x7f34f40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 shutdown_connections 2023-10-16T09:43:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.674+0000 7f34fa1a7700 1 -- 172.21.15.158:0/4014449612 wait complete. 2023-10-16T09:43:15.689 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2023-10-16T09:43:15.689 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:15.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f58652cd700 1 Processor -- start 2023-10-16T09:43:15.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f58652cd700 1 -- start start 2023-10-16T09:43:15.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f58652cd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f586015d7c0 0x7f58600d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f58652cd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58600d1210 0x7f58600ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f58652cd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600d3bd0 con 0x7f58600ce950 2023-10-16T09:43:15.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f58652cd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58600d4060 con 0x7f586015d7c0 2023-10-16T09:43:15.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f58652cd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58600d2cf0 con 0x7f58600d1210 2023-10-16T09:43:15.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.808+0000 7f585effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f586015d7c0 0x7f58600d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f586015d7c0 0x7f58600d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45666/0 (socket says 172.21.15.158:45666) 2023-10-16T09:43:15.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585effd700 1 -- 172.21.15.158:0/3364387410 learned_addr learned my addr 172.21.15.158:0/3364387410 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:15.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585e7fc700 1 --2- 172.21.15.158:0/3364387410 >> v2:172.21.15.172:3300/0 conn(0x7f58600d1210 0x7f58600ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585effd700 1 -- 172.21.15.158:0/3364387410 >> v2:172.21.15.172:3300/0 conn(0x7f58600d1210 msgr2=0x7f58600ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:15.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585effd700 1 --2- 172.21.15.158:0/3364387410 >> v2:172.21.15.172:3300/0 conn(0x7f58600d1210 0x7f58600ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585effd700 1 -- 172.21.15.158:0/3364387410 >> v1:172.21.15.172:6789/0 conn(0x7f58600ce950 legacy=0x7f58600b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585effd700 1 -- 172.21.15.158:0/3364387410 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600d2fc0 con 0x7f586015d7c0 2023-10-16T09:43:15.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585effd700 1 --2- 172.21.15.158:0/3364387410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f586015d7c0 0x7f58600d27b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f58600b9750 tx=0x7f5848009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61779cf561e66a0a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585dffb700 1 -- 172.21.15.158:0/3364387410 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800b870 con 0x7f586015d7c0 2023-10-16T09:43:15.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585dffb700 1 -- 172.21.15.158:0/3364387410 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584800b9d0 con 0x7f586015d7c0 2023-10-16T09:43:15.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.809+0000 7f585dffb700 1 -- 172.21.15.158:0/3364387410 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800bd60 con 0x7f586015d7c0 2023-10-16T09:43:15.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 -- 172.21.15.158:0/3364387410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f586015d7c0 msgr2=0x7f58600d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 --2- 172.21.15.158:0/3364387410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f586015d7c0 0x7f58600d27b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f58600b9750 tx=0x7f5848009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 -- 172.21.15.158:0/3364387410 shutdown_connections 2023-10-16T09:43:15.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 --2- 172.21.15.158:0/3364387410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f586015d7c0 0x7f58600d27b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 --2- 172.21.15.158:0/3364387410 >> v2:172.21.15.172:3300/0 conn(0x7f58600d1210 0x7f58600ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 -- 172.21.15.158:0/3364387410 >> 172.21.15.158:0/3364387410 conn(0x7f58600669f0 msgr2=0x7f5860066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:15.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 -- 172.21.15.158:0/3364387410 shutdown_connections 2023-10-16T09:43:15.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.810+0000 7f58652cd700 1 -- 172.21.15.158:0/3364387410 wait complete. 2023-10-16T09:43:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.811+0000 7f58652cd700 1 Processor -- start 2023-10-16T09:43:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.811+0000 7f58652cd700 1 -- start start 2023-10-16T09:43:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58652cd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58600ce950 0x7f586015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58652cd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58600d1210 0x7f586015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58652cd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f586015d7c0 0x7f58600b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58652cd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586015e410 con 0x7f58600ce950 2023-10-16T09:43:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58652cd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586015e6f0 con 0x7f58600d1210 2023-10-16T09:43:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58652cd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586015e9d0 con 0x7f586015d7c0 2023-10-16T09:43:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58600ce950 0x7f586015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58600ce950 0x7f586015dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45672/0 (socket says 172.21.15.158:45672) 2023-10-16T09:43:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 -- 172.21.15.158:0/3554401958 learned_addr learned my addr 172.21.15.158:0/3554401958 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:15.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585e7fc700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58600d1210 0x7f586015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585f7fe700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f586015d7c0 0x7f58600b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 -- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f586015d7c0 msgr2=0x7f58600b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f586015d7c0 0x7f58600b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 -- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58600d1210 msgr2=0x7f586015fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:15.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58600d1210 0x7f586015fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:15.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 -- 172.21.15.158:0/3554401958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5848009550 con 0x7f58600ce950 2023-10-16T09:43:15.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f585effd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58600ce950 0x7f586015dba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5848000c00 tx=0x7f5848004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58577fe700 1 -- 172.21.15.158:0/3554401958 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848004480 con 0x7f58600ce950 2023-10-16T09:43:15.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58577fe700 1 -- 172.21.15.158:0/3554401958 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58480045e0 con 0x7f58600ce950 2023-10-16T09:43:15.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.812+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586015ecb0 con 0x7f58600ce950 2023-10-16T09:43:15.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.813+0000 7f58577fe700 1 -- 172.21.15.158:0/3554401958 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58480038e0 con 0x7f58600ce950 2023-10-16T09:43:15.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.813+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586015e0e0 con 0x7f58600ce950 2023-10-16T09:43:15.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.814+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586015f1d0 con 0x7f58600ce950 2023-10-16T09:43:15.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.814+0000 7f58577fe700 1 -- 172.21.15.158:0/3554401958 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f5848021070 con 0x7f58600ce950 2023-10-16T09:43:15.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.814+0000 7f58577fe700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f584006ae70 0x7f584006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:15.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.814+0000 7f58577fe700 1 -- 172.21.15.158:0/3554401958 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f584808a650 con 0x7f58600ce950 2023-10-16T09:43:15.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.814+0000 7f585e7fc700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f584006ae70 0x7f584006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:15.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.815+0000 7f585e7fc700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f584006ae70 0x7f584006d330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58600b2920 tx=0x7f5850008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:15.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:15.819+0000 7f58577fe700 1 -- 172.21.15.158:0/3554401958 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f586015f1d0 con 0x7f58600ce950 2023-10-16T09:43:16.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.056+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f586015f1d0 con 0x7f58600ce950 2023-10-16T09:43:16.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.056+0000 7f58577fe700 1 -- 172.21.15.158:0/3554401958 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1327 (secure 0 0 0) 0x7f586015f1d0 con 0x7f58600ce950 2023-10-16T09:43:16.059 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:16.060 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-16T09:43:16.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f584006ae70 msgr2=0x7f584006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:16.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f584006ae70 0x7f584006d330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58600b2920 tx=0x7f5850008040 comp rx=0 tx=0).stop 2023-10-16T09:43:16.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58600ce950 msgr2=0x7f586015dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:16.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58600ce950 0x7f586015dba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5848000c00 tx=0x7f5848004750 comp rx=0 tx=0).stop 2023-10-16T09:43:16.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 shutdown_connections 2023-10-16T09:43:16.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f584006ae70 0x7f584006d330 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:16.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f586015d7c0 0x7f58600b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:16.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58600ce950 0x7f586015dba0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:16.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.058+0000 7f58652cd700 1 --2- 172.21.15.158:0/3554401958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58600d1210 0x7f586015fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:16.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.059+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 >> 172.21.15.158:0/3554401958 conn(0x7f58600669f0 msgr2=0x7f58600ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:16.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.059+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 shutdown_connections 2023-10-16T09:43:16.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:16.059+0000 7f58652cd700 1 -- 172.21.15.158:0/3554401958 wait complete. 2023-10-16T09:43:16.064 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 2 2023-10-16T09:43:18.074 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:18.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.194+0000 7fe232979700 1 Processor -- start 2023-10-16T09:43:18.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.194+0000 7fe232979700 1 -- start start 2023-10-16T09:43:18.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe232979700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe22c0ce640 0x7fe22c0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe232979700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c0cef60 con 0x7fe22c0c65b0 2023-10-16T09:43:18.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe232979700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c15d800 con 0x7fe22c0cae30 2023-10-16T09:43:18.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe232979700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c15e480 con 0x7fe22c0ce640 2023-10-16T09:43:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe22c0c65b0 0x7fe22c06f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45926/0 (socket says 172.21.15.158:45926) 2023-10-16T09:43:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22b7fe700 1 -- 172.21.15.158:0/1728970625 learned_addr learned my addr 172.21.15.158:0/1728970625 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22bfff700 1 --2- 172.21.15.158:0/1728970625 >> v2:172.21.15.172:3300/0 conn(0x7fe22c0ce640 0x7fe22c0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936198765 0 0) 0x7fe22c0cef60 con 0x7fe22c0c65b0 2023-10-16T09:43:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2180035b0 con 0x7fe22c0c65b0 2023-10-16T09:43:18.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1657515629 0 0) 0x7fe2180035b0 con 0x7fe22c0c65b0 2023-10-16T09:43:18.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 >> v2:172.21.15.172:3300/0 conn(0x7fe22c0ce640 msgr2=0x7fe22c0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:18.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22affd700 1 --2- 172.21.15.158:0/1728970625 >> v2:172.21.15.172:3300/0 conn(0x7fe22c0ce640 0x7fe22c0cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.195+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 >> v1:172.21.15.172:6789/0 conn(0x7fe22c0cae30 legacy=0x7fe22c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c15dae0 con 0x7fe22c0c65b0 2023-10-16T09:43:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe214001670 con 0x7fe22c0c65b0 2023-10-16T09:43:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 >> v1:172.21.15.158:6789/0 conn(0x7fe22c0c65b0 legacy=0x7fe22c06f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe218003a00 0x7fe218003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe218004850 0x7fe218004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe21800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c15e480 con 0x7fe218005210 2023-10-16T09:43:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21800c0b0 con 0x7fe218004850 2023-10-16T09:43:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21800c270 con 0x7fe218003a00 2023-10-16T09:43:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe21800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22b7fe700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe218004850 0x7fe218004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22bfff700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe218003a00 0x7fe218003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 -- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe218003a00 msgr2=0x7fe218003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:18.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe218003a00 0x7fe218003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 -- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe218004850 msgr2=0x7fe218004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:18.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe218004850 0x7fe218004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 -- 172.21.15.158:0/1728970625 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2180035b0 con 0x7fe218005210 2023-10-16T09:43:18.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22b7fe700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe218004850 0x7fe218004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:18.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 -- 172.21.15.158:0/1728970625 reap_dead start 2023-10-16T09:43:18.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe230f16700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe21800ba10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe22400c190 tx=0x7fe218005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45267092a8f1ff0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224001db0 con 0x7fe218005210 2023-10-16T09:43:18.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22400ccf0 con 0x7fe218005210 2023-10-16T09:43:18.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.196+0000 7fe22affd700 1 -- 172.21.15.158:0/1728970625 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224003c40 con 0x7fe218005210 2023-10-16T09:43:18.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.197+0000 7fe232979700 1 -- 172.21.15.158:0/1728970625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 msgr2=0x7fe21800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.197+0000 7fe232979700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe21800ba10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe22400c190 tx=0x7fe218005880 comp rx=0 tx=0).stop 2023-10-16T09:43:18.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.197+0000 7fe232979700 1 -- 172.21.15.158:0/1728970625 shutdown_connections 2023-10-16T09:43:18.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.197+0000 7fe232979700 1 --2- 172.21.15.158:0/1728970625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe21800ba10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.197+0000 7fe232979700 1 -- 172.21.15.158:0/1728970625 >> 172.21.15.158:0/1728970625 conn(0x7fe22c0cc4b0 msgr2=0x7fe22c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:18.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.197+0000 7fe232979700 1 -- 172.21.15.158:0/1728970625 shutdown_connections 2023-10-16T09:43:18.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.197+0000 7fe232979700 1 -- 172.21.15.158:0/1728970625 wait complete. 2023-10-16T09:43:18.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.198+0000 7fe232979700 1 Processor -- start 2023-10-16T09:43:18.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.198+0000 7fe232979700 1 -- start start 2023-10-16T09:43:18.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe232979700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe22c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe232979700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe22c13b510 0x7fe22c159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe232979700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe22c154200 0x7fe22c154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe232979700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c1f2ca0 con 0x7fe218005210 2023-10-16T09:43:18.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe232979700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0c8510 con 0x7fe22c154200 2023-10-16T09:43:18.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe232979700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c1f37a0 con 0x7fe22c13b510 2023-10-16T09:43:18.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe22c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe22c06d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45760/0 (socket says 172.21.15.158:45760) 2023-10-16T09:43:18.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 -- 172.21.15.158:0/3559802742 learned_addr learned my addr 172.21.15.158:0/3559802742 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:18.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe230f16700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe22c154200 0x7fe22c154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22b7fe700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe22c13b510 0x7fe22c159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 -- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe22c13b510 msgr2=0x7fe22c159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe22c13b510 0x7fe22c159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 -- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe22c154200 msgr2=0x7fe22c154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe22c154200 0x7fe22c154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 -- 172.21.15.158:0/3559802742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c1f2eb0 con 0x7fe218005210 2023-10-16T09:43:18.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.199+0000 7fe22bfff700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe22c06d880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009fd0 tx=0x7fe21c011c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.200+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00f750 con 0x7fe218005210 2023-10-16T09:43:18.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.200+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c00f8d0 con 0x7fe218005210 2023-10-16T09:43:18.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.200+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00fc80 con 0x7fe218005210 2023-10-16T09:43:18.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.200+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c06ddc0 con 0x7fe218005210 2023-10-16T09:43:18.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.200+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c1f3430 con 0x7fe218005210 2023-10-16T09:43:18.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.201+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c0681b0 con 0x7fe218005210 2023-10-16T09:43:18.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.201+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fe21c014440 con 0x7fe218005210 2023-10-16T09:43:18.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.202+0000 7fe228ff9700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe20c06ae70 0x7fe20c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.202+0000 7fe22b7fe700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe20c06ae70 0x7fe20c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.202+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe21c089770 con 0x7fe218005210 2023-10-16T09:43:18.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.202+0000 7fe22b7fe700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe20c06ae70 0x7fe20c06d330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe214001200 tx=0x7fe214001110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.206+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe21c054c50 con 0x7fe218005210 2023-10-16T09:43:18.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.218+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe21c055020 con 0x7fe218005210 2023-10-16T09:43:18.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.448+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe22c0c5300 con 0x7fe218005210 2023-10-16T09:43:18.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.449+0000 7fe228ff9700 1 -- 172.21.15.158:0/3559802742 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3706 (secure 0 0 0) 0x7fe21c022e90 con 0x7fe218005210 2023-10-16T09:43:18.452 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:18.452 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-16T09:43:18.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe20c06ae70 msgr2=0x7fe20c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe20c06ae70 0x7fe20c06d330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe214001200 tx=0x7fe214001110 comp rx=0 tx=0).stop 2023-10-16T09:43:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 msgr2=0x7fe22c06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe22c06d880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009fd0 tx=0x7fe21c011c80 comp rx=0 tx=0).stop 2023-10-16T09:43:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 shutdown_connections 2023-10-16T09:43:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe22c154200 0x7fe22c154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe218005210 0x7fe22c06d880 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe20c06ae70 0x7fe20c06d330 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 --2- 172.21.15.158:0/3559802742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe22c13b510 0x7fe22c159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 >> 172.21.15.158:0/3559802742 conn(0x7fe22c0cc4b0 msgr2=0x7fe22c06b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 shutdown_connections 2023-10-16T09:43:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.451+0000 7fe232979700 1 -- 172.21.15.158:0/3559802742 wait complete. 2023-10-16T09:43:18.457 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 3 2023-10-16T09:43:18.466 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:18.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.583+0000 7f45fe867700 1 Processor -- start 2023-10-16T09:43:18.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.583+0000 7f45fe867700 1 -- start start 2023-10-16T09:43:18.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.584+0000 7f45fe867700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f45f80d0b40 0x7f45f80cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.584+0000 7f45fe867700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f80cc110 con 0x7f45f80b3a60 2023-10-16T09:43:18.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.584+0000 7f45fe867700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f815f160 con 0x7f45f81524a0 2023-10-16T09:43:18.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.584+0000 7f45fe867700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f80cc350 con 0x7f45f80d0b40 2023-10-16T09:43:18.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.584+0000 7f45f77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f45f80b3a60 0x7f45f80d0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45944/0 (socket says 172.21.15.158:45944) 2023-10-16T09:43:18.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.584+0000 7f45f77fe700 1 -- 172.21.15.158:0/2641963243 learned_addr learned my addr 172.21.15.158:0/2641963243 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:18.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.584+0000 7f45fce04700 1 --2- 172.21.15.158:0/2641963243 >> v2:172.21.15.172:3300/0 conn(0x7f45f80d0b40 0x7f45f80cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2641963243 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651899165 0 0) 0x7f45e8001110 con 0x7f45f80b3a60 2023-10-16T09:43:18.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2641963243 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e00035b0 con 0x7f45f80b3a60 2023-10-16T09:43:18.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2641963243 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980359438 0 0) 0x7f45f815f160 con 0x7f45f81524a0 2023-10-16T09:43:18.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2641963243 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e8001110 con 0x7f45f81524a0 2023-10-16T09:43:18.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45fce04700 1 -- 172.21.15.158:0/2641963243 >> v1:172.21.15.172:6789/0 conn(0x7f45f81524a0 legacy=0x7f45f80cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45fce04700 1 -- 172.21.15.158:0/2641963243 >> v1:172.21.15.158:6789/0 conn(0x7f45f80b3a60 legacy=0x7f45f80d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45fce04700 1 -- 172.21.15.158:0/2641963243 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f815f440 con 0x7f45f80d0b40 2023-10-16T09:43:18.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45fce04700 1 --2- 172.21.15.158:0/2641963243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80d0b40 0x7f45f80cd720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009d90 tx=0x7f45f000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6116239464d3d31 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2641963243 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f000e070 con 0x7f45f80d0b40 2023-10-16T09:43:18.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2641963243 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f000bb20 con 0x7f45f80d0b40 2023-10-16T09:43:18.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.585+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2641963243 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0012430 con 0x7f45f80d0b40 2023-10-16T09:43:18.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.586+0000 7f45fe867700 1 -- 172.21.15.158:0/2641963243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80d0b40 msgr2=0x7f45f80cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.586+0000 7f45fe867700 1 --2- 172.21.15.158:0/2641963243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80d0b40 0x7f45f80cd720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009d90 tx=0x7f45f000b330 comp rx=0 tx=0).stop 2023-10-16T09:43:18.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.586+0000 7f45fe867700 1 -- 172.21.15.158:0/2641963243 shutdown_connections 2023-10-16T09:43:18.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.586+0000 7f45fe867700 1 --2- 172.21.15.158:0/2641963243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80d0b40 0x7f45f80cd720 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.586+0000 7f45fe867700 1 -- 172.21.15.158:0/2641963243 >> 172.21.15.158:0/2641963243 conn(0x7f45f806a8d0 msgr2=0x7f45f806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:18.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.586+0000 7f45fe867700 1 -- 172.21.15.158:0/2641963243 shutdown_connections 2023-10-16T09:43:18.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.586+0000 7f45fe867700 1 -- 172.21.15.158:0/2641963243 wait complete. 2023-10-16T09:43:18.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.587+0000 7f45fe867700 1 Processor -- start 2023-10-16T09:43:18.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.588+0000 7f45fe867700 1 -- start start 2023-10-16T09:43:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.588+0000 7f45fe867700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b3a60 0x7f45f80c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.588+0000 7f45fe867700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f80d0b40 0x7f45f814e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.588+0000 7f45fe867700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f81524a0 0x7f45f8156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.588+0000 7f45fe867700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f8157ac0 con 0x7f45f80d0b40 2023-10-16T09:43:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.588+0000 7f45fe867700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f814e6e0 con 0x7f45f81524a0 2023-10-16T09:43:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.588+0000 7f45fe867700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f8156770 con 0x7f45f80b3a60 2023-10-16T09:43:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f80d0b40 0x7f45f814e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f80d0b40 0x7f45f814e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45776/0 (socket says 172.21.15.158:45776) 2023-10-16T09:43:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 -- 172.21.15.158:0/2502275036 learned_addr learned my addr 172.21.15.158:0/2502275036 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f7fff700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b3a60 0x7f45f80c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45fce04700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f81524a0 0x7f45f8156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 -- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b3a60 msgr2=0x7f45f80c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b3a60 0x7f45f80c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 -- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f81524a0 msgr2=0x7f45f8156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f81524a0 0x7f45f8156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 -- 172.21.15.158:0/2502275036 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f0009a70 con 0x7f45f80d0b40 2023-10-16T09:43:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f77fe700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f80d0b40 0x7f45f814e1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45e8001eb0 tx=0x7f45e800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f4ff9700 1 -- 172.21.15.158:0/2502275036 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8005d70 con 0x7f45f80d0b40 2023-10-16T09:43:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f4ff9700 1 -- 172.21.15.158:0/2502275036 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e80070d0 con 0x7f45f80d0b40 2023-10-16T09:43:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45f4ff9700 1 -- 172.21.15.158:0/2502275036 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8007480 con 0x7f45f80d0b40 2023-10-16T09:43:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f8156a20 con 0x7f45f80d0b40 2023-10-16T09:43:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.589+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f8156fe0 con 0x7f45f80d0b40 2023-10-16T09:43:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.590+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f806f7e0 con 0x7f45f80d0b40 2023-10-16T09:43:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.591+0000 7f45f4ff9700 1 -- 172.21.15.158:0/2502275036 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f45e80075e0 con 0x7f45f80d0b40 2023-10-16T09:43:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.591+0000 7f45f4ff9700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f45e406af00 0x7f45e406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.591+0000 7f45f7fff700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f45e406af00 0x7f45e406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.591+0000 7f45f4ff9700 1 -- 172.21.15.158:0/2502275036 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f45e8088640 con 0x7f45f80d0b40 2023-10-16T09:43:18.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.591+0000 7f45f7fff700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f45e406af00 0x7f45e406d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0011e0 tx=0x7f45ec009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.596+0000 7f45f4ff9700 1 -- 172.21.15.158:0/2502275036 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f45e8053a40 con 0x7f45f80d0b40 2023-10-16T09:43:18.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.788+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45f80cc110 con 0x7f45f80d0b40 2023-10-16T09:43:18.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.789+0000 7f45f4ff9700 1 -- 172.21.15.158:0/2502275036 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12862 (secure 0 0 0) 0x7f45f80cc110 con 0x7f45f80d0b40 2023-10-16T09:43:18.791 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:18.792 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":16,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:17.886200+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:18.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f45e406af00 msgr2=0x7f45e406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f45e406af00 0x7f45e406d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0011e0 tx=0x7f45ec009040 comp rx=0 tx=0).stop 2023-10-16T09:43:18.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f80d0b40 msgr2=0x7f45f814e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f80d0b40 0x7f45f814e1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45e8001eb0 tx=0x7f45e800c9e0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 shutdown_connections 2023-10-16T09:43:18.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f81524a0 0x7f45f8156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f45e406af00 0x7f45e406d3c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b3a60 0x7f45f80c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 --2- 172.21.15.158:0/2502275036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f80d0b40 0x7f45f814e1a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 >> 172.21.15.158:0/2502275036 conn(0x7f45f806a8d0 msgr2=0x7f45f80b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:18.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 shutdown_connections 2023-10-16T09:43:18.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.791+0000 7f45fe867700 1 -- 172.21.15.158:0/2502275036 wait complete. 2023-10-16T09:43:18.805 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:43:18.806 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:43:18.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.928+0000 7f61c2198700 1 Processor -- start 2023-10-16T09:43:18.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.928+0000 7f61c2198700 1 -- start start 2023-10-16T09:43:18.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61c2198700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f61bc15b590 0x7f61bc0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61c2198700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61c2198700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61bc0cbaf0 0x7f61bc0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61c2198700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc0cc4c0 con 0x7f61bc0cbaf0 2023-10-16T09:43:18.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61c2198700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc0cf4c0 con 0x7f61bc15b590 2023-10-16T09:43:18.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61c2198700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc15f970 con 0x7f61bc15f1d0 2023-10-16T09:43:18.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61bbfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61bc0cbaf0 0x7f61bc0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61bbfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61bc0cbaf0 0x7f61bc0cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45794/0 (socket says 172.21.15.158:45794) 2023-10-16T09:43:18.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.929+0000 7f61bbfff700 1 -- 172.21.15.158:0/3155726466 learned_addr learned my addr 172.21.15.158:0/3155726466 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:18.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bb7fe700 1 --2- 172.21.15.158:0/3155726466 >> v2:172.21.15.172:3300/0 conn(0x7f61bc15b590 0x7f61bc0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61baffd700 1 --2- 172.21.15.158:0/3155726466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bbfff700 1 -- 172.21.15.158:0/3155726466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 msgr2=0x7f61bc0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bbfff700 1 --2- 172.21.15.158:0/3155726466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bbfff700 1 -- 172.21.15.158:0/3155726466 >> v2:172.21.15.172:3300/0 conn(0x7f61bc15b590 msgr2=0x7f61bc0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bbfff700 1 --2- 172.21.15.158:0/3155726466 >> v2:172.21.15.172:3300/0 conn(0x7f61bc15b590 0x7f61bc0ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bbfff700 1 -- 172.21.15.158:0/3155726466 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc15fc50 con 0x7f61bc0cbaf0 2023-10-16T09:43:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61baffd700 1 --2- 172.21.15.158:0/3155726466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bb7fe700 1 --2- 172.21.15.158:0/3155726466 >> v2:172.21.15.172:3300/0 conn(0x7f61bc15b590 0x7f61bc0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:18.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61bbfff700 1 --2- 172.21.15.158:0/3155726466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc0cbaf0 0x7f61bc0cbf80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0b3520 tx=0x7f61b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7c41026f30dd3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61ba7fc700 1 -- 172.21.15.158:0/3155726466 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b400e070 con 0x7f61bc0cbaf0 2023-10-16T09:43:18.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61ba7fc700 1 -- 172.21.15.158:0/3155726466 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b400b9c0 con 0x7f61bc0cbaf0 2023-10-16T09:43:18.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61ba7fc700 1 -- 172.21.15.158:0/3155726466 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b400bd70 con 0x7f61bc0cbaf0 2023-10-16T09:43:18.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61c2198700 1 -- 172.21.15.158:0/3155726466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc0cbaf0 msgr2=0x7f61bc0cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61c2198700 1 --2- 172.21.15.158:0/3155726466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc0cbaf0 0x7f61bc0cbf80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0b3520 tx=0x7f61b400b330 comp rx=0 tx=0).stop 2023-10-16T09:43:18.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61c2198700 1 -- 172.21.15.158:0/3155726466 shutdown_connections 2023-10-16T09:43:18.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61c2198700 1 --2- 172.21.15.158:0/3155726466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc0cbaf0 0x7f61bc0cbf80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61c2198700 1 --2- 172.21.15.158:0/3155726466 >> v2:172.21.15.172:3300/0 conn(0x7f61bc15b590 0x7f61bc0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.930+0000 7f61c2198700 1 --2- 172.21.15.158:0/3155726466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.931+0000 7f61c2198700 1 -- 172.21.15.158:0/3155726466 >> 172.21.15.158:0/3155726466 conn(0x7f61bc06a8d0 msgr2=0x7f61bc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:18.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.931+0000 7f61c2198700 1 -- 172.21.15.158:0/3155726466 shutdown_connections 2023-10-16T09:43:18.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.931+0000 7f61c2198700 1 -- 172.21.15.158:0/3155726466 wait complete. 2023-10-16T09:43:18.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.932+0000 7f61c2198700 1 Processor -- start 2023-10-16T09:43:18.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.932+0000 7f61c2198700 1 -- start start 2023-10-16T09:43:18.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61bc0cbaf0 0x7f61bc0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc15b590 0x7f61bc152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0c9660 con 0x7f61bc15b590 2023-10-16T09:43:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0c8dc0 con 0x7f61bc0cbaf0 2023-10-16T09:43:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0c90a0 con 0x7f61bc15f1d0 2023-10-16T09:43:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc15b590 0x7f61bc152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc15b590 0x7f61bc152500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45798/0 (socket says 172.21.15.158:45798) 2023-10-16T09:43:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 -- 172.21.15.158:0/3843276381 learned_addr learned my addr 172.21.15.158:0/3843276381 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61bb7fe700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61bc0cbaf0 0x7f61bc0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61bbfff700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 -- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 msgr2=0x7f61bc0b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 -- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61bc0cbaf0 msgr2=0x7f61bc0c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61bc0cbaf0 0x7f61bc0c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 -- 172.21.15.158:0/3843276381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4009b30 con 0x7f61bc15b590 2023-10-16T09:43:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61baffd700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc15b590 0x7f61bc152500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c7f0 tx=0x7f61ac00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61a3fff700 1 -- 172.21.15.158:0/3843276381 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac011800 con 0x7f61bc15b590 2023-10-16T09:43:18.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61a3fff700 1 -- 172.21.15.158:0/3843276381 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac011960 con 0x7f61bc15b590 2023-10-16T09:43:18.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61bc0c9380 con 0x7f61bc15b590 2023-10-16T09:43:18.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61a3fff700 1 -- 172.21.15.158:0/3843276381 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac011d10 con 0x7f61bc15b590 2023-10-16T09:43:18.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.933+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61bc0c8ab0 con 0x7f61bc15b590 2023-10-16T09:43:18.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.934+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61bc0c99c0 con 0x7f61bc15b590 2023-10-16T09:43:18.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.935+0000 7f61a3fff700 1 -- 172.21.15.158:0/3843276381 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f61ac018360 con 0x7f61bc15b590 2023-10-16T09:43:18.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.935+0000 7f61a3fff700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61a806af00 0x7f61a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:18.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.935+0000 7f61a3fff700 1 -- 172.21.15.158:0/3843276381 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f61ac089220 con 0x7f61bc15b590 2023-10-16T09:43:18.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.935+0000 7f61bb7fe700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61a806af00 0x7f61a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:18.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.935+0000 7f61bb7fe700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61a806af00 0x7f61a806d3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f61b0006ba0 tx=0x7f61b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:18.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:18.940+0000 7f61a3fff700 1 -- 172.21.15.158:0/3843276381 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61bc0c99c0 con 0x7f61bc15b590 2023-10-16T09:43:19.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.132+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61bc0c9da0 con 0x7f61bc15b590 2023-10-16T09:43:19.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.132+0000 7f61a3fff700 1 -- 172.21.15.158:0/3843276381 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61bc0c9da0 con 0x7f61bc15b590 2023-10-16T09:43:19.135 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:43:19.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61a806af00 msgr2=0x7f61a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61a806af00 0x7f61a806d3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f61b0006ba0 tx=0x7f61b0008040 comp rx=0 tx=0).stop 2023-10-16T09:43:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc15b590 msgr2=0x7f61bc152500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc15b590 0x7f61bc152500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c7f0 tx=0x7f61ac00cbb0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 shutdown_connections 2023-10-16T09:43:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61a806af00 0x7f61a806d3c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61bc15f1d0 0x7f61bc0b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61bc0cbaf0 0x7f61bc0c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 --2- 172.21.15.158:0/3843276381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61bc15b590 0x7f61bc152500 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 >> 172.21.15.158:0/3843276381 conn(0x7f61bc06a8d0 msgr2=0x7f61bc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 shutdown_connections 2023-10-16T09:43:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.134+0000 7f61c2198700 1 -- 172.21.15.158:0/3843276381 wait complete. 2023-10-16T09:43:19.149 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T09:43:19.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.267+0000 7f63d1fb9700 1 Processor -- start 2023-10-16T09:43:19.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.267+0000 7f63d1fb9700 1 -- start start 2023-10-16T09:43:19.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63d1fb9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63cc06e5d0 0x7f63cc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63d1fb9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc161270 con 0x7f63cc06ab40 2023-10-16T09:43:19.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63d1fb9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc161510 con 0x7f63cc0729b0 2023-10-16T09:43:19.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63d1fb9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc1617f0 con 0x7f63cc06e5d0 2023-10-16T09:43:19.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63cb7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f63cc06ab40 0x7f63cc0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45984/0 (socket says 172.21.15.158:45984) 2023-10-16T09:43:19.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63cb7fe700 1 -- 172.21.15.158:0/1725749377 learned_addr learned my addr 172.21.15.158:0/1725749377 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:19.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63caffd700 1 --2- 172.21.15.158:0/1725749377 >> v2:172.21.15.172:3300/0 conn(0x7f63cc06e5d0 0x7f63cc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258312474 0 0) 0x7f63cc161270 con 0x7f63cc06ab40 2023-10-16T09:43:19.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b40035b0 con 0x7f63cc06ab40 2023-10-16T09:43:19.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632132810 0 0) 0x7f63b40035b0 con 0x7f63cc06ab40 2023-10-16T09:43:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 >> v2:172.21.15.172:3300/0 conn(0x7f63cc06e5d0 msgr2=0x7f63cc06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 --2- 172.21.15.158:0/1725749377 >> v2:172.21.15.172:3300/0 conn(0x7f63cc06e5d0 0x7f63cc06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 >> v1:172.21.15.172:6789/0 conn(0x7f63cc0729b0 legacy=0x7f63cc15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc161ad0 con 0x7f63cc06ab40 2023-10-16T09:43:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f63bc001670 con 0x7f63cc06ab40 2023-10-16T09:43:19.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 >> v1:172.21.15.158:6789/0 conn(0x7f63cc06ab40 legacy=0x7f63cc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63b4003a00 0x7f63b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4004850 0x7f63b4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63b4005210 0x7f63b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1617f0 con 0x7f63b4005210 2023-10-16T09:43:19.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b400c0b0 con 0x7f63b4003a00 2023-10-16T09:43:19.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.268+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b400c270 con 0x7f63b4004850 2023-10-16T09:43:19.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63b4003a00 0x7f63b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63b4005210 0x7f63b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63caffd700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4004850 0x7f63b4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 -- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4004850 msgr2=0x7f63b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:19.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4004850 0x7f63b4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 -- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63b4003a00 msgr2=0x7f63b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:19.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63b4003a00 0x7f63b4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 -- 172.21.15.158:0/1725749377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40035b0 con 0x7f63b4005210 2023-10-16T09:43:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63b4003a00 0x7f63b4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 -- 172.21.15.158:0/1725749377 reap_dead start 2023-10-16T09:43:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63cbfff700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63b4005210 0x7f63b400ba10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63c400c210 tx=0x7f63b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485e4621a924bb03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4001df0 con 0x7f63b4005210 2023-10-16T09:43:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c400cd70 con 0x7f63b4005210 2023-10-16T09:43:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.269+0000 7f63ca7fc700 1 -- 172.21.15.158:0/1725749377 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4003c40 con 0x7f63b4005210 2023-10-16T09:43:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.270+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1725749377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63b4005210 msgr2=0x7f63b400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.270+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63b4005210 0x7f63b400ba10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63c400c210 tx=0x7f63b4005880 comp rx=0 tx=0).stop 2023-10-16T09:43:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.270+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1725749377 shutdown_connections 2023-10-16T09:43:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.270+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1725749377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63b4005210 0x7f63b400ba10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.270+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1725749377 >> 172.21.15.158:0/1725749377 conn(0x7f63cc0d02b0 msgr2=0x7f63cc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.270+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1725749377 shutdown_connections 2023-10-16T09:43:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.270+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1725749377 wait complete. 2023-10-16T09:43:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.271+0000 7f63d1fb9700 1 Processor -- start 2023-10-16T09:43:19.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.271+0000 7f63d1fb9700 1 -- start start 2023-10-16T09:43:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63d1fb9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4005210 0x7f63cc0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63d1fb9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc154bd0 0x7f63cc0c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63d1fb9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63cc0b50b0 0x7f63cc153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63d1fb9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b4c90 con 0x7f63cc154bd0 2023-10-16T09:43:19.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63d1fb9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b4010 con 0x7f63cc0b50b0 2023-10-16T09:43:19.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63d1fb9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b42f0 con 0x7f63b4005210 2023-10-16T09:43:19.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc154bd0 0x7f63cc0c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc154bd0 0x7f63cc0c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45822/0 (socket says 172.21.15.158:45822) 2023-10-16T09:43:19.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 -- 172.21.15.158:0/1651730388 learned_addr learned my addr 172.21.15.158:0/1651730388 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:19.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63cbfff700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63cc0b50b0 0x7f63cc153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4005210 0x7f63cc0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 -- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4005210 msgr2=0x7f63cc0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4005210 0x7f63cc0c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 -- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63cc0b50b0 msgr2=0x7f63cc153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63cc0b50b0 0x7f63cc153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 -- 172.21.15.158:0/1651730388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0b45d0 con 0x7f63cc154bd0 2023-10-16T09:43:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63caffd700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc154bd0 0x7f63cc0c4900 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f63c0011930 tx=0x7f63c0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63b3fff700 1 -- 172.21.15.158:0/1651730388 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000f730 con 0x7f63cc154bd0 2023-10-16T09:43:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63b3fff700 1 -- 172.21.15.158:0/1651730388 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c000f890 con 0x7f63cc154bd0 2023-10-16T09:43:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc0b48b0 con 0x7f63cc154bd0 2023-10-16T09:43:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.272+0000 7f63b3fff700 1 -- 172.21.15.158:0/1651730388 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000fc20 con 0x7f63cc154bd0 2023-10-16T09:43:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.273+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc0c4e40 con 0x7f63cc154bd0 2023-10-16T09:43:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.274+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc0681b0 con 0x7f63cc154bd0 2023-10-16T09:43:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.274+0000 7f63b3fff700 1 -- 172.21.15.158:0/1651730388 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f63c000fd80 con 0x7f63cc154bd0 2023-10-16T09:43:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.274+0000 7f63b3fff700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b806af00 0x7f63b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.274+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b806af00 0x7f63b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.274+0000 7f63b3fff700 1 -- 172.21.15.158:0/1651730388 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f63c0089800 con 0x7f63cc154bd0 2023-10-16T09:43:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.275+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b806af00 0x7f63b806d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f63bc001200 tx=0x7f63bc001110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.279+0000 7f63b3fff700 1 -- 172.21.15.158:0/1651730388 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63c0054d40 con 0x7f63cc154bd0 2023-10-16T09:43:19.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.472+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f63cc162060 con 0x7f63cc154bd0 2023-10-16T09:43:19.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.472+0000 7f63b3fff700 1 -- 172.21.15.158:0/1651730388 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f63c0058360 con 0x7f63cc154bd0 2023-10-16T09:43:19.475 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:19.476 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:43:19.476 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:43:19.476 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T09:43:19.476 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T09:43:19.476 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:43:19.476 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:43:19.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b806af00 msgr2=0x7f63b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b806af00 0x7f63b806d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f63bc001200 tx=0x7f63bc001110 comp rx=0 tx=0).stop 2023-10-16T09:43:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc154bd0 msgr2=0x7f63cc0c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc154bd0 0x7f63cc0c4900 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f63c0011930 tx=0x7f63c0011c40 comp rx=0 tx=0).stop 2023-10-16T09:43:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 shutdown_connections 2023-10-16T09:43:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63cc0b50b0 0x7f63cc153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b806af00 0x7f63b806d3c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63b4005210 0x7f63cc0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 --2- 172.21.15.158:0/1651730388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc154bd0 0x7f63cc0c4900 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.474+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 >> 172.21.15.158:0/1651730388 conn(0x7f63cc0d02b0 msgr2=0x7f63cc0cb790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:19.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.475+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 shutdown_connections 2023-10-16T09:43:19.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.475+0000 7f63d1fb9700 1 -- 172.21.15.158:0/1651730388 wait complete. 2023-10-16T09:43:19.489 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T09:43:19.489 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T09:43:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.610+0000 7f347dbfb700 1 Processor -- start 2023-10-16T09:43:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f347dbfb700 1 -- start start 2023-10-16T09:43:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f347dbfb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f347815da40 0x7f34780caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f347dbfb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34780cbd30 con 0x7f34780cc590 2023-10-16T09:43:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f347dbfb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34780cc1c0 con 0x7f347806afe0 2023-10-16T09:43:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f347dbfb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347815de20 con 0x7f347815da40 2023-10-16T09:43:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f34777fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f347815da40 0x7f34780caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f346ffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f34780cc590 0x7f347806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46004/0 (socket says 172.21.15.158:46004) 2023-10-16T09:43:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.611+0000 7f346ffff700 1 -- 172.21.15.158:0/2464355386 learned_addr learned my addr 172.21.15.158:0/2464355386 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f34777fe700 1 -- 172.21.15.158:0/2464355386 >> v1:172.21.15.172:6789/0 conn(0x7f347806afe0 legacy=0x7f34780afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:19.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f34777fe700 1 -- 172.21.15.158:0/2464355386 >> v1:172.21.15.158:6789/0 conn(0x7f34780cc590 legacy=0x7f347806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f34777fe700 1 -- 172.21.15.158:0/2464355386 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34780ce260 con 0x7f347815da40 2023-10-16T09:43:19.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f3476ffd700 1 -- 172.21.15.158:0/2464355386 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044683304 0 0) 0x7f34780cbd30 con 0x7f34780cc590 2023-10-16T09:43:19.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f34777fe700 1 --2- 172.21.15.158:0/2464355386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780caa20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3468009840 tx=0x7f3468009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d491dd86169d05e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:19.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f3476ffd700 1 -- 172.21.15.158:0/2464355386 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468011830 con 0x7f347815da40 2023-10-16T09:43:19.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f3476ffd700 1 -- 172.21.15.158:0/2464355386 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3468011990 con 0x7f347815da40 2023-10-16T09:43:19.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.612+0000 7f3476ffd700 1 -- 172.21.15.158:0/2464355386 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468011d20 con 0x7f347815da40 2023-10-16T09:43:19.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.613+0000 7f347dbfb700 1 -- 172.21.15.158:0/2464355386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 msgr2=0x7f34780caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.613+0000 7f347dbfb700 1 --2- 172.21.15.158:0/2464355386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780caa20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3468009840 tx=0x7f3468009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.613+0000 7f347dbfb700 1 -- 172.21.15.158:0/2464355386 shutdown_connections 2023-10-16T09:43:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.613+0000 7f347dbfb700 1 --2- 172.21.15.158:0/2464355386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780caa20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.613+0000 7f347dbfb700 1 -- 172.21.15.158:0/2464355386 >> 172.21.15.158:0/2464355386 conn(0x7f3478099a10 msgr2=0x7f34780d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.613+0000 7f347dbfb700 1 -- 172.21.15.158:0/2464355386 shutdown_connections 2023-10-16T09:43:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.613+0000 7f347dbfb700 1 -- 172.21.15.158:0/2464355386 wait complete. 2023-10-16T09:43:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.614+0000 7f347dbfb700 1 Processor -- start 2023-10-16T09:43:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.614+0000 7f347dbfb700 1 -- start start 2023-10-16T09:43:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f347806afe0 0x7f347815a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34780cc590 0x7f34781f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c4e00 con 0x7f347815da40 2023-10-16T09:43:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34781f7170 con 0x7f34780cc590 2023-10-16T09:43:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c41f0 con 0x7f347806afe0 2023-10-16T09:43:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45838/0 (socket says 172.21.15.158:45838) 2023-10-16T09:43:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 -- 172.21.15.158:0/4049171159 learned_addr learned my addr 172.21.15.158:0/4049171159 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f346ffff700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34780cc590 0x7f34781f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f34777fe700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f347806afe0 0x7f347815a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 -- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f347806afe0 msgr2=0x7f347815a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f347806afe0 0x7f347815a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 -- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34780cc590 msgr2=0x7f34781f6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34780cc590 0x7f34781f6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 -- 172.21.15.158:0/4049171159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680094a0 con 0x7f347815da40 2023-10-16T09:43:19.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3477fff700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780b38b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f347000cbc0 tx=0x7f347000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:19.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3474ff9700 1 -- 172.21.15.158:0/4049171159 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470003c00 con 0x7f347815da40 2023-10-16T09:43:19.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3474ff9700 1 -- 172.21.15.158:0/4049171159 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3470003d60 con 0x7f347815da40 2023-10-16T09:43:19.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34780c44d0 con 0x7f347815da40 2023-10-16T09:43:19.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f3474ff9700 1 -- 172.21.15.158:0/4049171159 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347000aae0 con 0x7f347815da40 2023-10-16T09:43:19.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.615+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34780c4a90 con 0x7f347815da40 2023-10-16T09:43:19.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.616+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34780c5190 con 0x7f347815da40 2023-10-16T09:43:19.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.617+0000 7f3474ff9700 1 -- 172.21.15.158:0/4049171159 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f347000ac40 con 0x7f347815da40 2023-10-16T09:43:19.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.617+0000 7f3474ff9700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f345806af00 0x7f345806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:19.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.617+0000 7f34777fe700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f345806af00 0x7f345806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:19.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.617+0000 7f3474ff9700 1 -- 172.21.15.158:0/4049171159 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f34700892c0 con 0x7f347815da40 2023-10-16T09:43:19.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.617+0000 7f34777fe700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f345806af00 0x7f345806d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34680090b0 tx=0x7f346801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:19.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.621+0000 7f3474ff9700 1 -- 172.21.15.158:0/4049171159 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34700546c0 con 0x7f347815da40 2023-10-16T09:43:19.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.812+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f34780681b0 con 0x7f347815da40 2023-10-16T09:43:19.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.895+0000 7f3474ff9700 1 -- 172.21.15.158:0/4049171159 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3470057ce0 con 0x7f347815da40 2023-10-16T09:43:19.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f345806af00 msgr2=0x7f345806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f345806af00 0x7f345806d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34680090b0 tx=0x7f346801c040 comp rx=0 tx=0).stop 2023-10-16T09:43:19.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 msgr2=0x7f34780b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:19.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780b38b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f347000cbc0 tx=0x7f347000ced0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 shutdown_connections 2023-10-16T09:43:19.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347815da40 0x7f34780b38b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f345806af00 0x7f345806d3c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f347806afe0 0x7f347815a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 --2- 172.21.15.158:0/4049171159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34780cc590 0x7f34781f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 >> 172.21.15.158:0/4049171159 conn(0x7f3478099a10 msgr2=0x7f34780b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 shutdown_connections 2023-10-16T09:43:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:19.897+0000 7f347dbfb700 1 -- 172.21.15.158:0/4049171159 wait complete. 2023-10-16T09:43:19.903 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T09:43:19.915 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-16T09:43:20.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.040+0000 7f5ec0a2d700 1 Processor -- start 2023-10-16T09:43:20.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.040+0000 7f5ec0a2d700 1 -- start start 2023-10-16T09:43:20.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5ec0a2d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:20.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5ec0a2d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ebc0cf6b0 0x7f5ebc0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:20.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5ec0a2d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc070db0 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5ec0a2d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ebc160220 con 0x7f5ebc0cf6b0 2023-10-16T09:43:20.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5ec0a2d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc15f6a0 con 0x7f5ebc070410 2023-10-16T09:43:20.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5eb9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:20.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5ebad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ebc0cf6b0 0x7f5ebc0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:20.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5eb9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:56676/0 (socket says 172.21.15.158:56676) 2023-10-16T09:43:20.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.041+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/481462640 learned_addr learned my addr 172.21.15.158:0/481462640 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:20.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/481462640 >> v2:172.21.15.172:3300/0 conn(0x7f5ebc0cf6b0 msgr2=0x7f5ebc0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:20.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb9d9b700 1 --2- 172.21.15.158:0/481462640 >> v2:172.21.15.172:3300/0 conn(0x7f5ebc0cf6b0 0x7f5ebc0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/481462640 >> v1:172.21.15.172:6789/0 conn(0x7f5ebc0cbf50 legacy=0x7f5ebc144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:20.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/481462640 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc15f980 con 0x7f5ebc070410 2023-10-16T09:43:20.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5ebad9d700 1 --2- 172.21.15.158:0/481462640 >> v2:172.21.15.172:3300/0 conn(0x7f5ebc0cf6b0 0x7f5ebc0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:20.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb9d9b700 1 --2- 172.21.15.158:0/481462640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc070870 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098f0 tx=0x7f5eac009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f694364c1ea46508 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:20.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb959a700 1 -- 172.21.15.158:0/481462640 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac011830 con 0x7f5ebc070410 2023-10-16T09:43:20.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb959a700 1 -- 172.21.15.158:0/481462640 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eac011990 con 0x7f5ebc070410 2023-10-16T09:43:20.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.042+0000 7f5eb959a700 1 -- 172.21.15.158:0/481462640 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac011d20 con 0x7f5ebc070410 2023-10-16T09:43:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/481462640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 msgr2=0x7f5ebc070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/481462640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc070870 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098f0 tx=0x7f5eac009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/481462640 shutdown_connections 2023-10-16T09:43:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/481462640 >> v2:172.21.15.172:3300/0 conn(0x7f5ebc0cf6b0 0x7f5ebc0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/481462640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc070870 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/481462640 >> 172.21.15.158:0/481462640 conn(0x7f5ebc0d1410 msgr2=0x7f5ebc0d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/481462640 shutdown_connections 2023-10-16T09:43:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.043+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/481462640 wait complete. 2023-10-16T09:43:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.044+0000 7f5ec0a2d700 1 Processor -- start 2023-10-16T09:43:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.044+0000 7f5ec0a2d700 1 -- start start 2023-10-16T09:43:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5ec0a2d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5ec0a2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ebc0cbf50 0x7f5ebc1590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5ec0a2d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ebc0cf6b0 0x7f5ebc1f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5ec0a2d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc0c7dc0 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5ec0a2d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc159610 con 0x7f5ebc0cf6b0 2023-10-16T09:43:20.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5ec0a2d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc0c9ad0 con 0x7f5ebc070410 2023-10-16T09:43:20.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ebc0cbf50 0x7f5ebc1590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:20.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ebc0cbf50 0x7f5ebc1590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45848/0 (socket says 172.21.15.158:45848) 2023-10-16T09:43:20.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/1478412272 learned_addr learned my addr 172.21.15.158:0/1478412272 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:20.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eba59c700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:20.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5ebad9d700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ebc0cf6b0 0x7f5ebc1f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:20.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 msgr2=0x7f5ebc0c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:20.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc0c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ebc0cf6b0 msgr2=0x7f5ebc1f0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:20.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ebc0cf6b0 0x7f5ebc1f0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.045+0000 7f5eb9d9b700 1 -- 172.21.15.158:0/1478412272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4009a70 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.046+0000 7f5eb9d9b700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ebc0cbf50 0x7f5ebc1590d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009c40 tx=0x7f5eac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:20.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.046+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1478412272 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac0044d0 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.046+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1478412272 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eac01e070 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.046+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac009550 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.046+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1478412272 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac0038e0 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.046+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc0ca090 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.047+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ebc0681b0 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.047+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1478412272 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5eac003a40 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.048+0000 7f5eaaffd700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e9c06af00 0x7f5e9c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:20.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.048+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1478412272 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5eac08a250 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.048+0000 7f5eba59c700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e9c06af00 0x7f5e9c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:20.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.048+0000 7f5eba59c700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e9c06af00 0x7f5e9c06d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0b7000 tx=0x7f5eb0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:20.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.052+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1478412272 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5eac0554d0 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.247+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f5ebc0d2300 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.906+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1478412272 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f5eac058af0 con 0x7f5ebc0cbf50 2023-10-16T09:43:20.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.914+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e9c06af00 msgr2=0x7f5e9c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:20.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.914+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e9c06af00 0x7f5e9c06d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0b7000 tx=0x7f5eb0009040 comp rx=0 tx=0).stop 2023-10-16T09:43:20.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.914+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ebc0cbf50 msgr2=0x7f5ebc1590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:20.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.914+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ebc0cbf50 0x7f5ebc1590d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009c40 tx=0x7f5eac004750 comp rx=0 tx=0).stop 2023-10-16T09:43:20.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.917+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 shutdown_connections 2023-10-16T09:43:20.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.917+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ebc0cf6b0 0x7f5ebc1f0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.917+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e9c06af00 0x7f5e9c06d3c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.917+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ebc070410 0x7f5ebc0c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.917+0000 7f5ec0a2d700 1 --2- 172.21.15.158:0/1478412272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ebc0cbf50 0x7f5ebc1590d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:20.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.917+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 >> 172.21.15.158:0/1478412272 conn(0x7f5ebc0d1410 msgr2=0x7f5ebc15a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:20.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.918+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 shutdown_connections 2023-10-16T09:43:20.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:20.918+0000 7f5ec0a2d700 1 -- 172.21.15.158:0/1478412272 wait complete. 2023-10-16T09:43:20.923 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T09:43:20.939 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T09:43:21.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.060+0000 7f46e7e2d700 1 Processor -- start 2023-10-16T09:43:21.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.061+0000 7f46e7e2d700 1 -- start start 2023-10-16T09:43:21.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.061+0000 7f46e7e2d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46e015b590 0x7f46e00ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:21.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.061+0000 7f46e7e2d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e00cbaf0 0x7f46e00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:21.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.061+0000 7f46e7e2d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00cf4c0 con 0x7f46e015f1d0 2023-10-16T09:43:21.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.061+0000 7f46e7e2d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e00cc4c0 con 0x7f46e015b590 2023-10-16T09:43:21.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.061+0000 7f46e7e2d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00b49e0 con 0x7f46e00cbaf0 2023-10-16T09:43:21.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46e015b590 0x7f46e00ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:21.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46e015b590 0x7f46e00ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45856/0 (socket says 172.21.15.158:45856) 2023-10-16T09:43:21.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2442989565 learned_addr learned my addr 172.21.15.158:0/2442989565 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:21.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e63ca700 1 --2- 172.21.15.158:0/2442989565 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e00cbaf0 0x7f46e00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:21.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2442989565 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e00cbaf0 msgr2=0x7f46e00cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:21.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 --2- 172.21.15.158:0/2442989565 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e00cbaf0 0x7f46e00cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2442989565 >> v1:172.21.15.172:6789/0 conn(0x7f46e015f1d0 legacy=0x7f46e00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:21.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2442989565 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e015f970 con 0x7f46e015b590 2023-10-16T09:43:21.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e63ca700 1 --2- 172.21.15.158:0/2442989565 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e00cbaf0 0x7f46e00cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:21.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e5bc9700 1 --2- 172.21.15.158:0/2442989565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e015b590 0x7f46e00ce510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009920 tx=0x7f46d0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c728d55d7714249 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e4bc7700 1 -- 172.21.15.158:0/2442989565 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0004230 con 0x7f46e015b590 2023-10-16T09:43:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e4bc7700 1 -- 172.21.15.158:0/2442989565 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0004390 con 0x7f46e015b590 2023-10-16T09:43:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.062+0000 7f46e4bc7700 1 -- 172.21.15.158:0/2442989565 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0009e70 con 0x7f46e015b590 2023-10-16T09:43:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2442989565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e015b590 msgr2=0x7f46e00ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2442989565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e015b590 0x7f46e00ce510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009920 tx=0x7f46d0009c30 comp rx=0 tx=0).stop 2023-10-16T09:43:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2442989565 shutdown_connections 2023-10-16T09:43:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2442989565 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e00cbaf0 0x7f46e00cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2442989565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e015b590 0x7f46e00ce510 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2442989565 >> 172.21.15.158:0/2442989565 conn(0x7f46e006a8d0 msgr2=0x7f46e006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:21.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2442989565 shutdown_connections 2023-10-16T09:43:21.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.063+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2442989565 wait complete. 2023-10-16T09:43:21.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.064+0000 7f46e7e2d700 1 Processor -- start 2023-10-16T09:43:21.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.064+0000 7f46e7e2d700 1 -- start start 2023-10-16T09:43:21.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e00cbaf0 0x7f46e01f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:21.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46e015b590 0x7f46e01f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:21.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e015f1d0 0x7f46e00b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:21.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e01f3560 con 0x7f46e00cbaf0 2023-10-16T09:43:21.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e01f3840 con 0x7f46e015b590 2023-10-16T09:43:21.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e01f6330 con 0x7f46e015f1d0 2023-10-16T09:43:21.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e00cbaf0 0x7f46e01f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:21.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e00cbaf0 0x7f46e01f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45862/0 (socket says 172.21.15.158:45862) 2023-10-16T09:43:21.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2344390321 learned_addr learned my addr 172.21.15.158:0/2344390321 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:21.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e53c8700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46e015b590 0x7f46e01f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:21.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e63ca700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e015f1d0 0x7f46e00b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:21.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e015f1d0 msgr2=0x7f46e00b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:21.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e015f1d0 0x7f46e00b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46e015b590 msgr2=0x7f46e01f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:21.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46e015b590 0x7f46e01f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 -- 172.21.15.158:0/2344390321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d0009550 con 0x7f46e00cbaf0 2023-10-16T09:43:21.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e5bc9700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e00cbaf0 0x7f46e01f6b30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009fd0 tx=0x7f46d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:21.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46d67fc700 1 -- 172.21.15.158:0/2344390321 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d000bc20 con 0x7f46e00cbaf0 2023-10-16T09:43:21.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46d67fc700 1 -- 172.21.15.158:0/2344390321 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d001f070 con 0x7f46e00cbaf0 2023-10-16T09:43:21.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46d67fc700 1 -- 172.21.15.158:0/2344390321 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00036a0 con 0x7f46e00cbaf0 2023-10-16T09:43:21.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e01f6610 con 0x7f46e00cbaf0 2023-10-16T09:43:21.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.065+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01f7070 con 0x7f46e00cbaf0 2023-10-16T09:43:21.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.066+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e015f630 con 0x7f46e00cbaf0 2023-10-16T09:43:21.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.067+0000 7f46d67fc700 1 -- 172.21.15.158:0/2344390321 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f46d0021030 con 0x7f46e00cbaf0 2023-10-16T09:43:21.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.067+0000 7f46d67fc700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f46c406af00 0x7f46c406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:21.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.067+0000 7f46d67fc700 1 -- 172.21.15.158:0/2344390321 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f46d008aa30 con 0x7f46e00cbaf0 2023-10-16T09:43:21.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.067+0000 7f46e53c8700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f46c406af00 0x7f46c406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:21.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.068+0000 7f46e53c8700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f46c406af00 0x7f46c406d3c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46e006b100 tx=0x7f46dc007ec0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:21.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.072+0000 7f46d67fc700 1 -- 172.21.15.158:0/2344390321 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46e015f630 con 0x7f46e00cbaf0 2023-10-16T09:43:21.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.307+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f46e01f6850 con 0x7f46e00cbaf0 2023-10-16T09:43:21.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.916+0000 7f46d67fc700 1 -- 172.21.15.158:0/2344390321 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f46e01f6850 con 0x7f46e00cbaf0 2023-10-16T09:43:21.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.921+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f46c406af00 msgr2=0x7f46c406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:21.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.921+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f46c406af00 0x7f46c406d3c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46e006b100 tx=0x7f46dc007ec0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.921+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e00cbaf0 msgr2=0x7f46e01f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:21.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.921+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e00cbaf0 0x7f46e01f6b30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009fd0 tx=0x7f46d0004750 comp rx=0 tx=0).stop 2023-10-16T09:43:21.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.923+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 shutdown_connections 2023-10-16T09:43:21.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.923+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f46c406af00 0x7f46c406d3c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.923+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46e015f1d0 0x7f46e00b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.923+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46e00cbaf0 0x7f46e01f6b30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.923+0000 7f46e7e2d700 1 --2- 172.21.15.158:0/2344390321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46e015b590 0x7f46e01f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:21.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.923+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 >> 172.21.15.158:0/2344390321 conn(0x7f46e006a8d0 msgr2=0x7f46e00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:21.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.924+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 shutdown_connections 2023-10-16T09:43:21.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:21.926+0000 7f46e7e2d700 1 -- 172.21.15.158:0/2344390321 wait complete. 2023-10-16T09:43:21.929 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 2 and data pool 3 2023-10-16T09:43:21.946 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-16T09:43:21.946 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-16T09:43:22.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.070+0000 7f7446616700 1 Processor -- start 2023-10-16T09:43:22.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.070+0000 7f7446616700 1 -- start start 2023-10-16T09:43:22.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7446616700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f744015b590 0x7f74400ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:22.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7446616700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f74400cbaf0 0x7f74400cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:22.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7446616700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74400cf4c0 con 0x7f744015f1d0 2023-10-16T09:43:22.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7446616700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400cc4c0 con 0x7f74400cbaf0 2023-10-16T09:43:22.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7446616700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400b49e0 con 0x7f744015b590 2023-10-16T09:43:22.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7444bb3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f74400cbaf0 0x7f74400cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:22.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f743f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f744015f1d0 0x7f74400b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46048/0 (socket says 172.21.15.158:46048) 2023-10-16T09:43:22.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f743f7fe700 1 -- 172.21.15.158:0/289051198 learned_addr learned my addr 172.21.15.158:0/289051198 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:22.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f743ffff700 1 --2- 172.21.15.158:0/289051198 >> v2:172.21.15.172:3300/0 conn(0x7f744015b590 0x7f74400ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:22.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7444bb3700 1 -- 172.21.15.158:0/289051198 >> v2:172.21.15.172:3300/0 conn(0x7f744015b590 msgr2=0x7f74400ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:22.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7444bb3700 1 --2- 172.21.15.158:0/289051198 >> v2:172.21.15.172:3300/0 conn(0x7f744015b590 0x7f74400ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7444bb3700 1 -- 172.21.15.158:0/289051198 >> v1:172.21.15.158:6789/0 conn(0x7f744015f1d0 legacy=0x7f74400b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:22.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7444bb3700 1 -- 172.21.15.158:0/289051198 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744015f970 con 0x7f74400cbaf0 2023-10-16T09:43:22.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f743effd700 1 -- 172.21.15.158:0/289051198 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450629601 0 0) 0x7f74400cf4c0 con 0x7f744015f1d0 2023-10-16T09:43:22.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f743ffff700 1 --2- 172.21.15.158:0/289051198 >> v2:172.21.15.172:3300/0 conn(0x7f744015b590 0x7f74400ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:22.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f7444bb3700 1 --2- 172.21.15.158:0/289051198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74400cbaf0 0x7f74400cbf80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74400b3520 tx=0x7f743800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84e415cb213f411 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:22.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.071+0000 7f743effd700 1 -- 172.21.15.158:0/289051198 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743800e070 con 0x7f74400cbaf0 2023-10-16T09:43:22.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f743effd700 1 -- 172.21.15.158:0/289051198 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743800bb90 con 0x7f74400cbaf0 2023-10-16T09:43:22.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f743effd700 1 -- 172.21.15.158:0/289051198 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438012430 con 0x7f74400cbaf0 2023-10-16T09:43:22.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 -- 172.21.15.158:0/289051198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74400cbaf0 msgr2=0x7f74400cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:22.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 --2- 172.21.15.158:0/289051198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74400cbaf0 0x7f74400cbf80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74400b3520 tx=0x7f743800b330 comp rx=0 tx=0).stop 2023-10-16T09:43:22.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 -- 172.21.15.158:0/289051198 shutdown_connections 2023-10-16T09:43:22.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 --2- 172.21.15.158:0/289051198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74400cbaf0 0x7f74400cbf80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 --2- 172.21.15.158:0/289051198 >> v2:172.21.15.172:3300/0 conn(0x7f744015b590 0x7f74400ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 -- 172.21.15.158:0/289051198 >> 172.21.15.158:0/289051198 conn(0x7f744006a8d0 msgr2=0x7f744006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:22.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 -- 172.21.15.158:0/289051198 shutdown_connections 2023-10-16T09:43:22.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.072+0000 7f7446616700 1 -- 172.21.15.158:0/289051198 wait complete. 2023-10-16T09:43:22.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.073+0000 7f7446616700 1 Processor -- start 2023-10-16T09:43:22.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.073+0000 7f7446616700 1 -- start start 2023-10-16T09:43:22.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f7446616700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74400cbaf0 0x7f744014f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:22.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f7446616700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f744015b590 0x7f7440152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:22.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f7446616700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f744015f1d0 0x7f74400cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:22.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f7446616700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74401522e0 con 0x7f744015b590 2023-10-16T09:43:22.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f7446616700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7440152770 con 0x7f744015f1d0 2023-10-16T09:43:22.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f7446616700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74401516d0 con 0x7f74400cbaf0 2023-10-16T09:43:22.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f743f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f744015b590 0x7f7440152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:22.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f743f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f744015b590 0x7f7440152b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45878/0 (socket says 172.21.15.158:45878) 2023-10-16T09:43:22.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f743f7fe700 1 -- 172.21.15.158:0/2328816531 learned_addr learned my addr 172.21.15.158:0/2328816531 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:22.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f743f7fe700 1 -- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74400cbaf0 msgr2=0x7f744014f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:22.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f7444bb3700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f744015f1d0 0x7f74400cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:22.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f743f7fe700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74400cbaf0 0x7f744014f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f743f7fe700 1 -- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f744015f1d0 msgr2=0x7f74400cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:22.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.074+0000 7f743f7fe700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f744015f1d0 0x7f74400cfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f743f7fe700 1 -- 172.21.15.158:0/2328816531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7438009a70 con 0x7f744015b590 2023-10-16T09:43:22.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f7444bb3700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f744015f1d0 0x7f74400cfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:22.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f743f7fe700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f744015b590 0x7f7440152b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f743000c550 tx=0x7f743000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:22.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f743cff9700 1 -- 172.21.15.158:0/2328816531 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430004c80 con 0x7f744015b590 2023-10-16T09:43:22.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f743cff9700 1 -- 172.21.15.158:0/2328816531 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7430004e00 con 0x7f744015b590 2023-10-16T09:43:22.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74401519b0 con 0x7f744015b590 2023-10-16T09:43:22.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f743cff9700 1 -- 172.21.15.158:0/2328816531 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430007310 con 0x7f744015b590 2023-10-16T09:43:22.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.075+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7440151f70 con 0x7f744015b590 2023-10-16T09:43:22.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.076+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74400681b0 con 0x7f744015b590 2023-10-16T09:43:22.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.076+0000 7f743cff9700 1 -- 172.21.15.158:0/2328816531 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7430007470 con 0x7f744015b590 2023-10-16T09:43:22.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.076+0000 7f743cff9700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f742c06af00 0x7f742c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:22.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.077+0000 7f743cff9700 1 -- 172.21.15.158:0/2328816531 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74300892b0 con 0x7f744015b590 2023-10-16T09:43:22.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.077+0000 7f743ffff700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f742c06af00 0x7f742c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:22.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.077+0000 7f743ffff700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f742c06af00 0x7f742c06d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7434007920 tx=0x7f7434008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:22.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.082+0000 7f743cff9700 1 -- 172.21.15.158:0/2328816531 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74300543f0 con 0x7f744015b590 2023-10-16T09:43:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.286+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f7440152420 con 0x7f744015b590 2023-10-16T09:43:22.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.920+0000 7f743cff9700 1 -- 172.21.15.158:0/2328816531 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f7430057a10 con 0x7f744015b590 2023-10-16T09:43:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f742c06af00 msgr2=0x7f742c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f742c06af00 0x7f742c06d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7434007920 tx=0x7f7434008040 comp rx=0 tx=0).stop 2023-10-16T09:43:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f744015b590 msgr2=0x7f7440152b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f744015b590 0x7f7440152b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f743000c550 tx=0x7f743000c910 comp rx=0 tx=0).stop 2023-10-16T09:43:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 shutdown_connections 2023-10-16T09:43:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f744015f1d0 0x7f74400cfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f742c06af00 0x7f742c06d3c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74400cbaf0 0x7f744014f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 --2- 172.21.15.158:0/2328816531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f744015b590 0x7f7440152b60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.923+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 >> 172.21.15.158:0/2328816531 conn(0x7f744006a8d0 msgr2=0x7f7440159a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.924+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 shutdown_connections 2023-10-16T09:43:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:22.924+0000 7f7446616700 1 -- 172.21.15.158:0/2328816531 wait complete. 2023-10-16T09:43:22.940 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T09:43:23.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.067+0000 7f34b95dc700 1 Processor -- start 2023-10-16T09:43:23.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.067+0000 7f34b95dc700 1 -- start start 2023-10-16T09:43:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.068+0000 7f34b95dc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b415a9c0 0x7f34b40ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.068+0000 7f34b95dc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b40ce730 con 0x7f34b40c3e40 2023-10-16T09:43:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.068+0000 7f34b95dc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b40cfc90 con 0x7f34b406fd50 2023-10-16T09:43:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.068+0000 7f34b95dc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40cbf80 con 0x7f34b415a9c0 2023-10-16T09:43:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.068+0000 7f34b37fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f34b40c3e40 0x7f34b406f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46062/0 (socket says 172.21.15.158:46062) 2023-10-16T09:43:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.068+0000 7f34b37fe700 1 -- 172.21.15.158:0/2147411741 learned_addr learned my addr 172.21.15.158:0/2147411741 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660781697 0 0) 0x7f34b40ce730 con 0x7f34b40c3e40 2023-10-16T09:43:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f349c0035b0 con 0x7f34b40c3e40 2023-10-16T09:43:23.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b415a9c0 0x7f34b40ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545795415 0 0) 0x7f349c0035b0 con 0x7f34b40c3e40 2023-10-16T09:43:23.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b415a9c0 msgr2=0x7f34b40ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:23.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b415a9c0 0x7f34b40ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:23.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 >> v1:172.21.15.172:6789/0 conn(0x7f34b406fd50 legacy=0x7f34b40cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:23.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40ce960 con 0x7f34b40c3e40 2023-10-16T09:43:23.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f34a4001670 con 0x7f34b40c3e40 2023-10-16T09:43:23.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 >> v1:172.21.15.158:6789/0 conn(0x7f34b40c3e40 legacy=0x7f34b406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:23.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f349c003a30 0x7f349c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c004850 0x7f349c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f349c005210 0x7f349c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40cbf80 con 0x7f349c003a30 2023-10-16T09:43:23.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c00bf50 con 0x7f349c004850 2023-10-16T09:43:23.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c00c090 con 0x7f349c005210 2023-10-16T09:43:23.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f349c003a30 0x7f349c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b37fe700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f349c005210 0x7f349c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b27fc700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c004850 0x7f349c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b2ffd700 1 -- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f349c005210 msgr2=0x7f349c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:23.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f349c005210 0x7f349c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:23.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b2ffd700 1 -- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c004850 msgr2=0x7f349c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:23.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c004850 0x7f349c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:23.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b2ffd700 1 -- 172.21.15.158:0/2147411741 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c0035b0 con 0x7f349c003a30 2023-10-16T09:43:23.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.069+0000 7f34b27fc700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c004850 0x7f349c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:23.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f349c003a30 0x7f349c003e50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0096f0 tx=0x7f349c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391edeb14ee1f1b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:23.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac00e660 con 0x7f349c003a30 2023-10-16T09:43:23.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac00e7c0 con 0x7f349c003a30 2023-10-16T09:43:23.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b1ffb700 1 -- 172.21.15.158:0/2147411741 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac00eb50 con 0x7f349c003a30 2023-10-16T09:43:23.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b95dc700 1 -- 172.21.15.158:0/2147411741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f349c003a30 msgr2=0x7f349c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:23.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b95dc700 1 --2- 172.21.15.158:0/2147411741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f349c003a30 0x7f349c003e50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0096f0 tx=0x7f349c005880 comp rx=0 tx=0).stop 2023-10-16T09:43:23.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b37fe700 1 -- 172.21.15.158:0/2147411741 reap_dead start 2023-10-16T09:43:23.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b95dc700 1 -- 172.21.15.158:0/2147411741 shutdown_connections 2023-10-16T09:43:23.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b95dc700 1 -- 172.21.15.158:0/2147411741 >> 172.21.15.158:0/2147411741 conn(0x7f34b40b1de0 msgr2=0x7f34b40b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:23.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b95dc700 1 -- 172.21.15.158:0/2147411741 shutdown_connections 2023-10-16T09:43:23.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.070+0000 7f34b95dc700 1 -- 172.21.15.158:0/2147411741 wait complete. 2023-10-16T09:43:23.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 Processor -- start 2023-10-16T09:43:23.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 -- start start 2023-10-16T09:43:23.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c003a30 0x7f34b4158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34b41f2e40 0x7f34b41f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b41f4d50 0x7f34b41f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b41f4300 con 0x7f34b41f2e40 2023-10-16T09:43:23.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b41f48c0 con 0x7f349c003a30 2023-10-16T09:43:23.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b95dc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b41f3b60 con 0x7f34b41f4d50 2023-10-16T09:43:23.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34b41f2e40 0x7f34b41f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34b41f2e40 0x7f34b41f32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45898/0 (socket says 172.21.15.158:45898) 2023-10-16T09:43:23.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.072+0000 7f34b27fc700 1 -- 172.21.15.158:0/2664436508 learned_addr learned my addr 172.21.15.158:0/2664436508 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:23.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c003a30 0x7f34b4158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b37fe700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b41f4d50 0x7f34b41f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b27fc700 1 -- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b41f4d50 msgr2=0x7f34b41f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:23.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b27fc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b41f4d50 0x7f34b41f51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:23.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b27fc700 1 -- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c003a30 msgr2=0x7f34b4158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:23.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b27fc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c003a30 0x7f34b4158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:23.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b27fc700 1 -- 172.21.15.158:0/2664436508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b41f3e40 con 0x7f34b41f2e40 2023-10-16T09:43:23.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c003a30 0x7f34b4158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:23.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b27fc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34b41f2e40 0x7f34b41f32c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34a8001fe0 tx=0x7f34a800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:23.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f349b7fe700 1 -- 172.21.15.158:0/2664436508 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8003e30 con 0x7f34b41f2e40 2023-10-16T09:43:23.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f349b7fe700 1 -- 172.21.15.158:0/2664436508 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a800b9c0 con 0x7f34b41f2e40 2023-10-16T09:43:23.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b41f4550 con 0x7f34b41f2e40 2023-10-16T09:43:23.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f349b7fe700 1 -- 172.21.15.158:0/2664436508 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a800bd90 con 0x7f34b41f2e40 2023-10-16T09:43:23.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.073+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b41586a0 con 0x7f34b41f2e40 2023-10-16T09:43:23.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.074+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b41f4a70 con 0x7f34b41f2e40 2023-10-16T09:43:23.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.074+0000 7f349b7fe700 1 -- 172.21.15.158:0/2664436508 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f34a8005070 con 0x7f34b41f2e40 2023-10-16T09:43:23.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.075+0000 7f349b7fe700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34a006af00 0x7f34a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:23.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.075+0000 7f349b7fe700 1 -- 172.21.15.158:0/2664436508 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f34a8089b80 con 0x7f34b41f2e40 2023-10-16T09:43:23.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.075+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34a006af00 0x7f34a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:23.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.075+0000 7f34b2ffd700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34a006af00 0x7f34a006d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f34ac01c040 tx=0x7f34ac015000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:23.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.079+0000 7f349b7fe700 1 -- 172.21.15.158:0/2664436508 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34b41f4a70 con 0x7f34b41f2e40 2023-10-16T09:43:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:23.272+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f34b4074a90 con 0x7f34b41f2e40 2023-10-16T09:43:24.043 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:43:24.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:43:24.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.935+0000 7f349b7fe700 1 -- 172.21.15.158:0/2664436508 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f34a8054be0 con 0x7f34b41f2e40 2023-10-16T09:43:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.940+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34a006af00 msgr2=0x7f34a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.940+0000 7f34b95dc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34a006af00 0x7f34a006d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f34ac01c040 tx=0x7f34ac015000 comp rx=0 tx=0).stop 2023-10-16T09:43:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.940+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34b41f2e40 msgr2=0x7f34b41f32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.940+0000 7f34b95dc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34b41f2e40 0x7f34b41f32c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34a8001fe0 tx=0x7f34a800b3b0 comp rx=0 tx=0).stop 2023-10-16T09:43:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.941+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 shutdown_connections 2023-10-16T09:43:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.941+0000 7f34b95dc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f34a006af00 0x7f34a006d3c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.941+0000 7f34b95dc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34b41f4d50 0x7f34b41f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.941+0000 7f34b95dc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f349c003a30 0x7f34b4158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.941+0000 7f34b95dc700 1 --2- 172.21.15.158:0/2664436508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34b41f2e40 0x7f34b41f32c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.941+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 >> 172.21.15.158:0/2664436508 conn(0x7f34b40b1de0 msgr2=0x7f34b4072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.943+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 shutdown_connections 2023-10-16T09:43:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:24.943+0000 7f34b95dc700 1 -- 172.21.15.158:0/2664436508 wait complete. 2023-10-16T09:43:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data_ec' created 2023-10-16T09:43:24.963 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-16T09:43:25.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.092+0000 7fbbfbeeb700 1 Processor -- start 2023-10-16T09:43:25.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.092+0000 7fbbfbeeb700 1 -- start start 2023-10-16T09:43:25.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfbeeb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbbf4159280 0x7fbbf406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:25.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfbeeb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf40cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:25.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfbeeb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf4159660 con 0x7fbbf40d0e10 2023-10-16T09:43:25.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfbeeb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf40cb710 con 0x7fbbf4159280 2023-10-16T09:43:25.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfbeeb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf406b120 con 0x7fbbf40cbef0 2023-10-16T09:43:25.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfa488700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf40cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:25.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbf9c87700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbbf4159280 0x7fbbf406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:25.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbf9c87700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbbf4159280 0x7fbbf406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47084/0 (socket says 172.21.15.158:47084) 2023-10-16T09:43:25.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfa488700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf40cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:56742/0 (socket says 172.21.15.158:56742) 2023-10-16T09:43:25.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbf9c87700 1 -- 172.21.15.158:0/2697860780 learned_addr learned my addr 172.21.15.158:0/2697860780 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:25.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfa488700 1 -- 172.21.15.158:0/2697860780 >> v2:172.21.15.172:3300/0 conn(0x7fbbf4159280 msgr2=0x7fbbf406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:25.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfa488700 1 --2- 172.21.15.158:0/2697860780 >> v2:172.21.15.172:3300/0 conn(0x7fbbf4159280 0x7fbbf406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfa488700 1 -- 172.21.15.158:0/2697860780 >> v1:172.21.15.172:6789/0 conn(0x7fbbf40d0e10 legacy=0x7fbbf415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:25.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbfa488700 1 -- 172.21.15.158:0/2697860780 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf406b400 con 0x7fbbf40cbef0 2023-10-16T09:43:25.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.093+0000 7fbbf9c87700 1 --2- 172.21.15.158:0/2697860780 >> v2:172.21.15.172:3300/0 conn(0x7fbbf4159280 0x7fbbf406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:25.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbfa488700 1 --2- 172.21.15.158:0/2697860780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf40cc380 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009d90 tx=0x7fbbec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce88ebe6444e46d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:25.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbf8c85700 1 -- 172.21.15.158:0/2697860780 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec00e070 con 0x7fbbf40cbef0 2023-10-16T09:43:25.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbf8c85700 1 -- 172.21.15.158:0/2697860780 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbec00bb20 con 0x7fbbf40cbef0 2023-10-16T09:43:25.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbf8c85700 1 -- 172.21.15.158:0/2697860780 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec012430 con 0x7fbbf40cbef0 2023-10-16T09:43:25.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/2697860780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 msgr2=0x7fbbf40cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:25.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/2697860780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf40cc380 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009d90 tx=0x7fbbec00b330 comp rx=0 tx=0).stop 2023-10-16T09:43:25.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/2697860780 shutdown_connections 2023-10-16T09:43:25.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/2697860780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf40cc380 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/2697860780 >> v2:172.21.15.172:3300/0 conn(0x7fbbf4159280 0x7fbbf406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.094+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/2697860780 >> 172.21.15.158:0/2697860780 conn(0x7fbbf4099a10 msgr2=0x7fbbf40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:25.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.095+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/2697860780 shutdown_connections 2023-10-16T09:43:25.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.095+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/2697860780 wait complete. 2023-10-16T09:43:25.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 Processor -- start 2023-10-16T09:43:25.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 -- start start 2023-10-16T09:43:25.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf415e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbf40d0e10 0x7fbbf415c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbf4159280 0x7fbbf40af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf41f5030 con 0x7fbbf40d0e10 2023-10-16T09:43:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf40aed40 con 0x7fbbf4159280 2023-10-16T09:43:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbfbeeb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf40ae170 con 0x7fbbf40cbef0 2023-10-16T09:43:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbf9486700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbf40d0e10 0x7fbbf415c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.096+0000 7fbbf9486700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbf40d0e10 0x7fbbf415c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45912/0 (socket says 172.21.15.158:45912) 2023-10-16T09:43:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9486700 1 -- 172.21.15.158:0/3478937265 learned_addr learned my addr 172.21.15.158:0/3478937265 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9c87700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf415e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:25.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbfa488700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbf4159280 0x7fbbf40af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:25.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9486700 1 -- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 msgr2=0x7fbbf415e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:25.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9486700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf415e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9486700 1 -- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbf4159280 msgr2=0x7fbbf40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:25.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9486700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbf4159280 0x7fbbf40af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9486700 1 -- 172.21.15.158:0/3478937265 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf0009550 con 0x7fbbf40d0e10 2023-10-16T09:43:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbf9486700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbf40d0e10 0x7fbbf415c450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c430 tx=0x7fbbe400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbea7fc700 1 -- 172.21.15.158:0/3478937265 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe40050d0 con 0x7fbbf40d0e10 2023-10-16T09:43:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbea7fc700 1 -- 172.21.15.158:0/3478937265 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe4005250 con 0x7fbbf40d0e10 2023-10-16T09:43:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec009a70 con 0x7fbbf40d0e10 2023-10-16T09:43:25.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbea7fc700 1 -- 172.21.15.158:0/3478937265 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4006860 con 0x7fbbf40d0e10 2023-10-16T09:43:25.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.097+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf40ae6f0 con 0x7fbbf40d0e10 2023-10-16T09:43:25.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.098+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf40aeef0 con 0x7fbbf40d0e10 2023-10-16T09:43:25.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.099+0000 7fbbea7fc700 1 -- 172.21.15.158:0/3478937265 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbbe4003680 con 0x7fbbf40d0e10 2023-10-16T09:43:25.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.099+0000 7fbbea7fc700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbbd806af00 0x7fbbd806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:25.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.099+0000 7fbbea7fc700 1 -- 172.21.15.158:0/3478937265 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6428+0+0 (secure 0 0 0) 0x7fbbe4088980 con 0x7fbbf40d0e10 2023-10-16T09:43:25.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.099+0000 7fbbf9c87700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbbd806af00 0x7fbbd806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:25.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.100+0000 7fbbf9c87700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbbd806af00 0x7fbbd806d3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0006d30 tx=0x7fbbf00092c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:25.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.104+0000 7fbbea7fc700 1 -- 172.21.15.158:0/3478937265 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbf40aeef0 con 0x7fbbf40d0e10 2023-10-16T09:43:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.296+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fbbf40aeef0 con 0x7fbbf40d0e10 2023-10-16T09:43:25.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.940+0000 7fbbea7fc700 1 -- 172.21.15.158:0/3478937265 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbbf40aeef0 con 0x7fbbf40d0e10 2023-10-16T09:43:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.944+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbbd806af00 msgr2=0x7fbbd806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.944+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbbd806af00 0x7fbbd806d3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0006d30 tx=0x7fbbf00092c0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.944+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbf40d0e10 msgr2=0x7fbbf415c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.944+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbf40d0e10 0x7fbbf415c450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c430 tx=0x7fbbe400c740 comp rx=0 tx=0).stop 2023-10-16T09:43:25.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.945+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 shutdown_connections 2023-10-16T09:43:25.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.945+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbf4159280 0x7fbbf40af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.945+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbbd806af00 0x7fbbd806d3c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.945+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbf40cbef0 0x7fbbf415e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.945+0000 7fbbfbeeb700 1 --2- 172.21.15.158:0/3478937265 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbf40d0e10 0x7fbbf415c450 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.945+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 >> 172.21.15.158:0/3478937265 conn(0x7fbbf4099a10 msgr2=0x7fbbf40cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.945+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 shutdown_connections 2023-10-16T09:43:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:25.949+0000 7fbbfbeeb700 1 -- 172.21.15.158:0/3478937265 wait complete. 2023-10-16T09:43:25.955 INFO:teuthology.orchestra.run.smithi158.stderr:set pool 4 allow_ec_overwrites to true 2023-10-16T09:43:25.979 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-16T09:43:26.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.105+0000 7f0d7cdcc700 1 Processor -- start 2023-10-16T09:43:26.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.105+0000 7f0d7cdcc700 1 -- start start 2023-10-16T09:43:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7cdcc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d7815da40 0x7f0d780caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7cdcc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0d780cc590 0x7f0d7806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7cdcc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d780cbd30 con 0x7f0d7806ae10 2023-10-16T09:43:26.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7cdcc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7806b280 con 0x7f0d780cc590 2023-10-16T09:43:26.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7cdcc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780cc1c0 con 0x7f0d7815da40 2023-10-16T09:43:26.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d75d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0d780cc590 0x7f0d7806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:26.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d76d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0d7806ae10 0x7f0d780ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46098/0 (socket says 172.21.15.158:46098) 2023-10-16T09:43:26.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d76d9d700 1 -- 172.21.15.158:0/3316134521 learned_addr learned my addr 172.21.15.158:0/3316134521 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:26.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7659c700 1 --2- 172.21.15.158:0/3316134521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d7815da40 0x7f0d780caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:26.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d75d9b700 1 -- 172.21.15.158:0/3316134521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d7815da40 msgr2=0x7f0d780caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:26.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d75d9b700 1 --2- 172.21.15.158:0/3316134521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d7815da40 0x7f0d780caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d75d9b700 1 -- 172.21.15.158:0/3316134521 >> v1:172.21.15.158:6789/0 conn(0x7f0d7806ae10 legacy=0x7f0d780ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:26.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d75d9b700 1 -- 172.21.15.158:0/3316134521 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7815de20 con 0x7f0d780cc590 2023-10-16T09:43:26.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7559a700 1 -- 172.21.15.158:0/3316134521 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740055716 0 0) 0x7f0d780cbd30 con 0x7f0d7806ae10 2023-10-16T09:43:26.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d75d9b700 1 --2- 172.21.15.158:0/3316134521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d780cc590 0x7f0d7806a8d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009840 tx=0x7f0d6c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fed032ea32571fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:26.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7559a700 1 -- 172.21.15.158:0/3316134521 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c011830 con 0x7f0d780cc590 2023-10-16T09:43:26.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7559a700 1 -- 172.21.15.158:0/3316134521 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c011990 con 0x7f0d780cc590 2023-10-16T09:43:26.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.106+0000 7f0d7559a700 1 -- 172.21.15.158:0/3316134521 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c011d20 con 0x7f0d780cc590 2023-10-16T09:43:26.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/3316134521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d780cc590 msgr2=0x7f0d7806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:26.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/3316134521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d780cc590 0x7f0d7806a8d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009840 tx=0x7f0d6c009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:26.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/3316134521 shutdown_connections 2023-10-16T09:43:26.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/3316134521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d7815da40 0x7f0d780caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/3316134521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d780cc590 0x7f0d7806a8d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/3316134521 >> 172.21.15.158:0/3316134521 conn(0x7f0d78099a10 msgr2=0x7f0d780d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:26.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/3316134521 shutdown_connections 2023-10-16T09:43:26.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.107+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/3316134521 wait complete. 2023-10-16T09:43:26.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.108+0000 7f0d7cdcc700 1 Processor -- start 2023-10-16T09:43:26.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.108+0000 7f0d7cdcc700 1 -- start start 2023-10-16T09:43:26.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d7cdcc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d7806ae10 0x7f0d780cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:26.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d7cdcc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d780cc590 0x7f0d781e74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:26.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d7cdcc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d7815da40 0x7f0d781e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:26.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d7cdcc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d781e4fd0 con 0x7f0d7815da40 2023-10-16T09:43:26.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d7cdcc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d781e7f00 con 0x7f0d7806ae10 2023-10-16T09:43:26.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d7cdcc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d781e5220 con 0x7f0d780cc590 2023-10-16T09:43:26.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d76d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d7815da40 0x7f0d781e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:26.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d76d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d7815da40 0x7f0d781e5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45928/0 (socket says 172.21.15.158:45928) 2023-10-16T09:43:26.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d76d9d700 1 -- 172.21.15.158:0/2394472866 learned_addr learned my addr 172.21.15.158:0/2394472866 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:26.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d75d9b700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d780cc590 0x7f0d781e74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:26.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.109+0000 7f0d7659c700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d7806ae10 0x7f0d780cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:26.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d76d9d700 1 -- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d780cc590 msgr2=0x7f0d781e74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:26.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d76d9d700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d780cc590 0x7f0d781e74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d76d9d700 1 -- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d7806ae10 msgr2=0x7f0d780cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:26.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d76d9d700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d7806ae10 0x7f0d780cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d76d9d700 1 -- 172.21.15.158:0/2394472866 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c0094a0 con 0x7f0d7815da40 2023-10-16T09:43:26.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d7659c700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d7806ae10 0x7f0d780cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:26.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d76d9d700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d7815da40 0x7f0d781e5d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d70002230 tx=0x7f0d7000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:26.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d66ffd700 1 -- 172.21.15.158:0/2394472866 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000ab70 con 0x7f0d7815da40 2023-10-16T09:43:26.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d66ffd700 1 -- 172.21.15.158:0/2394472866 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7000acf0 con 0x7f0d7815da40 2023-10-16T09:43:26.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d781e5500 con 0x7f0d7815da40 2023-10-16T09:43:26.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d66ffd700 1 -- 172.21.15.158:0/2394472866 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70007310 con 0x7f0d7815da40 2023-10-16T09:43:26.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.110+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d780c5dd0 con 0x7f0d7815da40 2023-10-16T09:43:26.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.111+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d781e5a20 con 0x7f0d7815da40 2023-10-16T09:43:26.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.111+0000 7f0d66ffd700 1 -- 172.21.15.158:0/2394472866 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0d70007470 con 0x7f0d7815da40 2023-10-16T09:43:26.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.112+0000 7f0d66ffd700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0d5806af00 0x7f0d5806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:26.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.112+0000 7f0d7659c700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0d5806af00 0x7f0d5806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:26.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.112+0000 7f0d66ffd700 1 -- 172.21.15.158:0/2394472866 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f0d70089750 con 0x7f0d7815da40 2023-10-16T09:43:26.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.112+0000 7f0d7659c700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0d5806af00 0x7f0d5806d3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d68007a50 tx=0x7f0d68008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:26.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.116+0000 7f0d66ffd700 1 -- 172.21.15.158:0/2394472866 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0d781e5a20 con 0x7f0d7815da40 2023-10-16T09:43:26.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.356+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f0d781e5a20 con 0x7f0d7815da40 2023-10-16T09:43:26.424 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T09:43:26.423+0000 7f0c79870700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T09:43:26.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.941+0000 7f0d66ffd700 1 -- 172.21.15.158:0/2394472866 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f0d781e5a20 con 0x7f0d7815da40 2023-10-16T09:43:26.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0d5806af00 msgr2=0x7f0d5806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0d5806af00 0x7f0d5806d3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d68007a50 tx=0x7f0d68008040 comp rx=0 tx=0).stop 2023-10-16T09:43:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d7815da40 msgr2=0x7f0d781e5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d7815da40 0x7f0d781e5d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d70002230 tx=0x7f0d7000cf50 comp rx=0 tx=0).stop 2023-10-16T09:43:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 shutdown_connections 2023-10-16T09:43:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d7815da40 0x7f0d781e5d00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d7806ae10 0x7f0d780cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0d5806af00 0x7f0d5806d3c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 --2- 172.21.15.158:0/2394472866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d780cc590 0x7f0d781e74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 >> 172.21.15.158:0/2394472866 conn(0x7f0d78099a10 msgr2=0x7f0d780b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:26.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 shutdown_connections 2023-10-16T09:43:26.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:26.945+0000 7f0d7cdcc700 1 -- 172.21.15.158:0/2394472866 wait complete. 2023-10-16T09:43:26.949 INFO:teuthology.orchestra.run.smithi158.stderr:added data pool 4 to fsmap 2023-10-16T09:43:26.962 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:27.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.086+0000 7f1ba5c8b700 1 Processor -- start 2023-10-16T09:43:27.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.086+0000 7f1ba5c8b700 1 -- start start 2023-10-16T09:43:27.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1ba5c8b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00d1210 0x7f1ba00ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1ba5c8b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ba00ce950 0x7f1ba00b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1ba5c8b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba00d3bd0 con 0x7f1ba015d7c0 2023-10-16T09:43:27.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1ba5c8b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba00d4060 con 0x7f1ba00ce950 2023-10-16T09:43:27.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1ba5c8b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00cede0 con 0x7f1ba00d1210 2023-10-16T09:43:27.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ba00ce950 0x7f1ba00b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1ba015d7c0 0x7f1ba00d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46108/0 (socket says 172.21.15.158:46108) 2023-10-16T09:43:27.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9f7fe700 1 -- 172.21.15.158:0/399796480 learned_addr learned my addr 172.21.15.158:0/399796480 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:27.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9effd700 1 --2- 172.21.15.158:0/399796480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00d1210 0x7f1ba00ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9ffff700 1 -- 172.21.15.158:0/399796480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00d1210 msgr2=0x7f1ba00ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:27.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9ffff700 1 --2- 172.21.15.158:0/399796480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00d1210 0x7f1ba00ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9ffff700 1 -- 172.21.15.158:0/399796480 >> v1:172.21.15.158:6789/0 conn(0x7f1ba015d7c0 legacy=0x7f1ba00d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9e7fc700 1 -- 172.21.15.158:0/399796480 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087836420 0 0) 0x7f1ba00d3bd0 con 0x7f1ba015d7c0 2023-10-16T09:43:27.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9ffff700 1 -- 172.21.15.158:0/399796480 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00d2fc0 con 0x7f1ba00ce950 2023-10-16T09:43:27.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.087+0000 7f1b9ffff700 1 --2- 172.21.15.158:0/399796480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba00ce950 0x7f1ba00b6800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1b98009d90 tx=0x7f1b9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0c50c054e442b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1b9e7fc700 1 -- 172.21.15.158:0/399796480 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9800e070 con 0x7f1ba00ce950 2023-10-16T09:43:27.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1b9e7fc700 1 -- 172.21.15.158:0/399796480 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9800bb20 con 0x7f1ba00ce950 2023-10-16T09:43:27.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1b9e7fc700 1 -- 172.21.15.158:0/399796480 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98012430 con 0x7f1ba00ce950 2023-10-16T09:43:27.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/399796480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba00ce950 msgr2=0x7f1ba00b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/399796480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba00ce950 0x7f1ba00b6800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1b98009d90 tx=0x7f1b9800b330 comp rx=0 tx=0).stop 2023-10-16T09:43:27.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/399796480 shutdown_connections 2023-10-16T09:43:27.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/399796480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba00ce950 0x7f1ba00b6800 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/399796480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00d1210 0x7f1ba00ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/399796480 >> 172.21.15.158:0/399796480 conn(0x7f1ba00669f0 msgr2=0x7f1ba0066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:27.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/399796480 shutdown_connections 2023-10-16T09:43:27.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.088+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/399796480 wait complete. 2023-10-16T09:43:27.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.089+0000 7f1ba5c8b700 1 Processor -- start 2023-10-16T09:43:27.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1ba5c8b700 1 -- start start 2023-10-16T09:43:27.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1ba5c8b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00ce950 0x7f1ba015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1ba5c8b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ba00d1210 0x7f1ba0153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1ba5c8b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba015d7c0 0x7f1ba0154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1ba5c8b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00c8b90 con 0x7f1ba015d7c0 2023-10-16T09:43:27.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1ba5c8b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba01604a0 con 0x7f1ba00d1210 2023-10-16T09:43:27.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1ba5c8b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00c8260 con 0x7f1ba00ce950 2023-10-16T09:43:27.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1b9ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba015d7c0 0x7f1ba0154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1b9ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba015d7c0 0x7f1ba0154140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45942/0 (socket says 172.21.15.158:45942) 2023-10-16T09:43:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.090+0000 7f1b9ffff700 1 -- 172.21.15.158:0/3278749987 learned_addr learned my addr 172.21.15.158:0/3278749987 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9f7fe700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00ce950 0x7f1ba015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9ffff700 1 -- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00ce950 msgr2=0x7f1ba015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9effd700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ba00d1210 0x7f1ba0153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9ffff700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00ce950 0x7f1ba015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9ffff700 1 -- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ba00d1210 msgr2=0x7f1ba0153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9ffff700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ba00d1210 0x7f1ba0153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9ffff700 1 -- 172.21.15.158:0/3278749987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98009a70 con 0x7f1ba015d7c0 2023-10-16T09:43:27.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9f7fe700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00ce950 0x7f1ba015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:27.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9effd700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ba00d1210 0x7f1ba0153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:27.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b9ffff700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba015d7c0 0x7f1ba0154140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b370 tx=0x7f1b98004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b87fff700 1 -- 172.21.15.158:0/3278749987 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98012430 con 0x7f1ba015d7c0 2023-10-16T09:43:27.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b87fff700 1 -- 172.21.15.158:0/3278749987 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98020070 con 0x7f1ba015d7c0 2023-10-16T09:43:27.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba00c8540 con 0x7f1ba015d7c0 2023-10-16T09:43:27.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1b87fff700 1 -- 172.21.15.158:0/3278749987 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98004280 con 0x7f1ba015d7c0 2023-10-16T09:43:27.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.091+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba00ccea0 con 0x7f1ba015d7c0 2023-10-16T09:43:27.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.092+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba00c8d40 con 0x7f1ba015d7c0 2023-10-16T09:43:27.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.092+0000 7f1b87fff700 1 -- 172.21.15.158:0/3278749987 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1b980043e0 con 0x7f1ba015d7c0 2023-10-16T09:43:27.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.093+0000 7f1b87fff700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b8c06af00 0x7f1b8c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.093+0000 7f1b87fff700 1 -- 172.21.15.158:0/3278749987 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1b98022070 con 0x7f1ba015d7c0 2023-10-16T09:43:27.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.093+0000 7f1b9f7fe700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b8c06af00 0x7f1b8c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.093+0000 7f1b9f7fe700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b8c06af00 0x7f1b8c06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1b94000f60 tx=0x7f1b94009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.097+0000 7f1b87fff700 1 -- 172.21.15.158:0/3278749987 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ba00c8d40 con 0x7f1ba015d7c0 2023-10-16T09:43:27.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.331+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba00681b0 con 0x7f1ba015d7c0 2023-10-16T09:43:27.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.332+0000 7f1b87fff700 1 -- 172.21.15.158:0/3278749987 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4921 (secure 0 0 0) 0x7f1b98056630 con 0x7f1ba015d7c0 2023-10-16T09:43:27.334 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:27.334 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:26.357666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-16T09:43:27.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b8c06af00 msgr2=0x7f1b8c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b8c06af00 0x7f1b8c06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1b94000f60 tx=0x7f1b94009040 comp rx=0 tx=0).stop 2023-10-16T09:43:27.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba015d7c0 msgr2=0x7f1ba0154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba015d7c0 0x7f1ba0154140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b370 tx=0x7f1b98004b20 comp rx=0 tx=0).stop 2023-10-16T09:43:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 shutdown_connections 2023-10-16T09:43:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ba015d7c0 0x7f1ba0154140 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b8c06af00 0x7f1b8c06d3c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ba00ce950 0x7f1ba015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 --2- 172.21.15.158:0/3278749987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ba00d1210 0x7f1ba0153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 >> 172.21.15.158:0/3278749987 conn(0x7f1ba00669f0 msgr2=0x7f1ba015b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 shutdown_connections 2023-10-16T09:43:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.334+0000 7f1ba5c8b700 1 -- 172.21.15.158:0/3278749987 wait complete. 2023-10-16T09:43:27.339 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 7 2023-10-16T09:43:27.348 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:27.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.467+0000 7f41a1870700 1 Processor -- start 2023-10-16T09:43:27.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.467+0000 7f41a1870700 1 -- start start 2023-10-16T09:43:27.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f41a1870700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f419c15d7c0 0x7f419c0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f41a1870700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f419c0d1210 0x7f419c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f41a1870700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c0d3bd0 con 0x7f419c0ce950 2023-10-16T09:43:27.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f41a1870700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c0d4060 con 0x7f419c0d1210 2023-10-16T09:43:27.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f41a1870700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c0d2cf0 con 0x7f419c15d7c0 2023-10-16T09:43:27.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f419c0d1210 0x7f419c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f419c0ce950 0x7f419c0b50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46124/0 (socket says 172.21.15.158:46124) 2023-10-16T09:43:27.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419b7fe700 1 -- 172.21.15.158:0/2045113420 learned_addr learned my addr 172.21.15.158:0/2045113420 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:27.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419affd700 1 --2- 172.21.15.158:0/2045113420 >> v2:172.21.15.172:3300/0 conn(0x7f419c15d7c0 0x7f419c0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419a7fc700 1 -- 172.21.15.158:0/2045113420 >> v2:172.21.15.172:3300/0 conn(0x7f419c15d7c0 msgr2=0x7f419c0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:27.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419a7fc700 1 --2- 172.21.15.158:0/2045113420 >> v2:172.21.15.172:3300/0 conn(0x7f419c15d7c0 0x7f419c0d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419a7fc700 1 -- 172.21.15.158:0/2045113420 >> v1:172.21.15.158:6789/0 conn(0x7f419c0ce950 legacy=0x7f419c0b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f419a7fc700 1 -- 172.21.15.158:0/2045113420 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0d2fc0 con 0x7f419c0d1210 2023-10-16T09:43:27.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.468+0000 7f4199ffb700 1 -- 172.21.15.158:0/2045113420 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384113091 0 0) 0x7f419c0d3bd0 con 0x7f419c0ce950 2023-10-16T09:43:27.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f419a7fc700 1 --2- 172.21.15.158:0/2045113420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0d1210 0x7f419c0ce410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f419c0d0bc0 tx=0x7f418c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc92df32ead60c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f4199ffb700 1 -- 172.21.15.158:0/2045113420 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c011830 con 0x7f419c0d1210 2023-10-16T09:43:27.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f4199ffb700 1 -- 172.21.15.158:0/2045113420 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c011990 con 0x7f419c0d1210 2023-10-16T09:43:27.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f4199ffb700 1 -- 172.21.15.158:0/2045113420 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c011d20 con 0x7f419c0d1210 2023-10-16T09:43:27.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 -- 172.21.15.158:0/2045113420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0d1210 msgr2=0x7f419c0ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 --2- 172.21.15.158:0/2045113420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0d1210 0x7f419c0ce410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f419c0d0bc0 tx=0x7f418c009b20 comp rx=0 tx=0).stop 2023-10-16T09:43:27.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 -- 172.21.15.158:0/2045113420 shutdown_connections 2023-10-16T09:43:27.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 --2- 172.21.15.158:0/2045113420 >> v2:172.21.15.172:3300/0 conn(0x7f419c15d7c0 0x7f419c0d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 --2- 172.21.15.158:0/2045113420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0d1210 0x7f419c0ce410 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 -- 172.21.15.158:0/2045113420 >> 172.21.15.158:0/2045113420 conn(0x7f419c0669f0 msgr2=0x7f419c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:27.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 -- 172.21.15.158:0/2045113420 shutdown_connections 2023-10-16T09:43:27.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.469+0000 7f41a1870700 1 -- 172.21.15.158:0/2045113420 wait complete. 2023-10-16T09:43:27.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.470+0000 7f41a1870700 1 Processor -- start 2023-10-16T09:43:27.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f41a1870700 1 -- start start 2023-10-16T09:43:27.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f41a1870700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0ce950 0x7f419c1f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f41a1870700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419c0d1210 0x7f419c1f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f41a1870700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419c15d7c0 0x7f419c0cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f41a1870700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c1602d0 con 0x7f419c0ce950 2023-10-16T09:43:27.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f41a1870700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c160b00 con 0x7f419c15d7c0 2023-10-16T09:43:27.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f41a1870700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c1f3a80 con 0x7f419c0d1210 2023-10-16T09:43:27.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f419affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0ce950 0x7f419c1f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f419affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0ce950 0x7f419c1f5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45952/0 (socket says 172.21.15.158:45952) 2023-10-16T09:43:27.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.471+0000 7f419affd700 1 -- 172.21.15.158:0/141759273 learned_addr learned my addr 172.21.15.158:0/141759273 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:27.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419b7fe700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419c15d7c0 0x7f419c0cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419a7fc700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419c0d1210 0x7f419c1f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419affd700 1 -- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419c0d1210 msgr2=0x7f419c1f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419affd700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419c0d1210 0x7f419c1f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419affd700 1 -- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419c15d7c0 msgr2=0x7f419c0cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419affd700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419c15d7c0 0x7f419c0cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419affd700 1 -- 172.21.15.158:0/141759273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0094a0 con 0x7f419c0ce950 2023-10-16T09:43:27.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419b7fe700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419c15d7c0 0x7f419c0cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:27.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f419affd700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0ce950 0x7f419c1f5aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f419000e810 tx=0x7f419000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f4183fff700 1 -- 172.21.15.158:0/141759273 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190013610 con 0x7f419c0ce950 2023-10-16T09:43:27.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f4183fff700 1 -- 172.21.15.158:0/141759273 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4190013770 con 0x7f419c0ce950 2023-10-16T09:43:27.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c1604b0 con 0x7f419c0ce950 2023-10-16T09:43:27.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f4183fff700 1 -- 172.21.15.158:0/141759273 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190013b20 con 0x7f419c0ce950 2023-10-16T09:43:27.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.472+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c151fa0 con 0x7f419c0ce950 2023-10-16T09:43:27.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.473+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c0681b0 con 0x7f419c0ce950 2023-10-16T09:43:27.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.473+0000 7f4183fff700 1 -- 172.21.15.158:0/141759273 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4190013c80 con 0x7f419c0ce950 2023-10-16T09:43:27.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.474+0000 7f4183fff700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f418806af00 0x7f418806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.474+0000 7f4183fff700 1 -- 172.21.15.158:0/141759273 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f41900895c0 con 0x7f419c0ce950 2023-10-16T09:43:27.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.474+0000 7f419a7fc700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f418806af00 0x7f418806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.474+0000 7f419a7fc700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f418806af00 0x7f418806d3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f419c0c6a90 tx=0x7f418c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.478+0000 7f4183fff700 1 -- 172.21.15.158:0/141759273 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4190054390 con 0x7f419c0ce950 2023-10-16T09:43:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.672+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f419c0d3550 con 0x7f419c0ce950 2023-10-16T09:43:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.673+0000 7f4183fff700 1 -- 172.21.15.158:0/141759273 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f41900579b0 con 0x7f419c0ce950 2023-10-16T09:43:27.675 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:27.675 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f418806af00 msgr2=0x7f418806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f418806af00 0x7f418806d3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f419c0c6a90 tx=0x7f418c008030 comp rx=0 tx=0).stop 2023-10-16T09:43:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0ce950 msgr2=0x7f419c1f5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0ce950 0x7f419c1f5aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f419000e810 tx=0x7f419000ebd0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 shutdown_connections 2023-10-16T09:43:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419c15d7c0 0x7f419c0cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f419c0ce950 0x7f419c1f5aa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f418806af00 0x7f418806d3c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 --2- 172.21.15.158:0/141759273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419c0d1210 0x7f419c1f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 >> 172.21.15.158:0/141759273 conn(0x7f419c0669f0 msgr2=0x7f419c15c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 shutdown_connections 2023-10-16T09:43:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.675+0000 7f41a1870700 1 -- 172.21.15.158:0/141759273 wait complete. 2023-10-16T09:43:27.689 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:27.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.808+0000 7fbda1c52700 1 Processor -- start 2023-10-16T09:43:27.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.808+0000 7fbda1c52700 1 -- start start 2023-10-16T09:43:27.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbda1c52700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbd9c0cc5d0 0x7fbd9c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbda1c52700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 0x7fbd9c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbda1c52700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c06e770 con 0x7fbd9c06f340 2023-10-16T09:43:27.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbda1c52700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c0cf270 con 0x7fbd9c0cc5d0 2023-10-16T09:43:27.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbda1c52700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c15e4e0 con 0x7fbd9c15dfb0 2023-10-16T09:43:27.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbd9c06f340 0x7fbd9c071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46140/0 (socket says 172.21.15.158:46140) 2023-10-16T09:43:27.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9bfff700 1 -- 172.21.15.158:0/3746957095 learned_addr learned my addr 172.21.15.158:0/3746957095 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:27.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/3746957095 >> v2:172.21.15.172:3300/0 conn(0x7fbd9c0cc5d0 0x7fbd9c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd92ffd700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 0x7fbd9c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386805803 0 0) 0x7fbd94001490 con 0x7fbd9c06f340 2023-10-16T09:43:27.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd880035b0 con 0x7fbd9c06f340 2023-10-16T09:43:27.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674320396 0 0) 0x7fbd880035b0 con 0x7fbd9c06f340 2023-10-16T09:43:27.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 msgr2=0x7fbd9c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:27.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 0x7fbd9c0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 >> v2:172.21.15.172:3300/0 conn(0x7fbd9c0cc5d0 msgr2=0x7fbd9c06e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:27.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 --2- 172.21.15.158:0/3746957095 >> v2:172.21.15.172:3300/0 conn(0x7fbd9c0cc5d0 0x7fbd9c06e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c15e7c0 con 0x7fbd9c06f340 2023-10-16T09:43:27.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fbd94001b80 con 0x7fbd9c06f340 2023-10-16T09:43:27.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 >> v1:172.21.15.158:6789/0 conn(0x7fbd9c06f340 legacy=0x7fbd9c071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd88003a00 0x7fbd88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 0x7fbd88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88004d90 0x7fbd88005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0cf270 con 0x7fbd88003a00 2023-10-16T09:43:27.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c15e4e0 con 0x7fbd88004d90 2023-10-16T09:43:27.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8800ba10 con 0x7fbd9c15dfb0 2023-10-16T09:43:27.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.809+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/3746957095 >> v2:172.21.15.172:3300/0 conn(0x7fbd9c0cc5d0 0x7fbd9c06e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:27.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd92ffd700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 0x7fbd88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd88003a00 0x7fbd88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9bfff700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88004d90 0x7fbd88005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9b7fe700 1 -- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 msgr2=0x7fbd88004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:27.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c15dfb0 0x7fbd88004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9b7fe700 1 -- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88004d90 msgr2=0x7fbd88005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:27.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88004d90 0x7fbd88005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9b7fe700 1 -- 172.21.15.158:0/3746957095 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd880035b0 con 0x7fbd88003a00 2023-10-16T09:43:27.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd88003a00 0x7fbd88003e20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009d10 tx=0x7fbd88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e91a2277c13c8fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c009e60 con 0x7fbd88003a00 2023-10-16T09:43:27.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c00e9a0 con 0x7fbd88003a00 2023-10-16T09:43:27.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.810+0000 7fbd9affd700 1 -- 172.21.15.158:0/3746957095 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c00ed10 con 0x7fbd88003a00 2023-10-16T09:43:27.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.811+0000 7fbda1c52700 1 -- 172.21.15.158:0/3746957095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd88003a00 msgr2=0x7fbd88003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.811+0000 7fbda1c52700 1 --2- 172.21.15.158:0/3746957095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd88003a00 0x7fbd88003e20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009d10 tx=0x7fbd88005880 comp rx=0 tx=0).stop 2023-10-16T09:43:27.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.811+0000 7fbd9bfff700 1 -- 172.21.15.158:0/3746957095 reap_dead start 2023-10-16T09:43:27.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.811+0000 7fbda1c52700 1 -- 172.21.15.158:0/3746957095 shutdown_connections 2023-10-16T09:43:27.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.811+0000 7fbda1c52700 1 -- 172.21.15.158:0/3746957095 >> 172.21.15.158:0/3746957095 conn(0x7fbd9c0d02b0 msgr2=0x7fbd9c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:27.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.811+0000 7fbda1c52700 1 -- 172.21.15.158:0/3746957095 shutdown_connections 2023-10-16T09:43:27.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.811+0000 7fbda1c52700 1 -- 172.21.15.158:0/3746957095 wait complete. 2023-10-16T09:43:27.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.812+0000 7fbda1c52700 1 Processor -- start 2023-10-16T09:43:27.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.812+0000 7fbda1c52700 1 -- start start 2023-10-16T09:43:27.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbda1c52700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88003a00 0x7fbd9c0c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbda1c52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd9c0c7380 0x7fbd9c14a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbda1c52700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c0c7800 0x7fbd9c14a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbda1c52700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c15ba90 con 0x7fbd9c0c7380 2023-10-16T09:43:27.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbda1c52700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c14abd0 con 0x7fbd88003a00 2023-10-16T09:43:27.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbda1c52700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0c3e90 con 0x7fbd9c0c7800 2023-10-16T09:43:27.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd9c0c7380 0x7fbd9c14a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd9c0c7380 0x7fbd9c14a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45974/0 (socket says 172.21.15.158:45974) 2023-10-16T09:43:27.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 -- 172.21.15.158:0/2042636750 learned_addr learned my addr 172.21.15.158:0/2042636750 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:27.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88003a00 0x7fbd9c0c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd9bfff700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c0c7800 0x7fbd9c14a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 -- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c0c7800 msgr2=0x7fbd9c14a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c0c7800 0x7fbd9c14a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 -- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88003a00 msgr2=0x7fbd9c0c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88003a00 0x7fbd9c0c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 -- 172.21.15.158:0/2042636750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c14ecb0 con 0x7fbd9c0c7380 2023-10-16T09:43:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88003a00 0x7fbd9c0c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd92ffd700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd9c0c7380 0x7fbd9c14a150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd840119c0 tx=0x7fbd84011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd98ff9700 1 -- 172.21.15.158:0/2042636750 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84004240 con 0x7fbd9c0c7380 2023-10-16T09:43:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd98ff9700 1 -- 172.21.15.158:0/2042636750 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840043a0 con 0x7fbd9c0c7380 2023-10-16T09:43:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.813+0000 7fbd98ff9700 1 -- 172.21.15.158:0/2042636750 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8400f730 con 0x7fbd9c0c7380 2023-10-16T09:43:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.814+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c14ef20 con 0x7fbd9c0c7380 2023-10-16T09:43:27.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.814+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c14f4e0 con 0x7fbd9c0c7380 2023-10-16T09:43:27.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.814+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c0afb00 con 0x7fbd9c0c7380 2023-10-16T09:43:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.815+0000 7fbd98ff9700 1 -- 172.21.15.158:0/2042636750 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbd8400f890 con 0x7fbd9c0c7380 2023-10-16T09:43:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.815+0000 7fbd98ff9700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbd7c06af00 0x7fbd7c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.815+0000 7fbd98ff9700 1 -- 172.21.15.158:0/2042636750 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbd84089e80 con 0x7fbd9c0c7380 2023-10-16T09:43:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.815+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbd7c06af00 0x7fbd7c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.816+0000 7fbd9b7fe700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbd7c06af00 0x7fbd7c06d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0090b0 tx=0x7fbd8c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:27.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:27.820+0000 7fbd98ff9700 1 -- 172.21.15.158:0/2042636750 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbd84054c40 con 0x7fbd9c0c7380 2023-10-16T09:43:28.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.019+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd9c0681b0 con 0x7fbd9c0c7380 2023-10-16T09:43:28.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.020+0000 7fbd98ff9700 1 -- 172.21.15.158:0/2042636750 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7fbd84023090 con 0x7fbd9c0c7380 2023-10-16T09:43:28.022 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:28.023 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:28.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbd7c06af00 msgr2=0x7fbd7c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbd7c06af00 0x7fbd7c06d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0090b0 tx=0x7fbd8c008030 comp rx=0 tx=0).stop 2023-10-16T09:43:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd9c0c7380 msgr2=0x7fbd9c14a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd9c0c7380 0x7fbd9c14a150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd840119c0 tx=0x7fbd84011d80 comp rx=0 tx=0).stop 2023-10-16T09:43:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 shutdown_connections 2023-10-16T09:43:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbd7c06af00 0x7fbd7c06d3c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd9c0c7800 0x7fbd9c14a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd88003a00 0x7fbd9c0c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 --2- 172.21.15.158:0/2042636750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd9c0c7380 0x7fbd9c14a150 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 >> 172.21.15.158:0/2042636750 conn(0x7fbd9c0d02b0 msgr2=0x7fbd9c150900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:28.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 shutdown_connections 2023-10-16T09:43:28.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.023+0000 7fbda1c52700 1 -- 172.21.15.158:0/2042636750 wait complete. 2023-10-16T09:43:28.051 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:28.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.170+0000 7f3c311a8700 1 Processor -- start 2023-10-16T09:43:28.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.170+0000 7f3c311a8700 1 -- start start 2023-10-16T09:43:28.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c311a8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c2c06ab40 0x7f3c2c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c311a8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3c2c0729b0 0x7f3c2c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c311a8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c071e50 con 0x7f3c2c06e5d0 2023-10-16T09:43:28.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c311a8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c161270 con 0x7f3c2c0729b0 2023-10-16T09:43:28.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c311a8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c161510 con 0x7f3c2c06ab40 2023-10-16T09:43:28.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3c2c06e5d0 0x7f3c2c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46156/0 (socket says 172.21.15.158:46156) 2023-10-16T09:43:28.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2a59c700 1 -- 172.21.15.158:0/2561286623 learned_addr learned my addr 172.21.15.158:0/2561286623 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:28.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2b59e700 1 --2- 172.21.15.158:0/2561286623 >> v2:172.21.15.158:3300/0 conn(0x7f3c2c0729b0 0x7f3c2c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c29d9b700 1 -- 172.21.15.158:0/2561286623 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630686970 0 0) 0x7f3c2c071e50 con 0x7f3c2c06e5d0 2023-10-16T09:43:28.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2ad9d700 1 --2- 172.21.15.158:0/2561286623 >> v2:172.21.15.172:3300/0 conn(0x7f3c2c06ab40 0x7f3c2c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c29d9b700 1 -- 172.21.15.158:0/2561286623 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c140035b0 con 0x7f3c2c06e5d0 2023-10-16T09:43:28.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2b59e700 1 -- 172.21.15.158:0/2561286623 >> v2:172.21.15.172:3300/0 conn(0x7f3c2c06ab40 msgr2=0x7f3c2c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:28.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2b59e700 1 --2- 172.21.15.158:0/2561286623 >> v2:172.21.15.172:3300/0 conn(0x7f3c2c06ab40 0x7f3c2c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2b59e700 1 -- 172.21.15.158:0/2561286623 >> v1:172.21.15.158:6789/0 conn(0x7f3c2c06e5d0 legacy=0x7f3c2c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2b59e700 1 -- 172.21.15.158:0/2561286623 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c1617f0 con 0x7f3c2c0729b0 2023-10-16T09:43:28.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.171+0000 7f3c2b59e700 1 --2- 172.21.15.158:0/2561286623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c0729b0 0x7f3c2c071910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c074d70 tx=0x7f3c2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d1e467cc8e499b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:28.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c29d9b700 1 -- 172.21.15.158:0/2561286623 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400e070 con 0x7f3c2c0729b0 2023-10-16T09:43:28.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c29d9b700 1 -- 172.21.15.158:0/2561286623 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2400bb20 con 0x7f3c2c0729b0 2023-10-16T09:43:28.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c29d9b700 1 -- 172.21.15.158:0/2561286623 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24012430 con 0x7f3c2c0729b0 2023-10-16T09:43:28.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 -- 172.21.15.158:0/2561286623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c0729b0 msgr2=0x7f3c2c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 --2- 172.21.15.158:0/2561286623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c0729b0 0x7f3c2c071910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c074d70 tx=0x7f3c2400b330 comp rx=0 tx=0).stop 2023-10-16T09:43:28.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 -- 172.21.15.158:0/2561286623 shutdown_connections 2023-10-16T09:43:28.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 --2- 172.21.15.158:0/2561286623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c0729b0 0x7f3c2c071910 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 --2- 172.21.15.158:0/2561286623 >> v2:172.21.15.172:3300/0 conn(0x7f3c2c06ab40 0x7f3c2c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 -- 172.21.15.158:0/2561286623 >> 172.21.15.158:0/2561286623 conn(0x7f3c2c0d02b0 msgr2=0x7f3c2c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:28.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 -- 172.21.15.158:0/2561286623 shutdown_connections 2023-10-16T09:43:28.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.172+0000 7f3c311a8700 1 -- 172.21.15.158:0/2561286623 wait complete. 2023-10-16T09:43:28.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 Processor -- start 2023-10-16T09:43:28.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 -- start start 2023-10-16T09:43:28.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c06ab40 0x7f3c2c1f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c2c06e5d0 0x7f3c2c0c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c2c0729b0 0x7f3c2c0c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0c3c40 con 0x7f3c2c06ab40 2023-10-16T09:43:28.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0c3e90 con 0x7f3c2c06e5d0 2023-10-16T09:43:28.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c311a8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0c4170 con 0x7f3c2c0729b0 2023-10-16T09:43:28.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.174+0000 7f3c2ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c06ab40 0x7f3c2c1f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c06ab40 0x7f3c2c1f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45986/0 (socket says 172.21.15.158:45986) 2023-10-16T09:43:28.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 -- 172.21.15.158:0/917553388 learned_addr learned my addr 172.21.15.158:0/917553388 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:28.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2a59c700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c2c06e5d0 0x7f3c2c0c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2b59e700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c2c0729b0 0x7f3c2c0c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 -- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c2c0729b0 msgr2=0x7f3c2c0c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c2c0729b0 0x7f3c2c0c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 -- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c2c06e5d0 msgr2=0x7f3c2c0c60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c2c06e5d0 0x7f3c2c0c60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 -- 172.21.15.158:0/917553388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c24009a70 con 0x7f3c2c06ab40 2023-10-16T09:43:28.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2a59c700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c2c06e5d0 0x7f3c2c0c60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:28.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c2ad9d700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c06ab40 0x7f3c2c1f2c00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000d800 tx=0x7f3c2000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:28.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c137fe700 1 -- 172.21.15.158:0/917553388 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c200125f0 con 0x7f3c2c06ab40 2023-10-16T09:43:28.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c137fe700 1 -- 172.21.15.158:0/917553388 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c20012750 con 0x7f3c2c06ab40 2023-10-16T09:43:28.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c0c4450 con 0x7f3c2c06ab40 2023-10-16T09:43:28.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c137fe700 1 -- 172.21.15.158:0/917553388 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20012b00 con 0x7f3c2c06ab40 2023-10-16T09:43:28.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.175+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c0c4a10 con 0x7f3c2c06ab40 2023-10-16T09:43:28.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.176+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c0c4bd0 con 0x7f3c2c06ab40 2023-10-16T09:43:28.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.177+0000 7f3c137fe700 1 -- 172.21.15.158:0/917553388 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c20012c60 con 0x7f3c2c06ab40 2023-10-16T09:43:28.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.177+0000 7f3c137fe700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c1806af00 0x7f3c1806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.177+0000 7f3c137fe700 1 -- 172.21.15.158:0/917553388 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3c20089c70 con 0x7f3c2c06ab40 2023-10-16T09:43:28.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.177+0000 7f3c2a59c700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c1806af00 0x7f3c1806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.177+0000 7f3c2a59c700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c1806af00 0x7f3c1806d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c073fa0 tx=0x7f3c1c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:28.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.182+0000 7f3c137fe700 1 -- 172.21.15.158:0/917553388 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c20054a30 con 0x7f3c2c06ab40 2023-10-16T09:43:28.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.417+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c2c0681b0 con 0x7f3c2c06ab40 2023-10-16T09:43:28.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.417+0000 7f3c137fe700 1 -- 172.21.15.158:0/917553388 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4921 (secure 0 0 0) 0x7f3c20058050 con 0x7f3c2c06ab40 2023-10-16T09:43:28.419 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:28.419 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:26.357666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-16T09:43:28.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c1806af00 msgr2=0x7f3c1806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c1806af00 0x7f3c1806d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c073fa0 tx=0x7f3c1c009040 comp rx=0 tx=0).stop 2023-10-16T09:43:28.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c06ab40 msgr2=0x7f3c2c1f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c06ab40 0x7f3c2c1f2c00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000d800 tx=0x7f3c2000dbc0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 shutdown_connections 2023-10-16T09:43:28.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c1806af00 0x7f3c1806d3c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c2c0729b0 0x7f3c2c0c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c2c06ab40 0x7f3c2c1f2c00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 --2- 172.21.15.158:0/917553388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c2c06e5d0 0x7f3c2c0c60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.419+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 >> 172.21.15.158:0/917553388 conn(0x7f3c2c0d02b0 msgr2=0x7f3c2c0aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:28.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.420+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 shutdown_connections 2023-10-16T09:43:28.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.420+0000 7f3c311a8700 1 -- 172.21.15.158:0/917553388 wait complete. 2023-10-16T09:43:28.424 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 7 2023-10-16T09:43:28.433 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:28.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.552+0000 7f875dcd9700 1 Processor -- start 2023-10-16T09:43:28.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.552+0000 7f875dcd9700 1 -- start start 2023-10-16T09:43:28.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f875dcd9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f875815b350 0x7f87580ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f875dcd9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f875815b730 0x7f87580cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f875dcd9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87580ca780 con 0x7f87580ce460 2023-10-16T09:43:28.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f875dcd9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f875806aa90 con 0x7f875815b350 2023-10-16T09:43:28.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f875dcd9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87580cf820 con 0x7f875815b730 2023-10-16T09:43:28.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87577fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f875815b350 0x7f87580ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f8757fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f87580ce460 0x7f875815f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46174/0 (socket says 172.21.15.158:46174) 2023-10-16T09:43:28.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f8757fff700 1 -- 172.21.15.158:0/1889833447 learned_addr learned my addr 172.21.15.158:0/1889833447 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:28.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f8756ffd700 1 --2- 172.21.15.158:0/1889833447 >> v2:172.21.15.172:3300/0 conn(0x7f875815b730 0x7f87580cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87577fe700 1 -- 172.21.15.158:0/1889833447 >> v2:172.21.15.172:3300/0 conn(0x7f875815b730 msgr2=0x7f87580cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:28.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87577fe700 1 --2- 172.21.15.158:0/1889833447 >> v2:172.21.15.172:3300/0 conn(0x7f875815b730 0x7f87580cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87577fe700 1 -- 172.21.15.158:0/1889833447 >> v1:172.21.15.158:6789/0 conn(0x7f87580ce460 legacy=0x7f875815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87577fe700 1 -- 172.21.15.158:0/1889833447 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87580cf260 con 0x7f875815b350 2023-10-16T09:43:28.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87567fc700 1 -- 172.21.15.158:0/1889833447 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177640855 0 0) 0x7f87580ca780 con 0x7f87580ce460 2023-10-16T09:43:28.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87577fe700 1 --2- 172.21.15.158:0/1889833447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580ca080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8758066ab0 tx=0x7f8748009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8351a5803d4ca93e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:28.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87567fc700 1 -- 172.21.15.158:0/1889833447 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748011830 con 0x7f875815b350 2023-10-16T09:43:28.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87567fc700 1 -- 172.21.15.158:0/1889833447 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8748011990 con 0x7f875815b350 2023-10-16T09:43:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.553+0000 7f87567fc700 1 -- 172.21.15.158:0/1889833447 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748011d20 con 0x7f875815b350 2023-10-16T09:43:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 -- 172.21.15.158:0/1889833447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 msgr2=0x7f87580ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 --2- 172.21.15.158:0/1889833447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580ca080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8758066ab0 tx=0x7f8748009b20 comp rx=0 tx=0).stop 2023-10-16T09:43:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 -- 172.21.15.158:0/1889833447 shutdown_connections 2023-10-16T09:43:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 --2- 172.21.15.158:0/1889833447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580ca080 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 --2- 172.21.15.158:0/1889833447 >> v2:172.21.15.172:3300/0 conn(0x7f875815b730 0x7f87580cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 -- 172.21.15.158:0/1889833447 >> 172.21.15.158:0/1889833447 conn(0x7f87580cc4b0 msgr2=0x7f87580cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 -- 172.21.15.158:0/1889833447 shutdown_connections 2023-10-16T09:43:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.554+0000 7f875dcd9700 1 -- 172.21.15.158:0/1889833447 wait complete. 2023-10-16T09:43:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.555+0000 7f875dcd9700 1 Processor -- start 2023-10-16T09:43:28.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.555+0000 7f875dcd9700 1 -- start start 2023-10-16T09:43:28.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f875dcd9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87580ce460 0x7f875813ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f875dcd9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f875dcd9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f875815b730 0x7f8758070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f875dcd9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87581598c0 con 0x7f875815b350 2023-10-16T09:43:28.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f875dcd9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87580b3470 con 0x7f87580ce460 2023-10-16T09:43:28.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f875dcd9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87581f0a90 con 0x7f875815b730 2023-10-16T09:43:28.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46000/0 (socket says 172.21.15.158:46000) 2023-10-16T09:43:28.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 -- 172.21.15.158:0/3287161157 learned_addr learned my addr 172.21.15.158:0/3287161157 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:28.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f87577fe700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87580ce460 0x7f875813ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8757fff700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f875815b730 0x7f8758070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 -- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f875815b730 msgr2=0x7f8758070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f875815b730 0x7f8758070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 -- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87580ce460 msgr2=0x7f875813ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87580ce460 0x7f875813ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8756ffd700 1 -- 172.21.15.158:0/3287161157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87480094a0 con 0x7f875815b350 2023-10-16T09:43:28.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.556+0000 7f8757fff700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f875815b730 0x7f8758070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:28.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.557+0000 7f8756ffd700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580c8560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f874c00e810 tx=0x7f874c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:28.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.557+0000 7f873ffff700 1 -- 172.21.15.158:0/3287161157 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c013610 con 0x7f875815b350 2023-10-16T09:43:28.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.557+0000 7f873ffff700 1 -- 172.21.15.158:0/3287161157 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874c013770 con 0x7f875815b350 2023-10-16T09:43:28.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.557+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875813b060 con 0x7f875815b350 2023-10-16T09:43:28.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.557+0000 7f873ffff700 1 -- 172.21.15.158:0/3287161157 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c013ae0 con 0x7f875815b350 2023-10-16T09:43:28.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.557+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8758158f20 con 0x7f875815b350 2023-10-16T09:43:28.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.558+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87580cf520 con 0x7f875815b350 2023-10-16T09:43:28.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.558+0000 7f873ffff700 1 -- 172.21.15.158:0/3287161157 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f874c013c40 con 0x7f875815b350 2023-10-16T09:43:28.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.559+0000 7f873ffff700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f874406af00 0x7f874406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:28.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.559+0000 7f87577fe700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f874406af00 0x7f874406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:28.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.559+0000 7f873ffff700 1 -- 172.21.15.158:0/3287161157 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f874c089440 con 0x7f875815b350 2023-10-16T09:43:28.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.559+0000 7f87577fe700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f874406af00 0x7f874406d3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87480090b0 tx=0x7f8748008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.563+0000 7f873ffff700 1 -- 172.21.15.158:0/3287161157 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87580cf520 con 0x7f875815b350 2023-10-16T09:43:28.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.755+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8758159a00 con 0x7f875815b350 2023-10-16T09:43:28.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.756+0000 7f873ffff700 1 -- 172.21.15.158:0/3287161157 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f8758159a00 con 0x7f875815b350 2023-10-16T09:43:28.759 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:28.759 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:28.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f874406af00 msgr2=0x7f874406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f874406af00 0x7f874406d3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87480090b0 tx=0x7f8748008030 comp rx=0 tx=0).stop 2023-10-16T09:43:28.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 msgr2=0x7f87580c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:28.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580c8560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f874c00e810 tx=0x7f874c00eb20 comp rx=0 tx=0).stop 2023-10-16T09:43:28.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 shutdown_connections 2023-10-16T09:43:28.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f874406af00 0x7f874406d3c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f875815b730 0x7f8758070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87580ce460 0x7f875813ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 --2- 172.21.15.158:0/3287161157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f875815b350 0x7f87580c8560 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:28.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.758+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 >> 172.21.15.158:0/3287161157 conn(0x7f87580cc4b0 msgr2=0x7f8758156130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:28.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.759+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 shutdown_connections 2023-10-16T09:43:28.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:28.759+0000 7f875dcd9700 1 -- 172.21.15.158:0/3287161157 wait complete. 2023-10-16T09:43:29.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T09:43:21.310458+0000', 'modified': '2023-10-16T09:43:26.357666+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4267}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4267': {'gid': 4267, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/3958845884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 3958845884}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 3958845884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-10-16T09:43:29.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T09:43:29.175 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T09:43:29.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.298+0000 7facea129700 1 Processor -- start 2023-10-16T09:43:29.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.298+0000 7facea129700 1 -- start start 2023-10-16T09:43:29.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7facea129700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7facea129700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7face415f4d0 0x7face406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7facea129700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face4071af0 con 0x7face415bac0 2023-10-16T09:43:29.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7facea129700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face40afb60 con 0x7face415f4d0 2023-10-16T09:43:29.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7facea129700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face415fa40 con 0x7face415b6e0 2023-10-16T09:43:29.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7face2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7face415f4d0 0x7face406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7face3fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7face415bac0 0x7face4071f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46188/0 (socket says 172.21.15.158:46188) 2023-10-16T09:43:29.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.299+0000 7face3fff700 1 -- 172.21.15.158:0/3995797763 learned_addr learned my addr 172.21.15.158:0/3995797763 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:29.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face37fe700 1 --2- 172.21.15.158:0/3995797763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face2ffd700 1 -- 172.21.15.158:0/3995797763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 msgr2=0x7face406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face2ffd700 1 --2- 172.21.15.158:0/3995797763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face406dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face2ffd700 1 -- 172.21.15.158:0/3995797763 >> v1:172.21.15.158:6789/0 conn(0x7face415bac0 legacy=0x7face4071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face27fc700 1 -- 172.21.15.158:0/3995797763 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974439809 0 0) 0x7face4071af0 con 0x7face415bac0 2023-10-16T09:43:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face2ffd700 1 -- 172.21.15.158:0/3995797763 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face415fd20 con 0x7face415f4d0 2023-10-16T09:43:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face37fe700 1 --2- 172.21.15.158:0/3995797763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face406dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face2ffd700 1 --2- 172.21.15.158:0/3995797763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415f4d0 0x7face406a8d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7facd8009840 tx=0x7facd8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da2ce6fa0b4c57e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face27fc700 1 -- 172.21.15.158:0/3995797763 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8011830 con 0x7face415f4d0 2023-10-16T09:43:29.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face27fc700 1 -- 172.21.15.158:0/3995797763 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd8011990 con 0x7face415f4d0 2023-10-16T09:43:29.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7face27fc700 1 -- 172.21.15.158:0/3995797763 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8011d20 con 0x7face415f4d0 2023-10-16T09:43:29.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7facea129700 1 -- 172.21.15.158:0/3995797763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415f4d0 msgr2=0x7face406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.300+0000 7facea129700 1 --2- 172.21.15.158:0/3995797763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415f4d0 0x7face406a8d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7facd8009840 tx=0x7facd8009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:29.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.301+0000 7facea129700 1 -- 172.21.15.158:0/3995797763 shutdown_connections 2023-10-16T09:43:29.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.301+0000 7facea129700 1 --2- 172.21.15.158:0/3995797763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face406dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.301+0000 7facea129700 1 --2- 172.21.15.158:0/3995797763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415f4d0 0x7face406a8d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.301+0000 7facea129700 1 -- 172.21.15.158:0/3995797763 >> 172.21.15.158:0/3995797763 conn(0x7face406b9e0 msgr2=0x7face406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.301+0000 7facea129700 1 -- 172.21.15.158:0/3995797763 shutdown_connections 2023-10-16T09:43:29.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.301+0000 7facea129700 1 -- 172.21.15.158:0/3995797763 wait complete. 2023-10-16T09:43:29.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.302+0000 7facea129700 1 Processor -- start 2023-10-16T09:43:29.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.302+0000 7facea129700 1 -- start start 2023-10-16T09:43:29.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face4148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415bac0 0x7face40c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7face415f4d0 0x7face40c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face414b730 con 0x7face415bac0 2023-10-16T09:43:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face414bf80 con 0x7face415f4d0 2023-10-16T09:43:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face40ca090 con 0x7face415b6e0 2023-10-16T09:43:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415bac0 0x7face40c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415bac0 0x7face40c9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46016/0 (socket says 172.21.15.158:46016) 2023-10-16T09:43:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 -- 172.21.15.158:0/2298976075 learned_addr learned my addr 172.21.15.158:0/2298976075 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face37fe700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face4148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face3fff700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7face415f4d0 0x7face40c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 -- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 msgr2=0x7face4148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face4148e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 -- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7face415f4d0 msgr2=0x7face40c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7face415f4d0 0x7face40c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 -- 172.21.15.158:0/2298976075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd80094a0 con 0x7face415bac0 2023-10-16T09:43:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face3fff700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7face415f4d0 0x7face40c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7face2ffd700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415bac0 0x7face40c9540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7facd8014040 tx=0x7facd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7faccbfff700 1 -- 172.21.15.158:0/2298976075 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80045f0 con 0x7face415bac0 2023-10-16T09:43:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7faccbfff700 1 -- 172.21.15.158:0/2298976075 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd801e070 con 0x7face415bac0 2023-10-16T09:43:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face414b930 con 0x7face415bac0 2023-10-16T09:43:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7faccbfff700 1 -- 172.21.15.158:0/2298976075 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8003ae0 con 0x7face415bac0 2023-10-16T09:43:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.303+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face4149390 con 0x7face415bac0 2023-10-16T09:43:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.305+0000 7faccbfff700 1 -- 172.21.15.158:0/2298976075 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7facd8003c40 con 0x7face415bac0 2023-10-16T09:43:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.305+0000 7faccbfff700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7facd006af30 0x7facd006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.305+0000 7faccbfff700 1 -- 172.21.15.158:0/2298976075 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7facd808a9c0 con 0x7face415bac0 2023-10-16T09:43:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.305+0000 7face37fe700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7facd006af30 0x7facd006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.306+0000 7face37fe700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7facd006af30 0x7facd006d3f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7facdc0075b0 tx=0x7facdc007540 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.307+0000 7facf6313b80 1 Processor -- start 2023-10-16T09:43:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.307+0000 7facf6313b80 1 -- start start 2023-10-16T09:43:29.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb906220 0x5629fb90da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb90b630 0x5629fb90dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7facf6313b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb8e16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7facf6313b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb857c20 con 0x5629fb8e1010 2023-10-16T09:43:29.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb83cc10 con 0x5629fb906220 2023-10-16T09:43:29.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5629fb83a530 con 0x5629fb90b630 2023-10-16T09:43:29.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7face3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb8e16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7face3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb8e16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46026/0 (socket says 172.21.15.158:46026) 2023-10-16T09:43:29.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7face3fff700 1 -- 172.21.15.158:0/708658036 learned_addr learned my addr 172.21.15.158:0/708658036 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:29.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7face37fe700 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb906220 0x5629fb90da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.308+0000 7face2ffd700 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb90b630 0x5629fb90dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7face3fff700 1 -- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb90b630 msgr2=0x5629fb90dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7face3fff700 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb90b630 0x5629fb90dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7face3fff700 1 -- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb906220 msgr2=0x5629fb90da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7face3fff700 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb906220 0x5629fb90da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7face3fff700 1 -- 172.21.15.158:0/708658036 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629fb7f33a0 con 0x5629fb8e1010 2023-10-16T09:43:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7face3fff700 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb8e16c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7facd4001870 tx=0x7facd4001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9d1d41b5660ac4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7facc97fa700 1 -- 172.21.15.158:0/708658036 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4004c80 con 0x5629fb8e1010 2023-10-16T09:43:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7facc97fa700 1 -- 172.21.15.158:0/708658036 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4004de0 con 0x5629fb8e1010 2023-10-16T09:43:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7facc97fa700 1 -- 172.21.15.158:0/708658036 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd40072d0 con 0x5629fb8e1010 2023-10-16T09:43:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7facf6313b80 1 -- 172.21.15.158:0/708658036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 msgr2=0x5629fb8e16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.309+0000 7facf6313b80 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb8e16c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7facd4001870 tx=0x7facd4001b80 comp rx=0 tx=0).stop 2023-10-16T09:43:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 -- 172.21.15.158:0/708658036 shutdown_connections 2023-10-16T09:43:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb8e16c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb906220 0x5629fb90da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 --2- 172.21.15.158:0/708658036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb90b630 0x5629fb90dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 -- 172.21.15.158:0/708658036 >> 172.21.15.158:0/708658036 conn(0x5629fb810d80 msgr2=0x5629fb92bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 -- 172.21.15.158:0/708658036 shutdown_connections 2023-10-16T09:43:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 -- 172.21.15.158:0/708658036 wait complete. 2023-10-16T09:43:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 -- start start 2023-10-16T09:43:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.310+0000 7facf6313b80 1 Processor -- start 2023-10-16T09:43:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7facf6313b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb90f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb906220 0x5629fb909c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7facf6313b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb835e20 con 0x5629fb8e1010 2023-10-16T09:43:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb7d26c0 con 0x5629fb90b630 2023-10-16T09:43:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5629fb7d2360 con 0x5629fb906220 2023-10-16T09:43:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7facf6313b80 10 client.0 _ll_register_callbacks cb 0x5629fb62f7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:43:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb90f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb90f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46028/0 (socket says 172.21.15.158:46028) 2023-10-16T09:43:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 -- 172.21.15.158:0/3999203254 learned_addr learned my addr 172.21.15.158:0/3999203254 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face2ffd700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb906220 0x5629fb909c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face3fff700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 -- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb906220 msgr2=0x5629fb909c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb906220 0x5629fb909c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 -- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 msgr2=0x5629fb9052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face37fe700 1 -- 172.21.15.158:0/3999203254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd40014a0 con 0x5629fb8e1010 2023-10-16T09:43:29.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.311+0000 7face2ffd700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb906220 0x5629fb909c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:29.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7face37fe700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb90f470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7facdc00cb40 tx=0x7facdc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d4dffbd2270cf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facc27fc700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:43:29.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facc27fc700 1 -- 172.21.15.158:0/3999203254 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc01e070 con 0x5629fb8e1010 2023-10-16T09:43:29.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facc27fc700 1 -- 172.21.15.158:0/3999203254 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7facdc019a20 con 0x5629fb8e1010 2023-10-16T09:43:29.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5629fb8d65b0 con 0x5629fb8e1010 2023-10-16T09:43:29.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facc27fc700 1 -- 172.21.15.158:0/3999203254 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7facdc019be0 con 0x5629fb8e1010 2023-10-16T09:43:29.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 10 client.4425 fetch_fsmap learned FSMap version 7 2023-10-16T09:43:29.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5629fb660d90 con 0x5629fb8e1010 2023-10-16T09:43:29.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facc27fc700 1 -- 172.21.15.158:0/3999203254 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5629fb8d65b0 con 0x5629fb8e1010 2023-10-16T09:43:29.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facc27fc700 1 -- 172.21.15.158:0/3999203254 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7facdc017450 con 0x5629fb8e1010 2023-10-16T09:43:29.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 10 client.4425 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T09:43:29.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 10 client.4425 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:43:29.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 20 client.4425 populate_metadata read hostname 'smithi158' 2023-10-16T09:43:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb910230 0x5629fb910610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 4 client.4425 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T09:43:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 --> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629fb82ee00 con 0x5629fb910230 2023-10-16T09:43:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.312+0000 7face2ffd700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb910230 0x5629fb910610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.313+0000 7facc27fc700 10 client.4425 ms_handle_connect on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.313+0000 7face2ffd700 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb910230 0x5629fb910610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59818de1ab0f0bbd server_cookie=a8803e8655135ec3 in_seq=0 out_seq=0 2023-10-16T09:43:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.313+0000 7facc27fc700 1 -- 172.21.15.158:0/3999203254 <== mds.0 v2:172.21.15.172:6833/3958845884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5629fb82ee00 con 0x5629fb910230 2023-10-16T09:43:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.313+0000 7facc27fc700 10 client.4425 handle_command_reply: tid=0 2023-10-16T09:43:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.313+0000 7facf6313b80 1 client.4425 shutdown 2023-10-16T09:43:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 msgr2=0x5629fb90f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb90f470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7facdc00cb40 tx=0x7facdc004b40 comp rx=0 tx=0).stop 2023-10-16T09:43:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 shutdown_connections 2023-10-16T09:43:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb910230 0x5629fb910610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facc27fc700 0 client.4425 ms_handle_reset on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb8e1010 0x5629fb90f470 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 --2- 172.21.15.158:0/3999203254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb906220 0x5629fb909c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 >> 172.21.15.158:0/3999203254 conn(0x5629fb810d80 msgr2=0x5629fb9294e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 shutdown_connections 2023-10-16T09:43:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 1 -- 172.21.15.158:0/3999203254 wait complete. 2023-10-16T09:43:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.314+0000 7facf6313b80 20 client.4425 trim_cache size 0 max 16384 2023-10-16T09:43:29.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.331+0000 7facf6313b80 1 Processor -- start 2023-10-16T09:43:29.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.331+0000 7facf6313b80 1 -- start start 2023-10-16T09:43:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb92c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7facf6313b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb8d8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9097e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7facf6313b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb6b68d0 con 0x5629fb906220 2023-10-16T09:43:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb904d50 con 0x5629fb90b630 2023-10-16T09:43:29.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5629fb9294e0 con 0x5629fb8e1010 2023-10-16T09:43:29.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb8d8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb8d8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46038/0 (socket says 172.21.15.158:46038) 2023-10-16T09:43:29.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 -- 172.21.15.158:0/1564363181 learned_addr learned my addr 172.21.15.158:0/1564363181 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:29.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face3fff700 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face37fe700 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb92c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 -- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 msgr2=0x5629fb92c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb92c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 -- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 msgr2=0x5629fb9097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9097e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.332+0000 7face2ffd700 1 -- 172.21.15.158:0/1564363181 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629fb649c10 con 0x5629fb906220 2023-10-16T09:43:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7face2ffd700 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb8d8130 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7facd80558f0 tx=0x7facd8026040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3f00517b602cbc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facc9ffb700 1 -- 172.21.15.158:0/1564363181 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8061740 con 0x5629fb906220 2023-10-16T09:43:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facc9ffb700 1 -- 172.21.15.158:0/1564363181 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd8057580 con 0x5629fb906220 2023-10-16T09:43:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facc9ffb700 1 -- 172.21.15.158:0/1564363181 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80577e0 con 0x5629fb906220 2023-10-16T09:43:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 -- 172.21.15.158:0/1564363181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 msgr2=0x5629fb8d8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb8d8130 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7facd80558f0 tx=0x7facd8026040 comp rx=0 tx=0).stop 2023-10-16T09:43:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 -- 172.21.15.158:0/1564363181 shutdown_connections 2023-10-16T09:43:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb9097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb92c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 --2- 172.21.15.158:0/1564363181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb8d8130 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 -- 172.21.15.158:0/1564363181 >> 172.21.15.158:0/1564363181 conn(0x5629fb810d80 msgr2=0x5629fb8df850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 -- 172.21.15.158:0/1564363181 shutdown_connections 2023-10-16T09:43:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.333+0000 7facf6313b80 1 -- 172.21.15.158:0/1564363181 wait complete. 2023-10-16T09:43:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.334+0000 7facf6313b80 1 -- start start 2023-10-16T09:43:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.334+0000 7facf6313b80 1 Processor -- start 2023-10-16T09:43:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb90e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb90eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb90f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb942510 con 0x5629fb906220 2023-10-16T09:43:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fb984670 con 0x5629fb90b630 2023-10-16T09:43:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5629fb986e20 con 0x5629fb8e1010 2023-10-16T09:43:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 10 client.0 _ll_register_callbacks cb 0x7facdc00c450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:43:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb90eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb90eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46044/0 (socket says 172.21.15.158:46044) 2023-10-16T09:43:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 -- 172.21.15.158:0/306523482 learned_addr learned my addr 172.21.15.158:0/306523482 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face37fe700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb90e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face3fff700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb90f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 -- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 msgr2=0x5629fb90e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb90e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 -- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 msgr2=0x5629fb90f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb90f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 -- 172.21.15.158:0/306523482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face4149390 con 0x5629fb906220 2023-10-16T09:43:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face3fff700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb90f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7face2ffd700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb90eac0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7facd8055940 tx=0x7facdc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48097ac62c921fb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facc27fc700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:43:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facc27fc700 1 -- 172.21.15.158:0/306523482 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd805d120 con 0x5629fb906220 2023-10-16T09:43:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facc27fc700 1 -- 172.21.15.158:0/306523482 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7facd807b060 con 0x5629fb906220 2023-10-16T09:43:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facd4004de0 con 0x5629fb906220 2023-10-16T09:43:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facc27fc700 1 -- 172.21.15.158:0/306523482 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7facd805d280 con 0x5629fb906220 2023-10-16T09:43:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 10 client.4431 fetch_fsmap learned FSMap version 7 2023-10-16T09:43:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.335+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5629fb687200 con 0x5629fb906220 2023-10-16T09:43:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facc27fc700 1 -- 172.21.15.158:0/306523482 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4004de0 con 0x5629fb906220 2023-10-16T09:43:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facc27fc700 1 -- 172.21.15.158:0/306523482 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7facd80596a0 con 0x5629fb906220 2023-10-16T09:43:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facf6313b80 10 client.4431 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T09:43:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facf6313b80 10 client.4431 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:43:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facf6313b80 20 client.4431 populate_metadata read hostname 'smithi158' 2023-10-16T09:43:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facf6313b80 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb8e36d0 0x5629fb8e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facf6313b80 4 client.4431 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T09:43:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 --> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5629fb82eb20 con 0x5629fb8e36d0 2023-10-16T09:43:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7face37fe700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb8e36d0 0x5629fb8e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facc27fc700 10 client.4431 ms_handle_connect on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:29.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7face37fe700 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb8e36d0 0x5629fb8e3ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82ed9f95715b705 server_cookie=f2b1b4be96df85a5 in_seq=0 out_seq=0 2023-10-16T09:43:29.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facc27fc700 1 -- 172.21.15.158:0/306523482 <== mds.0 v2:172.21.15.172:6833/3958845884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5629fb82eb20 con 0x5629fb8e36d0 2023-10-16T09:43:29.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facc27fc700 10 client.4431 handle_command_reply: tid=0 2023-10-16T09:43:29.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.336+0000 7facf6313b80 1 client.4431 shutdown 2023-10-16T09:43:29.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 msgr2=0x5629fb90eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb90eac0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7facd8055940 tx=0x7facdc011000 comp rx=0 tx=0).stop 2023-10-16T09:43:29.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 shutdown_connections 2023-10-16T09:43:29.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x5629fb8e36d0 0x5629fb8e3ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5629fb90b630 0x5629fb90f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5629fb8e1010 0x5629fb90e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facc27fc700 0 client.4431 ms_handle_reset on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 --2- 172.21.15.158:0/306523482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5629fb906220 0x5629fb90eac0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 >> 172.21.15.158:0/306523482 conn(0x5629fb810d80 msgr2=0x5629fb9097e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 shutdown_connections 2023-10-16T09:43:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 1 -- 172.21.15.158:0/306523482 wait complete. 2023-10-16T09:43:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facf6313b80 20 client.4431 trim_cache size 0 max 16384 2023-10-16T09:43:29.361 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:43:29.361 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T09:43:29.362 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T09:43:29.362 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 4267, 2023-10-16T09:43:29.362 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T09:43:29.362 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T09:43:29.362 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T09:43:29.363 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 7.4150789890000004, 2023-10-16T09:43:29.363 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 7, 2023-10-16T09:43:29.363 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 24, 2023-10-16T09:43:29.363 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 23, 2023-10-16T09:43:29.363 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 13.923875328999999 2023-10-16T09:43:29.363 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:43:29.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7facd006af30 msgr2=0x7facd006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7facd006af30 0x7facd006d3f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7facdc0075b0 tx=0x7facdc007540 comp rx=0 tx=0).stop 2023-10-16T09:43:29.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415bac0 msgr2=0x7face40c9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415bac0 0x7face40c9540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7facd8014040 tx=0x7facd8004750 comp rx=0 tx=0).stop 2023-10-16T09:43:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 shutdown_connections 2023-10-16T09:43:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7face415f4d0 0x7face40c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7facd006af30 0x7facd006d3f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7face415b6e0 0x7face4148e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 --2- 172.21.15.158:0/2298976075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7face415bac0 0x7face40c9540 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 >> 172.21.15.158:0/2298976075 conn(0x7face406b9e0 msgr2=0x7face406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 shutdown_connections 2023-10-16T09:43:29.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.337+0000 7facea129700 1 -- 172.21.15.158:0/2298976075 wait complete. 2023-10-16T09:43:29.368 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T09:43:29.368 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:29.369 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:43:29.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.487+0000 7f79fc3bc700 1 Processor -- start 2023-10-16T09:43:29.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.488+0000 7f79fc3bc700 1 -- start start 2023-10-16T09:43:29.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fc3bc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f79f406e5d0 0x7f79f406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fc3bc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 0x7f79f4071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fc3bc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f4071e50 con 0x7f79f406ab40 2023-10-16T09:43:29.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fc3bc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f4161270 con 0x7f79f406e5d0 2023-10-16T09:43:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fc3bc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4161510 con 0x7f79f40729b0 2023-10-16T09:43:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fa158700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f79f406ab40 0x7f79f40722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46226/0 (socket says 172.21.15.158:46226) 2023-10-16T09:43:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fa158700 1 -- 172.21.15.158:0/3747347777 learned_addr learned my addr 172.21.15.158:0/3747347777 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fa959700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 0x7f79f4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9957700 1 --2- 172.21.15.158:0/3747347777 >> v2:172.21.15.172:3300/0 conn(0x7f79f406e5d0 0x7f79f406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229864126 0 0) 0x7f79f4071e50 con 0x7f79f406ab40 2023-10-16T09:43:29.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79dc0035b0 con 0x7f79f406ab40 2023-10-16T09:43:29.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 744964791 0 0) 0x7f79dc0035b0 con 0x7f79f406ab40 2023-10-16T09:43:29.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 msgr2=0x7f79f4071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 0x7f79f4071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 >> v2:172.21.15.172:3300/0 conn(0x7f79f406e5d0 msgr2=0x7f79f406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 --2- 172.21.15.158:0/3747347777 >> v2:172.21.15.172:3300/0 conn(0x7f79f406e5d0 0x7f79f406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f41617f0 con 0x7f79f406ab40 2023-10-16T09:43:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f79ec0019f0 con 0x7f79f406ab40 2023-10-16T09:43:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 >> v1:172.21.15.158:6789/0 conn(0x7f79f406ab40 legacy=0x7f79f40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc003a00 0x7f79dc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 0x7f79dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79dc004d90 0x7f79dc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4161270 con 0x7f79dc004d90 2023-10-16T09:43:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4161510 con 0x7f79dc003a00 2023-10-16T09:43:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc00ba10 con 0x7f79f40729b0 2023-10-16T09:43:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fa959700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79dc004d90 0x7f79dc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79fa158700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc003a00 0x7f79dc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.489+0000 7f79f9957700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 0x7f79dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fa959700 1 -- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 msgr2=0x7f79dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fa959700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40729b0 0x7f79dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fa959700 1 -- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc003a00 msgr2=0x7f79dc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fa959700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc003a00 0x7f79dc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fa959700 1 -- 172.21.15.158:0/3747347777 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0035b0 con 0x7f79dc004d90 2023-10-16T09:43:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fa959700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79dc004d90 0x7f79dc005210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009b30 tx=0x7f79dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6254eca561a683cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e400e6c0 con 0x7f79dc004d90 2023-10-16T09:43:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e400e820 con 0x7f79dc004d90 2023-10-16T09:43:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79f9156700 1 -- 172.21.15.158:0/3747347777 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e400eb90 con 0x7f79dc004d90 2023-10-16T09:43:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fc3bc700 1 -- 172.21.15.158:0/3747347777 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79dc004d90 msgr2=0x7f79dc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fc3bc700 1 --2- 172.21.15.158:0/3747347777 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79dc004d90 0x7f79dc005210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009b30 tx=0x7f79dc005880 comp rx=0 tx=0).stop 2023-10-16T09:43:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.490+0000 7f79fa959700 1 -- 172.21.15.158:0/3747347777 reap_dead start 2023-10-16T09:43:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.491+0000 7f79fc3bc700 1 -- 172.21.15.158:0/3747347777 shutdown_connections 2023-10-16T09:43:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.491+0000 7f79fc3bc700 1 -- 172.21.15.158:0/3747347777 >> 172.21.15.158:0/3747347777 conn(0x7f79f40d02b0 msgr2=0x7f79f40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.491+0000 7f79fc3bc700 1 -- 172.21.15.158:0/3747347777 shutdown_connections 2023-10-16T09:43:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.491+0000 7f79fc3bc700 1 -- 172.21.15.158:0/3747347777 wait complete. 2023-10-16T09:43:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.492+0000 7f79fc3bc700 1 Processor -- start 2023-10-16T09:43:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.492+0000 7f79fc3bc700 1 -- start start 2023-10-16T09:43:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.492+0000 7f79fc3bc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79dc004d90 0x7f79f40b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fc3bc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f414f430 0x7f79f40ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fc3bc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f414e870 0x7f79f414ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fc3bc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f41e8de0 con 0x7f79f414f430 2023-10-16T09:43:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fc3bc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f414f230 con 0x7f79f414e870 2023-10-16T09:43:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fc3bc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f41e9050 con 0x7f79dc004d90 2023-10-16T09:43:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f414f430 0x7f79f40ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f414f430 0x7f79f40ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46064/0 (socket says 172.21.15.158:46064) 2023-10-16T09:43:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 -- 172.21.15.158:0/2850229143 learned_addr learned my addr 172.21.15.158:0/2850229143 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fa158700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79dc004d90 0x7f79f40b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fa959700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f414e870 0x7f79f414ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 -- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79dc004d90 msgr2=0x7f79f40b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79dc004d90 0x7f79f40b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 -- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f414e870 msgr2=0x7f79f414ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f414e870 0x7f79f414ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 -- 172.21.15.158:0/2850229143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f41e9330 con 0x7f79f414f430 2023-10-16T09:43:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fa959700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f414e870 0x7f79f414ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f9957700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f414f430 0x7f79f40ad490 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f79e8011a50 tx=0x7f79e8011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f2ffd700 1 -- 172.21.15.158:0/2850229143 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e800fa00 con 0x7f79f414f430 2023-10-16T09:43:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f2ffd700 1 -- 172.21.15.158:0/2850229143 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e800fb60 con 0x7f79f414f430 2023-10-16T09:43:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f41e9610 con 0x7f79f414f430 2023-10-16T09:43:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79f2ffd700 1 -- 172.21.15.158:0/2850229143 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e8014440 con 0x7f79f414f430 2023-10-16T09:43:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.493+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f40b2940 con 0x7f79f414f430 2023-10-16T09:43:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.494+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f40681b0 con 0x7f79f414f430 2023-10-16T09:43:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.495+0000 7f79f2ffd700 1 -- 172.21.15.158:0/2850229143 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f79e80145a0 con 0x7f79f414f430 2023-10-16T09:43:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.495+0000 7f79f2ffd700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f79e006af00 0x7f79e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.495+0000 7f79f2ffd700 1 -- 172.21.15.158:0/2850229143 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f79e808a030 con 0x7f79f414f430 2023-10-16T09:43:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.495+0000 7f79fa158700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f79e006af00 0x7f79e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.495+0000 7f79fa158700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f79e006af00 0x7f79e006d3c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0013d0 tx=0x7f79ec006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.499+0000 7f79f2ffd700 1 -- 172.21.15.158:0/2850229143 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79e8054e80 con 0x7f79f414f430 2023-10-16T09:43:29.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.737+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f79f40b2f60 con 0x7f79f414f430 2023-10-16T09:43:29.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.738+0000 7f79f2ffd700 1 -- 172.21.15.158:0/2850229143 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f79e80584a0 con 0x7f79f414f430 2023-10-16T09:43:29.740 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T09:43:29.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f79e006af00 msgr2=0x7f79e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f79e006af00 0x7f79e006d3c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0013d0 tx=0x7f79ec006040 comp rx=0 tx=0).stop 2023-10-16T09:43:29.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f414f430 msgr2=0x7f79f40ad490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:29.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f414f430 0x7f79f40ad490 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f79e8011a50 tx=0x7f79e8011e10 comp rx=0 tx=0).stop 2023-10-16T09:43:29.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 shutdown_connections 2023-10-16T09:43:29.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f414e870 0x7f79f414ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f79e006af00 0x7f79e006d3c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79dc004d90 0x7f79f40b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 --2- 172.21.15.158:0/2850229143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f414f430 0x7f79f40ad490 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:29.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 >> 172.21.15.158:0/2850229143 conn(0x7f79f40d02b0 msgr2=0x7f79f406d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:29.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 shutdown_connections 2023-10-16T09:43:29.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:29.740+0000 7f79fc3bc700 1 -- 172.21.15.158:0/2850229143 wait complete. 2023-10-16T09:43:29.753 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T09:43:29.754 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:43:29.754 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T09:43:29.754 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-16T09:43:29.755 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:29.755 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T09:43:29.755 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T09:43:29.755 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:43:29.756 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-16T09:43:29.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:29.757 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T09:43:29.774 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:43:29.774 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:43:29.774 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:43:29.774 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:29.774 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:43:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:29.775 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:29.775 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:29.776 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:29.776 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:43:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:43:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 42423sec preferred_lft 42423sec 2023-10-16T09:43:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T09:43:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:29.778 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T09:43:29.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:29.778 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:29.779 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:29.779 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add name ceph-brx type bridge 2023-10-16T09:43:29.779 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip addr flush dev ceph-brx 2023-10-16T09:43:29.780 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set ceph-brx up 2023-10-16T09:43:29.780 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T09:43:29.780 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:29.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:29.944 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:43:29.973 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T09:43:29.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:29.974 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T09:43:30.049 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T09:43:30.050 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:43:30.050 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:43:30.050 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:43:30.051 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:43:30.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:30.052 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:30.053 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:30.053 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:43:30.054 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:43:30.055 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:43:30.055 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:30.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:30.312 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:43:30.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:30.329 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list-id 2023-10-16T09:43:30.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:30.385 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:30.385 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:30.386 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T09:43:30.386 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-16T09:43:30.386 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:30.479 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-16T09:43:30.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:30.480 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:30.480 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:30.481 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-16T09:43:30.481 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T09:43:30.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-16T09:43:30.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-16T09:43:30.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-16T09:43:30.483 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:30.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:30.686 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:30.687 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:30.687 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T09:43:30.687 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T09:43:30.688 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:30.787 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-16T09:43:30.787 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:43:30.788 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:30.808 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-16T09:43:30.808 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:43:30.809 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:30.865 DEBUG:teuthology.orchestra.run.smithi158:> sudo modprobe fuse 2023-10-16T09:43:30.932 DEBUG:teuthology.orchestra.run.smithi158:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T09:43:30.989 INFO:teuthology.orchestra.run.smithi158.stdout:/sys 2023-10-16T09:43:30.990 INFO:teuthology.orchestra.run.smithi158.stdout:/proc 2023-10-16T09:43:30.990 INFO:teuthology.orchestra.run.smithi158.stdout:/dev 2023-10-16T09:43:30.990 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/security 2023-10-16T09:43:30.990 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/shm 2023-10-16T09:43:30.991 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/pts 2023-10-16T09:43:30.991 INFO:teuthology.orchestra.run.smithi158.stdout:/run 2023-10-16T09:43:30.991 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup 2023-10-16T09:43:30.991 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/systemd 2023-10-16T09:43:30.991 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/pstore 2023-10-16T09:43:30.992 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/bpf 2023-10-16T09:43:30.992 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/cpuset 2023-10-16T09:43:30.992 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/blkio 2023-10-16T09:43:30.992 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/freezer 2023-10-16T09:43:30.992 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/cpu,cpuacct 2023-10-16T09:43:30.993 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/hugetlb 2023-10-16T09:43:30.993 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/rdma 2023-10-16T09:43:30.993 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/pids 2023-10-16T09:43:30.993 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/net_cls,net_prio 2023-10-16T09:43:30.993 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/memory 2023-10-16T09:43:30.994 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/perf_event 2023-10-16T09:43:30.994 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/devices 2023-10-16T09:43:30.994 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/tracing 2023-10-16T09:43:30.994 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/config 2023-10-16T09:43:30.994 INFO:teuthology.orchestra.run.smithi158.stdout:/ 2023-10-16T09:43:30.995 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/selinux 2023-10-16T09:43:30.995 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/debug 2023-10-16T09:43:30.995 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/mqueue 2023-10-16T09:43:30.995 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/hugepages 2023-10-16T09:43:30.995 INFO:teuthology.orchestra.run.smithi158.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T09:43:30.995 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/fuse/connections 2023-10-16T09:43:30.996 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/nfs/rpc_pipefs 2023-10-16T09:43:30.996 INFO:teuthology.orchestra.run.smithi158.stdout:/run/user/1000 2023-10-16T09:43:30.996 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph 2023-10-16T09:43:30.996 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T09:43:30.996 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T09:43:30.997 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T09:43:30.997 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T09:43:30.997 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns 2023-10-16T09:43:30.997 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T09:43:30.997 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T09:43:30.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:30.998 DEBUG:teuthology.orchestra.run.smithi158:> ls /sys/fs/fuse/connections 2023-10-16T09:43:31.047 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-16T09:43:31.047 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-16T09:43:31.090 DEBUG:teuthology.orchestra.run.smithi158:> sudo modprobe fuse 2023-10-16T09:43:31.116 DEBUG:teuthology.orchestra.run.smithi158:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T09:43:31.174 INFO:teuthology.orchestra.run.smithi158.stdout:/sys 2023-10-16T09:43:31.174 INFO:teuthology.orchestra.run.smithi158.stdout:/proc 2023-10-16T09:43:31.175 INFO:teuthology.orchestra.run.smithi158.stdout:/dev 2023-10-16T09:43:31.175 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/security 2023-10-16T09:43:31.175 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/shm 2023-10-16T09:43:31.175 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/pts 2023-10-16T09:43:31.175 INFO:teuthology.orchestra.run.smithi158.stdout:/run 2023-10-16T09:43:31.176 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup 2023-10-16T09:43:31.176 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/systemd 2023-10-16T09:43:31.176 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/pstore 2023-10-16T09:43:31.176 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/bpf 2023-10-16T09:43:31.176 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/cpuset 2023-10-16T09:43:31.176 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/blkio 2023-10-16T09:43:31.177 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/freezer 2023-10-16T09:43:31.177 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/cpu,cpuacct 2023-10-16T09:43:31.177 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/hugetlb 2023-10-16T09:43:31.177 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/rdma 2023-10-16T09:43:31.177 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/pids 2023-10-16T09:43:31.178 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/net_cls,net_prio 2023-10-16T09:43:31.178 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/memory 2023-10-16T09:43:31.178 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/perf_event 2023-10-16T09:43:31.178 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup/devices 2023-10-16T09:43:31.179 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/tracing 2023-10-16T09:43:31.179 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/config 2023-10-16T09:43:31.179 INFO:teuthology.orchestra.run.smithi158.stdout:/ 2023-10-16T09:43:31.179 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/selinux 2023-10-16T09:43:31.179 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/debug 2023-10-16T09:43:31.179 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/mqueue 2023-10-16T09:43:31.180 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/hugepages 2023-10-16T09:43:31.180 INFO:teuthology.orchestra.run.smithi158.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T09:43:31.180 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/fuse/connections 2023-10-16T09:43:31.180 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/nfs/rpc_pipefs 2023-10-16T09:43:31.180 INFO:teuthology.orchestra.run.smithi158.stdout:/run/user/1000 2023-10-16T09:43:31.181 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph 2023-10-16T09:43:31.181 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T09:43:31.181 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T09:43:31.181 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T09:43:31.181 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T09:43:31.181 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns 2023-10-16T09:43:31.182 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T09:43:31.182 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T09:43:31.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:31.183 DEBUG:teuthology.orchestra.run.smithi158:> ls /sys/fs/fuse/connections 2023-10-16T09:43:31.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi158.stderr:2023-10-16T09:43:31.186+0000 7fdbfda565c0 -1 init, newargv = 0x562e6dd84760 newargc=15 2023-10-16T09:43:31.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi158.stderr:ceph-fuse[111211]: starting ceph client 2023-10-16T09:43:31.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi158.stderr:ceph-fuse[111211]: starting fuse 2023-10-16T09:43:31.203 INFO:teuthology.orchestra.run.smithi158.stdout:47 2023-10-16T09:43:31.204 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-10-16T09:43:31.204 DEBUG:teuthology.orchestra.run.smithi158:> sudo stdin-killer -- python3 -c ' 2023-10-16T09:43:31.204 DEBUG:teuthology.orchestra.run.smithi158:> import glob 2023-10-16T09:43:31.205 DEBUG:teuthology.orchestra.run.smithi158:> import re 2023-10-16T09:43:31.205 DEBUG:teuthology.orchestra.run.smithi158:> import os 2023-10-16T09:43:31.205 DEBUG:teuthology.orchestra.run.smithi158:> import subprocess 2023-10-16T09:43:31.206 DEBUG:teuthology.orchestra.run.smithi158:> 2023-10-16T09:43:31.206 DEBUG:teuthology.orchestra.run.smithi158:> def _find_admin_socket(client_name): 2023-10-16T09:43:31.206 DEBUG:teuthology.orchestra.run.smithi158:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-16T09:43:31.207 DEBUG:teuthology.orchestra.run.smithi158:> files = glob.glob(asok_path) 2023-10-16T09:43:31.207 DEBUG:teuthology.orchestra.run.smithi158:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-16T09:43:31.207 DEBUG:teuthology.orchestra.run.smithi158:> 2023-10-16T09:43:31.208 DEBUG:teuthology.orchestra.run.smithi158:> # Given a non-glob path, it better be there 2023-10-16T09:43:31.208 DEBUG:teuthology.orchestra.run.smithi158:> if "*" not in asok_path: 2023-10-16T09:43:31.208 DEBUG:teuthology.orchestra.run.smithi158:> assert(len(files) == 1) 2023-10-16T09:43:31.209 DEBUG:teuthology.orchestra.run.smithi158:> return files[0] 2023-10-16T09:43:31.209 DEBUG:teuthology.orchestra.run.smithi158:> 2023-10-16T09:43:31.209 DEBUG:teuthology.orchestra.run.smithi158:> for f in files: 2023-10-16T09:43:31.210 DEBUG:teuthology.orchestra.run.smithi158:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-16T09:43:31.210 DEBUG:teuthology.orchestra.run.smithi158:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-16T09:43:31.211 DEBUG:teuthology.orchestra.run.smithi158:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-16T09:43:31.211 DEBUG:teuthology.orchestra.run.smithi158:> contents = proc_f.read() 2023-10-16T09:43:31.211 DEBUG:teuthology.orchestra.run.smithi158:> if mountpoint in contents: 2023-10-16T09:43:31.211 DEBUG:teuthology.orchestra.run.smithi158:> return f 2023-10-16T09:43:31.212 DEBUG:teuthology.orchestra.run.smithi158:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-16T09:43:31.212 DEBUG:teuthology.orchestra.run.smithi158:> 2023-10-16T09:43:31.212 DEBUG:teuthology.orchestra.run.smithi158:> print(_find_admin_socket("client.admin")) 2023-10-16T09:43:31.213 DEBUG:teuthology.orchestra.run.smithi158:> ' 2023-10-16T09:43:31.327 INFO:teuthology.orchestra.run.smithi158.stdout:/var/run/ceph/ceph-client.admin.111211.asok 2023-10-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-16T09:43:31.336 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.111211.asok 2023-10-16T09:43:31.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:31.337 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.111211.asok status 2023-10-16T09:43:31.492 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:43:31.493 INFO:teuthology.orchestra.run.smithi158.stdout: "metadata": { 2023-10-16T09:43:31.493 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_sha1": "c7471068426370a3754c2250e7729280233d552b", 2023-10-16T09:43:31.493 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_version": "ceph version 18.0.0-6675-gc7471068 (c7471068426370a3754c2250e7729280233d552b) reef (dev)", 2023-10-16T09:43:31.493 INFO:teuthology.orchestra.run.smithi158.stdout: "entity_id": "admin", 2023-10-16T09:43:31.493 INFO:teuthology.orchestra.run.smithi158.stdout: "hostname": "smithi158", 2023-10-16T09:43:31.493 INFO:teuthology.orchestra.run.smithi158.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-16T09:43:31.494 INFO:teuthology.orchestra.run.smithi158.stdout: "pid": "111211", 2023-10-16T09:43:31.494 INFO:teuthology.orchestra.run.smithi158.stdout: "root": "/" 2023-10-16T09:43:31.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:43:31.494 INFO:teuthology.orchestra.run.smithi158.stdout: "dentry_count": 0, 2023-10-16T09:43:31.494 INFO:teuthology.orchestra.run.smithi158.stdout: "dentry_pinned_count": 0, 2023-10-16T09:43:31.495 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 4446, 2023-10-16T09:43:31.495 INFO:teuthology.orchestra.run.smithi158.stdout: "inst": { 2023-10-16T09:43:31.495 INFO:teuthology.orchestra.run.smithi158.stdout: "name": { 2023-10-16T09:43:31.495 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "client", 2023-10-16T09:43:31.495 INFO:teuthology.orchestra.run.smithi158.stdout: "num": 4446 2023-10-16T09:43:31.496 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:43:31.496 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": { 2023-10-16T09:43:31.496 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "v1", 2023-10-16T09:43:31.496 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": "192.168.0.1:0", 2023-10-16T09:43:31.496 INFO:teuthology.orchestra.run.smithi158.stdout: "nonce": 3710524148 2023-10-16T09:43:31.497 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:43:31.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:43:31.497 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": { 2023-10-16T09:43:31.497 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "v1", 2023-10-16T09:43:31.497 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": "192.168.0.1:0", 2023-10-16T09:43:31.497 INFO:teuthology.orchestra.run.smithi158.stdout: "nonce": 3710524148 2023-10-16T09:43:31.498 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:43:31.498 INFO:teuthology.orchestra.run.smithi158.stdout: "inst_str": "client.4446 192.168.0.1:0/3710524148", 2023-10-16T09:43:31.498 INFO:teuthology.orchestra.run.smithi158.stdout: "addr_str": "192.168.0.1:0/3710524148", 2023-10-16T09:43:31.498 INFO:teuthology.orchestra.run.smithi158.stdout: "inode_count": 1, 2023-10-16T09:43:31.498 INFO:teuthology.orchestra.run.smithi158.stdout: "mds_epoch": 7, 2023-10-16T09:43:31.498 INFO:teuthology.orchestra.run.smithi158.stdout: "osd_epoch": 24, 2023-10-16T09:43:31.499 INFO:teuthology.orchestra.run.smithi158.stdout: "osd_epoch_barrier": 0, 2023-10-16T09:43:31.499 INFO:teuthology.orchestra.run.smithi158.stdout: "blocklisted": false, 2023-10-16T09:43:31.499 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs" 2023-10-16T09:43:31.499 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:43:31.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:31.503 DEBUG:teuthology.orchestra.run.smithi158:> stat --file-system '--printf=%T 2023-10-16T09:43:31.503 DEBUG:teuthology.orchestra.run.smithi158:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:31.520 INFO:teuthology.orchestra.run.smithi158.stdout:fuseblk 2023-10-16T09:43:31.520 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:31.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:31.521 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:31.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:31.588 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-16T09:43:31.691 INFO:teuthology.orchestra.run.smithi158.stdout:# file: . 2023-10-16T09:43:31.692 INFO:teuthology.orchestra.run.smithi158.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-16T09:43:31.692 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:31.692 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:43:31.693 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:43:31.710 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:31.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.868+0000 7fdc73236700 1 Processor -- start 2023-10-16T09:43:31.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.868+0000 7fdc73236700 1 -- start start 2023-10-16T09:43:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc73236700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc6c15d7c0 0x7fdc6c161290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc73236700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdc6c0ca5b0 0x7fdc6c1617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc73236700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc73236700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c4d10 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc73236700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c5570 con 0x7fdc6c0ca5b0 2023-10-16T09:43:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc73236700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c15dba0 con 0x7fdc6c06b300 2023-10-16T09:43:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc70fd2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc6c15d7c0 0x7fdc6c161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc70fd2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc6c15d7c0 0x7fdc6c161290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46086/0 (socket says 172.21.15.158:46086) 2023-10-16T09:43:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc70fd2700 1 -- 172.21.15.158:0/2029200277 learned_addr learned my addr 172.21.15.158:0/2029200277 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc6bfff700 1 --2- 172.21.15.158:0/2029200277 >> v2:172.21.15.172:3300/0 conn(0x7fdc6c0ca5b0 0x7fdc6c1617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.869+0000 7fdc717d3700 1 --2- 172.21.15.158:0/2029200277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc70fd2700 1 -- 172.21.15.158:0/2029200277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 msgr2=0x7fdc6c06b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc70fd2700 1 --2- 172.21.15.158:0/2029200277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c06b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc70fd2700 1 -- 172.21.15.158:0/2029200277 >> v2:172.21.15.172:3300/0 conn(0x7fdc6c0ca5b0 msgr2=0x7fdc6c1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc70fd2700 1 --2- 172.21.15.158:0/2029200277 >> v2:172.21.15.172:3300/0 conn(0x7fdc6c0ca5b0 0x7fdc6c1617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc70fd2700 1 -- 172.21.15.158:0/2029200277 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c06c080 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc6bfff700 1 --2- 172.21.15.158:0/2029200277 >> v2:172.21.15.172:3300/0 conn(0x7fdc6c0ca5b0 0x7fdc6c1617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc70fd2700 1 --2- 172.21.15.158:0/2029200277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c161290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0725d0 tx=0x7fdc60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6e819bba481f23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc6b7fe700 1 -- 172.21.15.158:0/2029200277 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6000b870 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc6b7fe700 1 -- 172.21.15.158:0/2029200277 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6000b9d0 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc6b7fe700 1 -- 172.21.15.158:0/2029200277 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6000bd60 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc73236700 1 -- 172.21.15.158:0/2029200277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 msgr2=0x7fdc6c161290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:31.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.870+0000 7fdc73236700 1 --2- 172.21.15.158:0/2029200277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c161290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0725d0 tx=0x7fdc60009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.871+0000 7fdc73236700 1 -- 172.21.15.158:0/2029200277 shutdown_connections 2023-10-16T09:43:31.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.871+0000 7fdc73236700 1 --2- 172.21.15.158:0/2029200277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c06b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.871+0000 7fdc73236700 1 --2- 172.21.15.158:0/2029200277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c161290 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.871+0000 7fdc73236700 1 --2- 172.21.15.158:0/2029200277 >> v2:172.21.15.172:3300/0 conn(0x7fdc6c0ca5b0 0x7fdc6c1617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.871+0000 7fdc73236700 1 -- 172.21.15.158:0/2029200277 >> 172.21.15.158:0/2029200277 conn(0x7fdc6c0cc4b0 msgr2=0x7fdc6c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.871+0000 7fdc73236700 1 -- 172.21.15.158:0/2029200277 shutdown_connections 2023-10-16T09:43:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.871+0000 7fdc73236700 1 -- 172.21.15.158:0/2029200277 wait complete. 2023-10-16T09:43:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.872+0000 7fdc73236700 1 Processor -- start 2023-10-16T09:43:31.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.872+0000 7fdc73236700 1 -- start start 2023-10-16T09:43:31.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c143830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc6c0ca5b0 0x7fdc6c140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c1409a0 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c15abb0 con 0x7fdc6c0ca5b0 2023-10-16T09:43:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c6ba0 con 0x7fdc6c06b300 2023-10-16T09:43:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c15a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46096/0 (socket says 172.21.15.158:46096) 2023-10-16T09:43:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 -- 172.21.15.158:0/1742894339 learned_addr learned my addr 172.21.15.158:0/1742894339 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc6bfff700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc6c0ca5b0 0x7fdc6c140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc70fd2700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 -- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 msgr2=0x7fdc6c143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c143830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 -- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc6c0ca5b0 msgr2=0x7fdc6c140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc6c0ca5b0 0x7fdc6c140460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 -- 172.21.15.158:0/1742894339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc60009550 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc70fd2700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c143830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:31.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc717d3700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c15a670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdc64007e40 tx=0x7fdc6400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:31.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc697fa700 1 -- 172.21.15.158:0/1742894339 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400ede0 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc697fa700 1 -- 172.21.15.158:0/1742894339 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc64014420 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c0c6e80 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc697fa700 1 -- 172.21.15.158:0/1742894339 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc640147d0 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.873+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c0c8b10 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.874+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c0c73a0 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.875+0000 7fdc697fa700 1 -- 172.21.15.158:0/1742894339 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdc64014930 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.875+0000 7fdc697fa700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc5006af00 0x7fdc5006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.876+0000 7fdc697fa700 1 -- 172.21.15.158:0/1742894339 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdc64089fe0 con 0x7fdc6c15d7c0 2023-10-16T09:43:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.876+0000 7fdc70fd2700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc5006af00 0x7fdc5006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.876+0000 7fdc70fd2700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc5006af00 0x7fdc5006d3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdc60014040 tx=0x7fdc60008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:31.880+0000 7fdc697fa700 1 -- 172.21.15.158:0/1742894339 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc6c0c73a0 con 0x7fdc6c15d7c0 2023-10-16T09:43:32.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.074+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc6c0c73a0 con 0x7fdc6c15d7c0 2023-10-16T09:43:32.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.076+0000 7fdc697fa700 1 -- 172.21.15.158:0/1742894339 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7fdc6c0c73a0 con 0x7fdc6c15d7c0 2023-10-16T09:43:32.078 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:32.078 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:32.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc5006af00 msgr2=0x7fdc5006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc5006af00 0x7fdc5006d3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdc60014040 tx=0x7fdc60008030 comp rx=0 tx=0).stop 2023-10-16T09:43:32.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 msgr2=0x7fdc6c15a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c15a670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdc64007e40 tx=0x7fdc6400e3c0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 shutdown_connections 2023-10-16T09:43:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc6c15d7c0 0x7fdc6c15a670 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc5006af00 0x7fdc5006d3c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc6c06b300 0x7fdc6c143830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 --2- 172.21.15.158:0/1742894339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc6c0ca5b0 0x7fdc6c140460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 >> 172.21.15.158:0/1742894339 conn(0x7fdc6c0cc4b0 msgr2=0x7fdc6c0cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 shutdown_connections 2023-10-16T09:43:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.078+0000 7fdc73236700 1 -- 172.21.15.158:0/1742894339 wait complete. 2023-10-16T09:43:32.092 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:43:32.092 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:43:32.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.214+0000 7f717bf36700 1 Processor -- start 2023-10-16T09:43:32.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.215+0000 7f717bf36700 1 -- start start 2023-10-16T09:43:32.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.215+0000 7f717bf36700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f717415deb0 0x7f71740ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.215+0000 7f717bf36700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71740d0ae0 con 0x7f71740ccc00 2023-10-16T09:43:32.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.215+0000 7f717bf36700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71740d12d0 con 0x7f717415dad0 2023-10-16T09:43:32.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.215+0000 7f717bf36700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71740cbdf0 con 0x7f717415deb0 2023-10-16T09:43:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.215+0000 7f717a4d3700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f71740ccc00 0x7f71740cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46280/0 (socket says 172.21.15.158:46280) 2023-10-16T09:43:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f717a4d3700 1 -- 172.21.15.158:0/139382399 learned_addr learned my addr 172.21.15.158:0/139382399 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f71794d1700 1 --2- 172.21.15.158:0/139382399 >> v2:172.21.15.172:3300/0 conn(0x7f717415deb0 0x7f71740ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f7178cd0700 1 -- 172.21.15.158:0/139382399 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147006835 0 0) 0x7f71740d0ae0 con 0x7f71740ccc00 2023-10-16T09:43:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f7178cd0700 1 -- 172.21.15.158:0/139382399 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71600035b0 con 0x7f71740ccc00 2023-10-16T09:43:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f71794d1700 1 -- 172.21.15.158:0/139382399 >> v1:172.21.15.172:6789/0 conn(0x7f717415dad0 legacy=0x7f71740cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f71794d1700 1 -- 172.21.15.158:0/139382399 >> v1:172.21.15.158:6789/0 conn(0x7f71740ccc00 legacy=0x7f71740cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f71794d1700 1 -- 172.21.15.158:0/139382399 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71740cc280 con 0x7f717415deb0 2023-10-16T09:43:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f71794d1700 1 --2- 172.21.15.158:0/139382399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f717415deb0 0x7f71740ce6b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7170009870 tx=0x7f7170009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0dee8cd1193eb0e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f7178cd0700 1 -- 172.21.15.158:0/139382399 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170011830 con 0x7f717415deb0 2023-10-16T09:43:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f7178cd0700 1 -- 172.21.15.158:0/139382399 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170011990 con 0x7f717415deb0 2023-10-16T09:43:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.216+0000 7f7178cd0700 1 -- 172.21.15.158:0/139382399 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170011d20 con 0x7f717415deb0 2023-10-16T09:43:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.217+0000 7f717bf36700 1 -- 172.21.15.158:0/139382399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f717415deb0 msgr2=0x7f71740ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.217+0000 7f717bf36700 1 --2- 172.21.15.158:0/139382399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f717415deb0 0x7f71740ce6b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7170009870 tx=0x7f7170009b80 comp rx=0 tx=0).stop 2023-10-16T09:43:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.217+0000 7f717bf36700 1 -- 172.21.15.158:0/139382399 shutdown_connections 2023-10-16T09:43:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.217+0000 7f717bf36700 1 --2- 172.21.15.158:0/139382399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f717415deb0 0x7f71740ce6b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.217+0000 7f717bf36700 1 -- 172.21.15.158:0/139382399 >> 172.21.15.158:0/139382399 conn(0x7f7174099a10 msgr2=0x7f71740d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.217+0000 7f717bf36700 1 -- 172.21.15.158:0/139382399 shutdown_connections 2023-10-16T09:43:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.217+0000 7f717bf36700 1 -- 172.21.15.158:0/139382399 wait complete. 2023-10-16T09:43:32.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.218+0000 7f717bf36700 1 Processor -- start 2023-10-16T09:43:32.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.218+0000 7f717bf36700 1 -- start start 2023-10-16T09:43:32.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717bf36700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71740ccc00 0x7f71740c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717bf36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f717415dad0 0x7f71741f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717bf36700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f717415deb0 0x7f71741e5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717bf36700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740ca380 con 0x7f717415dad0 2023-10-16T09:43:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717bf36700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7174146310 con 0x7f71740ccc00 2023-10-16T09:43:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717bf36700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71741465d0 con 0x7f717415deb0 2023-10-16T09:43:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f717415dad0 0x7f71741f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f717415dad0 0x7f71741f2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46108/0 (socket says 172.21.15.158:46108) 2023-10-16T09:43:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 -- 172.21.15.158:0/4232861521 learned_addr learned my addr 172.21.15.158:0/4232861521 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f7179cd2700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71740ccc00 0x7f71740c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717a4d3700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f717415deb0 0x7f71741e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 -- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f717415deb0 msgr2=0x7f71741e5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f717415deb0 0x7f71741e5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 -- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71740ccc00 msgr2=0x7f71740c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71740ccc00 0x7f71740c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 -- 172.21.15.158:0/4232861521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71700094a0 con 0x7f717415dad0 2023-10-16T09:43:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f717a4d3700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f717415deb0 0x7f71741e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.219+0000 7f71794d1700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f717415dad0 0x7f71741f2cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7170014040 tx=0x7f7170004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.220+0000 7f716a7fc700 1 -- 172.21.15.158:0/4232861521 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71700036a0 con 0x7f717415dad0 2023-10-16T09:43:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.220+0000 7f716a7fc700 1 -- 172.21.15.158:0/4232861521 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717001e070 con 0x7f717415dad0 2023-10-16T09:43:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.220+0000 7f716a7fc700 1 -- 172.21.15.158:0/4232861521 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170003a00 con 0x7f717415dad0 2023-10-16T09:43:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.220+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71741468b0 con 0x7f717415dad0 2023-10-16T09:43:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.220+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7174146e70 con 0x7f717415dad0 2023-10-16T09:43:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.221+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71740681b0 con 0x7f717415dad0 2023-10-16T09:43:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.221+0000 7f716a7fc700 1 -- 172.21.15.158:0/4232861521 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7170003b60 con 0x7f717415dad0 2023-10-16T09:43:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.221+0000 7f716a7fc700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f715806af00 0x7f715806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.221+0000 7f716a7fc700 1 -- 172.21.15.158:0/4232861521 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f717008a930 con 0x7f717415dad0 2023-10-16T09:43:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.221+0000 7f7179cd2700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f715806af00 0x7f715806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.222+0000 7f7179cd2700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f715806af00 0x7f715806d3c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7164000f60 tx=0x7f7164009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.226+0000 7f716a7fc700 1 -- 172.21.15.158:0/4232861521 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7170055700 con 0x7f717415dad0 2023-10-16T09:43:32.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.419+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71740cf350 con 0x7f717415dad0 2023-10-16T09:43:32.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.419+0000 7f716a7fc700 1 -- 172.21.15.158:0/4232861521 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7170015020 con 0x7f717415dad0 2023-10-16T09:43:32.421 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:43:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f715806af00 msgr2=0x7f715806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f715806af00 0x7f715806d3c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7164000f60 tx=0x7f7164009040 comp rx=0 tx=0).stop 2023-10-16T09:43:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f717415dad0 msgr2=0x7f71741f2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f717415dad0 0x7f71741f2cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7170014040 tx=0x7f7170004750 comp rx=0 tx=0).stop 2023-10-16T09:43:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 shutdown_connections 2023-10-16T09:43:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f715806af00 0x7f715806d3c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f717415deb0 0x7f71741e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71740ccc00 0x7f71740c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 --2- 172.21.15.158:0/4232861521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f717415dad0 0x7f71741f2cc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 >> 172.21.15.158:0/4232861521 conn(0x7f7174099a10 msgr2=0x7f717406de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:32.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 shutdown_connections 2023-10-16T09:43:32.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.421+0000 7f717bf36700 1 -- 172.21.15.158:0/4232861521 wait complete. 2023-10-16T09:43:32.434 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:43:32.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.556+0000 7f2a08c14700 1 Processor -- start 2023-10-16T09:43:32.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.556+0000 7f2a08c14700 1 -- start start 2023-10-16T09:43:32.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a08c14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a0415da40 0x7f2a040caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a08c14700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a040cc590 0x7f2a0406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a08c14700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a0406ae10 0x7f2a040ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a08c14700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0406b280 con 0x7f2a0406ae10 2023-10-16T09:43:32.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a08c14700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a040cbd30 con 0x7f2a040cc590 2023-10-16T09:43:32.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a08c14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040cc1c0 con 0x7f2a0415da40 2023-10-16T09:43:32.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a0406ae10 0x7f2a040ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a0406ae10 0x7f2a040ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46118/0 (socket says 172.21.15.158:46118) 2023-10-16T09:43:32.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 -- 172.21.15.158:0/3770403145 learned_addr learned my addr 172.21.15.158:0/3770403145 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:32.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a01d9b700 1 --2- 172.21.15.158:0/3770403145 >> v2:172.21.15.172:3300/0 conn(0x7f2a040cc590 0x7f2a0406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a0259c700 1 --2- 172.21.15.158:0/3770403145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a0415da40 0x7f2a040caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 -- 172.21.15.158:0/3770403145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a0415da40 msgr2=0x7f2a040caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:32.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 --2- 172.21.15.158:0/3770403145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a0415da40 0x7f2a040caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 -- 172.21.15.158:0/3770403145 >> v2:172.21.15.172:3300/0 conn(0x7f2a040cc590 msgr2=0x7f2a0406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:32.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 --2- 172.21.15.158:0/3770403145 >> v2:172.21.15.172:3300/0 conn(0x7f2a040cc590 0x7f2a0406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 -- 172.21.15.158:0/3770403145 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0415de20 con 0x7f2a0406ae10 2023-10-16T09:43:32.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a02d9d700 1 --2- 172.21.15.158:0/3770403145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0406ae10 0x7f2a040ccd60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2a04070430 tx=0x7f29fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3bf3c35714f368 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.557+0000 7f2a0159a700 1 -- 172.21.15.158:0/3770403145 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00e070 con 0x7f2a0406ae10 2023-10-16T09:43:32.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a0159a700 1 -- 172.21.15.158:0/3770403145 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc00b910 con 0x7f2a0406ae10 2023-10-16T09:43:32.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a0159a700 1 -- 172.21.15.158:0/3770403145 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00bcc0 con 0x7f2a0406ae10 2023-10-16T09:43:32.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 -- 172.21.15.158:0/3770403145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0406ae10 msgr2=0x7f2a040ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 --2- 172.21.15.158:0/3770403145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0406ae10 0x7f2a040ccd60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2a04070430 tx=0x7f29fc00b330 comp rx=0 tx=0).stop 2023-10-16T09:43:32.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 -- 172.21.15.158:0/3770403145 shutdown_connections 2023-10-16T09:43:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 --2- 172.21.15.158:0/3770403145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0406ae10 0x7f2a040ccd60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 --2- 172.21.15.158:0/3770403145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a0415da40 0x7f2a040caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 --2- 172.21.15.158:0/3770403145 >> v2:172.21.15.172:3300/0 conn(0x7f2a040cc590 0x7f2a0406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 -- 172.21.15.158:0/3770403145 >> 172.21.15.158:0/3770403145 conn(0x7f2a04099a10 msgr2=0x7f2a040d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 -- 172.21.15.158:0/3770403145 shutdown_connections 2023-10-16T09:43:32.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.558+0000 7f2a08c14700 1 -- 172.21.15.158:0/3770403145 wait complete. 2023-10-16T09:43:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.560+0000 7f2a08c14700 1 Processor -- start 2023-10-16T09:43:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.560+0000 7f2a08c14700 1 -- start start 2023-10-16T09:43:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a08c14700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a0406ae10 0x7f2a04149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a08c14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cc590 0x7f2a041e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a08c14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0415da40 0x7f2a041e4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a08c14700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0415c0a0 con 0x7f2a0415da40 2023-10-16T09:43:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a08c14700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0415b000 con 0x7f2a0406ae10 2023-10-16T09:43:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a08c14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0415b2e0 con 0x7f2a040cc590 2023-10-16T09:43:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0415da40 0x7f2a041e4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0415da40 0x7f2a041e4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46124/0 (socket says 172.21.15.158:46124) 2023-10-16T09:43:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 -- 172.21.15.158:0/296457991 learned_addr learned my addr 172.21.15.158:0/296457991 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a01d9b700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cc590 0x7f2a041e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a0259c700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a0406ae10 0x7f2a04149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 -- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cc590 msgr2=0x7f2a041e79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cc590 0x7f2a041e79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 -- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a0406ae10 msgr2=0x7f2a04149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a0406ae10 0x7f2a04149790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 -- 172.21.15.158:0/296457991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc009b30 con 0x7f2a0415da40 2023-10-16T09:43:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.561+0000 7f2a02d9d700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0415da40 0x7f2a041e4f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2a040702f0 tx=0x7f29fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.562+0000 7f29f2ffd700 1 -- 172.21.15.158:0/296457991 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0099b0 con 0x7f2a0415da40 2023-10-16T09:43:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.562+0000 7f29f2ffd700 1 -- 172.21.15.158:0/296457991 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc00ce80 con 0x7f2a0415da40 2023-10-16T09:43:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.562+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0415b5c0 con 0x7f2a0415da40 2023-10-16T09:43:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.562+0000 7f29f2ffd700 1 -- 172.21.15.158:0/296457991 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc004490 con 0x7f2a0415da40 2023-10-16T09:43:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.562+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0415bb80 con 0x7f2a0415da40 2023-10-16T09:43:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.563+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a040681b0 con 0x7f2a0415da40 2023-10-16T09:43:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.563+0000 7f29f2ffd700 1 -- 172.21.15.158:0/296457991 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f29fc0045f0 con 0x7f2a0415da40 2023-10-16T09:43:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.564+0000 7f29f2ffd700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f29e406af00 0x7f29e406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.564+0000 7f29f2ffd700 1 -- 172.21.15.158:0/296457991 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f29fc08af00 con 0x7f2a0415da40 2023-10-16T09:43:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.564+0000 7f2a0259c700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f29e406af00 0x7f29e406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.564+0000 7f2a0259c700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f29e406af00 0x7f29e406d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009b40 tx=0x7f29f4009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.568+0000 7f29f2ffd700 1 -- 172.21.15.158:0/296457991 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29fc055df0 con 0x7f2a0415da40 2023-10-16T09:43:32.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.762+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2a0406f760 con 0x7f2a0415da40 2023-10-16T09:43:32.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.762+0000 7f29f2ffd700 1 -- 172.21.15.158:0/296457991 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29fc0200a0 con 0x7f2a0415da40 2023-10-16T09:43:32.764 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:32.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.764+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f29e406af00 msgr2=0x7f29e406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.764+0000 7f2a08c14700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f29e406af00 0x7f29e406d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009b40 tx=0x7f29f4009210 comp rx=0 tx=0).stop 2023-10-16T09:43:32.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0415da40 msgr2=0x7f2a041e4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0415da40 0x7f2a041e4f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2a040702f0 tx=0x7f29fc004ad0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 shutdown_connections 2023-10-16T09:43:32.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a0415da40 0x7f2a041e4f10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a0406ae10 0x7f2a04149790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f29e406af00 0x7f29e406d3c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 --2- 172.21.15.158:0/296457991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cc590 0x7f2a041e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 >> 172.21.15.158:0/296457991 conn(0x7f2a04099a10 msgr2=0x7f2a0415cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:32.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 shutdown_connections 2023-10-16T09:43:32.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.765+0000 7f2a08c14700 1 -- 172.21.15.158:0/296457991 wait complete. 2023-10-16T09:43:32.778 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:43:32.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.899+0000 7f23e938b700 1 Processor -- start 2023-10-16T09:43:32.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.900+0000 7f23e938b700 1 -- start start 2023-10-16T09:43:32.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e938b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f23e4159280 0x7f23e406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e938b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e4159660 con 0x7f23e40d0e10 2023-10-16T09:43:32.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e938b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e406b120 con 0x7f23e40cbef0 2023-10-16T09:43:32.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e938b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e40cb710 con 0x7f23e4159280 2023-10-16T09:43:32.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e27fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f23e40d0e10 0x7f23e415c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46306/0 (socket says 172.21.15.158:46306) 2023-10-16T09:43:32.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e27fc700 1 -- 172.21.15.158:0/4084837929 learned_addr learned my addr 172.21.15.158:0/4084837929 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:32.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e2ffd700 1 --2- 172.21.15.158:0/4084837929 >> v2:172.21.15.172:3300/0 conn(0x7f23e4159280 0x7f23e406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012073979 0 0) 0x7f23e4159660 con 0x7f23e40d0e10 2023-10-16T09:43:32.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc0035b0 con 0x7f23e40d0e10 2023-10-16T09:43:32.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3463123588 0 0) 0x7f23cc0035b0 con 0x7f23e40d0e10 2023-10-16T09:43:32.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 >> v2:172.21.15.172:3300/0 conn(0x7f23e4159280 msgr2=0x7f23e406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:32.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 --2- 172.21.15.158:0/4084837929 >> v2:172.21.15.172:3300/0 conn(0x7f23e4159280 0x7f23e406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 >> v1:172.21.15.172:6789/0 conn(0x7f23e40cbef0 legacy=0x7f23e40cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e406b400 con 0x7f23e40d0e10 2023-10-16T09:43:32.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f23d4001670 con 0x7f23e40d0e10 2023-10-16T09:43:32.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 >> v1:172.21.15.158:6789/0 conn(0x7f23e40d0e10 legacy=0x7f23e415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc003a00 0x7f23cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23cc004850 0x7f23cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23cc005210 0x7f23cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc00c080 con 0x7f23cc005210 2023-10-16T09:43:32.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc00c1f0 con 0x7f23cc004850 2023-10-16T09:43:32.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.901+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc00c3b0 con 0x7f23cc003a00 2023-10-16T09:43:32.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23cc005210 0x7f23cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e2ffd700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc003a00 0x7f23cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e27fc700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23cc004850 0x7f23cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 -- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc003a00 msgr2=0x7f23cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:32.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc003a00 0x7f23cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 -- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23cc004850 msgr2=0x7f23cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:32.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23cc004850 0x7f23cc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 -- 172.21.15.158:0/4084837929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0035b0 con 0x7f23cc005210 2023-10-16T09:43:32.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 -- 172.21.15.158:0/4084837929 reap_dead start 2023-10-16T09:43:32.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e37fe700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23cc005210 0x7f23cc00ba10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c190 tx=0x7f23cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d35216a8179504 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc001db0 con 0x7f23cc005210 2023-10-16T09:43:32.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23dc00ccf0 con 0x7f23cc005210 2023-10-16T09:43:32.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.902+0000 7f23e1ffb700 1 -- 172.21.15.158:0/4084837929 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc003c40 con 0x7f23cc005210 2023-10-16T09:43:32.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.903+0000 7f23e938b700 1 -- 172.21.15.158:0/4084837929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23cc005210 msgr2=0x7f23cc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.903+0000 7f23e938b700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23cc005210 0x7f23cc00ba10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c190 tx=0x7f23cc005880 comp rx=0 tx=0).stop 2023-10-16T09:43:32.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.903+0000 7f23e938b700 1 -- 172.21.15.158:0/4084837929 shutdown_connections 2023-10-16T09:43:32.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.903+0000 7f23e938b700 1 --2- 172.21.15.158:0/4084837929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23cc005210 0x7f23cc00ba10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.903+0000 7f23e938b700 1 -- 172.21.15.158:0/4084837929 >> 172.21.15.158:0/4084837929 conn(0x7f23e4099a10 msgr2=0x7f23e40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:32.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.903+0000 7f23e938b700 1 -- 172.21.15.158:0/4084837929 shutdown_connections 2023-10-16T09:43:32.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.903+0000 7f23e938b700 1 -- 172.21.15.158:0/4084837929 wait complete. 2023-10-16T09:43:32.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.904+0000 7f23e938b700 1 Processor -- start 2023-10-16T09:43:32.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.904+0000 7f23e938b700 1 -- start start 2023-10-16T09:43:32.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e938b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc005210 0x7f23e40c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e938b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23e40afdc0 0x7f23e4152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e938b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23e40b5560 0x7f23e40b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e938b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e40b5f50 con 0x7f23e40afdc0 2023-10-16T09:43:32.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e938b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e40c4270 con 0x7f23e40b5560 2023-10-16T09:43:32.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e938b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e40c4550 con 0x7f23cc005210 2023-10-16T09:43:32.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23e40afdc0 0x7f23e4152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23e40afdc0 0x7f23e4152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46140/0 (socket says 172.21.15.158:46140) 2023-10-16T09:43:32.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 -- 172.21.15.158:0/4032082598 learned_addr learned my addr 172.21.15.158:0/4032082598 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:32.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e2ffd700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc005210 0x7f23e40c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e37fe700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23e40b5560 0x7f23e40b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 -- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc005210 msgr2=0x7f23e40c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc005210 0x7f23e40c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 -- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23e40b5560 msgr2=0x7f23e40b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:32.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23e40b5560 0x7f23e40b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:32.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 -- 172.21.15.158:0/4032082598 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e40c4830 con 0x7f23e40afdc0 2023-10-16T09:43:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.905+0000 7f23e27fc700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23e40afdc0 0x7f23e4152ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f23d40072c0 tx=0x7f23d40075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.906+0000 7f23cb7fe700 1 -- 172.21.15.158:0/4032082598 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d400f6c0 con 0x7f23e40afdc0 2023-10-16T09:43:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.906+0000 7f23cb7fe700 1 -- 172.21.15.158:0/4032082598 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d400f840 con 0x7f23e40afdc0 2023-10-16T09:43:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.906+0000 7f23cb7fe700 1 -- 172.21.15.158:0/4032082598 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d400fbf0 con 0x7f23e40afdc0 2023-10-16T09:43:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.906+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e40c4b10 con 0x7f23e40afdc0 2023-10-16T09:43:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.906+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e4152ff0 con 0x7f23e40afdc0 2023-10-16T09:43:32.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.907+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e40c5030 con 0x7f23e40afdc0 2023-10-16T09:43:32.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.907+0000 7f23cb7fe700 1 -- 172.21.15.158:0/4032082598 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f23d400fd50 con 0x7f23e40afdc0 2023-10-16T09:43:32.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.907+0000 7f23cb7fe700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23d006af00 0x7f23d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:32.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.908+0000 7f23cb7fe700 1 -- 172.21.15.158:0/4032082598 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f23d4089790 con 0x7f23e40afdc0 2023-10-16T09:43:32.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.908+0000 7f23e2ffd700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23d006af00 0x7f23d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:32.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.908+0000 7f23e2ffd700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23d006af00 0x7f23d006d3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009be0 tx=0x7f23d8009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:32.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:32.912+0000 7f23cb7fe700 1 -- 172.21.15.158:0/4032082598 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f23e40c5030 con 0x7f23e40afdc0 2023-10-16T09:43:33.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.104+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f23e40c5030 con 0x7f23e40afdc0 2023-10-16T09:43:33.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.104+0000 7f23cb7fe700 1 -- 172.21.15.158:0/4032082598 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f23e40c5030 con 0x7f23e40afdc0 2023-10-16T09:43:33.106 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:43:33.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23d006af00 msgr2=0x7f23d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23d006af00 0x7f23d006d3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009be0 tx=0x7f23d8009210 comp rx=0 tx=0).stop 2023-10-16T09:43:33.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23e40afdc0 msgr2=0x7f23e4152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23e40afdc0 0x7f23e4152ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f23d40072c0 tx=0x7f23d40075d0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 shutdown_connections 2023-10-16T09:43:33.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23e40b5560 0x7f23e40b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23d006af00 0x7f23d006d3c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23cc005210 0x7f23e40c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.106+0000 7f23e938b700 1 --2- 172.21.15.158:0/4032082598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23e40afdc0 0x7f23e4152ab0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.107+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 >> 172.21.15.158:0/4032082598 conn(0x7f23e4099a10 msgr2=0x7f23e406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:33.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.107+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 shutdown_connections 2023-10-16T09:43:33.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.107+0000 7f23e938b700 1 -- 172.21.15.158:0/4032082598 wait complete. 2023-10-16T09:43:33.120 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:43:33.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.242+0000 7efefc1d2700 1 Processor -- start 2023-10-16T09:43:33.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.243+0000 7efefc1d2700 1 -- start start 2023-10-16T09:43:33.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efefc1d2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efef40d0e10 0x7efef415c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efefc1d2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef4159660 con 0x7efef4159280 2023-10-16T09:43:33.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efefc1d2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef406b120 con 0x7efef40cbef0 2023-10-16T09:43:33.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efefc1d2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef40cb710 con 0x7efef40d0e10 2023-10-16T09:43:33.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef976d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efef40d0e10 0x7efef415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef9f6e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efef4159280 0x7efef406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46324/0 (socket says 172.21.15.158:46324) 2023-10-16T09:43:33.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef976d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efef40d0e10 0x7efef415c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46148/0 (socket says 172.21.15.158:46148) 2023-10-16T09:43:33.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef9f6e700 1 -- 172.21.15.158:0/1597354289 learned_addr learned my addr 172.21.15.158:0/1597354289 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:33.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef976d700 1 -- 172.21.15.158:0/1597354289 >> v1:172.21.15.172:6789/0 conn(0x7efef40cbef0 legacy=0x7efef40cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef976d700 1 -- 172.21.15.158:0/1597354289 >> v1:172.21.15.158:6789/0 conn(0x7efef4159280 legacy=0x7efef406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef976d700 1 -- 172.21.15.158:0/1597354289 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef406b400 con 0x7efef40d0e10 2023-10-16T09:43:33.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef8f6c700 1 -- 172.21.15.158:0/1597354289 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396382908 0 0) 0x7efef4159660 con 0x7efef4159280 2023-10-16T09:43:33.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef976d700 1 --2- 172.21.15.158:0/1597354289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef415c590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efef0009840 tx=0x7efef0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4555377ea63a0eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:33.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef8f6c700 1 -- 172.21.15.158:0/1597354289 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0011830 con 0x7efef40d0e10 2023-10-16T09:43:33.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef8f6c700 1 -- 172.21.15.158:0/1597354289 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef0011990 con 0x7efef40d0e10 2023-10-16T09:43:33.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.244+0000 7efef8f6c700 1 -- 172.21.15.158:0/1597354289 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0011d20 con 0x7efef40d0e10 2023-10-16T09:43:33.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.245+0000 7efefc1d2700 1 -- 172.21.15.158:0/1597354289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 msgr2=0x7efef415c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.245+0000 7efefc1d2700 1 --2- 172.21.15.158:0/1597354289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef415c590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efef0009840 tx=0x7efef0009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:33.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.245+0000 7efefc1d2700 1 -- 172.21.15.158:0/1597354289 shutdown_connections 2023-10-16T09:43:33.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.245+0000 7efefc1d2700 1 --2- 172.21.15.158:0/1597354289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef415c590 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.245+0000 7efefc1d2700 1 -- 172.21.15.158:0/1597354289 >> 172.21.15.158:0/1597354289 conn(0x7efef4099a10 msgr2=0x7efef40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:33.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.245+0000 7efefc1d2700 1 -- 172.21.15.158:0/1597354289 shutdown_connections 2023-10-16T09:43:33.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.245+0000 7efefc1d2700 1 -- 172.21.15.158:0/1597354289 wait complete. 2023-10-16T09:43:33.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.246+0000 7efefc1d2700 1 Processor -- start 2023-10-16T09:43:33.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.246+0000 7efefc1d2700 1 -- start start 2023-10-16T09:43:33.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efefc1d2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efef40cbef0 0x7efef4155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efefc1d2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef40c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efefc1d2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef4159280 0x7efef40b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efefc1d2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef415e850 con 0x7efef40d0e10 2023-10-16T09:43:33.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efefc1d2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef415ece0 con 0x7efef4159280 2023-10-16T09:43:33.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efefc1d2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef4155d90 con 0x7efef40cbef0 2023-10-16T09:43:33.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef40c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef40c4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46156/0 (socket says 172.21.15.158:46156) 2023-10-16T09:43:33.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 -- 172.21.15.158:0/3625494805 learned_addr learned my addr 172.21.15.158:0/3625494805 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:33.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efefa76f700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef4159280 0x7efef40b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef9f6e700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efef40cbef0 0x7efef4155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 -- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efef40cbef0 msgr2=0x7efef4155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efef40cbef0 0x7efef4155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 -- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef4159280 msgr2=0x7efef40b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef4159280 0x7efef40b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef976d700 1 -- 172.21.15.158:0/3625494805 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef00094a0 con 0x7efef40d0e10 2023-10-16T09:43:33.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.247+0000 7efef9f6e700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efef40cbef0 0x7efef4155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:33.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.248+0000 7efef976d700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef40c4dd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efef0014040 tx=0x7efef0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:33.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.248+0000 7efeea7fc700 1 -- 172.21.15.158:0/3625494805 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef001a850 con 0x7efef40d0e10 2023-10-16T09:43:33.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.248+0000 7efeea7fc700 1 -- 172.21.15.158:0/3625494805 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef001a9b0 con 0x7efef40d0e10 2023-10-16T09:43:33.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.248+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef415dc40 con 0x7efef40d0e10 2023-10-16T09:43:33.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.248+0000 7efeea7fc700 1 -- 172.21.15.158:0/3625494805 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef001ad20 con 0x7efef40d0e10 2023-10-16T09:43:33.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.248+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef415e200 con 0x7efef40d0e10 2023-10-16T09:43:33.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.249+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef40681b0 con 0x7efef40d0e10 2023-10-16T09:43:33.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.249+0000 7efeea7fc700 1 -- 172.21.15.158:0/3625494805 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efef0022070 con 0x7efef40d0e10 2023-10-16T09:43:33.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.249+0000 7efeea7fc700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efed806af00 0x7efed806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.250+0000 7efeea7fc700 1 -- 172.21.15.158:0/3625494805 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efef008aad0 con 0x7efef40d0e10 2023-10-16T09:43:33.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.250+0000 7efef9f6e700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efed806af00 0x7efed806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.250+0000 7efef9f6e700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efed806af00 0x7efed806d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efee4000f60 tx=0x7efee4007e70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:33.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.254+0000 7efeea7fc700 1 -- 172.21.15.158:0/3625494805 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efef0055a40 con 0x7efef40d0e10 2023-10-16T09:43:33.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.445+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7efef415e990 con 0x7efef40d0e10 2023-10-16T09:43:33.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.446+0000 7efeea7fc700 1 -- 172.21.15.158:0/3625494805 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7efef00290a0 con 0x7efef40d0e10 2023-10-16T09:43:33.448 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:33.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efed806af00 msgr2=0x7efed806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efed806af00 0x7efed806d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efee4000f60 tx=0x7efee4007e70 comp rx=0 tx=0).stop 2023-10-16T09:43:33.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 msgr2=0x7efef40c4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef40c4dd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efef0014040 tx=0x7efef0004750 comp rx=0 tx=0).stop 2023-10-16T09:43:33.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 shutdown_connections 2023-10-16T09:43:33.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef4159280 0x7efef40b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efed806af00 0x7efed806d3c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efef40cbef0 0x7efef4155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 --2- 172.21.15.158:0/3625494805 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef40d0e10 0x7efef40c4dd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 >> 172.21.15.158:0/3625494805 conn(0x7efef4099a10 msgr2=0x7efef40b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:33.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 shutdown_connections 2023-10-16T09:43:33.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.448+0000 7efefc1d2700 1 -- 172.21.15.158:0/3625494805 wait complete. 2023-10-16T09:43:33.462 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:43:33.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.581+0000 7f1c887fd700 1 Processor -- start 2023-10-16T09:43:33.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.581+0000 7f1c887fd700 1 -- start start 2023-10-16T09:43:33.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c887fd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c8015d990 0x7f1c800b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c887fd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c8006e480 0x7f1c800b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c887fd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800b3760 0x7f1c800cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c887fd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c800b2c90 con 0x7f1c8006e480 2023-10-16T09:43:33.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c887fd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c800af600 con 0x7f1c8015d990 2023-10-16T09:43:33.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c887fd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8006e9b0 con 0x7f1c800b3760 2023-10-16T09:43:33.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c85d98700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c8006e480 0x7f1c800b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c85d98700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c8006e480 0x7f1c800b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46162/0 (socket says 172.21.15.158:46162) 2023-10-16T09:43:33.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.582+0000 7f1c85d98700 1 -- 172.21.15.158:0/3443746949 learned_addr learned my addr 172.21.15.158:0/3443746949 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:33.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85d98700 1 -- 172.21.15.158:0/3443746949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800b3760 msgr2=0x7f1c800cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:33.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85d98700 1 --2- 172.21.15.158:0/3443746949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800b3760 0x7f1c800cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85d98700 1 -- 172.21.15.158:0/3443746949 >> v2:172.21.15.172:3300/0 conn(0x7f1c8015d990 msgr2=0x7f1c800b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:33.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c86599700 1 --2- 172.21.15.158:0/3443746949 >> v2:172.21.15.172:3300/0 conn(0x7f1c8015d990 0x7f1c800b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85d98700 1 --2- 172.21.15.158:0/3443746949 >> v2:172.21.15.172:3300/0 conn(0x7f1c8015d990 0x7f1c800b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85d98700 1 -- 172.21.15.158:0/3443746949 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800ce890 con 0x7f1c8006e480 2023-10-16T09:43:33.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85d98700 1 --2- 172.21.15.158:0/3443746949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c800b3220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c800b2fc0 tx=0x7f1c7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd3025f6d3652df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:33.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85597700 1 -- 172.21.15.158:0/3443746949 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00b870 con 0x7f1c8006e480 2023-10-16T09:43:33.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85597700 1 -- 172.21.15.158:0/3443746949 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c00b9d0 con 0x7f1c8006e480 2023-10-16T09:43:33.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c85597700 1 -- 172.21.15.158:0/3443746949 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00bd60 con 0x7f1c8006e480 2023-10-16T09:43:33.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c887fd700 1 -- 172.21.15.158:0/3443746949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 msgr2=0x7f1c800b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.583+0000 7f1c887fd700 1 --2- 172.21.15.158:0/3443746949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c800b3220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c800b2fc0 tx=0x7f1c7c009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.584+0000 7f1c887fd700 1 -- 172.21.15.158:0/3443746949 shutdown_connections 2023-10-16T09:43:33.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.584+0000 7f1c887fd700 1 --2- 172.21.15.158:0/3443746949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800b3760 0x7f1c800cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.584+0000 7f1c887fd700 1 --2- 172.21.15.158:0/3443746949 >> v2:172.21.15.172:3300/0 conn(0x7f1c8015d990 0x7f1c800b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.584+0000 7f1c887fd700 1 --2- 172.21.15.158:0/3443746949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c800b3220 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.584+0000 7f1c887fd700 1 -- 172.21.15.158:0/3443746949 >> 172.21.15.158:0/3443746949 conn(0x7f1c8006b9e0 msgr2=0x7f1c8006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:33.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.584+0000 7f1c887fd700 1 -- 172.21.15.158:0/3443746949 shutdown_connections 2023-10-16T09:43:33.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.584+0000 7f1c887fd700 1 -- 172.21.15.158:0/3443746949 wait complete. 2023-10-16T09:43:33.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.585+0000 7f1c887fd700 1 Processor -- start 2023-10-16T09:43:33.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.585+0000 7f1c887fd700 1 -- start start 2023-10-16T09:43:33.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.585+0000 7f1c887fd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c8015ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.585+0000 7f1c887fd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3760 0x7f1c8015f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.585+0000 7f1c887fd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c8015d990 0x7f1c8015f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c887fd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c801515b0 con 0x7f1c8006e480 2023-10-16T09:43:33.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c887fd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8015dd70 con 0x7f1c800b3760 2023-10-16T09:43:33.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c887fd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8015e8f0 con 0x7f1c8015d990 2023-10-16T09:43:33.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c8015ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c8015ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46168/0 (socket says 172.21.15.158:46168) 2023-10-16T09:43:33.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 -- 172.21.15.158:0/1434670270 learned_addr learned my addr 172.21.15.158:0/1434670270 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:33.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c85d98700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3760 0x7f1c8015f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 -- 172.21.15.158:0/1434670270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c8015d990 msgr2=0x7f1c8015f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c8015d990 0x7f1c8015f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 -- 172.21.15.158:0/1434670270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3760 msgr2=0x7f1c8015f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3760 0x7f1c8015f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 -- 172.21.15.158:0/1434670270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c009550 con 0x7f1c8006e480 2023-10-16T09:43:33.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c86599700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c8015ed80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000c7f0 tx=0x7f1c7000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:33.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c76ffd700 1 -- 172.21.15.158:0/1434670270 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70011800 con 0x7f1c8006e480 2023-10-16T09:43:33.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c76ffd700 1 -- 172.21.15.158:0/1434670270 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c70011960 con 0x7f1c8006e480 2023-10-16T09:43:33.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8015dfc0 con 0x7f1c8006e480 2023-10-16T09:43:33.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c76ffd700 1 -- 172.21.15.158:0/1434670270 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70011cf0 con 0x7f1c8006e480 2023-10-16T09:43:33.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.586+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8015e580 con 0x7f1c8006e480 2023-10-16T09:43:33.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.587+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c800681b0 con 0x7f1c8006e480 2023-10-16T09:43:33.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.588+0000 7f1c76ffd700 1 -- 172.21.15.158:0/1434670270 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c70018360 con 0x7f1c8006e480 2023-10-16T09:43:33.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.588+0000 7f1c76ffd700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1c6406af00 0x7f1c6406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.588+0000 7f1c76ffd700 1 -- 172.21.15.158:0/1434670270 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1c700897e0 con 0x7f1c8006e480 2023-10-16T09:43:33.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.588+0000 7f1c85d98700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1c6406af00 0x7f1c6406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:33.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.588+0000 7f1c85d98700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1c6406af00 0x7f1c6406d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1c800b2fc0 tx=0x7f1c7c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:33.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.593+0000 7f1c76ffd700 1 -- 172.21.15.158:0/1434670270 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c700545a0 con 0x7f1c8006e480 2023-10-16T09:43:33.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.799+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1c80073c00 con 0x7f1c8006e480 2023-10-16T09:43:33.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.799+0000 7f1c76ffd700 1 -- 172.21.15.158:0/1434670270 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1c70057bc0 con 0x7f1c8006e480 2023-10-16T09:43:33.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1c6406af00 msgr2=0x7f1c6406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1c6406af00 0x7f1c6406d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1c800b2fc0 tx=0x7f1c7c008030 comp rx=0 tx=0).stop 2023-10-16T09:43:33.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 msgr2=0x7f1c8015ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:33.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c8015ed80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000c7f0 tx=0x7f1c7000cbb0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 shutdown_connections 2023-10-16T09:43:33.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1c6406af00 0x7f1c6406d3c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c8015d990 0x7f1c8015f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c8006e480 0x7f1c8015ed80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 --2- 172.21.15.158:0/1434670270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3760 0x7f1c8015f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:33.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 >> 172.21.15.158:0/1434670270 conn(0x7f1c8006b9e0 msgr2=0x7f1c8015cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:33.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 shutdown_connections 2023-10-16T09:43:33.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.801+0000 7f1c887fd700 1 -- 172.21.15.158:0/1434670270 wait complete. 2023-10-16T09:43:33.806 INFO:teuthology.orchestra.run.smithi158.stderr:listed 0 entries 2023-10-16T09:43:33.815 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:43:33.815 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:43:33.833 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:33.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.995+0000 7f1374a62700 1 Processor -- start 2023-10-16T09:43:33.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.995+0000 7f1374a62700 1 -- start start 2023-10-16T09:43:33.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f1374a62700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f13700729b0 0x7f137006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:33.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f1374a62700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1370161270 con 0x7f137006ab40 2023-10-16T09:43:33.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f1374a62700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1370161510 con 0x7f137006e5d0 2023-10-16T09:43:33.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f1374a62700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13701617f0 con 0x7f13700729b0 2023-10-16T09:43:33.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f13700729b0 0x7f137006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136e59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f137006ab40 0x7f13700722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46362/0 (socket says 172.21.15.158:46362) 2023-10-16T09:43:34.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f13700729b0 0x7f137006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46190/0 (socket says 172.21.15.158:46190) 2023-10-16T09:43:34.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136e59c700 1 -- 172.21.15.158:0/1238430516 learned_addr learned my addr 172.21.15.158:0/1238430516 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:34.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136ed9d700 1 -- 172.21.15.158:0/1238430516 >> v1:172.21.15.172:6789/0 conn(0x7f137006e5d0 legacy=0x7f137015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136ed9d700 1 -- 172.21.15.158:0/1238430516 >> v1:172.21.15.158:6789/0 conn(0x7f137006ab40 legacy=0x7f13700722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136ed9d700 1 -- 172.21.15.158:0/1238430516 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1370161ad0 con 0x7f13700729b0 2023-10-16T09:43:34.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136d59a700 1 -- 172.21.15.158:0/1238430516 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009082548 0 0) 0x7f1370161270 con 0x7f137006ab40 2023-10-16T09:43:34.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136ed9d700 1 --2- 172.21.15.158:0/1238430516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f137006d820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1368009d90 tx=0x7f136800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689a22941c416e52 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136d59a700 1 -- 172.21.15.158:0/1238430516 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136800e070 con 0x7f13700729b0 2023-10-16T09:43:34.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136d59a700 1 -- 172.21.15.158:0/1238430516 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136800bbd0 con 0x7f13700729b0 2023-10-16T09:43:34.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.996+0000 7f136d59a700 1 -- 172.21.15.158:0/1238430516 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368012430 con 0x7f13700729b0 2023-10-16T09:43:34.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.997+0000 7f1374a62700 1 -- 172.21.15.158:0/1238430516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 msgr2=0x7f137006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.997+0000 7f1374a62700 1 --2- 172.21.15.158:0/1238430516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f137006d820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1368009d90 tx=0x7f136800b330 comp rx=0 tx=0).stop 2023-10-16T09:43:34.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.997+0000 7f1374a62700 1 -- 172.21.15.158:0/1238430516 shutdown_connections 2023-10-16T09:43:34.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.997+0000 7f1374a62700 1 --2- 172.21.15.158:0/1238430516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f137006d820 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.997+0000 7f1374a62700 1 -- 172.21.15.158:0/1238430516 >> 172.21.15.158:0/1238430516 conn(0x7f13700d02b0 msgr2=0x7f13700d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:34.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.997+0000 7f1374a62700 1 -- 172.21.15.158:0/1238430516 shutdown_connections 2023-10-16T09:43:34.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.997+0000 7f1374a62700 1 -- 172.21.15.158:0/1238430516 wait complete. 2023-10-16T09:43:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.998+0000 7f1374a62700 1 Processor -- start 2023-10-16T09:43:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.998+0000 7f1374a62700 1 -- start start 2023-10-16T09:43:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f1374a62700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f137006ab40 0x7f13700aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f1374a62700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f137006e5d0 0x7f13700b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f1374a62700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f13700c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f1374a62700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700c5920 con 0x7f13700729b0 2023-10-16T09:43:34.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f1374a62700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700c5c00 con 0x7f137006ab40 2023-10-16T09:43:34.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f1374a62700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13700b2f40 con 0x7f137006e5d0 2023-10-16T09:43:34.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f13700c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f13700c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46194/0 (socket says 172.21.15.158:46194) 2023-10-16T09:43:34.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 -- 172.21.15.158:0/3177989630 learned_addr learned my addr 172.21.15.158:0/3177989630 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:34.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136e59c700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f137006ab40 0x7f13700aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136dd9b700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f137006e5d0 0x7f13700b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 -- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f137006e5d0 msgr2=0x7f13700b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f137006e5d0 0x7f13700b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 -- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f137006ab40 msgr2=0x7f13700aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f137006ab40 0x7f13700aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:33.999+0000 7f136ed9d700 1 -- 172.21.15.158:0/3177989630 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1368009a70 con 0x7f13700729b0 2023-10-16T09:43:34.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.000+0000 7f136ed9d700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f13700c4cd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1368014040 tx=0x7f1368004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.000+0000 7f135effd700 1 -- 172.21.15.158:0/3177989630 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136801b810 con 0x7f13700729b0 2023-10-16T09:43:34.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.000+0000 7f135effd700 1 -- 172.21.15.158:0/3177989630 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1368020070 con 0x7f13700729b0 2023-10-16T09:43:34.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.000+0000 7f135effd700 1 -- 172.21.15.158:0/3177989630 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136801bbb0 con 0x7f13700729b0 2023-10-16T09:43:34.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.000+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13700b3220 con 0x7f13700729b0 2023-10-16T09:43:34.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.000+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13700af230 con 0x7f13700729b0 2023-10-16T09:43:34.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.001+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13700af3c0 con 0x7f13700729b0 2023-10-16T09:43:34.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.001+0000 7f135effd700 1 -- 172.21.15.158:0/3177989630 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f136801bd10 con 0x7f13700729b0 2023-10-16T09:43:34.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.002+0000 7f135effd700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f135006af50 0x7f135006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.002+0000 7f136e59c700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f135006af50 0x7f135006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.002+0000 7f135effd700 1 -- 172.21.15.158:0/3177989630 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1368022070 con 0x7f13700729b0 2023-10-16T09:43:34.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.002+0000 7f136e59c700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f135006af50 0x7f135006d410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1360007b10 tx=0x7f1360007a60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.006+0000 7f135effd700 1 -- 172.21.15.158:0/3177989630 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1368056ed0 con 0x7f13700729b0 2023-10-16T09:43:34.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.200+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13700681b0 con 0x7f13700729b0 2023-10-16T09:43:34.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.201+0000 7f135effd700 1 -- 172.21.15.158:0/3177989630 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f136805a4f0 con 0x7f13700729b0 2023-10-16T09:43:34.203 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:34.203 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:34.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f135006af50 msgr2=0x7f135006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f135006af50 0x7f135006d410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1360007b10 tx=0x7f1360007a60 comp rx=0 tx=0).stop 2023-10-16T09:43:34.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 msgr2=0x7f13700c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f13700c4cd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1368014040 tx=0x7f1368004ad0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 shutdown_connections 2023-10-16T09:43:34.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13700729b0 0x7f13700c4cd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f137006ab40 0x7f13700aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f135006af50 0x7f135006d410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 --2- 172.21.15.158:0/3177989630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f137006e5d0 0x7f13700b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 >> 172.21.15.158:0/3177989630 conn(0x7f13700d02b0 msgr2=0x7f137015d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:34.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 shutdown_connections 2023-10-16T09:43:34.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.203+0000 7f1374a62700 1 -- 172.21.15.158:0/3177989630 wait complete. 2023-10-16T09:43:34.216 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:43:34.217 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:43:34.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.339+0000 7f9763b32700 1 Processor -- start 2023-10-16T09:43:34.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.339+0000 7f9763b32700 1 -- start start 2023-10-16T09:43:34.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f9763b32700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f975c0ce510 0x7f975c15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f9763b32700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f975c0cbaf0 0x7f975c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f9763b32700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0cf4c0 con 0x7f975c15b590 2023-10-16T09:43:34.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f9763b32700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975c0cc4c0 con 0x7f975c0ce510 2023-10-16T09:43:34.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f9763b32700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975c0b49e0 con 0x7f975c0cbaf0 2023-10-16T09:43:34.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97610cd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f975c0ce510 0x7f975c15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97618ce700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f975c15b590 0x7f975c0b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46376/0 (socket says 172.21.15.158:46376) 2023-10-16T09:43:34.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97618ce700 1 -- 172.21.15.158:0/748451542 learned_addr learned my addr 172.21.15.158:0/748451542 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:34.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97620cf700 1 --2- 172.21.15.158:0/748451542 >> v2:172.21.15.172:3300/0 conn(0x7f975c0cbaf0 0x7f975c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97610cd700 1 -- 172.21.15.158:0/748451542 >> v2:172.21.15.172:3300/0 conn(0x7f975c0cbaf0 msgr2=0x7f975c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:34.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97610cd700 1 --2- 172.21.15.158:0/748451542 >> v2:172.21.15.172:3300/0 conn(0x7f975c0cbaf0 0x7f975c0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97610cd700 1 -- 172.21.15.158:0/748451542 >> v1:172.21.15.158:6789/0 conn(0x7f975c15b590 legacy=0x7f975c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97610cd700 1 -- 172.21.15.158:0/748451542 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c15f710 con 0x7f975c0ce510 2023-10-16T09:43:34.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.340+0000 7f97608cc700 1 -- 172.21.15.158:0/748451542 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463284471 0 0) 0x7f975c0cf4c0 con 0x7f975c15b590 2023-10-16T09:43:34.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f97610cd700 1 --2- 172.21.15.158:0/748451542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c0ce510 0x7f975c15f1d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f975c0cf430 tx=0x7f9758009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec36d64257be403 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f97608cc700 1 -- 172.21.15.158:0/748451542 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758011830 con 0x7f975c0ce510 2023-10-16T09:43:34.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f97608cc700 1 -- 172.21.15.158:0/748451542 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9758011990 con 0x7f975c0ce510 2023-10-16T09:43:34.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f97608cc700 1 -- 172.21.15.158:0/748451542 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758011d20 con 0x7f975c0ce510 2023-10-16T09:43:34.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 -- 172.21.15.158:0/748451542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c0ce510 msgr2=0x7f975c15f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 --2- 172.21.15.158:0/748451542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c0ce510 0x7f975c15f1d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f975c0cf430 tx=0x7f9758009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:34.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 -- 172.21.15.158:0/748451542 shutdown_connections 2023-10-16T09:43:34.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 --2- 172.21.15.158:0/748451542 >> v2:172.21.15.172:3300/0 conn(0x7f975c0cbaf0 0x7f975c0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 --2- 172.21.15.158:0/748451542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c0ce510 0x7f975c15f1d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 -- 172.21.15.158:0/748451542 >> 172.21.15.158:0/748451542 conn(0x7f975c06a8d0 msgr2=0x7f975c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:34.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 -- 172.21.15.158:0/748451542 shutdown_connections 2023-10-16T09:43:34.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.341+0000 7f9763b32700 1 -- 172.21.15.158:0/748451542 wait complete. 2023-10-16T09:43:34.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 Processor -- start 2023-10-16T09:43:34.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 -- start start 2023-10-16T09:43:34.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f975c0cbaf0 0x7f975c15db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f975c0ce510 0x7f975c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c15b590 0x7f975c0c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c0c9cd0 con 0x7f975c15b590 2023-10-16T09:43:34.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c1f4420 con 0x7f975c0cbaf0 2023-10-16T09:43:34.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f9763b32700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c0c9430 con 0x7f975c0ce510 2023-10-16T09:43:34.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f97620cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c15b590 0x7f975c0c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f97620cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c15b590 0x7f975c0c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46210/0 (socket says 172.21.15.158:46210) 2023-10-16T09:43:34.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.343+0000 7f97620cf700 1 -- 172.21.15.158:0/539108999 learned_addr learned my addr 172.21.15.158:0/539108999 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:34.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97618ce700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f975c0cbaf0 0x7f975c15db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97610cd700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f975c0ce510 0x7f975c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97620cf700 1 -- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f975c0ce510 msgr2=0x7f975c15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97620cf700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f975c0ce510 0x7f975c15e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97620cf700 1 -- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f975c0cbaf0 msgr2=0x7f975c15db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97620cf700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f975c0cbaf0 0x7f975c15db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97620cf700 1 -- 172.21.15.158:0/539108999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97580094a0 con 0x7f975c15b590 2023-10-16T09:43:34.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97620cf700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c15b590 0x7f975c0c86c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9754007e40 tx=0x7f975400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97527fc700 1 -- 172.21.15.158:0/539108999 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975400ed30 con 0x7f975c15b590 2023-10-16T09:43:34.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97527fc700 1 -- 172.21.15.158:0/539108999 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9754014420 con 0x7f975c15b590 2023-10-16T09:43:34.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975c0c9710 con 0x7f975c15b590 2023-10-16T09:43:34.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f97527fc700 1 -- 172.21.15.158:0/539108999 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97540147d0 con 0x7f975c15b590 2023-10-16T09:43:34.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.344+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975c159de0 con 0x7f975c15b590 2023-10-16T09:43:34.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.345+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975c1f4280 con 0x7f975c15b590 2023-10-16T09:43:34.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.345+0000 7f97527fc700 1 -- 172.21.15.158:0/539108999 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9754014930 con 0x7f975c15b590 2023-10-16T09:43:34.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.346+0000 7f97527fc700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f974006af00 0x7f974006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.346+0000 7f97527fc700 1 -- 172.21.15.158:0/539108999 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9754089ee0 con 0x7f975c15b590 2023-10-16T09:43:34.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.346+0000 7f97618ce700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f974006af00 0x7f974006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.346+0000 7f97618ce700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f974006af00 0x7f974006d3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f974c000f60 tx=0x7f974c007e70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.350+0000 7f97527fc700 1 -- 172.21.15.158:0/539108999 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f975c1f4280 con 0x7f975c15b590 2023-10-16T09:43:34.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.540+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f975c1f4280 con 0x7f975c15b590 2023-10-16T09:43:34.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.541+0000 7f97527fc700 1 -- 172.21.15.158:0/539108999 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f975c1f4280 con 0x7f975c15b590 2023-10-16T09:43:34.543 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:43:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f974006af00 msgr2=0x7f974006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f974006af00 0x7f974006d3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f974c000f60 tx=0x7f974c007e70 comp rx=0 tx=0).stop 2023-10-16T09:43:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c15b590 msgr2=0x7f975c0c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c15b590 0x7f975c0c86c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9754007e40 tx=0x7f975400e3c0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 shutdown_connections 2023-10-16T09:43:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f975c15b590 0x7f975c0c86c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f975c0cbaf0 0x7f975c15db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f974006af00 0x7f974006d3c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 --2- 172.21.15.158:0/539108999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f975c0ce510 0x7f975c15e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.543+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 >> 172.21.15.158:0/539108999 conn(0x7f975c06a8d0 msgr2=0x7f975c0b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.544+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 shutdown_connections 2023-10-16T09:43:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.544+0000 7f9763b32700 1 -- 172.21.15.158:0/539108999 wait complete. 2023-10-16T09:43:34.557 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:43:34.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.679+0000 7f94fede8700 1 Processor -- start 2023-10-16T09:43:34.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.679+0000 7f94fede8700 1 -- start start 2023-10-16T09:43:34.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fede8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94f815da40 0x7f94f80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fede8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 0x7f94f806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fede8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f80cbd30 con 0x7f94f806ae10 2023-10-16T09:43:34.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fede8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f806b2c0 con 0x7f94f815da40 2023-10-16T09:43:34.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fede8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80cc1c0 con 0x7f94f80cc590 2023-10-16T09:43:34.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fd385700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f94f806ae10 0x7f94f80ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46394/0 (socket says 172.21.15.158:46394) 2023-10-16T09:43:34.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fd385700 1 -- 172.21.15.158:0/809240937 learned_addr learned my addr 172.21.15.158:0/809240937 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:34.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94fcb84700 1 --2- 172.21.15.158:0/809240937 >> v2:172.21.15.172:3300/0 conn(0x7f94f815da40 0x7f94f80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94f7fff700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 0x7f94f806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401695834 0 0) 0x7f94f80cbd30 con 0x7f94f806ae10 2023-10-16T09:43:34.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.680+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e80035b0 con 0x7f94f806ae10 2023-10-16T09:43:34.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3765534675 0 0) 0x7f94e80035b0 con 0x7f94f806ae10 2023-10-16T09:43:34.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 msgr2=0x7f94f806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:34.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 0x7f94f806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 >> v2:172.21.15.172:3300/0 conn(0x7f94f815da40 msgr2=0x7f94f80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:34.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 --2- 172.21.15.158:0/809240937 >> v2:172.21.15.172:3300/0 conn(0x7f94f815da40 0x7f94f80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f815de20 con 0x7f94f806ae10 2023-10-16T09:43:34.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f94f00019f0 con 0x7f94f806ae10 2023-10-16T09:43:34.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 --2- 172.21.15.158:0/809240937 >> v2:172.21.15.172:3300/0 conn(0x7f94f815da40 0x7f94f80caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:34.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 >> v1:172.21.15.158:6789/0 conn(0x7f94f806ae10 legacy=0x7f94f80ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e8004850 0x7f94e8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 0x7f94e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f806b2c0 con 0x7f94e8003a00 2023-10-16T09:43:34.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80cc1c0 con 0x7f94e8004850 2023-10-16T09:43:34.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e800ba10 con 0x7f94f80cc590 2023-10-16T09:43:34.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94e8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f7fff700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e8004850 0x7f94e8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fd385700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 0x7f94e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 -- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 msgr2=0x7f94e8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:34.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f80cc590 0x7f94e8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 -- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e8004850 msgr2=0x7f94e8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:34.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e8004850 0x7f94e8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 -- 172.21.15.158:0/809240937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80035b0 con 0x7f94e8003a00 2023-10-16T09:43:34.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94fcb84700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94e8003e20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009d30 tx=0x7f94e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ccd73113559769 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec009e80 con 0x7f94e8003a00 2023-10-16T09:43:34.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec00e9a0 con 0x7f94e8003a00 2023-10-16T09:43:34.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.681+0000 7f94f77fe700 1 -- 172.21.15.158:0/809240937 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec00ed30 con 0x7f94e8003a00 2023-10-16T09:43:34.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.682+0000 7f94fede8700 1 -- 172.21.15.158:0/809240937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 msgr2=0x7f94e8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.682+0000 7f94fede8700 1 --2- 172.21.15.158:0/809240937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94e8003e20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009d30 tx=0x7f94e8005880 comp rx=0 tx=0).stop 2023-10-16T09:43:34.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.682+0000 7f94fd385700 1 -- 172.21.15.158:0/809240937 reap_dead start 2023-10-16T09:43:34.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.682+0000 7f94fede8700 1 -- 172.21.15.158:0/809240937 shutdown_connections 2023-10-16T09:43:34.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.682+0000 7f94fede8700 1 -- 172.21.15.158:0/809240937 >> 172.21.15.158:0/809240937 conn(0x7f94f8099a10 msgr2=0x7f94f80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:34.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.682+0000 7f94fede8700 1 -- 172.21.15.158:0/809240937 shutdown_connections 2023-10-16T09:43:34.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.682+0000 7f94fede8700 1 -- 172.21.15.158:0/809240937 wait complete. 2023-10-16T09:43:34.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.684+0000 7f94fede8700 1 Processor -- start 2023-10-16T09:43:34.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.684+0000 7f94fede8700 1 -- start start 2023-10-16T09:43:34.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94f81597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f81f2c80 0x7f94f8159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f815b6c0 0x7f94f815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f8151350 con 0x7f94e8003a00 2023-10-16T09:43:34.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f815c080 con 0x7f94f815b6c0 2023-10-16T09:43:34.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f81567c0 con 0x7f94f81f2c80 2023-10-16T09:43:34.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94f81597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94f81597c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46226/0 (socket says 172.21.15.158:46226) 2023-10-16T09:43:34.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 -- 172.21.15.158:0/2485216200 learned_addr learned my addr 172.21.15.158:0/2485216200 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:34.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94f7fff700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f81f2c80 0x7f94f8159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fd385700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f815b6c0 0x7f94f815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 -- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f81f2c80 msgr2=0x7f94f8159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f81f2c80 0x7f94f8159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 -- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f815b6c0 msgr2=0x7f94f815bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f815b6c0 0x7f94f815bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 -- 172.21.15.158:0/2485216200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f815a280 con 0x7f94e8003a00 2023-10-16T09:43:34.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94f7fff700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f81f2c80 0x7f94f8159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:34.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fd385700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f815b6c0 0x7f94f815bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:34.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fcb84700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94f81597c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f94e80046d0 tx=0x7f94ec0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94f57fa700 1 -- 172.21.15.158:0/2485216200 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec006e60 con 0x7f94e8003a00 2023-10-16T09:43:34.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94f57fa700 1 -- 172.21.15.158:0/2485216200 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec003b10 con 0x7f94e8003a00 2023-10-16T09:43:34.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f815a550 con 0x7f94e8003a00 2023-10-16T09:43:34.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94f57fa700 1 -- 172.21.15.158:0/2485216200 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec009e80 con 0x7f94e8003a00 2023-10-16T09:43:34.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.685+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f815ab10 con 0x7f94e8003a00 2023-10-16T09:43:34.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.686+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f80681b0 con 0x7f94e8003a00 2023-10-16T09:43:34.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.687+0000 7f94f57fa700 1 -- 172.21.15.158:0/2485216200 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f94ec013470 con 0x7f94e8003a00 2023-10-16T09:43:34.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.687+0000 7f94f57fa700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f94dc06af00 0x7f94dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:34.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.687+0000 7f94f7fff700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f94dc06af00 0x7f94dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:34.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.687+0000 7f94f57fa700 1 -- 172.21.15.158:0/2485216200 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f94ec08b340 con 0x7f94e8003a00 2023-10-16T09:43:34.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.688+0000 7f94f7fff700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f94dc06af00 0x7f94dc06d3c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94e4006ba0 tx=0x7f94e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:34.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.692+0000 7f94f57fa700 1 -- 172.21.15.158:0/2485216200 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94ec0562b0 con 0x7f94e8003a00 2023-10-16T09:43:34.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.884+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94f806ea70 con 0x7f94e8003a00 2023-10-16T09:43:34.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.885+0000 7f94f57fa700 1 -- 172.21.15.158:0/2485216200 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f94ec0598d0 con 0x7f94e8003a00 2023-10-16T09:43:34.887 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:34.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f94dc06af00 msgr2=0x7f94dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f94dc06af00 0x7f94dc06d3c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94e4006ba0 tx=0x7f94e4008040 comp rx=0 tx=0).stop 2023-10-16T09:43:34.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 msgr2=0x7f94f81597c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:34.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94f81597c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f94e80046d0 tx=0x7f94ec0047c0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 shutdown_connections 2023-10-16T09:43:34.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f815b6c0 0x7f94f815bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e8003a00 0x7f94f81597c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f94dc06af00 0x7f94dc06d3c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 --2- 172.21.15.158:0/2485216200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94f81f2c80 0x7f94f8159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:34.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 >> 172.21.15.158:0/2485216200 conn(0x7f94f8099a10 msgr2=0x7f94f80cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:34.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 shutdown_connections 2023-10-16T09:43:34.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:34.887+0000 7f94fede8700 1 -- 172.21.15.158:0/2485216200 wait complete. 2023-10-16T09:43:34.900 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:43:35.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.022+0000 7f63d3220700 1 Processor -- start 2023-10-16T09:43:35.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.022+0000 7f63d3220700 1 -- start start 2023-10-16T09:43:35.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.023+0000 7f63d3220700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63cc0cc590 0x7f63cc0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.023+0000 7f63d3220700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc0cbd30 con 0x7f63cc15da40 2023-10-16T09:43:35.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.023+0000 7f63d3220700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc0cc1c0 con 0x7f63cc06afe0 2023-10-16T09:43:35.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.023+0000 7f63d3220700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc15de20 con 0x7f63cc0cc590 2023-10-16T09:43:35.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.023+0000 7f63d0fbc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f63cc15da40 0x7f63cc06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46410/0 (socket says 172.21.15.158:46410) 2023-10-16T09:43:35.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.023+0000 7f63d0fbc700 1 -- 172.21.15.158:0/2342447389 learned_addr learned my addr 172.21.15.158:0/2342447389 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:35.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.023+0000 7f63d17bd700 1 --2- 172.21.15.158:0/2342447389 >> v2:172.21.15.172:3300/0 conn(0x7f63cc0cc590 0x7f63cc0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189668969 0 0) 0x7f63cc0cbd30 con 0x7f63cc15da40 2023-10-16T09:43:35.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc0035b0 con 0x7f63cc15da40 2023-10-16T09:43:35.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910976775 0 0) 0x7f63bc0035b0 con 0x7f63cc15da40 2023-10-16T09:43:35.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 >> v2:172.21.15.172:3300/0 conn(0x7f63cc0cc590 msgr2=0x7f63cc0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:35.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/2342447389 >> v2:172.21.15.172:3300/0 conn(0x7f63cc0cc590 0x7f63cc0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 >> v1:172.21.15.172:6789/0 conn(0x7f63cc06afe0 legacy=0x7f63cc0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0ce260 con 0x7f63cc15da40 2023-10-16T09:43:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f63c0001670 con 0x7f63cc15da40 2023-10-16T09:43:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 >> v1:172.21.15.158:6789/0 conn(0x7f63cc15da40 legacy=0x7f63cc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc003a00 0x7f63bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63bc004850 0x7f63bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63bc005210 0x7f63bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc15de20 con 0x7f63bc004850 2023-10-16T09:43:35.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc00c0b0 con 0x7f63bc003a00 2023-10-16T09:43:35.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc00c270 con 0x7f63bc005210 2023-10-16T09:43:35.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cbfff700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63bc004850 0x7f63bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63d0fbc700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc003a00 0x7f63bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63d17bd700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63bc005210 0x7f63bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cbfff700 1 -- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63bc005210 msgr2=0x7f63bc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:35.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cbfff700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63bc005210 0x7f63bc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cbfff700 1 -- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc003a00 msgr2=0x7f63bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:35.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cbfff700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc003a00 0x7f63bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cbfff700 1 -- 172.21.15.158:0/2342447389 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0035b0 con 0x7f63bc004850 2023-10-16T09:43:35.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63d17bd700 1 -- 172.21.15.158:0/2342447389 reap_dead start 2023-10-16T09:43:35.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.024+0000 7f63cbfff700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63bc004850 0x7f63bc004cd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63b80017c0 tx=0x7f63bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474bb74b5aafac68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80052d0 con 0x7f63bc004850 2023-10-16T09:43:35.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8005430 con 0x7f63bc004850 2023-10-16T09:43:35.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63cb7fe700 1 -- 172.21.15.158:0/2342447389 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80038c0 con 0x7f63bc004850 2023-10-16T09:43:35.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63d3220700 1 -- 172.21.15.158:0/2342447389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63bc004850 msgr2=0x7f63bc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63d3220700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63bc004850 0x7f63bc004cd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63b80017c0 tx=0x7f63bc005880 comp rx=0 tx=0).stop 2023-10-16T09:43:35.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63d3220700 1 -- 172.21.15.158:0/2342447389 shutdown_connections 2023-10-16T09:43:35.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63d3220700 1 --2- 172.21.15.158:0/2342447389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63bc004850 0x7f63bc004cd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63d3220700 1 -- 172.21.15.158:0/2342447389 >> 172.21.15.158:0/2342447389 conn(0x7f63cc099a10 msgr2=0x7f63cc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:35.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63d3220700 1 -- 172.21.15.158:0/2342447389 shutdown_connections 2023-10-16T09:43:35.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.025+0000 7f63d3220700 1 -- 172.21.15.158:0/2342447389 wait complete. 2023-10-16T09:43:35.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.026+0000 7f63d3220700 1 Processor -- start 2023-10-16T09:43:35.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.026+0000 7f63d3220700 1 -- start start 2023-10-16T09:43:35.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d3220700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc004850 0x7f63cc150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d3220700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63cc15a840 0x7f63cc15acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d3220700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc0c8c20 0x7f63cc0c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d3220700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc15b200 con 0x7f63cc0c8c20 2023-10-16T09:43:35.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d3220700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0c7540 con 0x7f63bc004850 2023-10-16T09:43:35.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d3220700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0c7820 con 0x7f63cc15a840 2023-10-16T09:43:35.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d17bd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc0c8c20 0x7f63cc0c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d17bd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc0c8c20 0x7f63cc0c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46248/0 (socket says 172.21.15.158:46248) 2023-10-16T09:43:35.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.027+0000 7f63d17bd700 1 -- 172.21.15.158:0/1623883123 learned_addr learned my addr 172.21.15.158:0/1623883123 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:35.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d0fbc700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc004850 0x7f63cc150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63cbfff700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63cc15a840 0x7f63cc15acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d17bd700 1 -- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63cc15a840 msgr2=0x7f63cc15acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d17bd700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63cc15a840 0x7f63cc15acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d17bd700 1 -- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc004850 msgr2=0x7f63cc150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d17bd700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc004850 0x7f63cc150ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d17bd700 1 -- 172.21.15.158:0/1623883123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0c7b00 con 0x7f63cc0c8c20 2023-10-16T09:43:35.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63cbfff700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63cc15a840 0x7f63cc15acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:35.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d0fbc700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc004850 0x7f63cc150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:35.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d17bd700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc0c8c20 0x7f63cc0c90a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ccb90 tx=0x7f63c40114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63c97fa700 1 -- 172.21.15.158:0/1623883123 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c400e070 con 0x7f63cc0c8c20 2023-10-16T09:43:35.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63c97fa700 1 -- 172.21.15.158:0/1623883123 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4003c30 con 0x7f63cc0c8c20 2023-10-16T09:43:35.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc0c7de0 con 0x7f63cc0c8c20 2023-10-16T09:43:35.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63c97fa700 1 -- 172.21.15.158:0/1623883123 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4003e20 con 0x7f63cc0c8c20 2023-10-16T09:43:35.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.028+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc151020 con 0x7f63cc0c8c20 2023-10-16T09:43:35.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.029+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc0c8300 con 0x7f63cc0c8c20 2023-10-16T09:43:35.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.029+0000 7f63c97fa700 1 -- 172.21.15.158:0/1623883123 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f63c4011a10 con 0x7f63cc0c8c20 2023-10-16T09:43:35.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.030+0000 7f63c97fa700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b006af00 0x7f63b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.030+0000 7f63c97fa700 1 -- 172.21.15.158:0/1623883123 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f63c4089e90 con 0x7f63cc0c8c20 2023-10-16T09:43:35.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.030+0000 7f63d0fbc700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b006af00 0x7f63b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.030+0000 7f63d0fbc700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b006af00 0x7f63b006d3c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f63c0007cf0 tx=0x7f63c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.034+0000 7f63c97fa700 1 -- 172.21.15.158:0/1623883123 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63cc0c8300 con 0x7f63cc0c8c20 2023-10-16T09:43:35.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.225+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63cc0681b0 con 0x7f63cc0c8c20 2023-10-16T09:43:35.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.225+0000 7f63c97fa700 1 -- 172.21.15.158:0/1623883123 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63c4054c50 con 0x7f63cc0c8c20 2023-10-16T09:43:35.227 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:43:35.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b006af00 msgr2=0x7f63b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b006af00 0x7f63b006d3c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f63c0007cf0 tx=0x7f63c0009040 comp rx=0 tx=0).stop 2023-10-16T09:43:35.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc0c8c20 msgr2=0x7f63cc0c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc0c8c20 0x7f63cc0c90a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ccb90 tx=0x7f63c40114b0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 shutdown_connections 2023-10-16T09:43:35.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63cc0c8c20 0x7f63cc0c90a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63bc004850 0x7f63cc150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f63b006af00 0x7f63b006d3c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 --2- 172.21.15.158:0/1623883123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63cc15a840 0x7f63cc15acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 >> 172.21.15.158:0/1623883123 conn(0x7f63cc099a10 msgr2=0x7f63cc15b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:35.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 shutdown_connections 2023-10-16T09:43:35.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.227+0000 7f63d3220700 1 -- 172.21.15.158:0/1623883123 wait complete. 2023-10-16T09:43:35.241 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:43:35.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.364+0000 7f8032ea7700 1 Processor -- start 2023-10-16T09:43:35.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.364+0000 7f8032ea7700 1 -- start start 2023-10-16T09:43:35.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8032ea7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8032ea7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c0b2f80 con 0x7f802c15d7b0 2023-10-16T09:43:35.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8032ea7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c0ad0e0 con 0x7f802c0721d0 2023-10-16T09:43:35.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8032ea7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c072630 con 0x7f802c06bff0 2023-10-16T09:43:35.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8030c43700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f802c15d7b0 0x7f802c0ac9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46430/0 (socket says 172.21.15.158:46430) 2023-10-16T09:43:35.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8030c43700 1 -- 172.21.15.158:0/581072160 learned_addr learned my addr 172.21.15.158:0/581072160 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:35.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8031444700 1 --2- 172.21.15.158:0/581072160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f802b7fe700 1 -- 172.21.15.158:0/581072160 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721785122 0 0) 0x7f802c0b2f80 con 0x7f802c15d7b0 2023-10-16T09:43:35.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f802b7fe700 1 -- 172.21.15.158:0/581072160 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c0035b0 con 0x7f802c15d7b0 2023-10-16T09:43:35.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f802b7fe700 1 -- 172.21.15.158:0/581072160 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046692491 0 0) 0x7f8018001110 con 0x7f802c0721d0 2023-10-16T09:43:35.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f802b7fe700 1 -- 172.21.15.158:0/581072160 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802c0b2f80 con 0x7f802c0721d0 2023-10-16T09:43:35.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8031444700 1 -- 172.21.15.158:0/581072160 >> v1:172.21.15.172:6789/0 conn(0x7f802c0721d0 legacy=0x7f802c06b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8031444700 1 -- 172.21.15.158:0/581072160 >> v1:172.21.15.158:6789/0 conn(0x7f802c15d7b0 legacy=0x7f802c0ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.365+0000 7f8031444700 1 -- 172.21.15.158:0/581072160 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c0b3dc0 con 0x7f802c06bff0 2023-10-16T09:43:35.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8031444700 1 --2- 172.21.15.158:0/581072160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c0b2a40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8024009dc0 tx=0x7f802400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=869f6c797e99dd41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f802b7fe700 1 -- 172.21.15.158:0/581072160 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802400e070 con 0x7f802c06bff0 2023-10-16T09:43:35.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f802b7fe700 1 -- 172.21.15.158:0/581072160 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802400bb20 con 0x7f802c06bff0 2023-10-16T09:43:35.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f802b7fe700 1 -- 172.21.15.158:0/581072160 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024012430 con 0x7f802c06bff0 2023-10-16T09:43:35.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8032ea7700 1 -- 172.21.15.158:0/581072160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 msgr2=0x7f802c0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8032ea7700 1 --2- 172.21.15.158:0/581072160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c0b2a40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8024009dc0 tx=0x7f802400b330 comp rx=0 tx=0).stop 2023-10-16T09:43:35.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8032ea7700 1 -- 172.21.15.158:0/581072160 shutdown_connections 2023-10-16T09:43:35.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8032ea7700 1 --2- 172.21.15.158:0/581072160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c0b2a40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8032ea7700 1 -- 172.21.15.158:0/581072160 >> 172.21.15.158:0/581072160 conn(0x7f802c071110 msgr2=0x7f802c071520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:35.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8032ea7700 1 -- 172.21.15.158:0/581072160 shutdown_connections 2023-10-16T09:43:35.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.366+0000 7f8032ea7700 1 -- 172.21.15.158:0/581072160 wait complete. 2023-10-16T09:43:35.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.367+0000 7f8032ea7700 1 Processor -- start 2023-10-16T09:43:35.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.367+0000 7f8032ea7700 1 -- start start 2023-10-16T09:43:35.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f8032ea7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c13e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f8032ea7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f802c0721d0 0x7f802c13ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f8032ea7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f802c15d7b0 0x7f802c1400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f8032ea7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c13f470 con 0x7f802c0721d0 2023-10-16T09:43:35.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f8032ea7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c13dd80 con 0x7f802c15d7b0 2023-10-16T09:43:35.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f8032ea7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c14ef10 con 0x7f802c06bff0 2023-10-16T09:43:35.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f802bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f802c0721d0 0x7f802c13ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f802bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f802c0721d0 0x7f802c13ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46260/0 (socket says 172.21.15.158:46260) 2023-10-16T09:43:35.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.368+0000 7f802bfff700 1 -- 172.21.15.158:0/1061604027 learned_addr learned my addr 172.21.15.158:0/1061604027 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:35.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f8031444700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f802c15d7b0 0x7f802c1400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f8030c43700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c13e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f802bfff700 1 -- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 msgr2=0x7f802c13e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f802bfff700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c13e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f802bfff700 1 -- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f802c15d7b0 msgr2=0x7f802c1400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f802bfff700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f802c15d7b0 0x7f802c1400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f802bfff700 1 -- 172.21.15.158:0/1061604027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8024009a70 con 0x7f802c0721d0 2023-10-16T09:43:35.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f802bfff700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f802c0721d0 0x7f802c13ef30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f801800c700 tx=0x7f801800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f80297fa700 1 -- 172.21.15.158:0/1061604027 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80180052d0 con 0x7f802c0721d0 2023-10-16T09:43:35.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f80297fa700 1 -- 172.21.15.158:0/1061604027 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8018005450 con 0x7f802c0721d0 2023-10-16T09:43:35.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c13f6a0 con 0x7f802c0721d0 2023-10-16T09:43:35.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f80297fa700 1 -- 172.21.15.158:0/1061604027 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80180038c0 con 0x7f802c0721d0 2023-10-16T09:43:35.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.369+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c13fc60 con 0x7f802c0721d0 2023-10-16T09:43:35.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.370+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c14f130 con 0x7f802c0721d0 2023-10-16T09:43:35.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.371+0000 7f80297fa700 1 -- 172.21.15.158:0/1061604027 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8018003a20 con 0x7f802c0721d0 2023-10-16T09:43:35.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.371+0000 7f80297fa700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.371+0000 7f80297fa700 1 -- 172.21.15.158:0/1061604027 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8018088af0 con 0x7f802c0721d0 2023-10-16T09:43:35.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.371+0000 7f8030c43700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.372+0000 7f8030c43700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8020007fd0 tx=0x7f8020007dc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.376+0000 7f80297fa700 1 -- 172.21.15.158:0/1061604027 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f802c14f130 con 0x7f802c0721d0 2023-10-16T09:43:35.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.568+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f802c0ad0e0 con 0x7f802c0721d0 2023-10-16T09:43:35.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.568+0000 7f80297fa700 1 -- 172.21.15.158:0/1061604027 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f802c0ad0e0 con 0x7f802c0721d0 2023-10-16T09:43:35.570 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:35.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 msgr2=0x7f801006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8020007fd0 tx=0x7f8020007dc0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f802c0721d0 msgr2=0x7f802c13ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f802c0721d0 0x7f802c13ef30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f801800c700 tx=0x7f801800ca10 comp rx=0 tx=0).stop 2023-10-16T09:43:35.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 shutdown_connections 2023-10-16T09:43:35.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f802c15d7b0 0x7f802c1400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f802c06bff0 0x7f802c13e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 --2- 172.21.15.158:0/1061604027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f802c0721d0 0x7f802c13ef30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 >> 172.21.15.158:0/1061604027 conn(0x7f802c071110 msgr2=0x7f802c06dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:35.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 shutdown_connections 2023-10-16T09:43:35.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.571+0000 7f8032ea7700 1 -- 172.21.15.158:0/1061604027 wait complete. 2023-10-16T09:43:35.584 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:43:35.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.705+0000 7fabb249e700 1 Processor -- start 2023-10-16T09:43:35.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.705+0000 7fabb249e700 1 -- start start 2023-10-16T09:43:35.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabb249e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fabac0cfe30 0x7fabac0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabb249e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac15ec70 con 0x7fabac15a6d0 2023-10-16T09:43:35.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabb249e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac15f1b0 con 0x7fabac0cd190 2023-10-16T09:43:35.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabb249e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabac0afcd0 con 0x7fabac0cfe30 2023-10-16T09:43:35.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fababfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fabac15a6d0 0x7fabac0ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46444/0 (socket says 172.21.15.158:46444) 2023-10-16T09:43:35.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fababfff700 1 -- 172.21.15.158:0/4263135739 learned_addr learned my addr 172.21.15.158:0/4263135739 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:35.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760640440 0 0) 0x7fabac15ec70 con 0x7fabac15a6d0 2023-10-16T09:43:35.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab940035b0 con 0x7fabac15a6d0 2023-10-16T09:43:35.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabab7fe700 1 --2- 172.21.15.158:0/4263135739 >> v2:172.21.15.172:3300/0 conn(0x7fabac0cfe30 0x7fabac0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361664866 0 0) 0x7fab940035b0 con 0x7fabac15a6d0 2023-10-16T09:43:35.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 >> v2:172.21.15.172:3300/0 conn(0x7fabac0cfe30 msgr2=0x7fabac0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 --2- 172.21.15.158:0/4263135739 >> v2:172.21.15.172:3300/0 conn(0x7fabac0cfe30 0x7fabac0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 >> v1:172.21.15.172:6789/0 conn(0x7fabac0cd190 legacy=0x7fabac06ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0cab50 con 0x7fabac15a6d0 2023-10-16T09:43:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fab9c001670 con 0x7fabac15a6d0 2023-10-16T09:43:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 >> v1:172.21.15.158:6789/0 conn(0x7fabac15a6d0 legacy=0x7fabac0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab94003a00 0x7fab94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab94004850 0x7fab94004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94005210 0x7fab9400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0afcd0 con 0x7fab94004850 2023-10-16T09:43:35.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9400c0b0 con 0x7fab94005210 2023-10-16T09:43:35.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.706+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9400c270 con 0x7fab94003a00 2023-10-16T09:43:35.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabab7fe700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab94004850 0x7fab94004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fababfff700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab94003a00 0x7fab94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabb0a3b700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94005210 0x7fab9400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabab7fe700 1 -- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab94003a00 msgr2=0x7fab94003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:35.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabab7fe700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab94003a00 0x7fab94003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabab7fe700 1 -- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94005210 msgr2=0x7fab9400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:35.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabab7fe700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94005210 0x7fab9400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabb0a3b700 1 -- 172.21.15.158:0/4263135739 reap_dead start 2023-10-16T09:43:35.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabab7fe700 1 -- 172.21.15.158:0/4263135739 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab940035b0 con 0x7fab94004850 2023-10-16T09:43:35.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabab7fe700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab94004850 0x7fab94004cd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faba00096f0 tx=0x7fab94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349ff78791e89cf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba000e6c0 con 0x7fab94004850 2023-10-16T09:43:35.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba000e820 con 0x7fab94004850 2023-10-16T09:43:35.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.707+0000 7fabaaffd700 1 -- 172.21.15.158:0/4263135739 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba000ebb0 con 0x7fab94004850 2023-10-16T09:43:35.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.708+0000 7fabb249e700 1 -- 172.21.15.158:0/4263135739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab94004850 msgr2=0x7fab94004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.708+0000 7fabb249e700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab94004850 0x7fab94004cd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faba00096f0 tx=0x7fab94005880 comp rx=0 tx=0).stop 2023-10-16T09:43:35.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.708+0000 7fabb249e700 1 -- 172.21.15.158:0/4263135739 shutdown_connections 2023-10-16T09:43:35.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.708+0000 7fabb249e700 1 --2- 172.21.15.158:0/4263135739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab94004850 0x7fab94004cd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.708+0000 7fabb249e700 1 -- 172.21.15.158:0/4263135739 >> 172.21.15.158:0/4263135739 conn(0x7fabac06b9e0 msgr2=0x7fabac06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:35.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.708+0000 7fabb249e700 1 -- 172.21.15.158:0/4263135739 shutdown_connections 2023-10-16T09:43:35.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.708+0000 7fabb249e700 1 -- 172.21.15.158:0/4263135739 wait complete. 2023-10-16T09:43:35.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.709+0000 7fabb249e700 1 Processor -- start 2023-10-16T09:43:35.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.709+0000 7fabb249e700 1 -- start start 2023-10-16T09:43:35.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb249e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94004850 0x7fabac15b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb249e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabac0b09d0 0x7fabac0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb249e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabac1ee5f0 0x7fabac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb249e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac156fc0 con 0x7fabac0b09d0 2023-10-16T09:43:35.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb249e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0b1790 con 0x7fab94004850 2023-10-16T09:43:35.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb249e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0b0e50 con 0x7fabac1ee5f0 2023-10-16T09:43:35.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabac0b09d0 0x7fabac0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabac0b09d0 0x7fabac0b1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46282/0 (socket says 172.21.15.158:46282) 2023-10-16T09:43:35.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 -- 172.21.15.158:0/3304055910 learned_addr learned my addr 172.21.15.158:0/3304055910 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:35.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fababfff700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94004850 0x7fabac15b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb0a3b700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabac1ee5f0 0x7fabac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 -- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabac1ee5f0 msgr2=0x7fabac06ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabac1ee5f0 0x7fabac06ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 -- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94004850 msgr2=0x7fabac15b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94004850 0x7fabac15b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 -- 172.21.15.158:0/3304055910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0b2260 con 0x7fabac0b09d0 2023-10-16T09:43:35.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabab7fe700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabac0b09d0 0x7fabac0b1250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faba00090b0 tx=0x7faba00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7faba8ff9700 1 -- 172.21.15.158:0/3304055910 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba000e6c0 con 0x7fabac0b09d0 2023-10-16T09:43:35.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7faba8ff9700 1 -- 172.21.15.158:0/3304055910 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba00038c0 con 0x7fabac0b09d0 2023-10-16T09:43:35.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.710+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac0b19c0 con 0x7fabac0b09d0 2023-10-16T09:43:35.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.711+0000 7faba8ff9700 1 -- 172.21.15.158:0/3304055910 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba0006e60 con 0x7fabac0b09d0 2023-10-16T09:43:35.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.711+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac0b1f80 con 0x7fabac0b09d0 2023-10-16T09:43:35.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.712+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac0681b0 con 0x7fabac0b09d0 2023-10-16T09:43:35.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.712+0000 7faba8ff9700 1 -- 172.21.15.158:0/3304055910 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faba0020070 con 0x7fabac0b09d0 2023-10-16T09:43:35.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.712+0000 7faba8ff9700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fab9806af00 0x7fab9806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:35.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.712+0000 7faba8ff9700 1 -- 172.21.15.158:0/3304055910 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faba008ce60 con 0x7fabac0b09d0 2023-10-16T09:43:35.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.712+0000 7fababfff700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fab9806af00 0x7fab9806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:35.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.713+0000 7fababfff700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fab9806af00 0x7fab9806d3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab9c000c00 tx=0x7fab9c007980 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:35.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.717+0000 7faba8ff9700 1 -- 172.21.15.158:0/3304055910 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faba005b740 con 0x7fabac0b09d0 2023-10-16T09:43:35.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.924+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fabac0720b0 con 0x7fabac0b09d0 2023-10-16T09:43:35.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.924+0000 7faba8ff9700 1 -- 172.21.15.158:0/3304055910 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7faba001e080 con 0x7fabac0b09d0 2023-10-16T09:43:35.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fab9806af00 msgr2=0x7fab9806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fab9806af00 0x7fab9806d3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab9c000c00 tx=0x7fab9c007980 comp rx=0 tx=0).stop 2023-10-16T09:43:35.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabac0b09d0 msgr2=0x7fabac0b1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:35.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabac0b09d0 0x7fabac0b1250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faba00090b0 tx=0x7faba00047c0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 shutdown_connections 2023-10-16T09:43:35.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fab9806af00 0x7fab9806d3c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabac1ee5f0 0x7fabac06ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab94004850 0x7fabac15b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 --2- 172.21.15.158:0/3304055910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabac0b09d0 0x7fabac0b1250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:35.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 >> 172.21.15.158:0/3304055910 conn(0x7fabac06b9e0 msgr2=0x7fabac159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:35.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 shutdown_connections 2023-10-16T09:43:35.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:35.926+0000 7fabb249e700 1 -- 172.21.15.158:0/3304055910 wait complete. 2023-10-16T09:43:35.931 INFO:teuthology.orchestra.run.smithi158.stderr:listed 0 entries 2023-10-16T09:43:35.940 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi158.front.sepia.ceph.com... 2023-10-16T09:43:35.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:35.940 DEBUG:teuthology.orchestra.run.smithi158:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-16T09:43:35.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi158.stderr:ceph-fuse[111211]: fuse finished with error 0 and tester_r 0 2023-10-16T09:43:35.977 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:43:42.079 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:43:42.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:42.080 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-16T09:43:42.142 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:42.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.272+0000 7f1b75557700 1 Processor -- start 2023-10-16T09:43:42.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.272+0000 7f1b75557700 1 -- start start 2023-10-16T09:43:42.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b75557700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 0x7f1b700d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b75557700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1b700d1210 0x7f1b700ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b75557700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b700d3bd0 con 0x7f1b700ce950 2023-10-16T09:43:42.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b75557700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b700d4060 con 0x7f1b700d1210 2023-10-16T09:43:42.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b75557700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700cede0 con 0x7f1b7015d7c0 2023-10-16T09:43:42.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b6f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1b700ce950 0x7f1b700b50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46464/0 (socket says 172.21.15.158:46464) 2023-10-16T09:43:42.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b6f7fe700 1 -- 172.21.15.158:0/579737506 learned_addr learned my addr 172.21.15.158:0/579737506 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:42.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b6e7fc700 1 --2- 172.21.15.158:0/579737506 >> v2:172.21.15.172:3300/0 conn(0x7f1b700d1210 0x7f1b700ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b6effd700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 0x7f1b700d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347661336 0 0) 0x7f1b700d3bd0 con 0x7f1b700ce950 2023-10-16T09:43:42.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b580035b0 con 0x7f1b700ce950 2023-10-16T09:43:42.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.273+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4078996508 0 0) 0x7f1b580035b0 con 0x7f1b700ce950 2023-10-16T09:43:42.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 msgr2=0x7f1b700d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:42.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 0x7f1b700d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 >> v2:172.21.15.172:3300/0 conn(0x7f1b700d1210 msgr2=0x7f1b700ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:42.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 --2- 172.21.15.158:0/579737506 >> v2:172.21.15.172:3300/0 conn(0x7f1b700d1210 0x7f1b700ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700d2cf0 con 0x7f1b700ce950 2023-10-16T09:43:42.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1b68001a40 con 0x7f1b700ce950 2023-10-16T09:43:42.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 >> v1:172.21.15.158:6789/0 conn(0x7f1b700ce950 legacy=0x7f1b700b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58003a00 0x7f1b58003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 0x7f1b58004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b58004d90 0x7f1b58005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700d4060 con 0x7f1b58004d90 2023-10-16T09:43:42.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700cede0 con 0x7f1b58003a00 2023-10-16T09:43:42.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5800ba10 con 0x7f1b7015d7c0 2023-10-16T09:43:42.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6f7fe700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b58004d90 0x7f1b58005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6effd700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58003a00 0x7f1b58003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6e7fc700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 0x7f1b58004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6f7fe700 1 -- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 msgr2=0x7f1b58004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:42.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6f7fe700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b7015d7c0 0x7f1b58004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6f7fe700 1 -- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58003a00 msgr2=0x7f1b58003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:42.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6f7fe700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58003a00 0x7f1b58003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6f7fe700 1 -- 172.21.15.158:0/579737506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580035b0 con 0x7f1b58004d90 2023-10-16T09:43:42.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6f7fe700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b58004d90 0x7f1b58005210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800a870 tx=0x7f1b58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be65089ebcc5159 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b68012600 con 0x7f1b58004d90 2023-10-16T09:43:42.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b68012760 con 0x7f1b58004d90 2023-10-16T09:43:42.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.274+0000 7f1b6dffb700 1 -- 172.21.15.158:0/579737506 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b68012b10 con 0x7f1b58004d90 2023-10-16T09:43:42.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.275+0000 7f1b75557700 1 -- 172.21.15.158:0/579737506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b58004d90 msgr2=0x7f1b58005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.275+0000 7f1b75557700 1 --2- 172.21.15.158:0/579737506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b58004d90 0x7f1b58005210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800a870 tx=0x7f1b58005880 comp rx=0 tx=0).stop 2023-10-16T09:43:42.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.275+0000 7f1b6f7fe700 1 -- 172.21.15.158:0/579737506 reap_dead start 2023-10-16T09:43:42.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.275+0000 7f1b75557700 1 -- 172.21.15.158:0/579737506 shutdown_connections 2023-10-16T09:43:42.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.275+0000 7f1b75557700 1 -- 172.21.15.158:0/579737506 >> 172.21.15.158:0/579737506 conn(0x7f1b700669f0 msgr2=0x7f1b70066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:42.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.275+0000 7f1b75557700 1 -- 172.21.15.158:0/579737506 shutdown_connections 2023-10-16T09:43:42.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.275+0000 7f1b75557700 1 -- 172.21.15.158:0/579737506 wait complete. 2023-10-16T09:43:42.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.277+0000 7f1b75557700 1 Processor -- start 2023-10-16T09:43:42.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.277+0000 7f1b75557700 1 -- start start 2023-10-16T09:43:42.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b701f5aa0 0x7f1b700c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701f5e80 0x7f1b701e6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b70151b70 0x7f1b701e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c9df0 con 0x7f1b701f5aa0 2023-10-16T09:43:42.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701520c0 con 0x7f1b70151b70 2023-10-16T09:43:42.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701523a0 con 0x7f1b701f5e80 2023-10-16T09:43:42.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b701f5aa0 0x7f1b700c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b701f5aa0 0x7f1b700c84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46298/0 (socket says 172.21.15.158:46298) 2023-10-16T09:43:42.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 -- 172.21.15.158:0/3742761304 learned_addr learned my addr 172.21.15.158:0/3742761304 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:42.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6f7fe700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b70151b70 0x7f1b701e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6e7fc700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701f5e80 0x7f1b701e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 -- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701f5e80 msgr2=0x7f1b701e6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701f5e80 0x7f1b701e6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 -- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b70151b70 msgr2=0x7f1b701e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b70151b70 0x7f1b701e6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 -- 172.21.15.158:0/3742761304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70152680 con 0x7f1b701f5aa0 2023-10-16T09:43:42.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6f7fe700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b70151b70 0x7f1b701e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:42.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b6effd700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b701f5aa0 0x7f1b700c84e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b640119a0 tx=0x7f1b64011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b577fe700 1 -- 172.21.15.158:0/3742761304 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400f8e0 con 0x7f1b701f5aa0 2023-10-16T09:43:42.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b577fe700 1 -- 172.21.15.158:0/3742761304 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6400fa40 con 0x7f1b701f5aa0 2023-10-16T09:43:42.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b70152960 con 0x7f1b701f5aa0 2023-10-16T09:43:42.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b577fe700 1 -- 172.21.15.158:0/3742761304 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400fdd0 con 0x7f1b701f5aa0 2023-10-16T09:43:42.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.278+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b700c7860 con 0x7f1b701f5aa0 2023-10-16T09:43:42.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.279+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b70152e80 con 0x7f1b701f5aa0 2023-10-16T09:43:42.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.280+0000 7f1b577fe700 1 -- 172.21.15.158:0/3742761304 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1b64014440 con 0x7f1b701f5aa0 2023-10-16T09:43:42.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.280+0000 7f1b577fe700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b5c06af30 0x7f1b5c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.280+0000 7f1b577fe700 1 -- 172.21.15.158:0/3742761304 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1b64089f60 con 0x7f1b701f5aa0 2023-10-16T09:43:42.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.280+0000 7f1b6e7fc700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b5c06af30 0x7f1b5c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.281+0000 7f1b6e7fc700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b5c06af30 0x7f1b5c06d3f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b70160b50 tx=0x7f1b60008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.285+0000 7f1b577fe700 1 -- 172.21.15.158:0/3742761304 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b64054d20 con 0x7f1b701f5aa0 2023-10-16T09:43:42.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.479+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b700681b0 con 0x7f1b701f5aa0 2023-10-16T09:43:42.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.480+0000 7f1b577fe700 1 -- 172.21.15.158:0/3742761304 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f1b64058340 con 0x7f1b701f5aa0 2023-10-16T09:43:42.482 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:42.482 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:42.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b5c06af30 msgr2=0x7f1b5c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b5c06af30 0x7f1b5c06d3f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b70160b50 tx=0x7f1b60008040 comp rx=0 tx=0).stop 2023-10-16T09:43:42.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b701f5aa0 msgr2=0x7f1b700c84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b701f5aa0 0x7f1b700c84e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b640119a0 tx=0x7f1b64011d60 comp rx=0 tx=0).stop 2023-10-16T09:43:42.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 shutdown_connections 2023-10-16T09:43:42.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b70151b70 0x7f1b701e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b701f5aa0 0x7f1b700c84e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b5c06af30 0x7f1b5c06d3f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 --2- 172.21.15.158:0/3742761304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701f5e80 0x7f1b701e6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 >> 172.21.15.158:0/3742761304 conn(0x7f1b700669f0 msgr2=0x7f1b7015a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:42.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 shutdown_connections 2023-10-16T09:43:42.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.482+0000 7f1b75557700 1 -- 172.21.15.158:0/3742761304 wait complete. 2023-10-16T09:43:42.496 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:42.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.615+0000 7fa29bca1700 1 Processor -- start 2023-10-16T09:43:42.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.615+0000 7fa29bca1700 1 -- start start 2023-10-16T09:43:42.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29bca1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa29406b0a0 0x7fa29415f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29bca1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa2940ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29bca1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa294072700 con 0x7fa29406e2b0 2023-10-16T09:43:42.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29bca1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa294071910 con 0x7fa29406b0a0 2023-10-16T09:43:42.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29bca1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29406b480 con 0x7fa29406d820 2023-10-16T09:43:42.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa299a3d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa29406b0a0 0x7fa29415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29a23e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa29406e2b0 0x7fa29415bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46486/0 (socket says 172.21.15.158:46486) 2023-10-16T09:43:42.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29a23e700 1 -- 172.21.15.158:0/140713904 learned_addr learned my addr 172.21.15.158:0/140713904 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:42.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa29923c700 1 --2- 172.21.15.158:0/140713904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa2940ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa299a3d700 1 -- 172.21.15.158:0/140713904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 msgr2=0x7fa2940ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa299a3d700 1 --2- 172.21.15.158:0/140713904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa2940ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa299a3d700 1 -- 172.21.15.158:0/140713904 >> v1:172.21.15.158:6789/0 conn(0x7fa29406e2b0 legacy=0x7fa29415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa298a3b700 1 -- 172.21.15.158:0/140713904 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008813711 0 0) 0x7fa294072700 con 0x7fa29406e2b0 2023-10-16T09:43:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa299a3d700 1 -- 172.21.15.158:0/140713904 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940cca90 con 0x7fa29406b0a0 2023-10-16T09:43:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.616+0000 7fa299a3d700 1 --2- 172.21.15.158:0/140713904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406b0a0 0x7fa29415f010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa290009870 tx=0x7fa290009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c94eb31b77a4df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa298a3b700 1 -- 172.21.15.158:0/140713904 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290011830 con 0x7fa29406b0a0 2023-10-16T09:43:42.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa298a3b700 1 -- 172.21.15.158:0/140713904 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa290011990 con 0x7fa29406b0a0 2023-10-16T09:43:42.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa298a3b700 1 -- 172.21.15.158:0/140713904 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290011d20 con 0x7fa29406b0a0 2023-10-16T09:43:42.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 -- 172.21.15.158:0/140713904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406b0a0 msgr2=0x7fa29415f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 --2- 172.21.15.158:0/140713904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406b0a0 0x7fa29415f010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa290009870 tx=0x7fa290009b80 comp rx=0 tx=0).stop 2023-10-16T09:43:42.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 -- 172.21.15.158:0/140713904 shutdown_connections 2023-10-16T09:43:42.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 --2- 172.21.15.158:0/140713904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406b0a0 0x7fa29415f010 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 --2- 172.21.15.158:0/140713904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa2940ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 -- 172.21.15.158:0/140713904 >> 172.21.15.158:0/140713904 conn(0x7fa2940d02b0 msgr2=0x7fa2940d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:42.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 -- 172.21.15.158:0/140713904 shutdown_connections 2023-10-16T09:43:42.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.617+0000 7fa29bca1700 1 -- 172.21.15.158:0/140713904 wait complete. 2023-10-16T09:43:42.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.619+0000 7fa29bca1700 1 Processor -- start 2023-10-16T09:43:42.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.619+0000 7fa29bca1700 1 -- start start 2023-10-16T09:43:42.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29bca1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa29406b0a0 0x7fa29414a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29bca1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa294151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29bca1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406e2b0 0x7fa2940c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29bca1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa294150250 con 0x7fa29406e2b0 2023-10-16T09:43:42.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29bca1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa294151860 con 0x7fa29406b0a0 2023-10-16T09:43:42.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29bca1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940b0d60 con 0x7fa29406d820 2023-10-16T09:43:42.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406e2b0 0x7fa2940c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406e2b0 0x7fa2940c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46316/0 (socket says 172.21.15.158:46316) 2023-10-16T09:43:42.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 -- 172.21.15.158:0/457817215 learned_addr learned my addr 172.21.15.158:0/457817215 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:42.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29923c700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa294151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa299a3d700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa29406b0a0 0x7fa29414a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 -- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 msgr2=0x7fa294151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa294151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 -- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa29406b0a0 msgr2=0x7fa29414a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa29406b0a0 0x7fa29414a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 -- 172.21.15.158:0/457817215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900094a0 con 0x7fa29406e2b0 2023-10-16T09:43:42.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.620+0000 7fa29a23e700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406e2b0 0x7fa2940c5fb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa28c002230 tx=0x7fa28c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.621+0000 7fa28a7fc700 1 -- 172.21.15.158:0/457817215 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c00abc0 con 0x7fa29406e2b0 2023-10-16T09:43:42.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.621+0000 7fa28a7fc700 1 -- 172.21.15.158:0/457817215 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c00ad40 con 0x7fa29406e2b0 2023-10-16T09:43:42.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.621+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa294150460 con 0x7fa29406e2b0 2023-10-16T09:43:42.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.621+0000 7fa28a7fc700 1 -- 172.21.15.158:0/457817215 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c007310 con 0x7fa29406e2b0 2023-10-16T09:43:42.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.621+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa294150a20 con 0x7fa29406e2b0 2023-10-16T09:43:42.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.621+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2940681b0 con 0x7fa29406e2b0 2023-10-16T09:43:42.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.622+0000 7fa28a7fc700 1 -- 172.21.15.158:0/457817215 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa28c007470 con 0x7fa29406e2b0 2023-10-16T09:43:42.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.622+0000 7fa28a7fc700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa27806af00 0x7fa27806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.622+0000 7fa28a7fc700 1 -- 172.21.15.158:0/457817215 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa28c089770 con 0x7fa29406e2b0 2023-10-16T09:43:42.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.622+0000 7fa299a3d700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa27806af00 0x7fa27806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.623+0000 7fa299a3d700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa27806af00 0x7fa27806d3c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2900097c0 tx=0x7fa29001c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.627+0000 7fa28a7fc700 1 -- 172.21.15.158:0/457817215 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa28c054530 con 0x7fa29406e2b0 2023-10-16T09:43:42.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.820+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2940d10e0 con 0x7fa29406e2b0 2023-10-16T09:43:42.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.821+0000 7fa28a7fc700 1 -- 172.21.15.158:0/457817215 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7fa28c057b50 con 0x7fa29406e2b0 2023-10-16T09:43:42.823 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:42.824 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:42.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.823+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa27806af00 msgr2=0x7fa27806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.823+0000 7fa29bca1700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa27806af00 0x7fa27806d3c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2900097c0 tx=0x7fa29001c040 comp rx=0 tx=0).stop 2023-10-16T09:43:42.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.823+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406e2b0 msgr2=0x7fa2940c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.823+0000 7fa29bca1700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406e2b0 0x7fa2940c5fb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa28c002230 tx=0x7fa28c00a060 comp rx=0 tx=0).stop 2023-10-16T09:43:42.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 shutdown_connections 2023-10-16T09:43:42.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa29406e2b0 0x7fa2940c5fb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa29406b0a0 0x7fa29414a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa27806af00 0x7fa27806d3c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 --2- 172.21.15.158:0/457817215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa29406d820 0x7fa294151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 >> 172.21.15.158:0/457817215 conn(0x7fa2940d02b0 msgr2=0x7fa29415aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:42.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 shutdown_connections 2023-10-16T09:43:42.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.824+0000 7fa29bca1700 1 -- 172.21.15.158:0/457817215 wait complete. 2023-10-16T09:43:42.837 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T09:43:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.959+0000 7fc138426700 1 Processor -- start 2023-10-16T09:43:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.959+0000 7fc138426700 1 -- start start 2023-10-16T09:43:42.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc138426700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13015dad0 0x7fc1300cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc138426700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc13015deb0 0x7fc1300ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc138426700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1300d0ae0 con 0x7fc1300cea20 2023-10-16T09:43:42.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc138426700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1300d12d0 con 0x7fc13015deb0 2023-10-16T09:43:42.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc138426700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300cbdf0 con 0x7fc13015dad0 2023-10-16T09:43:42.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc13015deb0 0x7fc1300ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc13015deb0 0x7fc1300ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46320/0 (socket says 172.21.15.158:46320) 2023-10-16T09:43:42.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 -- 172.21.15.158:0/1829555301 learned_addr learned my addr 172.21.15.158:0/1829555301 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:42.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1361c2700 1 --2- 172.21.15.158:0/1829555301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13015dad0 0x7fc1300cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 -- 172.21.15.158:0/1829555301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13015dad0 msgr2=0x7fc1300cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:42.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 --2- 172.21.15.158:0/1829555301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13015dad0 0x7fc1300cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 -- 172.21.15.158:0/1829555301 >> v1:172.21.15.172:6789/0 conn(0x7fc1300cea20 legacy=0x7fc1300c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 -- 172.21.15.158:0/1829555301 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1300cc280 con 0x7fc13015deb0 2023-10-16T09:43:42.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1359c1700 1 --2- 172.21.15.158:0/1829555301 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300ce4e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc12000bb90 tx=0x7fc12000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b77e2dae9c0617 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.960+0000 7fc1351c0700 1 -- 172.21.15.158:0/1829555301 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12000dd50 con 0x7fc13015deb0 2023-10-16T09:43:42.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc1351c0700 1 -- 172.21.15.158:0/1829555301 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc120010470 con 0x7fc13015deb0 2023-10-16T09:43:42.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc1351c0700 1 -- 172.21.15.158:0/1829555301 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120010800 con 0x7fc13015deb0 2023-10-16T09:43:42.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 -- 172.21.15.158:0/1829555301 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 msgr2=0x7fc1300ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 --2- 172.21.15.158:0/1829555301 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300ce4e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc12000bb90 tx=0x7fc12000bea0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 -- 172.21.15.158:0/1829555301 shutdown_connections 2023-10-16T09:43:42.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 --2- 172.21.15.158:0/1829555301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13015dad0 0x7fc1300cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 --2- 172.21.15.158:0/1829555301 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300ce4e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 -- 172.21.15.158:0/1829555301 >> 172.21.15.158:0/1829555301 conn(0x7fc130099a10 msgr2=0x7fc1300d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:42.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 -- 172.21.15.158:0/1829555301 shutdown_connections 2023-10-16T09:43:42.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.961+0000 7fc138426700 1 -- 172.21.15.158:0/1829555301 wait complete. 2023-10-16T09:43:42.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.962+0000 7fc138426700 1 Processor -- start 2023-10-16T09:43:42.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.962+0000 7fc138426700 1 -- start start 2023-10-16T09:43:42.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc138426700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1300cea20 0x7fc1300b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc138426700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13015dad0 0x7fc1300c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc138426700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc138426700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300b3b80 con 0x7fc13015deb0 2023-10-16T09:43:42.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc138426700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300b3e60 con 0x7fc13015dad0 2023-10-16T09:43:42.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc138426700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300b4140 con 0x7fc1300cea20 2023-10-16T09:43:42.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc1369c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc1369c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46330/0 (socket says 172.21.15.158:46330) 2023-10-16T09:43:42.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc1369c3700 1 -- 172.21.15.158:0/978516157 learned_addr learned my addr 172.21.15.158:0/978516157 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:42.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc1361c2700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1300cea20 0x7fc1300b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.963+0000 7fc1359c1700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13015dad0 0x7fc1300c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc1369c3700 1 -- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1300cea20 msgr2=0x7fc1300b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc1369c3700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1300cea20 0x7fc1300b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc1369c3700 1 -- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13015dad0 msgr2=0x7fc1300c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:42.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc1369c3700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13015dad0 0x7fc1300c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:42.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc1369c3700 1 -- 172.21.15.158:0/978516157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12000b7f0 con 0x7fc13015deb0 2023-10-16T09:43:42.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc1369c3700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300c74a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc12800cc50 tx=0x7fc12800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc126ffd700 1 -- 172.21.15.158:0/978516157 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128003c00 con 0x7fc13015deb0 2023-10-16T09:43:42.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc126ffd700 1 -- 172.21.15.158:0/978516157 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128003d60 con 0x7fc13015deb0 2023-10-16T09:43:42.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1300b4420 con 0x7fc13015deb0 2023-10-16T09:43:42.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc126ffd700 1 -- 172.21.15.158:0/978516157 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12800aab0 con 0x7fc13015deb0 2023-10-16T09:43:42.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.964+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1300b4940 con 0x7fc13015deb0 2023-10-16T09:43:42.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.965+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1300681b0 con 0x7fc13015deb0 2023-10-16T09:43:42.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.965+0000 7fc126ffd700 1 -- 172.21.15.158:0/978516157 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc12800ac10 con 0x7fc13015deb0 2023-10-16T09:43:42.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.966+0000 7fc126ffd700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc11406af00 0x7fc11406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:42.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.966+0000 7fc126ffd700 1 -- 172.21.15.158:0/978516157 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc1280898b0 con 0x7fc13015deb0 2023-10-16T09:43:42.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.966+0000 7fc1361c2700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc11406af00 0x7fc11406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:42.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.966+0000 7fc1361c2700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc11406af00 0x7fc11406d3c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0099d0 tx=0x7fc12c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:42.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:42.970+0000 7fc126ffd700 1 -- 172.21.15.158:0/978516157 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc128054660 con 0x7fc13015deb0 2023-10-16T09:43:43.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.211+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc1300b3cc0 con 0x7fc13015deb0 2023-10-16T09:43:43.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.376+0000 7fc126ffd700 1 -- 172.21.15.158:0/978516157 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc1300b3cc0 con 0x7fc13015deb0 2023-10-16T09:43:43.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.378+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc11406af00 msgr2=0x7fc11406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:43.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.378+0000 7fc138426700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc11406af00 0x7fc11406d3c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0099d0 tx=0x7fc12c008040 comp rx=0 tx=0).stop 2023-10-16T09:43:43.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.378+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 msgr2=0x7fc1300c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:43.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300c74a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc12800cc50 tx=0x7fc12800cf60 comp rx=0 tx=0).stop 2023-10-16T09:43:43.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 shutdown_connections 2023-10-16T09:43:43.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13015deb0 0x7fc1300c74a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc11406af00 0x7fc11406d3c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1300cea20 0x7fc1300b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 --2- 172.21.15.158:0/978516157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13015dad0 0x7fc1300c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 >> 172.21.15.158:0/978516157 conn(0x7fc130099a10 msgr2=0x7fc1300ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:43.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 shutdown_connections 2023-10-16T09:43:43.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.379+0000 7fc138426700 1 -- 172.21.15.158:0/978516157 wait complete. 2023-10-16T09:43:43.392 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-10-16T09:43:43.393 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T09:43:43.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.515+0000 7f60d0d47700 1 Processor -- start 2023-10-16T09:43:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.515+0000 7f60d0d47700 1 -- start start 2023-10-16T09:43:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60d0d47700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60cc0aefa0 0x7f60cc161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60d0d47700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f60cc071cb0 0x7f60cc06ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60d0d47700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60cc0b3dc0 con 0x7f60cc15eae0 2023-10-16T09:43:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60d0d47700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60cc15ddf0 con 0x7f60cc0aefa0 2023-10-16T09:43:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60d0d47700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60cc1622f0 con 0x7f60cc071cb0 2023-10-16T09:43:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60cc0aefa0 0x7f60cc161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60cc0aefa0 0x7f60cc161db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46334/0 (socket says 172.21.15.158:46334) 2023-10-16T09:43:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 -- 172.21.15.158:0/1478699737 learned_addr learned my addr 172.21.15.158:0/1478699737 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:43.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60cad9d700 1 --2- 172.21.15.158:0/1478699737 >> v2:172.21.15.172:3300/0 conn(0x7f60cc071cb0 0x7f60cc06ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 -- 172.21.15.158:0/1478699737 >> v2:172.21.15.172:3300/0 conn(0x7f60cc071cb0 msgr2=0x7f60cc06ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 --2- 172.21.15.158:0/1478699737 >> v2:172.21.15.172:3300/0 conn(0x7f60cc071cb0 0x7f60cc06ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 -- 172.21.15.158:0/1478699737 >> v1:172.21.15.172:6789/0 conn(0x7f60cc15eae0 legacy=0x7f60cc0715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 -- 172.21.15.158:0/1478699737 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc0b3ae0 con 0x7f60cc0aefa0 2023-10-16T09:43:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60ca59c700 1 --2- 172.21.15.158:0/1478699737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc0aefa0 0x7f60cc161db0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0b4a50 tx=0x7f60bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9492ee4822946cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60c959a700 1 -- 172.21.15.158:0/1478699737 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00b870 con 0x7f60cc0aefa0 2023-10-16T09:43:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60c959a700 1 -- 172.21.15.158:0/1478699737 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc00b9d0 con 0x7f60cc0aefa0 2023-10-16T09:43:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.516+0000 7f60c959a700 1 -- 172.21.15.158:0/1478699737 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00bd60 con 0x7f60cc0aefa0 2023-10-16T09:43:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 -- 172.21.15.158:0/1478699737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc0aefa0 msgr2=0x7f60cc161db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1478699737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc0aefa0 0x7f60cc161db0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0b4a50 tx=0x7f60bc009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 -- 172.21.15.158:0/1478699737 shutdown_connections 2023-10-16T09:43:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1478699737 >> v2:172.21.15.172:3300/0 conn(0x7f60cc071cb0 0x7f60cc06ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1478699737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc0aefa0 0x7f60cc161db0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 -- 172.21.15.158:0/1478699737 >> 172.21.15.158:0/1478699737 conn(0x7f60cc0b1bd0 msgr2=0x7f60cc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 -- 172.21.15.158:0/1478699737 shutdown_connections 2023-10-16T09:43:43.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.517+0000 7f60d0d47700 1 -- 172.21.15.158:0/1478699737 wait complete. 2023-10-16T09:43:43.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.518+0000 7f60d0d47700 1 Processor -- start 2023-10-16T09:43:43.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.518+0000 7f60d0d47700 1 -- start start 2023-10-16T09:43:43.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60d0d47700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc071cb0 0x7f60cc1421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60d0d47700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60cc0aefa0 0x7f60cc13e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60d0d47700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60cc15eae0 0x7f60cc1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60d0d47700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc150f90 con 0x7f60cc071cb0 2023-10-16T09:43:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60d0d47700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc13eb20 con 0x7f60cc0aefa0 2023-10-16T09:43:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60d0d47700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc150030 con 0x7f60cc15eae0 2023-10-16T09:43:43.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc071cb0 0x7f60cc1421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:43.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc071cb0 0x7f60cc1421f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46342/0 (socket says 172.21.15.158:46342) 2023-10-16T09:43:43.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60ca59c700 1 -- 172.21.15.158:0/1491180985 learned_addr learned my addr 172.21.15.158:0/1491180985 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:43.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60c9d9b700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60cc0aefa0 0x7f60cc13e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:43.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.519+0000 7f60cad9d700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60cc15eae0 0x7f60cc1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:43.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60ca59c700 1 -- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60cc15eae0 msgr2=0x7f60cc1576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:43.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60ca59c700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60cc15eae0 0x7f60cc1576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60ca59c700 1 -- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60cc0aefa0 msgr2=0x7f60cc13e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:43.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60ca59c700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60cc0aefa0 0x7f60cc13e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:43.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60ca59c700 1 -- 172.21.15.158:0/1491180985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc009550 con 0x7f60cc071cb0 2023-10-16T09:43:43.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60c9d9b700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60cc0aefa0 0x7f60cc13e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:43.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60ca59c700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc071cb0 0x7f60cc1421f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f60bc000c00 tx=0x7f60bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:43.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60baffd700 1 -- 172.21.15.158:0/1491180985 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc004540 con 0x7f60cc071cb0 2023-10-16T09:43:43.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60baffd700 1 -- 172.21.15.158:0/1491180985 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc0036a0 con 0x7f60cc071cb0 2023-10-16T09:43:43.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60baffd700 1 -- 172.21.15.158:0/1491180985 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc003a30 con 0x7f60cc071cb0 2023-10-16T09:43:43.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60cc150310 con 0x7f60cc071cb0 2023-10-16T09:43:43.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.520+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60cc1508d0 con 0x7f60cc071cb0 2023-10-16T09:43:43.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.521+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60cc0681b0 con 0x7f60cc071cb0 2023-10-16T09:43:43.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.521+0000 7f60baffd700 1 -- 172.21.15.158:0/1491180985 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f60bc021070 con 0x7f60cc071cb0 2023-10-16T09:43:43.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.522+0000 7f60baffd700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f60ac06af00 0x7f60ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:43.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.522+0000 7f60baffd700 1 -- 172.21.15.158:0/1491180985 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f60bc08b010 con 0x7f60cc071cb0 2023-10-16T09:43:43.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.522+0000 7f60c9d9b700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f60ac06af00 0x7f60ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:43.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.522+0000 7f60c9d9b700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f60ac06af00 0x7f60ac06d3c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0cf760 tx=0x7f60c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:43.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.526+0000 7f60baffd700 1 -- 172.21.15.158:0/1491180985 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60bc055de0 con 0x7f60cc071cb0 2023-10-16T09:43:43.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:43.764+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f60cc06e6a0 con 0x7f60cc071cb0 2023-10-16T09:43:44.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.380+0000 7f60baffd700 1 -- 172.21.15.158:0/1491180985 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f60bc059400 con 0x7f60cc071cb0 2023-10-16T09:43:44.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f60ac06af00 msgr2=0x7f60ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f60ac06af00 0x7f60ac06d3c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0cf760 tx=0x7f60c0009040 comp rx=0 tx=0).stop 2023-10-16T09:43:44.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc071cb0 msgr2=0x7f60cc1421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc071cb0 0x7f60cc1421f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f60bc000c00 tx=0x7f60bc004750 comp rx=0 tx=0).stop 2023-10-16T09:43:44.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 shutdown_connections 2023-10-16T09:43:44.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f60ac06af00 0x7f60ac06d3c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60cc15eae0 0x7f60cc1576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60cc071cb0 0x7f60cc1421f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 --2- 172.21.15.158:0/1491180985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60cc0aefa0 0x7f60cc13e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 >> 172.21.15.158:0/1491180985 conn(0x7f60cc0b1bd0 msgr2=0x7f60cc0b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:44.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 shutdown_connections 2023-10-16T09:43:44.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.383+0000 7f60d0d47700 1 -- 172.21.15.158:0/1491180985 wait complete. 2023-10-16T09:43:44.399 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T09:43:44.400 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:44.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.524+0000 7f781a060700 1 Processor -- start 2023-10-16T09:43:44.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.524+0000 7f781a060700 1 -- start start 2023-10-16T09:43:44.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f781a060700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78140d3570 0x7f78140d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f781a060700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140ce4d0 con 0x7f781415daa0 2023-10-16T09:43:44.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f781a060700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140d3ec0 con 0x7f78140d0bd0 2023-10-16T09:43:44.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f781a060700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78140d1c20 con 0x7f78140d3570 2023-10-16T09:43:44.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f78137fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f781415daa0 0x7f781406b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46522/0 (socket says 172.21.15.158:46522) 2023-10-16T09:43:44.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f78137fe700 1 -- 172.21.15.158:0/2597328214 learned_addr learned my addr 172.21.15.158:0/2597328214 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:44.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f7813fff700 1 --2- 172.21.15.158:0/2597328214 >> v2:172.21.15.172:3300/0 conn(0x7f78140d3570 0x7f78140d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396380993 0 0) 0x7f78140ce4d0 con 0x7f781415daa0 2023-10-16T09:43:44.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.525+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78000035b0 con 0x7f781415daa0 2023-10-16T09:43:44.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615837937 0 0) 0x7f78140d3ec0 con 0x7f78140d0bd0 2023-10-16T09:43:44.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78140ce4d0 con 0x7f78140d0bd0 2023-10-16T09:43:44.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4249738278 0 0) 0x7f78000035b0 con 0x7f781415daa0 2023-10-16T09:43:44.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 >> v2:172.21.15.172:3300/0 conn(0x7f78140d3570 msgr2=0x7f78140d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:44.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 --2- 172.21.15.158:0/2597328214 >> v2:172.21.15.172:3300/0 conn(0x7f78140d3570 0x7f78140d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 >> v1:172.21.15.172:6789/0 conn(0x7f78140d0bd0 legacy=0x7f78140d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781406c1b0 con 0x7f781415daa0 2023-10-16T09:43:44.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f77fc0017f0 con 0x7f781415daa0 2023-10-16T09:43:44.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 >> v1:172.21.15.158:6789/0 conn(0x7f781415daa0 legacy=0x7f781406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7800003be0 0x7f7800003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800004850 0x7f7800004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7800005210 0x7f780000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78140d1c20 con 0x7f7800003be0 2023-10-16T09:43:44.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780000c0e0 con 0x7f7800005210 2023-10-16T09:43:44.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780000c2a0 con 0x7f7800004850 2023-10-16T09:43:44.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78137fe700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7800003be0 0x7f7800003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f7813fff700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7800005210 0x7f780000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f7812ffd700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800004850 0x7f7800004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78137fe700 1 -- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800004850 msgr2=0x7f7800004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:44.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78137fe700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800004850 0x7f7800004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78137fe700 1 -- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7800005210 msgr2=0x7f780000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:44.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78137fe700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7800005210 0x7f780000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f7813fff700 1 -- 172.21.15.158:0/2597328214 reap_dead start 2023-10-16T09:43:44.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78137fe700 1 -- 172.21.15.158:0/2597328214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000035b0 con 0x7f7800003be0 2023-10-16T09:43:44.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f7812ffd700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800004850 0x7f7800004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:44.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78137fe700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7800003be0 0x7f7800003fe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c110 tx=0x7f7800005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728a735c750f6bf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:44.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc00cc50 con 0x7f7800003be0 2023-10-16T09:43:44.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc00cdb0 con 0x7f7800003be0 2023-10-16T09:43:44.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.526+0000 7f78127fc700 1 -- 172.21.15.158:0/2597328214 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc007310 con 0x7f7800003be0 2023-10-16T09:43:44.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.527+0000 7f781a060700 1 -- 172.21.15.158:0/2597328214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7800003be0 msgr2=0x7f7800003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.527+0000 7f781a060700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7800003be0 0x7f7800003fe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c110 tx=0x7f7800005880 comp rx=0 tx=0).stop 2023-10-16T09:43:44.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.527+0000 7f781a060700 1 -- 172.21.15.158:0/2597328214 shutdown_connections 2023-10-16T09:43:44.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.527+0000 7f781a060700 1 --2- 172.21.15.158:0/2597328214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7800003be0 0x7f7800003fe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.527+0000 7f781a060700 1 -- 172.21.15.158:0/2597328214 >> 172.21.15.158:0/2597328214 conn(0x7f781408d040 msgr2=0x7f781409c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:44.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.527+0000 7f781a060700 1 -- 172.21.15.158:0/2597328214 shutdown_connections 2023-10-16T09:43:44.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.527+0000 7f781a060700 1 -- 172.21.15.158:0/2597328214 wait complete. 2023-10-16T09:43:44.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.528+0000 7f781a060700 1 Processor -- start 2023-10-16T09:43:44.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.528+0000 7f781a060700 1 -- start start 2023-10-16T09:43:44.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f781a060700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800003be0 0x7f781415b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f781a060700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f781415e310 0x7f78140b83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f781a060700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78140b8920 0x7f78140ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f781a060700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78140b8da0 con 0x7f781415e310 2023-10-16T09:43:44.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f781a060700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78141f3fc0 con 0x7f78140b8920 2023-10-16T09:43:44.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f781a060700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78140cbe30 con 0x7f7800003be0 2023-10-16T09:43:44.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f7812ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f781415e310 0x7f78140b83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f7812ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f781415e310 0x7f78140b83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46360/0 (socket says 172.21.15.158:46360) 2023-10-16T09:43:44.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f7812ffd700 1 -- 172.21.15.158:0/4216152346 learned_addr learned my addr 172.21.15.158:0/4216152346 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:44.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f78137fe700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800003be0 0x7f781415b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.529+0000 7f7813fff700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78140b8920 0x7f78140ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f7812ffd700 1 -- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800003be0 msgr2=0x7f781415b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f7812ffd700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800003be0 0x7f781415b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f7812ffd700 1 -- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78140b8920 msgr2=0x7f78140ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f7812ffd700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78140b8920 0x7f78140ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f7812ffd700 1 -- 172.21.15.158:0/4216152346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140cb640 con 0x7f781415e310 2023-10-16T09:43:44.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f78137fe700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800003be0 0x7f781415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:44.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f7812ffd700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f781415e310 0x7f78140b83e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f780400e780 tx=0x7f780400eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:44.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f780bfff700 1 -- 172.21.15.158:0/4216152346 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78040052e0 con 0x7f781415e310 2023-10-16T09:43:44.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f780bfff700 1 -- 172.21.15.158:0/4216152346 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7804005440 con 0x7f781415e310 2023-10-16T09:43:44.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78140cc080 con 0x7f781415e310 2023-10-16T09:43:44.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f780bfff700 1 -- 172.21.15.158:0/4216152346 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804003880 con 0x7f781415e310 2023-10-16T09:43:44.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.530+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78140cc640 con 0x7f781415e310 2023-10-16T09:43:44.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.531+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78140681b0 con 0x7f781415e310 2023-10-16T09:43:44.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.531+0000 7f780bfff700 1 -- 172.21.15.158:0/4216152346 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f78040039e0 con 0x7f781415e310 2023-10-16T09:43:44.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.532+0000 7f780bfff700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f77f406af00 0x7f77f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.532+0000 7f78137fe700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f77f406af00 0x7f77f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.532+0000 7f780bfff700 1 -- 172.21.15.158:0/4216152346 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7804089000 con 0x7f781415e310 2023-10-16T09:43:44.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.532+0000 7f78137fe700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f77f406af00 0x7f77f406d3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77fc015040 tx=0x7f77fc007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:44.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.536+0000 7f780bfff700 1 -- 172.21.15.158:0/4216152346 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7804053e80 con 0x7f781415e310 2023-10-16T09:43:44.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.774+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78140d4450 con 0x7f781415e310 2023-10-16T09:43:44.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.774+0000 7f780bfff700 1 -- 172.21.15.158:0/4216152346 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7f78040574a0 con 0x7f781415e310 2023-10-16T09:43:44.776 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:44.777 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:43:44.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.776+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f77f406af00 msgr2=0x7f77f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.776+0000 7f781a060700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f77f406af00 0x7f77f406d3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77fc015040 tx=0x7f77fc007000 comp rx=0 tx=0).stop 2023-10-16T09:43:44.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.776+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f781415e310 msgr2=0x7f78140b83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.776+0000 7f781a060700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f781415e310 0x7f78140b83e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f780400e780 tx=0x7f780400eb40 comp rx=0 tx=0).stop 2023-10-16T09:43:44.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 shutdown_connections 2023-10-16T09:43:44.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78140b8920 0x7f78140ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f77f406af00 0x7f77f406d3c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7800003be0 0x7f781415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 --2- 172.21.15.158:0/4216152346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f781415e310 0x7f78140b83e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 >> 172.21.15.158:0/4216152346 conn(0x7f781408d040 msgr2=0x7f78140d3980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:44.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 shutdown_connections 2023-10-16T09:43:44.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.777+0000 7f781a060700 1 -- 172.21.15.158:0/4216152346 wait complete. 2023-10-16T09:43:44.781 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:43:44.790 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:44.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.909+0000 7f6583bf0700 1 Processor -- start 2023-10-16T09:43:44.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.909+0000 7f6583bf0700 1 -- start start 2023-10-16T09:43:44.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f6583bf0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f657c0cdee0 0x7f657c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f6583bf0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0ce2c0 0x7f657c0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f6583bf0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c15c5d0 con 0x7f657c0b6e10 2023-10-16T09:43:44.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f6583bf0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c15d2c0 con 0x7f657c0cdee0 2023-10-16T09:43:44.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f6583bf0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c15c8b0 con 0x7f657c0ce2c0 2023-10-16T09:43:44.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f658218d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0ce2c0 0x7f657c0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f658198c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f657c0cdee0 0x7f657c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f658118b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f657c0b6e10 0x7f657c1521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:50446/0 (socket says 172.21.15.158:50446) 2023-10-16T09:43:44.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f658218d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0ce2c0 0x7f657c0cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57202/0 (socket says 172.21.15.158:57202) 2023-10-16T09:43:44.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f658198c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f657c0cdee0 0x7f657c15bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47546/0 (socket says 172.21.15.158:47546) 2023-10-16T09:43:44.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.910+0000 7f658118b700 1 -- 172.21.15.158:0/2650121474 learned_addr learned my addr 172.21.15.158:0/2650121474 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:44.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658098a700 1 -- 172.21.15.158:0/2650121474 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303924726 0 0) 0x7f657c15c5d0 con 0x7f657c0b6e10 2023-10-16T09:43:44.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658098a700 1 -- 172.21.15.158:0/2650121474 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65680035b0 con 0x7f657c0b6e10 2023-10-16T09:43:44.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658198c700 1 -- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0ce2c0 msgr2=0x7f657c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:44.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658198c700 1 --2- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0ce2c0 0x7f657c0cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658198c700 1 -- 172.21.15.158:0/2650121474 >> v1:172.21.15.172:6789/0 conn(0x7f657c0b6e10 legacy=0x7f657c1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658198c700 1 -- 172.21.15.158:0/2650121474 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c15cb90 con 0x7f657c0cdee0 2023-10-16T09:43:44.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658218d700 1 --2- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0ce2c0 0x7f657c0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:44.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658198c700 1 --2- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0cdee0 0x7f657c15bd10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f656c009840 tx=0x7f656c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e21b612edea89d01 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:44.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658098a700 1 -- 172.21.15.158:0/2650121474 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c011830 con 0x7f657c0cdee0 2023-10-16T09:43:44.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658098a700 1 -- 172.21.15.158:0/2650121474 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c011990 con 0x7f657c0cdee0 2023-10-16T09:43:44.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.911+0000 7f658098a700 1 -- 172.21.15.158:0/2650121474 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c011d00 con 0x7f657c0cdee0 2023-10-16T09:43:44.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 -- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0cdee0 msgr2=0x7f657c15bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0cdee0 0x7f657c15bd10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f656c009840 tx=0x7f656c009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:44.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 -- 172.21.15.158:0/2650121474 shutdown_connections 2023-10-16T09:43:44.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0ce2c0 0x7f657c0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2650121474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0cdee0 0x7f657c15bd10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 -- 172.21.15.158:0/2650121474 >> 172.21.15.158:0/2650121474 conn(0x7f657c06a8d0 msgr2=0x7f657c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:44.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 -- 172.21.15.158:0/2650121474 shutdown_connections 2023-10-16T09:43:44.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.912+0000 7f6583bf0700 1 -- 172.21.15.158:0/2650121474 wait complete. 2023-10-16T09:43:44.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.913+0000 7f6583bf0700 1 Processor -- start 2023-10-16T09:43:44.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.913+0000 7f6583bf0700 1 -- start start 2023-10-16T09:43:44.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0b6e10 0x7f657c153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0cdee0 0x7f657c0c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f657c0ce2c0 0x7f657c0c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c154260 con 0x7f657c0ce2c0 2023-10-16T09:43:44.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c154540 con 0x7f657c0b6e10 2023-10-16T09:43:44.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c154820 con 0x7f657c0cdee0 2023-10-16T09:43:44.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f657c0ce2c0 0x7f657c0c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f657c0ce2c0 0x7f657c0c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46374/0 (socket says 172.21.15.158:46374) 2023-10-16T09:43:44.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 -- 172.21.15.158:0/2042107497 learned_addr learned my addr 172.21.15.158:0/2042107497 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:44.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658198c700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0b6e10 0x7f657c153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658118b700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0cdee0 0x7f657c0c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 -- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0cdee0 msgr2=0x7f657c0c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0cdee0 0x7f657c0c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 -- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0b6e10 msgr2=0x7f657c153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:44.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0b6e10 0x7f657c153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:44.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 -- 172.21.15.158:0/2042107497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6574009a70 con 0x7f657c0ce2c0 2023-10-16T09:43:44.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f658218d700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f657c0ce2c0 0x7f657c0c4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f657400efc0 tx=0x7f657400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:44.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f65727fc700 1 -- 172.21.15.158:0/2042107497 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657400ca90 con 0x7f657c0ce2c0 2023-10-16T09:43:44.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f65727fc700 1 -- 172.21.15.158:0/2042107497 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657400f040 con 0x7f657c0ce2c0 2023-10-16T09:43:44.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f65727fc700 1 -- 172.21.15.158:0/2042107497 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657400ce30 con 0x7f657c0ce2c0 2023-10-16T09:43:44.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c0094a0 con 0x7f657c0ce2c0 2023-10-16T09:43:44.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.914+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c154de0 con 0x7f657c0ce2c0 2023-10-16T09:43:44.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.915+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c0681b0 con 0x7f657c0ce2c0 2023-10-16T09:43:44.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.916+0000 7f65727fc700 1 -- 172.21.15.158:0/2042107497 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6574014480 con 0x7f657c0ce2c0 2023-10-16T09:43:44.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.916+0000 7f65727fc700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f656006af00 0x7f656006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:44.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.916+0000 7f65727fc700 1 -- 172.21.15.158:0/2042107497 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6574089f40 con 0x7f657c0ce2c0 2023-10-16T09:43:44.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.916+0000 7f658198c700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f656006af00 0x7f656006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:44.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.917+0000 7f658198c700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f656006af00 0x7f656006d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f656c000c00 tx=0x7f656c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:44.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:44.921+0000 7f65727fc700 1 -- 172.21.15.158:0/2042107497 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6574054d00 con 0x7f657c0ce2c0 2023-10-16T09:43:45.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.115+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f657c074270 con 0x7f657c0ce2c0 2023-10-16T09:43:45.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.116+0000 7f65727fc700 1 -- 172.21.15.158:0/2042107497 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f6574058320 con 0x7f657c0ce2c0 2023-10-16T09:43:45.118 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:45.118 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:45.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f656006af00 msgr2=0x7f656006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f656006af00 0x7f656006d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f656c000c00 tx=0x7f656c008030 comp rx=0 tx=0).stop 2023-10-16T09:43:45.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f657c0ce2c0 msgr2=0x7f657c0c4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f657c0ce2c0 0x7f657c0c4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f657400efc0 tx=0x7f657400c3a0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 shutdown_connections 2023-10-16T09:43:45.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f657c0ce2c0 0x7f657c0c4f00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f657c0b6e10 0x7f657c153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f656006af00 0x7f656006d3c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 --2- 172.21.15.158:0/2042107497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f657c0cdee0 0x7f657c0c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 >> 172.21.15.158:0/2042107497 conn(0x7f657c06a8d0 msgr2=0x7f657c15d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:45.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 shutdown_connections 2023-10-16T09:43:45.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.118+0000 7f6583bf0700 1 -- 172.21.15.158:0/2042107497 wait complete. 2023-10-16T09:43:45.132 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T09:43:45.133 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:43:45.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.253+0000 7f2409ba3700 1 Processor -- start 2023-10-16T09:43:45.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.253+0000 7f2409ba3700 1 -- start start 2023-10-16T09:43:45.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.253+0000 7f2409ba3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f240406ab40 0x7f240406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.253+0000 7f2409ba3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f240406e5d0 0x7f2404071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.253+0000 7f2409ba3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2404071e50 con 0x7f240406af20 2023-10-16T09:43:45.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.253+0000 7f2409ba3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2404161270 con 0x7f240406ab40 2023-10-16T09:43:45.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.253+0000 7f2409ba3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2404161510 con 0x7f240406e5d0 2023-10-16T09:43:45.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f240406ab40 0x7f240406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f240406ab40 0x7f240406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46378/0 (socket says 172.21.15.158:46378) 2023-10-16T09:43:45.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 -- 172.21.15.158:0/2129115965 learned_addr learned my addr 172.21.15.158:0/2129115965 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:45.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f2402ffd700 1 --2- 172.21.15.158:0/2129115965 >> v2:172.21.15.172:3300/0 conn(0x7f240406e5d0 0x7f2404071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 -- 172.21.15.158:0/2129115965 >> v2:172.21.15.172:3300/0 conn(0x7f240406e5d0 msgr2=0x7f2404071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:45.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 --2- 172.21.15.158:0/2129115965 >> v2:172.21.15.172:3300/0 conn(0x7f240406e5d0 0x7f2404071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 -- 172.21.15.158:0/2129115965 >> v1:172.21.15.172:6789/0 conn(0x7f240406af20 legacy=0x7f24040722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:45.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 -- 172.21.15.158:0/2129115965 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24041617f0 con 0x7f240406ab40 2023-10-16T09:43:45.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24037fe700 1 --2- 172.21.15.158:0/2129115965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406ab40 0x7f240406d820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24040c3bf0 tx=0x7f23f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718a9afd3f62427d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:45.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24027fc700 1 -- 172.21.15.158:0/2129115965 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4004230 con 0x7f240406ab40 2023-10-16T09:43:45.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24027fc700 1 -- 172.21.15.158:0/2129115965 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f4004390 con 0x7f240406ab40 2023-10-16T09:43:45.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.254+0000 7f24027fc700 1 -- 172.21.15.158:0/2129115965 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f40036a0 con 0x7f240406ab40 2023-10-16T09:43:45.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 -- 172.21.15.158:0/2129115965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406ab40 msgr2=0x7f240406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 --2- 172.21.15.158:0/2129115965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406ab40 0x7f240406d820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24040c3bf0 tx=0x7f23f4009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 -- 172.21.15.158:0/2129115965 shutdown_connections 2023-10-16T09:43:45.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 --2- 172.21.15.158:0/2129115965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406ab40 0x7f240406d820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 --2- 172.21.15.158:0/2129115965 >> v2:172.21.15.172:3300/0 conn(0x7f240406e5d0 0x7f2404071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 -- 172.21.15.158:0/2129115965 >> 172.21.15.158:0/2129115965 conn(0x7f24040d02b0 msgr2=0x7f24040d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:45.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 -- 172.21.15.158:0/2129115965 shutdown_connections 2023-10-16T09:43:45.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.255+0000 7f2409ba3700 1 -- 172.21.15.158:0/2129115965 wait complete. 2023-10-16T09:43:45.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.256+0000 7f2409ba3700 1 Processor -- start 2023-10-16T09:43:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.256+0000 7f2409ba3700 1 -- start start 2023-10-16T09:43:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2409ba3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240406ab40 0x7f24041f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2409ba3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406af20 0x7f24040c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2409ba3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240406e5d0 0x7f24040c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2409ba3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040c95c0 con 0x7f240406af20 2023-10-16T09:43:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2409ba3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240415f6c0 con 0x7f240406e5d0 2023-10-16T09:43:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2409ba3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24040c4fb0 con 0x7f240406ab40 2023-10-16T09:43:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406af20 0x7f24040c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406af20 0x7f24040c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46386/0 (socket says 172.21.15.158:46386) 2023-10-16T09:43:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 -- 172.21.15.158:0/982684484 learned_addr learned my addr 172.21.15.158:0/982684484 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2403fff700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240406e5d0 0x7f24040c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f24037fe700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240406ab40 0x7f24041f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 -- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240406ab40 msgr2=0x7f24041f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240406ab40 0x7f24041f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 -- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240406e5d0 msgr2=0x7f24040c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240406e5d0 0x7f24040c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 -- 172.21.15.158:0/982684484 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4009550 con 0x7f240406af20 2023-10-16T09:43:45.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2403fff700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240406e5d0 0x7f24040c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:45.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f24037fe700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240406ab40 0x7f24041f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:45.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2402ffd700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406af20 0x7f24040c7d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00c820 tx=0x7f23ec00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:45.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f23fbfff700 1 -- 172.21.15.158:0/982684484 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec004220 con 0x7f240406af20 2023-10-16T09:43:45.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f23fbfff700 1 -- 172.21.15.158:0/982684484 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec004380 con 0x7f240406af20 2023-10-16T09:43:45.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24041f3420 con 0x7f240406af20 2023-10-16T09:43:45.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.257+0000 7f23fbfff700 1 -- 172.21.15.158:0/982684484 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec0036a0 con 0x7f240406af20 2023-10-16T09:43:45.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.258+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24041f39e0 con 0x7f240406af20 2023-10-16T09:43:45.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.258+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240415f870 con 0x7f240406af20 2023-10-16T09:43:45.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.259+0000 7f23fbfff700 1 -- 172.21.15.158:0/982684484 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f23ec003820 con 0x7f240406af20 2023-10-16T09:43:45.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.259+0000 7f23fbfff700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23e406af00 0x7f23e406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.259+0000 7f23fbfff700 1 -- 172.21.15.158:0/982684484 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f23ec089410 con 0x7f240406af20 2023-10-16T09:43:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.259+0000 7f24037fe700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23e406af00 0x7f23e406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.260+0000 7f24037fe700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23e406af00 0x7f23e406d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009130 tx=0x7f23f4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.264+0000 7f23fbfff700 1 -- 172.21.15.158:0/982684484 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f240415f870 con 0x7f240406af20 2023-10-16T09:43:45.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.452+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2404161db0 con 0x7f23e406af00 2023-10-16T09:43:45.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.462+0000 7f23fbfff700 1 -- 172.21.15.158:0/982684484 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345314 (secure 0 0 0) 0x7f23f4003850 con 0x7f23e406af00 2023-10-16T09:43:45.469 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23e406af00 msgr2=0x7f23e406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23e406af00 0x7f23e406d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009130 tx=0x7f23f4008030 comp rx=0 tx=0).stop 2023-10-16T09:43:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406af20 msgr2=0x7f24040c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406af20 0x7f24040c7d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00c820 tx=0x7f23ec00cbe0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 shutdown_connections 2023-10-16T09:43:45.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240406e5d0 0x7f24040c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f23e406af00 0x7f23e406d3c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240406ab40 0x7f24041f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 --2- 172.21.15.158:0/982684484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240406af20 0x7f24040c7d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 >> 172.21.15.158:0/982684484 conn(0x7f24040d02b0 msgr2=0x7f24040adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:45.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 shutdown_connections 2023-10-16T09:43:45.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.466+0000 7f2409ba3700 1 -- 172.21.15.158:0/982684484 wait complete. 2023-10-16T09:43:45.474 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:43:45.480 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-10-16T09:43:45.223013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3284992,"data_stored":1988256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944843+0000","last_change":"2023-10-16T09:43:21.932095+0000","last_active":"2023-10-16T09:43:25.944843+0000","last_peered":"2023-10-16T09:43:25.944843+0000","last_clean":"2023-10-16T09:43:25.944843+0000","last_became_active":"2023-10-16T09:43:21.931826+0000","last_became_peered":"2023-10-16T09:43:21.931826+0000","last_unstale":"2023-10-16T09:43:25.944843+0000","last_undegraded":"2023-10-16T09:43:25.944843+0000","last_fullsized":"2023-10-16T09:43:25.944843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955709+0000","last_change":"2023-10-16T09:43:21.921656+0000","last_active":"2023-10-16T09:43:25.955709+0000","last_peered":"2023-10-16T09:43:25.955709+0000","last_clean":"2023-10-16T09:43:25.955709+0000","last_became_active":"2023-10-16T09:43:21.921392+0000","last_became_peered":"2023-10-16T09:43:21.921392+0000","last_unstale":"2023-10-16T09:43:25.955709+0000","last_undegraded":"2023-10-16T09:43:25.955709+0000","last_fullsized":"2023-10-16T09:43:25.955709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951037+0000","last_change":"2023-10-16T09:43:21.925692+0000","last_active":"2023-10-16T09:43:25.951037+0000","last_peered":"2023-10-16T09:43:25.951037+0000","last_clean":"2023-10-16T09:43:25.951037+0000","last_became_active":"2023-10-16T09:43:21.925438+0000","last_became_peered":"2023-10-16T09:43:21.925438+0000","last_unstale":"2023-10-16T09:43:25.951037+0000","last_undegraded":"2023-10-16T09:43:25.951037+0000","last_fullsized":"2023-10-16T09:43:25.951037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950435+0000","last_change":"2023-10-16T09:43:21.927209+0000","last_active":"2023-10-16T09:43:25.950435+0000","last_peered":"2023-10-16T09:43:25.950435+0000","last_clean":"2023-10-16T09:43:25.950435+0000","last_became_active":"2023-10-16T09:43:21.926900+0000","last_became_peered":"2023-10-16T09:43:21.926900+0000","last_unstale":"2023-10-16T09:43:25.950435+0000","last_undegraded":"2023-10-16T09:43:25.950435+0000","last_fullsized":"2023-10-16T09:43:25.950435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944943+0000","last_change":"2023-10-16T09:43:21.924582+0000","last_active":"2023-10-16T09:43:25.944943+0000","last_peered":"2023-10-16T09:43:25.944943+0000","last_clean":"2023-10-16T09:43:25.944943+0000","last_became_active":"2023-10-16T09:43:21.924329+0000","last_became_peered":"2023-10-16T09:43:21.924329+0000","last_unstale":"2023-10-16T09:43:25.944943+0000","last_undegraded":"2023-10-16T09:43:25.944943+0000","last_fullsized":"2023-10-16T09:43:25.944943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950668+0000","last_change":"2023-10-16T09:43:21.937326+0000","last_active":"2023-10-16T09:43:25.950668+0000","last_peered":"2023-10-16T09:43:25.950668+0000","last_clean":"2023-10-16T09:43:25.950668+0000","last_became_active":"2023-10-16T09:43:21.936665+0000","last_became_peered":"2023-10-16T09:43:21.936665+0000","last_unstale":"2023-10-16T09:43:25.950668+0000","last_undegraded":"2023-10-16T09:43:25.950668+0000","last_fullsized":"2023-10-16T09:43:25.950668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:21.923235+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:21.923021+0000","last_became_peered":"2023-10-16T09:43:21.923021+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965916+0000","last_change":"2023-10-16T09:43:21.933111+0000","last_active":"2023-10-16T09:43:25.965916+0000","last_peered":"2023-10-16T09:43:25.965916+0000","last_clean":"2023-10-16T09:43:25.965916+0000","last_became_active":"2023-10-16T09:43:21.932835+0000","last_became_peered":"2023-10-16T09:43:21.932835+0000","last_unstale":"2023-10-16T09:43:25.965916+0000","last_undegraded":"2023-10-16T09:43:25.965916+0000","last_fullsized":"2023-10-16T09:43:25.965916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944721+0000","last_change":"2023-10-16T09:43:21.940444+0000","last_active":"2023-10-16T09:43:25.944721+0000","last_peered":"2023-10-16T09:43:25.944721+0000","last_clean":"2023-10-16T09:43:25.944721+0000","last_became_active":"2023-10-16T09:43:21.939823+0000","last_became_peered":"2023-10-16T09:43:21.939823+0000","last_unstale":"2023-10-16T09:43:25.944721+0000","last_undegraded":"2023-10-16T09:43:25.944721+0000","last_fullsized":"2023-10-16T09:43:25.944721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956248+0000","last_change":"2023-10-16T09:43:21.934573+0000","last_active":"2023-10-16T09:43:25.956248+0000","last_peered":"2023-10-16T09:43:25.956248+0000","last_clean":"2023-10-16T09:43:25.956248+0000","last_became_active":"2023-10-16T09:43:21.934285+0000","last_became_peered":"2023-10-16T09:43:21.934285+0000","last_unstale":"2023-10-16T09:43:25.956248+0000","last_undegraded":"2023-10-16T09:43:25.956248+0000","last_fullsized":"2023-10-16T09:43:25.956248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953996+0000","last_change":"2023-10-16T09:43:21.934669+0000","last_active":"2023-10-16T09:43:25.953996+0000","last_peered":"2023-10-16T09:43:25.953996+0000","last_clean":"2023-10-16T09:43:25.953996+0000","last_became_active":"2023-10-16T09:43:21.933918+0000","last_became_peered":"2023-10-16T09:43:21.933918+0000","last_unstale":"2023-10-16T09:43:25.953996+0000","last_undegraded":"2023-10-16T09:43:25.953996+0000","last_fullsized":"2023-10-16T09:43:25.953996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950011+0000","last_change":"2023-10-16T09:43:21.934285+0000","last_active":"2023-10-16T09:43:25.950011+0000","last_peered":"2023-10-16T09:43:25.950011+0000","last_clean":"2023-10-16T09:43:25.950011+0000","last_became_active":"2023-10-16T09:43:21.933516+0000","last_became_peered":"2023-10-16T09:43:21.933516+0000","last_unstale":"2023-10-16T09:43:25.950011+0000","last_undegraded":"2023-10-16T09:43:25.950011+0000","last_fullsized":"2023-10-16T09:43:25.950011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:56.814327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965961+0000","last_change":"2023-10-16T09:43:21.934162+0000","last_active":"2023-10-16T09:43:25.965961+0000","last_peered":"2023-10-16T09:43:25.965961+0000","last_clean":"2023-10-16T09:43:25.965961+0000","last_became_active":"2023-10-16T09:43:21.933898+0000","last_became_peered":"2023-10-16T09:43:21.933898+0000","last_unstale":"2023-10-16T09:43:25.965961+0000","last_undegraded":"2023-10-16T09:43:25.965961+0000","last_fullsized":"2023-10-16T09:43:25.965961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948340+0000","last_change":"2023-10-16T09:43:21.923364+0000","last_active":"2023-10-16T09:43:25.948340+0000","last_peered":"2023-10-16T09:43:25.948340+0000","last_clean":"2023-10-16T09:43:25.948340+0000","last_became_active":"2023-10-16T09:43:21.923123+0000","last_became_peered":"2023-10-16T09:43:21.923123+0000","last_unstale":"2023-10-16T09:43:25.948340+0000","last_undegraded":"2023-10-16T09:43:25.948340+0000","last_fullsized":"2023-10-16T09:43:25.948340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953943+0000","last_change":"2023-10-16T09:43:21.925350+0000","last_active":"2023-10-16T09:43:25.953943+0000","last_peered":"2023-10-16T09:43:25.953943+0000","last_clean":"2023-10-16T09:43:25.953943+0000","last_became_active":"2023-10-16T09:43:21.925107+0000","last_became_peered":"2023-10-16T09:43:21.925107+0000","last_unstale":"2023-10-16T09:43:25.953943+0000","last_undegraded":"2023-10-16T09:43:25.953943+0000","last_fullsized":"2023-10-16T09:43:25.953943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951212+0000","last_change":"2023-10-16T09:43:21.927864+0000","last_active":"2023-10-16T09:43:25.951212+0000","last_peered":"2023-10-16T09:43:25.951212+0000","last_clean":"2023-10-16T09:43:25.951212+0000","last_became_active":"2023-10-16T09:43:21.927411+0000","last_became_peered":"2023-10-16T09:43:21.927411+0000","last_unstale":"2023-10-16T09:43:25.951212+0000","last_undegraded":"2023-10-16T09:43:25.951212+0000","last_fullsized":"2023-10-16T09:43:25.951212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950524+0000","last_change":"2023-10-16T09:43:21.936884+0000","last_active":"2023-10-16T09:43:25.950524+0000","last_peered":"2023-10-16T09:43:25.950524+0000","last_clean":"2023-10-16T09:43:25.950524+0000","last_became_active":"2023-10-16T09:43:21.936569+0000","last_became_peered":"2023-10-16T09:43:21.936569+0000","last_unstale":"2023-10-16T09:43:25.950524+0000","last_undegraded":"2023-10-16T09:43:25.950524+0000","last_fullsized":"2023-10-16T09:43:25.950524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948848+0000","last_change":"2023-10-16T09:43:21.935147+0000","last_active":"2023-10-16T09:43:25.948848+0000","last_peered":"2023-10-16T09:43:25.948848+0000","last_clean":"2023-10-16T09:43:25.948848+0000","last_became_active":"2023-10-16T09:43:21.934786+0000","last_became_peered":"2023-10-16T09:43:21.934786+0000","last_unstale":"2023-10-16T09:43:25.948848+0000","last_undegraded":"2023-10-16T09:43:25.948848+0000","last_fullsized":"2023-10-16T09:43:25.948848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945003+0000","last_change":"2023-10-16T09:43:21.927238+0000","last_active":"2023-10-16T09:43:25.945003+0000","last_peered":"2023-10-16T09:43:25.945003+0000","last_clean":"2023-10-16T09:43:25.945003+0000","last_became_active":"2023-10-16T09:43:21.926944+0000","last_became_peered":"2023-10-16T09:43:21.926944+0000","last_unstale":"2023-10-16T09:43:25.945003+0000","last_undegraded":"2023-10-16T09:43:25.945003+0000","last_fullsized":"2023-10-16T09:43:25.945003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015597+0000","last_change":"2023-10-16T09:43:25.972270+0000","last_active":"2023-10-16T09:43:28.015597+0000","last_peered":"2023-10-16T09:43:28.015597+0000","last_clean":"2023-10-16T09:43:28.015597+0000","last_became_active":"2023-10-16T09:43:25.971856+0000","last_became_peered":"2023-10-16T09:43:25.971856+0000","last_unstale":"2023-10-16T09:43:28.015597+0000","last_undegraded":"2023-10-16T09:43:28.015597+0000","last_fullsized":"2023-10-16T09:43:28.015597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:00.427859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951697+0000","last_change":"2023-10-16T09:43:20.934516+0000","last_active":"2023-10-16T09:43:25.951697+0000","last_peered":"2023-10-16T09:43:25.951697+0000","last_clean":"2023-10-16T09:43:25.951697+0000","last_became_active":"2023-10-16T09:43:20.934196+0000","last_became_peered":"2023-10-16T09:43:20.934196+0000","last_unstale":"2023-10-16T09:43:25.951697+0000","last_undegraded":"2023-10-16T09:43:25.951697+0000","last_fullsized":"2023-10-16T09:43:25.951697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966598+0000","last_change":"2023-10-16T09:43:21.941097+0000","last_active":"2023-10-16T09:43:25.966598+0000","last_peered":"2023-10-16T09:43:25.966598+0000","last_clean":"2023-10-16T09:43:25.966598+0000","last_became_active":"2023-10-16T09:43:21.940820+0000","last_became_peered":"2023-10-16T09:43:21.940820+0000","last_unstale":"2023-10-16T09:43:25.966598+0000","last_undegraded":"2023-10-16T09:43:25.966598+0000","last_fullsized":"2023-10-16T09:43:25.966598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947760+0000","last_change":"2023-10-16T09:43:25.970409+0000","last_active":"2023-10-16T09:43:26.947760+0000","last_peered":"2023-10-16T09:43:26.947760+0000","last_clean":"2023-10-16T09:43:26.947760+0000","last_became_active":"2023-10-16T09:43:25.969848+0000","last_became_peered":"2023-10-16T09:43:25.969848+0000","last_unstale":"2023-10-16T09:43:26.947760+0000","last_undegraded":"2023-10-16T09:43:26.947760+0000","last_fullsized":"2023-10-16T09:43:26.947760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:00.972576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966712+0000","last_change":"2023-10-16T09:43:20.933965+0000","last_active":"2023-10-16T09:43:25.966712+0000","last_peered":"2023-10-16T09:43:25.966712+0000","last_clean":"2023-10-16T09:43:25.966712+0000","last_became_active":"2023-10-16T09:43:20.933635+0000","last_became_peered":"2023-10-16T09:43:20.933635+0000","last_unstale":"2023-10-16T09:43:25.966712+0000","last_undegraded":"2023-10-16T09:43:25.966712+0000","last_fullsized":"2023-10-16T09:43:25.966712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951601+0000","last_change":"2023-10-16T09:43:21.937374+0000","last_active":"2023-10-16T09:43:25.951601+0000","last_peered":"2023-10-16T09:43:25.951601+0000","last_clean":"2023-10-16T09:43:25.951601+0000","last_became_active":"2023-10-16T09:43:21.936989+0000","last_became_peered":"2023-10-16T09:43:21.936989+0000","last_unstale":"2023-10-16T09:43:25.951601+0000","last_undegraded":"2023-10-16T09:43:25.951601+0000","last_fullsized":"2023-10-16T09:43:25.951601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948251+0000","last_change":"2023-10-16T09:43:25.970011+0000","last_active":"2023-10-16T09:43:26.948251+0000","last_peered":"2023-10-16T09:43:26.948251+0000","last_clean":"2023-10-16T09:43:26.948251+0000","last_became_active":"2023-10-16T09:43:25.969570+0000","last_became_peered":"2023-10-16T09:43:25.969570+0000","last_unstale":"2023-10-16T09:43:26.948251+0000","last_undegraded":"2023-10-16T09:43:26.948251+0000","last_fullsized":"2023-10-16T09:43:26.948251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944734+0000","last_change":"2023-10-16T09:43:20.928578+0000","last_active":"2023-10-16T09:43:25.944734+0000","last_peered":"2023-10-16T09:43:25.944734+0000","last_clean":"2023-10-16T09:43:25.944734+0000","last_became_active":"2023-10-16T09:43:20.928283+0000","last_became_peered":"2023-10-16T09:43:20.928283+0000","last_unstale":"2023-10-16T09:43:25.944734+0000","last_undegraded":"2023-10-16T09:43:25.944734+0000","last_fullsized":"2023-10-16T09:43:25.944734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950360+0000","last_change":"2023-10-16T09:43:21.936827+0000","last_active":"2023-10-16T09:43:25.950360+0000","last_peered":"2023-10-16T09:43:25.950360+0000","last_clean":"2023-10-16T09:43:25.950360+0000","last_became_active":"2023-10-16T09:43:21.936545+0000","last_became_peered":"2023-10-16T09:43:21.936545+0000","last_unstale":"2023-10-16T09:43:25.950360+0000","last_undegraded":"2023-10-16T09:43:25.950360+0000","last_fullsized":"2023-10-16T09:43:25.950360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947909+0000","last_change":"2023-10-16T09:43:25.968566+0000","last_active":"2023-10-16T09:43:26.947909+0000","last_peered":"2023-10-16T09:43:26.947909+0000","last_clean":"2023-10-16T09:43:26.947909+0000","last_became_active":"2023-10-16T09:43:25.968188+0000","last_became_peered":"2023-10-16T09:43:25.968188+0000","last_unstale":"2023-10-16T09:43:26.947909+0000","last_undegraded":"2023-10-16T09:43:26.947909+0000","last_fullsized":"2023-10-16T09:43:26.947909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:38.074113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944439+0000","last_change":"2023-10-16T09:43:20.929934+0000","last_active":"2023-10-16T09:43:25.944439+0000","last_peered":"2023-10-16T09:43:25.944439+0000","last_clean":"2023-10-16T09:43:25.944439+0000","last_became_active":"2023-10-16T09:43:20.929639+0000","last_became_peered":"2023-10-16T09:43:20.929639+0000","last_unstale":"2023-10-16T09:43:25.944439+0000","last_undegraded":"2023-10-16T09:43:25.944439+0000","last_fullsized":"2023-10-16T09:43:25.944439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953109+0000","last_change":"2023-10-16T09:43:21.933199+0000","last_active":"2023-10-16T09:43:25.953109+0000","last_peered":"2023-10-16T09:43:25.953109+0000","last_clean":"2023-10-16T09:43:25.953109+0000","last_became_active":"2023-10-16T09:43:21.932906+0000","last_became_peered":"2023-10-16T09:43:21.932906+0000","last_unstale":"2023-10-16T09:43:25.953109+0000","last_undegraded":"2023-10-16T09:43:25.953109+0000","last_fullsized":"2023-10-16T09:43:25.953109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016130+0000","last_change":"2023-10-16T09:43:25.975356+0000","last_active":"2023-10-16T09:43:28.016130+0000","last_peered":"2023-10-16T09:43:28.016130+0000","last_clean":"2023-10-16T09:43:28.016130+0000","last_became_active":"2023-10-16T09:43:25.974334+0000","last_became_peered":"2023-10-16T09:43:25.974334+0000","last_unstale":"2023-10-16T09:43:28.016130+0000","last_undegraded":"2023-10-16T09:43:28.016130+0000","last_fullsized":"2023-10-16T09:43:28.016130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:20.175649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966133+0000","last_change":"2023-10-16T09:43:20.928498+0000","last_active":"2023-10-16T09:43:25.966133+0000","last_peered":"2023-10-16T09:43:25.966133+0000","last_clean":"2023-10-16T09:43:25.966133+0000","last_became_active":"2023-10-16T09:43:20.928242+0000","last_became_peered":"2023-10-16T09:43:20.928242+0000","last_unstale":"2023-10-16T09:43:25.966133+0000","last_undegraded":"2023-10-16T09:43:25.966133+0000","last_fullsized":"2023-10-16T09:43:25.966133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948266+0000","last_change":"2023-10-16T09:43:21.919769+0000","last_active":"2023-10-16T09:43:25.948266+0000","last_peered":"2023-10-16T09:43:25.948266+0000","last_clean":"2023-10-16T09:43:25.948266+0000","last_became_active":"2023-10-16T09:43:21.919395+0000","last_became_peered":"2023-10-16T09:43:21.919395+0000","last_unstale":"2023-10-16T09:43:25.948266+0000","last_undegraded":"2023-10-16T09:43:25.948266+0000","last_fullsized":"2023-10-16T09:43:25.948266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:06.700888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.944846+0000","last_change":"2023-10-16T09:43:25.970755+0000","last_active":"2023-10-16T09:43:26.944846+0000","last_peered":"2023-10-16T09:43:26.944846+0000","last_clean":"2023-10-16T09:43:26.944846+0000","last_became_active":"2023-10-16T09:43:25.970344+0000","last_became_peered":"2023-10-16T09:43:25.970344+0000","last_unstale":"2023-10-16T09:43:26.944846+0000","last_undegraded":"2023-10-16T09:43:26.944846+0000","last_fullsized":"2023-10-16T09:43:26.944846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:28.937851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957011+0000","last_change":"2023-10-16T09:43:20.924380+0000","last_active":"2023-10-16T09:43:25.957011+0000","last_peered":"2023-10-16T09:43:25.957011+0000","last_clean":"2023-10-16T09:43:25.957011+0000","last_became_active":"2023-10-16T09:43:20.924104+0000","last_became_peered":"2023-10-16T09:43:20.924104+0000","last_unstale":"2023-10-16T09:43:25.957011+0000","last_undegraded":"2023-10-16T09:43:25.957011+0000","last_fullsized":"2023-10-16T09:43:25.957011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948179+0000","last_change":"2023-10-16T09:43:21.929870+0000","last_active":"2023-10-16T09:43:25.948179+0000","last_peered":"2023-10-16T09:43:25.948179+0000","last_clean":"2023-10-16T09:43:25.948179+0000","last_became_active":"2023-10-16T09:43:21.929589+0000","last_became_peered":"2023-10-16T09:43:21.929589+0000","last_unstale":"2023-10-16T09:43:25.948179+0000","last_undegraded":"2023-10-16T09:43:25.948179+0000","last_fullsized":"2023-10-16T09:43:25.948179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949711+0000","last_change":"2023-10-16T09:43:25.966670+0000","last_active":"2023-10-16T09:43:26.949711+0000","last_peered":"2023-10-16T09:43:26.949711+0000","last_clean":"2023-10-16T09:43:26.949711+0000","last_became_active":"2023-10-16T09:43:25.966312+0000","last_became_peered":"2023-10-16T09:43:25.966312+0000","last_unstale":"2023-10-16T09:43:26.949711+0000","last_undegraded":"2023-10-16T09:43:26.949711+0000","last_fullsized":"2023-10-16T09:43:26.949711+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965835+0000","last_change":"2023-10-16T09:43:20.928383+0000","last_active":"2023-10-16T09:43:25.965835+0000","last_peered":"2023-10-16T09:43:25.965835+0000","last_clean":"2023-10-16T09:43:25.965835+0000","last_became_active":"2023-10-16T09:43:20.928096+0000","last_became_peered":"2023-10-16T09:43:20.928096+0000","last_unstale":"2023-10-16T09:43:25.965835+0000","last_undegraded":"2023-10-16T09:43:25.965835+0000","last_fullsized":"2023-10-16T09:43:25.965835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965672+0000","last_change":"2023-10-16T09:43:21.921213+0000","last_active":"2023-10-16T09:43:25.965672+0000","last_peered":"2023-10-16T09:43:25.965672+0000","last_clean":"2023-10-16T09:43:25.965672+0000","last_became_active":"2023-10-16T09:43:21.920920+0000","last_became_peered":"2023-10-16T09:43:21.920920+0000","last_unstale":"2023-10-16T09:43:25.965672+0000","last_undegraded":"2023-10-16T09:43:25.965672+0000","last_fullsized":"2023-10-16T09:43:25.965672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015970+0000","last_change":"2023-10-16T09:43:25.972702+0000","last_active":"2023-10-16T09:43:28.015970+0000","last_peered":"2023-10-16T09:43:28.015970+0000","last_clean":"2023-10-16T09:43:28.015970+0000","last_became_active":"2023-10-16T09:43:25.971784+0000","last_became_peered":"2023-10-16T09:43:25.971784+0000","last_unstale":"2023-10-16T09:43:28.015970+0000","last_undegraded":"2023-10-16T09:43:28.015970+0000","last_fullsized":"2023-10-16T09:43:28.015970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950235+0000","last_change":"2023-10-16T09:43:20.926495+0000","last_active":"2023-10-16T09:43:25.950235+0000","last_peered":"2023-10-16T09:43:25.950235+0000","last_clean":"2023-10-16T09:43:25.950235+0000","last_became_active":"2023-10-16T09:43:20.925998+0000","last_became_peered":"2023-10-16T09:43:20.925998+0000","last_unstale":"2023-10-16T09:43:25.950235+0000","last_undegraded":"2023-10-16T09:43:25.950235+0000","last_fullsized":"2023-10-16T09:43:25.950235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954132+0000","last_change":"2023-10-16T09:43:21.919652+0000","last_active":"2023-10-16T09:43:25.954132+0000","last_peered":"2023-10-16T09:43:25.954132+0000","last_clean":"2023-10-16T09:43:25.954132+0000","last_became_active":"2023-10-16T09:43:21.919397+0000","last_became_peered":"2023-10-16T09:43:21.919397+0000","last_unstale":"2023-10-16T09:43:25.954132+0000","last_undegraded":"2023-10-16T09:43:25.954132+0000","last_fullsized":"2023-10-16T09:43:25.954132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015120+0000","last_change":"2023-10-16T09:43:25.975457+0000","last_active":"2023-10-16T09:43:28.015120+0000","last_peered":"2023-10-16T09:43:28.015120+0000","last_clean":"2023-10-16T09:43:28.015120+0000","last_became_active":"2023-10-16T09:43:25.975098+0000","last_became_peered":"2023-10-16T09:43:25.975098+0000","last_unstale":"2023-10-16T09:43:28.015120+0000","last_undegraded":"2023-10-16T09:43:28.015120+0000","last_fullsized":"2023-10-16T09:43:28.015120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956270+0000","last_change":"2023-10-16T09:43:20.925087+0000","last_active":"2023-10-16T09:43:25.956270+0000","last_peered":"2023-10-16T09:43:25.956270+0000","last_clean":"2023-10-16T09:43:25.956270+0000","last_became_active":"2023-10-16T09:43:20.924798+0000","last_became_peered":"2023-10-16T09:43:20.924798+0000","last_unstale":"2023-10-16T09:43:25.956270+0000","last_undegraded":"2023-10-16T09:43:25.956270+0000","last_fullsized":"2023-10-16T09:43:25.956270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951125+0000","last_change":"2023-10-16T09:43:21.933170+0000","last_active":"2023-10-16T09:43:25.951125+0000","last_peered":"2023-10-16T09:43:25.951125+0000","last_clean":"2023-10-16T09:43:25.951125+0000","last_became_active":"2023-10-16T09:43:21.932876+0000","last_became_peered":"2023-10-16T09:43:21.932876+0000","last_unstale":"2023-10-16T09:43:25.951125+0000","last_undegraded":"2023-10-16T09:43:25.951125+0000","last_fullsized":"2023-10-16T09:43:25.951125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015261+0000","last_change":"2023-10-16T09:43:25.979174+0000","last_active":"2023-10-16T09:43:28.015261+0000","last_peered":"2023-10-16T09:43:28.015261+0000","last_clean":"2023-10-16T09:43:28.015261+0000","last_became_active":"2023-10-16T09:43:25.977030+0000","last_became_peered":"2023-10-16T09:43:25.977030+0000","last_unstale":"2023-10-16T09:43:28.015261+0000","last_undegraded":"2023-10-16T09:43:28.015261+0000","last_fullsized":"2023-10-16T09:43:28.015261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:47.151340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951034+0000","last_change":"2023-10-16T09:43:20.928828+0000","last_active":"2023-10-16T09:43:25.951034+0000","last_peered":"2023-10-16T09:43:25.951034+0000","last_clean":"2023-10-16T09:43:25.951034+0000","last_became_active":"2023-10-16T09:43:20.928550+0000","last_became_peered":"2023-10-16T09:43:20.928550+0000","last_unstale":"2023-10-16T09:43:25.951034+0000","last_undegraded":"2023-10-16T09:43:25.951034+0000","last_fullsized":"2023-10-16T09:43:25.951034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951780+0000","last_change":"2023-10-16T09:43:21.937614+0000","last_active":"2023-10-16T09:43:25.951780+0000","last_peered":"2023-10-16T09:43:25.951780+0000","last_clean":"2023-10-16T09:43:25.951780+0000","last_became_active":"2023-10-16T09:43:21.937335+0000","last_became_peered":"2023-10-16T09:43:21.937335+0000","last_unstale":"2023-10-16T09:43:25.951780+0000","last_undegraded":"2023-10-16T09:43:25.951780+0000","last_fullsized":"2023-10-16T09:43:25.951780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948709+0000","last_change":"2023-10-16T09:43:25.958049+0000","last_active":"2023-10-16T09:43:26.948709+0000","last_peered":"2023-10-16T09:43:26.948709+0000","last_clean":"2023-10-16T09:43:26.948709+0000","last_became_active":"2023-10-16T09:43:25.957672+0000","last_became_peered":"2023-10-16T09:43:25.957672+0000","last_unstale":"2023-10-16T09:43:26.948709+0000","last_undegraded":"2023-10-16T09:43:26.948709+0000","last_fullsized":"2023-10-16T09:43:26.948709+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:53.105902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955629+0000","last_change":"2023-10-16T09:43:20.926834+0000","last_active":"2023-10-16T09:43:25.955629+0000","last_peered":"2023-10-16T09:43:25.955629+0000","last_clean":"2023-10-16T09:43:25.955629+0000","last_became_active":"2023-10-16T09:43:20.926593+0000","last_became_peered":"2023-10-16T09:43:20.926593+0000","last_unstale":"2023-10-16T09:43:25.955629+0000","last_undegraded":"2023-10-16T09:43:25.955629+0000","last_fullsized":"2023-10-16T09:43:25.955629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947243+0000","last_change":"2023-10-16T09:43:21.925181+0000","last_active":"2023-10-16T09:43:25.947243+0000","last_peered":"2023-10-16T09:43:25.947243+0000","last_clean":"2023-10-16T09:43:25.947243+0000","last_became_active":"2023-10-16T09:43:21.924083+0000","last_became_peered":"2023-10-16T09:43:21.924083+0000","last_unstale":"2023-10-16T09:43:25.947243+0000","last_undegraded":"2023-10-16T09:43:25.947243+0000","last_fullsized":"2023-10-16T09:43:25.947243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:41.369421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995287+0000","last_change":"2023-10-16T09:43:25.960604+0000","last_active":"2023-10-16T09:43:27.995287+0000","last_peered":"2023-10-16T09:43:27.995287+0000","last_clean":"2023-10-16T09:43:27.995287+0000","last_became_active":"2023-10-16T09:43:25.960212+0000","last_became_peered":"2023-10-16T09:43:25.960212+0000","last_unstale":"2023-10-16T09:43:27.995287+0000","last_undegraded":"2023-10-16T09:43:27.995287+0000","last_fullsized":"2023-10-16T09:43:27.995287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:31.688570+0000","last_change":"2023-10-16T09:43:20.927651+0000","last_active":"2023-10-16T09:43:31.688570+0000","last_peered":"2023-10-16T09:43:31.688570+0000","last_clean":"2023-10-16T09:43:31.688570+0000","last_became_active":"2023-10-16T09:43:20.927393+0000","last_became_peered":"2023-10-16T09:43:20.927393+0000","last_unstale":"2023-10-16T09:43:31.688570+0000","last_undegraded":"2023-10-16T09:43:31.688570+0000","last_fullsized":"2023-10-16T09:43:31.688570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944933+0000","last_change":"2023-10-16T09:43:21.922547+0000","last_active":"2023-10-16T09:43:25.944933+0000","last_peered":"2023-10-16T09:43:25.944933+0000","last_clean":"2023-10-16T09:43:25.944933+0000","last_became_active":"2023-10-16T09:43:21.922309+0000","last_became_peered":"2023-10-16T09:43:21.922309+0000","last_unstale":"2023-10-16T09:43:25.944933+0000","last_undegraded":"2023-10-16T09:43:25.944933+0000","last_fullsized":"2023-10-16T09:43:25.944933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016142+0000","last_change":"2023-10-16T09:43:25.973055+0000","last_active":"2023-10-16T09:43:28.016142+0000","last_peered":"2023-10-16T09:43:28.016142+0000","last_clean":"2023-10-16T09:43:28.016142+0000","last_became_active":"2023-10-16T09:43:25.972664+0000","last_became_peered":"2023-10-16T09:43:25.972664+0000","last_unstale":"2023-10-16T09:43:28.016142+0000","last_undegraded":"2023-10-16T09:43:28.016142+0000","last_fullsized":"2023-10-16T09:43:28.016142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948288+0000","last_change":"2023-10-16T09:43:20.933659+0000","last_active":"2023-10-16T09:43:25.948288+0000","last_peered":"2023-10-16T09:43:25.948288+0000","last_clean":"2023-10-16T09:43:25.948288+0000","last_became_active":"2023-10-16T09:43:20.933376+0000","last_became_peered":"2023-10-16T09:43:20.933376+0000","last_unstale":"2023-10-16T09:43:25.948288+0000","last_undegraded":"2023-10-16T09:43:25.948288+0000","last_fullsized":"2023-10-16T09:43:25.948288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948438+0000","last_change":"2023-10-16T09:43:21.925525+0000","last_active":"2023-10-16T09:43:25.948438+0000","last_peered":"2023-10-16T09:43:25.948438+0000","last_clean":"2023-10-16T09:43:25.948438+0000","last_became_active":"2023-10-16T09:43:21.925269+0000","last_became_peered":"2023-10-16T09:43:21.925269+0000","last_unstale":"2023-10-16T09:43:25.948438+0000","last_undegraded":"2023-10-16T09:43:25.948438+0000","last_fullsized":"2023-10-16T09:43:25.948438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:39.205632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016287+0000","last_change":"2023-10-16T09:43:25.974917+0000","last_active":"2023-10-16T09:43:28.016287+0000","last_peered":"2023-10-16T09:43:28.016287+0000","last_clean":"2023-10-16T09:43:28.016287+0000","last_became_active":"2023-10-16T09:43:25.974512+0000","last_became_peered":"2023-10-16T09:43:25.974512+0000","last_unstale":"2023-10-16T09:43:28.016287+0000","last_undegraded":"2023-10-16T09:43:28.016287+0000","last_fullsized":"2023-10-16T09:43:28.016287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:24.186922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950390+0000","last_change":"2023-10-16T09:43:20.927369+0000","last_active":"2023-10-16T09:43:25.950390+0000","last_peered":"2023-10-16T09:43:25.950390+0000","last_clean":"2023-10-16T09:43:25.950390+0000","last_became_active":"2023-10-16T09:43:20.927043+0000","last_became_peered":"2023-10-16T09:43:20.927043+0000","last_unstale":"2023-10-16T09:43:25.950390+0000","last_undegraded":"2023-10-16T09:43:25.950390+0000","last_fullsized":"2023-10-16T09:43:25.950390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:11.929994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945029+0000","last_change":"2023-10-16T09:43:21.930235+0000","last_active":"2023-10-16T09:43:25.945029+0000","last_peered":"2023-10-16T09:43:25.945029+0000","last_clean":"2023-10-16T09:43:25.945029+0000","last_became_active":"2023-10-16T09:43:21.929925+0000","last_became_peered":"2023-10-16T09:43:21.929925+0000","last_unstale":"2023-10-16T09:43:25.945029+0000","last_undegraded":"2023-10-16T09:43:25.945029+0000","last_fullsized":"2023-10-16T09:43:25.945029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016497+0000","last_change":"2023-10-16T09:43:25.975868+0000","last_active":"2023-10-16T09:43:28.016497+0000","last_peered":"2023-10-16T09:43:28.016497+0000","last_clean":"2023-10-16T09:43:28.016497+0000","last_became_active":"2023-10-16T09:43:25.975476+0000","last_became_peered":"2023-10-16T09:43:25.975476+0000","last_unstale":"2023-10-16T09:43:28.016497+0000","last_undegraded":"2023-10-16T09:43:28.016497+0000","last_fullsized":"2023-10-16T09:43:28.016497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:36.344621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954142+0000","last_change":"2023-10-16T09:43:20.922948+0000","last_active":"2023-10-16T09:43:25.954142+0000","last_peered":"2023-10-16T09:43:25.954142+0000","last_clean":"2023-10-16T09:43:25.954142+0000","last_became_active":"2023-10-16T09:43:20.922660+0000","last_became_peered":"2023-10-16T09:43:20.922660+0000","last_unstale":"2023-10-16T09:43:25.954142+0000","last_undegraded":"2023-10-16T09:43:25.954142+0000","last_fullsized":"2023-10-16T09:43:25.954142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:24.786762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953944+0000","last_change":"2023-10-16T09:43:21.930450+0000","last_active":"2023-10-16T09:43:25.953944+0000","last_peered":"2023-10-16T09:43:25.953944+0000","last_clean":"2023-10-16T09:43:25.953944+0000","last_became_active":"2023-10-16T09:43:21.929848+0000","last_became_peered":"2023-10-16T09:43:21.929848+0000","last_unstale":"2023-10-16T09:43:25.953944+0000","last_undegraded":"2023-10-16T09:43:25.953944+0000","last_fullsized":"2023-10-16T09:43:25.953944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015695+0000","last_change":"2023-10-16T09:43:25.972951+0000","last_active":"2023-10-16T09:43:28.015695+0000","last_peered":"2023-10-16T09:43:28.015695+0000","last_clean":"2023-10-16T09:43:28.015695+0000","last_became_active":"2023-10-16T09:43:25.972321+0000","last_became_peered":"2023-10-16T09:43:25.972321+0000","last_unstale":"2023-10-16T09:43:28.015695+0000","last_undegraded":"2023-10-16T09:43:28.015695+0000","last_fullsized":"2023-10-16T09:43:28.015695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948561+0000","last_change":"2023-10-16T09:43:20.919381+0000","last_active":"2023-10-16T09:43:25.948561+0000","last_peered":"2023-10-16T09:43:25.948561+0000","last_clean":"2023-10-16T09:43:25.948561+0000","last_became_active":"2023-10-16T09:43:20.919099+0000","last_became_peered":"2023-10-16T09:43:20.919099+0000","last_unstale":"2023-10-16T09:43:25.948561+0000","last_undegraded":"2023-10-16T09:43:25.948561+0000","last_fullsized":"2023-10-16T09:43:25.948561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.153559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944510+0000","last_change":"2023-10-16T09:43:21.930254+0000","last_active":"2023-10-16T09:43:25.944510+0000","last_peered":"2023-10-16T09:43:25.944510+0000","last_clean":"2023-10-16T09:43:25.944510+0000","last_became_active":"2023-10-16T09:43:21.929903+0000","last_became_peered":"2023-10-16T09:43:21.929903+0000","last_unstale":"2023-10-16T09:43:25.944510+0000","last_undegraded":"2023-10-16T09:43:25.944510+0000","last_fullsized":"2023-10-16T09:43:25.944510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947704+0000","last_change":"2023-10-16T09:43:25.973595+0000","last_active":"2023-10-16T09:43:26.947704+0000","last_peered":"2023-10-16T09:43:26.947704+0000","last_clean":"2023-10-16T09:43:26.947704+0000","last_became_active":"2023-10-16T09:43:25.972877+0000","last_became_peered":"2023-10-16T09:43:25.972877+0000","last_unstale":"2023-10-16T09:43:26.947704+0000","last_undegraded":"2023-10-16T09:43:26.947704+0000","last_fullsized":"2023-10-16T09:43:26.947704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:59.042186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.946744+0000","last_change":"2023-10-16T09:43:20.933953+0000","last_active":"2023-10-16T09:43:25.946744+0000","last_peered":"2023-10-16T09:43:25.946744+0000","last_clean":"2023-10-16T09:43:25.946744+0000","last_became_active":"2023-10-16T09:43:20.933628+0000","last_became_peered":"2023-10-16T09:43:20.933628+0000","last_unstale":"2023-10-16T09:43:25.946744+0000","last_undegraded":"2023-10-16T09:43:25.946744+0000","last_fullsized":"2023-10-16T09:43:25.946744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:06.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944812+0000","last_change":"2023-10-16T09:43:21.922710+0000","last_active":"2023-10-16T09:43:25.944812+0000","last_peered":"2023-10-16T09:43:25.944812+0000","last_clean":"2023-10-16T09:43:25.944812+0000","last_became_active":"2023-10-16T09:43:21.921824+0000","last_became_peered":"2023-10-16T09:43:21.921824+0000","last_unstale":"2023-10-16T09:43:25.944812+0000","last_undegraded":"2023-10-16T09:43:25.944812+0000","last_fullsized":"2023-10-16T09:43:25.944812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995171+0000","last_change":"2023-10-16T09:43:25.960646+0000","last_active":"2023-10-16T09:43:27.995171+0000","last_peered":"2023-10-16T09:43:27.995171+0000","last_clean":"2023-10-16T09:43:27.995171+0000","last_became_active":"2023-10-16T09:43:25.960270+0000","last_became_peered":"2023-10-16T09:43:25.960270+0000","last_unstale":"2023-10-16T09:43:27.995171+0000","last_undegraded":"2023-10-16T09:43:27.995171+0000","last_fullsized":"2023-10-16T09:43:27.995171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955611+0000","last_change":"2023-10-16T09:43:20.935197+0000","last_active":"2023-10-16T09:43:25.955611+0000","last_peered":"2023-10-16T09:43:25.955611+0000","last_clean":"2023-10-16T09:43:25.955611+0000","last_became_active":"2023-10-16T09:43:20.934929+0000","last_became_peered":"2023-10-16T09:43:20.934929+0000","last_unstale":"2023-10-16T09:43:25.955611+0000","last_undegraded":"2023-10-16T09:43:25.955611+0000","last_fullsized":"2023-10-16T09:43:25.955611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953179+0000","last_change":"2023-10-16T09:43:21.936959+0000","last_active":"2023-10-16T09:43:25.953179+0000","last_peered":"2023-10-16T09:43:25.953179+0000","last_clean":"2023-10-16T09:43:25.953179+0000","last_became_active":"2023-10-16T09:43:21.932598+0000","last_became_peered":"2023-10-16T09:43:21.932598+0000","last_unstale":"2023-10-16T09:43:25.953179+0000","last_undegraded":"2023-10-16T09:43:25.953179+0000","last_fullsized":"2023-10-16T09:43:25.953179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995359+0000","last_change":"2023-10-16T09:43:25.976441+0000","last_active":"2023-10-16T09:43:27.995359+0000","last_peered":"2023-10-16T09:43:27.995359+0000","last_clean":"2023-10-16T09:43:27.995359+0000","last_became_active":"2023-10-16T09:43:25.975914+0000","last_became_peered":"2023-10-16T09:43:25.975914+0000","last_unstale":"2023-10-16T09:43:27.995359+0000","last_undegraded":"2023-10-16T09:43:27.995359+0000","last_fullsized":"2023-10-16T09:43:27.995359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950296+0000","last_change":"2023-10-16T09:43:20.925613+0000","last_active":"2023-10-16T09:43:25.950296+0000","last_peered":"2023-10-16T09:43:25.950296+0000","last_clean":"2023-10-16T09:43:25.950296+0000","last_became_active":"2023-10-16T09:43:20.925345+0000","last_became_peered":"2023-10-16T09:43:20.925345+0000","last_unstale":"2023-10-16T09:43:25.950296+0000","last_undegraded":"2023-10-16T09:43:25.950296+0000","last_fullsized":"2023-10-16T09:43:25.950296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:28.608301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948410+0000","last_change":"2023-10-16T09:43:21.925702+0000","last_active":"2023-10-16T09:43:25.948410+0000","last_peered":"2023-10-16T09:43:25.948410+0000","last_clean":"2023-10-16T09:43:25.948410+0000","last_became_active":"2023-10-16T09:43:21.925003+0000","last_became_peered":"2023-10-16T09:43:21.925003+0000","last_unstale":"2023-10-16T09:43:25.948410+0000","last_undegraded":"2023-10-16T09:43:25.948410+0000","last_fullsized":"2023-10-16T09:43:25.948410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995813+0000","last_change":"2023-10-16T09:43:25.976511+0000","last_active":"2023-10-16T09:43:27.995813+0000","last_peered":"2023-10-16T09:43:27.995813+0000","last_clean":"2023-10-16T09:43:27.995813+0000","last_became_active":"2023-10-16T09:43:25.975258+0000","last_became_peered":"2023-10-16T09:43:25.975258+0000","last_unstale":"2023-10-16T09:43:27.995813+0000","last_undegraded":"2023-10-16T09:43:27.995813+0000","last_fullsized":"2023-10-16T09:43:27.995813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953939+0000","last_change":"2023-10-16T09:43:21.924235+0000","last_active":"2023-10-16T09:43:25.953939+0000","last_peered":"2023-10-16T09:43:25.953939+0000","last_clean":"2023-10-16T09:43:25.953939+0000","last_became_active":"2023-10-16T09:43:21.923963+0000","last_became_peered":"2023-10-16T09:43:21.923963+0000","last_unstale":"2023-10-16T09:43:25.953939+0000","last_undegraded":"2023-10-16T09:43:25.953939+0000","last_fullsized":"2023-10-16T09:43:25.953939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948171+0000","last_change":"2023-10-16T09:43:20.922462+0000","last_active":"2023-10-16T09:43:25.948171+0000","last_peered":"2023-10-16T09:43:25.948171+0000","last_clean":"2023-10-16T09:43:25.948171+0000","last_became_active":"2023-10-16T09:43:20.922204+0000","last_became_peered":"2023-10-16T09:43:20.922204+0000","last_unstale":"2023-10-16T09:43:25.948171+0000","last_undegraded":"2023-10-16T09:43:25.948171+0000","last_fullsized":"2023-10-16T09:43:25.948171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949005+0000","last_change":"2023-10-16T09:43:25.975066+0000","last_active":"2023-10-16T09:43:26.949005+0000","last_peered":"2023-10-16T09:43:26.949005+0000","last_clean":"2023-10-16T09:43:26.949005+0000","last_became_active":"2023-10-16T09:43:25.974685+0000","last_became_peered":"2023-10-16T09:43:25.974685+0000","last_unstale":"2023-10-16T09:43:26.949005+0000","last_undegraded":"2023-10-16T09:43:26.949005+0000","last_fullsized":"2023-10-16T09:43:26.949005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966726+0000","last_change":"2023-10-16T09:43:21.940037+0000","last_active":"2023-10-16T09:43:25.966726+0000","last_peered":"2023-10-16T09:43:25.966726+0000","last_clean":"2023-10-16T09:43:25.966726+0000","last_became_active":"2023-10-16T09:43:21.939645+0000","last_became_peered":"2023-10-16T09:43:21.939645+0000","last_unstale":"2023-10-16T09:43:25.966726+0000","last_undegraded":"2023-10-16T09:43:25.966726+0000","last_fullsized":"2023-10-16T09:43:25.966726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950263+0000","last_change":"2023-10-16T09:43:20.925442+0000","last_active":"2023-10-16T09:43:25.950263+0000","last_peered":"2023-10-16T09:43:25.950263+0000","last_clean":"2023-10-16T09:43:25.950263+0000","last_became_active":"2023-10-16T09:43:20.925129+0000","last_became_peered":"2023-10-16T09:43:20.925129+0000","last_unstale":"2023-10-16T09:43:25.950263+0000","last_undegraded":"2023-10-16T09:43:25.950263+0000","last_fullsized":"2023-10-16T09:43:25.950263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015775+0000","last_change":"2023-10-16T09:43:25.961486+0000","last_active":"2023-10-16T09:43:28.015775+0000","last_peered":"2023-10-16T09:43:28.015775+0000","last_clean":"2023-10-16T09:43:28.015775+0000","last_became_active":"2023-10-16T09:43:25.961120+0000","last_became_peered":"2023-10-16T09:43:25.961120+0000","last_unstale":"2023-10-16T09:43:28.015775+0000","last_undegraded":"2023-10-16T09:43:28.015775+0000","last_fullsized":"2023-10-16T09:43:28.015775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947444+0000","last_change":"2023-10-16T09:43:21.929303+0000","last_active":"2023-10-16T09:43:25.947444+0000","last_peered":"2023-10-16T09:43:25.947444+0000","last_clean":"2023-10-16T09:43:25.947444+0000","last_became_active":"2023-10-16T09:43:21.929017+0000","last_became_peered":"2023-10-16T09:43:21.929017+0000","last_unstale":"2023-10-16T09:43:25.947444+0000","last_undegraded":"2023-10-16T09:43:25.947444+0000","last_fullsized":"2023-10-16T09:43:25.947444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:45.433460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:20.925419+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:20.924900+0000","last_became_peered":"2023-10-16T09:43:20.924900+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.960836+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.959879+0000","last_became_peered":"2023-10-16T09:43:25.959879+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944657+0000","last_change":"2023-10-16T09:43:20.935065+0000","last_active":"2023-10-16T09:43:25.944657+0000","last_peered":"2023-10-16T09:43:25.944657+0000","last_clean":"2023-10-16T09:43:25.944657+0000","last_became_active":"2023-10-16T09:43:20.934870+0000","last_became_peered":"2023-10-16T09:43:20.934870+0000","last_unstale":"2023-10-16T09:43:25.944657+0000","last_undegraded":"2023-10-16T09:43:25.944657+0000","last_fullsized":"2023-10-16T09:43:25.944657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955720+0000","last_change":"2023-10-16T09:43:21.924250+0000","last_active":"2023-10-16T09:43:25.955720+0000","last_peered":"2023-10-16T09:43:25.955720+0000","last_clean":"2023-10-16T09:43:25.955720+0000","last_became_active":"2023-10-16T09:43:21.923974+0000","last_became_peered":"2023-10-16T09:43:21.923974+0000","last_unstale":"2023-10-16T09:43:25.955720+0000","last_undegraded":"2023-10-16T09:43:25.955720+0000","last_fullsized":"2023-10-16T09:43:25.955720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.974974+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.974574+0000","last_became_peered":"2023-10-16T09:43:25.974574+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955603+0000","last_change":"2023-10-16T09:43:20.928340+0000","last_active":"2023-10-16T09:43:25.955603+0000","last_peered":"2023-10-16T09:43:25.955603+0000","last_clean":"2023-10-16T09:43:25.955603+0000","last_became_active":"2023-10-16T09:43:20.927748+0000","last_became_peered":"2023-10-16T09:43:20.927748+0000","last_unstale":"2023-10-16T09:43:25.955603+0000","last_undegraded":"2023-10-16T09:43:25.955603+0000","last_fullsized":"2023-10-16T09:43:25.955603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967050+0000","last_change":"2023-10-16T09:43:21.919707+0000","last_active":"2023-10-16T09:43:25.967050+0000","last_peered":"2023-10-16T09:43:25.967050+0000","last_clean":"2023-10-16T09:43:25.967050+0000","last_became_active":"2023-10-16T09:43:21.919454+0000","last_became_peered":"2023-10-16T09:43:21.919454+0000","last_unstale":"2023-10-16T09:43:25.967050+0000","last_undegraded":"2023-10-16T09:43:25.967050+0000","last_fullsized":"2023-10-16T09:43:25.967050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947855+0000","last_change":"2023-10-16T09:43:25.961225+0000","last_active":"2023-10-16T09:43:26.947855+0000","last_peered":"2023-10-16T09:43:26.947855+0000","last_clean":"2023-10-16T09:43:26.947855+0000","last_became_active":"2023-10-16T09:43:25.960336+0000","last_became_peered":"2023-10-16T09:43:25.960336+0000","last_unstale":"2023-10-16T09:43:26.947855+0000","last_undegraded":"2023-10-16T09:43:26.947855+0000","last_fullsized":"2023-10-16T09:43:26.947855+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:15.617593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956170+0000","last_change":"2023-10-16T09:43:20.928799+0000","last_active":"2023-10-16T09:43:25.956170+0000","last_peered":"2023-10-16T09:43:25.956170+0000","last_clean":"2023-10-16T09:43:25.956170+0000","last_became_active":"2023-10-16T09:43:20.928302+0000","last_became_peered":"2023-10-16T09:43:20.928302+0000","last_unstale":"2023-10-16T09:43:25.956170+0000","last_undegraded":"2023-10-16T09:43:25.956170+0000","last_fullsized":"2023-10-16T09:43:25.956170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951319+0000","last_change":"2023-10-16T09:43:21.924602+0000","last_active":"2023-10-16T09:43:25.951319+0000","last_peered":"2023-10-16T09:43:25.951319+0000","last_clean":"2023-10-16T09:43:25.951319+0000","last_became_active":"2023-10-16T09:43:21.924344+0000","last_became_peered":"2023-10-16T09:43:21.924344+0000","last_unstale":"2023-10-16T09:43:25.951319+0000","last_undegraded":"2023-10-16T09:43:25.951319+0000","last_fullsized":"2023-10-16T09:43:25.951319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948047+0000","last_change":"2023-10-16T09:43:25.969751+0000","last_active":"2023-10-16T09:43:26.948047+0000","last_peered":"2023-10-16T09:43:26.948047+0000","last_clean":"2023-10-16T09:43:26.948047+0000","last_became_active":"2023-10-16T09:43:25.969399+0000","last_became_peered":"2023-10-16T09:43:25.969399+0000","last_unstale":"2023-10-16T09:43:26.948047+0000","last_undegraded":"2023-10-16T09:43:26.948047+0000","last_fullsized":"2023-10-16T09:43:26.948047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:20.933882+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:20.933615+0000","last_became_peered":"2023-10-16T09:43:20.933615+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947408+0000","last_change":"2023-10-16T09:43:21.929135+0000","last_active":"2023-10-16T09:43:25.947408+0000","last_peered":"2023-10-16T09:43:25.947408+0000","last_clean":"2023-10-16T09:43:25.947408+0000","last_became_active":"2023-10-16T09:43:21.928849+0000","last_became_peered":"2023-10-16T09:43:21.928849+0000","last_unstale":"2023-10-16T09:43:25.947408+0000","last_undegraded":"2023-10-16T09:43:25.947408+0000","last_fullsized":"2023-10-16T09:43:25.947408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995596+0000","last_change":"2023-10-16T09:43:25.976682+0000","last_active":"2023-10-16T09:43:27.995596+0000","last_peered":"2023-10-16T09:43:27.995596+0000","last_clean":"2023-10-16T09:43:27.995596+0000","last_became_active":"2023-10-16T09:43:25.976274+0000","last_became_peered":"2023-10-16T09:43:25.976274+0000","last_unstale":"2023-10-16T09:43:27.995596+0000","last_undegraded":"2023-10-16T09:43:27.995596+0000","last_fullsized":"2023-10-16T09:43:27.995596+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954044+0000","last_change":"2023-10-16T09:43:20.926509+0000","last_active":"2023-10-16T09:43:25.954044+0000","last_peered":"2023-10-16T09:43:25.954044+0000","last_clean":"2023-10-16T09:43:25.954044+0000","last_became_active":"2023-10-16T09:43:20.926228+0000","last_became_peered":"2023-10-16T09:43:20.926228+0000","last_unstale":"2023-10-16T09:43:25.954044+0000","last_undegraded":"2023-10-16T09:43:25.954044+0000","last_fullsized":"2023-10-16T09:43:25.954044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965799+0000","last_change":"2023-10-16T09:43:21.924014+0000","last_active":"2023-10-16T09:43:25.965799+0000","last_peered":"2023-10-16T09:43:25.965799+0000","last_clean":"2023-10-16T09:43:25.965799+0000","last_became_active":"2023-10-16T09:43:21.922684+0000","last_became_peered":"2023-10-16T09:43:21.922684+0000","last_unstale":"2023-10-16T09:43:25.965799+0000","last_undegraded":"2023-10-16T09:43:25.965799+0000","last_fullsized":"2023-10-16T09:43:25.965799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948333+0000","last_change":"2023-10-16T09:43:25.970563+0000","last_active":"2023-10-16T09:43:26.948333+0000","last_peered":"2023-10-16T09:43:26.948333+0000","last_clean":"2023-10-16T09:43:26.948333+0000","last_became_active":"2023-10-16T09:43:25.970168+0000","last_became_peered":"2023-10-16T09:43:25.970168+0000","last_unstale":"2023-10-16T09:43:26.948333+0000","last_undegraded":"2023-10-16T09:43:26.948333+0000","last_fullsized":"2023-10-16T09:43:26.948333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:56.567461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953937+0000","last_change":"2023-10-16T09:43:20.923587+0000","last_active":"2023-10-16T09:43:25.953937+0000","last_peered":"2023-10-16T09:43:25.953937+0000","last_clean":"2023-10-16T09:43:25.953937+0000","last_became_active":"2023-10-16T09:43:20.923277+0000","last_became_peered":"2023-10-16T09:43:20.923277+0000","last_unstale":"2023-10-16T09:43:25.953937+0000","last_undegraded":"2023-10-16T09:43:25.953937+0000","last_fullsized":"2023-10-16T09:43:25.953937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966226+0000","last_change":"2023-10-16T09:43:21.925711+0000","last_active":"2023-10-16T09:43:25.966226+0000","last_peered":"2023-10-16T09:43:25.966226+0000","last_clean":"2023-10-16T09:43:25.966226+0000","last_became_active":"2023-10-16T09:43:21.921509+0000","last_became_peered":"2023-10-16T09:43:21.921509+0000","last_unstale":"2023-10-16T09:43:25.966226+0000","last_undegraded":"2023-10-16T09:43:25.966226+0000","last_fullsized":"2023-10-16T09:43:25.966226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948182+0000","last_change":"2023-10-16T09:43:25.955876+0000","last_active":"2023-10-16T09:43:26.948182+0000","last_peered":"2023-10-16T09:43:26.948182+0000","last_clean":"2023-10-16T09:43:26.948182+0000","last_became_active":"2023-10-16T09:43:25.955490+0000","last_became_peered":"2023-10-16T09:43:25.955490+0000","last_unstale":"2023-10-16T09:43:26.948182+0000","last_undegraded":"2023-10-16T09:43:26.948182+0000","last_fullsized":"2023-10-16T09:43:26.948182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945113+0000","last_change":"2023-10-16T09:43:20.926395+0000","last_active":"2023-10-16T09:43:25.945113+0000","last_peered":"2023-10-16T09:43:25.945113+0000","last_clean":"2023-10-16T09:43:25.945113+0000","last_became_active":"2023-10-16T09:43:20.926086+0000","last_became_peered":"2023-10-16T09:43:20.926086+0000","last_unstale":"2023-10-16T09:43:25.945113+0000","last_undegraded":"2023-10-16T09:43:25.945113+0000","last_fullsized":"2023-10-16T09:43:25.945113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:56.924185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957128+0000","last_change":"2023-10-16T09:43:21.932460+0000","last_active":"2023-10-16T09:43:25.957128+0000","last_peered":"2023-10-16T09:43:25.957128+0000","last_clean":"2023-10-16T09:43:25.957128+0000","last_became_active":"2023-10-16T09:43:21.932184+0000","last_became_peered":"2023-10-16T09:43:21.932184+0000","last_unstale":"2023-10-16T09:43:25.957128+0000","last_undegraded":"2023-10-16T09:43:25.957128+0000","last_fullsized":"2023-10-16T09:43:25.957128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948895+0000","last_change":"2023-10-16T09:43:25.967183+0000","last_active":"2023-10-16T09:43:26.948895+0000","last_peered":"2023-10-16T09:43:26.948895+0000","last_clean":"2023-10-16T09:43:26.948895+0000","last_became_active":"2023-10-16T09:43:25.966759+0000","last_became_peered":"2023-10-16T09:43:25.966759+0000","last_unstale":"2023-10-16T09:43:26.948895+0000","last_undegraded":"2023-10-16T09:43:26.948895+0000","last_fullsized":"2023-10-16T09:43:26.948895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948721+0000","last_change":"2023-10-16T09:43:20.925234+0000","last_active":"2023-10-16T09:43:25.948721+0000","last_peered":"2023-10-16T09:43:25.948721+0000","last_clean":"2023-10-16T09:43:25.948721+0000","last_became_active":"2023-10-16T09:43:20.924957+0000","last_became_peered":"2023-10-16T09:43:20.924957+0000","last_unstale":"2023-10-16T09:43:25.948721+0000","last_undegraded":"2023-10-16T09:43:25.948721+0000","last_fullsized":"2023-10-16T09:43:25.948721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:00.084223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944746+0000","last_change":"2023-10-16T09:43:21.936106+0000","last_active":"2023-10-16T09:43:25.944746+0000","last_peered":"2023-10-16T09:43:25.944746+0000","last_clean":"2023-10-16T09:43:25.944746+0000","last_became_active":"2023-10-16T09:43:21.935859+0000","last_became_peered":"2023-10-16T09:43:21.935859+0000","last_unstale":"2023-10-16T09:43:25.944746+0000","last_undegraded":"2023-10-16T09:43:25.944746+0000","last_fullsized":"2023-10-16T09:43:25.944746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.014901+0000","last_change":"2023-10-16T09:43:25.961369+0000","last_active":"2023-10-16T09:43:28.014901+0000","last_peered":"2023-10-16T09:43:28.014901+0000","last_clean":"2023-10-16T09:43:28.014901+0000","last_became_active":"2023-10-16T09:43:25.960465+0000","last_became_peered":"2023-10-16T09:43:25.960465+0000","last_unstale":"2023-10-16T09:43:28.014901+0000","last_undegraded":"2023-10-16T09:43:28.014901+0000","last_fullsized":"2023-10-16T09:43:28.014901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:40.952070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955514+0000","last_change":"2023-10-16T09:43:20.922927+0000","last_active":"2023-10-16T09:43:25.955514+0000","last_peered":"2023-10-16T09:43:25.955514+0000","last_clean":"2023-10-16T09:43:25.955514+0000","last_became_active":"2023-10-16T09:43:20.922658+0000","last_became_peered":"2023-10-16T09:43:20.922658+0000","last_unstale":"2023-10-16T09:43:25.955514+0000","last_undegraded":"2023-10-16T09:43:25.955514+0000","last_fullsized":"2023-10-16T09:43:25.955514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950404+0000","last_change":"2023-10-16T09:43:21.928420+0000","last_active":"2023-10-16T09:43:25.950404+0000","last_peered":"2023-10-16T09:43:25.950404+0000","last_clean":"2023-10-16T09:43:25.950404+0000","last_became_active":"2023-10-16T09:43:21.928180+0000","last_became_peered":"2023-10-16T09:43:21.928180+0000","last_unstale":"2023-10-16T09:43:25.950404+0000","last_undegraded":"2023-10-16T09:43:25.950404+0000","last_fullsized":"2023-10-16T09:43:25.950404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015214+0000","last_change":"2023-10-16T09:43:25.960304+0000","last_active":"2023-10-16T09:43:28.015214+0000","last_peered":"2023-10-16T09:43:28.015214+0000","last_clean":"2023-10-16T09:43:28.015214+0000","last_became_active":"2023-10-16T09:43:25.959916+0000","last_became_peered":"2023-10-16T09:43:25.959916+0000","last_unstale":"2023-10-16T09:43:28.015214+0000","last_undegraded":"2023-10-16T09:43:28.015214+0000","last_fullsized":"2023-10-16T09:43:28.015214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948747+0000","last_change":"2023-10-16T09:43:20.921666+0000","last_active":"2023-10-16T09:43:25.948747+0000","last_peered":"2023-10-16T09:43:25.948747+0000","last_clean":"2023-10-16T09:43:25.948747+0000","last_became_active":"2023-10-16T09:43:20.921373+0000","last_became_peered":"2023-10-16T09:43:20.921373+0000","last_unstale":"2023-10-16T09:43:25.948747+0000","last_undegraded":"2023-10-16T09:43:25.948747+0000","last_fullsized":"2023-10-16T09:43:25.948747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944941+0000","last_change":"2023-10-16T09:43:21.930965+0000","last_active":"2023-10-16T09:43:25.944941+0000","last_peered":"2023-10-16T09:43:25.944941+0000","last_clean":"2023-10-16T09:43:25.944941+0000","last_became_active":"2023-10-16T09:43:21.930701+0000","last_became_peered":"2023-10-16T09:43:21.930701+0000","last_unstale":"2023-10-16T09:43:25.944941+0000","last_undegraded":"2023-10-16T09:43:25.944941+0000","last_fullsized":"2023-10-16T09:43:25.944941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:20.329096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954036+0000","last_change":"2023-10-16T09:43:21.930641+0000","last_active":"2023-10-16T09:43:25.954036+0000","last_peered":"2023-10-16T09:43:25.954036+0000","last_clean":"2023-10-16T09:43:25.954036+0000","last_became_active":"2023-10-16T09:43:21.930194+0000","last_became_peered":"2023-10-16T09:43:21.930194+0000","last_unstale":"2023-10-16T09:43:25.954036+0000","last_undegraded":"2023-10-16T09:43:25.954036+0000","last_fullsized":"2023-10-16T09:43:25.954036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948076+0000","last_change":"2023-10-16T09:43:21.933068+0000","last_active":"2023-10-16T09:43:25.948076+0000","last_peered":"2023-10-16T09:43:25.948076+0000","last_clean":"2023-10-16T09:43:25.948076+0000","last_became_active":"2023-10-16T09:43:21.932774+0000","last_became_peered":"2023-10-16T09:43:21.932774+0000","last_unstale":"2023-10-16T09:43:25.948076+0000","last_undegraded":"2023-10-16T09:43:25.948076+0000","last_fullsized":"2023-10-16T09:43:25.948076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950489+0000","last_change":"2023-10-16T09:43:21.937509+0000","last_active":"2023-10-16T09:43:25.950489+0000","last_peered":"2023-10-16T09:43:25.950489+0000","last_clean":"2023-10-16T09:43:25.950489+0000","last_became_active":"2023-10-16T09:43:21.936970+0000","last_became_peered":"2023-10-16T09:43:21.936970+0000","last_unstale":"2023-10-16T09:43:25.950489+0000","last_undegraded":"2023-10-16T09:43:25.950489+0000","last_fullsized":"2023-10-16T09:43:25.950489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948478+0000","last_change":"2023-10-16T09:43:21.927275+0000","last_active":"2023-10-16T09:43:25.948478+0000","last_peered":"2023-10-16T09:43:25.948478+0000","last_clean":"2023-10-16T09:43:25.948478+0000","last_became_active":"2023-10-16T09:43:21.927024+0000","last_became_peered":"2023-10-16T09:43:21.927024+0000","last_unstale":"2023-10-16T09:43:25.948478+0000","last_undegraded":"2023-10-16T09:43:25.948478+0000","last_fullsized":"2023-10-16T09:43:25.948478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966374+0000","last_change":"2023-10-16T09:43:21.925070+0000","last_active":"2023-10-16T09:43:25.966374+0000","last_peered":"2023-10-16T09:43:25.966374+0000","last_clean":"2023-10-16T09:43:25.966374+0000","last_became_active":"2023-10-16T09:43:21.924799+0000","last_became_peered":"2023-10-16T09:43:21.924799+0000","last_unstale":"2023-10-16T09:43:25.966374+0000","last_undegraded":"2023-10-16T09:43:25.966374+0000","last_fullsized":"2023-10-16T09:43:25.966374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955685+0000","last_change":"2023-10-16T09:43:21.920367+0000","last_active":"2023-10-16T09:43:25.955685+0000","last_peered":"2023-10-16T09:43:25.955685+0000","last_clean":"2023-10-16T09:43:25.955685+0000","last_became_active":"2023-10-16T09:43:21.919980+0000","last_became_peered":"2023-10-16T09:43:21.919980+0000","last_unstale":"2023-10-16T09:43:25.955685+0000","last_undegraded":"2023-10-16T09:43:25.955685+0000","last_fullsized":"2023-10-16T09:43:25.955685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956021+0000","last_change":"2023-10-16T09:43:21.931239+0000","last_active":"2023-10-16T09:43:25.956021+0000","last_peered":"2023-10-16T09:43:25.956021+0000","last_clean":"2023-10-16T09:43:25.956021+0000","last_became_active":"2023-10-16T09:43:21.930923+0000","last_became_peered":"2023-10-16T09:43:21.930923+0000","last_unstale":"2023-10-16T09:43:25.956021+0000","last_undegraded":"2023-10-16T09:43:25.956021+0000","last_fullsized":"2023-10-16T09:43:25.956021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967144+0000","last_change":"2023-10-16T09:43:21.924310+0000","last_active":"2023-10-16T09:43:25.967144+0000","last_peered":"2023-10-16T09:43:25.967144+0000","last_clean":"2023-10-16T09:43:25.967144+0000","last_became_active":"2023-10-16T09:43:21.921665+0000","last_became_peered":"2023-10-16T09:43:21.921665+0000","last_unstale":"2023-10-16T09:43:25.967144+0000","last_undegraded":"2023-10-16T09:43:25.967144+0000","last_fullsized":"2023-10-16T09:43:25.967144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950321+0000","last_change":"2023-10-16T09:43:21.927572+0000","last_active":"2023-10-16T09:43:25.950321+0000","last_peered":"2023-10-16T09:43:25.950321+0000","last_clean":"2023-10-16T09:43:25.950321+0000","last_became_active":"2023-10-16T09:43:21.927298+0000","last_became_peered":"2023-10-16T09:43:21.927298+0000","last_unstale":"2023-10-16T09:43:25.950321+0000","last_undegraded":"2023-10-16T09:43:25.950321+0000","last_fullsized":"2023-10-16T09:43:25.950321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965935+0000","last_change":"2023-10-16T09:43:21.922799+0000","last_active":"2023-10-16T09:43:25.965935+0000","last_peered":"2023-10-16T09:43:25.965935+0000","last_clean":"2023-10-16T09:43:25.965935+0000","last_became_active":"2023-10-16T09:43:21.922126+0000","last_became_peered":"2023-10-16T09:43:21.922126+0000","last_unstale":"2023-10-16T09:43:25.965935+0000","last_undegraded":"2023-10-16T09:43:25.965935+0000","last_fullsized":"2023-10-16T09:43:25.965935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950579+0000","last_change":"2023-10-16T09:43:21.936429+0000","last_active":"2023-10-16T09:43:25.950579+0000","last_peered":"2023-10-16T09:43:25.950579+0000","last_clean":"2023-10-16T09:43:25.950579+0000","last_became_active":"2023-10-16T09:43:21.933542+0000","last_became_peered":"2023-10-16T09:43:21.933542+0000","last_unstale":"2023-10-16T09:43:25.950579+0000","last_undegraded":"2023-10-16T09:43:25.950579+0000","last_fullsized":"2023-10-16T09:43:25.950579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948591+0000","last_change":"2023-10-16T09:43:21.930439+0000","last_active":"2023-10-16T09:43:25.948591+0000","last_peered":"2023-10-16T09:43:25.948591+0000","last_clean":"2023-10-16T09:43:25.948591+0000","last_became_active":"2023-10-16T09:43:21.930237+0000","last_became_peered":"2023-10-16T09:43:21.930237+0000","last_unstale":"2023-10-16T09:43:25.948591+0000","last_undegraded":"2023-10-16T09:43:25.948591+0000","last_fullsized":"2023-10-16T09:43:25.948591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:17.181381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953918+0000","last_change":"2023-10-16T09:43:21.934041+0000","last_active":"2023-10-16T09:43:25.953918+0000","last_peered":"2023-10-16T09:43:25.953918+0000","last_clean":"2023-10-16T09:43:25.953918+0000","last_became_active":"2023-10-16T09:43:21.933451+0000","last_became_peered":"2023-10-16T09:43:21.933451+0000","last_unstale":"2023-10-16T09:43:25.953918+0000","last_undegraded":"2023-10-16T09:43:25.953918+0000","last_fullsized":"2023-10-16T09:43:25.953918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":99940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":99780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:43:45.487 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:43:45.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.641+0000 7f582c5be700 1 Processor -- start 2023-10-16T09:43:45.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.641+0000 7f582c5be700 1 -- start start 2023-10-16T09:43:45.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.641+0000 7f582c5be700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f582406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.641+0000 7f582c5be700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f582406f340 0x7f582406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.641+0000 7f582c5be700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582406e770 con 0x7f58240cc5d0 2023-10-16T09:43:45.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582c5be700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582406fd30 con 0x7f582406f340 2023-10-16T09:43:45.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582c5be700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58240cf270 con 0x7f582415dfb0 2023-10-16T09:43:45.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582a35a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f58240cc5d0 0x7f58240cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46568/0 (socket says 172.21.15.158:46568) 2023-10-16T09:43:45.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582a35a700 1 -- 172.21.15.158:0/2554419490 learned_addr learned my addr 172.21.15.158:0/2554419490 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:45.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582ab5b700 1 --2- 172.21.15.158:0/2554419490 >> v2:172.21.15.158:3300/0 conn(0x7f582406f340 0x7f582406f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f5829b59700 1 --2- 172.21.15.158:0/2554419490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f582406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f5829358700 1 -- 172.21.15.158:0/2554419490 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365243294 0 0) 0x7f581c001490 con 0x7f58240cc5d0 2023-10-16T09:43:45.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f5829358700 1 -- 172.21.15.158:0/2554419490 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58100035b0 con 0x7f58240cc5d0 2023-10-16T09:43:45.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582ab5b700 1 -- 172.21.15.158:0/2554419490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 msgr2=0x7f582406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:45.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582ab5b700 1 --2- 172.21.15.158:0/2554419490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f582406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582ab5b700 1 -- 172.21.15.158:0/2554419490 >> v1:172.21.15.158:6789/0 conn(0x7f58240cc5d0 legacy=0x7f58240cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582ab5b700 1 -- 172.21.15.158:0/2554419490 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58240cbe00 con 0x7f582406f340 2023-10-16T09:43:45.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f5829b59700 1 --2- 172.21.15.158:0/2554419490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f582406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:45.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f582ab5b700 1 --2- 172.21.15.158:0/2554419490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582406f7f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5820009840 tx=0x7f5820009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac91491a3a6a2db7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:45.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f5829358700 1 -- 172.21.15.158:0/2554419490 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820011830 con 0x7f582406f340 2023-10-16T09:43:45.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f5829358700 1 -- 172.21.15.158:0/2554419490 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5820011990 con 0x7f582406f340 2023-10-16T09:43:45.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.642+0000 7f5829358700 1 -- 172.21.15.158:0/2554419490 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820011d20 con 0x7f582406f340 2023-10-16T09:43:45.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 -- 172.21.15.158:0/2554419490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 msgr2=0x7f582406f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 --2- 172.21.15.158:0/2554419490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582406f7f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5820009840 tx=0x7f5820009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:45.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 -- 172.21.15.158:0/2554419490 shutdown_connections 2023-10-16T09:43:45.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 --2- 172.21.15.158:0/2554419490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582406f7f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 --2- 172.21.15.158:0/2554419490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f582406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 -- 172.21.15.158:0/2554419490 >> 172.21.15.158:0/2554419490 conn(0x7f58240d02b0 msgr2=0x7f58240d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:45.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 -- 172.21.15.158:0/2554419490 shutdown_connections 2023-10-16T09:43:45.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.643+0000 7f582c5be700 1 -- 172.21.15.158:0/2554419490 wait complete. 2023-10-16T09:43:45.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.644+0000 7f582c5be700 1 Processor -- start 2023-10-16T09:43:45.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.644+0000 7f582c5be700 1 -- start start 2023-10-16T09:43:45.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582c5be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582414d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582c5be700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58240cc5d0 0x7f5824157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582c5be700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f58241576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582c5be700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582415c650 con 0x7f582406f340 2023-10-16T09:43:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582c5be700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582415b0f0 con 0x7f58240cc5d0 2023-10-16T09:43:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582c5be700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58240c5fb0 con 0x7f582415dfb0 2023-10-16T09:43:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582414d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582414d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46398/0 (socket says 172.21.15.158:46398) 2023-10-16T09:43:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 -- 172.21.15.158:0/491968167 learned_addr learned my addr 172.21.15.158:0/491968167 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f5829b59700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58240cc5d0 0x7f5824157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582ab5b700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f58241576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 -- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 msgr2=0x7f58241576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f58241576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 -- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58240cc5d0 msgr2=0x7f5824157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58240cc5d0 0x7f5824157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 -- 172.21.15.158:0/491968167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58200094a0 con 0x7f582406f340 2023-10-16T09:43:45.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f5829b59700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58240cc5d0 0x7f5824157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:45.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.645+0000 7f582a35a700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582414d360 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f581c00cbb0 tx=0x7f581c0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:45.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.646+0000 7f581affd700 1 -- 172.21.15.158:0/491968167 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c00ab50 con 0x7f582406f340 2023-10-16T09:43:45.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.646+0000 7f581affd700 1 -- 172.21.15.158:0/491968167 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c00acb0 con 0x7f582406f340 2023-10-16T09:43:45.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.646+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582415aae0 con 0x7f582406f340 2023-10-16T09:43:45.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.646+0000 7f581affd700 1 -- 172.21.15.158:0/491968167 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c012600 con 0x7f582406f340 2023-10-16T09:43:45.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.646+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58240c6250 con 0x7f582406f340 2023-10-16T09:43:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.647+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58240681b0 con 0x7f582406f340 2023-10-16T09:43:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.647+0000 7f581affd700 1 -- 172.21.15.158:0/491968167 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f581c01b3b0 con 0x7f582406f340 2023-10-16T09:43:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.647+0000 7f581affd700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f580806af00 0x7f580806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.648+0000 7f581affd700 1 -- 172.21.15.158:0/491968167 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f581c0893e0 con 0x7f582406f340 2023-10-16T09:43:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.648+0000 7f5829b59700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f580806af00 0x7f580806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:45.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.648+0000 7f5829b59700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f580806af00 0x7f580806d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f58240cee90 tx=0x7f5814008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:45.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.652+0000 7f581affd700 1 -- 172.21.15.158:0/491968167 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f581c054250 con 0x7f582406f340 2023-10-16T09:43:45.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.841+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f582415e7c0 con 0x7f580806af00 2023-10-16T09:43:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.850+0000 7f581affd700 1 -- 172.21.15.158:0/491968167 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345314 (secure 0 0 0) 0x7f581400c050 con 0x7f580806af00 2023-10-16T09:43:45.855 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:45.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f580806af00 msgr2=0x7f580806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f580806af00 0x7f580806d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f58240cee90 tx=0x7f5814008040 comp rx=0 tx=0).stop 2023-10-16T09:43:45.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 msgr2=0x7f582414d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:45.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582414d360 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f581c00cbb0 tx=0x7f581c0021d0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 shutdown_connections 2023-10-16T09:43:45.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f580806af00 0x7f580806d3c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f582415dfb0 0x7f58241576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f582406f340 0x7f582414d360 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 --2- 172.21.15.158:0/491968167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58240cc5d0 0x7f5824157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:45.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 >> 172.21.15.158:0/491968167 conn(0x7f58240d02b0 msgr2=0x7f58240afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:45.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 shutdown_connections 2023-10-16T09:43:45.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:45.854+0000 7f582c5be700 1 -- 172.21.15.158:0/491968167 wait complete. 2023-10-16T09:43:45.861 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:43:45.867 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-10-16T09:43:45.223013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3284992,"data_stored":1988256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944843+0000","last_change":"2023-10-16T09:43:21.932095+0000","last_active":"2023-10-16T09:43:25.944843+0000","last_peered":"2023-10-16T09:43:25.944843+0000","last_clean":"2023-10-16T09:43:25.944843+0000","last_became_active":"2023-10-16T09:43:21.931826+0000","last_became_peered":"2023-10-16T09:43:21.931826+0000","last_unstale":"2023-10-16T09:43:25.944843+0000","last_undegraded":"2023-10-16T09:43:25.944843+0000","last_fullsized":"2023-10-16T09:43:25.944843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955709+0000","last_change":"2023-10-16T09:43:21.921656+0000","last_active":"2023-10-16T09:43:25.955709+0000","last_peered":"2023-10-16T09:43:25.955709+0000","last_clean":"2023-10-16T09:43:25.955709+0000","last_became_active":"2023-10-16T09:43:21.921392+0000","last_became_peered":"2023-10-16T09:43:21.921392+0000","last_unstale":"2023-10-16T09:43:25.955709+0000","last_undegraded":"2023-10-16T09:43:25.955709+0000","last_fullsized":"2023-10-16T09:43:25.955709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951037+0000","last_change":"2023-10-16T09:43:21.925692+0000","last_active":"2023-10-16T09:43:25.951037+0000","last_peered":"2023-10-16T09:43:25.951037+0000","last_clean":"2023-10-16T09:43:25.951037+0000","last_became_active":"2023-10-16T09:43:21.925438+0000","last_became_peered":"2023-10-16T09:43:21.925438+0000","last_unstale":"2023-10-16T09:43:25.951037+0000","last_undegraded":"2023-10-16T09:43:25.951037+0000","last_fullsized":"2023-10-16T09:43:25.951037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950435+0000","last_change":"2023-10-16T09:43:21.927209+0000","last_active":"2023-10-16T09:43:25.950435+0000","last_peered":"2023-10-16T09:43:25.950435+0000","last_clean":"2023-10-16T09:43:25.950435+0000","last_became_active":"2023-10-16T09:43:21.926900+0000","last_became_peered":"2023-10-16T09:43:21.926900+0000","last_unstale":"2023-10-16T09:43:25.950435+0000","last_undegraded":"2023-10-16T09:43:25.950435+0000","last_fullsized":"2023-10-16T09:43:25.950435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944943+0000","last_change":"2023-10-16T09:43:21.924582+0000","last_active":"2023-10-16T09:43:25.944943+0000","last_peered":"2023-10-16T09:43:25.944943+0000","last_clean":"2023-10-16T09:43:25.944943+0000","last_became_active":"2023-10-16T09:43:21.924329+0000","last_became_peered":"2023-10-16T09:43:21.924329+0000","last_unstale":"2023-10-16T09:43:25.944943+0000","last_undegraded":"2023-10-16T09:43:25.944943+0000","last_fullsized":"2023-10-16T09:43:25.944943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950668+0000","last_change":"2023-10-16T09:43:21.937326+0000","last_active":"2023-10-16T09:43:25.950668+0000","last_peered":"2023-10-16T09:43:25.950668+0000","last_clean":"2023-10-16T09:43:25.950668+0000","last_became_active":"2023-10-16T09:43:21.936665+0000","last_became_peered":"2023-10-16T09:43:21.936665+0000","last_unstale":"2023-10-16T09:43:25.950668+0000","last_undegraded":"2023-10-16T09:43:25.950668+0000","last_fullsized":"2023-10-16T09:43:25.950668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:21.923235+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:21.923021+0000","last_became_peered":"2023-10-16T09:43:21.923021+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965916+0000","last_change":"2023-10-16T09:43:21.933111+0000","last_active":"2023-10-16T09:43:25.965916+0000","last_peered":"2023-10-16T09:43:25.965916+0000","last_clean":"2023-10-16T09:43:25.965916+0000","last_became_active":"2023-10-16T09:43:21.932835+0000","last_became_peered":"2023-10-16T09:43:21.932835+0000","last_unstale":"2023-10-16T09:43:25.965916+0000","last_undegraded":"2023-10-16T09:43:25.965916+0000","last_fullsized":"2023-10-16T09:43:25.965916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944721+0000","last_change":"2023-10-16T09:43:21.940444+0000","last_active":"2023-10-16T09:43:25.944721+0000","last_peered":"2023-10-16T09:43:25.944721+0000","last_clean":"2023-10-16T09:43:25.944721+0000","last_became_active":"2023-10-16T09:43:21.939823+0000","last_became_peered":"2023-10-16T09:43:21.939823+0000","last_unstale":"2023-10-16T09:43:25.944721+0000","last_undegraded":"2023-10-16T09:43:25.944721+0000","last_fullsized":"2023-10-16T09:43:25.944721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956248+0000","last_change":"2023-10-16T09:43:21.934573+0000","last_active":"2023-10-16T09:43:25.956248+0000","last_peered":"2023-10-16T09:43:25.956248+0000","last_clean":"2023-10-16T09:43:25.956248+0000","last_became_active":"2023-10-16T09:43:21.934285+0000","last_became_peered":"2023-10-16T09:43:21.934285+0000","last_unstale":"2023-10-16T09:43:25.956248+0000","last_undegraded":"2023-10-16T09:43:25.956248+0000","last_fullsized":"2023-10-16T09:43:25.956248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953996+0000","last_change":"2023-10-16T09:43:21.934669+0000","last_active":"2023-10-16T09:43:25.953996+0000","last_peered":"2023-10-16T09:43:25.953996+0000","last_clean":"2023-10-16T09:43:25.953996+0000","last_became_active":"2023-10-16T09:43:21.933918+0000","last_became_peered":"2023-10-16T09:43:21.933918+0000","last_unstale":"2023-10-16T09:43:25.953996+0000","last_undegraded":"2023-10-16T09:43:25.953996+0000","last_fullsized":"2023-10-16T09:43:25.953996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950011+0000","last_change":"2023-10-16T09:43:21.934285+0000","last_active":"2023-10-16T09:43:25.950011+0000","last_peered":"2023-10-16T09:43:25.950011+0000","last_clean":"2023-10-16T09:43:25.950011+0000","last_became_active":"2023-10-16T09:43:21.933516+0000","last_became_peered":"2023-10-16T09:43:21.933516+0000","last_unstale":"2023-10-16T09:43:25.950011+0000","last_undegraded":"2023-10-16T09:43:25.950011+0000","last_fullsized":"2023-10-16T09:43:25.950011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:56.814327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965961+0000","last_change":"2023-10-16T09:43:21.934162+0000","last_active":"2023-10-16T09:43:25.965961+0000","last_peered":"2023-10-16T09:43:25.965961+0000","last_clean":"2023-10-16T09:43:25.965961+0000","last_became_active":"2023-10-16T09:43:21.933898+0000","last_became_peered":"2023-10-16T09:43:21.933898+0000","last_unstale":"2023-10-16T09:43:25.965961+0000","last_undegraded":"2023-10-16T09:43:25.965961+0000","last_fullsized":"2023-10-16T09:43:25.965961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948340+0000","last_change":"2023-10-16T09:43:21.923364+0000","last_active":"2023-10-16T09:43:25.948340+0000","last_peered":"2023-10-16T09:43:25.948340+0000","last_clean":"2023-10-16T09:43:25.948340+0000","last_became_active":"2023-10-16T09:43:21.923123+0000","last_became_peered":"2023-10-16T09:43:21.923123+0000","last_unstale":"2023-10-16T09:43:25.948340+0000","last_undegraded":"2023-10-16T09:43:25.948340+0000","last_fullsized":"2023-10-16T09:43:25.948340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953943+0000","last_change":"2023-10-16T09:43:21.925350+0000","last_active":"2023-10-16T09:43:25.953943+0000","last_peered":"2023-10-16T09:43:25.953943+0000","last_clean":"2023-10-16T09:43:25.953943+0000","last_became_active":"2023-10-16T09:43:21.925107+0000","last_became_peered":"2023-10-16T09:43:21.925107+0000","last_unstale":"2023-10-16T09:43:25.953943+0000","last_undegraded":"2023-10-16T09:43:25.953943+0000","last_fullsized":"2023-10-16T09:43:25.953943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951212+0000","last_change":"2023-10-16T09:43:21.927864+0000","last_active":"2023-10-16T09:43:25.951212+0000","last_peered":"2023-10-16T09:43:25.951212+0000","last_clean":"2023-10-16T09:43:25.951212+0000","last_became_active":"2023-10-16T09:43:21.927411+0000","last_became_peered":"2023-10-16T09:43:21.927411+0000","last_unstale":"2023-10-16T09:43:25.951212+0000","last_undegraded":"2023-10-16T09:43:25.951212+0000","last_fullsized":"2023-10-16T09:43:25.951212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950524+0000","last_change":"2023-10-16T09:43:21.936884+0000","last_active":"2023-10-16T09:43:25.950524+0000","last_peered":"2023-10-16T09:43:25.950524+0000","last_clean":"2023-10-16T09:43:25.950524+0000","last_became_active":"2023-10-16T09:43:21.936569+0000","last_became_peered":"2023-10-16T09:43:21.936569+0000","last_unstale":"2023-10-16T09:43:25.950524+0000","last_undegraded":"2023-10-16T09:43:25.950524+0000","last_fullsized":"2023-10-16T09:43:25.950524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948848+0000","last_change":"2023-10-16T09:43:21.935147+0000","last_active":"2023-10-16T09:43:25.948848+0000","last_peered":"2023-10-16T09:43:25.948848+0000","last_clean":"2023-10-16T09:43:25.948848+0000","last_became_active":"2023-10-16T09:43:21.934786+0000","last_became_peered":"2023-10-16T09:43:21.934786+0000","last_unstale":"2023-10-16T09:43:25.948848+0000","last_undegraded":"2023-10-16T09:43:25.948848+0000","last_fullsized":"2023-10-16T09:43:25.948848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945003+0000","last_change":"2023-10-16T09:43:21.927238+0000","last_active":"2023-10-16T09:43:25.945003+0000","last_peered":"2023-10-16T09:43:25.945003+0000","last_clean":"2023-10-16T09:43:25.945003+0000","last_became_active":"2023-10-16T09:43:21.926944+0000","last_became_peered":"2023-10-16T09:43:21.926944+0000","last_unstale":"2023-10-16T09:43:25.945003+0000","last_undegraded":"2023-10-16T09:43:25.945003+0000","last_fullsized":"2023-10-16T09:43:25.945003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015597+0000","last_change":"2023-10-16T09:43:25.972270+0000","last_active":"2023-10-16T09:43:28.015597+0000","last_peered":"2023-10-16T09:43:28.015597+0000","last_clean":"2023-10-16T09:43:28.015597+0000","last_became_active":"2023-10-16T09:43:25.971856+0000","last_became_peered":"2023-10-16T09:43:25.971856+0000","last_unstale":"2023-10-16T09:43:28.015597+0000","last_undegraded":"2023-10-16T09:43:28.015597+0000","last_fullsized":"2023-10-16T09:43:28.015597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:00.427859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951697+0000","last_change":"2023-10-16T09:43:20.934516+0000","last_active":"2023-10-16T09:43:25.951697+0000","last_peered":"2023-10-16T09:43:25.951697+0000","last_clean":"2023-10-16T09:43:25.951697+0000","last_became_active":"2023-10-16T09:43:20.934196+0000","last_became_peered":"2023-10-16T09:43:20.934196+0000","last_unstale":"2023-10-16T09:43:25.951697+0000","last_undegraded":"2023-10-16T09:43:25.951697+0000","last_fullsized":"2023-10-16T09:43:25.951697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966598+0000","last_change":"2023-10-16T09:43:21.941097+0000","last_active":"2023-10-16T09:43:25.966598+0000","last_peered":"2023-10-16T09:43:25.966598+0000","last_clean":"2023-10-16T09:43:25.966598+0000","last_became_active":"2023-10-16T09:43:21.940820+0000","last_became_peered":"2023-10-16T09:43:21.940820+0000","last_unstale":"2023-10-16T09:43:25.966598+0000","last_undegraded":"2023-10-16T09:43:25.966598+0000","last_fullsized":"2023-10-16T09:43:25.966598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947760+0000","last_change":"2023-10-16T09:43:25.970409+0000","last_active":"2023-10-16T09:43:26.947760+0000","last_peered":"2023-10-16T09:43:26.947760+0000","last_clean":"2023-10-16T09:43:26.947760+0000","last_became_active":"2023-10-16T09:43:25.969848+0000","last_became_peered":"2023-10-16T09:43:25.969848+0000","last_unstale":"2023-10-16T09:43:26.947760+0000","last_undegraded":"2023-10-16T09:43:26.947760+0000","last_fullsized":"2023-10-16T09:43:26.947760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:00.972576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966712+0000","last_change":"2023-10-16T09:43:20.933965+0000","last_active":"2023-10-16T09:43:25.966712+0000","last_peered":"2023-10-16T09:43:25.966712+0000","last_clean":"2023-10-16T09:43:25.966712+0000","last_became_active":"2023-10-16T09:43:20.933635+0000","last_became_peered":"2023-10-16T09:43:20.933635+0000","last_unstale":"2023-10-16T09:43:25.966712+0000","last_undegraded":"2023-10-16T09:43:25.966712+0000","last_fullsized":"2023-10-16T09:43:25.966712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951601+0000","last_change":"2023-10-16T09:43:21.937374+0000","last_active":"2023-10-16T09:43:25.951601+0000","last_peered":"2023-10-16T09:43:25.951601+0000","last_clean":"2023-10-16T09:43:25.951601+0000","last_became_active":"2023-10-16T09:43:21.936989+0000","last_became_peered":"2023-10-16T09:43:21.936989+0000","last_unstale":"2023-10-16T09:43:25.951601+0000","last_undegraded":"2023-10-16T09:43:25.951601+0000","last_fullsized":"2023-10-16T09:43:25.951601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948251+0000","last_change":"2023-10-16T09:43:25.970011+0000","last_active":"2023-10-16T09:43:26.948251+0000","last_peered":"2023-10-16T09:43:26.948251+0000","last_clean":"2023-10-16T09:43:26.948251+0000","last_became_active":"2023-10-16T09:43:25.969570+0000","last_became_peered":"2023-10-16T09:43:25.969570+0000","last_unstale":"2023-10-16T09:43:26.948251+0000","last_undegraded":"2023-10-16T09:43:26.948251+0000","last_fullsized":"2023-10-16T09:43:26.948251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944734+0000","last_change":"2023-10-16T09:43:20.928578+0000","last_active":"2023-10-16T09:43:25.944734+0000","last_peered":"2023-10-16T09:43:25.944734+0000","last_clean":"2023-10-16T09:43:25.944734+0000","last_became_active":"2023-10-16T09:43:20.928283+0000","last_became_peered":"2023-10-16T09:43:20.928283+0000","last_unstale":"2023-10-16T09:43:25.944734+0000","last_undegraded":"2023-10-16T09:43:25.944734+0000","last_fullsized":"2023-10-16T09:43:25.944734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950360+0000","last_change":"2023-10-16T09:43:21.936827+0000","last_active":"2023-10-16T09:43:25.950360+0000","last_peered":"2023-10-16T09:43:25.950360+0000","last_clean":"2023-10-16T09:43:25.950360+0000","last_became_active":"2023-10-16T09:43:21.936545+0000","last_became_peered":"2023-10-16T09:43:21.936545+0000","last_unstale":"2023-10-16T09:43:25.950360+0000","last_undegraded":"2023-10-16T09:43:25.950360+0000","last_fullsized":"2023-10-16T09:43:25.950360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947909+0000","last_change":"2023-10-16T09:43:25.968566+0000","last_active":"2023-10-16T09:43:26.947909+0000","last_peered":"2023-10-16T09:43:26.947909+0000","last_clean":"2023-10-16T09:43:26.947909+0000","last_became_active":"2023-10-16T09:43:25.968188+0000","last_became_peered":"2023-10-16T09:43:25.968188+0000","last_unstale":"2023-10-16T09:43:26.947909+0000","last_undegraded":"2023-10-16T09:43:26.947909+0000","last_fullsized":"2023-10-16T09:43:26.947909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:38.074113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944439+0000","last_change":"2023-10-16T09:43:20.929934+0000","last_active":"2023-10-16T09:43:25.944439+0000","last_peered":"2023-10-16T09:43:25.944439+0000","last_clean":"2023-10-16T09:43:25.944439+0000","last_became_active":"2023-10-16T09:43:20.929639+0000","last_became_peered":"2023-10-16T09:43:20.929639+0000","last_unstale":"2023-10-16T09:43:25.944439+0000","last_undegraded":"2023-10-16T09:43:25.944439+0000","last_fullsized":"2023-10-16T09:43:25.944439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953109+0000","last_change":"2023-10-16T09:43:21.933199+0000","last_active":"2023-10-16T09:43:25.953109+0000","last_peered":"2023-10-16T09:43:25.953109+0000","last_clean":"2023-10-16T09:43:25.953109+0000","last_became_active":"2023-10-16T09:43:21.932906+0000","last_became_peered":"2023-10-16T09:43:21.932906+0000","last_unstale":"2023-10-16T09:43:25.953109+0000","last_undegraded":"2023-10-16T09:43:25.953109+0000","last_fullsized":"2023-10-16T09:43:25.953109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016130+0000","last_change":"2023-10-16T09:43:25.975356+0000","last_active":"2023-10-16T09:43:28.016130+0000","last_peered":"2023-10-16T09:43:28.016130+0000","last_clean":"2023-10-16T09:43:28.016130+0000","last_became_active":"2023-10-16T09:43:25.974334+0000","last_became_peered":"2023-10-16T09:43:25.974334+0000","last_unstale":"2023-10-16T09:43:28.016130+0000","last_undegraded":"2023-10-16T09:43:28.016130+0000","last_fullsized":"2023-10-16T09:43:28.016130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:20.175649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966133+0000","last_change":"2023-10-16T09:43:20.928498+0000","last_active":"2023-10-16T09:43:25.966133+0000","last_peered":"2023-10-16T09:43:25.966133+0000","last_clean":"2023-10-16T09:43:25.966133+0000","last_became_active":"2023-10-16T09:43:20.928242+0000","last_became_peered":"2023-10-16T09:43:20.928242+0000","last_unstale":"2023-10-16T09:43:25.966133+0000","last_undegraded":"2023-10-16T09:43:25.966133+0000","last_fullsized":"2023-10-16T09:43:25.966133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948266+0000","last_change":"2023-10-16T09:43:21.919769+0000","last_active":"2023-10-16T09:43:25.948266+0000","last_peered":"2023-10-16T09:43:25.948266+0000","last_clean":"2023-10-16T09:43:25.948266+0000","last_became_active":"2023-10-16T09:43:21.919395+0000","last_became_peered":"2023-10-16T09:43:21.919395+0000","last_unstale":"2023-10-16T09:43:25.948266+0000","last_undegraded":"2023-10-16T09:43:25.948266+0000","last_fullsized":"2023-10-16T09:43:25.948266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:06.700888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.944846+0000","last_change":"2023-10-16T09:43:25.970755+0000","last_active":"2023-10-16T09:43:26.944846+0000","last_peered":"2023-10-16T09:43:26.944846+0000","last_clean":"2023-10-16T09:43:26.944846+0000","last_became_active":"2023-10-16T09:43:25.970344+0000","last_became_peered":"2023-10-16T09:43:25.970344+0000","last_unstale":"2023-10-16T09:43:26.944846+0000","last_undegraded":"2023-10-16T09:43:26.944846+0000","last_fullsized":"2023-10-16T09:43:26.944846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:28.937851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957011+0000","last_change":"2023-10-16T09:43:20.924380+0000","last_active":"2023-10-16T09:43:25.957011+0000","last_peered":"2023-10-16T09:43:25.957011+0000","last_clean":"2023-10-16T09:43:25.957011+0000","last_became_active":"2023-10-16T09:43:20.924104+0000","last_became_peered":"2023-10-16T09:43:20.924104+0000","last_unstale":"2023-10-16T09:43:25.957011+0000","last_undegraded":"2023-10-16T09:43:25.957011+0000","last_fullsized":"2023-10-16T09:43:25.957011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948179+0000","last_change":"2023-10-16T09:43:21.929870+0000","last_active":"2023-10-16T09:43:25.948179+0000","last_peered":"2023-10-16T09:43:25.948179+0000","last_clean":"2023-10-16T09:43:25.948179+0000","last_became_active":"2023-10-16T09:43:21.929589+0000","last_became_peered":"2023-10-16T09:43:21.929589+0000","last_unstale":"2023-10-16T09:43:25.948179+0000","last_undegraded":"2023-10-16T09:43:25.948179+0000","last_fullsized":"2023-10-16T09:43:25.948179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949711+0000","last_change":"2023-10-16T09:43:25.966670+0000","last_active":"2023-10-16T09:43:26.949711+0000","last_peered":"2023-10-16T09:43:26.949711+0000","last_clean":"2023-10-16T09:43:26.949711+0000","last_became_active":"2023-10-16T09:43:25.966312+0000","last_became_peered":"2023-10-16T09:43:25.966312+0000","last_unstale":"2023-10-16T09:43:26.949711+0000","last_undegraded":"2023-10-16T09:43:26.949711+0000","last_fullsized":"2023-10-16T09:43:26.949711+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965835+0000","last_change":"2023-10-16T09:43:20.928383+0000","last_active":"2023-10-16T09:43:25.965835+0000","last_peered":"2023-10-16T09:43:25.965835+0000","last_clean":"2023-10-16T09:43:25.965835+0000","last_became_active":"2023-10-16T09:43:20.928096+0000","last_became_peered":"2023-10-16T09:43:20.928096+0000","last_unstale":"2023-10-16T09:43:25.965835+0000","last_undegraded":"2023-10-16T09:43:25.965835+0000","last_fullsized":"2023-10-16T09:43:25.965835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965672+0000","last_change":"2023-10-16T09:43:21.921213+0000","last_active":"2023-10-16T09:43:25.965672+0000","last_peered":"2023-10-16T09:43:25.965672+0000","last_clean":"2023-10-16T09:43:25.965672+0000","last_became_active":"2023-10-16T09:43:21.920920+0000","last_became_peered":"2023-10-16T09:43:21.920920+0000","last_unstale":"2023-10-16T09:43:25.965672+0000","last_undegraded":"2023-10-16T09:43:25.965672+0000","last_fullsized":"2023-10-16T09:43:25.965672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015970+0000","last_change":"2023-10-16T09:43:25.972702+0000","last_active":"2023-10-16T09:43:28.015970+0000","last_peered":"2023-10-16T09:43:28.015970+0000","last_clean":"2023-10-16T09:43:28.015970+0000","last_became_active":"2023-10-16T09:43:25.971784+0000","last_became_peered":"2023-10-16T09:43:25.971784+0000","last_unstale":"2023-10-16T09:43:28.015970+0000","last_undegraded":"2023-10-16T09:43:28.015970+0000","last_fullsized":"2023-10-16T09:43:28.015970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950235+0000","last_change":"2023-10-16T09:43:20.926495+0000","last_active":"2023-10-16T09:43:25.950235+0000","last_peered":"2023-10-16T09:43:25.950235+0000","last_clean":"2023-10-16T09:43:25.950235+0000","last_became_active":"2023-10-16T09:43:20.925998+0000","last_became_peered":"2023-10-16T09:43:20.925998+0000","last_unstale":"2023-10-16T09:43:25.950235+0000","last_undegraded":"2023-10-16T09:43:25.950235+0000","last_fullsized":"2023-10-16T09:43:25.950235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954132+0000","last_change":"2023-10-16T09:43:21.919652+0000","last_active":"2023-10-16T09:43:25.954132+0000","last_peered":"2023-10-16T09:43:25.954132+0000","last_clean":"2023-10-16T09:43:25.954132+0000","last_became_active":"2023-10-16T09:43:21.919397+0000","last_became_peered":"2023-10-16T09:43:21.919397+0000","last_unstale":"2023-10-16T09:43:25.954132+0000","last_undegraded":"2023-10-16T09:43:25.954132+0000","last_fullsized":"2023-10-16T09:43:25.954132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015120+0000","last_change":"2023-10-16T09:43:25.975457+0000","last_active":"2023-10-16T09:43:28.015120+0000","last_peered":"2023-10-16T09:43:28.015120+0000","last_clean":"2023-10-16T09:43:28.015120+0000","last_became_active":"2023-10-16T09:43:25.975098+0000","last_became_peered":"2023-10-16T09:43:25.975098+0000","last_unstale":"2023-10-16T09:43:28.015120+0000","last_undegraded":"2023-10-16T09:43:28.015120+0000","last_fullsized":"2023-10-16T09:43:28.015120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956270+0000","last_change":"2023-10-16T09:43:20.925087+0000","last_active":"2023-10-16T09:43:25.956270+0000","last_peered":"2023-10-16T09:43:25.956270+0000","last_clean":"2023-10-16T09:43:25.956270+0000","last_became_active":"2023-10-16T09:43:20.924798+0000","last_became_peered":"2023-10-16T09:43:20.924798+0000","last_unstale":"2023-10-16T09:43:25.956270+0000","last_undegraded":"2023-10-16T09:43:25.956270+0000","last_fullsized":"2023-10-16T09:43:25.956270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951125+0000","last_change":"2023-10-16T09:43:21.933170+0000","last_active":"2023-10-16T09:43:25.951125+0000","last_peered":"2023-10-16T09:43:25.951125+0000","last_clean":"2023-10-16T09:43:25.951125+0000","last_became_active":"2023-10-16T09:43:21.932876+0000","last_became_peered":"2023-10-16T09:43:21.932876+0000","last_unstale":"2023-10-16T09:43:25.951125+0000","last_undegraded":"2023-10-16T09:43:25.951125+0000","last_fullsized":"2023-10-16T09:43:25.951125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015261+0000","last_change":"2023-10-16T09:43:25.979174+0000","last_active":"2023-10-16T09:43:28.015261+0000","last_peered":"2023-10-16T09:43:28.015261+0000","last_clean":"2023-10-16T09:43:28.015261+0000","last_became_active":"2023-10-16T09:43:25.977030+0000","last_became_peered":"2023-10-16T09:43:25.977030+0000","last_unstale":"2023-10-16T09:43:28.015261+0000","last_undegraded":"2023-10-16T09:43:28.015261+0000","last_fullsized":"2023-10-16T09:43:28.015261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:47.151340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951034+0000","last_change":"2023-10-16T09:43:20.928828+0000","last_active":"2023-10-16T09:43:25.951034+0000","last_peered":"2023-10-16T09:43:25.951034+0000","last_clean":"2023-10-16T09:43:25.951034+0000","last_became_active":"2023-10-16T09:43:20.928550+0000","last_became_peered":"2023-10-16T09:43:20.928550+0000","last_unstale":"2023-10-16T09:43:25.951034+0000","last_undegraded":"2023-10-16T09:43:25.951034+0000","last_fullsized":"2023-10-16T09:43:25.951034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951780+0000","last_change":"2023-10-16T09:43:21.937614+0000","last_active":"2023-10-16T09:43:25.951780+0000","last_peered":"2023-10-16T09:43:25.951780+0000","last_clean":"2023-10-16T09:43:25.951780+0000","last_became_active":"2023-10-16T09:43:21.937335+0000","last_became_peered":"2023-10-16T09:43:21.937335+0000","last_unstale":"2023-10-16T09:43:25.951780+0000","last_undegraded":"2023-10-16T09:43:25.951780+0000","last_fullsized":"2023-10-16T09:43:25.951780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948709+0000","last_change":"2023-10-16T09:43:25.958049+0000","last_active":"2023-10-16T09:43:26.948709+0000","last_peered":"2023-10-16T09:43:26.948709+0000","last_clean":"2023-10-16T09:43:26.948709+0000","last_became_active":"2023-10-16T09:43:25.957672+0000","last_became_peered":"2023-10-16T09:43:25.957672+0000","last_unstale":"2023-10-16T09:43:26.948709+0000","last_undegraded":"2023-10-16T09:43:26.948709+0000","last_fullsized":"2023-10-16T09:43:26.948709+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:53.105902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955629+0000","last_change":"2023-10-16T09:43:20.926834+0000","last_active":"2023-10-16T09:43:25.955629+0000","last_peered":"2023-10-16T09:43:25.955629+0000","last_clean":"2023-10-16T09:43:25.955629+0000","last_became_active":"2023-10-16T09:43:20.926593+0000","last_became_peered":"2023-10-16T09:43:20.926593+0000","last_unstale":"2023-10-16T09:43:25.955629+0000","last_undegraded":"2023-10-16T09:43:25.955629+0000","last_fullsized":"2023-10-16T09:43:25.955629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947243+0000","last_change":"2023-10-16T09:43:21.925181+0000","last_active":"2023-10-16T09:43:25.947243+0000","last_peered":"2023-10-16T09:43:25.947243+0000","last_clean":"2023-10-16T09:43:25.947243+0000","last_became_active":"2023-10-16T09:43:21.924083+0000","last_became_peered":"2023-10-16T09:43:21.924083+0000","last_unstale":"2023-10-16T09:43:25.947243+0000","last_undegraded":"2023-10-16T09:43:25.947243+0000","last_fullsized":"2023-10-16T09:43:25.947243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:41.369421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995287+0000","last_change":"2023-10-16T09:43:25.960604+0000","last_active":"2023-10-16T09:43:27.995287+0000","last_peered":"2023-10-16T09:43:27.995287+0000","last_clean":"2023-10-16T09:43:27.995287+0000","last_became_active":"2023-10-16T09:43:25.960212+0000","last_became_peered":"2023-10-16T09:43:25.960212+0000","last_unstale":"2023-10-16T09:43:27.995287+0000","last_undegraded":"2023-10-16T09:43:27.995287+0000","last_fullsized":"2023-10-16T09:43:27.995287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:31.688570+0000","last_change":"2023-10-16T09:43:20.927651+0000","last_active":"2023-10-16T09:43:31.688570+0000","last_peered":"2023-10-16T09:43:31.688570+0000","last_clean":"2023-10-16T09:43:31.688570+0000","last_became_active":"2023-10-16T09:43:20.927393+0000","last_became_peered":"2023-10-16T09:43:20.927393+0000","last_unstale":"2023-10-16T09:43:31.688570+0000","last_undegraded":"2023-10-16T09:43:31.688570+0000","last_fullsized":"2023-10-16T09:43:31.688570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944933+0000","last_change":"2023-10-16T09:43:21.922547+0000","last_active":"2023-10-16T09:43:25.944933+0000","last_peered":"2023-10-16T09:43:25.944933+0000","last_clean":"2023-10-16T09:43:25.944933+0000","last_became_active":"2023-10-16T09:43:21.922309+0000","last_became_peered":"2023-10-16T09:43:21.922309+0000","last_unstale":"2023-10-16T09:43:25.944933+0000","last_undegraded":"2023-10-16T09:43:25.944933+0000","last_fullsized":"2023-10-16T09:43:25.944933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016142+0000","last_change":"2023-10-16T09:43:25.973055+0000","last_active":"2023-10-16T09:43:28.016142+0000","last_peered":"2023-10-16T09:43:28.016142+0000","last_clean":"2023-10-16T09:43:28.016142+0000","last_became_active":"2023-10-16T09:43:25.972664+0000","last_became_peered":"2023-10-16T09:43:25.972664+0000","last_unstale":"2023-10-16T09:43:28.016142+0000","last_undegraded":"2023-10-16T09:43:28.016142+0000","last_fullsized":"2023-10-16T09:43:28.016142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948288+0000","last_change":"2023-10-16T09:43:20.933659+0000","last_active":"2023-10-16T09:43:25.948288+0000","last_peered":"2023-10-16T09:43:25.948288+0000","last_clean":"2023-10-16T09:43:25.948288+0000","last_became_active":"2023-10-16T09:43:20.933376+0000","last_became_peered":"2023-10-16T09:43:20.933376+0000","last_unstale":"2023-10-16T09:43:25.948288+0000","last_undegraded":"2023-10-16T09:43:25.948288+0000","last_fullsized":"2023-10-16T09:43:25.948288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948438+0000","last_change":"2023-10-16T09:43:21.925525+0000","last_active":"2023-10-16T09:43:25.948438+0000","last_peered":"2023-10-16T09:43:25.948438+0000","last_clean":"2023-10-16T09:43:25.948438+0000","last_became_active":"2023-10-16T09:43:21.925269+0000","last_became_peered":"2023-10-16T09:43:21.925269+0000","last_unstale":"2023-10-16T09:43:25.948438+0000","last_undegraded":"2023-10-16T09:43:25.948438+0000","last_fullsized":"2023-10-16T09:43:25.948438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:39.205632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016287+0000","last_change":"2023-10-16T09:43:25.974917+0000","last_active":"2023-10-16T09:43:28.016287+0000","last_peered":"2023-10-16T09:43:28.016287+0000","last_clean":"2023-10-16T09:43:28.016287+0000","last_became_active":"2023-10-16T09:43:25.974512+0000","last_became_peered":"2023-10-16T09:43:25.974512+0000","last_unstale":"2023-10-16T09:43:28.016287+0000","last_undegraded":"2023-10-16T09:43:28.016287+0000","last_fullsized":"2023-10-16T09:43:28.016287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:24.186922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950390+0000","last_change":"2023-10-16T09:43:20.927369+0000","last_active":"2023-10-16T09:43:25.950390+0000","last_peered":"2023-10-16T09:43:25.950390+0000","last_clean":"2023-10-16T09:43:25.950390+0000","last_became_active":"2023-10-16T09:43:20.927043+0000","last_became_peered":"2023-10-16T09:43:20.927043+0000","last_unstale":"2023-10-16T09:43:25.950390+0000","last_undegraded":"2023-10-16T09:43:25.950390+0000","last_fullsized":"2023-10-16T09:43:25.950390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:11.929994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945029+0000","last_change":"2023-10-16T09:43:21.930235+0000","last_active":"2023-10-16T09:43:25.945029+0000","last_peered":"2023-10-16T09:43:25.945029+0000","last_clean":"2023-10-16T09:43:25.945029+0000","last_became_active":"2023-10-16T09:43:21.929925+0000","last_became_peered":"2023-10-16T09:43:21.929925+0000","last_unstale":"2023-10-16T09:43:25.945029+0000","last_undegraded":"2023-10-16T09:43:25.945029+0000","last_fullsized":"2023-10-16T09:43:25.945029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016497+0000","last_change":"2023-10-16T09:43:25.975868+0000","last_active":"2023-10-16T09:43:28.016497+0000","last_peered":"2023-10-16T09:43:28.016497+0000","last_clean":"2023-10-16T09:43:28.016497+0000","last_became_active":"2023-10-16T09:43:25.975476+0000","last_became_peered":"2023-10-16T09:43:25.975476+0000","last_unstale":"2023-10-16T09:43:28.016497+0000","last_undegraded":"2023-10-16T09:43:28.016497+0000","last_fullsized":"2023-10-16T09:43:28.016497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:36.344621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954142+0000","last_change":"2023-10-16T09:43:20.922948+0000","last_active":"2023-10-16T09:43:25.954142+0000","last_peered":"2023-10-16T09:43:25.954142+0000","last_clean":"2023-10-16T09:43:25.954142+0000","last_became_active":"2023-10-16T09:43:20.922660+0000","last_became_peered":"2023-10-16T09:43:20.922660+0000","last_unstale":"2023-10-16T09:43:25.954142+0000","last_undegraded":"2023-10-16T09:43:25.954142+0000","last_fullsized":"2023-10-16T09:43:25.954142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:24.786762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953944+0000","last_change":"2023-10-16T09:43:21.930450+0000","last_active":"2023-10-16T09:43:25.953944+0000","last_peered":"2023-10-16T09:43:25.953944+0000","last_clean":"2023-10-16T09:43:25.953944+0000","last_became_active":"2023-10-16T09:43:21.929848+0000","last_became_peered":"2023-10-16T09:43:21.929848+0000","last_unstale":"2023-10-16T09:43:25.953944+0000","last_undegraded":"2023-10-16T09:43:25.953944+0000","last_fullsized":"2023-10-16T09:43:25.953944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015695+0000","last_change":"2023-10-16T09:43:25.972951+0000","last_active":"2023-10-16T09:43:28.015695+0000","last_peered":"2023-10-16T09:43:28.015695+0000","last_clean":"2023-10-16T09:43:28.015695+0000","last_became_active":"2023-10-16T09:43:25.972321+0000","last_became_peered":"2023-10-16T09:43:25.972321+0000","last_unstale":"2023-10-16T09:43:28.015695+0000","last_undegraded":"2023-10-16T09:43:28.015695+0000","last_fullsized":"2023-10-16T09:43:28.015695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948561+0000","last_change":"2023-10-16T09:43:20.919381+0000","last_active":"2023-10-16T09:43:25.948561+0000","last_peered":"2023-10-16T09:43:25.948561+0000","last_clean":"2023-10-16T09:43:25.948561+0000","last_became_active":"2023-10-16T09:43:20.919099+0000","last_became_peered":"2023-10-16T09:43:20.919099+0000","last_unstale":"2023-10-16T09:43:25.948561+0000","last_undegraded":"2023-10-16T09:43:25.948561+0000","last_fullsized":"2023-10-16T09:43:25.948561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.153559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944510+0000","last_change":"2023-10-16T09:43:21.930254+0000","last_active":"2023-10-16T09:43:25.944510+0000","last_peered":"2023-10-16T09:43:25.944510+0000","last_clean":"2023-10-16T09:43:25.944510+0000","last_became_active":"2023-10-16T09:43:21.929903+0000","last_became_peered":"2023-10-16T09:43:21.929903+0000","last_unstale":"2023-10-16T09:43:25.944510+0000","last_undegraded":"2023-10-16T09:43:25.944510+0000","last_fullsized":"2023-10-16T09:43:25.944510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947704+0000","last_change":"2023-10-16T09:43:25.973595+0000","last_active":"2023-10-16T09:43:26.947704+0000","last_peered":"2023-10-16T09:43:26.947704+0000","last_clean":"2023-10-16T09:43:26.947704+0000","last_became_active":"2023-10-16T09:43:25.972877+0000","last_became_peered":"2023-10-16T09:43:25.972877+0000","last_unstale":"2023-10-16T09:43:26.947704+0000","last_undegraded":"2023-10-16T09:43:26.947704+0000","last_fullsized":"2023-10-16T09:43:26.947704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:59.042186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.946744+0000","last_change":"2023-10-16T09:43:20.933953+0000","last_active":"2023-10-16T09:43:25.946744+0000","last_peered":"2023-10-16T09:43:25.946744+0000","last_clean":"2023-10-16T09:43:25.946744+0000","last_became_active":"2023-10-16T09:43:20.933628+0000","last_became_peered":"2023-10-16T09:43:20.933628+0000","last_unstale":"2023-10-16T09:43:25.946744+0000","last_undegraded":"2023-10-16T09:43:25.946744+0000","last_fullsized":"2023-10-16T09:43:25.946744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:06.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944812+0000","last_change":"2023-10-16T09:43:21.922710+0000","last_active":"2023-10-16T09:43:25.944812+0000","last_peered":"2023-10-16T09:43:25.944812+0000","last_clean":"2023-10-16T09:43:25.944812+0000","last_became_active":"2023-10-16T09:43:21.921824+0000","last_became_peered":"2023-10-16T09:43:21.921824+0000","last_unstale":"2023-10-16T09:43:25.944812+0000","last_undegraded":"2023-10-16T09:43:25.944812+0000","last_fullsized":"2023-10-16T09:43:25.944812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995171+0000","last_change":"2023-10-16T09:43:25.960646+0000","last_active":"2023-10-16T09:43:27.995171+0000","last_peered":"2023-10-16T09:43:27.995171+0000","last_clean":"2023-10-16T09:43:27.995171+0000","last_became_active":"2023-10-16T09:43:25.960270+0000","last_became_peered":"2023-10-16T09:43:25.960270+0000","last_unstale":"2023-10-16T09:43:27.995171+0000","last_undegraded":"2023-10-16T09:43:27.995171+0000","last_fullsized":"2023-10-16T09:43:27.995171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955611+0000","last_change":"2023-10-16T09:43:20.935197+0000","last_active":"2023-10-16T09:43:25.955611+0000","last_peered":"2023-10-16T09:43:25.955611+0000","last_clean":"2023-10-16T09:43:25.955611+0000","last_became_active":"2023-10-16T09:43:20.934929+0000","last_became_peered":"2023-10-16T09:43:20.934929+0000","last_unstale":"2023-10-16T09:43:25.955611+0000","last_undegraded":"2023-10-16T09:43:25.955611+0000","last_fullsized":"2023-10-16T09:43:25.955611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953179+0000","last_change":"2023-10-16T09:43:21.936959+0000","last_active":"2023-10-16T09:43:25.953179+0000","last_peered":"2023-10-16T09:43:25.953179+0000","last_clean":"2023-10-16T09:43:25.953179+0000","last_became_active":"2023-10-16T09:43:21.932598+0000","last_became_peered":"2023-10-16T09:43:21.932598+0000","last_unstale":"2023-10-16T09:43:25.953179+0000","last_undegraded":"2023-10-16T09:43:25.953179+0000","last_fullsized":"2023-10-16T09:43:25.953179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995359+0000","last_change":"2023-10-16T09:43:25.976441+0000","last_active":"2023-10-16T09:43:27.995359+0000","last_peered":"2023-10-16T09:43:27.995359+0000","last_clean":"2023-10-16T09:43:27.995359+0000","last_became_active":"2023-10-16T09:43:25.975914+0000","last_became_peered":"2023-10-16T09:43:25.975914+0000","last_unstale":"2023-10-16T09:43:27.995359+0000","last_undegraded":"2023-10-16T09:43:27.995359+0000","last_fullsized":"2023-10-16T09:43:27.995359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950296+0000","last_change":"2023-10-16T09:43:20.925613+0000","last_active":"2023-10-16T09:43:25.950296+0000","last_peered":"2023-10-16T09:43:25.950296+0000","last_clean":"2023-10-16T09:43:25.950296+0000","last_became_active":"2023-10-16T09:43:20.925345+0000","last_became_peered":"2023-10-16T09:43:20.925345+0000","last_unstale":"2023-10-16T09:43:25.950296+0000","last_undegraded":"2023-10-16T09:43:25.950296+0000","last_fullsized":"2023-10-16T09:43:25.950296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:28.608301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948410+0000","last_change":"2023-10-16T09:43:21.925702+0000","last_active":"2023-10-16T09:43:25.948410+0000","last_peered":"2023-10-16T09:43:25.948410+0000","last_clean":"2023-10-16T09:43:25.948410+0000","last_became_active":"2023-10-16T09:43:21.925003+0000","last_became_peered":"2023-10-16T09:43:21.925003+0000","last_unstale":"2023-10-16T09:43:25.948410+0000","last_undegraded":"2023-10-16T09:43:25.948410+0000","last_fullsized":"2023-10-16T09:43:25.948410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995813+0000","last_change":"2023-10-16T09:43:25.976511+0000","last_active":"2023-10-16T09:43:27.995813+0000","last_peered":"2023-10-16T09:43:27.995813+0000","last_clean":"2023-10-16T09:43:27.995813+0000","last_became_active":"2023-10-16T09:43:25.975258+0000","last_became_peered":"2023-10-16T09:43:25.975258+0000","last_unstale":"2023-10-16T09:43:27.995813+0000","last_undegraded":"2023-10-16T09:43:27.995813+0000","last_fullsized":"2023-10-16T09:43:27.995813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953939+0000","last_change":"2023-10-16T09:43:21.924235+0000","last_active":"2023-10-16T09:43:25.953939+0000","last_peered":"2023-10-16T09:43:25.953939+0000","last_clean":"2023-10-16T09:43:25.953939+0000","last_became_active":"2023-10-16T09:43:21.923963+0000","last_became_peered":"2023-10-16T09:43:21.923963+0000","last_unstale":"2023-10-16T09:43:25.953939+0000","last_undegraded":"2023-10-16T09:43:25.953939+0000","last_fullsized":"2023-10-16T09:43:25.953939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948171+0000","last_change":"2023-10-16T09:43:20.922462+0000","last_active":"2023-10-16T09:43:25.948171+0000","last_peered":"2023-10-16T09:43:25.948171+0000","last_clean":"2023-10-16T09:43:25.948171+0000","last_became_active":"2023-10-16T09:43:20.922204+0000","last_became_peered":"2023-10-16T09:43:20.922204+0000","last_unstale":"2023-10-16T09:43:25.948171+0000","last_undegraded":"2023-10-16T09:43:25.948171+0000","last_fullsized":"2023-10-16T09:43:25.948171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949005+0000","last_change":"2023-10-16T09:43:25.975066+0000","last_active":"2023-10-16T09:43:26.949005+0000","last_peered":"2023-10-16T09:43:26.949005+0000","last_clean":"2023-10-16T09:43:26.949005+0000","last_became_active":"2023-10-16T09:43:25.974685+0000","last_became_peered":"2023-10-16T09:43:25.974685+0000","last_unstale":"2023-10-16T09:43:26.949005+0000","last_undegraded":"2023-10-16T09:43:26.949005+0000","last_fullsized":"2023-10-16T09:43:26.949005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966726+0000","last_change":"2023-10-16T09:43:21.940037+0000","last_active":"2023-10-16T09:43:25.966726+0000","last_peered":"2023-10-16T09:43:25.966726+0000","last_clean":"2023-10-16T09:43:25.966726+0000","last_became_active":"2023-10-16T09:43:21.939645+0000","last_became_peered":"2023-10-16T09:43:21.939645+0000","last_unstale":"2023-10-16T09:43:25.966726+0000","last_undegraded":"2023-10-16T09:43:25.966726+0000","last_fullsized":"2023-10-16T09:43:25.966726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950263+0000","last_change":"2023-10-16T09:43:20.925442+0000","last_active":"2023-10-16T09:43:25.950263+0000","last_peered":"2023-10-16T09:43:25.950263+0000","last_clean":"2023-10-16T09:43:25.950263+0000","last_became_active":"2023-10-16T09:43:20.925129+0000","last_became_peered":"2023-10-16T09:43:20.925129+0000","last_unstale":"2023-10-16T09:43:25.950263+0000","last_undegraded":"2023-10-16T09:43:25.950263+0000","last_fullsized":"2023-10-16T09:43:25.950263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015775+0000","last_change":"2023-10-16T09:43:25.961486+0000","last_active":"2023-10-16T09:43:28.015775+0000","last_peered":"2023-10-16T09:43:28.015775+0000","last_clean":"2023-10-16T09:43:28.015775+0000","last_became_active":"2023-10-16T09:43:25.961120+0000","last_became_peered":"2023-10-16T09:43:25.961120+0000","last_unstale":"2023-10-16T09:43:28.015775+0000","last_undegraded":"2023-10-16T09:43:28.015775+0000","last_fullsized":"2023-10-16T09:43:28.015775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947444+0000","last_change":"2023-10-16T09:43:21.929303+0000","last_active":"2023-10-16T09:43:25.947444+0000","last_peered":"2023-10-16T09:43:25.947444+0000","last_clean":"2023-10-16T09:43:25.947444+0000","last_became_active":"2023-10-16T09:43:21.929017+0000","last_became_peered":"2023-10-16T09:43:21.929017+0000","last_unstale":"2023-10-16T09:43:25.947444+0000","last_undegraded":"2023-10-16T09:43:25.947444+0000","last_fullsized":"2023-10-16T09:43:25.947444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:45.433460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:20.925419+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:20.924900+0000","last_became_peered":"2023-10-16T09:43:20.924900+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.960836+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.959879+0000","last_became_peered":"2023-10-16T09:43:25.959879+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944657+0000","last_change":"2023-10-16T09:43:20.935065+0000","last_active":"2023-10-16T09:43:25.944657+0000","last_peered":"2023-10-16T09:43:25.944657+0000","last_clean":"2023-10-16T09:43:25.944657+0000","last_became_active":"2023-10-16T09:43:20.934870+0000","last_became_peered":"2023-10-16T09:43:20.934870+0000","last_unstale":"2023-10-16T09:43:25.944657+0000","last_undegraded":"2023-10-16T09:43:25.944657+0000","last_fullsized":"2023-10-16T09:43:25.944657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955720+0000","last_change":"2023-10-16T09:43:21.924250+0000","last_active":"2023-10-16T09:43:25.955720+0000","last_peered":"2023-10-16T09:43:25.955720+0000","last_clean":"2023-10-16T09:43:25.955720+0000","last_became_active":"2023-10-16T09:43:21.923974+0000","last_became_peered":"2023-10-16T09:43:21.923974+0000","last_unstale":"2023-10-16T09:43:25.955720+0000","last_undegraded":"2023-10-16T09:43:25.955720+0000","last_fullsized":"2023-10-16T09:43:25.955720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.974974+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.974574+0000","last_became_peered":"2023-10-16T09:43:25.974574+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955603+0000","last_change":"2023-10-16T09:43:20.928340+0000","last_active":"2023-10-16T09:43:25.955603+0000","last_peered":"2023-10-16T09:43:25.955603+0000","last_clean":"2023-10-16T09:43:25.955603+0000","last_became_active":"2023-10-16T09:43:20.927748+0000","last_became_peered":"2023-10-16T09:43:20.927748+0000","last_unstale":"2023-10-16T09:43:25.955603+0000","last_undegraded":"2023-10-16T09:43:25.955603+0000","last_fullsized":"2023-10-16T09:43:25.955603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967050+0000","last_change":"2023-10-16T09:43:21.919707+0000","last_active":"2023-10-16T09:43:25.967050+0000","last_peered":"2023-10-16T09:43:25.967050+0000","last_clean":"2023-10-16T09:43:25.967050+0000","last_became_active":"2023-10-16T09:43:21.919454+0000","last_became_peered":"2023-10-16T09:43:21.919454+0000","last_unstale":"2023-10-16T09:43:25.967050+0000","last_undegraded":"2023-10-16T09:43:25.967050+0000","last_fullsized":"2023-10-16T09:43:25.967050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947855+0000","last_change":"2023-10-16T09:43:25.961225+0000","last_active":"2023-10-16T09:43:26.947855+0000","last_peered":"2023-10-16T09:43:26.947855+0000","last_clean":"2023-10-16T09:43:26.947855+0000","last_became_active":"2023-10-16T09:43:25.960336+0000","last_became_peered":"2023-10-16T09:43:25.960336+0000","last_unstale":"2023-10-16T09:43:26.947855+0000","last_undegraded":"2023-10-16T09:43:26.947855+0000","last_fullsized":"2023-10-16T09:43:26.947855+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:15.617593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956170+0000","last_change":"2023-10-16T09:43:20.928799+0000","last_active":"2023-10-16T09:43:25.956170+0000","last_peered":"2023-10-16T09:43:25.956170+0000","last_clean":"2023-10-16T09:43:25.956170+0000","last_became_active":"2023-10-16T09:43:20.928302+0000","last_became_peered":"2023-10-16T09:43:20.928302+0000","last_unstale":"2023-10-16T09:43:25.956170+0000","last_undegraded":"2023-10-16T09:43:25.956170+0000","last_fullsized":"2023-10-16T09:43:25.956170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951319+0000","last_change":"2023-10-16T09:43:21.924602+0000","last_active":"2023-10-16T09:43:25.951319+0000","last_peered":"2023-10-16T09:43:25.951319+0000","last_clean":"2023-10-16T09:43:25.951319+0000","last_became_active":"2023-10-16T09:43:21.924344+0000","last_became_peered":"2023-10-16T09:43:21.924344+0000","last_unstale":"2023-10-16T09:43:25.951319+0000","last_undegraded":"2023-10-16T09:43:25.951319+0000","last_fullsized":"2023-10-16T09:43:25.951319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948047+0000","last_change":"2023-10-16T09:43:25.969751+0000","last_active":"2023-10-16T09:43:26.948047+0000","last_peered":"2023-10-16T09:43:26.948047+0000","last_clean":"2023-10-16T09:43:26.948047+0000","last_became_active":"2023-10-16T09:43:25.969399+0000","last_became_peered":"2023-10-16T09:43:25.969399+0000","last_unstale":"2023-10-16T09:43:26.948047+0000","last_undegraded":"2023-10-16T09:43:26.948047+0000","last_fullsized":"2023-10-16T09:43:26.948047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:20.933882+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:20.933615+0000","last_became_peered":"2023-10-16T09:43:20.933615+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947408+0000","last_change":"2023-10-16T09:43:21.929135+0000","last_active":"2023-10-16T09:43:25.947408+0000","last_peered":"2023-10-16T09:43:25.947408+0000","last_clean":"2023-10-16T09:43:25.947408+0000","last_became_active":"2023-10-16T09:43:21.928849+0000","last_became_peered":"2023-10-16T09:43:21.928849+0000","last_unstale":"2023-10-16T09:43:25.947408+0000","last_undegraded":"2023-10-16T09:43:25.947408+0000","last_fullsized":"2023-10-16T09:43:25.947408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995596+0000","last_change":"2023-10-16T09:43:25.976682+0000","last_active":"2023-10-16T09:43:27.995596+0000","last_peered":"2023-10-16T09:43:27.995596+0000","last_clean":"2023-10-16T09:43:27.995596+0000","last_became_active":"2023-10-16T09:43:25.976274+0000","last_became_peered":"2023-10-16T09:43:25.976274+0000","last_unstale":"2023-10-16T09:43:27.995596+0000","last_undegraded":"2023-10-16T09:43:27.995596+0000","last_fullsized":"2023-10-16T09:43:27.995596+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954044+0000","last_change":"2023-10-16T09:43:20.926509+0000","last_active":"2023-10-16T09:43:25.954044+0000","last_peered":"2023-10-16T09:43:25.954044+0000","last_clean":"2023-10-16T09:43:25.954044+0000","last_became_active":"2023-10-16T09:43:20.926228+0000","last_became_peered":"2023-10-16T09:43:20.926228+0000","last_unstale":"2023-10-16T09:43:25.954044+0000","last_undegraded":"2023-10-16T09:43:25.954044+0000","last_fullsized":"2023-10-16T09:43:25.954044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965799+0000","last_change":"2023-10-16T09:43:21.924014+0000","last_active":"2023-10-16T09:43:25.965799+0000","last_peered":"2023-10-16T09:43:25.965799+0000","last_clean":"2023-10-16T09:43:25.965799+0000","last_became_active":"2023-10-16T09:43:21.922684+0000","last_became_peered":"2023-10-16T09:43:21.922684+0000","last_unstale":"2023-10-16T09:43:25.965799+0000","last_undegraded":"2023-10-16T09:43:25.965799+0000","last_fullsized":"2023-10-16T09:43:25.965799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948333+0000","last_change":"2023-10-16T09:43:25.970563+0000","last_active":"2023-10-16T09:43:26.948333+0000","last_peered":"2023-10-16T09:43:26.948333+0000","last_clean":"2023-10-16T09:43:26.948333+0000","last_became_active":"2023-10-16T09:43:25.970168+0000","last_became_peered":"2023-10-16T09:43:25.970168+0000","last_unstale":"2023-10-16T09:43:26.948333+0000","last_undegraded":"2023-10-16T09:43:26.948333+0000","last_fullsized":"2023-10-16T09:43:26.948333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:56.567461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953937+0000","last_change":"2023-10-16T09:43:20.923587+0000","last_active":"2023-10-16T09:43:25.953937+0000","last_peered":"2023-10-16T09:43:25.953937+0000","last_clean":"2023-10-16T09:43:25.953937+0000","last_became_active":"2023-10-16T09:43:20.923277+0000","last_became_peered":"2023-10-16T09:43:20.923277+0000","last_unstale":"2023-10-16T09:43:25.953937+0000","last_undegraded":"2023-10-16T09:43:25.953937+0000","last_fullsized":"2023-10-16T09:43:25.953937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966226+0000","last_change":"2023-10-16T09:43:21.925711+0000","last_active":"2023-10-16T09:43:25.966226+0000","last_peered":"2023-10-16T09:43:25.966226+0000","last_clean":"2023-10-16T09:43:25.966226+0000","last_became_active":"2023-10-16T09:43:21.921509+0000","last_became_peered":"2023-10-16T09:43:21.921509+0000","last_unstale":"2023-10-16T09:43:25.966226+0000","last_undegraded":"2023-10-16T09:43:25.966226+0000","last_fullsized":"2023-10-16T09:43:25.966226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948182+0000","last_change":"2023-10-16T09:43:25.955876+0000","last_active":"2023-10-16T09:43:26.948182+0000","last_peered":"2023-10-16T09:43:26.948182+0000","last_clean":"2023-10-16T09:43:26.948182+0000","last_became_active":"2023-10-16T09:43:25.955490+0000","last_became_peered":"2023-10-16T09:43:25.955490+0000","last_unstale":"2023-10-16T09:43:26.948182+0000","last_undegraded":"2023-10-16T09:43:26.948182+0000","last_fullsized":"2023-10-16T09:43:26.948182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945113+0000","last_change":"2023-10-16T09:43:20.926395+0000","last_active":"2023-10-16T09:43:25.945113+0000","last_peered":"2023-10-16T09:43:25.945113+0000","last_clean":"2023-10-16T09:43:25.945113+0000","last_became_active":"2023-10-16T09:43:20.926086+0000","last_became_peered":"2023-10-16T09:43:20.926086+0000","last_unstale":"2023-10-16T09:43:25.945113+0000","last_undegraded":"2023-10-16T09:43:25.945113+0000","last_fullsized":"2023-10-16T09:43:25.945113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:56.924185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957128+0000","last_change":"2023-10-16T09:43:21.932460+0000","last_active":"2023-10-16T09:43:25.957128+0000","last_peered":"2023-10-16T09:43:25.957128+0000","last_clean":"2023-10-16T09:43:25.957128+0000","last_became_active":"2023-10-16T09:43:21.932184+0000","last_became_peered":"2023-10-16T09:43:21.932184+0000","last_unstale":"2023-10-16T09:43:25.957128+0000","last_undegraded":"2023-10-16T09:43:25.957128+0000","last_fullsized":"2023-10-16T09:43:25.957128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948895+0000","last_change":"2023-10-16T09:43:25.967183+0000","last_active":"2023-10-16T09:43:26.948895+0000","last_peered":"2023-10-16T09:43:26.948895+0000","last_clean":"2023-10-16T09:43:26.948895+0000","last_became_active":"2023-10-16T09:43:25.966759+0000","last_became_peered":"2023-10-16T09:43:25.966759+0000","last_unstale":"2023-10-16T09:43:26.948895+0000","last_undegraded":"2023-10-16T09:43:26.948895+0000","last_fullsized":"2023-10-16T09:43:26.948895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948721+0000","last_change":"2023-10-16T09:43:20.925234+0000","last_active":"2023-10-16T09:43:25.948721+0000","last_peered":"2023-10-16T09:43:25.948721+0000","last_clean":"2023-10-16T09:43:25.948721+0000","last_became_active":"2023-10-16T09:43:20.924957+0000","last_became_peered":"2023-10-16T09:43:20.924957+0000","last_unstale":"2023-10-16T09:43:25.948721+0000","last_undegraded":"2023-10-16T09:43:25.948721+0000","last_fullsized":"2023-10-16T09:43:25.948721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:00.084223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944746+0000","last_change":"2023-10-16T09:43:21.936106+0000","last_active":"2023-10-16T09:43:25.944746+0000","last_peered":"2023-10-16T09:43:25.944746+0000","last_clean":"2023-10-16T09:43:25.944746+0000","last_became_active":"2023-10-16T09:43:21.935859+0000","last_became_peered":"2023-10-16T09:43:21.935859+0000","last_unstale":"2023-10-16T09:43:25.944746+0000","last_undegraded":"2023-10-16T09:43:25.944746+0000","last_fullsized":"2023-10-16T09:43:25.944746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.014901+0000","last_change":"2023-10-16T09:43:25.961369+0000","last_active":"2023-10-16T09:43:28.014901+0000","last_peered":"2023-10-16T09:43:28.014901+0000","last_clean":"2023-10-16T09:43:28.014901+0000","last_became_active":"2023-10-16T09:43:25.960465+0000","last_became_peered":"2023-10-16T09:43:25.960465+0000","last_unstale":"2023-10-16T09:43:28.014901+0000","last_undegraded":"2023-10-16T09:43:28.014901+0000","last_fullsized":"2023-10-16T09:43:28.014901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:40.952070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955514+0000","last_change":"2023-10-16T09:43:20.922927+0000","last_active":"2023-10-16T09:43:25.955514+0000","last_peered":"2023-10-16T09:43:25.955514+0000","last_clean":"2023-10-16T09:43:25.955514+0000","last_became_active":"2023-10-16T09:43:20.922658+0000","last_became_peered":"2023-10-16T09:43:20.922658+0000","last_unstale":"2023-10-16T09:43:25.955514+0000","last_undegraded":"2023-10-16T09:43:25.955514+0000","last_fullsized":"2023-10-16T09:43:25.955514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950404+0000","last_change":"2023-10-16T09:43:21.928420+0000","last_active":"2023-10-16T09:43:25.950404+0000","last_peered":"2023-10-16T09:43:25.950404+0000","last_clean":"2023-10-16T09:43:25.950404+0000","last_became_active":"2023-10-16T09:43:21.928180+0000","last_became_peered":"2023-10-16T09:43:21.928180+0000","last_unstale":"2023-10-16T09:43:25.950404+0000","last_undegraded":"2023-10-16T09:43:25.950404+0000","last_fullsized":"2023-10-16T09:43:25.950404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015214+0000","last_change":"2023-10-16T09:43:25.960304+0000","last_active":"2023-10-16T09:43:28.015214+0000","last_peered":"2023-10-16T09:43:28.015214+0000","last_clean":"2023-10-16T09:43:28.015214+0000","last_became_active":"2023-10-16T09:43:25.959916+0000","last_became_peered":"2023-10-16T09:43:25.959916+0000","last_unstale":"2023-10-16T09:43:28.015214+0000","last_undegraded":"2023-10-16T09:43:28.015214+0000","last_fullsized":"2023-10-16T09:43:28.015214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948747+0000","last_change":"2023-10-16T09:43:20.921666+0000","last_active":"2023-10-16T09:43:25.948747+0000","last_peered":"2023-10-16T09:43:25.948747+0000","last_clean":"2023-10-16T09:43:25.948747+0000","last_became_active":"2023-10-16T09:43:20.921373+0000","last_became_peered":"2023-10-16T09:43:20.921373+0000","last_unstale":"2023-10-16T09:43:25.948747+0000","last_undegraded":"2023-10-16T09:43:25.948747+0000","last_fullsized":"2023-10-16T09:43:25.948747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944941+0000","last_change":"2023-10-16T09:43:21.930965+0000","last_active":"2023-10-16T09:43:25.944941+0000","last_peered":"2023-10-16T09:43:25.944941+0000","last_clean":"2023-10-16T09:43:25.944941+0000","last_became_active":"2023-10-16T09:43:21.930701+0000","last_became_peered":"2023-10-16T09:43:21.930701+0000","last_unstale":"2023-10-16T09:43:25.944941+0000","last_undegraded":"2023-10-16T09:43:25.944941+0000","last_fullsized":"2023-10-16T09:43:25.944941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:20.329096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954036+0000","last_change":"2023-10-16T09:43:21.930641+0000","last_active":"2023-10-16T09:43:25.954036+0000","last_peered":"2023-10-16T09:43:25.954036+0000","last_clean":"2023-10-16T09:43:25.954036+0000","last_became_active":"2023-10-16T09:43:21.930194+0000","last_became_peered":"2023-10-16T09:43:21.930194+0000","last_unstale":"2023-10-16T09:43:25.954036+0000","last_undegraded":"2023-10-16T09:43:25.954036+0000","last_fullsized":"2023-10-16T09:43:25.954036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948076+0000","last_change":"2023-10-16T09:43:21.933068+0000","last_active":"2023-10-16T09:43:25.948076+0000","last_peered":"2023-10-16T09:43:25.948076+0000","last_clean":"2023-10-16T09:43:25.948076+0000","last_became_active":"2023-10-16T09:43:21.932774+0000","last_became_peered":"2023-10-16T09:43:21.932774+0000","last_unstale":"2023-10-16T09:43:25.948076+0000","last_undegraded":"2023-10-16T09:43:25.948076+0000","last_fullsized":"2023-10-16T09:43:25.948076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950489+0000","last_change":"2023-10-16T09:43:21.937509+0000","last_active":"2023-10-16T09:43:25.950489+0000","last_peered":"2023-10-16T09:43:25.950489+0000","last_clean":"2023-10-16T09:43:25.950489+0000","last_became_active":"2023-10-16T09:43:21.936970+0000","last_became_peered":"2023-10-16T09:43:21.936970+0000","last_unstale":"2023-10-16T09:43:25.950489+0000","last_undegraded":"2023-10-16T09:43:25.950489+0000","last_fullsized":"2023-10-16T09:43:25.950489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948478+0000","last_change":"2023-10-16T09:43:21.927275+0000","last_active":"2023-10-16T09:43:25.948478+0000","last_peered":"2023-10-16T09:43:25.948478+0000","last_clean":"2023-10-16T09:43:25.948478+0000","last_became_active":"2023-10-16T09:43:21.927024+0000","last_became_peered":"2023-10-16T09:43:21.927024+0000","last_unstale":"2023-10-16T09:43:25.948478+0000","last_undegraded":"2023-10-16T09:43:25.948478+0000","last_fullsized":"2023-10-16T09:43:25.948478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966374+0000","last_change":"2023-10-16T09:43:21.925070+0000","last_active":"2023-10-16T09:43:25.966374+0000","last_peered":"2023-10-16T09:43:25.966374+0000","last_clean":"2023-10-16T09:43:25.966374+0000","last_became_active":"2023-10-16T09:43:21.924799+0000","last_became_peered":"2023-10-16T09:43:21.924799+0000","last_unstale":"2023-10-16T09:43:25.966374+0000","last_undegraded":"2023-10-16T09:43:25.966374+0000","last_fullsized":"2023-10-16T09:43:25.966374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955685+0000","last_change":"2023-10-16T09:43:21.920367+0000","last_active":"2023-10-16T09:43:25.955685+0000","last_peered":"2023-10-16T09:43:25.955685+0000","last_clean":"2023-10-16T09:43:25.955685+0000","last_became_active":"2023-10-16T09:43:21.919980+0000","last_became_peered":"2023-10-16T09:43:21.919980+0000","last_unstale":"2023-10-16T09:43:25.955685+0000","last_undegraded":"2023-10-16T09:43:25.955685+0000","last_fullsized":"2023-10-16T09:43:25.955685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956021+0000","last_change":"2023-10-16T09:43:21.931239+0000","last_active":"2023-10-16T09:43:25.956021+0000","last_peered":"2023-10-16T09:43:25.956021+0000","last_clean":"2023-10-16T09:43:25.956021+0000","last_became_active":"2023-10-16T09:43:21.930923+0000","last_became_peered":"2023-10-16T09:43:21.930923+0000","last_unstale":"2023-10-16T09:43:25.956021+0000","last_undegraded":"2023-10-16T09:43:25.956021+0000","last_fullsized":"2023-10-16T09:43:25.956021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967144+0000","last_change":"2023-10-16T09:43:21.924310+0000","last_active":"2023-10-16T09:43:25.967144+0000","last_peered":"2023-10-16T09:43:25.967144+0000","last_clean":"2023-10-16T09:43:25.967144+0000","last_became_active":"2023-10-16T09:43:21.921665+0000","last_became_peered":"2023-10-16T09:43:21.921665+0000","last_unstale":"2023-10-16T09:43:25.967144+0000","last_undegraded":"2023-10-16T09:43:25.967144+0000","last_fullsized":"2023-10-16T09:43:25.967144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950321+0000","last_change":"2023-10-16T09:43:21.927572+0000","last_active":"2023-10-16T09:43:25.950321+0000","last_peered":"2023-10-16T09:43:25.950321+0000","last_clean":"2023-10-16T09:43:25.950321+0000","last_became_active":"2023-10-16T09:43:21.927298+0000","last_became_peered":"2023-10-16T09:43:21.927298+0000","last_unstale":"2023-10-16T09:43:25.950321+0000","last_undegraded":"2023-10-16T09:43:25.950321+0000","last_fullsized":"2023-10-16T09:43:25.950321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965935+0000","last_change":"2023-10-16T09:43:21.922799+0000","last_active":"2023-10-16T09:43:25.965935+0000","last_peered":"2023-10-16T09:43:25.965935+0000","last_clean":"2023-10-16T09:43:25.965935+0000","last_became_active":"2023-10-16T09:43:21.922126+0000","last_became_peered":"2023-10-16T09:43:21.922126+0000","last_unstale":"2023-10-16T09:43:25.965935+0000","last_undegraded":"2023-10-16T09:43:25.965935+0000","last_fullsized":"2023-10-16T09:43:25.965935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950579+0000","last_change":"2023-10-16T09:43:21.936429+0000","last_active":"2023-10-16T09:43:25.950579+0000","last_peered":"2023-10-16T09:43:25.950579+0000","last_clean":"2023-10-16T09:43:25.950579+0000","last_became_active":"2023-10-16T09:43:21.933542+0000","last_became_peered":"2023-10-16T09:43:21.933542+0000","last_unstale":"2023-10-16T09:43:25.950579+0000","last_undegraded":"2023-10-16T09:43:25.950579+0000","last_fullsized":"2023-10-16T09:43:25.950579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948591+0000","last_change":"2023-10-16T09:43:21.930439+0000","last_active":"2023-10-16T09:43:25.948591+0000","last_peered":"2023-10-16T09:43:25.948591+0000","last_clean":"2023-10-16T09:43:25.948591+0000","last_became_active":"2023-10-16T09:43:21.930237+0000","last_became_peered":"2023-10-16T09:43:21.930237+0000","last_unstale":"2023-10-16T09:43:25.948591+0000","last_undegraded":"2023-10-16T09:43:25.948591+0000","last_fullsized":"2023-10-16T09:43:25.948591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:17.181381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953918+0000","last_change":"2023-10-16T09:43:21.934041+0000","last_active":"2023-10-16T09:43:25.953918+0000","last_peered":"2023-10-16T09:43:25.953918+0000","last_clean":"2023-10-16T09:43:25.953918+0000","last_became_active":"2023-10-16T09:43:21.933451+0000","last_became_peered":"2023-10-16T09:43:21.933451+0000","last_unstale":"2023-10-16T09:43:25.953918+0000","last_undegraded":"2023-10-16T09:43:25.953918+0000","last_fullsized":"2023-10-16T09:43:25.953918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":99940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":99780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:43:45.874 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T09:43:45.875 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-16T09:43:45.875 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-16T09:43:45.875 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T09:43:45.876 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-16T09:43:46.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.027+0000 7f200206d700 1 Processor -- start 2023-10-16T09:43:46.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.027+0000 7f200206d700 1 -- start start 2023-10-16T09:43:46.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f200206d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ffc06ab40 0x7f1ffc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f200206d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1ffc06e5d0 0x7f1ffc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f200206d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc071e50 con 0x7f1ffc06af20 2023-10-16T09:43:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f200206d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc161270 con 0x7f1ffc06ab40 2023-10-16T09:43:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f200206d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc161510 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ffc06ab40 0x7f1ffc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffbfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1ffc06af20 0x7f1ffc0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46586/0 (socket says 172.21.15.158:46586) 2023-10-16T09:43:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffbfff700 1 -- 172.21.15.158:0/808164803 learned_addr learned my addr 172.21.15.158:0/808164803 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffaffd700 1 --2- 172.21.15.158:0/808164803 >> v2:172.21.15.172:3300/0 conn(0x7f1ffc06e5d0 0x7f1ffc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffb7fe700 1 -- 172.21.15.158:0/808164803 >> v2:172.21.15.172:3300/0 conn(0x7f1ffc06e5d0 msgr2=0x7f1ffc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffb7fe700 1 --2- 172.21.15.158:0/808164803 >> v2:172.21.15.172:3300/0 conn(0x7f1ffc06e5d0 0x7f1ffc071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffb7fe700 1 -- 172.21.15.158:0/808164803 >> v1:172.21.15.158:6789/0 conn(0x7f1ffc06af20 legacy=0x7f1ffc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffb7fe700 1 -- 172.21.15.158:0/808164803 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc1617f0 con 0x7f1ffc06ab40 2023-10-16T09:43:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.028+0000 7f1ffa7fc700 1 -- 172.21.15.158:0/808164803 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481711260 0 0) 0x7f1ffc071e50 con 0x7f1ffc06af20 2023-10-16T09:43:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f1ffb7fe700 1 --2- 172.21.15.158:0/808164803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06ab40 0x7f1ffc06d820 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0c3bf0 tx=0x7f1ff0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df6f0b7a6902dfd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f1ffa7fc700 1 -- 172.21.15.158:0/808164803 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0011830 con 0x7f1ffc06ab40 2023-10-16T09:43:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f1ffa7fc700 1 -- 172.21.15.158:0/808164803 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff0011990 con 0x7f1ffc06ab40 2023-10-16T09:43:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f1ffa7fc700 1 -- 172.21.15.158:0/808164803 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0011d20 con 0x7f1ffc06ab40 2023-10-16T09:43:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 -- 172.21.15.158:0/808164803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06ab40 msgr2=0x7f1ffc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 --2- 172.21.15.158:0/808164803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06ab40 0x7f1ffc06d820 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0c3bf0 tx=0x7f1ff0009b20 comp rx=0 tx=0).stop 2023-10-16T09:43:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 -- 172.21.15.158:0/808164803 shutdown_connections 2023-10-16T09:43:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 --2- 172.21.15.158:0/808164803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06ab40 0x7f1ffc06d820 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 --2- 172.21.15.158:0/808164803 >> v2:172.21.15.172:3300/0 conn(0x7f1ffc06e5d0 0x7f1ffc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 -- 172.21.15.158:0/808164803 >> 172.21.15.158:0/808164803 conn(0x7f1ffc0d02b0 msgr2=0x7f1ffc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 -- 172.21.15.158:0/808164803 shutdown_connections 2023-10-16T09:43:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.029+0000 7f200206d700 1 -- 172.21.15.158:0/808164803 wait complete. 2023-10-16T09:43:46.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.031+0000 7f200206d700 1 Processor -- start 2023-10-16T09:43:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.031+0000 7f200206d700 1 -- start start 2023-10-16T09:43:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f200206d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ffc06ab40 0x7f1ffc15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f200206d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ffc06af20 0x7f1ffc1f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f200206d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06e5d0 0x7f1ffc1f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f200206d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc15de90 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f200206d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc1601f0 con 0x7f1ffc06ab40 2023-10-16T09:43:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f200206d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc1604d0 con 0x7f1ffc06af20 2023-10-16T09:43:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06e5d0 0x7f1ffc1f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06e5d0 0x7f1ffc1f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46416/0 (socket says 172.21.15.158:46416) 2023-10-16T09:43:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 -- 172.21.15.158:0/4156009494 learned_addr learned my addr 172.21.15.158:0/4156009494 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffb7fe700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ffc06ab40 0x7f1ffc15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffaffd700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ffc06af20 0x7f1ffc1f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 -- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ffc06af20 msgr2=0x7f1ffc1f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ffc06af20 0x7f1ffc1f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 -- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ffc06ab40 msgr2=0x7f1ffc15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ffc06ab40 0x7f1ffc15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffbfff700 1 -- 172.21.15.158:0/4156009494 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00094a0 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.032+0000 7f1ffaffd700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ffc06af20 0x7f1ffc1f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.033+0000 7f1ffbfff700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06e5d0 0x7f1ffc1f6a30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4002230 tx=0x7f1ff400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.033+0000 7f1fe3fff700 1 -- 172.21.15.158:0/4156009494 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff400ab70 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.033+0000 7f1fe3fff700 1 -- 172.21.15.158:0/4156009494 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff400acf0 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.033+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc1607b0 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.033+0000 7f1fe3fff700 1 -- 172.21.15.158:0/4156009494 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4007310 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.033+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc15fee0 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.034+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc0681b0 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.034+0000 7f1fe3fff700 1 -- 172.21.15.158:0/4156009494 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1ff4007470 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.034+0000 7f1fe3fff700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fe806af00 0x7f1fe806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.035+0000 7f1fe3fff700 1 -- 172.21.15.158:0/4156009494 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1ff4089670 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.035+0000 7f1ffb7fe700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fe806af00 0x7f1fe806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.035+0000 7f1ffb7fe700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fe806af00 0x7f1fe806d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00090b0 tx=0x7f1ff0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.039+0000 7f1fe3fff700 1 -- 172.21.15.158:0/4156009494 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ff40544c0 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.287+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1ffc157f40 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.290+0000 7f1fe3fff700 1 -- 172.21.15.158:0/4156009494 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171035 (secure 0 0 0) 0x7f1ff400ae60 con 0x7f1ffc06e5d0 2023-10-16T09:43:46.293 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:46.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fe806af00 msgr2=0x7f1fe806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fe806af00 0x7f1fe806d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00090b0 tx=0x7f1ff0008030 comp rx=0 tx=0).stop 2023-10-16T09:43:46.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06e5d0 msgr2=0x7f1ffc1f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06e5d0 0x7f1ffc1f6a30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4002230 tx=0x7f1ff400cf50 comp rx=0 tx=0).stop 2023-10-16T09:43:46.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 shutdown_connections 2023-10-16T09:43:46.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc06e5d0 0x7f1ffc1f6a30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ffc06ab40 0x7f1ffc15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fe806af00 0x7f1fe806d3c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 --2- 172.21.15.158:0/4156009494 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ffc06af20 0x7f1ffc1f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 >> 172.21.15.158:0/4156009494 conn(0x7f1ffc0d02b0 msgr2=0x7f1ffc072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:46.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 shutdown_connections 2023-10-16T09:43:46.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.292+0000 7f200206d700 1 -- 172.21.15.158:0/4156009494 wait complete. 2023-10-16T09:43:46.306 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":5,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4024924909},{"type":"v1","addr":"172.21.15.158:6833","nonce":4024924909}]},"active_addr":"172.21.15.158:6833/4024924909","active_change":"2023-10-16T09:43:09.210936+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":3457698273}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":251660717}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":686968952}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":2730663923}]}]} 2023-10-16T09:43:46.310 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-16T09:43:46.310 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-16T09:43:46.311 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:46.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.431+0000 7fb683bb6700 1 Processor -- start 2023-10-16T09:43:46.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.432+0000 7fb683bb6700 1 -- start start 2023-10-16T09:43:46.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.432+0000 7fb683bb6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 0x7fb67c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb683bb6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c161270 con 0x7fb67c0729b0 2023-10-16T09:43:46.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb683bb6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c161510 con 0x7fb67c06e5d0 2023-10-16T09:43:46.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb683bb6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1617f0 con 0x7fb67c06ab40 2023-10-16T09:43:46.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb682153700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb67c0729b0 0x7fb67c15df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46600/0 (socket says 172.21.15.158:46600) 2023-10-16T09:43:46.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb682153700 1 -- 172.21.15.158:0/2304984580 learned_addr learned my addr 172.21.15.158:0/2304984580 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:46.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb681952700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 0x7fb67c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148624602 0 0) 0x7fb67c161270 con 0x7fb67c0729b0 2023-10-16T09:43:46.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6680035b0 con 0x7fb67c0729b0 2023-10-16T09:43:46.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248681482 0 0) 0x7fb67c161510 con 0x7fb67c06e5d0 2023-10-16T09:43:46.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c161270 con 0x7fb67c06e5d0 2023-10-16T09:43:46.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003095393 0 0) 0x7fb6680035b0 con 0x7fb67c0729b0 2023-10-16T09:43:46.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 msgr2=0x7fb67c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:46.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 0x7fb67c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 >> v1:172.21.15.172:6789/0 conn(0x7fb67c06e5d0 legacy=0x7fb67c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c161ad0 con 0x7fb67c0729b0 2023-10-16T09:43:46.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fb6740019f0 con 0x7fb67c0729b0 2023-10-16T09:43:46.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 >> v1:172.21.15.158:6789/0 conn(0x7fb67c0729b0 legacy=0x7fb67c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb668003bc0 0x7fb668003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668004850 0x7fb668004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 0x7fb668005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1617f0 con 0x7fb668003bc0 2023-10-16T09:43:46.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66800bac0 con 0x7fb668004850 2023-10-16T09:43:46.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66800bc80 con 0x7fb67c06ab40 2023-10-16T09:43:46.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.433+0000 7fb681952700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb668003bc0 0x7fb668003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681151700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668004850 0x7fb668004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb682153700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 0x7fb668005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681952700 1 -- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 msgr2=0x7fb668005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:46.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681952700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 0x7fb668005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681952700 1 -- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668004850 msgr2=0x7fb668004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:46.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681952700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668004850 0x7fb668004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681952700 1 -- 172.21.15.158:0/2304984580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6680035b0 con 0x7fb668003bc0 2023-10-16T09:43:46.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681151700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668004850 0x7fb668004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:46.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb682153700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c06ab40 0x7fb668005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:46.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb681952700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb668003bc0 0x7fb668003fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009b60 tx=0x7fb668005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf582c300e1cbb0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00e6c0 con 0x7fb668003bc0 2023-10-16T09:43:46.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c00e820 con 0x7fb668003bc0 2023-10-16T09:43:46.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb680950700 1 -- 172.21.15.158:0/2304984580 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00eb90 con 0x7fb668003bc0 2023-10-16T09:43:46.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb683bb6700 1 -- 172.21.15.158:0/2304984580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb668003bc0 msgr2=0x7fb668003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb683bb6700 1 --2- 172.21.15.158:0/2304984580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb668003bc0 0x7fb668003fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009b60 tx=0x7fb668005880 comp rx=0 tx=0).stop 2023-10-16T09:43:46.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb682153700 1 -- 172.21.15.158:0/2304984580 reap_dead start 2023-10-16T09:43:46.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.434+0000 7fb683bb6700 1 -- 172.21.15.158:0/2304984580 shutdown_connections 2023-10-16T09:43:46.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.435+0000 7fb683bb6700 1 -- 172.21.15.158:0/2304984580 >> 172.21.15.158:0/2304984580 conn(0x7fb67c0d02b0 msgr2=0x7fb67c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:46.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.435+0000 7fb683bb6700 1 -- 172.21.15.158:0/2304984580 shutdown_connections 2023-10-16T09:43:46.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.435+0000 7fb683bb6700 1 -- 172.21.15.158:0/2304984580 wait complete. 2023-10-16T09:43:46.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.436+0000 7fb683bb6700 1 Processor -- start 2023-10-16T09:43:46.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.436+0000 7fb683bb6700 1 -- start start 2023-10-16T09:43:46.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668003bc0 0x7fb67c0c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb67c156590 0x7fb67c156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c0b30d0 0x7fb67c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c156f50 con 0x7fb67c156590 2023-10-16T09:43:46.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0c9ab0 con 0x7fb668003bc0 2023-10-16T09:43:46.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0b3d70 con 0x7fb67c0b30d0 2023-10-16T09:43:46.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb67c156590 0x7fb67c156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb67c156590 0x7fb67c156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46434/0 (socket says 172.21.15.158:46434) 2023-10-16T09:43:46.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 -- 172.21.15.158:0/2933349810 learned_addr learned my addr 172.21.15.158:0/2933349810 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:46.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681952700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668003bc0 0x7fb67c0c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb682153700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c0b30d0 0x7fb67c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 -- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c0b30d0 msgr2=0x7fb67c0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c0b30d0 0x7fb67c0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 -- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668003bc0 msgr2=0x7fb67c0c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668003bc0 0x7fb67c0c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 -- 172.21.15.158:0/2933349810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c0b4050 con 0x7fb67c156590 2023-10-16T09:43:46.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb681151700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb67c156590 0x7fb67c156a10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb678007a30 tx=0x7fb67800ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb6727fc700 1 -- 172.21.15.158:0/2933349810 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6780052e0 con 0x7fb67c156590 2023-10-16T09:43:46.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb6727fc700 1 -- 172.21.15.158:0/2933349810 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb678005440 con 0x7fb67c156590 2023-10-16T09:43:46.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c0c92b0 con 0x7fb67c156590 2023-10-16T09:43:46.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb6727fc700 1 -- 172.21.15.158:0/2933349810 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6780038a0 con 0x7fb67c156590 2023-10-16T09:43:46.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.437+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c0b3a90 con 0x7fb67c156590 2023-10-16T09:43:46.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.438+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c0c97d0 con 0x7fb67c156590 2023-10-16T09:43:46.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.439+0000 7fb6727fc700 1 -- 172.21.15.158:0/2933349810 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb678003a00 con 0x7fb67c156590 2023-10-16T09:43:46.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.439+0000 7fb6727fc700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb66006af00 0x7fb66006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.439+0000 7fb6727fc700 1 -- 172.21.15.158:0/2933349810 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb678088fd0 con 0x7fb67c156590 2023-10-16T09:43:46.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.439+0000 7fb681952700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb66006af00 0x7fb66006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.440+0000 7fb681952700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb66006af00 0x7fb66006d3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0090b0 tx=0x7fb66c01f040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.443+0000 7fb6727fc700 1 -- 172.21.15.158:0/2933349810 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb67c0c97d0 con 0x7fb67c156590 2023-10-16T09:43:46.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.638+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb67c0c97d0 con 0x7fb67c156590 2023-10-16T09:43:46.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.640+0000 7fb6727fc700 1 -- 172.21.15.158:0/2933349810 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7fb67c0c97d0 con 0x7fb67c156590 2023-10-16T09:43:46.642 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:46.642 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:46.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb66006af00 msgr2=0x7fb66006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb66006af00 0x7fb66006d3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0090b0 tx=0x7fb66c01f040 comp rx=0 tx=0).stop 2023-10-16T09:43:46.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb67c156590 msgr2=0x7fb67c156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb67c156590 0x7fb67c156a10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb678007a30 tx=0x7fb67800ea60 comp rx=0 tx=0).stop 2023-10-16T09:43:46.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 shutdown_connections 2023-10-16T09:43:46.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb66006af00 0x7fb66006d3c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb67c0b30d0 0x7fb67c0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb668003bc0 0x7fb67c0c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 --2- 172.21.15.158:0/2933349810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb67c156590 0x7fb67c156a10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 >> 172.21.15.158:0/2933349810 conn(0x7fb67c0d02b0 msgr2=0x7fb67c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:46.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 shutdown_connections 2023-10-16T09:43:46.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.642+0000 7fb683bb6700 1 -- 172.21.15.158:0/2933349810 wait complete. 2023-10-16T09:43:46.656 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-16T09:43:46.656 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.774+0000 7f22dfb92700 1 Processor -- start 2023-10-16T09:43:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.774+0000 7f22dfb92700 1 -- start start 2023-10-16T09:43:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.775+0000 7f22dfb92700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806ab40 0x7f22d806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.775+0000 7f22dfb92700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f22d806e5d0 0x7f22d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.775+0000 7f22dfb92700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d8071e50 con 0x7f22d806af20 2023-10-16T09:43:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.775+0000 7f22dfb92700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d8161270 con 0x7f22d806e5d0 2023-10-16T09:43:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.775+0000 7f22dfb92700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d8161510 con 0x7f22d806ab40 2023-10-16T09:43:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd12d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f22d806e5d0 0x7f22d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd92e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806ab40 0x7f22d806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22de12f700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f22d806af20 0x7f22d80722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:50524/0 (socket says 172.21.15.158:50524) 2023-10-16T09:43:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22de12f700 1 -- 172.21.15.158:0/3641761336 learned_addr learned my addr 172.21.15.158:0/3641761336 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd92e700 1 -- 172.21.15.158:0/3641761336 >> v2:172.21.15.172:3300/0 conn(0x7f22d806e5d0 msgr2=0x7f22d8071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd92e700 1 --2- 172.21.15.158:0/3641761336 >> v2:172.21.15.172:3300/0 conn(0x7f22d806e5d0 0x7f22d8071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd92e700 1 -- 172.21.15.158:0/3641761336 >> v1:172.21.15.172:6789/0 conn(0x7f22d806af20 legacy=0x7f22d80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd92e700 1 -- 172.21.15.158:0/3641761336 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d81617f0 con 0x7f22d806ab40 2023-10-16T09:43:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd12d700 1 --2- 172.21.15.158:0/3641761336 >> v2:172.21.15.172:3300/0 conn(0x7f22d806e5d0 0x7f22d8071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dd92e700 1 --2- 172.21.15.158:0/3641761336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806ab40 0x7f22d806d820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009840 tx=0x7f22d4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23442ea69595571a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dc92c700 1 -- 172.21.15.158:0/3641761336 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4011830 con 0x7f22d806ab40 2023-10-16T09:43:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dc92c700 1 -- 172.21.15.158:0/3641761336 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d4011990 con 0x7f22d806ab40 2023-10-16T09:43:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.776+0000 7f22dc92c700 1 -- 172.21.15.158:0/3641761336 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4011d20 con 0x7f22d806ab40 2023-10-16T09:43:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 -- 172.21.15.158:0/3641761336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806ab40 msgr2=0x7f22d806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 --2- 172.21.15.158:0/3641761336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806ab40 0x7f22d806d820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009840 tx=0x7f22d4009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 -- 172.21.15.158:0/3641761336 shutdown_connections 2023-10-16T09:43:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 --2- 172.21.15.158:0/3641761336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806ab40 0x7f22d806d820 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 --2- 172.21.15.158:0/3641761336 >> v2:172.21.15.172:3300/0 conn(0x7f22d806e5d0 0x7f22d8071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 -- 172.21.15.158:0/3641761336 >> 172.21.15.158:0/3641761336 conn(0x7f22d80d02b0 msgr2=0x7f22d80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 -- 172.21.15.158:0/3641761336 shutdown_connections 2023-10-16T09:43:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.777+0000 7f22dfb92700 1 -- 172.21.15.158:0/3641761336 wait complete. 2023-10-16T09:43:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.778+0000 7f22dfb92700 1 Processor -- start 2023-10-16T09:43:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.778+0000 7f22dfb92700 1 -- start start 2023-10-16T09:43:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dfb92700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d806ab40 0x7f22d80b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dfb92700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d806af20 0x7f22d80cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dfb92700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806e5d0 0x7f22d80cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dfb92700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d80b4bc0 con 0x7f22d806af20 2023-10-16T09:43:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dfb92700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d81460f0 con 0x7f22d806ab40 2023-10-16T09:43:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dfb92700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d80b5720 con 0x7f22d806e5d0 2023-10-16T09:43:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d806af20 0x7f22d80cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d806af20 0x7f22d80cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46448/0 (socket says 172.21.15.158:46448) 2023-10-16T09:43:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 -- 172.21.15.158:0/669187750 learned_addr learned my addr 172.21.15.158:0/669187750 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22de12f700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806e5d0 0x7f22d80cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd92e700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d806ab40 0x7f22d80b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 -- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806e5d0 msgr2=0x7f22d80cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806e5d0 0x7f22d80cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 -- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d806ab40 msgr2=0x7f22d80b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d806ab40 0x7f22d80b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 -- 172.21.15.158:0/669187750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc0094a0 con 0x7f22d806af20 2023-10-16T09:43:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dd12d700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d806af20 0x7f22d80cb750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00e9e0 tx=0x7f22cc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22ca7fc700 1 -- 172.21.15.158:0/669187750 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc00ee80 con 0x7f22d806af20 2023-10-16T09:43:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22ca7fc700 1 -- 172.21.15.158:0/669187750 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22cc00c850 con 0x7f22d806af20 2023-10-16T09:43:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d40094a0 con 0x7f22d806af20 2023-10-16T09:43:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.779+0000 7f22ca7fc700 1 -- 172.21.15.158:0/669187750 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc00cc00 con 0x7f22d806af20 2023-10-16T09:43:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.780+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d80b4ea0 con 0x7f22d806af20 2023-10-16T09:43:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.780+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d80681b0 con 0x7f22d806af20 2023-10-16T09:43:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.781+0000 7f22ca7fc700 1 -- 172.21.15.158:0/669187750 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f22cc00cd60 con 0x7f22d806af20 2023-10-16T09:43:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.781+0000 7f22ca7fc700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f22bc06af00 0x7f22bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.781+0000 7f22ca7fc700 1 -- 172.21.15.158:0/669187750 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f22cc089520 con 0x7f22d806af20 2023-10-16T09:43:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.781+0000 7f22dd92e700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f22bc06af00 0x7f22bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.782+0000 7f22dd92e700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f22bc06af00 0x7f22bc06d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f22d4000c00 tx=0x7f22d4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:46.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.786+0000 7f22ca7fc700 1 -- 172.21.15.158:0/669187750 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22cc0542e0 con 0x7f22d806af20 2023-10-16T09:43:46.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.978+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22d8161d80 con 0x7f22d806af20 2023-10-16T09:43:46.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.980+0000 7f22ca7fc700 1 -- 172.21.15.158:0/669187750 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f22cc057900 con 0x7f22d806af20 2023-10-16T09:43:46.981 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:46.982 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:46.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f22bc06af00 msgr2=0x7f22bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f22bc06af00 0x7f22bc06d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f22d4000c00 tx=0x7f22d4008030 comp rx=0 tx=0).stop 2023-10-16T09:43:46.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d806af20 msgr2=0x7f22d80cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:46.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d806af20 0x7f22d80cb750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00e9e0 tx=0x7f22cc00ecf0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 shutdown_connections 2023-10-16T09:43:46.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f22bc06af00 0x7f22bc06d3c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d806e5d0 0x7f22d80cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d806ab40 0x7f22d80b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 --2- 172.21.15.158:0/669187750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d806af20 0x7f22d80cb750 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:46.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 >> 172.21.15.158:0/669187750 conn(0x7f22d80d02b0 msgr2=0x7f22d80ac9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:46.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 shutdown_connections 2023-10-16T09:43:46.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:46.982+0000 7f22dfb92700 1 -- 172.21.15.158:0/669187750 wait complete. 2023-10-16T09:43:46.996 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-16T09:43:46.997 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-16T09:43:46.997 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-16T09:43:46.998 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-16T09:43:46.998 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-16T09:43:46.999 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-16T09:43:46.999 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-16T09:43:47.000 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-16T09:43:47.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.202+0000 7f69123ce700 1 Processor -- start 2023-10-16T09:43:47.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.204+0000 7f5e1f0fb700 1 Processor -- start 2023-10-16T09:43:47.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.204+0000 7f5e1f0fb700 1 -- start start 2023-10-16T09:43:47.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1f0fb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5e180ce9a0 0x7f5e1806e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1f0fb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5e1806e7c0 0x7f5e181657e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1f0fb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e180cce80 con 0x7f5e18165d20 2023-10-16T09:43:47.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1f0fb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e181667f0 con 0x7f5e180ce9a0 2023-10-16T09:43:47.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1f0fb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1806ec70 con 0x7f5e1806e7c0 2023-10-16T09:43:47.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5e180ce9a0 0x7f5e1806e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5e180ce9a0 0x7f5e1806e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46454/0 (socket says 172.21.15.158:46454) 2023-10-16T09:43:47.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 -- 172.21.15.158:0/4286448044 learned_addr learned my addr 172.21.15.158:0/4286448044 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e17fff700 1 --2- 172.21.15.158:0/4286448044 >> v2:172.21.15.172:3300/0 conn(0x7f5e1806e7c0 0x7f5e181657e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 -- 172.21.15.158:0/4286448044 >> v2:172.21.15.172:3300/0 conn(0x7f5e1806e7c0 msgr2=0x7f5e181657e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 --2- 172.21.15.158:0/4286448044 >> v2:172.21.15.172:3300/0 conn(0x7f5e1806e7c0 0x7f5e181657e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 -- 172.21.15.158:0/4286448044 >> v1:172.21.15.172:6789/0 conn(0x7f5e18165d20 legacy=0x7f5e180c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 -- 172.21.15.158:0/4286448044 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e18162120 con 0x7f5e180ce9a0 2023-10-16T09:43:47.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.206+0000 7f5e1ce97700 1 --2- 172.21.15.158:0/4286448044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e1806e280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5e000098f0 tx=0x7f5e00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9768c1f26147c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.207+0000 7f5e177fe700 1 -- 172.21.15.158:0/4286448044 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0000b860 con 0x7f5e180ce9a0 2023-10-16T09:43:47.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.207+0000 7f5e177fe700 1 -- 172.21.15.158:0/4286448044 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0000b9c0 con 0x7f5e180ce9a0 2023-10-16T09:43:47.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.207+0000 7f5e177fe700 1 -- 172.21.15.158:0/4286448044 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0000bd50 con 0x7f5e180ce9a0 2023-10-16T09:43:47.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/4286448044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 msgr2=0x7f5e1806e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f5e1f0fb700 1 --2- 172.21.15.158:0/4286448044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e1806e280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5e000098f0 tx=0x7f5e00009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:47.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/4286448044 shutdown_connections 2023-10-16T09:43:47.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f5e1f0fb700 1 --2- 172.21.15.158:0/4286448044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e1806e280 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f5e1f0fb700 1 --2- 172.21.15.158:0/4286448044 >> v2:172.21.15.172:3300/0 conn(0x7f5e1806e7c0 0x7f5e181657e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/4286448044 >> 172.21.15.158:0/4286448044 conn(0x7f5e180cf2b0 msgr2=0x7f5e180cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.207+0000 7f69123ce700 1 -- start start 2023-10-16T09:43:47.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f69123ce700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f690c06e990 0x7f690c15def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f69123ce700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f690c06c780 0x7f690c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f69123ce700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690c0cd6c0 con 0x7f690c15cde0 2023-10-16T09:43:47.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f69123ce700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f690c070b30 con 0x7f690c06e990 2023-10-16T09:43:47.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.208+0000 7f69123ce700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f690c06d750 con 0x7f690c06c780 2023-10-16T09:43:47.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f690c06e990 0x7f690c15def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f690c06e990 0x7f690c15def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46462/0 (socket says 172.21.15.158:46462) 2023-10-16T09:43:47.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 -- 172.21.15.158:0/884672691 learned_addr learned my addr 172.21.15.158:0/884672691 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 -- 172.21.15.158:0/884672691 >> v2:172.21.15.172:3300/0 conn(0x7f690c06c780 msgr2=0x7f690c071260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:47.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 --2- 172.21.15.158:0/884672691 >> v2:172.21.15.172:3300/0 conn(0x7f690c06c780 0x7f690c071260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 -- 172.21.15.158:0/884672691 >> v1:172.21.15.172:6789/0 conn(0x7f690c15cde0 legacy=0x7f690c06e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 -- 172.21.15.158:0/884672691 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c15d1c0 con 0x7f690c06e990 2023-10-16T09:43:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.209+0000 7f690b7fe700 1 --2- 172.21.15.158:0/884672691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c06e990 0x7f690c15def0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f690c070900 tx=0x7f6900009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20eb452fe8a636b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.210+0000 7f690affd700 1 -- 172.21.15.158:0/884672691 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900004230 con 0x7f690c06e990 2023-10-16T09:43:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.210+0000 7f690affd700 1 -- 172.21.15.158:0/884672691 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6900004390 con 0x7f690c06e990 2023-10-16T09:43:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.210+0000 7f690affd700 1 -- 172.21.15.158:0/884672691 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69000036a0 con 0x7f690c06e990 2023-10-16T09:43:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.210+0000 7f285d1d5700 1 Processor -- start 2023-10-16T09:43:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 -- 172.21.15.158:0/884672691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c06e990 msgr2=0x7f690c15def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 --2- 172.21.15.158:0/884672691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c06e990 0x7f690c15def0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f690c070900 tx=0x7f6900009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f285d1d5700 1 -- start start 2023-10-16T09:43:47.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 -- 172.21.15.158:0/884672691 shutdown_connections 2023-10-16T09:43:47.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 --2- 172.21.15.158:0/884672691 >> v2:172.21.15.172:3300/0 conn(0x7f690c06c780 0x7f690c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 --2- 172.21.15.158:0/884672691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c06e990 0x7f690c15def0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 -- 172.21.15.158:0/884672691 >> 172.21.15.158:0/884672691 conn(0x7f690c0cf2b0 msgr2=0x7f690c0cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/4286448044 shutdown_connections 2023-10-16T09:43:47.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 -- 172.21.15.158:0/884672691 shutdown_connections 2023-10-16T09:43:47.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f0c71fd5700 1 Processor -- start 2023-10-16T09:43:47.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285d1d5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28580c9b90 0x7f2858070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285d1d5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28580b0d30 0x7f28580b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285d1d5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2858162210 con 0x7f2858161e30 2023-10-16T09:43:47.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285d1d5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28580cdc30 con 0x7f28580c9b90 2023-10-16T09:43:47.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285d1d5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28580b16a0 con 0x7f28580b0d30 2023-10-16T09:43:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28580c9b90 0x7f2858070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28580c9b90 0x7f2858070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46468/0 (socket says 172.21.15.158:46468) 2023-10-16T09:43:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f285659c700 1 -- 172.21.15.158:0/699263843 learned_addr learned my addr 172.21.15.158:0/699263843 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f285659c700 1 -- 172.21.15.158:0/699263843 >> v2:172.21.15.172:3300/0 conn(0x7f28580b0d30 msgr2=0x7f28580b1160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T09:43:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f285659c700 1 --2- 172.21.15.158:0/699263843 >> v2:172.21.15.172:3300/0 conn(0x7f28580b0d30 0x7f28580b1160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f285659c700 1 -- 172.21.15.158:0/699263843 >> v1:172.21.15.172:6789/0 conn(0x7f2858161e30 legacy=0x7f28580ca090 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T09:43:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f285659c700 1 -- 172.21.15.158:0/699263843 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28580cd020 con 0x7f28580c9b90 2023-10-16T09:43:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.211+0000 7f69123ce700 1 -- 172.21.15.158:0/884672691 wait complete. 2023-10-16T09:43:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f69123ce700 1 Processor -- start 2023-10-16T09:43:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f0c71fd5700 1 -- start start 2023-10-16T09:43:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f0c71fd5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0c6c0b93f0 0x7f0c6c0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f0c71fd5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0cfff0 con 0x7f0c6c0d2040 2023-10-16T09:43:47.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f0c71fd5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0d2450 con 0x7f0c6c0b9010 2023-10-16T09:43:47.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f0c71fd5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6c0cf410 con 0x7f0c6c0b93f0 2023-10-16T09:43:47.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f0c6affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0c6c0b93f0 0x7f0c6c0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f0c6b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0c6c0b9010 0x7f0c6c0d4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:50550/0 (socket says 172.21.15.158:50550) 2023-10-16T09:43:47.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f0c6b7fe700 1 -- 172.21.15.158:0/1396200127 learned_addr learned my addr 172.21.15.158:0/1396200127 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:47.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f0c6affd700 1 -- 172.21.15.158:0/1396200127 >> v1:172.21.15.172:6789/0 conn(0x7f0c6c0b9010 legacy=0x7f0c6c0d4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f0c6affd700 1 -- 172.21.15.158:0/1396200127 >> v1:172.21.15.158:6789/0 conn(0x7f0c6c0d2040 legacy=0x7f0c6c0b6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f0c6affd700 1 -- 172.21.15.158:0/1396200127 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c0cfd10 con 0x7f0c6c0b93f0 2023-10-16T09:43:47.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f285659c700 1 --2- 172.21.15.158:0/699263843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580c9b90 0x7f2858070870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f285806ed10 tx=0x7f2848009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11da956583a273ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f0c6affd700 1 --2- 172.21.15.158:0/1396200127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c0b93f0 0x7f0c6c0d1b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0097c0 tx=0x7f0c5c009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a6cab299f1a2b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.213+0000 7f69123ce700 1 -- start start 2023-10-16T09:43:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f69123ce700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690c06c780 0x7f690c0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f69123ce700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f690c06e990 0x7f690c167530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f69123ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c15cde0 0x7f690c167a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f69123ce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c200b10 con 0x7f690c15cde0 2023-10-16T09:43:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f69123ce700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c200dd0 con 0x7f690c06c780 2023-10-16T09:43:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f69123ce700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c2010b0 con 0x7f690c06e990 2023-10-16T09:43:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f691096b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c15cde0 0x7f690c167a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f690b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f690c06e990 0x7f690c167530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f691096b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c15cde0 0x7f690c167a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46480/0 (socket says 172.21.15.158:46480) 2023-10-16T09:43:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f691096b700 1 -- 172.21.15.158:0/1443292699 learned_addr learned my addr 172.21.15.158:0/1443292699 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f690bfff700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690c06c780 0x7f690c0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f690b7fe700 1 -- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690c06c780 msgr2=0x7f690c0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f690b7fe700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690c06c780 0x7f690c0c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f690b7fe700 1 -- 172.21.15.158:0/1443292699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c15cde0 msgr2=0x7f690c167a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f690b7fe700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f690c15cde0 0x7f690c167a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f690b7fe700 1 -- 172.21.15.158:0/1443292699 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6900009550 con 0x7f690c06e990 2023-10-16T09:43:47.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f690b7fe700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f690c06e990 0x7f690c167530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6900015000 tx=0x7f6900004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900003e60 con 0x7f690c06e990 2023-10-16T09:43:47.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.212+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/4286448044 wait complete. 2023-10-16T09:43:47.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f5e1f0fb700 1 Processor -- start 2023-10-16T09:43:47.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.214+0000 7f5e1f0fb700 1 -- start start 2023-10-16T09:43:47.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1f0fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e18145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1f0fb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e18165d20 0x7f5e180c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1f0fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e180c68c0 0x7f5e180c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1f0fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e180c4610 con 0x7f5e180ce9a0 2023-10-16T09:43:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1f0fb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e181466c0 con 0x7f5e18165d20 2023-10-16T09:43:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1f0fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e180c3fc0 con 0x7f5e180c68c0 2023-10-16T09:43:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1ce97700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e18145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e17fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e18165d20 0x7f5e180c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1ce97700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e18145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46484/0 (socket says 172.21.15.158:46484) 2023-10-16T09:43:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.215+0000 7f5e1ce97700 1 -- 172.21.15.158:0/2821895274 learned_addr learned my addr 172.21.15.158:0/2821895274 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1d698700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e180c68c0 0x7f5e180c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1ce97700 1 -- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e180c68c0 msgr2=0x7f5e180c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1ce97700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e180c68c0 0x7f5e180c4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1ce97700 1 -- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e18165d20 msgr2=0x7f5e180c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1ce97700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e18165d20 0x7f5e180c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1ce97700 1 -- 172.21.15.158:0/2821895274 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00009550 con 0x7f5e180ce9a0 2023-10-16T09:43:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e17fff700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e18165d20 0x7f5e180c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1ce97700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e18145e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e00000c00 tx=0x7f5e00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c201390 con 0x7f690c06e990 2023-10-16T09:43:47.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f2855d9b700 1 -- 172.21.15.158:0/699263843 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848004230 con 0x7f28580c9b90 2023-10-16T09:43:47.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f2855d9b700 1 -- 172.21.15.158:0/699263843 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2848004390 con 0x7f28580c9b90 2023-10-16T09:43:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f2855d9b700 1 -- 172.21.15.158:0/699263843 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28480036a0 con 0x7f28580c9b90 2023-10-16T09:43:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c201950 con 0x7f690c06e990 2023-10-16T09:43:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.217+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f690c06d190 con 0x7f690c06e990 2023-10-16T09:43:47.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.217+0000 7f1267e08700 1 Processor -- start 2023-10-16T09:43:47.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.217+0000 7f285d1d5700 1 -- 172.21.15.158:0/699263843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580c9b90 msgr2=0x7f2858070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.217+0000 7f285d1d5700 1 --2- 172.21.15.158:0/699263843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580c9b90 0x7f2858070870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f285806ed10 tx=0x7f2848009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.218+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6900011630 con 0x7f690c06e990 2023-10-16T09:43:47.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.218+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69000119a0 con 0x7f690c06e990 2023-10-16T09:43:47.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.218+0000 7f1267e08700 1 -- start start 2023-10-16T09:43:47.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1267e08700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12600cd830 0x7f12600cb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1267e08700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 0x7f126006e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1267e08700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126006ec10 con 0x7f126006e7c0 2023-10-16T09:43:47.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1267e08700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126015ee10 con 0x7f12600cd830 2023-10-16T09:43:47.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1267e08700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1260158320 con 0x7f12600c9900 2023-10-16T09:43:47.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f0c6a7fc700 1 -- 172.21.15.158:0/1396200127 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c004230 con 0x7f0c6c0b93f0 2023-10-16T09:43:47.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f0c6a7fc700 1 -- 172.21.15.158:0/1396200127 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c004390 con 0x7f0c6c0b93f0 2023-10-16T09:43:47.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f0c6a7fc700 1 -- 172.21.15.158:0/1396200127 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c0036a0 con 0x7f0c6c0b93f0 2023-10-16T09:43:47.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1396200127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c0b93f0 msgr2=0x7f0c6c0d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f0c71fd5700 1 --2- 172.21.15.158:0/1396200127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c0b93f0 0x7f0c6c0d1b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0097c0 tx=0x7f0c5c009ad0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6900022070 con 0x7f690c06e990 2023-10-16T09:43:47.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f6908ff9700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f68f806af00 0x7f68f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f690008b750 con 0x7f690c06e990 2023-10-16T09:43:47.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f690bfff700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f68f806af00 0x7f68f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f6908ff9700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7f68f8074440 0x7f68f8076880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 --> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68f8076f50 con 0x7f68f8074440 2023-10-16T09:43:47.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f690008bba0 con 0x7f690c06e990 2023-10-16T09:43:47.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f690bfff700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f68f806af00 0x7f68f806d3c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f690c15b4a0 tx=0x7f68fc007ec0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1265ba4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12600cd830 0x7f12600cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1265ba4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12600cd830 0x7f12600cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46496/0 (socket says 172.21.15.158:46496) 2023-10-16T09:43:47.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f1265ba4700 1 -- 172.21.15.158:0/1648248882 learned_addr learned my addr 172.21.15.158:0/1648248882 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f12653a3700 1 --2- 172.21.15.158:0/1648248882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 0x7f126006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1265ba4700 1 -- 172.21.15.158:0/1648248882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 msgr2=0x7f126006e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1265ba4700 1 --2- 172.21.15.158:0/1648248882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 0x7f126006e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1265ba4700 1 -- 172.21.15.158:0/1648248882 >> v1:172.21.15.172:6789/0 conn(0x7f126006e7c0 legacy=0x7f1260161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1265ba4700 1 -- 172.21.15.158:0/1648248882 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126006f220 con 0x7f12600cd830 2023-10-16T09:43:47.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1265ba4700 1 --2- 172.21.15.158:0/1648248882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12600cd830 0x7f12600cb510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f125c0098f0 tx=0x7f125c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3c71f62bcb7c04 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1264ba2700 1 -- 172.21.15.158:0/1648248882 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c004230 con 0x7f12600cd830 2023-10-16T09:43:47.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1264ba2700 1 -- 172.21.15.158:0/1648248882 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c004390 con 0x7f12600cd830 2023-10-16T09:43:47.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f1264ba2700 1 -- 172.21.15.158:0/1648248882 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c0036a0 con 0x7f12600cd830 2023-10-16T09:43:47.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1396200127 shutdown_connections 2023-10-16T09:43:47.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f0c71fd5700 1 --2- 172.21.15.158:0/1396200127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c0b93f0 0x7f0c6c0d1b00 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0097c0 tx=0x7f0c5c009ad0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1396200127 >> 172.21.15.158:0/1396200127 conn(0x7f0c6c09ca30 msgr2=0x7f0c6c0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f691096b700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7f68f8074440 0x7f68f8076880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f285d1d5700 1 -- 172.21.15.158:0/699263843 shutdown_connections 2023-10-16T09:43:47.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f285d1d5700 1 --2- 172.21.15.158:0/699263843 >> v2:172.21.15.172:3300/0 conn(0x7f28580b0d30 0x7f28580b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f285d1d5700 1 --2- 172.21.15.158:0/699263843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580c9b90 0x7f2858070870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f285d1d5700 1 -- 172.21.15.158:0/699263843 >> 172.21.15.158:0/699263843 conn(0x7f28580cbd30 msgr2=0x7f28580cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.222+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1396200127 shutdown_connections 2023-10-16T09:43:47.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7fdad3279700 1 Processor -- start 2023-10-16T09:43:47.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7fdad3279700 1 -- start start 2023-10-16T09:43:47.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f691096b700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7f68f8074440 0x7f68f8076880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f1267e08700 1 -- 172.21.15.158:0/1648248882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12600cd830 msgr2=0x7f12600cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f1267e08700 1 --2- 172.21.15.158:0/1648248882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12600cd830 0x7f12600cb510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f125c0098f0 tx=0x7f125c009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:47.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f1267e08700 1 -- 172.21.15.158:0/1648248882 shutdown_connections 2023-10-16T09:43:47.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f1267e08700 1 --2- 172.21.15.158:0/1648248882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12600cd830 0x7f12600cb510 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f1267e08700 1 --2- 172.21.15.158:0/1648248882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 0x7f126006e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f1267e08700 1 -- 172.21.15.158:0/1648248882 >> 172.21.15.158:0/1648248882 conn(0x7f12600cf2b0 msgr2=0x7f12600cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.222+0000 7f1267e08700 1 -- 172.21.15.158:0/1648248882 shutdown_connections 2023-10-16T09:43:47.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00004300 con 0x7f5e180ce9a0 2023-10-16T09:43:47.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0001f070 con 0x7f5e180ce9a0 2023-10-16T09:43:47.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e000036a0 con 0x7f5e180ce9a0 2023-10-16T09:43:47.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e180c42a0 con 0x7f5e180ce9a0 2023-10-16T09:43:47.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.216+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e180c5b50 con 0x7f5e180ce9a0 2023-10-16T09:43:47.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.218+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e18166510 con 0x7f5e180ce9a0 2023-10-16T09:43:47.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.218+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5e00044d50 con 0x7f5e180ce9a0 2023-10-16T09:43:47.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f5e157fa700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5df806af00 0x7f5df806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5e0008aff0 con 0x7f5e180ce9a0 2023-10-16T09:43:47.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f5e17fff700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5df806af00 0x7f5df806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f5e157fa700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7f5df80744c0 0x7f5df8076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 --> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5df8076fd0 con 0x7f5df80744c0 2023-10-16T09:43:47.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e0008b420 con 0x7f5e180ce9a0 2023-10-16T09:43:47.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.219+0000 7f5e1d698700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7f5df80744c0 0x7f5df8076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f5e17fff700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5df806af00 0x7f5df806d3c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e180b4df0 tx=0x7f5e08009850 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.220+0000 7f5e1d698700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7f5df80744c0 0x7f5df8076900 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.221+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== osd.6 v2:172.21.15.172:6800/3389147915 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f5df8076fd0 con 0x7f5df80744c0 2023-10-16T09:43:47.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.222+0000 7f285d1d5700 1 -- 172.21.15.158:0/699263843 shutdown_connections 2023-10-16T09:43:47.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7f285d1d5700 1 -- 172.21.15.158:0/699263843 wait complete. 2023-10-16T09:43:47.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f285d1d5700 1 Processor -- start 2023-10-16T09:43:47.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f285d1d5700 1 -- start start 2023-10-16T09:43:47.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.222+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1396200127 wait complete. 2023-10-16T09:43:47.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7f0c71fd5700 1 Processor -- start 2023-10-16T09:43:47.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad3279700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdacc15d920 0x7fdacc0cf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad3279700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad3279700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0c73a0 con 0x7fdacc1615b0 2023-10-16T09:43:47.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad3279700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdacc0cf5a0 con 0x7fdacc15d920 2023-10-16T09:43:47.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad3279700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0ced00 con 0x7fdacc162120 2023-10-16T09:43:47.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad1816700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad1816700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc0c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57340/0 (socket says 172.21.15.158:57340) 2023-10-16T09:43:47.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad1816700 1 -- 172.21.15.158:0/2670930814 learned_addr learned my addr 172.21.15.158:0/2670930814 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad1816700 1 -- 172.21.15.158:0/2670930814 >> v2:172.21.15.172:3300/0 conn(0x7fdacc15d920 msgr2=0x7fdacc0cf930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T09:43:47.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad1816700 1 --2- 172.21.15.158:0/2670930814 >> v2:172.21.15.172:3300/0 conn(0x7fdacc15d920 0x7fdacc0cf930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad1816700 1 -- 172.21.15.158:0/2670930814 >> v1:172.21.15.172:6789/0 conn(0x7fdacc1615b0 legacy=0x7fdacc161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7fdad1816700 1 -- 172.21.15.158:0/2670930814 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc0cefe0 con 0x7fdacc162120 2023-10-16T09:43:47.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.224+0000 7fdad1816700 1 --2- 172.21.15.158:0/2670930814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc0c7880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e50 tx=0x7fdac400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d846c210b9e2f2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.224+0000 7fdac3fff700 1 -- 172.21.15.158:0/2670930814 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac400e070 con 0x7fdacc162120 2023-10-16T09:43:47.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.224+0000 7fdac3fff700 1 -- 172.21.15.158:0/2670930814 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac4003a20 con 0x7fdacc162120 2023-10-16T09:43:47.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.224+0000 7fdac3fff700 1 -- 172.21.15.158:0/2670930814 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4003dd0 con 0x7fdacc162120 2023-10-16T09:43:47.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fdad3279700 1 -- 172.21.15.158:0/2670930814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 msgr2=0x7fdacc0c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fdad3279700 1 --2- 172.21.15.158:0/2670930814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc0c7880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e50 tx=0x7fdac400b330 comp rx=0 tx=0).stop 2023-10-16T09:43:47.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fdad3279700 1 -- 172.21.15.158:0/2670930814 shutdown_connections 2023-10-16T09:43:47.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fdad3279700 1 --2- 172.21.15.158:0/2670930814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc0c7880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fdad3279700 1 --2- 172.21.15.158:0/2670930814 >> v2:172.21.15.172:3300/0 conn(0x7fdacc15d920 0x7fdacc0cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fdad3279700 1 -- 172.21.15.158:0/2670930814 >> 172.21.15.158:0/2670930814 conn(0x7fdacc09ca30 msgr2=0x7fdacc0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.223+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== osd.4 v2:172.21.15.172:6824/3174967294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f68f8076f50 con 0x7f68f8074440 2023-10-16T09:43:47.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.222+0000 7f1267e08700 1 -- 172.21.15.158:0/1648248882 wait complete. 2023-10-16T09:43:47.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.224+0000 7f1267e08700 1 Processor -- start 2023-10-16T09:43:47.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.224+0000 7f1267e08700 1 -- start start 2023-10-16T09:43:47.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.224+0000 7f1267e08700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006e7c0 0x7f12600ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1267e08700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 0x7f12600a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1267e08700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600cd830 0x7f12601e88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1267e08700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600aa540 con 0x7f126006e7c0 2023-10-16T09:43:47.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1267e08700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600a9930 con 0x7f12600cd830 2023-10-16T09:43:47.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1267e08700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600a9c10 con 0x7f12600c9900 2023-10-16T09:43:47.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006e7c0 0x7f12600ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006e7c0 0x7f12600ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46508/0 (socket says 172.21.15.158:46508) 2023-10-16T09:43:47.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 -- 172.21.15.158:0/1638930955 learned_addr learned my addr 172.21.15.158:0/1638930955 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f12663a5700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600cd830 0x7f12601e88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f12653a3700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 0x7f12600a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 -- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 msgr2=0x7f12600a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12600c9900 0x7f12600a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 -- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600cd830 msgr2=0x7f12601e88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600cd830 0x7f12601e88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7f1265ba4700 1 -- 172.21.15.158:0/1638930955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c009550 con 0x7f126006e7c0 2023-10-16T09:43:47.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f1265ba4700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006e7c0 0x7f12600ab330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f125c009c40 tx=0x7f125c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c0045f0 con 0x7f126006e7c0 2023-10-16T09:43:47.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12600a9ef0 con 0x7f126006e7c0 2023-10-16T09:43:47.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12600ca610 con 0x7f126006e7c0 2023-10-16T09:43:47.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fd7f22f6700 1 Processor -- start 2023-10-16T09:43:47.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.225+0000 7fd7f22f6700 1 -- start start 2023-10-16T09:43:47.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7fd7f22f6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd7ec0c4ae0 0x7fd7ec166010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7fd7f22f6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7ec0ceae0 0x7fd7ec070340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7fd7f22f6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ec071200 con 0x7fd7ec0cf0a0 2023-10-16T09:43:47.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7fd7f22f6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c4ec0 con 0x7fd7ec0c4ae0 2023-10-16T09:43:47.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7fd7f22f6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec165b80 con 0x7fd7ec0ceae0 2023-10-16T09:43:47.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7fd7f0893700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd7ec0cf0a0 0x7fd7ec0cbb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46694/0 (socket says 172.21.15.158:46694) 2023-10-16T09:43:47.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7fd7f0893700 1 -- 172.21.15.158:0/850925708 learned_addr learned my addr 172.21.15.158:0/850925708 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:47.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f285d1d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580b0d30 0x7f28580b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f285d1d5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28580c9b90 0x7f28581f9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f285d1d5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2858161e30 0x7f28580c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f285d1d5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580c83f0 con 0x7f28580b0d30 2023-10-16T09:43:47.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f285d1d5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580b2410 con 0x7f2858161e30 2023-10-16T09:43:47.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f285d1d5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28581fa310 con 0x7f28580c9b90 2023-10-16T09:43:47.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f126015e570 con 0x7f126006e7c0 2023-10-16T09:43:47.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.226+0000 7f0c71fd5700 1 -- start start 2023-10-16T09:43:47.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877664325 0 0) 0x7fd7ec071200 con 0x7fd7ec0cf0a0 2023-10-16T09:43:47.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c71fd5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c6c0b9010 0x7f0c6c06f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c71fd5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c6c0d2040 0x7f0c6c1ff770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c71fd5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c09e4d0 0x7f0c6c09f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c71fd5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0c8800 con 0x7f0c6c09e4d0 2023-10-16T09:43:47.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c71fd5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c06f910 con 0x7f0c6c0d2040 2023-10-16T09:43:47.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d40035b0 con 0x7fd7ec0cf0a0 2023-10-16T09:43:47.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c71fd5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0a00e0 con 0x7f0c6c0b9010 2023-10-16T09:43:47.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c6affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c6c0d2040 0x7f0c6c1ff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c6affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c6c0d2040 0x7f0c6c1ff770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47706/0 (socket says 172.21.15.158:47706) 2023-10-16T09:43:47.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f0c6affd700 1 -- 172.21.15.158:0/1023279136 learned_addr learned my addr 172.21.15.158:0/1023279136 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f2856d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580b0d30 0x7f28580b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580b0d30 0x7f28580b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46528/0 (socket says 172.21.15.158:46528) 2023-10-16T09:43:47.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 -- 172.21.15.158:0/3508444563 learned_addr learned my addr 172.21.15.158:0/3508444563 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fdad3279700 1 -- 172.21.15.158:0/2670930814 shutdown_connections 2023-10-16T09:43:47.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f0c6bfff700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c09e4d0 0x7f0c6c09f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 -- 172.21.15.158:0/3508444563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28580c9b90 msgr2=0x7f28581f9dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:47.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28580c9b90 0x7f28581f9dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 -- 172.21.15.158:0/3508444563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2858161e30 msgr2=0x7f28580c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2858161e30 0x7f28580c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 -- 172.21.15.158:0/3508444563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2848009550 con 0x7f28580b0d30 2023-10-16T09:43:47.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f0c6affd700 1 -- 172.21.15.158:0/1023279136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c6c0b9010 msgr2=0x7f0c6c06f3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:47.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f0c6affd700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c6c0b9010 0x7f0c6c06f3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f0c6affd700 1 -- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c09e4d0 msgr2=0x7f0c6c09f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f0c6affd700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c6c09e4d0 0x7f0c6c09f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f0c6affd700 1 -- 172.21.15.158:0/1023279136 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c0094a0 con 0x7f0c6c0d2040 2023-10-16T09:43:47.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f2856d9d700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580b0d30 0x7f28580b1ed0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f284c00b580 tx=0x7f284c00b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7f0c6affd700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c6c0d2040 0x7f0c6c1ff770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c08dc20 tx=0x7f0c5c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c00b660 con 0x7f126006e7c0 2023-10-16T09:43:47.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.227+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c00ba10 con 0x7f126006e7c0 2023-10-16T09:43:47.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7f283f7fe700 1 -- 172.21.15.158:0/3508444563 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c011830 con 0x7f28580b0d30 2023-10-16T09:43:47.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7f283f7fe700 1 -- 172.21.15.158:0/3508444563 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c0119b0 con 0x7f28580b0d30 2023-10-16T09:43:47.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28580c6dd0 con 0x7f28580b0d30 2023-10-16T09:43:47.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928719295 0 0) 0x7fd7d40035b0 con 0x7fd7ec0cf0a0 2023-10-16T09:43:47.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7ec0ceae0 msgr2=0x7fd7ec070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fd7eaffd700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7ec0ceae0 0x7fd7ec070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 >> v2:172.21.15.172:3300/0 conn(0x7fd7ec0c4ae0 msgr2=0x7fd7ec166010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T09:43:47.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fd7eaffd700 1 --2- 172.21.15.158:0/850925708 >> v2:172.21.15.172:3300/0 conn(0x7fd7ec0c4ae0 0x7fd7ec166010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec070c40 con 0x7fd7ec0cf0a0 2023-10-16T09:43:47.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd7e40019f0 con 0x7fd7ec0cf0a0 2023-10-16T09:43:47.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 >> v1:172.21.15.158:6789/0 conn(0x7fd7ec0cf0a0 legacy=0x7fd7ec0cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7d4003a00 0x7fd7d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7d4004850 0x7fd7d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7d4005330 0x7fd7d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c4ec0 con 0x7fd7d4003a00 2023-10-16T09:43:47.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec165b80 con 0x7fd7d4005330 2023-10-16T09:43:47.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d400c260 con 0x7fd7d4004850 2023-10-16T09:43:47.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28580c7390 con 0x7f28580b0d30 2023-10-16T09:43:47.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.228+0000 7fdad3279700 1 -- 172.21.15.158:0/2670930814 wait complete. 2023-10-16T09:43:47.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fd7f0893700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7d4005330 0x7fd7d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7f283f7fe700 1 -- 172.21.15.158:0/3508444563 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c011d60 con 0x7f28580b0d30 2023-10-16T09:43:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.229+0000 7fdad3279700 1 Processor -- start 2023-10-16T09:43:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f125c021070 con 0x7f126006e7c0 2023-10-16T09:43:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7f12567fc700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f124406af30 0x7f124406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f125c08add0 con 0x7f126006e7c0 2023-10-16T09:43:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7f12567fc700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f12440744f0 0x7f1244076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 --> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1244077000 con 0x7f12440744f0 2023-10-16T09:43:47.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7fd7f0893700 1 -- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7d4004850 msgr2=0x7fd7d4004cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:47.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7fd7f0893700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7d4004850 0x7fd7d4004cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fd7f0893700 1 -- 172.21.15.158:0/850925708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7d4003a00 msgr2=0x7fd7d4003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:47.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fd7f0893700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7d4003a00 0x7fd7d4003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fd7f0893700 1 -- 172.21.15.158:0/850925708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d40035b0 con 0x7fd7d4005330 2023-10-16T09:43:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7f12663a5700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f12440744f0 0x7f1244076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f125c055ca0 con 0x7f126006e7c0 2023-10-16T09:43:47.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7f283f7fe700 1 -- 172.21.15.158:0/3508444563 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f284c01a400 con 0x7f28580b0d30 2023-10-16T09:43:47.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.232+0000 7f283f7fe700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f284406af50 0x7f284406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.232+0000 7f283f7fe700 1 -- 172.21.15.158:0/3508444563 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f284c089820 con 0x7f28580b0d30 2023-10-16T09:43:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.232+0000 7f285659c700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f284406af50 0x7f284406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.232+0000 7f285d1d5700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f28380015e0 0x7f2838003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.232+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 --> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2838006bd0 con 0x7f28380015e0 2023-10-16T09:43:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.233+0000 7f285659c700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f284406af50 0x7f284406d410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28580b13b0 tx=0x7f2848008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.233+0000 7f285759e700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f28380015e0 0x7f2838003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.233+0000 7f285759e700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f28380015e0 0x7f2838003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c011630 con 0x7f0c6c0d2040 2023-10-16T09:43:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c011790 con 0x7f0c6c0d2040 2023-10-16T09:43:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.230+0000 7fdad3279700 1 -- start start 2023-10-16T09:43:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad3279700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdacc15d920 0x7fdacc0b8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad3279700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdacc1615b0 0x7fdacc160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad3279700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc06c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad3279700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0b89f0 con 0x7fdacc15d920 2023-10-16T09:43:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad3279700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0b7de0 con 0x7fdacc1615b0 2023-10-16T09:43:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad3279700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0b80c0 con 0x7fdacc162120 2023-10-16T09:43:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdacc15d920 0x7fdacc0b8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdacc15d920 0x7fdacc0b8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46538/0 (socket says 172.21.15.158:46538) 2023-10-16T09:43:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 -- 172.21.15.158:0/2064706363 learned_addr learned my addr 172.21.15.158:0/2064706363 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1816700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc06c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 -- 172.21.15.158:0/2064706363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 msgr2=0x7fdacc06c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdacc162120 0x7fdacc06c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 -- 172.21.15.158:0/2064706363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdacc1615b0 msgr2=0x7fdacc160b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdacc1615b0 0x7fdacc160b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.231+0000 7fdad1015700 1 -- 172.21.15.158:0/2064706363 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4009b30 con 0x7fdacc15d920 2023-10-16T09:43:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.232+0000 7fdad1015700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdacc15d920 0x7fdacc0b8e80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009fa0 tx=0x7fdac800b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7f12653a3700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f124406af30 0x7f124406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7fd7f0893700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7d4005330 0x7fd7d400ba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400a7d0 tx=0x7fd7d4005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d35e6622e790d86b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c0c8a10 con 0x7f0c6c0d2040 2023-10-16T09:43:47.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c0c8fd0 con 0x7f0c6c0d2040 2023-10-16T09:43:47.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7f12653a3700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f124406af30 0x7f124406d3f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f12600cac40 tx=0x7f1250008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.234+0000 7f12663a5700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f12440744f0 0x7f1244076930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.235+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c6c0cf030 con 0x7f0c6c0d2040 2023-10-16T09:43:47.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.235+0000 7f283f7fe700 1 -- 172.21.15.158:0/3508444563 <== osd.0 v2:172.21.15.158:6809/38337391 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f2838006bd0 con 0x7f28380015e0 2023-10-16T09:43:47.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.235+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== osd.5 v2:172.21.15.172:6816/2983726905 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f1244077000 con 0x7f12440744f0 2023-10-16T09:43:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.237+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c011b40 con 0x7f0c6c0d2040 2023-10-16T09:43:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.237+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0c5c022020 con 0x7f0c6c0d2040 2023-10-16T09:43:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.237+0000 7f0c53fff700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0c5806b050 0x7f0c5806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.237+0000 7f0c6b7fe700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0c5806b050 0x7f0c5806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.237+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0c5c08afd0 con 0x7f0c6c0d2040 2023-10-16T09:43:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.237+0000 7f0c53fff700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f0c58074690 0x7f0c58076ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.238+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 --> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c580771a0 con 0x7f0c58074690 2023-10-16T09:43:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.238+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c5c08d520 con 0x7f0c6c0d2040 2023-10-16T09:43:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.238+0000 7f0c6b7fe700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0c5806b050 0x7f0c5806d510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0c640021d0 tx=0x7f0c64001770 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.239+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8005a50 con 0x7fdacc15d920 2023-10-16T09:43:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.239+0000 7f0c6bfff700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f0c58074690 0x7f0c58076ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.239+0000 7f0c6bfff700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f0c58074690 0x7f0c58076ad0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.239+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac8005bd0 con 0x7fdacc15d920 2023-10-16T09:43:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.239+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8005d80 con 0x7fdacc15d920 2023-10-16T09:43:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.240+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc0b83a0 con 0x7fdacc15d920 2023-10-16T09:43:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.240+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc0b93c0 con 0x7fdacc15d920 2023-10-16T09:43:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.241+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdacc0cf2c0 con 0x7fdacc15d920 2023-10-16T09:43:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.242+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4012600 con 0x7fd7d4005330 2023-10-16T09:43:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.242+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e4012760 con 0x7fd7d4005330 2023-10-16T09:43:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.242+0000 7fd7eaffd700 1 -- 172.21.15.158:0/850925708 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4012b10 con 0x7fd7d4005330 2023-10-16T09:43:47.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.242+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdac8033640 con 0x7fdacc15d920 2023-10-16T09:43:47.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.242+0000 7fdac1ffb700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdab006af50 0x7fdab006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.242+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdac80aba60 con 0x7fdacc15d920 2023-10-16T09:43:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.243+0000 7fdad0814700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdab006af50 0x7fdab006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.243+0000 7fdac1ffb700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7fdab0074590 0x7fdab00769d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.243+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 --> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdab00770a0 con 0x7fdab0074590 2023-10-16T09:43:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.243+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdac80abf00 con 0x7fdacc15d920 2023-10-16T09:43:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.243+0000 7fdad0814700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdab006af50 0x7fdab006d410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdacc071fd0 tx=0x7fdabc007a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.243+0000 7fd7f22f6700 1 -- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7d4005330 msgr2=0x7fd7d400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.243+0000 7fd7f22f6700 1 --2- 172.21.15.158:0/850925708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7d4005330 0x7fd7d400ba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400a7d0 tx=0x7fd7d4005880 comp rx=0 tx=0).stop 2023-10-16T09:43:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.244+0000 7fd7f0893700 1 -- 172.21.15.158:0/850925708 reap_dead start 2023-10-16T09:43:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.244+0000 7fd7f22f6700 1 -- 172.21.15.158:0/850925708 shutdown_connections 2023-10-16T09:43:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.244+0000 7fd7f22f6700 1 -- 172.21.15.158:0/850925708 >> 172.21.15.158:0/850925708 conn(0x7fd7ec0cdf50 msgr2=0x7fd7ec0ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.244+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== osd.1 v2:172.21.15.158:6800/1822888471 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f0c580771a0 con 0x7f0c58074690 2023-10-16T09:43:47.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.244+0000 7fd7f22f6700 1 -- 172.21.15.158:0/850925708 shutdown_connections 2023-10-16T09:43:47.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.246+0000 7fdad1816700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7fdab0074590 0x7fdab00769d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.249+0000 7fd7f22f6700 1 -- 172.21.15.158:0/850925708 wait complete. 2023-10-16T09:43:47.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.249+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 --> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f690c06ceb0 con 0x7f68f8074440 2023-10-16T09:43:47.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.250+0000 7fd7f22f6700 1 Processor -- start 2023-10-16T09:43:47.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.250+0000 7fd7f22f6700 1 -- start start 2023-10-16T09:43:47.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.250+0000 7f6908ff9700 1 -- 172.21.15.158:0/1443292699 <== osd.4 v2:172.21.15.172:6824/3174967294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f690400f040 con 0x7f68f8074440 2023-10-16T09:43:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7f68f8074440 msgr2=0x7f68f8076880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7f69123ce700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7f68f8074440 0x7f68f8076880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f68f806af00 msgr2=0x7f68f806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7f69123ce700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f68f806af00 0x7f68f806d3c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f690c15b4a0 tx=0x7f68fc007ec0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f690c06e990 msgr2=0x7f690c167530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7f69123ce700 1 --2- 172.21.15.158:0/1443292699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f690c06e990 0x7f690c167530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6900015000 tx=0x7f6900004750 comp rx=0 tx=0).stop 2023-10-16T09:43:47.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7f691096b700 1 -- 172.21.15.158:0/1443292699 reap_dead start 2023-10-16T09:43:47.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7f22f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7ec1ffb10 0x7fd7ec0d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7f22f6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7ec164ee0 0x7fd7ec15c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7f22f6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7ec15ced0 0x7fd7ec2035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7f22f6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec15d380 con 0x7fd7ec1ffb10 2023-10-16T09:43:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7f22f6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec15c7e0 con 0x7fd7ec164ee0 2023-10-16T09:43:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7f22f6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c9720 con 0x7fd7ec15ced0 2023-10-16T09:43:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7ec164ee0 0x7fd7ec15c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7ec164ee0 0x7fd7ec15c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47736/0 (socket says 172.21.15.158:47736) 2023-10-16T09:43:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.251+0000 7fd7eb7fe700 1 -- 172.21.15.158:0/528637721 learned_addr learned my addr 172.21.15.158:0/528637721 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7f0893700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7ec15ced0 0x7fd7ec2035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7eb7fe700 1 -- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7ec15ced0 msgr2=0x7fd7ec2035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7eb7fe700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7ec15ced0 0x7fd7ec2035f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7eb7fe700 1 -- 172.21.15.158:0/528637721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7ec1ffb10 msgr2=0x7fd7ec0d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7eb7fe700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7ec1ffb10 0x7fd7ec0d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7eb7fe700 1 -- 172.21.15.158:0/528637721 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec1ffef0 con 0x7fd7ec164ee0 2023-10-16T09:43:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7eb7fe700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7ec164ee0 0x7fd7ec15c990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000b430 tx=0x7fd7e000b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0011830 con 0x7fd7ec164ee0 2023-10-16T09:43:47.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e0011990 con 0x7fd7ec164ee0 2023-10-16T09:43:47.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ec0c6ea0 con 0x7fd7ec164ee0 2023-10-16T09:43:47.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0011d20 con 0x7fd7ec164ee0 2023-10-16T09:43:47.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.252+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec0c7460 con 0x7fd7ec164ee0 2023-10-16T09:43:47.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.253+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 --> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e180ced80 con 0x7f5df80744c0 2023-10-16T09:43:47.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.253+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7ec165810 con 0x7fd7ec164ee0 2023-10-16T09:43:47.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd7e001a400 con 0x7fd7ec164ee0 2023-10-16T09:43:47.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd0dfd67700 1 Processor -- start 2023-10-16T09:43:47.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd7e8ff9700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd7d806af00 0x7fd7d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd0dfd67700 1 -- start start 2023-10-16T09:43:47.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd7e00897f0 con 0x7fd7ec164ee0 2023-10-16T09:43:47.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd7ebfff700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd7d806af00 0x7fd7d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd7e8ff9700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fd7d8074440 0x7fd7d8076880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.254+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 --> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7d8076f50 con 0x7fd7d8074440 2023-10-16T09:43:47.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.255+0000 7fd7f0893700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fd7d8074440 0x7fd7d8076880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.255+0000 7fd7ebfff700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd7d806af00 0x7fd7d806d3c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0096f0 tx=0x7fd7dc006cd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.255+0000 7fd7f0893700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fd7d8074440 0x7fd7d8076880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.255+0000 7fd0dfd67700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d80cfd60 0x7fd0d80b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.255+0000 7fd0dfd67700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d80af600 con 0x7fd0d815d990 2023-10-16T09:43:47.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0dfd67700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d806e9b0 con 0x7fd0d806e480 2023-10-16T09:43:47.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0dfd67700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80b2c90 con 0x7fd0d80cfd60 2023-10-16T09:43:47.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0ddb03700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd0d815d990 0x7fd0d80b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46740/0 (socket says 172.21.15.158:46740) 2023-10-16T09:43:47.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0ddb03700 1 -- 172.21.15.158:0/2850081858 learned_addr learned my addr 172.21.15.158:0/2850081858 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:47.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0de304700 1 --2- 172.21.15.158:0/2850081858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d80cfd60 0x7fd0d80b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== osd.7 v2:172.21.15.172:6808/2085135193 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fd7d8076f50 con 0x7fd7d8074440 2023-10-16T09:43:47.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0dcb01700 1 -- 172.21.15.158:0/2850081858 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530989885 0 0) 0x7fd0d806e9b0 con 0x7fd0d806e480 2023-10-16T09:43:47.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0dcb01700 1 -- 172.21.15.158:0/2850081858 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c40035b0 con 0x7fd0d806e480 2023-10-16T09:43:47.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0dcb01700 1 -- 172.21.15.158:0/2850081858 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906097398 0 0) 0x7fd0d80af600 con 0x7fd0d815d990 2023-10-16T09:43:47.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0dcb01700 1 -- 172.21.15.158:0/2850081858 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d806e9b0 con 0x7fd0d815d990 2023-10-16T09:43:47.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0de304700 1 -- 172.21.15.158:0/2850081858 >> v1:172.21.15.172:6789/0 conn(0x7fd0d806e480 legacy=0x7fd0d80cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0de304700 1 -- 172.21.15.158:0/2850081858 >> v1:172.21.15.158:6789/0 conn(0x7fd0d815d990 legacy=0x7fd0d80b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7f5e157fa700 1 -- 172.21.15.158:0/2821895274 <== osd.6 v2:172.21.15.172:6800/3389147915 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5e10001440 con 0x7f5df80744c0 2023-10-16T09:43:47.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.256+0000 7fd0de304700 1 -- 172.21.15.158:0/2850081858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80ce890 con 0x7fd0d80cfd60 2023-10-16T09:43:47.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 shutdown_connections 2023-10-16T09:43:47.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 >> 172.21.15.158:0/1443292699 conn(0x7f690c0cf2b0 msgr2=0x7f690c15a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7f5df80744c0 msgr2=0x7f5df8076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f5e1f0fb700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7f5df80744c0 0x7f5df8076900 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5df806af00 msgr2=0x7f5df806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f5e1f0fb700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5df806af00 0x7f5df806d3c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e180b4df0 tx=0x7f5e08009850 comp rx=0 tx=0).stop 2023-10-16T09:43:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 msgr2=0x7f5e18145e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f5e1f0fb700 1 --2- 172.21.15.158:0/2821895274 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e180ce9a0 0x7f5e18145e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e00000c00 tx=0x7f5e00004750 comp rx=0 tx=0).stop 2023-10-16T09:43:47.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7f5e1d698700 1 -- 172.21.15.158:0/2821895274 reap_dead start 2023-10-16T09:43:47.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7e0011e80 con 0x7fd7ec164ee0 2023-10-16T09:43:47.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.257+0000 7fdad1816700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7fdab0074590 0x7fdab00769d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.258+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 shutdown_connections 2023-10-16T09:43:47.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.258+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== osd.2 v2:172.21.15.158:6808/205317247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fdab00770a0 con 0x7fdab0074590 2023-10-16T09:43:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.260+0000 7f69123ce700 1 -- 172.21.15.158:0/1443292699 wait complete. 2023-10-16T09:43:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.260+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 shutdown_connections 2023-10-16T09:43:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.260+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 >> 172.21.15.158:0/2821895274 conn(0x7f5e180cf2b0 msgr2=0x7f5e180cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.261+0000 7fd0de304700 1 --2- 172.21.15.158:0/2850081858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d80cfd60 0x7fd0d80b2750 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009dc0 tx=0x7fd0d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac8211836c9218a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.261+0000 7fd0dcb01700 1 -- 172.21.15.158:0/2850081858 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000e070 con 0x7fd0d80cfd60 2023-10-16T09:43:47.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.263+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 shutdown_connections 2023-10-16T09:43:47.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.263+0000 7f5e1f0fb700 1 -- 172.21.15.158:0/2821895274 wait complete. 2023-10-16T09:43:47.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.263+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 --> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2838005cb0 con 0x7f28380015e0 2023-10-16T09:43:47.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.267+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 --> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f126015e290 con 0x7f12440744f0 2023-10-16T09:43:47.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.267+0000 7fd0dcb01700 1 -- 172.21.15.158:0/2850081858 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d000bb20 con 0x7fd0d80cfd60 2023-10-16T09:43:47.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.267+0000 7fd0dcb01700 1 -- 172.21.15.158:0/2850081858 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000be90 con 0x7fd0d80cfd60 2023-10-16T09:43:47.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.267+0000 7f283f7fe700 1 -- 172.21.15.158:0/3508444563 <== osd.0 v2:172.21.15.158:6809/38337391 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2838005cb0 con 0x7f28380015e0 2023-10-16T09:43:47.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.268+0000 7f12567fc700 1 -- 172.21.15.158:0/1638930955 <== osd.5 v2:172.21.15.172:6816/2983726905 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1258005600 con 0x7f12440744f0 2023-10-16T09:43:47.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.268+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2850081858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d80cfd60 msgr2=0x7fd0d80b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.268+0000 7fd0dfd67700 1 --2- 172.21.15.158:0/2850081858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d80cfd60 0x7fd0d80b2750 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009dc0 tx=0x7fd0d000b330 comp rx=0 tx=0).stop 2023-10-16T09:43:47.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.268+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f12440744f0 msgr2=0x7f1244076930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.268+0000 7f1267e08700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f12440744f0 0x7f1244076930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.270+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f124406af30 msgr2=0x7f124406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.270+0000 7f1267e08700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f124406af30 0x7f124406d3f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f12600cac40 tx=0x7f1250008040 comp rx=0 tx=0).stop 2023-10-16T09:43:47.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.270+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006e7c0 msgr2=0x7f12600ab330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.270+0000 7f1267e08700 1 --2- 172.21.15.158:0/1638930955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006e7c0 0x7f12600ab330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f125c009c40 tx=0x7f125c004750 comp rx=0 tx=0).stop 2023-10-16T09:43:47.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.270+0000 7f12663a5700 1 -- 172.21.15.158:0/1638930955 reap_dead start 2023-10-16T09:43:47.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f28380015e0 msgr2=0x7f2838003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f285d1d5700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f28380015e0 0x7f2838003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f284406af50 msgr2=0x7f284406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f285d1d5700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f284406af50 0x7f284406d410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28580b13b0 tx=0x7f2848008030 comp rx=0 tx=0).stop 2023-10-16T09:43:47.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580b0d30 msgr2=0x7f28580b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f285d1d5700 1 --2- 172.21.15.158:0/3508444563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28580b0d30 0x7f28580b1ed0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f284c00b580 tx=0x7f284c00b940 comp rx=0 tx=0).stop 2023-10-16T09:43:47.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f285759e700 1 -- 172.21.15.158:0/3508444563 reap_dead start 2023-10-16T09:43:47.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.273+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 --> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c6c0ced50 con 0x7f0c58074690 2023-10-16T09:43:47.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 shutdown_connections 2023-10-16T09:43:47.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 >> 172.21.15.158:0/1638930955 conn(0x7f12600cf2b0 msgr2=0x7f1260160840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f0c53fff700 1 -- 172.21.15.158:0/1023279136 <== osd.1 v2:172.21.15.158:6800/1822888471 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0c60009530 con 0x7f0c58074690 2023-10-16T09:43:47.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f0c58074690 msgr2=0x7f0c58076ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f0c71fd5700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f0c58074690 0x7f0c58076ad0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0c5806b050 msgr2=0x7f0c5806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f0c71fd5700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f0c5806b050 0x7f0c5806d510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0c640021d0 tx=0x7f0c64001770 comp rx=0 tx=0).stop 2023-10-16T09:43:47.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c6c0d2040 msgr2=0x7f0c6c1ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.274+0000 7f0c71fd5700 1 --2- 172.21.15.158:0/1023279136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c6c0d2040 0x7f0c6c1ff770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c08dc20 tx=0x7f0c5c004750 comp rx=0 tx=0).stop 2023-10-16T09:43:47.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.275+0000 7f0c6bfff700 1 -- 172.21.15.158:0/1023279136 reap_dead start 2023-10-16T09:43:47.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.275+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 shutdown_connections 2023-10-16T09:43:47.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.275+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 >> 172.21.15.158:0/1023279136 conn(0x7f0c6c09ca30 msgr2=0x7f0c6c0ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.275+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 shutdown_connections 2023-10-16T09:43:47.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.275+0000 7f0c71fd5700 1 -- 172.21.15.158:0/1023279136 wait complete. 2023-10-16T09:43:47.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.276+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 shutdown_connections 2023-10-16T09:43:47.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.276+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 >> 172.21.15.158:0/3508444563 conn(0x7f28580cbd30 msgr2=0x7f285806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.276+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 shutdown_connections 2023-10-16T09:43:47.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.276+0000 7f285d1d5700 1 -- 172.21.15.158:0/3508444563 wait complete. 2023-10-16T09:43:47.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.278+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 shutdown_connections 2023-10-16T09:43:47.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.278+0000 7f1267e08700 1 -- 172.21.15.158:0/1638930955 wait complete. 2023-10-16T09:43:47.352 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:47.352 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T09:43:47.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.279+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 --> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdacc0b8b30 con 0x7fdab0074590 2023-10-16T09:43:47.354 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:47.354 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T09:43:47.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.279+0000 7fdac1ffb700 1 -- 172.21.15.158:0/2064706363 <== osd.2 v2:172.21.15.158:6808/205317247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdac4003e50 con 0x7fdab0074590 2023-10-16T09:43:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7fdab0074590 msgr2=0x7fdab00769d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7fdab0074590 0x7fdab00769d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdab006af50 msgr2=0x7fdab006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdab006af50 0x7fdab006d410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdacc071fd0 tx=0x7fdabc007a20 comp rx=0 tx=0).stop 2023-10-16T09:43:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdacc15d920 msgr2=0x7fdacc0b8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 --2- 172.21.15.158:0/2064706363 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdacc15d920 0x7fdacc0b8e80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009fa0 tx=0x7fdac800b830 comp rx=0 tx=0).stop 2023-10-16T09:43:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad1816700 1 -- 172.21.15.158:0/2064706363 reap_dead start 2023-10-16T09:43:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 shutdown_connections 2023-10-16T09:43:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.280+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 >> 172.21.15.158:0/2064706363 conn(0x7fdacc09ca30 msgr2=0x7fdacc0c7910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.282+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 shutdown_connections 2023-10-16T09:43:47.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.282+0000 7fdad3279700 1 -- 172.21.15.158:0/2064706363 wait complete. 2023-10-16T09:43:47.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.282+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2850081858 shutdown_connections 2023-10-16T09:43:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.282+0000 7fd0dfd67700 1 --2- 172.21.15.158:0/2850081858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d80cfd60 0x7fd0d80b2750 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.282+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2850081858 >> 172.21.15.158:0/2850081858 conn(0x7fd0d806b9e0 msgr2=0x7fd0d806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.282+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2850081858 shutdown_connections 2023-10-16T09:43:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.282+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2850081858 wait complete. 2023-10-16T09:43:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.283+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 --> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd7ec070f20 con 0x7fd7d8074440 2023-10-16T09:43:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.284+0000 7fd7e8ff9700 1 -- 172.21.15.158:0/528637721 <== osd.7 v2:172.21.15.172:6808/2085135193 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd7e4012400 con 0x7fd7d8074440 2023-10-16T09:43:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.284+0000 7fd0dfd67700 1 Processor -- start 2023-10-16T09:43:47.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.284+0000 7fd0dfd67700 1 -- start start 2023-10-16T09:43:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.284+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fd7d8074440 msgr2=0x7fd7d8076880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.284+0000 7fd7f22f6700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fd7d8074440 0x7fd7d8076880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd7d806af00 msgr2=0x7fd7d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd7d806af00 0x7fd7d806d3c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0096f0 tx=0x7fd7dc006cd0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7ec164ee0 msgr2=0x7fd7ec15c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 --2- 172.21.15.158:0/528637721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7ec164ee0 0x7fd7ec15c990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000b430 tx=0x7fd7e000b7f0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f0893700 1 -- 172.21.15.158:0/528637721 reap_dead start 2023-10-16T09:43:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 shutdown_connections 2023-10-16T09:43:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 >> 172.21.15.158:0/528637721 conn(0x7fd7ec0cdf50 msgr2=0x7fd7ec0cd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 shutdown_connections 2023-10-16T09:43:47.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd7f22f6700 1 -- 172.21.15.158:0/528637721 wait complete. 2023-10-16T09:43:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd0dfd67700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0d806e480 0x7fd0d80c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd0dfd67700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d80cfd60 0x7fd0d815dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd0dfd67700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d815d990 0x7fd0d814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd0dfd67700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d815e2b0 con 0x7fd0d806e480 2023-10-16T09:43:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd0dfd67700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8160ed0 con 0x7fd0d80cfd60 2023-10-16T09:43:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.285+0000 7fd0dfd67700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80b3c80 con 0x7fd0d815d990 2023-10-16T09:43:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d80cfd60 0x7fd0d815dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d80cfd60 0x7fd0d815dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47750/0 (socket says 172.21.15.158:47750) 2023-10-16T09:43:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 -- 172.21.15.158:0/2185782792 learned_addr learned my addr 172.21.15.158:0/2185782792 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0ddb03700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0d806e480 0x7fd0d80c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 -- 172.21.15.158:0/2185782792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d815d990 msgr2=0x7fd0d814c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0d815d990 0x7fd0d814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 -- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0d806e480 msgr2=0x7fd0d80c4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0d806e480 0x7fd0d80c4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 -- 172.21.15.158:0/2185782792 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d0009a70 con 0x7fd0d80cfd60 2023-10-16T09:43:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dd302700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d80cfd60 0x7fd0d815dd70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c4f0 tx=0x7fd0c800c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80052d0 con 0x7fd0d80cfd60 2023-10-16T09:43:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c8005430 con 0x7fd0d80cfd60 2023-10-16T09:43:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80038c0 con 0x7fd0d80cfd60 2023-10-16T09:43:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.286+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d8160880 con 0x7fd0d80cfd60 2023-10-16T09:43:47.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.287+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d80b3f60 con 0x7fd0d80cfd60 2023-10-16T09:43:47.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.288+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd0c8003a20 con 0x7fd0d80cfd60 2023-10-16T09:43:47.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.288+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0d8160700 con 0x7fd0d80cfd60 2023-10-16T09:43:47.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.289+0000 7fd0ce7fc700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd0bc06af00 0x7fd0bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.289+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd0c8088e60 con 0x7fd0d80cfd60 2023-10-16T09:43:47.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.289+0000 7fd0ddb03700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd0bc06af00 0x7fd0bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.289+0000 7fd0ce7fc700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fd0bc0744c0 0x7fd0bc076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.289+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 --> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0bc076fd0 con 0x7fd0bc0744c0 2023-10-16T09:43:47.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.289+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0c808b520 con 0x7fd0d80cfd60 2023-10-16T09:43:47.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.290+0000 7fd0ddb03700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd0bc06af00 0x7fd0bc06d3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4005910 tx=0x7fd0d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.290+0000 7fd0de304700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fd0bc0744c0 0x7fd0bc076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.290+0000 7fd0de304700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fd0bc0744c0 0x7fd0bc076900 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.291+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== osd.3 v2:172.21.15.158:6824/969498875 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fd0bc076fd0 con 0x7fd0bc0744c0 2023-10-16T09:43:47.374 INFO:teuthology.orchestra.run.smithi158.stdout:55834574858 2023-10-16T09:43:47.374 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T09:43:47.375 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:47.375 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T09:43:47.376 INFO:teuthology.orchestra.run.smithi158.stdout:55834574858 2023-10-16T09:43:47.377 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T09:43:47.378 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:47.378 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T09:43:47.379 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:47.379 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T09:43:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.316+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 --> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0d80cee50 con 0x7fd0bc0744c0 2023-10-16T09:43:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.316+0000 7fd0ce7fc700 1 -- 172.21.15.158:0/2185782792 <== osd.3 v2:172.21.15.158:6824/969498875 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd0d0003a20 con 0x7fd0bc0744c0 2023-10-16T09:43:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fd0bc0744c0 msgr2=0x7fd0bc076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fd0bc0744c0 0x7fd0bc076900 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd0bc06af00 msgr2=0x7fd0bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd0bc06af00 0x7fd0bc06d3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4005910 tx=0x7fd0d4009040 comp rx=0 tx=0).stop 2023-10-16T09:43:47.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d80cfd60 msgr2=0x7fd0d815dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 --2- 172.21.15.158:0/2185782792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d80cfd60 0x7fd0d815dd70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c4f0 tx=0x7fd0c800c8b0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0de304700 1 -- 172.21.15.158:0/2185782792 reap_dead start 2023-10-16T09:43:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 shutdown_connections 2023-10-16T09:43:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 >> 172.21.15.158:0/2185782792 conn(0x7fd0d806b9e0 msgr2=0x7fd0d80b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 shutdown_connections 2023-10-16T09:43:47.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.317+0000 7fd0dfd67700 1 -- 172.21.15.158:0/2185782792 wait complete. 2023-10-16T09:43:47.385 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:47.386 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T09:43:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.598+0000 7f3ad6b09700 1 Processor -- start 2023-10-16T09:43:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.598+0000 7f3ad6b09700 1 -- start start 2023-10-16T09:43:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.599+0000 7f3ad6b09700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ad006e990 0x7f3ad015cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7f3ad6b09700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3ad015def0 0x7f3ad006c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7f3ad6b09700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad00cd6c0 con 0x7f3ad00cfdd0 2023-10-16T09:43:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7f3ad6b09700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad006ccc0 con 0x7f3ad015def0 2023-10-16T09:43:47.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7f3ad6b09700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad0070b30 con 0x7f3ad006e990 2023-10-16T09:43:47.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7f3acffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ad006e990 0x7f3ad015cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7f3acffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ad006e990 0x7f3ad015cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47762/0 (socket says 172.21.15.158:47762) 2023-10-16T09:43:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7f3acffff700 1 -- 172.21.15.158:0/3078918629 learned_addr learned my addr 172.21.15.158:0/3078918629 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7fe496378700 1 Processor -- start 2023-10-16T09:43:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.600+0000 7fe496378700 1 -- start start 2023-10-16T09:43:47.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7f3acffff700 1 -- 172.21.15.158:0/3078918629 >> v2:172.21.15.158:3300/0 conn(0x7f3ad015def0 msgr2=0x7f3ad006c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7f3acffff700 1 --2- 172.21.15.158:0/3078918629 >> v2:172.21.15.158:3300/0 conn(0x7f3ad015def0 0x7f3ad006c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7f3acffff700 1 -- 172.21.15.158:0/3078918629 >> v1:172.21.15.158:6789/0 conn(0x7f3ad00cfdd0 legacy=0x7f3ad006e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7f3acffff700 1 -- 172.21.15.158:0/3078918629 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad006d750 con 0x7f3ad006e990 2023-10-16T09:43:47.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7f3acffff700 1 --2- 172.21.15.158:0/3078918629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad006e990 0x7f3ad015cde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ad00703f0 tx=0x7f3ac400be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=398ad202ab9b097 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7fe496378700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 0x7fe49006de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7f4620ee6700 1 Processor -- start 2023-10-16T09:43:47.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7fe496378700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe49006afe0 0x7fe4900cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7fe496378700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4900afc70 con 0x7fe4900c3960 2023-10-16T09:43:47.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7fe496378700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe490071e40 con 0x7fe49006afe0 2023-10-16T09:43:47.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7fe496378700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe490072120 con 0x7fe4900af890 2023-10-16T09:43:47.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.601+0000 7f4620ee6700 1 -- start start 2023-10-16T09:43:47.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f4620ee6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c161e30 0x7f461c070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f4620ee6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f461c0c9b90 0x7f461c0ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f4620ee6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f461c162210 con 0x7f461c0b0d30 2023-10-16T09:43:47.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f4620ee6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c0cdc30 con 0x7f461c0c9b90 2023-10-16T09:43:47.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f4620ee6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0cd020 con 0x7f461c161e30 2023-10-16T09:43:47.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f461a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c161e30 0x7f461c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f461a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c161e30 0x7f461c070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57426/0 (socket says 172.21.15.158:57426) 2023-10-16T09:43:47.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f461a59c700 1 -- 172.21.15.158:0/1181091060 learned_addr learned my addr 172.21.15.158:0/1181091060 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f461a59c700 1 -- 172.21.15.158:0/1181091060 >> v2:172.21.15.172:3300/0 conn(0x7f461c0c9b90 msgr2=0x7f461c0ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f461a59c700 1 --2- 172.21.15.158:0/1181091060 >> v2:172.21.15.172:3300/0 conn(0x7f461c0c9b90 0x7f461c0ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f461a59c700 1 -- 172.21.15.158:0/1181091060 >> v1:172.21.15.158:6789/0 conn(0x7f461c0b0d30 legacy=0x7f461c0ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f461a59c700 1 -- 172.21.15.158:0/1181091060 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0cd300 con 0x7f461c161e30 2023-10-16T09:43:47.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f461a59c700 1 --2- 172.21.15.158:0/1181091060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c161e30 0x7f461c070870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f460c0098f0 tx=0x7f460c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5410333f96e9abae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe49006afe0 0x7fe4900cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe49006afe0 0x7fe4900cec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46586/0 (socket says 172.21.15.158:46586) 2023-10-16T09:43:47.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 -- 172.21.15.158:0/2300429372 learned_addr learned my addr 172.21.15.158:0/2300429372 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 -- 172.21.15.158:0/2300429372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 msgr2=0x7fe49006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 --2- 172.21.15.158:0/2300429372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 0x7fe49006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 -- 172.21.15.158:0/2300429372 >> v1:172.21.15.158:6789/0 conn(0x7fe4900c3960 legacy=0x7fe49006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 -- 172.21.15.158:0/2300429372 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe490072400 con 0x7fe49006afe0 2023-10-16T09:43:47.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.602+0000 7fe494915700 1 --2- 172.21.15.158:0/2300429372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49006afe0 0x7fe4900cec70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe488009e50 tx=0x7fe48800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e0d5af6c580047 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7fe48effd700 1 -- 172.21.15.158:0/2300429372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48800e070 con 0x7fe49006afe0 2023-10-16T09:43:47.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7fe48effd700 1 -- 172.21.15.158:0/2300429372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48800b9c0 con 0x7fe49006afe0 2023-10-16T09:43:47.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7fe48effd700 1 -- 172.21.15.158:0/2300429372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48800bd70 con 0x7fe49006afe0 2023-10-16T09:43:47.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7fe496378700 1 -- 172.21.15.158:0/2300429372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49006afe0 msgr2=0x7fe4900cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7fe496378700 1 --2- 172.21.15.158:0/2300429372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49006afe0 0x7fe4900cec70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe488009e50 tx=0x7fe48800b330 comp rx=0 tx=0).stop 2023-10-16T09:43:47.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7fe496378700 1 -- 172.21.15.158:0/2300429372 shutdown_connections 2023-10-16T09:43:47.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7fe496378700 1 --2- 172.21.15.158:0/2300429372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49006afe0 0x7fe4900cec70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7fe496378700 1 --2- 172.21.15.158:0/2300429372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 0x7fe49006de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7fe496378700 1 -- 172.21.15.158:0/2300429372 >> 172.21.15.158:0/2300429372 conn(0x7fe49006b9e0 msgr2=0x7fe49006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.606+0000 7fe496378700 1 -- 172.21.15.158:0/2300429372 shutdown_connections 2023-10-16T09:43:47.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.606+0000 7fe496378700 1 -- 172.21.15.158:0/2300429372 wait complete. 2023-10-16T09:43:47.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.607+0000 7fe496378700 1 Processor -- start 2023-10-16T09:43:47.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f3acf7fe700 1 -- 172.21.15.158:0/3078918629 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400dcf0 con 0x7f3ad006e990 2023-10-16T09:43:47.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f3acf7fe700 1 -- 172.21.15.158:0/3078918629 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac400de50 con 0x7f3ad006e990 2023-10-16T09:43:47.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.603+0000 7f3acf7fe700 1 -- 172.21.15.158:0/3078918629 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4004230 con 0x7f3ad006e990 2023-10-16T09:43:47.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f3ad6b09700 1 -- 172.21.15.158:0/3078918629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad006e990 msgr2=0x7f3ad015cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/3078918629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad006e990 0x7f3ad015cde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ad00703f0 tx=0x7f3ac400be70 comp rx=0 tx=0).stop 2023-10-16T09:43:47.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f3ad6b09700 1 -- 172.21.15.158:0/3078918629 shutdown_connections 2023-10-16T09:43:47.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/3078918629 >> v2:172.21.15.158:3300/0 conn(0x7f3ad015def0 0x7f3ad006c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/3078918629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad006e990 0x7f3ad015cde0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.604+0000 7f3ad6b09700 1 -- 172.21.15.158:0/3078918629 >> 172.21.15.158:0/3078918629 conn(0x7f3ad00cf2b0 msgr2=0x7f3ad00cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7f3ad6b09700 1 -- 172.21.15.158:0/3078918629 shutdown_connections 2023-10-16T09:43:47.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.605+0000 7f3ad6b09700 1 -- 172.21.15.158:0/3078918629 wait complete. 2023-10-16T09:43:47.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.606+0000 7f3ad6b09700 1 Processor -- start 2023-10-16T09:43:47.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.606+0000 7f3ad6b09700 1 -- start start 2023-10-16T09:43:47.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad6b09700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e990 0x7f3ad00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad6b09700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad00cfdd0 0x7f3ad01fa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad6b09700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad015def0 0x7f3ad0157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad6b09700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad00c6380 con 0x7f3ad006e990 2023-10-16T09:43:47.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad6b09700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01fb400 con 0x7f3ad00cfdd0 2023-10-16T09:43:47.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad6b09700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01fb6e0 con 0x7f3ad015def0 2023-10-16T09:43:47.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad015def0 0x7f3ad0157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad48a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e990 0x7f3ad00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad015def0 0x7f3ad0157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57432/0 (socket says 172.21.15.158:57432) 2023-10-16T09:43:47.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 -- 172.21.15.158:0/2885692203 learned_addr learned my addr 172.21.15.158:0/2885692203 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3acffff700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad00cfdd0 0x7f3ad01fa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7fe496378700 1 -- start start 2023-10-16T09:43:47.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 -- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad00cfdd0 msgr2=0x7f3ad01fa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad00cfdd0 0x7f3ad01fa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 -- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e990 msgr2=0x7f3ad00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e990 0x7f3ad00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.608+0000 7f3ad50a6700 1 -- 172.21.15.158:0/2885692203 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac400b7f0 con 0x7f3ad015def0 2023-10-16T09:43:47.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7f3ad50a6700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad015def0 0x7f3ad0157580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b970 tx=0x7f3ac800bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7f3a5b39e700 1 Processor -- start 2023-10-16T09:43:47.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7f3acd7fa700 1 -- 172.21.15.158:0/2885692203 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800d860 con 0x7f3ad015def0 2023-10-16T09:43:47.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7f3acd7fa700 1 -- 172.21.15.158:0/2885692203 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac800d9e0 con 0x7f3ad015def0 2023-10-16T09:43:47.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7f3acd7fa700 1 -- 172.21.15.158:0/2885692203 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800dd90 con 0x7f3ad015def0 2023-10-16T09:43:47.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7f3a5b39e700 1 -- start start 2023-10-16T09:43:47.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe496378700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49006afe0 0x7fe490150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe496378700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 0x7fe490151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe496378700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4900c3960 0x7fe4901569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe496378700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490151180 con 0x7fe4900c3960 2023-10-16T09:43:47.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe496378700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490150bc0 con 0x7fe49006afe0 2023-10-16T09:43:47.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe496378700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe490150ea0 con 0x7fe4900af890 2023-10-16T09:43:47.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe48f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 0x7fe490151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe48ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49006afe0 0x7fe490150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe494915700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4900c3960 0x7fe4901569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe494915700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4900c3960 0x7fe4901569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46608/0 (socket says 172.21.15.158:46608) 2023-10-16T09:43:47.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7fe494915700 1 -- 172.21.15.158:0/656630299 learned_addr learned my addr 172.21.15.158:0/656630299 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.609+0000 7f461959a700 1 -- 172.21.15.158:0/1181091060 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c004230 con 0x7f461c161e30 2023-10-16T09:43:47.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f461959a700 1 -- 172.21.15.158:0/1181091060 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c004390 con 0x7f461c161e30 2023-10-16T09:43:47.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f461959a700 1 -- 172.21.15.158:0/1181091060 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c0036a0 con 0x7f461c161e30 2023-10-16T09:43:47.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f4620ee6700 1 -- 172.21.15.158:0/1181091060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c161e30 msgr2=0x7f461c070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f4620ee6700 1 --2- 172.21.15.158:0/1181091060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c161e30 0x7f461c070870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f460c0098f0 tx=0x7f460c009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:47.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7fe494915700 1 -- 172.21.15.158:0/656630299 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 msgr2=0x7fe490151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7fe494915700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 0x7fe490151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7fe494915700 1 -- 172.21.15.158:0/656630299 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49006afe0 msgr2=0x7fe490150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7fe494915700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49006afe0 0x7fe490150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7fe494915700 1 -- 172.21.15.158:0/656630299 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe488009b30 con 0x7fe4900c3960 2023-10-16T09:43:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7fe494915700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4900c3960 0x7fe4901569a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe488012000 tx=0x7fe488004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad00c6590 con 0x7f3ad015def0 2023-10-16T09:43:47.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad0146f20 con 0x7f3ad015def0 2023-10-16T09:43:47.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f3a5b39e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a54066980 0x7f3a5406bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f3a5b39e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a540b3960 con 0x7f3a5406c000 2023-10-16T09:43:47.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f3a5b39e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a540b2a70 con 0x7f3a540665a0 2023-10-16T09:43:47.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.610+0000 7f3a5b39e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5415af70 con 0x7f3a54066980 2023-10-16T09:43:47.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a59b9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a54066980 0x7f3a5406bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a59b9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a54066980 0x7f3a5406bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46610/0 (socket says 172.21.15.158:46610) 2023-10-16T09:43:47.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a59b9b700 1 -- 172.21.15.158:0/279361939 learned_addr learned my addr 172.21.15.158:0/279361939 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a59b9b700 1 -- 172.21.15.158:0/279361939 >> v1:172.21.15.172:6789/0 conn(0x7f3a540665a0 legacy=0x7f3a540b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a59b9b700 1 -- 172.21.15.158:0/279361939 >> v1:172.21.15.158:6789/0 conn(0x7f3a5406c000 legacy=0x7f3a5415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a59b9b700 1 -- 172.21.15.158:0/279361939 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5415b250 con 0x7f3a54066980 2023-10-16T09:43:47.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad00c6ab0 con 0x7f3ad015def0 2023-10-16T09:43:47.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a59b9b700 1 --2- 172.21.15.158:0/279361939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a54066980 0x7f3a5406bac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3a500098f0 tx=0x7f3a50009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ea04d92e243c01 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7f3a5939a700 1 -- 172.21.15.158:0/279361939 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50004230 con 0x7f3a54066980 2023-10-16T09:43:47.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f3a5939a700 1 -- 172.21.15.158:0/279361939 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50004390 con 0x7f3a54066980 2023-10-16T09:43:47.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f3a5939a700 1 -- 172.21.15.158:0/279361939 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a500036a0 con 0x7f3a54066980 2023-10-16T09:43:47.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7fe48cff9700 1 -- 172.21.15.158:0/656630299 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488004250 con 0x7fe4900c3960 2023-10-16T09:43:47.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7fe48cff9700 1 -- 172.21.15.158:0/656630299 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4880043b0 con 0x7fe4900c3960 2023-10-16T09:43:47.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.611+0000 7fe48cff9700 1 -- 172.21.15.158:0/656630299 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488004740 con 0x7fe4900c3960 2023-10-16T09:43:47.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f4620ee6700 1 -- 172.21.15.158:0/1181091060 shutdown_connections 2023-10-16T09:43:47.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f4620ee6700 1 --2- 172.21.15.158:0/1181091060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c161e30 0x7f461c070870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f4620ee6700 1 --2- 172.21.15.158:0/1181091060 >> v2:172.21.15.172:3300/0 conn(0x7f461c0c9b90 0x7f461c0ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f4620ee6700 1 -- 172.21.15.158:0/1181091060 >> 172.21.15.158:0/1181091060 conn(0x7f461c0cbd30 msgr2=0x7f461c0cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f4620ee6700 1 -- 172.21.15.158:0/1181091060 shutdown_connections 2023-10-16T09:43:47.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4901514b0 con 0x7fe4900c3960 2023-10-16T09:43:47.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe490150790 con 0x7fe4900c3960 2023-10-16T09:43:47.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.612+0000 7f3a5b39e700 1 -- 172.21.15.158:0/279361939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a54066980 msgr2=0x7f3a5406bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.613+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/279361939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a54066980 0x7f3a5406bac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3a500098f0 tx=0x7f3a50009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:47.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.613+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4901519d0 con 0x7fe4900c3960 2023-10-16T09:43:47.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.613+0000 7f3acd7fa700 1 -- 172.21.15.158:0/2885692203 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3ac800e430 con 0x7f3ad015def0 2023-10-16T09:43:47.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7f3a5b39e700 1 -- 172.21.15.158:0/279361939 shutdown_connections 2023-10-16T09:43:47.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/279361939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a54066980 0x7f3a5406bac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7f3a5b39e700 1 -- 172.21.15.158:0/279361939 >> 172.21.15.158:0/279361939 conn(0x7f3a540cff90 msgr2=0x7f3a54070fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7fa647f7e700 1 Processor -- start 2023-10-16T09:43:47.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7f3acd7fa700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7f3acd7fa700 1 -- 172.21.15.158:0/2885692203 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3ac8089b00 con 0x7f3ad015def0 2023-10-16T09:43:47.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7f3ad48a5700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7f3ad48a5700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0007fd0 tx=0x7f3ac0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.614+0000 7fa647f7e700 1 -- start start 2023-10-16T09:43:47.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7fa647f7e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa64006e990 0x7fa64015def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7fa647f7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64006c780 0x7fa640071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7fa647f7e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6400cd6c0 con 0x7fa64015cde0 2023-10-16T09:43:47.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7fa647f7e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa640070b30 con 0x7fa64006e990 2023-10-16T09:43:47.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7fa647f7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64006d750 con 0x7fa64006c780 2023-10-16T09:43:47.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7f3a5b39e700 1 -- 172.21.15.158:0/279361939 shutdown_connections 2023-10-16T09:43:47.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64006c780 0x7fa640071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64006c780 0x7fa640071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57454/0 (socket says 172.21.15.158:57454) 2023-10-16T09:43:47.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 -- 172.21.15.158:0/2115839488 learned_addr learned my addr 172.21.15.158:0/2115839488 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.613+0000 7f4620ee6700 1 -- 172.21.15.158:0/1181091060 wait complete. 2023-10-16T09:43:47.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7f4620ee6700 1 Processor -- start 2023-10-16T09:43:47.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7f4620ee6700 1 -- start start 2023-10-16T09:43:47.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f4620ee6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c0b0d30 0x7f461c0b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f4620ee6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c0c9b90 0x7f461c164a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f4620ee6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f461c161e30 0x7f461c160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f4620ee6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c1643e0 con 0x7f461c161e30 2023-10-16T09:43:47.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f4620ee6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c1623a0 con 0x7f461c0b0d30 2023-10-16T09:43:47.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f4620ee6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0b4a40 con 0x7f461c0c9b90 2023-10-16T09:43:47.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 -- 172.21.15.158:0/2115839488 >> v2:172.21.15.172:3300/0 conn(0x7fa64006e990 msgr2=0x7fa64015def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 --2- 172.21.15.158:0/2115839488 >> v2:172.21.15.172:3300/0 conn(0x7fa64006e990 0x7fa64015def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 -- 172.21.15.158:0/2115839488 >> v1:172.21.15.172:6789/0 conn(0x7fa64015cde0 legacy=0x7fa64006e280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T09:43:47.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f461a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c0b0d30 0x7f461c0b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f461a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c0b0d30 0x7f461c0b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47804/0 (socket says 172.21.15.158:47804) 2023-10-16T09:43:47.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7f461a59c700 1 -- 172.21.15.158:0/332745536 learned_addr learned my addr 172.21.15.158:0/332745536 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 -- 172.21.15.158:0/2115839488 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64006cc10 con 0x7fa64006c780 2023-10-16T09:43:47.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.616+0000 7fa64651b700 1 --2- 172.21.15.158:0/2115839488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64006c780 0x7fa640071260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d370 tx=0x7fa63800d680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bef4a1d7618adfdb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7fe48cff9700 1 -- 172.21.15.158:0/656630299 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe48801c440 con 0x7fe4900c3960 2023-10-16T09:43:47.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f461a59c700 1 -- 172.21.15.158:0/332745536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c0c9b90 msgr2=0x7f461c164a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:47.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f461a59c700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c0c9b90 0x7f461c164a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f461a59c700 1 -- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f461c161e30 msgr2=0x7f461c160560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:47.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.615+0000 7f3a5b39e700 1 -- 172.21.15.158:0/279361939 wait complete. 2023-10-16T09:43:47.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f3a5b39e700 1 Processor -- start 2023-10-16T09:43:47.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f461ad9d700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f461c161e30 0x7f461c160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f461a59c700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f461c161e30 0x7f461c160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f461a59c700 1 -- 172.21.15.158:0/332745536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c009550 con 0x7f461c0b0d30 2023-10-16T09:43:47.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f461ad9d700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f461c161e30 0x7f461c160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:47.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7fe48cff9700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe47c06af50 0x7fe47c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7fe48cff9700 1 -- 172.21.15.158:0/656630299 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe48808ade0 con 0x7fe4900c3960 2023-10-16T09:43:47.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f461a59c700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c0b0d30 0x7f461c0b41a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f461c0cd160 tx=0x7f460c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f460affd700 1 -- 172.21.15.158:0/332745536 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c003db0 con 0x7f461c0b0d30 2023-10-16T09:43:47.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f460affd700 1 -- 172.21.15.158:0/332745536 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c020070 con 0x7f461c0b0d30 2023-10-16T09:43:47.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f460affd700 1 -- 172.21.15.158:0/332745536 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c011870 con 0x7f461c0b0d30 2023-10-16T09:43:47.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7fe48ffff700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe47c06af50 0x7fe47c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f3acd7fa700 1 -- 172.21.15.158:0/2885692203 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ac80549a0 con 0x7f3ad015def0 2023-10-16T09:43:47.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.617+0000 7f3a5b39e700 1 -- start start 2023-10-16T09:43:47.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f3a5b39e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a540665a0 0x7f3a54152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f3a5b39e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a54066980 0x7f3a54155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f3a5b39e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a5406c000 0x7f3a541583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f3a5b39e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54156280 con 0x7f3a540665a0 2023-10-16T09:43:47.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f3a5b39e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54156560 con 0x7f3a5406c000 2023-10-16T09:43:47.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.618+0000 7f3a5b39e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54156840 con 0x7f3a54066980 2023-10-16T09:43:47.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a540665a0 0x7f3a54152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a540665a0 0x7f3a54152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46632/0 (socket says 172.21.15.158:46632) 2023-10-16T09:43:47.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 -- 172.21.15.158:0/2371369621 learned_addr learned my addr 172.21.15.158:0/2371369621 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 -- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a54066980 msgr2=0x7f3a54155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a54066980 0x7f3a54155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 -- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a5406c000 msgr2=0x7f3a541583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5ab9d700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a5406c000 0x7f3a541583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a5406c000 0x7f3a541583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f3a5a39c700 1 -- 172.21.15.158:0/2371369621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a50009550 con 0x7f3a540665a0 2023-10-16T09:43:47.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c0b4d20 con 0x7f461c0b0d30 2023-10-16T09:43:47.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c1628d0 con 0x7f461c0b0d30 2023-10-16T09:43:47.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.619+0000 7fe48ffff700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe47c06af50 0x7fe47c06d410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe484006fd0 tx=0x7fe484008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.620+0000 7fa644d18700 1 -- 172.21.15.158:0/2115839488 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638010070 con 0x7fa64006c780 2023-10-16T09:43:47.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.620+0000 7fa644d18700 1 -- 172.21.15.158:0/2115839488 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63800dcd0 con 0x7fa64006c780 2023-10-16T09:43:47.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.620+0000 7fa644d18700 1 -- 172.21.15.158:0/2115839488 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638007d50 con 0x7fa64006c780 2023-10-16T09:43:47.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.621+0000 7f3a5ab9d700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a5406c000 0x7f3a541583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:47.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.621+0000 7f3a5a39c700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a540665a0 0x7f3a54152ad0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c4d0 tx=0x7f3a4400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.621+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c0c9750 con 0x7f461c0b0d30 2023-10-16T09:43:47.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.621+0000 7f460affd700 1 -- 172.21.15.158:0/332745536 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f460c022030 con 0x7f461c0b0d30 2023-10-16T09:43:47.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.621+0000 7f460affd700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460406b0b0 0x7f460406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.622+0000 7f460affd700 1 -- 172.21.15.158:0/332745536 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f460c08b120 con 0x7f461c0b0d30 2023-10-16T09:43:47.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.622+0000 7f4619d9b700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460406b0b0 0x7f460406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.622+0000 7f4619d9b700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460406b0b0 0x7f460406d570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f461c072c40 tx=0x7f4610006c10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7fa647f7e700 1 -- 172.21.15.158:0/2115839488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64006c780 msgr2=0x7fa640071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7fa647f7e700 1 --2- 172.21.15.158:0/2115839488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64006c780 0x7fa640071260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d370 tx=0x7fa63800d680 comp rx=0 tx=0).stop 2023-10-16T09:43:47.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7fa647f7e700 1 -- 172.21.15.158:0/2115839488 shutdown_connections 2023-10-16T09:43:47.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7fa647f7e700 1 --2- 172.21.15.158:0/2115839488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64006c780 0x7fa640071260 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7fa647f7e700 1 --2- 172.21.15.158:0/2115839488 >> v2:172.21.15.172:3300/0 conn(0x7fa64006e990 0x7fa64015def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7fa647f7e700 1 -- 172.21.15.158:0/2115839488 >> 172.21.15.158:0/2115839488 conn(0x7fa6400cf2b0 msgr2=0x7fa6400cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7f7bf8f11700 1 Processor -- start 2023-10-16T09:43:47.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.624+0000 7fa647f7e700 1 -- 172.21.15.158:0/2115839488 shutdown_connections 2023-10-16T09:43:47.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.625+0000 7f7bf8f11700 1 -- start start 2023-10-16T09:43:47.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7efcd87a0700 1 Processor -- start 2023-10-16T09:43:47.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf8f11700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7bf40c9900 0x7f7bf406e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf8f11700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf406e7c0 0x7f7bf4161780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf8f11700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf415ee10 con 0x7f7bf40cd830 2023-10-16T09:43:47.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf8f11700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf406ec50 con 0x7f7bf40c9900 2023-10-16T09:43:47.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf8f11700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4158320 con 0x7f7bf406e7c0 2023-10-16T09:43:47.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.625+0000 7fa647f7e700 1 -- 172.21.15.158:0/2115839488 wait complete. 2023-10-16T09:43:47.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7fa647f7e700 1 Processor -- start 2023-10-16T09:43:47.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f3a4affd700 1 -- 172.21.15.158:0/2371369621 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44011790 con 0x7f3a540665a0 2023-10-16T09:43:47.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f3a4affd700 1 -- 172.21.15.158:0/2371369621 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a440118f0 con 0x7f3a540665a0 2023-10-16T09:43:47.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f3a4affd700 1 -- 172.21.15.158:0/2371369621 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44011ca0 con 0x7f3a540665a0 2023-10-16T09:43:47.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7efcd87a0700 1 -- start start 2023-10-16T09:43:47.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf1d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7bf40c9900 0x7f7bf406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf1d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7bf40c9900 0x7f7bf406e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47822/0 (socket says 172.21.15.158:47822) 2023-10-16T09:43:47.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf1d9b700 1 -- 172.21.15.158:0/1013285685 learned_addr learned my addr 172.21.15.158:0/1013285685 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7f7bf2d9d700 1 --2- 172.21.15.158:0/1013285685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf406e7c0 0x7f7bf4161780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5415ad40 con 0x7f3a540665a0 2023-10-16T09:43:47.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a54156e00 con 0x7f3a540665a0 2023-10-16T09:43:47.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf2d9d700 1 -- 172.21.15.158:0/1013285685 >> v2:172.21.15.172:3300/0 conn(0x7f7bf40c9900 msgr2=0x7f7bf406e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf2d9d700 1 --2- 172.21.15.158:0/1013285685 >> v2:172.21.15.172:3300/0 conn(0x7f7bf40c9900 0x7f7bf406e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf2d9d700 1 -- 172.21.15.158:0/1013285685 >> v1:172.21.15.172:6789/0 conn(0x7f7bf40cd830 legacy=0x7f7bf40cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf2d9d700 1 -- 172.21.15.158:0/1013285685 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf406f220 con 0x7f7bf406e7c0 2023-10-16T09:43:47.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7fe48cff9700 1 -- 172.21.15.158:0/656630299 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4901519d0 con 0x7fe4900c3960 2023-10-16T09:43:47.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf2d9d700 1 --2- 172.21.15.158:0/1013285685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf406e7c0 0x7f7bf4161780 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009d90 tx=0x7f7bec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdd88956aee30eab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf159a700 1 -- 172.21.15.158:0/1013285685 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00e070 con 0x7f7bf406e7c0 2023-10-16T09:43:47.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf159a700 1 -- 172.21.15.158:0/1013285685 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec00bbd0 con 0x7f7bf406e7c0 2023-10-16T09:43:47.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f7bf159a700 1 -- 172.21.15.158:0/1013285685 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec012430 con 0x7f7bf406e7c0 2023-10-16T09:43:47.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.626+0000 7fa647f7e700 1 -- start start 2023-10-16T09:43:47.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa647f7e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa64006c780 0x7fa640156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa647f7e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa64006e990 0x7fa6400b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa647f7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64015cde0 0x7fa6400b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa647f7e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa640158930 con 0x7fa64006e990 2023-10-16T09:43:47.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa647f7e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa640158090 con 0x7fa64006c780 2023-10-16T09:43:47.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa647f7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa640158370 con 0x7fa64015cde0 2023-10-16T09:43:47.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645d1a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa64006c780 0x7fa640156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645d1a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa64006c780 0x7fa640156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47826/0 (socket says 172.21.15.158:47826) 2023-10-16T09:43:47.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645d1a700 1 -- 172.21.15.158:0/3718487340 learned_addr learned my addr 172.21.15.158:0/3718487340 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645519700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa64006e990 0x7fa6400b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645519700 1 -- 172.21.15.158:0/3718487340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64015cde0 msgr2=0x7fa6400b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645519700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64015cde0 0x7fa6400b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645519700 1 -- 172.21.15.158:0/3718487340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa64006c780 msgr2=0x7fa640156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645519700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa64006c780 0x7fa640156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7fa645519700 1 -- 172.21.15.158:0/3718487340 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63800d050 con 0x7fa64006e990 2023-10-16T09:43:47.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7fa645519700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa64006e990 0x7fa6400b3320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00b450 tx=0x7fa63c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7fa6367fc700 1 -- 172.21.15.158:0/3718487340 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c011830 con 0x7fa64006e990 2023-10-16T09:43:47.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7fa6367fc700 1 -- 172.21.15.158:0/3718487340 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c011990 con 0x7fa64006e990 2023-10-16T09:43:47.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7fa6367fc700 1 -- 172.21.15.158:0/3718487340 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c011d40 con 0x7fa64006e990 2023-10-16T09:43:47.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7efcd87a0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efcd00665a0 0x7efcd00ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7efcd87a0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efcd0066980 0x7efcd0071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7efcd87a0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd015d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7efcd87a0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd00725c0 con 0x7efcd00665a0 2023-10-16T09:43:47.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7efcd87a0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd00ce320 con 0x7efcd0066980 2023-10-16T09:43:47.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.628+0000 7efcd87a0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd00cd9f0 con 0x7efcd0072120 2023-10-16T09:43:47.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa640158650 con 0x7fa64006e990 2023-10-16T09:43:47.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa640156bb0 con 0x7fa64006e990 2023-10-16T09:43:47.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7efcd5d3b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efcd0066980 0x7efcd0071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7efcd5d3b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efcd0066980 0x7efcd0071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47830/0 (socket says 172.21.15.158:47830) 2023-10-16T09:43:47.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7efcd5d3b700 1 -- 172.21.15.158:0/3466328426 learned_addr learned my addr 172.21.15.158:0/3466328426 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.627+0000 7f460affd700 1 -- 172.21.15.158:0/332745536 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f460c056050 con 0x7f461c0b0d30 2023-10-16T09:43:47.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7efcd5d3b700 1 -- 172.21.15.158:0/3466328426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 msgr2=0x7efcd015d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7efcd5d3b700 1 --2- 172.21.15.158:0/3466328426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd015d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7efcd5d3b700 1 -- 172.21.15.158:0/3466328426 >> v2:172.21.15.158:3300/0 conn(0x7efcd00665a0 msgr2=0x7efcd00ce7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:47.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7efcd5d3b700 1 --2- 172.21.15.158:0/3466328426 >> v2:172.21.15.158:3300/0 conn(0x7efcd00665a0 0x7efcd00ce7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7efcd5d3b700 1 -- 172.21.15.158:0/3466328426 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd00cdcd0 con 0x7efcd0066980 2023-10-16T09:43:47.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.629+0000 7f3a4affd700 1 -- 172.21.15.158:0/2371369621 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3a4401c460 con 0x7f3a540665a0 2023-10-16T09:43:47.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7f3a4affd700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3a3806afa0 0x7f3a3806d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7f3a4affd700 1 -- 172.21.15.158:0/2371369621 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3a44089720 con 0x7f3a540665a0 2023-10-16T09:43:47.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5404ebb0 con 0x7f3a540665a0 2023-10-16T09:43:47.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7f7bf8f11700 1 -- 172.21.15.158:0/1013285685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf406e7c0 msgr2=0x7f7bf4161780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/1013285685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf406e7c0 0x7f7bf4161780 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009d90 tx=0x7f7bec00b330 comp rx=0 tx=0).stop 2023-10-16T09:43:47.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.631+0000 7f7bf8f11700 1 -- 172.21.15.158:0/1013285685 shutdown_connections 2023-10-16T09:43:47.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.631+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/1013285685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf406e7c0 0x7f7bf4161780 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.631+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/1013285685 >> v2:172.21.15.172:3300/0 conn(0x7f7bf40c9900 0x7f7bf406e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.631+0000 7f7bf8f11700 1 -- 172.21.15.158:0/1013285685 >> 172.21.15.158:0/1013285685 conn(0x7f7bf40cf2b0 msgr2=0x7f7bf40cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.631+0000 7f7bf8f11700 1 -- 172.21.15.158:0/1013285685 shutdown_connections 2023-10-16T09:43:47.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.631+0000 7f7bf8f11700 1 -- 172.21.15.158:0/1013285685 wait complete. 2023-10-16T09:43:47.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.633+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6400c4140 con 0x7fa64006e990 2023-10-16T09:43:47.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.633+0000 7f7bf8f11700 1 Processor -- start 2023-10-16T09:43:47.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.633+0000 7fa6367fc700 1 -- 172.21.15.158:0/3718487340 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa63c01a400 con 0x7fa64006e990 2023-10-16T09:43:47.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.633+0000 7fa6367fc700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa62406af50 0x7fa62406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.633+0000 7f7bf8f11700 1 -- start start 2023-10-16T09:43:47.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf8f11700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf406e7c0 0x7f7bf40ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf8f11700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf40c9900 0x7f7bf40c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf8f11700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf40cd830 0x7f7bf40a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf8f11700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40aba40 con 0x7f7bf40c9900 2023-10-16T09:43:47.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf8f11700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40c59f0 con 0x7f7bf406e7c0 2023-10-16T09:43:47.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf8f11700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40aa9a0 con 0x7f7bf40cd830 2023-10-16T09:43:47.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf40c9900 0x7f7bf40c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf40c9900 0x7f7bf40c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46660/0 (socket says 172.21.15.158:46660) 2023-10-16T09:43:47.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf1d9b700 1 -- 172.21.15.158:0/2578704158 learned_addr learned my addr 172.21.15.158:0/2578704158 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.634+0000 7f7bf259c700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf406e7c0 0x7f7bf40ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.635+0000 7f7bf1d9b700 1 -- 172.21.15.158:0/2578704158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf40cd830 msgr2=0x7f7bf40a85d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:47.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.635+0000 7f7bf1d9b700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf40cd830 0x7f7bf40a85d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.635+0000 7f7bf1d9b700 1 -- 172.21.15.158:0/2578704158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf406e7c0 msgr2=0x7f7bf40ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.635+0000 7f7bf1d9b700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf406e7c0 0x7f7bf40ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.635+0000 7f7bf1d9b700 1 -- 172.21.15.158:0/2578704158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec009a70 con 0x7f7bf40c9900 2023-10-16T09:43:47.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.635+0000 7f7bf1d9b700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf40c9900 0x7f7bf40c54b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e7e0 tx=0x7f7be800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.636+0000 7fa645d1a700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa62406af50 0x7fa62406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.636+0000 7fa645d1a700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa62406af50 0x7fa62406d410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa64015b2f0 tx=0x7fa630008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.637+0000 7fa6367fc700 1 -- 172.21.15.158:0/3718487340 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa63c058a50 con 0x7fa64006e990 2023-10-16T09:43:47.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.637+0000 7f3a4affd700 1 -- 172.21.15.158:0/2371369621 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a440544c0 con 0x7f3a540665a0 2023-10-16T09:43:47.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.639+0000 7f59536b1700 1 Processor -- start 2023-10-16T09:43:47.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.640+0000 7f59536b1700 1 -- start start 2023-10-16T09:43:47.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.630+0000 7efcd5d3b700 1 --2- 172.21.15.158:0/3466328426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd0071be0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efcd006cce0 tx=0x7efcc0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=defae31756be2081 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.639+0000 7efcd553a700 1 -- 172.21.15.158:0/3466328426 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc000b860 con 0x7efcd0066980 2023-10-16T09:43:47.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.639+0000 7efcd553a700 1 -- 172.21.15.158:0/3466328426 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc000b9c0 con 0x7efcd0066980 2023-10-16T09:43:47.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.639+0000 7efcd553a700 1 -- 172.21.15.158:0/3466328426 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc000bd50 con 0x7efcd0066980 2023-10-16T09:43:47.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.640+0000 7efcd87a0700 1 -- 172.21.15.158:0/3466328426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 msgr2=0x7efcd0071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.640+0000 7efcd87a0700 1 --2- 172.21.15.158:0/3466328426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd0071be0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efcd006cce0 tx=0x7efcc0009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:47.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f3a59b9b700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3a3806afa0 0x7f3a3806d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f59536b1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f594c15d950 0x7f594c15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f59536b1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c0ce320 con 0x7f594c0ce7b0 2023-10-16T09:43:47.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f59536b1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c15e270 con 0x7f594c0cec30 2023-10-16T09:43:47.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f59536b1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0725c0 con 0x7f594c15d950 2023-10-16T09:43:47.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f5951c4e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f594c0cec30 0x7f594c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:50748/0 (socket says 172.21.15.158:50748) 2023-10-16T09:43:47.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f5951c4e700 1 -- 172.21.15.158:0/1643465216 learned_addr learned my addr 172.21.15.158:0/1643465216 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:47.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f5950c4c700 1 --1- 172.21.15.158:0/1643465216 >> v1:172.21.15.158:6789/0 conn(0x7f594c0ce7b0 0x7f594c071be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46846/0 (socket says 172.21.15.158:46846) 2023-10-16T09:43:47.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881406689 0 0) 0x7f594c15e270 con 0x7f594c0cec30 2023-10-16T09:43:47.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.641+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59380046c0 con 0x7f594c0cec30 2023-10-16T09:43:47.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525305464 0 0) 0x7f59380046c0 con 0x7f594c0cec30 2023-10-16T09:43:47.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f594c15d950 msgr2=0x7f594c15dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T09:43:47.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f594c15d950 0x7f594c15dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 >> v1:172.21.15.158:6789/0 conn(0x7f594c0ce7b0 legacy=0x7f594c071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c0722f0 con 0x7f594c0cec30 2023-10-16T09:43:47.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f59440019f0 con 0x7f594c0cec30 2023-10-16T09:43:47.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 >> v1:172.21.15.172:6789/0 conn(0x7f594c0cec30 legacy=0x7f594c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5938004b10 0x7f5938004f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5938005960 0x7f5938005de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5938006440 0x7f593800cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593800d270 con 0x7f5938005960 2023-10-16T09:43:47.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593800d3e0 con 0x7f5938004b10 2023-10-16T09:43:47.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593800d5a0 con 0x7f5938006440 2023-10-16T09:43:47.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5950c4c700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5938005960 0x7f5938005de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5951c4e700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5938006440 0x7f593800cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5950c4c700 1 -- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5938006440 msgr2=0x7f593800cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:47.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5950c4c700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5938006440 0x7f593800cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5950c4c700 1 -- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5938004b10 msgr2=0x7f5938004f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:47.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5950c4c700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5938004b10 0x7f5938004f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.642+0000 7f5950c4c700 1 -- 172.21.15.158:0/1643465216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59380046c0 con 0x7f5938005960 2023-10-16T09:43:47.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7f5950c4c700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5938005960 0x7f5938005de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f59480017e0 tx=0x7f5938006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee5edf9773ece26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948004a70 con 0x7f5938005960 2023-10-16T09:43:47.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5948004bd0 con 0x7f5938005960 2023-10-16T09:43:47.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7f5943fff700 1 -- 172.21.15.158:0/1643465216 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59480070d0 con 0x7f5938005960 2023-10-16T09:43:47.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7fa6367fc700 1 -- 172.21.15.158:0/3718487340 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6400c4140 con 0x7fa64006e990 2023-10-16T09:43:47.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.644+0000 7f3a59b9b700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3a3806afa0 0x7f3a3806d460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3a5406f4e0 tx=0x7f3a50010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.644+0000 7f59536b1700 1 -- 172.21.15.158:0/1643465216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5938005960 msgr2=0x7f5938005de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.644+0000 7f59536b1700 1 --2- 172.21.15.158:0/1643465216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5938005960 0x7f5938005de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f59480017e0 tx=0x7f5938006990 comp rx=0 tx=0).stop 2023-10-16T09:43:47.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f5951c4e700 1 -- 172.21.15.158:0/1643465216 reap_dead start 2023-10-16T09:43:47.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f59536b1700 1 -- 172.21.15.158:0/1643465216 shutdown_connections 2023-10-16T09:43:47.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f59536b1700 1 -- 172.21.15.158:0/1643465216 >> 172.21.15.158:0/1643465216 conn(0x7f594c0cc700 msgr2=0x7f594c0ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7efcd87a0700 1 -- 172.21.15.158:0/3466328426 shutdown_connections 2023-10-16T09:43:47.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7efcd87a0700 1 --2- 172.21.15.158:0/3466328426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd015d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7efcd87a0700 1 --2- 172.21.15.158:0/3466328426 >> v2:172.21.15.158:3300/0 conn(0x7efcd00665a0 0x7efcd00ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7efcd87a0700 1 --2- 172.21.15.158:0/3466328426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd0071be0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.643+0000 7efcd87a0700 1 -- 172.21.15.158:0/3466328426 >> 172.21.15.158:0/3466328426 conn(0x7efcd00cc700 msgr2=0x7efcd00ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.644+0000 7efcd87a0700 1 -- 172.21.15.158:0/3466328426 shutdown_connections 2023-10-16T09:43:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.644+0000 7efcd87a0700 1 -- 172.21.15.158:0/3466328426 wait complete. 2023-10-16T09:43:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7efcd87a0700 1 Processor -- start 2023-10-16T09:43:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7efcd87a0700 1 -- start start 2023-10-16T09:43:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f7be2ffd700 1 -- 172.21.15.158:0/2578704158 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8013610 con 0x7f7bf40c9900 2023-10-16T09:43:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f7be2ffd700 1 -- 172.21.15.158:0/2578704158 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8013770 con 0x7f7bf40c9900 2023-10-16T09:43:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f7be2ffd700 1 -- 172.21.15.158:0/2578704158 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8013b00 con 0x7f7bf40c9900 2023-10-16T09:43:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd87a0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd00665a0 0x7efcd0089ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd87a0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd00c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd87a0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd87a0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd00b51f0 con 0x7efcd00665a0 2023-10-16T09:43:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd87a0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd008a6f0 con 0x7efcd0066980 2023-10-16T09:43:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd87a0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd00c8960 con 0x7efcd0072120 2023-10-16T09:43:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd5d3b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd00c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd6d3d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd5d3b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd00c3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47858/0 (socket says 172.21.15.158:47858) 2023-10-16T09:43:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd5d3b700 1 -- 172.21.15.158:0/1921388902 learned_addr learned my addr 172.21.15.158:0/1921388902 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd6d3d700 1 -- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 msgr2=0x7efcd00c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd6d3d700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd00c3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd6d3d700 1 -- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd00665a0 msgr2=0x7efcd0089ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd6d3d700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd00665a0 0x7efcd0089ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd6d3d700 1 -- 172.21.15.158:0/1921388902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0009550 con 0x7efcd0072120 2023-10-16T09:43:47.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.647+0000 7efcd5d3b700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd00c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:47.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.648+0000 7efcd6d3d700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd00c5ca0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efcd0073a60 tx=0x7efccc00b6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.648+0000 7efcc6ffd700 1 -- 172.21.15.158:0/1921388902 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc011830 con 0x7efcd0072120 2023-10-16T09:43:47.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.648+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd00c8c40 con 0x7efcd0072120 2023-10-16T09:43:47.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.648+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd008a400 con 0x7efcd0072120 2023-10-16T09:43:47.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf40aac80 con 0x7f7bf40c9900 2023-10-16T09:43:47.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.646+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf40ab240 con 0x7f7bf40c9900 2023-10-16T09:43:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.649+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd00681b0 con 0x7efcd0072120 2023-10-16T09:43:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.650+0000 7f59536b1700 1 -- 172.21.15.158:0/1643465216 shutdown_connections 2023-10-16T09:43:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.650+0000 7f7be2ffd700 1 -- 172.21.15.158:0/2578704158 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7be8013c60 con 0x7f7bf40c9900 2023-10-16T09:43:47.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.650+0000 7efcc6ffd700 1 -- 172.21.15.158:0/1921388902 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc011990 con 0x7efcd0072120 2023-10-16T09:43:47.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.650+0000 7efcc6ffd700 1 -- 172.21.15.158:0/1921388902 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc011d40 con 0x7efcd0072120 2023-10-16T09:43:47.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.650+0000 7f7be2ffd700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd406af50 0x7f7bd406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.651+0000 7f7be2ffd700 1 -- 172.21.15.158:0/2578704158 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7be8089330 con 0x7f7bf40c9900 2023-10-16T09:43:47.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.651+0000 7f7bf259c700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd406af50 0x7f7bd406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.651+0000 7f7bf259c700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd406af50 0x7f7bd406d410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7be4001800 tx=0x7f7be4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.651+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd80052f0 con 0x7f7bf40c9900 2023-10-16T09:43:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.652+0000 7f59536b1700 1 -- 172.21.15.158:0/1643465216 wait complete. 2023-10-16T09:43:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.653+0000 7f59536b1700 1 Processor -- start 2023-10-16T09:43:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.653+0000 7f59536b1700 1 -- start start 2023-10-16T09:43:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f59536b1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f594c0c5920 0x7f594c0c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f59536b1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f594c0c7a80 0x7f594c14dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f59536b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f594c0c5d00 0x7f594c14e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f59536b1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0c6530 con 0x7f594c0c5d00 2023-10-16T09:43:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f59536b1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c14cbd0 con 0x7f594c0c5920 2023-10-16T09:43:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f59536b1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c14ceb0 con 0x7f594c0c7a80 2023-10-16T09:43:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f595144d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f594c0c5920 0x7f594c0c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f595144d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f594c0c5920 0x7f594c0c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47866/0 (socket says 172.21.15.158:47866) 2023-10-16T09:43:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f595144d700 1 -- 172.21.15.158:0/1686191084 learned_addr learned my addr 172.21.15.158:0/1686191084 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:47.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.654+0000 7f5951c4e700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f594c0c5d00 0x7f594c14e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.652+0000 7efcc6ffd700 1 -- 172.21.15.158:0/1921388902 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efccc01a400 con 0x7efcd0072120 2023-10-16T09:43:47.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.653+0000 7efcc6ffd700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efcb406af00 0x7efcb406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.653+0000 7efcc6ffd700 1 -- 172.21.15.158:0/1921388902 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efccc089d20 con 0x7efcd0072120 2023-10-16T09:43:47.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.655+0000 7f5951c4e700 1 -- 172.21.15.158:0/1686191084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f594c0c7a80 msgr2=0x7f594c14dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:47.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.655+0000 7f5951c4e700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f594c0c7a80 0x7f594c14dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.655+0000 7f5951c4e700 1 -- 172.21.15.158:0/1686191084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f594c0c5920 msgr2=0x7f594c0c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.655+0000 7f5951c4e700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f594c0c5920 0x7f594c0c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.655+0000 7f5951c4e700 1 -- 172.21.15.158:0/1686191084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c14d190 con 0x7f594c0c5d00 2023-10-16T09:43:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.655+0000 7f5951c4e700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f594c0c5d00 0x7f594c14e1b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f594c0cf010 tx=0x7f594400f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.655+0000 7efcd653c700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efcb406af00 0x7efcb406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.656+0000 7efcc6ffd700 1 -- 172.21.15.158:0/1921388902 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efccc054b70 con 0x7efcd0072120 2023-10-16T09:43:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.664+0000 7f7be2ffd700 1 -- 172.21.15.158:0/2578704158 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7be8054180 con 0x7f7bf40c9900 2023-10-16T09:43:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.666+0000 7efcd653c700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efcb406af00 0x7efcb406d3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efcc8001ff0 tx=0x7efcc80015b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.673+0000 7f5941ffb700 1 -- 172.21.15.158:0/1686191084 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594400c050 con 0x7f594c0c5d00 2023-10-16T09:43:47.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.673+0000 7f5941ffb700 1 -- 172.21.15.158:0/1686191084 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5944001250 con 0x7f594c0c5d00 2023-10-16T09:43:47.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.673+0000 7f5941ffb700 1 -- 172.21.15.158:0/1686191084 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59440075c0 con 0x7f594c0c5d00 2023-10-16T09:43:47.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.673+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c14d470 con 0x7f594c0c5d00 2023-10-16T09:43:47.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.673+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c14bc40 con 0x7f594c0c5d00 2023-10-16T09:43:47.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.678+0000 7f5941ffb700 1 -- 172.21.15.158:0/1686191084 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5944007720 con 0x7f594c0c5d00 2023-10-16T09:43:47.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.678+0000 7f5941ffb700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f593006afd0 0x7f593006d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:47.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.678+0000 7f5941ffb700 1 -- 172.21.15.158:0/1686191084 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f594408a590 con 0x7f594c0c5d00 2023-10-16T09:43:47.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.679+0000 7f595144d700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f593006afd0 0x7f593006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:47.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.679+0000 7f595144d700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f593006afd0 0x7f593006d490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f593c009840 tx=0x7f593c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:47.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.683+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c0052f0 con 0x7f594c0c5d00 2023-10-16T09:43:47.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:47.693+0000 7f5941ffb700 1 -- 172.21.15.158:0/1686191084 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5944015080 con 0x7f594c0c5d00 2023-10-16T09:43:48.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.027+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3ad00c6ab0 con 0x7f3ad015def0 2023-10-16T09:43:48.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.032+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe4901519d0 con 0x7fe4900c3960 2023-10-16T09:43:48.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.033+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f461c0c9750 con 0x7f461c0b0d30 2023-10-16T09:43:48.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.032+0000 7fe48cff9700 1 -- 172.21.15.158:0/656630299 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4901519d0 con 0x7fe4900c3960 2023-10-16T09:43:48.035 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:48.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.033+0000 7f460affd700 1 -- 172.21.15.158:0/332745536 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f460c059670 con 0x7f461c0b0d30 2023-10-16T09:43:48.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.033+0000 7f3acd7fa700 1 -- 172.21.15.158:0/2885692203 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ad00c6ab0 con 0x7f3ad015def0 2023-10-16T09:43:48.038 INFO:teuthology.orchestra.run.smithi158.stdout:55834574856 2023-10-16T09:43:48.039 INFO:teuthology.orchestra.run.smithi158.stdout:55834574856 2023-10-16T09:43:48.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 msgr2=0x7f3abc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0007fd0 tx=0x7f3ac0009040 comp rx=0 tx=0).stop 2023-10-16T09:43:48.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad015def0 msgr2=0x7f3ad0157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad015def0 0x7f3ad0157580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b970 tx=0x7f3ac800bc80 comp rx=0 tx=0).stop 2023-10-16T09:43:48.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 shutdown_connections 2023-10-16T09:43:48.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad015def0 0x7f3ad0157580 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e990 0x7f3ad00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 --2- 172.21.15.158:0/2885692203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad00cfdd0 0x7f3ad01fa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 >> 172.21.15.158:0/2885692203 conn(0x7f3ad00cf2b0 msgr2=0x7f3ad01599f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe47c06af50 msgr2=0x7fe47c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7fe496378700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe47c06af50 0x7fe47c06d410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe484006fd0 tx=0x7fe484008040 comp rx=0 tx=0).stop 2023-10-16T09:43:48.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4900c3960 msgr2=0x7fe4901569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.037+0000 7fe496378700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4900c3960 0x7fe4901569a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe488012000 tx=0x7fe488004ad0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460406b0b0 msgr2=0x7f460406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7f4620ee6700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460406b0b0 0x7f460406d570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f461c072c40 tx=0x7f4610006c10 comp rx=0 tx=0).stop 2023-10-16T09:43:48.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c0b0d30 msgr2=0x7f461c0b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7f4620ee6700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c0b0d30 0x7f461c0b41a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f461c0cd160 tx=0x7f460c004750 comp rx=0 tx=0).stop 2023-10-16T09:43:48.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 shutdown_connections 2023-10-16T09:43:48.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7fe496378700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4900c3960 0x7fe4901569a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7fe496378700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49006afe0 0x7fe490150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7fe496378700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe47c06af50 0x7fe47c06d410 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7fe496378700 1 --2- 172.21.15.158:0/656630299 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4900af890 0x7fe490151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 >> 172.21.15.158:0/656630299 conn(0x7fe49006b9e0 msgr2=0x7fe49006a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 shutdown_connections 2023-10-16T09:43:48.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.038+0000 7f3ad6b09700 1 -- 172.21.15.158:0/2885692203 wait complete. 2023-10-16T09:43:48.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.039+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 shutdown_connections 2023-10-16T09:43:48.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.039+0000 7fe496378700 1 -- 172.21.15.158:0/656630299 wait complete. 2023-10-16T09:43:48.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 shutdown_connections 2023-10-16T09:43:48.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f461c161e30 0x7f461c160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c0b0d30 0x7f461c0b41a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460406b0b0 0x7f460406d570 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 --2- 172.21.15.158:0/332745536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c0c9b90 0x7f461c164a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 >> 172.21.15.158:0/332745536 conn(0x7f461c0cbd30 msgr2=0x7f461c14ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 shutdown_connections 2023-10-16T09:43:48.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.040+0000 7f4620ee6700 1 -- 172.21.15.158:0/332745536 wait complete. 2023-10-16T09:43:48.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.044+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa6400c4140 con 0x7fa64006e990 2023-10-16T09:43:48.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.045+0000 7fa6367fc700 1 -- 172.21.15.158:0/3718487340 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6400c4140 con 0x7fa64006e990 2023-10-16T09:43:48.049 INFO:teuthology.orchestra.run.smithi158.stdout:55834574856 2023-10-16T09:43:48.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa62406af50 msgr2=0x7fa62406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa62406af50 0x7fa62406d410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa64015b2f0 tx=0x7fa630008040 comp rx=0 tx=0).stop 2023-10-16T09:43:48.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa64006e990 msgr2=0x7fa6400b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa64006e990 0x7fa6400b3320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00b450 tx=0x7fa63c00b810 comp rx=0 tx=0).stop 2023-10-16T09:43:48.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 shutdown_connections 2023-10-16T09:43:48.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa62406af50 0x7fa62406d410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa64015cde0 0x7fa6400b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa64006c780 0x7fa640156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 --2- 172.21.15.158:0/3718487340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa64006e990 0x7fa6400b3320 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.049+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 >> 172.21.15.158:0/3718487340 conn(0x7fa6400cf2b0 msgr2=0x7fa64015c080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.050+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3a5415d300 con 0x7f3a540665a0 2023-10-16T09:43:48.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.050+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 shutdown_connections 2023-10-16T09:43:48.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.050+0000 7fa647f7e700 1 -- 172.21.15.158:0/3718487340 wait complete. 2023-10-16T09:43:48.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.050+0000 7f3a4affd700 1 -- 172.21.15.158:0/2371369621 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a44057ae0 con 0x7f3a540665a0 2023-10-16T09:43:48.054 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:48.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.055+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3a3806afa0 msgr2=0x7f3a3806d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.055+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3a3806afa0 0x7f3a3806d460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3a5406f4e0 tx=0x7f3a50010000 comp rx=0 tx=0).stop 2023-10-16T09:43:48.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.055+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a540665a0 msgr2=0x7f3a54152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.055+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a540665a0 0x7f3a54152ad0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c4d0 tx=0x7f3a4400c890 comp rx=0 tx=0).stop 2023-10-16T09:43:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.059+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 shutdown_connections 2023-10-16T09:43:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.059+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a5406c000 0x7f3a541583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.059+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a540665a0 0x7f3a54152ad0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.059+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3a3806afa0 0x7f3a3806d460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.059+0000 7f3a5b39e700 1 --2- 172.21.15.158:0/2371369621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a54066980 0x7f3a54155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.059+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 >> 172.21.15.158:0/2371369621 conn(0x7f3a540cff90 msgr2=0x7f3a540ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2023-10-16T09:43:48.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.062+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 shutdown_connections 2023-10-16T09:43:48.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.062+0000 7f3a5b39e700 1 -- 172.21.15.158:0/2371369621 wait complete. 2023-10-16T09:43:48.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2023-10-16T09:43:48.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.066+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7efcd00c8aa0 con 0x7efcd0072120 2023-10-16T09:43:48.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.067+0000 7efcc6ffd700 1 -- 172.21.15.158:0/1921388902 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efcd00c8aa0 con 0x7efcd0072120 2023-10-16T09:43:48.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2023-10-16T09:43:48.069 INFO:teuthology.orchestra.run.smithi158.stdout:55834574856 2023-10-16T09:43:48.070 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2023-10-16T09:43:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.070+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efcb406af00 msgr2=0x7efcb406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.070+0000 7efcd87a0700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efcb406af00 0x7efcb406d3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efcc8001ff0 tx=0x7efcc80015b0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.070+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 msgr2=0x7efcd00c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.070+0000 7efcd87a0700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd00c5ca0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efcd0073a60 tx=0x7efccc00b6a0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 shutdown_connections 2023-10-16T09:43:48.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efcb406af00 0x7efcb406d3c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd0072120 0x7efcd00c5ca0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd00665a0 0x7efcd0089ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 --2- 172.21.15.158:0/1921388902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd0066980 0x7efcd00c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 >> 172.21.15.158:0/1921388902 conn(0x7efcd00cc700 msgr2=0x7efcd0169a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 shutdown_connections 2023-10-16T09:43:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.071+0000 7efcd87a0700 1 -- 172.21.15.158:0/1921388902 wait complete. 2023-10-16T09:43:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.072+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7bd8005870 con 0x7f7bf40c9900 2023-10-16T09:43:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.072+0000 7f7be2ffd700 1 -- 172.21.15.158:0/2578704158 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7be8021090 con 0x7f7bf40c9900 2023-10-16T09:43:48.076 INFO:teuthology.orchestra.run.smithi158.stdout:55834574856 2023-10-16T09:43:48.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.075+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd406af50 msgr2=0x7f7bd406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.075+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd406af50 0x7f7bd406d410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7be4001800 tx=0x7f7be4009040 comp rx=0 tx=0).stop 2023-10-16T09:43:48.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.076+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf40c9900 msgr2=0x7f7bf40c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.076+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf40c9900 0x7f7bf40c54b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e7e0 tx=0x7f7be800eaf0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 shutdown_connections 2023-10-16T09:43:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd406af50 0x7f7bd406d410 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf40cd830 0x7f7bf40a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf406e7c0 0x7f7bf40ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 --2- 172.21.15.158:0/2578704158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf40c9900 0x7f7bf40c54b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 >> 172.21.15.158:0/2578704158 conn(0x7f7bf40cf2b0 msgr2=0x7f7bf4072030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 shutdown_connections 2023-10-16T09:43:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.077+0000 7f7bf8f11700 1 -- 172.21.15.158:0/2578704158 wait complete. 2023-10-16T09:43:48.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.0 2023-10-16T09:43:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.082+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f592c0055a0 con 0x7f594c0c5d00 2023-10-16T09:43:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.083+0000 7f5941ffb700 1 -- 172.21.15.158:0/1686191084 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5944058e90 con 0x7f594c0c5d00 2023-10-16T09:43:48.084 INFO:teuthology.orchestra.run.smithi158.stdout:55834574856 2023-10-16T09:43:48.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f593006afd0 msgr2=0x7f593006d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f593006afd0 0x7f593006d490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f593c009840 tx=0x7f593c008040 comp rx=0 tx=0).stop 2023-10-16T09:43:48.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f594c0c5d00 msgr2=0x7f594c14e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:48.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f594c0c5d00 0x7f594c14e1b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f594c0cf010 tx=0x7f594400f920 comp rx=0 tx=0).stop 2023-10-16T09:43:48.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 shutdown_connections 2023-10-16T09:43:48.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f594c0c5d00 0x7f594c14e1b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f594c0c5920 0x7f594c0c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f593006afd0 0x7f593006d490 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 --2- 172.21.15.158:0/1686191084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f594c0c7a80 0x7f594c14dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 >> 172.21.15.158:0/1686191084 conn(0x7f594c0cc700 msgr2=0x7f594c0af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 shutdown_connections 2023-10-16T09:43:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:48.085+0000 7f59536b1700 1 -- 172.21.15.158:0/1686191084 wait complete. 2023-10-16T09:43:48.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2023-10-16T09:43:48.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2023-10-16T09:43:48.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.2 2023-10-16T09:43:49.063 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T09:43:49.065 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T09:43:49.070 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T09:43:49.071 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T09:43:49.082 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T09:43:49.093 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T09:43:49.098 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T09:43:49.102 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T09:43:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.260+0000 7fa995745700 1 Processor -- start 2023-10-16T09:43:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.260+0000 7fa995745700 1 -- start start 2023-10-16T09:43:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.261+0000 7fa995745700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9900ce9a0 0x7fa99006e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.261+0000 7fa995745700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa99006e7c0 0x7fa9901657e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.261+0000 7fa995745700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9900cce80 con 0x7fa990165d20 2023-10-16T09:43:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.261+0000 7fa995745700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9901667f0 con 0x7fa99006e7c0 2023-10-16T09:43:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.261+0000 7fa995745700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99006ec70 con 0x7fa9900ce9a0 2023-10-16T09:43:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.262+0000 7fa98f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa990165d20 0x7fa9900c9900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46868/0 (socket says 172.21.15.158:46868) 2023-10-16T09:43:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.262+0000 7fa98f7fe700 1 -- 172.21.15.158:0/4133389469 learned_addr learned my addr 172.21.15.158:0/4133389469 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.263+0000 7fa98effd700 1 --2- 172.21.15.158:0/4133389469 >> v2:172.21.15.172:3300/0 conn(0x7fa9900ce9a0 0x7fa99006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.263+0000 7fa98dffb700 1 -- 172.21.15.158:0/4133389469 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908805311 0 0) 0x7fa9900cce80 con 0x7fa990165d20 2023-10-16T09:43:49.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.263+0000 7fa98dffb700 1 -- 172.21.15.158:0/4133389469 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9780035b0 con 0x7fa990165d20 2023-10-16T09:43:49.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.263+0000 7fa98effd700 1 -- 172.21.15.158:0/4133389469 >> v2:172.21.15.158:3300/0 conn(0x7fa99006e7c0 msgr2=0x7fa9901657e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:49.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.263+0000 7fa98effd700 1 --2- 172.21.15.158:0/4133389469 >> v2:172.21.15.158:3300/0 conn(0x7fa99006e7c0 0x7fa9901657e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.263+0000 7fa98effd700 1 -- 172.21.15.158:0/4133389469 >> v1:172.21.15.158:6789/0 conn(0x7fa990165d20 legacy=0x7fa9900c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.263+0000 7fa98effd700 1 -- 172.21.15.158:0/4133389469 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa990162120 con 0x7fa9900ce9a0 2023-10-16T09:43:49.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.264+0000 7f61d98ea700 1 Processor -- start 2023-10-16T09:43:49.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.264+0000 7f61d98ea700 1 -- start start 2023-10-16T09:43:49.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.265+0000 7f61d98ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 0x7f61d415def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.265+0000 7f61d98ea700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61d406c780 0x7f61d4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.265+0000 7f61d98ea700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d40cd6c0 con 0x7f61d415cde0 2023-10-16T09:43:49.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.265+0000 7f61d98ea700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d4070b30 con 0x7f61d406c780 2023-10-16T09:43:49.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.265+0000 7f61d98ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d406d750 con 0x7f61d406e990 2023-10-16T09:43:49.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.266+0000 7f61d2ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f61d415cde0 0x7f61d406e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46872/0 (socket says 172.21.15.158:46872) 2023-10-16T09:43:49.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.266+0000 7f61d2ffd700 1 -- 172.21.15.158:0/3834049251 learned_addr learned my addr 172.21.15.158:0/3834049251 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:49.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7f61d37fe700 1 --2- 172.21.15.158:0/3834049251 >> v2:172.21.15.158:3300/0 conn(0x7f61d406c780 0x7f61d4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7f61d37fe700 1 -- 172.21.15.158:0/3834049251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 msgr2=0x7f61d415def0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:49.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7f61d37fe700 1 --2- 172.21.15.158:0/3834049251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 0x7f61d415def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7f61d37fe700 1 -- 172.21.15.158:0/3834049251 >> v1:172.21.15.158:6789/0 conn(0x7f61d415cde0 legacy=0x7f61d406e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7f61d37fe700 1 -- 172.21.15.158:0/3834049251 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d406cc30 con 0x7f61d406c780 2023-10-16T09:43:49.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.268+0000 7f61d37fe700 1 --2- 172.21.15.158:0/3834049251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d4071260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f61c4010ba0 tx=0x7f61c4010eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a9f02743045d4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.268+0000 7f61d1ffb700 1 -- 172.21.15.158:0/3834049251 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c400bb90 con 0x7f61d406c780 2023-10-16T09:43:49.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.268+0000 7f61d1ffb700 1 -- 172.21.15.158:0/3834049251 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c400bcf0 con 0x7f61d406c780 2023-10-16T09:43:49.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.268+0000 7f61d1ffb700 1 -- 172.21.15.158:0/3834049251 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4004270 con 0x7f61d406c780 2023-10-16T09:43:49.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.269+0000 7f61d98ea700 1 -- 172.21.15.158:0/3834049251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 msgr2=0x7f61d4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.269+0000 7f61d98ea700 1 --2- 172.21.15.158:0/3834049251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d4071260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f61c4010ba0 tx=0x7f61c4010eb0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.264+0000 7fa98effd700 1 --2- 172.21.15.158:0/4133389469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9900ce9a0 0x7fa99006e280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa984009840 tx=0x7fa984009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7088952e6bd73dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.266+0000 7fa98dffb700 1 -- 172.21.15.158:0/4133389469 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984011830 con 0x7fa9900ce9a0 2023-10-16T09:43:49.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.266+0000 7fa98dffb700 1 -- 172.21.15.158:0/4133389469 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa984011990 con 0x7fa9900ce9a0 2023-10-16T09:43:49.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.266+0000 7fa98dffb700 1 -- 172.21.15.158:0/4133389469 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984011d20 con 0x7fa9900ce9a0 2023-10-16T09:43:49.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7fa995745700 1 -- 172.21.15.158:0/4133389469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9900ce9a0 msgr2=0x7fa99006e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7fa995745700 1 --2- 172.21.15.158:0/4133389469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9900ce9a0 0x7fa99006e280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa984009840 tx=0x7fa984009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:49.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7fa995745700 1 -- 172.21.15.158:0/4133389469 shutdown_connections 2023-10-16T09:43:49.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7fa995745700 1 --2- 172.21.15.158:0/4133389469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9900ce9a0 0x7fa99006e280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7fa995745700 1 --2- 172.21.15.158:0/4133389469 >> v2:172.21.15.158:3300/0 conn(0x7fa99006e7c0 0x7fa9901657e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.267+0000 7fa995745700 1 -- 172.21.15.158:0/4133389469 >> 172.21.15.158:0/4133389469 conn(0x7fa9900cf2b0 msgr2=0x7fa9900cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.270+0000 7f61d98ea700 1 -- 172.21.15.158:0/3834049251 shutdown_connections 2023-10-16T09:43:49.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.270+0000 7f61d98ea700 1 --2- 172.21.15.158:0/3834049251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d4071260 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.270+0000 7f61d98ea700 1 --2- 172.21.15.158:0/3834049251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 0x7f61d415def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.270+0000 7f61d98ea700 1 -- 172.21.15.158:0/3834049251 >> 172.21.15.158:0/3834049251 conn(0x7f61d40cf2b0 msgr2=0x7f61d40cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.270+0000 7fa995745700 1 -- 172.21.15.158:0/4133389469 shutdown_connections 2023-10-16T09:43:49.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.270+0000 7fa995745700 1 -- 172.21.15.158:0/4133389469 wait complete. 2023-10-16T09:43:49.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.272+0000 7fa995745700 1 Processor -- start 2023-10-16T09:43:49.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.272+0000 7fa995745700 1 -- start start 2023-10-16T09:43:49.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.273+0000 7fa995745700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa99006e7c0 0x7fa99017a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.273+0000 7fa995745700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9900ce9a0 0x7fa990152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.273+0000 7fa995745700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa990165d20 0x7fa9900c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.273+0000 7fa995745700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900c5050 con 0x7fa9900ce9a0 2023-10-16T09:43:49.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.273+0000 7fa995745700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900c5330 con 0x7fa990165d20 2023-10-16T09:43:49.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.273+0000 7fa995745700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900c6f30 con 0x7fa99006e7c0 2023-10-16T09:43:49.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9900ce9a0 0x7fa990152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9900ce9a0 0x7fa990152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46704/0 (socket says 172.21.15.158:46704) 2023-10-16T09:43:49.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 -- 172.21.15.158:0/2559940208 learned_addr learned my addr 172.21.15.158:0/2559940208 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7f61d98ea700 1 -- 172.21.15.158:0/3834049251 shutdown_connections 2023-10-16T09:43:49.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 -- 172.21.15.158:0/2559940208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa99006e7c0 msgr2=0x7fa99017a550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:49.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa99006e7c0 0x7fa99017a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 -- 172.21.15.158:0/2559940208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa990165d20 msgr2=0x7fa9900c7730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:49.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa990165d20 0x7fa9900c7730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7fa98e7fc700 1 -- 172.21.15.158:0/2559940208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840094a0 con 0x7fa9900ce9a0 2023-10-16T09:43:49.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.274+0000 7f61d98ea700 1 -- 172.21.15.158:0/3834049251 wait complete. 2023-10-16T09:43:49.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.276+0000 7f61d98ea700 1 Processor -- start 2023-10-16T09:43:49.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.277+0000 7fa98e7fc700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9900ce9a0 0x7fa990152590 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa98000b550 tx=0x7fa98000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.277+0000 7f61d98ea700 1 -- start start 2023-10-16T09:43:49.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.278+0000 7f61d98ea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d41f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.278+0000 7f61d98ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 0x7f61d41560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.278+0000 7f61d98ea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61d415cde0 0x7f61d40c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.278+0000 7f61d98ea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d40bed30 con 0x7f61d406c780 2023-10-16T09:43:49.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.278+0000 7f61d98ea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d40c3f60 con 0x7f61d415cde0 2023-10-16T09:43:49.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.278+0000 7f61d98ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d41f84f0 con 0x7f61d406e990 2023-10-16T09:43:49.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d41f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d41f7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46706/0 (socket says 172.21.15.158:46706) 2023-10-16T09:43:49.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 -- 172.21.15.158:0/1221141334 learned_addr learned my addr 172.21.15.158:0/1221141334 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 -- 172.21.15.158:0/1221141334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 msgr2=0x7f61d41560b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:49.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 0x7f61d41560b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 -- 172.21.15.158:0/1221141334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61d415cde0 msgr2=0x7f61d40c8390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:49.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61d415cde0 0x7f61d40c8390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.279+0000 7f61d2ffd700 1 -- 172.21.15.158:0/1221141334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4010800 con 0x7f61d406c780 2023-10-16T09:43:49.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.280+0000 7f61d2ffd700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d41f7fb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f61cc002080 tx=0x7f61cc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.280+0000 7f61bbfff700 1 -- 172.21.15.158:0/1221141334 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc00ae00 con 0x7f61d406c780 2023-10-16T09:43:49.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.280+0000 7f61bbfff700 1 -- 172.21.15.158:0/1221141334 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc0070d0 con 0x7f61d406c780 2023-10-16T09:43:49.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.280+0000 7f61bbfff700 1 -- 172.21.15.158:0/1221141334 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc007460 con 0x7f61d406c780 2023-10-16T09:43:49.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.280+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d40c6030 con 0x7f61d406c780 2023-10-16T09:43:49.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.280+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d40be390 con 0x7f61d406c780 2023-10-16T09:43:49.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.283+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b40052f0 con 0x7f61d406c780 2023-10-16T09:43:49.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.285+0000 7f61bbfff700 1 -- 172.21.15.158:0/1221141334 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f61cc0075c0 con 0x7f61d406c780 2023-10-16T09:43:49.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.286+0000 7f61bbfff700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61c006af50 0x7f61c006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.286+0000 7f61bbfff700 1 -- 172.21.15.158:0/1221141334 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f61cc089d80 con 0x7f61d406c780 2023-10-16T09:43:49.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.287+0000 7f61d27fc700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61c006af50 0x7f61c006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.284+0000 7fa9777fe700 1 -- 172.21.15.158:0/2559940208 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980011830 con 0x7fa9900ce9a0 2023-10-16T09:43:49.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.284+0000 7fa9777fe700 1 -- 172.21.15.158:0/2559940208 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa980011990 con 0x7fa9900ce9a0 2023-10-16T09:43:49.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.284+0000 7fa9777fe700 1 -- 172.21.15.158:0/2559940208 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980011d40 con 0x7fa9900ce9a0 2023-10-16T09:43:49.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.284+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9900c7210 con 0x7fa9900ce9a0 2023-10-16T09:43:49.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.284+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa990152ad0 con 0x7fa9900ce9a0 2023-10-16T09:43:49.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.287+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9700052f0 con 0x7fa9900ce9a0 2023-10-16T09:43:49.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.296+0000 7f61d27fc700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61c006af50 0x7f61c006d410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61d41faad0 tx=0x7f61c8006bd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.297+0000 7f61bbfff700 1 -- 172.21.15.158:0/1221141334 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61cc054c30 con 0x7f61d406c780 2023-10-16T09:43:49.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.296+0000 7fa9777fe700 1 -- 172.21.15.158:0/2559940208 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa98001a400 con 0x7fa9900ce9a0 2023-10-16T09:43:49.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.296+0000 7fa9777fe700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa97c06afd0 0x7fa97c06d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.296+0000 7fa9777fe700 1 -- 172.21.15.158:0/2559940208 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa98008a0a0 con 0x7fa9900ce9a0 2023-10-16T09:43:49.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.297+0000 7fa98effd700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa97c06afd0 0x7fa97c06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.303+0000 7fa98effd700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa97c06afd0 0x7fa97c06d490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa9840090b0 tx=0x7fa984010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.303+0000 7fa9777fe700 1 -- 172.21.15.158:0/2559940208 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa980054db0 con 0x7fa9900ce9a0 2023-10-16T09:43:49.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.328+0000 7f3ad5729700 1 Processor -- start 2023-10-16T09:43:49.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.328+0000 7f3ad5729700 1 -- start start 2023-10-16T09:43:49.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.329+0000 7f3ad5729700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad00ce9a0 0x7f3ad006e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.329+0000 7f3ad5729700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3ad006e7c0 0x7f3ad01657e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.329+0000 7f3ad5729700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad00cce80 con 0x7f3ad0165d20 2023-10-16T09:43:49.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.329+0000 7f3ad5729700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad01667f0 con 0x7f3ad006e7c0 2023-10-16T09:43:49.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.329+0000 7f3ad5729700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad0070870 con 0x7f3ad00ce9a0 2023-10-16T09:43:49.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3acf7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3ad0165d20 0x7f3ad00c9900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46890/0 (socket says 172.21.15.158:46890) 2023-10-16T09:43:49.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3acf7fe700 1 -- 172.21.15.158:0/2378165458 learned_addr learned my addr 172.21.15.158:0/2378165458 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:49.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3ace7fc700 1 --2- 172.21.15.158:0/2378165458 >> v2:172.21.15.158:3300/0 conn(0x7f3ad006e7c0 0x7f3ad01657e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3ace7fc700 1 -- 172.21.15.158:0/2378165458 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad00ce9a0 msgr2=0x7f3ad006e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:49.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3ace7fc700 1 --2- 172.21.15.158:0/2378165458 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad00ce9a0 0x7f3ad006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3ace7fc700 1 -- 172.21.15.158:0/2378165458 >> v1:172.21.15.158:6789/0 conn(0x7f3ad0165d20 legacy=0x7f3ad00c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3ace7fc700 1 -- 172.21.15.158:0/2378165458 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad0162120 con 0x7f3ad006e7c0 2023-10-16T09:43:49.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7f3ace7fc700 1 --2- 172.21.15.158:0/2378165458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e7c0 0x7f3ad01657e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009840 tx=0x7f3ac4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef2a3e93c055666 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7f3acdffb700 1 -- 172.21.15.158:0/2378165458 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4011830 con 0x7f3ad006e7c0 2023-10-16T09:43:49.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7f3acdffb700 1 -- 172.21.15.158:0/2378165458 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac4011990 con 0x7f3ad006e7c0 2023-10-16T09:43:49.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7f3acdffb700 1 -- 172.21.15.158:0/2378165458 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4011d00 con 0x7f3ad006e7c0 2023-10-16T09:43:49.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f3ad5729700 1 -- 172.21.15.158:0/2378165458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e7c0 msgr2=0x7f3ad01657e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f3ad5729700 1 --2- 172.21.15.158:0/2378165458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e7c0 0x7f3ad01657e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009840 tx=0x7f3ac4009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:49.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f3ad5729700 1 -- 172.21.15.158:0/2378165458 shutdown_connections 2023-10-16T09:43:49.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f3ad5729700 1 --2- 172.21.15.158:0/2378165458 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad00ce9a0 0x7f3ad006e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f3ad5729700 1 --2- 172.21.15.158:0/2378165458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad006e7c0 0x7f3ad01657e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f3ad5729700 1 -- 172.21.15.158:0/2378165458 >> 172.21.15.158:0/2378165458 conn(0x7f3ad00cf2b0 msgr2=0x7f3ad00cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f3ad5729700 1 -- 172.21.15.158:0/2378165458 shutdown_connections 2023-10-16T09:43:49.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f3ad5729700 1 -- 172.21.15.158:0/2378165458 wait complete. 2023-10-16T09:43:49.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.332+0000 7f81e4741700 1 Processor -- start 2023-10-16T09:43:49.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.332+0000 7f81e4741700 1 -- start start 2023-10-16T09:43:49.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7f81e4741700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc159860 0x7f81dc0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7f81e4741700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f81dc159c40 0x7f81dc15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7f81e4741700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f81dc15d9a0 0x7f81dc0b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7f81e4741700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0ca040 con 0x7f81dc15d9a0 2023-10-16T09:43:49.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7f81e4741700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0caa10 con 0x7f81dc159c40 2023-10-16T09:43:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7f81e4741700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc15de40 con 0x7f81dc159860 2023-10-16T09:43:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.334+0000 7f81e2cde700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f81dc15d9a0 0x7f81dc0b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.334+0000 7f81e2cde700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f81dc15d9a0 0x7f81dc0b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46724/0 (socket says 172.21.15.158:46724) 2023-10-16T09:43:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.334+0000 7f81e2cde700 1 -- 172.21.15.158:0/3916618699 learned_addr learned my addr 172.21.15.158:0/3916618699 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f81e2cde700 1 -- 172.21.15.158:0/3916618699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc159860 msgr2=0x7f81dc0ca4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:49.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f81e2cde700 1 --2- 172.21.15.158:0/3916618699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc159860 0x7f81dc0ca4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f81e2cde700 1 -- 172.21.15.158:0/3916618699 >> v2:172.21.15.172:3300/0 conn(0x7f81dc159c40 msgr2=0x7f81dc15d460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T09:43:49.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f81e2cde700 1 --2- 172.21.15.158:0/3916618699 >> v2:172.21.15.172:3300/0 conn(0x7f81dc159c40 0x7f81dc15d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f81e2cde700 1 -- 172.21.15.158:0/3916618699 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc15e120 con 0x7f81dc15d9a0 2023-10-16T09:43:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f81e2cde700 1 --2- 172.21.15.158:0/3916618699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc15d9a0 0x7f81dc0b0d30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00bbc0 tx=0x7f81cc00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a66108dfbfa21d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f81e14db700 1 -- 172.21.15.158:0/3916618699 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00dc50 con 0x7f81dc15d9a0 2023-10-16T09:43:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f81e14db700 1 -- 172.21.15.158:0/3916618699 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00ddb0 con 0x7f81dc15d9a0 2023-10-16T09:43:49.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f81e14db700 1 -- 172.21.15.158:0/3916618699 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc012990 con 0x7f81dc15d9a0 2023-10-16T09:43:49.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f81e4741700 1 -- 172.21.15.158:0/3916618699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc15d9a0 msgr2=0x7f81dc0b0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f81e4741700 1 --2- 172.21.15.158:0/3916618699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc15d9a0 0x7f81dc0b0d30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00bbc0 tx=0x7f81cc00bed0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f81e4741700 1 -- 172.21.15.158:0/3916618699 shutdown_connections 2023-10-16T09:43:49.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f81e4741700 1 --2- 172.21.15.158:0/3916618699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc15d9a0 0x7f81dc0b0d30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f81e4741700 1 --2- 172.21.15.158:0/3916618699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc159860 0x7f81dc0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f81e4741700 1 --2- 172.21.15.158:0/3916618699 >> v2:172.21.15.172:3300/0 conn(0x7f81dc159c40 0x7f81dc15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f81e4741700 1 -- 172.21.15.158:0/3916618699 >> 172.21.15.158:0/3916618699 conn(0x7f81dc0cbd30 msgr2=0x7f81dc0cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7f81e4741700 1 -- 172.21.15.158:0/3916618699 shutdown_connections 2023-10-16T09:43:49.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7f81e4741700 1 -- 172.21.15.158:0/3916618699 wait complete. 2023-10-16T09:43:49.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f81e4741700 1 Processor -- start 2023-10-16T09:43:49.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7ff7482c6700 1 Processor -- start 2023-10-16T09:43:49.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.330+0000 7ff7482c6700 1 -- start start 2023-10-16T09:43:49.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.331+0000 7ff7482c6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff74006e990 0x7ff74015def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.331+0000 7ff7482c6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006c780 0x7ff740071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.331+0000 7ff7482c6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7400cd6c0 con 0x7ff74015cde0 2023-10-16T09:43:49.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.331+0000 7ff7482c6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff740070b30 con 0x7ff74006e990 2023-10-16T09:43:49.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.331+0000 7ff7482c6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74006d750 con 0x7ff74006c780 2023-10-16T09:43:49.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.332+0000 7ff746062700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff74015cde0 0x7ff74006e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46894/0 (socket says 172.21.15.158:46894) 2023-10-16T09:43:49.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.332+0000 7ff746062700 1 -- 172.21.15.158:0/988390248 learned_addr learned my addr 172.21.15.158:0/988390248 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:49.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.332+0000 7ff745861700 1 --2- 172.21.15.158:0/988390248 >> v2:172.21.15.172:3300/0 conn(0x7ff74006e990 0x7ff74015def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.332+0000 7ff746863700 1 --2- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006c780 0x7ff740071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7ff745861700 1 -- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006c780 msgr2=0x7ff740071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:49.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7ff745861700 1 --2- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006c780 0x7ff740071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7ff745861700 1 -- 172.21.15.158:0/988390248 >> v1:172.21.15.158:6789/0 conn(0x7ff74015cde0 legacy=0x7ff74006e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7ff745861700 1 -- 172.21.15.158:0/988390248 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74006cc10 con 0x7ff74006e990 2023-10-16T09:43:49.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.333+0000 7ff745861700 1 --2- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006e990 0x7ff74015def0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff738009c80 tx=0x7ff738009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e3ebe7819d6350 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7ff745060700 1 -- 172.21.15.158:0/988390248 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738003a20 con 0x7ff74006e990 2023-10-16T09:43:49.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7ff745060700 1 -- 172.21.15.158:0/988390248 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff738003b80 con 0x7ff74006e990 2023-10-16T09:43:49.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.338+0000 7ff745060700 1 -- 172.21.15.158:0/988390248 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73800c9b0 con 0x7ff74006e990 2023-10-16T09:43:49.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7ff7482c6700 1 -- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006e990 msgr2=0x7ff74015def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7ff7482c6700 1 --2- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006e990 0x7ff74015def0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff738009c80 tx=0x7ff738009f90 comp rx=0 tx=0).stop 2023-10-16T09:43:49.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7ff7482c6700 1 -- 172.21.15.158:0/988390248 shutdown_connections 2023-10-16T09:43:49.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7ff7482c6700 1 --2- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006c780 0x7ff740071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7ff7482c6700 1 --2- 172.21.15.158:0/988390248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006e990 0x7ff74015def0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7ff7482c6700 1 -- 172.21.15.158:0/988390248 >> 172.21.15.158:0/988390248 conn(0x7ff7400cf2b0 msgr2=0x7ff7400cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.334+0000 7f1dca9c5700 1 Processor -- start 2023-10-16T09:43:49.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.334+0000 7f1dca9c5700 1 -- start start 2023-10-16T09:43:49.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dca9c5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dca9c5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1dc40b1220 0x7f1dc4159a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dca9c5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc415e760 con 0x7f1dc40b0da0 2023-10-16T09:43:49.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dca9c5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc40713f0 con 0x7f1dc40b1220 2023-10-16T09:43:49.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dca9c5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40716b0 con 0x7f1dc4071010 2023-10-16T09:43:49.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc8f62700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1dc40b1220 0x7f1dc4159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc415e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57560/0 (socket says 172.21.15.158:57560) 2023-10-16T09:43:49.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc3fff700 1 -- 172.21.15.158:0/1797122453 learned_addr learned my addr 172.21.15.158:0/1797122453 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc8f62700 1 -- 172.21.15.158:0/1797122453 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 msgr2=0x7f1dc415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:49.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc8f62700 1 --2- 172.21.15.158:0/1797122453 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc415e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc8f62700 1 -- 172.21.15.158:0/1797122453 >> v1:172.21.15.158:6789/0 conn(0x7f1dc40b0da0 legacy=0x7f1dc406c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.335+0000 7f1dc8f62700 1 -- 172.21.15.158:0/1797122453 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc4071990 con 0x7f1dc40b1220 2023-10-16T09:43:49.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f1dc8f62700 1 --2- 172.21.15.158:0/1797122453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b1220 0x7f1dc4159a10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009840 tx=0x7f1db4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f27931241c6568 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f1dc2ffd700 1 -- 172.21.15.158:0/1797122453 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4011830 con 0x7f1dc40b1220 2023-10-16T09:43:49.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f1dc2ffd700 1 -- 172.21.15.158:0/1797122453 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db4011990 con 0x7f1dc40b1220 2023-10-16T09:43:49.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.336+0000 7f1dc2ffd700 1 -- 172.21.15.158:0/1797122453 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4011d20 con 0x7f1dc40b1220 2023-10-16T09:43:49.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 -- 172.21.15.158:0/1797122453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b1220 msgr2=0x7f1dc4159a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/1797122453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b1220 0x7f1dc4159a10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009840 tx=0x7f1db4009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:49.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 -- 172.21.15.158:0/1797122453 shutdown_connections 2023-10-16T09:43:49.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/1797122453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b1220 0x7f1dc4159a10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/1797122453 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc415e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 -- 172.21.15.158:0/1797122453 >> 172.21.15.158:0/1797122453 conn(0x7f1dc40cbd30 msgr2=0x7f1dc40cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 -- 172.21.15.158:0/1797122453 shutdown_connections 2023-10-16T09:43:49.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.337+0000 7f1dca9c5700 1 -- 172.21.15.158:0/1797122453 wait complete. 2023-10-16T09:43:49.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f1dca9c5700 1 Processor -- start 2023-10-16T09:43:49.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.339+0000 7f1dca9c5700 1 -- start start 2023-10-16T09:43:49.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f1dca9c5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc40b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f1dca9c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b0da0 0x7f1dc4152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f1dca9c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dc40b1220 0x7f1dc40c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f1dca9c5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40b4500 con 0x7f1dc40b0da0 2023-10-16T09:43:49.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f1dca9c5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc41530e0 con 0x7f1dc40b1220 2023-10-16T09:43:49.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f1dca9c5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40b3880 con 0x7f1dc4071010 2023-10-16T09:43:49.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f1dc8f62700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dc40b1220 0x7f1dc40c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f1dc8f62700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dc40b1220 0x7f1dc40c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47916/0 (socket says 172.21.15.158:47916) 2023-10-16T09:43:49.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f1dc8f62700 1 -- 172.21.15.158:0/3461277244 learned_addr learned my addr 172.21.15.158:0/3461277244 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f3ad5729700 1 Processor -- start 2023-10-16T09:43:49.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f3ad5729700 1 -- start start 2023-10-16T09:43:49.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7ff7482c6700 1 -- 172.21.15.158:0/988390248 shutdown_connections 2023-10-16T09:43:49.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7ff7482c6700 1 -- 172.21.15.158:0/988390248 wait complete. 2023-10-16T09:43:49.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f1dc37fe700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b0da0 0x7f1dc4152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.340+0000 7f81e4741700 1 -- start start 2023-10-16T09:43:49.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f81e4741700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc15d9a0 0x7f81dc0c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f81e4741700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc1f60f0 0x7f81dc1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f81e4741700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc1f5450 0x7f81dc1f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f81e4741700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc1f0f10 con 0x7f81dc1f60f0 2023-10-16T09:43:49.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f81e4741700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc1f1160 con 0x7f81dc1f5450 2023-10-16T09:43:49.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.341+0000 7f81e4741700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0c8630 con 0x7f81dc15d9a0 2023-10-16T09:43:49.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f1dc3fff700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc40b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc1f5450 0x7f81dc1f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc1f5450 0x7f81dc1f58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47918/0 (socket says 172.21.15.158:47918) 2023-10-16T09:43:49.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 -- 172.21.15.158:0/3132847344 learned_addr learned my addr 172.21.15.158:0/3132847344 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f1dc37fe700 1 -- 172.21.15.158:0/3461277244 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 msgr2=0x7f1dc40b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f1dc37fe700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc40b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f1dc37fe700 1 -- 172.21.15.158:0/3461277244 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dc40b1220 msgr2=0x7f1dc40c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f1dc37fe700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dc40b1220 0x7f1dc40c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f1dc37fe700 1 -- 172.21.15.158:0/3461277244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db40094a0 con 0x7f1dc40b0da0 2023-10-16T09:43:49.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e24dd700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc15d9a0 0x7f81dc0c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e1cdc700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc1f60f0 0x7f81dc1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f1dc37fe700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b0da0 0x7f1dc4152ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1db8000f60 tx=0x7f1db800c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 -- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc15d9a0 msgr2=0x7f81dc0c5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc15d9a0 0x7f81dc0c5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 -- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc1f60f0 msgr2=0x7f81dc1f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc1f60f0 0x7f81dc1f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.342+0000 7f81e2cde700 1 -- 172.21.15.158:0/3132847344 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc00b7f0 con 0x7f81dc1f5450 2023-10-16T09:43:49.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3ad5729700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad006e7c0 0x7f3ad015cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3ad5729700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad00ce9a0 0x7f3ad00c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3ad5729700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad0165d20 0x7f3ad00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3ad5729700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad00c6520 con 0x7f3ad00ce9a0 2023-10-16T09:43:49.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3ad5729700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad00c6800 con 0x7f3ad0165d20 2023-10-16T09:43:49.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3ad5729700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad00c6ae0 con 0x7f3ad006e7c0 2023-10-16T09:43:49.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f81e2cde700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc1f5450 0x7f81dc1f58d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f81cc017000 tx=0x7f81cc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f81e24dd700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc15d9a0 0x7f81dc0c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:49.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad006e7c0 0x7f3ad015cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad006e7c0 0x7f3ad015cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57578/0 (socket says 172.21.15.158:57578) 2023-10-16T09:43:49.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 -- 172.21.15.158:0/3465991625 learned_addr learned my addr 172.21.15.158:0/3465991625 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3ace7fc700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad00ce9a0 0x7f3ad00c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3acf7fe700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad0165d20 0x7f3ad00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7ff7482c6700 1 Processor -- start 2023-10-16T09:43:49.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f81d2ffd700 1 -- 172.21.15.158:0/3132847344 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc003ab0 con 0x7f81dc1f5450 2023-10-16T09:43:49.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f81d2ffd700 1 -- 172.21.15.158:0/3132847344 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc003c10 con 0x7f81dc1f5450 2023-10-16T09:43:49.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f81d2ffd700 1 -- 172.21.15.158:0/3132847344 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc010450 con 0x7f81dc1f5450 2023-10-16T09:43:49.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 -- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad0165d20 msgr2=0x7f3ad00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad0165d20 0x7f3ad00c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 -- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad00ce9a0 msgr2=0x7f3ad00c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad00ce9a0 0x7f3ad00c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.343+0000 7f3aceffd700 1 -- 172.21.15.158:0/3465991625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40094a0 con 0x7f3ad006e7c0 2023-10-16T09:43:49.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f3aceffd700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad006e7c0 0x7f3ad015cb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800ba50 tx=0x7f3ac800bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f1dc0ff9700 1 -- 172.21.15.158:0/3461277244 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80052d0 con 0x7f1dc40b0da0 2023-10-16T09:43:49.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f1dc0ff9700 1 -- 172.21.15.158:0/3461277244 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db8005430 con 0x7f1dc40b0da0 2023-10-16T09:43:49.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc40b3b60 con 0x7f1dc40b0da0 2023-10-16T09:43:49.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7ff7482c6700 1 -- start start 2023-10-16T09:43:49.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff7482c6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006c780 0x7ff7401400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff7482c6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006e990 0x7ff7401418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff7482c6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff74015cde0 0x7ff7400c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff7482c6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7401405e0 con 0x7ff74015cde0 2023-10-16T09:43:49.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff7482c6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740140810 con 0x7ff74006c780 2023-10-16T09:43:49.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff7482c6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff740140af0 con 0x7ff74006e990 2023-10-16T09:43:49.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006c780 0x7ff7401400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006c780 0x7ff7401400a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:47930/0 (socket says 172.21.15.158:47930) 2023-10-16T09:43:49.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 -- 172.21.15.158:0/1084976478 learned_addr learned my addr 172.21.15.158:0/1084976478 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746863700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff74015cde0 0x7ff7400c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 -- 172.21.15.158:0/1084976478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006e990 msgr2=0x7ff7401418b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:49.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006e990 0x7ff7401418b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 -- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff74015cde0 msgr2=0x7ff7400c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff74015cde0 0x7ff7400c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746062700 1 -- 172.21.15.158:0/1084976478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73800d040 con 0x7ff74006c780 2023-10-16T09:43:49.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f372cd13700 1 Processor -- start 2023-10-16T09:43:49.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f372cd13700 1 -- start start 2023-10-16T09:43:49.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f372cd13700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37200e7830 0x7f3720054520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f372cd13700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f372004f1e0 0x7f3720037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f372cd13700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3720053a80 con 0x7f3720038200 2023-10-16T09:43:49.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f372cd13700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37200484f0 con 0x7f372004f1e0 2023-10-16T09:43:49.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f372cd13700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372004ff60 con 0x7f37200e7830 2023-10-16T09:43:49.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f3727fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3720038200 0x7f3720005f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:46934/0 (socket says 172.21.15.158:46934) 2023-10-16T09:43:49.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f3727fff700 1 -- 172.21.15.158:0/2233710849 learned_addr learned my addr 172.21.15.158:0/2233710849 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:49.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7ff746863700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff74015cde0 0x7ff7400c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:49.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7ff746062700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006c780 0x7ff7401400a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00c780 tx=0x7ff73c00ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7f3726ffd700 1 --2- 172.21.15.158:0/2233710849 >> v2:172.21.15.158:3300/0 conn(0x7f372004f1e0 0x7f3720037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f1dc0ff9700 1 -- 172.21.15.158:0/3461277244 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80038a0 con 0x7f1dc40b0da0 2023-10-16T09:43:49.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.344+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc40b4120 con 0x7f1dc40b0da0 2023-10-16T09:43:49.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.345+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc40681b0 con 0x7f1dc40b0da0 2023-10-16T09:43:49.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7ff736ffd700 1 -- 172.21.15.158:0/1084976478 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c004c80 con 0x7ff74006c780 2023-10-16T09:43:49.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7ff736ffd700 1 -- 172.21.15.158:0/1084976478 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73c004de0 con 0x7ff74006c780 2023-10-16T09:43:49.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7ff736ffd700 1 -- 172.21.15.158:0/1084976478 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c0072f0 con 0x7ff74006c780 2023-10-16T09:43:49.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff740140dd0 con 0x7ff74006c780 2023-10-16T09:43:49.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7f3726ffd700 1 -- 172.21.15.158:0/2233710849 >> v2:172.21.15.172:3300/0 conn(0x7f37200e7830 msgr2=0x7f3720054520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:49.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7f3726ffd700 1 --2- 172.21.15.158:0/2233710849 >> v2:172.21.15.172:3300/0 conn(0x7f37200e7830 0x7f3720054520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7f3726ffd700 1 -- 172.21.15.158:0/2233710849 >> v1:172.21.15.158:6789/0 conn(0x7f3720038200 legacy=0x7f3720005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7f3726ffd700 1 -- 172.21.15.158:0/2233710849 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3720053d20 con 0x7f372004f1e0 2023-10-16T09:43:49.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff740141390 con 0x7ff74006c780 2023-10-16T09:43:49.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.346+0000 7f3726ffd700 1 --2- 172.21.15.158:0/2233710849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f372004f1e0 0x7f3720037cc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3720007b00 tx=0x7f3714009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1b95d311a54ca5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f1dc0ff9700 1 -- 172.21.15.158:0/3461277244 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1db8003a00 con 0x7f1dc40b0da0 2023-10-16T09:43:49.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f3ab77fe700 1 -- 172.21.15.158:0/3465991625 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800c930 con 0x7f3ad006e7c0 2023-10-16T09:43:49.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f3ab77fe700 1 -- 172.21.15.158:0/3465991625 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac800cab0 con 0x7f3ad006e7c0 2023-10-16T09:43:49.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad00c6dc0 con 0x7f3ad006e7c0 2023-10-16T09:43:49.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc0c8910 con 0x7f81dc1f5450 2023-10-16T09:43:49.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc1f13b0 con 0x7f81dc1f5450 2023-10-16T09:43:49.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f1dc0ff9700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1dac06b060 0x7f1dac06d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f3ab77fe700 1 -- 172.21.15.158:0/3465991625 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800ce60 con 0x7f3ad006e7c0 2023-10-16T09:43:49.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.348+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc1f15d0 con 0x7f81dc1f5450 2023-10-16T09:43:49.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.348+0000 7f1dc3fff700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1dac06b060 0x7f1dac06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.348+0000 7ff736ffd700 1 -- 172.21.15.158:0/1084976478 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff73c007450 con 0x7ff74006c780 2023-10-16T09:43:49.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.348+0000 7ff736ffd700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff72406af00 0x7ff72406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.348+0000 7ff736ffd700 1 -- 172.21.15.158:0/1084976478 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff73c0895c0 con 0x7ff74006c780 2023-10-16T09:43:49.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.348+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7401415b0 con 0x7ff74006c780 2023-10-16T09:43:49.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f1dc3fff700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1dac06b060 0x7f1dac06d520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009ce0 tx=0x7f1dbc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f81d2ffd700 1 -- 172.21.15.158:0/3132847344 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f81cc0105b0 con 0x7f81dc1f5450 2023-10-16T09:43:49.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad00c5c60 con 0x7f3ad006e7c0 2023-10-16T09:43:49.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f81d2ffd700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f81c006af30 0x7f81c006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f81d2ffd700 1 -- 172.21.15.158:0/3132847344 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f81cc08b000 con 0x7f81dc1f5450 2023-10-16T09:43:49.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f81e24dd700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f81c006af30 0x7f81c006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.350+0000 7f81e24dd700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f81c006af30 0x7f81c006d3f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0cd6c0 tx=0x7f81d4009570 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.350+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad00c72e0 con 0x7f3ad006e7c0 2023-10-16T09:43:49.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.351+0000 7f37267fc700 1 -- 172.21.15.158:0/2233710849 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714011830 con 0x7f372004f1e0 2023-10-16T09:43:49.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.351+0000 7f37267fc700 1 -- 172.21.15.158:0/2233710849 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3714011990 con 0x7f372004f1e0 2023-10-16T09:43:49.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.351+0000 7f3ab77fe700 1 -- 172.21.15.158:0/3465991625 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3ac8014440 con 0x7f3ad006e7c0 2023-10-16T09:43:49.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.351+0000 7f1dc0ff9700 1 -- 172.21.15.158:0/3461277244 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1db8057b70 con 0x7f1dc40b0da0 2023-10-16T09:43:49.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.351+0000 7ff745861700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff72406af00 0x7ff72406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f3ab77fe700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.351+0000 7f37267fc700 1 -- 172.21.15.158:0/2233710849 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714011d00 con 0x7f372004f1e0 2023-10-16T09:43:49.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f372cd13700 1 -- 172.21.15.158:0/2233710849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f372004f1e0 msgr2=0x7f3720037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f372cd13700 1 --2- 172.21.15.158:0/2233710849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f372004f1e0 0x7f3720037cc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3720007b00 tx=0x7f3714009b20 comp rx=0 tx=0).stop 2023-10-16T09:43:49.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f3ab77fe700 1 -- 172.21.15.158:0/3465991625 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3ac8006e40 con 0x7f3ad006e7c0 2023-10-16T09:43:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7ff745861700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff72406af00 0x7ff72406d3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff738000f80 tx=0x7ff738009f90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f372cd13700 1 -- 172.21.15.158:0/2233710849 shutdown_connections 2023-10-16T09:43:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f372cd13700 1 --2- 172.21.15.158:0/2233710849 >> v2:172.21.15.172:3300/0 conn(0x7f37200e7830 0x7f3720054520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f372cd13700 1 --2- 172.21.15.158:0/2233710849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f372004f1e0 0x7f3720037cc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f372cd13700 1 -- 172.21.15.158:0/2233710849 >> 172.21.15.158:0/2233710849 conn(0x7f3720054b90 msgr2=0x7f3720054fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f3ace7fc700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.352+0000 7f3ace7fc700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4000c00 tx=0x7f3ac4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f6af09d1700 1 Processor -- start 2023-10-16T09:43:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.347+0000 7f6af09d1700 1 -- start start 2023-10-16T09:43:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6af09d1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6aec0ce6d0 0x7f6aec06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6af09d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ceab0 0x7f6aec06e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6af09d1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec16e450 con 0x7f6aec06e7c0 2023-10-16T09:43:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6af09d1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec06ec70 con 0x7f6aec0ce6d0 2023-10-16T09:43:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6af09d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec16e730 con 0x7f6aec0ceab0 2023-10-16T09:43:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6ae9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ceab0 0x7f6aec06e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6ae9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ceab0 0x7f6aec06e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57600/0 (socket says 172.21.15.158:57600) 2023-10-16T09:43:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6ae9d9b700 1 -- 172.21.15.158:0/2671717342 learned_addr learned my addr 172.21.15.158:0/2671717342 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.349+0000 7f6aea59c700 1 --2- 172.21.15.158:0/2671717342 >> v2:172.21.15.172:3300/0 conn(0x7f6aec0ce6d0 0x7f6aec06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.353+0000 7f6aea59c700 1 -- 172.21.15.158:0/2671717342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ceab0 msgr2=0x7f6aec06e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.353+0000 7f6aea59c700 1 --2- 172.21.15.158:0/2671717342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ceab0 0x7f6aec06e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.353+0000 7f6aea59c700 1 -- 172.21.15.158:0/2671717342 >> v1:172.21.15.158:6789/0 conn(0x7f6aec06e7c0 legacy=0x7f6aec169e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.353+0000 7f6aea59c700 1 -- 172.21.15.158:0/2671717342 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec16ea10 con 0x7f6aec0ce6d0 2023-10-16T09:43:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.353+0000 7f6aea59c700 1 --2- 172.21.15.158:0/2671717342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec0ce6d0 0x7f6aec06cff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6adc006280 tx=0x7f6adc00cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=535a1a5a338949c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.354+0000 7f1dc0ff9700 1 -- 172.21.15.158:0/3461277244 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1db80537b0 con 0x7f1dc40b0da0 2023-10-16T09:43:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.355+0000 7f372cd13700 1 -- 172.21.15.158:0/2233710849 shutdown_connections 2023-10-16T09:43:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.355+0000 7f372cd13700 1 -- 172.21.15.158:0/2233710849 wait complete. 2023-10-16T09:43:49.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.355+0000 7ff736ffd700 1 -- 172.21.15.158:0/1084976478 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff73c054380 con 0x7ff74006c780 2023-10-16T09:43:49.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.357+0000 7f372cd13700 1 Processor -- start 2023-10-16T09:43:49.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.357+0000 7f3ab77fe700 1 -- 172.21.15.158:0/3465991625 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ad00c72e0 con 0x7f3ad006e7c0 2023-10-16T09:43:49.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.357+0000 7f372cd13700 1 -- start start 2023-10-16T09:43:49.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f372cd13700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3720038200 0x7f3720048ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f372cd13700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f372004f1e0 0x7f37200d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f372cd13700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37200e7830 0x7f3720045c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f372cd13700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37200497b0 con 0x7f37200e7830 2023-10-16T09:43:49.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f372cd13700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37200e38f0 con 0x7f3720038200 2023-10-16T09:43:49.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f372cd13700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37200e3bd0 con 0x7f372004f1e0 2023-10-16T09:43:49.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37200e7830 0x7f3720045c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f37277fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3720038200 0x7f3720048ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37200e7830 0x7f3720045c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46780/0 (socket says 172.21.15.158:46780) 2023-10-16T09:43:49.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 -- 172.21.15.158:0/3038917822 learned_addr learned my addr 172.21.15.158:0/3038917822 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 -- 172.21.15.158:0/3038917822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f372004f1e0 msgr2=0x7f37200d49c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T09:43:49.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f372004f1e0 0x7f37200d49c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 -- 172.21.15.158:0/3038917822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3720038200 msgr2=0x7f3720048ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3720038200 0x7f3720048ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 -- 172.21.15.158:0/3038917822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37140094a0 con 0x7f37200e7830 2023-10-16T09:43:49.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f37277fe700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3720038200 0x7f3720048ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:49.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.358+0000 7f3727fff700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37200e7830 0x7f3720045c50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3728069b10 tx=0x7f3728067e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f3713fff700 1 -- 172.21.15.158:0/3038917822 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372806aae0 con 0x7f37200e7830 2023-10-16T09:43:49.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f3713fff700 1 -- 172.21.15.158:0/3038917822 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372808e070 con 0x7f37200e7830 2023-10-16T09:43:49.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37200e3eb0 con 0x7f37200e7830 2023-10-16T09:43:49.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f3713fff700 1 -- 172.21.15.158:0/3038917822 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372806ae60 con 0x7f37200e7830 2023-10-16T09:43:49.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3720049410 con 0x7f37200e7830 2023-10-16T09:43:49.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.360+0000 7f81d2ffd700 1 -- 172.21.15.158:0/3132847344 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81cc055e50 con 0x7f81dc1f5450 2023-10-16T09:43:49.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f6ae959a700 1 -- 172.21.15.158:0/2671717342 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc011070 con 0x7f6aec0ce6d0 2023-10-16T09:43:49.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f6ae959a700 1 -- 172.21.15.158:0/2671717342 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc00c2e0 con 0x7f6aec0ce6d0 2023-10-16T09:43:49.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.359+0000 7f6ae959a700 1 -- 172.21.15.158:0/2671717342 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc007a70 con 0x7f6aec0ce6d0 2023-10-16T09:43:49.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.360+0000 7f6af09d1700 1 -- 172.21.15.158:0/2671717342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec0ce6d0 msgr2=0x7f6aec06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.360+0000 7f6af09d1700 1 --2- 172.21.15.158:0/2671717342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec0ce6d0 0x7f6aec06cff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6adc006280 tx=0x7f6adc00cdb0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.360+0000 7f6af09d1700 1 -- 172.21.15.158:0/2671717342 shutdown_connections 2023-10-16T09:43:49.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.360+0000 7f6af09d1700 1 --2- 172.21.15.158:0/2671717342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec0ce6d0 0x7f6aec06cff0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.360+0000 7f6af09d1700 1 --2- 172.21.15.158:0/2671717342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ceab0 0x7f6aec06e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.360+0000 7f6af09d1700 1 -- 172.21.15.158:0/2671717342 >> 172.21.15.158:0/2671717342 conn(0x7f6aec0cf2b0 msgr2=0x7f6aec0cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.361+0000 7f6af09d1700 1 -- 172.21.15.158:0/2671717342 shutdown_connections 2023-10-16T09:43:49.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.361+0000 7f6af09d1700 1 -- 172.21.15.158:0/2671717342 wait complete. 2023-10-16T09:43:49.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.361+0000 7f3713fff700 1 -- 172.21.15.158:0/3038917822 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3728073c00 con 0x7f37200e7830 2023-10-16T09:43:49.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.361+0000 7f3713fff700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f370c06af00 0x7f370c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.362+0000 7f3713fff700 1 -- 172.21.15.158:0/3038917822 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f37281034d0 con 0x7f37200e7830 2023-10-16T09:43:49.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.363+0000 7f6af09d1700 1 Processor -- start 2023-10-16T09:43:49.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.363+0000 7f37277fe700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f370c06af00 0x7f370c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.363+0000 7f37277fe700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f370c06af00 0x7f370c06d3c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f371c009de0 tx=0x7f371c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.363+0000 7f6af09d1700 1 -- start start 2023-10-16T09:43:49.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6af09d1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec06e7c0 0x7f6aec08a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6af09d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ce6d0 0x7f6aec0a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6af09d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aec0ceab0 0x7f6aec2034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6af09d1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec08a590 con 0x7f6aec0ceab0 2023-10-16T09:43:49.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6af09d1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec169d50 con 0x7f6aec06e7c0 2023-10-16T09:43:49.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6af09d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec16a030 con 0x7f6aec0ce6d0 2023-10-16T09:43:49.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6aead9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aec0ceab0 0x7f6aec2034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6aead9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aec0ceab0 0x7f6aec2034b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46786/0 (socket says 172.21.15.158:46786) 2023-10-16T09:43:49.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.364+0000 7f6aead9d700 1 -- 172.21.15.158:0/987658996 learned_addr learned my addr 172.21.15.158:0/987658996 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aea59c700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec06e7c0 0x7f6aec08a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aead9d700 1 -- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ce6d0 msgr2=0x7f6aec0a8480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:43:49.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aead9d700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ce6d0 0x7f6aec0a8480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aead9d700 1 -- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec06e7c0 msgr2=0x7f6aec08a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aead9d700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec06e7c0 0x7f6aec08a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aead9d700 1 -- 172.21.15.158:0/987658996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc00c7b0 con 0x7f6aec0ceab0 2023-10-16T09:43:49.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aea59c700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec06e7c0 0x7f6aec08a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:49.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.365+0000 7f6aead9d700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aec0ceab0 0x7f6aec2034b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400cbc0 tx=0x7f6ae400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.366+0000 7f6adaffd700 1 -- 172.21.15.158:0/987658996 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4003c00 con 0x7f6aec0ceab0 2023-10-16T09:43:49.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.366+0000 7f6adaffd700 1 -- 172.21.15.158:0/987658996 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae4003d60 con 0x7f6aec0ceab0 2023-10-16T09:43:49.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.366+0000 7f6adaffd700 1 -- 172.21.15.158:0/987658996 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae400aa80 con 0x7f6aec0ceab0 2023-10-16T09:43:49.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.366+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec16a310 con 0x7f6aec0ceab0 2023-10-16T09:43:49.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.366+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec168a70 con 0x7f6aec0ceab0 2023-10-16T09:43:49.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.367+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec0681b0 con 0x7f6aec0ceab0 2023-10-16T09:43:49.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.368+0000 7f6adaffd700 1 -- 172.21.15.158:0/987658996 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6ae400abe0 con 0x7f6aec0ceab0 2023-10-16T09:43:49.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.368+0000 7f6adaffd700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6acc06af50 0x7f6acc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.369+0000 7f6aea59c700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6acc06af50 0x7f6acc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.369+0000 7f6aea59c700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6acc06af50 0x7f6acc06d410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc012000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.375+0000 7f6adaffd700 1 -- 172.21.15.158:0/987658996 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6ae40589b0 con 0x7f6aec0ceab0 2023-10-16T09:43:49.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.376+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37200e43d0 con 0x7f37200e7830 2023-10-16T09:43:49.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.377+0000 7f6adaffd700 1 -- 172.21.15.158:0/987658996 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ae40545f0 con 0x7f6aec0ceab0 2023-10-16T09:43:49.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.386+0000 7f3713fff700 1 -- 172.21.15.158:0/3038917822 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f37200e43d0 con 0x7f37200e7830 2023-10-16T09:43:49.706 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:49.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.704+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa970005870 con 0x7fa9900ce9a0 2023-10-16T09:43:49.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.704+0000 7fa9777fe700 1 -- 172.21.15.158:0/2559940208 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa980015020 con 0x7fa9900ce9a0 2023-10-16T09:43:49.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.707+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f61b4005870 con 0x7f61d406c780 2023-10-16T09:43:49.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.708+0000 7f61bbfff700 1 -- 172.21.15.158:0/1221141334 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f61cc018070 con 0x7f61d406c780 2023-10-16T09:43:49.710 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:49.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.708+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa97c06afd0 msgr2=0x7fa97c06d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.708+0000 7fa995745700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa97c06afd0 0x7fa97c06d490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa9840090b0 tx=0x7fa984010000 comp rx=0 tx=0).stop 2023-10-16T09:43:49.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.708+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9900ce9a0 msgr2=0x7fa990152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.708+0000 7fa995745700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9900ce9a0 0x7fa990152590 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa98000b550 tx=0x7fa98000b910 comp rx=0 tx=0).stop 2023-10-16T09:43:49.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 shutdown_connections 2023-10-16T09:43:49.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa990165d20 0x7fa9900c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa97c06afd0 0x7fa97c06d490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa99006e7c0 0x7fa99017a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 --2- 172.21.15.158:0/2559940208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9900ce9a0 0x7fa990152590 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 >> 172.21.15.158:0/2559940208 conn(0x7fa9900cf2b0 msgr2=0x7fa9900b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 shutdown_connections 2023-10-16T09:43:49.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.709+0000 7fa995745700 1 -- 172.21.15.158:0/2559940208 wait complete. 2023-10-16T09:43:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.716+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61c006af50 msgr2=0x7f61c006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.716+0000 7f61d98ea700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61c006af50 0x7f61c006d410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61d41faad0 tx=0x7f61c8006bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.716+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 msgr2=0x7f61d41f7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.716+0000 7f61d98ea700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d41f7fb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f61cc002080 tx=0x7f61cc00a340 comp rx=0 tx=0).stop 2023-10-16T09:43:49.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 shutdown_connections 2023-10-16T09:43:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61d415cde0 0x7f61d40c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61d406c780 0x7f61d41f7fb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f61c006af50 0x7f61c006d410 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 --2- 172.21.15.158:0/1221141334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61d406e990 0x7f61d41560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 >> 172.21.15.158:0/1221141334 conn(0x7f61d40cf2b0 msgr2=0x7f61d4156970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 shutdown_connections 2023-10-16T09:43:49.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.717+0000 7f61d98ea700 1 -- 172.21.15.158:0/1221141334 wait complete. 2023-10-16T09:43:49.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2023-10-16T09:43:49.731 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2023-10-16T09:43:49.738 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.754+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1dc40b4640 con 0x7f1dc40b0da0 2023-10-16T09:43:49.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.754+0000 7f1dc0ff9700 1 -- 172.21.15.158:0/3461277244 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1dc40b4640 con 0x7f1dc40b0da0 2023-10-16T09:43:49.757 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:49.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.755+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff7400681b0 con 0x7ff74006c780 2023-10-16T09:43:49.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.756+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f81dc0681b0 con 0x7f81dc1f5450 2023-10-16T09:43:49.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.756+0000 7ff736ffd700 1 -- 172.21.15.158:0/1084976478 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff73c0579a0 con 0x7ff74006c780 2023-10-16T09:43:49.759 INFO:teuthology.orchestra.run.smithi158.stdout:55834574858 2023-10-16T09:43:49.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.756+0000 7f81d2ffd700 1 -- 172.21.15.158:0/3132847344 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f81cc059470 con 0x7f81dc1f5450 2023-10-16T09:43:49.761 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:49.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.757+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1dac06b060 msgr2=0x7f1dac06d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.757+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1dac06b060 0x7f1dac06d520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009ce0 tx=0x7f1dbc00b040 comp rx=0 tx=0).stop 2023-10-16T09:43:49.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.757+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b0da0 msgr2=0x7f1dc4152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.757+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b0da0 0x7f1dc4152ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1db8000f60 tx=0x7f1db800c730 comp rx=0 tx=0).stop 2023-10-16T09:43:49.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff72406af00 msgr2=0x7ff72406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff72406af00 0x7ff72406d3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff738000f80 tx=0x7ff738009f90 comp rx=0 tx=0).stop 2023-10-16T09:43:49.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006c780 msgr2=0x7ff7401400a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006c780 0x7ff7401400a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00c780 tx=0x7ff73c00ca90 comp rx=0 tx=0).stop 2023-10-16T09:43:49.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 shutdown_connections 2023-10-16T09:43:49.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff74015cde0 0x7ff7400c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff74006c780 0x7ff7401400a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff72406af00 0x7ff72406d3c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 --2- 172.21.15.158:0/1084976478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff74006e990 0x7ff7401418b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 >> 172.21.15.158:0/1084976478 conn(0x7ff7400cf2b0 msgr2=0x7ff74015a950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 shutdown_connections 2023-10-16T09:43:49.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.759+0000 7ff7482c6700 1 -- 172.21.15.158:0/1084976478 wait complete. 2023-10-16T09:43:49.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 shutdown_connections 2023-10-16T09:43:49.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dc40b1220 0x7f1dc40c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1dac06b060 0x7f1dac06d520 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dc4071010 0x7f1dc40b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 --2- 172.21.15.158:0/3461277244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1dc40b0da0 0x7f1dc4152ba0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 >> 172.21.15.158:0/3461277244 conn(0x7f1dc40cbd30 msgr2=0x7f1dc406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 shutdown_connections 2023-10-16T09:43:49.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f81c006af30 msgr2=0x7f81c006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f81c006af30 0x7f81c006d3f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0cd6c0 tx=0x7f81d4009570 comp rx=0 tx=0).stop 2023-10-16T09:43:49.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc1f5450 msgr2=0x7f81dc1f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f1dca9c5700 1 -- 172.21.15.158:0/3461277244 wait complete. 2023-10-16T09:43:49.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc1f5450 0x7f81dc1f58d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f81cc017000 tx=0x7f81cc004750 comp rx=0 tx=0).stop 2023-10-16T09:43:49.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 shutdown_connections 2023-10-16T09:43:49.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc1f5450 0x7f81dc1f58d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f81c006af30 0x7f81c006d3f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc15d9a0 0x7f81dc0c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.760+0000 7f81e4741700 1 --2- 172.21.15.158:0/3132847344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81dc1f60f0 0x7f81dc1f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.761+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 >> 172.21.15.158:0/3132847344 conn(0x7f81dc0cbd30 msgr2=0x7f81dc0cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.765+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 shutdown_connections 2023-10-16T09:43:49.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.766+0000 7f81e4741700 1 -- 172.21.15.158:0/3132847344 wait complete. 2023-10-16T09:43:49.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.766+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f37200e43d0 con 0x7f37200e7830 2023-10-16T09:43:49.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.767+0000 7f3713fff700 1 -- 172.21.15.158:0/3038917822 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f37200e43d0 con 0x7f37200e7830 2023-10-16T09:43:49.775 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:49.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.770+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f370c06af00 msgr2=0x7f370c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.770+0000 7f372cd13700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f370c06af00 0x7f370c06d3c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f371c009de0 tx=0x7f371c009190 comp rx=0 tx=0).stop 2023-10-16T09:43:49.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.770+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37200e7830 msgr2=0x7f3720045c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.770+0000 7f372cd13700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37200e7830 0x7f3720045c50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3728069b10 tx=0x7f3728067e00 comp rx=0 tx=0).stop 2023-10-16T09:43:49.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.771+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 shutdown_connections 2023-10-16T09:43:49.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.771+0000 7f372cd13700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37200e7830 0x7f3720045c50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.771+0000 7f372cd13700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3720038200 0x7f3720048ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.771+0000 7f372cd13700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f370c06af00 0x7f370c06d3c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.771+0000 7f372cd13700 1 --2- 172.21.15.158:0/3038917822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f372004f1e0 0x7f37200d49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.771+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 >> 172.21.15.158:0/3038917822 conn(0x7f3720054b90 msgr2=0x7f37200c7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2023-10-16T09:43:49.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.774+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3ad00c6c20 con 0x7f3ad006e7c0 2023-10-16T09:43:49.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.775+0000 7f3ab77fe700 1 -- 172.21.15.158:0/3465991625 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ad00c6c20 con 0x7f3ad006e7c0 2023-10-16T09:43:49.780 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.781 INFO:teuthology.orchestra.run.smithi158.stdout:55834574858 2023-10-16T09:43:49.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.776+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 shutdown_connections 2023-10-16T09:43:49.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.776+0000 7f372cd13700 1 -- 172.21.15.158:0/3038917822 wait complete. 2023-10-16T09:43:49.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.777+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 msgr2=0x7f3abc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.777+0000 7f3ad5729700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4000c00 tx=0x7f3ac4008030 comp rx=0 tx=0).stop 2023-10-16T09:43:49.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.777+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad006e7c0 msgr2=0x7f3ad015cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.777+0000 7f3ad5729700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad006e7c0 0x7f3ad015cb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800ba50 tx=0x7f3ac800bd60 comp rx=0 tx=0).stop 2023-10-16T09:43:49.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 shutdown_connections 2023-10-16T09:43:49.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad0165d20 0x7f3ad00c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3abc06af50 0x7f3abc06d410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ad006e7c0 0x7f3ad015cb20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 --2- 172.21.15.158:0/3465991625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ad00ce9a0 0x7f3ad00c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 >> 172.21.15.158:0/3465991625 conn(0x7f3ad00cf2b0 msgr2=0x7f3ad015fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 shutdown_connections 2023-10-16T09:43:49.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.778+0000 7f3ad5729700 1 -- 172.21.15.158:0/3465991625 wait complete. 2023-10-16T09:43:49.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2023-10-16T09:43:49.787 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2023-10-16T09:43:49.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.786+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6aec0ae2c0 con 0x7f6aec0ceab0 2023-10-16T09:43:49.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.787+0000 7f6adaffd700 1 -- 172.21.15.158:0/987658996 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ae4057c10 con 0x7f6aec0ceab0 2023-10-16T09:43:49.789 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.790 INFO:teuthology.orchestra.run.smithi158.stdout:55834574857 2023-10-16T09:43:49.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6acc06af50 msgr2=0x7f6acc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6acc06af50 0x7f6acc06d410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc012000 comp rx=0 tx=0).stop 2023-10-16T09:43:49.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aec0ceab0 msgr2=0x7f6aec2034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aec0ceab0 0x7f6aec2034b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400cbc0 tx=0x7f6ae400ced0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 shutdown_connections 2023-10-16T09:43:49.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aec0ceab0 0x7f6aec2034b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aec06e7c0 0x7f6aec08a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6acc06af50 0x7f6acc06d410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 --2- 172.21.15.158:0/987658996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aec0ce6d0 0x7f6aec0a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 >> 172.21.15.158:0/987658996 conn(0x7f6aec0cf2b0 msgr2=0x7f6aec0cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 shutdown_connections 2023-10-16T09:43:49.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.789+0000 7f6af09d1700 1 -- 172.21.15.158:0/987658996 wait complete. 2023-10-16T09:43:49.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2023-10-16T09:43:49.795 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2023-10-16T09:43:49.796 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.804 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2023-10-16T09:43:49.804 DEBUG:teuthology.parallel:result is None 2023-10-16T09:43:49.804 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T09:43:49.805 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:43:49.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.963+0000 7f2201693700 1 Processor -- start 2023-10-16T09:43:49.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.963+0000 7f2201693700 1 -- start start 2023-10-16T09:43:49.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f2201693700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21fc0cdee0 0x7f21fc15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f2201693700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21fc0b6e10 0x7f21fc1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f2201693700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0ce2c0 0x7f21fc1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f2201693700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc15d2c0 con 0x7f21fc0cdee0 2023-10-16T09:43:49.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f2201693700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc152c20 con 0x7f21fc0b6e10 2023-10-16T09:43:49.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f2201693700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc15c5d0 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21faffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21fc0cdee0 0x7f21fc15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21faffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21fc0cdee0 0x7f21fc15bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46790/0 (socket says 172.21.15.158:46790) 2023-10-16T09:43:49.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21faffd700 1 -- 172.21.15.158:0/2958726370 learned_addr learned my addr 172.21.15.158:0/2958726370 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21fb7fe700 1 --2- 172.21.15.158:0/2958726370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0ce2c0 0x7f21fc1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21fa7fc700 1 --2- 172.21.15.158:0/2958726370 >> v2:172.21.15.172:3300/0 conn(0x7f21fc0b6e10 0x7f21fc1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21faffd700 1 -- 172.21.15.158:0/2958726370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0ce2c0 msgr2=0x7f21fc1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:49.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21faffd700 1 --2- 172.21.15.158:0/2958726370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0ce2c0 0x7f21fc1526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.964+0000 7f21faffd700 1 -- 172.21.15.158:0/2958726370 >> v2:172.21.15.172:3300/0 conn(0x7f21fc0b6e10 msgr2=0x7f21fc1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:49.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f21faffd700 1 --2- 172.21.15.158:0/2958726370 >> v2:172.21.15.172:3300/0 conn(0x7f21fc0b6e10 0x7f21fc1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f21faffd700 1 -- 172.21.15.158:0/2958726370 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc15c8b0 con 0x7f21fc0cdee0 2023-10-16T09:43:49.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f21fa7fc700 1 --2- 172.21.15.158:0/2958726370 >> v2:172.21.15.172:3300/0 conn(0x7f21fc0b6e10 0x7f21fc1521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:49.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f21faffd700 1 --2- 172.21.15.158:0/2958726370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0cdee0 0x7f21fc15bd10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0b3b90 tx=0x7f21ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0016fe2bcae9469 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f21f9ffb700 1 -- 172.21.15.158:0/2958726370 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00b860 con 0x7f21fc0cdee0 2023-10-16T09:43:49.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f21f9ffb700 1 -- 172.21.15.158:0/2958726370 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec00b9c0 con 0x7f21fc0cdee0 2023-10-16T09:43:49.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f21f9ffb700 1 -- 172.21.15.158:0/2958726370 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00bd50 con 0x7f21fc0cdee0 2023-10-16T09:43:49.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 -- 172.21.15.158:0/2958726370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0cdee0 msgr2=0x7f21fc15bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 --2- 172.21.15.158:0/2958726370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0cdee0 0x7f21fc15bd10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0b3b90 tx=0x7f21ec009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:49.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 -- 172.21.15.158:0/2958726370 shutdown_connections 2023-10-16T09:43:49.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 --2- 172.21.15.158:0/2958726370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0ce2c0 0x7f21fc1526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 --2- 172.21.15.158:0/2958726370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0cdee0 0x7f21fc15bd10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 --2- 172.21.15.158:0/2958726370 >> v2:172.21.15.172:3300/0 conn(0x7f21fc0b6e10 0x7f21fc1521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 -- 172.21.15.158:0/2958726370 >> 172.21.15.158:0/2958726370 conn(0x7f21fc06a8d0 msgr2=0x7f21fc06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:49.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.965+0000 7f2201693700 1 -- 172.21.15.158:0/2958726370 shutdown_connections 2023-10-16T09:43:49.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.966+0000 7f2201693700 1 -- 172.21.15.158:0/2958726370 wait complete. 2023-10-16T09:43:49.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.967+0000 7f2201693700 1 Processor -- start 2023-10-16T09:43:49.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.967+0000 7f2201693700 1 -- start start 2023-10-16T09:43:49.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f2201693700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21fc0b6e10 0x7f21fc14c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f2201693700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0cdee0 0x7f21fc0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f2201693700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0ce2c0 0x7f21fc0b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f2201693700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc1e62e0 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f2201693700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc152d60 con 0x7f21fc0b6e10 2023-10-16T09:43:49.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f2201693700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc153040 con 0x7f21fc0cdee0 2023-10-16T09:43:49.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0ce2c0 0x7f21fc0b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fa7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0cdee0 0x7f21fc0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21faffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21fc0b6e10 0x7f21fc14c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0ce2c0 0x7f21fc0b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46798/0 (socket says 172.21.15.158:46798) 2023-10-16T09:43:49.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 -- 172.21.15.158:0/3820658068 learned_addr learned my addr 172.21.15.158:0/3820658068 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:49.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 -- 172.21.15.158:0/3820658068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0cdee0 msgr2=0x7f21fc0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0cdee0 0x7f21fc0b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 -- 172.21.15.158:0/3820658068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21fc0b6e10 msgr2=0x7f21fc14c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:49.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21fc0b6e10 0x7f21fc14c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:49.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 -- 172.21.15.158:0/3820658068 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec009550 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21fb7fe700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0ce2c0 0x7f21fc0b23c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f21f400cc30 tx=0x7f21f400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21f37fe700 1 -- 172.21.15.158:0/3820658068 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f400aa50 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21f37fe700 1 -- 172.21.15.158:0/3820658068 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f400abb0 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc153320 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.968+0000 7f21f37fe700 1 -- 172.21.15.158:0/3820658068 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4003a20 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.969+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc14c980 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.970+0000 7f21f37fe700 1 -- 172.21.15.158:0/3820658068 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f21f400f3f0 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.970+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc0681b0 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.971+0000 7f21f37fe700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f21dc06af00 0x7f21dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:49.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.971+0000 7f21f37fe700 1 -- 172.21.15.158:0/3820658068 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f21f4089950 con 0x7f21fc0ce2c0 2023-10-16T09:43:49.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.971+0000 7f21faffd700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f21dc06af00 0x7f21dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:49.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.971+0000 7f21faffd700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f21dc06af00 0x7f21dc06d3c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f21ec000c00 tx=0x7f21ec008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:49.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:49.976+0000 7f21f37fe700 1 -- 172.21.15.158:0/3820658068 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21f4054710 con 0x7f21fc0ce2c0 2023-10-16T09:43:50.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.165+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21fc15cee0 con 0x7f21dc06af00 2023-10-16T09:43:50.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.174+0000 7f21f37fe700 1 -- 172.21.15.158:0/3820658068 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345229 (secure 0 0 0) 0x7f21ec00bce0 con 0x7f21dc06af00 2023-10-16T09:43:50.181 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:50.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f21dc06af00 msgr2=0x7f21dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f21dc06af00 0x7f21dc06d3c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f21ec000c00 tx=0x7f21ec008030 comp rx=0 tx=0).stop 2023-10-16T09:43:50.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0ce2c0 msgr2=0x7f21fc0b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0ce2c0 0x7f21fc0b23c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f21f400cc30 tx=0x7f21f400cf40 comp rx=0 tx=0).stop 2023-10-16T09:43:50.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 shutdown_connections 2023-10-16T09:43:50.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21fc0ce2c0 0x7f21fc0b23c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21fc0b6e10 0x7f21fc14c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f21dc06af00 0x7f21dc06d3c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 --2- 172.21.15.158:0/3820658068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21fc0cdee0 0x7f21fc0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 >> 172.21.15.158:0/3820658068 conn(0x7f21fc06a8d0 msgr2=0x7f21fc150d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:50.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 shutdown_connections 2023-10-16T09:43:50.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.178+0000 7f2201693700 1 -- 172.21.15.158:0/3820658068 wait complete. 2023-10-16T09:43:50.187 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:43:50.192 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-16T09:43:49.224077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3284992,"data_stored":1988256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003263"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944843+0000","last_change":"2023-10-16T09:43:21.932095+0000","last_active":"2023-10-16T09:43:25.944843+0000","last_peered":"2023-10-16T09:43:25.944843+0000","last_clean":"2023-10-16T09:43:25.944843+0000","last_became_active":"2023-10-16T09:43:21.931826+0000","last_became_peered":"2023-10-16T09:43:21.931826+0000","last_unstale":"2023-10-16T09:43:25.944843+0000","last_undegraded":"2023-10-16T09:43:25.944843+0000","last_fullsized":"2023-10-16T09:43:25.944843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955709+0000","last_change":"2023-10-16T09:43:21.921656+0000","last_active":"2023-10-16T09:43:25.955709+0000","last_peered":"2023-10-16T09:43:25.955709+0000","last_clean":"2023-10-16T09:43:25.955709+0000","last_became_active":"2023-10-16T09:43:21.921392+0000","last_became_peered":"2023-10-16T09:43:21.921392+0000","last_unstale":"2023-10-16T09:43:25.955709+0000","last_undegraded":"2023-10-16T09:43:25.955709+0000","last_fullsized":"2023-10-16T09:43:25.955709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951037+0000","last_change":"2023-10-16T09:43:21.925692+0000","last_active":"2023-10-16T09:43:25.951037+0000","last_peered":"2023-10-16T09:43:25.951037+0000","last_clean":"2023-10-16T09:43:25.951037+0000","last_became_active":"2023-10-16T09:43:21.925438+0000","last_became_peered":"2023-10-16T09:43:21.925438+0000","last_unstale":"2023-10-16T09:43:25.951037+0000","last_undegraded":"2023-10-16T09:43:25.951037+0000","last_fullsized":"2023-10-16T09:43:25.951037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950435+0000","last_change":"2023-10-16T09:43:21.927209+0000","last_active":"2023-10-16T09:43:25.950435+0000","last_peered":"2023-10-16T09:43:25.950435+0000","last_clean":"2023-10-16T09:43:25.950435+0000","last_became_active":"2023-10-16T09:43:21.926900+0000","last_became_peered":"2023-10-16T09:43:21.926900+0000","last_unstale":"2023-10-16T09:43:25.950435+0000","last_undegraded":"2023-10-16T09:43:25.950435+0000","last_fullsized":"2023-10-16T09:43:25.950435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944943+0000","last_change":"2023-10-16T09:43:21.924582+0000","last_active":"2023-10-16T09:43:25.944943+0000","last_peered":"2023-10-16T09:43:25.944943+0000","last_clean":"2023-10-16T09:43:25.944943+0000","last_became_active":"2023-10-16T09:43:21.924329+0000","last_became_peered":"2023-10-16T09:43:21.924329+0000","last_unstale":"2023-10-16T09:43:25.944943+0000","last_undegraded":"2023-10-16T09:43:25.944943+0000","last_fullsized":"2023-10-16T09:43:25.944943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950668+0000","last_change":"2023-10-16T09:43:21.937326+0000","last_active":"2023-10-16T09:43:25.950668+0000","last_peered":"2023-10-16T09:43:25.950668+0000","last_clean":"2023-10-16T09:43:25.950668+0000","last_became_active":"2023-10-16T09:43:21.936665+0000","last_became_peered":"2023-10-16T09:43:21.936665+0000","last_unstale":"2023-10-16T09:43:25.950668+0000","last_undegraded":"2023-10-16T09:43:25.950668+0000","last_fullsized":"2023-10-16T09:43:25.950668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:21.923235+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:21.923021+0000","last_became_peered":"2023-10-16T09:43:21.923021+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965916+0000","last_change":"2023-10-16T09:43:21.933111+0000","last_active":"2023-10-16T09:43:25.965916+0000","last_peered":"2023-10-16T09:43:25.965916+0000","last_clean":"2023-10-16T09:43:25.965916+0000","last_became_active":"2023-10-16T09:43:21.932835+0000","last_became_peered":"2023-10-16T09:43:21.932835+0000","last_unstale":"2023-10-16T09:43:25.965916+0000","last_undegraded":"2023-10-16T09:43:25.965916+0000","last_fullsized":"2023-10-16T09:43:25.965916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944721+0000","last_change":"2023-10-16T09:43:21.940444+0000","last_active":"2023-10-16T09:43:25.944721+0000","last_peered":"2023-10-16T09:43:25.944721+0000","last_clean":"2023-10-16T09:43:25.944721+0000","last_became_active":"2023-10-16T09:43:21.939823+0000","last_became_peered":"2023-10-16T09:43:21.939823+0000","last_unstale":"2023-10-16T09:43:25.944721+0000","last_undegraded":"2023-10-16T09:43:25.944721+0000","last_fullsized":"2023-10-16T09:43:25.944721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956248+0000","last_change":"2023-10-16T09:43:21.934573+0000","last_active":"2023-10-16T09:43:25.956248+0000","last_peered":"2023-10-16T09:43:25.956248+0000","last_clean":"2023-10-16T09:43:25.956248+0000","last_became_active":"2023-10-16T09:43:21.934285+0000","last_became_peered":"2023-10-16T09:43:21.934285+0000","last_unstale":"2023-10-16T09:43:25.956248+0000","last_undegraded":"2023-10-16T09:43:25.956248+0000","last_fullsized":"2023-10-16T09:43:25.956248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953996+0000","last_change":"2023-10-16T09:43:21.934669+0000","last_active":"2023-10-16T09:43:25.953996+0000","last_peered":"2023-10-16T09:43:25.953996+0000","last_clean":"2023-10-16T09:43:25.953996+0000","last_became_active":"2023-10-16T09:43:21.933918+0000","last_became_peered":"2023-10-16T09:43:21.933918+0000","last_unstale":"2023-10-16T09:43:25.953996+0000","last_undegraded":"2023-10-16T09:43:25.953996+0000","last_fullsized":"2023-10-16T09:43:25.953996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950011+0000","last_change":"2023-10-16T09:43:21.934285+0000","last_active":"2023-10-16T09:43:25.950011+0000","last_peered":"2023-10-16T09:43:25.950011+0000","last_clean":"2023-10-16T09:43:25.950011+0000","last_became_active":"2023-10-16T09:43:21.933516+0000","last_became_peered":"2023-10-16T09:43:21.933516+0000","last_unstale":"2023-10-16T09:43:25.950011+0000","last_undegraded":"2023-10-16T09:43:25.950011+0000","last_fullsized":"2023-10-16T09:43:25.950011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:56.814327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965961+0000","last_change":"2023-10-16T09:43:21.934162+0000","last_active":"2023-10-16T09:43:25.965961+0000","last_peered":"2023-10-16T09:43:25.965961+0000","last_clean":"2023-10-16T09:43:25.965961+0000","last_became_active":"2023-10-16T09:43:21.933898+0000","last_became_peered":"2023-10-16T09:43:21.933898+0000","last_unstale":"2023-10-16T09:43:25.965961+0000","last_undegraded":"2023-10-16T09:43:25.965961+0000","last_fullsized":"2023-10-16T09:43:25.965961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948340+0000","last_change":"2023-10-16T09:43:21.923364+0000","last_active":"2023-10-16T09:43:25.948340+0000","last_peered":"2023-10-16T09:43:25.948340+0000","last_clean":"2023-10-16T09:43:25.948340+0000","last_became_active":"2023-10-16T09:43:21.923123+0000","last_became_peered":"2023-10-16T09:43:21.923123+0000","last_unstale":"2023-10-16T09:43:25.948340+0000","last_undegraded":"2023-10-16T09:43:25.948340+0000","last_fullsized":"2023-10-16T09:43:25.948340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953943+0000","last_change":"2023-10-16T09:43:21.925350+0000","last_active":"2023-10-16T09:43:25.953943+0000","last_peered":"2023-10-16T09:43:25.953943+0000","last_clean":"2023-10-16T09:43:25.953943+0000","last_became_active":"2023-10-16T09:43:21.925107+0000","last_became_peered":"2023-10-16T09:43:21.925107+0000","last_unstale":"2023-10-16T09:43:25.953943+0000","last_undegraded":"2023-10-16T09:43:25.953943+0000","last_fullsized":"2023-10-16T09:43:25.953943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951212+0000","last_change":"2023-10-16T09:43:21.927864+0000","last_active":"2023-10-16T09:43:25.951212+0000","last_peered":"2023-10-16T09:43:25.951212+0000","last_clean":"2023-10-16T09:43:25.951212+0000","last_became_active":"2023-10-16T09:43:21.927411+0000","last_became_peered":"2023-10-16T09:43:21.927411+0000","last_unstale":"2023-10-16T09:43:25.951212+0000","last_undegraded":"2023-10-16T09:43:25.951212+0000","last_fullsized":"2023-10-16T09:43:25.951212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950524+0000","last_change":"2023-10-16T09:43:21.936884+0000","last_active":"2023-10-16T09:43:25.950524+0000","last_peered":"2023-10-16T09:43:25.950524+0000","last_clean":"2023-10-16T09:43:25.950524+0000","last_became_active":"2023-10-16T09:43:21.936569+0000","last_became_peered":"2023-10-16T09:43:21.936569+0000","last_unstale":"2023-10-16T09:43:25.950524+0000","last_undegraded":"2023-10-16T09:43:25.950524+0000","last_fullsized":"2023-10-16T09:43:25.950524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948848+0000","last_change":"2023-10-16T09:43:21.935147+0000","last_active":"2023-10-16T09:43:25.948848+0000","last_peered":"2023-10-16T09:43:25.948848+0000","last_clean":"2023-10-16T09:43:25.948848+0000","last_became_active":"2023-10-16T09:43:21.934786+0000","last_became_peered":"2023-10-16T09:43:21.934786+0000","last_unstale":"2023-10-16T09:43:25.948848+0000","last_undegraded":"2023-10-16T09:43:25.948848+0000","last_fullsized":"2023-10-16T09:43:25.948848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945003+0000","last_change":"2023-10-16T09:43:21.927238+0000","last_active":"2023-10-16T09:43:25.945003+0000","last_peered":"2023-10-16T09:43:25.945003+0000","last_clean":"2023-10-16T09:43:25.945003+0000","last_became_active":"2023-10-16T09:43:21.926944+0000","last_became_peered":"2023-10-16T09:43:21.926944+0000","last_unstale":"2023-10-16T09:43:25.945003+0000","last_undegraded":"2023-10-16T09:43:25.945003+0000","last_fullsized":"2023-10-16T09:43:25.945003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015597+0000","last_change":"2023-10-16T09:43:25.972270+0000","last_active":"2023-10-16T09:43:28.015597+0000","last_peered":"2023-10-16T09:43:28.015597+0000","last_clean":"2023-10-16T09:43:28.015597+0000","last_became_active":"2023-10-16T09:43:25.971856+0000","last_became_peered":"2023-10-16T09:43:25.971856+0000","last_unstale":"2023-10-16T09:43:28.015597+0000","last_undegraded":"2023-10-16T09:43:28.015597+0000","last_fullsized":"2023-10-16T09:43:28.015597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:00.427859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951697+0000","last_change":"2023-10-16T09:43:20.934516+0000","last_active":"2023-10-16T09:43:25.951697+0000","last_peered":"2023-10-16T09:43:25.951697+0000","last_clean":"2023-10-16T09:43:25.951697+0000","last_became_active":"2023-10-16T09:43:20.934196+0000","last_became_peered":"2023-10-16T09:43:20.934196+0000","last_unstale":"2023-10-16T09:43:25.951697+0000","last_undegraded":"2023-10-16T09:43:25.951697+0000","last_fullsized":"2023-10-16T09:43:25.951697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966598+0000","last_change":"2023-10-16T09:43:21.941097+0000","last_active":"2023-10-16T09:43:25.966598+0000","last_peered":"2023-10-16T09:43:25.966598+0000","last_clean":"2023-10-16T09:43:25.966598+0000","last_became_active":"2023-10-16T09:43:21.940820+0000","last_became_peered":"2023-10-16T09:43:21.940820+0000","last_unstale":"2023-10-16T09:43:25.966598+0000","last_undegraded":"2023-10-16T09:43:25.966598+0000","last_fullsized":"2023-10-16T09:43:25.966598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947760+0000","last_change":"2023-10-16T09:43:25.970409+0000","last_active":"2023-10-16T09:43:26.947760+0000","last_peered":"2023-10-16T09:43:26.947760+0000","last_clean":"2023-10-16T09:43:26.947760+0000","last_became_active":"2023-10-16T09:43:25.969848+0000","last_became_peered":"2023-10-16T09:43:25.969848+0000","last_unstale":"2023-10-16T09:43:26.947760+0000","last_undegraded":"2023-10-16T09:43:26.947760+0000","last_fullsized":"2023-10-16T09:43:26.947760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:00.972576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966712+0000","last_change":"2023-10-16T09:43:20.933965+0000","last_active":"2023-10-16T09:43:25.966712+0000","last_peered":"2023-10-16T09:43:25.966712+0000","last_clean":"2023-10-16T09:43:25.966712+0000","last_became_active":"2023-10-16T09:43:20.933635+0000","last_became_peered":"2023-10-16T09:43:20.933635+0000","last_unstale":"2023-10-16T09:43:25.966712+0000","last_undegraded":"2023-10-16T09:43:25.966712+0000","last_fullsized":"2023-10-16T09:43:25.966712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951601+0000","last_change":"2023-10-16T09:43:21.937374+0000","last_active":"2023-10-16T09:43:25.951601+0000","last_peered":"2023-10-16T09:43:25.951601+0000","last_clean":"2023-10-16T09:43:25.951601+0000","last_became_active":"2023-10-16T09:43:21.936989+0000","last_became_peered":"2023-10-16T09:43:21.936989+0000","last_unstale":"2023-10-16T09:43:25.951601+0000","last_undegraded":"2023-10-16T09:43:25.951601+0000","last_fullsized":"2023-10-16T09:43:25.951601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948251+0000","last_change":"2023-10-16T09:43:25.970011+0000","last_active":"2023-10-16T09:43:26.948251+0000","last_peered":"2023-10-16T09:43:26.948251+0000","last_clean":"2023-10-16T09:43:26.948251+0000","last_became_active":"2023-10-16T09:43:25.969570+0000","last_became_peered":"2023-10-16T09:43:25.969570+0000","last_unstale":"2023-10-16T09:43:26.948251+0000","last_undegraded":"2023-10-16T09:43:26.948251+0000","last_fullsized":"2023-10-16T09:43:26.948251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944734+0000","last_change":"2023-10-16T09:43:20.928578+0000","last_active":"2023-10-16T09:43:25.944734+0000","last_peered":"2023-10-16T09:43:25.944734+0000","last_clean":"2023-10-16T09:43:25.944734+0000","last_became_active":"2023-10-16T09:43:20.928283+0000","last_became_peered":"2023-10-16T09:43:20.928283+0000","last_unstale":"2023-10-16T09:43:25.944734+0000","last_undegraded":"2023-10-16T09:43:25.944734+0000","last_fullsized":"2023-10-16T09:43:25.944734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950360+0000","last_change":"2023-10-16T09:43:21.936827+0000","last_active":"2023-10-16T09:43:25.950360+0000","last_peered":"2023-10-16T09:43:25.950360+0000","last_clean":"2023-10-16T09:43:25.950360+0000","last_became_active":"2023-10-16T09:43:21.936545+0000","last_became_peered":"2023-10-16T09:43:21.936545+0000","last_unstale":"2023-10-16T09:43:25.950360+0000","last_undegraded":"2023-10-16T09:43:25.950360+0000","last_fullsized":"2023-10-16T09:43:25.950360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947909+0000","last_change":"2023-10-16T09:43:25.968566+0000","last_active":"2023-10-16T09:43:26.947909+0000","last_peered":"2023-10-16T09:43:26.947909+0000","last_clean":"2023-10-16T09:43:26.947909+0000","last_became_active":"2023-10-16T09:43:25.968188+0000","last_became_peered":"2023-10-16T09:43:25.968188+0000","last_unstale":"2023-10-16T09:43:26.947909+0000","last_undegraded":"2023-10-16T09:43:26.947909+0000","last_fullsized":"2023-10-16T09:43:26.947909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:38.074113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944439+0000","last_change":"2023-10-16T09:43:20.929934+0000","last_active":"2023-10-16T09:43:25.944439+0000","last_peered":"2023-10-16T09:43:25.944439+0000","last_clean":"2023-10-16T09:43:25.944439+0000","last_became_active":"2023-10-16T09:43:20.929639+0000","last_became_peered":"2023-10-16T09:43:20.929639+0000","last_unstale":"2023-10-16T09:43:25.944439+0000","last_undegraded":"2023-10-16T09:43:25.944439+0000","last_fullsized":"2023-10-16T09:43:25.944439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953109+0000","last_change":"2023-10-16T09:43:21.933199+0000","last_active":"2023-10-16T09:43:25.953109+0000","last_peered":"2023-10-16T09:43:25.953109+0000","last_clean":"2023-10-16T09:43:25.953109+0000","last_became_active":"2023-10-16T09:43:21.932906+0000","last_became_peered":"2023-10-16T09:43:21.932906+0000","last_unstale":"2023-10-16T09:43:25.953109+0000","last_undegraded":"2023-10-16T09:43:25.953109+0000","last_fullsized":"2023-10-16T09:43:25.953109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016130+0000","last_change":"2023-10-16T09:43:25.975356+0000","last_active":"2023-10-16T09:43:28.016130+0000","last_peered":"2023-10-16T09:43:28.016130+0000","last_clean":"2023-10-16T09:43:28.016130+0000","last_became_active":"2023-10-16T09:43:25.974334+0000","last_became_peered":"2023-10-16T09:43:25.974334+0000","last_unstale":"2023-10-16T09:43:28.016130+0000","last_undegraded":"2023-10-16T09:43:28.016130+0000","last_fullsized":"2023-10-16T09:43:28.016130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:20.175649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966133+0000","last_change":"2023-10-16T09:43:20.928498+0000","last_active":"2023-10-16T09:43:25.966133+0000","last_peered":"2023-10-16T09:43:25.966133+0000","last_clean":"2023-10-16T09:43:25.966133+0000","last_became_active":"2023-10-16T09:43:20.928242+0000","last_became_peered":"2023-10-16T09:43:20.928242+0000","last_unstale":"2023-10-16T09:43:25.966133+0000","last_undegraded":"2023-10-16T09:43:25.966133+0000","last_fullsized":"2023-10-16T09:43:25.966133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948266+0000","last_change":"2023-10-16T09:43:21.919769+0000","last_active":"2023-10-16T09:43:25.948266+0000","last_peered":"2023-10-16T09:43:25.948266+0000","last_clean":"2023-10-16T09:43:25.948266+0000","last_became_active":"2023-10-16T09:43:21.919395+0000","last_became_peered":"2023-10-16T09:43:21.919395+0000","last_unstale":"2023-10-16T09:43:25.948266+0000","last_undegraded":"2023-10-16T09:43:25.948266+0000","last_fullsized":"2023-10-16T09:43:25.948266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:06.700888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.944846+0000","last_change":"2023-10-16T09:43:25.970755+0000","last_active":"2023-10-16T09:43:26.944846+0000","last_peered":"2023-10-16T09:43:26.944846+0000","last_clean":"2023-10-16T09:43:26.944846+0000","last_became_active":"2023-10-16T09:43:25.970344+0000","last_became_peered":"2023-10-16T09:43:25.970344+0000","last_unstale":"2023-10-16T09:43:26.944846+0000","last_undegraded":"2023-10-16T09:43:26.944846+0000","last_fullsized":"2023-10-16T09:43:26.944846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:28.937851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957011+0000","last_change":"2023-10-16T09:43:20.924380+0000","last_active":"2023-10-16T09:43:25.957011+0000","last_peered":"2023-10-16T09:43:25.957011+0000","last_clean":"2023-10-16T09:43:25.957011+0000","last_became_active":"2023-10-16T09:43:20.924104+0000","last_became_peered":"2023-10-16T09:43:20.924104+0000","last_unstale":"2023-10-16T09:43:25.957011+0000","last_undegraded":"2023-10-16T09:43:25.957011+0000","last_fullsized":"2023-10-16T09:43:25.957011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948179+0000","last_change":"2023-10-16T09:43:21.929870+0000","last_active":"2023-10-16T09:43:25.948179+0000","last_peered":"2023-10-16T09:43:25.948179+0000","last_clean":"2023-10-16T09:43:25.948179+0000","last_became_active":"2023-10-16T09:43:21.929589+0000","last_became_peered":"2023-10-16T09:43:21.929589+0000","last_unstale":"2023-10-16T09:43:25.948179+0000","last_undegraded":"2023-10-16T09:43:25.948179+0000","last_fullsized":"2023-10-16T09:43:25.948179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949711+0000","last_change":"2023-10-16T09:43:25.966670+0000","last_active":"2023-10-16T09:43:26.949711+0000","last_peered":"2023-10-16T09:43:26.949711+0000","last_clean":"2023-10-16T09:43:26.949711+0000","last_became_active":"2023-10-16T09:43:25.966312+0000","last_became_peered":"2023-10-16T09:43:25.966312+0000","last_unstale":"2023-10-16T09:43:26.949711+0000","last_undegraded":"2023-10-16T09:43:26.949711+0000","last_fullsized":"2023-10-16T09:43:26.949711+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965835+0000","last_change":"2023-10-16T09:43:20.928383+0000","last_active":"2023-10-16T09:43:25.965835+0000","last_peered":"2023-10-16T09:43:25.965835+0000","last_clean":"2023-10-16T09:43:25.965835+0000","last_became_active":"2023-10-16T09:43:20.928096+0000","last_became_peered":"2023-10-16T09:43:20.928096+0000","last_unstale":"2023-10-16T09:43:25.965835+0000","last_undegraded":"2023-10-16T09:43:25.965835+0000","last_fullsized":"2023-10-16T09:43:25.965835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965672+0000","last_change":"2023-10-16T09:43:21.921213+0000","last_active":"2023-10-16T09:43:25.965672+0000","last_peered":"2023-10-16T09:43:25.965672+0000","last_clean":"2023-10-16T09:43:25.965672+0000","last_became_active":"2023-10-16T09:43:21.920920+0000","last_became_peered":"2023-10-16T09:43:21.920920+0000","last_unstale":"2023-10-16T09:43:25.965672+0000","last_undegraded":"2023-10-16T09:43:25.965672+0000","last_fullsized":"2023-10-16T09:43:25.965672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015970+0000","last_change":"2023-10-16T09:43:25.972702+0000","last_active":"2023-10-16T09:43:28.015970+0000","last_peered":"2023-10-16T09:43:28.015970+0000","last_clean":"2023-10-16T09:43:28.015970+0000","last_became_active":"2023-10-16T09:43:25.971784+0000","last_became_peered":"2023-10-16T09:43:25.971784+0000","last_unstale":"2023-10-16T09:43:28.015970+0000","last_undegraded":"2023-10-16T09:43:28.015970+0000","last_fullsized":"2023-10-16T09:43:28.015970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950235+0000","last_change":"2023-10-16T09:43:20.926495+0000","last_active":"2023-10-16T09:43:25.950235+0000","last_peered":"2023-10-16T09:43:25.950235+0000","last_clean":"2023-10-16T09:43:25.950235+0000","last_became_active":"2023-10-16T09:43:20.925998+0000","last_became_peered":"2023-10-16T09:43:20.925998+0000","last_unstale":"2023-10-16T09:43:25.950235+0000","last_undegraded":"2023-10-16T09:43:25.950235+0000","last_fullsized":"2023-10-16T09:43:25.950235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954132+0000","last_change":"2023-10-16T09:43:21.919652+0000","last_active":"2023-10-16T09:43:25.954132+0000","last_peered":"2023-10-16T09:43:25.954132+0000","last_clean":"2023-10-16T09:43:25.954132+0000","last_became_active":"2023-10-16T09:43:21.919397+0000","last_became_peered":"2023-10-16T09:43:21.919397+0000","last_unstale":"2023-10-16T09:43:25.954132+0000","last_undegraded":"2023-10-16T09:43:25.954132+0000","last_fullsized":"2023-10-16T09:43:25.954132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015120+0000","last_change":"2023-10-16T09:43:25.975457+0000","last_active":"2023-10-16T09:43:28.015120+0000","last_peered":"2023-10-16T09:43:28.015120+0000","last_clean":"2023-10-16T09:43:28.015120+0000","last_became_active":"2023-10-16T09:43:25.975098+0000","last_became_peered":"2023-10-16T09:43:25.975098+0000","last_unstale":"2023-10-16T09:43:28.015120+0000","last_undegraded":"2023-10-16T09:43:28.015120+0000","last_fullsized":"2023-10-16T09:43:28.015120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956270+0000","last_change":"2023-10-16T09:43:20.925087+0000","last_active":"2023-10-16T09:43:25.956270+0000","last_peered":"2023-10-16T09:43:25.956270+0000","last_clean":"2023-10-16T09:43:25.956270+0000","last_became_active":"2023-10-16T09:43:20.924798+0000","last_became_peered":"2023-10-16T09:43:20.924798+0000","last_unstale":"2023-10-16T09:43:25.956270+0000","last_undegraded":"2023-10-16T09:43:25.956270+0000","last_fullsized":"2023-10-16T09:43:25.956270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951125+0000","last_change":"2023-10-16T09:43:21.933170+0000","last_active":"2023-10-16T09:43:25.951125+0000","last_peered":"2023-10-16T09:43:25.951125+0000","last_clean":"2023-10-16T09:43:25.951125+0000","last_became_active":"2023-10-16T09:43:21.932876+0000","last_became_peered":"2023-10-16T09:43:21.932876+0000","last_unstale":"2023-10-16T09:43:25.951125+0000","last_undegraded":"2023-10-16T09:43:25.951125+0000","last_fullsized":"2023-10-16T09:43:25.951125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015261+0000","last_change":"2023-10-16T09:43:25.979174+0000","last_active":"2023-10-16T09:43:28.015261+0000","last_peered":"2023-10-16T09:43:28.015261+0000","last_clean":"2023-10-16T09:43:28.015261+0000","last_became_active":"2023-10-16T09:43:25.977030+0000","last_became_peered":"2023-10-16T09:43:25.977030+0000","last_unstale":"2023-10-16T09:43:28.015261+0000","last_undegraded":"2023-10-16T09:43:28.015261+0000","last_fullsized":"2023-10-16T09:43:28.015261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:47.151340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951034+0000","last_change":"2023-10-16T09:43:20.928828+0000","last_active":"2023-10-16T09:43:25.951034+0000","last_peered":"2023-10-16T09:43:25.951034+0000","last_clean":"2023-10-16T09:43:25.951034+0000","last_became_active":"2023-10-16T09:43:20.928550+0000","last_became_peered":"2023-10-16T09:43:20.928550+0000","last_unstale":"2023-10-16T09:43:25.951034+0000","last_undegraded":"2023-10-16T09:43:25.951034+0000","last_fullsized":"2023-10-16T09:43:25.951034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951780+0000","last_change":"2023-10-16T09:43:21.937614+0000","last_active":"2023-10-16T09:43:25.951780+0000","last_peered":"2023-10-16T09:43:25.951780+0000","last_clean":"2023-10-16T09:43:25.951780+0000","last_became_active":"2023-10-16T09:43:21.937335+0000","last_became_peered":"2023-10-16T09:43:21.937335+0000","last_unstale":"2023-10-16T09:43:25.951780+0000","last_undegraded":"2023-10-16T09:43:25.951780+0000","last_fullsized":"2023-10-16T09:43:25.951780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948709+0000","last_change":"2023-10-16T09:43:25.958049+0000","last_active":"2023-10-16T09:43:26.948709+0000","last_peered":"2023-10-16T09:43:26.948709+0000","last_clean":"2023-10-16T09:43:26.948709+0000","last_became_active":"2023-10-16T09:43:25.957672+0000","last_became_peered":"2023-10-16T09:43:25.957672+0000","last_unstale":"2023-10-16T09:43:26.948709+0000","last_undegraded":"2023-10-16T09:43:26.948709+0000","last_fullsized":"2023-10-16T09:43:26.948709+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:53.105902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955629+0000","last_change":"2023-10-16T09:43:20.926834+0000","last_active":"2023-10-16T09:43:25.955629+0000","last_peered":"2023-10-16T09:43:25.955629+0000","last_clean":"2023-10-16T09:43:25.955629+0000","last_became_active":"2023-10-16T09:43:20.926593+0000","last_became_peered":"2023-10-16T09:43:20.926593+0000","last_unstale":"2023-10-16T09:43:25.955629+0000","last_undegraded":"2023-10-16T09:43:25.955629+0000","last_fullsized":"2023-10-16T09:43:25.955629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947243+0000","last_change":"2023-10-16T09:43:21.925181+0000","last_active":"2023-10-16T09:43:25.947243+0000","last_peered":"2023-10-16T09:43:25.947243+0000","last_clean":"2023-10-16T09:43:25.947243+0000","last_became_active":"2023-10-16T09:43:21.924083+0000","last_became_peered":"2023-10-16T09:43:21.924083+0000","last_unstale":"2023-10-16T09:43:25.947243+0000","last_undegraded":"2023-10-16T09:43:25.947243+0000","last_fullsized":"2023-10-16T09:43:25.947243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:41.369421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995287+0000","last_change":"2023-10-16T09:43:25.960604+0000","last_active":"2023-10-16T09:43:27.995287+0000","last_peered":"2023-10-16T09:43:27.995287+0000","last_clean":"2023-10-16T09:43:27.995287+0000","last_became_active":"2023-10-16T09:43:25.960212+0000","last_became_peered":"2023-10-16T09:43:25.960212+0000","last_unstale":"2023-10-16T09:43:27.995287+0000","last_undegraded":"2023-10-16T09:43:27.995287+0000","last_fullsized":"2023-10-16T09:43:27.995287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:31.688570+0000","last_change":"2023-10-16T09:43:20.927651+0000","last_active":"2023-10-16T09:43:31.688570+0000","last_peered":"2023-10-16T09:43:31.688570+0000","last_clean":"2023-10-16T09:43:31.688570+0000","last_became_active":"2023-10-16T09:43:20.927393+0000","last_became_peered":"2023-10-16T09:43:20.927393+0000","last_unstale":"2023-10-16T09:43:31.688570+0000","last_undegraded":"2023-10-16T09:43:31.688570+0000","last_fullsized":"2023-10-16T09:43:31.688570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944933+0000","last_change":"2023-10-16T09:43:21.922547+0000","last_active":"2023-10-16T09:43:25.944933+0000","last_peered":"2023-10-16T09:43:25.944933+0000","last_clean":"2023-10-16T09:43:25.944933+0000","last_became_active":"2023-10-16T09:43:21.922309+0000","last_became_peered":"2023-10-16T09:43:21.922309+0000","last_unstale":"2023-10-16T09:43:25.944933+0000","last_undegraded":"2023-10-16T09:43:25.944933+0000","last_fullsized":"2023-10-16T09:43:25.944933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016142+0000","last_change":"2023-10-16T09:43:25.973055+0000","last_active":"2023-10-16T09:43:28.016142+0000","last_peered":"2023-10-16T09:43:28.016142+0000","last_clean":"2023-10-16T09:43:28.016142+0000","last_became_active":"2023-10-16T09:43:25.972664+0000","last_became_peered":"2023-10-16T09:43:25.972664+0000","last_unstale":"2023-10-16T09:43:28.016142+0000","last_undegraded":"2023-10-16T09:43:28.016142+0000","last_fullsized":"2023-10-16T09:43:28.016142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948288+0000","last_change":"2023-10-16T09:43:20.933659+0000","last_active":"2023-10-16T09:43:25.948288+0000","last_peered":"2023-10-16T09:43:25.948288+0000","last_clean":"2023-10-16T09:43:25.948288+0000","last_became_active":"2023-10-16T09:43:20.933376+0000","last_became_peered":"2023-10-16T09:43:20.933376+0000","last_unstale":"2023-10-16T09:43:25.948288+0000","last_undegraded":"2023-10-16T09:43:25.948288+0000","last_fullsized":"2023-10-16T09:43:25.948288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948438+0000","last_change":"2023-10-16T09:43:21.925525+0000","last_active":"2023-10-16T09:43:25.948438+0000","last_peered":"2023-10-16T09:43:25.948438+0000","last_clean":"2023-10-16T09:43:25.948438+0000","last_became_active":"2023-10-16T09:43:21.925269+0000","last_became_peered":"2023-10-16T09:43:21.925269+0000","last_unstale":"2023-10-16T09:43:25.948438+0000","last_undegraded":"2023-10-16T09:43:25.948438+0000","last_fullsized":"2023-10-16T09:43:25.948438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:39.205632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016287+0000","last_change":"2023-10-16T09:43:25.974917+0000","last_active":"2023-10-16T09:43:28.016287+0000","last_peered":"2023-10-16T09:43:28.016287+0000","last_clean":"2023-10-16T09:43:28.016287+0000","last_became_active":"2023-10-16T09:43:25.974512+0000","last_became_peered":"2023-10-16T09:43:25.974512+0000","last_unstale":"2023-10-16T09:43:28.016287+0000","last_undegraded":"2023-10-16T09:43:28.016287+0000","last_fullsized":"2023-10-16T09:43:28.016287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:24.186922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950390+0000","last_change":"2023-10-16T09:43:20.927369+0000","last_active":"2023-10-16T09:43:25.950390+0000","last_peered":"2023-10-16T09:43:25.950390+0000","last_clean":"2023-10-16T09:43:25.950390+0000","last_became_active":"2023-10-16T09:43:20.927043+0000","last_became_peered":"2023-10-16T09:43:20.927043+0000","last_unstale":"2023-10-16T09:43:25.950390+0000","last_undegraded":"2023-10-16T09:43:25.950390+0000","last_fullsized":"2023-10-16T09:43:25.950390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:11.929994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945029+0000","last_change":"2023-10-16T09:43:21.930235+0000","last_active":"2023-10-16T09:43:25.945029+0000","last_peered":"2023-10-16T09:43:25.945029+0000","last_clean":"2023-10-16T09:43:25.945029+0000","last_became_active":"2023-10-16T09:43:21.929925+0000","last_became_peered":"2023-10-16T09:43:21.929925+0000","last_unstale":"2023-10-16T09:43:25.945029+0000","last_undegraded":"2023-10-16T09:43:25.945029+0000","last_fullsized":"2023-10-16T09:43:25.945029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016497+0000","last_change":"2023-10-16T09:43:25.975868+0000","last_active":"2023-10-16T09:43:28.016497+0000","last_peered":"2023-10-16T09:43:28.016497+0000","last_clean":"2023-10-16T09:43:28.016497+0000","last_became_active":"2023-10-16T09:43:25.975476+0000","last_became_peered":"2023-10-16T09:43:25.975476+0000","last_unstale":"2023-10-16T09:43:28.016497+0000","last_undegraded":"2023-10-16T09:43:28.016497+0000","last_fullsized":"2023-10-16T09:43:28.016497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:36.344621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954142+0000","last_change":"2023-10-16T09:43:20.922948+0000","last_active":"2023-10-16T09:43:25.954142+0000","last_peered":"2023-10-16T09:43:25.954142+0000","last_clean":"2023-10-16T09:43:25.954142+0000","last_became_active":"2023-10-16T09:43:20.922660+0000","last_became_peered":"2023-10-16T09:43:20.922660+0000","last_unstale":"2023-10-16T09:43:25.954142+0000","last_undegraded":"2023-10-16T09:43:25.954142+0000","last_fullsized":"2023-10-16T09:43:25.954142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:24.786762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953944+0000","last_change":"2023-10-16T09:43:21.930450+0000","last_active":"2023-10-16T09:43:25.953944+0000","last_peered":"2023-10-16T09:43:25.953944+0000","last_clean":"2023-10-16T09:43:25.953944+0000","last_became_active":"2023-10-16T09:43:21.929848+0000","last_became_peered":"2023-10-16T09:43:21.929848+0000","last_unstale":"2023-10-16T09:43:25.953944+0000","last_undegraded":"2023-10-16T09:43:25.953944+0000","last_fullsized":"2023-10-16T09:43:25.953944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015695+0000","last_change":"2023-10-16T09:43:25.972951+0000","last_active":"2023-10-16T09:43:28.015695+0000","last_peered":"2023-10-16T09:43:28.015695+0000","last_clean":"2023-10-16T09:43:28.015695+0000","last_became_active":"2023-10-16T09:43:25.972321+0000","last_became_peered":"2023-10-16T09:43:25.972321+0000","last_unstale":"2023-10-16T09:43:28.015695+0000","last_undegraded":"2023-10-16T09:43:28.015695+0000","last_fullsized":"2023-10-16T09:43:28.015695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948561+0000","last_change":"2023-10-16T09:43:20.919381+0000","last_active":"2023-10-16T09:43:25.948561+0000","last_peered":"2023-10-16T09:43:25.948561+0000","last_clean":"2023-10-16T09:43:25.948561+0000","last_became_active":"2023-10-16T09:43:20.919099+0000","last_became_peered":"2023-10-16T09:43:20.919099+0000","last_unstale":"2023-10-16T09:43:25.948561+0000","last_undegraded":"2023-10-16T09:43:25.948561+0000","last_fullsized":"2023-10-16T09:43:25.948561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.153559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944510+0000","last_change":"2023-10-16T09:43:21.930254+0000","last_active":"2023-10-16T09:43:25.944510+0000","last_peered":"2023-10-16T09:43:25.944510+0000","last_clean":"2023-10-16T09:43:25.944510+0000","last_became_active":"2023-10-16T09:43:21.929903+0000","last_became_peered":"2023-10-16T09:43:21.929903+0000","last_unstale":"2023-10-16T09:43:25.944510+0000","last_undegraded":"2023-10-16T09:43:25.944510+0000","last_fullsized":"2023-10-16T09:43:25.944510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947704+0000","last_change":"2023-10-16T09:43:25.973595+0000","last_active":"2023-10-16T09:43:26.947704+0000","last_peered":"2023-10-16T09:43:26.947704+0000","last_clean":"2023-10-16T09:43:26.947704+0000","last_became_active":"2023-10-16T09:43:25.972877+0000","last_became_peered":"2023-10-16T09:43:25.972877+0000","last_unstale":"2023-10-16T09:43:26.947704+0000","last_undegraded":"2023-10-16T09:43:26.947704+0000","last_fullsized":"2023-10-16T09:43:26.947704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:59.042186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.946744+0000","last_change":"2023-10-16T09:43:20.933953+0000","last_active":"2023-10-16T09:43:25.946744+0000","last_peered":"2023-10-16T09:43:25.946744+0000","last_clean":"2023-10-16T09:43:25.946744+0000","last_became_active":"2023-10-16T09:43:20.933628+0000","last_became_peered":"2023-10-16T09:43:20.933628+0000","last_unstale":"2023-10-16T09:43:25.946744+0000","last_undegraded":"2023-10-16T09:43:25.946744+0000","last_fullsized":"2023-10-16T09:43:25.946744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:06.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944812+0000","last_change":"2023-10-16T09:43:21.922710+0000","last_active":"2023-10-16T09:43:25.944812+0000","last_peered":"2023-10-16T09:43:25.944812+0000","last_clean":"2023-10-16T09:43:25.944812+0000","last_became_active":"2023-10-16T09:43:21.921824+0000","last_became_peered":"2023-10-16T09:43:21.921824+0000","last_unstale":"2023-10-16T09:43:25.944812+0000","last_undegraded":"2023-10-16T09:43:25.944812+0000","last_fullsized":"2023-10-16T09:43:25.944812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995171+0000","last_change":"2023-10-16T09:43:25.960646+0000","last_active":"2023-10-16T09:43:27.995171+0000","last_peered":"2023-10-16T09:43:27.995171+0000","last_clean":"2023-10-16T09:43:27.995171+0000","last_became_active":"2023-10-16T09:43:25.960270+0000","last_became_peered":"2023-10-16T09:43:25.960270+0000","last_unstale":"2023-10-16T09:43:27.995171+0000","last_undegraded":"2023-10-16T09:43:27.995171+0000","last_fullsized":"2023-10-16T09:43:27.995171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955611+0000","last_change":"2023-10-16T09:43:20.935197+0000","last_active":"2023-10-16T09:43:25.955611+0000","last_peered":"2023-10-16T09:43:25.955611+0000","last_clean":"2023-10-16T09:43:25.955611+0000","last_became_active":"2023-10-16T09:43:20.934929+0000","last_became_peered":"2023-10-16T09:43:20.934929+0000","last_unstale":"2023-10-16T09:43:25.955611+0000","last_undegraded":"2023-10-16T09:43:25.955611+0000","last_fullsized":"2023-10-16T09:43:25.955611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953179+0000","last_change":"2023-10-16T09:43:21.936959+0000","last_active":"2023-10-16T09:43:25.953179+0000","last_peered":"2023-10-16T09:43:25.953179+0000","last_clean":"2023-10-16T09:43:25.953179+0000","last_became_active":"2023-10-16T09:43:21.932598+0000","last_became_peered":"2023-10-16T09:43:21.932598+0000","last_unstale":"2023-10-16T09:43:25.953179+0000","last_undegraded":"2023-10-16T09:43:25.953179+0000","last_fullsized":"2023-10-16T09:43:25.953179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995359+0000","last_change":"2023-10-16T09:43:25.976441+0000","last_active":"2023-10-16T09:43:27.995359+0000","last_peered":"2023-10-16T09:43:27.995359+0000","last_clean":"2023-10-16T09:43:27.995359+0000","last_became_active":"2023-10-16T09:43:25.975914+0000","last_became_peered":"2023-10-16T09:43:25.975914+0000","last_unstale":"2023-10-16T09:43:27.995359+0000","last_undegraded":"2023-10-16T09:43:27.995359+0000","last_fullsized":"2023-10-16T09:43:27.995359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950296+0000","last_change":"2023-10-16T09:43:20.925613+0000","last_active":"2023-10-16T09:43:25.950296+0000","last_peered":"2023-10-16T09:43:25.950296+0000","last_clean":"2023-10-16T09:43:25.950296+0000","last_became_active":"2023-10-16T09:43:20.925345+0000","last_became_peered":"2023-10-16T09:43:20.925345+0000","last_unstale":"2023-10-16T09:43:25.950296+0000","last_undegraded":"2023-10-16T09:43:25.950296+0000","last_fullsized":"2023-10-16T09:43:25.950296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:28.608301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948410+0000","last_change":"2023-10-16T09:43:21.925702+0000","last_active":"2023-10-16T09:43:25.948410+0000","last_peered":"2023-10-16T09:43:25.948410+0000","last_clean":"2023-10-16T09:43:25.948410+0000","last_became_active":"2023-10-16T09:43:21.925003+0000","last_became_peered":"2023-10-16T09:43:21.925003+0000","last_unstale":"2023-10-16T09:43:25.948410+0000","last_undegraded":"2023-10-16T09:43:25.948410+0000","last_fullsized":"2023-10-16T09:43:25.948410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995813+0000","last_change":"2023-10-16T09:43:25.976511+0000","last_active":"2023-10-16T09:43:27.995813+0000","last_peered":"2023-10-16T09:43:27.995813+0000","last_clean":"2023-10-16T09:43:27.995813+0000","last_became_active":"2023-10-16T09:43:25.975258+0000","last_became_peered":"2023-10-16T09:43:25.975258+0000","last_unstale":"2023-10-16T09:43:27.995813+0000","last_undegraded":"2023-10-16T09:43:27.995813+0000","last_fullsized":"2023-10-16T09:43:27.995813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953939+0000","last_change":"2023-10-16T09:43:21.924235+0000","last_active":"2023-10-16T09:43:25.953939+0000","last_peered":"2023-10-16T09:43:25.953939+0000","last_clean":"2023-10-16T09:43:25.953939+0000","last_became_active":"2023-10-16T09:43:21.923963+0000","last_became_peered":"2023-10-16T09:43:21.923963+0000","last_unstale":"2023-10-16T09:43:25.953939+0000","last_undegraded":"2023-10-16T09:43:25.953939+0000","last_fullsized":"2023-10-16T09:43:25.953939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948171+0000","last_change":"2023-10-16T09:43:20.922462+0000","last_active":"2023-10-16T09:43:25.948171+0000","last_peered":"2023-10-16T09:43:25.948171+0000","last_clean":"2023-10-16T09:43:25.948171+0000","last_became_active":"2023-10-16T09:43:20.922204+0000","last_became_peered":"2023-10-16T09:43:20.922204+0000","last_unstale":"2023-10-16T09:43:25.948171+0000","last_undegraded":"2023-10-16T09:43:25.948171+0000","last_fullsized":"2023-10-16T09:43:25.948171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949005+0000","last_change":"2023-10-16T09:43:25.975066+0000","last_active":"2023-10-16T09:43:26.949005+0000","last_peered":"2023-10-16T09:43:26.949005+0000","last_clean":"2023-10-16T09:43:26.949005+0000","last_became_active":"2023-10-16T09:43:25.974685+0000","last_became_peered":"2023-10-16T09:43:25.974685+0000","last_unstale":"2023-10-16T09:43:26.949005+0000","last_undegraded":"2023-10-16T09:43:26.949005+0000","last_fullsized":"2023-10-16T09:43:26.949005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966726+0000","last_change":"2023-10-16T09:43:21.940037+0000","last_active":"2023-10-16T09:43:25.966726+0000","last_peered":"2023-10-16T09:43:25.966726+0000","last_clean":"2023-10-16T09:43:25.966726+0000","last_became_active":"2023-10-16T09:43:21.939645+0000","last_became_peered":"2023-10-16T09:43:21.939645+0000","last_unstale":"2023-10-16T09:43:25.966726+0000","last_undegraded":"2023-10-16T09:43:25.966726+0000","last_fullsized":"2023-10-16T09:43:25.966726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950263+0000","last_change":"2023-10-16T09:43:20.925442+0000","last_active":"2023-10-16T09:43:25.950263+0000","last_peered":"2023-10-16T09:43:25.950263+0000","last_clean":"2023-10-16T09:43:25.950263+0000","last_became_active":"2023-10-16T09:43:20.925129+0000","last_became_peered":"2023-10-16T09:43:20.925129+0000","last_unstale":"2023-10-16T09:43:25.950263+0000","last_undegraded":"2023-10-16T09:43:25.950263+0000","last_fullsized":"2023-10-16T09:43:25.950263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015775+0000","last_change":"2023-10-16T09:43:25.961486+0000","last_active":"2023-10-16T09:43:28.015775+0000","last_peered":"2023-10-16T09:43:28.015775+0000","last_clean":"2023-10-16T09:43:28.015775+0000","last_became_active":"2023-10-16T09:43:25.961120+0000","last_became_peered":"2023-10-16T09:43:25.961120+0000","last_unstale":"2023-10-16T09:43:28.015775+0000","last_undegraded":"2023-10-16T09:43:28.015775+0000","last_fullsized":"2023-10-16T09:43:28.015775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947444+0000","last_change":"2023-10-16T09:43:21.929303+0000","last_active":"2023-10-16T09:43:25.947444+0000","last_peered":"2023-10-16T09:43:25.947444+0000","last_clean":"2023-10-16T09:43:25.947444+0000","last_became_active":"2023-10-16T09:43:21.929017+0000","last_became_peered":"2023-10-16T09:43:21.929017+0000","last_unstale":"2023-10-16T09:43:25.947444+0000","last_undegraded":"2023-10-16T09:43:25.947444+0000","last_fullsized":"2023-10-16T09:43:25.947444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:45.433460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:20.925419+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:20.924900+0000","last_became_peered":"2023-10-16T09:43:20.924900+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.960836+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.959879+0000","last_became_peered":"2023-10-16T09:43:25.959879+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944657+0000","last_change":"2023-10-16T09:43:20.935065+0000","last_active":"2023-10-16T09:43:25.944657+0000","last_peered":"2023-10-16T09:43:25.944657+0000","last_clean":"2023-10-16T09:43:25.944657+0000","last_became_active":"2023-10-16T09:43:20.934870+0000","last_became_peered":"2023-10-16T09:43:20.934870+0000","last_unstale":"2023-10-16T09:43:25.944657+0000","last_undegraded":"2023-10-16T09:43:25.944657+0000","last_fullsized":"2023-10-16T09:43:25.944657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955720+0000","last_change":"2023-10-16T09:43:21.924250+0000","last_active":"2023-10-16T09:43:25.955720+0000","last_peered":"2023-10-16T09:43:25.955720+0000","last_clean":"2023-10-16T09:43:25.955720+0000","last_became_active":"2023-10-16T09:43:21.923974+0000","last_became_peered":"2023-10-16T09:43:21.923974+0000","last_unstale":"2023-10-16T09:43:25.955720+0000","last_undegraded":"2023-10-16T09:43:25.955720+0000","last_fullsized":"2023-10-16T09:43:25.955720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.974974+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.974574+0000","last_became_peered":"2023-10-16T09:43:25.974574+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955603+0000","last_change":"2023-10-16T09:43:20.928340+0000","last_active":"2023-10-16T09:43:25.955603+0000","last_peered":"2023-10-16T09:43:25.955603+0000","last_clean":"2023-10-16T09:43:25.955603+0000","last_became_active":"2023-10-16T09:43:20.927748+0000","last_became_peered":"2023-10-16T09:43:20.927748+0000","last_unstale":"2023-10-16T09:43:25.955603+0000","last_undegraded":"2023-10-16T09:43:25.955603+0000","last_fullsized":"2023-10-16T09:43:25.955603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967050+0000","last_change":"2023-10-16T09:43:21.919707+0000","last_active":"2023-10-16T09:43:25.967050+0000","last_peered":"2023-10-16T09:43:25.967050+0000","last_clean":"2023-10-16T09:43:25.967050+0000","last_became_active":"2023-10-16T09:43:21.919454+0000","last_became_peered":"2023-10-16T09:43:21.919454+0000","last_unstale":"2023-10-16T09:43:25.967050+0000","last_undegraded":"2023-10-16T09:43:25.967050+0000","last_fullsized":"2023-10-16T09:43:25.967050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947855+0000","last_change":"2023-10-16T09:43:25.961225+0000","last_active":"2023-10-16T09:43:26.947855+0000","last_peered":"2023-10-16T09:43:26.947855+0000","last_clean":"2023-10-16T09:43:26.947855+0000","last_became_active":"2023-10-16T09:43:25.960336+0000","last_became_peered":"2023-10-16T09:43:25.960336+0000","last_unstale":"2023-10-16T09:43:26.947855+0000","last_undegraded":"2023-10-16T09:43:26.947855+0000","last_fullsized":"2023-10-16T09:43:26.947855+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:15.617593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956170+0000","last_change":"2023-10-16T09:43:20.928799+0000","last_active":"2023-10-16T09:43:25.956170+0000","last_peered":"2023-10-16T09:43:25.956170+0000","last_clean":"2023-10-16T09:43:25.956170+0000","last_became_active":"2023-10-16T09:43:20.928302+0000","last_became_peered":"2023-10-16T09:43:20.928302+0000","last_unstale":"2023-10-16T09:43:25.956170+0000","last_undegraded":"2023-10-16T09:43:25.956170+0000","last_fullsized":"2023-10-16T09:43:25.956170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951319+0000","last_change":"2023-10-16T09:43:21.924602+0000","last_active":"2023-10-16T09:43:25.951319+0000","last_peered":"2023-10-16T09:43:25.951319+0000","last_clean":"2023-10-16T09:43:25.951319+0000","last_became_active":"2023-10-16T09:43:21.924344+0000","last_became_peered":"2023-10-16T09:43:21.924344+0000","last_unstale":"2023-10-16T09:43:25.951319+0000","last_undegraded":"2023-10-16T09:43:25.951319+0000","last_fullsized":"2023-10-16T09:43:25.951319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948047+0000","last_change":"2023-10-16T09:43:25.969751+0000","last_active":"2023-10-16T09:43:26.948047+0000","last_peered":"2023-10-16T09:43:26.948047+0000","last_clean":"2023-10-16T09:43:26.948047+0000","last_became_active":"2023-10-16T09:43:25.969399+0000","last_became_peered":"2023-10-16T09:43:25.969399+0000","last_unstale":"2023-10-16T09:43:26.948047+0000","last_undegraded":"2023-10-16T09:43:26.948047+0000","last_fullsized":"2023-10-16T09:43:26.948047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:20.933882+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:20.933615+0000","last_became_peered":"2023-10-16T09:43:20.933615+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947408+0000","last_change":"2023-10-16T09:43:21.929135+0000","last_active":"2023-10-16T09:43:25.947408+0000","last_peered":"2023-10-16T09:43:25.947408+0000","last_clean":"2023-10-16T09:43:25.947408+0000","last_became_active":"2023-10-16T09:43:21.928849+0000","last_became_peered":"2023-10-16T09:43:21.928849+0000","last_unstale":"2023-10-16T09:43:25.947408+0000","last_undegraded":"2023-10-16T09:43:25.947408+0000","last_fullsized":"2023-10-16T09:43:25.947408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995596+0000","last_change":"2023-10-16T09:43:25.976682+0000","last_active":"2023-10-16T09:43:27.995596+0000","last_peered":"2023-10-16T09:43:27.995596+0000","last_clean":"2023-10-16T09:43:27.995596+0000","last_became_active":"2023-10-16T09:43:25.976274+0000","last_became_peered":"2023-10-16T09:43:25.976274+0000","last_unstale":"2023-10-16T09:43:27.995596+0000","last_undegraded":"2023-10-16T09:43:27.995596+0000","last_fullsized":"2023-10-16T09:43:27.995596+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954044+0000","last_change":"2023-10-16T09:43:20.926509+0000","last_active":"2023-10-16T09:43:25.954044+0000","last_peered":"2023-10-16T09:43:25.954044+0000","last_clean":"2023-10-16T09:43:25.954044+0000","last_became_active":"2023-10-16T09:43:20.926228+0000","last_became_peered":"2023-10-16T09:43:20.926228+0000","last_unstale":"2023-10-16T09:43:25.954044+0000","last_undegraded":"2023-10-16T09:43:25.954044+0000","last_fullsized":"2023-10-16T09:43:25.954044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965799+0000","last_change":"2023-10-16T09:43:21.924014+0000","last_active":"2023-10-16T09:43:25.965799+0000","last_peered":"2023-10-16T09:43:25.965799+0000","last_clean":"2023-10-16T09:43:25.965799+0000","last_became_active":"2023-10-16T09:43:21.922684+0000","last_became_peered":"2023-10-16T09:43:21.922684+0000","last_unstale":"2023-10-16T09:43:25.965799+0000","last_undegraded":"2023-10-16T09:43:25.965799+0000","last_fullsized":"2023-10-16T09:43:25.965799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948333+0000","last_change":"2023-10-16T09:43:25.970563+0000","last_active":"2023-10-16T09:43:26.948333+0000","last_peered":"2023-10-16T09:43:26.948333+0000","last_clean":"2023-10-16T09:43:26.948333+0000","last_became_active":"2023-10-16T09:43:25.970168+0000","last_became_peered":"2023-10-16T09:43:25.970168+0000","last_unstale":"2023-10-16T09:43:26.948333+0000","last_undegraded":"2023-10-16T09:43:26.948333+0000","last_fullsized":"2023-10-16T09:43:26.948333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:56.567461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953937+0000","last_change":"2023-10-16T09:43:20.923587+0000","last_active":"2023-10-16T09:43:25.953937+0000","last_peered":"2023-10-16T09:43:25.953937+0000","last_clean":"2023-10-16T09:43:25.953937+0000","last_became_active":"2023-10-16T09:43:20.923277+0000","last_became_peered":"2023-10-16T09:43:20.923277+0000","last_unstale":"2023-10-16T09:43:25.953937+0000","last_undegraded":"2023-10-16T09:43:25.953937+0000","last_fullsized":"2023-10-16T09:43:25.953937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966226+0000","last_change":"2023-10-16T09:43:21.925711+0000","last_active":"2023-10-16T09:43:25.966226+0000","last_peered":"2023-10-16T09:43:25.966226+0000","last_clean":"2023-10-16T09:43:25.966226+0000","last_became_active":"2023-10-16T09:43:21.921509+0000","last_became_peered":"2023-10-16T09:43:21.921509+0000","last_unstale":"2023-10-16T09:43:25.966226+0000","last_undegraded":"2023-10-16T09:43:25.966226+0000","last_fullsized":"2023-10-16T09:43:25.966226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948182+0000","last_change":"2023-10-16T09:43:25.955876+0000","last_active":"2023-10-16T09:43:26.948182+0000","last_peered":"2023-10-16T09:43:26.948182+0000","last_clean":"2023-10-16T09:43:26.948182+0000","last_became_active":"2023-10-16T09:43:25.955490+0000","last_became_peered":"2023-10-16T09:43:25.955490+0000","last_unstale":"2023-10-16T09:43:26.948182+0000","last_undegraded":"2023-10-16T09:43:26.948182+0000","last_fullsized":"2023-10-16T09:43:26.948182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945113+0000","last_change":"2023-10-16T09:43:20.926395+0000","last_active":"2023-10-16T09:43:25.945113+0000","last_peered":"2023-10-16T09:43:25.945113+0000","last_clean":"2023-10-16T09:43:25.945113+0000","last_became_active":"2023-10-16T09:43:20.926086+0000","last_became_peered":"2023-10-16T09:43:20.926086+0000","last_unstale":"2023-10-16T09:43:25.945113+0000","last_undegraded":"2023-10-16T09:43:25.945113+0000","last_fullsized":"2023-10-16T09:43:25.945113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:56.924185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957128+0000","last_change":"2023-10-16T09:43:21.932460+0000","last_active":"2023-10-16T09:43:25.957128+0000","last_peered":"2023-10-16T09:43:25.957128+0000","last_clean":"2023-10-16T09:43:25.957128+0000","last_became_active":"2023-10-16T09:43:21.932184+0000","last_became_peered":"2023-10-16T09:43:21.932184+0000","last_unstale":"2023-10-16T09:43:25.957128+0000","last_undegraded":"2023-10-16T09:43:25.957128+0000","last_fullsized":"2023-10-16T09:43:25.957128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948895+0000","last_change":"2023-10-16T09:43:25.967183+0000","last_active":"2023-10-16T09:43:26.948895+0000","last_peered":"2023-10-16T09:43:26.948895+0000","last_clean":"2023-10-16T09:43:26.948895+0000","last_became_active":"2023-10-16T09:43:25.966759+0000","last_became_peered":"2023-10-16T09:43:25.966759+0000","last_unstale":"2023-10-16T09:43:26.948895+0000","last_undegraded":"2023-10-16T09:43:26.948895+0000","last_fullsized":"2023-10-16T09:43:26.948895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948721+0000","last_change":"2023-10-16T09:43:20.925234+0000","last_active":"2023-10-16T09:43:25.948721+0000","last_peered":"2023-10-16T09:43:25.948721+0000","last_clean":"2023-10-16T09:43:25.948721+0000","last_became_active":"2023-10-16T09:43:20.924957+0000","last_became_peered":"2023-10-16T09:43:20.924957+0000","last_unstale":"2023-10-16T09:43:25.948721+0000","last_undegraded":"2023-10-16T09:43:25.948721+0000","last_fullsized":"2023-10-16T09:43:25.948721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:00.084223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944746+0000","last_change":"2023-10-16T09:43:21.936106+0000","last_active":"2023-10-16T09:43:25.944746+0000","last_peered":"2023-10-16T09:43:25.944746+0000","last_clean":"2023-10-16T09:43:25.944746+0000","last_became_active":"2023-10-16T09:43:21.935859+0000","last_became_peered":"2023-10-16T09:43:21.935859+0000","last_unstale":"2023-10-16T09:43:25.944746+0000","last_undegraded":"2023-10-16T09:43:25.944746+0000","last_fullsized":"2023-10-16T09:43:25.944746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.014901+0000","last_change":"2023-10-16T09:43:25.961369+0000","last_active":"2023-10-16T09:43:28.014901+0000","last_peered":"2023-10-16T09:43:28.014901+0000","last_clean":"2023-10-16T09:43:28.014901+0000","last_became_active":"2023-10-16T09:43:25.960465+0000","last_became_peered":"2023-10-16T09:43:25.960465+0000","last_unstale":"2023-10-16T09:43:28.014901+0000","last_undegraded":"2023-10-16T09:43:28.014901+0000","last_fullsized":"2023-10-16T09:43:28.014901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:40.952070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955514+0000","last_change":"2023-10-16T09:43:20.922927+0000","last_active":"2023-10-16T09:43:25.955514+0000","last_peered":"2023-10-16T09:43:25.955514+0000","last_clean":"2023-10-16T09:43:25.955514+0000","last_became_active":"2023-10-16T09:43:20.922658+0000","last_became_peered":"2023-10-16T09:43:20.922658+0000","last_unstale":"2023-10-16T09:43:25.955514+0000","last_undegraded":"2023-10-16T09:43:25.955514+0000","last_fullsized":"2023-10-16T09:43:25.955514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950404+0000","last_change":"2023-10-16T09:43:21.928420+0000","last_active":"2023-10-16T09:43:25.950404+0000","last_peered":"2023-10-16T09:43:25.950404+0000","last_clean":"2023-10-16T09:43:25.950404+0000","last_became_active":"2023-10-16T09:43:21.928180+0000","last_became_peered":"2023-10-16T09:43:21.928180+0000","last_unstale":"2023-10-16T09:43:25.950404+0000","last_undegraded":"2023-10-16T09:43:25.950404+0000","last_fullsized":"2023-10-16T09:43:25.950404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015214+0000","last_change":"2023-10-16T09:43:25.960304+0000","last_active":"2023-10-16T09:43:28.015214+0000","last_peered":"2023-10-16T09:43:28.015214+0000","last_clean":"2023-10-16T09:43:28.015214+0000","last_became_active":"2023-10-16T09:43:25.959916+0000","last_became_peered":"2023-10-16T09:43:25.959916+0000","last_unstale":"2023-10-16T09:43:28.015214+0000","last_undegraded":"2023-10-16T09:43:28.015214+0000","last_fullsized":"2023-10-16T09:43:28.015214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948747+0000","last_change":"2023-10-16T09:43:20.921666+0000","last_active":"2023-10-16T09:43:25.948747+0000","last_peered":"2023-10-16T09:43:25.948747+0000","last_clean":"2023-10-16T09:43:25.948747+0000","last_became_active":"2023-10-16T09:43:20.921373+0000","last_became_peered":"2023-10-16T09:43:20.921373+0000","last_unstale":"2023-10-16T09:43:25.948747+0000","last_undegraded":"2023-10-16T09:43:25.948747+0000","last_fullsized":"2023-10-16T09:43:25.948747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944941+0000","last_change":"2023-10-16T09:43:21.930965+0000","last_active":"2023-10-16T09:43:25.944941+0000","last_peered":"2023-10-16T09:43:25.944941+0000","last_clean":"2023-10-16T09:43:25.944941+0000","last_became_active":"2023-10-16T09:43:21.930701+0000","last_became_peered":"2023-10-16T09:43:21.930701+0000","last_unstale":"2023-10-16T09:43:25.944941+0000","last_undegraded":"2023-10-16T09:43:25.944941+0000","last_fullsized":"2023-10-16T09:43:25.944941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:20.329096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954036+0000","last_change":"2023-10-16T09:43:21.930641+0000","last_active":"2023-10-16T09:43:25.954036+0000","last_peered":"2023-10-16T09:43:25.954036+0000","last_clean":"2023-10-16T09:43:25.954036+0000","last_became_active":"2023-10-16T09:43:21.930194+0000","last_became_peered":"2023-10-16T09:43:21.930194+0000","last_unstale":"2023-10-16T09:43:25.954036+0000","last_undegraded":"2023-10-16T09:43:25.954036+0000","last_fullsized":"2023-10-16T09:43:25.954036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948076+0000","last_change":"2023-10-16T09:43:21.933068+0000","last_active":"2023-10-16T09:43:25.948076+0000","last_peered":"2023-10-16T09:43:25.948076+0000","last_clean":"2023-10-16T09:43:25.948076+0000","last_became_active":"2023-10-16T09:43:21.932774+0000","last_became_peered":"2023-10-16T09:43:21.932774+0000","last_unstale":"2023-10-16T09:43:25.948076+0000","last_undegraded":"2023-10-16T09:43:25.948076+0000","last_fullsized":"2023-10-16T09:43:25.948076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950489+0000","last_change":"2023-10-16T09:43:21.937509+0000","last_active":"2023-10-16T09:43:25.950489+0000","last_peered":"2023-10-16T09:43:25.950489+0000","last_clean":"2023-10-16T09:43:25.950489+0000","last_became_active":"2023-10-16T09:43:21.936970+0000","last_became_peered":"2023-10-16T09:43:21.936970+0000","last_unstale":"2023-10-16T09:43:25.950489+0000","last_undegraded":"2023-10-16T09:43:25.950489+0000","last_fullsized":"2023-10-16T09:43:25.950489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948478+0000","last_change":"2023-10-16T09:43:21.927275+0000","last_active":"2023-10-16T09:43:25.948478+0000","last_peered":"2023-10-16T09:43:25.948478+0000","last_clean":"2023-10-16T09:43:25.948478+0000","last_became_active":"2023-10-16T09:43:21.927024+0000","last_became_peered":"2023-10-16T09:43:21.927024+0000","last_unstale":"2023-10-16T09:43:25.948478+0000","last_undegraded":"2023-10-16T09:43:25.948478+0000","last_fullsized":"2023-10-16T09:43:25.948478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966374+0000","last_change":"2023-10-16T09:43:21.925070+0000","last_active":"2023-10-16T09:43:25.966374+0000","last_peered":"2023-10-16T09:43:25.966374+0000","last_clean":"2023-10-16T09:43:25.966374+0000","last_became_active":"2023-10-16T09:43:21.924799+0000","last_became_peered":"2023-10-16T09:43:21.924799+0000","last_unstale":"2023-10-16T09:43:25.966374+0000","last_undegraded":"2023-10-16T09:43:25.966374+0000","last_fullsized":"2023-10-16T09:43:25.966374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955685+0000","last_change":"2023-10-16T09:43:21.920367+0000","last_active":"2023-10-16T09:43:25.955685+0000","last_peered":"2023-10-16T09:43:25.955685+0000","last_clean":"2023-10-16T09:43:25.955685+0000","last_became_active":"2023-10-16T09:43:21.919980+0000","last_became_peered":"2023-10-16T09:43:21.919980+0000","last_unstale":"2023-10-16T09:43:25.955685+0000","last_undegraded":"2023-10-16T09:43:25.955685+0000","last_fullsized":"2023-10-16T09:43:25.955685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956021+0000","last_change":"2023-10-16T09:43:21.931239+0000","last_active":"2023-10-16T09:43:25.956021+0000","last_peered":"2023-10-16T09:43:25.956021+0000","last_clean":"2023-10-16T09:43:25.956021+0000","last_became_active":"2023-10-16T09:43:21.930923+0000","last_became_peered":"2023-10-16T09:43:21.930923+0000","last_unstale":"2023-10-16T09:43:25.956021+0000","last_undegraded":"2023-10-16T09:43:25.956021+0000","last_fullsized":"2023-10-16T09:43:25.956021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967144+0000","last_change":"2023-10-16T09:43:21.924310+0000","last_active":"2023-10-16T09:43:25.967144+0000","last_peered":"2023-10-16T09:43:25.967144+0000","last_clean":"2023-10-16T09:43:25.967144+0000","last_became_active":"2023-10-16T09:43:21.921665+0000","last_became_peered":"2023-10-16T09:43:21.921665+0000","last_unstale":"2023-10-16T09:43:25.967144+0000","last_undegraded":"2023-10-16T09:43:25.967144+0000","last_fullsized":"2023-10-16T09:43:25.967144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950321+0000","last_change":"2023-10-16T09:43:21.927572+0000","last_active":"2023-10-16T09:43:25.950321+0000","last_peered":"2023-10-16T09:43:25.950321+0000","last_clean":"2023-10-16T09:43:25.950321+0000","last_became_active":"2023-10-16T09:43:21.927298+0000","last_became_peered":"2023-10-16T09:43:21.927298+0000","last_unstale":"2023-10-16T09:43:25.950321+0000","last_undegraded":"2023-10-16T09:43:25.950321+0000","last_fullsized":"2023-10-16T09:43:25.950321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965935+0000","last_change":"2023-10-16T09:43:21.922799+0000","last_active":"2023-10-16T09:43:25.965935+0000","last_peered":"2023-10-16T09:43:25.965935+0000","last_clean":"2023-10-16T09:43:25.965935+0000","last_became_active":"2023-10-16T09:43:21.922126+0000","last_became_peered":"2023-10-16T09:43:21.922126+0000","last_unstale":"2023-10-16T09:43:25.965935+0000","last_undegraded":"2023-10-16T09:43:25.965935+0000","last_fullsized":"2023-10-16T09:43:25.965935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950579+0000","last_change":"2023-10-16T09:43:21.936429+0000","last_active":"2023-10-16T09:43:25.950579+0000","last_peered":"2023-10-16T09:43:25.950579+0000","last_clean":"2023-10-16T09:43:25.950579+0000","last_became_active":"2023-10-16T09:43:21.933542+0000","last_became_peered":"2023-10-16T09:43:21.933542+0000","last_unstale":"2023-10-16T09:43:25.950579+0000","last_undegraded":"2023-10-16T09:43:25.950579+0000","last_fullsized":"2023-10-16T09:43:25.950579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948591+0000","last_change":"2023-10-16T09:43:21.930439+0000","last_active":"2023-10-16T09:43:25.948591+0000","last_peered":"2023-10-16T09:43:25.948591+0000","last_clean":"2023-10-16T09:43:25.948591+0000","last_became_active":"2023-10-16T09:43:21.930237+0000","last_became_peered":"2023-10-16T09:43:21.930237+0000","last_unstale":"2023-10-16T09:43:25.948591+0000","last_undegraded":"2023-10-16T09:43:25.948591+0000","last_fullsized":"2023-10-16T09:43:25.948591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:17.181381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953918+0000","last_change":"2023-10-16T09:43:21.934041+0000","last_active":"2023-10-16T09:43:25.953918+0000","last_peered":"2023-10-16T09:43:25.953918+0000","last_clean":"2023-10-16T09:43:25.953918+0000","last_became_active":"2023-10-16T09:43:21.933451+0000","last_became_peered":"2023-10-16T09:43:21.933451+0000","last_unstale":"2023-10-16T09:43:25.953918+0000","last_undegraded":"2023-10-16T09:43:25.953918+0000","last_fullsized":"2023-10-16T09:43:25.953918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":99940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":99780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:43:50.199 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:43:50.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.351+0000 7f423df00700 1 Processor -- start 2023-10-16T09:43:50.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.351+0000 7f423df00700 1 -- start start 2023-10-16T09:43:50.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f423df00700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f423df00700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f423806dd70 0x7f42380cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f423df00700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42380caaa0 con 0x7f423806acb0 2023-10-16T09:43:50.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f423df00700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42380cc320 con 0x7f423806dd70 2023-10-16T09:43:50.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f423df00700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42380cad40 con 0x7f423806a8d0 2023-10-16T09:43:50.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f4236ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f423806dd70 0x7f42380cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f4236ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f423806dd70 0x7f42380cbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46804/0 (socket says 172.21.15.158:46804) 2023-10-16T09:43:50.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.352+0000 7f4236ffd700 1 -- 172.21.15.158:0/1292089409 learned_addr learned my addr 172.21.15.158:0/1292089409 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:50.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f42377fe700 1 --2- 172.21.15.158:0/1292089409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f4236ffd700 1 -- 172.21.15.158:0/1292089409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 msgr2=0x7f42380ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:50.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f4236ffd700 1 --2- 172.21.15.158:0/1292089409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f4236ffd700 1 -- 172.21.15.158:0/1292089409 >> v1:172.21.15.172:6789/0 conn(0x7f423806acb0 legacy=0x7f423806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f4236ffd700 1 -- 172.21.15.158:0/1292089409 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380cb020 con 0x7f423806dd70 2023-10-16T09:43:50.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f4236ffd700 1 --2- 172.21.15.158:0/1292089409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380cbde0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f42280098f0 tx=0x7f4228009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8966e546df13f402 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:50.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f42367fc700 1 -- 172.21.15.158:0/1292089409 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800b8a0 con 0x7f423806dd70 2023-10-16T09:43:50.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f42367fc700 1 -- 172.21.15.158:0/1292089409 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800ba00 con 0x7f423806dd70 2023-10-16T09:43:50.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.353+0000 7f42367fc700 1 -- 172.21.15.158:0/1292089409 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800bd90 con 0x7f423806dd70 2023-10-16T09:43:50.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 -- 172.21.15.158:0/1292089409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 msgr2=0x7f42380cbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 --2- 172.21.15.158:0/1292089409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380cbde0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f42280098f0 tx=0x7f4228009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:50.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 -- 172.21.15.158:0/1292089409 shutdown_connections 2023-10-16T09:43:50.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 --2- 172.21.15.158:0/1292089409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 --2- 172.21.15.158:0/1292089409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380cbde0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 -- 172.21.15.158:0/1292089409 >> 172.21.15.158:0/1292089409 conn(0x7f42380cc5a0 msgr2=0x7f42380cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:50.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 -- 172.21.15.158:0/1292089409 shutdown_connections 2023-10-16T09:43:50.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.354+0000 7f423df00700 1 -- 172.21.15.158:0/1292089409 wait complete. 2023-10-16T09:43:50.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.355+0000 7f423df00700 1 Processor -- start 2023-10-16T09:43:50.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.355+0000 7f423df00700 1 -- start start 2023-10-16T09:43:50.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f423df00700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f423df00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f423806acb0 0x7f42380cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f423df00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f423df00700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380c7ce0 con 0x7f423806dd70 2023-10-16T09:43:50.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f423df00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380c7f50 con 0x7f423806acb0 2023-10-16T09:43:50.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f423df00700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42380c8230 con 0x7f423806a8d0 2023-10-16T09:43:50.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46812/0 (socket says 172.21.15.158:46812) 2023-10-16T09:43:50.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 -- 172.21.15.158:0/937655234 learned_addr learned my addr 172.21.15.158:0/937655234 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:50.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4236ffd700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f423806acb0 0x7f42380cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f42377fe700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 -- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 msgr2=0x7f42380b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 -- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f423806acb0 msgr2=0x7f42380cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f423806acb0 0x7f42380cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 -- 172.21.15.158:0/937655234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228009550 con 0x7f423806dd70 2023-10-16T09:43:50.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f4237fff700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380b3040 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f423000cc50 tx=0x7f423000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:50.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.356+0000 7f421ffff700 1 -- 172.21.15.158:0/937655234 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230003c00 con 0x7f423806dd70 2023-10-16T09:43:50.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.357+0000 7f421ffff700 1 -- 172.21.15.158:0/937655234 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4230003d60 con 0x7f423806dd70 2023-10-16T09:43:50.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.357+0000 7f421ffff700 1 -- 172.21.15.158:0/937655234 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423000aab0 con 0x7f423806dd70 2023-10-16T09:43:50.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.357+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42380c8510 con 0x7f423806dd70 2023-10-16T09:43:50.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.357+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42381f3b80 con 0x7f423806dd70 2023-10-16T09:43:50.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.358+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42380b5990 con 0x7f423806dd70 2023-10-16T09:43:50.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.358+0000 7f421ffff700 1 -- 172.21.15.158:0/937655234 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f423000ac10 con 0x7f423806dd70 2023-10-16T09:43:50.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.358+0000 7f421ffff700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f422406af00 0x7f422406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.358+0000 7f421ffff700 1 -- 172.21.15.158:0/937655234 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f42300898a0 con 0x7f423806dd70 2023-10-16T09:43:50.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.358+0000 7f42377fe700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f422406af00 0x7f422406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.359+0000 7f42377fe700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f422406af00 0x7f422406d3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f422c0099d0 tx=0x7f422c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:50.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.363+0000 7f421ffff700 1 -- 172.21.15.158:0/937655234 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4230054660 con 0x7f423806dd70 2023-10-16T09:43:50.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.552+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f42380cb5e0 con 0x7f422406af00 2023-10-16T09:43:50.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.561+0000 7f421ffff700 1 -- 172.21.15.158:0/937655234 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345229 (secure 0 0 0) 0x7f422c009e40 con 0x7f422406af00 2023-10-16T09:43:50.568 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:50.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f422406af00 msgr2=0x7f422406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f422406af00 0x7f422406d3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f422c0099d0 tx=0x7f422c008040 comp rx=0 tx=0).stop 2023-10-16T09:43:50.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 msgr2=0x7f42380b3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380b3040 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f423000cc50 tx=0x7f423000cf60 comp rx=0 tx=0).stop 2023-10-16T09:43:50.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 shutdown_connections 2023-10-16T09:43:50.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f423806dd70 0x7f42380b3040 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f422406af00 0x7f422406d3c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f423806a8d0 0x7f42380b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 --2- 172.21.15.158:0/937655234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f423806acb0 0x7f42380cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 >> 172.21.15.158:0/937655234 conn(0x7f42380cc5a0 msgr2=0x7f42381564d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:50.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 shutdown_connections 2023-10-16T09:43:50.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.565+0000 7f423df00700 1 -- 172.21.15.158:0/937655234 wait complete. 2023-10-16T09:43:50.573 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:43:50.579 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-16T09:43:49.224077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3284992,"data_stored":1988256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003263"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944843+0000","last_change":"2023-10-16T09:43:21.932095+0000","last_active":"2023-10-16T09:43:25.944843+0000","last_peered":"2023-10-16T09:43:25.944843+0000","last_clean":"2023-10-16T09:43:25.944843+0000","last_became_active":"2023-10-16T09:43:21.931826+0000","last_became_peered":"2023-10-16T09:43:21.931826+0000","last_unstale":"2023-10-16T09:43:25.944843+0000","last_undegraded":"2023-10-16T09:43:25.944843+0000","last_fullsized":"2023-10-16T09:43:25.944843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955709+0000","last_change":"2023-10-16T09:43:21.921656+0000","last_active":"2023-10-16T09:43:25.955709+0000","last_peered":"2023-10-16T09:43:25.955709+0000","last_clean":"2023-10-16T09:43:25.955709+0000","last_became_active":"2023-10-16T09:43:21.921392+0000","last_became_peered":"2023-10-16T09:43:21.921392+0000","last_unstale":"2023-10-16T09:43:25.955709+0000","last_undegraded":"2023-10-16T09:43:25.955709+0000","last_fullsized":"2023-10-16T09:43:25.955709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951037+0000","last_change":"2023-10-16T09:43:21.925692+0000","last_active":"2023-10-16T09:43:25.951037+0000","last_peered":"2023-10-16T09:43:25.951037+0000","last_clean":"2023-10-16T09:43:25.951037+0000","last_became_active":"2023-10-16T09:43:21.925438+0000","last_became_peered":"2023-10-16T09:43:21.925438+0000","last_unstale":"2023-10-16T09:43:25.951037+0000","last_undegraded":"2023-10-16T09:43:25.951037+0000","last_fullsized":"2023-10-16T09:43:25.951037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950435+0000","last_change":"2023-10-16T09:43:21.927209+0000","last_active":"2023-10-16T09:43:25.950435+0000","last_peered":"2023-10-16T09:43:25.950435+0000","last_clean":"2023-10-16T09:43:25.950435+0000","last_became_active":"2023-10-16T09:43:21.926900+0000","last_became_peered":"2023-10-16T09:43:21.926900+0000","last_unstale":"2023-10-16T09:43:25.950435+0000","last_undegraded":"2023-10-16T09:43:25.950435+0000","last_fullsized":"2023-10-16T09:43:25.950435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944943+0000","last_change":"2023-10-16T09:43:21.924582+0000","last_active":"2023-10-16T09:43:25.944943+0000","last_peered":"2023-10-16T09:43:25.944943+0000","last_clean":"2023-10-16T09:43:25.944943+0000","last_became_active":"2023-10-16T09:43:21.924329+0000","last_became_peered":"2023-10-16T09:43:21.924329+0000","last_unstale":"2023-10-16T09:43:25.944943+0000","last_undegraded":"2023-10-16T09:43:25.944943+0000","last_fullsized":"2023-10-16T09:43:25.944943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950668+0000","last_change":"2023-10-16T09:43:21.937326+0000","last_active":"2023-10-16T09:43:25.950668+0000","last_peered":"2023-10-16T09:43:25.950668+0000","last_clean":"2023-10-16T09:43:25.950668+0000","last_became_active":"2023-10-16T09:43:21.936665+0000","last_became_peered":"2023-10-16T09:43:21.936665+0000","last_unstale":"2023-10-16T09:43:25.950668+0000","last_undegraded":"2023-10-16T09:43:25.950668+0000","last_fullsized":"2023-10-16T09:43:25.950668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:21.923235+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:21.923021+0000","last_became_peered":"2023-10-16T09:43:21.923021+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965916+0000","last_change":"2023-10-16T09:43:21.933111+0000","last_active":"2023-10-16T09:43:25.965916+0000","last_peered":"2023-10-16T09:43:25.965916+0000","last_clean":"2023-10-16T09:43:25.965916+0000","last_became_active":"2023-10-16T09:43:21.932835+0000","last_became_peered":"2023-10-16T09:43:21.932835+0000","last_unstale":"2023-10-16T09:43:25.965916+0000","last_undegraded":"2023-10-16T09:43:25.965916+0000","last_fullsized":"2023-10-16T09:43:25.965916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944721+0000","last_change":"2023-10-16T09:43:21.940444+0000","last_active":"2023-10-16T09:43:25.944721+0000","last_peered":"2023-10-16T09:43:25.944721+0000","last_clean":"2023-10-16T09:43:25.944721+0000","last_became_active":"2023-10-16T09:43:21.939823+0000","last_became_peered":"2023-10-16T09:43:21.939823+0000","last_unstale":"2023-10-16T09:43:25.944721+0000","last_undegraded":"2023-10-16T09:43:25.944721+0000","last_fullsized":"2023-10-16T09:43:25.944721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956248+0000","last_change":"2023-10-16T09:43:21.934573+0000","last_active":"2023-10-16T09:43:25.956248+0000","last_peered":"2023-10-16T09:43:25.956248+0000","last_clean":"2023-10-16T09:43:25.956248+0000","last_became_active":"2023-10-16T09:43:21.934285+0000","last_became_peered":"2023-10-16T09:43:21.934285+0000","last_unstale":"2023-10-16T09:43:25.956248+0000","last_undegraded":"2023-10-16T09:43:25.956248+0000","last_fullsized":"2023-10-16T09:43:25.956248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953996+0000","last_change":"2023-10-16T09:43:21.934669+0000","last_active":"2023-10-16T09:43:25.953996+0000","last_peered":"2023-10-16T09:43:25.953996+0000","last_clean":"2023-10-16T09:43:25.953996+0000","last_became_active":"2023-10-16T09:43:21.933918+0000","last_became_peered":"2023-10-16T09:43:21.933918+0000","last_unstale":"2023-10-16T09:43:25.953996+0000","last_undegraded":"2023-10-16T09:43:25.953996+0000","last_fullsized":"2023-10-16T09:43:25.953996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950011+0000","last_change":"2023-10-16T09:43:21.934285+0000","last_active":"2023-10-16T09:43:25.950011+0000","last_peered":"2023-10-16T09:43:25.950011+0000","last_clean":"2023-10-16T09:43:25.950011+0000","last_became_active":"2023-10-16T09:43:21.933516+0000","last_became_peered":"2023-10-16T09:43:21.933516+0000","last_unstale":"2023-10-16T09:43:25.950011+0000","last_undegraded":"2023-10-16T09:43:25.950011+0000","last_fullsized":"2023-10-16T09:43:25.950011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:56.814327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965961+0000","last_change":"2023-10-16T09:43:21.934162+0000","last_active":"2023-10-16T09:43:25.965961+0000","last_peered":"2023-10-16T09:43:25.965961+0000","last_clean":"2023-10-16T09:43:25.965961+0000","last_became_active":"2023-10-16T09:43:21.933898+0000","last_became_peered":"2023-10-16T09:43:21.933898+0000","last_unstale":"2023-10-16T09:43:25.965961+0000","last_undegraded":"2023-10-16T09:43:25.965961+0000","last_fullsized":"2023-10-16T09:43:25.965961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948340+0000","last_change":"2023-10-16T09:43:21.923364+0000","last_active":"2023-10-16T09:43:25.948340+0000","last_peered":"2023-10-16T09:43:25.948340+0000","last_clean":"2023-10-16T09:43:25.948340+0000","last_became_active":"2023-10-16T09:43:21.923123+0000","last_became_peered":"2023-10-16T09:43:21.923123+0000","last_unstale":"2023-10-16T09:43:25.948340+0000","last_undegraded":"2023-10-16T09:43:25.948340+0000","last_fullsized":"2023-10-16T09:43:25.948340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953943+0000","last_change":"2023-10-16T09:43:21.925350+0000","last_active":"2023-10-16T09:43:25.953943+0000","last_peered":"2023-10-16T09:43:25.953943+0000","last_clean":"2023-10-16T09:43:25.953943+0000","last_became_active":"2023-10-16T09:43:21.925107+0000","last_became_peered":"2023-10-16T09:43:21.925107+0000","last_unstale":"2023-10-16T09:43:25.953943+0000","last_undegraded":"2023-10-16T09:43:25.953943+0000","last_fullsized":"2023-10-16T09:43:25.953943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951212+0000","last_change":"2023-10-16T09:43:21.927864+0000","last_active":"2023-10-16T09:43:25.951212+0000","last_peered":"2023-10-16T09:43:25.951212+0000","last_clean":"2023-10-16T09:43:25.951212+0000","last_became_active":"2023-10-16T09:43:21.927411+0000","last_became_peered":"2023-10-16T09:43:21.927411+0000","last_unstale":"2023-10-16T09:43:25.951212+0000","last_undegraded":"2023-10-16T09:43:25.951212+0000","last_fullsized":"2023-10-16T09:43:25.951212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950524+0000","last_change":"2023-10-16T09:43:21.936884+0000","last_active":"2023-10-16T09:43:25.950524+0000","last_peered":"2023-10-16T09:43:25.950524+0000","last_clean":"2023-10-16T09:43:25.950524+0000","last_became_active":"2023-10-16T09:43:21.936569+0000","last_became_peered":"2023-10-16T09:43:21.936569+0000","last_unstale":"2023-10-16T09:43:25.950524+0000","last_undegraded":"2023-10-16T09:43:25.950524+0000","last_fullsized":"2023-10-16T09:43:25.950524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:36.921587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948848+0000","last_change":"2023-10-16T09:43:21.935147+0000","last_active":"2023-10-16T09:43:25.948848+0000","last_peered":"2023-10-16T09:43:25.948848+0000","last_clean":"2023-10-16T09:43:25.948848+0000","last_became_active":"2023-10-16T09:43:21.934786+0000","last_became_peered":"2023-10-16T09:43:21.934786+0000","last_unstale":"2023-10-16T09:43:25.948848+0000","last_undegraded":"2023-10-16T09:43:25.948848+0000","last_fullsized":"2023-10-16T09:43:25.948848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945003+0000","last_change":"2023-10-16T09:43:21.927238+0000","last_active":"2023-10-16T09:43:25.945003+0000","last_peered":"2023-10-16T09:43:25.945003+0000","last_clean":"2023-10-16T09:43:25.945003+0000","last_became_active":"2023-10-16T09:43:21.926944+0000","last_became_peered":"2023-10-16T09:43:21.926944+0000","last_unstale":"2023-10-16T09:43:25.945003+0000","last_undegraded":"2023-10-16T09:43:25.945003+0000","last_fullsized":"2023-10-16T09:43:25.945003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015597+0000","last_change":"2023-10-16T09:43:25.972270+0000","last_active":"2023-10-16T09:43:28.015597+0000","last_peered":"2023-10-16T09:43:28.015597+0000","last_clean":"2023-10-16T09:43:28.015597+0000","last_became_active":"2023-10-16T09:43:25.971856+0000","last_became_peered":"2023-10-16T09:43:25.971856+0000","last_unstale":"2023-10-16T09:43:28.015597+0000","last_undegraded":"2023-10-16T09:43:28.015597+0000","last_fullsized":"2023-10-16T09:43:28.015597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:00.427859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951697+0000","last_change":"2023-10-16T09:43:20.934516+0000","last_active":"2023-10-16T09:43:25.951697+0000","last_peered":"2023-10-16T09:43:25.951697+0000","last_clean":"2023-10-16T09:43:25.951697+0000","last_became_active":"2023-10-16T09:43:20.934196+0000","last_became_peered":"2023-10-16T09:43:20.934196+0000","last_unstale":"2023-10-16T09:43:25.951697+0000","last_undegraded":"2023-10-16T09:43:25.951697+0000","last_fullsized":"2023-10-16T09:43:25.951697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966598+0000","last_change":"2023-10-16T09:43:21.941097+0000","last_active":"2023-10-16T09:43:25.966598+0000","last_peered":"2023-10-16T09:43:25.966598+0000","last_clean":"2023-10-16T09:43:25.966598+0000","last_became_active":"2023-10-16T09:43:21.940820+0000","last_became_peered":"2023-10-16T09:43:21.940820+0000","last_unstale":"2023-10-16T09:43:25.966598+0000","last_undegraded":"2023-10-16T09:43:25.966598+0000","last_fullsized":"2023-10-16T09:43:25.966598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947760+0000","last_change":"2023-10-16T09:43:25.970409+0000","last_active":"2023-10-16T09:43:26.947760+0000","last_peered":"2023-10-16T09:43:26.947760+0000","last_clean":"2023-10-16T09:43:26.947760+0000","last_became_active":"2023-10-16T09:43:25.969848+0000","last_became_peered":"2023-10-16T09:43:25.969848+0000","last_unstale":"2023-10-16T09:43:26.947760+0000","last_undegraded":"2023-10-16T09:43:26.947760+0000","last_fullsized":"2023-10-16T09:43:26.947760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:00.972576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966712+0000","last_change":"2023-10-16T09:43:20.933965+0000","last_active":"2023-10-16T09:43:25.966712+0000","last_peered":"2023-10-16T09:43:25.966712+0000","last_clean":"2023-10-16T09:43:25.966712+0000","last_became_active":"2023-10-16T09:43:20.933635+0000","last_became_peered":"2023-10-16T09:43:20.933635+0000","last_unstale":"2023-10-16T09:43:25.966712+0000","last_undegraded":"2023-10-16T09:43:25.966712+0000","last_fullsized":"2023-10-16T09:43:25.966712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951601+0000","last_change":"2023-10-16T09:43:21.937374+0000","last_active":"2023-10-16T09:43:25.951601+0000","last_peered":"2023-10-16T09:43:25.951601+0000","last_clean":"2023-10-16T09:43:25.951601+0000","last_became_active":"2023-10-16T09:43:21.936989+0000","last_became_peered":"2023-10-16T09:43:21.936989+0000","last_unstale":"2023-10-16T09:43:25.951601+0000","last_undegraded":"2023-10-16T09:43:25.951601+0000","last_fullsized":"2023-10-16T09:43:25.951601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948251+0000","last_change":"2023-10-16T09:43:25.970011+0000","last_active":"2023-10-16T09:43:26.948251+0000","last_peered":"2023-10-16T09:43:26.948251+0000","last_clean":"2023-10-16T09:43:26.948251+0000","last_became_active":"2023-10-16T09:43:25.969570+0000","last_became_peered":"2023-10-16T09:43:25.969570+0000","last_unstale":"2023-10-16T09:43:26.948251+0000","last_undegraded":"2023-10-16T09:43:26.948251+0000","last_fullsized":"2023-10-16T09:43:26.948251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944734+0000","last_change":"2023-10-16T09:43:20.928578+0000","last_active":"2023-10-16T09:43:25.944734+0000","last_peered":"2023-10-16T09:43:25.944734+0000","last_clean":"2023-10-16T09:43:25.944734+0000","last_became_active":"2023-10-16T09:43:20.928283+0000","last_became_peered":"2023-10-16T09:43:20.928283+0000","last_unstale":"2023-10-16T09:43:25.944734+0000","last_undegraded":"2023-10-16T09:43:25.944734+0000","last_fullsized":"2023-10-16T09:43:25.944734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950360+0000","last_change":"2023-10-16T09:43:21.936827+0000","last_active":"2023-10-16T09:43:25.950360+0000","last_peered":"2023-10-16T09:43:25.950360+0000","last_clean":"2023-10-16T09:43:25.950360+0000","last_became_active":"2023-10-16T09:43:21.936545+0000","last_became_peered":"2023-10-16T09:43:21.936545+0000","last_unstale":"2023-10-16T09:43:25.950360+0000","last_undegraded":"2023-10-16T09:43:25.950360+0000","last_fullsized":"2023-10-16T09:43:25.950360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:47.422323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947909+0000","last_change":"2023-10-16T09:43:25.968566+0000","last_active":"2023-10-16T09:43:26.947909+0000","last_peered":"2023-10-16T09:43:26.947909+0000","last_clean":"2023-10-16T09:43:26.947909+0000","last_became_active":"2023-10-16T09:43:25.968188+0000","last_became_peered":"2023-10-16T09:43:25.968188+0000","last_unstale":"2023-10-16T09:43:26.947909+0000","last_undegraded":"2023-10-16T09:43:26.947909+0000","last_fullsized":"2023-10-16T09:43:26.947909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:38.074113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944439+0000","last_change":"2023-10-16T09:43:20.929934+0000","last_active":"2023-10-16T09:43:25.944439+0000","last_peered":"2023-10-16T09:43:25.944439+0000","last_clean":"2023-10-16T09:43:25.944439+0000","last_became_active":"2023-10-16T09:43:20.929639+0000","last_became_peered":"2023-10-16T09:43:20.929639+0000","last_unstale":"2023-10-16T09:43:25.944439+0000","last_undegraded":"2023-10-16T09:43:25.944439+0000","last_fullsized":"2023-10-16T09:43:25.944439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953109+0000","last_change":"2023-10-16T09:43:21.933199+0000","last_active":"2023-10-16T09:43:25.953109+0000","last_peered":"2023-10-16T09:43:25.953109+0000","last_clean":"2023-10-16T09:43:25.953109+0000","last_became_active":"2023-10-16T09:43:21.932906+0000","last_became_peered":"2023-10-16T09:43:21.932906+0000","last_unstale":"2023-10-16T09:43:25.953109+0000","last_undegraded":"2023-10-16T09:43:25.953109+0000","last_fullsized":"2023-10-16T09:43:25.953109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016130+0000","last_change":"2023-10-16T09:43:25.975356+0000","last_active":"2023-10-16T09:43:28.016130+0000","last_peered":"2023-10-16T09:43:28.016130+0000","last_clean":"2023-10-16T09:43:28.016130+0000","last_became_active":"2023-10-16T09:43:25.974334+0000","last_became_peered":"2023-10-16T09:43:25.974334+0000","last_unstale":"2023-10-16T09:43:28.016130+0000","last_undegraded":"2023-10-16T09:43:28.016130+0000","last_fullsized":"2023-10-16T09:43:28.016130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:20.175649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966133+0000","last_change":"2023-10-16T09:43:20.928498+0000","last_active":"2023-10-16T09:43:25.966133+0000","last_peered":"2023-10-16T09:43:25.966133+0000","last_clean":"2023-10-16T09:43:25.966133+0000","last_became_active":"2023-10-16T09:43:20.928242+0000","last_became_peered":"2023-10-16T09:43:20.928242+0000","last_unstale":"2023-10-16T09:43:25.966133+0000","last_undegraded":"2023-10-16T09:43:25.966133+0000","last_fullsized":"2023-10-16T09:43:25.966133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948266+0000","last_change":"2023-10-16T09:43:21.919769+0000","last_active":"2023-10-16T09:43:25.948266+0000","last_peered":"2023-10-16T09:43:25.948266+0000","last_clean":"2023-10-16T09:43:25.948266+0000","last_became_active":"2023-10-16T09:43:21.919395+0000","last_became_peered":"2023-10-16T09:43:21.919395+0000","last_unstale":"2023-10-16T09:43:25.948266+0000","last_undegraded":"2023-10-16T09:43:25.948266+0000","last_fullsized":"2023-10-16T09:43:25.948266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:06.700888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.944846+0000","last_change":"2023-10-16T09:43:25.970755+0000","last_active":"2023-10-16T09:43:26.944846+0000","last_peered":"2023-10-16T09:43:26.944846+0000","last_clean":"2023-10-16T09:43:26.944846+0000","last_became_active":"2023-10-16T09:43:25.970344+0000","last_became_peered":"2023-10-16T09:43:25.970344+0000","last_unstale":"2023-10-16T09:43:26.944846+0000","last_undegraded":"2023-10-16T09:43:26.944846+0000","last_fullsized":"2023-10-16T09:43:26.944846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:28.937851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957011+0000","last_change":"2023-10-16T09:43:20.924380+0000","last_active":"2023-10-16T09:43:25.957011+0000","last_peered":"2023-10-16T09:43:25.957011+0000","last_clean":"2023-10-16T09:43:25.957011+0000","last_became_active":"2023-10-16T09:43:20.924104+0000","last_became_peered":"2023-10-16T09:43:20.924104+0000","last_unstale":"2023-10-16T09:43:25.957011+0000","last_undegraded":"2023-10-16T09:43:25.957011+0000","last_fullsized":"2023-10-16T09:43:25.957011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948179+0000","last_change":"2023-10-16T09:43:21.929870+0000","last_active":"2023-10-16T09:43:25.948179+0000","last_peered":"2023-10-16T09:43:25.948179+0000","last_clean":"2023-10-16T09:43:25.948179+0000","last_became_active":"2023-10-16T09:43:21.929589+0000","last_became_peered":"2023-10-16T09:43:21.929589+0000","last_unstale":"2023-10-16T09:43:25.948179+0000","last_undegraded":"2023-10-16T09:43:25.948179+0000","last_fullsized":"2023-10-16T09:43:25.948179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949711+0000","last_change":"2023-10-16T09:43:25.966670+0000","last_active":"2023-10-16T09:43:26.949711+0000","last_peered":"2023-10-16T09:43:26.949711+0000","last_clean":"2023-10-16T09:43:26.949711+0000","last_became_active":"2023-10-16T09:43:25.966312+0000","last_became_peered":"2023-10-16T09:43:25.966312+0000","last_unstale":"2023-10-16T09:43:26.949711+0000","last_undegraded":"2023-10-16T09:43:26.949711+0000","last_fullsized":"2023-10-16T09:43:26.949711+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:54.507574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965835+0000","last_change":"2023-10-16T09:43:20.928383+0000","last_active":"2023-10-16T09:43:25.965835+0000","last_peered":"2023-10-16T09:43:25.965835+0000","last_clean":"2023-10-16T09:43:25.965835+0000","last_became_active":"2023-10-16T09:43:20.928096+0000","last_became_peered":"2023-10-16T09:43:20.928096+0000","last_unstale":"2023-10-16T09:43:25.965835+0000","last_undegraded":"2023-10-16T09:43:25.965835+0000","last_fullsized":"2023-10-16T09:43:25.965835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965672+0000","last_change":"2023-10-16T09:43:21.921213+0000","last_active":"2023-10-16T09:43:25.965672+0000","last_peered":"2023-10-16T09:43:25.965672+0000","last_clean":"2023-10-16T09:43:25.965672+0000","last_became_active":"2023-10-16T09:43:21.920920+0000","last_became_peered":"2023-10-16T09:43:21.920920+0000","last_unstale":"2023-10-16T09:43:25.965672+0000","last_undegraded":"2023-10-16T09:43:25.965672+0000","last_fullsized":"2023-10-16T09:43:25.965672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015970+0000","last_change":"2023-10-16T09:43:25.972702+0000","last_active":"2023-10-16T09:43:28.015970+0000","last_peered":"2023-10-16T09:43:28.015970+0000","last_clean":"2023-10-16T09:43:28.015970+0000","last_became_active":"2023-10-16T09:43:25.971784+0000","last_became_peered":"2023-10-16T09:43:25.971784+0000","last_unstale":"2023-10-16T09:43:28.015970+0000","last_undegraded":"2023-10-16T09:43:28.015970+0000","last_fullsized":"2023-10-16T09:43:28.015970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950235+0000","last_change":"2023-10-16T09:43:20.926495+0000","last_active":"2023-10-16T09:43:25.950235+0000","last_peered":"2023-10-16T09:43:25.950235+0000","last_clean":"2023-10-16T09:43:25.950235+0000","last_became_active":"2023-10-16T09:43:20.925998+0000","last_became_peered":"2023-10-16T09:43:20.925998+0000","last_unstale":"2023-10-16T09:43:25.950235+0000","last_undegraded":"2023-10-16T09:43:25.950235+0000","last_fullsized":"2023-10-16T09:43:25.950235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954132+0000","last_change":"2023-10-16T09:43:21.919652+0000","last_active":"2023-10-16T09:43:25.954132+0000","last_peered":"2023-10-16T09:43:25.954132+0000","last_clean":"2023-10-16T09:43:25.954132+0000","last_became_active":"2023-10-16T09:43:21.919397+0000","last_became_peered":"2023-10-16T09:43:21.919397+0000","last_unstale":"2023-10-16T09:43:25.954132+0000","last_undegraded":"2023-10-16T09:43:25.954132+0000","last_fullsized":"2023-10-16T09:43:25.954132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015120+0000","last_change":"2023-10-16T09:43:25.975457+0000","last_active":"2023-10-16T09:43:28.015120+0000","last_peered":"2023-10-16T09:43:28.015120+0000","last_clean":"2023-10-16T09:43:28.015120+0000","last_became_active":"2023-10-16T09:43:25.975098+0000","last_became_peered":"2023-10-16T09:43:25.975098+0000","last_unstale":"2023-10-16T09:43:28.015120+0000","last_undegraded":"2023-10-16T09:43:28.015120+0000","last_fullsized":"2023-10-16T09:43:28.015120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956270+0000","last_change":"2023-10-16T09:43:20.925087+0000","last_active":"2023-10-16T09:43:25.956270+0000","last_peered":"2023-10-16T09:43:25.956270+0000","last_clean":"2023-10-16T09:43:25.956270+0000","last_became_active":"2023-10-16T09:43:20.924798+0000","last_became_peered":"2023-10-16T09:43:20.924798+0000","last_unstale":"2023-10-16T09:43:25.956270+0000","last_undegraded":"2023-10-16T09:43:25.956270+0000","last_fullsized":"2023-10-16T09:43:25.956270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951125+0000","last_change":"2023-10-16T09:43:21.933170+0000","last_active":"2023-10-16T09:43:25.951125+0000","last_peered":"2023-10-16T09:43:25.951125+0000","last_clean":"2023-10-16T09:43:25.951125+0000","last_became_active":"2023-10-16T09:43:21.932876+0000","last_became_peered":"2023-10-16T09:43:21.932876+0000","last_unstale":"2023-10-16T09:43:25.951125+0000","last_undegraded":"2023-10-16T09:43:25.951125+0000","last_fullsized":"2023-10-16T09:43:25.951125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015261+0000","last_change":"2023-10-16T09:43:25.979174+0000","last_active":"2023-10-16T09:43:28.015261+0000","last_peered":"2023-10-16T09:43:28.015261+0000","last_clean":"2023-10-16T09:43:28.015261+0000","last_became_active":"2023-10-16T09:43:25.977030+0000","last_became_peered":"2023-10-16T09:43:25.977030+0000","last_unstale":"2023-10-16T09:43:28.015261+0000","last_undegraded":"2023-10-16T09:43:28.015261+0000","last_fullsized":"2023-10-16T09:43:28.015261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:47.151340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951034+0000","last_change":"2023-10-16T09:43:20.928828+0000","last_active":"2023-10-16T09:43:25.951034+0000","last_peered":"2023-10-16T09:43:25.951034+0000","last_clean":"2023-10-16T09:43:25.951034+0000","last_became_active":"2023-10-16T09:43:20.928550+0000","last_became_peered":"2023-10-16T09:43:20.928550+0000","last_unstale":"2023-10-16T09:43:25.951034+0000","last_undegraded":"2023-10-16T09:43:25.951034+0000","last_fullsized":"2023-10-16T09:43:25.951034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951780+0000","last_change":"2023-10-16T09:43:21.937614+0000","last_active":"2023-10-16T09:43:25.951780+0000","last_peered":"2023-10-16T09:43:25.951780+0000","last_clean":"2023-10-16T09:43:25.951780+0000","last_became_active":"2023-10-16T09:43:21.937335+0000","last_became_peered":"2023-10-16T09:43:21.937335+0000","last_unstale":"2023-10-16T09:43:25.951780+0000","last_undegraded":"2023-10-16T09:43:25.951780+0000","last_fullsized":"2023-10-16T09:43:25.951780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948709+0000","last_change":"2023-10-16T09:43:25.958049+0000","last_active":"2023-10-16T09:43:26.948709+0000","last_peered":"2023-10-16T09:43:26.948709+0000","last_clean":"2023-10-16T09:43:26.948709+0000","last_became_active":"2023-10-16T09:43:25.957672+0000","last_became_peered":"2023-10-16T09:43:25.957672+0000","last_unstale":"2023-10-16T09:43:26.948709+0000","last_undegraded":"2023-10-16T09:43:26.948709+0000","last_fullsized":"2023-10-16T09:43:26.948709+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:53.105902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955629+0000","last_change":"2023-10-16T09:43:20.926834+0000","last_active":"2023-10-16T09:43:25.955629+0000","last_peered":"2023-10-16T09:43:25.955629+0000","last_clean":"2023-10-16T09:43:25.955629+0000","last_became_active":"2023-10-16T09:43:20.926593+0000","last_became_peered":"2023-10-16T09:43:20.926593+0000","last_unstale":"2023-10-16T09:43:25.955629+0000","last_undegraded":"2023-10-16T09:43:25.955629+0000","last_fullsized":"2023-10-16T09:43:25.955629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:38.196651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947243+0000","last_change":"2023-10-16T09:43:21.925181+0000","last_active":"2023-10-16T09:43:25.947243+0000","last_peered":"2023-10-16T09:43:25.947243+0000","last_clean":"2023-10-16T09:43:25.947243+0000","last_became_active":"2023-10-16T09:43:21.924083+0000","last_became_peered":"2023-10-16T09:43:21.924083+0000","last_unstale":"2023-10-16T09:43:25.947243+0000","last_undegraded":"2023-10-16T09:43:25.947243+0000","last_fullsized":"2023-10-16T09:43:25.947243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:41.369421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995287+0000","last_change":"2023-10-16T09:43:25.960604+0000","last_active":"2023-10-16T09:43:27.995287+0000","last_peered":"2023-10-16T09:43:27.995287+0000","last_clean":"2023-10-16T09:43:27.995287+0000","last_became_active":"2023-10-16T09:43:25.960212+0000","last_became_peered":"2023-10-16T09:43:25.960212+0000","last_unstale":"2023-10-16T09:43:27.995287+0000","last_undegraded":"2023-10-16T09:43:27.995287+0000","last_fullsized":"2023-10-16T09:43:27.995287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:31.688570+0000","last_change":"2023-10-16T09:43:20.927651+0000","last_active":"2023-10-16T09:43:31.688570+0000","last_peered":"2023-10-16T09:43:31.688570+0000","last_clean":"2023-10-16T09:43:31.688570+0000","last_became_active":"2023-10-16T09:43:20.927393+0000","last_became_peered":"2023-10-16T09:43:20.927393+0000","last_unstale":"2023-10-16T09:43:31.688570+0000","last_undegraded":"2023-10-16T09:43:31.688570+0000","last_fullsized":"2023-10-16T09:43:31.688570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944933+0000","last_change":"2023-10-16T09:43:21.922547+0000","last_active":"2023-10-16T09:43:25.944933+0000","last_peered":"2023-10-16T09:43:25.944933+0000","last_clean":"2023-10-16T09:43:25.944933+0000","last_became_active":"2023-10-16T09:43:21.922309+0000","last_became_peered":"2023-10-16T09:43:21.922309+0000","last_unstale":"2023-10-16T09:43:25.944933+0000","last_undegraded":"2023-10-16T09:43:25.944933+0000","last_fullsized":"2023-10-16T09:43:25.944933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016142+0000","last_change":"2023-10-16T09:43:25.973055+0000","last_active":"2023-10-16T09:43:28.016142+0000","last_peered":"2023-10-16T09:43:28.016142+0000","last_clean":"2023-10-16T09:43:28.016142+0000","last_became_active":"2023-10-16T09:43:25.972664+0000","last_became_peered":"2023-10-16T09:43:25.972664+0000","last_unstale":"2023-10-16T09:43:28.016142+0000","last_undegraded":"2023-10-16T09:43:28.016142+0000","last_fullsized":"2023-10-16T09:43:28.016142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948288+0000","last_change":"2023-10-16T09:43:20.933659+0000","last_active":"2023-10-16T09:43:25.948288+0000","last_peered":"2023-10-16T09:43:25.948288+0000","last_clean":"2023-10-16T09:43:25.948288+0000","last_became_active":"2023-10-16T09:43:20.933376+0000","last_became_peered":"2023-10-16T09:43:20.933376+0000","last_unstale":"2023-10-16T09:43:25.948288+0000","last_undegraded":"2023-10-16T09:43:25.948288+0000","last_fullsized":"2023-10-16T09:43:25.948288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948438+0000","last_change":"2023-10-16T09:43:21.925525+0000","last_active":"2023-10-16T09:43:25.948438+0000","last_peered":"2023-10-16T09:43:25.948438+0000","last_clean":"2023-10-16T09:43:25.948438+0000","last_became_active":"2023-10-16T09:43:21.925269+0000","last_became_peered":"2023-10-16T09:43:21.925269+0000","last_unstale":"2023-10-16T09:43:25.948438+0000","last_undegraded":"2023-10-16T09:43:25.948438+0000","last_fullsized":"2023-10-16T09:43:25.948438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:39.205632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016287+0000","last_change":"2023-10-16T09:43:25.974917+0000","last_active":"2023-10-16T09:43:28.016287+0000","last_peered":"2023-10-16T09:43:28.016287+0000","last_clean":"2023-10-16T09:43:28.016287+0000","last_became_active":"2023-10-16T09:43:25.974512+0000","last_became_peered":"2023-10-16T09:43:25.974512+0000","last_unstale":"2023-10-16T09:43:28.016287+0000","last_undegraded":"2023-10-16T09:43:28.016287+0000","last_fullsized":"2023-10-16T09:43:28.016287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:24.186922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950390+0000","last_change":"2023-10-16T09:43:20.927369+0000","last_active":"2023-10-16T09:43:25.950390+0000","last_peered":"2023-10-16T09:43:25.950390+0000","last_clean":"2023-10-16T09:43:25.950390+0000","last_became_active":"2023-10-16T09:43:20.927043+0000","last_became_peered":"2023-10-16T09:43:20.927043+0000","last_unstale":"2023-10-16T09:43:25.950390+0000","last_undegraded":"2023-10-16T09:43:25.950390+0000","last_fullsized":"2023-10-16T09:43:25.950390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:11.929994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945029+0000","last_change":"2023-10-16T09:43:21.930235+0000","last_active":"2023-10-16T09:43:25.945029+0000","last_peered":"2023-10-16T09:43:25.945029+0000","last_clean":"2023-10-16T09:43:25.945029+0000","last_became_active":"2023-10-16T09:43:21.929925+0000","last_became_peered":"2023-10-16T09:43:21.929925+0000","last_unstale":"2023-10-16T09:43:25.945029+0000","last_undegraded":"2023-10-16T09:43:25.945029+0000","last_fullsized":"2023-10-16T09:43:25.945029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.016497+0000","last_change":"2023-10-16T09:43:25.975868+0000","last_active":"2023-10-16T09:43:28.016497+0000","last_peered":"2023-10-16T09:43:28.016497+0000","last_clean":"2023-10-16T09:43:28.016497+0000","last_became_active":"2023-10-16T09:43:25.975476+0000","last_became_peered":"2023-10-16T09:43:25.975476+0000","last_unstale":"2023-10-16T09:43:28.016497+0000","last_undegraded":"2023-10-16T09:43:28.016497+0000","last_fullsized":"2023-10-16T09:43:28.016497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:36.344621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954142+0000","last_change":"2023-10-16T09:43:20.922948+0000","last_active":"2023-10-16T09:43:25.954142+0000","last_peered":"2023-10-16T09:43:25.954142+0000","last_clean":"2023-10-16T09:43:25.954142+0000","last_became_active":"2023-10-16T09:43:20.922660+0000","last_became_peered":"2023-10-16T09:43:20.922660+0000","last_unstale":"2023-10-16T09:43:25.954142+0000","last_undegraded":"2023-10-16T09:43:25.954142+0000","last_fullsized":"2023-10-16T09:43:25.954142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:24.786762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953944+0000","last_change":"2023-10-16T09:43:21.930450+0000","last_active":"2023-10-16T09:43:25.953944+0000","last_peered":"2023-10-16T09:43:25.953944+0000","last_clean":"2023-10-16T09:43:25.953944+0000","last_became_active":"2023-10-16T09:43:21.929848+0000","last_became_peered":"2023-10-16T09:43:21.929848+0000","last_unstale":"2023-10-16T09:43:25.953944+0000","last_undegraded":"2023-10-16T09:43:25.953944+0000","last_fullsized":"2023-10-16T09:43:25.953944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015695+0000","last_change":"2023-10-16T09:43:25.972951+0000","last_active":"2023-10-16T09:43:28.015695+0000","last_peered":"2023-10-16T09:43:28.015695+0000","last_clean":"2023-10-16T09:43:28.015695+0000","last_became_active":"2023-10-16T09:43:25.972321+0000","last_became_peered":"2023-10-16T09:43:25.972321+0000","last_unstale":"2023-10-16T09:43:28.015695+0000","last_undegraded":"2023-10-16T09:43:28.015695+0000","last_fullsized":"2023-10-16T09:43:28.015695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:19.852112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948561+0000","last_change":"2023-10-16T09:43:20.919381+0000","last_active":"2023-10-16T09:43:25.948561+0000","last_peered":"2023-10-16T09:43:25.948561+0000","last_clean":"2023-10-16T09:43:25.948561+0000","last_became_active":"2023-10-16T09:43:20.919099+0000","last_became_peered":"2023-10-16T09:43:20.919099+0000","last_unstale":"2023-10-16T09:43:25.948561+0000","last_undegraded":"2023-10-16T09:43:25.948561+0000","last_fullsized":"2023-10-16T09:43:25.948561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.153559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944510+0000","last_change":"2023-10-16T09:43:21.930254+0000","last_active":"2023-10-16T09:43:25.944510+0000","last_peered":"2023-10-16T09:43:25.944510+0000","last_clean":"2023-10-16T09:43:25.944510+0000","last_became_active":"2023-10-16T09:43:21.929903+0000","last_became_peered":"2023-10-16T09:43:21.929903+0000","last_unstale":"2023-10-16T09:43:25.944510+0000","last_undegraded":"2023-10-16T09:43:25.944510+0000","last_fullsized":"2023-10-16T09:43:25.944510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947704+0000","last_change":"2023-10-16T09:43:25.973595+0000","last_active":"2023-10-16T09:43:26.947704+0000","last_peered":"2023-10-16T09:43:26.947704+0000","last_clean":"2023-10-16T09:43:26.947704+0000","last_became_active":"2023-10-16T09:43:25.972877+0000","last_became_peered":"2023-10-16T09:43:25.972877+0000","last_unstale":"2023-10-16T09:43:26.947704+0000","last_undegraded":"2023-10-16T09:43:26.947704+0000","last_fullsized":"2023-10-16T09:43:26.947704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:59.042186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.946744+0000","last_change":"2023-10-16T09:43:20.933953+0000","last_active":"2023-10-16T09:43:25.946744+0000","last_peered":"2023-10-16T09:43:25.946744+0000","last_clean":"2023-10-16T09:43:25.946744+0000","last_became_active":"2023-10-16T09:43:20.933628+0000","last_became_peered":"2023-10-16T09:43:20.933628+0000","last_unstale":"2023-10-16T09:43:25.946744+0000","last_undegraded":"2023-10-16T09:43:25.946744+0000","last_fullsized":"2023-10-16T09:43:25.946744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:06.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944812+0000","last_change":"2023-10-16T09:43:21.922710+0000","last_active":"2023-10-16T09:43:25.944812+0000","last_peered":"2023-10-16T09:43:25.944812+0000","last_clean":"2023-10-16T09:43:25.944812+0000","last_became_active":"2023-10-16T09:43:21.921824+0000","last_became_peered":"2023-10-16T09:43:21.921824+0000","last_unstale":"2023-10-16T09:43:25.944812+0000","last_undegraded":"2023-10-16T09:43:25.944812+0000","last_fullsized":"2023-10-16T09:43:25.944812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:50.477091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995171+0000","last_change":"2023-10-16T09:43:25.960646+0000","last_active":"2023-10-16T09:43:27.995171+0000","last_peered":"2023-10-16T09:43:27.995171+0000","last_clean":"2023-10-16T09:43:27.995171+0000","last_became_active":"2023-10-16T09:43:25.960270+0000","last_became_peered":"2023-10-16T09:43:25.960270+0000","last_unstale":"2023-10-16T09:43:27.995171+0000","last_undegraded":"2023-10-16T09:43:27.995171+0000","last_fullsized":"2023-10-16T09:43:27.995171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955611+0000","last_change":"2023-10-16T09:43:20.935197+0000","last_active":"2023-10-16T09:43:25.955611+0000","last_peered":"2023-10-16T09:43:25.955611+0000","last_clean":"2023-10-16T09:43:25.955611+0000","last_became_active":"2023-10-16T09:43:20.934929+0000","last_became_peered":"2023-10-16T09:43:20.934929+0000","last_unstale":"2023-10-16T09:43:25.955611+0000","last_undegraded":"2023-10-16T09:43:25.955611+0000","last_fullsized":"2023-10-16T09:43:25.955611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:49.049962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953179+0000","last_change":"2023-10-16T09:43:21.936959+0000","last_active":"2023-10-16T09:43:25.953179+0000","last_peered":"2023-10-16T09:43:25.953179+0000","last_clean":"2023-10-16T09:43:25.953179+0000","last_became_active":"2023-10-16T09:43:21.932598+0000","last_became_peered":"2023-10-16T09:43:21.932598+0000","last_unstale":"2023-10-16T09:43:25.953179+0000","last_undegraded":"2023-10-16T09:43:25.953179+0000","last_fullsized":"2023-10-16T09:43:25.953179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:41:49.075419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995359+0000","last_change":"2023-10-16T09:43:25.976441+0000","last_active":"2023-10-16T09:43:27.995359+0000","last_peered":"2023-10-16T09:43:27.995359+0000","last_clean":"2023-10-16T09:43:27.995359+0000","last_became_active":"2023-10-16T09:43:25.975914+0000","last_became_peered":"2023-10-16T09:43:25.975914+0000","last_unstale":"2023-10-16T09:43:27.995359+0000","last_undegraded":"2023-10-16T09:43:27.995359+0000","last_fullsized":"2023-10-16T09:43:27.995359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950296+0000","last_change":"2023-10-16T09:43:20.925613+0000","last_active":"2023-10-16T09:43:25.950296+0000","last_peered":"2023-10-16T09:43:25.950296+0000","last_clean":"2023-10-16T09:43:25.950296+0000","last_became_active":"2023-10-16T09:43:20.925345+0000","last_became_peered":"2023-10-16T09:43:20.925345+0000","last_unstale":"2023-10-16T09:43:25.950296+0000","last_undegraded":"2023-10-16T09:43:25.950296+0000","last_fullsized":"2023-10-16T09:43:25.950296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:28.608301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948410+0000","last_change":"2023-10-16T09:43:21.925702+0000","last_active":"2023-10-16T09:43:25.948410+0000","last_peered":"2023-10-16T09:43:25.948410+0000","last_clean":"2023-10-16T09:43:25.948410+0000","last_became_active":"2023-10-16T09:43:21.925003+0000","last_became_peered":"2023-10-16T09:43:21.925003+0000","last_unstale":"2023-10-16T09:43:25.948410+0000","last_undegraded":"2023-10-16T09:43:25.948410+0000","last_fullsized":"2023-10-16T09:43:25.948410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:57.933166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995813+0000","last_change":"2023-10-16T09:43:25.976511+0000","last_active":"2023-10-16T09:43:27.995813+0000","last_peered":"2023-10-16T09:43:27.995813+0000","last_clean":"2023-10-16T09:43:27.995813+0000","last_became_active":"2023-10-16T09:43:25.975258+0000","last_became_peered":"2023-10-16T09:43:25.975258+0000","last_unstale":"2023-10-16T09:43:27.995813+0000","last_undegraded":"2023-10-16T09:43:27.995813+0000","last_fullsized":"2023-10-16T09:43:27.995813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:59:15.277469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953939+0000","last_change":"2023-10-16T09:43:21.924235+0000","last_active":"2023-10-16T09:43:25.953939+0000","last_peered":"2023-10-16T09:43:25.953939+0000","last_clean":"2023-10-16T09:43:25.953939+0000","last_became_active":"2023-10-16T09:43:21.923963+0000","last_became_peered":"2023-10-16T09:43:21.923963+0000","last_unstale":"2023-10-16T09:43:25.953939+0000","last_undegraded":"2023-10-16T09:43:25.953939+0000","last_fullsized":"2023-10-16T09:43:25.953939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948171+0000","last_change":"2023-10-16T09:43:20.922462+0000","last_active":"2023-10-16T09:43:25.948171+0000","last_peered":"2023-10-16T09:43:25.948171+0000","last_clean":"2023-10-16T09:43:25.948171+0000","last_became_active":"2023-10-16T09:43:20.922204+0000","last_became_peered":"2023-10-16T09:43:20.922204+0000","last_unstale":"2023-10-16T09:43:25.948171+0000","last_undegraded":"2023-10-16T09:43:25.948171+0000","last_fullsized":"2023-10-16T09:43:25.948171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.949005+0000","last_change":"2023-10-16T09:43:25.975066+0000","last_active":"2023-10-16T09:43:26.949005+0000","last_peered":"2023-10-16T09:43:26.949005+0000","last_clean":"2023-10-16T09:43:26.949005+0000","last_became_active":"2023-10-16T09:43:25.974685+0000","last_became_peered":"2023-10-16T09:43:25.974685+0000","last_unstale":"2023-10-16T09:43:26.949005+0000","last_undegraded":"2023-10-16T09:43:26.949005+0000","last_fullsized":"2023-10-16T09:43:26.949005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966726+0000","last_change":"2023-10-16T09:43:21.940037+0000","last_active":"2023-10-16T09:43:25.966726+0000","last_peered":"2023-10-16T09:43:25.966726+0000","last_clean":"2023-10-16T09:43:25.966726+0000","last_became_active":"2023-10-16T09:43:21.939645+0000","last_became_peered":"2023-10-16T09:43:21.939645+0000","last_unstale":"2023-10-16T09:43:25.966726+0000","last_undegraded":"2023-10-16T09:43:25.966726+0000","last_fullsized":"2023-10-16T09:43:25.966726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950263+0000","last_change":"2023-10-16T09:43:20.925442+0000","last_active":"2023-10-16T09:43:25.950263+0000","last_peered":"2023-10-16T09:43:25.950263+0000","last_clean":"2023-10-16T09:43:25.950263+0000","last_became_active":"2023-10-16T09:43:20.925129+0000","last_became_peered":"2023-10-16T09:43:20.925129+0000","last_unstale":"2023-10-16T09:43:25.950263+0000","last_undegraded":"2023-10-16T09:43:25.950263+0000","last_fullsized":"2023-10-16T09:43:25.950263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015775+0000","last_change":"2023-10-16T09:43:25.961486+0000","last_active":"2023-10-16T09:43:28.015775+0000","last_peered":"2023-10-16T09:43:28.015775+0000","last_clean":"2023-10-16T09:43:28.015775+0000","last_became_active":"2023-10-16T09:43:25.961120+0000","last_became_peered":"2023-10-16T09:43:25.961120+0000","last_unstale":"2023-10-16T09:43:28.015775+0000","last_undegraded":"2023-10-16T09:43:28.015775+0000","last_fullsized":"2023-10-16T09:43:28.015775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:06.955721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947444+0000","last_change":"2023-10-16T09:43:21.929303+0000","last_active":"2023-10-16T09:43:25.947444+0000","last_peered":"2023-10-16T09:43:25.947444+0000","last_clean":"2023-10-16T09:43:25.947444+0000","last_became_active":"2023-10-16T09:43:21.929017+0000","last_became_peered":"2023-10-16T09:43:21.929017+0000","last_unstale":"2023-10-16T09:43:25.947444+0000","last_undegraded":"2023-10-16T09:43:25.947444+0000","last_fullsized":"2023-10-16T09:43:25.947444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:45.433460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944422+0000","last_change":"2023-10-16T09:43:20.925419+0000","last_active":"2023-10-16T09:43:25.944422+0000","last_peered":"2023-10-16T09:43:25.944422+0000","last_clean":"2023-10-16T09:43:25.944422+0000","last_became_active":"2023-10-16T09:43:20.924900+0000","last_became_peered":"2023-10-16T09:43:20.924900+0000","last_unstale":"2023-10-16T09:43:25.944422+0000","last_undegraded":"2023-10-16T09:43:25.944422+0000","last_fullsized":"2023-10-16T09:43:25.944422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:05.691906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.960836+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.959879+0000","last_became_peered":"2023-10-16T09:43:25.959879+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:24.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944657+0000","last_change":"2023-10-16T09:43:20.935065+0000","last_active":"2023-10-16T09:43:25.944657+0000","last_peered":"2023-10-16T09:43:25.944657+0000","last_clean":"2023-10-16T09:43:25.944657+0000","last_became_active":"2023-10-16T09:43:20.934870+0000","last_became_peered":"2023-10-16T09:43:20.934870+0000","last_unstale":"2023-10-16T09:43:25.944657+0000","last_undegraded":"2023-10-16T09:43:25.944657+0000","last_fullsized":"2023-10-16T09:43:25.944657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955720+0000","last_change":"2023-10-16T09:43:21.924250+0000","last_active":"2023-10-16T09:43:25.955720+0000","last_peered":"2023-10-16T09:43:25.955720+0000","last_clean":"2023-10-16T09:43:25.955720+0000","last_became_active":"2023-10-16T09:43:21.923974+0000","last_became_peered":"2023-10-16T09:43:21.923974+0000","last_unstale":"2023-10-16T09:43:25.955720+0000","last_undegraded":"2023-10-16T09:43:25.955720+0000","last_fullsized":"2023-10-16T09:43:25.955720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015042+0000","last_change":"2023-10-16T09:43:25.974974+0000","last_active":"2023-10-16T09:43:28.015042+0000","last_peered":"2023-10-16T09:43:28.015042+0000","last_clean":"2023-10-16T09:43:28.015042+0000","last_became_active":"2023-10-16T09:43:25.974574+0000","last_became_peered":"2023-10-16T09:43:25.974574+0000","last_unstale":"2023-10-16T09:43:28.015042+0000","last_undegraded":"2023-10-16T09:43:28.015042+0000","last_fullsized":"2023-10-16T09:43:28.015042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955603+0000","last_change":"2023-10-16T09:43:20.928340+0000","last_active":"2023-10-16T09:43:25.955603+0000","last_peered":"2023-10-16T09:43:25.955603+0000","last_clean":"2023-10-16T09:43:25.955603+0000","last_became_active":"2023-10-16T09:43:20.927748+0000","last_became_peered":"2023-10-16T09:43:20.927748+0000","last_unstale":"2023-10-16T09:43:25.955603+0000","last_undegraded":"2023-10-16T09:43:25.955603+0000","last_fullsized":"2023-10-16T09:43:25.955603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:16.172400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967050+0000","last_change":"2023-10-16T09:43:21.919707+0000","last_active":"2023-10-16T09:43:25.967050+0000","last_peered":"2023-10-16T09:43:25.967050+0000","last_clean":"2023-10-16T09:43:25.967050+0000","last_became_active":"2023-10-16T09:43:21.919454+0000","last_became_peered":"2023-10-16T09:43:21.919454+0000","last_unstale":"2023-10-16T09:43:25.967050+0000","last_undegraded":"2023-10-16T09:43:25.967050+0000","last_fullsized":"2023-10-16T09:43:25.967050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.947855+0000","last_change":"2023-10-16T09:43:25.961225+0000","last_active":"2023-10-16T09:43:26.947855+0000","last_peered":"2023-10-16T09:43:26.947855+0000","last_clean":"2023-10-16T09:43:26.947855+0000","last_became_active":"2023-10-16T09:43:25.960336+0000","last_became_peered":"2023-10-16T09:43:25.960336+0000","last_unstale":"2023-10-16T09:43:26.947855+0000","last_undegraded":"2023-10-16T09:43:26.947855+0000","last_fullsized":"2023-10-16T09:43:26.947855+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:15.617593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956170+0000","last_change":"2023-10-16T09:43:20.928799+0000","last_active":"2023-10-16T09:43:25.956170+0000","last_peered":"2023-10-16T09:43:25.956170+0000","last_clean":"2023-10-16T09:43:25.956170+0000","last_became_active":"2023-10-16T09:43:20.928302+0000","last_became_peered":"2023-10-16T09:43:20.928302+0000","last_unstale":"2023-10-16T09:43:25.956170+0000","last_undegraded":"2023-10-16T09:43:25.956170+0000","last_fullsized":"2023-10-16T09:43:25.956170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.951319+0000","last_change":"2023-10-16T09:43:21.924602+0000","last_active":"2023-10-16T09:43:25.951319+0000","last_peered":"2023-10-16T09:43:25.951319+0000","last_clean":"2023-10-16T09:43:25.951319+0000","last_became_active":"2023-10-16T09:43:21.924344+0000","last_became_peered":"2023-10-16T09:43:21.924344+0000","last_unstale":"2023-10-16T09:43:25.951319+0000","last_undegraded":"2023-10-16T09:43:25.951319+0000","last_fullsized":"2023-10-16T09:43:25.951319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948047+0000","last_change":"2023-10-16T09:43:25.969751+0000","last_active":"2023-10-16T09:43:26.948047+0000","last_peered":"2023-10-16T09:43:26.948047+0000","last_clean":"2023-10-16T09:43:26.948047+0000","last_became_active":"2023-10-16T09:43:25.969399+0000","last_became_peered":"2023-10-16T09:43:25.969399+0000","last_unstale":"2023-10-16T09:43:26.948047+0000","last_undegraded":"2023-10-16T09:43:26.948047+0000","last_fullsized":"2023-10-16T09:43:26.948047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944316+0000","last_change":"2023-10-16T09:43:20.933882+0000","last_active":"2023-10-16T09:43:25.944316+0000","last_peered":"2023-10-16T09:43:25.944316+0000","last_clean":"2023-10-16T09:43:25.944316+0000","last_became_active":"2023-10-16T09:43:20.933615+0000","last_became_peered":"2023-10-16T09:43:20.933615+0000","last_unstale":"2023-10-16T09:43:25.944316+0000","last_undegraded":"2023-10-16T09:43:25.944316+0000","last_fullsized":"2023-10-16T09:43:25.944316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.947408+0000","last_change":"2023-10-16T09:43:21.929135+0000","last_active":"2023-10-16T09:43:25.947408+0000","last_peered":"2023-10-16T09:43:25.947408+0000","last_clean":"2023-10-16T09:43:25.947408+0000","last_became_active":"2023-10-16T09:43:21.928849+0000","last_became_peered":"2023-10-16T09:43:21.928849+0000","last_unstale":"2023-10-16T09:43:25.947408+0000","last_undegraded":"2023-10-16T09:43:25.947408+0000","last_fullsized":"2023-10-16T09:43:25.947408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:27.995596+0000","last_change":"2023-10-16T09:43:25.976682+0000","last_active":"2023-10-16T09:43:27.995596+0000","last_peered":"2023-10-16T09:43:27.995596+0000","last_clean":"2023-10-16T09:43:27.995596+0000","last_became_active":"2023-10-16T09:43:25.976274+0000","last_became_peered":"2023-10-16T09:43:25.976274+0000","last_unstale":"2023-10-16T09:43:27.995596+0000","last_undegraded":"2023-10-16T09:43:27.995596+0000","last_fullsized":"2023-10-16T09:43:27.995596+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954044+0000","last_change":"2023-10-16T09:43:20.926509+0000","last_active":"2023-10-16T09:43:25.954044+0000","last_peered":"2023-10-16T09:43:25.954044+0000","last_clean":"2023-10-16T09:43:25.954044+0000","last_became_active":"2023-10-16T09:43:20.926228+0000","last_became_peered":"2023-10-16T09:43:20.926228+0000","last_unstale":"2023-10-16T09:43:25.954044+0000","last_undegraded":"2023-10-16T09:43:25.954044+0000","last_fullsized":"2023-10-16T09:43:25.954044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.424479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965799+0000","last_change":"2023-10-16T09:43:21.924014+0000","last_active":"2023-10-16T09:43:25.965799+0000","last_peered":"2023-10-16T09:43:25.965799+0000","last_clean":"2023-10-16T09:43:25.965799+0000","last_became_active":"2023-10-16T09:43:21.922684+0000","last_became_peered":"2023-10-16T09:43:21.922684+0000","last_unstale":"2023-10-16T09:43:25.965799+0000","last_undegraded":"2023-10-16T09:43:25.965799+0000","last_fullsized":"2023-10-16T09:43:25.965799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948333+0000","last_change":"2023-10-16T09:43:25.970563+0000","last_active":"2023-10-16T09:43:26.948333+0000","last_peered":"2023-10-16T09:43:26.948333+0000","last_clean":"2023-10-16T09:43:26.948333+0000","last_became_active":"2023-10-16T09:43:25.970168+0000","last_became_peered":"2023-10-16T09:43:25.970168+0000","last_unstale":"2023-10-16T09:43:26.948333+0000","last_undegraded":"2023-10-16T09:43:26.948333+0000","last_fullsized":"2023-10-16T09:43:26.948333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:56.567461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953937+0000","last_change":"2023-10-16T09:43:20.923587+0000","last_active":"2023-10-16T09:43:25.953937+0000","last_peered":"2023-10-16T09:43:25.953937+0000","last_clean":"2023-10-16T09:43:25.953937+0000","last_became_active":"2023-10-16T09:43:20.923277+0000","last_became_peered":"2023-10-16T09:43:20.923277+0000","last_unstale":"2023-10-16T09:43:25.953937+0000","last_undegraded":"2023-10-16T09:43:25.953937+0000","last_fullsized":"2023-10-16T09:43:25.953937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:40.360440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966226+0000","last_change":"2023-10-16T09:43:21.925711+0000","last_active":"2023-10-16T09:43:25.966226+0000","last_peered":"2023-10-16T09:43:25.966226+0000","last_clean":"2023-10-16T09:43:25.966226+0000","last_became_active":"2023-10-16T09:43:21.921509+0000","last_became_peered":"2023-10-16T09:43:21.921509+0000","last_unstale":"2023-10-16T09:43:25.966226+0000","last_undegraded":"2023-10-16T09:43:25.966226+0000","last_fullsized":"2023-10-16T09:43:25.966226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948182+0000","last_change":"2023-10-16T09:43:25.955876+0000","last_active":"2023-10-16T09:43:26.948182+0000","last_peered":"2023-10-16T09:43:26.948182+0000","last_clean":"2023-10-16T09:43:26.948182+0000","last_became_active":"2023-10-16T09:43:25.955490+0000","last_became_peered":"2023-10-16T09:43:25.955490+0000","last_unstale":"2023-10-16T09:43:26.948182+0000","last_undegraded":"2023-10-16T09:43:26.948182+0000","last_fullsized":"2023-10-16T09:43:26.948182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:49:51.452806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.945113+0000","last_change":"2023-10-16T09:43:20.926395+0000","last_active":"2023-10-16T09:43:25.945113+0000","last_peered":"2023-10-16T09:43:25.945113+0000","last_clean":"2023-10-16T09:43:25.945113+0000","last_became_active":"2023-10-16T09:43:20.926086+0000","last_became_peered":"2023-10-16T09:43:20.926086+0000","last_unstale":"2023-10-16T09:43:25.945113+0000","last_undegraded":"2023-10-16T09:43:25.945113+0000","last_fullsized":"2023-10-16T09:43:25.945113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:56.924185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.957128+0000","last_change":"2023-10-16T09:43:21.932460+0000","last_active":"2023-10-16T09:43:25.957128+0000","last_peered":"2023-10-16T09:43:25.957128+0000","last_clean":"2023-10-16T09:43:25.957128+0000","last_became_active":"2023-10-16T09:43:21.932184+0000","last_became_peered":"2023-10-16T09:43:21.932184+0000","last_unstale":"2023-10-16T09:43:25.957128+0000","last_undegraded":"2023-10-16T09:43:25.957128+0000","last_fullsized":"2023-10-16T09:43:25.957128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:26.948895+0000","last_change":"2023-10-16T09:43:25.967183+0000","last_active":"2023-10-16T09:43:26.948895+0000","last_peered":"2023-10-16T09:43:26.948895+0000","last_clean":"2023-10-16T09:43:26.948895+0000","last_became_active":"2023-10-16T09:43:25.966759+0000","last_became_peered":"2023-10-16T09:43:25.966759+0000","last_unstale":"2023-10-16T09:43:26.948895+0000","last_undegraded":"2023-10-16T09:43:26.948895+0000","last_fullsized":"2023-10-16T09:43:26.948895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:04.227287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948721+0000","last_change":"2023-10-16T09:43:20.925234+0000","last_active":"2023-10-16T09:43:25.948721+0000","last_peered":"2023-10-16T09:43:25.948721+0000","last_clean":"2023-10-16T09:43:25.948721+0000","last_became_active":"2023-10-16T09:43:20.924957+0000","last_became_peered":"2023-10-16T09:43:20.924957+0000","last_unstale":"2023-10-16T09:43:25.948721+0000","last_undegraded":"2023-10-16T09:43:25.948721+0000","last_fullsized":"2023-10-16T09:43:25.948721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:00.084223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944746+0000","last_change":"2023-10-16T09:43:21.936106+0000","last_active":"2023-10-16T09:43:25.944746+0000","last_peered":"2023-10-16T09:43:25.944746+0000","last_clean":"2023-10-16T09:43:25.944746+0000","last_became_active":"2023-10-16T09:43:21.935859+0000","last_became_peered":"2023-10-16T09:43:21.935859+0000","last_unstale":"2023-10-16T09:43:25.944746+0000","last_undegraded":"2023-10-16T09:43:25.944746+0000","last_fullsized":"2023-10-16T09:43:25.944746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:00.196804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.014901+0000","last_change":"2023-10-16T09:43:25.961369+0000","last_active":"2023-10-16T09:43:28.014901+0000","last_peered":"2023-10-16T09:43:28.014901+0000","last_clean":"2023-10-16T09:43:28.014901+0000","last_became_active":"2023-10-16T09:43:25.960465+0000","last_became_peered":"2023-10-16T09:43:25.960465+0000","last_unstale":"2023-10-16T09:43:28.014901+0000","last_undegraded":"2023-10-16T09:43:28.014901+0000","last_fullsized":"2023-10-16T09:43:28.014901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:40.952070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955514+0000","last_change":"2023-10-16T09:43:20.922927+0000","last_active":"2023-10-16T09:43:25.955514+0000","last_peered":"2023-10-16T09:43:25.955514+0000","last_clean":"2023-10-16T09:43:25.955514+0000","last_became_active":"2023-10-16T09:43:20.922658+0000","last_became_peered":"2023-10-16T09:43:20.922658+0000","last_unstale":"2023-10-16T09:43:25.955514+0000","last_undegraded":"2023-10-16T09:43:25.955514+0000","last_fullsized":"2023-10-16T09:43:25.955514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:23.898387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950404+0000","last_change":"2023-10-16T09:43:21.928420+0000","last_active":"2023-10-16T09:43:25.950404+0000","last_peered":"2023-10-16T09:43:25.950404+0000","last_clean":"2023-10-16T09:43:25.950404+0000","last_became_active":"2023-10-16T09:43:21.928180+0000","last_became_peered":"2023-10-16T09:43:21.928180+0000","last_unstale":"2023-10-16T09:43:25.950404+0000","last_undegraded":"2023-10-16T09:43:25.950404+0000","last_fullsized":"2023-10-16T09:43:25.950404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:56.942093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T09:43:28.015214+0000","last_change":"2023-10-16T09:43:25.960304+0000","last_active":"2023-10-16T09:43:28.015214+0000","last_peered":"2023-10-16T09:43:28.015214+0000","last_clean":"2023-10-16T09:43:28.015214+0000","last_became_active":"2023-10-16T09:43:25.959916+0000","last_became_peered":"2023-10-16T09:43:25.959916+0000","last_unstale":"2023-10-16T09:43:28.015214+0000","last_undegraded":"2023-10-16T09:43:28.015214+0000","last_fullsized":"2023-10-16T09:43:28.015214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:24.928047+0000","last_clean_scrub_stamp":"2023-10-16T09:43:24.928047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:14.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948747+0000","last_change":"2023-10-16T09:43:20.921666+0000","last_active":"2023-10-16T09:43:25.948747+0000","last_peered":"2023-10-16T09:43:25.948747+0000","last_clean":"2023-10-16T09:43:25.948747+0000","last_became_active":"2023-10-16T09:43:20.921373+0000","last_became_peered":"2023-10-16T09:43:20.921373+0000","last_unstale":"2023-10-16T09:43:25.948747+0000","last_undegraded":"2023-10-16T09:43:25.948747+0000","last_fullsized":"2023-10-16T09:43:25.948747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:19.888583+0000","last_clean_scrub_stamp":"2023-10-16T09:43:19.888583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:15.313940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.944941+0000","last_change":"2023-10-16T09:43:21.930965+0000","last_active":"2023-10-16T09:43:25.944941+0000","last_peered":"2023-10-16T09:43:25.944941+0000","last_clean":"2023-10-16T09:43:25.944941+0000","last_became_active":"2023-10-16T09:43:21.930701+0000","last_became_peered":"2023-10-16T09:43:21.930701+0000","last_unstale":"2023-10-16T09:43:25.944941+0000","last_undegraded":"2023-10-16T09:43:25.944941+0000","last_fullsized":"2023-10-16T09:43:25.944941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:20.329096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.954036+0000","last_change":"2023-10-16T09:43:21.930641+0000","last_active":"2023-10-16T09:43:25.954036+0000","last_peered":"2023-10-16T09:43:25.954036+0000","last_clean":"2023-10-16T09:43:25.954036+0000","last_became_active":"2023-10-16T09:43:21.930194+0000","last_became_peered":"2023-10-16T09:43:21.930194+0000","last_unstale":"2023-10-16T09:43:25.954036+0000","last_undegraded":"2023-10-16T09:43:25.954036+0000","last_fullsized":"2023-10-16T09:43:25.954036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948076+0000","last_change":"2023-10-16T09:43:21.933068+0000","last_active":"2023-10-16T09:43:25.948076+0000","last_peered":"2023-10-16T09:43:25.948076+0000","last_clean":"2023-10-16T09:43:25.948076+0000","last_became_active":"2023-10-16T09:43:21.932774+0000","last_became_peered":"2023-10-16T09:43:21.932774+0000","last_unstale":"2023-10-16T09:43:25.948076+0000","last_undegraded":"2023-10-16T09:43:25.948076+0000","last_fullsized":"2023-10-16T09:43:25.948076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:25.795743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950489+0000","last_change":"2023-10-16T09:43:21.937509+0000","last_active":"2023-10-16T09:43:25.950489+0000","last_peered":"2023-10-16T09:43:25.950489+0000","last_clean":"2023-10-16T09:43:25.950489+0000","last_became_active":"2023-10-16T09:43:21.936970+0000","last_became_peered":"2023-10-16T09:43:21.936970+0000","last_unstale":"2023-10-16T09:43:25.950489+0000","last_undegraded":"2023-10-16T09:43:25.950489+0000","last_fullsized":"2023-10-16T09:43:25.950489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:18:11.587110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948478+0000","last_change":"2023-10-16T09:43:21.927275+0000","last_active":"2023-10-16T09:43:25.948478+0000","last_peered":"2023-10-16T09:43:25.948478+0000","last_clean":"2023-10-16T09:43:25.948478+0000","last_became_active":"2023-10-16T09:43:21.927024+0000","last_became_peered":"2023-10-16T09:43:21.927024+0000","last_unstale":"2023-10-16T09:43:25.948478+0000","last_undegraded":"2023-10-16T09:43:25.948478+0000","last_fullsized":"2023-10-16T09:43:25.948478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:29.617282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.966374+0000","last_change":"2023-10-16T09:43:21.925070+0000","last_active":"2023-10-16T09:43:25.966374+0000","last_peered":"2023-10-16T09:43:25.966374+0000","last_clean":"2023-10-16T09:43:25.966374+0000","last_became_active":"2023-10-16T09:43:21.924799+0000","last_became_peered":"2023-10-16T09:43:21.924799+0000","last_unstale":"2023-10-16T09:43:25.966374+0000","last_undegraded":"2023-10-16T09:43:25.966374+0000","last_fullsized":"2023-10-16T09:43:25.966374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T09:54:55.011703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.955685+0000","last_change":"2023-10-16T09:43:21.920367+0000","last_active":"2023-10-16T09:43:25.955685+0000","last_peered":"2023-10-16T09:43:25.955685+0000","last_clean":"2023-10-16T09:43:25.955685+0000","last_became_active":"2023-10-16T09:43:21.919980+0000","last_became_peered":"2023-10-16T09:43:21.919980+0000","last_unstale":"2023-10-16T09:43:25.955685+0000","last_undegraded":"2023-10-16T09:43:25.955685+0000","last_fullsized":"2023-10-16T09:43:25.955685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:57.840800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.956021+0000","last_change":"2023-10-16T09:43:21.931239+0000","last_active":"2023-10-16T09:43:25.956021+0000","last_peered":"2023-10-16T09:43:25.956021+0000","last_clean":"2023-10-16T09:43:25.956021+0000","last_became_active":"2023-10-16T09:43:21.930923+0000","last_became_peered":"2023-10-16T09:43:21.930923+0000","last_unstale":"2023-10-16T09:43:25.956021+0000","last_undegraded":"2023-10-16T09:43:25.956021+0000","last_fullsized":"2023-10-16T09:43:25.956021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.967144+0000","last_change":"2023-10-16T09:43:21.924310+0000","last_active":"2023-10-16T09:43:25.967144+0000","last_peered":"2023-10-16T09:43:25.967144+0000","last_clean":"2023-10-16T09:43:25.967144+0000","last_became_active":"2023-10-16T09:43:21.921665+0000","last_became_peered":"2023-10-16T09:43:21.921665+0000","last_unstale":"2023-10-16T09:43:25.967144+0000","last_undegraded":"2023-10-16T09:43:25.967144+0000","last_fullsized":"2023-10-16T09:43:25.967144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:34.043630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950321+0000","last_change":"2023-10-16T09:43:21.927572+0000","last_active":"2023-10-16T09:43:25.950321+0000","last_peered":"2023-10-16T09:43:25.950321+0000","last_clean":"2023-10-16T09:43:25.950321+0000","last_became_active":"2023-10-16T09:43:21.927298+0000","last_became_peered":"2023-10-16T09:43:21.927298+0000","last_unstale":"2023-10-16T09:43:25.950321+0000","last_undegraded":"2023-10-16T09:43:25.950321+0000","last_fullsized":"2023-10-16T09:43:25.950321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:41.008923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.965935+0000","last_change":"2023-10-16T09:43:21.922799+0000","last_active":"2023-10-16T09:43:25.965935+0000","last_peered":"2023-10-16T09:43:25.965935+0000","last_clean":"2023-10-16T09:43:25.965935+0000","last_became_active":"2023-10-16T09:43:21.922126+0000","last_became_peered":"2023-10-16T09:43:21.922126+0000","last_unstale":"2023-10-16T09:43:25.965935+0000","last_undegraded":"2023-10-16T09:43:25.965935+0000","last_fullsized":"2023-10-16T09:43:25.965935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:12.938975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.950579+0000","last_change":"2023-10-16T09:43:21.936429+0000","last_active":"2023-10-16T09:43:25.950579+0000","last_peered":"2023-10-16T09:43:25.950579+0000","last_clean":"2023-10-16T09:43:25.950579+0000","last_became_active":"2023-10-16T09:43:21.933542+0000","last_became_peered":"2023-10-16T09:43:21.933542+0000","last_unstale":"2023-10-16T09:43:25.950579+0000","last_undegraded":"2023-10-16T09:43:25.950579+0000","last_fullsized":"2023-10-16T09:43:25.950579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:52.536978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.948591+0000","last_change":"2023-10-16T09:43:21.930439+0000","last_active":"2023-10-16T09:43:25.948591+0000","last_peered":"2023-10-16T09:43:25.948591+0000","last_clean":"2023-10-16T09:43:25.948591+0000","last_became_active":"2023-10-16T09:43:21.930237+0000","last_became_peered":"2023-10-16T09:43:21.930237+0000","last_unstale":"2023-10-16T09:43:25.948591+0000","last_undegraded":"2023-10-16T09:43:25.948591+0000","last_fullsized":"2023-10-16T09:43:25.948591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:17.181381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-16T09:43:25.953918+0000","last_change":"2023-10-16T09:43:21.934041+0000","last_active":"2023-10-16T09:43:25.953918+0000","last_peered":"2023-10-16T09:43:25.953918+0000","last_clean":"2023-10-16T09:43:25.953918+0000","last_became_active":"2023-10-16T09:43:21.933451+0000","last_became_peered":"2023-10-16T09:43:21.933451+0000","last_unstale":"2023-10-16T09:43:25.953918+0000","last_undegraded":"2023-10-16T09:43:25.953918+0000","last_fullsized":"2023-10-16T09:43:25.953918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:20.897564+0000","last_clean_scrub_stamp":"2023-10-16T09:43:20.897564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:24.160356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":99940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":99780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:43:50.585 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T09:43:50.585 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-16T09:43:50.585 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-16T09:43:50.586 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T09:43:50.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.736+0000 7fb6a8271700 1 Processor -- start 2023-10-16T09:43:50.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.736+0000 7fb6a8271700 1 -- start start 2023-10-16T09:43:50.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a8271700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a00c65d0 0x7fb6a006be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a8271700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6a006a8d0 0x7fb6a0161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a8271700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb6a00cde80 0x7fb6a00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a8271700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a0162240 con 0x7fb6a006a8d0 2023-10-16T09:43:50.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a8271700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a00ce830 con 0x7fb6a00cde80 2023-10-16T09:43:50.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a8271700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a006b970 con 0x7fb6a00c65d0 2023-10-16T09:43:50.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a580c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6a006a8d0 0x7fb6a0161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a580c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6a006a8d0 0x7fb6a0161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46818/0 (socket says 172.21.15.158:46818) 2023-10-16T09:43:50.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.737+0000 7fb6a580c700 1 -- 172.21.15.158:0/2978314489 learned_addr learned my addr 172.21.15.158:0/2978314489 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:50.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a680e700 1 --2- 172.21.15.158:0/2978314489 >> v2:172.21.15.172:3300/0 conn(0x7fb6a00cde80 0x7fb6a00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a600d700 1 --2- 172.21.15.158:0/2978314489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a00c65d0 0x7fb6a006be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a580c700 1 -- 172.21.15.158:0/2978314489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a00c65d0 msgr2=0x7fb6a006be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:50.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a580c700 1 --2- 172.21.15.158:0/2978314489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a00c65d0 0x7fb6a006be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a580c700 1 -- 172.21.15.158:0/2978314489 >> v2:172.21.15.172:3300/0 conn(0x7fb6a00cde80 msgr2=0x7fb6a00ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:50.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a580c700 1 --2- 172.21.15.158:0/2978314489 >> v2:172.21.15.172:3300/0 conn(0x7fb6a00cde80 0x7fb6a00ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a580c700 1 -- 172.21.15.158:0/2978314489 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a006ad60 con 0x7fb6a006a8d0 2023-10-16T09:43:50.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a680e700 1 --2- 172.21.15.158:0/2978314489 >> v2:172.21.15.172:3300/0 conn(0x7fb6a00cde80 0x7fb6a00ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:50.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a580c700 1 --2- 172.21.15.158:0/2978314489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a006a8d0 0x7fb6a0161d00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb6a015ed50 tx=0x7fb69c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85485133887727a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:50.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a500b700 1 -- 172.21.15.158:0/2978314489 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c00b860 con 0x7fb6a006a8d0 2023-10-16T09:43:50.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a500b700 1 -- 172.21.15.158:0/2978314489 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c00b9c0 con 0x7fb6a006a8d0 2023-10-16T09:43:50.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a500b700 1 -- 172.21.15.158:0/2978314489 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c00bd50 con 0x7fb6a006a8d0 2023-10-16T09:43:50.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a8271700 1 -- 172.21.15.158:0/2978314489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a006a8d0 msgr2=0x7fb6a0161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.738+0000 7fb6a8271700 1 --2- 172.21.15.158:0/2978314489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a006a8d0 0x7fb6a0161d00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb6a015ed50 tx=0x7fb69c009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:50.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.739+0000 7fb6a8271700 1 -- 172.21.15.158:0/2978314489 shutdown_connections 2023-10-16T09:43:50.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.739+0000 7fb6a8271700 1 --2- 172.21.15.158:0/2978314489 >> v2:172.21.15.172:3300/0 conn(0x7fb6a00cde80 0x7fb6a00ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.739+0000 7fb6a8271700 1 --2- 172.21.15.158:0/2978314489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a00c65d0 0x7fb6a006be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.739+0000 7fb6a8271700 1 --2- 172.21.15.158:0/2978314489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a006a8d0 0x7fb6a0161d00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.739+0000 7fb6a8271700 1 -- 172.21.15.158:0/2978314489 >> 172.21.15.158:0/2978314489 conn(0x7fb6a00cc4b0 msgr2=0x7fb6a00cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:50.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.739+0000 7fb6a8271700 1 -- 172.21.15.158:0/2978314489 shutdown_connections 2023-10-16T09:43:50.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.739+0000 7fb6a8271700 1 -- 172.21.15.158:0/2978314489 wait complete. 2023-10-16T09:43:50.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.740+0000 7fb6a8271700 1 Processor -- start 2023-10-16T09:43:50.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.740+0000 7fb6a8271700 1 -- start start 2023-10-16T09:43:50.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6a00c65d0 0x7fb6a01f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a00cde80 0x7fb6a00aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a014be60 0x7fb6a014b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00ae3d0 con 0x7fb6a00cde80 2023-10-16T09:43:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a0148e20 con 0x7fb6a00c65d0 2023-10-16T09:43:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00ad330 con 0x7fb6a014be60 2023-10-16T09:43:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a00cde80 0x7fb6a00aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a00cde80 0x7fb6a00aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46824/0 (socket says 172.21.15.158:46824) 2023-10-16T09:43:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 -- 172.21.15.158:0/1653011562 learned_addr learned my addr 172.21.15.158:0/1653011562 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a600d700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6a00c65d0 0x7fb6a01f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a680e700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a014be60 0x7fb6a014b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 -- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a014be60 msgr2=0x7fb6a014b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a014be60 0x7fb6a014b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 -- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6a00c65d0 msgr2=0x7fb6a01f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6a00c65d0 0x7fb6a01f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 -- 172.21.15.158:0/1653011562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c009550 con 0x7fb6a00cde80 2023-10-16T09:43:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a680e700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a014be60 0x7fb6a014b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a580c700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a00cde80 0x7fb6a00aea20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb69c000c00 tx=0x7fb69c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb696ffd700 1 -- 172.21.15.158:0/1653011562 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0042c0 con 0x7fb6a00cde80 2023-10-16T09:43:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb696ffd700 1 -- 172.21.15.158:0/1653011562 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c01f070 con 0x7fb6a00cde80 2023-10-16T09:43:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a00ad610 con 0x7fb6a00cde80 2023-10-16T09:43:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb696ffd700 1 -- 172.21.15.158:0/1653011562 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0036a0 con 0x7fb6a00cde80 2023-10-16T09:43:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.741+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a00adbd0 con 0x7fb6a00cde80 2023-10-16T09:43:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.742+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a00681b0 con 0x7fb6a00cde80 2023-10-16T09:43:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.743+0000 7fb696ffd700 1 -- 172.21.15.158:0/1653011562 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb69c021030 con 0x7fb6a00cde80 2023-10-16T09:43:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.743+0000 7fb696ffd700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb68406af00 0x7fb68406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.743+0000 7fb696ffd700 1 -- 172.21.15.158:0/1653011562 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb69c08ac70 con 0x7fb6a00cde80 2023-10-16T09:43:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.743+0000 7fb6a600d700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb68406af00 0x7fb68406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.743+0000 7fb6a600d700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb68406af00 0x7fb68406d3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb690009950 tx=0x7fb690008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:50.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.747+0000 7fb696ffd700 1 -- 172.21.15.158:0/1653011562 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb69c055a40 con 0x7fb6a00cde80 2023-10-16T09:43:50.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.980+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb6a00c4ba0 con 0x7fb6a00cde80 2023-10-16T09:43:50.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.980+0000 7fb696ffd700 1 -- 172.21.15.158:0/1653011562 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb69c059060 con 0x7fb6a00cde80 2023-10-16T09:43:50.982 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:50.983 INFO:teuthology.orchestra.run.smithi158.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-16T09:43:50.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb68406af00 msgr2=0x7fb68406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb68406af00 0x7fb68406d3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb690009950 tx=0x7fb690008040 comp rx=0 tx=0).stop 2023-10-16T09:43:50.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a00cde80 msgr2=0x7fb6a00aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a00cde80 0x7fb6a00aea20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb69c000c00 tx=0x7fb69c004750 comp rx=0 tx=0).stop 2023-10-16T09:43:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 shutdown_connections 2023-10-16T09:43:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb68406af00 0x7fb68406d3c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6a014be60 0x7fb6a014b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6a00c65d0 0x7fb6a01f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 --2- 172.21.15.158:0/1653011562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6a00cde80 0x7fb6a00aea20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 >> 172.21.15.158:0/1653011562 conn(0x7fb6a00cc4b0 msgr2=0x7fb6a006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 shutdown_connections 2023-10-16T09:43:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:50.982+0000 7fb6a8271700 1 -- 172.21.15.158:0/1653011562 wait complete. 2023-10-16T09:43:50.996 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-16T09:43:50.996 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:51.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.115+0000 7f67e4796700 1 Processor -- start 2023-10-16T09:43:51.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.115+0000 7f67e4796700 1 -- start start 2023-10-16T09:43:51.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e4796700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 0x7f67dc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e4796700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f67dc0729b0 0x7f67dc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e4796700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc071e50 con 0x7f67dc06e5d0 2023-10-16T09:43:51.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e4796700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67dc161270 con 0x7f67dc0729b0 2023-10-16T09:43:51.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e4796700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc161510 con 0x7f67dc06ab40 2023-10-16T09:43:51.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1d31700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f67dc06e5d0 0x7f67dc0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47008/0 (socket says 172.21.15.158:47008) 2023-10-16T09:43:51.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1d31700 1 -- 172.21.15.158:0/4079432216 learned_addr learned my addr 172.21.15.158:0/4079432216 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:51.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e2d33700 1 --2- 172.21.15.158:0/4079432216 >> v2:172.21.15.172:3300/0 conn(0x7f67dc0729b0 0x7f67dc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e2532700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 0x7f67dc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395416060 0 0) 0x7f67dc071e50 con 0x7f67dc06e5d0 2023-10-16T09:43:51.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c80035b0 con 0x7f67dc06e5d0 2023-10-16T09:43:51.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3907634110 0 0) 0x7f67c80035b0 con 0x7f67dc06e5d0 2023-10-16T09:43:51.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 msgr2=0x7f67dc06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 0x7f67dc06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 >> v2:172.21.15.172:3300/0 conn(0x7f67dc0729b0 msgr2=0x7f67dc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 --2- 172.21.15.158:0/4079432216 >> v2:172.21.15.172:3300/0 conn(0x7f67dc0729b0 0x7f67dc071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc1617f0 con 0x7f67dc06e5d0 2023-10-16T09:43:51.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f67cc001670 con 0x7f67dc06e5d0 2023-10-16T09:43:51.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.116+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 >> v1:172.21.15.158:6789/0 conn(0x7f67dc06e5d0 legacy=0x7f67dc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8004850 0x7f67c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 0x7f67c8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc161270 con 0x7f67c8003a00 2023-10-16T09:43:51.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc161510 con 0x7f67c8004850 2023-10-16T09:43:51.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c800ba10 con 0x7f67dc06ab40 2023-10-16T09:43:51.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2532700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1d31700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8004850 0x7f67c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2d33700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 0x7f67c8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2532700 1 -- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 msgr2=0x7f67c8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2532700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc06ab40 0x7f67c8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2532700 1 -- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8004850 msgr2=0x7f67c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2532700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8004850 0x7f67c8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2532700 1 -- 172.21.15.158:0/4079432216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c80035b0 con 0x7f67c8003a00 2023-10-16T09:43:51.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e2532700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67c8003e20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009b60 tx=0x7f67c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366924f5884f0a93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d800e6c0 con 0x7f67c8003a00 2023-10-16T09:43:51.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d800e820 con 0x7f67c8003a00 2023-10-16T09:43:51.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.117+0000 7f67e1530700 1 -- 172.21.15.158:0/4079432216 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d800eb90 con 0x7f67c8003a00 2023-10-16T09:43:51.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.118+0000 7f67e4796700 1 -- 172.21.15.158:0/4079432216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 msgr2=0x7f67c8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.118+0000 7f67e4796700 1 --2- 172.21.15.158:0/4079432216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67c8003e20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009b60 tx=0x7f67c8005880 comp rx=0 tx=0).stop 2023-10-16T09:43:51.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.118+0000 7f67e2d33700 1 -- 172.21.15.158:0/4079432216 reap_dead start 2023-10-16T09:43:51.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.118+0000 7f67e4796700 1 -- 172.21.15.158:0/4079432216 shutdown_connections 2023-10-16T09:43:51.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.118+0000 7f67e4796700 1 -- 172.21.15.158:0/4079432216 >> 172.21.15.158:0/4079432216 conn(0x7f67dc0d02b0 msgr2=0x7f67dc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:51.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.118+0000 7f67e4796700 1 -- 172.21.15.158:0/4079432216 shutdown_connections 2023-10-16T09:43:51.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.118+0000 7f67e4796700 1 -- 172.21.15.158:0/4079432216 wait complete. 2023-10-16T09:43:51.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.119+0000 7f67e4796700 1 Processor -- start 2023-10-16T09:43:51.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.119+0000 7f67e4796700 1 -- start start 2023-10-16T09:43:51.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e4796700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67dc0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e4796700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0b2e60 0x7f67dc149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e4796700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0b0bb0 0x7f67dc0b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e4796700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0b1570 con 0x7f67c8003a00 2023-10-16T09:43:51.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e4796700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc15de20 con 0x7f67dc0b2e60 2023-10-16T09:43:51.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e4796700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0c41c0 con 0x7f67dc0b0bb0 2023-10-16T09:43:51.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67dc0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67dc0c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46842/0 (socket says 172.21.15.158:46842) 2023-10-16T09:43:51.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 -- 172.21.15.158:0/1453121479 learned_addr learned my addr 172.21.15.158:0/1453121479 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:51.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e1d31700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0b2e60 0x7f67dc149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2d33700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0b0bb0 0x7f67dc0b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 -- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0b0bb0 msgr2=0x7f67dc0b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0b0bb0 0x7f67dc0b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 -- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0b2e60 msgr2=0x7f67dc149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0b2e60 0x7f67dc149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 -- 172.21.15.158:0/1453121479 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc0c4460 con 0x7f67c8003a00 2023-10-16T09:43:51.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e2532700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67dc0c3c80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f67d800e4b0 tx=0x7f67d80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67d2ffd700 1 -- 172.21.15.158:0/1453121479 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d800e6c0 con 0x7f67c8003a00 2023-10-16T09:43:51.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67d2ffd700 1 -- 172.21.15.158:0/1453121479 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d80038a0 con 0x7f67c8003a00 2023-10-16T09:43:51.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67d2ffd700 1 -- 172.21.15.158:0/1453121479 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8003c50 con 0x7f67c8003a00 2023-10-16T09:43:51.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.120+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc0c4740 con 0x7f67c8003a00 2023-10-16T09:43:51.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.121+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc0c5490 con 0x7f67c8003a00 2023-10-16T09:43:51.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.121+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc0c5800 con 0x7f67c8003a00 2023-10-16T09:43:51.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.122+0000 7f67d2ffd700 1 -- 172.21.15.158:0/1453121479 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f67d8013470 con 0x7f67c8003a00 2023-10-16T09:43:51.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.122+0000 7f67d2ffd700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f67c006af00 0x7f67c006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.122+0000 7f67d2ffd700 1 -- 172.21.15.158:0/1453121479 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f67d808b3d0 con 0x7f67c8003a00 2023-10-16T09:43:51.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.122+0000 7f67e1d31700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f67c006af00 0x7f67c006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.123+0000 7f67e1d31700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f67c006af00 0x7f67c006d3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0b35f0 tx=0x7f67cc007980 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.127+0000 7f67d2ffd700 1 -- 172.21.15.158:0/1453121479 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f67d8056340 con 0x7f67c8003a00 2023-10-16T09:43:51.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.319+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67dc0c52f0 con 0x7f67c8003a00 2023-10-16T09:43:51.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.320+0000 7f67d2ffd700 1 -- 172.21.15.158:0/1453121479 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f67dc0c52f0 con 0x7f67c8003a00 2023-10-16T09:43:51.322 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:51.322 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:51.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f67c006af00 msgr2=0x7f67c006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f67c006af00 0x7f67c006d3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0b35f0 tx=0x7f67cc007980 comp rx=0 tx=0).stop 2023-10-16T09:43:51.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 msgr2=0x7f67dc0c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67dc0c3c80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f67d800e4b0 tx=0x7f67d80047c0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 shutdown_connections 2023-10-16T09:43:51.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f67c006af00 0x7f67c006d3c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0b0bb0 0x7f67dc0b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8003a00 0x7f67dc0c3c80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 --2- 172.21.15.158:0/1453121479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0b2e60 0x7f67dc149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 >> 172.21.15.158:0/1453121479 conn(0x7f67dc0d02b0 msgr2=0x7f67dc06b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:51.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 shutdown_connections 2023-10-16T09:43:51.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.322+0000 7f67e4796700 1 -- 172.21.15.158:0/1453121479 wait complete. 2023-10-16T09:43:51.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:43:51.337 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:43:51.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.460+0000 7f3e3a7a9700 1 Processor -- start 2023-10-16T09:43:51.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.460+0000 7f3e3a7a9700 1 -- start start 2023-10-16T09:43:51.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e3a7a9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3e3415da40 0x7f3e340caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e3a7a9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e340cbd30 con 0x7f3e3406afe0 2023-10-16T09:43:51.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e3a7a9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e340cc1c0 con 0x7f3e340cc590 2023-10-16T09:43:51.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e3a7a9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3415de20 con 0x7f3e3415da40 2023-10-16T09:43:51.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e38d46700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3e3406afe0 0x7f3e340afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47030/0 (socket says 172.21.15.158:47030) 2023-10-16T09:43:51.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e38d46700 1 -- 172.21.15.158:0/2936617685 learned_addr learned my addr 172.21.15.158:0/2936617685 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:51.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588815736 0 0) 0x7f3e340cbd30 con 0x7f3e3406afe0 2023-10-16T09:43:51.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1c0035b0 con 0x7f3e3406afe0 2023-10-16T09:43:51.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e33fff700 1 --2- 172.21.15.158:0/2936617685 >> v2:172.21.15.172:3300/0 conn(0x7f3e3415da40 0x7f3e340caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47711014 0 0) 0x7f3e1c0035b0 con 0x7f3e3406afe0 2023-10-16T09:43:51.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 >> v2:172.21.15.172:3300/0 conn(0x7f3e3415da40 msgr2=0x7f3e340caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e32ffd700 1 --2- 172.21.15.158:0/2936617685 >> v2:172.21.15.172:3300/0 conn(0x7f3e3415da40 0x7f3e340caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 >> v1:172.21.15.172:6789/0 conn(0x7f3e340cc590 legacy=0x7f3e3406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.461+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e340ce260 con 0x7f3e3406afe0 2023-10-16T09:43:51.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3e2c0019f0 con 0x7f3e3406afe0 2023-10-16T09:43:51.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 >> v1:172.21.15.158:6789/0 conn(0x7f3e3406afe0 legacy=0x7f3e340afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e1c003a00 0x7f3e1c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e1c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e1c005210 0x7f3e1c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3415de20 con 0x7f3e1c004850 2023-10-16T09:43:51.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c00c0b0 con 0x7f3e1c003a00 2023-10-16T09:43:51.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c00c270 con 0x7f3e1c005210 2023-10-16T09:43:51.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e337fe700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e1c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e33fff700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e1c003a00 0x7f3e1c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e38d46700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e1c005210 0x7f3e1c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e337fe700 1 -- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e1c005210 msgr2=0x7f3e1c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e337fe700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e1c005210 0x7f3e1c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e337fe700 1 -- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e1c003a00 msgr2=0x7f3e1c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e337fe700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e1c003a00 0x7f3e1c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e337fe700 1 -- 172.21.15.158:0/2936617685 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0035b0 con 0x7f3e1c004850 2023-10-16T09:43:51.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e38d46700 1 -- 172.21.15.158:0/2936617685 reap_dead start 2023-10-16T09:43:51.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e337fe700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e1c004cd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e340cca40 tx=0x7f3e1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542735df63e84e06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e240052d0 con 0x7f3e1c004850 2023-10-16T09:43:51.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e24005450 con 0x7f3e1c004850 2023-10-16T09:43:51.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.462+0000 7f3e32ffd700 1 -- 172.21.15.158:0/2936617685 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e240038c0 con 0x7f3e1c004850 2023-10-16T09:43:51.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.463+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2936617685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 msgr2=0x7f3e1c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.463+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e1c004cd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e340cca40 tx=0x7f3e1c005880 comp rx=0 tx=0).stop 2023-10-16T09:43:51.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.463+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2936617685 shutdown_connections 2023-10-16T09:43:51.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.463+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2936617685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e1c004cd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.463+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2936617685 >> 172.21.15.158:0/2936617685 conn(0x7f3e34099a10 msgr2=0x7f3e340d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:51.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.463+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2936617685 shutdown_connections 2023-10-16T09:43:51.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.463+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2936617685 wait complete. 2023-10-16T09:43:51.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.464+0000 7f3e3a7a9700 1 Processor -- start 2023-10-16T09:43:51.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.464+0000 7f3e3a7a9700 1 -- start start 2023-10-16T09:43:51.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e3a7a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e3414e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e3a7a9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e341e7350 0x7f3e341e77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e3a7a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e340c5150 0x7f3e340c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e3a7a9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e340c78c0 con 0x7f3e1c004850 2023-10-16T09:43:51.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e3a7a9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e341e7d10 con 0x7f3e341e7350 2023-10-16T09:43:51.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e3a7a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e340c69b0 con 0x7f3e340c5150 2023-10-16T09:43:51.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e3414e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e3414e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46862/0 (socket says 172.21.15.158:46862) 2023-10-16T09:43:51.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 -- 172.21.15.158:0/2084763657 learned_addr learned my addr 172.21.15.158:0/2084763657 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:51.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e337fe700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e341e7350 0x7f3e341e77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e38d46700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e340c5150 0x7f3e340c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 -- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e340c5150 msgr2=0x7f3e340c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e340c5150 0x7f3e340c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 -- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e341e7350 msgr2=0x7f3e341e77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e341e7350 0x7f3e341e77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 -- 172.21.15.158:0/2084763657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e340c5b10 con 0x7f3e1c004850 2023-10-16T09:43:51.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.465+0000 7f3e33fff700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e3414e720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e28011ad0 tx=0x7f3e28011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.466+0000 7f3e30ff9700 1 -- 172.21.15.158:0/2084763657 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800f940 con 0x7f3e1c004850 2023-10-16T09:43:51.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.466+0000 7f3e30ff9700 1 -- 172.21.15.158:0/2084763657 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2800faa0 con 0x7f3e1c004850 2023-10-16T09:43:51.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.466+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e340c5da0 con 0x7f3e1c004850 2023-10-16T09:43:51.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.466+0000 7f3e30ff9700 1 -- 172.21.15.158:0/2084763657 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800fe50 con 0x7f3e1c004850 2023-10-16T09:43:51.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.466+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e340c6360 con 0x7f3e1c004850 2023-10-16T09:43:51.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.467+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e340c6c90 con 0x7f3e1c004850 2023-10-16T09:43:51.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.467+0000 7f3e30ff9700 1 -- 172.21.15.158:0/2084763657 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3e28014440 con 0x7f3e1c004850 2023-10-16T09:43:51.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.467+0000 7f3e30ff9700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3e2006af00 0x7f3e2006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.467+0000 7f3e30ff9700 1 -- 172.21.15.158:0/2084763657 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3e28089f40 con 0x7f3e1c004850 2023-10-16T09:43:51.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.467+0000 7f3e337fe700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3e2006af00 0x7f3e2006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.468+0000 7f3e337fe700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3e2006af00 0x7f3e2006d3c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005040 tx=0x7f3e24009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.472+0000 7f3e30ff9700 1 -- 172.21.15.158:0/2084763657 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e28054d90 con 0x7f3e1c004850 2023-10-16T09:43:51.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.663+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e340cb0f0 con 0x7f3e1c004850 2023-10-16T09:43:51.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.663+0000 7f3e30ff9700 1 -- 172.21.15.158:0/2084763657 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e280583b0 con 0x7f3e1c004850 2023-10-16T09:43:51.665 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:43:51.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3e2006af00 msgr2=0x7f3e2006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3e2006af00 0x7f3e2006d3c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005040 tx=0x7f3e24009030 comp rx=0 tx=0).stop 2023-10-16T09:43:51.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 msgr2=0x7f3e3414e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e3414e720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e28011ad0 tx=0x7f3e28011de0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 shutdown_connections 2023-10-16T09:43:51.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3e2006af00 0x7f3e2006d3c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e340c5150 0x7f3e340c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e1c004850 0x7f3e3414e720 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 --2- 172.21.15.158:0/2084763657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e341e7350 0x7f3e341e77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 >> 172.21.15.158:0/2084763657 conn(0x7f3e34099a10 msgr2=0x7f3e3415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:51.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 shutdown_connections 2023-10-16T09:43:51.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.665+0000 7f3e3a7a9700 1 -- 172.21.15.158:0/2084763657 wait complete. 2023-10-16T09:43:51.679 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:43:51.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.801+0000 7fa0459e7700 1 Processor -- start 2023-10-16T09:43:51.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.801+0000 7fa0459e7700 1 -- start start 2023-10-16T09:43:51.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa0459e7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa040160d30 0x7fa0400b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa0459e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0400d0530 0x7fa0400d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa0459e7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa0400d19a0 0x7fa0400d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa0459e7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0400b8910 con 0x7fa040160d30 2023-10-16T09:43:51.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa0459e7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0400d0ef0 con 0x7fa0400d19a0 2023-10-16T09:43:51.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa0459e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400b7c90 con 0x7fa0400d0530 2023-10-16T09:43:51.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa040160d30 0x7fa0400b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa040160d30 0x7fa0400b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46870/0 (socket says 172.21.15.158:46870) 2023-10-16T09:43:51.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 -- 172.21.15.158:0/3675259363 learned_addr learned my addr 172.21.15.158:0/3675259363 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:51.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03f7fe700 1 --2- 172.21.15.158:0/3675259363 >> v2:172.21.15.172:3300/0 conn(0x7fa0400d19a0 0x7fa0400d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03e7fc700 1 --2- 172.21.15.158:0/3675259363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0400d0530 0x7fa0400d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 -- 172.21.15.158:0/3675259363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0400d0530 msgr2=0x7fa0400d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 --2- 172.21.15.158:0/3675259363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0400d0530 0x7fa0400d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 -- 172.21.15.158:0/3675259363 >> v2:172.21.15.172:3300/0 conn(0x7fa0400d19a0 msgr2=0x7fa0400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:51.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 --2- 172.21.15.158:0/3675259363 >> v2:172.21.15.172:3300/0 conn(0x7fa0400d19a0 0x7fa0400d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03effd700 1 -- 172.21.15.158:0/3675259363 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0400b7f70 con 0x7fa040160d30 2023-10-16T09:43:51.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.802+0000 7fa03e7fc700 1 --2- 172.21.15.158:0/3675259363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0400d0530 0x7fa0400d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:51.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa03effd700 1 --2- 172.21.15.158:0/3675259363 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa040160d30 0x7fa0400b8f50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa04008ca40 tx=0x7fa034009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9b7584d2146d09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa03dffb700 1 -- 172.21.15.158:0/3675259363 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03400b870 con 0x7fa040160d30 2023-10-16T09:43:51.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa03dffb700 1 -- 172.21.15.158:0/3675259363 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03400b9d0 con 0x7fa040160d30 2023-10-16T09:43:51.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa03dffb700 1 -- 172.21.15.158:0/3675259363 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03400bd60 con 0x7fa040160d30 2023-10-16T09:43:51.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 -- 172.21.15.158:0/3675259363 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa040160d30 msgr2=0x7fa0400b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 --2- 172.21.15.158:0/3675259363 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa040160d30 0x7fa0400b8f50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa04008ca40 tx=0x7fa034009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 -- 172.21.15.158:0/3675259363 shutdown_connections 2023-10-16T09:43:51.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 --2- 172.21.15.158:0/3675259363 >> v2:172.21.15.172:3300/0 conn(0x7fa0400d19a0 0x7fa0400d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 --2- 172.21.15.158:0/3675259363 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa040160d30 0x7fa0400b8f50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 --2- 172.21.15.158:0/3675259363 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0400d0530 0x7fa0400d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 -- 172.21.15.158:0/3675259363 >> 172.21.15.158:0/3675259363 conn(0x7fa0400669f0 msgr2=0x7fa040066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:51.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 -- 172.21.15.158:0/3675259363 shutdown_connections 2023-10-16T09:43:51.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.803+0000 7fa0459e7700 1 -- 172.21.15.158:0/3675259363 wait complete. 2023-10-16T09:43:51.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.804+0000 7fa0459e7700 1 Processor -- start 2023-10-16T09:43:51.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa0459e7700 1 -- start start 2023-10-16T09:43:51.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa0459e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0400d0530 0x7fa0400c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa0459e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0400d19a0 0x7fa040146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa0459e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa040160d30 0x7fa0400cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa0459e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa040146dd0 con 0x7fa0400d19a0 2023-10-16T09:43:51.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa0459e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400cbb50 con 0x7fa0400d0530 2023-10-16T09:43:51.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa0459e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400cbe30 con 0x7fa040160d30 2023-10-16T09:43:51.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa03e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0400d19a0 0x7fa040146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa03e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0400d19a0 0x7fa040146890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46876/0 (socket says 172.21.15.158:46876) 2023-10-16T09:43:51.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.805+0000 7fa03e7fc700 1 -- 172.21.15.158:0/1853518083 learned_addr learned my addr 172.21.15.158:0/1853518083 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:51.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03effd700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0400d0530 0x7fa0400c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03f7fe700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa040160d30 0x7fa0400cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03e7fc700 1 -- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa040160d30 msgr2=0x7fa0400cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03e7fc700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa040160d30 0x7fa0400cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03e7fc700 1 -- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0400d0530 msgr2=0x7fa0400c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03e7fc700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0400d0530 0x7fa0400c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03e7fc700 1 -- 172.21.15.158:0/1853518083 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa034009550 con 0x7fa0400d19a0 2023-10-16T09:43:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa03e7fc700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0400d19a0 0x7fa040146890 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c740 tx=0x7fa03000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa027fff700 1 -- 172.21.15.158:0/1853518083 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030011800 con 0x7fa0400d19a0 2023-10-16T09:43:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa027fff700 1 -- 172.21.15.158:0/1853518083 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa030011960 con 0x7fa0400d19a0 2023-10-16T09:43:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0400cc110 con 0x7fa0400d19a0 2023-10-16T09:43:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa027fff700 1 -- 172.21.15.158:0/1853518083 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030011cd0 con 0x7fa0400d19a0 2023-10-16T09:43:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.806+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0400cc6d0 con 0x7fa0400d19a0 2023-10-16T09:43:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.807+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0400681b0 con 0x7fa0400d19a0 2023-10-16T09:43:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.807+0000 7fa027fff700 1 -- 172.21.15.158:0/1853518083 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa030021350 con 0x7fa0400d19a0 2023-10-16T09:43:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.808+0000 7fa027fff700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa02c06af00 0x7fa02c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.808+0000 7fa027fff700 1 -- 172.21.15.158:0/1853518083 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa030089790 con 0x7fa0400d19a0 2023-10-16T09:43:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.808+0000 7fa03effd700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa02c06af00 0x7fa02c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.808+0000 7fa03effd700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa02c06af00 0x7fa02c06d3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa034014040 tx=0x7fa034008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:51.812+0000 7fa027fff700 1 -- 172.21.15.158:0/1853518083 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0300544c0 con 0x7fa0400d19a0 2023-10-16T09:43:52.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.002+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa040158220 con 0x7fa0400d19a0 2023-10-16T09:43:52.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.003+0000 7fa027fff700 1 -- 172.21.15.158:0/1853518083 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa030057ae0 con 0x7fa0400d19a0 2023-10-16T09:43:52.005 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:52.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa02c06af00 msgr2=0x7fa02c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa02c06af00 0x7fa02c06d3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa034014040 tx=0x7fa034008030 comp rx=0 tx=0).stop 2023-10-16T09:43:52.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0400d19a0 msgr2=0x7fa040146890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0400d19a0 0x7fa040146890 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c740 tx=0x7fa03000ca50 comp rx=0 tx=0).stop 2023-10-16T09:43:52.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 shutdown_connections 2023-10-16T09:43:52.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa02c06af00 0x7fa02c06d3c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa040160d30 0x7fa0400cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0400d0530 0x7fa0400c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 --2- 172.21.15.158:0/1853518083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0400d19a0 0x7fa040146890 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 >> 172.21.15.158:0/1853518083 conn(0x7fa0400669f0 msgr2=0x7fa0400d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:52.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 shutdown_connections 2023-10-16T09:43:52.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.005+0000 7fa0459e7700 1 -- 172.21.15.158:0/1853518083 wait complete. 2023-10-16T09:43:52.019 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:43:52.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.141+0000 7f86b2016700 1 Processor -- start 2023-10-16T09:43:52.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.141+0000 7f86b2016700 1 -- start start 2023-10-16T09:43:52.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86b2016700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86ac15dad0 0x7f86ac0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86b2016700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac15deb0 0x7f86ac0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86b2016700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f86ac0cea20 0x7f86ac0c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86b2016700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac0d0ae0 con 0x7f86ac15dad0 2023-10-16T09:43:52.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86b2016700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac0d12d0 con 0x7f86ac0cea20 2023-10-16T09:43:52.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86b2016700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0cbdf0 con 0x7f86ac15deb0 2023-10-16T09:43:52.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86ac15dad0 0x7f86ac0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86ac15dad0 0x7f86ac0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46884/0 (socket says 172.21.15.158:46884) 2023-10-16T09:43:52.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 -- 172.21.15.158:0/1923950675 learned_addr learned my addr 172.21.15.158:0/1923950675 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:52.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86abfff700 1 --2- 172.21.15.158:0/1923950675 >> v2:172.21.15.172:3300/0 conn(0x7f86ac0cea20 0x7f86ac0c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86aaffd700 1 --2- 172.21.15.158:0/1923950675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac15deb0 0x7f86ac0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 -- 172.21.15.158:0/1923950675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac15deb0 msgr2=0x7f86ac0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:52.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 --2- 172.21.15.158:0/1923950675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac15deb0 0x7f86ac0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 -- 172.21.15.158:0/1923950675 >> v2:172.21.15.172:3300/0 conn(0x7f86ac0cea20 msgr2=0x7f86ac0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:52.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 --2- 172.21.15.158:0/1923950675 >> v2:172.21.15.172:3300/0 conn(0x7f86ac0cea20 0x7f86ac0c3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.142+0000 7f86ab7fe700 1 -- 172.21.15.158:0/1923950675 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac0cc280 con 0x7f86ac15dad0 2023-10-16T09:43:52.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86ab7fe700 1 --2- 172.21.15.158:0/1923950675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac0cdfa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f86ac071950 tx=0x7f86a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9d83125d9b0efd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86aa7fc700 1 -- 172.21.15.158:0/1923950675 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a000b870 con 0x7f86ac15dad0 2023-10-16T09:43:52.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86aa7fc700 1 -- 172.21.15.158:0/1923950675 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a000b9d0 con 0x7f86ac15dad0 2023-10-16T09:43:52.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86aa7fc700 1 -- 172.21.15.158:0/1923950675 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a000bd60 con 0x7f86ac15dad0 2023-10-16T09:43:52.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 -- 172.21.15.158:0/1923950675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 msgr2=0x7f86ac0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 --2- 172.21.15.158:0/1923950675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac0cdfa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f86ac071950 tx=0x7f86a0009bd0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 -- 172.21.15.158:0/1923950675 shutdown_connections 2023-10-16T09:43:52.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 --2- 172.21.15.158:0/1923950675 >> v2:172.21.15.172:3300/0 conn(0x7f86ac0cea20 0x7f86ac0c3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 --2- 172.21.15.158:0/1923950675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac0cdfa0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 --2- 172.21.15.158:0/1923950675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac15deb0 0x7f86ac0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 -- 172.21.15.158:0/1923950675 >> 172.21.15.158:0/1923950675 conn(0x7f86ac099a10 msgr2=0x7f86ac0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:52.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 -- 172.21.15.158:0/1923950675 shutdown_connections 2023-10-16T09:43:52.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.143+0000 7f86b2016700 1 -- 172.21.15.158:0/1923950675 wait complete. 2023-10-16T09:43:52.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 Processor -- start 2023-10-16T09:43:52.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 -- start start 2023-10-16T09:43:52.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac0cea20 0x7f86ac153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ac15deb0 0x7f86ac1563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac1e3200 con 0x7f86ac15dad0 2023-10-16T09:43:52.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac1568f0 con 0x7f86ac15deb0 2023-10-16T09:43:52.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.145+0000 7f86b2016700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac1e3450 con 0x7f86ac0cea20 2023-10-16T09:43:52.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46892/0 (socket says 172.21.15.158:46892) 2023-10-16T09:43:52.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 -- 172.21.15.158:0/198109990 learned_addr learned my addr 172.21.15.158:0/198109990 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:52.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86abfff700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ac15deb0 0x7f86ac1563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86ab7fe700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac0cea20 0x7f86ac153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 -- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac0cea20 msgr2=0x7f86ac153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac0cea20 0x7f86ac153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 -- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ac15deb0 msgr2=0x7f86ac1563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ac15deb0 0x7f86ac1563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 -- 172.21.15.158:0/198109990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0009550 con 0x7f86ac15dad0 2023-10-16T09:43:52.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86aaffd700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac155e70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f869c00d830 tx=0x7f869c00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f8693fff700 1 -- 172.21.15.158:0/198109990 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c0125f0 con 0x7f86ac15dad0 2023-10-16T09:43:52.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f8693fff700 1 -- 172.21.15.158:0/198109990 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c012750 con 0x7f86ac15dad0 2023-10-16T09:43:52.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac1e3730 con 0x7f86ac15dad0 2023-10-16T09:43:52.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f8693fff700 1 -- 172.21.15.158:0/198109990 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c012b00 con 0x7f86ac15dad0 2023-10-16T09:43:52.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.146+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac1e3cf0 con 0x7f86ac15dad0 2023-10-16T09:43:52.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.147+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac1e3eb0 con 0x7f86ac15dad0 2023-10-16T09:43:52.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.148+0000 7f8693fff700 1 -- 172.21.15.158:0/198109990 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f869c012c60 con 0x7f86ac15dad0 2023-10-16T09:43:52.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.148+0000 7f8693fff700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f869806af00 0x7f869806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.148+0000 7f8693fff700 1 -- 172.21.15.158:0/198109990 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f869c089bc0 con 0x7f86ac15dad0 2023-10-16T09:43:52.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.148+0000 7f86ab7fe700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f869806af00 0x7f869806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.148+0000 7f86ab7fe700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f869806af00 0x7f869806d3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f86a0000c00 tx=0x7f86a0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.152+0000 7f8693fff700 1 -- 172.21.15.158:0/198109990 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f869c054a10 con 0x7f86ac15dad0 2023-10-16T09:43:52.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.346+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f86ac1e4060 con 0x7f86ac15dad0 2023-10-16T09:43:52.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.346+0000 7f8693fff700 1 -- 172.21.15.158:0/198109990 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f869c058030 con 0x7f86ac15dad0 2023-10-16T09:43:52.348 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:43:52.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f869806af00 msgr2=0x7f869806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f869806af00 0x7f869806d3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f86a0000c00 tx=0x7f86a0008030 comp rx=0 tx=0).stop 2023-10-16T09:43:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 msgr2=0x7f86ac155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac155e70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f869c00d830 tx=0x7f869c00dbf0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 shutdown_connections 2023-10-16T09:43:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ac15deb0 0x7f86ac1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f869806af00 0x7f869806d3c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ac0cea20 0x7f86ac153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 --2- 172.21.15.158:0/198109990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ac15dad0 0x7f86ac155e70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 >> 172.21.15.158:0/198109990 conn(0x7f86ac099a10 msgr2=0x7f86ac0c5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 shutdown_connections 2023-10-16T09:43:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.348+0000 7f86b2016700 1 -- 172.21.15.158:0/198109990 wait complete. 2023-10-16T09:43:52.362 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:43:52.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.485+0000 7f462aece700 1 Processor -- start 2023-10-16T09:43:52.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.485+0000 7f462aece700 1 -- start start 2023-10-16T09:43:52.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462aece700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f462415b590 0x7f46240ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462aece700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46240cbaf0 0x7f46240cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462aece700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46240cf4c0 con 0x7f462415f1d0 2023-10-16T09:43:52.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462aece700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46240cc4c0 con 0x7f462415b590 2023-10-16T09:43:52.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462aece700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46240b49e0 con 0x7f46240cbaf0 2023-10-16T09:43:52.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4628c6a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f462415b590 0x7f46240ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462946b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46240cbaf0 0x7f46240cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4623fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f462415f1d0 0x7f46240b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:50978/0 (socket says 172.21.15.158:50978) 2023-10-16T09:43:52.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4628c6a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f462415b590 0x7f46240ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:48078/0 (socket says 172.21.15.158:48078) 2023-10-16T09:43:52.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462946b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46240cbaf0 0x7f46240cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:57734/0 (socket says 172.21.15.158:57734) 2023-10-16T09:43:52.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4628c6a700 1 -- 172.21.15.158:0/347271 learned_addr learned my addr 172.21.15.158:0/347271 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:52.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4628c6a700 1 -- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46240cbaf0 msgr2=0x7f46240cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:52.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4628c6a700 1 --2- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46240cbaf0 0x7f46240cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4628c6a700 1 -- 172.21.15.158:0/347271 >> v1:172.21.15.172:6789/0 conn(0x7f462415f1d0 legacy=0x7f46240b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f4628c6a700 1 -- 172.21.15.158:0/347271 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462415f970 con 0x7f462415b590 2023-10-16T09:43:52.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f46237fe700 1 -- 172.21.15.158:0/347271 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042410967 0 0) 0x7f46240cf4c0 con 0x7f462415f1d0 2023-10-16T09:43:52.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.486+0000 7f462946b700 1 --2- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46240cbaf0 0x7f46240cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:52.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f4628c6a700 1 --2- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f462415b590 0x7f46240ce510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4618009920 tx=0x7f4618009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f2da95d80df87d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f46237fe700 1 -- 172.21.15.158:0/347271 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618011830 con 0x7f462415b590 2023-10-16T09:43:52.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f46237fe700 1 -- 172.21.15.158:0/347271 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4618011990 con 0x7f462415b590 2023-10-16T09:43:52.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f46237fe700 1 -- 172.21.15.158:0/347271 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618011d20 con 0x7f462415b590 2023-10-16T09:43:52.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 -- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f462415b590 msgr2=0x7f46240ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 --2- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f462415b590 0x7f46240ce510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4618009920 tx=0x7f4618009c30 comp rx=0 tx=0).stop 2023-10-16T09:43:52.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 -- 172.21.15.158:0/347271 shutdown_connections 2023-10-16T09:43:52.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 --2- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46240cbaf0 0x7f46240cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 --2- 172.21.15.158:0/347271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f462415b590 0x7f46240ce510 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 -- 172.21.15.158:0/347271 >> 172.21.15.158:0/347271 conn(0x7f462406a8d0 msgr2=0x7f462406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:52.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 -- 172.21.15.158:0/347271 shutdown_connections 2023-10-16T09:43:52.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.487+0000 7f462aece700 1 -- 172.21.15.158:0/347271 wait complete. 2023-10-16T09:43:52.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.488+0000 7f462aece700 1 Processor -- start 2023-10-16T09:43:52.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.488+0000 7f462aece700 1 -- start start 2023-10-16T09:43:52.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462aece700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46240cbaf0 0x7f462415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462aece700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f462415b590 0x7f462415dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462aece700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f462415f1d0 0x7f462415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462aece700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240c92d0 con 0x7f462415f1d0 2023-10-16T09:43:52.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462aece700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462415e240 con 0x7f46240cbaf0 2023-10-16T09:43:52.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462aece700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462415e430 con 0x7f462415b590 2023-10-16T09:43:52.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462946b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f462415f1d0 0x7f462415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462946b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f462415f1d0 0x7f462415b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46906/0 (socket says 172.21.15.158:46906) 2023-10-16T09:43:52.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.489+0000 7f462946b700 1 -- 172.21.15.158:0/509747087 learned_addr learned my addr 172.21.15.158:0/509747087 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:52.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f4628c6a700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46240cbaf0 0x7f462415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f4623fff700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f462415b590 0x7f462415dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462946b700 1 -- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f462415b590 msgr2=0x7f462415dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462946b700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f462415b590 0x7f462415dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462946b700 1 -- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46240cbaf0 msgr2=0x7f462415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462946b700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46240cbaf0 0x7f462415d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462946b700 1 -- 172.21.15.158:0/509747087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c009a70 con 0x7f462415f1d0 2023-10-16T09:43:52.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f4623fff700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f462415b590 0x7f462415dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:52.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f4628c6a700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46240cbaf0 0x7f462415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:52.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462946b700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f462415f1d0 0x7f462415b970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f461c00efc0 tx=0x7f461c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f46217fa700 1 -- 172.21.15.158:0/509747087 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c009cd0 con 0x7f462415f1d0 2023-10-16T09:43:52.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f46217fa700 1 -- 172.21.15.158:0/509747087 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c00f040 con 0x7f462415f1d0 2023-10-16T09:43:52.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f46217fa700 1 -- 172.21.15.158:0/509747087 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00cd60 con 0x7f462415f1d0 2023-10-16T09:43:52.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4618009550 con 0x7f462415f1d0 2023-10-16T09:43:52.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.490+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46240c8ef0 con 0x7f462415f1d0 2023-10-16T09:43:52.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.491+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462415e950 con 0x7f462415f1d0 2023-10-16T09:43:52.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.491+0000 7f46217fa700 1 -- 172.21.15.158:0/509747087 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f461c014480 con 0x7f462415f1d0 2023-10-16T09:43:52.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.492+0000 7f46217fa700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460806af00 0x7f460806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.492+0000 7f46217fa700 1 -- 172.21.15.158:0/509747087 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f461c08a010 con 0x7f462415f1d0 2023-10-16T09:43:52.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.492+0000 7f4628c6a700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460806af00 0x7f460806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.492+0000 7f4628c6a700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460806af00 0x7f460806d3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f461800a010 tx=0x7f4618008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.496+0000 7f46217fa700 1 -- 172.21.15.158:0/509747087 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f462415e950 con 0x7f462415f1d0 2023-10-16T09:43:52.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.689+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f46240c8d50 con 0x7f462415f1d0 2023-10-16T09:43:52.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.690+0000 7f46217fa700 1 -- 172.21.15.158:0/509747087 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f46240c8d50 con 0x7f462415f1d0 2023-10-16T09:43:52.692 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:52.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460806af00 msgr2=0x7f460806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460806af00 0x7f460806d3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f461800a010 tx=0x7f4618008030 comp rx=0 tx=0).stop 2023-10-16T09:43:52.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f462415f1d0 msgr2=0x7f462415b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f462415f1d0 0x7f462415b970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f461c00efc0 tx=0x7f461c00c3a0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 shutdown_connections 2023-10-16T09:43:52.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f462415f1d0 0x7f462415b970 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46240cbaf0 0x7f462415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f460806af00 0x7f460806d3c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 --2- 172.21.15.158:0/509747087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f462415b590 0x7f462415dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 >> 172.21.15.158:0/509747087 conn(0x7f462406a8d0 msgr2=0x7f46240b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:52.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 shutdown_connections 2023-10-16T09:43:52.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.692+0000 7f462aece700 1 -- 172.21.15.158:0/509747087 wait complete. 2023-10-16T09:43:52.705 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:52.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.823+0000 7efe1fe3b700 1 Processor -- start 2023-10-16T09:43:52.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.824+0000 7efe1fe3b700 1 -- start start 2023-10-16T09:43:52.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.824+0000 7efe1fe3b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 0x7efe180723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.824+0000 7efe1fe3b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efe180af590 0x7efe1806d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.824+0000 7efe1fe3b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe18072910 con 0x7efe1806dec0 2023-10-16T09:43:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.824+0000 7efe1fe3b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe180ca7e0 con 0x7efe180af590 2023-10-16T09:43:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.824+0000 7efe1fe3b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1806e340 con 0x7efe1815a6c0 2023-10-16T09:43:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.824+0000 7efe1e3d8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efe1806dec0 0x7efe1815e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47088/0 (socket says 172.21.15.158:47088) 2023-10-16T09:43:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1e3d8700 1 -- 172.21.15.158:0/3126342124 learned_addr learned my addr 172.21.15.158:0/3126342124 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 0x7efe180723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1d3d6700 1 --2- 172.21.15.158:0/3126342124 >> v2:172.21.15.172:3300/0 conn(0x7efe180af590 0x7efe1806d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685979243 0 0) 0x7efe18072910 con 0x7efe1806dec0 2023-10-16T09:43:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe040035b0 con 0x7efe1806dec0 2023-10-16T09:43:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1979273061 0 0) 0x7efe040035b0 con 0x7efe1806dec0 2023-10-16T09:43:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 msgr2=0x7efe180723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 0x7efe180723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 >> v2:172.21.15.172:3300/0 conn(0x7efe180af590 msgr2=0x7efe1806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 --2- 172.21.15.158:0/3126342124 >> v2:172.21.15.172:3300/0 conn(0x7efe180af590 0x7efe1806d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe180b1e40 con 0x7efe1806dec0 2023-10-16T09:43:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7efe100019f0 con 0x7efe1806dec0 2023-10-16T09:43:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 >> v1:172.21.15.158:6789/0 conn(0x7efe1806dec0 legacy=0x7efe1815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe04003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe04004850 0x7efe04004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 0x7efe04005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe180ca7e0 con 0x7efe04003a00 2023-10-16T09:43:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1806e340 con 0x7efe04004850 2023-10-16T09:43:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0400ba10 con 0x7efe1815a6c0 2023-10-16T09:43:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe04003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1d3d6700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe04004850 0x7efe04004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.825+0000 7efe1e3d8700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 0x7efe04005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1dbd7700 1 -- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 msgr2=0x7efe04005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:52.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe1815a6c0 0x7efe04005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1dbd7700 1 -- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe04004850 msgr2=0x7efe04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:52.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe04004850 0x7efe04004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1dbd7700 1 -- 172.21.15.158:0/3126342124 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe040035b0 con 0x7efe04003a00 2023-10-16T09:43:52.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe04003e20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efe08009b40 tx=0x7efe04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e345d19d6c75a9a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08005a20 con 0x7efe04003a00 2023-10-16T09:43:52.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe08005b80 con 0x7efe04003a00 2023-10-16T09:43:52.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1cbd5700 1 -- 172.21.15.158:0/3126342124 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0800e6c0 con 0x7efe04003a00 2023-10-16T09:43:52.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1fe3b700 1 -- 172.21.15.158:0/3126342124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 msgr2=0x7efe04003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1fe3b700 1 --2- 172.21.15.158:0/3126342124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe04003e20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efe08009b40 tx=0x7efe04005880 comp rx=0 tx=0).stop 2023-10-16T09:43:52.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1e3d8700 1 -- 172.21.15.158:0/3126342124 reap_dead start 2023-10-16T09:43:52.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1fe3b700 1 -- 172.21.15.158:0/3126342124 shutdown_connections 2023-10-16T09:43:52.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.826+0000 7efe1fe3b700 1 -- 172.21.15.158:0/3126342124 >> 172.21.15.158:0/3126342124 conn(0x7efe180ccda0 msgr2=0x7efe180cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:52.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.827+0000 7efe1fe3b700 1 -- 172.21.15.158:0/3126342124 shutdown_connections 2023-10-16T09:43:52.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.827+0000 7efe1fe3b700 1 -- 172.21.15.158:0/3126342124 wait complete. 2023-10-16T09:43:52.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 Processor -- start 2023-10-16T09:43:52.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 -- start start 2023-10-16T09:43:52.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe180c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe180c4210 0x7efe1814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe180c4690 0x7efe1814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1814bbc0 con 0x7efe04003a00 2023-10-16T09:43:52.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe180cd300 con 0x7efe180c4210 2023-10-16T09:43:52.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.828+0000 7efe1fe3b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe181541b0 con 0x7efe180c4690 2023-10-16T09:43:52.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe180c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe180c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46920/0 (socket says 172.21.15.158:46920) 2023-10-16T09:43:52.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 -- 172.21.15.158:0/1304998473 learned_addr learned my addr 172.21.15.158:0/1304998473 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:52.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1e3d8700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe180c4690 0x7efe1814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1d3d6700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe180c4210 0x7efe1814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 -- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe180c4690 msgr2=0x7efe1814b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe180c4690 0x7efe1814b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 -- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe180c4210 msgr2=0x7efe1814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:52.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe180c4210 0x7efe1814b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:52.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 -- 172.21.15.158:0/1304998473 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18154490 con 0x7efe04003a00 2023-10-16T09:43:52.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1dbd7700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe180c6ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe040046a0 tx=0x7efe080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe0e7fc700 1 -- 172.21.15.158:0/1304998473 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08020070 con 0x7efe04003a00 2023-10-16T09:43:52.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe0e7fc700 1 -- 172.21.15.158:0/1304998473 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe08004030 con 0x7efe04003a00 2023-10-16T09:43:52.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe180cd5c0 con 0x7efe04003a00 2023-10-16T09:43:52.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe0e7fc700 1 -- 172.21.15.158:0/1304998473 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08004440 con 0x7efe04003a00 2023-10-16T09:43:52.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.829+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe180cdae0 con 0x7efe04003a00 2023-10-16T09:43:52.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.830+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe180cd930 con 0x7efe04003a00 2023-10-16T09:43:52.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.830+0000 7efe0e7fc700 1 -- 172.21.15.158:0/1304998473 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efe080036a0 con 0x7efe04003a00 2023-10-16T09:43:52.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.831+0000 7efe0e7fc700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdfc06af00 0x7efdfc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:52.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.831+0000 7efe0e7fc700 1 -- 172.21.15.158:0/1304998473 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efe0808ad20 con 0x7efe04003a00 2023-10-16T09:43:52.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.831+0000 7efe1d3d6700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdfc06af00 0x7efdfc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:52.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.831+0000 7efe1d3d6700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdfc06af00 0x7efdfc06d3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efe180ced70 tx=0x7efe14008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:52.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:52.835+0000 7efe0e7fc700 1 -- 172.21.15.158:0/1304998473 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe08055b70 con 0x7efe04003a00 2023-10-16T09:43:53.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.075+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe1814bbc0 con 0x7efe04003a00 2023-10-16T09:43:53.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.075+0000 7efe0e7fc700 1 -- 172.21.15.158:0/1304998473 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7efe08059190 con 0x7efe04003a00 2023-10-16T09:43:53.077 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:53.077 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:43:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdfc06af00 msgr2=0x7efdfc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdfc06af00 0x7efdfc06d3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efe180ced70 tx=0x7efe14008040 comp rx=0 tx=0).stop 2023-10-16T09:43:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 msgr2=0x7efe180c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe180c6ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe040046a0 tx=0x7efe080047c0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 shutdown_connections 2023-10-16T09:43:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdfc06af00 0x7efdfc06d3c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe180c4690 0x7efe1814b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe04003a00 0x7efe180c6ae0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 --2- 172.21.15.158:0/1304998473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe180c4210 0x7efe1814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 >> 172.21.15.158:0/1304998473 conn(0x7efe180ccda0 msgr2=0x7efe180ca920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 shutdown_connections 2023-10-16T09:43:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.077+0000 7efe1fe3b700 1 -- 172.21.15.158:0/1304998473 wait complete. 2023-10-16T09:43:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:43:53.091 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:53.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.209+0000 7f09e2165700 1 Processor -- start 2023-10-16T09:43:53.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.209+0000 7f09e2165700 1 -- start start 2023-10-16T09:43:53.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09e2165700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 0x7f09dc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09e2165700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc161270 con 0x7f09dc0729b0 2023-10-16T09:43:53.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09e2165700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc161510 con 0x7f09dc06ab40 2023-10-16T09:43:53.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09e2165700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc1617f0 con 0x7f09dc06e5d0 2023-10-16T09:43:53.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09dbfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f09dc0729b0 0x7f09dc15df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47108/0 (socket says 172.21.15.158:47108) 2023-10-16T09:43:53.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09dbfff700 1 -- 172.21.15.158:0/1470977445 learned_addr learned my addr 172.21.15.158:0/1470977445 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:53.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09daffd700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 0x7f09dc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304137364 0 0) 0x7f09dc161270 con 0x7f09dc0729b0 2023-10-16T09:43:53.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c40035b0 con 0x7f09dc0729b0 2023-10-16T09:43:53.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472995767 0 0) 0x7f09dc161510 con 0x7f09dc06ab40 2023-10-16T09:43:53.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09dc161270 con 0x7f09dc06ab40 2023-10-16T09:43:53.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406311073 0 0) 0x7f09c40035b0 con 0x7f09dc0729b0 2023-10-16T09:43:53.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 msgr2=0x7f09dc06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:53.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 0x7f09dc06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 >> v1:172.21.15.172:6789/0 conn(0x7f09dc06ab40 legacy=0x7f09dc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc161ad0 con 0x7f09dc0729b0 2023-10-16T09:43:53.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f09d40019f0 con 0x7f09dc0729b0 2023-10-16T09:43:53.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 >> v1:172.21.15.158:6789/0 conn(0x7f09dc0729b0 legacy=0x7f09dc15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 0x7f09c4003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c4004170 0x7f09c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09c4004d90 0x7f09c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc1617f0 con 0x7f09c4004d90 2023-10-16T09:43:53.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c400bb50 con 0x7f09c4004170 2023-10-16T09:43:53.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.210+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c400bd10 con 0x7f09dc06e5d0 2023-10-16T09:43:53.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09dbfff700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09c4004d90 0x7f09c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09db7fe700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 0x7f09c4003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09daffd700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c4004170 0x7f09c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09dbfff700 1 -- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 msgr2=0x7f09c4003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:53.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09dbfff700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc06e5d0 0x7f09c4003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09dbfff700 1 -- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c4004170 msgr2=0x7f09c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:53.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09dbfff700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c4004170 0x7f09c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09dbfff700 1 -- 172.21.15.158:0/1470977445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c40035b0 con 0x7f09c4004d90 2023-10-16T09:43:53.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09dbfff700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09c4004d90 0x7f09c4005210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f09d400a6e0 tx=0x7f09c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ab979d5431de6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d40072d0 con 0x7f09c4004d90 2023-10-16T09:43:53.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d4007430 con 0x7f09c4004d90 2023-10-16T09:43:53.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.211+0000 7f09da7fc700 1 -- 172.21.15.158:0/1470977445 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d40077a0 con 0x7f09c4004d90 2023-10-16T09:43:53.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.212+0000 7f09e2165700 1 -- 172.21.15.158:0/1470977445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09c4004d90 msgr2=0x7f09c4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.212+0000 7f09e2165700 1 --2- 172.21.15.158:0/1470977445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09c4004d90 0x7f09c4005210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f09d400a6e0 tx=0x7f09c4005880 comp rx=0 tx=0).stop 2023-10-16T09:43:53.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.212+0000 7f09dbfff700 1 -- 172.21.15.158:0/1470977445 reap_dead start 2023-10-16T09:43:53.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.212+0000 7f09e2165700 1 -- 172.21.15.158:0/1470977445 shutdown_connections 2023-10-16T09:43:53.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.212+0000 7f09e2165700 1 -- 172.21.15.158:0/1470977445 >> 172.21.15.158:0/1470977445 conn(0x7f09dc0d02b0 msgr2=0x7f09dc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:53.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.212+0000 7f09e2165700 1 -- 172.21.15.158:0/1470977445 shutdown_connections 2023-10-16T09:43:53.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.212+0000 7f09e2165700 1 -- 172.21.15.158:0/1470977445 wait complete. 2023-10-16T09:43:53.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.213+0000 7f09e2165700 1 Processor -- start 2023-10-16T09:43:53.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09e2165700 1 -- start start 2023-10-16T09:43:53.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09e2165700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09dc1f00b0 0x7f09dc1f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09e2165700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc1f2dd0 0x7f09dc160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09e2165700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09dc0b1f40 0x7f09dc0b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09e2165700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc1f3250 con 0x7f09dc0b1f40 2023-10-16T09:43:53.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09e2165700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0c4d70 con 0x7f09dc1f00b0 2023-10-16T09:43:53.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09e2165700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc160fe0 con 0x7f09dc1f2dd0 2023-10-16T09:43:53.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09dbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09dc0b1f40 0x7f09dc0b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09dbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09dc0b1f40 0x7f09dc0b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46944/0 (socket says 172.21.15.158:46944) 2023-10-16T09:43:53.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.214+0000 7f09dbfff700 1 -- 172.21.15.158:0/3253966970 learned_addr learned my addr 172.21.15.158:0/3253966970 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:53.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09db7fe700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09dc1f00b0 0x7f09dc1f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09daffd700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc1f2dd0 0x7f09dc160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09dbfff700 1 -- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc1f2dd0 msgr2=0x7f09dc160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09dbfff700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc1f2dd0 0x7f09dc160aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09dbfff700 1 -- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09dc1f00b0 msgr2=0x7f09dc1f0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09dbfff700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09dc1f00b0 0x7f09dc1f0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09dbfff700 1 -- 172.21.15.158:0/3253966970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc1f3480 con 0x7f09dc0b1f40 2023-10-16T09:43:53.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09dbfff700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09dc0b1f40 0x7f09dc0b23c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f09d4002230 tx=0x7f09d4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09c3fff700 1 -- 172.21.15.158:0/3253966970 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d40079b0 con 0x7f09dc0b1f40 2023-10-16T09:43:53.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09c3fff700 1 -- 172.21.15.158:0/3253966970 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d4007b10 con 0x7f09dc0b1f40 2023-10-16T09:43:53.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc1f3760 con 0x7f09dc0b1f40 2023-10-16T09:43:53.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09c3fff700 1 -- 172.21.15.158:0/3253966970 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4012640 con 0x7f09dc0b1f40 2023-10-16T09:43:53.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.215+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc0b2900 con 0x7f09dc0b1f40 2023-10-16T09:43:53.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.216+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc0681b0 con 0x7f09dc0b1f40 2023-10-16T09:43:53.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.216+0000 7f09c3fff700 1 -- 172.21.15.158:0/3253966970 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f09d401a070 con 0x7f09dc0b1f40 2023-10-16T09:43:53.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.217+0000 7f09c3fff700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c806af00 0x7f09c806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.217+0000 7f09c3fff700 1 -- 172.21.15.158:0/3253966970 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f09d408a7f0 con 0x7f09dc0b1f40 2023-10-16T09:43:53.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.217+0000 7f09db7fe700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c806af00 0x7f09c806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.217+0000 7f09db7fe700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c806af00 0x7f09c806d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f09cc007fd0 tx=0x7f09cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.221+0000 7f09c3fff700 1 -- 172.21.15.158:0/3253966970 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09d4055640 con 0x7f09dc0b1f40 2023-10-16T09:43:53.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.416+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09dc0b6230 con 0x7f09dc0b1f40 2023-10-16T09:43:53.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.417+0000 7f09c3fff700 1 -- 172.21.15.158:0/3253966970 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f09d4018090 con 0x7f09dc0b1f40 2023-10-16T09:43:53.419 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:53.419 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:53.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c806af00 msgr2=0x7f09c806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c806af00 0x7f09c806d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f09cc007fd0 tx=0x7f09cc009040 comp rx=0 tx=0).stop 2023-10-16T09:43:53.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09dc0b1f40 msgr2=0x7f09dc0b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09dc0b1f40 0x7f09dc0b23c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f09d4002230 tx=0x7f09d4001cb0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 shutdown_connections 2023-10-16T09:43:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09dc0b1f40 0x7f09dc0b23c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09dc1f00b0 0x7f09dc1f0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c806af00 0x7f09c806d3c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 --2- 172.21.15.158:0/3253966970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09dc1f2dd0 0x7f09dc160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 >> 172.21.15.158:0/3253966970 conn(0x7f09dc0d02b0 msgr2=0x7f09dc0ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 shutdown_connections 2023-10-16T09:43:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.419+0000 7f09e2165700 1 -- 172.21.15.158:0/3253966970 wait complete. 2023-10-16T09:43:53.433 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:43:53.434 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:43:53.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.556+0000 7f625f89d700 1 Processor -- start 2023-10-16T09:43:53.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.557+0000 7f625f89d700 1 -- start start 2023-10-16T09:43:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.557+0000 7f625f89d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6258160d40 0x7f62580ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.557+0000 7f625f89d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f62580c50c0 0x7f625806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.557+0000 7f625f89d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580ce110 con 0x7f62580cf420 2023-10-16T09:43:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.557+0000 7f625f89d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62580c61e0 con 0x7f6258160d40 2023-10-16T09:43:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.557+0000 7f625f89d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62580ce3f0 con 0x7f62580c50c0 2023-10-16T09:43:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.557+0000 7f625d639700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6258160d40 0x7f62580ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625ce38700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f62580cf420 0x7f62580c49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47126/0 (socket says 172.21.15.158:47126) 2023-10-16T09:43:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625ce38700 1 -- 172.21.15.158:0/3277401514 learned_addr learned my addr 172.21.15.158:0/3277401514 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625de3a700 1 --2- 172.21.15.158:0/3277401514 >> v2:172.21.15.172:3300/0 conn(0x7f62580c50c0 0x7f625806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625d639700 1 -- 172.21.15.158:0/3277401514 >> v2:172.21.15.172:3300/0 conn(0x7f62580c50c0 msgr2=0x7f625806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625d639700 1 --2- 172.21.15.158:0/3277401514 >> v2:172.21.15.172:3300/0 conn(0x7f62580c50c0 0x7f625806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625d639700 1 -- 172.21.15.158:0/3277401514 >> v1:172.21.15.158:6789/0 conn(0x7f62580cf420 legacy=0x7f62580c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625d639700 1 -- 172.21.15.158:0/3277401514 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580ce6d0 con 0x7f6258160d40 2023-10-16T09:43:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f624ffff700 1 -- 172.21.15.158:0/3277401514 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92637558 0 0) 0x7f62580ce110 con 0x7f62580cf420 2023-10-16T09:43:53.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f625d639700 1 --2- 172.21.15.158:0/3277401514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580ceee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62580ca110 tx=0x7f6248009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b343e1cb8e2d2c65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f624ffff700 1 -- 172.21.15.158:0/3277401514 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248011830 con 0x7f6258160d40 2023-10-16T09:43:53.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f624ffff700 1 -- 172.21.15.158:0/3277401514 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6248011990 con 0x7f6258160d40 2023-10-16T09:43:53.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.558+0000 7f624ffff700 1 -- 172.21.15.158:0/3277401514 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248011d20 con 0x7f6258160d40 2023-10-16T09:43:53.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 -- 172.21.15.158:0/3277401514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 msgr2=0x7f62580ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 --2- 172.21.15.158:0/3277401514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580ceee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62580ca110 tx=0x7f6248009b20 comp rx=0 tx=0).stop 2023-10-16T09:43:53.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 -- 172.21.15.158:0/3277401514 shutdown_connections 2023-10-16T09:43:53.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 --2- 172.21.15.158:0/3277401514 >> v2:172.21.15.172:3300/0 conn(0x7f62580c50c0 0x7f625806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 --2- 172.21.15.158:0/3277401514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580ceee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 -- 172.21.15.158:0/3277401514 >> 172.21.15.158:0/3277401514 conn(0x7f62580cc5b0 msgr2=0x7f62580cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:53.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 -- 172.21.15.158:0/3277401514 shutdown_connections 2023-10-16T09:43:53.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.559+0000 7f625f89d700 1 -- 172.21.15.158:0/3277401514 wait complete. 2023-10-16T09:43:53.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.560+0000 7f625f89d700 1 Processor -- start 2023-10-16T09:43:53.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.560+0000 7f625f89d700 1 -- start start 2023-10-16T09:43:53.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625f89d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62580c50c0 0x7f625815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625f89d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62580cf420 0x7f62580c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625f89d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625f89d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62580ae9b0 con 0x7f6258160d40 2023-10-16T09:43:53.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625f89d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62580aec90 con 0x7f62580cf420 2023-10-16T09:43:53.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625f89d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62580aef70 con 0x7f62580c50c0 2023-10-16T09:43:53.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46958/0 (socket says 172.21.15.158:46958) 2023-10-16T09:43:53.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 -- 172.21.15.158:0/3611302433 learned_addr learned my addr 172.21.15.158:0/3611302433 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:53.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625ce38700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62580cf420 0x7f62580c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625d639700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62580c50c0 0x7f625815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 -- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62580c50c0 msgr2=0x7f625815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62580c50c0 0x7f625815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 -- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62580cf420 msgr2=0x7f62580c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62580cf420 0x7f62580c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 -- 172.21.15.158:0/3611302433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62480094a0 con 0x7f6258160d40 2023-10-16T09:43:53.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625ce38700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62580cf420 0x7f62580c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:53.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625de3a700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580adf60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6250007e40 tx=0x7f625000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f624dffb700 1 -- 172.21.15.158:0/3611302433 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625000eea0 con 0x7f6258160d40 2023-10-16T09:43:53.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f624dffb700 1 -- 172.21.15.158:0/3611302433 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6250014420 con 0x7f6258160d40 2023-10-16T09:43:53.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f624dffb700 1 -- 172.21.15.158:0/3611302433 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62500147b0 con 0x7f6258160d40 2023-10-16T09:43:53.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.561+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62581612a0 con 0x7f6258160d40 2023-10-16T09:43:53.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.562+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625815be60 con 0x7f6258160d40 2023-10-16T09:43:53.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.562+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6258066db0 con 0x7f6258160d40 2023-10-16T09:43:53.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.563+0000 7f624dffb700 1 -- 172.21.15.158:0/3611302433 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6250014910 con 0x7f6258160d40 2023-10-16T09:43:53.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.563+0000 7f624dffb700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f623c06af00 0x7f623c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.563+0000 7f624dffb700 1 -- 172.21.15.158:0/3611302433 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f625008a0b0 con 0x7f6258160d40 2023-10-16T09:43:53.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.563+0000 7f625d639700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f623c06af00 0x7f623c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.564+0000 7f625d639700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f623c06af00 0x7f623c06d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f62480090b0 tx=0x7f624801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.568+0000 7f624dffb700 1 -- 172.21.15.158:0/3611302433 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6250054e70 con 0x7f6258160d40 2023-10-16T09:43:53.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.759+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62580681b0 con 0x7f6258160d40 2023-10-16T09:43:53.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.759+0000 7f624dffb700 1 -- 172.21.15.158:0/3611302433 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6250058490 con 0x7f6258160d40 2023-10-16T09:43:53.761 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:43:53.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f623c06af00 msgr2=0x7f623c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f623c06af00 0x7f623c06d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f62480090b0 tx=0x7f624801c040 comp rx=0 tx=0).stop 2023-10-16T09:43:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 msgr2=0x7f62580adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580adf60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6250007e40 tx=0x7f625000e3c0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 shutdown_connections 2023-10-16T09:43:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6258160d40 0x7f62580adf60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f623c06af00 0x7f623c06d3c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62580c50c0 0x7f625815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 --2- 172.21.15.158:0/3611302433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62580cf420 0x7f62580c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.761+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 >> 172.21.15.158:0/3611302433 conn(0x7f62580cc5b0 msgr2=0x7f625815eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.762+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 shutdown_connections 2023-10-16T09:43:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.762+0000 7f625f89d700 1 -- 172.21.15.158:0/3611302433 wait complete. 2023-10-16T09:43:53.775 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:43:53.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.896+0000 7fe595949700 1 Processor -- start 2023-10-16T09:43:53.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.896+0000 7fe595949700 1 -- start start 2023-10-16T09:43:53.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe595949700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe590160d30 0x7fe5900b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe595949700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5900d0530 0x7fe5900d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe595949700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5900b7c90 con 0x7fe5900d19a0 2023-10-16T09:43:53.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe595949700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5900b8910 con 0x7fe5900d0530 2023-10-16T09:43:53.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe595949700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5900d0ef0 con 0x7fe590160d30 2023-10-16T09:43:53.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5900d0530 0x7fe5900d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5900d0530 0x7fe5900d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46962/0 (socket says 172.21.15.158:46962) 2023-10-16T09:43:53.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 -- 172.21.15.158:0/3118353312 learned_addr learned my addr 172.21.15.158:0/3118353312 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:53.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58effd700 1 --2- 172.21.15.158:0/3118353312 >> v2:172.21.15.172:3300/0 conn(0x7fe590160d30 0x7fe5900b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 -- 172.21.15.158:0/3118353312 >> v2:172.21.15.172:3300/0 conn(0x7fe590160d30 msgr2=0x7fe5900b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:53.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 --2- 172.21.15.158:0/3118353312 >> v2:172.21.15.172:3300/0 conn(0x7fe590160d30 0x7fe5900b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 -- 172.21.15.158:0/3118353312 >> v1:172.21.15.172:6789/0 conn(0x7fe5900d19a0 legacy=0x7fe5900d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:53.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 -- 172.21.15.158:0/3118353312 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5900b7f70 con 0x7fe5900d0530 2023-10-16T09:43:53.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58e7fc700 1 --2- 172.21.15.158:0/3118353312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5900d0530 0x7fe5900d09b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe5840098f0 tx=0x7fe584009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc64ee555b2617c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58dffb700 1 -- 172.21.15.158:0/3118353312 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584004230 con 0x7fe5900d0530 2023-10-16T09:43:53.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58dffb700 1 -- 172.21.15.158:0/3118353312 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584004390 con 0x7fe5900d0530 2023-10-16T09:43:53.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.897+0000 7fe58dffb700 1 -- 172.21.15.158:0/3118353312 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5840036a0 con 0x7fe5900d0530 2023-10-16T09:43:53.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 -- 172.21.15.158:0/3118353312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5900d0530 msgr2=0x7fe5900d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 --2- 172.21.15.158:0/3118353312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5900d0530 0x7fe5900d09b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe5840098f0 tx=0x7fe584009c00 comp rx=0 tx=0).stop 2023-10-16T09:43:53.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 -- 172.21.15.158:0/3118353312 shutdown_connections 2023-10-16T09:43:53.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 --2- 172.21.15.158:0/3118353312 >> v2:172.21.15.172:3300/0 conn(0x7fe590160d30 0x7fe5900b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 --2- 172.21.15.158:0/3118353312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5900d0530 0x7fe5900d09b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 -- 172.21.15.158:0/3118353312 >> 172.21.15.158:0/3118353312 conn(0x7fe5900669f0 msgr2=0x7fe590066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:53.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 -- 172.21.15.158:0/3118353312 shutdown_connections 2023-10-16T09:43:53.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.898+0000 7fe595949700 1 -- 172.21.15.158:0/3118353312 wait complete. 2023-10-16T09:43:53.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.899+0000 7fe595949700 1 Processor -- start 2023-10-16T09:43:53.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.899+0000 7fe595949700 1 -- start start 2023-10-16T09:43:53.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe595949700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5900d0530 0x7fe5901f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe595949700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5900d19a0 0x7fe59015bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe595949700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe590160d30 0x7fe59014a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe595949700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900cabb0 con 0x7fe590160d30 2023-10-16T09:43:53.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe595949700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59015c500 con 0x7fe5900d0530 2023-10-16T09:43:53.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe595949700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59015e980 con 0x7fe5900d19a0 2023-10-16T09:43:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe590160d30 0x7fe59014a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe590160d30 0x7fe59014a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46972/0 (socket says 172.21.15.158:46972) 2023-10-16T09:43:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 -- 172.21.15.158:0/3217095955 learned_addr learned my addr 172.21.15.158:0/3217095955 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58effd700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5900d0530 0x7fe5901f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58e7fc700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5900d19a0 0x7fe59015bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 -- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5900d19a0 msgr2=0x7fe59015bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5900d19a0 0x7fe59015bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 -- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5900d0530 msgr2=0x7fe5901f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:53.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5900d0530 0x7fe5901f6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:53.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.900+0000 7fe58f7fe700 1 -- 172.21.15.158:0/3217095955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe584009550 con 0x7fe590160d30 2023-10-16T09:43:53.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.901+0000 7fe58f7fe700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe590160d30 0x7fe59014a0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe58800cc40 tx=0x7fe58800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.901+0000 7fe577fff700 1 -- 172.21.15.158:0/3217095955 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800aa80 con 0x7fe590160d30 2023-10-16T09:43:53.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.901+0000 7fe577fff700 1 -- 172.21.15.158:0/3217095955 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58800abe0 con 0x7fe590160d30 2023-10-16T09:43:53.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.901+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59015ec60 con 0x7fe590160d30 2023-10-16T09:43:53.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.901+0000 7fe577fff700 1 -- 172.21.15.158:0/3217095955 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588012420 con 0x7fe590160d30 2023-10-16T09:43:53.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.901+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59015e5c0 con 0x7fe590160d30 2023-10-16T09:43:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.902+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5900681b0 con 0x7fe590160d30 2023-10-16T09:43:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.902+0000 7fe577fff700 1 -- 172.21.15.158:0/3217095955 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe588012580 con 0x7fe590160d30 2023-10-16T09:43:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.902+0000 7fe577fff700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe57c06af00 0x7fe57c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.903+0000 7fe577fff700 1 -- 172.21.15.158:0/3217095955 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe588089ed0 con 0x7fe590160d30 2023-10-16T09:43:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.903+0000 7fe58effd700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe57c06af00 0x7fe57c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:53.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.903+0000 7fe58effd700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe57c06af00 0x7fe57c06d3c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe580006b20 tx=0x7fe580008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:53.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:53.907+0000 7fe577fff700 1 -- 172.21.15.158:0/3217095955 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe588054d20 con 0x7fe590160d30 2023-10-16T09:43:54.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.101+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe5900cce00 con 0x7fe590160d30 2023-10-16T09:43:54.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.101+0000 7fe577fff700 1 -- 172.21.15.158:0/3217095955 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe588016020 con 0x7fe590160d30 2023-10-16T09:43:54.103 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:54.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe57c06af00 msgr2=0x7fe57c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe57c06af00 0x7fe57c06d3c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe580006b20 tx=0x7fe580008040 comp rx=0 tx=0).stop 2023-10-16T09:43:54.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe590160d30 msgr2=0x7fe59014a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe590160d30 0x7fe59014a0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe58800cc40 tx=0x7fe58800cf50 comp rx=0 tx=0).stop 2023-10-16T09:43:54.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 shutdown_connections 2023-10-16T09:43:54.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe590160d30 0x7fe59014a0e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5900d0530 0x7fe5901f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe57c06af00 0x7fe57c06d3c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 --2- 172.21.15.158:0/3217095955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5900d19a0 0x7fe59015bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 >> 172.21.15.158:0/3217095955 conn(0x7fe5900669f0 msgr2=0x7fe5900b7400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:54.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 shutdown_connections 2023-10-16T09:43:54.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.103+0000 7fe595949700 1 -- 172.21.15.158:0/3217095955 wait complete. 2023-10-16T09:43:54.117 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:43:54.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.238+0000 7f5f6814b700 1 Processor -- start 2023-10-16T09:43:54.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.238+0000 7f5f6814b700 1 -- start start 2023-10-16T09:43:54.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f6814b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5f600cc590 0x7f5f600caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f6814b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600cbd30 con 0x7f5f6015da40 2023-10-16T09:43:54.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f6814b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600cc1c0 con 0x7f5f6006afe0 2023-10-16T09:43:54.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f6814b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6015de20 con 0x7f5f600cc590 2023-10-16T09:43:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f65ee7700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5f6015da40 0x7f5f6006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47152/0 (socket says 172.21.15.158:47152) 2023-10-16T09:43:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f65ee7700 1 -- 172.21.15.158:0/2004957812 learned_addr learned my addr 172.21.15.158:0/2004957812 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f666e8700 1 --2- 172.21.15.158:0/2004957812 >> v2:172.21.15.172:3300/0 conn(0x7f5f600cc590 0x7f5f600caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141637487 0 0) 0x7f5f600cbd30 con 0x7f5f6015da40 2023-10-16T09:43:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.239+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c0035b0 con 0x7f5f6015da40 2023-10-16T09:43:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3592500137 0 0) 0x7f5f4c0035b0 con 0x7f5f6015da40 2023-10-16T09:43:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 >> v2:172.21.15.172:3300/0 conn(0x7f5f600cc590 msgr2=0x7f5f600caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 --2- 172.21.15.158:0/2004957812 >> v2:172.21.15.172:3300/0 conn(0x7f5f600cc590 0x7f5f600caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 >> v1:172.21.15.172:6789/0 conn(0x7f5f6006afe0 legacy=0x7f5f600afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600ce260 con 0x7f5f6015da40 2023-10-16T09:43:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f5f50001670 con 0x7f5f6015da40 2023-10-16T09:43:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 >> v1:172.21.15.158:6789/0 conn(0x7f5f6015da40 legacy=0x7f5f6006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f4c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f4c005210 0x7f5f4c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6015de20 con 0x7f5f4c004850 2023-10-16T09:43:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c00c0b0 con 0x7f5f4c003a00 2023-10-16T09:43:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c00c270 con 0x7f5f4c005210 2023-10-16T09:43:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f656e6700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f4c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f65ee7700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f666e8700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f4c005210 0x7f5f4c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f656e6700 1 -- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f4c005210 msgr2=0x7f5f4c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f656e6700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f4c005210 0x7f5f4c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f656e6700 1 -- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c003a00 msgr2=0x7f5f4c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f656e6700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f666e8700 1 -- 172.21.15.158:0/2004957812 reap_dead start 2023-10-16T09:43:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f656e6700 1 -- 172.21.15.158:0/2004957812 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0035b0 con 0x7f5f4c004850 2023-10-16T09:43:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f65ee7700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.240+0000 7f5f656e6700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f4c004cd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0017c0 tx=0x7f5f4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2094115c2fafbd4f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0052d0 con 0x7f5f4c004850 2023-10-16T09:43:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c005450 con 0x7f5f4c004850 2023-10-16T09:43:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f64ee5700 1 -- 172.21.15.158:0/2004957812 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0038c0 con 0x7f5f4c004850 2023-10-16T09:43:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f6814b700 1 -- 172.21.15.158:0/2004957812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 msgr2=0x7f5f4c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f6814b700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f4c004cd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0017c0 tx=0x7f5f4c005880 comp rx=0 tx=0).stop 2023-10-16T09:43:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f6814b700 1 -- 172.21.15.158:0/2004957812 shutdown_connections 2023-10-16T09:43:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f6814b700 1 --2- 172.21.15.158:0/2004957812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f4c004cd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f6814b700 1 -- 172.21.15.158:0/2004957812 >> 172.21.15.158:0/2004957812 conn(0x7f5f60099a10 msgr2=0x7f5f600d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:54.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f6814b700 1 -- 172.21.15.158:0/2004957812 shutdown_connections 2023-10-16T09:43:54.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.241+0000 7f5f6814b700 1 -- 172.21.15.158:0/2004957812 wait complete. 2023-10-16T09:43:54.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.242+0000 7f5f6814b700 1 Processor -- start 2023-10-16T09:43:54.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.242+0000 7f5f6814b700 1 -- start start 2023-10-16T09:43:54.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f6814b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f60159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f6814b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f600c74a0 0x7f5f601f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f6814b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f6015f280 0x7f5f6015f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f6814b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600c8590 con 0x7f5f4c004850 2023-10-16T09:43:54.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f6814b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6015fc50 con 0x7f5f6015f280 2023-10-16T09:43:54.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f6814b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60159fb0 con 0x7f5f600c74a0 2023-10-16T09:43:54.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f65ee7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f60159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f65ee7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f60159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46988/0 (socket says 172.21.15.158:46988) 2023-10-16T09:43:54.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f65ee7700 1 -- 172.21.15.158:0/503523935 learned_addr learned my addr 172.21.15.158:0/503523935 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:54.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f656e6700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f600c74a0 0x7f5f601f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.243+0000 7f5f666e8700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f6015f280 0x7f5f6015f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f65ee7700 1 -- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f600c74a0 msgr2=0x7f5f601f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f65ee7700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f600c74a0 0x7f5f601f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f65ee7700 1 -- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f6015f280 msgr2=0x7f5f6015f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f65ee7700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f6015f280 0x7f5f6015f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f65ee7700 1 -- 172.21.15.158:0/503523935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f601f3e50 con 0x7f5f4c004850 2023-10-16T09:43:54.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f666e8700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f6015f280 0x7f5f6015f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:54.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f656e6700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f600c74a0 0x7f5f601f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:43:54.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f65ee7700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f60159a70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5f50007290 tx=0x7f5f50007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f567fc700 1 -- 172.21.15.158:0/503523935 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5000f7f0 con 0x7f5f4c004850 2023-10-16T09:43:54.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f567fc700 1 -- 172.21.15.158:0/503523935 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5000f970 con 0x7f5f4c004850 2023-10-16T09:43:54.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f600c7a10 con 0x7f5f4c004850 2023-10-16T09:43:54.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f567fc700 1 -- 172.21.15.158:0/503523935 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5000fd20 con 0x7f5f4c004850 2023-10-16T09:43:54.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.244+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f600c7fd0 con 0x7f5f4c004850 2023-10-16T09:43:54.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.245+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6006c6d0 con 0x7f5f4c004850 2023-10-16T09:43:54.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.245+0000 7f5f567fc700 1 -- 172.21.15.158:0/503523935 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5f5001d460 con 0x7f5f4c004850 2023-10-16T09:43:54.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.246+0000 7f5f567fc700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5f4406af00 0x7f5f4406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.246+0000 7f5f656e6700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5f4406af00 0x7f5f4406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.246+0000 7f5f567fc700 1 -- 172.21.15.158:0/503523935 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5f500898c0 con 0x7f5f4c004850 2023-10-16T09:43:54.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.246+0000 7f5f656e6700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5f4406af00 0x7f5f4406d3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c015040 tx=0x7f5f5c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.250+0000 7f5f567fc700 1 -- 172.21.15.158:0/503523935 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f500545e0 con 0x7f5f4c004850 2023-10-16T09:43:54.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.442+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f600681b0 con 0x7f5f4c004850 2023-10-16T09:43:54.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.443+0000 7f5f567fc700 1 -- 172.21.15.158:0/503523935 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5f50057c00 con 0x7f5f4c004850 2023-10-16T09:43:54.445 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:43:54.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5f4406af00 msgr2=0x7f5f4406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5f4406af00 0x7f5f4406d3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c015040 tx=0x7f5f5c009030 comp rx=0 tx=0).stop 2023-10-16T09:43:54.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 msgr2=0x7f5f60159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f60159a70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5f50007290 tx=0x7f5f50007650 comp rx=0 tx=0).stop 2023-10-16T09:43:54.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 shutdown_connections 2023-10-16T09:43:54.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f6015f280 0x7f5f6015f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c004850 0x7f5f60159a70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5f4406af00 0x7f5f4406d3c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 --2- 172.21.15.158:0/503523935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f600c74a0 0x7f5f601f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 >> 172.21.15.158:0/503523935 conn(0x7f5f60099a10 msgr2=0x7f5f6015b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:54.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 shutdown_connections 2023-10-16T09:43:54.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.445+0000 7f5f6814b700 1 -- 172.21.15.158:0/503523935 wait complete. 2023-10-16T09:43:54.460 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:43:54.485 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:43:54.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:43:54.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.587+0000 7f35d3836700 1 Processor -- start 2023-10-16T09:43:54.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.587+0000 7f35d3836700 1 -- start start 2023-10-16T09:43:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d3836700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d0c40 0x7f35cc0b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d3836700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc0b7c90 con 0x7f35cc160d30 2023-10-16T09:43:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d3836700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc0b7f70 con 0x7f35cc0d29b0 2023-10-16T09:43:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d3836700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0b8910 con 0x7f35cc0d0c40 2023-10-16T09:43:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d15d2700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f35cc160d30 0x7f35cc0b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47172/0 (socket says 172.21.15.158:47172) 2023-10-16T09:43:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d15d2700 1 -- 172.21.15.158:0/1865352600 learned_addr learned my addr 172.21.15.158:0/1865352600 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d1dd3700 1 --2- 172.21.15.158:0/1865352600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d0c40 0x7f35cc0b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35c3fff700 1 -- 172.21.15.158:0/1865352600 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441835003 0 0) 0x7f35cc0b7c90 con 0x7f35cc160d30 2023-10-16T09:43:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35c3fff700 1 -- 172.21.15.158:0/1865352600 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b80035b0 con 0x7f35cc160d30 2023-10-16T09:43:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35c3fff700 1 -- 172.21.15.158:0/1865352600 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668518434 0 0) 0x7f35cc0b7f70 con 0x7f35cc0d29b0 2023-10-16T09:43:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35c3fff700 1 -- 172.21.15.158:0/1865352600 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc0b7c90 con 0x7f35cc0d29b0 2023-10-16T09:43:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d1dd3700 1 -- 172.21.15.158:0/1865352600 >> v1:172.21.15.172:6789/0 conn(0x7f35cc0d29b0 legacy=0x7f35cc0d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d1dd3700 1 -- 172.21.15.158:0/1865352600 >> v1:172.21.15.158:6789/0 conn(0x7f35cc160d30 legacy=0x7f35cc0b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.588+0000 7f35d1dd3700 1 -- 172.21.15.158:0/1865352600 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0b8250 con 0x7f35cc0d0c40 2023-10-16T09:43:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d1dd3700 1 --2- 172.21.15.158:0/1865352600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d0c40 0x7f35cc0b7590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f35c4009dc0 tx=0x7f35c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c101600d7fe44f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35c3fff700 1 -- 172.21.15.158:0/1865352600 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c400e070 con 0x7f35cc0d0c40 2023-10-16T09:43:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35c3fff700 1 -- 172.21.15.158:0/1865352600 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c400bb20 con 0x7f35cc0d0c40 2023-10-16T09:43:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35c3fff700 1 -- 172.21.15.158:0/1865352600 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4012430 con 0x7f35cc0d0c40 2023-10-16T09:43:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d3836700 1 -- 172.21.15.158:0/1865352600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d0c40 msgr2=0x7f35cc0b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d3836700 1 --2- 172.21.15.158:0/1865352600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d0c40 0x7f35cc0b7590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f35c4009dc0 tx=0x7f35c400b330 comp rx=0 tx=0).stop 2023-10-16T09:43:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d3836700 1 -- 172.21.15.158:0/1865352600 shutdown_connections 2023-10-16T09:43:54.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d3836700 1 --2- 172.21.15.158:0/1865352600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d0c40 0x7f35cc0b7590 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d3836700 1 -- 172.21.15.158:0/1865352600 >> 172.21.15.158:0/1865352600 conn(0x7f35cc0669f0 msgr2=0x7f35cc066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:54.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d3836700 1 -- 172.21.15.158:0/1865352600 shutdown_connections 2023-10-16T09:43:54.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.589+0000 7f35d3836700 1 -- 172.21.15.158:0/1865352600 wait complete. 2023-10-16T09:43:54.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 Processor -- start 2023-10-16T09:43:54.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 -- start start 2023-10-16T09:43:54.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc0d0c40 0x7f35cc14c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d29b0 0x7f35cc15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35cc160d30 0x7f35cc0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0ca690 con 0x7f35cc0d0c40 2023-10-16T09:43:54.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc14d820 con 0x7f35cc160d30 2023-10-16T09:43:54.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d3836700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0ca8e0 con 0x7f35cc0d29b0 2023-10-16T09:43:54.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d15d2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc0d0c40 0x7f35cc14c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d15d2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc0d0c40 0x7f35cc14c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47002/0 (socket says 172.21.15.158:47002) 2023-10-16T09:43:54.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.591+0000 7f35d15d2700 1 -- 172.21.15.158:0/3083942544 learned_addr learned my addr 172.21.15.158:0/3083942544 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:54.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d1dd3700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35cc160d30 0x7f35cc0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d0dd1700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d29b0 0x7f35cc15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d15d2700 1 -- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d29b0 msgr2=0x7f35cc15f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d15d2700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d29b0 0x7f35cc15f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d15d2700 1 -- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35cc160d30 msgr2=0x7f35cc0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d15d2700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35cc160d30 0x7f35cc0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d15d2700 1 -- 172.21.15.158:0/3083942544 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c4009a70 con 0x7f35cc0d0c40 2023-10-16T09:43:54.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d15d2700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc0d0c40 0x7f35cc14c550 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f35c8001eb0 tx=0x7f35c800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35c1ffb700 1 -- 172.21.15.158:0/3083942544 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8005d70 con 0x7f35cc0d0c40 2023-10-16T09:43:54.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc0cabc0 con 0x7f35cc0d0c40 2023-10-16T09:43:54.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35c1ffb700 1 -- 172.21.15.158:0/3083942544 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c80070d0 con 0x7f35cc0d0c40 2023-10-16T09:43:54.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35c1ffb700 1 -- 172.21.15.158:0/3083942544 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8007460 con 0x7f35cc0d0c40 2023-10-16T09:43:54.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.592+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc0cb180 con 0x7f35cc0d0c40 2023-10-16T09:43:54.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.593+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc0681b0 con 0x7f35cc0d0c40 2023-10-16T09:43:54.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.593+0000 7f35c1ffb700 1 -- 172.21.15.158:0/3083942544 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f35c80075c0 con 0x7f35cc0d0c40 2023-10-16T09:43:54.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.594+0000 7f35c1ffb700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f35b006af00 0x7f35b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.594+0000 7f35c1ffb700 1 -- 172.21.15.158:0/3083942544 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f35c8088ca0 con 0x7f35cc0d0c40 2023-10-16T09:43:54.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.594+0000 7f35d0dd1700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f35b006af00 0x7f35b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.594+0000 7f35d0dd1700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f35b006af00 0x7f35b006d3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f35cc154a60 tx=0x7f35bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.598+0000 7f35c1ffb700 1 -- 172.21.15.158:0/3083942544 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35c8053a70 con 0x7f35cc0d0c40 2023-10-16T09:43:54.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.789+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f35cc151220 con 0x7f35cc0d0c40 2023-10-16T09:43:54.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.789+0000 7f35c1ffb700 1 -- 172.21.15.158:0/3083942544 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f35c8057090 con 0x7f35cc0d0c40 2023-10-16T09:43:54.791 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:43:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f35b006af00 msgr2=0x7f35b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f35b006af00 0x7f35b006d3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f35cc154a60 tx=0x7f35bc009040 comp rx=0 tx=0).stop 2023-10-16T09:43:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc0d0c40 msgr2=0x7f35cc14c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc0d0c40 0x7f35cc14c550 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f35c8001eb0 tx=0x7f35c800c7b0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 shutdown_connections 2023-10-16T09:43:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35cc160d30 0x7f35cc0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc0d0c40 0x7f35cc14c550 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f35b006af00 0x7f35b006d3c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 --2- 172.21.15.158:0/3083942544 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35cc0d29b0 0x7f35cc15f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 >> 172.21.15.158:0/3083942544 conn(0x7f35cc0669f0 msgr2=0x7f35cc150600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:54.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 shutdown_connections 2023-10-16T09:43:54.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.791+0000 7f35d3836700 1 -- 172.21.15.158:0/3083942544 wait complete. 2023-10-16T09:43:54.805 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:54.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.964+0000 7fbf72c74700 1 Processor -- start 2023-10-16T09:43:54.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.964+0000 7fbf72c74700 1 -- start start 2023-10-16T09:43:54.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.965+0000 7fbf72c74700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c06fa50 0x7fbf6c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.965+0000 7fbf72c74700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6c06e770 con 0x7fbf6c15dfb0 2023-10-16T09:43:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.965+0000 7fbf72c74700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6c0cbe00 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.965+0000 7fbf72c74700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0cf270 con 0x7fbf6c06fa50 2023-10-16T09:43:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.965+0000 7fbf6bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbf6c15dfb0 0x7fbf6c06f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47186/0 (socket says 172.21.15.158:47186) 2023-10-16T09:43:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.965+0000 7fbf6bfff700 1 -- 172.21.15.158:0/554403866 learned_addr learned my addr 172.21.15.158:0/554403866 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.965+0000 7fbf71211700 1 --2- 172.21.15.158:0/554403866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c06fa50 0x7fbf6c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf6b7fe700 1 -- 172.21.15.158:0/554403866 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942662175 0 0) 0x7fbf58001110 con 0x7fbf6c15dfb0 2023-10-16T09:43:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf6b7fe700 1 -- 172.21.15.158:0/554403866 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c0035b0 con 0x7fbf6c15dfb0 2023-10-16T09:43:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf6b7fe700 1 -- 172.21.15.158:0/554403866 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853191563 0 0) 0x7fbf6c0cbe00 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf6b7fe700 1 -- 172.21.15.158:0/554403866 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf58001110 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf71211700 1 -- 172.21.15.158:0/554403866 >> v1:172.21.15.172:6789/0 conn(0x7fbf6c0cc5d0 legacy=0x7fbf6c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf71211700 1 -- 172.21.15.158:0/554403866 >> v1:172.21.15.158:6789/0 conn(0x7fbf6c15dfb0 legacy=0x7fbf6c06f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf71211700 1 -- 172.21.15.158:0/554403866 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c15e4e0 con 0x7fbf6c06fa50 2023-10-16T09:43:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf71211700 1 --2- 172.21.15.158:0/554403866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c06fa50 0x7fbf6c06e230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009dc0 tx=0x7fbf6400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e4affeb82a4d37b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf6b7fe700 1 -- 172.21.15.158:0/554403866 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6400e070 con 0x7fbf6c06fa50 2023-10-16T09:43:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf6b7fe700 1 -- 172.21.15.158:0/554403866 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6400bb20 con 0x7fbf6c06fa50 2023-10-16T09:43:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.966+0000 7fbf6b7fe700 1 -- 172.21.15.158:0/554403866 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64012430 con 0x7fbf6c06fa50 2023-10-16T09:43:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.967+0000 7fbf72c74700 1 -- 172.21.15.158:0/554403866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c06fa50 msgr2=0x7fbf6c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.967+0000 7fbf72c74700 1 --2- 172.21.15.158:0/554403866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c06fa50 0x7fbf6c06e230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009dc0 tx=0x7fbf6400b330 comp rx=0 tx=0).stop 2023-10-16T09:43:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.967+0000 7fbf72c74700 1 -- 172.21.15.158:0/554403866 shutdown_connections 2023-10-16T09:43:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.967+0000 7fbf72c74700 1 --2- 172.21.15.158:0/554403866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c06fa50 0x7fbf6c06e230 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.967+0000 7fbf72c74700 1 -- 172.21.15.158:0/554403866 >> 172.21.15.158:0/554403866 conn(0x7fbf6c0d02b0 msgr2=0x7fbf6c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.967+0000 7fbf72c74700 1 -- 172.21.15.158:0/554403866 shutdown_connections 2023-10-16T09:43:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.967+0000 7fbf72c74700 1 -- 172.21.15.158:0/554403866 wait complete. 2023-10-16T09:43:54.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.968+0000 7fbf72c74700 1 Processor -- start 2023-10-16T09:43:54.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.968+0000 7fbf72c74700 1 -- start start 2023-10-16T09:43:54.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf72c74700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf6c06fa50 0x7fbf6c153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf72c74700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf6c0cc5d0 0x7fbf6c14fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf72c74700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c15dfb0 0x7fbf6c1569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf72c74700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c156560 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf72c74700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c154190 con 0x7fbf6c06fa50 2023-10-16T09:43:54.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf72c74700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c155950 con 0x7fbf6c15dfb0 2023-10-16T09:43:54.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf6c0cc5d0 0x7fbf6c14fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf6c0cc5d0 0x7fbf6c14fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47018/0 (socket says 172.21.15.158:47018) 2023-10-16T09:43:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 -- 172.21.15.158:0/609652111 learned_addr learned my addr 172.21.15.158:0/609652111 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf70a10700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf6c06fa50 0x7fbf6c153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf71211700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c15dfb0 0x7fbf6c1569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 -- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c15dfb0 msgr2=0x7fbf6c1569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c15dfb0 0x7fbf6c1569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 -- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf6c06fa50 msgr2=0x7fbf6c153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf6c06fa50 0x7fbf6c153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 -- 172.21.15.158:0/609652111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf64009a70 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf6bfff700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf6c0cc5d0 0x7fbf6c14fd90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800c680 tx=0x7fbf5800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf697fa700 1 -- 172.21.15.158:0/609652111 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58005d70 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf697fa700 1 -- 172.21.15.158:0/609652111 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf580070d0 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.969+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c155c30 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.970+0000 7fbf697fa700 1 -- 172.21.15.158:0/609652111 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58007460 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.970+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c1561f0 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.970+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c0681b0 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.971+0000 7fbf697fa700 1 -- 172.21.15.158:0/609652111 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbf580075c0 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.971+0000 7fbf697fa700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbf5006af00 0x7fbf5006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.971+0000 7fbf697fa700 1 -- 172.21.15.158:0/609652111 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbf58088c80 con 0x7fbf6c0cc5d0 2023-10-16T09:43:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.971+0000 7fbf70a10700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbf5006af00 0x7fbf5006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.972+0000 7fbf70a10700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbf5006af00 0x7fbf5006d3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbf60001220 tx=0x7fbf60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:54.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:54.976+0000 7fbf697fa700 1 -- 172.21.15.158:0/609652111 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf58053a40 con 0x7fbf6c0cc5d0 2023-10-16T09:43:55.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.211+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf6c06e770 con 0x7fbf6c0cc5d0 2023-10-16T09:43:55.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.212+0000 7fbf697fa700 1 -- 172.21.15.158:0/609652111 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7fbf6c06e770 con 0x7fbf6c0cc5d0 2023-10-16T09:43:55.214 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:55.214 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:43:55.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.214+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbf5006af00 msgr2=0x7fbf5006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.214+0000 7fbf72c74700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbf5006af00 0x7fbf5006d3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbf60001220 tx=0x7fbf60009040 comp rx=0 tx=0).stop 2023-10-16T09:43:55.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.214+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf6c0cc5d0 msgr2=0x7fbf6c14fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf6c0cc5d0 0x7fbf6c14fd90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800c680 tx=0x7fbf5800c990 comp rx=0 tx=0).stop 2023-10-16T09:43:55.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 shutdown_connections 2023-10-16T09:43:55.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbf5006af00 0x7fbf5006d3c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf6c15dfb0 0x7fbf6c1569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf6c06fa50 0x7fbf6c153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 --2- 172.21.15.158:0/609652111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf6c0cc5d0 0x7fbf6c14fd90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 >> 172.21.15.158:0/609652111 conn(0x7fbf6c0d02b0 msgr2=0x7fbf6c0d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:55.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 shutdown_connections 2023-10-16T09:43:55.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.215+0000 7fbf72c74700 1 -- 172.21.15.158:0/609652111 wait complete. 2023-10-16T09:43:55.219 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:43:55.228 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:43:55.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.347+0000 7f2a23368700 1 Processor -- start 2023-10-16T09:43:55.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.347+0000 7f2a23368700 1 -- start start 2023-10-16T09:43:55.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.348+0000 7f2a23368700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c06ab40 0x7f2a1c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.348+0000 7f2a23368700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a1c0729b0 0x7f2a1c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.348+0000 7f2a23368700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c071e50 con 0x7f2a1c06e5d0 2023-10-16T09:43:55.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.348+0000 7f2a23368700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c161270 con 0x7f2a1c0729b0 2023-10-16T09:43:55.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.348+0000 7f2a23368700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c161510 con 0x7f2a1c06ab40 2023-10-16T09:43:55.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21104700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c06ab40 0x7f2a1c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21905700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a1c0729b0 0x7f2a1c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a20903700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2a1c06e5d0 0x7f2a1c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:51104/0 (socket says 172.21.15.158:51104) 2023-10-16T09:43:55.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a20903700 1 -- 172.21.15.158:0/3432337069 learned_addr learned my addr 172.21.15.158:0/3432337069 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:55.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21905700 1 -- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c06ab40 msgr2=0x7f2a1c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:55.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21905700 1 --2- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c06ab40 0x7f2a1c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21905700 1 -- 172.21.15.158:0/3432337069 >> v1:172.21.15.172:6789/0 conn(0x7f2a1c06e5d0 legacy=0x7f2a1c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21905700 1 -- 172.21.15.158:0/3432337069 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c1617f0 con 0x7f2a1c0729b0 2023-10-16T09:43:55.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a13fff700 1 -- 172.21.15.158:0/3432337069 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108261035 0 0) 0x7f2a1c071e50 con 0x7f2a1c06e5d0 2023-10-16T09:43:55.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21104700 1 --2- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c06ab40 0x7f2a1c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:55.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a21905700 1 --2- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c0729b0 0x7f2a1c071910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a1400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27c6bba1b1768d48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:55.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a13fff700 1 -- 172.21.15.158:0/3432337069 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400e070 con 0x7f2a1c0729b0 2023-10-16T09:43:55.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a13fff700 1 -- 172.21.15.158:0/3432337069 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1400bb20 con 0x7f2a1c0729b0 2023-10-16T09:43:55.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.349+0000 7f2a13fff700 1 -- 172.21.15.158:0/3432337069 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14012430 con 0x7f2a1c0729b0 2023-10-16T09:43:55.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 -- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c0729b0 msgr2=0x7f2a1c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 --2- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c0729b0 0x7f2a1c071910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a1400b330 comp rx=0 tx=0).stop 2023-10-16T09:43:55.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 -- 172.21.15.158:0/3432337069 shutdown_connections 2023-10-16T09:43:55.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 --2- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c0729b0 0x7f2a1c071910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 --2- 172.21.15.158:0/3432337069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c06ab40 0x7f2a1c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 -- 172.21.15.158:0/3432337069 >> 172.21.15.158:0/3432337069 conn(0x7f2a1c0d02b0 msgr2=0x7f2a1c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:55.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 -- 172.21.15.158:0/3432337069 shutdown_connections 2023-10-16T09:43:55.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.350+0000 7f2a23368700 1 -- 172.21.15.158:0/3432337069 wait complete. 2023-10-16T09:43:55.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.351+0000 7f2a23368700 1 Processor -- start 2023-10-16T09:43:55.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.351+0000 7f2a23368700 1 -- start start 2023-10-16T09:43:55.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a23368700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c06ab40 0x7f2a1c0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a23368700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1c06e5d0 0x7f2a1c0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a23368700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c0729b0 0x7f2a1c0b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a23368700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c146280 con 0x7f2a1c06e5d0 2023-10-16T09:43:55.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a23368700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c146560 con 0x7f2a1c06ab40 2023-10-16T09:43:55.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a23368700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c146840 con 0x7f2a1c0729b0 2023-10-16T09:43:55.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a20903700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1c06e5d0 0x7f2a1c0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a20903700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1c06e5d0 0x7f2a1c0afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47030/0 (socket says 172.21.15.158:47030) 2023-10-16T09:43:55.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a21104700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c06ab40 0x7f2a1c0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a20903700 1 -- 172.21.15.158:0/794717264 learned_addr learned my addr 172.21.15.158:0/794717264 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:55.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a21905700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c0729b0 0x7f2a1c0b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a21905700 1 -- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c06ab40 msgr2=0x7f2a1c0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a21905700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c06ab40 0x7f2a1c0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a21905700 1 -- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1c06e5d0 msgr2=0x7f2a1c0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a21905700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1c06e5d0 0x7f2a1c0afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a21905700 1 -- 172.21.15.158:0/794717264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0094a0 con 0x7f2a1c0729b0 2023-10-16T09:43:55.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.352+0000 7f2a20903700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1c06e5d0 0x7f2a1c0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:43:55.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.353+0000 7f2a21905700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c0729b0 0x7f2a1c0b5070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a14016000 tx=0x7f2a14004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:55.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.353+0000 7f2a11ffb700 1 -- 172.21.15.158:0/794717264 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14012430 con 0x7f2a1c0729b0 2023-10-16T09:43:55.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.353+0000 7f2a11ffb700 1 -- 172.21.15.158:0/794717264 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a14020070 con 0x7f2a1c0729b0 2023-10-16T09:43:55.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.353+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a14009a70 con 0x7f2a1c0729b0 2023-10-16T09:43:55.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.353+0000 7f2a11ffb700 1 -- 172.21.15.158:0/794717264 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14004280 con 0x7f2a1c0729b0 2023-10-16T09:43:55.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.353+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c146e00 con 0x7f2a1c0729b0 2023-10-16T09:43:55.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.354+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c0681b0 con 0x7f2a1c0729b0 2023-10-16T09:43:55.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.355+0000 7f2a11ffb700 1 -- 172.21.15.158:0/794717264 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2a140043e0 con 0x7f2a1c0729b0 2023-10-16T09:43:55.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.355+0000 7f2a11ffb700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f2a0006af00 0x7f2a0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.355+0000 7f2a11ffb700 1 -- 172.21.15.158:0/794717264 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2a14022070 con 0x7f2a1c0729b0 2023-10-16T09:43:55.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.355+0000 7f2a21104700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f2a0006af00 0x7f2a0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.355+0000 7f2a21104700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f2a0006af00 0x7f2a0006d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009d70 tx=0x7f2a0c009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:55.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.360+0000 7f2a11ffb700 1 -- 172.21.15.158:0/794717264 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a14056270 con 0x7f2a1c0729b0 2023-10-16T09:43:55.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.554+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a1c06db50 con 0x7f2a1c0729b0 2023-10-16T09:43:55.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.556+0000 7f2a11ffb700 1 -- 172.21.15.158:0/794717264 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f2a14059890 con 0x7f2a1c0729b0 2023-10-16T09:43:55.558 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:55.558 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:43:55.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f2a0006af00 msgr2=0x7f2a0006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f2a0006af00 0x7f2a0006d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009d70 tx=0x7f2a0c009210 comp rx=0 tx=0).stop 2023-10-16T09:43:55.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c0729b0 msgr2=0x7f2a1c0b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c0729b0 0x7f2a1c0b5070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a14016000 tx=0x7f2a14004ad0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 shutdown_connections 2023-10-16T09:43:55.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f2a0006af00 0x7f2a0006d3c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a1c0729b0 0x7f2a1c0b5070 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a1c06ab40 0x7f2a1c0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 --2- 172.21.15.158:0/794717264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1c06e5d0 0x7f2a1c0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 >> 172.21.15.158:0/794717264 conn(0x7f2a1c0d02b0 msgr2=0x7f2a1c072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:55.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 shutdown_connections 2023-10-16T09:43:55.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.558+0000 7f2a23368700 1 -- 172.21.15.158:0/794717264 wait complete. 2023-10-16T09:43:55.572 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:43:55.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.692+0000 7f99f43f2700 1 Processor -- start 2023-10-16T09:43:55.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.692+0000 7f99f43f2700 1 -- start start 2023-10-16T09:43:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f43f2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99ec0ce3c0 0x7f99ec0cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f43f2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec15e7f0 con 0x7f99ec0cf4c0 2023-10-16T09:43:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f43f2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec06c3b0 con 0x7f99ec15e3a0 2023-10-16T09:43:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f43f2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ec0cf970 con 0x7f99ec0ce3c0 2023-10-16T09:43:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f198d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f99ec0cf4c0 0x7f99ec15dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47214/0 (socket says 172.21.15.158:47214) 2023-10-16T09:43:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f198d700 1 -- 172.21.15.158:0/2824244780 learned_addr learned my addr 172.21.15.158:0/2824244780 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436342769 0 0) 0x7f99dc001110 con 0x7f99ec0cf4c0 2023-10-16T09:43:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d80035b0 con 0x7f99ec0cf4c0 2023-10-16T09:43:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f218e700 1 --2- 172.21.15.158:0/2824244780 >> v2:172.21.15.172:3300/0 conn(0x7f99ec0ce3c0 0x7f99ec0cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2595706604 0 0) 0x7f99d80035b0 con 0x7f99ec0cf4c0 2023-10-16T09:43:55.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 >> v2:172.21.15.172:3300/0 conn(0x7f99ec0ce3c0 msgr2=0x7f99ec0cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:55.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 --2- 172.21.15.158:0/2824244780 >> v2:172.21.15.172:3300/0 conn(0x7f99ec0ce3c0 0x7f99ec0cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 >> v1:172.21.15.172:6789/0 conn(0x7f99ec15e3a0 legacy=0x7f99ec06c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec15ea30 con 0x7f99ec0cf4c0 2023-10-16T09:43:55.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f99dc001800 con 0x7f99ec0cf4c0 2023-10-16T09:43:55.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 >> v1:172.21.15.158:6789/0 conn(0x7f99ec0cf4c0 legacy=0x7f99ec15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.693+0000 7f99f118c700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d8003a00 0x7f99d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d8004850 0x7f99d8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8005210 0x7f99d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0cf970 con 0x7f99d8003a00 2023-10-16T09:43:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d800c0b0 con 0x7f99d8004850 2023-10-16T09:43:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d800c270 con 0x7f99d8005210 2023-10-16T09:43:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f218e700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d8003a00 0x7f99d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f198d700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d8004850 0x7f99d8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f298f700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8005210 0x7f99d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f218e700 1 -- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8005210 msgr2=0x7f99d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:55.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f218e700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8005210 0x7f99d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f218e700 1 -- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d8004850 msgr2=0x7f99d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:55.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f218e700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d8004850 0x7f99d8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f298f700 1 -- 172.21.15.158:0/2824244780 reap_dead start 2023-10-16T09:43:55.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f218e700 1 -- 172.21.15.158:0/2824244780 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d80035b0 con 0x7f99d8003a00 2023-10-16T09:43:55.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f218e700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d8003a00 0x7f99d8003e20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009b60 tx=0x7f99d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a921c7bbacfb6960 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:55.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e800e6c0 con 0x7f99d8003a00 2023-10-16T09:43:55.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e800e820 con 0x7f99d8003a00 2023-10-16T09:43:55.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.694+0000 7f99f118c700 1 -- 172.21.15.158:0/2824244780 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e800eb90 con 0x7f99d8003a00 2023-10-16T09:43:55.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.695+0000 7f99f43f2700 1 -- 172.21.15.158:0/2824244780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d8003a00 msgr2=0x7f99d8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.695+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d8003a00 0x7f99d8003e20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009b60 tx=0x7f99d8005880 comp rx=0 tx=0).stop 2023-10-16T09:43:55.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.695+0000 7f99f43f2700 1 -- 172.21.15.158:0/2824244780 shutdown_connections 2023-10-16T09:43:55.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.695+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2824244780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d8003a00 0x7f99d8003e20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.695+0000 7f99f43f2700 1 -- 172.21.15.158:0/2824244780 >> 172.21.15.158:0/2824244780 conn(0x7f99ec06a8d0 msgr2=0x7f99ec06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:55.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.695+0000 7f99f43f2700 1 -- 172.21.15.158:0/2824244780 shutdown_connections 2023-10-16T09:43:55.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.695+0000 7f99f43f2700 1 -- 172.21.15.158:0/2824244780 wait complete. 2023-10-16T09:43:55.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.696+0000 7f99f43f2700 1 Processor -- start 2023-10-16T09:43:55.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.696+0000 7f99f43f2700 1 -- start start 2023-10-16T09:43:55.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f43f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8003a00 0x7f99ec1f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f43f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99ec161260 0x7f99ec0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f43f2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99ec1616e0 0x7f99ec0c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f43f2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec157d20 con 0x7f99ec161260 2023-10-16T09:43:55.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f43f2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec1f66a0 con 0x7f99ec1616e0 2023-10-16T09:43:55.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f43f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0cb330 con 0x7f99d8003a00 2023-10-16T09:43:55.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99ec161260 0x7f99ec0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99ec161260 0x7f99ec0c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47050/0 (socket says 172.21.15.158:47050) 2023-10-16T09:43:55.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 -- 172.21.15.158:0/2049132622 learned_addr learned my addr 172.21.15.158:0/2049132622 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:55.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f298f700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99ec1616e0 0x7f99ec0c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f218e700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8003a00 0x7f99ec1f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 -- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8003a00 msgr2=0x7f99ec1f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8003a00 0x7f99ec1f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 -- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99ec1616e0 msgr2=0x7f99ec0c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99ec1616e0 0x7f99ec0c4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 -- 172.21.15.158:0/2049132622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec0cb610 con 0x7f99ec161260 2023-10-16T09:43:55.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.697+0000 7f99f198d700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99ec161260 0x7f99ec0c3e20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0070c0 tx=0x7f99dc0073d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:55.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.698+0000 7f99e2ffd700 1 -- 172.21.15.158:0/2049132622 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc011580 con 0x7f99ec161260 2023-10-16T09:43:55.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.698+0000 7f99e2ffd700 1 -- 172.21.15.158:0/2049132622 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc0116e0 con 0x7f99ec161260 2023-10-16T09:43:55.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.698+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ec1f68f0 con 0x7f99ec161260 2023-10-16T09:43:55.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.698+0000 7f99e2ffd700 1 -- 172.21.15.158:0/2049132622 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc011a90 con 0x7f99ec161260 2023-10-16T09:43:55.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.698+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec0ca5b0 con 0x7f99ec161260 2023-10-16T09:43:55.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.699+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ec15f1c0 con 0x7f99ec161260 2023-10-16T09:43:55.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.699+0000 7f99e2ffd700 1 -- 172.21.15.158:0/2049132622 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f99dc011bf0 con 0x7f99ec161260 2023-10-16T09:43:55.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.699+0000 7f99e2ffd700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f99d006af00 0x7f99d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:55.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.700+0000 7f99e2ffd700 1 -- 172.21.15.158:0/2049132622 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f99dc089730 con 0x7f99ec161260 2023-10-16T09:43:55.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.700+0000 7f99f218e700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f99d006af00 0x7f99d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:55.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.700+0000 7f99f218e700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f99d006af00 0x7f99d006d3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f99e80090b0 tx=0x7f99e801f040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:55.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.704+0000 7f99e2ffd700 1 -- 172.21.15.158:0/2049132622 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99dc054510 con 0x7f99ec161260 2023-10-16T09:43:55.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.939+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99ec15e7f0 con 0x7f99ec161260 2023-10-16T09:43:55.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.940+0000 7f99e2ffd700 1 -- 172.21.15.158:0/2049132622 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7f99ec15e7f0 con 0x7f99ec161260 2023-10-16T09:43:55.942 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:43:55.942 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:43:55.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f99d006af00 msgr2=0x7f99d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f99d006af00 0x7f99d006d3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f99e80090b0 tx=0x7f99e801f040 comp rx=0 tx=0).stop 2023-10-16T09:43:55.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99ec161260 msgr2=0x7f99ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:55.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99ec161260 0x7f99ec0c3e20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0070c0 tx=0x7f99dc0073d0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 shutdown_connections 2023-10-16T09:43:55.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99ec1616e0 0x7f99ec0c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f99d006af00 0x7f99d006d3c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d8003a00 0x7f99ec1f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 --2- 172.21.15.158:0/2049132622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99ec161260 0x7f99ec0c3e20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:55.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 >> 172.21.15.158:0/2049132622 conn(0x7f99ec06a8d0 msgr2=0x7f99ec0cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:55.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 shutdown_connections 2023-10-16T09:43:55.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:55.942+0000 7f99f43f2700 1 -- 172.21.15.158:0/2049132622 wait complete. 2023-10-16T09:43:55.946 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:43:56.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T09:43:21.310458+0000', 'modified': '2023-10-16T09:43:43.765523+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4267}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4267': {'gid': 4267, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/3958845884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 3958845884}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 3958845884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T09:43:56.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T09:43:56.359 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T09:43:56.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.484+0000 7f771c3f9700 1 Processor -- start 2023-10-16T09:43:56.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.484+0000 7f771c3f9700 1 -- start start 2023-10-16T09:43:56.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.485+0000 7f771c3f9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f771415d990 0x7f77140b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.485+0000 7f771c3f9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77140af600 con 0x7f77140cf650 2023-10-16T09:43:56.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.485+0000 7f771c3f9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771406e9b0 con 0x7f771406e480 2023-10-16T09:43:56.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.485+0000 7f771c3f9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77140b2c90 con 0x7f771415d990 2023-10-16T09:43:56.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.485+0000 7f771a195700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f771415d990 0x7f77140b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.485+0000 7f771a996700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f77140cf650 0x7f771406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47234/0 (socket says 172.21.15.158:47234) 2023-10-16T09:43:56.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.485+0000 7f771a996700 1 -- 172.21.15.158:0/311904862 learned_addr learned my addr 172.21.15.158:0/311904862 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:56.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f771a195700 1 -- 172.21.15.158:0/311904862 >> v1:172.21.15.172:6789/0 conn(0x7f771406e480 legacy=0x7f77140b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f771a195700 1 -- 172.21.15.158:0/311904862 >> v1:172.21.15.158:6789/0 conn(0x7f77140cf650 legacy=0x7f771406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f771a195700 1 -- 172.21.15.158:0/311904862 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77140ce890 con 0x7f771415d990 2023-10-16T09:43:56.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f771a195700 1 --2- 172.21.15.158:0/311904862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140b2750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7704009840 tx=0x7f7704009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596859fc3e379acc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:56.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f7719193700 1 -- 172.21.15.158:0/311904862 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704011830 con 0x7f771415d990 2023-10-16T09:43:56.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f7719193700 1 -- 172.21.15.158:0/311904862 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7704011990 con 0x7f771415d990 2023-10-16T09:43:56.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f7719193700 1 -- 172.21.15.158:0/311904862 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704011d20 con 0x7f771415d990 2023-10-16T09:43:56.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f771c3f9700 1 -- 172.21.15.158:0/311904862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 msgr2=0x7f77140b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.486+0000 7f771c3f9700 1 --2- 172.21.15.158:0/311904862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140b2750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7704009840 tx=0x7f7704009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:56.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.487+0000 7f771c3f9700 1 -- 172.21.15.158:0/311904862 shutdown_connections 2023-10-16T09:43:56.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.487+0000 7f771c3f9700 1 --2- 172.21.15.158:0/311904862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140b2750 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.487+0000 7f771c3f9700 1 -- 172.21.15.158:0/311904862 >> 172.21.15.158:0/311904862 conn(0x7f771406b9e0 msgr2=0x7f771406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:56.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.487+0000 7f771c3f9700 1 -- 172.21.15.158:0/311904862 shutdown_connections 2023-10-16T09:43:56.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.487+0000 7f771c3f9700 1 -- 172.21.15.158:0/311904862 wait complete. 2023-10-16T09:43:56.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.488+0000 7f771c3f9700 1 Processor -- start 2023-10-16T09:43:56.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.488+0000 7f771c3f9700 1 -- start start 2023-10-16T09:43:56.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f771406e480 0x7f77140ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77140cf650 0x7f77140c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77140cb350 con 0x7f771415d990 2023-10-16T09:43:56.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77140cb630 con 0x7f771406e480 2023-10-16T09:43:56.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77140cb910 con 0x7f77140cf650 2023-10-16T09:43:56.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47064/0 (socket says 172.21.15.158:47064) 2023-10-16T09:43:56.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 -- 172.21.15.158:0/359129853 learned_addr learned my addr 172.21.15.158:0/359129853 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:56.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a195700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f771406e480 0x7f77140ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f7719994700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77140cf650 0x7f77140c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 -- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77140cf650 msgr2=0x7f77140c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77140cf650 0x7f77140c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 -- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f771406e480 msgr2=0x7f77140ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f771406e480 0x7f77140ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 -- 172.21.15.158:0/359129853 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77040094a0 con 0x7f771415d990 2023-10-16T09:43:56.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771a996700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140cc140 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f770c00cbc0 tx=0x7f770c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:56.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f770affd700 1 -- 172.21.15.158:0/359129853 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c00a9e0 con 0x7f771415d990 2023-10-16T09:43:56.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f770affd700 1 -- 172.21.15.158:0/359129853 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c00ab40 con 0x7f771415d990 2023-10-16T09:43:56.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77140cbbf0 con 0x7f771415d990 2023-10-16T09:43:56.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f770affd700 1 -- 172.21.15.158:0/359129853 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c012400 con 0x7f771415d990 2023-10-16T09:43:56.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.489+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77140cb100 con 0x7f771415d990 2023-10-16T09:43:56.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.491+0000 7f770affd700 1 -- 172.21.15.158:0/359129853 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f770c01b3b0 con 0x7f771415d990 2023-10-16T09:43:56.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.491+0000 7f770affd700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f76f806af00 0x7f76f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.491+0000 7f770affd700 1 -- 172.21.15.158:0/359129853 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f770c089260 con 0x7f771415d990 2023-10-16T09:43:56.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.491+0000 7f771a195700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f76f806af00 0x7f76f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.492+0000 7f771a195700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f76f806af00 0x7f76f806d3c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f77040090b0 tx=0x7f770401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:56.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.493+0000 7f77285e3b80 1 Processor -- start 2023-10-16T09:43:56.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.493+0000 7f77285e3b80 1 -- start start 2023-10-16T09:43:56.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b084806220 0x55b08480da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b08480dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b0847e1010 0x55b0847e16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f77285e3b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084757c20 con 0x55b0847e1010 2023-10-16T09:43:56.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b08473cc10 con 0x55b08480b630 2023-10-16T09:43:56.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b08473a530 con 0x55b084806220 2023-10-16T09:43:56.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b0847e1010 0x55b0847e16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b0847e1010 0x55b0847e16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47070/0 (socket says 172.21.15.158:47070) 2023-10-16T09:43:56.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 -- 172.21.15.158:0/1786588131 learned_addr learned my addr 172.21.15.158:0/1786588131 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:56.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a195700 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b084806220 0x55b08480da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f7719994700 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b08480dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 -- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b084806220 msgr2=0x55b08480da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b084806220 0x55b08480da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 -- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 msgr2=0x55b08480dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b08480dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a996700 1 -- 172.21.15.158:0/1786588131 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0846f33a0 con 0x55b0847e1010 2023-10-16T09:43:56.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.494+0000 7f771a195700 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b084806220 0x55b08480da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:56.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f771a996700 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b0847e1010 0x55b0847e16c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f770c00afd0 tx=0x7f770c057800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979d7a4ce91d2852 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:56.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f76f7fff700 1 -- 172.21.15.158:0/1786588131 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0890d0 con 0x55b0847e1010 2023-10-16T09:43:56.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f76f7fff700 1 -- 172.21.15.158:0/1786588131 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c05bec0 con 0x55b0847e1010 2023-10-16T09:43:56.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f76f7fff700 1 -- 172.21.15.158:0/1786588131 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c060070 con 0x55b0847e1010 2023-10-16T09:43:56.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f77285e3b80 1 -- 172.21.15.158:0/1786588131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b0847e1010 msgr2=0x55b0847e16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b0847e1010 0x55b0847e16c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f770c00afd0 tx=0x7f770c057800 comp rx=0 tx=0).stop 2023-10-16T09:43:56.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f77285e3b80 1 -- 172.21.15.158:0/1786588131 shutdown_connections 2023-10-16T09:43:56.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b0847e1010 0x55b0847e16c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b084806220 0x55b08480da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1786588131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b08480dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.495+0000 7f77285e3b80 1 -- 172.21.15.158:0/1786588131 >> 172.21.15.158:0/1786588131 conn(0x55b084710d80 msgr2=0x55b08482bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:56.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.496+0000 7f77285e3b80 1 -- 172.21.15.158:0/1786588131 shutdown_connections 2023-10-16T09:43:56.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.496+0000 7f77285e3b80 1 -- 172.21.15.158:0/1786588131 wait complete. 2023-10-16T09:43:56.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.496+0000 7f77285e3b80 1 -- start start 2023-10-16T09:43:56.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.496+0000 7f77285e3b80 1 Processor -- start 2023-10-16T09:43:56.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b084809f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b084806220 0x55b084810910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b08480b630 0x55b0848374a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f77285e3b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084735e20 con 0x55b08480b630 2023-10-16T09:43:56.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0846d26c0 con 0x55b084806220 2023-10-16T09:43:56.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0846d2360 con 0x55b0847e1010 2023-10-16T09:43:56.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f77285e3b80 10 client.0 _ll_register_callbacks cb 0x55b08452f7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:43:56.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f771a996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b08480b630 0x55b0848374a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f771a996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b08480b630 0x55b0848374a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47078/0 (socket says 172.21.15.158:47078) 2023-10-16T09:43:56.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f771a996700 1 -- 172.21.15.158:0/1713743520 learned_addr learned my addr 172.21.15.158:0/1713743520 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:56.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f7719994700 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b084806220 0x55b084810910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.497+0000 7f771a195700 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b084809f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f771a996700 1 -- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 msgr2=0x55b084809f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f771a996700 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b084809f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f771a996700 1 -- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b084806220 msgr2=0x55b084810910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f771a996700 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b084806220 0x55b084810910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f771a996700 1 -- 172.21.15.158:0/1713743520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77140cb100 con 0x55b08480b630 2023-10-16T09:43:56.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f771a996700 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b08480b630 0x55b0848374a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f770c087580 tx=0x7f770c001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da45ad25bfc1ef5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:56.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f76f57fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:43:56.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f76f57fa700 1 -- 172.21.15.158:0/1713743520 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0890d0 con 0x55b08480b630 2023-10-16T09:43:56.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f76f57fa700 1 -- 172.21.15.158:0/1713743520 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f770c060070 con 0x55b08480b630 2023-10-16T09:43:56.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0847d65b0 con 0x55b08480b630 2023-10-16T09:43:56.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f76f57fa700 1 -- 172.21.15.158:0/1713743520 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f770c05fbe0 con 0x55b08480b630 2023-10-16T09:43:56.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 10 client.4830 fetch_fsmap learned FSMap version 9 2023-10-16T09:43:56.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55b084560d90 con 0x55b08480b630 2023-10-16T09:43:56.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f76f57fa700 1 -- 172.21.15.158:0/1713743520 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c07a080 con 0x55b08480b630 2023-10-16T09:43:56.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f76f57fa700 1 -- 172.21.15.158:0/1713743520 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f770c053c10 con 0x55b08480b630 2023-10-16T09:43:56.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 10 client.4830 fetch_fsmap finished waiting for FSMap version 9 2023-10-16T09:43:56.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 10 client.4830 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:43:56.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 20 client.4830 populate_metadata read hostname 'smithi158' 2023-10-16T09:43:56.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b0848047a0 0x55b084804b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 4 client.4830 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T09:43:56.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.498+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 --> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b08472ee00 con 0x55b0848047a0 2023-10-16T09:43:56.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.499+0000 7f771a195700 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b0848047a0 0x55b084804b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.499+0000 7f771a195700 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b0848047a0 0x55b084804b80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=600d46120eb321f9 server_cookie=3bc1e8cb002dec46 in_seq=0 out_seq=0 2023-10-16T09:43:56.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.499+0000 7f76f57fa700 10 client.4830 ms_handle_connect on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:56.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f76f57fa700 1 -- 172.21.15.158:0/1713743520 <== mds.0 v2:172.21.15.172:6833/3958845884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b08472ee00 con 0x55b0848047a0 2023-10-16T09:43:56.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f76f57fa700 10 client.4830 handle_command_reply: tid=0 2023-10-16T09:43:56.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 client.4830 shutdown 2023-10-16T09:43:56.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b08480b630 msgr2=0x55b0848374a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b08480b630 0x55b0848374a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f770c087580 tx=0x7f770c001250 comp rx=0 tx=0).stop 2023-10-16T09:43:56.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 shutdown_connections 2023-10-16T09:43:56.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b0848047a0 0x55b084804b80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b08480b630 0x55b0848374a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b084809f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 --2- 172.21.15.158:0/1713743520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b084806220 0x55b084810910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f76f57fa700 0 client.4830 ms_handle_reset on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:56.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 >> 172.21.15.158:0/1713743520 conn(0x55b084710d80 msgr2=0x55b0847d8640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:56.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 shutdown_connections 2023-10-16T09:43:56.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 1 -- 172.21.15.158:0/1713743520 wait complete. 2023-10-16T09:43:56.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.500+0000 7f77285e3b80 20 client.4830 trim_cache size 0 max 16384 2023-10-16T09:43:56.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.517+0000 7f77285e3b80 1 Processor -- start 2023-10-16T09:43:56.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f77285e3b80 1 -- start start 2023-10-16T09:43:56.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b0847dfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b084804fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b084836e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f77285e3b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0845b68d0 con 0x55b084806220 2023-10-16T09:43:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084804e70 con 0x55b08480b630 2023-10-16T09:43:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b08480b460 con 0x55b0847e1010 2023-10-16T09:43:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f7719994700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b084804fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f7719994700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b084804fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47084/0 (socket says 172.21.15.158:47084) 2023-10-16T09:43:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f7719994700 1 -- 172.21.15.158:0/4129540106 learned_addr learned my addr 172.21.15.158:0/4129540106 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f771a996700 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b084836e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.518+0000 7f771a195700 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b0847dfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7719994700 1 -- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 msgr2=0x55b0847dfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7719994700 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b0847dfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7719994700 1 -- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 msgr2=0x55b084836e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7719994700 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b084836e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7719994700 1 -- 172.21.15.158:0/4129540106 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b084549c10 con 0x55b084806220 2023-10-16T09:43:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f771a195700 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b0847dfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:43:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7719994700 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b084804fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7710000f60 tx=0x7f7710007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f4fc3c63e64fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7708d82700 1 -- 172.21.15.158:0/4129540106 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710005320 con 0x55b084806220 2023-10-16T09:43:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7708d82700 1 -- 172.21.15.158:0/4129540106 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771001a070 con 0x55b084806220 2023-10-16T09:43:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f7708d82700 1 -- 172.21.15.158:0/4129540106 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710003680 con 0x55b084806220 2023-10-16T09:43:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 -- 172.21.15.158:0/4129540106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 msgr2=0x55b084804fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b084804fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7710000f60 tx=0x7f7710007ec0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 -- 172.21.15.158:0/4129540106 shutdown_connections 2023-10-16T09:43:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b08480b630 0x55b084836e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b0847e1010 0x55b0847dfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 --2- 172.21.15.158:0/4129540106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b084804fd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 -- 172.21.15.158:0/4129540106 >> 172.21.15.158:0/4129540106 conn(0x55b084710d80 msgr2=0x55b0847e4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.519+0000 7f77285e3b80 1 -- 172.21.15.158:0/4129540106 shutdown_connections 2023-10-16T09:43:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.520+0000 7f77285e3b80 1 -- 172.21.15.158:0/4129540106 wait complete. 2023-10-16T09:43:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.520+0000 7f77285e3b80 1 -- start start 2023-10-16T09:43:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.520+0000 7f77285e3b80 1 Processor -- start 2023-10-16T09:43:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b0847e1010 0x55b084878960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b0848900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f77285e3b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b08480b630 0x55b0848371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f77285e3b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084890960 con 0x55b084806220 2023-10-16T09:43:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084810970 con 0x55b0847e1010 2023-10-16T09:43:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f77285e3b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0847e3400 con 0x55b08480b630 2023-10-16T09:43:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f77285e3b80 10 client.0 _ll_register_callbacks cb 0x7f770c057f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:43:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b0848900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b0848900e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47090/0 (socket says 172.21.15.158:47090) 2023-10-16T09:43:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 -- 172.21.15.158:0/2361772542 learned_addr learned my addr 172.21.15.158:0/2361772542 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f771a996700 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b08480b630 0x55b0848371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f771a195700 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b0847e1010 0x55b084878960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 -- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b08480b630 msgr2=0x55b0848371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b08480b630 0x55b0848371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 -- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b0847e1010 msgr2=0x55b084878960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b0847e1010 0x55b084878960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.521+0000 7f7719994700 1 -- 172.21.15.158:0/2361772542 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771406e9b0 con 0x55b084806220 2023-10-16T09:43:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f7719994700 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b0848900e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7710009010 tx=0x7f7710000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27523d0e733c631 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f76f57fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:43:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f76f57fa700 1 -- 172.21.15.158:0/2361772542 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710025070 con 0x55b084806220 2023-10-16T09:43:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f770c05bec0 con 0x55b084806220 2023-10-16T09:43:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f76f57fa700 1 -- 172.21.15.158:0/2361772542 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f77100039b0 con 0x55b084806220 2023-10-16T09:43:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f76f57fa700 1 -- 172.21.15.158:0/2361772542 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7710007b80 con 0x55b084806220 2023-10-16T09:43:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 10 client.4836 fetch_fsmap learned FSMap version 9 2023-10-16T09:43:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55b084587200 con 0x55b084806220 2023-10-16T09:43:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f76f57fa700 1 -- 172.21.15.158:0/2361772542 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c05bec0 con 0x55b084806220 2023-10-16T09:43:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f76f57fa700 1 -- 172.21.15.158:0/2361772542 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f7710012070 con 0x55b084806220 2023-10-16T09:43:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 10 client.4836 fetch_fsmap finished waiting for FSMap version 9 2023-10-16T09:43:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 10 client.4836 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:43:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 20 client.4836 populate_metadata read hostname 'smithi158' 2023-10-16T09:43:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b084810e50 0x55b084811230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 4 client.4836 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T09:43:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 --> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b08472eb20 con 0x55b084810e50 2023-10-16T09:43:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.522+0000 7f771a195700 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b084810e50 0x55b084811230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f76f57fa700 10 client.4836 ms_handle_connect on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f771a195700 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b084810e50 0x55b084811230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f8f4121cdbe9209 server_cookie=46c2936f85b1fbe9 in_seq=0 out_seq=0 2023-10-16T09:43:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f76f57fa700 1 -- 172.21.15.158:0/2361772542 <== mds.0 v2:172.21.15.172:6833/3958845884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55b08472eb20 con 0x55b084810e50 2023-10-16T09:43:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f76f57fa700 10 client.4836 handle_command_reply: tid=0 2023-10-16T09:43:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 client.4836 shutdown 2023-10-16T09:43:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 msgr2=0x55b0848900e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b0848900e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7710009010 tx=0x7f7710000ed0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 shutdown_connections 2023-10-16T09:43:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:6833/3958845884,v1:172.21.15.172:6835/3958845884] conn(0x55b084810e50 0x55b084811230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b08480b630 0x55b0848371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b0847e1010 0x55b084878960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 --2- 172.21.15.158:0/2361772542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b084806220 0x55b0848900e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f76f57fa700 0 client.4836 ms_handle_reset on v2:172.21.15.172:6833/3958845884 2023-10-16T09:43:56.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 >> 172.21.15.158:0/2361772542 conn(0x55b084710d80 msgr2=0x55b084803f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:56.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 shutdown_connections 2023-10-16T09:43:56.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 1 -- 172.21.15.158:0/2361772542 wait complete. 2023-10-16T09:43:56.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.523+0000 7f77285e3b80 20 client.4836 trim_cache size 0 max 16384 2023-10-16T09:43:56.545 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:43:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T09:43:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T09:43:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 4267, 2023-10-16T09:43:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T09:43:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T09:43:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T09:43:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 34.601511328999997, 2023-10-16T09:43:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 9, 2023-10-16T09:43:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 24, 2023-10-16T09:43:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 24, 2023-10-16T09:43:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 41.110308906999997 2023-10-16T09:43:56.548 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:43:56.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f76f806af00 msgr2=0x7f76f806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f76f806af00 0x7f76f806d3c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f77040090b0 tx=0x7f770401c040 comp rx=0 tx=0).stop 2023-10-16T09:43:56.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 msgr2=0x7f77140cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:56.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140cc140 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f770c00cbc0 tx=0x7f770c00ced0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 shutdown_connections 2023-10-16T09:43:56.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f771415d990 0x7f77140cc140 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f771406e480 0x7f77140ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f76f806af00 0x7f76f806d3c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 --2- 172.21.15.158:0/359129853 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77140cf650 0x7f77140c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:56.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 >> 172.21.15.158:0/359129853 conn(0x7f771406b9e0 msgr2=0x7f77140b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:56.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 shutdown_connections 2023-10-16T09:43:56.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:56.524+0000 7f771c3f9700 1 -- 172.21.15.158:0/359129853 wait complete. 2023-10-16T09:43:56.552 INFO:teuthology.run_tasks:Running task kclient... 2023-10-16T09:43:56.564 INFO:tasks.kclient:Mounting kernel clients... 2023-10-16T09:43:56.564 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-16T09:43:56.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.565 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:43:56.582 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-16T09:43:56.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.583 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T09:43:56.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.662 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link delete ceph-brx 2023-10-16T09:43:56.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.782 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:43:56.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.799 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link delete ceph-brx 2023-10-16T09:43:56.870 INFO:teuthology.orchestra.run.smithi158.stderr:Cannot find device "ceph-brx" 2023-10-16T09:43:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T09:43:56.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.872 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2023-10-16T09:43:56.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link delete ceph-brx 2023-10-16T09:43:56.966 INFO:teuthology.orchestra.run.smithi172.stderr:Cannot find device "ceph-brx" 2023-10-16T09:43:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T09:43:56.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:56.968 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2023-10-16T09:43:57.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link delete ceph-brx 2023-10-16T09:43:57.100 INFO:teuthology.orchestra.run.smithi172.stderr:Cannot find device "ceph-brx" 2023-10-16T09:43:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T09:43:57.101 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-10-16T09:43:57.102 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T09:43:57.102 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T09:43:57.103 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:43:57.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.223+0000 7f1b5d414700 1 Processor -- start 2023-10-16T09:43:57.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.223+0000 7f1b5d414700 1 -- start start 2023-10-16T09:43:57.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b5d414700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1b581599b0 0x7f1b580ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b5d414700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b58072250 con 0x7f1b5815d7b0 2023-10-16T09:43:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b5d414700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5806d370 con 0x7f1b58072430 2023-10-16T09:43:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b5d414700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5815dc90 con 0x7f1b581599b0 2023-10-16T09:43:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b567fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1b5815d7b0 0x7f1b5806c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47274/0 (socket says 172.21.15.158:47274) 2023-10-16T09:43:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b56ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1b581599b0 0x7f1b580ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b567fc700 1 -- 172.21.15.158:0/697385103 learned_addr learned my addr 172.21.15.158:0/697385103 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b56ffd700 1 -- 172.21.15.158:0/697385103 >> v1:172.21.15.172:6789/0 conn(0x7f1b58072430 legacy=0x7f1b5806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.224+0000 7f1b56ffd700 1 -- 172.21.15.158:0/697385103 >> v1:172.21.15.158:6789/0 conn(0x7f1b5815d7b0 legacy=0x7f1b5806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b56ffd700 1 -- 172.21.15.158:0/697385103 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5815df70 con 0x7f1b581599b0 2023-10-16T09:43:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b55ffb700 1 -- 172.21.15.158:0/697385103 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318433771 0 0) 0x7f1b58072250 con 0x7f1b5815d7b0 2023-10-16T09:43:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b56ffd700 1 --2- 172.21.15.158:0/697385103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b580ca0c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009840 tx=0x7f1b4c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5675be753e808dc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b55ffb700 1 -- 172.21.15.158:0/697385103 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c004230 con 0x7f1b581599b0 2023-10-16T09:43:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b55ffb700 1 -- 172.21.15.158:0/697385103 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c004390 con 0x7f1b581599b0 2023-10-16T09:43:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b55ffb700 1 -- 172.21.15.158:0/697385103 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0036a0 con 0x7f1b581599b0 2023-10-16T09:43:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b5d414700 1 -- 172.21.15.158:0/697385103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 msgr2=0x7f1b580ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.225+0000 7f1b5d414700 1 --2- 172.21.15.158:0/697385103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b580ca0c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009840 tx=0x7f1b4c009b50 comp rx=0 tx=0).stop 2023-10-16T09:43:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.226+0000 7f1b5d414700 1 -- 172.21.15.158:0/697385103 shutdown_connections 2023-10-16T09:43:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.226+0000 7f1b5d414700 1 --2- 172.21.15.158:0/697385103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b580ca0c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.226+0000 7f1b5d414700 1 -- 172.21.15.158:0/697385103 >> 172.21.15.158:0/697385103 conn(0x7f1b5806af00 msgr2=0x7f1b580cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.226+0000 7f1b5d414700 1 -- 172.21.15.158:0/697385103 shutdown_connections 2023-10-16T09:43:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.226+0000 7f1b5d414700 1 -- 172.21.15.158:0/697385103 wait complete. 2023-10-16T09:43:57.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.227+0000 7f1b5d414700 1 Processor -- start 2023-10-16T09:43:57.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.227+0000 7f1b5d414700 1 -- start start 2023-10-16T09:43:57.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58072430 0x7f1b5814ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:57.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b581f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:57.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b5815d7b0 0x7f1b581f7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5816ac90 con 0x7f1b581599b0 2023-10-16T09:43:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5816a110 con 0x7f1b58072430 2023-10-16T09:43:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5816a3f0 con 0x7f1b5815d7b0 2023-10-16T09:43:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b581f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b581f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47104/0 (socket says 172.21.15.158:47104) 2023-10-16T09:43:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 -- 172.21.15.158:0/289207832 learned_addr learned my addr 172.21.15.158:0/289207832 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b56ffd700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58072430 0x7f1b5814ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b577fe700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b5815d7b0 0x7f1b581f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 -- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b5815d7b0 msgr2=0x7f1b581f7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b5815d7b0 0x7f1b581f7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 -- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58072430 msgr2=0x7f1b5814ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:57.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58072430 0x7f1b5814ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 -- 172.21.15.158:0/289207832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c0094a0 con 0x7f1b581599b0 2023-10-16T09:43:57.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b567fc700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b581f6bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800c520 tx=0x7f1b4800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:57.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b3f7fe700 1 -- 172.21.15.158:0/289207832 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48004c80 con 0x7f1b581599b0 2023-10-16T09:43:57.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b3f7fe700 1 -- 172.21.15.158:0/289207832 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b48004e00 con 0x7f1b581599b0 2023-10-16T09:43:57.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5816a6d0 con 0x7f1b581599b0 2023-10-16T09:43:57.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b3f7fe700 1 -- 172.21.15.158:0/289207832 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48007310 con 0x7f1b581599b0 2023-10-16T09:43:57.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.228+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5814efb0 con 0x7f1b581599b0 2023-10-16T09:43:57.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.229+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b580ac920 con 0x7f1b581599b0 2023-10-16T09:43:57.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.230+0000 7f1b3f7fe700 1 -- 172.21.15.158:0/289207832 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1b48007470 con 0x7f1b581599b0 2023-10-16T09:43:57.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.230+0000 7f1b3f7fe700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b4406af00 0x7f1b4406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:57.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.230+0000 7f1b3f7fe700 1 -- 172.21.15.158:0/289207832 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1b48089620 con 0x7f1b581599b0 2023-10-16T09:43:57.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.230+0000 7f1b56ffd700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b4406af00 0x7f1b4406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:57.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.231+0000 7f1b56ffd700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b4406af00 0x7f1b4406d3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c000c00 tx=0x7f1b4c01c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:57.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.235+0000 7f1b3f7fe700 1 -- 172.21.15.158:0/289207832 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b580ac920 con 0x7f1b581599b0 2023-10-16T09:43:57.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.473+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1b580ac920 con 0x7f1b581599b0 2023-10-16T09:43:57.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.473+0000 7f1b3f7fe700 1 -- 172.21.15.158:0/289207832 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1b580ac920 con 0x7f1b581599b0 2023-10-16T09:43:57.475 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T09:43:57.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b4406af00 msgr2=0x7f1b4406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:57.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b4406af00 0x7f1b4406d3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c000c00 tx=0x7f1b4c01c040 comp rx=0 tx=0).stop 2023-10-16T09:43:57.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 msgr2=0x7f1b581f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:57.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b581f6bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800c520 tx=0x7f1b4800c8e0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 shutdown_connections 2023-10-16T09:43:57.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1b4406af00 0x7f1b4406d3c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b5815d7b0 0x7f1b581f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b58072430 0x7f1b5814ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 --2- 172.21.15.158:0/289207832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b581599b0 0x7f1b581f6bf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:57.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 >> 172.21.15.158:0/289207832 conn(0x7f1b5806af00 msgr2=0x7f1b5814d170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:57.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 shutdown_connections 2023-10-16T09:43:57.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:57.476+0000 7f1b5d414700 1 -- 172.21.15.158:0/289207832 wait complete. 2023-10-16T09:43:57.489 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T09:43:57.490 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:43:57.490 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T09:43:57.490 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T09:43:57.490 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T09:43:57.491 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T09:43:57.491 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T09:43:57.491 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:43:57.491 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:43:57.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.492 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T09:43:57.509 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:43:57.509 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:43:57.509 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:43:57.509 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:57.509 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:43:57.509 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:57.510 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:57.510 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:57.510 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:57.510 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:57.510 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:57.511 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:57.511 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:43:57.511 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:57.511 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:43:57.511 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 42396sec preferred_lft 42396sec 2023-10-16T09:43:57.511 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T09:43:57.512 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:57.512 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T09:43:57.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.513 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:57.513 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:57.513 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add name ceph-brx type bridge 2023-10-16T09:43:57.513 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip addr flush dev ceph-brx 2023-10-16T09:43:57.514 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set ceph-brx up 2023-10-16T09:43:57.514 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T09:43:57.514 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:57.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.626 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:43:57.651 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T09:43:57.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.653 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T09:43:57.712 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T09:43:57.713 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:43:57.713 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:43:57.713 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:43:57.713 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:43:57.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.714 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:57.714 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:57.715 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:43:57.715 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:43:57.715 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:43:57.715 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:57.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.817 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:43:57.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.834 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list-id 2023-10-16T09:43:57.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.890 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:57.891 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:57.891 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T09:43:57.891 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-16T09:43:57.891 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:57.985 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-16T09:43:57.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:57.986 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:57.986 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:57.987 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-16T09:43:57.987 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T09:43:57.987 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-16T09:43:57.988 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-16T09:43:57.988 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-16T09:43:57.988 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:58.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:58.196 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:58.197 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:58.197 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T09:43:58.197 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T09:43:58.198 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:58.315 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:43:58.316 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T09:43:58.334 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:43:58.335 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:43:58.336 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:43:58.390 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T09:43:58.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:58.391 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:43:58.476 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:43:58.476 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:43:58.594 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:43:58.595 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:43:58.595 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:43:58.595 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:43:58.595 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:43:58.596 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:43:58.596 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:43:58.596 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:43:58.628 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:43:58.628 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:43:58.628 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:43:58.629 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:43:58.629 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:43:58.629 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:43:58.629 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:43:58.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:43:58.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T09:43:58.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T09:43:58.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T09:43:58.640 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T09:43:58.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:58.641 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:43:58.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:58.671 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:43:58.743 INFO:teuthology.orchestra.run.smithi158.stdout:client4738 2023-10-16T09:43:58.744 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:43:58.744 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:43:58.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:58.801 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:43:58.873 INFO:teuthology.orchestra.run.smithi158.stdout:client4738 2023-10-16T09:43:58.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:58.874 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:43:58.946 INFO:teuthology.orchestra.run.smithi158.stdout:client4738 2023-10-16T09:43:58.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:58.947 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7cf363df-9988-44b1-be05-df2a6bf24f2d.client4738/status') 2023-10-16T09:43:59.026 INFO:teuthology.orchestra.run.smithi158.stdout:instance: client.4738 (3)172.21.15.158:0/2575797982 2023-10-16T09:43:59.026 INFO:teuthology.orchestra.run.smithi158.stdout:blocklisted: false 2023-10-16T09:43:59.027 INFO:teuthology.orchestra.run.smithi158.stderr:0+1 records in 2023-10-16T09:43:59.027 INFO:teuthology.orchestra.run.smithi158.stderr:0+1 records out 2023-10-16T09:43:59.028 INFO:teuthology.orchestra.run.smithi158.stderr:71 bytes copied, 2.3299e-05 s, 3.0 MB/s 2023-10-16T09:43:59.028 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2023-10-16T09:43:59.028 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T09:43:59.028 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T09:43:59.029 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:43:59.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.186+0000 7fb248a50700 1 Processor -- start 2023-10-16T09:43:59.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.186+0000 7fb248a50700 1 -- start start 2023-10-16T09:43:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb248a50700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb24415d7c0 0x7fb2440d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb248a50700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb2440d1210 0x7fb2440ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb248a50700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2440d3bd0 con 0x7fb2440ce950 2023-10-16T09:43:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb248a50700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2440d4060 con 0x7fb2440d1210 2023-10-16T09:43:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb248a50700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2440d2cf0 con 0x7fb24415d7c0 2023-10-16T09:43:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb241d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb2440d1210 0x7fb2440ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb242d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb2440ce950 0x7fb2440b50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47290/0 (socket says 172.21.15.158:47290) 2023-10-16T09:43:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb242d9d700 1 -- 172.21.15.158:0/1047093997 learned_addr learned my addr 172.21.15.158:0/1047093997 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:43:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb241d9b700 1 --2- 172.21.15.158:0/1047093997 >> v2:172.21.15.158:3300/0 conn(0x7fb2440d1210 0x7fb2440ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47114/0 (socket says 172.21.15.158:47114) 2023-10-16T09:43:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.187+0000 7fb24259c700 1 --2- 172.21.15.158:0/1047093997 >> v2:172.21.15.172:3300/0 conn(0x7fb24415d7c0 0x7fb2440d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb241d9b700 1 -- 172.21.15.158:0/1047093997 >> v2:172.21.15.172:3300/0 conn(0x7fb24415d7c0 msgr2=0x7fb2440d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:43:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb241d9b700 1 --2- 172.21.15.158:0/1047093997 >> v2:172.21.15.172:3300/0 conn(0x7fb24415d7c0 0x7fb2440d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb241d9b700 1 -- 172.21.15.158:0/1047093997 >> v1:172.21.15.158:6789/0 conn(0x7fb2440ce950 legacy=0x7fb2440b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb24159a700 1 -- 172.21.15.158:0/1047093997 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635825440 0 0) 0x7fb2440d3bd0 con 0x7fb2440ce950 2023-10-16T09:43:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb241d9b700 1 -- 172.21.15.158:0/1047093997 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440d2fc0 con 0x7fb2440d1210 2023-10-16T09:43:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb241d9b700 1 --2- 172.21.15.158:0/1047093997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440d1210 0x7fb2440ce410 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb2440d0bc0 tx=0x7fb234009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66882fd354164ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb24159a700 1 -- 172.21.15.158:0/1047093997 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234011830 con 0x7fb2440d1210 2023-10-16T09:43:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb24159a700 1 -- 172.21.15.158:0/1047093997 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb234011990 con 0x7fb2440d1210 2023-10-16T09:43:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb24159a700 1 -- 172.21.15.158:0/1047093997 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234011d20 con 0x7fb2440d1210 2023-10-16T09:43:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb248a50700 1 -- 172.21.15.158:0/1047093997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440d1210 msgr2=0x7fb2440ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb248a50700 1 --2- 172.21.15.158:0/1047093997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440d1210 0x7fb2440ce410 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb2440d0bc0 tx=0x7fb234009b20 comp rx=0 tx=0).stop 2023-10-16T09:43:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb248a50700 1 -- 172.21.15.158:0/1047093997 shutdown_connections 2023-10-16T09:43:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb248a50700 1 --2- 172.21.15.158:0/1047093997 >> v2:172.21.15.172:3300/0 conn(0x7fb24415d7c0 0x7fb2440d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb248a50700 1 --2- 172.21.15.158:0/1047093997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440d1210 0x7fb2440ce410 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb248a50700 1 -- 172.21.15.158:0/1047093997 >> 172.21.15.158:0/1047093997 conn(0x7fb2440669f0 msgr2=0x7fb244066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:59.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.188+0000 7fb248a50700 1 -- 172.21.15.158:0/1047093997 shutdown_connections 2023-10-16T09:43:59.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.189+0000 7fb248a50700 1 -- 172.21.15.158:0/1047093997 wait complete. 2023-10-16T09:43:59.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.190+0000 7fb248a50700 1 Processor -- start 2023-10-16T09:43:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.190+0000 7fb248a50700 1 -- start start 2023-10-16T09:43:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440ce950 0x7fb2440c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2440d1210 0x7fb2441511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb24415d7c0 0x7fb2440b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440c9070 con 0x7fb2440ce950 2023-10-16T09:43:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb244151f40 con 0x7fb24415d7c0 2023-10-16T09:43:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440c81a0 con 0x7fb2440d1210 2023-10-16T09:43:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440ce950 0x7fb2440c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440ce950 0x7fb2440c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47116/0 (socket says 172.21.15.158:47116) 2023-10-16T09:43:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 -- 172.21.15.158:0/3136312920 learned_addr learned my addr 172.21.15.158:0/3136312920 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:43:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb241d9b700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2440d1210 0x7fb2441511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb242d9d700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb24415d7c0 0x7fb2440b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 -- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2440d1210 msgr2=0x7fb2441511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2440d1210 0x7fb2441511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 -- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb24415d7c0 msgr2=0x7fb2440b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb24415d7c0 0x7fb2440b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 -- 172.21.15.158:0/3136312920 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340094a0 con 0x7fb2440ce950 2023-10-16T09:43:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb24259c700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440ce950 0x7fb2440c9cd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb23800e810 tx=0x7fb23800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb232ffd700 1 -- 172.21.15.158:0/3136312920 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238013610 con 0x7fb2440ce950 2023-10-16T09:43:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb232ffd700 1 -- 172.21.15.158:0/3136312920 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb238013770 con 0x7fb2440ce950 2023-10-16T09:43:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb232ffd700 1 -- 172.21.15.158:0/3136312920 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238013b20 con 0x7fb2440ce950 2023-10-16T09:43:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2440ab360 con 0x7fb2440ce950 2023-10-16T09:43:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.191+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2440c86d0 con 0x7fb2440ce950 2023-10-16T09:43:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.192+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2440681b0 con 0x7fb2440ce950 2023-10-16T09:43:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.193+0000 7fb232ffd700 1 -- 172.21.15.158:0/3136312920 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb238013c80 con 0x7fb2440ce950 2023-10-16T09:43:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.193+0000 7fb232ffd700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb22406af00 0x7fb22406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:43:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.193+0000 7fb232ffd700 1 -- 172.21.15.158:0/3136312920 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb238089570 con 0x7fb2440ce950 2023-10-16T09:43:59.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.193+0000 7fb241d9b700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb22406af00 0x7fb22406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:43:59.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.193+0000 7fb241d9b700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb22406af00 0x7fb22406d3c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb2340090b0 tx=0x7fb234008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:43:59.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.197+0000 7fb232ffd700 1 -- 172.21.15.158:0/3136312920 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb238054340 con 0x7fb2440ce950 2023-10-16T09:43:59.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.437+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb2440d22a0 con 0x7fb2440ce950 2023-10-16T09:43:59.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.437+0000 7fb232ffd700 1 -- 172.21.15.158:0/3136312920 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb238057960 con 0x7fb2440ce950 2023-10-16T09:43:59.439 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T09:43:59.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb22406af00 msgr2=0x7fb22406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb22406af00 0x7fb22406d3c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb2340090b0 tx=0x7fb234008030 comp rx=0 tx=0).stop 2023-10-16T09:43:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440ce950 msgr2=0x7fb2440c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:43:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440ce950 0x7fb2440c9cd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb23800e810 tx=0x7fb23800ebd0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 shutdown_connections 2023-10-16T09:43:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb24415d7c0 0x7fb2440b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2440ce950 0x7fb2440c9cd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb22406af00 0x7fb22406d3c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 --2- 172.21.15.158:0/3136312920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2440d1210 0x7fb2441511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:43:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 >> 172.21.15.158:0/3136312920 conn(0x7fb2440669f0 msgr2=0x7fb2440b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:43:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 shutdown_connections 2023-10-16T09:43:59.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:43:59.440+0000 7fb248a50700 1 -- 172.21.15.158:0/3136312920 wait complete. 2023-10-16T09:43:59.453 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T09:43:59.453 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:43:59.454 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T09:43:59.454 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-16T09:43:59.454 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T09:43:59.454 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T09:43:59.455 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T09:43:59.455 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:43:59.455 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-16T09:43:59.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.456 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T09:43:59.472 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:43:59.473 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:43:59.473 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:43:59.473 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.473 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:43:59.473 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.473 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:59.474 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:59.474 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:59.474 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:59.474 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:43:59.475 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:59.475 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:43:59.475 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:59.475 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:43:59.475 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 42394sec preferred_lft 42394sec 2023-10-16T09:43:59.475 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T09:43:59.476 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.476 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:43:59.476 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T09:43:59.476 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T09:43:59.476 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.476 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T09:43:59.477 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.477 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:43:59.477 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T09:43:59.477 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link tentative 2023-10-16T09:43:59.477 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.478 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:43:59.537 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T09:43:59.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.539 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T09:43:59.597 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T09:43:59.597 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:43:59.597 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:43:59.598 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:43:59.598 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:43:59.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.599 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:59.599 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:59.599 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:43:59.599 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:43:59.600 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:43:59.600 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:59.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.702 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:43:59.719 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T09:43:59.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.720 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list-id 2023-10-16T09:43:59.775 INFO:teuthology.orchestra.run.smithi158.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-16T09:43:59.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.777 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:43:59.777 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:43:59.777 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T09:43:59.778 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-16T09:43:59.778 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:43:59.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.869 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T09:43:59.909 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:43:59.909 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:43:59.910 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:43:59.910 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.910 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:43:59.910 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.910 INFO:teuthology.orchestra.run.smithi158.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:43:59.911 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 66:be:62:4d:61:3d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T09:43:59.911 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T09:43:59.911 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.911 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::64be:62ff:fe4d:613d/64 scope link 2023-10-16T09:43:59.911 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:43:59.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:43:59.922 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T09:44:00.003 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:44:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:44:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:44:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:44:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:00.005 INFO:teuthology.orchestra.run.smithi158.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:44:00.005 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 66:be:62:4d:61:3d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T09:44:00.005 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T09:44:00.005 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:00.006 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::64be:62ff:fe4d:613d/64 scope link 2023-10-16T09:44:00.006 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:00.024 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-16T09:44:00.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.025 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:44:00.025 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:44:00.025 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-16T09:44:00.026 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T09:44:00.026 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-16T09:44:00.026 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-16T09:44:00.027 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-16T09:44:00.027 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:44:00.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.275 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:44:00.275 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:44:00.275 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.1 up 2023-10-16T09:44:00.275 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T09:44:00.276 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:44:00.383 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:00.383 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-16T09:44:00.420 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-16T09:44:00.421 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:00.421 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-16T09:44:00.477 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2023-10-16T09:44:00.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.478 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:00.563 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:00.563 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:00.590 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:00.591 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:00.591 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:00.591 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 2@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:00.591 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:00.592 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:00.592 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:00.592 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:00.619 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:00.619 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:00.619 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:00.619 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 2@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:00.619 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:00.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:00.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:00.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:00.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T09:44:00.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T09:44:00.621 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,key=2,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T09:44:00.622 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T09:44:00.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.622 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-16T09:44:00.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.652 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:00.724 INFO:teuthology.orchestra.run.smithi158.stdout:client4857 2023-10-16T09:44:00.725 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:00.726 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:00.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.782 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:00.854 INFO:teuthology.orchestra.run.smithi158.stdout:client4857 2023-10-16T09:44:00.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.855 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:00.926 INFO:teuthology.orchestra.run.smithi158.stdout:client4857 2023-10-16T09:44:00.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:00.927 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7cf363df-9988-44b1-be05-df2a6bf24f2d.client4857/status') 2023-10-16T09:44:01.006 INFO:teuthology.orchestra.run.smithi158.stdout:instance: client.4857 (3)192.168.0.2:0/2137867621 2023-10-16T09:44:01.006 INFO:teuthology.orchestra.run.smithi158.stdout:blocklisted: false 2023-10-16T09:44:01.007 INFO:teuthology.orchestra.run.smithi158.stderr:0+1 records in 2023-10-16T09:44:01.008 INFO:teuthology.orchestra.run.smithi158.stderr:0+1 records out 2023-10-16T09:44:01.008 INFO:teuthology.orchestra.run.smithi158.stderr:69 bytes copied, 4.621e-05 s, 1.5 MB/s 2023-10-16T09:44:01.008 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2023-10-16T09:44:01.008 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T09:44:01.009 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T09:44:01.009 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:44:01.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.165+0000 7f1c0ccae700 1 Processor -- start 2023-10-16T09:44:01.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.165+0000 7f1c0ccae700 1 -- start start 2023-10-16T09:44:01.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.166+0000 7f1c0ccae700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c0815e6f0 0x7f1c080cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.166+0000 7f1c0ccae700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0815c5d0 con 0x7f1c080b6e10 2023-10-16T09:44:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.166+0000 7f1c0ccae700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0815c8b0 con 0x7f1c080cdee0 2023-10-16T09:44:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.166+0000 7f1c0ccae700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0815d2c0 con 0x7f1c0815e6f0 2023-10-16T09:44:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.166+0000 7f1c05d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1c080b6e10 0x7f1c081521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47306/0 (socket says 172.21.15.158:47306) 2023-10-16T09:44:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.166+0000 7f1c05d9b700 1 -- 172.21.15.158:0/1592755802 learned_addr learned my addr 172.21.15.158:0/1592755802 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c06d9d700 1 --2- 172.21.15.158:0/1592755802 >> v2:172.21.15.172:3300/0 conn(0x7f1c0815e6f0 0x7f1c080cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243301683 0 0) 0x7f1c0815c5d0 con 0x7f1c080b6e10 2023-10-16T09:44:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf00035b0 con 0x7f1c080b6e10 2023-10-16T09:44:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169234900 0 0) 0x7f1bf00035b0 con 0x7f1c080b6e10 2023-10-16T09:44:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 >> v2:172.21.15.172:3300/0 conn(0x7f1c0815e6f0 msgr2=0x7f1c080cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 --2- 172.21.15.158:0/1592755802 >> v2:172.21.15.172:3300/0 conn(0x7f1c0815e6f0 0x7f1c080cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 >> v1:172.21.15.172:6789/0 conn(0x7f1c080cdee0 legacy=0x7f1c0815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0815cb90 con 0x7f1c080b6e10 2023-10-16T09:44:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1bf8001670 con 0x7f1c080b6e10 2023-10-16T09:44:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 >> v1:172.21.15.158:6789/0 conn(0x7f1c080b6e10 legacy=0x7f1c081521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1bf0003a00 0x7f1bf0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0004850 0x7f1bf0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf0005210 0x7f1bf000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0815d2c0 con 0x7f1bf0003a00 2023-10-16T09:44:01.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf000c1b0 con 0x7f1bf0004850 2023-10-16T09:44:01.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf000c370 con 0x7f1bf0005210 2023-10-16T09:44:01.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0659c700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1bf0003a00 0x7f1bf0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c05d9b700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0004850 0x7f1bf0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c06d9d700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf0005210 0x7f1bf000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0659c700 1 -- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf0005210 msgr2=0x7f1bf000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0659c700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf0005210 0x7f1bf000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0659c700 1 -- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0004850 msgr2=0x7f1bf0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0659c700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0004850 0x7f1bf0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c0659c700 1 -- 172.21.15.158:0/1592755802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf00035b0 con 0x7f1bf0003a00 2023-10-16T09:44:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.167+0000 7f1c06d9d700 1 -- 172.21.15.158:0/1592755802 reap_dead start 2023-10-16T09:44:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0659c700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1bf0003a00 0x7f1bf0003e20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0017c0 tx=0x7f1bf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840ceed2c9f51357 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0052d0 con 0x7f1bf0003a00 2023-10-16T09:44:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc005450 con 0x7f1bf0003a00 2023-10-16T09:44:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0559a700 1 -- 172.21.15.158:0/1592755802 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0038c0 con 0x7f1bf0003a00 2023-10-16T09:44:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1592755802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1bf0003a00 msgr2=0x7f1bf0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1bf0003a00 0x7f1bf0003e20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0017c0 tx=0x7f1bf0005880 comp rx=0 tx=0).stop 2023-10-16T09:44:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1592755802 shutdown_connections 2023-10-16T09:44:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1592755802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1bf0003a00 0x7f1bf0003e20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1592755802 >> 172.21.15.158:0/1592755802 conn(0x7f1c0806a8d0 msgr2=0x7f1c0806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1592755802 shutdown_connections 2023-10-16T09:44:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.168+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1592755802 wait complete. 2023-10-16T09:44:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.170+0000 7f1c0ccae700 1 Processor -- start 2023-10-16T09:44:01.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.170+0000 7f1c0ccae700 1 -- start start 2023-10-16T09:44:01.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.170+0000 7f1c0ccae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0003a00 0x7f1c08148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.170+0000 7f1c0ccae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c080cb350 0x7f1c080c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.170+0000 7f1c0ccae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c080c70d0 0x7f1c080c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.170+0000 7f1c0ccae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080c7ac0 con 0x7f1c080c70d0 2023-10-16T09:44:01.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c0ccae700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080c93f0 con 0x7f1bf0003a00 2023-10-16T09:44:01.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c0ccae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080cb7d0 con 0x7f1c080cb350 2023-10-16T09:44:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c080c70d0 0x7f1c080c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c080c70d0 0x7f1c080c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47144/0 (socket says 172.21.15.158:47144) 2023-10-16T09:44:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 -- 172.21.15.158:0/1367002210 learned_addr learned my addr 172.21.15.158:0/1367002210 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:01.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c0659c700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0003a00 0x7f1c08148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c05d9b700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c080cb350 0x7f1c080c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 -- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c080cb350 msgr2=0x7f1c080c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:01.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c080cb350 0x7f1c080c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 -- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0003a00 msgr2=0x7f1c08148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:01.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0003a00 0x7f1c08148a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 -- 172.21.15.158:0/1367002210 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080cc2f0 con 0x7f1c080c70d0 2023-10-16T09:44:01.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c05d9b700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c080cb350 0x7f1c080c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:01.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c06d9d700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c080c70d0 0x7f1c080c7580 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1c080715e0 tx=0x7f1c000111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:01.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1bf6ffd700 1 -- 172.21.15.158:0/1367002210 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000e070 con 0x7f1c080c70d0 2023-10-16T09:44:01.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1bf6ffd700 1 -- 172.21.15.158:0/1367002210 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00003cf0 con 0x7f1c080c70d0 2023-10-16T09:44:01.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c080d0590 con 0x7f1c080c70d0 2023-10-16T09:44:01.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1bf6ffd700 1 -- 172.21.15.158:0/1367002210 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00003ea0 con 0x7f1c080c70d0 2023-10-16T09:44:01.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.171+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c080cbd30 con 0x7f1c080c70d0 2023-10-16T09:44:01.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.172+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c080681b0 con 0x7f1c080c70d0 2023-10-16T09:44:01.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.173+0000 7f1bf6ffd700 1 -- 172.21.15.158:0/1367002210 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c000117f0 con 0x7f1c080c70d0 2023-10-16T09:44:01.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.173+0000 7f1bf6ffd700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1be806af00 0x7f1be806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:01.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.173+0000 7f1bf6ffd700 1 -- 172.21.15.158:0/1367002210 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1c00089d00 con 0x7f1c080c70d0 2023-10-16T09:44:01.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.173+0000 7f1c0659c700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1be806af00 0x7f1be806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:01.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.173+0000 7f1c0659c700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1be806af00 0x7f1be806d3c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:01.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.177+0000 7f1bf6ffd700 1 -- 172.21.15.158:0/1367002210 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c00054b50 con 0x7f1c080c70d0 2023-10-16T09:44:01.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.414+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1c080d0720 con 0x7f1c080c70d0 2023-10-16T09:44:01.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.415+0000 7f1bf6ffd700 1 -- 172.21.15.158:0/1367002210 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1c00058170 con 0x7f1c080c70d0 2023-10-16T09:44:01.417 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T09:44:01.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1be806af00 msgr2=0x7f1be806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:01.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1be806af00 0x7f1be806d3c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc009030 comp rx=0 tx=0).stop 2023-10-16T09:44:01.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c080c70d0 msgr2=0x7f1c080c7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:01.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c080c70d0 0x7f1c080c7580 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1c080715e0 tx=0x7f1c000111d0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 shutdown_connections 2023-10-16T09:44:01.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c080c70d0 0x7f1c080c7580 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf0003a00 0x7f1c08148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1be806af00 0x7f1be806d3c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 --2- 172.21.15.158:0/1367002210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c080cb350 0x7f1c080c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:01.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 >> 172.21.15.158:0/1367002210 conn(0x7f1c0806a8d0 msgr2=0x7f1c08154670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:01.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 shutdown_connections 2023-10-16T09:44:01.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:01.417+0000 7f1c0ccae700 1 -- 172.21.15.158:0/1367002210 wait complete. 2023-10-16T09:44:01.431 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T09:44:01.431 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:44:01.431 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi172.front.sepia.ceph.com 2023-10-16T09:44:01.432 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-16T09:44:01.432 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T09:44:01.432 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T09:44:01.432 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T09:44:01.433 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:44:01.433 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-16T09:44:01.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:01.433 DEBUG:teuthology.orchestra.run.smithi172:> ip addr 2023-10-16T09:44:01.454 INFO:teuthology.orchestra.run.smithi172.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:44:01.455 INFO:teuthology.orchestra.run.smithi172.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:44:01.455 INFO:teuthology.orchestra.run.smithi172.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:44:01.455 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:01.455 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 ::1/128 scope host 2023-10-16T09:44:01.456 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:01.456 INFO:teuthology.orchestra.run.smithi172.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:01.456 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:d9:17:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:01.456 INFO:teuthology.orchestra.run.smithi172.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:01.456 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:d9:17:2d brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:01.457 INFO:teuthology.orchestra.run.smithi172.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:01.457 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:88:7f:90 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:01.457 INFO:teuthology.orchestra.run.smithi172.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:44:01.457 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:88:7f:91 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:01.457 INFO:teuthology.orchestra.run.smithi172.stdout: inet 172.21.15.172/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:44:01.458 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft 42365sec preferred_lft 42365sec 2023-10-16T09:44:01.458 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 fe80::382f:e61a:4284:c04d/64 scope link noprefixroute 2023-10-16T09:44:01.458 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:01.458 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T09:44:01.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:01.459 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:01.460 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:01.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link add name ceph-brx type bridge 2023-10-16T09:44:01.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip addr flush dev ceph-brx 2023-10-16T09:44:01.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set ceph-brx up 2023-10-16T09:44:01.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T09:44:01.461 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:01.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:01.632 DEBUG:teuthology.orchestra.run.smithi172:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:44:01.660 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-10-16T09:44:01.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:01.662 DEBUG:teuthology.orchestra.run.smithi172:> route 2023-10-16T09:44:01.749 INFO:teuthology.orchestra.run.smithi172.stdout:Kernel IP routing table 2023-10-16T09:44:01.749 INFO:teuthology.orchestra.run.smithi172.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:44:01.750 INFO:teuthology.orchestra.run.smithi172.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:44:01.750 INFO:teuthology.orchestra.run.smithi172.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:44:01.750 INFO:teuthology.orchestra.run.smithi172.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:44:01.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:01.751 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:01.752 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:01.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:44:01.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:44:01.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:44:01.753 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:01.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:01.944 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2023-10-16T09:44:01.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:01.962 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list-id 2023-10-16T09:44:02.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.019 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:02.019 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:02.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T09:44:02.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-16T09:44:02.020 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:02.113 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-16T09:44:02.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.114 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:02.114 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:02.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-16T09:44:02.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T09:44:02.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-16T09:44:02.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-16T09:44:02.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-16T09:44:02.116 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:02.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.319 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:02.320 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:02.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set brx.0 up 2023-10-16T09:44:02.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T09:44:02.321 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:02.420 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:02.420 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-16T09:44:02.440 INFO:teuthology.orchestra.run.smithi172.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-16T09:44:02.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:02.463 DEBUG:teuthology.orchestra.run.smithi172:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-16T09:44:02.497 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-10-16T09:44:02.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:02.588 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:02.588 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:02.702 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:02.703 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:02.703 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:02.703 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: trying mount with new device syntax: 1@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:02.703 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:02.704 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:02.704 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:02.704 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:02.739 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:02.739 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:02.740 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:02.740 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: trying mount with new device syntax: 1@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:02.740 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:02.740 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:02.740 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:02.741 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:02.741 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T09:44:02.741 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T09:44:02.741 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T09:44:02.743 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T09:44:02.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-16T09:44:02.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.774 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:02.845 INFO:teuthology.orchestra.run.smithi172.stdout:client34496 2023-10-16T09:44:02.846 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:02.846 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:02.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.903 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:02.974 INFO:teuthology.orchestra.run.smithi172.stdout:client34496 2023-10-16T09:44:02.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:02.975 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:03.046 INFO:teuthology.orchestra.run.smithi172.stdout:client34496 2023-10-16T09:44:03.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.047 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7cf363df-9988-44b1-be05-df2a6bf24f2d.client34496/status') 2023-10-16T09:44:03.126 INFO:teuthology.orchestra.run.smithi172.stdout:instance: client.34496 (3)192.168.0.1:0/3149689369 2023-10-16T09:44:03.127 INFO:teuthology.orchestra.run.smithi172.stdout:blocklisted: false 2023-10-16T09:44:03.127 INFO:teuthology.orchestra.run.smithi172.stderr:0+1 records in 2023-10-16T09:44:03.128 INFO:teuthology.orchestra.run.smithi172.stderr:0+1 records out 2023-10-16T09:44:03.128 INFO:teuthology.orchestra.run.smithi172.stderr:70 bytes copied, 2.4167e-05 s, 2.9 MB/s 2023-10-16T09:44:03.128 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2023-10-16T09:44:03.128 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T09:44:03.129 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T09:44:03.129 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:44:03.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.261+0000 7f076375e700 1 Processor -- start 2023-10-16T09:44:03.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.261+0000 7f076375e700 1 -- start start 2023-10-16T09:44:03.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.262+0000 7f076375e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f075c15dfb0 0x7f075c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:03.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.262+0000 7f076375e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c06e770 con 0x7f075c06f340 2023-10-16T09:44:03.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.262+0000 7f076375e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c0cbe00 con 0x7f075c0cc5d0 2023-10-16T09:44:03.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.262+0000 7f076375e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075c0cf270 con 0x7f075c15dfb0 2023-10-16T09:44:03.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.262+0000 7f0761cfb700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f075c06f340 0x7f075c071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47328/0 (socket says 172.21.15.158:47328) 2023-10-16T09:44:03.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.262+0000 7f0761cfb700 1 -- 172.21.15.158:0/1256340787 learned_addr learned my addr 172.21.15.158:0/1256340787 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:03.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.262+0000 7f0760cf9700 1 --2- 172.21.15.158:0/1256340787 >> v2:172.21.15.172:3300/0 conn(0x7f075c15dfb0 0x7f075c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:03.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454215873 0 0) 0x7f0754001490 con 0x7f075c06f340 2023-10-16T09:44:03.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07480035b0 con 0x7f075c06f340 2023-10-16T09:44:03.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543119045 0 0) 0x7f075c0cbe00 con 0x7f075c0cc5d0 2023-10-16T09:44:03.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0754001490 con 0x7f075c0cc5d0 2023-10-16T09:44:03.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 323592003 0 0) 0x7f07480035b0 con 0x7f075c06f340 2023-10-16T09:44:03.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0760cf9700 1 -- 172.21.15.158:0/1256340787 >> v1:172.21.15.172:6789/0 conn(0x7f075c0cc5d0 legacy=0x7f075c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:03.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0760cf9700 1 -- 172.21.15.158:0/1256340787 >> v1:172.21.15.158:6789/0 conn(0x7f075c06f340 legacy=0x7f075c071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:03.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0760cf9700 1 -- 172.21.15.158:0/1256340787 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c15e4e0 con 0x7f075c15dfb0 2023-10-16T09:44:03.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0760cf9700 1 --2- 172.21.15.158:0/1256340787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c15dfb0 0x7f075c06e230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0758009840 tx=0x7f0758009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e56497f748db0fd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:03.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754001b80 con 0x7f075c15dfb0 2023-10-16T09:44:03.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0758004030 con 0x7f075c15dfb0 2023-10-16T09:44:03.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.263+0000 7f0753fff700 1 -- 172.21.15.158:0/1256340787 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07580043c0 con 0x7f075c15dfb0 2023-10-16T09:44:03.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.264+0000 7f076375e700 1 -- 172.21.15.158:0/1256340787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c15dfb0 msgr2=0x7f075c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:03.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.264+0000 7f076375e700 1 --2- 172.21.15.158:0/1256340787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c15dfb0 0x7f075c06e230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0758009840 tx=0x7f0758009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:03.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.264+0000 7f076375e700 1 -- 172.21.15.158:0/1256340787 shutdown_connections 2023-10-16T09:44:03.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.264+0000 7f076375e700 1 --2- 172.21.15.158:0/1256340787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c15dfb0 0x7f075c06e230 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:03.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.264+0000 7f076375e700 1 -- 172.21.15.158:0/1256340787 >> 172.21.15.158:0/1256340787 conn(0x7f075c0d02b0 msgr2=0x7f075c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:03.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.264+0000 7f076375e700 1 -- 172.21.15.158:0/1256340787 shutdown_connections 2023-10-16T09:44:03.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.264+0000 7f076375e700 1 -- 172.21.15.158:0/1256340787 wait complete. 2023-10-16T09:44:03.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.265+0000 7f076375e700 1 Processor -- start 2023-10-16T09:44:03.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.265+0000 7f076375e700 1 -- start start 2023-10-16T09:44:03.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f076375e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c06f340 0x7f075c1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:03.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f076375e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f075c0cc5d0 0x7f075c0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:03.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f076375e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f075c15dfb0 0x7f075c1f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:03.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f076375e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c1f1b60 con 0x7f075c15dfb0 2023-10-16T09:44:03.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f076375e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c1f1090 con 0x7f075c06f340 2023-10-16T09:44:03.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f076375e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c153e80 con 0x7f075c0cc5d0 2023-10-16T09:44:03.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f075c15dfb0 0x7f075c1f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:03.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f075c15dfb0 0x7f075c1f0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47158/0 (socket says 172.21.15.158:47158) 2023-10-16T09:44:03.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 -- 172.21.15.158:0/1319123358 learned_addr learned my addr 172.21.15.158:0/1319123358 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:03.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0760cf9700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f075c0cc5d0 0x7f075c0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:03.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f07614fa700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c06f340 0x7f075c1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:03.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 -- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f075c0cc5d0 msgr2=0x7f075c0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:03.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f075c0cc5d0 0x7f075c0c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:03.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 -- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c06f340 msgr2=0x7f075c1560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:03.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c06f340 0x7f075c1560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:03.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 -- 172.21.15.158:0/1319123358 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580094a0 con 0x7f075c15dfb0 2023-10-16T09:44:03.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0760cf9700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f075c0cc5d0 0x7f075c0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:03.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f07614fa700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c06f340 0x7f075c1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:03.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.266+0000 7f0761cfb700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f075c15dfb0 0x7f075c1f0b50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0754001a90 tx=0x7f075400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:03.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.267+0000 7f0751ffb700 1 -- 172.21.15.158:0/1319123358 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075400ad70 con 0x7f075c15dfb0 2023-10-16T09:44:03.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.267+0000 7f0751ffb700 1 -- 172.21.15.158:0/1319123358 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07540070d0 con 0x7f075c15dfb0 2023-10-16T09:44:03.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.267+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c1f12c0 con 0x7f075c15dfb0 2023-10-16T09:44:03.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.267+0000 7f0751ffb700 1 -- 172.21.15.158:0/1319123358 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754007480 con 0x7f075c15dfb0 2023-10-16T09:44:03.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.267+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c1f1880 con 0x7f075c15dfb0 2023-10-16T09:44:03.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.268+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c153c50 con 0x7f075c15dfb0 2023-10-16T09:44:03.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.268+0000 7f0751ffb700 1 -- 172.21.15.158:0/1319123358 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f07540075e0 con 0x7f075c15dfb0 2023-10-16T09:44:03.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.268+0000 7f0751ffb700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f074006af00 0x7f074006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:03.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.268+0000 7f0751ffb700 1 -- 172.21.15.158:0/1319123358 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0754089770 con 0x7f075c15dfb0 2023-10-16T09:44:03.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.269+0000 7f07614fa700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f074006af00 0x7f074006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:03.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.269+0000 7f07614fa700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f074006af00 0x7f074006d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f074c0011e0 tx=0x7f074c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:03.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.273+0000 7f0751ffb700 1 -- 172.21.15.158:0/1319123358 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f075c153c50 con 0x7f075c15dfb0 2023-10-16T09:44:03.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.515+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f075c06e770 con 0x7f075c15dfb0 2023-10-16T09:44:03.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.515+0000 7f0751ffb700 1 -- 172.21.15.158:0/1319123358 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f0754018070 con 0x7f075c15dfb0 2023-10-16T09:44:03.517 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f074006af00 msgr2=0x7f074006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f074006af00 0x7f074006d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f074c0011e0 tx=0x7f074c009040 comp rx=0 tx=0).stop 2023-10-16T09:44:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f075c15dfb0 msgr2=0x7f075c1f0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f075c15dfb0 0x7f075c1f0b50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0754001a90 tx=0x7f075400cf90 comp rx=0 tx=0).stop 2023-10-16T09:44:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 shutdown_connections 2023-10-16T09:44:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f075c15dfb0 0x7f075c1f0b50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f075c06f340 0x7f075c1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f074006af00 0x7f074006d3c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 --2- 172.21.15.158:0/1319123358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f075c0cc5d0 0x7f075c0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.517+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 >> 172.21.15.158:0/1319123358 conn(0x7f075c0d02b0 msgr2=0x7f075c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.518+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 shutdown_connections 2023-10-16T09:44:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:03.518+0000 7f076375e700 1 -- 172.21.15.158:0/1319123358 wait complete. 2023-10-16T09:44:03.531 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T09:44:03.531 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:44:03.532 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi172.front.sepia.ceph.com 2023-10-16T09:44:03.532 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-16T09:44:03.532 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T09:44:03.532 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T09:44:03.533 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T09:44:03.533 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:44:03.533 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-16T09:44:03.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.534 DEBUG:teuthology.orchestra.run.smithi172:> ip addr 2023-10-16T09:44:03.551 INFO:teuthology.orchestra.run.smithi172.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:44:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:44:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:44:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 ::1/128 scope host 2023-10-16T09:44:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.553 INFO:teuthology.orchestra.run.smithi172.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:d9:17:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:03.553 INFO:teuthology.orchestra.run.smithi172.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:d9:17:2d brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:03.553 INFO:teuthology.orchestra.run.smithi172.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:88:7f:90 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:03.554 INFO:teuthology.orchestra.run.smithi172.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:44:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:88:7f:91 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: inet 172.21.15.172/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:44:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft 42362sec preferred_lft 42362sec 2023-10-16T09:44:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 fe80::382f:e61a:4284:c04d/64 scope link noprefixroute 2023-10-16T09:44:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.555 INFO:teuthology.orchestra.run.smithi172.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:44:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 8a:f2:3e:fe:e5:1b brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T09:44:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 fe80::9802:18ff:fed8:16b9/64 scope link 2023-10-16T09:44:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.556 INFO:teuthology.orchestra.run.smithi172.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:44:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 8a:f2:3e:fe:e5:1b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T09:44:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 fe80::88f2:3eff:fefe:e51b/64 scope link tentative 2023-10-16T09:44:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.557 DEBUG:teuthology.orchestra.run.smithi172:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:44:03.616 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-10-16T09:44:03.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.618 DEBUG:teuthology.orchestra.run.smithi172:> route 2023-10-16T09:44:03.678 INFO:teuthology.orchestra.run.smithi172.stdout:Kernel IP routing table 2023-10-16T09:44:03.678 INFO:teuthology.orchestra.run.smithi172.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:44:03.678 INFO:teuthology.orchestra.run.smithi172.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:44:03.679 INFO:teuthology.orchestra.run.smithi172.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:44:03.679 INFO:teuthology.orchestra.run.smithi172.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:44:03.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.679 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:03.680 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:03.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:44:03.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:44:03.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:44:03.681 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:03.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.782 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2023-10-16T09:44:03.799 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T09:44:03.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.800 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list-id 2023-10-16T09:44:03.856 INFO:teuthology.orchestra.run.smithi172.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-16T09:44:03.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.857 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:03.857 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:03.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T09:44:03.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-16T09:44:03.858 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:03.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:03.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T09:44:03.991 INFO:teuthology.orchestra.run.smithi172.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:44:03.991 INFO:teuthology.orchestra.run.smithi172.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:44:03.991 INFO:teuthology.orchestra.run.smithi172.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:44:03.992 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.992 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 ::1/128 scope host 2023-10-16T09:44:03.993 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.993 INFO:teuthology.orchestra.run.smithi172.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:44:03.993 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 72:63:08:1a:15:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T09:44:03.994 INFO:teuthology.orchestra.run.smithi172.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T09:44:03.994 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:03.995 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 fe80::7063:8ff:fe1a:15e0/64 scope link 2023-10-16T09:44:03.995 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:04.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T09:44:04.080 INFO:teuthology.orchestra.run.smithi172.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:44:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:44:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:44:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 ::1/128 scope host 2023-10-16T09:44:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:04.082 INFO:teuthology.orchestra.run.smithi172.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:44:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 72:63:08:1a:15:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T09:44:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T09:44:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 fe80::7063:8ff:fe1a:15e0/64 scope link 2023-10-16T09:44:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:04.092 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-16T09:44:04.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.093 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:04.093 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:04.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-16T09:44:04.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T09:44:04.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-16T09:44:04.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-16T09:44:04.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-16T09:44:04.095 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:04.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.339 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2023-10-16T09:44:04.339 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2023-10-16T09:44:04.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set brx.1 up 2023-10-16T09:44:04.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T09:44:04.340 DEBUG:teuthology.orchestra.run.smithi172:> ') 2023-10-16T09:44:04.453 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:04.454 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-16T09:44:04.474 INFO:teuthology.orchestra.run.smithi172.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-16T09:44:04.474 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:04.474 DEBUG:teuthology.orchestra.run.smithi172:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-16T09:44:04.531 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2023-10-16T09:44:04.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:04.617 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:04.617 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:04.640 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:04.640 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:04.641 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:04.641 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: trying mount with new device syntax: 3@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:04.641 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:04.641 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:04.642 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:04.642 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:04.668 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:04.668 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:04.668 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:04.669 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: trying mount with new device syntax: 3@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:04.669 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:04.669 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:04.669 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:04.670 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:04.670 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T09:44:04.670 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T09:44:04.670 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,key=3,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T09:44:04.671 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T09:44:04.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-16T09:44:04.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.706 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:04.777 INFO:teuthology.orchestra.run.smithi172.stdout:client4765 2023-10-16T09:44:04.777 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:04.777 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:04.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.834 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:04.905 INFO:teuthology.orchestra.run.smithi172.stdout:client4765 2023-10-16T09:44:04.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.905 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:04.976 INFO:teuthology.orchestra.run.smithi172.stdout:client4765 2023-10-16T09:44:04.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:04.977 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7cf363df-9988-44b1-be05-df2a6bf24f2d.client4765/status') 2023-10-16T09:44:05.056 INFO:teuthology.orchestra.run.smithi172.stdout:instance: client.4765 (3)192.168.0.2:0/3247445509 2023-10-16T09:44:05.056 INFO:teuthology.orchestra.run.smithi172.stdout:blocklisted: false 2023-10-16T09:44:05.057 INFO:teuthology.orchestra.run.smithi172.stderr:0+1 records in 2023-10-16T09:44:05.057 INFO:teuthology.orchestra.run.smithi172.stderr:0+1 records out 2023-10-16T09:44:05.057 INFO:teuthology.orchestra.run.smithi172.stderr:69 bytes copied, 2.0132e-05 s, 3.4 MB/s 2023-10-16T09:44:05.058 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-16T09:44:05.073 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:05.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.207+0000 7f2457d82700 1 Processor -- start 2023-10-16T09:44:05.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.207+0000 7f2457d82700 1 -- start start 2023-10-16T09:44:05.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2457d82700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f245006ab40 0x7f245006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2457d82700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2450161270 con 0x7f245006e5d0 2023-10-16T09:44:05.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2457d82700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2450161510 con 0x7f24500729b0 2023-10-16T09:44:05.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2457d82700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24501617f0 con 0x7f245006ab40 2023-10-16T09:44:05.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2455b1e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f245006ab40 0x7f245006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2455b1e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f245006ab40 0x7f245006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47162/0 (socket says 172.21.15.158:47162) 2023-10-16T09:44:05.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2455b1e700 1 -- 172.21.15.158:0/1066043278 learned_addr learned my addr 172.21.15.158:0/1066043278 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:05.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2455b1e700 1 -- 172.21.15.158:0/1066043278 >> v1:172.21.15.172:6789/0 conn(0x7f24500729b0 legacy=0x7f245015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2455b1e700 1 -- 172.21.15.158:0/1066043278 >> v1:172.21.15.158:6789/0 conn(0x7f245006e5d0 legacy=0x7f24500722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.208+0000 7f2454b1c700 1 -- 172.21.15.158:0/1066043278 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254835121 0 0) 0x7f2450161270 con 0x7f245006e5d0 2023-10-16T09:44:05.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2455b1e700 1 -- 172.21.15.158:0/1066043278 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2450161ad0 con 0x7f245006ab40 2023-10-16T09:44:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2455b1e700 1 --2- 172.21.15.158:0/1066043278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006ab40 0x7f245006d820 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f24400098f0 tx=0x7f2440009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2191d58d57cd2b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2454b1c700 1 -- 172.21.15.158:0/1066043278 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244000b8a0 con 0x7f245006ab40 2023-10-16T09:44:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2454b1c700 1 -- 172.21.15.158:0/1066043278 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244000ba00 con 0x7f245006ab40 2023-10-16T09:44:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2454b1c700 1 -- 172.21.15.158:0/1066043278 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244000bd90 con 0x7f245006ab40 2023-10-16T09:44:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2457d82700 1 -- 172.21.15.158:0/1066043278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006ab40 msgr2=0x7f245006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2457d82700 1 --2- 172.21.15.158:0/1066043278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006ab40 0x7f245006d820 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f24400098f0 tx=0x7f2440009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2457d82700 1 -- 172.21.15.158:0/1066043278 shutdown_connections 2023-10-16T09:44:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2457d82700 1 --2- 172.21.15.158:0/1066043278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006ab40 0x7f245006d820 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2457d82700 1 -- 172.21.15.158:0/1066043278 >> 172.21.15.158:0/1066043278 conn(0x7f24500d02b0 msgr2=0x7f24500d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.209+0000 7f2457d82700 1 -- 172.21.15.158:0/1066043278 shutdown_connections 2023-10-16T09:44:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.210+0000 7f2457d82700 1 -- 172.21.15.158:0/1066043278 wait complete. 2023-10-16T09:44:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 Processor -- start 2023-10-16T09:44:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 -- start start 2023-10-16T09:44:05.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f245006ab40 0x7f24501f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006e5d0 0x7f24501f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24500729b0 0x7f245015e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2450160800 con 0x7f245006e5d0 2023-10-16T09:44:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24500b1c70 con 0x7f24500729b0 2023-10-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.211+0000 7f2457d82700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24501f7300 con 0x7f245006ab40 2023-10-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006e5d0 0x7f24501f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006e5d0 0x7f24501f2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47170/0 (socket says 172.21.15.158:47170) 2023-10-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 -- 172.21.15.158:0/1603292999 learned_addr learned my addr 172.21.15.158:0/1603292999 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f2455b1e700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f245006ab40 0x7f24501f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245631f700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24500729b0 0x7f245015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 -- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f245006ab40 msgr2=0x7f24501f6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f245006ab40 0x7f24501f6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 -- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24500729b0 msgr2=0x7f245015e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24500729b0 0x7f245015e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 -- 172.21.15.158:0/1603292999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440009550 con 0x7f245006e5d0 2023-10-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f2455b1e700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f245006ab40 0x7f24501f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245631f700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24500729b0 0x7f245015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f245531d700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006e5d0 0x7f24501f2790 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f244c00c580 tx=0x7f244c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f24467fc700 1 -- 172.21.15.158:0/1603292999 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c0054e0 con 0x7f245006e5d0 2023-10-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f24467fc700 1 -- 172.21.15.158:0/1603292999 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244c003680 con 0x7f245006e5d0 2023-10-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24501f2cd0 con 0x7f245006e5d0 2023-10-16T09:44:05.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f24467fc700 1 -- 172.21.15.158:0/1603292999 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c003a30 con 0x7f245006e5d0 2023-10-16T09:44:05.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.212+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2450160a50 con 0x7f245006e5d0 2023-10-16T09:44:05.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.213+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24500b1e20 con 0x7f245006e5d0 2023-10-16T09:44:05.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.214+0000 7f24467fc700 1 -- 172.21.15.158:0/1603292999 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f244c003b90 con 0x7f245006e5d0 2023-10-16T09:44:05.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.214+0000 7f24467fc700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f243406af00 0x7f243406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.214+0000 7f24467fc700 1 -- 172.21.15.158:0/1603292999 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f244c088d60 con 0x7f245006e5d0 2023-10-16T09:44:05.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.214+0000 7f2455b1e700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f243406af00 0x7f243406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.214+0000 7f2455b1e700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f243406af00 0x7f243406d3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2440009130 tx=0x7f2440008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.219+0000 7f24467fc700 1 -- 172.21.15.158:0/1603292999 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f24500b1e20 con 0x7f245006e5d0 2023-10-16T09:44:05.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.414+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24500b1e20 con 0x7f245006e5d0 2023-10-16T09:44:05.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.415+0000 7f24467fc700 1 -- 172.21.15.158:0/1603292999 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f24500b1e20 con 0x7f245006e5d0 2023-10-16T09:44:05.418 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:05.419 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:05.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f243406af00 msgr2=0x7f243406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f243406af00 0x7f243406d3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2440009130 tx=0x7f2440008030 comp rx=0 tx=0).stop 2023-10-16T09:44:05.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006e5d0 msgr2=0x7f24501f2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006e5d0 0x7f24501f2790 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f244c00c580 tx=0x7f244c00c940 comp rx=0 tx=0).stop 2023-10-16T09:44:05.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 shutdown_connections 2023-10-16T09:44:05.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24500729b0 0x7f245015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f243406af00 0x7f243406d3c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f245006ab40 0x7f24501f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 --2- 172.21.15.158:0/1603292999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f245006e5d0 0x7f24501f2790 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 >> 172.21.15.158:0/1603292999 conn(0x7f24500d02b0 msgr2=0x7f245015d250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:05.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 shutdown_connections 2023-10-16T09:44:05.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.417+0000 7f2457d82700 1 -- 172.21.15.158:0/1603292999 wait complete. 2023-10-16T09:44:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:05.433 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.558+0000 7fd72ccf0700 1 Processor -- start 2023-10-16T09:44:05.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.558+0000 7fd72ccf0700 1 -- start start 2023-10-16T09:44:05.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd72ccf0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7281516c0 0x7fd7280cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd72ccf0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd728072800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd72ccf0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7280cc730 con 0x7fd728072370 2023-10-16T09:44:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd72ccf0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd728151aa0 con 0x7fd7281516c0 2023-10-16T09:44:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd72ccf0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280cbb60 con 0x7fd72806e030 2023-10-16T09:44:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd72659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7281516c0 0x7fd7280cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd725d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd728072370 0x7fd72806d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47354/0 (socket says 172.21.15.158:47354) 2023-10-16T09:44:05.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd725d9b700 1 -- 172.21.15.158:0/2544578997 learned_addr learned my addr 172.21.15.158:0/2544578997 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:05.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd72659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7281516c0 0x7fd7280cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47176/0 (socket says 172.21.15.158:47176) 2023-10-16T09:44:05.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.559+0000 7fd726d9d700 1 --2- 172.21.15.158:0/2544578997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd728072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72659c700 1 -- 172.21.15.158:0/2544578997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 msgr2=0x7fd728072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:05.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72659c700 1 --2- 172.21.15.158:0/2544578997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd728072800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72659c700 1 -- 172.21.15.158:0/2544578997 >> v1:172.21.15.158:6789/0 conn(0x7fd728072370 legacy=0x7fd72806d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72559a700 1 -- 172.21.15.158:0/2544578997 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294420597 0 0) 0x7fd7280cc730 con 0x7fd728072370 2023-10-16T09:44:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72659c700 1 -- 172.21.15.158:0/2544578997 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7280cbe00 con 0x7fd7281516c0 2023-10-16T09:44:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd726d9d700 1 --2- 172.21.15.158:0/2544578997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd728072800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72659c700 1 --2- 172.21.15.158:0/2544578997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280cdfa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009840 tx=0x7fd71c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a103644b4760fde server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72559a700 1 -- 172.21.15.158:0/2544578997 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c011830 con 0x7fd7281516c0 2023-10-16T09:44:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72559a700 1 -- 172.21.15.158:0/2544578997 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71c011990 con 0x7fd7281516c0 2023-10-16T09:44:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72559a700 1 -- 172.21.15.158:0/2544578997 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c011d20 con 0x7fd7281516c0 2023-10-16T09:44:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72ccf0700 1 -- 172.21.15.158:0/2544578997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 msgr2=0x7fd7280cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.560+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/2544578997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280cdfa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009840 tx=0x7fd71c009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.561+0000 7fd72ccf0700 1 -- 172.21.15.158:0/2544578997 shutdown_connections 2023-10-16T09:44:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.561+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/2544578997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280cdfa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.561+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/2544578997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd728072800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.561+0000 7fd72ccf0700 1 -- 172.21.15.158:0/2544578997 >> 172.21.15.158:0/2544578997 conn(0x7fd7280b1f00 msgr2=0x7fd7280b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.561+0000 7fd72ccf0700 1 -- 172.21.15.158:0/2544578997 shutdown_connections 2023-10-16T09:44:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.561+0000 7fd72ccf0700 1 -- 172.21.15.158:0/2544578997 wait complete. 2023-10-16T09:44:05.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.562+0000 7fd72ccf0700 1 Processor -- start 2023-10-16T09:44:05.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.562+0000 7fd72ccf0700 1 -- start start 2023-10-16T09:44:05.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd72ccf0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd72814fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd72ccf0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd728072370 0x7fd72815abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd72ccf0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd72ccf0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7281506c0 con 0x7fd7281516c0 2023-10-16T09:44:05.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd72ccf0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd728158b40 con 0x7fd728072370 2023-10-16T09:44:05.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd72ccf0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd728158e20 con 0x7fd72806e030 2023-10-16T09:44:05.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47186/0 (socket says 172.21.15.158:47186) 2023-10-16T09:44:05.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 -- 172.21.15.158:0/1266290007 learned_addr learned my addr 172.21.15.158:0/1266290007 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:05.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd72659c700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd72814fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd725d9b700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd728072370 0x7fd72815abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 -- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 msgr2=0x7fd72814fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd72814fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 -- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd728072370 msgr2=0x7fd72815abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd728072370 0x7fd72815abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 -- 172.21.15.158:0/1266290007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c0094a0 con 0x7fd7281516c0 2023-10-16T09:44:05.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd725d9b700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd728072370 0x7fd72815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:05.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.563+0000 7fd726d9d700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280c8150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd720007e30 tx=0x7fd72000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.564+0000 7fd716ffd700 1 -- 172.21.15.158:0/1266290007 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72000ddd0 con 0x7fd7281516c0 2023-10-16T09:44:05.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.564+0000 7fd716ffd700 1 -- 172.21.15.158:0/1266290007 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72000f040 con 0x7fd7281516c0 2023-10-16T09:44:05.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.564+0000 7fd716ffd700 1 -- 172.21.15.158:0/1266290007 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720015640 con 0x7fd7281516c0 2023-10-16T09:44:05.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.564+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd728159100 con 0x7fd7281516c0 2023-10-16T09:44:05.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.564+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7281596c0 con 0x7fd7281516c0 2023-10-16T09:44:05.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.564+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7281598e0 con 0x7fd7281516c0 2023-10-16T09:44:05.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.565+0000 7fd716ffd700 1 -- 172.21.15.158:0/1266290007 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd72001e3b0 con 0x7fd7281516c0 2023-10-16T09:44:05.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.565+0000 7fd716ffd700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd70806af00 0x7fd70806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.565+0000 7fd716ffd700 1 -- 172.21.15.158:0/1266290007 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd7200898a0 con 0x7fd7281516c0 2023-10-16T09:44:05.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.565+0000 7fd72659c700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd70806af00 0x7fd70806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.566+0000 7fd72659c700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd70806af00 0x7fd70806d3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0090b0 tx=0x7fd71c01c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.570+0000 7fd716ffd700 1 -- 172.21.15.158:0/1266290007 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7200546f0 con 0x7fd7281516c0 2023-10-16T09:44:05.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.761+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7280681b0 con 0x7fd7281516c0 2023-10-16T09:44:05.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.761+0000 7fd716ffd700 1 -- 172.21.15.158:0/1266290007 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd72001a020 con 0x7fd7281516c0 2023-10-16T09:44:05.763 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:05.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd70806af00 msgr2=0x7fd70806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd70806af00 0x7fd70806d3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0090b0 tx=0x7fd71c01c040 comp rx=0 tx=0).stop 2023-10-16T09:44:05.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 msgr2=0x7fd7280c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280c8150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd720007e30 tx=0x7fd72000d3b0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 shutdown_connections 2023-10-16T09:44:05.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7281516c0 0x7fd7280c8150 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd70806af00 0x7fd70806d3c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd72806e030 0x7fd72814fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 --2- 172.21.15.158:0/1266290007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd728072370 0x7fd72815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 >> 172.21.15.158:0/1266290007 conn(0x7fd7280b1f00 msgr2=0x7fd7280b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:05.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 shutdown_connections 2023-10-16T09:44:05.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.763+0000 7fd72ccf0700 1 -- 172.21.15.158:0/1266290007 wait complete. 2023-10-16T09:44:05.777 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:05.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.898+0000 7f27323de700 1 Processor -- start 2023-10-16T09:44:05.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.898+0000 7f27323de700 1 -- start start 2023-10-16T09:44:05.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f27323de700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f272c15dad0 0x7f272c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f27323de700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f272c15deb0 0x7f272c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f27323de700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c0d0ae0 con 0x7f272c0cea20 2023-10-16T09:44:05.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f27323de700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272c0cee50 con 0x7f272c15deb0 2023-10-16T09:44:05.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f27323de700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272c0d12d0 con 0x7f272c15dad0 2023-10-16T09:44:05.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f272b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f272c15deb0 0x7f272c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f272b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f272c15deb0 0x7f272c0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47190/0 (socket says 172.21.15.158:47190) 2023-10-16T09:44:05.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f273097b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f272c0cea20 0x7f272c0c3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47370/0 (socket says 172.21.15.158:47370) 2023-10-16T09:44:05.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f272b7fe700 1 -- 172.21.15.158:0/1063727226 learned_addr learned my addr 172.21.15.158:0/1063727226 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:05.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.899+0000 7f272bfff700 1 --2- 172.21.15.158:0/1063727226 >> v2:172.21.15.172:3300/0 conn(0x7f272c15dad0 0x7f272c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272b7fe700 1 -- 172.21.15.158:0/1063727226 >> v2:172.21.15.172:3300/0 conn(0x7f272c15dad0 msgr2=0x7f272c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:05.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272b7fe700 1 --2- 172.21.15.158:0/1063727226 >> v2:172.21.15.172:3300/0 conn(0x7f272c15dad0 0x7f272c0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272b7fe700 1 -- 172.21.15.158:0/1063727226 >> v1:172.21.15.158:6789/0 conn(0x7f272c0cea20 legacy=0x7f272c0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272b7fe700 1 -- 172.21.15.158:0/1063727226 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c0cbdf0 con 0x7f272c15deb0 2023-10-16T09:44:05.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272affd700 1 -- 172.21.15.158:0/1063727226 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983783479 0 0) 0x7f272c0d0ae0 con 0x7f272c0cea20 2023-10-16T09:44:05.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272b7fe700 1 --2- 172.21.15.158:0/1063727226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c0ce4e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f271c0098f0 tx=0x7f271c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe993da202d98e1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272affd700 1 -- 172.21.15.158:0/1063727226 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c011830 con 0x7f272c15deb0 2023-10-16T09:44:05.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272affd700 1 -- 172.21.15.158:0/1063727226 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c011990 con 0x7f272c15deb0 2023-10-16T09:44:05.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f272affd700 1 -- 172.21.15.158:0/1063727226 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c011d20 con 0x7f272c15deb0 2023-10-16T09:44:05.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f27323de700 1 -- 172.21.15.158:0/1063727226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 msgr2=0x7f272c0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f27323de700 1 --2- 172.21.15.158:0/1063727226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c0ce4e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f271c0098f0 tx=0x7f271c009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:05.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f27323de700 1 -- 172.21.15.158:0/1063727226 shutdown_connections 2023-10-16T09:44:05.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f27323de700 1 --2- 172.21.15.158:0/1063727226 >> v2:172.21.15.172:3300/0 conn(0x7f272c15dad0 0x7f272c0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f27323de700 1 --2- 172.21.15.158:0/1063727226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c0ce4e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f27323de700 1 -- 172.21.15.158:0/1063727226 >> 172.21.15.158:0/1063727226 conn(0x7f272c099a10 msgr2=0x7f272c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:05.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.900+0000 7f27323de700 1 -- 172.21.15.158:0/1063727226 shutdown_connections 2023-10-16T09:44:05.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.901+0000 7f27323de700 1 -- 172.21.15.158:0/1063727226 wait complete. 2023-10-16T09:44:05.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.902+0000 7f27323de700 1 Processor -- start 2023-10-16T09:44:05.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.902+0000 7f27323de700 1 -- start start 2023-10-16T09:44:05.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c0cea20 0x7f272c14fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c15dad0 0x7f272c0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c14e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c06f060 con 0x7f272c15deb0 2023-10-16T09:44:05.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c14e690 con 0x7f272c15dad0 2023-10-16T09:44:05.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272c0c5630 con 0x7f272c0cea20 2023-10-16T09:44:05.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c14e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c14e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47200/0 (socket says 172.21.15.158:47200) 2023-10-16T09:44:05.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 -- 172.21.15.158:0/1212515707 learned_addr learned my addr 172.21.15.158:0/1212515707 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:05.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f272b7fe700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c15dad0 0x7f272c0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 -- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c0cea20 msgr2=0x7f272c14fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f272bfff700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c0cea20 0x7f272c14fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c0cea20 0x7f272c14fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 -- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c15dad0 msgr2=0x7f272c0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:05.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c15dad0 0x7f272c0c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:05.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 -- 172.21.15.158:0/1212515707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c009550 con 0x7f272c15deb0 2023-10-16T09:44:05.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f273097b700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c14e150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2724002230 tx=0x7f272400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f2728ff9700 1 -- 172.21.15.158:0/1212515707 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272400ac20 con 0x7f272c15deb0 2023-10-16T09:44:05.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f2728ff9700 1 -- 172.21.15.158:0/1212515707 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272400ada0 con 0x7f272c15deb0 2023-10-16T09:44:05.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c0c5910 con 0x7f272c15deb0 2023-10-16T09:44:05.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f2728ff9700 1 -- 172.21.15.158:0/1212515707 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724007310 con 0x7f272c15deb0 2023-10-16T09:44:05.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.903+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c0c5ed0 con 0x7f272c15deb0 2023-10-16T09:44:05.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.904+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272c0c60c0 con 0x7f272c15deb0 2023-10-16T09:44:05.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.905+0000 7f2728ff9700 1 -- 172.21.15.158:0/1212515707 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2724007470 con 0x7f272c15deb0 2023-10-16T09:44:05.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.905+0000 7f2728ff9700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f271806af00 0x7f271806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:05.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.905+0000 7f2728ff9700 1 -- 172.21.15.158:0/1212515707 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f27240898c0 con 0x7f272c15deb0 2023-10-16T09:44:05.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.905+0000 7f272bfff700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f271806af00 0x7f271806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:05.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.906+0000 7f272bfff700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f271806af00 0x7f271806d3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2720009ab0 tx=0x7f2720008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:05.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:05.910+0000 7f2728ff9700 1 -- 172.21.15.158:0/1212515707 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27240545e0 con 0x7f272c15deb0 2023-10-16T09:44:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.108+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f272c0681b0 con 0x7f272c15deb0 2023-10-16T09:44:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.109+0000 7f2728ff9700 1 -- 172.21.15.158:0/1212515707 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2724018020 con 0x7f272c15deb0 2023-10-16T09:44:06.111 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f271806af00 msgr2=0x7f271806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f271806af00 0x7f271806d3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2720009ab0 tx=0x7f2720008040 comp rx=0 tx=0).stop 2023-10-16T09:44:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 msgr2=0x7f272c14e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c14e150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2724002230 tx=0x7f272400a060 comp rx=0 tx=0).stop 2023-10-16T09:44:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 shutdown_connections 2023-10-16T09:44:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15deb0 0x7f272c14e150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f271806af00 0x7f271806d3c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c0cea20 0x7f272c14fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 --2- 172.21.15.158:0/1212515707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c15dad0 0x7f272c0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 >> 172.21.15.158:0/1212515707 conn(0x7f272c099a10 msgr2=0x7f272c15cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 shutdown_connections 2023-10-16T09:44:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.111+0000 7f27323de700 1 -- 172.21.15.158:0/1212515707 wait complete. 2023-10-16T09:44:06.125 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:06.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.247+0000 7f92c15a5700 1 Processor -- start 2023-10-16T09:44:06.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.247+0000 7f92c15a5700 1 -- start start 2023-10-16T09:44:06.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92c15a5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f92bc160d30 0x7f92bc0b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92c15a5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92bc0d0530 0x7f92bc0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92c15a5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc0b7c90 con 0x7f92bc0d19a0 2023-10-16T09:44:06.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92c15a5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc0b8910 con 0x7f92bc0d0530 2023-10-16T09:44:06.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92c15a5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc0d0ef0 con 0x7f92bc160d30 2023-10-16T09:44:06.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92bc0d0530 0x7f92bc0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92bc0d0530 0x7f92bc0d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47204/0 (socket says 172.21.15.158:47204) 2023-10-16T09:44:06.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 -- 172.21.15.158:0/962779532 learned_addr learned my addr 172.21.15.158:0/962779532 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:06.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92baffd700 1 --2- 172.21.15.158:0/962779532 >> v2:172.21.15.172:3300/0 conn(0x7f92bc160d30 0x7f92bc0b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 -- 172.21.15.158:0/962779532 >> v2:172.21.15.172:3300/0 conn(0x7f92bc160d30 msgr2=0x7f92bc0b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:06.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 --2- 172.21.15.158:0/962779532 >> v2:172.21.15.172:3300/0 conn(0x7f92bc160d30 0x7f92bc0b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 -- 172.21.15.158:0/962779532 >> v1:172.21.15.172:6789/0 conn(0x7f92bc0d19a0 legacy=0x7f92bc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 -- 172.21.15.158:0/962779532 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc0b7f70 con 0x7f92bc0d0530 2023-10-16T09:44:06.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92ba7fc700 1 --2- 172.21.15.158:0/962779532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc0d09b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f92b00098f0 tx=0x7f92b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a589789f340a88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92b9ffb700 1 -- 172.21.15.158:0/962779532 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b000b860 con 0x7f92bc0d0530 2023-10-16T09:44:06.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.248+0000 7f92b9ffb700 1 -- 172.21.15.158:0/962779532 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b000b9c0 con 0x7f92bc0d0530 2023-10-16T09:44:06.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92b9ffb700 1 -- 172.21.15.158:0/962779532 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b000bd50 con 0x7f92bc0d0530 2023-10-16T09:44:06.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 -- 172.21.15.158:0/962779532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 msgr2=0x7f92bc0d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 --2- 172.21.15.158:0/962779532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc0d09b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f92b00098f0 tx=0x7f92b0009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:06.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 -- 172.21.15.158:0/962779532 shutdown_connections 2023-10-16T09:44:06.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 --2- 172.21.15.158:0/962779532 >> v2:172.21.15.172:3300/0 conn(0x7f92bc160d30 0x7f92bc0b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 --2- 172.21.15.158:0/962779532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc0d09b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 -- 172.21.15.158:0/962779532 >> 172.21.15.158:0/962779532 conn(0x7f92bc0669f0 msgr2=0x7f92bc066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:06.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 -- 172.21.15.158:0/962779532 shutdown_connections 2023-10-16T09:44:06.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.249+0000 7f92c15a5700 1 -- 172.21.15.158:0/962779532 wait complete. 2023-10-16T09:44:06.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.251+0000 7f92c15a5700 1 Processor -- start 2023-10-16T09:44:06.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.251+0000 7f92c15a5700 1 -- start start 2023-10-16T09:44:06.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc15acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92bc0d19a0 0x7f92bc0c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92bc160d30 0x7f92bc0c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc15e460 con 0x7f92bc0d0530 2023-10-16T09:44:06.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc1f2bd0 con 0x7f92bc160d30 2023-10-16T09:44:06.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc15d8e0 con 0x7f92bc0d19a0 2023-10-16T09:44:06.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc15acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc15acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47210/0 (socket says 172.21.15.158:47210) 2023-10-16T09:44:06.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 -- 172.21.15.158:0/4082728903 learned_addr learned my addr 172.21.15.158:0/4082728903 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:06.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92bb7fe700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92bc160d30 0x7f92bc0c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92ba7fc700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92bc0d19a0 0x7f92bc0c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 -- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92bc0d19a0 msgr2=0x7f92bc0c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92bc0d19a0 0x7f92bc0c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 -- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92bc160d30 msgr2=0x7f92bc0c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92bc160d30 0x7f92bc0c9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 -- 172.21.15.158:0/4082728903 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b0009550 con 0x7f92bc0d0530 2023-10-16T09:44:06.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92baffd700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc15acb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00d800 tx=0x7f92ac00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92a37fe700 1 -- 172.21.15.158:0/4082728903 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac0125f0 con 0x7f92bc0d0530 2023-10-16T09:44:06.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92a37fe700 1 -- 172.21.15.158:0/4082728903 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac012750 con 0x7f92bc0d0530 2023-10-16T09:44:06.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc15dbc0 con 0x7f92bc0d0530 2023-10-16T09:44:06.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92a37fe700 1 -- 172.21.15.158:0/4082728903 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac012b00 con 0x7f92bc0d0530 2023-10-16T09:44:06.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.252+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc15e180 con 0x7f92bc0d0530 2023-10-16T09:44:06.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.253+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc0681b0 con 0x7f92bc0d0530 2023-10-16T09:44:06.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.254+0000 7f92a37fe700 1 -- 172.21.15.158:0/4082728903 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f92ac012c60 con 0x7f92bc0d0530 2023-10-16T09:44:06.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.254+0000 7f92a37fe700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f92a806af00 0x7f92a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.254+0000 7f92ba7fc700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f92a806af00 0x7f92a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.254+0000 7f92a37fe700 1 -- 172.21.15.158:0/4082728903 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f92ac089c70 con 0x7f92bc0d0530 2023-10-16T09:44:06.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.255+0000 7f92ba7fc700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f92a806af00 0x7f92a806d3c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009c40 tx=0x7f92b0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.258+0000 7f92a37fe700 1 -- 172.21.15.158:0/4082728903 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f92ac054a30 con 0x7f92bc0d0530 2023-10-16T09:44:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.451+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92bc1f2ed0 con 0x7f92bc0d0530 2023-10-16T09:44:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.451+0000 7f92a37fe700 1 -- 172.21.15.158:0/4082728903 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f92ac058050 con 0x7f92bc0d0530 2023-10-16T09:44:06.454 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f92a806af00 msgr2=0x7f92a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f92a806af00 0x7f92a806d3c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009c40 tx=0x7f92b0008030 comp rx=0 tx=0).stop 2023-10-16T09:44:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 msgr2=0x7f92bc15acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc15acb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00d800 tx=0x7f92ac00dbc0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 shutdown_connections 2023-10-16T09:44:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92bc160d30 0x7f92bc0c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92bc0d0530 0x7f92bc15acb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f92a806af00 0x7f92a806d3c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 --2- 172.21.15.158:0/4082728903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92bc0d19a0 0x7f92bc0c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 >> 172.21.15.158:0/4082728903 conn(0x7f92bc0669f0 msgr2=0x7f92bc0d3690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 shutdown_connections 2023-10-16T09:44:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.454+0000 7f92c15a5700 1 -- 172.21.15.158:0/4082728903 wait complete. 2023-10-16T09:44:06.467 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:06.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.589+0000 7f12a8751700 1 Processor -- start 2023-10-16T09:44:06.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.589+0000 7f12a8751700 1 -- start start 2023-10-16T09:44:06.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a8751700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12a0160d30 0x7f12a00b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a8751700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12a00d0530 0x7f12a00d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a8751700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a00b7c90 con 0x7f12a00d19a0 2023-10-16T09:44:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a8751700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a00b8910 con 0x7f12a0160d30 2023-10-16T09:44:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a8751700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a00d0ef0 con 0x7f12a00d0530 2023-10-16T09:44:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a64ed700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12a0160d30 0x7f12a00b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a6cee700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f12a00d19a0 0x7f12a00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47398/0 (socket says 172.21.15.158:47398) 2023-10-16T09:44:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a6cee700 1 -- 172.21.15.158:0/1037168791 learned_addr learned my addr 172.21.15.158:0/1037168791 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a64ed700 1 -- 172.21.15.158:0/1037168791 >> v2:172.21.15.172:3300/0 conn(0x7f12a00d0530 msgr2=0x7f12a00d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a64ed700 1 --2- 172.21.15.158:0/1037168791 >> v2:172.21.15.172:3300/0 conn(0x7f12a00d0530 0x7f12a00d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a64ed700 1 -- 172.21.15.158:0/1037168791 >> v1:172.21.15.158:6789/0 conn(0x7f12a00d19a0 legacy=0x7f12a00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a64ed700 1 -- 172.21.15.158:0/1037168791 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a00b7f70 con 0x7f12a0160d30 2023-10-16T09:44:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a64ed700 1 --2- 172.21.15.158:0/1037168791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a0160d30 0x7f12a00b8f50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f129c009840 tx=0x7f129c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6f7b66d862d56d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a54eb700 1 -- 172.21.15.158:0/1037168791 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c011830 con 0x7f12a0160d30 2023-10-16T09:44:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a54eb700 1 -- 172.21.15.158:0/1037168791 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c011990 con 0x7f12a0160d30 2023-10-16T09:44:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.590+0000 7f12a54eb700 1 -- 172.21.15.158:0/1037168791 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c011d20 con 0x7f12a0160d30 2023-10-16T09:44:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 -- 172.21.15.158:0/1037168791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a0160d30 msgr2=0x7f12a00b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 --2- 172.21.15.158:0/1037168791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a0160d30 0x7f12a00b8f50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f129c009840 tx=0x7f129c009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 -- 172.21.15.158:0/1037168791 shutdown_connections 2023-10-16T09:44:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 --2- 172.21.15.158:0/1037168791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a0160d30 0x7f12a00b8f50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 --2- 172.21.15.158:0/1037168791 >> v2:172.21.15.172:3300/0 conn(0x7f12a00d0530 0x7f12a00d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 -- 172.21.15.158:0/1037168791 >> 172.21.15.158:0/1037168791 conn(0x7f12a00669f0 msgr2=0x7f12a0066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 -- 172.21.15.158:0/1037168791 shutdown_connections 2023-10-16T09:44:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.591+0000 7f12a8751700 1 -- 172.21.15.158:0/1037168791 wait complete. 2023-10-16T09:44:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.592+0000 7f12a8751700 1 Processor -- start 2023-10-16T09:44:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.592+0000 7f12a8751700 1 -- start start 2023-10-16T09:44:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a8751700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a00d0530 0x7f12a0159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a8751700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a00d19a0 0x7f12a0156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a8751700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a0160d30 0x7f12a0145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a8751700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0143430 con 0x7f12a00d0530 2023-10-16T09:44:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a8751700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0145ca0 con 0x7f12a0160d30 2023-10-16T09:44:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a8751700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0145f80 con 0x7f12a00d19a0 2023-10-16T09:44:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a00d0530 0x7f12a0159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a00d0530 0x7f12a0159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47224/0 (socket says 172.21.15.158:47224) 2023-10-16T09:44:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 -- 172.21.15.158:0/1966479921 learned_addr learned my addr 172.21.15.158:0/1966479921 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a6cee700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a0160d30 0x7f12a0145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a5cec700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a00d19a0 0x7f12a0156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 -- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a00d19a0 msgr2=0x7f12a0156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a00d19a0 0x7f12a0156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 -- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a0160d30 msgr2=0x7f12a0145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a0160d30 0x7f12a0145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 -- 172.21.15.158:0/1966479921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c0094a0 con 0x7f12a00d0530 2023-10-16T09:44:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a6cee700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a0160d30 0x7f12a0145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.593+0000 7f12a64ed700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a00d0530 0x7f12a0159020 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f129c014040 tx=0x7f129c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.594+0000 7f1296ffd700 1 -- 172.21.15.158:0/1966479921 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c0045f0 con 0x7f12a00d0530 2023-10-16T09:44:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.594+0000 7f1296ffd700 1 -- 172.21.15.158:0/1966479921 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c01e070 con 0x7f12a00d0530 2023-10-16T09:44:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.594+0000 7f1296ffd700 1 -- 172.21.15.158:0/1966479921 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c003b00 con 0x7f12a00d0530 2023-10-16T09:44:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.594+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a0146260 con 0x7f12a00d0530 2023-10-16T09:44:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.594+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a01459c0 con 0x7f12a00d0530 2023-10-16T09:44:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.595+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a0146450 con 0x7f12a00d0530 2023-10-16T09:44:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.595+0000 7f1296ffd700 1 -- 172.21.15.158:0/1966479921 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f129c003c60 con 0x7f12a00d0530 2023-10-16T09:44:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.595+0000 7f1296ffd700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f128406af00 0x7f128406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.596+0000 7f1296ffd700 1 -- 172.21.15.158:0/1966479921 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f129c08aa10 con 0x7f12a00d0530 2023-10-16T09:44:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.596+0000 7f12a5cec700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f128406af00 0x7f128406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.596+0000 7f12a5cec700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f128406af00 0x7f128406d3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1290006fd0 tx=0x7f1290008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.600+0000 7f1296ffd700 1 -- 172.21.15.158:0/1966479921 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12a0146450 con 0x7f12a00d0530 2023-10-16T09:44:06.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.790+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f12a0143880 con 0x7f12a00d0530 2023-10-16T09:44:06.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.791+0000 7f1296ffd700 1 -- 172.21.15.158:0/1966479921 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f12a0143880 con 0x7f12a00d0530 2023-10-16T09:44:06.793 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:06.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f128406af00 msgr2=0x7f128406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f128406af00 0x7f128406d3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1290006fd0 tx=0x7f1290008040 comp rx=0 tx=0).stop 2023-10-16T09:44:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a00d0530 msgr2=0x7f12a0159020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a00d0530 0x7f12a0159020 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f129c014040 tx=0x7f129c004750 comp rx=0 tx=0).stop 2023-10-16T09:44:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 shutdown_connections 2023-10-16T09:44:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a0160d30 0x7f12a0145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a00d0530 0x7f12a0159020 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f128406af00 0x7f128406d3c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 --2- 172.21.15.158:0/1966479921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a00d19a0 0x7f12a0156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 >> 172.21.15.158:0/1966479921 conn(0x7f12a00669f0 msgr2=0x7f12a0157060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 shutdown_connections 2023-10-16T09:44:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.793+0000 7f12a8751700 1 -- 172.21.15.158:0/1966479921 wait complete. 2023-10-16T09:44:06.807 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:06.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.926+0000 7f692cba2700 1 Processor -- start 2023-10-16T09:44:06.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.926+0000 7f692cba2700 1 -- start start 2023-10-16T09:44:06.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692cba2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f692815dbc0 0x7f69280cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692cba2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f69280d1d50 0x7f69280cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692cba2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69280d21e0 con 0x7f692806ba90 2023-10-16T09:44:06.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692cba2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6928162550 con 0x7f692815dbc0 2023-10-16T09:44:06.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692cba2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692815f320 con 0x7f69280d1d50 2023-10-16T09:44:06.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f692815dbc0 0x7f69280cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f692815dbc0 0x7f69280cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47232/0 (socket says 172.21.15.158:47232) 2023-10-16T09:44:06.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f6926d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f692806ba90 0x7f692806beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47412/0 (socket says 172.21.15.158:47412) 2023-10-16T09:44:06.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f692659c700 1 -- 172.21.15.158:0/1719728307 learned_addr learned my addr 172.21.15.158:0/1719728307 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:06.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.927+0000 7f6925d9b700 1 --2- 172.21.15.158:0/1719728307 >> v2:172.21.15.172:3300/0 conn(0x7f69280d1d50 0x7f69280cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692659c700 1 -- 172.21.15.158:0/1719728307 >> v2:172.21.15.172:3300/0 conn(0x7f69280d1d50 msgr2=0x7f69280cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:06.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692659c700 1 --2- 172.21.15.158:0/1719728307 >> v2:172.21.15.172:3300/0 conn(0x7f69280d1d50 0x7f69280cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692659c700 1 -- 172.21.15.158:0/1719728307 >> v1:172.21.15.158:6789/0 conn(0x7f692806ba90 legacy=0x7f692806beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692559a700 1 -- 172.21.15.158:0/1719728307 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875734323 0 0) 0x7f69280d21e0 con 0x7f692806ba90 2023-10-16T09:44:06.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692659c700 1 -- 172.21.15.158:0/1719728307 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692815ef10 con 0x7f692815dbc0 2023-10-16T09:44:06.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f6925d9b700 1 --2- 172.21.15.158:0/1719728307 >> v2:172.21.15.172:3300/0 conn(0x7f69280d1d50 0x7f69280cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:06.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692659c700 1 --2- 172.21.15.158:0/1719728307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f692815dbc0 0x7f69280cefa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f69280b36d0 tx=0x7f691c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b515811a594aa819 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692559a700 1 -- 172.21.15.158:0/1719728307 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c00b870 con 0x7f692815dbc0 2023-10-16T09:44:06.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692559a700 1 -- 172.21.15.158:0/1719728307 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c00b9d0 con 0x7f692815dbc0 2023-10-16T09:44:06.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692559a700 1 -- 172.21.15.158:0/1719728307 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c00bd60 con 0x7f692815dbc0 2023-10-16T09:44:06.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692cba2700 1 -- 172.21.15.158:0/1719728307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f692815dbc0 msgr2=0x7f69280cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.928+0000 7f692cba2700 1 --2- 172.21.15.158:0/1719728307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f692815dbc0 0x7f69280cefa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f69280b36d0 tx=0x7f691c009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.929+0000 7f692cba2700 1 -- 172.21.15.158:0/1719728307 shutdown_connections 2023-10-16T09:44:06.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.929+0000 7f692cba2700 1 --2- 172.21.15.158:0/1719728307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f692815dbc0 0x7f69280cefa0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.929+0000 7f692cba2700 1 --2- 172.21.15.158:0/1719728307 >> v2:172.21.15.172:3300/0 conn(0x7f69280d1d50 0x7f69280cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.929+0000 7f692cba2700 1 -- 172.21.15.158:0/1719728307 >> 172.21.15.158:0/1719728307 conn(0x7f692806a8d0 msgr2=0x7f692806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:06.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.929+0000 7f692cba2700 1 -- 172.21.15.158:0/1719728307 shutdown_connections 2023-10-16T09:44:06.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.929+0000 7f692cba2700 1 -- 172.21.15.158:0/1719728307 wait complete. 2023-10-16T09:44:06.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.930+0000 7f692cba2700 1 Processor -- start 2023-10-16T09:44:06.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.930+0000 7f692cba2700 1 -- start start 2023-10-16T09:44:06.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f692cba2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f692806ba90 0x7f6928150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f692cba2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6928151230 0x7f69281516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f692cba2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69280cc830 0x7f69280cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f692cba2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69280cd200 con 0x7f6928151230 2023-10-16T09:44:06.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f692cba2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69280b5cb0 con 0x7f69280cc830 2023-10-16T09:44:06.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f692cba2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69280b6140 con 0x7f692806ba90 2023-10-16T09:44:06.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f6925d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6928151230 0x7f69281516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f6925d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6928151230 0x7f69281516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47240/0 (socket says 172.21.15.158:47240) 2023-10-16T09:44:06.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.931+0000 7f6925d9b700 1 -- 172.21.15.158:0/771928522 learned_addr learned my addr 172.21.15.158:0/771928522 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:06.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6926d9d700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69280cc830 0x7f69280cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f692659c700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f692806ba90 0x7f6928150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6925d9b700 1 -- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f692806ba90 msgr2=0x7f6928150c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6925d9b700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f692806ba90 0x7f6928150c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6925d9b700 1 -- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69280cc830 msgr2=0x7f69280cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:06.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6925d9b700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69280cc830 0x7f69280cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:06.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6925d9b700 1 -- 172.21.15.158:0/771928522 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c009550 con 0x7f6928151230 2023-10-16T09:44:06.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6926d9d700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69280cc830 0x7f69280cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:06.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f692659c700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f692806ba90 0x7f6928150c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:06.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6925d9b700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6928151230 0x7f69281516b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f691800e810 tx=0x7f691800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6916ffd700 1 -- 172.21.15.158:0/771928522 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918004230 con 0x7f6928151230 2023-10-16T09:44:06.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6916ffd700 1 -- 172.21.15.158:0/771928522 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69180043b0 con 0x7f6928151230 2023-10-16T09:44:06.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69280b50a0 con 0x7f6928151230 2023-10-16T09:44:06.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f6916ffd700 1 -- 172.21.15.158:0/771928522 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180038e0 con 0x7f6928151230 2023-10-16T09:44:06.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.932+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69280b5660 con 0x7f6928151230 2023-10-16T09:44:06.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.933+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69280b5e20 con 0x7f6928151230 2023-10-16T09:44:06.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.934+0000 7f6916ffd700 1 -- 172.21.15.158:0/771928522 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6918003a40 con 0x7f6928151230 2023-10-16T09:44:06.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.934+0000 7f6916ffd700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f690806af00 0x7f690806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:06.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.934+0000 7f6916ffd700 1 -- 172.21.15.158:0/771928522 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6918089320 con 0x7f6928151230 2023-10-16T09:44:06.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.934+0000 7f692659c700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f690806af00 0x7f690806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:06.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.934+0000 7f692659c700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f690806af00 0x7f690806d3c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69280cd3f0 tx=0x7f691c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:06.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:06.938+0000 7f6916ffd700 1 -- 172.21.15.158:0/771928522 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69180540e0 con 0x7f6928151230 2023-10-16T09:44:07.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.131+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69280681b0 con 0x7f6928151230 2023-10-16T09:44:07.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.132+0000 7f6916ffd700 1 -- 172.21.15.158:0/771928522 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f6918057700 con 0x7f6928151230 2023-10-16T09:44:07.134 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:07.134 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:07.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f690806af00 msgr2=0x7f690806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f690806af00 0x7f690806d3c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69280cd3f0 tx=0x7f691c008030 comp rx=0 tx=0).stop 2023-10-16T09:44:07.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6928151230 msgr2=0x7f69281516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6928151230 0x7f69281516b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f691800e810 tx=0x7f691800ebd0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 shutdown_connections 2023-10-16T09:44:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69280cc830 0x7f69280cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f690806af00 0x7f690806d3c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f692806ba90 0x7f6928150c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 --2- 172.21.15.158:0/771928522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6928151230 0x7f69281516b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 >> 172.21.15.158:0/771928522 conn(0x7f692806a8d0 msgr2=0x7f69280cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:07.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 shutdown_connections 2023-10-16T09:44:07.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.134+0000 7f692cba2700 1 -- 172.21.15.158:0/771928522 wait complete. 2023-10-16T09:44:07.148 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:07.149 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:07.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.270+0000 7f166380a700 1 Processor -- start 2023-10-16T09:44:07.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.270+0000 7f166380a700 1 -- start start 2023-10-16T09:44:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.271+0000 7f166380a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 0x7f165c0cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.271+0000 7f166380a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165c15a870 con 0x7f165c15e990 2023-10-16T09:44:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.271+0000 7f166380a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165c071d10 con 0x7f165c15a490 2023-10-16T09:44:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.271+0000 7f166380a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0cab90 con 0x7f165c06cc60 2023-10-16T09:44:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.271+0000 7f1660da5700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f165c15e990 0x7f165c06c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47422/0 (socket says 172.21.15.158:47422) 2023-10-16T09:44:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.271+0000 7f1660da5700 1 -- 172.21.15.158:0/3920474983 learned_addr learned my addr 172.21.15.158:0/3920474983 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1661da7700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 0x7f165c0cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629707939 0 0) 0x7f165c15a870 con 0x7f165c15e990 2023-10-16T09:44:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16480035b0 con 0x7f165c15e990 2023-10-16T09:44:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 419336832 0 0) 0x7f16480035b0 con 0x7f165c15e990 2023-10-16T09:44:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 msgr2=0x7f165c0cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 0x7f165c0cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 >> v1:172.21.15.172:6789/0 conn(0x7f165c15a490 legacy=0x7f165c071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c0cae70 con 0x7f165c15e990 2023-10-16T09:44:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1658001670 con 0x7f165c15e990 2023-10-16T09:44:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 >> v1:172.21.15.158:6789/0 conn(0x7f165c15e990 legacy=0x7f165c06c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1648003a30 0x7f1648003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648004850 0x7f1648004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 0x7f1648005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0cab90 con 0x7f1648003a30 2023-10-16T09:44:07.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164800bac0 con 0x7f1648004850 2023-10-16T09:44:07.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164800bc80 con 0x7f165c06cc60 2023-10-16T09:44:07.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f16615a6700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1648003a30 0x7f1648003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1660da5700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648004850 0x7f1648004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1661da7700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 0x7f1648005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f16615a6700 1 -- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 msgr2=0x7f1648005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f16615a6700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c06cc60 0x7f1648005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f16615a6700 1 -- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648004850 msgr2=0x7f1648004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f16615a6700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648004850 0x7f1648004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f16615a6700 1 -- 172.21.15.158:0/3920474983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16480035b0 con 0x7f1648003a30 2023-10-16T09:44:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.272+0000 7f1660da5700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648004850 0x7f1648004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f16615a6700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1648003a30 0x7f1648003e50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f164c007fd0 tx=0x7f1648005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2afe17aa73ca7fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0052d0 con 0x7f1648003a30 2023-10-16T09:44:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c005470 con 0x7f1648003a30 2023-10-16T09:44:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f1653fff700 1 -- 172.21.15.158:0/3920474983 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0038c0 con 0x7f1648003a30 2023-10-16T09:44:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f166380a700 1 -- 172.21.15.158:0/3920474983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1648003a30 msgr2=0x7f1648003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f166380a700 1 --2- 172.21.15.158:0/3920474983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1648003a30 0x7f1648003e50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f164c007fd0 tx=0x7f1648005880 comp rx=0 tx=0).stop 2023-10-16T09:44:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f1661da7700 1 -- 172.21.15.158:0/3920474983 reap_dead start 2023-10-16T09:44:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f166380a700 1 -- 172.21.15.158:0/3920474983 shutdown_connections 2023-10-16T09:44:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f166380a700 1 -- 172.21.15.158:0/3920474983 >> 172.21.15.158:0/3920474983 conn(0x7f165c06af00 msgr2=0x7f165c0cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f166380a700 1 -- 172.21.15.158:0/3920474983 shutdown_connections 2023-10-16T09:44:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.273+0000 7f166380a700 1 -- 172.21.15.158:0/3920474983 wait complete. 2023-10-16T09:44:07.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.274+0000 7f166380a700 1 Processor -- start 2023-10-16T09:44:07.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f166380a700 1 -- start start 2023-10-16T09:44:07.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f166380a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648003a30 0x7f165c0b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f166380a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f165c1f6910 0x7f165c1f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f166380a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c1f6d90 0x7f165c1f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f166380a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0b32e0 con 0x7f165c1f6910 2023-10-16T09:44:07.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f166380a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c1f0a80 con 0x7f1648003a30 2023-10-16T09:44:07.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f166380a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0b26d0 con 0x7f165c1f6d90 2023-10-16T09:44:07.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f1660da5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f165c1f6910 0x7f165c1f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f1660da5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f165c1f6910 0x7f165c1f0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47262/0 (socket says 172.21.15.158:47262) 2023-10-16T09:44:07.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f1660da5700 1 -- 172.21.15.158:0/1283330356 learned_addr learned my addr 172.21.15.158:0/1283330356 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:07.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.275+0000 7f16615a6700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648003a30 0x7f165c0b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1661da7700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c1f6d90 0x7f165c1f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1660da5700 1 -- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c1f6d90 msgr2=0x7f165c1f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1660da5700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c1f6d90 0x7f165c1f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1660da5700 1 -- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648003a30 msgr2=0x7f165c0b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1660da5700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648003a30 0x7f165c0b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1660da5700 1 -- 172.21.15.158:0/1283330356 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c0b29b0 con 0x7f165c1f6910 2023-10-16T09:44:07.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f16615a6700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648003a30 0x7f165c0b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:07.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1661da7700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c1f6d90 0x7f165c1f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:07.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1660da5700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f165c1f6910 0x7f165c1f0000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f165800e820 tx=0x7f165800ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1651ffb700 1 -- 172.21.15.158:0/1283330356 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165800c620 con 0x7f165c1f6910 2023-10-16T09:44:07.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1651ffb700 1 -- 172.21.15.158:0/1283330356 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165800c780 con 0x7f165c1f6910 2023-10-16T09:44:07.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c0b2c90 con 0x7f165c1f6910 2023-10-16T09:44:07.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f1651ffb700 1 -- 172.21.15.158:0/1283330356 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165800cb30 con 0x7f165c1f6910 2023-10-16T09:44:07.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.276+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165c0b3cb0 con 0x7f165c1f6910 2023-10-16T09:44:07.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.277+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165c0681b0 con 0x7f165c1f6910 2023-10-16T09:44:07.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.277+0000 7f1651ffb700 1 -- 172.21.15.158:0/1283330356 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f165800cc90 con 0x7f165c1f6910 2023-10-16T09:44:07.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.278+0000 7f1651ffb700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f164006af00 0x7f164006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.278+0000 7f1651ffb700 1 -- 172.21.15.158:0/1283330356 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1658089750 con 0x7f165c1f6910 2023-10-16T09:44:07.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.278+0000 7f16615a6700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f164006af00 0x7f164006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.278+0000 7f16615a6700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f164006af00 0x7f164006d3c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f164c007fd0 tx=0x7f164c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.282+0000 7f1651ffb700 1 -- 172.21.15.158:0/1283330356 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1658054510 con 0x7f165c1f6910 2023-10-16T09:44:07.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.475+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f165c0cdf80 con 0x7f165c1f6910 2023-10-16T09:44:07.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.477+0000 7f1651ffb700 1 -- 172.21.15.158:0/1283330356 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1658057b30 con 0x7f165c1f6910 2023-10-16T09:44:07.479 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:07.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f164006af00 msgr2=0x7f164006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f164006af00 0x7f164006d3c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f164c007fd0 tx=0x7f164c009030 comp rx=0 tx=0).stop 2023-10-16T09:44:07.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f165c1f6910 msgr2=0x7f165c1f0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f165c1f6910 0x7f165c1f0000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f165800e820 tx=0x7f165800ebe0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 shutdown_connections 2023-10-16T09:44:07.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f164006af00 0x7f164006d3c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f165c1f6d90 0x7f165c1f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1648003a30 0x7f165c0b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 --2- 172.21.15.158:0/1283330356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f165c1f6910 0x7f165c1f0000 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 >> 172.21.15.158:0/1283330356 conn(0x7f165c06af00 msgr2=0x7f165c0cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:07.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 shutdown_connections 2023-10-16T09:44:07.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.479+0000 7f166380a700 1 -- 172.21.15.158:0/1283330356 wait complete. 2023-10-16T09:44:07.493 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:07.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.616+0000 7fe4504bc700 1 Processor -- start 2023-10-16T09:44:07.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.616+0000 7fe4504bc700 1 -- start start 2023-10-16T09:44:07.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe4504bc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe44806ec50 0x7fe4480c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe4504bc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44806c220 con 0x7fe448143790 2023-10-16T09:44:07.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe4504bc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44806aa70 con 0x7fe4480c50a0 2023-10-16T09:44:07.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe4504bc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe448143b70 con 0x7fe44806ec50 2023-10-16T09:44:07.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe44ea59700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe44806ec50 0x7fe4480c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe44e258700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe448143790 0x7fe44806bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47442/0 (socket says 172.21.15.158:47442) 2023-10-16T09:44:07.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe44ea59700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe44806ec50 0x7fe4480c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47270/0 (socket says 172.21.15.158:47270) 2023-10-16T09:44:07.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.617+0000 7fe44e258700 1 -- 172.21.15.158:0/2587226419 learned_addr learned my addr 172.21.15.158:0/2587226419 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:07.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44ea59700 1 -- 172.21.15.158:0/2587226419 >> v1:172.21.15.172:6789/0 conn(0x7fe4480c50a0 legacy=0x7fe44806e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44ea59700 1 -- 172.21.15.158:0/2587226419 >> v1:172.21.15.158:6789/0 conn(0x7fe448143790 legacy=0x7fe44806bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44ea59700 1 -- 172.21.15.158:0/2587226419 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44806ad50 con 0x7fe44806ec50 2023-10-16T09:44:07.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44d256700 1 -- 172.21.15.158:0/2587226419 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025002383 0 0) 0x7fe44806c220 con 0x7fe448143790 2023-10-16T09:44:07.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44ea59700 1 --2- 172.21.15.158:0/2587226419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44806ec50 0x7fe4480c5520 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe440009d90 tx=0x7fe44000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac13952bc338c48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44d256700 1 -- 172.21.15.158:0/2587226419 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44000e070 con 0x7fe44806ec50 2023-10-16T09:44:07.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44d256700 1 -- 172.21.15.158:0/2587226419 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44000bb90 con 0x7fe44806ec50 2023-10-16T09:44:07.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.618+0000 7fe44d256700 1 -- 172.21.15.158:0/2587226419 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440003a20 con 0x7fe44806ec50 2023-10-16T09:44:07.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.619+0000 7fe4504bc700 1 -- 172.21.15.158:0/2587226419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44806ec50 msgr2=0x7fe4480c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.619+0000 7fe4504bc700 1 --2- 172.21.15.158:0/2587226419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44806ec50 0x7fe4480c5520 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe440009d90 tx=0x7fe44000b330 comp rx=0 tx=0).stop 2023-10-16T09:44:07.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.619+0000 7fe4504bc700 1 -- 172.21.15.158:0/2587226419 shutdown_connections 2023-10-16T09:44:07.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.619+0000 7fe4504bc700 1 --2- 172.21.15.158:0/2587226419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44806ec50 0x7fe4480c5520 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.619+0000 7fe4504bc700 1 -- 172.21.15.158:0/2587226419 >> 172.21.15.158:0/2587226419 conn(0x7fe4480cc5b0 msgr2=0x7fe4480cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:07.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.619+0000 7fe4504bc700 1 -- 172.21.15.158:0/2587226419 shutdown_connections 2023-10-16T09:44:07.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.619+0000 7fe4504bc700 1 -- 172.21.15.158:0/2587226419 wait complete. 2023-10-16T09:44:07.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.620+0000 7fe4504bc700 1 Processor -- start 2023-10-16T09:44:07.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.620+0000 7fe4504bc700 1 -- start start 2023-10-16T09:44:07.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44806ec50 0x7fe4480713f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4480c50a0 0x7fe44814aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe448143790 0x7fe44814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4481373f0 con 0x7fe4480c50a0 2023-10-16T09:44:07.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44814cc90 con 0x7fe448143790 2023-10-16T09:44:07.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44814cf70 con 0x7fe44806ec50 2023-10-16T09:44:07.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4480c50a0 0x7fe44814aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4480c50a0 0x7fe44814aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47274/0 (socket says 172.21.15.158:47274) 2023-10-16T09:44:07.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 -- 172.21.15.158:0/1306396977 learned_addr learned my addr 172.21.15.158:0/1306396977 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:07.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44e258700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44806ec50 0x7fe4480713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44ea59700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe448143790 0x7fe44814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 -- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44806ec50 msgr2=0x7fe4480713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44806ec50 0x7fe4480713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 -- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe448143790 msgr2=0x7fe44814b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe448143790 0x7fe44814b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 -- 172.21.15.158:0/1306396977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe440009a70 con 0x7fe4480c50a0 2023-10-16T09:44:07.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44ea59700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe448143790 0x7fe44814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:07.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44e258700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44806ec50 0x7fe4480713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:07.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe44da57700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4480c50a0 0x7fe44814aed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c4a0 tx=0x7fe44400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe43effd700 1 -- 172.21.15.158:0/1306396977 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4440054e0 con 0x7fe4480c50a0 2023-10-16T09:44:07.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe43effd700 1 -- 172.21.15.158:0/1306396977 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe444003680 con 0x7fe4480c50a0 2023-10-16T09:44:07.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe448136da0 con 0x7fe4480c50a0 2023-10-16T09:44:07.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe43effd700 1 -- 172.21.15.158:0/1306396977 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444003a30 con 0x7fe4480c50a0 2023-10-16T09:44:07.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.621+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44814c8f0 con 0x7fe4480c50a0 2023-10-16T09:44:07.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.622+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe448074350 con 0x7fe4480c50a0 2023-10-16T09:44:07.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.623+0000 7fe43effd700 1 -- 172.21.15.158:0/1306396977 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe444003b90 con 0x7fe4480c50a0 2023-10-16T09:44:07.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.623+0000 7fe43effd700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe42c06af00 0x7fe42c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.623+0000 7fe43effd700 1 -- 172.21.15.158:0/1306396977 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe444088930 con 0x7fe4480c50a0 2023-10-16T09:44:07.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.623+0000 7fe44e258700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe42c06af00 0x7fe42c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.623+0000 7fe44e258700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe42c06af00 0x7fe42c06d3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe438007fd0 tx=0x7fe438009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.627+0000 7fe43effd700 1 -- 172.21.15.158:0/1306396977 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe444053780 con 0x7fe4480c50a0 2023-10-16T09:44:07.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.823+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4480681b0 con 0x7fe4480c50a0 2023-10-16T09:44:07.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.823+0000 7fe43effd700 1 -- 172.21.15.158:0/1306396977 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe444056da0 con 0x7fe4480c50a0 2023-10-16T09:44:07.825 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:07.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe42c06af00 msgr2=0x7fe42c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe42c06af00 0x7fe42c06d3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe438007fd0 tx=0x7fe438009040 comp rx=0 tx=0).stop 2023-10-16T09:44:07.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4480c50a0 msgr2=0x7fe44814aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4480c50a0 0x7fe44814aed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c4a0 tx=0x7fe44400c860 comp rx=0 tx=0).stop 2023-10-16T09:44:07.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 shutdown_connections 2023-10-16T09:44:07.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe448143790 0x7fe44814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe42c06af00 0x7fe42c06d3c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44806ec50 0x7fe4480713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 --2- 172.21.15.158:0/1306396977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4480c50a0 0x7fe44814aed0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.825+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 >> 172.21.15.158:0/1306396977 conn(0x7fe4480cc5b0 msgr2=0x7fe448073c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:07.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.826+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 shutdown_connections 2023-10-16T09:44:07.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.826+0000 7fe4504bc700 1 -- 172.21.15.158:0/1306396977 wait complete. 2023-10-16T09:44:07.840 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:07.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.962+0000 7fe3c29f4700 1 Processor -- start 2023-10-16T09:44:07.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.962+0000 7fe3c29f4700 1 -- start start 2023-10-16T09:44:07.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3c29f4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 0x7fe3bc15e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3c29f4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe3bc071600 0x7fe3bc06c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3c29f4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc15a870 con 0x7fe3bc071a80 2023-10-16T09:44:07.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3c29f4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc15eed0 con 0x7fe3bc071600 2023-10-16T09:44:07.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3c29f4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cab90 con 0x7fe3bc15a490 2023-10-16T09:44:07.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3c0f91700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe3bc071a80 0x7fe3bc0cd280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47460/0 (socket says 172.21.15.158:47460) 2023-10-16T09:44:07.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3c0f91700 1 -- 172.21.15.158:0/2111031257 learned_addr learned my addr 172.21.15.158:0/2111031257 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:07.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3bb7fe700 1 --2- 172.21.15.158:0/2111031257 >> v2:172.21.15.172:3300/0 conn(0x7fe3bc071600 0x7fe3bc06c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 0x7fe3bc15e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216302285 0 0) 0x7fe3bc15a870 con 0x7fe3bc071a80 2023-10-16T09:44:07.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.963+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a40035b0 con 0x7fe3bc071a80 2023-10-16T09:44:07.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2958056304 0 0) 0x7fe3a40035b0 con 0x7fe3bc071a80 2023-10-16T09:44:07.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 msgr2=0x7fe3bc15e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 0x7fe3bc15e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 >> v2:172.21.15.172:3300/0 conn(0x7fe3bc071600 msgr2=0x7fe3bc06c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 --2- 172.21.15.158:0/2111031257 >> v2:172.21.15.172:3300/0 conn(0x7fe3bc071600 0x7fe3bc06c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0cae70 con 0x7fe3bc071a80 2023-10-16T09:44:07.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe3b40019f0 con 0x7fe3bc071a80 2023-10-16T09:44:07.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 >> v1:172.21.15.158:6789/0 conn(0x7fe3bc071a80 legacy=0x7fe3bc0cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3a4003a00 0x7fe3a4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a4004850 0x7fe3a4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 0x7fe3a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc15eed0 con 0x7fe3a4003a00 2023-10-16T09:44:07.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cab90 con 0x7fe3a4004850 2023-10-16T09:44:07.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a400ba10 con 0x7fe3bc15a490 2023-10-16T09:44:07.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bb7fe700 1 --2- 172.21.15.158:0/2111031257 >> v2:172.21.15.172:3300/0 conn(0x7fe3bc071600 0x7fe3bc06c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:07.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3a4003a00 0x7fe3a4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3c0f91700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 0x7fe3a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bb7fe700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a4004850 0x7fe3a4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bbfff700 1 -- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 msgr2=0x7fe3a4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc15a490 0x7fe3a4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bbfff700 1 -- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a4004850 msgr2=0x7fe3a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:07.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a4004850 0x7fe3a4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bbfff700 1 -- 172.21.15.158:0/2111031257 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40035b0 con 0x7fe3a4003a00 2023-10-16T09:44:07.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bb7fe700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a4004850 0x7fe3a4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:07.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3a4003a00 0x7fe3a4003e20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009b40 tx=0x7fe3a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389c307049f3a4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000e6c0 con 0x7fe3a4003a00 2023-10-16T09:44:07.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b000e820 con 0x7fe3a4003a00 2023-10-16T09:44:07.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.964+0000 7fe3baffd700 1 -- 172.21.15.158:0/2111031257 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000ebd0 con 0x7fe3a4003a00 2023-10-16T09:44:07.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.965+0000 7fe3c29f4700 1 -- 172.21.15.158:0/2111031257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3a4003a00 msgr2=0x7fe3a4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.965+0000 7fe3c29f4700 1 --2- 172.21.15.158:0/2111031257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3a4003a00 0x7fe3a4003e20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009b40 tx=0x7fe3a4005880 comp rx=0 tx=0).stop 2023-10-16T09:44:07.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.965+0000 7fe3c0f91700 1 -- 172.21.15.158:0/2111031257 reap_dead start 2023-10-16T09:44:07.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.965+0000 7fe3c29f4700 1 -- 172.21.15.158:0/2111031257 shutdown_connections 2023-10-16T09:44:07.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.965+0000 7fe3c29f4700 1 -- 172.21.15.158:0/2111031257 >> 172.21.15.158:0/2111031257 conn(0x7fe3bc06af00 msgr2=0x7fe3bc0cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:07.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.965+0000 7fe3c29f4700 1 -- 172.21.15.158:0/2111031257 shutdown_connections 2023-10-16T09:44:07.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.965+0000 7fe3c29f4700 1 -- 172.21.15.158:0/2111031257 wait complete. 2023-10-16T09:44:07.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.966+0000 7fe3c29f4700 1 Processor -- start 2023-10-16T09:44:07.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.966+0000 7fe3c29f4700 1 -- start start 2023-10-16T09:44:07.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c29f4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3a4003a00 0x7fe3bc0aa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c29f4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc1f73e0 0x7fe3bc14e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c29f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3bc0af3f0 0x7fe3bc0af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c29f4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0afdb0 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c29f4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0aae70 con 0x7fe3bc1f73e0 2023-10-16T09:44:07.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c29f4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0aa4a0 con 0x7fe3a4003a00 2023-10-16T09:44:07.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3bc0af3f0 0x7fe3bc0af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3bc0af3f0 0x7fe3bc0af870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47296/0 (socket says 172.21.15.158:47296) 2023-10-16T09:44:07.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 -- 172.21.15.158:0/1236175472 learned_addr learned my addr 172.21.15.158:0/1236175472 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:07.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3a4003a00 0x7fe3bc0aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 -- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3a4003a00 msgr2=0x7fe3bc0aa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3a4003a00 0x7fe3bc0aa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3bb7fe700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc1f73e0 0x7fe3bc14e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 -- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc1f73e0 msgr2=0x7fe3bc14e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:07.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc1f73e0 0x7fe3bc14e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:07.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.967+0000 7fe3c0f91700 1 -- 172.21.15.158:0/1236175472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0a9890 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.968+0000 7fe3c0f91700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3bc0af3f0 0x7fe3bc0af870 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0d00f0 tx=0x7fe3b400a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.968+0000 7fe3b8ff9700 1 -- 172.21.15.158:0/1236175472 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400ad50 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.968+0000 7fe3b8ff9700 1 -- 172.21.15.158:0/1236175472 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b40133f0 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.968+0000 7fe3b8ff9700 1 -- 172.21.15.158:0/1236175472 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40137a0 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.968+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc0a9b70 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.968+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc0aa130 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.969+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc0aa630 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.969+0000 7fe3b8ff9700 1 -- 172.21.15.158:0/1236175472 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe3b4013900 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.970+0000 7fe3b8ff9700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe3a806af00 0x7fe3a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:07.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.970+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe3a806af00 0x7fe3a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:07.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.970+0000 7fe3b8ff9700 1 -- 172.21.15.158:0/1236175472 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe3b408a1e0 con 0x7fe3bc0af3f0 2023-10-16T09:44:07.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.970+0000 7fe3bbfff700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe3a806af00 0x7fe3a806d3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00090b0 tx=0x7fe3b0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:07.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:07.974+0000 7fe3b8ff9700 1 -- 172.21.15.158:0/1236175472 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3b4054fa0 con 0x7fe3bc0af3f0 2023-10-16T09:44:08.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.165+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3bc06f190 con 0x7fe3bc0af3f0 2023-10-16T09:44:08.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.165+0000 7fe3b8ff9700 1 -- 172.21.15.158:0/1236175472 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe3b40585c0 con 0x7fe3bc0af3f0 2023-10-16T09:44:08.167 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:08.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe3a806af00 msgr2=0x7fe3a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe3a806af00 0x7fe3a806d3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00090b0 tx=0x7fe3b0008030 comp rx=0 tx=0).stop 2023-10-16T09:44:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3bc0af3f0 msgr2=0x7fe3bc0af870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3bc0af3f0 0x7fe3bc0af870 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0d00f0 tx=0x7fe3b400a640 comp rx=0 tx=0).stop 2023-10-16T09:44:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 shutdown_connections 2023-10-16T09:44:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3bc0af3f0 0x7fe3bc0af870 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe3a806af00 0x7fe3a806d3c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3a4003a00 0x7fe3bc0aa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 --2- 172.21.15.158:0/1236175472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc1f73e0 0x7fe3bc14e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.167+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 >> 172.21.15.158:0/1236175472 conn(0x7fe3bc06af00 msgr2=0x7fe3bc0c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.168+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 shutdown_connections 2023-10-16T09:44:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.168+0000 7fe3c29f4700 1 -- 172.21.15.158:0/1236175472 wait complete. 2023-10-16T09:44:08.181 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:08.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.303+0000 7f5e98c60700 1 Processor -- start 2023-10-16T09:44:08.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.303+0000 7f5e98c60700 1 -- start start 2023-10-16T09:44:08.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e98c60700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5e9415d910 0x7f5e940cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e98c60700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9406a8d0 con 0x7f5e940700a0 2023-10-16T09:44:08.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e98c60700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e94161e50 con 0x7f5e940cc9e0 2023-10-16T09:44:08.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e98c60700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e94072970 con 0x7f5e9415d910 2023-10-16T09:44:08.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e92d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5e940700a0 0x7f5e9406f590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47480/0 (socket says 172.21.15.158:47480) 2023-10-16T09:44:08.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e92d9d700 1 -- 172.21.15.158:0/3834218414 learned_addr learned my addr 172.21.15.158:0/3834218414 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:08.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e9259c700 1 --2- 172.21.15.158:0/3834218414 >> v2:172.21.15.172:3300/0 conn(0x7f5e9415d910 0x7f5e940cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e9159a700 1 -- 172.21.15.158:0/3834218414 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386481898 0 0) 0x7f5e9406a8d0 con 0x7f5e940700a0 2023-10-16T09:44:08.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.304+0000 7f5e9159a700 1 -- 172.21.15.158:0/3834218414 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e7c0035b0 con 0x7f5e940700a0 2023-10-16T09:44:08.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9159a700 1 -- 172.21.15.158:0/3834218414 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157160791 0 0) 0x7f5e7c0035b0 con 0x7f5e940700a0 2023-10-16T09:44:08.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9259c700 1 -- 172.21.15.158:0/3834218414 >> v1:172.21.15.172:6789/0 conn(0x7f5e940cc9e0 legacy=0x7f5e9406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9259c700 1 -- 172.21.15.158:0/3834218414 >> v1:172.21.15.158:6789/0 conn(0x7f5e940700a0 legacy=0x7f5e9406f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9259c700 1 -- 172.21.15.158:0/3834218414 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9406fca0 con 0x7f5e9415d910 2023-10-16T09:44:08.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9259c700 1 --2- 172.21.15.158:0/3834218414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e9415d910 0x7f5e940cdf70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009840 tx=0x7f5e88009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b267573873971fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:08.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9159a700 1 -- 172.21.15.158:0/3834218414 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0019f0 con 0x7f5e9415d910 2023-10-16T09:44:08.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9159a700 1 -- 172.21.15.158:0/3834218414 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e88004030 con 0x7f5e9415d910 2023-10-16T09:44:08.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e9159a700 1 -- 172.21.15.158:0/3834218414 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880043c0 con 0x7f5e9415d910 2023-10-16T09:44:08.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e98c60700 1 -- 172.21.15.158:0/3834218414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e9415d910 msgr2=0x7f5e940cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.305+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3834218414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e9415d910 0x7f5e940cdf70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009840 tx=0x7f5e88009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:08.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.306+0000 7f5e98c60700 1 -- 172.21.15.158:0/3834218414 shutdown_connections 2023-10-16T09:44:08.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.306+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3834218414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e9415d910 0x7f5e940cdf70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.306+0000 7f5e98c60700 1 -- 172.21.15.158:0/3834218414 >> 172.21.15.158:0/3834218414 conn(0x7f5e940b1e90 msgr2=0x7f5e940b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:08.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.306+0000 7f5e98c60700 1 -- 172.21.15.158:0/3834218414 shutdown_connections 2023-10-16T09:44:08.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.306+0000 7f5e98c60700 1 -- 172.21.15.158:0/3834218414 wait complete. 2023-10-16T09:44:08.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.307+0000 7f5e98c60700 1 Processor -- start 2023-10-16T09:44:08.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.307+0000 7f5e98c60700 1 -- start start 2023-10-16T09:44:08.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e98c60700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e940700a0 0x7f5e9414f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e98c60700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e940cc9e0 0x7f5e94160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e98c60700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e9415d910 0x7f5e94160880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e98c60700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e940b3a50 con 0x7f5e940700a0 2023-10-16T09:44:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e98c60700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9415f580 con 0x7f5e940cc9e0 2023-10-16T09:44:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e98c60700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9415f860 con 0x7f5e9415d910 2023-10-16T09:44:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e940700a0 0x7f5e9414f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e940700a0 0x7f5e9414f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47306/0 (socket says 172.21.15.158:47306) 2023-10-16T09:44:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 -- 172.21.15.158:0/3742032066 learned_addr learned my addr 172.21.15.158:0/3742032066 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 -- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e9415d910 msgr2=0x7f5e94160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e92d9d700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e9415d910 0x7f5e94160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e91d9b700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e940cc9e0 0x7f5e94160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e9415d910 0x7f5e94160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 -- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e940cc9e0 msgr2=0x7f5e94160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e940cc9e0 0x7f5e94160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 -- 172.21.15.158:0/3742032066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880094a0 con 0x7f5e940700a0 2023-10-16T09:44:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e9259c700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e940700a0 0x7f5e9414f5c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800a010 tx=0x7f5e88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.308+0000 7f5e82ffd700 1 -- 172.21.15.158:0/3742032066 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88003e30 con 0x7f5e940700a0 2023-10-16T09:44:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.309+0000 7f5e82ffd700 1 -- 172.21.15.158:0/3742032066 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8801f070 con 0x7f5e940700a0 2023-10-16T09:44:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.309+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9415fb40 con 0x7f5e940700a0 2023-10-16T09:44:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.309+0000 7f5e82ffd700 1 -- 172.21.15.158:0/3742032066 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88011870 con 0x7f5e940700a0 2023-10-16T09:44:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.309+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e941f2b20 con 0x7f5e940700a0 2023-10-16T09:44:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.309+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e94160060 con 0x7f5e940700a0 2023-10-16T09:44:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.310+0000 7f5e82ffd700 1 -- 172.21.15.158:0/3742032066 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5e88021030 con 0x7f5e940700a0 2023-10-16T09:44:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.310+0000 7f5e82ffd700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e7406af50 0x7f5e7406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.310+0000 7f5e82ffd700 1 -- 172.21.15.158:0/3742032066 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5e8808abd0 con 0x7f5e940700a0 2023-10-16T09:44:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.310+0000 7f5e91d9b700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e7406af50 0x7f5e7406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.311+0000 7f5e91d9b700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e7406af50 0x7f5e7406d410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5e9406b230 tx=0x7f5e84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.315+0000 7f5e82ffd700 1 -- 172.21.15.158:0/3742032066 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5e94160060 con 0x7f5e940700a0 2023-10-16T09:44:08.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.508+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5e94160060 con 0x7f5e940700a0 2023-10-16T09:44:08.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.508+0000 7f5e82ffd700 1 -- 172.21.15.158:0/3742032066 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5e94160060 con 0x7f5e940700a0 2023-10-16T09:44:08.510 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:08.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e7406af50 msgr2=0x7f5e7406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e7406af50 0x7f5e7406d410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5e9406b230 tx=0x7f5e84009040 comp rx=0 tx=0).stop 2023-10-16T09:44:08.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e940700a0 msgr2=0x7f5e9414f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e940700a0 0x7f5e9414f5c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800a010 tx=0x7f5e88004750 comp rx=0 tx=0).stop 2023-10-16T09:44:08.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 shutdown_connections 2023-10-16T09:44:08.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f5e7406af50 0x7f5e7406d410 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e9415d910 0x7f5e94160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e940700a0 0x7f5e9414f5c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 --2- 172.21.15.158:0/3742032066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e940cc9e0 0x7f5e94160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 >> 172.21.15.158:0/3742032066 conn(0x7f5e940b1e90 msgr2=0x7f5e9406ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:08.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 shutdown_connections 2023-10-16T09:44:08.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.510+0000 7f5e98c60700 1 -- 172.21.15.158:0/3742032066 wait complete. 2023-10-16T09:44:08.524 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:08.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.643+0000 7f49bd397700 1 Processor -- start 2023-10-16T09:44:08.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.643+0000 7f49bd397700 1 -- start start 2023-10-16T09:44:08.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49bd397700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806b0a0 0x7f49b815f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49bd397700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49b80cb010 0x7f49b815bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49bd397700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b8072700 con 0x7f49b806d820 2023-10-16T09:44:08.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49bd397700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b8071910 con 0x7f49b80cb010 2023-10-16T09:44:08.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49bd397700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b806b480 con 0x7f49b806b0a0 2023-10-16T09:44:08.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49b80cb010 0x7f49b815bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49b80cb010 0x7f49b815bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47318/0 (socket says 172.21.15.158:47318) 2023-10-16T09:44:08.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 -- 172.21.15.158:0/1731774806 learned_addr learned my addr 172.21.15.158:0/1731774806 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:08.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b6ffd700 1 --2- 172.21.15.158:0/1731774806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806b0a0 0x7f49b815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 -- 172.21.15.158:0/1731774806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806b0a0 msgr2=0x7f49b815f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:08.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 --2- 172.21.15.158:0/1731774806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806b0a0 0x7f49b815f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 -- 172.21.15.158:0/1731774806 >> v1:172.21.15.172:6789/0 conn(0x7f49b806d820 legacy=0x7f49b80ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 -- 172.21.15.158:0/1731774806 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b80cca90 con 0x7f49b80cb010 2023-10-16T09:44:08.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b77fe700 1 --2- 172.21.15.158:0/1731774806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b80cb010 0x7f49b815bcd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009e50 tx=0x7f49b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446f9f6fe6db7186 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:08.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.644+0000 7f49b5ffb700 1 -- 172.21.15.158:0/1731774806 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b000e070 con 0x7f49b80cb010 2023-10-16T09:44:08.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49b5ffb700 1 -- 172.21.15.158:0/1731774806 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b000b9c0 con 0x7f49b80cb010 2023-10-16T09:44:08.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49b5ffb700 1 -- 172.21.15.158:0/1731774806 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b000bd70 con 0x7f49b80cb010 2023-10-16T09:44:08.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 -- 172.21.15.158:0/1731774806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b80cb010 msgr2=0x7f49b815bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 --2- 172.21.15.158:0/1731774806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b80cb010 0x7f49b815bcd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009e50 tx=0x7f49b000b330 comp rx=0 tx=0).stop 2023-10-16T09:44:08.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 -- 172.21.15.158:0/1731774806 shutdown_connections 2023-10-16T09:44:08.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 --2- 172.21.15.158:0/1731774806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b80cb010 0x7f49b815bcd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 --2- 172.21.15.158:0/1731774806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806b0a0 0x7f49b815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 -- 172.21.15.158:0/1731774806 >> 172.21.15.158:0/1731774806 conn(0x7f49b80d02b0 msgr2=0x7f49b80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:08.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 -- 172.21.15.158:0/1731774806 shutdown_connections 2023-10-16T09:44:08.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.645+0000 7f49bd397700 1 -- 172.21.15.158:0/1731774806 wait complete. 2023-10-16T09:44:08.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.647+0000 7f49bd397700 1 Processor -- start 2023-10-16T09:44:08.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.647+0000 7f49bd397700 1 -- start start 2023-10-16T09:44:08.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49bd397700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b806b0a0 0x7f49b815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49bd397700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806d820 0x7f49b8150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49bd397700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b80cb010 0x7f49b80c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49bd397700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b815dc10 con 0x7f49b806b0a0 2023-10-16T09:44:08.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49bd397700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b81f1d40 con 0x7f49b80cb010 2023-10-16T09:44:08.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49bd397700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b81f1a60 con 0x7f49b806d820 2023-10-16T09:44:08.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b806b0a0 0x7f49b815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b806b0a0 0x7f49b815d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47320/0 (socket says 172.21.15.158:47320) 2023-10-16T09:44:08.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 -- 172.21.15.158:0/3604329372 learned_addr learned my addr 172.21.15.158:0/3604329372 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:08.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b77fe700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b80cb010 0x7f49b80c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b67fc700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806d820 0x7f49b8150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 -- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806d820 msgr2=0x7f49b8150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806d820 0x7f49b8150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 -- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b80cb010 msgr2=0x7f49b80c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b80cb010 0x7f49b80c7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 -- 172.21.15.158:0/3604329372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0009b30 con 0x7f49b806b0a0 2023-10-16T09:44:08.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b77fe700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b80cb010 0x7f49b80c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:08.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49b6ffd700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b806b0a0 0x7f49b815d6d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00e810 tx=0x7f49ac00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:08.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f499f7fe700 1 -- 172.21.15.158:0/3604329372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac013610 con 0x7f49b806b0a0 2023-10-16T09:44:08.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f499f7fe700 1 -- 172.21.15.158:0/3604329372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ac013770 con 0x7f49b806b0a0 2023-10-16T09:44:08.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b815cbf0 con 0x7f49b806b0a0 2023-10-16T09:44:08.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.648+0000 7f499f7fe700 1 -- 172.21.15.158:0/3604329372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac013b20 con 0x7f49b806b0a0 2023-10-16T09:44:08.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.649+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b815d1b0 con 0x7f49b806b0a0 2023-10-16T09:44:08.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.649+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b815d370 con 0x7f49b806b0a0 2023-10-16T09:44:08.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.650+0000 7f499f7fe700 1 -- 172.21.15.158:0/3604329372 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f49ac013c80 con 0x7f49b806b0a0 2023-10-16T09:44:08.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.650+0000 7f499f7fe700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f49a406af00 0x7f49a406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:08.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.650+0000 7f499f7fe700 1 -- 172.21.15.158:0/3604329372 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f49ac089630 con 0x7f49b806b0a0 2023-10-16T09:44:08.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.650+0000 7f49b67fc700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f49a406af00 0x7f49a406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:08.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.651+0000 7f49b67fc700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f49a406af00 0x7f49a406d3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f49b80b2490 tx=0x7f49a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:08.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.655+0000 7f499f7fe700 1 -- 172.21.15.158:0/3604329372 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49ac054400 con 0x7f49b806b0a0 2023-10-16T09:44:08.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.848+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49b8072090 con 0x7f49b806b0a0 2023-10-16T09:44:08.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.849+0000 7f499f7fe700 1 -- 172.21.15.158:0/3604329372 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f49b8072090 con 0x7f49b806b0a0 2023-10-16T09:44:08.851 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:08.851 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":24,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:43:26.357500+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:08.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f49a406af00 msgr2=0x7f49a406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f49a406af00 0x7f49a406d3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f49b80b2490 tx=0x7f49a8009040 comp rx=0 tx=0).stop 2023-10-16T09:44:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b806b0a0 msgr2=0x7f49b815d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b806b0a0 0x7f49b815d6d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00e810 tx=0x7f49ac00ebd0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 shutdown_connections 2023-10-16T09:44:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b80cb010 0x7f49b80c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b806b0a0 0x7f49b815d6d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f49a406af00 0x7f49a406d3c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 --2- 172.21.15.158:0/3604329372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b806d820 0x7f49b8150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 >> 172.21.15.158:0/3604329372 conn(0x7f49b80d02b0 msgr2=0x7f49b80d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 shutdown_connections 2023-10-16T09:44:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.851+0000 7f49bd397700 1 -- 172.21.15.158:0/3604329372 wait complete. 2023-10-16T09:44:08.865 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:08.866 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:08.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.990+0000 7f326ac7e700 1 Processor -- start 2023-10-16T09:44:08.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.990+0000 7f326ac7e700 1 -- start start 2023-10-16T09:44:08.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f326ac7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 0x7f32640caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f326ac7e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32640cbd30 con 0x7f32640cc590 2023-10-16T09:44:09.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f326ac7e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32640cc1c0 con 0x7f326406afe0 2023-10-16T09:44:09.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f326ac7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326415de20 con 0x7f326415da40 2023-10-16T09:44:09.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f3263fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f32640cc590 0x7f326406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47504/0 (socket says 172.21.15.158:47504) 2023-10-16T09:44:09.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f3263fff700 1 -- 172.21.15.158:0/4200197714 learned_addr learned my addr 172.21.15.158:0/4200197714 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:09.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53917630 0 0) 0x7f32640cbd30 con 0x7f32640cc590 2023-10-16T09:44:09.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32540035b0 con 0x7f32640cc590 2023-10-16T09:44:09.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f3268a1a700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 0x7f32640caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 686906626 0 0) 0x7f32540035b0 con 0x7f32640cc590 2023-10-16T09:44:09.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 msgr2=0x7f32640caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:09.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 0x7f32640caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 >> v1:172.21.15.172:6789/0 conn(0x7f326406afe0 legacy=0x7f32640afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32640ce260 con 0x7f32640cc590 2023-10-16T09:44:09.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3250001670 con 0x7f32640cc590 2023-10-16T09:44:09.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 >> v1:172.21.15.158:6789/0 conn(0x7f32640cc590 legacy=0x7f326406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3254003a30 0x7f3254003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 0x7f3254004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3254004d90 0x7f3254005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326415de20 con 0x7f3254004d90 2023-10-16T09:44:09.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325400bac0 con 0x7f3254003a30 2023-10-16T09:44:09.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.991+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325400bc80 con 0x7f326415da40 2023-10-16T09:44:09.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f326921b700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3254004d90 0x7f3254005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f3268a1a700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3254003a30 0x7f3254003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f3263fff700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 0x7f3254004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f326921b700 1 -- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 msgr2=0x7f3254004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:09.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f326921b700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415da40 0x7f3254004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f326921b700 1 -- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3254003a30 msgr2=0x7f3254003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:09.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f326921b700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3254003a30 0x7f3254003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f326921b700 1 -- 172.21.15.158:0/4200197714 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540035b0 con 0x7f3254004d90 2023-10-16T09:44:09.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f326921b700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3254004d90 0x7f3254005210 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f32640cf500 tx=0x7f3254005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c63614f6f8abb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c001db0 con 0x7f3254004d90 2023-10-16T09:44:09.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c00cc40 con 0x7f3254004d90 2023-10-16T09:44:09.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.992+0000 7f32637fe700 1 -- 172.21.15.158:0/4200197714 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c003a00 con 0x7f3254004d90 2023-10-16T09:44:09.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.993+0000 7f326ac7e700 1 -- 172.21.15.158:0/4200197714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3254004d90 msgr2=0x7f3254005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.993+0000 7f326ac7e700 1 --2- 172.21.15.158:0/4200197714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3254004d90 0x7f3254005210 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f32640cf500 tx=0x7f3254005880 comp rx=0 tx=0).stop 2023-10-16T09:44:09.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.993+0000 7f326921b700 1 -- 172.21.15.158:0/4200197714 reap_dead start 2023-10-16T09:44:09.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.993+0000 7f326ac7e700 1 -- 172.21.15.158:0/4200197714 shutdown_connections 2023-10-16T09:44:09.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.993+0000 7f326ac7e700 1 -- 172.21.15.158:0/4200197714 >> 172.21.15.158:0/4200197714 conn(0x7f3264099a10 msgr2=0x7f32640d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:09.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.993+0000 7f326ac7e700 1 -- 172.21.15.158:0/4200197714 shutdown_connections 2023-10-16T09:44:09.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.993+0000 7f326ac7e700 1 -- 172.21.15.158:0/4200197714 wait complete. 2023-10-16T09:44:09.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.994+0000 7f326ac7e700 1 Processor -- start 2023-10-16T09:44:09.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.994+0000 7f326ac7e700 1 -- start start 2023-10-16T09:44:09.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326ac7e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3264160480 0x7f32640c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326ac7e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32640c4760 0x7f326415df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326ac7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415e4a0 0x7f3264160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326ac7e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32640c6240 con 0x7f32640c4760 2023-10-16T09:44:09.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326ac7e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326415e930 con 0x7f3264160480 2023-10-16T09:44:09.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326ac7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32640c5870 con 0x7f326415e4a0 2023-10-16T09:44:09.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32640c4760 0x7f326415df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32640c4760 0x7f326415df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47340/0 (socket says 172.21.15.158:47340) 2023-10-16T09:44:09.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 -- 172.21.15.158:0/3609006838 learned_addr learned my addr 172.21.15.158:0/3609006838 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:09.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326921b700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415e4a0 0x7f3264160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3268a1a700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3264160480 0x7f32640c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 -- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415e4a0 msgr2=0x7f3264160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415e4a0 0x7f3264160860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 -- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3264160480 msgr2=0x7f32640c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3264160480 0x7f32640c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 -- 172.21.15.158:0/3609006838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32640c4c60 con 0x7f32640c4760 2023-10-16T09:44:09.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3268a1a700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3264160480 0x7f32640c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:09.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f326921b700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415e4a0 0x7f3264160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:09.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.995+0000 7f3263fff700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32640c4760 0x7f326415df60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f32500072c0 tx=0x7f3250007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.996+0000 7f32617fa700 1 -- 172.21.15.158:0/3609006838 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325000f870 con 0x7f32640c4760 2023-10-16T09:44:09.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.996+0000 7f32617fa700 1 -- 172.21.15.158:0/3609006838 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325000f9d0 con 0x7f32640c4760 2023-10-16T09:44:09.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.996+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32640c4f40 con 0x7f32640c4760 2023-10-16T09:44:09.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.996+0000 7f32617fa700 1 -- 172.21.15.158:0/3609006838 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325000fd60 con 0x7f32640c4760 2023-10-16T09:44:09.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.996+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32640c5500 con 0x7f32640c4760 2023-10-16T09:44:09.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.997+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32640c5690 con 0x7f32640c4760 2023-10-16T09:44:09.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.997+0000 7f32617fa700 1 -- 172.21.15.158:0/3609006838 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f325001d460 con 0x7f32640c4760 2023-10-16T09:44:09.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.997+0000 7f32617fa700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f324806af30 0x7f324806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.998+0000 7f32617fa700 1 -- 172.21.15.158:0/3609006838 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3250089a90 con 0x7f32640c4760 2023-10-16T09:44:09.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.998+0000 7f3268a1a700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f324806af30 0x7f324806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:08.998+0000 7f3268a1a700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f324806af30 0x7f324806d3f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3258009c60 tx=0x7f3258008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.002+0000 7f32617fa700 1 -- 172.21.15.158:0/3609006838 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3250054720 con 0x7f32640c4760 2023-10-16T09:44:09.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.192+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32640681b0 con 0x7f32640c4760 2023-10-16T09:44:09.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.192+0000 7f32617fa700 1 -- 172.21.15.158:0/3609006838 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3250057d40 con 0x7f32640c4760 2023-10-16T09:44:09.194 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:09.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f324806af30 msgr2=0x7f324806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f324806af30 0x7f324806d3f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3258009c60 tx=0x7f3258008040 comp rx=0 tx=0).stop 2023-10-16T09:44:09.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32640c4760 msgr2=0x7f326415df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32640c4760 0x7f326415df60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f32500072c0 tx=0x7f3250007680 comp rx=0 tx=0).stop 2023-10-16T09:44:09.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 shutdown_connections 2023-10-16T09:44:09.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f324806af30 0x7f324806d3f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f326415e4a0 0x7f3264160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3264160480 0x7f32640c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 --2- 172.21.15.158:0/3609006838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32640c4760 0x7f326415df60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 >> 172.21.15.158:0/3609006838 conn(0x7f3264099a10 msgr2=0x7f32640cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:09.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 shutdown_connections 2023-10-16T09:44:09.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.194+0000 7f326ac7e700 1 -- 172.21.15.158:0/3609006838 wait complete. 2023-10-16T09:44:09.208 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:09.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.332+0000 7f7623fea700 1 Processor -- start 2023-10-16T09:44:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.332+0000 7f7623fea700 1 -- start start 2023-10-16T09:44:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7623fea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f761c15b590 0x7f761c0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7623fea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c0cbaf0 0x7f761c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7623fea700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0cf4c0 con 0x7f761c15f1d0 2023-10-16T09:44:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7623fea700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c0cc4c0 con 0x7f761c15b590 2023-10-16T09:44:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7623fea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c0b49e0 con 0x7f761c0cbaf0 2023-10-16T09:44:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7621d86700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f761c15b590 0x7f761c0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7622587700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c0cbaf0 0x7f761c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7621d86700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f761c15b590 0x7f761c0ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:48526/0 (socket says 172.21.15.158:48526) 2023-10-16T09:44:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7621585700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f761c15f1d0 0x7f761c0b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:51426/0 (socket says 172.21.15.158:51426) 2023-10-16T09:44:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.333+0000 7f7621d86700 1 -- 172.21.15.158:0/3746072486 learned_addr learned my addr 172.21.15.158:0/3746072486 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7621d86700 1 -- 172.21.15.158:0/3746072486 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c0cbaf0 msgr2=0x7f761c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7621d86700 1 --2- 172.21.15.158:0/3746072486 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c0cbaf0 0x7f761c0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7621d86700 1 -- 172.21.15.158:0/3746072486 >> v1:172.21.15.172:6789/0 conn(0x7f761c15f1d0 legacy=0x7f761c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7620d84700 1 -- 172.21.15.158:0/3746072486 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620343852 0 0) 0x7f761c0cf4c0 con 0x7f761c15f1d0 2023-10-16T09:44:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7621d86700 1 -- 172.21.15.158:0/3746072486 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c15f970 con 0x7f761c15b590 2023-10-16T09:44:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7621d86700 1 --2- 172.21.15.158:0/3746072486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15b590 0x7f761c0ce510 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7618009920 tx=0x7f7618009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e8b5fd4865aefb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7620d84700 1 -- 172.21.15.158:0/3746072486 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618011830 con 0x7f761c15b590 2023-10-16T09:44:09.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7620d84700 1 -- 172.21.15.158:0/3746072486 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618011990 con 0x7f761c15b590 2023-10-16T09:44:09.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7620d84700 1 -- 172.21.15.158:0/3746072486 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618011d20 con 0x7f761c15b590 2023-10-16T09:44:09.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7623fea700 1 -- 172.21.15.158:0/3746072486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15b590 msgr2=0x7f761c0ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7623fea700 1 --2- 172.21.15.158:0/3746072486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15b590 0x7f761c0ce510 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7618009920 tx=0x7f7618009c30 comp rx=0 tx=0).stop 2023-10-16T09:44:09.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7623fea700 1 -- 172.21.15.158:0/3746072486 shutdown_connections 2023-10-16T09:44:09.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7623fea700 1 --2- 172.21.15.158:0/3746072486 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c0cbaf0 0x7f761c0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7623fea700 1 --2- 172.21.15.158:0/3746072486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15b590 0x7f761c0ce510 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.334+0000 7f7623fea700 1 -- 172.21.15.158:0/3746072486 >> 172.21.15.158:0/3746072486 conn(0x7f761c06a8d0 msgr2=0x7f761c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:09.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.335+0000 7f7623fea700 1 -- 172.21.15.158:0/3746072486 shutdown_connections 2023-10-16T09:44:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.335+0000 7f7623fea700 1 -- 172.21.15.158:0/3746072486 wait complete. 2023-10-16T09:44:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.336+0000 7f7623fea700 1 Processor -- start 2023-10-16T09:44:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.336+0000 7f7623fea700 1 -- start start 2023-10-16T09:44:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.336+0000 7f7623fea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f761c0cbaf0 0x7f761c1f1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.336+0000 7f7623fea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c15b590 0x7f761c152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.336+0000 7f7623fea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15f1d0 0x7f761c154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.336+0000 7f7623fea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1f18c0 con 0x7f761c0cbaf0 2023-10-16T09:44:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7623fea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c153140 con 0x7f761c15f1d0 2023-10-16T09:44:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7623fea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1f08c0 con 0x7f761c15b590 2023-10-16T09:44:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f761c0cbaf0 0x7f761c1f1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f761c0cbaf0 0x7f761c1f1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47352/0 (socket says 172.21.15.158:47352) 2023-10-16T09:44:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 -- 172.21.15.158:0/941286227 learned_addr learned my addr 172.21.15.158:0/941286227 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7622587700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15f1d0 0x7f761c154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621585700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c15b590 0x7f761c152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 -- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c15b590 msgr2=0x7f761c152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c15b590 0x7f761c152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 -- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15f1d0 msgr2=0x7f761c154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15f1d0 0x7f761c154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 -- 172.21.15.158:0/941286227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7618009550 con 0x7f761c0cbaf0 2023-10-16T09:44:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7622587700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15f1d0 0x7f761c154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7621d86700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f761c0cbaf0 0x7f761c1f1380 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7618009870 tx=0x7f7618004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f76127fc700 1 -- 172.21.15.158:0/941286227 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180045d0 con 0x7f761c0cbaf0 2023-10-16T09:44:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f76127fc700 1 -- 172.21.15.158:0/941286227 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761801f070 con 0x7f761c0cbaf0 2023-10-16T09:44:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c154950 con 0x7f761c0cbaf0 2023-10-16T09:44:09.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f76127fc700 1 -- 172.21.15.158:0/941286227 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180038e0 con 0x7f761c0cbaf0 2023-10-16T09:44:09.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.337+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c1f0b80 con 0x7f761c0cbaf0 2023-10-16T09:44:09.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.338+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c0681b0 con 0x7f761c0cbaf0 2023-10-16T09:44:09.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.339+0000 7f76127fc700 1 -- 172.21.15.158:0/941286227 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7618003a40 con 0x7f761c0cbaf0 2023-10-16T09:44:09.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.339+0000 7f76127fc700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f760006af00 0x7f760006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.339+0000 7f76127fc700 1 -- 172.21.15.158:0/941286227 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7618021070 con 0x7f761c0cbaf0 2023-10-16T09:44:09.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.339+0000 7f7621585700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f760006af00 0x7f760006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.340+0000 7f7621585700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f760006af00 0x7f760006d3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f761c0d0b30 tx=0x7f760c007e70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.344+0000 7f76127fc700 1 -- 172.21.15.158:0/941286227 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7618055f60 con 0x7f761c0cbaf0 2023-10-16T09:44:09.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.539+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f761c0b07c0 con 0x7f761c0cbaf0 2023-10-16T09:44:09.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.539+0000 7f76127fc700 1 -- 172.21.15.158:0/941286227 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7618015020 con 0x7f761c0cbaf0 2023-10-16T09:44:09.541 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:09.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f760006af00 msgr2=0x7f760006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f760006af00 0x7f760006d3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f761c0d0b30 tx=0x7f760c007e70 comp rx=0 tx=0).stop 2023-10-16T09:44:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f761c0cbaf0 msgr2=0x7f761c1f1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f761c0cbaf0 0x7f761c1f1380 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7618009870 tx=0x7f7618004750 comp rx=0 tx=0).stop 2023-10-16T09:44:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 shutdown_connections 2023-10-16T09:44:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f761c15f1d0 0x7f761c154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f761c0cbaf0 0x7f761c1f1380 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f760006af00 0x7f760006d3c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 --2- 172.21.15.158:0/941286227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f761c15b590 0x7f761c152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 >> 172.21.15.158:0/941286227 conn(0x7f761c06a8d0 msgr2=0x7f761c0cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 shutdown_connections 2023-10-16T09:44:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.541+0000 7f7623fea700 1 -- 172.21.15.158:0/941286227 wait complete. 2023-10-16T09:44:09.555 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.677+0000 7f09ad5e7700 1 Processor -- start 2023-10-16T09:44:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.677+0000 7f09ad5e7700 1 -- start start 2023-10-16T09:44:09.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09ad5e7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09a806afe0 0x7f09a80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09ad5e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a80cc590 0x7f09a80afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09ad5e7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a80cbd30 con 0x7f09a815da40 2023-10-16T09:44:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09ad5e7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a80cc1c0 con 0x7f09a806afe0 2023-10-16T09:44:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09ad5e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a815de20 con 0x7f09a80cc590 2023-10-16T09:44:09.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09a67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09a806afe0 0x7f09a80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09a6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f09a815da40 0x7f09a806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47536/0 (socket says 172.21.15.158:47536) 2023-10-16T09:44:09.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09a67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09a806afe0 0x7f09a80caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47362/0 (socket says 172.21.15.158:47362) 2023-10-16T09:44:09.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.678+0000 7f09a6ffd700 1 -- 172.21.15.158:0/2426409482 learned_addr learned my addr 172.21.15.158:0/2426409482 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a67fc700 1 -- 172.21.15.158:0/2426409482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a80cc590 msgr2=0x7f09a80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a77fe700 1 --2- 172.21.15.158:0/2426409482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a80cc590 0x7f09a80afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a67fc700 1 --2- 172.21.15.158:0/2426409482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a80cc590 0x7f09a80afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a5ffb700 1 -- 172.21.15.158:0/2426409482 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799108828 0 0) 0x7f09a80cbd30 con 0x7f09a815da40 2023-10-16T09:44:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a67fc700 1 -- 172.21.15.158:0/2426409482 >> v1:172.21.15.158:6789/0 conn(0x7f09a815da40 legacy=0x7f09a806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a67fc700 1 -- 172.21.15.158:0/2426409482 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a80ce260 con 0x7f09a806afe0 2023-10-16T09:44:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a67fc700 1 --2- 172.21.15.158:0/2426409482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a806afe0 0x7f09a80caa20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f099c009840 tx=0x7f099c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a6c6f53805e0e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a5ffb700 1 -- 172.21.15.158:0/2426409482 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c011830 con 0x7f09a806afe0 2023-10-16T09:44:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a5ffb700 1 -- 172.21.15.158:0/2426409482 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c011990 con 0x7f09a806afe0 2023-10-16T09:44:09.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.679+0000 7f09a5ffb700 1 -- 172.21.15.158:0/2426409482 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c011d20 con 0x7f09a806afe0 2023-10-16T09:44:09.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2426409482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a806afe0 msgr2=0x7f09a80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2426409482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a806afe0 0x7f09a80caa20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f099c009840 tx=0x7f099c009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:09.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2426409482 shutdown_connections 2023-10-16T09:44:09.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2426409482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a80cc590 0x7f09a80afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2426409482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a806afe0 0x7f09a80caa20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2426409482 >> 172.21.15.158:0/2426409482 conn(0x7f09a8099a10 msgr2=0x7f09a80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:09.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2426409482 shutdown_connections 2023-10-16T09:44:09.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.680+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2426409482 wait complete. 2023-10-16T09:44:09.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.681+0000 7f09ad5e7700 1 Processor -- start 2023-10-16T09:44:09.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.681+0000 7f09ad5e7700 1 -- start start 2023-10-16T09:44:09.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09ad5e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a806afe0 0x7f09a815a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09ad5e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09a80cc590 0x7f09a80c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09ad5e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a815da40 0x7f09a80c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09ad5e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a815acb0 con 0x7f09a815da40 2023-10-16T09:44:09.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09ad5e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80c3f60 con 0x7f09a80cc590 2023-10-16T09:44:09.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09ad5e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80c4240 con 0x7f09a806afe0 2023-10-16T09:44:09.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a815da40 0x7f09a80c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a815da40 0x7f09a80c5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47370/0 (socket says 172.21.15.158:47370) 2023-10-16T09:44:09.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 -- 172.21.15.158:0/2637453881 learned_addr learned my addr 172.21.15.158:0/2637453881 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:09.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a6ffd700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a806afe0 0x7f09a815a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a67fc700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09a80cc590 0x7f09a80c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 -- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a806afe0 msgr2=0x7f09a815a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a806afe0 0x7f09a815a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 -- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09a80cc590 msgr2=0x7f09a80c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09a80cc590 0x7f09a80c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 -- 172.21.15.158:0/2637453881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c0094a0 con 0x7f09a815da40 2023-10-16T09:44:09.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a67fc700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09a80cc590 0x7f09a80c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:09.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09a77fe700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a815da40 0x7f09a80c5560 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f09a000cd70 tx=0x7f09a000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f098f7fe700 1 -- 172.21.15.158:0/2637453881 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0003a20 con 0x7f09a815da40 2023-10-16T09:44:09.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f098f7fe700 1 -- 172.21.15.158:0/2637453881 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a0003b80 con 0x7f09a815da40 2023-10-16T09:44:09.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.682+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a80c4520 con 0x7f09a815da40 2023-10-16T09:44:09.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.683+0000 7f098f7fe700 1 -- 172.21.15.158:0/2637453881 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0004020 con 0x7f09a815da40 2023-10-16T09:44:09.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.683+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a80ca440 con 0x7f09a815da40 2023-10-16T09:44:09.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.683+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a80c4a40 con 0x7f09a815da40 2023-10-16T09:44:09.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.684+0000 7f098f7fe700 1 -- 172.21.15.158:0/2637453881 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f09a0004180 con 0x7f09a815da40 2023-10-16T09:44:09.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.684+0000 7f098f7fe700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f099406af00 0x7f099406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:09.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.684+0000 7f098f7fe700 1 -- 172.21.15.158:0/2637453881 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f09a0012070 con 0x7f09a815da40 2023-10-16T09:44:09.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.684+0000 7f09a6ffd700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f099406af00 0x7f099406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:09.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.685+0000 7f09a6ffd700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f099406af00 0x7f099406d3c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0998007fd0 tx=0x7f0998007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:09.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.689+0000 7f098f7fe700 1 -- 172.21.15.158:0/2637453881 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09a80c4a40 con 0x7f09a815da40 2023-10-16T09:44:09.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.880+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09a80c4a40 con 0x7f09a815da40 2023-10-16T09:44:09.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.881+0000 7f098f7fe700 1 -- 172.21.15.158:0/2637453881 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09a80c4a40 con 0x7f09a815da40 2023-10-16T09:44:09.883 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f099406af00 msgr2=0x7f099406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f099406af00 0x7f099406d3c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0998007fd0 tx=0x7f0998007de0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a815da40 msgr2=0x7f09a80c5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a815da40 0x7f09a80c5560 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f09a000cd70 tx=0x7f09a000a570 comp rx=0 tx=0).stop 2023-10-16T09:44:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 shutdown_connections 2023-10-16T09:44:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09a815da40 0x7f09a80c5560 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f099406af00 0x7f099406d3c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09a806afe0 0x7f09a815a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 --2- 172.21.15.158:0/2637453881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09a80cc590 0x7f09a80c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 >> 172.21.15.158:0/2637453881 conn(0x7f09a8099a10 msgr2=0x7f09a815c970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 shutdown_connections 2023-10-16T09:44:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:09.883+0000 7f09ad5e7700 1 -- 172.21.15.158:0/2637453881 wait complete. 2023-10-16T09:44:09.897 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:10.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.020+0000 7f13c14c0700 1 Processor -- start 2023-10-16T09:44:10.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.020+0000 7f13c14c0700 1 -- start start 2023-10-16T09:44:10.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.021+0000 7f13c14c0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13bc15da40 0x7f13bc0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.021+0000 7f13c14c0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 0x7f13bc0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.021+0000 7f13c14c0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0cbd30 con 0x7f13bc0cc590 2023-10-16T09:44:10.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.021+0000 7f13c14c0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc0cc1c0 con 0x7f13bc15da40 2023-10-16T09:44:10.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.021+0000 7f13c14c0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc15de20 con 0x7f13bc06afe0 2023-10-16T09:44:10.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.021+0000 7f13ba7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f13bc0cc590 0x7f13bc06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47550/0 (socket says 172.21.15.158:47550) 2023-10-16T09:44:10.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.021+0000 7f13ba7fc700 1 -- 172.21.15.158:0/1925145895 learned_addr learned my addr 172.21.15.158:0/1925145895 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:10.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13baffd700 1 --2- 172.21.15.158:0/1925145895 >> v2:172.21.15.172:3300/0 conn(0x7f13bc15da40 0x7f13bc0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513374613 0 0) 0x7f13bc0cbd30 con 0x7f13bc0cc590 2023-10-16T09:44:10.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 0x7f13bc0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a40035b0 con 0x7f13bc0cc590 2023-10-16T09:44:10.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1159676930 0 0) 0x7f13a40035b0 con 0x7f13bc0cc590 2023-10-16T09:44:10.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 msgr2=0x7f13bc0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:10.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 0x7f13bc0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 >> v2:172.21.15.172:3300/0 conn(0x7f13bc15da40 msgr2=0x7f13bc0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:10.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 --2- 172.21.15.158:0/1925145895 >> v2:172.21.15.172:3300/0 conn(0x7f13bc15da40 0x7f13bc0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0ce260 con 0x7f13bc0cc590 2023-10-16T09:44:10.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f13ac001670 con 0x7f13bc0cc590 2023-10-16T09:44:10.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 >> v1:172.21.15.158:6789/0 conn(0x7f13bc0cc590 legacy=0x7f13bc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 0x7f13a4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a4003f80 0x7f13a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a4004d90 0x7f13a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0cc1c0 con 0x7f13a4004d90 2023-10-16T09:44:10.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc15de20 con 0x7f13a4003f80 2023-10-16T09:44:10.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a400ba10 con 0x7f13bc06afe0 2023-10-16T09:44:10.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a4004d90 0x7f13a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13baffd700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 0x7f13a4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13ba7fc700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a4003f80 0x7f13a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13bb7fe700 1 -- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 msgr2=0x7f13a4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:10.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.022+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc06afe0 0x7f13a4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13bb7fe700 1 -- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a4003f80 msgr2=0x7f13a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:10.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a4003f80 0x7f13a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13bb7fe700 1 -- 172.21.15.158:0/1925145895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a40035b0 con 0x7f13a4004d90 2023-10-16T09:44:10.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a4004d90 0x7f13a4005210 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f13bc072a00 tx=0x7f13a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0550741bb4e039e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400eb70 con 0x7f13a4004d90 2023-10-16T09:44:10.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b400f040 con 0x7f13a4004d90 2023-10-16T09:44:10.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13b9ffb700 1 -- 172.21.15.158:0/1925145895 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4014480 con 0x7f13a4004d90 2023-10-16T09:44:10.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13c14c0700 1 -- 172.21.15.158:0/1925145895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a4004d90 msgr2=0x7f13a4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13c14c0700 1 --2- 172.21.15.158:0/1925145895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a4004d90 0x7f13a4005210 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f13bc072a00 tx=0x7f13a4005880 comp rx=0 tx=0).stop 2023-10-16T09:44:10.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13bb7fe700 1 -- 172.21.15.158:0/1925145895 reap_dead start 2023-10-16T09:44:10.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13c14c0700 1 -- 172.21.15.158:0/1925145895 shutdown_connections 2023-10-16T09:44:10.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13c14c0700 1 -- 172.21.15.158:0/1925145895 >> 172.21.15.158:0/1925145895 conn(0x7f13bc099a10 msgr2=0x7f13bc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:10.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13c14c0700 1 -- 172.21.15.158:0/1925145895 shutdown_connections 2023-10-16T09:44:10.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.023+0000 7f13c14c0700 1 -- 172.21.15.158:0/1925145895 wait complete. 2023-10-16T09:44:10.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.025+0000 7f13c14c0700 1 Processor -- start 2023-10-16T09:44:10.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.025+0000 7f13c14c0700 1 -- start start 2023-10-16T09:44:10.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.025+0000 7f13c14c0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13bc1f5b80 0x7f13bc1f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13c14c0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc1f5f60 0x7f13bc0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13c14c0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13bc1e7530 0x7f13bc1e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13c14c0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1e7ef0 con 0x7f13bc1e7530 2023-10-16T09:44:10.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13c14c0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc156850 con 0x7f13bc1f5b80 2023-10-16T09:44:10.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13c14c0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1f3410 con 0x7f13bc1f5f60 2023-10-16T09:44:10.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13bc1e7530 0x7f13bc1e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13bc1e7530 0x7f13bc1e79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47390/0 (socket says 172.21.15.158:47390) 2023-10-16T09:44:10.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 -- 172.21.15.158:0/2093116560 learned_addr learned my addr 172.21.15.158:0/2093116560 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:10.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13baffd700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13bc1f5b80 0x7f13bc1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13ba7fc700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc1f5f60 0x7f13bc0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 -- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc1f5f60 msgr2=0x7f13bc0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc1f5f60 0x7f13bc0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 -- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13bc1f5b80 msgr2=0x7f13bc1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13bc1f5b80 0x7f13bc1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 -- 172.21.15.158:0/2093116560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc156aa0 con 0x7f13bc1e7530 2023-10-16T09:44:10.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13bb7fe700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13bc1e7530 0x7f13bc1e79b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f13b400e700 tx=0x7f13b4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13a37fe700 1 -- 172.21.15.158:0/2093116560 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4007000 con 0x7f13bc1e7530 2023-10-16T09:44:10.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13a37fe700 1 -- 172.21.15.158:0/2093116560 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b4022070 con 0x7f13bc1e7530 2023-10-16T09:44:10.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc156d80 con 0x7f13bc1e7530 2023-10-16T09:44:10.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13a37fe700 1 -- 172.21.15.158:0/2093116560 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4003ea0 con 0x7f13bc1e7530 2023-10-16T09:44:10.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.026+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc157340 con 0x7f13bc1e7530 2023-10-16T09:44:10.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.027+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc0681b0 con 0x7f13bc1e7530 2023-10-16T09:44:10.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.028+0000 7f13a37fe700 1 -- 172.21.15.158:0/2093116560 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f13b4024030 con 0x7f13bc1e7530 2023-10-16T09:44:10.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.028+0000 7f13a37fe700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f13a806af00 0x7f13a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.028+0000 7f13a37fe700 1 -- 172.21.15.158:0/2093116560 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f13b408b130 con 0x7f13bc1e7530 2023-10-16T09:44:10.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.028+0000 7f13baffd700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f13a806af00 0x7f13a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.029+0000 7f13baffd700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f13a806af00 0x7f13a806d3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009ce0 tx=0x7f13b0009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.033+0000 7f13a37fe700 1 -- 172.21.15.158:0/2093116560 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13b4055f30 con 0x7f13bc1e7530 2023-10-16T09:44:10.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.223+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f13bc06c6d0 con 0x7f13bc1e7530 2023-10-16T09:44:10.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.224+0000 7f13a37fe700 1 -- 172.21.15.158:0/2093116560 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f13b40180a0 con 0x7f13bc1e7530 2023-10-16T09:44:10.226 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:10.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f13a806af00 msgr2=0x7f13a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f13a806af00 0x7f13a806d3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009ce0 tx=0x7f13b0009210 comp rx=0 tx=0).stop 2023-10-16T09:44:10.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13bc1e7530 msgr2=0x7f13bc1e79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13bc1e7530 0x7f13bc1e79b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f13b400e700 tx=0x7f13b4009750 comp rx=0 tx=0).stop 2023-10-16T09:44:10.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 shutdown_connections 2023-10-16T09:44:10.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13bc1e7530 0x7f13bc1e79b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13bc1f5b80 0x7f13bc1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f13a806af00 0x7f13a806d3c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 --2- 172.21.15.158:0/2093116560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13bc1f5f60 0x7f13bc0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 >> 172.21.15.158:0/2093116560 conn(0x7f13bc099a10 msgr2=0x7f13bc0cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:10.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 shutdown_connections 2023-10-16T09:44:10.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.226+0000 7f13c14c0700 1 -- 172.21.15.158:0/2093116560 wait complete. 2023-10-16T09:44:10.240 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.359+0000 7f050de07700 1 Processor -- start 2023-10-16T09:44:10.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.359+0000 7f050de07700 1 -- start start 2023-10-16T09:44:10.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f050de07700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05080cdee0 0x7f050815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f050de07700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05080b6e10 0x7f05081521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f050de07700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f05081526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f050de07700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050815d2c0 con 0x7f05080cdee0 2023-10-16T09:44:10.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f050de07700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0508152c20 con 0x7f05080b6e10 2023-10-16T09:44:10.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f050de07700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050815c5d0 con 0x7f05080ce2c0 2023-10-16T09:44:10.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05080cdee0 0x7f050815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05080cdee0 0x7f050815bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47394/0 (socket says 172.21.15.158:47394) 2023-10-16T09:44:10.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 -- 172.21.15.158:0/3719844065 learned_addr learned my addr 172.21.15.158:0/3719844065 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:10.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f0506ffd700 1 --2- 172.21.15.158:0/3719844065 >> v2:172.21.15.172:3300/0 conn(0x7f05080b6e10 0x7f05081521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f0507fff700 1 --2- 172.21.15.158:0/3719844065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f05081526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 -- 172.21.15.158:0/3719844065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 msgr2=0x7f05081526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:10.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 --2- 172.21.15.158:0/3719844065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f05081526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 -- 172.21.15.158:0/3719844065 >> v2:172.21.15.172:3300/0 conn(0x7f05080b6e10 msgr2=0x7f05081521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:10.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 --2- 172.21.15.158:0/3719844065 >> v2:172.21.15.172:3300/0 conn(0x7f05080b6e10 0x7f05081521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 -- 172.21.15.158:0/3719844065 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050815c8b0 con 0x7f05080cdee0 2023-10-16T09:44:10.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f0506ffd700 1 --2- 172.21.15.158:0/3719844065 >> v2:172.21.15.172:3300/0 conn(0x7f05080b6e10 0x7f05081521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:10.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f0507fff700 1 --2- 172.21.15.158:0/3719844065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f05081526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:10.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05077fe700 1 --2- 172.21.15.158:0/3719844065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080cdee0 0x7f050815bd10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f05080b3b90 tx=0x7f04fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636a07df7eaf5353 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.360+0000 7f05067fc700 1 -- 172.21.15.158:0/3719844065 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00b860 con 0x7f05080cdee0 2023-10-16T09:44:10.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f05067fc700 1 -- 172.21.15.158:0/3719844065 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc00b9c0 con 0x7f05080cdee0 2023-10-16T09:44:10.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f05067fc700 1 -- 172.21.15.158:0/3719844065 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00bd50 con 0x7f05080cdee0 2023-10-16T09:44:10.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 -- 172.21.15.158:0/3719844065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080cdee0 msgr2=0x7f050815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 --2- 172.21.15.158:0/3719844065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080cdee0 0x7f050815bd10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f05080b3b90 tx=0x7f04fc009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 -- 172.21.15.158:0/3719844065 shutdown_connections 2023-10-16T09:44:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 --2- 172.21.15.158:0/3719844065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f05081526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 --2- 172.21.15.158:0/3719844065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080cdee0 0x7f050815bd10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 --2- 172.21.15.158:0/3719844065 >> v2:172.21.15.172:3300/0 conn(0x7f05080b6e10 0x7f05081521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 -- 172.21.15.158:0/3719844065 >> 172.21.15.158:0/3719844065 conn(0x7f050806a8d0 msgr2=0x7f050806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:10.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 -- 172.21.15.158:0/3719844065 shutdown_connections 2023-10-16T09:44:10.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.361+0000 7f050de07700 1 -- 172.21.15.158:0/3719844065 wait complete. 2023-10-16T09:44:10.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.362+0000 7f050de07700 1 Processor -- start 2023-10-16T09:44:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.362+0000 7f050de07700 1 -- start start 2023-10-16T09:44:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f050de07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080b6e10 0x7f05080b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f050de07700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05080cdee0 0x7f05081e3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f050de07700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f050814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f050de07700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05080b3ff0 con 0x7f05080b6e10 2023-10-16T09:44:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f050de07700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05080b45b0 con 0x7f05080cdee0 2023-10-16T09:44:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f050de07700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05081e4620 con 0x7f05080ce2c0 2023-10-16T09:44:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f05077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080b6e10 0x7f05080b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f05077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080b6e10 0x7f05080b4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47400/0 (socket says 172.21.15.158:47400) 2023-10-16T09:44:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.363+0000 7f05077fe700 1 -- 172.21.15.158:0/2539062810 learned_addr learned my addr 172.21.15.158:0/2539062810 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f0507fff700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f050814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f0506ffd700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05080cdee0 0x7f05081e3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f05077fe700 1 -- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 msgr2=0x7f050814b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f05077fe700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f050814b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f05077fe700 1 -- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05080cdee0 msgr2=0x7f05081e3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f05077fe700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05080cdee0 0x7f05081e3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f05077fe700 1 -- 172.21.15.158:0/2539062810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc009550 con 0x7f05080b6e10 2023-10-16T09:44:10.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f05077fe700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080b6e10 0x7f05080b4a40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009c40 tx=0x7f04fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f04effff700 1 -- 172.21.15.158:0/2539062810 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc004380 con 0x7f05080b6e10 2023-10-16T09:44:10.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f04effff700 1 -- 172.21.15.158:0/2539062810 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc01f070 con 0x7f05080b6e10 2023-10-16T09:44:10.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05081e4900 con 0x7f05080b6e10 2023-10-16T09:44:10.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f04effff700 1 -- 172.21.15.158:0/2539062810 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc0036a0 con 0x7f05080b6e10 2023-10-16T09:44:10.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.364+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05081e4310 con 0x7f05080b6e10 2023-10-16T09:44:10.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.365+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05080b4890 con 0x7f05080b6e10 2023-10-16T09:44:10.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.366+0000 7f04effff700 1 -- 172.21.15.158:0/2539062810 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f04fc021030 con 0x7f05080b6e10 2023-10-16T09:44:10.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.366+0000 7f04effff700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f04f406af00 0x7f04f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.366+0000 7f04effff700 1 -- 172.21.15.158:0/2539062810 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f04fc08ad20 con 0x7f05080b6e10 2023-10-16T09:44:10.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.366+0000 7f0506ffd700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f04f406af00 0x7f04f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.366+0000 7f0506ffd700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f04f406af00 0x7f04f406d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f05081e6660 tx=0x7f04f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.370+0000 7f04effff700 1 -- 172.21.15.158:0/2539062810 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f04fc055a50 con 0x7f05080b6e10 2023-10-16T09:44:10.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.574+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05081e4b10 con 0x7f05080b6e10 2023-10-16T09:44:10.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.575+0000 7f04effff700 1 -- 172.21.15.158:0/2539062810 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7f04fc015020 con 0x7f05080b6e10 2023-10-16T09:44:10.577 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:10.577 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f04f406af00 msgr2=0x7f04f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f04f406af00 0x7f04f406d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f05081e6660 tx=0x7f04f8008040 comp rx=0 tx=0).stop 2023-10-16T09:44:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080b6e10 msgr2=0x7f05080b4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080b6e10 0x7f05080b4a40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009c40 tx=0x7f04fc004750 comp rx=0 tx=0).stop 2023-10-16T09:44:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 shutdown_connections 2023-10-16T09:44:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f04f406af00 0x7f04f406d3c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05080ce2c0 0x7f050814b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05080b6e10 0x7f05080b4a40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 --2- 172.21.15.158:0/2539062810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05080cdee0 0x7f05081e3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.577+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 >> 172.21.15.158:0/2539062810 conn(0x7f050806a8d0 msgr2=0x7f05080b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:10.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.578+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 shutdown_connections 2023-10-16T09:44:10.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.578+0000 7f050de07700 1 -- 172.21.15.158:0/2539062810 wait complete. 2023-10-16T09:44:10.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:10.592 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:10.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.714+0000 7f3884858700 1 Processor -- start 2023-10-16T09:44:10.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.714+0000 7f3884858700 1 -- start start 2023-10-16T09:44:10.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f3884858700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f388015da40 0x7f38800caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f3884858700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f38800cc590 0x7f388006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f3884858700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38800cbd30 con 0x7f388006ae10 2023-10-16T09:44:10.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f3884858700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388006b2c0 con 0x7f388015da40 2023-10-16T09:44:10.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f3884858700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38800cc1c0 con 0x7f38800cc590 2023-10-16T09:44:10.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387ed9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f388006ae10 0x7f38800ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47588/0 (socket says 172.21.15.158:47588) 2023-10-16T09:44:10.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387ed9d700 1 -- 172.21.15.158:0/3202659292 learned_addr learned my addr 172.21.15.158:0/3202659292 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:10.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387e59c700 1 --2- 172.21.15.158:0/3202659292 >> v2:172.21.15.172:3300/0 conn(0x7f388015da40 0x7f38800caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387dd9b700 1 --2- 172.21.15.158:0/3202659292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f38800cc590 0x7f388006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387d59a700 1 -- 172.21.15.158:0/3202659292 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365568772 0 0) 0x7f38800cbd30 con 0x7f388006ae10 2023-10-16T09:44:10.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387d59a700 1 -- 172.21.15.158:0/3202659292 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38680035b0 con 0x7f388006ae10 2023-10-16T09:44:10.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387d59a700 1 -- 172.21.15.158:0/3202659292 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111293920 0 0) 0x7f38680035b0 con 0x7f388006ae10 2023-10-16T09:44:10.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387dd9b700 1 -- 172.21.15.158:0/3202659292 >> v2:172.21.15.172:3300/0 conn(0x7f388015da40 msgr2=0x7f38800caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:10.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387dd9b700 1 --2- 172.21.15.158:0/3202659292 >> v2:172.21.15.172:3300/0 conn(0x7f388015da40 0x7f38800caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387dd9b700 1 -- 172.21.15.158:0/3202659292 >> v1:172.21.15.158:6789/0 conn(0x7f388006ae10 legacy=0x7f38800ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387dd9b700 1 -- 172.21.15.158:0/3202659292 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388015de20 con 0x7f38800cc590 2023-10-16T09:44:10.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.715+0000 7f387e59c700 1 --2- 172.21.15.158:0/3202659292 >> v2:172.21.15.172:3300/0 conn(0x7f388015da40 0x7f38800caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:10.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f387dd9b700 1 --2- 172.21.15.158:0/3202659292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f38800cc590 0x7f388006a8d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3874009840 tx=0x7f3874009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=667bb306ec600664 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f387d59a700 1 -- 172.21.15.158:0/3202659292 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38780019f0 con 0x7f38800cc590 2023-10-16T09:44:10.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f387d59a700 1 -- 172.21.15.158:0/3202659292 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3874004030 con 0x7f38800cc590 2023-10-16T09:44:10.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f387d59a700 1 -- 172.21.15.158:0/3202659292 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740043a0 con 0x7f38800cc590 2023-10-16T09:44:10.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 -- 172.21.15.158:0/3202659292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f38800cc590 msgr2=0x7f388006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 --2- 172.21.15.158:0/3202659292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f38800cc590 0x7f388006a8d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3874009840 tx=0x7f3874009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:10.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 -- 172.21.15.158:0/3202659292 shutdown_connections 2023-10-16T09:44:10.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 --2- 172.21.15.158:0/3202659292 >> v2:172.21.15.172:3300/0 conn(0x7f388015da40 0x7f38800caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 --2- 172.21.15.158:0/3202659292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f38800cc590 0x7f388006a8d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 -- 172.21.15.158:0/3202659292 >> 172.21.15.158:0/3202659292 conn(0x7f3880099a10 msgr2=0x7f38800d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:10.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 -- 172.21.15.158:0/3202659292 shutdown_connections 2023-10-16T09:44:10.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.716+0000 7f3884858700 1 -- 172.21.15.158:0/3202659292 wait complete. 2023-10-16T09:44:10.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 Processor -- start 2023-10-16T09:44:10.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 -- start start 2023-10-16T09:44:10.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f388006ae10 0x7f38800b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38800cc590 0x7f38801e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f388015da40 0x7f38801e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880159f50 con 0x7f388006ae10 2023-10-16T09:44:10.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880159920 con 0x7f38800cc590 2023-10-16T09:44:10.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f3884858700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38800ca2b0 con 0x7f388015da40 2023-10-16T09:44:10.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.718+0000 7f387e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f388006ae10 0x7f38800b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f388006ae10 0x7f38800b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47414/0 (socket says 172.21.15.158:47414) 2023-10-16T09:44:10.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 -- 172.21.15.158:0/3653656445 learned_addr learned my addr 172.21.15.158:0/3653656445 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:10.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387ed9d700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f388015da40 0x7f38801e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387dd9b700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38800cc590 0x7f38801e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 -- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f388015da40 msgr2=0x7f38801e4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f388015da40 0x7f38801e4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 -- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38800cc590 msgr2=0x7f38801e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38800cc590 0x7f38801e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 -- 172.21.15.158:0/3653656445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38700094a0 con 0x7f388006ae10 2023-10-16T09:44:10.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f387e59c700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f388006ae10 0x7f38800b3760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f387000ebe0 tx=0x7f3870009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f386effd700 1 -- 172.21.15.158:0/3653656445 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387000c8b0 con 0x7f388006ae10 2023-10-16T09:44:10.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f386effd700 1 -- 172.21.15.158:0/3653656445 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387000ca10 con 0x7f388006ae10 2023-10-16T09:44:10.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38740094a0 con 0x7f388006ae10 2023-10-16T09:44:10.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f386effd700 1 -- 172.21.15.158:0/3653656445 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387000cda0 con 0x7f388006ae10 2023-10-16T09:44:10.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.719+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38801e5d20 con 0x7f388006ae10 2023-10-16T09:44:10.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.720+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388015a1c0 con 0x7f388006ae10 2023-10-16T09:44:10.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.721+0000 7f386effd700 1 -- 172.21.15.158:0/3653656445 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3870013470 con 0x7f388006ae10 2023-10-16T09:44:10.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.721+0000 7f386effd700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f386006af00 0x7f386006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:10.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.721+0000 7f386effd700 1 -- 172.21.15.158:0/3653656445 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f3870089a70 con 0x7f388006ae10 2023-10-16T09:44:10.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.721+0000 7f387dd9b700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f386006af00 0x7f386006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:10.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.721+0000 7f387dd9b700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f386006af00 0x7f386006d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3874009840 tx=0x7f3874008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:10.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.725+0000 7f386effd700 1 -- 172.21.15.158:0/3653656445 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38700547a0 con 0x7f388006ae10 2023-10-16T09:44:10.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.914+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38800681b0 con 0x7f388006ae10 2023-10-16T09:44:10.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.915+0000 7f386effd700 1 -- 172.21.15.158:0/3653656445 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3870057dc0 con 0x7f388006ae10 2023-10-16T09:44:10.917 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:10.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f386006af00 msgr2=0x7f386006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f386006af00 0x7f386006d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3874009840 tx=0x7f3874008030 comp rx=0 tx=0).stop 2023-10-16T09:44:10.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f388006ae10 msgr2=0x7f38800b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:10.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f388006ae10 0x7f38800b3760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f387000ebe0 tx=0x7f3870009e20 comp rx=0 tx=0).stop 2023-10-16T09:44:10.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 shutdown_connections 2023-10-16T09:44:10.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f386006af00 0x7f386006d3c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f388015da40 0x7f38801e4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f388006ae10 0x7f38800b3760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 --2- 172.21.15.158:0/3653656445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38800cc590 0x7f38801e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:10.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 >> 172.21.15.158:0/3653656445 conn(0x7f3880099a10 msgr2=0x7f38800b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:10.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 shutdown_connections 2023-10-16T09:44:10.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:10.917+0000 7f3884858700 1 -- 172.21.15.158:0/3653656445 wait complete. 2023-10-16T09:44:10.932 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:11.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.054+0000 7fb0da81b700 1 Processor -- start 2023-10-16T09:44:11.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.054+0000 7fb0da81b700 1 -- start start 2023-10-16T09:44:11.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0da81b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0d4160d30 0x7fb0d40b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0da81b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d40d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0da81b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d40b7c90 con 0x7fb0d40d19a0 2023-10-16T09:44:11.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0da81b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d40b8910 con 0x7fb0d4160d30 2023-10-16T09:44:11.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0da81b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40d0ef0 con 0x7fb0d40d0530 2023-10-16T09:44:11.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0d4160d30 0x7fb0d40b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0d4160d30 0x7fb0d40b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47422/0 (socket says 172.21.15.158:47422) 2023-10-16T09:44:11.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 -- 172.21.15.158:0/1453090440 learned_addr learned my addr 172.21.15.158:0/1453090440 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:11.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d37fe700 1 --2- 172.21.15.158:0/1453090440 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d40d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 -- 172.21.15.158:0/1453090440 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 msgr2=0x7fb0d40d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:11.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 --2- 172.21.15.158:0/1453090440 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d40d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 -- 172.21.15.158:0/1453090440 >> v1:172.21.15.172:6789/0 conn(0x7fb0d40d19a0 legacy=0x7fb0d40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 -- 172.21.15.158:0/1453090440 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40b7f70 con 0x7fb0d4160d30 2023-10-16T09:44:11.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d3fff700 1 --2- 172.21.15.158:0/1453090440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d4160d30 0x7fb0d40b8f50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40098f0 tx=0x7fb0c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d1762d4008eacd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d2ffd700 1 -- 172.21.15.158:0/1453090440 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4011830 con 0x7fb0d4160d30 2023-10-16T09:44:11.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d2ffd700 1 -- 172.21.15.158:0/1453090440 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c4011990 con 0x7fb0d4160d30 2023-10-16T09:44:11.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.055+0000 7fb0d2ffd700 1 -- 172.21.15.158:0/1453090440 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4011d20 con 0x7fb0d4160d30 2023-10-16T09:44:11.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 -- 172.21.15.158:0/1453090440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d4160d30 msgr2=0x7fb0d40b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1453090440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d4160d30 0x7fb0d40b8f50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40098f0 tx=0x7fb0c4009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:11.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 -- 172.21.15.158:0/1453090440 shutdown_connections 2023-10-16T09:44:11.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1453090440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d4160d30 0x7fb0d40b8f50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1453090440 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d40d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 -- 172.21.15.158:0/1453090440 >> 172.21.15.158:0/1453090440 conn(0x7fb0d40669f0 msgr2=0x7fb0d4066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:11.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 -- 172.21.15.158:0/1453090440 shutdown_connections 2023-10-16T09:44:11.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.056+0000 7fb0da81b700 1 -- 172.21.15.158:0/1453090440 wait complete. 2023-10-16T09:44:11.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.057+0000 7fb0da81b700 1 Processor -- start 2023-10-16T09:44:11.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.057+0000 7fb0da81b700 1 -- start start 2023-10-16T09:44:11.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0da81b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d41f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0da81b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d40d19a0 0x7fb0d4148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0da81b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0d4160d30 0x7fb0d4156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0da81b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40c9f20 con 0x7fb0d40d19a0 2023-10-16T09:44:11.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0da81b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d41588d0 con 0x7fb0d4160d30 2023-10-16T09:44:11.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0da81b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d4158bb0 con 0x7fb0d40d0530 2023-10-16T09:44:11.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d40d19a0 0x7fb0d4148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d40d19a0 0x7fb0d4148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47430/0 (socket says 172.21.15.158:47430) 2023-10-16T09:44:11.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 -- 172.21.15.158:0/1802846239 learned_addr learned my addr 172.21.15.158:0/1802846239 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:11.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d3fff700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d41f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d8db8700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0d4160d30 0x7fb0d4156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 -- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 msgr2=0x7fb0d41f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d41f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 -- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0d4160d30 msgr2=0x7fb0d4156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0d4160d30 0x7fb0d4156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d37fe700 1 -- 172.21.15.158:0/1802846239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c4009550 con 0x7fb0d40d19a0 2023-10-16T09:44:11.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.058+0000 7fb0d3fff700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d41f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:11.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.059+0000 7fb0d37fe700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d40d19a0 0x7fb0d4148610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e9b0 tx=0x7fb0c800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.059+0000 7fb0d0ff9700 1 -- 172.21.15.158:0/1802846239 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8009710 con 0x7fb0d40d19a0 2023-10-16T09:44:11.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.059+0000 7fb0d0ff9700 1 -- 172.21.15.158:0/1802846239 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c800c850 con 0x7fb0d40d19a0 2023-10-16T09:44:11.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.059+0000 7fb0d0ff9700 1 -- 172.21.15.158:0/1802846239 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c800cbc0 con 0x7fb0d40d19a0 2023-10-16T09:44:11.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.059+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d4158e90 con 0x7fb0d40d19a0 2023-10-16T09:44:11.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.059+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d4148b50 con 0x7fb0d40d19a0 2023-10-16T09:44:11.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.060+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d4148d70 con 0x7fb0d40d19a0 2023-10-16T09:44:11.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.060+0000 7fb0d0ff9700 1 -- 172.21.15.158:0/1802846239 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb0c800cd20 con 0x7fb0d40d19a0 2023-10-16T09:44:11.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.061+0000 7fb0d0ff9700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb0c006af00 0x7fb0c006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.061+0000 7fb0d3fff700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb0c006af00 0x7fb0c006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.061+0000 7fb0d0ff9700 1 -- 172.21.15.158:0/1802846239 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fb0c8089a50 con 0x7fb0d40d19a0 2023-10-16T09:44:11.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.061+0000 7fb0d3fff700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb0c006af00 0x7fb0c006d3c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4014040 tx=0x7fb0c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.065+0000 7fb0d0ff9700 1 -- 172.21.15.158:0/1802846239 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0d4148d70 con 0x7fb0d40d19a0 2023-10-16T09:44:11.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.256+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0d4148d70 con 0x7fb0d40d19a0 2023-10-16T09:44:11.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.256+0000 7fb0d0ff9700 1 -- 172.21.15.158:0/1802846239 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0d4148d70 con 0x7fb0d40d19a0 2023-10-16T09:44:11.258 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:11.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb0c006af00 msgr2=0x7fb0c006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb0c006af00 0x7fb0c006d3c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4014040 tx=0x7fb0c4008030 comp rx=0 tx=0).stop 2023-10-16T09:44:11.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d40d19a0 msgr2=0x7fb0d4148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d40d19a0 0x7fb0d4148610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e9b0 tx=0x7fb0c800ed70 comp rx=0 tx=0).stop 2023-10-16T09:44:11.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 shutdown_connections 2023-10-16T09:44:11.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0d4160d30 0x7fb0d4156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb0c006af00 0x7fb0c006d3c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0d40d0530 0x7fb0d41f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 --2- 172.21.15.158:0/1802846239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d40d19a0 0x7fb0d4148610 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.258+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 >> 172.21.15.158:0/1802846239 conn(0x7fb0d40669f0 msgr2=0x7fb0d4066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:11.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.259+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 shutdown_connections 2023-10-16T09:44:11.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.259+0000 7fb0da81b700 1 -- 172.21.15.158:0/1802846239 wait complete. 2023-10-16T09:44:11.272 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:11.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.393+0000 7f98eb9a3700 1 Processor -- start 2023-10-16T09:44:11.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.393+0000 7f98eb9a3700 1 -- start start 2023-10-16T09:44:11.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98eb9a3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98e415da40 0x7f98e40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98eb9a3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 0x7f98e406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98eb9a3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e40cbd30 con 0x7f98e406ae10 2023-10-16T09:44:11.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98eb9a3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e406b2c0 con 0x7f98e415da40 2023-10-16T09:44:11.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98eb9a3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40cc1c0 con 0x7f98e40cc590 2023-10-16T09:44:11.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98e9f40700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f98e406ae10 0x7f98e40ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47616/0 (socket says 172.21.15.158:47616) 2023-10-16T09:44:11.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98e9f40700 1 -- 172.21.15.158:0/3811828945 learned_addr learned my addr 172.21.15.158:0/3811828945 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:11.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98e973f700 1 --2- 172.21.15.158:0/3811828945 >> v2:172.21.15.172:3300/0 conn(0x7f98e415da40 0x7f98e40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98e8f3e700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 0x7f98e406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365289706 0 0) 0x7f98e40cbd30 con 0x7f98e406ae10 2023-10-16T09:44:11.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d00035b0 con 0x7f98e406ae10 2023-10-16T09:44:11.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4032887476 0 0) 0x7f98d00035b0 con 0x7f98e406ae10 2023-10-16T09:44:11.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 msgr2=0x7f98e406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:11.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 0x7f98e406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 >> v2:172.21.15.172:3300/0 conn(0x7f98e415da40 msgr2=0x7f98e40caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:11.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 --2- 172.21.15.158:0/3811828945 >> v2:172.21.15.172:3300/0 conn(0x7f98e415da40 0x7f98e40caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e415de20 con 0x7f98e406ae10 2023-10-16T09:44:11.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f98dc0019f0 con 0x7f98e406ae10 2023-10-16T09:44:11.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 >> v1:172.21.15.158:6789/0 conn(0x7f98e406ae10 legacy=0x7f98e40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98d0003a00 0x7f98d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 0x7f98d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98d0004d90 0x7f98d0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e406b2c0 con 0x7f98d0004d90 2023-10-16T09:44:11.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40cc1c0 con 0x7f98d0003a00 2023-10-16T09:44:11.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.394+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d000ba10 con 0x7f98e40cc590 2023-10-16T09:44:11.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98d0004d90 0x7f98d0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e973f700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98d0003a00 0x7f98d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e8f3e700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 0x7f98d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 -- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 msgr2=0x7f98d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:11.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98e40cc590 0x7f98d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 -- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98d0003a00 msgr2=0x7f98d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:11.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98d0003a00 0x7f98d0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 -- 172.21.15.158:0/3811828945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00035b0 con 0x7f98d0004d90 2023-10-16T09:44:11.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98d0004d90 0x7f98d0005210 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00a900 tx=0x7f98d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c270d1646fc60ad7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc00a340 con 0x7f98d0004d90 2023-10-16T09:44:11.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc012620 con 0x7f98d0004d90 2023-10-16T09:44:11.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98dbfff700 1 -- 172.21.15.158:0/3811828945 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0129d0 con 0x7f98d0004d90 2023-10-16T09:44:11.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98eb9a3700 1 -- 172.21.15.158:0/3811828945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98d0004d90 msgr2=0x7f98d0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98eb9a3700 1 --2- 172.21.15.158:0/3811828945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98d0004d90 0x7f98d0005210 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00a900 tx=0x7f98d0005880 comp rx=0 tx=0).stop 2023-10-16T09:44:11.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.395+0000 7f98e9f40700 1 -- 172.21.15.158:0/3811828945 reap_dead start 2023-10-16T09:44:11.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.396+0000 7f98eb9a3700 1 -- 172.21.15.158:0/3811828945 shutdown_connections 2023-10-16T09:44:11.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.396+0000 7f98eb9a3700 1 -- 172.21.15.158:0/3811828945 >> 172.21.15.158:0/3811828945 conn(0x7f98e4099a10 msgr2=0x7f98e40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:11.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.396+0000 7f98eb9a3700 1 -- 172.21.15.158:0/3811828945 shutdown_connections 2023-10-16T09:44:11.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.396+0000 7f98eb9a3700 1 -- 172.21.15.158:0/3811828945 wait complete. 2023-10-16T09:44:11.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.397+0000 7f98eb9a3700 1 Processor -- start 2023-10-16T09:44:11.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.397+0000 7f98eb9a3700 1 -- start start 2023-10-16T09:44:11.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98eb9a3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98d0004d90 0x7f98e4150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98eb9a3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98e40aff70 0x7f98e40b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98eb9a3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406eab0 0x7f98e406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98eb9a3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e406f4a0 con 0x7f98e40aff70 2023-10-16T09:44:11.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98eb9a3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40c88f0 con 0x7f98e406eab0 2023-10-16T09:44:11.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98eb9a3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40b0a30 con 0x7f98d0004d90 2023-10-16T09:44:11.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98e40aff70 0x7f98e40b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98e40aff70 0x7f98e40b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47450/0 (socket says 172.21.15.158:47450) 2023-10-16T09:44:11.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 -- 172.21.15.158:0/1776871033 learned_addr learned my addr 172.21.15.158:0/1776871033 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:11.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e973f700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98d0004d90 0x7f98e4150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e9f40700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406eab0 0x7f98e406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 -- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98d0004d90 msgr2=0x7f98e4150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98d0004d90 0x7f98e4150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 -- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406eab0 msgr2=0x7f98e406ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406eab0 0x7f98e406ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 -- 172.21.15.158:0/1776871033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40b0d10 con 0x7f98e40aff70 2023-10-16T09:44:11.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e9f40700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406eab0 0x7f98e406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:11.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98e8f3e700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98e40aff70 0x7f98e40b03f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f98e00119c0 tx=0x7f98e0011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98d9ffb700 1 -- 172.21.15.158:0/1776871033 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e000f940 con 0x7f98e40aff70 2023-10-16T09:44:11.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98d9ffb700 1 -- 172.21.15.158:0/1776871033 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e000faa0 con 0x7f98e40aff70 2023-10-16T09:44:11.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98d9ffb700 1 -- 172.21.15.158:0/1776871033 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e000fe50 con 0x7f98e40aff70 2023-10-16T09:44:11.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.398+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e40c8bb0 con 0x7f98e40aff70 2023-10-16T09:44:11.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.399+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e40ad970 con 0x7f98e40aff70 2023-10-16T09:44:11.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.400+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e40c90d0 con 0x7f98e40aff70 2023-10-16T09:44:11.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.400+0000 7f98d9ffb700 1 -- 172.21.15.158:0/1776871033 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f98e0014440 con 0x7f98e40aff70 2023-10-16T09:44:11.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.401+0000 7f98d9ffb700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f98c806af30 0x7f98c806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.401+0000 7f98d9ffb700 1 -- 172.21.15.158:0/1776871033 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f98e008a0e0 con 0x7f98e40aff70 2023-10-16T09:44:11.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.401+0000 7f98e973f700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f98c806af30 0x7f98c806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.401+0000 7f98e973f700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f98c806af30 0x7f98c806d3f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009ce0 tx=0x7f98d4009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.405+0000 7f98d9ffb700 1 -- 172.21.15.158:0/1776871033 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98e0054ef0 con 0x7f98e40aff70 2023-10-16T09:44:11.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.598+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98e40c90d0 con 0x7f98e40aff70 2023-10-16T09:44:11.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.598+0000 7f98d9ffb700 1 -- 172.21.15.158:0/1776871033 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f98e40c90d0 con 0x7f98e40aff70 2023-10-16T09:44:11.601 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:11.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.600+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f98c806af30 msgr2=0x7f98c806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.600+0000 7f98eb9a3700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f98c806af30 0x7f98c806d3f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009ce0 tx=0x7f98d4009210 comp rx=0 tx=0).stop 2023-10-16T09:44:11.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.600+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98e40aff70 msgr2=0x7f98e40b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.600+0000 7f98eb9a3700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98e40aff70 0x7f98e40b03f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f98e00119c0 tx=0x7f98e0011d80 comp rx=0 tx=0).stop 2023-10-16T09:44:11.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 shutdown_connections 2023-10-16T09:44:11.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406eab0 0x7f98e406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f98c806af30 0x7f98c806d3f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98d0004d90 0x7f98e4150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 --2- 172.21.15.158:0/1776871033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98e40aff70 0x7f98e40b03f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 >> 172.21.15.158:0/1776871033 conn(0x7f98e4099a10 msgr2=0x7f98e40c5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:11.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 shutdown_connections 2023-10-16T09:44:11.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.601+0000 7f98eb9a3700 1 -- 172.21.15.158:0/1776871033 wait complete. 2023-10-16T09:44:11.614 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:11.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.736+0000 7fcb799df700 1 Processor -- start 2023-10-16T09:44:11.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.736+0000 7fcb799df700 1 -- start start 2023-10-16T09:44:11.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb799df700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb740d29b0 0x7fcb740d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb799df700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcb740d0a70 0x7fcb740b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb799df700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740b7c90 con 0x7fcb74160d30 2023-10-16T09:44:11.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb799df700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb740b8910 con 0x7fcb740d29b0 2023-10-16T09:44:11.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb799df700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb740d0f00 con 0x7fcb740d0a70 2023-10-16T09:44:11.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb740d29b0 0x7fcb740d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb740d29b0 0x7fcb740d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47458/0 (socket says 172.21.15.158:47458) 2023-10-16T09:44:11.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 -- 172.21.15.158:0/3524640814 learned_addr learned my addr 172.21.15.158:0/3524640814 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:11.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb737fe700 1 --2- 172.21.15.158:0/3524640814 >> v2:172.21.15.172:3300/0 conn(0x7fcb740d0a70 0x7fcb740b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 -- 172.21.15.158:0/3524640814 >> v2:172.21.15.172:3300/0 conn(0x7fcb740d0a70 msgr2=0x7fcb740b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:11.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 --2- 172.21.15.158:0/3524640814 >> v2:172.21.15.172:3300/0 conn(0x7fcb740d0a70 0x7fcb740b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 -- 172.21.15.158:0/3524640814 >> v1:172.21.15.172:6789/0 conn(0x7fcb74160d30 legacy=0x7fcb740b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:11.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 -- 172.21.15.158:0/3524640814 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb740b7f70 con 0x7fcb740d29b0 2023-10-16T09:44:11.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb737fe700 1 --2- 172.21.15.158:0/3524640814 >> v2:172.21.15.172:3300/0 conn(0x7fcb740d0a70 0x7fcb740b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:11.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.737+0000 7fcb727fc700 1 --2- 172.21.15.158:0/3524640814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d29b0 0x7fcb740d0530 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcb640098f0 tx=0x7fcb64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e62aa2445984a0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb71ffb700 1 -- 172.21.15.158:0/3524640814 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64004230 con 0x7fcb740d29b0 2023-10-16T09:44:11.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb71ffb700 1 -- 172.21.15.158:0/3524640814 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb64004390 con 0x7fcb740d29b0 2023-10-16T09:44:11.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb71ffb700 1 -- 172.21.15.158:0/3524640814 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb640036a0 con 0x7fcb740d29b0 2023-10-16T09:44:11.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 -- 172.21.15.158:0/3524640814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d29b0 msgr2=0x7fcb740d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 --2- 172.21.15.158:0/3524640814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d29b0 0x7fcb740d0530 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcb640098f0 tx=0x7fcb64009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:11.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 -- 172.21.15.158:0/3524640814 shutdown_connections 2023-10-16T09:44:11.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 --2- 172.21.15.158:0/3524640814 >> v2:172.21.15.172:3300/0 conn(0x7fcb740d0a70 0x7fcb740b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 --2- 172.21.15.158:0/3524640814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d29b0 0x7fcb740d0530 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 -- 172.21.15.158:0/3524640814 >> 172.21.15.158:0/3524640814 conn(0x7fcb740669f0 msgr2=0x7fcb74066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:11.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 -- 172.21.15.158:0/3524640814 shutdown_connections 2023-10-16T09:44:11.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.738+0000 7fcb799df700 1 -- 172.21.15.158:0/3524640814 wait complete. 2023-10-16T09:44:11.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.739+0000 7fcb799df700 1 Processor -- start 2023-10-16T09:44:11.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb799df700 1 -- start start 2023-10-16T09:44:11.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb799df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d0a70 0x7fcb741454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb799df700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb740d29b0 0x7fcb74158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb799df700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb74160d30 0x7fcb74158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb799df700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb74146ab0 con 0x7fcb740d0a70 2023-10-16T09:44:11.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb799df700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb74145ea0 con 0x7fcb740d29b0 2023-10-16T09:44:11.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb799df700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb74146180 con 0x7fcb74160d30 2023-10-16T09:44:11.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb72ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d0a70 0x7fcb741454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb72ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d0a70 0x7fcb741454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47462/0 (socket says 172.21.15.158:47462) 2023-10-16T09:44:11.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.740+0000 7fcb72ffd700 1 -- 172.21.15.158:0/3425229821 learned_addr learned my addr 172.21.15.158:0/3425229821 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:11.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb72ffd700 1 -- 172.21.15.158:0/3425229821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb74160d30 msgr2=0x7fcb74158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb72ffd700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb74160d30 0x7fcb74158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb72ffd700 1 -- 172.21.15.158:0/3425229821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb740d29b0 msgr2=0x7fcb74158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb72ffd700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb740d29b0 0x7fcb74158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb72ffd700 1 -- 172.21.15.158:0/3425229821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb64009550 con 0x7fcb740d0a70 2023-10-16T09:44:11.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb72ffd700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d0a70 0x7fcb741454e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800b4a0 tx=0x7fcb6800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb5bfff700 1 -- 172.21.15.158:0/3425229821 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb68011830 con 0x7fcb740d0a70 2023-10-16T09:44:11.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb5bfff700 1 -- 172.21.15.158:0/3425229821 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb68011990 con 0x7fcb740d0a70 2023-10-16T09:44:11.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb74146460 con 0x7fcb740d0a70 2023-10-16T09:44:11.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb5bfff700 1 -- 172.21.15.158:0/3425229821 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb68011d40 con 0x7fcb740d0a70 2023-10-16T09:44:11.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.741+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7415a470 con 0x7fcb740d0a70 2023-10-16T09:44:11.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.742+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb740681b0 con 0x7fcb740d0a70 2023-10-16T09:44:11.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.742+0000 7fcb5bfff700 1 -- 172.21.15.158:0/3425229821 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcb6801a400 con 0x7fcb740d0a70 2023-10-16T09:44:11.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.743+0000 7fcb5bfff700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcb6006af00 0x7fcb6006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:11.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.743+0000 7fcb5bfff700 1 -- 172.21.15.158:0/3425229821 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fcb68089790 con 0x7fcb740d0a70 2023-10-16T09:44:11.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.743+0000 7fcb727fc700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcb6006af00 0x7fcb6006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:11.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.743+0000 7fcb727fc700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcb6006af00 0x7fcb6006d3c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb740674b0 tx=0x7fcb64010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:11.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.747+0000 7fcb5bfff700 1 -- 172.21.15.158:0/3425229821 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb68054430 con 0x7fcb740d0a70 2023-10-16T09:44:11.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.939+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcb740b78a0 con 0x7fcb740d0a70 2023-10-16T09:44:11.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.940+0000 7fcb5bfff700 1 -- 172.21.15.158:0/3425229821 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcb68057a50 con 0x7fcb740d0a70 2023-10-16T09:44:11.942 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcb6006af00 msgr2=0x7fcb6006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcb6006af00 0x7fcb6006d3c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb740674b0 tx=0x7fcb64010000 comp rx=0 tx=0).stop 2023-10-16T09:44:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d0a70 msgr2=0x7fcb741454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d0a70 0x7fcb741454e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800b4a0 tx=0x7fcb6800b860 comp rx=0 tx=0).stop 2023-10-16T09:44:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 shutdown_connections 2023-10-16T09:44:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcb6006af00 0x7fcb6006d3c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb74160d30 0x7fcb74158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb740d0a70 0x7fcb741454e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 --2- 172.21.15.158:0/3425229821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb740d29b0 0x7fcb74158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 >> 172.21.15.158:0/3425229821 conn(0x7fcb740669f0 msgr2=0x7fcb740cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:11.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 shutdown_connections 2023-10-16T09:44:11.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:11.942+0000 7fcb799df700 1 -- 172.21.15.158:0/3425229821 wait complete. 2023-10-16T09:44:11.966 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-10-16T09:44:11.967 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T09:44:11.968 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:12.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.086+0000 7f805c4a9700 1 Processor -- start 2023-10-16T09:44:12.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.086+0000 7f805c4a9700 1 -- start start 2023-10-16T09:44:12.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.087+0000 7f805c4a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 0x7f805406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.087+0000 7f805c4a9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8054161c80 con 0x7f8054162110 2023-10-16T09:44:12.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.087+0000 7f805c4a9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540b2bd0 con 0x7f80540cbfd0 2023-10-16T09:44:12.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.087+0000 7f805c4a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805406e090 con 0x7f805415db30 2023-10-16T09:44:12.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059a44700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8054162110 0x7f80540b1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47646/0 (socket says 172.21.15.158:47646) 2023-10-16T09:44:12.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059a44700 1 -- 172.21.15.158:0/3195340968 learned_addr learned my addr 172.21.15.158:0/3195340968 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:12.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f805a245700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 0x7f805406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562398609 0 0) 0x7f8054161c80 con 0x7f8054162110 2023-10-16T09:44:12.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80400035b0 con 0x7f8054162110 2023-10-16T09:44:12.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288141526 0 0) 0x7f80400035b0 con 0x7f8054162110 2023-10-16T09:44:12.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 msgr2=0x7f805406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 0x7f805406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 >> v1:172.21.15.172:6789/0 conn(0x7f80540cbfd0 legacy=0x7f80540b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805406ddb0 con 0x7f8054162110 2023-10-16T09:44:12.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f8050001670 con 0x7f8054162110 2023-10-16T09:44:12.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 >> v1:172.21.15.158:6789/0 conn(0x7f8054162110 legacy=0x7f80540b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f8040003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8040004850 0x7f8040004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 0x7f80400052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805406e090 con 0x7f8040003a30 2023-10-16T09:44:12.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804000bbd0 con 0x7f8040004850 2023-10-16T09:44:12.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804000bd90 con 0x7f805415db30 2023-10-16T09:44:12.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f805a245700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f8040003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f8059a44700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8040004850 0x7f8040004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.088+0000 7f805aa46700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 0x7f80400052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805a245700 1 -- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 msgr2=0x7f80400052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805a245700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f805415db30 0x7f80400052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805a245700 1 -- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8040004850 msgr2=0x7f8040004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805a245700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8040004850 0x7f8040004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805a245700 1 -- 172.21.15.158:0/3195340968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80400035b0 con 0x7f8040003a30 2023-10-16T09:44:12.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f8059a44700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8040004850 0x7f8040004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:12.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805a245700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f8040003e50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8044009b60 tx=0x7f8040005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e29ec898c43b7e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804400e6c0 con 0x7f8040003a30 2023-10-16T09:44:12.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804400e820 con 0x7f8040003a30 2023-10-16T09:44:12.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f8059243700 1 -- 172.21.15.158:0/3195340968 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804400eb90 con 0x7f8040003a30 2023-10-16T09:44:12.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805c4a9700 1 -- 172.21.15.158:0/3195340968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 msgr2=0x7f8040003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805c4a9700 1 --2- 172.21.15.158:0/3195340968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f8040003e50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8044009b60 tx=0x7f8040005880 comp rx=0 tx=0).stop 2023-10-16T09:44:12.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805aa46700 1 -- 172.21.15.158:0/3195340968 reap_dead start 2023-10-16T09:44:12.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805c4a9700 1 -- 172.21.15.158:0/3195340968 shutdown_connections 2023-10-16T09:44:12.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805c4a9700 1 -- 172.21.15.158:0/3195340968 >> 172.21.15.158:0/3195340968 conn(0x7f80540ccdb0 msgr2=0x7f80540cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:12.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.089+0000 7f805c4a9700 1 -- 172.21.15.158:0/3195340968 shutdown_connections 2023-10-16T09:44:12.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.090+0000 7f805c4a9700 1 -- 172.21.15.158:0/3195340968 wait complete. 2023-10-16T09:44:12.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 Processor -- start 2023-10-16T09:44:12.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 -- start start 2023-10-16T09:44:12.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f80540c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80540add90 0x7f805415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80540ae210 0x7f805415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80540c52f0 con 0x7f8040003a30 2023-10-16T09:44:12.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80540c6c90 con 0x7f80540ae210 2023-10-16T09:44:12.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805c4a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805415d410 con 0x7f80540add90 2023-10-16T09:44:12.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.091+0000 7f805a245700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f80540c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f80540c4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47482/0 (socket says 172.21.15.158:47482) 2023-10-16T09:44:12.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 -- 172.21.15.158:0/819938015 learned_addr learned my addr 172.21.15.158:0/819938015 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:12.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805aa46700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80540ae210 0x7f805415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f8059a44700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80540add90 0x7f805415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 -- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80540add90 msgr2=0x7f805415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80540add90 0x7f805415c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 -- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80540ae210 msgr2=0x7f805415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80540ae210 0x7f805415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 -- 172.21.15.158:0/819938015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540c4770 con 0x7f8040003a30 2023-10-16T09:44:12.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805a245700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f80540c4070 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f804000bba0 tx=0x7f80440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f804affd700 1 -- 172.21.15.158:0/819938015 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044021070 con 0x7f8040003a30 2023-10-16T09:44:12.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f804affd700 1 -- 172.21.15.158:0/819938015 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80440045d0 con 0x7f8040003a30 2023-10-16T09:44:12.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80540c4a50 con 0x7f8040003a30 2023-10-16T09:44:12.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f804affd700 1 -- 172.21.15.158:0/819938015 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804400e6c0 con 0x7f8040003a30 2023-10-16T09:44:12.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.092+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80540c5010 con 0x7f8040003a30 2023-10-16T09:44:12.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.093+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80540681b0 con 0x7f8040003a30 2023-10-16T09:44:12.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.094+0000 7f804affd700 1 -- 172.21.15.158:0/819938015 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8044025070 con 0x7f8040003a30 2023-10-16T09:44:12.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.094+0000 7f804affd700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f803806af00 0x7f803806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.094+0000 7f804affd700 1 -- 172.21.15.158:0/819938015 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f804408aca0 con 0x7f8040003a30 2023-10-16T09:44:12.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.094+0000 7f8059a44700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f803806af00 0x7f803806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.094+0000 7f8059a44700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f803806af00 0x7f803806d3c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f80500010e0 tx=0x7f8050009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.099+0000 7f804affd700 1 -- 172.21.15.158:0/819938015 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8044055af0 con 0x7f8040003a30 2023-10-16T09:44:12.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.289+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f805406da60 con 0x7f8040003a30 2023-10-16T09:44:12.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.291+0000 7f804affd700 1 -- 172.21.15.158:0/819938015 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7f804402c090 con 0x7f8040003a30 2023-10-16T09:44:12.293 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:12.293 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:12.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f803806af00 msgr2=0x7f803806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f803806af00 0x7f803806d3c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f80500010e0 tx=0x7f8050009040 comp rx=0 tx=0).stop 2023-10-16T09:44:12.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 msgr2=0x7f80540c4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f80540c4070 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f804000bba0 tx=0x7f80440047c0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 shutdown_connections 2023-10-16T09:44:12.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80540ae210 0x7f805415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8040003a30 0x7f80540c4070 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f803806af00 0x7f803806d3c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 --2- 172.21.15.158:0/819938015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80540add90 0x7f805415c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 >> 172.21.15.158:0/819938015 conn(0x7f80540ccdb0 msgr2=0x7f80540b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:12.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 shutdown_connections 2023-10-16T09:44:12.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.293+0000 7f805c4a9700 1 -- 172.21.15.158:0/819938015 wait complete. 2023-10-16T09:44:12.307 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:12.307 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.430+0000 7efd58a9c700 1 Processor -- start 2023-10-16T09:44:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.430+0000 7efd58a9c700 1 -- start start 2023-10-16T09:44:12.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd58a9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd540cc200 0x7efd5415f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd58a9c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd540cf4c0 con 0x7efd540ce510 2023-10-16T09:44:12.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd58a9c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5415f710 con 0x7efd5415b590 2023-10-16T09:44:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd58a9c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd540b49e0 con 0x7efd540cc200 2023-10-16T09:44:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd51d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efd540ce510 0x7efd540cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47668/0 (socket says 172.21.15.158:47668) 2023-10-16T09:44:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd51d9b700 1 -- 172.21.15.158:0/1731051764 learned_addr learned my addr 172.21.15.158:0/1731051764 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:12.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd52d9d700 1 --2- 172.21.15.158:0/1731051764 >> v2:172.21.15.172:3300/0 conn(0x7efd540cc200 0x7efd5415f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.431+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840023677 0 0) 0x7efd540cf4c0 con 0x7efd540ce510 2023-10-16T09:44:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c0035b0 con 0x7efd540ce510 2023-10-16T09:44:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291795375 0 0) 0x7efd3c0035b0 con 0x7efd540ce510 2023-10-16T09:44:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 >> v2:172.21.15.172:3300/0 conn(0x7efd540cc200 msgr2=0x7efd5415f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 --2- 172.21.15.158:0/1731051764 >> v2:172.21.15.172:3300/0 conn(0x7efd540cc200 0x7efd5415f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 >> v1:172.21.15.172:6789/0 conn(0x7efd5415b590 legacy=0x7efd540b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5415f970 con 0x7efd540ce510 2023-10-16T09:44:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7efd44001670 con 0x7efd540ce510 2023-10-16T09:44:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd52d9d700 1 --2- 172.21.15.158:0/1731051764 >> v2:172.21.15.172:3300/0 conn(0x7efd540cc200 0x7efd5415f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 >> v1:172.21.15.158:6789/0 conn(0x7efd540ce510 legacy=0x7efd540cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd3c003a30 0x7efd3c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd3c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd3c005260 0x7efd3c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd540b49e0 con 0x7efd3c004850 2023-10-16T09:44:12.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c00c190 con 0x7efd3c005260 2023-10-16T09:44:12.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c00c350 con 0x7efd3c003a30 2023-10-16T09:44:12.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd51d9b700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd3c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5259c700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd3c003a30 0x7efd3c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd52d9d700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd3c005260 0x7efd3c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd51d9b700 1 -- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd3c003a30 msgr2=0x7efd3c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd51d9b700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd3c003a30 0x7efd3c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd51d9b700 1 -- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd3c005260 msgr2=0x7efd3c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd51d9b700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd3c005260 0x7efd3c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd51d9b700 1 -- 172.21.15.158:0/1731051764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c0035b0 con 0x7efd3c004850 2023-10-16T09:44:12.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd52d9d700 1 -- 172.21.15.158:0/1731051764 reap_dead start 2023-10-16T09:44:12.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd51d9b700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd3c004cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efd4400c240 tx=0x7efd3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5bbfedefbfb858 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.432+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44007c10 con 0x7efd3c004850 2023-10-16T09:44:12.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4400cdc0 con 0x7efd3c004850 2023-10-16T09:44:12.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd5159a700 1 -- 172.21.15.158:0/1731051764 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44005ba0 con 0x7efd3c004850 2023-10-16T09:44:12.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd58a9c700 1 -- 172.21.15.158:0/1731051764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 msgr2=0x7efd3c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd3c004cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efd4400c240 tx=0x7efd3c005880 comp rx=0 tx=0).stop 2023-10-16T09:44:12.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd58a9c700 1 -- 172.21.15.158:0/1731051764 shutdown_connections 2023-10-16T09:44:12.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1731051764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd3c004cd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd58a9c700 1 -- 172.21.15.158:0/1731051764 >> 172.21.15.158:0/1731051764 conn(0x7efd5406a8d0 msgr2=0x7efd5406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:12.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd58a9c700 1 -- 172.21.15.158:0/1731051764 shutdown_connections 2023-10-16T09:44:12.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.433+0000 7efd58a9c700 1 -- 172.21.15.158:0/1731051764 wait complete. 2023-10-16T09:44:12.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 Processor -- start 2023-10-16T09:44:12.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 -- start start 2023-10-16T09:44:12.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd541f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd540afdd0 0x7efd540ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd540cac80 0x7efd540a8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd540cb0f0 con 0x7efd3c004850 2023-10-16T09:44:12.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd541f1ce0 con 0x7efd540cac80 2023-10-16T09:44:12.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd58a9c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5415bf10 con 0x7efd540afdd0 2023-10-16T09:44:12.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd5259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd541f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd5259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd541f17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47502/0 (socket says 172.21.15.158:47502) 2023-10-16T09:44:12.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.435+0000 7efd5259c700 1 -- 172.21.15.158:0/1302845383 learned_addr learned my addr 172.21.15.158:0/1302845383 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:12.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd5259c700 1 -- 172.21.15.158:0/1302845383 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd540afdd0 msgr2=0x7efd540ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd5259c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd540afdd0 0x7efd540ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd5259c700 1 -- 172.21.15.158:0/1302845383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd540cac80 msgr2=0x7efd540a8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd5259c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd540cac80 0x7efd540a8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd5259c700 1 -- 172.21.15.158:0/1302845383 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5415c1f0 con 0x7efd3c004850 2023-10-16T09:44:12.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd5259c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd541f17a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f820 tx=0x7efd4800fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd42ffd700 1 -- 172.21.15.158:0/1302845383 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd480052e0 con 0x7efd3c004850 2023-10-16T09:44:12.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd42ffd700 1 -- 172.21.15.158:0/1302845383 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd48005440 con 0x7efd3c004850 2023-10-16T09:44:12.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5415c4d0 con 0x7efd3c004850 2023-10-16T09:44:12.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd42ffd700 1 -- 172.21.15.158:0/1302845383 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd480038a0 con 0x7efd3c004850 2023-10-16T09:44:12.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.436+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5415ca90 con 0x7efd3c004850 2023-10-16T09:44:12.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.437+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd540681b0 con 0x7efd3c004850 2023-10-16T09:44:12.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.437+0000 7efd42ffd700 1 -- 172.21.15.158:0/1302845383 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efd48003a00 con 0x7efd3c004850 2023-10-16T09:44:12.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.438+0000 7efd42ffd700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efd3406af00 0x7efd3406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.438+0000 7efd42ffd700 1 -- 172.21.15.158:0/1302845383 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7efd48089750 con 0x7efd3c004850 2023-10-16T09:44:12.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.438+0000 7efd51d9b700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efd3406af00 0x7efd3406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.438+0000 7efd51d9b700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efd3406af00 0x7efd3406d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd44001930 tx=0x7efd4400a000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.442+0000 7efd42ffd700 1 -- 172.21.15.158:0/1302845383 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd480543f0 con 0x7efd3c004850 2023-10-16T09:44:12.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.635+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd5415af40 con 0x7efd3c004850 2023-10-16T09:44:12.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.635+0000 7efd42ffd700 1 -- 172.21.15.158:0/1302845383 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd48003cb0 con 0x7efd3c004850 2023-10-16T09:44:12.637 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:12.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efd3406af00 msgr2=0x7efd3406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efd3406af00 0x7efd3406d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd44001930 tx=0x7efd4400a000 comp rx=0 tx=0).stop 2023-10-16T09:44:12.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 msgr2=0x7efd541f17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd541f17a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f820 tx=0x7efd4800fbe0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 shutdown_connections 2023-10-16T09:44:12.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd540cac80 0x7efd540a8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd3c004850 0x7efd541f17a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efd3406af00 0x7efd3406d3c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 --2- 172.21.15.158:0/1302845383 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd540afdd0 0x7efd540ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 >> 172.21.15.158:0/1302845383 conn(0x7efd5406a8d0 msgr2=0x7efd540b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:12.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 shutdown_connections 2023-10-16T09:44:12.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.637+0000 7efd58a9c700 1 -- 172.21.15.158:0/1302845383 wait complete. 2023-10-16T09:44:12.651 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:12.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.773+0000 7fad7e15d700 1 Processor -- start 2023-10-16T09:44:12.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.773+0000 7fad7e15d700 1 -- start start 2023-10-16T09:44:12.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.773+0000 7fad7e15d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 0x7fad780caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.773+0000 7fad7e15d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fad780cc590 0x7fad780afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad7e15d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780cbd30 con 0x7fad7815da40 2023-10-16T09:44:12.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad7e15d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad780cc1c0 con 0x7fad780cc590 2023-10-16T09:44:12.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad7e15d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7815de20 con 0x7fad7806afe0 2023-10-16T09:44:12.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad777fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fad7815da40 0x7fad7806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47686/0 (socket says 172.21.15.158:47686) 2023-10-16T09:44:12.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad777fe700 1 -- 172.21.15.158:0/2815616465 learned_addr learned my addr 172.21.15.158:0/2815616465 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:12.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383192870 0 0) 0x7fad780cbd30 con 0x7fad7815da40 2023-10-16T09:44:12.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad600035b0 con 0x7fad7815da40 2023-10-16T09:44:12.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad76ffd700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 0x7fad780caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad77fff700 1 --2- 172.21.15.158:0/2815616465 >> v2:172.21.15.172:3300/0 conn(0x7fad780cc590 0x7fad780afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330014475 0 0) 0x7fad600035b0 con 0x7fad7815da40 2023-10-16T09:44:12.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 msgr2=0x7fad780caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 0x7fad780caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 >> v2:172.21.15.172:3300/0 conn(0x7fad780cc590 msgr2=0x7fad780afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 --2- 172.21.15.158:0/2815616465 >> v2:172.21.15.172:3300/0 conn(0x7fad780cc590 0x7fad780afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780ce260 con 0x7fad7815da40 2023-10-16T09:44:12.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fad68001670 con 0x7fad7815da40 2023-10-16T09:44:12.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad77fff700 1 --2- 172.21.15.158:0/2815616465 >> v2:172.21.15.172:3300/0 conn(0x7fad780cc590 0x7fad780afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:12.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 >> v1:172.21.15.158:6789/0 conn(0x7fad7815da40 legacy=0x7fad7806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad60003a00 0x7fad60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad60004850 0x7fad60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 0x7fad60005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad780cc1c0 con 0x7fad60004850 2023-10-16T09:44:12.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7815de20 con 0x7fad60003a00 2023-10-16T09:44:12.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6000ba90 con 0x7fad7806afe0 2023-10-16T09:44:12.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad76ffd700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad60004850 0x7fad60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.774+0000 7fad777fe700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad60003a00 0x7fad60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad77fff700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 0x7fad60005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad76ffd700 1 -- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 msgr2=0x7fad60005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad76ffd700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad7806afe0 0x7fad60005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad76ffd700 1 -- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad60003a00 msgr2=0x7fad60003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:12.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad76ffd700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad60003a00 0x7fad60003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad76ffd700 1 -- 172.21.15.158:0/2815616465 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad600035b0 con 0x7fad60004850 2023-10-16T09:44:12.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad76ffd700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad60004850 0x7fad60004cd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009b60 tx=0x7fad60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbabdf1532d94a69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c00e6c0 con 0x7fad60004850 2023-10-16T09:44:12.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c00e820 con 0x7fad60004850 2023-10-16T09:44:12.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad767fc700 1 -- 172.21.15.158:0/2815616465 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c00eb90 con 0x7fad60004850 2023-10-16T09:44:12.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad7e15d700 1 -- 172.21.15.158:0/2815616465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad60004850 msgr2=0x7fad60004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad7e15d700 1 --2- 172.21.15.158:0/2815616465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad60004850 0x7fad60004cd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009b60 tx=0x7fad60005880 comp rx=0 tx=0).stop 2023-10-16T09:44:12.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad77fff700 1 -- 172.21.15.158:0/2815616465 reap_dead start 2023-10-16T09:44:12.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad7e15d700 1 -- 172.21.15.158:0/2815616465 shutdown_connections 2023-10-16T09:44:12.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.775+0000 7fad7e15d700 1 -- 172.21.15.158:0/2815616465 >> 172.21.15.158:0/2815616465 conn(0x7fad78099a10 msgr2=0x7fad780d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:12.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.776+0000 7fad7e15d700 1 -- 172.21.15.158:0/2815616465 shutdown_connections 2023-10-16T09:44:12.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.776+0000 7fad7e15d700 1 -- 172.21.15.158:0/2815616465 wait complete. 2023-10-16T09:44:12.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.777+0000 7fad7e15d700 1 Processor -- start 2023-10-16T09:44:12.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.777+0000 7fad7e15d700 1 -- start start 2023-10-16T09:44:12.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad60004850 0x7fad781493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad781f2d60 0x7fad78158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad78159450 0x7fad781e6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad781498f0 con 0x7fad78159450 2023-10-16T09:44:12.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78159a80 con 0x7fad781f2d60 2023-10-16T09:44:12.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7815a710 con 0x7fad60004850 2023-10-16T09:44:12.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad78159450 0x7fad781e6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad78159450 0x7fad781e6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47526/0 (socket says 172.21.15.158:47526) 2023-10-16T09:44:12.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 -- 172.21.15.158:0/1509154562 learned_addr learned my addr 172.21.15.158:0/1509154562 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:12.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad777fe700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad60004850 0x7fad781493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad76ffd700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad781f2d60 0x7fad78158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 -- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad60004850 msgr2=0x7fad781493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad60004850 0x7fad781493b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 -- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad781f2d60 msgr2=0x7fad78158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad781f2d60 0x7fad78158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 -- 172.21.15.158:0/1509154562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78149d40 con 0x7fad78159450 2023-10-16T09:44:12.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad77fff700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad78159450 0x7fad781e6f50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fad7806c970 tx=0x7fad70010440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad5ffff700 1 -- 172.21.15.158:0/1509154562 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7001b070 con 0x7fad78159450 2023-10-16T09:44:12.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad5ffff700 1 -- 172.21.15.158:0/1509154562 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad70003a20 con 0x7fad78159450 2023-10-16T09:44:12.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad5ffff700 1 -- 172.21.15.158:0/1509154562 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70003bb0 con 0x7fad78159450 2023-10-16T09:44:12.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad78159de0 con 0x7fad78159450 2023-10-16T09:44:12.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.778+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7815a3a0 con 0x7fad78159450 2023-10-16T09:44:12.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.779+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7815a530 con 0x7fad78159450 2023-10-16T09:44:12.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.780+0000 7fad5ffff700 1 -- 172.21.15.158:0/1509154562 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fad70003d10 con 0x7fad78159450 2023-10-16T09:44:12.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.780+0000 7fad5ffff700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fad6406af00 0x7fad6406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:12.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.780+0000 7fad777fe700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fad6406af00 0x7fad6406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:12.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.780+0000 7fad5ffff700 1 -- 172.21.15.158:0/1509154562 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fad70089f10 con 0x7fad78159450 2023-10-16T09:44:12.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.781+0000 7fad777fe700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fad6406af00 0x7fad6406d3c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad68001cf0 tx=0x7fad68007980 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:12.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.785+0000 7fad5ffff700 1 -- 172.21.15.158:0/1509154562 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fad70054bb0 con 0x7fad78159450 2023-10-16T09:44:12.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.977+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad780681b0 con 0x7fad78159450 2023-10-16T09:44:12.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.977+0000 7fad5ffff700 1 -- 172.21.15.158:0/1509154562 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad700581d0 con 0x7fad78159450 2023-10-16T09:44:12.979 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:12.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fad6406af00 msgr2=0x7fad6406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fad6406af00 0x7fad6406d3c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad68001cf0 tx=0x7fad68007980 comp rx=0 tx=0).stop 2023-10-16T09:44:12.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad78159450 msgr2=0x7fad781e6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:12.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad78159450 0x7fad781e6f50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fad7806c970 tx=0x7fad70010440 comp rx=0 tx=0).stop 2023-10-16T09:44:12.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 shutdown_connections 2023-10-16T09:44:12.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad78159450 0x7fad781e6f50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fad6406af00 0x7fad6406d3c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad60004850 0x7fad781493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 --2- 172.21.15.158:0/1509154562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad781f2d60 0x7fad78158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:12.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 >> 172.21.15.158:0/1509154562 conn(0x7fad78099a10 msgr2=0x7fad780c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:12.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 shutdown_connections 2023-10-16T09:44:12.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:12.979+0000 7fad7e15d700 1 -- 172.21.15.158:0/1509154562 wait complete. 2023-10-16T09:44:12.993 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:13.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.115+0000 7f85126de700 1 Processor -- start 2023-10-16T09:44:13.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.115+0000 7f85126de700 1 -- start start 2023-10-16T09:44:13.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f85126de700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15da40 0x7f850c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f85126de700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f850c06afe0 0x7f850c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f85126de700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0cbd30 con 0x7f850c0cc590 2023-10-16T09:44:13.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f85126de700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0cc1c0 con 0x7f850c06afe0 2023-10-16T09:44:13.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f85126de700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c15de20 con 0x7f850c15da40 2023-10-16T09:44:13.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f850c06afe0 0x7f850c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f850c06afe0 0x7f850c0afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47532/0 (socket says 172.21.15.158:47532) 2023-10-16T09:44:13.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 -- 172.21.15.158:0/4078522283 learned_addr learned my addr 172.21.15.158:0/4078522283 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:13.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f850bfff700 1 --2- 172.21.15.158:0/4078522283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15da40 0x7f850c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 -- 172.21.15.158:0/4078522283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15da40 msgr2=0x7f850c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:13.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 --2- 172.21.15.158:0/4078522283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15da40 0x7f850c0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 -- 172.21.15.158:0/4078522283 >> v1:172.21.15.172:6789/0 conn(0x7f850c0cc590 legacy=0x7f850c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 -- 172.21.15.158:0/4078522283 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0ce260 con 0x7f850c06afe0 2023-10-16T09:44:13.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f8510c7b700 1 --2- 172.21.15.158:0/4078522283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06afe0 0x7f850c0afa40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8504009e50 tx=0x7f850400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9361e8d7b6e51e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.116+0000 7f850affd700 1 -- 172.21.15.158:0/4078522283 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850400e070 con 0x7f850c06afe0 2023-10-16T09:44:13.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f850affd700 1 -- 172.21.15.158:0/4078522283 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850400b910 con 0x7f850c06afe0 2023-10-16T09:44:13.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f850affd700 1 -- 172.21.15.158:0/4078522283 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850400bcc0 con 0x7f850c06afe0 2023-10-16T09:44:13.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 -- 172.21.15.158:0/4078522283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06afe0 msgr2=0x7f850c0afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 --2- 172.21.15.158:0/4078522283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06afe0 0x7f850c0afa40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8504009e50 tx=0x7f850400b330 comp rx=0 tx=0).stop 2023-10-16T09:44:13.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 -- 172.21.15.158:0/4078522283 shutdown_connections 2023-10-16T09:44:13.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 --2- 172.21.15.158:0/4078522283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06afe0 0x7f850c0afa40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 --2- 172.21.15.158:0/4078522283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15da40 0x7f850c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 -- 172.21.15.158:0/4078522283 >> 172.21.15.158:0/4078522283 conn(0x7f850c099a10 msgr2=0x7f850c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:13.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 -- 172.21.15.158:0/4078522283 shutdown_connections 2023-10-16T09:44:13.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.117+0000 7f85126de700 1 -- 172.21.15.158:0/4078522283 wait complete. 2023-10-16T09:44:13.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.118+0000 7f85126de700 1 Processor -- start 2023-10-16T09:44:13.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.118+0000 7f85126de700 1 -- start start 2023-10-16T09:44:13.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f85126de700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c06afe0 0x7f850c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f85126de700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c0cc590 0x7f850c0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f85126de700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15da40 0x7f850c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f85126de700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1f2a70 con 0x7f850c0cc590 2023-10-16T09:44:13.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f85126de700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1f2ce0 con 0x7f850c15da40 2023-10-16T09:44:13.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f85126de700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1f2fc0 con 0x7f850c06afe0 2023-10-16T09:44:13.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f850b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c0cc590 0x7f850c0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f850b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c0cc590 0x7f850c0b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47536/0 (socket says 172.21.15.158:47536) 2023-10-16T09:44:13.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f850b7fe700 1 -- 172.21.15.158:0/1462425442 learned_addr learned my addr 172.21.15.158:0/1462425442 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:13.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f850bfff700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c06afe0 0x7f850c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.119+0000 7f8510c7b700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15da40 0x7f850c0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f850b7fe700 1 -- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c06afe0 msgr2=0x7f850c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f850b7fe700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c06afe0 0x7f850c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f850b7fe700 1 -- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15da40 msgr2=0x7f850c0b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f850b7fe700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15da40 0x7f850c0b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f850b7fe700 1 -- 172.21.15.158:0/1462425442 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8504009b30 con 0x7f850c0cc590 2023-10-16T09:44:13.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f8510c7b700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15da40 0x7f850c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:13.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f850b7fe700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c0cc590 0x7f850c0b3480 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c4e0 tx=0x7f84fc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f8508ff9700 1 -- 172.21.15.158:0/1462425442 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc0052d0 con 0x7f850c0cc590 2023-10-16T09:44:13.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f8508ff9700 1 -- 172.21.15.158:0/1462425442 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc005430 con 0x7f850c0cc590 2023-10-16T09:44:13.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c1f32a0 con 0x7f850c0cc590 2023-10-16T09:44:13.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f8508ff9700 1 -- 172.21.15.158:0/1462425442 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc0038c0 con 0x7f850c0cc590 2023-10-16T09:44:13.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.120+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c1f3860 con 0x7f850c0cc590 2023-10-16T09:44:13.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.121+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c0681b0 con 0x7f850c0cc590 2023-10-16T09:44:13.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.121+0000 7f8508ff9700 1 -- 172.21.15.158:0/1462425442 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f84fc003a20 con 0x7f850c0cc590 2023-10-16T09:44:13.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.122+0000 7f8508ff9700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f84f806af00 0x7f84f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.122+0000 7f850bfff700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f84f806af00 0x7f84f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.122+0000 7f8508ff9700 1 -- 172.21.15.158:0/1462425442 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f84fc088db0 con 0x7f850c0cc590 2023-10-16T09:44:13.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.122+0000 7f850bfff700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f84f806af00 0x7f84f806d3c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8500009b40 tx=0x7f8500009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.126+0000 7f8508ff9700 1 -- 172.21.15.158:0/1462425442 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84fc053ae0 con 0x7f850c0cc590 2023-10-16T09:44:13.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.318+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f850c0b5ac0 con 0x7f850c0cc590 2023-10-16T09:44:13.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.319+0000 7f8508ff9700 1 -- 172.21.15.158:0/1462425442 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84fc0170a0 con 0x7f850c0cc590 2023-10-16T09:44:13.321 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:13.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f84f806af00 msgr2=0x7f84f806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f84f806af00 0x7f84f806d3c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8500009b40 tx=0x7f8500009210 comp rx=0 tx=0).stop 2023-10-16T09:44:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c0cc590 msgr2=0x7f850c0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c0cc590 0x7f850c0b3480 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c4e0 tx=0x7f84fc00c7f0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 shutdown_connections 2023-10-16T09:44:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15da40 0x7f850c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f84f806af00 0x7f84f806d3c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c06afe0 0x7f850c0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 --2- 172.21.15.158:0/1462425442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c0cc590 0x7f850c0b3480 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 >> 172.21.15.158:0/1462425442 conn(0x7f850c099a10 msgr2=0x7f850c15c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 shutdown_connections 2023-10-16T09:44:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.321+0000 7f85126de700 1 -- 172.21.15.158:0/1462425442 wait complete. 2023-10-16T09:44:13.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:13.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.456+0000 7fa19bdd5700 1 Processor -- start 2023-10-16T09:44:13.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.456+0000 7fa19bdd5700 1 -- start start 2023-10-16T09:44:13.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa19bdd5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa194159280 0x7fa19406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa19bdd5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa19415c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa19bdd5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa1940cbef0 0x7fa1940cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa19bdd5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1940cb710 con 0x7fa194159280 2023-10-16T09:44:13.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa19bdd5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa194159660 con 0x7fa1940cbef0 2023-10-16T09:44:13.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa19bdd5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19406b120 con 0x7fa1940d0e10 2023-10-16T09:44:13.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa194159280 0x7fa19406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa194159280 0x7fa19406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47542/0 (socket says 172.21.15.158:47542) 2023-10-16T09:44:13.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 -- 172.21.15.158:0/2298599312 learned_addr learned my addr 172.21.15.158:0/2298599312 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:13.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199370700 1 --2- 172.21.15.158:0/2298599312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa19415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 -- 172.21.15.158:0/2298599312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 msgr2=0x7fa19415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:13.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa19a372700 1 --2- 172.21.15.158:0/2298599312 >> v2:172.21.15.172:3300/0 conn(0x7fa1940cbef0 0x7fa1940cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 --2- 172.21.15.158:0/2298599312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa19415c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 -- 172.21.15.158:0/2298599312 >> v2:172.21.15.172:3300/0 conn(0x7fa1940cbef0 msgr2=0x7fa1940cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:13.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 --2- 172.21.15.158:0/2298599312 >> v2:172.21.15.172:3300/0 conn(0x7fa1940cbef0 0x7fa1940cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 -- 172.21.15.158:0/2298599312 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19406b400 con 0x7fa194159280 2023-10-16T09:44:13.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa199b71700 1 --2- 172.21.15.158:0/2298599312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa19406a8d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa1940729e0 tx=0x7fa190009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0708b7a77ea7106 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.457+0000 7fa198b6f700 1 -- 172.21.15.158:0/2298599312 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19000b870 con 0x7fa194159280 2023-10-16T09:44:13.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa198b6f700 1 -- 172.21.15.158:0/2298599312 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19000b9d0 con 0x7fa194159280 2023-10-16T09:44:13.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa198b6f700 1 -- 172.21.15.158:0/2298599312 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19000bd60 con 0x7fa194159280 2023-10-16T09:44:13.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 -- 172.21.15.158:0/2298599312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 msgr2=0x7fa19406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/2298599312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa19406a8d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa1940729e0 tx=0x7fa190009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 -- 172.21.15.158:0/2298599312 shutdown_connections 2023-10-16T09:44:13.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/2298599312 >> v2:172.21.15.172:3300/0 conn(0x7fa1940cbef0 0x7fa1940cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/2298599312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa19406a8d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/2298599312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa19415c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 -- 172.21.15.158:0/2298599312 >> 172.21.15.158:0/2298599312 conn(0x7fa194099a10 msgr2=0x7fa1940d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:13.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 -- 172.21.15.158:0/2298599312 shutdown_connections 2023-10-16T09:44:13.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.458+0000 7fa19bdd5700 1 -- 172.21.15.158:0/2298599312 wait complete. 2023-10-16T09:44:13.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.460+0000 7fa19bdd5700 1 Processor -- start 2023-10-16T09:44:13.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.460+0000 7fa19bdd5700 1 -- start start 2023-10-16T09:44:13.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19bdd5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1940cbef0 0x7fa1940af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19bdd5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa1941568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19bdd5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa1940b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19bdd5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1940ae4c0 con 0x7fa194159280 2023-10-16T09:44:13.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19bdd5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1940adfb0 con 0x7fa1940cbef0 2023-10-16T09:44:13.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19bdd5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1940ae780 con 0x7fa1940d0e10 2023-10-16T09:44:13.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa1940b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa1940b5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47552/0 (socket says 172.21.15.158:47552) 2023-10-16T09:44:13.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 -- 172.21.15.158:0/397259750 learned_addr learned my addr 172.21.15.158:0/397259750 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:13.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa199b71700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1940cbef0 0x7fa1940af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa199370700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa1941568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 -- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 msgr2=0x7fa1941568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa1941568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 -- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1940cbef0 msgr2=0x7fa1940af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1940cbef0 0x7fa1940af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 -- 172.21.15.158:0/397259750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190009550 con 0x7fa194159280 2023-10-16T09:44:13.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19a372700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa1940b5520 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00cb80 tx=0x7fa18c00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa18a7fc700 1 -- 172.21.15.158:0/397259750 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c00a820 con 0x7fa194159280 2023-10-16T09:44:13.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa18a7fc700 1 -- 172.21.15.158:0/397259750 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c00a980 con 0x7fa194159280 2023-10-16T09:44:13.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1940aea60 con 0x7fa194159280 2023-10-16T09:44:13.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.461+0000 7fa18a7fc700 1 -- 172.21.15.158:0/397259750 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c00ad70 con 0x7fa194159280 2023-10-16T09:44:13.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.462+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa194156df0 con 0x7fa194159280 2023-10-16T09:44:13.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.462+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1940681b0 con 0x7fa194159280 2023-10-16T09:44:13.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.463+0000 7fa18a7fc700 1 -- 172.21.15.158:0/397259750 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa18c011410 con 0x7fa194159280 2023-10-16T09:44:13.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.463+0000 7fa18a7fc700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa17806af00 0x7fa17806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.463+0000 7fa199b71700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa17806af00 0x7fa17806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.463+0000 7fa18a7fc700 1 -- 172.21.15.158:0/397259750 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fa18c089600 con 0x7fa194159280 2023-10-16T09:44:13.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.464+0000 7fa199b71700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa17806af00 0x7fa17806d3c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa190000c00 tx=0x7fa190008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.468+0000 7fa18a7fc700 1 -- 172.21.15.158:0/397259750 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa18c054200 con 0x7fa194159280 2023-10-16T09:44:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.660+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa1940b2d30 con 0x7fa194159280 2023-10-16T09:44:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.660+0000 7fa18a7fc700 1 -- 172.21.15.158:0/397259750 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa18c057820 con 0x7fa194159280 2023-10-16T09:44:13.662 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa17806af00 msgr2=0x7fa17806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa17806af00 0x7fa17806d3c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa190000c00 tx=0x7fa190008030 comp rx=0 tx=0).stop 2023-10-16T09:44:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 msgr2=0x7fa1940b5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa1940b5520 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00cb80 tx=0x7fa18c00ce90 comp rx=0 tx=0).stop 2023-10-16T09:44:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 shutdown_connections 2023-10-16T09:44:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa194159280 0x7fa1940b5520 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1940cbef0 0x7fa1940af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa17806af00 0x7fa17806d3c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 --2- 172.21.15.158:0/397259750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1940d0e10 0x7fa1941568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 >> 172.21.15.158:0/397259750 conn(0x7fa194099a10 msgr2=0x7fa194159fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:13.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 shutdown_connections 2023-10-16T09:44:13.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.662+0000 7fa19bdd5700 1 -- 172.21.15.158:0/397259750 wait complete. 2023-10-16T09:44:13.676 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T09:44:13.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.796+0000 7f901d5af700 1 Processor -- start 2023-10-16T09:44:13.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.796+0000 7f901d5af700 1 -- start start 2023-10-16T09:44:13.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f901d5af700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f901806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f901d5af700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90180cddb0 0x7f90180ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f901d5af700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180ce760 con 0x7f901815db90 2023-10-16T09:44:13.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f901d5af700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90180d0fb0 con 0x7f90180cddb0 2023-10-16T09:44:13.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f901d5af700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901815e100 con 0x7f901815d7b0 2023-10-16T09:44:13.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90167fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f901815db90 0x7f90180b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47732/0 (socket says 172.21.15.158:47732) 2023-10-16T09:44:13.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90167fc700 1 -- 172.21.15.158:0/1307928169 learned_addr learned my addr 172.21.15.158:0/1307928169 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:13.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90177fe700 1 --2- 172.21.15.158:0/1307928169 >> v2:172.21.15.158:3300/0 conn(0x7f90180cddb0 0x7f90180ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f9016ffd700 1 --2- 172.21.15.158:0/1307928169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f901806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90177fe700 1 -- 172.21.15.158:0/1307928169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 msgr2=0x7f901806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:13.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90177fe700 1 --2- 172.21.15.158:0/1307928169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f901806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90177fe700 1 -- 172.21.15.158:0/1307928169 >> v1:172.21.15.158:6789/0 conn(0x7f901815db90 legacy=0x7f90180b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90177fe700 1 -- 172.21.15.158:0/1307928169 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901815e3e0 con 0x7f90180cddb0 2023-10-16T09:44:13.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f9015ffb700 1 -- 172.21.15.158:0/1307928169 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637128009 0 0) 0x7f90180ce760 con 0x7f901815db90 2023-10-16T09:44:13.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f90177fe700 1 --2- 172.21.15.158:0/1307928169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90180cddb0 0x7f90180ce220 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9010009d90 tx=0x7f901000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4295069d81780ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f9015ffb700 1 -- 172.21.15.158:0/1307928169 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901000e070 con 0x7f90180cddb0 2023-10-16T09:44:13.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f9015ffb700 1 -- 172.21.15.158:0/1307928169 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901000bbd0 con 0x7f90180cddb0 2023-10-16T09:44:13.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.797+0000 7f9015ffb700 1 -- 172.21.15.158:0/1307928169 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010012430 con 0x7f90180cddb0 2023-10-16T09:44:13.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 -- 172.21.15.158:0/1307928169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90180cddb0 msgr2=0x7f90180ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 --2- 172.21.15.158:0/1307928169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90180cddb0 0x7f90180ce220 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9010009d90 tx=0x7f901000b330 comp rx=0 tx=0).stop 2023-10-16T09:44:13.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 -- 172.21.15.158:0/1307928169 shutdown_connections 2023-10-16T09:44:13.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 --2- 172.21.15.158:0/1307928169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90180cddb0 0x7f90180ce220 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 --2- 172.21.15.158:0/1307928169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f901806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 -- 172.21.15.158:0/1307928169 >> 172.21.15.158:0/1307928169 conn(0x7f9018071220 msgr2=0x7f901806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:13.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 -- 172.21.15.158:0/1307928169 shutdown_connections 2023-10-16T09:44:13.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.798+0000 7f901d5af700 1 -- 172.21.15.158:0/1307928169 wait complete. 2023-10-16T09:44:13.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.799+0000 7f901d5af700 1 Processor -- start 2023-10-16T09:44:13.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.799+0000 7f901d5af700 1 -- start start 2023-10-16T09:44:13.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f901d5af700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90180cddb0 0x7f90180af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f901d5af700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f90180ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f901d5af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f901815db90 0x7f901814d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f901d5af700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90180b0760 con 0x7f901815db90 2023-10-16T09:44:13.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f901d5af700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90180ad830 con 0x7f90180cddb0 2023-10-16T09:44:13.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f901d5af700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901815ee10 con 0x7f901815d7b0 2023-10-16T09:44:13.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f901815db90 0x7f901814d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f901815db90 0x7f901814d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47566/0 (socket says 172.21.15.158:47566) 2023-10-16T09:44:13.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 -- 172.21.15.158:0/1647502778 learned_addr learned my addr 172.21.15.158:0/1647502778 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:13.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f9016ffd700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90180cddb0 0x7f90180af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90167fc700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f90180ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 -- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 msgr2=0x7f90180ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f90180ad2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 -- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90180cddb0 msgr2=0x7f90180af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:13.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90180cddb0 0x7f90180af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:13.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.800+0000 7f90177fe700 1 -- 172.21.15.158:0/1647502778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010009a70 con 0x7f901815db90 2023-10-16T09:44:13.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.801+0000 7f90177fe700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f901815db90 0x7f901814d750 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f90180ceab0 tx=0x7f9010004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.801+0000 7f8fff7fe700 1 -- 172.21.15.158:0/1647502778 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010003ea0 con 0x7f901815db90 2023-10-16T09:44:13.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.801+0000 7f8fff7fe700 1 -- 172.21.15.158:0/1647502778 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9010020070 con 0x7f901815db90 2023-10-16T09:44:13.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.801+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90180afec0 con 0x7f901815db90 2023-10-16T09:44:13.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.801+0000 7f8fff7fe700 1 -- 172.21.15.158:0/1647502778 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010004280 con 0x7f901815db90 2023-10-16T09:44:13.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.801+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90180b0480 con 0x7f901815db90 2023-10-16T09:44:13.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.802+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901815ebe0 con 0x7f901815db90 2023-10-16T09:44:13.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.802+0000 7f8fff7fe700 1 -- 172.21.15.158:0/1647502778 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f90100043e0 con 0x7f901815db90 2023-10-16T09:44:13.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.803+0000 7f8fff7fe700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f900406af00 0x7f900406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:13.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.803+0000 7f9016ffd700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f900406af00 0x7f900406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:13.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.803+0000 7f8fff7fe700 1 -- 172.21.15.158:0/1647502778 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f9010022070 con 0x7f901815db90 2023-10-16T09:44:13.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.803+0000 7f9016ffd700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f900406af00 0x7f900406d3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90180cf510 tx=0x7f9008008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:13.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:13.807+0000 7f8fff7fe700 1 -- 172.21.15.158:0/1647502778 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f901815ebe0 con 0x7f901815db90 2023-10-16T09:44:14.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.081+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f901815ebe0 con 0x7f901815db90 2023-10-16T09:44:14.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.505+0000 7f8fff7fe700 1 -- 172.21.15.158:0/1647502778 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v64) v1 ==== 161+0+0 (secure 0 0 0) 0x7f901815ebe0 con 0x7f901815db90 2023-10-16T09:44:14.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f900406af00 msgr2=0x7f900406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f900406af00 0x7f900406d3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90180cf510 tx=0x7f9008008040 comp rx=0 tx=0).stop 2023-10-16T09:44:14.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f901815db90 msgr2=0x7f901814d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f901815db90 0x7f901814d750 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f90180ceab0 tx=0x7f9010004ad0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 shutdown_connections 2023-10-16T09:44:14.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f901815db90 0x7f901814d750 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90180cddb0 0x7f90180af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f900406af00 0x7f900406d3c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 --2- 172.21.15.158:0/1647502778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f901815d7b0 0x7f90180ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 >> 172.21.15.158:0/1647502778 conn(0x7f9018071220 msgr2=0x7f901806ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:14.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 shutdown_connections 2023-10-16T09:44:14.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.507+0000 7f901d5af700 1 -- 172.21.15.158:0/1647502778 wait complete. 2023-10-16T09:44:14.522 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T09:44:14.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.647+0000 7f30c40ae700 1 Processor -- start 2023-10-16T09:44:14.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.647+0000 7f30c40ae700 1 -- start start 2023-10-16T09:44:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c40ae700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30bc072070 0x7f30bc0cef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c40ae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 0x7f30bc0cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c40ae700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30bc0ce150 con 0x7f30bc15a5b0 2023-10-16T09:44:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c40ae700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30bc0cdf70 con 0x7f30bc072070 2023-10-16T09:44:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c40ae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0ce430 con 0x7f30bc0cb900 2023-10-16T09:44:14.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c1e4a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f30bc15a5b0 0x7f30bc071960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47746/0 (socket says 172.21.15.158:47746) 2023-10-16T09:44:14.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c1e4a700 1 -- 172.21.15.158:0/3143354591 learned_addr learned my addr 172.21.15.158:0/3143354591 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:14.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942765674 0 0) 0x7f30bc0ce150 con 0x7f30bc15a5b0 2023-10-16T09:44:14.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c1649700 1 --2- 172.21.15.158:0/3143354591 >> v2:172.21.15.172:3300/0 conn(0x7f30bc072070 0x7f30bc0cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a80035b0 con 0x7f30bc15a5b0 2023-10-16T09:44:14.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c264b700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 0x7f30bc0cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236538305 0 0) 0x7f30a80035b0 con 0x7f30bc15a5b0 2023-10-16T09:44:14.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 msgr2=0x7f30bc0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:14.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 0x7f30bc0cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 >> v2:172.21.15.172:3300/0 conn(0x7f30bc072070 msgr2=0x7f30bc0cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:14.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 --2- 172.21.15.158:0/3143354591 >> v2:172.21.15.172:3300/0 conn(0x7f30bc072070 0x7f30bc0cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc0ce710 con 0x7f30bc15a5b0 2023-10-16T09:44:14.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f30b8001670 con 0x7f30bc15a5b0 2023-10-16T09:44:14.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 >> v1:172.21.15.158:6789/0 conn(0x7f30bc15a5b0 legacy=0x7f30bc071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 0x7f30a8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.648+0000 7f30c0e48700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a8004d90 0x7f30a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0cdf70 con 0x7f30a8003f80 2023-10-16T09:44:14.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0ce430 con 0x7f30a8004d90 2023-10-16T09:44:14.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a800ba10 con 0x7f30bc0cb900 2023-10-16T09:44:14.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1649700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1e4a700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 0x7f30a8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c264b700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a8004d90 0x7f30a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1649700 1 -- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 msgr2=0x7f30a8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:14.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1649700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 0x7f30a8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1649700 1 -- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a8004d90 msgr2=0x7f30a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:14.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1649700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a8004d90 0x7f30a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1649700 1 -- 172.21.15.158:0/3143354591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a80035b0 con 0x7f30a8003f80 2023-10-16T09:44:14.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c264b700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a8004d90 0x7f30a8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:14.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1e4a700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0cb900 0x7f30a8003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:14.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c1649700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30a8004850 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009b60 tx=0x7f30a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ccec0cd8084ca6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:14.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac00e660 con 0x7f30a8003f80 2023-10-16T09:44:14.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac00e7c0 con 0x7f30a8003f80 2023-10-16T09:44:14.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.649+0000 7f30c0e48700 1 -- 172.21.15.158:0/3143354591 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac00eb30 con 0x7f30a8003f80 2023-10-16T09:44:14.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.650+0000 7f30c40ae700 1 -- 172.21.15.158:0/3143354591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 msgr2=0x7f30a8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.650+0000 7f30c40ae700 1 --2- 172.21.15.158:0/3143354591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30a8004850 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009b60 tx=0x7f30a8005880 comp rx=0 tx=0).stop 2023-10-16T09:44:14.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.650+0000 7f30c264b700 1 -- 172.21.15.158:0/3143354591 reap_dead start 2023-10-16T09:44:14.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.650+0000 7f30c40ae700 1 -- 172.21.15.158:0/3143354591 shutdown_connections 2023-10-16T09:44:14.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.650+0000 7f30c40ae700 1 -- 172.21.15.158:0/3143354591 >> 172.21.15.158:0/3143354591 conn(0x7f30bc0b1e90 msgr2=0x7f30bc0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:14.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.650+0000 7f30c40ae700 1 -- 172.21.15.158:0/3143354591 shutdown_connections 2023-10-16T09:44:14.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.650+0000 7f30c40ae700 1 -- 172.21.15.158:0/3143354591 wait complete. 2023-10-16T09:44:14.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.651+0000 7f30c40ae700 1 Processor -- start 2023-10-16T09:44:14.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.651+0000 7f30c40ae700 1 -- start start 2023-10-16T09:44:14.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30bc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0b0950 0x7f30bc0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30bc0c6610 0x7f30bc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0c6d20 con 0x7f30a8003f80 2023-10-16T09:44:14.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0b0dd0 con 0x7f30bc0c6610 2023-10-16T09:44:14.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0aea40 con 0x7f30bc0b0950 2023-10-16T09:44:14.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30bc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30bc0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47582/0 (socket says 172.21.15.158:47582) 2023-10-16T09:44:14.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 -- 172.21.15.158:0/1038030467 learned_addr learned my addr 172.21.15.158:0/1038030467 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:14.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c264b700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30bc0c6610 0x7f30bc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1649700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0b0950 0x7f30bc0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 -- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0b0950 msgr2=0x7f30bc0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0b0950 0x7f30bc0c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 -- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30bc0c6610 msgr2=0x7f30bc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30bc0c6610 0x7f30bc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 -- 172.21.15.158:0/1038030467 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc0c6a90 con 0x7f30a8003f80 2023-10-16T09:44:14.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c1e4a700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30bc0b0410 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f30b800eaa0 tx=0x7f30b800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:14.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30b27fc700 1 -- 172.21.15.158:0/1038030467 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b800c700 con 0x7f30a8003f80 2023-10-16T09:44:14.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30b27fc700 1 -- 172.21.15.158:0/1038030467 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b800c860 con 0x7f30a8003f80 2023-10-16T09:44:14.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30bc0ae110 con 0x7f30a8003f80 2023-10-16T09:44:14.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30b27fc700 1 -- 172.21.15.158:0/1038030467 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b800cbf0 con 0x7f30a8003f80 2023-10-16T09:44:14.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.652+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30bc0ae6d0 con 0x7f30a8003f80 2023-10-16T09:44:14.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.653+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30bc0681b0 con 0x7f30a8003f80 2023-10-16T09:44:14.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.654+0000 7f30b27fc700 1 -- 172.21.15.158:0/1038030467 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f30b800cd50 con 0x7f30a8003f80 2023-10-16T09:44:14.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.654+0000 7f30b27fc700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a006af00 0x7f30a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:14.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.654+0000 7f30b27fc700 1 -- 172.21.15.158:0/1038030467 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f30b80898f0 con 0x7f30a8003f80 2023-10-16T09:44:14.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.654+0000 7f30c1649700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a006af00 0x7f30a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:14.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.655+0000 7f30c1649700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a006af00 0x7f30a006d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0090b0 tx=0x7f30ac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:14.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.659+0000 7f30b27fc700 1 -- 172.21.15.158:0/1038030467 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30b8054590 con 0x7f30a8003f80 2023-10-16T09:44:14.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.852+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f30bc0cc090 con 0x7f30a8003f80 2023-10-16T09:44:14.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.858+0000 7f30b27fc700 1 -- 172.21.15.158:0/1038030467 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f30b8057bb0 con 0x7f30a8003f80 2023-10-16T09:44:14.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a006af00 msgr2=0x7f30a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a006af00 0x7f30a006d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0090b0 tx=0x7f30ac008030 comp rx=0 tx=0).stop 2023-10-16T09:44:14.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 msgr2=0x7f30bc0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:14.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30bc0b0410 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f30b800eaa0 tx=0x7f30b800ee60 comp rx=0 tx=0).stop 2023-10-16T09:44:14.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 shutdown_connections 2023-10-16T09:44:14.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30bc0c6610 0x7f30bc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30a8003f80 0x7f30bc0b0410 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a006af00 0x7f30a006d3c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 --2- 172.21.15.158:0/1038030467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30bc0b0950 0x7f30bc0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:14.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 >> 172.21.15.158:0/1038030467 conn(0x7f30bc0b1e90 msgr2=0x7f30bc06d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:14.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 shutdown_connections 2023-10-16T09:44:14.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:14.860+0000 7f30c40ae700 1 -- 172.21.15.158:0/1038030467 wait complete. 2023-10-16T09:44:14.874 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T09:44:14.874 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:44:14.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:44:14.964 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T09:44:14.965 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:14.965 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:14.983 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:14.983 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:15.040 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:15.041 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:15.097 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:15.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.256+0000 7fc8179fb700 1 Processor -- start 2023-10-16T09:44:15.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.256+0000 7fc8179fb700 1 -- start start 2023-10-16T09:44:15.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc8179fb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc81006ab40 0x7fc81006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc8179fb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc81006e5d0 0x7fc810071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc8179fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006af20 0x7fc8100722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc8179fb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc810071e50 con 0x7fc81006ab40 2023-10-16T09:44:15.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc8179fb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc810161270 con 0x7fc81006e5d0 2023-10-16T09:44:15.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc8179fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc810161510 con 0x7fc81006af20 2023-10-16T09:44:15.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc815797700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc81006ab40 0x7fc81006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc815797700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc81006ab40 0x7fc81006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47590/0 (socket says 172.21.15.158:47590) 2023-10-16T09:44:15.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.257+0000 7fc815797700 1 -- 172.21.15.158:0/3455866879 learned_addr learned my addr 172.21.15.158:0/3455866879 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:15.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc814f96700 1 --2- 172.21.15.158:0/3455866879 >> v2:172.21.15.172:3300/0 conn(0x7fc81006e5d0 0x7fc810071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc815f98700 1 --2- 172.21.15.158:0/3455866879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006af20 0x7fc8100722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc815797700 1 -- 172.21.15.158:0/3455866879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006af20 msgr2=0x7fc8100722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:15.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc815797700 1 --2- 172.21.15.158:0/3455866879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006af20 0x7fc8100722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc815797700 1 -- 172.21.15.158:0/3455866879 >> v2:172.21.15.172:3300/0 conn(0x7fc81006e5d0 msgr2=0x7fc810071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:15.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc815797700 1 --2- 172.21.15.158:0/3455866879 >> v2:172.21.15.172:3300/0 conn(0x7fc81006e5d0 0x7fc810071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc815797700 1 -- 172.21.15.158:0/3455866879 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8101617f0 con 0x7fc81006ab40 2023-10-16T09:44:15.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc814f96700 1 --2- 172.21.15.158:0/3455866879 >> v2:172.21.15.172:3300/0 conn(0x7fc81006e5d0 0x7fc810071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:15.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc815797700 1 --2- 172.21.15.158:0/3455866879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc81006d820 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc8100c3bf0 tx=0x7fc800009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fb76a6e66b3055 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc807fff700 1 -- 172.21.15.158:0/3455866879 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80000b870 con 0x7fc81006ab40 2023-10-16T09:44:15.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc807fff700 1 -- 172.21.15.158:0/3455866879 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80000b9d0 con 0x7fc81006ab40 2023-10-16T09:44:15.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc807fff700 1 -- 172.21.15.158:0/3455866879 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80000bd60 con 0x7fc81006ab40 2023-10-16T09:44:15.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc8179fb700 1 -- 172.21.15.158:0/3455866879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 msgr2=0x7fc81006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc8179fb700 1 --2- 172.21.15.158:0/3455866879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc81006d820 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc8100c3bf0 tx=0x7fc800009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc8179fb700 1 -- 172.21.15.158:0/3455866879 shutdown_connections 2023-10-16T09:44:15.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc8179fb700 1 --2- 172.21.15.158:0/3455866879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006af20 0x7fc8100722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc8179fb700 1 --2- 172.21.15.158:0/3455866879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc81006d820 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc8179fb700 1 --2- 172.21.15.158:0/3455866879 >> v2:172.21.15.172:3300/0 conn(0x7fc81006e5d0 0x7fc810071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.258+0000 7fc8179fb700 1 -- 172.21.15.158:0/3455866879 >> 172.21.15.158:0/3455866879 conn(0x7fc8100d02b0 msgr2=0x7fc8100d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:15.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.259+0000 7fc8179fb700 1 -- 172.21.15.158:0/3455866879 shutdown_connections 2023-10-16T09:44:15.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.259+0000 7fc8179fb700 1 -- 172.21.15.158:0/3455866879 wait complete. 2023-10-16T09:44:15.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 Processor -- start 2023-10-16T09:44:15.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 -- start start 2023-10-16T09:44:15.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc8100b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc81006af20 0x7fc8100c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006e5d0 0x7fc810155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100c4520 con 0x7fc81006ab40 2023-10-16T09:44:15.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100c7870 con 0x7fc81006af20 2023-10-16T09:44:15.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.260+0000 7fc8179fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100b2560 con 0x7fc81006e5d0 2023-10-16T09:44:15.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc8100b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc8100b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47596/0 (socket says 172.21.15.158:47596) 2023-10-16T09:44:15.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 -- 172.21.15.158:0/506503521 learned_addr learned my addr 172.21.15.158:0/506503521 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:15.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc814f96700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc81006af20 0x7fc8100c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815f98700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006e5d0 0x7fc810155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 -- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006e5d0 msgr2=0x7fc810155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006e5d0 0x7fc810155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 -- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc81006af20 msgr2=0x7fc8100c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc81006af20 0x7fc8100c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 -- 172.21.15.158:0/506503521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc800009550 con 0x7fc81006ab40 2023-10-16T09:44:15.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc814f96700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc81006af20 0x7fc8100c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:15.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815f98700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006e5d0 0x7fc810155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:15.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc815797700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc8100b2020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc810074ec0 tx=0x7fc800004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc805ffb700 1 -- 172.21.15.158:0/506503521 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800004310 con 0x7fc81006ab40 2023-10-16T09:44:15.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc805ffb700 1 -- 172.21.15.158:0/506503521 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80001f070 con 0x7fc81006ab40 2023-10-16T09:44:15.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc805ffb700 1 -- 172.21.15.158:0/506503521 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000036a0 con 0x7fc81006ab40 2023-10-16T09:44:15.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc810150d20 con 0x7fc81006ab40 2023-10-16T09:44:15.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.261+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8100c3f60 con 0x7fc81006ab40 2023-10-16T09:44:15.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.262+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8100681b0 con 0x7fc81006ab40 2023-10-16T09:44:15.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.263+0000 7fc805ffb700 1 -- 172.21.15.158:0/506503521 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc800021030 con 0x7fc81006ab40 2023-10-16T09:44:15.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.263+0000 7fc805ffb700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc7f406af00 0x7fc7f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.263+0000 7fc805ffb700 1 -- 172.21.15.158:0/506503521 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fc80008ae40 con 0x7fc81006ab40 2023-10-16T09:44:15.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.263+0000 7fc814f96700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc7f406af00 0x7fc7f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.264+0000 7fc814f96700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc7f406af00 0x7fc7f406d3c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc80c006fd0 tx=0x7fc80c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.268+0000 7fc805ffb700 1 -- 172.21.15.158:0/506503521 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc800055c90 con 0x7fc81006ab40 2023-10-16T09:44:15.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.459+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc810161aa0 con 0x7fc81006ab40 2023-10-16T09:44:15.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.460+0000 7fc805ffb700 1 -- 172.21.15.158:0/506503521 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7fc800015090 con 0x7fc81006ab40 2023-10-16T09:44:15.462 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:15.462 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:15.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc7f406af00 msgr2=0x7fc7f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc7f406af00 0x7fc7f406d3c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc80c006fd0 tx=0x7fc80c008040 comp rx=0 tx=0).stop 2023-10-16T09:44:15.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 msgr2=0x7fc8100b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc8100b2020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc810074ec0 tx=0x7fc800004750 comp rx=0 tx=0).stop 2023-10-16T09:44:15.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 shutdown_connections 2023-10-16T09:44:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc7f406af00 0x7fc7f406d3c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc81006e5d0 0x7fc810155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc81006ab40 0x7fc8100b2020 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 --2- 172.21.15.158:0/506503521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc81006af20 0x7fc8100c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 >> 172.21.15.158:0/506503521 conn(0x7fc8100d02b0 msgr2=0x7fc81006d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:15.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 shutdown_connections 2023-10-16T09:44:15.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.463+0000 7fc8179fb700 1 -- 172.21.15.158:0/506503521 wait complete. 2023-10-16T09:44:15.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:15.477 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:15.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.602+0000 7fe844558700 1 Processor -- start 2023-10-16T09:44:15.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.602+0000 7fe844558700 1 -- start start 2023-10-16T09:44:15.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe844558700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe844558700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe83c0cc9e0 0x7fe83c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe844558700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83c06a8d0 con 0x7fe83c0700a0 2023-10-16T09:44:15.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe844558700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe83c161e50 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe844558700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c072970 con 0x7fe83c15d910 2023-10-16T09:44:15.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe841af3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe83c0cc9e0 0x7fe83c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe842af5700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe83c0700a0 0x7fe83c06ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47784/0 (socket says 172.21.15.158:47784) 2023-10-16T09:44:15.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe842af5700 1 -- 172.21.15.158:0/3486532583 learned_addr learned my addr 172.21.15.158:0/3486532583 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:15.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe8422f4700 1 --2- 172.21.15.158:0/3486532583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe841af3700 1 -- 172.21.15.158:0/3486532583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 msgr2=0x7fe83c0cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:15.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe841af3700 1 --2- 172.21.15.158:0/3486532583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c0cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe841af3700 1 -- 172.21.15.158:0/3486532583 >> v1:172.21.15.158:6789/0 conn(0x7fe83c0700a0 legacy=0x7fe83c06ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe841af3700 1 -- 172.21.15.158:0/3486532583 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c15dcf0 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe8412f2700 1 -- 172.21.15.158:0/3486532583 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051240201 0 0) 0x7fe83c06a8d0 con 0x7fe83c0700a0 2023-10-16T09:44:15.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe841af3700 1 --2- 172.21.15.158:0/3486532583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c0ce4b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009840 tx=0x7fe82c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970e4b997ad3cd5c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe8412f2700 1 -- 172.21.15.158:0/3486532583 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c011830 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe8412f2700 1 -- 172.21.15.158:0/3486532583 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c011990 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.603+0000 7fe8412f2700 1 -- 172.21.15.158:0/3486532583 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c011d20 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 -- 172.21.15.158:0/3486532583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 msgr2=0x7fe83c0ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 --2- 172.21.15.158:0/3486532583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c0ce4b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009840 tx=0x7fe82c009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:15.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 -- 172.21.15.158:0/3486532583 shutdown_connections 2023-10-16T09:44:15.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 --2- 172.21.15.158:0/3486532583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 --2- 172.21.15.158:0/3486532583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c0ce4b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 -- 172.21.15.158:0/3486532583 >> 172.21.15.158:0/3486532583 conn(0x7fe83c0b1e90 msgr2=0x7fe83c0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:15.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 -- 172.21.15.158:0/3486532583 shutdown_connections 2023-10-16T09:44:15.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.604+0000 7fe844558700 1 -- 172.21.15.158:0/3486532583 wait complete. 2023-10-16T09:44:15.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.605+0000 7fe844558700 1 Processor -- start 2023-10-16T09:44:15.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.605+0000 7fe844558700 1 -- start start 2023-10-16T09:44:15.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe844558700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe83c0700a0 0x7fe83c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe844558700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c1f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe844558700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c15fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe844558700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c0ca310 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe844558700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c1f7260 con 0x7fe83c0700a0 2023-10-16T09:44:15.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe844558700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c1f7540 con 0x7fe83c15d910 2023-10-16T09:44:15.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c1f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c1f6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47612/0 (socket says 172.21.15.158:47612) 2023-10-16T09:44:15.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 -- 172.21.15.158:0/354642346 learned_addr learned my addr 172.21.15.158:0/354642346 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:15.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe8422f4700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe83c0700a0 0x7fe83c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe842af5700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c15fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 -- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 msgr2=0x7fe83c15fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c15fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 -- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe83c0700a0 msgr2=0x7fe83c0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe83c0700a0 0x7fe83c0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe841af3700 1 -- 172.21.15.158:0/354642346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c0094a0 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe842af5700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c15fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:15.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.606+0000 7fe8422f4700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe83c0700a0 0x7fe83c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:15.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.607+0000 7fe841af3700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c1f6920 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe82c000c00 tx=0x7fe82c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.607+0000 7fe832ffd700 1 -- 172.21.15.158:0/354642346 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c0036a0 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.607+0000 7fe832ffd700 1 -- 172.21.15.158:0/354642346 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c01f070 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.607+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83c1f7820 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.607+0000 7fe832ffd700 1 -- 172.21.15.158:0/354642346 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c003a20 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.607+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83c0c9f50 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.608+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83c0ca0e0 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.608+0000 7fe832ffd700 1 -- 172.21.15.158:0/354642346 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe82c003b80 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.609+0000 7fe832ffd700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe82006af00 0x7fe82006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.609+0000 7fe8422f4700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe82006af00 0x7fe82006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.609+0000 7fe832ffd700 1 -- 172.21.15.158:0/354642346 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fe82c021070 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.609+0000 7fe8422f4700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe82006af00 0x7fe82006d3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe8380099f0 tx=0x7fe838008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.613+0000 7fe832ffd700 1 -- 172.21.15.158:0/354642346 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe82c056170 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.804+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe83c1f7d40 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.804+0000 7fe832ffd700 1 -- 172.21.15.158:0/354642346 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe83c1f7d40 con 0x7fe83c0cc9e0 2023-10-16T09:44:15.807 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:15.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe82006af00 msgr2=0x7fe82006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe82006af00 0x7fe82006d3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe8380099f0 tx=0x7fe838008040 comp rx=0 tx=0).stop 2023-10-16T09:44:15.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 msgr2=0x7fe83c1f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c1f6920 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe82c000c00 tx=0x7fe82c004750 comp rx=0 tx=0).stop 2023-10-16T09:44:15.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 shutdown_connections 2023-10-16T09:44:15.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe82006af00 0x7fe82006d3c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83c15d910 0x7fe83c15fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe83c0700a0 0x7fe83c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 --2- 172.21.15.158:0/354642346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83c0cc9e0 0x7fe83c1f6920 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 >> 172.21.15.158:0/354642346 conn(0x7fe83c0b1e90 msgr2=0x7fe83c0cb030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:15.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 shutdown_connections 2023-10-16T09:44:15.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.807+0000 7fe844558700 1 -- 172.21.15.158:0/354642346 wait complete. 2023-10-16T09:44:15.820 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:15.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.943+0000 7fa80af90700 1 Processor -- start 2023-10-16T09:44:15.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.943+0000 7fa80af90700 1 -- start start 2023-10-16T09:44:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa80af90700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa80415b590 0x7fa8040ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa80af90700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa80415f1d0 0x7fa8040b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa80af90700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8040cf4c0 con 0x7fa8040cbaf0 2023-10-16T09:44:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa80af90700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80415f970 con 0x7fa80415b590 2023-10-16T09:44:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa80af90700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80415fc50 con 0x7fa80415f1d0 2023-10-16T09:44:15.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa808d2c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa80415b590 0x7fa8040ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa80952d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa8040cbaf0 0x7fa8040cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47798/0 (socket says 172.21.15.158:47798) 2023-10-16T09:44:15.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa808d2c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa80415b590 0x7fa8040ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47618/0 (socket says 172.21.15.158:47618) 2023-10-16T09:44:15.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa80952d700 1 -- 172.21.15.158:0/3426550769 learned_addr learned my addr 172.21.15.158:0/3426550769 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:15.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa803fff700 1 --2- 172.21.15.158:0/3426550769 >> v2:172.21.15.172:3300/0 conn(0x7fa80415f1d0 0x7fa8040b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa808d2c700 1 -- 172.21.15.158:0/3426550769 >> v2:172.21.15.172:3300/0 conn(0x7fa80415f1d0 msgr2=0x7fa8040b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:15.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa808d2c700 1 --2- 172.21.15.158:0/3426550769 >> v2:172.21.15.172:3300/0 conn(0x7fa80415f1d0 0x7fa8040b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa808d2c700 1 -- 172.21.15.158:0/3426550769 >> v1:172.21.15.158:6789/0 conn(0x7fa8040cbaf0 legacy=0x7fa8040cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa8037fe700 1 -- 172.21.15.158:0/3426550769 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337773566 0 0) 0x7fa8040cf4c0 con 0x7fa8040cbaf0 2023-10-16T09:44:15.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa808d2c700 1 -- 172.21.15.158:0/3426550769 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80415ff30 con 0x7fa80415b590 2023-10-16T09:44:15.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa808d2c700 1 --2- 172.21.15.158:0/3426550769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415b590 0x7fa8040ce510 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa8040d1360 tx=0x7fa7f8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefa7fcd63d2f195 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.944+0000 7fa8037fe700 1 -- 172.21.15.158:0/3426550769 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8011830 con 0x7fa80415b590 2023-10-16T09:44:15.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa8037fe700 1 -- 172.21.15.158:0/3426550769 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f8011990 con 0x7fa80415b590 2023-10-16T09:44:15.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa8037fe700 1 -- 172.21.15.158:0/3426550769 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8011d20 con 0x7fa80415b590 2023-10-16T09:44:15.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 -- 172.21.15.158:0/3426550769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415b590 msgr2=0x7fa8040ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 --2- 172.21.15.158:0/3426550769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415b590 0x7fa8040ce510 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa8040d1360 tx=0x7fa7f8009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 -- 172.21.15.158:0/3426550769 shutdown_connections 2023-10-16T09:44:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 --2- 172.21.15.158:0/3426550769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415b590 0x7fa8040ce510 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 --2- 172.21.15.158:0/3426550769 >> v2:172.21.15.172:3300/0 conn(0x7fa80415f1d0 0x7fa8040b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 -- 172.21.15.158:0/3426550769 >> 172.21.15.158:0/3426550769 conn(0x7fa80406a8d0 msgr2=0x7fa80406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 -- 172.21.15.158:0/3426550769 shutdown_connections 2023-10-16T09:44:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.945+0000 7fa80af90700 1 -- 172.21.15.158:0/3426550769 wait complete. 2023-10-16T09:44:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.946+0000 7fa80af90700 1 Processor -- start 2023-10-16T09:44:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.946+0000 7fa80af90700 1 -- start start 2023-10-16T09:44:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80af90700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8040cbaf0 0x7fa804153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80af90700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa80415b590 0x7fa80406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80af90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415f1d0 0x7fa80415e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80af90700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80415cf50 con 0x7fa80415f1d0 2023-10-16T09:44:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80af90700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80415d1f0 con 0x7fa8040cbaf0 2023-10-16T09:44:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80af90700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80415d4d0 con 0x7fa80415b590 2023-10-16T09:44:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80952d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415f1d0 0x7fa80415e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80952d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415f1d0 0x7fa80415e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47628/0 (socket says 172.21.15.158:47628) 2023-10-16T09:44:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.947+0000 7fa80952d700 1 -- 172.21.15.158:0/2779384878 learned_addr learned my addr 172.21.15.158:0/2779384878 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa808d2c700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8040cbaf0 0x7fa804153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa803fff700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa80415b590 0x7fa80406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80952d700 1 -- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa80415b590 msgr2=0x7fa80406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80952d700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa80415b590 0x7fa80406c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80952d700 1 -- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8040cbaf0 msgr2=0x7fa804153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:15.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80952d700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8040cbaf0 0x7fa804153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:15.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80952d700 1 -- 172.21.15.158:0/2779384878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f80094a0 con 0x7fa80415f1d0 2023-10-16T09:44:15.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa808d2c700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8040cbaf0 0x7fa804153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:15.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80952d700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415f1d0 0x7fa80415e290 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc002230 tx=0x7fa7fc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa8017fa700 1 -- 172.21.15.158:0/2779384878 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc00ac20 con 0x7fa80415f1d0 2023-10-16T09:44:15.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa8017fa700 1 -- 172.21.15.158:0/2779384878 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7fc00ada0 con 0x7fa80415f1d0 2023-10-16T09:44:15.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80415d7b0 con 0x7fa80415f1d0 2023-10-16T09:44:15.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa8017fa700 1 -- 172.21.15.158:0/2779384878 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc007310 con 0x7fa80415f1d0 2023-10-16T09:44:15.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.948+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80415dd70 con 0x7fa80415f1d0 2023-10-16T09:44:15.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.949+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8041541c0 con 0x7fa80415f1d0 2023-10-16T09:44:15.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.950+0000 7fa8017fa700 1 -- 172.21.15.158:0/2779384878 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa7fc007470 con 0x7fa80415f1d0 2023-10-16T09:44:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.950+0000 7fa8017fa700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa7e806af00 0x7fa7e806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.950+0000 7fa8017fa700 1 -- 172.21.15.158:0/2779384878 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fa7fc089850 con 0x7fa80415f1d0 2023-10-16T09:44:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.950+0000 7fa808d2c700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa7e806af00 0x7fa7e806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.951+0000 7fa808d2c700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa7e806af00 0x7fa7e806d3c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009b60 tx=0x7fa7f8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:15.954+0000 7fa8017fa700 1 -- 172.21.15.158:0/2779384878 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa7fc0545d0 con 0x7fa80415f1d0 2023-10-16T09:44:16.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.146+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa80415df60 con 0x7fa80415f1d0 2023-10-16T09:44:16.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.146+0000 7fa8017fa700 1 -- 172.21.15.158:0/2779384878 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7fc018020 con 0x7fa80415f1d0 2023-10-16T09:44:16.148 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa7e806af00 msgr2=0x7fa7e806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa7e806af00 0x7fa7e806d3c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009b60 tx=0x7fa7f8008030 comp rx=0 tx=0).stop 2023-10-16T09:44:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415f1d0 msgr2=0x7fa80415e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415f1d0 0x7fa80415e290 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc002230 tx=0x7fa7fc00a060 comp rx=0 tx=0).stop 2023-10-16T09:44:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 shutdown_connections 2023-10-16T09:44:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa80415f1d0 0x7fa80415e290 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8040cbaf0 0x7fa804153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa7e806af00 0x7fa7e806d3c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 --2- 172.21.15.158:0/2779384878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa80415b590 0x7fa80406c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 >> 172.21.15.158:0/2779384878 conn(0x7fa80406a8d0 msgr2=0x7fa8040cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 shutdown_connections 2023-10-16T09:44:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.148+0000 7fa80af90700 1 -- 172.21.15.158:0/2779384878 wait complete. 2023-10-16T09:44:16.162 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.284+0000 7f1ff7833700 1 Processor -- start 2023-10-16T09:44:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.284+0000 7f1ff7833700 1 -- start start 2023-10-16T09:44:16.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff7833700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ff015b590 0x7f1ff00ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff7833700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1ff00cbaf0 0x7f1ff00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff7833700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff00cf4c0 con 0x7f1ff015f1d0 2023-10-16T09:44:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff7833700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00cc4c0 con 0x7f1ff015b590 2023-10-16T09:44:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff7833700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00b49e0 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ff015b590 0x7f1ff00ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ff015b590 0x7f1ff00ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47632/0 (socket says 172.21.15.158:47632) 2023-10-16T09:44:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 -- 172.21.15.158:0/486091188 learned_addr learned my addr 172.21.15.158:0/486091188 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff5dd0700 1 --2- 172.21.15.158:0/486091188 >> v2:172.21.15.172:3300/0 conn(0x7f1ff00cbaf0 0x7f1ff00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 -- 172.21.15.158:0/486091188 >> v2:172.21.15.172:3300/0 conn(0x7f1ff00cbaf0 msgr2=0x7f1ff00cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 --2- 172.21.15.158:0/486091188 >> v2:172.21.15.172:3300/0 conn(0x7f1ff00cbaf0 0x7f1ff00cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 -- 172.21.15.158:0/486091188 >> v1:172.21.15.172:6789/0 conn(0x7f1ff015f1d0 legacy=0x7f1ff00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 -- 172.21.15.158:0/486091188 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff015f970 con 0x7f1ff015b590 2023-10-16T09:44:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1ff55cf700 1 --2- 172.21.15.158:0/486091188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff015b590 0x7f1ff00ce510 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00cea50 tx=0x7f1fec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c05ea4da0d46da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1fe7fff700 1 -- 172.21.15.158:0/486091188 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00b860 con 0x7f1ff015b590 2023-10-16T09:44:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1fe7fff700 1 -- 172.21.15.158:0/486091188 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec00b9c0 con 0x7f1ff015b590 2023-10-16T09:44:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.285+0000 7f1fe7fff700 1 -- 172.21.15.158:0/486091188 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00bd50 con 0x7f1ff015b590 2023-10-16T09:44:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 -- 172.21.15.158:0/486091188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff015b590 msgr2=0x7f1ff00ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 --2- 172.21.15.158:0/486091188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff015b590 0x7f1ff00ce510 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00cea50 tx=0x7f1fec009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:16.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 -- 172.21.15.158:0/486091188 shutdown_connections 2023-10-16T09:44:16.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 --2- 172.21.15.158:0/486091188 >> v2:172.21.15.172:3300/0 conn(0x7f1ff00cbaf0 0x7f1ff00cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 --2- 172.21.15.158:0/486091188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff015b590 0x7f1ff00ce510 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 -- 172.21.15.158:0/486091188 >> 172.21.15.158:0/486091188 conn(0x7f1ff006a8d0 msgr2=0x7f1ff006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:16.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 -- 172.21.15.158:0/486091188 shutdown_connections 2023-10-16T09:44:16.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.286+0000 7f1ff7833700 1 -- 172.21.15.158:0/486091188 wait complete. 2023-10-16T09:44:16.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.287+0000 7f1ff7833700 1 Processor -- start 2023-10-16T09:44:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.287+0000 7f1ff7833700 1 -- start start 2023-10-16T09:44:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff7833700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff00cbaf0 0x7f1ff00af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff7833700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff015b590 0x7f1ff00c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff7833700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff015f1d0 0x7f1ff00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff7833700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00ac890 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff7833700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00ad410 con 0x7f1ff015b590 2023-10-16T09:44:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff7833700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00af710 con 0x7f1ff015f1d0 2023-10-16T09:44:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff00cbaf0 0x7f1ff00af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff00cbaf0 0x7f1ff00af1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47638/0 (socket says 172.21.15.158:47638) 2023-10-16T09:44:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 -- 172.21.15.158:0/2461943898 learned_addr learned my addr 172.21.15.158:0/2461943898 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff5dd0700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff015f1d0 0x7f1ff00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff4dce700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff015b590 0x7f1ff00c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 -- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff015f1d0 msgr2=0x7f1ff00b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff015f1d0 0x7f1ff00b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 -- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff015b590 msgr2=0x7f1ff00c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff015b590 0x7f1ff00c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff55cf700 1 -- 172.21.15.158:0/2461943898 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec009550 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.288+0000 7f1ff4dce700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff015b590 0x7f1ff00c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.289+0000 7f1ff55cf700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff00cbaf0 0x7f1ff00af1d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00d25a0 tx=0x7f1fec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.289+0000 7f1fe5ffb700 1 -- 172.21.15.158:0/2461943898 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec004400 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.289+0000 7f1fe5ffb700 1 -- 172.21.15.158:0/2461943898 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec01f070 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.289+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff00aeb50 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.289+0000 7f1fe5ffb700 1 -- 172.21.15.158:0/2461943898 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0038e0 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.289+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff00acdc0 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.290+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff00681b0 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.290+0000 7f1fe5ffb700 1 -- 172.21.15.158:0/2461943898 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1fec021030 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.291+0000 7f1fe5ffb700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fd406af00 0x7f1fd406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.291+0000 7f1fe5ffb700 1 -- 172.21.15.158:0/2461943898 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f1fec08ae00 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.291+0000 7f1ff4dce700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fd406af00 0x7f1fd406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.291+0000 7f1ff4dce700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fd406af00 0x7f1fd406d3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ff006cd90 tx=0x7f1fe0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.295+0000 7f1fe5ffb700 1 -- 172.21.15.158:0/2461943898 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1fec055ab0 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.486+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ff015fc20 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.487+0000 7f1fe5ffb700 1 -- 172.21.15.158:0/2461943898 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1fec0590d0 con 0x7f1ff00cbaf0 2023-10-16T09:44:16.489 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:16.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fd406af00 msgr2=0x7f1fd406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fd406af00 0x7f1fd406d3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ff006cd90 tx=0x7f1fe0009040 comp rx=0 tx=0).stop 2023-10-16T09:44:16.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff00cbaf0 msgr2=0x7f1ff00af1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff00cbaf0 0x7f1ff00af1d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00d25a0 tx=0x7f1fec004750 comp rx=0 tx=0).stop 2023-10-16T09:44:16.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 shutdown_connections 2023-10-16T09:44:16.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1fd406af00 0x7f1fd406d3c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff015f1d0 0x7f1ff00b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff00cbaf0 0x7f1ff00af1d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 --2- 172.21.15.158:0/2461943898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff015b590 0x7f1ff00c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 >> 172.21.15.158:0/2461943898 conn(0x7f1ff006a8d0 msgr2=0x7f1ff00cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:16.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 shutdown_connections 2023-10-16T09:44:16.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.489+0000 7f1ff7833700 1 -- 172.21.15.158:0/2461943898 wait complete. 2023-10-16T09:44:16.503 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:16.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.625+0000 7f9580e6e700 1 Processor -- start 2023-10-16T09:44:16.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.625+0000 7f9580e6e700 1 -- start start 2023-10-16T09:44:16.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f9580e6e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f957c15da40 0x7f957c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f9580e6e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c06afe0 0x7f957c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f9580e6e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c0cbd30 con 0x7f957c0cc590 2023-10-16T09:44:16.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f9580e6e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f957c0cc1c0 con 0x7f957c15da40 2023-10-16T09:44:16.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f9580e6e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957c15de20 con 0x7f957c06afe0 2023-10-16T09:44:16.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f957c15da40 0x7f957c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f957c15da40 0x7f957c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47646/0 (socket says 172.21.15.158:47646) 2023-10-16T09:44:16.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 -- 172.21.15.158:0/422306303 learned_addr learned my addr 172.21.15.158:0/422306303 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:16.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 -- 172.21.15.158:0/422306303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c06afe0 msgr2=0x7f957c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:16.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 --2- 172.21.15.158:0/422306303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c06afe0 0x7f957c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 -- 172.21.15.158:0/422306303 >> v1:172.21.15.172:6789/0 conn(0x7f957c0cc590 legacy=0x7f957c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:16.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 -- 172.21.15.158:0/422306303 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c0ce260 con 0x7f957c15da40 2023-10-16T09:44:16.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.626+0000 7f957a59c700 1 --2- 172.21.15.158:0/422306303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c15da40 0x7f957c0caa20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f956c0098f0 tx=0x7f956c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ebe56f6a9d9b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f957959a700 1 -- 172.21.15.158:0/422306303 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c004230 con 0x7f957c15da40 2023-10-16T09:44:16.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f957959a700 1 -- 172.21.15.158:0/422306303 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c004390 con 0x7f957c15da40 2023-10-16T09:44:16.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f957959a700 1 -- 172.21.15.158:0/422306303 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c0036a0 con 0x7f957c15da40 2023-10-16T09:44:16.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 -- 172.21.15.158:0/422306303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c15da40 msgr2=0x7f957c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 --2- 172.21.15.158:0/422306303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c15da40 0x7f957c0caa20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f956c0098f0 tx=0x7f956c009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:16.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 -- 172.21.15.158:0/422306303 shutdown_connections 2023-10-16T09:44:16.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 --2- 172.21.15.158:0/422306303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c06afe0 0x7f957c0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 --2- 172.21.15.158:0/422306303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c15da40 0x7f957c0caa20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 -- 172.21.15.158:0/422306303 >> 172.21.15.158:0/422306303 conn(0x7f957c099a10 msgr2=0x7f957c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:16.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 -- 172.21.15.158:0/422306303 shutdown_connections 2023-10-16T09:44:16.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.627+0000 7f9580e6e700 1 -- 172.21.15.158:0/422306303 wait complete. 2023-10-16T09:44:16.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.628+0000 7f9580e6e700 1 Processor -- start 2023-10-16T09:44:16.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f9580e6e700 1 -- start start 2023-10-16T09:44:16.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f9580e6e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c06afe0 0x7f957c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f9580e6e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c0cc590 0x7f957c0c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f9580e6e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f957c15da40 0x7f957c1e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f9580e6e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c0c8890 con 0x7f957c06afe0 2023-10-16T09:44:16.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f9580e6e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c0c79e0 con 0x7f957c15da40 2023-10-16T09:44:16.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f9580e6e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957c0c7c80 con 0x7f957c0cc590 2023-10-16T09:44:16.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f957a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c06afe0 0x7f957c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f957a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c06afe0 0x7f957c0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47652/0 (socket says 172.21.15.158:47652) 2023-10-16T09:44:16.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.629+0000 7f957a59c700 1 -- 172.21.15.158:0/2283599968 learned_addr learned my addr 172.21.15.158:0/2283599968 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:16.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f9579d9b700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c0cc590 0x7f957c0c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f957ad9d700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f957c15da40 0x7f957c1e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f957a59c700 1 -- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c0cc590 msgr2=0x7f957c0c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f957a59c700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c0cc590 0x7f957c0c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f957a59c700 1 -- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f957c15da40 msgr2=0x7f957c1e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f957a59c700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f957c15da40 0x7f957c1e6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f957a59c700 1 -- 172.21.15.158:0/2283599968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c009550 con 0x7f957c06afe0 2023-10-16T09:44:16.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f957a59c700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c06afe0 0x7f957c0c74a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f956c000c00 tx=0x7f956c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f956affd700 1 -- 172.21.15.158:0/2283599968 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c003db0 con 0x7f957c06afe0 2023-10-16T09:44:16.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f956affd700 1 -- 172.21.15.158:0/2283599968 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c01f070 con 0x7f957c06afe0 2023-10-16T09:44:16.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957c0c7f60 con 0x7f957c06afe0 2023-10-16T09:44:16.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f956affd700 1 -- 172.21.15.158:0/2283599968 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c00b8a0 con 0x7f957c06afe0 2023-10-16T09:44:16.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.630+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c0c8520 con 0x7f957c06afe0 2023-10-16T09:44:16.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.631+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c0c86b0 con 0x7f957c06afe0 2023-10-16T09:44:16.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.632+0000 7f956affd700 1 -- 172.21.15.158:0/2283599968 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f956c021070 con 0x7f957c06afe0 2023-10-16T09:44:16.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.632+0000 7f956affd700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f955c06af00 0x7f955c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.633+0000 7f956affd700 1 -- 172.21.15.158:0/2283599968 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f956c08b5b0 con 0x7f957c06afe0 2023-10-16T09:44:16.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.633+0000 7f9579d9b700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f955c06af00 0x7f955c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.633+0000 7f9579d9b700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f955c06af00 0x7f955c06d3c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f957c0c42f0 tx=0x7f9570009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.636+0000 7f956affd700 1 -- 172.21.15.158:0/2283599968 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f956c0562e0 con 0x7f957c06afe0 2023-10-16T09:44:16.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.828+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f957c0c86b0 con 0x7f957c06afe0 2023-10-16T09:44:16.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.829+0000 7f956affd700 1 -- 172.21.15.158:0/2283599968 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f956c059900 con 0x7f957c06afe0 2023-10-16T09:44:16.831 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:16.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f955c06af00 msgr2=0x7f955c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f955c06af00 0x7f955c06d3c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f957c0c42f0 tx=0x7f9570009040 comp rx=0 tx=0).stop 2023-10-16T09:44:16.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c06afe0 msgr2=0x7f957c0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c06afe0 0x7f957c0c74a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f956c000c00 tx=0x7f956c004750 comp rx=0 tx=0).stop 2023-10-16T09:44:16.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 shutdown_connections 2023-10-16T09:44:16.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f957c15da40 0x7f957c1e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f957c06afe0 0x7f957c0c74a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f955c06af00 0x7f955c06d3c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 --2- 172.21.15.158:0/2283599968 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f957c0cc590 0x7f957c0c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 >> 172.21.15.158:0/2283599968 conn(0x7f957c099a10 msgr2=0x7f957c0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:16.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 shutdown_connections 2023-10-16T09:44:16.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.831+0000 7f9580e6e700 1 -- 172.21.15.158:0/2283599968 wait complete. 2023-10-16T09:44:16.845 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:44:16.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.966+0000 7f43f219b700 1 Processor -- start 2023-10-16T09:44:16.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.967+0000 7f43f219b700 1 -- start start 2023-10-16T09:44:16.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.967+0000 7f43f219b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f43ec15a6d0 0x7f43ec0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.967+0000 7f43f219b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cd360 0x7f43ec06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.967+0000 7f43f219b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec15ec70 con 0x7f43ec0cfe30 2023-10-16T09:44:16.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.967+0000 7f43f219b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec0afcd0 con 0x7f43ec15a6d0 2023-10-16T09:44:16.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.967+0000 7f43f219b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec15f1b0 con 0x7f43ec0cd360 2023-10-16T09:44:16.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f43ec15a6d0 0x7f43ec0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43ebfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cd360 0x7f43ec06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f43ec15a6d0 0x7f43ec0ce2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:48840/0 (socket says 172.21.15.158:48840) 2023-10-16T09:44:16.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43ebfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cd360 0x7f43ec06ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:58496/0 (socket says 172.21.15.158:58496) 2023-10-16T09:44:16.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 -- 172.21.15.158:0/3634237847 learned_addr learned my addr 172.21.15.158:0/3634237847 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:16.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 -- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cd360 msgr2=0x7f43ec06ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:16.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 --2- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cd360 0x7f43ec06ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 -- 172.21.15.158:0/3634237847 >> v1:172.21.15.172:6789/0 conn(0x7f43ec0cfe30 legacy=0x7f43ec0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 -- 172.21.15.158:0/3634237847 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec0cab50 con 0x7f43ec15a6d0 2023-10-16T09:44:16.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43ebfff700 1 --2- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cd360 0x7f43ec06ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:16.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43eb7fe700 1 --2- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec15a6d0 0x7f43ec0ce2d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0098f0 tx=0x7f43dc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3024629908f97fe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.968+0000 7f43ea7fc700 1 -- 172.21.15.158:0/3634237847 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc011830 con 0x7f43ec15a6d0 2023-10-16T09:44:16.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43ea7fc700 1 -- 172.21.15.158:0/3634237847 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc011990 con 0x7f43ec15a6d0 2023-10-16T09:44:16.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43ea7fc700 1 -- 172.21.15.158:0/3634237847 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc011d20 con 0x7f43ec15a6d0 2023-10-16T09:44:16.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 -- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec15a6d0 msgr2=0x7f43ec0ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 --2- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec15a6d0 0x7f43ec0ce2d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0098f0 tx=0x7f43dc009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:16.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 -- 172.21.15.158:0/3634237847 shutdown_connections 2023-10-16T09:44:16.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 --2- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cd360 0x7f43ec06ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 --2- 172.21.15.158:0/3634237847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec15a6d0 0x7f43ec0ce2d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 -- 172.21.15.158:0/3634237847 >> 172.21.15.158:0/3634237847 conn(0x7f43ec06b9e0 msgr2=0x7f43ec06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:16.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 -- 172.21.15.158:0/3634237847 shutdown_connections 2023-10-16T09:44:16.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.969+0000 7f43f219b700 1 -- 172.21.15.158:0/3634237847 wait complete. 2023-10-16T09:44:16.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.971+0000 7f43f219b700 1 Processor -- start 2023-10-16T09:44:16.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.971+0000 7f43f219b700 1 -- start start 2023-10-16T09:44:16.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43f219b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec0cd360 0x7f43ec06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43f219b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cfe30 0x7f43ec15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43f219b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43ec15a6d0 0x7f43ec147600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43f219b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec157fd0 con 0x7f43ec15a6d0 2023-10-16T09:44:16.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43f219b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec15d6d0 con 0x7f43ec0cd360 2023-10-16T09:44:16.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43f219b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec15db60 con 0x7f43ec0cfe30 2023-10-16T09:44:16.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43ec15a6d0 0x7f43ec147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec0cd360 0x7f43ec06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43ec15a6d0 0x7f43ec147600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47668/0 (socket says 172.21.15.158:47668) 2023-10-16T09:44:16.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43ebfff700 1 -- 172.21.15.158:0/105232947 learned_addr learned my addr 172.21.15.158:0/105232947 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:16.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43eaffd700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cfe30 0x7f43ec15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43ebfff700 1 -- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cfe30 msgr2=0x7f43ec15aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43ebfff700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cfe30 0x7f43ec15aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43ebfff700 1 -- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec0cd360 msgr2=0x7f43ec06e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:16.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.972+0000 7f43ebfff700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec0cd360 0x7f43ec06e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:16.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43ebfff700 1 -- 172.21.15.158:0/105232947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e4009a70 con 0x7f43ec15a6d0 2023-10-16T09:44:16.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43eb7fe700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec0cd360 0x7f43ec06e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:16.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43ebfff700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43ec15a6d0 0x7f43ec147600 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f43e400efc0 tx=0x7f43e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43e3fff700 1 -- 172.21.15.158:0/105232947 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e400ca90 con 0x7f43ec15a6d0 2023-10-16T09:44:16.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43e3fff700 1 -- 172.21.15.158:0/105232947 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e400f040 con 0x7f43ec15a6d0 2023-10-16T09:44:16.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43e3fff700 1 -- 172.21.15.158:0/105232947 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e400ce30 con 0x7f43ec15a6d0 2023-10-16T09:44:16.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43dc009550 con 0x7f43ec15a6d0 2023-10-16T09:44:16.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.973+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec15cda0 con 0x7f43ec15a6d0 2023-10-16T09:44:16.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.974+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec15d840 con 0x7f43ec15a6d0 2023-10-16T09:44:16.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.974+0000 7f43e3fff700 1 -- 172.21.15.158:0/105232947 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f43e4014480 con 0x7f43ec15a6d0 2023-10-16T09:44:16.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.975+0000 7f43e3fff700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f43cc06af00 0x7f43cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:16.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.975+0000 7f43eb7fe700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f43cc06af00 0x7f43cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:16.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.975+0000 7f43e3fff700 1 -- 172.21.15.158:0/105232947 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f43e408a060 con 0x7f43ec15a6d0 2023-10-16T09:44:16.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.975+0000 7f43eb7fe700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f43cc06af00 0x7f43cc06d3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f43dc009c40 tx=0x7f43dc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:16.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:16.979+0000 7f43e3fff700 1 -- 172.21.15.158:0/105232947 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43e4054d00 con 0x7f43ec15a6d0 2023-10-16T09:44:17.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.185+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f43ec0681b0 con 0x7f43ec15a6d0 2023-10-16T09:44:17.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.186+0000 7f43e3fff700 1 -- 172.21.15.158:0/105232947 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f43e4058320 con 0x7f43ec15a6d0 2023-10-16T09:44:17.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f43cc06af00 msgr2=0x7f43cc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f43cc06af00 0x7f43cc06d3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f43dc009c40 tx=0x7f43dc008030 comp rx=0 tx=0).stop 2023-10-16T09:44:17.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43ec15a6d0 msgr2=0x7f43ec147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43ec15a6d0 0x7f43ec147600 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f43e400efc0 tx=0x7f43e400c3a0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 shutdown_connections 2023-10-16T09:44:17.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43ec15a6d0 0x7f43ec147600 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43ec0cd360 0x7f43ec06e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f43cc06af00 0x7f43cc06d3c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 --2- 172.21.15.158:0/105232947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43ec0cfe30 0x7f43ec15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 >> 172.21.15.158:0/105232947 conn(0x7f43ec06b9e0 msgr2=0x7f43ec0b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:17.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 shutdown_connections 2023-10-16T09:44:17.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.188+0000 7f43f219b700 1 -- 172.21.15.158:0/105232947 wait complete. 2023-10-16T09:44:17.192 INFO:teuthology.orchestra.run.smithi158.stderr:listed 0 entries 2023-10-16T09:44:17.201 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T09:44:17.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:17.202 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-16T09:44:17.243 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:44:17.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:17.244 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T09:44:17.300 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:17.300 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:17.322 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:17.323 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:17.383 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:17.384 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:17.444 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:17.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.564+0000 7fd91ee31700 1 Processor -- start 2023-10-16T09:44:17.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.564+0000 7fd91ee31700 1 -- start start 2023-10-16T09:44:17.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91ee31700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd91815a6c0 0x7fd9180723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91ee31700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd91806e090 0x7fd9180ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91ee31700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd918072910 con 0x7fd9180af590 2023-10-16T09:44:17.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91ee31700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180ca7e0 con 0x7fd91815a6c0 2023-10-16T09:44:17.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91ee31700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180b1e40 con 0x7fd91806e090 2023-10-16T09:44:17.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91cbcd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd91815a6c0 0x7fd9180723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91cbcd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd91815a6c0 0x7fd9180723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47672/0 (socket says 172.21.15.158:47672) 2023-10-16T09:44:17.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1158800856 learned_addr learned my addr 172.21.15.158:0/1158800856 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:17.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd917fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd9180af590 0x7fd91806d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47848/0 (socket says 172.21.15.158:47848) 2023-10-16T09:44:17.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91d3ce700 1 --2- 172.21.15.158:0/1158800856 >> v2:172.21.15.172:3300/0 conn(0x7fd91806e090 0x7fd9180ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1158800856 >> v2:172.21.15.172:3300/0 conn(0x7fd91806e090 msgr2=0x7fd9180ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:17.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91cbcd700 1 --2- 172.21.15.158:0/1158800856 >> v2:172.21.15.172:3300/0 conn(0x7fd91806e090 0x7fd9180ad430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1158800856 >> v1:172.21.15.158:6789/0 conn(0x7fd9180af590 legacy=0x7fd91806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd9177fe700 1 -- 172.21.15.158:0/1158800856 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362598024 0 0) 0x7fd918072910 con 0x7fd9180af590 2023-10-16T09:44:17.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.565+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1158800856 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9180b2120 con 0x7fd91815a6c0 2023-10-16T09:44:17.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91cbcd700 1 --2- 172.21.15.158:0/1158800856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91815a6c0 0x7fd9180723d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd9180b29f0 tx=0x7fd90c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaefa82d0c586089 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:17.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd9177fe700 1 -- 172.21.15.158:0/1158800856 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c005e30 con 0x7fd91815a6c0 2023-10-16T09:44:17.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd9177fe700 1 -- 172.21.15.158:0/1158800856 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c004030 con 0x7fd91815a6c0 2023-10-16T09:44:17.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd9177fe700 1 -- 172.21.15.158:0/1158800856 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0043e0 con 0x7fd91815a6c0 2023-10-16T09:44:17.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 -- 172.21.15.158:0/1158800856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91815a6c0 msgr2=0x7fd9180723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1158800856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91815a6c0 0x7fd9180723d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd9180b29f0 tx=0x7fd90c009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 -- 172.21.15.158:0/1158800856 shutdown_connections 2023-10-16T09:44:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1158800856 >> v2:172.21.15.172:3300/0 conn(0x7fd91806e090 0x7fd9180ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1158800856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91815a6c0 0x7fd9180723d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 -- 172.21.15.158:0/1158800856 >> 172.21.15.158:0/1158800856 conn(0x7fd9180ccda0 msgr2=0x7fd9180cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 -- 172.21.15.158:0/1158800856 shutdown_connections 2023-10-16T09:44:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.566+0000 7fd91ee31700 1 -- 172.21.15.158:0/1158800856 wait complete. 2023-10-16T09:44:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.567+0000 7fd91ee31700 1 Processor -- start 2023-10-16T09:44:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91ee31700 1 -- start start 2023-10-16T09:44:17.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91ee31700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91806e090 0x7fd91815aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91ee31700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9180af590 0x7fd9181f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91ee31700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd91815a6c0 0x7fd9181f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91ee31700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91814f890 con 0x7fd91806e090 2023-10-16T09:44:17.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91ee31700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180c59e0 con 0x7fd9180af590 2023-10-16T09:44:17.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91ee31700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180c5cc0 con 0x7fd91815a6c0 2023-10-16T09:44:17.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91cbcd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91806e090 0x7fd91815aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.568+0000 7fd91cbcd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91806e090 0x7fd91815aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47676/0 (socket says 172.21.15.158:47676) 2023-10-16T09:44:17.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1816989704 learned_addr learned my addr 172.21.15.158:0/1816989704 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:17.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd917fff700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9180af590 0x7fd9181f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91d3ce700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd91815a6c0 0x7fd9181f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd91815a6c0 msgr2=0x7fd9181f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91cbcd700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd91815a6c0 0x7fd9181f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9180af590 msgr2=0x7fd9181f4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91cbcd700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9180af590 0x7fd9181f4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91cbcd700 1 -- 172.21.15.158:0/1816989704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c009500 con 0x7fd91806e090 2023-10-16T09:44:17.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd917fff700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9180af590 0x7fd9181f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:17.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91cbcd700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91806e090 0x7fd91815aaa0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009c10 tx=0x7fd90c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:17.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd9157fa700 1 -- 172.21.15.158:0/1816989704 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c003e70 con 0x7fd91806e090 2023-10-16T09:44:17.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd9157fa700 1 -- 172.21.15.158:0/1816989704 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c011420 con 0x7fd91806e090 2023-10-16T09:44:17.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9180c5fa0 con 0x7fd91806e090 2023-10-16T09:44:17.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd9157fa700 1 -- 172.21.15.158:0/1816989704 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0117f0 con 0x7fd91806e090 2023-10-16T09:44:17.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.569+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9180c6560 con 0x7fd91806e090 2023-10-16T09:44:17.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.570+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9180681b0 con 0x7fd91806e090 2023-10-16T09:44:17.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.571+0000 7fd9157fa700 1 -- 172.21.15.158:0/1816989704 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd90c011950 con 0x7fd91806e090 2023-10-16T09:44:17.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.571+0000 7fd9157fa700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd8fc06af00 0x7fd8fc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.571+0000 7fd917fff700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd8fc06af00 0x7fd8fc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.571+0000 7fd9157fa700 1 -- 172.21.15.158:0/1816989704 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fd90c08ad70 con 0x7fd91806e090 2023-10-16T09:44:17.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.572+0000 7fd917fff700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd8fc06af00 0x7fd8fc06d3c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd918159720 tx=0x7fd904007ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:17.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.576+0000 7fd9157fa700 1 -- 172.21.15.158:0/1816989704 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd90c055bc0 con 0x7fd91806e090 2023-10-16T09:44:17.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.768+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9180c36b0 con 0x7fd91806e090 2023-10-16T09:44:17.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.770+0000 7fd9157fa700 1 -- 172.21.15.158:0/1816989704 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7fd90c0591e0 con 0x7fd91806e090 2023-10-16T09:44:17.772 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:17.772 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:17.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd8fc06af00 msgr2=0x7fd8fc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd8fc06af00 0x7fd8fc06d3c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd918159720 tx=0x7fd904007ca0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91806e090 msgr2=0x7fd91815aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91806e090 0x7fd91815aaa0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009c10 tx=0x7fd90c004750 comp rx=0 tx=0).stop 2023-10-16T09:44:17.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 shutdown_connections 2023-10-16T09:44:17.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fd8fc06af00 0x7fd8fc06d3c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd91815a6c0 0x7fd9181f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd91806e090 0x7fd91815aaa0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 --2- 172.21.15.158:0/1816989704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9180af590 0x7fd9181f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 >> 172.21.15.158:0/1816989704 conn(0x7fd9180ccda0 msgr2=0x7fd9180cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 shutdown_connections 2023-10-16T09:44:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.772+0000 7fd91ee31700 1 -- 172.21.15.158:0/1816989704 wait complete. 2023-10-16T09:44:17.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:17.790 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.912+0000 7fe932bbf700 1 Processor -- start 2023-10-16T09:44:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.912+0000 7fe932bbf700 1 -- start start 2023-10-16T09:44:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe932bbf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe92c15dad0 0x7fe92c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe932bbf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c0d0ae0 con 0x7fe92c0ccc00 2023-10-16T09:44:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe932bbf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c0d12d0 con 0x7fe92c15deb0 2023-10-16T09:44:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe932bbf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c0cbdf0 con 0x7fe92c15dad0 2023-10-16T09:44:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe93095b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe92c15dad0 0x7fe92c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe93095b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe92c15dad0 0x7fe92c0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47684/0 (socket says 172.21.15.158:47684) 2023-10-16T09:44:17.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe93095b700 1 -- 172.21.15.158:0/609052048 learned_addr learned my addr 172.21.15.158:0/609052048 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:17.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe93115c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe92c0ccc00 0x7fe92c0cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47864/0 (socket says 172.21.15.158:47864) 2023-10-16T09:44:17.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe93095b700 1 -- 172.21.15.158:0/609052048 >> v1:172.21.15.172:6789/0 conn(0x7fe92c15deb0 legacy=0x7fe92c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:17.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe93095b700 1 -- 172.21.15.158:0/609052048 >> v1:172.21.15.158:6789/0 conn(0x7fe92c0ccc00 legacy=0x7fe92c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.913+0000 7fe92b7fe700 1 -- 172.21.15.158:0/609052048 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046809576 0 0) 0x7fe92c0d0ae0 con 0x7fe92c0ccc00 2023-10-16T09:44:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe93095b700 1 -- 172.21.15.158:0/609052048 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c0cc280 con 0x7fe92c15dad0 2023-10-16T09:44:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe93095b700 1 --2- 172.21.15.158:0/609052048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c15dad0 0x7fe92c0cdfa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe92000bbc0 tx=0x7fe92000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a880f31731adc3ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe92b7fe700 1 -- 172.21.15.158:0/609052048 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92000dd50 con 0x7fe92c15dad0 2023-10-16T09:44:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe92b7fe700 1 -- 172.21.15.158:0/609052048 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920010470 con 0x7fe92c15dad0 2023-10-16T09:44:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe92b7fe700 1 -- 172.21.15.158:0/609052048 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920010800 con 0x7fe92c15dad0 2023-10-16T09:44:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe932bbf700 1 -- 172.21.15.158:0/609052048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c15dad0 msgr2=0x7fe92c0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe932bbf700 1 --2- 172.21.15.158:0/609052048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c15dad0 0x7fe92c0cdfa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe92000bbc0 tx=0x7fe92000bed0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe932bbf700 1 -- 172.21.15.158:0/609052048 shutdown_connections 2023-10-16T09:44:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe932bbf700 1 --2- 172.21.15.158:0/609052048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c15dad0 0x7fe92c0cdfa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe932bbf700 1 -- 172.21.15.158:0/609052048 >> 172.21.15.158:0/609052048 conn(0x7fe92c099a10 msgr2=0x7fe92c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe932bbf700 1 -- 172.21.15.158:0/609052048 shutdown_connections 2023-10-16T09:44:17.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.914+0000 7fe932bbf700 1 -- 172.21.15.158:0/609052048 wait complete. 2023-10-16T09:44:17.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.916+0000 7fe932bbf700 1 Processor -- start 2023-10-16T09:44:17.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.916+0000 7fe932bbf700 1 -- start start 2023-10-16T09:44:17.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe932bbf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c0ccc00 0x7fe92c1ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe932bbf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe92c15dad0 0x7fe92c0b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe932bbf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe92c15deb0 0x7fe92c0b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe932bbf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c14c050 con 0x7fe92c0ccc00 2023-10-16T09:44:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe932bbf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0aef50 con 0x7fe92c15dad0 2023-10-16T09:44:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe932bbf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0b55a0 con 0x7fe92c15deb0 2023-10-16T09:44:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe93095b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c0ccc00 0x7fe92c1ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe93095b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c0ccc00 0x7fe92c1ee520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47692/0 (socket says 172.21.15.158:47692) 2023-10-16T09:44:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe93095b700 1 -- 172.21.15.158:0/132446761 learned_addr learned my addr 172.21.15.158:0/132446761 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe92bfff700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe92c15dad0 0x7fe92c0b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.917+0000 7fe93115c700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe92c15deb0 0x7fe92c0b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe93095b700 1 -- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe92c15deb0 msgr2=0x7fe92c0b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe93095b700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe92c15deb0 0x7fe92c0b4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe93095b700 1 -- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe92c15dad0 msgr2=0x7fe92c0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe93095b700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe92c15dad0 0x7fe92c0b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe93095b700 1 -- 172.21.15.158:0/132446761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92000b7f0 con 0x7fe92c0ccc00 2023-10-16T09:44:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe93095b700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c0ccc00 0x7fe92c1ee520 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe92000b340 tx=0x7fe920004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:17.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe9297fa700 1 -- 172.21.15.158:0/132446761 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92001f070 con 0x7fe92c0ccc00 2023-10-16T09:44:17.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe9297fa700 1 -- 172.21.15.158:0/132446761 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920024470 con 0x7fe92c0ccc00 2023-10-16T09:44:17.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe9297fa700 1 -- 172.21.15.158:0/132446761 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920024800 con 0x7fe92c0ccc00 2023-10-16T09:44:17.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c0b5880 con 0x7fe92c0ccc00 2023-10-16T09:44:17.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.918+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c0aebe0 con 0x7fe92c0ccc00 2023-10-16T09:44:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.919+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c0ae760 con 0x7fe92c0ccc00 2023-10-16T09:44:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.919+0000 7fe9297fa700 1 -- 172.21.15.158:0/132446761 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe920019400 con 0x7fe92c0ccc00 2023-10-16T09:44:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.920+0000 7fe9297fa700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe91006af00 0x7fe91006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.920+0000 7fe9297fa700 1 -- 172.21.15.158:0/132446761 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fe92008d2d0 con 0x7fe92c0ccc00 2023-10-16T09:44:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.920+0000 7fe92bfff700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe91006af00 0x7fe91006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.920+0000 7fe92bfff700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe91006af00 0x7fe91006d3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe92c1f2180 tx=0x7fe918009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:17.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:17.925+0000 7fe9297fa700 1 -- 172.21.15.158:0/132446761 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe92c0ae760 con 0x7fe92c0ccc00 2023-10-16T09:44:18.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.116+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe92c0ae760 con 0x7fe92c0ccc00 2023-10-16T09:44:18.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.117+0000 7fe9297fa700 1 -- 172.21.15.158:0/132446761 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe92c0ae760 con 0x7fe92c0ccc00 2023-10-16T09:44:18.119 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:18.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe91006af00 msgr2=0x7fe91006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe91006af00 0x7fe91006d3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe92c1f2180 tx=0x7fe918009040 comp rx=0 tx=0).stop 2023-10-16T09:44:18.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c0ccc00 msgr2=0x7fe92c1ee520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c0ccc00 0x7fe92c1ee520 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe92000b340 tx=0x7fe920004750 comp rx=0 tx=0).stop 2023-10-16T09:44:18.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 shutdown_connections 2023-10-16T09:44:18.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe91006af00 0x7fe91006d3c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe92c15deb0 0x7fe92c0b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe92c0ccc00 0x7fe92c1ee520 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 --2- 172.21.15.158:0/132446761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe92c15dad0 0x7fe92c0b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 >> 172.21.15.158:0/132446761 conn(0x7fe92c099a10 msgr2=0x7fe92c15b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 shutdown_connections 2023-10-16T09:44:18.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.119+0000 7fe932bbf700 1 -- 172.21.15.158:0/132446761 wait complete. 2023-10-16T09:44:18.132 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.254+0000 7f05b21b5700 1 Processor -- start 2023-10-16T09:44:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.254+0000 7f05b21b5700 1 -- start start 2023-10-16T09:44:18.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05b21b5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05ac0cf070 0x7f05ac0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05b21b5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac0cbdd0 0x7f05ac0cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05b21b5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac15dc20 con 0x7f05ac15d840 2023-10-16T09:44:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05b21b5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac0cfa60 con 0x7f05ac0cf070 2023-10-16T09:44:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05b21b5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac06c0e0 con 0x7f05ac0cbdd0 2023-10-16T09:44:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05ac0cf070 0x7f05ac0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05ac0cf070 0x7f05ac0cf520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47700/0 (socket says 172.21.15.158:47700) 2023-10-16T09:44:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05ab7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f05ac15d840 0x7f05ac161d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47874/0 (socket says 172.21.15.158:47874) 2023-10-16T09:44:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aaffd700 1 -- 172.21.15.158:0/2707007791 learned_addr learned my addr 172.21.15.158:0/2707007791 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05abfff700 1 --2- 172.21.15.158:0/2707007791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac0cbdd0 0x7f05ac0cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aaffd700 1 -- 172.21.15.158:0/2707007791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac0cbdd0 msgr2=0x7f05ac0cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aaffd700 1 --2- 172.21.15.158:0/2707007791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac0cbdd0 0x7f05ac0cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aaffd700 1 -- 172.21.15.158:0/2707007791 >> v1:172.21.15.158:6789/0 conn(0x7f05ac15d840 legacy=0x7f05ac161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aa7fc700 1 -- 172.21.15.158:0/2707007791 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594069497 0 0) 0x7f05a4001110 con 0x7f05ac15d840 2023-10-16T09:44:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.255+0000 7f05aaffd700 1 -- 172.21.15.158:0/2707007791 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac06c3c0 con 0x7f05ac0cf070 2023-10-16T09:44:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05aaffd700 1 --2- 172.21.15.158:0/2707007791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac0cf520 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0594009e50 tx=0x7f059400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af47d021ea4650d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05aa7fc700 1 -- 172.21.15.158:0/2707007791 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059400e070 con 0x7f05ac0cf070 2023-10-16T09:44:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05aa7fc700 1 -- 172.21.15.158:0/2707007791 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059400b9c0 con 0x7f05ac0cf070 2023-10-16T09:44:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05aa7fc700 1 -- 172.21.15.158:0/2707007791 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059400bd70 con 0x7f05ac0cf070 2023-10-16T09:44:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 -- 172.21.15.158:0/2707007791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 msgr2=0x7f05ac0cf520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 --2- 172.21.15.158:0/2707007791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac0cf520 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0594009e50 tx=0x7f059400b330 comp rx=0 tx=0).stop 2023-10-16T09:44:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 -- 172.21.15.158:0/2707007791 shutdown_connections 2023-10-16T09:44:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 --2- 172.21.15.158:0/2707007791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac0cbdd0 0x7f05ac0cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 --2- 172.21.15.158:0/2707007791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac0cf520 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 -- 172.21.15.158:0/2707007791 >> 172.21.15.158:0/2707007791 conn(0x7f05ac06a8d0 msgr2=0x7f05ac06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 -- 172.21.15.158:0/2707007791 shutdown_connections 2023-10-16T09:44:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.256+0000 7f05b21b5700 1 -- 172.21.15.158:0/2707007791 wait complete. 2023-10-16T09:44:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.257+0000 7f05b21b5700 1 Processor -- start 2023-10-16T09:44:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.258+0000 7f05b21b5700 1 -- start start 2023-10-16T09:44:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.258+0000 7f05b21b5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05ac0cbdd0 0x7f05ac15a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.258+0000 7f05b21b5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac15ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.258+0000 7f05b21b5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac15d840 0x7f05ac0b6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.258+0000 7f05b21b5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac15a890 con 0x7f05ac0cf070 2023-10-16T09:44:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.258+0000 7f05b21b5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0c8f80 con 0x7f05ac0cbdd0 2023-10-16T09:44:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.258+0000 7f05b21b5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0b4ff0 con 0x7f05ac15d840 2023-10-16T09:44:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac15ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac15ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47706/0 (socket says 172.21.15.158:47706) 2023-10-16T09:44:18.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 -- 172.21.15.158:0/3418384922 learned_addr learned my addr 172.21.15.158:0/3418384922 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:18.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05ab7fe700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05ac0cbdd0 0x7f05ac15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05abfff700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac15d840 0x7f05ac0b6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 -- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac15d840 msgr2=0x7f05ac0b6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac15d840 0x7f05ac0b6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 -- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05ac0cbdd0 msgr2=0x7f05ac15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05ac0cbdd0 0x7f05ac15a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 -- 172.21.15.158:0/3418384922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594009b30 con 0x7f05ac0cf070 2023-10-16T09:44:18.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05aaffd700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac15ff20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f059400b730 tx=0x7f0594004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05a3fff700 1 -- 172.21.15.158:0/3418384922 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594003ea0 con 0x7f05ac0cf070 2023-10-16T09:44:18.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05a3fff700 1 -- 172.21.15.158:0/3418384922 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05940099b0 con 0x7f05ac0cf070 2023-10-16T09:44:18.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05a3fff700 1 -- 172.21.15.158:0/3418384922 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059401c660 con 0x7f05ac0cf070 2023-10-16T09:44:18.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac0b52d0 con 0x7f05ac0cf070 2023-10-16T09:44:18.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.259+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac0b5890 con 0x7f05ac0cf070 2023-10-16T09:44:18.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.260+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac0c9110 con 0x7f05ac0cf070 2023-10-16T09:44:18.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.261+0000 7f05a3fff700 1 -- 172.21.15.158:0/3418384922 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f059401c7c0 con 0x7f05ac0cf070 2023-10-16T09:44:18.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.261+0000 7f05a3fff700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f058c06af00 0x7f058c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.261+0000 7f05a3fff700 1 -- 172.21.15.158:0/3418384922 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f059408b400 con 0x7f05ac0cf070 2023-10-16T09:44:18.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.261+0000 7f05ab7fe700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f058c06af00 0x7f058c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.261+0000 7f05ab7fe700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f058c06af00 0x7f058c06d3c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f05a4000f60 tx=0x7f05a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.265+0000 7f05a3fff700 1 -- 172.21.15.158:0/3418384922 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0594056130 con 0x7f05ac0cf070 2023-10-16T09:44:18.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.456+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05ac0b0c00 con 0x7f05ac0cf070 2023-10-16T09:44:18.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.456+0000 7f05a3fff700 1 -- 172.21.15.158:0/3418384922 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0594059750 con 0x7f05ac0cf070 2023-10-16T09:44:18.458 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:18.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f058c06af00 msgr2=0x7f058c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f058c06af00 0x7f058c06d3c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f05a4000f60 tx=0x7f05a4009040 comp rx=0 tx=0).stop 2023-10-16T09:44:18.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 msgr2=0x7f05ac15ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac15ff20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f059400b730 tx=0x7f0594004ad0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 shutdown_connections 2023-10-16T09:44:18.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f058c06af00 0x7f058c06d3c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05ac15d840 0x7f05ac0b6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.458+0000 7f05b21b5700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05ac0cbdd0 0x7f05ac15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.459+0000 7f05b21b5700 1 --2- 172.21.15.158:0/3418384922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05ac0cf070 0x7f05ac15ff20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.459+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 >> 172.21.15.158:0/3418384922 conn(0x7f05ac06a8d0 msgr2=0x7f05ac0cdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:18.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.459+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 shutdown_connections 2023-10-16T09:44:18.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.459+0000 7f05b21b5700 1 -- 172.21.15.158:0/3418384922 wait complete. 2023-10-16T09:44:18.473 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:18.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.595+0000 7f14b45f4700 1 Processor -- start 2023-10-16T09:44:18.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.595+0000 7f14b45f4700 1 -- start start 2023-10-16T09:44:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b45f4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b45f4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14ac0cc590 0x7f14ac0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b45f4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac0cbd30 con 0x7f14ac15da40 2023-10-16T09:44:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b45f4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ac0cc1c0 con 0x7f14ac0cc590 2023-10-16T09:44:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b45f4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac15de20 con 0x7f14ac06afe0 2023-10-16T09:44:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2390700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f14ac15da40 0x7f14ac06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47888/0 (socket says 172.21.15.158:47888) 2023-10-16T09:44:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2390700 1 -- 172.21.15.158:0/2877083165 learned_addr learned my addr 172.21.15.158:0/2877083165 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2b91700 1 --2- 172.21.15.158:0/2877083165 >> v2:172.21.15.158:3300/0 conn(0x7f14ac0cc590 0x7f14ac0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b1b8f700 1 --2- 172.21.15.158:0/2877083165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2b91700 1 -- 172.21.15.158:0/2877083165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 msgr2=0x7f14ac0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2b91700 1 --2- 172.21.15.158:0/2877083165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2b91700 1 -- 172.21.15.158:0/2877083165 >> v1:172.21.15.158:6789/0 conn(0x7f14ac15da40 legacy=0x7f14ac06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2b91700 1 -- 172.21.15.158:0/2877083165 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac0ce260 con 0x7f14ac0cc590 2023-10-16T09:44:18.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b138e700 1 -- 172.21.15.158:0/2877083165 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485552615 0 0) 0x7f14ac0cbd30 con 0x7f14ac15da40 2023-10-16T09:44:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b1b8f700 1 --2- 172.21.15.158:0/2877083165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b2b91700 1 --2- 172.21.15.158:0/2877083165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac0cc590 0x7f14ac0afa40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009d90 tx=0x7f14a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f788c13e086e65f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b138e700 1 -- 172.21.15.158:0/2877083165 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400e070 con 0x7f14ac0cc590 2023-10-16T09:44:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b138e700 1 -- 172.21.15.158:0/2877083165 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a400bbd0 con 0x7f14ac0cc590 2023-10-16T09:44:18.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.596+0000 7f14b138e700 1 -- 172.21.15.158:0/2877083165 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4012430 con 0x7f14ac0cc590 2023-10-16T09:44:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 -- 172.21.15.158:0/2877083165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac0cc590 msgr2=0x7f14ac0afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 --2- 172.21.15.158:0/2877083165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac0cc590 0x7f14ac0afa40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009d90 tx=0x7f14a400b330 comp rx=0 tx=0).stop 2023-10-16T09:44:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 -- 172.21.15.158:0/2877083165 shutdown_connections 2023-10-16T09:44:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 --2- 172.21.15.158:0/2877083165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac0cc590 0x7f14ac0afa40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 --2- 172.21.15.158:0/2877083165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 -- 172.21.15.158:0/2877083165 >> 172.21.15.158:0/2877083165 conn(0x7f14ac099a10 msgr2=0x7f14ac0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:18.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 -- 172.21.15.158:0/2877083165 shutdown_connections 2023-10-16T09:44:18.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.597+0000 7f14b45f4700 1 -- 172.21.15.158:0/2877083165 wait complete. 2023-10-16T09:44:18.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.598+0000 7f14b45f4700 1 Processor -- start 2023-10-16T09:44:18.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.598+0000 7f14b45f4700 1 -- start start 2023-10-16T09:44:18.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b45f4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b45f4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14ac0cc590 0x7f14ac0d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b45f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac15da40 0x7f14ac0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b45f4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0c6710 con 0x7f14ac15da40 2023-10-16T09:44:18.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b45f4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0c5b00 con 0x7f14ac0cc590 2023-10-16T09:44:18.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b45f4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0c5de0 con 0x7f14ac06afe0 2023-10-16T09:44:18.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac15da40 0x7f14ac0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac15da40 0x7f14ac0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47722/0 (socket says 172.21.15.158:47722) 2023-10-16T09:44:18.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 -- 172.21.15.158:0/1195237336 learned_addr learned my addr 172.21.15.158:0/1195237336 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:18.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2390700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b1b8f700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14ac0cc590 0x7f14ac0d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 -- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 msgr2=0x7f14ac15aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac15aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 -- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14ac0cc590 msgr2=0x7f14ac0d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14ac0cc590 0x7f14ac0d09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.599+0000 7f14b2b91700 1 -- 172.21.15.158:0/1195237336 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4009a70 con 0x7f14ac15da40 2023-10-16T09:44:18.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.600+0000 7f14b2b91700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac15da40 0x7f14ac0b2fc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0702f0 tx=0x7f14a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.600+0000 7f14a2ffd700 1 -- 172.21.15.158:0/1195237336 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4012430 con 0x7f14ac15da40 2023-10-16T09:44:18.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.600+0000 7f14a2ffd700 1 -- 172.21.15.158:0/1195237336 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4020070 con 0x7f14ac15da40 2023-10-16T09:44:18.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.600+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac0c60c0 con 0x7f14ac15da40 2023-10-16T09:44:18.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.600+0000 7f14a2ffd700 1 -- 172.21.15.158:0/1195237336 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4004280 con 0x7f14ac15da40 2023-10-16T09:44:18.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.600+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac15f900 con 0x7f14ac15da40 2023-10-16T09:44:18.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.601+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac15fac0 con 0x7f14ac15da40 2023-10-16T09:44:18.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.601+0000 7f14a2ffd700 1 -- 172.21.15.158:0/1195237336 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f14a40043e0 con 0x7f14ac15da40 2023-10-16T09:44:18.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.601+0000 7f14a2ffd700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f149006af00 0x7f149006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.601+0000 7f14a2ffd700 1 -- 172.21.15.158:0/1195237336 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f14a4022070 con 0x7f14ac15da40 2023-10-16T09:44:18.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.601+0000 7f14b2390700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f149006af00 0x7f149006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.602+0000 7f14b2390700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f149006af00 0x7f149006d3c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f14a8007fd0 tx=0x7f14a8007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.606+0000 7f14a2ffd700 1 -- 172.21.15.158:0/1195237336 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14a40568d0 con 0x7f14ac15da40 2023-10-16T09:44:18.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.797+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14ac0c6850 con 0x7f14ac15da40 2023-10-16T09:44:18.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.797+0000 7f14a2ffd700 1 -- 172.21.15.158:0/1195237336 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f14ac0c6850 con 0x7f14ac15da40 2023-10-16T09:44:18.799 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:18.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f149006af00 msgr2=0x7f149006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f149006af00 0x7f149006d3c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f14a8007fd0 tx=0x7f14a8007de0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac15da40 msgr2=0x7f14ac0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac15da40 0x7f14ac0b2fc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0702f0 tx=0x7f14a4004ad0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 shutdown_connections 2023-10-16T09:44:18.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14ac15da40 0x7f14ac0b2fc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f149006af00 0x7f149006d3c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14ac06afe0 0x7f14ac15aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 --2- 172.21.15.158:0/1195237336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14ac0cc590 0x7f14ac0d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.799+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 >> 172.21.15.158:0/1195237336 conn(0x7f14ac099a10 msgr2=0x7f14ac15b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:18.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.800+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 shutdown_connections 2023-10-16T09:44:18.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.800+0000 7f14b45f4700 1 -- 172.21.15.158:0/1195237336 wait complete. 2023-10-16T09:44:18.813 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.935+0000 7fbac0cdf700 1 Processor -- start 2023-10-16T09:44:18.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.935+0000 7fbac0cdf700 1 -- start start 2023-10-16T09:44:18.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbac0cdf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbabc151220 0x7fbabc0665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbac0cdf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc0cf190 0x7fbabc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbac0cdf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbabc0cf930 con 0x7fbabc06d950 2023-10-16T09:44:18.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbac0cdf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbabc0cf650 con 0x7fbabc151220 2023-10-16T09:44:18.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbac0cdf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc0c51b0 con 0x7fbabc0cf190 2023-10-16T09:44:18.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbabad9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbabc06d950 0x7fbabc0c5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47906/0 (socket says 172.21.15.158:47906) 2023-10-16T09:44:18.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbabad9d700 1 -- 172.21.15.158:0/2180724204 learned_addr learned my addr 172.21.15.158:0/2180724204 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:18.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbaba59c700 1 --2- 172.21.15.158:0/2180724204 >> v2:172.21.15.172:3300/0 conn(0x7fbabc151220 0x7fbabc0665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbab9d9b700 1 --2- 172.21.15.158:0/2180724204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc0cf190 0x7fbabc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbab959a700 1 -- 172.21.15.158:0/2180724204 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340429997 0 0) 0x7fbabc0cf930 con 0x7fbabc06d950 2023-10-16T09:44:18.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbab959a700 1 -- 172.21.15.158:0/2180724204 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa40035b0 con 0x7fbabc06d950 2023-10-16T09:44:18.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbab9d9b700 1 -- 172.21.15.158:0/2180724204 >> v2:172.21.15.172:3300/0 conn(0x7fbabc151220 msgr2=0x7fbabc0665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:18.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbab9d9b700 1 --2- 172.21.15.158:0/2180724204 >> v2:172.21.15.172:3300/0 conn(0x7fbabc151220 0x7fbabc0665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbab9d9b700 1 -- 172.21.15.158:0/2180724204 >> v1:172.21.15.158:6789/0 conn(0x7fbabc06d950 legacy=0x7fbabc0c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbab9d9b700 1 -- 172.21.15.158:0/2180724204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc0c5490 con 0x7fbabc0cf190 2023-10-16T09:44:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.936+0000 7fbaba59c700 1 --2- 172.21.15.158:0/2180724204 >> v2:172.21.15.172:3300/0 conn(0x7fbabc151220 0x7fbabc0665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbab9d9b700 1 --2- 172.21.15.158:0/2180724204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc0cf190 0x7fbabc06d410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009840 tx=0x7fbab0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7a2edd254cdedd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbab959a700 1 -- 172.21.15.158:0/2180724204 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0011830 con 0x7fbabc0cf190 2023-10-16T09:44:18.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbab959a700 1 -- 172.21.15.158:0/2180724204 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab0011990 con 0x7fbabc0cf190 2023-10-16T09:44:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbab959a700 1 -- 172.21.15.158:0/2180724204 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0011d20 con 0x7fbabc0cf190 2023-10-16T09:44:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 -- 172.21.15.158:0/2180724204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc0cf190 msgr2=0x7fbabc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/2180724204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc0cf190 0x7fbabc06d410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009840 tx=0x7fbab0009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:18.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 -- 172.21.15.158:0/2180724204 shutdown_connections 2023-10-16T09:44:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/2180724204 >> v2:172.21.15.172:3300/0 conn(0x7fbabc151220 0x7fbabc0665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/2180724204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc0cf190 0x7fbabc06d410 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 -- 172.21.15.158:0/2180724204 >> 172.21.15.158:0/2180724204 conn(0x7fbabc0cc5b0 msgr2=0x7fbabc0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:18.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 -- 172.21.15.158:0/2180724204 shutdown_connections 2023-10-16T09:44:18.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.937+0000 7fbac0cdf700 1 -- 172.21.15.158:0/2180724204 wait complete. 2023-10-16T09:44:18.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 Processor -- start 2023-10-16T09:44:18.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 -- start start 2023-10-16T09:44:18.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbabc06d950 0x7fbabc148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbabc0cf190 0x7fbabc153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc151220 0x7fbabc15d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc1543c0 con 0x7fbabc06d950 2023-10-16T09:44:18.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc148eb0 con 0x7fbabc0cf190 2023-10-16T09:44:18.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbac0cdf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc0b3dd0 con 0x7fbabc151220 2023-10-16T09:44:18.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.939+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbabc06d950 0x7fbabc148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbabc06d950 0x7fbabc148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47732/0 (socket says 172.21.15.158:47732) 2023-10-16T09:44:18.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 -- 172.21.15.158:0/848813046 learned_addr learned my addr 172.21.15.158:0/848813046 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:18.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbabad9d700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc151220 0x7fbabc15d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbab9d9b700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbabc0cf190 0x7fbabc153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 -- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc151220 msgr2=0x7fbabc15d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc151220 0x7fbabc15d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 -- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbabc0cf190 msgr2=0x7fbabc153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:18.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbabc0cf190 0x7fbabc153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:18.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 -- 172.21.15.158:0/848813046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac0094a0 con 0x7fbabc06d950 2023-10-16T09:44:18.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbab9d9b700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbabc0cf190 0x7fbabc153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:18.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaba59c700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbabc06d950 0x7fbabc148970 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00ebe0 tx=0x7fbaac00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaaaffd700 1 -- 172.21.15.158:0/848813046 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00c850 con 0x7fbabc06d950 2023-10-16T09:44:18.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaaaffd700 1 -- 172.21.15.158:0/848813046 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac00c9b0 con 0x7fbabc06d950 2023-10-16T09:44:18.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbaaaffd700 1 -- 172.21.15.158:0/848813046 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00cd40 con 0x7fbabc06d950 2023-10-16T09:44:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab00094a0 con 0x7fbabc06d950 2023-10-16T09:44:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.940+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc0b4390 con 0x7fbabc06d950 2023-10-16T09:44:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.941+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc0681b0 con 0x7fbabc06d950 2023-10-16T09:44:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.941+0000 7fbaaaffd700 1 -- 172.21.15.158:0/848813046 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbaac013470 con 0x7fbabc06d950 2023-10-16T09:44:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.942+0000 7fbaaaffd700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fba9c06af00 0x7fba9c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.942+0000 7fbaaaffd700 1 -- 172.21.15.158:0/848813046 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fbaac089a00 con 0x7fbabc06d950 2023-10-16T09:44:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.942+0000 7fbab9d9b700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fba9c06af00 0x7fba9c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.942+0000 7fbab9d9b700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fba9c06af00 0x7fba9c06d3c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbab0000c00 tx=0x7fbab0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:18.946+0000 7fbaaaffd700 1 -- 172.21.15.158:0/848813046 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbaac054730 con 0x7fbabc06d950 2023-10-16T09:44:19.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.139+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbabc0723f0 con 0x7fbabc06d950 2023-10-16T09:44:19.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.140+0000 7fbaaaffd700 1 -- 172.21.15.158:0/848813046 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbaac00cea0 con 0x7fbabc06d950 2023-10-16T09:44:19.142 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:19.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fba9c06af00 msgr2=0x7fba9c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fba9c06af00 0x7fba9c06d3c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbab0000c00 tx=0x7fbab0008030 comp rx=0 tx=0).stop 2023-10-16T09:44:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbabc06d950 msgr2=0x7fbabc148970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbabc06d950 0x7fbabc148970 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00ebe0 tx=0x7fbaac00eef0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 shutdown_connections 2023-10-16T09:44:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fba9c06af00 0x7fba9c06d3c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbabc151220 0x7fbabc15d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbabc06d950 0x7fbabc148970 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 --2- 172.21.15.158:0/848813046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbabc0cf190 0x7fbabc153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 >> 172.21.15.158:0/848813046 conn(0x7fbabc0cc5b0 msgr2=0x7fbabc0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 shutdown_connections 2023-10-16T09:44:19.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.142+0000 7fbac0cdf700 1 -- 172.21.15.158:0/848813046 wait complete. 2023-10-16T09:44:19.156 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:44:19.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.276+0000 7fabb9a29700 1 Processor -- start 2023-10-16T09:44:19.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.276+0000 7fabb9a29700 1 -- start start 2023-10-16T09:44:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb9a29700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabb415da60 0x7fabb40cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb9a29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 0x7fabb40d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb9a29700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb40d1db0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb9a29700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb40cfd60 con 0x7fabb415da60 2023-10-16T09:44:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb9a29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb40d1760 con 0x7fabb40b5870 2023-10-16T09:44:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabb415da60 0x7fabb40cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb27fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fabb40ce5b0 0x7fabb40b5160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47916/0 (socket says 172.21.15.158:47916) 2023-10-16T09:44:19.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb27fc700 1 -- 172.21.15.158:0/3097596564 learned_addr learned my addr 172.21.15.158:0/3097596564 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb2ffd700 1 -- 172.21.15.158:0/3097596564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 msgr2=0x7fabb40d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb37fe700 1 --2- 172.21.15.158:0/3097596564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 0x7fabb40d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb2ffd700 1 --2- 172.21.15.158:0/3097596564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 0x7fabb40d2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb2ffd700 1 -- 172.21.15.158:0/3097596564 >> v1:172.21.15.158:6789/0 conn(0x7fabb40ce5b0 legacy=0x7fabb40b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb1ffb700 1 -- 172.21.15.158:0/3097596564 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802807875 0 0) 0x7fabb40d1db0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb2ffd700 1 -- 172.21.15.158:0/3097596564 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40d1a40 con 0x7fabb415da60 2023-10-16T09:44:19.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb2ffd700 1 --2- 172.21.15.158:0/3097596564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb415da60 0x7fabb40cf430 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faba4009820 tx=0x7faba4009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf81646b31cd2d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:19.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb1ffb700 1 -- 172.21.15.158:0/3097596564 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4005e30 con 0x7fabb415da60 2023-10-16T09:44:19.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb1ffb700 1 -- 172.21.15.158:0/3097596564 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba4006030 con 0x7fabb415da60 2023-10-16T09:44:19.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.277+0000 7fabb1ffb700 1 -- 172.21.15.158:0/3097596564 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba40063e0 con 0x7fabb415da60 2023-10-16T09:44:19.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 -- 172.21.15.158:0/3097596564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb415da60 msgr2=0x7fabb40cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3097596564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb415da60 0x7fabb40cf430 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faba4009820 tx=0x7faba4009b30 comp rx=0 tx=0).stop 2023-10-16T09:44:19.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 -- 172.21.15.158:0/3097596564 shutdown_connections 2023-10-16T09:44:19.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3097596564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb415da60 0x7fabb40cf430 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3097596564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 0x7fabb40d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 -- 172.21.15.158:0/3097596564 >> 172.21.15.158:0/3097596564 conn(0x7fabb409c9b0 msgr2=0x7fabb40d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 -- 172.21.15.158:0/3097596564 shutdown_connections 2023-10-16T09:44:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.278+0000 7fabb9a29700 1 -- 172.21.15.158:0/3097596564 wait complete. 2023-10-16T09:44:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.279+0000 7fabb9a29700 1 Processor -- start 2023-10-16T09:44:19.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.279+0000 7fabb9a29700 1 -- start start 2023-10-16T09:44:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb9a29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 0x7fabb415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb9a29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb40ce5b0 0x7fabb40b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb9a29700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabb415da60 0x7fabb40b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb9a29700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb41f3db0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb9a29700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb40b6230 con 0x7fabb415da60 2023-10-16T09:44:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb9a29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb415b1a0 con 0x7fabb40b5870 2023-10-16T09:44:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb40ce5b0 0x7fabb40b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb40ce5b0 0x7fabb40b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47746/0 (socket says 172.21.15.158:47746) 2023-10-16T09:44:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 -- 172.21.15.158:0/3183023044 learned_addr learned my addr 172.21.15.158:0/3183023044 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:19.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 -- 172.21.15.158:0/3183023044 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 msgr2=0x7fabb415ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 0x7fabb415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 -- 172.21.15.158:0/3183023044 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabb415da60 msgr2=0x7fabb40b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabb415da60 0x7fabb40b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.280+0000 7fabb27fc700 1 -- 172.21.15.158:0/3183023044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba4009430 con 0x7fabb40ce5b0 2023-10-16T09:44:19.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.281+0000 7fabb27fc700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb40ce5b0 0x7fabb40b0aa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faba800c540 tx=0x7faba800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:19.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.281+0000 7fab9bfff700 1 -- 172.21.15.158:0/3183023044 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba8004c80 con 0x7fabb40ce5b0 2023-10-16T09:44:19.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.281+0000 7fab9bfff700 1 -- 172.21.15.158:0/3183023044 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba8004de0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.281+0000 7fab9bfff700 1 -- 172.21.15.158:0/3183023044 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba8007310 con 0x7fabb40ce5b0 2023-10-16T09:44:19.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.281+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb41f3410 con 0x7fabb40ce5b0 2023-10-16T09:44:19.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.281+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb41f39d0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.282+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb41f3ef0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.282+0000 7fab9bfff700 1 -- 172.21.15.158:0/3183023044 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faba8007470 con 0x7fabb40ce5b0 2023-10-16T09:44:19.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.282+0000 7fab9bfff700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7faba006af00 0x7faba006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.283+0000 7fab9bfff700 1 -- 172.21.15.158:0/3183023044 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7faba80896e0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.283+0000 7fabb2ffd700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7faba006af00 0x7faba006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.283+0000 7fabb2ffd700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7faba006af00 0x7faba006d3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faba4013040 tx=0x7faba4010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:19.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.287+0000 7fab9bfff700 1 -- 172.21.15.158:0/3183023044 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fabb41f3ef0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.493+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fabb41f3ef0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.493+0000 7fab9bfff700 1 -- 172.21.15.158:0/3183023044 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fabb41f3ef0 con 0x7fabb40ce5b0 2023-10-16T09:44:19.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.495+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7faba006af00 msgr2=0x7faba006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.495+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7faba006af00 0x7faba006d3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faba4013040 tx=0x7faba4010000 comp rx=0 tx=0).stop 2023-10-16T09:44:19.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.495+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb40ce5b0 msgr2=0x7fabb40b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.495+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb40ce5b0 0x7fabb40b0aa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faba800c540 tx=0x7faba800c900 comp rx=0 tx=0).stop 2023-10-16T09:44:19.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 shutdown_connections 2023-10-16T09:44:19.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabb415da60 0x7fabb40b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7faba006af00 0x7faba006d3c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabb40b5870 0x7fabb415ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 --2- 172.21.15.158:0/3183023044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb40ce5b0 0x7fabb40b0aa0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 >> 172.21.15.158:0/3183023044 conn(0x7fabb409c9b0 msgr2=0x7fabb40d2240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:19.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 shutdown_connections 2023-10-16T09:44:19.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.496+0000 7fabb9a29700 1 -- 172.21.15.158:0/3183023044 wait complete. 2023-10-16T09:44:19.500 INFO:teuthology.orchestra.run.smithi158.stderr:listed 0 entries 2023-10-16T09:44:19.509 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-16T09:44:19.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:19.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-16T09:44:19.550 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:44:19.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:19.551 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-16T09:44:19.621 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:19.622 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:19.639 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:19.640 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:19.696 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:19.697 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:19.753 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.912+0000 7fbde6939700 1 Processor -- start 2023-10-16T09:44:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.912+0000 7fbde6939700 1 -- start start 2023-10-16T09:44:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbde6939700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbde00cc5d0 0x7fbde006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbde6939700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde006f340 0x7fbde006f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbde6939700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde006e770 con 0x7fbde015dfb0 2023-10-16T09:44:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbde6939700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde00cf270 con 0x7fbde00cc5d0 2023-10-16T09:44:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbde6939700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde006fd10 con 0x7fbde006f340 2023-10-16T09:44:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbde00cc5d0 0x7fbde006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbde00cc5d0 0x7fbde006e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47752/0 (socket says 172.21.15.158:47752) 2023-10-16T09:44:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 -- 172.21.15.158:0/3781168660 learned_addr learned my addr 172.21.15.158:0/3781168660 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbde4ed6700 1 --2- 172.21.15.158:0/3781168660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde006f340 0x7fbde006f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 -- 172.21.15.158:0/3781168660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde006f340 msgr2=0x7fbde006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 --2- 172.21.15.158:0/3781168660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde006f340 0x7fbde006f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 -- 172.21.15.158:0/3781168660 >> v1:172.21.15.172:6789/0 conn(0x7fbde015dfb0 legacy=0x7fbde00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 -- 172.21.15.158:0/3781168660 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde00cbe00 con 0x7fbde00cc5d0 2023-10-16T09:44:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddffff700 1 --2- 172.21.15.158:0/3781168660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde00cc5d0 0x7fbde006e230 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000bb90 tx=0x7fbdd000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a394fa6e854de6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddeffd700 1 -- 172.21.15.158:0/3781168660 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd000dd50 con 0x7fbde00cc5d0 2023-10-16T09:44:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddeffd700 1 -- 172.21.15.158:0/3781168660 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd0010470 con 0x7fbde00cc5d0 2023-10-16T09:44:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.913+0000 7fbddeffd700 1 -- 172.21.15.158:0/3781168660 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0010800 con 0x7fbde00cc5d0 2023-10-16T09:44:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 -- 172.21.15.158:0/3781168660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde00cc5d0 msgr2=0x7fbde006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 --2- 172.21.15.158:0/3781168660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde00cc5d0 0x7fbde006e230 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000bb90 tx=0x7fbdd000bea0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 -- 172.21.15.158:0/3781168660 shutdown_connections 2023-10-16T09:44:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 --2- 172.21.15.158:0/3781168660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde006f340 0x7fbde006f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 --2- 172.21.15.158:0/3781168660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde00cc5d0 0x7fbde006e230 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 -- 172.21.15.158:0/3781168660 >> 172.21.15.158:0/3781168660 conn(0x7fbde00d02b0 msgr2=0x7fbde00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 -- 172.21.15.158:0/3781168660 shutdown_connections 2023-10-16T09:44:19.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.914+0000 7fbde6939700 1 -- 172.21.15.158:0/3781168660 wait complete. 2023-10-16T09:44:19.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.915+0000 7fbde6939700 1 Processor -- start 2023-10-16T09:44:19.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.915+0000 7fbde6939700 1 -- start start 2023-10-16T09:44:19.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbde6939700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde006f340 0x7fbde00b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbde6939700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde00cc5d0 0x7fbde00c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbde6939700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde015dfb0 0x7fbde00b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbde6939700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde0153240 con 0x7fbde006f340 2023-10-16T09:44:19.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbde6939700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00b5140 con 0x7fbde015dfb0 2023-10-16T09:44:19.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbde6939700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00b42d0 con 0x7fbde00cc5d0 2023-10-16T09:44:19.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbddffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde006f340 0x7fbde00b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbddffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde006f340 0x7fbde00b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47758/0 (socket says 172.21.15.158:47758) 2023-10-16T09:44:19.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbddffff700 1 -- 172.21.15.158:0/647259102 learned_addr learned my addr 172.21.15.158:0/647259102 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:19.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbde4ed6700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde015dfb0 0x7fbde00b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.916+0000 7fbddf7fe700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde00cc5d0 0x7fbde00c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddffff700 1 -- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde00cc5d0 msgr2=0x7fbde00c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddffff700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde00cc5d0 0x7fbde00c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddffff700 1 -- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde015dfb0 msgr2=0x7fbde00b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddffff700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde015dfb0 0x7fbde00b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddffff700 1 -- 172.21.15.158:0/647259102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd000b7f0 con 0x7fbde006f340 2023-10-16T09:44:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddf7fe700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde00cc5d0 0x7fbde00c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddffff700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde006f340 0x7fbde00b3d90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbde0074ec0 tx=0x7fbdd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddcff9700 1 -- 172.21.15.158:0/647259102 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd001f070 con 0x7fbde006f340 2023-10-16T09:44:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddcff9700 1 -- 172.21.15.158:0/647259102 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd00038b0 con 0x7fbde006f340 2023-10-16T09:44:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde00c6500 con 0x7fbde006f340 2023-10-16T09:44:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbddcff9700 1 -- 172.21.15.158:0/647259102 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0003c40 con 0x7fbde006f340 2023-10-16T09:44:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.917+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde00b4810 con 0x7fbde006f340 2023-10-16T09:44:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.918+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde00b5420 con 0x7fbde006f340 2023-10-16T09:44:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.918+0000 7fbddcff9700 1 -- 172.21.15.158:0/647259102 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbdd002d430 con 0x7fbde006f340 2023-10-16T09:44:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.919+0000 7fbddcff9700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.919+0000 7fbddcff9700 1 -- 172.21.15.158:0/647259102 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fbdd008d3d0 con 0x7fbde006f340 2023-10-16T09:44:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.919+0000 7fbddf7fe700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:19.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.919+0000 7fbddf7fe700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbde006e420 tx=0x7fbdd4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:19.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:19.923+0000 7fbddcff9700 1 -- 172.21.15.158:0/647259102 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbdd0058100 con 0x7fbde006f340 2023-10-16T09:44:20.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.118+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbde00681b0 con 0x7fbde006f340 2023-10-16T09:44:20.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.119+0000 7fbddcff9700 1 -- 172.21.15.158:0/647259102 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7fbdd005b720 con 0x7fbde006f340 2023-10-16T09:44:20.121 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:20.122 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:20.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbdcc06af00 msgr2=0x7fbdcc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbde006e420 tx=0x7fbdd4009040 comp rx=0 tx=0).stop 2023-10-16T09:44:20.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde006f340 msgr2=0x7fbde00b3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde006f340 0x7fbde00b3d90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbde0074ec0 tx=0x7fbdd0004750 comp rx=0 tx=0).stop 2023-10-16T09:44:20.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 shutdown_connections 2023-10-16T09:44:20.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde015dfb0 0x7fbde00b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde006f340 0x7fbde00b3d90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 --2- 172.21.15.158:0/647259102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbde00cc5d0 0x7fbde00c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 >> 172.21.15.158:0/647259102 conn(0x7fbde00d02b0 msgr2=0x7fbde015ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:20.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 shutdown_connections 2023-10-16T09:44:20.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.122+0000 7fbde6939700 1 -- 172.21.15.158:0/647259102 wait complete. 2023-10-16T09:44:20.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:20.136 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:20.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.258+0000 7ffaf7679700 1 Processor -- start 2023-10-16T09:44:20.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.259+0000 7ffaf7679700 1 -- start start 2023-10-16T09:44:20.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.259+0000 7ffaf7679700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffaf0141920 0x7ffaf006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.259+0000 7ffaf7679700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffaf00cde10 0x7ffaf006da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.259+0000 7ffaf7679700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf006c210 con 0x7ffaf0144840 2023-10-16T09:44:20.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.259+0000 7ffaf7679700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf0141d00 con 0x7ffaf0141920 2023-10-16T09:44:20.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.259+0000 7ffaf7679700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf006b970 con 0x7ffaf00cde10 2023-10-16T09:44:20.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.259+0000 7ffaf5415700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffaf0141920 0x7ffaf006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5415700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffaf0141920 0x7ffaf006b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47766/0 (socket says 172.21.15.158:47766) 2023-10-16T09:44:20.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5415700 1 -- 172.21.15.158:0/811957194 learned_addr learned my addr 172.21.15.158:0/811957194 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:20.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5c16700 1 --2- 172.21.15.158:0/811957194 >> v2:172.21.15.172:3300/0 conn(0x7ffaf00cde10 0x7ffaf006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5415700 1 -- 172.21.15.158:0/811957194 >> v2:172.21.15.172:3300/0 conn(0x7ffaf00cde10 msgr2=0x7ffaf006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:20.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5415700 1 --2- 172.21.15.158:0/811957194 >> v2:172.21.15.172:3300/0 conn(0x7ffaf00cde10 0x7ffaf006da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5415700 1 -- 172.21.15.158:0/811957194 >> v1:172.21.15.172:6789/0 conn(0x7ffaf0144840 legacy=0x7ffaf00cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5415700 1 -- 172.21.15.158:0/811957194 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf006bc50 con 0x7ffaf0141920 2023-10-16T09:44:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffaf5415700 1 --2- 172.21.15.158:0/811957194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0141920 0x7ffaf006b2e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0066630 tx=0x7ffae0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67cd35a603fd429 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffae7fff700 1 -- 172.21.15.158:0/811957194 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0004230 con 0x7ffaf0141920 2023-10-16T09:44:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffae7fff700 1 -- 172.21.15.158:0/811957194 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0004390 con 0x7ffaf0141920 2023-10-16T09:44:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.260+0000 7ffae7fff700 1 -- 172.21.15.158:0/811957194 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae00036a0 con 0x7ffaf0141920 2023-10-16T09:44:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 -- 172.21.15.158:0/811957194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0141920 msgr2=0x7ffaf006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 --2- 172.21.15.158:0/811957194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0141920 0x7ffaf006b2e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0066630 tx=0x7ffae0009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 -- 172.21.15.158:0/811957194 shutdown_connections 2023-10-16T09:44:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 --2- 172.21.15.158:0/811957194 >> v2:172.21.15.172:3300/0 conn(0x7ffaf00cde10 0x7ffaf006da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 --2- 172.21.15.158:0/811957194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0141920 0x7ffaf006b2e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 -- 172.21.15.158:0/811957194 >> 172.21.15.158:0/811957194 conn(0x7ffaf00cc5b0 msgr2=0x7ffaf00cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:20.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 -- 172.21.15.158:0/811957194 shutdown_connections 2023-10-16T09:44:20.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.261+0000 7ffaf7679700 1 -- 172.21.15.158:0/811957194 wait complete. 2023-10-16T09:44:20.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.262+0000 7ffaf7679700 1 Processor -- start 2023-10-16T09:44:20.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.262+0000 7ffaf7679700 1 -- start start 2023-10-16T09:44:20.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf7679700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffaf00cde10 0x7ffaf0136730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf7679700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffaf0141920 0x7ffaf00a8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf7679700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0144840 0x7ffaf0142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf7679700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00a8f50 con 0x7ffaf0144840 2023-10-16T09:44:20.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf7679700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf0133f80 con 0x7ffaf00cde10 2023-10-16T09:44:20.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf7679700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf0134260 con 0x7ffaf0141920 2023-10-16T09:44:20.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0144840 0x7ffaf0142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0144840 0x7ffaf0142220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47774/0 (socket says 172.21.15.158:47774) 2023-10-16T09:44:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 -- 172.21.15.158:0/3489428751 learned_addr learned my addr 172.21.15.158:0/3489428751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5415700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffaf00cde10 0x7ffaf0136730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf4c14700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffaf0141920 0x7ffaf00a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 -- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffaf0141920 msgr2=0x7ffaf00a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffaf0141920 0x7ffaf00a8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 -- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffaf00cde10 msgr2=0x7ffaf0136730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffaf00cde10 0x7ffaf0136730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 -- 172.21.15.158:0/3489428751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0009550 con 0x7ffaf0144840 2023-10-16T09:44:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf5c16700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0144840 0x7ffaf0142220 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffae800cd20 tx=0x7ffae800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffae5ffb700 1 -- 172.21.15.158:0/3489428751 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae800ce70 con 0x7ffaf0144840 2023-10-16T09:44:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffae5ffb700 1 -- 172.21.15.158:0/3489428751 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8003a20 con 0x7ffaf0144840 2023-10-16T09:44:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf0134540 con 0x7ffaf0144840 2023-10-16T09:44:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.263+0000 7ffae5ffb700 1 -- 172.21.15.158:0/3489428751 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8003db0 con 0x7ffaf0144840 2023-10-16T09:44:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.264+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf00c7c50 con 0x7ffaf0144840 2023-10-16T09:44:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.264+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf0134a60 con 0x7ffaf0144840 2023-10-16T09:44:20.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.265+0000 7ffae5ffb700 1 -- 172.21.15.158:0/3489428751 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ffae8003f10 con 0x7ffaf0144840 2023-10-16T09:44:20.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.265+0000 7ffae5ffb700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ffad406af00 0x7ffad406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.265+0000 7ffae5ffb700 1 -- 172.21.15.158:0/3489428751 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7ffae8012070 con 0x7ffaf0144840 2023-10-16T09:44:20.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.265+0000 7ffaf5415700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ffad406af00 0x7ffad406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.266+0000 7ffaf5415700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ffad406af00 0x7ffad406d3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009130 tx=0x7ffae0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.269+0000 7ffae5ffb700 1 -- 172.21.15.158:0/3489428751 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffae8054f10 con 0x7ffaf0144840 2023-10-16T09:44:20.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.460+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaf00681b0 con 0x7ffaf0144840 2023-10-16T09:44:20.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.460+0000 7ffae5ffb700 1 -- 172.21.15.158:0/3489428751 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffae8058530 con 0x7ffaf0144840 2023-10-16T09:44:20.462 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:20.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.462+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ffad406af00 msgr2=0x7ffad406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.462+0000 7ffaf7679700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ffad406af00 0x7ffad406d3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009130 tx=0x7ffae0008030 comp rx=0 tx=0).stop 2023-10-16T09:44:20.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.462+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0144840 msgr2=0x7ffaf0142220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.462+0000 7ffaf7679700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0144840 0x7ffaf0142220 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffae800cd20 tx=0x7ffae800a570 comp rx=0 tx=0).stop 2023-10-16T09:44:20.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.462+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 shutdown_connections 2023-10-16T09:44:20.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.463+0000 7ffaf7679700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffaf0144840 0x7ffaf0142220 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.463+0000 7ffaf7679700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffaf00cde10 0x7ffaf0136730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.463+0000 7ffaf7679700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ffad406af00 0x7ffad406d3c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.463+0000 7ffaf7679700 1 --2- 172.21.15.158:0/3489428751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffaf0141920 0x7ffaf00a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.463+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 >> 172.21.15.158:0/3489428751 conn(0x7ffaf00cc5b0 msgr2=0x7ffaf0072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:20.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.463+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 shutdown_connections 2023-10-16T09:44:20.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.463+0000 7ffaf7679700 1 -- 172.21.15.158:0/3489428751 wait complete. 2023-10-16T09:44:20.476 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:20.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.599+0000 7f7ca4647700 1 Processor -- start 2023-10-16T09:44:20.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.599+0000 7f7ca4647700 1 -- start start 2023-10-16T09:44:20.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.600+0000 7f7ca4647700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7c9c151220 0x7f7c9c0665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.600+0000 7f7ca4647700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7c9c151600 0x7f7c9c0c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.600+0000 7f7ca4647700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9c0cf930 con 0x7f7c9c0cf190 2023-10-16T09:44:20.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.600+0000 7f7ca4647700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9c0cf650 con 0x7f7c9c151220 2023-10-16T09:44:20.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.600+0000 7f7ca4647700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9c0c51b0 con 0x7f7c9c151600 2023-10-16T09:44:20.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7c9c151220 0x7f7c9c0665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7c9c151220 0x7f7c9c0665a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47778/0 (socket says 172.21.15.158:47778) 2023-10-16T09:44:20.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 -- 172.21.15.158:0/932162935 learned_addr learned my addr 172.21.15.158:0/932162935 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:20.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 -- 172.21.15.158:0/932162935 >> v2:172.21.15.172:3300/0 conn(0x7f7c9c151600 msgr2=0x7f7c9c0c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:20.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 --2- 172.21.15.158:0/932162935 >> v2:172.21.15.172:3300/0 conn(0x7f7c9c151600 0x7f7c9c0c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 -- 172.21.15.158:0/932162935 >> v1:172.21.15.172:6789/0 conn(0x7f7c9c0cf190 legacy=0x7f7c9c06d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:20.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 -- 172.21.15.158:0/932162935 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c0c5490 con 0x7f7c9c151220 2023-10-16T09:44:20.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca23e3700 1 --2- 172.21.15.158:0/932162935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c0665a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7c980098f0 tx=0x7f7c98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871ea904bf5c7ef1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca13e1700 1 -- 172.21.15.158:0/932162935 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98004230 con 0x7f7c9c151220 2023-10-16T09:44:20.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca13e1700 1 -- 172.21.15.158:0/932162935 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c98004390 con 0x7f7c9c151220 2023-10-16T09:44:20.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.601+0000 7f7ca13e1700 1 -- 172.21.15.158:0/932162935 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980036a0 con 0x7f7c9c151220 2023-10-16T09:44:20.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 -- 172.21.15.158:0/932162935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 msgr2=0x7f7c9c0665a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 --2- 172.21.15.158:0/932162935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c0665a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7c980098f0 tx=0x7f7c98009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:20.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 -- 172.21.15.158:0/932162935 shutdown_connections 2023-10-16T09:44:20.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 --2- 172.21.15.158:0/932162935 >> v2:172.21.15.172:3300/0 conn(0x7f7c9c151600 0x7f7c9c0c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 --2- 172.21.15.158:0/932162935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c0665a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 -- 172.21.15.158:0/932162935 >> 172.21.15.158:0/932162935 conn(0x7f7c9c0cc5b0 msgr2=0x7f7c9c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:20.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 -- 172.21.15.158:0/932162935 shutdown_connections 2023-10-16T09:44:20.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.602+0000 7f7ca4647700 1 -- 172.21.15.158:0/932162935 wait complete. 2023-10-16T09:44:20.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.603+0000 7f7ca4647700 1 Processor -- start 2023-10-16T09:44:20.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.603+0000 7f7ca4647700 1 -- start start 2023-10-16T09:44:20.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca4647700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c9c0cf190 0x7f7c9c14c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca4647700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c1f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca4647700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c9c151600 0x7f7c9c14a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca4647700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1f2d90 con 0x7f7c9c151220 2023-10-16T09:44:20.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca4647700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c14ae60 con 0x7f7c9c151600 2023-10-16T09:44:20.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca4647700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c14ab80 con 0x7f7c9c0cf190 2023-10-16T09:44:20.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c1f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c1f2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47786/0 (socket says 172.21.15.158:47786) 2023-10-16T09:44:20.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 -- 172.21.15.158:0/2851283128 learned_addr learned my addr 172.21.15.158:0/2851283128 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:20.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca2be4700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c9c151600 0x7f7c9c14a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca23e3700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c9c0cf190 0x7f7c9c14c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 -- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c9c0cf190 msgr2=0x7f7c9c14c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c9c0cf190 0x7f7c9c14c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 -- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c9c151600 msgr2=0x7f7c9c14a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c9c151600 0x7f7c9c14a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 -- 172.21.15.158:0/2851283128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98009550 con 0x7f7c9c151220 2023-10-16T09:44:20.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca2be4700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c9c151600 0x7f7c9c14a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:20.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca23e3700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c9c0cf190 0x7f7c9c14c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:20.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.604+0000 7f7ca1be2700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c1f2850 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00c4a0 tx=0x7f7c8c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.605+0000 7f7c92ffd700 1 -- 172.21.15.158:0/2851283128 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c0054e0 con 0x7f7c9c151220 2023-10-16T09:44:20.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.605+0000 7f7c92ffd700 1 -- 172.21.15.158:0/2851283128 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8c003680 con 0x7f7c9c151220 2023-10-16T09:44:20.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.605+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c1f6550 con 0x7f7c9c151220 2023-10-16T09:44:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.605+0000 7f7c92ffd700 1 -- 172.21.15.158:0/2851283128 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c003a10 con 0x7f7c9c151220 2023-10-16T09:44:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.605+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c1f6b10 con 0x7f7c9c151220 2023-10-16T09:44:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.605+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9c15c670 con 0x7f7c9c151220 2023-10-16T09:44:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.606+0000 7f7c92ffd700 1 -- 172.21.15.158:0/2851283128 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7c8c003b70 con 0x7f7c9c151220 2023-10-16T09:44:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.606+0000 7f7c92ffd700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7c8006af00 0x7f7c8006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.607+0000 7f7ca23e3700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7c8006af00 0x7f7c8006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.607+0000 7f7c92ffd700 1 -- 172.21.15.158:0/2851283128 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f7c8c088d20 con 0x7f7c9c151220 2023-10-16T09:44:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.607+0000 7f7ca23e3700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7c8006af00 0x7f7c8006d3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009c40 tx=0x7f7c98008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.611+0000 7f7c92ffd700 1 -- 172.21.15.158:0/2851283128 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7c8c053b00 con 0x7f7c9c151220 2023-10-16T09:44:20.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.802+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c9c0681b0 con 0x7f7c9c151220 2023-10-16T09:44:20.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.803+0000 7f7c92ffd700 1 -- 172.21.15.158:0/2851283128 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c8c0170a0 con 0x7f7c9c151220 2023-10-16T09:44:20.805 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:20.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7c8006af00 msgr2=0x7f7c8006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7c8006af00 0x7f7c8006d3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009c40 tx=0x7f7c98008030 comp rx=0 tx=0).stop 2023-10-16T09:44:20.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 msgr2=0x7f7c9c1f2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c1f2850 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00c4a0 tx=0x7f7c8c00c860 comp rx=0 tx=0).stop 2023-10-16T09:44:20.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 shutdown_connections 2023-10-16T09:44:20.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c9c151600 0x7f7c9c14a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7c8006af00 0x7f7c8006d3c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c9c0cf190 0x7f7c9c14c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 --2- 172.21.15.158:0/2851283128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c9c151220 0x7f7c9c1f2850 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 >> 172.21.15.158:0/2851283128 conn(0x7f7c9c0cc5b0 msgr2=0x7f7c9c15e050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:20.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 shutdown_connections 2023-10-16T09:44:20.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.805+0000 7f7ca4647700 1 -- 172.21.15.158:0/2851283128 wait complete. 2023-10-16T09:44:20.818 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.941+0000 7f6c921b9700 1 Processor -- start 2023-10-16T09:44:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.941+0000 7f6c921b9700 1 -- start start 2023-10-16T09:44:20.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c921b9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c8c143790 0x7f6c8c06bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c921b9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c0c50a0 0x7f6c8c0c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c921b9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c06aa70 con 0x7f6c8c06e540 2023-10-16T09:44:20.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c921b9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c143b70 con 0x7f6c8c143790 2023-10-16T09:44:20.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c921b9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0c5a60 con 0x7f6c8c0c50a0 2023-10-16T09:44:20.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c8c143790 0x7f6c8c06bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c8c143790 0x7f6c8c06bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47792/0 (socket says 172.21.15.158:47792) 2023-10-16T09:44:20.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2309402675 learned_addr learned my addr 172.21.15.158:0/2309402675 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:20.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8affd700 1 --2- 172.21.15.158:0/2309402675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c0c50a0 0x7f6c8c0c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2309402675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c0c50a0 msgr2=0x7f6c8c0c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:20.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 --2- 172.21.15.158:0/2309402675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c0c50a0 0x7f6c8c0c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2309402675 >> v1:172.21.15.172:6789/0 conn(0x7f6c8c06e540 legacy=0x7f6c8c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2309402675 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c06ad50 con 0x7f6c8c143790 2023-10-16T09:44:20.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8b7fe700 1 --2- 172.21.15.158:0/2309402675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c143790 0x7f6c8c06bb10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0098f0 tx=0x7f6c7c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a46dfa33d0740d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8a7fc700 1 -- 172.21.15.158:0/2309402675 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00b8a0 con 0x7f6c8c143790 2023-10-16T09:44:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8a7fc700 1 -- 172.21.15.158:0/2309402675 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c00ba00 con 0x7f6c8c143790 2023-10-16T09:44:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.942+0000 7f6c8a7fc700 1 -- 172.21.15.158:0/2309402675 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00bd90 con 0x7f6c8c143790 2023-10-16T09:44:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 -- 172.21.15.158:0/2309402675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c143790 msgr2=0x7f6c8c06bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2309402675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c143790 0x7f6c8c06bb10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0098f0 tx=0x7f6c7c009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 -- 172.21.15.158:0/2309402675 shutdown_connections 2023-10-16T09:44:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2309402675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c143790 0x7f6c8c06bb10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2309402675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c0c50a0 0x7f6c8c0c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 -- 172.21.15.158:0/2309402675 >> 172.21.15.158:0/2309402675 conn(0x7f6c8c0cc5b0 msgr2=0x7f6c8c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 -- 172.21.15.158:0/2309402675 shutdown_connections 2023-10-16T09:44:20.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.943+0000 7f6c921b9700 1 -- 172.21.15.158:0/2309402675 wait complete. 2023-10-16T09:44:20.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.944+0000 7f6c921b9700 1 Processor -- start 2023-10-16T09:44:20.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.944+0000 7f6c921b9700 1 -- start start 2023-10-16T09:44:20.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c921b9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c06e540 0x7f6c8c156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c921b9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c8c0c50a0 0x7f6c8c1425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c921b9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c143790 0x7f6c8c158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c921b9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0b56c0 con 0x7f6c8c06e540 2023-10-16T09:44:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c921b9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0b2d50 con 0x7f6c8c0c50a0 2023-10-16T09:44:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c921b9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0b5100 con 0x7f6c8c143790 2023-10-16T09:44:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c06e540 0x7f6c8c156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c06e540 0x7f6c8c156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47798/0 (socket says 172.21.15.158:47798) 2023-10-16T09:44:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2264802059 learned_addr learned my addr 172.21.15.158:0/2264802059 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8affd700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c8c0c50a0 0x7f6c8c1425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8bfff700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c143790 0x7f6c8c158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c143790 msgr2=0x7f6c8c158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c143790 0x7f6c8c158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c8c0c50a0 msgr2=0x7f6c8c1425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:20.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c8c0c50a0 0x7f6c8c1425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:20.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/2264802059 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c009550 con 0x7f6c8c06e540 2023-10-16T09:44:20.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8affd700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c8c0c50a0 0x7f6c8c1425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:20.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.945+0000 7f6c8b7fe700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c06e540 0x7f6c8c156a50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c009c40 tx=0x7f6c7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.946+0000 7f6c73fff700 1 -- 172.21.15.158:0/2264802059 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c004400 con 0x7f6c8c06e540 2023-10-16T09:44:20.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.946+0000 7f6c73fff700 1 -- 172.21.15.158:0/2264802059 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c01f070 con 0x7f6c8c06e540 2023-10-16T09:44:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.946+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c0b53e0 con 0x7f6c8c06e540 2023-10-16T09:44:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.946+0000 7f6c73fff700 1 -- 172.21.15.158:0/2264802059 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c0038a0 con 0x7f6c8c06e540 2023-10-16T09:44:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.946+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c156f90 con 0x7f6c8c06e540 2023-10-16T09:44:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.947+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c0b27d0 con 0x7f6c8c06e540 2023-10-16T09:44:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.947+0000 7f6c73fff700 1 -- 172.21.15.158:0/2264802059 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6c7c021030 con 0x7f6c8c06e540 2023-10-16T09:44:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.947+0000 7f6c73fff700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c7806af00 0x7f6c7806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.948+0000 7f6c8affd700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c7806af00 0x7f6c7806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.948+0000 7f6c73fff700 1 -- 172.21.15.158:0/2264802059 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f6c7c08af70 con 0x7f6c8c06e540 2023-10-16T09:44:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.948+0000 7f6c8affd700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c7806af00 0x7f6c7806d3c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c1f3e90 tx=0x7f6c80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:20.952+0000 7f6c73fff700 1 -- 172.21.15.158:0/2264802059 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c7c0bdfe0 con 0x7f6c8c06e540 2023-10-16T09:44:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.145+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c8c0681b0 con 0x7f6c8c06e540 2023-10-16T09:44:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.146+0000 7f6c73fff700 1 -- 172.21.15.158:0/2264802059 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c8c0b27d0 con 0x7f6c8c06e540 2023-10-16T09:44:21.148 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c7806af00 msgr2=0x7f6c7806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c7806af00 0x7f6c7806d3c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c1f3e90 tx=0x7f6c80008040 comp rx=0 tx=0).stop 2023-10-16T09:44:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c06e540 msgr2=0x7f6c8c156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c06e540 0x7f6c8c156a50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c009c40 tx=0x7f6c7c004750 comp rx=0 tx=0).stop 2023-10-16T09:44:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 shutdown_connections 2023-10-16T09:44:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c7806af00 0x7f6c7806d3c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c8c143790 0x7f6c8c158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c06e540 0x7f6c8c156a50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 --2- 172.21.15.158:0/2264802059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c8c0c50a0 0x7f6c8c1425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 >> 172.21.15.158:0/2264802059 conn(0x7f6c8c0cc5b0 msgr2=0x7f6c8c0c8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:21.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 shutdown_connections 2023-10-16T09:44:21.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.148+0000 7f6c921b9700 1 -- 172.21.15.158:0/2264802059 wait complete. 2023-10-16T09:44:21.161 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:21.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.284+0000 7fdc25b45700 1 Processor -- start 2023-10-16T09:44:21.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.284+0000 7fdc25b45700 1 -- start start 2023-10-16T09:44:21.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.285+0000 7fdc25b45700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 0x7fdc200b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.285+0000 7fdc25b45700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdc200b3460 0x7fdc200b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.285+0000 7fdc25b45700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc200b4420 con 0x7fdc2015b6b0 2023-10-16T09:44:21.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.285+0000 7fdc25b45700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2015ba90 con 0x7fdc200b3460 2023-10-16T09:44:21.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.285+0000 7fdc25b45700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc200b38f0 con 0x7fdc200d0c60 2023-10-16T09:44:21.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.285+0000 7fdc1f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdc2015b6b0 0x7fdc200cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:47982/0 (socket says 172.21.15.158:47982) 2023-10-16T09:44:21.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.285+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/1884958236 learned_addr learned my addr 172.21.15.158:0/1884958236 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:21.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1ffff700 1 --2- 172.21.15.158:0/1884958236 >> v2:172.21.15.172:3300/0 conn(0x7fdc200b3460 0x7fdc200b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1effd700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 0x7fdc200b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188930730 0 0) 0x7fdc200b4420 con 0x7fdc2015b6b0 2023-10-16T09:44:21.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc080035b0 con 0x7fdc2015b6b0 2023-10-16T09:44:21.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1849200373 0 0) 0x7fdc080035b0 con 0x7fdc2015b6b0 2023-10-16T09:44:21.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 msgr2=0x7fdc200b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:21.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 0x7fdc200b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 >> v2:172.21.15.172:3300/0 conn(0x7fdc200b3460 msgr2=0x7fdc200b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:21.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 --2- 172.21.15.158:0/1884958236 >> v2:172.21.15.172:3300/0 conn(0x7fdc200b3460 0x7fdc200b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc200cbc20 con 0x7fdc2015b6b0 2023-10-16T09:44:21.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fdc14001670 con 0x7fdc2015b6b0 2023-10-16T09:44:21.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 >> v1:172.21.15.158:6789/0 conn(0x7fdc2015b6b0 legacy=0x7fdc200cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 0x7fdc08004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc08004d90 0x7fdc08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc200b38f0 con 0x7fdc08003a00 2023-10-16T09:44:21.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0800ba10 con 0x7fdc08004d90 2023-10-16T09:44:21.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0800bb50 con 0x7fdc200d0c60 2023-10-16T09:44:21.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1f7fe700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1effd700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 0x7fdc08004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1ffff700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc08004d90 0x7fdc08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 msgr2=0x7fdc08004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:21.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1f7fe700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200d0c60 0x7fdc08004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc08004d90 msgr2=0x7fdc08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:21.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1f7fe700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc08004d90 0x7fdc08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.286+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/1884958236 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc080035b0 con 0x7fdc08003a00 2023-10-16T09:44:21.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc1f7fe700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc08003e20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400c320 tx=0x7fdc08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf0b05fe7d677dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:21.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1400ce60 con 0x7fdc08003a00 2023-10-16T09:44:21.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14005960 con 0x7fdc08003a00 2023-10-16T09:44:21.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc1e7fc700 1 -- 172.21.15.158:0/1884958236 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14005cf0 con 0x7fdc08003a00 2023-10-16T09:44:21.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc25b45700 1 -- 172.21.15.158:0/1884958236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 msgr2=0x7fdc08003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc25b45700 1 --2- 172.21.15.158:0/1884958236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc08003e20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400c320 tx=0x7fdc08005880 comp rx=0 tx=0).stop 2023-10-16T09:44:21.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc1ffff700 1 -- 172.21.15.158:0/1884958236 reap_dead start 2023-10-16T09:44:21.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc25b45700 1 -- 172.21.15.158:0/1884958236 shutdown_connections 2023-10-16T09:44:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc25b45700 1 -- 172.21.15.158:0/1884958236 >> 172.21.15.158:0/1884958236 conn(0x7fdc200cdf20 msgr2=0x7fdc200ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc25b45700 1 -- 172.21.15.158:0/1884958236 shutdown_connections 2023-10-16T09:44:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.287+0000 7fdc25b45700 1 -- 172.21.15.158:0/1884958236 wait complete. 2023-10-16T09:44:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.289+0000 7fdc25b45700 1 Processor -- start 2023-10-16T09:44:21.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.289+0000 7fdc25b45700 1 -- start start 2023-10-16T09:44:21.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc25b45700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc201f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc25b45700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200c95c0 0x7fdc200b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc25b45700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc200cab60 0x7fdc200cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc25b45700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc200b6e50 con 0x7fdc08003a00 2023-10-16T09:44:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc25b45700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201f4ca0 con 0x7fdc200cab60 2023-10-16T09:44:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc25b45700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc200c9ac0 con 0x7fdc200c95c0 2023-10-16T09:44:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc201f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc201f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47818/0 (socket says 172.21.15.158:47818) 2023-10-16T09:44:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/3657872355 learned_addr learned my addr 172.21.15.158:0/3657872355 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1ffff700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc200cab60 0x7fdc200cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200c95c0 msgr2=0x7fdc200b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1effd700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200c95c0 0x7fdc200b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200c95c0 0x7fdc200b54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc200cab60 msgr2=0x7fdc200cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc200cab60 0x7fdc200cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1f7fe700 1 -- 172.21.15.158:0/3657872355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc200c9da0 con 0x7fdc08003a00 2023-10-16T09:44:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.290+0000 7fdc1ffff700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc200cab60 0x7fdc200cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.291+0000 7fdc1f7fe700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc201f4760 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdc14008fd0 tx=0x7fdc14001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.291+0000 7fdc07fff700 1 -- 172.21.15.158:0/3657872355 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14001680 con 0x7fdc08003a00 2023-10-16T09:44:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.291+0000 7fdc07fff700 1 -- 172.21.15.158:0/3657872355 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14004a80 con 0x7fdc08003a00 2023-10-16T09:44:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.291+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc200ca080 con 0x7fdc08003a00 2023-10-16T09:44:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.291+0000 7fdc07fff700 1 -- 172.21.15.158:0/3657872355 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14004e30 con 0x7fdc08003a00 2023-10-16T09:44:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.291+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc200ca640 con 0x7fdc08003a00 2023-10-16T09:44:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.292+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc200681b0 con 0x7fdc08003a00 2023-10-16T09:44:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.292+0000 7fdc07fff700 1 -- 172.21.15.158:0/3657872355 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdc140070d0 con 0x7fdc08003a00 2023-10-16T09:44:21.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.292+0000 7fdc07fff700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc0c06af00 0x7fdc0c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.293+0000 7fdc07fff700 1 -- 172.21.15.158:0/3657872355 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fdc1408aec0 con 0x7fdc08003a00 2023-10-16T09:44:21.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.293+0000 7fdc1effd700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc0c06af00 0x7fdc0c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.293+0000 7fdc1effd700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc0c06af00 0x7fdc0c06d3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdc200b21a0 tx=0x7fdc10008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:21.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.297+0000 7fdc07fff700 1 -- 172.21.15.158:0/3657872355 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc14055bf0 con 0x7fdc08003a00 2023-10-16T09:44:21.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.488+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdc200d2050 con 0x7fdc08003a00 2023-10-16T09:44:21.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.489+0000 7fdc07fff700 1 -- 172.21.15.158:0/3657872355 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdc14007da0 con 0x7fdc08003a00 2023-10-16T09:44:21.491 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:21.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc0c06af00 msgr2=0x7fdc0c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc0c06af00 0x7fdc0c06d3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdc200b21a0 tx=0x7fdc10008040 comp rx=0 tx=0).stop 2023-10-16T09:44:21.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 msgr2=0x7fdc201f4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc201f4760 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdc14008fd0 tx=0x7fdc14001a00 comp rx=0 tx=0).stop 2023-10-16T09:44:21.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 shutdown_connections 2023-10-16T09:44:21.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc200cab60 0x7fdc200cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc08003a00 0x7fdc201f4760 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fdc0c06af00 0x7fdc0c06d3c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 --2- 172.21.15.158:0/3657872355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc200c95c0 0x7fdc200b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 >> 172.21.15.158:0/3657872355 conn(0x7fdc200cdf20 msgr2=0x7fdc2015e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:21.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 shutdown_connections 2023-10-16T09:44:21.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.491+0000 7fdc25b45700 1 -- 172.21.15.158:0/3657872355 wait complete. 2023-10-16T09:44:21.505 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:44:21.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.627+0000 7fa491910700 1 Processor -- start 2023-10-16T09:44:21.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.627+0000 7fa491910700 1 -- start start 2023-10-16T09:44:21.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa491910700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa48c15dab0 0x7fa48c06c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa491910700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa491910700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c161aa0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa491910700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48c1622c0 con 0x7fa48c15dab0 2023-10-16T09:44:21.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa491910700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c161d00 con 0x7fa48c06d640 2023-10-16T09:44:21.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa48c15dab0 0x7fa48c06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa48c15d6d0 0x7fa48c161390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48002/0 (socket says 172.21.15.158:48002) 2023-10-16T09:44:21.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa48c15dab0 0x7fa48c06c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47828/0 (socket says 172.21.15.158:47828) 2023-10-16T09:44:21.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48affd700 1 -- 172.21.15.158:0/309825205 learned_addr learned my addr 172.21.15.158:0/309825205 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:21.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48b7fe700 1 --2- 172.21.15.158:0/309825205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48a7fc700 1 -- 172.21.15.158:0/309825205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 msgr2=0x7fa48c0cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:21.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48a7fc700 1 --2- 172.21.15.158:0/309825205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48a7fc700 1 -- 172.21.15.158:0/309825205 >> v1:172.21.15.158:6789/0 conn(0x7fa48c15d6d0 legacy=0x7fa48c161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48a7fc700 1 -- 172.21.15.158:0/309825205 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c161fe0 con 0x7fa48c15dab0 2023-10-16T09:44:21.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa489ffb700 1 -- 172.21.15.158:0/309825205 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95339671 0 0) 0x7fa48c161aa0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48b7fe700 1 --2- 172.21.15.158:0/309825205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:21.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa48a7fc700 1 --2- 172.21.15.158:0/309825205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15dab0 0x7fa48c06c490 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009840 tx=0x7fa47c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c0447dc0a1a708 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:21.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.628+0000 7fa489ffb700 1 -- 172.21.15.158:0/309825205 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c011830 con 0x7fa48c15dab0 2023-10-16T09:44:21.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa489ffb700 1 -- 172.21.15.158:0/309825205 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c011990 con 0x7fa48c15dab0 2023-10-16T09:44:21.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa489ffb700 1 -- 172.21.15.158:0/309825205 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c011d20 con 0x7fa48c15dab0 2023-10-16T09:44:21.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 -- 172.21.15.158:0/309825205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15dab0 msgr2=0x7fa48c06c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 --2- 172.21.15.158:0/309825205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15dab0 0x7fa48c06c490 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009840 tx=0x7fa47c009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:21.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 -- 172.21.15.158:0/309825205 shutdown_connections 2023-10-16T09:44:21.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 --2- 172.21.15.158:0/309825205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 --2- 172.21.15.158:0/309825205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15dab0 0x7fa48c06c490 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 -- 172.21.15.158:0/309825205 >> 172.21.15.158:0/309825205 conn(0x7fa48c0ce040 msgr2=0x7fa48c0ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:21.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 -- 172.21.15.158:0/309825205 shutdown_connections 2023-10-16T09:44:21.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.629+0000 7fa491910700 1 -- 172.21.15.158:0/309825205 wait complete. 2023-10-16T09:44:21.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.630+0000 7fa491910700 1 Processor -- start 2023-10-16T09:44:21.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.630+0000 7fa491910700 1 -- start start 2023-10-16T09:44:21.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa491910700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa491910700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15d6d0 0x7fa48c1f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa491910700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa48c15dab0 0x7fa48c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa491910700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c0c82d0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa491910700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c155450 con 0x7fa48c15dab0 2023-10-16T09:44:21.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa491910700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c154840 con 0x7fa48c06d640 2023-10-16T09:44:21.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa48a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15d6d0 0x7fa48c1f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa48a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15d6d0 0x7fa48c1f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47834/0 (socket says 172.21.15.158:47834) 2023-10-16T09:44:21.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.631+0000 7fa48a7fc700 1 -- 172.21.15.158:0/198775170 learned_addr learned my addr 172.21.15.158:0/198775170 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:21.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48b7fe700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa48c15dab0 0x7fa48c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48affd700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48a7fc700 1 -- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 msgr2=0x7fa48c0c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48a7fc700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48a7fc700 1 -- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa48c15dab0 msgr2=0x7fa48c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48a7fc700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa48c15dab0 0x7fa48c0b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48a7fc700 1 -- 172.21.15.158:0/198775170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0094a0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48b7fe700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa48c15dab0 0x7fa48c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:21.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa48a7fc700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15d6d0 0x7fa48c1f6b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa47c000c00 tx=0x7fa47c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:21.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa473fff700 1 -- 172.21.15.158:0/198775170 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0038a0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa473fff700 1 -- 172.21.15.158:0/198775170 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c01e070 con 0x7fa48c15d6d0 2023-10-16T09:44:21.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48c154b20 con 0x7fa48c15d6d0 2023-10-16T09:44:21.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa473fff700 1 -- 172.21.15.158:0/198775170 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c003c60 con 0x7fa48c15d6d0 2023-10-16T09:44:21.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.632+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c1550e0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.633+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48c1555e0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.633+0000 7fa473fff700 1 -- 172.21.15.158:0/198775170 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa47c02d450 con 0x7fa48c15d6d0 2023-10-16T09:44:21.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.634+0000 7fa473fff700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa47806af00 0x7fa47806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:21.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.634+0000 7fa473fff700 1 -- 172.21.15.158:0/198775170 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fa47c08aa80 con 0x7fa48c15d6d0 2023-10-16T09:44:21.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.634+0000 7fa48affd700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa47806af00 0x7fa47806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:21.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.634+0000 7fa48affd700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa47806af00 0x7fa47806d3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa480000f60 tx=0x7fa480007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:21.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.638+0000 7fa473fff700 1 -- 172.21.15.158:0/198775170 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa47c0557b0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.846+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa48c0681b0 con 0x7fa48c15d6d0 2023-10-16T09:44:21.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.846+0000 7fa473fff700 1 -- 172.21.15.158:0/198775170 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa47c020080 con 0x7fa48c15d6d0 2023-10-16T09:44:21.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa47806af00 msgr2=0x7fa47806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa47806af00 0x7fa47806d3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa480000f60 tx=0x7fa480007be0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15d6d0 msgr2=0x7fa48c1f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:21.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15d6d0 0x7fa48c1f6b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa47c000c00 tx=0x7fa47c004750 comp rx=0 tx=0).stop 2023-10-16T09:44:21.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 shutdown_connections 2023-10-16T09:44:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa48c15dab0 0x7fa48c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa47806af00 0x7fa47806d3c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa48c06d640 0x7fa48c0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 --2- 172.21.15.158:0/198775170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa48c15d6d0 0x7fa48c1f6b50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 >> 172.21.15.158:0/198775170 conn(0x7fa48c0ce040 msgr2=0x7fa48c1604f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:21.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 shutdown_connections 2023-10-16T09:44:21.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:21.849+0000 7fa491910700 1 -- 172.21.15.158:0/198775170 wait complete. 2023-10-16T09:44:21.853 INFO:teuthology.orchestra.run.smithi158.stderr:listed 0 entries 2023-10-16T09:44:21.862 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-16T09:44:21.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:21.863 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-16T09:44:21.904 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:44:21.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:21.904 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-16T09:44:21.961 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:21.961 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:21.983 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:21.983 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:22.043 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:44:22.043 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:22.102 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:22.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.221+0000 7f494d321700 1 Processor -- start 2023-10-16T09:44:22.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.221+0000 7f494d321700 1 -- start start 2023-10-16T09:44:22.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f494d321700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f494806ab40 0x7f494806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f494d321700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 0x7f4948071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f494d321700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4948071e50 con 0x7f494806e5d0 2023-10-16T09:44:22.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f494d321700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4948161270 con 0x7f494806ab40 2023-10-16T09:44:22.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f494d321700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4948161510 con 0x7f49480729b0 2023-10-16T09:44:22.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f49467fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f494806e5d0 0x7f49480722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48018/0 (socket says 172.21.15.158:48018) 2023-10-16T09:44:22.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f49467fc700 1 -- 172.21.15.158:0/867953507 learned_addr learned my addr 172.21.15.158:0/867953507 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:22.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f4946ffd700 1 --2- 172.21.15.158:0/867953507 >> v2:172.21.15.172:3300/0 conn(0x7f494806ab40 0x7f494806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f49477fe700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 0x7f4948071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431384337 0 0) 0x7f4948071e50 con 0x7f494806e5d0 2023-10-16T09:44:22.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.222+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49300035b0 con 0x7f494806e5d0 2023-10-16T09:44:22.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202325552 0 0) 0x7f49300035b0 con 0x7f494806e5d0 2023-10-16T09:44:22.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 msgr2=0x7f4948071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:22.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 0x7f4948071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 >> v2:172.21.15.172:3300/0 conn(0x7f494806ab40 msgr2=0x7f494806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:22.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 --2- 172.21.15.158:0/867953507 >> v2:172.21.15.172:3300/0 conn(0x7f494806ab40 0x7f494806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49481617f0 con 0x7f494806e5d0 2023-10-16T09:44:22.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4938001670 con 0x7f494806e5d0 2023-10-16T09:44:22.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 >> v1:172.21.15.158:6789/0 conn(0x7f494806e5d0 legacy=0x7f49480722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930003a00 0x7f4930003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4930004850 0x7f4930004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 0x7f4930005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948161270 con 0x7f4930004850 2023-10-16T09:44:22.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948161510 con 0x7f4930003a00 2023-10-16T09:44:22.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493000ba10 con 0x7f49480729b0 2023-10-16T09:44:22.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49467fc700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4930004850 0x7f4930004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4946ffd700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930003a00 0x7f4930003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49477fe700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 0x7f4930005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49467fc700 1 -- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 msgr2=0x7f4930005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:22.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49467fc700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49480729b0 0x7f4930005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49467fc700 1 -- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930003a00 msgr2=0x7f4930003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:22.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49467fc700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930003a00 0x7f4930003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49467fc700 1 -- 172.21.15.158:0/867953507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300035b0 con 0x7f4930004850 2023-10-16T09:44:22.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4946ffd700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930003a00 0x7f4930003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:22.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f49467fc700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4930004850 0x7f4930004cd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f494806b560 tx=0x7f4930005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f115ece1eb6346 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.223+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938007c60 con 0x7f4930004850 2023-10-16T09:44:22.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493800ce80 con 0x7f4930004850 2023-10-16T09:44:22.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f4945ffb700 1 -- 172.21.15.158:0/867953507 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938005ba0 con 0x7f4930004850 2023-10-16T09:44:22.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f494d321700 1 -- 172.21.15.158:0/867953507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4930004850 msgr2=0x7f4930004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f494d321700 1 --2- 172.21.15.158:0/867953507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4930004850 0x7f4930004cd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f494806b560 tx=0x7f4930005880 comp rx=0 tx=0).stop 2023-10-16T09:44:22.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f49477fe700 1 -- 172.21.15.158:0/867953507 reap_dead start 2023-10-16T09:44:22.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f494d321700 1 -- 172.21.15.158:0/867953507 shutdown_connections 2023-10-16T09:44:22.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f494d321700 1 -- 172.21.15.158:0/867953507 >> 172.21.15.158:0/867953507 conn(0x7f49480d02b0 msgr2=0x7f49480d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:22.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f494d321700 1 -- 172.21.15.158:0/867953507 shutdown_connections 2023-10-16T09:44:22.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.224+0000 7f494d321700 1 -- 172.21.15.158:0/867953507 wait complete. 2023-10-16T09:44:22.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.225+0000 7f494d321700 1 Processor -- start 2023-10-16T09:44:22.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.225+0000 7f494d321700 1 -- start start 2023-10-16T09:44:22.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f494d321700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930004850 0x7f49481475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f494d321700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4948158780 0x7f494815a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f494d321700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494815a940 0x7f4948148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f494d321700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948159110 con 0x7f4948158780 2023-10-16T09:44:22.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f494d321700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948159f70 con 0x7f4930004850 2023-10-16T09:44:22.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f494d321700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4948159360 con 0x7f494815a940 2023-10-16T09:44:22.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f49467fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4948158780 0x7f494815a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f49467fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4948158780 0x7f494815a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47852/0 (socket says 172.21.15.158:47852) 2023-10-16T09:44:22.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f49467fc700 1 -- 172.21.15.158:0/712296160 learned_addr learned my addr 172.21.15.158:0/712296160 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:22.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f4946ffd700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930004850 0x7f49481475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.226+0000 7f49477fe700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494815a940 0x7f4948148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f49467fc700 1 -- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494815a940 msgr2=0x7f4948148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f49467fc700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494815a940 0x7f4948148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f49467fc700 1 -- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930004850 msgr2=0x7f49481475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f49467fc700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930004850 0x7f49481475c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f49467fc700 1 -- 172.21.15.158:0/712296160 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4948159640 con 0x7f4948158780 2023-10-16T09:44:22.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f4946ffd700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930004850 0x7f49481475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:22.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f49467fc700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4948158780 0x7f494815a400 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4938004b20 tx=0x7f4938000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f492f7fe700 1 -- 172.21.15.158:0/712296160 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49380070d0 con 0x7f4948158780 2023-10-16T09:44:22.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f492f7fe700 1 -- 172.21.15.158:0/712296160 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938007230 con 0x7f4948158780 2023-10-16T09:44:22.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4948159920 con 0x7f4948158780 2023-10-16T09:44:22.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f492f7fe700 1 -- 172.21.15.158:0/712296160 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49380075e0 con 0x7f4948158780 2023-10-16T09:44:22.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.227+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4948154f80 con 0x7f4948158780 2023-10-16T09:44:22.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.229+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49480727d0 con 0x7f4948158780 2023-10-16T09:44:22.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.229+0000 7f492f7fe700 1 -- 172.21.15.158:0/712296160 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f493801e430 con 0x7f4948158780 2023-10-16T09:44:22.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.229+0000 7f492f7fe700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f493406af00 0x7f493406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.229+0000 7f492f7fe700 1 -- 172.21.15.158:0/712296160 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f493808ac00 con 0x7f4948158780 2023-10-16T09:44:22.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.229+0000 7f4946ffd700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f493406af00 0x7f493406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.229+0000 7f4946ffd700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f493406af00 0x7f493406d3c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f493c009d80 tx=0x7f493c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.234+0000 7f492f7fe700 1 -- 172.21.15.158:0/712296160 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4938091080 con 0x7f4948158780 2023-10-16T09:44:22.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.427+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49480681b0 con 0x7f4948158780 2023-10-16T09:44:22.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.428+0000 7f492f7fe700 1 -- 172.21.15.158:0/712296160 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7f49380558b0 con 0x7f4948158780 2023-10-16T09:44:22.430 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:22.430 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:22.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f493406af00 msgr2=0x7f493406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f493406af00 0x7f493406d3c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f493c009d80 tx=0x7f493c008040 comp rx=0 tx=0).stop 2023-10-16T09:44:22.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4948158780 msgr2=0x7f494815a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4948158780 0x7f494815a400 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4938004b20 tx=0x7f4938000fe0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 shutdown_connections 2023-10-16T09:44:22.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f493406af00 0x7f493406d3c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494815a940 0x7f4948148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4930004850 0x7f49481475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 --2- 172.21.15.158:0/712296160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4948158780 0x7f494815a400 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 >> 172.21.15.158:0/712296160 conn(0x7f49480d02b0 msgr2=0x7f494815e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:22.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 shutdown_connections 2023-10-16T09:44:22.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.430+0000 7f494d321700 1 -- 172.21.15.158:0/712296160 wait complete. 2023-10-16T09:44:22.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:22.445 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:22.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.571+0000 7f702c05f700 1 Processor -- start 2023-10-16T09:44:22.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.571+0000 7f702c05f700 1 -- start start 2023-10-16T09:44:22.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f702c05f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f70240cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f702c05f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70240cf360 0x7f70240d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f702c05f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240d1ee0 con 0x7f702415daa0 2023-10-16T09:44:22.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f702c05f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70240d40a0 con 0x7f70240cf360 2023-10-16T09:44:22.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f702c05f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70240d12d0 con 0x7f702415d6c0 2023-10-16T09:44:22.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f702a5fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70240cf360 0x7f70240d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f7029dfb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f70240cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f70295fa700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f702415daa0 0x7f70240cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:51938/0 (socket says 172.21.15.158:51938) 2023-10-16T09:44:22.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f70295fa700 1 -- 172.21.15.158:0/4257731399 learned_addr learned my addr 172.21.15.158:0/4257731399 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:22.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.572+0000 7f7028df9700 1 -- 172.21.15.158:0/4257731399 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764528599 0 0) 0x7f70240d1ee0 con 0x7f702415daa0 2023-10-16T09:44:22.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f7028df9700 1 -- 172.21.15.158:0/4257731399 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70100035b0 con 0x7f702415daa0 2023-10-16T09:44:22.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f702a5fc700 1 -- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 msgr2=0x7f70240cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:22.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f702a5fc700 1 --2- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f70240cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f702a5fc700 1 -- 172.21.15.158:0/4257731399 >> v1:172.21.15.172:6789/0 conn(0x7f702415daa0 legacy=0x7f70240cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f702a5fc700 1 -- 172.21.15.158:0/4257731399 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70240d15b0 con 0x7f70240cf360 2023-10-16T09:44:22.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f7029dfb700 1 --2- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f70240cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:22.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f702a5fc700 1 --2- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cf360 0x7f70240d0470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f701c009d90 tx=0x7f701c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f66e3ad5a404c93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f7028df9700 1 -- 172.21.15.158:0/4257731399 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c00e070 con 0x7f70240cf360 2023-10-16T09:44:22.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f7028df9700 1 -- 172.21.15.158:0/4257731399 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c00bb20 con 0x7f70240cf360 2023-10-16T09:44:22.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.573+0000 7f7028df9700 1 -- 172.21.15.158:0/4257731399 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c012430 con 0x7f70240cf360 2023-10-16T09:44:22.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 -- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cf360 msgr2=0x7f70240d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 --2- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cf360 0x7f70240d0470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f701c009d90 tx=0x7f701c00b330 comp rx=0 tx=0).stop 2023-10-16T09:44:22.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 -- 172.21.15.158:0/4257731399 shutdown_connections 2023-10-16T09:44:22.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 --2- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cf360 0x7f70240d0470 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 --2- 172.21.15.158:0/4257731399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f70240cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 -- 172.21.15.158:0/4257731399 >> 172.21.15.158:0/4257731399 conn(0x7f702409c940 msgr2=0x7f70240d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:22.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 -- 172.21.15.158:0/4257731399 shutdown_connections 2023-10-16T09:44:22.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.574+0000 7f702c05f700 1 -- 172.21.15.158:0/4257731399 wait complete. 2023-10-16T09:44:22.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.575+0000 7f702c05f700 1 Processor -- start 2023-10-16T09:44:22.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.575+0000 7f702c05f700 1 -- start start 2023-10-16T09:44:22.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f702c05f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70240cf360 0x7f702415e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f702c05f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f7024159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f702c05f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f702415daa0 0x7f702415a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f702c05f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7024160f00 con 0x7f70240cf360 2023-10-16T09:44:22.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f702c05f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7024160c20 con 0x7f702415daa0 2023-10-16T09:44:22.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f702c05f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7024144840 con 0x7f702415d6c0 2023-10-16T09:44:22.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f7029dfb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70240cf360 0x7f702415e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f7029dfb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70240cf360 0x7f702415e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47862/0 (socket says 172.21.15.158:47862) 2023-10-16T09:44:22.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f7029dfb700 1 -- 172.21.15.158:0/536110779 learned_addr learned my addr 172.21.15.158:0/536110779 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:22.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f70295fa700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f7024159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.576+0000 7f702a5fc700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f702415daa0 0x7f702415a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f7029dfb700 1 -- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 msgr2=0x7f7024159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f7029dfb700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f7024159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f7029dfb700 1 -- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f702415daa0 msgr2=0x7f702415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f7029dfb700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f702415daa0 0x7f702415a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f7029dfb700 1 -- 172.21.15.158:0/536110779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70200094a0 con 0x7f70240cf360 2023-10-16T09:44:22.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f7029dfb700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70240cf360 0x7f702415e350 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f702000ebe0 tx=0x7f702000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f701a7fc700 1 -- 172.21.15.158:0/536110779 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702000c850 con 0x7f70240cf360 2023-10-16T09:44:22.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f701a7fc700 1 -- 172.21.15.158:0/536110779 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702000c9b0 con 0x7f70240cf360 2023-10-16T09:44:22.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c009a70 con 0x7f70240cf360 2023-10-16T09:44:22.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f701a7fc700 1 -- 172.21.15.158:0/536110779 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702000cd40 con 0x7f70240cf360 2023-10-16T09:44:22.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.577+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7024144e00 con 0x7f70240cf360 2023-10-16T09:44:22.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.578+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70240681b0 con 0x7f70240cf360 2023-10-16T09:44:22.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.579+0000 7f701a7fc700 1 -- 172.21.15.158:0/536110779 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7020013470 con 0x7f70240cf360 2023-10-16T09:44:22.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.579+0000 7f701a7fc700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f700806af00 0x7f700806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.579+0000 7f701a7fc700 1 -- 172.21.15.158:0/536110779 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f7020089a60 con 0x7f70240cf360 2023-10-16T09:44:22.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.579+0000 7f70295fa700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f700806af00 0x7f700806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.580+0000 7f70295fa700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f700806af00 0x7f700806d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f70240b2da0 tx=0x7f7014007dc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.584+0000 7f701a7fc700 1 -- 172.21.15.158:0/536110779 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7020054710 con 0x7f70240cf360 2023-10-16T09:44:22.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.778+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f70240cda90 con 0x7f70240cf360 2023-10-16T09:44:22.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.778+0000 7f701a7fc700 1 -- 172.21.15.158:0/536110779 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f702000cea0 con 0x7f70240cf360 2023-10-16T09:44:22.780 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:22.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f700806af00 msgr2=0x7f700806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f700806af00 0x7f700806d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f70240b2da0 tx=0x7f7014007dc0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70240cf360 msgr2=0x7f702415e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70240cf360 0x7f702415e350 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f702000ebe0 tx=0x7f702000eef0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 shutdown_connections 2023-10-16T09:44:22.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f702415daa0 0x7f702415a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70240cf360 0x7f702415e350 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f700806af00 0x7f700806d3c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 --2- 172.21.15.158:0/536110779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f702415d6c0 0x7f7024159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 >> 172.21.15.158:0/536110779 conn(0x7f702409c940 msgr2=0x7f70240b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:22.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 shutdown_connections 2023-10-16T09:44:22.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.780+0000 7f702c05f700 1 -- 172.21.15.158:0/536110779 wait complete. 2023-10-16T09:44:22.794 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:22.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.917+0000 7f98bbe17700 1 Processor -- start 2023-10-16T09:44:22.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.917+0000 7f98bbe17700 1 -- start start 2023-10-16T09:44:22.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98bbe17700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b406afe0 0x7f98b40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98bbe17700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98b40cc590 0x7f98b40afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98bbe17700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b40cbd30 con 0x7f98b415da40 2023-10-16T09:44:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98bbe17700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b40cc1c0 con 0x7f98b40cc590 2023-10-16T09:44:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98bbe17700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b415de20 con 0x7f98b406afe0 2023-10-16T09:44:22.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98b40cc590 0x7f98b40afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98b40cc590 0x7f98b40afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47874/0 (socket says 172.21.15.158:47874) 2023-10-16T09:44:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 -- 172.21.15.158:0/1830584469 learned_addr learned my addr 172.21.15.158:0/1830584469 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98b93b2700 1 --2- 172.21.15.158:0/1830584469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b406afe0 0x7f98b40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 -- 172.21.15.158:0/1830584469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b406afe0 msgr2=0x7f98b40caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:22.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 --2- 172.21.15.158:0/1830584469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b406afe0 0x7f98b40caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 -- 172.21.15.158:0/1830584469 >> v1:172.21.15.172:6789/0 conn(0x7f98b415da40 legacy=0x7f98b406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 -- 172.21.15.158:0/1830584469 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40ce260 con 0x7f98b40cc590 2023-10-16T09:44:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98ba3b4700 1 --2- 172.21.15.158:0/1830584469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b40cc590 0x7f98b40afa40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009e50 tx=0x7f98ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b705404117689eb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98b8bb1700 1 -- 172.21.15.158:0/1830584469 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac00e070 con 0x7f98b40cc590 2023-10-16T09:44:22.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98b8bb1700 1 -- 172.21.15.158:0/1830584469 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac00b9c0 con 0x7f98b40cc590 2023-10-16T09:44:22.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.918+0000 7f98b8bb1700 1 -- 172.21.15.158:0/1830584469 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac00bd70 con 0x7f98b40cc590 2023-10-16T09:44:22.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 -- 172.21.15.158:0/1830584469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b40cc590 msgr2=0x7f98b40afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 --2- 172.21.15.158:0/1830584469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b40cc590 0x7f98b40afa40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009e50 tx=0x7f98ac00b330 comp rx=0 tx=0).stop 2023-10-16T09:44:22.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 -- 172.21.15.158:0/1830584469 shutdown_connections 2023-10-16T09:44:22.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 --2- 172.21.15.158:0/1830584469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b40cc590 0x7f98b40afa40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 --2- 172.21.15.158:0/1830584469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b406afe0 0x7f98b40caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 -- 172.21.15.158:0/1830584469 >> 172.21.15.158:0/1830584469 conn(0x7f98b4099a10 msgr2=0x7f98b40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:22.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 -- 172.21.15.158:0/1830584469 shutdown_connections 2023-10-16T09:44:22.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.919+0000 7f98bbe17700 1 -- 172.21.15.158:0/1830584469 wait complete. 2023-10-16T09:44:22.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.920+0000 7f98bbe17700 1 Processor -- start 2023-10-16T09:44:22.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.920+0000 7f98bbe17700 1 -- start start 2023-10-16T09:44:22.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98bbe17700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b406afe0 0x7f98b414e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98bbe17700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b40cc590 0x7f98b40ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98bbe17700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b415da40 0x7f98b41f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98bbe17700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40ca5a0 con 0x7f98b406afe0 2023-10-16T09:44:22.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98bbe17700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40c5370 con 0x7f98b40cc590 2023-10-16T09:44:22.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98bbe17700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40c5650 con 0x7f98b415da40 2023-10-16T09:44:22.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b406afe0 0x7f98b414e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b406afe0 0x7f98b414e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47876/0 (socket says 172.21.15.158:47876) 2023-10-16T09:44:22.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 -- 172.21.15.158:0/4183116896 learned_addr learned my addr 172.21.15.158:0/4183116896 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:22.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b93b2700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b40cc590 0x7f98b40ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98ba3b4700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b415da40 0x7f98b41f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 -- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b415da40 msgr2=0x7f98b41f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b415da40 0x7f98b41f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 -- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b40cc590 msgr2=0x7f98b40ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:22.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b40cc590 0x7f98b40ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:22.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 -- 172.21.15.158:0/4183116896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac009b30 con 0x7f98b406afe0 2023-10-16T09:44:22.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98ba3b4700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b415da40 0x7f98b41f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:22.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.921+0000 7f98b9bb3700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b406afe0 0x7f98b414e8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f98a400c4e0 tx=0x7f98a400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.922+0000 7f98aa7fc700 1 -- 172.21.15.158:0/4183116896 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a40052d0 con 0x7f98b406afe0 2023-10-16T09:44:22.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.922+0000 7f98aa7fc700 1 -- 172.21.15.158:0/4183116896 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a4005430 con 0x7f98b406afe0 2023-10-16T09:44:22.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.922+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b40c5da0 con 0x7f98b406afe0 2023-10-16T09:44:22.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.922+0000 7f98aa7fc700 1 -- 172.21.15.158:0/4183116896 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a40038c0 con 0x7f98b406afe0 2023-10-16T09:44:22.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.922+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b40c8b20 con 0x7f98b406afe0 2023-10-16T09:44:22.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.923+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b40c62c0 con 0x7f98b406afe0 2023-10-16T09:44:22.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.923+0000 7f98aa7fc700 1 -- 172.21.15.158:0/4183116896 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f98a4003a20 con 0x7f98b406afe0 2023-10-16T09:44:22.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.923+0000 7f98aa7fc700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f989806af00 0x7f989806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:22.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.924+0000 7f98aa7fc700 1 -- 172.21.15.158:0/4183116896 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f98a4088ec0 con 0x7f98b406afe0 2023-10-16T09:44:22.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.924+0000 7f98b93b2700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f989806af00 0x7f989806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:22.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.924+0000 7f98b93b2700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f989806af00 0x7f989806d3c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98b40c5300 tx=0x7f98b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:22.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:22.928+0000 7f98aa7fc700 1 -- 172.21.15.158:0/4183116896 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98b40c62c0 con 0x7f98b406afe0 2023-10-16T09:44:23.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.118+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98b40681b0 con 0x7f98b406afe0 2023-10-16T09:44:23.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.118+0000 7f98aa7fc700 1 -- 172.21.15.158:0/4183116896 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f98a4017020 con 0x7f98b406afe0 2023-10-16T09:44:23.120 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:23.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f989806af00 msgr2=0x7f989806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f989806af00 0x7f989806d3c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98b40c5300 tx=0x7f98b0008040 comp rx=0 tx=0).stop 2023-10-16T09:44:23.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b406afe0 msgr2=0x7f98b414e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b406afe0 0x7f98b414e8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f98a400c4e0 tx=0x7f98a400c8a0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 shutdown_connections 2023-10-16T09:44:23.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f989806af00 0x7f989806d3c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b415da40 0x7f98b41f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b406afe0 0x7f98b414e8c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 --2- 172.21.15.158:0/4183116896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b40cc590 0x7f98b40ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 >> 172.21.15.158:0/4183116896 conn(0x7f98b4099a10 msgr2=0x7f98b40b47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 shutdown_connections 2023-10-16T09:44:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.121+0000 7f98bbe17700 1 -- 172.21.15.158:0/4183116896 wait complete. 2023-10-16T09:44:23.134 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:23.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.257+0000 7f3278b6b700 1 Processor -- start 2023-10-16T09:44:23.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.257+0000 7f3278b6b700 1 -- start start 2023-10-16T09:44:23.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3278b6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f32740d0e10 0x7f327415c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3278b6b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3274159660 con 0x7f32740cbef0 2023-10-16T09:44:23.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3278b6b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327406b120 con 0x7f3274159280 2023-10-16T09:44:23.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3278b6b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32740cb710 con 0x7f32740d0e10 2023-10-16T09:44:23.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3271d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f32740d0e10 0x7f327415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3272d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f32740cbef0 0x7f32740cc340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48064/0 (socket says 172.21.15.158:48064) 2023-10-16T09:44:23.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3272d9d700 1 -- 172.21.15.158:0/2752249654 learned_addr learned my addr 172.21.15.158:0/2752249654 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:23.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3271d9b700 1 -- 172.21.15.158:0/2752249654 >> v1:172.21.15.172:6789/0 conn(0x7f3274159280 legacy=0x7f327406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3271d9b700 1 -- 172.21.15.158:0/2752249654 >> v1:172.21.15.158:6789/0 conn(0x7f32740cbef0 legacy=0x7f32740cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3271d9b700 1 -- 172.21.15.158:0/2752249654 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327406b400 con 0x7f32740d0e10 2023-10-16T09:44:23.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f327159a700 1 -- 172.21.15.158:0/2752249654 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851538648 0 0) 0x7f3274159660 con 0x7f32740cbef0 2023-10-16T09:44:23.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f3271d9b700 1 --2- 172.21.15.158:0/2752249654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32740d0e10 0x7f327415c590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3268009840 tx=0x7f3268009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20eb9bc694d23ff2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f327159a700 1 -- 172.21.15.158:0/2752249654 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268011830 con 0x7f32740d0e10 2023-10-16T09:44:23.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f327159a700 1 -- 172.21.15.158:0/2752249654 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268011990 con 0x7f32740d0e10 2023-10-16T09:44:23.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.258+0000 7f327159a700 1 -- 172.21.15.158:0/2752249654 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268011d20 con 0x7f32740d0e10 2023-10-16T09:44:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.259+0000 7f3278b6b700 1 -- 172.21.15.158:0/2752249654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32740d0e10 msgr2=0x7f327415c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.259+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2752249654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32740d0e10 0x7f327415c590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3268009840 tx=0x7f3268009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.259+0000 7f3278b6b700 1 -- 172.21.15.158:0/2752249654 shutdown_connections 2023-10-16T09:44:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.259+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2752249654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32740d0e10 0x7f327415c590 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.259+0000 7f3278b6b700 1 -- 172.21.15.158:0/2752249654 >> 172.21.15.158:0/2752249654 conn(0x7f3274099a10 msgr2=0x7f32740d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.259+0000 7f3278b6b700 1 -- 172.21.15.158:0/2752249654 shutdown_connections 2023-10-16T09:44:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.259+0000 7f3278b6b700 1 -- 172.21.15.158:0/2752249654 wait complete. 2023-10-16T09:44:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.260+0000 7f3278b6b700 1 Processor -- start 2023-10-16T09:44:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.260+0000 7f3278b6b700 1 -- start start 2023-10-16T09:44:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3278b6b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32740cbef0 0x7f32741f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3278b6b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32740d0e10 0x7f32740c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3278b6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274159280 0x7f32740c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3278b6b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32741f41b0 con 0x7f3274159280 2023-10-16T09:44:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3278b6b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32741f4fd0 con 0x7f32740d0e10 2023-10-16T09:44:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3278b6b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32741f43c0 con 0x7f32740cbef0 2023-10-16T09:44:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274159280 0x7f32740c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274159280 0x7f32740c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47894/0 (socket says 172.21.15.158:47894) 2023-10-16T09:44:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 -- 172.21.15.158:0/2181382760 learned_addr learned my addr 172.21.15.158:0/2181382760 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3271d9b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32740d0e10 0x7f32740c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f327259c700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32740cbef0 0x7f32741f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 -- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32740cbef0 msgr2=0x7f32741f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32740cbef0 0x7f32741f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 -- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32740d0e10 msgr2=0x7f32740c6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32740d0e10 0x7f32740c6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3272d9d700 1 -- 172.21.15.158:0/2181382760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32680094a0 con 0x7f3274159280 2023-10-16T09:44:23.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.261+0000 7f3271d9b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32740d0e10 0x7f32740c6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:23.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.262+0000 7f3272d9d700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274159280 0x7f32740c4910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f326c00cb60 tx=0x7f326c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.262+0000 7f3262ffd700 1 -- 172.21.15.158:0/2181382760 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c00ab50 con 0x7f3274159280 2023-10-16T09:44:23.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.262+0000 7f3262ffd700 1 -- 172.21.15.158:0/2181382760 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c00acb0 con 0x7f3274159280 2023-10-16T09:44:23.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.262+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32741f46a0 con 0x7f3274159280 2023-10-16T09:44:23.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.262+0000 7f3262ffd700 1 -- 172.21.15.158:0/2181382760 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c007310 con 0x7f3274159280 2023-10-16T09:44:23.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.262+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32741f4c60 con 0x7f3274159280 2023-10-16T09:44:23.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.263+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32740681b0 con 0x7f3274159280 2023-10-16T09:44:23.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.263+0000 7f3262ffd700 1 -- 172.21.15.158:0/2181382760 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f326c007470 con 0x7f3274159280 2023-10-16T09:44:23.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.264+0000 7f3262ffd700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f325406af00 0x7f325406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.264+0000 7f3262ffd700 1 -- 172.21.15.158:0/2181382760 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f326c089800 con 0x7f3274159280 2023-10-16T09:44:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.264+0000 7f327259c700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f325406af00 0x7f325406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.264+0000 7f327259c700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f325406af00 0x7f325406d3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3264000f60 tx=0x7f3264009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.268+0000 7f3262ffd700 1 -- 172.21.15.158:0/2181382760 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f326c0544a0 con 0x7f3274159280 2023-10-16T09:44:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.459+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f32740c46f0 con 0x7f3274159280 2023-10-16T09:44:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.460+0000 7f3262ffd700 1 -- 172.21.15.158:0/2181382760 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f326c0180a0 con 0x7f3274159280 2023-10-16T09:44:23.462 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f325406af00 msgr2=0x7f325406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f325406af00 0x7f325406d3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3264000f60 tx=0x7f3264009040 comp rx=0 tx=0).stop 2023-10-16T09:44:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274159280 msgr2=0x7f32740c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274159280 0x7f32740c4910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f326c00cb60 tx=0x7f326c00cf20 comp rx=0 tx=0).stop 2023-10-16T09:44:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 shutdown_connections 2023-10-16T09:44:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274159280 0x7f32740c4910 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f325406af00 0x7f325406d3c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32740cbef0 0x7f32741f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 --2- 172.21.15.158:0/2181382760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32740d0e10 0x7f32740c6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 >> 172.21.15.158:0/2181382760 conn(0x7f3274099a10 msgr2=0x7f327406f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 shutdown_connections 2023-10-16T09:44:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.462+0000 7f3278b6b700 1 -- 172.21.15.158:0/2181382760 wait complete. 2023-10-16T09:44:23.476 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:23.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.599+0000 7f9ea369d700 1 Processor -- start 2023-10-16T09:44:23.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.599+0000 7f9ea369d700 1 -- start start 2023-10-16T09:44:23.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea369d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e9c15b540 0x7f9e9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea369d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9e9c06f180 0x7f9e9c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea369d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06fea0 0x7f9e9c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea369d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c15b920 con 0x7f9e9c15b540 2023-10-16T09:44:23.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea369d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c15ee60 con 0x7f9e9c06f180 2023-10-16T09:44:23.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea369d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c15e2e0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea1439700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e9c15b540 0x7f9e9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea1439700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e9c15b540 0x7f9e9c06d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47898/0 (socket says 172.21.15.158:47898) 2023-10-16T09:44:23.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea1439700 1 -- 172.21.15.158:0/3915456529 learned_addr learned my addr 172.21.15.158:0/3915456529 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:23.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea1c3a700 1 --2- 172.21.15.158:0/3915456529 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06fea0 0x7f9e9c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.600+0000 7f9ea0c38700 1 --2- 172.21.15.158:0/3915456529 >> v2:172.21.15.172:3300/0 conn(0x7f9e9c06f180 0x7f9e9c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea1439700 1 -- 172.21.15.158:0/3915456529 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06fea0 msgr2=0x7f9e9c06ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:23.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea1439700 1 --2- 172.21.15.158:0/3915456529 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06fea0 0x7f9e9c06ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea1439700 1 -- 172.21.15.158:0/3915456529 >> v2:172.21.15.172:3300/0 conn(0x7f9e9c06f180 msgr2=0x7f9e9c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:23.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea1439700 1 --2- 172.21.15.158:0/3915456529 >> v2:172.21.15.172:3300/0 conn(0x7f9e9c06f180 0x7f9e9c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea1439700 1 -- 172.21.15.158:0/3915456529 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c15e5c0 con 0x7f9e9c15b540 2023-10-16T09:44:23.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea0c38700 1 --2- 172.21.15.158:0/3915456529 >> v2:172.21.15.172:3300/0 conn(0x7f9e9c06f180 0x7f9e9c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:23.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea1439700 1 --2- 172.21.15.158:0/3915456529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c15b540 0x7f9e9c06d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0ca990 tx=0x7f9e8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4c49dd21dd366d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9e93fff700 1 -- 172.21.15.158:0/3915456529 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c011830 con 0x7f9e9c15b540 2023-10-16T09:44:23.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9e93fff700 1 -- 172.21.15.158:0/3915456529 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c011990 con 0x7f9e9c15b540 2023-10-16T09:44:23.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9e93fff700 1 -- 172.21.15.158:0/3915456529 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c011d20 con 0x7f9e9c15b540 2023-10-16T09:44:23.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea369d700 1 -- 172.21.15.158:0/3915456529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c15b540 msgr2=0x7f9e9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3915456529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c15b540 0x7f9e9c06d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0ca990 tx=0x7f9e8c009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea369d700 1 -- 172.21.15.158:0/3915456529 shutdown_connections 2023-10-16T09:44:23.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3915456529 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06fea0 0x7f9e9c06ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3915456529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c15b540 0x7f9e9c06d9c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3915456529 >> v2:172.21.15.172:3300/0 conn(0x7f9e9c06f180 0x7f9e9c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.601+0000 7f9ea369d700 1 -- 172.21.15.158:0/3915456529 >> 172.21.15.158:0/3915456529 conn(0x7f9e9c0d03b0 msgr2=0x7f9e9c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:23.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.602+0000 7f9ea369d700 1 -- 172.21.15.158:0/3915456529 shutdown_connections 2023-10-16T09:44:23.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.602+0000 7f9ea369d700 1 -- 172.21.15.158:0/3915456529 wait complete. 2023-10-16T09:44:23.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 Processor -- start 2023-10-16T09:44:23.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 -- start start 2023-10-16T09:44:23.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06f180 0x7f9e9c1451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c06fea0 0x7f9e9c157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e9c15b540 0x7f9e9c1f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c1f2da0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c0ae4b0 con 0x7f9e9c15b540 2023-10-16T09:44:23.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.603+0000 7f9ea369d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c0adb80 con 0x7f9e9c06f180 2023-10-16T09:44:23.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c06fea0 0x7f9e9c157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c06fea0 0x7f9e9c157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47906/0 (socket says 172.21.15.158:47906) 2023-10-16T09:44:23.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 -- 172.21.15.158:0/3917080872 learned_addr learned my addr 172.21.15.158:0/3917080872 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:23.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea1c3a700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e9c15b540 0x7f9e9c1f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea1439700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06f180 0x7f9e9c1451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 -- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06f180 msgr2=0x7f9e9c1451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06f180 0x7f9e9c1451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 -- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e9c15b540 msgr2=0x7f9e9c1f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e9c15b540 0x7f9e9c1f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 -- 172.21.15.158:0/3917080872 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c009550 con 0x7f9e9c06fea0 2023-10-16T09:44:23.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea1439700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06f180 0x7f9e9c1451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:23.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea1c3a700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e9c15b540 0x7f9e9c1f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:23.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea0c38700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c06fea0 0x7f9e9c157fc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800d800 tx=0x7f9e9800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9e91ffb700 1 -- 172.21.15.158:0/3917080872 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98004030 con 0x7f9e9c06fea0 2023-10-16T09:44:23.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9e91ffb700 1 -- 172.21.15.158:0/3917080872 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e98004190 con 0x7f9e9c06fea0 2023-10-16T09:44:23.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c0ade60 con 0x7f9e9c06fea0 2023-10-16T09:44:23.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9e91ffb700 1 -- 172.21.15.158:0/3917080872 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98004520 con 0x7f9e9c06fea0 2023-10-16T09:44:23.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.604+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c0ad830 con 0x7f9e9c06fea0 2023-10-16T09:44:23.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.605+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c0681b0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.605+0000 7f9e91ffb700 1 -- 172.21.15.158:0/3917080872 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9e980036a0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.606+0000 7f9e91ffb700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f9e8006af00 0x7f9e8006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.606+0000 7f9e91ffb700 1 -- 172.21.15.158:0/3917080872 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f9e98089fa0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.606+0000 7f9ea1439700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f9e8006af00 0x7f9e8006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.606+0000 7f9ea1439700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f9e8006af00 0x7f9e8006d3c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c014040 tx=0x7f9e8c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.610+0000 7f9e91ffb700 1 -- 172.21.15.158:0/3917080872 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e980bcfe0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.803+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9e9c0ae5f0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.803+0000 7f9e91ffb700 1 -- 172.21.15.158:0/3917080872 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9e980586d0 con 0x7f9e9c06fea0 2023-10-16T09:44:23.805 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:23.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f9e8006af00 msgr2=0x7f9e8006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f9e8006af00 0x7f9e8006d3c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c014040 tx=0x7f9e8c008030 comp rx=0 tx=0).stop 2023-10-16T09:44:23.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c06fea0 msgr2=0x7f9e9c157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c06fea0 0x7f9e9c157fc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800d800 tx=0x7f9e9800dbc0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 shutdown_connections 2023-10-16T09:44:23.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e9c15b540 0x7f9e9c1f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f9e8006af00 0x7f9e8006d3c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e9c06f180 0x7f9e9c1451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 --2- 172.21.15.158:0/3917080872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e9c06fea0 0x7f9e9c157fc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 >> 172.21.15.158:0/3917080872 conn(0x7f9e9c0d03b0 msgr2=0x7f9e9c073c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:23.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 shutdown_connections 2023-10-16T09:44:23.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.806+0000 7f9ea369d700 1 -- 172.21.15.158:0/3917080872 wait complete. 2023-10-16T09:44:23.820 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:44:23.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.939+0000 7f78f6796700 1 Processor -- start 2023-10-16T09:44:23.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78f6796700 1 -- start start 2023-10-16T09:44:23.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78f6796700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78f015d990 0x7f78f00b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78f6796700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f78f006e480 0x7f78f00b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78f6796700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f00af600 con 0x7f78f00b3760 2023-10-16T09:44:23.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78f6796700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f00b2c90 con 0x7f78f006e480 2023-10-16T09:44:23.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78f6796700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f006e9b0 con 0x7f78f015d990 2023-10-16T09:44:23.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78ef7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f78f006e480 0x7f78f00b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.940+0000 7f78f4d33700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f78f00b3760 0x7f78f00cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48092/0 (socket says 172.21.15.158:48092) 2023-10-16T09:44:23.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78f4d33700 1 -- 172.21.15.158:0/1188164166 learned_addr learned my addr 172.21.15.158:0/1188164166 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:23.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78effff700 1 --2- 172.21.15.158:0/1188164166 >> v2:172.21.15.172:3300/0 conn(0x7f78f015d990 0x7f78f00b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78ef7fe700 1 -- 172.21.15.158:0/1188164166 >> v2:172.21.15.172:3300/0 conn(0x7f78f015d990 msgr2=0x7f78f00b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:23.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78ef7fe700 1 --2- 172.21.15.158:0/1188164166 >> v2:172.21.15.172:3300/0 conn(0x7f78f015d990 0x7f78f00b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78ef7fe700 1 -- 172.21.15.158:0/1188164166 >> v1:172.21.15.158:6789/0 conn(0x7f78f00b3760 legacy=0x7f78f00cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78ef7fe700 1 -- 172.21.15.158:0/1188164166 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f00ce890 con 0x7f78f006e480 2023-10-16T09:44:23.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78ef7fe700 1 --2- 172.21.15.158:0/1188164166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00b3220 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f78f00b2fc0 tx=0x7f78e0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e7453b5547d1ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78eeffd700 1 -- 172.21.15.158:0/1188164166 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0011830 con 0x7f78f006e480 2023-10-16T09:44:23.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78eeffd700 1 -- 172.21.15.158:0/1188164166 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e0011990 con 0x7f78f006e480 2023-10-16T09:44:23.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.941+0000 7f78eeffd700 1 -- 172.21.15.158:0/1188164166 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0011d20 con 0x7f78f006e480 2023-10-16T09:44:23.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 -- 172.21.15.158:0/1188164166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 msgr2=0x7f78f00b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 --2- 172.21.15.158:0/1188164166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00b3220 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f78f00b2fc0 tx=0x7f78e0009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:23.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 -- 172.21.15.158:0/1188164166 shutdown_connections 2023-10-16T09:44:23.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 --2- 172.21.15.158:0/1188164166 >> v2:172.21.15.172:3300/0 conn(0x7f78f015d990 0x7f78f00b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 --2- 172.21.15.158:0/1188164166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00b3220 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 -- 172.21.15.158:0/1188164166 >> 172.21.15.158:0/1188164166 conn(0x7f78f006b9e0 msgr2=0x7f78f006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:23.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 -- 172.21.15.158:0/1188164166 shutdown_connections 2023-10-16T09:44:23.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.942+0000 7f78f6796700 1 -- 172.21.15.158:0/1188164166 wait complete. 2023-10-16T09:44:23.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.943+0000 7f78f6796700 1 Processor -- start 2023-10-16T09:44:23.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.943+0000 7f78f6796700 1 -- start start 2023-10-16T09:44:23.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78f6796700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78f6796700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78f00b3760 0x7f78f014e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78f6796700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f015d990 0x7f78f01f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78f6796700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00c9f20 con 0x7f78f006e480 2023-10-16T09:44:23.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78f6796700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f014edc0 con 0x7f78f015d990 2023-10-16T09:44:23.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78f6796700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00c9310 con 0x7f78f00b3760 2023-10-16T09:44:23.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47918/0 (socket says 172.21.15.158:47918) 2023-10-16T09:44:23.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 -- 172.21.15.158:0/2537865455 learned_addr learned my addr 172.21.15.158:0/2537865455 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:23.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78f4d33700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f015d990 0x7f78f01f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78ef7fe700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78f00b3760 0x7f78f014e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 -- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78f00b3760 msgr2=0x7f78f014e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78f00b3760 0x7f78f014e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 -- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f015d990 msgr2=0x7f78f01f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:23.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f015d990 0x7f78f01f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:23.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 -- 172.21.15.158:0/2537865455 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e00094a0 con 0x7f78f006e480 2023-10-16T09:44:23.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.944+0000 7f78effff700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00ca3b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f78d800d800 tx=0x7f78d800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.945+0000 7f78ecff9700 1 -- 172.21.15.158:0/2537865455 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d80125f0 con 0x7f78f006e480 2023-10-16T09:44:23.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.945+0000 7f78ecff9700 1 -- 172.21.15.158:0/2537865455 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d8012750 con 0x7f78f006e480 2023-10-16T09:44:23.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.945+0000 7f78ecff9700 1 -- 172.21.15.158:0/2537865455 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8012b00 con 0x7f78f006e480 2023-10-16T09:44:23.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.945+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f00c95f0 con 0x7f78f006e480 2023-10-16T09:44:23.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.945+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f00c9bb0 con 0x7f78f006e480 2023-10-16T09:44:23.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.946+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f00ca200 con 0x7f78f006e480 2023-10-16T09:44:23.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.946+0000 7f78ecff9700 1 -- 172.21.15.158:0/2537865455 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f78d8012c60 con 0x7f78f006e480 2023-10-16T09:44:23.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.947+0000 7f78ecff9700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f78d006af00 0x7f78d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:23.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.947+0000 7f78ef7fe700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f78d006af00 0x7f78d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:23.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.947+0000 7f78ecff9700 1 -- 172.21.15.158:0/2537865455 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f78d8089d90 con 0x7f78f006e480 2023-10-16T09:44:23.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.947+0000 7f78ef7fe700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f78d006af00 0x7f78d006d3c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f78f00b2fc0 tx=0x7f78e0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:23.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:23.951+0000 7f78ecff9700 1 -- 172.21.15.158:0/2537865455 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78d8054a30 con 0x7f78f006e480 2023-10-16T09:44:24.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.160+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f78f00681b0 con 0x7f78f006e480 2023-10-16T09:44:24.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.160+0000 7f78ecff9700 1 -- 172.21.15.158:0/2537865455 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f78d8058050 con 0x7f78f006e480 2023-10-16T09:44:24.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f78d006af00 msgr2=0x7f78d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f78d006af00 0x7f78d006d3c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f78f00b2fc0 tx=0x7f78e0008030 comp rx=0 tx=0).stop 2023-10-16T09:44:24.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 msgr2=0x7f78f00ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00ca3b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f78d800d800 tx=0x7f78d800dbc0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 shutdown_connections 2023-10-16T09:44:24.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f015d990 0x7f78f01f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78f006e480 0x7f78f00ca3b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f78d006af00 0x7f78d006d3c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 --2- 172.21.15.158:0/2537865455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78f00b3760 0x7f78f014e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 >> 172.21.15.158:0/2537865455 conn(0x7f78f006b9e0 msgr2=0x7f78f015cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:24.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.162+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 shutdown_connections 2023-10-16T09:44:24.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.163+0000 7f78f6796700 1 -- 172.21.15.158:0/2537865455 wait complete. 2023-10-16T09:44:24.167 INFO:teuthology.orchestra.run.smithi158.stderr:listed 0 entries 2023-10-16T09:44:24.176 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-16T09:44:24.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:24.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-16T09:44:24.218 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T09:44:24.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:24.219 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-16T09:44:24.278 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T09:44:24.279 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:44:24.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.397+0000 7fcf3f998700 1 Processor -- start 2023-10-16T09:44:24.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.398+0000 7fcf3f998700 1 -- start start 2023-10-16T09:44:24.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.398+0000 7fcf3f998700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf380cdee0 0x7fcf3815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.398+0000 7fcf3f998700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3815c5d0 con 0x7fcf380b6e10 2023-10-16T09:44:24.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.398+0000 7fcf3f998700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3815c8b0 con 0x7fcf380ce2c0 2023-10-16T09:44:24.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.398+0000 7fcf3f998700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3815d2c0 con 0x7fcf380cdee0 2023-10-16T09:44:24.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf3d734700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf380cdee0 0x7fcf3815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf3cf33700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcf380b6e10 0x7fcf381521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48104/0 (socket says 172.21.15.158:48104) 2023-10-16T09:44:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf3cf33700 1 -- 172.21.15.158:0/1743843455 learned_addr learned my addr 172.21.15.158:0/1743843455 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf3d734700 1 -- 172.21.15.158:0/1743843455 >> v1:172.21.15.172:6789/0 conn(0x7fcf380ce2c0 legacy=0x7fcf380cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf3d734700 1 -- 172.21.15.158:0/1743843455 >> v1:172.21.15.158:6789/0 conn(0x7fcf380b6e10 legacy=0x7fcf381521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf3d734700 1 -- 172.21.15.158:0/1743843455 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3815cb90 con 0x7fcf380cdee0 2023-10-16T09:44:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf2ffff700 1 -- 172.21.15.158:0/1743843455 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825574631 0 0) 0x7fcf3815c5d0 con 0x7fcf380b6e10 2023-10-16T09:44:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf3d734700 1 --2- 172.21.15.158:0/1743843455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380cdee0 0x7fcf3815bd10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009840 tx=0x7fcf28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7513cbeaf913f344 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf2ffff700 1 -- 172.21.15.158:0/1743843455 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28004230 con 0x7fcf380cdee0 2023-10-16T09:44:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf2ffff700 1 -- 172.21.15.158:0/1743843455 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28004390 con 0x7fcf380cdee0 2023-10-16T09:44:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.399+0000 7fcf2ffff700 1 -- 172.21.15.158:0/1743843455 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280036a0 con 0x7fcf380cdee0 2023-10-16T09:44:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.400+0000 7fcf3f998700 1 -- 172.21.15.158:0/1743843455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380cdee0 msgr2=0x7fcf3815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.400+0000 7fcf3f998700 1 --2- 172.21.15.158:0/1743843455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380cdee0 0x7fcf3815bd10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009840 tx=0x7fcf28009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.400+0000 7fcf3f998700 1 -- 172.21.15.158:0/1743843455 shutdown_connections 2023-10-16T09:44:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.400+0000 7fcf3f998700 1 --2- 172.21.15.158:0/1743843455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380cdee0 0x7fcf3815bd10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.400+0000 7fcf3f998700 1 -- 172.21.15.158:0/1743843455 >> 172.21.15.158:0/1743843455 conn(0x7fcf3806a8d0 msgr2=0x7fcf3806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.400+0000 7fcf3f998700 1 -- 172.21.15.158:0/1743843455 shutdown_connections 2023-10-16T09:44:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.400+0000 7fcf3f998700 1 -- 172.21.15.158:0/1743843455 wait complete. 2023-10-16T09:44:24.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.401+0000 7fcf3f998700 1 Processor -- start 2023-10-16T09:44:24.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3f998700 1 -- start start 2023-10-16T09:44:24.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3f998700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380b6e10 0x7fcf3814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3f998700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf380cdee0 0x7fcf380ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3f998700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf380ce2c0 0x7fcf380b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3f998700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38150350 con 0x7fcf380b6e10 2023-10-16T09:44:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3f998700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380d01d0 con 0x7fcf380cdee0 2023-10-16T09:44:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3f998700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380cf930 con 0x7fcf380ce2c0 2023-10-16T09:44:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.402+0000 7fcf3d734700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380b6e10 0x7fcf3814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380b6e10 0x7fcf3814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47932/0 (socket says 172.21.15.158:47932) 2023-10-16T09:44:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 -- 172.21.15.158:0/2446720454 learned_addr learned my addr 172.21.15.158:0/2446720454 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 -- 172.21.15.158:0/2446720454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf380ce2c0 msgr2=0x7fcf380b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf380ce2c0 0x7fcf380b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 -- 172.21.15.158:0/2446720454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf380cdee0 msgr2=0x7fcf380ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf380cdee0 0x7fcf380ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 -- 172.21.15.158:0/2446720454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf280094a0 con 0x7fcf380b6e10 2023-10-16T09:44:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3d734700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380b6e10 0x7fcf3814fe10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcf280090b0 tx=0x7fcf28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf2dffb700 1 -- 172.21.15.158:0/2446720454 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28011830 con 0x7fcf380b6e10 2023-10-16T09:44:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf2dffb700 1 -- 172.21.15.158:0/2446720454 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2801e070 con 0x7fcf380b6e10 2023-10-16T09:44:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf380cfc10 con 0x7fcf380b6e10 2023-10-16T09:44:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf2dffb700 1 -- 172.21.15.158:0/2446720454 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28011bb0 con 0x7fcf380b6e10 2023-10-16T09:44:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.403+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf380d06b0 con 0x7fcf380b6e10 2023-10-16T09:44:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.404+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf380d08a0 con 0x7fcf380b6e10 2023-10-16T09:44:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.405+0000 7fcf2dffb700 1 -- 172.21.15.158:0/2446720454 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcf28011d10 con 0x7fcf380b6e10 2023-10-16T09:44:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.405+0000 7fcf2dffb700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcf1c06af00 0x7fcf1c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.405+0000 7fcf2dffb700 1 -- 172.21.15.158:0/2446720454 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fcf2808ad90 con 0x7fcf380b6e10 2023-10-16T09:44:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.405+0000 7fcf3cf33700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcf1c06af00 0x7fcf1c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.406+0000 7fcf3cf33700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcf1c06af00 0x7fcf1c06d3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcf340011e0 tx=0x7fcf340079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.410+0000 7fcf2dffb700 1 -- 172.21.15.158:0/2446720454 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf28055ac0 con 0x7fcf380b6e10 2023-10-16T09:44:24.567 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:44:24.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:44:24.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.654+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf380681b0 con 0x7fcf380b6e10 2023-10-16T09:44:24.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.654+0000 7fcf2dffb700 1 -- 172.21.15.158:0/2446720454 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7fcf28020020 con 0x7fcf380b6e10 2023-10-16T09:44:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:24.659 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:44:24.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcf1c06af00 msgr2=0x7fcf1c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcf1c06af00 0x7fcf1c06d3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcf340011e0 tx=0x7fcf340079b0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380b6e10 msgr2=0x7fcf3814fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380b6e10 0x7fcf3814fe10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcf280090b0 tx=0x7fcf28004750 comp rx=0 tx=0).stop 2023-10-16T09:44:24.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 shutdown_connections 2023-10-16T09:44:24.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcf1c06af00 0x7fcf1c06d3c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf380ce2c0 0x7fcf380b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf380b6e10 0x7fcf3814fe10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 --2- 172.21.15.158:0/2446720454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf380cdee0 0x7fcf380ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.656+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 >> 172.21.15.158:0/2446720454 conn(0x7fcf3806a8d0 msgr2=0x7fcf380b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:24.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.657+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 shutdown_connections 2023-10-16T09:44:24.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.657+0000 7fcf3f998700 1 -- 172.21.15.158:0/2446720454 wait complete. 2023-10-16T09:44:24.663 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:44:24.670 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:24.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.829+0000 7f31e021f700 1 Processor -- start 2023-10-16T09:44:24.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.830+0000 7f31e021f700 1 -- start start 2023-10-16T09:44:24.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.830+0000 7f31e021f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f31d815db30 0x7f31d806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.830+0000 7f31e021f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d8161c80 con 0x7f31d8162110 2023-10-16T09:44:24.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.830+0000 7f31e021f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d80b2bd0 con 0x7f31d80cbfd0 2023-10-16T09:44:24.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.830+0000 7f31e021f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d806e090 con 0x7f31d815db30 2023-10-16T09:44:24.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31ddfbb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f31d815db30 0x7f31d806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31dd7ba700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f31d8162110 0x7f31d80b1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48116/0 (socket says 172.21.15.158:48116) 2023-10-16T09:44:24.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31dd7ba700 1 -- 172.21.15.158:0/590245019 learned_addr learned my addr 172.21.15.158:0/590245019 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:24.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31ddfbb700 1 -- 172.21.15.158:0/590245019 >> v1:172.21.15.172:6789/0 conn(0x7f31d80cbfd0 legacy=0x7f31d80b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31ddfbb700 1 -- 172.21.15.158:0/590245019 >> v1:172.21.15.158:6789/0 conn(0x7f31d8162110 legacy=0x7f31d80b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31dcfb9700 1 -- 172.21.15.158:0/590245019 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848048852 0 0) 0x7f31d8161c80 con 0x7f31d8162110 2023-10-16T09:44:24.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31ddfbb700 1 -- 172.21.15.158:0/590245019 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d806ddb0 con 0x7f31d815db30 2023-10-16T09:44:24.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31ddfbb700 1 --2- 172.21.15.158:0/590245019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d806a8d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009840 tx=0x7f31c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3ca9e70a7d0e9c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:24.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31dcfb9700 1 -- 172.21.15.158:0/590245019 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8011830 con 0x7f31d815db30 2023-10-16T09:44:24.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31dcfb9700 1 -- 172.21.15.158:0/590245019 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c8011990 con 0x7f31d815db30 2023-10-16T09:44:24.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.831+0000 7f31dcfb9700 1 -- 172.21.15.158:0/590245019 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8011d20 con 0x7f31d815db30 2023-10-16T09:44:24.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.832+0000 7f31e021f700 1 -- 172.21.15.158:0/590245019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 msgr2=0x7f31d806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.832+0000 7f31e021f700 1 --2- 172.21.15.158:0/590245019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d806a8d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009840 tx=0x7f31c8009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:24.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.832+0000 7f31e021f700 1 -- 172.21.15.158:0/590245019 shutdown_connections 2023-10-16T09:44:24.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.832+0000 7f31e021f700 1 --2- 172.21.15.158:0/590245019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d806a8d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.832+0000 7f31e021f700 1 -- 172.21.15.158:0/590245019 >> 172.21.15.158:0/590245019 conn(0x7f31d80ccdb0 msgr2=0x7f31d80cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:24.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.832+0000 7f31e021f700 1 -- 172.21.15.158:0/590245019 shutdown_connections 2023-10-16T09:44:24.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.832+0000 7f31e021f700 1 -- 172.21.15.158:0/590245019 wait complete. 2023-10-16T09:44:24.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.833+0000 7f31e021f700 1 Processor -- start 2023-10-16T09:44:24.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.833+0000 7f31e021f700 1 -- start start 2023-10-16T09:44:24.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31e021f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80cbfd0 0x7f31d80acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31e021f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d814a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31e021f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d8162110 0x7f31d80c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31e021f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d814ad10 con 0x7f31d815db30 2023-10-16T09:44:24.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31e021f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80c6de0 con 0x7f31d8162110 2023-10-16T09:44:24.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31e021f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80c3f00 con 0x7f31d80cbfd0 2023-10-16T09:44:24.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d814a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d814a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47948/0 (socket says 172.21.15.158:47948) 2023-10-16T09:44:24.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 -- 172.21.15.158:0/2099126397 learned_addr learned my addr 172.21.15.158:0/2099126397 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31de7bc700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d8162110 0x7f31d80c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31ddfbb700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80cbfd0 0x7f31d80acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 -- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80cbfd0 msgr2=0x7f31d80acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80cbfd0 0x7f31d80acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 -- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d8162110 msgr2=0x7f31d80c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d8162110 0x7f31d80c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 -- 172.21.15.158:0/2099126397 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c80094a0 con 0x7f31d815db30 2023-10-16T09:44:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31ddfbb700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80cbfd0 0x7f31d80acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31dd7ba700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d814a7d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f31d400c520 tx=0x7f31d400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.834+0000 7f31ce7fc700 1 -- 172.21.15.158:0/2099126397 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4004c80 con 0x7f31d815db30 2023-10-16T09:44:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.835+0000 7f31ce7fc700 1 -- 172.21.15.158:0/2099126397 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d4004de0 con 0x7f31d815db30 2023-10-16T09:44:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.835+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d80c41e0 con 0x7f31d815db30 2023-10-16T09:44:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.835+0000 7f31ce7fc700 1 -- 172.21.15.158:0/2099126397 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4007310 con 0x7f31d815db30 2023-10-16T09:44:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.835+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d80c68c0 con 0x7f31d815db30 2023-10-16T09:44:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.835+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d80681b0 con 0x7f31d815db30 2023-10-16T09:44:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.836+0000 7f31ce7fc700 1 -- 172.21.15.158:0/2099126397 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f31d4007470 con 0x7f31d815db30 2023-10-16T09:44:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.836+0000 7f31ce7fc700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f31bc06af00 0x7f31bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.836+0000 7f31ce7fc700 1 -- 172.21.15.158:0/2099126397 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f31d40896b0 con 0x7f31d815db30 2023-10-16T09:44:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.836+0000 7f31ddfbb700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f31bc06af00 0x7f31bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.837+0000 7f31ddfbb700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f31bc06af00 0x7f31bc06d3c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f31c8000c00 tx=0x7f31c801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:24.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:24.841+0000 7f31ce7fc700 1 -- 172.21.15.158:0/2099126397 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31d4054350 con 0x7f31d815db30 2023-10-16T09:44:25.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.036+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31d80cea60 con 0x7f31d815db30 2023-10-16T09:44:25.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.037+0000 7f31ce7fc700 1 -- 172.21.15.158:0/2099126397 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7f31d4057970 con 0x7f31d815db30 2023-10-16T09:44:25.039 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:25.039 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f31bc06af00 msgr2=0x7f31bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f31bc06af00 0x7f31bc06d3c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f31c8000c00 tx=0x7f31c801c040 comp rx=0 tx=0).stop 2023-10-16T09:44:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 msgr2=0x7f31d814a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d814a7d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f31d400c520 tx=0x7f31d400c830 comp rx=0 tx=0).stop 2023-10-16T09:44:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 shutdown_connections 2023-10-16T09:44:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d8162110 0x7f31d80c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f31bc06af00 0x7f31bc06d3c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80cbfd0 0x7f31d80acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 --2- 172.21.15.158:0/2099126397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d815db30 0x7f31d814a7d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 >> 172.21.15.158:0/2099126397 conn(0x7f31d80ccdb0 msgr2=0x7f31d8158160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:25.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 shutdown_connections 2023-10-16T09:44:25.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.039+0000 7f31e021f700 1 -- 172.21.15.158:0/2099126397 wait complete. 2023-10-16T09:44:25.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:43:19.832012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:43:20.266773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T09:43:23.948527+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T09:44:25.054 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:25.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.176+0000 7f2432293700 1 Processor -- start 2023-10-16T09:44:25.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.176+0000 7f2432293700 1 -- start start 2023-10-16T09:44:25.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f2432293700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f242c159280 0x7f242c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f2432293700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f2432293700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242c159660 con 0x7f242c0cbef0 2023-10-16T09:44:25.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f2432293700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f242c0cb710 con 0x7f242c159280 2023-10-16T09:44:25.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f2432293700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c06b120 con 0x7f242c0d0e10 2023-10-16T09:44:25.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f242c159280 0x7f242c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f2430830700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f242c0cbef0 0x7f242c0cc380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:52038/0 (socket says 172.21.15.158:52038) 2023-10-16T09:44:25.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:58790/0 (socket says 172.21.15.158:58790) 2023-10-16T09:44:25.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f2430830700 1 -- 172.21.15.158:0/179819630 learned_addr learned my addr 172.21.15.158:0/179819630 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:25.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f242c159280 0x7f242c06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:49134/0 (socket says 172.21.15.158:49134) 2023-10-16T09:44:25.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242bfff700 1 -- 172.21.15.158:0/179819630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 msgr2=0x7f242c15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242bfff700 1 --2- 172.21.15.158:0/179819630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242bfff700 1 -- 172.21.15.158:0/179819630 >> v1:172.21.15.172:6789/0 conn(0x7f242c0cbef0 legacy=0x7f242c0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242bfff700 1 -- 172.21.15.158:0/179819630 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c06b400 con 0x7f242c159280 2023-10-16T09:44:25.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.177+0000 7f242bfff700 1 --2- 172.21.15.158:0/179819630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c159280 0x7f242c06a8d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2420009840 tx=0x7f2420009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e888c5d134d8109 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f242affd700 1 -- 172.21.15.158:0/179819630 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420011830 con 0x7f242c159280 2023-10-16T09:44:25.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f242affd700 1 -- 172.21.15.158:0/179819630 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420011990 con 0x7f242c159280 2023-10-16T09:44:25.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f242affd700 1 -- 172.21.15.158:0/179819630 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420011d20 con 0x7f242c159280 2023-10-16T09:44:25.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 -- 172.21.15.158:0/179819630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c159280 msgr2=0x7f242c06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 --2- 172.21.15.158:0/179819630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c159280 0x7f242c06a8d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2420009840 tx=0x7f2420009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:25.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 -- 172.21.15.158:0/179819630 shutdown_connections 2023-10-16T09:44:25.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 --2- 172.21.15.158:0/179819630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c159280 0x7f242c06a8d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 --2- 172.21.15.158:0/179819630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 -- 172.21.15.158:0/179819630 >> 172.21.15.158:0/179819630 conn(0x7f242c099a10 msgr2=0x7f242c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:25.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 -- 172.21.15.158:0/179819630 shutdown_connections 2023-10-16T09:44:25.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.178+0000 7f2432293700 1 -- 172.21.15.158:0/179819630 wait complete. 2023-10-16T09:44:25.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.179+0000 7f2432293700 1 Processor -- start 2023-10-16T09:44:25.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2432293700 1 -- start start 2023-10-16T09:44:25.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2432293700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c0cbef0 0x7f242c0c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2432293700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2432293700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159280 0x7f242c0ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2432293700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c15cfa0 con 0x7f242c159280 2023-10-16T09:44:25.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2432293700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0cffb0 con 0x7f242c0cbef0 2023-10-16T09:44:25.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2432293700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c1f3b00 con 0x7f242c0d0e10 2023-10-16T09:44:25.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2430830700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159280 0x7f242c0ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2430830700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159280 0x7f242c0ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47964/0 (socket says 172.21.15.158:47964) 2023-10-16T09:44:25.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.180+0000 7f2430830700 1 -- 172.21.15.158:0/104876895 learned_addr learned my addr 172.21.15.158:0/104876895 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:25.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f242bfff700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c0cbef0 0x7f242c0c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f242b7fe700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2430830700 1 -- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 msgr2=0x7f242c15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2430830700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2430830700 1 -- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c0cbef0 msgr2=0x7f242c0c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2430830700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c0cbef0 0x7f242c0c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2430830700 1 -- 172.21.15.158:0/104876895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24200094a0 con 0x7f242c159280 2023-10-16T09:44:25.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f242b7fe700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:25.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2430830700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159280 0x7f242c0ccc00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f242400ca80 tx=0x7f242400cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2428ff9700 1 -- 172.21.15.158:0/104876895 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242400a9e0 con 0x7f242c159280 2023-10-16T09:44:25.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2428ff9700 1 -- 172.21.15.158:0/104876895 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242400ab40 con 0x7f242c159280 2023-10-16T09:44:25.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c1f2d40 con 0x7f242c159280 2023-10-16T09:44:25.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2428ff9700 1 -- 172.21.15.158:0/104876895 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424012400 con 0x7f242c159280 2023-10-16T09:44:25.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.181+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c1f3300 con 0x7f242c159280 2023-10-16T09:44:25.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.182+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c0681b0 con 0x7f242c159280 2023-10-16T09:44:25.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.182+0000 7f2428ff9700 1 -- 172.21.15.158:0/104876895 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f242401b3b0 con 0x7f242c159280 2023-10-16T09:44:25.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.183+0000 7f2428ff9700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f241806af00 0x7f241806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.183+0000 7f2428ff9700 1 -- 172.21.15.158:0/104876895 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f2424089510 con 0x7f242c159280 2023-10-16T09:44:25.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.183+0000 7f242bfff700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f241806af00 0x7f241806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.183+0000 7f242bfff700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f241806af00 0x7f241806d3c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2420000c00 tx=0x7f2420008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.187+0000 7f2428ff9700 1 -- 172.21.15.158:0/104876895 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f24240541d0 con 0x7f242c159280 2023-10-16T09:44:25.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.378+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f242c1598b0 con 0x7f242c159280 2023-10-16T09:44:25.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.379+0000 7f2428ff9700 1 -- 172.21.15.158:0/104876895 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24240577f0 con 0x7f242c159280 2023-10-16T09:44:25.381 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:25.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f241806af00 msgr2=0x7f241806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f241806af00 0x7f241806d3c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2420000c00 tx=0x7f2420008030 comp rx=0 tx=0).stop 2023-10-16T09:44:25.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159280 msgr2=0x7f242c0ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159280 0x7f242c0ccc00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f242400ca80 tx=0x7f242400cd90 comp rx=0 tx=0).stop 2023-10-16T09:44:25.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 shutdown_connections 2023-10-16T09:44:25.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159280 0x7f242c0ccc00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c0cbef0 0x7f242c0c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f241806af00 0x7f241806d3c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 --2- 172.21.15.158:0/104876895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c0d0e10 0x7f242c15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 >> 172.21.15.158:0/104876895 conn(0x7f242c099a10 msgr2=0x7f242c15b460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:25.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 shutdown_connections 2023-10-16T09:44:25.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.381+0000 7f2432293700 1 -- 172.21.15.158:0/104876895 wait complete. 2023-10-16T09:44:25.395 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:44:25.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.518+0000 7f1da8cda700 1 Processor -- start 2023-10-16T09:44:25.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.518+0000 7f1da8cda700 1 -- start start 2023-10-16T09:44:25.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.518+0000 7f1da8cda700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 0x7f1da406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.518+0000 7f1da8cda700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1da40d0e10 0x7f1da415c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.518+0000 7f1da8cda700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da4159660 con 0x7f1da40cbef0 2023-10-16T09:44:25.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.518+0000 7f1da8cda700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da40cb710 con 0x7f1da40d0e10 2023-10-16T09:44:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.518+0000 7f1da8cda700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da406b120 con 0x7f1da4159280 2023-10-16T09:44:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da2d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1da40cbef0 0x7f1da40cc380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48148/0 (socket says 172.21.15.158:48148) 2023-10-16T09:44:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da2d9d700 1 -- 172.21.15.158:0/2448448589 learned_addr learned my addr 172.21.15.158:0/2448448589 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da259c700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 0x7f1da406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/2448448589 >> v2:172.21.15.172:3300/0 conn(0x7f1da40d0e10 0x7f1da415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013341028 0 0) 0x7f1da4159660 con 0x7f1da40cbef0 2023-10-16T09:44:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c0035b0 con 0x7f1da40cbef0 2023-10-16T09:44:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806917368 0 0) 0x7f1d8c0035b0 con 0x7f1da40cbef0 2023-10-16T09:44:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 msgr2=0x7f1da406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 0x7f1da406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 >> v2:172.21.15.172:3300/0 conn(0x7f1da40d0e10 msgr2=0x7f1da415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 --2- 172.21.15.158:0/2448448589 >> v2:172.21.15.172:3300/0 conn(0x7f1da40d0e10 0x7f1da415c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da406b400 con 0x7f1da40cbef0 2023-10-16T09:44:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1d9c0019f0 con 0x7f1da40cbef0 2023-10-16T09:44:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 >> v1:172.21.15.158:6789/0 conn(0x7f1da40cbef0 legacy=0x7f1da40cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 0x7f1d8c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1d8c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d8c004d90 0x7f1d8c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da406b120 con 0x7f1d8c003f80 2023-10-16T09:44:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c00ba10 con 0x7f1d8c004d90 2023-10-16T09:44:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c00bb50 con 0x7f1da4159280 2023-10-16T09:44:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1d8c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da259c700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 0x7f1d8c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.519+0000 7f1da2d9d700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d8c004d90 0x7f1d8c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da1d9b700 1 -- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 msgr2=0x7f1d8c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da4159280 0x7f1d8c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da1d9b700 1 -- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d8c004d90 msgr2=0x7f1d8c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d8c004d90 0x7f1d8c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da1d9b700 1 -- 172.21.15.158:0/2448448589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c0035b0 con 0x7f1d8c003f80 2023-10-16T09:44:25.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1d8c004850 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009b60 tx=0x7f1d8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269f321ea0aac33b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9400e660 con 0x7f1d8c003f80 2023-10-16T09:44:25.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9400e7c0 con 0x7f1d8c003f80 2023-10-16T09:44:25.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da159a700 1 -- 172.21.15.158:0/2448448589 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9400eb30 con 0x7f1d8c003f80 2023-10-16T09:44:25.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da8cda700 1 -- 172.21.15.158:0/2448448589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 msgr2=0x7f1d8c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da8cda700 1 --2- 172.21.15.158:0/2448448589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1d8c004850 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009b60 tx=0x7f1d8c005880 comp rx=0 tx=0).stop 2023-10-16T09:44:25.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.520+0000 7f1da2d9d700 1 -- 172.21.15.158:0/2448448589 reap_dead start 2023-10-16T09:44:25.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.521+0000 7f1da8cda700 1 -- 172.21.15.158:0/2448448589 shutdown_connections 2023-10-16T09:44:25.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.521+0000 7f1da8cda700 1 -- 172.21.15.158:0/2448448589 >> 172.21.15.158:0/2448448589 conn(0x7f1da4099a10 msgr2=0x7f1da40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:25.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.521+0000 7f1da8cda700 1 -- 172.21.15.158:0/2448448589 shutdown_connections 2023-10-16T09:44:25.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.521+0000 7f1da8cda700 1 -- 172.21.15.158:0/2448448589 wait complete. 2023-10-16T09:44:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.522+0000 7f1da8cda700 1 Processor -- start 2023-10-16T09:44:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.522+0000 7f1da8cda700 1 -- start start 2023-10-16T09:44:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1da414e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da40ae1b0 0x7f1da40c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da40af730 0x7f1da40c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40c8b10 con 0x7f1d8c003f80 2023-10-16T09:44:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40ca420 con 0x7f1da40af730 2023-10-16T09:44:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40ae6b0 con 0x7f1da40ae1b0 2023-10-16T09:44:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1da414e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1da414e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:47980/0 (socket says 172.21.15.158:47980) 2023-10-16T09:44:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 -- 172.21.15.158:0/3083115561 learned_addr learned my addr 172.21.15.158:0/3083115561 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da2d9d700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da40af730 0x7f1da40c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da40ae1b0 0x7f1da40c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 -- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da40ae1b0 msgr2=0x7f1da40c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da40ae1b0 0x7f1da40c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 -- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da40af730 msgr2=0x7f1da40c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da40af730 0x7f1da40c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 -- 172.21.15.158:0/3083115561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40ae990 con 0x7f1d8c003f80 2023-10-16T09:44:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da40ae1b0 0x7f1da40c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da259c700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1da414e400 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1d98011ad0 tx=0x7f1d98011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1d92ffd700 1 -- 172.21.15.158:0/3083115561 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9800f940 con 0x7f1d8c003f80 2023-10-16T09:44:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1d92ffd700 1 -- 172.21.15.158:0/3083115561 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9800faa0 con 0x7f1d8c003f80 2023-10-16T09:44:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1d92ffd700 1 -- 172.21.15.158:0/3083115561 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9800fe50 con 0x7f1d8c003f80 2023-10-16T09:44:25.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da40aec70 con 0x7f1d8c003f80 2023-10-16T09:44:25.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.523+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da40af230 con 0x7f1d8c003f80 2023-10-16T09:44:25.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.524+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da40681b0 con 0x7f1d8c003f80 2023-10-16T09:44:25.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.525+0000 7f1d92ffd700 1 -- 172.21.15.158:0/3083115561 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1d98014440 con 0x7f1d8c003f80 2023-10-16T09:44:25.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.525+0000 7f1d92ffd700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1d8406af00 0x7f1d8406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.525+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1d8406af00 0x7f1d8406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.525+0000 7f1d92ffd700 1 -- 172.21.15.158:0/3083115561 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f1d9808a000 con 0x7f1d8c003f80 2023-10-16T09:44:25.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.526+0000 7f1da1d9b700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1d8406af00 0x7f1d8406d3c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1d940090b0 tx=0x7f1d94008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.530+0000 7f1d92ffd700 1 -- 172.21.15.158:0/3083115561 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1d98054d60 con 0x7f1d8c003f80 2023-10-16T09:44:25.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.728+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1da40c3810 con 0x7f1d8c003f80 2023-10-16T09:44:25.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.729+0000 7f1d92ffd700 1 -- 172.21.15.158:0/3083115561 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1d98022020 con 0x7f1d8c003f80 2023-10-16T09:44:25.731 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:25.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1d8406af00 msgr2=0x7f1d8406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1d8406af00 0x7f1d8406d3c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1d940090b0 tx=0x7f1d94008030 comp rx=0 tx=0).stop 2023-10-16T09:44:25.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 msgr2=0x7f1da414e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1da414e400 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1d98011ad0 tx=0x7f1d98011de0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 shutdown_connections 2023-10-16T09:44:25.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da40af730 0x7f1da40c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d8c003f80 0x7f1da414e400 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1d8406af00 0x7f1d8406d3c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 --2- 172.21.15.158:0/3083115561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1da40ae1b0 0x7f1da40c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 >> 172.21.15.158:0/3083115561 conn(0x7f1da4099a10 msgr2=0x7f1da4158730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:25.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 shutdown_connections 2023-10-16T09:44:25.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.731+0000 7f1da8cda700 1 -- 172.21.15.158:0/3083115561 wait complete. 2023-10-16T09:44:25.745 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:44:25.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.866+0000 7f7d3e3bd700 1 Processor -- start 2023-10-16T09:44:25.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.866+0000 7f7d3e3bd700 1 -- start start 2023-10-16T09:44:25.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.867+0000 7f7d3e3bd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7d380cc590 0x7f7d380caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.867+0000 7f7d3e3bd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d380cbd30 con 0x7f7d3806afe0 2023-10-16T09:44:25.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.867+0000 7f7d3e3bd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d380cc1c0 con 0x7f7d3815da40 2023-10-16T09:44:25.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.867+0000 7f7d3e3bd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3815de20 con 0x7f7d380cc590 2023-10-16T09:44:25.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.867+0000 7f7d377fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7d3806afe0 0x7f7d380afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48166/0 (socket says 172.21.15.158:48166) 2023-10-16T09:44:25.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.867+0000 7f7d377fe700 1 -- 172.21.15.158:0/1895889993 learned_addr learned my addr 172.21.15.158:0/1895889993 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:25.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.867+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576957677 0 0) 0x7f7d380cbd30 con 0x7f7d3806afe0 2023-10-16T09:44:25.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d200035b0 con 0x7f7d3806afe0 2023-10-16T09:44:25.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d3c95a700 1 --2- 172.21.15.158:0/1895889993 >> v2:172.21.15.172:3300/0 conn(0x7f7d380cc590 0x7f7d380caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2734219437 0 0) 0x7f7d200035b0 con 0x7f7d3806afe0 2023-10-16T09:44:25.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 >> v2:172.21.15.172:3300/0 conn(0x7f7d380cc590 msgr2=0x7f7d380caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 --2- 172.21.15.158:0/1895889993 >> v2:172.21.15.172:3300/0 conn(0x7f7d380cc590 0x7f7d380caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 >> v1:172.21.15.172:6789/0 conn(0x7f7d3815da40 legacy=0x7f7d3806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d380ce260 con 0x7f7d3806afe0 2023-10-16T09:44:25.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7d28001670 con 0x7f7d3806afe0 2023-10-16T09:44:25.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 >> v1:172.21.15.158:6789/0 conn(0x7f7d3806afe0 legacy=0x7f7d380afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20003a00 0x7f7d20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20004850 0x7f7d20004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20005210 0x7f7d2000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815de20 con 0x7f7d20003a00 2023-10-16T09:44:25.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2000c0b0 con 0x7f7d20004850 2023-10-16T09:44:25.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2000c270 con 0x7f7d20005210 2023-10-16T09:44:25.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d37fff700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20003a00 0x7f7d20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d377fe700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20004850 0x7f7d20004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d3c95a700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20005210 0x7f7d2000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d37fff700 1 -- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20005210 msgr2=0x7f7d2000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d37fff700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20005210 0x7f7d2000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d37fff700 1 -- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20004850 msgr2=0x7f7d20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:25.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d37fff700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20004850 0x7f7d20004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d37fff700 1 -- 172.21.15.158:0/1895889993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200035b0 con 0x7f7d20003a00 2023-10-16T09:44:25.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d3c95a700 1 -- 172.21.15.158:0/1895889993 reap_dead start 2023-10-16T09:44:25.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.868+0000 7f7d37fff700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20003a00 0x7f7d20003e20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0017c0 tx=0x7f7d20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0a8cc08b178a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0052d0 con 0x7f7d20003a00 2023-10-16T09:44:25.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c005430 con 0x7f7d20003a00 2023-10-16T09:44:25.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d36ffd700 1 -- 172.21.15.158:0/1895889993 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0038c0 con 0x7f7d20003a00 2023-10-16T09:44:25.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/1895889993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20003a00 msgr2=0x7f7d20003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20003a00 0x7f7d20003e20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0017c0 tx=0x7f7d20005880 comp rx=0 tx=0).stop 2023-10-16T09:44:25.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/1895889993 shutdown_connections 2023-10-16T09:44:25.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/1895889993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20003a00 0x7f7d20003e20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/1895889993 >> 172.21.15.158:0/1895889993 conn(0x7f7d38099a10 msgr2=0x7f7d380d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:25.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/1895889993 shutdown_connections 2023-10-16T09:44:25.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.869+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/1895889993 wait complete. 2023-10-16T09:44:25.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.870+0000 7f7d3e3bd700 1 Processor -- start 2023-10-16T09:44:25.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3e3bd700 1 -- start start 2023-10-16T09:44:25.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3e3bd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20003a00 0x7f7d381e8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3e3bd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d380c48c0 0x7f7d3814f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3e3bd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d380c67b0 0x7f7d380c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3e3bd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c7170 con 0x7f7d380c67b0 2023-10-16T09:44:25.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3e3bd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380b3470 con 0x7f7d380c48c0 2023-10-16T09:44:25.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3e3bd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c5140 con 0x7f7d20003a00 2023-10-16T09:44:25.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3c95a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d380c67b0 0x7f7d380c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3c95a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d380c67b0 0x7f7d380c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48004/0 (socket says 172.21.15.158:48004) 2023-10-16T09:44:25.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.871+0000 7f7d3c95a700 1 -- 172.21.15.158:0/3689091367 learned_addr learned my addr 172.21.15.158:0/3689091367 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:25.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d377fe700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d380c48c0 0x7f7d3814f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d37fff700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20003a00 0x7f7d381e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3c95a700 1 -- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20003a00 msgr2=0x7f7d381e8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3c95a700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20003a00 0x7f7d381e8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3c95a700 1 -- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d380c48c0 msgr2=0x7f7d3814f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:25.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3c95a700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d380c48c0 0x7f7d3814f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:25.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3c95a700 1 -- 172.21.15.158:0/3689091367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d380c5420 con 0x7f7d380c67b0 2023-10-16T09:44:25.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3c95a700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d380c67b0 0x7f7d380c6c30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7d380ccb90 tx=0x7f7d30009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d34ff9700 1 -- 172.21.15.158:0/3689091367 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000e070 con 0x7f7d380c67b0 2023-10-16T09:44:25.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d34ff9700 1 -- 172.21.15.158:0/3689091367 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30003cf0 con 0x7f7d380c67b0 2023-10-16T09:44:25.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d380c5700 con 0x7f7d380c67b0 2023-10-16T09:44:25.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d34ff9700 1 -- 172.21.15.158:0/3689091367 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300117e0 con 0x7f7d380c67b0 2023-10-16T09:44:25.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.872+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3814fb00 con 0x7f7d380c67b0 2023-10-16T09:44:25.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.873+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d380681b0 con 0x7f7d380c67b0 2023-10-16T09:44:25.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.873+0000 7f7d34ff9700 1 -- 172.21.15.158:0/3689091367 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7d30011940 con 0x7f7d380c67b0 2023-10-16T09:44:25.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.874+0000 7f7d34ff9700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7d2406af00 0x7f7d2406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:25.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.874+0000 7f7d34ff9700 1 -- 172.21.15.158:0/3689091367 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f7d30089d80 con 0x7f7d380c67b0 2023-10-16T09:44:25.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.874+0000 7f7d37fff700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7d2406af00 0x7f7d2406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:25.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.874+0000 7f7d37fff700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7d2406af00 0x7f7d2406d3c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c005040 tx=0x7f7d2c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:25.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:25.878+0000 7f7d34ff9700 1 -- 172.21.15.158:0/3689091367 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d30054ab0 con 0x7f7d380c67b0 2023-10-16T09:44:26.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.071+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d380b3600 con 0x7f7d380c67b0 2023-10-16T09:44:26.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.071+0000 7f7d34ff9700 1 -- 172.21.15.158:0/3689091367 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d300580d0 con 0x7f7d380c67b0 2023-10-16T09:44:26.073 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:44:26.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7d2406af00 msgr2=0x7f7d2406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7d2406af00 0x7f7d2406d3c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c005040 tx=0x7f7d2c009030 comp rx=0 tx=0).stop 2023-10-16T09:44:26.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d380c67b0 msgr2=0x7f7d380c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d380c67b0 0x7f7d380c6c30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7d380ccb90 tx=0x7f7d30009f70 comp rx=0 tx=0).stop 2023-10-16T09:44:26.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 shutdown_connections 2023-10-16T09:44:26.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d380c67b0 0x7f7d380c6c30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7d2406af00 0x7f7d2406d3c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d20003a00 0x7f7d381e8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 --2- 172.21.15.158:0/3689091367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d380c48c0 0x7f7d3814f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 >> 172.21.15.158:0/3689091367 conn(0x7f7d38099a10 msgr2=0x7f7d381609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:26.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 shutdown_connections 2023-10-16T09:44:26.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.073+0000 7f7d3e3bd700 1 -- 172.21.15.158:0/3689091367 wait complete. 2023-10-16T09:44:26.087 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T09:44:26.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.209+0000 7f2035d20700 1 Processor -- start 2023-10-16T09:44:26.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.209+0000 7f2035d20700 1 -- start start 2023-10-16T09:44:26.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.209+0000 7f2035d20700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f20300cf500 0x7f20300cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.209+0000 7f2035d20700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203015dc20 con 0x7f203015d840 2023-10-16T09:44:26.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.209+0000 7f2035d20700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203006c0e0 con 0x7f20300cf070 2023-10-16T09:44:26.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.209+0000 7f2035d20700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f203006c3c0 con 0x7f20300cf500 2023-10-16T09:44:26.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f203015d840 0x7f2030161d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48184/0 (socket says 172.21.15.158:48184) 2023-10-16T09:44:26.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202f7fe700 1 -- 172.21.15.158:0/3179113489 learned_addr learned my addr 172.21.15.158:0/3179113489 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:26.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202ffff700 1 --2- 172.21.15.158:0/3179113489 >> v2:172.21.15.172:3300/0 conn(0x7f20300cf500 0x7f20300cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202e7fc700 1 -- 172.21.15.158:0/3179113489 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466915733 0 0) 0x7f2020001110 con 0x7f203015d840 2023-10-16T09:44:26.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202e7fc700 1 -- 172.21.15.158:0/3179113489 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f201c0035b0 con 0x7f203015d840 2023-10-16T09:44:26.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202ffff700 1 -- 172.21.15.158:0/3179113489 >> v1:172.21.15.172:6789/0 conn(0x7f20300cf070 legacy=0x7f20300cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202ffff700 1 -- 172.21.15.158:0/3179113489 >> v1:172.21.15.158:6789/0 conn(0x7f203015d840 legacy=0x7f2030161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202e7fc700 1 -- 172.21.15.158:0/3179113489 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364567484 0 0) 0x7f203006c0e0 con 0x7f20300cf070 2023-10-16T09:44:26.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202ffff700 1 -- 172.21.15.158:0/3179113489 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203006c6a0 con 0x7f20300cf500 2023-10-16T09:44:26.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202ffff700 1 --2- 172.21.15.158:0/3179113489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf500 0x7f20300cc4e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2028009d90 tx=0x7f202800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61ea93dad7793957 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.210+0000 7f202e7fc700 1 -- 172.21.15.158:0/3179113489 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202800e070 con 0x7f20300cf500 2023-10-16T09:44:26.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f202e7fc700 1 -- 172.21.15.158:0/3179113489 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202800bb20 con 0x7f20300cf500 2023-10-16T09:44:26.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f202e7fc700 1 -- 172.21.15.158:0/3179113489 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202800bed0 con 0x7f20300cf500 2023-10-16T09:44:26.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f2035d20700 1 -- 172.21.15.158:0/3179113489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf500 msgr2=0x7f20300cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f2035d20700 1 --2- 172.21.15.158:0/3179113489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf500 0x7f20300cc4e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2028009d90 tx=0x7f202800b330 comp rx=0 tx=0).stop 2023-10-16T09:44:26.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f2035d20700 1 -- 172.21.15.158:0/3179113489 shutdown_connections 2023-10-16T09:44:26.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f2035d20700 1 --2- 172.21.15.158:0/3179113489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf500 0x7f20300cc4e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f2035d20700 1 -- 172.21.15.158:0/3179113489 >> 172.21.15.158:0/3179113489 conn(0x7f203006a8d0 msgr2=0x7f203006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:26.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f2035d20700 1 -- 172.21.15.158:0/3179113489 shutdown_connections 2023-10-16T09:44:26.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.211+0000 7f2035d20700 1 -- 172.21.15.158:0/3179113489 wait complete. 2023-10-16T09:44:26.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.212+0000 7f2035d20700 1 Processor -- start 2023-10-16T09:44:26.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.212+0000 7f2035d20700 1 -- start start 2023-10-16T09:44:26.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f2035d20700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf070 0x7f2030147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f2035d20700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20300cf500 0x7f20301485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f2035d20700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f203015d840 0x7f203014b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f2035d20700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20300ca520 con 0x7f20300cf500 2023-10-16T09:44:26.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f2035d20700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20300c7f30 con 0x7f20300cf070 2023-10-16T09:44:26.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f2035d20700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203014b540 con 0x7f203015d840 2023-10-16T09:44:26.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f202effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20300cf500 0x7f20301485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f202effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20300cf500 0x7f20301485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48016/0 (socket says 172.21.15.158:48016) 2023-10-16T09:44:26.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.213+0000 7f202effd700 1 -- 172.21.15.158:0/3472028564 learned_addr learned my addr 172.21.15.158:0/3472028564 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:26.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f202f7fe700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf070 0x7f2030147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f202effd700 1 -- 172.21.15.158:0/3472028564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f203015d840 msgr2=0x7f203014b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f202effd700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f203015d840 0x7f203014b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f202effd700 1 -- 172.21.15.158:0/3472028564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf070 msgr2=0x7f2030147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f202effd700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf070 0x7f2030147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f202effd700 1 -- 172.21.15.158:0/3472028564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2028009a70 con 0x7f20300cf500 2023-10-16T09:44:26.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f202effd700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20300cf500 0x7f20301485a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f20180011e0 tx=0x7f201800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f2027fff700 1 -- 172.21.15.158:0/3472028564 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20180054e0 con 0x7f20300cf500 2023-10-16T09:44:26.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f2027fff700 1 -- 172.21.15.158:0/3472028564 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2018003680 con 0x7f20300cf500 2023-10-16T09:44:26.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20300c7c50 con 0x7f20300cf500 2023-10-16T09:44:26.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f2027fff700 1 -- 172.21.15.158:0/3472028564 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018003a30 con 0x7f20300cf500 2023-10-16T09:44:26.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.214+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20300ca140 con 0x7f20300cf500 2023-10-16T09:44:26.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.215+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20300c9cc0 con 0x7f20300cf500 2023-10-16T09:44:26.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.215+0000 7f2027fff700 1 -- 172.21.15.158:0/3472028564 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2018003b90 con 0x7f20300cf500 2023-10-16T09:44:26.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.216+0000 7f2027fff700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f201006af00 0x7f201006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.216+0000 7f2027fff700 1 -- 172.21.15.158:0/3472028564 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f2018088fa0 con 0x7f20300cf500 2023-10-16T09:44:26.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.216+0000 7f202f7fe700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f201006af00 0x7f201006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.216+0000 7f202f7fe700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f201006af00 0x7f201006d3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2020007fd0 tx=0x7f2020009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.220+0000 7f2027fff700 1 -- 172.21.15.158:0/3472028564 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f20300c9cc0 con 0x7f20300cf500 2023-10-16T09:44:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.412+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f20300ca660 con 0x7f20300cf500 2023-10-16T09:44:26.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.412+0000 7f2027fff700 1 -- 172.21.15.158:0/3472028564 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f20300ca660 con 0x7f20300cf500 2023-10-16T09:44:26.414 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:44:26.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f201006af00 msgr2=0x7f201006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f201006af00 0x7f201006d3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2020007fd0 tx=0x7f2020009040 comp rx=0 tx=0).stop 2023-10-16T09:44:26.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20300cf500 msgr2=0x7f20301485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20300cf500 0x7f20301485a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f20180011e0 tx=0x7f201800cae0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 shutdown_connections 2023-10-16T09:44:26.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f201006af00 0x7f201006d3c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f203015d840 0x7f203014b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20300cf070 0x7f2030147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 --2- 172.21.15.158:0/3472028564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20300cf500 0x7f20301485a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 >> 172.21.15.158:0/3472028564 conn(0x7f203006a8d0 msgr2=0x7f2030161a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:26.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 shutdown_connections 2023-10-16T09:44:26.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.414+0000 7f2035d20700 1 -- 172.21.15.158:0/3472028564 wait complete. 2023-10-16T09:44:26.428 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:44:26.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.549+0000 7f1ef122a700 1 Processor -- start 2023-10-16T09:44:26.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.549+0000 7f1ef122a700 1 -- start start 2023-10-16T09:44:26.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ef122a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1eec06ab40 0x7f1eec06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ef122a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06e5d0 0x7f1eec071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ef122a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1eec06af20 0x7f1eec0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ef122a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec071e50 con 0x7f1eec06ab40 2023-10-16T09:44:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ef122a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec161270 con 0x7f1eec06af20 2023-10-16T09:44:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ef122a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec161510 con 0x7f1eec06e5d0 2023-10-16T09:44:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1eec06ab40 0x7f1eec06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1eec06ab40 0x7f1eec06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48022/0 (socket says 172.21.15.158:48022) 2023-10-16T09:44:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 -- 172.21.15.158:0/4041044431 learned_addr learned my addr 172.21.15.158:0/4041044431 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eea59c700 1 --2- 172.21.15.158:0/4041044431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06e5d0 0x7f1eec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eeb59e700 1 --2- 172.21.15.158:0/4041044431 >> v2:172.21.15.172:3300/0 conn(0x7f1eec06af20 0x7f1eec0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 -- 172.21.15.158:0/4041044431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06e5d0 msgr2=0x7f1eec071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 --2- 172.21.15.158:0/4041044431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06e5d0 0x7f1eec071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 -- 172.21.15.158:0/4041044431 >> v2:172.21.15.172:3300/0 conn(0x7f1eec06af20 msgr2=0x7f1eec0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 --2- 172.21.15.158:0/4041044431 >> v2:172.21.15.172:3300/0 conn(0x7f1eec06af20 0x7f1eec0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 -- 172.21.15.158:0/4041044431 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec1617f0 con 0x7f1eec06ab40 2023-10-16T09:44:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eeb59e700 1 --2- 172.21.15.158:0/4041044431 >> v2:172.21.15.172:3300/0 conn(0x7f1eec06af20 0x7f1eec0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1eead9d700 1 --2- 172.21.15.158:0/4041044431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06ab40 0x7f1eec06d820 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1eec06eaf0 tx=0x7f1edc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82645ff80e21f6fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ee9d9b700 1 -- 172.21.15.158:0/4041044431 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc011830 con 0x7f1eec06ab40 2023-10-16T09:44:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ee9d9b700 1 -- 172.21.15.158:0/4041044431 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc011990 con 0x7f1eec06ab40 2023-10-16T09:44:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.550+0000 7f1ee9d9b700 1 -- 172.21.15.158:0/4041044431 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc011d20 con 0x7f1eec06ab40 2023-10-16T09:44:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 -- 172.21.15.158:0/4041044431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06ab40 msgr2=0x7f1eec06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 --2- 172.21.15.158:0/4041044431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06ab40 0x7f1eec06d820 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1eec06eaf0 tx=0x7f1edc009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 -- 172.21.15.158:0/4041044431 shutdown_connections 2023-10-16T09:44:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 --2- 172.21.15.158:0/4041044431 >> v2:172.21.15.172:3300/0 conn(0x7f1eec06af20 0x7f1eec0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 --2- 172.21.15.158:0/4041044431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06ab40 0x7f1eec06d820 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 --2- 172.21.15.158:0/4041044431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06e5d0 0x7f1eec071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 -- 172.21.15.158:0/4041044431 >> 172.21.15.158:0/4041044431 conn(0x7f1eec0d02b0 msgr2=0x7f1eec0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 -- 172.21.15.158:0/4041044431 shutdown_connections 2023-10-16T09:44:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.551+0000 7f1ef122a700 1 -- 172.21.15.158:0/4041044431 wait complete. 2023-10-16T09:44:26.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.552+0000 7f1ef122a700 1 Processor -- start 2023-10-16T09:44:26.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.552+0000 7f1ef122a700 1 -- start start 2023-10-16T09:44:26.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1ef122a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06ab40 0x7f1eec0b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1ef122a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06af20 0x7f1eec0c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1ef122a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eec06e5d0 0x7f1eec0c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1ef122a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0c7380 con 0x7f1eec06af20 2023-10-16T09:44:26.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1ef122a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec1f66c0 con 0x7f1eec06e5d0 2023-10-16T09:44:26.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1ef122a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec1f69a0 con 0x7f1eec06ab40 2023-10-16T09:44:26.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1eea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06af20 0x7f1eec0c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1eea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06af20 0x7f1eec0c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48028/0 (socket says 172.21.15.158:48028) 2023-10-16T09:44:26.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.553+0000 7f1eea59c700 1 -- 172.21.15.158:0/2674384097 learned_addr learned my addr 172.21.15.158:0/2674384097 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:26.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eead9d700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06ab40 0x7f1eec0b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eeb59e700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eec06e5d0 0x7f1eec0c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eea59c700 1 -- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06ab40 msgr2=0x7f1eec0b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eea59c700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06ab40 0x7f1eec0b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eea59c700 1 -- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eec06e5d0 msgr2=0x7f1eec0c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eea59c700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eec06e5d0 0x7f1eec0c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eea59c700 1 -- 172.21.15.158:0/2674384097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc009550 con 0x7f1eec06af20 2023-10-16T09:44:26.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eeb59e700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eec06e5d0 0x7f1eec0c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:26.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1eea59c700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06af20 0x7f1eec0c9120 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1ee000e870 tx=0x7f1ee000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1ed37fe700 1 -- 172.21.15.158:0/2674384097 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee000c640 con 0x7f1eec06af20 2023-10-16T09:44:26.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1ed37fe700 1 -- 172.21.15.158:0/2674384097 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee000c7a0 con 0x7f1eec06af20 2023-10-16T09:44:26.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec1f6c80 con 0x7f1eec06af20 2023-10-16T09:44:26.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1ed37fe700 1 -- 172.21.15.158:0/2674384097 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee000cb10 con 0x7f1eec06af20 2023-10-16T09:44:26.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.554+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec0b1f40 con 0x7f1eec06af20 2023-10-16T09:44:26.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.555+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec0c7660 con 0x7f1eec06af20 2023-10-16T09:44:26.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.555+0000 7f1ed37fe700 1 -- 172.21.15.158:0/2674384097 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1ee000cc70 con 0x7f1eec06af20 2023-10-16T09:44:26.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.556+0000 7f1ed37fe700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1ed806af00 0x7f1ed806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.556+0000 7f1eead9d700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1ed806af00 0x7f1ed806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.556+0000 7f1ed37fe700 1 -- 172.21.15.158:0/2674384097 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f1ee0089880 con 0x7f1eec06af20 2023-10-16T09:44:26.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.556+0000 7f1eead9d700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1ed806af00 0x7f1ed806d3c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009c10 tx=0x7f1edc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.560+0000 7f1ed37fe700 1 -- 172.21.15.158:0/2674384097 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ee00545b0 con 0x7f1eec06af20 2023-10-16T09:44:26.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.796+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eec0681b0 con 0x7f1eec06af20 2023-10-16T09:44:26.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.797+0000 7f1ed37fe700 1 -- 172.21.15.158:0/2674384097 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7f1ee0057bd0 con 0x7f1eec06af20 2023-10-16T09:44:26.799 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:26.799 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:44:26.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1ed806af00 msgr2=0x7f1ed806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1ed806af00 0x7f1ed806d3c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009c10 tx=0x7f1edc008030 comp rx=0 tx=0).stop 2023-10-16T09:44:26.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06af20 msgr2=0x7f1eec0c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06af20 0x7f1eec0c9120 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1ee000e870 tx=0x7f1ee000ec30 comp rx=0 tx=0).stop 2023-10-16T09:44:26.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 shutdown_connections 2023-10-16T09:44:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eec06e5d0 0x7f1eec0c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f1ed806af00 0x7f1ed806d3c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eec06ab40 0x7f1eec0b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 --2- 172.21.15.158:0/2674384097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eec06af20 0x7f1eec0c9120 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 >> 172.21.15.158:0/2674384097 conn(0x7f1eec0d02b0 msgr2=0x7f1eec072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 shutdown_connections 2023-10-16T09:44:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.799+0000 7f1ef122a700 1 -- 172.21.15.158:0/2674384097 wait complete. 2023-10-16T09:44:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:44:26.813 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:26.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.932+0000 7ff36b8db700 1 Processor -- start 2023-10-16T09:44:26.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.932+0000 7ff36b8db700 1 -- start start 2023-10-16T09:44:26.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff36b8db700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff36406ab40 0x7ff36406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff36b8db700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff3640729b0 0x7ff364071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff36b8db700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff364071e50 con 0x7ff36406e5d0 2023-10-16T09:44:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff36b8db700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff364161270 con 0x7ff36406ab40 2023-10-16T09:44:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff36b8db700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff364161510 con 0x7ff3640729b0 2023-10-16T09:44:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff369677700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff36406ab40 0x7ff36406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff368e76700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff36406e5d0 0x7ff3640722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48210/0 (socket says 172.21.15.158:48210) 2023-10-16T09:44:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff368e76700 1 -- 172.21.15.158:0/2565937794 learned_addr learned my addr 172.21.15.158:0/2565937794 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff369e78700 1 --2- 172.21.15.158:0/2565937794 >> v2:172.21.15.172:3300/0 conn(0x7ff3640729b0 0x7ff364071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff369677700 1 -- 172.21.15.158:0/2565937794 >> v2:172.21.15.172:3300/0 conn(0x7ff3640729b0 msgr2=0x7ff364071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff369677700 1 --2- 172.21.15.158:0/2565937794 >> v2:172.21.15.172:3300/0 conn(0x7ff3640729b0 0x7ff364071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.933+0000 7ff35bfff700 1 -- 172.21.15.158:0/2565937794 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662000336 0 0) 0x7ff364071e50 con 0x7ff36406e5d0 2023-10-16T09:44:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff369677700 1 -- 172.21.15.158:0/2565937794 >> v1:172.21.15.158:6789/0 conn(0x7ff36406e5d0 legacy=0x7ff3640722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff369677700 1 -- 172.21.15.158:0/2565937794 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3641617f0 con 0x7ff36406ab40 2023-10-16T09:44:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff369677700 1 --2- 172.21.15.158:0/2565937794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff36406ab40 0x7ff36406d820 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff364158240 tx=0x7ff354009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff6102727664af3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff35bfff700 1 -- 172.21.15.158:0/2565937794 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354011830 con 0x7ff36406ab40 2023-10-16T09:44:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff35bfff700 1 -- 172.21.15.158:0/2565937794 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354011990 con 0x7ff36406ab40 2023-10-16T09:44:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff35bfff700 1 -- 172.21.15.158:0/2565937794 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354011d20 con 0x7ff36406ab40 2023-10-16T09:44:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 -- 172.21.15.158:0/2565937794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff36406ab40 msgr2=0x7ff36406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2565937794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff36406ab40 0x7ff36406d820 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff364158240 tx=0x7ff354009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 -- 172.21.15.158:0/2565937794 shutdown_connections 2023-10-16T09:44:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2565937794 >> v2:172.21.15.172:3300/0 conn(0x7ff3640729b0 0x7ff364071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2565937794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff36406ab40 0x7ff36406d820 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 -- 172.21.15.158:0/2565937794 >> 172.21.15.158:0/2565937794 conn(0x7ff3640d02b0 msgr2=0x7ff3640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:26.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 -- 172.21.15.158:0/2565937794 shutdown_connections 2023-10-16T09:44:26.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.934+0000 7ff36b8db700 1 -- 172.21.15.158:0/2565937794 wait complete. 2023-10-16T09:44:26.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 Processor -- start 2023-10-16T09:44:26.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 -- start start 2023-10-16T09:44:26.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff36406ab40 0x7ff36414c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff36406e5d0 0x7ff36415ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3640729b0 0x7ff3640c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3640b3960 con 0x7ff3640729b0 2023-10-16T09:44:26.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3640b3c40 con 0x7ff36406ab40 2023-10-16T09:44:26.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.936+0000 7ff36b8db700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3640b3f20 con 0x7ff36406e5d0 2023-10-16T09:44:26.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3640729b0 0x7ff3640c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3640729b0 0x7ff3640c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48042/0 (socket says 172.21.15.158:48042) 2023-10-16T09:44:26.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 -- 172.21.15.158:0/2745655644 learned_addr learned my addr 172.21.15.158:0/2745655644 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:26.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 -- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff36406e5d0 msgr2=0x7ff36415ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369677700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff36406ab40 0x7ff36414c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff368e76700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff36406e5d0 0x7ff36415ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff36406e5d0 0x7ff36415ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 -- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff36406ab40 msgr2=0x7ff36414c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff36406ab40 0x7ff36414c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 -- 172.21.15.158:0/2745655644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3540094a0 con 0x7ff3640729b0 2023-10-16T09:44:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff369e78700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3640729b0 0x7ff3640c3c40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00efc0 tx=0x7ff35c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff359ffb700 1 -- 172.21.15.158:0/2745655644 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c009ca0 con 0x7ff3640729b0 2023-10-16T09:44:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff359ffb700 1 -- 172.21.15.158:0/2745655644 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c00f040 con 0x7ff3640729b0 2023-10-16T09:44:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3640b4200 con 0x7ff3640729b0 2023-10-16T09:44:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff359ffb700 1 -- 172.21.15.158:0/2745655644 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c014480 con 0x7ff3640729b0 2023-10-16T09:44:26.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.937+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3640b3550 con 0x7ff3640729b0 2023-10-16T09:44:26.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.938+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3640681b0 con 0x7ff3640729b0 2023-10-16T09:44:26.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.939+0000 7ff359ffb700 1 -- 172.21.15.158:0/2745655644 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff35c0145e0 con 0x7ff3640729b0 2023-10-16T09:44:26.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.939+0000 7ff359ffb700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff34806af00 0x7ff34806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:26.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.939+0000 7ff359ffb700 1 -- 172.21.15.158:0/2745655644 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7ff35c08a690 con 0x7ff3640729b0 2023-10-16T09:44:26.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.939+0000 7ff369677700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff34806af00 0x7ff34806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:26.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.940+0000 7ff369677700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff34806af00 0x7ff34806d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff3540090b0 tx=0x7ff354008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:26.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:26.944+0000 7ff359ffb700 1 -- 172.21.15.158:0/2745655644 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff35c0553c0 con 0x7ff3640729b0 2023-10-16T09:44:27.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.138+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3640b6230 con 0x7ff3640729b0 2023-10-16T09:44:27.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.139+0000 7ff359ffb700 1 -- 172.21.15.158:0/2745655644 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7ff35c0589e0 con 0x7ff3640729b0 2023-10-16T09:44:27.141 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:27.142 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff34806af00 msgr2=0x7ff34806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff34806af00 0x7ff34806d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff3540090b0 tx=0x7ff354008030 comp rx=0 tx=0).stop 2023-10-16T09:44:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3640729b0 msgr2=0x7ff3640c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3640729b0 0x7ff3640c3c40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00efc0 tx=0x7ff35c00c3a0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 shutdown_connections 2023-10-16T09:44:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3640729b0 0x7ff3640c3c40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff36406ab40 0x7ff36414c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff34806af00 0x7ff34806d3c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 --2- 172.21.15.158:0/2745655644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff36406e5d0 0x7ff36415ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 >> 172.21.15.158:0/2745655644 conn(0x7ff3640d02b0 msgr2=0x7ff3640adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:27.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 shutdown_connections 2023-10-16T09:44:27.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.142+0000 7ff36b8db700 1 -- 172.21.15.158:0/2745655644 wait complete. 2023-10-16T09:44:27.156 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T09:44:27.157 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:44:27.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.276+0000 7f4fb9f19700 1 Processor -- start 2023-10-16T09:44:27.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.276+0000 7f4fb9f19700 1 -- start start 2023-10-16T09:44:27.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb9f19700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4fb406d820 0x7f4fb415f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb9f19700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb4072700 con 0x7f4fb406b0a0 2023-10-16T09:44:27.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb9f19700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb40cca90 con 0x7f4fb40cb010 2023-10-16T09:44:27.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb9f19700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb4071910 con 0x7f4fb406d820 2023-10-16T09:44:27.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4fb406d820 0x7f4fb415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb37fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4fb406b0a0 0x7f4fb40ca900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48222/0 (socket says 172.21.15.158:48222) 2023-10-16T09:44:27.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb37fe700 1 -- 172.21.15.158:0/1778381072 learned_addr learned my addr 172.21.15.158:0/1778381072 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:27.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb27fc700 1 -- 172.21.15.158:0/1778381072 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017700913 0 0) 0x7f4fb4072700 con 0x7f4fb406b0a0 2023-10-16T09:44:27.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb27fc700 1 -- 172.21.15.158:0/1778381072 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa00035b0 con 0x7f4fb406b0a0 2023-10-16T09:44:27.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb3fff700 1 -- 172.21.15.158:0/1778381072 >> v1:172.21.15.172:6789/0 conn(0x7f4fb40cb010 legacy=0x7f4fb415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.277+0000 7f4fb3fff700 1 -- 172.21.15.158:0/1778381072 >> v1:172.21.15.158:6789/0 conn(0x7f4fb406b0a0 legacy=0x7f4fb40ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb3fff700 1 -- 172.21.15.158:0/1778381072 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb4071af0 con 0x7f4fb406d820 2023-10-16T09:44:27.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb3fff700 1 --2- 172.21.15.158:0/1778381072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415f010 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009d90 tx=0x7f4fac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d329e98cfca412d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:27.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb27fc700 1 -- 172.21.15.158:0/1778381072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00e070 con 0x7f4fb406d820 2023-10-16T09:44:27.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb27fc700 1 -- 172.21.15.158:0/1778381072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac00bb20 con 0x7f4fb406d820 2023-10-16T09:44:27.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb27fc700 1 -- 172.21.15.158:0/1778381072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac012430 con 0x7f4fb406d820 2023-10-16T09:44:27.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb9f19700 1 -- 172.21.15.158:0/1778381072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 msgr2=0x7f4fb415f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/1778381072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415f010 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009d90 tx=0x7f4fac00b330 comp rx=0 tx=0).stop 2023-10-16T09:44:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb9f19700 1 -- 172.21.15.158:0/1778381072 shutdown_connections 2023-10-16T09:44:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/1778381072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415f010 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb9f19700 1 -- 172.21.15.158:0/1778381072 >> 172.21.15.158:0/1778381072 conn(0x7f4fb40d02b0 msgr2=0x7f4fb40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb9f19700 1 -- 172.21.15.158:0/1778381072 shutdown_connections 2023-10-16T09:44:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.278+0000 7f4fb9f19700 1 -- 172.21.15.158:0/1778381072 wait complete. 2023-10-16T09:44:27.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.280+0000 7f4fb9f19700 1 Processor -- start 2023-10-16T09:44:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.280+0000 7f4fb9f19700 1 -- start start 2023-10-16T09:44:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.280+0000 7f4fb9f19700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fb406b0a0 0x7f4fb41ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.280+0000 7f4fb9f19700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb9f19700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fb40cb010 0x7f4fb41f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb9f19700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb415e960 con 0x7f4fb406d820 2023-10-16T09:44:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb9f19700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb415daa0 con 0x7f4fb406b0a0 2023-10-16T09:44:27.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb9f19700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40c7810 con 0x7f4fb40cb010 2023-10-16T09:44:27.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48052/0 (socket says 172.21.15.158:48052) 2023-10-16T09:44:27.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 -- 172.21.15.158:0/2969750725 learned_addr learned my addr 172.21.15.158:0/2969750725 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:27.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb3fff700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fb40cb010 0x7f4fb41f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 -- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fb40cb010 msgr2=0x7f4fb41f2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fb40cb010 0x7f4fb41f2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb37fe700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fb406b0a0 0x7f4fb41ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 -- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fb406b0a0 msgr2=0x7f4fb41ee4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fb406b0a0 0x7f4fb41ee4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 -- 172.21.15.158:0/2969750725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac009a70 con 0x7f4fb406d820 2023-10-16T09:44:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb2ffd700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415d560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c4e0 tx=0x7f4fa400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fabfff700 1 -- 172.21.15.158:0/2969750725 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40052d0 con 0x7f4fb406d820 2023-10-16T09:44:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fabfff700 1 -- 172.21.15.158:0/2969750725 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa4005430 con 0x7f4fb406d820 2023-10-16T09:44:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fabfff700 1 -- 172.21.15.158:0/2969750725 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40038c0 con 0x7f4fb406d820 2023-10-16T09:44:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb40c7af0 con 0x7f4fb406d820 2023-10-16T09:44:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.281+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb40c80b0 con 0x7f4fb406d820 2023-10-16T09:44:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.282+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb4074b40 con 0x7f4fb406d820 2023-10-16T09:44:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.283+0000 7f4fabfff700 1 -- 172.21.15.158:0/2969750725 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4fa4003a20 con 0x7f4fb406d820 2023-10-16T09:44:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.283+0000 7f4fabfff700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f4f9406af00 0x7f4f9406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.283+0000 7f4fabfff700 1 -- 172.21.15.158:0/2969750725 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f4fa4088e30 con 0x7f4fb406d820 2023-10-16T09:44:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.283+0000 7f4fb37fe700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f4f9406af00 0x7f4f9406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.283+0000 7f4fb37fe700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f4f9406af00 0x7f4f9406d3c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0011e0 tx=0x7f4f9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.287+0000 7f4fabfff700 1 -- 172.21.15.158:0/2969750725 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4fa4053b60 con 0x7f4fb406d820 2023-10-16T09:44:27.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.523+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fb40681b0 con 0x7f4fb406d820 2023-10-16T09:44:27.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.524+0000 7f4fabfff700 1 -- 172.21.15.158:0/2969750725 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7f4fa4017090 con 0x7f4fb406d820 2023-10-16T09:44:27.526 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:27.526 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:44:27.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f4f9406af00 msgr2=0x7f4f9406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f4f9406af00 0x7f4f9406d3c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0011e0 tx=0x7f4f9c009040 comp rx=0 tx=0).stop 2023-10-16T09:44:27.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 msgr2=0x7f4fb415d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415d560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c4e0 tx=0x7f4fa400c8a0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 shutdown_connections 2023-10-16T09:44:27.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f4f9406af00 0x7f4f9406d3c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fb40cb010 0x7f4fb41f2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fb406b0a0 0x7f4fb41ee4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 --2- 172.21.15.158:0/2969750725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fb406d820 0x7f4fb415d560 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 >> 172.21.15.158:0/2969750725 conn(0x7f4fb40d02b0 msgr2=0x7f4fb415bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:27.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 shutdown_connections 2023-10-16T09:44:27.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.526+0000 7f4fb9f19700 1 -- 172.21.15.158:0/2969750725 wait complete. 2023-10-16T09:44:27.530 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:44:27.540 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:44:27.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.659+0000 7fa351c5c700 1 Processor -- start 2023-10-16T09:44:27.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.660+0000 7fa351c5c700 1 -- start start 2023-10-16T09:44:27.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.660+0000 7fa351c5c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c0d0bd0 0x7fa34c0d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.660+0000 7fa351c5c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa34c0d3570 0x7fa34c0d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.660+0000 7fa351c5c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0ce4d0 con 0x7fa34c15daa0 2023-10-16T09:44:27.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.660+0000 7fa351c5c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34c0d3f00 con 0x7fa34c0d3570 2023-10-16T09:44:27.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.660+0000 7fa351c5c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c06c1b0 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa34c0d3570 0x7fa34c0d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c0d0bd0 0x7fa34c0d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa34c15daa0 0x7fa34c06b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:52138/0 (socket says 172.21.15.158:52138) 2023-10-16T09:44:27.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34b7fe700 1 -- 172.21.15.158:0/928443403 learned_addr learned my addr 172.21.15.158:0/928443403 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:27.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34a7fc700 1 -- 172.21.15.158:0/928443403 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407223689 0 0) 0x7fa34c0ce4d0 con 0x7fa34c15daa0 2023-10-16T09:44:27.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34a7fc700 1 -- 172.21.15.158:0/928443403 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3380035b0 con 0x7fa34c15daa0 2023-10-16T09:44:27.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34affd700 1 -- 172.21.15.158:0/928443403 >> v2:172.21.15.172:3300/0 conn(0x7fa34c0d3570 msgr2=0x7fa34c0d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:27.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34affd700 1 --2- 172.21.15.158:0/928443403 >> v2:172.21.15.172:3300/0 conn(0x7fa34c0d3570 0x7fa34c0d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34affd700 1 -- 172.21.15.158:0/928443403 >> v1:172.21.15.172:6789/0 conn(0x7fa34c15daa0 legacy=0x7fa34c06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34affd700 1 -- 172.21.15.158:0/928443403 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c06c490 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34bfff700 1 --2- 172.21.15.158:0/928443403 >> v2:172.21.15.172:3300/0 conn(0x7fa34c0d3570 0x7fa34c0d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:27.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.661+0000 7fa34affd700 1 --2- 172.21.15.158:0/928443403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c0d0bd0 0x7fa34c0d1510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa334009840 tx=0x7fa334009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=850c64c44d27c0ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:27.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa34a7fc700 1 -- 172.21.15.158:0/928443403 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334011830 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa34a7fc700 1 -- 172.21.15.158:0/928443403 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334011990 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa34a7fc700 1 -- 172.21.15.158:0/928443403 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334011d20 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 -- 172.21.15.158:0/928443403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c0d0bd0 msgr2=0x7fa34c0d1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 --2- 172.21.15.158:0/928443403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c0d0bd0 0x7fa34c0d1510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa334009840 tx=0x7fa334009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:27.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 -- 172.21.15.158:0/928443403 shutdown_connections 2023-10-16T09:44:27.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 --2- 172.21.15.158:0/928443403 >> v2:172.21.15.172:3300/0 conn(0x7fa34c0d3570 0x7fa34c0d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 --2- 172.21.15.158:0/928443403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c0d0bd0 0x7fa34c0d1510 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 -- 172.21.15.158:0/928443403 >> 172.21.15.158:0/928443403 conn(0x7fa34c08d040 msgr2=0x7fa34c09c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:27.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 -- 172.21.15.158:0/928443403 shutdown_connections 2023-10-16T09:44:27.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.662+0000 7fa351c5c700 1 -- 172.21.15.158:0/928443403 wait complete. 2023-10-16T09:44:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.663+0000 7fa351c5c700 1 Processor -- start 2023-10-16T09:44:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.664+0000 7fa351c5c700 1 -- start start 2023-10-16T09:44:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.664+0000 7fa351c5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa34c0d0bd0 0x7fa34c15f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.664+0000 7fa351c5c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa34c0d3570 0x7fa34c152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.664+0000 7fa351c5c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c15daa0 0x7fa34c0d1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.664+0000 7fa351c5c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c15fa40 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.664+0000 7fa351c5c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c15fc90 con 0x7fa34c0d3570 2023-10-16T09:44:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.664+0000 7fa351c5c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c15ff70 con 0x7fa34c15daa0 2023-10-16T09:44:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa34c0d0bd0 0x7fa34c15f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa34c0d0bd0 0x7fa34c15f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48066/0 (socket says 172.21.15.158:48066) 2023-10-16T09:44:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 -- 172.21.15.158:0/3679197580 learned_addr learned my addr 172.21.15.158:0/3679197580 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34affd700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa34c0d3570 0x7fa34c152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34bfff700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c15daa0 0x7fa34c0d1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 -- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c15daa0 msgr2=0x7fa34c0d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c15daa0 0x7fa34c0d1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 -- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa34c0d3570 msgr2=0x7fa34c152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa34c0d3570 0x7fa34c152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 -- 172.21.15.158:0/3679197580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa344009a70 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34affd700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa34c0d3570 0x7fa34c152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa34b7fe700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa34c0d0bd0 0x7fa34c15f500 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa33c001130 tx=0x7fa33c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa343fff700 1 -- 172.21.15.158:0/3679197580 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c005dc0 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa343fff700 1 -- 172.21.15.158:0/3679197580 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c0070d0 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3340094a0 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa343fff700 1 -- 172.21.15.158:0/3679197580 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c007460 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.665+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c160530 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.666+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c0681b0 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.666+0000 7fa343fff700 1 -- 172.21.15.158:0/3679197580 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa33c0075c0 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.667+0000 7fa343fff700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa32c06af00 0x7fa32c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.667+0000 7fa343fff700 1 -- 172.21.15.158:0/3679197580 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fa33c088f40 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.667+0000 7fa34affd700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa32c06af00 0x7fa32c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.667+0000 7fa34affd700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa32c06af00 0x7fa32c06d3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa3340090b0 tx=0x7fa334008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.671+0000 7fa343fff700 1 -- 172.21.15.158:0/3679197580 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa33c053c70 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.905+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa34c0d4450 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.906+0000 7fa343fff700 1 -- 172.21.15.158:0/3679197580 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4921 (secure 0 0 0) 0x7fa33c017020 con 0x7fa34c0d0bd0 2023-10-16T09:44:27.908 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:27.908 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3981192744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3981192744},{"type":"v1","addr":"172.21.15.158:6836","nonce":3981192744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1171129002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1171129002},{"type":"v1","addr":"172.21.15.158:6837","nonce":1171129002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2891125818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2891125818},{"type":"v1","addr":"172.21.15.172:6834","nonce":2891125818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:43:21.310458+0000","modified":"2023-10-16T09:43:43.765523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/3958845884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3958845884},{"type":"v1","addr":"172.21.15.172:6835","nonce":3958845884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T09:44:27.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa32c06af00 msgr2=0x7fa32c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa32c06af00 0x7fa32c06d3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa3340090b0 tx=0x7fa334008030 comp rx=0 tx=0).stop 2023-10-16T09:44:27.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa34c0d0bd0 msgr2=0x7fa34c15f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:27.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa34c0d0bd0 0x7fa34c15f500 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa33c001130 tx=0x7fa33c00ca90 comp rx=0 tx=0).stop 2023-10-16T09:44:27.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 shutdown_connections 2023-10-16T09:44:27.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa32c06af00 0x7fa32c06d3c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa34c15daa0 0x7fa34c0d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa34c0d0bd0 0x7fa34c15f500 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 --2- 172.21.15.158:0/3679197580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa34c0d3570 0x7fa34c152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 >> 172.21.15.158:0/3679197580 conn(0x7fa34c08d040 msgr2=0x7fa34c0b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 shutdown_connections 2023-10-16T09:44:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:27.908+0000 7fa351c5c700 1 -- 172.21.15.158:0/3679197580 wait complete. 2023-10-16T09:44:27.913 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 9 2023-10-16T09:44:27.923 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:28.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.042+0000 7f3c86db8700 1 Processor -- start 2023-10-16T09:44:28.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.042+0000 7f3c86db8700 1 -- start start 2023-10-16T09:44:28.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c86db8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c8015dbc0 0x7f3c800cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c86db8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c8006ba90 0x7f3c800cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c86db8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800d21e0 con 0x7f3c800d1d50 2023-10-16T09:44:28.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c86db8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c80162550 con 0x7f3c8015dbc0 2023-10-16T09:44:28.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c86db8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015f320 con 0x7f3c8006ba90 2023-10-16T09:44:28.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c85355700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c8006ba90 0x7f3c800cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c84b54700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c8015dbc0 0x7f3c800cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c85355700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c8006ba90 0x7f3c800cf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:58910/0 (socket says 172.21.15.158:58910) 2023-10-16T09:44:28.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c85355700 1 -- 172.21.15.158:0/1220170262 learned_addr learned my addr 172.21.15.158:0/1220170262 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:28.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c7ffff700 1 --1- 172.21.15.158:0/1220170262 >> v1:172.21.15.172:6789/0 conn(0x7f3c800d1d50 0x7f3c8006beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:52154/0 (socket says 172.21.15.158:52154) 2023-10-16T09:44:28.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c85355700 1 -- 172.21.15.158:0/1220170262 >> v2:172.21.15.172:3300/0 conn(0x7f3c8015dbc0 msgr2=0x7f3c800cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:28.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c85355700 1 --2- 172.21.15.158:0/1220170262 >> v2:172.21.15.172:3300/0 conn(0x7f3c8015dbc0 0x7f3c800cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c85355700 1 -- 172.21.15.158:0/1220170262 >> v1:172.21.15.172:6789/0 conn(0x7f3c800d1d50 legacy=0x7f3c8006beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.043+0000 7f3c85355700 1 -- 172.21.15.158:0/1220170262 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8015ef10 con 0x7f3c8006ba90 2023-10-16T09:44:28.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c85355700 1 --2- 172.21.15.158:0/1220170262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c8006ba90 0x7f3c800cf4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e50 tx=0x7f3c7800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76558662f87f433e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c7f7fe700 1 -- 172.21.15.158:0/1220170262 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800e070 con 0x7f3c8006ba90 2023-10-16T09:44:28.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c7f7fe700 1 -- 172.21.15.158:0/1220170262 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7800b910 con 0x7f3c8006ba90 2023-10-16T09:44:28.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c7f7fe700 1 -- 172.21.15.158:0/1220170262 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800bcc0 con 0x7f3c8006ba90 2023-10-16T09:44:28.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 -- 172.21.15.158:0/1220170262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c8006ba90 msgr2=0x7f3c800cf4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1220170262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c8006ba90 0x7f3c800cf4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e50 tx=0x7f3c7800b330 comp rx=0 tx=0).stop 2023-10-16T09:44:28.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 -- 172.21.15.158:0/1220170262 shutdown_connections 2023-10-16T09:44:28.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1220170262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c8006ba90 0x7f3c800cf4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1220170262 >> v2:172.21.15.172:3300/0 conn(0x7f3c8015dbc0 0x7f3c800cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 -- 172.21.15.158:0/1220170262 >> 172.21.15.158:0/1220170262 conn(0x7f3c8006a8d0 msgr2=0x7f3c8006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:28.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 -- 172.21.15.158:0/1220170262 shutdown_connections 2023-10-16T09:44:28.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.044+0000 7f3c86db8700 1 -- 172.21.15.158:0/1220170262 wait complete. 2023-10-16T09:44:28.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 Processor -- start 2023-10-16T09:44:28.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 -- start start 2023-10-16T09:44:28.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c8006ba90 0x7f3c8014f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c80155150 0x7f3c8015ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c801555d0 0x7f3c8015cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015c230 con 0x7f3c801555d0 2023-10-16T09:44:28.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015b620 con 0x7f3c8006ba90 2023-10-16T09:44:28.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.046+0000 7f3c86db8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015b900 con 0x7f3c80155150 2023-10-16T09:44:28.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c801555d0 0x7f3c8015cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c801555d0 0x7f3c8015cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48082/0 (socket says 172.21.15.158:48082) 2023-10-16T09:44:28.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 -- 172.21.15.158:0/1469935689 learned_addr learned my addr 172.21.15.158:0/1469935689 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:28.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c84b54700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c8006ba90 0x7f3c8014f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c7ffff700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c80155150 0x7f3c8015ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 -- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c80155150 msgr2=0x7f3c8015ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c80155150 0x7f3c8015ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 -- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c8006ba90 msgr2=0x7f3c8014f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c8006ba90 0x7f3c8014f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 -- 172.21.15.158:0/1469935689 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78009b30 con 0x7f3c801555d0 2023-10-16T09:44:28.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c85355700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c801555d0 0x7f3c8015cfd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c78012000 tx=0x7f3c78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c7d7fa700 1 -- 172.21.15.158:0/1469935689 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78004800 con 0x7f3c801555d0 2023-10-16T09:44:28.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c7d7fa700 1 -- 172.21.15.158:0/1469935689 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c78004960 con 0x7f3c801555d0 2023-10-16T09:44:28.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c7d7fa700 1 -- 172.21.15.158:0/1469935689 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7801c680 con 0x7f3c801555d0 2023-10-16T09:44:28.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8015bbe0 con 0x7f3c801555d0 2023-10-16T09:44:28.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.047+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8015b1b0 con 0x7f3c801555d0 2023-10-16T09:44:28.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.048+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8015b370 con 0x7f3c801555d0 2023-10-16T09:44:28.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.049+0000 7f3c7d7fa700 1 -- 172.21.15.158:0/1469935689 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c7801c7e0 con 0x7f3c801555d0 2023-10-16T09:44:28.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.049+0000 7f3c7d7fa700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c6406af00 0x7f3c6406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.049+0000 7f3c84b54700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c6406af00 0x7f3c6406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.049+0000 7f3c7d7fa700 1 -- 172.21.15.158:0/1469935689 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f3c7808b540 con 0x7f3c801555d0 2023-10-16T09:44:28.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.050+0000 7f3c84b54700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c6406af00 0x7f3c6406d3c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c8014ff70 tx=0x7f3c74009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.054+0000 7f3c7d7fa700 1 -- 172.21.15.158:0/1469935689 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c78056270 con 0x7f3c801555d0 2023-10-16T09:44:28.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.247+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c800681b0 con 0x7f3c801555d0 2023-10-16T09:44:28.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.248+0000 7f3c7d7fa700 1 -- 172.21.15.158:0/1469935689 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18831 (secure 0 0 0) 0x7f3c78059890 con 0x7f3c801555d0 2023-10-16T09:44:28.250 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:28.250 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":26,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:10.487901+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:43:19.832012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T09:43:20.266773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T09:43:23.948527+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:28.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c6406af00 msgr2=0x7f3c6406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c6406af00 0x7f3c6406d3c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c8014ff70 tx=0x7f3c74009210 comp rx=0 tx=0).stop 2023-10-16T09:44:28.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c801555d0 msgr2=0x7f3c8015cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c801555d0 0x7f3c8015cfd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c78012000 tx=0x7f3c78004ad0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 shutdown_connections 2023-10-16T09:44:28.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c801555d0 0x7f3c8015cfd0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c8006ba90 0x7f3c8014f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f3c6406af00 0x7f3c6406d3c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 --2- 172.21.15.158:0/1469935689 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c80155150 0x7f3c8015ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 >> 172.21.15.158:0/1469935689 conn(0x7f3c8006a8d0 msgr2=0x7f3c8006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:28.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 shutdown_connections 2023-10-16T09:44:28.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.250+0000 7f3c86db8700 1 -- 172.21.15.158:0/1469935689 wait complete. 2023-10-16T09:44:28.265 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T09:44:28.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.386+0000 7efe6ccc7700 1 Processor -- start 2023-10-16T09:44:28.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.386+0000 7efe6ccc7700 1 -- start start 2023-10-16T09:44:28.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6ccc7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 0x7efe680b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6ccc7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe680af600 con 0x7efe6815d990 2023-10-16T09:44:28.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6ccc7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6806e9b0 con 0x7efe6806e480 2023-10-16T09:44:28.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6ccc7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe680b2c90 con 0x7efe680cfd60 2023-10-16T09:44:28.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6659c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efe6815d990 0x7efe680b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48262/0 (socket says 172.21.15.158:48262) 2023-10-16T09:44:28.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6659c700 1 -- 172.21.15.158:0/3318111489 learned_addr learned my addr 172.21.15.158:0/3318111489 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:28.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 0x7efe680b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481675035 0 0) 0x7efe680af600 con 0x7efe6815d990 2023-10-16T09:44:28.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe500035b0 con 0x7efe6815d990 2023-10-16T09:44:28.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924865186 0 0) 0x7efe6806e9b0 con 0x7efe6806e480 2023-10-16T09:44:28.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe680af600 con 0x7efe6806e480 2023-10-16T09:44:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986035647 0 0) 0x7efe500035b0 con 0x7efe6815d990 2023-10-16T09:44:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 msgr2=0x7efe680b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 0x7efe680b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 >> v1:172.21.15.172:6789/0 conn(0x7efe6806e480 legacy=0x7efe680cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680ce890 con 0x7efe6815d990 2023-10-16T09:44:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7efe58001670 con 0x7efe6815d990 2023-10-16T09:44:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 >> v1:172.21.15.158:6789/0 conn(0x7efe6815d990 legacy=0x7efe680b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.387+0000 7efe6559a700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 0x7efe50003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe50004120 0x7efe50004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe50004d90 0x7efe50005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680b2c90 con 0x7efe50004d90 2023-10-16T09:44:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5000baf0 con 0x7efe50004120 2023-10-16T09:44:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5000bcb0 con 0x7efe680cfd60 2023-10-16T09:44:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe50004d90 0x7efe50005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6659c700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 0x7efe50003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe65d9b700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe50004120 0x7efe50004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe66d9d700 1 -- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 msgr2=0x7efe50003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe680cfd60 0x7efe50003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe66d9d700 1 -- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe50004120 msgr2=0x7efe50004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe50004120 0x7efe50004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe66d9d700 1 -- 172.21.15.158:0/3318111489 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe500035b0 con 0x7efe50004d90 2023-10-16T09:44:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe50004d90 0x7efe50005210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efe680b2ee0 tx=0x7efe50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4c69c0dc34b585 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6000ebd0 con 0x7efe50004d90 2023-10-16T09:44:28.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6000f040 con 0x7efe50004d90 2023-10-16T09:44:28.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.388+0000 7efe6559a700 1 -- 172.21.15.158:0/3318111489 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60014480 con 0x7efe50004d90 2023-10-16T09:44:28.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.389+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3318111489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe50004d90 msgr2=0x7efe50005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.389+0000 7efe6ccc7700 1 --2- 172.21.15.158:0/3318111489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe50004d90 0x7efe50005210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efe680b2ee0 tx=0x7efe50005880 comp rx=0 tx=0).stop 2023-10-16T09:44:28.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.389+0000 7efe66d9d700 1 -- 172.21.15.158:0/3318111489 reap_dead start 2023-10-16T09:44:28.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.389+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3318111489 shutdown_connections 2023-10-16T09:44:28.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.389+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3318111489 >> 172.21.15.158:0/3318111489 conn(0x7efe6806b9e0 msgr2=0x7efe6806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:28.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.389+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3318111489 shutdown_connections 2023-10-16T09:44:28.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.389+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3318111489 wait complete. 2023-10-16T09:44:28.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.390+0000 7efe6ccc7700 1 Processor -- start 2023-10-16T09:44:28.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.390+0000 7efe6ccc7700 1 -- start start 2023-10-16T09:44:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6ccc7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe50004d90 0x7efe680cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6ccc7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe680cbba0 0x7efe68152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6ccc7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe680b38b0 0x7efe680b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6ccc7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680c99a0 con 0x7efe680b38b0 2023-10-16T09:44:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6ccc7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680b4270 con 0x7efe680cbba0 2023-10-16T09:44:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6ccc7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe68152f50 con 0x7efe50004d90 2023-10-16T09:44:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe680b38b0 0x7efe680b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe680b38b0 0x7efe680b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48102/0 (socket says 172.21.15.158:48102) 2023-10-16T09:44:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 -- 172.21.15.158:0/3131182531 learned_addr learned my addr 172.21.15.158:0/3131182531 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe65d9b700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe680cbba0 0x7efe68152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6659c700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe50004d90 0x7efe680cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 -- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe50004d90 msgr2=0x7efe680cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe50004d90 0x7efe680cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 -- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe680cbba0 msgr2=0x7efe68152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe680cbba0 0x7efe68152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 -- 172.21.15.158:0/3131182531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680cc2f0 con 0x7efe680b38b0 2023-10-16T09:44:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe65d9b700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe680cbba0 0x7efe68152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe6659c700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe50004d90 0x7efe680cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.391+0000 7efe66d9d700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe680b38b0 0x7efe680b3d30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efe5000bac0 tx=0x7efe60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.392+0000 7efe56ffd700 1 -- 172.21.15.158:0/3131182531 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60007000 con 0x7efe680b38b0 2023-10-16T09:44:28.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.392+0000 7efe56ffd700 1 -- 172.21.15.158:0/3131182531 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60014d40 con 0x7efe680b38b0 2023-10-16T09:44:28.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.392+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe680cb960 con 0x7efe680b38b0 2023-10-16T09:44:28.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.392+0000 7efe56ffd700 1 -- 172.21.15.158:0/3131182531 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60003c60 con 0x7efe680b38b0 2023-10-16T09:44:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.392+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe680caba0 con 0x7efe680b38b0 2023-10-16T09:44:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.393+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe680681b0 con 0x7efe680b38b0 2023-10-16T09:44:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.393+0000 7efe56ffd700 1 -- 172.21.15.158:0/3131182531 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efe60004030 con 0x7efe680b38b0 2023-10-16T09:44:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.394+0000 7efe56ffd700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efe4806af30 0x7efe4806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.394+0000 7efe6659c700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efe4806af30 0x7efe4806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.394+0000 7efe56ffd700 1 -- 172.21.15.158:0/3131182531 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6750+0+0 (secure 0 0 0) 0x7efe6008af40 con 0x7efe680b38b0 2023-10-16T09:44:28.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.394+0000 7efe6659c700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efe4806af30 0x7efe4806d3f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efe58001eb0 tx=0x7efe58009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.398+0000 7efe56ffd700 1 -- 172.21.15.158:0/3131182531 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe60059820 con 0x7efe680b38b0 2023-10-16T09:44:28.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.637+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efe680b3390 con 0x7efe680b38b0 2023-10-16T09:44:28.639 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T09:44:28.637+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T09:44:28.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.639+0000 7efe56ffd700 1 -- 172.21.15.158:0/3131182531 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7efe60018020 con 0x7efe680b38b0 2023-10-16T09:44:28.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efe4806af30 msgr2=0x7efe4806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efe4806af30 0x7efe4806d3f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efe58001eb0 tx=0x7efe58009040 comp rx=0 tx=0).stop 2023-10-16T09:44:28.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe680b38b0 msgr2=0x7efe680b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe680b38b0 0x7efe680b3d30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efe5000bac0 tx=0x7efe60009750 comp rx=0 tx=0).stop 2023-10-16T09:44:28.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 shutdown_connections 2023-10-16T09:44:28.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe680b38b0 0x7efe680b3d30 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efe4806af30 0x7efe4806d3f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe50004d90 0x7efe680cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 --2- 172.21.15.158:0/3131182531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe680cbba0 0x7efe68152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 >> 172.21.15.158:0/3131182531 conn(0x7efe6806b9e0 msgr2=0x7efe680cf550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:28.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 shutdown_connections 2023-10-16T09:44:28.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.641+0000 7efe6ccc7700 1 -- 172.21.15.158:0/3131182531 wait complete. 2023-10-16T09:44:28.646 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T09:44:28.648 INFO:tasks.ceph.mds.b.smithi172.stderr: -5771> 2023-10-16T09:43:26.423+0000 7f0c79870700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T09:44:28.655 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T09:44:28.681 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:44:28.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.777+0000 7efc7857e700 1 Processor -- start 2023-10-16T09:44:28.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.777+0000 7efc7857e700 1 -- start start 2023-10-16T09:44:28.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.778+0000 7efc7857e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc70141920 0x7efc7006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.778+0000 7efc7857e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc700cde10 0x7efc7006da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.778+0000 7efc7857e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7006c210 con 0x7efc70144840 2023-10-16T09:44:28.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.778+0000 7efc7857e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc70141d00 con 0x7efc70141920 2023-10-16T09:44:28.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.778+0000 7efc7857e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7006b970 con 0x7efc700cde10 2023-10-16T09:44:28.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.778+0000 7efc7631a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc70141920 0x7efc7006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.778+0000 7efc7631a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc70141920 0x7efc7006b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48106/0 (socket says 172.21.15.158:48106) 2023-10-16T09:44:28.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc7631a700 1 -- 172.21.15.158:0/1764598020 learned_addr learned my addr 172.21.15.158:0/1764598020 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:28.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc76b1b700 1 --2- 172.21.15.158:0/1764598020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc700cde10 0x7efc7006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc7631a700 1 -- 172.21.15.158:0/1764598020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc700cde10 msgr2=0x7efc7006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:28.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc7631a700 1 --2- 172.21.15.158:0/1764598020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc700cde10 0x7efc7006da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc7631a700 1 -- 172.21.15.158:0/1764598020 >> v1:172.21.15.172:6789/0 conn(0x7efc70144840 legacy=0x7efc700cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc7631a700 1 -- 172.21.15.158:0/1764598020 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7006bc50 con 0x7efc70141920 2023-10-16T09:44:28.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc76b1b700 1 --2- 172.21.15.158:0/1764598020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc700cde10 0x7efc7006da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:28.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc7631a700 1 --2- 172.21.15.158:0/1764598020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc70141920 0x7efc7006b2e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7efc600098f0 tx=0x7efc60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce38300d5cd8306c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc75318700 1 -- 172.21.15.158:0/1764598020 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6000b8a0 con 0x7efc70141920 2023-10-16T09:44:28.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc75318700 1 -- 172.21.15.158:0/1764598020 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6000ba00 con 0x7efc70141920 2023-10-16T09:44:28.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.779+0000 7efc75318700 1 -- 172.21.15.158:0/1764598020 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6000bd90 con 0x7efc70141920 2023-10-16T09:44:28.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 -- 172.21.15.158:0/1764598020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc70141920 msgr2=0x7efc7006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 --2- 172.21.15.158:0/1764598020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc70141920 0x7efc7006b2e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7efc600098f0 tx=0x7efc60009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:28.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 -- 172.21.15.158:0/1764598020 shutdown_connections 2023-10-16T09:44:28.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 --2- 172.21.15.158:0/1764598020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc700cde10 0x7efc7006da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 --2- 172.21.15.158:0/1764598020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc70141920 0x7efc7006b2e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 -- 172.21.15.158:0/1764598020 >> 172.21.15.158:0/1764598020 conn(0x7efc700cc5b0 msgr2=0x7efc700cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:28.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 -- 172.21.15.158:0/1764598020 shutdown_connections 2023-10-16T09:44:28.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.780+0000 7efc7857e700 1 -- 172.21.15.158:0/1764598020 wait complete. 2023-10-16T09:44:28.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.781+0000 7efc7857e700 1 Processor -- start 2023-10-16T09:44:28.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.781+0000 7efc7857e700 1 -- start start 2023-10-16T09:44:28.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc700cde10 0x7efc701408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc70141920 0x7efc70142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc70144840 0x7efc70142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc700b4030 con 0x7efc700cde10 2023-10-16T09:44:28.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc701e3c30 con 0x7efc70144840 2023-10-16T09:44:28.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc701e3f10 con 0x7efc70141920 2023-10-16T09:44:28.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc700cde10 0x7efc701408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc700cde10 0x7efc701408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48112/0 (socket says 172.21.15.158:48112) 2023-10-16T09:44:28.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 -- 172.21.15.158:0/386550764 learned_addr learned my addr 172.21.15.158:0/386550764 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:28.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc76b1b700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc70144840 0x7efc70142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc75b19700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc70141920 0x7efc70142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 -- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc70141920 msgr2=0x7efc70142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc70141920 0x7efc70142510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 -- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc70144840 msgr2=0x7efc70142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:28.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc70144840 0x7efc70142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:28.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 -- 172.21.15.158:0/386550764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60009550 con 0x7efc700cde10 2023-10-16T09:44:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7631a700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc700cde10 0x7efc701408d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc66ffd700 1 -- 172.21.15.158:0/386550764 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60004300 con 0x7efc700cde10 2023-10-16T09:44:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc66ffd700 1 -- 172.21.15.158:0/386550764 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6001f070 con 0x7efc700cde10 2023-10-16T09:44:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 -- 172.21.15.158:0/386550764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc701e41f0 con 0x7efc700cde10 2023-10-16T09:44:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc66ffd700 1 -- 172.21.15.158:0/386550764 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc600036a0 con 0x7efc700cde10 2023-10-16T09:44:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.782+0000 7efc7857e700 1 -- 172.21.15.158:0/386550764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc701e44d0 con 0x7efc700cde10 2023-10-16T09:44:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.783+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c0052f0 con 0x7efc700cde10 2023-10-16T09:44:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.784+0000 7efc66ffd700 1 -- 172.21.15.158:0/386550764 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efc60021030 con 0x7efc700cde10 2023-10-16T09:44:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.784+0000 7efc66ffd700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efc5406af00 0x7efc5406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.784+0000 7efc66ffd700 1 -- 172.21.15.158:0/386550764 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6836+0+0 (secure 0 0 0) 0x7efc6008ad70 con 0x7efc700cde10 2023-10-16T09:44:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.784+0000 7efc75b19700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efc5406af00 0x7efc5406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.785+0000 7efc75b19700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efc5406af00 0x7efc5406d3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efc701e60a0 tx=0x7efc6c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:28.789+0000 7efc66ffd700 1 -- 172.21.15.158:0/386550764 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc600559c0 con 0x7efc700cde10 2023-10-16T09:44:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.027+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efc4c005b20 con 0x7efc700cde10 2023-10-16T09:44:29.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.647+0000 7efc66ffd700 1 -- 172.21.15.158:0/386550764 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7efc60058fe0 con 0x7efc700cde10 2023-10-16T09:44:29.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efc5406af00 msgr2=0x7efc5406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:29.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efc5406af00 0x7efc5406d3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efc701e60a0 tx=0x7efc6c009040 comp rx=0 tx=0).stop 2023-10-16T09:44:29.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc700cde10 msgr2=0x7efc701408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:29.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc700cde10 0x7efc701408d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc60004750 comp rx=0 tx=0).stop 2023-10-16T09:44:29.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 shutdown_connections 2023-10-16T09:44:29.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc70144840 0x7efc70142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc700cde10 0x7efc701408d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efc5406af00 0x7efc5406d3c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 --2- 172.21.15.158:0/386550764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc70141920 0x7efc70142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.649+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 >> 172.21.15.158:0/386550764 conn(0x7efc700cc5b0 msgr2=0x7efc700ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:29.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.650+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 shutdown_connections 2023-10-16T09:44:29.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.650+0000 7efc64ff9700 1 -- 172.21.15.158:0/386550764 wait complete. 2023-10-16T09:44:29.664 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T09:44:29.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.789+0000 7fc92930a700 1 Processor -- start 2023-10-16T09:44:29.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.789+0000 7fc92930a700 1 -- start start 2023-10-16T09:44:29.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc92930a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc92415dc60 0x7fc92406b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:29.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc92930a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9240cb260 0x7fc9240cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:29.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc92930a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9240cd9f0 con 0x7fc92406be20 2023-10-16T09:44:29.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc92930a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92415e040 con 0x7fc92415dc60 2023-10-16T09:44:29.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc92930a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92406c2a0 con 0x7fc9240cb260 2023-10-16T09:44:29.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc92415dc60 0x7fc92406b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:29.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc92415dc60 0x7fc92406b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48120/0 (socket says 172.21.15.158:48120) 2023-10-16T09:44:29.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 -- 172.21.15.158:0/351890144 learned_addr learned my addr 172.21.15.158:0/351890144 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:29.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc9237fe700 1 --2- 172.21.15.158:0/351890144 >> v2:172.21.15.172:3300/0 conn(0x7fc9240cb260 0x7fc9240cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:29.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 -- 172.21.15.158:0/351890144 >> v2:172.21.15.172:3300/0 conn(0x7fc9240cb260 msgr2=0x7fc9240cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:29.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 --2- 172.21.15.158:0/351890144 >> v2:172.21.15.172:3300/0 conn(0x7fc9240cb260 0x7fc9240cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 -- 172.21.15.158:0/351890144 >> v1:172.21.15.172:6789/0 conn(0x7fc92406be20 legacy=0x7fc9240cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:29.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 -- 172.21.15.158:0/351890144 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240cb6f0 con 0x7fc92415dc60 2023-10-16T09:44:29.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc922ffd700 1 --2- 172.21.15.158:0/351890144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc92406b8e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc924070cb0 tx=0x7fc918009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85ff2c244f93292 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:29.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc921ffb700 1 -- 172.21.15.158:0/351890144 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91800b870 con 0x7fc92415dc60 2023-10-16T09:44:29.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.790+0000 7fc921ffb700 1 -- 172.21.15.158:0/351890144 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91800b9d0 con 0x7fc92415dc60 2023-10-16T09:44:29.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc921ffb700 1 -- 172.21.15.158:0/351890144 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91800bd60 con 0x7fc92415dc60 2023-10-16T09:44:29.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 -- 172.21.15.158:0/351890144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 msgr2=0x7fc92406b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:29.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 --2- 172.21.15.158:0/351890144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc92406b8e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc924070cb0 tx=0x7fc918009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 -- 172.21.15.158:0/351890144 shutdown_connections 2023-10-16T09:44:29.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 --2- 172.21.15.158:0/351890144 >> v2:172.21.15.172:3300/0 conn(0x7fc9240cb260 0x7fc9240cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 --2- 172.21.15.158:0/351890144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc92406b8e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 -- 172.21.15.158:0/351890144 >> 172.21.15.158:0/351890144 conn(0x7fc924070fe0 msgr2=0x7fc9240713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:29.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 -- 172.21.15.158:0/351890144 shutdown_connections 2023-10-16T09:44:29.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.791+0000 7fc92930a700 1 -- 172.21.15.158:0/351890144 wait complete. 2023-10-16T09:44:29.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.792+0000 7fc92930a700 1 Processor -- start 2023-10-16T09:44:29.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.792+0000 7fc92930a700 1 -- start start 2023-10-16T09:44:29.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc92930a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc92406be20 0x7fc92406cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:29.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc92930a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240cb260 0x7fc9241f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:29.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc92930a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc9240b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:29.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc92930a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92406ef60 con 0x7fc92415dc60 2023-10-16T09:44:29.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc92930a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241f36c0 con 0x7fc92406be20 2023-10-16T09:44:29.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc92930a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241f39a0 con 0x7fc9240cb260 2023-10-16T09:44:29.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc9240b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:29.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc9240b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48128/0 (socket says 172.21.15.158:48128) 2023-10-16T09:44:29.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 -- 172.21.15.158:0/3890182636 learned_addr learned my addr 172.21.15.158:0/3890182636 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:29.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc922ffd700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc92406be20 0x7fc92406cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:29.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9227fc700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240cb260 0x7fc9241f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:29.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 -- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240cb260 msgr2=0x7fc9241f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:29.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240cb260 0x7fc9241f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 -- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc92406be20 msgr2=0x7fc92406cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:29.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc92406be20 0x7fc92406cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:29.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9237fe700 1 -- 172.21.15.158:0/3890182636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc918009550 con 0x7fc92415dc60 2023-10-16T09:44:29.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.793+0000 7fc9227fc700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240cb260 0x7fc9241f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:29.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.794+0000 7fc9237fe700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc9240b0b70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc91c007e60 tx=0x7fc91c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:29.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.794+0000 7fc90b7fe700 1 -- 172.21.15.158:0/3890182636 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00ede0 con 0x7fc92415dc60 2023-10-16T09:44:29.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.794+0000 7fc90b7fe700 1 -- 172.21.15.158:0/3890182636 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c014420 con 0x7fc92415dc60 2023-10-16T09:44:29.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.794+0000 7fc90b7fe700 1 -- 172.21.15.158:0/3890182636 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c0147d0 con 0x7fc92415dc60 2023-10-16T09:44:29.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.794+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9241f3c80 con 0x7fc92415dc60 2023-10-16T09:44:29.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.794+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9241f4240 con 0x7fc92415dc60 2023-10-16T09:44:29.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.795+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9241f69f0 con 0x7fc92415dc60 2023-10-16T09:44:29.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.795+0000 7fc90b7fe700 1 -- 172.21.15.158:0/3890182636 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc91c014930 con 0x7fc92415dc60 2023-10-16T09:44:29.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.795+0000 7fc90b7fe700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc91006af00 0x7fc91006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:29.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.796+0000 7fc90b7fe700 1 -- 172.21.15.158:0/3890182636 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6836+0+0 (secure 0 0 0) 0x7fc91c08a160 con 0x7fc92415dc60 2023-10-16T09:44:29.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.796+0000 7fc922ffd700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc91006af00 0x7fc91006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:29.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.796+0000 7fc922ffd700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc91006af00 0x7fc91006d3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc918000c00 tx=0x7fc918008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:29.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.800+0000 7fc90b7fe700 1 -- 172.21.15.158:0/3890182636 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9241f69f0 con 0x7fc92415dc60 2023-10-16T09:44:29.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:29.996+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc9241f69f0 con 0x7fc92415dc60 2023-10-16T09:44:30.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.649+0000 7fc90b7fe700 1 -- 172.21.15.158:0/3890182636 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc9241f69f0 con 0x7fc92415dc60 2023-10-16T09:44:30.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc91006af00 msgr2=0x7fc91006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:30.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc91006af00 0x7fc91006d3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc918000c00 tx=0x7fc918008030 comp rx=0 tx=0).stop 2023-10-16T09:44:30.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 msgr2=0x7fc9240b0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:30.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc9240b0b70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc91c007e60 tx=0x7fc91c00e3c0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 shutdown_connections 2023-10-16T09:44:30.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92415dc60 0x7fc9240b0b70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc92406be20 0x7fc92406cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc91006af00 0x7fc91006d3c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 --2- 172.21.15.158:0/3890182636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240cb260 0x7fc9241f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 >> 172.21.15.158:0/3890182636 conn(0x7fc924070fe0 msgr2=0x7fc92415ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:30.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.653+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 shutdown_connections 2023-10-16T09:44:30.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.654+0000 7fc92930a700 1 -- 172.21.15.158:0/3890182636 wait complete. 2023-10-16T09:44:30.658 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T09:44:30.671 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T09:44:30.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.801+0000 7f022e709700 1 Processor -- start 2023-10-16T09:44:30.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.801+0000 7f022e709700 1 -- start start 2023-10-16T09:44:30.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f022e709700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02280ce510 0x7f022815f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:30.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f022e709700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f02280cbaf0 0x7f02280cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:30.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f022e709700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02280cf4c0 con 0x7f022815b590 2023-10-16T09:44:30.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f022e709700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02280cc4c0 con 0x7f02280ce510 2023-10-16T09:44:30.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f022e709700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02280b49e0 con 0x7f02280cbaf0 2023-10-16T09:44:30.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f02277fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02280ce510 0x7f022815f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:30.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f0227fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f022815b590 0x7f02280b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48308/0 (socket says 172.21.15.158:48308) 2023-10-16T09:44:30.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.802+0000 7f0227fff700 1 -- 172.21.15.158:0/1347351573 learned_addr learned my addr 172.21.15.158:0/1347351573 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:30.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f022cca6700 1 --2- 172.21.15.158:0/1347351573 >> v2:172.21.15.172:3300/0 conn(0x7f02280cbaf0 0x7f02280cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:30.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f02277fe700 1 -- 172.21.15.158:0/1347351573 >> v2:172.21.15.172:3300/0 conn(0x7f02280cbaf0 msgr2=0x7f02280cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:30.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f02277fe700 1 --2- 172.21.15.158:0/1347351573 >> v2:172.21.15.172:3300/0 conn(0x7f02280cbaf0 0x7f02280cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f02277fe700 1 -- 172.21.15.158:0/1347351573 >> v1:172.21.15.158:6789/0 conn(0x7f022815b590 legacy=0x7f02280b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:30.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f02277fe700 1 -- 172.21.15.158:0/1347351573 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022815f710 con 0x7f02280ce510 2023-10-16T09:44:30.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f0226ffd700 1 -- 172.21.15.158:0/1347351573 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083039103 0 0) 0x7f02280cf4c0 con 0x7f022815b590 2023-10-16T09:44:30.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f022cca6700 1 --2- 172.21.15.158:0/1347351573 >> v2:172.21.15.172:3300/0 conn(0x7f02280cbaf0 0x7f02280cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:30.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f02277fe700 1 --2- 172.21.15.158:0/1347351573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02280ce510 0x7f022815f1d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f02280cf430 tx=0x7f0218009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e7a27deb1dd29a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:30.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f0226ffd700 1 -- 172.21.15.158:0/1347351573 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218011830 con 0x7f02280ce510 2023-10-16T09:44:30.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f0226ffd700 1 -- 172.21.15.158:0/1347351573 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218011990 con 0x7f02280ce510 2023-10-16T09:44:30.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.803+0000 7f0226ffd700 1 -- 172.21.15.158:0/1347351573 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218011d20 con 0x7f02280ce510 2023-10-16T09:44:30.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 -- 172.21.15.158:0/1347351573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02280ce510 msgr2=0x7f022815f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:30.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 --2- 172.21.15.158:0/1347351573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02280ce510 0x7f022815f1d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f02280cf430 tx=0x7f0218009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:30.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 -- 172.21.15.158:0/1347351573 shutdown_connections 2023-10-16T09:44:30.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 --2- 172.21.15.158:0/1347351573 >> v2:172.21.15.172:3300/0 conn(0x7f02280cbaf0 0x7f02280cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 --2- 172.21.15.158:0/1347351573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02280ce510 0x7f022815f1d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 -- 172.21.15.158:0/1347351573 >> 172.21.15.158:0/1347351573 conn(0x7f022806a8d0 msgr2=0x7f022806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:30.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 -- 172.21.15.158:0/1347351573 shutdown_connections 2023-10-16T09:44:30.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.804+0000 7f022e709700 1 -- 172.21.15.158:0/1347351573 wait complete. 2023-10-16T09:44:30.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.805+0000 7f022e709700 1 Processor -- start 2023-10-16T09:44:30.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.805+0000 7f022e709700 1 -- start start 2023-10-16T09:44:30.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022e709700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02280cbaf0 0x7f0228159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:30.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022e709700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02280ce510 0x7f022815db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:30.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022e709700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f022815b590 0x7f022815e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:30.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022e709700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02280aad00 con 0x7f022815b590 2023-10-16T09:44:30.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022e709700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02280ca6f0 con 0x7f02280ce510 2023-10-16T09:44:30.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022e709700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02280ca9d0 con 0x7f02280cbaf0 2023-10-16T09:44:30.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f022815b590 0x7f022815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:30.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f022815b590 0x7f022815e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48142/0 (socket says 172.21.15.158:48142) 2023-10-16T09:44:30.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 -- 172.21.15.158:0/1869591992 learned_addr learned my addr 172.21.15.158:0/1869591992 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:30.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f0227fff700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02280cbaf0 0x7f0228159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:30.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 -- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02280cbaf0 msgr2=0x7f0228159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:30.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f02277fe700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02280ce510 0x7f022815db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:30.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02280cbaf0 0x7f0228159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 -- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02280ce510 msgr2=0x7f022815db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:30.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02280ce510 0x7f022815db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:30.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 -- 172.21.15.158:0/1869591992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180094a0 con 0x7f022815b590 2023-10-16T09:44:30.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f02277fe700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02280ce510 0x7f022815db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:30.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022cca6700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f022815b590 0x7f022815e080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f022000cef0 tx=0x7f022000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:30.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f0224ff9700 1 -- 172.21.15.158:0/1869591992 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022000ae50 con 0x7f022815b590 2023-10-16T09:44:30.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f0224ff9700 1 -- 172.21.15.158:0/1869591992 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0220015440 con 0x7f022815b590 2023-10-16T09:44:30.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f0224ff9700 1 -- 172.21.15.158:0/1869591992 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02200157f0 con 0x7f022815b590 2023-10-16T09:44:30.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.806+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02280cacb0 con 0x7f022815b590 2023-10-16T09:44:30.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.807+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02280cb270 con 0x7f022815b590 2023-10-16T09:44:30.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.807+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022815ff10 con 0x7f022815b590 2023-10-16T09:44:30.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.808+0000 7f0224ff9700 1 -- 172.21.15.158:0/1869591992 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f022000d430 con 0x7f022815b590 2023-10-16T09:44:30.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.808+0000 7f0224ff9700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f021406af00 0x7f021406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:30.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.808+0000 7f0224ff9700 1 -- 172.21.15.158:0/1869591992 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6398+0+0 (secure 0 0 0) 0x7f0220089df0 con 0x7f022815b590 2023-10-16T09:44:30.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.808+0000 7f0227fff700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f021406af00 0x7f021406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:30.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.809+0000 7f0227fff700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f021406af00 0x7f021406d3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f021c000f60 tx=0x7f021c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:30.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:30.813+0000 7f0224ff9700 1 -- 172.21.15.158:0/1869591992 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f022815ff10 con 0x7f022815b590 2023-10-16T09:44:31.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.006+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f022815ff10 con 0x7f022815b590 2023-10-16T09:44:31.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.653+0000 7f0224ff9700 1 -- 172.21.15.158:0/1869591992 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7f022815ff10 con 0x7f022815b590 2023-10-16T09:44:31.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f021406af00 msgr2=0x7f021406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:31.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f021406af00 0x7f021406d3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f021c000f60 tx=0x7f021c009040 comp rx=0 tx=0).stop 2023-10-16T09:44:31.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f022815b590 msgr2=0x7f022815e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:31.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f022815b590 0x7f022815e080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f022000cef0 tx=0x7f022000a340 comp rx=0 tx=0).stop 2023-10-16T09:44:31.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 shutdown_connections 2023-10-16T09:44:31.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f022815b590 0x7f022815e080 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f021406af00 0x7f021406d3c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02280cbaf0 0x7f0228159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 --2- 172.21.15.158:0/1869591992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02280ce510 0x7f022815db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.658+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 >> 172.21.15.158:0/1869591992 conn(0x7f022806a8d0 msgr2=0x7f02280b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:31.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.659+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 shutdown_connections 2023-10-16T09:44:31.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.659+0000 7f022e709700 1 -- 172.21.15.158:0/1869591992 wait complete. 2023-10-16T09:44:31.663 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T09:44:31.679 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-16T09:44:31.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.810+0000 7fc7bf260700 1 Processor -- start 2023-10-16T09:44:31.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.810+0000 7fc7bf260700 1 -- start start 2023-10-16T09:44:31.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7bf260700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 0x7fc7b81686e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7bf260700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b80d1ee0 con 0x7fc7b815daa0 2023-10-16T09:44:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7bf260700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b80d12d0 con 0x7fc7b815d6c0 2023-10-16T09:44:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7bf260700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80d40a0 con 0x7fc7b80d0640 2023-10-16T09:44:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc7b815daa0 0x7fc7b80cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48322/0 (socket says 172.21.15.158:48322) 2023-10-16T09:44:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b7fff700 1 -- 172.21.15.158:0/3576308587 learned_addr learned my addr 172.21.15.158:0/3576308587 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7bd7fd700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 0x7fc7b81686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678462479 0 0) 0x7fc7b80d1ee0 con 0x7fc7b815daa0 2023-10-16T09:44:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a80035b0 con 0x7fc7b815daa0 2023-10-16T09:44:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 829605105 0 0) 0x7fc7a80035b0 con 0x7fc7b815daa0 2023-10-16T09:44:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 msgr2=0x7fc7b81686e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 0x7fc7b81686e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 >> v1:172.21.15.172:6789/0 conn(0x7fc7b815d6c0 legacy=0x7fc7b80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b80d15b0 con 0x7fc7b815daa0 2023-10-16T09:44:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.811+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc7a4001670 con 0x7fc7b815daa0 2023-10-16T09:44:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 >> v1:172.21.15.158:6789/0 conn(0x7fc7b815daa0 legacy=0x7fc7b80cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a8003a30 0x7fc7a8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 0x7fc7a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a800bb40 con 0x7fc7a8004d90 2023-10-16T09:44:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a800bcb0 con 0x7fc7a8003a30 2023-10-16T09:44:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a800be70 con 0x7fc7b80d0640 2023-10-16T09:44:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bd7fd700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bcffc700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a8003a30 0x7fc7a8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b7fff700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 0x7fc7a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bd7fd700 1 -- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 msgr2=0x7fc7a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bd7fd700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b80d0640 0x7fc7a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bd7fd700 1 -- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a8003a30 msgr2=0x7fc7a8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bd7fd700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a8003a30 0x7fc7a8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bd7fd700 1 -- 172.21.15.158:0/3576308587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a80035b0 con 0x7fc7a8004d90 2023-10-16T09:44:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bcffc700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a8003a30 0x7fc7a8003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7bd7fd700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7a8005210 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009f90 tx=0x7fc7a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c88e90eeed6bc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b000ec30 con 0x7fc7a8004d90 2023-10-16T09:44:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b000f040 con 0x7fc7a8004d90 2023-10-16T09:44:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.812+0000 7fc7b77fe700 1 -- 172.21.15.158:0/3576308587 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0014480 con 0x7fc7a8004d90 2023-10-16T09:44:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.813+0000 7fc7bf260700 1 -- 172.21.15.158:0/3576308587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 msgr2=0x7fc7a8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.813+0000 7fc7bf260700 1 --2- 172.21.15.158:0/3576308587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7a8005210 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009f90 tx=0x7fc7a8005880 comp rx=0 tx=0).stop 2023-10-16T09:44:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.813+0000 7fc7bd7fd700 1 -- 172.21.15.158:0/3576308587 reap_dead start 2023-10-16T09:44:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.813+0000 7fc7bf260700 1 -- 172.21.15.158:0/3576308587 shutdown_connections 2023-10-16T09:44:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.813+0000 7fc7bf260700 1 -- 172.21.15.158:0/3576308587 >> 172.21.15.158:0/3576308587 conn(0x7fc7b809c940 msgr2=0x7fc7b80d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.813+0000 7fc7bf260700 1 -- 172.21.15.158:0/3576308587 shutdown_connections 2023-10-16T09:44:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.813+0000 7fc7bf260700 1 -- 172.21.15.158:0/3576308587 wait complete. 2023-10-16T09:44:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.814+0000 7fc7bf260700 1 Processor -- start 2023-10-16T09:44:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.814+0000 7fc7bf260700 1 -- start start 2023-10-16T09:44:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bf260700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7b815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bf260700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b8152af0 0x7fc7b80b8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bf260700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7b80b8560 0x7fc7b8160310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bf260700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80c70e0 con 0x7fc7a8004d90 2023-10-16T09:44:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bf260700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80c7980 con 0x7fc7b80b8560 2023-10-16T09:44:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bf260700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80b8a50 con 0x7fc7b8152af0 2023-10-16T09:44:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7b815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7b815deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48158/0 (socket says 172.21.15.158:48158) 2023-10-16T09:44:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 -- 172.21.15.158:0/81389376 learned_addr learned my addr 172.21.15.158:0/81389376 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bd7fd700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7b80b8560 0x7fc7b8160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7b7fff700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b8152af0 0x7fc7b80b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 -- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b8152af0 msgr2=0x7fc7b80b8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b8152af0 0x7fc7b80b8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 -- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7b80b8560 msgr2=0x7fc7b8160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7b80b8560 0x7fc7b8160310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 -- 172.21.15.158:0/81389376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b80c7330 con 0x7fc7a8004d90 2023-10-16T09:44:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.815+0000 7fc7bcffc700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7b815deb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00e7b0 tx=0x7fc7ac00eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.816+0000 7fc7b57fa700 1 -- 172.21.15.158:0/81389376 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0052e0 con 0x7fc7a8004d90 2023-10-16T09:44:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.816+0000 7fc7b57fa700 1 -- 172.21.15.158:0/81389376 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac005440 con 0x7fc7a8004d90 2023-10-16T09:44:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.816+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b80c7610 con 0x7fc7a8004d90 2023-10-16T09:44:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.816+0000 7fc7b57fa700 1 -- 172.21.15.158:0/81389376 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac003880 con 0x7fc7a8004d90 2023-10-16T09:44:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.816+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b80c9f60 con 0x7fc7a8004d90 2023-10-16T09:44:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.817+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b80c7c60 con 0x7fc7a8004d90 2023-10-16T09:44:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.817+0000 7fc7b57fa700 1 -- 172.21.15.158:0/81389376 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc7ac0039e0 con 0x7fc7a8004d90 2023-10-16T09:44:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.817+0000 7fc7b57fa700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc79c06af00 0x7fc79c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.817+0000 7fc7b57fa700 1 -- 172.21.15.158:0/81389376 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fc7ac088c10 con 0x7fc7a8004d90 2023-10-16T09:44:31.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.818+0000 7fc7b7fff700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc79c06af00 0x7fc79c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:31.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.818+0000 7fc7b7fff700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc79c06af00 0x7fc79c06d3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc7b806c0d0 tx=0x7fc7a4007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:31.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:31.822+0000 7fc7b57fa700 1 -- 172.21.15.158:0/81389376 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7ac053db0 con 0x7fc7a8004d90 2023-10-16T09:44:32.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.015+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc7b80681b0 con 0x7fc7a8004d90 2023-10-16T09:44:32.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.654+0000 7fc7b57fa700 1 -- 172.21.15.158:0/81389376 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7fc7ac019020 con 0x7fc7a8004d90 2023-10-16T09:44:32.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc79c06af00 msgr2=0x7fc79c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:32.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc79c06af00 0x7fc79c06d3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc7b806c0d0 tx=0x7fc7a4007040 comp rx=0 tx=0).stop 2023-10-16T09:44:32.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 msgr2=0x7fc7b815deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:32.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7b815deb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00e7b0 tx=0x7fc7ac00eac0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 shutdown_connections 2023-10-16T09:44:32.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7b80b8560 0x7fc7b8160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7a8004d90 0x7fc7b815deb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fc79c06af00 0x7fc79c06d3c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 --2- 172.21.15.158:0/81389376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7b8152af0 0x7fc7b80b8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 >> 172.21.15.158:0/81389376 conn(0x7fc7b809c940 msgr2=0x7fc7b806c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:32.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 shutdown_connections 2023-10-16T09:44:32.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.657+0000 7fc7bf260700 1 -- 172.21.15.158:0/81389376 wait complete. 2023-10-16T09:44:32.661 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data_ec' removed 2023-10-16T09:44:32.676 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T09:44:32.676 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T09:44:32.677 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T09:44:32.677 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:44:32.677 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T09:44:32.678 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T09:44:32.678 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T09:44:32.678 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:44:32.679 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T09:44:32.679 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T09:44:32.679 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T09:44:32.679 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:44:32.680 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T09:44:32.680 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T09:44:32.680 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T09:44:32.680 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:44:32.686 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T09:44:32.686 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T09:44:32.687 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T09:44:32.688 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T09:44:32.690 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T09:44:32.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T09:44:32.691 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T09:44:32.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T09:44:32.728 INFO:tasks.ceph.mds.a:Started 2023-10-16T09:44:32.729 DEBUG:teuthology.parallel:result is None 2023-10-16T09:44:32.730 INFO:tasks.ceph.mds.c:Started 2023-10-16T09:44:32.730 DEBUG:teuthology.parallel:result is None 2023-10-16T09:44:32.732 INFO:tasks.ceph.mds.b:Started 2023-10-16T09:44:32.732 DEBUG:teuthology.parallel:result is None 2023-10-16T09:44:32.733 INFO:tasks.ceph.mds.d:Started 2023-10-16T09:44:32.733 DEBUG:teuthology.parallel:result is None 2023-10-16T09:44:32.734 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T09:44:32.812 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T09:44:32.817 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T09:44:32.818 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:44:32.818 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T09:44:32.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.865+0000 7f83b92b4700 1 Processor -- start 2023-10-16T09:44:32.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.865+0000 7f83b92b4700 1 -- start start 2023-10-16T09:44:32.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b92b4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415d7c0 0x7f83b406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:32.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b92b4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f83b40cfd90 0x7f83b40719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:32.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b92b4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b406b320 con 0x7f83b415dba0 2023-10-16T09:44:32.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b92b4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b415e020 con 0x7f83b40cfd90 2023-10-16T09:44:32.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b92b4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b415e300 con 0x7f83b415d7c0 2023-10-16T09:44:32.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b37fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f83b40cfd90 0x7f83b40719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:32.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b27fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f83b415dba0 0x7f83b40cf680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48368/0 (socket says 172.21.15.158:48368) 2023-10-16T09:44:32.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b27fc700 1 -- 172.21.15.158:0/1568147068 learned_addr learned my addr 172.21.15.158:0/1568147068 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:32.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b2ffd700 1 --2- 172.21.15.158:0/1568147068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415d7c0 0x7f83b406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:32.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b37fe700 1 -- 172.21.15.158:0/1568147068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415d7c0 msgr2=0x7f83b406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:32.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b37fe700 1 --2- 172.21.15.158:0/1568147068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415d7c0 0x7f83b406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b37fe700 1 -- 172.21.15.158:0/1568147068 >> v1:172.21.15.158:6789/0 conn(0x7f83b415dba0 legacy=0x7f83b40cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:32.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b37fe700 1 -- 172.21.15.158:0/1568147068 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b415e5e0 con 0x7f83b40cfd90 2023-10-16T09:44:32.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b1ffb700 1 -- 172.21.15.158:0/1568147068 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306948285 0 0) 0x7f83b406b320 con 0x7f83b415dba0 2023-10-16T09:44:32.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b2ffd700 1 --2- 172.21.15.158:0/1568147068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415d7c0 0x7f83b406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:32.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b37fe700 1 --2- 172.21.15.158:0/1568147068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b40cfd90 0x7f83b40719f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009d90 tx=0x7f83ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a114316de5e51e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:32.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.866+0000 7f83b1ffb700 1 -- 172.21.15.158:0/1568147068 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00e070 con 0x7f83b40cfd90 2023-10-16T09:44:32.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b1ffb700 1 -- 172.21.15.158:0/1568147068 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac00bbd0 con 0x7f83b40cfd90 2023-10-16T09:44:32.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b1ffb700 1 -- 172.21.15.158:0/1568147068 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac012430 con 0x7f83b40cfd90 2023-10-16T09:44:32.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 -- 172.21.15.158:0/1568147068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b40cfd90 msgr2=0x7f83b40719f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:32.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 --2- 172.21.15.158:0/1568147068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b40cfd90 0x7f83b40719f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009d90 tx=0x7f83ac00b330 comp rx=0 tx=0).stop 2023-10-16T09:44:32.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 -- 172.21.15.158:0/1568147068 shutdown_connections 2023-10-16T09:44:32.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 --2- 172.21.15.158:0/1568147068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b40cfd90 0x7f83b40719f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 --2- 172.21.15.158:0/1568147068 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415d7c0 0x7f83b406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 -- 172.21.15.158:0/1568147068 >> 172.21.15.158:0/1568147068 conn(0x7f83b406da00 msgr2=0x7f83b406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:32.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 -- 172.21.15.158:0/1568147068 shutdown_connections 2023-10-16T09:44:32.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.867+0000 7f83b92b4700 1 -- 172.21.15.158:0/1568147068 wait complete. 2023-10-16T09:44:32.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.868+0000 7f83b92b4700 1 Processor -- start 2023-10-16T09:44:32.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.868+0000 7f83b92b4700 1 -- start start 2023-10-16T09:44:32.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b92b4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83b40cfd90 0x7f83b4145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:32.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b92b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b415d7c0 0x7f83b4148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:32.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b92b4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415dba0 0x7f83b41f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:32.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b92b4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40acb00 con 0x7f83b415d7c0 2023-10-16T09:44:32.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b92b4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4148680 con 0x7f83b40cfd90 2023-10-16T09:44:32.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b92b4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40cf780 con 0x7f83b415dba0 2023-10-16T09:44:32.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b415d7c0 0x7f83b4148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:32.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b415d7c0 0x7f83b4148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48198/0 (socket says 172.21.15.158:48198) 2023-10-16T09:44:32.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b27fc700 1 -- 172.21.15.158:0/2032981326 learned_addr learned my addr 172.21.15.158:0/2032981326 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:32.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b37fe700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415dba0 0x7f83b41f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:32.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.869+0000 7f83b2ffd700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83b40cfd90 0x7f83b4145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:32.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b27fc700 1 -- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415dba0 msgr2=0x7f83b41f60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:32.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b27fc700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415dba0 0x7f83b41f60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b27fc700 1 -- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83b40cfd90 msgr2=0x7f83b4145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:32.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b27fc700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83b40cfd90 0x7f83b4145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:32.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b27fc700 1 -- 172.21.15.158:0/2032981326 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac009a70 con 0x7f83b415d7c0 2023-10-16T09:44:32.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b27fc700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b415d7c0 0x7f83b4148140 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f83a800c550 tx=0x7f83a800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:32.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f839b7fe700 1 -- 172.21.15.158:0/2032981326 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8004c80 con 0x7f83b415d7c0 2023-10-16T09:44:32.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f839b7fe700 1 -- 172.21.15.158:0/2032981326 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8004e00 con 0x7f83b415d7c0 2023-10-16T09:44:32.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b415ec80 con 0x7f83b415d7c0 2023-10-16T09:44:32.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f839b7fe700 1 -- 172.21.15.158:0/2032981326 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8007310 con 0x7f83b415d7c0 2023-10-16T09:44:32.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.870+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b41f65e0 con 0x7f83b415d7c0 2023-10-16T09:44:32.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.871+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b415cff0 con 0x7f83b415d7c0 2023-10-16T09:44:32.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.871+0000 7f839b7fe700 1 -- 172.21.15.158:0/2032981326 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f83a8007470 con 0x7f83b415d7c0 2023-10-16T09:44:32.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.872+0000 7f839b7fe700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f83a006af00 0x7f83a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:32.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.872+0000 7f839b7fe700 1 -- 172.21.15.158:0/2032981326 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f83a8088fc0 con 0x7f83b415d7c0 2023-10-16T09:44:32.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.872+0000 7f83b2ffd700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f83a006af00 0x7f83a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:32.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.872+0000 7f83b2ffd700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f83a006af00 0x7f83a006d3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f83a4006fd0 tx=0x7f83a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:32.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:32.876+0000 7f839b7fe700 1 -- 172.21.15.158:0/2032981326 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83a80bbfe0 con 0x7f83b415d7c0 2023-10-16T09:44:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:33.088+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f83b40681b0 con 0x7f83b415d7c0 2023-10-16T09:44:34.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.380+0000 7f839b7fe700 1 -- 172.21.15.158:0/2032981326 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v35) v1 ==== 97+0+0 (secure 0 0 0) 0x7f83a8054230 con 0x7f83b415d7c0 2023-10-16T09:44:34.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.383+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f83a006af00 msgr2=0x7f83a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.383+0000 7f83b92b4700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f83a006af00 0x7f83a006d3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f83a4006fd0 tx=0x7f83a4008040 comp rx=0 tx=0).stop 2023-10-16T09:44:34.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.383+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b415d7c0 msgr2=0x7f83b4148140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.383+0000 7f83b92b4700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b415d7c0 0x7f83b4148140 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f83a800c550 tx=0x7f83a800c860 comp rx=0 tx=0).stop 2023-10-16T09:44:34.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 shutdown_connections 2023-10-16T09:44:34.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f83a006af00 0x7f83a006d3c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83b415dba0 0x7f83b41f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83b40cfd90 0x7f83b4145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 --2- 172.21.15.158:0/2032981326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b415d7c0 0x7f83b4148140 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 >> 172.21.15.158:0/2032981326 conn(0x7f83b406da00 msgr2=0x7f83b4072440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:34.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 shutdown_connections 2023-10-16T09:44:34.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.384+0000 7f83b92b4700 1 -- 172.21.15.158:0/2032981326 wait complete. 2023-10-16T09:44:34.388 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T09:44:34.400 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T09:44:34.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.524+0000 7fb11a576700 1 Processor -- start 2023-10-16T09:44:34.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.524+0000 7fb11a576700 1 -- start start 2023-10-16T09:44:34.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb11a576700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb11406ab40 0x7fb11406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb11a576700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 0x7fb114071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb11a576700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb114071e50 con 0x7fb11406e5d0 2023-10-16T09:44:34.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb11a576700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb114161270 con 0x7fb11406ab40 2023-10-16T09:44:34.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb11a576700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb114161510 con 0x7fb1140729b0 2023-10-16T09:44:34.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb1137fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb11406e5d0 0x7fb1140722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48386/0 (socket says 172.21.15.158:48386) 2023-10-16T09:44:34.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb1137fe700 1 -- 172.21.15.158:0/1543320073 learned_addr learned my addr 172.21.15.158:0/1543320073 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:34.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb118b13700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 0x7fb114071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb113fff700 1 --2- 172.21.15.158:0/1543320073 >> v2:172.21.15.172:3300/0 conn(0x7fb11406ab40 0x7fb11406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528571970 0 0) 0x7fb114071e50 con 0x7fb11406e5d0 2023-10-16T09:44:34.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0fc0035b0 con 0x7fb11406e5d0 2023-10-16T09:44:34.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095728229 0 0) 0x7fb0fc0035b0 con 0x7fb11406e5d0 2023-10-16T09:44:34.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 msgr2=0x7fb114071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:34.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 0x7fb114071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 >> v2:172.21.15.172:3300/0 conn(0x7fb11406ab40 msgr2=0x7fb11406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:34.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 --2- 172.21.15.158:0/1543320073 >> v2:172.21.15.172:3300/0 conn(0x7fb11406ab40 0x7fb11406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1141617f0 con 0x7fb11406e5d0 2023-10-16T09:44:34.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fb104001670 con 0x7fb11406e5d0 2023-10-16T09:44:34.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 >> v1:172.21.15.158:6789/0 conn(0x7fb11406e5d0 legacy=0x7fb1140722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc003a00 0x7fb0fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0fc004850 0x7fb0fc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 0x7fb0fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.525+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb114161270 con 0x7fb0fc004850 2023-10-16T09:44:34.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb114161510 con 0x7fb0fc003a00 2023-10-16T09:44:34.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc00ba10 con 0x7fb1140729b0 2023-10-16T09:44:34.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb1137fe700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0fc004850 0x7fb0fc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb113fff700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc003a00 0x7fb0fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb118b13700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 0x7fb0fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb1137fe700 1 -- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 msgr2=0x7fb0fc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:34.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb1137fe700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 0x7fb0fc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb1137fe700 1 -- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc003a00 msgr2=0x7fb0fc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:34.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb1137fe700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc003a00 0x7fb0fc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb1137fe700 1 -- 172.21.15.158:0/1543320073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc0035b0 con 0x7fb0fc004850 2023-10-16T09:44:34.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb118b13700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140729b0 0x7fb0fc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:34.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb113fff700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc003a00 0x7fb0fc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:34.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb1137fe700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0fc004850 0x7fb0fc004cd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb11406b560 tx=0x7fb0fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc17a160b66cbb4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:34.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104007c60 con 0x7fb0fc004850 2023-10-16T09:44:34.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400ce80 con 0x7fb0fc004850 2023-10-16T09:44:34.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.526+0000 7fb112ffd700 1 -- 172.21.15.158:0/1543320073 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104005ba0 con 0x7fb0fc004850 2023-10-16T09:44:34.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.527+0000 7fb11a576700 1 -- 172.21.15.158:0/1543320073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0fc004850 msgr2=0x7fb0fc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.527+0000 7fb11a576700 1 --2- 172.21.15.158:0/1543320073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0fc004850 0x7fb0fc004cd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb11406b560 tx=0x7fb0fc005880 comp rx=0 tx=0).stop 2023-10-16T09:44:34.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.527+0000 7fb118b13700 1 -- 172.21.15.158:0/1543320073 reap_dead start 2023-10-16T09:44:34.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.527+0000 7fb11a576700 1 -- 172.21.15.158:0/1543320073 shutdown_connections 2023-10-16T09:44:34.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.527+0000 7fb11a576700 1 -- 172.21.15.158:0/1543320073 >> 172.21.15.158:0/1543320073 conn(0x7fb1140d02b0 msgr2=0x7fb1140d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:34.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.527+0000 7fb11a576700 1 -- 172.21.15.158:0/1543320073 shutdown_connections 2023-10-16T09:44:34.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.527+0000 7fb11a576700 1 -- 172.21.15.158:0/1543320073 wait complete. 2023-10-16T09:44:34.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.528+0000 7fb11a576700 1 Processor -- start 2023-10-16T09:44:34.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.528+0000 7fb11a576700 1 -- start start 2023-10-16T09:44:34.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb11a576700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc004850 0x7fb1141f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb11a576700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140c6dd0 0x7fb1140cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb11a576700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1140c9780 0x7fb1140c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb11a576700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11414e290 con 0x7fb1140c9780 2023-10-16T09:44:34.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb11a576700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1140c59f0 con 0x7fb0fc004850 2023-10-16T09:44:34.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb11a576700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1140ca170 con 0x7fb1140c6dd0 2023-10-16T09:44:34.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1140c9780 0x7fb1140c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1140c9780 0x7fb1140c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48224/0 (socket says 172.21.15.158:48224) 2023-10-16T09:44:34.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 -- 172.21.15.158:0/1594315999 learned_addr learned my addr 172.21.15.158:0/1594315999 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:34.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb1137fe700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140c6dd0 0x7fb1140cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb113fff700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc004850 0x7fb1141f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 -- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140c6dd0 msgr2=0x7fb1140cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140c6dd0 0x7fb1140cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 -- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc004850 msgr2=0x7fb1141f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc004850 0x7fb1141f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 -- 172.21.15.158:0/1594315999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1140c4de0 con 0x7fb1140c9780 2023-10-16T09:44:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb1137fe700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140c6dd0 0x7fb1140cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.529+0000 7fb118b13700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1140c9780 0x7fb1140c9c30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb114072db0 tx=0x7fb10c0104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.530+0000 7fb110ff9700 1 -- 172.21.15.158:0/1594315999 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c01b070 con 0x7fb1140c9780 2023-10-16T09:44:34.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.530+0000 7fb110ff9700 1 -- 172.21.15.158:0/1594315999 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c00f040 con 0x7fb1140c9780 2023-10-16T09:44:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.530+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1140c50c0 con 0x7fb1140c9780 2023-10-16T09:44:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.530+0000 7fb110ff9700 1 -- 172.21.15.158:0/1594315999 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c00e050 con 0x7fb1140c9780 2023-10-16T09:44:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.530+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1140c5680 con 0x7fb1140c9780 2023-10-16T09:44:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.531+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1140c5840 con 0x7fb1140c9780 2023-10-16T09:44:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.531+0000 7fb110ff9700 1 -- 172.21.15.158:0/1594315999 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb10c015440 con 0x7fb1140c9780 2023-10-16T09:44:34.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.531+0000 7fb110ff9700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb10006af00 0x7fb10006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.532+0000 7fb113fff700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb10006af00 0x7fb10006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.532+0000 7fb110ff9700 1 -- 172.21.15.158:0/1594315999 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb10c0892f0 con 0x7fb1140c9780 2023-10-16T09:44:34.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.532+0000 7fb113fff700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb10006af00 0x7fb10006d3c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb108009b40 tx=0x7fb108008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:34.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.536+0000 7fb110ff9700 1 -- 172.21.15.158:0/1594315999 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb10c0545b0 con 0x7fb1140c9780 2023-10-16T09:44:34.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.807+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb1140c5b80 con 0x7fb1140c9780 2023-10-16T09:44:34.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.807+0000 7fb110ff9700 1 -- 172.21.15.158:0/1594315999 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v5) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fb10c057bd0 con 0x7fb1140c9780 2023-10-16T09:44:34.810 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:34.810 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:44:34.810 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T09:44:34.810 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.811 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T09:44:34.811 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T09:44:34.811 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.811 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.812 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:44:34.812 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:44:34.812 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.812 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.812 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.813 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T09:44:34.813 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T09:44:34.813 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.814 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.814 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:44:34.814 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:44:34.814 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.815 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.815 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.818 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T09:44:34.818 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T09:44:34.818 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.818 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.819 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:44:34.819 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:44:34.819 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.819 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.819 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.820 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.820 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T09:44:34.820 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T09:44:34.820 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.821 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.821 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:44:34.821 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:44:34.821 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.821 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.821 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.822 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.822 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T09:44:34.822 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T09:44:34.822 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.823 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.823 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.823 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.823 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.824 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.824 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T09:44:34.824 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T09:44:34.824 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.824 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.825 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.825 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.825 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.825 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.825 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.826 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T09:44:34.826 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T09:44:34.826 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.826 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.826 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.827 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.827 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.827 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.827 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.827 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T09:44:34.828 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T09:44:34.828 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.828 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.828 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.828 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.829 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.829 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.829 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.829 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T09:44:34.829 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T09:44:34.829 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.830 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.830 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.830 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.830 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.830 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.831 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.831 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T09:44:34.831 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T09:44:34.831 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.831 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T09:44:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T09:44:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T09:44:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T09:44:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.835 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:44:34.835 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:44:34.835 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.835 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.836 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.836 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.836 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T09:44:34.836 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T09:44:34.836 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.836 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.837 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:44:34.837 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:44:34.837 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:44:34.837 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.838 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.838 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.838 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T09:44:34.838 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T09:44:34.838 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.839 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.839 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:44:34.839 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:44:34.839 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:44:34.839 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.840 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.840 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.840 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T09:44:34.840 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T09:44:34.840 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.841 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.841 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:44:34.841 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:44:34.841 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:44:34.841 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.841 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.842 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.842 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T09:44:34.842 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T09:44:34.842 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.842 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:44:34.843 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:44:34.843 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:44:34.843 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:44:34.843 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.843 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.844 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.844 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T09:44:34.844 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T09:44:34.844 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.844 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:44:34.844 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:44:34.845 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:44:34.845 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.845 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.845 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.845 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.846 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.bootstrap-mds", 2023-10-16T09:44:34.846 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCnBS1lGAa4IxAA+5Y+noavwacFApC8B+AE4A==", 2023-10-16T09:44:34.846 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.846 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile bootstrap-mds" 2023-10-16T09:44:34.846 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.847 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.847 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.847 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.bootstrap-mgr", 2023-10-16T09:44:34.847 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCnBS1lCA+4IxAANPSlSftjDd+z147GXoLN4A==", 2023-10-16T09:44:34.847 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.848 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-16T09:44:34.848 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.848 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.848 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.848 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.bootstrap-osd", 2023-10-16T09:44:34.849 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCnBS1l8ha4IxAAU0MtWFPGVyV3xJme9/cnRA==", 2023-10-16T09:44:34.849 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.849 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile bootstrap-osd" 2023-10-16T09:44:34.849 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.849 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.849 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.850 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.bootstrap-rbd", 2023-10-16T09:44:34.850 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCnBS1lnh64IxAAvNgMFKAsRz5B+N00fUxE0g==", 2023-10-16T09:44:34.850 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.850 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-16T09:44:34.850 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.851 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.851 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.851 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-16T09:44:34.851 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCnBS1lqya4IxAAkgPyUQteaK5gCDWzLmUBaw==", 2023-10-16T09:44:34.851 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.852 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-16T09:44:34.852 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.852 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.852 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.852 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.bootstrap-rgw", 2023-10-16T09:44:34.853 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCnBS1lni64IxAAA3B9WbzNazLdJJGB50WJNA==", 2023-10-16T09:44:34.853 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.853 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-16T09:44:34.853 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.853 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.853 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.854 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T09:44:34.854 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T09:44:34.854 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.854 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:44:34.855 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:44:34.855 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.855 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.855 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:44:34.855 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:34.856 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T09:44:34.856 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T09:44:34.856 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:44:34.856 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:44:34.856 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:44:34.856 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:44:34.857 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.857 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:34.857 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T09:44:34.857 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:44:34.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb10006af00 msgr2=0x7fb10006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb10006af00 0x7fb10006d3c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb108009b40 tx=0x7fb108008040 comp rx=0 tx=0).stop 2023-10-16T09:44:34.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1140c9780 msgr2=0x7fb1140c9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1140c9780 0x7fb1140c9c30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb114072db0 tx=0x7fb10c0104a0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 shutdown_connections 2023-10-16T09:44:34.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1140c9780 0x7fb1140c9c30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0fc004850 0x7fb1141f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb10006af00 0x7fb10006d3c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 --2- 172.21.15.158:0/1594315999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1140c6dd0 0x7fb1140cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 >> 172.21.15.158:0/1594315999 conn(0x7fb1140d02b0 msgr2=0x7fb1140ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:34.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 shutdown_connections 2023-10-16T09:44:34.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.810+0000 7fb11a576700 1 -- 172.21.15.158:0/1594315999 wait complete. 2023-10-16T09:44:34.861 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-16T09:44:34.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.981+0000 7f56bb5ca700 1 Processor -- start 2023-10-16T09:44:34.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.981+0000 7f56bb5ca700 1 -- start start 2023-10-16T09:44:34.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56bb5ca700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56b406e480 0x7f56b40b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56bb5ca700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f56b40cf650 0x7f56b40cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56bb5ca700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b40af600 con 0x7f56b415d990 2023-10-16T09:44:34.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56bb5ca700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40d0020 con 0x7f56b406e480 2023-10-16T09:44:34.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56bb5ca700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40b2c90 con 0x7f56b40cf650 2023-10-16T09:44:34.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56b406e480 0x7f56b40b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56b406e480 0x7f56b40b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48228/0 (socket says 172.21.15.158:48228) 2023-10-16T09:44:34.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 -- 172.21.15.158:0/3306738866 learned_addr learned my addr 172.21.15.158:0/3306738866 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:34.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b9b67700 1 --2- 172.21.15.158:0/3306738866 >> v2:172.21.15.172:3300/0 conn(0x7f56b40cf650 0x7f56b40cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 -- 172.21.15.158:0/3306738866 >> v2:172.21.15.172:3300/0 conn(0x7f56b40cf650 msgr2=0x7f56b40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:34.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 --2- 172.21.15.158:0/3306738866 >> v2:172.21.15.172:3300/0 conn(0x7f56b40cf650 0x7f56b40cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 -- 172.21.15.158:0/3306738866 >> v1:172.21.15.172:6789/0 conn(0x7f56b415d990 legacy=0x7f56b40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:34.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 -- 172.21.15.158:0/3306738866 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b406e9b0 con 0x7f56b406e480 2023-10-16T09:44:34.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b9b67700 1 --2- 172.21.15.158:0/3306738866 >> v2:172.21.15.172:3300/0 conn(0x7f56b40cf650 0x7f56b40cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:34.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56b8b65700 1 --2- 172.21.15.158:0/3306738866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b406e480 0x7f56b40b2750 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f56b4070bb0 tx=0x7f56a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14581a6af7366d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:34.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56abfff700 1 -- 172.21.15.158:0/3306738866 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4004230 con 0x7f56b406e480 2023-10-16T09:44:34.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56abfff700 1 -- 172.21.15.158:0/3306738866 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a4004390 con 0x7f56b406e480 2023-10-16T09:44:34.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.982+0000 7f56abfff700 1 -- 172.21.15.158:0/3306738866 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a40036a0 con 0x7f56b406e480 2023-10-16T09:44:34.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3306738866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b406e480 msgr2=0x7f56b40b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3306738866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b406e480 0x7f56b40b2750 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f56b4070bb0 tx=0x7f56a4009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3306738866 shutdown_connections 2023-10-16T09:44:34.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3306738866 >> v2:172.21.15.172:3300/0 conn(0x7f56b40cf650 0x7f56b40cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3306738866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b406e480 0x7f56b40b2750 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3306738866 >> 172.21.15.158:0/3306738866 conn(0x7f56b406b9e0 msgr2=0x7f56b406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:34.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3306738866 shutdown_connections 2023-10-16T09:44:34.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.983+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3306738866 wait complete. 2023-10-16T09:44:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.984+0000 7f56bb5ca700 1 Processor -- start 2023-10-16T09:44:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.984+0000 7f56bb5ca700 1 -- start start 2023-10-16T09:44:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56bb5ca700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56b406e480 0x7f56b40c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56bb5ca700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b40cf650 0x7f56b41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56bb5ca700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56b415d990 0x7f56b40cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56bb5ca700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40ca380 con 0x7f56b40cf650 2023-10-16T09:44:34.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56bb5ca700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40c74a0 con 0x7f56b406e480 2023-10-16T09:44:34.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56bb5ca700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40cb490 con 0x7f56b415d990 2023-10-16T09:44:34.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b40cf650 0x7f56b41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b40cf650 0x7f56b41529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48234/0 (socket says 172.21.15.158:48234) 2023-10-16T09:44:34.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 -- 172.21.15.158:0/3752852582 learned_addr learned my addr 172.21.15.158:0/3752852582 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:34.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b9b67700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56b415d990 0x7f56b40cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b9366700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56b406e480 0x7f56b40c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 -- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56b415d990 msgr2=0x7f56b40cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56b415d990 0x7f56b40cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 -- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56b406e480 msgr2=0x7f56b40c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:34.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56b406e480 0x7f56b40c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:34.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b8b65700 1 -- 172.21.15.158:0/3752852582 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a4009550 con 0x7f56b40cf650 2023-10-16T09:44:34.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.985+0000 7f56b9366700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56b406e480 0x7f56b40c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:34.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.986+0000 7f56b8b65700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b40cf650 0x7f56b41529a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f56a4004000 tx=0x7f56a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:34.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.986+0000 7f56a9ffb700 1 -- 172.21.15.158:0/3752852582 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4003d80 con 0x7f56b40cf650 2023-10-16T09:44:34.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.986+0000 7f56a9ffb700 1 -- 172.21.15.158:0/3752852582 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a401f070 con 0x7f56b40cf650 2023-10-16T09:44:34.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.986+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b40cb770 con 0x7f56b40cf650 2023-10-16T09:44:34.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.986+0000 7f56a9ffb700 1 -- 172.21.15.158:0/3752852582 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4011870 con 0x7f56b40cf650 2023-10-16T09:44:34.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.986+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b40cbd30 con 0x7f56b40cf650 2023-10-16T09:44:34.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.987+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b40c7650 con 0x7f56b40cf650 2023-10-16T09:44:34.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.987+0000 7f56a9ffb700 1 -- 172.21.15.158:0/3752852582 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f56a4021030 con 0x7f56b40cf650 2023-10-16T09:44:34.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.988+0000 7f56a9ffb700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f569806af00 0x7f569806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:34.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.988+0000 7f56a9ffb700 1 -- 172.21.15.158:0/3752852582 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f56a408a980 con 0x7f56b40cf650 2023-10-16T09:44:34.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.988+0000 7f56b9366700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f569806af00 0x7f569806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:34.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.988+0000 7f56b9366700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f569806af00 0x7f569806d3c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009930 tx=0x7f56b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:34.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:34.992+0000 7f56a9ffb700 1 -- 172.21.15.158:0/3752852582 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56b40c7650 con 0x7f56b40cf650 2023-10-16T09:44:35.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.266+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f56b40c7610 con 0x7f56b40cf650 2023-10-16T09:44:35.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.268+0000 7f56a9ffb700 1 -- 172.21.15.158:0/3752852582 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f56a4055cd0 con 0x7f56b40cf650 2023-10-16T09:44:35.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f569806af00 msgr2=0x7f569806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f569806af00 0x7f569806d3c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009930 tx=0x7f56b0008040 comp rx=0 tx=0).stop 2023-10-16T09:44:35.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b40cf650 msgr2=0x7f56b41529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b40cf650 0x7f56b41529a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f56a4004000 tx=0x7f56a4004750 comp rx=0 tx=0).stop 2023-10-16T09:44:35.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 shutdown_connections 2023-10-16T09:44:35.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f569806af00 0x7f569806d3c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56b415d990 0x7f56b40cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56b406e480 0x7f56b40c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 --2- 172.21.15.158:0/3752852582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56b40cf650 0x7f56b41529a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 >> 172.21.15.158:0/3752852582 conn(0x7f56b406b9e0 msgr2=0x7f56b40b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:35.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.270+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 shutdown_connections 2023-10-16T09:44:35.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.271+0000 7f56bb5ca700 1 -- 172.21.15.158:0/3752852582 wait complete. 2023-10-16T09:44:35.284 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-16T09:44:35.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.403+0000 7f205f40e700 1 Processor -- start 2023-10-16T09:44:35.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.404+0000 7f205f40e700 1 -- start start 2023-10-16T09:44:35.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.404+0000 7f205f40e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f205806e480 0x7f20580b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.404+0000 7f205f40e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20580cf650 0x7f20580cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.404+0000 7f205f40e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20580af600 con 0x7f205815d990 2023-10-16T09:44:35.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.404+0000 7f205f40e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20580b2c90 con 0x7f20580cf650 2023-10-16T09:44:35.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.404+0000 7f205f40e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205806e9b0 con 0x7f205806e480 2023-10-16T09:44:35.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d1aa700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f205815d990 0x7f20580b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48416/0 (socket says 172.21.15.158:48416) 2023-10-16T09:44:35.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d1aa700 1 -- 172.21.15.158:0/2456393313 learned_addr learned my addr 172.21.15.158:0/2456393313 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:35.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d9ab700 1 --2- 172.21.15.158:0/2456393313 >> v2:172.21.15.158:3300/0 conn(0x7f20580cf650 0x7f20580cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f204ffff700 1 -- 172.21.15.158:0/2456393313 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302589115 0 0) 0x7f20580af600 con 0x7f205815d990 2023-10-16T09:44:35.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f204ffff700 1 -- 172.21.15.158:0/2456393313 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20440035b0 con 0x7f205815d990 2023-10-16T09:44:35.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d9ab700 1 -- 172.21.15.158:0/2456393313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f205806e480 msgr2=0x7f20580b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:35.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205c9a9700 1 --2- 172.21.15.158:0/2456393313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f205806e480 0x7f20580b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d9ab700 1 --2- 172.21.15.158:0/2456393313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f205806e480 0x7f20580b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d9ab700 1 -- 172.21.15.158:0/2456393313 >> v1:172.21.15.158:6789/0 conn(0x7f205815d990 legacy=0x7f20580b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d9ab700 1 -- 172.21.15.158:0/2456393313 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20580ce890 con 0x7f20580cf650 2023-10-16T09:44:35.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205c9a9700 1 --2- 172.21.15.158:0/2456393313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f205806e480 0x7f20580b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:35.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f205d9ab700 1 --2- 172.21.15.158:0/2456393313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20580cf650 0x7f20580cfb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2050009d90 tx=0x7f205000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f09023494f3c99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:35.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f204ffff700 1 -- 172.21.15.158:0/2456393313 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205000e070 con 0x7f20580cf650 2023-10-16T09:44:35.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f204ffff700 1 -- 172.21.15.158:0/2456393313 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205000bb20 con 0x7f20580cf650 2023-10-16T09:44:35.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.405+0000 7f204ffff700 1 -- 172.21.15.158:0/2456393313 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050012430 con 0x7f20580cf650 2023-10-16T09:44:35.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 -- 172.21.15.158:0/2456393313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20580cf650 msgr2=0x7f20580cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 --2- 172.21.15.158:0/2456393313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20580cf650 0x7f20580cfb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2050009d90 tx=0x7f205000b330 comp rx=0 tx=0).stop 2023-10-16T09:44:35.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 -- 172.21.15.158:0/2456393313 shutdown_connections 2023-10-16T09:44:35.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 --2- 172.21.15.158:0/2456393313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20580cf650 0x7f20580cfb00 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 --2- 172.21.15.158:0/2456393313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f205806e480 0x7f20580b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 -- 172.21.15.158:0/2456393313 >> 172.21.15.158:0/2456393313 conn(0x7f205806b9e0 msgr2=0x7f205806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:35.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 -- 172.21.15.158:0/2456393313 shutdown_connections 2023-10-16T09:44:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.406+0000 7f205f40e700 1 -- 172.21.15.158:0/2456393313 wait complete. 2023-10-16T09:44:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.407+0000 7f205f40e700 1 Processor -- start 2023-10-16T09:44:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.407+0000 7f205f40e700 1 -- start start 2023-10-16T09:44:35.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205f40e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f205806e480 0x7f20580af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205f40e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20580cf650 0x7f20580ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205f40e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f205815d990 0x7f20580ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205f40e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20580ca830 con 0x7f205815d990 2023-10-16T09:44:35.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205f40e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20580afd90 con 0x7f205806e480 2023-10-16T09:44:35.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205f40e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20580c9cb0 con 0x7f20580cf650 2023-10-16T09:44:35.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f205815d990 0x7f20580ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f205815d990 0x7f20580ad9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48250/0 (socket says 172.21.15.158:48250) 2023-10-16T09:44:35.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 -- 172.21.15.158:0/372498589 learned_addr learned my addr 172.21.15.158:0/372498589 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:35.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205c9a9700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20580cf650 0x7f20580ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d1aa700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f205806e480 0x7f20580af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 -- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20580cf650 msgr2=0x7f20580ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20580cf650 0x7f20580ad470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 -- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f205806e480 msgr2=0x7f20580af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f205806e480 0x7f20580af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 -- 172.21.15.158:0/372498589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2050009a70 con 0x7f205815d990 2023-10-16T09:44:35.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d1aa700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f205806e480 0x7f20580af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:35.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205c9a9700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20580cf650 0x7f20580ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:35.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205d9ab700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f205815d990 0x7f20580ad9b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2050014040 tx=0x7f2050004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:35.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f204dffb700 1 -- 172.21.15.158:0/372498589 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050012430 con 0x7f205815d990 2023-10-16T09:44:35.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f204dffb700 1 -- 172.21.15.158:0/372498589 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2050020070 con 0x7f205815d990 2023-10-16T09:44:35.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.408+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20580c9f90 con 0x7f205815d990 2023-10-16T09:44:35.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.409+0000 7f204dffb700 1 -- 172.21.15.158:0/372498589 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050003ea0 con 0x7f205815d990 2023-10-16T09:44:35.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.409+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20580ca550 con 0x7f205815d990 2023-10-16T09:44:35.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.410+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20580b5060 con 0x7f205815d990 2023-10-16T09:44:35.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.410+0000 7f204dffb700 1 -- 172.21.15.158:0/372498589 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2050004280 con 0x7f205815d990 2023-10-16T09:44:35.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.411+0000 7f204dffb700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f203c06af00 0x7f203c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.411+0000 7f204dffb700 1 -- 172.21.15.158:0/372498589 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f2050022070 con 0x7f205815d990 2023-10-16T09:44:35.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.411+0000 7f205d1aa700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f203c06af00 0x7f203c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.411+0000 7f205d1aa700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f203c06af00 0x7f203c06d3c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20480011e0 tx=0x7f2048009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:35.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.415+0000 7f204dffb700 1 -- 172.21.15.158:0/372498589 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f20500bdfe0 con 0x7f205815d990 2023-10-16T09:44:35.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.690+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f20580681b0 con 0x7f205815d990 2023-10-16T09:44:35.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.692+0000 7f204dffb700 1 -- 172.21.15.158:0/372498589 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2050056480 con 0x7f205815d990 2023-10-16T09:44:35.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f203c06af00 msgr2=0x7f203c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f203c06af00 0x7f203c06d3c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20480011e0 tx=0x7f2048009040 comp rx=0 tx=0).stop 2023-10-16T09:44:35.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f205815d990 msgr2=0x7f20580ad9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f205815d990 0x7f20580ad9b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2050014040 tx=0x7f2050004ad0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 shutdown_connections 2023-10-16T09:44:35.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f205815d990 0x7f20580ad9b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f205806e480 0x7f20580af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f203c06af00 0x7f203c06d3c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 --2- 172.21.15.158:0/372498589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20580cf650 0x7f20580ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 >> 172.21.15.158:0/372498589 conn(0x7f205806b9e0 msgr2=0x7f20580b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:35.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 shutdown_connections 2023-10-16T09:44:35.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.694+0000 7f205f40e700 1 -- 172.21.15.158:0/372498589 wait complete. 2023-10-16T09:44:35.707 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-16T09:44:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.827+0000 7f09e8fdd700 1 Processor -- start 2023-10-16T09:44:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.827+0000 7f09e8fdd700 1 -- start start 2023-10-16T09:44:35.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e8fdd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09e415a6d0 0x7f09e40ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e8fdd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f09e40cd360 0x7f09e406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e8fdd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e415ec70 con 0x7f09e40cfe30 2023-10-16T09:44:35.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e8fdd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e40afcd0 con 0x7f09e415a6d0 2023-10-16T09:44:35.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e8fdd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e415f1b0 con 0x7f09e40cd360 2023-10-16T09:44:35.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09e415a6d0 0x7f09e40ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e1d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f09e40cfe30 0x7f09e40ccc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48430/0 (socket says 172.21.15.158:48430) 2023-10-16T09:44:35.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.828+0000 7f09e1d9b700 1 -- 172.21.15.158:0/410473560 learned_addr learned my addr 172.21.15.158:0/410473560 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:35.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e159a700 1 -- 172.21.15.158:0/410473560 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123100728 0 0) 0x7f09e415ec70 con 0x7f09e40cfe30 2023-10-16T09:44:35.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e2d9d700 1 --2- 172.21.15.158:0/410473560 >> v2:172.21.15.172:3300/0 conn(0x7f09e40cd360 0x7f09e406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e159a700 1 -- 172.21.15.158:0/410473560 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc0035b0 con 0x7f09e40cfe30 2023-10-16T09:44:35.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e259c700 1 -- 172.21.15.158:0/410473560 >> v2:172.21.15.172:3300/0 conn(0x7f09e40cd360 msgr2=0x7f09e406ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:35.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e259c700 1 --2- 172.21.15.158:0/410473560 >> v2:172.21.15.172:3300/0 conn(0x7f09e40cd360 0x7f09e406ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e259c700 1 -- 172.21.15.158:0/410473560 >> v1:172.21.15.158:6789/0 conn(0x7f09e40cfe30 legacy=0x7f09e40ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e259c700 1 -- 172.21.15.158:0/410473560 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e40cab50 con 0x7f09e415a6d0 2023-10-16T09:44:35.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e2d9d700 1 --2- 172.21.15.158:0/410473560 >> v2:172.21.15.172:3300/0 conn(0x7f09e40cd360 0x7f09e406ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:35.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e259c700 1 --2- 172.21.15.158:0/410473560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e415a6d0 0x7f09e40ce2d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f09e40c3ed0 tx=0x7f09d8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962665320ce43c03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:35.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e159a700 1 -- 172.21.15.158:0/410473560 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8011830 con 0x7f09e415a6d0 2023-10-16T09:44:35.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e159a700 1 -- 172.21.15.158:0/410473560 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d8011990 con 0x7f09e415a6d0 2023-10-16T09:44:35.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.829+0000 7f09e159a700 1 -- 172.21.15.158:0/410473560 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8011d00 con 0x7f09e415a6d0 2023-10-16T09:44:35.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 -- 172.21.15.158:0/410473560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e415a6d0 msgr2=0x7f09e40ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/410473560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e415a6d0 0x7f09e40ce2d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f09e40c3ed0 tx=0x7f09d8009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:35.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 -- 172.21.15.158:0/410473560 shutdown_connections 2023-10-16T09:44:35.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/410473560 >> v2:172.21.15.172:3300/0 conn(0x7f09e40cd360 0x7f09e406ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/410473560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e415a6d0 0x7f09e40ce2d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 -- 172.21.15.158:0/410473560 >> 172.21.15.158:0/410473560 conn(0x7f09e406b9e0 msgr2=0x7f09e406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:35.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 -- 172.21.15.158:0/410473560 shutdown_connections 2023-10-16T09:44:35.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.830+0000 7f09e8fdd700 1 -- 172.21.15.158:0/410473560 wait complete. 2023-10-16T09:44:35.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.831+0000 7f09e8fdd700 1 Processor -- start 2023-10-16T09:44:35.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.831+0000 7f09e8fdd700 1 -- start start 2023-10-16T09:44:35.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e8fdd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09e40cd360 0x7f09e41586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e8fdd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e40cfe30 0x7f09e41474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e8fdd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09e415a6d0 0x7f09e4147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e8fdd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40c5d70 con 0x7f09e40cfe30 2023-10-16T09:44:35.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e8fdd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40c68b0 con 0x7f09e40cd360 2023-10-16T09:44:35.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e8fdd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e4147f60 con 0x7f09e415a6d0 2023-10-16T09:44:35.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e40cfe30 0x7f09e41474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e40cfe30 0x7f09e41474e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48262/0 (socket says 172.21.15.158:48262) 2023-10-16T09:44:35.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 -- 172.21.15.158:0/1130443589 learned_addr learned my addr 172.21.15.158:0/1130443589 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e2d9d700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09e415a6d0 0x7f09e4147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e259c700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09e40cd360 0x7f09e41586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 -- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09e415a6d0 msgr2=0x7f09e4147a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09e415a6d0 0x7f09e4147a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 -- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09e40cd360 msgr2=0x7f09e41586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09e40cd360 0x7f09e41586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:35.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 -- 172.21.15.158:0/1130443589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d80094a0 con 0x7f09e40cfe30 2023-10-16T09:44:35.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e259c700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09e40cd360 0x7f09e41586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:35.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e2d9d700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09e415a6d0 0x7f09e4147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:35.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e1d9b700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e40cfe30 0x7f09e41474e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f09d400c4d0 tx=0x7f09d400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:35.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09d2ffd700 1 -- 172.21.15.158:0/1130443589 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4005d70 con 0x7f09e40cfe30 2023-10-16T09:44:35.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09d2ffd700 1 -- 172.21.15.158:0/1130443589 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d40070d0 con 0x7f09e40cfe30 2023-10-16T09:44:35.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e41f5130 con 0x7f09e40cfe30 2023-10-16T09:44:35.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.832+0000 7f09d2ffd700 1 -- 172.21.15.158:0/1130443589 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4007480 con 0x7f09e40cfe30 2023-10-16T09:44:35.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.833+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e40c6540 con 0x7f09e40cfe30 2023-10-16T09:44:35.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.834+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e40c66d0 con 0x7f09e40cfe30 2023-10-16T09:44:35.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.834+0000 7f09d2ffd700 1 -- 172.21.15.158:0/1130443589 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f09d40075e0 con 0x7f09e40cfe30 2023-10-16T09:44:35.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.834+0000 7f09d2ffd700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c406af00 0x7f09c406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:35.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.834+0000 7f09d2ffd700 1 -- 172.21.15.158:0/1130443589 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f09d40888a0 con 0x7f09e40cfe30 2023-10-16T09:44:35.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.834+0000 7f09e259c700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c406af00 0x7f09c406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:35.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.835+0000 7f09e259c700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c406af00 0x7f09c406d3c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f09d80090b0 tx=0x7f09d801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:35.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:35.839+0000 7f09d2ffd700 1 -- 172.21.15.158:0/1130443589 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09d4053cb0 con 0x7f09e40cfe30 2023-10-16T09:44:36.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.109+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f09e40c6090 con 0x7f09e40cfe30 2023-10-16T09:44:36.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.111+0000 7f09d2ffd700 1 -- 172.21.15.158:0/1130443589 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f09d4017020 con 0x7f09e40cfe30 2023-10-16T09:44:36.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c406af00 msgr2=0x7f09c406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c406af00 0x7f09c406d3c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f09d80090b0 tx=0x7f09d801c040 comp rx=0 tx=0).stop 2023-10-16T09:44:36.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e40cfe30 msgr2=0x7f09e41474e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e40cfe30 0x7f09e41474e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f09d400c4d0 tx=0x7f09d400c890 comp rx=0 tx=0).stop 2023-10-16T09:44:36.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 shutdown_connections 2023-10-16T09:44:36.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f09c406af00 0x7f09c406d3c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09e415a6d0 0x7f09e4147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09e40cd360 0x7f09e41586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 --2- 172.21.15.158:0/1130443589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09e40cfe30 0x7f09e41474e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 >> 172.21.15.158:0/1130443589 conn(0x7f09e406b9e0 msgr2=0x7f09e415e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:36.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 shutdown_connections 2023-10-16T09:44:36.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.113+0000 7f09e8fdd700 1 -- 172.21.15.158:0/1130443589 wait complete. 2023-10-16T09:44:36.127 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.248+0000 7f803498f700 1 Processor -- start 2023-10-16T09:44:36.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.248+0000 7f803498f700 1 -- start start 2023-10-16T09:44:36.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f803498f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f803498f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f803006e480 0x7f80300b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f803498f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80300af600 con 0x7f80300b3760 2023-10-16T09:44:36.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f803498f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80300b2c90 con 0x7f803006e480 2023-10-16T09:44:36.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f803498f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803006e9b0 con 0x7f803015d990 2023-10-16T09:44:36.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802dd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f803006e480 0x7f80300b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802dd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f803006e480 0x7f80300b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:49450/0 (socket says 172.21.15.158:49450) 2023-10-16T09:44:36.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802dd9b700 1 -- 172.21.15.158:0/958797651 learned_addr learned my addr 172.21.15.158:0/958797651 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:36.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802ed9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f80300b3760 0x7f80300cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:52350/0 (socket says 172.21.15.158:52350) 2023-10-16T09:44:36.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802e59c700 1 -- 172.21.15.158:0/958797651 >> v2:172.21.15.172:3300/0 conn(0x7f803006e480 msgr2=0x7f80300b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:36.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802e59c700 1 --2- 172.21.15.158:0/958797651 >> v2:172.21.15.172:3300/0 conn(0x7f803006e480 0x7f80300b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802e59c700 1 -- 172.21.15.158:0/958797651 >> v1:172.21.15.172:6789/0 conn(0x7f80300b3760 legacy=0x7f80300cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802d59a700 1 -- 172.21.15.158:0/958797651 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361564040 0 0) 0x7f80300af600 con 0x7f80300b3760 2023-10-16T09:44:36.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802e59c700 1 -- 172.21.15.158:0/958797651 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80300ce890 con 0x7f803015d990 2023-10-16T09:44:36.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.249+0000 7f802dd9b700 1 --2- 172.21.15.158:0/958797651 >> v2:172.21.15.172:3300/0 conn(0x7f803006e480 0x7f80300b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:36.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f802e59c700 1 --2- 172.21.15.158:0/958797651 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300b2750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8024009840 tx=0x7f8024009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b964c63e2899edb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:36.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f802d59a700 1 -- 172.21.15.158:0/958797651 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024011830 con 0x7f803015d990 2023-10-16T09:44:36.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f802d59a700 1 -- 172.21.15.158:0/958797651 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8024011990 con 0x7f803015d990 2023-10-16T09:44:36.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f802d59a700 1 -- 172.21.15.158:0/958797651 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024011d20 con 0x7f803015d990 2023-10-16T09:44:36.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f803498f700 1 -- 172.21.15.158:0/958797651 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 msgr2=0x7f80300b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f803498f700 1 --2- 172.21.15.158:0/958797651 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300b2750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8024009840 tx=0x7f8024009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:36.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f803498f700 1 -- 172.21.15.158:0/958797651 shutdown_connections 2023-10-16T09:44:36.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f803498f700 1 --2- 172.21.15.158:0/958797651 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300b2750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f803498f700 1 --2- 172.21.15.158:0/958797651 >> v2:172.21.15.172:3300/0 conn(0x7f803006e480 0x7f80300b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f803498f700 1 -- 172.21.15.158:0/958797651 >> 172.21.15.158:0/958797651 conn(0x7f803006b9e0 msgr2=0x7f803006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:36.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.250+0000 7f803498f700 1 -- 172.21.15.158:0/958797651 shutdown_connections 2023-10-16T09:44:36.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.251+0000 7f803498f700 1 -- 172.21.15.158:0/958797651 wait complete. 2023-10-16T09:44:36.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.252+0000 7f803498f700 1 Processor -- start 2023-10-16T09:44:36.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.252+0000 7f803498f700 1 -- start start 2023-10-16T09:44:36.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f803498f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803006e480 0x7f8030152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f803498f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80300b3760 0x7f80300afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f803498f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f803498f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80300b08b0 con 0x7f80300b3760 2023-10-16T09:44:36.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f803498f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80300b0b90 con 0x7f803006e480 2023-10-16T09:44:36.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f803498f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80301544d0 con 0x7f803015d990 2023-10-16T09:44:36.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80300b3760 0x7f80300afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80300b3760 0x7f80300afe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48276/0 (socket says 172.21.15.158:48276) 2023-10-16T09:44:36.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 -- 172.21.15.158:0/3108789255 learned_addr learned my addr 172.21.15.158:0/3108789255 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:36.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802ed9d700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802e59c700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803006e480 0x7f8030152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 -- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 msgr2=0x7f80300cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 -- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803006e480 msgr2=0x7f8030152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803006e480 0x7f8030152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 -- 172.21.15.158:0/3108789255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8020009550 con 0x7f80300b3760 2023-10-16T09:44:36.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f802dd9b700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80300b3760 0x7f80300afe10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f802000ebe0 tx=0x7f802000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:36.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f801effd700 1 -- 172.21.15.158:0/3108789255 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802000c850 con 0x7f80300b3760 2023-10-16T09:44:36.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f801effd700 1 -- 172.21.15.158:0/3108789255 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802000c9b0 con 0x7f80300b3760 2023-10-16T09:44:36.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.253+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80240094a0 con 0x7f80300b3760 2023-10-16T09:44:36.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.254+0000 7f801effd700 1 -- 172.21.15.158:0/3108789255 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802000cd60 con 0x7f80300b3760 2023-10-16T09:44:36.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.254+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8030153070 con 0x7f80300b3760 2023-10-16T09:44:36.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.255+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80300681b0 con 0x7f80300b3760 2023-10-16T09:44:36.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.255+0000 7f801effd700 1 -- 172.21.15.158:0/3108789255 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8020013470 con 0x7f80300b3760 2023-10-16T09:44:36.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.255+0000 7f801effd700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.255+0000 7f801effd700 1 -- 172.21.15.158:0/3108789255 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f80200893d0 con 0x7f80300b3760 2023-10-16T09:44:36.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.255+0000 7f802e59c700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.256+0000 7f802e59c700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8024000c00 tx=0x7f802401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:36.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.260+0000 7f801effd700 1 -- 172.21.15.158:0/3108789255 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8020054720 con 0x7f80300b3760 2023-10-16T09:44:36.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.534+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f80300d0f30 con 0x7f80300b3760 2023-10-16T09:44:36.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.536+0000 7f801effd700 1 -- 172.21.15.158:0/3108789255 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8020057d40 con 0x7f80300b3760 2023-10-16T09:44:36.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 msgr2=0x7f801006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8024000c00 tx=0x7f802401c040 comp rx=0 tx=0).stop 2023-10-16T09:44:36.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80300b3760 msgr2=0x7f80300afe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80300b3760 0x7f80300afe10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f802000ebe0 tx=0x7f802000eef0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 shutdown_connections 2023-10-16T09:44:36.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f801006af00 0x7f801006d3c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f803015d990 0x7f80300cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803006e480 0x7f8030152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 --2- 172.21.15.158:0/3108789255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80300b3760 0x7f80300afe10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 >> 172.21.15.158:0/3108789255 conn(0x7f803006b9e0 msgr2=0x7f80300acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:36.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 shutdown_connections 2023-10-16T09:44:36.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.539+0000 7f803498f700 1 -- 172.21.15.158:0/3108789255 wait complete. 2023-10-16T09:44:36.553 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-16T09:44:36.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.673+0000 7f7d1824c700 1 Processor -- start 2023-10-16T09:44:36.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.673+0000 7f7d1824c700 1 -- start start 2023-10-16T09:44:36.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d1824c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d100d0ae0 0x7f7d100b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d1824c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7d100b4950 0x7f7d100cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d1824c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1015f990 con 0x7f7d1015eb10 2023-10-16T09:44:36.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d1824c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100cdef0 con 0x7f7d100d0ae0 2023-10-16T09:44:36.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d1824c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100cd070 con 0x7f7d100b4950 2023-10-16T09:44:36.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d157e7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d100d0ae0 0x7f7d100b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d15fe8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7d1015eb10 0x7f7d1015f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48458/0 (socket says 172.21.15.158:48458) 2023-10-16T09:44:36.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d15fe8700 1 -- 172.21.15.158:0/2730120112 learned_addr learned my addr 172.21.15.158:0/2730120112 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:36.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d167e9700 1 --2- 172.21.15.158:0/2730120112 >> v2:172.21.15.172:3300/0 conn(0x7f7d100b4950 0x7f7d100cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d157e7700 1 -- 172.21.15.158:0/2730120112 >> v2:172.21.15.172:3300/0 conn(0x7f7d100b4950 msgr2=0x7f7d100cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:36.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d157e7700 1 --2- 172.21.15.158:0/2730120112 >> v2:172.21.15.172:3300/0 conn(0x7f7d100b4950 0x7f7d100cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d157e7700 1 -- 172.21.15.158:0/2730120112 >> v1:172.21.15.158:6789/0 conn(0x7f7d1015eb10 legacy=0x7f7d1015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d157e7700 1 -- 172.21.15.158:0/2730120112 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1015fc70 con 0x7f7d100d0ae0 2023-10-16T09:44:36.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d14fe6700 1 -- 172.21.15.158:0/2730120112 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237157773 0 0) 0x7f7d1015f990 con 0x7f7d1015eb10 2023-10-16T09:44:36.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.674+0000 7f7d167e9700 1 --2- 172.21.15.158:0/2730120112 >> v2:172.21.15.172:3300/0 conn(0x7f7d100b4950 0x7f7d100cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:36.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.675+0000 7f7d157e7700 1 --2- 172.21.15.158:0/2730120112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100d0ae0 0x7f7d100b4410 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7d10071670 tx=0x7f7d00009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b98743f5d907397 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:36.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d14fe6700 1 -- 172.21.15.158:0/2730120112 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00011830 con 0x7f7d100d0ae0 2023-10-16T09:44:36.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d14fe6700 1 -- 172.21.15.158:0/2730120112 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d00011990 con 0x7f7d100d0ae0 2023-10-16T09:44:36.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d14fe6700 1 -- 172.21.15.158:0/2730120112 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00011d20 con 0x7f7d100d0ae0 2023-10-16T09:44:36.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 -- 172.21.15.158:0/2730120112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100d0ae0 msgr2=0x7f7d100b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 --2- 172.21.15.158:0/2730120112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100d0ae0 0x7f7d100b4410 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7d10071670 tx=0x7f7d00009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 -- 172.21.15.158:0/2730120112 shutdown_connections 2023-10-16T09:44:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 --2- 172.21.15.158:0/2730120112 >> v2:172.21.15.172:3300/0 conn(0x7f7d100b4950 0x7f7d100cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 --2- 172.21.15.158:0/2730120112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100d0ae0 0x7f7d100b4410 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 -- 172.21.15.158:0/2730120112 >> 172.21.15.158:0/2730120112 conn(0x7f7d100ce040 msgr2=0x7f7d100ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:36.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 -- 172.21.15.158:0/2730120112 shutdown_connections 2023-10-16T09:44:36.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.677+0000 7f7d1824c700 1 -- 172.21.15.158:0/2730120112 wait complete. 2023-10-16T09:44:36.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 Processor -- start 2023-10-16T09:44:36.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 -- start start 2023-10-16T09:44:36.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100b4950 0x7f7d100c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d100d0ae0 0x7f7d101573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d1015eb10 0x7f7d10151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d10154990 con 0x7f7d100b4950 2023-10-16T09:44:36.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d101f1c00 con 0x7f7d1015eb10 2023-10-16T09:44:36.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d1824c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d101f1310 con 0x7f7d100d0ae0 2023-10-16T09:44:36.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d15fe8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100b4950 0x7f7d100c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.679+0000 7f7d15fe8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100b4950 0x7f7d100c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48288/0 (socket says 172.21.15.158:48288) 2023-10-16T09:44:36.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d15fe8700 1 -- 172.21.15.158:0/1271491222 learned_addr learned my addr 172.21.15.158:0/1271491222 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:36.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d157e7700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d100d0ae0 0x7f7d101573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d167e9700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d1015eb10 0x7f7d10151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d15fe8700 1 -- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d100d0ae0 msgr2=0x7f7d101573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d15fe8700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d100d0ae0 0x7f7d101573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d15fe8700 1 -- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d1015eb10 msgr2=0x7f7d10151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d15fe8700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d1015eb10 0x7f7d10151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d15fe8700 1 -- 172.21.15.158:0/1271491222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d000094a0 con 0x7f7d100b4950 2023-10-16T09:44:36.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d15fe8700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100b4950 0x7f7d100c7700 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00c550 tx=0x7f7d0c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:36.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d067fc700 1 -- 172.21.15.158:0/1271491222 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c004c80 con 0x7f7d100b4950 2023-10-16T09:44:36.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d067fc700 1 -- 172.21.15.158:0/1271491222 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c004e00 con 0x7f7d100b4950 2023-10-16T09:44:36.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d101f2090 con 0x7f7d100b4950 2023-10-16T09:44:36.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d067fc700 1 -- 172.21.15.158:0/1271491222 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c007310 con 0x7f7d100b4950 2023-10-16T09:44:36.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.680+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d101f1890 con 0x7f7d100b4950 2023-10-16T09:44:36.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.681+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d100681b0 con 0x7f7d100b4950 2023-10-16T09:44:36.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.681+0000 7f7d067fc700 1 -- 172.21.15.158:0/1271491222 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7d0c007470 con 0x7f7d100b4950 2023-10-16T09:44:36.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.682+0000 7f7d067fc700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7cf406af00 0x7f7cf406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:36.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.682+0000 7f7d157e7700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7cf406af00 0x7f7cf406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:36.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.682+0000 7f7d067fc700 1 -- 172.21.15.158:0/1271491222 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7d0c088f70 con 0x7f7d100b4950 2023-10-16T09:44:36.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.682+0000 7f7d157e7700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7cf406af00 0x7f7cf406d3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d10071670 tx=0x7f7d00008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:36.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.686+0000 7f7d067fc700 1 -- 172.21.15.158:0/1271491222 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d0c0bbfe0 con 0x7f7d100b4950 2023-10-16T09:44:36.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.958+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f7d100b0500 con 0x7f7d100b4950 2023-10-16T09:44:36.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.960+0000 7f7d067fc700 1 -- 172.21.15.158:0/1271491222 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7d0c054230 con 0x7f7d100b4950 2023-10-16T09:44:36.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.962+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7cf406af00 msgr2=0x7f7cf406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.962+0000 7f7d1824c700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7cf406af00 0x7f7cf406d3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d10071670 tx=0x7f7d00008030 comp rx=0 tx=0).stop 2023-10-16T09:44:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.962+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100b4950 msgr2=0x7f7d100c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.962+0000 7f7d1824c700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100b4950 0x7f7d100c7700 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00c550 tx=0x7f7d0c00c860 comp rx=0 tx=0).stop 2023-10-16T09:44:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 shutdown_connections 2023-10-16T09:44:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d1015eb10 0x7f7d10151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d100b4950 0x7f7d100c7700 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7cf406af00 0x7f7cf406d3c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 --2- 172.21.15.158:0/1271491222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d100d0ae0 0x7f7d101573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 >> 172.21.15.158:0/1271491222 conn(0x7f7d100ce040 msgr2=0x7f7d100b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 shutdown_connections 2023-10-16T09:44:36.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:36.963+0000 7f7d1824c700 1 -- 172.21.15.158:0/1271491222 wait complete. 2023-10-16T09:44:36.976 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-16T09:44:37.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.096+0000 7f7bf6022700 1 Processor -- start 2023-10-16T09:44:37.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7bf6022700 1 -- start start 2023-10-16T09:44:37.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7bf6022700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf015a500 0x7f7bf00b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7bf6022700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7bf00b3c70 0x7f7bf00b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7bf6022700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf00b4240 con 0x7f7bf00cfa70 2023-10-16T09:44:37.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7bf6022700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf00cd7a0 con 0x7f7bf00b3c70 2023-10-16T09:44:37.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7bf6022700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0071b20 con 0x7f7bf015a500 2023-10-16T09:44:37.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7beffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7bf00b3c70 0x7f7bf00b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7beffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7bf00b3c70 0x7f7bf00b4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48300/0 (socket says 172.21.15.158:48300) 2023-10-16T09:44:37.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.097+0000 7f7beffff700 1 -- 172.21.15.158:0/1345388866 learned_addr learned my addr 172.21.15.158:0/1345388866 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:37.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7beffff700 1 -- 172.21.15.158:0/1345388866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf015a500 msgr2=0x7f7bf00b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:37.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7beffff700 1 --2- 172.21.15.158:0/1345388866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf015a500 0x7f7bf00b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7beffff700 1 -- 172.21.15.158:0/1345388866 >> v1:172.21.15.172:6789/0 conn(0x7f7bf00cfa70 legacy=0x7f7bf015e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:37.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7beffff700 1 -- 172.21.15.158:0/1345388866 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf0071dc0 con 0x7f7bf00b3c70 2023-10-16T09:44:37.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7beffff700 1 --2- 172.21.15.158:0/1345388866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf00b3c70 0x7f7bf00b4f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7be800d150 tx=0x7f7be800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b10f19e1e9b4e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7bee7fc700 1 -- 172.21.15.158:0/1345388866 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8010070 con 0x7f7bf00b3c70 2023-10-16T09:44:37.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7bee7fc700 1 -- 172.21.15.158:0/1345388866 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be800dd80 con 0x7f7bf00b3c70 2023-10-16T09:44:37.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.098+0000 7f7bee7fc700 1 -- 172.21.15.158:0/1345388866 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be800b310 con 0x7f7bf00b3c70 2023-10-16T09:44:37.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 -- 172.21.15.158:0/1345388866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf00b3c70 msgr2=0x7f7bf00b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 --2- 172.21.15.158:0/1345388866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf00b3c70 0x7f7bf00b4f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7be800d150 tx=0x7f7be800d460 comp rx=0 tx=0).stop 2023-10-16T09:44:37.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 -- 172.21.15.158:0/1345388866 shutdown_connections 2023-10-16T09:44:37.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 --2- 172.21.15.158:0/1345388866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf00b3c70 0x7f7bf00b4f40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 --2- 172.21.15.158:0/1345388866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf015a500 0x7f7bf00b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 -- 172.21.15.158:0/1345388866 >> 172.21.15.158:0/1345388866 conn(0x7f7bf006da00 msgr2=0x7f7bf006dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:37.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 -- 172.21.15.158:0/1345388866 shutdown_connections 2023-10-16T09:44:37.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.099+0000 7f7bf6022700 1 -- 172.21.15.158:0/1345388866 wait complete. 2023-10-16T09:44:37.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.100+0000 7f7bf6022700 1 Processor -- start 2023-10-16T09:44:37.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.100+0000 7f7bf6022700 1 -- start start 2023-10-16T09:44:37.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bf6022700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf00b3c70 0x7f7bf0149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bf6022700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf00cfa70 0x7f7bf00b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bf6022700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf015a500 0x7f7bf01ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bf6022700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00b1210 con 0x7f7bf015a500 2023-10-16T09:44:37.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bf6022700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01491a0 con 0x7f7bf00cfa70 2023-10-16T09:44:37.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bf6022700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0149480 con 0x7f7bf00b3c70 2023-10-16T09:44:37.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf015a500 0x7f7bf01ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf015a500 0x7f7bf01ee280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48306/0 (socket says 172.21.15.158:48306) 2023-10-16T09:44:37.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 -- 172.21.15.158:0/509942653 learned_addr learned my addr 172.21.15.158:0/509942653 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:37.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bef7fe700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf00b3c70 0x7f7bf0149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beeffd700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf00cfa70 0x7f7bf00b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 -- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf00b3c70 msgr2=0x7f7bf0149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf00b3c70 0x7f7bf0149f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 -- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf00cfa70 msgr2=0x7f7bf00b0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf00cfa70 0x7f7bf00b0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 -- 172.21.15.158:0/509942653 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be800bd40 con 0x7f7bf015a500 2023-10-16T09:44:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beeffd700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf00cfa70 0x7f7bf00b0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7beffff700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf015a500 0x7f7bf01ee280 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00d1480 tx=0x7f7be8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7be7fff700 1 -- 172.21.15.158:0/509942653 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be800ab20 con 0x7f7bf015a500 2023-10-16T09:44:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7be7fff700 1 -- 172.21.15.158:0/509942653 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be800ac80 con 0x7f7bf015a500 2023-10-16T09:44:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.101+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf0149760 con 0x7f7bf015a500 2023-10-16T09:44:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.102+0000 7f7be7fff700 1 -- 172.21.15.158:0/509942653 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8004030 con 0x7f7bf015a500 2023-10-16T09:44:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.102+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf00b0e50 con 0x7f7bf015a500 2023-10-16T09:44:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.102+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf00b0fe0 con 0x7f7bf015a500 2023-10-16T09:44:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.103+0000 7f7be7fff700 1 -- 172.21.15.158:0/509942653 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7be801b050 con 0x7f7bf015a500 2023-10-16T09:44:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.103+0000 7f7be7fff700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd006af00 0x7f7bd006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.103+0000 7f7be7fff700 1 -- 172.21.15.158:0/509942653 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7be808c9e0 con 0x7f7bf015a500 2023-10-16T09:44:37.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.103+0000 7f7bef7fe700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd006af00 0x7f7bd006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.104+0000 7f7bef7fe700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd006af00 0x7f7bd006d3c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8006fd0 tx=0x7f7bd8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.108+0000 7f7be7fff700 1 -- 172.21.15.158:0/509942653 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7be8057d30 con 0x7f7bf015a500 2023-10-16T09:44:37.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.379+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f7bf00b4870 con 0x7f7bf015a500 2023-10-16T09:44:37.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.381+0000 7f7be7fff700 1 -- 172.21.15.158:0/509942653 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7be80190a0 con 0x7f7bf015a500 2023-10-16T09:44:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd006af00 msgr2=0x7f7bd006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd006af00 0x7f7bd006d3c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8006fd0 tx=0x7f7bd8008040 comp rx=0 tx=0).stop 2023-10-16T09:44:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf015a500 msgr2=0x7f7bf01ee280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf015a500 0x7f7bf01ee280 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00d1480 tx=0x7f7be8004ad0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 shutdown_connections 2023-10-16T09:44:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7bf015a500 0x7f7bf01ee280 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7bd006af00 0x7f7bd006d3c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bf00b3c70 0x7f7bf0149f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 --2- 172.21.15.158:0/509942653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bf00cfa70 0x7f7bf00b0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 >> 172.21.15.158:0/509942653 conn(0x7f7bf006da00 msgr2=0x7f7bf006f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 shutdown_connections 2023-10-16T09:44:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.383+0000 7f7bf6022700 1 -- 172.21.15.158:0/509942653 wait complete. 2023-10-16T09:44:37.397 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:37.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.516+0000 7ff513dfa700 1 Processor -- start 2023-10-16T09:44:37.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.516+0000 7ff513dfa700 1 -- start start 2023-10-16T09:44:37.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff513dfa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06ab40 0x7ff50c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff513dfa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff50c06e5d0 0x7ff50c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff513dfa700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff50c06af20 0x7ff50c0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff513dfa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50c071e50 con 0x7ff50c06e5d0 2023-10-16T09:44:37.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff513dfa700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50c161270 con 0x7ff50c06af20 2023-10-16T09:44:37.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff513dfa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c161510 con 0x7ff50c06ab40 2023-10-16T09:44:37.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff50c06e5d0 0x7ff50c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff50c06e5d0 0x7ff50c071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48310/0 (socket says 172.21.15.158:48310) 2023-10-16T09:44:37.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 -- 172.21.15.158:0/1729009255 learned_addr learned my addr 172.21.15.158:0/1729009255 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:37.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff512397700 1 --2- 172.21.15.158:0/1729009255 >> v2:172.21.15.172:3300/0 conn(0x7ff50c06af20 0x7ff50c0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511b96700 1 --2- 172.21.15.158:0/1729009255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06ab40 0x7ff50c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 -- 172.21.15.158:0/1729009255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06ab40 msgr2=0x7ff50c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:37.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 --2- 172.21.15.158:0/1729009255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06ab40 0x7ff50c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 -- 172.21.15.158:0/1729009255 >> v2:172.21.15.172:3300/0 conn(0x7ff50c06af20 msgr2=0x7ff50c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 --2- 172.21.15.158:0/1729009255 >> v2:172.21.15.172:3300/0 conn(0x7ff50c06af20 0x7ff50c0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 -- 172.21.15.158:0/1729009255 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c1617f0 con 0x7ff50c06e5d0 2023-10-16T09:44:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511b96700 1 --2- 172.21.15.158:0/1729009255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06ab40 0x7ff50c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff511395700 1 --2- 172.21.15.158:0/1729009255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06e5d0 0x7ff50c071910 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff50c06eaf0 tx=0x7ff508009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59792038d8b4a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff510b94700 1 -- 172.21.15.158:0/1729009255 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800b870 con 0x7ff50c06e5d0 2023-10-16T09:44:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff510b94700 1 -- 172.21.15.158:0/1729009255 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50800b9d0 con 0x7ff50c06e5d0 2023-10-16T09:44:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.517+0000 7ff510b94700 1 -- 172.21.15.158:0/1729009255 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800bd60 con 0x7ff50c06e5d0 2023-10-16T09:44:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 -- 172.21.15.158:0/1729009255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06e5d0 msgr2=0x7ff50c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 --2- 172.21.15.158:0/1729009255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06e5d0 0x7ff50c071910 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff50c06eaf0 tx=0x7ff508009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 -- 172.21.15.158:0/1729009255 shutdown_connections 2023-10-16T09:44:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 --2- 172.21.15.158:0/1729009255 >> v2:172.21.15.172:3300/0 conn(0x7ff50c06af20 0x7ff50c0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 --2- 172.21.15.158:0/1729009255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06ab40 0x7ff50c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 --2- 172.21.15.158:0/1729009255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06e5d0 0x7ff50c071910 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 -- 172.21.15.158:0/1729009255 >> 172.21.15.158:0/1729009255 conn(0x7ff50c0d02b0 msgr2=0x7ff50c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 -- 172.21.15.158:0/1729009255 shutdown_connections 2023-10-16T09:44:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.518+0000 7ff513dfa700 1 -- 172.21.15.158:0/1729009255 wait complete. 2023-10-16T09:44:37.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.519+0000 7ff513dfa700 1 Processor -- start 2023-10-16T09:44:37.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.519+0000 7ff513dfa700 1 -- start start 2023-10-16T09:44:37.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff513dfa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff50c06ab40 0x7ff50c15f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff513dfa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06af20 0x7ff50c1f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff513dfa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06e5d0 0x7ff50c1f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff513dfa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c160840 con 0x7ff50c06af20 2023-10-16T09:44:37.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff513dfa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c15f970 con 0x7ff50c06ab40 2023-10-16T09:44:37.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff513dfa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c14c450 con 0x7ff50c06e5d0 2023-10-16T09:44:37.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06af20 0x7ff50c1f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06af20 0x7ff50c1f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48316/0 (socket says 172.21.15.158:48316) 2023-10-16T09:44:37.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 -- 172.21.15.158:0/4156784030 learned_addr learned my addr 172.21.15.158:0/4156784030 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:37.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff512397700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06e5d0 0x7ff50c1f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511b96700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff50c06ab40 0x7ff50c15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 -- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06e5d0 msgr2=0x7ff50c1f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06e5d0 0x7ff50c1f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 -- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff50c06ab40 msgr2=0x7ff50c15f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff50c06ab40 0x7ff50c15f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 -- 172.21.15.158:0/4156784030 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508009550 con 0x7ff50c06af20 2023-10-16T09:44:37.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff512397700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06e5d0 0x7ff50c1f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:37.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.520+0000 7ff511395700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06af20 0x7ff50c1f65f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff508000c00 tx=0x7ff508004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.521+0000 7ff5027fc700 1 -- 172.21.15.158:0/4156784030 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508004520 con 0x7ff50c06af20 2023-10-16T09:44:37.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.521+0000 7ff5027fc700 1 -- 172.21.15.158:0/4156784030 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50801e070 con 0x7ff50c06af20 2023-10-16T09:44:37.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.521+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c160c60 con 0x7ff50c06af20 2023-10-16T09:44:37.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.521+0000 7ff5027fc700 1 -- 172.21.15.158:0/4156784030 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5080038c0 con 0x7ff50c06af20 2023-10-16T09:44:37.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.521+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c15fea0 con 0x7ff50c06af20 2023-10-16T09:44:37.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.522+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c160980 con 0x7ff50c06af20 2023-10-16T09:44:37.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.522+0000 7ff5027fc700 1 -- 172.21.15.158:0/4156784030 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff508003a20 con 0x7ff50c06af20 2023-10-16T09:44:37.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.522+0000 7ff5027fc700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff4f006af00 0x7ff4f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.522+0000 7ff5027fc700 1 -- 172.21.15.158:0/4156784030 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff508020070 con 0x7ff50c06af20 2023-10-16T09:44:37.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.523+0000 7ff511b96700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff4f006af00 0x7ff4f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.523+0000 7ff511b96700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff4f006af00 0x7ff4f006d3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc006fd0 tx=0x7ff4fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.527+0000 7ff5027fc700 1 -- 172.21.15.158:0/4156784030 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff50c160980 con 0x7ff50c06af20 2023-10-16T09:44:37.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.722+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff50c160980 con 0x7ff50c06af20 2023-10-16T09:44:37.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.723+0000 7ff5027fc700 1 -- 172.21.15.158:0/4156784030 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13074 (secure 0 0 0) 0x7ff50c160980 con 0x7ff50c06af20 2023-10-16T09:44:37.725 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:37.725 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":35,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:34.379961+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff4f006af00 msgr2=0x7ff4f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff4f006af00 0x7ff4f006d3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc006fd0 tx=0x7ff4fc008040 comp rx=0 tx=0).stop 2023-10-16T09:44:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06af20 msgr2=0x7ff50c1f65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06af20 0x7ff50c1f65f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff508000c00 tx=0x7ff508004750 comp rx=0 tx=0).stop 2023-10-16T09:44:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 shutdown_connections 2023-10-16T09:44:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff4f006af00 0x7ff4f006d3c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50c06e5d0 0x7ff50c1f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff50c06ab40 0x7ff50c15f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 --2- 172.21.15.158:0/4156784030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff50c06af20 0x7ff50c1f65f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 >> 172.21.15.158:0/4156784030 conn(0x7ff50c0d02b0 msgr2=0x7ff50c0c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 shutdown_connections 2023-10-16T09:44:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.725+0000 7ff513dfa700 1 -- 172.21.15.158:0/4156784030 wait complete. 2023-10-16T09:44:37.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:44:37.740 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:44:37.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.861+0000 7f6c654d1700 1 Processor -- start 2023-10-16T09:44:37.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c654d1700 1 -- start start 2023-10-16T09:44:37.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c654d1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c6006afe0 0x7f6c600caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c654d1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c600cc590 0x7f6c600afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c654d1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c600cbd30 con 0x7f6c6015da40 2023-10-16T09:44:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c654d1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c600cc1c0 con 0x7f6c6006afe0 2023-10-16T09:44:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c654d1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6015de20 con 0x7f6c600cc590 2023-10-16T09:44:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c5e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c6006afe0 0x7f6c600caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c5e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c6006afe0 0x7f6c600caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48324/0 (socket says 172.21.15.158:48324) 2023-10-16T09:44:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.862+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/1847805317 learned_addr learned my addr 172.21.15.158:0/1847805317 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5f7fe700 1 --2- 172.21.15.158:0/1847805317 >> v2:172.21.15.172:3300/0 conn(0x7f6c600cc590 0x7f6c600afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/1847805317 >> v2:172.21.15.172:3300/0 conn(0x7f6c600cc590 msgr2=0x7f6c600afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5e7fc700 1 --2- 172.21.15.158:0/1847805317 >> v2:172.21.15.172:3300/0 conn(0x7f6c600cc590 0x7f6c600afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/1847805317 >> v1:172.21.15.172:6789/0 conn(0x7f6c6015da40 legacy=0x7f6c6006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/1847805317 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c600ce260 con 0x7f6c6006afe0 2023-10-16T09:44:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5e7fc700 1 --2- 172.21.15.158:0/1847805317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c6006afe0 0x7f6c600caa20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6c600cdd70 tx=0x7f6c50009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78310a66d6817a4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5dffb700 1 -- 172.21.15.158:0/1847805317 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c50004230 con 0x7f6c6006afe0 2023-10-16T09:44:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5dffb700 1 -- 172.21.15.158:0/1847805317 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c50004390 con 0x7f6c6006afe0 2023-10-16T09:44:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.863+0000 7f6c5dffb700 1 -- 172.21.15.158:0/1847805317 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c500036a0 con 0x7f6c6006afe0 2023-10-16T09:44:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 -- 172.21.15.158:0/1847805317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c6006afe0 msgr2=0x7f6c600caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 --2- 172.21.15.158:0/1847805317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c6006afe0 0x7f6c600caa20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6c600cdd70 tx=0x7f6c50009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 -- 172.21.15.158:0/1847805317 shutdown_connections 2023-10-16T09:44:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 --2- 172.21.15.158:0/1847805317 >> v2:172.21.15.172:3300/0 conn(0x7f6c600cc590 0x7f6c600afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 --2- 172.21.15.158:0/1847805317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c6006afe0 0x7f6c600caa20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 -- 172.21.15.158:0/1847805317 >> 172.21.15.158:0/1847805317 conn(0x7f6c60099a10 msgr2=0x7f6c600d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 -- 172.21.15.158:0/1847805317 shutdown_connections 2023-10-16T09:44:37.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.864+0000 7f6c654d1700 1 -- 172.21.15.158:0/1847805317 wait complete. 2023-10-16T09:44:37.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.865+0000 7f6c654d1700 1 Processor -- start 2023-10-16T09:44:37.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.865+0000 7f6c654d1700 1 -- start start 2023-10-16T09:44:37.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c654d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c6006afe0 0x7f6c600c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c654d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c600cc590 0x7f6c6014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c654d1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c6015da40 0x7f6c600c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c654d1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600c74a0 con 0x7f6c600cc590 2023-10-16T09:44:37.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c654d1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600c76f0 con 0x7f6c6015da40 2023-10-16T09:44:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c654d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600c79d0 con 0x7f6c6006afe0 2023-10-16T09:44:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c600cc590 0x7f6c6014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c600cc590 0x7f6c6014b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48330/0 (socket says 172.21.15.158:48330) 2023-10-16T09:44:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/846688096 learned_addr learned my addr 172.21.15.158:0/846688096 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5f7fe700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c6015da40 0x7f6c600c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5effd700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c6006afe0 0x7f6c600c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c6006afe0 msgr2=0x7f6c600c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c6006afe0 0x7f6c600c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c6015da40 msgr2=0x7f6c600c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c6015da40 0x7f6c600c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 -- 172.21.15.158:0/846688096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c50009550 con 0x7f6c600cc590 2023-10-16T09:44:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5effd700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c6006afe0 0x7f6c600c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c5e7fc700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c600cc590 0x7f6c6014b790 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009130 tx=0x7f6c50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.866+0000 7f6c477fe700 1 -- 172.21.15.158:0/846688096 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c50003d10 con 0x7f6c600cc590 2023-10-16T09:44:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.867+0000 7f6c477fe700 1 -- 172.21.15.158:0/846688096 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5001f070 con 0x7f6c600cc590 2023-10-16T09:44:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.867+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c600c7cb0 con 0x7f6c600cc590 2023-10-16T09:44:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.867+0000 7f6c477fe700 1 -- 172.21.15.158:0/846688096 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c50011870 con 0x7f6c600cc590 2023-10-16T09:44:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.867+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c600c8270 con 0x7f6c600cc590 2023-10-16T09:44:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.867+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c600681b0 con 0x7f6c600cc590 2023-10-16T09:44:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.868+0000 7f6c477fe700 1 -- 172.21.15.158:0/846688096 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6c50021030 con 0x7f6c600cc590 2023-10-16T09:44:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.868+0000 7f6c477fe700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c4c06af00 0x7f6c4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.868+0000 7f6c477fe700 1 -- 172.21.15.158:0/846688096 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6c5008a950 con 0x7f6c600cc590 2023-10-16T09:44:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.868+0000 7f6c5effd700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c4c06af00 0x7f6c4c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.869+0000 7f6c5effd700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c4c06af00 0x7f6c4c06d3c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009910 tx=0x7f6c54008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:37.873+0000 7f6c477fe700 1 -- 172.21.15.158:0/846688096 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c50055ca0 con 0x7f6c600cc590 2023-10-16T09:44:38.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.064+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c6006ace0 con 0x7f6c600cc590 2023-10-16T09:44:38.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.065+0000 7f6c477fe700 1 -- 172.21.15.158:0/846688096 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c500592c0 con 0x7f6c600cc590 2023-10-16T09:44:38.067 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:44:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c4c06af00 msgr2=0x7f6c4c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c4c06af00 0x7f6c4c06d3c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009910 tx=0x7f6c54008040 comp rx=0 tx=0).stop 2023-10-16T09:44:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c600cc590 msgr2=0x7f6c6014b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c600cc590 0x7f6c6014b790 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009130 tx=0x7f6c50004750 comp rx=0 tx=0).stop 2023-10-16T09:44:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 shutdown_connections 2023-10-16T09:44:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c6015da40 0x7f6c600c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f6c4c06af00 0x7f6c4c06d3c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c6006afe0 0x7f6c600c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 --2- 172.21.15.158:0/846688096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c600cc590 0x7f6c6014b790 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 >> 172.21.15.158:0/846688096 conn(0x7f6c60099a10 msgr2=0x7f6c6015cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 shutdown_connections 2023-10-16T09:44:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.067+0000 7f6c654d1700 1 -- 172.21.15.158:0/846688096 wait complete. 2023-10-16T09:44:38.081 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T09:44:38.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.201+0000 7f15a6b9e700 1 Processor -- start 2023-10-16T09:44:38.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.201+0000 7f15a6b9e700 1 -- start start 2023-10-16T09:44:38.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.201+0000 7f15a6b9e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f15a00b24c0 0x7f15a00ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.201+0000 7f15a6b9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a015e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.201+0000 7f15a6b9e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a015eb40 con 0x7f15a015a5d0 2023-10-16T09:44:38.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.201+0000 7f15a6b9e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a015edd0 con 0x7f15a00b24c0 2023-10-16T09:44:38.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.201+0000 7f15a6b9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a00ce080 con 0x7f15a015e150 2023-10-16T09:44:38.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f15a00b24c0 0x7f15a00ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f15a513b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a015e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f15a513b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a015e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:59172/0 (socket says 172.21.15.158:59172) 2023-10-16T09:44:38.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f15a00b24c0 0x7f15a00ceb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:49520/0 (socket says 172.21.15.158:49520) 2023-10-16T09:44:38.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f15a513b700 1 -- 172.21.15.158:0/849874600 learned_addr learned my addr 172.21.15.158:0/849874600 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:38.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f15a493a700 1 --1- 172.21.15.158:0/849874600 >> v1:172.21.15.172:6789/0 conn(0x7f15a015a5d0 0x7f15a00b1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:52416/0 (socket says 172.21.15.158:52416) 2023-10-16T09:44:38.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159ffff700 1 -- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 msgr2=0x7f15a015e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:38.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159ffff700 1 --2- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a015e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159ffff700 1 -- 172.21.15.158:0/849874600 >> v1:172.21.15.172:6789/0 conn(0x7f15a015a5d0 legacy=0x7f15a00b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159ffff700 1 -- 172.21.15.158:0/849874600 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a00ce360 con 0x7f15a00b24c0 2023-10-16T09:44:38.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f15a513b700 1 --2- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a015e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:38.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159ffff700 1 --2- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a00b24c0 0x7f15a00ceb60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f158c009840 tx=0x7f158c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e83721d4d6434541 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:38.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.202+0000 7f159f7fe700 1 -- 172.21.15.158:0/849874600 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c011830 con 0x7f15a00b24c0 2023-10-16T09:44:38.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f159f7fe700 1 -- 172.21.15.158:0/849874600 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c011990 con 0x7f15a00b24c0 2023-10-16T09:44:38.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f159f7fe700 1 -- 172.21.15.158:0/849874600 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c011d20 con 0x7f15a00b24c0 2023-10-16T09:44:38.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 -- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a00b24c0 msgr2=0x7f15a00ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a00b24c0 0x7f15a00ceb60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f158c009840 tx=0x7f158c009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:38.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 -- 172.21.15.158:0/849874600 shutdown_connections 2023-10-16T09:44:38.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a015e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/849874600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a00b24c0 0x7f15a00ceb60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 -- 172.21.15.158:0/849874600 >> 172.21.15.158:0/849874600 conn(0x7f15a00ccdb0 msgr2=0x7f15a00cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:38.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 -- 172.21.15.158:0/849874600 shutdown_connections 2023-10-16T09:44:38.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.203+0000 7f15a6b9e700 1 -- 172.21.15.158:0/849874600 wait complete. 2023-10-16T09:44:38.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.204+0000 7f15a6b9e700 1 Processor -- start 2023-10-16T09:44:38.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.204+0000 7f15a6b9e700 1 -- start start 2023-10-16T09:44:38.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a6b9e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a00b24c0 0x7f15a014a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a6b9e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a015a5d0 0x7f15a015d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a6b9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a014d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a6b9e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015d100 con 0x7f15a00b24c0 2023-10-16T09:44:38.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a6b9e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a00accd0 con 0x7f15a015a5d0 2023-10-16T09:44:38.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a6b9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015cb20 con 0x7f15a015e150 2023-10-16T09:44:38.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a493a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a00b24c0 0x7f15a014a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a493a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a00b24c0 0x7f15a014a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48342/0 (socket says 172.21.15.158:48342) 2023-10-16T09:44:38.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.205+0000 7f15a493a700 1 -- 172.21.15.158:0/2486629493 learned_addr learned my addr 172.21.15.158:0/2486629493 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:38.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f159ffff700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a015a5d0 0x7f15a015d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a513b700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a014d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a493a700 1 -- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 msgr2=0x7f15a014d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a493a700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a014d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a493a700 1 -- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a015a5d0 msgr2=0x7f15a015d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a493a700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a015a5d0 0x7f15a015d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a493a700 1 -- 172.21.15.158:0/2486629493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159800bd40 con 0x7f15a00b24c0 2023-10-16T09:44:38.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a493a700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a00b24c0 0x7f15a014a890 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f159400c520 tx=0x7f159400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:38.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f159d7fa700 1 -- 172.21.15.158:0/2486629493 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594005cb0 con 0x7f15a00b24c0 2023-10-16T09:44:38.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f159d7fa700 1 -- 172.21.15.158:0/2486629493 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1594005e10 con 0x7f15a00b24c0 2023-10-16T09:44:38.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158c0094a0 con 0x7f15a00b24c0 2023-10-16T09:44:38.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f159d7fa700 1 -- 172.21.15.158:0/2486629493 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15940072d0 con 0x7f15a00b24c0 2023-10-16T09:44:38.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.206+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a015cd90 con 0x7f15a00b24c0 2023-10-16T09:44:38.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.207+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a015cf20 con 0x7f15a00b24c0 2023-10-16T09:44:38.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.207+0000 7f159d7fa700 1 -- 172.21.15.158:0/2486629493 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1594007430 con 0x7f15a00b24c0 2023-10-16T09:44:38.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.208+0000 7f159d7fa700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f158406af00 0x7f158406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.208+0000 7f159ffff700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f158406af00 0x7f158406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.208+0000 7f159d7fa700 1 -- 172.21.15.158:0/2486629493 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1594088860 con 0x7f15a00b24c0 2023-10-16T09:44:38.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.208+0000 7f159ffff700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f158406af00 0x7f158406d3c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15a0158fb0 tx=0x7f158c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:38.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.212+0000 7f159d7fa700 1 -- 172.21.15.158:0/2486629493 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1594053b40 con 0x7f15a00b24c0 2023-10-16T09:44:38.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.408+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f15a00681b0 con 0x7f15a00b24c0 2023-10-16T09:44:38.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.408+0000 7f159d7fa700 1 -- 172.21.15.158:0/2486629493 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1594057160 con 0x7f15a00b24c0 2023-10-16T09:44:38.411 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:38.411 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:44:38.411 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:38.411 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T09:44:38.412 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T09:44:38.412 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:38.412 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:44:38.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f158406af00 msgr2=0x7f158406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f158406af00 0x7f158406d3c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15a0158fb0 tx=0x7f158c008030 comp rx=0 tx=0).stop 2023-10-16T09:44:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a00b24c0 msgr2=0x7f15a014a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a00b24c0 0x7f15a014a890 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f159400c520 tx=0x7f159400c8e0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 shutdown_connections 2023-10-16T09:44:38.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f158406af00 0x7f158406d3c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a015e150 0x7f15a014d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a00b24c0 0x7f15a014a890 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 --2- 172.21.15.158:0/2486629493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a015a5d0 0x7f15a015d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 >> 172.21.15.158:0/2486629493 conn(0x7f15a00ccdb0 msgr2=0x7f15a00b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:38.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 shutdown_connections 2023-10-16T09:44:38.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.411+0000 7f15a6b9e700 1 -- 172.21.15.158:0/2486629493 wait complete. 2023-10-16T09:44:38.424 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T09:44:38.425 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T09:44:38.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.547+0000 7ff3c7b2d700 1 Processor -- start 2023-10-16T09:44:38.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.547+0000 7ff3c7b2d700 1 -- start start 2023-10-16T09:44:38.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c7b2d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3c015da40 0x7ff3c00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c7b2d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 0x7ff3c00afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c7b2d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00cbd30 con 0x7ff3c00cc590 2023-10-16T09:44:38.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c7b2d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c00cc1c0 con 0x7ff3c015da40 2023-10-16T09:44:38.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c7b2d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c015de20 con 0x7ff3c006afe0 2023-10-16T09:44:38.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c58c9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3c015da40 0x7ff3c00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c50c8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff3c00cc590 0x7ff3c006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48528/0 (socket says 172.21.15.158:48528) 2023-10-16T09:44:38.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c50c8700 1 -- 172.21.15.158:0/3534124731 learned_addr learned my addr 172.21.15.158:0/3534124731 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:38.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c60ca700 1 --2- 172.21.15.158:0/3534124731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 0x7ff3c00afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c58c9700 1 -- 172.21.15.158:0/3534124731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 msgr2=0x7ff3c00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:38.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c58c9700 1 --2- 172.21.15.158:0/3534124731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 0x7ff3c00afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c58c9700 1 -- 172.21.15.158:0/3534124731 >> v1:172.21.15.158:6789/0 conn(0x7ff3c00cc590 legacy=0x7ff3c006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c58c9700 1 -- 172.21.15.158:0/3534124731 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00ce260 con 0x7ff3c015da40 2023-10-16T09:44:38.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.548+0000 7ff3c48c7700 1 -- 172.21.15.158:0/3534124731 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154057110 0 0) 0x7ff3c00cbd30 con 0x7ff3c00cc590 2023-10-16T09:44:38.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c58c9700 1 --2- 172.21.15.158:0/3534124731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c00caa20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009840 tx=0x7ff3bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd360d109a800f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:38.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c48c7700 1 -- 172.21.15.158:0/3534124731 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc011830 con 0x7ff3c015da40 2023-10-16T09:44:38.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c48c7700 1 -- 172.21.15.158:0/3534124731 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc011990 con 0x7ff3c015da40 2023-10-16T09:44:38.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c48c7700 1 -- 172.21.15.158:0/3534124731 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc011d20 con 0x7ff3c015da40 2023-10-16T09:44:38.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/3534124731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 msgr2=0x7ff3c00caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/3534124731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c00caa20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009840 tx=0x7ff3bc009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:38.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/3534124731 shutdown_connections 2023-10-16T09:44:38.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/3534124731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c00caa20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/3534124731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 0x7ff3c00afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/3534124731 >> 172.21.15.158:0/3534124731 conn(0x7ff3c0099a10 msgr2=0x7ff3c00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:38.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/3534124731 shutdown_connections 2023-10-16T09:44:38.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.549+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/3534124731 wait complete. 2023-10-16T09:44:38.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.550+0000 7ff3c7b2d700 1 Processor -- start 2023-10-16T09:44:38.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c7b2d700 1 -- start start 2023-10-16T09:44:38.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c7b2d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 0x7ff3c01e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c7b2d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3c00cc590 0x7ff3c01e6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c7b2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c014e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c7b2d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c015acd0 con 0x7ff3c015da40 2023-10-16T09:44:38.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c7b2d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c01e5560 con 0x7ff3c00cc590 2023-10-16T09:44:38.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c7b2d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c01e5840 con 0x7ff3c006afe0 2023-10-16T09:44:38.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c60ca700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c014e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c60ca700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c014e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48360/0 (socket says 172.21.15.158:48360) 2023-10-16T09:44:38.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.551+0000 7ff3c60ca700 1 -- 172.21.15.158:0/1622322712 learned_addr learned my addr 172.21.15.158:0/1622322712 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:38.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c60ca700 1 -- 172.21.15.158:0/1622322712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 msgr2=0x7ff3c01e4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c60ca700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 0x7ff3c01e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c60ca700 1 -- 172.21.15.158:0/1622322712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3c00cc590 msgr2=0x7ff3c01e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:38.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c60ca700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3c00cc590 0x7ff3c01e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:38.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c60ca700 1 -- 172.21.15.158:0/1622322712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc0094a0 con 0x7ff3c015da40 2023-10-16T09:44:38.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c60ca700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c014e690 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007d40 tx=0x7ff3b800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:38.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3b67fc700 1 -- 172.21.15.158:0/1622322712 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800ede0 con 0x7ff3c015da40 2023-10-16T09:44:38.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3b67fc700 1 -- 172.21.15.158:0/1622322712 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b8014420 con 0x7ff3c015da40 2023-10-16T09:44:38.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c01e5b20 con 0x7ff3c015da40 2023-10-16T09:44:38.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3b67fc700 1 -- 172.21.15.158:0/1622322712 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b80147d0 con 0x7ff3c015da40 2023-10-16T09:44:38.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.552+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c01e60e0 con 0x7ff3c015da40 2023-10-16T09:44:38.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.553+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c00681b0 con 0x7ff3c015da40 2023-10-16T09:44:38.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.553+0000 7ff3b67fc700 1 -- 172.21.15.158:0/1622322712 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff3b8014930 con 0x7ff3c015da40 2023-10-16T09:44:38.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.554+0000 7ff3b67fc700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3a406af00 0x7ff3a406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:38.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.554+0000 7ff3b67fc700 1 -- 172.21.15.158:0/1622322712 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff3b8089af0 con 0x7ff3c015da40 2023-10-16T09:44:38.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.554+0000 7ff3c58c9700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3a406af00 0x7ff3a406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:38.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.554+0000 7ff3c58c9700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3a406af00 0x7ff3a406d3c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0090b0 tx=0x7ff3bc010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:38.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.558+0000 7ff3b67fc700 1 -- 172.21.15.158:0/1622322712 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3b8054db0 con 0x7ff3c015da40 2023-10-16T09:44:38.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:38.752+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff3c01e62a0 con 0x7ff3c015da40 2023-10-16T09:44:39.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.682+0000 7ff3b67fc700 1 -- 172.21.15.158:0/1622322712 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff3b80583d0 con 0x7ff3c015da40 2023-10-16T09:44:39.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.686+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3a406af00 msgr2=0x7ff3a406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:39.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.686+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3a406af00 0x7ff3a406d3c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0090b0 tx=0x7ff3bc010000 comp rx=0 tx=0).stop 2023-10-16T09:44:39.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.686+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 msgr2=0x7ff3c014e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:39.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.686+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c014e690 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007d40 tx=0x7ff3b800e3c0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 shutdown_connections 2023-10-16T09:44:39.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3c015da40 0x7ff3c014e690 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3a406af00 0x7ff3a406d3c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3c006afe0 0x7ff3c01e4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 --2- 172.21.15.158:0/1622322712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3c00cc590 0x7ff3c01e6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 >> 172.21.15.158:0/1622322712 conn(0x7ff3c0099a10 msgr2=0x7ff3c015c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:39.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 shutdown_connections 2023-10-16T09:44:39.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.687+0000 7ff3c7b2d700 1 -- 172.21.15.158:0/1622322712 wait complete. 2023-10-16T09:44:39.692 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T09:44:39.706 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T09:44:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.838+0000 7f30cd3ce700 1 Processor -- start 2023-10-16T09:44:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.838+0000 7f30cd3ce700 1 -- start start 2023-10-16T09:44:39.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30cd3ce700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30c80cbf50 0x7f30c8144b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:39.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30cd3ce700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f30c8145040 0x7f30c8070410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:39.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30cd3ce700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c8070d90 con 0x7f30c8070950 2023-10-16T09:44:39.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30cd3ce700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c8160220 con 0x7f30c8145040 2023-10-16T09:44:39.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30cd3ce700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c815f6a0 con 0x7f30c80cbf50 2023-10-16T09:44:39.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30c67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f30c8145040 0x7f30c8070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:39.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30c67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f30c8145040 0x7f30c8070410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48364/0 (socket says 172.21.15.158:48364) 2023-10-16T09:44:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30c67fc700 1 -- 172.21.15.158:0/2523358684 learned_addr learned my addr 172.21.15.158:0/2523358684 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.839+0000 7f30c6ffd700 1 --2- 172.21.15.158:0/2523358684 >> v2:172.21.15.172:3300/0 conn(0x7f30c80cbf50 0x7f30c8144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c67fc700 1 -- 172.21.15.158:0/2523358684 >> v2:172.21.15.172:3300/0 conn(0x7f30c80cbf50 msgr2=0x7f30c8144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c67fc700 1 --2- 172.21.15.158:0/2523358684 >> v2:172.21.15.172:3300/0 conn(0x7f30c80cbf50 0x7f30c8144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c67fc700 1 -- 172.21.15.158:0/2523358684 >> v1:172.21.15.172:6789/0 conn(0x7f30c8070950 legacy=0x7f30c806ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c67fc700 1 -- 172.21.15.158:0/2523358684 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c815f980 con 0x7f30c8145040 2023-10-16T09:44:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c67fc700 1 --2- 172.21.15.158:0/2523358684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c8070410 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f30c8067750 tx=0x7f30b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0242adb6c816b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c5ffb700 1 -- 172.21.15.158:0/2523358684 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8004230 con 0x7f30c8145040 2023-10-16T09:44:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c5ffb700 1 -- 172.21.15.158:0/2523358684 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b8004390 con 0x7f30c8145040 2023-10-16T09:44:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30c5ffb700 1 -- 172.21.15.158:0/2523358684 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80036a0 con 0x7f30c8145040 2023-10-16T09:44:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 -- 172.21.15.158:0/2523358684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 msgr2=0x7f30c8070410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/2523358684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c8070410 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f30c8067750 tx=0x7f30b8009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 -- 172.21.15.158:0/2523358684 shutdown_connections 2023-10-16T09:44:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/2523358684 >> v2:172.21.15.172:3300/0 conn(0x7f30c80cbf50 0x7f30c8144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/2523358684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c8070410 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 -- 172.21.15.158:0/2523358684 >> 172.21.15.158:0/2523358684 conn(0x7f30c80d1410 msgr2=0x7f30c80d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 -- 172.21.15.158:0/2523358684 shutdown_connections 2023-10-16T09:44:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.840+0000 7f30cd3ce700 1 -- 172.21.15.158:0/2523358684 wait complete. 2023-10-16T09:44:39.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.842+0000 7f30cd3ce700 1 Processor -- start 2023-10-16T09:44:39.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.842+0000 7f30cd3ce700 1 -- start start 2023-10-16T09:44:39.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c8070950 0x7f30c815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:39.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cbf50 0x7f30c81f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:39.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c80c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:39.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81f22d0 con 0x7f30c8145040 2023-10-16T09:44:39.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81f28b0 con 0x7f30c8070950 2023-10-16T09:44:39.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81f2b90 con 0x7f30c80cbf50 2023-10-16T09:44:39.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c80c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:39.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c80c3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48374/0 (socket says 172.21.15.158:48374) 2023-10-16T09:44:39.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 -- 172.21.15.158:0/318295272 learned_addr learned my addr 172.21.15.158:0/318295272 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:39.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c6ffd700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c8070950 0x7f30c815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:39.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c67fc700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cbf50 0x7f30c81f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:39.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 -- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cbf50 msgr2=0x7f30c81f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:39.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cbf50 0x7f30c81f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 -- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c8070950 msgr2=0x7f30c815bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:39.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c8070950 0x7f30c815bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:39.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 -- 172.21.15.158:0/318295272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8009550 con 0x7f30c8145040 2023-10-16T09:44:39.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30c77fe700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c80c3a80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f30c000b970 tx=0x7f30c000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:39.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30bf7fe700 1 -- 172.21.15.158:0/318295272 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c000d860 con 0x7f30c8145040 2023-10-16T09:44:39.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30bf7fe700 1 -- 172.21.15.158:0/318295272 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c000d9c0 con 0x7f30c8145040 2023-10-16T09:44:39.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c81f2e70 con 0x7f30c8145040 2023-10-16T09:44:39.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30bf7fe700 1 -- 172.21.15.158:0/318295272 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c000dd70 con 0x7f30c8145040 2023-10-16T09:44:39.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.843+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c81f3430 con 0x7f30c8145040 2023-10-16T09:44:39.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.844+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c80d11f0 con 0x7f30c8145040 2023-10-16T09:44:39.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.845+0000 7f30bf7fe700 1 -- 172.21.15.158:0/318295272 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f30c000e430 con 0x7f30c8145040 2023-10-16T09:44:39.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.845+0000 7f30bf7fe700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a806af00 0x7f30a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:39.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.845+0000 7f30bf7fe700 1 -- 172.21.15.158:0/318295272 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f30c00892a0 con 0x7f30c8145040 2023-10-16T09:44:39.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.845+0000 7f30c6ffd700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a806af00 0x7f30a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:39.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.846+0000 7f30c6ffd700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a806af00 0x7f30a806d3c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f30c815b310 tx=0x7f30b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:39.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:39.850+0000 7f30bf7fe700 1 -- 172.21.15.158:0/318295272 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30c00543f0 con 0x7f30c8145040 2023-10-16T09:44:40.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.047+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f30c80cbcd0 con 0x7f30c8145040 2023-10-16T09:44:40.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.684+0000 7f30bf7fe700 1 -- 172.21.15.158:0/318295272 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f30c000ded0 con 0x7f30c8145040 2023-10-16T09:44:40.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a806af00 msgr2=0x7f30a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:40.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a806af00 0x7f30a806d3c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f30c815b310 tx=0x7f30b0008040 comp rx=0 tx=0).stop 2023-10-16T09:44:40.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 msgr2=0x7f30c80c3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c80c3a80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f30c000b970 tx=0x7f30c000bc80 comp rx=0 tx=0).stop 2023-10-16T09:44:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 shutdown_connections 2023-10-16T09:44:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c8145040 0x7f30c80c3a80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c8070950 0x7f30c815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f30a806af00 0x7f30a806d3c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 --2- 172.21.15.158:0/318295272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cbf50 0x7f30c81f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.687+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 >> 172.21.15.158:0/318295272 conn(0x7f30c80d1410 msgr2=0x7f30c80c4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:40.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.688+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 shutdown_connections 2023-10-16T09:44:40.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.688+0000 7f30cd3ce700 1 -- 172.21.15.158:0/318295272 wait complete. 2023-10-16T09:44:40.701 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T09:44:40.716 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T09:44:40.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.840+0000 7f489b4e8700 1 Processor -- start 2023-10-16T09:44:40.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.840+0000 7f489b4e8700 1 -- start start 2023-10-16T09:44:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f489b4e8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f48940cc200 0x7f489415f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f489b4e8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48940cf4c0 con 0x7f489415b590 2023-10-16T09:44:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f489b4e8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489415f710 con 0x7f48940ce510 2023-10-16T09:44:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f489b4e8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48940b49e0 con 0x7f48940cc200 2023-10-16T09:44:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f4899284700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f489415b590 0x7f48940b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48554/0 (socket says 172.21.15.158:48554) 2023-10-16T09:44:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f4899284700 1 -- 172.21.15.158:0/3208340705 learned_addr learned my addr 172.21.15.158:0/3208340705 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f4899a85700 1 --2- 172.21.15.158:0/3208340705 >> v2:172.21.15.172:3300/0 conn(0x7f48940cc200 0x7f489415f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175022989 0 0) 0x7f48940cf4c0 con 0x7f489415b590 2023-10-16T09:44:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48800035b0 con 0x7f489415b590 2023-10-16T09:44:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533988199 0 0) 0x7f48800035b0 con 0x7f489415b590 2023-10-16T09:44:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 >> v2:172.21.15.172:3300/0 conn(0x7f48940cc200 msgr2=0x7f489415f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 --2- 172.21.15.158:0/3208340705 >> v2:172.21.15.172:3300/0 conn(0x7f48940cc200 0x7f489415f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 >> v1:172.21.15.172:6789/0 conn(0x7f48940ce510 legacy=0x7f48940cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489415f970 con 0x7f489415b590 2023-10-16T09:44:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4890001670 con 0x7f489415b590 2023-10-16T09:44:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 >> v1:172.21.15.158:6789/0 conn(0x7f489415b590 legacy=0x7f48940b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4880003a30 0x7f4880003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4880004850 0x7f4880004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880005260 0x7f488000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48940b49e0 con 0x7f4880003a30 2023-10-16T09:44:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488000c190 con 0x7f4880005260 2023-10-16T09:44:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488000c350 con 0x7f4880004850 2023-10-16T09:44:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.841+0000 7f4899284700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4880003a30 0x7f4880003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899a85700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880005260 0x7f488000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4898a83700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4880004850 0x7f4880004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899284700 1 -- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4880004850 msgr2=0x7f4880004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899284700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4880004850 0x7f4880004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899284700 1 -- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880005260 msgr2=0x7f488000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899284700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880005260 0x7f488000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899284700 1 -- 172.21.15.158:0/3208340705 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48800035b0 con 0x7f4880003a30 2023-10-16T09:44:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899a85700 1 -- 172.21.15.158:0/3208340705 reap_dead start 2023-10-16T09:44:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f4899284700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4880003a30 0x7f4880003e50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f489000c240 tx=0x7f4880005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbdc70ec9b9a024 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4890007c10 con 0x7f4880003a30 2023-10-16T09:44:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489000cda0 con 0x7f4880003a30 2023-10-16T09:44:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f488bfff700 1 -- 172.21.15.158:0/3208340705 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4890005ba0 con 0x7f4880003a30 2023-10-16T09:44:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f489b4e8700 1 -- 172.21.15.158:0/3208340705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4880003a30 msgr2=0x7f4880003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4880003a30 0x7f4880003e50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f489000c240 tx=0x7f4880005880 comp rx=0 tx=0).stop 2023-10-16T09:44:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f489b4e8700 1 -- 172.21.15.158:0/3208340705 shutdown_connections 2023-10-16T09:44:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3208340705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4880003a30 0x7f4880003e50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.842+0000 7f489b4e8700 1 -- 172.21.15.158:0/3208340705 >> 172.21.15.158:0/3208340705 conn(0x7f489406a8d0 msgr2=0x7f489406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.843+0000 7f489b4e8700 1 -- 172.21.15.158:0/3208340705 shutdown_connections 2023-10-16T09:44:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.843+0000 7f489b4e8700 1 -- 172.21.15.158:0/3208340705 wait complete. 2023-10-16T09:44:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 Processor -- start 2023-10-16T09:44:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 -- start start 2023-10-16T09:44:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880003a30 0x7f48940ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48941f44f0 0x7f48941f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f489415dbf0 0x7f489415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48940a8480 con 0x7f48941f44f0 2023-10-16T09:44:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489415e5e0 con 0x7f4880003a30 2023-10-16T09:44:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.844+0000 7f489b4e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48941f4c50 con 0x7f489415dbf0 2023-10-16T09:44:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48941f44f0 0x7f48941f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48941f44f0 0x7f48941f07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48392/0 (socket says 172.21.15.158:48392) 2023-10-16T09:44:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 -- 172.21.15.158:0/3964574986 learned_addr learned my addr 172.21.15.158:0/3964574986 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4899284700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880003a30 0x7f48940ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4899a85700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f489415dbf0 0x7f489415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 -- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f489415dbf0 msgr2=0x7f489415e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f489415dbf0 0x7f489415e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 -- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880003a30 msgr2=0x7f48940ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880003a30 0x7f48940ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 -- 172.21.15.158:0/3964574986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48941f4f30 con 0x7f48941f44f0 2023-10-16T09:44:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4898a83700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48941f44f0 0x7f48941f07b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f488400e4d0 tx=0x7f488400e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4889ffb700 1 -- 172.21.15.158:0/3964574986 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488400ee60 con 0x7f48941f44f0 2023-10-16T09:44:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4889ffb700 1 -- 172.21.15.158:0/3964574986 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48840052e0 con 0x7f48941f44f0 2023-10-16T09:44:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48941f5210 con 0x7f48941f44f0 2023-10-16T09:44:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f4889ffb700 1 -- 172.21.15.158:0/3964574986 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884003680 con 0x7f48941f44f0 2023-10-16T09:44:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.845+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48941f4970 con 0x7f48941f44f0 2023-10-16T09:44:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.846+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48941f5430 con 0x7f48941f44f0 2023-10-16T09:44:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.847+0000 7f4889ffb700 1 -- 172.21.15.158:0/3964574986 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f48840037e0 con 0x7f48941f44f0 2023-10-16T09:44:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.847+0000 7f4889ffb700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f487806af00 0x7f487806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.847+0000 7f4889ffb700 1 -- 172.21.15.158:0/3964574986 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f4884088ce0 con 0x7f48941f44f0 2023-10-16T09:44:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.847+0000 7f4899284700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f487806af00 0x7f487806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:40.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.848+0000 7f4899284700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f487806af00 0x7f487806d3c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4890000c00 tx=0x7f4890007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:40.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:40.852+0000 7f4889ffb700 1 -- 172.21.15.158:0/3964574986 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4884053c40 con 0x7f48941f44f0 2023-10-16T09:44:41.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.088+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f48940681b0 con 0x7f48941f44f0 2023-10-16T09:44:41.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.712+0000 7f4889ffb700 1 -- 172.21.15.158:0/3964574986 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4884057260 con 0x7f48941f44f0 2023-10-16T09:44:41.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f487806af00 msgr2=0x7f487806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:41.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f487806af00 0x7f487806d3c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4890000c00 tx=0x7f4890007000 comp rx=0 tx=0).stop 2023-10-16T09:44:41.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48941f44f0 msgr2=0x7f48941f07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:41.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48941f44f0 0x7f48941f07b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f488400e4d0 tx=0x7f488400e7e0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 shutdown_connections 2023-10-16T09:44:41.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f487806af00 0x7f487806d3c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f489415dbf0 0x7f489415e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4880003a30 0x7f48940ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 --2- 172.21.15.158:0/3964574986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48941f44f0 0x7f48941f07b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 >> 172.21.15.158:0/3964574986 conn(0x7f489406a8d0 msgr2=0x7f48940b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:41.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 shutdown_connections 2023-10-16T09:44:41.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.720+0000 7f489b4e8700 1 -- 172.21.15.158:0/3964574986 wait complete. 2023-10-16T09:44:41.725 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 5 and data pool 6 2023-10-16T09:44:41.740 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:41.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.862+0000 7fbb7a4ad700 1 Processor -- start 2023-10-16T09:44:41.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.862+0000 7fbb7a4ad700 1 -- start start 2023-10-16T09:44:41.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb7a4ad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb740b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:41.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb7a4ad700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbb7415f0f0 0x7fbb740cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:41.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb7a4ad700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7415b930 con 0x7fbb740ce530 2023-10-16T09:44:41.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb7a4ad700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb740cf020 con 0x7fbb7415f0f0 2023-10-16T09:44:41.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb7a4ad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7415f5f0 con 0x7fbb740cc1b0 2023-10-16T09:44:41.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb73fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbb740ce530 0x7fbb740cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48574/0 (socket says 172.21.15.158:48574) 2023-10-16T09:44:41.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb73fff700 1 -- 172.21.15.158:0/591084863 learned_addr learned my addr 172.21.15.158:0/591084863 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb78a4a700 1 --2- 172.21.15.158:0/591084863 >> v2:172.21.15.158:3300/0 conn(0x7fbb7415f0f0 0x7fbb740cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb737fe700 1 --2- 172.21.15.158:0/591084863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb740b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb72ffd700 1 -- 172.21.15.158:0/591084863 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947446692 0 0) 0x7fbb7415b930 con 0x7fbb740ce530 2023-10-16T09:44:41.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb72ffd700 1 -- 172.21.15.158:0/591084863 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c0035b0 con 0x7fbb740ce530 2023-10-16T09:44:41.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb78a4a700 1 -- 172.21.15.158:0/591084863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 msgr2=0x7fbb740b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:41.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb78a4a700 1 --2- 172.21.15.158:0/591084863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb740b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb78a4a700 1 -- 172.21.15.158:0/591084863 >> v1:172.21.15.158:6789/0 conn(0x7fbb740ce530 legacy=0x7fbb740cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.863+0000 7fbb78a4a700 1 -- 172.21.15.158:0/591084863 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7415f8d0 con 0x7fbb7415f0f0 2023-10-16T09:44:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb78a4a700 1 --2- 172.21.15.158:0/591084863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb7415f0f0 0x7fbb740cf470 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009d90 tx=0x7fbb6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fcbd3407e1a7ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb72ffd700 1 -- 172.21.15.158:0/591084863 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00e070 con 0x7fbb7415f0f0 2023-10-16T09:44:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb72ffd700 1 -- 172.21.15.158:0/591084863 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c00bb20 con 0x7fbb7415f0f0 2023-10-16T09:44:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb72ffd700 1 -- 172.21.15.158:0/591084863 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00be90 con 0x7fbb7415f0f0 2023-10-16T09:44:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/591084863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb7415f0f0 msgr2=0x7fbb740cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/591084863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb7415f0f0 0x7fbb740cf470 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009d90 tx=0x7fbb6c00b330 comp rx=0 tx=0).stop 2023-10-16T09:44:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/591084863 shutdown_connections 2023-10-16T09:44:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/591084863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb7415f0f0 0x7fbb740cf470 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/591084863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb740b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/591084863 >> 172.21.15.158:0/591084863 conn(0x7fbb7406a8d0 msgr2=0x7fbb7406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/591084863 shutdown_connections 2023-10-16T09:44:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.864+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/591084863 wait complete. 2023-10-16T09:44:41.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.865+0000 7fbb7a4ad700 1 Processor -- start 2023-10-16T09:44:41.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.866+0000 7fbb7a4ad700 1 -- start start 2023-10-16T09:44:41.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.866+0000 7fbb7a4ad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb7415a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:41.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.866+0000 7fbb7a4ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb740ce530 0x7fbb740b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:41.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.866+0000 7fbb7a4ad700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb7415f0f0 0x7fbb740c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:41.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.866+0000 7fbb7a4ad700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb741f20f0 con 0x7fbb740ce530 2023-10-16T09:44:41.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.866+0000 7fbb7a4ad700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7415af20 con 0x7fbb7415f0f0 2023-10-16T09:44:41.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.866+0000 7fbb7a4ad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740b6a20 con 0x7fbb740cc1b0 2023-10-16T09:44:41.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb740ce530 0x7fbb740b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb740ce530 0x7fbb740b5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48404/0 (socket says 172.21.15.158:48404) 2023-10-16T09:44:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 -- 172.21.15.158:0/1972350133 learned_addr learned my addr 172.21.15.158:0/1972350133 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb73fff700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb7415a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:41.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb78a4a700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb7415f0f0 0x7fbb740c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:41.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 -- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 msgr2=0x7fbb7415a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:41.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb7415a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 -- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb7415f0f0 msgr2=0x7fbb740c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:41.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb7415f0f0 0x7fbb740c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 -- 172.21.15.158:0/1972350133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c009a70 con 0x7fbb740ce530 2023-10-16T09:44:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb737fe700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb740ce530 0x7fbb740b5e40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400e870 tx=0x7fbb6400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb70ff9700 1 -- 172.21.15.158:0/1972350133 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64009590 con 0x7fbb740ce530 2023-10-16T09:44:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb70ff9700 1 -- 172.21.15.158:0/1972350133 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6400ee70 con 0x7fbb740ce530 2023-10-16T09:44:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb70ff9700 1 -- 172.21.15.158:0/1972350133 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6400ca50 con 0x7fbb740ce530 2023-10-16T09:44:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb740b6740 con 0x7fbb740ce530 2023-10-16T09:44:41.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.867+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb741f1d80 con 0x7fbb740ce530 2023-10-16T09:44:41.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.868+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb740681b0 con 0x7fbb740ce530 2023-10-16T09:44:41.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.868+0000 7fbb70ff9700 1 -- 172.21.15.158:0/1972350133 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbb6400cbb0 con 0x7fbb740ce530 2023-10-16T09:44:41.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.869+0000 7fbb70ff9700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbb6006af00 0x7fbb6006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:41.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.869+0000 7fbb70ff9700 1 -- 172.21.15.158:0/1972350133 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbb64089610 con 0x7fbb740ce530 2023-10-16T09:44:41.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.869+0000 7fbb73fff700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbb6006af00 0x7fbb6006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:41.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.869+0000 7fbb73fff700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbb6006af00 0x7fbb6006d3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbb68005910 tx=0x7fbb68009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:41.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:41.873+0000 7fbb70ff9700 1 -- 172.21.15.158:0/1972350133 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb64054610 con 0x7fbb740ce530 2023-10-16T09:44:42.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.064+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb740b0a10 con 0x7fbb740ce530 2023-10-16T09:44:42.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.065+0000 7fbb70ff9700 1 -- 172.21.15.158:0/1972350133 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16799 (secure 0 0 0) 0x7fbb6400ce60 con 0x7fbb740ce530 2023-10-16T09:44:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:42.067 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":38,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:41.089845+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:44:38.770851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T09:44:40.065625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:42.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.067+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbb6006af00 msgr2=0x7fbb6006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.067+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbb6006af00 0x7fbb6006d3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbb68005910 tx=0x7fbb68009040 comp rx=0 tx=0).stop 2023-10-16T09:44:42.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.067+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb740ce530 msgr2=0x7fbb740b5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.067+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb740ce530 0x7fbb740b5e40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400e870 tx=0x7fbb6400ec30 comp rx=0 tx=0).stop 2023-10-16T09:44:42.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 shutdown_connections 2023-10-16T09:44:42.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb7415f0f0 0x7fbb740c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fbb6006af00 0x7fbb6006d3c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cc1b0 0x7fbb7415a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 --2- 172.21.15.158:0/1972350133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb740ce530 0x7fbb740b5e40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 >> 172.21.15.158:0/1972350133 conn(0x7fbb7406a8d0 msgr2=0x7fbb7415bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:42.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 shutdown_connections 2023-10-16T09:44:42.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.068+0000 7fbb7a4ad700 1 -- 172.21.15.158:0/1972350133 wait complete. 2023-10-16T09:44:42.082 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:42.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.200+0000 7fe5a8983700 1 Processor -- start 2023-10-16T09:44:42.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.200+0000 7fe5a8983700 1 -- start start 2023-10-16T09:44:42.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a8983700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5a40b24c0 0x7fe5a40ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a8983700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe5a415e150 0x7fe5a415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a8983700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a415eb20 con 0x7fe5a415a5d0 2023-10-16T09:44:42.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a8983700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a415edd0 con 0x7fe5a40b24c0 2023-10-16T09:44:42.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a8983700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a40ce080 con 0x7fe5a415e150 2023-10-16T09:44:42.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5a40b24c0 0x7fe5a40ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe5a415a5d0 0x7fe5a40b1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48588/0 (socket says 172.21.15.158:48588) 2023-10-16T09:44:42.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a259c700 1 -- 172.21.15.158:0/2716255459 learned_addr learned my addr 172.21.15.158:0/2716255459 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:42.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a2d9d700 1 --2- 172.21.15.158:0/2716255459 >> v2:172.21.15.172:3300/0 conn(0x7fe5a415e150 0x7fe5a415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a1d9b700 1 -- 172.21.15.158:0/2716255459 >> v2:172.21.15.172:3300/0 conn(0x7fe5a415e150 msgr2=0x7fe5a415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:42.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a1d9b700 1 --2- 172.21.15.158:0/2716255459 >> v2:172.21.15.172:3300/0 conn(0x7fe5a415e150 0x7fe5a415e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a1d9b700 1 -- 172.21.15.158:0/2716255459 >> v1:172.21.15.158:6789/0 conn(0x7fe5a415a5d0 legacy=0x7fe5a40b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.201+0000 7fe5a159a700 1 -- 172.21.15.158:0/2716255459 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822723630 0 0) 0x7fe5a415eb20 con 0x7fe5a415a5d0 2023-10-16T09:44:42.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a1d9b700 1 -- 172.21.15.158:0/2716255459 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40ce360 con 0x7fe5a40b24c0 2023-10-16T09:44:42.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a1d9b700 1 --2- 172.21.15.158:0/2716255459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a40b24c0 0x7fe5a40ceb60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4072d80 tx=0x7fe594009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc784f322b50dcad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:42.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a159a700 1 -- 172.21.15.158:0/2716255459 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594011830 con 0x7fe5a40b24c0 2023-10-16T09:44:42.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a159a700 1 -- 172.21.15.158:0/2716255459 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe594011990 con 0x7fe5a40b24c0 2023-10-16T09:44:42.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a159a700 1 -- 172.21.15.158:0/2716255459 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594011d20 con 0x7fe5a40b24c0 2023-10-16T09:44:42.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a8983700 1 -- 172.21.15.158:0/2716255459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a40b24c0 msgr2=0x7fe5a40ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a8983700 1 --2- 172.21.15.158:0/2716255459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a40b24c0 0x7fe5a40ceb60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4072d80 tx=0x7fe594009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:42.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a8983700 1 -- 172.21.15.158:0/2716255459 shutdown_connections 2023-10-16T09:44:42.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a8983700 1 --2- 172.21.15.158:0/2716255459 >> v2:172.21.15.172:3300/0 conn(0x7fe5a415e150 0x7fe5a415e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a8983700 1 --2- 172.21.15.158:0/2716255459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a40b24c0 0x7fe5a40ceb60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a8983700 1 -- 172.21.15.158:0/2716255459 >> 172.21.15.158:0/2716255459 conn(0x7fe5a40ccdb0 msgr2=0x7fe5a40cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:42.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.202+0000 7fe5a8983700 1 -- 172.21.15.158:0/2716255459 shutdown_connections 2023-10-16T09:44:42.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.203+0000 7fe5a8983700 1 -- 172.21.15.158:0/2716255459 wait complete. 2023-10-16T09:44:42.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.204+0000 7fe5a8983700 1 Processor -- start 2023-10-16T09:44:42.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.204+0000 7fe5a8983700 1 -- start start 2023-10-16T09:44:42.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5a40b24c0 0x7fe5a415cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5a415a5d0 0x7fe5a41ef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a415e150 0x7fe5a41f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a415dcf0 con 0x7fe5a415e150 2023-10-16T09:44:42.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41efa00 con 0x7fe5a415a5d0 2023-10-16T09:44:42.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41f4090 con 0x7fe5a40b24c0 2023-10-16T09:44:42.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a415e150 0x7fe5a41f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a415e150 0x7fe5a41f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48422/0 (socket says 172.21.15.158:48422) 2023-10-16T09:44:42.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 -- 172.21.15.158:0/1992076779 learned_addr learned my addr 172.21.15.158:0/1992076779 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:42.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a1d9b700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5a415a5d0 0x7fe5a41ef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a259c700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5a40b24c0 0x7fe5a415cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 -- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5a40b24c0 msgr2=0x7fe5a415cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5a40b24c0 0x7fe5a415cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 -- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5a415a5d0 msgr2=0x7fe5a41ef4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5a415a5d0 0x7fe5a41ef4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 -- 172.21.15.158:0/1992076779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940094a0 con 0x7fe5a415e150 2023-10-16T09:44:42.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a1d9b700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5a415a5d0 0x7fe5a41ef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:42.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a259c700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5a40b24c0 0x7fe5a415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:42.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a2d9d700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a415e150 0x7fe5a41f3b50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00cc30 tx=0x7fe59c00a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:42.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe592ffd700 1 -- 172.21.15.158:0/1992076779 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00ab70 con 0x7fe5a415e150 2023-10-16T09:44:42.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe592ffd700 1 -- 172.21.15.158:0/1992076779 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c00acd0 con 0x7fe5a415e150 2023-10-16T09:44:42.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a415d170 con 0x7fe5a415e150 2023-10-16T09:44:42.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe592ffd700 1 -- 172.21.15.158:0/1992076779 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c003c40 con 0x7fe5a415e150 2023-10-16T09:44:42.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.205+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a415d730 con 0x7fe5a415e150 2023-10-16T09:44:42.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.206+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a40c4890 con 0x7fe5a415e150 2023-10-16T09:44:42.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.207+0000 7fe592ffd700 1 -- 172.21.15.158:0/1992076779 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe59c00f3f0 con 0x7fe5a415e150 2023-10-16T09:44:42.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.207+0000 7fe592ffd700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe58406af00 0x7fe58406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.207+0000 7fe592ffd700 1 -- 172.21.15.158:0/1992076779 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe59c0898f0 con 0x7fe5a415e150 2023-10-16T09:44:42.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.207+0000 7fe5a259c700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe58406af00 0x7fe58406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.208+0000 7fe5a259c700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe58406af00 0x7fe58406d3c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe598000f60 tx=0x7fe598009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:42.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.211+0000 7fe592ffd700 1 -- 172.21.15.158:0/1992076779 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5a40c4890 con 0x7fe5a415e150 2023-10-16T09:44:42.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.403+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5a40c4890 con 0x7fe5a415e150 2023-10-16T09:44:42.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.404+0000 7fe592ffd700 1 -- 172.21.15.158:0/1992076779 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16799 (secure 0 0 0) 0x7fe5a40c4890 con 0x7fe5a415e150 2023-10-16T09:44:42.406 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:42.407 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":38,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:41.089845+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:44:38.770851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T09:44:40.065625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:42.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.406+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe58406af00 msgr2=0x7fe58406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.406+0000 7fe5a8983700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe58406af00 0x7fe58406d3c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe598000f60 tx=0x7fe598009040 comp rx=0 tx=0).stop 2023-10-16T09:44:42.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.406+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a415e150 msgr2=0x7fe5a41f3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.406+0000 7fe5a8983700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a415e150 0x7fe5a41f3b50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00cc30 tx=0x7fe59c00a1f0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 shutdown_connections 2023-10-16T09:44:42.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5a415e150 0x7fe5a41f3b50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fe58406af00 0x7fe58406d3c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5a40b24c0 0x7fe5a415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 --2- 172.21.15.158:0/1992076779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5a415a5d0 0x7fe5a41ef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 >> 172.21.15.158:0/1992076779 conn(0x7fe5a40ccdb0 msgr2=0x7fe5a40cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:42.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 shutdown_connections 2023-10-16T09:44:42.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.407+0000 7fe5a8983700 1 -- 172.21.15.158:0/1992076779 wait complete. 2023-10-16T09:44:42.421 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T09:44:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.542+0000 7f8a98348700 1 Processor -- start 2023-10-16T09:44:42.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.543+0000 7f8a98348700 1 -- start start 2023-10-16T09:44:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.543+0000 7f8a98348700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8a9015d990 0x7f8a9006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.543+0000 7f8a98348700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a90070300 0x7f8a90161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.543+0000 7f8a98348700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a900cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.543+0000 7f8a98348700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a901616b0 con 0x7f8a90070300 2023-10-16T09:44:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.543+0000 7f8a98348700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a900ced70 con 0x7f8a9015d990 2023-10-16T09:44:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.543+0000 7f8a98348700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9015dd70 con 0x7f8a90162010 2023-10-16T09:44:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a90070300 0x7f8a90161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a90070300 0x7f8a90161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48426/0 (socket says 172.21.15.158:48426) 2023-10-16T09:44:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 -- 172.21.15.158:0/629410092 learned_addr learned my addr 172.21.15.158:0/629410092 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a968e5700 1 --2- 172.21.15.158:0/629410092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a900cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a960e4700 1 --2- 172.21.15.158:0/629410092 >> v2:172.21.15.172:3300/0 conn(0x7f8a9015d990 0x7f8a9006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 -- 172.21.15.158:0/629410092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 msgr2=0x7f8a900cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:42.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 --2- 172.21.15.158:0/629410092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a900cbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 -- 172.21.15.158:0/629410092 >> v2:172.21.15.172:3300/0 conn(0x7f8a9015d990 msgr2=0x7f8a9006d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:42.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 --2- 172.21.15.158:0/629410092 >> v2:172.21.15.172:3300/0 conn(0x7f8a9015d990 0x7f8a9006d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 -- 172.21.15.158:0/629410092 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a90070780 con 0x7f8a90070300 2023-10-16T09:44:42.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a958e3700 1 --2- 172.21.15.158:0/629410092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a90070300 0x7f8a90161ad0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8a900d25a0 tx=0x7f8a8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c044cc87488f89bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:42.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a950e2700 1 -- 172.21.15.158:0/629410092 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c00b860 con 0x7f8a90070300 2023-10-16T09:44:42.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a950e2700 1 -- 172.21.15.158:0/629410092 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c00b9c0 con 0x7f8a90070300 2023-10-16T09:44:42.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.544+0000 7f8a950e2700 1 -- 172.21.15.158:0/629410092 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c00bd50 con 0x7f8a90070300 2023-10-16T09:44:42.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 -- 172.21.15.158:0/629410092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a90070300 msgr2=0x7f8a90161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 --2- 172.21.15.158:0/629410092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a90070300 0x7f8a90161ad0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8a900d25a0 tx=0x7f8a8c009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 -- 172.21.15.158:0/629410092 shutdown_connections 2023-10-16T09:44:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 --2- 172.21.15.158:0/629410092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a900cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 --2- 172.21.15.158:0/629410092 >> v2:172.21.15.172:3300/0 conn(0x7f8a9015d990 0x7f8a9006d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 --2- 172.21.15.158:0/629410092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a90070300 0x7f8a90161ad0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 -- 172.21.15.158:0/629410092 >> 172.21.15.158:0/629410092 conn(0x7f8a900cdf20 msgr2=0x7f8a900ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 -- 172.21.15.158:0/629410092 shutdown_connections 2023-10-16T09:44:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.545+0000 7f8a98348700 1 -- 172.21.15.158:0/629410092 wait complete. 2023-10-16T09:44:42.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.546+0000 7f8a98348700 1 Processor -- start 2023-10-16T09:44:42.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.546+0000 7f8a98348700 1 -- start start 2023-10-16T09:44:42.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a98348700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a90070300 0x7f8a901592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a98348700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9015d990 0x7f8a901f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a98348700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a901f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a98348700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901f7b80 con 0x7f8a9015d990 2023-10-16T09:44:42.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a98348700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a90157bb0 con 0x7f8a90070300 2023-10-16T09:44:42.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a98348700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a90153290 con 0x7f8a90162010 2023-10-16T09:44:42.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9015d990 0x7f8a901f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9015d990 0x7f8a901f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48434/0 (socket says 172.21.15.158:48434) 2023-10-16T09:44:42.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 -- 172.21.15.158:0/2248000838 learned_addr learned my addr 172.21.15.158:0/2248000838 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:42.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a968e5700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a901f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a960e4700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a90070300 0x7f8a901592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 -- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 msgr2=0x7f8a901f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a901f71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 -- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a90070300 msgr2=0x7f8a901592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:42.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a90070300 0x7f8a901592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:42.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 -- 172.21.15.158:0/2248000838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c009550 con 0x7f8a9015d990 2023-10-16T09:44:42.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a960e4700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a90070300 0x7f8a901592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:42.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a958e3700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9015d990 0x7f8a901f6c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009840 tx=0x7f8a8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:42.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.547+0000 7f8a86ffd700 1 -- 172.21.15.158:0/2248000838 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c004510 con 0x7f8a9015d990 2023-10-16T09:44:42.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.548+0000 7f8a86ffd700 1 -- 172.21.15.158:0/2248000838 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c01e070 con 0x7f8a9015d990 2023-10-16T09:44:42.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.548+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a900c6580 con 0x7f8a9015d990 2023-10-16T09:44:42.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.548+0000 7f8a86ffd700 1 -- 172.21.15.158:0/2248000838 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c0038c0 con 0x7f8a9015d990 2023-10-16T09:44:42.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.548+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a90158530 con 0x7f8a9015d990 2023-10-16T09:44:42.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.548+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a900681b0 con 0x7f8a9015d990 2023-10-16T09:44:42.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.549+0000 7f8a86ffd700 1 -- 172.21.15.158:0/2248000838 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8a8c003a20 con 0x7f8a9015d990 2023-10-16T09:44:42.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.549+0000 7f8a86ffd700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8a7406af00 0x7f8a7406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:42.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.549+0000 7f8a86ffd700 1 -- 172.21.15.158:0/2248000838 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8a8c020070 con 0x7f8a9015d990 2023-10-16T09:44:42.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.549+0000 7f8a960e4700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8a7406af00 0x7f8a7406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:42.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.550+0000 7f8a960e4700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8a7406af00 0x7f8a7406d3c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a901ea790 tx=0x7f8a80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:42.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.554+0000 7f8a86ffd700 1 -- 172.21.15.158:0/2248000838 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a8c0559e0 con 0x7f8a9015d990 2023-10-16T09:44:42.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:42.795+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8a900b70a0 con 0x7f8a9015d990 2023-10-16T09:44:43.675 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T09:44:43.674+0000 7fee99b9a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T09:44:43.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.731+0000 7f8a86ffd700 1 -- 172.21.15.158:0/2248000838 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8a8c02a020 con 0x7f8a9015d990 2023-10-16T09:44:43.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8a7406af00 msgr2=0x7f8a7406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:43.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8a7406af00 0x7f8a7406d3c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a901ea790 tx=0x7f8a80008040 comp rx=0 tx=0).stop 2023-10-16T09:44:43.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9015d990 msgr2=0x7f8a901f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:43.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9015d990 0x7f8a901f6c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009840 tx=0x7f8a8c004750 comp rx=0 tx=0).stop 2023-10-16T09:44:43.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 shutdown_connections 2023-10-16T09:44:43.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8a7406af00 0x7f8a7406d3c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a90162010 0x7f8a901f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a90070300 0x7f8a901592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 --2- 172.21.15.158:0/2248000838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9015d990 0x7f8a901f6c90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 >> 172.21.15.158:0/2248000838 conn(0x7f8a900cdf20 msgr2=0x7f8a9015df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:43.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 shutdown_connections 2023-10-16T09:44:43.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.734+0000 7f8a98348700 1 -- 172.21.15.158:0/2248000838 wait complete. 2023-10-16T09:44:43.751 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T09:44:43.751 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T09:44:43.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.879+0000 7feb45f45700 1 Processor -- start 2023-10-16T09:44:43.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.879+0000 7feb45f45700 1 -- start start 2023-10-16T09:44:43.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb45f45700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb4015dc40 0x7feb4015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:43.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb45f45700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb4006de70 0x7feb4006f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:43.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb45f45700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4015f380 con 0x7feb4006d9c0 2023-10-16T09:44:43.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb45f45700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4006e2e0 con 0x7feb4015dc40 2023-10-16T09:44:43.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb45f45700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb40071e70 con 0x7feb4006de70 2023-10-16T09:44:43.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb4015dc40 0x7feb4015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:43.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7feb4006d9c0 0x7feb4006f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48616/0 (socket says 172.21.15.158:48616) 2023-10-16T09:44:43.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3effd700 1 -- 172.21.15.158:0/3357893589 learned_addr learned my addr 172.21.15.158:0/3357893589 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:43.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3ffff700 1 --2- 172.21.15.158:0/3357893589 >> v2:172.21.15.172:3300/0 conn(0x7feb4006de70 0x7feb4006f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:43.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3f7fe700 1 -- 172.21.15.158:0/3357893589 >> v2:172.21.15.172:3300/0 conn(0x7feb4006de70 msgr2=0x7feb4006f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:43.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3f7fe700 1 --2- 172.21.15.158:0/3357893589 >> v2:172.21.15.172:3300/0 conn(0x7feb4006de70 0x7feb4006f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3f7fe700 1 -- 172.21.15.158:0/3357893589 >> v1:172.21.15.158:6789/0 conn(0x7feb4006d9c0 legacy=0x7feb4006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:43.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3e7fc700 1 -- 172.21.15.158:0/3357893589 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437182099 0 0) 0x7feb28001490 con 0x7feb4006d9c0 2023-10-16T09:44:43.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3f7fe700 1 -- 172.21.15.158:0/3357893589 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40072150 con 0x7feb4015dc40 2023-10-16T09:44:43.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3ffff700 1 --2- 172.21.15.158:0/3357893589 >> v2:172.21.15.172:3300/0 conn(0x7feb4006de70 0x7feb4006f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:43.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3f7fe700 1 --2- 172.21.15.158:0/3357893589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4015dc40 0x7feb4015ee40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feb400afe20 tx=0x7feb30009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781bf0cd044e45b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:43.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.880+0000 7feb3e7fc700 1 -- 172.21.15.158:0/3357893589 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30011830 con 0x7feb4015dc40 2023-10-16T09:44:43.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb3e7fc700 1 -- 172.21.15.158:0/3357893589 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb30011990 con 0x7feb4015dc40 2023-10-16T09:44:43.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb3e7fc700 1 -- 172.21.15.158:0/3357893589 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30011d20 con 0x7feb4015dc40 2023-10-16T09:44:43.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 -- 172.21.15.158:0/3357893589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4015dc40 msgr2=0x7feb4015ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:43.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 --2- 172.21.15.158:0/3357893589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4015dc40 0x7feb4015ee40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feb400afe20 tx=0x7feb30009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:43.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 -- 172.21.15.158:0/3357893589 shutdown_connections 2023-10-16T09:44:43.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 --2- 172.21.15.158:0/3357893589 >> v2:172.21.15.172:3300/0 conn(0x7feb4006de70 0x7feb4006f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 --2- 172.21.15.158:0/3357893589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4015dc40 0x7feb4015ee40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 -- 172.21.15.158:0/3357893589 >> 172.21.15.158:0/3357893589 conn(0x7feb400d03b0 msgr2=0x7feb400d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:43.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 -- 172.21.15.158:0/3357893589 shutdown_connections 2023-10-16T09:44:43.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.881+0000 7feb45f45700 1 -- 172.21.15.158:0/3357893589 wait complete. 2023-10-16T09:44:43.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.882+0000 7feb45f45700 1 Processor -- start 2023-10-16T09:44:43.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb45f45700 1 -- start start 2023-10-16T09:44:43.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb45f45700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb4006d9c0 0x7feb400ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:43.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb45f45700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006de70 0x7feb401585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:43.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb45f45700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb4015dc40 0x7feb400c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:43.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb45f45700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400af340 con 0x7feb4006de70 2023-10-16T09:44:43.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb45f45700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400ae730 con 0x7feb4015dc40 2023-10-16T09:44:43.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb45f45700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb400aea10 con 0x7feb4006d9c0 2023-10-16T09:44:43.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb3effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006de70 0x7feb401585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:43.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb3effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006de70 0x7feb401585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48448/0 (socket says 172.21.15.158:48448) 2023-10-16T09:44:43.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.883+0000 7feb3effd700 1 -- 172.21.15.158:0/2031874980 learned_addr learned my addr 172.21.15.158:0/2031874980 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:43.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3ffff700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb4015dc40 0x7feb400c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:43.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3effd700 1 -- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb4006d9c0 msgr2=0x7feb400ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:43.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3effd700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb4006d9c0 0x7feb400ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3effd700 1 -- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb4015dc40 msgr2=0x7feb400c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:43.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3effd700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb4015dc40 0x7feb400c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:43.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3effd700 1 -- 172.21.15.158:0/2031874980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb300094a0 con 0x7feb4006de70 2023-10-16T09:44:43.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3ffff700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb4015dc40 0x7feb400c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:43.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb3effd700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006de70 0x7feb401585a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7feb28001f80 tx=0x7feb2800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:43.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb37fff700 1 -- 172.21.15.158:0/2031874980 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800ad80 con 0x7feb4006de70 2023-10-16T09:44:43.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb37fff700 1 -- 172.21.15.158:0/2031874980 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb280070d0 con 0x7feb4006de70 2023-10-16T09:44:43.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb400aecf0 con 0x7feb4006de70 2023-10-16T09:44:43.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb37fff700 1 -- 172.21.15.158:0/2031874980 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28007480 con 0x7feb4006de70 2023-10-16T09:44:43.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.884+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb400af7d0 con 0x7feb4006de70 2023-10-16T09:44:43.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.885+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb400681b0 con 0x7feb4006de70 2023-10-16T09:44:43.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.885+0000 7feb37fff700 1 -- 172.21.15.158:0/2031874980 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feb280075e0 con 0x7feb4006de70 2023-10-16T09:44:43.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.886+0000 7feb37fff700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb2406af00 0x7feb2406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:43.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.886+0000 7feb37fff700 1 -- 172.21.15.158:0/2031874980 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feb28089570 con 0x7feb4006de70 2023-10-16T09:44:43.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.886+0000 7feb3f7fe700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb2406af00 0x7feb2406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:43.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.886+0000 7feb3f7fe700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb2406af00 0x7feb2406d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7feb400cde20 tx=0x7feb30010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:43.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:43.890+0000 7feb37fff700 1 -- 172.21.15.158:0/2031874980 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb280544e0 con 0x7feb4006de70 2023-10-16T09:44:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.130+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7feb4015f380 con 0x7feb4006de70 2023-10-16T09:44:44.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.732+0000 7feb37fff700 1 -- 172.21.15.158:0/2031874980 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7feb4015f380 con 0x7feb4006de70 2023-10-16T09:44:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb2406af00 msgr2=0x7feb2406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb2406af00 0x7feb2406d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7feb400cde20 tx=0x7feb30010000 comp rx=0 tx=0).stop 2023-10-16T09:44:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006de70 msgr2=0x7feb401585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006de70 0x7feb401585a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7feb28001f80 tx=0x7feb2800a340 comp rx=0 tx=0).stop 2023-10-16T09:44:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 shutdown_connections 2023-10-16T09:44:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb4015dc40 0x7feb400c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb2406af00 0x7feb2406d3c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:44.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb4006d9c0 0x7feb400ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:44.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 --2- 172.21.15.158:0/2031874980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006de70 0x7feb401585a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:44.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 >> 172.21.15.158:0/2031874980 conn(0x7feb400d03b0 msgr2=0x7feb400b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:44.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 shutdown_connections 2023-10-16T09:44:44.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.735+0000 7feb45f45700 1 -- 172.21.15.158:0/2031874980 wait complete. 2023-10-16T09:44:44.751 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T09:44:44.752 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:44:44.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.876+0000 7f8f3a1e4700 1 Processor -- start 2023-10-16T09:44:44.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.876+0000 7f8f3a1e4700 1 -- start start 2023-10-16T09:44:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f3a1e4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f340cdee0 0x7f8f3415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f3a1e4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3415c5d0 con 0x7f8f340ce2c0 2023-10-16T09:44:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f3a1e4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3415c8b0 con 0x7f8f340b6e10 2023-10-16T09:44:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f3a1e4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3415d2c0 con 0x7f8f340cdee0 2023-10-16T09:44:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f337fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f340cdee0 0x7f8f3415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f33fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8f340ce2c0 0x7f8f340cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48634/0 (socket says 172.21.15.158:48634) 2023-10-16T09:44:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f33fff700 1 -- 172.21.15.158:0/3490802397 learned_addr learned my addr 172.21.15.158:0/3490802397 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f337fe700 1 -- 172.21.15.158:0/3490802397 >> v1:172.21.15.172:6789/0 conn(0x7f8f340b6e10 legacy=0x7f8f341521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f337fe700 1 -- 172.21.15.158:0/3490802397 >> v1:172.21.15.158:6789/0 conn(0x7f8f340ce2c0 legacy=0x7f8f340cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.877+0000 7f8f337fe700 1 -- 172.21.15.158:0/3490802397 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3415cb90 con 0x7f8f340cdee0 2023-10-16T09:44:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f337fe700 1 --2- 172.21.15.158:0/3490802397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340cdee0 0x7f8f3415bd10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009840 tx=0x7f8f1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b8d442c60f2185 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f327fc700 1 -- 172.21.15.158:0/3490802397 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c004230 con 0x7f8f340cdee0 2023-10-16T09:44:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f327fc700 1 -- 172.21.15.158:0/3490802397 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c004390 con 0x7f8f340cdee0 2023-10-16T09:44:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f327fc700 1 -- 172.21.15.158:0/3490802397 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c0036a0 con 0x7f8f340cdee0 2023-10-16T09:44:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/3490802397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340cdee0 msgr2=0x7f8f3415bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:44.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/3490802397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340cdee0 0x7f8f3415bd10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009840 tx=0x7f8f1c009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:44.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/3490802397 shutdown_connections 2023-10-16T09:44:44.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/3490802397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340cdee0 0x7f8f3415bd10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:44.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/3490802397 >> 172.21.15.158:0/3490802397 conn(0x7f8f3406a8d0 msgr2=0x7f8f3406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:44.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/3490802397 shutdown_connections 2023-10-16T09:44:44.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.878+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/3490802397 wait complete. 2023-10-16T09:44:44.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.880+0000 7f8f3a1e4700 1 Processor -- start 2023-10-16T09:44:44.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.880+0000 7f8f3a1e4700 1 -- start start 2023-10-16T09:44:44.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.880+0000 7f8f3a1e4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f340b6e10 0x7f8f34152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:44.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f3a1e4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f340cdee0 0x7f8f3414d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:44.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f3a1e4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340ce2c0 0x7f8f3414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:44.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f3a1e4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3414e1b0 con 0x7f8f340ce2c0 2023-10-16T09:44:44.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f3a1e4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340c5f80 con 0x7f8f340cdee0 2023-10-16T09:44:44.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f3a1e4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340c6c40 con 0x7f8f340b6e10 2023-10-16T09:44:44.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340ce2c0 0x7f8f3414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:44.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340ce2c0 0x7f8f3414dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48464/0 (socket says 172.21.15.158:48464) 2023-10-16T09:44:44.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 -- 172.21.15.158:0/2305959396 learned_addr learned my addr 172.21.15.158:0/2305959396 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:44.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f337fe700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f340b6e10 0x7f8f34152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:44.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f32ffd700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f340cdee0 0x7f8f3414d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:44.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 -- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f340b6e10 msgr2=0x7f8f34152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:44.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f340b6e10 0x7f8f34152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:44.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 -- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f340cdee0 msgr2=0x7f8f3414d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:44.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f340cdee0 0x7f8f3414d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:44.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 -- 172.21.15.158:0/2305959396 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0094a0 con 0x7f8f340ce2c0 2023-10-16T09:44:44.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f337fe700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f340b6e10 0x7f8f34152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:44.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f32ffd700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f340cdee0 0x7f8f3414d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:44.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f33fff700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340ce2c0 0x7f8f3414dc70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00ca80 tx=0x7f8f2c00ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:44.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f2bfff700 1 -- 172.21.15.158:0/2305959396 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c00aaa0 con 0x7f8f340ce2c0 2023-10-16T09:44:44.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f2bfff700 1 -- 172.21.15.158:0/2305959396 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c00ac00 con 0x7f8f340ce2c0 2023-10-16T09:44:44.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f340c6f20 con 0x7f8f340ce2c0 2023-10-16T09:44:44.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f2bfff700 1 -- 172.21.15.158:0/2305959396 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c012400 con 0x7f8f340ce2c0 2023-10-16T09:44:44.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.881+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f340c74e0 con 0x7f8f340ce2c0 2023-10-16T09:44:44.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.882+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f340681b0 con 0x7f8f340ce2c0 2023-10-16T09:44:44.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.883+0000 7f8f2bfff700 1 -- 172.21.15.158:0/2305959396 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8f2c01b3b0 con 0x7f8f340ce2c0 2023-10-16T09:44:44.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.883+0000 7f8f2bfff700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f1406af00 0x7f8f1406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:44.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.883+0000 7f8f2bfff700 1 -- 172.21.15.158:0/2305959396 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8f2c089270 con 0x7f8f340ce2c0 2023-10-16T09:44:44.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.883+0000 7f8f337fe700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f1406af00 0x7f8f1406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:44.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.884+0000 7f8f337fe700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f1406af00 0x7f8f1406d3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8f341537b0 tx=0x7f8f1c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:44.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:44.888+0000 7f8f2bfff700 1 -- 172.21.15.158:0/2305959396 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f2c054290 con 0x7f8f340ce2c0 2023-10-16T09:44:45.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.122+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f340b61e0 con 0x7f8f340ce2c0 2023-10-16T09:44:45.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.123+0000 7f8f2bfff700 1 -- 172.21.15.158:0/2305959396 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4928 (secure 0 0 0) 0x7f8f2c0578b0 con 0x7f8f340ce2c0 2023-10-16T09:44:45.125 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:45.125 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4283661376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4283661376},{"type":"v1","addr":"172.21.15.158:6835","nonce":4283661376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/3069959147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":3069959147},{"type":"v1","addr":"172.21.15.158:6837","nonce":3069959147}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1583191131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1583191131},{"type":"v1","addr":"172.21.15.172:6835","nonce":1583191131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:44:41.090180+0000","modified":"2023-10-16T09:44:44.131783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34760":{"gid":34760,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.172:6834/1706659835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1706659835},{"type":"v1","addr":"172.21.15.172:6834","nonce":1706659835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T09:44:45.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f1406af00 msgr2=0x7f8f1406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f1406af00 0x7f8f1406d3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8f341537b0 tx=0x7f8f1c008030 comp rx=0 tx=0).stop 2023-10-16T09:44:45.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340ce2c0 msgr2=0x7f8f3414dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340ce2c0 0x7f8f3414dc70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00ca80 tx=0x7f8f2c00ce40 comp rx=0 tx=0).stop 2023-10-16T09:44:45.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 shutdown_connections 2023-10-16T09:44:45.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f340ce2c0 0x7f8f3414dc70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f1406af00 0x7f8f1406d3c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f340b6e10 0x7f8f34152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 --2- 172.21.15.158:0/2305959396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f340cdee0 0x7f8f3414d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 >> 172.21.15.158:0/2305959396 conn(0x7f8f3406a8d0 msgr2=0x7f8f340b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:45.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 shutdown_connections 2023-10-16T09:44:45.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.125+0000 7f8f3a1e4700 1 -- 172.21.15.158:0/2305959396 wait complete. 2023-10-16T09:44:45.131 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 17 2023-10-16T09:44:45.139 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:44:45.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.259+0000 7ff4b5950700 1 Processor -- start 2023-10-16T09:44:45.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.259+0000 7ff4b5950700 1 -- start start 2023-10-16T09:44:45.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.260+0000 7ff4b5950700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff4b00b6e10 0x7ff4b01521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.260+0000 7ff4b5950700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b015c5d0 con 0x7ff4b00cdee0 2023-10-16T09:44:45.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.260+0000 7ff4b5950700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b015c8b0 con 0x7ff4b00ce2c0 2023-10-16T09:44:45.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.260+0000 7ff4b5950700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b015d2c0 con 0x7ff4b00b6e10 2023-10-16T09:44:45.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.260+0000 7ff4ae7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff4b00b6e10 0x7ff4b01521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.260+0000 7ff4aeffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff4b00cdee0 0x7ff4b015bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48644/0 (socket says 172.21.15.158:48644) 2023-10-16T09:44:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.260+0000 7ff4aeffd700 1 -- 172.21.15.158:0/1600667329 learned_addr learned my addr 172.21.15.158:0/1600667329 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4ae7fc700 1 -- 172.21.15.158:0/1600667329 >> v1:172.21.15.172:6789/0 conn(0x7ff4b00ce2c0 legacy=0x7ff4b00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4ae7fc700 1 -- 172.21.15.158:0/1600667329 >> v1:172.21.15.158:6789/0 conn(0x7ff4b00cdee0 legacy=0x7ff4b015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4adffb700 1 -- 172.21.15.158:0/1600667329 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858145771 0 0) 0x7ff4b015c5d0 con 0x7ff4b00cdee0 2023-10-16T09:44:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4ae7fc700 1 -- 172.21.15.158:0/1600667329 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b015cb90 con 0x7ff4b00b6e10 2023-10-16T09:44:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4ae7fc700 1 --2- 172.21.15.158:0/1600667329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b01521a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009840 tx=0x7ff4a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55000ae9979a01c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4adffb700 1 -- 172.21.15.158:0/1600667329 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0011830 con 0x7ff4b00b6e10 2023-10-16T09:44:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4adffb700 1 -- 172.21.15.158:0/1600667329 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a0011990 con 0x7ff4b00b6e10 2023-10-16T09:44:45.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4adffb700 1 -- 172.21.15.158:0/1600667329 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0011d20 con 0x7ff4b00b6e10 2023-10-16T09:44:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4b5950700 1 -- 172.21.15.158:0/1600667329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 msgr2=0x7ff4b01521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4b5950700 1 --2- 172.21.15.158:0/1600667329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b01521a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009840 tx=0x7ff4a0009b50 comp rx=0 tx=0).stop 2023-10-16T09:44:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4b5950700 1 -- 172.21.15.158:0/1600667329 shutdown_connections 2023-10-16T09:44:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4b5950700 1 --2- 172.21.15.158:0/1600667329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b01521a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4b5950700 1 -- 172.21.15.158:0/1600667329 >> 172.21.15.158:0/1600667329 conn(0x7ff4b006a8d0 msgr2=0x7ff4b006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:45.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4b5950700 1 -- 172.21.15.158:0/1600667329 shutdown_connections 2023-10-16T09:44:45.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.261+0000 7ff4b5950700 1 -- 172.21.15.158:0/1600667329 wait complete. 2023-10-16T09:44:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 Processor -- start 2023-10-16T09:44:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 -- start start 2023-10-16T09:44:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b0148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4b00cdee0 0x7ff4b0156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4b00ce2c0 0x7ff4b0156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b00cf9f0 con 0x7ff4b00b6e10 2023-10-16T09:44:45.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b00cfcd0 con 0x7ff4b00cdee0 2023-10-16T09:44:45.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.263+0000 7ff4b5950700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b00cffb0 con 0x7ff4b00ce2c0 2023-10-16T09:44:45.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b0148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b0148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48474/0 (socket says 172.21.15.158:48474) 2023-10-16T09:44:45.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 -- 172.21.15.158:0/2363312053 learned_addr learned my addr 172.21.15.158:0/2363312053 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:45.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4ae7fc700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4b00cdee0 0x7ff4b0156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4af7fe700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4b00ce2c0 0x7ff4b0156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 -- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4b00ce2c0 msgr2=0x7ff4b0156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4b00ce2c0 0x7ff4b0156880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 -- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4b00cdee0 msgr2=0x7ff4b0156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4b00cdee0 0x7ff4b0156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 -- 172.21.15.158:0/2363312053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00094a0 con 0x7ff4b00b6e10 2023-10-16T09:44:45.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4aeffd700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b0148970 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c520 tx=0x7ff4a400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:45.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff497fff700 1 -- 172.21.15.158:0/2363312053 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4004c80 con 0x7ff4b00b6e10 2023-10-16T09:44:45.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff497fff700 1 -- 172.21.15.158:0/2363312053 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a4004e00 con 0x7ff4b00b6e10 2023-10-16T09:44:45.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b00d0290 con 0x7ff4b00b6e10 2023-10-16T09:44:45.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff497fff700 1 -- 172.21.15.158:0/2363312053 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4007310 con 0x7ff4b00b6e10 2023-10-16T09:44:45.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.264+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b00c6e10 con 0x7ff4b00b6e10 2023-10-16T09:44:45.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.265+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b00d07b0 con 0x7ff4b00b6e10 2023-10-16T09:44:45.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.266+0000 7ff497fff700 1 -- 172.21.15.158:0/2363312053 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff4a4007470 con 0x7ff4b00b6e10 2023-10-16T09:44:45.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.266+0000 7ff497fff700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff49c06af00 0x7ff49c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.266+0000 7ff497fff700 1 -- 172.21.15.158:0/2363312053 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff4a4089300 con 0x7ff4b00b6e10 2023-10-16T09:44:45.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.266+0000 7ff4ae7fc700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff49c06af00 0x7ff49c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.267+0000 7ff4ae7fc700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff49c06af00 0x7ff49c06d3c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00090b0 tx=0x7ff4a001c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:45.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.271+0000 7ff497fff700 1 -- 172.21.15.158:0/2363312053 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4b00d07b0 con 0x7ff4b00b6e10 2023-10-16T09:44:45.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.465+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4b00d07b0 con 0x7ff4b00b6e10 2023-10-16T09:44:45.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.466+0000 7ff497fff700 1 -- 172.21.15.158:0/2363312053 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16786 (secure 0 0 0) 0x7ff4b00d07b0 con 0x7ff4b00b6e10 2023-10-16T09:44:45.468 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:45.468 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":39,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:44:42.695441+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:44:38.770851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T09:44:40.065625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:44:45.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff49c06af00 msgr2=0x7ff49c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff49c06af00 0x7ff49c06d3c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00090b0 tx=0x7ff4a001c040 comp rx=0 tx=0).stop 2023-10-16T09:44:45.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 msgr2=0x7ff4b0148970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b0148970 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c520 tx=0x7ff4a400c830 comp rx=0 tx=0).stop 2023-10-16T09:44:45.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 shutdown_connections 2023-10-16T09:44:45.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff49c06af00 0x7ff49c06d3c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4b00ce2c0 0x7ff4b0156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4b00b6e10 0x7ff4b0148970 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 --2- 172.21.15.158:0/2363312053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4b00cdee0 0x7ff4b0156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 >> 172.21.15.158:0/2363312053 conn(0x7ff4b006a8d0 msgr2=0x7ff4b015f100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 shutdown_connections 2023-10-16T09:44:45.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.468+0000 7ff4b5950700 1 -- 172.21.15.158:0/2363312053 wait complete. 2023-10-16T09:44:45.482 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T09:44:45.483 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:44:45.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.603+0000 7f8231da5700 1 Processor -- start 2023-10-16T09:44:45.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.603+0000 7f8231da5700 1 -- start start 2023-10-16T09:44:45.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.603+0000 7f8231da5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06ab40 0x7f822c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.603+0000 7f8231da5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c161270 con 0x7f822c06e5d0 2023-10-16T09:44:45.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.603+0000 7f8231da5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c161510 con 0x7f822c0729b0 2023-10-16T09:44:45.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.603+0000 7f8231da5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c1617f0 con 0x7f822c06ab40 2023-10-16T09:44:45.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f822c06e5d0 0x7f822c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48660/0 (socket says 172.21.15.158:48660) 2023-10-16T09:44:45.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822affd700 1 -- 172.21.15.158:0/3823776944 learned_addr learned my addr 172.21.15.158:0/3823776944 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:45.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822b7fe700 1 --2- 172.21.15.158:0/3823776944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06ab40 0x7f822c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239379257 0 0) 0x7f822c161270 con 0x7f822c06e5d0 2023-10-16T09:44:45.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82140035b0 con 0x7f822c06e5d0 2023-10-16T09:44:45.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565916999 0 0) 0x7f822c161510 con 0x7f822c0729b0 2023-10-16T09:44:45.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c161270 con 0x7f822c0729b0 2023-10-16T09:44:45.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1558144896 0 0) 0x7f82140035b0 con 0x7f822c06e5d0 2023-10-16T09:44:45.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822b7fe700 1 -- 172.21.15.158:0/3823776944 >> v1:172.21.15.172:6789/0 conn(0x7f822c0729b0 legacy=0x7f822c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822b7fe700 1 -- 172.21.15.158:0/3823776944 >> v1:172.21.15.158:6789/0 conn(0x7f822c06e5d0 legacy=0x7f822c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822b7fe700 1 -- 172.21.15.158:0/3823776944 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c161ad0 con 0x7f822c06ab40 2023-10-16T09:44:45.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822b7fe700 1 --2- 172.21.15.158:0/3823776944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06ab40 0x7f822c06d820 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f821c009870 tx=0x7f821c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f88ea072d56e543 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:45.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.604+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8220001670 con 0x7f822c06ab40 2023-10-16T09:44:45.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821c004030 con 0x7f822c06ab40 2023-10-16T09:44:45.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f822a7fc700 1 -- 172.21.15.158:0/3823776944 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c0043a0 con 0x7f822c06ab40 2023-10-16T09:44:45.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f8231da5700 1 -- 172.21.15.158:0/3823776944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06ab40 msgr2=0x7f822c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f8231da5700 1 --2- 172.21.15.158:0/3823776944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06ab40 0x7f822c06d820 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f821c009870 tx=0x7f821c009b80 comp rx=0 tx=0).stop 2023-10-16T09:44:45.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f8231da5700 1 -- 172.21.15.158:0/3823776944 shutdown_connections 2023-10-16T09:44:45.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f8231da5700 1 --2- 172.21.15.158:0/3823776944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06ab40 0x7f822c06d820 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f8231da5700 1 -- 172.21.15.158:0/3823776944 >> 172.21.15.158:0/3823776944 conn(0x7f822c0d02b0 msgr2=0x7f822c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:45.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f8231da5700 1 -- 172.21.15.158:0/3823776944 shutdown_connections 2023-10-16T09:44:45.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.605+0000 7f8231da5700 1 -- 172.21.15.158:0/3823776944 wait complete. 2023-10-16T09:44:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.606+0000 7f8231da5700 1 Processor -- start 2023-10-16T09:44:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.606+0000 7f8231da5700 1 -- start start 2023-10-16T09:44:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f8231da5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f822c06ab40 0x7f822c0b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f8231da5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06e5d0 0x7f822c0aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f8231da5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822c0729b0 0x7f822c0af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f8231da5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c0b1e40 con 0x7f822c0729b0 2023-10-16T09:44:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f8231da5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c0b3590 con 0x7f822c06ab40 2023-10-16T09:44:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f8231da5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c0b2990 con 0x7f822c06e5d0 2023-10-16T09:44:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f822bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822c0729b0 0x7f822c0af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f822bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822c0729b0 0x7f822c0af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48492/0 (socket says 172.21.15.158:48492) 2023-10-16T09:44:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.607+0000 7f822bfff700 1 -- 172.21.15.158:0/3234633419 learned_addr learned my addr 172.21.15.158:0/3234633419 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822b7fe700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f822c06ab40 0x7f822c0b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822affd700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06e5d0 0x7f822c0aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822bfff700 1 -- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06e5d0 msgr2=0x7f822c0aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822bfff700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06e5d0 0x7f822c0aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822bfff700 1 -- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f822c06ab40 msgr2=0x7f822c0b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822bfff700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f822c06ab40 0x7f822c0b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822bfff700 1 -- 172.21.15.158:0/3234633419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c0094a0 con 0x7f822c0729b0 2023-10-16T09:44:45.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822affd700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06e5d0 0x7f822c0aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:45.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f822bfff700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822c0729b0 0x7f822c0af030 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f822400cb50 tx=0x7f822400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:45.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f8213fff700 1 -- 172.21.15.158:0/3234633419 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8224003c00 con 0x7f822c0729b0 2023-10-16T09:44:45.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f8213fff700 1 -- 172.21.15.158:0/3234633419 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8224003d60 con 0x7f822c0729b0 2023-10-16T09:44:45.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c1f39b0 con 0x7f822c0729b0 2023-10-16T09:44:45.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f8213fff700 1 -- 172.21.15.158:0/3234633419 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822400aac0 con 0x7f822c0729b0 2023-10-16T09:44:45.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.608+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822c0b3220 con 0x7f822c0729b0 2023-10-16T09:44:45.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.609+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822c0b3720 con 0x7f822c0729b0 2023-10-16T09:44:45.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.609+0000 7f8213fff700 1 -- 172.21.15.158:0/3234633419 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f822400ac20 con 0x7f822c0729b0 2023-10-16T09:44:45.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.610+0000 7f8213fff700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f821806af00 0x7f821806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.610+0000 7f8213fff700 1 -- 172.21.15.158:0/3234633419 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8224089760 con 0x7f822c0729b0 2023-10-16T09:44:45.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.610+0000 7f822b7fe700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f821806af00 0x7f821806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.610+0000 7f822b7fe700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f821806af00 0x7f821806d3c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097c0 tx=0x7f821c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:45.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.614+0000 7f8213fff700 1 -- 172.21.15.158:0/3234633419 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82240546d0 con 0x7f822c0729b0 2023-10-16T09:44:45.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.806+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f822c162090 con 0x7f821806af00 2023-10-16T09:44:45.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.814+0000 7f8213fff700 1 -- 172.21.15.158:0/3234633419 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276888 (secure 0 0 0) 0x7f821c0042c0 con 0x7f821806af00 2023-10-16T09:44:45.819 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:45.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f821806af00 msgr2=0x7f821806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f821806af00 0x7f821806d3c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097c0 tx=0x7f821c008030 comp rx=0 tx=0).stop 2023-10-16T09:44:45.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822c0729b0 msgr2=0x7f822c0af030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822c0729b0 0x7f822c0af030 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f822400cb50 tx=0x7f822400ce60 comp rx=0 tx=0).stop 2023-10-16T09:44:45.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 shutdown_connections 2023-10-16T09:44:45.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822c0729b0 0x7f822c0af030 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f822c06ab40 0x7f822c0b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f821806af00 0x7f821806d3c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 --2- 172.21.15.158:0/3234633419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f822c06e5d0 0x7f822c0aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 >> 172.21.15.158:0/3234633419 conn(0x7f822c0d02b0 msgr2=0x7f822c160830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:45.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 shutdown_connections 2023-10-16T09:44:45.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.817+0000 7f8231da5700 1 -- 172.21.15.158:0/3234633419 wait complete. 2023-10-16T09:44:45.823 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:44:45.831 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2023-10-16T09:44:45.235635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220552,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754168,"statfs":{"total":773094113280,"available":772868268032,"internally_reserved":0,"allocated":4497408,"data_stored":2582784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552013"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690157+0000","last_change":"2023-10-16T09:44:39.690157+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690157+0000","last_undegraded":"2023-10-16T09:44:39.690157+0000","last_fullsized":"2023-10-16T09:44:39.690157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689987+0000","last_change":"2023-10-16T09:44:39.689987+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689987+0000","last_undegraded":"2023-10-16T09:44:39.689987+0000","last_fullsized":"2023-10-16T09:44:39.689987+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689036+0000","last_change":"2023-10-16T09:44:39.689036+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689036+0000","last_undegraded":"2023-10-16T09:44:39.689036+0000","last_fullsized":"2023-10-16T09:44:39.689036+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686535+0000","last_change":"2023-10-16T09:44:39.686535+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686535+0000","last_undegraded":"2023-10-16T09:44:39.686535+0000","last_fullsized":"2023-10-16T09:44:39.686535+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689557+0000","last_change":"2023-10-16T09:44:39.689557+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689557+0000","last_undegraded":"2023-10-16T09:44:39.689557+0000","last_fullsized":"2023-10-16T09:44:39.689557+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.688204+0000","last_change":"2023-10-16T09:44:39.688204+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.688204+0000","last_undegraded":"2023-10-16T09:44:39.688204+0000","last_fullsized":"2023-10-16T09:44:39.688204+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690430+0000","last_change":"2023-10-16T09:44:39.690430+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690430+0000","last_undegraded":"2023-10-16T09:44:39.690430+0000","last_fullsized":"2023-10-16T09:44:39.690430+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686522+0000","last_change":"2023-10-16T09:44:39.686522+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686522+0000","last_undegraded":"2023-10-16T09:44:39.686522+0000","last_fullsized":"2023-10-16T09:44:39.686522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690184+0000","last_change":"2023-10-16T09:44:39.690184+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690184+0000","last_undegraded":"2023-10-16T09:44:39.690184+0000","last_fullsized":"2023-10-16T09:44:39.690184+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689380+0000","last_change":"2023-10-16T09:44:39.689380+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689380+0000","last_undegraded":"2023-10-16T09:44:39.689380+0000","last_fullsized":"2023-10-16T09:44:39.689380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686325+0000","last_change":"2023-10-16T09:44:39.686325+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686325+0000","last_undegraded":"2023-10-16T09:44:39.686325+0000","last_fullsized":"2023-10-16T09:44:39.686325+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.688939+0000","last_change":"2023-10-16T09:44:39.688939+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.688939+0000","last_undegraded":"2023-10-16T09:44:39.688939+0000","last_fullsized":"2023-10-16T09:44:39.688939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690190+0000","last_change":"2023-10-16T09:44:39.690190+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690190+0000","last_undegraded":"2023-10-16T09:44:39.690190+0000","last_fullsized":"2023-10-16T09:44:39.690190+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687872+0000","last_change":"2023-10-16T09:44:39.687872+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687872+0000","last_undegraded":"2023-10-16T09:44:39.687872+0000","last_fullsized":"2023-10-16T09:44:39.687872+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690013+0000","last_change":"2023-10-16T09:44:39.690013+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690013+0000","last_undegraded":"2023-10-16T09:44:39.690013+0000","last_fullsized":"2023-10-16T09:44:39.690013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686521+0000","last_change":"2023-10-16T09:44:39.686521+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686521+0000","last_undegraded":"2023-10-16T09:44:39.686521+0000","last_fullsized":"2023-10-16T09:44:39.686521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692427+0000","last_change":"2023-10-16T09:44:39.692427+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692427+0000","last_undegraded":"2023-10-16T09:44:39.692427+0000","last_fullsized":"2023-10-16T09:44:39.692427+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689530+0000","last_change":"2023-10-16T09:44:39.689530+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689530+0000","last_undegraded":"2023-10-16T09:44:39.689530+0000","last_fullsized":"2023-10-16T09:44:39.689530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.693948+0000","last_change":"2023-10-16T09:44:39.693948+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.693948+0000","last_undegraded":"2023-10-16T09:44:39.693948+0000","last_fullsized":"2023-10-16T09:44:39.693948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692069+0000","last_change":"2023-10-16T09:44:39.692069+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692069+0000","last_undegraded":"2023-10-16T09:44:39.692069+0000","last_fullsized":"2023-10-16T09:44:39.692069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687912+0000","last_change":"2023-10-16T09:44:39.687912+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687912+0000","last_undegraded":"2023-10-16T09:44:39.687912+0000","last_fullsized":"2023-10-16T09:44:39.687912+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692055+0000","last_change":"2023-10-16T09:44:39.692055+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692055+0000","last_undegraded":"2023-10-16T09:44:39.692055+0000","last_fullsized":"2023-10-16T09:44:39.692055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689543+0000","last_change":"2023-10-16T09:44:39.689543+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689543+0000","last_undegraded":"2023-10-16T09:44:39.689543+0000","last_fullsized":"2023-10-16T09:44:39.689543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692248+0000","last_change":"2023-10-16T09:44:39.692248+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692248+0000","last_undegraded":"2023-10-16T09:44:39.692248+0000","last_fullsized":"2023-10-16T09:44:39.692248+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689081+0000","last_change":"2023-10-16T09:44:39.689081+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689081+0000","last_undegraded":"2023-10-16T09:44:39.689081+0000","last_fullsized":"2023-10-16T09:44:39.689081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689742+0000","last_change":"2023-10-16T09:44:39.689742+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689742+0000","last_undegraded":"2023-10-16T09:44:39.689742+0000","last_fullsized":"2023-10-16T09:44:39.689742+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.691440+0000","last_change":"2023-10-16T09:44:39.691440+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.691440+0000","last_undegraded":"2023-10-16T09:44:39.691440+0000","last_fullsized":"2023-10-16T09:44:39.691440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686535+0000","last_change":"2023-10-16T09:44:39.686535+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686535+0000","last_undegraded":"2023-10-16T09:44:39.686535+0000","last_fullsized":"2023-10-16T09:44:39.686535+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687880+0000","last_change":"2023-10-16T09:44:39.687880+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687880+0000","last_undegraded":"2023-10-16T09:44:39.687880+0000","last_fullsized":"2023-10-16T09:44:39.687880+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-16T09:44:34.383810+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:44:34.383810+0000","last_peered":"2023-10-16T09:44:34.383810+0000","last_clean":"2023-10-16T09:44:34.383810+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:44:34.383810+0000","last_undegraded":"2023-10-16T09:44:34.383810+0000","last_fullsized":"2023-10-16T09:44:34.383810+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":765624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":11.246,"5min":11.246,"15min":11.246},"last":0.56200000000000006}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.245},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.29799999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.73499999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.1760000000000002,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.72099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":765624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.376}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56399999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.629}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.39300000000000002}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.253}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.497}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.61499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.499},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.435}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.68100000000000005}]},{"osd":1,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.81499999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.84299999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.748},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.73099999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.79600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:44:45.837 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:44:45.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.990+0000 7ff40c248700 1 Processor -- start 2023-10-16T09:44:45.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.990+0000 7ff40c248700 1 -- start start 2023-10-16T09:44:45.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff40c248700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff404160e20 0x7ff4040cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff40c248700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca9f0 0x7ff40406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:45.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff40c248700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040afb60 con 0x7ff40406ae10 2023-10-16T09:44:45.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff40c248700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40406b2a0 con 0x7ff404160e20 2023-10-16T09:44:45.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff40c248700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040cfcf0 con 0x7ff4040ca9f0 2023-10-16T09:44:45.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff409fe4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff404160e20 0x7ff4040cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff409fe4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff404160e20 0x7ff4040cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48496/0 (socket says 172.21.15.158:48496) 2023-10-16T09:44:45.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.991+0000 7ff409fe4700 1 -- 172.21.15.158:0/2526853179 learned_addr learned my addr 172.21.15.158:0/2526853179 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:45.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff4097e3700 1 --2- 172.21.15.158:0/2526853179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca9f0 0x7ff40406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:45.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff409fe4700 1 -- 172.21.15.158:0/2526853179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca9f0 msgr2=0x7ff40406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:45.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff409fe4700 1 --2- 172.21.15.158:0/2526853179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca9f0 0x7ff40406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff409fe4700 1 -- 172.21.15.158:0/2526853179 >> v1:172.21.15.172:6789/0 conn(0x7ff40406ae10 legacy=0x7ff4040b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:45.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff409fe4700 1 -- 172.21.15.158:0/2526853179 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040cffd0 con 0x7ff404160e20 2023-10-16T09:44:45.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff4097e3700 1 --2- 172.21.15.158:0/2526853179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca9f0 0x7ff40406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:45.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff409fe4700 1 --2- 172.21.15.158:0/2526853179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff404160e20 0x7ff4040cf610 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40098f0 tx=0x7ff3f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3da3e0a2ad8d928 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:45.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff408fe2700 1 -- 172.21.15.158:0/2526853179 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4004230 con 0x7ff404160e20 2023-10-16T09:44:45.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff408fe2700 1 -- 172.21.15.158:0/2526853179 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4004390 con 0x7ff404160e20 2023-10-16T09:44:45.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.992+0000 7ff408fe2700 1 -- 172.21.15.158:0/2526853179 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40036a0 con 0x7ff404160e20 2023-10-16T09:44:45.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 -- 172.21.15.158:0/2526853179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff404160e20 msgr2=0x7ff4040cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:45.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 --2- 172.21.15.158:0/2526853179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff404160e20 0x7ff4040cf610 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40098f0 tx=0x7ff3f4009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:45.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 -- 172.21.15.158:0/2526853179 shutdown_connections 2023-10-16T09:44:45.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 --2- 172.21.15.158:0/2526853179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff404160e20 0x7ff4040cf610 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 --2- 172.21.15.158:0/2526853179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca9f0 0x7ff40406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:45.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 -- 172.21.15.158:0/2526853179 >> 172.21.15.158:0/2526853179 conn(0x7ff4040d03a0 msgr2=0x7ff4040d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:46.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 -- 172.21.15.158:0/2526853179 shutdown_connections 2023-10-16T09:44:46.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.993+0000 7ff40c248700 1 -- 172.21.15.158:0/2526853179 wait complete. 2023-10-16T09:44:46.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.994+0000 7ff40c248700 1 Processor -- start 2023-10-16T09:44:46.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.994+0000 7ff40c248700 1 -- start start 2023-10-16T09:44:46.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff40c248700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40406ae10 0x7ff4040c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff40c248700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4040ca9f0 0x7ff4040c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff40c248700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff404160e20 0x7ff4040b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff40c248700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4041f72d0 con 0x7ff4040ca9f0 2023-10-16T09:44:46.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff40c248700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4041f6420 con 0x7ff40406ae10 2023-10-16T09:44:46.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff40c248700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4041f66c0 con 0x7ff404160e20 2023-10-16T09:44:46.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff4097e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4040ca9f0 0x7ff4040c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff4097e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4040ca9f0 0x7ff4040c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48504/0 (socket says 172.21.15.158:48504) 2023-10-16T09:44:46.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff4097e3700 1 -- 172.21.15.158:0/2509553078 learned_addr learned my addr 172.21.15.158:0/2509553078 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:46.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff40a7e5700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff404160e20 0x7ff4040b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.995+0000 7ff409fe4700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40406ae10 0x7ff4040c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff4097e3700 1 -- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff404160e20 msgr2=0x7ff4040b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff4097e3700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff404160e20 0x7ff4040b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff4097e3700 1 -- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40406ae10 msgr2=0x7ff4040c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff4097e3700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40406ae10 0x7ff4040c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff4097e3700 1 -- 172.21.15.158:0/2509553078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4009550 con 0x7ff4040ca9f0 2023-10-16T09:44:46.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff4097e3700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4040ca9f0 0x7ff4040c3ed0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c750 tx=0x7ff40000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff3fa7fc700 1 -- 172.21.15.158:0/2509553078 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400011800 con 0x7ff4040ca9f0 2023-10-16T09:44:46.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff3fa7fc700 1 -- 172.21.15.158:0/2509553078 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff400011960 con 0x7ff4040ca9f0 2023-10-16T09:44:46.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4041f69a0 con 0x7ff4040ca9f0 2023-10-16T09:44:46.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff3fa7fc700 1 -- 172.21.15.158:0/2509553078 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400011cd0 con 0x7ff4040ca9f0 2023-10-16T09:44:46.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.996+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4041f6f60 con 0x7ff4040ca9f0 2023-10-16T09:44:46.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.997+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4040668b0 con 0x7ff4040ca9f0 2023-10-16T09:44:46.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.997+0000 7ff3fa7fc700 1 -- 172.21.15.158:0/2509553078 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff400021350 con 0x7ff4040ca9f0 2023-10-16T09:44:46.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.998+0000 7ff3fa7fc700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3e806af00 0x7ff3e806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.998+0000 7ff3fa7fc700 1 -- 172.21.15.158:0/2509553078 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff400089530 con 0x7ff4040ca9f0 2023-10-16T09:44:46.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.998+0000 7ff409fe4700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3e806af00 0x7ff3e806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:45.998+0000 7ff409fe4700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3e806af00 0x7ff3e806d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009130 tx=0x7ff3f4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.002+0000 7ff3fa7fc700 1 -- 172.21.15.158:0/2509553078 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff400054530 con 0x7ff4040ca9f0 2023-10-16T09:44:46.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.191+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4040b0090 con 0x7ff3e806af00 2023-10-16T09:44:46.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.199+0000 7ff3fa7fc700 1 -- 172.21.15.158:0/2509553078 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276888 (secure 0 0 0) 0x7ff3f40036a0 con 0x7ff3e806af00 2023-10-16T09:44:46.203 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:46.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3e806af00 msgr2=0x7ff3e806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3e806af00 0x7ff3e806d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009130 tx=0x7ff3f4008030 comp rx=0 tx=0).stop 2023-10-16T09:44:46.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4040ca9f0 msgr2=0x7ff4040c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4040ca9f0 0x7ff4040c3ed0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c750 tx=0x7ff40000ca60 comp rx=0 tx=0).stop 2023-10-16T09:44:46.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 shutdown_connections 2023-10-16T09:44:46.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7ff3e806af00 0x7ff3e806d3c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff404160e20 0x7ff4040b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40406ae10 0x7ff4040c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 --2- 172.21.15.158:0/2509553078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4040ca9f0 0x7ff4040c3ed0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 >> 172.21.15.158:0/2509553078 conn(0x7ff4040d03a0 msgr2=0x7ff4040cd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:46.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 shutdown_connections 2023-10-16T09:44:46.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.202+0000 7ff40c248700 1 -- 172.21.15.158:0/2509553078 wait complete. 2023-10-16T09:44:46.208 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:44:46.216 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2023-10-16T09:44:45.235635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220552,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754168,"statfs":{"total":773094113280,"available":772868268032,"internally_reserved":0,"allocated":4497408,"data_stored":2582784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552013"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690157+0000","last_change":"2023-10-16T09:44:39.690157+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690157+0000","last_undegraded":"2023-10-16T09:44:39.690157+0000","last_fullsized":"2023-10-16T09:44:39.690157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689987+0000","last_change":"2023-10-16T09:44:39.689987+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689987+0000","last_undegraded":"2023-10-16T09:44:39.689987+0000","last_fullsized":"2023-10-16T09:44:39.689987+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689036+0000","last_change":"2023-10-16T09:44:39.689036+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689036+0000","last_undegraded":"2023-10-16T09:44:39.689036+0000","last_fullsized":"2023-10-16T09:44:39.689036+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686535+0000","last_change":"2023-10-16T09:44:39.686535+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686535+0000","last_undegraded":"2023-10-16T09:44:39.686535+0000","last_fullsized":"2023-10-16T09:44:39.686535+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689557+0000","last_change":"2023-10-16T09:44:39.689557+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689557+0000","last_undegraded":"2023-10-16T09:44:39.689557+0000","last_fullsized":"2023-10-16T09:44:39.689557+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.688204+0000","last_change":"2023-10-16T09:44:39.688204+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.688204+0000","last_undegraded":"2023-10-16T09:44:39.688204+0000","last_fullsized":"2023-10-16T09:44:39.688204+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690430+0000","last_change":"2023-10-16T09:44:39.690430+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690430+0000","last_undegraded":"2023-10-16T09:44:39.690430+0000","last_fullsized":"2023-10-16T09:44:39.690430+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686522+0000","last_change":"2023-10-16T09:44:39.686522+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686522+0000","last_undegraded":"2023-10-16T09:44:39.686522+0000","last_fullsized":"2023-10-16T09:44:39.686522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690184+0000","last_change":"2023-10-16T09:44:39.690184+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690184+0000","last_undegraded":"2023-10-16T09:44:39.690184+0000","last_fullsized":"2023-10-16T09:44:39.690184+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689380+0000","last_change":"2023-10-16T09:44:39.689380+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689380+0000","last_undegraded":"2023-10-16T09:44:39.689380+0000","last_fullsized":"2023-10-16T09:44:39.689380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686325+0000","last_change":"2023-10-16T09:44:39.686325+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686325+0000","last_undegraded":"2023-10-16T09:44:39.686325+0000","last_fullsized":"2023-10-16T09:44:39.686325+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.688939+0000","last_change":"2023-10-16T09:44:39.688939+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.688939+0000","last_undegraded":"2023-10-16T09:44:39.688939+0000","last_fullsized":"2023-10-16T09:44:39.688939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690190+0000","last_change":"2023-10-16T09:44:39.690190+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690190+0000","last_undegraded":"2023-10-16T09:44:39.690190+0000","last_fullsized":"2023-10-16T09:44:39.690190+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687872+0000","last_change":"2023-10-16T09:44:39.687872+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687872+0000","last_undegraded":"2023-10-16T09:44:39.687872+0000","last_fullsized":"2023-10-16T09:44:39.687872+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690013+0000","last_change":"2023-10-16T09:44:39.690013+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690013+0000","last_undegraded":"2023-10-16T09:44:39.690013+0000","last_fullsized":"2023-10-16T09:44:39.690013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686521+0000","last_change":"2023-10-16T09:44:39.686521+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686521+0000","last_undegraded":"2023-10-16T09:44:39.686521+0000","last_fullsized":"2023-10-16T09:44:39.686521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692427+0000","last_change":"2023-10-16T09:44:39.692427+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692427+0000","last_undegraded":"2023-10-16T09:44:39.692427+0000","last_fullsized":"2023-10-16T09:44:39.692427+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689530+0000","last_change":"2023-10-16T09:44:39.689530+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689530+0000","last_undegraded":"2023-10-16T09:44:39.689530+0000","last_fullsized":"2023-10-16T09:44:39.689530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.693948+0000","last_change":"2023-10-16T09:44:39.693948+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.693948+0000","last_undegraded":"2023-10-16T09:44:39.693948+0000","last_fullsized":"2023-10-16T09:44:39.693948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692069+0000","last_change":"2023-10-16T09:44:39.692069+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692069+0000","last_undegraded":"2023-10-16T09:44:39.692069+0000","last_fullsized":"2023-10-16T09:44:39.692069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687912+0000","last_change":"2023-10-16T09:44:39.687912+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687912+0000","last_undegraded":"2023-10-16T09:44:39.687912+0000","last_fullsized":"2023-10-16T09:44:39.687912+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692055+0000","last_change":"2023-10-16T09:44:39.692055+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692055+0000","last_undegraded":"2023-10-16T09:44:39.692055+0000","last_fullsized":"2023-10-16T09:44:39.692055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689543+0000","last_change":"2023-10-16T09:44:39.689543+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689543+0000","last_undegraded":"2023-10-16T09:44:39.689543+0000","last_fullsized":"2023-10-16T09:44:39.689543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692248+0000","last_change":"2023-10-16T09:44:39.692248+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692248+0000","last_undegraded":"2023-10-16T09:44:39.692248+0000","last_fullsized":"2023-10-16T09:44:39.692248+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689081+0000","last_change":"2023-10-16T09:44:39.689081+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689081+0000","last_undegraded":"2023-10-16T09:44:39.689081+0000","last_fullsized":"2023-10-16T09:44:39.689081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689742+0000","last_change":"2023-10-16T09:44:39.689742+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689742+0000","last_undegraded":"2023-10-16T09:44:39.689742+0000","last_fullsized":"2023-10-16T09:44:39.689742+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.691440+0000","last_change":"2023-10-16T09:44:39.691440+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.691440+0000","last_undegraded":"2023-10-16T09:44:39.691440+0000","last_fullsized":"2023-10-16T09:44:39.691440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686535+0000","last_change":"2023-10-16T09:44:39.686535+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686535+0000","last_undegraded":"2023-10-16T09:44:39.686535+0000","last_fullsized":"2023-10-16T09:44:39.686535+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687880+0000","last_change":"2023-10-16T09:44:39.687880+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687880+0000","last_undegraded":"2023-10-16T09:44:39.687880+0000","last_fullsized":"2023-10-16T09:44:39.687880+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-16T09:44:34.383810+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:44:34.383810+0000","last_peered":"2023-10-16T09:44:34.383810+0000","last_clean":"2023-10-16T09:44:34.383810+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:44:34.383810+0000","last_undegraded":"2023-10-16T09:44:34.383810+0000","last_fullsized":"2023-10-16T09:44:34.383810+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":765624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":11.246,"5min":11.246,"15min":11.246},"last":0.56200000000000006}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.245},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.29799999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.73499999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.1760000000000002,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.72099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":765624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.376}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56399999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.629}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.39300000000000002}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.253}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.497}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.61499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.499},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.435}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.68100000000000005}]},{"osd":1,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.81499999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.84299999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.748},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.73099999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.79600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:44:46.222 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:44:46.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106d0d4700 1 Processor -- start 2023-10-16T09:44:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106d0d4700 1 -- start start 2023-10-16T09:44:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106d0d4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f106815e6f0 0x7f10680cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106d0d4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106815c5d0 con 0x7f10680b6e10 2023-10-16T09:44:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106d0d4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106815c8b0 con 0x7f10680cdee0 2023-10-16T09:44:46.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106d0d4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106815d2c0 con 0x7f106815e6f0 2023-10-16T09:44:46.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106759e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f106815e6f0 0x7f10680cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.376+0000 7f106659c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f10680b6e10 0x7f10681521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48688/0 (socket says 172.21.15.158:48688) 2023-10-16T09:44:46.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f106659c700 1 -- 172.21.15.158:0/2167608446 learned_addr learned my addr 172.21.15.158:0/2167608446 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f106759e700 1 -- 172.21.15.158:0/2167608446 >> v1:172.21.15.172:6789/0 conn(0x7f10680cdee0 legacy=0x7f106815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f106759e700 1 -- 172.21.15.158:0/2167608446 >> v1:172.21.15.158:6789/0 conn(0x7f10680b6e10 legacy=0x7f10681521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f1065d9b700 1 -- 172.21.15.158:0/2167608446 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255730736 0 0) 0x7f106815c5d0 con 0x7f10680b6e10 2023-10-16T09:44:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f106759e700 1 -- 172.21.15.158:0/2167608446 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106815cb90 con 0x7f106815e6f0 2023-10-16T09:44:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f106759e700 1 --2- 172.21.15.158:0/2167608446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106815e6f0 0x7f10680cba20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1060009d90 tx=0x7f106000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b047fcb9ce0db1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f1065d9b700 1 -- 172.21.15.158:0/2167608446 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106000e070 con 0x7f106815e6f0 2023-10-16T09:44:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f1065d9b700 1 -- 172.21.15.158:0/2167608446 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106000bbd0 con 0x7f106815e6f0 2023-10-16T09:44:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.377+0000 7f1065d9b700 1 -- 172.21.15.158:0/2167608446 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060012430 con 0x7f106815e6f0 2023-10-16T09:44:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.378+0000 7f106d0d4700 1 -- 172.21.15.158:0/2167608446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106815e6f0 msgr2=0x7f10680cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.378+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2167608446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106815e6f0 0x7f10680cba20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1060009d90 tx=0x7f106000b330 comp rx=0 tx=0).stop 2023-10-16T09:44:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.378+0000 7f106d0d4700 1 -- 172.21.15.158:0/2167608446 shutdown_connections 2023-10-16T09:44:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.378+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2167608446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106815e6f0 0x7f10680cba20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.378+0000 7f106d0d4700 1 -- 172.21.15.158:0/2167608446 >> 172.21.15.158:0/2167608446 conn(0x7f106806a8d0 msgr2=0x7f106806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:46.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.378+0000 7f106d0d4700 1 -- 172.21.15.158:0/2167608446 shutdown_connections 2023-10-16T09:44:46.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.378+0000 7f106d0d4700 1 -- 172.21.15.158:0/2167608446 wait complete. 2023-10-16T09:44:46.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.379+0000 7f106d0d4700 1 Processor -- start 2023-10-16T09:44:46.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.379+0000 7f106d0d4700 1 -- start start 2023-10-16T09:44:46.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106d0d4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10680b6e10 0x7f10681411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106d0d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10680cdee0 0x7f10680cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106d0d4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106815e6f0 0x7f1068151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106d0d4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680d1350 con 0x7f10680cdee0 2023-10-16T09:44:46.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106d0d4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680d0470 con 0x7f106815e6f0 2023-10-16T09:44:46.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106d0d4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10680d06d0 con 0x7f10680b6e10 2023-10-16T09:44:46.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10680cdee0 0x7f10680cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10680cdee0 0x7f10680cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48518/0 (socket says 172.21.15.158:48518) 2023-10-16T09:44:46.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 -- 172.21.15.158:0/2942271131 learned_addr learned my addr 172.21.15.158:0/2942271131 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:46.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106759e700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106815e6f0 0x7f1068151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f1066d9d700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10680b6e10 0x7f10681411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 -- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10680b6e10 msgr2=0x7f10681411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10680b6e10 0x7f10681411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 -- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106815e6f0 msgr2=0x7f1068151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106815e6f0 0x7f1068151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.380+0000 7f106659c700 1 -- 172.21.15.158:0/2942271131 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060009a70 con 0x7f10680cdee0 2023-10-16T09:44:46.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.381+0000 7f106659c700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10680cdee0 0x7f10680cff30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f105800c520 tx=0x7f105800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.381+0000 7f104f7fe700 1 -- 172.21.15.158:0/2942271131 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058004c80 con 0x7f10680cdee0 2023-10-16T09:44:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.381+0000 7f104f7fe700 1 -- 172.21.15.158:0/2942271131 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1058004e00 con 0x7f10680cdee0 2023-10-16T09:44:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.381+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10680d09b0 con 0x7f10680cdee0 2023-10-16T09:44:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.381+0000 7f104f7fe700 1 -- 172.21.15.158:0/2942271131 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058007310 con 0x7f10680cdee0 2023-10-16T09:44:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.381+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10680d0f70 con 0x7f10680cdee0 2023-10-16T09:44:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.382+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10680681b0 con 0x7f10680cdee0 2023-10-16T09:44:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.382+0000 7f104f7fe700 1 -- 172.21.15.158:0/2942271131 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1058007470 con 0x7f10680cdee0 2023-10-16T09:44:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.382+0000 7f104f7fe700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f105406af00 0x7f105406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.383+0000 7f104f7fe700 1 -- 172.21.15.158:0/2942271131 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f10580892f0 con 0x7f10680cdee0 2023-10-16T09:44:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.383+0000 7f1066d9d700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f105406af00 0x7f105406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.383+0000 7f1066d9d700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f105406af00 0x7f105406d3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f105c007fd0 tx=0x7f105c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.387+0000 7f104f7fe700 1 -- 172.21.15.158:0/2942271131 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1058054260 con 0x7f10680cdee0 2023-10-16T09:44:46.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.573+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f106815cee0 con 0x7f105406af00 2023-10-16T09:44:46.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.581+0000 7f104f7fe700 1 -- 172.21.15.158:0/2942271131 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276888 (secure 0 0 0) 0x7f105c001560 con 0x7f105406af00 2023-10-16T09:44:46.589 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:46.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f105406af00 msgr2=0x7f105406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f105406af00 0x7f105406d3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f105c007fd0 tx=0x7f105c009040 comp rx=0 tx=0).stop 2023-10-16T09:44:46.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10680cdee0 msgr2=0x7f10680cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10680cdee0 0x7f10680cff30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f105800c520 tx=0x7f105800c830 comp rx=0 tx=0).stop 2023-10-16T09:44:46.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 shutdown_connections 2023-10-16T09:44:46.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106815e6f0 0x7f1068151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f105406af00 0x7f105406d3c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10680b6e10 0x7f10681411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 --2- 172.21.15.158:0/2942271131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10680cdee0 0x7f10680cff30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 >> 172.21.15.158:0/2942271131 conn(0x7f106806a8d0 msgr2=0x7f106815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:46.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 shutdown_connections 2023-10-16T09:44:46.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.584+0000 7f106d0d4700 1 -- 172.21.15.158:0/2942271131 wait complete. 2023-10-16T09:44:46.594 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:44:46.597 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2023-10-16T09:44:45.235635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220552,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754168,"statfs":{"total":773094113280,"available":772868268032,"internally_reserved":0,"allocated":4497408,"data_stored":2582784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552013"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690157+0000","last_change":"2023-10-16T09:44:39.690157+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690157+0000","last_undegraded":"2023-10-16T09:44:39.690157+0000","last_fullsized":"2023-10-16T09:44:39.690157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689987+0000","last_change":"2023-10-16T09:44:39.689987+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689987+0000","last_undegraded":"2023-10-16T09:44:39.689987+0000","last_fullsized":"2023-10-16T09:44:39.689987+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689036+0000","last_change":"2023-10-16T09:44:39.689036+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689036+0000","last_undegraded":"2023-10-16T09:44:39.689036+0000","last_fullsized":"2023-10-16T09:44:39.689036+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686535+0000","last_change":"2023-10-16T09:44:39.686535+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686535+0000","last_undegraded":"2023-10-16T09:44:39.686535+0000","last_fullsized":"2023-10-16T09:44:39.686535+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689557+0000","last_change":"2023-10-16T09:44:39.689557+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689557+0000","last_undegraded":"2023-10-16T09:44:39.689557+0000","last_fullsized":"2023-10-16T09:44:39.689557+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.688204+0000","last_change":"2023-10-16T09:44:39.688204+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.688204+0000","last_undegraded":"2023-10-16T09:44:39.688204+0000","last_fullsized":"2023-10-16T09:44:39.688204+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690430+0000","last_change":"2023-10-16T09:44:39.690430+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690430+0000","last_undegraded":"2023-10-16T09:44:39.690430+0000","last_fullsized":"2023-10-16T09:44:39.690430+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686522+0000","last_change":"2023-10-16T09:44:39.686522+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686522+0000","last_undegraded":"2023-10-16T09:44:39.686522+0000","last_fullsized":"2023-10-16T09:44:39.686522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690184+0000","last_change":"2023-10-16T09:44:39.690184+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690184+0000","last_undegraded":"2023-10-16T09:44:39.690184+0000","last_fullsized":"2023-10-16T09:44:39.690184+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689380+0000","last_change":"2023-10-16T09:44:39.689380+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689380+0000","last_undegraded":"2023-10-16T09:44:39.689380+0000","last_fullsized":"2023-10-16T09:44:39.689380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686325+0000","last_change":"2023-10-16T09:44:39.686325+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686325+0000","last_undegraded":"2023-10-16T09:44:39.686325+0000","last_fullsized":"2023-10-16T09:44:39.686325+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.688939+0000","last_change":"2023-10-16T09:44:39.688939+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.688939+0000","last_undegraded":"2023-10-16T09:44:39.688939+0000","last_fullsized":"2023-10-16T09:44:39.688939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690190+0000","last_change":"2023-10-16T09:44:39.690190+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690190+0000","last_undegraded":"2023-10-16T09:44:39.690190+0000","last_fullsized":"2023-10-16T09:44:39.690190+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687872+0000","last_change":"2023-10-16T09:44:39.687872+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687872+0000","last_undegraded":"2023-10-16T09:44:39.687872+0000","last_fullsized":"2023-10-16T09:44:39.687872+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.690013+0000","last_change":"2023-10-16T09:44:39.690013+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.690013+0000","last_undegraded":"2023-10-16T09:44:39.690013+0000","last_fullsized":"2023-10-16T09:44:39.690013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686521+0000","last_change":"2023-10-16T09:44:39.686521+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686521+0000","last_undegraded":"2023-10-16T09:44:39.686521+0000","last_fullsized":"2023-10-16T09:44:39.686521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692427+0000","last_change":"2023-10-16T09:44:39.692427+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692427+0000","last_undegraded":"2023-10-16T09:44:39.692427+0000","last_fullsized":"2023-10-16T09:44:39.692427+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689530+0000","last_change":"2023-10-16T09:44:39.689530+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689530+0000","last_undegraded":"2023-10-16T09:44:39.689530+0000","last_fullsized":"2023-10-16T09:44:39.689530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.693948+0000","last_change":"2023-10-16T09:44:39.693948+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.693948+0000","last_undegraded":"2023-10-16T09:44:39.693948+0000","last_fullsized":"2023-10-16T09:44:39.693948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692069+0000","last_change":"2023-10-16T09:44:39.692069+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692069+0000","last_undegraded":"2023-10-16T09:44:39.692069+0000","last_fullsized":"2023-10-16T09:44:39.692069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687912+0000","last_change":"2023-10-16T09:44:39.687912+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687912+0000","last_undegraded":"2023-10-16T09:44:39.687912+0000","last_fullsized":"2023-10-16T09:44:39.687912+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692055+0000","last_change":"2023-10-16T09:44:39.692055+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692055+0000","last_undegraded":"2023-10-16T09:44:39.692055+0000","last_fullsized":"2023-10-16T09:44:39.692055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689543+0000","last_change":"2023-10-16T09:44:39.689543+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689543+0000","last_undegraded":"2023-10-16T09:44:39.689543+0000","last_fullsized":"2023-10-16T09:44:39.689543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.692248+0000","last_change":"2023-10-16T09:44:39.692248+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.692248+0000","last_undegraded":"2023-10-16T09:44:39.692248+0000","last_fullsized":"2023-10-16T09:44:39.692248+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689081+0000","last_change":"2023-10-16T09:44:39.689081+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689081+0000","last_undegraded":"2023-10-16T09:44:39.689081+0000","last_fullsized":"2023-10-16T09:44:39.689081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.689742+0000","last_change":"2023-10-16T09:44:39.689742+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.689742+0000","last_undegraded":"2023-10-16T09:44:39.689742+0000","last_fullsized":"2023-10-16T09:44:39.689742+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.691440+0000","last_change":"2023-10-16T09:44:39.691440+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.691440+0000","last_undegraded":"2023-10-16T09:44:39.691440+0000","last_fullsized":"2023-10-16T09:44:39.691440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686535+0000","last_change":"2023-10-16T09:44:39.686535+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686535+0000","last_undegraded":"2023-10-16T09:44:39.686535+0000","last_fullsized":"2023-10-16T09:44:39.686535+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.686324+0000","last_change":"2023-10-16T09:44:39.686324+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.686324+0000","last_undegraded":"2023-10-16T09:44:39.686324+0000","last_fullsized":"2023-10-16T09:44:39.686324+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T09:44:40.683025+0000","last_change":"2023-10-16T09:44:40.683025+0000","last_active":"2023-10-16T09:44:40.683025+0000","last_peered":"2023-10-16T09:44:40.683025+0000","last_clean":"2023-10-16T09:44:40.683025+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:40.683025+0000","last_undegraded":"2023-10-16T09:44:40.683025+0000","last_fullsized":"2023-10-16T09:44:40.683025+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-10-16T09:44:39.687880+0000","last_change":"2023-10-16T09:44:39.687880+0000","last_active":"2023-10-16T09:44:39.678349+0000","last_peered":"2023-10-16T09:44:39.678349+0000","last_clean":"2023-10-16T09:44:39.678349+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T09:44:39.687880+0000","last_undegraded":"2023-10-16T09:44:39.687880+0000","last_fullsized":"2023-10-16T09:44:39.687880+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-16T09:44:34.383810+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:44:34.383810+0000","last_peered":"2023-10-16T09:44:34.383810+0000","last_clean":"2023-10-16T09:44:34.383810+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:44:34.383810+0000","last_undegraded":"2023-10-16T09:44:34.383810+0000","last_fullsized":"2023-10-16T09:44:34.383810+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":765624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":11.246,"5min":11.246,"15min":11.246},"last":0.56200000000000006}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.245},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.29799999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.73499999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.1760000000000002,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.72099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":765624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.376}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56399999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.629}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.39300000000000002}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.253}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.497}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.61499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.499},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.435}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":175256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.68100000000000005}]},{"osd":1,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.81499999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.84299999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.748},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.73099999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.79600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:44:46.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2023-10-16T09:44:46.604 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.605 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2023-10-16T09:44:46.605 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.606 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2023-10-16T09:44:46.606 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.606 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2023-10-16T09:44:46.607 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.607 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2023-10-16T09:44:46.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.608 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2023-10-16T09:44:46.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.609 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2023-10-16T09:44:46.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.610 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2023-10-16T09:44:46.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.611 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2023-10-16T09:44:46.611 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.612 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2023-10-16T09:44:46.612 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2023-10-16T09:44:46.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2023-10-16T09:44:46.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2023-10-16T09:44:46.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.615 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2023-10-16T09:44:46.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.616 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2023-10-16T09:44:46.616 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.617 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2023-10-16T09:44:46.617 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.617 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2023-10-16T09:44:46.618 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.618 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2023-10-16T09:44:46.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2023-10-16T09:44:46.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2023-10-16T09:44:46.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2023-10-16T09:44:46.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2023-10-16T09:44:46.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2023-10-16T09:44:46.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2023-10-16T09:44:46.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2023-10-16T09:44:46.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2023-10-16T09:44:46.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2023-10-16T09:44:46.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2023-10-16T09:44:46.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.627 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2023-10-16T09:44:46.627 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2023-10-16T09:44:46.628 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2023-10-16T09:44:46.629 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.629 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2023-10-16T09:44:46.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.630 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-10-16T09:44:46.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.631 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2023-10-16T09:44:46.631 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.690157+0000', 'last_change': '2023-10-16T09:44:39.690157+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.690157+0000', 'last_undegraded': '2023-10-16T09:44:39.690157+0000', 'last_fullsized': '2023-10-16T09:44:39.690157+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-16T09:44:46.632 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2023-10-16T09:44:46.632 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.632 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2023-10-16T09:44:46.633 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689987+0000', 'last_change': '2023-10-16T09:44:39.689987+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689987+0000', 'last_undegraded': '2023-10-16T09:44:39.689987+0000', 'last_fullsized': '2023-10-16T09:44:39.689987+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-16T09:44:46.633 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-10-16T09:44:46.634 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.634 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2023-10-16T09:44:46.635 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689036+0000', 'last_change': '2023-10-16T09:44:39.689036+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689036+0000', 'last_undegraded': '2023-10-16T09:44:39.689036+0000', 'last_fullsized': '2023-10-16T09:44:39.689036+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-16T09:44:46.635 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-10-16T09:44:46.635 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.636 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2023-10-16T09:44:46.636 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686535+0000', 'last_change': '2023-10-16T09:44:39.686535+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686535+0000', 'last_undegraded': '2023-10-16T09:44:39.686535+0000', 'last_fullsized': '2023-10-16T09:44:39.686535+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T09:44:46.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2023-10-16T09:44:46.637 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2023-10-16T09:44:46.638 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689557+0000', 'last_change': '2023-10-16T09:44:39.689557+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689557+0000', 'last_undegraded': '2023-10-16T09:44:39.689557+0000', 'last_fullsized': '2023-10-16T09:44:39.689557+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-16T09:44:46.638 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-10-16T09:44:46.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2023-10-16T09:44:46.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.688204+0000', 'last_change': '2023-10-16T09:44:39.688204+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.688204+0000', 'last_undegraded': '2023-10-16T09:44:39.688204+0000', 'last_fullsized': '2023-10-16T09:44:39.688204+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-16T09:44:46.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-10-16T09:44:46.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2023-10-16T09:44:46.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.690430+0000', 'last_change': '2023-10-16T09:44:39.690430+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.690430+0000', 'last_undegraded': '2023-10-16T09:44:39.690430+0000', 'last_fullsized': '2023-10-16T09:44:39.690430+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-16T09:44:46.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-10-16T09:44:46.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2023-10-16T09:44:46.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686522+0000', 'last_change': '2023-10-16T09:44:39.686522+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686522+0000', 'last_undegraded': '2023-10-16T09:44:39.686522+0000', 'last_fullsized': '2023-10-16T09:44:39.686522+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T09:44:46.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2023-10-16T09:44:46.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2023-10-16T09:44:46.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686324+0000', 'last_change': '2023-10-16T09:44:39.686324+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686324+0000', 'last_undegraded': '2023-10-16T09:44:39.686324+0000', 'last_fullsized': '2023-10-16T09:44:39.686324+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-16T09:44:46.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-10-16T09:44:46.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2023-10-16T09:44:46.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.690184+0000', 'last_change': '2023-10-16T09:44:39.690184+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.690184+0000', 'last_undegraded': '2023-10-16T09:44:39.690184+0000', 'last_fullsized': '2023-10-16T09:44:39.690184+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-16T09:44:46.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2023-10-16T09:44:46.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2023-10-16T09:44:46.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689380+0000', 'last_change': '2023-10-16T09:44:39.689380+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689380+0000', 'last_undegraded': '2023-10-16T09:44:39.689380+0000', 'last_fullsized': '2023-10-16T09:44:39.689380+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-16T09:44:46.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-10-16T09:44:46.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2023-10-16T09:44:46.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686325+0000', 'last_change': '2023-10-16T09:44:39.686325+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686325+0000', 'last_undegraded': '2023-10-16T09:44:39.686325+0000', 'last_fullsized': '2023-10-16T09:44:39.686325+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-16T09:44:46.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2023-10-16T09:44:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2023-10-16T09:44:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686324+0000', 'last_change': '2023-10-16T09:44:39.686324+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686324+0000', 'last_undegraded': '2023-10-16T09:44:39.686324+0000', 'last_fullsized': '2023-10-16T09:44:39.686324+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-16T09:44:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-10-16T09:44:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2023-10-16T09:44:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.688939+0000', 'last_change': '2023-10-16T09:44:39.688939+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.688939+0000', 'last_undegraded': '2023-10-16T09:44:39.688939+0000', 'last_fullsized': '2023-10-16T09:44:39.688939+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-16T09:44:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-10-16T09:44:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2023-10-16T09:44:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.690190+0000', 'last_change': '2023-10-16T09:44:39.690190+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.690190+0000', 'last_undegraded': '2023-10-16T09:44:39.690190+0000', 'last_fullsized': '2023-10-16T09:44:39.690190+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-16T09:44:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-10-16T09:44:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2023-10-16T09:44:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.687872+0000', 'last_change': '2023-10-16T09:44:39.687872+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.687872+0000', 'last_undegraded': '2023-10-16T09:44:39.687872+0000', 'last_fullsized': '2023-10-16T09:44:39.687872+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-16T09:44:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-10-16T09:44:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2023-10-16T09:44:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.690013+0000', 'last_change': '2023-10-16T09:44:39.690013+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.690013+0000', 'last_undegraded': '2023-10-16T09:44:39.690013+0000', 'last_fullsized': '2023-10-16T09:44:39.690013+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-16T09:44:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-10-16T09:44:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2023-10-16T09:44:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686521+0000', 'last_change': '2023-10-16T09:44:39.686521+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686521+0000', 'last_undegraded': '2023-10-16T09:44:39.686521+0000', 'last_fullsized': '2023-10-16T09:44:39.686521+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T09:44:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-10-16T09:44:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2023-10-16T09:44:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.692427+0000', 'last_change': '2023-10-16T09:44:39.692427+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.692427+0000', 'last_undegraded': '2023-10-16T09:44:39.692427+0000', 'last_fullsized': '2023-10-16T09:44:39.692427+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-16T09:44:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2023-10-16T09:44:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2023-10-16T09:44:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689530+0000', 'last_change': '2023-10-16T09:44:39.689530+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689530+0000', 'last_undegraded': '2023-10-16T09:44:39.689530+0000', 'last_fullsized': '2023-10-16T09:44:39.689530+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-16T09:44:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-10-16T09:44:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2023-10-16T09:44:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.693948+0000', 'last_change': '2023-10-16T09:44:39.693948+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.693948+0000', 'last_undegraded': '2023-10-16T09:44:39.693948+0000', 'last_fullsized': '2023-10-16T09:44:39.693948+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-16T09:44:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2023-10-16T09:44:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2023-10-16T09:44:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.692069+0000', 'last_change': '2023-10-16T09:44:39.692069+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.692069+0000', 'last_undegraded': '2023-10-16T09:44:39.692069+0000', 'last_fullsized': '2023-10-16T09:44:39.692069+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-16T09:44:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-10-16T09:44:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.667 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2023-10-16T09:44:46.667 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.687912+0000', 'last_change': '2023-10-16T09:44:39.687912+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.687912+0000', 'last_undegraded': '2023-10-16T09:44:39.687912+0000', 'last_fullsized': '2023-10-16T09:44:39.687912+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-16T09:44:46.668 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-10-16T09:44:46.668 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.669 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2023-10-16T09:44:46.669 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.692055+0000', 'last_change': '2023-10-16T09:44:39.692055+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.692055+0000', 'last_undegraded': '2023-10-16T09:44:39.692055+0000', 'last_fullsized': '2023-10-16T09:44:39.692055+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-16T09:44:46.669 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2023-10-16T09:44:46.670 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.670 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2023-10-16T09:44:46.671 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689543+0000', 'last_change': '2023-10-16T09:44:39.689543+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689543+0000', 'last_undegraded': '2023-10-16T09:44:39.689543+0000', 'last_fullsized': '2023-10-16T09:44:39.689543+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-16T09:44:46.671 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-10-16T09:44:46.671 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.672 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2023-10-16T09:44:46.672 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.692248+0000', 'last_change': '2023-10-16T09:44:39.692248+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.692248+0000', 'last_undegraded': '2023-10-16T09:44:39.692248+0000', 'last_fullsized': '2023-10-16T09:44:39.692248+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-16T09:44:46.673 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-10-16T09:44:46.673 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.674 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2023-10-16T09:44:46.674 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689081+0000', 'last_change': '2023-10-16T09:44:39.689081+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689081+0000', 'last_undegraded': '2023-10-16T09:44:39.689081+0000', 'last_fullsized': '2023-10-16T09:44:39.689081+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-16T09:44:46.675 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2023-10-16T09:44:46.675 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.676 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2023-10-16T09:44:46.676 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.689742+0000', 'last_change': '2023-10-16T09:44:39.689742+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.689742+0000', 'last_undegraded': '2023-10-16T09:44:39.689742+0000', 'last_fullsized': '2023-10-16T09:44:39.689742+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-16T09:44:46.676 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2023-10-16T09:44:46.677 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.677 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2023-10-16T09:44:46.678 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.691440+0000', 'last_change': '2023-10-16T09:44:39.691440+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.691440+0000', 'last_undegraded': '2023-10-16T09:44:39.691440+0000', 'last_fullsized': '2023-10-16T09:44:39.691440+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-16T09:44:46.679 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-10-16T09:44:46.679 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.680 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2023-10-16T09:44:46.681 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686535+0000', 'last_change': '2023-10-16T09:44:39.686535+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686535+0000', 'last_undegraded': '2023-10-16T09:44:39.686535+0000', 'last_fullsized': '2023-10-16T09:44:39.686535+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T09:44:46.681 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-10-16T09:44:46.682 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.682 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2023-10-16T09:44:46.683 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.686324+0000', 'last_change': '2023-10-16T09:44:39.686324+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.686324+0000', 'last_undegraded': '2023-10-16T09:44:39.686324+0000', 'last_fullsized': '2023-10-16T09:44:39.686324+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-16T09:44:46.683 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-10-16T09:44:46.684 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T09:44:40.683025+0000', 'last_change': '2023-10-16T09:44:40.683025+0000', 'last_active': '2023-10-16T09:44:40.683025+0000', 'last_peered': '2023-10-16T09:44:40.683025+0000', 'last_clean': '2023-10-16T09:44:40.683025+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:40.683025+0000', 'last_undegraded': '2023-10-16T09:44:40.683025+0000', 'last_fullsized': '2023-10-16T09:44:40.683025+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:40.683025+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T09:44:46.684 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2023-10-16T09:44:46.684 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-10-16T09:44:39.687880+0000', 'last_change': '2023-10-16T09:44:39.687880+0000', 'last_active': '2023-10-16T09:44:39.678349+0000', 'last_peered': '2023-10-16T09:44:39.678349+0000', 'last_clean': '2023-10-16T09:44:39.678349+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T09:44:39.687880+0000', 'last_undegraded': '2023-10-16T09:44:39.687880+0000', 'last_fullsized': '2023-10-16T09:44:39.687880+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'last_clean_scrub_stamp': '2023-10-16T09:44:39.678349+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-16T09:44:46.685 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-16T09:44:46.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.804+0000 7f7ab3dd5700 1 Processor -- start 2023-10-16T09:44:46.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.804+0000 7f7ab3dd5700 1 -- start start 2023-10-16T09:44:46.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab3dd5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7aac1433b0 0x7f7aac06bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab3dd5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7aac0c5920 0x7f7aac071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab3dd5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac06b780 con 0x7f7aac0711a0 2023-10-16T09:44:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab3dd5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac071b90 con 0x7f7aac0c5920 2023-10-16T09:44:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab3dd5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac0c5d70 con 0x7f7aac1433b0 2023-10-16T09:44:46.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab2372700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7aac0c5920 0x7f7aac071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab2372700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7aac0c5920 0x7f7aac071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48528/0 (socket says 172.21.15.158:48528) 2023-10-16T09:44:46.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab2372700 1 -- 172.21.15.158:0/1745731544 learned_addr learned my addr 172.21.15.158:0/1745731544 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:46.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab2372700 1 -- 172.21.15.158:0/1745731544 >> v2:172.21.15.172:3300/0 conn(0x7f7aac1433b0 msgr2=0x7f7aac06bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:46.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab2372700 1 --2- 172.21.15.158:0/1745731544 >> v2:172.21.15.172:3300/0 conn(0x7f7aac1433b0 0x7f7aac06bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab2372700 1 -- 172.21.15.158:0/1745731544 >> v1:172.21.15.172:6789/0 conn(0x7f7aac0711a0 legacy=0x7f7aac0c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:46.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.805+0000 7f7ab2372700 1 -- 172.21.15.158:0/1745731544 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac06ab70 con 0x7f7aac0c5920 2023-10-16T09:44:46.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab2372700 1 --2- 172.21.15.158:0/1745731544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0c5920 0x7f7aac071650 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009e50 tx=0x7f7a9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1246d12d8817ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab0b6f700 1 -- 172.21.15.158:0/1745731544 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c003a20 con 0x7f7aac0c5920 2023-10-16T09:44:46.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab0b6f700 1 -- 172.21.15.158:0/1745731544 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c003b80 con 0x7f7aac0c5920 2023-10-16T09:44:46.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab0b6f700 1 -- 172.21.15.158:0/1745731544 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c00b940 con 0x7f7aac0c5920 2023-10-16T09:44:46.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/1745731544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0c5920 msgr2=0x7f7aac071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/1745731544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0c5920 0x7f7aac071650 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009e50 tx=0x7f7a9c00b330 comp rx=0 tx=0).stop 2023-10-16T09:44:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/1745731544 shutdown_connections 2023-10-16T09:44:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/1745731544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0c5920 0x7f7aac071650 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/1745731544 >> v2:172.21.15.172:3300/0 conn(0x7f7aac1433b0 0x7f7aac06bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/1745731544 >> 172.21.15.158:0/1745731544 conn(0x7f7aac0cfaa0 msgr2=0x7f7aac0cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.806+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/1745731544 shutdown_connections 2023-10-16T09:44:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.807+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/1745731544 wait complete. 2023-10-16T09:44:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 Processor -- start 2023-10-16T09:44:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 -- start start 2023-10-16T09:44:46.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0711a0 0x7f7aac157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aac0c5920 0x7f7aac1f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aac1433b0 0x7f7aac1f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0c8a80 con 0x7f7aac0711a0 2023-10-16T09:44:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac158080 con 0x7f7aac0c5920 2023-10-16T09:44:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab3dd5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac1e4180 con 0x7f7aac1433b0 2023-10-16T09:44:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.808+0000 7f7ab1b71700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0711a0 0x7f7aac157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0711a0 0x7f7aac157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48530/0 (socket says 172.21.15.158:48530) 2023-10-16T09:44:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 -- 172.21.15.158:0/3787210502 learned_addr learned my addr 172.21.15.158:0/3787210502 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 -- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aac1433b0 msgr2=0x7f7aac1f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab2372700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aac1433b0 0x7f7aac1f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aac1433b0 0x7f7aac1f2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1370700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aac0c5920 0x7f7aac1f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 -- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aac0c5920 msgr2=0x7f7aac1f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:46.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aac0c5920 0x7f7aac1f2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:46.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 -- 172.21.15.158:0/3787210502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c009b30 con 0x7f7aac0711a0 2023-10-16T09:44:46.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab1b71700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0711a0 0x7f7aac157b40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b530 tx=0x7f7aa400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7aaa7fc700 1 -- 172.21.15.158:0/3787210502 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4011830 con 0x7f7aac0711a0 2023-10-16T09:44:46.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7aaa7fc700 1 -- 172.21.15.158:0/3787210502 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa40119b0 con 0x7f7aac0711a0 2023-10-16T09:44:46.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac0c7e00 con 0x7f7aac0711a0 2023-10-16T09:44:46.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7aaa7fc700 1 -- 172.21.15.158:0/3787210502 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4011d60 con 0x7f7aac0711a0 2023-10-16T09:44:46.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.809+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac0c83c0 con 0x7f7aac0711a0 2023-10-16T09:44:46.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.810+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac0c8bc0 con 0x7f7aac0711a0 2023-10-16T09:44:46.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.811+0000 7f7aaa7fc700 1 -- 172.21.15.158:0/3787210502 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7aa401a400 con 0x7f7aac0711a0 2023-10-16T09:44:46.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.811+0000 7f7aaa7fc700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7a9806af00 0x7f7a9806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:46.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.811+0000 7f7aaa7fc700 1 -- 172.21.15.158:0/3787210502 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7aa4089750 con 0x7f7aac0711a0 2023-10-16T09:44:46.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.811+0000 7f7ab1370700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7a9806af00 0x7f7a9806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:46.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.812+0000 7f7ab1370700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7a9806af00 0x7f7a9806d3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7aac1df390 tx=0x7f7aa0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:46.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:46.816+0000 7f7aaa7fc700 1 -- 172.21.15.158:0/3787210502 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7aac0c8bc0 con 0x7f7aac0711a0 2023-10-16T09:44:47.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.089+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7aac0c8bc0 con 0x7f7aac0711a0 2023-10-16T09:44:47.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.090+0000 7f7aaa7fc700 1 -- 172.21.15.158:0/3787210502 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1109 (secure 0 0 0) 0x7f7aac0c8bc0 con 0x7f7aac0711a0 2023-10-16T09:44:47.092 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:47.092 INFO:teuthology.orchestra.run.smithi158.stdout:{"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":102,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1697449394,"num_in_osds":8,"osd_in_since":1697449386,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225845248,"bytes_avail":772868268032,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"b","status":"up:active","gid":34760}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2023-10-16T09:44:27.232710+0000","services":{}},"progress_events":{}} 2023-10-16T09:44:47.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7a9806af00 msgr2=0x7f7a9806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7a9806af00 0x7f7a9806d3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7aac1df390 tx=0x7f7aa0008040 comp rx=0 tx=0).stop 2023-10-16T09:44:47.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0711a0 msgr2=0x7f7aac157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0711a0 0x7f7aac157b40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b530 tx=0x7f7aa400b8f0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 shutdown_connections 2023-10-16T09:44:47.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f7a9806af00 0x7f7a9806d3c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aac1433b0 0x7f7aac1f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aac0711a0 0x7f7aac157b40 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 --2- 172.21.15.158:0/3787210502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aac0c5920 0x7f7aac1f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 >> 172.21.15.158:0/3787210502 conn(0x7f7aac0cfaa0 msgr2=0x7f7aac137e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:47.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 shutdown_connections 2023-10-16T09:44:47.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.092+0000 7f7ab3dd5700 1 -- 172.21.15.158:0/3787210502 wait complete. 2023-10-16T09:44:47.106 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-16T09:44:47.106 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:44:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.224+0000 7f65d91dd700 1 Processor -- start 2023-10-16T09:44:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.224+0000 7f65d91dd700 1 -- start start 2023-10-16T09:44:47.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d91dd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65d415b4c0 0x7f65d40cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:47.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d91dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d40ce440 0x7f65d415f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:47.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d91dd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d40ce8d0 con 0x7f65d40cdf90 2023-10-16T09:44:47.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d91dd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40caa90 con 0x7f65d415b4c0 2023-10-16T09:44:47.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d91dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d406a970 con 0x7f65d40ce440 2023-10-16T09:44:47.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d40ce440 0x7f65d415f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d2d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65d415b4c0 0x7f65d40cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d40ce440 0x7f65d415f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:59374/0 (socket says 172.21.15.158:59374) 2023-10-16T09:44:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d2d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65d415b4c0 0x7f65d40cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:49718/0 (socket says 172.21.15.158:49718) 2023-10-16T09:44:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d259c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f65d40cdf90 0x7f65d415f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:52622/0 (socket says 172.21.15.158:52622) 2023-10-16T09:44:47.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.225+0000 7f65d359e700 1 -- 172.21.15.158:0/317331751 learned_addr learned my addr 172.21.15.158:0/317331751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d359e700 1 -- 172.21.15.158:0/317331751 >> v2:172.21.15.172:3300/0 conn(0x7f65d415b4c0 msgr2=0x7f65d40cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d359e700 1 --2- 172.21.15.158:0/317331751 >> v2:172.21.15.172:3300/0 conn(0x7f65d415b4c0 0x7f65d40cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d359e700 1 -- 172.21.15.158:0/317331751 >> v1:172.21.15.172:6789/0 conn(0x7f65d40cdf90 legacy=0x7f65d415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d359e700 1 -- 172.21.15.158:0/317331751 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40cac70 con 0x7f65d40ce440 2023-10-16T09:44:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d2d9d700 1 --2- 172.21.15.158:0/317331751 >> v2:172.21.15.172:3300/0 conn(0x7f65d415b4c0 0x7f65d40cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:47.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d359e700 1 --2- 172.21.15.158:0/317331751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d40ce440 0x7f65d415f840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009e50 tx=0x7f65cc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=100d3013636a3241 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d1d9b700 1 -- 172.21.15.158:0/317331751 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00e070 con 0x7f65d40ce440 2023-10-16T09:44:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d1d9b700 1 -- 172.21.15.158:0/317331751 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc00b910 con 0x7f65d40ce440 2023-10-16T09:44:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d1d9b700 1 -- 172.21.15.158:0/317331751 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00bcc0 con 0x7f65d40ce440 2023-10-16T09:44:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d91dd700 1 -- 172.21.15.158:0/317331751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d40ce440 msgr2=0x7f65d415f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d91dd700 1 --2- 172.21.15.158:0/317331751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d40ce440 0x7f65d415f840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009e50 tx=0x7f65cc00b330 comp rx=0 tx=0).stop 2023-10-16T09:44:47.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.226+0000 7f65d91dd700 1 -- 172.21.15.158:0/317331751 shutdown_connections 2023-10-16T09:44:47.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.227+0000 7f65d91dd700 1 --2- 172.21.15.158:0/317331751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d40ce440 0x7f65d415f840 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.227+0000 7f65d91dd700 1 --2- 172.21.15.158:0/317331751 >> v2:172.21.15.172:3300/0 conn(0x7f65d415b4c0 0x7f65d40cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.227+0000 7f65d91dd700 1 -- 172.21.15.158:0/317331751 >> 172.21.15.158:0/317331751 conn(0x7f65d40cc4b0 msgr2=0x7f65d40cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:47.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.227+0000 7f65d91dd700 1 -- 172.21.15.158:0/317331751 shutdown_connections 2023-10-16T09:44:47.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.227+0000 7f65d91dd700 1 -- 172.21.15.158:0/317331751 wait complete. 2023-10-16T09:44:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.228+0000 7f65d91dd700 1 Processor -- start 2023-10-16T09:44:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.228+0000 7f65d91dd700 1 -- start start 2023-10-16T09:44:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d91dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d40cdf90 0x7f65d41512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d91dd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d40ce440 0x7f65d41f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:47.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d91dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d415b4c0 0x7f65d4158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d91dd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d41531f0 con 0x7f65d40cdf90 2023-10-16T09:44:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d91dd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d41517f0 con 0x7f65d40ce440 2023-10-16T09:44:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d91dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d41f5420 con 0x7f65d415b4c0 2023-10-16T09:44:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d40cdf90 0x7f65d41512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d40cdf90 0x7f65d41512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48544/0 (socket says 172.21.15.158:48544) 2023-10-16T09:44:47.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 -- 172.21.15.158:0/3104957304 learned_addr learned my addr 172.21.15.158:0/3104957304 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d259c700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d40ce440 0x7f65d41f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d359e700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d415b4c0 0x7f65d4158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 -- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d415b4c0 msgr2=0x7f65d4158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d415b4c0 0x7f65d4158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 -- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d40ce440 msgr2=0x7f65d41f4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d40ce440 0x7f65d41f4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 -- 172.21.15.158:0/3104957304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80094a0 con 0x7f65d40cdf90 2023-10-16T09:44:47.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.229+0000 7f65d2d9d700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d40cdf90 0x7f65d41512b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f65c800ebe0 tx=0x7f65c800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:47.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.230+0000 7f65bb7fe700 1 -- 172.21.15.158:0/3104957304 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8009760 con 0x7f65d40cdf90 2023-10-16T09:44:47.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.230+0000 7f65bb7fe700 1 -- 172.21.15.158:0/3104957304 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c800c850 con 0x7f65d40cdf90 2023-10-16T09:44:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.230+0000 7f65bb7fe700 1 -- 172.21.15.158:0/3104957304 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c800cbe0 con 0x7f65d40cdf90 2023-10-16T09:44:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.230+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc009b30 con 0x7f65d40cdf90 2023-10-16T09:44:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.230+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d4152430 con 0x7f65d40cdf90 2023-10-16T09:44:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.231+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d40681b0 con 0x7f65d40cdf90 2023-10-16T09:44:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.231+0000 7f65bb7fe700 1 -- 172.21.15.158:0/3104957304 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f65c800cd40 con 0x7f65d40cdf90 2023-10-16T09:44:47.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.232+0000 7f65bb7fe700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f65c006af00 0x7f65c006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:47.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.232+0000 7f65bb7fe700 1 -- 172.21.15.158:0/3104957304 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f65c8089e60 con 0x7f65d40cdf90 2023-10-16T09:44:47.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.232+0000 7f65d259c700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f65c006af00 0x7f65c006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:47.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.232+0000 7f65d259c700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f65c006af00 0x7f65c006d3c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f65c4000f60 tx=0x7f65c4007ed0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:47.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.236+0000 7f65bb7fe700 1 -- 172.21.15.158:0/3104957304 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65c80b6fe0 con 0x7f65d40cdf90 2023-10-16T09:44:47.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.421+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65d406ac10 con 0x7f65c006af00 2023-10-16T09:44:47.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.429+0000 7f65bb7fe700 1 -- 172.21.15.158:0/3104957304 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288753 (secure 0 0 0) 0x7f65c4001330 con 0x7f65c006af00 2023-10-16T09:44:47.434 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:47.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f65c006af00 msgr2=0x7f65c006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f65c006af00 0x7f65c006d3c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f65c4000f60 tx=0x7f65c4007ed0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d40cdf90 msgr2=0x7f65d41512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:47.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d40cdf90 0x7f65d41512b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f65c800ebe0 tx=0x7f65c800eef0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 shutdown_connections 2023-10-16T09:44:47.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f65c006af00 0x7f65c006d3c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65d415b4c0 0x7f65d4158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d40cdf90 0x7f65d41512b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 --2- 172.21.15.158:0/3104957304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d40ce440 0x7f65d41f4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:47.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 >> 172.21.15.158:0/3104957304 conn(0x7f65d40cc4b0 msgr2=0x7f65d40715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:47.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 shutdown_connections 2023-10-16T09:44:47.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:47.432+0000 7f65d91dd700 1 -- 172.21.15.158:0/3104957304 wait complete. 2023-10-16T09:44:47.439 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:44:47.446 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2023-10-16T09:44:47.235944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220940,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753780,"statfs":{"total":773094113280,"available":772867870720,"internally_reserved":0,"allocated":4857856,"data_stored":2750084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.550287"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302288+0000","last_change":"2023-10-16T09:44:41.715983+0000","last_active":"2023-10-16T09:44:43.302288+0000","last_peered":"2023-10-16T09:44:43.302288+0000","last_clean":"2023-10-16T09:44:43.302288+0000","last_became_active":"2023-10-16T09:44:41.715679+0000","last_became_peered":"2023-10-16T09:44:41.715679+0000","last_unstale":"2023-10-16T09:44:43.302288+0000","last_undegraded":"2023-10-16T09:44:43.302288+0000","last_fullsized":"2023-10-16T09:44:43.302288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:49.835781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302206+0000","last_change":"2023-10-16T09:44:41.714384+0000","last_active":"2023-10-16T09:44:43.302206+0000","last_peered":"2023-10-16T09:44:43.302206+0000","last_clean":"2023-10-16T09:44:43.302206+0000","last_became_active":"2023-10-16T09:44:41.714139+0000","last_became_peered":"2023-10-16T09:44:41.714139+0000","last_unstale":"2023-10-16T09:44:43.302206+0000","last_undegraded":"2023-10-16T09:44:43.302206+0000","last_fullsized":"2023-10-16T09:44:43.302206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:20.696205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301376+0000","last_change":"2023-10-16T09:44:41.722778+0000","last_active":"2023-10-16T09:44:43.301376+0000","last_peered":"2023-10-16T09:44:43.301376+0000","last_clean":"2023-10-16T09:44:43.301376+0000","last_became_active":"2023-10-16T09:44:41.722491+0000","last_became_peered":"2023-10-16T09:44:41.722491+0000","last_unstale":"2023-10-16T09:44:43.301376+0000","last_undegraded":"2023-10-16T09:44:43.301376+0000","last_fullsized":"2023-10-16T09:44:43.301376+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:10.486173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823911+0000","last_change":"2023-10-16T09:44:41.711778+0000","last_active":"2023-10-16T09:44:42.823911+0000","last_peered":"2023-10-16T09:44:42.823911+0000","last_clean":"2023-10-16T09:44:42.823911+0000","last_became_active":"2023-10-16T09:44:41.711501+0000","last_became_peered":"2023-10-16T09:44:41.711501+0000","last_unstale":"2023-10-16T09:44:42.823911+0000","last_undegraded":"2023-10-16T09:44:42.823911+0000","last_fullsized":"2023-10-16T09:44:42.823911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:36.319832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823945+0000","last_change":"2023-10-16T09:44:41.711009+0000","last_active":"2023-10-16T09:44:42.823945+0000","last_peered":"2023-10-16T09:44:42.823945+0000","last_clean":"2023-10-16T09:44:42.823945+0000","last_became_active":"2023-10-16T09:44:41.710670+0000","last_became_peered":"2023-10-16T09:44:41.710670+0000","last_unstale":"2023-10-16T09:44:42.823945+0000","last_undegraded":"2023-10-16T09:44:42.823945+0000","last_fullsized":"2023-10-16T09:44:42.823945+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:43.077817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300621+0000","last_change":"2023-10-16T09:44:41.708805+0000","last_active":"2023-10-16T09:44:43.300621+0000","last_peered":"2023-10-16T09:44:43.300621+0000","last_clean":"2023-10-16T09:44:43.300621+0000","last_became_active":"2023-10-16T09:44:41.708534+0000","last_became_peered":"2023-10-16T09:44:41.708534+0000","last_unstale":"2023-10-16T09:44:43.300621+0000","last_undegraded":"2023-10-16T09:44:43.300621+0000","last_fullsized":"2023-10-16T09:44:43.300621+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:10.486173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826195+0000","last_change":"2023-10-16T09:44:41.721814+0000","last_active":"2023-10-16T09:44:42.826195+0000","last_peered":"2023-10-16T09:44:42.826195+0000","last_clean":"2023-10-16T09:44:42.826195+0000","last_became_active":"2023-10-16T09:44:41.721529+0000","last_became_peered":"2023-10-16T09:44:41.721529+0000","last_unstale":"2023-10-16T09:44:42.826195+0000","last_undegraded":"2023-10-16T09:44:42.826195+0000","last_fullsized":"2023-10-16T09:44:42.826195+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:02.867835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301633+0000","last_change":"2023-10-16T09:44:41.726475+0000","last_active":"2023-10-16T09:44:43.301633+0000","last_peered":"2023-10-16T09:44:43.301633+0000","last_clean":"2023-10-16T09:44:43.301633+0000","last_became_active":"2023-10-16T09:44:41.726201+0000","last_became_peered":"2023-10-16T09:44:41.726201+0000","last_unstale":"2023-10-16T09:44:43.301633+0000","last_undegraded":"2023-10-16T09:44:43.301633+0000","last_fullsized":"2023-10-16T09:44:43.301633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:02.867835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302079+0000","last_change":"2023-10-16T09:44:41.716157+0000","last_active":"2023-10-16T09:44:43.302079+0000","last_peered":"2023-10-16T09:44:43.302079+0000","last_clean":"2023-10-16T09:44:43.302079+0000","last_became_active":"2023-10-16T09:44:41.715886+0000","last_became_peered":"2023-10-16T09:44:41.715886+0000","last_unstale":"2023-10-16T09:44:43.302079+0000","last_undegraded":"2023-10-16T09:44:43.302079+0000","last_fullsized":"2023-10-16T09:44:43.302079+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:06.221143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826655+0000","last_change":"2023-10-16T09:44:41.712662+0000","last_active":"2023-10-16T09:44:42.826655+0000","last_peered":"2023-10-16T09:44:42.826655+0000","last_clean":"2023-10-16T09:44:42.826655+0000","last_became_active":"2023-10-16T09:44:41.712272+0000","last_became_peered":"2023-10-16T09:44:41.712272+0000","last_unstale":"2023-10-16T09:44:42.826655+0000","last_undegraded":"2023-10-16T09:44:42.826655+0000","last_fullsized":"2023-10-16T09:44:42.826655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:51.986693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826543+0000","last_change":"2023-10-16T09:44:41.712812+0000","last_active":"2023-10-16T09:44:42.826543+0000","last_peered":"2023-10-16T09:44:42.826543+0000","last_clean":"2023-10-16T09:44:42.826543+0000","last_became_active":"2023-10-16T09:44:41.710569+0000","last_became_peered":"2023-10-16T09:44:41.710569+0000","last_unstale":"2023-10-16T09:44:42.826543+0000","last_undegraded":"2023-10-16T09:44:42.826543+0000","last_fullsized":"2023-10-16T09:44:42.826543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:54.034136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301438+0000","last_change":"2023-10-16T09:44:41.713449+0000","last_active":"2023-10-16T09:44:43.301438+0000","last_peered":"2023-10-16T09:44:43.301438+0000","last_clean":"2023-10-16T09:44:43.301438+0000","last_became_active":"2023-10-16T09:44:41.713171+0000","last_became_peered":"2023-10-16T09:44:41.713171+0000","last_unstale":"2023-10-16T09:44:43.301438+0000","last_undegraded":"2023-10-16T09:44:43.301438+0000","last_fullsized":"2023-10-16T09:44:43.301438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:55.334657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826186+0000","last_change":"2023-10-16T09:44:41.712647+0000","last_active":"2023-10-16T09:44:42.826186+0000","last_peered":"2023-10-16T09:44:42.826186+0000","last_clean":"2023-10-16T09:44:42.826186+0000","last_became_active":"2023-10-16T09:44:41.710678+0000","last_became_peered":"2023-10-16T09:44:41.710678+0000","last_unstale":"2023-10-16T09:44:42.826186+0000","last_undegraded":"2023-10-16T09:44:42.826186+0000","last_fullsized":"2023-10-16T09:44:42.826186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:41.255334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701225+0000","last_change":"2023-10-16T09:44:41.723197+0000","last_active":"2023-10-16T09:44:42.701225+0000","last_peered":"2023-10-16T09:44:42.701225+0000","last_clean":"2023-10-16T09:44:42.701225+0000","last_became_active":"2023-10-16T09:44:41.722914+0000","last_became_peered":"2023-10-16T09:44:41.722914+0000","last_unstale":"2023-10-16T09:44:42.701225+0000","last_undegraded":"2023-10-16T09:44:42.701225+0000","last_fullsized":"2023-10-16T09:44:42.701225+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:12.843916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.825993+0000","last_change":"2023-10-16T09:44:41.721767+0000","last_active":"2023-10-16T09:44:42.825993+0000","last_peered":"2023-10-16T09:44:42.825993+0000","last_clean":"2023-10-16T09:44:42.825993+0000","last_became_active":"2023-10-16T09:44:41.721319+0000","last_became_peered":"2023-10-16T09:44:41.721319+0000","last_unstale":"2023-10-16T09:44:42.825993+0000","last_undegraded":"2023-10-16T09:44:42.825993+0000","last_fullsized":"2023-10-16T09:44:42.825993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:50.971484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699414+0000","last_change":"2023-10-16T09:44:41.716090+0000","last_active":"2023-10-16T09:44:42.699414+0000","last_peered":"2023-10-16T09:44:42.699414+0000","last_clean":"2023-10-16T09:44:42.699414+0000","last_became_active":"2023-10-16T09:44:41.715799+0000","last_became_peered":"2023-10-16T09:44:41.715799+0000","last_unstale":"2023-10-16T09:44:42.699414+0000","last_undegraded":"2023-10-16T09:44:42.699414+0000","last_fullsized":"2023-10-16T09:44:42.699414+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:06.221143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700044+0000","last_change":"2023-10-16T09:44:41.726858+0000","last_active":"2023-10-16T09:44:42.700044+0000","last_peered":"2023-10-16T09:44:42.700044+0000","last_clean":"2023-10-16T09:44:42.700044+0000","last_became_active":"2023-10-16T09:44:41.726362+0000","last_became_peered":"2023-10-16T09:44:41.726362+0000","last_unstale":"2023-10-16T09:44:42.700044+0000","last_undegraded":"2023-10-16T09:44:42.700044+0000","last_fullsized":"2023-10-16T09:44:42.700044+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:02.867835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701622+0000","last_change":"2023-10-16T09:44:41.718454+0000","last_active":"2023-10-16T09:44:42.701622+0000","last_peered":"2023-10-16T09:44:42.701622+0000","last_clean":"2023-10-16T09:44:42.701622+0000","last_became_active":"2023-10-16T09:44:41.718250+0000","last_became_peered":"2023-10-16T09:44:41.718250+0000","last_unstale":"2023-10-16T09:44:42.701622+0000","last_undegraded":"2023-10-16T09:44:42.701622+0000","last_fullsized":"2023-10-16T09:44:42.701622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:34.020959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823949+0000","last_change":"2023-10-16T09:44:41.711102+0000","last_active":"2023-10-16T09:44:42.823949+0000","last_peered":"2023-10-16T09:44:42.823949+0000","last_clean":"2023-10-16T09:44:42.823949+0000","last_became_active":"2023-10-16T09:44:41.710779+0000","last_became_peered":"2023-10-16T09:44:41.710779+0000","last_unstale":"2023-10-16T09:44:42.823949+0000","last_undegraded":"2023-10-16T09:44:42.823949+0000","last_fullsized":"2023-10-16T09:44:42.823949+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:09.602576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701420+0000","last_change":"2023-10-16T09:44:41.711811+0000","last_active":"2023-10-16T09:44:42.701420+0000","last_peered":"2023-10-16T09:44:42.701420+0000","last_clean":"2023-10-16T09:44:42.701420+0000","last_became_active":"2023-10-16T09:44:41.711492+0000","last_became_peered":"2023-10-16T09:44:41.711492+0000","last_unstale":"2023-10-16T09:44:42.701420+0000","last_undegraded":"2023-10-16T09:44:42.701420+0000","last_fullsized":"2023-10-16T09:44:42.701420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:25.433203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826200+0000","last_change":"2023-10-16T09:44:41.718297+0000","last_active":"2023-10-16T09:44:42.826200+0000","last_peered":"2023-10-16T09:44:42.826200+0000","last_clean":"2023-10-16T09:44:42.826200+0000","last_became_active":"2023-10-16T09:44:41.715805+0000","last_became_peered":"2023-10-16T09:44:41.715805+0000","last_unstale":"2023-10-16T09:44:42.826200+0000","last_undegraded":"2023-10-16T09:44:42.826200+0000","last_fullsized":"2023-10-16T09:44:42.826200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:47.928959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700916+0000","last_change":"2023-10-16T09:44:41.714375+0000","last_active":"2023-10-16T09:44:42.700916+0000","last_peered":"2023-10-16T09:44:42.700916+0000","last_clean":"2023-10-16T09:44:42.700916+0000","last_became_active":"2023-10-16T09:44:41.714135+0000","last_became_peered":"2023-10-16T09:44:41.714135+0000","last_unstale":"2023-10-16T09:44:42.700916+0000","last_undegraded":"2023-10-16T09:44:42.700916+0000","last_fullsized":"2023-10-16T09:44:42.700916+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:50.795711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699603+0000","last_change":"2023-10-16T09:44:41.713529+0000","last_active":"2023-10-16T09:44:42.699603+0000","last_peered":"2023-10-16T09:44:42.699603+0000","last_clean":"2023-10-16T09:44:42.699603+0000","last_became_active":"2023-10-16T09:44:41.713092+0000","last_became_peered":"2023-10-16T09:44:41.713092+0000","last_unstale":"2023-10-16T09:44:42.699603+0000","last_undegraded":"2023-10-16T09:44:42.699603+0000","last_fullsized":"2023-10-16T09:44:42.699603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:33.948283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823773+0000","last_change":"2023-10-16T09:44:41.712401+0000","last_active":"2023-10-16T09:44:42.823773+0000","last_peered":"2023-10-16T09:44:42.823773+0000","last_clean":"2023-10-16T09:44:42.823773+0000","last_became_active":"2023-10-16T09:44:41.712015+0000","last_became_peered":"2023-10-16T09:44:41.712015+0000","last_unstale":"2023-10-16T09:44:42.823773+0000","last_undegraded":"2023-10-16T09:44:42.823773+0000","last_fullsized":"2023-10-16T09:44:42.823773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:36.417995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701692+0000","last_change":"2023-10-16T09:44:41.713591+0000","last_active":"2023-10-16T09:44:42.701692+0000","last_peered":"2023-10-16T09:44:42.701692+0000","last_clean":"2023-10-16T09:44:42.701692+0000","last_became_active":"2023-10-16T09:44:41.713324+0000","last_became_peered":"2023-10-16T09:44:41.713324+0000","last_unstale":"2023-10-16T09:44:42.701692+0000","last_undegraded":"2023-10-16T09:44:42.701692+0000","last_fullsized":"2023-10-16T09:44:42.701692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:59.353526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301122+0000","last_change":"2023-10-16T09:44:41.716327+0000","last_active":"2023-10-16T09:44:43.301122+0000","last_peered":"2023-10-16T09:44:43.301122+0000","last_clean":"2023-10-16T09:44:43.301122+0000","last_became_active":"2023-10-16T09:44:41.716035+0000","last_became_peered":"2023-10-16T09:44:41.716035+0000","last_unstale":"2023-10-16T09:44:43.301122+0000","last_undegraded":"2023-10-16T09:44:43.301122+0000","last_fullsized":"2023-10-16T09:44:43.301122+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:06.221143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701628+0000","last_change":"2023-10-16T09:44:41.718065+0000","last_active":"2023-10-16T09:44:42.701628+0000","last_peered":"2023-10-16T09:44:42.701628+0000","last_clean":"2023-10-16T09:44:42.701628+0000","last_became_active":"2023-10-16T09:44:41.717899+0000","last_became_peered":"2023-10-16T09:44:41.717899+0000","last_unstale":"2023-10-16T09:44:42.701628+0000","last_undegraded":"2023-10-16T09:44:42.701628+0000","last_fullsized":"2023-10-16T09:44:42.701628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:34:28.701882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826306+0000","last_change":"2023-10-16T09:44:41.725435+0000","last_active":"2023-10-16T09:44:42.826306+0000","last_peered":"2023-10-16T09:44:42.826306+0000","last_clean":"2023-10-16T09:44:42.826306+0000","last_became_active":"2023-10-16T09:44:41.725037+0000","last_became_peered":"2023-10-16T09:44:41.725037+0000","last_unstale":"2023-10-16T09:44:42.826306+0000","last_undegraded":"2023-10-16T09:44:42.826306+0000","last_fullsized":"2023-10-16T09:44:42.826306+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.965241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701735+0000","last_change":"2023-10-16T09:44:41.721031+0000","last_active":"2023-10-16T09:44:42.701735+0000","last_peered":"2023-10-16T09:44:42.701735+0000","last_clean":"2023-10-16T09:44:42.701735+0000","last_became_active":"2023-10-16T09:44:41.720732+0000","last_became_peered":"2023-10-16T09:44:41.720732+0000","last_unstale":"2023-10-16T09:44:42.701735+0000","last_undegraded":"2023-10-16T09:44:42.701735+0000","last_fullsized":"2023-10-16T09:44:42.701735+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:39:08.048466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701430+0000","last_change":"2023-10-16T09:44:41.713476+0000","last_active":"2023-10-16T09:44:42.701430+0000","last_peered":"2023-10-16T09:44:42.701430+0000","last_clean":"2023-10-16T09:44:42.701430+0000","last_became_active":"2023-10-16T09:44:41.713176+0000","last_became_peered":"2023-10-16T09:44:41.713176+0000","last_unstale":"2023-10-16T09:44:42.701430+0000","last_undegraded":"2023-10-16T09:44:42.701430+0000","last_fullsized":"2023-10-16T09:44:42.701430+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:23.401056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.698987+0000","last_change":"2023-10-16T09:44:41.714041+0000","last_active":"2023-10-16T09:44:42.698987+0000","last_peered":"2023-10-16T09:44:42.698987+0000","last_clean":"2023-10-16T09:44:42.698987+0000","last_became_active":"2023-10-16T09:44:41.713753+0000","last_became_peered":"2023-10-16T09:44:41.713753+0000","last_unstale":"2023-10-16T09:44:42.698987+0000","last_undegraded":"2023-10-16T09:44:42.698987+0000","last_fullsized":"2023-10-16T09:44:42.698987+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:10.486173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300591+0000","last_change":"2023-10-16T09:44:41.713988+0000","last_active":"2023-10-16T09:44:43.300591+0000","last_peered":"2023-10-16T09:44:43.300591+0000","last_clean":"2023-10-16T09:44:43.300591+0000","last_became_active":"2023-10-16T09:44:41.713590+0000","last_became_peered":"2023-10-16T09:44:41.713590+0000","last_unstale":"2023-10-16T09:44:43.300591+0000","last_undegraded":"2023-10-16T09:44:43.300591+0000","last_fullsized":"2023-10-16T09:44:43.300591+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.848266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699607+0000","last_change":"2023-10-16T09:44:41.722709+0000","last_active":"2023-10-16T09:44:42.699607+0000","last_peered":"2023-10-16T09:44:42.699607+0000","last_clean":"2023-10-16T09:44:42.699607+0000","last_became_active":"2023-10-16T09:44:41.722451+0000","last_became_peered":"2023-10-16T09:44:41.722451+0000","last_unstale":"2023-10-16T09:44:42.699607+0000","last_undegraded":"2023-10-16T09:44:42.699607+0000","last_fullsized":"2023-10-16T09:44:42.699607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:41.255334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733573+0000","last_change":"2023-10-16T09:44:40.702880+0000","last_active":"2023-10-16T09:44:41.733573+0000","last_peered":"2023-10-16T09:44:41.733573+0000","last_clean":"2023-10-16T09:44:41.733573+0000","last_became_active":"2023-10-16T09:44:40.702634+0000","last_became_peered":"2023-10-16T09:44:40.702634+0000","last_unstale":"2023-10-16T09:44:41.733573+0000","last_undegraded":"2023-10-16T09:44:41.733573+0000","last_fullsized":"2023-10-16T09:44:41.733573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:10.979826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300965+0000","last_change":"2023-10-16T09:44:41.721756+0000","last_active":"2023-10-16T09:44:43.300965+0000","last_peered":"2023-10-16T09:44:43.300965+0000","last_clean":"2023-10-16T09:44:43.300965+0000","last_became_active":"2023-10-16T09:44:41.721482+0000","last_became_peered":"2023-10-16T09:44:41.721482+0000","last_unstale":"2023-10-16T09:44:43.300965+0000","last_undegraded":"2023-10-16T09:44:43.300965+0000","last_fullsized":"2023-10-16T09:44:43.300965+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:55.334657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733855+0000","last_change":"2023-10-16T09:44:40.704794+0000","last_active":"2023-10-16T09:44:41.733855+0000","last_peered":"2023-10-16T09:44:41.733855+0000","last_clean":"2023-10-16T09:44:41.733855+0000","last_became_active":"2023-10-16T09:44:40.704567+0000","last_became_peered":"2023-10-16T09:44:40.704567+0000","last_unstale":"2023-10-16T09:44:41.733855+0000","last_undegraded":"2023-10-16T09:44:41.733855+0000","last_fullsized":"2023-10-16T09:44:41.733855+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:27.249703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700288+0000","last_change":"2023-10-16T09:44:41.711461+0000","last_active":"2023-10-16T09:44:42.700288+0000","last_peered":"2023-10-16T09:44:42.700288+0000","last_clean":"2023-10-16T09:44:42.700288+0000","last_became_active":"2023-10-16T09:44:41.711179+0000","last_became_peered":"2023-10-16T09:44:41.711179+0000","last_unstale":"2023-10-16T09:44:42.700288+0000","last_undegraded":"2023-10-16T09:44:42.700288+0000","last_fullsized":"2023-10-16T09:44:42.700288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:39:08.048466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732996+0000","last_change":"2023-10-16T09:44:40.710707+0000","last_active":"2023-10-16T09:44:41.732996+0000","last_peered":"2023-10-16T09:44:41.732996+0000","last_clean":"2023-10-16T09:44:41.732996+0000","last_became_active":"2023-10-16T09:44:40.710264+0000","last_became_peered":"2023-10-16T09:44:40.710264+0000","last_unstale":"2023-10-16T09:44:41.732996+0000","last_undegraded":"2023-10-16T09:44:41.732996+0000","last_fullsized":"2023-10-16T09:44:41.732996+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:49.791036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302117+0000","last_change":"2023-10-16T09:44:41.722861+0000","last_active":"2023-10-16T09:44:43.302117+0000","last_peered":"2023-10-16T09:44:43.302117+0000","last_clean":"2023-10-16T09:44:43.302117+0000","last_became_active":"2023-10-16T09:44:41.722533+0000","last_became_peered":"2023-10-16T09:44:41.722533+0000","last_unstale":"2023-10-16T09:44:43.302117+0000","last_undegraded":"2023-10-16T09:44:43.302117+0000","last_fullsized":"2023-10-16T09:44:43.302117+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:51.986693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732577+0000","last_change":"2023-10-16T09:44:40.707210+0000","last_active":"2023-10-16T09:44:41.732577+0000","last_peered":"2023-10-16T09:44:41.732577+0000","last_clean":"2023-10-16T09:44:41.732577+0000","last_became_active":"2023-10-16T09:44:40.706931+0000","last_became_peered":"2023-10-16T09:44:40.706931+0000","last_unstale":"2023-10-16T09:44:41.732577+0000","last_undegraded":"2023-10-16T09:44:41.732577+0000","last_fullsized":"2023-10-16T09:44:41.732577+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:22.049243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301259+0000","last_change":"2023-10-16T09:44:41.704334+0000","last_active":"2023-10-16T09:44:43.301259+0000","last_peered":"2023-10-16T09:44:43.301259+0000","last_clean":"2023-10-16T09:44:43.301259+0000","last_became_active":"2023-10-16T09:44:41.703863+0000","last_became_peered":"2023-10-16T09:44:41.703863+0000","last_unstale":"2023-10-16T09:44:43.301259+0000","last_undegraded":"2023-10-16T09:44:43.301259+0000","last_fullsized":"2023-10-16T09:44:43.301259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.848266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733020+0000","last_change":"2023-10-16T09:44:40.698975+0000","last_active":"2023-10-16T09:44:41.733020+0000","last_peered":"2023-10-16T09:44:41.733020+0000","last_clean":"2023-10-16T09:44:41.733020+0000","last_became_active":"2023-10-16T09:44:40.698667+0000","last_became_peered":"2023-10-16T09:44:40.698667+0000","last_unstale":"2023-10-16T09:44:41.733020+0000","last_undegraded":"2023-10-16T09:44:41.733020+0000","last_fullsized":"2023-10-16T09:44:41.733020+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:11.839241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.825986+0000","last_change":"2023-10-16T09:44:41.721594+0000","last_active":"2023-10-16T09:44:42.825986+0000","last_peered":"2023-10-16T09:44:42.825986+0000","last_clean":"2023-10-16T09:44:42.825986+0000","last_became_active":"2023-10-16T09:44:41.721220+0000","last_became_peered":"2023-10-16T09:44:41.721220+0000","last_unstale":"2023-10-16T09:44:42.825986+0000","last_undegraded":"2023-10-16T09:44:42.825986+0000","last_fullsized":"2023-10-16T09:44:42.825986+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:33.948283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732515+0000","last_change":"2023-10-16T09:44:40.705524+0000","last_active":"2023-10-16T09:44:41.732515+0000","last_peered":"2023-10-16T09:44:41.732515+0000","last_clean":"2023-10-16T09:44:41.732515+0000","last_became_active":"2023-10-16T09:44:40.705221+0000","last_became_peered":"2023-10-16T09:44:40.705221+0000","last_unstale":"2023-10-16T09:44:41.732515+0000","last_undegraded":"2023-10-16T09:44:41.732515+0000","last_fullsized":"2023-10-16T09:44:41.732515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:22.396380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700911+0000","last_change":"2023-10-16T09:44:41.721767+0000","last_active":"2023-10-16T09:44:42.700911+0000","last_peered":"2023-10-16T09:44:42.700911+0000","last_clean":"2023-10-16T09:44:42.700911+0000","last_became_active":"2023-10-16T09:44:41.721473+0000","last_became_peered":"2023-10-16T09:44:41.721473+0000","last_unstale":"2023-10-16T09:44:42.700911+0000","last_undegraded":"2023-10-16T09:44:42.700911+0000","last_fullsized":"2023-10-16T09:44:42.700911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:04.467315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701396+0000","last_change":"2023-10-16T09:44:40.704004+0000","last_active":"2023-10-16T09:44:41.701396+0000","last_peered":"2023-10-16T09:44:41.701396+0000","last_clean":"2023-10-16T09:44:41.701396+0000","last_became_active":"2023-10-16T09:44:40.703251+0000","last_became_peered":"2023-10-16T09:44:40.703251+0000","last_unstale":"2023-10-16T09:44:41.701396+0000","last_undegraded":"2023-10-16T09:44:41.701396+0000","last_fullsized":"2023-10-16T09:44:41.701396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:03.462640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301858+0000","last_change":"2023-10-16T09:44:41.711239+0000","last_active":"2023-10-16T09:44:43.301858+0000","last_peered":"2023-10-16T09:44:43.301858+0000","last_clean":"2023-10-16T09:44:43.301858+0000","last_became_active":"2023-10-16T09:44:41.711053+0000","last_became_peered":"2023-10-16T09:44:41.711053+0000","last_unstale":"2023-10-16T09:44:43.301858+0000","last_undegraded":"2023-10-16T09:44:43.301858+0000","last_fullsized":"2023-10-16T09:44:43.301858+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:54.034136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.703133+0000","last_change":"2023-10-16T09:44:40.701904+0000","last_active":"2023-10-16T09:44:41.703133+0000","last_peered":"2023-10-16T09:44:41.703133+0000","last_clean":"2023-10-16T09:44:41.703133+0000","last_became_active":"2023-10-16T09:44:40.701600+0000","last_became_peered":"2023-10-16T09:44:40.701600+0000","last_unstale":"2023-10-16T09:44:41.703133+0000","last_undegraded":"2023-10-16T09:44:41.703133+0000","last_fullsized":"2023-10-16T09:44:41.703133+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:17.296825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823915+0000","last_change":"2023-10-16T09:44:41.711297+0000","last_active":"2023-10-16T09:44:42.823915+0000","last_peered":"2023-10-16T09:44:42.823915+0000","last_clean":"2023-10-16T09:44:42.823915+0000","last_became_active":"2023-10-16T09:44:41.711030+0000","last_became_peered":"2023-10-16T09:44:41.711030+0000","last_unstale":"2023-10-16T09:44:42.823915+0000","last_undegraded":"2023-10-16T09:44:42.823915+0000","last_fullsized":"2023-10-16T09:44:42.823915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:37:15.717236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.708552+0000","last_change":"2023-10-16T09:44:40.710741+0000","last_active":"2023-10-16T09:44:41.708552+0000","last_peered":"2023-10-16T09:44:41.708552+0000","last_clean":"2023-10-16T09:44:41.708552+0000","last_became_active":"2023-10-16T09:44:40.710468+0000","last_became_peered":"2023-10-16T09:44:40.710468+0000","last_unstale":"2023-10-16T09:44:41.708552+0000","last_undegraded":"2023-10-16T09:44:41.708552+0000","last_fullsized":"2023-10-16T09:44:41.708552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:49.966809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700618+0000","last_change":"2023-10-16T09:44:41.722862+0000","last_active":"2023-10-16T09:44:42.700618+0000","last_peered":"2023-10-16T09:44:42.700618+0000","last_clean":"2023-10-16T09:44:42.700618+0000","last_became_active":"2023-10-16T09:44:41.722611+0000","last_became_peered":"2023-10-16T09:44:41.722611+0000","last_unstale":"2023-10-16T09:44:42.700618+0000","last_undegraded":"2023-10-16T09:44:42.700618+0000","last_fullsized":"2023-10-16T09:44:42.700618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:16.187818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.710537+0000","last_change":"2023-10-16T09:44:40.704708+0000","last_active":"2023-10-16T09:44:41.710537+0000","last_peered":"2023-10-16T09:44:41.710537+0000","last_clean":"2023-10-16T09:44:41.710537+0000","last_became_active":"2023-10-16T09:44:40.704330+0000","last_became_peered":"2023-10-16T09:44:40.704330+0000","last_unstale":"2023-10-16T09:44:41.710537+0000","last_undegraded":"2023-10-16T09:44:41.710537+0000","last_fullsized":"2023-10-16T09:44:41.710537+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:30.548816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301378+0000","last_change":"2023-10-16T09:44:41.711025+0000","last_active":"2023-10-16T09:44:43.301378+0000","last_peered":"2023-10-16T09:44:43.301378+0000","last_clean":"2023-10-16T09:44:43.301378+0000","last_became_active":"2023-10-16T09:44:41.710739+0000","last_became_peered":"2023-10-16T09:44:41.710739+0000","last_unstale":"2023-10-16T09:44:43.301378+0000","last_undegraded":"2023-10-16T09:44:43.301378+0000","last_fullsized":"2023-10-16T09:44:43.301378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:31.553491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733396+0000","last_change":"2023-10-16T09:44:40.710791+0000","last_active":"2023-10-16T09:44:41.733396+0000","last_peered":"2023-10-16T09:44:41.733396+0000","last_clean":"2023-10-16T09:44:41.733396+0000","last_became_active":"2023-10-16T09:44:40.710491+0000","last_became_peered":"2023-10-16T09:44:40.710491+0000","last_unstale":"2023-10-16T09:44:41.733396+0000","last_undegraded":"2023-10-16T09:44:41.733396+0000","last_fullsized":"2023-10-16T09:44:41.733396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:09.481497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700080+0000","last_change":"2023-10-16T09:44:41.713551+0000","last_active":"2023-10-16T09:44:42.700080+0000","last_peered":"2023-10-16T09:44:42.700080+0000","last_clean":"2023-10-16T09:44:42.700080+0000","last_became_active":"2023-10-16T09:44:41.713270+0000","last_became_peered":"2023-10-16T09:44:41.713270+0000","last_unstale":"2023-10-16T09:44:42.700080+0000","last_undegraded":"2023-10-16T09:44:42.700080+0000","last_fullsized":"2023-10-16T09:44:42.700080+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:59.353526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.735080+0000","last_change":"2023-10-16T09:44:40.709967+0000","last_active":"2023-10-16T09:44:41.735080+0000","last_peered":"2023-10-16T09:44:41.735080+0000","last_clean":"2023-10-16T09:44:41.735080+0000","last_became_active":"2023-10-16T09:44:40.709745+0000","last_became_peered":"2023-10-16T09:44:40.709745+0000","last_unstale":"2023-10-16T09:44:41.735080+0000","last_undegraded":"2023-10-16T09:44:41.735080+0000","last_fullsized":"2023-10-16T09:44:41.735080+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:09.481497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301010+0000","last_change":"2023-10-16T09:44:41.715601+0000","last_active":"2023-10-16T09:44:43.301010+0000","last_peered":"2023-10-16T09:44:43.301010+0000","last_clean":"2023-10-16T09:44:43.301010+0000","last_became_active":"2023-10-16T09:44:41.715332+0000","last_became_peered":"2023-10-16T09:44:41.715332+0000","last_unstale":"2023-10-16T09:44:43.301010+0000","last_undegraded":"2023-10-16T09:44:43.301010+0000","last_fullsized":"2023-10-16T09:44:43.301010+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:04.467315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.731405+0000","last_change":"2023-10-16T09:44:40.706782+0000","last_active":"2023-10-16T09:44:41.731405+0000","last_peered":"2023-10-16T09:44:41.731405+0000","last_clean":"2023-10-16T09:44:41.731405+0000","last_became_active":"2023-10-16T09:44:40.706515+0000","last_became_peered":"2023-10-16T09:44:40.706515+0000","last_unstale":"2023-10-16T09:44:41.731405+0000","last_undegraded":"2023-10-16T09:44:41.731405+0000","last_fullsized":"2023-10-16T09:44:41.731405+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:30.548816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301679+0000","last_change":"2023-10-16T09:44:41.712044+0000","last_active":"2023-10-16T09:44:43.301679+0000","last_peered":"2023-10-16T09:44:43.301679+0000","last_clean":"2023-10-16T09:44:43.301679+0000","last_became_active":"2023-10-16T09:44:41.711733+0000","last_became_peered":"2023-10-16T09:44:41.711733+0000","last_unstale":"2023-10-16T09:44:43.301679+0000","last_undegraded":"2023-10-16T09:44:43.301679+0000","last_fullsized":"2023-10-16T09:44:43.301679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:16.187818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700855+0000","last_change":"2023-10-16T09:44:40.704624+0000","last_active":"2023-10-16T09:44:41.700855+0000","last_peered":"2023-10-16T09:44:41.700855+0000","last_clean":"2023-10-16T09:44:41.700855+0000","last_became_active":"2023-10-16T09:44:40.704319+0000","last_became_peered":"2023-10-16T09:44:40.704319+0000","last_unstale":"2023-10-16T09:44:41.700855+0000","last_undegraded":"2023-10-16T09:44:41.700855+0000","last_fullsized":"2023-10-16T09:44:41.700855+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:22.396380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699259+0000","last_change":"2023-10-16T09:44:41.713526+0000","last_active":"2023-10-16T09:44:42.699259+0000","last_peered":"2023-10-16T09:44:42.699259+0000","last_clean":"2023-10-16T09:44:42.699259+0000","last_became_active":"2023-10-16T09:44:41.713235+0000","last_became_peered":"2023-10-16T09:44:41.713235+0000","last_unstale":"2023-10-16T09:44:42.699259+0000","last_undegraded":"2023-10-16T09:44:42.699259+0000","last_fullsized":"2023-10-16T09:44:42.699259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:55.334657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.709763+0000","last_change":"2023-10-16T09:44:40.706416+0000","last_active":"2023-10-16T09:44:41.709763+0000","last_peered":"2023-10-16T09:44:41.709763+0000","last_clean":"2023-10-16T09:44:41.709763+0000","last_became_active":"2023-10-16T09:44:40.706121+0000","last_became_peered":"2023-10-16T09:44:40.706121+0000","last_unstale":"2023-10-16T09:44:41.709763+0000","last_undegraded":"2023-10-16T09:44:41.709763+0000","last_fullsized":"2023-10-16T09:44:41.709763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:22.049243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301719+0000","last_change":"2023-10-16T09:44:41.712153+0000","last_active":"2023-10-16T09:44:43.301719+0000","last_peered":"2023-10-16T09:44:43.301719+0000","last_clean":"2023-10-16T09:44:43.301719+0000","last_became_active":"2023-10-16T09:44:41.711872+0000","last_became_peered":"2023-10-16T09:44:41.711872+0000","last_unstale":"2023-10-16T09:44:43.301719+0000","last_undegraded":"2023-10-16T09:44:43.301719+0000","last_fullsized":"2023-10-16T09:44:43.301719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:47.928959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733928+0000","last_change":"2023-10-16T09:44:40.708822+0000","last_active":"2023-10-16T09:44:41.733928+0000","last_peered":"2023-10-16T09:44:41.733928+0000","last_clean":"2023-10-16T09:44:41.733928+0000","last_became_active":"2023-10-16T09:44:40.708413+0000","last_became_peered":"2023-10-16T09:44:40.708413+0000","last_unstale":"2023-10-16T09:44:41.733928+0000","last_undegraded":"2023-10-16T09:44:41.733928+0000","last_fullsized":"2023-10-16T09:44:41.733928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:49.791036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301796+0000","last_change":"2023-10-16T09:44:41.722908+0000","last_active":"2023-10-16T09:44:43.301796+0000","last_peered":"2023-10-16T09:44:43.301796+0000","last_clean":"2023-10-16T09:44:43.301796+0000","last_became_active":"2023-10-16T09:44:41.722590+0000","last_became_peered":"2023-10-16T09:44:41.722590+0000","last_unstale":"2023-10-16T09:44:43.301796+0000","last_undegraded":"2023-10-16T09:44:43.301796+0000","last_fullsized":"2023-10-16T09:44:43.301796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.965241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733019+0000","last_change":"2023-10-16T09:44:40.702783+0000","last_active":"2023-10-16T09:44:41.733019+0000","last_peered":"2023-10-16T09:44:41.733019+0000","last_clean":"2023-10-16T09:44:41.733019+0000","last_became_active":"2023-10-16T09:44:40.702513+0000","last_became_peered":"2023-10-16T09:44:40.702513+0000","last_unstale":"2023-10-16T09:44:41.733019+0000","last_undegraded":"2023-10-16T09:44:41.733019+0000","last_fullsized":"2023-10-16T09:44:41.733019+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:54.329982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826464+0000","last_change":"2023-10-16T09:44:41.715529+0000","last_active":"2023-10-16T09:44:42.826464+0000","last_peered":"2023-10-16T09:44:42.826464+0000","last_clean":"2023-10-16T09:44:42.826464+0000","last_became_active":"2023-10-16T09:44:41.715257+0000","last_became_peered":"2023-10-16T09:44:41.715257+0000","last_unstale":"2023-10-16T09:44:42.826464+0000","last_undegraded":"2023-10-16T09:44:42.826464+0000","last_fullsized":"2023-10-16T09:44:42.826464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:57.428175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733510+0000","last_change":"2023-10-16T09:44:40.702916+0000","last_active":"2023-10-16T09:44:41.733510+0000","last_peered":"2023-10-16T09:44:41.733510+0000","last_clean":"2023-10-16T09:44:41.733510+0000","last_became_active":"2023-10-16T09:44:40.702630+0000","last_became_peered":"2023-10-16T09:44:40.702630+0000","last_unstale":"2023-10-16T09:44:41.733510+0000","last_undegraded":"2023-10-16T09:44:41.733510+0000","last_fullsized":"2023-10-16T09:44:41.733510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:22.396380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700405+0000","last_change":"2023-10-16T09:44:41.707197+0000","last_active":"2023-10-16T09:44:42.700405+0000","last_peered":"2023-10-16T09:44:42.700405+0000","last_clean":"2023-10-16T09:44:42.700405+0000","last_became_active":"2023-10-16T09:44:41.706555+0000","last_became_peered":"2023-10-16T09:44:41.706555+0000","last_unstale":"2023-10-16T09:44:42.700405+0000","last_undegraded":"2023-10-16T09:44:42.700405+0000","last_fullsized":"2023-10-16T09:44:42.700405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:28.254378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.729255+0000","last_change":"2023-10-16T09:44:40.705548+0000","last_active":"2023-10-16T09:44:41.729255+0000","last_peered":"2023-10-16T09:44:41.729255+0000","last_clean":"2023-10-16T09:44:41.729255+0000","last_became_active":"2023-10-16T09:44:40.705254+0000","last_became_peered":"2023-10-16T09:44:40.705254+0000","last_unstale":"2023-10-16T09:44:41.729255+0000","last_undegraded":"2023-10-16T09:44:41.729255+0000","last_fullsized":"2023-10-16T09:44:41.729255+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:51.383595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823954+0000","last_change":"2023-10-16T09:44:41.714436+0000","last_active":"2023-10-16T09:44:42.823954+0000","last_peered":"2023-10-16T09:44:42.823954+0000","last_clean":"2023-10-16T09:44:42.823954+0000","last_became_active":"2023-10-16T09:44:41.714142+0000","last_became_peered":"2023-10-16T09:44:41.714142+0000","last_unstale":"2023-10-16T09:44:42.823954+0000","last_undegraded":"2023-10-16T09:44:42.823954+0000","last_fullsized":"2023-10-16T09:44:42.823954+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:05.899379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701238+0000","last_change":"2023-10-16T09:44:40.703548+0000","last_active":"2023-10-16T09:44:41.701238+0000","last_peered":"2023-10-16T09:44:41.701238+0000","last_clean":"2023-10-16T09:44:41.701238+0000","last_became_active":"2023-10-16T09:44:40.703299+0000","last_became_peered":"2023-10-16T09:44:40.703299+0000","last_unstale":"2023-10-16T09:44:41.701238+0000","last_undegraded":"2023-10-16T09:44:41.701238+0000","last_fullsized":"2023-10-16T09:44:41.701238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:27.249703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699135+0000","last_change":"2023-10-16T09:44:41.713675+0000","last_active":"2023-10-16T09:44:42.699135+0000","last_peered":"2023-10-16T09:44:42.699135+0000","last_clean":"2023-10-16T09:44:42.699135+0000","last_became_active":"2023-10-16T09:44:41.713321+0000","last_became_peered":"2023-10-16T09:44:41.713321+0000","last_unstale":"2023-10-16T09:44:42.699135+0000","last_undegraded":"2023-10-16T09:44:42.699135+0000","last_fullsized":"2023-10-16T09:44:42.699135+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:31.553491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.729125+0000","last_change":"2023-10-16T09:44:40.702817+0000","last_active":"2023-10-16T09:44:41.729125+0000","last_peered":"2023-10-16T09:44:41.729125+0000","last_clean":"2023-10-16T09:44:41.729125+0000","last_became_active":"2023-10-16T09:44:40.700380+0000","last_became_peered":"2023-10-16T09:44:40.700380+0000","last_unstale":"2023-10-16T09:44:41.729125+0000","last_undegraded":"2023-10-16T09:44:41.729125+0000","last_fullsized":"2023-10-16T09:44:41.729125+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:39.230846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301084+0000","last_change":"2023-10-16T09:44:41.712532+0000","last_active":"2023-10-16T09:44:43.301084+0000","last_peered":"2023-10-16T09:44:43.301084+0000","last_clean":"2023-10-16T09:44:43.301084+0000","last_became_active":"2023-10-16T09:44:41.712280+0000","last_became_peered":"2023-10-16T09:44:41.712280+0000","last_unstale":"2023-10-16T09:44:43.301084+0000","last_undegraded":"2023-10-16T09:44:43.301084+0000","last_fullsized":"2023-10-16T09:44:43.301084+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:50.795711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700031+0000","last_change":"2023-10-16T09:44:40.702242+0000","last_active":"2023-10-16T09:44:41.700031+0000","last_peered":"2023-10-16T09:44:41.700031+0000","last_clean":"2023-10-16T09:44:41.700031+0000","last_became_active":"2023-10-16T09:44:40.702008+0000","last_became_peered":"2023-10-16T09:44:40.702008+0000","last_unstale":"2023-10-16T09:44:41.700031+0000","last_undegraded":"2023-10-16T09:44:41.700031+0000","last_fullsized":"2023-10-16T09:44:41.700031+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:15:13.705856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826207+0000","last_change":"2023-10-16T09:44:41.725281+0000","last_active":"2023-10-16T09:44:42.826207+0000","last_peered":"2023-10-16T09:44:42.826207+0000","last_clean":"2023-10-16T09:44:42.826207+0000","last_became_active":"2023-10-16T09:44:41.724987+0000","last_became_peered":"2023-10-16T09:44:41.724987+0000","last_unstale":"2023-10-16T09:44:42.826207+0000","last_undegraded":"2023-10-16T09:44:42.826207+0000","last_fullsized":"2023-10-16T09:44:42.826207+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:16.187818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701536+0000","last_change":"2023-10-16T09:44:40.708773+0000","last_active":"2023-10-16T09:44:41.701536+0000","last_peered":"2023-10-16T09:44:41.701536+0000","last_clean":"2023-10-16T09:44:41.701536+0000","last_became_active":"2023-10-16T09:44:40.708467+0000","last_became_peered":"2023-10-16T09:44:40.708467+0000","last_unstale":"2023-10-16T09:44:41.701536+0000","last_undegraded":"2023-10-16T09:44:41.701536+0000","last_fullsized":"2023-10-16T09:44:41.701536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:11.839241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301805+0000","last_change":"2023-10-16T09:44:41.711587+0000","last_active":"2023-10-16T09:44:43.301805+0000","last_peered":"2023-10-16T09:44:43.301805+0000","last_clean":"2023-10-16T09:44:43.301805+0000","last_became_active":"2023-10-16T09:44:41.711310+0000","last_became_peered":"2023-10-16T09:44:41.711310+0000","last_unstale":"2023-10-16T09:44:43.301805+0000","last_undegraded":"2023-10-16T09:44:43.301805+0000","last_fullsized":"2023-10-16T09:44:43.301805+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:57.428175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701467+0000","last_change":"2023-10-16T09:44:40.700966+0000","last_active":"2023-10-16T09:44:41.701467+0000","last_peered":"2023-10-16T09:44:41.701467+0000","last_clean":"2023-10-16T09:44:41.701467+0000","last_became_active":"2023-10-16T09:44:40.700697+0000","last_became_peered":"2023-10-16T09:44:40.700697+0000","last_unstale":"2023-10-16T09:44:41.701467+0000","last_undegraded":"2023-10-16T09:44:41.701467+0000","last_fullsized":"2023-10-16T09:44:41.701467+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:48.054971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300819+0000","last_change":"2023-10-16T09:44:41.727134+0000","last_active":"2023-10-16T09:44:43.300819+0000","last_peered":"2023-10-16T09:44:43.300819+0000","last_clean":"2023-10-16T09:44:43.300819+0000","last_became_active":"2023-10-16T09:44:41.726887+0000","last_became_peered":"2023-10-16T09:44:41.726887+0000","last_unstale":"2023-10-16T09:44:43.300819+0000","last_undegraded":"2023-10-16T09:44:43.300819+0000","last_fullsized":"2023-10-16T09:44:43.300819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:31.553491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732941+0000","last_change":"2023-10-16T09:44:40.703790+0000","last_active":"2023-10-16T09:44:41.732941+0000","last_peered":"2023-10-16T09:44:41.732941+0000","last_clean":"2023-10-16T09:44:41.732941+0000","last_became_active":"2023-10-16T09:44:40.703520+0000","last_became_peered":"2023-10-16T09:44:40.703520+0000","last_unstale":"2023-10-16T09:44:41.732941+0000","last_undegraded":"2023-10-16T09:44:41.732941+0000","last_fullsized":"2023-10-16T09:44:41.732941+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:05.216468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700677+0000","last_change":"2023-10-16T09:44:41.717136+0000","last_active":"2023-10-16T09:44:42.700677+0000","last_peered":"2023-10-16T09:44:42.700677+0000","last_clean":"2023-10-16T09:44:42.700677+0000","last_became_active":"2023-10-16T09:44:41.716885+0000","last_became_peered":"2023-10-16T09:44:41.716885+0000","last_unstale":"2023-10-16T09:44:42.700677+0000","last_undegraded":"2023-10-16T09:44:42.700677+0000","last_fullsized":"2023-10-16T09:44:42.700677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:47.928959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823864+0000","last_change":"2023-10-16T09:44:40.702669+0000","last_active":"2023-10-16T09:44:42.823864+0000","last_peered":"2023-10-16T09:44:42.823864+0000","last_clean":"2023-10-16T09:44:42.823864+0000","last_became_active":"2023-10-16T09:44:40.702410+0000","last_became_peered":"2023-10-16T09:44:40.702410+0000","last_unstale":"2023-10-16T09:44:42.823864+0000","last_undegraded":"2023-10-16T09:44:42.823864+0000","last_fullsized":"2023-10-16T09:44:42.823864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:15.630799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302324+0000","last_change":"2023-10-16T09:44:41.712832+0000","last_active":"2023-10-16T09:44:43.302324+0000","last_peered":"2023-10-16T09:44:43.302324+0000","last_clean":"2023-10-16T09:44:43.302324+0000","last_became_active":"2023-10-16T09:44:41.711860+0000","last_became_peered":"2023-10-16T09:44:41.711860+0000","last_unstale":"2023-10-16T09:44:43.302324+0000","last_undegraded":"2023-10-16T09:44:43.302324+0000","last_fullsized":"2023-10-16T09:44:43.302324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:08:34.357566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700121+0000","last_change":"2023-10-16T09:44:40.703799+0000","last_active":"2023-10-16T09:44:41.700121+0000","last_peered":"2023-10-16T09:44:41.700121+0000","last_clean":"2023-10-16T09:44:41.700121+0000","last_became_active":"2023-10-16T09:44:40.703433+0000","last_became_peered":"2023-10-16T09:44:40.703433+0000","last_unstale":"2023-10-16T09:44:41.700121+0000","last_undegraded":"2023-10-16T09:44:41.700121+0000","last_fullsized":"2023-10-16T09:44:41.700121+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:17.296825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301345+0000","last_change":"2023-10-16T09:44:41.709494+0000","last_active":"2023-10-16T09:44:43.301345+0000","last_peered":"2023-10-16T09:44:43.301345+0000","last_clean":"2023-10-16T09:44:43.301345+0000","last_became_active":"2023-10-16T09:44:41.709228+0000","last_became_peered":"2023-10-16T09:44:41.709228+0000","last_unstale":"2023-10-16T09:44:43.301345+0000","last_undegraded":"2023-10-16T09:44:43.301345+0000","last_fullsized":"2023-10-16T09:44:43.301345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:12.843916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.699909+0000","last_change":"2023-10-16T09:44:40.704621+0000","last_active":"2023-10-16T09:44:41.699909+0000","last_peered":"2023-10-16T09:44:41.699909+0000","last_clean":"2023-10-16T09:44:41.699909+0000","last_became_active":"2023-10-16T09:44:40.703625+0000","last_became_peered":"2023-10-16T09:44:40.703625+0000","last_unstale":"2023-10-16T09:44:41.699909+0000","last_undegraded":"2023-10-16T09:44:41.699909+0000","last_fullsized":"2023-10-16T09:44:41.699909+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:10.979826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300684+0000","last_change":"2023-10-16T09:44:41.710756+0000","last_active":"2023-10-16T09:44:43.300684+0000","last_peered":"2023-10-16T09:44:43.300684+0000","last_clean":"2023-10-16T09:44:43.300684+0000","last_became_active":"2023-10-16T09:44:41.710489+0000","last_became_peered":"2023-10-16T09:44:41.710489+0000","last_unstale":"2023-10-16T09:44:43.300684+0000","last_undegraded":"2023-10-16T09:44:43.300684+0000","last_fullsized":"2023-10-16T09:44:43.300684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:12.843916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.710491+0000","last_change":"2023-10-16T09:44:40.700385+0000","last_active":"2023-10-16T09:44:41.710491+0000","last_peered":"2023-10-16T09:44:41.710491+0000","last_clean":"2023-10-16T09:44:41.710491+0000","last_became_active":"2023-10-16T09:44:40.700118+0000","last_became_peered":"2023-10-16T09:44:40.700118+0000","last_unstale":"2023-10-16T09:44:41.710491+0000","last_undegraded":"2023-10-16T09:44:41.710491+0000","last_fullsized":"2023-10-16T09:44:41.710491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:25.169653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701067+0000","last_change":"2023-10-16T09:44:41.712317+0000","last_active":"2023-10-16T09:44:42.701067+0000","last_peered":"2023-10-16T09:44:42.701067+0000","last_clean":"2023-10-16T09:44:42.701067+0000","last_became_active":"2023-10-16T09:44:41.712046+0000","last_became_peered":"2023-10-16T09:44:41.712046+0000","last_unstale":"2023-10-16T09:44:42.701067+0000","last_undegraded":"2023-10-16T09:44:42.701067+0000","last_fullsized":"2023-10-16T09:44:42.701067+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.848266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.702645+0000","last_change":"2023-10-16T09:44:40.706560+0000","last_active":"2023-10-16T09:44:41.702645+0000","last_peered":"2023-10-16T09:44:41.702645+0000","last_clean":"2023-10-16T09:44:41.702645+0000","last_became_active":"2023-10-16T09:44:40.706126+0000","last_became_peered":"2023-10-16T09:44:40.706126+0000","last_unstale":"2023-10-16T09:44:41.702645+0000","last_undegraded":"2023-10-16T09:44:41.702645+0000","last_fullsized":"2023-10-16T09:44:41.702645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:35.480015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699880+0000","last_change":"2023-10-16T09:44:41.721849+0000","last_active":"2023-10-16T09:44:42.699880+0000","last_peered":"2023-10-16T09:44:42.699880+0000","last_clean":"2023-10-16T09:44:42.699880+0000","last_became_active":"2023-10-16T09:44:41.721581+0000","last_became_peered":"2023-10-16T09:44:41.721581+0000","last_unstale":"2023-10-16T09:44:42.699880+0000","last_undegraded":"2023-10-16T09:44:42.699880+0000","last_fullsized":"2023-10-16T09:44:42.699880+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:50.971484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.707272+0000","last_change":"2023-10-16T09:44:40.710752+0000","last_active":"2023-10-16T09:44:41.707272+0000","last_peered":"2023-10-16T09:44:41.707272+0000","last_clean":"2023-10-16T09:44:41.707272+0000","last_became_active":"2023-10-16T09:44:40.710456+0000","last_became_peered":"2023-10-16T09:44:40.710456+0000","last_unstale":"2023-10-16T09:44:41.707272+0000","last_undegraded":"2023-10-16T09:44:41.707272+0000","last_fullsized":"2023-10-16T09:44:41.707272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:54.329982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826708+0000","last_change":"2023-10-16T09:44:41.714926+0000","last_active":"2023-10-16T09:44:42.826708+0000","last_peered":"2023-10-16T09:44:42.826708+0000","last_clean":"2023-10-16T09:44:42.826708+0000","last_became_active":"2023-10-16T09:44:41.714681+0000","last_became_peered":"2023-10-16T09:44:41.714681+0000","last_unstale":"2023-10-16T09:44:42.826708+0000","last_undegraded":"2023-10-16T09:44:42.826708+0000","last_fullsized":"2023-10-16T09:44:42.826708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:20.696205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700514+0000","last_change":"2023-10-16T09:44:40.705265+0000","last_active":"2023-10-16T09:44:41.700514+0000","last_peered":"2023-10-16T09:44:41.700514+0000","last_clean":"2023-10-16T09:44:41.700514+0000","last_became_active":"2023-10-16T09:44:40.704988+0000","last_became_peered":"2023-10-16T09:44:40.704988+0000","last_unstale":"2023-10-16T09:44:41.700514+0000","last_undegraded":"2023-10-16T09:44:41.700514+0000","last_fullsized":"2023-10-16T09:44:41.700514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:24.428528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701207+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:44:41.701207+0000","last_peered":"2023-10-16T09:44:41.701207+0000","last_clean":"2023-10-16T09:44:41.701207+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:44:41.701207+0000","last_undegraded":"2023-10-16T09:44:41.701207+0000","last_fullsized":"2023-10-16T09:44:41.701207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":787616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":11.246,"5min":11.246,"15min":11.246},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.39800000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.379},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.442}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":196148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.435}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.496}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":195656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.255},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.28799999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.1760000000000002,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.76500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":785968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.5}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53300000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":195636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.376},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.32300000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.26700000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.51300000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47099999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.45900000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":197212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.20999999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.313},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42399999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.434},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":196170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59599999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.625},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":195678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:44:50.453 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:44:50.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.585+0000 7fb2dec1b700 1 Processor -- start 2023-10-16T09:44:50.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.586+0000 7fb2dec1b700 1 -- start start 2023-10-16T09:44:50.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dec1b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb2d80cdee0 0x7fb2d815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dec1b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb2d80b6e10 0x7fb2d81521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dec1b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d815c5d0 con 0x7fb2d80ce2c0 2023-10-16T09:44:50.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dec1b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d815d2c0 con 0x7fb2d80b6e10 2023-10-16T09:44:50.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dec1b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d815c8b0 con 0x7fb2d80cdee0 2023-10-16T09:44:50.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb2d80b6e10 0x7fb2d81521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dd1b8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb2d80ce2c0 0x7fb2d80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48732/0 (socket says 172.21.15.158:48732) 2023-10-16T09:44:50.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dd1b8700 1 -- 172.21.15.158:0/695557217 learned_addr learned my addr 172.21.15.158:0/695557217 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:50.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dc9b7700 1 --2- 172.21.15.158:0/695557217 >> v2:172.21.15.172:3300/0 conn(0x7fb2d80cdee0 0x7fb2d815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d7fff700 1 -- 172.21.15.158:0/695557217 >> v2:172.21.15.172:3300/0 conn(0x7fb2d80cdee0 msgr2=0x7fb2d815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:50.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d7fff700 1 --2- 172.21.15.158:0/695557217 >> v2:172.21.15.172:3300/0 conn(0x7fb2d80cdee0 0x7fb2d815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d7fff700 1 -- 172.21.15.158:0/695557217 >> v1:172.21.15.158:6789/0 conn(0x7fb2d80ce2c0 legacy=0x7fb2d80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d7fff700 1 -- 172.21.15.158:0/695557217 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d815cb90 con 0x7fb2d80b6e10 2023-10-16T09:44:50.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d77fe700 1 -- 172.21.15.158:0/695557217 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769244735 0 0) 0x7fb2d815c5d0 con 0x7fb2d80ce2c0 2023-10-16T09:44:50.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2dc9b7700 1 --2- 172.21.15.158:0/695557217 >> v2:172.21.15.172:3300/0 conn(0x7fb2d80cdee0 0x7fb2d815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:50.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d7fff700 1 --2- 172.21.15.158:0/695557217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d81521a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80d2470 tx=0x7fb2c4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906f704c46affb50 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:50.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.587+0000 7fb2d77fe700 1 -- 172.21.15.158:0/695557217 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4011830 con 0x7fb2d80b6e10 2023-10-16T09:44:50.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2d77fe700 1 -- 172.21.15.158:0/695557217 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c4011990 con 0x7fb2d80b6e10 2023-10-16T09:44:50.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2d77fe700 1 -- 172.21.15.158:0/695557217 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4011d20 con 0x7fb2d80b6e10 2023-10-16T09:44:50.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 -- 172.21.15.158:0/695557217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 msgr2=0x7fb2d81521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/695557217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d81521a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80d2470 tx=0x7fb2c4009b20 comp rx=0 tx=0).stop 2023-10-16T09:44:50.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 -- 172.21.15.158:0/695557217 shutdown_connections 2023-10-16T09:44:50.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/695557217 >> v2:172.21.15.172:3300/0 conn(0x7fb2d80cdee0 0x7fb2d815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/695557217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d81521a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 -- 172.21.15.158:0/695557217 >> 172.21.15.158:0/695557217 conn(0x7fb2d806a8d0 msgr2=0x7fb2d806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:50.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 -- 172.21.15.158:0/695557217 shutdown_connections 2023-10-16T09:44:50.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.588+0000 7fb2dec1b700 1 -- 172.21.15.158:0/695557217 wait complete. 2023-10-16T09:44:50.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.589+0000 7fb2dec1b700 1 Processor -- start 2023-10-16T09:44:50.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.589+0000 7fb2dec1b700 1 -- start start 2023-10-16T09:44:50.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dec1b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d80d1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dec1b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2d80cdee0 0x7fb2d80cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dec1b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d80ce2c0 0x7fb2d815f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dec1b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80d08a0 con 0x7fb2d80b6e10 2023-10-16T09:44:50.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dec1b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80d0b80 con 0x7fb2d80ce2c0 2023-10-16T09:44:50.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dec1b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80d0e60 con 0x7fb2d80cdee0 2023-10-16T09:44:50.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dc9b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d80d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dc9b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d80d1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48558/0 (socket says 172.21.15.158:48558) 2023-10-16T09:44:50.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.590+0000 7fb2dc9b7700 1 -- 172.21.15.158:0/645877880 learned_addr learned my addr 172.21.15.158:0/645877880 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:50.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dd1b8700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d80ce2c0 0x7fb2d815f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2d7fff700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2d80cdee0 0x7fb2d80cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dc9b7700 1 -- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2d80cdee0 msgr2=0x7fb2d80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dc9b7700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2d80cdee0 0x7fb2d80cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dc9b7700 1 -- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d80ce2c0 msgr2=0x7fb2d815f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dc9b7700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d80ce2c0 0x7fb2d815f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dc9b7700 1 -- 172.21.15.158:0/645877880 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40094a0 con 0x7fb2d80b6e10 2023-10-16T09:44:50.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2d7fff700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2d80cdee0 0x7fb2d80cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:50.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dc9b7700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d80d1940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c750 tx=0x7fb2cc00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:50.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2d57fa700 1 -- 172.21.15.158:0/645877880 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc011800 con 0x7fb2d80b6e10 2023-10-16T09:44:50.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2d57fa700 1 -- 172.21.15.158:0/645877880 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc011960 con 0x7fb2d80b6e10 2023-10-16T09:44:50.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d80d1140 con 0x7fb2d80b6e10 2023-10-16T09:44:50.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2d57fa700 1 -- 172.21.15.158:0/645877880 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc011cd0 con 0x7fb2d80b6e10 2023-10-16T09:44:50.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.591+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d80d1e80 con 0x7fb2d80b6e10 2023-10-16T09:44:50.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.592+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d80681b0 con 0x7fb2d80b6e10 2023-10-16T09:44:50.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.592+0000 7fb2d57fa700 1 -- 172.21.15.158:0/645877880 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb2cc021350 con 0x7fb2d80b6e10 2023-10-16T09:44:50.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.593+0000 7fb2d57fa700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb2bc06af00 0x7fb2bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.593+0000 7fb2d57fa700 1 -- 172.21.15.158:0/645877880 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb2cc089750 con 0x7fb2d80b6e10 2023-10-16T09:44:50.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.593+0000 7fb2d7fff700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb2bc06af00 0x7fb2bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.593+0000 7fb2d7fff700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb2bc06af00 0x7fb2bc06d3c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb2d815eff0 tx=0x7fb2c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:50.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.597+0000 7fb2d57fa700 1 -- 172.21.15.158:0/645877880 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2cc054620 con 0x7fb2d80b6e10 2023-10-16T09:44:50.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.785+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2d815c250 con 0x7fb2bc06af00 2023-10-16T09:44:50.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.793+0000 7fb2d57fa700 1 -- 172.21.15.158:0/645877880 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288753 (secure 0 0 0) 0x7fb2c4011c00 con 0x7fb2bc06af00 2023-10-16T09:44:50.799 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:50.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.796+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb2bc06af00 msgr2=0x7fb2bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.796+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb2bc06af00 0x7fb2bc06d3c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb2d815eff0 tx=0x7fb2c4008030 comp rx=0 tx=0).stop 2023-10-16T09:44:50.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.796+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 msgr2=0x7fb2d80d1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.796+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d80d1940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c750 tx=0x7fb2cc00cb10 comp rx=0 tx=0).stop 2023-10-16T09:44:50.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 shutdown_connections 2023-10-16T09:44:50.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d80ce2c0 0x7fb2d815f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2d80b6e10 0x7fb2d80d1940 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fb2bc06af00 0x7fb2bc06d3c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 --2- 172.21.15.158:0/645877880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2d80cdee0 0x7fb2d80cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 >> 172.21.15.158:0/645877880 conn(0x7fb2d806a8d0 msgr2=0x7fb2d80b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:50.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 shutdown_connections 2023-10-16T09:44:50.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.797+0000 7fb2dec1b700 1 -- 172.21.15.158:0/645877880 wait complete. 2023-10-16T09:44:50.804 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:44:50.811 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-10-16T09:44:49.236350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220940,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753780,"statfs":{"total":773094113280,"available":772867870720,"internally_reserved":0,"allocated":4857856,"data_stored":2750084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001344"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302288+0000","last_change":"2023-10-16T09:44:41.715983+0000","last_active":"2023-10-16T09:44:43.302288+0000","last_peered":"2023-10-16T09:44:43.302288+0000","last_clean":"2023-10-16T09:44:43.302288+0000","last_became_active":"2023-10-16T09:44:41.715679+0000","last_became_peered":"2023-10-16T09:44:41.715679+0000","last_unstale":"2023-10-16T09:44:43.302288+0000","last_undegraded":"2023-10-16T09:44:43.302288+0000","last_fullsized":"2023-10-16T09:44:43.302288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:49.835781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302206+0000","last_change":"2023-10-16T09:44:41.714384+0000","last_active":"2023-10-16T09:44:43.302206+0000","last_peered":"2023-10-16T09:44:43.302206+0000","last_clean":"2023-10-16T09:44:43.302206+0000","last_became_active":"2023-10-16T09:44:41.714139+0000","last_became_peered":"2023-10-16T09:44:41.714139+0000","last_unstale":"2023-10-16T09:44:43.302206+0000","last_undegraded":"2023-10-16T09:44:43.302206+0000","last_fullsized":"2023-10-16T09:44:43.302206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:20.696205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301376+0000","last_change":"2023-10-16T09:44:41.722778+0000","last_active":"2023-10-16T09:44:43.301376+0000","last_peered":"2023-10-16T09:44:43.301376+0000","last_clean":"2023-10-16T09:44:43.301376+0000","last_became_active":"2023-10-16T09:44:41.722491+0000","last_became_peered":"2023-10-16T09:44:41.722491+0000","last_unstale":"2023-10-16T09:44:43.301376+0000","last_undegraded":"2023-10-16T09:44:43.301376+0000","last_fullsized":"2023-10-16T09:44:43.301376+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:10.486173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823911+0000","last_change":"2023-10-16T09:44:41.711778+0000","last_active":"2023-10-16T09:44:42.823911+0000","last_peered":"2023-10-16T09:44:42.823911+0000","last_clean":"2023-10-16T09:44:42.823911+0000","last_became_active":"2023-10-16T09:44:41.711501+0000","last_became_peered":"2023-10-16T09:44:41.711501+0000","last_unstale":"2023-10-16T09:44:42.823911+0000","last_undegraded":"2023-10-16T09:44:42.823911+0000","last_fullsized":"2023-10-16T09:44:42.823911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:36.319832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823945+0000","last_change":"2023-10-16T09:44:41.711009+0000","last_active":"2023-10-16T09:44:42.823945+0000","last_peered":"2023-10-16T09:44:42.823945+0000","last_clean":"2023-10-16T09:44:42.823945+0000","last_became_active":"2023-10-16T09:44:41.710670+0000","last_became_peered":"2023-10-16T09:44:41.710670+0000","last_unstale":"2023-10-16T09:44:42.823945+0000","last_undegraded":"2023-10-16T09:44:42.823945+0000","last_fullsized":"2023-10-16T09:44:42.823945+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:43.077817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300621+0000","last_change":"2023-10-16T09:44:41.708805+0000","last_active":"2023-10-16T09:44:43.300621+0000","last_peered":"2023-10-16T09:44:43.300621+0000","last_clean":"2023-10-16T09:44:43.300621+0000","last_became_active":"2023-10-16T09:44:41.708534+0000","last_became_peered":"2023-10-16T09:44:41.708534+0000","last_unstale":"2023-10-16T09:44:43.300621+0000","last_undegraded":"2023-10-16T09:44:43.300621+0000","last_fullsized":"2023-10-16T09:44:43.300621+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:10.486173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826195+0000","last_change":"2023-10-16T09:44:41.721814+0000","last_active":"2023-10-16T09:44:42.826195+0000","last_peered":"2023-10-16T09:44:42.826195+0000","last_clean":"2023-10-16T09:44:42.826195+0000","last_became_active":"2023-10-16T09:44:41.721529+0000","last_became_peered":"2023-10-16T09:44:41.721529+0000","last_unstale":"2023-10-16T09:44:42.826195+0000","last_undegraded":"2023-10-16T09:44:42.826195+0000","last_fullsized":"2023-10-16T09:44:42.826195+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:02.867835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301633+0000","last_change":"2023-10-16T09:44:41.726475+0000","last_active":"2023-10-16T09:44:43.301633+0000","last_peered":"2023-10-16T09:44:43.301633+0000","last_clean":"2023-10-16T09:44:43.301633+0000","last_became_active":"2023-10-16T09:44:41.726201+0000","last_became_peered":"2023-10-16T09:44:41.726201+0000","last_unstale":"2023-10-16T09:44:43.301633+0000","last_undegraded":"2023-10-16T09:44:43.301633+0000","last_fullsized":"2023-10-16T09:44:43.301633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:02.867835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302079+0000","last_change":"2023-10-16T09:44:41.716157+0000","last_active":"2023-10-16T09:44:43.302079+0000","last_peered":"2023-10-16T09:44:43.302079+0000","last_clean":"2023-10-16T09:44:43.302079+0000","last_became_active":"2023-10-16T09:44:41.715886+0000","last_became_peered":"2023-10-16T09:44:41.715886+0000","last_unstale":"2023-10-16T09:44:43.302079+0000","last_undegraded":"2023-10-16T09:44:43.302079+0000","last_fullsized":"2023-10-16T09:44:43.302079+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:06.221143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826655+0000","last_change":"2023-10-16T09:44:41.712662+0000","last_active":"2023-10-16T09:44:42.826655+0000","last_peered":"2023-10-16T09:44:42.826655+0000","last_clean":"2023-10-16T09:44:42.826655+0000","last_became_active":"2023-10-16T09:44:41.712272+0000","last_became_peered":"2023-10-16T09:44:41.712272+0000","last_unstale":"2023-10-16T09:44:42.826655+0000","last_undegraded":"2023-10-16T09:44:42.826655+0000","last_fullsized":"2023-10-16T09:44:42.826655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:51.986693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826543+0000","last_change":"2023-10-16T09:44:41.712812+0000","last_active":"2023-10-16T09:44:42.826543+0000","last_peered":"2023-10-16T09:44:42.826543+0000","last_clean":"2023-10-16T09:44:42.826543+0000","last_became_active":"2023-10-16T09:44:41.710569+0000","last_became_peered":"2023-10-16T09:44:41.710569+0000","last_unstale":"2023-10-16T09:44:42.826543+0000","last_undegraded":"2023-10-16T09:44:42.826543+0000","last_fullsized":"2023-10-16T09:44:42.826543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:54.034136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301438+0000","last_change":"2023-10-16T09:44:41.713449+0000","last_active":"2023-10-16T09:44:43.301438+0000","last_peered":"2023-10-16T09:44:43.301438+0000","last_clean":"2023-10-16T09:44:43.301438+0000","last_became_active":"2023-10-16T09:44:41.713171+0000","last_became_peered":"2023-10-16T09:44:41.713171+0000","last_unstale":"2023-10-16T09:44:43.301438+0000","last_undegraded":"2023-10-16T09:44:43.301438+0000","last_fullsized":"2023-10-16T09:44:43.301438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:55.334657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826186+0000","last_change":"2023-10-16T09:44:41.712647+0000","last_active":"2023-10-16T09:44:42.826186+0000","last_peered":"2023-10-16T09:44:42.826186+0000","last_clean":"2023-10-16T09:44:42.826186+0000","last_became_active":"2023-10-16T09:44:41.710678+0000","last_became_peered":"2023-10-16T09:44:41.710678+0000","last_unstale":"2023-10-16T09:44:42.826186+0000","last_undegraded":"2023-10-16T09:44:42.826186+0000","last_fullsized":"2023-10-16T09:44:42.826186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:41.255334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701225+0000","last_change":"2023-10-16T09:44:41.723197+0000","last_active":"2023-10-16T09:44:42.701225+0000","last_peered":"2023-10-16T09:44:42.701225+0000","last_clean":"2023-10-16T09:44:42.701225+0000","last_became_active":"2023-10-16T09:44:41.722914+0000","last_became_peered":"2023-10-16T09:44:41.722914+0000","last_unstale":"2023-10-16T09:44:42.701225+0000","last_undegraded":"2023-10-16T09:44:42.701225+0000","last_fullsized":"2023-10-16T09:44:42.701225+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:12.843916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.825993+0000","last_change":"2023-10-16T09:44:41.721767+0000","last_active":"2023-10-16T09:44:42.825993+0000","last_peered":"2023-10-16T09:44:42.825993+0000","last_clean":"2023-10-16T09:44:42.825993+0000","last_became_active":"2023-10-16T09:44:41.721319+0000","last_became_peered":"2023-10-16T09:44:41.721319+0000","last_unstale":"2023-10-16T09:44:42.825993+0000","last_undegraded":"2023-10-16T09:44:42.825993+0000","last_fullsized":"2023-10-16T09:44:42.825993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:50.971484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699414+0000","last_change":"2023-10-16T09:44:41.716090+0000","last_active":"2023-10-16T09:44:42.699414+0000","last_peered":"2023-10-16T09:44:42.699414+0000","last_clean":"2023-10-16T09:44:42.699414+0000","last_became_active":"2023-10-16T09:44:41.715799+0000","last_became_peered":"2023-10-16T09:44:41.715799+0000","last_unstale":"2023-10-16T09:44:42.699414+0000","last_undegraded":"2023-10-16T09:44:42.699414+0000","last_fullsized":"2023-10-16T09:44:42.699414+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:06.221143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700044+0000","last_change":"2023-10-16T09:44:41.726858+0000","last_active":"2023-10-16T09:44:42.700044+0000","last_peered":"2023-10-16T09:44:42.700044+0000","last_clean":"2023-10-16T09:44:42.700044+0000","last_became_active":"2023-10-16T09:44:41.726362+0000","last_became_peered":"2023-10-16T09:44:41.726362+0000","last_unstale":"2023-10-16T09:44:42.700044+0000","last_undegraded":"2023-10-16T09:44:42.700044+0000","last_fullsized":"2023-10-16T09:44:42.700044+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:02.867835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701622+0000","last_change":"2023-10-16T09:44:41.718454+0000","last_active":"2023-10-16T09:44:42.701622+0000","last_peered":"2023-10-16T09:44:42.701622+0000","last_clean":"2023-10-16T09:44:42.701622+0000","last_became_active":"2023-10-16T09:44:41.718250+0000","last_became_peered":"2023-10-16T09:44:41.718250+0000","last_unstale":"2023-10-16T09:44:42.701622+0000","last_undegraded":"2023-10-16T09:44:42.701622+0000","last_fullsized":"2023-10-16T09:44:42.701622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:34.020959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823949+0000","last_change":"2023-10-16T09:44:41.711102+0000","last_active":"2023-10-16T09:44:42.823949+0000","last_peered":"2023-10-16T09:44:42.823949+0000","last_clean":"2023-10-16T09:44:42.823949+0000","last_became_active":"2023-10-16T09:44:41.710779+0000","last_became_peered":"2023-10-16T09:44:41.710779+0000","last_unstale":"2023-10-16T09:44:42.823949+0000","last_undegraded":"2023-10-16T09:44:42.823949+0000","last_fullsized":"2023-10-16T09:44:42.823949+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:09.602576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701420+0000","last_change":"2023-10-16T09:44:41.711811+0000","last_active":"2023-10-16T09:44:42.701420+0000","last_peered":"2023-10-16T09:44:42.701420+0000","last_clean":"2023-10-16T09:44:42.701420+0000","last_became_active":"2023-10-16T09:44:41.711492+0000","last_became_peered":"2023-10-16T09:44:41.711492+0000","last_unstale":"2023-10-16T09:44:42.701420+0000","last_undegraded":"2023-10-16T09:44:42.701420+0000","last_fullsized":"2023-10-16T09:44:42.701420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:25.433203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826200+0000","last_change":"2023-10-16T09:44:41.718297+0000","last_active":"2023-10-16T09:44:42.826200+0000","last_peered":"2023-10-16T09:44:42.826200+0000","last_clean":"2023-10-16T09:44:42.826200+0000","last_became_active":"2023-10-16T09:44:41.715805+0000","last_became_peered":"2023-10-16T09:44:41.715805+0000","last_unstale":"2023-10-16T09:44:42.826200+0000","last_undegraded":"2023-10-16T09:44:42.826200+0000","last_fullsized":"2023-10-16T09:44:42.826200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:47.928959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700916+0000","last_change":"2023-10-16T09:44:41.714375+0000","last_active":"2023-10-16T09:44:42.700916+0000","last_peered":"2023-10-16T09:44:42.700916+0000","last_clean":"2023-10-16T09:44:42.700916+0000","last_became_active":"2023-10-16T09:44:41.714135+0000","last_became_peered":"2023-10-16T09:44:41.714135+0000","last_unstale":"2023-10-16T09:44:42.700916+0000","last_undegraded":"2023-10-16T09:44:42.700916+0000","last_fullsized":"2023-10-16T09:44:42.700916+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:50.795711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699603+0000","last_change":"2023-10-16T09:44:41.713529+0000","last_active":"2023-10-16T09:44:42.699603+0000","last_peered":"2023-10-16T09:44:42.699603+0000","last_clean":"2023-10-16T09:44:42.699603+0000","last_became_active":"2023-10-16T09:44:41.713092+0000","last_became_peered":"2023-10-16T09:44:41.713092+0000","last_unstale":"2023-10-16T09:44:42.699603+0000","last_undegraded":"2023-10-16T09:44:42.699603+0000","last_fullsized":"2023-10-16T09:44:42.699603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:33.948283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823773+0000","last_change":"2023-10-16T09:44:41.712401+0000","last_active":"2023-10-16T09:44:42.823773+0000","last_peered":"2023-10-16T09:44:42.823773+0000","last_clean":"2023-10-16T09:44:42.823773+0000","last_became_active":"2023-10-16T09:44:41.712015+0000","last_became_peered":"2023-10-16T09:44:41.712015+0000","last_unstale":"2023-10-16T09:44:42.823773+0000","last_undegraded":"2023-10-16T09:44:42.823773+0000","last_fullsized":"2023-10-16T09:44:42.823773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:45:36.417995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701692+0000","last_change":"2023-10-16T09:44:41.713591+0000","last_active":"2023-10-16T09:44:42.701692+0000","last_peered":"2023-10-16T09:44:42.701692+0000","last_clean":"2023-10-16T09:44:42.701692+0000","last_became_active":"2023-10-16T09:44:41.713324+0000","last_became_peered":"2023-10-16T09:44:41.713324+0000","last_unstale":"2023-10-16T09:44:42.701692+0000","last_undegraded":"2023-10-16T09:44:42.701692+0000","last_fullsized":"2023-10-16T09:44:42.701692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:59.353526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301122+0000","last_change":"2023-10-16T09:44:41.716327+0000","last_active":"2023-10-16T09:44:43.301122+0000","last_peered":"2023-10-16T09:44:43.301122+0000","last_clean":"2023-10-16T09:44:43.301122+0000","last_became_active":"2023-10-16T09:44:41.716035+0000","last_became_peered":"2023-10-16T09:44:41.716035+0000","last_unstale":"2023-10-16T09:44:43.301122+0000","last_undegraded":"2023-10-16T09:44:43.301122+0000","last_fullsized":"2023-10-16T09:44:43.301122+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:06.221143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701628+0000","last_change":"2023-10-16T09:44:41.718065+0000","last_active":"2023-10-16T09:44:42.701628+0000","last_peered":"2023-10-16T09:44:42.701628+0000","last_clean":"2023-10-16T09:44:42.701628+0000","last_became_active":"2023-10-16T09:44:41.717899+0000","last_became_peered":"2023-10-16T09:44:41.717899+0000","last_unstale":"2023-10-16T09:44:42.701628+0000","last_undegraded":"2023-10-16T09:44:42.701628+0000","last_fullsized":"2023-10-16T09:44:42.701628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:34:28.701882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826306+0000","last_change":"2023-10-16T09:44:41.725435+0000","last_active":"2023-10-16T09:44:42.826306+0000","last_peered":"2023-10-16T09:44:42.826306+0000","last_clean":"2023-10-16T09:44:42.826306+0000","last_became_active":"2023-10-16T09:44:41.725037+0000","last_became_peered":"2023-10-16T09:44:41.725037+0000","last_unstale":"2023-10-16T09:44:42.826306+0000","last_undegraded":"2023-10-16T09:44:42.826306+0000","last_fullsized":"2023-10-16T09:44:42.826306+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.965241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701735+0000","last_change":"2023-10-16T09:44:41.721031+0000","last_active":"2023-10-16T09:44:42.701735+0000","last_peered":"2023-10-16T09:44:42.701735+0000","last_clean":"2023-10-16T09:44:42.701735+0000","last_became_active":"2023-10-16T09:44:41.720732+0000","last_became_peered":"2023-10-16T09:44:41.720732+0000","last_unstale":"2023-10-16T09:44:42.701735+0000","last_undegraded":"2023-10-16T09:44:42.701735+0000","last_fullsized":"2023-10-16T09:44:42.701735+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:39:08.048466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701430+0000","last_change":"2023-10-16T09:44:41.713476+0000","last_active":"2023-10-16T09:44:42.701430+0000","last_peered":"2023-10-16T09:44:42.701430+0000","last_clean":"2023-10-16T09:44:42.701430+0000","last_became_active":"2023-10-16T09:44:41.713176+0000","last_became_peered":"2023-10-16T09:44:41.713176+0000","last_unstale":"2023-10-16T09:44:42.701430+0000","last_undegraded":"2023-10-16T09:44:42.701430+0000","last_fullsized":"2023-10-16T09:44:42.701430+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:23.401056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.698987+0000","last_change":"2023-10-16T09:44:41.714041+0000","last_active":"2023-10-16T09:44:42.698987+0000","last_peered":"2023-10-16T09:44:42.698987+0000","last_clean":"2023-10-16T09:44:42.698987+0000","last_became_active":"2023-10-16T09:44:41.713753+0000","last_became_peered":"2023-10-16T09:44:41.713753+0000","last_unstale":"2023-10-16T09:44:42.698987+0000","last_undegraded":"2023-10-16T09:44:42.698987+0000","last_fullsized":"2023-10-16T09:44:42.698987+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:10.486173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300591+0000","last_change":"2023-10-16T09:44:41.713988+0000","last_active":"2023-10-16T09:44:43.300591+0000","last_peered":"2023-10-16T09:44:43.300591+0000","last_clean":"2023-10-16T09:44:43.300591+0000","last_became_active":"2023-10-16T09:44:41.713590+0000","last_became_peered":"2023-10-16T09:44:41.713590+0000","last_unstale":"2023-10-16T09:44:43.300591+0000","last_undegraded":"2023-10-16T09:44:43.300591+0000","last_fullsized":"2023-10-16T09:44:43.300591+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.848266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699607+0000","last_change":"2023-10-16T09:44:41.722709+0000","last_active":"2023-10-16T09:44:42.699607+0000","last_peered":"2023-10-16T09:44:42.699607+0000","last_clean":"2023-10-16T09:44:42.699607+0000","last_became_active":"2023-10-16T09:44:41.722451+0000","last_became_peered":"2023-10-16T09:44:41.722451+0000","last_unstale":"2023-10-16T09:44:42.699607+0000","last_undegraded":"2023-10-16T09:44:42.699607+0000","last_fullsized":"2023-10-16T09:44:42.699607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:41.255334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733573+0000","last_change":"2023-10-16T09:44:40.702880+0000","last_active":"2023-10-16T09:44:41.733573+0000","last_peered":"2023-10-16T09:44:41.733573+0000","last_clean":"2023-10-16T09:44:41.733573+0000","last_became_active":"2023-10-16T09:44:40.702634+0000","last_became_peered":"2023-10-16T09:44:40.702634+0000","last_unstale":"2023-10-16T09:44:41.733573+0000","last_undegraded":"2023-10-16T09:44:41.733573+0000","last_fullsized":"2023-10-16T09:44:41.733573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:10.979826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300965+0000","last_change":"2023-10-16T09:44:41.721756+0000","last_active":"2023-10-16T09:44:43.300965+0000","last_peered":"2023-10-16T09:44:43.300965+0000","last_clean":"2023-10-16T09:44:43.300965+0000","last_became_active":"2023-10-16T09:44:41.721482+0000","last_became_peered":"2023-10-16T09:44:41.721482+0000","last_unstale":"2023-10-16T09:44:43.300965+0000","last_undegraded":"2023-10-16T09:44:43.300965+0000","last_fullsized":"2023-10-16T09:44:43.300965+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:55.334657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733855+0000","last_change":"2023-10-16T09:44:40.704794+0000","last_active":"2023-10-16T09:44:41.733855+0000","last_peered":"2023-10-16T09:44:41.733855+0000","last_clean":"2023-10-16T09:44:41.733855+0000","last_became_active":"2023-10-16T09:44:40.704567+0000","last_became_peered":"2023-10-16T09:44:40.704567+0000","last_unstale":"2023-10-16T09:44:41.733855+0000","last_undegraded":"2023-10-16T09:44:41.733855+0000","last_fullsized":"2023-10-16T09:44:41.733855+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:27.249703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700288+0000","last_change":"2023-10-16T09:44:41.711461+0000","last_active":"2023-10-16T09:44:42.700288+0000","last_peered":"2023-10-16T09:44:42.700288+0000","last_clean":"2023-10-16T09:44:42.700288+0000","last_became_active":"2023-10-16T09:44:41.711179+0000","last_became_peered":"2023-10-16T09:44:41.711179+0000","last_unstale":"2023-10-16T09:44:42.700288+0000","last_undegraded":"2023-10-16T09:44:42.700288+0000","last_fullsized":"2023-10-16T09:44:42.700288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:39:08.048466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732996+0000","last_change":"2023-10-16T09:44:40.710707+0000","last_active":"2023-10-16T09:44:41.732996+0000","last_peered":"2023-10-16T09:44:41.732996+0000","last_clean":"2023-10-16T09:44:41.732996+0000","last_became_active":"2023-10-16T09:44:40.710264+0000","last_became_peered":"2023-10-16T09:44:40.710264+0000","last_unstale":"2023-10-16T09:44:41.732996+0000","last_undegraded":"2023-10-16T09:44:41.732996+0000","last_fullsized":"2023-10-16T09:44:41.732996+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:49.791036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302117+0000","last_change":"2023-10-16T09:44:41.722861+0000","last_active":"2023-10-16T09:44:43.302117+0000","last_peered":"2023-10-16T09:44:43.302117+0000","last_clean":"2023-10-16T09:44:43.302117+0000","last_became_active":"2023-10-16T09:44:41.722533+0000","last_became_peered":"2023-10-16T09:44:41.722533+0000","last_unstale":"2023-10-16T09:44:43.302117+0000","last_undegraded":"2023-10-16T09:44:43.302117+0000","last_fullsized":"2023-10-16T09:44:43.302117+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:51.986693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732577+0000","last_change":"2023-10-16T09:44:40.707210+0000","last_active":"2023-10-16T09:44:41.732577+0000","last_peered":"2023-10-16T09:44:41.732577+0000","last_clean":"2023-10-16T09:44:41.732577+0000","last_became_active":"2023-10-16T09:44:40.706931+0000","last_became_peered":"2023-10-16T09:44:40.706931+0000","last_unstale":"2023-10-16T09:44:41.732577+0000","last_undegraded":"2023-10-16T09:44:41.732577+0000","last_fullsized":"2023-10-16T09:44:41.732577+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:22.049243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301259+0000","last_change":"2023-10-16T09:44:41.704334+0000","last_active":"2023-10-16T09:44:43.301259+0000","last_peered":"2023-10-16T09:44:43.301259+0000","last_clean":"2023-10-16T09:44:43.301259+0000","last_became_active":"2023-10-16T09:44:41.703863+0000","last_became_peered":"2023-10-16T09:44:41.703863+0000","last_unstale":"2023-10-16T09:44:43.301259+0000","last_undegraded":"2023-10-16T09:44:43.301259+0000","last_fullsized":"2023-10-16T09:44:43.301259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.848266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733020+0000","last_change":"2023-10-16T09:44:40.698975+0000","last_active":"2023-10-16T09:44:41.733020+0000","last_peered":"2023-10-16T09:44:41.733020+0000","last_clean":"2023-10-16T09:44:41.733020+0000","last_became_active":"2023-10-16T09:44:40.698667+0000","last_became_peered":"2023-10-16T09:44:40.698667+0000","last_unstale":"2023-10-16T09:44:41.733020+0000","last_undegraded":"2023-10-16T09:44:41.733020+0000","last_fullsized":"2023-10-16T09:44:41.733020+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:11.839241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.825986+0000","last_change":"2023-10-16T09:44:41.721594+0000","last_active":"2023-10-16T09:44:42.825986+0000","last_peered":"2023-10-16T09:44:42.825986+0000","last_clean":"2023-10-16T09:44:42.825986+0000","last_became_active":"2023-10-16T09:44:41.721220+0000","last_became_peered":"2023-10-16T09:44:41.721220+0000","last_unstale":"2023-10-16T09:44:42.825986+0000","last_undegraded":"2023-10-16T09:44:42.825986+0000","last_fullsized":"2023-10-16T09:44:42.825986+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:33.948283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732515+0000","last_change":"2023-10-16T09:44:40.705524+0000","last_active":"2023-10-16T09:44:41.732515+0000","last_peered":"2023-10-16T09:44:41.732515+0000","last_clean":"2023-10-16T09:44:41.732515+0000","last_became_active":"2023-10-16T09:44:40.705221+0000","last_became_peered":"2023-10-16T09:44:40.705221+0000","last_unstale":"2023-10-16T09:44:41.732515+0000","last_undegraded":"2023-10-16T09:44:41.732515+0000","last_fullsized":"2023-10-16T09:44:41.732515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:22.396380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700911+0000","last_change":"2023-10-16T09:44:41.721767+0000","last_active":"2023-10-16T09:44:42.700911+0000","last_peered":"2023-10-16T09:44:42.700911+0000","last_clean":"2023-10-16T09:44:42.700911+0000","last_became_active":"2023-10-16T09:44:41.721473+0000","last_became_peered":"2023-10-16T09:44:41.721473+0000","last_unstale":"2023-10-16T09:44:42.700911+0000","last_undegraded":"2023-10-16T09:44:42.700911+0000","last_fullsized":"2023-10-16T09:44:42.700911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:04.467315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701396+0000","last_change":"2023-10-16T09:44:40.704004+0000","last_active":"2023-10-16T09:44:41.701396+0000","last_peered":"2023-10-16T09:44:41.701396+0000","last_clean":"2023-10-16T09:44:41.701396+0000","last_became_active":"2023-10-16T09:44:40.703251+0000","last_became_peered":"2023-10-16T09:44:40.703251+0000","last_unstale":"2023-10-16T09:44:41.701396+0000","last_undegraded":"2023-10-16T09:44:41.701396+0000","last_fullsized":"2023-10-16T09:44:41.701396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:03.462640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301858+0000","last_change":"2023-10-16T09:44:41.711239+0000","last_active":"2023-10-16T09:44:43.301858+0000","last_peered":"2023-10-16T09:44:43.301858+0000","last_clean":"2023-10-16T09:44:43.301858+0000","last_became_active":"2023-10-16T09:44:41.711053+0000","last_became_peered":"2023-10-16T09:44:41.711053+0000","last_unstale":"2023-10-16T09:44:43.301858+0000","last_undegraded":"2023-10-16T09:44:43.301858+0000","last_fullsized":"2023-10-16T09:44:43.301858+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:54.034136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.703133+0000","last_change":"2023-10-16T09:44:40.701904+0000","last_active":"2023-10-16T09:44:41.703133+0000","last_peered":"2023-10-16T09:44:41.703133+0000","last_clean":"2023-10-16T09:44:41.703133+0000","last_became_active":"2023-10-16T09:44:40.701600+0000","last_became_peered":"2023-10-16T09:44:40.701600+0000","last_unstale":"2023-10-16T09:44:41.703133+0000","last_undegraded":"2023-10-16T09:44:41.703133+0000","last_fullsized":"2023-10-16T09:44:41.703133+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:17.296825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823915+0000","last_change":"2023-10-16T09:44:41.711297+0000","last_active":"2023-10-16T09:44:42.823915+0000","last_peered":"2023-10-16T09:44:42.823915+0000","last_clean":"2023-10-16T09:44:42.823915+0000","last_became_active":"2023-10-16T09:44:41.711030+0000","last_became_peered":"2023-10-16T09:44:41.711030+0000","last_unstale":"2023-10-16T09:44:42.823915+0000","last_undegraded":"2023-10-16T09:44:42.823915+0000","last_fullsized":"2023-10-16T09:44:42.823915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:37:15.717236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.708552+0000","last_change":"2023-10-16T09:44:40.710741+0000","last_active":"2023-10-16T09:44:41.708552+0000","last_peered":"2023-10-16T09:44:41.708552+0000","last_clean":"2023-10-16T09:44:41.708552+0000","last_became_active":"2023-10-16T09:44:40.710468+0000","last_became_peered":"2023-10-16T09:44:40.710468+0000","last_unstale":"2023-10-16T09:44:41.708552+0000","last_undegraded":"2023-10-16T09:44:41.708552+0000","last_fullsized":"2023-10-16T09:44:41.708552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:49.966809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700618+0000","last_change":"2023-10-16T09:44:41.722862+0000","last_active":"2023-10-16T09:44:42.700618+0000","last_peered":"2023-10-16T09:44:42.700618+0000","last_clean":"2023-10-16T09:44:42.700618+0000","last_became_active":"2023-10-16T09:44:41.722611+0000","last_became_peered":"2023-10-16T09:44:41.722611+0000","last_unstale":"2023-10-16T09:44:42.700618+0000","last_undegraded":"2023-10-16T09:44:42.700618+0000","last_fullsized":"2023-10-16T09:44:42.700618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:16.187818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.710537+0000","last_change":"2023-10-16T09:44:40.704708+0000","last_active":"2023-10-16T09:44:41.710537+0000","last_peered":"2023-10-16T09:44:41.710537+0000","last_clean":"2023-10-16T09:44:41.710537+0000","last_became_active":"2023-10-16T09:44:40.704330+0000","last_became_peered":"2023-10-16T09:44:40.704330+0000","last_unstale":"2023-10-16T09:44:41.710537+0000","last_undegraded":"2023-10-16T09:44:41.710537+0000","last_fullsized":"2023-10-16T09:44:41.710537+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:30.548816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301378+0000","last_change":"2023-10-16T09:44:41.711025+0000","last_active":"2023-10-16T09:44:43.301378+0000","last_peered":"2023-10-16T09:44:43.301378+0000","last_clean":"2023-10-16T09:44:43.301378+0000","last_became_active":"2023-10-16T09:44:41.710739+0000","last_became_peered":"2023-10-16T09:44:41.710739+0000","last_unstale":"2023-10-16T09:44:43.301378+0000","last_undegraded":"2023-10-16T09:44:43.301378+0000","last_fullsized":"2023-10-16T09:44:43.301378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:31.553491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733396+0000","last_change":"2023-10-16T09:44:40.710791+0000","last_active":"2023-10-16T09:44:41.733396+0000","last_peered":"2023-10-16T09:44:41.733396+0000","last_clean":"2023-10-16T09:44:41.733396+0000","last_became_active":"2023-10-16T09:44:40.710491+0000","last_became_peered":"2023-10-16T09:44:40.710491+0000","last_unstale":"2023-10-16T09:44:41.733396+0000","last_undegraded":"2023-10-16T09:44:41.733396+0000","last_fullsized":"2023-10-16T09:44:41.733396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:09.481497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700080+0000","last_change":"2023-10-16T09:44:41.713551+0000","last_active":"2023-10-16T09:44:42.700080+0000","last_peered":"2023-10-16T09:44:42.700080+0000","last_clean":"2023-10-16T09:44:42.700080+0000","last_became_active":"2023-10-16T09:44:41.713270+0000","last_became_peered":"2023-10-16T09:44:41.713270+0000","last_unstale":"2023-10-16T09:44:42.700080+0000","last_undegraded":"2023-10-16T09:44:42.700080+0000","last_fullsized":"2023-10-16T09:44:42.700080+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:59.353526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.735080+0000","last_change":"2023-10-16T09:44:40.709967+0000","last_active":"2023-10-16T09:44:41.735080+0000","last_peered":"2023-10-16T09:44:41.735080+0000","last_clean":"2023-10-16T09:44:41.735080+0000","last_became_active":"2023-10-16T09:44:40.709745+0000","last_became_peered":"2023-10-16T09:44:40.709745+0000","last_unstale":"2023-10-16T09:44:41.735080+0000","last_undegraded":"2023-10-16T09:44:41.735080+0000","last_fullsized":"2023-10-16T09:44:41.735080+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:09.481497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301010+0000","last_change":"2023-10-16T09:44:41.715601+0000","last_active":"2023-10-16T09:44:43.301010+0000","last_peered":"2023-10-16T09:44:43.301010+0000","last_clean":"2023-10-16T09:44:43.301010+0000","last_became_active":"2023-10-16T09:44:41.715332+0000","last_became_peered":"2023-10-16T09:44:41.715332+0000","last_unstale":"2023-10-16T09:44:43.301010+0000","last_undegraded":"2023-10-16T09:44:43.301010+0000","last_fullsized":"2023-10-16T09:44:43.301010+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:04.467315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.731405+0000","last_change":"2023-10-16T09:44:40.706782+0000","last_active":"2023-10-16T09:44:41.731405+0000","last_peered":"2023-10-16T09:44:41.731405+0000","last_clean":"2023-10-16T09:44:41.731405+0000","last_became_active":"2023-10-16T09:44:40.706515+0000","last_became_peered":"2023-10-16T09:44:40.706515+0000","last_unstale":"2023-10-16T09:44:41.731405+0000","last_undegraded":"2023-10-16T09:44:41.731405+0000","last_fullsized":"2023-10-16T09:44:41.731405+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:30.548816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301679+0000","last_change":"2023-10-16T09:44:41.712044+0000","last_active":"2023-10-16T09:44:43.301679+0000","last_peered":"2023-10-16T09:44:43.301679+0000","last_clean":"2023-10-16T09:44:43.301679+0000","last_became_active":"2023-10-16T09:44:41.711733+0000","last_became_peered":"2023-10-16T09:44:41.711733+0000","last_unstale":"2023-10-16T09:44:43.301679+0000","last_undegraded":"2023-10-16T09:44:43.301679+0000","last_fullsized":"2023-10-16T09:44:43.301679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:16.187818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700855+0000","last_change":"2023-10-16T09:44:40.704624+0000","last_active":"2023-10-16T09:44:41.700855+0000","last_peered":"2023-10-16T09:44:41.700855+0000","last_clean":"2023-10-16T09:44:41.700855+0000","last_became_active":"2023-10-16T09:44:40.704319+0000","last_became_peered":"2023-10-16T09:44:40.704319+0000","last_unstale":"2023-10-16T09:44:41.700855+0000","last_undegraded":"2023-10-16T09:44:41.700855+0000","last_fullsized":"2023-10-16T09:44:41.700855+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:22.396380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699259+0000","last_change":"2023-10-16T09:44:41.713526+0000","last_active":"2023-10-16T09:44:42.699259+0000","last_peered":"2023-10-16T09:44:42.699259+0000","last_clean":"2023-10-16T09:44:42.699259+0000","last_became_active":"2023-10-16T09:44:41.713235+0000","last_became_peered":"2023-10-16T09:44:41.713235+0000","last_unstale":"2023-10-16T09:44:42.699259+0000","last_undegraded":"2023-10-16T09:44:42.699259+0000","last_fullsized":"2023-10-16T09:44:42.699259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:55.334657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.709763+0000","last_change":"2023-10-16T09:44:40.706416+0000","last_active":"2023-10-16T09:44:41.709763+0000","last_peered":"2023-10-16T09:44:41.709763+0000","last_clean":"2023-10-16T09:44:41.709763+0000","last_became_active":"2023-10-16T09:44:40.706121+0000","last_became_peered":"2023-10-16T09:44:40.706121+0000","last_unstale":"2023-10-16T09:44:41.709763+0000","last_undegraded":"2023-10-16T09:44:41.709763+0000","last_fullsized":"2023-10-16T09:44:41.709763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:22.049243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301719+0000","last_change":"2023-10-16T09:44:41.712153+0000","last_active":"2023-10-16T09:44:43.301719+0000","last_peered":"2023-10-16T09:44:43.301719+0000","last_clean":"2023-10-16T09:44:43.301719+0000","last_became_active":"2023-10-16T09:44:41.711872+0000","last_became_peered":"2023-10-16T09:44:41.711872+0000","last_unstale":"2023-10-16T09:44:43.301719+0000","last_undegraded":"2023-10-16T09:44:43.301719+0000","last_fullsized":"2023-10-16T09:44:43.301719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:47.928959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733928+0000","last_change":"2023-10-16T09:44:40.708822+0000","last_active":"2023-10-16T09:44:41.733928+0000","last_peered":"2023-10-16T09:44:41.733928+0000","last_clean":"2023-10-16T09:44:41.733928+0000","last_became_active":"2023-10-16T09:44:40.708413+0000","last_became_peered":"2023-10-16T09:44:40.708413+0000","last_unstale":"2023-10-16T09:44:41.733928+0000","last_undegraded":"2023-10-16T09:44:41.733928+0000","last_fullsized":"2023-10-16T09:44:41.733928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:49.791036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301796+0000","last_change":"2023-10-16T09:44:41.722908+0000","last_active":"2023-10-16T09:44:43.301796+0000","last_peered":"2023-10-16T09:44:43.301796+0000","last_clean":"2023-10-16T09:44:43.301796+0000","last_became_active":"2023-10-16T09:44:41.722590+0000","last_became_peered":"2023-10-16T09:44:41.722590+0000","last_unstale":"2023-10-16T09:44:43.301796+0000","last_undegraded":"2023-10-16T09:44:43.301796+0000","last_fullsized":"2023-10-16T09:44:43.301796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.965241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733019+0000","last_change":"2023-10-16T09:44:40.702783+0000","last_active":"2023-10-16T09:44:41.733019+0000","last_peered":"2023-10-16T09:44:41.733019+0000","last_clean":"2023-10-16T09:44:41.733019+0000","last_became_active":"2023-10-16T09:44:40.702513+0000","last_became_peered":"2023-10-16T09:44:40.702513+0000","last_unstale":"2023-10-16T09:44:41.733019+0000","last_undegraded":"2023-10-16T09:44:41.733019+0000","last_fullsized":"2023-10-16T09:44:41.733019+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:54.329982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826464+0000","last_change":"2023-10-16T09:44:41.715529+0000","last_active":"2023-10-16T09:44:42.826464+0000","last_peered":"2023-10-16T09:44:42.826464+0000","last_clean":"2023-10-16T09:44:42.826464+0000","last_became_active":"2023-10-16T09:44:41.715257+0000","last_became_peered":"2023-10-16T09:44:41.715257+0000","last_unstale":"2023-10-16T09:44:42.826464+0000","last_undegraded":"2023-10-16T09:44:42.826464+0000","last_fullsized":"2023-10-16T09:44:42.826464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:57.428175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.733510+0000","last_change":"2023-10-16T09:44:40.702916+0000","last_active":"2023-10-16T09:44:41.733510+0000","last_peered":"2023-10-16T09:44:41.733510+0000","last_clean":"2023-10-16T09:44:41.733510+0000","last_became_active":"2023-10-16T09:44:40.702630+0000","last_became_peered":"2023-10-16T09:44:40.702630+0000","last_unstale":"2023-10-16T09:44:41.733510+0000","last_undegraded":"2023-10-16T09:44:41.733510+0000","last_fullsized":"2023-10-16T09:44:41.733510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:19:22.396380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700405+0000","last_change":"2023-10-16T09:44:41.707197+0000","last_active":"2023-10-16T09:44:42.700405+0000","last_peered":"2023-10-16T09:44:42.700405+0000","last_clean":"2023-10-16T09:44:42.700405+0000","last_became_active":"2023-10-16T09:44:41.706555+0000","last_became_peered":"2023-10-16T09:44:41.706555+0000","last_unstale":"2023-10-16T09:44:42.700405+0000","last_undegraded":"2023-10-16T09:44:42.700405+0000","last_fullsized":"2023-10-16T09:44:42.700405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:28.254378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.729255+0000","last_change":"2023-10-16T09:44:40.705548+0000","last_active":"2023-10-16T09:44:41.729255+0000","last_peered":"2023-10-16T09:44:41.729255+0000","last_clean":"2023-10-16T09:44:41.729255+0000","last_became_active":"2023-10-16T09:44:40.705254+0000","last_became_peered":"2023-10-16T09:44:40.705254+0000","last_unstale":"2023-10-16T09:44:41.729255+0000","last_undegraded":"2023-10-16T09:44:41.729255+0000","last_fullsized":"2023-10-16T09:44:41.729255+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:51.383595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823954+0000","last_change":"2023-10-16T09:44:41.714436+0000","last_active":"2023-10-16T09:44:42.823954+0000","last_peered":"2023-10-16T09:44:42.823954+0000","last_clean":"2023-10-16T09:44:42.823954+0000","last_became_active":"2023-10-16T09:44:41.714142+0000","last_became_peered":"2023-10-16T09:44:41.714142+0000","last_unstale":"2023-10-16T09:44:42.823954+0000","last_undegraded":"2023-10-16T09:44:42.823954+0000","last_fullsized":"2023-10-16T09:44:42.823954+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:05.899379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701238+0000","last_change":"2023-10-16T09:44:40.703548+0000","last_active":"2023-10-16T09:44:41.701238+0000","last_peered":"2023-10-16T09:44:41.701238+0000","last_clean":"2023-10-16T09:44:41.701238+0000","last_became_active":"2023-10-16T09:44:40.703299+0000","last_became_peered":"2023-10-16T09:44:40.703299+0000","last_unstale":"2023-10-16T09:44:41.701238+0000","last_undegraded":"2023-10-16T09:44:41.701238+0000","last_fullsized":"2023-10-16T09:44:41.701238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:27.249703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699135+0000","last_change":"2023-10-16T09:44:41.713675+0000","last_active":"2023-10-16T09:44:42.699135+0000","last_peered":"2023-10-16T09:44:42.699135+0000","last_clean":"2023-10-16T09:44:42.699135+0000","last_became_active":"2023-10-16T09:44:41.713321+0000","last_became_peered":"2023-10-16T09:44:41.713321+0000","last_unstale":"2023-10-16T09:44:42.699135+0000","last_undegraded":"2023-10-16T09:44:42.699135+0000","last_fullsized":"2023-10-16T09:44:42.699135+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:31.553491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.729125+0000","last_change":"2023-10-16T09:44:40.702817+0000","last_active":"2023-10-16T09:44:41.729125+0000","last_peered":"2023-10-16T09:44:41.729125+0000","last_clean":"2023-10-16T09:44:41.729125+0000","last_became_active":"2023-10-16T09:44:40.700380+0000","last_became_peered":"2023-10-16T09:44:40.700380+0000","last_unstale":"2023-10-16T09:44:41.729125+0000","last_undegraded":"2023-10-16T09:44:41.729125+0000","last_fullsized":"2023-10-16T09:44:41.729125+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:39.230846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301084+0000","last_change":"2023-10-16T09:44:41.712532+0000","last_active":"2023-10-16T09:44:43.301084+0000","last_peered":"2023-10-16T09:44:43.301084+0000","last_clean":"2023-10-16T09:44:43.301084+0000","last_became_active":"2023-10-16T09:44:41.712280+0000","last_became_peered":"2023-10-16T09:44:41.712280+0000","last_unstale":"2023-10-16T09:44:43.301084+0000","last_undegraded":"2023-10-16T09:44:43.301084+0000","last_fullsized":"2023-10-16T09:44:43.301084+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:50.795711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700031+0000","last_change":"2023-10-16T09:44:40.702242+0000","last_active":"2023-10-16T09:44:41.700031+0000","last_peered":"2023-10-16T09:44:41.700031+0000","last_clean":"2023-10-16T09:44:41.700031+0000","last_became_active":"2023-10-16T09:44:40.702008+0000","last_became_peered":"2023-10-16T09:44:40.702008+0000","last_unstale":"2023-10-16T09:44:41.700031+0000","last_undegraded":"2023-10-16T09:44:41.700031+0000","last_fullsized":"2023-10-16T09:44:41.700031+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:15:13.705856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826207+0000","last_change":"2023-10-16T09:44:41.725281+0000","last_active":"2023-10-16T09:44:42.826207+0000","last_peered":"2023-10-16T09:44:42.826207+0000","last_clean":"2023-10-16T09:44:42.826207+0000","last_became_active":"2023-10-16T09:44:41.724987+0000","last_became_peered":"2023-10-16T09:44:41.724987+0000","last_unstale":"2023-10-16T09:44:42.826207+0000","last_undegraded":"2023-10-16T09:44:42.826207+0000","last_fullsized":"2023-10-16T09:44:42.826207+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:16.187818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701536+0000","last_change":"2023-10-16T09:44:40.708773+0000","last_active":"2023-10-16T09:44:41.701536+0000","last_peered":"2023-10-16T09:44:41.701536+0000","last_clean":"2023-10-16T09:44:41.701536+0000","last_became_active":"2023-10-16T09:44:40.708467+0000","last_became_peered":"2023-10-16T09:44:40.708467+0000","last_unstale":"2023-10-16T09:44:41.701536+0000","last_undegraded":"2023-10-16T09:44:41.701536+0000","last_fullsized":"2023-10-16T09:44:41.701536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:11.839241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301805+0000","last_change":"2023-10-16T09:44:41.711587+0000","last_active":"2023-10-16T09:44:43.301805+0000","last_peered":"2023-10-16T09:44:43.301805+0000","last_clean":"2023-10-16T09:44:43.301805+0000","last_became_active":"2023-10-16T09:44:41.711310+0000","last_became_peered":"2023-10-16T09:44:41.711310+0000","last_unstale":"2023-10-16T09:44:43.301805+0000","last_undegraded":"2023-10-16T09:44:43.301805+0000","last_fullsized":"2023-10-16T09:44:43.301805+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:57.428175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701467+0000","last_change":"2023-10-16T09:44:40.700966+0000","last_active":"2023-10-16T09:44:41.701467+0000","last_peered":"2023-10-16T09:44:41.701467+0000","last_clean":"2023-10-16T09:44:41.701467+0000","last_became_active":"2023-10-16T09:44:40.700697+0000","last_became_peered":"2023-10-16T09:44:40.700697+0000","last_unstale":"2023-10-16T09:44:41.701467+0000","last_undegraded":"2023-10-16T09:44:41.701467+0000","last_fullsized":"2023-10-16T09:44:41.701467+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:48.054971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300819+0000","last_change":"2023-10-16T09:44:41.727134+0000","last_active":"2023-10-16T09:44:43.300819+0000","last_peered":"2023-10-16T09:44:43.300819+0000","last_clean":"2023-10-16T09:44:43.300819+0000","last_became_active":"2023-10-16T09:44:41.726887+0000","last_became_peered":"2023-10-16T09:44:41.726887+0000","last_unstale":"2023-10-16T09:44:43.300819+0000","last_undegraded":"2023-10-16T09:44:43.300819+0000","last_fullsized":"2023-10-16T09:44:43.300819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:31.553491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.732941+0000","last_change":"2023-10-16T09:44:40.703790+0000","last_active":"2023-10-16T09:44:41.732941+0000","last_peered":"2023-10-16T09:44:41.732941+0000","last_clean":"2023-10-16T09:44:41.732941+0000","last_became_active":"2023-10-16T09:44:40.703520+0000","last_became_peered":"2023-10-16T09:44:40.703520+0000","last_unstale":"2023-10-16T09:44:41.732941+0000","last_undegraded":"2023-10-16T09:44:41.732941+0000","last_fullsized":"2023-10-16T09:44:41.732941+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:05.216468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.700677+0000","last_change":"2023-10-16T09:44:41.717136+0000","last_active":"2023-10-16T09:44:42.700677+0000","last_peered":"2023-10-16T09:44:42.700677+0000","last_clean":"2023-10-16T09:44:42.700677+0000","last_became_active":"2023-10-16T09:44:41.716885+0000","last_became_peered":"2023-10-16T09:44:41.716885+0000","last_unstale":"2023-10-16T09:44:42.700677+0000","last_undegraded":"2023-10-16T09:44:42.700677+0000","last_fullsized":"2023-10-16T09:44:42.700677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:47.928959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.823864+0000","last_change":"2023-10-16T09:44:40.702669+0000","last_active":"2023-10-16T09:44:42.823864+0000","last_peered":"2023-10-16T09:44:42.823864+0000","last_clean":"2023-10-16T09:44:42.823864+0000","last_became_active":"2023-10-16T09:44:40.702410+0000","last_became_peered":"2023-10-16T09:44:40.702410+0000","last_unstale":"2023-10-16T09:44:42.823864+0000","last_undegraded":"2023-10-16T09:44:42.823864+0000","last_fullsized":"2023-10-16T09:44:42.823864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:15.630799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.302324+0000","last_change":"2023-10-16T09:44:41.712832+0000","last_active":"2023-10-16T09:44:43.302324+0000","last_peered":"2023-10-16T09:44:43.302324+0000","last_clean":"2023-10-16T09:44:43.302324+0000","last_became_active":"2023-10-16T09:44:41.711860+0000","last_became_peered":"2023-10-16T09:44:41.711860+0000","last_unstale":"2023-10-16T09:44:43.302324+0000","last_undegraded":"2023-10-16T09:44:43.302324+0000","last_fullsized":"2023-10-16T09:44:43.302324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:08:34.357566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700121+0000","last_change":"2023-10-16T09:44:40.703799+0000","last_active":"2023-10-16T09:44:41.700121+0000","last_peered":"2023-10-16T09:44:41.700121+0000","last_clean":"2023-10-16T09:44:41.700121+0000","last_became_active":"2023-10-16T09:44:40.703433+0000","last_became_peered":"2023-10-16T09:44:40.703433+0000","last_unstale":"2023-10-16T09:44:41.700121+0000","last_undegraded":"2023-10-16T09:44:41.700121+0000","last_fullsized":"2023-10-16T09:44:41.700121+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:17.296825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.301345+0000","last_change":"2023-10-16T09:44:41.709494+0000","last_active":"2023-10-16T09:44:43.301345+0000","last_peered":"2023-10-16T09:44:43.301345+0000","last_clean":"2023-10-16T09:44:43.301345+0000","last_became_active":"2023-10-16T09:44:41.709228+0000","last_became_peered":"2023-10-16T09:44:41.709228+0000","last_unstale":"2023-10-16T09:44:43.301345+0000","last_undegraded":"2023-10-16T09:44:43.301345+0000","last_fullsized":"2023-10-16T09:44:43.301345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:12.843916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.699909+0000","last_change":"2023-10-16T09:44:40.704621+0000","last_active":"2023-10-16T09:44:41.699909+0000","last_peered":"2023-10-16T09:44:41.699909+0000","last_clean":"2023-10-16T09:44:41.699909+0000","last_became_active":"2023-10-16T09:44:40.703625+0000","last_became_peered":"2023-10-16T09:44:40.703625+0000","last_unstale":"2023-10-16T09:44:41.699909+0000","last_undegraded":"2023-10-16T09:44:41.699909+0000","last_fullsized":"2023-10-16T09:44:41.699909+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:10.979826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:43.300684+0000","last_change":"2023-10-16T09:44:41.710756+0000","last_active":"2023-10-16T09:44:43.300684+0000","last_peered":"2023-10-16T09:44:43.300684+0000","last_clean":"2023-10-16T09:44:43.300684+0000","last_became_active":"2023-10-16T09:44:41.710489+0000","last_became_peered":"2023-10-16T09:44:41.710489+0000","last_unstale":"2023-10-16T09:44:43.300684+0000","last_undegraded":"2023-10-16T09:44:43.300684+0000","last_fullsized":"2023-10-16T09:44:43.300684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:12.843916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.710491+0000","last_change":"2023-10-16T09:44:40.700385+0000","last_active":"2023-10-16T09:44:41.710491+0000","last_peered":"2023-10-16T09:44:41.710491+0000","last_clean":"2023-10-16T09:44:41.710491+0000","last_became_active":"2023-10-16T09:44:40.700118+0000","last_became_peered":"2023-10-16T09:44:40.700118+0000","last_unstale":"2023-10-16T09:44:41.710491+0000","last_undegraded":"2023-10-16T09:44:41.710491+0000","last_fullsized":"2023-10-16T09:44:41.710491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:25.169653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.701067+0000","last_change":"2023-10-16T09:44:41.712317+0000","last_active":"2023-10-16T09:44:42.701067+0000","last_peered":"2023-10-16T09:44:42.701067+0000","last_clean":"2023-10-16T09:44:42.701067+0000","last_became_active":"2023-10-16T09:44:41.712046+0000","last_became_peered":"2023-10-16T09:44:41.712046+0000","last_unstale":"2023-10-16T09:44:42.701067+0000","last_undegraded":"2023-10-16T09:44:42.701067+0000","last_fullsized":"2023-10-16T09:44:42.701067+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.848266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.702645+0000","last_change":"2023-10-16T09:44:40.706560+0000","last_active":"2023-10-16T09:44:41.702645+0000","last_peered":"2023-10-16T09:44:41.702645+0000","last_clean":"2023-10-16T09:44:41.702645+0000","last_became_active":"2023-10-16T09:44:40.706126+0000","last_became_peered":"2023-10-16T09:44:40.706126+0000","last_unstale":"2023-10-16T09:44:41.702645+0000","last_undegraded":"2023-10-16T09:44:41.702645+0000","last_fullsized":"2023-10-16T09:44:41.702645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:35.480015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.699880+0000","last_change":"2023-10-16T09:44:41.721849+0000","last_active":"2023-10-16T09:44:42.699880+0000","last_peered":"2023-10-16T09:44:42.699880+0000","last_clean":"2023-10-16T09:44:42.699880+0000","last_became_active":"2023-10-16T09:44:41.721581+0000","last_became_peered":"2023-10-16T09:44:41.721581+0000","last_unstale":"2023-10-16T09:44:42.699880+0000","last_undegraded":"2023-10-16T09:44:42.699880+0000","last_fullsized":"2023-10-16T09:44:42.699880+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:50.971484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.707272+0000","last_change":"2023-10-16T09:44:40.710752+0000","last_active":"2023-10-16T09:44:41.707272+0000","last_peered":"2023-10-16T09:44:41.707272+0000","last_clean":"2023-10-16T09:44:41.707272+0000","last_became_active":"2023-10-16T09:44:40.710456+0000","last_became_peered":"2023-10-16T09:44:40.710456+0000","last_unstale":"2023-10-16T09:44:41.707272+0000","last_undegraded":"2023-10-16T09:44:41.707272+0000","last_fullsized":"2023-10-16T09:44:41.707272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:54.329982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T09:44:42.826708+0000","last_change":"2023-10-16T09:44:41.714926+0000","last_active":"2023-10-16T09:44:42.826708+0000","last_peered":"2023-10-16T09:44:42.826708+0000","last_clean":"2023-10-16T09:44:42.826708+0000","last_became_active":"2023-10-16T09:44:41.714681+0000","last_became_peered":"2023-10-16T09:44:41.714681+0000","last_unstale":"2023-10-16T09:44:42.826708+0000","last_undegraded":"2023-10-16T09:44:42.826708+0000","last_fullsized":"2023-10-16T09:44:42.826708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:40.683025+0000","last_clean_scrub_stamp":"2023-10-16T09:44:40.683025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:20.696205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.700514+0000","last_change":"2023-10-16T09:44:40.705265+0000","last_active":"2023-10-16T09:44:41.700514+0000","last_peered":"2023-10-16T09:44:41.700514+0000","last_clean":"2023-10-16T09:44:41.700514+0000","last_became_active":"2023-10-16T09:44:40.704988+0000","last_became_peered":"2023-10-16T09:44:40.704988+0000","last_unstale":"2023-10-16T09:44:41.700514+0000","last_undegraded":"2023-10-16T09:44:41.700514+0000","last_fullsized":"2023-10-16T09:44:41.700514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:44:39.678349+0000","last_clean_scrub_stamp":"2023-10-16T09:44:39.678349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:24.428528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T09:44:41.701207+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:44:41.701207+0000","last_peered":"2023-10-16T09:44:41.701207+0000","last_clean":"2023-10-16T09:44:41.701207+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:44:41.701207+0000","last_undegraded":"2023-10-16T09:44:41.701207+0000","last_fullsized":"2023-10-16T09:44:41.701207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":787616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":11.246,"5min":11.246,"15min":11.246},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.39800000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.379},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.442}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":196148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.435}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.496}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":195656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.255},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.28799999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.1760000000000002,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Oct 16 09:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.76500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":785968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.5}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53300000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":195636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.376},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.32300000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.26700000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.51300000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47099999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.45900000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":197212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.20999999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.313},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42399999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.434},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":196170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59599999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.625},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":195678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:44:50.817 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T09:44:50.817 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T09:44:50.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.977+0000 7f574c637700 1 Processor -- start 2023-10-16T09:44:50.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.977+0000 7f574c637700 1 -- start start 2023-10-16T09:44:50.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f574c637700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5744072b60 0x7f574406d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f574c637700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f574406d960 0x7f57440b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f574c637700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440b5040 con 0x7f574415a780 2023-10-16T09:44:50.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f574c637700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574406de10 con 0x7f5744072b60 2023-10-16T09:44:50.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f574c637700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574415e190 con 0x7f574406d960 2023-10-16T09:44:50.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f5749bd2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5744072b60 0x7f574406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f5749bd2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5744072b60 0x7f574406d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48570/0 (socket says 172.21.15.158:48570) 2023-10-16T09:44:50.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f5749bd2700 1 -- 172.21.15.158:0/1082773734 learned_addr learned my addr 172.21.15.158:0/1082773734 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f574abd4700 1 --2- 172.21.15.158:0/1082773734 >> v2:172.21.15.172:3300/0 conn(0x7f574406d960 0x7f57440b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f5749bd2700 1 -- 172.21.15.158:0/1082773734 >> v2:172.21.15.172:3300/0 conn(0x7f574406d960 msgr2=0x7f57440b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f5749bd2700 1 --2- 172.21.15.158:0/1082773734 >> v2:172.21.15.172:3300/0 conn(0x7f574406d960 0x7f57440b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f5749bd2700 1 -- 172.21.15.158:0/1082773734 >> v1:172.21.15.172:6789/0 conn(0x7f574415a780 legacy=0x7f57440b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.978+0000 7f5749bd2700 1 -- 172.21.15.158:0/1082773734 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574415e470 con 0x7f5744072b60 2023-10-16T09:44:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f5749bd2700 1 --2- 172.21.15.158:0/1082773734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5744072b60 0x7f574406d420 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f574406d2a0 tx=0x7f5740009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c908eca82db1bf43 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f57493d1700 1 -- 172.21.15.158:0/1082773734 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740011830 con 0x7f5744072b60 2023-10-16T09:44:50.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f57493d1700 1 -- 172.21.15.158:0/1082773734 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5740011990 con 0x7f5744072b60 2023-10-16T09:44:50.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f57493d1700 1 -- 172.21.15.158:0/1082773734 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740011d20 con 0x7f5744072b60 2023-10-16T09:44:50.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 -- 172.21.15.158:0/1082773734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5744072b60 msgr2=0x7f574406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 --2- 172.21.15.158:0/1082773734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5744072b60 0x7f574406d420 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f574406d2a0 tx=0x7f5740009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 -- 172.21.15.158:0/1082773734 shutdown_connections 2023-10-16T09:44:50.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 --2- 172.21.15.158:0/1082773734 >> v2:172.21.15.172:3300/0 conn(0x7f574406d960 0x7f57440b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 --2- 172.21.15.158:0/1082773734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5744072b60 0x7f574406d420 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 -- 172.21.15.158:0/1082773734 >> 172.21.15.158:0/1082773734 conn(0x7f57440ccd30 msgr2=0x7f57440cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:50.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 -- 172.21.15.158:0/1082773734 shutdown_connections 2023-10-16T09:44:50.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.979+0000 7f574c637700 1 -- 172.21.15.158:0/1082773734 wait complete. 2023-10-16T09:44:50.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 Processor -- start 2023-10-16T09:44:50.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 -- start start 2023-10-16T09:44:50.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f574406d960 0x7f5744150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5744072b60 0x7f57440c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f574415a780 0x7f574406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574415b0d0 con 0x7f574406d960 2023-10-16T09:44:50.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440c83c0 con 0x7f5744072b60 2023-10-16T09:44:50.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574c637700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57440c9060 con 0x7f574415a780 2023-10-16T09:44:50.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574a3d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f574406d960 0x7f5744150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574a3d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f574406d960 0x7f5744150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48572/0 (socket says 172.21.15.158:48572) 2023-10-16T09:44:50.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.981+0000 7f574a3d3700 1 -- 172.21.15.158:0/1309443774 learned_addr learned my addr 172.21.15.158:0/1309443774 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:50.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574abd4700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f574415a780 0x7f574406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f5749bd2700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5744072b60 0x7f57440c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574a3d3700 1 -- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f574415a780 msgr2=0x7f574406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574a3d3700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f574415a780 0x7f574406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574a3d3700 1 -- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5744072b60 msgr2=0x7f57440c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:50.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574a3d3700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5744072b60 0x7f57440c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:50.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574a3d3700 1 -- 172.21.15.158:0/1309443774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740009550 con 0x7f574406d960 2023-10-16T09:44:50.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f5749bd2700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5744072b60 0x7f57440c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:44:50.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574a3d3700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f574406d960 0x7f5744150620 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f573400c510 tx=0x7f573400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:50.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f573affd700 1 -- 172.21.15.158:0/1309443774 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340052d0 con 0x7f574406d960 2023-10-16T09:44:50.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f573affd700 1 -- 172.21.15.158:0/1309443774 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5734005430 con 0x7f574406d960 2023-10-16T09:44:50.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57440c7740 con 0x7f574406d960 2023-10-16T09:44:50.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f573affd700 1 -- 172.21.15.158:0/1309443774 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340038c0 con 0x7f574406d960 2023-10-16T09:44:50.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.982+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57440c7d00 con 0x7f574406d960 2023-10-16T09:44:50.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.983+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57440681b0 con 0x7f574406d960 2023-10-16T09:44:50.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.984+0000 7f573affd700 1 -- 172.21.15.158:0/1309443774 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5734003a20 con 0x7f574406d960 2023-10-16T09:44:50.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.984+0000 7f573affd700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f572806af30 0x7f572806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:50.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.984+0000 7f5749bd2700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f572806af30 0x7f572806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:50.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.984+0000 7f573affd700 1 -- 172.21.15.158:0/1309443774 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5734088b10 con 0x7f574406d960 2023-10-16T09:44:50.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.985+0000 7f5749bd2700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f572806af30 0x7f572806d3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f574406d2a0 tx=0x7f5740008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:50.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:50.989+0000 7f573affd700 1 -- 172.21.15.158:0/1309443774 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5734053a80 con 0x7f574406d960 2023-10-16T09:44:51.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.264+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f57440728b0 con 0x7f574406d960 2023-10-16T09:44:51.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.266+0000 7f573affd700 1 -- 172.21.15.158:0/1309443774 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f57340170e0 con 0x7f574406d960 2023-10-16T09:44:51.268 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T09:44:51.269 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T09:44:51.269 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T09:44:51.269 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T09:44:51.269 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T09:44:51.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f572806af30 msgr2=0x7f572806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f572806af30 0x7f572806d3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f574406d2a0 tx=0x7f5740008030 comp rx=0 tx=0).stop 2023-10-16T09:44:51.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f574406d960 msgr2=0x7f5744150620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f574406d960 0x7f5744150620 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f573400c510 tx=0x7f573400c820 comp rx=0 tx=0).stop 2023-10-16T09:44:51.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 shutdown_connections 2023-10-16T09:44:51.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f572806af30 0x7f572806d3f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f574415a780 0x7f574406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f574406d960 0x7f5744150620 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 --2- 172.21.15.158:0/1309443774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5744072b60 0x7f57440c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 >> 172.21.15.158:0/1309443774 conn(0x7f57440ccd30 msgr2=0x7f57440d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:51.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 shutdown_connections 2023-10-16T09:44:51.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.268+0000 7f574c637700 1 -- 172.21.15.158:0/1309443774 wait complete. 2023-10-16T09:44:51.273 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T09:44:51.283 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:44:51.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.401+0000 7f6697d14700 1 Processor -- start 2023-10-16T09:44:51.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.402+0000 7f6697d14700 1 -- start start 2023-10-16T09:44:51.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6697d14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f669015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:51.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6697d14700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66900b6e10 0x7f66901521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:51.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6697d14700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669015c5d0 con 0x7f66900ce2c0 2023-10-16T09:44:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6697d14700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669015d2c0 con 0x7f66900b6e10 2023-10-16T09:44:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6697d14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669015c8b0 con 0x7f66900cdee0 2023-10-16T09:44:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f669015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f66952af700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66900b6e10 0x7f66901521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f669015bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:59414/0 (socket says 172.21.15.158:59414) 2023-10-16T09:44:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f66962b1700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f66900ce2c0 0x7f66900cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:52664/0 (socket says 172.21.15.158:52664) 2023-10-16T09:44:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f66952af700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66900b6e10 0x7f66901521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:49762/0 (socket says 172.21.15.158:49762) 2023-10-16T09:44:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 -- 172.21.15.158:0/789222316 learned_addr learned my addr 172.21.15.158:0/789222316 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 -- 172.21.15.158:0/789222316 >> v2:172.21.15.172:3300/0 conn(0x7f66900b6e10 msgr2=0x7f66901521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 --2- 172.21.15.158:0/789222316 >> v2:172.21.15.172:3300/0 conn(0x7f66900b6e10 0x7f66901521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 -- 172.21.15.158:0/789222316 >> v1:172.21.15.172:6789/0 conn(0x7f66900ce2c0 legacy=0x7f66900cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 -- 172.21.15.158:0/789222316 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669015cb90 con 0x7f66900cdee0 2023-10-16T09:44:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f66952af700 1 --2- 172.21.15.158:0/789222316 >> v2:172.21.15.172:3300/0 conn(0x7f66900b6e10 0x7f66901521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:44:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.403+0000 7f6695ab0700 1 --2- 172.21.15.158:0/789222316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f669015bd10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f66800098f0 tx=0x7f6680009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aedb62d04efe0252 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6694aae700 1 -- 172.21.15.158:0/789222316 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680011830 con 0x7f66900cdee0 2023-10-16T09:44:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6694aae700 1 -- 172.21.15.158:0/789222316 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680011990 con 0x7f66900cdee0 2023-10-16T09:44:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6694aae700 1 -- 172.21.15.158:0/789222316 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680011d20 con 0x7f66900cdee0 2023-10-16T09:44:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 -- 172.21.15.158:0/789222316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 msgr2=0x7f669015bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 --2- 172.21.15.158:0/789222316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f669015bd10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f66800098f0 tx=0x7f6680009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 -- 172.21.15.158:0/789222316 shutdown_connections 2023-10-16T09:44:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 --2- 172.21.15.158:0/789222316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f669015bd10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 --2- 172.21.15.158:0/789222316 >> v2:172.21.15.172:3300/0 conn(0x7f66900b6e10 0x7f66901521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 -- 172.21.15.158:0/789222316 >> 172.21.15.158:0/789222316 conn(0x7f669006a8d0 msgr2=0x7f669006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:51.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 -- 172.21.15.158:0/789222316 shutdown_connections 2023-10-16T09:44:51.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.404+0000 7f6697d14700 1 -- 172.21.15.158:0/789222316 wait complete. 2023-10-16T09:44:51.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.405+0000 7f6697d14700 1 Processor -- start 2023-10-16T09:44:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.405+0000 7f6697d14700 1 -- start start 2023-10-16T09:44:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f6697d14700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66900b6e10 0x7f669013f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f6697d14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f6690155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f6697d14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66900ce2c0 0x7f669015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f6697d14700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6690154b90 con 0x7f66900ce2c0 2023-10-16T09:44:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f6697d14700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66900d0b50 con 0x7f66900b6e10 2023-10-16T09:44:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f6697d14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6690154010 con 0x7f66900cdee0 2023-10-16T09:44:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f66962b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66900ce2c0 0x7f669015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f66962b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66900ce2c0 0x7f669015a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48588/0 (socket says 172.21.15.158:48588) 2023-10-16T09:44:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f66962b1700 1 -- 172.21.15.158:0/65735537 learned_addr learned my addr 172.21.15.158:0/65735537 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f66952af700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f6690155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.406+0000 7f6695ab0700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66900b6e10 0x7f669013f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66962b1700 1 -- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 msgr2=0x7f6690155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66962b1700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f6690155260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66962b1700 1 -- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66900b6e10 msgr2=0x7f669013f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66962b1700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66900b6e10 0x7f669013f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66962b1700 1 -- 172.21.15.158:0/65735537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c0094a0 con 0x7f66900ce2c0 2023-10-16T09:44:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66962b1700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66900ce2c0 0x7f669015a9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f668800cbc0 tx=0x7f668800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66867fc700 1 -- 172.21.15.158:0/65735537 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668800a9e0 con 0x7f66900ce2c0 2023-10-16T09:44:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66867fc700 1 -- 172.21.15.158:0/65735537 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668800ab40 con 0x7f66900ce2c0 2023-10-16T09:44:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6680009550 con 0x7f66900ce2c0 2023-10-16T09:44:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f66867fc700 1 -- 172.21.15.158:0/65735537 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688007070 con 0x7f66900ce2c0 2023-10-16T09:44:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.407+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66901545d0 con 0x7f66900ce2c0 2023-10-16T09:44:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.408+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66900d0d50 con 0x7f66900ce2c0 2023-10-16T09:44:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.408+0000 7f66867fc700 1 -- 172.21.15.158:0/65735537 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f66880071d0 con 0x7f66900ce2c0 2023-10-16T09:44:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.409+0000 7f66867fc700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f667406af00 0x7f667406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.409+0000 7f66867fc700 1 -- 172.21.15.158:0/65735537 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6688089340 con 0x7f66900ce2c0 2023-10-16T09:44:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.409+0000 7f6695ab0700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f667406af00 0x7f667406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.409+0000 7f6695ab0700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f667406af00 0x7f667406d3c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6680009c40 tx=0x7f6680008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.413+0000 7f66867fc700 1 -- 172.21.15.158:0/65735537 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66900d0d50 con 0x7f66900ce2c0 2023-10-16T09:44:51.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.651+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66900681b0 con 0x7f66900ce2c0 2023-10-16T09:44:51.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.652+0000 7f66867fc700 1 -- 172.21.15.158:0/65735537 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4928 (secure 0 0 0) 0x7f66880542e0 con 0x7f66900ce2c0 2023-10-16T09:44:51.655 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:44:51.655 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4283661376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4283661376},{"type":"v1","addr":"172.21.15.158:6835","nonce":4283661376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/3069959147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":3069959147},{"type":"v1","addr":"172.21.15.158:6837","nonce":3069959147}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1583191131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1583191131},{"type":"v1","addr":"172.21.15.172:6835","nonce":1583191131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:44:41.090180+0000","modified":"2023-10-16T09:44:44.131783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34760":{"gid":34760,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.172:6834/1706659835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1706659835},{"type":"v1","addr":"172.21.15.172:6834","nonce":1706659835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T09:44:51.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f667406af00 msgr2=0x7f667406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f667406af00 0x7f667406d3c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6680009c40 tx=0x7f6680008030 comp rx=0 tx=0).stop 2023-10-16T09:44:51.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66900ce2c0 msgr2=0x7f669015a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:51.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66900ce2c0 0x7f669015a9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f668800cbc0 tx=0x7f668800ced0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 shutdown_connections 2023-10-16T09:44:51.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66900ce2c0 0x7f669015a9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66900b6e10 0x7f669013f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f667406af00 0x7f667406d3c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 --2- 172.21.15.158:0/65735537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66900cdee0 0x7f6690155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:51.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 >> 172.21.15.158:0/65735537 conn(0x7f669006a8d0 msgr2=0x7f66901509a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:51.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 shutdown_connections 2023-10-16T09:44:51.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:51.654+0000 7f6697d14700 1 -- 172.21.15.158:0/65735537 wait complete. 2023-10-16T09:44:51.660 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 17 2023-10-16T09:44:52.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T09:44:41.090180+0000', 'modified': '2023-10-16T09:44:44.131783+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34760}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34760': {'gid': 34760, 'name': 'b', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.172:6834/1706659835', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6832', 'nonce': 1706659835}, {'type': 'v1', 'addr': '172.21.15.172:6834', 'nonce': 1706659835}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T09:44:52.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T09:44:52.072 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T09:44:52.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.196+0000 7efdecfce700 1 Processor -- start 2023-10-16T09:44:52.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.196+0000 7efdecfce700 1 -- start start 2023-10-16T09:44:52.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efdecfce700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efde815eb10 0x7efde815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efdecfce700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efde80d0ae0 0x7efde80b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efdecfce700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde815f6f0 con 0x7efde80b4950 2023-10-16T09:44:52.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efdecfce700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde80cdef0 con 0x7efde815eb10 2023-10-16T09:44:52.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efdecfce700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde80cd070 con 0x7efde80d0ae0 2023-10-16T09:44:52.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efde659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efde815eb10 0x7efde815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efde659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efde815eb10 0x7efde815f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48594/0 (socket says 172.21.15.158:48594) 2023-10-16T09:44:52.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efde659c700 1 -- 172.21.15.158:0/2961794424 learned_addr learned my addr 172.21.15.158:0/2961794424 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efde659c700 1 -- 172.21.15.158:0/2961794424 >> v2:172.21.15.172:3300/0 conn(0x7efde80d0ae0 msgr2=0x7efde80b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efde659c700 1 --2- 172.21.15.158:0/2961794424 >> v2:172.21.15.172:3300/0 conn(0x7efde80d0ae0 0x7efde80b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efde659c700 1 -- 172.21.15.158:0/2961794424 >> v1:172.21.15.172:6789/0 conn(0x7efde80b4950 legacy=0x7efde80cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.197+0000 7efde659c700 1 -- 172.21.15.158:0/2961794424 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde815f990 con 0x7efde815eb10 2023-10-16T09:44:52.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efde659c700 1 --2- 172.21.15.158:0/2961794424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde815eb10 0x7efde815f1b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efde8071670 tx=0x7efddc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6134570fe5bfe3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efde559a700 1 -- 172.21.15.158:0/2961794424 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc004230 con 0x7efde815eb10 2023-10-16T09:44:52.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efde559a700 1 -- 172.21.15.158:0/2961794424 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc004390 con 0x7efde815eb10 2023-10-16T09:44:52.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efde559a700 1 -- 172.21.15.158:0/2961794424 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc0036a0 con 0x7efde815eb10 2023-10-16T09:44:52.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 -- 172.21.15.158:0/2961794424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde815eb10 msgr2=0x7efde815f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 --2- 172.21.15.158:0/2961794424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde815eb10 0x7efde815f1b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efde8071670 tx=0x7efddc009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 -- 172.21.15.158:0/2961794424 shutdown_connections 2023-10-16T09:44:52.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 --2- 172.21.15.158:0/2961794424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde815eb10 0x7efde815f1b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 --2- 172.21.15.158:0/2961794424 >> v2:172.21.15.172:3300/0 conn(0x7efde80d0ae0 0x7efde80b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 -- 172.21.15.158:0/2961794424 >> 172.21.15.158:0/2961794424 conn(0x7efde80ce040 msgr2=0x7efde80ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 -- 172.21.15.158:0/2961794424 shutdown_connections 2023-10-16T09:44:52.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.198+0000 7efdecfce700 1 -- 172.21.15.158:0/2961794424 wait complete. 2023-10-16T09:44:52.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 Processor -- start 2023-10-16T09:44:52.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 -- start start 2023-10-16T09:44:52.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde80b4950 0x7efde8155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efde80d0ae0 0x7efde8154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efde815eb10 0x7efde806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde806cb80 con 0x7efde80b4950 2023-10-16T09:44:52.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde8155d40 con 0x7efde80d0ae0 2023-10-16T09:44:52.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efdecfce700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde806c2e0 con 0x7efde815eb10 2023-10-16T09:44:52.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efde659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde80b4950 0x7efde8155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efde659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde80b4950 0x7efde8155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48598/0 (socket says 172.21.15.158:48598) 2023-10-16T09:44:52.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.200+0000 7efde659c700 1 -- 172.21.15.158:0/3148038443 learned_addr learned my addr 172.21.15.158:0/3148038443 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde6d9d700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efde815eb10 0x7efde806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde5d9b700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efde80d0ae0 0x7efde8154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde659c700 1 -- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efde815eb10 msgr2=0x7efde806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde659c700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efde815eb10 0x7efde806b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde659c700 1 -- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efde80d0ae0 msgr2=0x7efde8154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde659c700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efde80d0ae0 0x7efde8154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde659c700 1 -- 172.21.15.158:0/3148038443 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efddc009550 con 0x7efde80b4950 2023-10-16T09:44:52.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efde659c700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde80b4950 0x7efde8155800 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efddc009fd0 tx=0x7efddc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efdd6ffd700 1 -- 172.21.15.158:0/3148038443 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc011630 con 0x7efde80b4950 2023-10-16T09:44:52.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efdd6ffd700 1 -- 172.21.15.158:0/3148038443 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc011790 con 0x7efde80b4950 2023-10-16T09:44:52.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde806c5c0 con 0x7efde80b4950 2023-10-16T09:44:52.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efdd6ffd700 1 -- 172.21.15.158:0/3148038443 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc011b40 con 0x7efde80b4950 2023-10-16T09:44:52.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.201+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde8154d40 con 0x7efde80b4950 2023-10-16T09:44:52.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.203+0000 7efdd6ffd700 1 -- 172.21.15.158:0/3148038443 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efddc011ca0 con 0x7efde80b4950 2023-10-16T09:44:52.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.203+0000 7efdd6ffd700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdc806af00 0x7efdc806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.203+0000 7efdd6ffd700 1 -- 172.21.15.158:0/3148038443 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efddc08ad70 con 0x7efde80b4950 2023-10-16T09:44:52.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.203+0000 7efde5d9b700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdc806af00 0x7efdc806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.204+0000 7efde5d9b700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdc806af00 0x7efdc806d3c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdd8006fd0 tx=0x7efdd8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.205+0000 7efdf91b8b80 1 Processor -- start 2023-10-16T09:44:52.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.205+0000 7efdf91b8b80 1 -- start start 2023-10-16T09:44:52.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317fc7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317f9b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a317f11c20 con 0x55a317f9b010 2023-10-16T09:44:52.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a317ef6c10 con 0x55a317fc5630 2023-10-16T09:44:52.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a317ef4530 con 0x55a317fc0220 2023-10-16T09:44:52.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317f9b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317f9b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48608/0 (socket says 172.21.15.158:48608) 2023-10-16T09:44:52.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 -- 172.21.15.158:0/563389352 learned_addr learned my addr 172.21.15.158:0/563389352 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde659c700 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde5d9b700 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317fc7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 -- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 msgr2=0x55a317fc7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 -- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 msgr2=0x55a317fc7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317fc7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde6d9d700 1 -- 172.21.15.158:0/563389352 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a317ead3a0 con 0x55a317f9b010 2023-10-16T09:44:52.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.206+0000 7efde659c700 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:52.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efde6d9d700 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317f9b6c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efde0001ce0 tx=0x7efde0001ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497bba990e418d33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdc7fff700 1 -- 172.21.15.158:0/563389352 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0003c10 con 0x55a317f9b010 2023-10-16T09:44:52.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdc7fff700 1 -- 172.21.15.158:0/563389352 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0003db0 con 0x55a317f9b010 2023-10-16T09:44:52.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdc7fff700 1 -- 172.21.15.158:0/563389352 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0005310 con 0x55a317f9b010 2023-10-16T09:44:52.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdf91b8b80 1 -- 172.21.15.158:0/563389352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 msgr2=0x55a317f9b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317f9b6c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efde0001ce0 tx=0x7efde0001ff0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdf91b8b80 1 -- 172.21.15.158:0/563389352 shutdown_connections 2023-10-16T09:44:52.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317f9b6c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/563389352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317fc7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.207+0000 7efdf91b8b80 1 -- 172.21.15.158:0/563389352 >> 172.21.15.158:0/563389352 conn(0x55a317ecad80 msgr2=0x55a317fe5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.208+0000 7efdf91b8b80 1 -- 172.21.15.158:0/563389352 shutdown_connections 2023-10-16T09:44:52.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.208+0000 7efdf91b8b80 1 -- 172.21.15.158:0/563389352 wait complete. 2023-10-16T09:44:52.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.208+0000 7efdf91b8b80 1 -- start start 2023-10-16T09:44:52.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.208+0000 7efdf91b8b80 1 Processor -- start 2023-10-16T09:44:52.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fbf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc5630 0x55a317fc3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a317eefe20 con 0x55a317fc5630 2023-10-16T09:44:52.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a317e8c6c0 con 0x55a317f9b010 2023-10-16T09:44:52.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a317e8c360 con 0x55a317fc0220 2023-10-16T09:44:52.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efdf91b8b80 10 client.0 _ll_register_callbacks cb 0x55a317ce97b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:44:52.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efde6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc5630 0x55a317fc3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efde6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc5630 0x55a317fc3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48616/0 (socket says 172.21.15.158:48616) 2023-10-16T09:44:52.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efde6d9d700 1 -- 172.21.15.158:0/704375384 learned_addr learned my addr 172.21.15.158:0/704375384 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efde659c700 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fbf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efde5d9b700 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.209+0000 7efde6d9d700 1 -- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 msgr2=0x55a317fc3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efde6d9d700 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efde6d9d700 1 -- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 msgr2=0x55a317fbf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efde6d9d700 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fbf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efde6d9d700 1 -- 172.21.15.158:0/704375384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde000f040 con 0x55a317fc5630 2023-10-16T09:44:52.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efde6d9d700 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc5630 0x55a317fc3a80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7efde0013000 tx=0x7efde0001330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694ac74d339bb781 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdc57fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:44:52.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdc57fa700 1 -- 172.21.15.158:0/704375384 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0001930 con 0x55a317fc5630 2023-10-16T09:44:52.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdc57fa700 1 -- 172.21.15.158:0/704375384 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efde00052e0 con 0x55a317fc5630 2023-10-16T09:44:52.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a317f905b0 con 0x55a317fc5630 2023-10-16T09:44:52.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdc57fa700 1 -- 172.21.15.158:0/704375384 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7efde00054a0 con 0x55a317fc5630 2023-10-16T09:44:52.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 10 client.5553 fetch_fsmap learned FSMap version 17 2023-10-16T09:44:52.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55a317d1ad90 con 0x55a317fc5630 2023-10-16T09:44:52.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdc57fa700 1 -- 172.21.15.158:0/704375384 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a317f905b0 con 0x55a317fc5630 2023-10-16T09:44:52.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdc57fa700 1 -- 172.21.15.158:0/704375384 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efde0005600 con 0x55a317fc5630 2023-10-16T09:44:52.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 10 client.5553 fetch_fsmap finished waiting for FSMap version 17 2023-10-16T09:44:52.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 10 client.5553 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:44:52.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 20 client.5553 populate_metadata read hostname 'smithi158' 2023-10-16T09:44:52.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317fcabd0 0x55a317fcafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 4 client.5553 mds_command: new command op to 34760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T09:44:52.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.210+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 --> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a317ee8e00 con 0x55a317fcabd0 2023-10-16T09:44:52.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.211+0000 7efde659c700 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317fcabd0 0x55a317fcafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.211+0000 7efde659c700 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317fcabd0 0x55a317fcafb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8a10b24a744a6ce server_cookie=d0dcfc45f4fc24f7 in_seq=0 out_seq=0 2023-10-16T09:44:52.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.211+0000 7efdc57fa700 10 client.5553 ms_handle_connect on v2:172.21.15.172:6832/1706659835 2023-10-16T09:44:52.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdc57fa700 1 -- 172.21.15.158:0/704375384 <== mds.0 v2:172.21.15.172:6832/1706659835 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a317ee8e00 con 0x55a317fcabd0 2023-10-16T09:44:52.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdc57fa700 10 client.5553 handle_command_reply: tid=0 2023-10-16T09:44:52.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 client.5553 shutdown 2023-10-16T09:44:52.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc5630 msgr2=0x55a317fc3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc5630 0x55a317fc3a80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7efde0013000 tx=0x7efde0001330 comp rx=0 tx=0).stop 2023-10-16T09:44:52.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 shutdown_connections 2023-10-16T09:44:52.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317fcabd0 0x55a317fcafb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc5630 0x55a317fc3a80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fbf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/704375384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317fc3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdc57fa700 0 client.5553 ms_handle_reset on v2:172.21.15.172:6832/1706659835 2023-10-16T09:44:52.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 >> 172.21.15.158:0/704375384 conn(0x55a317ecad80 msgr2=0x55a317f92640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 shutdown_connections 2023-10-16T09:44:52.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 1 -- 172.21.15.158:0/704375384 wait complete. 2023-10-16T09:44:52.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.212+0000 7efdf91b8b80 20 client.5553 trim_cache size 0 max 16384 2023-10-16T09:44:52.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 Processor -- start 2023-10-16T09:44:52.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 -- start start 2023-10-16T09:44:52.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317fc3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317f9ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317f92160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a317d708d0 con 0x55a317f9b010 2023-10-16T09:44:52.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a317fcb7d0 con 0x55a317fc5630 2023-10-16T09:44:52.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a317f91820 con 0x55a317fc0220 2023-10-16T09:44:52.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efde659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317fc3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efde659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317fc3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48620/0 (socket says 172.21.15.158:48620) 2023-10-16T09:44:52.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.230+0000 7efde659c700 1 -- 172.21.15.158:0/1233463616 learned_addr learned my addr 172.21.15.158:0/1233463616 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde5d9b700 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317f9ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde659c700 1 -- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 msgr2=0x55a317f9ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde6d9d700 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317f92160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde659c700 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317f9ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde659c700 1 -- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 msgr2=0x55a317f92160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde659c700 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317f92160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde659c700 1 -- 172.21.15.158:0/1233463616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a317d03c10 con 0x55a317f9b010 2023-10-16T09:44:52.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efde659c700 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317fc3820 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efddc009c10 tx=0x7efddc0890c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3e2c6a78b40a0b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efdd4ff9700 1 -- 172.21.15.158:0/1233463616 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc061d30 con 0x55a317f9b010 2023-10-16T09:44:52.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efdd4ff9700 1 -- 172.21.15.158:0/1233463616 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc057b70 con 0x55a317f9b010 2023-10-16T09:44:52.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efdd4ff9700 1 -- 172.21.15.158:0/1233463616 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc0711c0 con 0x55a317f9b010 2023-10-16T09:44:52.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efdf91b8b80 1 -- 172.21.15.158:0/1233463616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 msgr2=0x55a317fc3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.231+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317fc3820 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efddc009c10 tx=0x7efddc0890c0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 -- 172.21.15.158:0/1233463616 shutdown_connections 2023-10-16T09:44:52.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317fc5630 0x55a317f92160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317f9b010 0x55a317fc3820 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/1233463616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc0220 0x55a317f9ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 -- 172.21.15.158:0/1233463616 >> 172.21.15.158:0/1233463616 conn(0x55a317ecad80 msgr2=0x55a317fe5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 -- 172.21.15.158:0/1233463616 shutdown_connections 2023-10-16T09:44:52.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 -- 172.21.15.158:0/1233463616 wait complete. 2023-10-16T09:44:52.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 -- start start 2023-10-16T09:44:52.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.232+0000 7efdf91b8b80 1 Processor -- start 2023-10-16T09:44:52.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fe2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc0220 0x55a318049a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efdf91b8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc5630 0x55a317fc7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a317f99ba0 con 0x55a317fc0220 2023-10-16T09:44:52.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a31804a520 con 0x55a317f9b010 2023-10-16T09:44:52.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efdf91b8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a317fe6320 con 0x55a317fc5630 2023-10-16T09:44:52.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efdf91b8b80 10 client.0 _ll_register_callbacks cb 0x7efde0017070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:44:52.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efde5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc0220 0x55a318049a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efde5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc0220 0x55a318049a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48628/0 (socket says 172.21.15.158:48628) 2023-10-16T09:44:52.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efde5d9b700 1 -- 172.21.15.158:0/2492501708 learned_addr learned my addr 172.21.15.158:0/2492501708 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efde659c700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fe2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.233+0000 7efde6d9d700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc5630 0x55a317fc7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde5d9b700 1 -- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc5630 msgr2=0x55a317fc7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde5d9b700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc5630 0x55a317fc7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde5d9b700 1 -- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 msgr2=0x55a317fe2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde5d9b700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fe2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde5d9b700 1 -- 172.21.15.158:0/2492501708 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde8154d40 con 0x55a317fc0220 2023-10-16T09:44:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde659c700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fe2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde6d9d700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc5630 0x55a317fc7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efde5d9b700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc0220 0x55a318049a20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efde8152450 tx=0x7efdd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb55c5af2af712a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdc57fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:44:52.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdc57fa700 1 -- 172.21.15.158:0/2492501708 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8019040 con 0x55a317fc0220 2023-10-16T09:44:52.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdc57fa700 1 -- 172.21.15.158:0/2492501708 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efdd8011c40 con 0x55a317fc0220 2023-10-16T09:44:52.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efde0003db0 con 0x55a317fc0220 2023-10-16T09:44:52.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdc57fa700 1 -- 172.21.15.158:0/2492501708 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdd8011e00 con 0x55a317fc0220 2023-10-16T09:44:52.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 10 client.5559 fetch_fsmap learned FSMap version 17 2023-10-16T09:44:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55a317d41200 con 0x55a317fc0220 2023-10-16T09:44:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdc57fa700 1 -- 172.21.15.158:0/2492501708 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0003db0 con 0x55a317fc0220 2023-10-16T09:44:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdc57fa700 1 -- 172.21.15.158:0/2492501708 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efdd8029440 con 0x55a317fc0220 2023-10-16T09:44:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 10 client.5559 fetch_fsmap finished waiting for FSMap version 17 2023-10-16T09:44:52.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 10 client.5559 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:44:52.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 20 client.5559 populate_metadata read hostname 'smithi158' 2023-10-16T09:44:52.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317f923d0 0x55a317ff1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 4 client.5559 mds_command: new command op to 34760 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T09:44:52.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.234+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 --> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a317ee8b20 con 0x55a317f923d0 2023-10-16T09:44:52.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efde659c700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317f923d0 0x55a317ff1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efde659c700 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317f923d0 0x55a317ff1140 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c616cd18ed8e6fe8 server_cookie=9c42b2c34cc422d7 in_seq=0 out_seq=0 2023-10-16T09:44:52.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdc57fa700 10 client.5559 ms_handle_connect on v2:172.21.15.172:6832/1706659835 2023-10-16T09:44:52.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdc57fa700 1 -- 172.21.15.158:0/2492501708 <== mds.0 v2:172.21.15.172:6832/1706659835 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a317ee8b20 con 0x55a317f923d0 2023-10-16T09:44:52.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdc57fa700 10 client.5559 handle_command_reply: tid=0 2023-10-16T09:44:52.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 client.5559 shutdown 2023-10-16T09:44:52.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc0220 msgr2=0x55a318049a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc0220 0x55a318049a20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efde8152450 tx=0x7efdd8004750 comp rx=0 tx=0).stop 2023-10-16T09:44:52.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 shutdown_connections 2023-10-16T09:44:52.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:6832/1706659835,v1:172.21.15.172:6834/1706659835] conn(0x55a317f923d0 0x55a317ff1140 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55a317fc5630 0x55a317fc7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdc57fa700 0 client.5559 ms_handle_reset on v2:172.21.15.172:6832/1706659835 2023-10-16T09:44:52.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a317f9b010 0x55a317fe2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 --2- 172.21.15.158:0/2492501708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55a317fc0220 0x55a318049a20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.235+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 >> 172.21.15.158:0/2492501708 conn(0x55a317ecad80 msgr2=0x55a317fca430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 shutdown_connections 2023-10-16T09:44:52.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdf91b8b80 1 -- 172.21.15.158:0/2492501708 wait complete. 2023-10-16T09:44:52.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdf91b8b80 20 client.5559 trim_cache size 0 max 16384 2023-10-16T09:44:52.268 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:44:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T09:44:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T09:44:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 34760, 2023-10-16T09:44:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T09:44:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T09:44:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T09:44:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 10.52099836, 2023-10-16T09:44:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 17, 2023-10-16T09:44:52.270 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 39, 2023-10-16T09:44:52.270 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 39, 2023-10-16T09:44:52.270 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 19.417950821000002 2023-10-16T09:44:52.270 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:44:52.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdc806af00 msgr2=0x7efdc806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdc806af00 0x7efdc806d3c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdd8006fd0 tx=0x7efdd8008040 comp rx=0 tx=0).stop 2023-10-16T09:44:52.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde80b4950 msgr2=0x7efde8155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde80b4950 0x7efde8155800 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efddc009fd0 tx=0x7efddc004750 comp rx=0 tx=0).stop 2023-10-16T09:44:52.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 shutdown_connections 2023-10-16T09:44:52.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7efdc806af00 0x7efdc806d3c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efde815eb10 0x7efde806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde80b4950 0x7efde8155800 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 --2- 172.21.15.158:0/3148038443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efde80d0ae0 0x7efde8154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 >> 172.21.15.158:0/3148038443 conn(0x7efde80ce040 msgr2=0x7efde80ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 shutdown_connections 2023-10-16T09:44:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.236+0000 7efdecfce700 1 -- 172.21.15.158:0/3148038443 wait complete. 2023-10-16T09:44:52.275 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:44:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.394+0000 7f42ae0d3700 1 Processor -- start 2023-10-16T09:44:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.394+0000 7f42ae0d3700 1 -- start start 2023-10-16T09:44:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42ae0d3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42a80aefc0 0x7f42a80cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42ae0d3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f42a815f0b0 0x7f42a815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42ae0d3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a815fa00 con 0x7f42a8152580 2023-10-16T09:44:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42ae0d3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a80af450 con 0x7f42a80aefc0 2023-10-16T09:44:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42ae0d3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a806aa70 con 0x7f42a815f0b0 2023-10-16T09:44:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42a80aefc0 0x7f42a80cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42a80aefc0 0x7f42a80cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48636/0 (socket says 172.21.15.158:48636) 2023-10-16T09:44:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 -- 172.21.15.158:0/72615406 learned_addr learned my addr 172.21.15.158:0/72615406 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a7fff700 1 --2- 172.21.15.158:0/72615406 >> v2:172.21.15.172:3300/0 conn(0x7f42a815f0b0 0x7f42a815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 -- 172.21.15.158:0/72615406 >> v2:172.21.15.172:3300/0 conn(0x7f42a815f0b0 msgr2=0x7f42a815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 --2- 172.21.15.158:0/72615406 >> v2:172.21.15.172:3300/0 conn(0x7f42a815f0b0 0x7f42a815f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 -- 172.21.15.158:0/72615406 >> v1:172.21.15.172:6789/0 conn(0x7f42a8152580 legacy=0x7f42a80ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:52.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 -- 172.21.15.158:0/72615406 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a806ad50 con 0x7f42a80aefc0 2023-10-16T09:44:52.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.395+0000 7f42a6ffd700 1 --2- 172.21.15.158:0/72615406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a80aefc0 0x7f42a80cdaf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f42a80cf9e0 tx=0x7f4298009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6537d57ed490ba6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42a67fc700 1 -- 172.21.15.158:0/72615406 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4298004230 con 0x7f42a80aefc0 2023-10-16T09:44:52.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42a67fc700 1 -- 172.21.15.158:0/72615406 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4298004390 con 0x7f42a80aefc0 2023-10-16T09:44:52.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42a67fc700 1 -- 172.21.15.158:0/72615406 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42980036a0 con 0x7f42a80aefc0 2023-10-16T09:44:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 -- 172.21.15.158:0/72615406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a80aefc0 msgr2=0x7f42a80cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/72615406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a80aefc0 0x7f42a80cdaf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f42a80cf9e0 tx=0x7f4298009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 -- 172.21.15.158:0/72615406 shutdown_connections 2023-10-16T09:44:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/72615406 >> v2:172.21.15.172:3300/0 conn(0x7f42a815f0b0 0x7f42a815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/72615406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a80aefc0 0x7f42a80cdaf0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 -- 172.21.15.158:0/72615406 >> 172.21.15.158:0/72615406 conn(0x7f42a80cc4b0 msgr2=0x7f42a80cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 -- 172.21.15.158:0/72615406 shutdown_connections 2023-10-16T09:44:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.396+0000 7f42ae0d3700 1 -- 172.21.15.158:0/72615406 wait complete. 2023-10-16T09:44:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.397+0000 7f42ae0d3700 1 Processor -- start 2023-10-16T09:44:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42ae0d3700 1 -- start start 2023-10-16T09:44:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42ae0d3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a80aefc0 0x7f42a80b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42ae0d3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a8152580 0x7f42a814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42ae0d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a815f0b0 0x7f42a80c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42ae0d3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80b3e90 con 0x7f42a815f0b0 2023-10-16T09:44:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42ae0d3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80c4cc0 con 0x7f42a8152580 2023-10-16T09:44:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42ae0d3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80c4fa0 con 0x7f42a80aefc0 2023-10-16T09:44:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a815f0b0 0x7f42a80c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a815f0b0 0x7f42a80c6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48644/0 (socket says 172.21.15.158:48644) 2023-10-16T09:44:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.398+0000 7f42a7fff700 1 -- 172.21.15.158:0/910713377 learned_addr learned my addr 172.21.15.158:0/910713377 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a6ffd700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a8152580 0x7f42a814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a77fe700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a80aefc0 0x7f42a80b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a7fff700 1 -- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a80aefc0 msgr2=0x7f42a80b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a7fff700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a80aefc0 0x7f42a80b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a7fff700 1 -- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a8152580 msgr2=0x7f42a814e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a7fff700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a8152580 0x7f42a814e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a7fff700 1 -- 172.21.15.158:0/910713377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298009550 con 0x7f42a815f0b0 2023-10-16T09:44:52.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a6ffd700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a8152580 0x7f42a814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:52.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a77fe700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a80aefc0 0x7f42a80b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:52.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42a7fff700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a815f0b0 0x7f42a80c6b00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f42a0007e60 tx=0x7f42a000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f428ffff700 1 -- 172.21.15.158:0/910713377 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a000ede0 con 0x7f42a815f0b0 2023-10-16T09:44:52.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f428ffff700 1 -- 172.21.15.158:0/910713377 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0014420 con 0x7f42a815f0b0 2023-10-16T09:44:52.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f428ffff700 1 -- 172.21.15.158:0/910713377 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00147d0 con 0x7f42a815f0b0 2023-10-16T09:44:52.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a80c5280 con 0x7f42a815f0b0 2023-10-16T09:44:52.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.399+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a80c5840 con 0x7f42a815f0b0 2023-10-16T09:44:52.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.400+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a80681b0 con 0x7f42a815f0b0 2023-10-16T09:44:52.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.400+0000 7f428ffff700 1 -- 172.21.15.158:0/910713377 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f42a0014930 con 0x7f42a815f0b0 2023-10-16T09:44:52.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.401+0000 7f428ffff700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f429406af00 0x7f429406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:52.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.401+0000 7f42a77fe700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f429406af00 0x7f429406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:52.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.401+0000 7f428ffff700 1 -- 172.21.15.158:0/910713377 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f42a0089e80 con 0x7f42a815f0b0 2023-10-16T09:44:52.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.401+0000 7f42a77fe700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f429406af00 0x7f429406d3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f42a80b2c30 tx=0x7f429c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:52.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.405+0000 7f428ffff700 1 -- 172.21.15.158:0/910713377 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f42a0054de0 con 0x7f42a815f0b0 2023-10-16T09:44:52.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.643+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f42a806fb20 con 0x7f42a815f0b0 2023-10-16T09:44:52.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.643+0000 7f428ffff700 1 -- 172.21.15.158:0/910713377 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f42a0058400 con 0x7f42a815f0b0 2023-10-16T09:44:52.646 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T09:44:52.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f429406af00 msgr2=0x7f429406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f429406af00 0x7f429406d3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f42a80b2c30 tx=0x7f429c009040 comp rx=0 tx=0).stop 2023-10-16T09:44:52.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a815f0b0 msgr2=0x7f42a80c6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:52.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a815f0b0 0x7f42a80c6b00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f42a0007e60 tx=0x7f42a000e3c0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 shutdown_connections 2023-10-16T09:44:52.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a815f0b0 0x7f42a80c6b00 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f429406af00 0x7f429406d3c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a80aefc0 0x7f42a80b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 --2- 172.21.15.158:0/910713377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a8152580 0x7f42a814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:52.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 >> 172.21.15.158:0/910713377 conn(0x7f42a80cc4b0 msgr2=0x7f42a806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:52.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 shutdown_connections 2023-10-16T09:44:52.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:52.645+0000 7f42ae0d3700 1 -- 172.21.15.158:0/910713377 wait complete. 2023-10-16T09:44:52.660 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T09:44:52.660 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:44:52.661 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T09:44:52.661 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T09:44:52.661 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T09:44:52.662 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T09:44:52.662 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T09:44:52.662 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:44:52.663 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:44:52.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:52.663 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T09:44:52.680 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:44:52.681 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:44:52.681 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:44:52.681 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:52.681 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:44:52.681 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:52.682 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:52.682 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:52.682 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:52.682 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:52.682 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:44:52.683 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:52.683 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:44:52.683 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:52.683 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:44:52.683 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 42340sec preferred_lft 42340sec 2023-10-16T09:44:52.684 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T09:44:52.684 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:52.684 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:44:52.684 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T09:44:52.684 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T09:44:52.685 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:52.685 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T09:44:52.685 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:52.685 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:44:52.685 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T09:44:52.686 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T09:44:52.686 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:52.686 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:44:52.686 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T09:44:52.686 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T09:44:52.687 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:44:52.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:52.688 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:44:52.746 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T09:44:52.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:52.748 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T09:44:52.806 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T09:44:52.806 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:44:52.807 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:44:52.807 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:44:52.807 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:44:52.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:52.808 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:44:52.809 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:44:52.809 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:44:52.810 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:44:52.810 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:44:52.810 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:44:52.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:52.912 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:44:52.928 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T09:44:52.929 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T09:44:52.929 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:44:52.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:52.930 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T09:44:52.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:52.996 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T09:44:53.052 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:44:53.052 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:44:53.053 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:44:53.108 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T09:44:53.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:53.109 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:53.193 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:53.194 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:53.215 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:53.216 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:53.216 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:53.216 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:53.216 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:53.216 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:53.217 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:53.217 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:53.243 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:44:53.243 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:44:53.244 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:44:53.244 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:44:53.244 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:44:53.244 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:44:53.244 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:44:53.245 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:44:53.245 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T09:44:53.245 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T09:44:53.245 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T09:44:53.248 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T09:44:53.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:53.248 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:44:53.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:53.278 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:44:53.367 INFO:teuthology.orchestra.run.smithi158.stdout:client5568 2023-10-16T09:44:53.367 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:44:53.368 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:44:53.385 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T09:44:53.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.544+0000 7f00629a1700 1 Processor -- start 2023-10-16T09:44:53.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.544+0000 7f00629a1700 1 -- start start 2023-10-16T09:44:53.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f00629a1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f005c158280 0x7f005c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f00629a1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f00629a1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0cb2d0 con 0x7f005c0722e0 2023-10-16T09:44:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f00629a1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005c15bc40 con 0x7f005c158280 2023-10-16T09:44:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f00629a1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c158660 con 0x7f005c0cf180 2023-10-16T09:44:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f005bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f005c158280 0x7f005c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f005bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f005c158280 0x7f005c06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48654/0 (socket says 172.21.15.158:48654) 2023-10-16T09:44:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f005bfff700 1 -- 172.21.15.158:0/2246850427 learned_addr learned my addr 172.21.15.158:0/2246850427 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f005bfff700 1 -- 172.21.15.158:0/2246850427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 msgr2=0x7f005c0cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.545+0000 7f0060f3e700 1 --2- 172.21.15.158:0/2246850427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f005bfff700 1 --2- 172.21.15.158:0/2246850427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f005bfff700 1 -- 172.21.15.158:0/2246850427 >> v1:172.21.15.172:6789/0 conn(0x7f005c0722e0 legacy=0x7f005c0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f005bfff700 1 -- 172.21.15.158:0/2246850427 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c15b030 con 0x7f005c158280 2023-10-16T09:44:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f005bfff700 1 --2- 172.21.15.158:0/2246850427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c06a8d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f00500098f0 tx=0x7f0050009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e36fc8f5f3ec45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f005affd700 1 -- 172.21.15.158:0/2246850427 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050004230 con 0x7f005c158280 2023-10-16T09:44:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f005affd700 1 -- 172.21.15.158:0/2246850427 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050004390 con 0x7f005c158280 2023-10-16T09:44:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f005affd700 1 -- 172.21.15.158:0/2246850427 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00500036a0 con 0x7f005c158280 2023-10-16T09:44:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f00629a1700 1 -- 172.21.15.158:0/2246850427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 msgr2=0x7f005c06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f00629a1700 1 --2- 172.21.15.158:0/2246850427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c06a8d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f00500098f0 tx=0x7f0050009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f00629a1700 1 -- 172.21.15.158:0/2246850427 shutdown_connections 2023-10-16T09:44:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f00629a1700 1 --2- 172.21.15.158:0/2246850427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f00629a1700 1 --2- 172.21.15.158:0/2246850427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c06a8d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f00629a1700 1 -- 172.21.15.158:0/2246850427 >> 172.21.15.158:0/2246850427 conn(0x7f005c0cc700 msgr2=0x7f005c0ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.546+0000 7f00629a1700 1 -- 172.21.15.158:0/2246850427 shutdown_connections 2023-10-16T09:44:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.547+0000 7f00629a1700 1 -- 172.21.15.158:0/2246850427 wait complete. 2023-10-16T09:44:53.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.548+0000 7f00629a1700 1 Processor -- start 2023-10-16T09:44:53.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.548+0000 7f00629a1700 1 -- start start 2023-10-16T09:44:53.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f005c0722e0 0x7f005c1491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c0c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0c5530 con 0x7f005c158280 2023-10-16T09:44:53.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0c4c60 con 0x7f005c0722e0 2023-10-16T09:44:53.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1f78e0 con 0x7f005c0cf180 2023-10-16T09:44:53.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c0c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c0c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48662/0 (socket says 172.21.15.158:48662) 2023-10-16T09:44:53.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 -- 172.21.15.158:0/2660535693 learned_addr learned my addr 172.21.15.158:0/2660535693 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:53.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f005b7fe700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f005bfff700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f005c0722e0 0x7f005c1491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 -- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 msgr2=0x7f005c0c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 -- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f005c0722e0 msgr2=0x7f005c1491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f005c0722e0 0x7f005c1491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 -- 172.21.15.158:0/2660535693 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0050009550 con 0x7f005c158280 2023-10-16T09:44:53.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f005bfff700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f005c0722e0 0x7f005c1491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f005b7fe700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:44:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0060f3e700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c0c6d50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f005400cd50 tx=0x7f005400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0058ff9700 1 -- 172.21.15.158:0/2660535693 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054003a20 con 0x7f005c158280 2023-10-16T09:44:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0058ff9700 1 -- 172.21.15.158:0/2660535693 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0054003b80 con 0x7f005c158280 2023-10-16T09:44:53.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c0c4ee0 con 0x7f005c158280 2023-10-16T09:44:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f0058ff9700 1 -- 172.21.15.158:0/2660535693 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054004020 con 0x7f005c158280 2023-10-16T09:44:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.549+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c1f75d0 con 0x7f005c158280 2023-10-16T09:44:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.550+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005c0c56e0 con 0x7f005c158280 2023-10-16T09:44:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.551+0000 7f0058ff9700 1 -- 172.21.15.158:0/2660535693 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0054004180 con 0x7f005c158280 2023-10-16T09:44:53.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.551+0000 7f0058ff9700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f004806af00 0x7f004806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.551+0000 7f0058ff9700 1 -- 172.21.15.158:0/2660535693 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0054012070 con 0x7f005c158280 2023-10-16T09:44:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.551+0000 7f005bfff700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f004806af00 0x7f004806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.552+0000 7f005bfff700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f004806af00 0x7f004806d3c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0050009c40 tx=0x7f0050008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:53.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.556+0000 7f0058ff9700 1 -- 172.21.15.158:0/2660535693 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f005c0c56e0 con 0x7f005c158280 2023-10-16T09:44:53.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.776+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f005c15b5f0 con 0x7f004806af00 2023-10-16T09:44:53.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.777+0000 7f0058ff9700 1 -- 172.21.15.158:0/2660535693 <== mgr.4101 v2:172.21.15.158:6832/4024924909 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f005001f070 con 0x7f004806af00 2023-10-16T09:44:53.780 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:44:53.780 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:44:53.780 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T09:44:53.781 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:44:53.781 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:44:53.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f004806af00 msgr2=0x7f004806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f004806af00 0x7f004806d3c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0050009c40 tx=0x7f0050008030 comp rx=0 tx=0).stop 2023-10-16T09:44:53.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 msgr2=0x7f005c0c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c0c6d50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f005400cd50 tx=0x7f005400a570 comp rx=0 tx=0).stop 2023-10-16T09:44:53.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 shutdown_connections 2023-10-16T09:44:53.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f005c158280 0x7f005c0c6d50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f005c0722e0 0x7f005c1491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f004806af00 0x7f004806d3c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 --2- 172.21.15.158:0/2660535693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f005c0cf180 0x7f005c0c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.779+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 >> 172.21.15.158:0/2660535693 conn(0x7f005c0cc700 msgr2=0x7f005c155c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:53.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.780+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 shutdown_connections 2023-10-16T09:44:53.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.780+0000 7f00629a1700 1 -- 172.21.15.158:0/2660535693 wait complete. 2023-10-16T09:44:53.794 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T09:44:53.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.915+0000 7fcc231e8700 1 Processor -- start 2023-10-16T09:44:53.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.915+0000 7fcc231e8700 1 -- start start 2023-10-16T09:44:53.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc231e8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcc1c15dad0 0x7fcc1c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc231e8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc1c15deb0 0x7fcc1c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc231e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c0cea20 0x7fcc1c0c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc231e8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c0cee50 con 0x7fcc1c15dad0 2023-10-16T09:44:53.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc231e8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c0d0ae0 con 0x7fcc1c15deb0 2023-10-16T09:44:53.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc231e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0d12d0 con 0x7fcc1c0cea20 2023-10-16T09:44:53.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc20f84700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcc1c15dad0 0x7fcc1c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc20f84700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcc1c15dad0 0x7fcc1c0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48666/0 (socket says 172.21.15.158:48666) 2023-10-16T09:44:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.916+0000 7fcc20f84700 1 -- 172.21.15.158:0/2623462235 learned_addr learned my addr 172.21.15.158:0/2623462235 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc1bfff700 1 --2- 172.21.15.158:0/2623462235 >> v2:172.21.15.172:3300/0 conn(0x7fcc1c15deb0 0x7fcc1c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc21785700 1 --2- 172.21.15.158:0/2623462235 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c0cea20 0x7fcc1c0c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc20f84700 1 -- 172.21.15.158:0/2623462235 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c0cea20 msgr2=0x7fcc1c0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc20f84700 1 --2- 172.21.15.158:0/2623462235 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c0cea20 0x7fcc1c0c3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc20f84700 1 -- 172.21.15.158:0/2623462235 >> v2:172.21.15.172:3300/0 conn(0x7fcc1c15deb0 msgr2=0x7fcc1c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc20f84700 1 --2- 172.21.15.158:0/2623462235 >> v2:172.21.15.172:3300/0 conn(0x7fcc1c15deb0 0x7fcc1c0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc20f84700 1 -- 172.21.15.158:0/2623462235 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0cbdf0 con 0x7fcc1c15dad0 2023-10-16T09:44:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc20f84700 1 --2- 172.21.15.158:0/2623462235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c15dad0 0x7fcc1c0cdfa0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c072c30 tx=0x7fcc10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b999b3bd4d85a5f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc1b7fe700 1 -- 172.21.15.158:0/2623462235 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000b860 con 0x7fcc1c15dad0 2023-10-16T09:44:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc1b7fe700 1 -- 172.21.15.158:0/2623462235 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1000b9c0 con 0x7fcc1c15dad0 2023-10-16T09:44:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc1b7fe700 1 -- 172.21.15.158:0/2623462235 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000bd50 con 0x7fcc1c15dad0 2023-10-16T09:44:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 -- 172.21.15.158:0/2623462235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c15dad0 msgr2=0x7fcc1c0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 --2- 172.21.15.158:0/2623462235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c15dad0 0x7fcc1c0cdfa0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c072c30 tx=0x7fcc10009c00 comp rx=0 tx=0).stop 2023-10-16T09:44:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 -- 172.21.15.158:0/2623462235 shutdown_connections 2023-10-16T09:44:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 --2- 172.21.15.158:0/2623462235 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c0cea20 0x7fcc1c0c3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 --2- 172.21.15.158:0/2623462235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c15dad0 0x7fcc1c0cdfa0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 --2- 172.21.15.158:0/2623462235 >> v2:172.21.15.172:3300/0 conn(0x7fcc1c15deb0 0x7fcc1c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 -- 172.21.15.158:0/2623462235 >> 172.21.15.158:0/2623462235 conn(0x7fcc1c099a10 msgr2=0x7fcc1c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.917+0000 7fcc231e8700 1 -- 172.21.15.158:0/2623462235 shutdown_connections 2023-10-16T09:44:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.918+0000 7fcc231e8700 1 -- 172.21.15.158:0/2623462235 wait complete. 2023-10-16T09:44:53.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.919+0000 7fcc231e8700 1 Processor -- start 2023-10-16T09:44:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.919+0000 7fcc231e8700 1 -- start start 2023-10-16T09:44:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c0cea20 0x7fcc1c145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c15dad0 0x7fcc1c1f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc1c15deb0 0x7fcc1c06e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c149de0 con 0x7fcc1c0cea20 2023-10-16T09:44:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0c8f60 con 0x7fcc1c15deb0 2023-10-16T09:44:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c146780 con 0x7fcc1c15dad0 2023-10-16T09:44:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c0cea20 0x7fcc1c145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c0cea20 0x7fcc1c145f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48672/0 (socket says 172.21.15.158:48672) 2023-10-16T09:44:53.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 -- 172.21.15.158:0/1643259792 learned_addr learned my addr 172.21.15.158:0/1643259792 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:53.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc21785700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc1c15deb0 0x7fcc1c06e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc1bfff700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c15dad0 0x7fcc1c1f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 -- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c15dad0 msgr2=0x7fcc1c1f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c15dad0 0x7fcc1c1f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 -- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc1c15deb0 msgr2=0x7fcc1c06e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:53.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc1c15deb0 0x7fcc1c06e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:53.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 -- 172.21.15.158:0/1643259792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10009550 con 0x7fcc1c0cea20 2023-10-16T09:44:53.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc20f84700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c0cea20 0x7fcc1c145f50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000b0e0 tx=0x7fcc10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:53.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc197fa700 1 -- 172.21.15.158:0/1643259792 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10004480 con 0x7fcc1c0cea20 2023-10-16T09:44:53.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc197fa700 1 -- 172.21.15.158:0/1643259792 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1001f070 con 0x7fcc1c0cea20 2023-10-16T09:44:53.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c146a60 con 0x7fcc1c0cea20 2023-10-16T09:44:53.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc197fa700 1 -- 172.21.15.158:0/1643259792 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc100038e0 con 0x7fcc1c0cea20 2023-10-16T09:44:53.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.920+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c146490 con 0x7fcc1c0cea20 2023-10-16T09:44:53.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.921+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c0cc260 con 0x7fcc1c0cea20 2023-10-16T09:44:53.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.922+0000 7fcc197fa700 1 -- 172.21.15.158:0/1643259792 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc10021030 con 0x7fcc1c0cea20 2023-10-16T09:44:53.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.922+0000 7fcc197fa700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcc0006af00 0x7fcc0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:53.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.922+0000 7fcc197fa700 1 -- 172.21.15.158:0/1643259792 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcc1008ab80 con 0x7fcc1c0cea20 2023-10-16T09:44:53.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.922+0000 7fcc1bfff700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcc0006af00 0x7fcc0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:53.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.923+0000 7fcc1bfff700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcc0006af00 0x7fcc0006d3c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c09e3a0 tx=0x7fcc08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:53.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:53.927+0000 7fcc197fa700 1 -- 172.21.15.158:0/1643259792 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc1c0cc260 con 0x7fcc1c0cea20 2023-10-16T09:44:54.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.171+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fcc1c0cc260 con 0x7fcc1c0cea20 2023-10-16T09:44:54.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.391+0000 7fcc197fa700 1 -- 172.21.15.158:0/1643259792 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7fcc1c0cc260 con 0x7fcc1c0cea20 2023-10-16T09:44:54.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.393+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcc0006af00 msgr2=0x7fcc0006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.393+0000 7fcc231e8700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcc0006af00 0x7fcc0006d3c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c09e3a0 tx=0x7fcc08008040 comp rx=0 tx=0).stop 2023-10-16T09:44:54.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.393+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c0cea20 msgr2=0x7fcc1c145f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.393+0000 7fcc231e8700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c0cea20 0x7fcc1c145f50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000b0e0 tx=0x7fcc10004750 comp rx=0 tx=0).stop 2023-10-16T09:44:54.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.394+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 shutdown_connections 2023-10-16T09:44:54.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.394+0000 7fcc231e8700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc1c15deb0 0x7fcc1c06e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.394+0000 7fcc231e8700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc1c0cea20 0x7fcc1c145f50 secure :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000b0e0 tx=0x7fcc10004750 comp rx=0 tx=0).stop 2023-10-16T09:44:54.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.394+0000 7fcc231e8700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fcc0006af00 0x7fcc0006d3c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.394+0000 7fcc231e8700 1 --2- 172.21.15.158:0/1643259792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc1c15dad0 0x7fcc1c1f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.394+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 >> 172.21.15.158:0/1643259792 conn(0x7fcc1c099a10 msgr2=0x7fcc1c06f220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:54.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.396+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 shutdown_connections 2023-10-16T09:44:54.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.396+0000 7fcc231e8700 1 -- 172.21.15.158:0/1643259792 wait complete. 2023-10-16T09:44:54.412 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T09:44:54.538 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:54.537+0000 7fcbcd32c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:44:54.545 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:54.542+0000 7f5df8e3b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:44:54.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.544+0000 7f8f6c2b2700 1 Processor -- start 2023-10-16T09:44:54.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.544+0000 7f8f6c2b2700 1 -- start start 2023-10-16T09:44:54.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.545+0000 7f8f6c2b2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.545+0000 7f8f6c2b2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6415ec50 con 0x7f8f6406af20 2023-10-16T09:44:54.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.545+0000 7f8f6c2b2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f64161450 con 0x7f8f640ce7f0 2023-10-16T09:44:54.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.545+0000 7f8f6c2b2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64161730 con 0x7f8f6406ab40 2023-10-16T09:44:54.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.545+0000 7f8f6984d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8f6406af20 0x7f8f640719f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48882/0 (socket says 172.21.15.158:48882) 2023-10-16T09:44:54.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.545+0000 7f8f6984d700 1 -- 172.21.15.158:0/698529083 learned_addr learned my addr 172.21.15.158:0/698529083 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:44:54.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.545+0000 7f8f6a04e700 1 --2- 172.21.15.158:0/698529083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162506803 0 0) 0x7f8f54001110 con 0x7f8f6406af20 2023-10-16T09:44:54.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f500035b0 con 0x7f8f6406af20 2023-10-16T09:44:54.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059688045 0 0) 0x7f8f64161450 con 0x7f8f640ce7f0 2023-10-16T09:44:54.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f54001110 con 0x7f8f640ce7f0 2023-10-16T09:44:54.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6a04e700 1 -- 172.21.15.158:0/698529083 >> v1:172.21.15.172:6789/0 conn(0x7f8f640ce7f0 legacy=0x7f8f6415e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6a04e700 1 -- 172.21.15.158:0/698529083 >> v1:172.21.15.158:6789/0 conn(0x7f8f6406af20 legacy=0x7f8f640719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776553968 0 0) 0x7f8f500035b0 con 0x7f8f6406af20 2023-10-16T09:44:54.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6a04e700 1 -- 172.21.15.158:0/698529083 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64161a10 con 0x7f8f6406ab40 2023-10-16T09:44:54.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6a04e700 1 --2- 172.21.15.158:0/698529083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ce2b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009870 tx=0x7f8f60009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc0d95f90faf372f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:54.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54001800 con 0x7f8f6406ab40 2023-10-16T09:44:54.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f60004030 con 0x7f8f6406ab40 2023-10-16T09:44:54.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.546+0000 7f8f6904c700 1 -- 172.21.15.158:0/698529083 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f600043c0 con 0x7f8f6406ab40 2023-10-16T09:44:54.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.547+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/698529083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 msgr2=0x7f8f640ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.547+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/698529083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ce2b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009870 tx=0x7f8f60009b80 comp rx=0 tx=0).stop 2023-10-16T09:44:54.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.547+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/698529083 shutdown_connections 2023-10-16T09:44:54.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.547+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/698529083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ce2b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.547+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/698529083 >> 172.21.15.158:0/698529083 conn(0x7f8f640b1f00 msgr2=0x7f8f640b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:54.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.547+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/698529083 shutdown_connections 2023-10-16T09:44:54.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.547+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/698529083 wait complete. 2023-10-16T09:44:54.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.548+0000 7f8f6c2b2700 1 Processor -- start 2023-10-16T09:44:54.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.548+0000 7f8f6c2b2700 1 -- start start 2023-10-16T09:44:54.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6c2b2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ae450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6c2b2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f6406af20 0x7f8f6415b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6c2b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f640ce7f0 0x7f8f64159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6c2b2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640ae990 con 0x7f8f640ce7f0 2023-10-16T09:44:54.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6c2b2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640aec10 con 0x7f8f6406af20 2023-10-16T09:44:54.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6c2b2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640aeef0 con 0x7f8f6406ab40 2023-10-16T09:44:54.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f640ce7f0 0x7f8f64159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f640ce7f0 0x7f8f64159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48716/0 (socket says 172.21.15.158:48716) 2023-10-16T09:44:54.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 -- 172.21.15.158:0/1581474361 learned_addr learned my addr 172.21.15.158:0/1581474361 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:54.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a04e700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6984d700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f6406af20 0x7f8f6415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 -- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 msgr2=0x7f8f640ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 -- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f6406af20 msgr2=0x7f8f6415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f6406af20 0x7f8f6415b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.549+0000 7f8f6a84f700 1 -- 172.21.15.158:0/1581474361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f600094a0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.550+0000 7f8f6a84f700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f640ce7f0 0x7f8f64159820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00cc20 tx=0x7f8f5c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:54.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.550+0000 7f8f5affd700 1 -- 172.21.15.158:0/1581474361 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c003c00 con 0x7f8f640ce7f0 2023-10-16T09:44:54.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.550+0000 7f8f5affd700 1 -- 172.21.15.158:0/1581474361 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c003d60 con 0x7f8f640ce7f0 2023-10-16T09:44:54.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.550+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f640af1d0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.550+0000 7f8f5affd700 1 -- 172.21.15.158:0/1581474361 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00aae0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.550+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6415ba80 con 0x7f8f640ce7f0 2023-10-16T09:44:54.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.551+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f640af6f0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.551+0000 7f8f5affd700 1 -- 172.21.15.158:0/1581474361 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f8f5c00ac40 con 0x7f8f640ce7f0 2023-10-16T09:44:54.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.552+0000 7f8f5affd700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 0x7f8f4806d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.552+0000 7f8f6a04e700 1 -- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 msgr2=0x7f8f4806d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:54.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.552+0000 7f8f5affd700 1 -- 172.21.15.158:0/1581474361 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8f5c0897b0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.552+0000 7f8f6a04e700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 0x7f8f4806d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:44:54.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.556+0000 7f8f5affd700 1 -- 172.21.15.158:0/1581474361 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f640af6f0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.605 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:54.603+0000 7fcbcd32c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:44:54.615 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:54.612+0000 7f5df8e3b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:44:54.651 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:44:54.654 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:44:54.680 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:54.679+0000 7fcbcd32c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:44:54.691 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:54.688+0000 7f5df8e3b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:44:54.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.753+0000 7f8f6a04e700 1 -- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 msgr2=0x7f8f4806d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:54.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.753+0000 7f8f6a04e700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 0x7f8f4806d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:44:54.759 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:54.758+0000 7fcbcd32c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:44:54.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.759+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f8f640af6f0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.761+0000 7f8f5affd700 1 -- 172.21.15.158:0/1581474361 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8f640af6f0 con 0x7f8f640ce7f0 2023-10-16T09:44:54.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 msgr2=0x7f8f4806d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:44:54.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 0x7f8f4806d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f640ce7f0 msgr2=0x7f8f64159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f640ce7f0 0x7f8f64159820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00cc20 tx=0x7f8f5c00cf30 comp rx=0 tx=0).stop 2023-10-16T09:44:54.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 shutdown_connections 2023-10-16T09:44:54.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f640ce7f0 0x7f8f64159820 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7f8f4806af50 0x7f8f4806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f6406ab40 0x7f8f640ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 --2- 172.21.15.158:0/1581474361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f6406af20 0x7f8f6415b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.763+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 >> 172.21.15.158:0/1581474361 conn(0x7f8f640b1f00 msgr2=0x7f8f6415c970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:54.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.764+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 shutdown_connections 2023-10-16T09:44:54.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.764+0000 7f8f6c2b2700 1 -- 172.21.15.158:0/1581474361 wait complete. 2023-10-16T09:44:54.773 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:54.770+0000 7f5df8e3b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:44:54.778 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T09:44:54.902 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:54.900+0000 7fcbcd32c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:44:54.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.902+0000 7feb24e5e700 1 Processor -- start 2023-10-16T09:44:54.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.902+0000 7feb24e5e700 1 -- start start 2023-10-16T09:44:54.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb24e5e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb2015dad0 0x7feb200cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb24e5e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb200ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb24e5e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb200cea20 0x7feb200c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb24e5e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200d0ae0 con 0x7feb2015dad0 2023-10-16T09:44:54.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb24e5e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200d12d0 con 0x7feb200cea20 2023-10-16T09:44:54.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb24e5e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb200cbdf0 con 0x7feb2015deb0 2023-10-16T09:44:54.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb2015dad0 0x7feb200cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb2015dad0 0x7feb200cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48744/0 (socket says 172.21.15.158:48744) 2023-10-16T09:44:54.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 -- 172.21.15.158:0/2317431725 learned_addr learned my addr 172.21.15.158:0/2317431725 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:54.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1ed9d700 1 --2- 172.21.15.158:0/2317431725 >> v2:172.21.15.172:3300/0 conn(0x7feb200cea20 0x7feb200c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1dd9b700 1 --2- 172.21.15.158:0/2317431725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb200ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 -- 172.21.15.158:0/2317431725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 msgr2=0x7feb200ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:54.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 --2- 172.21.15.158:0/2317431725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb200ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 -- 172.21.15.158:0/2317431725 >> v2:172.21.15.172:3300/0 conn(0x7feb200cea20 msgr2=0x7feb200c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:54.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 --2- 172.21.15.158:0/2317431725 >> v2:172.21.15.172:3300/0 conn(0x7feb200cea20 0x7feb200c3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 -- 172.21.15.158:0/2317431725 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200cc280 con 0x7feb2015dad0 2023-10-16T09:44:54.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1e59c700 1 --2- 172.21.15.158:0/2317431725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb200cdfa0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7feb20071950 tx=0x7feb10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb156e748bba5844 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:54.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.904+0000 7feb1d59a700 1 -- 172.21.15.158:0/2317431725 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10011830 con 0x7feb2015dad0 2023-10-16T09:44:54.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb1d59a700 1 -- 172.21.15.158:0/2317431725 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10011990 con 0x7feb2015dad0 2023-10-16T09:44:54.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb1d59a700 1 -- 172.21.15.158:0/2317431725 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10011d20 con 0x7feb2015dad0 2023-10-16T09:44:54.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb24e5e700 1 -- 172.21.15.158:0/2317431725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 msgr2=0x7feb200cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb24e5e700 1 --2- 172.21.15.158:0/2317431725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb200cdfa0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7feb20071950 tx=0x7feb10009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb24e5e700 1 -- 172.21.15.158:0/2317431725 shutdown_connections 2023-10-16T09:44:54.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb24e5e700 1 --2- 172.21.15.158:0/2317431725 >> v2:172.21.15.172:3300/0 conn(0x7feb200cea20 0x7feb200c3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb24e5e700 1 --2- 172.21.15.158:0/2317431725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb200cdfa0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb24e5e700 1 --2- 172.21.15.158:0/2317431725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb200ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.905+0000 7feb24e5e700 1 -- 172.21.15.158:0/2317431725 >> 172.21.15.158:0/2317431725 conn(0x7feb20099a10 msgr2=0x7feb200d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:54.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.906+0000 7feb24e5e700 1 -- 172.21.15.158:0/2317431725 shutdown_connections 2023-10-16T09:44:54.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.906+0000 7feb24e5e700 1 -- 172.21.15.158:0/2317431725 wait complete. 2023-10-16T09:44:54.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.908+0000 7feb24e5e700 1 Processor -- start 2023-10-16T09:44:54.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.908+0000 7feb24e5e700 1 -- start start 2023-10-16T09:44:54.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb24e5e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb200cea20 0x7feb20159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb24e5e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb201ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb24e5e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb20146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb24e5e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb20159a60 con 0x7feb2015dad0 2023-10-16T09:44:54.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb24e5e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb20158480 con 0x7feb200cea20 2023-10-16T09:44:54.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb24e5e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb20158760 con 0x7feb2015deb0 2023-10-16T09:44:54.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb201ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb201ef5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48752/0 (socket says 172.21.15.158:48752) 2023-10-16T09:44:54.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 -- 172.21.15.158:0/3698838717 learned_addr learned my addr 172.21.15.158:0/3698838717 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:54.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1e59c700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb200cea20 0x7feb20159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1ed9d700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb20146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:54.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 -- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 msgr2=0x7feb20146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb20146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 -- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb200cea20 msgr2=0x7feb20159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:54.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb200cea20 0x7feb20159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:54.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 -- 172.21.15.158:0/3698838717 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb10009550 con 0x7feb2015dad0 2023-10-16T09:44:54.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.909+0000 7feb1dd9b700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb201ef5a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7feb14010a80 tx=0x7feb14010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:54.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.910+0000 7feb0effd700 1 -- 172.21.15.158:0/3698838717 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14009590 con 0x7feb2015dad0 2023-10-16T09:44:54.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.910+0000 7feb0effd700 1 -- 172.21.15.158:0/3698838717 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1400ec60 con 0x7feb2015dad0 2023-10-16T09:44:54.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.910+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb20158a40 con 0x7feb2015dad0 2023-10-16T09:44:54.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.910+0000 7feb0effd700 1 -- 172.21.15.158:0/3698838717 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14011450 con 0x7feb2015dad0 2023-10-16T09:44:54.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.910+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb20159000 con 0x7feb2015dad0 2023-10-16T09:44:54.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.911+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb201591c0 con 0x7feb2015dad0 2023-10-16T09:44:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.911+0000 7feb0effd700 1 -- 172.21.15.158:0/3698838717 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7feb140115b0 con 0x7feb2015dad0 2023-10-16T09:44:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.912+0000 7feb0effd700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 0x7feb0006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.912+0000 7feb1e59c700 1 -- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 msgr2=0x7feb0006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.912+0000 7feb0effd700 1 -- 172.21.15.158:0/3698838717 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feb1408b860 con 0x7feb2015dad0 2023-10-16T09:44:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.912+0000 7feb1e59c700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 0x7feb0006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:44:54.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:54.916+0000 7feb0effd700 1 -- 172.21.15.158:0/3698838717 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb140567d0 con 0x7feb2015dad0 2023-10-16T09:44:54.926 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:54.924+0000 7f5df8e3b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:44:55.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.113+0000 7feb1e59c700 1 -- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 msgr2=0x7feb0006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:55.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.113+0000 7feb1e59c700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 0x7feb0006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:44:55.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.122+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7feb200681b0 con 0x7feb2015dad0 2023-10-16T09:44:55.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.125+0000 7feb0effd700 1 -- 172.21.15.158:0/3698838717 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7feb1401e020 con 0x7feb2015dad0 2023-10-16T09:44:55.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 msgr2=0x7feb0006d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:44:55.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 0x7feb0006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 msgr2=0x7feb201ef5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb201ef5a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7feb14010a80 tx=0x7feb14010d90 comp rx=0 tx=0).stop 2023-10-16T09:44:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 shutdown_connections 2023-10-16T09:44:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7feb0006af50 0x7feb0006d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb2015deb0 0x7feb20146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb200cea20 0x7feb20159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 --2- 172.21.15.158:0/3698838717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2015dad0 0x7feb201ef5a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 >> 172.21.15.158:0/3698838717 conn(0x7feb20099a10 msgr2=0x7feb200c3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 shutdown_connections 2023-10-16T09:44:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.127+0000 7feb24e5e700 1 -- 172.21.15.158:0/3698838717 wait complete. 2023-10-16T09:44:55.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:55.143 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T09:44:55.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:44:55.165 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-10-16T09:44:55.222 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2023-10-16T09:44:55.227 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:55.225+0000 7fcbcd32c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:44:55.265 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:55.262+0000 7f5df8e3b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:44:55.341 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:55.339+0000 7fcbcd32c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:44:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.387+0000 7fa0a2c6b700 1 Processor -- start 2023-10-16T09:44:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.387+0000 7fa0a2c6b700 1 -- start start 2023-10-16T09:44:55.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a2c6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa09c15b540 0x7fa09c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a2c6b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa09c06f180 0x7fa09c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a2c6b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c15ee60 con 0x7fa09c06fea0 2023-10-16T09:44:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a2c6b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09c15b920 con 0x7fa09c15b540 2023-10-16T09:44:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a2c6b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09c15e2e0 con 0x7fa09c06f180 2023-10-16T09:44:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a0a07700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa09c15b540 0x7fa09c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a0a07700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa09c15b540 0x7fa09c06d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48786/0 (socket says 172.21.15.158:48786) 2023-10-16T09:44:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3067547214 learned_addr learned my addr 172.21.15.158:0/3067547214 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa0a1208700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa09c06fea0 0x7fa09c161860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:48966/0 (socket says 172.21.15.158:48966) 2023-10-16T09:44:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.388+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3067547214 >> v2:172.21.15.172:3300/0 conn(0x7fa09c06f180 0x7fa09c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3067547214 >> v2:172.21.15.172:3300/0 conn(0x7fa09c06f180 msgr2=0x7fa09c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:44:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa0a0a07700 1 --2- 172.21.15.158:0/3067547214 >> v2:172.21.15.172:3300/0 conn(0x7fa09c06f180 0x7fa09c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3067547214 >> v1:172.21.15.158:6789/0 conn(0x7fa09c06fea0 legacy=0x7fa09c161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3067547214 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c15e5c0 con 0x7fa09c15b540 2023-10-16T09:44:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa09b7fe700 1 -- 172.21.15.158:0/3067547214 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921655635 0 0) 0x7fa09c15ee60 con 0x7fa09c06fea0 2023-10-16T09:44:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa0a0a07700 1 --2- 172.21.15.158:0/3067547214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c15b540 0x7fa09c06d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa09c15ba60 tx=0x7fa090009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7cb43a0a467ac6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa09b7fe700 1 -- 172.21.15.158:0/3067547214 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090004230 con 0x7fa09c15b540 2023-10-16T09:44:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa09b7fe700 1 -- 172.21.15.158:0/3067547214 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa090004390 con 0x7fa09c15b540 2023-10-16T09:44:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa09b7fe700 1 -- 172.21.15.158:0/3067547214 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900036a0 con 0x7fa09c15b540 2023-10-16T09:44:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3067547214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c15b540 msgr2=0x7fa09c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.389+0000 7fa0a2c6b700 1 --2- 172.21.15.158:0/3067547214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c15b540 0x7fa09c06d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa09c15ba60 tx=0x7fa090009bd0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.390+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3067547214 shutdown_connections 2023-10-16T09:44:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.390+0000 7fa0a2c6b700 1 --2- 172.21.15.158:0/3067547214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c15b540 0x7fa09c06d9c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.390+0000 7fa0a2c6b700 1 --2- 172.21.15.158:0/3067547214 >> v2:172.21.15.172:3300/0 conn(0x7fa09c06f180 0x7fa09c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.390+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3067547214 >> 172.21.15.158:0/3067547214 conn(0x7fa09c0d03b0 msgr2=0x7fa09c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:44:55.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.390+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3067547214 shutdown_connections 2023-10-16T09:44:55.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.390+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3067547214 wait complete. 2023-10-16T09:44:55.398 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:55.391+0000 7f5df8e3b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:44:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.391+0000 7fa0a2c6b700 1 Processor -- start 2023-10-16T09:44:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.391+0000 7fa0a2c6b700 1 -- start start 2023-10-16T09:44:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a2c6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c06f180 0x7fa09c0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a2c6b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa09c06fea0 0x7fa09c0b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a2c6b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa09c15b540 0x7fa09c1f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a2c6b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0acbe0 con 0x7fa09c06f180 2023-10-16T09:44:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a2c6b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0acec0 con 0x7fa09c15b540 2023-10-16T09:44:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a2c6b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0ad1a0 con 0x7fa09c06fea0 2023-10-16T09:44:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c06f180 0x7fa09c0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c06f180 0x7fa09c0b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48792/0 (socket says 172.21.15.158:48792) 2023-10-16T09:44:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3540914305 learned_addr learned my addr 172.21.15.158:0/3540914305 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:44:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa09c06fea0 0x7fa09c0b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a1208700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa09c15b540 0x7fa09c1f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:44:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa09c06fea0 msgr2=0x7fa09c0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa09c06fea0 0x7fa09c0b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa09c15b540 msgr2=0x7fa09c1f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:44:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa09c15b540 0x7fa09c1f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 -- 172.21.15.158:0/3540914305 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa090009550 con 0x7fa09c06f180 2023-10-16T09:44:55.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.392+0000 7fa0a0a07700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c06f180 0x7fa09c0b3730 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa090009c10 tx=0x7fa090004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:44:55.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.394+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090003d20 con 0x7fa09c06f180 2023-10-16T09:44:55.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.394+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa090011630 con 0x7fa09c06f180 2023-10-16T09:44:55.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.394+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c0ad480 con 0x7fa09c06f180 2023-10-16T09:44:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.394+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090003ea0 con 0x7fa09c06f180 2023-10-16T09:44:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.394+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c0ac890 con 0x7fa09c06f180 2023-10-16T09:44:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.395+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c1612a0 con 0x7fa09c06f180 2023-10-16T09:44:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.395+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fa0900119e0 con 0x7fa09c06f180 2023-10-16T09:44:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.396+0000 7fa0997fa700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 0x7fa08006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:44:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.396+0000 7fa09bfff700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 msgr2=0x7fa08006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.396+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa09008ae80 con 0x7fa09c06f180 2023-10-16T09:44:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.396+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 0x7fa08006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:44:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.400+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa090055e80 con 0x7fa09c06f180 2023-10-16T09:44:55.414 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:55.413+0000 7fcbcd32c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:44:55.470 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:55.468+0000 7f5df8e3b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:44:55.494 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:55.492+0000 7fcbcd32c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:44:55.553 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:55.550+0000 7f5df8e3b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:44:55.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.597+0000 7fa09bfff700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 msgr2=0x7fa08006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:55.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.597+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 0x7fa08006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:44:55.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.630+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 --> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa09c0ad030 con 0x7fa08006af50 2023-10-16T09:44:55.861 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:55.859+0000 7fcbcd32c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:44:55.923 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:55.921+0000 7f5df8e3b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:44:56.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.998+0000 7fa09bfff700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 msgr2=0x7fa08006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:56.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:55.998+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 0x7fa08006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T09:44:56.427 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:56.426+0000 7fcbcd32c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:44:56.491 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:56.488+0000 7f5df8e3b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:44:56.495 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:56.495+0000 7fcbcd32c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:44:56.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:56.558+0000 7f5df8e3b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:44:56.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:56.799+0000 7fa09bfff700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 msgr2=0x7fa08006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:56.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:56.799+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 0x7fa08006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T09:44:57.129 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:57.128+0000 7fcbcd32c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:44:57.190 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:57.187+0000 7f5df8e3b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:44:57.207 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:57.207+0000 7fcbcd32c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:44:57.266 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:57.263+0000 7f5df8e3b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:44:57.350 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:57.349+0000 7fcbcd32c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:44:57.408 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:57.405+0000 7f5df8e3b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:44:57.575 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:57.574+0000 7fcbcd32c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:44:57.632 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:57.629+0000 7f5df8e3b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:44:57.648 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:57.647+0000 7fcbcd32c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:44:57.705 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:57.702+0000 7f5df8e3b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:44:57.917 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:57.916+0000 7fcbcd32c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:44:57.974 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:57.971+0000 7f5df8e3b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:44:58.134 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:58.133+0000 7fcbcd32c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:44:58.190 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:58.187+0000 7f5df8e3b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:44:58.279 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:58.278+0000 7fcbcd32c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:44:58.336 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:58.334+0000 7f5df8e3b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:44:58.391 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:58.391+0000 7fcbcd32c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:44:58.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:58.400+0000 7fa09bfff700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 msgr2=0x7fa08006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4024924909 2023-10-16T09:44:58.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:58.400+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 0x7fa08006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T09:44:58.449 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:58.447+0000 7f5df8e3b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:44:58.544 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:58.543+0000 7fcbcd32c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:44:58.601 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:58.599+0000 7f5df8e3b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:44:58.806 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:58.806+0000 7fcbcd32c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:44:58.864 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:58.862+0000 7f5df8e3b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:44:58.964 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:58.963+0000 7fcbcd32c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:44:59.021 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.019+0000 7f5df8e3b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:44:59.482 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:44:59.481+0000 7fcbcd32c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:44:59.537 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.536+0000 7f5df8e3b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:44:59.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:59.551+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 7) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fa090056250 con 0x7fa09c06f180 2023-10-16T09:44:59.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:59.561+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 8) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fa090059da0 con 0x7fa09c06f180 2023-10-16T09:44:59.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:59.561+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 msgr2=0x7fa08006d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:44:59.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:44:59.561+0000 7fa0997fa700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/4024924909,v1:172.21.15.158:6833/4024924909] conn(0x7fa08006af50 0x7fa08006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:44:59.732 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6be16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.732 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6be16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.732 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6be16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.732 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6be16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.733 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6be16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.733 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6ee1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.733 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6ee1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.733 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6ee1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.733 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6ee1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:44:59.734 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:44:59.730+0000 7f5d6ee1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:45:00.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.567+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 9) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa09006ff20 con 0x7fa09c06f180 2023-10-16T09:45:00.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.568+0000 7fa0997fa700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa08001ca90 0x7fa08001ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:00.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.568+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa09c0ad030 con 0x7fa08001ca90 2023-10-16T09:45:00.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.568+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa08001ca90 0x7fa08001ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:00.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.569+0000 7fa09bfff700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa08001ca90 0x7fa08001ce70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa088009800 tx=0x7fa088009210 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:00.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.651+0000 7fa0997fa700 1 -- 172.21.15.158:0/3540914305 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa09c0ad030 con 0x7fa08001ca90 2023-10-16T09:45:00.653 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir0 2023-10-16T09:45:00.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa08001ca90 msgr2=0x7fa08001ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:00.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa08001ca90 0x7fa08001ce70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa088009800 tx=0x7fa088009210 comp rx=0 tx=0).stop 2023-10-16T09:45:00.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c06f180 msgr2=0x7fa09c0b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:00.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 --2- 172.21.15.158:0/3540914305 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c06f180 0x7fa09c0b3730 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa090009c10 tx=0x7fa090004750 comp rx=0 tx=0).stop 2023-10-16T09:45:00.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a1208700 1 -- 172.21.15.158:0/3540914305 reap_dead start 2023-10-16T09:45:00.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 shutdown_connections 2023-10-16T09:45:00.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 >> 172.21.15.158:0/3540914305 conn(0x7fa09c0d03b0 msgr2=0x7fa09c0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:00.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 shutdown_connections 2023-10-16T09:45:00.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.654+0000 7fa0a2c6b700 1 -- 172.21.15.158:0/3540914305 wait complete. 2023-10-16T09:45:00.668 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-10-16T09:45:00.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:00.669 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-10-16T09:45:00.687 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2023-10-16T09:45:00.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.849+0000 7ff963efa700 1 Processor -- start 2023-10-16T09:45:00.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.849+0000 7ff963efa700 1 -- start start 2023-10-16T09:45:00.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff963efa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff95c15dc40 0x7ff95c15ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:00.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff963efa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 0x7ff95c06f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:00.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff963efa700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95c15f380 con 0x7ff95c06d9c0 2023-10-16T09:45:00.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff963efa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95c06e300 con 0x7ff95c15dc40 2023-10-16T09:45:00.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff963efa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c071e70 con 0x7ff95c06de70 2023-10-16T09:45:00.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961c96700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff95c15dc40 0x7ff95c15ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:00.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961495700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff95c06d9c0 0x7ff95c06f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49144/0 (socket says 172.21.15.158:49144) 2023-10-16T09:45:00.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961c96700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff95c15dc40 0x7ff95c15ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48966/0 (socket says 172.21.15.158:48966) 2023-10-16T09:45:00.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961495700 1 -- 172.21.15.158:0/747940672 learned_addr learned my addr 172.21.15.158:0/747940672 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:00.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961c96700 1 -- 172.21.15.158:0/747940672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 msgr2=0x7ff95c06f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:00.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961c96700 1 --2- 172.21.15.158:0/747940672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 0x7ff95c06f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961c96700 1 -- 172.21.15.158:0/747940672 >> v1:172.21.15.158:6789/0 conn(0x7ff95c06d9c0 legacy=0x7ff95c06f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961c96700 1 -- 172.21.15.158:0/747940672 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c072150 con 0x7ff95c15dc40 2023-10-16T09:45:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff960c94700 1 -- 172.21.15.158:0/747940672 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178336172 0 0) 0x7ff94c001110 con 0x7ff95c06d9c0 2023-10-16T09:45:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.850+0000 7ff961c96700 1 --2- 172.21.15.158:0/747940672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c15dc40 0x7ff95c15ee40 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff958009840 tx=0x7ff958009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531cff5add0ebd28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:00.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff960c94700 1 -- 172.21.15.158:0/747940672 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958011830 con 0x7ff95c15dc40 2023-10-16T09:45:00.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff960c94700 1 -- 172.21.15.158:0/747940672 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff958011990 con 0x7ff95c15dc40 2023-10-16T09:45:00.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff960c94700 1 -- 172.21.15.158:0/747940672 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958011d20 con 0x7ff95c15dc40 2023-10-16T09:45:00.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 -- 172.21.15.158:0/747940672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c15dc40 msgr2=0x7ff95c15ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:00.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 --2- 172.21.15.158:0/747940672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c15dc40 0x7ff95c15ee40 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff958009840 tx=0x7ff958009b50 comp rx=0 tx=0).stop 2023-10-16T09:45:00.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 -- 172.21.15.158:0/747940672 shutdown_connections 2023-10-16T09:45:00.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 --2- 172.21.15.158:0/747940672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c15dc40 0x7ff95c15ee40 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:00.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 --2- 172.21.15.158:0/747940672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 0x7ff95c06f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:00.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 -- 172.21.15.158:0/747940672 >> 172.21.15.158:0/747940672 conn(0x7ff95c0d03b0 msgr2=0x7ff95c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:00.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 -- 172.21.15.158:0/747940672 shutdown_connections 2023-10-16T09:45:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.851+0000 7ff963efa700 1 -- 172.21.15.158:0/747940672 wait complete. 2023-10-16T09:45:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.852+0000 7ff963efa700 1 Processor -- start 2023-10-16T09:45:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.852+0000 7ff963efa700 1 -- start start 2023-10-16T09:45:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff963efa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c06d9c0 0x7ff95c0cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff963efa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 0x7ff95c15c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff963efa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff95c15dc40 0x7ff95c0cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff963efa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0cdd30 con 0x7ff95c06d9c0 2023-10-16T09:45:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff963efa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c15cd90 con 0x7ff95c15dc40 2023-10-16T09:45:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff963efa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0ae760 con 0x7ff95c06de70 2023-10-16T09:45:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c06d9c0 0x7ff95c0cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c06d9c0 0x7ff95c0cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48972/0 (socket says 172.21.15.158:48972) 2023-10-16T09:45:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 -- 172.21.15.158:0/2385639724 learned_addr learned my addr 172.21.15.158:0/2385639724 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961495700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 0x7ff95c15c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff962497700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff95c15dc40 0x7ff95c0cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 -- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 msgr2=0x7ff95c15c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 0x7ff95c15c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 -- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff95c15dc40 msgr2=0x7ff95c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff95c15dc40 0x7ff95c0cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.853+0000 7ff961c96700 1 -- 172.21.15.158:0/2385639724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9580094a0 con 0x7ff95c06d9c0 2023-10-16T09:45:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.854+0000 7ff961c96700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c06d9c0 0x7ff95c0cd7f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff9580090b0 tx=0x7ff958004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.854+0000 7ff9527fc700 1 -- 172.21.15.158:0/2385639724 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9580036a0 con 0x7ff95c06d9c0 2023-10-16T09:45:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.854+0000 7ff9527fc700 1 -- 172.21.15.158:0/2385639724 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95801e070 con 0x7ff95c06d9c0 2023-10-16T09:45:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.854+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95c0ae9e0 con 0x7ff95c06d9c0 2023-10-16T09:45:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.854+0000 7ff9527fc700 1 -- 172.21.15.158:0/2385639724 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958003a20 con 0x7ff95c06d9c0 2023-10-16T09:45:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.854+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c0aefa0 con 0x7ff95c06d9c0 2023-10-16T09:45:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.855+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c0681b0 con 0x7ff95c06d9c0 2023-10-16T09:45:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.855+0000 7ff9527fc700 1 -- 172.21.15.158:0/2385639724 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 9) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff958003b80 con 0x7ff95c06d9c0 2023-10-16T09:45:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.855+0000 7ff9527fc700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff94006ada0 0x7ff94006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.856+0000 7ff9527fc700 1 -- 172.21.15.158:0/2385639724 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7ff95808a6b0 con 0x7ff95c06d9c0 2023-10-16T09:45:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.856+0000 7ff961495700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff94006ada0 0x7ff94006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:00.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.856+0000 7ff961495700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff94006ada0 0x7ff94006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff95c15c7e0 tx=0x7ff94c007a40 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:00.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:00.860+0000 7ff9527fc700 1 -- 172.21.15.158:0/2385639724 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff958055860 con 0x7ff95c06d9c0 2023-10-16T09:45:01.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.078+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff95c072710 con 0x7ff94006ada0 2023-10-16T09:45:01.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.138+0000 7ff9527fc700 1 -- 172.21.15.158:0/2385639724 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ff94c005920 con 0x7ff94006ada0 2023-10-16T09:45:01.140 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir1 2023-10-16T09:45:01.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff94006ada0 msgr2=0x7ff94006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff94006ada0 0x7ff94006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff95c15c7e0 tx=0x7ff94c007a40 comp rx=0 tx=0).stop 2023-10-16T09:45:01.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c06d9c0 msgr2=0x7ff95c0cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c06d9c0 0x7ff95c0cd7f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff9580090b0 tx=0x7ff958004750 comp rx=0 tx=0).stop 2023-10-16T09:45:01.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 shutdown_connections 2023-10-16T09:45:01.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff95c15dc40 0x7ff95c0cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff94006ada0 0x7ff94006d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff95c06d9c0 0x7ff95c0cd7f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 --2- 172.21.15.158:0/2385639724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff95c06de70 0x7ff95c15c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 >> 172.21.15.158:0/2385639724 conn(0x7ff95c0d03b0 msgr2=0x7ff95c0b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:01.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 shutdown_connections 2023-10-16T09:45:01.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.140+0000 7ff963efa700 1 -- 172.21.15.158:0/2385639724 wait complete. 2023-10-16T09:45:01.153 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-10-16T09:45:01.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:01.154 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-10-16T09:45:01.173 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2023-10-16T09:45:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.336+0000 7fddda235700 1 Processor -- start 2023-10-16T09:45:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.336+0000 7fddda235700 1 -- start start 2023-10-16T09:45:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddda235700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fddd4160d30 0x7fddd40b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddda235700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fddd40b7590 0x7fddd40d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddda235700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd40b7a20 con 0x7fddd40d0530 2023-10-16T09:45:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddda235700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd40b8910 con 0x7fddd4160d30 2023-10-16T09:45:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddda235700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd40d0ef0 con 0x7fddd40b7590 2023-10-16T09:45:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddd37fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fddd4160d30 0x7fddd40b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddd2ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fddd40d0530 0x7fddd40d19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49160/0 (socket says 172.21.15.158:49160) 2023-10-16T09:45:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.337+0000 7fddd2ffd700 1 -- 172.21.15.158:0/1193918754 learned_addr learned my addr 172.21.15.158:0/1193918754 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd37fe700 1 -- 172.21.15.158:0/1193918754 >> v2:172.21.15.172:3300/0 conn(0x7fddd40b7590 msgr2=0x7fddd40d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd37fe700 1 --2- 172.21.15.158:0/1193918754 >> v2:172.21.15.172:3300/0 conn(0x7fddd40b7590 0x7fddd40d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd37fe700 1 -- 172.21.15.158:0/1193918754 >> v1:172.21.15.158:6789/0 conn(0x7fddd40d0530 legacy=0x7fddd40d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd27fc700 1 -- 172.21.15.158:0/1193918754 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671292548 0 0) 0x7fddd40b7a20 con 0x7fddd40d0530 2023-10-16T09:45:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd37fe700 1 -- 172.21.15.158:0/1193918754 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd40b7c90 con 0x7fddd4160d30 2023-10-16T09:45:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd37fe700 1 --2- 172.21.15.158:0/1193918754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd4160d30 0x7fddd40b8f50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009840 tx=0x7fddc4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ec2ba3de9d009c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd27fc700 1 -- 172.21.15.158:0/1193918754 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4011830 con 0x7fddd4160d30 2023-10-16T09:45:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd27fc700 1 -- 172.21.15.158:0/1193918754 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc4011990 con 0x7fddd4160d30 2023-10-16T09:45:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.338+0000 7fddd27fc700 1 -- 172.21.15.158:0/1193918754 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4011d20 con 0x7fddd4160d30 2023-10-16T09:45:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 -- 172.21.15.158:0/1193918754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd4160d30 msgr2=0x7fddd40b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 --2- 172.21.15.158:0/1193918754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd4160d30 0x7fddd40b8f50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009840 tx=0x7fddc4009b50 comp rx=0 tx=0).stop 2023-10-16T09:45:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 -- 172.21.15.158:0/1193918754 shutdown_connections 2023-10-16T09:45:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 --2- 172.21.15.158:0/1193918754 >> v2:172.21.15.172:3300/0 conn(0x7fddd40b7590 0x7fddd40d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 --2- 172.21.15.158:0/1193918754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd4160d30 0x7fddd40b8f50 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 -- 172.21.15.158:0/1193918754 >> 172.21.15.158:0/1193918754 conn(0x7fddd40669f0 msgr2=0x7fddd4066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 -- 172.21.15.158:0/1193918754 shutdown_connections 2023-10-16T09:45:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.339+0000 7fddda235700 1 -- 172.21.15.158:0/1193918754 wait complete. 2023-10-16T09:45:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.340+0000 7fddda235700 1 Processor -- start 2023-10-16T09:45:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.340+0000 7fddda235700 1 -- start start 2023-10-16T09:45:01.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddda235700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd40b7590 0x7fddd41f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddda235700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddd40d0530 0x7fddd4159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddda235700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddd4160d30 0x7fddd415a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddda235700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd40c6c70 con 0x7fddd40b7590 2023-10-16T09:45:01.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddda235700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4159bb0 con 0x7fddd4160d30 2023-10-16T09:45:01.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddda235700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4159e90 con 0x7fddd40d0530 2023-10-16T09:45:01.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd40b7590 0x7fddd41f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd40b7590 0x7fddd41f25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:48986/0 (socket says 172.21.15.158:48986) 2023-10-16T09:45:01.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 -- 172.21.15.158:0/2871583309 learned_addr learned my addr 172.21.15.158:0/2871583309 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:01.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd2ffd700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddd40d0530 0x7fddd4159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 -- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddd40d0530 msgr2=0x7fddd4159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddd40d0530 0x7fddd4159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd3fff700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddd4160d30 0x7fddd415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 -- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddd4160d30 msgr2=0x7fddd415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddd4160d30 0x7fddd415a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 -- 172.21.15.158:0/2871583309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40094a0 con 0x7fddd40b7590 2023-10-16T09:45:01.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd2ffd700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddd40d0530 0x7fddd4159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:45:01.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd3fff700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddd4160d30 0x7fddd415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:45:01.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddd37fe700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd40b7590 0x7fddd41f25d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009b90 tx=0x7fddc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:01.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.341+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40045f0 con 0x7fddd40b7590 2023-10-16T09:45:01.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.342+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc401e070 con 0x7fddd40b7590 2023-10-16T09:45:01.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.342+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4003ac0 con 0x7fddd40b7590 2023-10-16T09:45:01.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.342+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd40c6e80 con 0x7fddd40b7590 2023-10-16T09:45:01.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.342+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd41453c0 con 0x7fddd40b7590 2023-10-16T09:45:01.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.342+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd406c1a0 con 0x7fddd40b7590 2023-10-16T09:45:01.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.343+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 9) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fddc4003c20 con 0x7fddd40b7590 2023-10-16T09:45:01.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.343+0000 7fddbbfff700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fddc006ada0 0x7fddc006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.343+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fddc408a670 con 0x7fddd40b7590 2023-10-16T09:45:01.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.343+0000 7fddd2ffd700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fddc006ada0 0x7fddc006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.344+0000 7fddd2ffd700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fddc006ada0 0x7fddc006d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddd415ae20 tx=0x7fddc8009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:01.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.348+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fddc4055820 con 0x7fddd40b7590 2023-10-16T09:45:01.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.568+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fddd40b8250 con 0x7fddc006ada0 2023-10-16T09:45:01.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.571+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fddc4055bf0 con 0x7fddd40b7590 2023-10-16T09:45:01.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.640+0000 7fddbbfff700 1 -- 172.21.15.158:0/2871583309 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fddc8001cc0 con 0x7fddc006ada0 2023-10-16T09:45:01.642 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir2 2023-10-16T09:45:01.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fddc006ada0 msgr2=0x7fddc006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fddc006ada0 0x7fddc006d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddd415ae20 tx=0x7fddc8009040 comp rx=0 tx=0).stop 2023-10-16T09:45:01.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd40b7590 msgr2=0x7fddd41f25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd40b7590 0x7fddd41f25d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009b90 tx=0x7fddc4004750 comp rx=0 tx=0).stop 2023-10-16T09:45:01.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 shutdown_connections 2023-10-16T09:45:01.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddd4160d30 0x7fddd415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fddc006ada0 0x7fddc006d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fddd40b7590 0x7fddd41f25d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 --2- 172.21.15.158:0/2871583309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddd40d0530 0x7fddd4159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.642+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 >> 172.21.15.158:0/2871583309 conn(0x7fddd40669f0 msgr2=0x7fddd40cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:01.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.643+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 shutdown_connections 2023-10-16T09:45:01.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.643+0000 7fddda235700 1 -- 172.21.15.158:0/2871583309 wait complete. 2023-10-16T09:45:01.656 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-10-16T09:45:01.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:01.657 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-10-16T09:45:01.675 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2023-10-16T09:45:01.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.837+0000 7f92eed34700 1 Processor -- start 2023-10-16T09:45:01.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.837+0000 7f92eed34700 1 -- start start 2023-10-16T09:45:01.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92eed34700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92e806afe0 0x7f92e80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92eed34700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f92e80cc590 0x7f92e80afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92eed34700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e80cbd30 con 0x7f92e815da40 2023-10-16T09:45:01.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92eed34700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e80cc1c0 con 0x7f92e806afe0 2023-10-16T09:45:01.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92eed34700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e815de20 con 0x7f92e80cc590 2023-10-16T09:45:01.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92e806afe0 0x7f92e80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92e806afe0 0x7f92e80caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49012/0 (socket says 172.21.15.158:49012) 2023-10-16T09:45:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 -- 172.21.15.158:0/3930751096 learned_addr learned my addr 172.21.15.158:0/3930751096 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92ed2d1700 1 --2- 172.21.15.158:0/3930751096 >> v2:172.21.15.172:3300/0 conn(0x7f92e80cc590 0x7f92e80afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 -- 172.21.15.158:0/3930751096 >> v2:172.21.15.172:3300/0 conn(0x7f92e80cc590 msgr2=0x7f92e80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 --2- 172.21.15.158:0/3930751096 >> v2:172.21.15.172:3300/0 conn(0x7f92e80cc590 0x7f92e80afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 -- 172.21.15.158:0/3930751096 >> v1:172.21.15.172:6789/0 conn(0x7f92e815da40 legacy=0x7f92e806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 -- 172.21.15.158:0/3930751096 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80ce260 con 0x7f92e806afe0 2023-10-16T09:45:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.838+0000 7f92e7fff700 1 --2- 172.21.15.158:0/3930751096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e806afe0 0x7f92e80caa20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f92e80cdd70 tx=0x7f92d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d1bed1e54a17da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92e77fe700 1 -- 172.21.15.158:0/3930751096 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d400b870 con 0x7f92e806afe0 2023-10-16T09:45:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92e77fe700 1 -- 172.21.15.158:0/3930751096 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d400b9d0 con 0x7f92e806afe0 2023-10-16T09:45:01.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92e77fe700 1 -- 172.21.15.158:0/3930751096 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d400bd60 con 0x7f92e806afe0 2023-10-16T09:45:01.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 -- 172.21.15.158:0/3930751096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e806afe0 msgr2=0x7f92e80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 --2- 172.21.15.158:0/3930751096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e806afe0 0x7f92e80caa20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f92e80cdd70 tx=0x7f92d4009bd0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 -- 172.21.15.158:0/3930751096 shutdown_connections 2023-10-16T09:45:01.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 --2- 172.21.15.158:0/3930751096 >> v2:172.21.15.172:3300/0 conn(0x7f92e80cc590 0x7f92e80afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 --2- 172.21.15.158:0/3930751096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e806afe0 0x7f92e80caa20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 -- 172.21.15.158:0/3930751096 >> 172.21.15.158:0/3930751096 conn(0x7f92e8099a10 msgr2=0x7f92e80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 -- 172.21.15.158:0/3930751096 shutdown_connections 2023-10-16T09:45:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.839+0000 7f92eed34700 1 -- 172.21.15.158:0/3930751096 wait complete. 2023-10-16T09:45:01.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.840+0000 7f92eed34700 1 Processor -- start 2023-10-16T09:45:01.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.840+0000 7f92eed34700 1 -- start start 2023-10-16T09:45:01.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92eed34700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92e806afe0 0x7f92e809de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92eed34700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e80cc590 0x7f92e809a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92eed34700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92e815da40 0x7f92e815a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92eed34700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e809a5d0 con 0x7f92e80cc590 2023-10-16T09:45:01.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92eed34700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e809a800 con 0x7f92e815da40 2023-10-16T09:45:01.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92eed34700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e809aae0 con 0x7f92e806afe0 2023-10-16T09:45:01.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92e7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e80cc590 0x7f92e809a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92e7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e80cc590 0x7f92e809a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49018/0 (socket says 172.21.15.158:49018) 2023-10-16T09:45:01.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.841+0000 7f92e7fff700 1 -- 172.21.15.158:0/3973594954 learned_addr learned my addr 172.21.15.158:0/3973594954 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:01.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92ecad0700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92e806afe0 0x7f92e809de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92ed2d1700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92e815da40 0x7f92e815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e7fff700 1 -- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92e806afe0 msgr2=0x7f92e809de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e7fff700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92e806afe0 0x7f92e809de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e7fff700 1 -- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92e815da40 msgr2=0x7f92e815a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:01.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e7fff700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92e815da40 0x7f92e815a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:01.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e7fff700 1 -- 172.21.15.158:0/3973594954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d4009550 con 0x7f92e80cc590 2023-10-16T09:45:01.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e7fff700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e80cc590 0x7f92e809a090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f92d4000c00 tx=0x7f92d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:01.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e57fa700 1 -- 172.21.15.158:0/3973594954 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4004480 con 0x7f92e80cc590 2023-10-16T09:45:01.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e57fa700 1 -- 172.21.15.158:0/3973594954 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d40045e0 con 0x7f92e80cc590 2023-10-16T09:45:01.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e809adc0 con 0x7f92e80cc590 2023-10-16T09:45:01.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92e57fa700 1 -- 172.21.15.158:0/3973594954 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d40038e0 con 0x7f92e80cc590 2023-10-16T09:45:01.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.842+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e809b380 con 0x7f92e80cc590 2023-10-16T09:45:01.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.843+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e80681b0 con 0x7f92e80cc590 2023-10-16T09:45:01.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.843+0000 7f92e57fa700 1 -- 172.21.15.158:0/3973594954 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f92d4021070 con 0x7f92e80cc590 2023-10-16T09:45:01.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.844+0000 7f92e57fa700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f92cc06b070 0x7f92cc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:01.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.844+0000 7f92e57fa700 1 -- 172.21.15.158:0/3973594954 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f92d408b050 con 0x7f92e80cc590 2023-10-16T09:45:01.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.844+0000 7f92ecad0700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f92cc06b070 0x7f92cc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:01.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.844+0000 7f92ecad0700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f92cc06b070 0x7f92cc06d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f92dc007fd0 tx=0x7f92dc009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:01.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:01.848+0000 7f92e57fa700 1 -- 172.21.15.158:0/3973594954 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f92d4055ee0 con 0x7f92e80cc590 2023-10-16T09:45:02.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.065+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f92e80cb400 con 0x7f92cc06b070 2023-10-16T09:45:02.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.122+0000 7f92e57fa700 1 -- 172.21.15.158:0/3973594954 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f92dc0014c0 con 0x7f92cc06b070 2023-10-16T09:45:02.124 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir3 2023-10-16T09:45:02.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.124+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f92cc06b070 msgr2=0x7f92cc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.124+0000 7f92eed34700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f92cc06b070 0x7f92cc06d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f92dc007fd0 tx=0x7f92dc009040 comp rx=0 tx=0).stop 2023-10-16T09:45:02.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.124+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e80cc590 msgr2=0x7f92e809a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.124+0000 7f92eed34700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e80cc590 0x7f92e809a090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f92d4000c00 tx=0x7f92d4004750 comp rx=0 tx=0).stop 2023-10-16T09:45:02.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 shutdown_connections 2023-10-16T09:45:02.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92e815da40 0x7f92e815a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92e806afe0 0x7f92e809de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f92cc06b070 0x7f92cc06d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 --2- 172.21.15.158:0/3973594954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e80cc590 0x7f92e809a090 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 >> 172.21.15.158:0/3973594954 conn(0x7f92e8099a10 msgr2=0x7f92e80b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:02.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 shutdown_connections 2023-10-16T09:45:02.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.125+0000 7f92eed34700 1 -- 172.21.15.158:0/3973594954 wait complete. 2023-10-16T09:45:02.138 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-10-16T09:45:02.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:02.139 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-10-16T09:45:02.159 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2023-10-16T09:45:02.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.321+0000 7fafe36b4700 1 Processor -- start 2023-10-16T09:45:02.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.321+0000 7fafe36b4700 1 -- start start 2023-10-16T09:45:02.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe36b4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fafdc160d40 0x7fafdc0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe36b4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 0x7fafdc0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe36b4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0ce110 con 0x7fafdc0c4ef0 2023-10-16T09:45:02.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe36b4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafdc0c53a0 con 0x7fafdc160d40 2023-10-16T09:45:02.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe36b4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0c61e0 con 0x7fafdc0cf420 2023-10-16T09:45:02.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe1c51700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fafdc0c4ef0 0x7fafdc06d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49202/0 (socket says 172.21.15.158:49202) 2023-10-16T09:45:02.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe1c51700 1 -- 172.21.15.158:0/334799254 learned_addr learned my addr 172.21.15.158:0/334799254 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:02.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404674893 0 0) 0x7fafdc0ce110 con 0x7fafdc0c4ef0 2023-10-16T09:45:02.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe1450700 1 --2- 172.21.15.158:0/334799254 >> v2:172.21.15.172:3300/0 conn(0x7fafdc160d40 0x7fafdc0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc80035b0 con 0x7fafdc0c4ef0 2023-10-16T09:45:02.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe0c4f700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 0x7fafdc0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 190439130 0 0) 0x7fafc80035b0 con 0x7fafdc0c4ef0 2023-10-16T09:45:02.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 msgr2=0x7fafdc0c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:02.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 0x7fafdc0c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 >> v2:172.21.15.172:3300/0 conn(0x7fafdc160d40 msgr2=0x7fafdc0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:02.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 --2- 172.21.15.158:0/334799254 >> v2:172.21.15.172:3300/0 conn(0x7fafdc160d40 0x7fafdc0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0ce3f0 con 0x7fafdc0c4ef0 2023-10-16T09:45:02.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fafd40019f0 con 0x7fafdc0c4ef0 2023-10-16T09:45:02.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafe0c4f700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 0x7fafdc0c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:02.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.322+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 >> v1:172.21.15.158:6789/0 conn(0x7fafdc0c4ef0 legacy=0x7fafdc06d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 0x7fafc8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc8004d90 0x7fafc8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0c53a0 con 0x7fafc8003a00 2023-10-16T09:45:02.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc800ba10 con 0x7fafc8004d90 2023-10-16T09:45:02.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc800bb80 con 0x7fafdc0cf420 2023-10-16T09:45:02.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1450700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe0c4f700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 0x7fafc8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1c51700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc8004d90 0x7fafc8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1450700 1 -- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 msgr2=0x7fafc8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:02.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1450700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc0cf420 0x7fafc8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1450700 1 -- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc8004d90 msgr2=0x7fafc8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:02.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1450700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc8004d90 0x7fafc8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1450700 1 -- 172.21.15.158:0/334799254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80035b0 con 0x7fafc8003a00 2023-10-16T09:45:02.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafe1450700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafc8003e20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009b60 tx=0x7fafc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e2d7ede013a665 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:02.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc00e6c0 con 0x7fafc8003a00 2023-10-16T09:45:02.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc00e820 con 0x7fafc8003a00 2023-10-16T09:45:02.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.323+0000 7fafd3fff700 1 -- 172.21.15.158:0/334799254 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc00eb90 con 0x7fafc8003a00 2023-10-16T09:45:02.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.324+0000 7fafe36b4700 1 -- 172.21.15.158:0/334799254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 msgr2=0x7fafc8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.324+0000 7fafe36b4700 1 --2- 172.21.15.158:0/334799254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafc8003e20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009b60 tx=0x7fafc8005880 comp rx=0 tx=0).stop 2023-10-16T09:45:02.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.324+0000 7fafe1c51700 1 -- 172.21.15.158:0/334799254 reap_dead start 2023-10-16T09:45:02.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.324+0000 7fafe36b4700 1 -- 172.21.15.158:0/334799254 shutdown_connections 2023-10-16T09:45:02.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.324+0000 7fafe36b4700 1 -- 172.21.15.158:0/334799254 >> 172.21.15.158:0/334799254 conn(0x7fafdc0cc5b0 msgr2=0x7fafdc0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:02.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.324+0000 7fafe36b4700 1 -- 172.21.15.158:0/334799254 shutdown_connections 2023-10-16T09:45:02.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.324+0000 7fafe36b4700 1 -- 172.21.15.158:0/334799254 wait complete. 2023-10-16T09:45:02.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.325+0000 7fafe36b4700 1 Processor -- start 2023-10-16T09:45:02.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.325+0000 7fafe36b4700 1 -- start start 2023-10-16T09:45:02.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe36b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafdc1e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe36b4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafdc0c92f0 0x7fafdc0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe36b4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc1549b0 0x7fafdc154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe36b4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc1f3620 con 0x7fafc8003a00 2023-10-16T09:45:02.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe36b4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0c9f40 con 0x7fafdc0c92f0 2023-10-16T09:45:02.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe36b4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0c9cb0 con 0x7fafdc1549b0 2023-10-16T09:45:02.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafdc1e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafdc1e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49036/0 (socket says 172.21.15.158:49036) 2023-10-16T09:45:02.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 -- 172.21.15.158:0/1311795241 learned_addr learned my addr 172.21.15.158:0/1311795241 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:02.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe0c4f700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafdc0c92f0 0x7fafdc0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1c51700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc1549b0 0x7fafdc154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 -- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc1549b0 msgr2=0x7fafdc154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc1549b0 0x7fafdc154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 -- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafdc0c92f0 msgr2=0x7fafdc0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafdc0c92f0 0x7fafdc0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 -- 172.21.15.158:0/1311795241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc155370 con 0x7fafc8003a00 2023-10-16T09:45:02.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe1450700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafdc1e78d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fafc8004780 tx=0x7fafcc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:02.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafd1ffb700 1 -- 172.21.15.158:0/1311795241 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc00e6c0 con 0x7fafc8003a00 2023-10-16T09:45:02.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafd1ffb700 1 -- 172.21.15.158:0/1311795241 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc0038c0 con 0x7fafc8003a00 2023-10-16T09:45:02.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc1f2860 con 0x7fafc8003a00 2023-10-16T09:45:02.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.326+0000 7fafd1ffb700 1 -- 172.21.15.158:0/1311795241 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc003c70 con 0x7fafc8003a00 2023-10-16T09:45:02.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.327+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafdc1f2e20 con 0x7fafc8003a00 2023-10-16T09:45:02.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.328+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafdc066db0 con 0x7fafc8003a00 2023-10-16T09:45:02.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.328+0000 7fafd1ffb700 1 -- 172.21.15.158:0/1311795241 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fafcc013470 con 0x7fafc8003a00 2023-10-16T09:45:02.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.329+0000 7fafd1ffb700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fafc006b070 0x7fafc006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.329+0000 7fafe0c4f700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fafc006b070 0x7fafc006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.329+0000 7fafd1ffb700 1 -- 172.21.15.158:0/1311795241 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fafcc08b370 con 0x7fafc8003a00 2023-10-16T09:45:02.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.329+0000 7fafe0c4f700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fafc006b070 0x7fafc006d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafdc0adc00 tx=0x7fafd8008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:02.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.333+0000 7fafd1ffb700 1 -- 172.21.15.158:0/1311795241 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fafcc0561e0 con 0x7fafc8003a00 2023-10-16T09:45:02.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.553+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fafdc0ce9b0 con 0x7fafc006b070 2023-10-16T09:45:02.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.620+0000 7fafd1ffb700 1 -- 172.21.15.158:0/1311795241 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fafd8009560 con 0x7fafc006b070 2023-10-16T09:45:02.622 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir4 2023-10-16T09:45:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fafc006b070 msgr2=0x7fafc006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fafc006b070 0x7fafc006d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafdc0adc00 tx=0x7fafd8008040 comp rx=0 tx=0).stop 2023-10-16T09:45:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 msgr2=0x7fafdc1e78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafdc1e78d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fafc8004780 tx=0x7fafcc0047c0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 shutdown_connections 2023-10-16T09:45:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafdc1549b0 0x7fafdc154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fafc006b070 0x7fafc006d530 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc8003a00 0x7fafdc1e78d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 --2- 172.21.15.158:0/1311795241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafdc0c92f0 0x7fafdc0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 >> 172.21.15.158:0/1311795241 conn(0x7fafdc0cc5b0 msgr2=0x7fafdc06b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 shutdown_connections 2023-10-16T09:45:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.622+0000 7fafe36b4700 1 -- 172.21.15.158:0/1311795241 wait complete. 2023-10-16T09:45:02.636 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-10-16T09:45:02.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:02.637 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-10-16T09:45:02.655 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2023-10-16T09:45:02.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.817+0000 7efe9d1b7700 1 Processor -- start 2023-10-16T09:45:02.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.817+0000 7efe9d1b7700 1 -- start start 2023-10-16T09:45:02.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe9d1b7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe98159280 0x7efe9806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe9d1b7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efe980cbef0 0x7efe980cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe9d1b7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe98159660 con 0x7efe980d0e10 2023-10-16T09:45:02.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe9d1b7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe980cb710 con 0x7efe980cbef0 2023-10-16T09:45:02.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe9d1b7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9806b120 con 0x7efe98159280 2023-10-16T09:45:02.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe96d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe98159280 0x7efe9806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe9759e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efe980cbef0 0x7efe980cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe96d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe98159280 0x7efe9806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:59878/0 (socket says 172.21.15.158:59878) 2023-10-16T09:45:02.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe96d9d700 1 -- 172.21.15.158:0/1106601460 learned_addr learned my addr 172.21.15.158:0/1106601460 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:02.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe9659c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7efe980d0e10 0x7efe9815c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:53126/0 (socket says 172.21.15.158:53126) 2023-10-16T09:45:02.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe96d9d700 1 -- 172.21.15.158:0/1106601460 >> v2:172.21.15.172:3300/0 conn(0x7efe980cbef0 msgr2=0x7efe980cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:02.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.818+0000 7efe96d9d700 1 --2- 172.21.15.158:0/1106601460 >> v2:172.21.15.172:3300/0 conn(0x7efe980cbef0 0x7efe980cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe96d9d700 1 -- 172.21.15.158:0/1106601460 >> v1:172.21.15.172:6789/0 conn(0x7efe980d0e10 legacy=0x7efe9815c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe96d9d700 1 -- 172.21.15.158:0/1106601460 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9806b400 con 0x7efe98159280 2023-10-16T09:45:02.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9759e700 1 --2- 172.21.15.158:0/1106601460 >> v2:172.21.15.172:3300/0 conn(0x7efe980cbef0 0x7efe980cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:45:02.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe96d9d700 1 --2- 172.21.15.158:0/1106601460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe98159280 0x7efe9806a8d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe900098f0 tx=0x7efe90009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=479c92995247fb47 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:02.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe95d9b700 1 -- 172.21.15.158:0/1106601460 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90011830 con 0x7efe98159280 2023-10-16T09:45:02.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe95d9b700 1 -- 172.21.15.158:0/1106601460 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe90011990 con 0x7efe98159280 2023-10-16T09:45:02.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe95d9b700 1 -- 172.21.15.158:0/1106601460 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90011d20 con 0x7efe98159280 2023-10-16T09:45:02.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9d1b7700 1 -- 172.21.15.158:0/1106601460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe98159280 msgr2=0x7efe9806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/1106601460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe98159280 0x7efe9806a8d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe900098f0 tx=0x7efe90009c00 comp rx=0 tx=0).stop 2023-10-16T09:45:02.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9d1b7700 1 -- 172.21.15.158:0/1106601460 shutdown_connections 2023-10-16T09:45:02.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/1106601460 >> v2:172.21.15.172:3300/0 conn(0x7efe980cbef0 0x7efe980cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/1106601460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe98159280 0x7efe9806a8d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9d1b7700 1 -- 172.21.15.158:0/1106601460 >> 172.21.15.158:0/1106601460 conn(0x7efe98099a10 msgr2=0x7efe980d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:02.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.819+0000 7efe9d1b7700 1 -- 172.21.15.158:0/1106601460 shutdown_connections 2023-10-16T09:45:02.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.820+0000 7efe9d1b7700 1 -- 172.21.15.158:0/1106601460 wait complete. 2023-10-16T09:45:02.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.821+0000 7efe9d1b7700 1 Processor -- start 2023-10-16T09:45:02.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.821+0000 7efe9d1b7700 1 -- start start 2023-10-16T09:45:02.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9d1b7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe980cbef0 0x7efe981f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9d1b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe980d0e10 0x7efe98152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9d1b7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe98159280 0x7efe980ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9d1b7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe981f7480 con 0x7efe980d0e10 2023-10-16T09:45:02.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9d1b7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe980ce040 con 0x7efe98159280 2023-10-16T09:45:02.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9d1b7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe98152750 con 0x7efe980cbef0 2023-10-16T09:45:02.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe980d0e10 0x7efe98152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe980d0e10 0x7efe98152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49052/0 (socket says 172.21.15.158:49052) 2023-10-16T09:45:02.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9659c700 1 -- 172.21.15.158:0/2509852096 learned_addr learned my addr 172.21.15.158:0/2509852096 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:02.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe96d9d700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe980cbef0 0x7efe981f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.822+0000 7efe9759e700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe98159280 0x7efe980ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9659c700 1 -- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe980cbef0 msgr2=0x7efe981f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9659c700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe980cbef0 0x7efe981f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9659c700 1 -- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe98159280 msgr2=0x7efe980ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:02.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9659c700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe98159280 0x7efe980ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:02.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9659c700 1 -- 172.21.15.158:0/2509852096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe88009a70 con 0x7efe980d0e10 2023-10-16T09:45:02.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9659c700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe980d0e10 0x7efe98152210 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7efe8000c500 tx=0x7efe8000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:02.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe8f7fe700 1 -- 172.21.15.158:0/2509852096 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8000cea0 con 0x7efe980d0e10 2023-10-16T09:45:02.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe8f7fe700 1 -- 172.21.15.158:0/2509852096 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe80005cb0 con 0x7efe980d0e10 2023-10-16T09:45:02.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe8f7fe700 1 -- 172.21.15.158:0/2509852096 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe800072d0 con 0x7efe980d0e10 2023-10-16T09:45:02.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe90009550 con 0x7efe980d0e10 2023-10-16T09:45:02.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.823+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe980c4280 con 0x7efe980d0e10 2023-10-16T09:45:02.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.824+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe980681b0 con 0x7efe980d0e10 2023-10-16T09:45:02.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.824+0000 7efe8f7fe700 1 -- 172.21.15.158:0/2509852096 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7efe80007430 con 0x7efe980d0e10 2023-10-16T09:45:02.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.825+0000 7efe8f7fe700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efe7806b070 0x7efe7806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:02.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.825+0000 7efe8f7fe700 1 -- 172.21.15.158:0/2509852096 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7efe80088b90 con 0x7efe980d0e10 2023-10-16T09:45:02.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.825+0000 7efe96d9d700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efe7806b070 0x7efe7806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:02.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.825+0000 7efe96d9d700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efe7806b070 0x7efe7806d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe90000c00 tx=0x7efe90008030 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:02.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:02.829+0000 7efe8f7fe700 1 -- 172.21.15.158:0/2509852096 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe800539a0 con 0x7efe980d0e10 2023-10-16T09:45:03.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.046+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efe9815bd90 con 0x7efe7806b070 2023-10-16T09:45:03.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.102+0000 7efe8f7fe700 1 -- 172.21.15.158:0/2509852096 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7efe90011b80 con 0x7efe7806b070 2023-10-16T09:45:03.104 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir5 2023-10-16T09:45:03.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efe7806b070 msgr2=0x7efe7806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efe7806b070 0x7efe7806d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe90000c00 tx=0x7efe90008030 comp rx=0 tx=0).stop 2023-10-16T09:45:03.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe980d0e10 msgr2=0x7efe98152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe980d0e10 0x7efe98152210 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7efe8000c500 tx=0x7efe8000c810 comp rx=0 tx=0).stop 2023-10-16T09:45:03.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 shutdown_connections 2023-10-16T09:45:03.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe98159280 0x7efe980ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe980cbef0 0x7efe981f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efe7806b070 0x7efe7806d530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 --2- 172.21.15.158:0/2509852096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe980d0e10 0x7efe98152210 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 >> 172.21.15.158:0/2509852096 conn(0x7efe98099a10 msgr2=0x7efe9815a880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:03.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 shutdown_connections 2023-10-16T09:45:03.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.104+0000 7efe9d1b7700 1 -- 172.21.15.158:0/2509852096 wait complete. 2023-10-16T09:45:03.118 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-10-16T09:45:03.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:03.119 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-10-16T09:45:03.137 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2023-10-16T09:45:03.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.300+0000 7f0eb33c8700 1 Processor -- start 2023-10-16T09:45:03.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.300+0000 7f0eb33c8700 1 -- start start 2023-10-16T09:45:03.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb33c8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 0x7f0eac0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb33c8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac0caa80 con 0x7f0eac15d770 2023-10-16T09:45:03.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb33c8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac0cad60 con 0x7f0eac0cf680 2023-10-16T09:45:03.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb33c8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0cb040 con 0x7f0eac15db50 2023-10-16T09:45:03.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb1164700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0eac15d770 0x7f0eac0cd7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49234/0 (socket says 172.21.15.158:49234) 2023-10-16T09:45:03.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb1164700 1 -- 172.21.15.158:0/882882190 learned_addr learned my addr 172.21.15.158:0/882882190 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:03.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542734267 0 0) 0x7f0eac0caa80 con 0x7f0eac15d770 2023-10-16T09:45:03.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb0963700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 0x7f0eac0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e980035b0 con 0x7f0eac15d770 2023-10-16T09:45:03.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1341724445 0 0) 0x7f0e980035b0 con 0x7f0eac15d770 2023-10-16T09:45:03.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 msgr2=0x7f0eac0cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:03.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 0x7f0eac0cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 >> v1:172.21.15.172:6789/0 conn(0x7f0eac0cf680 legacy=0x7f0eac0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0cb320 con 0x7f0eac15d770 2023-10-16T09:45:03.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0e9c001670 con 0x7f0eac15d770 2023-10-16T09:45:03.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 >> v1:172.21.15.158:6789/0 conn(0x7f0eac15d770 legacy=0x7f0eac0cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 0x7f0e98003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98003fb0 0x7f0e98004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e98004d90 0x7f0e98005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0cb040 con 0x7f0e98004d90 2023-10-16T09:45:03.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9800bb80 con 0x7f0e98003fb0 2023-10-16T09:45:03.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9800bd40 con 0x7f0eac15db50 2023-10-16T09:45:03.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb1965700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e98004d90 0x7f0e98005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.301+0000 7f0eb0963700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98003fb0 0x7f0e98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb1164700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 0x7f0e98003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb1965700 1 -- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 msgr2=0x7f0e98003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:03.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb1965700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac15db50 0x7f0e98003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb1965700 1 -- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98003fb0 msgr2=0x7f0e98004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:03.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb1965700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98003fb0 0x7f0e98004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb1965700 1 -- 172.21.15.158:0/882882190 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980035b0 con 0x7f0e98004d90 2023-10-16T09:45:03.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb1965700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e98004d90 0x7f0e98005210 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400c190 tx=0x7f0e98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8282244ed234fb9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:03.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4001db0 con 0x7f0e98004d90 2023-10-16T09:45:03.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea400ccd0 con 0x7f0e98004d90 2023-10-16T09:45:03.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0ea3fff700 1 -- 172.21.15.158:0/882882190 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4003c40 con 0x7f0e98004d90 2023-10-16T09:45:03.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.302+0000 7f0eb33c8700 1 -- 172.21.15.158:0/882882190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e98004d90 msgr2=0x7f0e98005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.303+0000 7f0eb33c8700 1 --2- 172.21.15.158:0/882882190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e98004d90 0x7f0e98005210 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400c190 tx=0x7f0e98005880 comp rx=0 tx=0).stop 2023-10-16T09:45:03.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.303+0000 7f0eb1965700 1 -- 172.21.15.158:0/882882190 reap_dead start 2023-10-16T09:45:03.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.303+0000 7f0eb33c8700 1 -- 172.21.15.158:0/882882190 shutdown_connections 2023-10-16T09:45:03.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.303+0000 7f0eb33c8700 1 -- 172.21.15.158:0/882882190 >> 172.21.15.158:0/882882190 conn(0x7f0eac099a10 msgr2=0x7f0eac0d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:03.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.303+0000 7f0eb33c8700 1 -- 172.21.15.158:0/882882190 shutdown_connections 2023-10-16T09:45:03.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.303+0000 7f0eb33c8700 1 -- 172.21.15.158:0/882882190 wait complete. 2023-10-16T09:45:03.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.304+0000 7f0eb33c8700 1 Processor -- start 2023-10-16T09:45:03.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.304+0000 7f0eb33c8700 1 -- start start 2023-10-16T09:45:03.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb33c8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98004d90 0x7f0eac150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb33c8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0eac1f30c0 0x7f0eac0c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb33c8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac0c7a40 0x7f0eac0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb33c8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0c8560 con 0x7f0eac1f30c0 2023-10-16T09:45:03.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb33c8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0c9080 con 0x7f0e98004d90 2023-10-16T09:45:03.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb33c8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1510c0 con 0x7f0eac0c7a40 2023-10-16T09:45:03.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0eac1f30c0 0x7f0eac0c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0eac1f30c0 0x7f0eac0c7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49072/0 (socket says 172.21.15.158:49072) 2023-10-16T09:45:03.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 -- 172.21.15.158:0/266279170 learned_addr learned my addr 172.21.15.158:0/266279170 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:03.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb1164700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98004d90 0x7f0eac150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb1965700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac0c7a40 0x7f0eac0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 -- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac0c7a40 msgr2=0x7f0eac0c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac0c7a40 0x7f0eac0c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 -- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98004d90 msgr2=0x7f0eac150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98004d90 0x7f0eac150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 -- 172.21.15.158:0/266279170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0c7fa0 con 0x7f0eac1f30c0 2023-10-16T09:45:03.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb1164700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98004d90 0x7f0eac150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:45:03.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb0963700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0eac1f30c0 0x7f0eac0c7500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8011aa0 tx=0x7f0ea8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:03.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/266279170 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800f980 con 0x7f0eac1f30c0 2023-10-16T09:45:03.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/266279170 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea800fae0 con 0x7f0eac1f30c0 2023-10-16T09:45:03.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/266279170 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800fe90 con 0x7f0eac1f30c0 2023-10-16T09:45:03.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.305+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac0c8280 con 0x7f0eac1f30c0 2023-10-16T09:45:03.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.306+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac154210 con 0x7f0eac1f30c0 2023-10-16T09:45:03.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.306+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac0c8b70 con 0x7f0eac1f30c0 2023-10-16T09:45:03.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.307+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/266279170 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f0ea8014440 con 0x7f0eac1f30c0 2023-10-16T09:45:03.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.307+0000 7f0ea1ffb700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f0e9006b070 0x7f0e9006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.307+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/266279170 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f0ea808a1f0 con 0x7f0eac1f30c0 2023-10-16T09:45:03.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.307+0000 7f0eb1164700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f0e9006b070 0x7f0e9006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.308+0000 7f0eb1164700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f0e9006b070 0x7f0e9006d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0017a0 tx=0x7f0e9c009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:03.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.312+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/266279170 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0eac0c8b70 con 0x7f0eac1f30c0 2023-10-16T09:45:03.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.531+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0eac0cb8e0 con 0x7f0e9006b070 2023-10-16T09:45:03.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.599+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/266279170 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0e9c001620 con 0x7f0e9006b070 2023-10-16T09:45:03.601 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir6 2023-10-16T09:45:03.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f0e9006b070 msgr2=0x7f0e9006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f0e9006b070 0x7f0e9006d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0017a0 tx=0x7f0e9c009040 comp rx=0 tx=0).stop 2023-10-16T09:45:03.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0eac1f30c0 msgr2=0x7f0eac0c7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0eac1f30c0 0x7f0eac0c7500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8011aa0 tx=0x7f0ea8011e60 comp rx=0 tx=0).stop 2023-10-16T09:45:03.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 shutdown_connections 2023-10-16T09:45:03.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0eac0c7a40 0x7f0eac0c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e98004d90 0x7f0eac150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f0e9006b070 0x7f0e9006d530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 --2- 172.21.15.158:0/266279170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0eac1f30c0 0x7f0eac0c7500 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 >> 172.21.15.158:0/266279170 conn(0x7f0eac099a10 msgr2=0x7f0eac0cd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.601+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 shutdown_connections 2023-10-16T09:45:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.602+0000 7f0eb33c8700 1 -- 172.21.15.158:0/266279170 wait complete. 2023-10-16T09:45:03.615 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-10-16T09:45:03.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:03.616 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-10-16T09:45:03.634 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2023-10-16T09:45:03.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.797+0000 7f6ac4735700 1 Processor -- start 2023-10-16T09:45:03.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.797+0000 7f6ac4735700 1 -- start start 2023-10-16T09:45:03.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.798+0000 7f6ac4735700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6abc163380 0x7f6abc165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.798+0000 7f6ac4735700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0d0ae0 con 0x7f6abc15deb0 2023-10-16T09:45:03.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.798+0000 7f6ac4735700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0d12d0 con 0x7f6abc15dad0 2023-10-16T09:45:03.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.798+0000 7f6ac4735700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc0cbdf0 con 0x7f6abc163380 2023-10-16T09:45:03.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.798+0000 7f6ac1cd0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6abc15deb0 0x7f6abc0ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49254/0 (socket says 172.21.15.158:49254) 2023-10-16T09:45:03.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.798+0000 7f6ac1cd0700 1 -- 172.21.15.158:0/1308584636 learned_addr learned my addr 172.21.15.158:0/1308584636 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:03.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399751508 0 0) 0x7f6abc0d0ae0 con 0x7f6abc15deb0 2023-10-16T09:45:03.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa80035b0 con 0x7f6abc15deb0 2023-10-16T09:45:03.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 --2- 172.21.15.158:0/1308584636 >> v2:172.21.15.172:3300/0 conn(0x7f6abc163380 0x7f6abc165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4229919123 0 0) 0x7f6aa80035b0 con 0x7f6abc15deb0 2023-10-16T09:45:03.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 >> v2:172.21.15.172:3300/0 conn(0x7f6abc163380 msgr2=0x7f6abc165770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:03.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 --2- 172.21.15.158:0/1308584636 >> v2:172.21.15.172:3300/0 conn(0x7f6abc163380 0x7f6abc165770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 >> v1:172.21.15.172:6789/0 conn(0x7f6abc15dad0 legacy=0x7f6abc0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0cc280 con 0x7f6abc15deb0 2023-10-16T09:45:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f6ab8001670 con 0x7f6abc15deb0 2023-10-16T09:45:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 >> v1:172.21.15.158:6789/0 conn(0x7f6abc15deb0 legacy=0x7f6abc0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aa8003a00 0x7f6aa8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aa8004850 0x7f6aa8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6aa800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0cbdf0 con 0x7f6aa8005330 2023-10-16T09:45:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa800c200 con 0x7f6aa8004850 2023-10-16T09:45:03.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa800c3c0 con 0x7f6aa8003a00 2023-10-16T09:45:03.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6aa800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac24d1700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aa8003a00 0x7f6aa8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac1cd0700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aa8004850 0x7f6aa8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 -- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aa8003a00 msgr2=0x7f6aa8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:03.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6aa8003a00 0x7f6aa8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 -- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aa8004850 msgr2=0x7f6aa8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:03.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6aa8004850 0x7f6aa8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 -- 172.21.15.158:0/1308584636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa80035b0 con 0x7f6aa8005330 2023-10-16T09:45:03.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.799+0000 7f6ac2cd2700 1 -- 172.21.15.158:0/1308584636 reap_dead start 2023-10-16T09:45:03.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac2cd2700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6aa800ba10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009f90 tx=0x7f6aa8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b04c6a3012073e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:03.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4003a20 con 0x7f6aa8005330 2023-10-16T09:45:03.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab400f040 con 0x7f6aa8005330 2023-10-16T09:45:03.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac14cf700 1 -- 172.21.15.158:0/1308584636 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4003dc0 con 0x7f6aa8005330 2023-10-16T09:45:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac4735700 1 -- 172.21.15.158:0/1308584636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 msgr2=0x7f6aa800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac4735700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6aa800ba10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009f90 tx=0x7f6aa8005880 comp rx=0 tx=0).stop 2023-10-16T09:45:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac4735700 1 -- 172.21.15.158:0/1308584636 shutdown_connections 2023-10-16T09:45:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac4735700 1 --2- 172.21.15.158:0/1308584636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6aa800ba10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac4735700 1 -- 172.21.15.158:0/1308584636 >> 172.21.15.158:0/1308584636 conn(0x7f6abc099a10 msgr2=0x7f6abc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:03.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac4735700 1 -- 172.21.15.158:0/1308584636 shutdown_connections 2023-10-16T09:45:03.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.800+0000 7f6ac4735700 1 -- 172.21.15.158:0/1308584636 wait complete. 2023-10-16T09:45:03.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 Processor -- start 2023-10-16T09:45:03.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 -- start start 2023-10-16T09:45:03.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6abc0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc154c80 0x7f6abc150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc0c5460 0x7f6abc0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0c5e20 con 0x7f6aa8005330 2023-10-16T09:45:03.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc1514d0 con 0x7f6abc0c5460 2023-10-16T09:45:03.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac4735700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc155460 con 0x7f6abc154c80 2023-10-16T09:45:03.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac24d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6abc0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac24d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6abc0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49088/0 (socket says 172.21.15.158:49088) 2023-10-16T09:45:03.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.802+0000 7f6ac24d1700 1 -- 172.21.15.158:0/4135571969 learned_addr learned my addr 172.21.15.158:0/4135571969 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:03.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac1cd0700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc154c80 0x7f6abc150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac2cd2700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc0c5460 0x7f6abc0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac24d1700 1 -- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc154c80 msgr2=0x7f6abc150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac24d1700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc154c80 0x7f6abc150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac24d1700 1 -- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc0c5460 msgr2=0x7f6abc0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:03.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac24d1700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc0c5460 0x7f6abc0c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:03.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac24d1700 1 -- 172.21.15.158:0/4135571969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc155740 con 0x7f6aa8005330 2023-10-16T09:45:03.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac24d1700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6abc0cd830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6aac011a30 tx=0x7f6aac011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:03.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ab2ffd700 1 -- 172.21.15.158:0/4135571969 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0054e0 con 0x7f6aa8005330 2023-10-16T09:45:03.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ab2ffd700 1 -- 172.21.15.158:0/4135571969 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac00f850 con 0x7f6aa8005330 2023-10-16T09:45:03.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc155a20 con 0x7f6aa8005330 2023-10-16T09:45:03.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ab2ffd700 1 -- 172.21.15.158:0/4135571969 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac00fbe0 con 0x7f6aa8005330 2023-10-16T09:45:03.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.803+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc155100 con 0x7f6aa8005330 2023-10-16T09:45:03.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.804+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc0681b0 con 0x7f6aa8005330 2023-10-16T09:45:03.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.804+0000 7f6ab2ffd700 1 -- 172.21.15.158:0/4135571969 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6aac018020 con 0x7f6aa8005330 2023-10-16T09:45:03.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.805+0000 7f6ab2ffd700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f6aa006b070 0x7f6aa006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:03.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.805+0000 7f6ab2ffd700 1 -- 172.21.15.158:0/4135571969 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f6aac08b720 con 0x7f6aa8005330 2023-10-16T09:45:03.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.805+0000 7f6ac1cd0700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f6aa006b070 0x7f6aa006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:03.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.805+0000 7f6ac1cd0700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f6aa006b070 0x7f6aa006d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0b3150 tx=0x7f6ab8009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:03.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:03.809+0000 7f6ab2ffd700 1 -- 172.21.15.158:0/4135571969 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6aac056480 con 0x7f6aa8005330 2023-10-16T09:45:04.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.025+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6abc0d0ff0 con 0x7f6aa006b070 2023-10-16T09:45:04.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.083+0000 7f6ab2ffd700 1 -- 172.21.15.158:0/4135571969 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f6ab8004d90 con 0x7f6aa006b070 2023-10-16T09:45:04.085 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir7 2023-10-16T09:45:04.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f6aa006b070 msgr2=0x7f6aa006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f6aa006b070 0x7f6aa006d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0b3150 tx=0x7f6ab8009040 comp rx=0 tx=0).stop 2023-10-16T09:45:04.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 msgr2=0x7f6abc0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6abc0cd830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6aac011a30 tx=0x7f6aac011df0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 shutdown_connections 2023-10-16T09:45:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc0c5460 0x7f6abc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f6aa006b070 0x7f6aa006d530 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6aa8005330 0x7f6abc0cd830 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 --2- 172.21.15.158:0/4135571969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc154c80 0x7f6abc150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 >> 172.21.15.158:0/4135571969 conn(0x7f6abc099a10 msgr2=0x7f6abc15ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:04.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 shutdown_connections 2023-10-16T09:45:04.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.085+0000 7f6ac4735700 1 -- 172.21.15.158:0/4135571969 wait complete. 2023-10-16T09:45:04.099 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-10-16T09:45:04.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:04.100 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-10-16T09:45:04.119 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2023-10-16T09:45:04.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.282+0000 7fa218b81700 1 Processor -- start 2023-10-16T09:45:04.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.282+0000 7fa218b81700 1 -- start start 2023-10-16T09:45:04.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa218b81700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa214072070 0x7fa2140cef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa218b81700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 0x7fa2140cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa218b81700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2140ce150 con 0x7fa21415a5b0 2023-10-16T09:45:04.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa218b81700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2140cdf70 con 0x7fa214072070 2023-10-16T09:45:04.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa218b81700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140ce430 con 0x7fa2140cb900 2023-10-16T09:45:04.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa21415a5b0 0x7fa214071960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49272/0 (socket says 172.21.15.158:49272) 2023-10-16T09:45:04.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21259c700 1 -- 172.21.15.158:0/1200782886 learned_addr learned my addr 172.21.15.158:0/1200782886 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:04.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811399574 0 0) 0x7fa2140ce150 con 0x7fa21415a5b0 2023-10-16T09:45:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc0035b0 con 0x7fa21415a5b0 2023-10-16T09:45:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa212d9d700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 0x7fa2140cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa211d9b700 1 --2- 172.21.15.158:0/1200782886 >> v2:172.21.15.172:3300/0 conn(0x7fa214072070 0x7fa2140cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 778087277 0 0) 0x7fa1fc0035b0 con 0x7fa21415a5b0 2023-10-16T09:45:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 msgr2=0x7fa2140cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 0x7fa2140cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 >> v2:172.21.15.172:3300/0 conn(0x7fa214072070 msgr2=0x7fa2140cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 --2- 172.21.15.158:0/1200782886 >> v2:172.21.15.172:3300/0 conn(0x7fa214072070 0x7fa2140cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140ce710 con 0x7fa21415a5b0 2023-10-16T09:45:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fa204001670 con 0x7fa21415a5b0 2023-10-16T09:45:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 >> v1:172.21.15.158:6789/0 conn(0x7fa21415a5b0 legacy=0x7fa214071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc003a00 0x7fa1fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 0x7fa1fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc004d90 0x7fa1fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140cdf70 con 0x7fa1fc003a00 2023-10-16T09:45:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140ce430 con 0x7fa1fc004d90 2023-10-16T09:45:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.283+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00ba10 con 0x7fa2140cb900 2023-10-16T09:45:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21259c700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc003a00 0x7fa1fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa211d9b700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 0x7fa1fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa212d9d700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc004d90 0x7fa1fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21259c700 1 -- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 msgr2=0x7fa1fc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21259c700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 0x7fa1fc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21259c700 1 -- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc004d90 msgr2=0x7fa1fc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21259c700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc004d90 0x7fa1fc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21259c700 1 -- 172.21.15.158:0/1200782886 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc0035b0 con 0x7fa1fc003a00 2023-10-16T09:45:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa211d9b700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2140cb900 0x7fa1fc004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21259c700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc003a00 0x7fa1fc003e20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa20400c320 tx=0x7fa1fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8ca5703750d672 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20400ce60 con 0x7fa1fc003a00 2023-10-16T09:45:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa204005960 con 0x7fa1fc003a00 2023-10-16T09:45:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.284+0000 7fa21159a700 1 -- 172.21.15.158:0/1200782886 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa204005cf0 con 0x7fa1fc003a00 2023-10-16T09:45:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.285+0000 7fa218b81700 1 -- 172.21.15.158:0/1200782886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc003a00 msgr2=0x7fa1fc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.285+0000 7fa218b81700 1 --2- 172.21.15.158:0/1200782886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc003a00 0x7fa1fc003e20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa20400c320 tx=0x7fa1fc005880 comp rx=0 tx=0).stop 2023-10-16T09:45:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.285+0000 7fa212d9d700 1 -- 172.21.15.158:0/1200782886 reap_dead start 2023-10-16T09:45:04.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.285+0000 7fa218b81700 1 -- 172.21.15.158:0/1200782886 shutdown_connections 2023-10-16T09:45:04.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.285+0000 7fa218b81700 1 -- 172.21.15.158:0/1200782886 >> 172.21.15.158:0/1200782886 conn(0x7fa2140b1e90 msgr2=0x7fa2140b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:04.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.285+0000 7fa218b81700 1 -- 172.21.15.158:0/1200782886 shutdown_connections 2023-10-16T09:45:04.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.285+0000 7fa218b81700 1 -- 172.21.15.158:0/1200782886 wait complete. 2023-10-16T09:45:04.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.286+0000 7fa218b81700 1 Processor -- start 2023-10-16T09:45:04.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.287+0000 7fa218b81700 1 -- start start 2023-10-16T09:45:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.287+0000 7fa218b81700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc003a00 0x7fa214158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.287+0000 7fa218b81700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa214158790 0x7fa2140b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.287+0000 7fa218b81700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2140b3df0 0x7fa2141ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.287+0000 7fa218b81700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140c6ca0 con 0x7fa2140b3df0 2023-10-16T09:45:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.287+0000 7fa218b81700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140b3960 con 0x7fa1fc003a00 2023-10-16T09:45:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.287+0000 7fa218b81700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140b2d50 con 0x7fa214158790 2023-10-16T09:45:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2140b3df0 0x7fa2141ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2140b3df0 0x7fa2141ef4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49112/0 (socket says 172.21.15.158:49112) 2023-10-16T09:45:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 -- 172.21.15.158:0/660845190 learned_addr learned my addr 172.21.15.158:0/660845190 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa21259c700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc003a00 0x7fa214158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa211d9b700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa214158790 0x7fa2140b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 -- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa214158790 msgr2=0x7fa2140b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa214158790 0x7fa2140b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 -- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc003a00 msgr2=0x7fa214158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc003a00 0x7fa214158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 -- 172.21.15.158:0/660845190 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140b3030 con 0x7fa2140b3df0 2023-10-16T09:45:04.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa21259c700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc003a00 0x7fa214158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:45:04.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa211d9b700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa214158790 0x7fa2140b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:45:04.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa212d9d700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2140b3df0 0x7fa2141ef4a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa2140cbd30 tx=0x7fa20c0114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:04.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c00e070 con 0x7fa2140b3df0 2023-10-16T09:45:04.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c003c30 con 0x7fa2140b3df0 2023-10-16T09:45:04.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c003de0 con 0x7fa2140b3df0 2023-10-16T09:45:04.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2140b3310 con 0x7fa2140b3df0 2023-10-16T09:45:04.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.288+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2140b5070 con 0x7fa2140b3df0 2023-10-16T09:45:04.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.289+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2140b3af0 con 0x7fa2140b3df0 2023-10-16T09:45:04.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.290+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa20c011a10 con 0x7fa2140b3df0 2023-10-16T09:45:04.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.290+0000 7fa202ffd700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa1f406b070 0x7fa1f406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.290+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fa20c08a050 con 0x7fa2140b3df0 2023-10-16T09:45:04.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.290+0000 7fa21259c700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa1f406b070 0x7fa1f406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.291+0000 7fa21259c700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa1f406b070 0x7fa1f406d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa2141f2900 tx=0x7fa20400a000 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:04.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.295+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa20c054d30 con 0x7fa2140b3df0 2023-10-16T09:45:04.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.381+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fa20c088330 con 0x7fa2140b3df0 2023-10-16T09:45:04.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.520+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa21415a990 con 0x7fa1f406b070 2023-10-16T09:45:04.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.582+0000 7fa202ffd700 1 -- 172.21.15.158:0/660845190 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa20400cce0 con 0x7fa1f406b070 2023-10-16T09:45:04.584 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir8 2023-10-16T09:45:04.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.584+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa1f406b070 msgr2=0x7fa1f406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.584+0000 7fa218b81700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa1f406b070 0x7fa1f406d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa2141f2900 tx=0x7fa20400a000 comp rx=0 tx=0).stop 2023-10-16T09:45:04.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.584+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2140b3df0 msgr2=0x7fa2141ef4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.584+0000 7fa218b81700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2140b3df0 0x7fa2141ef4a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa2140cbd30 tx=0x7fa20c0114b0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 shutdown_connections 2023-10-16T09:45:04.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa1f406b070 0x7fa1f406d530 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2140b3df0 0x7fa2141ef4a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc003a00 0x7fa214158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 --2- 172.21.15.158:0/660845190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa214158790 0x7fa2140b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 >> 172.21.15.158:0/660845190 conn(0x7fa2140b1e90 msgr2=0x7fa2140b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:04.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 shutdown_connections 2023-10-16T09:45:04.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.585+0000 7fa218b81700 1 -- 172.21.15.158:0/660845190 wait complete. 2023-10-16T09:45:04.598 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-10-16T09:45:04.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:45:04.599 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-10-16T09:45:04.618 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2023-10-16T09:45:04.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.782+0000 7f14e603f700 1 Processor -- start 2023-10-16T09:45:04.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.782+0000 7f14e603f700 1 -- start start 2023-10-16T09:45:04.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14e603f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f14e015da40 0x7f14e00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14e603f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14e006afe0 0x7f14e00afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14e603f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e00cbd30 con 0x7f14e00cc590 2023-10-16T09:45:04.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14e603f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e00cc1c0 con 0x7f14e006afe0 2023-10-16T09:45:04.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14e603f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e015de20 con 0x7f14e015da40 2023-10-16T09:45:04.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14e006afe0 0x7f14e00afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14e006afe0 0x7f14e00afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49120/0 (socket says 172.21.15.158:49120) 2023-10-16T09:45:04.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 -- 172.21.15.158:0/1154158975 learned_addr learned my addr 172.21.15.158:0/1154158975 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:04.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14df7fe700 1 --2- 172.21.15.158:0/1154158975 >> v2:172.21.15.172:3300/0 conn(0x7f14e015da40 0x7f14e00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 -- 172.21.15.158:0/1154158975 >> v2:172.21.15.172:3300/0 conn(0x7f14e015da40 msgr2=0x7f14e00caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:04.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 --2- 172.21.15.158:0/1154158975 >> v2:172.21.15.172:3300/0 conn(0x7f14e015da40 0x7f14e00caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 -- 172.21.15.158:0/1154158975 >> v1:172.21.15.172:6789/0 conn(0x7f14e00cc590 legacy=0x7f14e006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 -- 172.21.15.158:0/1154158975 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e00ce260 con 0x7f14e006afe0 2023-10-16T09:45:04.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.783+0000 7f14dffff700 1 --2- 172.21.15.158:0/1154158975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e006afe0 0x7f14e00afa40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f14e006c9c0 tx=0x7f14d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41fea738632f487a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:04.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14deffd700 1 -- 172.21.15.158:0/1154158975 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d800e070 con 0x7f14e006afe0 2023-10-16T09:45:04.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14deffd700 1 -- 172.21.15.158:0/1154158975 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d800b9c0 con 0x7f14e006afe0 2023-10-16T09:45:04.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14deffd700 1 -- 172.21.15.158:0/1154158975 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d800bd70 con 0x7f14e006afe0 2023-10-16T09:45:04.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 -- 172.21.15.158:0/1154158975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e006afe0 msgr2=0x7f14e00afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 --2- 172.21.15.158:0/1154158975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e006afe0 0x7f14e00afa40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f14e006c9c0 tx=0x7f14d800b330 comp rx=0 tx=0).stop 2023-10-16T09:45:04.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 -- 172.21.15.158:0/1154158975 shutdown_connections 2023-10-16T09:45:04.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 --2- 172.21.15.158:0/1154158975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e006afe0 0x7f14e00afa40 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 --2- 172.21.15.158:0/1154158975 >> v2:172.21.15.172:3300/0 conn(0x7f14e015da40 0x7f14e00caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 -- 172.21.15.158:0/1154158975 >> 172.21.15.158:0/1154158975 conn(0x7f14e0099a10 msgr2=0x7f14e00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:04.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 -- 172.21.15.158:0/1154158975 shutdown_connections 2023-10-16T09:45:04.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.784+0000 7f14e603f700 1 -- 172.21.15.158:0/1154158975 wait complete. 2023-10-16T09:45:04.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.785+0000 7f14e603f700 1 Processor -- start 2023-10-16T09:45:04.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14e603f700 1 -- start start 2023-10-16T09:45:04.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14e603f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14e006afe0 0x7f14e00c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14e603f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14e00cc590 0x7f14e01e72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14e603f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e015da40 0x7f14e015b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14e603f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e01e56d0 con 0x7f14e015da40 2023-10-16T09:45:04.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14e603f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e01e50a0 con 0x7f14e006afe0 2023-10-16T09:45:04.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14e603f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00c5ed0 con 0x7f14e00cc590 2023-10-16T09:45:04.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.786+0000 7f14dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e015da40 0x7f14e015b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e015da40 0x7f14e015b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49126/0 (socket says 172.21.15.158:49126) 2023-10-16T09:45:04.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 -- 172.21.15.158:0/1316646015 learned_addr learned my addr 172.21.15.158:0/1316646015 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:04.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14df7fe700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14e006afe0 0x7f14e00c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14d6ffd700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14e00cc590 0x7f14e01e72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 -- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14e00cc590 msgr2=0x7f14e01e72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14e00cc590 0x7f14e01e72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 -- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14e006afe0 msgr2=0x7f14e00c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:04.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14e006afe0 0x7f14e00c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:04.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 -- 172.21.15.158:0/1316646015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8009b30 con 0x7f14e015da40 2023-10-16T09:45:04.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dffff700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e015da40 0x7f14e015b770 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f14d800b370 tx=0x7f14d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:04.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dcff9700 1 -- 172.21.15.158:0/1316646015 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8004900 con 0x7f14e015da40 2023-10-16T09:45:04.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dcff9700 1 -- 172.21.15.158:0/1316646015 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d80099b0 con 0x7f14e015da40 2023-10-16T09:45:04.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e00c61b0 con 0x7f14e015da40 2023-10-16T09:45:04.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14dcff9700 1 -- 172.21.15.158:0/1316646015 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d801c680 con 0x7f14e015da40 2023-10-16T09:45:04.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.787+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e01e7810 con 0x7f14e015da40 2023-10-16T09:45:04.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.788+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e01e5810 con 0x7f14e015da40 2023-10-16T09:45:04.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.788+0000 7f14dcff9700 1 -- 172.21.15.158:0/1316646015 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f14d801c7e0 con 0x7f14e015da40 2023-10-16T09:45:04.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.789+0000 7f14dcff9700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f14c006b100 0x7f14c006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:04.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.789+0000 7f14dcff9700 1 -- 172.21.15.158:0/1316646015 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f14d808b4a0 con 0x7f14e015da40 2023-10-16T09:45:04.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.789+0000 7f14df7fe700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f14c006b100 0x7f14c006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:04.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.789+0000 7f14df7fe700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f14c006b100 0x7f14c006d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14d0006b20 tx=0x7f14d0008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:04.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:04.793+0000 7f14dcff9700 1 -- 172.21.15.158:0/1316646015 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14d8056010 con 0x7f14e015da40 2023-10-16T09:45:05.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.015+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f14e00cb400 con 0x7f14c006b100 2023-10-16T09:45:05.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.076+0000 7f14dcff9700 1 -- 172.21.15.158:0/1316646015 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f14d00076b0 con 0x7f14c006b100 2023-10-16T09:45:05.079 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/dir9 2023-10-16T09:45:05.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.078+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f14c006b100 msgr2=0x7f14c006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.078+0000 7f14e603f700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f14c006b100 0x7f14c006d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14d0006b20 tx=0x7f14d0008040 comp rx=0 tx=0).stop 2023-10-16T09:45:05.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e015da40 msgr2=0x7f14e015b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e015da40 0x7f14e015b770 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f14d800b370 tx=0x7f14d8004ad0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 shutdown_connections 2023-10-16T09:45:05.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f14c006b100 0x7f14c006d5c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e015da40 0x7f14e015b770 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14e006afe0 0x7f14e00c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 --2- 172.21.15.158:0/1316646015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14e00cc590 0x7f14e01e72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 >> 172.21.15.158:0/1316646015 conn(0x7f14e0099a10 msgr2=0x7f14e00b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:05.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 shutdown_connections 2023-10-16T09:45:05.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.079+0000 7f14e603f700 1 -- 172.21.15.158:0/1316646015 wait complete. 2023-10-16T09:45:05.092 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-10-16T09:45:05.093 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2023-10-16T09:45:05.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.214+0000 7f01a7d73700 1 Processor -- start 2023-10-16T09:45:05.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.215+0000 7f01a7d73700 1 -- start start 2023-10-16T09:45:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.215+0000 7f01a7d73700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 0x7f01a0161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.215+0000 7f01a7d73700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a015ee60 con 0x7f01a015b540 2023-10-16T09:45:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.215+0000 7f01a7d73700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a015e2e0 con 0x7f01a006f180 2023-10-16T09:45:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.215+0000 7f01a7d73700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a015b920 con 0x7f01a006afe0 2023-10-16T09:45:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a5b0f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f01a015b540 0x7f01a006d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49306/0 (socket says 172.21.15.158:49306) 2023-10-16T09:45:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a5b0f700 1 -- 172.21.15.158:0/2686661872 learned_addr learned my addr 172.21.15.158:0/2686661872 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015786273 0 0) 0x7f01a015ee60 con 0x7f01a015b540 2023-10-16T09:45:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a6310700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 0x7f01a0161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018c0035b0 con 0x7f01a015b540 2023-10-16T09:45:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2277461316 0 0) 0x7f018c0035b0 con 0x7f01a015b540 2023-10-16T09:45:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 msgr2=0x7f01a0161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 0x7f01a0161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 >> v1:172.21.15.172:6789/0 conn(0x7f01a006f180 legacy=0x7f01a006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a015e5c0 con 0x7f01a015b540 2023-10-16T09:45:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0190001670 con 0x7f01a015b540 2023-10-16T09:45:05.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 >> v1:172.21.15.158:6789/0 conn(0x7f01a015b540 legacy=0x7f01a006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c003a30 0x7f018c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 0x7f018c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c004d90 0x7f018c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a015b920 con 0x7f018c004d90 2023-10-16T09:45:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c00bb70 con 0x7f018c003a30 2023-10-16T09:45:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c00bd30 con 0x7f01a006afe0 2023-10-16T09:45:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a6310700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c004d90 0x7f018c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a530e700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 0x7f018c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.216+0000 7f01a5b0f700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c003a30 0x7f018c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a6310700 1 -- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 msgr2=0x7f018c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a6310700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a006afe0 0x7f018c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a6310700 1 -- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c003a30 msgr2=0x7f018c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a6310700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c003a30 0x7f018c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a6310700 1 -- 172.21.15.158:0/2686661872 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c0035b0 con 0x7f018c004d90 2023-10-16T09:45:05.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a6310700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c004d90 0x7f018c005210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0198009fc0 tx=0x7f018c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f02d60dca88bee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:05.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019800ec30 con 0x7f018c004d90 2023-10-16T09:45:05.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019800f040 con 0x7f018c004d90 2023-10-16T09:45:05.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a4b0d700 1 -- 172.21.15.158:0/2686661872 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198014480 con 0x7f018c004d90 2023-10-16T09:45:05.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a7d73700 1 -- 172.21.15.158:0/2686661872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c004d90 msgr2=0x7f018c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a7d73700 1 --2- 172.21.15.158:0/2686661872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c004d90 0x7f018c005210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0198009fc0 tx=0x7f018c005880 comp rx=0 tx=0).stop 2023-10-16T09:45:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a6310700 1 -- 172.21.15.158:0/2686661872 reap_dead start 2023-10-16T09:45:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.217+0000 7f01a7d73700 1 -- 172.21.15.158:0/2686661872 shutdown_connections 2023-10-16T09:45:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.218+0000 7f01a7d73700 1 -- 172.21.15.158:0/2686661872 >> 172.21.15.158:0/2686661872 conn(0x7f01a00d03b0 msgr2=0x7f01a00d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.218+0000 7f01a7d73700 1 -- 172.21.15.158:0/2686661872 shutdown_connections 2023-10-16T09:45:05.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.218+0000 7f01a7d73700 1 -- 172.21.15.158:0/2686661872 wait complete. 2023-10-16T09:45:05.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.219+0000 7f01a7d73700 1 Processor -- start 2023-10-16T09:45:05.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.219+0000 7f01a7d73700 1 -- start start 2023-10-16T09:45:05.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.219+0000 7f01a7d73700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c004d90 0x7f01a00c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.219+0000 7f01a7d73700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01a00af4b0 0x7f01a0160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a7d73700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a01605c0 0x7f01a00c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a7d73700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00af090 con 0x7f01a00af4b0 2023-10-16T09:45:05.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a7d73700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0160a70 con 0x7f018c004d90 2023-10-16T09:45:05.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a7d73700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00c6d20 con 0x7f01a01605c0 2023-10-16T09:45:05.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01a00af4b0 0x7f01a0160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01a00af4b0 0x7f01a0160080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49144/0 (socket says 172.21.15.158:49144) 2023-10-16T09:45:05.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 -- 172.21.15.158:0/3004866132 learned_addr learned my addr 172.21.15.158:0/3004866132 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:05.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a5b0f700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c004d90 0x7f01a00c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a6310700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a01605c0 0x7f01a00c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 -- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a01605c0 msgr2=0x7f01a00c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a01605c0 0x7f01a00c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 -- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c004d90 msgr2=0x7f01a00c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c004d90 0x7f01a00c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 -- 172.21.15.158:0/3004866132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a00b5b70 con 0x7f01a00af4b0 2023-10-16T09:45:05.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a530e700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01a00af4b0 0x7f01a0160080 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f019c00f7f0 tx=0x7f019c00fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:05.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01967fc700 1 -- 172.21.15.158:0/3004866132 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0052e0 con 0x7f01a00af4b0 2023-10-16T09:45:05.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01967fc700 1 -- 172.21.15.158:0/3004866132 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c005440 con 0x7f01a00af4b0 2023-10-16T09:45:05.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a00ae410 con 0x7f01a00af4b0 2023-10-16T09:45:05.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01967fc700 1 -- 172.21.15.158:0/3004866132 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0038c0 con 0x7f01a00af4b0 2023-10-16T09:45:05.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.220+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a00ae9d0 con 0x7f01a00af4b0 2023-10-16T09:45:05.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.221+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a00681b0 con 0x7f01a00af4b0 2023-10-16T09:45:05.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.222+0000 7f01967fc700 1 -- 172.21.15.158:0/3004866132 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f019c003a20 con 0x7f01a00af4b0 2023-10-16T09:45:05.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.222+0000 7f01967fc700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f018406b100 0x7f018406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.222+0000 7f01967fc700 1 -- 172.21.15.158:0/3004866132 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f019c089820 con 0x7f01a00af4b0 2023-10-16T09:45:05.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.222+0000 7f01a5b0f700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f018406b100 0x7f018406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.223+0000 7f01a5b0f700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f018406b100 0x7f018406d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0190001050 tx=0x7f0190000f80 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:05.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.226+0000 7f01967fc700 1 -- 172.21.15.158:0/3004866132 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f019c0544a0 con 0x7f01a00af4b0 2023-10-16T09:45:05.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.453+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f01a015eb80 con 0x7f018406b100 2023-10-16T09:45:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.503+0000 7f01967fc700 1 -- 172.21.15.158:0/3004866132 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f01900070c0 con 0x7f018406b100 2023-10-16T09:45:05.505 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-10-16T09:45:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f018406b100 msgr2=0x7f018406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f018406b100 0x7f018406d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0190001050 tx=0x7f0190000f80 comp rx=0 tx=0).stop 2023-10-16T09:45:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01a00af4b0 msgr2=0x7f01a0160080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01a00af4b0 0x7f01a0160080 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f019c00f7f0 tx=0x7f019c00fb00 comp rx=0 tx=0).stop 2023-10-16T09:45:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 shutdown_connections 2023-10-16T09:45:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01a01605c0 0x7f01a00c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f018c004d90 0x7f01a00c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f018406b100 0x7f018406d5c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 --2- 172.21.15.158:0/3004866132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01a00af4b0 0x7f01a0160080 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 >> 172.21.15.158:0/3004866132 conn(0x7f01a00d03b0 msgr2=0x7f01a00d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:05.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 shutdown_connections 2023-10-16T09:45:05.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.505+0000 7f01a7d73700 1 -- 172.21.15.158:0/3004866132 wait complete. 2023-10-16T09:45:05.519 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-10-16T09:45:05.519 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2023-10-16T09:45:05.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.643+0000 7fcd4f648700 1 Processor -- start 2023-10-16T09:45:05.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.643+0000 7fcd4f648700 1 -- start start 2023-10-16T09:45:05.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4f648700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcd480ce510 0x7fcd4815f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4f648700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 0x7fcd480cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4f648700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480cf4c0 con 0x7fcd4815b590 2023-10-16T09:45:05.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4f648700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd480cc4c0 con 0x7fcd480ce510 2023-10-16T09:45:05.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4f648700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480b49e0 con 0x7fcd480cbaf0 2023-10-16T09:45:05.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4d3e4700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcd4815b590 0x7fcd480b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49326/0 (socket says 172.21.15.158:49326) 2023-10-16T09:45:05.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4d3e4700 1 -- 172.21.15.158:0/3163552294 learned_addr learned my addr 172.21.15.158:0/3163552294 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:05.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739406697 0 0) 0x7fcd480cf4c0 con 0x7fcd4815b590 2023-10-16T09:45:05.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4cbe3700 1 --2- 172.21.15.158:0/3163552294 >> v2:172.21.15.172:3300/0 conn(0x7fcd480ce510 0x7fcd4815f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd4dbe5700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 0x7fcd480cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd340035b0 con 0x7fcd4815b590 2023-10-16T09:45:05.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3296011664 0 0) 0x7fcd340035b0 con 0x7fcd4815b590 2023-10-16T09:45:05.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 msgr2=0x7fcd480cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:05.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 0x7fcd480cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 >> v2:172.21.15.172:3300/0 conn(0x7fcd480ce510 msgr2=0x7fcd4815f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:05.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 --2- 172.21.15.158:0/3163552294 >> v2:172.21.15.172:3300/0 conn(0x7fcd480ce510 0x7fcd4815f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4815f710 con 0x7fcd4815b590 2023-10-16T09:45:05.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.644+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcd38001670 con 0x7fcd4815b590 2023-10-16T09:45:05.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 >> v1:172.21.15.158:6789/0 conn(0x7fcd4815b590 legacy=0x7fcd480b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34003a30 0x7fcd34003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 0x7fcd34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004d90 0x7fcd34005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480cc4c0 con 0x7fcd34004d90 2023-10-16T09:45:05.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480b49e0 con 0x7fcd34003a30 2023-10-16T09:45:05.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3400bac0 con 0x7fcd480cbaf0 2023-10-16T09:45:05.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4dbe5700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004d90 0x7fcd34005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4cbe3700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 0x7fcd34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4d3e4700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34003a30 0x7fcd34003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4dbe5700 1 -- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 msgr2=0x7fcd34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:05.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4dbe5700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 0x7fcd34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4dbe5700 1 -- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34003a30 msgr2=0x7fcd34003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:05.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4dbe5700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34003a30 0x7fcd34003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4dbe5700 1 -- 172.21.15.158:0/3163552294 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd340035b0 con 0x7fcd34004d90 2023-10-16T09:45:05.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4cbe3700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd480cbaf0 0x7fcd34004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:05.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4d3e4700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34003a30 0x7fcd34003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:05.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd4dbe5700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004d90 0x7fcd34005210 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009fc0 tx=0x7fcd34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894a37a229ee5332 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:05.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4000ec30 con 0x7fcd34004d90 2023-10-16T09:45:05.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4000f040 con 0x7fcd34004d90 2023-10-16T09:45:05.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.645+0000 7fcd3ffff700 1 -- 172.21.15.158:0/3163552294 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40014480 con 0x7fcd34004d90 2023-10-16T09:45:05.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.646+0000 7fcd4f648700 1 -- 172.21.15.158:0/3163552294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004d90 msgr2=0x7fcd34005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.646+0000 7fcd4f648700 1 --2- 172.21.15.158:0/3163552294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004d90 0x7fcd34005210 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009fc0 tx=0x7fcd34005880 comp rx=0 tx=0).stop 2023-10-16T09:45:05.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.646+0000 7fcd4dbe5700 1 -- 172.21.15.158:0/3163552294 reap_dead start 2023-10-16T09:45:05.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.646+0000 7fcd4f648700 1 -- 172.21.15.158:0/3163552294 shutdown_connections 2023-10-16T09:45:05.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.646+0000 7fcd4f648700 1 -- 172.21.15.158:0/3163552294 >> 172.21.15.158:0/3163552294 conn(0x7fcd4806a8d0 msgr2=0x7fcd4806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:05.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.646+0000 7fcd4f648700 1 -- 172.21.15.158:0/3163552294 shutdown_connections 2023-10-16T09:45:05.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.646+0000 7fcd4f648700 1 -- 172.21.15.158:0/3163552294 wait complete. 2023-10-16T09:45:05.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.647+0000 7fcd4f648700 1 Processor -- start 2023-10-16T09:45:05.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.647+0000 7fcd4f648700 1 -- start start 2023-10-16T09:45:05.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4f648700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd34004d90 0x7fcd4814e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4f648700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd4815e600 0x7fcd480c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4f648700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd4815c7d0 0x7fcd480b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4f648700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4815ba10 con 0x7fcd4815e600 2023-10-16T09:45:05.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4f648700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4815bcf0 con 0x7fcd4815c7d0 2023-10-16T09:45:05.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4f648700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4815bfd0 con 0x7fcd34004d90 2023-10-16T09:45:05.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd4815e600 0x7fcd480c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd4815e600 0x7fcd480c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49164/0 (socket says 172.21.15.158:49164) 2023-10-16T09:45:05.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 -- 172.21.15.158:0/261757896 learned_addr learned my addr 172.21.15.158:0/261757896 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:05.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4d3e4700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd34004d90 0x7fcd4814e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4dbe5700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd4815c7d0 0x7fcd480b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 -- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd34004d90 msgr2=0x7fcd4814e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd34004d90 0x7fcd4814e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 -- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd4815c7d0 msgr2=0x7fcd480b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd4815c7d0 0x7fcd480b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 -- 172.21.15.158:0/261757896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4815c2b0 con 0x7fcd4815e600 2023-10-16T09:45:05.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.648+0000 7fcd4cbe3700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd4815e600 0x7fcd480c6c50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcd44010950 tx=0x7fcd44010c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:05.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.649+0000 7fcd3dffb700 1 -- 172.21.15.158:0/261757896 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44004020 con 0x7fcd4815e600 2023-10-16T09:45:05.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.649+0000 7fcd3dffb700 1 -- 172.21.15.158:0/261757896 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44004180 con 0x7fcd4815e600 2023-10-16T09:45:05.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.649+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd480c7550 con 0x7fcd4815e600 2023-10-16T09:45:05.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.649+0000 7fcd3dffb700 1 -- 172.21.15.158:0/261757896 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44004510 con 0x7fcd4815e600 2023-10-16T09:45:05.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.649+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd480c7190 con 0x7fcd4815e600 2023-10-16T09:45:05.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.650+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd480681b0 con 0x7fcd4815e600 2023-10-16T09:45:05.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.650+0000 7fcd3dffb700 1 -- 172.21.15.158:0/261757896 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fcd440036a0 con 0x7fcd4815e600 2023-10-16T09:45:05.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.651+0000 7fcd3dffb700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcd2c06b100 0x7fcd2c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:05.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.651+0000 7fcd4d3e4700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcd2c06b100 0x7fcd2c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:05.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.651+0000 7fcd3dffb700 1 -- 172.21.15.158:0/261757896 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fcd44089e70 con 0x7fcd4815e600 2023-10-16T09:45:05.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.651+0000 7fcd4d3e4700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcd2c06b100 0x7fcd2c06d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd380017a0 tx=0x7fcd38007040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:05.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.655+0000 7fcd3dffb700 1 -- 172.21.15.158:0/261757896 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd44054a60 con 0x7fcd4815e600 2023-10-16T09:45:05.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.882+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd4815be60 con 0x7fcd2c06b100 2023-10-16T09:45:05.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.932+0000 7fcd3dffb700 1 -- 172.21.15.158:0/261757896 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcd4815be60 con 0x7fcd2c06b100 2023-10-16T09:45:05.934 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-10-16T09:45:05.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcd2c06b100 msgr2=0x7fcd2c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcd2c06b100 0x7fcd2c06d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd380017a0 tx=0x7fcd38007040 comp rx=0 tx=0).stop 2023-10-16T09:45:05.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd4815e600 msgr2=0x7fcd480c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:05.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd4815e600 0x7fcd480c6c50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcd44010950 tx=0x7fcd44010c60 comp rx=0 tx=0).stop 2023-10-16T09:45:05.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 shutdown_connections 2023-10-16T09:45:05.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd4815c7d0 0x7fcd480b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd34004d90 0x7fcd4814e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcd2c06b100 0x7fcd2c06d5c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 --2- 172.21.15.158:0/261757896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd4815e600 0x7fcd480c6c50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:05.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.934+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 >> 172.21.15.158:0/261757896 conn(0x7fcd4806a8d0 msgr2=0x7fcd480b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:05.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.935+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 shutdown_connections 2023-10-16T09:45:05.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:05.935+0000 7fcd4f648700 1 -- 172.21.15.158:0/261757896 wait complete. 2023-10-16T09:45:05.948 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-10-16T09:45:05.948 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2023-10-16T09:45:06.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.070+0000 7f11c1721700 1 Processor -- start 2023-10-16T09:45:06.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.070+0000 7f11c1721700 1 -- start start 2023-10-16T09:45:06.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.070+0000 7f11c1721700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f11bc15d840 0x7f11bc15dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.070+0000 7f11c1721700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0b38f0 0x7f11bc0d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.070+0000 7f11c1721700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc15e160 con 0x7f11bc0d2010 2023-10-16T09:45:06.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.070+0000 7f11c1721700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc15ec80 con 0x7f11bc15d840 2023-10-16T09:45:06.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.070+0000 7f11c1721700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0d13b0 con 0x7f11bc0b38f0 2023-10-16T09:45:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f11bc15d840 0x7f11bc15dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f11bc15d840 0x7f11bc15dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49170/0 (socket says 172.21.15.158:49170) 2023-10-16T09:45:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 -- 172.21.15.158:0/3202174148 learned_addr learned my addr 172.21.15.158:0/3202174148 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11ba7fc700 1 --2- 172.21.15.158:0/3202174148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0b38f0 0x7f11bc0d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 -- 172.21.15.158:0/3202174148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0b38f0 msgr2=0x7f11bc0d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 --2- 172.21.15.158:0/3202174148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0b38f0 0x7f11bc0d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 -- 172.21.15.158:0/3202174148 >> v1:172.21.15.172:6789/0 conn(0x7f11bc0d2010 legacy=0x7f11bc06bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 -- 172.21.15.158:0/3202174148 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc15e3e0 con 0x7f11bc15d840 2023-10-16T09:45:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11baffd700 1 --2- 172.21.15.158:0/3202174148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc15d840 0x7f11bc15dc20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0098f0 tx=0x7f11ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc87a45c6854ad68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11b9ffb700 1 -- 172.21.15.158:0/3202174148 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac004230 con 0x7f11bc15d840 2023-10-16T09:45:06.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11b9ffb700 1 -- 172.21.15.158:0/3202174148 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac004390 con 0x7f11bc15d840 2023-10-16T09:45:06.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.071+0000 7f11b9ffb700 1 -- 172.21.15.158:0/3202174148 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0036a0 con 0x7f11bc15d840 2023-10-16T09:45:06.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 -- 172.21.15.158:0/3202174148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc15d840 msgr2=0x7f11bc15dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 --2- 172.21.15.158:0/3202174148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc15d840 0x7f11bc15dc20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0098f0 tx=0x7f11ac009c00 comp rx=0 tx=0).stop 2023-10-16T09:45:06.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 -- 172.21.15.158:0/3202174148 shutdown_connections 2023-10-16T09:45:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 --2- 172.21.15.158:0/3202174148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc15d840 0x7f11bc15dc20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 --2- 172.21.15.158:0/3202174148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0b38f0 0x7f11bc0d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 -- 172.21.15.158:0/3202174148 >> 172.21.15.158:0/3202174148 conn(0x7f11bc06ec00 msgr2=0x7f11bc06efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 -- 172.21.15.158:0/3202174148 shutdown_connections 2023-10-16T09:45:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.072+0000 7f11c1721700 1 -- 172.21.15.158:0/3202174148 wait complete. 2023-10-16T09:45:06.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.073+0000 7f11c1721700 1 Processor -- start 2023-10-16T09:45:06.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.073+0000 7f11c1721700 1 -- start start 2023-10-16T09:45:06.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11c1721700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc0b38f0 0x7f11bc0b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11c1721700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0d2010 0x7f11bc1f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11c1721700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11bc15d840 0x7f11bc1e8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11c1721700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc1e85f0 con 0x7f11bc0b38f0 2023-10-16T09:45:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11c1721700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc1e7a90 con 0x7f11bc15d840 2023-10-16T09:45:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11c1721700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc1e7d50 con 0x7f11bc0d2010 2023-10-16T09:45:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc0b38f0 0x7f11bc0b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc0b38f0 0x7f11bc0b5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49176/0 (socket says 172.21.15.158:49176) 2023-10-16T09:45:06.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 -- 172.21.15.158:0/3344340820 learned_addr learned my addr 172.21.15.158:0/3344340820 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:06.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11ba7fc700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0d2010 0x7f11bc1f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11bb7fe700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11bc15d840 0x7f11bc1e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 -- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0d2010 msgr2=0x7f11bc1f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0d2010 0x7f11bc1f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 -- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11bc15d840 msgr2=0x7f11bc1e8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11bc15d840 0x7f11bc1e8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.074+0000 7f11baffd700 1 -- 172.21.15.158:0/3344340820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac009550 con 0x7f11bc0b38f0 2023-10-16T09:45:06.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.075+0000 7f11baffd700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc0b38f0 0x7f11bc0b5dd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009130 tx=0x7f11ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.075+0000 7f11a37fe700 1 -- 172.21.15.158:0/3344340820 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac00b860 con 0x7f11bc0b38f0 2023-10-16T09:45:06.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.075+0000 7f11a37fe700 1 -- 172.21.15.158:0/3344340820 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac00b9c0 con 0x7f11bc0b38f0 2023-10-16T09:45:06.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.075+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc1e8030 con 0x7f11bc0b38f0 2023-10-16T09:45:06.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.075+0000 7f11a37fe700 1 -- 172.21.15.158:0/3344340820 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac00bd70 con 0x7f11bc0b38f0 2023-10-16T09:45:06.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.075+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc0b6310 con 0x7f11bc0b38f0 2023-10-16T09:45:06.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.076+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc0681b0 con 0x7f11bc0b38f0 2023-10-16T09:45:06.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.077+0000 7f11a37fe700 1 -- 172.21.15.158:0/3344340820 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f11ac01b440 con 0x7f11bc0b38f0 2023-10-16T09:45:06.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.078+0000 7f11a37fe700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f11a806b100 0x7f11a806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.078+0000 7f11a37fe700 1 -- 172.21.15.158:0/3344340820 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f11ac08b6f0 con 0x7f11bc0b38f0 2023-10-16T09:45:06.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.078+0000 7f11ba7fc700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f11a806b100 0x7f11a806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.078+0000 7f11ba7fc700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f11a806b100 0x7f11a806d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f11bc06e030 tx=0x7f11b0008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.081+0000 7f11a37fe700 1 -- 172.21.15.158:0/3344340820 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11ac056470 con 0x7f11bc0b38f0 2023-10-16T09:45:06.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.302+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f11bc15e9a0 con 0x7f11a806b100 2023-10-16T09:45:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.349+0000 7f11a37fe700 1 -- 172.21.15.158:0/3344340820 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f11b000ae30 con 0x7f11a806b100 2023-10-16T09:45:06.351 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-10-16T09:45:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.351+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f11a806b100 msgr2=0x7f11a806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.351+0000 7f11c1721700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f11a806b100 0x7f11a806d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f11bc06e030 tx=0x7f11b0008040 comp rx=0 tx=0).stop 2023-10-16T09:45:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc0b38f0 msgr2=0x7f11bc0b5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc0b38f0 0x7f11bc0b5dd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009130 tx=0x7f11ac004750 comp rx=0 tx=0).stop 2023-10-16T09:45:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 shutdown_connections 2023-10-16T09:45:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11bc15d840 0x7f11bc1e8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f11a806b100 0x7f11a806d5c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11bc0b38f0 0x7f11bc0b5dd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 --2- 172.21.15.158:0/3344340820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11bc0d2010 0x7f11bc1f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 >> 172.21.15.158:0/3344340820 conn(0x7f11bc06ec00 msgr2=0x7f11bc159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 shutdown_connections 2023-10-16T09:45:06.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.352+0000 7f11c1721700 1 -- 172.21.15.158:0/3344340820 wait complete. 2023-10-16T09:45:06.365 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-10-16T09:45:06.366 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2023-10-16T09:45:06.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.489+0000 7f706469f700 1 Processor -- start 2023-10-16T09:45:06.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.489+0000 7f706469f700 1 -- start start 2023-10-16T09:45:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706469f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f705c15da40 0x7f705c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706469f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 0x7f705c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706469f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c0cbd30 con 0x7f705c06ae10 2023-10-16T09:45:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706469f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705c06b2c0 con 0x7f705c15da40 2023-10-16T09:45:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706469f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705c0cc1c0 con 0x7f705c0cc590 2023-10-16T09:45:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f705c15da40 0x7f705c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f705c15da40 0x7f705c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49184/0 (socket says 172.21.15.158:49184) 2023-10-16T09:45:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 -- 172.21.15.158:0/2568981056 learned_addr learned my addr 172.21.15.158:0/2568981056 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 -- 172.21.15.158:0/2568981056 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 msgr2=0x7f705c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 --2- 172.21.15.158:0/2568981056 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 0x7f705c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 -- 172.21.15.158:0/2568981056 >> v1:172.21.15.158:6789/0 conn(0x7f705c06ae10 legacy=0x7f705c0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 -- 172.21.15.158:0/2568981056 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c15de20 con 0x7f705c15da40 2023-10-16T09:45:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f706243b700 1 --2- 172.21.15.158:0/2568981056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c15da40 0x7f705c0caa20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f70580098f0 tx=0x7f7058009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f91d87ca1fdaf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f7061439700 1 -- 172.21.15.158:0/2568981056 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800b8a0 con 0x7f705c15da40 2023-10-16T09:45:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f7061439700 1 -- 172.21.15.158:0/2568981056 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705800ba00 con 0x7f705c15da40 2023-10-16T09:45:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.490+0000 7f7061439700 1 -- 172.21.15.158:0/2568981056 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800bd90 con 0x7f705c15da40 2023-10-16T09:45:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 -- 172.21.15.158:0/2568981056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c15da40 msgr2=0x7f705c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 --2- 172.21.15.158:0/2568981056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c15da40 0x7f705c0caa20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f70580098f0 tx=0x7f7058009c00 comp rx=0 tx=0).stop 2023-10-16T09:45:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 -- 172.21.15.158:0/2568981056 shutdown_connections 2023-10-16T09:45:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 --2- 172.21.15.158:0/2568981056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c15da40 0x7f705c0caa20 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 --2- 172.21.15.158:0/2568981056 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 0x7f705c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 -- 172.21.15.158:0/2568981056 >> 172.21.15.158:0/2568981056 conn(0x7f705c099a10 msgr2=0x7f705c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 -- 172.21.15.158:0/2568981056 shutdown_connections 2023-10-16T09:45:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.491+0000 7f706469f700 1 -- 172.21.15.158:0/2568981056 wait complete. 2023-10-16T09:45:06.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.492+0000 7f706469f700 1 Processor -- start 2023-10-16T09:45:06.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.492+0000 7f706469f700 1 -- start start 2023-10-16T09:45:06.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706469f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c06ae10 0x7f705c14c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706469f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 0x7f705c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706469f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f705c15da40 0x7f705c0b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706469f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c14cd70 con 0x7f705c06ae10 2023-10-16T09:45:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706469f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c15e030 con 0x7f705c15da40 2023-10-16T09:45:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706469f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705c15e310 con 0x7f705c0cc590 2023-10-16T09:45:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c06ae10 0x7f705c14c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c06ae10 0x7f705c14c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49190/0 (socket says 172.21.15.158:49190) 2023-10-16T09:45:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 -- 172.21.15.158:0/472786049 learned_addr learned my addr 172.21.15.158:0/472786049 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 -- 172.21.15.158:0/472786049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 msgr2=0x7f705c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 0x7f705c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 -- 172.21.15.158:0/472786049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f705c15da40 msgr2=0x7f705c0b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f705c15da40 0x7f705c0b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 -- 172.21.15.158:0/472786049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7058009550 con 0x7f705c06ae10 2023-10-16T09:45:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.493+0000 7f706243b700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c06ae10 0x7f705c14c830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7058014040 tx=0x7f7058004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.494+0000 7f7052ffd700 1 -- 172.21.15.158:0/472786049 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70580036a0 con 0x7f705c06ae10 2023-10-16T09:45:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.494+0000 7f7052ffd700 1 -- 172.21.15.158:0/472786049 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705801e070 con 0x7f705c06ae10 2023-10-16T09:45:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.494+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c15e5f0 con 0x7f705c06ae10 2023-10-16T09:45:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.494+0000 7f7052ffd700 1 -- 172.21.15.158:0/472786049 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058003a60 con 0x7f705c06ae10 2023-10-16T09:45:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.494+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705c15ebb0 con 0x7f705c06ae10 2023-10-16T09:45:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.495+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705c0681b0 con 0x7f705c06ae10 2023-10-16T09:45:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.495+0000 7f7052ffd700 1 -- 172.21.15.158:0/472786049 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f7058003bc0 con 0x7f705c06ae10 2023-10-16T09:45:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.495+0000 7f7052ffd700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f704006b100 0x7f704006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.496+0000 7f7052ffd700 1 -- 172.21.15.158:0/472786049 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f7058020070 con 0x7f705c06ae10 2023-10-16T09:45:06.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.496+0000 7f7061c3a700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f704006b100 0x7f704006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.496+0000 7f7061c3a700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f704006b100 0x7f704006d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f705c0b5dc0 tx=0x7f704c006d20 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.500+0000 7f7052ffd700 1 -- 172.21.15.158:0/472786049 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7058055d70 con 0x7f705c06ae10 2023-10-16T09:45:06.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.718+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f705c0cb120 con 0x7f704006b100 2023-10-16T09:45:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.769+0000 7f7052ffd700 1 -- 172.21.15.158:0/472786049 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f704c0093c0 con 0x7f704006b100 2023-10-16T09:45:06.771 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-10-16T09:45:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.771+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f704006b100 msgr2=0x7f704006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.771+0000 7f706469f700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f704006b100 0x7f704006d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f705c0b5dc0 tx=0x7f704c006d20 comp rx=0 tx=0).stop 2023-10-16T09:45:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.771+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c06ae10 msgr2=0x7f705c14c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.771+0000 7f706469f700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c06ae10 0x7f705c14c830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7058014040 tx=0x7f7058004750 comp rx=0 tx=0).stop 2023-10-16T09:45:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 shutdown_connections 2023-10-16T09:45:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f705c15da40 0x7f705c0b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f704006b100 0x7f704006d5c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f705c06ae10 0x7f705c14c830 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 --2- 172.21.15.158:0/472786049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f705c0cc590 0x7f705c0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 >> 172.21.15.158:0/472786049 conn(0x7f705c099a10 msgr2=0x7f705c0b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 shutdown_connections 2023-10-16T09:45:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.772+0000 7f706469f700 1 -- 172.21.15.158:0/472786049 wait complete. 2023-10-16T09:45:06.785 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-10-16T09:45:06.785 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2023-10-16T09:45:06.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 Processor -- start 2023-10-16T09:45:06.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 -- start start 2023-10-16T09:45:06.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f469015d840 0x7f469015dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46900b38f0 0x7f46900d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46900d2010 0x7f469006bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469015ec80 con 0x7f46900b38f0 2023-10-16T09:45:06.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46900d13b0 con 0x7f469015d840 2023-10-16T09:45:06.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.908+0000 7f4695652700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469015e160 con 0x7f46900d2010 2023-10-16T09:45:06.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46900b38f0 0x7f46900d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46900b38f0 0x7f46900d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49200/0 (socket says 172.21.15.158:49200) 2023-10-16T09:45:06.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 -- 172.21.15.158:0/1316387871 learned_addr learned my addr 172.21.15.158:0/1316387871 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:06.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468effd700 1 --2- 172.21.15.158:0/1316387871 >> v2:172.21.15.172:3300/0 conn(0x7f469015d840 0x7f469015dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 -- 172.21.15.158:0/1316387871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46900d2010 msgr2=0x7f469006bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:06.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468f7fe700 1 --2- 172.21.15.158:0/1316387871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46900d2010 0x7f469006bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 --2- 172.21.15.158:0/1316387871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46900d2010 0x7f469006bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 -- 172.21.15.158:0/1316387871 >> v2:172.21.15.172:3300/0 conn(0x7f469015d840 msgr2=0x7f469015dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:06.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 --2- 172.21.15.158:0/1316387871 >> v2:172.21.15.172:3300/0 conn(0x7f469015d840 0x7f469015dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 -- 172.21.15.158:0/1316387871 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469015e3e0 con 0x7f46900b38f0 2023-10-16T09:45:06.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468e7fc700 1 --2- 172.21.15.158:0/1316387871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900b38f0 0x7f46900d1ad0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f46900b3090 tx=0x7f4684009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aff1257741ec332 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468dffb700 1 -- 172.21.15.158:0/1316387871 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400b870 con 0x7f46900b38f0 2023-10-16T09:45:06.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468dffb700 1 -- 172.21.15.158:0/1316387871 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468400b9d0 con 0x7f46900b38f0 2023-10-16T09:45:06.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.909+0000 7f468dffb700 1 -- 172.21.15.158:0/1316387871 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400bd60 con 0x7f46900b38f0 2023-10-16T09:45:06.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 -- 172.21.15.158:0/1316387871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900b38f0 msgr2=0x7f46900d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 --2- 172.21.15.158:0/1316387871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900b38f0 0x7f46900d1ad0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f46900b3090 tx=0x7f4684009bd0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 -- 172.21.15.158:0/1316387871 shutdown_connections 2023-10-16T09:45:06.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 --2- 172.21.15.158:0/1316387871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46900d2010 0x7f469006bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 --2- 172.21.15.158:0/1316387871 >> v2:172.21.15.172:3300/0 conn(0x7f469015d840 0x7f469015dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 --2- 172.21.15.158:0/1316387871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900b38f0 0x7f46900d1ad0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 -- 172.21.15.158:0/1316387871 >> 172.21.15.158:0/1316387871 conn(0x7f469006ec00 msgr2=0x7f469006efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:06.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 -- 172.21.15.158:0/1316387871 shutdown_connections 2023-10-16T09:45:06.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.910+0000 7f4695652700 1 -- 172.21.15.158:0/1316387871 wait complete. 2023-10-16T09:45:06.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.911+0000 7f4695652700 1 Processor -- start 2023-10-16T09:45:06.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.911+0000 7f4695652700 1 -- start start 2023-10-16T09:45:06.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f4695652700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46900b38f0 0x7f4690156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f4695652700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900d2010 0x7f46900c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f4695652700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f469015d840 0x7f4690152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f4695652700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4690156230 con 0x7f46900d2010 2023-10-16T09:45:06.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f4695652700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46900c5ea0 con 0x7f46900b38f0 2023-10-16T09:45:06.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f4695652700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46900b4be0 con 0x7f469015d840 2023-10-16T09:45:06.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900d2010 0x7f46900c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900d2010 0x7f46900c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49206/0 (socket says 172.21.15.158:49206) 2023-10-16T09:45:06.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 -- 172.21.15.158:0/3912960733 learned_addr learned my addr 172.21.15.158:0/3912960733 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:06.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468effd700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46900b38f0 0x7f4690156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468f7fe700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f469015d840 0x7f4690152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 -- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f469015d840 msgr2=0x7f4690152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f469015d840 0x7f4690152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 -- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46900b38f0 msgr2=0x7f4690156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:06.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46900b38f0 0x7f4690156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:06.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468e7fc700 1 -- 172.21.15.158:0/3912960733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684009550 con 0x7f46900d2010 2023-10-16T09:45:06.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468effd700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46900b38f0 0x7f4690156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:06.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.912+0000 7f468f7fe700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f469015d840 0x7f4690152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:06.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.913+0000 7f468e7fc700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900d2010 0x7f46900c57a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4684014040 tx=0x7f4684004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.913+0000 7f46777fe700 1 -- 172.21.15.158:0/3912960733 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684004520 con 0x7f46900d2010 2023-10-16T09:45:06.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.913+0000 7f46777fe700 1 -- 172.21.15.158:0/3912960733 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468401e070 con 0x7f46900d2010 2023-10-16T09:45:06.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.913+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46900b4ec0 con 0x7f46900d2010 2023-10-16T09:45:06.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.913+0000 7f46777fe700 1 -- 172.21.15.158:0/3912960733 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840038e0 con 0x7f46900d2010 2023-10-16T09:45:06.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.913+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46900b5480 con 0x7f46900d2010 2023-10-16T09:45:06.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.914+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46900681b0 con 0x7f46900d2010 2023-10-16T09:45:06.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.914+0000 7f46777fe700 1 -- 172.21.15.158:0/3912960733 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f4684003a40 con 0x7f46900d2010 2023-10-16T09:45:06.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.915+0000 7f46777fe700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f467c06b100 0x7f467c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:06.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.915+0000 7f46777fe700 1 -- 172.21.15.158:0/3912960733 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f4684020070 con 0x7f46900d2010 2023-10-16T09:45:06.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.915+0000 7f468effd700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f467c06b100 0x7f467c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:06.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.915+0000 7f468effd700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f467c06b100 0x7f467c06d5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f468000aa50 tx=0x7f4680008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:06.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:06.919+0000 7f46777fe700 1 -- 172.21.15.158:0/3912960733 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4684055d90 con 0x7f46900d2010 2023-10-16T09:45:07.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.138+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f46900b4d50 con 0x7f467c06b100 2023-10-16T09:45:07.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.183+0000 7f46777fe700 1 -- 172.21.15.158:0/3912960733 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f46900b4d50 con 0x7f467c06b100 2023-10-16T09:45:07.185 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-10-16T09:45:07.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.185+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f467c06b100 msgr2=0x7f467c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.185+0000 7f4695652700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f467c06b100 0x7f467c06d5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f468000aa50 tx=0x7f4680008040 comp rx=0 tx=0).stop 2023-10-16T09:45:07.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.185+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900d2010 msgr2=0x7f46900c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.185+0000 7f4695652700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900d2010 0x7f46900c57a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4684014040 tx=0x7f4684004750 comp rx=0 tx=0).stop 2023-10-16T09:45:07.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.185+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 shutdown_connections 2023-10-16T09:45:07.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.186+0000 7f4695652700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f469015d840 0x7f4690152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.186+0000 7f4695652700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46900b38f0 0x7f4690156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.186+0000 7f4695652700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f467c06b100 0x7f467c06d5c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.186+0000 7f4695652700 1 --2- 172.21.15.158:0/3912960733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46900d2010 0x7f46900c57a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.186+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 >> 172.21.15.158:0/3912960733 conn(0x7f469006ec00 msgr2=0x7f469006efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:07.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.186+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 shutdown_connections 2023-10-16T09:45:07.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.186+0000 7f4695652700 1 -- 172.21.15.158:0/3912960733 wait complete. 2023-10-16T09:45:07.199 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-10-16T09:45:07.200 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2023-10-16T09:45:07.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.322+0000 7f7e3ae23700 1 Processor -- start 2023-10-16T09:45:07.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.322+0000 7f7e3ae23700 1 -- start start 2023-10-16T09:45:07.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e3ae23700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7e3406e3b0 0x7f7e3406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e3ae23700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3406ed00 con 0x7f7e3415e870 2023-10-16T09:45:07.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e3ae23700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340ce160 con 0x7f7e3415e490 2023-10-16T09:45:07.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e3ae23700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3406ccd0 con 0x7f7e3406e3b0 2023-10-16T09:45:07.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e33fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7e3415e870 0x7f7e340cd6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49390/0 (socket says 172.21.15.158:49390) 2023-10-16T09:45:07.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e33fff700 1 -- 172.21.15.158:0/4235345214 learned_addr learned my addr 172.21.15.158:0/4235345214 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:07.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e393c0700 1 --2- 172.21.15.158:0/4235345214 >> v2:172.21.15.172:3300/0 conn(0x7f7e3406e3b0 0x7f7e3406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312503400 0 0) 0x7f7e3406ed00 con 0x7f7e3415e870 2023-10-16T09:45:07.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.323+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e240035b0 con 0x7f7e3415e870 2023-10-16T09:45:07.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281421154 0 0) 0x7f7e340ce160 con 0x7f7e3415e490 2023-10-16T09:45:07.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3406ed00 con 0x7f7e3415e490 2023-10-16T09:45:07.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720591016 0 0) 0x7f7e240035b0 con 0x7f7e3415e870 2023-10-16T09:45:07.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 >> v2:172.21.15.172:3300/0 conn(0x7f7e3406e3b0 msgr2=0x7f7e3406e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:07.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 --2- 172.21.15.158:0/4235345214 >> v2:172.21.15.172:3300/0 conn(0x7f7e3406e3b0 0x7f7e3406e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 >> v1:172.21.15.172:6789/0 conn(0x7f7e3415e490 legacy=0x7f7e3406da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3406cfb0 con 0x7f7e3415e870 2023-10-16T09:45:07.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7e20001670 con 0x7f7e3415e870 2023-10-16T09:45:07.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 >> v1:172.21.15.158:6789/0 conn(0x7f7e3415e870 legacy=0x7f7e340cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24003b90 0x7f7e24003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e24004850 0x7f7e24004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e24005210 0x7f7e2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3406ccd0 con 0x7f7e24004850 2023-10-16T09:45:07.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2400c090 con 0x7f7e24005210 2023-10-16T09:45:07.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2400c250 con 0x7f7e24003b90 2023-10-16T09:45:07.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e33fff700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e24004850 0x7f7e24004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e393c0700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e24005210 0x7f7e2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e38bbf700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24003b90 0x7f7e24003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e33fff700 1 -- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24003b90 msgr2=0x7f7e24003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:07.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e33fff700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24003b90 0x7f7e24003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e33fff700 1 -- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e24005210 msgr2=0x7f7e2400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:07.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e33fff700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e24005210 0x7f7e2400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e393c0700 1 -- 172.21.15.158:0/4235345214 reap_dead start 2023-10-16T09:45:07.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e33fff700 1 -- 172.21.15.158:0/4235345214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e240035b0 con 0x7f7e24004850 2023-10-16T09:45:07.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.324+0000 7f7e33fff700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e24004850 0x7f7e24004cd0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000c210 tx=0x7f7e24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43d5fe1e1d0d157 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:07.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2000cd10 con 0x7f7e24004850 2023-10-16T09:45:07.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2000ce90 con 0x7f7e24004850 2023-10-16T09:45:07.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e337fe700 1 -- 172.21.15.158:0/4235345214 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20005ba0 con 0x7f7e24004850 2023-10-16T09:45:07.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4235345214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e24004850 msgr2=0x7f7e24004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e24004850 0x7f7e24004cd0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000c210 tx=0x7f7e24005880 comp rx=0 tx=0).stop 2023-10-16T09:45:07.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4235345214 shutdown_connections 2023-10-16T09:45:07.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4235345214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e24004850 0x7f7e24004cd0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4235345214 >> 172.21.15.158:0/4235345214 conn(0x7f7e340b15c0 msgr2=0x7f7e340b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:07.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4235345214 shutdown_connections 2023-10-16T09:45:07.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.325+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4235345214 wait complete. 2023-10-16T09:45:07.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.326+0000 7f7e3ae23700 1 Processor -- start 2023-10-16T09:45:07.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.326+0000 7f7e3ae23700 1 -- start start 2023-10-16T09:45:07.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e3ae23700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24004850 0x7f7e34152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e3ae23700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e3415bfd0 0x7f7e340c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e3ae23700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3415ee40 0x7f7e3415f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e3ae23700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e340c78d0 con 0x7f7e3415ee40 2023-10-16T09:45:07.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e3ae23700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3415c870 con 0x7f7e3415bfd0 2023-10-16T09:45:07.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e3ae23700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3415f800 con 0x7f7e24004850 2023-10-16T09:45:07.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e393c0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3415ee40 0x7f7e3415f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e393c0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3415ee40 0x7f7e3415f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49228/0 (socket says 172.21.15.158:49228) 2023-10-16T09:45:07.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.327+0000 7f7e393c0700 1 -- 172.21.15.158:0/4218512783 learned_addr learned my addr 172.21.15.158:0/4218512783 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:07.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e38bbf700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24004850 0x7f7e34152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e33fff700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e3415bfd0 0x7f7e340c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e393c0700 1 -- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24004850 msgr2=0x7f7e34152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e393c0700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24004850 0x7f7e34152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e393c0700 1 -- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e3415bfd0 msgr2=0x7f7e340c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e393c0700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e3415bfd0 0x7f7e340c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e393c0700 1 -- 172.21.15.158:0/4218512783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340c70e0 con 0x7f7e3415ee40 2023-10-16T09:45:07.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e33fff700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e3415bfd0 0x7f7e340c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:07.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e38bbf700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24004850 0x7f7e34152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:07.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e393c0700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3415ee40 0x7f7e3415f2c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7e34066880 tx=0x7f7e2c011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:07.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e317fa700 1 -- 172.21.15.158:0/4218512783 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c00e040 con 0x7f7e3415ee40 2023-10-16T09:45:07.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e317fa700 1 -- 172.21.15.158:0/4218512783 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c00f040 con 0x7f7e3415ee40 2023-10-16T09:45:07.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e340b32f0 con 0x7f7e3415ee40 2023-10-16T09:45:07.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e317fa700 1 -- 172.21.15.158:0/4218512783 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c00fd50 con 0x7f7e3415ee40 2023-10-16T09:45:07.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.328+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e340c75f0 con 0x7f7e3415ee40 2023-10-16T09:45:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.329+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e340b3480 con 0x7f7e3415ee40 2023-10-16T09:45:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.330+0000 7f7e317fa700 1 -- 172.21.15.158:0/4218512783 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f7e2c022460 con 0x7f7e3415ee40 2023-10-16T09:45:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.330+0000 7f7e317fa700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e1806b100 0x7f7e1806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.330+0000 7f7e317fa700 1 -- 172.21.15.158:0/4218512783 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f7e2c08a200 con 0x7f7e3415ee40 2023-10-16T09:45:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.330+0000 7f7e38bbf700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e1806b100 0x7f7e1806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.330+0000 7f7e38bbf700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e1806b100 0x7f7e1806d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000c00 tx=0x7f7e28009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.334+0000 7f7e317fa700 1 -- 172.21.15.158:0/4218512783 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e2c054ce0 con 0x7f7e3415ee40 2023-10-16T09:45:07.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.555+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e3406d290 con 0x7f7e1806b100 2023-10-16T09:45:07.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.604+0000 7f7e317fa700 1 -- 172.21.15.158:0/4218512783 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7e28005230 con 0x7f7e1806b100 2023-10-16T09:45:07.606 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-10-16T09:45:07.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e1806b100 msgr2=0x7f7e1806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e1806b100 0x7f7e1806d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000c00 tx=0x7f7e28009040 comp rx=0 tx=0).stop 2023-10-16T09:45:07.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3415ee40 msgr2=0x7f7e3415f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3415ee40 0x7f7e3415f2c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7e34066880 tx=0x7f7e2c011f80 comp rx=0 tx=0).stop 2023-10-16T09:45:07.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 shutdown_connections 2023-10-16T09:45:07.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e1806b100 0x7f7e1806d5c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3415ee40 0x7f7e3415f2c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e24004850 0x7f7e34152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 --2- 172.21.15.158:0/4218512783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e3415bfd0 0x7f7e340c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 >> 172.21.15.158:0/4218512783 conn(0x7f7e340b15c0 msgr2=0x7f7e34160490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:07.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 shutdown_connections 2023-10-16T09:45:07.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.607+0000 7f7e3ae23700 1 -- 172.21.15.158:0/4218512783 wait complete. 2023-10-16T09:45:07.620 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-10-16T09:45:07.621 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2023-10-16T09:45:07.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.743+0000 7fdefba7e700 1 Processor -- start 2023-10-16T09:45:07.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.744+0000 7fdefba7e700 1 -- start start 2023-10-16T09:45:07.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.744+0000 7fdefba7e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdef40cc590 0x7fdef40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.744+0000 7fdefba7e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40cbd30 con 0x7fdef406afe0 2023-10-16T09:45:07.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.744+0000 7fdefba7e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40cc1c0 con 0x7fdef415da40 2023-10-16T09:45:07.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.744+0000 7fdefba7e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef415de20 con 0x7fdef40cc590 2023-10-16T09:45:07.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdefa01b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdef40cc590 0x7fdef40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdefa01b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdef40cc590 0x7fdef40caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49236/0 (socket says 172.21.15.158:49236) 2023-10-16T09:45:07.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdefa01b700 1 -- 172.21.15.158:0/2581993908 learned_addr learned my addr 172.21.15.158:0/2581993908 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:07.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdefa01b700 1 -- 172.21.15.158:0/2581993908 >> v1:172.21.15.172:6789/0 conn(0x7fdef415da40 legacy=0x7fdef406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:07.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdefa01b700 1 -- 172.21.15.158:0/2581993908 >> v1:172.21.15.158:6789/0 conn(0x7fdef406afe0 legacy=0x7fdef40afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdefa01b700 1 -- 172.21.15.158:0/2581993908 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40ce260 con 0x7fdef40cc590 2023-10-16T09:45:07.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdefa01b700 1 --2- 172.21.15.158:0/2581993908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef40cc590 0x7fdef40caa20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009e50 tx=0x7fdeec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b8a6841c9915c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:07.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdef8818700 1 -- 172.21.15.158:0/2581993908 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00e070 con 0x7fdef40cc590 2023-10-16T09:45:07.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdef8818700 1 -- 172.21.15.158:0/2581993908 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec003a20 con 0x7fdef40cc590 2023-10-16T09:45:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.745+0000 7fdef8818700 1 -- 172.21.15.158:0/2581993908 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec003dd0 con 0x7fdef40cc590 2023-10-16T09:45:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.746+0000 7fdefba7e700 1 -- 172.21.15.158:0/2581993908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef40cc590 msgr2=0x7fdef40caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.746+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2581993908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef40cc590 0x7fdef40caa20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009e50 tx=0x7fdeec00b330 comp rx=0 tx=0).stop 2023-10-16T09:45:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.746+0000 7fdefba7e700 1 -- 172.21.15.158:0/2581993908 shutdown_connections 2023-10-16T09:45:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.746+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2581993908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef40cc590 0x7fdef40caa20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.746+0000 7fdefba7e700 1 -- 172.21.15.158:0/2581993908 >> 172.21.15.158:0/2581993908 conn(0x7fdef4099a10 msgr2=0x7fdef40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:07.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.746+0000 7fdefba7e700 1 -- 172.21.15.158:0/2581993908 shutdown_connections 2023-10-16T09:45:07.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.746+0000 7fdefba7e700 1 -- 172.21.15.158:0/2581993908 wait complete. 2023-10-16T09:45:07.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.747+0000 7fdefba7e700 1 Processor -- start 2023-10-16T09:45:07.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.747+0000 7fdefba7e700 1 -- start start 2023-10-16T09:45:07.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefba7e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef406afe0 0x7fdef41583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefba7e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdef40cc590 0x7fdef40c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefba7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdef415da40 0x7fdef4156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefba7e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40c8de0 con 0x7fdef406afe0 2023-10-16T09:45:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefba7e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40c9080 con 0x7fdef40cc590 2023-10-16T09:45:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefba7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40c9360 con 0x7fdef415da40 2023-10-16T09:45:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef406afe0 0x7fdef41583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef406afe0 0x7fdef41583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49238/0 (socket says 172.21.15.158:49238) 2023-10-16T09:45:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 -- 172.21.15.158:0/2633504409 learned_addr learned my addr 172.21.15.158:0/2633504409 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef9019700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdef40cc590 0x7fdef40c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefa01b700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdef415da40 0x7fdef4156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 -- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdef415da40 msgr2=0x7fdef4156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdef415da40 0x7fdef4156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 -- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdef40cc590 msgr2=0x7fdef40c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdef40cc590 0x7fdef40c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 -- 172.21.15.158:0/2633504409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec009b30 con 0x7fdef406afe0 2023-10-16T09:45:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdef981a700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef406afe0 0x7fdef41583e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdee400b470 tx=0x7fdee400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdeea7fc700 1 -- 172.21.15.158:0/2633504409 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4011830 con 0x7fdef406afe0 2023-10-16T09:45:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdeea7fc700 1 -- 172.21.15.158:0/2633504409 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee4011990 con 0x7fdef406afe0 2023-10-16T09:45:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdeea7fc700 1 -- 172.21.15.158:0/2633504409 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4011d40 con 0x7fdef406afe0 2023-10-16T09:45:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.748+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef40c9640 con 0x7fdef406afe0 2023-10-16T09:45:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.749+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef40c9c00 con 0x7fdef406afe0 2023-10-16T09:45:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.749+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef40681b0 con 0x7fdef406afe0 2023-10-16T09:45:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.750+0000 7fdeea7fc700 1 -- 172.21.15.158:0/2633504409 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fdee401a400 con 0x7fdef406afe0 2023-10-16T09:45:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.750+0000 7fdeea7fc700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fded806b100 0x7fded806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.750+0000 7fdeea7fc700 1 -- 172.21.15.158:0/2633504409 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fdee4089bb0 con 0x7fdef406afe0 2023-10-16T09:45:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.750+0000 7fdef9019700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fded806b100 0x7fded806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.751+0000 7fdef9019700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fded806b100 0x7fded806d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdef41577e0 tx=0x7fdef0009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:07.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.754+0000 7fdeea7fc700 1 -- 172.21.15.158:0/2633504409 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdee40547a0 con 0x7fdef406afe0 2023-10-16T09:45:07.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:07.978+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef40cb400 con 0x7fded806b100 2023-10-16T09:45:08.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.038+0000 7fdeea7fc700 1 -- 172.21.15.158:0/2633504409 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fdef0001340 con 0x7fded806b100 2023-10-16T09:45:08.040 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-10-16T09:45:08.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fded806b100 msgr2=0x7fded806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fded806b100 0x7fded806d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdef41577e0 tx=0x7fdef0009040 comp rx=0 tx=0).stop 2023-10-16T09:45:08.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef406afe0 msgr2=0x7fdef41583e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef406afe0 0x7fdef41583e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdee400b470 tx=0x7fdee400b830 comp rx=0 tx=0).stop 2023-10-16T09:45:08.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 shutdown_connections 2023-10-16T09:45:08.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdef415da40 0x7fdef4156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fded806b100 0x7fded806d5c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef406afe0 0x7fdef41583e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 --2- 172.21.15.158:0/2633504409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdef40cc590 0x7fdef40c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 >> 172.21.15.158:0/2633504409 conn(0x7fdef4099a10 msgr2=0x7fdef415a720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:08.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 shutdown_connections 2023-10-16T09:45:08.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.040+0000 7fdefba7e700 1 -- 172.21.15.158:0/2633504409 wait complete. 2023-10-16T09:45:08.054 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-10-16T09:45:08.054 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2023-10-16T09:45:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.177+0000 7f54225b0700 1 Processor -- start 2023-10-16T09:45:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.177+0000 7f54225b0700 1 -- start start 2023-10-16T09:45:08.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f54225b0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f541c15dad0 0x7f541c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f54225b0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f54225b0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0d0ae0 con 0x7f541c0cea20 2023-10-16T09:45:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f54225b0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c0d12d0 con 0x7f541c15dad0 2023-10-16T09:45:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f54225b0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0cbdf0 con 0x7f541c15deb0 2023-10-16T09:45:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f541c15dad0 0x7f541c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f541c15dad0 0x7f541c0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49246/0 (socket says 172.21.15.158:49246) 2023-10-16T09:45:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 -- 172.21.15.158:0/3271366989 learned_addr learned my addr 172.21.15.158:0/3271366989 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541b7fe700 1 --2- 172.21.15.158:0/3271366989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 -- 172.21.15.158:0/3271366989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 msgr2=0x7f541c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 --2- 172.21.15.158:0/3271366989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 -- 172.21.15.158:0/3271366989 >> v1:172.21.15.158:6789/0 conn(0x7f541c0cea20 legacy=0x7f541c0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 -- 172.21.15.158:0/3271366989 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0cc280 con 0x7f541c15dad0 2023-10-16T09:45:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541bfff700 1 --2- 172.21.15.158:0/3271366989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0cdfa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f540c0098f0 tx=0x7f540c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9433280704478a95 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541affd700 1 -- 172.21.15.158:0/3271366989 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c00b8a0 con 0x7f541c15dad0 2023-10-16T09:45:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541affd700 1 -- 172.21.15.158:0/3271366989 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540c00ba00 con 0x7f541c15dad0 2023-10-16T09:45:08.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.178+0000 7f541affd700 1 -- 172.21.15.158:0/3271366989 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c00bd90 con 0x7f541c15dad0 2023-10-16T09:45:08.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 -- 172.21.15.158:0/3271366989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 msgr2=0x7f541c0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 --2- 172.21.15.158:0/3271366989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0cdfa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f540c0098f0 tx=0x7f540c009c00 comp rx=0 tx=0).stop 2023-10-16T09:45:08.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 -- 172.21.15.158:0/3271366989 shutdown_connections 2023-10-16T09:45:08.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 --2- 172.21.15.158:0/3271366989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0cdfa0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 --2- 172.21.15.158:0/3271366989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 -- 172.21.15.158:0/3271366989 >> 172.21.15.158:0/3271366989 conn(0x7f541c099a10 msgr2=0x7f541c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:08.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 -- 172.21.15.158:0/3271366989 shutdown_connections 2023-10-16T09:45:08.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.179+0000 7f54225b0700 1 -- 172.21.15.158:0/3271366989 wait complete. 2023-10-16T09:45:08.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.180+0000 7f54225b0700 1 Processor -- start 2023-10-16T09:45:08.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.180+0000 7f54225b0700 1 -- start start 2023-10-16T09:45:08.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f54225b0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c0cea20 0x7f541c152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f54225b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f54225b0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f54225b0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b3060 con 0x7f541c15dad0 2023-10-16T09:45:08.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f54225b0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b4e90 con 0x7f541c0cea20 2023-10-16T09:45:08.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f54225b0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0cd3d0 con 0x7f541c15deb0 2023-10-16T09:45:08.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f541b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f541b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49252/0 (socket says 172.21.15.158:49252) 2023-10-16T09:45:08.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f541b7fe700 1 -- 172.21.15.158:0/1927773151 learned_addr learned my addr 172.21.15.158:0/1927773151 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:08.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f541bfff700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c0cea20 0x7f541c152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.181+0000 7f5420b4d700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f541b7fe700 1 -- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 msgr2=0x7f541c0cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f541b7fe700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f541b7fe700 1 -- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c0cea20 msgr2=0x7f541c152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f541b7fe700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c0cea20 0x7f541c152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f541b7fe700 1 -- 172.21.15.158:0/1927773151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c009550 con 0x7f541c15dad0 2023-10-16T09:45:08.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f5420b4d700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:08.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f541b7fe700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0c8850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f541000d800 tx=0x7f541000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:08.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f5418ff9700 1 -- 172.21.15.158:0/1927773151 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54100125f0 con 0x7f541c15dad0 2023-10-16T09:45:08.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f5418ff9700 1 -- 172.21.15.158:0/1927773151 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410012750 con 0x7f541c15dad0 2023-10-16T09:45:08.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c0cd6b0 con 0x7f541c15dad0 2023-10-16T09:45:08.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f5418ff9700 1 -- 172.21.15.158:0/1927773151 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410012b00 con 0x7f541c15dad0 2023-10-16T09:45:08.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.182+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c0cd990 con 0x7f541c15dad0 2023-10-16T09:45:08.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.183+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c0681b0 con 0x7f541c15dad0 2023-10-16T09:45:08.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.183+0000 7f5418ff9700 1 -- 172.21.15.158:0/1927773151 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f5410012c60 con 0x7f541c15dad0 2023-10-16T09:45:08.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.184+0000 7f5418ff9700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f540806b100 0x7f540806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.184+0000 7f5418ff9700 1 -- 172.21.15.158:0/1927773151 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f5410089e10 con 0x7f541c15dad0 2023-10-16T09:45:08.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.184+0000 7f541bfff700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f540806b100 0x7f540806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.184+0000 7f541bfff700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f540806b100 0x7f540806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f540c009c40 tx=0x7f540c008030 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:08.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.188+0000 7f5418ff9700 1 -- 172.21.15.158:0/1927773151 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5410054a90 con 0x7f541c15dad0 2023-10-16T09:45:08.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.409+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f541c0d0ff0 con 0x7f540806b100 2023-10-16T09:45:08.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.457+0000 7f5418ff9700 1 -- 172.21.15.158:0/1927773151 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f540c00bb70 con 0x7f540806b100 2023-10-16T09:45:08.459 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-10-16T09:45:08.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f540806b100 msgr2=0x7f540806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f540806b100 0x7f540806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f540c009c40 tx=0x7f540c008030 comp rx=0 tx=0).stop 2023-10-16T09:45:08.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 msgr2=0x7f541c0c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0c8850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f541000d800 tx=0x7f541000db10 comp rx=0 tx=0).stop 2023-10-16T09:45:08.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 shutdown_connections 2023-10-16T09:45:08.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f541c15deb0 0x7f541c0cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c0cea20 0x7f541c152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f540806b100 0x7f540806d5c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 --2- 172.21.15.158:0/1927773151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f541c15dad0 0x7f541c0c8850 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 >> 172.21.15.158:0/1927773151 conn(0x7f541c099a10 msgr2=0x7f541c1e4850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:08.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 shutdown_connections 2023-10-16T09:45:08.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.459+0000 7f54225b0700 1 -- 172.21.15.158:0/1927773151 wait complete. 2023-10-16T09:45:08.473 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-10-16T09:45:08.473 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2023-10-16T09:45:08.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.595+0000 7f1ca4a2a700 1 Processor -- start 2023-10-16T09:45:08.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.595+0000 7f1ca4a2a700 1 -- start start 2023-10-16T09:45:08.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.596+0000 7f1ca4a2a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1ca00cf890 0x7f1ca00b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.596+0000 7f1ca4a2a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca00cac90 con 0x7f1ca00ce3e0 2023-10-16T09:45:08.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.596+0000 7f1ca4a2a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca00cd3d0 con 0x7f1ca00ca8b0 2023-10-16T09:45:08.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.596+0000 7f1ca4a2a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca015e560 con 0x7f1ca00cf890 2023-10-16T09:45:08.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.596+0000 7f1c9dd9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1ca00ce3e0 0x7f1ca015de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49436/0 (socket says 172.21.15.158:49436) 2023-10-16T09:45:08.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.596+0000 7f1c9dd9b700 1 -- 172.21.15.158:0/3717552084 learned_addr learned my addr 172.21.15.158:0/3717552084 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:08.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9ed9d700 1 --2- 172.21.15.158:0/3717552084 >> v2:172.21.15.172:3300/0 conn(0x7f1ca00cf890 0x7f1ca00b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104462301 0 0) 0x7f1c90001110 con 0x7f1ca00ce3e0 2023-10-16T09:45:08.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c880035b0 con 0x7f1ca00ce3e0 2023-10-16T09:45:08.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668768011 0 0) 0x7f1ca00cd3d0 con 0x7f1ca00ca8b0 2023-10-16T09:45:08.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c90001110 con 0x7f1ca00ca8b0 2023-10-16T09:45:08.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3402767847 0 0) 0x7f1c880035b0 con 0x7f1ca00ce3e0 2023-10-16T09:45:08.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 >> v2:172.21.15.172:3300/0 conn(0x7f1ca00cf890 msgr2=0x7f1ca00b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:08.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 --2- 172.21.15.158:0/3717552084 >> v2:172.21.15.172:3300/0 conn(0x7f1ca00cf890 0x7f1ca00b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 >> v1:172.21.15.172:6789/0 conn(0x7f1ca00ca8b0 legacy=0x7f1ca0159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca00cdf50 con 0x7f1ca00ce3e0 2023-10-16T09:45:08.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1c90001800 con 0x7f1ca00ce3e0 2023-10-16T09:45:08.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 >> v1:172.21.15.158:6789/0 conn(0x7f1ca00ce3e0 legacy=0x7f1ca015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c88003be0 0x7f1c88003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c88004850 0x7f1c88004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88005210 0x7f1c8800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8800c0e0 con 0x7f1c88004850 2023-10-16T09:45:08.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8800c220 con 0x7f1c88005210 2023-10-16T09:45:08.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8800c3e0 con 0x7f1c88003be0 2023-10-16T09:45:08.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9dd9b700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c88004850 0x7f1c88004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9ed9d700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88005210 0x7f1c8800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9e59c700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c88003be0 0x7f1c88003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9dd9b700 1 -- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c88003be0 msgr2=0x7f1c88003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:08.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9dd9b700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c88003be0 0x7f1c88003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9dd9b700 1 -- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88005210 msgr2=0x7f1c8800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:08.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9dd9b700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88005210 0x7f1c8800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9ed9d700 1 -- 172.21.15.158:0/3717552084 reap_dead start 2023-10-16T09:45:08.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9dd9b700 1 -- 172.21.15.158:0/3717552084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c880035b0 con 0x7f1c88004850 2023-10-16T09:45:08.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9e59c700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c88003be0 0x7f1c88003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:45:08.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.597+0000 7f1c9dd9b700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c88004850 0x7f1c88004cd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000c110 tx=0x7f1c88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2299770f0849be20 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:08.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000cc70 con 0x7f1c88004850 2023-10-16T09:45:08.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9000cdf0 con 0x7f1c88004850 2023-10-16T09:45:08.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1c9d59a700 1 -- 172.21.15.158:0/3717552084 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c90005ba0 con 0x7f1c88004850 2023-10-16T09:45:08.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/3717552084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c88004850 msgr2=0x7f1c88004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c88004850 0x7f1c88004cd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000c110 tx=0x7f1c88005880 comp rx=0 tx=0).stop 2023-10-16T09:45:08.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/3717552084 shutdown_connections 2023-10-16T09:45:08.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/3717552084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c88004850 0x7f1c88004cd0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/3717552084 >> 172.21.15.158:0/3717552084 conn(0x7f1ca006af00 msgr2=0x7f1ca00cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:08.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/3717552084 shutdown_connections 2023-10-16T09:45:08.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.598+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/3717552084 wait complete. 2023-10-16T09:45:08.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 Processor -- start 2023-10-16T09:45:08.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 -- start start 2023-10-16T09:45:08.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88004850 0x7f1ca01e6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ca01e58b0 0x7f1ca0155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ca01f0a50 0x7f1ca01f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca01e6a80 con 0x7f1ca01f0a50 2023-10-16T09:45:08.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca01e7450 con 0x7f1c88004850 2023-10-16T09:45:08.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.600+0000 7f1ca4a2a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca01e5e70 con 0x7f1ca01e58b0 2023-10-16T09:45:08.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ca01f0a50 0x7f1ca01f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ca01f0a50 0x7f1ca01f0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49274/0 (socket says 172.21.15.158:49274) 2023-10-16T09:45:08.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 -- 172.21.15.158:0/1949935005 learned_addr learned my addr 172.21.15.158:0/1949935005 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:08.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9dd9b700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ca01e58b0 0x7f1ca0155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9e59c700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88004850 0x7f1ca01e6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 -- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ca01e58b0 msgr2=0x7f1ca0155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ca01e58b0 0x7f1ca0155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 -- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88004850 msgr2=0x7f1ca01e6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88004850 0x7f1ca01e6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 -- 172.21.15.158:0/1949935005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca01e6150 con 0x7f1ca01f0a50 2023-10-16T09:45:08.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c9ed9d700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ca01f0a50 0x7f1ca01f0ee0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00b18e0 tx=0x7f1c98011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:08.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c8effd700 1 -- 172.21.15.158:0/1949935005 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9800e040 con 0x7f1ca01f0a50 2023-10-16T09:45:08.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c8effd700 1 -- 172.21.15.158:0/1949935005 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9800f040 con 0x7f1ca01f0a50 2023-10-16T09:45:08.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca01e6430 con 0x7f1ca01f0a50 2023-10-16T09:45:08.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1c8effd700 1 -- 172.21.15.158:0/1949935005 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9800fe90 con 0x7f1ca01f0a50 2023-10-16T09:45:08.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.601+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca0156a90 con 0x7f1ca01f0a50 2023-10-16T09:45:08.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.602+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca00681b0 con 0x7f1ca01f0a50 2023-10-16T09:45:08.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.603+0000 7f1c8effd700 1 -- 172.21.15.158:0/1949935005 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f1c98015430 con 0x7f1ca01f0a50 2023-10-16T09:45:08.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.603+0000 7f1c8effd700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1c8006b100 0x7f1c8006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:08.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.603+0000 7f1c9e59c700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1c8006b100 0x7f1c8006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:08.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.603+0000 7f1c8effd700 1 -- 172.21.15.158:0/1949935005 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f1c9808a930 con 0x7f1ca01f0a50 2023-10-16T09:45:08.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.604+0000 7f1c9e59c700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1c8006b100 0x7f1c8006d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ca004fec0 tx=0x7f1c94009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:08.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.608+0000 7f1c8effd700 1 -- 172.21.15.158:0/1949935005 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c980554a0 con 0x7f1ca01f0a50 2023-10-16T09:45:08.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.829+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ca00cd900 con 0x7f1c8006b100 2023-10-16T09:45:08.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.877+0000 7f1c8effd700 1 -- 172.21.15.158:0/1949935005 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1c94005170 con 0x7f1c8006b100 2023-10-16T09:45:08.879 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-10-16T09:45:08.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1c8006b100 msgr2=0x7f1c8006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1c8006b100 0x7f1c8006d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ca004fec0 tx=0x7f1c94009040 comp rx=0 tx=0).stop 2023-10-16T09:45:08.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ca01f0a50 msgr2=0x7f1ca01f0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:08.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ca01f0a50 0x7f1ca01f0ee0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00b18e0 tx=0x7f1c98011f80 comp rx=0 tx=0).stop 2023-10-16T09:45:08.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 shutdown_connections 2023-10-16T09:45:08.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1c8006b100 0x7f1c8006d5c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ca01f0a50 0x7f1ca01f0ee0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c88004850 0x7f1ca01e6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 --2- 172.21.15.158:0/1949935005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ca01e58b0 0x7f1ca0155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:08.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 >> 172.21.15.158:0/1949935005 conn(0x7f1ca006af00 msgr2=0x7f1ca0070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:08.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 shutdown_connections 2023-10-16T09:45:08.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:08.879+0000 7f1ca4a2a700 1 -- 172.21.15.158:0/1949935005 wait complete. 2023-10-16T09:45:08.893 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-10-16T09:45:08.894 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2023-10-16T09:45:09.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.016+0000 7fcec39f9700 1 Processor -- start 2023-10-16T09:45:09.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.016+0000 7fcec39f9700 1 -- start start 2023-10-16T09:45:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fcec39f9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 0x7fcebc15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fcec39f9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0cf4c0 con 0x7fcebc0ce510 2023-10-16T09:45:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fcec39f9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc15f710 con 0x7fcebc15b590 2023-10-16T09:45:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fcec39f9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b49e0 con 0x7fcebc0cc200 2023-10-16T09:45:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fcec0f94700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcebc0ce510 0x7fcebc0cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49456/0 (socket says 172.21.15.158:49456) 2023-10-16T09:45:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fcec0f94700 1 -- 172.21.15.158:0/1807400982 learned_addr learned my addr 172.21.15.158:0/1807400982 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:45:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fcec1f96700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 0x7fcebc15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359877380 0 0) 0x7fcebc0cf4c0 con 0x7fcebc0ce510 2023-10-16T09:45:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea80035b0 con 0x7fcebc0ce510 2023-10-16T09:45:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1585479399 0 0) 0x7fcea80035b0 con 0x7fcebc0ce510 2023-10-16T09:45:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 msgr2=0x7fcebc15f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 0x7fcebc15f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 >> v1:172.21.15.172:6789/0 conn(0x7fcebc15b590 legacy=0x7fcebc0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc15f970 con 0x7fcebc0ce510 2023-10-16T09:45:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fceac001670 con 0x7fcebc0ce510 2023-10-16T09:45:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 >> v1:172.21.15.158:6789/0 conn(0x7fcebc0ce510 legacy=0x7fcebc0cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 0x7fcea8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea8003fb0 0x7fcea8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8004d90 0x7fcea8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b49e0 con 0x7fcea8003fb0 2023-10-16T09:45:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea800bba0 con 0x7fcea8004d90 2023-10-16T09:45:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.017+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea800bd60 con 0x7fcebc0cc200 2023-10-16T09:45:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec0f94700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea8003fb0 0x7fcea8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec1795700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 0x7fcea8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec1f96700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8004d90 0x7fcea8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec0f94700 1 -- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 msgr2=0x7fcea8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec0f94700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0cc200 0x7fcea8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec0f94700 1 -- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8004d90 msgr2=0x7fcea8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:45:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec0f94700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8004d90 0x7fcea8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec0f94700 1 -- 172.21.15.158:0/1807400982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea80035b0 con 0x7fcea8003fb0 2023-10-16T09:45:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fcec0f94700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea8003fb0 0x7fcea8004850 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fceac00c2a0 tx=0x7fcea8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6742731b5a1dc8f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac007c10 con 0x7fcea8003fb0 2023-10-16T09:45:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac00cdc0 con 0x7fcea8003fb0 2023-10-16T09:45:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.018+0000 7fceb3fff700 1 -- 172.21.15.158:0/1807400982 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac005ba0 con 0x7fcea8003fb0 2023-10-16T09:45:09.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.019+0000 7fcec39f9700 1 -- 172.21.15.158:0/1807400982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea8003fb0 msgr2=0x7fcea8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:09.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.019+0000 7fcec39f9700 1 --2- 172.21.15.158:0/1807400982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea8003fb0 0x7fcea8004850 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fceac00c2a0 tx=0x7fcea8005880 comp rx=0 tx=0).stop 2023-10-16T09:45:09.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.019+0000 7fcec1f96700 1 -- 172.21.15.158:0/1807400982 reap_dead start 2023-10-16T09:45:09.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.019+0000 7fcec39f9700 1 -- 172.21.15.158:0/1807400982 shutdown_connections 2023-10-16T09:45:09.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.019+0000 7fcec39f9700 1 -- 172.21.15.158:0/1807400982 >> 172.21.15.158:0/1807400982 conn(0x7fcebc06a8d0 msgr2=0x7fcebc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:09.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.019+0000 7fcec39f9700 1 -- 172.21.15.158:0/1807400982 shutdown_connections 2023-10-16T09:45:09.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.019+0000 7fcec39f9700 1 -- 172.21.15.158:0/1807400982 wait complete. 2023-10-16T09:45:09.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.020+0000 7fcec39f9700 1 Processor -- start 2023-10-16T09:45:09.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.020+0000 7fcec39f9700 1 -- start start 2023-10-16T09:45:09.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec39f9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8003fb0 0x7fcebc0c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec39f9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0b2290 0x7fcebc0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec39f9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcebc0b2710 0x7fcebc0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec39f9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0c6450 con 0x7fcebc0b2710 2023-10-16T09:45:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec39f9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0c8570 con 0x7fcea8003fb0 2023-10-16T09:45:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec39f9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc15dd70 con 0x7fcebc0b2290 2023-10-16T09:45:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec1f96700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcebc0b2710 0x7fcebc0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec1f96700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcebc0b2710 0x7fcebc0cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49294/0 (socket says 172.21.15.158:49294) 2023-10-16T09:45:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.021+0000 7fcec1f96700 1 -- 172.21.15.158:0/2159616123 learned_addr learned my addr 172.21.15.158:0/2159616123 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:45:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec0f94700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0b2290 0x7fcebc0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec1795700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8003fb0 0x7fcebc0c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec1f96700 1 -- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0b2290 msgr2=0x7fcebc0ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec1f96700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0b2290 0x7fcebc0ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec1f96700 1 -- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8003fb0 msgr2=0x7fcebc0c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec1f96700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8003fb0 0x7fcebc0c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec1f96700 1 -- 172.21.15.158:0/2159616123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0cb350 con 0x7fcebc0b2710 2023-10-16T09:45:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec1f96700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcebc0b2710 0x7fcebc0cad20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0cee80 tx=0x7fceb4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fceb1ffb700 1 -- 172.21.15.158:0/2159616123 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb400e040 con 0x7fcebc0b2710 2023-10-16T09:45:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fceb1ffb700 1 -- 172.21.15.158:0/2159616123 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb400f040 con 0x7fcebc0b2710 2023-10-16T09:45:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc0c8810 con 0x7fcebc0b2710 2023-10-16T09:45:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fceb1ffb700 1 -- 172.21.15.158:0/2159616123 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb400fd00 con 0x7fcebc0b2710 2023-10-16T09:45:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.022+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc0c8dd0 con 0x7fcebc0b2710 2023-10-16T09:45:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.023+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc0c8ff0 con 0x7fcebc0b2710 2023-10-16T09:45:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.024+0000 7fceb1ffb700 1 -- 172.21.15.158:0/2159616123 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 11) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fceb4022460 con 0x7fcebc0b2710 2023-10-16T09:45:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.024+0000 7fceb1ffb700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcea006b100 0x7fcea006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:45:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.024+0000 7fceb1ffb700 1 -- 172.21.15.158:0/2159616123 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fceb408a1a0 con 0x7fcebc0b2710 2023-10-16T09:45:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.024+0000 7fcec1795700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcea006b100 0x7fcea006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:45:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.024+0000 7fcec1795700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcea006b100 0x7fcea006d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fceb8007fd0 tx=0x7fceb8009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:45:09.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.029+0000 7fceb1ffb700 1 -- 172.21.15.158:0/2159616123 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fceb4054c80 con 0x7fcebc0b2710 2023-10-16T09:45:09.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.253+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcebc15ff30 con 0x7fcea006b100 2023-10-16T09:45:09.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.318+0000 7fceb1ffb700 1 -- 172.21.15.158:0/2159616123 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fceb8005350 con 0x7fcea006b100 2023-10-16T09:45:09.321 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-10-16T09:45:09.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcea006b100 msgr2=0x7fcea006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:09.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcea006b100 0x7fcea006d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fceb8007fd0 tx=0x7fceb8009040 comp rx=0 tx=0).stop 2023-10-16T09:45:09.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcebc0b2710 msgr2=0x7fcebc0cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:45:09.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcebc0b2710 0x7fcebc0cad20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0cee80 tx=0x7fceb4011ed0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 shutdown_connections 2023-10-16T09:45:09.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcea006b100 0x7fcea006d5c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcebc0b2710 0x7fcebc0cad20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcea8003fb0 0x7fcebc0c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 --2- 172.21.15.158:0/2159616123 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcebc0b2290 0x7fcebc0ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:45:09.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 >> 172.21.15.158:0/2159616123 conn(0x7fcebc06a8d0 msgr2=0x7fcebc0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:45:09.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 shutdown_connections 2023-10-16T09:45:09.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:45:09.321+0000 7fcec39f9700 1 -- 172.21.15.158:0/2159616123 wait complete. 2023-10-16T09:45:09.334 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-10-16T09:45:24.746 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:45:24.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:45:54.822 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:45:54.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:46:00.050 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:46:00.048+0000 7f5d57d6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:46:00.050 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:46:00.048+0000 7f5d57d6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:46:00.050 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:46:00.048+0000 7f5d57d6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:46:00.051 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:46:00.048+0000 7f5d57d6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:46:00.051 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:46:00.048+0000 7f5d57d6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:46:24.897 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:46:24.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:46:54.964 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:46:54.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:47:25.232 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:47:25.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:47:55.320 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:47:55.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:48:25.390 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:48:25.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:48:55.457 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:48:55.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:49:25.536 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:49:25.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:49:55.660 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:49:55.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:50:00.654 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:50:00.652+0000 7f5d50d60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:50:00.654 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:50:00.652+0000 7f5d50d60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:50:00.654 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:50:00.652+0000 7f5d50d60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:50:00.655 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:50:00.652+0000 7f5d50d60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:50:00.655 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:50:00.652+0000 7f5d50d60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:50:25.743 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:50:25.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:50:55.816 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:50:55.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:51:26.128 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:51:26.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:51:56.191 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:51:56.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:52:26.295 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:52:26.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:52:56.385 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:52:56.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:53:26.456 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:53:26.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:53:56.527 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:53:56.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:54:26.815 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:54:26.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:54:56.887 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:54:56.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:55:27.223 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:55:27.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:55:57.287 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:55:57.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:56:19.336 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2023-10-16T09:56:19.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.471+0000 7fd63723c700 1 Processor -- start 2023-10-16T09:56:19.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.472+0000 7fd63723c700 1 -- start start 2023-10-16T09:56:19.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.472+0000 7fd63723c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6301516c0 0x7fd6300cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.472+0000 7fd63723c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6300cc730 con 0x7fd63006e030 2023-10-16T09:56:19.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.472+0000 7fd63723c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd630151aa0 con 0x7fd630072370 2023-10-16T09:56:19.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.472+0000 7fd63723c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6300cbb60 con 0x7fd6301516c0 2023-10-16T09:56:19.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd634fd8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6301516c0 0x7fd6300cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd6357d9700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd63006e030 0x7fd6300ccbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49504/0 (socket says 172.21.15.158:49504) 2023-10-16T09:56:19.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd6357d9700 1 -- 172.21.15.158:0/3238194428 learned_addr learned my addr 172.21.15.158:0/3238194428 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:19.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd634fd8700 1 -- 172.21.15.158:0/3238194428 >> v1:172.21.15.172:6789/0 conn(0x7fd630072370 legacy=0x7fd63006d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:19.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd634fd8700 1 -- 172.21.15.158:0/3238194428 >> v1:172.21.15.158:6789/0 conn(0x7fd63006e030 legacy=0x7fd6300ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd62f7fe700 1 -- 172.21.15.158:0/3238194428 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316228012 0 0) 0x7fd6300cc730 con 0x7fd63006e030 2023-10-16T09:56:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd634fd8700 1 -- 172.21.15.158:0/3238194428 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6300cbe00 con 0x7fd6301516c0 2023-10-16T09:56:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd634fd8700 1 --2- 172.21.15.158:0/3238194428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6301516c0 0x7fd6300cdfa0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd624009840 tx=0x7fd624009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb768e2dbb12dd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd62f7fe700 1 -- 172.21.15.158:0/3238194428 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624004230 con 0x7fd6301516c0 2023-10-16T09:56:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd62f7fe700 1 -- 172.21.15.158:0/3238194428 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd624004390 con 0x7fd6301516c0 2023-10-16T09:56:19.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.473+0000 7fd62f7fe700 1 -- 172.21.15.158:0/3238194428 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6240036a0 con 0x7fd6301516c0 2023-10-16T09:56:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.474+0000 7fd63723c700 1 -- 172.21.15.158:0/3238194428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6301516c0 msgr2=0x7fd6300cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.474+0000 7fd63723c700 1 --2- 172.21.15.158:0/3238194428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6301516c0 0x7fd6300cdfa0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd624009840 tx=0x7fd624009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.474+0000 7fd63723c700 1 -- 172.21.15.158:0/3238194428 shutdown_connections 2023-10-16T09:56:19.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.474+0000 7fd63723c700 1 --2- 172.21.15.158:0/3238194428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6301516c0 0x7fd6300cdfa0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.474+0000 7fd63723c700 1 -- 172.21.15.158:0/3238194428 >> 172.21.15.158:0/3238194428 conn(0x7fd6300b1f00 msgr2=0x7fd6300b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:19.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.474+0000 7fd63723c700 1 -- 172.21.15.158:0/3238194428 shutdown_connections 2023-10-16T09:56:19.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.474+0000 7fd63723c700 1 -- 172.21.15.158:0/3238194428 wait complete. 2023-10-16T09:56:19.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.475+0000 7fd63723c700 1 Processor -- start 2023-10-16T09:56:19.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.475+0000 7fd63723c700 1 -- start start 2023-10-16T09:56:19.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd63723c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd63006e030 0x7fd63015b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd63723c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd630072370 0x7fd6301f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd63723c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6301516c0 0x7fd63015d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd63723c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300c5fd0 con 0x7fd63006e030 2023-10-16T09:56:19.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd63723c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300c6750 con 0x7fd630072370 2023-10-16T09:56:19.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd63723c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63015bd90 con 0x7fd6301516c0 2023-10-16T09:56:19.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd63006e030 0x7fd63015b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd63006e030 0x7fd63015b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49330/0 (socket says 172.21.15.158:49330) 2023-10-16T09:56:19.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 -- 172.21.15.158:0/1739169862 learned_addr learned my addr 172.21.15.158:0/1739169862 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:19.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd62ffff700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd630072370 0x7fd6301f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd6357d9700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6301516c0 0x7fd63015d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 -- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6301516c0 msgr2=0x7fd63015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6301516c0 0x7fd63015d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 -- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd630072370 msgr2=0x7fd6301f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd630072370 0x7fd6301f3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.476+0000 7fd634fd8700 1 -- 172.21.15.158:0/1739169862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6240094a0 con 0x7fd63006e030 2023-10-16T09:56:19.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.477+0000 7fd634fd8700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd63006e030 0x7fd63015b850 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd624009b90 tx=0x7fd624004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:19.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.477+0000 7fd62d7fa700 1 -- 172.21.15.158:0/1739169862 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624011630 con 0x7fd63006e030 2023-10-16T09:56:19.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.477+0000 7fd62d7fa700 1 -- 172.21.15.158:0/1739169862 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62401e070 con 0x7fd63006e030 2023-10-16T09:56:19.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.477+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6301f3a90 con 0x7fd63006e030 2023-10-16T09:56:19.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.477+0000 7fd62d7fa700 1 -- 172.21.15.158:0/1739169862 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6240119b0 con 0x7fd63006e030 2023-10-16T09:56:19.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.477+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6300c5730 con 0x7fd63006e030 2023-10-16T09:56:19.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.478+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6300c8cf0 con 0x7fd63006e030 2023-10-16T09:56:19.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.478+0000 7fd62d7fa700 1 -- 172.21.15.158:0/1739169862 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd624011b10 con 0x7fd63006e030 2023-10-16T09:56:19.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.479+0000 7fd62d7fa700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd61406b130 0x7fd61406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.479+0000 7fd62ffff700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd61406b130 0x7fd61406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.479+0000 7fd62d7fa700 1 -- 172.21.15.158:0/1739169862 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd62408af90 con 0x7fd63006e030 2023-10-16T09:56:19.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.479+0000 7fd62ffff700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd61406b130 0x7fd61406d5f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009870 tx=0x7fd61c008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:19.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.484+0000 7fd62d7fa700 1 -- 172.21.15.158:0/1739169862 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6240bdfe0 con 0x7fd63006e030 2023-10-16T09:56:19.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.706+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6300cc0e0 con 0x7fd61406b130 2023-10-16T09:56:19.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.775+0000 7fd62d7fa700 1 -- 172.21.15.158:0/1739169862 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd61c009460 con 0x7fd61406b130 2023-10-16T09:56:19.777 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T09:56:19.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd61406b130 msgr2=0x7fd61406d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd61406b130 0x7fd61406d5f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009870 tx=0x7fd61c008040 comp rx=0 tx=0).stop 2023-10-16T09:56:19.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd63006e030 msgr2=0x7fd63015b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd63006e030 0x7fd63015b850 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd624009b90 tx=0x7fd624004750 comp rx=0 tx=0).stop 2023-10-16T09:56:19.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 shutdown_connections 2023-10-16T09:56:19.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6301516c0 0x7fd63015d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd61406b130 0x7fd61406d5f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd63006e030 0x7fd63015b850 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 --2- 172.21.15.158:0/1739169862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd630072370 0x7fd6301f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 >> 172.21.15.158:0/1739169862 conn(0x7fd6300b1f00 msgr2=0x7fd63006d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:19.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 shutdown_connections 2023-10-16T09:56:19.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.777+0000 7fd63723c700 1 -- 172.21.15.158:0/1739169862 wait complete. 2023-10-16T09:56:19.790 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T09:56:19.791 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2023-10-16T09:56:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.913+0000 7f2f90ea0700 1 Processor -- start 2023-10-16T09:56:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.914+0000 7f2f90ea0700 1 -- start start 2023-10-16T09:56:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.914+0000 7f2f90ea0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2f8c15da40 0x7f2f8c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.914+0000 7f2f90ea0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f8c06afe0 0x7f2f8c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.914+0000 7f2f90ea0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0cbd30 con 0x7f2f8c0cc590 2023-10-16T09:56:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.914+0000 7f2f90ea0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8c0cc1c0 con 0x7f2f8c15da40 2023-10-16T09:56:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.914+0000 7f2f90ea0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8c15de20 con 0x7f2f8c06afe0 2023-10-16T09:56:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2f8c15da40 0x7f2f8c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2f8c15da40 0x7f2f8c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49338/0 (socket says 172.21.15.158:49338) 2023-10-16T09:56:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 -- 172.21.15.158:0/1879688899 learned_addr learned my addr 172.21.15.158:0/1879688899 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8ad9d700 1 --2- 172.21.15.158:0/1879688899 >> v2:172.21.15.172:3300/0 conn(0x7f2f8c06afe0 0x7f2f8c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 -- 172.21.15.158:0/1879688899 >> v2:172.21.15.172:3300/0 conn(0x7f2f8c06afe0 msgr2=0x7f2f8c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 --2- 172.21.15.158:0/1879688899 >> v2:172.21.15.172:3300/0 conn(0x7f2f8c06afe0 0x7f2f8c0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 -- 172.21.15.158:0/1879688899 >> v1:172.21.15.172:6789/0 conn(0x7f2f8c0cc590 legacy=0x7f2f8c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 -- 172.21.15.158:0/1879688899 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c0ce260 con 0x7f2f8c15da40 2023-10-16T09:56:19.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8ad9d700 1 --2- 172.21.15.158:0/1879688899 >> v2:172.21.15.172:3300/0 conn(0x7f2f8c06afe0 0x7f2f8c0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:19.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8a59c700 1 --2- 172.21.15.158:0/1879688899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0caa20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0729e0 tx=0x7f2f80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2571f0b0215547c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:19.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8959a700 1 -- 172.21.15.158:0/1879688899 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000b870 con 0x7f2f8c15da40 2023-10-16T09:56:19.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8959a700 1 -- 172.21.15.158:0/1879688899 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8000b9d0 con 0x7f2f8c15da40 2023-10-16T09:56:19.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.915+0000 7f2f8959a700 1 -- 172.21.15.158:0/1879688899 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000bd60 con 0x7f2f8c15da40 2023-10-16T09:56:19.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1879688899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 msgr2=0x7f2f8c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1879688899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0caa20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0729e0 tx=0x7f2f80009bd0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1879688899 shutdown_connections 2023-10-16T09:56:19.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1879688899 >> v2:172.21.15.172:3300/0 conn(0x7f2f8c06afe0 0x7f2f8c0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1879688899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0caa20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1879688899 >> 172.21.15.158:0/1879688899 conn(0x7f2f8c099a10 msgr2=0x7f2f8c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:19.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1879688899 shutdown_connections 2023-10-16T09:56:19.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.916+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1879688899 wait complete. 2023-10-16T09:56:19.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.918+0000 7f2f90ea0700 1 Processor -- start 2023-10-16T09:56:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.918+0000 7f2f90ea0700 1 -- start start 2023-10-16T09:56:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f8c06afe0 0x7f2f8c148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f8c0cc590 0x7f2f8c0c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0d0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c1e5080 con 0x7f2f8c15da40 2023-10-16T09:56:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c0c9bc0 con 0x7f2f8c0cc590 2023-10-16T09:56:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c0c9ea0 con 0x7f2f8c06afe0 2023-10-16T09:56:19.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0d0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49348/0 (socket says 172.21.15.158:49348) 2023-10-16T09:56:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 -- 172.21.15.158:0/1908052763 learned_addr learned my addr 172.21.15.158:0/1908052763 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8a59c700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f8c06afe0 0x7f2f8c148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f89d9b700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f8c0cc590 0x7f2f8c0c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 -- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f8c06afe0 msgr2=0x7f2f8c148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f8c06afe0 0x7f2f8c148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 -- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f8c0cc590 msgr2=0x7f2f8c0c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f8c0cc590 0x7f2f8c0c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:19.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 -- 172.21.15.158:0/1908052763 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f80009550 con 0x7f2f8c15da40 2023-10-16T09:56:19.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f89d9b700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f8c0cc590 0x7f2f8c0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:19.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f8ad9d700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0d0220 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2f84007e70 tx=0x7f2f8400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:19.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f7affd700 1 -- 172.21.15.158:0/1908052763 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8400de40 con 0x7f2f8c15da40 2023-10-16T09:56:19.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f7affd700 1 -- 172.21.15.158:0/1908052763 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8400f040 con 0x7f2f8c15da40 2023-10-16T09:56:19.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c0ca180 con 0x7f2f8c15da40 2023-10-16T09:56:19.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f7affd700 1 -- 172.21.15.158:0/1908052763 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84015640 con 0x7f2f8c15da40 2023-10-16T09:56:19.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.919+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c0c9740 con 0x7f2f8c15da40 2023-10-16T09:56:19.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.920+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c1e8360 con 0x7f2f8c15da40 2023-10-16T09:56:19.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.921+0000 7f2f7affd700 1 -- 172.21.15.158:0/1908052763 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f2f8401e3b0 con 0x7f2f8c15da40 2023-10-16T09:56:19.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.921+0000 7f2f7affd700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2f6c06b100 0x7f2f6c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:19.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.921+0000 7f2f7affd700 1 -- 172.21.15.158:0/1908052763 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f2f84089e70 con 0x7f2f8c15da40 2023-10-16T09:56:19.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.921+0000 7f2f8a59c700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2f6c06b100 0x7f2f6c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:19.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.922+0000 7f2f8a59c700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2f6c06b100 0x7f2f6c06d5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f80014040 tx=0x7f2f80008030 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:19.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:19.926+0000 7f2f7affd700 1 -- 172.21.15.158:0/1908052763 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2f840549c0 con 0x7f2f8c15da40 2023-10-16T09:56:20.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.147+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f8c0cb400 con 0x7f2f6c06b100 2023-10-16T09:56:20.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.220+0000 7f2f7affd700 1 -- 172.21.15.158:0/1908052763 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2f8000bb40 con 0x7f2f6c06b100 2023-10-16T09:56:20.222 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T09:56:20.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2f6c06b100 msgr2=0x7f2f6c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2f6c06b100 0x7f2f6c06d5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f80014040 tx=0x7f2f80008030 comp rx=0 tx=0).stop 2023-10-16T09:56:20.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 msgr2=0x7f2f8c0d0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0d0220 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2f84007e70 tx=0x7f2f8400d3b0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 shutdown_connections 2023-10-16T09:56:20.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2f6c06b100 0x7f2f6c06d5c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f8c15da40 0x7f2f8c0d0220 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f8c06afe0 0x7f2f8c148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 --2- 172.21.15.158:0/1908052763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f8c0cc590 0x7f2f8c0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 >> 172.21.15.158:0/1908052763 conn(0x7f2f8c099a10 msgr2=0x7f2f8c06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:20.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 shutdown_connections 2023-10-16T09:56:20.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.223+0000 7f2f90ea0700 1 -- 172.21.15.158:0/1908052763 wait complete. 2023-10-16T09:56:20.236 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T09:56:20.236 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2023-10-16T09:56:20.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.359+0000 7f8085880700 1 Processor -- start 2023-10-16T09:56:20.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.359+0000 7f8085880700 1 -- start start 2023-10-16T09:56:20.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.360+0000 7f8085880700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f808015a350 0x7f808015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.360+0000 7f8085880700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808006b230 con 0x7f808015a730 2023-10-16T09:56:20.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.360+0000 7f8085880700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808006a970 con 0x7f808006db00 2023-10-16T09:56:20.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.360+0000 7f8085880700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808015f000 con 0x7f808015a350 2023-10-16T09:56:20.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807e7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f808015a730 0x7f808006d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49528/0 (socket says 172.21.15.158:49528) 2023-10-16T09:56:20.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807e7fc700 1 -- 172.21.15.158:0/1672672352 learned_addr learned my addr 172.21.15.158:0/1672672352 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:20.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807effd700 1 --2- 172.21.15.158:0/1672672352 >> v2:172.21.15.172:3300/0 conn(0x7f808015a350 0x7f808015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500703185 0 0) 0x7f808006b230 con 0x7f808015a730 2023-10-16T09:56:20.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80680035b0 con 0x7f808015a730 2023-10-16T09:56:20.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380531916 0 0) 0x7f80680035b0 con 0x7f808015a730 2023-10-16T09:56:20.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 >> v2:172.21.15.172:3300/0 conn(0x7f808015a350 msgr2=0x7f808015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:20.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 --2- 172.21.15.158:0/1672672352 >> v2:172.21.15.172:3300/0 conn(0x7f808015a350 0x7f808015eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 >> v1:172.21.15.172:6789/0 conn(0x7f808006db00 legacy=0x7f80800b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808006abe0 con 0x7f808015a730 2023-10-16T09:56:20.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f8070001670 con 0x7f808015a730 2023-10-16T09:56:20.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 >> v1:172.21.15.158:6789/0 conn(0x7f808015a730 legacy=0x7f808006d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068003a00 0x7f8068003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8068004850 0x7f8068004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8068005210 0x7f806800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808015f000 con 0x7f8068005210 2023-10-16T09:56:20.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806800c160 con 0x7f8068004850 2023-10-16T09:56:20.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806800c320 con 0x7f8068003a00 2023-10-16T09:56:20.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807f7fe700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8068005210 0x7f806800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807e7fc700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8068004850 0x7f8068004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.361+0000 7f807effd700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068003a00 0x7f8068003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807f7fe700 1 -- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068003a00 msgr2=0x7f8068003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:20.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807f7fe700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068003a00 0x7f8068003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807f7fe700 1 -- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8068004850 msgr2=0x7f8068004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:20.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807f7fe700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8068004850 0x7f8068004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807f7fe700 1 -- 172.21.15.158:0/1672672352 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80680035b0 con 0x7f8068005210 2023-10-16T09:56:20.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807effd700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068003a00 0x7f8068003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:20.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807f7fe700 1 -- 172.21.15.158:0/1672672352 reap_dead start 2023-10-16T09:56:20.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807f7fe700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8068005210 0x7f806800ba10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f807800c190 tx=0x7f8068005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b600beaafde27ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:20.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078001db0 con 0x7f8068005210 2023-10-16T09:56:20.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807800ccf0 con 0x7f8068005210 2023-10-16T09:56:20.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f807dffb700 1 -- 172.21.15.158:0/1672672352 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078003c40 con 0x7f8068005210 2023-10-16T09:56:20.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f8085880700 1 -- 172.21.15.158:0/1672672352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8068005210 msgr2=0x7f806800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f8085880700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8068005210 0x7f806800ba10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f807800c190 tx=0x7f8068005880 comp rx=0 tx=0).stop 2023-10-16T09:56:20.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f8085880700 1 -- 172.21.15.158:0/1672672352 shutdown_connections 2023-10-16T09:56:20.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f8085880700 1 --2- 172.21.15.158:0/1672672352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8068005210 0x7f806800ba10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.362+0000 7f8085880700 1 -- 172.21.15.158:0/1672672352 >> 172.21.15.158:0/1672672352 conn(0x7f80800b1bd0 msgr2=0x7f80800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:20.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.363+0000 7f8085880700 1 -- 172.21.15.158:0/1672672352 shutdown_connections 2023-10-16T09:56:20.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.363+0000 7f8085880700 1 -- 172.21.15.158:0/1672672352 wait complete. 2023-10-16T09:56:20.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.364+0000 7f8085880700 1 Processor -- start 2023-10-16T09:56:20.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.364+0000 7f8085880700 1 -- start start 2023-10-16T09:56:20.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f8085880700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068005210 0x7f80801502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f8085880700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8080159510 0x7f808014e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f8085880700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f808014cdf0 0x7f8080150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f8085880700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808014c960 con 0x7f808014cdf0 2023-10-16T09:56:20.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f8085880700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80801599a0 con 0x7f8080159510 2023-10-16T09:56:20.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f8085880700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808014c310 con 0x7f8068005210 2023-10-16T09:56:20.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f807f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f808014cdf0 0x7f8080150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f807f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f808014cdf0 0x7f8080150dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49368/0 (socket says 172.21.15.158:49368) 2023-10-16T09:56:20.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.365+0000 7f807f7fe700 1 -- 172.21.15.158:0/4184314737 learned_addr learned my addr 172.21.15.158:0/4184314737 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:20.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807e7fc700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8080159510 0x7f808014e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807effd700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068005210 0x7f80801502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807f7fe700 1 -- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068005210 msgr2=0x7f80801502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807f7fe700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068005210 0x7f80801502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807f7fe700 1 -- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8080159510 msgr2=0x7f808014e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807f7fe700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8080159510 0x7f808014e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807f7fe700 1 -- 172.21.15.158:0/4184314737 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808014c5f0 con 0x7f808014cdf0 2023-10-16T09:56:20.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807effd700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068005210 0x7f80801502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:20.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807e7fc700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8080159510 0x7f808014e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:20.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f807f7fe700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f808014cdf0 0x7f8080150dc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f80800cfb30 tx=0x7f80780014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:20.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f8067fff700 1 -- 172.21.15.158:0/4184314737 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780050d0 con 0x7f808014cdf0 2023-10-16T09:56:20.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f8067fff700 1 -- 172.21.15.158:0/4184314737 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8078005230 con 0x7f808014cdf0 2023-10-16T09:56:20.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80801507e0 con 0x7f808014cdf0 2023-10-16T09:56:20.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f8067fff700 1 -- 172.21.15.158:0/4184314737 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780055a0 con 0x7f808014cdf0 2023-10-16T09:56:20.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.366+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8080151300 con 0x7f808014cdf0 2023-10-16T09:56:20.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.367+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808014c170 con 0x7f808014cdf0 2023-10-16T09:56:20.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.367+0000 7f8067fff700 1 -- 172.21.15.158:0/4184314737 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8078005700 con 0x7f808014cdf0 2023-10-16T09:56:20.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.368+0000 7f8067fff700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f806c06b100 0x7f806c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.368+0000 7f8067fff700 1 -- 172.21.15.158:0/4184314737 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f807808cb40 con 0x7f808014cdf0 2023-10-16T09:56:20.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.368+0000 7f807effd700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f806c06b100 0x7f806c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.368+0000 7f807effd700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f806c06b100 0x7f806c06d5c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f807400c850 tx=0x7f8074008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:20.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.372+0000 7f8067fff700 1 -- 172.21.15.158:0/4184314737 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f808014c170 con 0x7f808014cdf0 2023-10-16T09:56:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.598+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f808006eaf0 con 0x7f806c06b100 2023-10-16T09:56:20.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.671+0000 7f8067fff700 1 -- 172.21.15.158:0/4184314737 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f807400c040 con 0x7f806c06b100 2023-10-16T09:56:20.672 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T09:56:20.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f806c06b100 msgr2=0x7f806c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f806c06b100 0x7f806c06d5c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f807400c850 tx=0x7f8074008040 comp rx=0 tx=0).stop 2023-10-16T09:56:20.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f808014cdf0 msgr2=0x7f8080150dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f808014cdf0 0x7f8080150dc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f80800cfb30 tx=0x7f80780014a0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 shutdown_connections 2023-10-16T09:56:20.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f806c06b100 0x7f806c06d5c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f808014cdf0 0x7f8080150dc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8068005210 0x7f80801502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 --2- 172.21.15.158:0/4184314737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8080159510 0x7f808014e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 >> 172.21.15.158:0/4184314737 conn(0x7f80800b1bd0 msgr2=0x7f80800cd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:20.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 shutdown_connections 2023-10-16T09:56:20.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.673+0000 7f8085880700 1 -- 172.21.15.158:0/4184314737 wait complete. 2023-10-16T09:56:20.687 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T09:56:20.687 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2023-10-16T09:56:20.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.811+0000 7f96fd1cd700 1 Processor -- start 2023-10-16T09:56:20.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.811+0000 7f96fd1cd700 1 -- start start 2023-10-16T09:56:20.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.811+0000 7f96fd1cd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f96f8141920 0x7f96f806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.811+0000 7f96fd1cd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f8144840 0x7f96f80cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96fd1cd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f806c210 con 0x7f96f80cdc40 2023-10-16T09:56:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96fd1cd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f80ce0d0 con 0x7f96f8141920 2023-10-16T09:56:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96fd1cd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f8141d00 con 0x7f96f8144840 2023-10-16T09:56:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f759e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f96f80cdc40 0x7f96f815f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49548/0 (socket says 172.21.15.158:49548) 2023-10-16T09:56:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f759e700 1 -- 172.21.15.158:0/2421142870 learned_addr learned my addr 172.21.15.158:0/2421142870 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873088381 0 0) 0x7f96f806c210 con 0x7f96f80cdc40 2023-10-16T09:56:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96e00035b0 con 0x7f96f80cdc40 2023-10-16T09:56:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f659c700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f8144840 0x7f96f80cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/2421142870 >> v2:172.21.15.172:3300/0 conn(0x7f96f8141920 0x7f96f806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2495129848 0 0) 0x7f96e00035b0 con 0x7f96f80cdc40 2023-10-16T09:56:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f8144840 msgr2=0x7f96f80cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f8144840 0x7f96f80cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 >> v2:172.21.15.172:3300/0 conn(0x7f96f8141920 msgr2=0x7f96f806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 --2- 172.21.15.158:0/2421142870 >> v2:172.21.15.172:3300/0 conn(0x7f96f8141920 0x7f96f806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f806b970 con 0x7f96f80cdc40 2023-10-16T09:56:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f96f00019f0 con 0x7f96f80cdc40 2023-10-16T09:56:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 >> v1:172.21.15.158:6789/0 conn(0x7f96f80cdc40 legacy=0x7f96f815f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96e0004850 0x7f96e0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96e0005210 0x7f96e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f80ce0d0 con 0x7f96e0003a00 2023-10-16T09:56:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f8141d00 con 0x7f96e0004850 2023-10-16T09:56:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e000c0e0 con 0x7f96e0005210 2023-10-16T09:56:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f659c700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f8144840 0x7f96f80cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f759e700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96e0005210 0x7f96e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.812+0000 7f96f659c700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96e0004850 0x7f96e0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f6d9d700 1 -- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96e0005210 msgr2=0x7f96e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96e0005210 0x7f96e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f6d9d700 1 -- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96e0004850 msgr2=0x7f96e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96e0004850 0x7f96e0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f6d9d700 1 -- 172.21.15.158:0/2421142870 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e00035b0 con 0x7f96e0003a00 2023-10-16T09:56:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f759e700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96e0005210 0x7f96e000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96e0003e20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098c0 tx=0x7f96e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5a215b8721e1d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e800c070 con 0x7f96e0003a00 2023-10-16T09:56:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e8013610 con 0x7f96e0003a00 2023-10-16T09:56:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f5d9b700 1 -- 172.21.15.158:0/2421142870 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e80139a0 con 0x7f96e0003a00 2023-10-16T09:56:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96fd1cd700 1 -- 172.21.15.158:0/2421142870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 msgr2=0x7f96e0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96fd1cd700 1 --2- 172.21.15.158:0/2421142870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96e0003e20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098c0 tx=0x7f96e0005880 comp rx=0 tx=0).stop 2023-10-16T09:56:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96f759e700 1 -- 172.21.15.158:0/2421142870 reap_dead start 2023-10-16T09:56:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96fd1cd700 1 -- 172.21.15.158:0/2421142870 shutdown_connections 2023-10-16T09:56:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96fd1cd700 1 -- 172.21.15.158:0/2421142870 >> 172.21.15.158:0/2421142870 conn(0x7f96f80cc5b0 msgr2=0x7f96f80cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96fd1cd700 1 -- 172.21.15.158:0/2421142870 shutdown_connections 2023-10-16T09:56:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.813+0000 7f96fd1cd700 1 -- 172.21.15.158:0/2421142870 wait complete. 2023-10-16T09:56:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.815+0000 7f96fd1cd700 1 Processor -- start 2023-10-16T09:56:20.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.815+0000 7f96fd1cd700 1 -- start start 2023-10-16T09:56:20.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96f813ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96f813b160 0x7f96f81df5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f81dfb10 0x7f96f81e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f81dffa0 con 0x7f96e0003a00 2023-10-16T09:56:20.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f81e4930 con 0x7f96f813b160 2023-10-16T09:56:20.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f81e4c10 con 0x7f96f81dfb10 2023-10-16T09:56:20.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96f813ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96f813ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49382/0 (socket says 172.21.15.158:49382) 2023-10-16T09:56:20.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 -- 172.21.15.158:0/117894507 learned_addr learned my addr 172.21.15.158:0/117894507 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:20.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f759e700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f81dfb10 0x7f96f81e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f659c700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96f813b160 0x7f96f81df5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 -- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f81dfb10 msgr2=0x7f96f81e3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f81dfb10 0x7f96f81e3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 -- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96f813b160 msgr2=0x7f96f81df5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:20.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96f813b160 0x7f96f81df5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:20.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 -- 172.21.15.158:0/117894507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f81e4ef0 con 0x7f96e0003a00 2023-10-16T09:56:20.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96f6d9d700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96f813ac20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f96e8016040 tx=0x7f96e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:20.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96df7fe700 1 -- 172.21.15.158:0/117894507 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e801c630 con 0x7f96e0003a00 2023-10-16T09:56:20.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96df7fe700 1 -- 172.21.15.158:0/117894507 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e801c790 con 0x7f96e0003a00 2023-10-16T09:56:20.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f81e51d0 con 0x7f96e0003a00 2023-10-16T09:56:20.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96df7fe700 1 -- 172.21.15.158:0/117894507 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e801cb20 con 0x7f96e0003a00 2023-10-16T09:56:20.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.816+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f81e41b0 con 0x7f96e0003a00 2023-10-16T09:56:20.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.817+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f80681b0 con 0x7f96e0003a00 2023-10-16T09:56:20.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.818+0000 7f96df7fe700 1 -- 172.21.15.158:0/117894507 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f96e801cc80 con 0x7f96e0003a00 2023-10-16T09:56:20.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.819+0000 7f96df7fe700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f96e406b100 0x7f96e406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:20.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.819+0000 7f96df7fe700 1 -- 172.21.15.158:0/117894507 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f96e808ac90 con 0x7f96e0003a00 2023-10-16T09:56:20.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.819+0000 7f96f659c700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f96e406b100 0x7f96e406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:20.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.819+0000 7f96f659c700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f96e406b100 0x7f96e406d5c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f96f80b5d00 tx=0x7f96ec008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:20.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:20.822+0000 7f96df7fe700 1 -- 172.21.15.158:0/117894507 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96e80558c0 con 0x7f96e0003a00 2023-10-16T09:56:21.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.049+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f96f806bf30 con 0x7f96e406b100 2023-10-16T09:56:21.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.118+0000 7f96df7fe700 1 -- 172.21.15.158:0/117894507 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f96ec00e510 con 0x7f96e406b100 2023-10-16T09:56:21.120 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T09:56:21.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f96e406b100 msgr2=0x7f96e406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f96e406b100 0x7f96e406d5c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f96f80b5d00 tx=0x7f96ec008040 comp rx=0 tx=0).stop 2023-10-16T09:56:21.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 msgr2=0x7f96f813ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96f813ac20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f96e8016040 tx=0x7f96e8004750 comp rx=0 tx=0).stop 2023-10-16T09:56:21.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 shutdown_connections 2023-10-16T09:56:21.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96f81dfb10 0x7f96f81e3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f96e406b100 0x7f96e406d5c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e0003a00 0x7f96f813ac20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 --2- 172.21.15.158:0/117894507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96f813b160 0x7f96f81df5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 >> 172.21.15.158:0/117894507 conn(0x7f96f80cc5b0 msgr2=0x7f96f81428b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:21.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 shutdown_connections 2023-10-16T09:56:21.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.120+0000 7f96fd1cd700 1 -- 172.21.15.158:0/117894507 wait complete. 2023-10-16T09:56:21.134 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T09:56:21.134 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2023-10-16T09:56:21.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 Processor -- start 2023-10-16T09:56:21.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 -- start start 2023-10-16T09:56:21.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f616c15d840 0x7f616c15dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c0b38f0 0x7f616c0d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f616c0d2010 0x7f616c06bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c15ec80 con 0x7f616c15d840 2023-10-16T09:56:21.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0d13b0 con 0x7f616c0d2010 2023-10-16T09:56:21.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.257+0000 7f6171718700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c15e160 con 0x7f616c0b38f0 2023-10-16T09:56:21.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f616c15d840 0x7f616c15dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f616c15d840 0x7f616c15dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49390/0 (socket says 172.21.15.158:49390) 2023-10-16T09:56:21.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 -- 172.21.15.158:0/3228192828 learned_addr learned my addr 172.21.15.158:0/3228192828 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:21.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 -- 172.21.15.158:0/3228192828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c0b38f0 msgr2=0x7f616c0d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:21.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616a7fc700 1 --2- 172.21.15.158:0/3228192828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c0b38f0 0x7f616c0d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616b7fe700 1 --2- 172.21.15.158:0/3228192828 >> v2:172.21.15.172:3300/0 conn(0x7f616c0d2010 0x7f616c06bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 --2- 172.21.15.158:0/3228192828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c0b38f0 0x7f616c0d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 -- 172.21.15.158:0/3228192828 >> v2:172.21.15.172:3300/0 conn(0x7f616c0d2010 msgr2=0x7f616c06bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:21.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 --2- 172.21.15.158:0/3228192828 >> v2:172.21.15.172:3300/0 conn(0x7f616c0d2010 0x7f616c06bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 -- 172.21.15.158:0/3228192828 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c15e3e0 con 0x7f616c15d840 2023-10-16T09:56:21.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f616affd700 1 --2- 172.21.15.158:0/3228192828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c15d840 0x7f616c15dc20 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f616c0cea30 tx=0x7f615c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af08b1680b66df1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:21.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f6169ffb700 1 -- 172.21.15.158:0/3228192828 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c00b870 con 0x7f616c15d840 2023-10-16T09:56:21.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f6169ffb700 1 -- 172.21.15.158:0/3228192828 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c00b9d0 con 0x7f616c15d840 2023-10-16T09:56:21.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.258+0000 7f6169ffb700 1 -- 172.21.15.158:0/3228192828 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c00bd60 con 0x7f616c15d840 2023-10-16T09:56:21.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 -- 172.21.15.158:0/3228192828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c15d840 msgr2=0x7f616c15dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 --2- 172.21.15.158:0/3228192828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c15d840 0x7f616c15dc20 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f616c0cea30 tx=0x7f615c009bd0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 -- 172.21.15.158:0/3228192828 shutdown_connections 2023-10-16T09:56:21.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 --2- 172.21.15.158:0/3228192828 >> v2:172.21.15.172:3300/0 conn(0x7f616c0d2010 0x7f616c06bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 --2- 172.21.15.158:0/3228192828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c15d840 0x7f616c15dc20 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 --2- 172.21.15.158:0/3228192828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c0b38f0 0x7f616c0d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 -- 172.21.15.158:0/3228192828 >> 172.21.15.158:0/3228192828 conn(0x7f616c06ec00 msgr2=0x7f616c06efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:21.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 -- 172.21.15.158:0/3228192828 shutdown_connections 2023-10-16T09:56:21.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.259+0000 7f6171718700 1 -- 172.21.15.158:0/3228192828 wait complete. 2023-10-16T09:56:21.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.260+0000 7f6171718700 1 Processor -- start 2023-10-16T09:56:21.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.261+0000 7f6171718700 1 -- start start 2023-10-16T09:56:21.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.261+0000 7f6171718700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c0b38f0 0x7f616c0b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.261+0000 7f6171718700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0d2010 0x7f616c15ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.261+0000 7f6171718700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c15d840 0x7f616c160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.261+0000 7f6171718700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0b7070 con 0x7f616c0d2010 2023-10-16T09:56:21.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.261+0000 7f6171718700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1511c0 con 0x7f616c0b38f0 2023-10-16T09:56:21.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.261+0000 7f6171718700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0b1e60 con 0x7f616c15d840 2023-10-16T09:56:21.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0d2010 0x7f616c15ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0d2010 0x7f616c15ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49396/0 (socket says 172.21.15.158:49396) 2023-10-16T09:56:21.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 -- 172.21.15.158:0/1281998216 learned_addr learned my addr 172.21.15.158:0/1281998216 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:21.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616affd700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c0b38f0 0x7f616c0b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616b7fe700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c15d840 0x7f616c160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 -- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c15d840 msgr2=0x7f616c160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c15d840 0x7f616c160480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 -- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c0b38f0 msgr2=0x7f616c0b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c0b38f0 0x7f616c0b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 -- 172.21.15.158:0/1281998216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c009550 con 0x7f616c0d2010 2023-10-16T09:56:21.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616b7fe700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c15d840 0x7f616c160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:21.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f616a7fc700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0d2010 0x7f616c15ff40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f616000c7f0 tx=0x7f616000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:21.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f61537fe700 1 -- 172.21.15.158:0/1281998216 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160011800 con 0x7f616c0d2010 2023-10-16T09:56:21.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f61537fe700 1 -- 172.21.15.158:0/1281998216 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6160011960 con 0x7f616c0d2010 2023-10-16T09:56:21.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c0b1250 con 0x7f616c0d2010 2023-10-16T09:56:21.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f61537fe700 1 -- 172.21.15.158:0/1281998216 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160011cf0 con 0x7f616c0d2010 2023-10-16T09:56:21.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.262+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0b1810 con 0x7f616c0d2010 2023-10-16T09:56:21.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.263+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c0681b0 con 0x7f616c0d2010 2023-10-16T09:56:21.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.264+0000 7f61537fe700 1 -- 172.21.15.158:0/1281998216 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f6160018360 con 0x7f616c0d2010 2023-10-16T09:56:21.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.264+0000 7f61537fe700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f615806b100 0x7f615806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.264+0000 7f61537fe700 1 -- 172.21.15.158:0/1281998216 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6160089ac0 con 0x7f616c0d2010 2023-10-16T09:56:21.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.264+0000 7f616affd700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f615806b100 0x7f615806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.264+0000 7f616affd700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f615806b100 0x7f615806d5c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f615c000c00 tx=0x7f615c008030 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:21.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.268+0000 7f61537fe700 1 -- 172.21.15.158:0/1281998216 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6160054590 con 0x7f616c0d2010 2023-10-16T09:56:21.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.489+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f616c15e9a0 con 0x7f615806b100 2023-10-16T09:56:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.564+0000 7f61537fe700 1 -- 172.21.15.158:0/1281998216 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f615c006c60 con 0x7f615806b100 2023-10-16T09:56:21.566 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T09:56:21.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.566+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f615806b100 msgr2=0x7f615806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.566+0000 7f6171718700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f615806b100 0x7f615806d5c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f615c000c00 tx=0x7f615c008030 comp rx=0 tx=0).stop 2023-10-16T09:56:21.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.566+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0d2010 msgr2=0x7f616c15ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.566+0000 7f6171718700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0d2010 0x7f616c15ff40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f616000c7f0 tx=0x7f616000cb00 comp rx=0 tx=0).stop 2023-10-16T09:56:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.566+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 shutdown_connections 2023-10-16T09:56:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.567+0000 7f6171718700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c15d840 0x7f616c160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.567+0000 7f6171718700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c0b38f0 0x7f616c0b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.567+0000 7f6171718700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f615806b100 0x7f615806d5c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.567+0000 7f6171718700 1 --2- 172.21.15.158:0/1281998216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0d2010 0x7f616c15ff40 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.567+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 >> 172.21.15.158:0/1281998216 conn(0x7f616c06ec00 msgr2=0x7f616c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:21.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.567+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 shutdown_connections 2023-10-16T09:56:21.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.567+0000 7f6171718700 1 -- 172.21.15.158:0/1281998216 wait complete. 2023-10-16T09:56:21.580 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T09:56:21.580 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2023-10-16T09:56:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.703+0000 7f20e9a53700 1 Processor -- start 2023-10-16T09:56:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.703+0000 7f20e9a53700 1 -- start start 2023-10-16T09:56:21.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.703+0000 7f20e9a53700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20e40d29b0 0x7f20e40d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.703+0000 7f20e9a53700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e40b7c90 con 0x7f20e40d0a70 2023-10-16T09:56:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.703+0000 7f20e9a53700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e40b7f70 con 0x7f20e4160d30 2023-10-16T09:56:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.703+0000 7f20e9a53700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e40b8910 con 0x7f20e40d29b0 2023-10-16T09:56:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e27fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20e40d29b0 0x7f20e40d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e37fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f20e40d0a70 0x7f20e40b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49582/0 (socket says 172.21.15.158:49582) 2023-10-16T09:56:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e27fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20e40d29b0 0x7f20e40d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49404/0 (socket says 172.21.15.158:49404) 2023-10-16T09:56:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e37fe700 1 -- 172.21.15.158:0/803340687 learned_addr learned my addr 172.21.15.158:0/803340687 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e27fc700 1 -- 172.21.15.158:0/803340687 >> v1:172.21.15.172:6789/0 conn(0x7f20e4160d30 legacy=0x7f20e40b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e27fc700 1 -- 172.21.15.158:0/803340687 >> v1:172.21.15.158:6789/0 conn(0x7f20e40d0a70 legacy=0x7f20e40b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e27fc700 1 -- 172.21.15.158:0/803340687 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40b8250 con 0x7f20e40d29b0 2023-10-16T09:56:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e1ffb700 1 -- 172.21.15.158:0/803340687 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500376957 0 0) 0x7f20e40b7c90 con 0x7f20e40d0a70 2023-10-16T09:56:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e27fc700 1 --2- 172.21.15.158:0/803340687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e40d29b0 0x7f20e40d0530 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009870 tx=0x7f20d4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b6ad85999ad381 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e1ffb700 1 -- 172.21.15.158:0/803340687 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4004230 con 0x7f20e40d29b0 2023-10-16T09:56:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e1ffb700 1 -- 172.21.15.158:0/803340687 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d4004390 con 0x7f20e40d29b0 2023-10-16T09:56:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.704+0000 7f20e1ffb700 1 -- 172.21.15.158:0/803340687 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d40036a0 con 0x7f20e40d29b0 2023-10-16T09:56:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.705+0000 7f20e9a53700 1 -- 172.21.15.158:0/803340687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e40d29b0 msgr2=0x7f20e40d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.705+0000 7f20e9a53700 1 --2- 172.21.15.158:0/803340687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e40d29b0 0x7f20e40d0530 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009870 tx=0x7f20d4009b80 comp rx=0 tx=0).stop 2023-10-16T09:56:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.705+0000 7f20e9a53700 1 -- 172.21.15.158:0/803340687 shutdown_connections 2023-10-16T09:56:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.705+0000 7f20e9a53700 1 --2- 172.21.15.158:0/803340687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e40d29b0 0x7f20e40d0530 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.705+0000 7f20e9a53700 1 -- 172.21.15.158:0/803340687 >> 172.21.15.158:0/803340687 conn(0x7f20e40669f0 msgr2=0x7f20e4066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.705+0000 7f20e9a53700 1 -- 172.21.15.158:0/803340687 shutdown_connections 2023-10-16T09:56:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.705+0000 7f20e9a53700 1 -- 172.21.15.158:0/803340687 wait complete. 2023-10-16T09:56:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.706+0000 7f20e9a53700 1 Processor -- start 2023-10-16T09:56:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.706+0000 7f20e9a53700 1 -- start start 2023-10-16T09:56:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e9a53700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e40d0a70 0x7f20e40c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e9a53700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e40d29b0 0x7f20e4153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e9a53700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e4160d30 0x7f20e415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e9a53700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e415a8c0 con 0x7f20e4160d30 2023-10-16T09:56:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e9a53700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e4153bc0 con 0x7f20e40d29b0 2023-10-16T09:56:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e9a53700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e415b930 con 0x7f20e40d0a70 2023-10-16T09:56:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e4160d30 0x7f20e415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e4160d30 0x7f20e415b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49412/0 (socket says 172.21.15.158:49412) 2023-10-16T09:56:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 -- 172.21.15.158:0/117123399 learned_addr learned my addr 172.21.15.158:0/117123399 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 -- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e40d0a70 msgr2=0x7f20e40c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e27fc700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e40d29b0 0x7f20e4153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e2ffd700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e40d0a70 0x7f20e40c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e40d0a70 0x7f20e40c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 -- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e40d29b0 msgr2=0x7f20e4153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e40d29b0 0x7f20e4153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 -- 172.21.15.158:0/117123399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d40094a0 con 0x7f20e4160d30 2023-10-16T09:56:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e27fc700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e40d29b0 0x7f20e4153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.707+0000 7f20e37fe700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e4160d30 0x7f20e415b3f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f20dc002230 tx=0x7f20dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.708+0000 7f20cbfff700 1 -- 172.21.15.158:0/117123399 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc00ac20 con 0x7f20e4160d30 2023-10-16T09:56:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.708+0000 7f20cbfff700 1 -- 172.21.15.158:0/117123399 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20dc00ada0 con 0x7f20e4160d30 2023-10-16T09:56:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.708+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e41589b0 con 0x7f20e4160d30 2023-10-16T09:56:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.708+0000 7f20cbfff700 1 -- 172.21.15.158:0/117123399 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc007310 con 0x7f20e4160d30 2023-10-16T09:56:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.708+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e41f2e60 con 0x7f20e4160d30 2023-10-16T09:56:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.709+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e40681b0 con 0x7f20e4160d30 2023-10-16T09:56:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.710+0000 7f20cbfff700 1 -- 172.21.15.158:0/117123399 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f20dc007470 con 0x7f20e4160d30 2023-10-16T09:56:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.710+0000 7f20cbfff700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f20d006b100 0x7f20d006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.710+0000 7f20e2ffd700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f20d006b100 0x7f20d006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.710+0000 7f20cbfff700 1 -- 172.21.15.158:0/117123399 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f20dc089d20 con 0x7f20e4160d30 2023-10-16T09:56:21.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.711+0000 7f20e2ffd700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f20d006b100 0x7f20d006d5c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20d800a7d0 tx=0x7f20d8008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:21.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.714+0000 7f20cbfff700 1 -- 172.21.15.158:0/117123399 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f20dc0549e0 con 0x7f20e4160d30 2023-10-16T09:56:21.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:21.938+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f20e40b8530 con 0x7f20d006b100 2023-10-16T09:56:22.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.007+0000 7f20cbfff700 1 -- 172.21.15.158:0/117123399 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f20d800a420 con 0x7f20d006b100 2023-10-16T09:56:22.009 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T09:56:22.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.009+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f20d006b100 msgr2=0x7f20d006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.009+0000 7f20e9a53700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f20d006b100 0x7f20d006d5c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20d800a7d0 tx=0x7f20d8008040 comp rx=0 tx=0).stop 2023-10-16T09:56:22.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.009+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e4160d30 msgr2=0x7f20e415b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.009+0000 7f20e9a53700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e4160d30 0x7f20e415b3f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f20dc002230 tx=0x7f20dc00a060 comp rx=0 tx=0).stop 2023-10-16T09:56:22.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.009+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 shutdown_connections 2023-10-16T09:56:22.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.010+0000 7f20e9a53700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f20d006b100 0x7f20d006d5c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.010+0000 7f20e9a53700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20e4160d30 0x7f20e415b3f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.010+0000 7f20e9a53700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e40d0a70 0x7f20e40c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.010+0000 7f20e9a53700 1 --2- 172.21.15.158:0/117123399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e40d29b0 0x7f20e4153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.010+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 >> 172.21.15.158:0/117123399 conn(0x7f20e40669f0 msgr2=0x7f20e40d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:22.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.010+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 shutdown_connections 2023-10-16T09:56:22.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.010+0000 7f20e9a53700 1 -- 172.21.15.158:0/117123399 wait complete. 2023-10-16T09:56:22.022 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T09:56:22.023 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2023-10-16T09:56:22.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.145+0000 7f3c0bf52700 1 Processor -- start 2023-10-16T09:56:22.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.145+0000 7f3c0bf52700 1 -- start start 2023-10-16T09:56:22.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c0bf52700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c0415d830 0x7f3c04070320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c0bf52700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c04070860 con 0x7f3c0406d4d0 2023-10-16T09:56:22.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c0bf52700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0406bca0 con 0x7f3c040b2f10 2023-10-16T09:56:22.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c0bf52700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c04070a90 con 0x7f3c0415d830 2023-10-16T09:56:22.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c094ed700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3c0406d4d0 0x7f3c04161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49592/0 (socket says 172.21.15.158:49592) 2023-10-16T09:56:22.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c094ed700 1 -- 172.21.15.158:0/2267145619 learned_addr learned my addr 172.21.15.158:0/2267145619 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:22.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c09cee700 1 --2- 172.21.15.158:0/2267145619 >> v2:172.21.15.172:3300/0 conn(0x7f3c0415d830 0x7f3c04070320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245981655 0 0) 0x7f3c04070860 con 0x7f3c0406d4d0 2023-10-16T09:56:22.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf00035b0 con 0x7f3c0406d4d0 2023-10-16T09:56:22.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589421460 0 0) 0x7f3bf00035b0 con 0x7f3c0406d4d0 2023-10-16T09:56:22.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 >> v2:172.21.15.172:3300/0 conn(0x7f3c0415d830 msgr2=0x7f3c04070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:22.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 --2- 172.21.15.158:0/2267145619 >> v2:172.21.15.172:3300/0 conn(0x7f3c0415d830 0x7f3c04070320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 >> v1:172.21.15.172:6789/0 conn(0x7f3c040b2f10 legacy=0x7f3c040cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c04070d70 con 0x7f3c0406d4d0 2023-10-16T09:56:22.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3bf4001670 con 0x7f3c0406d4d0 2023-10-16T09:56:22.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 >> v1:172.21.15.158:6789/0 conn(0x7f3c0406d4d0 legacy=0x7f3c04161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.146+0000 7f3c08cec700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bf0003a00 0x7f3bf0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf0004850 0x7f3bf0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0005210 0x7f3bf000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c04070a90 con 0x7f3bf0003a00 2023-10-16T09:56:22.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf000c0b0 con 0x7f3bf0004850 2023-10-16T09:56:22.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf000c270 con 0x7f3bf0005210 2023-10-16T09:56:22.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c09cee700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bf0003a00 0x7f3bf0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c094ed700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf0004850 0x7f3bf0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c0a4ef700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0005210 0x7f3bf000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c09cee700 1 -- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0005210 msgr2=0x7f3bf000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:22.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c09cee700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0005210 0x7f3bf000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c09cee700 1 -- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf0004850 msgr2=0x7f3bf0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:22.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c09cee700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf0004850 0x7f3bf0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c09cee700 1 -- 172.21.15.158:0/2267145619 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf00035b0 con 0x7f3bf0003a00 2023-10-16T09:56:22.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c0a4ef700 1 -- 172.21.15.158:0/2267145619 reap_dead start 2023-10-16T09:56:22.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c09cee700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bf0003a00 0x7f3bf0003e20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009b60 tx=0x7f3bf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d5877a680fe46 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:22.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000e6c0 con 0x7f3bf0003a00 2023-10-16T09:56:22.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0000e820 con 0x7f3bf0003a00 2023-10-16T09:56:22.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.147+0000 7f3c08cec700 1 -- 172.21.15.158:0/2267145619 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000eb90 con 0x7f3bf0003a00 2023-10-16T09:56:22.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.148+0000 7f3c0bf52700 1 -- 172.21.15.158:0/2267145619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bf0003a00 msgr2=0x7f3bf0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.148+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bf0003a00 0x7f3bf0003e20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009b60 tx=0x7f3bf0005880 comp rx=0 tx=0).stop 2023-10-16T09:56:22.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.148+0000 7f3c0bf52700 1 -- 172.21.15.158:0/2267145619 shutdown_connections 2023-10-16T09:56:22.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.148+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/2267145619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bf0003a00 0x7f3bf0003e20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.148+0000 7f3c0bf52700 1 -- 172.21.15.158:0/2267145619 >> 172.21.15.158:0/2267145619 conn(0x7f3c040cdf10 msgr2=0x7f3c040ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:22.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.148+0000 7f3c0bf52700 1 -- 172.21.15.158:0/2267145619 shutdown_connections 2023-10-16T09:56:22.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.148+0000 7f3c0bf52700 1 -- 172.21.15.158:0/2267145619 wait complete. 2023-10-16T09:56:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.149+0000 7f3c0bf52700 1 Processor -- start 2023-10-16T09:56:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.150+0000 7f3c0bf52700 1 -- start start 2023-10-16T09:56:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.150+0000 7f3c0bf52700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0003a00 0x7f3c04155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.150+0000 7f3c0bf52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c0415dc10 0x7f3c040c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.150+0000 7f3c0bf52700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c040c7020 0x7f3c040cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.150+0000 7f3c0bf52700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041f7b90 con 0x7f3c0415dc10 2023-10-16T09:56:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.150+0000 7f3c0bf52700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041f6f10 con 0x7f3c040c7020 2023-10-16T09:56:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.150+0000 7f3c0bf52700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041f71f0 con 0x7f3bf0003a00 2023-10-16T09:56:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c0415dc10 0x7f3c040c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c0415dc10 0x7f3c040c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49430/0 (socket says 172.21.15.158:49430) 2023-10-16T09:56:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 -- 172.21.15.158:0/1231524350 learned_addr learned my addr 172.21.15.158:0/1231524350 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c09cee700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0003a00 0x7f3c04155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c0a4ef700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c040c7020 0x7f3c040cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 -- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0003a00 msgr2=0x7f3c04155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0003a00 0x7f3c04155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 -- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c040c7020 msgr2=0x7f3c040cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c040c7020 0x7f3c040cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 -- 172.21.15.158:0/1231524350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c041f74d0 con 0x7f3c0415dc10 2023-10-16T09:56:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c094ed700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c0415dc10 0x7f3c040c6ae0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4007290 tx=0x7f3bf40075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3bfa7fc700 1 -- 172.21.15.158:0/1231524350 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4011580 con 0x7f3c0415dc10 2023-10-16T09:56:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3bfa7fc700 1 -- 172.21.15.158:0/1231524350 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf40116e0 con 0x7f3c0415dc10 2023-10-16T09:56:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c041f77b0 con 0x7f3c0415dc10 2023-10-16T09:56:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3bfa7fc700 1 -- 172.21.15.158:0/1231524350 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4011a70 con 0x7f3c0415dc10 2023-10-16T09:56:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.151+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c04155c80 con 0x7f3c0415dc10 2023-10-16T09:56:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.152+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c040c4dd0 con 0x7f3c0415dc10 2023-10-16T09:56:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.153+0000 7f3bfa7fc700 1 -- 172.21.15.158:0/1231524350 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f3bf4011bd0 con 0x7f3c0415dc10 2023-10-16T09:56:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.153+0000 7f3bfa7fc700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f3be806b100 0x7f3be806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.153+0000 7f3bfa7fc700 1 -- 172.21.15.158:0/1231524350 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f3bf4089b50 con 0x7f3c0415dc10 2023-10-16T09:56:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.153+0000 7f3c09cee700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f3be806b100 0x7f3be806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.153+0000 7f3c09cee700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f3be806b100 0x7f3be806d5c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c00000c00 tx=0x7f3c0001e040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.157+0000 7f3bfa7fc700 1 -- 172.21.15.158:0/1231524350 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c040c4dd0 con 0x7f3c0415dc10 2023-10-16T09:56:22.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.381+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c041612b0 con 0x7f3be806b100 2023-10-16T09:56:22.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.451+0000 7f3bfa7fc700 1 -- 172.21.15.158:0/1231524350 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3c0000e990 con 0x7f3be806b100 2023-10-16T09:56:22.453 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T09:56:22.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f3be806b100 msgr2=0x7f3be806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f3be806b100 0x7f3be806d5c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c00000c00 tx=0x7f3c0001e040 comp rx=0 tx=0).stop 2023-10-16T09:56:22.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c0415dc10 msgr2=0x7f3c040c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c0415dc10 0x7f3c040c6ae0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4007290 tx=0x7f3bf40075a0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 shutdown_connections 2023-10-16T09:56:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c040c7020 0x7f3c040cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bf0003a00 0x7f3c04155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f3be806b100 0x7f3be806d5c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 --2- 172.21.15.158:0/1231524350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c0415dc10 0x7f3c040c6ae0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 >> 172.21.15.158:0/1231524350 conn(0x7f3c040cdf10 msgr2=0x7f3c040b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.453+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 shutdown_connections 2023-10-16T09:56:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.454+0000 7f3c0bf52700 1 -- 172.21.15.158:0/1231524350 wait complete. 2023-10-16T09:56:22.466 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T09:56:22.467 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2023-10-16T09:56:22.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.590+0000 7f157582b700 1 Processor -- start 2023-10-16T09:56:22.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.590+0000 7f157582b700 1 -- start start 2023-10-16T09:56:22.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f157582b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f15700d29b0 0x7f15700d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f157582b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f15700d0a70 0x7f15700b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f157582b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700b7c90 con 0x7f1570160d30 2023-10-16T09:56:22.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f157582b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15700b8910 con 0x7f15700d0a70 2023-10-16T09:56:22.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f157582b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15700d0f00 con 0x7f15700d29b0 2023-10-16T09:56:22.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f15700d0a70 0x7f15700b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f15700d0a70 0x7f15700b6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49440/0 (socket says 172.21.15.158:49440) 2023-10-16T09:56:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 -- 172.21.15.158:0/151032968 learned_addr learned my addr 172.21.15.158:0/151032968 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156e7fc700 1 --2- 172.21.15.158:0/151032968 >> v2:172.21.15.172:3300/0 conn(0x7f15700d29b0 0x7f15700d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 -- 172.21.15.158:0/151032968 >> v2:172.21.15.172:3300/0 conn(0x7f15700d29b0 msgr2=0x7f15700d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 --2- 172.21.15.158:0/151032968 >> v2:172.21.15.172:3300/0 conn(0x7f15700d29b0 0x7f15700d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 -- 172.21.15.158:0/151032968 >> v1:172.21.15.172:6789/0 conn(0x7f1570160d30 legacy=0x7f15700b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 -- 172.21.15.158:0/151032968 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15700b7f70 con 0x7f15700d0a70 2023-10-16T09:56:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156f7fe700 1 --2- 172.21.15.158:0/151032968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15700b6580 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1568009e50 tx=0x7f156800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7f3526e8fac065 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156dffb700 1 -- 172.21.15.158:0/151032968 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156800e070 con 0x7f15700d0a70 2023-10-16T09:56:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156dffb700 1 -- 172.21.15.158:0/151032968 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1568003a20 con 0x7f15700d0a70 2023-10-16T09:56:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.591+0000 7f156dffb700 1 -- 172.21.15.158:0/151032968 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568003dd0 con 0x7f15700d0a70 2023-10-16T09:56:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 -- 172.21.15.158:0/151032968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 msgr2=0x7f15700b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 --2- 172.21.15.158:0/151032968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15700b6580 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1568009e50 tx=0x7f156800b330 comp rx=0 tx=0).stop 2023-10-16T09:56:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 -- 172.21.15.158:0/151032968 shutdown_connections 2023-10-16T09:56:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 --2- 172.21.15.158:0/151032968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15700b6580 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 --2- 172.21.15.158:0/151032968 >> v2:172.21.15.172:3300/0 conn(0x7f15700d29b0 0x7f15700d0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 -- 172.21.15.158:0/151032968 >> 172.21.15.158:0/151032968 conn(0x7f15700669f0 msgr2=0x7f1570066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 -- 172.21.15.158:0/151032968 shutdown_connections 2023-10-16T09:56:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.592+0000 7f157582b700 1 -- 172.21.15.158:0/151032968 wait complete. 2023-10-16T09:56:22.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.593+0000 7f157582b700 1 Processor -- start 2023-10-16T09:56:22.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.594+0000 7f157582b700 1 -- start start 2023-10-16T09:56:22.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15701f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15700d29b0 0x7f1570159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1570160d30 0x7f157015c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570155430 con 0x7f15700d0a70 2023-10-16T09:56:22.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15701f2db0 con 0x7f1570160d30 2023-10-16T09:56:22.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15701556f0 con 0x7f15700d29b0 2023-10-16T09:56:22.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15701f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15701f2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49442/0 (socket says 172.21.15.158:49442) 2023-10-16T09:56:22.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 -- 172.21.15.158:0/4158605312 learned_addr learned my addr 172.21.15.158:0/4158605312 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:22.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156e7fc700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15700d29b0 0x7f1570159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156f7fe700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1570160d30 0x7f157015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 -- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15700d29b0 msgr2=0x7f1570159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15700d29b0 0x7f1570159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 -- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1570160d30 msgr2=0x7f157015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1570160d30 0x7f157015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 -- 172.21.15.158:0/4158605312 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568009b30 con 0x7f15700d0a70 2023-10-16T09:56:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f156effd700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15701f2870 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f156000b470 tx=0x7f156000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f1557fff700 1 -- 172.21.15.158:0/4158605312 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560011830 con 0x7f15700d0a70 2023-10-16T09:56:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f1557fff700 1 -- 172.21.15.158:0/4158605312 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1560011990 con 0x7f15700d0a70 2023-10-16T09:56:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15701559d0 con 0x7f15700d0a70 2023-10-16T09:56:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f1557fff700 1 -- 172.21.15.158:0/4158605312 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560011d40 con 0x7f15700d0a70 2023-10-16T09:56:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.595+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1570159780 con 0x7f15700d0a70 2023-10-16T09:56:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.596+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15700681b0 con 0x7f15700d0a70 2023-10-16T09:56:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.597+0000 7f1557fff700 1 -- 172.21.15.158:0/4158605312 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f156001a400 con 0x7f15700d0a70 2023-10-16T09:56:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.597+0000 7f1557fff700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f155c06b100 0x7f155c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.597+0000 7f1557fff700 1 -- 172.21.15.158:0/4158605312 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f1560089c30 con 0x7f15700d0a70 2023-10-16T09:56:22.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.597+0000 7f156e7fc700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f155c06b100 0x7f155c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:22.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.598+0000 7f156e7fc700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f155c06b100 0x7f155c06d5c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f15700b5c50 tx=0x7f1564008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:22.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.602+0000 7f1557fff700 1 -- 172.21.15.158:0/4158605312 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f15600547f0 con 0x7f15700d0a70 2023-10-16T09:56:22.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.825+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f15700b8530 con 0x7f155c06b100 2023-10-16T09:56:22.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.895+0000 7f1557fff700 1 -- 172.21.15.158:0/4158605312 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f156400ae30 con 0x7f155c06b100 2023-10-16T09:56:22.896 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T09:56:22.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f155c06b100 msgr2=0x7f155c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f155c06b100 0x7f155c06d5c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f15700b5c50 tx=0x7f1564008040 comp rx=0 tx=0).stop 2023-10-16T09:56:22.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 msgr2=0x7f15701f2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:22.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15701f2870 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f156000b470 tx=0x7f156000b830 comp rx=0 tx=0).stop 2023-10-16T09:56:22.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 shutdown_connections 2023-10-16T09:56:22.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1570160d30 0x7f157015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f155c06b100 0x7f155c06d5c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15700d0a70 0x7f15701f2870 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 --2- 172.21.15.158:0/4158605312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15700d29b0 0x7f1570159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:22.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 >> 172.21.15.158:0/4158605312 conn(0x7f15700669f0 msgr2=0x7f157006b870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:22.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 shutdown_connections 2023-10-16T09:56:22.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:22.897+0000 7f157582b700 1 -- 172.21.15.158:0/4158605312 wait complete. 2023-10-16T09:56:22.910 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T09:56:22.910 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2023-10-16T09:56:23.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.033+0000 7fe86c175700 1 Processor -- start 2023-10-16T09:56:23.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.033+0000 7fe86c175700 1 -- start start 2023-10-16T09:56:23.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe86c175700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 0x7fe8640caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe86c175700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe8640cc590 0x7fe86406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe86c175700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8640cbd30 con 0x7fe86406ae10 2023-10-16T09:56:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe86c175700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86406b280 con 0x7fe8640cc590 2023-10-16T09:56:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe86c175700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640cc1c0 con 0x7fe86415da40 2023-10-16T09:56:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe86a712700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe86406ae10 0x7fe8640ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49630/0 (socket says 172.21.15.158:49630) 2023-10-16T09:56:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe86a712700 1 -- 172.21.15.158:0/4265653468 learned_addr learned my addr 172.21.15.158:0/4265653468 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe869f11700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 0x7fe8640caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe869710700 1 --2- 172.21.15.158:0/4265653468 >> v2:172.21.15.172:3300/0 conn(0x7fe8640cc590 0x7fe86406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975998398 0 0) 0x7fe8640cbd30 con 0x7fe86406ae10 2023-10-16T09:56:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.034+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8500035b0 con 0x7fe86406ae10 2023-10-16T09:56:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920998642 0 0) 0x7fe8500035b0 con 0x7fe86406ae10 2023-10-16T09:56:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 msgr2=0x7fe8640caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 0x7fe8640caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 >> v2:172.21.15.172:3300/0 conn(0x7fe8640cc590 msgr2=0x7fe86406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 --2- 172.21.15.158:0/4265653468 >> v2:172.21.15.172:3300/0 conn(0x7fe8640cc590 0x7fe86406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86415de20 con 0x7fe86406ae10 2023-10-16T09:56:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe85c0019f0 con 0x7fe86406ae10 2023-10-16T09:56:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 >> v1:172.21.15.158:6789/0 conn(0x7fe86406ae10 legacy=0x7fe8640ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe850003a00 0x7fe850003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe850004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 0x7fe850005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86406b280 con 0x7fe850004850 2023-10-16T09:56:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640cc1c0 con 0x7fe850003a00 2023-10-16T09:56:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85000ba10 con 0x7fe86415da40 2023-10-16T09:56:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869710700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe850004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869f11700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe850003a00 0x7fe850003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe86a712700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 0x7fe850005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869710700 1 -- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 msgr2=0x7fe850005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869710700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe86415da40 0x7fe850005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869710700 1 -- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe850003a00 msgr2=0x7fe850003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869710700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe850003a00 0x7fe850003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869710700 1 -- 172.21.15.158:0/4265653468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500035b0 con 0x7fe850004850 2023-10-16T09:56:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.035+0000 7fe869710700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe850004cd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe860009b60 tx=0x7fe850005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9f909b39227e22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86000e6c0 con 0x7fe850004850 2023-10-16T09:56:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86000e820 con 0x7fe850004850 2023-10-16T09:56:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe868f0f700 1 -- 172.21.15.158:0/4265653468 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86000eb90 con 0x7fe850004850 2023-10-16T09:56:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe86c175700 1 -- 172.21.15.158:0/4265653468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 msgr2=0x7fe850004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe86c175700 1 --2- 172.21.15.158:0/4265653468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe850004cd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe860009b60 tx=0x7fe850005880 comp rx=0 tx=0).stop 2023-10-16T09:56:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe86a712700 1 -- 172.21.15.158:0/4265653468 reap_dead start 2023-10-16T09:56:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe86c175700 1 -- 172.21.15.158:0/4265653468 shutdown_connections 2023-10-16T09:56:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe86c175700 1 -- 172.21.15.158:0/4265653468 >> 172.21.15.158:0/4265653468 conn(0x7fe864099a10 msgr2=0x7fe8640d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe86c175700 1 -- 172.21.15.158:0/4265653468 shutdown_connections 2023-10-16T09:56:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.036+0000 7fe86c175700 1 -- 172.21.15.158:0/4265653468 wait complete. 2023-10-16T09:56:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.037+0000 7fe86c175700 1 Processor -- start 2023-10-16T09:56:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.037+0000 7fe86c175700 1 -- start start 2023-10-16T09:56:23.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe86c175700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe86415c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe86c175700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8641e6510 0x7fe8641e6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe86c175700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8640b2d20 0x7fe8640b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe86c175700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8641e6ed0 con 0x7fe850004850 2023-10-16T09:56:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe86c175700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640b36f0 con 0x7fe8641e6510 2023-10-16T09:56:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe86c175700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86409e0a0 con 0x7fe8640b2d20 2023-10-16T09:56:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe86415c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe86415c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49462/0 (socket says 172.21.15.158:49462) 2023-10-16T09:56:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 -- 172.21.15.158:0/3803759636 learned_addr learned my addr 172.21.15.158:0/3803759636 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 -- 172.21.15.158:0/3803759636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8640b2d20 msgr2=0x7fe8640b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8640b2d20 0x7fe8640b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 -- 172.21.15.158:0/3803759636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8641e6510 msgr2=0x7fe8641e6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8641e6510 0x7fe8641e6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.038+0000 7fe869f11700 1 -- 172.21.15.158:0/3803759636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8640c8b60 con 0x7fe850004850 2023-10-16T09:56:23.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.039+0000 7fe869f11700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe86415c930 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe854011aa0 tx=0x7fe854011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.039+0000 7fe85a7fc700 1 -- 172.21.15.158:0/3803759636 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85400f900 con 0x7fe850004850 2023-10-16T09:56:23.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.039+0000 7fe85a7fc700 1 -- 172.21.15.158:0/3803759636 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85400fa60 con 0x7fe850004850 2023-10-16T09:56:23.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.039+0000 7fe86c175700 1 -- 172.21.15.158:0/3803759636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8640b3900 con 0x7fe850004850 2023-10-16T09:56:23.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.039+0000 7fe85a7fc700 1 -- 172.21.15.158:0/3803759636 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85400fe10 con 0x7fe850004850 2023-10-16T09:56:23.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.039+0000 7fe86c175700 1 -- 172.21.15.158:0/3803759636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8641f6820 con 0x7fe850004850 2023-10-16T09:56:23.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.040+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8400052f0 con 0x7fe850004850 2023-10-16T09:56:23.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.040+0000 7fe85a7fc700 1 -- 172.21.15.158:0/3803759636 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fe854014440 con 0x7fe850004850 2023-10-16T09:56:23.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.041+0000 7fe85a7fc700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fe84806b100 0x7fe84806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.041+0000 7fe85a7fc700 1 -- 172.21.15.158:0/3803759636 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fe85408a1c0 con 0x7fe850004850 2023-10-16T09:56:23.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.041+0000 7fe869710700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fe84806b100 0x7fe84806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.041+0000 7fe869710700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fe84806b100 0x7fe84806d5c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe860000c00 tx=0x7fe860013000 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.045+0000 7fe85a7fc700 1 -- 172.21.15.158:0/3803759636 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe854054ee0 con 0x7fe850004850 2023-10-16T09:56:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.263+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe840000bc0 con 0x7fe84806b100 2023-10-16T09:56:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.332+0000 7fe85a7fc700 1 -- 172.21.15.158:0/3803759636 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe840000bc0 con 0x7fe84806b100 2023-10-16T09:56:23.334 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T09:56:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.334+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fe84806b100 msgr2=0x7fe84806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.334+0000 7fe847fff700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fe84806b100 0x7fe84806d5c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe860000c00 tx=0x7fe860013000 comp rx=0 tx=0).stop 2023-10-16T09:56:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.334+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 msgr2=0x7fe86415c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.334+0000 7fe847fff700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe86415c930 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe854011aa0 tx=0x7fe854011e60 comp rx=0 tx=0).stop 2023-10-16T09:56:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.334+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 shutdown_connections 2023-10-16T09:56:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.334+0000 7fe847fff700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8640b2d20 0x7fe8640b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.335+0000 7fe847fff700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fe84806b100 0x7fe84806d5c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.335+0000 7fe847fff700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe850004850 0x7fe86415c930 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.335+0000 7fe847fff700 1 --2- 172.21.15.158:0/3803759636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8641e6510 0x7fe8641e6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.335+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 >> 172.21.15.158:0/3803759636 conn(0x7fe864099a10 msgr2=0x7fe8640ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.335+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 shutdown_connections 2023-10-16T09:56:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.335+0000 7fe847fff700 1 -- 172.21.15.158:0/3803759636 wait complete. 2023-10-16T09:56:23.347 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T09:56:23.348 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2023-10-16T09:56:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.470+0000 7ffbb2d44700 1 Processor -- start 2023-10-16T09:56:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.470+0000 7ffbb2d44700 1 -- start start 2023-10-16T09:56:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbb2d44700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffbac15d840 0x7ffbac161d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbb2d44700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffbac0ccde0 0x7ffbac0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbb2d44700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac15dc20 con 0x7ffbac0cf070 2023-10-16T09:56:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbb2d44700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac0cfa60 con 0x7ffbac0ccde0 2023-10-16T09:56:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbb2d44700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac162290 con 0x7ffbac15d840 2023-10-16T09:56:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbabfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffbac0cf070 0x7ffbac0d1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49648/0 (socket says 172.21.15.158:49648) 2023-10-16T09:56:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbabfff700 1 -- 172.21.15.158:0/1289650248 learned_addr learned my addr 172.21.15.158:0/1289650248 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.471+0000 7ffbb12e1700 1 --2- 172.21.15.158:0/1289650248 >> v2:172.21.15.158:3300/0 conn(0x7ffbac0ccde0 0x7ffbac0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbb0ae0700 1 --2- 172.21.15.158:0/1289650248 >> v2:172.21.15.172:3300/0 conn(0x7ffbac15d840 0x7ffbac161d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbab7fe700 1 -- 172.21.15.158:0/1289650248 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000673115 0 0) 0x7ffb98001110 con 0x7ffbac0cf070 2023-10-16T09:56:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbab7fe700 1 -- 172.21.15.158:0/1289650248 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb9c0035b0 con 0x7ffbac0cf070 2023-10-16T09:56:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbb12e1700 1 -- 172.21.15.158:0/1289650248 >> v2:172.21.15.172:3300/0 conn(0x7ffbac15d840 msgr2=0x7ffbac161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbb12e1700 1 --2- 172.21.15.158:0/1289650248 >> v2:172.21.15.172:3300/0 conn(0x7ffbac15d840 0x7ffbac161d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbb12e1700 1 -- 172.21.15.158:0/1289650248 >> v1:172.21.15.158:6789/0 conn(0x7ffbac0cf070 legacy=0x7ffbac0d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbb12e1700 1 -- 172.21.15.158:0/1289650248 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac06c0e0 con 0x7ffbac0ccde0 2023-10-16T09:56:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbb12e1700 1 --2- 172.21.15.158:0/1289650248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0ccde0 0x7ffbac0cf520 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0cebb0 tx=0x7ffba400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9651807f2852c917 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbab7fe700 1 -- 172.21.15.158:0/1289650248 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba400e070 con 0x7ffbac0ccde0 2023-10-16T09:56:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbab7fe700 1 -- 172.21.15.158:0/1289650248 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba400bb20 con 0x7ffbac0ccde0 2023-10-16T09:56:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.472+0000 7ffbab7fe700 1 -- 172.21.15.158:0/1289650248 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4012430 con 0x7ffbac0ccde0 2023-10-16T09:56:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 -- 172.21.15.158:0/1289650248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0ccde0 msgr2=0x7ffbac0cf520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/1289650248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0ccde0 0x7ffbac0cf520 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0cebb0 tx=0x7ffba400b330 comp rx=0 tx=0).stop 2023-10-16T09:56:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 -- 172.21.15.158:0/1289650248 shutdown_connections 2023-10-16T09:56:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/1289650248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0ccde0 0x7ffbac0cf520 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/1289650248 >> v2:172.21.15.172:3300/0 conn(0x7ffbac15d840 0x7ffbac161d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 -- 172.21.15.158:0/1289650248 >> 172.21.15.158:0/1289650248 conn(0x7ffbac06a8d0 msgr2=0x7ffbac06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 -- 172.21.15.158:0/1289650248 shutdown_connections 2023-10-16T09:56:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.473+0000 7ffbb2d44700 1 -- 172.21.15.158:0/1289650248 wait complete. 2023-10-16T09:56:23.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.474+0000 7ffbb2d44700 1 Processor -- start 2023-10-16T09:56:23.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.474+0000 7ffbb2d44700 1 -- start start 2023-10-16T09:56:23.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbb2d44700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbac0ccde0 0x7ffbac1f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbb2d44700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0cf070 0x7ffbac0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbb2d44700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbac15d840 0x7ffbac0c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbb2d44700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0c97f0 con 0x7ffbac0cf070 2023-10-16T09:56:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbb2d44700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac1f3030 con 0x7ffbac15d840 2023-10-16T09:56:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbb2d44700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0c8f50 con 0x7ffbac0ccde0 2023-10-16T09:56:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbabfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0cf070 0x7ffbac0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbabfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0cf070 0x7ffbac0c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49478/0 (socket says 172.21.15.158:49478) 2023-10-16T09:56:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.475+0000 7ffbabfff700 1 -- 172.21.15.158:0/850699002 learned_addr learned my addr 172.21.15.158:0/850699002 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbb12e1700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbac15d840 0x7ffbac0c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbb0ae0700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbac0ccde0 0x7ffbac1f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbabfff700 1 -- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbac0ccde0 msgr2=0x7ffbac1f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbabfff700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbac0ccde0 0x7ffbac1f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbabfff700 1 -- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbac15d840 msgr2=0x7ffbac0c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbabfff700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbac15d840 0x7ffbac0c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbabfff700 1 -- 172.21.15.158:0/850699002 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4009a70 con 0x7ffbac0cf070 2023-10-16T09:56:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbb12e1700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbac15d840 0x7ffbac0c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbb0ae0700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbac0ccde0 0x7ffbac1f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbabfff700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0cf070 0x7ffbac0c8570 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ffb98007fd0 tx=0x7ffb9800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffba97fa700 1 -- 172.21.15.158:0/850699002 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98005d70 con 0x7ffbac0cf070 2023-10-16T09:56:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffba97fa700 1 -- 172.21.15.158:0/850699002 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb980070d0 con 0x7ffbac0cf070 2023-10-16T09:56:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffba97fa700 1 -- 172.21.15.158:0/850699002 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98007460 con 0x7ffbac0cf070 2023-10-16T09:56:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac0c9230 con 0x7ffbac0cf070 2023-10-16T09:56:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.476+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac0c5170 con 0x7ffbac0cf070 2023-10-16T09:56:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.477+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac0c4fd0 con 0x7ffbac0cf070 2023-10-16T09:56:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.477+0000 7ffba97fa700 1 -- 172.21.15.158:0/850699002 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ffb980075c0 con 0x7ffbac0cf070 2023-10-16T09:56:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.478+0000 7ffba97fa700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb9006b100 0x7ffb9006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.478+0000 7ffbb0ae0700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb9006b100 0x7ffb9006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.478+0000 7ffba97fa700 1 -- 172.21.15.158:0/850699002 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7ffb98088f30 con 0x7ffbac0cf070 2023-10-16T09:56:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.478+0000 7ffbb0ae0700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb9006b100 0x7ffb9006d5c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffbac1607a0 tx=0x7ffba0008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.482+0000 7ffba97fa700 1 -- 172.21.15.158:0/850699002 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffbac0c4fd0 con 0x7ffbac0cf070 2023-10-16T09:56:23.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.708+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbac06c6a0 con 0x7ffb9006b100 2023-10-16T09:56:23.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.780+0000 7ffba97fa700 1 -- 172.21.15.158:0/850699002 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ffba0009800 con 0x7ffb9006b100 2023-10-16T09:56:23.781 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T09:56:23.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb9006b100 msgr2=0x7ffb9006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb9006b100 0x7ffb9006d5c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffbac1607a0 tx=0x7ffba0008040 comp rx=0 tx=0).stop 2023-10-16T09:56:23.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0cf070 msgr2=0x7ffbac0c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0cf070 0x7ffbac0c8570 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ffb98007fd0 tx=0x7ffb9800ca90 comp rx=0 tx=0).stop 2023-10-16T09:56:23.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 shutdown_connections 2023-10-16T09:56:23.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbac15d840 0x7ffbac0c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbac0ccde0 0x7ffbac1f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb9006b100 0x7ffb9006d5c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 --2- 172.21.15.158:0/850699002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbac0cf070 0x7ffbac0c8570 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 >> 172.21.15.158:0/850699002 conn(0x7ffbac06a8d0 msgr2=0x7ffbac15b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:23.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 shutdown_connections 2023-10-16T09:56:23.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.782+0000 7ffbb2d44700 1 -- 172.21.15.158:0/850699002 wait complete. 2023-10-16T09:56:23.795 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T09:56:23.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:23.796 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T09:56:23.815 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:23.815 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:23.815 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:23.816 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:23.816 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:23.816 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:23.816 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:23.816 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:23.817 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:23.817 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:23.817 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:23.817 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:23.818 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2023-10-16T09:56:23.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.976+0000 7f2cdea90700 1 Processor -- start 2023-10-16T09:56:23.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.976+0000 7f2cdea90700 1 -- start start 2023-10-16T09:56:23.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdea90700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2cd80aefa0 0x7f2cd8161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdea90700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2cd8071cb0 0x7f2cd806ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdea90700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80b3dc0 con 0x7f2cd815eae0 2023-10-16T09:56:23.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdea90700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd815ddf0 con 0x7f2cd8071cb0 2023-10-16T09:56:23.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdea90700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd81622f0 con 0x7f2cd80aefa0 2023-10-16T09:56:23.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdd02d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2cd8071cb0 0x7f2cd806ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cd7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2cd815eae0 0x7f2cd80715a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49660/0 (socket says 172.21.15.158:49660) 2023-10-16T09:56:23.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cd7fff700 1 -- 172.21.15.158:0/5176635 learned_addr learned my addr 172.21.15.158:0/5176635 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:23.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdd02d700 1 -- 172.21.15.158:0/5176635 >> v2:172.21.15.172:3300/0 conn(0x7f2cd80aefa0 msgr2=0x7f2cd8161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:23.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdc82c700 1 --2- 172.21.15.158:0/5176635 >> v2:172.21.15.172:3300/0 conn(0x7f2cd80aefa0 0x7f2cd8161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cd77fe700 1 -- 172.21.15.158:0/5176635 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858053256 0 0) 0x7f2cd80b3dc0 con 0x7f2cd815eae0 2023-10-16T09:56:23.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdd02d700 1 --2- 172.21.15.158:0/5176635 >> v2:172.21.15.172:3300/0 conn(0x7f2cd80aefa0 0x7f2cd8161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdd02d700 1 -- 172.21.15.158:0/5176635 >> v1:172.21.15.158:6789/0 conn(0x7f2cd815eae0 legacy=0x7f2cd80715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdd02d700 1 -- 172.21.15.158:0/5176635 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80b3ae0 con 0x7f2cd8071cb0 2023-10-16T09:56:23.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.977+0000 7f2cdc82c700 1 --2- 172.21.15.158:0/5176635 >> v2:172.21.15.172:3300/0 conn(0x7f2cd80aefa0 0x7f2cd8161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:23.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdd02d700 1 --2- 172.21.15.158:0/5176635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd8071cb0 0x7f2cd806ada0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8072180 tx=0x7f2cd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422e7800e8db0e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cd77fe700 1 -- 172.21.15.158:0/5176635 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd000e070 con 0x7f2cd8071cb0 2023-10-16T09:56:23.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cd77fe700 1 -- 172.21.15.158:0/5176635 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd000bbd0 con 0x7f2cd8071cb0 2023-10-16T09:56:23.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cd77fe700 1 -- 172.21.15.158:0/5176635 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0012430 con 0x7f2cd8071cb0 2023-10-16T09:56:23.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 -- 172.21.15.158:0/5176635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd8071cb0 msgr2=0x7f2cd806ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 --2- 172.21.15.158:0/5176635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd8071cb0 0x7f2cd806ada0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8072180 tx=0x7f2cd000b330 comp rx=0 tx=0).stop 2023-10-16T09:56:23.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 -- 172.21.15.158:0/5176635 shutdown_connections 2023-10-16T09:56:23.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 --2- 172.21.15.158:0/5176635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd8071cb0 0x7f2cd806ada0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 --2- 172.21.15.158:0/5176635 >> v2:172.21.15.172:3300/0 conn(0x7f2cd80aefa0 0x7f2cd8161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 -- 172.21.15.158:0/5176635 >> 172.21.15.158:0/5176635 conn(0x7f2cd80b1bd0 msgr2=0x7f2cd80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:23.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 -- 172.21.15.158:0/5176635 shutdown_connections 2023-10-16T09:56:23.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.978+0000 7f2cdea90700 1 -- 172.21.15.158:0/5176635 wait complete. 2023-10-16T09:56:23.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 Processor -- start 2023-10-16T09:56:23.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 -- start start 2023-10-16T09:56:23.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cd8071cb0 0x7f2cd8158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd80aefa0 0x7f2cd80ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd815eae0 0x7f2cd80ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd815bfa0 con 0x7f2cd815eae0 2023-10-16T09:56:23.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd815b390 con 0x7f2cd80aefa0 2023-10-16T09:56:23.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.980+0000 7f2cdea90700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd815b670 con 0x7f2cd8071cb0 2023-10-16T09:56:23.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd815eae0 0x7f2cd80ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd815eae0 0x7f2cd80ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49494/0 (socket says 172.21.15.158:49494) 2023-10-16T09:56:23.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 -- 172.21.15.158:0/2531957268 learned_addr learned my addr 172.21.15.158:0/2531957268 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:23.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cd7fff700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd80aefa0 0x7f2cd80ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdc82c700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cd8071cb0 0x7f2cd8158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 -- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cd8071cb0 msgr2=0x7f2cd8158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cd8071cb0 0x7f2cd8158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 -- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd80aefa0 msgr2=0x7f2cd80ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:23.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd80aefa0 0x7f2cd80ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:23.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 -- 172.21.15.158:0/2531957268 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0009a70 con 0x7f2cd815eae0 2023-10-16T09:56:23.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdc82c700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cd8071cb0 0x7f2cd8158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:23.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdd02d700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd815eae0 0x7f2cd80ade10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8072500 tx=0x7f2cd0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cd57fa700 1 -- 172.21.15.158:0/2531957268 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0012430 con 0x7f2cd815eae0 2023-10-16T09:56:23.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cd57fa700 1 -- 172.21.15.158:0/2531957268 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd0020070 con 0x7f2cd815eae0 2023-10-16T09:56:23.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cd57fa700 1 -- 172.21.15.158:0/2531957268 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0003ea0 con 0x7f2cd815eae0 2023-10-16T09:56:23.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd815b950 con 0x7f2cd815eae0 2023-10-16T09:56:23.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.981+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd8151c30 con 0x7f2cd815eae0 2023-10-16T09:56:23.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.982+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd815c150 con 0x7f2cd815eae0 2023-10-16T09:56:23.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.983+0000 7f2cd57fa700 1 -- 172.21.15.158:0/2531957268 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f2cd0004280 con 0x7f2cd815eae0 2023-10-16T09:56:23.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.983+0000 7f2cd57fa700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2cbc06b100 0x7f2cbc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:23.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.983+0000 7f2cd57fa700 1 -- 172.21.15.158:0/2531957268 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f2cd0022070 con 0x7f2cd815eae0 2023-10-16T09:56:23.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.983+0000 7f2cdc82c700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2cbc06b100 0x7f2cbc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:23.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.983+0000 7f2cdc82c700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2cbc06b100 0x7f2cbc06d5c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc006fd0 tx=0x7f2ccc008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:23.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:23.987+0000 7f2cd57fa700 1 -- 172.21.15.158:0/2531957268 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2cd815c150 con 0x7f2cd815eae0 2023-10-16T09:56:24.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.202+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2cd815e2e0 con 0x7f2cbc06b100 2023-10-16T09:56:24.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.210+0000 7f2cd57fa700 1 -- 172.21.15.158:0/2531957268 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2ccc00c050 con 0x7f2cbc06b100 2023-10-16T09:56:24.211 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:24.211 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:00", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:24.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2cbc06b100 msgr2=0x7f2cbc06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2cbc06b100 0x7f2cbc06d5c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc006fd0 tx=0x7f2ccc008040 comp rx=0 tx=0).stop 2023-10-16T09:56:24.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd815eae0 msgr2=0x7f2cd80ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd815eae0 0x7f2cd80ade10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8072500 tx=0x7f2cd0004ad0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 shutdown_connections 2023-10-16T09:56:24.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f2cbc06b100 0x7f2cbc06d5c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cd815eae0 0x7f2cd80ade10 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cd8071cb0 0x7f2cd8158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 --2- 172.21.15.158:0/2531957268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd80aefa0 0x7f2cd80ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 >> 172.21.15.158:0/2531957268 conn(0x7f2cd80b1bd0 msgr2=0x7f2cd81526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:24.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 shutdown_connections 2023-10-16T09:56:24.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.212+0000 7f2cdea90700 1 -- 172.21.15.158:0/2531957268 wait complete. 2023-10-16T09:56:24.225 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:00", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:24.225 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:00', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:24.226 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:24.226 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:24.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:24.227 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T09:56:24.245 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:24.245 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:24.245 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:24.246 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:24.246 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:24.246 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:24.246 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:24.246 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:24.247 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:24.247 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:24.247 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:24.247 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:24.248 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2023-10-16T09:56:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.405+0000 7f394b0f3700 1 Processor -- start 2023-10-16T09:56:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.405+0000 7f394b0f3700 1 -- start start 2023-10-16T09:56:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f394b0f3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 0x7f39440c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f394b0f3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394406c220 con 0x7f394406e540 2023-10-16T09:56:24.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f394b0f3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394406aa70 con 0x7f3944143790 2023-10-16T09:56:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f394b0f3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3944143b70 con 0x7f39440c50a0 2023-10-16T09:56:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f3949690700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f394406e540 0x7f394406e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49678/0 (socket says 172.21.15.158:49678) 2023-10-16T09:56:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f3949690700 1 -- 172.21.15.158:0/274315266 learned_addr learned my addr 172.21.15.158:0/274315266 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f3943fff700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 0x7f39440c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204962272 0 0) 0x7f394406c220 con 0x7f394406e540 2023-10-16T09:56:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39340035b0 con 0x7f394406e540 2023-10-16T09:56:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1126213512 0 0) 0x7f39340035b0 con 0x7f394406e540 2023-10-16T09:56:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 msgr2=0x7f39440c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 0x7f39440c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 >> v1:172.21.15.172:6789/0 conn(0x7f3944143790 legacy=0x7f394406bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394406ad50 con 0x7f394406e540 2023-10-16T09:56:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f393c0019f0 con 0x7f394406e540 2023-10-16T09:56:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 >> v1:172.21.15.158:6789/0 conn(0x7f394406e540 legacy=0x7f394406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.406+0000 7f39437fe700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3934003a30 0x7f3934003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3934004850 0x7f3934004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 0x7f3934005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3944143b70 con 0x7f3934004850 2023-10-16T09:56:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393400bac0 con 0x7f3934003a30 2023-10-16T09:56:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393400bc80 con 0x7f39440c50a0 2023-10-16T09:56:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3943fff700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3934004850 0x7f3934004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3948e8f700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3934003a30 0x7f3934003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3949690700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 0x7f3934005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3943fff700 1 -- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 msgr2=0x7f3934005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3943fff700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 0x7f3934005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3943fff700 1 -- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3934003a30 msgr2=0x7f3934003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3943fff700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3934003a30 0x7f3934003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3943fff700 1 -- 172.21.15.158:0/274315266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340035b0 con 0x7f3934004850 2023-10-16T09:56:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3949690700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39440c50a0 0x7f3934005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f3943fff700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3934004850 0x7f3934004cd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f393000ab30 tx=0x7f3934005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d11731b94fa06f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393000f9d0 con 0x7f3934004850 2023-10-16T09:56:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393000fb30 con 0x7f3934004850 2023-10-16T09:56:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.407+0000 7f39437fe700 1 -- 172.21.15.158:0/274315266 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393000fea0 con 0x7f3934004850 2023-10-16T09:56:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.408+0000 7f394b0f3700 1 -- 172.21.15.158:0/274315266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3934004850 msgr2=0x7f3934004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.408+0000 7f394b0f3700 1 --2- 172.21.15.158:0/274315266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3934004850 0x7f3934004cd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f393000ab30 tx=0x7f3934005880 comp rx=0 tx=0).stop 2023-10-16T09:56:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.408+0000 7f3949690700 1 -- 172.21.15.158:0/274315266 reap_dead start 2023-10-16T09:56:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.408+0000 7f394b0f3700 1 -- 172.21.15.158:0/274315266 shutdown_connections 2023-10-16T09:56:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.408+0000 7f394b0f3700 1 -- 172.21.15.158:0/274315266 >> 172.21.15.158:0/274315266 conn(0x7f39440cc5b0 msgr2=0x7f39440cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.408+0000 7f394b0f3700 1 -- 172.21.15.158:0/274315266 shutdown_connections 2023-10-16T09:56:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.408+0000 7f394b0f3700 1 -- 172.21.15.158:0/274315266 wait complete. 2023-10-16T09:56:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.409+0000 7f394b0f3700 1 Processor -- start 2023-10-16T09:56:24.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.409+0000 7f394b0f3700 1 -- start start 2023-10-16T09:56:24.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f394b0f3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3934004850 0x7f39440c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f394b0f3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3944158c30 0x7f39440c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f394b0f3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39440c8e70 0x7f39440cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f394b0f3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39440c6c00 con 0x7f39440c8e70 2023-10-16T09:56:24.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f394b0f3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39440c74a0 con 0x7f3944158c30 2023-10-16T09:56:24.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f394b0f3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39440c93d0 con 0x7f3934004850 2023-10-16T09:56:24.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39440c8e70 0x7f39440cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39440c8e70 0x7f39440cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49514/0 (socket says 172.21.15.158:49514) 2023-10-16T09:56:24.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 -- 172.21.15.158:0/2418232065 learned_addr learned my addr 172.21.15.158:0/2418232065 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:24.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3948e8f700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3934004850 0x7f39440c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3943fff700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3944158c30 0x7f39440c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 -- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3934004850 msgr2=0x7f39440c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3934004850 0x7f39440c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 -- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3944158c30 msgr2=0x7f39440c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3944158c30 0x7f39440c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 -- 172.21.15.158:0/2418232065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39440c6e50 con 0x7f39440c8e70 2023-10-16T09:56:24.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.410+0000 7f3949690700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39440c8e70 0x7f39440cb1b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f393c000f80 tx=0x7f393c007560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:24.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.411+0000 7f39417fa700 1 -- 172.21.15.158:0/2418232065 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c012650 con 0x7f39440c8e70 2023-10-16T09:56:24.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.411+0000 7f39417fa700 1 -- 172.21.15.158:0/2418232065 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c0127d0 con 0x7f39440c8e70 2023-10-16T09:56:24.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.411+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39440c7130 con 0x7f39440c8e70 2023-10-16T09:56:24.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.411+0000 7f39417fa700 1 -- 172.21.15.158:0/2418232065 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c012b80 con 0x7f39440c8e70 2023-10-16T09:56:24.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.411+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39441590b0 con 0x7f39440c8e70 2023-10-16T09:56:24.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.412+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39440681b0 con 0x7f39440c8e70 2023-10-16T09:56:24.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.412+0000 7f39417fa700 1 -- 172.21.15.158:0/2418232065 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f393c012ce0 con 0x7f39440c8e70 2023-10-16T09:56:24.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.412+0000 7f39417fa700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f392806b100 0x7f392806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.413+0000 7f39417fa700 1 -- 172.21.15.158:0/2418232065 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f393c08c630 con 0x7f39440c8e70 2023-10-16T09:56:24.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.413+0000 7f3948e8f700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f392806b100 0x7f392806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.413+0000 7f3948e8f700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f392806b100 0x7f392806d5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3944159df0 tx=0x7f3938009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:24.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.417+0000 7f39417fa700 1 -- 172.21.15.158:0/2418232065 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f393c0570f0 con 0x7f39440c8e70 2023-10-16T09:56:24.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.633+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f394406b310 con 0x7f392806b100 2023-10-16T09:56:24.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.641+0000 7f39417fa700 1 -- 172.21.15.158:0/2418232065 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f3938001640 con 0x7f392806b100 2023-10-16T09:56:24.643 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:24.643 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:01", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:24.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f392806b100 msgr2=0x7f392806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f392806b100 0x7f392806d5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3944159df0 tx=0x7f3938009040 comp rx=0 tx=0).stop 2023-10-16T09:56:24.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39440c8e70 msgr2=0x7f39440cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39440c8e70 0x7f39440cb1b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f393c000f80 tx=0x7f393c007560 comp rx=0 tx=0).stop 2023-10-16T09:56:24.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 shutdown_connections 2023-10-16T09:56:24.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f392806b100 0x7f392806d5c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39440c8e70 0x7f39440cb1b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3934004850 0x7f39440c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 --2- 172.21.15.158:0/2418232065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3944158c30 0x7f39440c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 >> 172.21.15.158:0/2418232065 conn(0x7f39440cc5b0 msgr2=0x7f39440ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:24.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 shutdown_connections 2023-10-16T09:56:24.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.644+0000 7f394b0f3700 1 -- 172.21.15.158:0/2418232065 wait complete. 2023-10-16T09:56:24.657 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:01", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:24.657 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:01', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:24.657 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:24.658 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:24.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:24.659 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T09:56:24.677 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:24.677 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:24.678 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:24.678 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:24.678 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:24.679 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:24.679 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:24.679 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:24.679 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:24.680 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:24.680 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:24.680 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:24.681 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2023-10-16T09:56:24.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.839+0000 7f1267b00700 1 Processor -- start 2023-10-16T09:56:24.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.839+0000 7f1267b00700 1 -- start start 2023-10-16T09:56:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f1267b00700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f126015b750 0x7f12600c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f1267b00700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12600d0d20 0x7f126006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f1267b00700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126006b260 con 0x7f126006ae10 2023-10-16T09:56:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f1267b00700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600cb480 con 0x7f126015b750 2023-10-16T09:56:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f1267b00700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126015bb30 con 0x7f12600d0d20 2023-10-16T09:56:24.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f126589c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f126015b750 0x7f12600c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f126609d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f126006ae10 0x7f126015f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49698/0 (socket says 172.21.15.158:49698) 2023-10-16T09:56:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f126609d700 1 -- 172.21.15.158:0/425525073 learned_addr learned my addr 172.21.15.158:0/425525073 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.840+0000 7f126589c700 1 -- 172.21.15.158:0/425525073 >> v2:172.21.15.172:3300/0 conn(0x7f12600d0d20 msgr2=0x7f126006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126509b700 1 --2- 172.21.15.158:0/425525073 >> v2:172.21.15.172:3300/0 conn(0x7f12600d0d20 0x7f126006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126489a700 1 -- 172.21.15.158:0/425525073 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132112604 0 0) 0x7f1258001490 con 0x7f126006ae10 2023-10-16T09:56:24.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126589c700 1 --2- 172.21.15.158:0/425525073 >> v2:172.21.15.172:3300/0 conn(0x7f12600d0d20 0x7f126006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126589c700 1 -- 172.21.15.158:0/425525073 >> v1:172.21.15.158:6789/0 conn(0x7f126006ae10 legacy=0x7f126015f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126589c700 1 -- 172.21.15.158:0/425525073 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600cca40 con 0x7f126015b750 2023-10-16T09:56:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126509b700 1 --2- 172.21.15.158:0/425525073 >> v2:172.21.15.172:3300/0 conn(0x7f12600d0d20 0x7f126006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126589c700 1 --2- 172.21.15.158:0/425525073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126015b750 0x7f12600c3d50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f12600675b0 tx=0x7f1250009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8c7b1521980724 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:24.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126489a700 1 -- 172.21.15.158:0/425525073 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250011830 con 0x7f126015b750 2023-10-16T09:56:24.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126489a700 1 -- 172.21.15.158:0/425525073 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250011990 con 0x7f126015b750 2023-10-16T09:56:24.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f126489a700 1 -- 172.21.15.158:0/425525073 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250011d20 con 0x7f126015b750 2023-10-16T09:56:24.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f1267b00700 1 -- 172.21.15.158:0/425525073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126015b750 msgr2=0x7f12600c3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f1267b00700 1 --2- 172.21.15.158:0/425525073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126015b750 0x7f12600c3d50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f12600675b0 tx=0x7f1250009b20 comp rx=0 tx=0).stop 2023-10-16T09:56:24.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f1267b00700 1 -- 172.21.15.158:0/425525073 shutdown_connections 2023-10-16T09:56:24.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f1267b00700 1 --2- 172.21.15.158:0/425525073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126015b750 0x7f12600c3d50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f1267b00700 1 --2- 172.21.15.158:0/425525073 >> v2:172.21.15.172:3300/0 conn(0x7f12600d0d20 0x7f126006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.841+0000 7f1267b00700 1 -- 172.21.15.158:0/425525073 >> 172.21.15.158:0/425525073 conn(0x7f12600d0200 msgr2=0x7f12600d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:24.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.842+0000 7f1267b00700 1 -- 172.21.15.158:0/425525073 shutdown_connections 2023-10-16T09:56:24.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.842+0000 7f1267b00700 1 -- 172.21.15.158:0/425525073 wait complete. 2023-10-16T09:56:24.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 Processor -- start 2023-10-16T09:56:24.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 -- start start 2023-10-16T09:56:24.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006ae10 0x7f12601f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600d0d20 0x7f12600ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f126015b750 0x7f12600b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600ac6a0 con 0x7f126006ae10 2023-10-16T09:56:24.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600b2520 con 0x7f12600d0d20 2023-10-16T09:56:24.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.843+0000 7f1267b00700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600aa530 con 0x7f126015b750 2023-10-16T09:56:24.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006ae10 0x7f12601f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006ae10 0x7f12601f0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49524/0 (socket says 172.21.15.158:49524) 2023-10-16T09:56:24.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 -- 172.21.15.158:0/136066394 learned_addr learned my addr 172.21.15.158:0/136066394 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:24.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126509b700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600d0d20 0x7f12600ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 -- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f126015b750 msgr2=0x7f12600b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126609d700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f126015b750 0x7f12600b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f126015b750 0x7f12600b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 -- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600d0d20 msgr2=0x7f12600ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:24.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600d0d20 0x7f12600ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:24.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 -- 172.21.15.158:0/136066394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12500094a0 con 0x7f126006ae10 2023-10-16T09:56:24.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126509b700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600d0d20 0x7f12600ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:24.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126609d700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f126015b750 0x7f12600b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:24.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f126589c700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006ae10 0x7f12601f0b80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1250000c00 tx=0x7f1250004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:24.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f12567fc700 1 -- 172.21.15.158:0/136066394 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500038a0 con 0x7f126006ae10 2023-10-16T09:56:24.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f12567fc700 1 -- 172.21.15.158:0/136066394 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125001f070 con 0x7f126006ae10 2023-10-16T09:56:24.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12600aa810 con 0x7f126006ae10 2023-10-16T09:56:24.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f12567fc700 1 -- 172.21.15.158:0/136066394 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250003c40 con 0x7f126006ae10 2023-10-16T09:56:24.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.844+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12600aadd0 con 0x7f126006ae10 2023-10-16T09:56:24.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.845+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126004ec20 con 0x7f126006ae10 2023-10-16T09:56:24.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.846+0000 7f12567fc700 1 -- 172.21.15.158:0/136066394 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f1250026480 con 0x7f126006ae10 2023-10-16T09:56:24.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.846+0000 7f12567fc700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f124406b100 0x7f124406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:24.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.846+0000 7f12567fc700 1 -- 172.21.15.158:0/136066394 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f1250021070 con 0x7f126006ae10 2023-10-16T09:56:24.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.846+0000 7f126509b700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f124406b100 0x7f124406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:24.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.846+0000 7f126509b700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f124406b100 0x7f124406d5c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12601581a0 tx=0x7f125c008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:24.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:24.851+0000 7f12567fc700 1 -- 172.21.15.158:0/136066394 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1250056280 con 0x7f126006ae10 2023-10-16T09:56:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.068+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f126006f040 con 0x7f124406b100 2023-10-16T09:56:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.076+0000 7f12567fc700 1 -- 172.21.15.158:0/136066394 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f125c00c040 con 0x7f124406b100 2023-10-16T09:56:25.077 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:25.078 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:01", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f124406b100 msgr2=0x7f124406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f124406b100 0x7f124406d5c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12601581a0 tx=0x7f125c008040 comp rx=0 tx=0).stop 2023-10-16T09:56:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006ae10 msgr2=0x7f12601f0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006ae10 0x7f12601f0b80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1250000c00 tx=0x7f1250004750 comp rx=0 tx=0).stop 2023-10-16T09:56:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 shutdown_connections 2023-10-16T09:56:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f126015b750 0x7f12600b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f124406b100 0x7f124406d5c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f126006ae10 0x7f12601f0b80 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 --2- 172.21.15.158:0/136066394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12600d0d20 0x7f12600ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 >> 172.21.15.158:0/136066394 conn(0x7f12600d0200 msgr2=0x7f12600c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 shutdown_connections 2023-10-16T09:56:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.078+0000 7f1267b00700 1 -- 172.21.15.158:0/136066394 wait complete. 2023-10-16T09:56:25.091 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:01", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:25.092 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:01', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:25.092 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:25.092 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:25.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:25.093 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T09:56:25.111 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:25.112 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:25.112 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:25.112 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:25.112 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:25.112 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:25.113 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:25.113 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:25.113 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:25.113 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:25.113 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:25.114 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:25.115 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2023-10-16T09:56:25.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.276+0000 7f32e794f700 1 Processor -- start 2023-10-16T09:56:25.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.276+0000 7f32e794f700 1 -- start start 2023-10-16T09:56:25.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e794f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32e00ce510 0x7f32e015f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e794f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00cbaf0 0x7f32e00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e794f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00cf4c0 con 0x7f32e015b590 2023-10-16T09:56:25.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e794f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e00cc4c0 con 0x7f32e00ce510 2023-10-16T09:56:25.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e794f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b49e0 con 0x7f32e00cbaf0 2023-10-16T09:56:25.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e4eea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32e00ce510 0x7f32e015f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e5eec700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00cbaf0 0x7f32e00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e4eea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32e00ce510 0x7f32e015f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:50714/0 (socket says 172.21.15.158:50714) 2023-10-16T09:56:25.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e5eec700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00cbaf0 0x7f32e00cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:60370/0 (socket says 172.21.15.158:60370) 2023-10-16T09:56:25.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e4eea700 1 -- 172.21.15.158:0/3184630606 learned_addr learned my addr 172.21.15.158:0/3184630606 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:25.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e5eec700 1 -- 172.21.15.158:0/3184630606 >> v2:172.21.15.172:3300/0 conn(0x7f32e00ce510 msgr2=0x7f32e015f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:25.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e5eec700 1 --2- 172.21.15.158:0/3184630606 >> v2:172.21.15.172:3300/0 conn(0x7f32e00ce510 0x7f32e015f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e5eec700 1 -- 172.21.15.158:0/3184630606 >> v1:172.21.15.172:6789/0 conn(0x7f32e015b590 legacy=0x7f32e00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.277+0000 7f32e5eec700 1 -- 172.21.15.158:0/3184630606 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e015f710 con 0x7f32e00cbaf0 2023-10-16T09:56:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e4eea700 1 --2- 172.21.15.158:0/3184630606 >> v2:172.21.15.172:3300/0 conn(0x7f32e00ce510 0x7f32e015f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e5eec700 1 --2- 172.21.15.158:0/3184630606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00cbaf0 0x7f32e00cbf80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009d90 tx=0x7f32d800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e6312649b2fce6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32d7fff700 1 -- 172.21.15.158:0/3184630606 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800e070 con 0x7f32e00cbaf0 2023-10-16T09:56:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32d7fff700 1 -- 172.21.15.158:0/3184630606 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d800bb20 con 0x7f32e00cbaf0 2023-10-16T09:56:25.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32d7fff700 1 -- 172.21.15.158:0/3184630606 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8012430 con 0x7f32e00cbaf0 2023-10-16T09:56:25.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e794f700 1 -- 172.21.15.158:0/3184630606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00cbaf0 msgr2=0x7f32e00cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e794f700 1 --2- 172.21.15.158:0/3184630606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00cbaf0 0x7f32e00cbf80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009d90 tx=0x7f32d800b330 comp rx=0 tx=0).stop 2023-10-16T09:56:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e794f700 1 -- 172.21.15.158:0/3184630606 shutdown_connections 2023-10-16T09:56:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e794f700 1 --2- 172.21.15.158:0/3184630606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00cbaf0 0x7f32e00cbf80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e794f700 1 --2- 172.21.15.158:0/3184630606 >> v2:172.21.15.172:3300/0 conn(0x7f32e00ce510 0x7f32e015f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.278+0000 7f32e794f700 1 -- 172.21.15.158:0/3184630606 >> 172.21.15.158:0/3184630606 conn(0x7f32e006a8d0 msgr2=0x7f32e006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.279+0000 7f32e794f700 1 -- 172.21.15.158:0/3184630606 shutdown_connections 2023-10-16T09:56:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.279+0000 7f32e794f700 1 -- 172.21.15.158:0/3184630606 wait complete. 2023-10-16T09:56:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 Processor -- start 2023-10-16T09:56:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 -- start start 2023-10-16T09:56:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32e00cbaf0 0x7f32e00c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00ce510 0x7f32e0148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32e015b590 0x7f32e00c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00c9e80 con 0x7f32e015b590 2023-10-16T09:56:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e0149ba0 con 0x7f32e00cbaf0 2023-10-16T09:56:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.280+0000 7f32e794f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00c8ab0 con 0x7f32e00ce510 2023-10-16T09:56:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32e015b590 0x7f32e00c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32e015b590 0x7f32e00c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49542/0 (socket says 172.21.15.158:49542) 2023-10-16T09:56:25.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 -- 172.21.15.158:0/3404109447 learned_addr learned my addr 172.21.15.158:0/3404109447 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:25.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e4eea700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00ce510 0x7f32e0148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e56eb700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32e00cbaf0 0x7f32e00c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 -- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00ce510 msgr2=0x7f32e0148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00ce510 0x7f32e0148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 -- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32e00cbaf0 msgr2=0x7f32e00c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32e00cbaf0 0x7f32e00c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 -- 172.21.15.158:0/3404109447 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d0009550 con 0x7f32e015b590 2023-10-16T09:56:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e5eec700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32e015b590 0x7f32e00c91a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f32d800b680 tx=0x7f32d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32d5ffb700 1 -- 172.21.15.158:0/3404109447 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8012430 con 0x7f32e015b590 2023-10-16T09:56:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32d5ffb700 1 -- 172.21.15.158:0/3404109447 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8020070 con 0x7f32e015b590 2023-10-16T09:56:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d8009a70 con 0x7f32e015b590 2023-10-16T09:56:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.281+0000 7f32d5ffb700 1 -- 172.21.15.158:0/3404109447 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8004280 con 0x7f32e015b590 2023-10-16T09:56:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.282+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e014a0e0 con 0x7f32e015b590 2023-10-16T09:56:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.282+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e00681b0 con 0x7f32e015b590 2023-10-16T09:56:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.283+0000 7f32d5ffb700 1 -- 172.21.15.158:0/3404109447 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f32d80043e0 con 0x7f32e015b590 2023-10-16T09:56:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.283+0000 7f32d5ffb700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32c406b100 0x7f32c406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.283+0000 7f32d5ffb700 1 -- 172.21.15.158:0/3404109447 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f32d8022070 con 0x7f32e015b590 2023-10-16T09:56:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.283+0000 7f32e56eb700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32c406b100 0x7f32c406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.284+0000 7f32e56eb700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32c406b100 0x7f32c406d5c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f32dc007fd0 tx=0x7f32dc009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.288+0000 7f32d5ffb700 1 -- 172.21.15.158:0/3404109447 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32d8056830 con 0x7f32e015b590 2023-10-16T09:56:25.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.503+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32e015fc50 con 0x7f32c406b100 2023-10-16T09:56:25.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.511+0000 7f32d5ffb700 1 -- 172.21.15.158:0/3404109447 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f32dc0014c0 con 0x7f32c406b100 2023-10-16T09:56:25.512 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:25.512 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:02", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:25.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32c406b100 msgr2=0x7f32c406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32c406b100 0x7f32c406d5c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f32dc007fd0 tx=0x7f32dc009040 comp rx=0 tx=0).stop 2023-10-16T09:56:25.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32e015b590 msgr2=0x7f32e00c91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32e015b590 0x7f32e00c91a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f32d800b680 tx=0x7f32d8004ad0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 shutdown_connections 2023-10-16T09:56:25.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32c406b100 0x7f32c406d5c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32e015b590 0x7f32e00c91a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32e00cbaf0 0x7f32e00c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 --2- 172.21.15.158:0/3404109447 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32e00ce510 0x7f32e0148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 >> 172.21.15.158:0/3404109447 conn(0x7f32e006a8d0 msgr2=0x7f32e006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:25.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 shutdown_connections 2023-10-16T09:56:25.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.513+0000 7f32e794f700 1 -- 172.21.15.158:0/3404109447 wait complete. 2023-10-16T09:56:25.526 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:02", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:25.526 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:02', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:25.527 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:25.527 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:25.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:25.528 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T09:56:25.546 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:25.546 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:25.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:25.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:25.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:25.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:25.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:25.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:25.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:25.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:25.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:25.548 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:25.549 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2023-10-16T09:56:25.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.709+0000 7fd6be8ee700 1 Processor -- start 2023-10-16T09:56:25.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.709+0000 7fd6be8ee700 1 -- start start 2023-10-16T09:56:25.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6be8ee700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd6b8160d40 0x7fd6b80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6be8ee700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 0x7fd6b806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6be8ee700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b80ce110 con 0x7fd6b80cf420 2023-10-16T09:56:25.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6be8ee700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b80c61e0 con 0x7fd6b8160d40 2023-10-16T09:56:25.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6be8ee700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80ce3f0 con 0x7fd6b80c50c0 2023-10-16T09:56:25.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd6b80cf420 0x7fd6b80c49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49724/0 (socket says 172.21.15.158:49724) 2023-10-16T09:56:25.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b77fe700 1 -- 172.21.15.158:0/4021887379 learned_addr learned my addr 172.21.15.158:0/4021887379 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:25.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712362716 0 0) 0x7fd6b80ce110 con 0x7fd6b80cf420 2023-10-16T09:56:25.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6bce8b700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 0x7fd6b806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b7fff700 1 --2- 172.21.15.158:0/4021887379 >> v2:172.21.15.172:3300/0 conn(0x7fd6b8160d40 0x7fd6b80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a00035b0 con 0x7fd6b80cf420 2023-10-16T09:56:25.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3744526369 0 0) 0x7fd6a00035b0 con 0x7fd6b80cf420 2023-10-16T09:56:25.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 msgr2=0x7fd6b806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:25.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 0x7fd6b806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 >> v2:172.21.15.172:3300/0 conn(0x7fd6b8160d40 msgr2=0x7fd6b80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:25.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 --2- 172.21.15.158:0/4021887379 >> v2:172.21.15.172:3300/0 conn(0x7fd6b8160d40 0x7fd6b80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80ce6d0 con 0x7fd6b80cf420 2023-10-16T09:56:25.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd6ac001670 con 0x7fd6b80cf420 2023-10-16T09:56:25.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 >> v1:172.21.15.158:6789/0 conn(0x7fd6b80cf420 legacy=0x7fd6b80c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6a0003a00 0x7fd6a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 0x7fd6a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80ce3f0 con 0x7fd6a0004d90 2023-10-16T09:56:25.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a000ba10 con 0x7fd6a0003a00 2023-10-16T09:56:25.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a000bb50 con 0x7fd6b80c50c0 2023-10-16T09:56:25.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.710+0000 7fd6bce8b700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6b7fff700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6a0003a00 0x7fd6a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6b77fe700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 0x7fd6a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6bce8b700 1 -- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 msgr2=0x7fd6a0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:25.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6bce8b700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b80c50c0 0x7fd6a0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6bce8b700 1 -- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6a0003a00 msgr2=0x7fd6a0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:25.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6bce8b700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6a0003a00 0x7fd6a0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6bce8b700 1 -- 172.21.15.158:0/4021887379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a00035b0 con 0x7fd6a0004d90 2023-10-16T09:56:25.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6bce8b700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6a0005210 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009f90 tx=0x7fd6a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30d6017fa0dc9f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:25.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b000ebd0 con 0x7fd6a0004d90 2023-10-16T09:56:25.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b000f040 con 0x7fd6a0004d90 2023-10-16T09:56:25.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6b6ffd700 1 -- 172.21.15.158:0/4021887379 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0014480 con 0x7fd6a0004d90 2023-10-16T09:56:25.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.711+0000 7fd6be8ee700 1 -- 172.21.15.158:0/4021887379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 msgr2=0x7fd6a0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.712+0000 7fd6be8ee700 1 --2- 172.21.15.158:0/4021887379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6a0005210 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009f90 tx=0x7fd6a0005880 comp rx=0 tx=0).stop 2023-10-16T09:56:25.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.712+0000 7fd6bce8b700 1 -- 172.21.15.158:0/4021887379 reap_dead start 2023-10-16T09:56:25.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.712+0000 7fd6be8ee700 1 -- 172.21.15.158:0/4021887379 shutdown_connections 2023-10-16T09:56:25.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.712+0000 7fd6be8ee700 1 -- 172.21.15.158:0/4021887379 >> 172.21.15.158:0/4021887379 conn(0x7fd6b80cc5b0 msgr2=0x7fd6b80cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:25.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.712+0000 7fd6be8ee700 1 -- 172.21.15.158:0/4021887379 shutdown_connections 2023-10-16T09:56:25.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.712+0000 7fd6be8ee700 1 -- 172.21.15.158:0/4021887379 wait complete. 2023-10-16T09:56:25.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.713+0000 7fd6be8ee700 1 Processor -- start 2023-10-16T09:56:25.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.713+0000 7fd6be8ee700 1 -- start start 2023-10-16T09:56:25.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6b81f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b815e900 0x7fd6b80ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6b80710a0 0x7fd6b8071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b8071a90 con 0x7fd6a0004d90 2023-10-16T09:56:25.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b81f6100 con 0x7fd6b80710a0 2023-10-16T09:56:25.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80ae9b0 con 0x7fd6b815e900 2023-10-16T09:56:25.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6b81f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6b81f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49558/0 (socket says 172.21.15.158:49558) 2023-10-16T09:56:25.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 -- 172.21.15.158:0/1233980072 learned_addr learned my addr 172.21.15.158:0/1233980072 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:25.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6bce8b700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6b80710a0 0x7fd6b8071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b77fe700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b815e900 0x7fd6b80ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 -- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b815e900 msgr2=0x7fd6b80ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b815e900 0x7fd6b80ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 -- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6b80710a0 msgr2=0x7fd6b8071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6b80710a0 0x7fd6b8071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 -- 172.21.15.158:0/1233980072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b81612a0 con 0x7fd6a0004d90 2023-10-16T09:56:25.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b7fff700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6b81f5bc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8011aa0 tx=0x7fd6a8011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:25.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b4ff9700 1 -- 172.21.15.158:0/1233980072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a800f940 con 0x7fd6a0004d90 2023-10-16T09:56:25.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b4ff9700 1 -- 172.21.15.158:0/1233980072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a800faa0 con 0x7fd6a0004d90 2023-10-16T09:56:25.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b80b55a0 con 0x7fd6a0004d90 2023-10-16T09:56:25.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6b4ff9700 1 -- 172.21.15.158:0/1233980072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a800fe50 con 0x7fd6a0004d90 2023-10-16T09:56:25.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.714+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b80b4ac0 con 0x7fd6a0004d90 2023-10-16T09:56:25.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.715+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b8066db0 con 0x7fd6a0004d90 2023-10-16T09:56:25.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.716+0000 7fd6b4ff9700 1 -- 172.21.15.158:0/1233980072 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd6a801d400 con 0x7fd6a0004d90 2023-10-16T09:56:25.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.717+0000 7fd6b4ff9700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd6a406b100 0x7fd6a406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:25.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.717+0000 7fd6b77fe700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd6a406b100 0x7fd6a406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:25.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.717+0000 7fd6b4ff9700 1 -- 172.21.15.158:0/1233980072 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd6a808aa10 con 0x7fd6a0004d90 2023-10-16T09:56:25.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.717+0000 7fd6b77fe700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd6a406b100 0x7fd6a406d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6b815ed20 tx=0x7fd6ac009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:25.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.721+0000 7fd6b4ff9700 1 -- 172.21.15.158:0/1233980072 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6a80556d0 con 0x7fd6a0004d90 2023-10-16T09:56:25.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.937+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6b80ce9b0 con 0x7fd6a406b100 2023-10-16T09:56:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.945+0000 7fd6b4ff9700 1 -- 172.21.15.158:0/1233980072 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fd6ac007e20 con 0x7fd6a406b100 2023-10-16T09:56:25.947 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:25.947 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:02", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd6a406b100 msgr2=0x7fd6a406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd6a406b100 0x7fd6a406d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6b815ed20 tx=0x7fd6ac009040 comp rx=0 tx=0).stop 2023-10-16T09:56:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 msgr2=0x7fd6b81f5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6b81f5bc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8011aa0 tx=0x7fd6a8011db0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 shutdown_connections 2023-10-16T09:56:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6b80710a0 0x7fd6b8071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd6a406b100 0x7fd6a406d5c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a0004d90 0x7fd6b81f5bc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 --2- 172.21.15.158:0/1233980072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6b815e900 0x7fd6b80ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:25.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 >> 172.21.15.158:0/1233980072 conn(0x7fd6b80cc5b0 msgr2=0x7fd6b80aebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:25.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 shutdown_connections 2023-10-16T09:56:25.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:25.947+0000 7fd6be8ee700 1 -- 172.21.15.158:0/1233980072 wait complete. 2023-10-16T09:56:25.961 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:02", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:25.961 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:02', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:25.961 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:25.962 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:25.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:25.962 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T09:56:25.981 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:25.981 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:25.981 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:25.981 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:25.981 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:25.982 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:25.982 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:25.982 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:25.982 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:25.982 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:25.983 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:25.983 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:25.983 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2023-10-16T09:56:26.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.142+0000 7f229c8d7700 1 Processor -- start 2023-10-16T09:56:26.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.142+0000 7f229c8d7700 1 -- start start 2023-10-16T09:56:26.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f229c8d7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980b36c0 0x7f229815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f229c8d7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22980d0cd0 0x7f229815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f229c8d7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229815f730 con 0x7f229815c720 2023-10-16T09:56:26.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f229c8d7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229815ffd0 con 0x7f22980d0cd0 2023-10-16T09:56:26.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f229c8d7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229815bab0 con 0x7f22980b36c0 2023-10-16T09:56:26.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f2295d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22980d0cd0 0x7f229815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f2296d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f229815c720 0x7f22980cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49742/0 (socket says 172.21.15.158:49742) 2023-10-16T09:56:26.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.143+0000 7f2296d9d700 1 -- 172.21.15.158:0/3195787590 learned_addr learned my addr 172.21.15.158:0/3195787590 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:26.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229659c700 1 --2- 172.21.15.158:0/3195787590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980b36c0 0x7f229815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f2295d9b700 1 -- 172.21.15.158:0/3195787590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980b36c0 msgr2=0x7f229815f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:26.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f2295d9b700 1 --2- 172.21.15.158:0/3195787590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980b36c0 0x7f229815f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f2295d9b700 1 -- 172.21.15.158:0/3195787590 >> v1:172.21.15.158:6789/0 conn(0x7f229815c720 legacy=0x7f22980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229559a700 1 -- 172.21.15.158:0/3195787590 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282908884 0 0) 0x7f229815f730 con 0x7f229815c720 2023-10-16T09:56:26.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f2295d9b700 1 -- 172.21.15.158:0/3195787590 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229815fa10 con 0x7f22980d0cd0 2023-10-16T09:56:26.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229659c700 1 --2- 172.21.15.158:0/3195787590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980b36c0 0x7f229815f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:26.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f2295d9b700 1 --2- 172.21.15.158:0/3195787590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980d0cd0 0x7f229815c1e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2288009840 tx=0x7f2288009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2509b4ea2c62b2ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:26.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229559a700 1 -- 172.21.15.158:0/3195787590 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288011830 con 0x7f22980d0cd0 2023-10-16T09:56:26.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229559a700 1 -- 172.21.15.158:0/3195787590 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2288011990 con 0x7f22980d0cd0 2023-10-16T09:56:26.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229559a700 1 -- 172.21.15.158:0/3195787590 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288011d20 con 0x7f22980d0cd0 2023-10-16T09:56:26.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229c8d7700 1 -- 172.21.15.158:0/3195787590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980d0cd0 msgr2=0x7f229815c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.144+0000 7f229c8d7700 1 --2- 172.21.15.158:0/3195787590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980d0cd0 0x7f229815c1e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2288009840 tx=0x7f2288009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:26.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.145+0000 7f229c8d7700 1 -- 172.21.15.158:0/3195787590 shutdown_connections 2023-10-16T09:56:26.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.145+0000 7f229c8d7700 1 --2- 172.21.15.158:0/3195787590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980b36c0 0x7f229815f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.145+0000 7f229c8d7700 1 --2- 172.21.15.158:0/3195787590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980d0cd0 0x7f229815c1e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.145+0000 7f229c8d7700 1 -- 172.21.15.158:0/3195787590 >> 172.21.15.158:0/3195787590 conn(0x7f22980cdf90 msgr2=0x7f22980ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:26.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.145+0000 7f229c8d7700 1 -- 172.21.15.158:0/3195787590 shutdown_connections 2023-10-16T09:56:26.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.145+0000 7f229c8d7700 1 -- 172.21.15.158:0/3195787590 wait complete. 2023-10-16T09:56:26.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.146+0000 7f229c8d7700 1 Processor -- start 2023-10-16T09:56:26.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.146+0000 7f229c8d7700 1 -- start start 2023-10-16T09:56:26.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229c8d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980b36c0 0x7f22981f0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229c8d7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980d0cd0 0x7f22980b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229c8d7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22980b14e0 0x7f22980b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229c8d7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22980b1970 con 0x7f22980b36c0 2023-10-16T09:56:26.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229c8d7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22981f0c40 con 0x7f22980b14e0 2023-10-16T09:56:26.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229c8d7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2298149570 con 0x7f22980d0cd0 2023-10-16T09:56:26.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980b36c0 0x7f22981f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980b36c0 0x7f22981f0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49570/0 (socket says 172.21.15.158:49570) 2023-10-16T09:56:26.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.147+0000 7f229659c700 1 -- 172.21.15.158:0/2737523924 learned_addr learned my addr 172.21.15.158:0/2737523924 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:26.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229659c700 1 -- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980d0cd0 msgr2=0x7f22980b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f2295d9b700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980d0cd0 0x7f22980b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f2296d9d700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22980b14e0 0x7f22980b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229659c700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980d0cd0 0x7f22980b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229659c700 1 -- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22980b14e0 msgr2=0x7f22980b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229659c700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22980b14e0 0x7f22980b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229659c700 1 -- 172.21.15.158:0/2737523924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880094a0 con 0x7f22980b36c0 2023-10-16T09:56:26.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229659c700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980b36c0 0x7f22981f0700 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f228c00e820 tx=0x7f228c00ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:26.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f2286ffd700 1 -- 172.21.15.158:0/2737523924 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c005c20 con 0x7f22980b36c0 2023-10-16T09:56:26.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f2286ffd700 1 -- 172.21.15.158:0/2737523924 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c005d80 con 0x7f22980b36c0 2023-10-16T09:56:26.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22980c4ad0 con 0x7f22980b36c0 2023-10-16T09:56:26.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f2286ffd700 1 -- 172.21.15.158:0/2737523924 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c006270 con 0x7f22980b36c0 2023-10-16T09:56:26.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.148+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22980b62f0 con 0x7f22980b36c0 2023-10-16T09:56:26.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.149+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22980681b0 con 0x7f22980b36c0 2023-10-16T09:56:26.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.149+0000 7f2286ffd700 1 -- 172.21.15.158:0/2737523924 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f228c0063d0 con 0x7f22980b36c0 2023-10-16T09:56:26.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.150+0000 7f2286ffd700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f227806b100 0x7f227806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.150+0000 7f2286ffd700 1 -- 172.21.15.158:0/2737523924 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f228c089810 con 0x7f22980b36c0 2023-10-16T09:56:26.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.150+0000 7f2295d9b700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f227806b100 0x7f227806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.150+0000 7f2295d9b700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f227806b100 0x7f227806d5c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f22980b6fa0 tx=0x7f2288008030 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:26.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.154+0000 7f2286ffd700 1 -- 172.21.15.158:0/2737523924 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f228c0542d0 con 0x7f22980b36c0 2023-10-16T09:56:26.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.370+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f229815bcc0 con 0x7f227806b100 2023-10-16T09:56:26.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.378+0000 7f2286ffd700 1 -- 172.21.15.158:0/2737523924 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2288011630 con 0x7f227806b100 2023-10-16T09:56:26.379 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:26.379 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:03", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:26.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f227806b100 msgr2=0x7f227806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f227806b100 0x7f227806d5c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f22980b6fa0 tx=0x7f2288008030 comp rx=0 tx=0).stop 2023-10-16T09:56:26.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980b36c0 msgr2=0x7f22981f0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980b36c0 0x7f22981f0700 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f228c00e820 tx=0x7f228c00ebe0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 shutdown_connections 2023-10-16T09:56:26.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22980b14e0 0x7f22980b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f227806b100 0x7f227806d5c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22980b36c0 0x7f22981f0700 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 --2- 172.21.15.158:0/2737523924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22980d0cd0 0x7f22980b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 >> 172.21.15.158:0/2737523924 conn(0x7f22980cdf90 msgr2=0x7f229815cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:26.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 shutdown_connections 2023-10-16T09:56:26.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.380+0000 7f229c8d7700 1 -- 172.21.15.158:0/2737523924 wait complete. 2023-10-16T09:56:26.393 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:03", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:26.393 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:03', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:26.394 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:26.394 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:26.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:26.395 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T09:56:26.413 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:26.413 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:26.413 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:26.414 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:26.414 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:26.414 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:26.414 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:26.414 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:26.415 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:26.415 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:26.415 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:26.415 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:26.416 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2023-10-16T09:56:26.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.577+0000 7efc0f06c700 1 Processor -- start 2023-10-16T09:56:26.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.577+0000 7efc0f06c700 1 -- start start 2023-10-16T09:56:26.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0f06c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efc0815db50 0x7efc080cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0f06c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 0x7efc080cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0f06c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc080caa80 con 0x7efc0815d770 2023-10-16T09:56:26.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0f06c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc080cad60 con 0x7efc0815db50 2023-10-16T09:56:26.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0f06c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080cb040 con 0x7efc080cf680 2023-10-16T09:56:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0ce08700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efc0815d770 0x7efc080cd7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49754/0 (socket says 172.21.15.158:49754) 2023-10-16T09:56:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0ce08700 1 -- 172.21.15.158:0/83448536 learned_addr learned my addr 172.21.15.158:0/83448536 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc0d609700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 0x7efc080cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc07fff700 1 --2- 172.21.15.158:0/83448536 >> v2:172.21.15.172:3300/0 conn(0x7efc0815db50 0x7efc080cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876695240 0 0) 0x7efc080caa80 con 0x7efc0815d770 2023-10-16T09:56:26.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.578+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf80035b0 con 0x7efc0815d770 2023-10-16T09:56:26.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001635762 0 0) 0x7efbf80035b0 con 0x7efc0815d770 2023-10-16T09:56:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 msgr2=0x7efc080cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 0x7efc080cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 >> v2:172.21.15.172:3300/0 conn(0x7efc0815db50 msgr2=0x7efc080cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 --2- 172.21.15.158:0/83448536 >> v2:172.21.15.172:3300/0 conn(0x7efc0815db50 0x7efc080cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080cb320 con 0x7efc0815d770 2023-10-16T09:56:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7efbfc001670 con 0x7efc0815d770 2023-10-16T09:56:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 >> v1:172.21.15.158:6789/0 conn(0x7efc0815d770 legacy=0x7efc080cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efbf8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 0x7efbf8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efbf8004d90 0x7efbf8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080cad60 con 0x7efbf8003a00 2023-10-16T09:56:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080cb040 con 0x7efbf8004d90 2023-10-16T09:56:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf800ba10 con 0x7efc080cf680 2023-10-16T09:56:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0ce08700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efbf8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc07fff700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 0x7efbf8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0d609700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efbf8004d90 0x7efbf8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0ce08700 1 -- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 msgr2=0x7efbf8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:26.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0ce08700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc080cf680 0x7efbf8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0ce08700 1 -- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efbf8004d90 msgr2=0x7efbf8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:26.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0ce08700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efbf8004d90 0x7efbf8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0ce08700 1 -- 172.21.15.158:0/83448536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf80035b0 con 0x7efbf8003a00 2023-10-16T09:56:26.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0d609700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efbf8004d90 0x7efbf8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:26.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc0ce08700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efbf8003e20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c2f0 tx=0x7efbf8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eeef7d3d9e6d951 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:26.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00ce30 con 0x7efbf8003a00 2023-10-16T09:56:26.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc005960 con 0x7efbf8003a00 2023-10-16T09:56:26.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.579+0000 7efc077fe700 1 -- 172.21.15.158:0/83448536 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc005d10 con 0x7efbf8003a00 2023-10-16T09:56:26.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.580+0000 7efc0f06c700 1 -- 172.21.15.158:0/83448536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 msgr2=0x7efbf8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.580+0000 7efc0f06c700 1 --2- 172.21.15.158:0/83448536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efbf8003e20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c2f0 tx=0x7efbf8005880 comp rx=0 tx=0).stop 2023-10-16T09:56:26.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.580+0000 7efc0d609700 1 -- 172.21.15.158:0/83448536 reap_dead start 2023-10-16T09:56:26.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.580+0000 7efc0f06c700 1 -- 172.21.15.158:0/83448536 shutdown_connections 2023-10-16T09:56:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.580+0000 7efc0f06c700 1 -- 172.21.15.158:0/83448536 >> 172.21.15.158:0/83448536 conn(0x7efc08099a10 msgr2=0x7efc080d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.580+0000 7efc0f06c700 1 -- 172.21.15.158:0/83448536 shutdown_connections 2023-10-16T09:56:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.580+0000 7efc0f06c700 1 -- 172.21.15.158:0/83448536 wait complete. 2023-10-16T09:56:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.581+0000 7efc0f06c700 1 Processor -- start 2023-10-16T09:56:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.581+0000 7efc0f06c700 1 -- start start 2023-10-16T09:56:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0f06c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efc08152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0f06c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc08160290 0x7efc081557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0f06c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc08160710 0x7efc08155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0f06c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08160ca0 con 0x7efbf8003a00 2023-10-16T09:56:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0f06c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080c8810 con 0x7efc08160290 2023-10-16T09:56:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0f06c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080c8af0 con 0x7efc08160710 2023-10-16T09:56:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efc08152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efc08152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49590/0 (socket says 172.21.15.158:49590) 2023-10-16T09:56:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 -- 172.21.15.158:0/139233970 learned_addr learned my addr 172.21.15.158:0/139233970 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 -- 172.21.15.158:0/139233970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc08160710 msgr2=0x7efc08155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc08160710 0x7efc08155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 -- 172.21.15.158:0/139233970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc08160290 msgr2=0x7efc081557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc08160290 0x7efc081557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.582+0000 7efc0ce08700 1 -- 172.21.15.158:0/139233970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080c8dd0 con 0x7efbf8003a00 2023-10-16T09:56:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.583+0000 7efc0ce08700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efc08152960 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7efbfc008fd0 tx=0x7efc080ce670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.583+0000 7efc057fa700 1 -- 172.21.15.158:0/139233970 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00cea0 con 0x7efbf8003a00 2023-10-16T09:56:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.583+0000 7efc057fa700 1 -- 172.21.15.158:0/139233970 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc004a80 con 0x7efbf8003a00 2023-10-16T09:56:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.583+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc080c90b0 con 0x7efbf8003a00 2023-10-16T09:56:26.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.583+0000 7efc057fa700 1 -- 172.21.15.158:0/139233970 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc004df0 con 0x7efbf8003a00 2023-10-16T09:56:26.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.583+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0815f4f0 con 0x7efbf8003a00 2023-10-16T09:56:26.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.584+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc080681b0 con 0x7efbf8003a00 2023-10-16T09:56:26.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.584+0000 7efc057fa700 1 -- 172.21.15.158:0/139233970 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7efbfc0070d0 con 0x7efbf8003a00 2023-10-16T09:56:26.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.584+0000 7efc057fa700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efbec06b100 0x7efbec06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:26.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.585+0000 7efc057fa700 1 -- 172.21.15.158:0/139233970 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7efbfc08aef0 con 0x7efbf8003a00 2023-10-16T09:56:26.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.585+0000 7efc07fff700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efbec06b100 0x7efbec06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:26.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.585+0000 7efc07fff700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efbec06b100 0x7efbec06d5c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efbf4006f10 tx=0x7efbf4006e80 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:26.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.589+0000 7efc057fa700 1 -- 172.21.15.158:0/139233970 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efbfc055c10 con 0x7efbf8003a00 2023-10-16T09:56:26.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.806+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc080cb8e0 con 0x7efbec06b100 2023-10-16T09:56:26.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.815+0000 7efc057fa700 1 -- 172.21.15.158:0/139233970 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7efbf4006c60 con 0x7efbec06b100 2023-10-16T09:56:26.816 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:26.816 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:03", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:26.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.817+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efbec06b100 msgr2=0x7efbec06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.817+0000 7efc0f06c700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efbec06b100 0x7efbec06d5c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efbf4006f10 tx=0x7efbf4006e80 comp rx=0 tx=0).stop 2023-10-16T09:56:26.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.817+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 msgr2=0x7efc08152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:26.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.817+0000 7efc0f06c700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efc08152960 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7efbfc008fd0 tx=0x7efc080ce670 comp rx=0 tx=0).stop 2023-10-16T09:56:26.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 shutdown_connections 2023-10-16T09:56:26.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc08160710 0x7efc08155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7efbec06b100 0x7efbec06d5c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efbf8003a00 0x7efc08152960 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 --2- 172.21.15.158:0/139233970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc08160290 0x7efc081557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:26.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 >> 172.21.15.158:0/139233970 conn(0x7efc08099a10 msgr2=0x7efc080c4130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:26.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 shutdown_connections 2023-10-16T09:56:26.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:26.818+0000 7efc0f06c700 1 -- 172.21.15.158:0/139233970 wait complete. 2023-10-16T09:56:26.831 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:03", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:26.831 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:03', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:26.831 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:26.832 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:26.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:26.832 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T09:56:26.850 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:26.850 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:26.850 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:26.850 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:26.851 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:26.851 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:26.851 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:26.851 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:26.851 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:26.852 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:26.852 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:26.853 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:26.853 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2023-10-16T09:56:27.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.014+0000 7fb7f3cfb700 1 Processor -- start 2023-10-16T09:56:27.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.014+0000 7fb7f3cfb700 1 -- start start 2023-10-16T09:56:27.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.015+0000 7fb7f3cfb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.015+0000 7fb7f3cfb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec159660 con 0x7fb7ec159280 2023-10-16T09:56:27.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.015+0000 7fb7f3cfb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec06b120 con 0x7fb7ec0cbef0 2023-10-16T09:56:27.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.015+0000 7fb7f3cfb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0cb710 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.015+0000 7fb7f1a97700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb7ec159280 0x7fb7ec06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49774/0 (socket says 172.21.15.158:49774) 2023-10-16T09:56:27.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.015+0000 7fb7f1a97700 1 -- 172.21.15.158:0/2295209261 learned_addr learned my addr 172.21.15.158:0/2295209261 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:27.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.015+0000 7fb7f1296700 1 --2- 172.21.15.158:0/2295209261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f0a95700 1 -- 172.21.15.158:0/2295209261 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688609418 0 0) 0x7fb7ec159660 con 0x7fb7ec159280 2023-10-16T09:56:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f0a95700 1 -- 172.21.15.158:0/2295209261 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d80035b0 con 0x7fb7ec159280 2023-10-16T09:56:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f0a95700 1 -- 172.21.15.158:0/2295209261 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072723880 0 0) 0x7fb7ec06b120 con 0x7fb7ec0cbef0 2023-10-16T09:56:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f0a95700 1 -- 172.21.15.158:0/2295209261 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ec159660 con 0x7fb7ec0cbef0 2023-10-16T09:56:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f1296700 1 -- 172.21.15.158:0/2295209261 >> v1:172.21.15.172:6789/0 conn(0x7fb7ec0cbef0 legacy=0x7fb7ec0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f1296700 1 -- 172.21.15.158:0/2295209261 >> v1:172.21.15.158:6789/0 conn(0x7fb7ec159280 legacy=0x7fb7ec06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f1296700 1 -- 172.21.15.158:0/2295209261 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec06b400 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f1296700 1 --2- 172.21.15.158:0/2295209261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009870 tx=0x7fb7e8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc38d261fc29766 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f0a95700 1 -- 172.21.15.158:0/2295209261 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8011830 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f0a95700 1 -- 172.21.15.158:0/2295209261 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e8011990 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.016+0000 7fb7f0a95700 1 -- 172.21.15.158:0/2295209261 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8011d20 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.017+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/2295209261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec0d0e10 msgr2=0x7fb7ec15c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.017+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/2295209261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009870 tx=0x7fb7e8009b80 comp rx=0 tx=0).stop 2023-10-16T09:56:27.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.017+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/2295209261 shutdown_connections 2023-10-16T09:56:27.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.017+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/2295209261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.017+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/2295209261 >> 172.21.15.158:0/2295209261 conn(0x7fb7ec099a10 msgr2=0x7fb7ec0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:27.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.017+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/2295209261 shutdown_connections 2023-10-16T09:56:27.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.017+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/2295209261 wait complete. 2023-10-16T09:56:27.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.018+0000 7fb7f3cfb700 1 Processor -- start 2023-10-16T09:56:27.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.018+0000 7fb7f3cfb700 1 -- start start 2023-10-16T09:56:27.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f3cfb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7ec0cbef0 0x7fb7ec158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f3cfb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f3cfb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec159280 0x7fb7ec15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f3cfb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec15d030 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f3cfb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec15d310 con 0x7fb7ec0cbef0 2023-10-16T09:56:27.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f3cfb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec15d5f0 con 0x7fb7ec159280 2023-10-16T09:56:27.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49606/0 (socket says 172.21.15.158:49606) 2023-10-16T09:56:27.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 -- 172.21.15.158:0/1399554759 learned_addr learned my addr 172.21.15.158:0/1399554759 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:27.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1a97700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7ec0cbef0 0x7fb7ec158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f2298700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec159280 0x7fb7ec15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 -- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec159280 msgr2=0x7fb7ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec159280 0x7fb7ec15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 -- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7ec0cbef0 msgr2=0x7fb7ec158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7ec0cbef0 0x7fb7ec158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 -- 172.21.15.158:0/1399554759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80094a0 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7f1296700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80090b0 tx=0x7fb7e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.019+0000 7fb7e27fc700 1 -- 172.21.15.158:0/1399554759 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e801a760 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.020+0000 7fb7e27fc700 1 -- 172.21.15.158:0/1399554759 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e801a8c0 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.020+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec15d8d0 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.020+0000 7fb7e27fc700 1 -- 172.21.15.158:0/1399554759 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e801ac70 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.020+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec158dc0 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.020+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec0681b0 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.021+0000 7fb7e27fc700 1 -- 172.21.15.158:0/1399554759 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fb7e8022070 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.021+0000 7fb7e27fc700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb7d006b100 0x7fb7d006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.021+0000 7fb7f1a97700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb7d006b100 0x7fb7d006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.022+0000 7fb7e27fc700 1 -- 172.21.15.158:0/1399554759 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fb7e808a870 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.022+0000 7fb7f1a97700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb7d006b100 0x7fb7d006d5c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc007fd0 tx=0x7fb7dc007e00 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.026+0000 7fb7e27fc700 1 -- 172.21.15.158:0/1399554759 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb7e8055450 con 0x7fb7ec0d0e10 2023-10-16T09:56:27.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.243+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7ec15bd90 con 0x7fb7d006b100 2023-10-16T09:56:27.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.251+0000 7fb7e27fc700 1 -- 172.21.15.158:0/1399554759 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fb7dc006bb0 con 0x7fb7d006b100 2023-10-16T09:56:27.252 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:27.252 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:04", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:27.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb7d006b100 msgr2=0x7fb7d006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb7d006b100 0x7fb7d006d5c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc007fd0 tx=0x7fb7dc007e00 comp rx=0 tx=0).stop 2023-10-16T09:56:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7ec0d0e10 msgr2=0x7fb7ec15c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80090b0 tx=0x7fb7e8004750 comp rx=0 tx=0).stop 2023-10-16T09:56:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 shutdown_connections 2023-10-16T09:56:27.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7ec159280 0x7fb7ec15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7ec0cbef0 0x7fb7ec158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb7d006b100 0x7fb7d006d5c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 --2- 172.21.15.158:0/1399554759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7ec0d0e10 0x7fb7ec15c590 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 >> 172.21.15.158:0/1399554759 conn(0x7fb7ec099a10 msgr2=0x7fb7ec06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:27.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 shutdown_connections 2023-10-16T09:56:27.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.253+0000 7fb7f3cfb700 1 -- 172.21.15.158:0/1399554759 wait complete. 2023-10-16T09:56:27.266 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:04", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:27.266 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:04', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:27.267 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:27.267 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:27.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:27.268 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T09:56:27.286 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:27.286 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:27.286 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:27.287 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:27.287 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:27.287 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:27.287 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:27.287 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:27.288 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:27.288 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:27.288 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:27.288 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:27.289 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2023-10-16T09:56:27.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.447+0000 7fd08efc1700 1 Processor -- start 2023-10-16T09:56:27.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.447+0000 7fd08efc1700 1 -- start start 2023-10-16T09:56:27.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd08efc1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 0x7fd08806b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd08efc1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880c52f0 con 0x7fd088160d40 2023-10-16T09:56:27.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd08efc1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880ce110 con 0x7fd0880c49b0 2023-10-16T09:56:27.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd08efc1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880c61e0 con 0x7fd0880c4e60 2023-10-16T09:56:27.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd08cd5d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd088160d40 0x7fd0880ceee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49788/0 (socket says 172.21.15.158:49788) 2023-10-16T09:56:27.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd08cd5d700 1 -- 172.21.15.158:0/931662850 learned_addr learned my addr 172.21.15.158:0/931662850 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:27.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930788482 0 0) 0x7fd0880c52f0 con 0x7fd088160d40 2023-10-16T09:56:27.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0780035b0 con 0x7fd088160d40 2023-10-16T09:56:27.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd08d55e700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 0x7fd08806b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3514059861 0 0) 0x7fd0780035b0 con 0x7fd088160d40 2023-10-16T09:56:27.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 msgr2=0x7fd08806b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:27.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 0x7fd08806b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 >> v1:172.21.15.172:6789/0 conn(0x7fd0880c49b0 legacy=0x7fd08806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880ce3f0 con 0x7fd088160d40 2023-10-16T09:56:27.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd07c001670 con 0x7fd088160d40 2023-10-16T09:56:27.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 >> v1:172.21.15.158:6789/0 conn(0x7fd088160d40 legacy=0x7fd0880ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 0x7fd078003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd078003fb0 0x7fd078004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd078004d90 0x7fd078005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07800bac0 con 0x7fd078004d90 2023-10-16T09:56:27.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07800bc30 con 0x7fd078003fb0 2023-10-16T09:56:27.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.448+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07800bdf0 con 0x7fd0880c4e60 2023-10-16T09:56:27.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08d55e700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd078004d90 0x7fd078005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08cd5d700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 0x7fd078003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd087fff700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd078003fb0 0x7fd078004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08d55e700 1 -- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 msgr2=0x7fd078003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:27.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08d55e700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0880c4e60 0x7fd078003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08d55e700 1 -- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd078003fb0 msgr2=0x7fd078004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:27.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08d55e700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd078003fb0 0x7fd078004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08d55e700 1 -- 172.21.15.158:0/931662850 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0780035b0 con 0x7fd078004d90 2023-10-16T09:56:27.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd087fff700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd078003fb0 0x7fd078004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:27.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd08d55e700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd078004d90 0x7fd078005210 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd080009fc0 tx=0x7fd078005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4580c3fffb85f5e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08000ec30 con 0x7fd078004d90 2023-10-16T09:56:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08000f040 con 0x7fd078004d90 2023-10-16T09:56:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.449+0000 7fd0877fe700 1 -- 172.21.15.158:0/931662850 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080014480 con 0x7fd078004d90 2023-10-16T09:56:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.450+0000 7fd08efc1700 1 -- 172.21.15.158:0/931662850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd078004d90 msgr2=0x7fd078005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.450+0000 7fd08efc1700 1 --2- 172.21.15.158:0/931662850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd078004d90 0x7fd078005210 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd080009fc0 tx=0x7fd078005880 comp rx=0 tx=0).stop 2023-10-16T09:56:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.450+0000 7fd08d55e700 1 -- 172.21.15.158:0/931662850 reap_dead start 2023-10-16T09:56:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.450+0000 7fd08efc1700 1 -- 172.21.15.158:0/931662850 shutdown_connections 2023-10-16T09:56:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.450+0000 7fd08efc1700 1 -- 172.21.15.158:0/931662850 >> 172.21.15.158:0/931662850 conn(0x7fd0880cc5b0 msgr2=0x7fd0880cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.450+0000 7fd08efc1700 1 -- 172.21.15.158:0/931662850 shutdown_connections 2023-10-16T09:56:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.450+0000 7fd08efc1700 1 -- 172.21.15.158:0/931662850 wait complete. 2023-10-16T09:56:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.451+0000 7fd08efc1700 1 Processor -- start 2023-10-16T09:56:27.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.451+0000 7fd08efc1700 1 -- start start 2023-10-16T09:56:27.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08efc1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd078004d90 0x7fd0881564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08efc1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0880c6e80 0x7fd0880c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08efc1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd088155380 0x7fd0880cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08efc1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08815dc30 con 0x7fd0880c6e80 2023-10-16T09:56:27.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08efc1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880c7e20 con 0x7fd088155380 2023-10-16T09:56:27.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08efc1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880c9ae0 con 0x7fd078004d90 2023-10-16T09:56:27.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0880c6e80 0x7fd0880c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0880c6e80 0x7fd0880c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49626/0 (socket says 172.21.15.158:49626) 2023-10-16T09:56:27.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 -- 172.21.15.158:0/2519784958 learned_addr learned my addr 172.21.15.158:0/2519784958 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:27.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08d55e700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd088155380 0x7fd0880cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd08cd5d700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd078004d90 0x7fd0881564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 -- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd078004d90 msgr2=0x7fd0881564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd078004d90 0x7fd0881564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 -- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd088155380 msgr2=0x7fd0880cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd088155380 0x7fd0880cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 -- 172.21.15.158:0/2519784958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0881569f0 con 0x7fd0880c6e80 2023-10-16T09:56:27.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd087fff700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0880c6e80 0x7fd0880c75f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e7b0 tx=0x7fd07400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.452+0000 7fd0857fa700 1 -- 172.21.15.158:0/2519784958 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740052e0 con 0x7fd0880c6e80 2023-10-16T09:56:27.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.453+0000 7fd0857fa700 1 -- 172.21.15.158:0/2519784958 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074005440 con 0x7fd0880c6e80 2023-10-16T09:56:27.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.453+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0880c9520 con 0x7fd0880c6e80 2023-10-16T09:56:27.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.453+0000 7fd0857fa700 1 -- 172.21.15.158:0/2519784958 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740038c0 con 0x7fd0880c6e80 2023-10-16T09:56:27.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.453+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0880c7b30 con 0x7fd0880c6e80 2023-10-16T09:56:27.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.453+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08815dd70 con 0x7fd0880c6e80 2023-10-16T09:56:27.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.454+0000 7fd0857fa700 1 -- 172.21.15.158:0/2519784958 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd074003a20 con 0x7fd0880c6e80 2023-10-16T09:56:27.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.454+0000 7fd0857fa700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd06c06b130 0x7fd06c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.455+0000 7fd08cd5d700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd06c06b130 0x7fd06c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.455+0000 7fd0857fa700 1 -- 172.21.15.158:0/2519784958 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd074089090 con 0x7fd0880c6e80 2023-10-16T09:56:27.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.455+0000 7fd08cd5d700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd06c06b130 0x7fd06c06d5f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd07c001050 tx=0x7fd07c000f80 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.459+0000 7fd0857fa700 1 -- 172.21.15.158:0/2519784958 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd08815dd70 con 0x7fd0880c6e80 2023-10-16T09:56:27.615 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:56:27.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:56:27.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.682+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0880ce6d0 con 0x7fd06c06b130 2023-10-16T09:56:27.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.690+0000 7fd0857fa700 1 -- 172.21.15.158:0/2519784958 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fd07c0070c0 con 0x7fd06c06b130 2023-10-16T09:56:27.692 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:27.692 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:04", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:27.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd06c06b130 msgr2=0x7fd06c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd06c06b130 0x7fd06c06d5f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd07c001050 tx=0x7fd07c000f80 comp rx=0 tx=0).stop 2023-10-16T09:56:27.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0880c6e80 msgr2=0x7fd0880c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0880c6e80 0x7fd0880c75f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e7b0 tx=0x7fd07400eac0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 shutdown_connections 2023-10-16T09:56:27.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd088155380 0x7fd0880cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd078004d90 0x7fd0881564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd06c06b130 0x7fd06c06d5f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 --2- 172.21.15.158:0/2519784958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0880c6e80 0x7fd0880c75f0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.692+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 >> 172.21.15.158:0/2519784958 conn(0x7fd0880cc5b0 msgr2=0x7fd08806b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:27.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.693+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 shutdown_connections 2023-10-16T09:56:27.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.693+0000 7fd08efc1700 1 -- 172.21.15.158:0/2519784958 wait complete. 2023-10-16T09:56:27.705 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:04", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:27.706 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:04', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:27.706 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:27.707 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:27.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:27.707 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T09:56:27.764 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:27.764 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:27.765 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:27.765 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:27.765 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:27.765 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:27.766 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:27.766 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:27.766 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:27.766 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:27.766 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:27.767 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-09_46_00_UTC', 'scheduled-2023-10-16-09_47_00_UTC', 'scheduled-2023-10-16-09_48_00_UTC', 'scheduled-2023-10-16-09_49_00_UTC', 'scheduled-2023-10-16-09_50_00_UTC', 'scheduled-2023-10-16-09_51_00_UTC', 'scheduled-2023-10-16-09_52_00_UTC', 'scheduled-2023-10-16-09_53_00_UTC', 'scheduled-2023-10-16-09_54_00_UTC', 'scheduled-2023-10-16-09_55_00_UTC', 'scheduled-2023-10-16-09_56_00_UTC'] 2023-10-16T09:56:27.767 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2023-10-16T09:56:27.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.925+0000 7ffb7b4c6700 1 Processor -- start 2023-10-16T09:56:27.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.925+0000 7ffb7b4c6700 1 -- start start 2023-10-16T09:56:27.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb7b4c6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb7406afe0 0x7ffb740caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb7b4c6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffb740cc590 0x7ffb740afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb7b4c6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740cbd30 con 0x7ffb7415da40 2023-10-16T09:56:27.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb7b4c6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb740cc1c0 con 0x7ffb740cc590 2023-10-16T09:56:27.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb7b4c6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb7415de20 con 0x7ffb7406afe0 2023-10-16T09:56:27.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb79a63700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffb740cc590 0x7ffb740afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb79a63700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffb740cc590 0x7ffb740afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49636/0 (socket says 172.21.15.158:49636) 2023-10-16T09:56:27.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.926+0000 7ffb79a63700 1 -- 172.21.15.158:0/1431878062 learned_addr learned my addr 172.21.15.158:0/1431878062 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:27.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb78a61700 1 --2- 172.21.15.158:0/1431878062 >> v2:172.21.15.172:3300/0 conn(0x7ffb7406afe0 0x7ffb740caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb79a63700 1 -- 172.21.15.158:0/1431878062 >> v2:172.21.15.172:3300/0 conn(0x7ffb7406afe0 msgr2=0x7ffb740caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:27.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb79a63700 1 --2- 172.21.15.158:0/1431878062 >> v2:172.21.15.172:3300/0 conn(0x7ffb7406afe0 0x7ffb740caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb79a63700 1 -- 172.21.15.158:0/1431878062 >> v1:172.21.15.172:6789/0 conn(0x7ffb7415da40 legacy=0x7ffb7406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:27.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb79a63700 1 -- 172.21.15.158:0/1431878062 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb740ce260 con 0x7ffb740cc590 2023-10-16T09:56:27.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb79a63700 1 --2- 172.21.15.158:0/1431878062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740afa40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ffb740729e0 tx=0x7ffb64009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e6481909770403 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb73fff700 1 -- 172.21.15.158:0/1431878062 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64004230 con 0x7ffb740cc590 2023-10-16T09:56:27.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb73fff700 1 -- 172.21.15.158:0/1431878062 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64004390 con 0x7ffb740cc590 2023-10-16T09:56:27.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb73fff700 1 -- 172.21.15.158:0/1431878062 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640036a0 con 0x7ffb740cc590 2023-10-16T09:56:27.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1431878062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 msgr2=0x7ffb740afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.927+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1431878062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740afa40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ffb740729e0 tx=0x7ffb64009bd0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.928+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1431878062 shutdown_connections 2023-10-16T09:56:27.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.928+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1431878062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740afa40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.928+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1431878062 >> v2:172.21.15.172:3300/0 conn(0x7ffb7406afe0 0x7ffb740caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.928+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1431878062 >> 172.21.15.158:0/1431878062 conn(0x7ffb74099a10 msgr2=0x7ffb740d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:27.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.928+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1431878062 shutdown_connections 2023-10-16T09:56:27.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.928+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1431878062 wait complete. 2023-10-16T09:56:27.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.929+0000 7ffb7b4c6700 1 Processor -- start 2023-10-16T09:56:27.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.929+0000 7ffb7b4c6700 1 -- start start 2023-10-16T09:56:27.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb7b4c6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb7406afe0 0x7ffb741508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb7b4c6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb7b4c6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb7415da40 0x7ffb740ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb7b4c6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb74150e20 con 0x7ffb740cc590 2023-10-16T09:56:27.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb7b4c6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740c8f60 con 0x7ffb7415da40 2023-10-16T09:56:27.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb7b4c6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740c8c80 con 0x7ffb7406afe0 2023-10-16T09:56:27.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740c8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49640/0 (socket says 172.21.15.158:49640) 2023-10-16T09:56:27.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 -- 172.21.15.158:0/1727712972 learned_addr learned my addr 172.21.15.158:0/1727712972 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:27.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb79a63700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb7415da40 0x7ffb740ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb79262700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb7406afe0 0x7ffb741508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 -- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb7406afe0 msgr2=0x7ffb741508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb7406afe0 0x7ffb741508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 -- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb7415da40 msgr2=0x7ffb740ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:27.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb7415da40 0x7ffb740ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:27.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 -- 172.21.15.158:0/1727712972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb64009550 con 0x7ffb740cc590 2023-10-16T09:56:27.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb79262700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb7406afe0 0x7ffb741508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:27.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb78a61700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740c8200 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800e810 tx=0x7ffb6800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb71ffb700 1 -- 172.21.15.158:0/1727712972 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68013610 con 0x7ffb740cc590 2023-10-16T09:56:27.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb71ffb700 1 -- 172.21.15.158:0/1727712972 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb68013770 con 0x7ffb740cc590 2023-10-16T09:56:27.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb741f6370 con 0x7ffb740cc590 2023-10-16T09:56:27.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.930+0000 7ffb71ffb700 1 -- 172.21.15.158:0/1727712972 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68013b20 con 0x7ffb740cc590 2023-10-16T09:56:27.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.931+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb741f6930 con 0x7ffb740cc590 2023-10-16T09:56:27.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.931+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb740681b0 con 0x7ffb740cc590 2023-10-16T09:56:27.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.932+0000 7ffb71ffb700 1 -- 172.21.15.158:0/1727712972 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ffb68013c80 con 0x7ffb740cc590 2023-10-16T09:56:27.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.932+0000 7ffb71ffb700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb6006b100 0x7ffb6006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:27.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.932+0000 7ffb71ffb700 1 -- 172.21.15.158:0/1727712972 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7ffb68089920 con 0x7ffb740cc590 2023-10-16T09:56:27.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.932+0000 7ffb79262700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb6006b100 0x7ffb6006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:27.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.933+0000 7ffb79262700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb6006b100 0x7ffb6006d5c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009fc0 tx=0x7ffb6c009550 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:27.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:27.937+0000 7ffb71ffb700 1 -- 172.21.15.158:0/1727712972 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb680543f0 con 0x7ffb740cc590 2023-10-16T09:56:28.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.154+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb740cb400 con 0x7ffb6006b100 2023-10-16T09:56:28.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.162+0000 7ffb71ffb700 1 -- 172.21.15.158:0/1727712972 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7ffb6c009bb0 con 0x7ffb6006b100 2023-10-16T09:56:28.164 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:28.164 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:05", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:28.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb6006b100 msgr2=0x7ffb6006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb6006b100 0x7ffb6006d5c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009fc0 tx=0x7ffb6c009550 comp rx=0 tx=0).stop 2023-10-16T09:56:28.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 msgr2=0x7ffb740c8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740c8200 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800e810 tx=0x7ffb6800eb20 comp rx=0 tx=0).stop 2023-10-16T09:56:28.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 shutdown_connections 2023-10-16T09:56:28.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb7415da40 0x7ffb740ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb7406afe0 0x7ffb741508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ffb6006b100 0x7ffb6006d5c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 --2- 172.21.15.158:0/1727712972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb740cc590 0x7ffb740c8200 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 >> 172.21.15.158:0/1727712972 conn(0x7ffb74099a10 msgr2=0x7ffb740b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:28.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 shutdown_connections 2023-10-16T09:56:28.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.165+0000 7ffb7b4c6700 1 -- 172.21.15.158:0/1727712972 wait complete. 2023-10-16T09:56:28.178 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T09:45:05", "first": "2023-10-16T09:46:00", "last": "2023-10-16T09:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T09:56:28.179 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T09:45:05', 'first': '2023-10-16T09:46:00', 'last': '2023-10-16T09:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T09:56:28.179 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T09:56:28.179 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T09:56:28.180 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2023-10-16T09:56:28.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.302+0000 7fc2bd9b3700 1 Processor -- start 2023-10-16T09:56:28.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2bd9b3700 1 -- start start 2023-10-16T09:56:28.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2bd9b3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc2b806afe0 0x7fc2b80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:28.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2bd9b3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b80cbd30 con 0x7fc2b80cc590 2023-10-16T09:56:28.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2bd9b3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b80cc1c0 con 0x7fc2b815da40 2023-10-16T09:56:28.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2bd9b3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b815de20 con 0x7fc2b806afe0 2023-10-16T09:56:28.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2b67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc2b806afe0 0x7fc2b80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:28.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2b77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc2b80cc590 0x7fc2b80afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49826/0 (socket says 172.21.15.158:49826) 2023-10-16T09:56:28.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.303+0000 7fc2b77fe700 1 -- 172.21.15.158:0/2654920712 learned_addr learned my addr 172.21.15.158:0/2654920712 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:28.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b67fc700 1 -- 172.21.15.158:0/2654920712 >> v1:172.21.15.172:6789/0 conn(0x7fc2b815da40 legacy=0x7fc2b806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:28.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b67fc700 1 -- 172.21.15.158:0/2654920712 >> v1:172.21.15.158:6789/0 conn(0x7fc2b80cc590 legacy=0x7fc2b80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b67fc700 1 -- 172.21.15.158:0/2654920712 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b80ce260 con 0x7fc2b806afe0 2023-10-16T09:56:28.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b5ffb700 1 -- 172.21.15.158:0/2654920712 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835042616 0 0) 0x7fc2b80cbd30 con 0x7fc2b80cc590 2023-10-16T09:56:28.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b67fc700 1 --2- 172.21.15.158:0/2654920712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b806afe0 0x7fc2b80caa20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009840 tx=0x7fc2a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1197f7aad9aa6c21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:28.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b5ffb700 1 -- 172.21.15.158:0/2654920712 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8004230 con 0x7fc2b806afe0 2023-10-16T09:56:28.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b5ffb700 1 -- 172.21.15.158:0/2654920712 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a8004390 con 0x7fc2b806afe0 2023-10-16T09:56:28.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.304+0000 7fc2b5ffb700 1 -- 172.21.15.158:0/2654920712 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80036a0 con 0x7fc2b806afe0 2023-10-16T09:56:28.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.305+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/2654920712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b806afe0 msgr2=0x7fc2b80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.305+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/2654920712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b806afe0 0x7fc2b80caa20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009840 tx=0x7fc2a8009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:28.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.305+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/2654920712 shutdown_connections 2023-10-16T09:56:28.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.305+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/2654920712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b806afe0 0x7fc2b80caa20 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.305+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/2654920712 >> 172.21.15.158:0/2654920712 conn(0x7fc2b8099a10 msgr2=0x7fc2b80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:28.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.305+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/2654920712 shutdown_connections 2023-10-16T09:56:28.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.305+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/2654920712 wait complete. 2023-10-16T09:56:28.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.306+0000 7fc2bd9b3700 1 Processor -- start 2023-10-16T09:56:28.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.307+0000 7fc2bd9b3700 1 -- start start 2023-10-16T09:56:28.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2bd9b3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b806afe0 0x7fc2b8159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:28.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2bd9b3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b80cc590 0x7fc2b80c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:28.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2bd9b3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b815da40 0x7fc2b80c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:28.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2bd9b3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b815ab30 con 0x7fc2b815da40 2023-10-16T09:56:28.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2bd9b3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80c5260 con 0x7fc2b80cc590 2023-10-16T09:56:28.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2bd9b3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b8159fb0 con 0x7fc2b806afe0 2023-10-16T09:56:28.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b815da40 0x7fc2b80c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:28.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b815da40 0x7fc2b80c99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49656/0 (socket says 172.21.15.158:49656) 2023-10-16T09:56:28.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 -- 172.21.15.158:0/3707010019 learned_addr learned my addr 172.21.15.158:0/3707010019 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:28.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b67fc700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b80cc590 0x7fc2b80c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:28.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b6ffd700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b806afe0 0x7fc2b8159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:28.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 -- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b806afe0 msgr2=0x7fc2b8159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b806afe0 0x7fc2b8159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 -- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b80cc590 msgr2=0x7fc2b80c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b80cc590 0x7fc2b80c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 -- 172.21.15.158:0/3707010019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80094a0 con 0x7fc2b815da40 2023-10-16T09:56:28.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2b77fe700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b815da40 0x7fc2b80c99f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000cca0 tx=0x7fc2b000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:28.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc29ffff700 1 -- 172.21.15.158:0/3707010019 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b000cea0 con 0x7fc2b815da40 2023-10-16T09:56:28.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc29ffff700 1 -- 172.21.15.158:0/3707010019 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b000ade0 con 0x7fc2b815da40 2023-10-16T09:56:28.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b815a290 con 0x7fc2b815da40 2023-10-16T09:56:28.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.308+0000 7fc29ffff700 1 -- 172.21.15.158:0/3707010019 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0007310 con 0x7fc2b815da40 2023-10-16T09:56:28.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.309+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b815a850 con 0x7fc2b815da40 2023-10-16T09:56:28.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.309+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b809a650 con 0x7fc2b815da40 2023-10-16T09:56:28.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.310+0000 7fc29ffff700 1 -- 172.21.15.158:0/3707010019 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc2b0007470 con 0x7fc2b815da40 2023-10-16T09:56:28.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.310+0000 7fc29ffff700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc2a406b130 0x7fc2a406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:28.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.310+0000 7fc29ffff700 1 -- 172.21.15.158:0/3707010019 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fc2b0089b70 con 0x7fc2b815da40 2023-10-16T09:56:28.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.310+0000 7fc2b6ffd700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc2a406b130 0x7fc2a406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:28.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.311+0000 7fc2b6ffd700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc2a406b130 0x7fc2a406d5f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009870 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:28.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.315+0000 7fc29ffff700 1 -- 172.21.15.158:0/3707010019 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc2b00546b0 con 0x7fc2b815da40 2023-10-16T09:56:28.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.531+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2b80cb400 con 0x7fc2a406b130 2023-10-16T09:56:28.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.613+0000 7fc29ffff700 1 -- 172.21.15.158:0/3707010019 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fc2ac009460 con 0x7fc2a406b130 2023-10-16T09:56:28.615 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-10-16T09:56:28.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc2a406b130 msgr2=0x7fc2a406d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc2a406b130 0x7fc2a406d5f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009870 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2023-10-16T09:56:28.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b815da40 msgr2=0x7fc2b80c99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:28.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b815da40 0x7fc2b80c99f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000cca0 tx=0x7fc2b000a340 comp rx=0 tx=0).stop 2023-10-16T09:56:28.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 shutdown_connections 2023-10-16T09:56:28.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc2a406b130 0x7fc2a406d5f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2b815da40 0x7fc2b80c99f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b806afe0 0x7fc2b8159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 --2- 172.21.15.158:0/3707010019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b80cc590 0x7fc2b80c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:28.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.615+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 >> 172.21.15.158:0/3707010019 conn(0x7fc2b8099a10 msgr2=0x7fc2b815c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:28.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.616+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 shutdown_connections 2023-10-16T09:56:28.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:28.616+0000 7fc2bd9b3700 1 -- 172.21.15.158:0/3707010019 wait complete. 2023-10-16T09:56:28.629 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-10-16T09:56:28.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:28.629 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T09:56:28.647 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:28.647 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:28.648 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:28.648 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:28.648 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:28.648 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:28.649 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:28.649 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:28.649 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:28.649 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:28.649 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:28.649 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:28.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:28.650 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:28.711 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:28.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:28.712 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:28.774 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:28.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:28.775 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:28.838 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:28.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:28.839 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:28.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:28.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:28.904 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:28.966 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:28.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:28.967 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:29.031 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:29.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.031 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:29.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:29.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.095 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:29.158 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:29.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.159 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:29.222 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:29.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.223 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:29.286 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:29.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.287 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:29.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.350 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-10-16T09:56:29.410 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2023-10-16T09:56:29.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.574+0000 7fd82c0e5700 1 Processor -- start 2023-10-16T09:56:29.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.574+0000 7fd82c0e5700 1 -- start start 2023-10-16T09:56:29.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.575+0000 7fd82c0e5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd8240d1830 0x7fd82406b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:29.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.575+0000 7fd82c0e5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8240d1da0 con 0x7fd82415d7c0 2023-10-16T09:56:29.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.575+0000 7fd82c0e5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8240d2080 con 0x7fd82415dba0 2023-10-16T09:56:29.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.575+0000 7fd82c0e5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8240b2dc0 con 0x7fd8240d1830 2023-10-16T09:56:29.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd829e81700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd82415d7c0 0x7fd8240ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49836/0 (socket says 172.21.15.158:49836) 2023-10-16T09:56:29.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd829e81700 1 -- 172.21.15.158:0/308803825 learned_addr learned my addr 172.21.15.158:0/308803825 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:29.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd82a682700 1 --2- 172.21.15.158:0/308803825 >> v2:172.21.15.158:3300/0 conn(0x7fd8240d1830 0x7fd82406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:29.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd828e7f700 1 -- 172.21.15.158:0/308803825 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805933779 0 0) 0x7fd8240d1da0 con 0x7fd82415d7c0 2023-10-16T09:56:29.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd828e7f700 1 -- 172.21.15.158:0/308803825 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8100035b0 con 0x7fd82415d7c0 2023-10-16T09:56:29.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd82a682700 1 -- 172.21.15.158:0/308803825 >> v1:172.21.15.172:6789/0 conn(0x7fd82415dba0 legacy=0x7fd8240d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:29.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd82a682700 1 -- 172.21.15.158:0/308803825 >> v1:172.21.15.158:6789/0 conn(0x7fd82415d7c0 legacy=0x7fd8240ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:29.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd82a682700 1 -- 172.21.15.158:0/308803825 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8240d2360 con 0x7fd8240d1830 2023-10-16T09:56:29.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd82a682700 1 --2- 172.21.15.158:0/308803825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8240d1830 0x7fd82406b890 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009d90 tx=0x7fd81c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fffb901bbb156b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:29.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd828e7f700 1 -- 172.21.15.158:0/308803825 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00e070 con 0x7fd8240d1830 2023-10-16T09:56:29.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd828e7f700 1 -- 172.21.15.158:0/308803825 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c00bae0 con 0x7fd8240d1830 2023-10-16T09:56:29.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.576+0000 7fd828e7f700 1 -- 172.21.15.158:0/308803825 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00be50 con 0x7fd8240d1830 2023-10-16T09:56:29.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.577+0000 7fd82c0e5700 1 -- 172.21.15.158:0/308803825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8240d1830 msgr2=0x7fd82406b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:29.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.577+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/308803825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8240d1830 0x7fd82406b890 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009d90 tx=0x7fd81c00b330 comp rx=0 tx=0).stop 2023-10-16T09:56:29.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.577+0000 7fd82c0e5700 1 -- 172.21.15.158:0/308803825 shutdown_connections 2023-10-16T09:56:29.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.577+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/308803825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8240d1830 0x7fd82406b890 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:29.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.577+0000 7fd82c0e5700 1 -- 172.21.15.158:0/308803825 >> 172.21.15.158:0/308803825 conn(0x7fd824066bd0 msgr2=0x7fd8240d3660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:29.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.577+0000 7fd82c0e5700 1 -- 172.21.15.158:0/308803825 shutdown_connections 2023-10-16T09:56:29.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.577+0000 7fd82c0e5700 1 -- 172.21.15.158:0/308803825 wait complete. 2023-10-16T09:56:29.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.578+0000 7fd82c0e5700 1 Processor -- start 2023-10-16T09:56:29.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.578+0000 7fd82c0e5700 1 -- start start 2023-10-16T09:56:29.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd82c0e5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8240d1830 0x7fd8240ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:29.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd82c0e5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd82415d7c0 0x7fd82415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:29.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd82c0e5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd82415dba0 0x7fd8240caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:29.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd82c0e5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240c7ac0 con 0x7fd82415d7c0 2023-10-16T09:56:29.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd82c0e5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240cda10 con 0x7fd82415dba0 2023-10-16T09:56:29.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd82c0e5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240cabe0 con 0x7fd8240d1830 2023-10-16T09:56:29.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd82415d7c0 0x7fd82415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:29.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd82415d7c0 0x7fd82415aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49668/0 (socket says 172.21.15.158:49668) 2023-10-16T09:56:29.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 -- 172.21.15.158:0/227008676 learned_addr learned my addr 172.21.15.158:0/227008676 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:29.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd82a682700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd82415dba0 0x7fd8240caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:29.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829e81700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8240d1830 0x7fd8240ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:29.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 -- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8240d1830 msgr2=0x7fd8240ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:29.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8240d1830 0x7fd8240ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:29.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 -- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd82415dba0 msgr2=0x7fd8240caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:29.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd82415dba0 0x7fd8240caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:29.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 -- 172.21.15.158:0/227008676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c009a70 con 0x7fd82415d7c0 2023-10-16T09:56:29.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.579+0000 7fd829680700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd82415d7c0 0x7fd82415aa40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b450 tx=0x7fd82000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:29.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.580+0000 7fd81a7fc700 1 -- 172.21.15.158:0/227008676 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820011830 con 0x7fd82415d7c0 2023-10-16T09:56:29.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.580+0000 7fd81a7fc700 1 -- 172.21.15.158:0/227008676 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd820011990 con 0x7fd82415d7c0 2023-10-16T09:56:29.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.580+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8240cdc60 con 0x7fd82415d7c0 2023-10-16T09:56:29.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.580+0000 7fd81a7fc700 1 -- 172.21.15.158:0/227008676 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820011d00 con 0x7fd82415d7c0 2023-10-16T09:56:29.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.580+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8240ce220 con 0x7fd82415d7c0 2023-10-16T09:56:29.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.580+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8240c7c00 con 0x7fd82415d7c0 2023-10-16T09:56:29.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.581+0000 7fd81a7fc700 1 -- 172.21.15.158:0/227008676 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd82001a400 con 0x7fd82415d7c0 2023-10-16T09:56:29.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.581+0000 7fd81a7fc700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd80806b100 0x7fd80806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:29.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.581+0000 7fd81a7fc700 1 -- 172.21.15.158:0/227008676 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd820089960 con 0x7fd82415d7c0 2023-10-16T09:56:29.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.581+0000 7fd829e81700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd80806b100 0x7fd80806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:29.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.582+0000 7fd829e81700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd80806b100 0x7fd80806d5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd814007fd0 tx=0x7fd814009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:29.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.586+0000 7fd81a7fc700 1 -- 172.21.15.158:0/227008676 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8240c7c00 con 0x7fd82415d7c0 2023-10-16T09:56:29.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.810+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8240d2920 con 0x7fd80806b100 2023-10-16T09:56:29.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.889+0000 7fd81a7fc700 1 -- 172.21.15.158:0/227008676 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fd81400f690 con 0x7fd80806b100 2023-10-16T09:56:29.890 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-10-16T09:56:29.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd80806b100 msgr2=0x7fd80806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:29.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd80806b100 0x7fd80806d5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd814007fd0 tx=0x7fd814009040 comp rx=0 tx=0).stop 2023-10-16T09:56:29.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd82415d7c0 msgr2=0x7fd82415aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:29.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd82415d7c0 0x7fd82415aa40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b450 tx=0x7fd82000b760 comp rx=0 tx=0).stop 2023-10-16T09:56:29.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 shutdown_connections 2023-10-16T09:56:29.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd82415dba0 0x7fd8240caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:29.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8240d1830 0x7fd8240ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:29.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd80806b100 0x7fd80806d5c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:29.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 --2- 172.21.15.158:0/227008676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd82415d7c0 0x7fd82415aa40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:29.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 >> 172.21.15.158:0/227008676 conn(0x7fd824066bd0 msgr2=0x7fd82415bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:29.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.891+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 shutdown_connections 2023-10-16T09:56:29.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:29.892+0000 7fd82c0e5700 1 -- 172.21.15.158:0/227008676 wait complete. 2023-10-16T09:56:29.905 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-10-16T09:56:29.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.905 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T09:56:29.923 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:29.923 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:29.924 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:29.924 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:29.924 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:29.924 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:29.925 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:29.925 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:29.925 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:29.925 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:29.925 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:29.926 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:29.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:29.926 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:29.989 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:29.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.000 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:30.052 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:30.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.053 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:30.116 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:30.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.117 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:30.180 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:30.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.181 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:30.243 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:30.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.244 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:30.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:30.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.308 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:30.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:30.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.372 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:30.435 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:30.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.436 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:30.498 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:30.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.499 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:30.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:30.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.562 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:30.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:30.625 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-10-16T09:56:30.686 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2023-10-16T09:56:30.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.847+0000 7fa73c163700 1 Processor -- start 2023-10-16T09:56:30.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.848+0000 7fa73c163700 1 -- start start 2023-10-16T09:56:30.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73c163700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73415b660 0x7fa7340718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73c163700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa7340ce2b0 0x7fa7340ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73c163700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73415f250 con 0x7fa73406fbc0 2023-10-16T09:56:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73c163700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7340cec80 con 0x7fa7340ce2b0 2023-10-16T09:56:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73c163700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa734072140 con 0x7fa73415b660 2023-10-16T09:56:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa7340ce2b0 0x7fa7340ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa7340ce2b0 0x7fa7340ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49678/0 (socket says 172.21.15.158:49678) 2023-10-16T09:56:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 -- 172.21.15.158:0/3339795158 learned_addr learned my addr 172.21.15.158:0/3339795158 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 -- 172.21.15.158:0/3339795158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73415b660 msgr2=0x7fa7340718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 --2- 172.21.15.158:0/3339795158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73415b660 0x7fa7340718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 -- 172.21.15.158:0/3339795158 >> v1:172.21.15.172:6789/0 conn(0x7fa73406fbc0 legacy=0x7fa73415fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 -- 172.21.15.158:0/3339795158 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734072420 con 0x7fa7340ce2b0 2023-10-16T09:56:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa73a700700 1 --2- 172.21.15.158:0/3339795158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7340ce2b0 0x7fa7340ce740 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009e50 tx=0x7fa72c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8466ace150276f64 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa738efd700 1 -- 172.21.15.158:0/3339795158 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c00e070 con 0x7fa7340ce2b0 2023-10-16T09:56:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa738efd700 1 -- 172.21.15.158:0/3339795158 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c003a20 con 0x7fa7340ce2b0 2023-10-16T09:56:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.849+0000 7fa738efd700 1 -- 172.21.15.158:0/3339795158 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c003dd0 con 0x7fa7340ce2b0 2023-10-16T09:56:30.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 -- 172.21.15.158:0/3339795158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7340ce2b0 msgr2=0x7fa7340ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:30.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 --2- 172.21.15.158:0/3339795158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7340ce2b0 0x7fa7340ce740 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009e50 tx=0x7fa72c00b330 comp rx=0 tx=0).stop 2023-10-16T09:56:30.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 -- 172.21.15.158:0/3339795158 shutdown_connections 2023-10-16T09:56:30.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 --2- 172.21.15.158:0/3339795158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7340ce2b0 0x7fa7340ce740 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 --2- 172.21.15.158:0/3339795158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73415b660 0x7fa7340718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 -- 172.21.15.158:0/3339795158 >> 172.21.15.158:0/3339795158 conn(0x7fa7340d03b0 msgr2=0x7fa7340d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 -- 172.21.15.158:0/3339795158 shutdown_connections 2023-10-16T09:56:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.850+0000 7fa73c163700 1 -- 172.21.15.158:0/3339795158 wait complete. 2023-10-16T09:56:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.851+0000 7fa73c163700 1 Processor -- start 2023-10-16T09:56:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.851+0000 7fa73c163700 1 -- start start 2023-10-16T09:56:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73c163700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73406fbc0 0x7fa7340b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73c163700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7340ce2b0 0x7fa7341f1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73c163700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa73415b660 0x7fa734157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73c163700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7341f1b50 con 0x7fa73415b660 2023-10-16T09:56:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73c163700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7340b1cf0 con 0x7fa7340ce2b0 2023-10-16T09:56:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73c163700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7340c8600 con 0x7fa73406fbc0 2023-10-16T09:56:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa73415b660 0x7fa734157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa73415b660 0x7fa734157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49684/0 (socket says 172.21.15.158:49684) 2023-10-16T09:56:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 -- 172.21.15.158:0/2688429419 learned_addr learned my addr 172.21.15.158:0/2688429419 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa739eff700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73406fbc0 0x7fa7340b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa7396fe700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7340ce2b0 0x7fa7341f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 -- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73406fbc0 msgr2=0x7fa7340b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:30.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73406fbc0 0x7fa7340b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:30.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 -- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7340ce2b0 msgr2=0x7fa7341f1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:30.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7340ce2b0 0x7fa7341f1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:30.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa73a700700 1 -- 172.21.15.158:0/2688429419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c009b30 con 0x7fa73415b660 2023-10-16T09:56:30.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.852+0000 7fa739eff700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73406fbc0 0x7fa7340b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:30.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.853+0000 7fa73a700700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa73415b660 0x7fa734157c70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa7340ccd00 tx=0x7fa72c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:30.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.853+0000 7fa72a7fc700 1 -- 172.21.15.158:0/2688429419 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c004240 con 0x7fa73415b660 2023-10-16T09:56:30.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.853+0000 7fa72a7fc700 1 -- 172.21.15.158:0/2688429419 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c0043a0 con 0x7fa73415b660 2023-10-16T09:56:30.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.853+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7340c88e0 con 0x7fa73415b660 2023-10-16T09:56:30.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.853+0000 7fa72a7fc700 1 -- 172.21.15.158:0/2688429419 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c004750 con 0x7fa73415b660 2023-10-16T09:56:30.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.853+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7340c8ea0 con 0x7fa73415b660 2023-10-16T09:56:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.854+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7340681b0 con 0x7fa73415b660 2023-10-16T09:56:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.854+0000 7fa72a7fc700 1 -- 172.21.15.158:0/2688429419 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa72c019070 con 0x7fa73415b660 2023-10-16T09:56:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.854+0000 7fa72a7fc700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa71806b100 0x7fa71806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.854+0000 7fa72a7fc700 1 -- 172.21.15.158:0/2688429419 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fa72c08ac60 con 0x7fa73415b660 2023-10-16T09:56:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.855+0000 7fa739eff700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa71806b100 0x7fa71806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:30.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.855+0000 7fa739eff700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa71806b100 0x7fa71806d5c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7341ee710 tx=0x7fa724008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:30.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:30.859+0000 7fa72a7fc700 1 -- 172.21.15.158:0/2688429419 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa72c055730 con 0x7fa73415b660 2023-10-16T09:56:31.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.078+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa73415f4f0 con 0x7fa71806b100 2023-10-16T09:56:31.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.155+0000 7fa72a7fc700 1 -- 172.21.15.158:0/2688429419 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fa724013420 con 0x7fa71806b100 2023-10-16T09:56:31.156 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-10-16T09:56:31.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa71806b100 msgr2=0x7fa71806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:31.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa71806b100 0x7fa71806d5c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7341ee710 tx=0x7fa724008040 comp rx=0 tx=0).stop 2023-10-16T09:56:31.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa73415b660 msgr2=0x7fa734157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:31.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa73415b660 0x7fa734157c70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa7340ccd00 tx=0x7fa72c004ad0 comp rx=0 tx=0).stop 2023-10-16T09:56:31.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 shutdown_connections 2023-10-16T09:56:31.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fa71806b100 0x7fa71806d5c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:31.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa73415b660 0x7fa734157c70 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:31.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa73406fbc0 0x7fa7340b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:31.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 --2- 172.21.15.158:0/2688429419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7340ce2b0 0x7fa7341f1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:31.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 >> 172.21.15.158:0/2688429419 conn(0x7fa7340d03b0 msgr2=0x7fa7340b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:31.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 shutdown_connections 2023-10-16T09:56:31.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:31.157+0000 7fa73c163700 1 -- 172.21.15.158:0/2688429419 wait complete. 2023-10-16T09:56:31.170 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-10-16T09:56:31.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.171 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T09:56:31.189 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:31.190 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:31.190 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:31.190 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:31.190 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:31.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:31.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:31.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:31.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:31.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:31.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:31.192 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.192 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:31.254 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:31.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.255 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:31.317 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:31.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.318 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:31.380 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:31.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.381 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:31.443 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:31.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.444 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:31.505 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:31.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.506 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:31.569 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:31.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.570 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:31.633 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:31.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.634 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:31.696 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:31.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.696 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:31.759 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:31.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.760 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:31.822 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:31.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.823 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:31.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:31.886 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-10-16T09:56:31.946 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2023-10-16T09:56:32.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.109+0000 7fc878239700 1 Processor -- start 2023-10-16T09:56:32.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.109+0000 7fc878239700 1 -- start start 2023-10-16T09:56:32.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc878239700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8700d0e10 0x7fc87015c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:32.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc878239700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc870159660 con 0x7fc870159280 2023-10-16T09:56:32.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc878239700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87006b120 con 0x7fc8700cbef0 2023-10-16T09:56:32.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc878239700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8700cb710 con 0x7fc8700d0e10 2023-10-16T09:56:32.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc8757d4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8700d0e10 0x7fc87015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:32.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc875fd5700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc870159280 0x7fc87006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49864/0 (socket says 172.21.15.158:49864) 2023-10-16T09:56:32.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc875fd5700 1 -- 172.21.15.158:0/4203999308 learned_addr learned my addr 172.21.15.158:0/4203999308 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:32.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc8757d4700 1 -- 172.21.15.158:0/4203999308 >> v1:172.21.15.172:6789/0 conn(0x7fc8700cbef0 legacy=0x7fc8700cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:32.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc8757d4700 1 -- 172.21.15.158:0/4203999308 >> v1:172.21.15.158:6789/0 conn(0x7fc870159280 legacy=0x7fc87006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:32.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc8757d4700 1 -- 172.21.15.158:0/4203999308 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87006b400 con 0x7fc8700d0e10 2023-10-16T09:56:32.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc874fd3700 1 -- 172.21.15.158:0/4203999308 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304600499 0 0) 0x7fc870159660 con 0x7fc870159280 2023-10-16T09:56:32.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.110+0000 7fc8757d4700 1 --2- 172.21.15.158:0/4203999308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8700d0e10 0x7fc87015c590 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc860009840 tx=0x7fc860009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4896bd6a9e819157 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:32.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc874fd3700 1 -- 172.21.15.158:0/4203999308 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860004230 con 0x7fc8700d0e10 2023-10-16T09:56:32.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc874fd3700 1 -- 172.21.15.158:0/4203999308 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc860004390 con 0x7fc8700d0e10 2023-10-16T09:56:32.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc874fd3700 1 -- 172.21.15.158:0/4203999308 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8600036a0 con 0x7fc8700d0e10 2023-10-16T09:56:32.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc878239700 1 -- 172.21.15.158:0/4203999308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8700d0e10 msgr2=0x7fc87015c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:32.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc878239700 1 --2- 172.21.15.158:0/4203999308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8700d0e10 0x7fc87015c590 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc860009840 tx=0x7fc860009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:32.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc878239700 1 -- 172.21.15.158:0/4203999308 shutdown_connections 2023-10-16T09:56:32.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc878239700 1 --2- 172.21.15.158:0/4203999308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8700d0e10 0x7fc87015c590 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:32.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc878239700 1 -- 172.21.15.158:0/4203999308 >> 172.21.15.158:0/4203999308 conn(0x7fc870099a10 msgr2=0x7fc8700d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:32.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc878239700 1 -- 172.21.15.158:0/4203999308 shutdown_connections 2023-10-16T09:56:32.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.111+0000 7fc878239700 1 -- 172.21.15.158:0/4203999308 wait complete. 2023-10-16T09:56:32.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.112+0000 7fc878239700 1 Processor -- start 2023-10-16T09:56:32.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.112+0000 7fc878239700 1 -- start start 2023-10-16T09:56:32.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc878239700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8700cbef0 0x7fc8700cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:32.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc878239700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8700d0e10 0x7fc8700b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:32.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc878239700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc870159280 0x7fc870155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:32.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc878239700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87015dc50 con 0x7fc870159280 2023-10-16T09:56:32.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc878239700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87015df10 con 0x7fc8700d0e10 2023-10-16T09:56:32.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc878239700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87015e1f0 con 0x7fc8700cbef0 2023-10-16T09:56:32.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc8767d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc870159280 0x7fc870155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:32.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc8767d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc870159280 0x7fc870155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49698/0 (socket says 172.21.15.158:49698) 2023-10-16T09:56:32.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc8767d6700 1 -- 172.21.15.158:0/1302696742 learned_addr learned my addr 172.21.15.158:0/1302696742 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:32.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc8757d4700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8700d0e10 0x7fc8700b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:32.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc8767d6700 1 -- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8700cbef0 msgr2=0x7fc8700cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:32.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.113+0000 7fc8767d6700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8700cbef0 0x7fc8700cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:32.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8767d6700 1 -- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8700d0e10 msgr2=0x7fc8700b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:32.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8767d6700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8700d0e10 0x7fc8700b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:32.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8767d6700 1 -- 172.21.15.158:0/1302696742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8600094a0 con 0x7fc870159280 2023-10-16T09:56:32.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8757d4700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8700d0e10 0x7fc8700b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:32.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8767d6700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc870159280 0x7fc870155470 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc86800b990 tx=0x7fc86800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:32.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8667fc700 1 -- 172.21.15.158:0/1302696742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800c9d0 con 0x7fc870159280 2023-10-16T09:56:32.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8667fc700 1 -- 172.21.15.158:0/1302696742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86800cb30 con 0x7fc870159280 2023-10-16T09:56:32.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87015e4d0 con 0x7fc870159280 2023-10-16T09:56:32.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc8667fc700 1 -- 172.21.15.158:0/1302696742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868014440 con 0x7fc870159280 2023-10-16T09:56:32.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.114+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8700cf9d0 con 0x7fc870159280 2023-10-16T09:56:32.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.115+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87015e9f0 con 0x7fc870159280 2023-10-16T09:56:32.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.115+0000 7fc8667fc700 1 -- 172.21.15.158:0/1302696742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc868006d50 con 0x7fc870159280 2023-10-16T09:56:32.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.116+0000 7fc8667fc700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc85406b100 0x7fc85406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:32.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.116+0000 7fc8667fc700 1 -- 172.21.15.158:0/1302696742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fc868089660 con 0x7fc870159280 2023-10-16T09:56:32.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.116+0000 7fc875fd5700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc85406b100 0x7fc85406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:32.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.116+0000 7fc875fd5700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc85406b100 0x7fc85406d5c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc86c000c00 tx=0x7fc86c007a50 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:32.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.120+0000 7fc8667fc700 1 -- 172.21.15.158:0/1302696742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc87015e9f0 con 0x7fc870159280 2023-10-16T09:56:32.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.337+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc87015bd90 con 0x7fc85406b100 2023-10-16T09:56:32.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.419+0000 7fc8667fc700 1 -- 172.21.15.158:0/1302696742 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fc86c001270 con 0x7fc85406b100 2023-10-16T09:56:32.420 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-10-16T09:56:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc85406b100 msgr2=0x7fc85406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc85406b100 0x7fc85406d5c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc86c000c00 tx=0x7fc86c007a50 comp rx=0 tx=0).stop 2023-10-16T09:56:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc870159280 msgr2=0x7fc870155470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc870159280 0x7fc870155470 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc86800b990 tx=0x7fc86800bd50 comp rx=0 tx=0).stop 2023-10-16T09:56:32.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 shutdown_connections 2023-10-16T09:56:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fc85406b100 0x7fc85406d5c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc870159280 0x7fc870155470 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8700cbef0 0x7fc8700cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 --2- 172.21.15.158:0/1302696742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8700d0e10 0x7fc8700b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 >> 172.21.15.158:0/1302696742 conn(0x7fc870099a10 msgr2=0x7fc8700b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:32.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 shutdown_connections 2023-10-16T09:56:32.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:32.421+0000 7fc878239700 1 -- 172.21.15.158:0/1302696742 wait complete. 2023-10-16T09:56:32.434 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-10-16T09:56:32.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.435 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T09:56:32.453 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:32.453 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:32.454 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:32.454 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:32.454 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:32.454 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:32.454 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:32.455 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:32.455 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:32.455 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:32.455 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:32.455 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:32.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.456 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:32.517 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:32.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.518 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:32.580 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:32.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.581 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:32.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:32.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.643 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:32.706 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:32.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.707 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:32.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:32.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.770 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:32.833 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:32.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.833 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:32.896 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:32.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.897 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:32.959 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:32.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:32.960 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:33.022 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:33.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:33.023 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:33.085 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:33.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:33.086 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:33.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:33.150 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-10-16T09:56:33.210 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2023-10-16T09:56:33.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.374+0000 7fcbd15fd700 1 Processor -- start 2023-10-16T09:56:33.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.374+0000 7fcbd15fd700 1 -- start start 2023-10-16T09:56:33.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.374+0000 7fcbd15fd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcbcc0d0c40 0x7fcbcc0b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.374+0000 7fcbd15fd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc0b7c90 con 0x7fcbcc0d29b0 2023-10-16T09:56:33.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.374+0000 7fcbd15fd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc0b7f70 con 0x7fcbcc160d30 2023-10-16T09:56:33.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.374+0000 7fcbd15fd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbcc0b8910 con 0x7fcbcc0d0c40 2023-10-16T09:56:33.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbca7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcbcc0d29b0 0x7fcbcc0d0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49880/0 (socket says 172.21.15.158:49880) 2023-10-16T09:56:33.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbca7fc700 1 -- 172.21.15.158:0/1213903550 learned_addr learned my addr 172.21.15.158:0/1213903550 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:33.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836870917 0 0) 0x7fcbcc0b7c90 con 0x7fcbcc0d29b0 2023-10-16T09:56:33.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbcb7fe700 1 --2- 172.21.15.158:0/1213903550 >> v2:172.21.15.172:3300/0 conn(0x7fcbcc0d0c40 0x7fcbcc0b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb40035b0 con 0x7fcbcc0d29b0 2023-10-16T09:56:33.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 399075208 0 0) 0x7fcbb40035b0 con 0x7fcbcc0d29b0 2023-10-16T09:56:33.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 >> v2:172.21.15.172:3300/0 conn(0x7fcbcc0d0c40 msgr2=0x7fcbcc0b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:33.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 --2- 172.21.15.158:0/1213903550 >> v2:172.21.15.172:3300/0 conn(0x7fcbcc0d0c40 0x7fcbcc0b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 >> v1:172.21.15.172:6789/0 conn(0x7fcbcc160d30 legacy=0x7fcbcc0b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:33.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc0b8250 con 0x7fcbcc0d29b0 2023-10-16T09:56:33.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcbbc001670 con 0x7fcbcc0d29b0 2023-10-16T09:56:33.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 >> v1:172.21.15.158:6789/0 conn(0x7fcbcc0d29b0 legacy=0x7fcbcc0d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:33.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbb4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbb4004850 0x7fcbb4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbb4005210 0x7fcbb400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0b8910 con 0x7fcbb4003a00 2023-10-16T09:56:33.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb400c1b0 con 0x7fcbb4005210 2023-10-16T09:56:33.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb400c370 con 0x7fcbb4004850 2023-10-16T09:56:33.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbcaffd700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbb4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbcb7fe700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbb4005210 0x7fcbb400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.375+0000 7fcbca7fc700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbb4004850 0x7fcbb4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbcaffd700 1 -- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbb4004850 msgr2=0x7fcbb4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:33.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbcaffd700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbb4004850 0x7fcbb4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbcaffd700 1 -- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbb4005210 msgr2=0x7fcbb400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:33.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbcaffd700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbb4005210 0x7fcbb400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbcaffd700 1 -- 172.21.15.158:0/1213903550 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40035b0 con 0x7fcbb4003a00 2023-10-16T09:56:33.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbcb7fe700 1 -- 172.21.15.158:0/1213903550 reap_dead start 2023-10-16T09:56:33.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbca7fc700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbb4004850 0x7fcbb4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:33.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbcaffd700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbb4003e20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00017c0 tx=0x7fcbb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996de4dc95e36629 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:33.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00052d0 con 0x7fcbb4003a00 2023-10-16T09:56:33.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc0005450 con 0x7fcbb4003a00 2023-10-16T09:56:33.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbc9ffb700 1 -- 172.21.15.158:0/1213903550 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00038c0 con 0x7fcbb4003a00 2023-10-16T09:56:33.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbd15fd700 1 -- 172.21.15.158:0/1213903550 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 msgr2=0x7fcbb4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:33.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.376+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbb4003e20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00017c0 tx=0x7fcbb4005880 comp rx=0 tx=0).stop 2023-10-16T09:56:33.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.377+0000 7fcbd15fd700 1 -- 172.21.15.158:0/1213903550 shutdown_connections 2023-10-16T09:56:33.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.377+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/1213903550 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbb4003e20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.377+0000 7fcbd15fd700 1 -- 172.21.15.158:0/1213903550 >> 172.21.15.158:0/1213903550 conn(0x7fcbcc0669f0 msgr2=0x7fcbcc066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:33.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.377+0000 7fcbd15fd700 1 -- 172.21.15.158:0/1213903550 shutdown_connections 2023-10-16T09:56:33.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.377+0000 7fcbd15fd700 1 -- 172.21.15.158:0/1213903550 wait complete. 2023-10-16T09:56:33.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.378+0000 7fcbd15fd700 1 Processor -- start 2023-10-16T09:56:33.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.378+0000 7fcbd15fd700 1 -- start start 2023-10-16T09:56:33.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbd15fd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbcc15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbd15fd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbcc156500 0x7fcbcc0c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbd15fd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc156980 0x7fcbcc0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbd15fd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc157110 con 0x7fcbb4003a00 2023-10-16T09:56:33.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbd15fd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc157f30 con 0x7fcbcc156980 2023-10-16T09:56:33.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbd15fd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc158210 con 0x7fcbcc156500 2023-10-16T09:56:33.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbcc15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbcc15b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49714/0 (socket says 172.21.15.158:49714) 2023-10-16T09:56:33.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 -- 172.21.15.158:0/2696894384 learned_addr learned my addr 172.21.15.158:0/2696894384 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:33.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbca7fc700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbcc156500 0x7fcbcc0c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcb7fe700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc156980 0x7fcbcc0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 -- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbcc156500 msgr2=0x7fcbcc0c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:33.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbcc156500 0x7fcbcc0c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 -- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc156980 msgr2=0x7fcbcc0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:33.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc156980 0x7fcbcc0c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 -- 172.21.15.158:0/2696894384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc1584f0 con 0x7fcbb4003a00 2023-10-16T09:56:33.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcb7fe700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc156980 0x7fcbcc0c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:33.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbcaffd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbcc15b2f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0000c00 tx=0x7fcbc0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:33.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.379+0000 7fcbb37fe700 1 -- 172.21.15.158:0/2696894384 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0001290 con 0x7fcbb4003a00 2023-10-16T09:56:33.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.380+0000 7fcbb37fe700 1 -- 172.21.15.158:0/2696894384 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc0007040 con 0x7fcbb4003a00 2023-10-16T09:56:33.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.380+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbcc1587d0 con 0x7fcbb4003a00 2023-10-16T09:56:33.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.380+0000 7fcbb37fe700 1 -- 172.21.15.158:0/2696894384 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00073f0 con 0x7fcbb4003a00 2023-10-16T09:56:33.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.380+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc15b830 con 0x7fcbb4003a00 2023-10-16T09:56:33.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.380+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc15c250 con 0x7fcbb4003a00 2023-10-16T09:56:33.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.381+0000 7fcbb37fe700 1 -- 172.21.15.158:0/2696894384 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fcbc0007550 con 0x7fcbb4003a00 2023-10-16T09:56:33.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.381+0000 7fcbb37fe700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcbb806b100 0x7fcbb806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:33.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.381+0000 7fcbb37fe700 1 -- 172.21.15.158:0/2696894384 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fcbc008a380 con 0x7fcbb4003a00 2023-10-16T09:56:33.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.381+0000 7fcbca7fc700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcbb806b100 0x7fcbb806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:33.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.382+0000 7fcbca7fc700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcbb806b100 0x7fcbb806d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0017a0 tx=0x7fcbbc007040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:33.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.386+0000 7fcbb37fe700 1 -- 172.21.15.158:0/2696894384 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbc0055100 con 0x7fcbb4003a00 2023-10-16T09:56:33.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.653+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbcc158cf0 con 0x7fcbb806b100 2023-10-16T09:56:33.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.942+0000 7fcbb37fe700 1 -- 172.21.15.158:0/2696894384 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fcbcc158cf0 con 0x7fcbb806b100 2023-10-16T09:56:33.944 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-10-16T09:56:33.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcbb806b100 msgr2=0x7fcbb806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:33.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcbb806b100 0x7fcbb806d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0017a0 tx=0x7fcbbc007040 comp rx=0 tx=0).stop 2023-10-16T09:56:33.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 msgr2=0x7fcbcc15b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:33.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbcc15b2f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0000c00 tx=0x7fcbc0001120 comp rx=0 tx=0).stop 2023-10-16T09:56:33.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 shutdown_connections 2023-10-16T09:56:33.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc156980 0x7fcbcc0c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcbb806b100 0x7fcbb806d5c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbb4003a00 0x7fcbcc15b2f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 --2- 172.21.15.158:0/2696894384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbcc156500 0x7fcbcc0c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:33.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 >> 172.21.15.158:0/2696894384 conn(0x7fcbcc0669f0 msgr2=0x7fcbcc0ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:33.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 shutdown_connections 2023-10-16T09:56:33.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:33.945+0000 7fcbd15fd700 1 -- 172.21.15.158:0/2696894384 wait complete. 2023-10-16T09:56:33.958 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-10-16T09:56:33.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:33.959 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T09:56:33.976 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:33.977 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:33.977 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:33.977 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:33.977 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:33.977 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:33.978 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:33.978 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:33.978 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:33.978 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:33.979 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:33.979 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:33.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:33.979 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:34.041 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:34.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.042 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:34.104 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:34.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.105 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:34.167 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:34.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.168 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:34.230 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:34.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.231 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:34.292 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:34.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.293 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:34.354 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:34.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.355 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:34.418 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:34.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.419 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:34.481 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:34.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.482 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:34.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:34.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.546 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:34.617 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:34.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.618 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:34.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:34.681 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-10-16T09:56:34.740 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2023-10-16T09:56:34.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.908+0000 7fd257ecf700 1 Processor -- start 2023-10-16T09:56:34.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.908+0000 7fd257ecf700 1 -- start start 2023-10-16T09:56:34.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd257ecf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd25015deb0 0x7fd2500ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:34.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd257ecf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2500ccc00 0x7fd2500cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:34.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd257ecf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500d0ae0 con 0x7fd25015dad0 2023-10-16T09:56:34.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd257ecf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2500d12d0 con 0x7fd2500ccc00 2023-10-16T09:56:34.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd257ecf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2500cbdf0 con 0x7fd25015deb0 2023-10-16T09:56:34.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25646c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2500ccc00 0x7fd2500cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:34.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25646c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2500ccc00 0x7fd2500cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49726/0 (socket says 172.21.15.158:49726) 2023-10-16T09:56:34.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25646c700 1 -- 172.21.15.158:0/3575587873 learned_addr learned my addr 172.21.15.158:0/3575587873 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:34.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25546a700 1 --2- 172.21.15.158:0/3575587873 >> v2:172.21.15.172:3300/0 conn(0x7fd25015deb0 0x7fd2500ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:34.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25646c700 1 -- 172.21.15.158:0/3575587873 >> v2:172.21.15.172:3300/0 conn(0x7fd25015deb0 msgr2=0x7fd2500ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:34.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25646c700 1 --2- 172.21.15.158:0/3575587873 >> v2:172.21.15.172:3300/0 conn(0x7fd25015deb0 0x7fd2500ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:34.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25646c700 1 -- 172.21.15.158:0/3575587873 >> v1:172.21.15.172:6789/0 conn(0x7fd25015dad0 legacy=0x7fd2500cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:34.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.909+0000 7fd25646c700 1 -- 172.21.15.158:0/3575587873 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2500cc280 con 0x7fd2500ccc00 2023-10-16T09:56:34.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd25646c700 1 --2- 172.21.15.158:0/3575587873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2500cd030 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd24800d150 tx=0x7fd24800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d685e7ea8881698 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:34.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd254c69700 1 -- 172.21.15.158:0/3575587873 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248010070 con 0x7fd2500ccc00 2023-10-16T09:56:34.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd254c69700 1 -- 172.21.15.158:0/3575587873 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24800dd40 con 0x7fd2500ccc00 2023-10-16T09:56:34.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd254c69700 1 -- 172.21.15.158:0/3575587873 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24800b310 con 0x7fd2500ccc00 2023-10-16T09:56:34.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 -- 172.21.15.158:0/3575587873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 msgr2=0x7fd2500cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:34.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3575587873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2500cd030 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd24800d150 tx=0x7fd24800d460 comp rx=0 tx=0).stop 2023-10-16T09:56:34.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 -- 172.21.15.158:0/3575587873 shutdown_connections 2023-10-16T09:56:34.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3575587873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2500cd030 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:34.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3575587873 >> v2:172.21.15.172:3300/0 conn(0x7fd25015deb0 0x7fd2500ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:34.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 -- 172.21.15.158:0/3575587873 >> 172.21.15.158:0/3575587873 conn(0x7fd250099a10 msgr2=0x7fd2500d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:34.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 -- 172.21.15.158:0/3575587873 shutdown_connections 2023-10-16T09:56:34.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.910+0000 7fd257ecf700 1 -- 172.21.15.158:0/3575587873 wait complete. 2023-10-16T09:56:34.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 Processor -- start 2023-10-16T09:56:34.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 -- start start 2023-10-16T09:56:34.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2501594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:34.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd25015dad0 0x7fd2500b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:34.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25015deb0 0x7fd2500b5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:34.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2500b44d0 con 0x7fd2500ccc00 2023-10-16T09:56:34.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2501599f0 con 0x7fd25015dad0 2023-10-16T09:56:34.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd257ecf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2500b38c0 con 0x7fd25015deb0 2023-10-16T09:56:34.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd255c6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2501594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:34.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd255c6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2501594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49728/0 (socket says 172.21.15.158:49728) 2023-10-16T09:56:34.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.912+0000 7fd255c6b700 1 -- 172.21.15.158:0/3488291818 learned_addr learned my addr 172.21.15.158:0/3488291818 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:34.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd25546a700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd25015dad0 0x7fd2500b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:34.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd25646c700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25015deb0 0x7fd2500b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:34.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd255c6b700 1 -- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25015deb0 msgr2=0x7fd2500b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:34.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd255c6b700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25015deb0 0x7fd2500b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:34.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd255c6b700 1 -- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd25015dad0 msgr2=0x7fd2500b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:34.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd255c6b700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd25015dad0 0x7fd2500b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:34.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd255c6b700 1 -- 172.21.15.158:0/3488291818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24800bd40 con 0x7fd2500ccc00 2023-10-16T09:56:34.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd25546a700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd25015dad0 0x7fd2500b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:34.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd25646c700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25015deb0 0x7fd2500b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:34.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd255c6b700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2501594b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c4a0 tx=0x7fd24000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:34.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd2467fc700 1 -- 172.21.15.158:0/3488291818 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2400054e0 con 0x7fd2500ccc00 2023-10-16T09:56:34.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd2467fc700 1 -- 172.21.15.158:0/3488291818 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240003680 con 0x7fd2500ccc00 2023-10-16T09:56:34.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2500b3ba0 con 0x7fd2500ccc00 2023-10-16T09:56:34.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd2467fc700 1 -- 172.21.15.158:0/3488291818 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240003a30 con 0x7fd2500ccc00 2023-10-16T09:56:34.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.913+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2500b4160 con 0x7fd2500ccc00 2023-10-16T09:56:34.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.914+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2500681b0 con 0x7fd2500ccc00 2023-10-16T09:56:34.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.914+0000 7fd2467fc700 1 -- 172.21.15.158:0/3488291818 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd240003b90 con 0x7fd2500ccc00 2023-10-16T09:56:34.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.915+0000 7fd2467fc700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd23406b100 0x7fd23406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:34.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.915+0000 7fd2467fc700 1 -- 172.21.15.158:0/3488291818 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 7076+0+0 (secure 0 0 0) 0x7fd2400893a0 con 0x7fd2500ccc00 2023-10-16T09:56:34.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.915+0000 7fd25546a700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd23406b100 0x7fd23406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:34.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.915+0000 7fd25546a700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd23406b100 0x7fd23406d5c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00aa10 tx=0x7fd24c008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:34.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:34.919+0000 7fd2467fc700 1 -- 172.21.15.158:0/3488291818 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd240053d00 con 0x7fd2500ccc00 2023-10-16T09:56:35.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.138+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2500d0ff0 con 0x7fd23406b100 2023-10-16T09:56:35.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.214+0000 7fd2467fc700 1 -- 172.21.15.158:0/3488291818 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fd24c00c050 con 0x7fd23406b100 2023-10-16T09:56:35.216 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-10-16T09:56:35.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd23406b100 msgr2=0x7fd23406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:35.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd23406b100 0x7fd23406d5c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00aa10 tx=0x7fd24c008040 comp rx=0 tx=0).stop 2023-10-16T09:56:35.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 msgr2=0x7fd2501594b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:35.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2501594b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c4a0 tx=0x7fd24000c860 comp rx=0 tx=0).stop 2023-10-16T09:56:35.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 shutdown_connections 2023-10-16T09:56:35.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25015deb0 0x7fd2500b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:35.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fd23406b100 0x7fd23406d5c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:35.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2500ccc00 0x7fd2501594b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:35.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 --2- 172.21.15.158:0/3488291818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd25015dad0 0x7fd2500b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:35.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 >> 172.21.15.158:0/3488291818 conn(0x7fd250099a10 msgr2=0x7fd25015aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:35.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 shutdown_connections 2023-10-16T09:56:35.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:35.217+0000 7fd257ecf700 1 -- 172.21.15.158:0/3488291818 wait complete. 2023-10-16T09:56:35.230 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-10-16T09:56:35.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.230 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T09:56:35.249 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:35.249 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:35.249 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:35.249 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:35.250 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:35.250 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:35.250 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:35.250 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:35.250 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:35.251 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:35.251 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:35.251 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:35.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.252 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:35.313 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:35.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.314 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:35.376 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:35.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.377 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:35.439 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:35.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.440 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:35.502 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:35.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.503 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:35.564 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:35.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.565 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:35.633 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:35.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.634 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:35.701 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:35.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.702 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:35.763 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:35.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.764 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:35.826 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:35.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.827 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:35.889 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:35.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.890 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:35.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:35.951 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-10-16T09:56:36.011 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2023-10-16T09:56:36.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.177+0000 7fee07b40700 1 Processor -- start 2023-10-16T09:56:36.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.177+0000 7fee07b40700 1 -- start start 2023-10-16T09:56:36.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee07b40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:36.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee07b40700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee000cc590 0x7fee0006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:36.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee07b40700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee000cbd30 con 0x7fee0006ae10 2023-10-16T09:56:36.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee07b40700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0006b280 con 0x7fee000cc590 2023-10-16T09:56:36.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee07b40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee000cc1c0 con 0x7fee0015da40 2023-10-16T09:56:36.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee000cc590 0x7fee0006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:36.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee000cc590 0x7fee0006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49736/0 (socket says 172.21.15.158:49736) 2023-10-16T09:56:36.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 -- 172.21.15.158:0/2933737086 learned_addr learned my addr 172.21.15.158:0/2933737086 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:36.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee058dc700 1 --2- 172.21.15.158:0/2933737086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:36.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 -- 172.21.15.158:0/2933737086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 msgr2=0x7fee000caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:36.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 --2- 172.21.15.158:0/2933737086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 -- 172.21.15.158:0/2933737086 >> v1:172.21.15.172:6789/0 conn(0x7fee0006ae10 legacy=0x7fee000ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:36.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 -- 172.21.15.158:0/2933737086 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0015de20 con 0x7fee000cc590 2023-10-16T09:56:36.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee050db700 1 --2- 172.21.15.158:0/2933737086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0006a8d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0098f0 tx=0x7fedfc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc14440bfcb141b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:36.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee048da700 1 -- 172.21.15.158:0/2933737086 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc004230 con 0x7fee000cc590 2023-10-16T09:56:36.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee048da700 1 -- 172.21.15.158:0/2933737086 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc004390 con 0x7fee000cc590 2023-10-16T09:56:36.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.178+0000 7fee048da700 1 -- 172.21.15.158:0/2933737086 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0036a0 con 0x7fee000cc590 2023-10-16T09:56:36.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 -- 172.21.15.158:0/2933737086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 msgr2=0x7fee0006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:36.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 --2- 172.21.15.158:0/2933737086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0006a8d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0098f0 tx=0x7fedfc009c00 comp rx=0 tx=0).stop 2023-10-16T09:56:36.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 -- 172.21.15.158:0/2933737086 shutdown_connections 2023-10-16T09:56:36.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 --2- 172.21.15.158:0/2933737086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 --2- 172.21.15.158:0/2933737086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0006a8d0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 -- 172.21.15.158:0/2933737086 >> 172.21.15.158:0/2933737086 conn(0x7fee00099a10 msgr2=0x7fee000d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:36.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 -- 172.21.15.158:0/2933737086 shutdown_connections 2023-10-16T09:56:36.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.179+0000 7fee07b40700 1 -- 172.21.15.158:0/2933737086 wait complete. 2023-10-16T09:56:36.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.180+0000 7fee07b40700 1 Processor -- start 2023-10-16T09:56:36.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.180+0000 7fee07b40700 1 -- start start 2023-10-16T09:56:36.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee07b40700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0006ae10 0x7fee000c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:36.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee07b40700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0014e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:36.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee07b40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:36.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee07b40700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee000c8900 con 0x7fee000cc590 2023-10-16T09:56:36.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee07b40700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0014ed90 con 0x7fee0006ae10 2023-10-16T09:56:36.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee07b40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee000c79e0 con 0x7fee0015da40 2023-10-16T09:56:36.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0014e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:36.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0014e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49740/0 (socket says 172.21.15.158:49740) 2023-10-16T09:56:36.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 -- 172.21.15.158:0/3942204323 learned_addr learned my addr 172.21.15.158:0/3942204323 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:36.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee060dd700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:36.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee058dc700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0006ae10 0x7fee000c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:36.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 -- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 msgr2=0x7fee000c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:36.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 -- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0006ae10 msgr2=0x7fee000c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:36.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0006ae10 0x7fee000c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 -- 172.21.15.158:0/3942204323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc009550 con 0x7fee000cc590 2023-10-16T09:56:36.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee058dc700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0006ae10 0x7fee000c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:36.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.181+0000 7fee050db700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0014e850 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009130 tx=0x7fedfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:36.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.182+0000 7fedf67fc700 1 -- 172.21.15.158:0/3942204323 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc003e70 con 0x7fee000cc590 2023-10-16T09:56:36.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.182+0000 7fedf67fc700 1 -- 172.21.15.158:0/3942204323 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc01f070 con 0x7fee000cc590 2023-10-16T09:56:36.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.182+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee000c7c80 con 0x7fee000cc590 2023-10-16T09:56:36.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.182+0000 7fedf67fc700 1 -- 172.21.15.158:0/3942204323 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc00b8a0 con 0x7fee000cc590 2023-10-16T09:56:36.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.182+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee000c8240 con 0x7fee000cc590 2023-10-16T09:56:36.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.183+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee000681b0 con 0x7fee000cc590 2023-10-16T09:56:36.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.183+0000 7fedf67fc700 1 -- 172.21.15.158:0/3942204323 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fedfc021030 con 0x7fee000cc590 2023-10-16T09:56:36.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.183+0000 7fedf67fc700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fede406b100 0x7fede406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:36.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.184+0000 7fee058dc700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fede406b100 0x7fede406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:36.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.184+0000 7fedf67fc700 1 -- 172.21.15.158:0/3942204323 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7284+0+0 (secure 0 0 0) 0x7fedfc08b680 con 0x7fee000cc590 2023-10-16T09:56:36.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.184+0000 7fee058dc700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fede406b100 0x7fede406d5c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fedf0006fd0 tx=0x7fedf0008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:36.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.188+0000 7fedf67fc700 1 -- 172.21.15.158:0/3942204323 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fedfc055f10 con 0x7fee000cc590 2023-10-16T09:56:36.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.406+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fee000cb120 con 0x7fede406b100 2023-10-16T09:56:36.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.482+0000 7fedf67fc700 1 -- 172.21.15.158:0/3942204323 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fedf000c040 con 0x7fede406b100 2023-10-16T09:56:36.484 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-10-16T09:56:36.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fede406b100 msgr2=0x7fede406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:36.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fede406b100 0x7fede406d5c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fedf0006fd0 tx=0x7fedf0008040 comp rx=0 tx=0).stop 2023-10-16T09:56:36.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 msgr2=0x7fee0014e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:36.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0014e850 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009130 tx=0x7fedfc004750 comp rx=0 tx=0).stop 2023-10-16T09:56:36.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 shutdown_connections 2023-10-16T09:56:36.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0015da40 0x7fee000c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0006ae10 0x7fee000c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fede406b100 0x7fede406d5c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 --2- 172.21.15.158:0/3942204323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee000cc590 0x7fee0014e850 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:36.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 >> 172.21.15.158:0/3942204323 conn(0x7fee00099a10 msgr2=0x7fee000ca890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:36.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 shutdown_connections 2023-10-16T09:56:36.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:36.484+0000 7fee07b40700 1 -- 172.21.15.158:0/3942204323 wait complete. 2023-10-16T09:56:36.497 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-10-16T09:56:36.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.498 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T09:56:36.516 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:36.516 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:36.517 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:36.517 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:36.517 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:36.517 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:36.517 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:36.518 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:36.518 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:36.518 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:36.518 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:36.519 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:36.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.519 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:36.582 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:36.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.583 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:36.663 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:36.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.664 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:36.725 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:36.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.726 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:36.788 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:36.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.789 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:36.851 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:36.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.852 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:36.913 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:36.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.914 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:36.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:36.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:36.977 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:37.038 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:37.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.039 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:37.101 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:37.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.102 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:37.164 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:37.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.165 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:37.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.228 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-10-16T09:56:37.287 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2023-10-16T09:56:37.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.450+0000 7f8b2f244700 1 Processor -- start 2023-10-16T09:56:37.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.450+0000 7f8b2f244700 1 -- start start 2023-10-16T09:56:37.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.450+0000 7f8b2f244700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b2806e0d0 0x7f8b2815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:37.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.450+0000 7f8b2f244700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2815f380 con 0x7f8b280cbcf0 2023-10-16T09:56:37.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.450+0000 7f8b2f244700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b28071e70 con 0x7f8b2815dc40 2023-10-16T09:56:37.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.450+0000 7f8b2f244700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b28072150 con 0x7f8b2806e0d0 2023-10-16T09:56:37.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b2d7e1700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8b280cbcf0 0x7f8b280cc140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49928/0 (socket says 172.21.15.158:49928) 2023-10-16T09:56:37.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b2d7e1700 1 -- 172.21.15.158:0/2380312563 learned_addr learned my addr 172.21.15.158:0/2380312563 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:37.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b27fff700 1 --2- 172.21.15.158:0/2380312563 >> v2:172.21.15.172:3300/0 conn(0x7f8b2806e0d0 0x7f8b2815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:37.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b277fe700 1 -- 172.21.15.158:0/2380312563 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536461752 0 0) 0x7f8b20001490 con 0x7f8b280cbcf0 2023-10-16T09:56:37.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b277fe700 1 -- 172.21.15.158:0/2380312563 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b100035b0 con 0x7f8b280cbcf0 2023-10-16T09:56:37.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b27fff700 1 -- 172.21.15.158:0/2380312563 >> v1:172.21.15.172:6789/0 conn(0x7f8b2815dc40 legacy=0x7f8b2806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:37.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b27fff700 1 -- 172.21.15.158:0/2380312563 >> v1:172.21.15.158:6789/0 conn(0x7f8b280cbcf0 legacy=0x7f8b280cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:37.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b27fff700 1 -- 172.21.15.158:0/2380312563 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b28072430 con 0x7f8b2806e0d0 2023-10-16T09:56:37.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b27fff700 1 --2- 172.21.15.158:0/2380312563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b2806e0d0 0x7f8b2815ee40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c009840 tx=0x7f8b1c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15422f2c03457e2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:37.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b277fe700 1 -- 172.21.15.158:0/2380312563 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c00f8a0 con 0x7f8b2806e0d0 2023-10-16T09:56:37.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b277fe700 1 -- 172.21.15.158:0/2380312563 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1c00fa00 con 0x7f8b2806e0d0 2023-10-16T09:56:37.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.451+0000 7f8b277fe700 1 -- 172.21.15.158:0/2380312563 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c00fd70 con 0x7f8b2806e0d0 2023-10-16T09:56:37.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.452+0000 7f8b2f244700 1 -- 172.21.15.158:0/2380312563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b2806e0d0 msgr2=0x7f8b2815ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:37.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.452+0000 7f8b2f244700 1 --2- 172.21.15.158:0/2380312563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b2806e0d0 0x7f8b2815ee40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c009840 tx=0x7f8b1c009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:37.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.452+0000 7f8b2f244700 1 -- 172.21.15.158:0/2380312563 shutdown_connections 2023-10-16T09:56:37.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.452+0000 7f8b2f244700 1 --2- 172.21.15.158:0/2380312563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b2806e0d0 0x7f8b2815ee40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:37.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.452+0000 7f8b2f244700 1 -- 172.21.15.158:0/2380312563 >> 172.21.15.158:0/2380312563 conn(0x7f8b280d03b0 msgr2=0x7f8b280d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:37.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.452+0000 7f8b2f244700 1 -- 172.21.15.158:0/2380312563 shutdown_connections 2023-10-16T09:56:37.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.452+0000 7f8b2f244700 1 -- 172.21.15.158:0/2380312563 wait complete. 2023-10-16T09:56:37.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.453+0000 7f8b2f244700 1 Processor -- start 2023-10-16T09:56:37.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.453+0000 7f8b2f244700 1 -- start start 2023-10-16T09:56:37.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2f244700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b2806e0d0 0x7f8b281f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:37.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2f244700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b280cbcf0 0x7f8b28158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:37.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2f244700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b2815dc40 0x7f8b281591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:37.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2f244700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b280af070 con 0x7f8b2815dc40 2023-10-16T09:56:37.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2f244700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b281f2fb0 con 0x7f8b280cbcf0 2023-10-16T09:56:37.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2f244700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b28152110 con 0x7f8b2806e0d0 2023-10-16T09:56:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b2815dc40 0x7f8b281591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b2815dc40 0x7f8b281591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49758/0 (socket says 172.21.15.158:49758) 2023-10-16T09:56:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 -- 172.21.15.158:0/1720699357 learned_addr learned my addr 172.21.15.158:0/1720699357 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2cfe0700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b2806e0d0 0x7f8b281f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b27fff700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b280cbcf0 0x7f8b28158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 -- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b2806e0d0 msgr2=0x7f8b281f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b2806e0d0 0x7f8b281f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 -- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b280cbcf0 msgr2=0x7f8b28158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b280cbcf0 0x7f8b28158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.454+0000 7f8b2d7e1700 1 -- 172.21.15.158:0/1720699357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c0094a0 con 0x7f8b2815dc40 2023-10-16T09:56:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.455+0000 7f8b2d7e1700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b2815dc40 0x7f8b281591a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000cb40 tx=0x7f8b2000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:37.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.455+0000 7f8b257fa700 1 -- 172.21.15.158:0/1720699357 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000a9e0 con 0x7f8b2815dc40 2023-10-16T09:56:37.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.455+0000 7f8b257fa700 1 -- 172.21.15.158:0/1720699357 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2000ab40 con 0x7f8b2815dc40 2023-10-16T09:56:37.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.455+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b280ae7d0 con 0x7f8b2815dc40 2023-10-16T09:56:37.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.455+0000 7f8b257fa700 1 -- 172.21.15.158:0/1720699357 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20012400 con 0x7f8b2815dc40 2023-10-16T09:56:37.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.455+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b280aed90 con 0x7f8b2815dc40 2023-10-16T09:56:37.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.456+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b280681b0 con 0x7f8b2815dc40 2023-10-16T09:56:37.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.456+0000 7f8b257fa700 1 -- 172.21.15.158:0/1720699357 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8b2001b3b0 con 0x7f8b2815dc40 2023-10-16T09:56:37.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.456+0000 7f8b257fa700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8b1406b100 0x7f8b1406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:37.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.457+0000 7f8b257fa700 1 -- 172.21.15.158:0/1720699357 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7236+0+0 (secure 0 0 0) 0x7f8b20089910 con 0x7f8b2815dc40 2023-10-16T09:56:37.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.457+0000 7f8b2cfe0700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8b1406b100 0x7f8b1406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:37.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.457+0000 7f8b2cfe0700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8b1406b100 0x7f8b1406d5c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b18000f60 tx=0x7f8b18009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:37.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.461+0000 7f8b257fa700 1 -- 172.21.15.158:0/1720699357 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b200541f0 con 0x7f8b2815dc40 2023-10-16T09:56:37.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.718+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b280729f0 con 0x7f8b1406b100 2023-10-16T09:56:37.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.791+0000 7f8b257fa700 1 -- 172.21.15.158:0/1720699357 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f8b18001500 con 0x7f8b1406b100 2023-10-16T09:56:37.793 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-10-16T09:56:37.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.793+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8b1406b100 msgr2=0x7f8b1406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:37.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.793+0000 7f8b2f244700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8b1406b100 0x7f8b1406d5c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b18000f60 tx=0x7f8b18009040 comp rx=0 tx=0).stop 2023-10-16T09:56:37.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.793+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b2815dc40 msgr2=0x7f8b281591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:37.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.793+0000 7f8b2f244700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b2815dc40 0x7f8b281591a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000cb40 tx=0x7f8b2000ce50 comp rx=0 tx=0).stop 2023-10-16T09:56:37.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 shutdown_connections 2023-10-16T09:56:37.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8b1406b100 0x7f8b1406d5c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:37.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b2815dc40 0x7f8b281591a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:37.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b2806e0d0 0x7f8b281f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:37.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 --2- 172.21.15.158:0/1720699357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b280cbcf0 0x7f8b28158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:37.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 >> 172.21.15.158:0/1720699357 conn(0x7f8b280d03b0 msgr2=0x7f8b280afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:37.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 shutdown_connections 2023-10-16T09:56:37.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:37.794+0000 7f8b2f244700 1 -- 172.21.15.158:0/1720699357 wait complete. 2023-10-16T09:56:37.807 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-10-16T09:56:37.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.807 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T09:56:37.825 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:37.826 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:37.826 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:37.826 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:37.826 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:37.827 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:37.827 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:37.827 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:37.827 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:37.827 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:37.828 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:37.828 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:37.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.829 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:37.893 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:37.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.894 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:37.956 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:37.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:37.957 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:38.020 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:38.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.021 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:38.085 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:38.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.085 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:38.149 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:38.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.149 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:38.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:38.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.214 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:38.277 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:38.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.277 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:38.341 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:38.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.342 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:38.405 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:38.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.406 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:38.469 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:38.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.470 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:38.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:38.565 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-10-16T09:56:38.599 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2023-10-16T09:56:38.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.768+0000 7f8fc50fb700 1 Processor -- start 2023-10-16T09:56:38.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.768+0000 7f8fc50fb700 1 -- start start 2023-10-16T09:56:38.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fc50fb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8fc015dbd0 0x7f8fc006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:38.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fc50fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc006afe0 0x7f8fc00718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:38.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fc50fb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc015e510 con 0x7f8fc015dfc0 2023-10-16T09:56:38.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fc50fb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc00ccbb0 con 0x7f8fc015dbd0 2023-10-16T09:56:38.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fc50fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc015e7f0 con 0x7f8fc006afe0 2023-10-16T09:56:38.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fbed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8fc015dbd0 0x7f8fc006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:38.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fbe59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8fc015dfc0 0x7f8fc006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:49938/0 (socket says 172.21.15.158:49938) 2023-10-16T09:56:38.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fbe59c700 1 -- 172.21.15.158:0/369025760 learned_addr learned my addr 172.21.15.158:0/369025760 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:38.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.769+0000 7f8fbf59e700 1 --2- 172.21.15.158:0/369025760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc006afe0 0x7f8fc00718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:38.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbed9d700 1 -- 172.21.15.158:0/369025760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc006afe0 msgr2=0x7f8fc00718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:38.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbed9d700 1 --2- 172.21.15.158:0/369025760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc006afe0 0x7f8fc00718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:38.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbed9d700 1 -- 172.21.15.158:0/369025760 >> v1:172.21.15.158:6789/0 conn(0x7f8fc015dfc0 legacy=0x7f8fc006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:38.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbdd9b700 1 -- 172.21.15.158:0/369025760 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629945561 0 0) 0x7f8fc015e510 con 0x7f8fc015dfc0 2023-10-16T09:56:38.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbed9d700 1 -- 172.21.15.158:0/369025760 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc015ead0 con 0x7f8fc015dbd0 2023-10-16T09:56:38.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbed9d700 1 --2- 172.21.15.158:0/369025760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc015dbd0 0x7f8fc006df00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009840 tx=0x7f8fb4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb032a872db06eb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:38.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbdd9b700 1 -- 172.21.15.158:0/369025760 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4011830 con 0x7f8fc015dbd0 2023-10-16T09:56:38.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbdd9b700 1 -- 172.21.15.158:0/369025760 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb4011990 con 0x7f8fc015dbd0 2023-10-16T09:56:38.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fbdd9b700 1 -- 172.21.15.158:0/369025760 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4011d20 con 0x7f8fc015dbd0 2023-10-16T09:56:38.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fc50fb700 1 -- 172.21.15.158:0/369025760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc015dbd0 msgr2=0x7f8fc006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:38.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/369025760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc015dbd0 0x7f8fc006df00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009840 tx=0x7f8fb4009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:38.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fc50fb700 1 -- 172.21.15.158:0/369025760 shutdown_connections 2023-10-16T09:56:38.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/369025760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc015dbd0 0x7f8fc006df00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:38.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/369025760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc006afe0 0x7f8fc00718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:38.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fc50fb700 1 -- 172.21.15.158:0/369025760 >> 172.21.15.158:0/369025760 conn(0x7f8fc00b1f00 msgr2=0x7f8fc00b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:38.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.770+0000 7f8fc50fb700 1 -- 172.21.15.158:0/369025760 shutdown_connections 2023-10-16T09:56:38.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.771+0000 7f8fc50fb700 1 -- 172.21.15.158:0/369025760 wait complete. 2023-10-16T09:56:38.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.772+0000 7f8fc50fb700 1 Processor -- start 2023-10-16T09:56:38.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.772+0000 7f8fc50fb700 1 -- start start 2023-10-16T09:56:38.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fc50fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc006afe0 0x7f8fc0152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:38.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fc50fb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fc015dfc0 0x7f8fc00c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:38.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fc50fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc00c5b10 0x7f8fc00c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:38.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fc50fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00c4a70 con 0x7f8fc006afe0 2023-10-16T09:56:38.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fc50fb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00c4d50 con 0x7f8fc015dfc0 2023-10-16T09:56:38.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fc50fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00c5030 con 0x7f8fc00c5b10 2023-10-16T09:56:38.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fbed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc006afe0 0x7f8fc0152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:38.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fbed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc006afe0 0x7f8fc0152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49766/0 (socket says 172.21.15.158:49766) 2023-10-16T09:56:38.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.773+0000 7f8fbed9d700 1 -- 172.21.15.158:0/1339153000 learned_addr learned my addr 172.21.15.158:0/1339153000 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:38.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbe59c700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fc015dfc0 0x7f8fc00c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:38.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbf59e700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc00c5b10 0x7f8fc00c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:38.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbed9d700 1 -- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc00c5b10 msgr2=0x7f8fc00c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:38.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbed9d700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc00c5b10 0x7f8fc00c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:38.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbed9d700 1 -- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fc015dfc0 msgr2=0x7f8fc00c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:38.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbed9d700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fc015dfc0 0x7f8fc00c3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:38.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbed9d700 1 -- 172.21.15.158:0/1339153000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb40094a0 con 0x7f8fc006afe0 2023-10-16T09:56:38.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbe59c700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fc015dfc0 0x7f8fc00c3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:38.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbf59e700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc00c5b10 0x7f8fc00c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:38.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fbed9d700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc006afe0 0x7f8fc0152dc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40090b0 tx=0x7f8fb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:38.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fa77fe700 1 -- 172.21.15.158:0/1339153000 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40038a0 con 0x7f8fc006afe0 2023-10-16T09:56:38.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fa77fe700 1 -- 172.21.15.158:0/1339153000 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb401f070 con 0x7f8fc006afe0 2023-10-16T09:56:38.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc00c5310 con 0x7f8fc006afe0 2023-10-16T09:56:38.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fa77fe700 1 -- 172.21.15.158:0/1339153000 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4003c00 con 0x7f8fc006afe0 2023-10-16T09:56:38.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.774+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc00ca1e0 con 0x7f8fc006afe0 2023-10-16T09:56:38.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.775+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc00c5830 con 0x7f8fc006afe0 2023-10-16T09:56:38.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.775+0000 7f8fa77fe700 1 -- 172.21.15.158:0/1339153000 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8fb401a430 con 0x7f8fc006afe0 2023-10-16T09:56:38.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.776+0000 7f8fa77fe700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8fac06b100 0x7f8fac06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:38.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.776+0000 7f8fa77fe700 1 -- 172.21.15.158:0/1339153000 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7520+0+0 (secure 0 0 0) 0x7f8fb4021070 con 0x7f8fc006afe0 2023-10-16T09:56:38.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.776+0000 7f8fbe59c700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8fac06b100 0x7f8fac06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:38.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.776+0000 7f8fbe59c700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8fac06b100 0x7f8fac06d5c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00cb770 tx=0x7f8fb0007d50 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:38.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:38.780+0000 7f8fa77fe700 1 -- 172.21.15.158:0/1339153000 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fc00c5830 con 0x7f8fc006afe0 2023-10-16T09:56:39.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.001+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fc006e900 con 0x7f8fac06b100 2023-10-16T09:56:39.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.079+0000 7f8fa77fe700 1 -- 172.21.15.158:0/1339153000 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f8fb0004870 con 0x7f8fac06b100 2023-10-16T09:56:39.081 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-10-16T09:56:39.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8fac06b100 msgr2=0x7f8fac06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:39.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8fac06b100 0x7f8fac06d5c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00cb770 tx=0x7f8fb0007d50 comp rx=0 tx=0).stop 2023-10-16T09:56:39.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc006afe0 msgr2=0x7f8fc0152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:39.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc006afe0 0x7f8fc0152dc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40090b0 tx=0x7f8fb4004750 comp rx=0 tx=0).stop 2023-10-16T09:56:39.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 shutdown_connections 2023-10-16T09:56:39.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fc00c5b10 0x7f8fc00c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:39.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f8fac06b100 0x7f8fac06d5c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:39.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fc006afe0 0x7f8fc0152dc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:39.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 --2- 172.21.15.158:0/1339153000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fc015dfc0 0x7f8fc00c3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:39.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 >> 172.21.15.158:0/1339153000 conn(0x7f8fc00b1f00 msgr2=0x7f8fc00cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:39.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 shutdown_connections 2023-10-16T09:56:39.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:39.082+0000 7f8fc50fb700 1 -- 172.21.15.158:0/1339153000 wait complete. 2023-10-16T09:56:39.095 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-10-16T09:56:39.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.096 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T09:56:39.114 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:39.114 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:39.114 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:39.114 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:39.115 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:39.115 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:39.115 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:39.115 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:39.115 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:39.116 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:39.116 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:39.116 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:39.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.117 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:39.178 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:39.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.179 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:39.242 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:39.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.243 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:39.304 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:39.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.305 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:39.366 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:39.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.366 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:39.428 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:39.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.429 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:39.491 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:39.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.492 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:39.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:39.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.574 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:39.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:39.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.643 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:39.707 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:39.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.708 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:39.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:39.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.769 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:39.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:39.834 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-10-16T09:56:39.893 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2023-10-16T09:56:40.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.055+0000 7f05232f8700 1 Processor -- start 2023-10-16T09:56:40.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.056+0000 7f05232f8700 1 -- start start 2023-10-16T09:56:40.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.056+0000 7f05232f8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15dbd0 0x7f051c06df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:40.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.056+0000 7f05232f8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f051c06afe0 0x7f051c0718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:40.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.056+0000 7f05232f8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c15e510 con 0x7f051c15dfc0 2023-10-16T09:56:40.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.056+0000 7f05232f8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051c0ccbb0 con 0x7f051c06afe0 2023-10-16T09:56:40.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.056+0000 7f05232f8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c15e7f0 con 0x7f051c15dbd0 2023-10-16T09:56:40.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521895700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f051c06afe0 0x7f051c0718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:40.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521094700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15dbd0 0x7f051c06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:40.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521895700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f051c06afe0 0x7f051c0718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:50958/0 (socket says 172.21.15.158:50958) 2023-10-16T09:56:40.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521895700 1 -- 172.21.15.158:0/4087196947 learned_addr learned my addr 172.21.15.158:0/4087196947 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:40.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521094700 1 -- 172.21.15.158:0/4087196947 >> v2:172.21.15.172:3300/0 conn(0x7f051c06afe0 msgr2=0x7f051c0718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:40.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521094700 1 --2- 172.21.15.158:0/4087196947 >> v2:172.21.15.172:3300/0 conn(0x7f051c06afe0 0x7f051c0718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521094700 1 -- 172.21.15.158:0/4087196947 >> v1:172.21.15.172:6789/0 conn(0x7f051c15dfc0 legacy=0x7f051c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:40.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521094700 1 -- 172.21.15.158:0/4087196947 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c15ead0 con 0x7f051c15dbd0 2023-10-16T09:56:40.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521895700 1 --2- 172.21.15.158:0/4087196947 >> v2:172.21.15.172:3300/0 conn(0x7f051c06afe0 0x7f051c0718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:40.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0521094700 1 --2- 172.21.15.158:0/4087196947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15dbd0 0x7f051c06df00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f050c009840 tx=0x7f050c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29595fb02fcd6696 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:40.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.057+0000 7f0513fff700 1 -- 172.21.15.158:0/4087196947 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c011830 con 0x7f051c15dbd0 2023-10-16T09:56:40.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f0513fff700 1 -- 172.21.15.158:0/4087196947 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c011990 con 0x7f051c15dbd0 2023-10-16T09:56:40.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f0513fff700 1 -- 172.21.15.158:0/4087196947 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c011d20 con 0x7f051c15dbd0 2023-10-16T09:56:40.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 -- 172.21.15.158:0/4087196947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15dbd0 msgr2=0x7f051c06df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:40.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 --2- 172.21.15.158:0/4087196947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15dbd0 0x7f051c06df00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f050c009840 tx=0x7f050c009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:40.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 -- 172.21.15.158:0/4087196947 shutdown_connections 2023-10-16T09:56:40.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 --2- 172.21.15.158:0/4087196947 >> v2:172.21.15.172:3300/0 conn(0x7f051c06afe0 0x7f051c0718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 --2- 172.21.15.158:0/4087196947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15dbd0 0x7f051c06df00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 -- 172.21.15.158:0/4087196947 >> 172.21.15.158:0/4087196947 conn(0x7f051c0b1f00 msgr2=0x7f051c0b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:40.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 -- 172.21.15.158:0/4087196947 shutdown_connections 2023-10-16T09:56:40.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.058+0000 7f05232f8700 1 -- 172.21.15.158:0/4087196947 wait complete. 2023-10-16T09:56:40.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.060+0000 7f05232f8700 1 Processor -- start 2023-10-16T09:56:40.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.060+0000 7f05232f8700 1 -- start start 2023-10-16T09:56:40.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f05232f8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f051c06afe0 0x7f051c0c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:40.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f05232f8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f051c15dfc0 0x7f051c15b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:40.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f05232f8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15b9e0 0x7f051c1f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:40.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f05232f8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c15be90 con 0x7f051c06afe0 2023-10-16T09:56:40.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f05232f8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c0c5810 con 0x7f051c15dfc0 2023-10-16T09:56:40.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f05232f8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c0c4730 con 0x7f051c15b9e0 2023-10-16T09:56:40.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f051c06afe0 0x7f051c0c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:40.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f051c06afe0 0x7f051c0c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49780/0 (socket says 172.21.15.158:49780) 2023-10-16T09:56:40.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 -- 172.21.15.158:0/1553175907 learned_addr learned my addr 172.21.15.158:0/1553175907 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:40.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521895700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15b9e0 0x7f051c1f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:40.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0520893700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f051c15dfc0 0x7f051c15b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:40.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 -- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15b9e0 msgr2=0x7f051c1f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:40.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15b9e0 0x7f051c1f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 -- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f051c15dfc0 msgr2=0x7f051c15b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:40.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f051c15dfc0 0x7f051c15b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 -- 172.21.15.158:0/1553175907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514009b30 con 0x7f051c06afe0 2023-10-16T09:56:40.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0521094700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f051c06afe0 0x7f051c0c3f00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f050c009fd0 tx=0x7f050c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:40.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0511ffb700 1 -- 172.21.15.158:0/1553175907 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c0036a0 con 0x7f051c06afe0 2023-10-16T09:56:40.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0511ffb700 1 -- 172.21.15.158:0/1553175907 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c01f070 con 0x7f051c06afe0 2023-10-16T09:56:40.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c0094a0 con 0x7f051c06afe0 2023-10-16T09:56:40.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.061+0000 7f0511ffb700 1 -- 172.21.15.158:0/1553175907 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c003a40 con 0x7f051c06afe0 2023-10-16T09:56:40.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.062+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c0c4cf0 con 0x7f051c06afe0 2023-10-16T09:56:40.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.063+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051c0681b0 con 0x7f051c06afe0 2023-10-16T09:56:40.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.063+0000 7f0511ffb700 1 -- 172.21.15.158:0/1553175907 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f050c003ba0 con 0x7f051c06afe0 2023-10-16T09:56:40.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.063+0000 7f0511ffb700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f050006b100 0x7f050006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:40.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.063+0000 7f0511ffb700 1 -- 172.21.15.158:0/1553175907 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7012+0+0 (secure 0 0 0) 0x7f050c021070 con 0x7f051c06afe0 2023-10-16T09:56:40.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.063+0000 7f0520893700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f050006b100 0x7f050006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:40.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.064+0000 7f0520893700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f050006b100 0x7f050006d5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f051c0af350 tx=0x7f0518009040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:40.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.068+0000 7f0511ffb700 1 -- 172.21.15.158:0/1553175907 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f050c056030 con 0x7f051c06afe0 2023-10-16T09:56:40.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.297+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 --> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f051c06e900 con 0x7f050006b100 2023-10-16T09:56:40.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.389+0000 7f0511ffb700 1 -- 172.21.15.158:0/1553175907 <== mgr.5592 v2:172.21.15.158:6832/3552602298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f05180014c0 con 0x7f050006b100 2023-10-16T09:56:40.391 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-10-16T09:56:40.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.391+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f050006b100 msgr2=0x7f050006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:40.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.391+0000 7f05232f8700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f050006b100 0x7f050006d5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f051c0af350 tx=0x7f0518009040 comp rx=0 tx=0).stop 2023-10-16T09:56:40.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f051c06afe0 msgr2=0x7f051c0c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:40.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f051c06afe0 0x7f051c0c3f00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f050c009fd0 tx=0x7f050c004750 comp rx=0 tx=0).stop 2023-10-16T09:56:40.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 shutdown_connections 2023-10-16T09:56:40.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f051c15b9e0 0x7f051c1f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f050006b100 0x7f050006d5c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f051c06afe0 0x7f051c0c3f00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 --2- 172.21.15.158:0/1553175907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f051c15dfc0 0x7f051c15b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:40.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 >> 172.21.15.158:0/1553175907 conn(0x7f051c0b1f00 msgr2=0x7f051c0ce460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:40.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 shutdown_connections 2023-10-16T09:56:40.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:40.392+0000 7f05232f8700 1 -- 172.21.15.158:0/1553175907 wait complete. 2023-10-16T09:56:40.405 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-10-16T09:56:40.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.405 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T09:56:40.424 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:40.424 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:40.424 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:40.425 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:40.425 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:40.425 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:40.425 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:40.425 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:40.426 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:40.426 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:40.426 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:40.426 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_46_00_UTC 2023-10-16T09:56:40.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.427 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_46_00_UTC) 2023-10-16T09:56:40.487 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_47_00_UTC 2023-10-16T09:56:40.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.488 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_47_00_UTC) 2023-10-16T09:56:40.579 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_48_00_UTC 2023-10-16T09:56:40.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.580 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_48_00_UTC) 2023-10-16T09:56:40.603 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_49_00_UTC 2023-10-16T09:56:40.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.604 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_49_00_UTC) 2023-10-16T09:56:40.667 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_50_00_UTC 2023-10-16T09:56:40.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.667 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_50_00_UTC) 2023-10-16T09:56:40.732 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_51_00_UTC 2023-10-16T09:56:40.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.733 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_51_00_UTC) 2023-10-16T09:56:40.794 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_52_00_UTC 2023-10-16T09:56:40.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.795 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_52_00_UTC) 2023-10-16T09:56:40.859 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_53_00_UTC 2023-10-16T09:56:40.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.860 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_53_00_UTC) 2023-10-16T09:56:40.921 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_54_00_UTC 2023-10-16T09:56:40.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.922 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_54_00_UTC) 2023-10-16T09:56:40.984 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_55_00_UTC 2023-10-16T09:56:40.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:40.985 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_55_00_UTC) 2023-10-16T09:56:41.047 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_56_00_UTC 2023-10-16T09:56:41.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:41.048 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-09_56_00_UTC) 2023-10-16T09:56:41.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:41.111 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-10-16T09:56:41.170 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T09:56:41.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.331+0000 7f33dc11e700 1 Processor -- start 2023-10-16T09:56:41.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.332+0000 7f33dc11e700 1 -- start start 2023-10-16T09:56:41.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.332+0000 7f33dc11e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f33d415da40 0x7f33d40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:41.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.332+0000 7f33dc11e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:41.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.332+0000 7f33dc11e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33d406ae10 0x7f33d40ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:41.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.332+0000 7f33dc11e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d406b2c0 con 0x7f33d406ae10 2023-10-16T09:56:41.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33dc11e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d40cbd30 con 0x7f33d415da40 2023-10-16T09:56:41.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33dc11e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40cc1c0 con 0x7f33d40cc590 2023-10-16T09:56:41.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33d406ae10 0x7f33d40ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:41.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33d406ae10 0x7f33d40ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49792/0 (socket says 172.21.15.158:49792) 2023-10-16T09:56:41.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 -- 172.21.15.158:0/4253365854 learned_addr learned my addr 172.21.15.158:0/4253365854 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:41.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33d9eba700 1 --2- 172.21.15.158:0/4253365854 >> v2:172.21.15.172:3300/0 conn(0x7f33d415da40 0x7f33d40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:41.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33d96b9700 1 --2- 172.21.15.158:0/4253365854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:41.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 -- 172.21.15.158:0/4253365854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 msgr2=0x7f33d406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:41.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 --2- 172.21.15.158:0/4253365854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:41.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 -- 172.21.15.158:0/4253365854 >> v2:172.21.15.172:3300/0 conn(0x7f33d415da40 msgr2=0x7f33d40caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:41.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 --2- 172.21.15.158:0/4253365854 >> v2:172.21.15.172:3300/0 conn(0x7f33d415da40 0x7f33d40caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:41.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 -- 172.21.15.158:0/4253365854 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d415de20 con 0x7f33d406ae10 2023-10-16T09:56:41.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33da6bb700 1 --2- 172.21.15.158:0/4253365854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40ccd60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33d406c9c0 tx=0x7f33cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5d9fda65dd646f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:41.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33d8eb8700 1 -- 172.21.15.158:0/4253365854 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00e070 con 0x7f33d406ae10 2023-10-16T09:56:41.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33d8eb8700 1 -- 172.21.15.158:0/4253365854 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc00b9c0 con 0x7f33d406ae10 2023-10-16T09:56:41.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.333+0000 7f33d8eb8700 1 -- 172.21.15.158:0/4253365854 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00bd70 con 0x7f33d406ae10 2023-10-16T09:56:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 -- 172.21.15.158:0/4253365854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 msgr2=0x7f33d40ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 --2- 172.21.15.158:0/4253365854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40ccd60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33d406c9c0 tx=0x7f33cc00b330 comp rx=0 tx=0).stop 2023-10-16T09:56:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 -- 172.21.15.158:0/4253365854 shutdown_connections 2023-10-16T09:56:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 --2- 172.21.15.158:0/4253365854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40ccd60 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 --2- 172.21.15.158:0/4253365854 >> v2:172.21.15.172:3300/0 conn(0x7f33d415da40 0x7f33d40caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 --2- 172.21.15.158:0/4253365854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 -- 172.21.15.158:0/4253365854 >> 172.21.15.158:0/4253365854 conn(0x7f33d4099a10 msgr2=0x7f33d40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 -- 172.21.15.158:0/4253365854 shutdown_connections 2023-10-16T09:56:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.334+0000 7f33dc11e700 1 -- 172.21.15.158:0/4253365854 wait complete. 2023-10-16T09:56:41.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.335+0000 7f33dc11e700 1 Processor -- start 2023-10-16T09:56:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.335+0000 7f33dc11e700 1 -- start start 2023-10-16T09:56:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33dc11e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33dc11e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d4160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33dc11e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33d415da40 0x7f33d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:41.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33dc11e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40c63c0 con 0x7f33d406ae10 2023-10-16T09:56:41.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33dc11e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40c44c0 con 0x7f33d415da40 2023-10-16T09:56:41.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33dc11e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4160d60 con 0x7f33d40cc590 2023-10-16T09:56:41.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:41.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49794/0 (socket says 172.21.15.158:49794) 2023-10-16T09:56:41.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 -- 172.21.15.158:0/2032696676 learned_addr learned my addr 172.21.15.158:0/2032696676 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:41.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33da6bb700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33d415da40 0x7f33d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:41.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d96b9700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d4160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:41.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 -- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 msgr2=0x7f33d4160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:41.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d4160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 -- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33d415da40 msgr2=0x7f33d406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33d415da40 0x7f33d406dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 -- 172.21.15.158:0/2032696676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc009b30 con 0x7f33d406ae10 2023-10-16T09:56:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.336+0000 7f33d9eba700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40c3f80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c820 tx=0x7f33c400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.337+0000 7f33ca7fc700 1 -- 172.21.15.158:0/2032696676 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4011800 con 0x7f33d406ae10 2023-10-16T09:56:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.337+0000 7f33ca7fc700 1 -- 172.21.15.158:0/2032696676 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c4011960 con 0x7f33d406ae10 2023-10-16T09:56:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.337+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d415e0e0 con 0x7f33d406ae10 2023-10-16T09:56:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.337+0000 7f33ca7fc700 1 -- 172.21.15.158:0/2032696676 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4011cd0 con 0x7f33d406ae10 2023-10-16T09:56:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.337+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d40c68b0 con 0x7f33d406ae10 2023-10-16T09:56:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.337+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d415e270 con 0x7f33d406ae10 2023-10-16T09:56:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.338+0000 7f33ca7fc700 1 -- 172.21.15.158:0/2032696676 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 14) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f33c4004020 con 0x7f33d406ae10 2023-10-16T09:56:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.338+0000 7f33ca7fc700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f33b806b100 0x7f33b806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.338+0000 7f33ca7fc700 1 -- 172.21.15.158:0/2032696676 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f33c4089de0 con 0x7f33d406ae10 2023-10-16T09:56:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.338+0000 7f33d96b9700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f33b806b100 0x7f33b806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:41.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.339+0000 7f33d96b9700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f33b806b100 0x7f33b806d5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33d41f5e80 tx=0x7f33d0008040 comp rx=0 tx=0).ready entity=mgr.5592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:41.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.343+0000 7f33ca7fc700 1 -- 172.21.15.158:0/2032696676 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33c4054540 con 0x7f33d406ae10 2023-10-16T09:56:41.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:41.632+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f33d40681b0 con 0x7f33d406ae10 2023-10-16T09:56:42.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.540+0000 7f33ca7fc700 1 -- 172.21.15.158:0/2032696676 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v15) v1 ==== 93+0+0 (secure 0 0 0) 0x7f33c401f070 con 0x7f33d406ae10 2023-10-16T09:56:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.544+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f33b806b100 msgr2=0x7f33b806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.544+0000 7f33dc11e700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f33b806b100 0x7f33b806d5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33d41f5e80 tx=0x7f33d0008040 comp rx=0 tx=0).stop 2023-10-16T09:56:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.544+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 msgr2=0x7f33d40c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.544+0000 7f33dc11e700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40c3f80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c820 tx=0x7f33c400cb30 comp rx=0 tx=0).stop 2023-10-16T09:56:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.545+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 shutdown_connections 2023-10-16T09:56:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.545+0000 7f33dc11e700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33d415da40 0x7f33d406dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.545+0000 7f33dc11e700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f33b806b100 0x7f33b806d5c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.545+0000 7f33dc11e700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33d406ae10 0x7f33d40c3f80 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.545+0000 7f33dc11e700 1 --2- 172.21.15.158:0/2032696676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33d40cc590 0x7f33d4160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.545+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 >> 172.21.15.158:0/2032696676 conn(0x7f33d4099a10 msgr2=0x7f33d40b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:42.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.549+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 shutdown_connections 2023-10-16T09:56:42.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.549+0000 7f33dc11e700 1 -- 172.21.15.158:0/2032696676 wait complete. 2023-10-16T09:56:42.574 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T09:56:42.575 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:56:42.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:56:42.673 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T09:56:42.674 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:56:42.674 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:42.693 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:56:42.693 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:42.715 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:42.713+0000 7fa54c167200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:56:42.717 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:42.716+0000 7f0849d4f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:56:42.734 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:56:42.781 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:42.780+0000 7fa54c167200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:56:42.786 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:42.785+0000 7f0849d4f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:56:42.864 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:42.863+0000 7fa54c167200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:56:42.886 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:42.881+0000 7f0849d4f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:56:42.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.922+0000 7f1d634b0700 1 Processor -- start 2023-10-16T09:56:42.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.922+0000 7f1d634b0700 1 -- start start 2023-10-16T09:56:42.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d634b0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1d5c15f0f0 0x7f1d5c0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:42.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d634b0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c15b930 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d634b0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c15f5f0 con 0x7f1d5c0ce530 2023-10-16T09:56:42.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d634b0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d5c0cf020 con 0x7f1d5c15f0f0 2023-10-16T09:56:42.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d61a4d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1d5c15f0f0 0x7f1d5c0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:42.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d61a4d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1d5c15f0f0 0x7f1d5c0b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49876/0 (socket says 172.21.15.158:49876) 2023-10-16T09:56:42.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d61a4d700 1 -- 172.21.15.158:0/668402256 learned_addr learned my addr 172.21.15.158:0/668402256 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:42.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d60a4b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1d5c0cc1b0 0x7f1d5c0cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50048/0 (socket says 172.21.15.158:50048) 2023-10-16T09:56:42.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d61a4d700 1 -- 172.21.15.158:0/668402256 >> v1:172.21.15.172:6789/0 conn(0x7f1d5c0ce530 legacy=0x7f1d5c0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:42.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d61a4d700 1 -- 172.21.15.158:0/668402256 >> v1:172.21.15.158:6789/0 conn(0x7f1d5c0cc1b0 legacy=0x7f1d5c0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:42.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d53fff700 1 -- 172.21.15.158:0/668402256 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010283894 0 0) 0x7f1d5c15b930 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.923+0000 7f1d61a4d700 1 -- 172.21.15.158:0/668402256 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c15f8d0 con 0x7f1d5c15f0f0 2023-10-16T09:56:42.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d61a4d700 1 --2- 172.21.15.158:0/668402256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c15f0f0 0x7f1d5c0b3b20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1d54009e50 tx=0x7f1d5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642d5252263b70b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:42.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d53fff700 1 -- 172.21.15.158:0/668402256 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400e070 con 0x7f1d5c15f0f0 2023-10-16T09:56:42.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d53fff700 1 -- 172.21.15.158:0/668402256 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5400b9c0 con 0x7f1d5c15f0f0 2023-10-16T09:56:42.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d53fff700 1 -- 172.21.15.158:0/668402256 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400bd70 con 0x7f1d5c15f0f0 2023-10-16T09:56:42.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d634b0700 1 -- 172.21.15.158:0/668402256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c15f0f0 msgr2=0x7f1d5c0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:42.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d634b0700 1 --2- 172.21.15.158:0/668402256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c15f0f0 0x7f1d5c0b3b20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1d54009e50 tx=0x7f1d5400b330 comp rx=0 tx=0).stop 2023-10-16T09:56:42.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d634b0700 1 -- 172.21.15.158:0/668402256 shutdown_connections 2023-10-16T09:56:42.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d634b0700 1 --2- 172.21.15.158:0/668402256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c15f0f0 0x7f1d5c0b3b20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:42.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d634b0700 1 -- 172.21.15.158:0/668402256 >> 172.21.15.158:0/668402256 conn(0x7f1d5c06a8d0 msgr2=0x7f1d5c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:42.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.924+0000 7f1d634b0700 1 -- 172.21.15.158:0/668402256 shutdown_connections 2023-10-16T09:56:42.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.925+0000 7f1d634b0700 1 -- 172.21.15.158:0/668402256 wait complete. 2023-10-16T09:56:42.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.926+0000 7f1d634b0700 1 Processor -- start 2023-10-16T09:56:42.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.926+0000 7f1d634b0700 1 -- start start 2023-10-16T09:56:42.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d634b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c0cc1b0 0x7f1d5c1f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:42.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d634b0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d5c0ce530 0x7f1d5c0b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:42.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d634b0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d5c15f0f0 0x7f1d5c0b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:42.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d634b0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c1f1d40 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d634b0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c1f10c0 con 0x7f1d5c15f0f0 2023-10-16T09:56:42.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d634b0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c1f13a0 con 0x7f1d5c0ce530 2023-10-16T09:56:42.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c0cc1b0 0x7f1d5c1f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:42.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c0cc1b0 0x7f1d5c1f1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49878/0 (socket says 172.21.15.158:49878) 2023-10-16T09:56:42.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 -- 172.21.15.158:0/906866404 learned_addr learned my addr 172.21.15.158:0/906866404 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:42.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d60a4b700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d5c0ce530 0x7f1d5c0b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:42.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d61a4d700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d5c15f0f0 0x7f1d5c0b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:42.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 -- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d5c0ce530 msgr2=0x7f1d5c0b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:42.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d5c0ce530 0x7f1d5c0b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:42.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 -- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d5c15f0f0 msgr2=0x7f1d5c0b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:42.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d5c15f0f0 0x7f1d5c0b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:42.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 -- 172.21.15.158:0/906866404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d54009b30 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d6124c700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c0cc1b0 0x7f1d5c1f1ea0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800c4e0 tx=0x7f1d5800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:42.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d51ffb700 1 -- 172.21.15.158:0/906866404 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d580052d0 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d51ffb700 1 -- 172.21.15.158:0/906866404 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d58005430 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.928+0000 7f1d51ffb700 1 -- 172.21.15.158:0/906866404 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d580038c0 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.929+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c1f1680 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.929+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c0b1420 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.930+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5c0c9570 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.930+0000 7f1d51ffb700 1 -- 172.21.15.158:0/906866404 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f1d58003a20 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.930+0000 7f1d51ffb700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 0x7f1d4006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:42.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.930+0000 7f1d60a4b700 1 -- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 msgr2=0x7f1d4006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:42.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.931+0000 7f1d51ffb700 1 -- 172.21.15.158:0/906866404 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7172+0+0 (secure 0 0 0) 0x7f1d580891c0 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.931+0000 7f1d60a4b700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 0x7f1d4006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:42.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:42.935+0000 7f1d51ffb700 1 -- 172.21.15.158:0/906866404 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1d5c0c9570 con 0x7f1d5c0cc1b0 2023-10-16T09:56:42.958 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:42.957+0000 7fa54c167200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:56:42.973 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:42.972+0000 7f0849d4f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:56:43.100 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:43.098+0000 7fa54c167200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:56:43.120 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:43.119+0000 7f0849d4f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:56:43.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.132+0000 7f1d60a4b700 1 -- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 msgr2=0x7f1d4006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:43.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.132+0000 7f1d60a4b700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 0x7f1d4006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:43.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.138+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d5c0c9570 con 0x7f1d5c0cc1b0 2023-10-16T09:56:43.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.140+0000 7f1d51ffb700 1 -- 172.21.15.158:0/906866404 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+18532 (secure 0 0 0) 0x7f1d58080e60 con 0x7f1d5c0cc1b0 2023-10-16T09:56:43.141 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:43.141 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":52,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:56:42.535830+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:44:38.770851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T09:44:40.065625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":52,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.38","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2730663923":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6832/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6833/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/3457698273":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/686968952":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/251660717":"2023-10-17T09:44:59.554471+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":3},{"begin":6,"length":1},{"begin":8,"length":1},{"begin":10,"length":1},{"begin":13,"length":3},{"begin":17,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":3},{"begin":27,"length":1},{"begin":29,"length":1},{"begin":31,"length":2},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":40,"length":1},{"begin":43,"length":1},{"begin":45,"length":3},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":55,"length":3},{"begin":59,"length":1},{"begin":61,"length":1},{"begin":63,"length":1},{"begin":65,"length":3},{"begin":69,"length":1},{"begin":71,"length":2},{"begin":74,"length":1},{"begin":76,"length":3},{"begin":80,"length":1},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":91,"length":2},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":3},{"begin":103,"length":1},{"begin":105,"length":3},{"begin":109,"length":2},{"begin":155,"length":66}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":15,"length":1},{"begin":25,"length":1},{"begin":36,"length":1},{"begin":47,"length":1},{"begin":57,"length":1},{"begin":67,"length":1},{"begin":78,"length":1},{"begin":89,"length":1},{"begin":100,"length":1},{"begin":211,"length":10}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:56:43.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 msgr2=0x7f1d4006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:43.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 0x7f1d4006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c0cc1b0 msgr2=0x7f1d5c1f1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c0cc1b0 0x7f1d5c1f1ea0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800c4e0 tx=0x7f1d5800c7f0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 shutdown_connections 2023-10-16T09:56:43.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d5c15f0f0 0x7f1d5c0b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1d4006b0b0 0x7f1d4006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d5c0cc1b0 0x7f1d5c1f1ea0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 --2- 172.21.15.158:0/906866404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d5c0ce530 0x7f1d5c0b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 >> 172.21.15.158:0/906866404 conn(0x7f1d5c06a8d0 msgr2=0x7f1d5c0cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:43.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.142+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 shutdown_connections 2023-10-16T09:56:43.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.143+0000 7f1d634b0700 1 -- 172.21.15.158:0/906866404 wait complete. 2023-10-16T09:56:43.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:44:38.770851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:44:40.065625+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 52, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:56:43.157 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:56:43.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.321+0000 7fb8e4449700 1 Processor -- start 2023-10-16T09:56:43.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.321+0000 7fb8e4449700 1 -- start start 2023-10-16T09:56:43.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e4449700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e4449700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8dc0cf360 0x7fb8dc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e4449700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8dc0d1ee0 con 0x7fb8dc15daa0 2023-10-16T09:56:43.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e4449700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8dc0d40a0 con 0x7fb8dc0cf360 2023-10-16T09:56:43.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e4449700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0d12d0 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e21e5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e29e6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8dc0cf360 0x7fb8dc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e19e4700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb8dc15daa0 0x7fb8dc0cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:54002/0 (socket says 172.21.15.158:54002) 2023-10-16T09:56:43.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e19e4700 1 -- 172.21.15.158:0/2198160336 learned_addr learned my addr 172.21.15.158:0/2198160336 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:43.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e21e5700 1 -- 172.21.15.158:0/2198160336 >> v2:172.21.15.172:3300/0 conn(0x7fb8dc0cf360 msgr2=0x7fb8dc0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:43.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e21e5700 1 --2- 172.21.15.158:0/2198160336 >> v2:172.21.15.172:3300/0 conn(0x7fb8dc0cf360 0x7fb8dc0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e21e5700 1 -- 172.21.15.158:0/2198160336 >> v1:172.21.15.172:6789/0 conn(0x7fb8dc15daa0 legacy=0x7fb8dc0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e21e5700 1 -- 172.21.15.158:0/2198160336 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc0d15b0 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e11e3700 1 -- 172.21.15.158:0/2198160336 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193770997 0 0) 0x7fb8dc0d1ee0 con 0x7fb8dc15daa0 2023-10-16T09:56:43.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e29e6700 1 --2- 172.21.15.158:0/2198160336 >> v2:172.21.15.172:3300/0 conn(0x7fb8dc0cf360 0x7fb8dc0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:43.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.322+0000 7fb8e21e5700 1 --2- 172.21.15.158:0/2198160336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cff30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009840 tx=0x7fb8d8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5922d8ae98f54ff8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:43.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e11e3700 1 -- 172.21.15.158:0/2198160336 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8011830 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e11e3700 1 -- 172.21.15.158:0/2198160336 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d8011990 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e11e3700 1 -- 172.21.15.158:0/2198160336 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8011d20 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e4449700 1 -- 172.21.15.158:0/2198160336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc15d6c0 msgr2=0x7fb8dc0cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e4449700 1 --2- 172.21.15.158:0/2198160336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cff30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009840 tx=0x7fb8d8009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:43.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e4449700 1 -- 172.21.15.158:0/2198160336 shutdown_connections 2023-10-16T09:56:43.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e4449700 1 --2- 172.21.15.158:0/2198160336 >> v2:172.21.15.172:3300/0 conn(0x7fb8dc0cf360 0x7fb8dc0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e4449700 1 --2- 172.21.15.158:0/2198160336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cff30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e4449700 1 -- 172.21.15.158:0/2198160336 >> 172.21.15.158:0/2198160336 conn(0x7fb8dc09c940 msgr2=0x7fb8dc0d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:43.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.323+0000 7fb8e4449700 1 -- 172.21.15.158:0/2198160336 shutdown_connections 2023-10-16T09:56:43.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.324+0000 7fb8e4449700 1 -- 172.21.15.158:0/2198160336 wait complete. 2023-10-16T09:56:43.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 Processor -- start 2023-10-16T09:56:43.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 -- start start 2023-10-16T09:56:43.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc0cf360 0x7fb8dc150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc15daa0 0x7fb8dc0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0b4f00 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc1513a0 con 0x7fb8dc15daa0 2023-10-16T09:56:43.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.325+0000 7fb8e4449700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0b31b0 con 0x7fb8dc0cf360 2023-10-16T09:56:43.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49930/0 (socket says 172.21.15.158:49930) 2023-10-16T09:56:43.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e21e5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc0cf360 0x7fb8dc150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 -- 172.21.15.158:0/3453076304 learned_addr learned my addr 172.21.15.158:0/3453076304 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:43.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e29e6700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc15daa0 0x7fb8dc0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 -- 172.21.15.158:0/3453076304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc0cf360 msgr2=0x7fb8dc150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc0cf360 0x7fb8dc150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 -- 172.21.15.158:0/3453076304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc15daa0 msgr2=0x7fb8dc0b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc15daa0 0x7fb8dc0b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 -- 172.21.15.158:0/3453076304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d4009a70 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e19e4700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cce50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00c520 tx=0x7fb8cc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:43.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8d2ffd700 1 -- 172.21.15.158:0/3453076304 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc004c80 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8d2ffd700 1 -- 172.21.15.158:0/3453076304 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc004de0 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d80094a0 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8d2ffd700 1 -- 172.21.15.158:0/3453076304 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0072d0 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.326+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc160130 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.327+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8dc0d2350 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.328+0000 7fb8d2ffd700 1 -- 172.21.15.158:0/3453076304 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fb8cc007430 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.329+0000 7fb8d2ffd700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 0x7fb8c006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.329+0000 7fb8e21e5700 1 -- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 msgr2=0x7fb8c006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:43.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.329+0000 7fb8d2ffd700 1 -- 172.21.15.158:0/3453076304 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7172+0+0 (secure 0 0 0) 0x7fb8cc089b20 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.329+0000 7fb8e21e5700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 0x7fb8c006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:43.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.333+0000 7fb8d2ffd700 1 -- 172.21.15.158:0/3453076304 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8dc0d2350 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.424 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:43.423+0000 7fa54c167200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:56:43.462 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:43.461+0000 7f0849d4f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:56:43.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.530+0000 7fb8e21e5700 1 -- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 msgr2=0x7fb8c006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:43.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.530+0000 7fb8e21e5700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 0x7fb8c006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:43.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.533+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8dc0d2350 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.534+0000 7fb8d2ffd700 1 -- 172.21.15.158:0/3453076304 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8dc0d2350 con 0x7fb8dc15d6c0 2023-10-16T09:56:43.535 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:56:43.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 msgr2=0x7fb8c006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:43.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 0x7fb8c006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc15d6c0 msgr2=0x7fb8dc0cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cce50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00c520 tx=0x7fb8cc00c830 comp rx=0 tx=0).stop 2023-10-16T09:56:43.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 shutdown_connections 2023-10-16T09:56:43.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc15daa0 0x7fb8dc0b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8dc0cf360 0x7fb8dc150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fb8c006b0b0 0x7fb8c006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 --2- 172.21.15.158:0/3453076304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc15d6c0 0x7fb8dc0cce50 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.536+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 >> 172.21.15.158:0/3453076304 conn(0x7fb8dc09c940 msgr2=0x7fb8dc0ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:43.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.537+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 shutdown_connections 2023-10-16T09:56:43.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.537+0000 7fb8e4449700 1 -- 172.21.15.158:0/3453076304 wait complete. 2023-10-16T09:56:43.543 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:43.542+0000 7fa54c167200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:56:43.553 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:56:43.589 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:43.588+0000 7f0849d4f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:56:43.619 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:43.618+0000 7fa54c167200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:56:43.666 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:43.664+0000 7f0849d4f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:56:43.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.680+0000 7f331c9d4700 1 Processor -- start 2023-10-16T09:56:43.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.680+0000 7f331c9d4700 1 -- start start 2023-10-16T09:56:43.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.681+0000 7f331c9d4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f33180ce2b0 0x7f3318160a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.681+0000 7f331c9d4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180cee50 con 0x7f331806ab40 2023-10-16T09:56:43.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.681+0000 7f331c9d4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180ce890 con 0x7f331806af20 2023-10-16T09:56:43.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.681+0000 7f331c9d4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331806d9c0 con 0x7f33180ce2b0 2023-10-16T09:56:43.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.681+0000 7f331659c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f331806ab40 0x7f33181620e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50114/0 (socket says 172.21.15.158:50114) 2023-10-16T09:56:43.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.681+0000 7f331659c700 1 -- 172.21.15.158:0/1468133006 learned_addr learned my addr 172.21.15.158:0/1468133006 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:43.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1468133006 >> v2:172.21.15.172:3300/0 conn(0x7f33180ce2b0 0x7f3318160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073203816 0 0) 0x7f33180cee50 con 0x7f331806ab40 2023-10-16T09:56:43.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33000035b0 con 0x7f331806ab40 2023-10-16T09:56:43.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258018745 0 0) 0x7f33180ce890 con 0x7f331806af20 2023-10-16T09:56:43.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33180cee50 con 0x7f331806af20 2023-10-16T09:56:43.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815799027 0 0) 0x7f33000035b0 con 0x7f331806ab40 2023-10-16T09:56:43.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 >> v2:172.21.15.172:3300/0 conn(0x7f33180ce2b0 msgr2=0x7f3318160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:43.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 --2- 172.21.15.158:0/1468133006 >> v2:172.21.15.172:3300/0 conn(0x7f33180ce2b0 0x7f3318160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 >> v1:172.21.15.172:6789/0 conn(0x7f331806af20 legacy=0x7f331806e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180ceb70 con 0x7f331806ab40 2023-10-16T09:56:43.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3308001670 con 0x7f331806ab40 2023-10-16T09:56:43.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1468133006 >> v2:172.21.15.172:3300/0 conn(0x7f33180ce2b0 0x7f3318160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:43.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 >> v1:172.21.15.158:6789/0 conn(0x7f331806ab40 legacy=0x7f33181620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300003be0 0x7f3300003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3300004850 0x7f3300004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3300005260 0x7f330000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331806d9c0 con 0x7f3300005260 2023-10-16T09:56:43.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330000c190 con 0x7f3300004850 2023-10-16T09:56:43.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330000c350 con 0x7f3300003be0 2023-10-16T09:56:43.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3300005260 0x7f330000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3315d9b700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3300004850 0x7f3300004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f331659c700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300003be0 0x7f3300003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 -- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300003be0 msgr2=0x7f3300003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:43.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300003be0 0x7f3300003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 -- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3300004850 msgr2=0x7f3300004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:43.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3300004850 0x7f3300004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.682+0000 7f3316d9d700 1 -- 172.21.15.158:0/1468133006 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33000035b0 con 0x7f3300005260 2023-10-16T09:56:43.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.683+0000 7f3316d9d700 1 -- 172.21.15.158:0/1468133006 reap_dead start 2023-10-16T09:56:43.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.683+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3300005260 0x7f330000ba10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f331000e350 tx=0x7f3300005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608be2d37d833544 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:43.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.683+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331000edf0 con 0x7f3300005260 2023-10-16T09:56:43.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.683+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331000f040 con 0x7f3300005260 2023-10-16T09:56:43.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.683+0000 7f331559a700 1 -- 172.21.15.158:0/1468133006 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33100146c0 con 0x7f3300005260 2023-10-16T09:56:43.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.683+0000 7f331c9d4700 1 -- 172.21.15.158:0/1468133006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3300005260 msgr2=0x7f330000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.683+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3300005260 0x7f330000ba10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f331000e350 tx=0x7f3300005880 comp rx=0 tx=0).stop 2023-10-16T09:56:43.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.684+0000 7f331c9d4700 1 -- 172.21.15.158:0/1468133006 shutdown_connections 2023-10-16T09:56:43.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.684+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1468133006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3300005260 0x7f330000ba10 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.684+0000 7f331c9d4700 1 -- 172.21.15.158:0/1468133006 >> 172.21.15.158:0/1468133006 conn(0x7f33180d03b0 msgr2=0x7f33180d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:43.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.684+0000 7f331c9d4700 1 -- 172.21.15.158:0/1468133006 shutdown_connections 2023-10-16T09:56:43.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.684+0000 7f331c9d4700 1 -- 172.21.15.158:0/1468133006 wait complete. 2023-10-16T09:56:43.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.685+0000 7f331c9d4700 1 Processor -- start 2023-10-16T09:56:43.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.685+0000 7f331c9d4700 1 -- start start 2023-10-16T09:56:43.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331c9d4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300005260 0x7f33180ae530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331c9d4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33180b0dc0 0x7f33180b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331c9d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33180acf60 0x7f33180ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331c9d4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318152110 con 0x7f33180acf60 2023-10-16T09:56:43.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331c9d4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33180b1780 con 0x7f33180b0dc0 2023-10-16T09:56:43.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331c9d4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33180b38a0 con 0x7f3300005260 2023-10-16T09:56:43.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33180acf60 0x7f33180ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33180acf60 0x7f33180ad3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:49952/0 (socket says 172.21.15.158:49952) 2023-10-16T09:56:43.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 -- 172.21.15.158:0/1529296349 learned_addr learned my addr 172.21.15.158:0/1529296349 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:43.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3315d9b700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33180b0dc0 0x7f33180b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331659c700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300005260 0x7f33180ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:43.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 -- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300005260 msgr2=0x7f33180ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300005260 0x7f33180ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 -- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33180b0dc0 msgr2=0x7f33180b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33180b0dc0 0x7f33180b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 -- 172.21.15.158:0/1529296349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180aed80 con 0x7f33180acf60 2023-10-16T09:56:43.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3316d9d700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33180acf60 0x7f33180ad3e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f331000e940 tx=0x7f3310009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:43.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3306ffd700 1 -- 172.21.15.158:0/1529296349 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310007000 con 0x7f33180acf60 2023-10-16T09:56:43.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3306ffd700 1 -- 172.21.15.158:0/1529296349 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3310021070 con 0x7f33180acf60 2023-10-16T09:56:43.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33180b2f70 con 0x7f33180acf60 2023-10-16T09:56:43.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.686+0000 7f3306ffd700 1 -- 172.21.15.158:0/1529296349 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331001d660 con 0x7f33180acf60 2023-10-16T09:56:43.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.687+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33180b3530 con 0x7f33180acf60 2023-10-16T09:56:43.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.687+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33180681b0 con 0x7f33180acf60 2023-10-16T09:56:43.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.688+0000 7f3306ffd700 1 -- 172.21.15.158:0/1529296349 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f331001d7c0 con 0x7f33180acf60 2023-10-16T09:56:43.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.689+0000 7f3306ffd700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 0x7f32f806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:43.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.689+0000 7f331659c700 1 -- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 msgr2=0x7f32f806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:43.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.689+0000 7f3306ffd700 1 -- 172.21.15.158:0/1529296349 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7204+0+0 (secure 0 0 0) 0x7f331008bd70 con 0x7f33180acf60 2023-10-16T09:56:43.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.689+0000 7f331659c700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 0x7f32f806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:43.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.692+0000 7f3306ffd700 1 -- 172.21.15.158:0/1529296349 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3310056630 con 0x7f33180acf60 2023-10-16T09:56:43.703 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:43.697+0000 7fa54c167200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:56:43.751 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:43.750+0000 7f0849d4f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:56:43.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.890+0000 7f331659c700 1 -- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 msgr2=0x7f32f806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:43.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.890+0000 7f331659c700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 0x7f32f806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:43.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.893+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33180b6330 con 0x7f33180acf60 2023-10-16T09:56:43.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.893+0000 7f3306ffd700 1 -- 172.21.15.158:0/1529296349 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3310018020 con 0x7f33180acf60 2023-10-16T09:56:43.895 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:56:43.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 msgr2=0x7f32f806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:43.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 0x7f32f806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33180acf60 msgr2=0x7f33180ad3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:43.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33180acf60 0x7f33180ad3e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f331000e940 tx=0x7f3310009750 comp rx=0 tx=0).stop 2023-10-16T09:56:43.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 shutdown_connections 2023-10-16T09:56:43.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f32f806b0b0 0x7f32f806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33180acf60 0x7f33180ad3e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3300005260 0x7f33180ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 --2- 172.21.15.158:0/1529296349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33180b0dc0 0x7f33180b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:43.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 >> 172.21.15.158:0/1529296349 conn(0x7f33180d03b0 msgr2=0x7f3318071830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:43.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 shutdown_connections 2023-10-16T09:56:43.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:43.896+0000 7f331c9d4700 1 -- 172.21.15.158:0/1529296349 wait complete. 2023-10-16T09:56:43.910 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:56:44.067 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:44.065+0000 7fa54c167200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:56:44.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.078+0000 7f2901f79700 1 Processor -- start 2023-10-16T09:56:44.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.078+0000 7f2901f79700 1 -- start start 2023-10-16T09:56:44.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f2901f79700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28fc15b6b0 0x7f28fc0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f2901f79700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0b4420 con 0x7f28fc0b3630 2023-10-16T09:56:44.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f2901f79700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0cbc20 con 0x7f28fc0d0c60 2023-10-16T09:56:44.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f2901f79700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc15ba90 con 0x7f28fc15b6b0 2023-10-16T09:56:44.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fbfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f28fc0b3630 0x7f28fc0b48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50182/0 (socket says 172.21.15.158:50182) 2023-10-16T09:56:44.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fbfff700 1 -- 172.21.15.158:0/2189546492 learned_addr learned my addr 172.21.15.158:0/2189546492 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:44.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fb7fe700 1 --2- 172.21.15.158:0/2189546492 >> v2:172.21.15.172:3300/0 conn(0x7f28fc15b6b0 0x7f28fc0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fa7fc700 1 -- 172.21.15.158:0/2189546492 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841763289 0 0) 0x7f28fc0b4420 con 0x7f28fc0b3630 2023-10-16T09:56:44.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fa7fc700 1 -- 172.21.15.158:0/2189546492 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e80035b0 con 0x7f28fc0b3630 2023-10-16T09:56:44.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fb7fe700 1 -- 172.21.15.158:0/2189546492 >> v1:172.21.15.172:6789/0 conn(0x7f28fc0d0c60 legacy=0x7f28fc0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fa7fc700 1 -- 172.21.15.158:0/2189546492 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673822840 0 0) 0x7f28fc0cbc20 con 0x7f28fc0d0c60 2023-10-16T09:56:44.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fb7fe700 1 -- 172.21.15.158:0/2189546492 >> v1:172.21.15.158:6789/0 conn(0x7f28fc0b3630 legacy=0x7f28fc0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.079+0000 7f28fb7fe700 1 -- 172.21.15.158:0/2189546492 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0cbf00 con 0x7f28fc15b6b0 2023-10-16T09:56:44.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.080+0000 7f28fb7fe700 1 --2- 172.21.15.158:0/2189546492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc0cfdb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009d90 tx=0x7f28ec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8459efd5c7bbe784 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:44.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.080+0000 7f28fa7fc700 1 -- 172.21.15.158:0/2189546492 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00e070 con 0x7f28fc15b6b0 2023-10-16T09:56:44.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.080+0000 7f28fa7fc700 1 -- 172.21.15.158:0/2189546492 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec00bb20 con 0x7f28fc15b6b0 2023-10-16T09:56:44.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.080+0000 7f28fa7fc700 1 -- 172.21.15.158:0/2189546492 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00be90 con 0x7f28fc15b6b0 2023-10-16T09:56:44.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.080+0000 7f2901f79700 1 -- 172.21.15.158:0/2189546492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 msgr2=0x7f28fc0cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.080+0000 7f2901f79700 1 --2- 172.21.15.158:0/2189546492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc0cfdb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009d90 tx=0x7f28ec00b330 comp rx=0 tx=0).stop 2023-10-16T09:56:44.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.081+0000 7f2901f79700 1 -- 172.21.15.158:0/2189546492 shutdown_connections 2023-10-16T09:56:44.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.081+0000 7f2901f79700 1 --2- 172.21.15.158:0/2189546492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc0cfdb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.081+0000 7f2901f79700 1 -- 172.21.15.158:0/2189546492 >> 172.21.15.158:0/2189546492 conn(0x7f28fc0cdf20 msgr2=0x7f28fc0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:44.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.081+0000 7f2901f79700 1 -- 172.21.15.158:0/2189546492 shutdown_connections 2023-10-16T09:56:44.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.081+0000 7f2901f79700 1 -- 172.21.15.158:0/2189546492 wait complete. 2023-10-16T09:56:44.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.082+0000 7f2901f79700 1 Processor -- start 2023-10-16T09:56:44.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.082+0000 7f2901f79700 1 -- start start 2023-10-16T09:56:44.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f2901f79700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28fc0b3630 0x7f28fc1509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f2901f79700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28fc0d0c60 0x7f28fc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f2901f79700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc1520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f2901f79700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0cac40 con 0x7f28fc0d0c60 2023-10-16T09:56:44.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f2901f79700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc150f00 con 0x7f28fc15b6b0 2023-10-16T09:56:44.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f2901f79700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0ca070 con 0x7f28fc0b3630 2023-10-16T09:56:44.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28faffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28fc0d0c60 0x7f28fc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc1520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28faffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28fc0d0c60 0x7f28fc0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50010/0 (socket says 172.21.15.158:50010) 2023-10-16T09:56:44.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28faffd700 1 -- 172.21.15.158:0/32282725 learned_addr learned my addr 172.21.15.158:0/32282725 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:44.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fb7fe700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28fc0b3630 0x7f28fc1509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fbfff700 1 -- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28fc0b3630 msgr2=0x7f28fc1509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fbfff700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28fc0b3630 0x7f28fc1509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fbfff700 1 -- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28fc0d0c60 msgr2=0x7f28fc0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fbfff700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28fc0d0c60 0x7f28fc0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fbfff700 1 -- 172.21.15.158:0/32282725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec009a70 con 0x7f28fc15b6b0 2023-10-16T09:56:44.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28faffd700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28fc0d0c60 0x7f28fc0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:44.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.083+0000 7f28fbfff700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc1520d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0ced20 tx=0x7f28f400c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:44.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.084+0000 7f28f3fff700 1 -- 172.21.15.158:0/32282725 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4005b40 con 0x7f28fc15b6b0 2023-10-16T09:56:44.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.084+0000 7f28f3fff700 1 -- 172.21.15.158:0/32282725 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4005ca0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.084+0000 7f28f3fff700 1 -- 172.21.15.158:0/32282725 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f40072d0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.084+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc0ca310 con 0x7f28fc15b6b0 2023-10-16T09:56:44.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.084+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc0ca8d0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.085+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc0cafd0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.085+0000 7f28f3fff700 1 -- 172.21.15.158:0/32282725 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f28f4007430 con 0x7f28fc15b6b0 2023-10-16T09:56:44.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.086+0000 7f28f3fff700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 0x7f28dc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.086+0000 7f28fb7fe700 1 -- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 msgr2=0x7f28dc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:44.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.086+0000 7f28f3fff700 1 -- 172.21.15.158:0/32282725 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7204+0+0 (secure 0 0 0) 0x7f28f4088fd0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.086+0000 7f28fb7fe700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 0x7f28dc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:44.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.091+0000 7f28f3fff700 1 -- 172.21.15.158:0/32282725 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f28fc0cafd0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.151 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:44.150+0000 7f0849d4f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:56:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.287+0000 7f28fb7fe700 1 -- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 msgr2=0x7f28dc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.287+0000 7f28fb7fe700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 0x7f28dc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:44.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.315+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28fc0cafd0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.316+0000 7f28f3fff700 1 -- 172.21.15.158:0/32282725 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f28fc0cafd0 con 0x7f28fc15b6b0 2023-10-16T09:56:44.317 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:56:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 msgr2=0x7f28dc06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 0x7f28dc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 msgr2=0x7f28fc1520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc1520d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0ced20 tx=0x7f28f400c670 comp rx=0 tx=0).stop 2023-10-16T09:56:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 shutdown_connections 2023-10-16T09:56:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28fc15b6b0 0x7f28fc1520d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28fc0b3630 0x7f28fc1509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f28dc06b0b0 0x7f28dc06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 --2- 172.21.15.158:0/32282725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28fc0d0c60 0x7f28fc0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 >> 172.21.15.158:0/32282725 conn(0x7f28fc0cdf20 msgr2=0x7f28fc0b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:44.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 shutdown_connections 2023-10-16T09:56:44.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.318+0000 7f2901f79700 1 -- 172.21.15.158:0/32282725 wait complete. 2023-10-16T09:56:44.332 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:56:44.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.455+0000 7f02b72a5700 1 Processor -- start 2023-10-16T09:56:44.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.455+0000 7f02b72a5700 1 -- start start 2023-10-16T09:56:44.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b72a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b015d670 0x7f02b006d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b72a5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02b00cf0a0 0x7f02b00cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b72a5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b0161320 con 0x7f02b0070340 2023-10-16T09:56:44.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b72a5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b00cfa70 con 0x7f02b00cf0a0 2023-10-16T09:56:44.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b72a5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0161600 con 0x7f02b015d670 2023-10-16T09:56:44.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02b00cf0a0 0x7f02b00cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02b00cf0a0 0x7f02b00cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50022/0 (socket says 172.21.15.158:50022) 2023-10-16T09:56:44.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b4840700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f02b0070340 0x7f02b0161a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50196/0 (socket says 172.21.15.158:50196) 2023-10-16T09:56:44.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 -- 172.21.15.158:0/3818887541 learned_addr learned my addr 172.21.15.158:0/3818887541 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:44.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5041700 1 --2- 172.21.15.158:0/3818887541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b015d670 0x7f02b006d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 -- 172.21.15.158:0/3818887541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b015d670 msgr2=0x7f02b006d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:44.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 --2- 172.21.15.158:0/3818887541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b015d670 0x7f02b006d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 -- 172.21.15.158:0/3818887541 >> v1:172.21.15.158:6789/0 conn(0x7f02b0070340 legacy=0x7f02b0161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02a7fff700 1 -- 172.21.15.158:0/3818887541 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606390929 0 0) 0x7f02b0161320 con 0x7f02b0070340 2023-10-16T09:56:44.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 -- 172.21.15.158:0/3818887541 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b01621a0 con 0x7f02b00cf0a0 2023-10-16T09:56:44.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02b5842700 1 --2- 172.21.15.158:0/3818887541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b00cf0a0 0x7f02b00cf530 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009e50 tx=0x7f02a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c06edd685b7deae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:44.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.457+0000 7f02a7fff700 1 -- 172.21.15.158:0/3818887541 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a800e070 con 0x7f02b00cf0a0 2023-10-16T09:56:44.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02a7fff700 1 -- 172.21.15.158:0/3818887541 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a800b910 con 0x7f02b00cf0a0 2023-10-16T09:56:44.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02a7fff700 1 -- 172.21.15.158:0/3818887541 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a800bcc0 con 0x7f02b00cf0a0 2023-10-16T09:56:44.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02b72a5700 1 -- 172.21.15.158:0/3818887541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b00cf0a0 msgr2=0x7f02b00cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02b72a5700 1 --2- 172.21.15.158:0/3818887541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b00cf0a0 0x7f02b00cf530 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009e50 tx=0x7f02a800b330 comp rx=0 tx=0).stop 2023-10-16T09:56:44.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02b72a5700 1 -- 172.21.15.158:0/3818887541 shutdown_connections 2023-10-16T09:56:44.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02b72a5700 1 --2- 172.21.15.158:0/3818887541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b00cf0a0 0x7f02b00cf530 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02b72a5700 1 --2- 172.21.15.158:0/3818887541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b015d670 0x7f02b006d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.458+0000 7f02b72a5700 1 -- 172.21.15.158:0/3818887541 >> 172.21.15.158:0/3818887541 conn(0x7f02b00cdf50 msgr2=0x7f02b00ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:44.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.459+0000 7f02b72a5700 1 -- 172.21.15.158:0/3818887541 shutdown_connections 2023-10-16T09:56:44.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.459+0000 7f02b72a5700 1 -- 172.21.15.158:0/3818887541 wait complete. 2023-10-16T09:56:44.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.461+0000 7f02b72a5700 1 Processor -- start 2023-10-16T09:56:44.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.461+0000 7f02b72a5700 1 -- start start 2023-10-16T09:56:44.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b72a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b0070340 0x7f02b00c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b72a5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02b00cf0a0 0x7f02b00b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b72a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b015d670 0x7f02b00b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b72a5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0160b90 con 0x7f02b015d670 2023-10-16T09:56:44.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b72a5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b015f850 con 0x7f02b00cf0a0 2023-10-16T09:56:44.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b72a5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b015faf0 con 0x7f02b0070340 2023-10-16T09:56:44.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b015d670 0x7f02b00b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b015d670 0x7f02b00b6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50028/0 (socket says 172.21.15.158:50028) 2023-10-16T09:56:44.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 -- 172.21.15.158:0/2205376392 learned_addr learned my addr 172.21.15.158:0/2205376392 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:44.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5041700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b0070340 0x7f02b00c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b4840700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02b00cf0a0 0x7f02b00b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:44.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 -- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b0070340 msgr2=0x7f02b00c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b0070340 0x7f02b00c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 -- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02b00cf0a0 msgr2=0x7f02b00b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02b00cf0a0 0x7f02b00b5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 -- 172.21.15.158:0/2205376392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8009b30 con 0x7f02b015d670 2023-10-16T09:56:44.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.462+0000 7f02b5842700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b015d670 0x7f02b00b6300 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f02a800b680 tx=0x7f02a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:44.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.463+0000 7f02a5ffb700 1 -- 172.21.15.158:0/2205376392 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80048b0 con 0x7f02b015d670 2023-10-16T09:56:44.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.463+0000 7f02a5ffb700 1 -- 172.21.15.158:0/2205376392 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80099b0 con 0x7f02b015d670 2023-10-16T09:56:44.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.463+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b015fdd0 con 0x7f02b015d670 2023-10-16T09:56:44.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.463+0000 7f02a5ffb700 1 -- 172.21.15.158:0/2205376392 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a801c680 con 0x7f02b015d670 2023-10-16T09:56:44.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.463+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0160390 con 0x7f02b015d670 2023-10-16T09:56:44.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.464+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b00681b0 con 0x7f02b015d670 2023-10-16T09:56:44.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.464+0000 7f02a5ffb700 1 -- 172.21.15.158:0/2205376392 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f02a801c7e0 con 0x7f02b015d670 2023-10-16T09:56:44.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.465+0000 7f02a5ffb700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 0x7f029406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:44.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.465+0000 7f02b5041700 1 -- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 msgr2=0x7f029406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:44.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.465+0000 7f02a5ffb700 1 -- 172.21.15.158:0/2205376392 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7204+0+0 (secure 0 0 0) 0x7f02a808b7e0 con 0x7f02b015d670 2023-10-16T09:56:44.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.465+0000 7f02b5041700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 0x7f029406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:44.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.469+0000 7f02a5ffb700 1 -- 172.21.15.158:0/2205376392 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02a8055fb0 con 0x7f02b015d670 2023-10-16T09:56:44.641 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:44.640+0000 7fa54c167200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:56:44.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.666+0000 7f02b5041700 1 -- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 msgr2=0x7f029406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:44.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.666+0000 7f02b5041700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 0x7f029406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:44.713 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:44.711+0000 7fa54c167200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:56:44.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.733+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f02b00c5a20 con 0x7f02b015d670 2023-10-16T09:56:44.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.733+0000 7f02a5ffb700 1 -- 172.21.15.158:0/2205376392 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v54) v1 ==== 80+0+358 (secure 0 0 0) 0x7f02a80595d0 con 0x7f02b015d670 2023-10-16T09:56:44.735 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2730663923 2023-10-17T09:44:59.554471+0000 2023-10-16T09:56:44.735 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/4024924909 2023-10-17T09:44:59.554471+0000 2023-10-16T09:56:44.735 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/4024924909 2023-10-17T09:44:59.554471+0000 2023-10-16T09:56:44.735 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3457698273 2023-10-17T09:44:59.554471+0000 2023-10-16T09:56:44.735 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/686968952 2023-10-17T09:44:59.554471+0000 2023-10-16T09:56:44.736 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/251660717 2023-10-17T09:44:59.554471+0000 2023-10-16T09:56:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 msgr2=0x7f029406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 0x7f029406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b015d670 msgr2=0x7f02b00b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b015d670 0x7f02b00b6300 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f02a800b680 tx=0x7f02a8004ad0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 shutdown_connections 2023-10-16T09:56:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f029406b0b0 0x7f029406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02b015d670 0x7f02b00b6300 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02b0070340 0x7f02b00c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 --2- 172.21.15.158:0/2205376392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02b00cf0a0 0x7f02b00b5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:44.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 >> 172.21.15.158:0/2205376392 conn(0x7f02b00cdf50 msgr2=0x7f02b0161990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:44.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 shutdown_connections 2023-10-16T09:56:44.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:44.735+0000 7f02b72a5700 1 -- 172.21.15.158:0/2205376392 wait complete. 2023-10-16T09:56:44.739 INFO:teuthology.orchestra.run.smithi158.stderr:listed 6 entries 2023-10-16T09:56:44.750 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T09:56:44.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:44.751 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T09:56:44.759 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:44.758+0000 7f0849d4f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:56:44.796 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:56:44.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:56:44.796 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T09:56:44.833 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:44.832+0000 7f0849d4f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:56:44.852 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:56:44.852 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:44.870 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:56:44.870 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:56:44.871 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:44.910 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:56:44.911 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:56:44.911 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:44.928 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:56:44.929 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:56:45.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.075+0000 7fcee5ba2700 1 Processor -- start 2023-10-16T09:56:45.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.075+0000 7fcee5ba2700 1 -- start start 2023-10-16T09:56:45.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcee5ba2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcee5ba2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcee006e5d0 0x7fcee0071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcee5ba2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee0071e50 con 0x7fcee006af20 2023-10-16T09:56:45.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcee5ba2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee0161270 con 0x7fcee006e5d0 2023-10-16T09:56:45.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcee5ba2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0161510 con 0x7fcee006ab40 2023-10-16T09:56:45.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedf7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fced7fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcee006e5d0 0x7fcee0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcee006af20 0x7fcee00722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:54118/0 (socket says 172.21.15.158:54118) 2023-10-16T09:56:45.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedffff700 1 -- 172.21.15.158:0/339047722 learned_addr learned my addr 172.21.15.158:0/339047722 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:45.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedf7fe700 1 -- 172.21.15.158:0/339047722 >> v2:172.21.15.172:3300/0 conn(0x7fcee006e5d0 msgr2=0x7fcee0071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:45.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedf7fe700 1 --2- 172.21.15.158:0/339047722 >> v2:172.21.15.172:3300/0 conn(0x7fcee006e5d0 0x7fcee0071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedf7fe700 1 -- 172.21.15.158:0/339047722 >> v1:172.21.15.172:6789/0 conn(0x7fcee006af20 legacy=0x7fcee00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedf7fe700 1 -- 172.21.15.158:0/339047722 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee01617f0 con 0x7fcee006ab40 2023-10-16T09:56:45.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedeffd700 1 -- 172.21.15.158:0/339047722 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302039504 0 0) 0x7fcee0071e50 con 0x7fcee006af20 2023-10-16T09:56:45.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fced7fff700 1 --2- 172.21.15.158:0/339047722 >> v2:172.21.15.172:3300/0 conn(0x7fcee006e5d0 0x7fcee0071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:45.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.076+0000 7fcedf7fe700 1 --2- 172.21.15.158:0/339047722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee006d820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fced0009840 tx=0x7fced0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1af84b985a7540e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:45.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcedeffd700 1 -- 172.21.15.158:0/339047722 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0011830 con 0x7fcee006ab40 2023-10-16T09:56:45.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcedeffd700 1 -- 172.21.15.158:0/339047722 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0011990 con 0x7fcee006ab40 2023-10-16T09:56:45.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcedeffd700 1 -- 172.21.15.158:0/339047722 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0011d20 con 0x7fcee006ab40 2023-10-16T09:56:45.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 -- 172.21.15.158:0/339047722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 msgr2=0x7fcee006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/339047722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee006d820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fced0009840 tx=0x7fced0009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:45.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 -- 172.21.15.158:0/339047722 shutdown_connections 2023-10-16T09:56:45.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/339047722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee006d820 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/339047722 >> v2:172.21.15.172:3300/0 conn(0x7fcee006e5d0 0x7fcee0071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 -- 172.21.15.158:0/339047722 >> 172.21.15.158:0/339047722 conn(0x7fcee00d02b0 msgr2=0x7fcee00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:45.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 -- 172.21.15.158:0/339047722 shutdown_connections 2023-10-16T09:56:45.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.077+0000 7fcee5ba2700 1 -- 172.21.15.158:0/339047722 wait complete. 2023-10-16T09:56:45.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.078+0000 7fcee5ba2700 1 Processor -- start 2023-10-16T09:56:45.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.078+0000 7fcee5ba2700 1 -- start start 2023-10-16T09:56:45.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcee5ba2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee00cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcee5ba2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee006af20 0x7fcee0155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcee5ba2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcee006e5d0 0x7fcee0155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcee5ba2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c45b0 con 0x7fcee006e5d0 2023-10-16T09:56:45.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcee5ba2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00b5cd0 con 0x7fcee006af20 2023-10-16T09:56:45.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcee5ba2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c3c80 con 0x7fcee006ab40 2023-10-16T09:56:45.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcedffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcee006e5d0 0x7fcee0155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcedf7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee00cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcedffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcee006e5d0 0x7fcee0155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50044/0 (socket says 172.21.15.158:50044) 2023-10-16T09:56:45.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fced7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee006af20 0x7fcee0155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.079+0000 7fcedffff700 1 -- 172.21.15.158:0/3147559774 learned_addr learned my addr 172.21.15.158:0/3147559774 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:45.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedf7fe700 1 -- 172.21.15.158:0/3147559774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee006af20 msgr2=0x7fcee0155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedf7fe700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee006af20 0x7fcee0155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedf7fe700 1 -- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcee006e5d0 msgr2=0x7fcee0155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedf7fe700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcee006e5d0 0x7fcee0155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedf7fe700 1 -- 172.21.15.158:0/3147559774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec80094a0 con 0x7fcee006ab40 2023-10-16T09:56:45.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedf7fe700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee00cb9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fced0016000 tx=0x7fced0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:45.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedcff9700 1 -- 172.21.15.158:0/3147559774 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0004550 con 0x7fcee006ab40 2023-10-16T09:56:45.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedcff9700 1 -- 172.21.15.158:0/3147559774 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced001f070 con 0x7fcee006ab40 2023-10-16T09:56:45.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced00094a0 con 0x7fcee006ab40 2023-10-16T09:56:45.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcedcff9700 1 -- 172.21.15.158:0/3147559774 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced00038e0 con 0x7fcee006ab40 2023-10-16T09:56:45.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.080+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee00c4240 con 0x7fcee006ab40 2023-10-16T09:56:45.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.081+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee00c43d0 con 0x7fcee006ab40 2023-10-16T09:56:45.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.081+0000 7fcedcff9700 1 -- 172.21.15.158:0/3147559774 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fced0003a40 con 0x7fcee006ab40 2023-10-16T09:56:45.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.082+0000 7fcedcff9700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 0x7fcec006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.082+0000 7fced7fff700 1 -- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 msgr2=0x7fcec006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:45.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.082+0000 7fcedcff9700 1 -- 172.21.15.158:0/3147559774 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7fced0021070 con 0x7fcee006ab40 2023-10-16T09:56:45.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.082+0000 7fced7fff700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 0x7fcec006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:45.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.087+0000 7fcedcff9700 1 -- 172.21.15.158:0/3147559774 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fced0055e90 con 0x7fcee006ab40 2023-10-16T09:56:45.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.283+0000 7fced7fff700 1 -- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 msgr2=0x7fcec006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:45.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.283+0000 7fced7fff700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 0x7fcec006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:45.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.335+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcee00681b0 con 0x7fcee006ab40 2023-10-16T09:56:45.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.335+0000 7fcedcff9700 1 -- 172.21.15.158:0/3147559774 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4928 (secure 0 0 0) 0x7fced0015090 con 0x7fcee006ab40 2023-10-16T09:56:45.337 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:45.337 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4283661376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4283661376},{"type":"v1","addr":"172.21.15.158:6835","nonce":4283661376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/3069959147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":3069959147},{"type":"v1","addr":"172.21.15.158:6837","nonce":3069959147}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1583191131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1583191131},{"type":"v1","addr":"172.21.15.172:6835","nonce":1583191131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:44:41.090180+0000","modified":"2023-10-16T09:44:44.131783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34760":{"gid":34760,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.172:6834/1706659835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1706659835},{"type":"v1","addr":"172.21.15.172:6834","nonce":1706659835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T09:56:45.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 msgr2=0x7fcec006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:45.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 0x7fcec006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 msgr2=0x7fcee00cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee00cb9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fced0016000 tx=0x7fced0004750 comp rx=0 tx=0).stop 2023-10-16T09:56:45.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 shutdown_connections 2023-10-16T09:56:45.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7fcec006b0b0 0x7fcec006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcee006e5d0 0x7fcee0155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee006ab40 0x7fcee00cb9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 --2- 172.21.15.158:0/3147559774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee006af20 0x7fcee0155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 >> 172.21.15.158:0/3147559774 conn(0x7fcee00d02b0 msgr2=0x7fcee00ada00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:45.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 shutdown_connections 2023-10-16T09:56:45.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.338+0000 7fcee5ba2700 1 -- 172.21.15.158:0/3147559774 wait complete. 2023-10-16T09:56:45.342 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 17 2023-10-16T09:56:45.349 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:45.349+0000 7fa54c167200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:56:45.353 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:56:45.426 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:45.425+0000 7fa54c167200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:56:45.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.479+0000 7f1aeef07700 1 Processor -- start 2023-10-16T09:56:45.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.479+0000 7f1aeef07700 1 -- start start 2023-10-16T09:56:45.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aeef07700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae80d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aeef07700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ae80d1210 0x7f1ae80ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aeef07700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1ae80ce950 0x7f1ae80b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aeef07700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80d4060 con 0x7f1ae80d1210 2023-10-16T09:56:45.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aeef07700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80d3bd0 con 0x7f1ae80ce950 2023-10-16T09:56:45.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aeef07700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80cede0 con 0x7f1ae815d7c0 2023-10-16T09:56:45.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ae80d1210 0x7f1ae80ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ae80d1210 0x7f1ae80ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50052/0 (socket says 172.21.15.158:50052) 2023-10-16T09:56:45.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 -- 172.21.15.158:0/303131909 learned_addr learned my addr 172.21.15.158:0/303131909 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:45.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aed4a4700 1 --2- 172.21.15.158:0/303131909 >> v2:172.21.15.172:3300/0 conn(0x7f1ae80ce950 0x7f1ae80b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aecca3700 1 --2- 172.21.15.158:0/303131909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae80d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 -- 172.21.15.158:0/303131909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 msgr2=0x7f1ae80d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:45.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 --2- 172.21.15.158:0/303131909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae80d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 -- 172.21.15.158:0/303131909 >> v2:172.21.15.172:3300/0 conn(0x7f1ae80ce950 msgr2=0x7f1ae80b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:45.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 --2- 172.21.15.158:0/303131909 >> v2:172.21.15.172:3300/0 conn(0x7f1ae80ce950 0x7f1ae80b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 -- 172.21.15.158:0/303131909 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80d2cf0 con 0x7f1ae80d1210 2023-10-16T09:56:45.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1aed4a4700 1 --2- 172.21.15.158:0/303131909 >> v2:172.21.15.172:3300/0 conn(0x7f1ae80ce950 0x7f1ae80b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:45.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae7fff700 1 --2- 172.21.15.158:0/303131909 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae80ce410 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80b6190 tx=0x7f1ad4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1890fbdecacc4650 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:45.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae77fe700 1 -- 172.21.15.158:0/303131909 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad400b870 con 0x7f1ae80d1210 2023-10-16T09:56:45.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae77fe700 1 -- 172.21.15.158:0/303131909 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad400b9d0 con 0x7f1ae80d1210 2023-10-16T09:56:45.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.480+0000 7f1ae77fe700 1 -- 172.21.15.158:0/303131909 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad400bd60 con 0x7f1ae80d1210 2023-10-16T09:56:45.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 -- 172.21.15.158:0/303131909 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 msgr2=0x7f1ae80ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 --2- 172.21.15.158:0/303131909 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae80ce410 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80b6190 tx=0x7f1ad4009bd0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 -- 172.21.15.158:0/303131909 shutdown_connections 2023-10-16T09:56:45.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 --2- 172.21.15.158:0/303131909 >> v2:172.21.15.172:3300/0 conn(0x7f1ae80ce950 0x7f1ae80b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 --2- 172.21.15.158:0/303131909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae80d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 --2- 172.21.15.158:0/303131909 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae80ce410 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 -- 172.21.15.158:0/303131909 >> 172.21.15.158:0/303131909 conn(0x7f1ae80669f0 msgr2=0x7f1ae8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:45.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 -- 172.21.15.158:0/303131909 shutdown_connections 2023-10-16T09:56:45.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.481+0000 7f1aeef07700 1 -- 172.21.15.158:0/303131909 wait complete. 2023-10-16T09:56:45.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.482+0000 7f1aeef07700 1 Processor -- start 2023-10-16T09:56:45.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.482+0000 7f1aeef07700 1 -- start start 2023-10-16T09:56:45.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aeef07700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ae80ce950 0x7f1ae81f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aeef07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae8154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aeef07700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae81515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aeef07700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae81f2ba0 con 0x7f1ae80d1210 2023-10-16T09:56:45.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aeef07700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8143260 con 0x7f1ae80ce950 2023-10-16T09:56:45.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aeef07700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae815ef80 con 0x7f1ae815d7c0 2023-10-16T09:56:45.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae8154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae8154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50058/0 (socket says 172.21.15.158:50058) 2023-10-16T09:56:45.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1ae7fff700 1 -- 172.21.15.158:0/140812424 learned_addr learned my addr 172.21.15.158:0/140812424 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:45.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aecca3700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ae80ce950 0x7f1ae81f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.483+0000 7f1aed4a4700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae81515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae7fff700 1 -- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 msgr2=0x7f1ae81515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae7fff700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae81515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae7fff700 1 -- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ae80ce950 msgr2=0x7f1ae81f6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae7fff700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ae80ce950 0x7f1ae81f6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae7fff700 1 -- 172.21.15.158:0/140812424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad4009550 con 0x7f1ae80d1210 2023-10-16T09:56:45.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae7fff700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae8154d90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4000c00 tx=0x7f1ad4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:45.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae57fa700 1 -- 172.21.15.158:0/140812424 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4011830 con 0x7f1ae80d1210 2023-10-16T09:56:45.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae57fa700 1 -- 172.21.15.158:0/140812424 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad401e070 con 0x7f1ae80d1210 2023-10-16T09:56:45.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1ae57fa700 1 -- 172.21.15.158:0/140812424 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4011bd0 con 0x7f1ae80d1210 2023-10-16T09:56:45.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae815e0a0 con 0x7f1ae80d1210 2023-10-16T09:56:45.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.484+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae815e660 con 0x7f1ae80d1210 2023-10-16T09:56:45.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.485+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae8143420 con 0x7f1ae80d1210 2023-10-16T09:56:45.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.485+0000 7f1ae57fa700 1 -- 172.21.15.158:0/140812424 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f1ad4011d30 con 0x7f1ae80d1210 2023-10-16T09:56:45.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.486+0000 7f1ae57fa700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 0x7f1acc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.486+0000 7f1aecca3700 1 -- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 msgr2=0x7f1acc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:45.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.486+0000 7f1ae57fa700 1 -- 172.21.15.158:0/140812424 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f1ad4020070 con 0x7f1ae80d1210 2023-10-16T09:56:45.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.486+0000 7f1aecca3700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 0x7f1acc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:45.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.490+0000 7f1ae57fa700 1 -- 172.21.15.158:0/140812424 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ad4055bd0 con 0x7f1ae80d1210 2023-10-16T09:56:45.504 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:45.503+0000 7f0849d4f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:56:45.568 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:45.568+0000 7fa54c167200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:56:45.584 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:45.583+0000 7f0849d4f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:56:45.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.687+0000 7f1aecca3700 1 -- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 msgr2=0x7f1acc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:45.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.687+0000 7f1aecca3700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 0x7f1acc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:45.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.693+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ae80681b0 con 0x7f1ae80d1210 2023-10-16T09:56:45.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.694+0000 7f1ae57fa700 1 -- 172.21.15.158:0/140812424 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17911 (secure 0 0 0) 0x7f1ad40591f0 con 0x7f1ae80d1210 2023-10-16T09:56:45.696 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:45.696 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":54,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:56:44.527750+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:44:38.770851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T09:44:40.065625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.38","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2730663923":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6832/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6833/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/3457698273":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/686968952":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/251660717":"2023-10-17T09:44:59.554471+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:56:45.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.696+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 msgr2=0x7f1acc06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:45.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.696+0000 7f1aeef07700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 0x7f1acc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.696+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 msgr2=0x7f1ae8154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.696+0000 7f1aeef07700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae8154d90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4000c00 tx=0x7f1ad4004750 comp rx=0 tx=0).stop 2023-10-16T09:56:45.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 shutdown_connections 2023-10-16T09:56:45.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ae815d7c0 0x7f1ae81515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ae80ce950 0x7f1ae81f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f1acc06b0b0 0x7f1acc06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 --2- 172.21.15.158:0/140812424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ae80d1210 0x7f1ae8154d90 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 >> 172.21.15.158:0/140812424 conn(0x7f1ae80669f0 msgr2=0x7f1ae80c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:45.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 shutdown_connections 2023-10-16T09:56:45.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.697+0000 7f1aeef07700 1 -- 172.21.15.158:0/140812424 wait complete. 2023-10-16T09:56:45.711 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:44:38.770851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:44:40.065625+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 221, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:56:45.712 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:56:45.736 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:45.735+0000 7f0849d4f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:56:45.792 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:45.791+0000 7fa54c167200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:56:45.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.838+0000 7f595589f700 1 Processor -- start 2023-10-16T09:56:45.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.838+0000 7f595589f700 1 -- start start 2023-10-16T09:56:45.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f595589f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59500ce5b0 0x7f59500cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f595589f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f59500cc010 0x7f595015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f595589f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59500cd200 con 0x7f595015c2b0 2023-10-16T09:56:45.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f595589f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59500cc490 con 0x7f59500ce5b0 2023-10-16T09:56:45.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f595589f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59500cfca0 con 0x7f59500cc010 2023-10-16T09:56:45.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59500ce5b0 0x7f59500cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f595015c2b0 0x7f595015f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50270/0 (socket says 172.21.15.158:50270) 2023-10-16T09:56:45.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594f7fe700 1 -- 172.21.15.158:0/10240045 learned_addr learned my addr 172.21.15.158:0/10240045 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:45.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594e7fc700 1 --2- 172.21.15.158:0/10240045 >> v2:172.21.15.172:3300/0 conn(0x7f59500cc010 0x7f595015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594effd700 1 -- 172.21.15.158:0/10240045 >> v2:172.21.15.172:3300/0 conn(0x7f59500cc010 msgr2=0x7f595015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:45.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594effd700 1 --2- 172.21.15.158:0/10240045 >> v2:172.21.15.172:3300/0 conn(0x7f59500cc010 0x7f595015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594effd700 1 -- 172.21.15.158:0/10240045 >> v1:172.21.15.158:6789/0 conn(0x7f595015c2b0 legacy=0x7f595015f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594effd700 1 -- 172.21.15.158:0/10240045 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595015caa0 con 0x7f59500ce5b0 2023-10-16T09:56:45.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.839+0000 7f594dffb700 1 -- 172.21.15.158:0/10240045 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761706536 0 0) 0x7f59500cd200 con 0x7f595015c2b0 2023-10-16T09:56:45.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.840+0000 7f594effd700 1 --2- 172.21.15.158:0/10240045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59500ce5b0 0x7f59500cbad0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f59500b0b70 tx=0x7f5940009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4991ba2cd665e86c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:45.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.840+0000 7f594dffb700 1 -- 172.21.15.158:0/10240045 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940011830 con 0x7f59500ce5b0 2023-10-16T09:56:45.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.840+0000 7f594dffb700 1 -- 172.21.15.158:0/10240045 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5940011990 con 0x7f59500ce5b0 2023-10-16T09:56:45.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.840+0000 7f594dffb700 1 -- 172.21.15.158:0/10240045 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940011d20 con 0x7f59500ce5b0 2023-10-16T09:56:45.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 -- 172.21.15.158:0/10240045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59500ce5b0 msgr2=0x7f59500cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 --2- 172.21.15.158:0/10240045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59500ce5b0 0x7f59500cbad0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f59500b0b70 tx=0x7f5940009b20 comp rx=0 tx=0).stop 2023-10-16T09:56:45.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 -- 172.21.15.158:0/10240045 shutdown_connections 2023-10-16T09:56:45.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 --2- 172.21.15.158:0/10240045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59500ce5b0 0x7f59500cbad0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 --2- 172.21.15.158:0/10240045 >> v2:172.21.15.172:3300/0 conn(0x7f59500cc010 0x7f595015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 -- 172.21.15.158:0/10240045 >> 172.21.15.158:0/10240045 conn(0x7f595006ec00 msgr2=0x7f595006efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:45.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 -- 172.21.15.158:0/10240045 shutdown_connections 2023-10-16T09:56:45.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.841+0000 7f595589f700 1 -- 172.21.15.158:0/10240045 wait complete. 2023-10-16T09:56:45.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.842+0000 7f595589f700 1 Processor -- start 2023-10-16T09:56:45.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.842+0000 7f595589f700 1 -- start start 2023-10-16T09:56:45.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f595589f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59500cc010 0x7f5950156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f595589f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59500ce5b0 0x7f59501f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f595589f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f595015c2b0 0x7f5950154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f595589f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5950153f40 con 0x7f595015c2b0 2023-10-16T09:56:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f595589f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5950156ff0 con 0x7f59500ce5b0 2023-10-16T09:56:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f595589f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5950153880 con 0x7f59500cc010 2023-10-16T09:56:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f595015c2b0 0x7f5950154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f595015c2b0 0x7f5950154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50100/0 (socket says 172.21.15.158:50100) 2023-10-16T09:56:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 -- 172.21.15.158:0/1457384309 learned_addr learned my addr 172.21.15.158:0/1457384309 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:45.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594effd700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59500cc010 0x7f5950156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594e7fc700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59500ce5b0 0x7f59501f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:45.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 -- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59500cc010 msgr2=0x7f5950156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59500cc010 0x7f5950156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 -- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59500ce5b0 msgr2=0x7f59501f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:45.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59500ce5b0 0x7f59501f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:45.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 -- 172.21.15.158:0/1457384309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400094a0 con 0x7f595015c2b0 2023-10-16T09:56:45.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f594f7fe700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f595015c2b0 0x7f5950154360 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f594800cca0 tx=0x7f594800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:45.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.843+0000 7f5937fff700 1 -- 172.21.15.158:0/1457384309 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594800ab70 con 0x7f595015c2b0 2023-10-16T09:56:45.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.844+0000 7f5937fff700 1 -- 172.21.15.158:0/1457384309 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594800acd0 con 0x7f595015c2b0 2023-10-16T09:56:45.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.844+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5950153b60 con 0x7f595015c2b0 2023-10-16T09:56:45.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.844+0000 7f5937fff700 1 -- 172.21.15.158:0/1457384309 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948007310 con 0x7f595015c2b0 2023-10-16T09:56:45.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.844+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59501566d0 con 0x7f595015c2b0 2023-10-16T09:56:45.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.845+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5950154080 con 0x7f595015c2b0 2023-10-16T09:56:45.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.845+0000 7f5937fff700 1 -- 172.21.15.158:0/1457384309 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f5948007470 con 0x7f595015c2b0 2023-10-16T09:56:45.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.845+0000 7f5937fff700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:45.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.845+0000 7f594effd700 1 -- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 msgr2=0x7f593c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:45.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.846+0000 7f5937fff700 1 -- 172.21.15.158:0/1457384309 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f5948089870 con 0x7f595015c2b0 2023-10-16T09:56:45.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.846+0000 7f594effd700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:45.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:45.854+0000 7f5937fff700 1 -- 172.21.15.158:0/1457384309 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5950154080 con 0x7f595015c2b0 2023-10-16T09:56:45.865 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:45.864+0000 7fa54c167200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:56:45.978 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:45.976+0000 7f0849d4f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:56:46.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.047+0000 7f594effd700 1 -- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 msgr2=0x7f593c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:46.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.047+0000 7f594effd700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:46.053 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:46.052+0000 7f0849d4f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:56:46.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.053+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5950154080 con 0x7f595015c2b0 2023-10-16T09:56:46.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.054+0000 7f5937fff700 1 -- 172.21.15.158:0/1457384309 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5948018020 con 0x7f595015c2b0 2023-10-16T09:56:46.055 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:56:46.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 msgr2=0x7f593c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:46.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f595015c2b0 msgr2=0x7f5950154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f595015c2b0 0x7f5950154360 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f594800cca0 tx=0x7f594800a060 comp rx=0 tx=0).stop 2023-10-16T09:56:46.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 shutdown_connections 2023-10-16T09:56:46.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f595015c2b0 0x7f5950154360 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59500cc010 0x7f5950156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 --2- 172.21.15.158:0/1457384309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59500ce5b0 0x7f59501f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 >> 172.21.15.158:0/1457384309 conn(0x7f595006ec00 msgr2=0x7f595015b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:46.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 shutdown_connections 2023-10-16T09:56:46.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.058+0000 7f595589f700 1 -- 172.21.15.158:0/1457384309 wait complete. 2023-10-16T09:56:46.071 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:56:46.129 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:46.129+0000 7fa54c167200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:56:46.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.241+0000 7f7ad7feb700 1 Processor -- start 2023-10-16T09:56:46.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.241+0000 7f7ad7feb700 1 -- start start 2023-10-16T09:56:46.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad7feb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ad015dc40 0x7f7ad015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad7feb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ad006d9c0 0x7f7ad006de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad7feb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad015f380 con 0x7f7ad006f180 2023-10-16T09:56:46.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad7feb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad006e3b0 con 0x7f7ad015dc40 2023-10-16T09:56:46.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad7feb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad0071e70 con 0x7f7ad006d9c0 2023-10-16T09:56:46.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad5d87700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ad015dc40 0x7f7ad015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad6588700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7ad006f180 0x7f7ad006f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50286/0 (socket says 172.21.15.158:50286) 2023-10-16T09:56:46.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad6588700 1 -- 172.21.15.158:0/3249833699 learned_addr learned my addr 172.21.15.158:0/3249833699 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:46.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.242+0000 7f7ad5586700 1 --2- 172.21.15.158:0/3249833699 >> v2:172.21.15.172:3300/0 conn(0x7f7ad006d9c0 0x7f7ad006de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad5d87700 1 -- 172.21.15.158:0/3249833699 >> v2:172.21.15.172:3300/0 conn(0x7f7ad006d9c0 msgr2=0x7f7ad006de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:46.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad5d87700 1 --2- 172.21.15.158:0/3249833699 >> v2:172.21.15.172:3300/0 conn(0x7f7ad006d9c0 0x7f7ad006de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad5d87700 1 -- 172.21.15.158:0/3249833699 >> v1:172.21.15.158:6789/0 conn(0x7f7ad006f180 legacy=0x7f7ad006f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad5d87700 1 -- 172.21.15.158:0/3249833699 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad0072150 con 0x7f7ad015dc40 2023-10-16T09:56:46.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad5d87700 1 --2- 172.21.15.158:0/3249833699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad015dc40 0x7f7ad015ee40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7ad004fc30 tx=0x7f7acc00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e662cebf285f3eaa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:46.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad4d85700 1 -- 172.21.15.158:0/3249833699 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc00dc70 con 0x7f7ad015dc40 2023-10-16T09:56:46.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad4d85700 1 -- 172.21.15.158:0/3249833699 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc00ddd0 con 0x7f7ad015dc40 2023-10-16T09:56:46.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.243+0000 7f7ad4d85700 1 -- 172.21.15.158:0/3249833699 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc010690 con 0x7f7ad015dc40 2023-10-16T09:56:46.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 -- 172.21.15.158:0/3249833699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad015dc40 msgr2=0x7f7ad015ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/3249833699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad015dc40 0x7f7ad015ee40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7ad004fc30 tx=0x7f7acc00bdc0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 -- 172.21.15.158:0/3249833699 shutdown_connections 2023-10-16T09:56:46.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/3249833699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad015dc40 0x7f7ad015ee40 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/3249833699 >> v2:172.21.15.172:3300/0 conn(0x7f7ad006d9c0 0x7f7ad006de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 -- 172.21.15.158:0/3249833699 >> 172.21.15.158:0/3249833699 conn(0x7f7ad00d03b0 msgr2=0x7f7ad00d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:46.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 -- 172.21.15.158:0/3249833699 shutdown_connections 2023-10-16T09:56:46.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.244+0000 7f7ad7feb700 1 -- 172.21.15.158:0/3249833699 wait complete. 2023-10-16T09:56:46.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.246+0000 7f7ad7feb700 1 Processor -- start 2023-10-16T09:56:46.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.246+0000 7f7ad7feb700 1 -- start start 2023-10-16T09:56:46.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad7feb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad006d9c0 0x7f7ad00aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad7feb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad006f180 0x7f7ad006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad7feb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad015dc40 0x7f7ad006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad7feb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0156730 con 0x7f7ad006f180 2023-10-16T09:56:46.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad7feb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0157590 con 0x7f7ad015dc40 2023-10-16T09:56:46.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad7feb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0157a20 con 0x7f7ad006d9c0 2023-10-16T09:56:46.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad5586700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad006f180 0x7f7ad006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad5586700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad006f180 0x7f7ad006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50114/0 (socket says 172.21.15.158:50114) 2023-10-16T09:56:46.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad5586700 1 -- 172.21.15.158:0/4131617627 learned_addr learned my addr 172.21.15.158:0/4131617627 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:46.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad5d87700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad006d9c0 0x7f7ad00aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.247+0000 7f7ad6588700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad015dc40 0x7f7ad006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad5586700 1 -- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad006d9c0 msgr2=0x7f7ad00aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad5586700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad006d9c0 0x7f7ad00aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad5586700 1 -- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad015dc40 msgr2=0x7f7ad006ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad5586700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad015dc40 0x7f7ad006ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad5586700 1 -- 172.21.15.158:0/4131617627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc00b740 con 0x7f7ad006f180 2023-10-16T09:56:46.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad5586700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad006f180 0x7f7ad006a8d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e9e0 tx=0x7f7ac000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:46.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ac67fc700 1 -- 172.21.15.158:0/4131617627 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0009710 con 0x7f7ad006f180 2023-10-16T09:56:46.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ac67fc700 1 -- 172.21.15.158:0/4131617627 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac000ee80 con 0x7f7ad006f180 2023-10-16T09:56:46.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad00af070 con 0x7f7ad006f180 2023-10-16T09:56:46.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ac67fc700 1 -- 172.21.15.158:0/4131617627 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000ca50 con 0x7f7ad006f180 2023-10-16T09:56:46.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.248+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad0156c60 con 0x7f7ad006f180 2023-10-16T09:56:46.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.249+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad00681b0 con 0x7f7ad006f180 2023-10-16T09:56:46.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.249+0000 7f7ac67fc700 1 -- 172.21.15.158:0/4131617627 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f7ac000cbb0 con 0x7f7ad006f180 2023-10-16T09:56:46.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.250+0000 7f7ac67fc700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 0x7f7ab406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.250+0000 7f7ad5d87700 1 -- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 msgr2=0x7f7ab406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:46.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.250+0000 7f7ac67fc700 1 -- 172.21.15.158:0/4131617627 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f7ac0089dc0 con 0x7f7ad006f180 2023-10-16T09:56:46.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.250+0000 7f7ad5d87700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 0x7f7ab406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:46.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.254+0000 7f7ac67fc700 1 -- 172.21.15.158:0/4131617627 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7ac0054740 con 0x7f7ad006f180 2023-10-16T09:56:46.348 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:46.348+0000 7fa54c167200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:56:46.349 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:46.347+0000 7f0849d4f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:56:46.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.451+0000 7f7ad5d87700 1 -- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 msgr2=0x7f7ab406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:46.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.451+0000 7f7ad5d87700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 0x7f7ab406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:46.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.467+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7ad006e920 con 0x7f7ad006f180 2023-10-16T09:56:46.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.468+0000 7f7ac67fc700 1 -- 172.21.15.158:0/4131617627 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7ac00210a0 con 0x7f7ad006f180 2023-10-16T09:56:46.469 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:56:46.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 msgr2=0x7f7ab406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:46.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 0x7f7ab406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad006f180 msgr2=0x7f7ad006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad006f180 0x7f7ad006a8d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e9e0 tx=0x7f7ac000ecf0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 shutdown_connections 2023-10-16T09:56:46.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad015dc40 0x7f7ad006ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad006d9c0 0x7f7ad00aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7ab406b0b0 0x7f7ab406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 --2- 172.21.15.158:0/4131617627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ad006f180 0x7f7ad006a8d0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 >> 172.21.15.158:0/4131617627 conn(0x7f7ad00d03b0 msgr2=0x7f7ad015ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:46.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 shutdown_connections 2023-10-16T09:56:46.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.470+0000 7f7ad7feb700 1 -- 172.21.15.158:0/4131617627 wait complete. 2023-10-16T09:56:46.483 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:56:46.495 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:46.495+0000 7fa54c167200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:56:46.577 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:46.576+0000 7f0849d4f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:56:46.607 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:46.607+0000 7fa54c167200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:56:46.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.609+0000 7f03f8ede700 1 Processor -- start 2023-10-16T09:56:46.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.609+0000 7f03f8ede700 1 -- start start 2023-10-16T09:56:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f8ede700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03f406d5d0 0x7f03f4070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f8ede700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f40cbaf0 0x7f03f40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f8ede700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f415dd70 con 0x7f03f415d990 2023-10-16T09:56:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f8ede700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f41616b0 con 0x7f03f406d5d0 2023-10-16T09:56:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f8ede700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f406da50 con 0x7f03f40cbaf0 2023-10-16T09:56:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03f406d5d0 0x7f03f4070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f03f415d990 0x7f03f4161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50298/0 (socket says 172.21.15.158:50298) 2023-10-16T09:56:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f259c700 1 -- 172.21.15.158:0/844695215 learned_addr learned my addr 172.21.15.158:0/844695215 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f2d9d700 1 --2- 172.21.15.158:0/844695215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f40cbaf0 0x7f03f40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f1d9b700 1 -- 172.21.15.158:0/844695215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f40cbaf0 msgr2=0x7f03f40cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f1d9b700 1 --2- 172.21.15.158:0/844695215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f40cbaf0 0x7f03f40cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f1d9b700 1 -- 172.21.15.158:0/844695215 >> v1:172.21.15.158:6789/0 conn(0x7f03f415d990 legacy=0x7f03f4161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f1d9b700 1 -- 172.21.15.158:0/844695215 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f4070a00 con 0x7f03f406d5d0 2023-10-16T09:56:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f1d9b700 1 --2- 172.21.15.158:0/844695215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f406d5d0 0x7f03f4070300 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009870 tx=0x7f03e8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce2c98706b00df5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f159a700 1 -- 172.21.15.158:0/844695215 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8011830 con 0x7f03f406d5d0 2023-10-16T09:56:46.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f159a700 1 -- 172.21.15.158:0/844695215 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e8011990 con 0x7f03f406d5d0 2023-10-16T09:56:46.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.610+0000 7f03f159a700 1 -- 172.21.15.158:0/844695215 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8011d20 con 0x7f03f406d5d0 2023-10-16T09:56:46.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 -- 172.21.15.158:0/844695215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f406d5d0 msgr2=0x7f03f4070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 --2- 172.21.15.158:0/844695215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f406d5d0 0x7f03f4070300 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009870 tx=0x7f03e8009b80 comp rx=0 tx=0).stop 2023-10-16T09:56:46.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 -- 172.21.15.158:0/844695215 shutdown_connections 2023-10-16T09:56:46.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 --2- 172.21.15.158:0/844695215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f40cbaf0 0x7f03f40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 --2- 172.21.15.158:0/844695215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f406d5d0 0x7f03f4070300 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 -- 172.21.15.158:0/844695215 >> 172.21.15.158:0/844695215 conn(0x7f03f40cdf20 msgr2=0x7f03f40ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:46.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 -- 172.21.15.158:0/844695215 shutdown_connections 2023-10-16T09:56:46.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.611+0000 7f03f8ede700 1 -- 172.21.15.158:0/844695215 wait complete. 2023-10-16T09:56:46.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 Processor -- start 2023-10-16T09:56:46.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 -- start start 2023-10-16T09:56:46.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f406d5d0 0x7f03f414d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03f40cbaf0 0x7f03f40c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f415d990 0x7f03f40cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40b5210 con 0x7f03f415d990 2023-10-16T09:56:46.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40b5460 con 0x7f03f40cbaf0 2023-10-16T09:56:46.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f8ede700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40b5740 con 0x7f03f406d5d0 2023-10-16T09:56:46.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.613+0000 7f03f2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f415d990 0x7f03f40cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f406d5d0 0x7f03f414d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f415d990 0x7f03f40cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50130/0 (socket says 172.21.15.158:50130) 2023-10-16T09:56:46.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 -- 172.21.15.158:0/2858456531 learned_addr learned my addr 172.21.15.158:0/2858456531 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:46.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f1d9b700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03f40cbaf0 0x7f03f40c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 -- 172.21.15.158:0/2858456531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f406d5d0 msgr2=0x7f03f414d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f406d5d0 0x7f03f414d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 -- 172.21.15.158:0/2858456531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03f40cbaf0 msgr2=0x7f03f40c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03f40cbaf0 0x7f03f40c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 -- 172.21.15.158:0/2858456531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e80094a0 con 0x7f03f415d990 2023-10-16T09:56:46.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f2d9d700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f415d990 0x7f03f40cc4d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00efc0 tx=0x7f03ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:46.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03e2ffd700 1 -- 172.21.15.158:0/2858456531 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec009ea0 con 0x7f03f415d990 2023-10-16T09:56:46.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03e2ffd700 1 -- 172.21.15.158:0/2858456531 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec00f040 con 0x7f03f415d990 2023-10-16T09:56:46.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f40b5a20 con 0x7f03f415d990 2023-10-16T09:56:46.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03e2ffd700 1 -- 172.21.15.158:0/2858456531 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec00ced0 con 0x7f03f415d990 2023-10-16T09:56:46.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.614+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f40c8ab0 con 0x7f03f415d990 2023-10-16T09:56:46.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.615+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f40b5f40 con 0x7f03f415d990 2023-10-16T09:56:46.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.616+0000 7f03e2ffd700 1 -- 172.21.15.158:0/2858456531 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f03ec014480 con 0x7f03f415d990 2023-10-16T09:56:46.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.617+0000 7f03e2ffd700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 0x7f03d406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.617+0000 7f03f259c700 1 -- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 msgr2=0x7f03d406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.617+0000 7f03e2ffd700 1 -- 172.21.15.158:0/2858456531 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f03ec08a6b0 con 0x7f03f415d990 2023-10-16T09:56:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.617+0000 7f03f259c700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 0x7f03d406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.621+0000 7f03e2ffd700 1 -- 172.21.15.158:0/2858456531 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03f40b5f40 con 0x7f03f415d990 2023-10-16T09:56:46.734 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:46.733+0000 7f0849d4f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:56:46.760 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:46.760+0000 7fa54c167200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:56:46.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.818+0000 7f03f259c700 1 -- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 msgr2=0x7f03d406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:46.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.818+0000 7f03f259c700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 0x7f03d406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.822+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03f40b5f40 con 0x7f03f415d990 2023-10-16T09:56:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.822+0000 7f03e2ffd700 1 -- 172.21.15.158:0/2858456531 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f03f40b5f40 con 0x7f03f415d990 2023-10-16T09:56:46.824 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:56:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 msgr2=0x7f03d406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 0x7f03d406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f415d990 msgr2=0x7f03f40cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f415d990 0x7f03f40cc4d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00efc0 tx=0x7f03ec00c3a0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 shutdown_connections 2023-10-16T09:56:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f03d406b0b0 0x7f03d406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03f415d990 0x7f03f40cc4d0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03f406d5d0 0x7f03f414d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 --2- 172.21.15.158:0/2858456531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03f40cbaf0 0x7f03f40c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 >> 172.21.15.158:0/2858456531 conn(0x7f03f40cdf20 msgr2=0x7f03f415ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 shutdown_connections 2023-10-16T09:56:46.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.825+0000 7f03f8ede700 1 -- 172.21.15.158:0/2858456531 wait complete. 2023-10-16T09:56:46.839 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:56:46.853 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:46.852+0000 7f0849d4f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:56:46.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.961+0000 7ff384ff7700 1 Processor -- start 2023-10-16T09:56:46.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.961+0000 7ff384ff7700 1 -- start start 2023-10-16T09:56:46.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff384ff7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff3800ce530 0x7ff3800b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff384ff7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3800cc1b0 0x7ff3800cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff384ff7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38015b930 con 0x7ff38015f0f0 2023-10-16T09:56:46.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff384ff7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800cf020 con 0x7ff3800cc1b0 2023-10-16T09:56:46.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff384ff7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38015f5f0 con 0x7ff3800ce530 2023-10-16T09:56:46.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3800cc1b0 0x7ff3800cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3800cc1b0 0x7ff3800cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50140/0 (socket says 172.21.15.158:50140) 2023-10-16T09:56:46.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 -- 172.21.15.158:0/1143452063 learned_addr learned my addr 172.21.15.158:0/1143452063 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:46.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37e59c700 1 --2- 172.21.15.158:0/1143452063 >> v2:172.21.15.172:3300/0 conn(0x7ff3800ce530 0x7ff3800b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 -- 172.21.15.158:0/1143452063 >> v2:172.21.15.172:3300/0 conn(0x7ff3800ce530 msgr2=0x7ff3800b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:46.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 --2- 172.21.15.158:0/1143452063 >> v2:172.21.15.172:3300/0 conn(0x7ff3800ce530 0x7ff3800b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 -- 172.21.15.158:0/1143452063 >> v1:172.21.15.172:6789/0 conn(0x7ff38015f0f0 legacy=0x7ff3800cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 -- 172.21.15.158:0/1143452063 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38015f8d0 con 0x7ff3800cc1b0 2023-10-16T09:56:46.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.962+0000 7ff37ed9d700 1 --2- 172.21.15.158:0/1143452063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff3800cf470 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff38006b8e0 tx=0x7ff37800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed204da213300fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:46.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff37d59a700 1 -- 172.21.15.158:0/1143452063 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37800e070 con 0x7ff3800cc1b0 2023-10-16T09:56:46.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff37d59a700 1 -- 172.21.15.158:0/1143452063 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37800b910 con 0x7ff3800cc1b0 2023-10-16T09:56:46.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff37d59a700 1 -- 172.21.15.158:0/1143452063 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37800bcc0 con 0x7ff3800cc1b0 2023-10-16T09:56:46.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 -- 172.21.15.158:0/1143452063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 msgr2=0x7ff3800cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 --2- 172.21.15.158:0/1143452063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff3800cf470 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff38006b8e0 tx=0x7ff37800b330 comp rx=0 tx=0).stop 2023-10-16T09:56:46.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 -- 172.21.15.158:0/1143452063 shutdown_connections 2023-10-16T09:56:46.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 --2- 172.21.15.158:0/1143452063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff3800cf470 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 --2- 172.21.15.158:0/1143452063 >> v2:172.21.15.172:3300/0 conn(0x7ff3800ce530 0x7ff3800b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 -- 172.21.15.158:0/1143452063 >> 172.21.15.158:0/1143452063 conn(0x7ff38006a8d0 msgr2=0x7ff38006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:46.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 -- 172.21.15.158:0/1143452063 shutdown_connections 2023-10-16T09:56:46.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.963+0000 7ff384ff7700 1 -- 172.21.15.158:0/1143452063 wait complete. 2023-10-16T09:56:46.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.965+0000 7ff384ff7700 1 Processor -- start 2023-10-16T09:56:46.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.965+0000 7ff384ff7700 1 -- start start 2023-10-16T09:56:46.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.965+0000 7ff384ff7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff380155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.965+0000 7ff384ff7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3800ce530 0x7ff38006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.965+0000 7ff384ff7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff38015f0f0 0x7ff3800c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.965+0000 7ff384ff7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380159cd0 con 0x7ff3800cc1b0 2023-10-16T09:56:46.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.965+0000 7ff384ff7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380156420 con 0x7ff3800ce530 2023-10-16T09:56:46.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff384ff7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38006cab0 con 0x7ff38015f0f0 2023-10-16T09:56:46.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff380155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff380155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50142/0 (socket says 172.21.15.158:50142) 2023-10-16T09:56:46.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 -- 172.21.15.158:0/4222983198 learned_addr learned my addr 172.21.15.158:0/4222983198 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:46.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37dd9b700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3800ce530 0x7ff38006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37ed9d700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff38015f0f0 0x7ff3800c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:46.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 -- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff38015f0f0 msgr2=0x7ff3800c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff38015f0f0 0x7ff3800c6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 -- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3800ce530 msgr2=0x7ff38006b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:46.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3800ce530 0x7ff38006b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:46.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 -- 172.21.15.158:0/4222983198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378009b30 con 0x7ff3800cc1b0 2023-10-16T09:56:46.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37ed9d700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff38015f0f0 0x7ff3800c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:46.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff37e59c700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff380155ee0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff37000e9b0 tx=0x7ff37000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:46.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff36effd700 1 -- 172.21.15.158:0/4222983198 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370009710 con 0x7ff3800cc1b0 2023-10-16T09:56:46.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff36effd700 1 -- 172.21.15.158:0/4222983198 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37000c850 con 0x7ff3800cc1b0 2023-10-16T09:56:46.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff36effd700 1 -- 172.21.15.158:0/4222983198 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37000cbc0 con 0x7ff3800cc1b0 2023-10-16T09:56:46.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38006c210 con 0x7ff3800cc1b0 2023-10-16T09:56:46.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.966+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38006c7d0 con 0x7ff3800cc1b0 2023-10-16T09:56:46.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.967+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff380159fb0 con 0x7ff3800cc1b0 2023-10-16T09:56:46.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.968+0000 7ff36effd700 1 -- 172.21.15.158:0/4222983198 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7ff37000cd20 con 0x7ff3800cc1b0 2023-10-16T09:56:46.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.968+0000 7ff36effd700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 0x7ff36006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:46.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.968+0000 7ff37dd9b700 1 -- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 msgr2=0x7ff36006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:46.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.968+0000 7ff36effd700 1 -- 172.21.15.158:0/4222983198 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7ff370089d40 con 0x7ff3800cc1b0 2023-10-16T09:56:46.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.968+0000 7ff37dd9b700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 0x7ff36006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:46.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:46.972+0000 7ff36effd700 1 -- 172.21.15.158:0/4222983198 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3700547e0 con 0x7ff3800cc1b0 2023-10-16T09:56:47.015 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:47.014+0000 7f0849d4f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:56:47.021 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:47.021+0000 7fa54c167200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:56:47.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.169+0000 7ff37dd9b700 1 -- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 msgr2=0x7ff36006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:47.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.169+0000 7ff37dd9b700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 0x7ff36006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:47.177 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:47.177+0000 7fa54c167200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:56:47.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.219+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff380074a90 con 0x7ff3800cc1b0 2023-10-16T09:56:47.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.220+0000 7ff36effd700 1 -- 172.21.15.158:0/4222983198 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4928 (secure 0 0 0) 0x7ff370021090 con 0x7ff3800cc1b0 2023-10-16T09:56:47.221 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:47.222 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4283661376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4283661376},{"type":"v1","addr":"172.21.15.158:6835","nonce":4283661376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/3069959147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":3069959147},{"type":"v1","addr":"172.21.15.158:6837","nonce":3069959147}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1583191131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1583191131},{"type":"v1","addr":"172.21.15.172:6835","nonce":1583191131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:44:41.090180+0000","modified":"2023-10-16T09:44:44.131783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34760":{"gid":34760,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.172:6834/1706659835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1706659835},{"type":"v1","addr":"172.21.15.172:6834","nonce":1706659835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T09:56:47.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 msgr2=0x7ff36006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 0x7ff36006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 msgr2=0x7ff380155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff380155ee0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff37000e9b0 tx=0x7ff37000ecc0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 shutdown_connections 2023-10-16T09:56:47.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff38015f0f0 0x7ff3800c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7ff36006b0b0 0x7ff36006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3800cc1b0 0x7ff380155ee0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 --2- 172.21.15.158:0/4222983198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3800ce530 0x7ff38006b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 >> 172.21.15.158:0/4222983198 conn(0x7ff38006a8d0 msgr2=0x7ff3800b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:47.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 shutdown_connections 2023-10-16T09:56:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.222+0000 7ff384ff7700 1 -- 172.21.15.158:0/4222983198 wait complete. 2023-10-16T09:56:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 17 2023-10-16T09:56:47.236 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:56:47.290 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:47.288+0000 7f0849d4f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:56:47.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.398+0000 7f7ea18ee700 1 Processor -- start 2023-10-16T09:56:47.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.398+0000 7f7ea18ee700 1 -- start start 2023-10-16T09:56:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.399+0000 7f7ea18ee700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7e9c15d7c0 0x7f7e9c0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.399+0000 7f7ea18ee700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7e9c0ceb20 0x7f7e9c0b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.399+0000 7f7ea18ee700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c0d3bd0 con 0x7f7e9c0d1210 2023-10-16T09:56:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.399+0000 7f7ea18ee700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c0d4060 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.399+0000 7f7ea18ee700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c0d2cf0 con 0x7f7e9c15d7c0 2023-10-16T09:56:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.399+0000 7f7e9b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7e9c0ceb20 0x7f7e9c0b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.399+0000 7f7e9b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7e9c0ceb20 0x7f7e9c0b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50156/0 (socket says 172.21.15.158:50156) 2023-10-16T09:56:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e9b7fe700 1 -- 172.21.15.158:0/3023214986 learned_addr learned my addr 172.21.15.158:0/3023214986 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e9affd700 1 --2- 172.21.15.158:0/3023214986 >> v2:172.21.15.172:3300/0 conn(0x7f7e9c15d7c0 0x7f7e9c0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e9b7fe700 1 -- 172.21.15.158:0/3023214986 >> v2:172.21.15.172:3300/0 conn(0x7f7e9c15d7c0 msgr2=0x7f7e9c0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e9b7fe700 1 --2- 172.21.15.158:0/3023214986 >> v2:172.21.15.172:3300/0 conn(0x7f7e9c15d7c0 0x7f7e9c0d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e9b7fe700 1 -- 172.21.15.158:0/3023214986 >> v1:172.21.15.172:6789/0 conn(0x7f7e9c0d1210 legacy=0x7f7e9c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e9b7fe700 1 -- 172.21.15.158:0/3023214986 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c0d2fc0 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e9b7fe700 1 --2- 172.21.15.158:0/3023214986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c0b6800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0d0bc0 tx=0x7f7e9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa642dd7046d011e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e99ffb700 1 -- 172.21.15.158:0/3023214986 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9400e070 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e99ffb700 1 -- 172.21.15.158:0/3023214986 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e94003a70 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.400+0000 7f7e99ffb700 1 -- 172.21.15.158:0/3023214986 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94003e20 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 -- 172.21.15.158:0/3023214986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 msgr2=0x7f7e9c0b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/3023214986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c0b6800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0d0bc0 tx=0x7f7e9400b330 comp rx=0 tx=0).stop 2023-10-16T09:56:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 -- 172.21.15.158:0/3023214986 shutdown_connections 2023-10-16T09:56:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/3023214986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c0b6800 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/3023214986 >> v2:172.21.15.172:3300/0 conn(0x7f7e9c15d7c0 0x7f7e9c0d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 -- 172.21.15.158:0/3023214986 >> 172.21.15.158:0/3023214986 conn(0x7f7e9c0669f0 msgr2=0x7f7e9c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 -- 172.21.15.158:0/3023214986 shutdown_connections 2023-10-16T09:56:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.401+0000 7f7ea18ee700 1 -- 172.21.15.158:0/3023214986 wait complete. 2023-10-16T09:56:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.402+0000 7f7ea18ee700 1 Processor -- start 2023-10-16T09:56:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.402+0000 7f7ea18ee700 1 -- start start 2023-10-16T09:56:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7ea18ee700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c1f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7ea18ee700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e9c0d1210 0x7f7e9c1f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7ea18ee700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e9c15d7c0 0x7f7e9c14e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7ea18ee700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c14f2b0 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7ea18ee700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c1f6900 con 0x7f7e9c15d7c0 2023-10-16T09:56:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7ea18ee700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c14ecf0 con 0x7f7e9c0d1210 2023-10-16T09:56:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c1f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c1f6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50158/0 (socket says 172.21.15.158:50158) 2023-10-16T09:56:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 -- 172.21.15.158:0/2683762549 learned_addr learned my addr 172.21.15.158:0/2683762549 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9b7fe700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e9c15d7c0 0x7f7e9c14e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9a7fc700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e9c0d1210 0x7f7e9c1f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 -- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e9c0d1210 msgr2=0x7f7e9c1f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e9c0d1210 0x7f7e9c1f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 -- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e9c15d7c0 msgr2=0x7f7e9c14e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e9c15d7c0 0x7f7e9c14e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.403+0000 7f7e9affd700 1 -- 172.21.15.158:0/2683762549 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e94009b30 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.404+0000 7f7e9affd700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c1f6d20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00e9b0 tx=0x7f7e8c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.404+0000 7f7e83fff700 1 -- 172.21.15.158:0/2683762549 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c009710 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.404+0000 7f7e83fff700 1 -- 172.21.15.158:0/2683762549 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00ee50 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.404+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c14efd0 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.404+0000 7f7e83fff700 1 -- 172.21.15.158:0/2683762549 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00ca50 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.404+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c1f7260 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.405+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c1f6380 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.405+0000 7f7e83fff700 1 -- 172.21.15.158:0/2683762549 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f7e8c00cbb0 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.405+0000 7f7e83fff700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 0x7f7e8806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.405+0000 7f7e9a7fc700 1 -- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 msgr2=0x7f7e8806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:47.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.406+0000 7f7e83fff700 1 -- 172.21.15.158:0/2683762549 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f7e8c089d00 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.406+0000 7f7e9a7fc700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 0x7f7e8806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:47.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.410+0000 7f7e83fff700 1 -- 172.21.15.158:0/2683762549 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e9c1f6380 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.470 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:47.469+0000 7f0849d4f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:56:47.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.607+0000 7f7e9a7fc700 1 -- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 msgr2=0x7f7e8806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:47.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.607+0000 7f7e9a7fc700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 0x7f7e8806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:47.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.625+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e9c1f6380 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.627+0000 7f7e83fff700 1 -- 172.21.15.158:0/2683762549 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17911 (secure 0 0 0) 0x7f7e9c1f6380 con 0x7f7e9c0ceb20 2023-10-16T09:56:47.628 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:47.629 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":54,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:56:44.527750+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:44:38.770851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T09:44:40.065625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.38","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2730663923":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6832/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6833/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/3457698273":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/686968952":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/251660717":"2023-10-17T09:44:59.554471+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:56:47.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 msgr2=0x7f7e8806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:47.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 0x7f7e8806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 msgr2=0x7f7e9c1f6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c1f6d20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00e9b0 tx=0x7f7e8c00ecc0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 shutdown_connections 2023-10-16T09:56:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e9c15d7c0 0x7f7e9c14e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f7e8806b0b0 0x7f7e8806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e9c0ceb20 0x7f7e9c1f6d20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 --2- 172.21.15.158:0/2683762549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e9c0d1210 0x7f7e9c1f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 >> 172.21.15.158:0/2683762549 conn(0x7f7e9c0669f0 msgr2=0x7f7e9c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:47.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 shutdown_connections 2023-10-16T09:56:47.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.629+0000 7f7ea18ee700 1 -- 172.21.15.158:0/2683762549 wait complete. 2023-10-16T09:56:47.642 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T09:56:47.643 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:56:47.696 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:56:47.696+0000 7fa54c167200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:56:47.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.804+0000 7f92920aa700 1 Processor -- start 2023-10-16T09:56:47.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.804+0000 7f92920aa700 1 -- start start 2023-10-16T09:56:47.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f92920aa700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f928c15a6c0 0x7f928c0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f92920aa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f928c0af590 0x7f928c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f92920aa700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928c072910 con 0x7f928c06dec0 2023-10-16T09:56:47.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f92920aa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928c06e370 con 0x7f928c0af590 2023-10-16T09:56:47.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f92920aa700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928c0ca7e0 con 0x7f928c15a6c0 2023-10-16T09:56:47.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f9282ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f928c0af590 0x7f928c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f928bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f928c06dec0 0x7f928c15e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50348/0 (socket says 172.21.15.158:50348) 2023-10-16T09:56:47.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f928bfff700 1 -- 172.21.15.158:0/1212302479 learned_addr learned my addr 172.21.15.158:0/1212302479 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:47.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f928b7fe700 1 --2- 172.21.15.158:0/1212302479 >> v2:172.21.15.172:3300/0 conn(0x7f928c15a6c0 0x7f928c0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f9282ffd700 1 -- 172.21.15.158:0/1212302479 >> v2:172.21.15.172:3300/0 conn(0x7f928c15a6c0 msgr2=0x7f928c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:47.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f9282ffd700 1 --2- 172.21.15.158:0/1212302479 >> v2:172.21.15.172:3300/0 conn(0x7f928c15a6c0 0x7f928c0723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f9282ffd700 1 -- 172.21.15.158:0/1212302479 >> v1:172.21.15.158:6789/0 conn(0x7f928c06dec0 legacy=0x7f928c15e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f9282ffd700 1 -- 172.21.15.158:0/1212302479 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c0b1e40 con 0x7f928c0af590 2023-10-16T09:56:47.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f928affd700 1 -- 172.21.15.158:0/1212302479 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419961211 0 0) 0x7f928c072910 con 0x7f928c06dec0 2023-10-16T09:56:47.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f9282ffd700 1 --2- 172.21.15.158:0/1212302479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c0af590 0x7f928c06d980 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f928c0737c0 tx=0x7f9274009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143cf1e80967fb79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:47.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.806+0000 7f928affd700 1 -- 172.21.15.158:0/1212302479 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274011830 con 0x7f928c0af590 2023-10-16T09:56:47.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f928affd700 1 -- 172.21.15.158:0/1212302479 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9274011990 con 0x7f928c0af590 2023-10-16T09:56:47.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f928affd700 1 -- 172.21.15.158:0/1212302479 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274011d20 con 0x7f928c0af590 2023-10-16T09:56:47.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 -- 172.21.15.158:0/1212302479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c0af590 msgr2=0x7f928c06d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 --2- 172.21.15.158:0/1212302479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c0af590 0x7f928c06d980 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f928c0737c0 tx=0x7f9274009b20 comp rx=0 tx=0).stop 2023-10-16T09:56:47.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 -- 172.21.15.158:0/1212302479 shutdown_connections 2023-10-16T09:56:47.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 --2- 172.21.15.158:0/1212302479 >> v2:172.21.15.172:3300/0 conn(0x7f928c15a6c0 0x7f928c0723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 --2- 172.21.15.158:0/1212302479 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c0af590 0x7f928c06d980 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 -- 172.21.15.158:0/1212302479 >> 172.21.15.158:0/1212302479 conn(0x7f928c0ccda0 msgr2=0x7f928c0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:47.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 -- 172.21.15.158:0/1212302479 shutdown_connections 2023-10-16T09:56:47.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.807+0000 7f92920aa700 1 -- 172.21.15.158:0/1212302479 wait complete. 2023-10-16T09:56:47.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.809+0000 7f92920aa700 1 Processor -- start 2023-10-16T09:56:47.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.809+0000 7f92920aa700 1 -- start start 2023-10-16T09:56:47.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f92920aa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f928c06dec0 0x7f928c0c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f92920aa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f928c0af590 0x7f928c0cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f92920aa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c15a6c0 0x7f928c153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f92920aa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0c5230 con 0x7f928c15a6c0 2023-10-16T09:56:47.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f92920aa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0c6030 con 0x7f928c06dec0 2023-10-16T09:56:47.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f92920aa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0cb990 con 0x7f928c0af590 2023-10-16T09:56:47.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c15a6c0 0x7f928c153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c15a6c0 0x7f928c153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50178/0 (socket says 172.21.15.158:50178) 2023-10-16T09:56:47.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 -- 172.21.15.158:0/1426268429 learned_addr learned my addr 172.21.15.158:0/1426268429 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:47.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f9282ffd700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f928c0af590 0x7f928c0cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928b7fe700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f928c06dec0 0x7f928c0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:47.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 -- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f928c0af590 msgr2=0x7f928c0cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f928c0af590 0x7f928c0cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 -- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f928c06dec0 msgr2=0x7f928c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:47.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f928c06dec0 0x7f928c0c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:47.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 -- 172.21.15.158:0/1426268429 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92740094a0 con 0x7f928c15a6c0 2023-10-16T09:56:47.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.810+0000 7f928bfff700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c15a6c0 0x7f928c153b80 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f928400cbd0 tx=0x7f928400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:47.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.811+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928400aa60 con 0x7f928c15a6c0 2023-10-16T09:56:47.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.811+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928400abe0 con 0x7f928c15a6c0 2023-10-16T09:56:47.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.811+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928c1540c0 con 0x7f928c15a6c0 2023-10-16T09:56:47.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.811+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840070d0 con 0x7f928c15a6c0 2023-10-16T09:56:47.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.811+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928c0c5420 con 0x7f928c15a6c0 2023-10-16T09:56:47.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.812+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928c0681b0 con 0x7f928c15a6c0 2023-10-16T09:56:47.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.813+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f9284007230 con 0x7f928c15a6c0 2023-10-16T09:56:47.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.813+0000 7f9288ff9700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 0x7f926c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:47.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.813+0000 7f928b7fe700 1 -- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 msgr2=0x7f926c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:47.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.813+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f9284089a40 con 0x7f928c15a6c0 2023-10-16T09:56:47.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.813+0000 7f928b7fe700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 0x7f926c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:56:47.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:47.817+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f928408be60 con 0x7f928c15a6c0 2023-10-16T09:56:48.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.014+0000 7f928b7fe700 1 -- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 msgr2=0x7f926c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3552602298 2023-10-16T09:56:48.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.014+0000 7f928b7fe700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 0x7f926c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:56:48.022 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.021+0000 7f0849d4f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:56:48.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.041+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 16) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9284054820 con 0x7f928c15a6c0 2023-10-16T09:56:48.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.042+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 msgr2=0x7f926c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:56:48.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.042+0000 7f9288ff9700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 0x7f926c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.074+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f928c0c6170 con 0x7f928c15a6c0 2023-10-16T09:56:48.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.075+0000 7f9288ff9700 1 -- 172.21.15.158:0/1426268429 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4928 (secure 0 0 0) 0x7f9284074090 con 0x7f928c15a6c0 2023-10-16T09:56:48.076 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:48.077 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4283661376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4283661376},{"type":"v1","addr":"172.21.15.158:6835","nonce":4283661376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/3069959147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":3069959147},{"type":"v1","addr":"172.21.15.158:6837","nonce":3069959147}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1583191131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1583191131},{"type":"v1","addr":"172.21.15.172:6835","nonce":1583191131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:44:41.090180+0000","modified":"2023-10-16T09:44:44.131783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34760":{"gid":34760,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.172:6834/1706659835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1706659835},{"type":"v1","addr":"172.21.15.172:6834","nonce":1706659835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T09:56:48.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c15a6c0 msgr2=0x7f928c153b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c15a6c0 0x7f928c153b80 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f928400cbd0 tx=0x7f928400cee0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 shutdown_connections 2023-10-16T09:56:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:6832/3552602298,v1:172.21.15.158:6833/3552602298] conn(0x7f926c06b0b0 0x7f926c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f928c15a6c0 0x7f928c153b80 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f928c06dec0 0x7f928c0c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 --2- 172.21.15.158:0/1426268429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f928c0af590 0x7f928c0cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 >> 172.21.15.158:0/1426268429 conn(0x7f928c0ccda0 msgr2=0x7f928c154340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 shutdown_connections 2023-10-16T09:56:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.077+0000 7f92920aa700 1 -- 172.21.15.158:0/1426268429 wait complete. 2023-10-16T09:56:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 17 2023-10-16T09:56:48.092 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:56:48.105 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c1d34700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.105 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c1d34700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.106 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c1d34700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.106 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c1d34700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.106 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c1d34700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.107 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c557b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.107 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c557b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.107 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.104+0000 7f07c557b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.107 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.105+0000 7f07c557b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.108 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:56:48.105+0000 7f07c557b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:56:48.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.213+0000 7f73c5c3b700 1 Processor -- start 2023-10-16T09:56:48.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.213+0000 7f73c5c3b700 1 -- start start 2023-10-16T09:56:48.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73c5c3b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f73c00cdee0 0x7f73c015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73c5c3b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f73c00ce2c0 0x7f73c00cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73c5c3b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c015c5d0 con 0x7f73c00b6e10 2023-10-16T09:56:48.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73c5c3b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c015d2c0 con 0x7f73c00cdee0 2023-10-16T09:56:48.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73c5c3b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c015c8b0 con 0x7f73c00ce2c0 2023-10-16T09:56:48.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73bf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f73c00cdee0 0x7f73c015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73bf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f73c00cdee0 0x7f73c015bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50252/0 (socket says 172.21.15.158:50252) 2023-10-16T09:56:48.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.214+0000 7f73bf7fe700 1 -- 172.21.15.158:0/3544292532 learned_addr learned my addr 172.21.15.158:0/3544292532 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:48.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73bffff700 1 --2- 172.21.15.158:0/3544292532 >> v2:172.21.15.172:3300/0 conn(0x7f73c00ce2c0 0x7f73c00cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73bf7fe700 1 -- 172.21.15.158:0/3544292532 >> v2:172.21.15.172:3300/0 conn(0x7f73c00ce2c0 msgr2=0x7f73c00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:48.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73bf7fe700 1 --2- 172.21.15.158:0/3544292532 >> v2:172.21.15.172:3300/0 conn(0x7f73c00ce2c0 0x7f73c00cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73bf7fe700 1 -- 172.21.15.158:0/3544292532 >> v1:172.21.15.158:6789/0 conn(0x7f73c00b6e10 legacy=0x7f73c01521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73bf7fe700 1 -- 172.21.15.158:0/3544292532 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c015cb90 con 0x7f73c00cdee0 2023-10-16T09:56:48.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73beffd700 1 -- 172.21.15.158:0/3544292532 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697760836 0 0) 0x7f73c015c5d0 con 0x7f73c00b6e10 2023-10-16T09:56:48.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73bf7fe700 1 --2- 172.21.15.158:0/3544292532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00cdee0 0x7f73c015bd10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f73c00b6630 tx=0x7f73b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7516664c4f0637d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:48.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73beffd700 1 -- 172.21.15.158:0/3544292532 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0011830 con 0x7f73c00cdee0 2023-10-16T09:56:48.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73beffd700 1 -- 172.21.15.158:0/3544292532 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0011990 con 0x7f73c00cdee0 2023-10-16T09:56:48.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.215+0000 7f73beffd700 1 -- 172.21.15.158:0/3544292532 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0011d20 con 0x7f73c00cdee0 2023-10-16T09:56:48.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 -- 172.21.15.158:0/3544292532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00cdee0 msgr2=0x7f73c015bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 --2- 172.21.15.158:0/3544292532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00cdee0 0x7f73c015bd10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f73c00b6630 tx=0x7f73b0009bd0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 -- 172.21.15.158:0/3544292532 shutdown_connections 2023-10-16T09:56:48.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 --2- 172.21.15.158:0/3544292532 >> v2:172.21.15.172:3300/0 conn(0x7f73c00ce2c0 0x7f73c00cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 --2- 172.21.15.158:0/3544292532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00cdee0 0x7f73c015bd10 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 -- 172.21.15.158:0/3544292532 >> 172.21.15.158:0/3544292532 conn(0x7f73c006a8d0 msgr2=0x7f73c006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:48.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 -- 172.21.15.158:0/3544292532 shutdown_connections 2023-10-16T09:56:48.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.216+0000 7f73c5c3b700 1 -- 172.21.15.158:0/3544292532 wait complete. 2023-10-16T09:56:48.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.217+0000 7f73c5c3b700 1 Processor -- start 2023-10-16T09:56:48.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.217+0000 7f73c5c3b700 1 -- start start 2023-10-16T09:56:48.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73c5c3b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00b6e10 0x7f73c00c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73c5c3b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73c00cdee0 0x7f73c00d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73c5c3b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73c00ce2c0 0x7f73c00d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73c5c3b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c014e310 con 0x7f73c00b6e10 2023-10-16T09:56:48.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73c5c3b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00ceee0 con 0x7f73c00cdee0 2023-10-16T09:56:48.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73c5c3b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00cf1c0 con 0x7f73c00ce2c0 2023-10-16T09:56:48.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00b6e10 0x7f73c00c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00b6e10 0x7f73c00c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50256/0 (socket says 172.21.15.158:50256) 2023-10-16T09:56:48.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 -- 172.21.15.158:0/2708471411 learned_addr learned my addr 172.21.15.158:0/2708471411 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:48.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73b7fff700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73c00cdee0 0x7f73c00d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bffff700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73c00ce2c0 0x7f73c00d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 -- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73c00ce2c0 msgr2=0x7f73c00d0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73c00ce2c0 0x7f73c00d0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 -- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73c00cdee0 msgr2=0x7f73c00d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73c00cdee0 0x7f73c00d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 -- 172.21.15.158:0/2708471411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b0009550 con 0x7f73c00b6e10 2023-10-16T09:56:48.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73b7fff700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73c00cdee0 0x7f73c00d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:48.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bf7fe700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00b6e10 0x7f73c00c67b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f73c00ce770 tx=0x7f73b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:48.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.218+0000 7f73bcff9700 1 -- 172.21.15.158:0/2708471411 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b00036a0 con 0x7f73c00b6e10 2023-10-16T09:56:48.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.219+0000 7f73bcff9700 1 -- 172.21.15.158:0/2708471411 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0003800 con 0x7f73c00b6e10 2023-10-16T09:56:48.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.219+0000 7f73bcff9700 1 -- 172.21.15.158:0/2708471411 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0003b70 con 0x7f73c00b6e10 2023-10-16T09:56:48.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.219+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c01546a0 con 0x7f73c00b6e10 2023-10-16T09:56:48.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.219+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c0154c60 con 0x7f73c00b6e10 2023-10-16T09:56:48.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.220+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c00681b0 con 0x7f73c00b6e10 2023-10-16T09:56:48.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.220+0000 7f73bcff9700 1 -- 172.21.15.158:0/2708471411 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f73b0003cd0 con 0x7f73c00b6e10 2023-10-16T09:56:48.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.220+0000 7f73bcff9700 1 -- 172.21.15.158:0/2708471411 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7099+0+0 (secure 0 0 0) 0x7f73b0021030 con 0x7f73c00b6e10 2023-10-16T09:56:48.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.225+0000 7f73bcff9700 1 -- 172.21.15.158:0/2708471411 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73b0055cc0 con 0x7f73c00b6e10 2023-10-16T09:56:48.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.463+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73c00b3bf0 con 0x7f73c00b6e10 2023-10-16T09:56:48.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.464+0000 7f73bcff9700 1 -- 172.21.15.158:0/2708471411 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4928 (secure 0 0 0) 0x7f73b002b020 con 0x7f73c00b6e10 2023-10-16T09:56:48.465 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:48.465 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4283661376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4283661376},{"type":"v1","addr":"172.21.15.158:6835","nonce":4283661376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/3069959147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":3069959147},{"type":"v1","addr":"172.21.15.158:6837","nonce":3069959147}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1583191131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1583191131},{"type":"v1","addr":"172.21.15.172:6835","nonce":1583191131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:44:41.090180+0000","modified":"2023-10-16T09:44:44.131783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34760":{"gid":34760,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.172:6834/1706659835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1706659835},{"type":"v1","addr":"172.21.15.172:6834","nonce":1706659835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T09:56:48.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00b6e10 msgr2=0x7f73c00c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00b6e10 0x7f73c00c67b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f73c00ce770 tx=0x7f73b0004750 comp rx=0 tx=0).stop 2023-10-16T09:56:48.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 shutdown_connections 2023-10-16T09:56:48.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73c00ce2c0 0x7f73c00d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73c00b6e10 0x7f73c00c67b0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 --2- 172.21.15.158:0/2708471411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f73c00cdee0 0x7f73c00d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 >> 172.21.15.158:0/2708471411 conn(0x7f73c006a8d0 msgr2=0x7f73c015d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:48.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 shutdown_connections 2023-10-16T09:56:48.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.466+0000 7f73c5c3b700 1 -- 172.21.15.158:0/2708471411 wait complete. 2023-10-16T09:56:48.470 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 17 2023-10-16T09:56:48.479 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:56:48.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.640+0000 7f98cec56700 1 Processor -- start 2023-10-16T09:56:48.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.640+0000 7f98cec56700 1 -- start start 2023-10-16T09:56:48.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98cec56700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 0x7f98c806b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98cec56700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c80ce4d0 con 0x7f98c80d0bd0 2023-10-16T09:56:48.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98cec56700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c806c1b0 con 0x7f98c80d3570 2023-10-16T09:56:48.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98cec56700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80d1c20 con 0x7f98c815daa0 2023-10-16T09:56:48.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98c7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f98c80d0bd0 0x7f98c80d1510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50440/0 (socket says 172.21.15.158:50440) 2023-10-16T09:56:48.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98c7fff700 1 -- 172.21.15.158:0/1378579876 learned_addr learned my addr 172.21.15.158:0/1378579876 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:48.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881885166 0 0) 0x7f98c80ce4d0 con 0x7f98c80d0bd0 2023-10-16T09:56:48.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98cc9f2700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 0x7f98c806b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.641+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b80035b0 con 0x7f98c80d0bd0 2023-10-16T09:56:48.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4294558025 0 0) 0x7f98b80035b0 con 0x7f98c80d0bd0 2023-10-16T09:56:48.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 msgr2=0x7f98c806b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:48.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 0x7f98c806b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 >> v1:172.21.15.172:6789/0 conn(0x7f98c80d3570 legacy=0x7f98c80d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c806c490 con 0x7f98c80d0bd0 2023-10-16T09:56:48.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f98b4001670 con 0x7f98c80d0bd0 2023-10-16T09:56:48.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 >> v1:172.21.15.158:6789/0 conn(0x7f98c80d0bd0 legacy=0x7f98c80d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b8003a30 0x7f98b8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 0x7f98b8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b8004d90 0x7f98b8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80d1c20 con 0x7f98b8004d90 2023-10-16T09:56:48.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b800bac0 con 0x7f98b8003a30 2023-10-16T09:56:48.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b800bc80 con 0x7f98c815daa0 2023-10-16T09:56:48.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cd1f3700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b8004d90 0x7f98b8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cc9f2700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b8003a30 0x7f98b8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c7fff700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 0x7f98b8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cd1f3700 1 -- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 msgr2=0x7f98b8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:48.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cd1f3700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c815daa0 0x7f98b8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cd1f3700 1 -- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b8003a30 msgr2=0x7f98b8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:48.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cd1f3700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b8003a30 0x7f98b8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cd1f3700 1 -- 172.21.15.158:0/1378579876 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b80035b0 con 0x7f98b8004d90 2023-10-16T09:56:48.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98cd1f3700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b8004d90 0x7f98b8005210 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f98c000c210 tx=0x7f98b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fa3aa8ee5ac255 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:48.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0001df0 con 0x7f98b8004d90 2023-10-16T09:56:48.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000cd70 con 0x7f98b8004d90 2023-10-16T09:56:48.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.642+0000 7f98c77fe700 1 -- 172.21.15.158:0/1378579876 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0003c40 con 0x7f98b8004d90 2023-10-16T09:56:48.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.643+0000 7f98cec56700 1 -- 172.21.15.158:0/1378579876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b8004d90 msgr2=0x7f98b8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.643+0000 7f98cec56700 1 --2- 172.21.15.158:0/1378579876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b8004d90 0x7f98b8005210 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f98c000c210 tx=0x7f98b8005880 comp rx=0 tx=0).stop 2023-10-16T09:56:48.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.643+0000 7f98cd1f3700 1 -- 172.21.15.158:0/1378579876 reap_dead start 2023-10-16T09:56:48.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.643+0000 7f98cec56700 1 -- 172.21.15.158:0/1378579876 shutdown_connections 2023-10-16T09:56:48.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.643+0000 7f98cec56700 1 -- 172.21.15.158:0/1378579876 >> 172.21.15.158:0/1378579876 conn(0x7f98c808d040 msgr2=0x7f98c809c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:48.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.643+0000 7f98cec56700 1 -- 172.21.15.158:0/1378579876 shutdown_connections 2023-10-16T09:56:48.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.643+0000 7f98cec56700 1 -- 172.21.15.158:0/1378579876 wait complete. 2023-10-16T09:56:48.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.644+0000 7f98cec56700 1 Processor -- start 2023-10-16T09:56:48.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.644+0000 7f98cec56700 1 -- start start 2023-10-16T09:56:48.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cec56700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c814bab0 0x7f98c80cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cec56700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80cbf30 0x7f98c80b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cec56700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c81f3350 0x7f98c80b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cec56700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f3850 con 0x7f98c80cbf30 2023-10-16T09:56:48.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cec56700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f3b30 con 0x7f98c81f3350 2023-10-16T09:56:48.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cec56700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f3e10 con 0x7f98c814bab0 2023-10-16T09:56:48.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80cbf30 0x7f98c80b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80cbf30 0x7f98c80b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50274/0 (socket says 172.21.15.158:50274) 2023-10-16T09:56:48.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 -- 172.21.15.158:0/1810193482 learned_addr learned my addr 172.21.15.158:0/1810193482 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:48.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cc9f2700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c814bab0 0x7f98c80cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98cd1f3700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c81f3350 0x7f98c80b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 -- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c814bab0 msgr2=0x7f98c80cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c814bab0 0x7f98c80cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 -- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c81f3350 msgr2=0x7f98c80b7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c81f3350 0x7f98c80b7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.645+0000 7f98c7fff700 1 -- 172.21.15.158:0/1810193482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c81f40f0 con 0x7f98c80cbf30 2023-10-16T09:56:48.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.646+0000 7f98c7fff700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80cbf30 0x7f98c80b7830 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f98b4007290 tx=0x7f98b40075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.646+0000 7f98c57fa700 1 -- 172.21.15.158:0/1810193482 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b400f5c0 con 0x7f98c80cbf30 2023-10-16T09:56:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.646+0000 7f98c57fa700 1 -- 172.21.15.158:0/1810193482 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b400f720 con 0x7f98c80cbf30 2023-10-16T09:56:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.646+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c814be90 con 0x7f98c80cbf30 2023-10-16T09:56:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.646+0000 7f98c57fa700 1 -- 172.21.15.158:0/1810193482 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b400fad0 con 0x7f98c80cbf30 2023-10-16T09:56:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.646+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c815e1e0 con 0x7f98c80cbf30 2023-10-16T09:56:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.647+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c814c190 con 0x7f98c80cbf30 2023-10-16T09:56:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.647+0000 7f98c57fa700 1 -- 172.21.15.158:0/1810193482 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f98b400fc30 con 0x7f98c80cbf30 2023-10-16T09:56:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.648+0000 7f98c57fa700 1 -- 172.21.15.158:0/1810193482 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7099+0+0 (secure 0 0 0) 0x7f98b40895c0 con 0x7f98c80cbf30 2023-10-16T09:56:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.652+0000 7f98c57fa700 1 -- 172.21.15.158:0/1810193482 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98b4054550 con 0x7f98c80cbf30 2023-10-16T09:56:48.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.847+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98c80681b0 con 0x7f98c80cbf30 2023-10-16T09:56:48.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.849+0000 7f98c57fa700 1 -- 172.21.15.158:0/1810193482 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18410 (secure 0 0 0) 0x7f98b4057b70 con 0x7f98c80cbf30 2023-10-16T09:56:48.850 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:48.850 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":55,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:56:48.035759+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:44:38.770851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T09:44:40.065625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.38","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/277874160":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/102656464":"2023-10-17T09:56:48.035698+0000","172.21.15.158:6833/3552602298":"2023-10-17T09:56:48.035698+0000","172.21.15.158:6832/3552602298":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/73320321":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/1636480876":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/3774227013":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/251660717":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/3457698273":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/1760463794":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/686968952":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6833/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6832/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/4158145300":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/2730663923":"2023-10-17T09:44:59.554471+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:56:48.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80cbf30 msgr2=0x7f98c80b7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80cbf30 0x7f98c80b7830 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f98b4007290 tx=0x7f98b40075a0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 shutdown_connections 2023-10-16T09:56:48.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c81f3350 0x7f98c80b7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c814bab0 0x7f98c80cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 --2- 172.21.15.158:0/1810193482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80cbf30 0x7f98c80b7830 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 >> 172.21.15.158:0/1810193482 conn(0x7f98c808d040 msgr2=0x7f98c80c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:48.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 shutdown_connections 2023-10-16T09:56:48.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.851+0000 7f98cec56700 1 -- 172.21.15.158:0/1810193482 wait complete. 2023-10-16T09:56:48.864 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T09:56:48.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.985+0000 7fbe3e6b0700 1 Processor -- start 2023-10-16T09:56:48.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.985+0000 7fbe3e6b0700 1 -- start start 2023-10-16T09:56:48.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe3e6b0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbe380b3040 0x7fbe380b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe3e6b0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbe3806bcc0 0x7fbe3806c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe3e6b0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3815e150 con 0x7fbe3815d920 2023-10-16T09:56:48.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe3e6b0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe380b3a00 con 0x7fbe380b3040 2023-10-16T09:56:48.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe3e6b0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe3815e430 con 0x7fbe3806bcc0 2023-10-16T09:56:48.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbe380b3040 0x7fbe380b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbe380b3040 0x7fbe380b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50280/0 (socket says 172.21.15.158:50280) 2023-10-16T09:56:48.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 -- 172.21.15.158:0/371335060 learned_addr learned my addr 172.21.15.158:0/371335060 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:48.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe3cc4d700 1 --2- 172.21.15.158:0/371335060 >> v2:172.21.15.172:3300/0 conn(0x7fbe3806bcc0 0x7fbe3806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 -- 172.21.15.158:0/371335060 >> v2:172.21.15.172:3300/0 conn(0x7fbe3806bcc0 msgr2=0x7fbe3806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:48.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 --2- 172.21.15.158:0/371335060 >> v2:172.21.15.172:3300/0 conn(0x7fbe3806bcc0 0x7fbe3806c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 -- 172.21.15.158:0/371335060 >> v1:172.21.15.172:6789/0 conn(0x7fbe3815d920 legacy=0x7fbe380cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 -- 172.21.15.158:0/371335060 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3815e710 con 0x7fbe380b3040 2023-10-16T09:56:48.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe3cc4d700 1 --2- 172.21.15.158:0/371335060 >> v2:172.21.15.172:3300/0 conn(0x7fbe3806bcc0 0x7fbe3806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:48.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.986+0000 7fbe377fe700 1 --2- 172.21.15.158:0/371335060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b34c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbe380cbb10 tx=0x7fbe2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4c018552c188d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:48.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe36ffd700 1 -- 172.21.15.158:0/371335060 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c00b870 con 0x7fbe380b3040 2023-10-16T09:56:48.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe36ffd700 1 -- 172.21.15.158:0/371335060 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c00b9d0 con 0x7fbe380b3040 2023-10-16T09:56:48.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe36ffd700 1 -- 172.21.15.158:0/371335060 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c00bd60 con 0x7fbe380b3040 2023-10-16T09:56:48.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/371335060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 msgr2=0x7fbe380b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:48.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/371335060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b34c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbe380cbb10 tx=0x7fbe2c009bd0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/371335060 shutdown_connections 2023-10-16T09:56:48.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/371335060 >> v2:172.21.15.172:3300/0 conn(0x7fbe3806bcc0 0x7fbe3806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/371335060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b34c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:48.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/371335060 >> 172.21.15.158:0/371335060 conn(0x7fbe380ce040 msgr2=0x7fbe380ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/371335060 shutdown_connections 2023-10-16T09:56:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.987+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/371335060 wait complete. 2023-10-16T09:56:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.988+0000 7fbe3e6b0700 1 Processor -- start 2023-10-16T09:56:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe3e6b0700 1 -- start start 2023-10-16T09:56:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe3e6b0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe3806bcc0 0x7fbe381557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe3e6b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe3e6b0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe3815d920 0x7fbe380c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe3e6b0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380b61a0 con 0x7fbe380b3040 2023-10-16T09:56:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe3e6b0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380b5520 con 0x7fbe3815d920 2023-10-16T09:56:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe3e6b0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380b5800 con 0x7fbe3806bcc0 2023-10-16T09:56:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe377fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe377fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50286/0 (socket says 172.21.15.158:50286) 2023-10-16T09:56:49.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.989+0000 7fbe377fe700 1 -- 172.21.15.158:0/591797033 learned_addr learned my addr 172.21.15.158:0/591797033 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:49.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe3cc4d700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe3815d920 0x7fbe380c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe37fff700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe3806bcc0 0x7fbe381557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe377fe700 1 -- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe3806bcc0 msgr2=0x7fbe381557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe377fe700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe3806bcc0 0x7fbe381557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe377fe700 1 -- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe3815d920 msgr2=0x7fbe380c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe377fe700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe3815d920 0x7fbe380c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe377fe700 1 -- 172.21.15.158:0/591797033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c009550 con 0x7fbe380b3040 2023-10-16T09:56:49.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe377fe700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b65c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbe3806b830 tx=0x7fbe2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:49.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c004310 con 0x7fbe380b3040 2023-10-16T09:56:49.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c01f070 con 0x7fbe380b3040 2023-10-16T09:56:49.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe380b5ae0 con 0x7fbe380b3040 2023-10-16T09:56:49.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c0036a0 con 0x7fbe380b3040 2023-10-16T09:56:49.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.990+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe38155d30 con 0x7fbe380b3040 2023-10-16T09:56:49.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.991+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe380b62e0 con 0x7fbe380b3040 2023-10-16T09:56:49.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.991+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbe2c021030 con 0x7fbe380b3040 2023-10-16T09:56:49.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.992+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7099+0+0 (secure 0 0 0) 0x7fbe2c08a940 con 0x7fbe380b3040 2023-10-16T09:56:49.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:48.996+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe380b62e0 con 0x7fbe380b3040 2023-10-16T09:56:49.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.045+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 17) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fbe2c055380 con 0x7fbe380b3040 2023-10-16T09:56:49.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.045+0000 7fbe34ff9700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fbe24071100 0x7fbe240734f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:49.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.046+0000 7fbe37fff700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fbe24071100 0x7fbe240734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.049+0000 7fbe37fff700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fbe24071100 0x7fbe240734f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000f60 tx=0x7fbe28009040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:49.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.243+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbe380b62e0 con 0x7fbe380b3040 2023-10-16T09:56:49.244 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T09:56:49.243+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T09:56:49.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.244+0000 7fbe34ff9700 1 -- 172.21.15.158:0/591797033 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbe380b62e0 con 0x7fbe380b3040 2023-10-16T09:56:49.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fbe24071100 msgr2=0x7fbe240734f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fbe24071100 0x7fbe240734f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000f60 tx=0x7fbe28009040 comp rx=0 tx=0).stop 2023-10-16T09:56:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 msgr2=0x7fbe380b65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b65c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbe3806b830 tx=0x7fbe2c004750 comp rx=0 tx=0).stop 2023-10-16T09:56:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 shutdown_connections 2023-10-16T09:56:49.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fbe24071100 0x7fbe240734f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe3815d920 0x7fbe380c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe3806bcc0 0x7fbe381557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 --2- 172.21.15.158:0/591797033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe380b3040 0x7fbe380b65c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 >> 172.21.15.158:0/591797033 conn(0x7fbe380ce040 msgr2=0x7fbe380cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:49.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 shutdown_connections 2023-10-16T09:56:49.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.247+0000 7fbe3e6b0700 1 -- 172.21.15.158:0/591797033 wait complete. 2023-10-16T09:56:49.251 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T09:56:49.261 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T09:56:49.285 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:56:49.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.383+0000 7f02d871c700 1 Processor -- start 2023-10-16T09:56:49.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.383+0000 7f02d871c700 1 -- start start 2023-10-16T09:56:49.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d871c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02d015da40 0x7f02d00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:49.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d871c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00cbd30 con 0x7f02d00cc590 2023-10-16T09:56:49.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d871c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00cc1c0 con 0x7f02d006afe0 2023-10-16T09:56:49.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d871c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d015de20 con 0x7f02d015da40 2023-10-16T09:56:49.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d64b8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02d015da40 0x7f02d00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d5cb7700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f02d00cc590 0x7f02d006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50490/0 (socket says 172.21.15.158:50490) 2023-10-16T09:56:49.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d64b8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02d015da40 0x7f02d00caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50314/0 (socket says 172.21.15.158:50314) 2023-10-16T09:56:49.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d5cb7700 1 -- 172.21.15.158:0/4020405121 learned_addr learned my addr 172.21.15.158:0/4020405121 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:49.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d64b8700 1 -- 172.21.15.158:0/4020405121 >> v1:172.21.15.172:6789/0 conn(0x7f02d006afe0 legacy=0x7f02d00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d64b8700 1 -- 172.21.15.158:0/4020405121 >> v1:172.21.15.158:6789/0 conn(0x7f02d00cc590 legacy=0x7f02d006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d64b8700 1 -- 172.21.15.158:0/4020405121 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00ce260 con 0x7f02d015da40 2023-10-16T09:56:49.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d54b6700 1 -- 172.21.15.158:0/4020405121 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263306121 0 0) 0x7f02d00cbd30 con 0x7f02d00cc590 2023-10-16T09:56:49.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d64b8700 1 --2- 172.21.15.158:0/4020405121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d015da40 0x7f02d00caa20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009840 tx=0x7f02cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa566b0495f766a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:49.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d54b6700 1 -- 172.21.15.158:0/4020405121 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc011830 con 0x7f02d015da40 2023-10-16T09:56:49.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d54b6700 1 -- 172.21.15.158:0/4020405121 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc011990 con 0x7f02d015da40 2023-10-16T09:56:49.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.384+0000 7f02d54b6700 1 -- 172.21.15.158:0/4020405121 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc011d20 con 0x7f02d015da40 2023-10-16T09:56:49.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.385+0000 7f02d871c700 1 -- 172.21.15.158:0/4020405121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d015da40 msgr2=0x7f02d00caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.385+0000 7f02d871c700 1 --2- 172.21.15.158:0/4020405121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d015da40 0x7f02d00caa20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009840 tx=0x7f02cc009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:49.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.385+0000 7f02d871c700 1 -- 172.21.15.158:0/4020405121 shutdown_connections 2023-10-16T09:56:49.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.385+0000 7f02d871c700 1 --2- 172.21.15.158:0/4020405121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d015da40 0x7f02d00caa20 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.385+0000 7f02d871c700 1 -- 172.21.15.158:0/4020405121 >> 172.21.15.158:0/4020405121 conn(0x7f02d0099a10 msgr2=0x7f02d00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:49.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.385+0000 7f02d871c700 1 -- 172.21.15.158:0/4020405121 shutdown_connections 2023-10-16T09:56:49.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.385+0000 7f02d871c700 1 -- 172.21.15.158:0/4020405121 wait complete. 2023-10-16T09:56:49.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.386+0000 7f02d871c700 1 Processor -- start 2023-10-16T09:56:49.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.386+0000 7f02d871c700 1 -- start start 2023-10-16T09:56:49.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d871c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02d006afe0 0x7f02d015a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:49.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d871c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d00cc590 0x7f02d01f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:49.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d871c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02d015da40 0x7f02d00b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:49.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d871c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00c4e00 con 0x7f02d00cc590 2023-10-16T09:56:49.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d871c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d01f7170 con 0x7f02d015da40 2023-10-16T09:56:49.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d871c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00c41f0 con 0x7f02d006afe0 2023-10-16T09:56:49.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d00cc590 0x7f02d01f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d00cc590 0x7f02d01f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50318/0 (socket says 172.21.15.158:50318) 2023-10-16T09:56:49.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 -- 172.21.15.158:0/3212867477 learned_addr learned my addr 172.21.15.158:0/3212867477 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:49.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d64b8700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02d006afe0 0x7f02d015a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d6cb9700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02d015da40 0x7f02d00b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 -- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02d006afe0 msgr2=0x7f02d015a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02d006afe0 0x7f02d015a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 -- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02d015da40 msgr2=0x7f02d00b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:49.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02d015da40 0x7f02d00b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:49.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 -- 172.21.15.158:0/3212867477 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0094a0 con 0x7f02d00cc590 2023-10-16T09:56:49.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.387+0000 7f02d5cb7700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d00cc590 0x7f02d01f6c30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f02c0001eb0 tx=0x7f02c000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:49.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.388+0000 7f02c6ffd700 1 -- 172.21.15.158:0/3212867477 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0004c80 con 0x7f02d00cc590 2023-10-16T09:56:49.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.388+0000 7f02c6ffd700 1 -- 172.21.15.158:0/3212867477 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0004de0 con 0x7f02d00cc590 2023-10-16T09:56:49.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.388+0000 7f02c6ffd700 1 -- 172.21.15.158:0/3212867477 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0007310 con 0x7f02d00cc590 2023-10-16T09:56:49.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.388+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d00c44d0 con 0x7f02d00cc590 2023-10-16T09:56:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.388+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d00c4a90 con 0x7f02d00cc590 2023-10-16T09:56:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.389+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d00c5190 con 0x7f02d00cc590 2023-10-16T09:56:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.389+0000 7f02c6ffd700 1 -- 172.21.15.158:0/3212867477 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 17) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f02c0007470 con 0x7f02d00cc590 2023-10-16T09:56:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.389+0000 7f02c6ffd700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f02b406ad50 0x7f02b406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:49.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.390+0000 7f02c6ffd700 1 -- 172.21.15.158:0/3212867477 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7185+0+0 (secure 0 0 0) 0x7f02c00893d0 con 0x7f02d00cc590 2023-10-16T09:56:49.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.390+0000 7f02d64b8700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f02b406ad50 0x7f02b406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:49.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.390+0000 7f02d64b8700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f02b406ad50 0x7f02b406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0090b0 tx=0x7f02cc01c040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:49.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.394+0000 7f02c6ffd700 1 -- 172.21.15.158:0/3212867477 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02c00541f0 con 0x7f02d00cc590 2023-10-16T09:56:49.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:49.633+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f02d00681b0 con 0x7f02d00cc590 2023-10-16T09:56:50.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.246+0000 7f02c6ffd700 1 -- 172.21.15.158:0/3212867477 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f02c0057810 con 0x7f02d00cc590 2023-10-16T09:56:50.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f02b406ad50 msgr2=0x7f02b406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:50.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f02b406ad50 0x7f02b406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0090b0 tx=0x7f02cc01c040 comp rx=0 tx=0).stop 2023-10-16T09:56:50.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d00cc590 msgr2=0x7f02d01f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:50.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d00cc590 0x7f02d01f6c30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f02c0001eb0 tx=0x7f02c000c800 comp rx=0 tx=0).stop 2023-10-16T09:56:50.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 shutdown_connections 2023-10-16T09:56:50.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f02b406ad50 0x7f02b406d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02d015da40 0x7f02d00b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02d006afe0 0x7f02d015a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 --2- 172.21.15.158:0/3212867477 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d00cc590 0x7f02d01f6c30 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 >> 172.21.15.158:0/3212867477 conn(0x7f02d0099a10 msgr2=0x7f02d00b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:50.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 shutdown_connections 2023-10-16T09:56:50.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.248+0000 7f02d871c700 1 -- 172.21.15.158:0/3212867477 wait complete. 2023-10-16T09:56:50.263 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T09:56:50.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.387+0000 7f708fce4700 1 Processor -- start 2023-10-16T09:56:50.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.387+0000 7f708fce4700 1 -- start start 2023-10-16T09:56:50.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708fce4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70880cde10 0x7f708806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:50.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708fce4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7088144840 0x7f708806da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:50.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708fce4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708806c210 con 0x7f7088141920 2023-10-16T09:56:50.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708fce4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7088141d00 con 0x7f7088144840 2023-10-16T09:56:50.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708fce4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708806b970 con 0x7f70880cde10 2023-10-16T09:56:50.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7088144840 0x7f708806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:50.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7088144840 0x7f708806da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50346/0 (socket says 172.21.15.158:50346) 2023-10-16T09:56:50.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 -- 172.21.15.158:0/2665123353 learned_addr learned my addr 172.21.15.158:0/2665123353 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:50.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708d27f700 1 --2- 172.21.15.158:0/2665123353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70880cde10 0x7f708806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:50.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 -- 172.21.15.158:0/2665123353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70880cde10 msgr2=0x7f708806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:50.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 --2- 172.21.15.158:0/2665123353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70880cde10 0x7f708806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 -- 172.21.15.158:0/2665123353 >> v1:172.21.15.158:6789/0 conn(0x7f7088141920 legacy=0x7f70880cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:50.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 -- 172.21.15.158:0/2665123353 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708806bc50 con 0x7f7088144840 2023-10-16T09:56:50.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708ca7e700 1 -- 172.21.15.158:0/2665123353 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439905315 0 0) 0x7f708806c210 con 0x7f7088141920 2023-10-16T09:56:50.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.388+0000 7f708e281700 1 --2- 172.21.15.158:0/2665123353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7088144840 0x7f708806da60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7080009e50 tx=0x7f708000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f25c96db701a087 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:50.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708ca7e700 1 -- 172.21.15.158:0/2665123353 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708000e070 con 0x7f7088144840 2023-10-16T09:56:50.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708ca7e700 1 -- 172.21.15.158:0/2665123353 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708000b9c0 con 0x7f7088144840 2023-10-16T09:56:50.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708ca7e700 1 -- 172.21.15.158:0/2665123353 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708000bd70 con 0x7f7088144840 2023-10-16T09:56:50.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 -- 172.21.15.158:0/2665123353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7088144840 msgr2=0x7f708806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:50.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 --2- 172.21.15.158:0/2665123353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7088144840 0x7f708806da60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7080009e50 tx=0x7f708000b330 comp rx=0 tx=0).stop 2023-10-16T09:56:50.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 -- 172.21.15.158:0/2665123353 shutdown_connections 2023-10-16T09:56:50.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 --2- 172.21.15.158:0/2665123353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7088144840 0x7f708806da60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 --2- 172.21.15.158:0/2665123353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70880cde10 0x7f708806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 -- 172.21.15.158:0/2665123353 >> 172.21.15.158:0/2665123353 conn(0x7f70880cc5b0 msgr2=0x7f70880cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:50.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 -- 172.21.15.158:0/2665123353 shutdown_connections 2023-10-16T09:56:50.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.389+0000 7f708fce4700 1 -- 172.21.15.158:0/2665123353 wait complete. 2023-10-16T09:56:50.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.390+0000 7f708fce4700 1 Processor -- start 2023-10-16T09:56:50.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.390+0000 7f708fce4700 1 -- start start 2023-10-16T09:56:50.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708fce4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70880cde10 0x7f708813b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:50.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708fce4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7088141920 0x7f7088144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:50.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708fce4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7088144840 0x7f70880b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:50.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708fce4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7088142000 con 0x7f70880cde10 2023-10-16T09:56:50.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708fce4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708813d230 con 0x7f7088144840 2023-10-16T09:56:50.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708fce4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708813c990 con 0x7f7088141920 2023-10-16T09:56:50.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708da80700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70880cde10 0x7f708813b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:50.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708da80700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70880cde10 0x7f708813b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50348/0 (socket says 172.21.15.158:50348) 2023-10-16T09:56:50.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.391+0000 7f708da80700 1 -- 172.21.15.158:0/160917404 learned_addr learned my addr 172.21.15.158:0/160917404 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:50.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708da80700 1 -- 172.21.15.158:0/160917404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7088141920 msgr2=0x7f7088144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:50.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708da80700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7088141920 0x7f7088144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708da80700 1 -- 172.21.15.158:0/160917404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7088144840 msgr2=0x7f70880b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:50.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708da80700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7088144840 0x7f70880b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:50.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708da80700 1 -- 172.21.15.158:0/160917404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080009b30 con 0x7f70880cde10 2023-10-16T09:56:50.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708da80700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70880cde10 0x7f708813b930 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f70840011e0 tx=0x7f708400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:50.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70840054e0 con 0x7f70880cde10 2023-10-16T09:56:50.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7084003680 con 0x7f70880cde10 2023-10-16T09:56:50.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708813cc70 con 0x7f70880cde10 2023-10-16T09:56:50.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084003a30 con 0x7f70880cde10 2023-10-16T09:56:50.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.392+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7088144540 con 0x7f70880cde10 2023-10-16T09:56:50.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.393+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70880681b0 con 0x7f70880cde10 2023-10-16T09:56:50.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.393+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 17) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7084003b90 con 0x7f70880cde10 2023-10-16T09:56:50.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.393+0000 7f707e7fc700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f706c06ad50 0x7f706c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:50.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.394+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7185+0+0 (secure 0 0 0) 0x7f7084088ce0 con 0x7f70880cde10 2023-10-16T09:56:50.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.394+0000 7f708d27f700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f706c06ad50 0x7f706c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:50.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.394+0000 7f708d27f700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f706c06ad50 0x7f706c06d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f708813dc20 tx=0x7f7078008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:50.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.398+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f708408d070 con 0x7f70880cde10 2023-10-16T09:56:50.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:50.592+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7088072e30 con 0x7f70880cde10 2023-10-16T09:56:51.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.056+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 18) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f7084053ed0 con 0x7f70880cde10 2023-10-16T09:56:51.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.558+0000 7f707e7fc700 1 -- 172.21.15.158:0/160917404 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7084072580 con 0x7f70880cde10 2023-10-16T09:56:51.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f706c06ad50 msgr2=0x7f706c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:51.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f706c06ad50 0x7f706c06d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f708813dc20 tx=0x7f7078008040 comp rx=0 tx=0).stop 2023-10-16T09:56:51.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70880cde10 msgr2=0x7f708813b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:51.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70880cde10 0x7f708813b930 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f70840011e0 tx=0x7f708400cae0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 shutdown_connections 2023-10-16T09:56:51.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f706c06ad50 0x7f706c06d210 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f708813dc20 tx=0x7f7078008040 comp rx=0 tx=0).stop 2023-10-16T09:56:51.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7088144840 0x7f70880b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70880cde10 0x7f708813b930 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 --2- 172.21.15.158:0/160917404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7088141920 0x7f7088144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.563+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 >> 172.21.15.158:0/160917404 conn(0x7f70880cc5b0 msgr2=0x7f70880719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:51.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.564+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 shutdown_connections 2023-10-16T09:56:51.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.564+0000 7f708fce4700 1 -- 172.21.15.158:0/160917404 wait complete. 2023-10-16T09:56:51.568 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T09:56:51.581 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T09:56:51.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.712+0000 7f6ac3d9e700 1 Processor -- start 2023-10-16T09:56:51.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.712+0000 7f6ac3d9e700 1 -- start start 2023-10-16T09:56:51.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac3d9e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6abc15dad0 0x7f6abc0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:51.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac3d9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:51.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac3d9e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0d0ae0 con 0x7f6abc0cea20 2023-10-16T09:56:51.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac3d9e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc0d12d0 con 0x7f6abc15dad0 2023-10-16T09:56:51.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac3d9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0cbdf0 con 0x7f6abc15deb0 2023-10-16T09:56:51.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac233b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6abc0cea20 0x7f6abc0c3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:54440/0 (socket says 172.21.15.158:54440) 2023-10-16T09:56:51.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac1339700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:51.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac1b3a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6abc15dad0 0x7f6abc0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:51.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.713+0000 7f6ac233b700 1 -- 172.21.15.158:0/3916399591 learned_addr learned my addr 172.21.15.158:0/3916399591 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:51.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac0b38700 1 -- 172.21.15.158:0/3916399591 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700303789 0 0) 0x7f6abc0d0ae0 con 0x7f6abc0cea20 2023-10-16T09:56:51.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac0b38700 1 -- 172.21.15.158:0/3916399591 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa80035b0 con 0x7f6abc0cea20 2023-10-16T09:56:51.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac1b3a700 1 -- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 msgr2=0x7f6abc0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:51.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac1b3a700 1 --2- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac1b3a700 1 -- 172.21.15.158:0/3916399591 >> v1:172.21.15.172:6789/0 conn(0x7f6abc0cea20 legacy=0x7f6abc0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:51.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac1b3a700 1 -- 172.21.15.158:0/3916399591 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0cc280 con 0x7f6abc15dad0 2023-10-16T09:56:51.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac1339700 1 --2- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:51.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac1b3a700 1 --2- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0cdfa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009840 tx=0x7f6ab8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=790bcdc487b6ccb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:51.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac0b38700 1 -- 172.21.15.158:0/3916399591 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8011830 con 0x7f6abc15dad0 2023-10-16T09:56:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac0b38700 1 -- 172.21.15.158:0/3916399591 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8011990 con 0x7f6abc15dad0 2023-10-16T09:56:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.714+0000 7f6ac0b38700 1 -- 172.21.15.158:0/3916399591 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8011d00 con 0x7f6abc15dad0 2023-10-16T09:56:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 msgr2=0x7f6abc0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0cdfa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009840 tx=0x7f6ab8009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3916399591 shutdown_connections 2023-10-16T09:56:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0cdfa0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3916399591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3916399591 >> 172.21.15.158:0/3916399591 conn(0x7f6abc099a10 msgr2=0x7f6abc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3916399591 shutdown_connections 2023-10-16T09:56:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.715+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3916399591 wait complete. 2023-10-16T09:56:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.716+0000 7f6ac3d9e700 1 Processor -- start 2023-10-16T09:56:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.716+0000 7f6ac3d9e700 1 -- start start 2023-10-16T09:56:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc0cea20 0x7f6abc153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:51.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc1564f0 con 0x7f6abc0cea20 2023-10-16T09:56:51.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc143250 con 0x7f6abc15dad0 2023-10-16T09:56:51.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc155c50 con 0x7f6abc15deb0 2023-10-16T09:56:51.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc0cea20 0x7f6abc153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:51.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc0cea20 0x7f6abc153800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50362/0 (socket says 172.21.15.158:50362) 2023-10-16T09:56:51.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 -- 172.21.15.158:0/3152444836 learned_addr learned my addr 172.21.15.158:0/3152444836 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:51.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1339700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:51.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac233b700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:51.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 -- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 msgr2=0x7f6abc0b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:51.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 -- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 msgr2=0x7f6abc0b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:51.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:51.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 -- 172.21.15.158:0/3152444836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac0094a0 con 0x7f6abc0cea20 2023-10-16T09:56:51.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1339700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:51.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac1b3a700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc0cea20 0x7f6abc153800 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80090b0 tx=0x7f6ab8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:51.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ab27fc700 1 -- 172.21.15.158:0/3152444836 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80045d0 con 0x7f6abc0cea20 2023-10-16T09:56:51.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ab27fc700 1 -- 172.21.15.158:0/3152444836 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab801f070 con 0x7f6abc0cea20 2023-10-16T09:56:51.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab80094a0 con 0x7f6abc0cea20 2023-10-16T09:56:51.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ab27fc700 1 -- 172.21.15.158:0/3152444836 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80038a0 con 0x7f6abc0cea20 2023-10-16T09:56:51.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.717+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc156210 con 0x7f6abc0cea20 2023-10-16T09:56:51.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.718+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc142fb0 con 0x7f6abc0cea20 2023-10-16T09:56:51.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.719+0000 7f6ab27fc700 1 -- 172.21.15.158:0/3152444836 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 18) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6ab8003a00 con 0x7f6abc0cea20 2023-10-16T09:56:51.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.719+0000 7f6ab27fc700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f6aa006af90 0x7f6aa006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:51.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.720+0000 7f6ab27fc700 1 -- 172.21.15.158:0/3152444836 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f6ab8021070 con 0x7f6abc0cea20 2023-10-16T09:56:51.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.720+0000 7f6ac1339700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f6aa006af90 0x7f6aa006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:51.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.720+0000 7f6ac1339700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f6aa006af90 0x7f6aa006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0cd190 tx=0x7f6aac008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:51.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.724+0000 7f6ab27fc700 1 -- 172.21.15.158:0/3152444836 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ab80561b0 con 0x7f6abc0cea20 2023-10-16T09:56:51.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:51.916+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6abc142fb0 con 0x7f6abc0cea20 2023-10-16T09:56:52.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.563+0000 7f6ab27fc700 1 -- 172.21.15.158:0/3152444836 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6abc142fb0 con 0x7f6abc0cea20 2023-10-16T09:56:52.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f6aa006af90 msgr2=0x7f6aa006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f6aa006af90 0x7f6aa006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0cd190 tx=0x7f6aac008040 comp rx=0 tx=0).stop 2023-10-16T09:56:52.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc0cea20 msgr2=0x7f6abc153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc0cea20 0x7f6abc153800 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80090b0 tx=0x7f6ab8004750 comp rx=0 tx=0).stop 2023-10-16T09:56:52.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 shutdown_connections 2023-10-16T09:56:52.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f6aa006af90 0x7f6aa006d450 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0cd190 tx=0x7f6aac008040 comp rx=0 tx=0).stop 2023-10-16T09:56:52.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc15deb0 0x7f6abc0b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.566+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc0cea20 0x7f6abc153800 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.567+0000 7f6ac3d9e700 1 --2- 172.21.15.158:0/3152444836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc15dad0 0x7f6abc0b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.567+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 >> 172.21.15.158:0/3152444836 conn(0x7f6abc099a10 msgr2=0x7f6abc1526a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:52.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.568+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 shutdown_connections 2023-10-16T09:56:52.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.568+0000 7f6ac3d9e700 1 -- 172.21.15.158:0/3152444836 wait complete. 2023-10-16T09:56:52.570 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T09:56:52.584 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T09:56:52.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.718+0000 7ff2ac6e6700 1 Processor -- start 2023-10-16T09:56:52.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.718+0000 7ff2ac6e6700 1 -- start start 2023-10-16T09:56:52.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2ac6e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff2a40ccaf0 0x7ff2a406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:52.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2ac6e6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a406ae10 0x7ff2a40b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:52.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2ac6e6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a40ccf50 con 0x7ff2a415b780 2023-10-16T09:56:52.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2ac6e6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a415bb60 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2ac6e6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cb710 con 0x7ff2a406ae10 2023-10-16T09:56:52.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff2a40ccaf0 0x7ff2a406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:52.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff2a40ccaf0 0x7ff2a406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50372/0 (socket says 172.21.15.158:50372) 2023-10-16T09:56:52.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 -- 172.21.15.158:0/3556474954 learned_addr learned my addr 172.21.15.158:0/3556474954 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:52.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2aa482700 1 --1- 172.21.15.158:0/3556474954 >> v1:172.21.15.158:6789/0 conn(0x7ff2a415b780 0x7ff2a406e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50546/0 (socket says 172.21.15.158:50546) 2023-10-16T09:56:52.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 -- 172.21.15.158:0/3556474954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a406ae10 msgr2=0x7ff2a40b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:52.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 --2- 172.21.15.158:0/3556474954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a406ae10 0x7ff2a40b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 -- 172.21.15.158:0/3556474954 >> v1:172.21.15.158:6789/0 conn(0x7ff2a415b780 legacy=0x7ff2a406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9480700 1 -- 172.21.15.158:0/3556474954 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679426694 0 0) 0x7ff2a0001110 con 0x7ff2a415b780 2023-10-16T09:56:52.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 -- 172.21.15.158:0/3556474954 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40cd190 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9c81700 1 --2- 172.21.15.158:0/3556474954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a406a8d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff2940098f0 tx=0x7ff294009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec45eb0b138056ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:52.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.719+0000 7ff2a9480700 1 -- 172.21.15.158:0/3556474954 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400b8a0 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2a9480700 1 -- 172.21.15.158:0/3556474954 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29400ba00 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2a9480700 1 -- 172.21.15.158:0/3556474954 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400bd90 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/3556474954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 msgr2=0x7ff2a406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/3556474954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a406a8d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff2940098f0 tx=0x7ff294009c00 comp rx=0 tx=0).stop 2023-10-16T09:56:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/3556474954 shutdown_connections 2023-10-16T09:56:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/3556474954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a406ae10 0x7ff2a40b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/3556474954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a406a8d0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/3556474954 >> 172.21.15.158:0/3556474954 conn(0x7ff2a40b1f00 msgr2=0x7ff2a40b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/3556474954 shutdown_connections 2023-10-16T09:56:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.720+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/3556474954 wait complete. 2023-10-16T09:56:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.721+0000 7ff2ac6e6700 1 Processor -- start 2023-10-16T09:56:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.721+0000 7ff2ac6e6700 1 -- start start 2023-10-16T09:56:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2ac6e6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2a406ae10 0x7ff2a414ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:52.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2ac6e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:52.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2ac6e6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a415b780 0x7ff2a40c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:52.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2ac6e6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4152640 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2ac6e6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a414bf80 con 0x7ff2a406ae10 2023-10-16T09:56:52.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2ac6e6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40c5f60 con 0x7ff2a415b780 2023-10-16T09:56:52.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:52.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a415bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50378/0 (socket says 172.21.15.158:50378) 2023-10-16T09:56:52.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 -- 172.21.15.158:0/347153929 learned_addr learned my addr 172.21.15.158:0/347153929 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:52.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2aac83700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a415b780 0x7ff2a40c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:52.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 -- 172.21.15.158:0/347153929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a415b780 msgr2=0x7ff2a40c4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a415b780 0x7ff2a40c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 -- 172.21.15.158:0/347153929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2a406ae10 msgr2=0x7ff2a414ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2a406ae10 0x7ff2a414ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.722+0000 7ff2a9c81700 1 -- 172.21.15.158:0/347153929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff294009550 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.723+0000 7ff2a9c81700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a415bca0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff294009c40 tx=0x7ff294004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:52.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.723+0000 7ff29affd700 1 -- 172.21.15.158:0/347153929 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294004480 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.723+0000 7ff29affd700 1 -- 172.21.15.158:0/347153929 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29401f070 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.723+0000 7ff29affd700 1 -- 172.21.15.158:0/347153929 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2940038a0 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.723+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a415c1e0 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.723+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a40c5c80 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.724+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a415ebc0 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.724+0000 7ff29affd700 1 -- 172.21.15.158:0/347153929 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff294021030 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.725+0000 7ff29affd700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff28806b020 0x7ff28806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:52.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.725+0000 7ff29affd700 1 -- 172.21.15.158:0/347153929 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7ff29408b060 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.725+0000 7ff2aa482700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff28806b020 0x7ff28806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:52.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.725+0000 7ff2aa482700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff28806b020 0x7ff28806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0001610 tx=0x7ff2a00079f0 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:52.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.729+0000 7ff29affd700 1 -- 172.21.15.158:0/347153929 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2940bdfe0 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.923+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7ff2a40681b0 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.926+0000 7ff29affd700 1 -- 172.21.15.158:0/347153929 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff294015020 con 0x7ff2a40ccaf0 2023-10-16T09:56:52.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.928+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff28806b020 msgr2=0x7ff28806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.928+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff28806b020 0x7ff28806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0001610 tx=0x7ff2a00079f0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.928+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 msgr2=0x7ff2a415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:52.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.928+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a415bca0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff294009c40 tx=0x7ff294004750 comp rx=0 tx=0).stop 2023-10-16T09:56:52.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.928+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 shutdown_connections 2023-10-16T09:56:52.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.928+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff28806b020 0x7ff28806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.928+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2a415b780 0x7ff2a40c4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.929+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2a406ae10 0x7ff2a414ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.929+0000 7ff2ac6e6700 1 --2- 172.21.15.158:0/347153929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2a40ccaf0 0x7ff2a415bca0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:52.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.929+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 >> 172.21.15.158:0/347153929 conn(0x7ff2a40b1f00 msgr2=0x7ff2a40b5740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:52.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.929+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 shutdown_connections 2023-10-16T09:56:52.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:52.929+0000 7ff2ac6e6700 1 -- 172.21.15.158:0/347153929 wait complete. 2023-10-16T09:56:52.942 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T09:56:53.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.063+0000 7f343edd0700 1 Processor -- start 2023-10-16T09:56:53.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.064+0000 7f343edd0700 1 -- start start 2023-10-16T09:56:53.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.064+0000 7f343edd0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f343815da40 0x7f34380caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.064+0000 7f343edd0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f343806afe0 0x7f34380afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.064+0000 7f343edd0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34380cbd30 con 0x7f34380cc590 2023-10-16T09:56:53.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.064+0000 7f343edd0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34380cc1c0 con 0x7f343806afe0 2023-10-16T09:56:53.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.064+0000 7f343edd0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343815de20 con 0x7f343815da40 2023-10-16T09:56:53.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.064+0000 7f343d36d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f343806afe0 0x7f34380afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343d36d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f343806afe0 0x7f34380afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50388/0 (socket says 172.21.15.158:50388) 2023-10-16T09:56:53.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343d36d700 1 -- 172.21.15.158:0/2635460602 learned_addr learned my addr 172.21.15.158:0/2635460602 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:53.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343cb6c700 1 --2- 172.21.15.158:0/2635460602 >> v2:172.21.15.172:3300/0 conn(0x7f343815da40 0x7f34380caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343d36d700 1 -- 172.21.15.158:0/2635460602 >> v2:172.21.15.172:3300/0 conn(0x7f343815da40 msgr2=0x7f34380caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:53.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343d36d700 1 --2- 172.21.15.158:0/2635460602 >> v2:172.21.15.172:3300/0 conn(0x7f343815da40 0x7f34380caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343d36d700 1 -- 172.21.15.158:0/2635460602 >> v1:172.21.15.172:6789/0 conn(0x7f34380cc590 legacy=0x7f343806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343d36d700 1 -- 172.21.15.158:0/2635460602 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380ce260 con 0x7f343806afe0 2023-10-16T09:56:53.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f343d36d700 1 --2- 172.21.15.158:0/2635460602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343806afe0 0x7f34380afa40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f343806c9c0 tx=0x7f343000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8772937f489af630 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f34377fe700 1 -- 172.21.15.158:0/2635460602 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000e070 con 0x7f343806afe0 2023-10-16T09:56:53.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f34377fe700 1 -- 172.21.15.158:0/2635460602 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343000b9c0 con 0x7f343806afe0 2023-10-16T09:56:53.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.065+0000 7f34377fe700 1 -- 172.21.15.158:0/2635460602 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000bd70 con 0x7f343806afe0 2023-10-16T09:56:53.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 -- 172.21.15.158:0/2635460602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343806afe0 msgr2=0x7f34380afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 --2- 172.21.15.158:0/2635460602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343806afe0 0x7f34380afa40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f343806c9c0 tx=0x7f343000b330 comp rx=0 tx=0).stop 2023-10-16T09:56:53.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 -- 172.21.15.158:0/2635460602 shutdown_connections 2023-10-16T09:56:53.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 --2- 172.21.15.158:0/2635460602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343806afe0 0x7f34380afa40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 --2- 172.21.15.158:0/2635460602 >> v2:172.21.15.172:3300/0 conn(0x7f343815da40 0x7f34380caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 -- 172.21.15.158:0/2635460602 >> 172.21.15.158:0/2635460602 conn(0x7f3438099a10 msgr2=0x7f34380d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:53.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 -- 172.21.15.158:0/2635460602 shutdown_connections 2023-10-16T09:56:53.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.066+0000 7f343edd0700 1 -- 172.21.15.158:0/2635460602 wait complete. 2023-10-16T09:56:53.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.067+0000 7f343edd0700 1 Processor -- start 2023-10-16T09:56:53.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.067+0000 7f343edd0700 1 -- start start 2023-10-16T09:56:53.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343edd0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f343806afe0 0x7f34380c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343edd0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34380cc590 0x7f34381e4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343edd0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343815da40 0x7f34381e3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343edd0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34380c9970 con 0x7f34380cc590 2023-10-16T09:56:53.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343edd0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34380c8d60 con 0x7f343815da40 2023-10-16T09:56:53.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343edd0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34380c9040 con 0x7f343806afe0 2023-10-16T09:56:53.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34380cc590 0x7f34381e4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34380cc590 0x7f34381e4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50392/0 (socket says 172.21.15.158:50392) 2023-10-16T09:56:53.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 -- 172.21.15.158:0/3710962813 learned_addr learned my addr 172.21.15.158:0/3710962813 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:53.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343d36d700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343815da40 0x7f34381e3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f343cb6c700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f343806afe0 0x7f34380c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 -- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f343806afe0 msgr2=0x7f34380c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f343806afe0 0x7f34380c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 -- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343815da40 msgr2=0x7f34381e3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343815da40 0x7f34381e3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 -- 172.21.15.158:0/3710962813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3430009b30 con 0x7f34380cc590 2023-10-16T09:56:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f3437fff700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34380cc590 0x7f34381e4f80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f342400c4b0 tx=0x7f342400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.068+0000 7f34357fa700 1 -- 172.21.15.158:0/3710962813 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34240052d0 con 0x7f34380cc590 2023-10-16T09:56:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.069+0000 7f34357fa700 1 -- 172.21.15.158:0/3710962813 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3424005430 con 0x7f34380cc590 2023-10-16T09:56:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.069+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34380c9320 con 0x7f34380cc590 2023-10-16T09:56:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.069+0000 7f34357fa700 1 -- 172.21.15.158:0/3710962813 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34240038c0 con 0x7f34380cc590 2023-10-16T09:56:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.069+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34381e54c0 con 0x7f34380cc590 2023-10-16T09:56:53.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.069+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34380ca340 con 0x7f34380cc590 2023-10-16T09:56:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.070+0000 7f34357fa700 1 -- 172.21.15.158:0/3710962813 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3424003a20 con 0x7f34380cc590 2023-10-16T09:56:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.070+0000 7f34357fa700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f341c06b050 0x7f341c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.070+0000 7f34357fa700 1 -- 172.21.15.158:0/3710962813 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f3424088dc0 con 0x7f34380cc590 2023-10-16T09:56:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.070+0000 7f343cb6c700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f341c06b050 0x7f341c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.071+0000 7f343cb6c700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f341c06b050 0x7f341c06d510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f342c006b20 tx=0x7f342c008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.075+0000 7f34357fa700 1 -- 172.21.15.158:0/3710962813 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3424053a80 con 0x7f34380cc590 2023-10-16T09:56:53.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.270+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f34380681b0 con 0x7f34380cc590 2023-10-16T09:56:53.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.273+0000 7f34357fa700 1 -- 172.21.15.158:0/3710962813 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f34240170a0 con 0x7f34380cc590 2023-10-16T09:56:53.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f341c06b050 msgr2=0x7f341c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f341c06b050 0x7f341c06d510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f342c006b20 tx=0x7f342c008040 comp rx=0 tx=0).stop 2023-10-16T09:56:53.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34380cc590 msgr2=0x7f34381e4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34380cc590 0x7f34381e4f80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f342400c4b0 tx=0x7f342400c7c0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 shutdown_connections 2023-10-16T09:56:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f341c06b050 0x7f341c06d510 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343815da40 0x7f34381e3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f343806afe0 0x7f34380c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 --2- 172.21.15.158:0/3710962813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34380cc590 0x7f34381e4f80 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 >> 172.21.15.158:0/3710962813 conn(0x7f3438099a10 msgr2=0x7f343815ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 shutdown_connections 2023-10-16T09:56:53.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.275+0000 7f343edd0700 1 -- 172.21.15.158:0/3710962813 wait complete. 2023-10-16T09:56:53.289 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T09:56:53.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.410+0000 7febb7af7700 1 Processor -- start 2023-10-16T09:56:53.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.411+0000 7febb7af7700 1 -- start start 2023-10-16T09:56:53.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.411+0000 7febb7af7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febb00cdfa0 0x7febb006d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.411+0000 7febb7af7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00cc730 con 0x7febb01516c0 2023-10-16T09:56:53.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.411+0000 7febb7af7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb006e290 con 0x7febb006de60 2023-10-16T09:56:53.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.411+0000 7febb7af7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb00ce430 con 0x7febb00cdfa0 2023-10-16T09:56:53.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb5092700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febb00cdfa0 0x7febb006d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb5092700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febb00cdfa0 0x7febb006d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50400/0 (socket says 172.21.15.158:50400) 2023-10-16T09:56:53.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb5092700 1 -- 172.21.15.158:0/2568044004 learned_addr learned my addr 172.21.15.158:0/2568044004 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:53.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb5092700 1 -- 172.21.15.158:0/2568044004 >> v1:172.21.15.172:6789/0 conn(0x7febb006de60 legacy=0x7febb00ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb5092700 1 -- 172.21.15.158:0/2568044004 >> v1:172.21.15.158:6789/0 conn(0x7febb01516c0 legacy=0x7febb0072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb5092700 1 -- 172.21.15.158:0/2568044004 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb0151aa0 con 0x7febb00cdfa0 2023-10-16T09:56:53.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb5092700 1 --2- 172.21.15.158:0/2568044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb00cdfa0 0x7febb006d920 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7febac0098f0 tx=0x7febac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a3ad9cbb288eaa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb4891700 1 -- 172.21.15.158:0/2568044004 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac00b8a0 con 0x7febb00cdfa0 2023-10-16T09:56:53.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb4891700 1 -- 172.21.15.158:0/2568044004 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febac00ba00 con 0x7febb00cdfa0 2023-10-16T09:56:53.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.412+0000 7febb4891700 1 -- 172.21.15.158:0/2568044004 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac00bd90 con 0x7febb00cdfa0 2023-10-16T09:56:53.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.413+0000 7febb7af7700 1 -- 172.21.15.158:0/2568044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb00cdfa0 msgr2=0x7febb006d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.413+0000 7febb7af7700 1 --2- 172.21.15.158:0/2568044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb00cdfa0 0x7febb006d920 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7febac0098f0 tx=0x7febac009c00 comp rx=0 tx=0).stop 2023-10-16T09:56:53.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.413+0000 7febb7af7700 1 -- 172.21.15.158:0/2568044004 shutdown_connections 2023-10-16T09:56:53.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.413+0000 7febb7af7700 1 --2- 172.21.15.158:0/2568044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb00cdfa0 0x7febb006d920 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.413+0000 7febb7af7700 1 -- 172.21.15.158:0/2568044004 >> 172.21.15.158:0/2568044004 conn(0x7febb00b1f00 msgr2=0x7febb00b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:53.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.413+0000 7febb7af7700 1 -- 172.21.15.158:0/2568044004 shutdown_connections 2023-10-16T09:56:53.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.413+0000 7febb7af7700 1 -- 172.21.15.158:0/2568044004 wait complete. 2023-10-16T09:56:53.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.414+0000 7febb7af7700 1 Processor -- start 2023-10-16T09:56:53.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.414+0000 7febb7af7700 1 -- start start 2023-10-16T09:56:53.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb7af7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006de60 0x7febb00c4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb7af7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb00cdfa0 0x7febb014f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb7af7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb01516c0 0x7febb0153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb7af7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb00c9230 con 0x7febb006de60 2023-10-16T09:56:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb7af7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb00c4c50 con 0x7febb01516c0 2023-10-16T09:56:53.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb7af7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb00c8620 con 0x7febb00cdfa0 2023-10-16T09:56:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb5893700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006de60 0x7febb00c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.415+0000 7febb5893700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006de60 0x7febb00c4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50404/0 (socket says 172.21.15.158:50404) 2023-10-16T09:56:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb5893700 1 -- 172.21.15.158:0/3740454072 learned_addr learned my addr 172.21.15.158:0/3740454072 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb5893700 1 -- 172.21.15.158:0/3740454072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb00cdfa0 msgr2=0x7febb014f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb5893700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb00cdfa0 0x7febb014f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb5893700 1 -- 172.21.15.158:0/3740454072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb01516c0 msgr2=0x7febb0153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb5893700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb01516c0 0x7febb0153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb5893700 1 -- 172.21.15.158:0/3740454072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac009550 con 0x7febb006de60 2023-10-16T09:56:53.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb5893700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006de60 0x7febb00c4710 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7feba000e940 tx=0x7feba000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7feba67fc700 1 -- 172.21.15.158:0/3740454072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba00052d0 con 0x7febb006de60 2023-10-16T09:56:53.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7feba67fc700 1 -- 172.21.15.158:0/3740454072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba0005430 con 0x7febb006de60 2023-10-16T09:56:53.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7feba67fc700 1 -- 172.21.15.158:0/3740454072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba000cbd0 con 0x7febb006de60 2023-10-16T09:56:53.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb00c8900 con 0x7febb006de60 2023-10-16T09:56:53.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.416+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb00c8ec0 con 0x7febb006de60 2023-10-16T09:56:53.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.417+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb00681b0 con 0x7febb006de60 2023-10-16T09:56:53.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.417+0000 7feba67fc700 1 -- 172.21.15.158:0/3740454072 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feba000cd30 con 0x7febb006de60 2023-10-16T09:56:53.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.418+0000 7feba67fc700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7feb9406b020 0x7feb9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.418+0000 7feba67fc700 1 -- 172.21.15.158:0/3740454072 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7feba008bbe0 con 0x7febb006de60 2023-10-16T09:56:53.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.418+0000 7febb5092700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7feb9406b020 0x7feb9406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.418+0000 7febb5092700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7feb9406b020 0x7feb9406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7febb00cfb00 tx=0x7febac010000 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.422+0000 7feba67fc700 1 -- 172.21.15.158:0/3740454072 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feba0056830 con 0x7febb006de60 2023-10-16T09:56:53.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.616+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7febb006e7b0 con 0x7febb006de60 2023-10-16T09:56:53.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.619+0000 7feba67fc700 1 -- 172.21.15.158:0/3740454072 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7feba001f0b0 con 0x7febb006de60 2023-10-16T09:56:53.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7feb9406b020 msgr2=0x7feb9406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7feb9406b020 0x7feb9406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7febb00cfb00 tx=0x7febac010000 comp rx=0 tx=0).stop 2023-10-16T09:56:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006de60 msgr2=0x7febb00c4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006de60 0x7febb00c4710 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7feba000e940 tx=0x7feba000ed00 comp rx=0 tx=0).stop 2023-10-16T09:56:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 shutdown_connections 2023-10-16T09:56:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7feb9406b020 0x7feb9406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb01516c0 0x7febb0153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006de60 0x7febb00c4710 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 --2- 172.21.15.158:0/3740454072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb00cdfa0 0x7febb014f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 >> 172.21.15.158:0/3740454072 conn(0x7febb00b1f00 msgr2=0x7febb00b0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 shutdown_connections 2023-10-16T09:56:53.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.622+0000 7febb7af7700 1 -- 172.21.15.158:0/3740454072 wait complete. 2023-10-16T09:56:53.636 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T09:56:53.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.756+0000 7ff98b367700 1 Processor -- start 2023-10-16T09:56:53.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.757+0000 7ff98b367700 1 -- start start 2023-10-16T09:56:53.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.757+0000 7ff98b367700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff98415dbc0 0x7ff9840cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.757+0000 7ff98b367700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff9840d1d50 0x7ff9840cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.757+0000 7ff98b367700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9840d21e0 con 0x7ff98406ba90 2023-10-16T09:56:53.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff98b367700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff984162550 con 0x7ff9840d1d50 2023-10-16T09:56:53.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff98b367700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98415f320 con 0x7ff98415dbc0 2023-10-16T09:56:53.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff989904700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff98406ba90 0x7ff98406beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50592/0 (socket says 172.21.15.158:50592) 2023-10-16T09:56:53.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff989904700 1 -- 172.21.15.158:0/4150680981 learned_addr learned my addr 172.21.15.158:0/4150680981 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:53.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760747632 0 0) 0x7ff9840d21e0 con 0x7ff98406ba90 2023-10-16T09:56:53.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9700035b0 con 0x7ff98406ba90 2023-10-16T09:56:53.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff988902700 1 --2- 172.21.15.158:0/4150680981 >> v2:172.21.15.172:3300/0 conn(0x7ff9840d1d50 0x7ff9840cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff989103700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff98415dbc0 0x7ff9840cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2758160721 0 0) 0x7ff9700035b0 con 0x7ff98406ba90 2023-10-16T09:56:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff98415dbc0 msgr2=0x7ff9840cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff98415dbc0 0x7ff9840cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 >> v2:172.21.15.172:3300/0 conn(0x7ff9840d1d50 msgr2=0x7ff9840cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 --2- 172.21.15.158:0/4150680981 >> v2:172.21.15.172:3300/0 conn(0x7ff9840d1d50 0x7ff9840cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98415ef10 con 0x7ff98406ba90 2023-10-16T09:56:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff97c0019f0 con 0x7ff98406ba90 2023-10-16T09:56:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 >> v1:172.21.15.158:6789/0 conn(0x7ff98406ba90 legacy=0x7ff98406beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff970003a00 0x7ff970003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff970004360 0x7ff970004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff970004ea0 0x7ff970005320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff984162550 con 0x7ff970004ea0 2023-10-16T09:56:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97000bdb0 con 0x7ff970004360 2023-10-16T09:56:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97000bf70 con 0x7ff970003a00 2023-10-16T09:56:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff989904700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff970004ea0 0x7ff970005320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff989103700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff970003a00 0x7ff970003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.758+0000 7ff988902700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff970004360 0x7ff970004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff989904700 1 -- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff970003a00 msgr2=0x7ff970003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:53.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff989904700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff970003a00 0x7ff970003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff989904700 1 -- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff970004360 msgr2=0x7ff970004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:53.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff989904700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff970004360 0x7ff970004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff989904700 1 -- 172.21.15.158:0/4150680981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700035b0 con 0x7ff970004ea0 2023-10-16T09:56:53.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff989904700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff970004ea0 0x7ff970005320 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff9840b3700 tx=0x7ff970005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3ee5908fa8b176 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00a320 con 0x7ff970004ea0 2023-10-16T09:56:53.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c012600 con 0x7ff970004ea0 2023-10-16T09:56:53.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff97bfff700 1 -- 172.21.15.158:0/4150680981 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0129b0 con 0x7ff970004ea0 2023-10-16T09:56:53.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff98b367700 1 -- 172.21.15.158:0/4150680981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff970004ea0 msgr2=0x7ff970005320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff98b367700 1 --2- 172.21.15.158:0/4150680981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff970004ea0 0x7ff970005320 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff9840b3700 tx=0x7ff970005880 comp rx=0 tx=0).stop 2023-10-16T09:56:53.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff989904700 1 -- 172.21.15.158:0/4150680981 reap_dead start 2023-10-16T09:56:53.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff98b367700 1 -- 172.21.15.158:0/4150680981 shutdown_connections 2023-10-16T09:56:53.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.759+0000 7ff98b367700 1 -- 172.21.15.158:0/4150680981 >> 172.21.15.158:0/4150680981 conn(0x7ff98406a8d0 msgr2=0x7ff98406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:53.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.760+0000 7ff98b367700 1 -- 172.21.15.158:0/4150680981 shutdown_connections 2023-10-16T09:56:53.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.760+0000 7ff98b367700 1 -- 172.21.15.158:0/4150680981 wait complete. 2023-10-16T09:56:53.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 Processor -- start 2023-10-16T09:56:53.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 -- start start 2023-10-16T09:56:53.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9840cbd20 0x7ff9840b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9840cafc0 0x7ff984160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9840cc100 0x7ff98414e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff984161960 con 0x7ff9840cc100 2023-10-16T09:56:53.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840b6bd0 con 0x7ff9840cbd20 2023-10-16T09:56:53.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.761+0000 7ff98b367700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff984160ff0 con 0x7ff9840cafc0 2023-10-16T09:56:53.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9840cc100 0x7ff98414e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9840cc100 0x7ff98414e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50428/0 (socket says 172.21.15.158:50428) 2023-10-16T09:56:53.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 -- 172.21.15.158:0/3673239912 learned_addr learned my addr 172.21.15.158:0/3673239912 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:53.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989103700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9840cbd20 0x7ff9840b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff988902700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9840cafc0 0x7ff984160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 -- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9840cafc0 msgr2=0x7ff984160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9840cafc0 0x7ff984160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 -- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9840cbd20 msgr2=0x7ff9840b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:53.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9840cbd20 0x7ff9840b6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:53.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 -- 172.21.15.158:0/3673239912 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984161bb0 con 0x7ff9840cc100 2023-10-16T09:56:53.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff989904700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9840cc100 0x7ff98414e410 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff97c012000 tx=0x7ff9840d2370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff979ffb700 1 -- 172.21.15.158:0/3673239912 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00a320 con 0x7ff9840cc100 2023-10-16T09:56:53.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff979ffb700 1 -- 172.21.15.158:0/3673239912 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c0019c0 con 0x7ff9840cc100 2023-10-16T09:56:53.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff984161e90 con 0x7ff9840cc100 2023-10-16T09:56:53.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff979ffb700 1 -- 172.21.15.158:0/3673239912 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c027410 con 0x7ff9840cc100 2023-10-16T09:56:53.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.762+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9840b5e90 con 0x7ff9840cc100 2023-10-16T09:56:53.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.763+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98415e410 con 0x7ff9840cc100 2023-10-16T09:56:53.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.764+0000 7ff979ffb700 1 -- 172.21.15.158:0/3673239912 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff97c019070 con 0x7ff9840cc100 2023-10-16T09:56:53.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.764+0000 7ff979ffb700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff96806b020 0x7ff96806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:53.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.764+0000 7ff979ffb700 1 -- 172.21.15.158:0/3673239912 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7ff97c08ad60 con 0x7ff9840cc100 2023-10-16T09:56:53.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.764+0000 7ff989103700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff96806b020 0x7ff96806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:53.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.765+0000 7ff989103700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff96806b020 0x7ff96806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff974009be0 tx=0x7ff974009210 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:53.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:53.769+0000 7ff979ffb700 1 -- 172.21.15.158:0/3673239912 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff98415e410 con 0x7ff9840cc100 2023-10-16T09:56:54.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.042+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7ff98415e410 con 0x7ff9840cc100 2023-10-16T09:56:54.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.565+0000 7ff979ffb700 1 -- 172.21.15.158:0/3673239912 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v485) v1 ==== 158+0+0 (secure 0 0 0) 0x7ff98415e410 con 0x7ff9840cc100 2023-10-16T09:56:54.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff96806b020 msgr2=0x7ff96806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff96806b020 0x7ff96806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff974009be0 tx=0x7ff974009210 comp rx=0 tx=0).stop 2023-10-16T09:56:54.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9840cc100 msgr2=0x7ff98414e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9840cc100 0x7ff98414e410 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff97c012000 tx=0x7ff9840d2370 comp rx=0 tx=0).stop 2023-10-16T09:56:54.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 shutdown_connections 2023-10-16T09:56:54.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff96806b020 0x7ff96806d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9840cc100 0x7ff98414e410 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9840cbd20 0x7ff9840b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 --2- 172.21.15.158:0/3673239912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9840cafc0 0x7ff984160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 >> 172.21.15.158:0/3673239912 conn(0x7ff98406a8d0 msgr2=0x7ff98415cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:54.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 shutdown_connections 2023-10-16T09:56:54.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.569+0000 7ff98b367700 1 -- 172.21.15.158:0/3673239912 wait complete. 2023-10-16T09:56:54.585 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-10-16T09:56:54.586 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-10-16T09:56:54.586 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T09:56:54.587 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:56:54.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.712+0000 7f202e40d700 1 Processor -- start 2023-10-16T09:56:54.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f202e40d700 1 -- start start 2023-10-16T09:56:54.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f202e40d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20280ca980 0x7f20280cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:54.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f202e40d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20280cb2a0 con 0x7f2028071980 2023-10-16T09:56:54.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f202e40d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202815f1a0 con 0x7f20280cc910 2023-10-16T09:56:54.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f202e40d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2028071e80 con 0x7f20280ca980 2023-10-16T09:56:54.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f2027fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20280ca980 0x7f20280cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:54.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f2027fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20280ca980 0x7f20280cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50432/0 (socket says 172.21.15.158:50432) 2023-10-16T09:56:54.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.713+0000 7f2027fff700 1 -- 172.21.15.158:0/2905620488 learned_addr learned my addr 172.21.15.158:0/2905620488 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:54.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f2027fff700 1 -- 172.21.15.158:0/2905620488 >> v1:172.21.15.172:6789/0 conn(0x7f20280cc910 legacy=0x7f202815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:54.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f2027fff700 1 -- 172.21.15.158:0/2905620488 >> v1:172.21.15.158:6789/0 conn(0x7f2028071980 legacy=0x7f202815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f2027fff700 1 -- 172.21.15.158:0/2905620488 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2028072160 con 0x7f20280ca980 2023-10-16T09:56:54.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f2027fff700 1 --2- 172.21.15.158:0/2905620488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20280ca980 0x7f20280cad60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f20180098f0 tx=0x7f2018009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6de37b1f2dba07 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:54.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f2026ffd700 1 -- 172.21.15.158:0/2905620488 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018004230 con 0x7f20280ca980 2023-10-16T09:56:54.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f2026ffd700 1 -- 172.21.15.158:0/2905620488 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2018004390 con 0x7f20280ca980 2023-10-16T09:56:54.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f2026ffd700 1 -- 172.21.15.158:0/2905620488 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20180036a0 con 0x7f20280ca980 2023-10-16T09:56:54.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.714+0000 7f202e40d700 1 -- 172.21.15.158:0/2905620488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20280ca980 msgr2=0x7f20280cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.715+0000 7f202e40d700 1 --2- 172.21.15.158:0/2905620488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20280ca980 0x7f20280cad60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f20180098f0 tx=0x7f2018009c00 comp rx=0 tx=0).stop 2023-10-16T09:56:54.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.715+0000 7f202e40d700 1 -- 172.21.15.158:0/2905620488 shutdown_connections 2023-10-16T09:56:54.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.715+0000 7f202e40d700 1 --2- 172.21.15.158:0/2905620488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20280ca980 0x7f20280cad60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.715+0000 7f202e40d700 1 -- 172.21.15.158:0/2905620488 >> 172.21.15.158:0/2905620488 conn(0x7f20280ccdb0 msgr2=0x7f20280cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:54.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.715+0000 7f202e40d700 1 -- 172.21.15.158:0/2905620488 shutdown_connections 2023-10-16T09:56:54.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.715+0000 7f202e40d700 1 -- 172.21.15.158:0/2905620488 wait complete. 2023-10-16T09:56:54.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.716+0000 7f202e40d700 1 Processor -- start 2023-10-16T09:56:54.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.716+0000 7f202e40d700 1 -- start start 2023-10-16T09:56:54.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2028071980 0x7f20280c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:54.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20280ca980 0x7f20280b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:54.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20280cc910 0x7f202815a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:54.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028156f00 con 0x7f2028071980 2023-10-16T09:56:54.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280b5930 con 0x7f20280ca980 2023-10-16T09:56:54.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20280b5c10 con 0x7f20280cc910 2023-10-16T09:56:54.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2028071980 0x7f20280c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:54.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2028071980 0x7f20280c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50436/0 (socket says 172.21.15.158:50436) 2023-10-16T09:56:54.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 -- 172.21.15.158:0/1503391315 learned_addr learned my addr 172.21.15.158:0/1503391315 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:54.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 -- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20280cc910 msgr2=0x7f202815a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202c9aa700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20280cc910 0x7f202815a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:54.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f20277fe700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20280ca980 0x7f20280b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:54.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20280cc910 0x7f202815a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 -- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20280ca980 msgr2=0x7f20280b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20280ca980 0x7f20280b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 -- 172.21.15.158:0/1503391315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2018009550 con 0x7f2028071980 2023-10-16T09:56:54.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f20277fe700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20280ca980 0x7f20280b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:54.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2027fff700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2028071980 0x7f20280c6260 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f202806cdc0 tx=0x7f2018004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:54.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2024ff9700 1 -- 172.21.15.158:0/1503391315 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201800b660 con 0x7f2028071980 2023-10-16T09:56:54.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2024ff9700 1 -- 172.21.15.158:0/1503391315 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201800b800 con 0x7f2028071980 2023-10-16T09:56:54.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20280afbd0 con 0x7f2028071980 2023-10-16T09:56:54.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f2024ff9700 1 -- 172.21.15.158:0/1503391315 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201800bbb0 con 0x7f2028071980 2023-10-16T09:56:54.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.717+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20280b33f0 con 0x7f2028071980 2023-10-16T09:56:54.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.718+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20280b00f0 con 0x7f2028071980 2023-10-16T09:56:54.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.719+0000 7f2024ff9700 1 -- 172.21.15.158:0/1503391315 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f201800bd10 con 0x7f2028071980 2023-10-16T09:56:54.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.719+0000 7f2024ff9700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f201406b020 0x7f201406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:54.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.720+0000 7f20277fe700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f201406b020 0x7f201406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:54.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.720+0000 7f2024ff9700 1 -- 172.21.15.158:0/1503391315 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f201808b3b0 con 0x7f2028071980 2023-10-16T09:56:54.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.720+0000 7f20277fe700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f201406b020 0x7f201406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f201c006fd0 tx=0x7f201c008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:54.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.724+0000 7f2024ff9700 1 -- 172.21.15.158:0/1503391315 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2018056230 con 0x7f2028071980 2023-10-16T09:56:54.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.916+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20280681b0 con 0x7f2028071980 2023-10-16T09:56:54.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.917+0000 7f2024ff9700 1 -- 172.21.15.158:0/1503391315 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+14757 (secure 0 0 0) 0x7f2018059850 con 0x7f2028071980 2023-10-16T09:56:54.919 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:54.919 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":58,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:56:52.559625+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6834/1706659835":"2023-10-17T09:56:49.244235+0000","172.21.15.158:6832/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/2730663923":"2023-10-17T09:44:59.554471+0000","172.21.15.158:6833/4024924909":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/1760463794":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/3457698273":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/3774227013":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/1636480876":"2023-10-17T09:56:48.035698+0000","172.21.15.158:6832/3552602298":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/4158145300":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/251660717":"2023-10-17T09:44:59.554471+0000","172.21.15.158:0/73320321":"2023-10-17T09:56:48.035698+0000","172.21.15.158:6833/3552602298":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/686968952":"2023-10-17T09:44:59.554471+0000","172.21.15.172:6832/1706659835":"2023-10-17T09:56:49.244235+0000","172.21.15.158:0/102656464":"2023-10-17T09:56:48.035698+0000","172.21.15.158:0/277874160":"2023-10-17T09:56:48.035698+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:56:54.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.919+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f201406b020 msgr2=0x7f201406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.919+0000 7f202e40d700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f201406b020 0x7f201406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f201c006fd0 tx=0x7f201c008040 comp rx=0 tx=0).stop 2023-10-16T09:56:54.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.919+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2028071980 msgr2=0x7f20280c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.919+0000 7f202e40d700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2028071980 0x7f20280c6260 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f202806cdc0 tx=0x7f2018004750 comp rx=0 tx=0).stop 2023-10-16T09:56:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.919+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 shutdown_connections 2023-10-16T09:56:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.919+0000 7f202e40d700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f201406b020 0x7f201406d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.920+0000 7f202e40d700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20280cc910 0x7f202815a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.920+0000 7f202e40d700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2028071980 0x7f20280c6260 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.920+0000 7f202e40d700 1 --2- 172.21.15.158:0/1503391315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20280ca980 0x7f20280b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:54.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.920+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 >> 172.21.15.158:0/1503391315 conn(0x7f20280ccdb0 msgr2=0x7f2028158970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:54.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.920+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 shutdown_connections 2023-10-16T09:56:54.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:54.920+0000 7f202e40d700 1 -- 172.21.15.158:0/1503391315 wait complete. 2023-10-16T09:56:54.933 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:56:54.934 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:56:55.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.056+0000 7f6173668700 1 Processor -- start 2023-10-16T09:56:55.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.056+0000 7f6173668700 1 -- start start 2023-10-16T09:56:55.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6173668700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f616c141920 0x7f616c06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6173668700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c144840 0x7f616c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6173668700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c06c210 con 0x7f616c0cdc40 2023-10-16T09:56:55.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6173668700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0ce0d0 con 0x7f616c141920 2023-10-16T09:56:55.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6173668700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c141d00 con 0x7f616c144840 2023-10-16T09:56:55.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6170c03700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c144840 0x7f616c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6171404700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f616c141920 0x7f616c06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6171c05700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f616c0cdc40 0x7f616c15f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:54524/0 (socket says 172.21.15.158:54524) 2023-10-16T09:56:55.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.057+0000 7f6171c05700 1 -- 172.21.15.158:0/3564466668 learned_addr learned my addr 172.21.15.158:0/3564466668 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:55.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6170c03700 1 -- 172.21.15.158:0/3564466668 >> v2:172.21.15.172:3300/0 conn(0x7f616c141920 msgr2=0x7f616c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:55.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6170c03700 1 --2- 172.21.15.158:0/3564466668 >> v2:172.21.15.172:3300/0 conn(0x7f616c141920 0x7f616c06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6170c03700 1 -- 172.21.15.158:0/3564466668 >> v1:172.21.15.172:6789/0 conn(0x7f616c0cdc40 legacy=0x7f616c15f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6170c03700 1 -- 172.21.15.158:0/3564466668 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c06b970 con 0x7f616c144840 2023-10-16T09:56:55.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6171404700 1 --2- 172.21.15.158:0/3564466668 >> v2:172.21.15.172:3300/0 conn(0x7f616c141920 0x7f616c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:55.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6170c03700 1 --2- 172.21.15.158:0/3564466668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c144840 0x7f616c0cd700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6168009840 tx=0x7f6168009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dadf7d37a1b14b30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:55.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6163fff700 1 -- 172.21.15.158:0/3564466668 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168011830 con 0x7f616c144840 2023-10-16T09:56:55.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6163fff700 1 -- 172.21.15.158:0/3564466668 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168011990 con 0x7f616c144840 2023-10-16T09:56:55.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.058+0000 7f6163fff700 1 -- 172.21.15.158:0/3564466668 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168011d20 con 0x7f616c144840 2023-10-16T09:56:55.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 -- 172.21.15.158:0/3564466668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c144840 msgr2=0x7f616c0cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 --2- 172.21.15.158:0/3564466668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c144840 0x7f616c0cd700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6168009840 tx=0x7f6168009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:55.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 -- 172.21.15.158:0/3564466668 shutdown_connections 2023-10-16T09:56:55.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 --2- 172.21.15.158:0/3564466668 >> v2:172.21.15.172:3300/0 conn(0x7f616c141920 0x7f616c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 --2- 172.21.15.158:0/3564466668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c144840 0x7f616c0cd700 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 -- 172.21.15.158:0/3564466668 >> 172.21.15.158:0/3564466668 conn(0x7f616c0cc5b0 msgr2=0x7f616c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:55.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 -- 172.21.15.158:0/3564466668 shutdown_connections 2023-10-16T09:56:55.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.059+0000 7f6173668700 1 -- 172.21.15.158:0/3564466668 wait complete. 2023-10-16T09:56:55.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.060+0000 7f6173668700 1 Processor -- start 2023-10-16T09:56:55.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.060+0000 7f6173668700 1 -- start start 2023-10-16T09:56:55.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6173668700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0cdc40 0x7f616c0aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6173668700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c141920 0x7f616c0c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6173668700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c144840 0x7f616c1e1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6173668700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1dc5e0 con 0x7f616c0cdc40 2023-10-16T09:56:55.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6173668700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0af310 con 0x7f616c144840 2023-10-16T09:56:55.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6173668700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0c6270 con 0x7f616c141920 2023-10-16T09:56:55.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0cdc40 0x7f616c0aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0cdc40 0x7f616c0aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50450/0 (socket says 172.21.15.158:50450) 2023-10-16T09:56:55.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 -- 172.21.15.158:0/1704064143 learned_addr learned my addr 172.21.15.158:0/1704064143 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:55.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6170c03700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c141920 0x7f616c0c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171c05700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c144840 0x7f616c1e1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 -- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c141920 msgr2=0x7f616c0c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c141920 0x7f616c0c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 -- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c144840 msgr2=0x7f616c1e1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c144840 0x7f616c1e1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 -- 172.21.15.158:0/1704064143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c0094a0 con 0x7f616c0cdc40 2023-10-16T09:56:55.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171c05700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c144840 0x7f616c1e1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:55.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.061+0000 7f6171404700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0cdc40 0x7f616c0aedd0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f615c00ebe0 tx=0x7f615c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:55.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.062+0000 7f6161ffb700 1 -- 172.21.15.158:0/1704064143 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c009800 con 0x7f616c0cdc40 2023-10-16T09:56:55.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.062+0000 7f6161ffb700 1 -- 172.21.15.158:0/1704064143 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c00c910 con 0x7f616c0cdc40 2023-10-16T09:56:55.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.062+0000 7f6161ffb700 1 -- 172.21.15.158:0/1704064143 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c00cc80 con 0x7f616c0cdc40 2023-10-16T09:56:55.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.062+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61680094a0 con 0x7f616c0cdc40 2023-10-16T09:56:55.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.062+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c1dc200 con 0x7f616c0cdc40 2023-10-16T09:56:55.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.063+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c0681b0 con 0x7f616c0cdc40 2023-10-16T09:56:55.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.063+0000 7f6161ffb700 1 -- 172.21.15.158:0/1704064143 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f615c013470 con 0x7f616c0cdc40 2023-10-16T09:56:55.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.064+0000 7f6161ffb700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f615006b020 0x7f615006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.064+0000 7f6161ffb700 1 -- 172.21.15.158:0/1704064143 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f615c089ab0 con 0x7f616c0cdc40 2023-10-16T09:56:55.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.064+0000 7f6170c03700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f615006b020 0x7f615006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.064+0000 7f6170c03700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f615006b020 0x7f615006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6168000c00 tx=0x7f6168008030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:55.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.068+0000 7f6161ffb700 1 -- 172.21.15.158:0/1704064143 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f615c054890 con 0x7f616c0cdc40 2023-10-16T09:56:55.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.257+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f616c0b5800 con 0x7f616c0cdc40 2023-10-16T09:56:55.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.258+0000 7f6161ffb700 1 -- 172.21.15.158:0/1704064143 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f615c0170a0 con 0x7f616c0cdc40 2023-10-16T09:56:55.259 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:56:55.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f615006b020 msgr2=0x7f615006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f615006b020 0x7f615006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6168000c00 tx=0x7f6168008030 comp rx=0 tx=0).stop 2023-10-16T09:56:55.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0cdc40 msgr2=0x7f616c0aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0cdc40 0x7f616c0aedd0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f615c00ebe0 tx=0x7f615c00eef0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 shutdown_connections 2023-10-16T09:56:55.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f615006b020 0x7f615006d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f616c144840 0x7f616c1e1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f616c0cdc40 0x7f616c0aedd0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 --2- 172.21.15.158:0/1704064143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f616c141920 0x7f616c0c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 >> 172.21.15.158:0/1704064143 conn(0x7f616c0cc5b0 msgr2=0x7f616c1e61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:55.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 shutdown_connections 2023-10-16T09:56:55.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.260+0000 7f6173668700 1 -- 172.21.15.158:0/1704064143 wait complete. 2023-10-16T09:56:55.273 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T09:56:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.394+0000 7ffbea20c700 1 Processor -- start 2023-10-16T09:56:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.394+0000 7ffbea20c700 1 -- start start 2023-10-16T09:56:55.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbea20c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffbe40cdee0 0x7ffbe415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbea20c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40b6e10 0x7ffbe41521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbea20c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe415c5d0 con 0x7ffbe40ce2c0 2023-10-16T09:56:55.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbea20c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe415d2c0 con 0x7ffbe40cdee0 2023-10-16T09:56:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbea20c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe415c8b0 con 0x7ffbe40b6e10 2023-10-16T09:56:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbe37fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffbe40cdee0 0x7ffbe415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbe3fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ffbe40ce2c0 0x7ffbe40cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:54542/0 (socket says 172.21.15.158:54542) 2023-10-16T09:56:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbe3fff700 1 -- 172.21.15.158:0/3870303346 learned_addr learned my addr 172.21.15.158:0/3870303346 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:56:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.395+0000 7ffbe2ffd700 1 --2- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40b6e10 0x7ffbe41521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe37fe700 1 -- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40b6e10 msgr2=0x7ffbe41521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe37fe700 1 --2- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40b6e10 0x7ffbe41521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe37fe700 1 -- 172.21.15.158:0/3870303346 >> v1:172.21.15.172:6789/0 conn(0x7ffbe40ce2c0 legacy=0x7ffbe40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe27fc700 1 -- 172.21.15.158:0/3870303346 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626506657 0 0) 0x7ffbe415c5d0 con 0x7ffbe40ce2c0 2023-10-16T09:56:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe37fe700 1 -- 172.21.15.158:0/3870303346 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe415cb90 con 0x7ffbe40cdee0 2023-10-16T09:56:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe2ffd700 1 --2- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40b6e10 0x7ffbe41521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:56:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe37fe700 1 --2- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40cdee0 0x7ffbe415bd10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009840 tx=0x7ffbd8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30978564950fb1b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe27fc700 1 -- 172.21.15.158:0/3870303346 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8011830 con 0x7ffbe40cdee0 2023-10-16T09:56:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe27fc700 1 -- 172.21.15.158:0/3870303346 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd8011990 con 0x7ffbe40cdee0 2023-10-16T09:56:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbe27fc700 1 -- 172.21.15.158:0/3870303346 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8011d20 con 0x7ffbe40cdee0 2023-10-16T09:56:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbea20c700 1 -- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40cdee0 msgr2=0x7ffbe415bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.396+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40cdee0 0x7ffbe415bd10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009840 tx=0x7ffbd8009b50 comp rx=0 tx=0).stop 2023-10-16T09:56:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.397+0000 7ffbea20c700 1 -- 172.21.15.158:0/3870303346 shutdown_connections 2023-10-16T09:56:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.397+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40cdee0 0x7ffbe415bd10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.397+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3870303346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40b6e10 0x7ffbe41521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.397+0000 7ffbea20c700 1 -- 172.21.15.158:0/3870303346 >> 172.21.15.158:0/3870303346 conn(0x7ffbe406a8d0 msgr2=0x7ffbe406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.397+0000 7ffbea20c700 1 -- 172.21.15.158:0/3870303346 shutdown_connections 2023-10-16T09:56:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.397+0000 7ffbea20c700 1 -- 172.21.15.158:0/3870303346 wait complete. 2023-10-16T09:56:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.398+0000 7ffbea20c700 1 Processor -- start 2023-10-16T09:56:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.398+0000 7ffbea20c700 1 -- start start 2023-10-16T09:56:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbea20c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40b6e10 0x7ffbe4153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbea20c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cdee0 0x7ffbe41f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbea20c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40ce2c0 0x7ffbe40d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbea20c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe4153bb0 con 0x7ffbe40cdee0 2023-10-16T09:56:55.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbea20c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe41f65f0 con 0x7ffbe40b6e10 2023-10-16T09:56:55.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbea20c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe41f6da0 con 0x7ffbe40ce2c0 2023-10-16T09:56:55.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cdee0 0x7ffbe41f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cdee0 0x7ffbe41f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50466/0 (socket says 172.21.15.158:50466) 2023-10-16T09:56:55.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 -- 172.21.15.158:0/3606873382 learned_addr learned my addr 172.21.15.158:0/3606873382 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:55.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe37fe700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40b6e10 0x7ffbe4153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe3fff700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40ce2c0 0x7ffbe40d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 -- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40ce2c0 msgr2=0x7ffbe40d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40ce2c0 0x7ffbe40d1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 -- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40b6e10 msgr2=0x7ffbe4153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:55.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40b6e10 0x7ffbe4153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:55.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 -- 172.21.15.158:0/3606873382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40094a0 con 0x7ffbe40cdee0 2023-10-16T09:56:55.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe3fff700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40ce2c0 0x7ffbe40d1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:56:55.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbe2ffd700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cdee0 0x7ffbe41f60b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400ebe0 tx=0x7ffbd400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:55.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbcbfff700 1 -- 172.21.15.158:0/3606873382 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4009800 con 0x7ffbe40cdee0 2023-10-16T09:56:55.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbcbfff700 1 -- 172.21.15.158:0/3606873382 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd400c910 con 0x7ffbe40cdee0 2023-10-16T09:56:55.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd80094a0 con 0x7ffbe40cdee0 2023-10-16T09:56:55.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.399+0000 7ffbcbfff700 1 -- 172.21.15.158:0/3606873382 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400cc80 con 0x7ffbe40cdee0 2023-10-16T09:56:55.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.400+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe41f7360 con 0x7ffbe40cdee0 2023-10-16T09:56:55.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.400+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe40681b0 con 0x7ffbe40cdee0 2023-10-16T09:56:55.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.401+0000 7ffbcbfff700 1 -- 172.21.15.158:0/3606873382 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffbd4013470 con 0x7ffbe40cdee0 2023-10-16T09:56:55.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.401+0000 7ffbcbfff700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffbd006b020 0x7ffbd006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:55.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.401+0000 7ffbe37fe700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffbd006b020 0x7ffbd006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:55.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.401+0000 7ffbcbfff700 1 -- 172.21.15.158:0/3606873382 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7ffbd4089b60 con 0x7ffbe40cdee0 2023-10-16T09:56:55.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.402+0000 7ffbe37fe700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffbd006b020 0x7ffbd006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffbd80090b0 tx=0x7ffbd8008030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:55.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.406+0000 7ffbcbfff700 1 -- 172.21.15.158:0/3606873382 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffbd4054830 con 0x7ffbe40cdee0 2023-10-16T09:56:55.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:55.677+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7ffbe40b3bf0 con 0x7ffbe40cdee0 2023-10-16T09:56:56.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.571+0000 7ffbcbfff700 1 -- 172.21.15.158:0/3606873382 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v487) v1 ==== 157+0+0 (secure 0 0 0) 0x7ffbd4057e50 con 0x7ffbe40cdee0 2023-10-16T09:56:56.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffbd006b020 msgr2=0x7ffbd006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:56.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffbd006b020 0x7ffbd006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffbd80090b0 tx=0x7ffbd8008030 comp rx=0 tx=0).stop 2023-10-16T09:56:56.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cdee0 msgr2=0x7ffbe41f60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:56.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cdee0 0x7ffbe41f60b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400ebe0 tx=0x7ffbd400eef0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 shutdown_connections 2023-10-16T09:56:56.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffbd006b020 0x7ffbd006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe40ce2c0 0x7ffbe40d1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe40b6e10 0x7ffbe4153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 --2- 172.21.15.158:0/3606873382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cdee0 0x7ffbe41f60b0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 >> 172.21.15.158:0/3606873382 conn(0x7ffbe406a8d0 msgr2=0x7ffbe415d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:56.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 shutdown_connections 2023-10-16T09:56:56.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.574+0000 7ffbea20c700 1 -- 172.21.15.158:0/3606873382 wait complete. 2023-10-16T09:56:56.590 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T09:56:56.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.718+0000 7ff9e682e700 1 Processor -- start 2023-10-16T09:56:56.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.718+0000 7ff9e682e700 1 -- start start 2023-10-16T09:56:56.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e682e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006d660 0x7ff9e015ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:56.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e682e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff9e006e550 0x7ff9e0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:56.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e682e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff9e006da40 0x7ff9e015a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:56.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e682e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e015a9b0 con 0x7ff9e006da40 2023-10-16T09:56:56.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e682e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e006ead0 con 0x7ff9e006e550 2023-10-16T09:56:56.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e682e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e006a970 con 0x7ff9e006d660 2023-10-16T09:56:56.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff9e006da40 0x7ff9e015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:56.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff9e006da40 0x7ff9e015a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50476/0 (socket says 172.21.15.158:50476) 2023-10-16T09:56:56.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 -- 172.21.15.158:0/3899453427 learned_addr learned my addr 172.21.15.158:0/3899453427 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:56.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 -- 172.21.15.158:0/3899453427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006d660 msgr2=0x7ff9e015ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:56.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 --2- 172.21.15.158:0/3899453427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006d660 0x7ff9e015ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 -- 172.21.15.158:0/3899453427 >> v2:172.21.15.172:3300/0 conn(0x7ff9e006e550 msgr2=0x7ff9e0159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:56.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 --2- 172.21.15.158:0/3899453427 >> v2:172.21.15.172:3300/0 conn(0x7ff9e006e550 0x7ff9e0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 -- 172.21.15.158:0/3899453427 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e006ac50 con 0x7ff9e006da40 2023-10-16T09:56:56.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.719+0000 7ff9e4dcb700 1 --2- 172.21.15.158:0/3899453427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006da40 0x7ff9e015a470 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0072790 tx=0x7ff9d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe75f2f7e970fad6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:56.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9deffd700 1 -- 172.21.15.158:0/3899453427 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800e070 con 0x7ff9e006da40 2023-10-16T09:56:56.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9deffd700 1 -- 172.21.15.158:0/3899453427 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d800b9c0 con 0x7ff9e006da40 2023-10-16T09:56:56.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9deffd700 1 -- 172.21.15.158:0/3899453427 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800bd70 con 0x7ff9e006da40 2023-10-16T09:56:56.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 -- 172.21.15.158:0/3899453427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006da40 msgr2=0x7ff9e015a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:56.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 --2- 172.21.15.158:0/3899453427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006da40 0x7ff9e015a470 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0072790 tx=0x7ff9d800b330 comp rx=0 tx=0).stop 2023-10-16T09:56:56.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 -- 172.21.15.158:0/3899453427 shutdown_connections 2023-10-16T09:56:56.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 --2- 172.21.15.158:0/3899453427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006da40 0x7ff9e015a470 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 --2- 172.21.15.158:0/3899453427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006d660 0x7ff9e015ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 --2- 172.21.15.158:0/3899453427 >> v2:172.21.15.172:3300/0 conn(0x7ff9e006e550 0x7ff9e0159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 -- 172.21.15.158:0/3899453427 >> 172.21.15.158:0/3899453427 conn(0x7ff9e00b1bd0 msgr2=0x7ff9e00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:56.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 -- 172.21.15.158:0/3899453427 shutdown_connections 2023-10-16T09:56:56.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.720+0000 7ff9e682e700 1 -- 172.21.15.158:0/3899453427 wait complete. 2023-10-16T09:56:56.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.721+0000 7ff9e682e700 1 Processor -- start 2023-10-16T09:56:56.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9e682e700 1 -- start start 2023-10-16T09:56:56.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9e682e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006d660 0x7ff9e014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:56.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9e682e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9e006da40 0x7ff9e01f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:56.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9e682e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006e550 0x7ff9e0155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:56.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9e682e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0156700 con 0x7ff9e006d660 2023-10-16T09:56:56.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9e682e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e01f5890 con 0x7ff9e006da40 2023-10-16T09:56:56.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9e682e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0155ea0 con 0x7ff9e006e550 2023-10-16T09:56:56.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006d660 0x7ff9e014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:56.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006d660 0x7ff9e014d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50478/0 (socket says 172.21.15.158:50478) 2023-10-16T09:56:56.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.722+0000 7ff9dffff700 1 -- 172.21.15.158:0/6821120 learned_addr learned my addr 172.21.15.158:0/6821120 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:56.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9e4dcb700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006e550 0x7ff9e0155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:56.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9df7fe700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9e006da40 0x7ff9e01f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:56.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dffff700 1 -- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006e550 msgr2=0x7ff9e0155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:56.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dffff700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006e550 0x7ff9e0155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dffff700 1 -- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9e006da40 msgr2=0x7ff9e01f5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:56.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dffff700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9e006da40 0x7ff9e01f5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dffff700 1 -- 172.21.15.158:0/6821120 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8009b30 con 0x7ff9e006d660 2023-10-16T09:56:56.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9df7fe700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9e006da40 0x7ff9e01f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:56.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9e4dcb700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006e550 0x7ff9e0155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dffff700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006d660 0x7ff9e014d260 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400b530 tx=0x7ff9d400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dcff9700 1 -- 172.21.15.158:0/6821120 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4011830 con 0x7ff9e006d660 2023-10-16T09:56:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dcff9700 1 -- 172.21.15.158:0/6821120 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d40119b0 con 0x7ff9e006d660 2023-10-16T09:56:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9dcff9700 1 -- 172.21.15.158:0/6821120 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4011d60 con 0x7ff9e006d660 2023-10-16T09:56:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e0156140 con 0x7ff9e006d660 2023-10-16T09:56:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.723+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e01f4b50 con 0x7ff9e006d660 2023-10-16T09:56:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.724+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e01f49b0 con 0x7ff9e006d660 2023-10-16T09:56:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.724+0000 7ff9dcff9700 1 -- 172.21.15.158:0/6821120 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9d401a400 con 0x7ff9e006d660 2023-10-16T09:56:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.725+0000 7ff9dcff9700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff9cc06b020 0x7ff9cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.725+0000 7ff9dcff9700 1 -- 172.21.15.158:0/6821120 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7ff9d4089ac0 con 0x7ff9e006d660 2023-10-16T09:56:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.725+0000 7ff9df7fe700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff9cc06b020 0x7ff9cc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.725+0000 7ff9df7fe700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff9cc06b020 0x7ff9cc06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000a8f0 tx=0x7ff9d0008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.729+0000 7ff9dcff9700 1 -- 172.21.15.158:0/6821120 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9e01f49b0 con 0x7ff9e006d660 2023-10-16T09:56:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.922+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff9e01f49b0 con 0x7ff9e006d660 2023-10-16T09:56:56.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.925+0000 7ff9dcff9700 1 -- 172.21.15.158:0/6821120 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff9e01f49b0 con 0x7ff9e006d660 2023-10-16T09:56:56.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff9cc06b020 msgr2=0x7ff9cc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:56.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff9cc06b020 0x7ff9cc06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000a8f0 tx=0x7ff9d0008040 comp rx=0 tx=0).stop 2023-10-16T09:56:56.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006d660 msgr2=0x7ff9e014d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:56.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006d660 0x7ff9e014d260 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400b530 tx=0x7ff9d400b8f0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 shutdown_connections 2023-10-16T09:56:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff9cc06b020 0x7ff9cc06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9e006e550 0x7ff9e0155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9e006d660 0x7ff9e014d260 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 --2- 172.21.15.158:0/6821120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9e006da40 0x7ff9e01f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 >> 172.21.15.158:0/6821120 conn(0x7ff9e00b1bd0 msgr2=0x7ff9e00b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:56.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 shutdown_connections 2023-10-16T09:56:56.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:56.928+0000 7ff9e682e700 1 -- 172.21.15.158:0/6821120 wait complete. 2023-10-16T09:56:56.941 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T09:56:56.942 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:56:56.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:56:57.032 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T09:56:57.032 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:56:57.033 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:57.050 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:56:57.051 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:56:57.051 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:57.071 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:56:57.071 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:56:57.072 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:57.107 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:56:57.108 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:56:57.108 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:56:57.130 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:56:57.130 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T09:56:57.131 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:56:57.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.267+0000 7ff2694f7700 1 Processor -- start 2023-10-16T09:56:57.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.267+0000 7ff2694f7700 1 -- start start 2023-10-16T09:56:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.267+0000 7ff2694f7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff26415d7b0 0x7ff26406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.267+0000 7ff2694f7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2640d0fb0 con 0x7ff26415db90 2023-10-16T09:56:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.267+0000 7ff2694f7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26415e100 con 0x7ff2640cddb0 2023-10-16T09:56:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.267+0000 7ff2694f7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26415e3e0 con 0x7ff26415d7b0 2023-10-16T09:56:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff262ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff26415d7b0 0x7ff26406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff262ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff26415d7b0 0x7ff26406b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50486/0 (socket says 172.21.15.158:50486) 2023-10-16T09:56:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff262ffd700 1 -- 172.21.15.158:0/3746450432 learned_addr learned my addr 172.21.15.158:0/3746450432 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff2627fc700 1 --1- 172.21.15.158:0/3746450432 >> v1:172.21.15.158:6789/0 conn(0x7ff26415db90 0x7ff2640b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50664/0 (socket says 172.21.15.158:50664) 2023-10-16T09:56:57.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff262ffd700 1 -- 172.21.15.158:0/3746450432 >> v1:172.21.15.172:6789/0 conn(0x7ff2640cddb0 legacy=0x7ff2640ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:57.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff262ffd700 1 -- 172.21.15.158:0/3746450432 >> v1:172.21.15.158:6789/0 conn(0x7ff26415db90 legacy=0x7ff2640b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff262ffd700 1 -- 172.21.15.158:0/3746450432 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26415e6c0 con 0x7ff26415d7b0 2023-10-16T09:56:57.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff262ffd700 1 --2- 172.21.15.158:0/3746450432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff26415d7b0 0x7ff26406b9e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff2540098f0 tx=0x7ff254009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9a49654c2b9caa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff261ffb700 1 -- 172.21.15.158:0/3746450432 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254004230 con 0x7ff26415d7b0 2023-10-16T09:56:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff261ffb700 1 -- 172.21.15.158:0/3746450432 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff254004390 con 0x7ff26415d7b0 2023-10-16T09:56:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.268+0000 7ff261ffb700 1 -- 172.21.15.158:0/3746450432 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2540036a0 con 0x7ff26415d7b0 2023-10-16T09:56:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.269+0000 7ff2694f7700 1 -- 172.21.15.158:0/3746450432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff26415d7b0 msgr2=0x7ff26406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.269+0000 7ff2694f7700 1 --2- 172.21.15.158:0/3746450432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff26415d7b0 0x7ff26406b9e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff2540098f0 tx=0x7ff254009c00 comp rx=0 tx=0).stop 2023-10-16T09:56:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.269+0000 7ff2694f7700 1 -- 172.21.15.158:0/3746450432 shutdown_connections 2023-10-16T09:56:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.269+0000 7ff2694f7700 1 --2- 172.21.15.158:0/3746450432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff26415d7b0 0x7ff26406b9e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.269+0000 7ff2694f7700 1 -- 172.21.15.158:0/3746450432 >> 172.21.15.158:0/3746450432 conn(0x7ff264071220 msgr2=0x7ff26406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.269+0000 7ff2694f7700 1 -- 172.21.15.158:0/3746450432 shutdown_connections 2023-10-16T09:56:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.269+0000 7ff2694f7700 1 -- 172.21.15.158:0/3746450432 wait complete. 2023-10-16T09:56:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.271+0000 7ff2694f7700 1 Processor -- start 2023-10-16T09:56:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.271+0000 7ff2694f7700 1 -- start start 2023-10-16T09:56:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.271+0000 7ff2694f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2640cddb0 0x7ff264151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.271+0000 7ff2694f7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26415d7b0 0x7ff264152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.271+0000 7ff2694f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26415db90 0x7ff264153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.271+0000 7ff2694f7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff264158a20 con 0x7ff2640cddb0 2023-10-16T09:56:57.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2694f7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640b5710 con 0x7ff26415d7b0 2023-10-16T09:56:57.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2694f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640c7990 con 0x7ff26415db90 2023-10-16T09:56:57.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2640cddb0 0x7ff264151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2640cddb0 0x7ff264151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50492/0 (socket says 172.21.15.158:50492) 2023-10-16T09:56:57.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 -- 172.21.15.158:0/1485666894 learned_addr learned my addr 172.21.15.158:0/1485666894 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:57.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2627fc700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26415d7b0 0x7ff264152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2637fe700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26415db90 0x7ff264153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 -- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26415db90 msgr2=0x7ff264153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26415db90 0x7ff264153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 -- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26415d7b0 msgr2=0x7ff264152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26415d7b0 0x7ff264152dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 -- 172.21.15.158:0/1485666894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff254009550 con 0x7ff2640cddb0 2023-10-16T09:56:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2627fc700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26415d7b0 0x7ff264152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2637fe700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26415db90 0x7ff264153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:56:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff262ffd700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2640cddb0 0x7ff264151820 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff2640ceab0 tx=0x7ff254004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff24b7fe700 1 -- 172.21.15.158:0/1485666894 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254003d80 con 0x7ff2640cddb0 2023-10-16T09:56:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff24b7fe700 1 -- 172.21.15.158:0/1485666894 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25401f070 con 0x7ff2640cddb0 2023-10-16T09:56:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff24b7fe700 1 -- 172.21.15.158:0/1485666894 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400b8e0 con 0x7ff2640cddb0 2023-10-16T09:56:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640b59b0 con 0x7ff2640cddb0 2023-10-16T09:56:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.272+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2640c7c70 con 0x7ff2640cddb0 2023-10-16T09:56:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.273+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2640b5ed0 con 0x7ff2640cddb0 2023-10-16T09:56:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.274+0000 7ff24b7fe700 1 -- 172.21.15.158:0/1485666894 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff254021030 con 0x7ff2640cddb0 2023-10-16T09:56:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.274+0000 7ff24b7fe700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff25006b020 0x7ff25006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.274+0000 7ff24b7fe700 1 -- 172.21.15.158:0/1485666894 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7ff25408b0d0 con 0x7ff2640cddb0 2023-10-16T09:56:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.274+0000 7ff2627fc700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff25006b020 0x7ff25006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.275+0000 7ff2627fc700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff25006b020 0x7ff25006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff264155910 tx=0x7ff258009040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:57.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.278+0000 7ff24b7fe700 1 -- 172.21.15.158:0/1485666894 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff254055eb0 con 0x7ff2640cddb0 2023-10-16T09:56:57.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.516+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2640b5ed0 con 0x7ff2640cddb0 2023-10-16T09:56:57.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.517+0000 7ff24b7fe700 1 -- 172.21.15.158:0/1485666894 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3711 (secure 0 0 0) 0x7ff2540594d0 con 0x7ff2640cddb0 2023-10-16T09:56:57.519 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:57.519 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4283661376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4283661376},{"type":"v1","addr":"172.21.15.158:6835","nonce":4283661376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/3069959147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":3069959147},{"type":"v1","addr":"172.21.15.158:6837","nonce":3069959147}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1583191131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1583191131},{"type":"v1","addr":"172.21.15.172:6835","nonce":1583191131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3008474018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3008474018},{"type":"v1","addr":"172.21.15.172:6834","nonce":3008474018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2023-10-16T09:56:57.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff25006b020 msgr2=0x7ff25006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff25006b020 0x7ff25006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff264155910 tx=0x7ff258009040 comp rx=0 tx=0).stop 2023-10-16T09:56:57.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2640cddb0 msgr2=0x7ff264151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2640cddb0 0x7ff264151820 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff2640ceab0 tx=0x7ff254004750 comp rx=0 tx=0).stop 2023-10-16T09:56:57.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 shutdown_connections 2023-10-16T09:56:57.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ff25006b020 0x7ff25006d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26415db90 0x7ff264153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2640cddb0 0x7ff264151820 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 --2- 172.21.15.158:0/1485666894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26415d7b0 0x7ff264152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 >> 172.21.15.158:0/1485666894 conn(0x7ff264071220 msgr2=0x7ff2640b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:57.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 shutdown_connections 2023-10-16T09:56:57.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.519+0000 7ff2694f7700 1 -- 172.21.15.158:0/1485666894 wait complete. 2023-10-16T09:56:57.524 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 19 2023-10-16T09:56:57.533 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T09:56:57.533 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T09:56:57.534 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T09:56:57.534 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:56:57.534 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T09:56:57.535 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T09:56:57.535 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T09:56:57.535 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:56:57.536 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T09:56:57.536 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T09:56:57.536 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T09:56:57.536 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:56:57.537 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T09:56:57.538 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T09:56:57.538 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T09:56:57.538 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:56:57.544 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T09:56:57.545 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T09:56:57.546 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T09:56:57.546 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T09:56:57.547 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T09:56:57.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T09:56:57.549 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T09:56:57.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T09:56:57.589 INFO:tasks.ceph.mds.a:Started 2023-10-16T09:56:57.590 DEBUG:teuthology.parallel:result is None 2023-10-16T09:56:57.591 INFO:tasks.ceph.mds.b:Started 2023-10-16T09:56:57.592 INFO:tasks.ceph.mds.c:Started 2023-10-16T09:56:57.592 DEBUG:teuthology.parallel:result is None 2023-10-16T09:56:57.593 DEBUG:teuthology.parallel:result is None 2023-10-16T09:56:57.593 INFO:tasks.ceph.mds.d:Started 2023-10-16T09:56:57.594 DEBUG:teuthology.parallel:result is None 2023-10-16T09:56:57.594 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T09:56:57.667 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T09:56:57.669 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T09:56:57.678 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:56:57.678 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T09:56:57.680 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:56:57.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:56:57.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.722+0000 7f7db77c1700 1 Processor -- start 2023-10-16T09:56:57.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.722+0000 7f7db77c1700 1 -- start start 2023-10-16T09:56:57.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db77c1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db00b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db77c1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7db00b4950 0x7f7db00cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db77c1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db015f990 con 0x7f7db015eb10 2023-10-16T09:56:57.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db77c1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db00cdef0 con 0x7f7db00b4950 2023-10-16T09:56:57.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db77c1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cd070 con 0x7f7db00d0ae0 2023-10-16T09:56:57.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db5d5e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7db00b4950 0x7f7db00cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db4d5c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db00b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db5d5e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7db00b4950 0x7f7db00cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:51708/0 (socket says 172.21.15.158:51708) 2023-10-16T09:56:57.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.723+0000 7f7db5d5e700 1 -- 172.21.15.158:0/1279863955 learned_addr learned my addr 172.21.15.158:0/1279863955 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:57.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7db5d5e700 1 -- 172.21.15.158:0/1279863955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 msgr2=0x7f7db00b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:57.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7db5d5e700 1 --2- 172.21.15.158:0/1279863955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db00b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7db5d5e700 1 -- 172.21.15.158:0/1279863955 >> v1:172.21.15.172:6789/0 conn(0x7f7db015eb10 legacy=0x7f7db015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7db5d5e700 1 -- 172.21.15.158:0/1279863955 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db015fc70 con 0x7f7db00b4950 2023-10-16T09:56:57.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7db5d5e700 1 --2- 172.21.15.158:0/1279863955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00cfad0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7da80098f0 tx=0x7f7da8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c5b9fb475637b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:57.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7da7fff700 1 -- 172.21.15.158:0/1279863955 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8011830 con 0x7f7db00b4950 2023-10-16T09:56:57.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7da7fff700 1 -- 172.21.15.158:0/1279863955 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da8011990 con 0x7f7db00b4950 2023-10-16T09:56:57.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7da7fff700 1 -- 172.21.15.158:0/1279863955 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8011d20 con 0x7f7db00b4950 2023-10-16T09:56:57.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7db77c1700 1 -- 172.21.15.158:0/1279863955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 msgr2=0x7f7db00cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.724+0000 7f7db77c1700 1 --2- 172.21.15.158:0/1279863955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00cfad0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7da80098f0 tx=0x7f7da8009c00 comp rx=0 tx=0).stop 2023-10-16T09:56:57.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.725+0000 7f7db77c1700 1 -- 172.21.15.158:0/1279863955 shutdown_connections 2023-10-16T09:56:57.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.725+0000 7f7db77c1700 1 --2- 172.21.15.158:0/1279863955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00cfad0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.725+0000 7f7db77c1700 1 --2- 172.21.15.158:0/1279863955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db00b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.725+0000 7f7db77c1700 1 -- 172.21.15.158:0/1279863955 >> 172.21.15.158:0/1279863955 conn(0x7f7db00ce040 msgr2=0x7f7db00ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:57.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.725+0000 7f7db77c1700 1 -- 172.21.15.158:0/1279863955 shutdown_connections 2023-10-16T09:56:57.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.725+0000 7f7db77c1700 1 -- 172.21.15.158:0/1279863955 wait complete. 2023-10-16T09:56:57.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.726+0000 7f7db77c1700 1 Processor -- start 2023-10-16T09:56:57.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.726+0000 7f7db77c1700 1 -- start start 2023-10-16T09:56:57.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db77c1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db77c1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db01519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db77c1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db015eb10 0x7f7db0151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db77c1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00c81b0 con 0x7f7db015eb10 2023-10-16T09:56:57.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db77c1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cc2e0 con 0x7f7db00b4950 2023-10-16T09:56:57.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db77c1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cbd20 con 0x7f7db00d0ae0 2023-10-16T09:56:57.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:51710/0 (socket says 172.21.15.158:51710) 2023-10-16T09:56:57.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 -- 172.21.15.158:0/2665221538 learned_addr learned my addr 172.21.15.158:0/2665221538 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:57.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db5d5e700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db015eb10 0x7f7db0151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db4d5c700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db01519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 -- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 msgr2=0x7f7db01519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db01519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 -- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db015eb10 msgr2=0x7f7db0151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:57.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db015eb10 0x7f7db0151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:57.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 -- 172.21.15.158:0/2665221538 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da8009550 con 0x7f7db00b4950 2023-10-16T09:56:57.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db555d700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00c7c70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7dac001ed0 tx=0x7f7dac0021e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:57.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7da5ffb700 1 -- 172.21.15.158:0/2665221538 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac003c00 con 0x7f7db00b4950 2023-10-16T09:56:57.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7da5ffb700 1 -- 172.21.15.158:0/2665221538 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac003d60 con 0x7f7db00b4950 2023-10-16T09:56:57.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db00cc000 con 0x7f7db00b4950 2023-10-16T09:56:57.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.727+0000 7f7da5ffb700 1 -- 172.21.15.158:0/2665221538 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac00ab40 con 0x7f7db00b4950 2023-10-16T09:56:57.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.728+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db014d860 con 0x7f7db00b4950 2023-10-16T09:56:57.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.729+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db014d3e0 con 0x7f7db00b4950 2023-10-16T09:56:57.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.729+0000 7f7da5ffb700 1 -- 172.21.15.158:0/2665221538 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7dac00aca0 con 0x7f7db00b4950 2023-10-16T09:56:57.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.729+0000 7f7da5ffb700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7d9406b020 0x7f7d9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:57.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.729+0000 7f7da5ffb700 1 -- 172.21.15.158:0/2665221538 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f7dac0899d0 con 0x7f7db00b4950 2023-10-16T09:56:57.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.729+0000 7f7db4d5c700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7d9406b020 0x7f7d9406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:57.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.730+0000 7f7db4d5c700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7d9406b020 0x7f7d9406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7db00b4780 tx=0x7f7da0009210 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:57.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.734+0000 7f7da5ffb700 1 -- 172.21.15.158:0/2665221538 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7db014d3e0 con 0x7f7db00b4950 2023-10-16T09:56:57.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:57.943+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7db014d3e0 con 0x7f7db00b4950 2023-10-16T09:56:59.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.544+0000 7f7da5ffb700 1 -- 172.21.15.158:0/2665221538 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7db014d3e0 con 0x7f7db00b4950 2023-10-16T09:56:59.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7d9406b020 msgr2=0x7f7d9406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:59.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7d9406b020 0x7f7d9406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7db00b4780 tx=0x7f7da0009210 comp rx=0 tx=0).stop 2023-10-16T09:56:59.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 msgr2=0x7f7db00c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:59.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00c7c70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7dac001ed0 tx=0x7f7dac0021e0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 shutdown_connections 2023-10-16T09:56:59.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7d9406b020 0x7f7d9406d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db015eb10 0x7f7db0151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db00b4950 0x7f7db00c7c70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 --2- 172.21.15.158:0/2665221538 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00d0ae0 0x7f7db01519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 >> 172.21.15.158:0/2665221538 conn(0x7f7db00ce040 msgr2=0x7f7db00b0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:59.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 shutdown_connections 2023-10-16T09:56:59.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.547+0000 7f7db77c1700 1 -- 172.21.15.158:0/2665221538 wait complete. 2023-10-16T09:56:59.551 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T09:56:59.563 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T09:56:59.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.688+0000 7f83c5eb3700 1 Processor -- start 2023-10-16T09:56:59.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.688+0000 7f83c5eb3700 1 -- start start 2023-10-16T09:56:59.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83c5eb3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c00cc5d0 0x7f83c006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:59.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83c5eb3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83c015dfb0 0x7f83c00cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:59.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83c5eb3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c006e770 con 0x7f83c006f340 2023-10-16T09:56:59.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83c5eb3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c00cf270 con 0x7f83c015dfb0 2023-10-16T09:56:59.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83c5eb3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c015e4e0 con 0x7f83c00cc5d0 2023-10-16T09:56:59.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83bf7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c00cc5d0 0x7f83c006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:59.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83beffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83c015dfb0 0x7f83c00cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:59.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83bf7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c00cc5d0 0x7f83c006e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:33144/0 (socket says 172.21.15.158:33144) 2023-10-16T09:56:59.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.689+0000 7f83bf7fe700 1 -- 172.21.15.158:0/419832362 learned_addr learned my addr 172.21.15.158:0/419832362 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:59.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83bf7fe700 1 -- 172.21.15.158:0/419832362 >> v2:172.21.15.172:3300/0 conn(0x7f83c015dfb0 msgr2=0x7f83c00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:56:59.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83bf7fe700 1 --2- 172.21.15.158:0/419832362 >> v2:172.21.15.172:3300/0 conn(0x7f83c015dfb0 0x7f83c00cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83bf7fe700 1 -- 172.21.15.158:0/419832362 >> v1:172.21.15.172:6789/0 conn(0x7f83c006f340 legacy=0x7f83c0071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:59.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83bf7fe700 1 -- 172.21.15.158:0/419832362 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c015e7c0 con 0x7f83c00cc5d0 2023-10-16T09:56:59.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83bf7fe700 1 --2- 172.21.15.158:0/419832362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c00cc5d0 0x7f83c006e230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f83a80098f0 tx=0x7f83a8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e56e9f1ced6992dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:59.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83be7fc700 1 -- 172.21.15.158:0/419832362 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800f8a0 con 0x7f83c00cc5d0 2023-10-16T09:56:59.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83be7fc700 1 -- 172.21.15.158:0/419832362 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a800fa00 con 0x7f83c00cc5d0 2023-10-16T09:56:59.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.690+0000 7f83be7fc700 1 -- 172.21.15.158:0/419832362 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800fd90 con 0x7f83c00cc5d0 2023-10-16T09:56:59.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 -- 172.21.15.158:0/419832362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c00cc5d0 msgr2=0x7f83c006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:59.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/419832362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c00cc5d0 0x7f83c006e230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f83a80098f0 tx=0x7f83a8009c00 comp rx=0 tx=0).stop 2023-10-16T09:56:59.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 -- 172.21.15.158:0/419832362 shutdown_connections 2023-10-16T09:56:59.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/419832362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c00cc5d0 0x7f83c006e230 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/419832362 >> v2:172.21.15.172:3300/0 conn(0x7f83c015dfb0 0x7f83c00cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 -- 172.21.15.158:0/419832362 >> 172.21.15.158:0/419832362 conn(0x7f83c00d02b0 msgr2=0x7f83c00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:56:59.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 -- 172.21.15.158:0/419832362 shutdown_connections 2023-10-16T09:56:59.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.691+0000 7f83c5eb3700 1 -- 172.21.15.158:0/419832362 wait complete. 2023-10-16T09:56:59.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.692+0000 7f83c5eb3700 1 Processor -- start 2023-10-16T09:56:59.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.692+0000 7f83c5eb3700 1 -- start start 2023-10-16T09:56:59.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83c5eb3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c006f340 0x7f83c0154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:59.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83c5eb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83c00cc5d0 0x7f83c00ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:59.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83c5eb3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c015dfb0 0x7f83c00cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:59.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83c5eb3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00c8510 con 0x7f83c00cc5d0 2023-10-16T09:56:59.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83c5eb3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00c74b0 con 0x7f83c006f340 2023-10-16T09:56:59.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83c5eb3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c0154760 con 0x7f83c015dfb0 2023-10-16T09:56:59.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83c00cc5d0 0x7f83c00ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:59.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83c00cc5d0 0x7f83c00ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50548/0 (socket says 172.21.15.158:50548) 2023-10-16T09:56:59.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 -- 172.21.15.158:0/3389668327 learned_addr learned my addr 172.21.15.158:0/3389668327 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:56:59.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83bffff700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c015dfb0 0x7f83c00cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:59.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83bf7fe700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c006f340 0x7f83c0154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:59.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 -- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c015dfb0 msgr2=0x7f83c00cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:59.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c015dfb0 0x7f83c00cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 -- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c006f340 msgr2=0x7f83c0154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:56:59.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c006f340 0x7f83c0154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:56:59.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 -- 172.21.15.158:0/3389668327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8009550 con 0x7f83c00cc5d0 2023-10-16T09:56:59.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.693+0000 7f83beffd700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83c00cc5d0 0x7f83c00ce6a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f83b000e980 tx=0x7f83b000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:59.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.694+0000 7f83b7fff700 1 -- 172.21.15.158:0/3389668327 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0009710 con 0x7f83c00cc5d0 2023-10-16T09:56:59.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.694+0000 7f83b7fff700 1 -- 172.21.15.158:0/3389668327 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b000c850 con 0x7f83c00cc5d0 2023-10-16T09:56:59.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.694+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c00c7750 con 0x7f83c00cc5d0 2023-10-16T09:56:59.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.694+0000 7f83b7fff700 1 -- 172.21.15.158:0/3389668327 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b000cc00 con 0x7f83c00cc5d0 2023-10-16T09:56:59.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.694+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c00c7d10 con 0x7f83c00cc5d0 2023-10-16T09:56:59.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.695+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c00c8360 con 0x7f83c00cc5d0 2023-10-16T09:56:59.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.695+0000 7f83b7fff700 1 -- 172.21.15.158:0/3389668327 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f83b0013470 con 0x7f83c00cc5d0 2023-10-16T09:56:59.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.695+0000 7f83b7fff700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f83a006b020 0x7f83a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:56:59.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.696+0000 7f83b7fff700 1 -- 172.21.15.158:0/3389668327 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83b00897d0 con 0x7f83c00cc5d0 2023-10-16T09:56:59.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.696+0000 7f83bf7fe700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f83a006b020 0x7f83a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:56:59.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.696+0000 7f83bf7fe700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f83a006b020 0x7f83a006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009c40 tx=0x7f83a8008030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:56:59.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.700+0000 7f83b7fff700 1 -- 172.21.15.158:0/3389668327 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83b0054940 con 0x7f83c00cc5d0 2023-10-16T09:56:59.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.972+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f83c00681b0 con 0x7f83c00cc5d0 2023-10-16T09:56:59.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.973+0000 7f83b7fff700 1 -- 172.21.15.158:0/3389668327 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f83b0021090 con 0x7f83c00cc5d0 2023-10-16T09:56:59.975 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:56:59.975 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:56:59.975 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T09:56:59.975 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.975 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T09:56:59.976 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T09:56:59.976 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.976 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:56:59.976 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:56:59.977 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:56:59.977 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.977 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.977 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.977 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.978 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T09:56:59.978 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T09:56:59.978 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.978 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:56:59.978 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:56:59.979 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:56:59.979 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.979 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.979 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.979 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.980 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T09:56:59.980 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T09:56:59.980 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.980 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:56:59.980 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:56:59.981 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:56:59.981 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.981 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.981 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.981 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.982 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T09:56:59.982 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T09:56:59.982 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.982 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:56:59.982 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:56:59.983 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:56:59.983 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.983 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.983 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.984 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T09:56:59.984 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T09:56:59.984 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.984 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.984 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T09:56:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T09:56:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T09:56:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T09:56:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T09:56:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T09:56:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T09:56:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T09:56:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T09:56:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T09:56:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T09:56:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T09:56:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T09:56:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T09:56:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:56:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:56:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:56:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:56:59.999 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:56:59.999 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:56:59.999 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:56:59.999 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T09:56:59.999 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T09:57:00.000 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:57:00.000 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:57:00.000 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T09:57:00.000 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T09:57:00.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.001 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:57:00.001 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:00.001 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T09:57:00.001 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T09:57:00.001 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:57:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:57:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:57:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:57:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:57:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:57:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T09:57:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T09:57:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:57:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:57:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:57:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:57:00.004 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:57:00.005 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.005 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:57:00.005 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:00.005 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T09:57:00.006 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T09:57:00.006 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:57:00.006 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:57:00.006 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:57:00.007 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:57:00.007 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:57:00.007 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.007 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:57:00.007 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:00.008 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T09:57:00.008 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T09:57:00.008 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:57:00.008 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:57:00.008 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:57:00.009 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:57:00.009 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:57:00.009 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.009 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:57:00.010 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:00.010 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T09:57:00.010 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T09:57:00.010 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:57:00.010 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:57:00.011 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:57:00.011 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:57:00.011 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.011 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:57:00.011 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:00.012 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T09:57:00.012 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T09:57:00.012 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:57:00.012 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:57:00.013 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:57:00.013 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:57:00.013 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.013 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:00.013 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T09:57:00.014 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:57:00.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f83a006b020 msgr2=0x7f83a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f83a006b020 0x7f83a006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009c40 tx=0x7f83a8008030 comp rx=0 tx=0).stop 2023-10-16T09:57:00.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83c00cc5d0 msgr2=0x7f83c00ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83c00cc5d0 0x7f83c00ce6a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f83b000e980 tx=0x7f83b000ed40 comp rx=0 tx=0).stop 2023-10-16T09:57:00.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 shutdown_connections 2023-10-16T09:57:00.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f83a006b020 0x7f83a006d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83c015dfb0 0x7f83c00cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c006f340 0x7f83c0154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 --2- 172.21.15.158:0/3389668327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83c00cc5d0 0x7f83c00ce6a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 >> 172.21.15.158:0/3389668327 conn(0x7f83c00d02b0 msgr2=0x7f83c00afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:00.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 shutdown_connections 2023-10-16T09:57:00.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:56:59.975+0000 7f83c5eb3700 1 -- 172.21.15.158:0/3389668327 wait complete. 2023-10-16T09:57:00.018 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:57:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 Processor -- start 2023-10-16T09:57:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 -- start start 2023-10-16T09:57:00.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f345006e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f34501612a0 0x7f345006acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f34500718d0 0x7f345015e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345015dcb0 con 0x7f34500718d0 2023-10-16T09:57:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345006b1f0 con 0x7f34501612a0 2023-10-16T09:57:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.139+0000 7f345590e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3450161740 con 0x7f345006a8d0 2023-10-16T09:57:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f34500718d0 0x7f345015e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f34500718d0 0x7f345015e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50558/0 (socket says 172.21.15.158:50558) 2023-10-16T09:57:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 -- 172.21.15.158:0/3414582080 learned_addr learned my addr 172.21.15.158:0/3414582080 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344e7fc700 1 --2- 172.21.15.158:0/3414582080 >> v2:172.21.15.172:3300/0 conn(0x7f34501612a0 0x7f345006acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344effd700 1 --2- 172.21.15.158:0/3414582080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f345006e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 -- 172.21.15.158:0/3414582080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 msgr2=0x7f345006e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 --2- 172.21.15.158:0/3414582080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f345006e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 -- 172.21.15.158:0/3414582080 >> v2:172.21.15.172:3300/0 conn(0x7f34501612a0 msgr2=0x7f345006acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 --2- 172.21.15.158:0/3414582080 >> v2:172.21.15.172:3300/0 conn(0x7f34501612a0 0x7f345006acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 -- 172.21.15.158:0/3414582080 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3450161a20 con 0x7f34500718d0 2023-10-16T09:57:00.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344f7fe700 1 --2- 172.21.15.158:0/3414582080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f345015e140 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3450073720 tx=0x7f344800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440a161ed43d4cb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344dffb700 1 -- 172.21.15.158:0/3414582080 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800e070 con 0x7f34500718d0 2023-10-16T09:57:00.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344dffb700 1 -- 172.21.15.158:0/3414582080 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344800b9c0 con 0x7f34500718d0 2023-10-16T09:57:00.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.140+0000 7f344dffb700 1 -- 172.21.15.158:0/3414582080 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800bd70 con 0x7f34500718d0 2023-10-16T09:57:00.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 -- 172.21.15.158:0/3414582080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 msgr2=0x7f345015e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 --2- 172.21.15.158:0/3414582080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f345015e140 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3450073720 tx=0x7f344800b330 comp rx=0 tx=0).stop 2023-10-16T09:57:00.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 -- 172.21.15.158:0/3414582080 shutdown_connections 2023-10-16T09:57:00.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 --2- 172.21.15.158:0/3414582080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f345015e140 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 --2- 172.21.15.158:0/3414582080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f345006e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 --2- 172.21.15.158:0/3414582080 >> v2:172.21.15.172:3300/0 conn(0x7f34501612a0 0x7f345006acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 -- 172.21.15.158:0/3414582080 >> 172.21.15.158:0/3414582080 conn(0x7f34500d02a0 msgr2=0x7f34500d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:00.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 -- 172.21.15.158:0/3414582080 shutdown_connections 2023-10-16T09:57:00.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.141+0000 7f345590e700 1 -- 172.21.15.158:0/3414582080 wait complete. 2023-10-16T09:57:00.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.142+0000 7f345590e700 1 Processor -- start 2023-10-16T09:57:00.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.142+0000 7f345590e700 1 -- start start 2023-10-16T09:57:00.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f345590e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f3450160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f345590e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f34501553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f345590e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34501612a0 0x7f34501f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f345590e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345015e040 con 0x7f34500718d0 2023-10-16T09:57:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f345590e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345015e590 con 0x7f34501612a0 2023-10-16T09:57:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f345590e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34500b5860 con 0x7f345006a8d0 2023-10-16T09:57:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f34501553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f34501553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50562/0 (socket says 172.21.15.158:50562) 2023-10-16T09:57:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 -- 172.21.15.158:0/876448602 learned_addr learned my addr 172.21.15.158:0/876448602 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344f7fe700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34501612a0 0x7f34501f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344effd700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f3450160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 -- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 msgr2=0x7f3450160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f3450160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 -- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34501612a0 msgr2=0x7f34501f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34501612a0 0x7f34501f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 -- 172.21.15.158:0/876448602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448009b30 con 0x7f34500718d0 2023-10-16T09:57:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f344e7fc700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f34501553d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f344400c7f0 tx=0x7f344400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f3437fff700 1 -- 172.21.15.158:0/876448602 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444004220 con 0x7f34500718d0 2023-10-16T09:57:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.143+0000 7f3437fff700 1 -- 172.21.15.158:0/876448602 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444004380 con 0x7f34500718d0 2023-10-16T09:57:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.144+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34500b5b40 con 0x7f34500718d0 2023-10-16T09:57:00.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.144+0000 7f3437fff700 1 -- 172.21.15.158:0/876448602 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34440036a0 con 0x7f34500718d0 2023-10-16T09:57:00.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.144+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34500b4150 con 0x7f34500718d0 2023-10-16T09:57:00.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.144+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3450161ce0 con 0x7f34500718d0 2023-10-16T09:57:00.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.145+0000 7f3437fff700 1 -- 172.21.15.158:0/876448602 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3444003800 con 0x7f34500718d0 2023-10-16T09:57:00.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.145+0000 7f3437fff700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f343c06b020 0x7f343c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.145+0000 7f3437fff700 1 -- 172.21.15.158:0/876448602 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f34440892c0 con 0x7f34500718d0 2023-10-16T09:57:00.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.145+0000 7f344effd700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f343c06b020 0x7f343c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.146+0000 7f344effd700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f343c06b020 0x7f343c06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3450160a60 tx=0x7f3440008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.150+0000 7f3437fff700 1 -- 172.21.15.158:0/876448602 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3450161ce0 con 0x7f34500718d0 2023-10-16T09:57:00.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.342+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3450161ce0 con 0x7f34500718d0 2023-10-16T09:57:00.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.343+0000 7f3437fff700 1 -- 172.21.15.158:0/876448602 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13675 (secure 0 0 0) 0x7f3450161ce0 con 0x7f34500718d0 2023-10-16T09:57:00.344 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:00.344 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":63,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:56:59.537254+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:57:00.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f343c06b020 msgr2=0x7f343c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f343c06b020 0x7f343c06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3450160a60 tx=0x7f3440008040 comp rx=0 tx=0).stop 2023-10-16T09:57:00.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 msgr2=0x7f34501553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f34501553d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f344400c7f0 tx=0x7f344400cb00 comp rx=0 tx=0).stop 2023-10-16T09:57:00.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 shutdown_connections 2023-10-16T09:57:00.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f343c06b020 0x7f343c06d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34501612a0 0x7f34501f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f345006a8d0 0x7f3450160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 --2- 172.21.15.158:0/876448602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34500718d0 0x7f34501553d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 >> 172.21.15.158:0/876448602 conn(0x7f34500d02a0 msgr2=0x7f34500cb610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:00.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 shutdown_connections 2023-10-16T09:57:00.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.345+0000 7f345590e700 1 -- 172.21.15.158:0/876448602 wait complete. 2023-10-16T09:57:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:57:00.359 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:57:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.482+0000 7fc51712f700 1 Processor -- start 2023-10-16T09:57:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.482+0000 7fc51712f700 1 -- start start 2023-10-16T09:57:00.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.482+0000 7fc51712f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc51015b590 0x7fc5100ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.482+0000 7fc51712f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 0x7fc5100cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.482+0000 7fc51712f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100cf4c0 con 0x7fc51015f1d0 2023-10-16T09:57:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.482+0000 7fc51712f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5100cc4c0 con 0x7fc51015b590 2023-10-16T09:57:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.482+0000 7fc51712f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100b49e0 con 0x7fc5100cbaf0 2023-10-16T09:57:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50ffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc51015f1d0 0x7fc5100b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50746/0 (socket says 172.21.15.158:50746) 2023-10-16T09:57:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50ffff700 1 -- 172.21.15.158:0/1955368854 learned_addr learned my addr 172.21.15.158:0/1955368854 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:00.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993742537 0 0) 0x7fc5100cf4c0 con 0x7fc51015f1d0 2023-10-16T09:57:00.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5000035b0 con 0x7fc51015f1d0 2023-10-16T09:57:00.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc514ecb700 1 --2- 172.21.15.158:0/1955368854 >> v2:172.21.15.172:3300/0 conn(0x7fc51015b590 0x7fc5100ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc5156cc700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 0x7fc5100cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023453634 0 0) 0x7fc5000035b0 con 0x7fc51015f1d0 2023-10-16T09:57:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 msgr2=0x7fc5100cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 0x7fc5100cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 >> v2:172.21.15.172:3300/0 conn(0x7fc51015b590 msgr2=0x7fc5100ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 --2- 172.21.15.158:0/1955368854 >> v2:172.21.15.172:3300/0 conn(0x7fc51015b590 0x7fc5100ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51015f970 con 0x7fc51015f1d0 2023-10-16T09:57:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc4fc001670 con 0x7fc51015f1d0 2023-10-16T09:57:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 >> v1:172.21.15.158:6789/0 conn(0x7fc51015f1d0 legacy=0x7fc5100b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc500003a30 0x7fc500003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 0x7fc500004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc500004d90 0x7fc500005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100cc4c0 con 0x7fc500004d90 2023-10-16T09:57:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100b49e0 con 0x7fc500003a30 2023-10-16T09:57:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000ba60 con 0x7fc5100cbaf0 2023-10-16T09:57:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc5156cc700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc500004d90 0x7fc500005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc514ecb700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc500003a30 0x7fc500003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.483+0000 7fc50ffff700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 0x7fc500004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc5156cc700 1 -- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 msgr2=0x7fc500004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc5156cc700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5100cbaf0 0x7fc500004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc5156cc700 1 -- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc500003a30 msgr2=0x7fc500003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc5156cc700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc500003a30 0x7fc500003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc5156cc700 1 -- 172.21.15.158:0/1955368854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5000035b0 con 0x7fc500004d90 2023-10-16T09:57:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc5156cc700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc500004d90 0x7fc500005210 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e170 tx=0x7fc500005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5a1d7b6f246f96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50800ec90 con 0x7fc500004d90 2023-10-16T09:57:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50800f040 con 0x7fc500004d90 2023-10-16T09:57:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc50f7fe700 1 -- 172.21.15.158:0/1955368854 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508014480 con 0x7fc500004d90 2023-10-16T09:57:00.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc51712f700 1 -- 172.21.15.158:0/1955368854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc500004d90 msgr2=0x7fc500005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc51712f700 1 --2- 172.21.15.158:0/1955368854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc500004d90 0x7fc500005210 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e170 tx=0x7fc500005880 comp rx=0 tx=0).stop 2023-10-16T09:57:00.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc5156cc700 1 -- 172.21.15.158:0/1955368854 reap_dead start 2023-10-16T09:57:00.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc51712f700 1 -- 172.21.15.158:0/1955368854 shutdown_connections 2023-10-16T09:57:00.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc51712f700 1 -- 172.21.15.158:0/1955368854 >> 172.21.15.158:0/1955368854 conn(0x7fc51006a8d0 msgr2=0x7fc51006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:00.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.484+0000 7fc51712f700 1 -- 172.21.15.158:0/1955368854 shutdown_connections 2023-10-16T09:57:00.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.485+0000 7fc51712f700 1 -- 172.21.15.158:0/1955368854 wait complete. 2023-10-16T09:57:00.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.486+0000 7fc51712f700 1 Processor -- start 2023-10-16T09:57:00.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.486+0000 7fc51712f700 1 -- start start 2023-10-16T09:57:00.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc510156900 0x7fc5100c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc510156ce0 0x7fc5101f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5101f3660 0x7fc5100b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5101f3ca0 con 0x7fc510156ce0 2023-10-16T09:57:00.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5101f3f80 con 0x7fc510156900 2023-10-16T09:57:00.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100ca4c0 con 0x7fc5101f3660 2023-10-16T09:57:00.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc510156ce0 0x7fc5101f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc510156ce0 0x7fc5101f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50582/0 (socket says 172.21.15.158:50582) 2023-10-16T09:57:00.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 -- 172.21.15.158:0/1390636611 learned_addr learned my addr 172.21.15.158:0/1390636611 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:00.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc514ecb700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc510156900 0x7fc5100c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc5156cc700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5101f3660 0x7fc5100b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 -- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5101f3660 msgr2=0x7fc5100b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5101f3660 0x7fc5100b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 -- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc510156900 msgr2=0x7fc5100c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc510156900 0x7fc5100c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 -- 172.21.15.158:0/1390636611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100ca7a0 con 0x7fc510156ce0 2023-10-16T09:57:00.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc5156cc700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5101f3660 0x7fc5100b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50ffff700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc510156ce0 0x7fc5101f3120 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc004fd0 tx=0x7fc4fc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50d7fa700 1 -- 172.21.15.158:0/1390636611 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0014a0 con 0x7fc510156ce0 2023-10-16T09:57:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50d7fa700 1 -- 172.21.15.158:0/1390636611 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc0077a0 con 0x7fc510156ce0 2023-10-16T09:57:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc50d7fa700 1 -- 172.21.15.158:0/1390636611 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc00fa90 con 0x7fc510156ce0 2023-10-16T09:57:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5100c8800 con 0x7fc510156ce0 2023-10-16T09:57:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.487+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5100b5640 con 0x7fc510156ce0 2023-10-16T09:57:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.488+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5101f3a70 con 0x7fc510156ce0 2023-10-16T09:57:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.489+0000 7fc50d7fa700 1 -- 172.21.15.158:0/1390636611 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc4fc00fbf0 con 0x7fc510156ce0 2023-10-16T09:57:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.489+0000 7fc50d7fa700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc4f406b020 0x7fc4f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.489+0000 7fc514ecb700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc4f406b020 0x7fc4f406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.489+0000 7fc50d7fa700 1 -- 172.21.15.158:0/1390636611 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc4fc089740 con 0x7fc510156ce0 2023-10-16T09:57:00.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.490+0000 7fc514ecb700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc4f406b020 0x7fc4f406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc51006d130 tx=0x7fc504009210 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.494+0000 7fc50d7fa700 1 -- 172.21.15.158:0/1390636611 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4fc054760 con 0x7fc510156ce0 2023-10-16T09:57:00.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.684+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5100ca9e0 con 0x7fc510156ce0 2023-10-16T09:57:00.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.685+0000 7fc50d7fa700 1 -- 172.21.15.158:0/1390636611 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5100ca9e0 con 0x7fc510156ce0 2023-10-16T09:57:00.721 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:57:00.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc4f406b020 msgr2=0x7fc4f406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc4f406b020 0x7fc4f406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc51006d130 tx=0x7fc504009210 comp rx=0 tx=0).stop 2023-10-16T09:57:00.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc510156ce0 msgr2=0x7fc5101f3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc510156ce0 0x7fc5101f3120 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc004fd0 tx=0x7fc4fc007620 comp rx=0 tx=0).stop 2023-10-16T09:57:00.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 shutdown_connections 2023-10-16T09:57:00.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc4f406b020 0x7fc4f406d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5101f3660 0x7fc5100b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc510156900 0x7fc5100c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 --2- 172.21.15.158:0/1390636611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc510156ce0 0x7fc5101f3120 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 >> 172.21.15.158:0/1390636611 conn(0x7fc51006a8d0 msgr2=0x7fc5100b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:00.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 shutdown_connections 2023-10-16T09:57:00.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.687+0000 7fc51712f700 1 -- 172.21.15.158:0/1390636611 wait complete. 2023-10-16T09:57:00.726 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T09:57:00.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.846+0000 7fc89e316700 1 Processor -- start 2023-10-16T09:57:00.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.846+0000 7fc89e316700 1 -- start start 2023-10-16T09:57:00.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.846+0000 7fc89e316700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc89806e5d0 0x7fc89806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc89e316700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc898161270 con 0x7fc8980729b0 2023-10-16T09:57:00.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc89e316700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc898161510 con 0x7fc89806ab40 2023-10-16T09:57:00.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc89e316700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8981617f0 con 0x7fc89806e5d0 2023-10-16T09:57:00.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc89c8b3700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc8980729b0 0x7fc89815df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50768/0 (socket says 172.21.15.158:50768) 2023-10-16T09:57:00.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc89c8b3700 1 -- 172.21.15.158:0/1794145114 learned_addr learned my addr 172.21.15.158:0/1794145114 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:00.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc8977fe700 1 --2- 172.21.15.158:0/1794145114 >> v2:172.21.15.172:3300/0 conn(0x7fc89806e5d0 0x7fc89806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605491083 0 0) 0x7fc898161270 con 0x7fc8980729b0 2023-10-16T09:57:00.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8800035b0 con 0x7fc8980729b0 2023-10-16T09:57:00.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 486448879 0 0) 0x7fc8800035b0 con 0x7fc8980729b0 2023-10-16T09:57:00.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 >> v2:172.21.15.172:3300/0 conn(0x7fc89806e5d0 msgr2=0x7fc89806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 --2- 172.21.15.158:0/1794145114 >> v2:172.21.15.172:3300/0 conn(0x7fc89806e5d0 0x7fc89806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 >> v1:172.21.15.172:6789/0 conn(0x7fc89806ab40 legacy=0x7fc8980722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc898161ad0 con 0x7fc8980729b0 2023-10-16T09:57:00.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc8900019f0 con 0x7fc8980729b0 2023-10-16T09:57:00.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 >> v1:172.21.15.158:6789/0 conn(0x7fc8980729b0 legacy=0x7fc89815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc880003a00 0x7fc880003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880004850 0x7fc880004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc880005210 0x7fc88000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981617f0 con 0x7fc880003a00 2023-10-16T09:57:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88000c0b0 con 0x7fc880004850 2023-10-16T09:57:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88000c270 con 0x7fc880005210 2023-10-16T09:57:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc8977fe700 1 --2- 172.21.15.158:0/1794145114 >> v2:172.21.15.172:3300/0 conn(0x7fc89806e5d0 0x7fc89806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:57:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.847+0000 7fc897fff700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc880003a00 0x7fc880003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc89c8b3700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc880005210 0x7fc88000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc8977fe700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880004850 0x7fc880004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc897fff700 1 -- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc880005210 msgr2=0x7fc88000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc897fff700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc880005210 0x7fc88000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc897fff700 1 -- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880004850 msgr2=0x7fc880004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:00.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc897fff700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880004850 0x7fc880004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc897fff700 1 -- 172.21.15.158:0/1794145114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8800035b0 con 0x7fc880003a00 2023-10-16T09:57:00.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc89c8b3700 1 -- 172.21.15.158:0/1794145114 reap_dead start 2023-10-16T09:57:00.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc897fff700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc880003a00 0x7fc880003e20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc888001800 tx=0x7fc880005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f10f6e1293e29cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8880012f0 con 0x7fc880003a00 2023-10-16T09:57:00.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc888005310 con 0x7fc880003a00 2023-10-16T09:57:00.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc896ffd700 1 -- 172.21.15.158:0/1794145114 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888006860 con 0x7fc880003a00 2023-10-16T09:57:00.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc89e316700 1 -- 172.21.15.158:0/1794145114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc880003a00 msgr2=0x7fc880003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.848+0000 7fc89e316700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc880003a00 0x7fc880003e20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc888001800 tx=0x7fc880005880 comp rx=0 tx=0).stop 2023-10-16T09:57:00.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.849+0000 7fc89e316700 1 -- 172.21.15.158:0/1794145114 shutdown_connections 2023-10-16T09:57:00.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.849+0000 7fc89e316700 1 --2- 172.21.15.158:0/1794145114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc880003a00 0x7fc880003e20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.849+0000 7fc89e316700 1 -- 172.21.15.158:0/1794145114 >> 172.21.15.158:0/1794145114 conn(0x7fc8980d02b0 msgr2=0x7fc8980d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.849+0000 7fc89e316700 1 -- 172.21.15.158:0/1794145114 shutdown_connections 2023-10-16T09:57:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.849+0000 7fc89e316700 1 -- 172.21.15.158:0/1794145114 wait complete. 2023-10-16T09:57:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.850+0000 7fc89e316700 1 Processor -- start 2023-10-16T09:57:00.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.850+0000 7fc89e316700 1 -- start start 2023-10-16T09:57:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880003a00 0x7fc8980c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8980c75b0 0x7fc8980c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8980c7f50 0x7fc8980b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981f3870 con 0x7fc8980c7f50 2023-10-16T09:57:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981f2bf0 con 0x7fc880003a00 2023-10-16T09:57:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981f2ed0 con 0x7fc8980c75b0 2023-10-16T09:57:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8980c7f50 0x7fc8980b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8980c7f50 0x7fc8980b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50604/0 (socket says 172.21.15.158:50604) 2023-10-16T09:57:00.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 -- 172.21.15.158:0/371408915 learned_addr learned my addr 172.21.15.158:0/371408915 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc897fff700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880003a00 0x7fc8980c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc8977fe700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8980c75b0 0x7fc8980c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 -- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8980c75b0 msgr2=0x7fc8980c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8980c75b0 0x7fc8980c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 -- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880003a00 msgr2=0x7fc8980c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880003a00 0x7fc8980c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 -- 172.21.15.158:0/371408915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8981f31b0 con 0x7fc8980c7f50 2023-10-16T09:57:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89c8b3700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8980c7f50 0x7fc8980b0bb0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc890000f80 tx=0x7fc89000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc894ff9700 1 -- 172.21.15.158:0/371408915 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8900019e0 con 0x7fc8980c7f50 2023-10-16T09:57:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc894ff9700 1 -- 172.21.15.158:0/371408915 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89000ad00 con 0x7fc8980c7f50 2023-10-16T09:57:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8981f3490 con 0x7fc8980c7f50 2023-10-16T09:57:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc894ff9700 1 -- 172.21.15.158:0/371408915 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890013630 con 0x7fc8980c7f50 2023-10-16T09:57:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.851+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8980ca050 con 0x7fc8980c7f50 2023-10-16T09:57:00.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.852+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8981f39b0 con 0x7fc8980c7f50 2023-10-16T09:57:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.853+0000 7fc894ff9700 1 -- 172.21.15.158:0/371408915 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc890013790 con 0x7fc8980c7f50 2023-10-16T09:57:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.853+0000 7fc894ff9700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc88406b020 0x7fc88406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.853+0000 7fc894ff9700 1 -- 172.21.15.158:0/371408915 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc890089c60 con 0x7fc8980c7f50 2023-10-16T09:57:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.853+0000 7fc897fff700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc88406b020 0x7fc88406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:00.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.854+0000 7fc897fff700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc88406b020 0x7fc88406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc888001800 tx=0x7fc888009030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:00.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:00.858+0000 7fc894ff9700 1 -- 172.21.15.158:0/371408915 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc8981f39b0 con 0x7fc8980c7f50 2023-10-16T09:57:01.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.053+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc8980ca050 con 0x7fc8980c7f50 2023-10-16T09:57:01.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.054+0000 7fc894ff9700 1 -- 172.21.15.158:0/371408915 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc8980ca050 con 0x7fc8980c7f50 2023-10-16T09:57:01.055 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:01.056 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:57:01.056 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:01.056 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T09:57:01.056 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T09:57:01.056 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:01.057 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:57:01.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc88406b020 msgr2=0x7fc88406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc88406b020 0x7fc88406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc888001800 tx=0x7fc888009030 comp rx=0 tx=0).stop 2023-10-16T09:57:01.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8980c7f50 msgr2=0x7fc8980b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8980c7f50 0x7fc8980b0bb0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc890000f80 tx=0x7fc89000a5f0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 shutdown_connections 2023-10-16T09:57:01.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fc88406b020 0x7fc88406d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8980c7f50 0x7fc8980b0bb0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc880003a00 0x7fc8980c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 --2- 172.21.15.158:0/371408915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8980c75b0 0x7fc8980c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 >> 172.21.15.158:0/371408915 conn(0x7fc8980d02b0 msgr2=0x7fc89815d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:01.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 shutdown_connections 2023-10-16T09:57:01.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.056+0000 7fc89e316700 1 -- 172.21.15.158:0/371408915 wait complete. 2023-10-16T09:57:01.069 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T09:57:01.070 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T09:57:01.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.193+0000 7f1971b06700 1 Processor -- start 2023-10-16T09:57:01.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.193+0000 7f1971b06700 1 -- start start 2023-10-16T09:57:01.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f1971b06700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f196c15dad0 0x7f196c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f1971b06700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f1971b06700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f196c0cea20 0x7f196c0c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f1971b06700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196c0d0ae0 con 0x7f196c15dad0 2023-10-16T09:57:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f1971b06700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196c0d12d0 con 0x7f196c0cea20 2023-10-16T09:57:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f1971b06700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196c0cbdf0 con 0x7f196c15deb0 2023-10-16T09:57:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f196c15dad0 0x7f196c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f196c15dad0 0x7f196c0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50608/0 (socket says 172.21.15.158:50608) 2023-10-16T09:57:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 -- 172.21.15.158:0/3358868365 learned_addr learned my addr 172.21.15.158:0/3358868365 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196affd700 1 --2- 172.21.15.158:0/3358868365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 -- 172.21.15.158:0/3358868365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 msgr2=0x7f196c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 --2- 172.21.15.158:0/3358868365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196bfff700 1 --2- 172.21.15.158:0/3358868365 >> v2:172.21.15.172:3300/0 conn(0x7f196c0cea20 0x7f196c0c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 -- 172.21.15.158:0/3358868365 >> v2:172.21.15.172:3300/0 conn(0x7f196c0cea20 msgr2=0x7f196c0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 --2- 172.21.15.158:0/3358868365 >> v2:172.21.15.172:3300/0 conn(0x7f196c0cea20 0x7f196c0c3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.194+0000 7f196b7fe700 1 -- 172.21.15.158:0/3358868365 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c0cc280 con 0x7f196c15dad0 2023-10-16T09:57:01.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f196b7fe700 1 --2- 172.21.15.158:0/3358868365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c15dad0 0x7f196c0cdfa0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f196c071950 tx=0x7f196000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8958acc814ea67 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:01.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f196a7fc700 1 -- 172.21.15.158:0/3358868365 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196000dd30 con 0x7f196c15dad0 2023-10-16T09:57:01.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f196a7fc700 1 -- 172.21.15.158:0/3358868365 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196000de90 con 0x7f196c15dad0 2023-10-16T09:57:01.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f196a7fc700 1 -- 172.21.15.158:0/3358868365 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960010690 con 0x7f196c15dad0 2023-10-16T09:57:01.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 -- 172.21.15.158:0/3358868365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c15dad0 msgr2=0x7f196c0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 --2- 172.21.15.158:0/3358868365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c15dad0 0x7f196c0cdfa0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f196c071950 tx=0x7f196000be70 comp rx=0 tx=0).stop 2023-10-16T09:57:01.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 -- 172.21.15.158:0/3358868365 shutdown_connections 2023-10-16T09:57:01.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 --2- 172.21.15.158:0/3358868365 >> v2:172.21.15.172:3300/0 conn(0x7f196c0cea20 0x7f196c0c3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 --2- 172.21.15.158:0/3358868365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c15dad0 0x7f196c0cdfa0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 --2- 172.21.15.158:0/3358868365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 -- 172.21.15.158:0/3358868365 >> 172.21.15.158:0/3358868365 conn(0x7f196c099a10 msgr2=0x7f196c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:01.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 -- 172.21.15.158:0/3358868365 shutdown_connections 2023-10-16T09:57:01.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.195+0000 7f1971b06700 1 -- 172.21.15.158:0/3358868365 wait complete. 2023-10-16T09:57:01.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.196+0000 7f1971b06700 1 Processor -- start 2023-10-16T09:57:01.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.197+0000 7f1971b06700 1 -- start start 2023-10-16T09:57:01.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.197+0000 7f1971b06700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c0cea20 0x7f196c155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.197+0000 7f1971b06700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196c15dad0 0x7f196c1597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.197+0000 7f1971b06700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.197+0000 7f1971b06700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c15be90 con 0x7f196c0cea20 2023-10-16T09:57:01.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.197+0000 7f1971b06700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c15bbb0 con 0x7f196c15dad0 2023-10-16T09:57:01.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.197+0000 7f1971b06700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196c157e50 con 0x7f196c15deb0 2023-10-16T09:57:01.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c0cea20 0x7f196c155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c0cea20 0x7f196c155640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50612/0 (socket says 172.21.15.158:50612) 2023-10-16T09:57:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 -- 172.21.15.158:0/556139893 learned_addr learned my addr 172.21.15.158:0/556139893 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196affd700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196c15dad0 0x7f196c1597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196bfff700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 -- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 msgr2=0x7f196c159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 -- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196c15dad0 msgr2=0x7f196c1597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196c15dad0 0x7f196c1597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 -- 172.21.15.158:0/556139893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196000b7f0 con 0x7f196c0cea20 2023-10-16T09:57:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f196b7fe700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c0cea20 0x7f196c155640 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f196000beb0 tx=0x7f1960004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:01.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f1953fff700 1 -- 172.21.15.158:0/556139893 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960003cf0 con 0x7f196c0cea20 2023-10-16T09:57:01.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f1953fff700 1 -- 172.21.15.158:0/556139893 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1960003e70 con 0x7f196c0cea20 2023-10-16T09:57:01.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196c158130 con 0x7f196c0cea20 2023-10-16T09:57:01.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f1953fff700 1 -- 172.21.15.158:0/556139893 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960010690 con 0x7f196c0cea20 2023-10-16T09:57:01.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.198+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196c151e40 con 0x7f196c0cea20 2023-10-16T09:57:01.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.199+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196c157cb0 con 0x7f196c0cea20 2023-10-16T09:57:01.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.200+0000 7f1953fff700 1 -- 172.21.15.158:0/556139893 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1960019400 con 0x7f196c0cea20 2023-10-16T09:57:01.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.200+0000 7f1953fff700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f195806b020 0x7f195806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.200+0000 7f1953fff700 1 -- 172.21.15.158:0/556139893 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f196008ab90 con 0x7f196c0cea20 2023-10-16T09:57:01.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.200+0000 7f196affd700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f195806b020 0x7f195806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.200+0000 7f196affd700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f195806b020 0x7f195806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f196c1591c0 tx=0x7f195c008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:01.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.204+0000 7f1953fff700 1 -- 172.21.15.158:0/556139893 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f196c157cb0 con 0x7f196c0cea20 2023-10-16T09:57:01.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.397+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f196c157cb0 con 0x7f196c0cea20 2023-10-16T09:57:01.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.591+0000 7f1953fff700 1 -- 172.21.15.158:0/556139893 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f196c157cb0 con 0x7f196c0cea20 2023-10-16T09:57:01.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f195806b020 msgr2=0x7f195806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f195806b020 0x7f195806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f196c1591c0 tx=0x7f195c008040 comp rx=0 tx=0).stop 2023-10-16T09:57:01.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c0cea20 msgr2=0x7f196c155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c0cea20 0x7f196c155640 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f196000beb0 tx=0x7f1960004750 comp rx=0 tx=0).stop 2023-10-16T09:57:01.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 shutdown_connections 2023-10-16T09:57:01.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f195806b020 0x7f195806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196c15deb0 0x7f196c159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f196c0cea20 0x7f196c155640 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 --2- 172.21.15.158:0/556139893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196c15dad0 0x7f196c1597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 >> 172.21.15.158:0/556139893 conn(0x7f196c099a10 msgr2=0x7f196c0cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:01.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 shutdown_connections 2023-10-16T09:57:01.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.593+0000 7f1971b06700 1 -- 172.21.15.158:0/556139893 wait complete. 2023-10-16T09:57:01.597 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T09:57:01.610 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T09:57:01.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.737+0000 7fd3ee233700 1 Processor -- start 2023-10-16T09:57:01.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.737+0000 7fd3ee233700 1 -- start start 2023-10-16T09:57:01.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3ee233700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3e806a8d0 0x7fd3e806ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3ee233700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e8071d70 con 0x7fd3e80af5f0 2023-10-16T09:57:01.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3ee233700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e80af9d0 con 0x7fd3e80b3b60 2023-10-16T09:57:01.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3ee233700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e806b2c0 con 0x7fd3e806a8d0 2023-10-16T09:57:01.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3e77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd3e80af5f0 0x7fd3e8161e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50796/0 (socket says 172.21.15.158:50796) 2023-10-16T09:57:01.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3e77fe700 1 -- 172.21.15.158:0/4290507185 learned_addr learned my addr 172.21.15.158:0/4290507185 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:01.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3e7fff700 1 --2- 172.21.15.158:0/4290507185 >> v2:172.21.15.172:3300/0 conn(0x7fd3e806a8d0 0x7fd3e806ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3e67fc700 1 -- 172.21.15.158:0/4290507185 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069485527 0 0) 0x7fd3e8071d70 con 0x7fd3e80af5f0 2023-10-16T09:57:01.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.738+0000 7fd3e67fc700 1 -- 172.21.15.158:0/4290507185 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d00035b0 con 0x7fd3e80af5f0 2023-10-16T09:57:01.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e67fc700 1 -- 172.21.15.158:0/4290507185 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356038642 0 0) 0x7fd3d00035b0 con 0x7fd3e80af5f0 2023-10-16T09:57:01.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e7fff700 1 -- 172.21.15.158:0/4290507185 >> v1:172.21.15.172:6789/0 conn(0x7fd3e80b3b60 legacy=0x7fd3e80724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e7fff700 1 -- 172.21.15.158:0/4290507185 >> v1:172.21.15.158:6789/0 conn(0x7fd3e80af5f0 legacy=0x7fd3e8161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e7fff700 1 -- 172.21.15.158:0/4290507185 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e80b3970 con 0x7fd3e806a8d0 2023-10-16T09:57:01.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e7fff700 1 --2- 172.21.15.158:0/4290507185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e806a8d0 0x7fd3e806ad80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009d90 tx=0x7fd3e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cb3272cf8d5b919 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:01.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e67fc700 1 -- 172.21.15.158:0/4290507185 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e000e070 con 0x7fd3e806a8d0 2023-10-16T09:57:01.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e67fc700 1 -- 172.21.15.158:0/4290507185 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e0003a20 con 0x7fd3e806a8d0 2023-10-16T09:57:01.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.739+0000 7fd3e67fc700 1 -- 172.21.15.158:0/4290507185 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc001670 con 0x7fd3e806a8d0 2023-10-16T09:57:01.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.740+0000 7fd3ee233700 1 -- 172.21.15.158:0/4290507185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e806a8d0 msgr2=0x7fd3e806ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.740+0000 7fd3ee233700 1 --2- 172.21.15.158:0/4290507185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e806a8d0 0x7fd3e806ad80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009d90 tx=0x7fd3e000b330 comp rx=0 tx=0).stop 2023-10-16T09:57:01.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.740+0000 7fd3ee233700 1 -- 172.21.15.158:0/4290507185 shutdown_connections 2023-10-16T09:57:01.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.740+0000 7fd3ee233700 1 --2- 172.21.15.158:0/4290507185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e806a8d0 0x7fd3e806ad80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.740+0000 7fd3ee233700 1 -- 172.21.15.158:0/4290507185 >> 172.21.15.158:0/4290507185 conn(0x7fd3e80ccd30 msgr2=0x7fd3e80cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:01.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.740+0000 7fd3ee233700 1 -- 172.21.15.158:0/4290507185 shutdown_connections 2023-10-16T09:57:01.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.740+0000 7fd3ee233700 1 -- 172.21.15.158:0/4290507185 wait complete. 2023-10-16T09:57:01.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.741+0000 7fd3ee233700 1 Processor -- start 2023-10-16T09:57:01.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.741+0000 7fd3ee233700 1 -- start start 2023-10-16T09:57:01.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3ee233700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e806a8d0 0x7fd3e815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3ee233700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3e80af5f0 0x7fd3e80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3ee233700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e80b3b60 0x7fd3e80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3ee233700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80b2c40 con 0x7fd3e806a8d0 2023-10-16T09:57:01.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3ee233700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80b2960 con 0x7fd3e80b3b60 2023-10-16T09:57:01.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3ee233700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80afeb0 con 0x7fd3e80af5f0 2023-10-16T09:57:01.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e806a8d0 0x7fd3e815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e806a8d0 0x7fd3e815bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50626/0 (socket says 172.21.15.158:50626) 2023-10-16T09:57:01.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 -- 172.21.15.158:0/3556890053 learned_addr learned my addr 172.21.15.158:0/3556890053 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:01.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e6ffd700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3e80af5f0 0x7fd3e80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e7fff700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e80b3b60 0x7fd3e80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 -- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3e80af5f0 msgr2=0x7fd3e80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3e80af5f0 0x7fd3e80c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 -- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e80b3b60 msgr2=0x7fd3e80b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:01.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e80b3b60 0x7fd3e80b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:01.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 -- 172.21.15.158:0/3556890053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0009a70 con 0x7fd3e806a8d0 2023-10-16T09:57:01.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.742+0000 7fd3e77fe700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e806a8d0 0x7fd3e815bdc0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0011e0 tx=0x7fd3dc00ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:01.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.743+0000 7fd3cffff700 1 -- 172.21.15.158:0/3556890053 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc004c80 con 0x7fd3e806a8d0 2023-10-16T09:57:01.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.743+0000 7fd3cffff700 1 -- 172.21.15.158:0/3556890053 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc004de0 con 0x7fd3e806a8d0 2023-10-16T09:57:01.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.743+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e80b0190 con 0x7fd3e806a8d0 2023-10-16T09:57:01.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.743+0000 7fd3cffff700 1 -- 172.21.15.158:0/3556890053 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0072d0 con 0x7fd3e806a8d0 2023-10-16T09:57:01.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.743+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e80c5580 con 0x7fd3e806a8d0 2023-10-16T09:57:01.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.744+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e80afd10 con 0x7fd3e806a8d0 2023-10-16T09:57:01.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.744+0000 7fd3cffff700 1 -- 172.21.15.158:0/3556890053 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd3dc007430 con 0x7fd3e806a8d0 2023-10-16T09:57:01.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.745+0000 7fd3cffff700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fd3d406b020 0x7fd3d406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:01.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.745+0000 7fd3cffff700 1 -- 172.21.15.158:0/3556890053 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5943+0+0 (secure 0 0 0) 0x7fd3dc089580 con 0x7fd3e806a8d0 2023-10-16T09:57:01.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.745+0000 7fd3e6ffd700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fd3d406b020 0x7fd3d406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:01.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.745+0000 7fd3e6ffd700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fd3d406b020 0x7fd3d406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80c6bc0 tx=0x7fd3d8009040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:01.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.750+0000 7fd3cffff700 1 -- 172.21.15.158:0/3556890053 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3e80afd10 con 0x7fd3e806a8d0 2023-10-16T09:57:01.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:01.943+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd3e80afd10 con 0x7fd3e806a8d0 2023-10-16T09:57:02.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.595+0000 7fd3cffff700 1 -- 172.21.15.158:0/3556890053 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd3e80afd10 con 0x7fd3e806a8d0 2023-10-16T09:57:02.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.600+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fd3d406b020 msgr2=0x7fd3d406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:02.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.600+0000 7fd3ee233700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fd3d406b020 0x7fd3d406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80c6bc0 tx=0x7fd3d8009040 comp rx=0 tx=0).stop 2023-10-16T09:57:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.600+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e806a8d0 msgr2=0x7fd3e815bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.600+0000 7fd3ee233700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e806a8d0 0x7fd3e815bdc0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0011e0 tx=0x7fd3dc00ca50 comp rx=0 tx=0).stop 2023-10-16T09:57:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.601+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 shutdown_connections 2023-10-16T09:57:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.601+0000 7fd3ee233700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fd3d406b020 0x7fd3d406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.601+0000 7fd3ee233700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e80b3b60 0x7fd3e80b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.601+0000 7fd3ee233700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e806a8d0 0x7fd3e815bdc0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.601+0000 7fd3ee233700 1 --2- 172.21.15.158:0/3556890053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3e80af5f0 0x7fd3e80c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.601+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 >> 172.21.15.158:0/3556890053 conn(0x7fd3e80ccd30 msgr2=0x7fd3e815c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.603+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 shutdown_connections 2023-10-16T09:57:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.603+0000 7fd3ee233700 1 -- 172.21.15.158:0/3556890053 wait complete. 2023-10-16T09:57:02.618 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T09:57:02.632 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T09:57:02.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.756+0000 7f4c17085700 1 Processor -- start 2023-10-16T09:57:02.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.756+0000 7f4c17085700 1 -- start start 2023-10-16T09:57:02.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c17085700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c1015da40 0x7f4c100caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c17085700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100cbd30 con 0x7f4c100cc590 2023-10-16T09:57:02.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c17085700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100cc1c0 con 0x7f4c1006afe0 2023-10-16T09:57:02.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c17085700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1015de20 con 0x7f4c1015da40 2023-10-16T09:57:02.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c0ffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4c100cc590 0x7f4c1006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50810/0 (socket says 172.21.15.158:50810) 2023-10-16T09:57:02.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c0ffff700 1 -- 172.21.15.158:0/1447617986 learned_addr learned my addr 172.21.15.158:0/1447617986 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:02.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c14e21700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c1015da40 0x7f4c100caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.757+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549255965 0 0) 0x7f4c100cbd30 con 0x7f4c100cc590 2023-10-16T09:57:02.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c000035b0 con 0x7f4c100cc590 2023-10-16T09:57:02.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206834575 0 0) 0x7f4c100cc1c0 con 0x7f4c1006afe0 2023-10-16T09:57:02.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c100cbd30 con 0x7f4c1006afe0 2023-10-16T09:57:02.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3048178503 0 0) 0x7f4c000035b0 con 0x7f4c100cc590 2023-10-16T09:57:02.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c1015da40 msgr2=0x7f4c100caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:02.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c1015da40 0x7f4c100caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 >> v1:172.21.15.172:6789/0 conn(0x7f4c1006afe0 legacy=0x7f4c100afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:02.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100ce260 con 0x7f4c100cc590 2023-10-16T09:57:02.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4bfc001670 con 0x7f4c100cc590 2023-10-16T09:57:02.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 >> v1:172.21.15.158:6789/0 conn(0x7f4c100cc590 legacy=0x7f4c1006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:02.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c00003bc0 0x7f4c00003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c00004850 0x7f4c00004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c00005210 0x7f4c0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1015de20 con 0x7f4c00005210 2023-10-16T09:57:02.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0000c000 con 0x7f4c00003bc0 2023-10-16T09:57:02.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0000c1c0 con 0x7f4c00004850 2023-10-16T09:57:02.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c14e21700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c1015da40 0x7f4c100caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:57:02.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c15622700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c00005210 0x7f4c0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c0ffff700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c00004850 0x7f4c00004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c14e21700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c00003bc0 0x7f4c00003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c15622700 1 -- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c00004850 msgr2=0x7f4c00004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:02.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c15622700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c00004850 0x7f4c00004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c15622700 1 -- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c00003bc0 msgr2=0x7f4c00003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:02.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c15622700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c00003bc0 0x7f4c00003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c15622700 1 -- 172.21.15.158:0/1447617986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c000035b0 con 0x7f4c00005210 2023-10-16T09:57:02.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.758+0000 7f4c15622700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c00005210 0x7f4c0000ba10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4c10072cc0 tx=0x7f4c00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b495b124ce981e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:02.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08001f20 con 0x7f4c00005210 2023-10-16T09:57:02.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0800ce40 con 0x7f4c00005210 2023-10-16T09:57:02.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c0f7fe700 1 -- 172.21.15.158:0/1447617986 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08003c40 con 0x7f4c00005210 2023-10-16T09:57:02.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c17085700 1 -- 172.21.15.158:0/1447617986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c00005210 msgr2=0x7f4c0000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:02.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c17085700 1 --2- 172.21.15.158:0/1447617986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c00005210 0x7f4c0000ba10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4c10072cc0 tx=0x7f4c00005880 comp rx=0 tx=0).stop 2023-10-16T09:57:02.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c15622700 1 -- 172.21.15.158:0/1447617986 reap_dead start 2023-10-16T09:57:02.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c17085700 1 -- 172.21.15.158:0/1447617986 shutdown_connections 2023-10-16T09:57:02.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c17085700 1 -- 172.21.15.158:0/1447617986 >> 172.21.15.158:0/1447617986 conn(0x7f4c10099a10 msgr2=0x7f4c100d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:02.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c17085700 1 -- 172.21.15.158:0/1447617986 shutdown_connections 2023-10-16T09:57:02.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.759+0000 7f4c17085700 1 -- 172.21.15.158:0/1447617986 wait complete. 2023-10-16T09:57:02.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.760+0000 7f4c17085700 1 Processor -- start 2023-10-16T09:57:02.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.760+0000 7f4c17085700 1 -- start start 2023-10-16T09:57:02.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c17085700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1014e850 0x7f4c100c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c17085700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c100ca2c0 0x7f4c100c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c17085700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c101f63b0 0x7f4c100c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c17085700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100b1f50 con 0x7f4c100ca2c0 2023-10-16T09:57:02.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c17085700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101f6950 con 0x7f4c1014e850 2023-10-16T09:57:02.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c17085700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101f6c30 con 0x7f4c101f63b0 2023-10-16T09:57:02.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c0ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c100ca2c0 0x7f4c100c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c0ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c100ca2c0 0x7f4c100c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50646/0 (socket says 172.21.15.158:50646) 2023-10-16T09:57:02.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.761+0000 7f4c0ffff700 1 -- 172.21.15.158:0/510842768 learned_addr learned my addr 172.21.15.158:0/510842768 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:02.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c14e21700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1014e850 0x7f4c100c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c15622700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c101f63b0 0x7f4c100c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0ffff700 1 -- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c101f63b0 msgr2=0x7f4c100c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:02.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0ffff700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c101f63b0 0x7f4c100c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0ffff700 1 -- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1014e850 msgr2=0x7f4c100c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:02.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0ffff700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1014e850 0x7f4c100c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:02.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0ffff700 1 -- 172.21.15.158:0/510842768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c101f6f10 con 0x7f4c100ca2c0 2023-10-16T09:57:02.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c14e21700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1014e850 0x7f4c100c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:57:02.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c15622700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c101f63b0 0x7f4c100c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:57:02.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0ffff700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c100ca2c0 0x7f4c100c6380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc007340 tx=0x7f4bfc007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:02.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0d7fa700 1 -- 172.21.15.158:0/510842768 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc007820 con 0x7f4c100ca2c0 2023-10-16T09:57:02.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0d7fa700 1 -- 172.21.15.158:0/510842768 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc00f870 con 0x7f4c100ca2c0 2023-10-16T09:57:02.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c101f71f0 con 0x7f4c100ca2c0 2023-10-16T09:57:02.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c0d7fa700 1 -- 172.21.15.158:0/510842768 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc00fc00 con 0x7f4c100ca2c0 2023-10-16T09:57:02.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.762+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1014ec30 con 0x7f4c100ca2c0 2023-10-16T09:57:02.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.763+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1014ee20 con 0x7f4c100ca2c0 2023-10-16T09:57:02.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.763+0000 7f4c0d7fa700 1 -- 172.21.15.158:0/510842768 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4bfc00fd60 con 0x7f4c100ca2c0 2023-10-16T09:57:02.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.764+0000 7f4c0d7fa700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f4bf406b020 0x7f4bf406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:02.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.764+0000 7f4c0d7fa700 1 -- 172.21.15.158:0/510842768 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6325+0+0 (secure 0 0 0) 0x7f4bfc089b30 con 0x7f4c100ca2c0 2023-10-16T09:57:02.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.764+0000 7f4c14e21700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f4bf406b020 0x7f4bf406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:02.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.764+0000 7f4c14e21700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f4bf406b020 0x7f4bf406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4c100d0060 tx=0x7f4c04008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:02.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:02.768+0000 7f4c0d7fa700 1 -- 172.21.15.158:0/510842768 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bfc054860 con 0x7f4c100ca2c0 2023-10-16T09:57:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.004+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4c100681b0 con 0x7f4c100ca2c0 2023-10-16T09:57:03.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.601+0000 7f4c0d7fa700 1 -- 172.21.15.158:0/510842768 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4bfc057e80 con 0x7f4c100ca2c0 2023-10-16T09:57:03.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f4bf406b020 msgr2=0x7f4bf406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f4bf406b020 0x7f4bf406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4c100d0060 tx=0x7f4c04008040 comp rx=0 tx=0).stop 2023-10-16T09:57:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c100ca2c0 msgr2=0x7f4c100c6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c100ca2c0 0x7f4c100c6380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc007340 tx=0x7f4bfc007700 comp rx=0 tx=0).stop 2023-10-16T09:57:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 shutdown_connections 2023-10-16T09:57:03.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f4bf406b020 0x7f4bf406d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c101f63b0 0x7f4c100c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1014e850 0x7f4c100c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 --2- 172.21.15.158:0/510842768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c100ca2c0 0x7f4c100c6380 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 >> 172.21.15.158:0/510842768 conn(0x7f4c10099a10 msgr2=0x7f4c1015cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:03.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 shutdown_connections 2023-10-16T09:57:03.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.604+0000 7f4c17085700 1 -- 172.21.15.158:0/510842768 wait complete. 2023-10-16T09:57:03.608 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 7 and data pool 8 2023-10-16T09:57:03.622 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:57:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.742+0000 7ffb9bb67700 1 Processor -- start 2023-10-16T09:57:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.742+0000 7ffb9bb67700 1 -- start start 2023-10-16T09:57:03.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb9bb67700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb9406e5d0 0x7ffb9406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb9bb67700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb94161270 con 0x7ffb940729b0 2023-10-16T09:57:03.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb9bb67700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb94161510 con 0x7ffb9406ab40 2023-10-16T09:57:03.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb9bb67700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb941617f0 con 0x7ffb9406e5d0 2023-10-16T09:57:03.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb9a104700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffb940729b0 0x7ffb9415df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50834/0 (socket says 172.21.15.158:50834) 2023-10-16T09:57:03.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb9a104700 1 -- 172.21.15.158:0/1840618624 learned_addr learned my addr 172.21.15.158:0/1840618624 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:03.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb99102700 1 --2- 172.21.15.158:0/1840618624 >> v2:172.21.15.172:3300/0 conn(0x7ffb9406e5d0 0x7ffb9406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309125518 0 0) 0x7ffb94161270 con 0x7ffb940729b0 2023-10-16T09:57:03.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb800035b0 con 0x7ffb940729b0 2023-10-16T09:57:03.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859172426 0 0) 0x7ffb800035b0 con 0x7ffb940729b0 2023-10-16T09:57:03.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 >> v2:172.21.15.172:3300/0 conn(0x7ffb9406e5d0 msgr2=0x7ffb9406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:03.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 --2- 172.21.15.158:0/1840618624 >> v2:172.21.15.172:3300/0 conn(0x7ffb9406e5d0 0x7ffb9406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 >> v1:172.21.15.172:6789/0 conn(0x7ffb9406ab40 legacy=0x7ffb940722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94161ad0 con 0x7ffb940729b0 2023-10-16T09:57:03.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ffb8c0019f0 con 0x7ffb940729b0 2023-10-16T09:57:03.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 >> v1:172.21.15.158:6789/0 conn(0x7ffb940729b0 legacy=0x7ffb9415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb80003a00 0x7ffb80003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb80004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb80005210 0x7ffb8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb941617f0 con 0x7ffb80004850 2023-10-16T09:57:03.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8000c0b0 con 0x7ffb80003a00 2023-10-16T09:57:03.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8000c270 con 0x7ffb80005210 2023-10-16T09:57:03.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.743+0000 7ffb99102700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb80004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb9a104700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb80005210 0x7ffb8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb99903700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb80003a00 0x7ffb80003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb99102700 1 -- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb80005210 msgr2=0x7ffb8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:03.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb99102700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb80005210 0x7ffb8000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb99102700 1 -- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb80003a00 msgr2=0x7ffb80003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:03.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb99102700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb80003a00 0x7ffb80003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb9a104700 1 -- 172.21.15.158:0/1840618624 reap_dead start 2023-10-16T09:57:03.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb99102700 1 -- 172.21.15.158:0/1840618624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb800035b0 con 0x7ffb80004850 2023-10-16T09:57:03.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb99102700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb80004cd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009b30 tx=0x7ffb80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3969e309c7fe4a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:03.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9000e6c0 con 0x7ffb80004850 2023-10-16T09:57:03.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9000e820 con 0x7ffb80004850 2023-10-16T09:57:03.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb98901700 1 -- 172.21.15.158:0/1840618624 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9000eb90 con 0x7ffb80004850 2023-10-16T09:57:03.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb9bb67700 1 -- 172.21.15.158:0/1840618624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 msgr2=0x7ffb80004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.744+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb80004cd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009b30 tx=0x7ffb80005880 comp rx=0 tx=0).stop 2023-10-16T09:57:03.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.745+0000 7ffb9bb67700 1 -- 172.21.15.158:0/1840618624 shutdown_connections 2023-10-16T09:57:03.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.745+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/1840618624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb80004cd0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.745+0000 7ffb9bb67700 1 -- 172.21.15.158:0/1840618624 >> 172.21.15.158:0/1840618624 conn(0x7ffb940d02b0 msgr2=0x7ffb940d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:03.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.745+0000 7ffb9bb67700 1 -- 172.21.15.158:0/1840618624 shutdown_connections 2023-10-16T09:57:03.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.745+0000 7ffb9bb67700 1 -- 172.21.15.158:0/1840618624 wait complete. 2023-10-16T09:57:03.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.746+0000 7ffb9bb67700 1 Processor -- start 2023-10-16T09:57:03.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.746+0000 7ffb9bb67700 1 -- start start 2023-10-16T09:57:03.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.746+0000 7ffb9bb67700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb940b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9bb67700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb941f2b50 0x7ffb940ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9bb67700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb9414a2b0 0x7ffb940acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9bb67700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940b1fe0 con 0x7ffb80004850 2023-10-16T09:57:03.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9bb67700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940b29b0 con 0x7ffb941f2b50 2023-10-16T09:57:03.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9bb67700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940b13d0 con 0x7ffb9414a2b0 2023-10-16T09:57:03.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb940b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb940b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50666/0 (socket says 172.21.15.158:50666) 2023-10-16T09:57:03.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 -- 172.21.15.158:0/799580667 learned_addr learned my addr 172.21.15.158:0/799580667 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:03.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9a104700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb9414a2b0 0x7ffb940acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99102700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb941f2b50 0x7ffb940ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 -- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb9414a2b0 msgr2=0x7ffb940acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb9414a2b0 0x7ffb940acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 -- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb941f2b50 msgr2=0x7ffb940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb941f2b50 0x7ffb940ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 -- 172.21.15.158:0/799580667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940b16b0 con 0x7ffb80004850 2023-10-16T09:57:03.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb99903700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb940b2470 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ffb8400f820 tx=0x7ffb8400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:03.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb8a7fc700 1 -- 172.21.15.158:0/799580667 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84005320 con 0x7ffb80004850 2023-10-16T09:57:03.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb8a7fc700 1 -- 172.21.15.158:0/799580667 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb84005480 con 0x7ffb80004850 2023-10-16T09:57:03.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb940b1990 con 0x7ffb80004850 2023-10-16T09:57:03.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb8a7fc700 1 -- 172.21.15.158:0/799580667 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84006aa0 con 0x7ffb80004850 2023-10-16T09:57:03.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.747+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb940b1060 con 0x7ffb80004850 2023-10-16T09:57:03.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.748+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb940b22c0 con 0x7ffb80004850 2023-10-16T09:57:03.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.749+0000 7ffb8a7fc700 1 -- 172.21.15.158:0/799580667 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffb84006c00 con 0x7ffb80004850 2023-10-16T09:57:03.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.749+0000 7ffb8a7fc700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffb7806b020 0x7ffb7806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:03.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.749+0000 7ffb8a7fc700 1 -- 172.21.15.158:0/799580667 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6425+0+0 (secure 0 0 0) 0x7ffb840890c0 con 0x7ffb80004850 2023-10-16T09:57:03.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.749+0000 7ffb99102700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffb7806b020 0x7ffb7806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:03.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.750+0000 7ffb99102700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffb7806b020 0x7ffb7806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffb900090b0 tx=0x7ffb9001e040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:03.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.754+0000 7ffb8a7fc700 1 -- 172.21.15.158:0/799580667 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb84053e20 con 0x7ffb80004850 2023-10-16T09:57:03.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.947+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb940681b0 con 0x7ffb80004850 2023-10-16T09:57:03.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.948+0000 7ffb8a7fc700 1 -- 172.21.15.158:0/799580667 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17400 (secure 0 0 0) 0x7ffb84057440 con 0x7ffb80004850 2023-10-16T09:57:03.949 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:03.950 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":66,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:57:03.005431+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:57:01.416542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T09:57:01.962552+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:57:03.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffb7806b020 msgr2=0x7ffb7806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffb7806b020 0x7ffb7806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffb900090b0 tx=0x7ffb9001e040 comp rx=0 tx=0).stop 2023-10-16T09:57:03.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 msgr2=0x7ffb940b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:03.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb940b2470 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ffb8400f820 tx=0x7ffb8400fb30 comp rx=0 tx=0).stop 2023-10-16T09:57:03.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 shutdown_connections 2023-10-16T09:57:03.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7ffb7806b020 0x7ffb7806d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb9414a2b0 0x7ffb940acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb80004850 0x7ffb940b2470 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 --2- 172.21.15.158:0/799580667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb941f2b50 0x7ffb940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:03.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 >> 172.21.15.158:0/799580667 conn(0x7ffb940d02b0 msgr2=0x7ffb940ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:03.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 shutdown_connections 2023-10-16T09:57:03.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:03.950+0000 7ffb9bb67700 1 -- 172.21.15.158:0/799580667 wait complete. 2023-10-16T09:57:03.964 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:57:04.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.084+0000 7f275620c700 1 Processor -- start 2023-10-16T09:57:04.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.084+0000 7f275620c700 1 -- start start 2023-10-16T09:57:04.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f275620c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f275006e5d0 0x7f275006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f275620c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27500729b0 0x7f2750071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f275620c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2750071e50 con 0x7f275006ab40 2023-10-16T09:57:04.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f275620c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2750161270 con 0x7f275006e5d0 2023-10-16T09:57:04.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f275620c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2750161510 con 0x7f27500729b0 2023-10-16T09:57:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27500729b0 0x7f2750071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274effd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f275006e5d0 0x7f275006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27500729b0 0x7f2750071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:33280/0 (socket says 172.21.15.158:33280) 2023-10-16T09:57:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274ffff700 1 -- 172.21.15.158:0/1713486836 learned_addr learned my addr 172.21.15.158:0/1713486836 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:04.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274ffff700 1 -- 172.21.15.158:0/1713486836 >> v2:172.21.15.172:3300/0 conn(0x7f275006e5d0 msgr2=0x7f275006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274ffff700 1 --2- 172.21.15.158:0/1713486836 >> v2:172.21.15.172:3300/0 conn(0x7f275006e5d0 0x7f275006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274ffff700 1 -- 172.21.15.158:0/1713486836 >> v1:172.21.15.172:6789/0 conn(0x7f275006ab40 legacy=0x7f27500722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274ffff700 1 -- 172.21.15.158:0/1713486836 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27501617f0 con 0x7f27500729b0 2023-10-16T09:57:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.085+0000 7f274effd700 1 --2- 172.21.15.158:0/1713486836 >> v2:172.21.15.172:3300/0 conn(0x7f275006e5d0 0x7f275006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:57:04.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f274ffff700 1 --2- 172.21.15.158:0/1713486836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27500729b0 0x7f2750071910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2748009e50 tx=0x7f274800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c437adfd363dc0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:04.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f274e7fc700 1 -- 172.21.15.158:0/1713486836 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274800e070 con 0x7f27500729b0 2023-10-16T09:57:04.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f274e7fc700 1 -- 172.21.15.158:0/1713486836 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274800b910 con 0x7f27500729b0 2023-10-16T09:57:04.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f274e7fc700 1 -- 172.21.15.158:0/1713486836 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274800bcc0 con 0x7f27500729b0 2023-10-16T09:57:04.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 -- 172.21.15.158:0/1713486836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27500729b0 msgr2=0x7f2750071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 --2- 172.21.15.158:0/1713486836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27500729b0 0x7f2750071910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2748009e50 tx=0x7f274800b330 comp rx=0 tx=0).stop 2023-10-16T09:57:04.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 -- 172.21.15.158:0/1713486836 shutdown_connections 2023-10-16T09:57:04.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 --2- 172.21.15.158:0/1713486836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27500729b0 0x7f2750071910 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 --2- 172.21.15.158:0/1713486836 >> v2:172.21.15.172:3300/0 conn(0x7f275006e5d0 0x7f275006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 -- 172.21.15.158:0/1713486836 >> 172.21.15.158:0/1713486836 conn(0x7f27500d02b0 msgr2=0x7f27500d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:04.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 -- 172.21.15.158:0/1713486836 shutdown_connections 2023-10-16T09:57:04.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.086+0000 7f275620c700 1 -- 172.21.15.158:0/1713486836 wait complete. 2023-10-16T09:57:04.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.087+0000 7f275620c700 1 Processor -- start 2023-10-16T09:57:04.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f275620c700 1 -- start start 2023-10-16T09:57:04.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f275620c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f275006ab40 0x7f2750155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f275620c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f275006e5d0 0x7f27500b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f275620c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27500729b0 0x7f27500ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f275620c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27500c57f0 con 0x7f275006e5d0 2023-10-16T09:57:04.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f275620c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27500c4b70 con 0x7f27500729b0 2023-10-16T09:57:04.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f275620c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27500c4e50 con 0x7f275006ab40 2023-10-16T09:57:04.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.088+0000 7f274effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f275006e5d0 0x7f27500b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f275006e5d0 0x7f27500b1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50682/0 (socket says 172.21.15.158:50682) 2023-10-16T09:57:04.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 -- 172.21.15.158:0/2919583387 learned_addr learned my addr 172.21.15.158:0/2919583387 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:04.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274f7fe700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f275006ab40 0x7f2750155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274ffff700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27500729b0 0x7f27500ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 -- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f275006ab40 msgr2=0x7f2750155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f275006ab40 0x7f2750155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 -- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27500729b0 msgr2=0x7f27500ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27500729b0 0x7f27500ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 -- 172.21.15.158:0/2919583387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27440094a0 con 0x7f275006e5d0 2023-10-16T09:57:04.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f274effd700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f275006e5d0 0x7f27500b1440 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f274400ebb0 tx=0x7f274400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:04.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f2737fff700 1 -- 172.21.15.158:0/2919583387 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274400c910 con 0x7f275006e5d0 2023-10-16T09:57:04.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f2737fff700 1 -- 172.21.15.158:0/2919583387 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274400ca70 con 0x7f275006e5d0 2023-10-16T09:57:04.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f2737fff700 1 -- 172.21.15.158:0/2919583387 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274400ce00 con 0x7f275006e5d0 2023-10-16T09:57:04.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2748009b30 con 0x7f275006e5d0 2023-10-16T09:57:04.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.089+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27500c5410 con 0x7f275006e5d0 2023-10-16T09:57:04.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.090+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27500c7430 con 0x7f275006e5d0 2023-10-16T09:57:04.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.091+0000 7f2737fff700 1 -- 172.21.15.158:0/2919583387 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2744013470 con 0x7f275006e5d0 2023-10-16T09:57:04.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.091+0000 7f2737fff700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f273c06b020 0x7f273c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.091+0000 7f2737fff700 1 -- 172.21.15.158:0/2919583387 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f2744089c20 con 0x7f275006e5d0 2023-10-16T09:57:04.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.091+0000 7f274f7fe700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f273c06b020 0x7f273c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.091+0000 7f274f7fe700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f273c06b020 0x7f273c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f27400016c0 tx=0x7f2740009040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:04.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.095+0000 7f2737fff700 1 -- 172.21.15.158:0/2919583387 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27440548f0 con 0x7f275006e5d0 2023-10-16T09:57:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.289+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27500681b0 con 0x7f275006e5d0 2023-10-16T09:57:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.290+0000 7f2737fff700 1 -- 172.21.15.158:0/2919583387 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17400 (secure 0 0 0) 0x7f2744017090 con 0x7f275006e5d0 2023-10-16T09:57:04.292 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:04.292 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":66,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:57:03.005431+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:57:01.416542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T09:57:01.962552+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:57:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f273c06b020 msgr2=0x7f273c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f273c06b020 0x7f273c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f27400016c0 tx=0x7f2740009040 comp rx=0 tx=0).stop 2023-10-16T09:57:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f275006e5d0 msgr2=0x7f27500b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f275006e5d0 0x7f27500b1440 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f274400ebb0 tx=0x7f274400ef70 comp rx=0 tx=0).stop 2023-10-16T09:57:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 shutdown_connections 2023-10-16T09:57:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f273c06b020 0x7f273c06d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27500729b0 0x7f27500ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f275006ab40 0x7f2750155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 --2- 172.21.15.158:0/2919583387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f275006e5d0 0x7f27500b1440 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.292+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 >> 172.21.15.158:0/2919583387 conn(0x7f27500d02b0 msgr2=0x7f275006d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.293+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 shutdown_connections 2023-10-16T09:57:04.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.293+0000 7f275620c700 1 -- 172.21.15.158:0/2919583387 wait complete. 2023-10-16T09:57:04.306 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T09:57:04.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.428+0000 7f9df045b700 1 Processor -- start 2023-10-16T09:57:04.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.428+0000 7f9df045b700 1 -- start start 2023-10-16T09:57:04.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.429+0000 7f9df045b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de80cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.429+0000 7f9df045b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9de80cf360 0x7f9de80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.429+0000 7f9df045b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80d1ee0 con 0x7f9de815daa0 2023-10-16T09:57:04.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.429+0000 7f9df045b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de80d40a0 con 0x7f9de80cf360 2023-10-16T09:57:04.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.429+0000 7f9df045b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80d12d0 con 0x7f9de815d6c0 2023-10-16T09:57:04.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9de80cf360 0x7f9de80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9de80cf360 0x7f9de80d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50692/0 (socket says 172.21.15.158:50692) 2023-10-16T09:57:04.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 -- 172.21.15.158:0/3204602812 learned_addr learned my addr 172.21.15.158:0/3204602812 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:04.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee1f7700 1 --2- 172.21.15.158:0/3204602812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de80cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 -- 172.21.15.158:0/3204602812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 msgr2=0x7f9de80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:04.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 --2- 172.21.15.158:0/3204602812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de80cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 -- 172.21.15.158:0/3204602812 >> v1:172.21.15.172:6789/0 conn(0x7f9de815daa0 legacy=0x7f9de80cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:04.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 -- 172.21.15.158:0/3204602812 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80d15b0 con 0x7f9de80cf360 2023-10-16T09:57:04.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9dee9f8700 1 --2- 172.21.15.158:0/3204602812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de80cf360 0x7f9de80d0470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009e50 tx=0x7f9de000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3920fed065ef17ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:04.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9ded1f5700 1 -- 172.21.15.158:0/3204602812 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de000e070 con 0x7f9de80cf360 2023-10-16T09:57:04.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9ded1f5700 1 -- 172.21.15.158:0/3204602812 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de0003a20 con 0x7f9de80cf360 2023-10-16T09:57:04.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.430+0000 7f9ded1f5700 1 -- 172.21.15.158:0/3204602812 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0003dd0 con 0x7f9de80cf360 2023-10-16T09:57:04.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 -- 172.21.15.158:0/3204602812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de80cf360 msgr2=0x7f9de80d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 --2- 172.21.15.158:0/3204602812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de80cf360 0x7f9de80d0470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009e50 tx=0x7f9de000b330 comp rx=0 tx=0).stop 2023-10-16T09:57:04.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 -- 172.21.15.158:0/3204602812 shutdown_connections 2023-10-16T09:57:04.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 --2- 172.21.15.158:0/3204602812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de80cf360 0x7f9de80d0470 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 --2- 172.21.15.158:0/3204602812 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de80cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 -- 172.21.15.158:0/3204602812 >> 172.21.15.158:0/3204602812 conn(0x7f9de809c940 msgr2=0x7f9de80d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:04.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 -- 172.21.15.158:0/3204602812 shutdown_connections 2023-10-16T09:57:04.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.431+0000 7f9df045b700 1 -- 172.21.15.158:0/3204602812 wait complete. 2023-10-16T09:57:04.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.432+0000 7f9df045b700 1 Processor -- start 2023-10-16T09:57:04.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.432+0000 7f9df045b700 1 -- start start 2023-10-16T09:57:04.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9de80cf360 0x7f9de80cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de815daa0 0x7f9de80ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de815e3f0 con 0x7f9de815daa0 2023-10-16T09:57:04.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81f2d00 con 0x7f9de80cf360 2023-10-16T09:57:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80b4080 con 0x7f9de815d6c0 2023-10-16T09:57:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de815daa0 0x7f9de80ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de815daa0 0x7f9de80ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50698/0 (socket says 172.21.15.158:50698) 2023-10-16T09:57:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 -- 172.21.15.158:0/2006371779 learned_addr learned my addr 172.21.15.158:0/2006371779 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:04.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee1f7700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9de80cf360 0x7f9de80cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9ded9f6700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 -- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 msgr2=0x7f9de815deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de815deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 -- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9de80cf360 msgr2=0x7f9de80cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:04.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9de80cf360 0x7f9de80cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 -- 172.21.15.158:0/2006371779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de0009b30 con 0x7f9de815daa0 2023-10-16T09:57:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee1f7700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9de80cf360 0x7f9de80cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9dee9f8700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de815daa0 0x7f9de80ccb90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9de000beb0 tx=0x7f9de0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9ddeffd700 1 -- 172.21.15.158:0/2006371779 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de00042c0 con 0x7f9de815daa0 2023-10-16T09:57:04.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9ddeffd700 1 -- 172.21.15.158:0/2006371779 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de0004420 con 0x7f9de815daa0 2023-10-16T09:57:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de815e600 con 0x7f9de815daa0 2023-10-16T09:57:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9ddeffd700 1 -- 172.21.15.158:0/2006371779 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de00047d0 con 0x7f9de815daa0 2023-10-16T09:57:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.433+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de8160a40 con 0x7f9de815daa0 2023-10-16T09:57:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.434+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de80681b0 con 0x7f9de815daa0 2023-10-16T09:57:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.435+0000 7f9ddeffd700 1 -- 172.21.15.158:0/2006371779 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9de0019070 con 0x7f9de815daa0 2023-10-16T09:57:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.435+0000 7f9ddeffd700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9dcc06b020 0x7f9dcc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.435+0000 7f9ddeffd700 1 -- 172.21.15.158:0/2006371779 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f9de008aa90 con 0x7f9de815daa0 2023-10-16T09:57:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.435+0000 7f9dee1f7700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9dcc06b020 0x7f9dcc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.436+0000 7f9dee1f7700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9dcc06b020 0x7f9dcc06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800aa50 tx=0x7f9dd8008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:04.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.440+0000 7f9ddeffd700 1 -- 172.21.15.158:0/2006371779 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9de00556e0 con 0x7f9de815daa0 2023-10-16T09:57:04.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:04.680+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9de80b41c0 con 0x7f9de815daa0 2023-10-16T09:57:05.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.616+0000 7f9ddeffd700 1 -- 172.21.15.158:0/2006371779 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9de0020020 con 0x7f9de815daa0 2023-10-16T09:57:05.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9dcc06b020 msgr2=0x7f9dcc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:05.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9dcc06b020 0x7f9dcc06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800aa50 tx=0x7f9dd8008040 comp rx=0 tx=0).stop 2023-10-16T09:57:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de815daa0 msgr2=0x7f9de80ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de815daa0 0x7f9de80ccb90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9de000beb0 tx=0x7f9de0004ad0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 shutdown_connections 2023-10-16T09:57:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9dcc06b020 0x7f9dcc06d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9de815daa0 0x7f9de80ccb90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9de80cf360 0x7f9de80cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 --2- 172.21.15.158:0/2006371779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9de815d6c0 0x7f9de815deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 >> 172.21.15.158:0/2006371779 conn(0x7f9de809c940 msgr2=0x7f9de80b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 shutdown_connections 2023-10-16T09:57:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.619+0000 7f9df045b700 1 -- 172.21.15.158:0/2006371779 wait complete. 2023-10-16T09:57:05.635 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T09:57:05.635 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T09:57:05.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.763+0000 7f1016349700 1 Processor -- start 2023-10-16T09:57:05.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.763+0000 7f1016349700 1 -- start start 2023-10-16T09:57:05.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f1016349700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f101015b6b0 0x7f10100cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:05.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f1016349700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10100d0c60 0x7f10100b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:05.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f1016349700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100b4420 con 0x7f10100b3460 2023-10-16T09:57:05.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f1016349700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101015ba90 con 0x7f101015b6b0 2023-10-16T09:57:05.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f1016349700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10100b38f0 con 0x7f10100d0c60 2023-10-16T09:57:05.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f100ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f101015b6b0 0x7f10100cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:05.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f100ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f101015b6b0 0x7f10100cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50702/0 (socket says 172.21.15.158:50702) 2023-10-16T09:57:05.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f100ffff700 1 -- 172.21.15.158:0/3638354429 learned_addr learned my addr 172.21.15.158:0/3638354429 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:05.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.764+0000 7f10148e6700 1 --1- 172.21.15.158:0/3638354429 >> v1:172.21.15.158:6789/0 conn(0x7f10100b3460 0x7f1010070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50882/0 (socket says 172.21.15.158:50882) 2023-10-16T09:57:05.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100f7fe700 1 --2- 172.21.15.158:0/3638354429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10100d0c60 0x7f10100b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:05.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100ffff700 1 -- 172.21.15.158:0/3638354429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10100d0c60 msgr2=0x7f10100b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:05.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100ffff700 1 --2- 172.21.15.158:0/3638354429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10100d0c60 0x7f10100b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100ffff700 1 -- 172.21.15.158:0/3638354429 >> v1:172.21.15.158:6789/0 conn(0x7f10100b3460 legacy=0x7f1010070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:05.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100ffff700 1 -- 172.21.15.158:0/3638354429 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10100cbc20 con 0x7f101015b6b0 2023-10-16T09:57:05.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100effd700 1 -- 172.21.15.158:0/3638354429 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012163145 0 0) 0x7f10100b4420 con 0x7f10100b3460 2023-10-16T09:57:05.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100f7fe700 1 --2- 172.21.15.158:0/3638354429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10100d0c60 0x7f10100b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:05.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100ffff700 1 --2- 172.21.15.158:0/3638354429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f101015b6b0 0x7f10100cfdb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80098f0 tx=0x7f0ff8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bae2a48c02131a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:05.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100effd700 1 -- 172.21.15.158:0/3638354429 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff800b8a0 con 0x7f101015b6b0 2023-10-16T09:57:05.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100effd700 1 -- 172.21.15.158:0/3638354429 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff800ba00 con 0x7f101015b6b0 2023-10-16T09:57:05.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.765+0000 7f100effd700 1 -- 172.21.15.158:0/3638354429 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff800bd90 con 0x7f101015b6b0 2023-10-16T09:57:05.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 -- 172.21.15.158:0/3638354429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f101015b6b0 msgr2=0x7f10100cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:05.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 --2- 172.21.15.158:0/3638354429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f101015b6b0 0x7f10100cfdb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80098f0 tx=0x7f0ff8009c00 comp rx=0 tx=0).stop 2023-10-16T09:57:05.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 -- 172.21.15.158:0/3638354429 shutdown_connections 2023-10-16T09:57:05.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 --2- 172.21.15.158:0/3638354429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f101015b6b0 0x7f10100cfdb0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 --2- 172.21.15.158:0/3638354429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10100d0c60 0x7f10100b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 -- 172.21.15.158:0/3638354429 >> 172.21.15.158:0/3638354429 conn(0x7f10100cdf20 msgr2=0x7f10100ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:05.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 -- 172.21.15.158:0/3638354429 shutdown_connections 2023-10-16T09:57:05.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.766+0000 7f1016349700 1 -- 172.21.15.158:0/3638354429 wait complete. 2023-10-16T09:57:05.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.767+0000 7f1016349700 1 Processor -- start 2023-10-16T09:57:05.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.767+0000 7f1016349700 1 -- start start 2023-10-16T09:57:05.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10100b3460 0x7f10101f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:05.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10100d0c60 0x7f10101f6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:05.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f101015b6b0 0x7f10100ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:05.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10100aeab0 con 0x7f10100b3460 2023-10-16T09:57:05.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10100adea0 con 0x7f10100d0c60 2023-10-16T09:57:05.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10100ae180 con 0x7f101015b6b0 2023-10-16T09:57:05.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10100b3460 0x7f10101f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:05.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10100b3460 0x7f10101f5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50708/0 (socket says 172.21.15.158:50708) 2023-10-16T09:57:05.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 -- 172.21.15.158:0/537281751 learned_addr learned my addr 172.21.15.158:0/537281751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:05.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f10148e6700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f101015b6b0 0x7f10100ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:05.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100f7fe700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10100d0c60 0x7f10101f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:05.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 -- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f101015b6b0 msgr2=0x7f10100ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:05.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f101015b6b0 0x7f10100ad0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 -- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10100d0c60 msgr2=0x7f10101f6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:05.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10100d0c60 0x7f10101f6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:05.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 -- 172.21.15.158:0/537281751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff8009550 con 0x7f10100b3460 2023-10-16T09:57:05.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100ffff700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10100b3460 0x7f10101f5ff0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009c40 tx=0x7f0ff8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:05.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100cff9700 1 -- 172.21.15.158:0/537281751 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80045d0 con 0x7f10100b3460 2023-10-16T09:57:05.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100cff9700 1 -- 172.21.15.158:0/537281751 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff801e070 con 0x7f10100b3460 2023-10-16T09:57:05.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f100cff9700 1 -- 172.21.15.158:0/537281751 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80038c0 con 0x7f10100b3460 2023-10-16T09:57:05.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10100ae460 con 0x7f10100b3460 2023-10-16T09:57:05.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.768+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10101f2690 con 0x7f10100b3460 2023-10-16T09:57:05.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.769+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10100681b0 con 0x7f10100b3460 2023-10-16T09:57:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.770+0000 7f100cff9700 1 -- 172.21.15.158:0/537281751 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0ff8003a20 con 0x7f10100b3460 2023-10-16T09:57:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.770+0000 7f100cff9700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f0ff006b020 0x7f0ff006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.770+0000 7f100cff9700 1 -- 172.21.15.158:0/537281751 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f0ff8020070 con 0x7f10100b3460 2023-10-16T09:57:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.770+0000 7f100f7fe700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f0ff006b020 0x7f0ff006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.771+0000 7f100f7fe700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f0ff006b020 0x7f0ff006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f10100aca40 tx=0x7f1000008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:05.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:05.775+0000 7f100cff9700 1 -- 172.21.15.158:0/537281751 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ff8055cd0 con 0x7f10100b3460 2023-10-16T09:57:06.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.012+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f10100acdc0 con 0x7f10100b3460 2023-10-16T09:57:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.623+0000 7f100cff9700 1 -- 172.21.15.158:0/537281751 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ff802a0c0 con 0x7f10100b3460 2023-10-16T09:57:06.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.625+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f0ff006b020 msgr2=0x7f0ff006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:06.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.625+0000 7f1016349700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f0ff006b020 0x7f0ff006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f10100aca40 tx=0x7f1000008040 comp rx=0 tx=0).stop 2023-10-16T09:57:06.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10100b3460 msgr2=0x7f10101f5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:06.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10100b3460 0x7f10101f5ff0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009c40 tx=0x7f0ff8004750 comp rx=0 tx=0).stop 2023-10-16T09:57:06.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 shutdown_connections 2023-10-16T09:57:06.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f0ff006b020 0x7f0ff006d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f101015b6b0 0x7f10100ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10100b3460 0x7f10101f5ff0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 --2- 172.21.15.158:0/537281751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10100d0c60 0x7f10101f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 >> 172.21.15.158:0/537281751 conn(0x7f10100cdf20 msgr2=0x7f10100b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:06.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 shutdown_connections 2023-10-16T09:57:06.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.626+0000 7f1016349700 1 -- 172.21.15.158:0/537281751 wait complete. 2023-10-16T09:57:06.641 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T09:57:06.641 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:57:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.762+0000 7f7b96d5e700 1 Processor -- start 2023-10-16T09:57:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.762+0000 7f7b96d5e700 1 -- start start 2023-10-16T09:57:06.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b96d5e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7b9006ab40 0x7f7b9006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b96d5e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b900729b0 0x7f7b90071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b96d5e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b90071e50 con 0x7f7b9006e5d0 2023-10-16T09:57:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b96d5e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b90161270 con 0x7f7b9006ab40 2023-10-16T09:57:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b96d5e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b90161510 con 0x7f7b900729b0 2023-10-16T09:57:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b94afa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7b9006ab40 0x7f7b9006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b94afa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7b9006ab40 0x7f7b9006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50716/0 (socket says 172.21.15.158:50716) 2023-10-16T09:57:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.763+0000 7f7b94afa700 1 -- 172.21.15.158:0/2872522288 learned_addr learned my addr 172.21.15.158:0/2872522288 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b952fb700 1 --2- 172.21.15.158:0/2872522288 >> v2:172.21.15.172:3300/0 conn(0x7f7b900729b0 0x7f7b90071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b94afa700 1 -- 172.21.15.158:0/2872522288 >> v2:172.21.15.172:3300/0 conn(0x7f7b900729b0 msgr2=0x7f7b90071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b94afa700 1 --2- 172.21.15.158:0/2872522288 >> v2:172.21.15.172:3300/0 conn(0x7f7b900729b0 0x7f7b90071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b94afa700 1 -- 172.21.15.158:0/2872522288 >> v1:172.21.15.172:6789/0 conn(0x7f7b9006e5d0 legacy=0x7f7b900722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b94afa700 1 -- 172.21.15.158:0/2872522288 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b901617f0 con 0x7f7b9006ab40 2023-10-16T09:57:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b94afa700 1 --2- 172.21.15.158:0/2872522288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b9006ab40 0x7f7b9006d820 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7b90158240 tx=0x7f7b84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7304207c5c04979 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b8f7fe700 1 -- 172.21.15.158:0/2872522288 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84004230 con 0x7f7b9006ab40 2023-10-16T09:57:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b8f7fe700 1 -- 172.21.15.158:0/2872522288 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b84004390 con 0x7f7b9006ab40 2023-10-16T09:57:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.764+0000 7f7b8f7fe700 1 -- 172.21.15.158:0/2872522288 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840036a0 con 0x7f7b9006ab40 2023-10-16T09:57:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 -- 172.21.15.158:0/2872522288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b9006ab40 msgr2=0x7f7b9006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/2872522288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b9006ab40 0x7f7b9006d820 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7b90158240 tx=0x7f7b84009bd0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 -- 172.21.15.158:0/2872522288 shutdown_connections 2023-10-16T09:57:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/2872522288 >> v2:172.21.15.172:3300/0 conn(0x7f7b900729b0 0x7f7b90071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/2872522288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b9006ab40 0x7f7b9006d820 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 -- 172.21.15.158:0/2872522288 >> 172.21.15.158:0/2872522288 conn(0x7f7b900d02b0 msgr2=0x7f7b900d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 -- 172.21.15.158:0/2872522288 shutdown_connections 2023-10-16T09:57:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.765+0000 7f7b96d5e700 1 -- 172.21.15.158:0/2872522288 wait complete. 2023-10-16T09:57:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.766+0000 7f7b96d5e700 1 Processor -- start 2023-10-16T09:57:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.766+0000 7f7b96d5e700 1 -- start start 2023-10-16T09:57:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b96d5e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b9006ab40 0x7f7b900b2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b96d5e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9006e5d0 0x7f7b900c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b96d5e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b900729b0 0x7f7b900c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b96d5e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015e790 con 0x7f7b900729b0 2023-10-16T09:57:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b96d5e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015ea70 con 0x7f7b9006e5d0 2023-10-16T09:57:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b96d5e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015ed50 con 0x7f7b9006ab40 2023-10-16T09:57:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b952fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b900729b0 0x7f7b900c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b952fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b900729b0 0x7f7b900c8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50726/0 (socket says 172.21.15.158:50726) 2023-10-16T09:57:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.767+0000 7f7b952fb700 1 -- 172.21.15.158:0/1539979969 learned_addr learned my addr 172.21.15.158:0/1539979969 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b94afa700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b9006ab40 0x7f7b900b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b8ffff700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9006e5d0 0x7f7b900c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b952fb700 1 -- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b9006ab40 msgr2=0x7f7b900b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b952fb700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b9006ab40 0x7f7b900b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b952fb700 1 -- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9006e5d0 msgr2=0x7f7b900c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b952fb700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9006e5d0 0x7f7b900c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b952fb700 1 -- 172.21.15.158:0/1539979969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84009550 con 0x7f7b900729b0 2023-10-16T09:57:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b952fb700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b900729b0 0x7f7b900c8eb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f7b88007e30 tx=0x7f7b8800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b8d7fa700 1 -- 172.21.15.158:0/1539979969 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800eea0 con 0x7f7b900729b0 2023-10-16T09:57:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b8d7fa700 1 -- 172.21.15.158:0/1539979969 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88014420 con 0x7f7b900729b0 2023-10-16T09:57:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9015f030 con 0x7f7b900729b0 2023-10-16T09:57:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b8d7fa700 1 -- 172.21.15.158:0/1539979969 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b880147d0 con 0x7f7b900729b0 2023-10-16T09:57:06.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.768+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b900b0a50 con 0x7f7b900729b0 2023-10-16T09:57:06.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.769+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9015e8d0 con 0x7f7b900729b0 2023-10-16T09:57:06.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.769+0000 7f7b8d7fa700 1 -- 172.21.15.158:0/1539979969 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b88014930 con 0x7f7b900729b0 2023-10-16T09:57:06.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.770+0000 7f7b8d7fa700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7b7406b020 0x7f7b7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:06.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.770+0000 7f7b8d7fa700 1 -- 172.21.15.158:0/1539979969 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7b8808a2b0 con 0x7f7b900729b0 2023-10-16T09:57:06.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.770+0000 7f7b94afa700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7b7406b020 0x7f7b7406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:06.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.770+0000 7f7b94afa700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7b7406b020 0x7f7b7406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009130 tx=0x7f7b84008030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:06.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:06.774+0000 7f7b8d7fa700 1 -- 172.21.15.158:0/1539979969 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b9015e8d0 con 0x7f7b900729b0 2023-10-16T09:57:07.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.012+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b9015f270 con 0x7f7b900729b0 2023-10-16T09:57:07.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.012+0000 7f7b8d7fa700 1 -- 172.21.15.158:0/1539979969 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f7b9015f270 con 0x7f7b900729b0 2023-10-16T09:57:07.014 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:07.015 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4264509745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4264509745},{"type":"v1","addr":"172.21.15.158:6835","nonce":4264509745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1091298830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1091298830},{"type":"v1","addr":"172.21.15.158:6837","nonce":1091298830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":35261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2630165052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2630165052},{"type":"v1","addr":"172.21.15.172:6834","nonce":2630165052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:57:03.005758+0000","modified":"2023-10-16T09:57:06.014112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35264":{"gid":35264,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1262687320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1262687320},{"type":"v1","addr":"172.21.15.172:6835","nonce":1262687320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T09:57:07.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7b7406b020 msgr2=0x7f7b7406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7b7406b020 0x7f7b7406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009130 tx=0x7f7b84008030 comp rx=0 tx=0).stop 2023-10-16T09:57:07.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b900729b0 msgr2=0x7f7b900c8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b900729b0 0x7f7b900c8eb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f7b88007e30 tx=0x7f7b8800e3c0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 shutdown_connections 2023-10-16T09:57:07.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7b7406b020 0x7f7b7406d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b900729b0 0x7f7b900c8eb0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b9006ab40 0x7f7b900b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 --2- 172.21.15.158:0/1539979969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9006e5d0 0x7f7b900c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 >> 172.21.15.158:0/1539979969 conn(0x7f7b900d02b0 msgr2=0x7f7b900ac920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:07.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 shutdown_connections 2023-10-16T09:57:07.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.015+0000 7f7b96d5e700 1 -- 172.21.15.158:0/1539979969 wait complete. 2023-10-16T09:57:07.019 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 25 2023-10-16T09:57:07.028 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:57:07.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.147+0000 7f75c9d03700 1 Processor -- start 2023-10-16T09:57:07.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.147+0000 7f75c9d03700 1 -- start start 2023-10-16T09:57:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c9d03700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f75c406e5d0 0x7f75c406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c9d03700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c40729b0 0x7f75c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c9d03700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c4071e50 con 0x7f75c406ab40 2023-10-16T09:57:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c9d03700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c4161270 con 0x7f75c406e5d0 2023-10-16T09:57:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c9d03700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4161510 con 0x7f75c40729b0 2023-10-16T09:57:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c37fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f75c406ab40 0x7f75c40722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50906/0 (socket says 172.21.15.158:50906) 2023-10-16T09:57:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c37fe700 1 -- 172.21.15.158:0/1714644038 learned_addr learned my addr 172.21.15.158:0/1714644038 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c3fff700 1 --2- 172.21.15.158:0/1714644038 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c40729b0 0x7f75c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.148+0000 7f75c2ffd700 1 --2- 172.21.15.158:0/1714644038 >> v2:172.21.15.172:3300/0 conn(0x7f75c406e5d0 0x7f75c406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c27fc700 1 -- 172.21.15.158:0/1714644038 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171056275 0 0) 0x7f75c4071e50 con 0x7f75c406ab40 2023-10-16T09:57:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c27fc700 1 -- 172.21.15.158:0/1714644038 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b00035b0 con 0x7f75c406ab40 2023-10-16T09:57:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c3fff700 1 -- 172.21.15.158:0/1714644038 >> v2:172.21.15.172:3300/0 conn(0x7f75c406e5d0 msgr2=0x7f75c406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c3fff700 1 --2- 172.21.15.158:0/1714644038 >> v2:172.21.15.172:3300/0 conn(0x7f75c406e5d0 0x7f75c406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c3fff700 1 -- 172.21.15.158:0/1714644038 >> v1:172.21.15.158:6789/0 conn(0x7f75c406ab40 legacy=0x7f75c40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c3fff700 1 -- 172.21.15.158:0/1714644038 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c41617f0 con 0x7f75c40729b0 2023-10-16T09:57:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c3fff700 1 --2- 172.21.15.158:0/1714644038 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c40729b0 0x7f75c4071910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009d90 tx=0x7f75b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d228b8601621d3cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c27fc700 1 -- 172.21.15.158:0/1714644038 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b400e070 con 0x7f75c40729b0 2023-10-16T09:57:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c27fc700 1 -- 172.21.15.158:0/1714644038 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b400bb20 con 0x7f75c40729b0 2023-10-16T09:57:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.149+0000 7f75c27fc700 1 -- 172.21.15.158:0/1714644038 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4012430 con 0x7f75c40729b0 2023-10-16T09:57:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 -- 172.21.15.158:0/1714644038 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c40729b0 msgr2=0x7f75c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 --2- 172.21.15.158:0/1714644038 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c40729b0 0x7f75c4071910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009d90 tx=0x7f75b400b330 comp rx=0 tx=0).stop 2023-10-16T09:57:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 -- 172.21.15.158:0/1714644038 shutdown_connections 2023-10-16T09:57:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 --2- 172.21.15.158:0/1714644038 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c40729b0 0x7f75c4071910 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 --2- 172.21.15.158:0/1714644038 >> v2:172.21.15.172:3300/0 conn(0x7f75c406e5d0 0x7f75c406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 -- 172.21.15.158:0/1714644038 >> 172.21.15.158:0/1714644038 conn(0x7f75c40d02b0 msgr2=0x7f75c40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 -- 172.21.15.158:0/1714644038 shutdown_connections 2023-10-16T09:57:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.150+0000 7f75c9d03700 1 -- 172.21.15.158:0/1714644038 wait complete. 2023-10-16T09:57:07.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.151+0000 7f75c9d03700 1 Processor -- start 2023-10-16T09:57:07.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.151+0000 7f75c9d03700 1 -- start start 2023-10-16T09:57:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c9d03700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75c406ab40 0x7f75c40c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c9d03700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c406e5d0 0x7f75c4152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c9d03700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c40729b0 0x7f75c40b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c9d03700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c415de20 con 0x7f75c40729b0 2023-10-16T09:57:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c9d03700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40b20e0 con 0x7f75c406ab40 2023-10-16T09:57:07.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c9d03700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40b1560 con 0x7f75c406e5d0 2023-10-16T09:57:07.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c40729b0 0x7f75c40b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c40729b0 0x7f75c40b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50738/0 (socket says 172.21.15.158:50738) 2023-10-16T09:57:07.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c37fe700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75c406ab40 0x7f75c40c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 -- 172.21.15.158:0/963582290 learned_addr learned my addr 172.21.15.158:0/963582290 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c2ffd700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c406e5d0 0x7f75c4152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 -- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c406e5d0 msgr2=0x7f75c4152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c406e5d0 0x7f75c4152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 -- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75c406ab40 msgr2=0x7f75c40c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75c406ab40 0x7f75c40c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.152+0000 7f75c3fff700 1 -- 172.21.15.158:0/963582290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b4009a70 con 0x7f75c40729b0 2023-10-16T09:57:07.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.153+0000 7f75c3fff700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c40729b0 0x7f75c40b0d10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f75b4014040 tx=0x7f75b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.153+0000 7f75bbfff700 1 -- 172.21.15.158:0/963582290 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4012430 con 0x7f75c40729b0 2023-10-16T09:57:07.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.153+0000 7f75bbfff700 1 -- 172.21.15.158:0/963582290 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b4020070 con 0x7f75c40729b0 2023-10-16T09:57:07.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.153+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c40b1840 con 0x7f75c40729b0 2023-10-16T09:57:07.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.153+0000 7f75bbfff700 1 -- 172.21.15.158:0/963582290 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4004280 con 0x7f75c40729b0 2023-10-16T09:57:07.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.153+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c40b1e00 con 0x7f75c40729b0 2023-10-16T09:57:07.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.154+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c415f880 con 0x7f75c40729b0 2023-10-16T09:57:07.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.154+0000 7f75bbfff700 1 -- 172.21.15.158:0/963582290 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75b40043e0 con 0x7f75c40729b0 2023-10-16T09:57:07.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.154+0000 7f75bbfff700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f75a406b020 0x7f75a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.155+0000 7f75bbfff700 1 -- 172.21.15.158:0/963582290 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f75b4022070 con 0x7f75c40729b0 2023-10-16T09:57:07.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.155+0000 7f75c37fe700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f75a406b020 0x7f75a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.155+0000 7f75c37fe700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f75a406b020 0x7f75a406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f75bc007fd0 tx=0x7f75bc009040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.159+0000 7f75bbfff700 1 -- 172.21.15.158:0/963582290 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75c415f880 con 0x7f75c40729b0 2023-10-16T09:57:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.353+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75c415f880 con 0x7f75c40729b0 2023-10-16T09:57:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.354+0000 7f75bbfff700 1 -- 172.21.15.158:0/963582290 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+17387 (secure 0 0 0) 0x7f75c415f880 con 0x7f75c40729b0 2023-10-16T09:57:07.356 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:07.356 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":67,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:57:04.524350+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:57:01.416542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T09:57:01.962552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:57:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f75a406b020 msgr2=0x7f75a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f75a406b020 0x7f75a406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f75bc007fd0 tx=0x7f75bc009040 comp rx=0 tx=0).stop 2023-10-16T09:57:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c40729b0 msgr2=0x7f75c40b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c40729b0 0x7f75c40b0d10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f75b4014040 tx=0x7f75b4004ad0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 shutdown_connections 2023-10-16T09:57:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f75a406b020 0x7f75a406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c40729b0 0x7f75c40b0d10 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75c406ab40 0x7f75c40c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 --2- 172.21.15.158:0/963582290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75c406e5d0 0x7f75c4152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 >> 172.21.15.158:0/963582290 conn(0x7f75c40d02b0 msgr2=0x7f75c4072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 shutdown_connections 2023-10-16T09:57:07.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.357+0000 7f75c9d03700 1 -- 172.21.15.158:0/963582290 wait complete. 2023-10-16T09:57:07.371 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T09:57:07.371 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:57:07.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.490+0000 7f6333f92700 1 Processor -- start 2023-10-16T09:57:07.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.491+0000 7f6333f92700 1 -- start start 2023-10-16T09:57:07.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.491+0000 7f6333f92700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f632c06ab40 0x7f632c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.491+0000 7f6333f92700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f632c06e5d0 0x7f632c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.491+0000 7f6333f92700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c071e50 con 0x7f632c06af20 2023-10-16T09:57:07.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.491+0000 7f6333f92700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c161270 con 0x7f632c06e5d0 2023-10-16T09:57:07.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.491+0000 7f6333f92700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c161510 con 0x7f632c06ab40 2023-10-16T09:57:07.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633152d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f632c06e5d0 0x7f632c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633252f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f632c06af20 0x7f632c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50920/0 (socket says 172.21.15.158:50920) 2023-10-16T09:57:07.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633252f700 1 -- 172.21.15.158:0/3501670560 learned_addr learned my addr 172.21.15.158:0/3501670560 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:07.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633152d700 1 -- 172.21.15.158:0/3501670560 >> v2:172.21.15.172:3300/0 conn(0x7f632c06ab40 msgr2=0x7f632c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:07.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f6331d2e700 1 --2- 172.21.15.158:0/3501670560 >> v2:172.21.15.172:3300/0 conn(0x7f632c06ab40 0x7f632c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f6330d2c700 1 -- 172.21.15.158:0/3501670560 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783371390 0 0) 0x7f632c071e50 con 0x7f632c06af20 2023-10-16T09:57:07.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633152d700 1 --2- 172.21.15.158:0/3501670560 >> v2:172.21.15.172:3300/0 conn(0x7f632c06ab40 0x7f632c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633152d700 1 -- 172.21.15.158:0/3501670560 >> v1:172.21.15.158:6789/0 conn(0x7f632c06af20 legacy=0x7f632c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633152d700 1 -- 172.21.15.158:0/3501670560 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c1617f0 con 0x7f632c06e5d0 2023-10-16T09:57:07.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f6331d2e700 1 --2- 172.21.15.158:0/3501670560 >> v2:172.21.15.172:3300/0 conn(0x7f632c06ab40 0x7f632c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:57:07.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f633152d700 1 --2- 172.21.15.158:0/3501670560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c071910 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f632c074d70 tx=0x7f6328009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0d9b0606453efd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f6330d2c700 1 -- 172.21.15.158:0/3501670560 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328011830 con 0x7f632c06e5d0 2023-10-16T09:57:07.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f6330d2c700 1 -- 172.21.15.158:0/3501670560 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6328011990 con 0x7f632c06e5d0 2023-10-16T09:57:07.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.492+0000 7f6330d2c700 1 -- 172.21.15.158:0/3501670560 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328011d20 con 0x7f632c06e5d0 2023-10-16T09:57:07.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 -- 172.21.15.158:0/3501670560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 msgr2=0x7f632c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 --2- 172.21.15.158:0/3501670560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c071910 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f632c074d70 tx=0x7f6328009b20 comp rx=0 tx=0).stop 2023-10-16T09:57:07.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 -- 172.21.15.158:0/3501670560 shutdown_connections 2023-10-16T09:57:07.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 --2- 172.21.15.158:0/3501670560 >> v2:172.21.15.172:3300/0 conn(0x7f632c06ab40 0x7f632c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 --2- 172.21.15.158:0/3501670560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c071910 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 -- 172.21.15.158:0/3501670560 >> 172.21.15.158:0/3501670560 conn(0x7f632c0d02b0 msgr2=0x7f632c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:07.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 -- 172.21.15.158:0/3501670560 shutdown_connections 2023-10-16T09:57:07.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.493+0000 7f6333f92700 1 -- 172.21.15.158:0/3501670560 wait complete. 2023-10-16T09:57:07.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.494+0000 7f6333f92700 1 Processor -- start 2023-10-16T09:57:07.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.494+0000 7f6333f92700 1 -- start start 2023-10-16T09:57:07.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6333f92700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f632c06ab40 0x7f632c0c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6333f92700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f632c06af20 0x7f632c0b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6333f92700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c1f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6333f92700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0c5a20 con 0x7f632c06e5d0 2023-10-16T09:57:07.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6333f92700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0ca430 con 0x7f632c06ab40 2023-10-16T09:57:07.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6333f92700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0c9220 con 0x7f632c06af20 2023-10-16T09:57:07.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c1f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c1f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50750/0 (socket says 172.21.15.158:50750) 2023-10-16T09:57:07.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 -- 172.21.15.158:0/854706842 learned_addr learned my addr 172.21.15.158:0/854706842 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:07.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6331d2e700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f632c06ab40 0x7f632c0c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633152d700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f632c06af20 0x7f632c0b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 -- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f632c06af20 msgr2=0x7f632c0b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f632c06af20 0x7f632c0b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 -- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f632c06ab40 msgr2=0x7f632c0c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f632c06ab40 0x7f632c0c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 -- 172.21.15.158:0/854706842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63280094a0 con 0x7f632c06e5d0 2023-10-16T09:57:07.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f633252f700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c1f2620 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f632400cc70 tx=0x7f632400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f63227fc700 1 -- 172.21.15.158:0/854706842 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632400ada0 con 0x7f632c06e5d0 2023-10-16T09:57:07.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f63227fc700 1 -- 172.21.15.158:0/854706842 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632400ce70 con 0x7f632c06e5d0 2023-10-16T09:57:07.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c0c94c0 con 0x7f632c06e5d0 2023-10-16T09:57:07.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.495+0000 7f63227fc700 1 -- 172.21.15.158:0/854706842 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63240072f0 con 0x7f632c06e5d0 2023-10-16T09:57:07.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.496+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c1f2b60 con 0x7f632c06e5d0 2023-10-16T09:57:07.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.497+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c0681b0 con 0x7f632c06e5d0 2023-10-16T09:57:07.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.497+0000 7f63227fc700 1 -- 172.21.15.158:0/854706842 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6324007450 con 0x7f632c06e5d0 2023-10-16T09:57:07.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.497+0000 7f63227fc700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f631006b020 0x7f631006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.497+0000 7f63227fc700 1 -- 172.21.15.158:0/854706842 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f6324089880 con 0x7f632c06e5d0 2023-10-16T09:57:07.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.497+0000 7f6331d2e700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f631006b020 0x7f631006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.498+0000 7f6331d2e700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f631006b020 0x7f631006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f631c006fd0 tx=0x7f631c008040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.502+0000 7f63227fc700 1 -- 172.21.15.158:0/854706842 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63240545e0 con 0x7f632c06e5d0 2023-10-16T09:57:07.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.689+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 --> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f632c1f2d50 con 0x7f631006b020 2023-10-16T09:57:07.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.698+0000 7f63227fc700 1 -- 172.21.15.158:0/854706842 <== mgr.6009 v2:172.21.15.158:6832/211595975 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288907 (secure 0 0 0) 0x7f631c00c050 con 0x7f631006b020 2023-10-16T09:57:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:07.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f631006b020 msgr2=0x7f631006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f631006b020 0x7f631006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f631c006fd0 tx=0x7f631c008040 comp rx=0 tx=0).stop 2023-10-16T09:57:07.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 msgr2=0x7f632c1f2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c1f2620 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f632400cc70 tx=0x7f632400a340 comp rx=0 tx=0).stop 2023-10-16T09:57:07.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 shutdown_connections 2023-10-16T09:57:07.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f631006b020 0x7f631006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f632c06e5d0 0x7f632c1f2620 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f632c06ab40 0x7f632c0c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 --2- 172.21.15.158:0/854706842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f632c06af20 0x7f632c0b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 >> 172.21.15.158:0/854706842 conn(0x7f632c0d02b0 msgr2=0x7f632c0af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:07.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 shutdown_connections 2023-10-16T09:57:07.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.701+0000 7f6333f92700 1 -- 172.21.15.158:0/854706842 wait complete. 2023-10-16T09:57:07.708 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:57:07.714 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T09:57:06.054954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315052,"kb_used_data":7288,"kb_used_omap":12,"kb_used_meta":307635,"kb_avail":754659668,"statfs":{"total":773094113280,"available":772771500032,"internally_reserved":0,"allocated":7462912,"data_stored":4293940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":315018807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.462661"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545189+0000","last_change":"2023-10-16T09:57:03.544690+0000","last_active":"2023-10-16T09:57:03.545189+0000","last_peered":"2023-10-16T09:57:03.545189+0000","last_clean":"2023-10-16T09:57:03.545189+0000","last_became_active":"2023-10-16T09:57:03.544417+0000","last_became_peered":"2023-10-16T09:57:03.544417+0000","last_unstale":"2023-10-16T09:57:03.545189+0000","last_undegraded":"2023-10-16T09:57:03.545189+0000","last_fullsized":"2023-10-16T09:57:03.545189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:15.106362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545803+0000","last_change":"2023-10-16T09:57:03.544520+0000","last_active":"2023-10-16T09:57:03.545803+0000","last_peered":"2023-10-16T09:57:03.545803+0000","last_clean":"2023-10-16T09:57:03.545803+0000","last_became_active":"2023-10-16T09:57:03.544248+0000","last_became_peered":"2023-10-16T09:57:03.544248+0000","last_unstale":"2023-10-16T09:57:03.545803+0000","last_undegraded":"2023-10-16T09:57:03.545803+0000","last_fullsized":"2023-10-16T09:57:03.545803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:42.266310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545041+0000","last_change":"2023-10-16T09:57:03.539579+0000","last_active":"2023-10-16T09:57:03.545041+0000","last_peered":"2023-10-16T09:57:03.545041+0000","last_clean":"2023-10-16T09:57:03.545041+0000","last_became_active":"2023-10-16T09:57:03.537238+0000","last_became_peered":"2023-10-16T09:57:03.537238+0000","last_unstale":"2023-10-16T09:57:03.545041+0000","last_undegraded":"2023-10-16T09:57:03.545041+0000","last_fullsized":"2023-10-16T09:57:03.545041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:22.204911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543498+0000","last_change":"2023-10-16T09:57:03.541526+0000","last_active":"2023-10-16T09:57:03.543498+0000","last_peered":"2023-10-16T09:57:03.543498+0000","last_clean":"2023-10-16T09:57:03.543498+0000","last_became_active":"2023-10-16T09:57:03.541262+0000","last_became_peered":"2023-10-16T09:57:03.541262+0000","last_unstale":"2023-10-16T09:57:03.543498+0000","last_undegraded":"2023-10-16T09:57:03.543498+0000","last_fullsized":"2023-10-16T09:57:03.543498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:34:42.033143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.538472+0000","last_change":"2023-10-16T09:57:03.533867+0000","last_active":"2023-10-16T09:57:03.538472+0000","last_peered":"2023-10-16T09:57:03.538472+0000","last_clean":"2023-10-16T09:57:03.538472+0000","last_became_active":"2023-10-16T09:57:03.533564+0000","last_became_peered":"2023-10-16T09:57:03.533564+0000","last_unstale":"2023-10-16T09:57:03.538472+0000","last_undegraded":"2023-10-16T09:57:03.538472+0000","last_fullsized":"2023-10-16T09:57:03.538472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:19:07.590052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.534609+0000","last_change":"2023-10-16T09:57:03.533226+0000","last_active":"2023-10-16T09:57:03.534609+0000","last_peered":"2023-10-16T09:57:03.534609+0000","last_clean":"2023-10-16T09:57:03.534609+0000","last_became_active":"2023-10-16T09:57:03.532964+0000","last_became_peered":"2023-10-16T09:57:03.532964+0000","last_unstale":"2023-10-16T09:57:03.534609+0000","last_undegraded":"2023-10-16T09:57:03.534609+0000","last_fullsized":"2023-10-16T09:57:03.534609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:10.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544984+0000","last_change":"2023-10-16T09:57:03.537264+0000","last_active":"2023-10-16T09:57:03.544984+0000","last_peered":"2023-10-16T09:57:03.544984+0000","last_clean":"2023-10-16T09:57:03.544984+0000","last_became_active":"2023-10-16T09:57:03.536628+0000","last_became_peered":"2023-10-16T09:57:03.536628+0000","last_unstale":"2023-10-16T09:57:03.544984+0000","last_undegraded":"2023-10-16T09:57:03.544984+0000","last_fullsized":"2023-10-16T09:57:03.544984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:02.825517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548788+0000","last_change":"2023-10-16T09:57:03.547578+0000","last_active":"2023-10-16T09:57:03.548788+0000","last_peered":"2023-10-16T09:57:03.548788+0000","last_clean":"2023-10-16T09:57:03.548788+0000","last_became_active":"2023-10-16T09:57:03.544708+0000","last_became_peered":"2023-10-16T09:57:03.544708+0000","last_unstale":"2023-10-16T09:57:03.548788+0000","last_undegraded":"2023-10-16T09:57:03.548788+0000","last_fullsized":"2023-10-16T09:57:03.548788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:48.406276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548774+0000","last_change":"2023-10-16T09:57:03.547542+0000","last_active":"2023-10-16T09:57:03.548774+0000","last_peered":"2023-10-16T09:57:03.548774+0000","last_clean":"2023-10-16T09:57:03.548774+0000","last_became_active":"2023-10-16T09:57:03.547283+0000","last_became_peered":"2023-10-16T09:57:03.547283+0000","last_unstale":"2023-10-16T09:57:03.548774+0000","last_undegraded":"2023-10-16T09:57:03.548774+0000","last_fullsized":"2023-10-16T09:57:03.548774+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:47.246485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537472+0000","last_change":"2023-10-16T09:57:03.536112+0000","last_active":"2023-10-16T09:57:03.537472+0000","last_peered":"2023-10-16T09:57:03.537472+0000","last_clean":"2023-10-16T09:57:03.537472+0000","last_became_active":"2023-10-16T09:57:03.535848+0000","last_became_peered":"2023-10-16T09:57:03.535848+0000","last_unstale":"2023-10-16T09:57:03.537472+0000","last_undegraded":"2023-10-16T09:57:03.537472+0000","last_fullsized":"2023-10-16T09:57:03.537472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:04:05.329862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535942+0000","last_change":"2023-10-16T09:57:03.533782+0000","last_active":"2023-10-16T09:57:03.535942+0000","last_peered":"2023-10-16T09:57:03.535942+0000","last_clean":"2023-10-16T09:57:03.535942+0000","last_became_active":"2023-10-16T09:57:03.533442+0000","last_became_peered":"2023-10-16T09:57:03.533442+0000","last_unstale":"2023-10-16T09:57:03.535942+0000","last_undegraded":"2023-10-16T09:57:03.535942+0000","last_fullsized":"2023-10-16T09:57:03.535942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:49.900014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548161+0000","last_change":"2023-10-16T09:57:03.543816+0000","last_active":"2023-10-16T09:57:03.548161+0000","last_peered":"2023-10-16T09:57:03.548161+0000","last_clean":"2023-10-16T09:57:03.548161+0000","last_became_active":"2023-10-16T09:57:03.543522+0000","last_became_peered":"2023-10-16T09:57:03.543522+0000","last_unstale":"2023-10-16T09:57:03.548161+0000","last_undegraded":"2023-10-16T09:57:03.548161+0000","last_fullsized":"2023-10-16T09:57:03.548161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:08:07.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537018+0000","last_change":"2023-10-16T09:57:03.535848+0000","last_active":"2023-10-16T09:57:03.537018+0000","last_peered":"2023-10-16T09:57:03.537018+0000","last_clean":"2023-10-16T09:57:03.537018+0000","last_became_active":"2023-10-16T09:57:03.535472+0000","last_became_peered":"2023-10-16T09:57:03.535472+0000","last_unstale":"2023-10-16T09:57:03.537018+0000","last_undegraded":"2023-10-16T09:57:03.537018+0000","last_fullsized":"2023-10-16T09:57:03.537018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:31.324049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543944+0000","last_change":"2023-10-16T09:57:03.537377+0000","last_active":"2023-10-16T09:57:03.543944+0000","last_peered":"2023-10-16T09:57:03.543944+0000","last_clean":"2023-10-16T09:57:03.543944+0000","last_became_active":"2023-10-16T09:57:03.536574+0000","last_became_peered":"2023-10-16T09:57:03.536574+0000","last_unstale":"2023-10-16T09:57:03.543944+0000","last_undegraded":"2023-10-16T09:57:03.543944+0000","last_fullsized":"2023-10-16T09:57:03.543944+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:12.180755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543916+0000","last_change":"2023-10-16T09:57:03.542794+0000","last_active":"2023-10-16T09:57:03.543916+0000","last_peered":"2023-10-16T09:57:03.543916+0000","last_clean":"2023-10-16T09:57:03.543916+0000","last_became_active":"2023-10-16T09:57:03.542496+0000","last_became_peered":"2023-10-16T09:57:03.542496+0000","last_unstale":"2023-10-16T09:57:03.543916+0000","last_undegraded":"2023-10-16T09:57:03.543916+0000","last_fullsized":"2023-10-16T09:57:03.543916+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:59.073896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535336+0000","last_change":"2023-10-16T09:57:03.534346+0000","last_active":"2023-10-16T09:57:03.535336+0000","last_peered":"2023-10-16T09:57:03.535336+0000","last_clean":"2023-10-16T09:57:03.535336+0000","last_became_active":"2023-10-16T09:57:03.533839+0000","last_became_peered":"2023-10-16T09:57:03.533839+0000","last_unstale":"2023-10-16T09:57:03.535336+0000","last_undegraded":"2023-10-16T09:57:03.535336+0000","last_fullsized":"2023-10-16T09:57:03.535336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:40.250963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546574+0000","last_change":"2023-10-16T09:57:03.545846+0000","last_active":"2023-10-16T09:57:03.546574+0000","last_peered":"2023-10-16T09:57:03.546574+0000","last_clean":"2023-10-16T09:57:03.546574+0000","last_became_active":"2023-10-16T09:57:03.545622+0000","last_became_peered":"2023-10-16T09:57:03.545622+0000","last_unstale":"2023-10-16T09:57:03.546574+0000","last_undegraded":"2023-10-16T09:57:03.546574+0000","last_fullsized":"2023-10-16T09:57:03.546574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:16.034527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545821+0000","last_change":"2023-10-16T09:57:03.544582+0000","last_active":"2023-10-16T09:57:03.545821+0000","last_peered":"2023-10-16T09:57:03.545821+0000","last_clean":"2023-10-16T09:57:03.545821+0000","last_became_active":"2023-10-16T09:57:03.544311+0000","last_became_peered":"2023-10-16T09:57:03.544311+0000","last_unstale":"2023-10-16T09:57:03.545821+0000","last_undegraded":"2023-10-16T09:57:03.545821+0000","last_fullsized":"2023-10-16T09:57:03.545821+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:52.358095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545289+0000","last_change":"2023-10-16T09:57:03.540295+0000","last_active":"2023-10-16T09:57:03.545289+0000","last_peered":"2023-10-16T09:57:03.545289+0000","last_clean":"2023-10-16T09:57:03.545289+0000","last_became_active":"2023-10-16T09:57:03.539862+0000","last_became_peered":"2023-10-16T09:57:03.539862+0000","last_unstale":"2023-10-16T09:57:03.545289+0000","last_undegraded":"2023-10-16T09:57:03.545289+0000","last_fullsized":"2023-10-16T09:57:03.545289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:40.484150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.541100+0000","last_change":"2023-10-16T09:57:03.539611+0000","last_active":"2023-10-16T09:57:03.541100+0000","last_peered":"2023-10-16T09:57:03.541100+0000","last_clean":"2023-10-16T09:57:03.541100+0000","last_became_active":"2023-10-16T09:57:03.539310+0000","last_became_peered":"2023-10-16T09:57:03.539310+0000","last_unstale":"2023-10-16T09:57:03.541100+0000","last_undegraded":"2023-10-16T09:57:03.541100+0000","last_fullsized":"2023-10-16T09:57:03.541100+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:23:03.103348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.540295+0000","last_change":"2023-10-16T09:57:03.538319+0000","last_active":"2023-10-16T09:57:03.540295+0000","last_peered":"2023-10-16T09:57:03.540295+0000","last_clean":"2023-10-16T09:57:03.540295+0000","last_became_active":"2023-10-16T09:57:03.538063+0000","last_became_peered":"2023-10-16T09:57:03.538063+0000","last_unstale":"2023-10-16T09:57:03.540295+0000","last_undegraded":"2023-10-16T09:57:03.540295+0000","last_fullsized":"2023-10-16T09:57:03.540295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:35.415124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548139+0000","last_change":"2023-10-16T09:57:03.544396+0000","last_active":"2023-10-16T09:57:03.548139+0000","last_peered":"2023-10-16T09:57:03.548139+0000","last_clean":"2023-10-16T09:57:03.548139+0000","last_became_active":"2023-10-16T09:57:03.544153+0000","last_became_peered":"2023-10-16T09:57:03.544153+0000","last_unstale":"2023-10-16T09:57:03.548139+0000","last_undegraded":"2023-10-16T09:57:03.548139+0000","last_fullsized":"2023-10-16T09:57:03.548139+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:39.983421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535999+0000","last_change":"2023-10-16T09:57:03.533975+0000","last_active":"2023-10-16T09:57:03.535999+0000","last_peered":"2023-10-16T09:57:03.535999+0000","last_clean":"2023-10-16T09:57:03.535999+0000","last_became_active":"2023-10-16T09:57:03.533678+0000","last_became_peered":"2023-10-16T09:57:03.533678+0000","last_unstale":"2023-10-16T09:57:03.535999+0000","last_undegraded":"2023-10-16T09:57:03.535999+0000","last_fullsized":"2023-10-16T09:57:03.535999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:07:04.086320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542891+0000","last_change":"2023-10-16T09:57:03.535833+0000","last_active":"2023-10-16T09:57:03.542891+0000","last_peered":"2023-10-16T09:57:03.542891+0000","last_clean":"2023-10-16T09:57:03.542891+0000","last_became_active":"2023-10-16T09:57:03.535573+0000","last_became_peered":"2023-10-16T09:57:03.535573+0000","last_unstale":"2023-10-16T09:57:03.542891+0000","last_undegraded":"2023-10-16T09:57:03.542891+0000","last_fullsized":"2023-10-16T09:57:03.542891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:53.618793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545757+0000","last_change":"2023-10-16T09:57:03.544305+0000","last_active":"2023-10-16T09:57:03.545757+0000","last_peered":"2023-10-16T09:57:03.545757+0000","last_clean":"2023-10-16T09:57:03.545757+0000","last_became_active":"2023-10-16T09:57:03.543963+0000","last_became_peered":"2023-10-16T09:57:03.543963+0000","last_unstale":"2023-10-16T09:57:03.545757+0000","last_undegraded":"2023-10-16T09:57:03.545757+0000","last_fullsized":"2023-10-16T09:57:03.545757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:24.213653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543426+0000","last_change":"2023-10-16T09:57:03.541179+0000","last_active":"2023-10-16T09:57:03.543426+0000","last_peered":"2023-10-16T09:57:03.543426+0000","last_clean":"2023-10-16T09:57:03.543426+0000","last_became_active":"2023-10-16T09:57:03.539053+0000","last_became_peered":"2023-10-16T09:57:03.539053+0000","last_unstale":"2023-10-16T09:57:03.543426+0000","last_undegraded":"2023-10-16T09:57:03.543426+0000","last_fullsized":"2023-10-16T09:57:03.543426+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:20.158090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.541936+0000","last_change":"2023-10-16T09:57:03.541376+0000","last_active":"2023-10-16T09:57:03.541936+0000","last_peered":"2023-10-16T09:57:03.541936+0000","last_clean":"2023-10-16T09:57:03.541936+0000","last_became_active":"2023-10-16T09:57:03.540991+0000","last_became_peered":"2023-10-16T09:57:03.540991+0000","last_unstale":"2023-10-16T09:57:03.541936+0000","last_undegraded":"2023-10-16T09:57:03.541936+0000","last_fullsized":"2023-10-16T09:57:03.541936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:44.313131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544513+0000","last_change":"2023-10-16T09:57:03.541998+0000","last_active":"2023-10-16T09:57:03.544513+0000","last_peered":"2023-10-16T09:57:03.544513+0000","last_clean":"2023-10-16T09:57:03.544513+0000","last_became_active":"2023-10-16T09:57:03.541733+0000","last_became_peered":"2023-10-16T09:57:03.541733+0000","last_unstale":"2023-10-16T09:57:03.544513+0000","last_undegraded":"2023-10-16T09:57:03.544513+0000","last_fullsized":"2023-10-16T09:57:03.544513+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:12:21.668013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545336+0000","last_change":"2023-10-16T09:57:03.542489+0000","last_active":"2023-10-16T09:57:03.545336+0000","last_peered":"2023-10-16T09:57:03.545336+0000","last_clean":"2023-10-16T09:57:03.545336+0000","last_became_active":"2023-10-16T09:57:03.540455+0000","last_became_peered":"2023-10-16T09:57:03.540455+0000","last_unstale":"2023-10-16T09:57:03.545336+0000","last_undegraded":"2023-10-16T09:57:03.545336+0000","last_fullsized":"2023-10-16T09:57:03.545336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:17.011859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537870+0000","last_change":"2023-10-16T09:57:03.536927+0000","last_active":"2023-10-16T09:57:03.537870+0000","last_peered":"2023-10-16T09:57:03.537870+0000","last_clean":"2023-10-16T09:57:03.537870+0000","last_became_active":"2023-10-16T09:57:03.536452+0000","last_became_peered":"2023-10-16T09:57:03.536452+0000","last_unstale":"2023-10-16T09:57:03.537870+0000","last_undegraded":"2023-10-16T09:57:03.537870+0000","last_fullsized":"2023-10-16T09:57:03.537870+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:20.158090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545014+0000","last_change":"2023-10-16T09:57:03.537437+0000","last_active":"2023-10-16T09:57:03.545014+0000","last_peered":"2023-10-16T09:57:03.545014+0000","last_clean":"2023-10-16T09:57:03.545014+0000","last_became_active":"2023-10-16T09:57:03.536929+0000","last_became_peered":"2023-10-16T09:57:03.536929+0000","last_unstale":"2023-10-16T09:57:03.545014+0000","last_undegraded":"2023-10-16T09:57:03.545014+0000","last_fullsized":"2023-10-16T09:57:03.545014+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:27:01.674563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.549859+0000","last_change":"2023-10-16T09:57:03.549454+0000","last_active":"2023-10-16T09:57:03.549859+0000","last_peered":"2023-10-16T09:57:03.549859+0000","last_clean":"2023-10-16T09:57:03.549859+0000","last_became_active":"2023-10-16T09:57:03.549249+0000","last_became_peered":"2023-10-16T09:57:03.549249+0000","last_unstale":"2023-10-16T09:57:03.549859+0000","last_undegraded":"2023-10-16T09:57:03.549859+0000","last_fullsized":"2023-10-16T09:57:03.549859+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:48.406276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542060+0000","last_change":"2023-10-16T09:57:03.540217+0000","last_active":"2023-10-16T09:57:03.542060+0000","last_peered":"2023-10-16T09:57:03.542060+0000","last_clean":"2023-10-16T09:57:03.542060+0000","last_became_active":"2023-10-16T09:57:03.539940+0000","last_became_peered":"2023-10-16T09:57:03.539940+0000","last_unstale":"2023-10-16T09:57:03.542060+0000","last_undegraded":"2023-10-16T09:57:03.542060+0000","last_fullsized":"2023-10-16T09:57:03.542060+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:25.347861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"66'4","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611342+0000","last_change":"2023-10-16T09:57:02.614048+0000","last_active":"2023-10-16T09:57:03.611342+0000","last_peered":"2023-10-16T09:57:03.611342+0000","last_clean":"2023-10-16T09:57:03.611342+0000","last_became_active":"2023-10-16T09:57:02.613493+0000","last_became_peered":"2023-10-16T09:57:02.613493+0000","last_unstale":"2023-10-16T09:57:03.611342+0000","last_undegraded":"2023-10-16T09:57:03.611342+0000","last_fullsized":"2023-10-16T09:57:03.611342+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:10.790453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543293+0000","last_change":"2023-10-16T09:57:03.539869+0000","last_active":"2023-10-16T09:57:03.543293+0000","last_peered":"2023-10-16T09:57:03.543293+0000","last_clean":"2023-10-16T09:57:03.543293+0000","last_became_active":"2023-10-16T09:57:03.539480+0000","last_became_peered":"2023-10-16T09:57:03.539480+0000","last_unstale":"2023-10-16T09:57:03.543293+0000","last_undegraded":"2023-10-16T09:57:03.543293+0000","last_fullsized":"2023-10-16T09:57:03.543293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:08:07.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613481+0000","last_change":"2023-10-16T09:57:02.616346+0000","last_active":"2023-10-16T09:57:03.613481+0000","last_peered":"2023-10-16T09:57:03.613481+0000","last_clean":"2023-10-16T09:57:03.613481+0000","last_became_active":"2023-10-16T09:57:02.615999+0000","last_became_peered":"2023-10-16T09:57:02.615999+0000","last_unstale":"2023-10-16T09:57:03.613481+0000","last_undegraded":"2023-10-16T09:57:03.613481+0000","last_fullsized":"2023-10-16T09:57:03.613481+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:33.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544662+0000","last_change":"2023-10-16T09:57:03.542777+0000","last_active":"2023-10-16T09:57:03.544662+0000","last_peered":"2023-10-16T09:57:03.544662+0000","last_clean":"2023-10-16T09:57:03.544662+0000","last_became_active":"2023-10-16T09:57:03.542461+0000","last_became_peered":"2023-10-16T09:57:03.542461+0000","last_unstale":"2023-10-16T09:57:03.544662+0000","last_undegraded":"2023-10-16T09:57:03.544662+0000","last_fullsized":"2023-10-16T09:57:03.544662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:29:02.805935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613637+0000","last_change":"2023-10-16T09:57:02.613855+0000","last_active":"2023-10-16T09:57:03.613637+0000","last_peered":"2023-10-16T09:57:03.613637+0000","last_clean":"2023-10-16T09:57:03.613637+0000","last_became_active":"2023-10-16T09:57:02.613664+0000","last_became_peered":"2023-10-16T09:57:02.613664+0000","last_unstale":"2023-10-16T09:57:03.613637+0000","last_undegraded":"2023-10-16T09:57:03.613637+0000","last_fullsized":"2023-10-16T09:57:03.613637+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:11:43.214775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547042+0000","last_change":"2023-10-16T09:57:03.542012+0000","last_active":"2023-10-16T09:57:03.547042+0000","last_peered":"2023-10-16T09:57:03.547042+0000","last_clean":"2023-10-16T09:57:03.547042+0000","last_became_active":"2023-10-16T09:57:03.541736+0000","last_became_peered":"2023-10-16T09:57:03.541736+0000","last_unstale":"2023-10-16T09:57:03.547042+0000","last_undegraded":"2023-10-16T09:57:03.547042+0000","last_fullsized":"2023-10-16T09:57:03.547042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:39.770670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611020+0000","last_change":"2023-10-16T09:57:02.619617+0000","last_active":"2023-10-16T09:57:03.611020+0000","last_peered":"2023-10-16T09:57:03.611020+0000","last_clean":"2023-10-16T09:57:03.611020+0000","last_became_active":"2023-10-16T09:57:02.619399+0000","last_became_peered":"2023-10-16T09:57:02.619399+0000","last_unstale":"2023-10-16T09:57:03.611020+0000","last_undegraded":"2023-10-16T09:57:03.611020+0000","last_fullsized":"2023-10-16T09:57:03.611020+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:46.237844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542447+0000","last_change":"2023-10-16T09:57:03.538893+0000","last_active":"2023-10-16T09:57:03.542447+0000","last_peered":"2023-10-16T09:57:03.542447+0000","last_clean":"2023-10-16T09:57:03.542447+0000","last_became_active":"2023-10-16T09:57:03.538590+0000","last_became_peered":"2023-10-16T09:57:03.538590+0000","last_unstale":"2023-10-16T09:57:03.542447+0000","last_undegraded":"2023-10-16T09:57:03.542447+0000","last_fullsized":"2023-10-16T09:57:03.542447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.534359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.614362+0000","last_change":"2023-10-16T09:57:02.619914+0000","last_active":"2023-10-16T09:57:03.614362+0000","last_peered":"2023-10-16T09:57:03.614362+0000","last_clean":"2023-10-16T09:57:03.614362+0000","last_became_active":"2023-10-16T09:57:02.619654+0000","last_became_peered":"2023-10-16T09:57:02.619654+0000","last_unstale":"2023-10-16T09:57:03.614362+0000","last_undegraded":"2023-10-16T09:57:03.614362+0000","last_fullsized":"2023-10-16T09:57:03.614362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:20:53.160834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537751+0000","last_change":"2023-10-16T09:57:03.536603+0000","last_active":"2023-10-16T09:57:03.537751+0000","last_peered":"2023-10-16T09:57:03.537751+0000","last_clean":"2023-10-16T09:57:03.537751+0000","last_became_active":"2023-10-16T09:57:03.536325+0000","last_became_peered":"2023-10-16T09:57:03.536325+0000","last_unstale":"2023-10-16T09:57:03.537751+0000","last_undegraded":"2023-10-16T09:57:03.537751+0000","last_fullsized":"2023-10-16T09:57:03.537751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:17.967654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.610519+0000","last_change":"2023-10-16T09:57:02.614065+0000","last_active":"2023-10-16T09:57:03.610519+0000","last_peered":"2023-10-16T09:57:03.610519+0000","last_clean":"2023-10-16T09:57:03.610519+0000","last_became_active":"2023-10-16T09:57:02.613779+0000","last_became_peered":"2023-10-16T09:57:02.613779+0000","last_unstale":"2023-10-16T09:57:03.610519+0000","last_undegraded":"2023-10-16T09:57:03.610519+0000","last_fullsized":"2023-10-16T09:57:03.610519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:08.774560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548135+0000","last_change":"2023-10-16T09:57:03.543837+0000","last_active":"2023-10-16T09:57:03.548135+0000","last_peered":"2023-10-16T09:57:03.548135+0000","last_clean":"2023-10-16T09:57:03.548135+0000","last_became_active":"2023-10-16T09:57:03.543616+0000","last_became_peered":"2023-10-16T09:57:03.543616+0000","last_unstale":"2023-10-16T09:57:03.548135+0000","last_undegraded":"2023-10-16T09:57:03.548135+0000","last_fullsized":"2023-10-16T09:57:03.548135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:29:58.824360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.534784+0000","last_change":"2023-10-16T09:57:02.615083+0000","last_active":"2023-10-16T09:57:03.534784+0000","last_peered":"2023-10-16T09:57:03.534784+0000","last_clean":"2023-10-16T09:57:03.534784+0000","last_became_active":"2023-10-16T09:57:02.614789+0000","last_became_peered":"2023-10-16T09:57:02.614789+0000","last_unstale":"2023-10-16T09:57:03.534784+0000","last_undegraded":"2023-10-16T09:57:03.534784+0000","last_fullsized":"2023-10-16T09:57:03.534784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:21.527620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.539773+0000","last_change":"2023-10-16T09:57:03.538989+0000","last_active":"2023-10-16T09:57:03.539773+0000","last_peered":"2023-10-16T09:57:03.539773+0000","last_clean":"2023-10-16T09:57:03.539773+0000","last_became_active":"2023-10-16T09:57:03.538719+0000","last_became_peered":"2023-10-16T09:57:03.538719+0000","last_unstale":"2023-10-16T09:57:03.539773+0000","last_undegraded":"2023-10-16T09:57:03.539773+0000","last_fullsized":"2023-10-16T09:57:03.539773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:58.548809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.529775+0000","last_change":"2023-10-16T09:57:02.618144+0000","last_active":"2023-10-16T09:57:03.529775+0000","last_peered":"2023-10-16T09:57:03.529775+0000","last_clean":"2023-10-16T09:57:03.529775+0000","last_became_active":"2023-10-16T09:57:02.617740+0000","last_became_peered":"2023-10-16T09:57:02.617740+0000","last_unstale":"2023-10-16T09:57:03.529775+0000","last_undegraded":"2023-10-16T09:57:03.529775+0000","last_fullsized":"2023-10-16T09:57:03.529775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:56.994666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542996+0000","last_change":"2023-10-16T09:57:03.540825+0000","last_active":"2023-10-16T09:57:03.542996+0000","last_peered":"2023-10-16T09:57:03.542996+0000","last_clean":"2023-10-16T09:57:03.542996+0000","last_became_active":"2023-10-16T09:57:03.540557+0000","last_became_peered":"2023-10-16T09:57:03.540557+0000","last_unstale":"2023-10-16T09:57:03.542996+0000","last_undegraded":"2023-10-16T09:57:03.542996+0000","last_fullsized":"2023-10-16T09:57:03.542996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:03.569703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.526606+0000","last_change":"2023-10-16T09:57:02.616421+0000","last_active":"2023-10-16T09:57:03.526606+0000","last_peered":"2023-10-16T09:57:03.526606+0000","last_clean":"2023-10-16T09:57:03.526606+0000","last_became_active":"2023-10-16T09:57:02.616141+0000","last_became_peered":"2023-10-16T09:57:02.616141+0000","last_unstale":"2023-10-16T09:57:03.526606+0000","last_undegraded":"2023-10-16T09:57:03.526606+0000","last_fullsized":"2023-10-16T09:57:03.526606+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:56.856285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547450+0000","last_change":"2023-10-16T09:57:03.546464+0000","last_active":"2023-10-16T09:57:03.547450+0000","last_peered":"2023-10-16T09:57:03.547450+0000","last_clean":"2023-10-16T09:57:03.547450+0000","last_became_active":"2023-10-16T09:57:03.546194+0000","last_became_peered":"2023-10-16T09:57:03.546194+0000","last_unstale":"2023-10-16T09:57:03.547450+0000","last_undegraded":"2023-10-16T09:57:03.547450+0000","last_fullsized":"2023-10-16T09:57:03.547450+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:39.983421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.528789+0000","last_change":"2023-10-16T09:57:02.620311+0000","last_active":"2023-10-16T09:57:03.528789+0000","last_peered":"2023-10-16T09:57:03.528789+0000","last_clean":"2023-10-16T09:57:03.528789+0000","last_became_active":"2023-10-16T09:57:02.620059+0000","last_became_peered":"2023-10-16T09:57:02.620059+0000","last_unstale":"2023-10-16T09:57:03.528789+0000","last_undegraded":"2023-10-16T09:57:03.528789+0000","last_fullsized":"2023-10-16T09:57:03.528789+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:20:43.320545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546313+0000","last_change":"2023-10-16T09:57:03.544882+0000","last_active":"2023-10-16T09:57:03.546313+0000","last_peered":"2023-10-16T09:57:03.546313+0000","last_clean":"2023-10-16T09:57:03.546313+0000","last_became_active":"2023-10-16T09:57:03.544598+0000","last_became_peered":"2023-10-16T09:57:03.544598+0000","last_unstale":"2023-10-16T09:57:03.546313+0000","last_undegraded":"2023-10-16T09:57:03.546313+0000","last_fullsized":"2023-10-16T09:57:03.546313+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:09.800917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"66'3","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.614186+0000","last_change":"2023-10-16T09:57:02.615041+0000","last_active":"2023-10-16T09:57:03.614186+0000","last_peered":"2023-10-16T09:57:03.614186+0000","last_clean":"2023-10-16T09:57:03.614186+0000","last_became_active":"2023-10-16T09:57:02.614767+0000","last_became_peered":"2023-10-16T09:57:02.614767+0000","last_unstale":"2023-10-16T09:57:03.614186+0000","last_undegraded":"2023-10-16T09:57:03.614186+0000","last_fullsized":"2023-10-16T09:57:03.614186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:29:57.815719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542323+0000","last_change":"2023-10-16T09:57:03.538117+0000","last_active":"2023-10-16T09:57:03.542323+0000","last_peered":"2023-10-16T09:57:03.542323+0000","last_clean":"2023-10-16T09:57:03.542323+0000","last_became_active":"2023-10-16T09:57:03.537208+0000","last_became_peered":"2023-10-16T09:57:03.537208+0000","last_unstale":"2023-10-16T09:57:03.542323+0000","last_undegraded":"2023-10-16T09:57:03.542323+0000","last_fullsized":"2023-10-16T09:57:03.542323+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:32.636528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"66'6","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.616009+0000","last_change":"2023-10-16T09:57:02.614842+0000","last_active":"2023-10-16T09:57:03.616009+0000","last_peered":"2023-10-16T09:57:03.616009+0000","last_clean":"2023-10-16T09:57:03.616009+0000","last_became_active":"2023-10-16T09:57:02.614567+0000","last_became_peered":"2023-10-16T09:57:02.614567+0000","last_unstale":"2023-10-16T09:57:03.616009+0000","last_undegraded":"2023-10-16T09:57:03.616009+0000","last_fullsized":"2023-10-16T09:57:03.616009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:06:00.596063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543164+0000","last_change":"2023-10-16T09:57:03.536597+0000","last_active":"2023-10-16T09:57:03.543164+0000","last_peered":"2023-10-16T09:57:03.543164+0000","last_clean":"2023-10-16T09:57:03.543164+0000","last_became_active":"2023-10-16T09:57:03.536336+0000","last_became_peered":"2023-10-16T09:57:03.536336+0000","last_unstale":"2023-10-16T09:57:03.543164+0000","last_undegraded":"2023-10-16T09:57:03.543164+0000","last_fullsized":"2023-10-16T09:57:03.543164+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:53.843985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.610621+0000","last_change":"2023-10-16T09:57:02.616859+0000","last_active":"2023-10-16T09:57:03.610621+0000","last_peered":"2023-10-16T09:57:03.610621+0000","last_clean":"2023-10-16T09:57:03.610621+0000","last_became_active":"2023-10-16T09:57:02.616570+0000","last_became_peered":"2023-10-16T09:57:02.616570+0000","last_unstale":"2023-10-16T09:57:03.610621+0000","last_undegraded":"2023-10-16T09:57:03.610621+0000","last_fullsized":"2023-10-16T09:57:03.610621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:47.795378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545266+0000","last_change":"2023-10-16T09:57:03.540132+0000","last_active":"2023-10-16T09:57:03.545266+0000","last_peered":"2023-10-16T09:57:03.545266+0000","last_clean":"2023-10-16T09:57:03.545266+0000","last_became_active":"2023-10-16T09:57:03.537599+0000","last_became_peered":"2023-10-16T09:57:03.537599+0000","last_unstale":"2023-10-16T09:57:03.545266+0000","last_undegraded":"2023-10-16T09:57:03.545266+0000","last_fullsized":"2023-10-16T09:57:03.545266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:02.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.538513+0000","last_change":"2023-10-16T09:57:02.622456+0000","last_active":"2023-10-16T09:57:03.538513+0000","last_peered":"2023-10-16T09:57:03.538513+0000","last_clean":"2023-10-16T09:57:03.538513+0000","last_became_active":"2023-10-16T09:57:02.622127+0000","last_became_peered":"2023-10-16T09:57:02.622127+0000","last_unstale":"2023-10-16T09:57:03.538513+0000","last_undegraded":"2023-10-16T09:57:03.538513+0000","last_fullsized":"2023-10-16T09:57:03.538513+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:05.254985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543406+0000","last_change":"2023-10-16T09:57:03.540378+0000","last_active":"2023-10-16T09:57:03.543406+0000","last_peered":"2023-10-16T09:57:03.543406+0000","last_clean":"2023-10-16T09:57:03.543406+0000","last_became_active":"2023-10-16T09:57:03.539016+0000","last_became_peered":"2023-10-16T09:57:03.539016+0000","last_unstale":"2023-10-16T09:57:03.543406+0000","last_undegraded":"2023-10-16T09:57:03.543406+0000","last_fullsized":"2023-10-16T09:57:03.543406+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:59.566663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.526798+0000","last_change":"2023-10-16T09:57:02.614023+0000","last_active":"2023-10-16T09:57:03.526798+0000","last_peered":"2023-10-16T09:57:03.526798+0000","last_clean":"2023-10-16T09:57:03.526798+0000","last_became_active":"2023-10-16T09:57:02.613724+0000","last_became_peered":"2023-10-16T09:57:02.613724+0000","last_unstale":"2023-10-16T09:57:03.526798+0000","last_undegraded":"2023-10-16T09:57:03.526798+0000","last_fullsized":"2023-10-16T09:57:03.526798+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:52.267242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546556+0000","last_change":"2023-10-16T09:57:03.545703+0000","last_active":"2023-10-16T09:57:03.546556+0000","last_peered":"2023-10-16T09:57:03.546556+0000","last_clean":"2023-10-16T09:57:03.546556+0000","last_became_active":"2023-10-16T09:57:03.545504+0000","last_became_peered":"2023-10-16T09:57:03.545504+0000","last_unstale":"2023-10-16T09:57:03.546556+0000","last_undegraded":"2023-10-16T09:57:03.546556+0000","last_fullsized":"2023-10-16T09:57:03.546556+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:11.429284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"66'8","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613602+0000","last_change":"2023-10-16T09:57:02.614852+0000","last_active":"2023-10-16T09:57:03.613602+0000","last_peered":"2023-10-16T09:57:03.613602+0000","last_clean":"2023-10-16T09:57:03.613602+0000","last_became_active":"2023-10-16T09:57:02.614607+0000","last_became_peered":"2023-10-16T09:57:02.614607+0000","last_unstale":"2023-10-16T09:57:03.613602+0000","last_undegraded":"2023-10-16T09:57:03.613602+0000","last_fullsized":"2023-10-16T09:57:03.613602+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:29.506540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548906+0000","last_change":"2023-10-16T09:57:03.548143+0000","last_active":"2023-10-16T09:57:03.548906+0000","last_peered":"2023-10-16T09:57:03.548906+0000","last_clean":"2023-10-16T09:57:03.548906+0000","last_became_active":"2023-10-16T09:57:03.547880+0000","last_became_peered":"2023-10-16T09:57:03.547880+0000","last_unstale":"2023-10-16T09:57:03.548906+0000","last_undegraded":"2023-10-16T09:57:03.548906+0000","last_fullsized":"2023-10-16T09:57:03.548906+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:53.212827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611183+0000","last_change":"2023-10-16T09:57:02.619514+0000","last_active":"2023-10-16T09:57:03.611183+0000","last_peered":"2023-10-16T09:57:03.611183+0000","last_clean":"2023-10-16T09:57:03.611183+0000","last_became_active":"2023-10-16T09:57:02.619257+0000","last_became_peered":"2023-10-16T09:57:02.619257+0000","last_unstale":"2023-10-16T09:57:03.611183+0000","last_undegraded":"2023-10-16T09:57:03.611183+0000","last_fullsized":"2023-10-16T09:57:03.611183+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:54.986229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.539369+0000","last_change":"2023-10-16T09:57:03.538345+0000","last_active":"2023-10-16T09:57:03.539369+0000","last_peered":"2023-10-16T09:57:03.539369+0000","last_clean":"2023-10-16T09:57:03.539369+0000","last_became_active":"2023-10-16T09:57:03.538076+0000","last_became_peered":"2023-10-16T09:57:03.538076+0000","last_unstale":"2023-10-16T09:57:03.539369+0000","last_undegraded":"2023-10-16T09:57:03.539369+0000","last_fullsized":"2023-10-16T09:57:03.539369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:24.030251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.614934+0000","last_change":"2023-10-16T09:57:02.616270+0000","last_active":"2023-10-16T09:57:03.614934+0000","last_peered":"2023-10-16T09:57:03.614934+0000","last_clean":"2023-10-16T09:57:03.614934+0000","last_became_active":"2023-10-16T09:57:02.616015+0000","last_became_peered":"2023-10-16T09:57:02.616015+0000","last_unstale":"2023-10-16T09:57:03.614934+0000","last_undegraded":"2023-10-16T09:57:03.614934+0000","last_fullsized":"2023-10-16T09:57:03.614934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:42.361094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542313+0000","last_change":"2023-10-16T09:57:03.537753+0000","last_active":"2023-10-16T09:57:03.542313+0000","last_peered":"2023-10-16T09:57:03.542313+0000","last_clean":"2023-10-16T09:57:03.542313+0000","last_became_active":"2023-10-16T09:57:03.537462+0000","last_became_peered":"2023-10-16T09:57:03.537462+0000","last_unstale":"2023-10-16T09:57:03.542313+0000","last_undegraded":"2023-10-16T09:57:03.542313+0000","last_fullsized":"2023-10-16T09:57:03.542313+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:34.384510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613906+0000","last_change":"2023-10-16T09:57:02.614932+0000","last_active":"2023-10-16T09:57:03.613906+0000","last_peered":"2023-10-16T09:57:03.613906+0000","last_clean":"2023-10-16T09:57:03.613906+0000","last_became_active":"2023-10-16T09:57:02.614667+0000","last_became_peered":"2023-10-16T09:57:02.614667+0000","last_unstale":"2023-10-16T09:57:03.613906+0000","last_undegraded":"2023-10-16T09:57:03.613906+0000","last_fullsized":"2023-10-16T09:57:03.613906+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545162+0000","last_change":"2023-10-16T09:57:03.544679+0000","last_active":"2023-10-16T09:57:03.545162+0000","last_peered":"2023-10-16T09:57:03.545162+0000","last_clean":"2023-10-16T09:57:03.545162+0000","last_became_active":"2023-10-16T09:57:03.544402+0000","last_became_peered":"2023-10-16T09:57:03.544402+0000","last_unstale":"2023-10-16T09:57:03.545162+0000","last_undegraded":"2023-10-16T09:57:03.545162+0000","last_fullsized":"2023-10-16T09:57:03.545162+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:09:56.421946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.534659+0000","last_change":"2023-10-16T09:57:02.616040+0000","last_active":"2023-10-16T09:57:03.534659+0000","last_peered":"2023-10-16T09:57:03.534659+0000","last_clean":"2023-10-16T09:57:03.534659+0000","last_became_active":"2023-10-16T09:57:02.615773+0000","last_became_peered":"2023-10-16T09:57:02.615773+0000","last_unstale":"2023-10-16T09:57:03.534659+0000","last_undegraded":"2023-10-16T09:57:03.534659+0000","last_fullsized":"2023-10-16T09:57:03.534659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:34.219976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548054+0000","last_change":"2023-10-16T09:57:03.547154+0000","last_active":"2023-10-16T09:57:03.548054+0000","last_peered":"2023-10-16T09:57:03.548054+0000","last_clean":"2023-10-16T09:57:03.548054+0000","last_became_active":"2023-10-16T09:57:03.546842+0000","last_became_peered":"2023-10-16T09:57:03.546842+0000","last_unstale":"2023-10-16T09:57:03.548054+0000","last_undegraded":"2023-10-16T09:57:03.548054+0000","last_fullsized":"2023-10-16T09:57:03.548054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:34.354433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.615036+0000","last_change":"2023-10-16T09:57:02.617412+0000","last_active":"2023-10-16T09:57:03.615036+0000","last_peered":"2023-10-16T09:57:03.615036+0000","last_clean":"2023-10-16T09:57:03.615036+0000","last_became_active":"2023-10-16T09:57:02.617141+0000","last_became_peered":"2023-10-16T09:57:02.617141+0000","last_unstale":"2023-10-16T09:57:03.615036+0000","last_undegraded":"2023-10-16T09:57:03.615036+0000","last_fullsized":"2023-10-16T09:57:03.615036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:20:43.320545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543397+0000","last_change":"2023-10-16T09:57:03.540745+0000","last_active":"2023-10-16T09:57:03.543397+0000","last_peered":"2023-10-16T09:57:03.543397+0000","last_clean":"2023-10-16T09:57:03.543397+0000","last_became_active":"2023-10-16T09:57:03.539364+0000","last_became_peered":"2023-10-16T09:57:03.539364+0000","last_unstale":"2023-10-16T09:57:03.543397+0000","last_undegraded":"2023-10-16T09:57:03.543397+0000","last_fullsized":"2023-10-16T09:57:03.543397+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:50:01.772995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535056+0000","last_change":"2023-10-16T09:57:02.615394+0000","last_active":"2023-10-16T09:57:03.535056+0000","last_peered":"2023-10-16T09:57:03.535056+0000","last_clean":"2023-10-16T09:57:03.535056+0000","last_became_active":"2023-10-16T09:57:02.615117+0000","last_became_peered":"2023-10-16T09:57:02.615117+0000","last_unstale":"2023-10-16T09:57:03.535056+0000","last_undegraded":"2023-10-16T09:57:03.535056+0000","last_fullsized":"2023-10-16T09:57:03.535056+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:33.345792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545889+0000","last_change":"2023-10-16T09:57:03.543453+0000","last_active":"2023-10-16T09:57:03.545889+0000","last_peered":"2023-10-16T09:57:03.545889+0000","last_clean":"2023-10-16T09:57:03.545889+0000","last_became_active":"2023-10-16T09:57:03.543204+0000","last_became_peered":"2023-10-16T09:57:03.543204+0000","last_unstale":"2023-10-16T09:57:03.545889+0000","last_undegraded":"2023-10-16T09:57:03.545889+0000","last_fullsized":"2023-10-16T09:57:03.545889+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:33:35.109415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.530640+0000","last_change":"2023-10-16T09:57:02.620572+0000","last_active":"2023-10-16T09:57:03.530640+0000","last_peered":"2023-10-16T09:57:03.530640+0000","last_clean":"2023-10-16T09:57:03.530640+0000","last_became_active":"2023-10-16T09:57:02.620313+0000","last_became_peered":"2023-10-16T09:57:02.620313+0000","last_unstale":"2023-10-16T09:57:03.530640+0000","last_undegraded":"2023-10-16T09:57:03.530640+0000","last_fullsized":"2023-10-16T09:57:03.530640+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:42.440435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543073+0000","last_change":"2023-10-16T09:57:03.541477+0000","last_active":"2023-10-16T09:57:03.543073+0000","last_peered":"2023-10-16T09:57:03.543073+0000","last_clean":"2023-10-16T09:57:03.543073+0000","last_became_active":"2023-10-16T09:57:03.540938+0000","last_became_peered":"2023-10-16T09:57:03.540938+0000","last_unstale":"2023-10-16T09:57:03.543073+0000","last_undegraded":"2023-10-16T09:57:03.543073+0000","last_fullsized":"2023-10-16T09:57:03.543073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:03.569703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.528211+0000","last_change":"2023-10-16T09:57:02.611493+0000","last_active":"2023-10-16T09:57:03.528211+0000","last_peered":"2023-10-16T09:57:03.528211+0000","last_clean":"2023-10-16T09:57:03.528211+0000","last_became_active":"2023-10-16T09:57:02.611205+0000","last_became_peered":"2023-10-16T09:57:02.611205+0000","last_unstale":"2023-10-16T09:57:03.528211+0000","last_undegraded":"2023-10-16T09:57:03.528211+0000","last_fullsized":"2023-10-16T09:57:03.528211+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:50:30.035578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.549517+0000","last_change":"2023-10-16T09:57:03.549165+0000","last_active":"2023-10-16T09:57:03.549517+0000","last_peered":"2023-10-16T09:57:03.549517+0000","last_clean":"2023-10-16T09:57:03.549517+0000","last_became_active":"2023-10-16T09:57:03.548897+0000","last_became_peered":"2023-10-16T09:57:03.548897+0000","last_unstale":"2023-10-16T09:57:03.549517+0000","last_undegraded":"2023-10-16T09:57:03.549517+0000","last_fullsized":"2023-10-16T09:57:03.549517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:30.515181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611485+0000","last_change":"2023-10-16T09:57:02.622458+0000","last_active":"2023-10-16T09:57:03.611485+0000","last_peered":"2023-10-16T09:57:03.611485+0000","last_clean":"2023-10-16T09:57:03.611485+0000","last_became_active":"2023-10-16T09:57:02.622132+0000","last_became_peered":"2023-10-16T09:57:02.622132+0000","last_unstale":"2023-10-16T09:57:03.611485+0000","last_undegraded":"2023-10-16T09:57:03.611485+0000","last_fullsized":"2023-10-16T09:57:03.611485+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:39.242322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.538060+0000","last_change":"2023-10-16T09:57:03.536803+0000","last_active":"2023-10-16T09:57:03.538060+0000","last_peered":"2023-10-16T09:57:03.538060+0000","last_clean":"2023-10-16T09:57:03.538060+0000","last_became_active":"2023-10-16T09:57:03.536504+0000","last_became_peered":"2023-10-16T09:57:03.536504+0000","last_unstale":"2023-10-16T09:57:03.538060+0000","last_undegraded":"2023-10-16T09:57:03.538060+0000","last_fullsized":"2023-10-16T09:57:03.538060+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:06.263626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.527217+0000","last_change":"2023-10-16T09:57:02.605820+0000","last_active":"2023-10-16T09:57:03.527217+0000","last_peered":"2023-10-16T09:57:03.527217+0000","last_clean":"2023-10-16T09:57:03.527217+0000","last_became_active":"2023-10-16T09:57:02.605364+0000","last_became_peered":"2023-10-16T09:57:02.605364+0000","last_unstale":"2023-10-16T09:57:03.527217+0000","last_undegraded":"2023-10-16T09:57:03.527217+0000","last_fullsized":"2023-10-16T09:57:03.527217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:38.762029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547928+0000","last_change":"2023-10-16T09:57:03.543926+0000","last_active":"2023-10-16T09:57:03.547928+0000","last_peered":"2023-10-16T09:57:03.547928+0000","last_clean":"2023-10-16T09:57:03.547928+0000","last_became_active":"2023-10-16T09:57:03.543639+0000","last_became_peered":"2023-10-16T09:57:03.543639+0000","last_unstale":"2023-10-16T09:57:03.547928+0000","last_undegraded":"2023-10-16T09:57:03.547928+0000","last_fullsized":"2023-10-16T09:57:03.547928+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:52.358095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.532257+0000","last_change":"2023-10-16T09:57:02.613008+0000","last_active":"2023-10-16T09:57:03.532257+0000","last_peered":"2023-10-16T09:57:03.532257+0000","last_clean":"2023-10-16T09:57:03.532257+0000","last_became_active":"2023-10-16T09:57:02.612700+0000","last_became_peered":"2023-10-16T09:57:02.612700+0000","last_unstale":"2023-10-16T09:57:03.532257+0000","last_undegraded":"2023-10-16T09:57:03.532257+0000","last_fullsized":"2023-10-16T09:57:03.532257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:05.422941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546031+0000","last_change":"2023-10-16T09:57:03.543332+0000","last_active":"2023-10-16T09:57:03.546031+0000","last_peered":"2023-10-16T09:57:03.546031+0000","last_clean":"2023-10-16T09:57:03.546031+0000","last_became_active":"2023-10-16T09:57:03.543083+0000","last_became_peered":"2023-10-16T09:57:03.543083+0000","last_unstale":"2023-10-16T09:57:03.546031+0000","last_undegraded":"2023-10-16T09:57:03.546031+0000","last_fullsized":"2023-10-16T09:57:03.546031+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:14:22.011175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.532962+0000","last_change":"2023-10-16T09:57:02.614732+0000","last_active":"2023-10-16T09:57:03.532962+0000","last_peered":"2023-10-16T09:57:03.532962+0000","last_clean":"2023-10-16T09:57:03.532962+0000","last_became_active":"2023-10-16T09:57:02.614451+0000","last_became_peered":"2023-10-16T09:57:02.614451+0000","last_unstale":"2023-10-16T09:57:03.532962+0000","last_undegraded":"2023-10-16T09:57:03.532962+0000","last_fullsized":"2023-10-16T09:57:03.532962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:28:07.404143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544978+0000","last_change":"2023-10-16T09:57:03.543563+0000","last_active":"2023-10-16T09:57:03.544978+0000","last_peered":"2023-10-16T09:57:03.544978+0000","last_clean":"2023-10-16T09:57:03.544978+0000","last_became_active":"2023-10-16T09:57:03.543272+0000","last_became_peered":"2023-10-16T09:57:03.543272+0000","last_unstale":"2023-10-16T09:57:03.544978+0000","last_undegraded":"2023-10-16T09:57:03.544978+0000","last_fullsized":"2023-10-16T09:57:03.544978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:06:01.604704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.536127+0000","last_change":"2023-10-16T09:57:02.617260+0000","last_active":"2023-10-16T09:57:03.536127+0000","last_peered":"2023-10-16T09:57:03.536127+0000","last_clean":"2023-10-16T09:57:03.536127+0000","last_became_active":"2023-10-16T09:57:02.614712+0000","last_became_peered":"2023-10-16T09:57:02.614712+0000","last_unstale":"2023-10-16T09:57:03.536127+0000","last_undegraded":"2023-10-16T09:57:03.536127+0000","last_fullsized":"2023-10-16T09:57:03.536127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:52.835344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545871+0000","last_change":"2023-10-16T09:57:03.542728+0000","last_active":"2023-10-16T09:57:03.545871+0000","last_peered":"2023-10-16T09:57:03.545871+0000","last_clean":"2023-10-16T09:57:03.545871+0000","last_became_active":"2023-10-16T09:57:03.540575+0000","last_became_peered":"2023-10-16T09:57:03.540575+0000","last_unstale":"2023-10-16T09:57:03.545871+0000","last_undegraded":"2023-10-16T09:57:03.545871+0000","last_fullsized":"2023-10-16T09:57:03.545871+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:51.431983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.528593+0000","last_change":"2023-10-16T09:57:02.614970+0000","last_active":"2023-10-16T09:57:03.528593+0000","last_peered":"2023-10-16T09:57:03.528593+0000","last_clean":"2023-10-16T09:57:03.528593+0000","last_became_active":"2023-10-16T09:57:02.614672+0000","last_became_peered":"2023-10-16T09:57:02.614672+0000","last_unstale":"2023-10-16T09:57:03.528593+0000","last_undegraded":"2023-10-16T09:57:03.528593+0000","last_fullsized":"2023-10-16T09:57:03.528593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:04:04.321220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547915+0000","last_change":"2023-10-16T09:57:03.543826+0000","last_active":"2023-10-16T09:57:03.547915+0000","last_peered":"2023-10-16T09:57:03.547915+0000","last_clean":"2023-10-16T09:57:03.547915+0000","last_became_active":"2023-10-16T09:57:03.543559+0000","last_became_peered":"2023-10-16T09:57:03.543559+0000","last_unstale":"2023-10-16T09:57:03.547915+0000","last_undegraded":"2023-10-16T09:57:03.547915+0000","last_fullsized":"2023-10-16T09:57:03.547915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:43.191367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535069+0000","last_change":"2023-10-16T09:57:02.616106+0000","last_active":"2023-10-16T09:57:03.535069+0000","last_peered":"2023-10-16T09:57:03.535069+0000","last_clean":"2023-10-16T09:57:03.535069+0000","last_became_active":"2023-10-16T09:57:02.615851+0000","last_became_peered":"2023-10-16T09:57:02.615851+0000","last_unstale":"2023-10-16T09:57:03.535069+0000","last_undegraded":"2023-10-16T09:57:03.535069+0000","last_fullsized":"2023-10-16T09:57:03.535069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:42.182726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542053+0000","last_change":"2023-10-16T09:57:03.540665+0000","last_active":"2023-10-16T09:57:03.542053+0000","last_peered":"2023-10-16T09:57:03.542053+0000","last_clean":"2023-10-16T09:57:03.542053+0000","last_became_active":"2023-10-16T09:57:03.540390+0000","last_became_peered":"2023-10-16T09:57:03.540390+0000","last_unstale":"2023-10-16T09:57:03.542053+0000","last_undegraded":"2023-10-16T09:57:03.542053+0000","last_fullsized":"2023-10-16T09:57:03.542053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:19.148669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.531853+0000","last_change":"2023-10-16T09:57:02.616890+0000","last_active":"2023-10-16T09:57:03.531853+0000","last_peered":"2023-10-16T09:57:03.531853+0000","last_clean":"2023-10-16T09:57:03.531853+0000","last_became_active":"2023-10-16T09:57:02.616455+0000","last_became_peered":"2023-10-16T09:57:02.616455+0000","last_unstale":"2023-10-16T09:57:03.531853+0000","last_undegraded":"2023-10-16T09:57:03.531853+0000","last_fullsized":"2023-10-16T09:57:03.531853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:50:23.686105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"56'139","reported_seq":291,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T09:57:02.600417+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:57:02.600417+0000","last_peered":"2023-10-16T09:57:02.600417+0000","last_clean":"2023-10-16T09:57:02.600417+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:57:02.600417+0000","last_undegraded":"2023-10-16T09:57:02.600417+0000","last_fullsized":"2023-10-16T09:57:02.600417+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575015,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54748,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317092,"statfs":{"total":96636764160,"available":96580702208,"internally_reserved":0,"allocated":1392640,"data_stored":985041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.66900000000000004},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":11.246},"last":0.373}]},{"osd":1,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.61599999999999999},"min":{"1min":0.36199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.75,"5min":0.79800000000000004,"15min":1.1439999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.212},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64600000000000002,"15min":0.63},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66200000000000003,"15min":0.67100000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.58399999999999996}]},{"osd":4,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.51700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":0.439}]},{"osd":5,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50600000000000001,"15min":0.53000000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81100000000000005,"5min":0.82099999999999995,"15min":1.056},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55800000000000005,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34000000000000002},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.59199999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.748,"5min":0.76900000000000002,"15min":0.96399999999999997},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834575015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.69399999999999995,"5min":0.80800000000000005,"15min":0.83199999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.745,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.32100000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.76600000000000001,"5min":0.81799999999999995,"15min":0.83199999999999996},"last":0.69599999999999995}]},{"osd":2,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60099999999999998,"15min":0.57899999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.317},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.495}]},{"osd":3,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59799999999999998,"15min":0.58899999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65100000000000002,"5min":0.78900000000000003,"15min":0.81699999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59599999999999997,"15min":0.59699999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.77100000000000002,"5min":0.80100000000000005,"15min":0.82999999999999996},"last":0.62}]},{"osd":4,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45700000000000002,"15min":0.504},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.71599999999999997,"15min":0.93200000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.5,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":0.67200000000000004,"5min":0.76900000000000002,"15min":0.85399999999999998},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.497,"15min":0.51300000000000001},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33100000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53000000000000003,"15min":0.56999999999999995},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.34599999999999997},"max":{"1min":0.73399999999999999,"5min":0.80000000000000004,"15min":0.89200000000000002},"last":0.71699999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55200000000000005,"15min":0.57199999999999995},"min":{"1min":0.41899999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76800000000000002,"5min":0.84299999999999997,"15min":0.90900000000000003},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.58799999999999997},"min":{"1min":0.38300000000000001,"5min":0.375,"15min":0.374},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.72999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575016,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31944,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339896,"statfs":{"total":96636764160,"available":96604053504,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39900000000000002,"15min":0.39800000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.58199999999999996,"5min":0.65400000000000003,"15min":1.5509999999999999},"last":1.6000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.45400000000000001,"15min":0.436},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.55500000000000005,"5min":0.70199999999999996,"15min":1.593},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39900000000000002,"15min":0.38500000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.55700000000000005,"5min":0.67800000000000005,"15min":0.77400000000000002},"last":4.923},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.216},"max":{"1min":0.625,"5min":0.72899999999999998,"15min":0.81999999999999995},"last":1.712}]},{"osd":3,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.46400000000000002,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.63200000000000001,"5min":0.68799999999999994,"15min":0.69099999999999995},"last":1.984},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.439,"15min":0.42199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":0.61399999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.629},"min":{"1min":0.377,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.80200000000000005,"5min":0.83499999999999996,"15min":0.90600000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.63900000000000001,"15min":0.66100000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.89800000000000002,"5min":0.91400000000000003,"15min":5.141},"last":5.0069999999999997}]},{"osd":5,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65900000000000003,"15min":0.68300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":0.94799999999999995,"15min":5.0789999999999997},"last":4.9749999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40300000000000002},"max":{"1min":0.89000000000000001,"5min":0.93300000000000005,"15min":0.996},"last":1.7769999999999999}]},{"osd":6,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64600000000000002,"15min":0.64100000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.85199999999999998,"5min":0.873,"15min":0.95099999999999996},"last":0.628},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.65200000000000002,"15min":0.67400000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.371},"max":{"1min":0.86099999999999999,"5min":0.97699999999999998,"15min":5.1760000000000002},"last":5.0419999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67100000000000004,"15min":0.66200000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.92000000000000004,"5min":0.96499999999999997,"15min":0.98499999999999999},"last":1.744},{"interface":"front","average":{"1min":0.626,"5min":0.66400000000000003,"15min":0.65800000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.38500000000000001},"max":{"1min":0.84799999999999998,"5min":0.91400000000000003,"15min":1.0329999999999999},"last":0.52900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1372160,"data_stored":977426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.373},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46999999999999997,"15min":0.45100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.217},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47799999999999998,"15min":0.45500000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.624,"15min":0.60799999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33700000000000002},"max":{"1min":0.81299999999999994,"5min":0.92900000000000005,"15min":0.93700000000000006},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.64000000000000001,"15min":0.61699999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64600000000000002,"15min":0.625},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85399999999999998,"5min":0.99299999999999999,"15min":0.998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":5.2240000000000002},"last":0.623}]},{"osd":6,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68899999999999995,"15min":0.66200000000000003},"min":{"1min":0.39900000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94199999999999995,"5min":0.95999999999999996,"15min":0.96799999999999997},"last":0.755}]},{"osd":7,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64900000000000002,"15min":0.63500000000000001},"min":{"1min":0.35199999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.90600000000000003,"5min":1.02,"15min":1.02},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66000000000000003,"15min":0.63700000000000001},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.73099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575015,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54148,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317692,"statfs":{"total":96636764160,"available":96581316608,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.60199999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41299999999999998,"15min":0.40100000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.502,"5min":0.65400000000000003,"15min":0.67500000000000004},"last":0.317}]},{"osd":1,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24099999999999999},"max":{"1min":0.56399999999999995,"5min":0.63700000000000001,"15min":0.754},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.251,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54900000000000004,"5min":0.624,"15min":0.624},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42399999999999999,"15min":0.41399999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.46500000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48699999999999999,"15min":0.47699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.36299999999999999},"max":{"1min":0.84299999999999997,"5min":1.0309999999999999,"15min":1.0700000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.65100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81799999999999995,"5min":0.95099999999999996,"15min":1.071},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":0.77500000000000002,"5min":1.103,"15min":1.103},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.755,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.71499999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65200000000000002,"15min":0.64600000000000002},"min":{"1min":0.45700000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85799999999999998,"5min":1.159,"15min":1.159},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.39200000000000002},"max":{"1min":0.88400000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.69599999999999995,"15min":0.68100000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89900000000000002,"5min":1.0069999999999999,"15min":1.107},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.67900000000000005,"15min":0.66800000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81000000000000005,"5min":1.123,"15min":1.123},"last":0.67700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575015,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":806912,"data_stored":396183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38200000000000001,"15min":0.36599999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.56799999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.40300000000000002,"15min":0.38200000000000001},"min":{"1min":0.23100000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.67000000000000004},"last":0.29499999999999998}]},{"osd":1,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.41199999999999998,"15min":0.38800000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.58399999999999996,"5min":0.65400000000000003,"15min":0.66100000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.438,"15min":0.42399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27400000000000002,"15min":0.22700000000000001},"max":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.67500000000000004},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47799999999999998,"15min":0.45900000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.253},"max":{"1min":0.63300000000000001,"5min":0.69399999999999995,"15min":1.0760000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.66100000000000003,"5min":0.71199999999999997,"15min":0.93799999999999994},"last":0.376}]},{"osd":4,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64300000000000002,"15min":0.60899999999999999},"min":{"1min":0.42699999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.66500000000000004,"15min":0.63100000000000001},"min":{"1min":0.46300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86199999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66500000000000004,"15min":0.63200000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97399999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.50600000000000001,"5min":0.35999999999999999,"15min":0.33200000000000002},"max":{"1min":0.90200000000000002,"5min":1.036,"15min":1.036},"last":0.66800000000000004}]},{"osd":6,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66900000000000004,"15min":0.64000000000000001},"min":{"1min":0.44500000000000001,"5min":0.374,"15min":0.35099999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68999999999999995,"15min":0.65800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69199999999999995,"15min":0.66200000000000003},"min":{"1min":0.438,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.88900000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67700000000000005,"15min":0.65000000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.92800000000000005,"5min":0.999,"15min":0.999},"last":0.56299999999999994}]}]},{"osd":4,"up_from":13,"seq":55834575015,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.58299999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":1.3029999999999999},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.38500000000000001,"15min":0.36599999999999999},"max":{"1min":0.77800000000000002,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.312},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.36299999999999999,"15min":0.27600000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.318},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":1.8899999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.39400000000000002,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.7909999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.42499999999999999,"5min":0.38,"15min":0.34899999999999998},"max":{"1min":0.79200000000000004,"5min":0.79900000000000004,"15min":1.1839999999999999},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.50800000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.28599999999999998},"max":{"1min":0.70199999999999996,"5min":0.75,"15min":0.88900000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52400000000000002,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.41999999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.505},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79200000000000004,"5min":0.86299999999999999,"15min":0.92000000000000004},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.621,"15min":0.621},"min":{"1min":0.42599999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81599999999999995,"5min":0.879,"15min":0.93799999999999994},"last":0.53100000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575015,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45828,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326012,"statfs":{"total":96636764160,"available":96589836288,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63600000000000001,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.63900000000000001,"15min":0.60499999999999998},"min":{"1min":0.34499999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.58699999999999997}]},{"osd":1,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.56699999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.249},"max":{"1min":0.71899999999999997,"5min":0.875,"15min":1.1279999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.82899999999999996,"15min":1.0720000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58299999999999996,"15min":0.55900000000000005},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.76900000000000002,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57899999999999996,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.36899999999999999,"15min":0.35499999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.84499999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60999999999999999,"15min":0.58499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.65800000000000003,"5min":0.747,"15min":0.84199999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69599999999999995,"5min":0.85399999999999998,"15min":0.86899999999999999},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.48699999999999999,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67100000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65200000000000002,"5min":0.82999999999999996,"15min":0.88200000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.751,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.315},"max":{"1min":0.73099999999999998,"5min":0.876,"15min":0.88500000000000001},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:57:07.721 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:57:07.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.874+0000 7f3b87209700 1 Processor -- start 2023-10-16T09:57:07.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.874+0000 7f3b87209700 1 -- start start 2023-10-16T09:57:07.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.874+0000 7f3b87209700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8015d7c0 0x7f3b80161290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.874+0000 7f3b87209700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3b800ca5b0 0x7f3b801617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.874+0000 7f3b87209700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b800c4d10 con 0x7f3b8006b300 2023-10-16T09:57:07.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.874+0000 7f3b87209700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8006bea0 con 0x7f3b800ca5b0 2023-10-16T09:57:07.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.874+0000 7f3b87209700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800c55b0 con 0x7f3b8015d7c0 2023-10-16T09:57:07.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8015d7c0 0x7f3b80161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b7ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3b800ca5b0 0x7f3b801617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8015d7c0 0x7f3b80161290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:33356/0 (socket says 172.21.15.158:33356) 2023-10-16T09:57:07.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 -- 172.21.15.158:0/2174632006 learned_addr learned my addr 172.21.15.158:0/2174632006 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:07.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 -- 172.21.15.158:0/2174632006 >> v2:172.21.15.172:3300/0 conn(0x7f3b800ca5b0 msgr2=0x7f3b801617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:07.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 --2- 172.21.15.158:0/2174632006 >> v2:172.21.15.172:3300/0 conn(0x7f3b800ca5b0 0x7f3b801617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 -- 172.21.15.158:0/2174632006 >> v1:172.21.15.172:6789/0 conn(0x7f3b8006b300 legacy=0x7f3b8006b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 -- 172.21.15.158:0/2174632006 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8015dba0 con 0x7f3b8015d7c0 2023-10-16T09:57:07.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b7ffff700 1 --2- 172.21.15.158:0/2174632006 >> v2:172.21.15.172:3300/0 conn(0x7f3b800ca5b0 0x7f3b801617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:57:07.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.875+0000 7f3b84fa5700 1 --2- 172.21.15.158:0/2174632006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8015d7c0 0x7f3b80161290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3b740098f0 tx=0x7f3b74009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d8bc21c0ef691 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b7f7fe700 1 -- 172.21.15.158:0/2174632006 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74011830 con 0x7f3b8015d7c0 2023-10-16T09:57:07.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b7f7fe700 1 -- 172.21.15.158:0/2174632006 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b74011990 con 0x7f3b8015d7c0 2023-10-16T09:57:07.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b7f7fe700 1 -- 172.21.15.158:0/2174632006 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74011d20 con 0x7f3b8015d7c0 2023-10-16T09:57:07.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 -- 172.21.15.158:0/2174632006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8015d7c0 msgr2=0x7f3b80161290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 --2- 172.21.15.158:0/2174632006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8015d7c0 0x7f3b80161290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3b740098f0 tx=0x7f3b74009c00 comp rx=0 tx=0).stop 2023-10-16T09:57:07.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 -- 172.21.15.158:0/2174632006 shutdown_connections 2023-10-16T09:57:07.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 --2- 172.21.15.158:0/2174632006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8015d7c0 0x7f3b80161290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 --2- 172.21.15.158:0/2174632006 >> v2:172.21.15.172:3300/0 conn(0x7f3b800ca5b0 0x7f3b801617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 -- 172.21.15.158:0/2174632006 >> 172.21.15.158:0/2174632006 conn(0x7f3b800cc4b0 msgr2=0x7f3b800cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:07.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 -- 172.21.15.158:0/2174632006 shutdown_connections 2023-10-16T09:57:07.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.876+0000 7f3b87209700 1 -- 172.21.15.158:0/2174632006 wait complete. 2023-10-16T09:57:07.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.877+0000 7f3b87209700 1 Processor -- start 2023-10-16T09:57:07.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.877+0000 7f3b87209700 1 -- start start 2023-10-16T09:57:07.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b87209700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8006b300 0x7f3b801557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b87209700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b800ca5b0 0x7f3b800c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b87209700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b8015d7c0 0x7f3b800cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b87209700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b80155d10 con 0x7f3b800ca5b0 2023-10-16T09:57:07.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b87209700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8015f6f0 con 0x7f3b8015d7c0 2023-10-16T09:57:07.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b87209700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800c82e0 con 0x7f3b8006b300 2023-10-16T09:57:07.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b7ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b800ca5b0 0x7f3b800c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b7ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b800ca5b0 0x7f3b800c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50760/0 (socket says 172.21.15.158:50760) 2023-10-16T09:57:07.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.878+0000 7f3b7ffff700 1 -- 172.21.15.158:0/4232232685 learned_addr learned my addr 172.21.15.158:0/4232232685 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:07.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b84fa5700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8006b300 0x7f3b801557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b857a6700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b8015d7c0 0x7f3b800cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7ffff700 1 -- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8006b300 msgr2=0x7f3b801557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7ffff700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8006b300 0x7f3b801557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7ffff700 1 -- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b8015d7c0 msgr2=0x7f3b800cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:07.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7ffff700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b8015d7c0 0x7f3b800cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:07.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7ffff700 1 -- 172.21.15.158:0/4232232685 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c0094a0 con 0x7f3b800ca5b0 2023-10-16T09:57:07.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b857a6700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b8015d7c0 0x7f3b800cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:07.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7ffff700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b800ca5b0 0x7f3b800c6f10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00ec10 tx=0x7f3b6c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7d7fa700 1 -- 172.21.15.158:0/4232232685 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00c910 con 0x7f3b800ca5b0 2023-10-16T09:57:07.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7d7fa700 1 -- 172.21.15.158:0/4232232685 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c00ca70 con 0x7f3b800ca5b0 2023-10-16T09:57:07.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b7d7fa700 1 -- 172.21.15.158:0/4232232685 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00cde0 con 0x7f3b800ca5b0 2023-10-16T09:57:07.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b74009550 con 0x7f3b800ca5b0 2023-10-16T09:57:07.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.879+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b800c7760 con 0x7f3b800ca5b0 2023-10-16T09:57:07.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.880+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b800681b0 con 0x7f3b800ca5b0 2023-10-16T09:57:07.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.881+0000 7f3b7d7fa700 1 -- 172.21.15.158:0/4232232685 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3b6c013470 con 0x7f3b800ca5b0 2023-10-16T09:57:07.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.881+0000 7f3b7d7fa700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f3b6406b020 0x7f3b6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:07.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.881+0000 7f3b7d7fa700 1 -- 172.21.15.158:0/4232232685 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f3b6c089b70 con 0x7f3b800ca5b0 2023-10-16T09:57:07.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.881+0000 7f3b84fa5700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f3b6406b020 0x7f3b6406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:07.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.881+0000 7f3b84fa5700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f3b6406b020 0x7f3b6406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009c40 tx=0x7f3b74008030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:07.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:07.885+0000 7f3b7d7fa700 1 -- 172.21.15.158:0/4232232685 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b6c0547c0 con 0x7f3b800ca5b0 2023-10-16T09:57:08.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.070+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 --> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b800c4f60 con 0x7f3b6406b020 2023-10-16T09:57:08.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.078+0000 7f3b7d7fa700 1 -- 172.21.15.158:0/4232232685 <== mgr.6009 v2:172.21.15.158:6832/211595975 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288907 (secure 0 0 0) 0x7f3b74011b80 con 0x7f3b6406b020 2023-10-16T09:57:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:08.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f3b6406b020 msgr2=0x7f3b6406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f3b6406b020 0x7f3b6406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009c40 tx=0x7f3b74008030 comp rx=0 tx=0).stop 2023-10-16T09:57:08.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b800ca5b0 msgr2=0x7f3b800c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b800ca5b0 0x7f3b800c6f10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00ec10 tx=0x7f3b6c00ef20 comp rx=0 tx=0).stop 2023-10-16T09:57:08.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 shutdown_connections 2023-10-16T09:57:08.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f3b6406b020 0x7f3b6406d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b8015d7c0 0x7f3b800cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b8006b300 0x7f3b801557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 --2- 172.21.15.158:0/4232232685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b800ca5b0 0x7f3b800c6f10 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 >> 172.21.15.158:0/4232232685 conn(0x7f3b800cc4b0 msgr2=0x7f3b8006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:08.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 shutdown_connections 2023-10-16T09:57:08.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.081+0000 7f3b87209700 1 -- 172.21.15.158:0/4232232685 wait complete. 2023-10-16T09:57:08.087 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:57:08.095 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T09:57:08.055377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315052,"kb_used_data":7288,"kb_used_omap":12,"kb_used_meta":307635,"kb_avail":754659668,"statfs":{"total":773094113280,"available":772771500032,"internally_reserved":0,"allocated":7462912,"data_stored":4293940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":315018807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001309"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545189+0000","last_change":"2023-10-16T09:57:03.544690+0000","last_active":"2023-10-16T09:57:03.545189+0000","last_peered":"2023-10-16T09:57:03.545189+0000","last_clean":"2023-10-16T09:57:03.545189+0000","last_became_active":"2023-10-16T09:57:03.544417+0000","last_became_peered":"2023-10-16T09:57:03.544417+0000","last_unstale":"2023-10-16T09:57:03.545189+0000","last_undegraded":"2023-10-16T09:57:03.545189+0000","last_fullsized":"2023-10-16T09:57:03.545189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:15.106362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545803+0000","last_change":"2023-10-16T09:57:03.544520+0000","last_active":"2023-10-16T09:57:03.545803+0000","last_peered":"2023-10-16T09:57:03.545803+0000","last_clean":"2023-10-16T09:57:03.545803+0000","last_became_active":"2023-10-16T09:57:03.544248+0000","last_became_peered":"2023-10-16T09:57:03.544248+0000","last_unstale":"2023-10-16T09:57:03.545803+0000","last_undegraded":"2023-10-16T09:57:03.545803+0000","last_fullsized":"2023-10-16T09:57:03.545803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:42.266310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545041+0000","last_change":"2023-10-16T09:57:03.539579+0000","last_active":"2023-10-16T09:57:03.545041+0000","last_peered":"2023-10-16T09:57:03.545041+0000","last_clean":"2023-10-16T09:57:03.545041+0000","last_became_active":"2023-10-16T09:57:03.537238+0000","last_became_peered":"2023-10-16T09:57:03.537238+0000","last_unstale":"2023-10-16T09:57:03.545041+0000","last_undegraded":"2023-10-16T09:57:03.545041+0000","last_fullsized":"2023-10-16T09:57:03.545041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:22.204911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543498+0000","last_change":"2023-10-16T09:57:03.541526+0000","last_active":"2023-10-16T09:57:03.543498+0000","last_peered":"2023-10-16T09:57:03.543498+0000","last_clean":"2023-10-16T09:57:03.543498+0000","last_became_active":"2023-10-16T09:57:03.541262+0000","last_became_peered":"2023-10-16T09:57:03.541262+0000","last_unstale":"2023-10-16T09:57:03.543498+0000","last_undegraded":"2023-10-16T09:57:03.543498+0000","last_fullsized":"2023-10-16T09:57:03.543498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:34:42.033143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.538472+0000","last_change":"2023-10-16T09:57:03.533867+0000","last_active":"2023-10-16T09:57:03.538472+0000","last_peered":"2023-10-16T09:57:03.538472+0000","last_clean":"2023-10-16T09:57:03.538472+0000","last_became_active":"2023-10-16T09:57:03.533564+0000","last_became_peered":"2023-10-16T09:57:03.533564+0000","last_unstale":"2023-10-16T09:57:03.538472+0000","last_undegraded":"2023-10-16T09:57:03.538472+0000","last_fullsized":"2023-10-16T09:57:03.538472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:19:07.590052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.534609+0000","last_change":"2023-10-16T09:57:03.533226+0000","last_active":"2023-10-16T09:57:03.534609+0000","last_peered":"2023-10-16T09:57:03.534609+0000","last_clean":"2023-10-16T09:57:03.534609+0000","last_became_active":"2023-10-16T09:57:03.532964+0000","last_became_peered":"2023-10-16T09:57:03.532964+0000","last_unstale":"2023-10-16T09:57:03.534609+0000","last_undegraded":"2023-10-16T09:57:03.534609+0000","last_fullsized":"2023-10-16T09:57:03.534609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:10.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544984+0000","last_change":"2023-10-16T09:57:03.537264+0000","last_active":"2023-10-16T09:57:03.544984+0000","last_peered":"2023-10-16T09:57:03.544984+0000","last_clean":"2023-10-16T09:57:03.544984+0000","last_became_active":"2023-10-16T09:57:03.536628+0000","last_became_peered":"2023-10-16T09:57:03.536628+0000","last_unstale":"2023-10-16T09:57:03.544984+0000","last_undegraded":"2023-10-16T09:57:03.544984+0000","last_fullsized":"2023-10-16T09:57:03.544984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:02.825517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548788+0000","last_change":"2023-10-16T09:57:03.547578+0000","last_active":"2023-10-16T09:57:03.548788+0000","last_peered":"2023-10-16T09:57:03.548788+0000","last_clean":"2023-10-16T09:57:03.548788+0000","last_became_active":"2023-10-16T09:57:03.544708+0000","last_became_peered":"2023-10-16T09:57:03.544708+0000","last_unstale":"2023-10-16T09:57:03.548788+0000","last_undegraded":"2023-10-16T09:57:03.548788+0000","last_fullsized":"2023-10-16T09:57:03.548788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:48.406276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548774+0000","last_change":"2023-10-16T09:57:03.547542+0000","last_active":"2023-10-16T09:57:03.548774+0000","last_peered":"2023-10-16T09:57:03.548774+0000","last_clean":"2023-10-16T09:57:03.548774+0000","last_became_active":"2023-10-16T09:57:03.547283+0000","last_became_peered":"2023-10-16T09:57:03.547283+0000","last_unstale":"2023-10-16T09:57:03.548774+0000","last_undegraded":"2023-10-16T09:57:03.548774+0000","last_fullsized":"2023-10-16T09:57:03.548774+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:47.246485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537472+0000","last_change":"2023-10-16T09:57:03.536112+0000","last_active":"2023-10-16T09:57:03.537472+0000","last_peered":"2023-10-16T09:57:03.537472+0000","last_clean":"2023-10-16T09:57:03.537472+0000","last_became_active":"2023-10-16T09:57:03.535848+0000","last_became_peered":"2023-10-16T09:57:03.535848+0000","last_unstale":"2023-10-16T09:57:03.537472+0000","last_undegraded":"2023-10-16T09:57:03.537472+0000","last_fullsized":"2023-10-16T09:57:03.537472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:04:05.329862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535942+0000","last_change":"2023-10-16T09:57:03.533782+0000","last_active":"2023-10-16T09:57:03.535942+0000","last_peered":"2023-10-16T09:57:03.535942+0000","last_clean":"2023-10-16T09:57:03.535942+0000","last_became_active":"2023-10-16T09:57:03.533442+0000","last_became_peered":"2023-10-16T09:57:03.533442+0000","last_unstale":"2023-10-16T09:57:03.535942+0000","last_undegraded":"2023-10-16T09:57:03.535942+0000","last_fullsized":"2023-10-16T09:57:03.535942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:49.900014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548161+0000","last_change":"2023-10-16T09:57:03.543816+0000","last_active":"2023-10-16T09:57:03.548161+0000","last_peered":"2023-10-16T09:57:03.548161+0000","last_clean":"2023-10-16T09:57:03.548161+0000","last_became_active":"2023-10-16T09:57:03.543522+0000","last_became_peered":"2023-10-16T09:57:03.543522+0000","last_unstale":"2023-10-16T09:57:03.548161+0000","last_undegraded":"2023-10-16T09:57:03.548161+0000","last_fullsized":"2023-10-16T09:57:03.548161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:08:07.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537018+0000","last_change":"2023-10-16T09:57:03.535848+0000","last_active":"2023-10-16T09:57:03.537018+0000","last_peered":"2023-10-16T09:57:03.537018+0000","last_clean":"2023-10-16T09:57:03.537018+0000","last_became_active":"2023-10-16T09:57:03.535472+0000","last_became_peered":"2023-10-16T09:57:03.535472+0000","last_unstale":"2023-10-16T09:57:03.537018+0000","last_undegraded":"2023-10-16T09:57:03.537018+0000","last_fullsized":"2023-10-16T09:57:03.537018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:31.324049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543944+0000","last_change":"2023-10-16T09:57:03.537377+0000","last_active":"2023-10-16T09:57:03.543944+0000","last_peered":"2023-10-16T09:57:03.543944+0000","last_clean":"2023-10-16T09:57:03.543944+0000","last_became_active":"2023-10-16T09:57:03.536574+0000","last_became_peered":"2023-10-16T09:57:03.536574+0000","last_unstale":"2023-10-16T09:57:03.543944+0000","last_undegraded":"2023-10-16T09:57:03.543944+0000","last_fullsized":"2023-10-16T09:57:03.543944+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:12.180755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543916+0000","last_change":"2023-10-16T09:57:03.542794+0000","last_active":"2023-10-16T09:57:03.543916+0000","last_peered":"2023-10-16T09:57:03.543916+0000","last_clean":"2023-10-16T09:57:03.543916+0000","last_became_active":"2023-10-16T09:57:03.542496+0000","last_became_peered":"2023-10-16T09:57:03.542496+0000","last_unstale":"2023-10-16T09:57:03.543916+0000","last_undegraded":"2023-10-16T09:57:03.543916+0000","last_fullsized":"2023-10-16T09:57:03.543916+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:59.073896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535336+0000","last_change":"2023-10-16T09:57:03.534346+0000","last_active":"2023-10-16T09:57:03.535336+0000","last_peered":"2023-10-16T09:57:03.535336+0000","last_clean":"2023-10-16T09:57:03.535336+0000","last_became_active":"2023-10-16T09:57:03.533839+0000","last_became_peered":"2023-10-16T09:57:03.533839+0000","last_unstale":"2023-10-16T09:57:03.535336+0000","last_undegraded":"2023-10-16T09:57:03.535336+0000","last_fullsized":"2023-10-16T09:57:03.535336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:40.250963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546574+0000","last_change":"2023-10-16T09:57:03.545846+0000","last_active":"2023-10-16T09:57:03.546574+0000","last_peered":"2023-10-16T09:57:03.546574+0000","last_clean":"2023-10-16T09:57:03.546574+0000","last_became_active":"2023-10-16T09:57:03.545622+0000","last_became_peered":"2023-10-16T09:57:03.545622+0000","last_unstale":"2023-10-16T09:57:03.546574+0000","last_undegraded":"2023-10-16T09:57:03.546574+0000","last_fullsized":"2023-10-16T09:57:03.546574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:16.034527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545821+0000","last_change":"2023-10-16T09:57:03.544582+0000","last_active":"2023-10-16T09:57:03.545821+0000","last_peered":"2023-10-16T09:57:03.545821+0000","last_clean":"2023-10-16T09:57:03.545821+0000","last_became_active":"2023-10-16T09:57:03.544311+0000","last_became_peered":"2023-10-16T09:57:03.544311+0000","last_unstale":"2023-10-16T09:57:03.545821+0000","last_undegraded":"2023-10-16T09:57:03.545821+0000","last_fullsized":"2023-10-16T09:57:03.545821+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:52.358095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545289+0000","last_change":"2023-10-16T09:57:03.540295+0000","last_active":"2023-10-16T09:57:03.545289+0000","last_peered":"2023-10-16T09:57:03.545289+0000","last_clean":"2023-10-16T09:57:03.545289+0000","last_became_active":"2023-10-16T09:57:03.539862+0000","last_became_peered":"2023-10-16T09:57:03.539862+0000","last_unstale":"2023-10-16T09:57:03.545289+0000","last_undegraded":"2023-10-16T09:57:03.545289+0000","last_fullsized":"2023-10-16T09:57:03.545289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:40.484150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.541100+0000","last_change":"2023-10-16T09:57:03.539611+0000","last_active":"2023-10-16T09:57:03.541100+0000","last_peered":"2023-10-16T09:57:03.541100+0000","last_clean":"2023-10-16T09:57:03.541100+0000","last_became_active":"2023-10-16T09:57:03.539310+0000","last_became_peered":"2023-10-16T09:57:03.539310+0000","last_unstale":"2023-10-16T09:57:03.541100+0000","last_undegraded":"2023-10-16T09:57:03.541100+0000","last_fullsized":"2023-10-16T09:57:03.541100+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:23:03.103348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.540295+0000","last_change":"2023-10-16T09:57:03.538319+0000","last_active":"2023-10-16T09:57:03.540295+0000","last_peered":"2023-10-16T09:57:03.540295+0000","last_clean":"2023-10-16T09:57:03.540295+0000","last_became_active":"2023-10-16T09:57:03.538063+0000","last_became_peered":"2023-10-16T09:57:03.538063+0000","last_unstale":"2023-10-16T09:57:03.540295+0000","last_undegraded":"2023-10-16T09:57:03.540295+0000","last_fullsized":"2023-10-16T09:57:03.540295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:35.415124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548139+0000","last_change":"2023-10-16T09:57:03.544396+0000","last_active":"2023-10-16T09:57:03.548139+0000","last_peered":"2023-10-16T09:57:03.548139+0000","last_clean":"2023-10-16T09:57:03.548139+0000","last_became_active":"2023-10-16T09:57:03.544153+0000","last_became_peered":"2023-10-16T09:57:03.544153+0000","last_unstale":"2023-10-16T09:57:03.548139+0000","last_undegraded":"2023-10-16T09:57:03.548139+0000","last_fullsized":"2023-10-16T09:57:03.548139+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:39.983421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535999+0000","last_change":"2023-10-16T09:57:03.533975+0000","last_active":"2023-10-16T09:57:03.535999+0000","last_peered":"2023-10-16T09:57:03.535999+0000","last_clean":"2023-10-16T09:57:03.535999+0000","last_became_active":"2023-10-16T09:57:03.533678+0000","last_became_peered":"2023-10-16T09:57:03.533678+0000","last_unstale":"2023-10-16T09:57:03.535999+0000","last_undegraded":"2023-10-16T09:57:03.535999+0000","last_fullsized":"2023-10-16T09:57:03.535999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:07:04.086320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542891+0000","last_change":"2023-10-16T09:57:03.535833+0000","last_active":"2023-10-16T09:57:03.542891+0000","last_peered":"2023-10-16T09:57:03.542891+0000","last_clean":"2023-10-16T09:57:03.542891+0000","last_became_active":"2023-10-16T09:57:03.535573+0000","last_became_peered":"2023-10-16T09:57:03.535573+0000","last_unstale":"2023-10-16T09:57:03.542891+0000","last_undegraded":"2023-10-16T09:57:03.542891+0000","last_fullsized":"2023-10-16T09:57:03.542891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:53.618793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545757+0000","last_change":"2023-10-16T09:57:03.544305+0000","last_active":"2023-10-16T09:57:03.545757+0000","last_peered":"2023-10-16T09:57:03.545757+0000","last_clean":"2023-10-16T09:57:03.545757+0000","last_became_active":"2023-10-16T09:57:03.543963+0000","last_became_peered":"2023-10-16T09:57:03.543963+0000","last_unstale":"2023-10-16T09:57:03.545757+0000","last_undegraded":"2023-10-16T09:57:03.545757+0000","last_fullsized":"2023-10-16T09:57:03.545757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:24.213653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543426+0000","last_change":"2023-10-16T09:57:03.541179+0000","last_active":"2023-10-16T09:57:03.543426+0000","last_peered":"2023-10-16T09:57:03.543426+0000","last_clean":"2023-10-16T09:57:03.543426+0000","last_became_active":"2023-10-16T09:57:03.539053+0000","last_became_peered":"2023-10-16T09:57:03.539053+0000","last_unstale":"2023-10-16T09:57:03.543426+0000","last_undegraded":"2023-10-16T09:57:03.543426+0000","last_fullsized":"2023-10-16T09:57:03.543426+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:20.158090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.541936+0000","last_change":"2023-10-16T09:57:03.541376+0000","last_active":"2023-10-16T09:57:03.541936+0000","last_peered":"2023-10-16T09:57:03.541936+0000","last_clean":"2023-10-16T09:57:03.541936+0000","last_became_active":"2023-10-16T09:57:03.540991+0000","last_became_peered":"2023-10-16T09:57:03.540991+0000","last_unstale":"2023-10-16T09:57:03.541936+0000","last_undegraded":"2023-10-16T09:57:03.541936+0000","last_fullsized":"2023-10-16T09:57:03.541936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:44.313131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544513+0000","last_change":"2023-10-16T09:57:03.541998+0000","last_active":"2023-10-16T09:57:03.544513+0000","last_peered":"2023-10-16T09:57:03.544513+0000","last_clean":"2023-10-16T09:57:03.544513+0000","last_became_active":"2023-10-16T09:57:03.541733+0000","last_became_peered":"2023-10-16T09:57:03.541733+0000","last_unstale":"2023-10-16T09:57:03.544513+0000","last_undegraded":"2023-10-16T09:57:03.544513+0000","last_fullsized":"2023-10-16T09:57:03.544513+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:12:21.668013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545336+0000","last_change":"2023-10-16T09:57:03.542489+0000","last_active":"2023-10-16T09:57:03.545336+0000","last_peered":"2023-10-16T09:57:03.545336+0000","last_clean":"2023-10-16T09:57:03.545336+0000","last_became_active":"2023-10-16T09:57:03.540455+0000","last_became_peered":"2023-10-16T09:57:03.540455+0000","last_unstale":"2023-10-16T09:57:03.545336+0000","last_undegraded":"2023-10-16T09:57:03.545336+0000","last_fullsized":"2023-10-16T09:57:03.545336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:17.011859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537870+0000","last_change":"2023-10-16T09:57:03.536927+0000","last_active":"2023-10-16T09:57:03.537870+0000","last_peered":"2023-10-16T09:57:03.537870+0000","last_clean":"2023-10-16T09:57:03.537870+0000","last_became_active":"2023-10-16T09:57:03.536452+0000","last_became_peered":"2023-10-16T09:57:03.536452+0000","last_unstale":"2023-10-16T09:57:03.537870+0000","last_undegraded":"2023-10-16T09:57:03.537870+0000","last_fullsized":"2023-10-16T09:57:03.537870+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:16:20.158090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545014+0000","last_change":"2023-10-16T09:57:03.537437+0000","last_active":"2023-10-16T09:57:03.545014+0000","last_peered":"2023-10-16T09:57:03.545014+0000","last_clean":"2023-10-16T09:57:03.545014+0000","last_became_active":"2023-10-16T09:57:03.536929+0000","last_became_peered":"2023-10-16T09:57:03.536929+0000","last_unstale":"2023-10-16T09:57:03.545014+0000","last_undegraded":"2023-10-16T09:57:03.545014+0000","last_fullsized":"2023-10-16T09:57:03.545014+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:27:01.674563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.549859+0000","last_change":"2023-10-16T09:57:03.549454+0000","last_active":"2023-10-16T09:57:03.549859+0000","last_peered":"2023-10-16T09:57:03.549859+0000","last_clean":"2023-10-16T09:57:03.549859+0000","last_became_active":"2023-10-16T09:57:03.549249+0000","last_became_peered":"2023-10-16T09:57:03.549249+0000","last_unstale":"2023-10-16T09:57:03.549859+0000","last_undegraded":"2023-10-16T09:57:03.549859+0000","last_fullsized":"2023-10-16T09:57:03.549859+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:48.406276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542060+0000","last_change":"2023-10-16T09:57:03.540217+0000","last_active":"2023-10-16T09:57:03.542060+0000","last_peered":"2023-10-16T09:57:03.542060+0000","last_clean":"2023-10-16T09:57:03.542060+0000","last_became_active":"2023-10-16T09:57:03.539940+0000","last_became_peered":"2023-10-16T09:57:03.539940+0000","last_unstale":"2023-10-16T09:57:03.542060+0000","last_undegraded":"2023-10-16T09:57:03.542060+0000","last_fullsized":"2023-10-16T09:57:03.542060+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:25.347861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"66'4","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611342+0000","last_change":"2023-10-16T09:57:02.614048+0000","last_active":"2023-10-16T09:57:03.611342+0000","last_peered":"2023-10-16T09:57:03.611342+0000","last_clean":"2023-10-16T09:57:03.611342+0000","last_became_active":"2023-10-16T09:57:02.613493+0000","last_became_peered":"2023-10-16T09:57:02.613493+0000","last_unstale":"2023-10-16T09:57:03.611342+0000","last_undegraded":"2023-10-16T09:57:03.611342+0000","last_fullsized":"2023-10-16T09:57:03.611342+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:10.790453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543293+0000","last_change":"2023-10-16T09:57:03.539869+0000","last_active":"2023-10-16T09:57:03.543293+0000","last_peered":"2023-10-16T09:57:03.543293+0000","last_clean":"2023-10-16T09:57:03.543293+0000","last_became_active":"2023-10-16T09:57:03.539480+0000","last_became_peered":"2023-10-16T09:57:03.539480+0000","last_unstale":"2023-10-16T09:57:03.543293+0000","last_undegraded":"2023-10-16T09:57:03.543293+0000","last_fullsized":"2023-10-16T09:57:03.543293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:08:07.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613481+0000","last_change":"2023-10-16T09:57:02.616346+0000","last_active":"2023-10-16T09:57:03.613481+0000","last_peered":"2023-10-16T09:57:03.613481+0000","last_clean":"2023-10-16T09:57:03.613481+0000","last_became_active":"2023-10-16T09:57:02.615999+0000","last_became_peered":"2023-10-16T09:57:02.615999+0000","last_unstale":"2023-10-16T09:57:03.613481+0000","last_undegraded":"2023-10-16T09:57:03.613481+0000","last_fullsized":"2023-10-16T09:57:03.613481+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:33.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544662+0000","last_change":"2023-10-16T09:57:03.542777+0000","last_active":"2023-10-16T09:57:03.544662+0000","last_peered":"2023-10-16T09:57:03.544662+0000","last_clean":"2023-10-16T09:57:03.544662+0000","last_became_active":"2023-10-16T09:57:03.542461+0000","last_became_peered":"2023-10-16T09:57:03.542461+0000","last_unstale":"2023-10-16T09:57:03.544662+0000","last_undegraded":"2023-10-16T09:57:03.544662+0000","last_fullsized":"2023-10-16T09:57:03.544662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:29:02.805935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613637+0000","last_change":"2023-10-16T09:57:02.613855+0000","last_active":"2023-10-16T09:57:03.613637+0000","last_peered":"2023-10-16T09:57:03.613637+0000","last_clean":"2023-10-16T09:57:03.613637+0000","last_became_active":"2023-10-16T09:57:02.613664+0000","last_became_peered":"2023-10-16T09:57:02.613664+0000","last_unstale":"2023-10-16T09:57:03.613637+0000","last_undegraded":"2023-10-16T09:57:03.613637+0000","last_fullsized":"2023-10-16T09:57:03.613637+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:11:43.214775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547042+0000","last_change":"2023-10-16T09:57:03.542012+0000","last_active":"2023-10-16T09:57:03.547042+0000","last_peered":"2023-10-16T09:57:03.547042+0000","last_clean":"2023-10-16T09:57:03.547042+0000","last_became_active":"2023-10-16T09:57:03.541736+0000","last_became_peered":"2023-10-16T09:57:03.541736+0000","last_unstale":"2023-10-16T09:57:03.547042+0000","last_undegraded":"2023-10-16T09:57:03.547042+0000","last_fullsized":"2023-10-16T09:57:03.547042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:39.770670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611020+0000","last_change":"2023-10-16T09:57:02.619617+0000","last_active":"2023-10-16T09:57:03.611020+0000","last_peered":"2023-10-16T09:57:03.611020+0000","last_clean":"2023-10-16T09:57:03.611020+0000","last_became_active":"2023-10-16T09:57:02.619399+0000","last_became_peered":"2023-10-16T09:57:02.619399+0000","last_unstale":"2023-10-16T09:57:03.611020+0000","last_undegraded":"2023-10-16T09:57:03.611020+0000","last_fullsized":"2023-10-16T09:57:03.611020+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:46.237844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542447+0000","last_change":"2023-10-16T09:57:03.538893+0000","last_active":"2023-10-16T09:57:03.542447+0000","last_peered":"2023-10-16T09:57:03.542447+0000","last_clean":"2023-10-16T09:57:03.542447+0000","last_became_active":"2023-10-16T09:57:03.538590+0000","last_became_peered":"2023-10-16T09:57:03.538590+0000","last_unstale":"2023-10-16T09:57:03.542447+0000","last_undegraded":"2023-10-16T09:57:03.542447+0000","last_fullsized":"2023-10-16T09:57:03.542447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.534359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.614362+0000","last_change":"2023-10-16T09:57:02.619914+0000","last_active":"2023-10-16T09:57:03.614362+0000","last_peered":"2023-10-16T09:57:03.614362+0000","last_clean":"2023-10-16T09:57:03.614362+0000","last_became_active":"2023-10-16T09:57:02.619654+0000","last_became_peered":"2023-10-16T09:57:02.619654+0000","last_unstale":"2023-10-16T09:57:03.614362+0000","last_undegraded":"2023-10-16T09:57:03.614362+0000","last_fullsized":"2023-10-16T09:57:03.614362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:20:53.160834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.537751+0000","last_change":"2023-10-16T09:57:03.536603+0000","last_active":"2023-10-16T09:57:03.537751+0000","last_peered":"2023-10-16T09:57:03.537751+0000","last_clean":"2023-10-16T09:57:03.537751+0000","last_became_active":"2023-10-16T09:57:03.536325+0000","last_became_peered":"2023-10-16T09:57:03.536325+0000","last_unstale":"2023-10-16T09:57:03.537751+0000","last_undegraded":"2023-10-16T09:57:03.537751+0000","last_fullsized":"2023-10-16T09:57:03.537751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:17.967654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.610519+0000","last_change":"2023-10-16T09:57:02.614065+0000","last_active":"2023-10-16T09:57:03.610519+0000","last_peered":"2023-10-16T09:57:03.610519+0000","last_clean":"2023-10-16T09:57:03.610519+0000","last_became_active":"2023-10-16T09:57:02.613779+0000","last_became_peered":"2023-10-16T09:57:02.613779+0000","last_unstale":"2023-10-16T09:57:03.610519+0000","last_undegraded":"2023-10-16T09:57:03.610519+0000","last_fullsized":"2023-10-16T09:57:03.610519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:08.774560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548135+0000","last_change":"2023-10-16T09:57:03.543837+0000","last_active":"2023-10-16T09:57:03.548135+0000","last_peered":"2023-10-16T09:57:03.548135+0000","last_clean":"2023-10-16T09:57:03.548135+0000","last_became_active":"2023-10-16T09:57:03.543616+0000","last_became_peered":"2023-10-16T09:57:03.543616+0000","last_unstale":"2023-10-16T09:57:03.548135+0000","last_undegraded":"2023-10-16T09:57:03.548135+0000","last_fullsized":"2023-10-16T09:57:03.548135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:29:58.824360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.534784+0000","last_change":"2023-10-16T09:57:02.615083+0000","last_active":"2023-10-16T09:57:03.534784+0000","last_peered":"2023-10-16T09:57:03.534784+0000","last_clean":"2023-10-16T09:57:03.534784+0000","last_became_active":"2023-10-16T09:57:02.614789+0000","last_became_peered":"2023-10-16T09:57:02.614789+0000","last_unstale":"2023-10-16T09:57:03.534784+0000","last_undegraded":"2023-10-16T09:57:03.534784+0000","last_fullsized":"2023-10-16T09:57:03.534784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:21.527620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.539773+0000","last_change":"2023-10-16T09:57:03.538989+0000","last_active":"2023-10-16T09:57:03.539773+0000","last_peered":"2023-10-16T09:57:03.539773+0000","last_clean":"2023-10-16T09:57:03.539773+0000","last_became_active":"2023-10-16T09:57:03.538719+0000","last_became_peered":"2023-10-16T09:57:03.538719+0000","last_unstale":"2023-10-16T09:57:03.539773+0000","last_undegraded":"2023-10-16T09:57:03.539773+0000","last_fullsized":"2023-10-16T09:57:03.539773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:58.548809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.529775+0000","last_change":"2023-10-16T09:57:02.618144+0000","last_active":"2023-10-16T09:57:03.529775+0000","last_peered":"2023-10-16T09:57:03.529775+0000","last_clean":"2023-10-16T09:57:03.529775+0000","last_became_active":"2023-10-16T09:57:02.617740+0000","last_became_peered":"2023-10-16T09:57:02.617740+0000","last_unstale":"2023-10-16T09:57:03.529775+0000","last_undegraded":"2023-10-16T09:57:03.529775+0000","last_fullsized":"2023-10-16T09:57:03.529775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:56.994666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542996+0000","last_change":"2023-10-16T09:57:03.540825+0000","last_active":"2023-10-16T09:57:03.542996+0000","last_peered":"2023-10-16T09:57:03.542996+0000","last_clean":"2023-10-16T09:57:03.542996+0000","last_became_active":"2023-10-16T09:57:03.540557+0000","last_became_peered":"2023-10-16T09:57:03.540557+0000","last_unstale":"2023-10-16T09:57:03.542996+0000","last_undegraded":"2023-10-16T09:57:03.542996+0000","last_fullsized":"2023-10-16T09:57:03.542996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:03.569703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.526606+0000","last_change":"2023-10-16T09:57:02.616421+0000","last_active":"2023-10-16T09:57:03.526606+0000","last_peered":"2023-10-16T09:57:03.526606+0000","last_clean":"2023-10-16T09:57:03.526606+0000","last_became_active":"2023-10-16T09:57:02.616141+0000","last_became_peered":"2023-10-16T09:57:02.616141+0000","last_unstale":"2023-10-16T09:57:03.526606+0000","last_undegraded":"2023-10-16T09:57:03.526606+0000","last_fullsized":"2023-10-16T09:57:03.526606+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:56.856285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547450+0000","last_change":"2023-10-16T09:57:03.546464+0000","last_active":"2023-10-16T09:57:03.547450+0000","last_peered":"2023-10-16T09:57:03.547450+0000","last_clean":"2023-10-16T09:57:03.547450+0000","last_became_active":"2023-10-16T09:57:03.546194+0000","last_became_peered":"2023-10-16T09:57:03.546194+0000","last_unstale":"2023-10-16T09:57:03.547450+0000","last_undegraded":"2023-10-16T09:57:03.547450+0000","last_fullsized":"2023-10-16T09:57:03.547450+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:39.983421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.528789+0000","last_change":"2023-10-16T09:57:02.620311+0000","last_active":"2023-10-16T09:57:03.528789+0000","last_peered":"2023-10-16T09:57:03.528789+0000","last_clean":"2023-10-16T09:57:03.528789+0000","last_became_active":"2023-10-16T09:57:02.620059+0000","last_became_peered":"2023-10-16T09:57:02.620059+0000","last_unstale":"2023-10-16T09:57:03.528789+0000","last_undegraded":"2023-10-16T09:57:03.528789+0000","last_fullsized":"2023-10-16T09:57:03.528789+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:20:43.320545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546313+0000","last_change":"2023-10-16T09:57:03.544882+0000","last_active":"2023-10-16T09:57:03.546313+0000","last_peered":"2023-10-16T09:57:03.546313+0000","last_clean":"2023-10-16T09:57:03.546313+0000","last_became_active":"2023-10-16T09:57:03.544598+0000","last_became_peered":"2023-10-16T09:57:03.544598+0000","last_unstale":"2023-10-16T09:57:03.546313+0000","last_undegraded":"2023-10-16T09:57:03.546313+0000","last_fullsized":"2023-10-16T09:57:03.546313+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:09.800917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"66'3","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.614186+0000","last_change":"2023-10-16T09:57:02.615041+0000","last_active":"2023-10-16T09:57:03.614186+0000","last_peered":"2023-10-16T09:57:03.614186+0000","last_clean":"2023-10-16T09:57:03.614186+0000","last_became_active":"2023-10-16T09:57:02.614767+0000","last_became_peered":"2023-10-16T09:57:02.614767+0000","last_unstale":"2023-10-16T09:57:03.614186+0000","last_undegraded":"2023-10-16T09:57:03.614186+0000","last_fullsized":"2023-10-16T09:57:03.614186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:29:57.815719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542323+0000","last_change":"2023-10-16T09:57:03.538117+0000","last_active":"2023-10-16T09:57:03.542323+0000","last_peered":"2023-10-16T09:57:03.542323+0000","last_clean":"2023-10-16T09:57:03.542323+0000","last_became_active":"2023-10-16T09:57:03.537208+0000","last_became_peered":"2023-10-16T09:57:03.537208+0000","last_unstale":"2023-10-16T09:57:03.542323+0000","last_undegraded":"2023-10-16T09:57:03.542323+0000","last_fullsized":"2023-10-16T09:57:03.542323+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:32.636528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"66'6","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.616009+0000","last_change":"2023-10-16T09:57:02.614842+0000","last_active":"2023-10-16T09:57:03.616009+0000","last_peered":"2023-10-16T09:57:03.616009+0000","last_clean":"2023-10-16T09:57:03.616009+0000","last_became_active":"2023-10-16T09:57:02.614567+0000","last_became_peered":"2023-10-16T09:57:02.614567+0000","last_unstale":"2023-10-16T09:57:03.616009+0000","last_undegraded":"2023-10-16T09:57:03.616009+0000","last_fullsized":"2023-10-16T09:57:03.616009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:06:00.596063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543164+0000","last_change":"2023-10-16T09:57:03.536597+0000","last_active":"2023-10-16T09:57:03.543164+0000","last_peered":"2023-10-16T09:57:03.543164+0000","last_clean":"2023-10-16T09:57:03.543164+0000","last_became_active":"2023-10-16T09:57:03.536336+0000","last_became_peered":"2023-10-16T09:57:03.536336+0000","last_unstale":"2023-10-16T09:57:03.543164+0000","last_undegraded":"2023-10-16T09:57:03.543164+0000","last_fullsized":"2023-10-16T09:57:03.543164+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:53.843985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.610621+0000","last_change":"2023-10-16T09:57:02.616859+0000","last_active":"2023-10-16T09:57:03.610621+0000","last_peered":"2023-10-16T09:57:03.610621+0000","last_clean":"2023-10-16T09:57:03.610621+0000","last_became_active":"2023-10-16T09:57:02.616570+0000","last_became_peered":"2023-10-16T09:57:02.616570+0000","last_unstale":"2023-10-16T09:57:03.610621+0000","last_undegraded":"2023-10-16T09:57:03.610621+0000","last_fullsized":"2023-10-16T09:57:03.610621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:47.795378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545266+0000","last_change":"2023-10-16T09:57:03.540132+0000","last_active":"2023-10-16T09:57:03.545266+0000","last_peered":"2023-10-16T09:57:03.545266+0000","last_clean":"2023-10-16T09:57:03.545266+0000","last_became_active":"2023-10-16T09:57:03.537599+0000","last_became_peered":"2023-10-16T09:57:03.537599+0000","last_unstale":"2023-10-16T09:57:03.545266+0000","last_undegraded":"2023-10-16T09:57:03.545266+0000","last_fullsized":"2023-10-16T09:57:03.545266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:02.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.538513+0000","last_change":"2023-10-16T09:57:02.622456+0000","last_active":"2023-10-16T09:57:03.538513+0000","last_peered":"2023-10-16T09:57:03.538513+0000","last_clean":"2023-10-16T09:57:03.538513+0000","last_became_active":"2023-10-16T09:57:02.622127+0000","last_became_peered":"2023-10-16T09:57:02.622127+0000","last_unstale":"2023-10-16T09:57:03.538513+0000","last_undegraded":"2023-10-16T09:57:03.538513+0000","last_fullsized":"2023-10-16T09:57:03.538513+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:05.254985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543406+0000","last_change":"2023-10-16T09:57:03.540378+0000","last_active":"2023-10-16T09:57:03.543406+0000","last_peered":"2023-10-16T09:57:03.543406+0000","last_clean":"2023-10-16T09:57:03.543406+0000","last_became_active":"2023-10-16T09:57:03.539016+0000","last_became_peered":"2023-10-16T09:57:03.539016+0000","last_unstale":"2023-10-16T09:57:03.543406+0000","last_undegraded":"2023-10-16T09:57:03.543406+0000","last_fullsized":"2023-10-16T09:57:03.543406+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:59.566663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.526798+0000","last_change":"2023-10-16T09:57:02.614023+0000","last_active":"2023-10-16T09:57:03.526798+0000","last_peered":"2023-10-16T09:57:03.526798+0000","last_clean":"2023-10-16T09:57:03.526798+0000","last_became_active":"2023-10-16T09:57:02.613724+0000","last_became_peered":"2023-10-16T09:57:02.613724+0000","last_unstale":"2023-10-16T09:57:03.526798+0000","last_undegraded":"2023-10-16T09:57:03.526798+0000","last_fullsized":"2023-10-16T09:57:03.526798+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:52.267242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546556+0000","last_change":"2023-10-16T09:57:03.545703+0000","last_active":"2023-10-16T09:57:03.546556+0000","last_peered":"2023-10-16T09:57:03.546556+0000","last_clean":"2023-10-16T09:57:03.546556+0000","last_became_active":"2023-10-16T09:57:03.545504+0000","last_became_peered":"2023-10-16T09:57:03.545504+0000","last_unstale":"2023-10-16T09:57:03.546556+0000","last_undegraded":"2023-10-16T09:57:03.546556+0000","last_fullsized":"2023-10-16T09:57:03.546556+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:11.429284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"66'8","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613602+0000","last_change":"2023-10-16T09:57:02.614852+0000","last_active":"2023-10-16T09:57:03.613602+0000","last_peered":"2023-10-16T09:57:03.613602+0000","last_clean":"2023-10-16T09:57:03.613602+0000","last_became_active":"2023-10-16T09:57:02.614607+0000","last_became_peered":"2023-10-16T09:57:02.614607+0000","last_unstale":"2023-10-16T09:57:03.613602+0000","last_undegraded":"2023-10-16T09:57:03.613602+0000","last_fullsized":"2023-10-16T09:57:03.613602+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:29.506540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548906+0000","last_change":"2023-10-16T09:57:03.548143+0000","last_active":"2023-10-16T09:57:03.548906+0000","last_peered":"2023-10-16T09:57:03.548906+0000","last_clean":"2023-10-16T09:57:03.548906+0000","last_became_active":"2023-10-16T09:57:03.547880+0000","last_became_peered":"2023-10-16T09:57:03.547880+0000","last_unstale":"2023-10-16T09:57:03.548906+0000","last_undegraded":"2023-10-16T09:57:03.548906+0000","last_fullsized":"2023-10-16T09:57:03.548906+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:53.212827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611183+0000","last_change":"2023-10-16T09:57:02.619514+0000","last_active":"2023-10-16T09:57:03.611183+0000","last_peered":"2023-10-16T09:57:03.611183+0000","last_clean":"2023-10-16T09:57:03.611183+0000","last_became_active":"2023-10-16T09:57:02.619257+0000","last_became_peered":"2023-10-16T09:57:02.619257+0000","last_unstale":"2023-10-16T09:57:03.611183+0000","last_undegraded":"2023-10-16T09:57:03.611183+0000","last_fullsized":"2023-10-16T09:57:03.611183+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:54.986229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.539369+0000","last_change":"2023-10-16T09:57:03.538345+0000","last_active":"2023-10-16T09:57:03.539369+0000","last_peered":"2023-10-16T09:57:03.539369+0000","last_clean":"2023-10-16T09:57:03.539369+0000","last_became_active":"2023-10-16T09:57:03.538076+0000","last_became_peered":"2023-10-16T09:57:03.538076+0000","last_unstale":"2023-10-16T09:57:03.539369+0000","last_undegraded":"2023-10-16T09:57:03.539369+0000","last_fullsized":"2023-10-16T09:57:03.539369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:24.030251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.614934+0000","last_change":"2023-10-16T09:57:02.616270+0000","last_active":"2023-10-16T09:57:03.614934+0000","last_peered":"2023-10-16T09:57:03.614934+0000","last_clean":"2023-10-16T09:57:03.614934+0000","last_became_active":"2023-10-16T09:57:02.616015+0000","last_became_peered":"2023-10-16T09:57:02.616015+0000","last_unstale":"2023-10-16T09:57:03.614934+0000","last_undegraded":"2023-10-16T09:57:03.614934+0000","last_fullsized":"2023-10-16T09:57:03.614934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:42.361094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542313+0000","last_change":"2023-10-16T09:57:03.537753+0000","last_active":"2023-10-16T09:57:03.542313+0000","last_peered":"2023-10-16T09:57:03.542313+0000","last_clean":"2023-10-16T09:57:03.542313+0000","last_became_active":"2023-10-16T09:57:03.537462+0000","last_became_peered":"2023-10-16T09:57:03.537462+0000","last_unstale":"2023-10-16T09:57:03.542313+0000","last_undegraded":"2023-10-16T09:57:03.542313+0000","last_fullsized":"2023-10-16T09:57:03.542313+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:34.384510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.613906+0000","last_change":"2023-10-16T09:57:02.614932+0000","last_active":"2023-10-16T09:57:03.613906+0000","last_peered":"2023-10-16T09:57:03.613906+0000","last_clean":"2023-10-16T09:57:03.613906+0000","last_became_active":"2023-10-16T09:57:02.614667+0000","last_became_peered":"2023-10-16T09:57:02.614667+0000","last_unstale":"2023-10-16T09:57:03.613906+0000","last_undegraded":"2023-10-16T09:57:03.613906+0000","last_fullsized":"2023-10-16T09:57:03.613906+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545162+0000","last_change":"2023-10-16T09:57:03.544679+0000","last_active":"2023-10-16T09:57:03.545162+0000","last_peered":"2023-10-16T09:57:03.545162+0000","last_clean":"2023-10-16T09:57:03.545162+0000","last_became_active":"2023-10-16T09:57:03.544402+0000","last_became_peered":"2023-10-16T09:57:03.544402+0000","last_unstale":"2023-10-16T09:57:03.545162+0000","last_undegraded":"2023-10-16T09:57:03.545162+0000","last_fullsized":"2023-10-16T09:57:03.545162+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:09:56.421946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.534659+0000","last_change":"2023-10-16T09:57:02.616040+0000","last_active":"2023-10-16T09:57:03.534659+0000","last_peered":"2023-10-16T09:57:03.534659+0000","last_clean":"2023-10-16T09:57:03.534659+0000","last_became_active":"2023-10-16T09:57:02.615773+0000","last_became_peered":"2023-10-16T09:57:02.615773+0000","last_unstale":"2023-10-16T09:57:03.534659+0000","last_undegraded":"2023-10-16T09:57:03.534659+0000","last_fullsized":"2023-10-16T09:57:03.534659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:34.219976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.548054+0000","last_change":"2023-10-16T09:57:03.547154+0000","last_active":"2023-10-16T09:57:03.548054+0000","last_peered":"2023-10-16T09:57:03.548054+0000","last_clean":"2023-10-16T09:57:03.548054+0000","last_became_active":"2023-10-16T09:57:03.546842+0000","last_became_peered":"2023-10-16T09:57:03.546842+0000","last_unstale":"2023-10-16T09:57:03.548054+0000","last_undegraded":"2023-10-16T09:57:03.548054+0000","last_fullsized":"2023-10-16T09:57:03.548054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:34.354433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.615036+0000","last_change":"2023-10-16T09:57:02.617412+0000","last_active":"2023-10-16T09:57:03.615036+0000","last_peered":"2023-10-16T09:57:03.615036+0000","last_clean":"2023-10-16T09:57:03.615036+0000","last_became_active":"2023-10-16T09:57:02.617141+0000","last_became_peered":"2023-10-16T09:57:02.617141+0000","last_unstale":"2023-10-16T09:57:03.615036+0000","last_undegraded":"2023-10-16T09:57:03.615036+0000","last_fullsized":"2023-10-16T09:57:03.615036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:20:43.320545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543397+0000","last_change":"2023-10-16T09:57:03.540745+0000","last_active":"2023-10-16T09:57:03.543397+0000","last_peered":"2023-10-16T09:57:03.543397+0000","last_clean":"2023-10-16T09:57:03.543397+0000","last_became_active":"2023-10-16T09:57:03.539364+0000","last_became_peered":"2023-10-16T09:57:03.539364+0000","last_unstale":"2023-10-16T09:57:03.543397+0000","last_undegraded":"2023-10-16T09:57:03.543397+0000","last_fullsized":"2023-10-16T09:57:03.543397+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:50:01.772995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535056+0000","last_change":"2023-10-16T09:57:02.615394+0000","last_active":"2023-10-16T09:57:03.535056+0000","last_peered":"2023-10-16T09:57:03.535056+0000","last_clean":"2023-10-16T09:57:03.535056+0000","last_became_active":"2023-10-16T09:57:02.615117+0000","last_became_peered":"2023-10-16T09:57:02.615117+0000","last_unstale":"2023-10-16T09:57:03.535056+0000","last_undegraded":"2023-10-16T09:57:03.535056+0000","last_fullsized":"2023-10-16T09:57:03.535056+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:33.345792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545889+0000","last_change":"2023-10-16T09:57:03.543453+0000","last_active":"2023-10-16T09:57:03.545889+0000","last_peered":"2023-10-16T09:57:03.545889+0000","last_clean":"2023-10-16T09:57:03.545889+0000","last_became_active":"2023-10-16T09:57:03.543204+0000","last_became_peered":"2023-10-16T09:57:03.543204+0000","last_unstale":"2023-10-16T09:57:03.545889+0000","last_undegraded":"2023-10-16T09:57:03.545889+0000","last_fullsized":"2023-10-16T09:57:03.545889+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:33:35.109415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.530640+0000","last_change":"2023-10-16T09:57:02.620572+0000","last_active":"2023-10-16T09:57:03.530640+0000","last_peered":"2023-10-16T09:57:03.530640+0000","last_clean":"2023-10-16T09:57:03.530640+0000","last_became_active":"2023-10-16T09:57:02.620313+0000","last_became_peered":"2023-10-16T09:57:02.620313+0000","last_unstale":"2023-10-16T09:57:03.530640+0000","last_undegraded":"2023-10-16T09:57:03.530640+0000","last_fullsized":"2023-10-16T09:57:03.530640+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:42.440435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.543073+0000","last_change":"2023-10-16T09:57:03.541477+0000","last_active":"2023-10-16T09:57:03.543073+0000","last_peered":"2023-10-16T09:57:03.543073+0000","last_clean":"2023-10-16T09:57:03.543073+0000","last_became_active":"2023-10-16T09:57:03.540938+0000","last_became_peered":"2023-10-16T09:57:03.540938+0000","last_unstale":"2023-10-16T09:57:03.543073+0000","last_undegraded":"2023-10-16T09:57:03.543073+0000","last_fullsized":"2023-10-16T09:57:03.543073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:03.569703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.528211+0000","last_change":"2023-10-16T09:57:02.611493+0000","last_active":"2023-10-16T09:57:03.528211+0000","last_peered":"2023-10-16T09:57:03.528211+0000","last_clean":"2023-10-16T09:57:03.528211+0000","last_became_active":"2023-10-16T09:57:02.611205+0000","last_became_peered":"2023-10-16T09:57:02.611205+0000","last_unstale":"2023-10-16T09:57:03.528211+0000","last_undegraded":"2023-10-16T09:57:03.528211+0000","last_fullsized":"2023-10-16T09:57:03.528211+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:50:30.035578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.549517+0000","last_change":"2023-10-16T09:57:03.549165+0000","last_active":"2023-10-16T09:57:03.549517+0000","last_peered":"2023-10-16T09:57:03.549517+0000","last_clean":"2023-10-16T09:57:03.549517+0000","last_became_active":"2023-10-16T09:57:03.548897+0000","last_became_peered":"2023-10-16T09:57:03.548897+0000","last_unstale":"2023-10-16T09:57:03.549517+0000","last_undegraded":"2023-10-16T09:57:03.549517+0000","last_fullsized":"2023-10-16T09:57:03.549517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:30.515181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"66'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.611485+0000","last_change":"2023-10-16T09:57:02.622458+0000","last_active":"2023-10-16T09:57:03.611485+0000","last_peered":"2023-10-16T09:57:03.611485+0000","last_clean":"2023-10-16T09:57:03.611485+0000","last_became_active":"2023-10-16T09:57:02.622132+0000","last_became_peered":"2023-10-16T09:57:02.622132+0000","last_unstale":"2023-10-16T09:57:03.611485+0000","last_undegraded":"2023-10-16T09:57:03.611485+0000","last_fullsized":"2023-10-16T09:57:03.611485+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:39.242322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.538060+0000","last_change":"2023-10-16T09:57:03.536803+0000","last_active":"2023-10-16T09:57:03.538060+0000","last_peered":"2023-10-16T09:57:03.538060+0000","last_clean":"2023-10-16T09:57:03.538060+0000","last_became_active":"2023-10-16T09:57:03.536504+0000","last_became_peered":"2023-10-16T09:57:03.536504+0000","last_unstale":"2023-10-16T09:57:03.538060+0000","last_undegraded":"2023-10-16T09:57:03.538060+0000","last_fullsized":"2023-10-16T09:57:03.538060+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:06.263626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.527217+0000","last_change":"2023-10-16T09:57:02.605820+0000","last_active":"2023-10-16T09:57:03.527217+0000","last_peered":"2023-10-16T09:57:03.527217+0000","last_clean":"2023-10-16T09:57:03.527217+0000","last_became_active":"2023-10-16T09:57:02.605364+0000","last_became_peered":"2023-10-16T09:57:02.605364+0000","last_unstale":"2023-10-16T09:57:03.527217+0000","last_undegraded":"2023-10-16T09:57:03.527217+0000","last_fullsized":"2023-10-16T09:57:03.527217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:38.762029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547928+0000","last_change":"2023-10-16T09:57:03.543926+0000","last_active":"2023-10-16T09:57:03.547928+0000","last_peered":"2023-10-16T09:57:03.547928+0000","last_clean":"2023-10-16T09:57:03.547928+0000","last_became_active":"2023-10-16T09:57:03.543639+0000","last_became_peered":"2023-10-16T09:57:03.543639+0000","last_unstale":"2023-10-16T09:57:03.547928+0000","last_undegraded":"2023-10-16T09:57:03.547928+0000","last_fullsized":"2023-10-16T09:57:03.547928+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:52.358095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.532257+0000","last_change":"2023-10-16T09:57:02.613008+0000","last_active":"2023-10-16T09:57:03.532257+0000","last_peered":"2023-10-16T09:57:03.532257+0000","last_clean":"2023-10-16T09:57:03.532257+0000","last_became_active":"2023-10-16T09:57:02.612700+0000","last_became_peered":"2023-10-16T09:57:02.612700+0000","last_unstale":"2023-10-16T09:57:03.532257+0000","last_undegraded":"2023-10-16T09:57:03.532257+0000","last_fullsized":"2023-10-16T09:57:03.532257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:05.422941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.546031+0000","last_change":"2023-10-16T09:57:03.543332+0000","last_active":"2023-10-16T09:57:03.546031+0000","last_peered":"2023-10-16T09:57:03.546031+0000","last_clean":"2023-10-16T09:57:03.546031+0000","last_became_active":"2023-10-16T09:57:03.543083+0000","last_became_peered":"2023-10-16T09:57:03.543083+0000","last_unstale":"2023-10-16T09:57:03.546031+0000","last_undegraded":"2023-10-16T09:57:03.546031+0000","last_fullsized":"2023-10-16T09:57:03.546031+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:14:22.011175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.532962+0000","last_change":"2023-10-16T09:57:02.614732+0000","last_active":"2023-10-16T09:57:03.532962+0000","last_peered":"2023-10-16T09:57:03.532962+0000","last_clean":"2023-10-16T09:57:03.532962+0000","last_became_active":"2023-10-16T09:57:02.614451+0000","last_became_peered":"2023-10-16T09:57:02.614451+0000","last_unstale":"2023-10-16T09:57:03.532962+0000","last_undegraded":"2023-10-16T09:57:03.532962+0000","last_fullsized":"2023-10-16T09:57:03.532962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:28:07.404143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.544978+0000","last_change":"2023-10-16T09:57:03.543563+0000","last_active":"2023-10-16T09:57:03.544978+0000","last_peered":"2023-10-16T09:57:03.544978+0000","last_clean":"2023-10-16T09:57:03.544978+0000","last_became_active":"2023-10-16T09:57:03.543272+0000","last_became_peered":"2023-10-16T09:57:03.543272+0000","last_unstale":"2023-10-16T09:57:03.544978+0000","last_undegraded":"2023-10-16T09:57:03.544978+0000","last_fullsized":"2023-10-16T09:57:03.544978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:06:01.604704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.536127+0000","last_change":"2023-10-16T09:57:02.617260+0000","last_active":"2023-10-16T09:57:03.536127+0000","last_peered":"2023-10-16T09:57:03.536127+0000","last_clean":"2023-10-16T09:57:03.536127+0000","last_became_active":"2023-10-16T09:57:02.614712+0000","last_became_peered":"2023-10-16T09:57:02.614712+0000","last_unstale":"2023-10-16T09:57:03.536127+0000","last_undegraded":"2023-10-16T09:57:03.536127+0000","last_fullsized":"2023-10-16T09:57:03.536127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:52.835344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.545871+0000","last_change":"2023-10-16T09:57:03.542728+0000","last_active":"2023-10-16T09:57:03.545871+0000","last_peered":"2023-10-16T09:57:03.545871+0000","last_clean":"2023-10-16T09:57:03.545871+0000","last_became_active":"2023-10-16T09:57:03.540575+0000","last_became_peered":"2023-10-16T09:57:03.540575+0000","last_unstale":"2023-10-16T09:57:03.545871+0000","last_undegraded":"2023-10-16T09:57:03.545871+0000","last_fullsized":"2023-10-16T09:57:03.545871+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:51.431983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.528593+0000","last_change":"2023-10-16T09:57:02.614970+0000","last_active":"2023-10-16T09:57:03.528593+0000","last_peered":"2023-10-16T09:57:03.528593+0000","last_clean":"2023-10-16T09:57:03.528593+0000","last_became_active":"2023-10-16T09:57:02.614672+0000","last_became_peered":"2023-10-16T09:57:02.614672+0000","last_unstale":"2023-10-16T09:57:03.528593+0000","last_undegraded":"2023-10-16T09:57:03.528593+0000","last_fullsized":"2023-10-16T09:57:03.528593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:04:04.321220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.547915+0000","last_change":"2023-10-16T09:57:03.543826+0000","last_active":"2023-10-16T09:57:03.547915+0000","last_peered":"2023-10-16T09:57:03.547915+0000","last_clean":"2023-10-16T09:57:03.547915+0000","last_became_active":"2023-10-16T09:57:03.543559+0000","last_became_peered":"2023-10-16T09:57:03.543559+0000","last_unstale":"2023-10-16T09:57:03.547915+0000","last_undegraded":"2023-10-16T09:57:03.547915+0000","last_fullsized":"2023-10-16T09:57:03.547915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:43.191367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.535069+0000","last_change":"2023-10-16T09:57:02.616106+0000","last_active":"2023-10-16T09:57:03.535069+0000","last_peered":"2023-10-16T09:57:03.535069+0000","last_clean":"2023-10-16T09:57:03.535069+0000","last_became_active":"2023-10-16T09:57:02.615851+0000","last_became_peered":"2023-10-16T09:57:02.615851+0000","last_unstale":"2023-10-16T09:57:03.535069+0000","last_undegraded":"2023-10-16T09:57:03.535069+0000","last_fullsized":"2023-10-16T09:57:03.535069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:42.182726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.542053+0000","last_change":"2023-10-16T09:57:03.540665+0000","last_active":"2023-10-16T09:57:03.542053+0000","last_peered":"2023-10-16T09:57:03.542053+0000","last_clean":"2023-10-16T09:57:03.542053+0000","last_became_active":"2023-10-16T09:57:03.540390+0000","last_became_peered":"2023-10-16T09:57:03.540390+0000","last_unstale":"2023-10-16T09:57:03.542053+0000","last_undegraded":"2023-10-16T09:57:03.542053+0000","last_fullsized":"2023-10-16T09:57:03.542053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:02.592350+0000","last_clean_scrub_stamp":"2023-10-16T09:57:02.592350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:19.148669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T09:57:03.531853+0000","last_change":"2023-10-16T09:57:02.616890+0000","last_active":"2023-10-16T09:57:03.531853+0000","last_peered":"2023-10-16T09:57:03.531853+0000","last_clean":"2023-10-16T09:57:03.531853+0000","last_became_active":"2023-10-16T09:57:02.616455+0000","last_became_peered":"2023-10-16T09:57:02.616455+0000","last_unstale":"2023-10-16T09:57:03.531853+0000","last_undegraded":"2023-10-16T09:57:03.531853+0000","last_fullsized":"2023-10-16T09:57:03.531853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:57:01.583709+0000","last_clean_scrub_stamp":"2023-10-16T09:57:01.583709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:50:23.686105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"56'139","reported_seq":291,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T09:57:02.600417+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:57:02.600417+0000","last_peered":"2023-10-16T09:57:02.600417+0000","last_clean":"2023-10-16T09:57:02.600417+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:57:02.600417+0000","last_undegraded":"2023-10-16T09:57:02.600417+0000","last_fullsized":"2023-10-16T09:57:02.600417+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575015,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54748,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317092,"statfs":{"total":96636764160,"available":96580702208,"internally_reserved":0,"allocated":1392640,"data_stored":985041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.66900000000000004},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":11.246},"last":0.373}]},{"osd":1,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.61599999999999999},"min":{"1min":0.36199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.75,"5min":0.79800000000000004,"15min":1.1439999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.212},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64600000000000002,"15min":0.63},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66200000000000003,"15min":0.67100000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.58399999999999996}]},{"osd":4,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.51700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":0.439}]},{"osd":5,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50600000000000001,"15min":0.53000000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81100000000000005,"5min":0.82099999999999995,"15min":1.056},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55800000000000005,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34000000000000002},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.59199999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.748,"5min":0.76900000000000002,"15min":0.96399999999999997},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834575015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.69399999999999995,"5min":0.80800000000000005,"15min":0.83199999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.745,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.32100000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.76600000000000001,"5min":0.81799999999999995,"15min":0.83199999999999996},"last":0.69599999999999995}]},{"osd":2,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60099999999999998,"15min":0.57899999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.317},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.495}]},{"osd":3,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59799999999999998,"15min":0.58899999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65100000000000002,"5min":0.78900000000000003,"15min":0.81699999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59599999999999997,"15min":0.59699999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.77100000000000002,"5min":0.80100000000000005,"15min":0.82999999999999996},"last":0.62}]},{"osd":4,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45700000000000002,"15min":0.504},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.71599999999999997,"15min":0.93200000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.5,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":0.67200000000000004,"5min":0.76900000000000002,"15min":0.85399999999999998},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.497,"15min":0.51300000000000001},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33100000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53000000000000003,"15min":0.56999999999999995},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.34599999999999997},"max":{"1min":0.73399999999999999,"5min":0.80000000000000004,"15min":0.89200000000000002},"last":0.71699999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55200000000000005,"15min":0.57199999999999995},"min":{"1min":0.41899999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76800000000000002,"5min":0.84299999999999997,"15min":0.90900000000000003},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.58799999999999997},"min":{"1min":0.38300000000000001,"5min":0.375,"15min":0.374},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.72999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575016,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31944,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339896,"statfs":{"total":96636764160,"available":96604053504,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39900000000000002,"15min":0.39800000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.58199999999999996,"5min":0.65400000000000003,"15min":1.5509999999999999},"last":1.6000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.45400000000000001,"15min":0.436},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.55500000000000005,"5min":0.70199999999999996,"15min":1.593},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39900000000000002,"15min":0.38500000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.55700000000000005,"5min":0.67800000000000005,"15min":0.77400000000000002},"last":4.923},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.216},"max":{"1min":0.625,"5min":0.72899999999999998,"15min":0.81999999999999995},"last":1.712}]},{"osd":3,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.46400000000000002,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.63200000000000001,"5min":0.68799999999999994,"15min":0.69099999999999995},"last":1.984},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.439,"15min":0.42199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":0.61399999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.629},"min":{"1min":0.377,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.80200000000000005,"5min":0.83499999999999996,"15min":0.90600000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.63900000000000001,"15min":0.66100000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.89800000000000002,"5min":0.91400000000000003,"15min":5.141},"last":5.0069999999999997}]},{"osd":5,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65900000000000003,"15min":0.68300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":0.94799999999999995,"15min":5.0789999999999997},"last":4.9749999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40300000000000002},"max":{"1min":0.89000000000000001,"5min":0.93300000000000005,"15min":0.996},"last":1.7769999999999999}]},{"osd":6,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64600000000000002,"15min":0.64100000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.85199999999999998,"5min":0.873,"15min":0.95099999999999996},"last":0.628},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.65200000000000002,"15min":0.67400000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.371},"max":{"1min":0.86099999999999999,"5min":0.97699999999999998,"15min":5.1760000000000002},"last":5.0419999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67100000000000004,"15min":0.66200000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.92000000000000004,"5min":0.96499999999999997,"15min":0.98499999999999999},"last":1.744},{"interface":"front","average":{"1min":0.626,"5min":0.66400000000000003,"15min":0.65800000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.38500000000000001},"max":{"1min":0.84799999999999998,"5min":0.91400000000000003,"15min":1.0329999999999999},"last":0.52900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1372160,"data_stored":977426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.373},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46999999999999997,"15min":0.45100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.217},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47799999999999998,"15min":0.45500000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.624,"15min":0.60799999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33700000000000002},"max":{"1min":0.81299999999999994,"5min":0.92900000000000005,"15min":0.93700000000000006},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.64000000000000001,"15min":0.61699999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64600000000000002,"15min":0.625},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85399999999999998,"5min":0.99299999999999999,"15min":0.998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":5.2240000000000002},"last":0.623}]},{"osd":6,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68899999999999995,"15min":0.66200000000000003},"min":{"1min":0.39900000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94199999999999995,"5min":0.95999999999999996,"15min":0.96799999999999997},"last":0.755}]},{"osd":7,"last update":"Mon Oct 16 09:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64900000000000002,"15min":0.63500000000000001},"min":{"1min":0.35199999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.90600000000000003,"5min":1.02,"15min":1.02},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66000000000000003,"15min":0.63700000000000001},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.73099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575015,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54148,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317692,"statfs":{"total":96636764160,"available":96581316608,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.60199999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41299999999999998,"15min":0.40100000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.502,"5min":0.65400000000000003,"15min":0.67500000000000004},"last":0.317}]},{"osd":1,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24099999999999999},"max":{"1min":0.56399999999999995,"5min":0.63700000000000001,"15min":0.754},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.251,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54900000000000004,"5min":0.624,"15min":0.624},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42399999999999999,"15min":0.41399999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.46500000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48699999999999999,"15min":0.47699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.36299999999999999},"max":{"1min":0.84299999999999997,"5min":1.0309999999999999,"15min":1.0700000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.65100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81799999999999995,"5min":0.95099999999999996,"15min":1.071},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":0.77500000000000002,"5min":1.103,"15min":1.103},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.755,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.71499999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65200000000000002,"15min":0.64600000000000002},"min":{"1min":0.45700000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85799999999999998,"5min":1.159,"15min":1.159},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.39200000000000002},"max":{"1min":0.88400000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.69599999999999995,"15min":0.68100000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89900000000000002,"5min":1.0069999999999999,"15min":1.107},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.67900000000000005,"15min":0.66800000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81000000000000005,"5min":1.123,"15min":1.123},"last":0.67700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575015,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":806912,"data_stored":396183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38200000000000001,"15min":0.36599999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.56799999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.40300000000000002,"15min":0.38200000000000001},"min":{"1min":0.23100000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.67000000000000004},"last":0.29499999999999998}]},{"osd":1,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.41199999999999998,"15min":0.38800000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.58399999999999996,"5min":0.65400000000000003,"15min":0.66100000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.438,"15min":0.42399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27400000000000002,"15min":0.22700000000000001},"max":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.67500000000000004},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47799999999999998,"15min":0.45900000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.253},"max":{"1min":0.63300000000000001,"5min":0.69399999999999995,"15min":1.0760000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.66100000000000003,"5min":0.71199999999999997,"15min":0.93799999999999994},"last":0.376}]},{"osd":4,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64300000000000002,"15min":0.60899999999999999},"min":{"1min":0.42699999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.66500000000000004,"15min":0.63100000000000001},"min":{"1min":0.46300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86199999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66500000000000004,"15min":0.63200000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97399999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.50600000000000001,"5min":0.35999999999999999,"15min":0.33200000000000002},"max":{"1min":0.90200000000000002,"5min":1.036,"15min":1.036},"last":0.66800000000000004}]},{"osd":6,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66900000000000004,"15min":0.64000000000000001},"min":{"1min":0.44500000000000001,"5min":0.374,"15min":0.35099999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68999999999999995,"15min":0.65800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:56:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69199999999999995,"15min":0.66200000000000003},"min":{"1min":0.438,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.88900000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67700000000000005,"15min":0.65000000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.92800000000000005,"5min":0.999,"15min":0.999},"last":0.56299999999999994}]}]},{"osd":4,"up_from":13,"seq":55834575015,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.58299999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":1.3029999999999999},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.38500000000000001,"15min":0.36599999999999999},"max":{"1min":0.77800000000000002,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.312},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.36299999999999999,"15min":0.27600000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.318},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":1.8899999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.39400000000000002,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.7909999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.42499999999999999,"5min":0.38,"15min":0.34899999999999998},"max":{"1min":0.79200000000000004,"5min":0.79900000000000004,"15min":1.1839999999999999},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.50800000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.28599999999999998},"max":{"1min":0.70199999999999996,"5min":0.75,"15min":0.88900000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52400000000000002,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.41999999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.505},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79200000000000004,"5min":0.86299999999999999,"15min":0.92000000000000004},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Oct 16 09:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.621,"15min":0.621},"min":{"1min":0.42599999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81599999999999995,"5min":0.879,"15min":0.93799999999999994},"last":0.53100000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575015,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45828,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326012,"statfs":{"total":96636764160,"available":96589836288,"internally_reserved":0,"allocated":778240,"data_stored":387058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63600000000000001,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.63900000000000001,"15min":0.60499999999999998},"min":{"1min":0.34499999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.58699999999999997}]},{"osd":1,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.56699999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.249},"max":{"1min":0.71899999999999997,"5min":0.875,"15min":1.1279999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.82899999999999996,"15min":1.0720000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58299999999999996,"15min":0.55900000000000005},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.76900000000000002,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57899999999999996,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.36899999999999999,"15min":0.35499999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.84499999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60999999999999999,"15min":0.58499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.65800000000000003,"5min":0.747,"15min":0.84199999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69599999999999995,"5min":0.85399999999999998,"15min":0.86899999999999999},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.48699999999999999,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67100000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65200000000000002,"5min":0.82999999999999996,"15min":0.88200000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.751,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.315},"max":{"1min":0.73099999999999998,"5min":0.876,"15min":0.88500000000000001},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:57:08.100 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T09:57:08.101 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T09:57:08.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.262+0000 7f10fd929700 1 Processor -- start 2023-10-16T09:57:08.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.262+0000 7f10fd929700 1 -- start start 2023-10-16T09:57:08.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.263+0000 7f10fd929700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10f8161380 0x7f10f80b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.263+0000 7f10fd929700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80b2c70 con 0x7f10f80cf220 2023-10-16T09:57:08.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.263+0000 7f10fd929700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80d4520 con 0x7f10f80cfff0 2023-10-16T09:57:08.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.263+0000 7f10fd929700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f80cf6b0 con 0x7f10f8161380 2023-10-16T09:57:08.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.263+0000 7f10f77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f10f80cf220 0x7f10f80b3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:50946/0 (socket says 172.21.15.158:50946) 2023-10-16T09:57:08.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.263+0000 7f10f77fe700 1 -- 172.21.15.158:0/291247018 learned_addr learned my addr 172.21.15.158:0/291247018 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:08.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.263+0000 7f10f6ffd700 1 --2- 172.21.15.158:0/291247018 >> v2:172.21.15.172:3300/0 conn(0x7f10f8161380 0x7f10f80b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f5ffb700 1 -- 172.21.15.158:0/291247018 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470678269 0 0) 0x7f10f0001490 con 0x7f10f80cf220 2023-10-16T09:57:08.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f5ffb700 1 -- 172.21.15.158:0/291247018 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e00035b0 con 0x7f10f80cf220 2023-10-16T09:57:08.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f6ffd700 1 -- 172.21.15.158:0/291247018 >> v1:172.21.15.172:6789/0 conn(0x7f10f80cfff0 legacy=0x7f10f80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f6ffd700 1 -- 172.21.15.158:0/291247018 >> v1:172.21.15.158:6789/0 conn(0x7f10f80cf220 legacy=0x7f10f80b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f5ffb700 1 -- 172.21.15.158:0/291247018 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2283328051 0 0) 0x7f10e00035b0 con 0x7f10f80cf220 2023-10-16T09:57:08.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f6ffd700 1 -- 172.21.15.158:0/291247018 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80b29d0 con 0x7f10f8161380 2023-10-16T09:57:08.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f6ffd700 1 --2- 172.21.15.158:0/291247018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f8161380 0x7f10f80b2490 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009820 tx=0x7f10ec009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0be605582565bb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:08.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f5ffb700 1 -- 172.21.15.158:0/291247018 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0001b80 con 0x7f10f8161380 2023-10-16T09:57:08.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f5ffb700 1 -- 172.21.15.158:0/291247018 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec005c30 con 0x7f10f8161380 2023-10-16T09:57:08.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.264+0000 7f10f5ffb700 1 -- 172.21.15.158:0/291247018 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec004030 con 0x7f10f8161380 2023-10-16T09:57:08.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.265+0000 7f10fd929700 1 -- 172.21.15.158:0/291247018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f8161380 msgr2=0x7f10f80b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.265+0000 7f10fd929700 1 --2- 172.21.15.158:0/291247018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f8161380 0x7f10f80b2490 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009820 tx=0x7f10ec009b30 comp rx=0 tx=0).stop 2023-10-16T09:57:08.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.265+0000 7f10fd929700 1 -- 172.21.15.158:0/291247018 shutdown_connections 2023-10-16T09:57:08.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.265+0000 7f10fd929700 1 --2- 172.21.15.158:0/291247018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f8161380 0x7f10f80b2490 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.265+0000 7f10fd929700 1 -- 172.21.15.158:0/291247018 >> 172.21.15.158:0/291247018 conn(0x7f10f809c960 msgr2=0x7f10f80d36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:08.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.265+0000 7f10fd929700 1 -- 172.21.15.158:0/291247018 shutdown_connections 2023-10-16T09:57:08.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.265+0000 7f10fd929700 1 -- 172.21.15.158:0/291247018 wait complete. 2023-10-16T09:57:08.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.266+0000 7f10fd929700 1 Processor -- start 2023-10-16T09:57:08.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.266+0000 7f10fd929700 1 -- start start 2023-10-16T09:57:08.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10fd929700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80cf220 0x7f10f80cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10fd929700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f80cfff0 0x7f10f80c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10fd929700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f8161380 0x7f10f8152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10fd929700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80ccd80 con 0x7f10f8161380 2023-10-16T09:57:08.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10fd929700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80c8130 con 0x7f10f80cfff0 2023-10-16T09:57:08.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10fd929700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8153160 con 0x7f10f80cf220 2023-10-16T09:57:08.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f8161380 0x7f10f8152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f8161380 0x7f10f8152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50776/0 (socket says 172.21.15.158:50776) 2023-10-16T09:57:08.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 -- 172.21.15.158:0/1117885799 learned_addr learned my addr 172.21.15.158:0/1117885799 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:08.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f67fc700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f80cfff0 0x7f10f80c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f6ffd700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80cf220 0x7f10f80cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 -- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80cf220 msgr2=0x7f10f80cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80cf220 0x7f10f80cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 -- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f80cfff0 msgr2=0x7f10f80c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f80cfff0 0x7f10f80c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 -- 172.21.15.158:0/1117885799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec009430 con 0x7f10f8161380 2023-10-16T09:57:08.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.267+0000 7f10f77fe700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f8161380 0x7f10f8152c20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f10f000cc00 tx=0x7f10f000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:08.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.268+0000 7f10dffff700 1 -- 172.21.15.158:0/1117885799 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f000aa80 con 0x7f10f8161380 2023-10-16T09:57:08.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.268+0000 7f10dffff700 1 -- 172.21.15.158:0/1117885799 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f000ac00 con 0x7f10f8161380 2023-10-16T09:57:08.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.268+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f80c8f70 con 0x7f10f8161380 2023-10-16T09:57:08.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.268+0000 7f10dffff700 1 -- 172.21.15.158:0/1117885799 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00070d0 con 0x7f10f8161380 2023-10-16T09:57:08.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.268+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f80c8360 con 0x7f10f8161380 2023-10-16T09:57:08.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.269+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f80b4fa0 con 0x7f10f8161380 2023-10-16T09:57:08.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.269+0000 7f10dffff700 1 -- 172.21.15.158:0/1117885799 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f10f0007230 con 0x7f10f8161380 2023-10-16T09:57:08.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.269+0000 7f10dffff700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f10e406b020 0x7f10e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.270+0000 7f10dffff700 1 -- 172.21.15.158:0/1117885799 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f10f00896e0 con 0x7f10f8161380 2023-10-16T09:57:08.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.270+0000 7f10f6ffd700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f10e406b020 0x7f10e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.270+0000 7f10f6ffd700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f10e406b020 0x7f10e406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009fd0 tx=0x7f10ec008070 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:08.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.274+0000 7f10dffff700 1 -- 172.21.15.158:0/1117885799 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10f00543b0 con 0x7f10f8161380 2023-10-16T09:57:08.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.546+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f10f80b2c70 con 0x7f10f8161380 2023-10-16T09:57:08.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.548+0000 7f10dffff700 1 -- 172.21.15.158:0/1117885799 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f10f80b2c70 con 0x7f10f8161380 2023-10-16T09:57:08.549 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T09:57:08.550 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T09:57:08.550 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T09:57:08.550 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T09:57:08.550 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T09:57:08.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f10e406b020 msgr2=0x7f10e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f10e406b020 0x7f10e406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009fd0 tx=0x7f10ec008070 comp rx=0 tx=0).stop 2023-10-16T09:57:08.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f8161380 msgr2=0x7f10f8152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f8161380 0x7f10f8152c20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f10f000cc00 tx=0x7f10f000cf10 comp rx=0 tx=0).stop 2023-10-16T09:57:08.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 shutdown_connections 2023-10-16T09:57:08.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f10e406b020 0x7f10e406d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f8161380 0x7f10f8152c20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80cf220 0x7f10f80cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 --2- 172.21.15.158:0/1117885799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f80cfff0 0x7f10f80c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 >> 172.21.15.158:0/1117885799 conn(0x7f10f809c960 msgr2=0x7f10f80b5130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:08.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 shutdown_connections 2023-10-16T09:57:08.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.551+0000 7f10fd929700 1 -- 172.21.15.158:0/1117885799 wait complete. 2023-10-16T09:57:08.555 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T09:57:08.564 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:57:08.583 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T09:57:08.582+0000 7f1d2e6ca700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T09:57:08.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.684+0000 7f27f94b1700 1 Processor -- start 2023-10-16T09:57:08.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.685+0000 7f27f94b1700 1 -- start start 2023-10-16T09:57:08.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.685+0000 7f27f94b1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f27f415d7c0 0x7f27f40d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.685+0000 7f27f94b1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40ceb20 0x7f27f40b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.685+0000 7f27f94b1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40d3bd0 con 0x7f27f40d1210 2023-10-16T09:57:08.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.685+0000 7f27f94b1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f40d4060 con 0x7f27f415d7c0 2023-10-16T09:57:08.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.685+0000 7f27f94b1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40d2cf0 con 0x7f27f40ceb20 2023-10-16T09:57:08.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f27f415d7c0 0x7f27f40d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f27f415d7c0 0x7f27f40d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50780/0 (socket says 172.21.15.158:50780) 2023-10-16T09:57:08.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 -- 172.21.15.158:0/3719394863 learned_addr learned my addr 172.21.15.158:0/3719394863 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:08.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 -- 172.21.15.158:0/3719394863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40ceb20 msgr2=0x7f27f40b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:08.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 --2- 172.21.15.158:0/3719394863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40ceb20 0x7f27f40b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 -- 172.21.15.158:0/3719394863 >> v1:172.21.15.158:6789/0 conn(0x7f27f40d1210 legacy=0x7f27f40ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 -- 172.21.15.158:0/3719394863 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40d2fc0 con 0x7f27f415d7c0 2023-10-16T09:57:08.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f1ffb700 1 -- 172.21.15.158:0/3719394863 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010124184 0 0) 0x7f27f40d3bd0 con 0x7f27f40d1210 2023-10-16T09:57:08.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f2ffd700 1 --2- 172.21.15.158:0/3719394863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40d27b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f27e80098f0 tx=0x7f27e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5453ae24324e6126 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:08.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f1ffb700 1 -- 172.21.15.158:0/3719394863 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e800b8a0 con 0x7f27f415d7c0 2023-10-16T09:57:08.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f1ffb700 1 -- 172.21.15.158:0/3719394863 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e800ba00 con 0x7f27f415d7c0 2023-10-16T09:57:08.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.686+0000 7f27f1ffb700 1 -- 172.21.15.158:0/3719394863 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e800bd90 con 0x7f27f415d7c0 2023-10-16T09:57:08.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 -- 172.21.15.158:0/3719394863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 msgr2=0x7f27f40d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3719394863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40d27b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f27e80098f0 tx=0x7f27e8009c00 comp rx=0 tx=0).stop 2023-10-16T09:57:08.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 -- 172.21.15.158:0/3719394863 shutdown_connections 2023-10-16T09:57:08.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3719394863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40d27b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3719394863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40ceb20 0x7f27f40b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 -- 172.21.15.158:0/3719394863 >> 172.21.15.158:0/3719394863 conn(0x7f27f40669f0 msgr2=0x7f27f4066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:08.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 -- 172.21.15.158:0/3719394863 shutdown_connections 2023-10-16T09:57:08.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.687+0000 7f27f94b1700 1 -- 172.21.15.158:0/3719394863 wait complete. 2023-10-16T09:57:08.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.688+0000 7f27f94b1700 1 Processor -- start 2023-10-16T09:57:08.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.688+0000 7f27f94b1700 1 -- start start 2023-10-16T09:57:08.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f94b1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27f40ceb20 0x7f27f4160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f94b1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40d1210 0x7f27f40b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f94b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f94b1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40b6030 con 0x7f27f415d7c0 2023-10-16T09:57:08.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f94b1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40c81a0 con 0x7f27f40ceb20 2023-10-16T09:57:08.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f94b1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40c83b0 con 0x7f27f40d1210 2023-10-16T09:57:08.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40b3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50790/0 (socket says 172.21.15.158:50790) 2023-10-16T09:57:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 -- 172.21.15.158:0/3462423891 learned_addr learned my addr 172.21.15.158:0/3462423891 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f2ffd700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27f40ceb20 0x7f27f4160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f27fc700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40d1210 0x7f27f40b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 -- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40d1210 msgr2=0x7f27f40b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40d1210 0x7f27f40b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 -- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27f40ceb20 msgr2=0x7f27f4160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27f40ceb20 0x7f27f4160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 -- 172.21.15.158:0/3462423891 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8009550 con 0x7f27f415d7c0 2023-10-16T09:57:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f27fc700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40d1210 0x7f27f40b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27f37fe700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40b3bf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b970 tx=0x7f27ec00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.689+0000 7f27db7fe700 1 -- 172.21.15.158:0/3462423891 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00c930 con 0x7f27f415d7c0 2023-10-16T09:57:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.690+0000 7f27db7fe700 1 -- 172.21.15.158:0/3462423891 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec00cab0 con 0x7f27f415d7c0 2023-10-16T09:57:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.690+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f40c8690 con 0x7f27f415d7c0 2023-10-16T09:57:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.690+0000 7f27db7fe700 1 -- 172.21.15.158:0/3462423891 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00ce60 con 0x7f27f415d7c0 2023-10-16T09:57:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.690+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f40c8c50 con 0x7f27f415d7c0 2023-10-16T09:57:08.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.690+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f40c8e70 con 0x7f27f415d7c0 2023-10-16T09:57:08.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.691+0000 7f27db7fe700 1 -- 172.21.15.158:0/3462423891 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f27ec01d3b0 con 0x7f27f415d7c0 2023-10-16T09:57:08.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.691+0000 7f27db7fe700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f27e006b020 0x7f27e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:08.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.692+0000 7f27db7fe700 1 -- 172.21.15.158:0/3462423891 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f27ec0896d0 con 0x7f27f415d7c0 2023-10-16T09:57:08.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.692+0000 7f27f2ffd700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f27e006b020 0x7f27e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:08.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.692+0000 7f27f2ffd700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f27e006b020 0x7f27e006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f27e8014040 tx=0x7f27e8008030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:08.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.696+0000 7f27db7fe700 1 -- 172.21.15.158:0/3462423891 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27ec0543a0 con 0x7f27f415d7c0 2023-10-16T09:57:08.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.931+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27f40d1ee0 con 0x7f27f415d7c0 2023-10-16T09:57:08.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.932+0000 7f27db7fe700 1 -- 172.21.15.158:0/3462423891 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f27ec0579c0 con 0x7f27f415d7c0 2023-10-16T09:57:08.933 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:08.934 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4264509745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4264509745},{"type":"v1","addr":"172.21.15.158:6835","nonce":4264509745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1091298830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1091298830},{"type":"v1","addr":"172.21.15.158:6837","nonce":1091298830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":35261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2630165052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2630165052},{"type":"v1","addr":"172.21.15.172:6834","nonce":2630165052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:57:03.005758+0000","modified":"2023-10-16T09:57:06.014112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35264":{"gid":35264,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1262687320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1262687320},{"type":"v1","addr":"172.21.15.172:6835","nonce":1262687320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T09:57:08.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f27e006b020 msgr2=0x7f27e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f27e006b020 0x7f27e006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f27e8014040 tx=0x7f27e8008030 comp rx=0 tx=0).stop 2023-10-16T09:57:08.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 msgr2=0x7f27f40b3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:08.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40b3bf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b970 tx=0x7f27ec00bc80 comp rx=0 tx=0).stop 2023-10-16T09:57:08.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 shutdown_connections 2023-10-16T09:57:08.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f27e006b020 0x7f27e006d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27f415d7c0 0x7f27f40b3bf0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27f40ceb20 0x7f27f4160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 --2- 172.21.15.158:0/3462423891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27f40d1210 0x7f27f40b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:08.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 >> 172.21.15.158:0/3462423891 conn(0x7f27f40669f0 msgr2=0x7f27f40ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:08.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 shutdown_connections 2023-10-16T09:57:08.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:08.934+0000 7f27f94b1700 1 -- 172.21.15.158:0/3462423891 wait complete. 2023-10-16T09:57:08.938 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 25 2023-10-16T09:57:09.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T09:57:03.005758+0000', 'modified': '2023-10-16T09:57:06.014112+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 35264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_35264': {'gid': 35264, 'name': 'b', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/1262687320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 1262687320}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 1262687320}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T09:57:09.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T09:57:09.351 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T09:57:09.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.473+0000 7f7aa7312700 1 Processor -- start 2023-10-16T09:57:09.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.473+0000 7f7aa7312700 1 -- start start 2023-10-16T09:57:09.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.473+0000 7f7aa7312700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa015e4d0 0x7f7aa00ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.473+0000 7f7aa7312700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7aa00711b0 0x7f7aa0071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.473+0000 7f7aa7312700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa00cb0f0 con 0x7f7aa0159c30 2023-10-16T09:57:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.473+0000 7f7aa7312700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa00cbc90 con 0x7f7aa00711b0 2023-10-16T09:57:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.473+0000 7f7aa7312700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa0071b80 con 0x7f7aa015e4d0 2023-10-16T09:57:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7aa00711b0 0x7f7aa0071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7aa00711b0 0x7f7aa0071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50798/0 (socket says 172.21.15.158:50798) 2023-10-16T09:57:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 -- 172.21.15.158:0/2302156236 learned_addr learned my addr 172.21.15.158:0/2302156236 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/2302156236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa015e4d0 0x7f7aa00ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 -- 172.21.15.158:0/2302156236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa015e4d0 msgr2=0x7f7aa00ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 --2- 172.21.15.158:0/2302156236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa015e4d0 0x7f7aa00ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 -- 172.21.15.158:0/2302156236 >> v1:172.21.15.172:6789/0 conn(0x7f7aa0159c30 legacy=0x7f7aa00cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 -- 172.21.15.158:0/2302156236 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa00ca800 con 0x7f7aa00711b0 2023-10-16T09:57:09.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7aa58af700 1 --2- 172.21.15.158:0/2302156236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa00711b0 0x7f7aa0071640 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7a980098f0 tx=0x7f7a98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f89a1378b48a431 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7a9ffff700 1 -- 172.21.15.158:0/2302156236 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9800b8a0 con 0x7f7aa00711b0 2023-10-16T09:57:09.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7a9ffff700 1 -- 172.21.15.158:0/2302156236 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9800ba00 con 0x7f7aa00711b0 2023-10-16T09:57:09.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.474+0000 7f7a9ffff700 1 -- 172.21.15.158:0/2302156236 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9800bd90 con 0x7f7aa00711b0 2023-10-16T09:57:09.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 -- 172.21.15.158:0/2302156236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa00711b0 msgr2=0x7f7aa0071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 --2- 172.21.15.158:0/2302156236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa00711b0 0x7f7aa0071640 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7a980098f0 tx=0x7f7a98009c00 comp rx=0 tx=0).stop 2023-10-16T09:57:09.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 -- 172.21.15.158:0/2302156236 shutdown_connections 2023-10-16T09:57:09.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 --2- 172.21.15.158:0/2302156236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa00711b0 0x7f7aa0071640 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 --2- 172.21.15.158:0/2302156236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa015e4d0 0x7f7aa00ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 -- 172.21.15.158:0/2302156236 >> 172.21.15.158:0/2302156236 conn(0x7f7aa00cfb40 msgr2=0x7f7aa00cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 -- 172.21.15.158:0/2302156236 shutdown_connections 2023-10-16T09:57:09.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.475+0000 7f7aa7312700 1 -- 172.21.15.158:0/2302156236 wait complete. 2023-10-16T09:57:09.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.476+0000 7f7aa7312700 1 Processor -- start 2023-10-16T09:57:09.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.476+0000 7f7aa7312700 1 -- start start 2023-10-16T09:57:09.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa7312700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa00711b0 0x7f7aa014f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa7312700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa0159c30 0x7f7aa00c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa7312700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa015e4d0 0x7f7aa006ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa7312700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa01e3c30 con 0x7f7aa015e4d0 2023-10-16T09:57:09.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa7312700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa01e3f10 con 0x7f7aa00711b0 2023-10-16T09:57:09.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa7312700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa01e41f0 con 0x7f7aa0159c30 2023-10-16T09:57:09.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa015e4d0 0x7f7aa006ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa015e4d0 0x7f7aa006ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50804/0 (socket says 172.21.15.158:50804) 2023-10-16T09:57:09.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 -- 172.21.15.158:0/240529164 learned_addr learned my addr 172.21.15.158:0/240529164 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:09.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa0159c30 0x7f7aa00c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa00711b0 0x7f7aa014f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 -- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa0159c30 msgr2=0x7f7aa00c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa0159c30 0x7f7aa00c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 -- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa00711b0 msgr2=0x7f7aa014f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa00711b0 0x7f7aa014f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 -- 172.21.15.158:0/240529164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a98009550 con 0x7f7aa015e4d0 2023-10-16T09:57:09.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa58af700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa015e4d0 0x7f7aa006ac00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7aa006e5c0 tx=0x7f7a98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7a9dffb700 1 -- 172.21.15.158:0/240529164 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98004300 con 0x7f7aa015e4d0 2023-10-16T09:57:09.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7a9dffb700 1 -- 172.21.15.158:0/240529164 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9801f070 con 0x7f7aa015e4d0 2023-10-16T09:57:09.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7a9dffb700 1 -- 172.21.15.158:0/240529164 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a980036a0 con 0x7f7aa015e4d0 2023-10-16T09:57:09.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.477+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa01e44d0 con 0x7f7aa015e4d0 2023-10-16T09:57:09.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.478+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa00c99a0 con 0x7f7aa015e4d0 2023-10-16T09:57:09.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.479+0000 7f7a9dffb700 1 -- 172.21.15.158:0/240529164 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a98021030 con 0x7f7aa015e4d0 2023-10-16T09:57:09.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.479+0000 7f7a9dffb700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7a8c06b020 0x7f7a8c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.480+0000 7f7a9dffb700 1 -- 172.21.15.158:0/240529164 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7a9808ad20 con 0x7f7aa015e4d0 2023-10-16T09:57:09.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.480+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7a8c06b020 0x7f7a8c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.480+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7a8c06b020 0x7f7a8c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a900012e0 tx=0x7f7a90009040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.482+0000 7f7ab34fcb80 1 Processor -- start 2023-10-16T09:57:09.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.482+0000 7f7ab34fcb80 1 -- start start 2023-10-16T09:57:09.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f367fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f367ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36536c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f35c9c20 con 0x5587f367d630 2023-10-16T09:57:09.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f35aec10 con 0x5587f3653010 2023-10-16T09:57:09.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587f35ac530 con 0x5587f3678220 2023-10-16T09:57:09.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f367ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f367ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50810/0 (socket says 172.21.15.158:50810) 2023-10-16T09:57:09.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 -- 172.21.15.158:0/934285008 learned_addr learned my addr 172.21.15.158:0/934285008 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:09.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f367fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa58af700 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 -- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 msgr2=0x5587f367fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f367fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 -- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 msgr2=0x5587f36536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 -- 172.21.15.158:0/934285008 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587f35653a0 con 0x5587f367d630 2023-10-16T09:57:09.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f367ffb0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7a94007920 tx=0x7f7a94007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e971b98e2cd70ea3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7a86ffd700 1 -- 172.21.15.158:0/934285008 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94010950 con 0x5587f367d630 2023-10-16T09:57:09.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7a86ffd700 1 -- 172.21.15.158:0/934285008 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a94010ab0 con 0x5587f367d630 2023-10-16T09:57:09.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.483+0000 7f7a86ffd700 1 -- 172.21.15.158:0/934285008 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94010e40 con 0x5587f367d630 2023-10-16T09:57:09.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/934285008 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 msgr2=0x5587f367ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f367ffb0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7a94007920 tx=0x7f7a94007c30 comp rx=0 tx=0).stop 2023-10-16T09:57:09.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/934285008 shutdown_connections 2023-10-16T09:57:09.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f367fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/934285008 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f367ffb0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/934285008 >> 172.21.15.158:0/934285008 conn(0x5587f3582d80 msgr2=0x5587f369dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/934285008 shutdown_connections 2023-10-16T09:57:09.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.484+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/934285008 wait complete. 2023-10-16T09:57:09.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.485+0000 7f7ab34fcb80 1 -- start start 2023-10-16T09:57:09.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.485+0000 7f7ab34fcb80 1 Processor -- start 2023-10-16T09:57:09.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36821a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f3649f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f364a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f35a7e20 con 0x5587f367d630 2023-10-16T09:57:09.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f35446c0 con 0x5587f3653010 2023-10-16T09:57:09.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587f3544360 con 0x5587f3678220 2023-10-16T09:57:09.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 10 client.0 _ll_register_callbacks cb 0x5587f33a17b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:57:09.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f364a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f364a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50818/0 (socket says 172.21.15.158:50818) 2023-10-16T09:57:09.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 -- 172.21.15.158:0/386913543 learned_addr learned my addr 172.21.15.158:0/386913543 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:09.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36821a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f3649f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 -- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 msgr2=0x5587f3649f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f3649f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 -- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 msgr2=0x5587f36821a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36821a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 -- 172.21.15.158:0/386913543 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9400e040 con 0x5587f367d630 2023-10-16T09:57:09.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7aa58af700 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f364a480 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009c40 tx=0x7f7a98059250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deac74819d7ba87a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7a7bfff700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:57:09.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7a7bfff700 1 -- 172.21.15.158:0/386913543 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98088070 con 0x5587f367d630 2023-10-16T09:57:09.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7a7bfff700 1 -- 172.21.15.158:0/386913543 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7a980596a0 con 0x5587f367d630 2023-10-16T09:57:09.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5587f36485b0 con 0x5587f367d630 2023-10-16T09:57:09.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7a7bfff700 1 -- 172.21.15.158:0/386913543 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a980195b0 con 0x5587f367d630 2023-10-16T09:57:09.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 10 client.6387 fetch_fsmap learned FSMap version 25 2023-10-16T09:57:09.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.486+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5587f33d2d90 con 0x5587f367d630 2023-10-16T09:57:09.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7a7bfff700 1 -- 172.21.15.158:0/386913543 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5587f36485b0 con 0x5587f367d630 2023-10-16T09:57:09.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7a7bfff700 1 -- 172.21.15.158:0/386913543 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7a98059900 con 0x5587f367d630 2023-10-16T09:57:09.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7ab34fcb80 10 client.6387 fetch_fsmap finished waiting for FSMap version 25 2023-10-16T09:57:09.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7ab34fcb80 10 client.6387 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:57:09.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7ab34fcb80 20 client.6387 populate_metadata read hostname 'smithi158' 2023-10-16T09:57:09.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f3679010 0x5587f36ad500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7ab34fcb80 4 client.6387 mds_command: new command op to 35264 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T09:57:09.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 --> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587f35a0e00 con 0x5587f3679010 2023-10-16T09:57:09.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f3679010 0x5587f36ad500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7a7bfff700 10 client.6387 ms_handle_connect on v2:172.21.15.172:6833/1262687320 2023-10-16T09:57:09.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.487+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f3679010 0x5587f36ad500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67f51872f85d9d1f server_cookie=c4d3f59a5bf626e6 in_seq=0 out_seq=0 2023-10-16T09:57:09.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7a7bfff700 1 -- 172.21.15.158:0/386913543 <== mds.0 v2:172.21.15.172:6833/1262687320 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5587f35a0e00 con 0x5587f3679010 2023-10-16T09:57:09.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7a7bfff700 10 client.6387 handle_command_reply: tid=0 2023-10-16T09:57:09.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 client.6387 shutdown 2023-10-16T09:57:09.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 msgr2=0x5587f364a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f364a480 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009c40 tx=0x7f7a98059250 comp rx=0 tx=0).stop 2023-10-16T09:57:09.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 shutdown_connections 2023-10-16T09:57:09.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f3679010 0x5587f36ad500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f364a480 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f36821a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/386913543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f3649f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7a7bfff700 0 client.6387 ms_handle_reset on v2:172.21.15.172:6833/1262687320 2023-10-16T09:57:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 >> 172.21.15.158:0/386913543 conn(0x5587f3582d80 msgr2=0x5587f369b520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 shutdown_connections 2023-10-16T09:57:09.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.488+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/386913543 wait complete. 2023-10-16T09:57:09.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.489+0000 7f7ab34fcb80 20 client.6387 trim_cache size 0 max 16384 2023-10-16T09:57:09.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.506+0000 7f7ab34fcb80 1 Processor -- start 2023-10-16T09:57:09.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.506+0000 7f7ab34fcb80 1 -- start start 2023-10-16T09:57:09.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f367c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f36535e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f3655a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f34288d0 con 0x5587f367d630 2023-10-16T09:57:09.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f36808f0 con 0x5587f3653010 2023-10-16T09:57:09.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587f3649a50 con 0x5587f3678220 2023-10-16T09:57:09.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f3655a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f3655a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50824/0 (socket says 172.21.15.158:50824) 2023-10-16T09:57:09.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 -- 172.21.15.158:0/3740191380 learned_addr learned my addr 172.21.15.158:0/3740191380 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:09.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f36535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f367c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 -- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 msgr2=0x5587f36535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f36535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 -- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 msgr2=0x5587f367c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f367c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 -- 172.21.15.158:0/3740191380 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587f33bbc10 con 0x5587f367d630 2023-10-16T09:57:09.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7aa58af700 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f3655a00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7a980593a0 tx=0x7f7a9805c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d21ad274757b3dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7a877fe700 1 -- 172.21.15.158:0/3740191380 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98088070 con 0x5587f367d630 2023-10-16T09:57:09.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7a877fe700 1 -- 172.21.15.158:0/3740191380 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a980591e0 con 0x5587f367d630 2023-10-16T09:57:09.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.507+0000 7f7a877fe700 1 -- 172.21.15.158:0/3740191380 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a980195b0 con 0x5587f367d630 2023-10-16T09:57:09.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/3740191380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 msgr2=0x5587f3655a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f3655a00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7a980593a0 tx=0x7f7a9805c070 comp rx=0 tx=0).stop 2023-10-16T09:57:09.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/3740191380 shutdown_connections 2023-10-16T09:57:09.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f367d630 0x5587f3655a00 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f3653010 0x5587f367c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/3740191380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3678220 0x5587f36535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/3740191380 >> 172.21.15.158:0/3740191380 conn(0x5587f3582d80 msgr2=0x5587f3651850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/3740191380 shutdown_connections 2023-10-16T09:57:09.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/3740191380 wait complete. 2023-10-16T09:57:09.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.508+0000 7f7ab34fcb80 1 -- start start 2023-10-16T09:57:09.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 1 Processor -- start 2023-10-16T09:57:09.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3653010 0x5587f37008e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f3678220 0x5587f36abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f367d630 0x5587f3681b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f37023e0 con 0x5587f3678220 2023-10-16T09:57:09.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f36ac860 con 0x5587f367d630 2023-10-16T09:57:09.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587f3650740 con 0x5587f3653010 2023-10-16T09:57:09.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7ab34fcb80 10 client.0 _ll_register_callbacks cb 0x7f7a98055ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:57:09.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.509+0000 7f7aa48ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f3678220 0x5587f36abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f3678220 0x5587f36abff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50830/0 (socket says 172.21.15.158:50830) 2023-10-16T09:57:09.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 -- 172.21.15.158:0/1050484557 learned_addr learned my addr 172.21.15.158:0/1050484557 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:09.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3653010 0x5587f37008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa58af700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f367d630 0x5587f3681b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 -- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3653010 msgr2=0x5587f37008e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3653010 0x5587f37008e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 -- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f367d630 msgr2=0x5587f3681b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f367d630 0x5587f3681b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 -- 172.21.15.158:0/1050484557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587f33d2d90 con 0x5587f3678220 2023-10-16T09:57:09.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa58af700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f367d630 0x5587f3681b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:09.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7a7bfff700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:57:09.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7aa48ad700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f3678220 0x5587f36abff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x5587f3656fa0 tx=0x7f7a940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c12e264e7ab5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7a7bfff700 1 -- 172.21.15.158:0/1050484557 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94023040 con 0x5587f3678220 2023-10-16T09:57:09.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7a7bfff700 1 -- 172.21.15.158:0/1050484557 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7a940036a0 con 0x5587f3678220 2023-10-16T09:57:09.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a94010ab0 con 0x5587f3678220 2023-10-16T09:57:09.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7a7bfff700 1 -- 172.21.15.158:0/1050484557 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a94010950 con 0x5587f3678220 2023-10-16T09:57:09.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7ab34fcb80 10 client.6393 fetch_fsmap learned FSMap version 25 2023-10-16T09:57:09.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5587f33f9200 con 0x5587f3678220 2023-10-16T09:57:09.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7a7bfff700 1 -- 172.21.15.158:0/1050484557 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a94010ab0 con 0x5587f3678220 2023-10-16T09:57:09.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7a7bfff700 1 -- 172.21.15.158:0/1050484557 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7a94005a20 con 0x5587f3678220 2023-10-16T09:57:09.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7ab34fcb80 10 client.6393 fetch_fsmap finished waiting for FSMap version 25 2023-10-16T09:57:09.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7ab34fcb80 10 client.6393 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T09:57:09.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7ab34fcb80 20 client.6393 populate_metadata read hostname 'smithi158' 2023-10-16T09:57:09.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.510+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f37016c0 0x5587f3701aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 4 client.6393 mds_command: new command op to 35264 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T09:57:09.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 --> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5587f35a0b20 con 0x5587f37016c0 2023-10-16T09:57:09.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f37016c0 0x5587f3701aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7a7bfff700 10 client.6393 ms_handle_connect on v2:172.21.15.172:6833/1262687320 2023-10-16T09:57:09.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7aa50ae700 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f37016c0 0x5587f3701aa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b89718c8b0f24d server_cookie=ca1470cc5fa4a063 in_seq=0 out_seq=0 2023-10-16T09:57:09.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7a7bfff700 1 -- 172.21.15.158:0/1050484557 <== mds.0 v2:172.21.15.172:6833/1262687320 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5587f35a0b20 con 0x5587f37016c0 2023-10-16T09:57:09.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7a7bfff700 10 client.6393 handle_command_reply: tid=0 2023-10-16T09:57:09.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 client.6393 shutdown 2023-10-16T09:57:09.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f3678220 msgr2=0x5587f36abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f3678220 0x5587f36abff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x5587f3656fa0 tx=0x7f7a940047c0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 shutdown_connections 2023-10-16T09:57:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587f367d630 0x5587f3681b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5587f3653010 0x5587f37008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.172:6833/1262687320,v1:172.21.15.172:6835/1262687320] conn(0x5587f37016c0 0x5587f3701aa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 --2- 172.21.15.158:0/1050484557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5587f3678220 0x5587f36abff0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7a7bfff700 0 client.6393 ms_handle_reset on v2:172.21.15.172:6833/1262687320 2023-10-16T09:57:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.511+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 >> 172.21.15.158:0/1050484557 conn(0x5587f3582d80 msgr2=0x5587f3680850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 shutdown_connections 2023-10-16T09:57:09.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7ab34fcb80 1 -- 172.21.15.158:0/1050484557 wait complete. 2023-10-16T09:57:09.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7ab34fcb80 20 client.6393 trim_cache size 0 max 16384 2023-10-16T09:57:09.534 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:57:09.535 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T09:57:09.535 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T09:57:09.535 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 35264, 2023-10-16T09:57:09.535 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T09:57:09.535 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T09:57:09.536 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T09:57:09.536 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9098918940000003, 2023-10-16T09:57:09.536 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 25, 2023-10-16T09:57:09.536 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 67, 2023-10-16T09:57:09.536 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 67, 2023-10-16T09:57:09.537 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 11.833039685999999 2023-10-16T09:57:09.537 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:57:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7a8c06b020 msgr2=0x7f7a8c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7a8c06b020 0x7f7a8c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a900012e0 tx=0x7f7a90009040 comp rx=0 tx=0).stop 2023-10-16T09:57:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa015e4d0 msgr2=0x7f7aa006ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa015e4d0 0x7f7aa006ac00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7aa006e5c0 tx=0x7f7a98004750 comp rx=0 tx=0).stop 2023-10-16T09:57:09.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 shutdown_connections 2023-10-16T09:57:09.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f7a8c06b020 0x7f7a8c06d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7aa015e4d0 0x7f7aa006ac00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa00711b0 0x7f7aa014f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 --2- 172.21.15.158:0/240529164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa0159c30 0x7f7aa00c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 >> 172.21.15.158:0/240529164 conn(0x7f7aa00cfb40 msgr2=0x7f7aa00665a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 shutdown_connections 2023-10-16T09:57:09.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.512+0000 7f7aa7312700 1 -- 172.21.15.158:0/240529164 wait complete. 2023-10-16T09:57:09.541 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:57:09.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.660+0000 7fef332c3700 1 Processor -- start 2023-10-16T09:57:09.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.661+0000 7fef332c3700 1 -- start start 2023-10-16T09:57:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.661+0000 7fef332c3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 0x7fef2c168620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.661+0000 7fef332c3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c15db90 con 0x7fef2c0b3770 2023-10-16T09:57:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.661+0000 7fef332c3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c06bea0 con 0x7fef2c15d7b0 2023-10-16T09:57:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.661+0000 7fef332c3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c06c140 con 0x7fef2c0cf760 2023-10-16T09:57:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef3085e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fef2c0b3770 0x7fef2c0cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51014/0 (socket says 172.21.15.158:51014) 2023-10-16T09:57:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef3085e700 1 -- 172.21.15.158:0/1376889714 learned_addr learned my addr 172.21.15.158:0/1376889714 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465938946 0 0) 0x7fef2c15db90 con 0x7fef2c0b3770 2023-10-16T09:57:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef180035b0 con 0x7fef2c0b3770 2023-10-16T09:57:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef31860700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 0x7fef2c168620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2954369824 0 0) 0x7fef180035b0 con 0x7fef2c0b3770 2023-10-16T09:57:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 msgr2=0x7fef2c168620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 0x7fef2c168620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 >> v1:172.21.15.172:6789/0 conn(0x7fef2c15d7b0 legacy=0x7fef2c161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c06c420 con 0x7fef2c0b3770 2023-10-16T09:57:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fef28001670 con 0x7fef2c0b3770 2023-10-16T09:57:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 >> v1:172.21.15.158:6789/0 conn(0x7fef2c0b3770 legacy=0x7fef2c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef18003a30 0x7fef18003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef18004850 0x7fef18004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 0x7fef180052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c06c140 con 0x7fef18003a30 2023-10-16T09:57:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1800bc00 con 0x7fef18004850 2023-10-16T09:57:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1800bdc0 con 0x7fef2c0cf760 2023-10-16T09:57:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef3105f700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef18003a30 0x7fef18003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef3085e700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef18004850 0x7fef18004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.662+0000 7fef31860700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 0x7fef180052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef3105f700 1 -- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 msgr2=0x7fef180052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef3105f700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef2c0cf760 0x7fef180052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef3105f700 1 -- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef18004850 msgr2=0x7fef18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:09.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef3105f700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef18004850 0x7fef18004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef3105f700 1 -- 172.21.15.158:0/1376889714 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef180035b0 con 0x7fef18003a30 2023-10-16T09:57:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef3105f700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef18003a30 0x7fef18003e50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fef1c001800 tx=0x7fef18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24794b5c1ca0854e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0012f0 con 0x7fef18003a30 2023-10-16T09:57:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c0052f0 con 0x7fef18003a30 2023-10-16T09:57:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef23fff700 1 -- 172.21.15.158:0/1376889714 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c006860 con 0x7fef18003a30 2023-10-16T09:57:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef332c3700 1 -- 172.21.15.158:0/1376889714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef18003a30 msgr2=0x7fef18003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef332c3700 1 --2- 172.21.15.158:0/1376889714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef18003a30 0x7fef18003e50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fef1c001800 tx=0x7fef18005880 comp rx=0 tx=0).stop 2023-10-16T09:57:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef31860700 1 -- 172.21.15.158:0/1376889714 reap_dead start 2023-10-16T09:57:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef332c3700 1 -- 172.21.15.158:0/1376889714 shutdown_connections 2023-10-16T09:57:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef332c3700 1 -- 172.21.15.158:0/1376889714 >> 172.21.15.158:0/1376889714 conn(0x7fef2c06a8d0 msgr2=0x7fef2c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.663+0000 7fef332c3700 1 -- 172.21.15.158:0/1376889714 shutdown_connections 2023-10-16T09:57:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.664+0000 7fef332c3700 1 -- 172.21.15.158:0/1376889714 wait complete. 2023-10-16T09:57:09.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 Processor -- start 2023-10-16T09:57:09.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 -- start start 2023-10-16T09:57:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef18003a30 0x7fef2c0b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef2c1e2a60 0x7fef2c1f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef2c0c9de0 0x7fef2c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0ca7b0 con 0x7fef2c0c9de0 2023-10-16T09:57:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0b0ce0 con 0x7fef2c1e2a60 2023-10-16T09:57:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.665+0000 7fef332c3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c14b650 con 0x7fef18003a30 2023-10-16T09:57:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef2c0c9de0 0x7fef2c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef2c0c9de0 0x7fef2c0ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50852/0 (socket says 172.21.15.158:50852) 2023-10-16T09:57:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 -- 172.21.15.158:0/897527765 learned_addr learned my addr 172.21.15.158:0/897527765 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef3105f700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef18003a30 0x7fef2c0b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef3085e700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef2c1e2a60 0x7fef2c1f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 -- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef18003a30 msgr2=0x7fef2c0b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef18003a30 0x7fef2c0b07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 -- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef2c1e2a60 msgr2=0x7fef2c1f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef2c1e2a60 0x7fef2c1f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 -- 172.21.15.158:0/897527765 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c14bbc0 con 0x7fef2c0c9de0 2023-10-16T09:57:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef31860700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef2c0c9de0 0x7fef2c0ca270 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fef2c06b750 tx=0x7fef24011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef21ffb700 1 -- 172.21.15.158:0/897527765 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2400e040 con 0x7fef2c0c9de0 2023-10-16T09:57:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef21ffb700 1 -- 172.21.15.158:0/897527765 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2400f040 con 0x7fef2c0c9de0 2023-10-16T09:57:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c148d20 con 0x7fef2c0c9de0 2023-10-16T09:57:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef21ffb700 1 -- 172.21.15.158:0/897527765 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2400fe40 con 0x7fef2c0c9de0 2023-10-16T09:57:09.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.666+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c148fc0 con 0x7fef2c0c9de0 2023-10-16T09:57:09.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.667+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c0681b0 con 0x7fef2c0c9de0 2023-10-16T09:57:09.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.668+0000 7fef21ffb700 1 -- 172.21.15.158:0/897527765 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef24015430 con 0x7fef2c0c9de0 2023-10-16T09:57:09.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.668+0000 7fef21ffb700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fef1006b020 0x7fef1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:09.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.668+0000 7fef21ffb700 1 -- 172.21.15.158:0/897527765 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fef2408a1f0 con 0x7fef2c0c9de0 2023-10-16T09:57:09.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.668+0000 7fef3105f700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fef1006b020 0x7fef1006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:09.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.669+0000 7fef3105f700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fef1006b020 0x7fef1006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef1c014040 tx=0x7fef1c009030 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:09.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.673+0000 7fef21ffb700 1 -- 172.21.15.158:0/897527765 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef24054da0 con 0x7fef2c0c9de0 2023-10-16T09:57:09.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.911+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fef2c0b7400 con 0x7fef2c0c9de0 2023-10-16T09:57:09.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.911+0000 7fef21ffb700 1 -- 172.21.15.158:0/897527765 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fef240583c0 con 0x7fef2c0c9de0 2023-10-16T09:57:09.913 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T09:57:09.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fef1006b020 msgr2=0x7fef1006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fef1006b020 0x7fef1006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef1c014040 tx=0x7fef1c009030 comp rx=0 tx=0).stop 2023-10-16T09:57:09.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef2c0c9de0 msgr2=0x7fef2c0ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:09.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef2c0c9de0 0x7fef2c0ca270 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fef2c06b750 tx=0x7fef24011f60 comp rx=0 tx=0).stop 2023-10-16T09:57:09.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 shutdown_connections 2023-10-16T09:57:09.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fef1006b020 0x7fef1006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef2c0c9de0 0x7fef2c0ca270 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef18003a30 0x7fef2c0b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 --2- 172.21.15.158:0/897527765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef2c1e2a60 0x7fef2c1f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:09.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 >> 172.21.15.158:0/897527765 conn(0x7fef2c06a8d0 msgr2=0x7fef2c0b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:09.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 shutdown_connections 2023-10-16T09:57:09.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:09.914+0000 7fef332c3700 1 -- 172.21.15.158:0/897527765 wait complete. 2023-10-16T09:57:09.927 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T09:57:09.927 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:57:09.928 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T09:57:09.928 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T09:57:09.928 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T09:57:09.929 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T09:57:09.929 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T09:57:09.929 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:57:09.930 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:57:09.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:09.931 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T09:57:09.948 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:57:09.948 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:57:09.948 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:57:09.948 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:57:09.949 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:57:09.949 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:57:09.949 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:57:09.949 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:57:09.949 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:57:09.950 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:57:09.950 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:57:09.950 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T09:57:09.950 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:57:09.950 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T09:57:09.951 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:57:09.951 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 41603sec preferred_lft 41603sec 2023-10-16T09:57:09.951 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T09:57:09.951 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:57:09.951 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:57:09.952 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T09:57:09.952 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T09:57:09.952 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:57:09.952 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T09:57:09.952 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:57:09.953 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:57:09.953 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T09:57:09.953 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T09:57:09.953 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:57:09.953 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:57:09.954 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T09:57:09.954 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T09:57:09.954 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:57:09.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:09.955 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:57:10.013 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T09:57:10.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:10.015 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T09:57:10.074 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T09:57:10.074 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:57:10.074 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:57:10.075 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:57:10.075 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:57:10.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:10.076 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:57:10.076 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:57:10.076 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:57:10.077 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:57:10.077 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:57:10.078 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:57:10.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:10.180 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:57:10.197 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T09:57:10.197 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T09:57:10.198 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:57:10.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:10.199 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T09:57:10.263 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:57:10.264 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T09:57:10.321 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:57:10.321 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:57:10.321 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:57:10.377 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T09:57:10.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:10.377 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:57:10.462 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:57:10.462 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:57:10.486 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:57:10.486 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:57:10.486 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:57:10.486 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:57:10.486 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:57:10.487 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:57:10.487 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:57:10.487 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:57:10.513 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:57:10.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:57:10.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:57:10.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:57:10.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:57:10.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:57:10.515 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:57:10.515 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:57:10.515 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T09:57:10.515 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T09:57:10.516 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T09:57:10.517 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T09:57:10.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:10.518 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:57:10.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:10.550 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:57:10.637 INFO:teuthology.orchestra.run.smithi158.stdout:client6402 2023-10-16T09:57:10.637 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:57:10.637 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:57:10.655 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T09:57:10.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.816+0000 7effc01dc700 1 Processor -- start 2023-10-16T09:57:10.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.816+0000 7effc01dc700 1 -- start start 2023-10-16T09:57:10.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effc01dc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7effb80c4e30 0x7effb81669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effc01dc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb8071be0 con 0x7effb80ca540 2023-10-16T09:57:10.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effc01dc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb80726d0 con 0x7effb80cd430 2023-10-16T09:57:10.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effc01dc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb806b6f0 con 0x7effb80c4e30 2023-10-16T09:57:10.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbdf78700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7effb80ca540 0x7effb806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51036/0 (socket says 172.21.15.158:51036) 2023-10-16T09:57:10.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbdf78700 1 -- 172.21.15.158:0/1910893463 learned_addr learned my addr 172.21.15.158:0/1910893463 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:10.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbe779700 1 --2- 172.21.15.158:0/1910893463 >> v2:172.21.15.172:3300/0 conn(0x7effb80c4e30 0x7effb81669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055008316 0 0) 0x7effb8071be0 con 0x7effb80ca540 2023-10-16T09:57:10.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa40035b0 con 0x7effb80ca540 2023-10-16T09:57:10.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174316266 0 0) 0x7effa40035b0 con 0x7effb80ca540 2023-10-16T09:57:10.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 >> v2:172.21.15.172:3300/0 conn(0x7effb80c4e30 msgr2=0x7effb81669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:10.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 --2- 172.21.15.158:0/1910893463 >> v2:172.21.15.172:3300/0 conn(0x7effb80c4e30 0x7effb81669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:10.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 >> v1:172.21.15.172:6789/0 conn(0x7effb80cd430 legacy=0x7effb806afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:10.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb8071e30 con 0x7effb80ca540 2023-10-16T09:57:10.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7effb4001670 con 0x7effb80ca540 2023-10-16T09:57:10.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 >> v1:172.21.15.158:6789/0 conn(0x7effb80ca540 legacy=0x7effb806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:10.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4004850 0x7effa4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa4005330 0x7effa400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb806b6f0 con 0x7effa4005330 2023-10-16T09:57:10.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c200 con 0x7effa4003a00 2023-10-16T09:57:10.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c3c0 con 0x7effa4004850 2023-10-16T09:57:10.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbe779700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa4005330 0x7effa400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbdf78700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.817+0000 7effbd777700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4004850 0x7effa4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbe779700 1 -- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4004850 msgr2=0x7effa4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:10.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbe779700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4004850 0x7effa4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:10.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbe779700 1 -- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa4003a00 msgr2=0x7effa4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:10.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbe779700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:10.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbe779700 1 -- 172.21.15.158:0/1910893463 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa40035b0 con 0x7effa4005330 2023-10-16T09:57:10.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbdf78700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:57:10.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbe779700 1 -- 172.21.15.158:0/1910893463 reap_dead start 2023-10-16T09:57:10.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbe779700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa4005330 0x7effa400ba10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7effb80c52c0 tx=0x7effa4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1e3b55631b4d54 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:10.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0018070 con 0x7effa4005330 2023-10-16T09:57:10.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb000f040 con 0x7effa4005330 2023-10-16T09:57:10.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effbcf76700 1 -- 172.21.15.158:0/1910893463 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000ec10 con 0x7effa4005330 2023-10-16T09:57:10.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effc01dc700 1 -- 172.21.15.158:0/1910893463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa4005330 msgr2=0x7effa400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:10.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.818+0000 7effc01dc700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa4005330 0x7effa400ba10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7effb80c52c0 tx=0x7effa4005880 comp rx=0 tx=0).stop 2023-10-16T09:57:10.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.819+0000 7effc01dc700 1 -- 172.21.15.158:0/1910893463 shutdown_connections 2023-10-16T09:57:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.819+0000 7effc01dc700 1 --2- 172.21.15.158:0/1910893463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa4005330 0x7effa400ba10 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.819+0000 7effc01dc700 1 -- 172.21.15.158:0/1910893463 >> 172.21.15.158:0/1910893463 conn(0x7effb80cc700 msgr2=0x7effb80ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.819+0000 7effc01dc700 1 -- 172.21.15.158:0/1910893463 shutdown_connections 2023-10-16T09:57:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.819+0000 7effc01dc700 1 -- 172.21.15.158:0/1910893463 wait complete. 2023-10-16T09:57:10.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.820+0000 7effc01dc700 1 Processor -- start 2023-10-16T09:57:10.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.820+0000 7effc01dc700 1 -- start start 2023-10-16T09:57:10.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.820+0000 7effc01dc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4005330 0x7effb8151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.820+0000 7effc01dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effb8151130 0x7effb81f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.820+0000 7effc01dc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb81f1180 0x7effb815b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effc01dc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb80b3ae0 con 0x7effb8151130 2023-10-16T09:57:10.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effc01dc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb80b3dc0 con 0x7effb81f1180 2023-10-16T09:57:10.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effc01dc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb80b40a0 con 0x7effa4005330 2023-10-16T09:57:10.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effb8151130 0x7effb81f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effb8151130 0x7effb81f0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50874/0 (socket says 172.21.15.158:50874) 2023-10-16T09:57:10.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 -- 172.21.15.158:0/3127633794 learned_addr learned my addr 172.21.15.158:0/3127633794 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:10.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbdf78700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4005330 0x7effb8151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbe779700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb81f1180 0x7effb815b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 -- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4005330 msgr2=0x7effb8151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:10.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4005330 0x7effb8151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:10.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 -- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb81f1180 msgr2=0x7effb815b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:10.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb81f1180 0x7effb815b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:10.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 -- 172.21.15.158:0/3127633794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80b4380 con 0x7effb8151130 2023-10-16T09:57:10.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effbd777700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effb8151130 0x7effb81f0c40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7effac00f610 tx=0x7effac00f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:10.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effaa7fc700 1 -- 172.21.15.158:0/3127633794 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac0052e0 con 0x7effb8151130 2023-10-16T09:57:10.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effaa7fc700 1 -- 172.21.15.158:0/3127633794 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac005440 con 0x7effb8151130 2023-10-16T09:57:10.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effaa7fc700 1 -- 172.21.15.158:0/3127633794 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac0038a0 con 0x7effb8151130 2023-10-16T09:57:10.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb80b4660 con 0x7effb8151130 2023-10-16T09:57:10.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.821+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb81549b0 con 0x7effb8151130 2023-10-16T09:57:10.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.822+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb80681b0 con 0x7effb8151130 2023-10-16T09:57:10.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.823+0000 7effaa7fc700 1 -- 172.21.15.158:0/3127633794 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7effac003a00 con 0x7effb8151130 2023-10-16T09:57:10.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.823+0000 7effaa7fc700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7eff9c06b020 0x7eff9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:10.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.823+0000 7effaa7fc700 1 -- 172.21.15.158:0/3127633794 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7effac0897b0 con 0x7effb8151130 2023-10-16T09:57:10.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.823+0000 7effbdf78700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7eff9c06b020 0x7eff9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:10.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.823+0000 7effbdf78700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7eff9c06b020 0x7eff9c06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7effb4000ff0 tx=0x7effb4009040 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:10.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:10.827+0000 7effaa7fc700 1 -- 172.21.15.158:0/3127633794 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7effac054480 con 0x7effb8151130 2023-10-16T09:57:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.046+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 --> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7effb80723f0 con 0x7eff9c06b020 2023-10-16T09:57:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.047+0000 7effaa7fc700 1 -- 172.21.15.158:0/3127633794 <== mgr.6009 v2:172.21.15.158:6832/211595975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7effb4001660 con 0x7eff9c06b020 2023-10-16T09:57:11.050 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:57:11.050 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:57:11.050 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T09:57:11.050 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:57:11.051 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:57:11.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7eff9c06b020 msgr2=0x7eff9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7eff9c06b020 0x7eff9c06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7effb4000ff0 tx=0x7effb4009040 comp rx=0 tx=0).stop 2023-10-16T09:57:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effb8151130 msgr2=0x7effb81f0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effb8151130 0x7effb81f0c40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7effac00f610 tx=0x7effac00f9d0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 shutdown_connections 2023-10-16T09:57:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7eff9c06b020 0x7eff9c06d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb81f1180 0x7effb815b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effa4005330 0x7effb8151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 --2- 172.21.15.158:0/3127633794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effb8151130 0x7effb81f0c40 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 >> 172.21.15.158:0/3127633794 conn(0x7effb80cc700 msgr2=0x7effb815dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 shutdown_connections 2023-10-16T09:57:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.050+0000 7effc01dc700 1 -- 172.21.15.158:0/3127633794 wait complete. 2023-10-16T09:57:11.064 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T09:57:11.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.184+0000 7f2b135d1700 1 Processor -- start 2023-10-16T09:57:11.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.184+0000 7f2b135d1700 1 -- start start 2023-10-16T09:57:11.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b135d1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b0c0b36c0 0x7f2b0c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b135d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 0x7f2b0c15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b135d1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c15f730 con 0x7f2b0c15c720 2023-10-16T09:57:11.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b135d1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b0c15ffd0 con 0x7f2b0c0b36c0 2023-10-16T09:57:11.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b135d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c15bab0 con 0x7f2b0c0d0cd0 2023-10-16T09:57:11.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b11b6e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2b0c15c720 0x7f2b0c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51058/0 (socket says 172.21.15.158:51058) 2023-10-16T09:57:11.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b11b6e700 1 -- 172.21.15.158:0/1730612484 learned_addr learned my addr 172.21.15.158:0/1730612484 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:11.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046674027 0 0) 0x7f2b0c15f730 con 0x7f2b0c15c720 2023-10-16T09:57:11.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b10b6c700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 0x7f2b0c15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af80035b0 con 0x7f2b0c15c720 2023-10-16T09:57:11.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b1136d700 1 --2- 172.21.15.158:0/1730612484 >> v2:172.21.15.172:3300/0 conn(0x7f2b0c0b36c0 0x7f2b0c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314424742 0 0) 0x7f2af80035b0 con 0x7f2b0c15c720 2023-10-16T09:57:11.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 msgr2=0x7f2b0c15c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:11.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 0x7f2b0c15c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 >> v2:172.21.15.172:3300/0 conn(0x7f2b0c0b36c0 msgr2=0x7f2b0c15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:11.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 --2- 172.21.15.158:0/1730612484 >> v2:172.21.15.172:3300/0 conn(0x7f2b0c0b36c0 0x7f2b0c15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c15fa10 con 0x7f2b0c15c720 2023-10-16T09:57:11.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.185+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f2b040019f0 con 0x7f2b0c15c720 2023-10-16T09:57:11.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 >> v1:172.21.15.158:6789/0 conn(0x7f2b0c15c720 legacy=0x7f2b0c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2af8003a00 0x7f2af8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 0x7f2af8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2af8004d90 0x7f2af8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c15ffd0 con 0x7f2af8004d90 2023-10-16T09:57:11.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c15bab0 con 0x7f2af8003a00 2023-10-16T09:57:11.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af800ba10 con 0x7f2b0c0d0cd0 2023-10-16T09:57:11.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b11b6e700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2af8004d90 0x7f2af8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b1136d700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2af8003a00 0x7f2af8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b10b6c700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 0x7f2af8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b11b6e700 1 -- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 msgr2=0x7f2af8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:11.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b11b6e700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c0d0cd0 0x7f2af8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b11b6e700 1 -- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2af8003a00 msgr2=0x7f2af8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:11.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b11b6e700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2af8003a00 0x7f2af8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b11b6e700 1 -- 172.21.15.158:0/1730612484 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80035b0 con 0x7f2af8004d90 2023-10-16T09:57:11.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b11b6e700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2af8004d90 0x7f2af8005210 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400a830 tx=0x7f2af8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c702019e0d6bae4b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:11.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04012600 con 0x7f2af8004d90 2023-10-16T09:57:11.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b040127a0 con 0x7f2af8004d90 2023-10-16T09:57:11.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.186+0000 7f2b03fff700 1 -- 172.21.15.158:0/1730612484 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04012b50 con 0x7f2af8004d90 2023-10-16T09:57:11.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.187+0000 7f2b135d1700 1 -- 172.21.15.158:0/1730612484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2af8004d90 msgr2=0x7f2af8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.187+0000 7f2b135d1700 1 --2- 172.21.15.158:0/1730612484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2af8004d90 0x7f2af8005210 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400a830 tx=0x7f2af8005880 comp rx=0 tx=0).stop 2023-10-16T09:57:11.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.187+0000 7f2b11b6e700 1 -- 172.21.15.158:0/1730612484 reap_dead start 2023-10-16T09:57:11.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.187+0000 7f2b135d1700 1 -- 172.21.15.158:0/1730612484 shutdown_connections 2023-10-16T09:57:11.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.187+0000 7f2b135d1700 1 -- 172.21.15.158:0/1730612484 >> 172.21.15.158:0/1730612484 conn(0x7f2b0c0cdf90 msgr2=0x7f2b0c0ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:11.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.187+0000 7f2b135d1700 1 -- 172.21.15.158:0/1730612484 shutdown_connections 2023-10-16T09:57:11.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.187+0000 7f2b135d1700 1 -- 172.21.15.158:0/1730612484 wait complete. 2023-10-16T09:57:11.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.188+0000 7f2b135d1700 1 Processor -- start 2023-10-16T09:57:11.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.188+0000 7f2b135d1700 1 -- start start 2023-10-16T09:57:11.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b135d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b0c14f970 0x7f2b0c0b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b135d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c14fd50 0x7f2b0c0b6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b135d1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b0c0b3ce0 0x7f2b0c0b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b135d1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c1547b0 con 0x7f2b0c14f970 2023-10-16T09:57:11.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b135d1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c0b4ab0 con 0x7f2b0c0b3ce0 2023-10-16T09:57:11.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b135d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c0b4d90 con 0x7f2b0c14fd50 2023-10-16T09:57:11.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b0c14f970 0x7f2b0c0b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b0c14f970 0x7f2b0c0b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50894/0 (socket says 172.21.15.158:50894) 2023-10-16T09:57:11.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 -- 172.21.15.158:0/2114508322 learned_addr learned my addr 172.21.15.158:0/2114508322 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:11.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b10b6c700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c14fd50 0x7f2b0c0b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b11b6e700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b0c0b3ce0 0x7f2b0c0b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 -- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c14fd50 msgr2=0x7f2b0c0b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c14fd50 0x7f2b0c0b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 -- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b0c0b3ce0 msgr2=0x7f2b0c0b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b0c0b3ce0 0x7f2b0c0b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 -- 172.21.15.158:0/2114508322 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c0b5070 con 0x7f2b0c14f970 2023-10-16T09:57:11.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b1136d700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b0c14f970 0x7f2b0c0b61d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2b08011a70 tx=0x7f2b08011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:11.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.189+0000 7f2b01ffb700 1 -- 172.21.15.158:0/2114508322 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08011ea0 con 0x7f2b0c14f970 2023-10-16T09:57:11.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.190+0000 7f2b01ffb700 1 -- 172.21.15.158:0/2114508322 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0800f940 con 0x7f2b0c14f970 2023-10-16T09:57:11.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.190+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0c0b5350 con 0x7f2b0c14f970 2023-10-16T09:57:11.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.190+0000 7f2b01ffb700 1 -- 172.21.15.158:0/2114508322 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0800fcd0 con 0x7f2b0c14f970 2023-10-16T09:57:11.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.190+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0c150bc0 con 0x7f2b0c14f970 2023-10-16T09:57:11.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.191+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0c074a90 con 0x7f2b0c14f970 2023-10-16T09:57:11.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.191+0000 7f2b01ffb700 1 -- 172.21.15.158:0/2114508322 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 19) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b08014440 con 0x7f2b0c14f970 2023-10-16T09:57:11.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.191+0000 7f2b01ffb700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f2af006b020 0x7f2af006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.192+0000 7f2b01ffb700 1 -- 172.21.15.158:0/2114508322 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f2b0808a010 con 0x7f2b0c14f970 2023-10-16T09:57:11.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.192+0000 7f2b10b6c700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f2af006b020 0x7f2af006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.192+0000 7f2b10b6c700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f2af006b020 0x7f2af006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0b5e40 tx=0x7f2afc009210 comp rx=0 tx=0).ready entity=mgr.6009 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:11.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.196+0000 7f2b01ffb700 1 -- 172.21.15.158:0/2114508322 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b08054ce0 con 0x7f2b0c14f970 2023-10-16T09:57:11.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.444+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2b0c0681b0 con 0x7f2b0c14f970 2023-10-16T09:57:11.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.533+0000 7f2b01ffb700 1 -- 172.21.15.158:0/2114508322 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2b08058300 con 0x7f2b0c14f970 2023-10-16T09:57:11.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.536+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f2af006b020 msgr2=0x7f2af006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.536+0000 7f2b135d1700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f2af006b020 0x7f2af006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0b5e40 tx=0x7f2afc009210 comp rx=0 tx=0).stop 2023-10-16T09:57:11.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.536+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b0c14f970 msgr2=0x7f2b0c0b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.536+0000 7f2b135d1700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b0c14f970 0x7f2b0c0b61d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2b08011a70 tx=0x7f2b08011d80 comp rx=0 tx=0).stop 2023-10-16T09:57:11.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.538+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 shutdown_connections 2023-10-16T09:57:11.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.538+0000 7f2b135d1700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f2af006b020 0x7f2af006d4e0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0b5e40 tx=0x7f2afc009210 comp rx=0 tx=0).stop 2023-10-16T09:57:11.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.538+0000 7f2b135d1700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b0c0b3ce0 0x7f2b0c0b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.538+0000 7f2b135d1700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b0c14f970 0x7f2b0c0b61d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.538+0000 7f2b135d1700 1 --2- 172.21.15.158:0/2114508322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b0c14fd50 0x7f2b0c0b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.538+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 >> 172.21.15.158:0/2114508322 conn(0x7f2b0c0cdf90 msgr2=0x7f2b0c159b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:11.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.540+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 shutdown_connections 2023-10-16T09:57:11.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.540+0000 7f2b135d1700 1 -- 172.21.15.158:0/2114508322 wait complete. 2023-10-16T09:57:11.558 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T09:57:11.678 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:11.676+0000 7f676f992200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:57:11.684 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:11.682+0000 7f249f052200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:57:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.683+0000 7fb0588fc700 1 Processor -- start 2023-10-16T09:57:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.683+0000 7fb0588fc700 1 -- start start 2023-10-16T09:57:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb0588fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb0540cc590 0x7fb0540caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb0588fc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0540cbd30 con 0x7fb05415da40 2023-10-16T09:57:11.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb0588fc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0540cc1c0 con 0x7fb05406afe0 2023-10-16T09:57:11.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb0588fc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05415de20 con 0x7fb0540cc590 2023-10-16T09:57:11.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb05259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb05415da40 0x7fb05406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51100/0 (socket says 172.21.15.158:51100) 2023-10-16T09:57:11.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb05259c700 1 -- 172.21.15.158:0/547712941 learned_addr learned my addr 172.21.15.158:0/547712941 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:11.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb052d9d700 1 --2- 172.21.15.158:0/547712941 >> v2:172.21.15.172:3300/0 conn(0x7fb0540cc590 0x7fb0540caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb05159a700 1 -- 172.21.15.158:0/547712941 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094699580 0 0) 0x7fb0540cbd30 con 0x7fb05415da40 2023-10-16T09:57:11.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.684+0000 7fb05159a700 1 -- 172.21.15.158:0/547712941 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c0035b0 con 0x7fb05415da40 2023-10-16T09:57:11.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb05159a700 1 -- 172.21.15.158:0/547712941 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275578369 0 0) 0x7fb0540cc1c0 con 0x7fb05406afe0 2023-10-16T09:57:11.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb05159a700 1 -- 172.21.15.158:0/547712941 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0540cbd30 con 0x7fb05406afe0 2023-10-16T09:57:11.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb052d9d700 1 -- 172.21.15.158:0/547712941 >> v1:172.21.15.172:6789/0 conn(0x7fb05406afe0 legacy=0x7fb0540afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb052d9d700 1 -- 172.21.15.158:0/547712941 >> v1:172.21.15.158:6789/0 conn(0x7fb05415da40 legacy=0x7fb05406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb052d9d700 1 -- 172.21.15.158:0/547712941 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0540ce260 con 0x7fb0540cc590 2023-10-16T09:57:11.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb052d9d700 1 --2- 172.21.15.158:0/547712941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 0x7fb0540caa20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009d90 tx=0x7fb04c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5331ce7b89103e22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:11.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb05159a700 1 -- 172.21.15.158:0/547712941 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c00e070 con 0x7fb0540cc590 2023-10-16T09:57:11.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb05159a700 1 -- 172.21.15.158:0/547712941 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c00bb20 con 0x7fb0540cc590 2023-10-16T09:57:11.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.685+0000 7fb05159a700 1 -- 172.21.15.158:0/547712941 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c012430 con 0x7fb0540cc590 2023-10-16T09:57:11.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.686+0000 7fb0588fc700 1 -- 172.21.15.158:0/547712941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 msgr2=0x7fb0540caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.686+0000 7fb0588fc700 1 --2- 172.21.15.158:0/547712941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 0x7fb0540caa20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009d90 tx=0x7fb04c00b330 comp rx=0 tx=0).stop 2023-10-16T09:57:11.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.686+0000 7fb0588fc700 1 -- 172.21.15.158:0/547712941 shutdown_connections 2023-10-16T09:57:11.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.686+0000 7fb0588fc700 1 --2- 172.21.15.158:0/547712941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 0x7fb0540caa20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.686+0000 7fb0588fc700 1 -- 172.21.15.158:0/547712941 >> 172.21.15.158:0/547712941 conn(0x7fb054099a10 msgr2=0x7fb0540d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:11.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.686+0000 7fb0588fc700 1 -- 172.21.15.158:0/547712941 shutdown_connections 2023-10-16T09:57:11.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.686+0000 7fb0588fc700 1 -- 172.21.15.158:0/547712941 wait complete. 2023-10-16T09:57:11.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.687+0000 7fb0588fc700 1 Processor -- start 2023-10-16T09:57:11.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.687+0000 7fb0588fc700 1 -- start start 2023-10-16T09:57:11.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb0588fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb05406afe0 0x7fb05415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb0588fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 0x7fb0540c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb0588fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb05415da40 0x7fb0540c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb0588fc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540c5710 con 0x7fb05406afe0 2023-10-16T09:57:11.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb0588fc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540c87d0 con 0x7fb0540cc590 2023-10-16T09:57:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb0588fc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540c4810 con 0x7fb05415da40 2023-10-16T09:57:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb05406afe0 0x7fb05415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb05406afe0 0x7fb05415a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50930/0 (socket says 172.21.15.158:50930) 2023-10-16T09:57:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 -- 172.21.15.158:0/1071785261 learned_addr learned my addr 172.21.15.158:0/1071785261 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb051d9b700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 0x7fb0540c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb052d9d700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb05415da40 0x7fb0540c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:11.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 -- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb05415da40 msgr2=0x7fb0540c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb05415da40 0x7fb0540c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 -- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 msgr2=0x7fb0540c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 0x7fb0540c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 -- 172.21.15.158:0/1071785261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c009a70 con 0x7fb05406afe0 2023-10-16T09:57:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb052d9d700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb05415da40 0x7fb0540c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.688+0000 7fb05259c700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb05406afe0 0x7fb05415a7e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb044001eb0 tx=0x7fb04400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.689+0000 7fb042ffd700 1 -- 172.21.15.158:0/1071785261 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044005d70 con 0x7fb05406afe0 2023-10-16T09:57:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.689+0000 7fb042ffd700 1 -- 172.21.15.158:0/1071785261 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0440070d0 con 0x7fb05406afe0 2023-10-16T09:57:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.689+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0540c4a90 con 0x7fb05406afe0 2023-10-16T09:57:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.689+0000 7fb042ffd700 1 -- 172.21.15.158:0/1071785261 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044007440 con 0x7fb05406afe0 2023-10-16T09:57:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.689+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0540c5050 con 0x7fb05406afe0 2023-10-16T09:57:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.690+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0540681b0 con 0x7fb05406afe0 2023-10-16T09:57:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.690+0000 7fb042ffd700 1 -- 172.21.15.158:0/1071785261 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 20) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fb0440075a0 con 0x7fb05406afe0 2023-10-16T09:57:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.691+0000 7fb042ffd700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 0x7fb03406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:11.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.691+0000 7fb051d9b700 1 -- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 msgr2=0x7fb03406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:11.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.691+0000 7fb042ffd700 1 -- 172.21.15.158:0/1071785261 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fb044088d90 con 0x7fb05406afe0 2023-10-16T09:57:11.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.691+0000 7fb051d9b700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 0x7fb03406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:57:11.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.695+0000 7fb042ffd700 1 -- 172.21.15.158:0/1071785261 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb044053b10 con 0x7fb05406afe0 2023-10-16T09:57:11.744 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:11.743+0000 7f676f992200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:57:11.754 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:11.752+0000 7f249f052200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:57:11.816 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:11.815+0000 7f676f992200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:57:11.829 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:11.826+0000 7f249f052200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:57:11.893 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:11.892+0000 7f676f992200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:57:11.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.892+0000 7fb051d9b700 1 -- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 msgr2=0x7fb03406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:11.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.892+0000 7fb051d9b700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 0x7fb03406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:57:11.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.898+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb05406ee40 con 0x7fb05406afe0 2023-10-16T09:57:11.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.901+0000 7fb042ffd700 1 -- 172.21.15.158:0/1071785261 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb0440170d0 con 0x7fb05406afe0 2023-10-16T09:57:11.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 msgr2=0x7fb03406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:57:11.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 0x7fb03406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb05406afe0 msgr2=0x7fb05415a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:11.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb05406afe0 0x7fb05415a7e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb044001eb0 tx=0x7fb04400c7b0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 shutdown_connections 2023-10-16T09:57:11.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7fb03406b070 0x7fb03406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb05415da40 0x7fb0540c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb05406afe0 0x7fb05415a7e0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 --2- 172.21.15.158:0/1071785261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0540cc590 0x7fb0540c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:11.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 >> 172.21.15.158:0/1071785261 conn(0x7fb054099a10 msgr2=0x7fb05406f360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:11.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 shutdown_connections 2023-10-16T09:57:11.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:11.904+0000 7fb0588fc700 1 -- 172.21.15.158:0/1071785261 wait complete. 2023-10-16T09:57:11.909 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:11.906+0000 7f249f052200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:57:11.919 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T09:57:12.035 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:12.034+0000 7f676f992200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:57:12.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.042+0000 7f9f7e856700 1 Processor -- start 2023-10-16T09:57:12.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.042+0000 7f9f7e856700 1 -- start start 2023-10-16T09:57:12.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7e856700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f7815d6c0 0x7f9f780cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7e856700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f780cf360 0x7f9f780d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7e856700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f780d1ee0 con 0x7f9f7815daa0 2023-10-16T09:57:12.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7e856700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780d40a0 con 0x7f9f780cf360 2023-10-16T09:57:12.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7e856700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780d12d0 con 0x7f9f7815d6c0 2023-10-16T09:57:12.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7cdf3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f780cf360 0x7f9f780d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7cdf3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f780cf360 0x7f9f780d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50988/0 (socket says 172.21.15.158:50988) 2023-10-16T09:57:12.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f7cdf3700 1 -- 172.21.15.158:0/1283061355 learned_addr learned my addr 172.21.15.158:0/1283061355 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:12.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.043+0000 7f9f77fff700 1 --2- 172.21.15.158:0/1283061355 >> v2:172.21.15.172:3300/0 conn(0x7f9f7815d6c0 0x7f9f780cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f7cdf3700 1 -- 172.21.15.158:0/1283061355 >> v2:172.21.15.172:3300/0 conn(0x7f9f7815d6c0 msgr2=0x7f9f780cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:12.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f7cdf3700 1 --2- 172.21.15.158:0/1283061355 >> v2:172.21.15.172:3300/0 conn(0x7f9f7815d6c0 0x7f9f780cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f7cdf3700 1 -- 172.21.15.158:0/1283061355 >> v1:172.21.15.172:6789/0 conn(0x7f9f7815daa0 legacy=0x7f9f780cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f7cdf3700 1 -- 172.21.15.158:0/1283061355 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f780d15b0 con 0x7f9f780cf360 2023-10-16T09:57:12.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f7cdf3700 1 --2- 172.21.15.158:0/1283061355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f780cf360 0x7f9f780d0470 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9f780674f0 tx=0x7f9f7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad810a0f2f8e761 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:12.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f76ffd700 1 -- 172.21.15.158:0/1283061355 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000e070 con 0x7f9f780cf360 2023-10-16T09:57:12.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f76ffd700 1 -- 172.21.15.158:0/1283061355 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7000b910 con 0x7f9f780cf360 2023-10-16T09:57:12.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.044+0000 7f9f76ffd700 1 -- 172.21.15.158:0/1283061355 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000bcc0 con 0x7f9f780cf360 2023-10-16T09:57:12.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 -- 172.21.15.158:0/1283061355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f780cf360 msgr2=0x7f9f780d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1283061355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f780cf360 0x7f9f780d0470 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9f780674f0 tx=0x7f9f7000b330 comp rx=0 tx=0).stop 2023-10-16T09:57:12.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 -- 172.21.15.158:0/1283061355 shutdown_connections 2023-10-16T09:57:12.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1283061355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f780cf360 0x7f9f780d0470 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1283061355 >> v2:172.21.15.172:3300/0 conn(0x7f9f7815d6c0 0x7f9f780cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 -- 172.21.15.158:0/1283061355 >> 172.21.15.158:0/1283061355 conn(0x7f9f7809c940 msgr2=0x7f9f780d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:12.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 -- 172.21.15.158:0/1283061355 shutdown_connections 2023-10-16T09:57:12.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.045+0000 7f9f7e856700 1 -- 172.21.15.158:0/1283061355 wait complete. 2023-10-16T09:57:12.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.046+0000 7f9f7e856700 1 Processor -- start 2023-10-16T09:57:12.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.047+0000 7f9f7e856700 1 -- start start 2023-10-16T09:57:12.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.047+0000 7f9f7e856700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cf360 0x7f9f780cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.047+0000 7f9f7e856700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7815d6c0 0x7f9f7815f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.047+0000 7f9f7e856700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f7815daa0 0x7f9f7815fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.047+0000 7f9f7e856700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780b49d0 con 0x7f9f7815d6c0 2023-10-16T09:57:12.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.047+0000 7f9f7e856700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780cb6c0 con 0x7f9f7815daa0 2023-10-16T09:57:12.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.047+0000 7f9f7e856700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780cb080 con 0x7f9f780cf360 2023-10-16T09:57:12.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7815d6c0 0x7f9f7815f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7815d6c0 0x7f9f7815f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:50992/0 (socket says 172.21.15.158:50992) 2023-10-16T09:57:12.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 -- 172.21.15.158:0/1541682211 learned_addr learned my addr 172.21.15.158:0/1541682211 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:12.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f77fff700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cf360 0x7f9f780cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f7cdf3700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f7815daa0 0x7f9f7815fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 -- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cf360 msgr2=0x7f9f780cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cf360 0x7f9f780cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 -- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f7815daa0 msgr2=0x7f9f7815fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f7815daa0 0x7f9f7815fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 -- 172.21.15.158:0/1541682211 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f70009b30 con 0x7f9f7815d6c0 2023-10-16T09:57:12.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f7cdf3700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f7815daa0 0x7f9f7815fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:57:12.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f777fe700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7815d6c0 0x7f9f7815f4e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c420 tx=0x7f9f6c00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:12.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f74ff9700 1 -- 172.21.15.158:0/1541682211 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0052d0 con 0x7f9f7815d6c0 2023-10-16T09:57:12.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f74ff9700 1 -- 172.21.15.158:0/1541682211 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c005430 con 0x7f9f7815d6c0 2023-10-16T09:57:12.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f781604b0 con 0x7f9f7815d6c0 2023-10-16T09:57:12.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f74ff9700 1 -- 172.21.15.158:0/1541682211 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0038c0 con 0x7f9f7815d6c0 2023-10-16T09:57:12.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.048+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f780cb350 con 0x7f9f7815d6c0 2023-10-16T09:57:12.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.049+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f780cb510 con 0x7f9f7815d6c0 2023-10-16T09:57:12.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.050+0000 7f9f74ff9700 1 -- 172.21.15.158:0/1541682211 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 20) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9f6c003a20 con 0x7f9f7815d6c0 2023-10-16T09:57:12.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.051+0000 7f9f74ff9700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.051+0000 7f9f77fff700 1 -- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 msgr2=0x7f9f6406d560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:12.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.051+0000 7f9f74ff9700 1 -- 172.21.15.158:0/1541682211 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f9f6c088d50 con 0x7f9f7815d6c0 2023-10-16T09:57:12.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.051+0000 7f9f77fff700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:57:12.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.059+0000 7f9f74ff9700 1 -- 172.21.15.158:0/1541682211 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f6c08c030 con 0x7f9f7815d6c0 2023-10-16T09:57:12.064 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:12.060+0000 7f249f052200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:57:12.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.252+0000 7f9f77fff700 1 -- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 msgr2=0x7f9f6406d560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:12.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.252+0000 7f9f77fff700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:57:12.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.257+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9f780681b0 con 0x7f9f7815d6c0 2023-10-16T09:57:12.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.260+0000 7f9f74ff9700 1 -- 172.21.15.158:0/1541682211 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9f6c053a20 con 0x7f9f7815d6c0 2023-10-16T09:57:12.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.262+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 msgr2=0x7f9f6406d560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:57:12.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.262+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.262+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7815d6c0 msgr2=0x7f9f7815f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.262+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7815d6c0 0x7f9f7815f4e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c420 tx=0x7f9f6c00c730 comp rx=0 tx=0).stop 2023-10-16T09:57:12.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 shutdown_connections 2023-10-16T09:57:12.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f7815daa0 0x7f9f7815fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cf360 0x7f9f780cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 --2- 172.21.15.158:0/1541682211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7815d6c0 0x7f9f7815f4e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 >> 172.21.15.158:0/1541682211 conn(0x7f9f7809c940 msgr2=0x7f9f7815c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:12.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 shutdown_connections 2023-10-16T09:57:12.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.263+0000 7f9f7e856700 1 -- 172.21.15.158:0/1541682211 wait complete. 2023-10-16T09:57:12.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:12.278 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T09:57:12.299 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T09:57:12.359 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:12.358+0000 7f676f992200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:57:12.403 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:12.401+0000 7f249f052200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:57:12.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.465+0000 7f757b4e4700 1 Processor -- start 2023-10-16T09:57:12.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.465+0000 7f757b4e4700 1 -- start start 2023-10-16T09:57:12.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.466+0000 7f757b4e4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f757415dbd0 0x7f757406df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.466+0000 7f757b4e4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f757406afe0 0x7f75740718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.466+0000 7f757b4e4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757415e510 con 0x7f757415dfc0 2023-10-16T09:57:12.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.466+0000 7f757b4e4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75740ccbb0 con 0x7f757415dbd0 2023-10-16T09:57:12.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.466+0000 7f757b4e4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757415e7f0 con 0x7f757406afe0 2023-10-16T09:57:12.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f757415dbd0 0x7f757406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f757415dbd0 0x7f757406df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51044/0 (socket says 172.21.15.158:51044) 2023-10-16T09:57:12.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 -- 172.21.15.158:0/1079252753 learned_addr learned my addr 172.21.15.158:0/1079252753 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:12.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579a81700 1 --2- 172.21.15.158:0/1079252753 >> v2:172.21.15.172:3300/0 conn(0x7f757406afe0 0x7f75740718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 -- 172.21.15.158:0/1079252753 >> v2:172.21.15.172:3300/0 conn(0x7f757406afe0 msgr2=0x7f75740718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:12.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 --2- 172.21.15.158:0/1079252753 >> v2:172.21.15.172:3300/0 conn(0x7f757406afe0 0x7f75740718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 -- 172.21.15.158:0/1079252753 >> v1:172.21.15.172:6789/0 conn(0x7f757415dfc0 legacy=0x7f757406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 -- 172.21.15.158:0/1079252753 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757415ead0 con 0x7f757415dbd0 2023-10-16T09:57:12.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f7579280700 1 --2- 172.21.15.158:0/1079252753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dbd0 0x7f757406df00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f75740ccad0 tx=0x7f7570009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475e478f5fe02c23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:12.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f756bfff700 1 -- 172.21.15.158:0/1079252753 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570011830 con 0x7f757415dbd0 2023-10-16T09:57:12.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f756bfff700 1 -- 172.21.15.158:0/1079252753 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7570011990 con 0x7f757415dbd0 2023-10-16T09:57:12.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.467+0000 7f756bfff700 1 -- 172.21.15.158:0/1079252753 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570011d20 con 0x7f757415dbd0 2023-10-16T09:57:12.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 -- 172.21.15.158:0/1079252753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dbd0 msgr2=0x7f757406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 --2- 172.21.15.158:0/1079252753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dbd0 0x7f757406df00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f75740ccad0 tx=0x7f7570009bd0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 -- 172.21.15.158:0/1079252753 shutdown_connections 2023-10-16T09:57:12.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 --2- 172.21.15.158:0/1079252753 >> v2:172.21.15.172:3300/0 conn(0x7f757406afe0 0x7f75740718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 --2- 172.21.15.158:0/1079252753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dbd0 0x7f757406df00 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 -- 172.21.15.158:0/1079252753 >> 172.21.15.158:0/1079252753 conn(0x7f75740b1f00 msgr2=0x7f75740b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:12.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 -- 172.21.15.158:0/1079252753 shutdown_connections 2023-10-16T09:57:12.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.468+0000 7f757b4e4700 1 -- 172.21.15.158:0/1079252753 wait complete. 2023-10-16T09:57:12.476 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:12.473+0000 7f676f992200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:57:12.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.469+0000 7f757b4e4700 1 Processor -- start 2023-10-16T09:57:12.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.469+0000 7f757b4e4700 1 -- start start 2023-10-16T09:57:12.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f757b4e4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f757406afe0 0x7f75740c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f757b4e4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dfc0 0x7f75740c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f757b4e4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740c8a90 0x7f75740c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f757b4e4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75740c9480 con 0x7f757415dfc0 2023-10-16T09:57:12.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f757b4e4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75740ca360 con 0x7f757406afe0 2023-10-16T09:57:12.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f757b4e4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75740c3f00 con 0x7f75740c8a90 2023-10-16T09:57:12.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f757406afe0 0x7f75740c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7578a7f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dfc0 0x7f75740c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f757406afe0 0x7f75740c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:52230/0 (socket says 172.21.15.158:52230) 2023-10-16T09:57:12.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 -- 172.21.15.158:0/821229978 learned_addr learned my addr 172.21.15.158:0/821229978 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:12.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579a81700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740c8a90 0x7f75740c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:12.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740c8a90 msgr2=0x7f75740c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740c8a90 0x7f75740c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dfc0 msgr2=0x7f75740c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:12.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dfc0 0x7f75740c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:12.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7579280700 1 -- 172.21.15.158:0/821229978 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7570009550 con 0x7f757406afe0 2023-10-16T09:57:12.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.470+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415dfc0 0x7f75740c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:57:12.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.471+0000 7f7579280700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f757406afe0 0x7f75740c9e20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7570009c10 tx=0x7f7570004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:12.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.471+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570004380 con 0x7f757406afe0 2023-10-16T09:57:12.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.471+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757001f070 con 0x7f757406afe0 2023-10-16T09:57:12.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.471+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75740c41a0 con 0x7f757406afe0 2023-10-16T09:57:12.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.471+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75700036a0 con 0x7f757406afe0 2023-10-16T09:57:12.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.471+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75740c4760 con 0x7f757406afe0 2023-10-16T09:57:12.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.472+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75740681b0 con 0x7f757406afe0 2023-10-16T09:57:12.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.472+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 20) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f7570021030 con 0x7f757406afe0 2023-10-16T09:57:12.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.473+0000 7f7569ffb700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 0x7f755806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:12.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.473+0000 7f7578a7f700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 msgr2=0x7f755806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:12.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.473+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f757008ad30 con 0x7f757406afe0 2023-10-16T09:57:12.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.473+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 0x7f755806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:57:12.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.478+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7570055a90 con 0x7f757406afe0 2023-10-16T09:57:12.524 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:12.522+0000 7f249f052200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:57:12.548 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:12.546+0000 7f676f992200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:57:12.599 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:12.597+0000 7f249f052200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:57:12.629 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:12.627+0000 7f676f992200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:57:12.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.674+0000 7f7578a7f700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 msgr2=0x7f755806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:12.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.674+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 0x7f755806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:57:12.681 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:12.679+0000 7f249f052200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:57:12.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:12.704+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 --> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f757406e900 con 0x7f755806b070 2023-10-16T09:57:12.995 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:12.994+0000 7f676f992200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:57:13.053 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:13.051+0000 7f249f052200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:57:13.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:13.075+0000 7f7578a7f700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 msgr2=0x7f755806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:13.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:13.075+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 0x7f755806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T09:57:13.562 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:13.561+0000 7f676f992200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:57:13.624 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:13.622+0000 7f249f052200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:57:13.631 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:13.629+0000 7f676f992200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:57:13.694 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:13.692+0000 7f249f052200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:57:13.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:13.876+0000 7f7578a7f700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 msgr2=0x7f755806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:13.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:13.876+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 0x7f755806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T09:57:14.260 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:14.259+0000 7f676f992200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:57:14.327 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:14.325+0000 7f249f052200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:57:14.337 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:14.336+0000 7f676f992200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:57:14.405 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:14.403+0000 7f249f052200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:57:14.480 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:14.479+0000 7f676f992200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:57:14.548 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:14.546+0000 7f249f052200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:57:14.705 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:14.704+0000 7f676f992200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:57:14.775 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:14.773+0000 7f249f052200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:57:14.778 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:14.777+0000 7f676f992200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:57:14.848 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:14.846+0000 7f249f052200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:57:15.041 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:15.039+0000 7f676f992200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:57:15.112 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:15.110+0000 7f249f052200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:57:15.259 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:15.258+0000 7f676f992200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:57:15.330 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:15.328+0000 7f249f052200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:57:15.406 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:15.405+0000 7f676f992200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:57:15.478 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:15.476+0000 7f249f052200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:57:15.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:15.477+0000 7f7578a7f700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 msgr2=0x7f755806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/211595975 2023-10-16T09:57:15.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:15.477+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 0x7f755806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T09:57:15.519 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:15.518+0000 7f676f992200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:57:15.591 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:15.589+0000 7f249f052200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:57:15.672 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:15.671+0000 7f676f992200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:57:15.747 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:15.744+0000 7f249f052200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:57:15.934 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:15.933+0000 7f676f992200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:57:16.008 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.006+0000 7f249f052200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:57:16.093 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:16.091+0000 7f676f992200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:57:16.166 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.164+0000 7f249f052200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:57:16.611 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:57:16.610+0000 7f676f992200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:57:16.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:16.678+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 21) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f7570055e60 con 0x7f757406afe0 2023-10-16T09:57:16.693 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.691+0000 7f249f052200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:57:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:16.712+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 8 ==== mgrmap(e 22) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f7570088f50 con 0x7f757406afe0 2023-10-16T09:57:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:16.712+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 msgr2=0x7f755806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:57:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:16.712+0000 7f7569ffb700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/211595975,v1:172.21.15.158:6833/211595975] conn(0x7f755806b070 0x7f755806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:16.860 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f2415033700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.861 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f2415033700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.861 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f2415033700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.861 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f2415033700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.861 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f2415033700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.861 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f241202d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.862 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f241202d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.862 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f241202d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.862 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f241202d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:16.862 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:57:16.858+0000 7f241202d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:57:17.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.719+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mon.1 v2:172.21.15.172:3300/0 9 ==== mgrmap(e 23) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f7570082aa0 con 0x7f757406afe0 2023-10-16T09:57:17.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.719+0000 7f7569ffb700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f75580279c0 0x7f7558027da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.719+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 --> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7558071580 con 0x7f75580279c0 2023-10-16T09:57:17.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.721+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f75580279c0 0x7f7558027da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.722+0000 7f7578a7f700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f75580279c0 0x7f7558027da0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f757406ae40 tx=0x7f7564007ec0 comp rx=0 tx=0).ready entity=mgr.6429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:17.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.786+0000 7f7569ffb700 1 -- 172.21.15.158:0/821229978 <== mgr.6429 v2:172.21.15.158:6832/1799865685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7558071580 con 0x7f75580279c0 2023-10-16T09:57:17.788 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1 2023-10-16T09:57:17.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.788+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f75580279c0 msgr2=0x7f7558027da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:17.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.788+0000 7f757b4e4700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f75580279c0 0x7f7558027da0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f757406ae40 tx=0x7f7564007ec0 comp rx=0 tx=0).stop 2023-10-16T09:57:17.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.788+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f757406afe0 msgr2=0x7f75740c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:17.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.788+0000 7f757b4e4700 1 --2- 172.21.15.158:0/821229978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f757406afe0 0x7f75740c9e20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7570009c10 tx=0x7f7570004750 comp rx=0 tx=0).stop 2023-10-16T09:57:17.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.788+0000 7f7579a81700 1 -- 172.21.15.158:0/821229978 reap_dead start 2023-10-16T09:57:17.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.788+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 shutdown_connections 2023-10-16T09:57:17.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.788+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 >> 172.21.15.158:0/821229978 conn(0x7f75740b1f00 msgr2=0x7f75740718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:17.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.789+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 shutdown_connections 2023-10-16T09:57:17.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.789+0000 7f757b4e4700 1 -- 172.21.15.158:0/821229978 wait complete. 2023-10-16T09:57:17.804 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T09:57:17.804 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2023-10-16T09:57:17.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.927+0000 7f1cfbda9700 1 Processor -- start 2023-10-16T09:57:17.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.928+0000 7f1cfbda9700 1 -- start start 2023-10-16T09:57:17.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.928+0000 7f1cfbda9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 0x7f1cf4161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.928+0000 7f1cfbda9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf415ee60 con 0x7f1cf415b540 2023-10-16T09:57:17.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.928+0000 7f1cfbda9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf415e2e0 con 0x7f1cf406f180 2023-10-16T09:57:17.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.928+0000 7f1cfbda9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf415b920 con 0x7f1cf406afe0 2023-10-16T09:57:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.928+0000 7f1cf9b45700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1cf415b540 0x7f1cf406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51406/0 (socket says 172.21.15.158:51406) 2023-10-16T09:57:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf9b45700 1 -- 172.21.15.158:0/2183504754 learned_addr learned my addr 172.21.15.158:0/2183504754 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:57:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cfa346700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 0x7f1cf4161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949606720 0 0) 0x7f1cf415ee60 con 0x7f1cf415b540 2023-10-16T09:57:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce00035b0 con 0x7f1cf415b540 2023-10-16T09:57:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035524914 0 0) 0x7f1cf415e2e0 con 0x7f1cf406f180 2023-10-16T09:57:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf415ee60 con 0x7f1cf406f180 2023-10-16T09:57:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 268066107 0 0) 0x7f1ce00035b0 con 0x7f1cf415b540 2023-10-16T09:57:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 msgr2=0x7f1cf4161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 0x7f1cf4161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:17.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 >> v1:172.21.15.172:6789/0 conn(0x7f1cf406f180 legacy=0x7f1cf406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:17.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf415e5c0 con 0x7f1cf415b540 2023-10-16T09:57:17.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1ce4001670 con 0x7f1cf415b540 2023-10-16T09:57:17.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 >> v1:172.21.15.158:6789/0 conn(0x7f1cf415b540 legacy=0x7f1cf406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:17.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce0003bc0 0x7f1ce0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 0x7f1ce0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce0004d90 0x7f1ce0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf415b920 con 0x7f1ce0003bc0 2023-10-16T09:57:17.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce000bba0 con 0x7f1ce0004d90 2023-10-16T09:57:17.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce000bd60 con 0x7f1cf406afe0 2023-10-16T09:57:17.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce0003bc0 0x7f1ce0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf9344700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 0x7f1ce0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cfa346700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce0004d90 0x7f1ce0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.929+0000 7f1cf9b45700 1 -- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 msgr2=0x7f1ce0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:17.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf406afe0 0x7f1ce0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:17.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf9b45700 1 -- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce0004d90 msgr2=0x7f1ce0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:17.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce0004d90 0x7f1ce0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:17.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf9b45700 1 -- 172.21.15.158:0/2183504754 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00035b0 con 0x7f1ce0003bc0 2023-10-16T09:57:17.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfa346700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce0004d90 0x7f1ce0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:17.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce0003bc0 0x7f1ce0003fc0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400c160 tx=0x7f1ce0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2e7485d50ff8f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:17.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce400cca0 con 0x7f1ce0003bc0 2023-10-16T09:57:17.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce400ce20 con 0x7f1ce0003bc0 2023-10-16T09:57:17.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cf8b43700 1 -- 172.21.15.158:0/2183504754 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4005ba0 con 0x7f1ce0003bc0 2023-10-16T09:57:17.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfbda9700 1 -- 172.21.15.158:0/2183504754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce0003bc0 msgr2=0x7f1ce0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:17.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfbda9700 1 --2- 172.21.15.158:0/2183504754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce0003bc0 0x7f1ce0003fc0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400c160 tx=0x7f1ce0005880 comp rx=0 tx=0).stop 2023-10-16T09:57:17.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfa346700 1 -- 172.21.15.158:0/2183504754 reap_dead start 2023-10-16T09:57:17.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfbda9700 1 -- 172.21.15.158:0/2183504754 shutdown_connections 2023-10-16T09:57:17.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfbda9700 1 -- 172.21.15.158:0/2183504754 >> 172.21.15.158:0/2183504754 conn(0x7f1cf40d03b0 msgr2=0x7f1cf40d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:17.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfbda9700 1 -- 172.21.15.158:0/2183504754 shutdown_connections 2023-10-16T09:57:17.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.930+0000 7f1cfbda9700 1 -- 172.21.15.158:0/2183504754 wait complete. 2023-10-16T09:57:17.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 Processor -- start 2023-10-16T09:57:17.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 -- start start 2023-10-16T09:57:17.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce0003bc0 0x7f1cf415f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf40b2fd0 0x7f1cf40b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf40b52e0 0x7f1cf40b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4160420 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40b3990 con 0x7f1cf40b52e0 2023-10-16T09:57:17.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cfbda9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf415f570 con 0x7f1ce0003bc0 2023-10-16T09:57:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cf9344700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf40b2fd0 0x7f1cf40b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cf9344700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf40b2fd0 0x7f1cf40b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51244/0 (socket says 172.21.15.158:51244) 2023-10-16T09:57:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.932+0000 7f1cf9344700 1 -- 172.21.15.158:0/3922329350 learned_addr learned my addr 172.21.15.158:0/3922329350 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cfa346700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf40b52e0 0x7f1cf40b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce0003bc0 0x7f1cf415f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9344700 1 -- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce0003bc0 msgr2=0x7f1cf415f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9344700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce0003bc0 0x7f1cf415f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9344700 1 -- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf40b52e0 msgr2=0x7f1cf40b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9344700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf40b52e0 0x7f1cf40b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9344700 1 -- 172.21.15.158:0/3922329350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf415f7a0 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cfa346700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf40b52e0 0x7f1cf40b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:57:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce0003bc0 0x7f1cf415f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:57:17.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cf9344700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf40b2fd0 0x7f1cf40b3450 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000f7f0 tx=0x7f1cf000fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:17.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cea7fc700 1 -- 172.21.15.158:0/3922329350 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0005320 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cea7fc700 1 -- 172.21.15.158:0/3922329350 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf0005480 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf415fa80 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cea7fc700 1 -- 172.21.15.158:0/3922329350 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0006aa0 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.933+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf4160040 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.934+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf4160560 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.934+0000 7f1cea7fc700 1 -- 172.21.15.158:0/3922329350 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 23) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1cf0006c00 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.935+0000 7f1cea7fc700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1cd806ada0 0x7f1cd806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:17.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.935+0000 7f1cea7fc700 1 -- 172.21.15.158:0/3922329350 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6769+0+0 (secure 0 0 0) 0x7f1cf0088bf0 con 0x7f1cf40b2fd0 2023-10-16T09:57:17.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.935+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1cd806ada0 0x7f1cd806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:17.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.935+0000 7f1cf9b45700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1cd806ada0 0x7f1cd806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400d010 tx=0x7f1ce4007000 comp rx=0 tx=0).ready entity=mgr.6429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:17.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:17.939+0000 7f1cea7fc700 1 -- 172.21.15.158:0/3922329350 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cf4160560 con 0x7f1cf40b2fd0 2023-10-16T09:57:18.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.157+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 --> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cf415eb80 con 0x7f1cd806ada0 2023-10-16T09:57:18.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.213+0000 7f1cea7fc700 1 -- 172.21.15.158:0/3922329350 <== mgr.6429 v2:172.21.15.158:6832/1799865685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1ce4022070 con 0x7f1cd806ada0 2023-10-16T09:57:18.216 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1 2023-10-16T09:57:18.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1cd806ada0 msgr2=0x7f1cd806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1cd806ada0 0x7f1cd806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400d010 tx=0x7f1ce4007000 comp rx=0 tx=0).stop 2023-10-16T09:57:18.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf40b2fd0 msgr2=0x7f1cf40b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf40b2fd0 0x7f1cf40b3450 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000f7f0 tx=0x7f1cf000fbb0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 shutdown_connections 2023-10-16T09:57:18.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1cd806ada0 0x7f1cd806d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf40b52e0 0x7f1cf40b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce0003bc0 0x7f1cf415f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 --2- 172.21.15.158:0/3922329350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf40b2fd0 0x7f1cf40b3450 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.215+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 >> 172.21.15.158:0/3922329350 conn(0x7f1cf40d03b0 msgr2=0x7f1cf415a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:18.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.216+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 shutdown_connections 2023-10-16T09:57:18.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.216+0000 7f1cfbda9700 1 -- 172.21.15.158:0/3922329350 wait complete. 2023-10-16T09:57:18.230 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T09:57:18.230 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-09_58_00 in ~42s... 2023-10-16T09:57:18.231 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-09_59_00 in ~102s... 2023-10-16T09:57:18.231 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2023-10-16T09:57:18.231 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T09:57:18.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.353+0000 7f1c3501f700 1 Processor -- start 2023-10-16T09:57:18.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.353+0000 7f1c3501f700 1 -- start start 2023-10-16T09:57:18.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c3501f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c30159280 0x7f1c3006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:18.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c3501f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c300cbef0 0x7f1c300cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:18.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c3501f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c30159660 con 0x7f1c300d0e10 2023-10-16T09:57:18.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c3501f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c300cb710 con 0x7f1c300cbef0 2023-10-16T09:57:18.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c3501f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3006b120 con 0x7f1c30159280 2023-10-16T09:57:18.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c300cbef0 0x7f1c300cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:18.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c300cbef0 0x7f1c300cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51254/0 (socket says 172.21.15.158:51254) 2023-10-16T09:57:18.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 -- 172.21.15.158:0/4205426606 learned_addr learned my addr 172.21.15.158:0/4205426606 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:18.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2ed9d700 1 --2- 172.21.15.158:0/4205426606 >> v2:172.21.15.172:3300/0 conn(0x7f1c30159280 0x7f1c3006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:18.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 -- 172.21.15.158:0/4205426606 >> v2:172.21.15.172:3300/0 conn(0x7f1c30159280 msgr2=0x7f1c3006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:57:18.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 --2- 172.21.15.158:0/4205426606 >> v2:172.21.15.172:3300/0 conn(0x7f1c30159280 0x7f1c3006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 -- 172.21.15.158:0/4205426606 >> v1:172.21.15.172:6789/0 conn(0x7f1c300d0e10 legacy=0x7f1c3015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 -- 172.21.15.158:0/4205426606 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3006b400 con 0x7f1c300cbef0 2023-10-16T09:57:18.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2f59e700 1 --2- 172.21.15.158:0/4205426606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c300cc340 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1c300cdf50 tx=0x7f1c2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a351da45985d5d4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:18.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2dd9b700 1 -- 172.21.15.158:0/4205426606 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2800e070 con 0x7f1c300cbef0 2023-10-16T09:57:18.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2dd9b700 1 -- 172.21.15.158:0/4205426606 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2800b9c0 con 0x7f1c300cbef0 2023-10-16T09:57:18.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.354+0000 7f1c2dd9b700 1 -- 172.21.15.158:0/4205426606 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2800bd70 con 0x7f1c300cbef0 2023-10-16T09:57:18.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 -- 172.21.15.158:0/4205426606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 msgr2=0x7f1c300cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 --2- 172.21.15.158:0/4205426606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c300cc340 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1c300cdf50 tx=0x7f1c2800b330 comp rx=0 tx=0).stop 2023-10-16T09:57:18.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 -- 172.21.15.158:0/4205426606 shutdown_connections 2023-10-16T09:57:18.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 --2- 172.21.15.158:0/4205426606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c300cc340 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 --2- 172.21.15.158:0/4205426606 >> v2:172.21.15.172:3300/0 conn(0x7f1c30159280 0x7f1c3006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 -- 172.21.15.158:0/4205426606 >> 172.21.15.158:0/4205426606 conn(0x7f1c30099a10 msgr2=0x7f1c300d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:18.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 -- 172.21.15.158:0/4205426606 shutdown_connections 2023-10-16T09:57:18.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.355+0000 7f1c3501f700 1 -- 172.21.15.158:0/4205426606 wait complete. 2023-10-16T09:57:18.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.356+0000 7f1c3501f700 1 Processor -- start 2023-10-16T09:57:18.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c3501f700 1 -- start start 2023-10-16T09:57:18.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c3501f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c3015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:18.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c3501f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c300d0e10 0x7f1c3015acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:18.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c3501f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c30159280 0x7f1c3015b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:18.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c3501f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3015c2b0 con 0x7f1c300cbef0 2023-10-16T09:57:18.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c3501f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3015c570 con 0x7f1c30159280 2023-10-16T09:57:18.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c3501f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3015c850 con 0x7f1c300d0e10 2023-10-16T09:57:18.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c2ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c3015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:18.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c2ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c3015eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51258/0 (socket says 172.21.15.158:51258) 2023-10-16T09:57:18.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c2ed9d700 1 -- 172.21.15.158:0/2204921262 learned_addr learned my addr 172.21.15.158:0/2204921262 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:57:18.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2e59c700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c300d0e10 0x7f1c3015acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:18.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.357+0000 7f1c2f59e700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c30159280 0x7f1c3015b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2ed9d700 1 -- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c300d0e10 msgr2=0x7f1c3015acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2ed9d700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c300d0e10 0x7f1c3015acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2ed9d700 1 -- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c30159280 msgr2=0x7f1c3015b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2ed9d700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c30159280 0x7f1c3015b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2ed9d700 1 -- 172.21.15.158:0/2204921262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28009b30 con 0x7f1c300cbef0 2023-10-16T09:57:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2e59c700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c300d0e10 0x7f1c3015acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2f59e700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c30159280 0x7f1c3015b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:57:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c2ed9d700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c3015eca0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e810 tx=0x7f1c2000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c177fe700 1 -- 172.21.15.158:0/2204921262 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20013610 con 0x7f1c300cbef0 2023-10-16T09:57:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c177fe700 1 -- 172.21.15.158:0/2204921262 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20013770 con 0x7f1c300cbef0 2023-10-16T09:57:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3015cb30 con 0x7f1c300cbef0 2023-10-16T09:57:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c177fe700 1 -- 172.21.15.158:0/2204921262 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20013b20 con 0x7f1c300cbef0 2023-10-16T09:57:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.358+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3015f1e0 con 0x7f1c300cbef0 2023-10-16T09:57:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.359+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3015bd70 con 0x7f1c300cbef0 2023-10-16T09:57:18.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.360+0000 7f1c177fe700 1 -- 172.21.15.158:0/2204921262 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 23) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1c20013c80 con 0x7f1c300cbef0 2023-10-16T09:57:18.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.360+0000 7f1c177fe700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1c1c06ada0 0x7f1c1c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:57:18.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.360+0000 7f1c177fe700 1 -- 172.21.15.158:0/2204921262 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6769+0+0 (secure 0 0 0) 0x7f1c200893f0 con 0x7f1c300cbef0 2023-10-16T09:57:18.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.360+0000 7f1c2e59c700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1c1c06ada0 0x7f1c1c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:57:18.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.360+0000 7f1c2e59c700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1c1c06ada0 0x7f1c1c06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c300c3a00 tx=0x7f1c24009040 comp rx=0 tx=0).ready entity=mgr.6429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:57:18.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.364+0000 7f1c177fe700 1 -- 172.21.15.158:0/2204921262 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c3015bd70 con 0x7f1c300cbef0 2023-10-16T09:57:18.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.583+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 --> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c3006b6e0 con 0x7f1c1c06ada0 2023-10-16T09:57:18.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.593+0000 7f1c177fe700 1 -- 172.21.15.158:0/2204921262 <== mgr.6429 v2:172.21.15.158:6832/1799865685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f1c24001440 con 0x7f1c1c06ada0 2023-10-16T09:57:18.595 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:57:18.596 INFO:teuthology.orchestra.run.smithi158.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T09:57:18.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1c1c06ada0 msgr2=0x7f1c1c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1c1c06ada0 0x7f1c1c06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c300c3a00 tx=0x7f1c24009040 comp rx=0 tx=0).stop 2023-10-16T09:57:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 msgr2=0x7f1c3015eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:57:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c3015eca0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e810 tx=0x7f1c2000ebd0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 shutdown_connections 2023-10-16T09:57:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1c1c06ada0 0x7f1c1c06d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c30159280 0x7f1c3015b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c300cbef0 0x7f1c3015eca0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 --2- 172.21.15.158:0/2204921262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c300d0e10 0x7f1c3015acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:57:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 >> 172.21.15.158:0/2204921262 conn(0x7f1c30099a10 msgr2=0x7f1c300b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:57:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 shutdown_connections 2023-10-16T09:57:18.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:57:18.595+0000 7f1c3501f700 1 -- 172.21.15.158:0/2204921262 wait complete. 2023-10-16T09:57:18.609 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T09:57:18.610 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2023-10-16T09:57:18.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:18.611 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:18.628 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:18.629 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:18.629 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:19.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:19.630 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:19.652 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:19.652 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:19.653 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:20.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:20.654 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:20.677 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:20.677 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:20.677 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:21.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:21.679 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:21.700 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:21.701 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:21.701 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:22.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:22.703 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:22.724 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:22.725 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:22.725 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:23.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:23.727 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:23.749 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:23.749 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:23.750 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:24.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:24.752 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:24.773 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:24.773 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:24.774 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:25.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:25.775 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:25.798 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:25.798 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:25.798 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:26.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:26.800 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:26.822 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:26.823 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:26.823 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:27.759 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:57:27.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:57:27.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:27.825 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:27.845 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:27.845 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:27.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:28.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:28.847 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:28.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:28.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:28.870 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:29.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:29.872 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:29.893 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:29.894 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:29.894 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:30.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:30.895 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:30.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:30.918 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:30.918 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:31.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:31.920 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:31.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:31.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:31.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:32.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:32.945 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:32.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:32.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:32.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:33.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:33.970 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:33.992 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:33.993 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:33.993 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:34.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:34.995 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:35.017 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:35.018 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:35.018 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:36.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:36.020 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:36.042 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:36.042 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:36.042 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:37.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:37.044 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:37.066 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:37.066 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:37.067 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:38.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:38.068 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:38.091 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:38.091 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:38.092 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:39.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:39.094 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:39.115 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:39.116 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:39.116 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:40.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:40.117 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:40.140 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:40.140 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:40.141 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:41.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:41.143 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:41.165 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:41.165 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:41.165 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:42.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:42.168 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:42.189 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:42.190 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:42.190 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:43.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:43.192 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:43.214 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:43.214 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:43.215 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:44.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:44.217 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:44.239 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:44.239 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:44.239 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:45.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:45.241 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:45.263 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:45.264 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:45.264 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:46.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:46.266 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:46.288 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:46.288 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:46.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:47.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:47.290 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:47.312 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:47.313 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:47.313 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:48.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:48.314 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:48.336 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:48.336 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:48.337 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:49.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:49.338 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:49.360 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:49.360 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:49.361 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:50.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:50.363 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:50.385 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:50.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:50.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:51.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:51.387 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:51.409 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:51.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:51.410 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:52.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:52.411 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:52.433 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:52.434 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:52.434 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:53.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:53.435 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:53.458 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:53.458 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:53.459 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:54.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:54.461 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:54.483 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:54.483 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:54.483 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:55.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:55.485 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:55.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:55.507 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:55.508 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:56.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:56.509 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:56.531 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:56.532 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:56.532 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:57.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:57.534 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:57.556 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:57.556 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:57.557 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:58.143 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:57:58.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:57:58.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:58.559 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:58.581 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:58.581 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:58.582 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:57:59.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:57:59.583 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:57:59.604 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T09:57:59.604 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:57:59.605 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:00.226 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:58:00.225+0000 7f2404011700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:58:00.227 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:58:00.225+0000 7f2404011700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:58:00.227 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:58:00.225+0000 7f2404011700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:58:00.227 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:58:00.225+0000 7f2404011700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:58:00.227 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:58:00.225+0000 7f2404011700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:58:00.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:00.606 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:00.626 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:00.627 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:00.628 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:00.628 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:00.628 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-09_58_00_UTC'] 2023-10-16T09:58:00.629 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697450238.2306273, now = 1697450280.6290958, timo = 42 2023-10-16T09:58:01.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:01.630 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:01.650 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:01.650 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:01.651 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:01.651 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:02.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:02.653 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:02.675 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:02.676 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:02.676 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:02.677 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:03.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:03.678 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:03.699 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:03.701 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:03.701 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:03.701 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:04.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:04.704 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:04.725 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:04.726 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:04.726 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:04.726 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:05.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:05.728 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:05.749 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:05.750 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:05.750 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:05.750 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:06.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:06.752 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:06.774 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:06.775 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:06.775 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:06.776 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:07.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:07.777 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:07.798 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:07.799 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:07.799 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:07.800 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:08.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:08.802 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:08.824 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:08.824 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:08.824 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:08.825 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:09.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:09.827 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:09.848 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:09.849 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:09.849 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:09.849 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:10.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:10.851 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:10.873 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:10.874 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:10.874 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:10.875 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:11.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:11.877 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:11.898 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:11.899 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:11.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:11.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:12.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:12.901 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:12.924 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:12.924 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:12.924 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:12.925 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:13.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:13.927 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:13.949 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:13.949 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:13.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:13.950 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:14.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:14.952 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:14.973 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:14.974 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:14.974 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:14.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:15.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:15.977 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:15.998 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:15.999 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:15.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:15.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:17.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:17.001 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:17.023 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:17.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:17.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:17.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:18.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:18.026 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:18.048 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:18.049 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:18.049 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:18.049 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:19.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:19.051 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:19.072 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:19.073 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:19.074 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:19.074 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:20.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:20.076 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:20.097 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:20.098 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:20.099 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:20.099 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:21.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:21.101 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:21.123 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:21.124 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:21.124 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:21.125 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:22.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:22.127 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:22.148 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:22.149 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:22.149 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:22.150 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:23.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:23.152 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:23.174 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:23.174 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:23.175 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:23.175 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:24.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:24.177 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:24.199 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:24.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:24.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:24.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:25.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:25.203 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:25.225 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:25.225 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:25.225 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:25.226 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:26.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:26.227 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:26.248 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:26.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:26.249 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:26.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:27.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:27.251 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:27.273 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:27.274 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:27.274 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:27.274 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:28.231 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:58:28.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:58:28.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:28.276 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:28.295 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:28.297 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:28.297 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:28.297 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:29.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:29.299 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:29.320 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:29.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:29.322 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:29.322 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:30.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:30.325 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:30.346 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:30.347 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:30.347 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:30.348 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:31.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:31.350 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:31.371 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:31.372 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:31.373 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:31.373 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:32.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:32.375 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:32.397 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:32.398 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:32.398 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:32.398 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:33.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:33.400 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:33.421 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:33.422 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:33.423 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:33.423 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:34.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:34.425 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:34.447 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:34.448 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:34.448 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:34.448 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:35.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:35.450 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:35.472 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:35.473 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:35.473 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:35.473 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:36.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:36.475 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:36.497 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:36.497 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:36.497 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:36.498 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:37.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:37.500 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:37.522 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:37.522 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:37.523 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:37.523 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:38.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:38.525 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:38.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:38.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:38.548 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:38.548 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:39.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:39.550 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:39.571 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:39.571 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:39.571 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:39.572 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:40.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:40.574 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:40.595 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:40.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:40.596 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:40.596 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:41.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:41.598 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:41.620 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:41.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:41.621 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:41.621 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:42.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:42.624 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:42.645 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:42.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:42.646 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:42.646 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:43.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:43.648 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:43.669 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:43.670 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:43.670 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:43.670 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:44.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:44.672 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:44.694 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:44.695 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:44.695 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:44.695 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:45.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:45.697 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:45.719 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:45.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:45.720 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:45.720 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:46.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:46.722 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:46.744 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:46.744 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:46.745 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:46.745 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:47.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:47.747 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:47.768 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:47.769 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:47.770 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:47.770 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:48.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:48.772 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:48.794 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:48.794 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:48.795 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:48.795 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:49.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:49.797 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:49.818 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:49.819 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:49.819 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:49.820 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:50.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:50.821 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:50.843 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:50.844 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:50.844 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:50.845 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:51.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:51.846 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:51.868 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:51.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:51.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:51.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:52.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:52.871 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:52.892 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:52.893 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:52.894 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:52.894 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:53.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:53.895 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:53.916 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:53.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:53.918 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:53.918 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:54.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:54.920 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:54.941 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:54.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:54.942 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:54.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:55.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:55.944 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:55.966 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:55.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:55.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:55.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:56.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:56.969 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:56.991 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:56.992 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:56.992 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:56.993 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:57.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:57.994 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:58.015 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:58.016 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:58.016 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:58.016 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:58:58.299 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:58:58.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:58:59.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:58:59.018 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:58:59.041 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:58:59.041 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:58:59.042 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:58:59.042 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:59:00.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:00.044 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:59:00.066 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:59:00.067 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC'} 2023-10-16T09:59:00.067 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T09:59:00.068 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:59:01.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:01.070 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:59:01.091 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:59:01.091 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_59_00_UTC 2023-10-16T09:59:01.092 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-09_58_00_UTC', 'scheduled-2023-10-16-09_59_00_UTC'} 2023-10-16T09:59:01.092 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-09_59_00_UTC'} 2023-10-16T09:59:01.093 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T09:59:01.093 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-09_59_00_UTC'] 2023-10-16T09:59:01.093 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697450238.2306273, now = 1697450341.0938323, timo = 102 2023-10-16T09:59:02.095 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T09:59:02.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.229+0000 7f481ce79700 1 Processor -- start 2023-10-16T09:59:02.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.229+0000 7f481ce79700 1 -- start start 2023-10-16T09:59:02.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.230+0000 7f481ce79700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.230+0000 7f481ce79700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f481815deb0 0x7f48180ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.230+0000 7f481ce79700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180d0ae0 con 0x7f48180cea20 2023-10-16T09:59:02.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.230+0000 7f481ce79700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48180d12d0 con 0x7f481815deb0 2023-10-16T09:59:02.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.230+0000 7f481ce79700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180cbdf0 con 0x7f481815dad0 2023-10-16T09:59:02.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f481815deb0 0x7f48180ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f481815deb0 0x7f48180ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51296/0 (socket says 172.21.15.158:51296) 2023-10-16T09:59:02.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 -- 172.21.15.158:0/3973875754 learned_addr learned my addr 172.21.15.158:0/3973875754 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:02.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f481659c700 1 --2- 172.21.15.158:0/3973875754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 -- 172.21.15.158:0/3973875754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 msgr2=0x7f48180cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:02.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 --2- 172.21.15.158:0/3973875754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 -- 172.21.15.158:0/3973875754 >> v1:172.21.15.172:6789/0 conn(0x7f48180cea20 legacy=0x7f48180c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:02.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 -- 172.21.15.158:0/3973875754 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48180cc280 con 0x7f481815deb0 2023-10-16T09:59:02.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f481659c700 1 --2- 172.21.15.158:0/3973875754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:02.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f4815d9b700 1 --2- 172.21.15.158:0/3973875754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f48180ce4e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f48080098f0 tx=0x7f4808009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6eadd8a8f44b25a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:02.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f481559a700 1 -- 172.21.15.158:0/3973875754 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808004230 con 0x7f481815deb0 2023-10-16T09:59:02.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f481559a700 1 -- 172.21.15.158:0/3973875754 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4808004390 con 0x7f481815deb0 2023-10-16T09:59:02.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.231+0000 7f481559a700 1 -- 172.21.15.158:0/3973875754 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48080036a0 con 0x7f481815deb0 2023-10-16T09:59:02.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 -- 172.21.15.158:0/3973875754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 msgr2=0x7f48180ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 --2- 172.21.15.158:0/3973875754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f48180ce4e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f48080098f0 tx=0x7f4808009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:02.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 -- 172.21.15.158:0/3973875754 shutdown_connections 2023-10-16T09:59:02.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 --2- 172.21.15.158:0/3973875754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 --2- 172.21.15.158:0/3973875754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f48180ce4e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 -- 172.21.15.158:0/3973875754 >> 172.21.15.158:0/3973875754 conn(0x7f4818099a10 msgr2=0x7f48180d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:02.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 -- 172.21.15.158:0/3973875754 shutdown_connections 2023-10-16T09:59:02.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.232+0000 7f481ce79700 1 -- 172.21.15.158:0/3973875754 wait complete. 2023-10-16T09:59:02.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.233+0000 7f481ce79700 1 Processor -- start 2023-10-16T09:59:02.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.233+0000 7f481ce79700 1 -- start start 2023-10-16T09:59:02.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f481ce79700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48180cea20 0x7f4818152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f481ce79700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f481ce79700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f4818156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f481ce79700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180c8c70 con 0x7f481815deb0 2023-10-16T09:59:02.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f481ce79700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180b4230 con 0x7f48180cea20 2023-10-16T09:59:02.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f481ce79700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180b4a90 con 0x7f481815dad0 2023-10-16T09:59:02.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f4816d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f4818156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f4816d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f4818156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51302/0 (socket says 172.21.15.158:51302) 2023-10-16T09:59:02.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.234+0000 7f4816d9d700 1 -- 172.21.15.158:0/2426915870 learned_addr learned my addr 172.21.15.158:0/2426915870 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:02.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f481659c700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48180cea20 0x7f4818152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4815d9b700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4816d9d700 1 -- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 msgr2=0x7f48180ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4816d9d700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4816d9d700 1 -- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48180cea20 msgr2=0x7f4818152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4816d9d700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48180cea20 0x7f4818152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4816d9d700 1 -- 172.21.15.158:0/2426915870 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4808009550 con 0x7f481815deb0 2023-10-16T09:59:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f481659c700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48180cea20 0x7f4818152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4816d9d700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f4818156870 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f481000cc10 tx=0x7f481000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4806ffd700 1 -- 172.21.15.158:0/2426915870 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481000abb0 con 0x7f481815deb0 2023-10-16T09:59:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4806ffd700 1 -- 172.21.15.158:0/2426915870 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481000ad10 con 0x7f481815deb0 2023-10-16T09:59:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48180c8990 con 0x7f481815deb0 2023-10-16T09:59:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f4806ffd700 1 -- 172.21.15.158:0/2426915870 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810012660 con 0x7f481815deb0 2023-10-16T09:59:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.235+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48180b4720 con 0x7f481815deb0 2023-10-16T09:59:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.236+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481806b800 con 0x7f481815deb0 2023-10-16T09:59:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.236+0000 7f4806ffd700 1 -- 172.21.15.158:0/2426915870 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 27) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f48100127c0 con 0x7f481815deb0 2023-10-16T09:59:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.237+0000 7f4806ffd700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f47f806b070 0x7f47f806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.237+0000 7f4806ffd700 1 -- 172.21.15.158:0/2426915870 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6827+0+0 (secure 0 0 0) 0x7f481008a4b0 con 0x7f481815deb0 2023-10-16T09:59:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.237+0000 7f481659c700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f47f806b070 0x7f47f806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.237+0000 7f481659c700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f47f806b070 0x7f47f806d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f480c0089b0 tx=0x7f480c008040 comp rx=0 tx=0).ready entity=mgr.6429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.241+0000 7f4806ffd700 1 -- 172.21.15.158:0/2426915870 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4810054f50 con 0x7f481815deb0 2023-10-16T09:59:02.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.460+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 --> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f48180d0ff0 con 0x7f47f806b070 2023-10-16T09:59:02.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.526+0000 7f4806ffd700 1 -- 172.21.15.158:0/2426915870 <== mgr.6429 v2:172.21.15.158:6832/1799865685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f480c00c040 con 0x7f47f806b070 2023-10-16T09:59:02.527 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path snap_test_dir1 2023-10-16T09:59:02.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f47f806b070 msgr2=0x7f47f806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f47f806b070 0x7f47f806d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f480c0089b0 tx=0x7f480c008040 comp rx=0 tx=0).stop 2023-10-16T09:59:02.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 msgr2=0x7f4818156870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f4818156870 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f481000cc10 tx=0x7f481000a060 comp rx=0 tx=0).stop 2023-10-16T09:59:02.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 shutdown_connections 2023-10-16T09:59:02.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f47f806b070 0x7f47f806d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f481815deb0 0x7f4818156870 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48180cea20 0x7f4818152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 --2- 172.21.15.158:0/2426915870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f481815dad0 0x7f48180ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 >> 172.21.15.158:0/2426915870 conn(0x7f4818099a10 msgr2=0x7f4818154130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:02.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 shutdown_connections 2023-10-16T09:59:02.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.528+0000 7f481ce79700 1 -- 172.21.15.158:0/2426915870 wait complete. 2023-10-16T09:59:02.541 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T09:59:02.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:02.542 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T09:59:02.560 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:59:02.560 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-09_59_00_UTC 2023-10-16T09:59:02.560 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-09_58_00_UTC 2023-10-16T09:59:02.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:02.561 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-09_58_00_UTC) 2023-10-16T09:59:02.622 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-09_59_00_UTC 2023-10-16T09:59:02.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:02.622 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-09_59_00_UTC) 2023-10-16T09:59:02.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:02.684 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T09:59:02.744 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T09:59:02.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.905+0000 7fc6e4049700 1 Processor -- start 2023-10-16T09:59:02.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.905+0000 7fc6e4049700 1 -- start start 2023-10-16T09:59:02.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e4049700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc6dc15b740 0x7fc6dc0ca260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e4049700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc0b46f0 0x7fc6dc0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e4049700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc15bb20 con 0x7fc6dc06dde0 2023-10-16T09:59:02.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e4049700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc0727a0 con 0x7fc6dc15b740 2023-10-16T09:59:02.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e4049700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0724c0 con 0x7fc6dc0b46f0 2023-10-16T09:59:02.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc6dc15b740 0x7fc6dc0ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc6dc15b740 0x7fc6dc0ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51308/0 (socket says 172.21.15.158:51308) 2023-10-16T09:59:02.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 -- 172.21.15.158:0/3536838284 learned_addr learned my addr 172.21.15.158:0/3536838284 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:02.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e25e6700 1 --2- 172.21.15.158:0/3536838284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc0b46f0 0x7fc6dc0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 -- 172.21.15.158:0/3536838284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc0b46f0 msgr2=0x7fc6dc0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:02.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 --2- 172.21.15.158:0/3536838284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc0b46f0 0x7fc6dc0ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 -- 172.21.15.158:0/3536838284 >> v1:172.21.15.172:6789/0 conn(0x7fc6dc06dde0 legacy=0x7fc6dc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:02.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 -- 172.21.15.158:0/3536838284 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0b2c00 con 0x7fc6dc15b740 2023-10-16T09:59:02.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e1de5700 1 --2- 172.21.15.158:0/3536838284 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc0ca260 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800dd10 tx=0x7fc6d800f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1480402b8deb330 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:02.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.906+0000 7fc6e0de3700 1 -- 172.21.15.158:0/3536838284 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8012070 con 0x7fc6dc15b740 2023-10-16T09:59:02.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e0de3700 1 -- 172.21.15.158:0/3536838284 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d800d2e0 con 0x7fc6dc15b740 2023-10-16T09:59:02.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e0de3700 1 -- 172.21.15.158:0/3536838284 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d800fb80 con 0x7fc6dc15b740 2023-10-16T09:59:02.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 -- 172.21.15.158:0/3536838284 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 msgr2=0x7fc6dc0ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 --2- 172.21.15.158:0/3536838284 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc0ca260 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800dd10 tx=0x7fc6d800f330 comp rx=0 tx=0).stop 2023-10-16T09:59:02.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 -- 172.21.15.158:0/3536838284 shutdown_connections 2023-10-16T09:59:02.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 --2- 172.21.15.158:0/3536838284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc0b46f0 0x7fc6dc0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 --2- 172.21.15.158:0/3536838284 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc0ca260 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 -- 172.21.15.158:0/3536838284 >> 172.21.15.158:0/3536838284 conn(0x7fc6dc0cc690 msgr2=0x7fc6dc0cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:02.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 -- 172.21.15.158:0/3536838284 shutdown_connections 2023-10-16T09:59:02.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.907+0000 7fc6e4049700 1 -- 172.21.15.158:0/3536838284 wait complete. 2023-10-16T09:59:02.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.908+0000 7fc6e4049700 1 Processor -- start 2023-10-16T09:59:02.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.908+0000 7fc6e4049700 1 -- start start 2023-10-16T09:59:02.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e4049700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc06dde0 0x7fc6dc153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e4049700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6dc0b46f0 0x7fc6dc150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e4049700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc1edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e4049700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1522d0 con 0x7fc6dc15b740 2023-10-16T09:59:02.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e4049700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc151650 con 0x7fc6dc0b46f0 2023-10-16T09:59:02.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e4049700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc151930 con 0x7fc6dc06dde0 2023-10-16T09:59:02.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e25e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc1edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e25e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc1edee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51318/0 (socket says 172.21.15.158:51318) 2023-10-16T09:59:02.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e25e6700 1 -- 172.21.15.158:0/304890019 learned_addr learned my addr 172.21.15.158:0/304890019 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:02.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e1de5700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc06dde0 0x7fc6dc153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.909+0000 7fc6e15e4700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6dc0b46f0 0x7fc6dc150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e25e6700 1 -- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc06dde0 msgr2=0x7fc6dc153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e25e6700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc06dde0 0x7fc6dc153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e25e6700 1 -- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6dc0b46f0 msgr2=0x7fc6dc150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:02.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e25e6700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6dc0b46f0 0x7fc6dc150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:02.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e25e6700 1 -- 172.21.15.158:0/304890019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d800d9f0 con 0x7fc6dc15b740 2023-10-16T09:59:02.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e25e6700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc1edee0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400cd60 tx=0x7fc6d400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:02.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4003a20 con 0x7fc6dc15b740 2023-10-16T09:59:02.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4003ba0 con 0x7fc6dc15b740 2023-10-16T09:59:02.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc151c10 con 0x7fc6dc15b740 2023-10-16T09:59:02.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4004020 con 0x7fc6dc15b740 2023-10-16T09:59:02.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.910+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc1eed00 con 0x7fc6dc15b740 2023-10-16T09:59:02.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.911+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc06c280 con 0x7fc6dc15b740 2023-10-16T09:59:02.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.911+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 27) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fc6d4004180 con 0x7fc6dc15b740 2023-10-16T09:59:02.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.912+0000 7fc6d27fc700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fc6c006b070 0x7fc6c006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:02.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.912+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6827+0+0 (secure 0 0 0) 0x7fc6d4012070 con 0x7fc6dc15b740 2023-10-16T09:59:02.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.912+0000 7fc6e1de5700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fc6c006b070 0x7fc6c006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:02.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.912+0000 7fc6e1de5700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fc6c006b070 0x7fc6c006d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8019040 tx=0x7fc6d800d000 comp rx=0 tx=0).ready entity=mgr.6429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:02.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:02.916+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc6d4054db0 con 0x7fc6dc15b740 2023-10-16T09:59:03.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.164+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc6dc152410 con 0x7fc6dc15b740 2023-10-16T09:59:03.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.616+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v28) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc6dc152410 con 0x7fc6dc15b740 2023-10-16T09:59:03.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.622+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fc6c006b070 msgr2=0x7fc6c006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:03.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.622+0000 7fc6e4049700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fc6c006b070 0x7fc6c006d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8019040 tx=0x7fc6d800d000 comp rx=0 tx=0).stop 2023-10-16T09:59:03.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.622+0000 7fc6d27fc700 1 -- 172.21.15.158:0/304890019 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fc6d408a0c0 con 0x7fc6dc15b740 2023-10-16T09:59:03.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.622+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 msgr2=0x7fc6dc1edee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:03.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.622+0000 7fc6e4049700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc1edee0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400cd60 tx=0x7fc6d400a570 comp rx=0 tx=0).stop 2023-10-16T09:59:03.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.624+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 shutdown_connections 2023-10-16T09:59:03.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.624+0000 7fc6e4049700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fc6c006b070 0x7fc6c006d530 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8019040 tx=0x7fc6d800d000 comp rx=0 tx=0).stop 2023-10-16T09:59:03.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.624+0000 7fc6e4049700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6dc15b740 0x7fc6dc1edee0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.624+0000 7fc6e4049700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6dc06dde0 0x7fc6dc153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.624+0000 7fc6e4049700 1 --2- 172.21.15.158:0/304890019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6dc0b46f0 0x7fc6dc150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.624+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 >> 172.21.15.158:0/304890019 conn(0x7fc6dc0cc690 msgr2=0x7fc6dc0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:03.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.624+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 shutdown_connections 2023-10-16T09:59:03.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.625+0000 7fc6e4049700 1 -- 172.21.15.158:0/304890019 wait complete. 2023-10-16T09:59:03.642 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T09:59:03.643 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:03.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:03.727 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T09:59:03.727 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:03.728 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:03.746 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:03.746 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:03.773 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:03.771+0000 7f194ef5b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:59:03.774 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:03.773+0000 7f68d0037200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:59:03.791 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:03.840 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:03.838+0000 7f194ef5b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:03.843 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:03.842+0000 7f68d0037200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:03.913 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:03.912+0000 7f194ef5b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:59:03.922 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:03.921+0000 7f68d0037200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:59:03.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.957+0000 7f29d0bdf700 1 Processor -- start 2023-10-16T09:59:03.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.957+0000 7f29d0bdf700 1 -- start start 2023-10-16T09:59:03.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29d0bdf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f29cc06f470 0x7f29cc06e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:03.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29d0bdf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f29cc06e810 0x7f29cc072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:03.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29d0bdf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29cc06f850 con 0x7f29cc0728a0 2023-10-16T09:59:03.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29d0bdf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29cc072cb0 con 0x7f29cc06e810 2023-10-16T09:59:03.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29d0bdf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29cc0b4260 con 0x7f29cc06f470 2023-10-16T09:59:03.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f29cc06e810 0x7f29cc072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:03.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f29cc06e810 0x7f29cc072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51384/0 (socket says 172.21.15.158:51384) 2023-10-16T09:59:03.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 -- 172.21.15.158:0/3736717892 learned_addr learned my addr 172.21.15.158:0/3736717892 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:03.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29ca59c700 1 --2- 172.21.15.158:0/3736717892 >> v2:172.21.15.172:3300/0 conn(0x7f29cc06f470 0x7f29cc06e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:03.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 -- 172.21.15.158:0/3736717892 >> v2:172.21.15.172:3300/0 conn(0x7f29cc06f470 msgr2=0x7f29cc06e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:03.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 --2- 172.21.15.158:0/3736717892 >> v2:172.21.15.172:3300/0 conn(0x7f29cc06f470 0x7f29cc06e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 -- 172.21.15.158:0/3736717892 >> v1:172.21.15.158:6789/0 conn(0x7f29cc0728a0 legacy=0x7f29cc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:03.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 -- 172.21.15.158:0/3736717892 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc0b4540 con 0x7f29cc06e810 2023-10-16T09:59:03.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.958+0000 7f29c9d9b700 1 --2- 172.21.15.158:0/3736717892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc06e810 0x7f29cc072360 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0737c0 tx=0x7f29c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819b0307c95e0727 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:03.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29c959a700 1 -- 172.21.15.158:0/3736717892 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0011830 con 0x7f29cc06e810 2023-10-16T09:59:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29c959a700 1 -- 172.21.15.158:0/3736717892 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c0011990 con 0x7f29cc06e810 2023-10-16T09:59:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29c959a700 1 -- 172.21.15.158:0/3736717892 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0011d20 con 0x7f29cc06e810 2023-10-16T09:59:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3736717892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc06e810 msgr2=0x7f29cc072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3736717892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc06e810 0x7f29cc072360 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0737c0 tx=0x7f29c0009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3736717892 shutdown_connections 2023-10-16T09:59:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3736717892 >> v2:172.21.15.172:3300/0 conn(0x7f29cc06f470 0x7f29cc06e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3736717892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc06e810 0x7f29cc072360 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3736717892 >> 172.21.15.158:0/3736717892 conn(0x7f29cc0ccda0 msgr2=0x7f29cc0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3736717892 shutdown_connections 2023-10-16T09:59:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.959+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3736717892 wait complete. 2023-10-16T09:59:03.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 Processor -- start 2023-10-16T09:59:03.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 -- start start 2023-10-16T09:59:03.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29cc06e810 0x7f29cc1f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:03.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29cc06f470 0x7f29cc1f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:03.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc0728a0 0x7f29cc0d0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:03.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc0cded0 con 0x7f29cc0728a0 2023-10-16T09:59:03.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc0ce0e0 con 0x7f29cc06f470 2023-10-16T09:59:03.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.961+0000 7f29d0bdf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc0ce3c0 con 0x7f29cc06e810 2023-10-16T09:59:03.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc0728a0 0x7f29cc0d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:03.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc0728a0 0x7f29cc0d0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51392/0 (socket says 172.21.15.158:51392) 2023-10-16T09:59:03.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 -- 172.21.15.158:0/3108199369 learned_addr learned my addr 172.21.15.158:0/3108199369 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:03.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29c9d9b700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29cc06f470 0x7f29cc1f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29ca59c700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29cc06e810 0x7f29cc1f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 -- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29cc06e810 msgr2=0x7f29cc1f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29cc06e810 0x7f29cc1f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 -- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29cc06f470 msgr2=0x7f29cc1f36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29cc06f470 0x7f29cc1f36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 -- 172.21.15.158:0/3108199369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c0009550 con 0x7f29cc0728a0 2023-10-16T09:59:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29c9d9b700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29cc06f470 0x7f29cc1f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29cad9d700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc0728a0 0x7f29cc0d0250 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f29c400cb50 tx=0x7f29c400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29baffd700 1 -- 172.21.15.158:0/3108199369 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4003c00 con 0x7f29cc0728a0 2023-10-16T09:59:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29baffd700 1 -- 172.21.15.158:0/3108199369 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4003d60 con 0x7f29cc0728a0 2023-10-16T09:59:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29baffd700 1 -- 172.21.15.158:0/3108199369 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c400aa20 con 0x7f29cc0728a0 2023-10-16T09:59:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc0ce6a0 con 0x7f29cc0728a0 2023-10-16T09:59:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.962+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc1f4640 con 0x7f29cc0728a0 2023-10-16T09:59:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.963+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc1f4800 con 0x7f29cc0728a0 2023-10-16T09:59:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.964+0000 7f29baffd700 1 -- 172.21.15.158:0/3108199369 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f29c400ab80 con 0x7f29cc0728a0 2023-10-16T09:59:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.965+0000 7f29baffd700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 0x7f29ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.965+0000 7f29ca59c700 1 -- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 msgr2=0x7f29ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.965+0000 7f29baffd700 1 -- 172.21.15.158:0/3108199369 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f29c4089b80 con 0x7f29cc0728a0 2023-10-16T09:59:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.965+0000 7f29ca59c700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 0x7f29ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:03.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:03.973+0000 7f29baffd700 1 -- 172.21.15.158:0/3108199369 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29c40546f0 con 0x7f29cc0728a0 2023-10-16T09:59:03.991 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:03.990+0000 7f194ef5b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:59:04.010 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:04.009+0000 7f68d0037200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:59:04.131 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:04.130+0000 7f194ef5b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:59:04.154 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:04.153+0000 7f68d0037200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:59:04.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.166+0000 7f29ca59c700 1 -- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 msgr2=0x7f29ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:04.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.166+0000 7f29ca59c700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 0x7f29ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:04.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.172+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29cc071e20 con 0x7f29cc0728a0 2023-10-16T09:59:04.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.174+0000 7f29baffd700 1 -- 172.21.15.158:0/3108199369 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+18144 (secure 0 0 0) 0x7f29cc071e20 con 0x7f29cc0728a0 2023-10-16T09:59:04.176 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:04.176 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":71,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:03.606967+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:57:01.416542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T09:57:01.962552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.28","mappings":[{"from":0,"to":6}]},{"pgid":"8.3d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3405184500":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/488409315":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2953589465":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2437477336":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6833/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6832/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2185855121":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3427024404":"2023-10-17T09:57:16.707111+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:04.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.176+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 msgr2=0x7f29ac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:04.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.176+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 0x7f29ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.176+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc0728a0 msgr2=0x7f29cc0d0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.176+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc0728a0 0x7f29cc0d0250 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f29c400cb50 tx=0x7f29c400ce60 comp rx=0 tx=0).stop 2023-10-16T09:59:04.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 shutdown_connections 2023-10-16T09:59:04.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f29ac06b020 0x7f29ac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29cc0728a0 0x7f29cc0d0250 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29cc06e810 0x7f29cc1f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 --2- 172.21.15.158:0/3108199369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29cc06f470 0x7f29cc1f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 >> 172.21.15.158:0/3108199369 conn(0x7f29cc0ccda0 msgr2=0x7f29cc0b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:04.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 shutdown_connections 2023-10-16T09:59:04.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.177+0000 7f29d0bdf700 1 -- 172.21.15.158:0/3108199369 wait complete. 2023-10-16T09:59:04.191 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:57:01.416542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:57:01.962552+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:04.192 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:04.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.355+0000 7faa0a6ea700 1 Processor -- start 2023-10-16T09:59:04.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.355+0000 7faa0a6ea700 1 -- start start 2023-10-16T09:59:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.355+0000 7faa0a6ea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faa0415d6c0 0x7faa040cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.355+0000 7faa0a6ea700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa040cf360 0x7faa040d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.355+0000 7faa0a6ea700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa040d1ee0 con 0x7faa0415daa0 2023-10-16T09:59:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.355+0000 7faa0a6ea700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa040d40a0 con 0x7faa040cf360 2023-10-16T09:59:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.355+0000 7faa0a6ea700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa040d12d0 con 0x7faa0415d6c0 2023-10-16T09:59:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa040cf360 0x7faa040d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa040cf360 0x7faa040d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51446/0 (socket says 172.21.15.158:51446) 2023-10-16T09:59:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 -- 172.21.15.158:0/1182191943 learned_addr learned my addr 172.21.15.158:0/1182191943 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa037fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7faa0415daa0 0x7faa040cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51618/0 (socket says 172.21.15.158:51618) 2023-10-16T09:59:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa03fff700 1 --2- 172.21.15.158:0/1182191943 >> v2:172.21.15.172:3300/0 conn(0x7faa0415d6c0 0x7faa040cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 -- 172.21.15.158:0/1182191943 >> v2:172.21.15.172:3300/0 conn(0x7faa0415d6c0 msgr2=0x7faa040cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 --2- 172.21.15.158:0/1182191943 >> v2:172.21.15.172:3300/0 conn(0x7faa0415d6c0 0x7faa040cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 -- 172.21.15.158:0/1182191943 >> v1:172.21.15.158:6789/0 conn(0x7faa0415daa0 legacy=0x7faa040cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 -- 172.21.15.158:0/1182191943 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa040d15b0 con 0x7faa040cf360 2023-10-16T09:59:04.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa08c87700 1 --2- 172.21.15.158:0/1182191943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040d0470 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faa040674f0 tx=0x7fa9fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea30bcbca2b2e0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:04.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa02ffd700 1 -- 172.21.15.158:0/1182191943 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc00e070 con 0x7faa040cf360 2023-10-16T09:59:04.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa02ffd700 1 -- 172.21.15.158:0/1182191943 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc00b910 con 0x7faa040cf360 2023-10-16T09:59:04.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.356+0000 7faa02ffd700 1 -- 172.21.15.158:0/1182191943 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc00bcc0 con 0x7faa040cf360 2023-10-16T09:59:04.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 -- 172.21.15.158:0/1182191943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 msgr2=0x7faa040d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/1182191943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040d0470 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faa040674f0 tx=0x7fa9fc00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:04.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 -- 172.21.15.158:0/1182191943 shutdown_connections 2023-10-16T09:59:04.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/1182191943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040d0470 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/1182191943 >> v2:172.21.15.172:3300/0 conn(0x7faa0415d6c0 0x7faa040cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 -- 172.21.15.158:0/1182191943 >> 172.21.15.158:0/1182191943 conn(0x7faa0409c940 msgr2=0x7faa040d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:04.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 -- 172.21.15.158:0/1182191943 shutdown_connections 2023-10-16T09:59:04.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.357+0000 7faa0a6ea700 1 -- 172.21.15.158:0/1182191943 wait complete. 2023-10-16T09:59:04.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.358+0000 7faa0a6ea700 1 Processor -- start 2023-10-16T09:59:04.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.358+0000 7faa0a6ea700 1 -- start start 2023-10-16T09:59:04.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa0a6ea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa0a6ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa0415d6c0 0x7faa040ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa0a6ea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa0415daa0 0x7faa040ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa0a6ea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa040cd790 con 0x7faa040cf360 2023-10-16T09:59:04.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa0a6ea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa040cd1d0 con 0x7faa0415daa0 2023-10-16T09:59:04.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa0a6ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa040cd4b0 con 0x7faa0415d6c0 2023-10-16T09:59:04.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa03fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa03fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51448/0 (socket says 172.21.15.158:51448) 2023-10-16T09:59:04.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa03fff700 1 -- 172.21.15.158:0/64998724 learned_addr learned my addr 172.21.15.158:0/64998724 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:04.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa08c87700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa0415daa0 0x7faa040ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.359+0000 7faa037fe700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa0415d6c0 0x7faa040ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa03fff700 1 -- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa0415d6c0 msgr2=0x7faa040ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa03fff700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa0415d6c0 0x7faa040ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa03fff700 1 -- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa0415daa0 msgr2=0x7faa040ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa03fff700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa0415daa0 0x7faa040ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa03fff700 1 -- 172.21.15.158:0/64998724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc009b30 con 0x7faa040cf360 2023-10-16T09:59:04.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa08c87700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa0415daa0 0x7faa040ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:04.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa03fff700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040cc650 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e9b0 tx=0x7fa9f400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:04.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa00ff9700 1 -- 172.21.15.158:0/64998724 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4009710 con 0x7faa040cf360 2023-10-16T09:59:04.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa00ff9700 1 -- 172.21.15.158:0/64998724 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f400ee50 con 0x7faa040cf360 2023-10-16T09:59:04.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa00ff9700 1 -- 172.21.15.158:0/64998724 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f400ca50 con 0x7faa040cf360 2023-10-16T09:59:04.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0415f0a0 con 0x7faa040cf360 2023-10-16T09:59:04.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.360+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa040b3f70 con 0x7faa040cf360 2023-10-16T09:59:04.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.361+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0415f410 con 0x7faa040cf360 2023-10-16T09:59:04.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.362+0000 7faa00ff9700 1 -- 172.21.15.158:0/64998724 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa9f400cbb0 con 0x7faa040cf360 2023-10-16T09:59:04.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.362+0000 7faa00ff9700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 0x7fa9f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.362+0000 7faa037fe700 1 -- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 msgr2=0x7fa9f006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:04.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.362+0000 7faa00ff9700 1 -- 172.21.15.158:0/64998724 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6915+0+0 (secure 0 0 0) 0x7fa9f408a300 con 0x7faa040cf360 2023-10-16T09:59:04.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.362+0000 7faa037fe700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 0x7fa9f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:04.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.367+0000 7faa00ff9700 1 -- 172.21.15.158:0/64998724 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa9f4054ec0 con 0x7faa040cf360 2023-10-16T09:59:04.454 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:04.453+0000 7f194ef5b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:59:04.496 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:04.495+0000 7f68d0037200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:59:04.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.563+0000 7faa037fe700 1 -- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 msgr2=0x7fa9f006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:04.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.563+0000 7faa037fe700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 0x7fa9f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:04.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.566+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa0415ef00 con 0x7faa040cf360 2023-10-16T09:59:04.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.566+0000 7faa00ff9700 1 -- 172.21.15.158:0/64998724 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa0415ef00 con 0x7faa040cf360 2023-10-16T09:59:04.568 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:04.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 msgr2=0x7fa9f006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:04.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 0x7fa9f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 msgr2=0x7faa040cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040cc650 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e9b0 tx=0x7fa9f400ecc0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 shutdown_connections 2023-10-16T09:59:04.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa9f006b020 0x7fa9f006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa0415daa0 0x7faa040ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa040cf360 0x7faa040cc650 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 --2- 172.21.15.158:0/64998724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa0415d6c0 0x7faa040ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 >> 172.21.15.158:0/64998724 conn(0x7faa0409c940 msgr2=0x7faa040b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:04.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 shutdown_connections 2023-10-16T09:59:04.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.569+0000 7faa0a6ea700 1 -- 172.21.15.158:0/64998724 wait complete. 2023-10-16T09:59:04.573 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:04.570+0000 7f194ef5b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:59:04.583 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:59:04.618 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:04.617+0000 7f68d0037200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:59:04.646 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:04.645+0000 7f194ef5b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:59:04.699 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:04.698+0000 7f68d0037200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:59:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 Processor -- start 2023-10-16T09:59:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 -- start start 2023-10-16T09:59:04.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa21c15b6b0 0x7fa21c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa21c0d0c60 0x7fa21c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c0b3460 0x7fa21c070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c15ba90 con 0x7fa21c0d0c60 2023-10-16T09:59:04.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c0b38f0 con 0x7fa21c15b6b0 2023-10-16T09:59:04.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.715+0000 7fa22252e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0b4420 con 0x7fa21c0b3460 2023-10-16T09:59:04.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa21c0d0c60 0x7fa21c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa21c0d0c60 0x7fa21c0b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51460/0 (socket says 172.21.15.158:51460) 2023-10-16T09:59:04.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 -- 172.21.15.158:0/1003522428 learned_addr learned my addr 172.21.15.158:0/1003522428 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:04.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21bfff700 1 --2- 172.21.15.158:0/1003522428 >> v2:172.21.15.172:3300/0 conn(0x7fa21c15b6b0 0x7fa21c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa220acb700 1 --2- 172.21.15.158:0/1003522428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c0b3460 0x7fa21c070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 -- 172.21.15.158:0/1003522428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c0b3460 msgr2=0x7fa21c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 --2- 172.21.15.158:0/1003522428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c0b3460 0x7fa21c070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 -- 172.21.15.158:0/1003522428 >> v2:172.21.15.172:3300/0 conn(0x7fa21c15b6b0 msgr2=0x7fa21c0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 --2- 172.21.15.158:0/1003522428 >> v2:172.21.15.172:3300/0 conn(0x7fa21c15b6b0 0x7fa21c0cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 -- 172.21.15.158:0/1003522428 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0cbc20 con 0x7fa21c0d0c60 2023-10-16T09:59:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21bfff700 1 --2- 172.21.15.158:0/1003522428 >> v2:172.21.15.172:3300/0 conn(0x7fa21c15b6b0 0x7fa21c0cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21b7fe700 1 --2- 172.21.15.158:0/1003522428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c0b2f20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa21c06ffb0 tx=0x7fa210009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87bbc2d85136e42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21affd700 1 -- 172.21.15.158:0/1003522428 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21000b870 con 0x7fa21c0d0c60 2023-10-16T09:59:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21affd700 1 -- 172.21.15.158:0/1003522428 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21000b9d0 con 0x7fa21c0d0c60 2023-10-16T09:59:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.716+0000 7fa21affd700 1 -- 172.21.15.158:0/1003522428 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21000bd60 con 0x7fa21c0d0c60 2023-10-16T09:59:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 -- 172.21.15.158:0/1003522428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 msgr2=0x7fa21c0b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 --2- 172.21.15.158:0/1003522428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c0b2f20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa21c06ffb0 tx=0x7fa210009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 -- 172.21.15.158:0/1003522428 shutdown_connections 2023-10-16T09:59:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 --2- 172.21.15.158:0/1003522428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c0b3460 0x7fa21c070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 --2- 172.21.15.158:0/1003522428 >> v2:172.21.15.172:3300/0 conn(0x7fa21c15b6b0 0x7fa21c0cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 --2- 172.21.15.158:0/1003522428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c0b2f20 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 -- 172.21.15.158:0/1003522428 >> 172.21.15.158:0/1003522428 conn(0x7fa21c0cdf20 msgr2=0x7fa21c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 -- 172.21.15.158:0/1003522428 shutdown_connections 2023-10-16T09:59:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.717+0000 7fa22252e700 1 -- 172.21.15.158:0/1003522428 wait complete. 2023-10-16T09:59:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.718+0000 7fa22252e700 1 Processor -- start 2023-10-16T09:59:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.718+0000 7fa22252e700 1 -- start start 2023-10-16T09:59:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa22252e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa21c0b3460 0x7fa21c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa22252e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa22252e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c15b6b0 0x7fa21c0cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa22252e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c06c920 con 0x7fa21c0d0c60 2023-10-16T09:59:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa22252e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c06bca0 con 0x7fa21c0b3460 2023-10-16T09:59:04.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa22252e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c06bf80 con 0x7fa21c15b6b0 2023-10-16T09:59:04.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c1578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51466/0 (socket says 172.21.15.158:51466) 2023-10-16T09:59:04.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 -- 172.21.15.158:0/37458832 learned_addr learned my addr 172.21.15.158:0/37458832 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:04.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21bfff700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa21c0b3460 0x7fa21c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa220acb700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c15b6b0 0x7fa21c0cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:04.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 -- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c15b6b0 msgr2=0x7fa21c0cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c15b6b0 0x7fa21c0cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 -- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa21c0b3460 msgr2=0x7fa21c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa21c0b3460 0x7fa21c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.719+0000 7fa21b7fe700 1 -- 172.21.15.158:0/37458832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa210009550 con 0x7fa21c0d0c60 2023-10-16T09:59:04.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.720+0000 7fa21b7fe700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c1578a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa210000c00 tx=0x7fa210004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:04.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.720+0000 7fa218ff9700 1 -- 172.21.15.158:0/37458832 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210004310 con 0x7fa21c0d0c60 2023-10-16T09:59:04.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.720+0000 7fa218ff9700 1 -- 172.21.15.158:0/37458832 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21001e070 con 0x7fa21c0d0c60 2023-10-16T09:59:04.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.720+0000 7fa218ff9700 1 -- 172.21.15.158:0/37458832 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2100036a0 con 0x7fa21c0d0c60 2023-10-16T09:59:04.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.720+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c06c260 con 0x7fa21c0d0c60 2023-10-16T09:59:04.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.720+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c157de0 con 0x7fa21c0d0c60 2023-10-16T09:59:04.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.721+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21c06ca60 con 0x7fa21c0d0c60 2023-10-16T09:59:04.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.722+0000 7fa218ff9700 1 -- 172.21.15.158:0/37458832 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa210020030 con 0x7fa21c0d0c60 2023-10-16T09:59:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.722+0000 7fa218ff9700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 0x7fa20806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.722+0000 7fa21bfff700 1 -- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 msgr2=0x7fa20806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.722+0000 7fa218ff9700 1 -- 172.21.15.158:0/37458832 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fa21008ad80 con 0x7fa21c0d0c60 2023-10-16T09:59:04.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.722+0000 7fa21bfff700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 0x7fa20806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:04.736 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:04.726+0000 7f194ef5b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:59:04.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.730+0000 7fa218ff9700 1 -- 172.21.15.158:0/37458832 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa210055900 con 0x7fa21c0d0c60 2023-10-16T09:59:04.787 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:04.786+0000 7f68d0037200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:59:04.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.923+0000 7fa21bfff700 1 -- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 msgr2=0x7fa20806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:04.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.923+0000 7fa21bfff700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 0x7fa20806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:04.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.928+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa21c06ca60 con 0x7fa21c0d0c60 2023-10-16T09:59:04.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.929+0000 7fa218ff9700 1 -- 172.21.15.158:0/37458832 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa21c06ca60 con 0x7fa21c0d0c60 2023-10-16T09:59:04.930 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:59:04.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 msgr2=0x7fa20806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:04.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 0x7fa20806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 msgr2=0x7fa21c1578a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:04.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c1578a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa210000c00 tx=0x7fa210004750 comp rx=0 tx=0).stop 2023-10-16T09:59:04.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 shutdown_connections 2023-10-16T09:59:04.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fa20806b020 0x7fa20806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa21c15b6b0 0x7fa21c0cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa21c0b3460 0x7fa21c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 --2- 172.21.15.158:0/37458832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa21c0d0c60 0x7fa21c1578a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:04.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 >> 172.21.15.158:0/37458832 conn(0x7fa21c0cdf20 msgr2=0x7fa21c0b52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:04.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 shutdown_connections 2023-10-16T09:59:04.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:04.931+0000 7fa22252e700 1 -- 172.21.15.158:0/37458832 wait complete. 2023-10-16T09:59:04.945 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:59:05.095 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:05.093+0000 7f194ef5b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:59:05.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.110+0000 7f9206ab5700 1 Processor -- start 2023-10-16T09:59:05.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.111+0000 7f9206ab5700 1 -- start start 2023-10-16T09:59:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.111+0000 7f9206ab5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f920015d990 0x7f920006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.111+0000 7f9206ab5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9200070300 0x7f9200161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.111+0000 7f9206ab5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920015dd70 con 0x7f9200162010 2023-10-16T09:59:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.111+0000 7f9206ab5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92001616b0 con 0x7f9200070300 2023-10-16T09:59:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.111+0000 7f9206ab5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92000ced70 con 0x7f920015d990 2023-10-16T09:59:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.111+0000 7f91fffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9200070300 0x7f9200161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91fffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9200070300 0x7f9200161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51530/0 (socket says 172.21.15.158:51530) 2023-10-16T09:59:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91fffff700 1 -- 172.21.15.158:0/4196940039 learned_addr learned my addr 172.21.15.158:0/4196940039 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f9204851700 1 --2- 172.21.15.158:0/4196940039 >> v2:172.21.15.172:3300/0 conn(0x7f920015d990 0x7f920006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91fffff700 1 -- 172.21.15.158:0/4196940039 >> v2:172.21.15.172:3300/0 conn(0x7f920015d990 msgr2=0x7f920006d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91fffff700 1 --2- 172.21.15.158:0/4196940039 >> v2:172.21.15.172:3300/0 conn(0x7f920015d990 0x7f920006d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91fffff700 1 -- 172.21.15.158:0/4196940039 >> v1:172.21.15.172:6789/0 conn(0x7f9200162010 legacy=0x7f92000cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91fffff700 1 -- 172.21.15.158:0/4196940039 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9200070780 con 0x7f9200070300 2023-10-16T09:59:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f9204851700 1 --2- 172.21.15.158:0/4196940039 >> v2:172.21.15.172:3300/0 conn(0x7f920015d990 0x7f920006d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:05.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91fffff700 1 --2- 172.21.15.158:0/4196940039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9200070300 0x7f9200161ad0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f92000d25a0 tx=0x7f91ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb3aed7bc698505 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:05.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91ff7fe700 1 -- 172.21.15.158:0/4196940039 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec00b860 con 0x7f9200070300 2023-10-16T09:59:05.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91ff7fe700 1 -- 172.21.15.158:0/4196940039 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ec00b9c0 con 0x7f9200070300 2023-10-16T09:59:05.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.112+0000 7f91ff7fe700 1 -- 172.21.15.158:0/4196940039 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec00bd50 con 0x7f9200070300 2023-10-16T09:59:05.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 -- 172.21.15.158:0/4196940039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9200070300 msgr2=0x7f9200161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 --2- 172.21.15.158:0/4196940039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9200070300 0x7f9200161ad0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f92000d25a0 tx=0x7f91ec009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:05.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 -- 172.21.15.158:0/4196940039 shutdown_connections 2023-10-16T09:59:05.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 --2- 172.21.15.158:0/4196940039 >> v2:172.21.15.172:3300/0 conn(0x7f920015d990 0x7f920006d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 --2- 172.21.15.158:0/4196940039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9200070300 0x7f9200161ad0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 -- 172.21.15.158:0/4196940039 >> 172.21.15.158:0/4196940039 conn(0x7f92000cdf20 msgr2=0x7f92000ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:05.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 -- 172.21.15.158:0/4196940039 shutdown_connections 2023-10-16T09:59:05.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.113+0000 7f9206ab5700 1 -- 172.21.15.158:0/4196940039 wait complete. 2023-10-16T09:59:05.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.114+0000 7f9206ab5700 1 Processor -- start 2023-10-16T09:59:05.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.114+0000 7f9206ab5700 1 -- start start 2023-10-16T09:59:05.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9206ab5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9200070300 0x7f92000c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9206ab5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f920015d990 0x7f920015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9206ab5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9200162010 0x7f920015c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9206ab5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92000ac890 con 0x7f920015d990 2023-10-16T09:59:05.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9206ab5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92001f7230 con 0x7f9200162010 2023-10-16T09:59:05.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9206ab5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92000acaa0 con 0x7f9200070300 2023-10-16T09:59:05.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f920015d990 0x7f920015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f920015d990 0x7f920015bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51538/0 (socket says 172.21.15.158:51538) 2023-10-16T09:59:05.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 -- 172.21.15.158:0/241791762 learned_addr learned my addr 172.21.15.158:0/241791762 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:05.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9205052700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9200162010 0x7f920015c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f9204851700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9200070300 0x7f92000c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 -- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9200070300 msgr2=0x7f92000c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9200070300 0x7f92000c72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 -- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9200162010 msgr2=0x7f920015c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9200162010 0x7f920015c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.115+0000 7f91fffff700 1 -- 172.21.15.158:0/241791762 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec009550 con 0x7f920015d990 2023-10-16T09:59:05.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.116+0000 7f91fffff700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f920015d990 0x7f920015bb20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f91ec014040 tx=0x7f91ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:05.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.116+0000 7f91fd7fa700 1 -- 172.21.15.158:0/241791762 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec004450 con 0x7f920015d990 2023-10-16T09:59:05.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.116+0000 7f91fd7fa700 1 -- 172.21.15.158:0/241791762 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ec01e070 con 0x7f920015d990 2023-10-16T09:59:05.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.116+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92000acd80 con 0x7f920015d990 2023-10-16T09:59:05.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.116+0000 7f91fd7fa700 1 -- 172.21.15.158:0/241791762 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0038e0 con 0x7f920015d990 2023-10-16T09:59:05.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.116+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92000c77f0 con 0x7f920015d990 2023-10-16T09:59:05.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.117+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92000ad2a0 con 0x7f920015d990 2023-10-16T09:59:05.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.118+0000 7f91fd7fa700 1 -- 172.21.15.158:0/241791762 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f91ec003a40 con 0x7f920015d990 2023-10-16T09:59:05.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.118+0000 7f91fd7fa700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 0x7f91e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.118+0000 7f9204851700 1 -- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 msgr2=0x7f91e406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:05.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.118+0000 7f91fd7fa700 1 -- 172.21.15.158:0/241791762 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f91ec020070 con 0x7f920015d990 2023-10-16T09:59:05.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.118+0000 7f9204851700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 0x7f91e406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:05.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.122+0000 7f91fd7fa700 1 -- 172.21.15.158:0/241791762 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91ec055c70 con 0x7f920015d990 2023-10-16T09:59:05.178 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:05.177+0000 7f68d0037200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:59:05.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.319+0000 7f9204851700 1 -- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 msgr2=0x7f91e406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:05.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.319+0000 7f9204851700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 0x7f91e406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:05.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.320+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92000ad2a0 con 0x7f920015d990 2023-10-16T09:59:05.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.321+0000 7f91fd7fa700 1 -- 172.21.15.158:0/241791762 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+11 (secure 0 0 0) 0x7f92000ad2a0 con 0x7f920015d990 2023-10-16T09:59:05.322 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:59:05.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 msgr2=0x7f91e406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:05.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 0x7f91e406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f920015d990 msgr2=0x7f920015bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f920015d990 0x7f920015bb20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f91ec014040 tx=0x7f91ec004750 comp rx=0 tx=0).stop 2023-10-16T09:59:05.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 shutdown_connections 2023-10-16T09:59:05.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f91e406b020 0x7f91e406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9200162010 0x7f920015c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9200070300 0x7f92000c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 --2- 172.21.15.158:0/241791762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f920015d990 0x7f920015bb20 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 >> 172.21.15.158:0/241791762 conn(0x7f92000cdf20 msgr2=0x7f920015cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:05.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 shutdown_connections 2023-10-16T09:59:05.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.323+0000 7f9206ab5700 1 -- 172.21.15.158:0/241791762 wait complete. 2023-10-16T09:59:05.337 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:59:05.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.461+0000 7f1182b4a700 1 Processor -- start 2023-10-16T09:59:05.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.461+0000 7f1182b4a700 1 -- start start 2023-10-16T09:59:05.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f1182b4a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c144e80 0x7f117c15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f1182b4a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c06ff10 con 0x7f117c072b40 2023-10-16T09:59:05.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f1182b4a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c06f060 con 0x7f117c06afe0 2023-10-16T09:59:05.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f1182b4a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c145260 con 0x7f117c144e80 2023-10-16T09:59:05.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f117bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f117c072b40 0x7f117c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51722/0 (socket says 172.21.15.158:51722) 2023-10-16T09:59:05.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f117bfff700 1 -- 172.21.15.158:0/901690952 learned_addr learned my addr 172.21.15.158:0/901690952 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:05.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f11808e6700 1 --2- 172.21.15.158:0/901690952 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c144e80 0x7f117c15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964099591 0 0) 0x7f117c06ff10 con 0x7f117c072b40 2023-10-16T09:59:05.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f116c0035b0 con 0x7f117c072b40 2023-10-16T09:59:05.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246048325 0 0) 0x7f117c06f060 con 0x7f117c06afe0 2023-10-16T09:59:05.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f117c06ff10 con 0x7f117c06afe0 2023-10-16T09:59:05.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549797864 0 0) 0x7f116c0035b0 con 0x7f117c072b40 2023-10-16T09:59:05.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f11808e6700 1 -- 172.21.15.158:0/901690952 >> v1:172.21.15.172:6789/0 conn(0x7f117c06afe0 legacy=0x7f117c0cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f11808e6700 1 -- 172.21.15.158:0/901690952 >> v1:172.21.15.158:6789/0 conn(0x7f117c072b40 legacy=0x7f117c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.462+0000 7f11808e6700 1 -- 172.21.15.158:0/901690952 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c06f300 con 0x7f117c144e80 2023-10-16T09:59:05.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.463+0000 7f11808e6700 1 --2- 172.21.15.158:0/901690952 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c144e80 0x7f117c15bc40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f117c06fec0 tx=0x7f1170009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f97c6f1b4ad7027f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:05.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.463+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168001670 con 0x7f117c144e80 2023-10-16T09:59:05.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.463+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1170004030 con 0x7f117c144e80 2023-10-16T09:59:05.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.463+0000 7f117b7fe700 1 -- 172.21.15.158:0/901690952 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700043c0 con 0x7f117c144e80 2023-10-16T09:59:05.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.463+0000 7f1182b4a700 1 -- 172.21.15.158:0/901690952 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c144e80 msgr2=0x7f117c15bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.463+0000 7f1182b4a700 1 --2- 172.21.15.158:0/901690952 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c144e80 0x7f117c15bc40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f117c06fec0 tx=0x7f1170009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:05.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.464+0000 7f1182b4a700 1 -- 172.21.15.158:0/901690952 shutdown_connections 2023-10-16T09:59:05.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.464+0000 7f1182b4a700 1 --2- 172.21.15.158:0/901690952 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c144e80 0x7f117c15bc40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.464+0000 7f1182b4a700 1 -- 172.21.15.158:0/901690952 >> 172.21.15.158:0/901690952 conn(0x7f117c06da00 msgr2=0x7f117c06dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:05.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.464+0000 7f1182b4a700 1 -- 172.21.15.158:0/901690952 shutdown_connections 2023-10-16T09:59:05.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.464+0000 7f1182b4a700 1 -- 172.21.15.158:0/901690952 wait complete. 2023-10-16T09:59:05.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.465+0000 7f1182b4a700 1 Processor -- start 2023-10-16T09:59:05.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.465+0000 7f1182b4a700 1 -- start start 2023-10-16T09:59:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f1182b4a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c06afe0 0x7f117c0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f1182b4a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f117c072b40 0x7f117c0c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f1182b4a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117c144e80 0x7f117c140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f1182b4a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c6ca0 con 0x7f117c144e80 2023-10-16T09:59:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f1182b4a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c5c00 con 0x7f117c072b40 2023-10-16T09:59:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f1182b4a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c5ee0 con 0x7f117c06afe0 2023-10-16T09:59:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117c144e80 0x7f117c140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117c144e80 0x7f117c140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51556/0 (socket says 172.21.15.158:51556) 2023-10-16T09:59:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 -- 172.21.15.158:0/3233806164 learned_addr learned my addr 172.21.15.158:0/3233806164 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11808e6700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c06afe0 0x7f117c0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f117bfff700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f117c072b40 0x7f117c0c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 -- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c06afe0 msgr2=0x7f117c0c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c06afe0 0x7f117c0c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 -- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f117c072b40 msgr2=0x7f117c0c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f117c072b40 0x7f117c0c53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 -- 172.21.15.158:0/3233806164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11700094a0 con 0x7f117c144e80 2023-10-16T09:59:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.466+0000 7f11810e7700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117c144e80 0x7f117c140f40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f117400cbc0 tx=0x7f117400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.467+0000 7f11797fa700 1 -- 172.21.15.158:0/3233806164 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174003c00 con 0x7f117c144e80 2023-10-16T09:59:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.467+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c0c61c0 con 0x7f117c144e80 2023-10-16T09:59:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.467+0000 7f11797fa700 1 -- 172.21.15.158:0/3233806164 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174003d60 con 0x7f117c144e80 2023-10-16T09:59:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.467+0000 7f11797fa700 1 -- 172.21.15.158:0/3233806164 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117400aae0 con 0x7f117c144e80 2023-10-16T09:59:05.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.467+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c0c6780 con 0x7f117c144e80 2023-10-16T09:59:05.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.468+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c15d500 con 0x7f117c144e80 2023-10-16T09:59:05.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.469+0000 7f11797fa700 1 -- 172.21.15.158:0/3233806164 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f117400ac40 con 0x7f117c144e80 2023-10-16T09:59:05.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.469+0000 7f11797fa700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 0x7f116006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:05.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.469+0000 7f11808e6700 1 -- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 msgr2=0x7f116006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:05.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.469+0000 7f11797fa700 1 -- 172.21.15.158:0/3233806164 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f1174089be0 con 0x7f117c144e80 2023-10-16T09:59:05.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.469+0000 7f11808e6700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 0x7f116006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:05.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.477+0000 7f11797fa700 1 -- 172.21.15.158:0/3233806164 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11740546e0 con 0x7f117c144e80 2023-10-16T09:59:05.664 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:05.662+0000 7f194ef5b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:59:05.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.670+0000 7f11808e6700 1 -- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 msgr2=0x7f116006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:05.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.670+0000 7f11808e6700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 0x7f116006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:05.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.727+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f117c0681b0 con 0x7f117c144e80 2023-10-16T09:59:05.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.728+0000 7f11797fa700 1 -- 172.21.15.158:0/3233806164 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v72) v1 ==== 80+0+475 (secure 0 0 0) 0x7f1174057d00 con 0x7f117c144e80 2023-10-16T09:59:05.729 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3405184500 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.730 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/488409315 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.730 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2953589465 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.730 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2437477336 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.730 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/211595975 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.731 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/211595975 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.731 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2185855121 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.731 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3427024404 2023-10-17T09:57:16.707111+0000 2023-10-16T09:59:05.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 msgr2=0x7f116006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:05.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 0x7f116006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117c144e80 msgr2=0x7f117c140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:05.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117c144e80 0x7f117c140f40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f117400cbc0 tx=0x7f117400ced0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 shutdown_connections 2023-10-16T09:59:05.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f116006b020 0x7f116006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117c144e80 0x7f117c140f40 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f117c06afe0 0x7f117c0c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 --2- 172.21.15.158:0/3233806164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f117c072b40 0x7f117c0c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:05.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 >> 172.21.15.158:0/3233806164 conn(0x7f117c06da00 msgr2=0x7f117c142df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:05.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 shutdown_connections 2023-10-16T09:59:05.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:05.730+0000 7f1182b4a700 1 -- 172.21.15.158:0/3233806164 wait complete. 2023-10-16T09:59:05.737 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T09:59:05.737 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:05.732+0000 7f194ef5b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:59:05.745 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T09:59:05.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:05.746 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T09:59:05.783 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:05.782+0000 7f68d0037200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:59:05.793 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:59:05.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:05.794 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T09:59:05.852 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:05.853 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:05.859 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:05.858+0000 7f68d0037200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:59:05.870 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:05.870 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:05.871 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:05.889 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:59:05.890 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:05.890 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:05.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:05.928 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:06.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.054+0000 7f1eb652c700 1 Processor -- start 2023-10-16T09:59:06.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.054+0000 7f1eb652c700 1 -- start start 2023-10-16T09:59:06.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.055+0000 7f1eb652c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.055+0000 7f1eb652c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00cd5a0 con 0x7f1eb00b1250 2023-10-16T09:59:06.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.055+0000 7f1eb652c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00cc9f0 con 0x7f1eb006ae10 2023-10-16T09:59:06.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.055+0000 7f1eb652c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00ccc70 con 0x7f1eb015b7c0 2023-10-16T09:59:06.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.055+0000 7f1eaffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1eb00b1250 0x7f1eb00b1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51738/0 (socket says 172.21.15.158:51738) 2023-10-16T09:59:06.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.055+0000 7f1eaffff700 1 -- 172.21.15.158:0/1090999093 learned_addr learned my addr 172.21.15.158:0/1090999093 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:06.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.055+0000 7f1eaf7fe700 1 --2- 172.21.15.158:0/1090999093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaeffd700 1 -- 172.21.15.158:0/1090999093 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599218318 0 0) 0x7f1eb00cd5a0 con 0x7f1eb00b1250 2023-10-16T09:59:06.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaeffd700 1 -- 172.21.15.158:0/1090999093 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e980035b0 con 0x7f1eb00b1250 2023-10-16T09:59:06.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaf7fe700 1 -- 172.21.15.158:0/1090999093 >> v1:172.21.15.172:6789/0 conn(0x7f1eb006ae10 legacy=0x7f1eb00ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaeffd700 1 -- 172.21.15.158:0/1090999093 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714071830 0 0) 0x7f1eb00cc9f0 con 0x7f1eb006ae10 2023-10-16T09:59:06.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaf7fe700 1 -- 172.21.15.158:0/1090999093 >> v1:172.21.15.158:6789/0 conn(0x7f1eb00b1250 legacy=0x7f1eb00b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaf7fe700 1 -- 172.21.15.158:0/1090999093 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00ccf50 con 0x7f1eb015b7c0 2023-10-16T09:59:06.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaf7fe700 1 --2- 172.21.15.158:0/1090999093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb006a8d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009870 tx=0x7f1ea4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edd5163049f7019d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:06.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaeffd700 1 -- 172.21.15.158:0/1090999093 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4011830 con 0x7f1eb015b7c0 2023-10-16T09:59:06.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaeffd700 1 -- 172.21.15.158:0/1090999093 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea4011990 con 0x7f1eb015b7c0 2023-10-16T09:59:06.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.056+0000 7f1eaeffd700 1 -- 172.21.15.158:0/1090999093 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4011d20 con 0x7f1eb015b7c0 2023-10-16T09:59:06.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.057+0000 7f1eb652c700 1 -- 172.21.15.158:0/1090999093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 msgr2=0x7f1eb006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.057+0000 7f1eb652c700 1 --2- 172.21.15.158:0/1090999093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb006a8d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009870 tx=0x7f1ea4009b80 comp rx=0 tx=0).stop 2023-10-16T09:59:06.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.057+0000 7f1eb652c700 1 -- 172.21.15.158:0/1090999093 shutdown_connections 2023-10-16T09:59:06.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.057+0000 7f1eb652c700 1 --2- 172.21.15.158:0/1090999093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb006a8d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.057+0000 7f1eb652c700 1 -- 172.21.15.158:0/1090999093 >> 172.21.15.158:0/1090999093 conn(0x7f1eb00d03a0 msgr2=0x7f1eb00d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:06.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.057+0000 7f1eb652c700 1 -- 172.21.15.158:0/1090999093 shutdown_connections 2023-10-16T09:59:06.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.057+0000 7f1eb652c700 1 -- 172.21.15.158:0/1090999093 wait complete. 2023-10-16T09:59:06.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.058+0000 7f1eb652c700 1 Processor -- start 2023-10-16T09:59:06.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.058+0000 7f1eb652c700 1 -- start start 2023-10-16T09:59:06.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eb652c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eb006ae10 0x7f1eb00c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eb652c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eb00b1250 0x7f1eb0150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eb652c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb00cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eb652c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00c9540 con 0x7f1eb00b1250 2023-10-16T09:59:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eb652c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00c8930 con 0x7f1eb006ae10 2023-10-16T09:59:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eb652c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00c8c10 con 0x7f1eb015b7c0 2023-10-16T09:59:06.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eb00b1250 0x7f1eb0150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eb00b1250 0x7f1eb0150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51570/0 (socket says 172.21.15.158:51570) 2023-10-16T09:59:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 -- 172.21.15.158:0/3537167761 learned_addr learned my addr 172.21.15.158:0/3537167761 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:06.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaffff700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eb006ae10 0x7f1eb00c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eb4ac9700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb00cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 -- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 msgr2=0x7f1eb00cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb00cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 -- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eb006ae10 msgr2=0x7f1eb00c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eb006ae10 0x7f1eb00c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 -- 172.21.15.158:0/3537167761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea40094a0 con 0x7f1eb00b1250 2023-10-16T09:59:06.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.059+0000 7f1eaf7fe700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eb00b1250 0x7f1eb0150840 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40090b0 tx=0x7f1ea4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:06.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.060+0000 7f1eacff9700 1 -- 172.21.15.158:0/3537167761 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea401a7e0 con 0x7f1eb00b1250 2023-10-16T09:59:06.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.060+0000 7f1eacff9700 1 -- 172.21.15.158:0/3537167761 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea401a940 con 0x7f1eb00b1250 2023-10-16T09:59:06.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.060+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb00c8ef0 con 0x7f1eb00b1250 2023-10-16T09:59:06.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.060+0000 7f1eacff9700 1 -- 172.21.15.158:0/3537167761 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea401acd0 con 0x7f1eb00b1250 2023-10-16T09:59:06.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.060+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb0150d80 con 0x7f1eb00b1250 2023-10-16T09:59:06.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.061+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb00c96b0 con 0x7f1eb00b1250 2023-10-16T09:59:06.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.061+0000 7f1eacff9700 1 -- 172.21.15.158:0/3537167761 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1ea4022070 con 0x7f1eb00b1250 2023-10-16T09:59:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.061+0000 7f1eacff9700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 0x7f1e9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.061+0000 7f1eaffff700 1 -- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 msgr2=0x7f1e9c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:06.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.062+0000 7f1eacff9700 1 -- 172.21.15.158:0/3537167761 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f1ea408a900 con 0x7f1eb00b1250 2023-10-16T09:59:06.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.062+0000 7f1eaffff700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 0x7f1e9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:06.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.066+0000 7f1eacff9700 1 -- 172.21.15.158:0/3537167761 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ea4055520 con 0x7f1eb00b1250 2023-10-16T09:59:06.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.263+0000 7f1eaffff700 1 -- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 msgr2=0x7f1e9c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:06.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.263+0000 7f1eaffff700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 0x7f1e9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:06.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.307+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eb00681b0 con 0x7f1eb00b1250 2023-10-16T09:59:06.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.308+0000 7f1eacff9700 1 -- 172.21.15.158:0/3537167761 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f1ea4029090 con 0x7f1eb00b1250 2023-10-16T09:59:06.310 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:06.310 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4264509745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4264509745},{"type":"v1","addr":"172.21.15.158:6835","nonce":4264509745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1091298830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1091298830},{"type":"v1","addr":"172.21.15.158:6837","nonce":1091298830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":35261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2630165052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2630165052},{"type":"v1","addr":"172.21.15.172:6834","nonce":2630165052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:57:03.005758+0000","modified":"2023-10-16T09:57:06.014112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35264":{"gid":35264,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1262687320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1262687320},{"type":"v1","addr":"172.21.15.172:6835","nonce":1262687320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T09:59:06.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 msgr2=0x7f1e9c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:06.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 0x7f1e9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eb00b1250 msgr2=0x7f1eb0150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eb00b1250 0x7f1eb0150840 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40090b0 tx=0x7f1ea4004750 comp rx=0 tx=0).stop 2023-10-16T09:59:06.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 shutdown_connections 2023-10-16T09:59:06.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1e9c06b020 0x7f1e9c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1eb015b7c0 0x7f1eb00cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eb006ae10 0x7f1eb00c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 --2- 172.21.15.158:0/3537167761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1eb00b1250 0x7f1eb0150840 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.310+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 >> 172.21.15.158:0/3537167761 conn(0x7f1eb00d03a0 msgr2=0x7f1eb00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:06.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.311+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 shutdown_connections 2023-10-16T09:59:06.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.311+0000 7f1eb652c700 1 -- 172.21.15.158:0/3537167761 wait complete. 2023-10-16T09:59:06.315 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 25 2023-10-16T09:59:06.325 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:06.366 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:06.365+0000 7f194ef5b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:06.443 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:06.442+0000 7f194ef5b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:59:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.450+0000 7f5f9fdfa700 1 Processor -- start 2023-10-16T09:59:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.451+0000 7f5f9fdfa700 1 -- start start 2023-10-16T09:59:06.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.451+0000 7f5f9fdfa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f98072360 0x7f5f980727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9fdfa700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5f9806a8d0 0x7f5f9806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9fdfa700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9806b260 con 0x7f5f9806f470 2023-10-16T09:59:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9fdfa700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f98072d20 con 0x7f5f9806a8d0 2023-10-16T09:59:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9fdfa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9806f850 con 0x7f5f98072360 2023-10-16T09:59:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9db96700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5f9806f470 0x7f5f9806e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51754/0 (socket says 172.21.15.158:51754) 2023-10-16T09:59:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9db96700 1 -- 172.21.15.158:0/2305662432 learned_addr learned my addr 172.21.15.158:0/2305662432 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9d395700 1 --2- 172.21.15.158:0/2305662432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f98072360 0x7f5f980727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9e397700 1 --2- 172.21.15.158:0/2305662432 >> v2:172.21.15.172:3300/0 conn(0x7f5f9806a8d0 0x7f5f9806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9cb94700 1 -- 172.21.15.158:0/2305662432 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901312894 0 0) 0x7f5f94001290 con 0x7f5f9806f470 2023-10-16T09:59:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9cb94700 1 -- 172.21.15.158:0/2305662432 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f840035b0 con 0x7f5f9806f470 2023-10-16T09:59:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9d395700 1 -- 172.21.15.158:0/2305662432 >> v2:172.21.15.172:3300/0 conn(0x7f5f9806a8d0 msgr2=0x7f5f9806ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9d395700 1 --2- 172.21.15.158:0/2305662432 >> v2:172.21.15.172:3300/0 conn(0x7f5f9806a8d0 0x7f5f9806ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9d395700 1 -- 172.21.15.158:0/2305662432 >> v1:172.21.15.158:6789/0 conn(0x7f5f9806f470 legacy=0x7f5f9806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9d395700 1 -- 172.21.15.158:0/2305662432 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980b4260 con 0x7f5f98072360 2023-10-16T09:59:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.452+0000 7f5f9d395700 1 --2- 172.21.15.158:0/2305662432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f98072360 0x7f5f980727e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009840 tx=0x7f5f88009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=520b02fb59b6e9f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9cb94700 1 -- 172.21.15.158:0/2305662432 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88011830 con 0x7f5f98072360 2023-10-16T09:59:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9cb94700 1 -- 172.21.15.158:0/2305662432 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f88011990 con 0x7f5f98072360 2023-10-16T09:59:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9cb94700 1 -- 172.21.15.158:0/2305662432 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88011d20 con 0x7f5f98072360 2023-10-16T09:59:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/2305662432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f98072360 msgr2=0x7f5f980727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/2305662432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f98072360 0x7f5f980727e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009840 tx=0x7f5f88009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/2305662432 shutdown_connections 2023-10-16T09:59:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/2305662432 >> v2:172.21.15.172:3300/0 conn(0x7f5f9806a8d0 0x7f5f9806ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/2305662432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f98072360 0x7f5f980727e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/2305662432 >> 172.21.15.158:0/2305662432 conn(0x7f5f980ccda0 msgr2=0x7f5f980cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/2305662432 shutdown_connections 2023-10-16T09:59:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.453+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/2305662432 wait complete. 2023-10-16T09:59:06.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.454+0000 7f5f9fdfa700 1 Processor -- start 2023-10-16T09:59:06.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9fdfa700 1 -- start start 2023-10-16T09:59:06.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9fdfa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f9806a8d0 0x7f5f98160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9fdfa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f9806f470 0x7f5f980adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9fdfa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98072360 0x7f5f980ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9fdfa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980cea50 con 0x7f5f98072360 2023-10-16T09:59:06.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9fdfa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980c9010 con 0x7f5f9806a8d0 2023-10-16T09:59:06.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9fdfa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980ce390 con 0x7f5f9806f470 2023-10-16T09:59:06.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9e397700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98072360 0x7f5f980ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9e397700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98072360 0x7f5f980ae320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51588/0 (socket says 172.21.15.158:51588) 2023-10-16T09:59:06.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9e397700 1 -- 172.21.15.158:0/581349511 learned_addr learned my addr 172.21.15.158:0/581349511 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:06.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9db96700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f9806a8d0 0x7f5f98160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.455+0000 7f5f9d395700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f9806f470 0x7f5f980adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9e397700 1 -- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f9806f470 msgr2=0x7f5f980adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9e397700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f9806f470 0x7f5f980adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9e397700 1 -- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f9806a8d0 msgr2=0x7f5f98160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9e397700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f9806a8d0 0x7f5f98160830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9e397700 1 -- 172.21.15.158:0/581349511 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f880094a0 con 0x7f5f98072360 2023-10-16T09:59:06.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9e397700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98072360 0x7f5f980ae320 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000efc0 tx=0x7f5f9000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:06.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/581349511 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90009ea0 con 0x7f5f98072360 2023-10-16T09:59:06.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/581349511 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9000f040 con 0x7f5f98072360 2023-10-16T09:59:06.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/581349511 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000ced0 con 0x7f5f98072360 2023-10-16T09:59:06.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f980ce670 con 0x7f5f98072360 2023-10-16T09:59:06.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.456+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f98160d70 con 0x7f5f98072360 2023-10-16T09:59:06.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.457+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f98071b40 con 0x7f5f98072360 2023-10-16T09:59:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.458+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/581349511 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5f90014480 con 0x7f5f98072360 2023-10-16T09:59:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.458+0000 7f5f8e7fc700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.458+0000 7f5f9db96700 1 -- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 msgr2=0x7f5f7c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.458+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/581349511 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f5f9008a770 con 0x7f5f98072360 2023-10-16T09:59:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.458+0000 7f5f9db96700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:06.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.462+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/581349511 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f98071b40 con 0x7f5f98072360 2023-10-16T09:59:06.530 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:06.529+0000 7f68d0037200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:06.587 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:06.585+0000 7f194ef5b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:06.610 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:06.609+0000 7f68d0037200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:59:06.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.659+0000 7f5f9db96700 1 -- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 msgr2=0x7f5f7c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:06.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.659+0000 7f5f9db96700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:06.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.666+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f980ceb90 con 0x7f5f98072360 2023-10-16T09:59:06.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.668+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/581349511 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+18121 (secure 0 0 0) 0x7f5f980ceb90 con 0x7f5f98072360 2023-10-16T09:59:06.669 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:06.670 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":72,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:04.616345+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:57:01.416542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T09:57:01.962552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.28","mappings":[{"from":0,"to":6}]},{"pgid":"8.3d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3405184500":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/488409315":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2953589465":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2437477336":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6833/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6832/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2185855121":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3427024404":"2023-10-17T09:57:16.707111+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:06.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 msgr2=0x7f5f7c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:06.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98072360 msgr2=0x7f5f980ae320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98072360 0x7f5f980ae320 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000efc0 tx=0x7f5f9000c3a0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 shutdown_connections 2023-10-16T09:59:06.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98072360 0x7f5f980ae320 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f9806a8d0 0x7f5f98160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 --2- 172.21.15.158:0/581349511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f9806f470 0x7f5f980adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 >> 172.21.15.158:0/581349511 conn(0x7f5f980ccda0 msgr2=0x7f5f980b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:06.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 shutdown_connections 2023-10-16T09:59:06.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.670+0000 7f5f9fdfa700 1 -- 172.21.15.158:0/581349511 wait complete. 2023-10-16T09:59:06.684 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:57:01.416542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:57:01.962552+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:06.685 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:06.762 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:06.761+0000 7f68d0037200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:06.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.810+0000 7f08797a2700 1 Processor -- start 2023-10-16T09:59:06.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.810+0000 7f08797a2700 1 -- start start 2023-10-16T09:59:06.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f08797a2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08740b27e0 0x7f08740cc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f08797a2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f08740cffd0 0x7f08740b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f08797a2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08740b2bc0 con 0x7f08740d2030 2023-10-16T09:59:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f08797a2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08740cd250 con 0x7f08740b27e0 2023-10-16T09:59:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f08797a2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08740b42d0 con 0x7f08740cffd0 2023-10-16T09:59:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08740b27e0 0x7f08740cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08740b27e0 0x7f08740cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51622/0 (socket says 172.21.15.158:51622) 2023-10-16T09:59:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 -- 172.21.15.158:0/4112926852 learned_addr learned my addr 172.21.15.158:0/4112926852 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0878fa1700 1 --2- 172.21.15.158:0/4112926852 >> v2:172.21.15.172:3300/0 conn(0x7f08740cffd0 0x7f08740b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 -- 172.21.15.158:0/4112926852 >> v2:172.21.15.172:3300/0 conn(0x7f08740cffd0 msgr2=0x7f08740b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 --2- 172.21.15.158:0/4112926852 >> v2:172.21.15.172:3300/0 conn(0x7f08740cffd0 0x7f08740b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 -- 172.21.15.158:0/4112926852 >> v1:172.21.15.172:6789/0 conn(0x7f08740d2030 legacy=0x7f08740cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 -- 172.21.15.158:0/4112926852 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08740b45b0 con 0x7f08740b27e0 2023-10-16T09:59:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0878fa1700 1 --2- 172.21.15.158:0/4112926852 >> v2:172.21.15.172:3300/0 conn(0x7f08740cffd0 0x7f08740b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0873fff700 1 --2- 172.21.15.158:0/4112926852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740b27e0 0x7f08740cc900 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0874072e20 tx=0x7f0864009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebd4b1f402a3d62 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0872ffd700 1 -- 172.21.15.158:0/4112926852 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086400b870 con 0x7f08740b27e0 2023-10-16T09:59:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0872ffd700 1 -- 172.21.15.158:0/4112926852 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086400b9d0 con 0x7f08740b27e0 2023-10-16T09:59:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.811+0000 7f0872ffd700 1 -- 172.21.15.158:0/4112926852 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086400bd60 con 0x7f08740b27e0 2023-10-16T09:59:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 -- 172.21.15.158:0/4112926852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740b27e0 msgr2=0x7f08740cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 --2- 172.21.15.158:0/4112926852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740b27e0 0x7f08740cc900 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0874072e20 tx=0x7f0864009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 -- 172.21.15.158:0/4112926852 shutdown_connections 2023-10-16T09:59:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 --2- 172.21.15.158:0/4112926852 >> v2:172.21.15.172:3300/0 conn(0x7f08740cffd0 0x7f08740b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 --2- 172.21.15.158:0/4112926852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740b27e0 0x7f08740cc900 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 -- 172.21.15.158:0/4112926852 >> 172.21.15.158:0/4112926852 conn(0x7f08740d12e0 msgr2=0x7f08740d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 -- 172.21.15.158:0/4112926852 shutdown_connections 2023-10-16T09:59:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.812+0000 7f08797a2700 1 -- 172.21.15.158:0/4112926852 wait complete. 2023-10-16T09:59:06.820 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:06.811+0000 7f194ef5b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:59:06.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.814+0000 7f08797a2700 1 Processor -- start 2023-10-16T09:59:06.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.814+0000 7f08797a2700 1 -- start start 2023-10-16T09:59:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f08797a2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08740b27e0 0x7f0874152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f08797a2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08740cffd0 0x7f087414f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f08797a2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740d2030 0x7f087406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f08797a2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087414d230 con 0x7f08740d2030 2023-10-16T09:59:06.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f08797a2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874150480 con 0x7f08740b27e0 2023-10-16T09:59:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f08797a2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087414fec0 con 0x7f08740cffd0 2023-10-16T09:59:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740d2030 0x7f087406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0873fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08740b27e0 0x7f0874152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0873fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08740b27e0 0x7f0874152f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:52808/0 (socket says 172.21.15.158:52808) 2023-10-16T09:59:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740d2030 0x7f087406ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51632/0 (socket says 172.21.15.158:51632) 2023-10-16T09:59:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0873fff700 1 -- 172.21.15.158:0/1843411128 learned_addr learned my addr 172.21.15.158:0/1843411128 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f08737fe700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08740cffd0 0x7f087414f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 -- 172.21.15.158:0/1843411128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08740cffd0 msgr2=0x7f087414f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08740cffd0 0x7f087414f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 -- 172.21.15.158:0/1843411128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08740b27e0 msgr2=0x7f0874152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08740b27e0 0x7f0874152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 -- 172.21.15.158:0/1843411128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0864009550 con 0x7f08740d2030 2023-10-16T09:59:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.815+0000 7f0878fa1700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740d2030 0x7f087406ea20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f086c007e60 tx=0x7f086c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.816+0000 7f0870ff9700 1 -- 172.21.15.158:0/1843411128 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00dd20 con 0x7f08740d2030 2023-10-16T09:59:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.816+0000 7f0870ff9700 1 -- 172.21.15.158:0/1843411128 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c00f040 con 0x7f08740d2030 2023-10-16T09:59:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.816+0000 7f0870ff9700 1 -- 172.21.15.158:0/1843411128 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c003c60 con 0x7f08740d2030 2023-10-16T09:59:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.816+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08741501a0 con 0x7f08740d2030 2023-10-16T09:59:06.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.816+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087414d8d0 con 0x7f08740d2030 2023-10-16T09:59:06.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.817+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087404ebb0 con 0x7f08740d2030 2023-10-16T09:59:06.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.817+0000 7f0870ff9700 1 -- 172.21.15.158:0/1843411128 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f086c004030 con 0x7f08740d2030 2023-10-16T09:59:06.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.818+0000 7f0870ff9700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:06.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.818+0000 7f0873fff700 1 -- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 msgr2=0x7f086006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:06.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.818+0000 7f0870ff9700 1 -- 172.21.15.158:0/1843411128 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f086c089c80 con 0x7f08740d2030 2023-10-16T09:59:06.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.818+0000 7f0873fff700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:06.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:06.822+0000 7f0870ff9700 1 -- 172.21.15.158:0/1843411128 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f086c0546f0 con 0x7f08740d2030 2023-10-16T09:59:06.887 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:06.886+0000 7f194ef5b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:59:06.999 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:06.998+0000 7f68d0037200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:59:07.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.019+0000 7f0873fff700 1 -- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 msgr2=0x7f086006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:07.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.019+0000 7f0873fff700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:07.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.021+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08740d0790 con 0x7f08740d2030 2023-10-16T09:59:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.021+0000 7f0870ff9700 1 -- 172.21.15.158:0/1843411128 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7f086c01a0a0 con 0x7f08740d2030 2023-10-16T09:59:07.023 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:07.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 msgr2=0x7f086006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:07.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740d2030 msgr2=0x7f087406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740d2030 0x7f087406ea20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f086c007e60 tx=0x7f086c00d3b0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 shutdown_connections 2023-10-16T09:59:07.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08740d2030 0x7f087406ea20 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08740b27e0 0x7f0874152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 --2- 172.21.15.158:0/1843411128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08740cffd0 0x7f087414f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 >> 172.21.15.158:0/1843411128 conn(0x7f08740d12e0 msgr2=0x7f08740ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:07.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 shutdown_connections 2023-10-16T09:59:07.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.024+0000 7f08797a2700 1 -- 172.21.15.158:0/1843411128 wait complete. 2023-10-16T09:59:07.038 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:59:07.078 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:07.077+0000 7f68d0037200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:59:07.151 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:07.150+0000 7f194ef5b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:59:07.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.166+0000 7fd31da2f700 1 Processor -- start 2023-10-16T09:59:07.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.166+0000 7fd31da2f700 1 -- start start 2023-10-16T09:59:07.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd31da2f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd31815dad0 0x7fd3180cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd31da2f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd31815deb0 0x7fd3180ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd31da2f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd3180cea20 0x7fd3180c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd31da2f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3180d0ae0 con 0x7fd3180cea20 2023-10-16T09:59:07.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd31da2f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3180d12d0 con 0x7fd31815deb0 2023-10-16T09:59:07.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd31da2f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180cbdf0 con 0x7fd31815dad0 2023-10-16T09:59:07.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd3180cea20 0x7fd3180c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd3180cea20 0x7fd3180c3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51642/0 (socket says 172.21.15.158:51642) 2023-10-16T09:59:07.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 -- 172.21.15.158:0/1443822102 learned_addr learned my addr 172.21.15.158:0/1443822102 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:07.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3167fc700 1 --2- 172.21.15.158:0/1443822102 >> v2:172.21.15.172:3300/0 conn(0x7fd31815deb0 0x7fd3180ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd316ffd700 1 --2- 172.21.15.158:0/1443822102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd31815dad0 0x7fd3180cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 -- 172.21.15.158:0/1443822102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd31815dad0 msgr2=0x7fd3180cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:07.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 --2- 172.21.15.158:0/1443822102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd31815dad0 0x7fd3180cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 -- 172.21.15.158:0/1443822102 >> v2:172.21.15.172:3300/0 conn(0x7fd31815deb0 msgr2=0x7fd3180ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:07.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 --2- 172.21.15.158:0/1443822102 >> v2:172.21.15.172:3300/0 conn(0x7fd31815deb0 0x7fd3180ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.167+0000 7fd3177fe700 1 -- 172.21.15.158:0/1443822102 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3180cc280 con 0x7fd3180cea20 2023-10-16T09:59:07.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.168+0000 7fd3177fe700 1 --2- 172.21.15.158:0/1443822102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3180cea20 0x7fd3180c3f20 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd318071580 tx=0x7fd31000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177b31d5d41d4399 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:07.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.168+0000 7fd315ffb700 1 -- 172.21.15.158:0/1443822102 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31000e070 con 0x7fd3180cea20 2023-10-16T09:59:07.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.168+0000 7fd315ffb700 1 -- 172.21.15.158:0/1443822102 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31000b910 con 0x7fd3180cea20 2023-10-16T09:59:07.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.168+0000 7fd315ffb700 1 -- 172.21.15.158:0/1443822102 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31000bcc0 con 0x7fd3180cea20 2023-10-16T09:59:07.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.168+0000 7fd31da2f700 1 -- 172.21.15.158:0/1443822102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3180cea20 msgr2=0x7fd3180c3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.168+0000 7fd31da2f700 1 --2- 172.21.15.158:0/1443822102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3180cea20 0x7fd3180c3f20 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd318071580 tx=0x7fd31000b330 comp rx=0 tx=0).stop 2023-10-16T09:59:07.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.169+0000 7fd31da2f700 1 -- 172.21.15.158:0/1443822102 shutdown_connections 2023-10-16T09:59:07.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.169+0000 7fd31da2f700 1 --2- 172.21.15.158:0/1443822102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3180cea20 0x7fd3180c3f20 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.169+0000 7fd31da2f700 1 --2- 172.21.15.158:0/1443822102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd31815dad0 0x7fd3180cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.169+0000 7fd31da2f700 1 --2- 172.21.15.158:0/1443822102 >> v2:172.21.15.172:3300/0 conn(0x7fd31815deb0 0x7fd3180ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.169+0000 7fd31da2f700 1 -- 172.21.15.158:0/1443822102 >> 172.21.15.158:0/1443822102 conn(0x7fd318099a10 msgr2=0x7fd3180d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:07.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.169+0000 7fd31da2f700 1 -- 172.21.15.158:0/1443822102 shutdown_connections 2023-10-16T09:59:07.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.169+0000 7fd31da2f700 1 -- 172.21.15.158:0/1443822102 wait complete. 2023-10-16T09:59:07.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.170+0000 7fd31da2f700 1 Processor -- start 2023-10-16T09:59:07.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.171+0000 7fd31da2f700 1 -- start start 2023-10-16T09:59:07.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.171+0000 7fd31da2f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3180cea20 0x7fd3180b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.171+0000 7fd31da2f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31815dad0 0x7fd3180b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.171+0000 7fd31da2f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31815deb0 0x7fd3180c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.171+0000 7fd31da2f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180b4840 con 0x7fd31815dad0 2023-10-16T09:59:07.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.171+0000 7fd31da2f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180b4b20 con 0x7fd31815deb0 2023-10-16T09:59:07.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.171+0000 7fd31da2f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180b4e00 con 0x7fd3180cea20 2023-10-16T09:59:07.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31815dad0 0x7fd3180b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd316ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3180cea20 0x7fd3180b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31815dad0 0x7fd3180b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51646/0 (socket says 172.21.15.158:51646) 2023-10-16T09:59:07.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd316ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3180cea20 0x7fd3180b58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:34246/0 (socket says 172.21.15.158:34246) 2023-10-16T09:59:07.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 -- 172.21.15.158:0/38583646 learned_addr learned my addr 172.21.15.158:0/38583646 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:07.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3177fe700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31815deb0 0x7fd3180c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 -- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3180cea20 msgr2=0x7fd3180b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3180cea20 0x7fd3180b58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 -- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31815deb0 msgr2=0x7fd3180c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31815deb0 0x7fd3180c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 -- 172.21.15.158:0/38583646 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310009b30 con 0x7fd31815dad0 2023-10-16T09:59:07.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd316ffd700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3180cea20 0x7fd3180b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:07.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd3167fc700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31815dad0 0x7fd3180b3f60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd30c010a50 tx=0x7fd30c010d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:07.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd2fffff700 1 -- 172.21.15.158:0/38583646 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c009590 con 0x7fd31815dad0 2023-10-16T09:59:07.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd2fffff700 1 -- 172.21.15.158:0/38583646 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c00ec60 con 0x7fd31815dad0 2023-10-16T09:59:07.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3180b50e0 con 0x7fd31815dad0 2023-10-16T09:59:07.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd2fffff700 1 -- 172.21.15.158:0/38583646 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c011450 con 0x7fd31815dad0 2023-10-16T09:59:07.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.172+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3180b44a0 con 0x7fd31815dad0 2023-10-16T09:59:07.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.173+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3180681b0 con 0x7fd31815dad0 2023-10-16T09:59:07.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.174+0000 7fd2fffff700 1 -- 172.21.15.158:0/38583646 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd30c0115b0 con 0x7fd31815dad0 2023-10-16T09:59:07.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.174+0000 7fd2fffff700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 0x7fd30406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.174+0000 7fd316ffd700 1 -- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 msgr2=0x7fd30406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:07.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.174+0000 7fd2fffff700 1 -- 172.21.15.158:0/38583646 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fd30c08bd40 con 0x7fd31815dad0 2023-10-16T09:59:07.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.174+0000 7fd316ffd700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 0x7fd30406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:07.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.179+0000 7fd2fffff700 1 -- 172.21.15.158:0/38583646 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd30c0567b0 con 0x7fd31815dad0 2023-10-16T09:59:07.355 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:07.354+0000 7f68d0037200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:59:07.371 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:07.370+0000 7f194ef5b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:59:07.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.375+0000 7fd316ffd700 1 -- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 msgr2=0x7fd30406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:07.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.375+0000 7fd316ffd700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 0x7fd30406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:07.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.378+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3180c62a0 con 0x7fd31815dad0 2023-10-16T09:59:07.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.379+0000 7fd2fffff700 1 -- 172.21.15.158:0/38583646 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd30c01e020 con 0x7fd31815dad0 2023-10-16T09:59:07.380 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:59:07.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 msgr2=0x7fd30406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:07.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 0x7fd30406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31815dad0 msgr2=0x7fd3180b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31815dad0 0x7fd3180b3f60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd30c010a50 tx=0x7fd30c010d60 comp rx=0 tx=0).stop 2023-10-16T09:59:07.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 shutdown_connections 2023-10-16T09:59:07.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7fd30406b020 0x7fd30406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31815deb0 0x7fd3180c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3180cea20 0x7fd3180b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 --2- 172.21.15.158:0/38583646 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31815dad0 0x7fd3180b3f60 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 >> 172.21.15.158:0/38583646 conn(0x7fd318099a10 msgr2=0x7fd3180c5520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:07.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 shutdown_connections 2023-10-16T09:59:07.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.381+0000 7fd31da2f700 1 -- 172.21.15.158:0/38583646 wait complete. 2023-10-16T09:59:07.395 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:59:07.518 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:07.516+0000 7f194ef5b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:59:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.562+0000 7f1f99d98700 1 Processor -- start 2023-10-16T09:59:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.562+0000 7f1f99d98700 1 -- start start 2023-10-16T09:59:07.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f99d98700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f940d0c60 0x7f1f940b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f99d98700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940b3460 0x7f1f940b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f99d98700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f940b4420 con 0x7f1f9415b6b0 2023-10-16T09:59:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f99d98700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9415ba90 con 0x7f1f940d0c60 2023-10-16T09:59:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f99d98700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940b38f0 con 0x7f1f940b3460 2023-10-16T09:59:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f937fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1f9415b6b0 0x7f1f940cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:51830/0 (socket says 172.21.15.158:51830) 2023-10-16T09:59:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f937fe700 1 -- 172.21.15.158:0/175066344 learned_addr learned my addr 172.21.15.158:0/175066344 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f93fff700 1 --2- 172.21.15.158:0/175066344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940b3460 0x7f1f940b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/175066344 >> v2:172.21.15.172:3300/0 conn(0x7f1f940d0c60 0x7f1f940b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f927fc700 1 -- 172.21.15.158:0/175066344 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726898892 0 0) 0x7f1f940b4420 con 0x7f1f9415b6b0 2023-10-16T09:59:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f927fc700 1 -- 172.21.15.158:0/175066344 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f7c0035b0 con 0x7f1f9415b6b0 2023-10-16T09:59:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f93fff700 1 -- 172.21.15.158:0/175066344 >> v2:172.21.15.172:3300/0 conn(0x7f1f940d0c60 msgr2=0x7f1f940b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f93fff700 1 --2- 172.21.15.158:0/175066344 >> v2:172.21.15.172:3300/0 conn(0x7f1f940d0c60 0x7f1f940b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f93fff700 1 -- 172.21.15.158:0/175066344 >> v1:172.21.15.158:6789/0 conn(0x7f1f9415b6b0 legacy=0x7f1f940cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f93fff700 1 -- 172.21.15.158:0/175066344 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f940cbc20 con 0x7f1f940b3460 2023-10-16T09:59:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.563+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/175066344 >> v2:172.21.15.172:3300/0 conn(0x7f1f940d0c60 0x7f1f940b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.564+0000 7f1f93fff700 1 --2- 172.21.15.158:0/175066344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940b3460 0x7f1f940b48b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009d90 tx=0x7f1f8c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5005d08e9b688d5f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.564+0000 7f1f927fc700 1 -- 172.21.15.158:0/175066344 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c00e070 con 0x7f1f940b3460 2023-10-16T09:59:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.564+0000 7f1f927fc700 1 -- 172.21.15.158:0/175066344 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c00bb20 con 0x7f1f940b3460 2023-10-16T09:59:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.564+0000 7f1f927fc700 1 -- 172.21.15.158:0/175066344 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c012430 con 0x7f1f940b3460 2023-10-16T09:59:07.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 -- 172.21.15.158:0/175066344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940b3460 msgr2=0x7f1f940b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 --2- 172.21.15.158:0/175066344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940b3460 0x7f1f940b48b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009d90 tx=0x7f1f8c00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:07.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 -- 172.21.15.158:0/175066344 shutdown_connections 2023-10-16T09:59:07.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 --2- 172.21.15.158:0/175066344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940b3460 0x7f1f940b48b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 --2- 172.21.15.158:0/175066344 >> v2:172.21.15.172:3300/0 conn(0x7f1f940d0c60 0x7f1f940b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 -- 172.21.15.158:0/175066344 >> 172.21.15.158:0/175066344 conn(0x7f1f940cdf20 msgr2=0x7f1f940ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:07.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 -- 172.21.15.158:0/175066344 shutdown_connections 2023-10-16T09:59:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.565+0000 7f1f99d98700 1 -- 172.21.15.158:0/175066344 wait complete. 2023-10-16T09:59:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.566+0000 7f1f99d98700 1 Processor -- start 2023-10-16T09:59:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.567+0000 7f1f99d98700 1 -- start start 2023-10-16T09:59:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.567+0000 7f1f99d98700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940b3460 0x7f1f940a93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.567+0000 7f1f99d98700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940d0c60 0x7f1f940b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f99d98700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415b6b0 0x7f1f940b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f99d98700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9406be70 con 0x7f1f9415b6b0 2023-10-16T09:59:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f99d98700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9406c150 con 0x7f1f940b3460 2023-10-16T09:59:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f99d98700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9406c430 con 0x7f1f940d0c60 2023-10-16T09:59:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415b6b0 0x7f1f940b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415b6b0 0x7f1f940b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51664/0 (socket says 172.21.15.158:51664) 2023-10-16T09:59:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 -- 172.21.15.158:0/3142152180 learned_addr learned my addr 172.21.15.158:0/3142152180 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940d0c60 0x7f1f940b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f937fe700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940b3460 0x7f1f940a93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 -- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940d0c60 msgr2=0x7f1f940b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940d0c60 0x7f1f940b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 -- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940b3460 msgr2=0x7f1f940a93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940b3460 0x7f1f940a93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 -- 172.21.15.158:0/3142152180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f880094a0 con 0x7f1f9415b6b0 2023-10-16T09:59:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f93fff700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415b6b0 0x7f1f940b2bf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c014040 tx=0x7f1f8c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f7bfff700 1 -- 172.21.15.158:0/3142152180 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0129f0 con 0x7f1f9415b6b0 2023-10-16T09:59:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f7bfff700 1 -- 172.21.15.158:0/3142152180 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c020070 con 0x7f1f9415b6b0 2023-10-16T09:59:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c009a70 con 0x7f1f9415b6b0 2023-10-16T09:59:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f7bfff700 1 -- 172.21.15.158:0/3142152180 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c012db0 con 0x7f1f9415b6b0 2023-10-16T09:59:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.568+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9406c9f0 con 0x7f1f9415b6b0 2023-10-16T09:59:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.570+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f940681b0 con 0x7f1f9415b6b0 2023-10-16T09:59:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.570+0000 7f1f7bfff700 1 -- 172.21.15.158:0/3142152180 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1f8c004040 con 0x7f1f9415b6b0 2023-10-16T09:59:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.570+0000 7f1f7bfff700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.570+0000 7f1f937fe700 1 -- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 msgr2=0x7f1f8006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.570+0000 7f1f7bfff700 1 -- 172.21.15.158:0/3142152180 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f1f8c022070 con 0x7f1f9415b6b0 2023-10-16T09:59:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.571+0000 7f1f937fe700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:07.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.575+0000 7f1f7bfff700 1 -- 172.21.15.158:0/3142152180 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f8c0565f0 con 0x7f1f9415b6b0 2023-10-16T09:59:07.592 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:07.591+0000 7f68d0037200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:59:07.631 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:07.630+0000 7f194ef5b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:59:07.748 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:07.747+0000 7f68d0037200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:59:07.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.772+0000 7f1f937fe700 1 -- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 msgr2=0x7f1f8006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:07.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.772+0000 7f1f937fe700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:07.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.778+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1f940b1a20 con 0x7f1f9415b6b0 2023-10-16T09:59:07.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.778+0000 7f1f7bfff700 1 -- 172.21.15.158:0/3142152180 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1f8c059c10 con 0x7f1f9415b6b0 2023-10-16T09:59:07.780 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:59:07.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 msgr2=0x7f1f8006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:07.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415b6b0 msgr2=0x7f1f940b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415b6b0 0x7f1f940b2bf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c014040 tx=0x7f1f8c004ad0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 shutdown_connections 2023-10-16T09:59:07.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415b6b0 0x7f1f940b2bf0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940b3460 0x7f1f940a93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.780+0000 7f1f99d98700 1 --2- 172.21.15.158:0/3142152180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940d0c60 0x7f1f940b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.781+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 >> 172.21.15.158:0/3142152180 conn(0x7f1f940cdf20 msgr2=0x7f1f940b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:07.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.781+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 shutdown_connections 2023-10-16T09:59:07.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.781+0000 7f1f99d98700 1 -- 172.21.15.158:0/3142152180 wait complete. 2023-10-16T09:59:07.785 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:07.783+0000 7f194ef5b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:59:07.794 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:07.870 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:07.869+0000 7f68d0037200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:59:07.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.917+0000 7f135fc9c700 1 Processor -- start 2023-10-16T09:59:07.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.917+0000 7f135fc9c700 1 -- start start 2023-10-16T09:59:07.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135fc9c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f135815a6c0 0x7f13580723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135fc9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f135806d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135fc9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f135806dec0 0x7f135815e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135fc9c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13580ca7e0 con 0x7f135815a6c0 2023-10-16T09:59:07.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135fc9c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135806e340 con 0x7f135806dec0 2023-10-16T09:59:07.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135fc9c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1358072910 con 0x7f13580af590 2023-10-16T09:59:07.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135da38700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f135815a6c0 0x7f13580723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135da38700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f135815a6c0 0x7f13580723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51670/0 (socket says 172.21.15.158:51670) 2023-10-16T09:59:07.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135da38700 1 -- 172.21.15.158:0/3409384013 learned_addr learned my addr 172.21.15.158:0/3409384013 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:07.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135e239700 1 --2- 172.21.15.158:0/3409384013 >> v2:172.21.15.172:3300/0 conn(0x7f135806dec0 0x7f135815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.918+0000 7f135d237700 1 --2- 172.21.15.158:0/3409384013 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f135806d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135da38700 1 -- 172.21.15.158:0/3409384013 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 msgr2=0x7f135806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:07.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135da38700 1 --2- 172.21.15.158:0/3409384013 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f135806d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135da38700 1 -- 172.21.15.158:0/3409384013 >> v2:172.21.15.172:3300/0 conn(0x7f135806dec0 msgr2=0x7f135815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:07.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135da38700 1 --2- 172.21.15.158:0/3409384013 >> v2:172.21.15.172:3300/0 conn(0x7f135806dec0 0x7f135815e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135da38700 1 -- 172.21.15.158:0/3409384013 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580b1e40 con 0x7f135815a6c0 2023-10-16T09:59:07.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135da38700 1 --2- 172.21.15.158:0/3409384013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135815a6c0 0x7f13580723d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f135809de90 tx=0x7f1354009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82c90026a95a1eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:07.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135ca36700 1 -- 172.21.15.158:0/3409384013 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354005e30 con 0x7f135815a6c0 2023-10-16T09:59:07.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135ca36700 1 -- 172.21.15.158:0/3409384013 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354004030 con 0x7f135815a6c0 2023-10-16T09:59:07.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.919+0000 7f135ca36700 1 -- 172.21.15.158:0/3409384013 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13540043e0 con 0x7f135815a6c0 2023-10-16T09:59:07.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 -- 172.21.15.158:0/3409384013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135815a6c0 msgr2=0x7f13580723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 --2- 172.21.15.158:0/3409384013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135815a6c0 0x7f13580723d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f135809de90 tx=0x7f1354009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 -- 172.21.15.158:0/3409384013 shutdown_connections 2023-10-16T09:59:07.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 --2- 172.21.15.158:0/3409384013 >> v2:172.21.15.172:3300/0 conn(0x7f135806dec0 0x7f135815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 --2- 172.21.15.158:0/3409384013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135815a6c0 0x7f13580723d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 --2- 172.21.15.158:0/3409384013 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f135806d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 -- 172.21.15.158:0/3409384013 >> 172.21.15.158:0/3409384013 conn(0x7f13580ccda0 msgr2=0x7f13580cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:07.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 -- 172.21.15.158:0/3409384013 shutdown_connections 2023-10-16T09:59:07.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.920+0000 7f135fc9c700 1 -- 172.21.15.158:0/3409384013 wait complete. 2023-10-16T09:59:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.921+0000 7f135fc9c700 1 Processor -- start 2023-10-16T09:59:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.921+0000 7f135fc9c700 1 -- start start 2023-10-16T09:59:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135fc9c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135806dec0 0x7f1358156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135fc9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f13580c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135fc9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815a6c0 0x7f13580ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135fc9c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580c7630 con 0x7f135806dec0 2023-10-16T09:59:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135fc9c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580c6410 con 0x7f135815a6c0 2023-10-16T09:59:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135fc9c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13580c66f0 con 0x7f13580af590 2023-10-16T09:59:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135806dec0 0x7f1358156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135806dec0 0x7f1358156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51676/0 (socket says 172.21.15.158:51676) 2023-10-16T09:59:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 -- 172.21.15.158:0/205789973 learned_addr learned my addr 172.21.15.158:0/205789973 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135e239700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815a6c0 0x7f13580ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135d237700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f13580c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 -- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 msgr2=0x7f13580c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f13580c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 -- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815a6c0 msgr2=0x7f13580ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815a6c0 0x7f13580ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 -- 172.21.15.158:0/205789973 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1354009500 con 0x7f135806dec0 2023-10-16T09:59:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f135da38700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135806dec0 0x7f1358156b40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1354009c10 tx=0x7f1354004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.922+0000 7f134e7fc700 1 -- 172.21.15.158:0/205789973 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354003bb0 con 0x7f135806dec0 2023-10-16T09:59:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.923+0000 7f134e7fc700 1 -- 172.21.15.158:0/205789973 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354003d10 con 0x7f135806dec0 2023-10-16T09:59:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.923+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13580c69d0 con 0x7f135806dec0 2023-10-16T09:59:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.923+0000 7f134e7fc700 1 -- 172.21.15.158:0/205789973 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135401c660 con 0x7f135806dec0 2023-10-16T09:59:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.923+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13580c6f90 con 0x7f135806dec0 2023-10-16T09:59:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.923+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13580681b0 con 0x7f135806dec0 2023-10-16T09:59:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.924+0000 7f134e7fc700 1 -- 172.21.15.158:0/205789973 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f135401c7c0 con 0x7f135806dec0 2023-10-16T09:59:07.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.924+0000 7f134e7fc700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:07.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.924+0000 7f135d237700 1 -- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 msgr2=0x7f133c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:07.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.925+0000 7f134e7fc700 1 -- 172.21.15.158:0/205789973 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f135408aaa0 con 0x7f135806dec0 2023-10-16T09:59:07.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.925+0000 7f135d237700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:07.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:07.929+0000 7f134e7fc700 1 -- 172.21.15.158:0/205789973 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1354055580 con 0x7f135806dec0 2023-10-16T09:59:08.034 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:08.033+0000 7f68d0037200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:59:08.047 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:08.046+0000 7f194ef5b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:08.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.126+0000 7f135d237700 1 -- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 msgr2=0x7f133c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:08.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.126+0000 7f135d237700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:08.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.172+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13580c9d50 con 0x7f135806dec0 2023-10-16T09:59:08.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.173+0000 7f134e7fc700 1 -- 172.21.15.158:0/205789973 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f1354058ba0 con 0x7f135806dec0 2023-10-16T09:59:08.175 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:08.175 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4264509745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4264509745},{"type":"v1","addr":"172.21.15.158:6835","nonce":4264509745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1091298830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1091298830},{"type":"v1","addr":"172.21.15.158:6837","nonce":1091298830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":35261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2630165052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2630165052},{"type":"v1","addr":"172.21.15.172:6834","nonce":2630165052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:57:03.005758+0000","modified":"2023-10-16T09:57:06.014112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35264":{"gid":35264,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1262687320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1262687320},{"type":"v1","addr":"172.21.15.172:6835","nonce":1262687320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T09:59:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 msgr2=0x7f133c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135806dec0 msgr2=0x7f1358156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135806dec0 0x7f1358156b40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1354009c10 tx=0x7f1354004750 comp rx=0 tx=0).stop 2023-10-16T09:59:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 shutdown_connections 2023-10-16T09:59:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815a6c0 0x7f13580ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f135806dec0 0x7f1358156b40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 --2- 172.21.15.158:0/205789973 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13580af590 0x7f13580c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 >> 172.21.15.158:0/205789973 conn(0x7f13580ccda0 msgr2=0x7f13580ad9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 shutdown_connections 2023-10-16T09:59:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.175+0000 7f135fc9c700 1 -- 172.21.15.158:0/205789973 wait complete. 2023-10-16T09:59:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 25 2023-10-16T09:59:08.190 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:08.204 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:08.203+0000 7f194ef5b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:59:08.311 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:08.309+0000 7f68d0037200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:08.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.311+0000 7f09f220b700 1 Processor -- start 2023-10-16T09:59:08.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.311+0000 7f09f220b700 1 -- start start 2023-10-16T09:59:08.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.312+0000 7f09f220b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f09ec0ce530 0x7f09ec0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.312+0000 7f09f220b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.312+0000 7f09f220b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec15b930 con 0x7f09ec15f0f0 2023-10-16T09:59:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.312+0000 7f09f220b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0cf020 con 0x7f09ec0ce530 2023-10-16T09:59:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.312+0000 7f09f220b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec15f5f0 con 0x7f09ec0cc1b0 2023-10-16T09:59:08.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f09ec0ce530 0x7f09ec0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09ebfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eaffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f09ec15f0f0 0x7f09ec0cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:55768/0 (socket says 172.21.15.158:55768) 2023-10-16T09:59:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eaffd700 1 -- 172.21.15.158:0/1645221251 learned_addr learned my addr 172.21.15.158:0/1645221251 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eb7fe700 1 --2- 172.21.15.158:0/1645221251 >> v2:172.21.15.172:3300/0 conn(0x7f09ec0ce530 0x7f09ec0b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:52866/0 (socket says 172.21.15.158:52866) 2023-10-16T09:59:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eb7fe700 1 -- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 msgr2=0x7f09ec0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eb7fe700 1 --2- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eb7fe700 1 -- 172.21.15.158:0/1645221251 >> v1:172.21.15.172:6789/0 conn(0x7f09ec15f0f0 legacy=0x7f09ec0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eb7fe700 1 -- 172.21.15.158:0/1645221251 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec15f8d0 con 0x7f09ec0ce530 2023-10-16T09:59:08.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09ea7fc700 1 -- 172.21.15.158:0/1645221251 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824371075 0 0) 0x7f09ec15b930 con 0x7f09ec15f0f0 2023-10-16T09:59:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09ebfff700 1 --2- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09eb7fe700 1 --2- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 0x7f09ec0b3b20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009d90 tx=0x7f09dc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26379e38820b5055 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:08.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09ea7fc700 1 -- 172.21.15.158:0/1645221251 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc00e070 con 0x7f09ec0ce530 2023-10-16T09:59:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09ea7fc700 1 -- 172.21.15.158:0/1645221251 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc00bb20 con 0x7f09ec0ce530 2023-10-16T09:59:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.313+0000 7f09ea7fc700 1 -- 172.21.15.158:0/1645221251 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc012430 con 0x7f09ec0ce530 2023-10-16T09:59:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 -- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 msgr2=0x7f09ec0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 --2- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 0x7f09ec0b3b20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009d90 tx=0x7f09dc00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 -- 172.21.15.158:0/1645221251 shutdown_connections 2023-10-16T09:59:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 --2- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 --2- 172.21.15.158:0/1645221251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 0x7f09ec0b3b20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 -- 172.21.15.158:0/1645221251 >> 172.21.15.158:0/1645221251 conn(0x7f09ec06a8d0 msgr2=0x7f09ec06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 -- 172.21.15.158:0/1645221251 shutdown_connections 2023-10-16T09:59:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.314+0000 7f09f220b700 1 -- 172.21.15.158:0/1645221251 wait complete. 2023-10-16T09:59:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.315+0000 7f09f220b700 1 Processor -- start 2023-10-16T09:59:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.315+0000 7f09f220b700 1 -- start start 2023-10-16T09:59:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09f220b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09f220b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 0x7f09ec1f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09f220b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ec15f0f0 0x7f09ec0b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09f220b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1f3e20 con 0x7f09ec15f0f0 2023-10-16T09:59:08.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09f220b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0b41d0 con 0x7f09ec0ce530 2023-10-16T09:59:08.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09f220b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0b6b80 con 0x7f09ec0cc1b0 2023-10-16T09:59:08.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ec15f0f0 0x7f09ec0b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ec15f0f0 0x7f09ec0b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51696/0 (socket says 172.21.15.158:51696) 2023-10-16T09:59:08.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 -- 172.21.15.158:0/4102421130 learned_addr learned my addr 172.21.15.158:0/4102421130 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:08.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09eb7fe700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09eaffd700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 0x7f09ec1f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 -- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 msgr2=0x7f09ec0b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0b6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 -- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 msgr2=0x7f09ec1f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 0x7f09ec1f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.316+0000 7f09ebfff700 1 -- 172.21.15.158:0/4102421130 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e40094a0 con 0x7f09ec15f0f0 2023-10-16T09:59:08.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.317+0000 7f09ebfff700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ec15f0f0 0x7f09ec0b3c90 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f09e400e8c0 tx=0x7f09e400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.317+0000 7f09e3fff700 1 -- 172.21.15.158:0/4102421130 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4013680 con 0x7f09ec15f0f0 2023-10-16T09:59:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.317+0000 7f09e3fff700 1 -- 172.21.15.158:0/4102421130 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e4013800 con 0x7f09ec15f0f0 2023-10-16T09:59:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.317+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc009a70 con 0x7f09ec15f0f0 2023-10-16T09:59:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.317+0000 7f09e3fff700 1 -- 172.21.15.158:0/4102421130 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4013bb0 con 0x7f09ec15f0f0 2023-10-16T09:59:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.317+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec0ca970 con 0x7f09ec15f0f0 2023-10-16T09:59:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.318+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec144af0 con 0x7f09ec15f0f0 2023-10-16T09:59:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.318+0000 7f09e3fff700 1 -- 172.21.15.158:0/4102421130 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f09e4013d10 con 0x7f09ec15f0f0 2023-10-16T09:59:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.319+0000 7f09e3fff700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 0x7f09cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.319+0000 7f09eb7fe700 1 -- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 msgr2=0x7f09cc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.319+0000 7f09e3fff700 1 -- 172.21.15.158:0/4102421130 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f09e4089b90 con 0x7f09ec15f0f0 2023-10-16T09:59:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.319+0000 7f09eb7fe700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 0x7f09cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:08.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.323+0000 7f09e3fff700 1 -- 172.21.15.158:0/4102421130 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09e4054600 con 0x7f09ec15f0f0 2023-10-16T09:59:08.479 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:08.478+0000 7f68d0037200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:59:08.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.520+0000 7f09eb7fe700 1 -- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 msgr2=0x7f09cc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:08.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.520+0000 7f09eb7fe700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 0x7f09cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:08.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.522+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09ec0681b0 con 0x7f09ec15f0f0 2023-10-16T09:59:08.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.524+0000 7f09e3fff700 1 -- 172.21.15.158:0/4102421130 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+18121 (secure 0 0 0) 0x7f09e4017090 con 0x7f09ec15f0f0 2023-10-16T09:59:08.525 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:08.526 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":72,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:04.616345+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:57:01.416542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T09:57:01.962552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.28","mappings":[{"from":0,"to":6}]},{"pgid":"8.3d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3405184500":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/488409315":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2953589465":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2437477336":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6833/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6832/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2185855121":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3427024404":"2023-10-17T09:57:16.707111+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:08.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 msgr2=0x7f09cc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 0x7f09cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ec15f0f0 msgr2=0x7f09ec0b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ec15f0f0 0x7f09ec0b3c90 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f09e400e8c0 tx=0x7f09e400ebd0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 shutdown_connections 2023-10-16T09:59:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f09cc06b020 0x7f09cc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ec15f0f0 0x7f09ec0b3c90 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ec0cc1b0 0x7f09ec0b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 --2- 172.21.15.158:0/4102421130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ec0ce530 0x7f09ec1f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 >> 172.21.15.158:0/4102421130 conn(0x7f09ec06a8d0 msgr2=0x7f09ec15e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:08.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 shutdown_connections 2023-10-16T09:59:08.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.526+0000 7f09f220b700 1 -- 172.21.15.158:0/4102421130 wait complete. 2023-10-16T09:59:08.540 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T09:59:08.541 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:08.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.664+0000 7f85ac3ac700 1 Processor -- start 2023-10-16T09:59:08.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.664+0000 7f85ac3ac700 1 -- start start 2023-10-16T09:59:08.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85ac3ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40ce530 0x7f85a40b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85ac3ac700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85a40cc1b0 0x7f85a40cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85ac3ac700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a415b930 con 0x7f85a415f0f0 2023-10-16T09:59:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85ac3ac700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a40cf020 con 0x7f85a40cc1b0 2023-10-16T09:59:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85ac3ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a415f5f0 con 0x7f85a40ce530 2023-10-16T09:59:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa949700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85a40cc1b0 0x7f85a40cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa148700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40ce530 0x7f85a40b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa949700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85a40cc1b0 0x7f85a40cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51704/0 (socket says 172.21.15.158:51704) 2023-10-16T09:59:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa949700 1 -- 172.21.15.158:0/3647171405 learned_addr learned my addr 172.21.15.158:0/3647171405 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85a9947700 1 --1- 172.21.15.158:0/3647171405 >> v1:172.21.15.172:6789/0 conn(0x7f85a415f0f0 0x7f85a40cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:55782/0 (socket says 172.21.15.158:55782) 2023-10-16T09:59:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85a9146700 1 -- 172.21.15.158:0/3647171405 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570604802 0 0) 0x7f85a415b930 con 0x7f85a415f0f0 2023-10-16T09:59:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa949700 1 -- 172.21.15.158:0/3647171405 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40ce530 msgr2=0x7f85a40b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa949700 1 --2- 172.21.15.158:0/3647171405 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40ce530 0x7f85a40b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa949700 1 -- 172.21.15.158:0/3647171405 >> v1:172.21.15.172:6789/0 conn(0x7f85a415f0f0 legacy=0x7f85a40cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.665+0000 7f85aa949700 1 -- 172.21.15.158:0/3647171405 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a415f8d0 con 0x7f85a40cc1b0 2023-10-16T09:59:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85aa949700 1 --2- 172.21.15.158:0/3647171405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40cc1b0 0x7f85a40cf470 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f859c009e50 tx=0x7f859c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73586b6696863db0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85a9146700 1 -- 172.21.15.158:0/3647171405 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00e070 con 0x7f85a40cc1b0 2023-10-16T09:59:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85a9146700 1 -- 172.21.15.158:0/3647171405 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c00b910 con 0x7f85a40cc1b0 2023-10-16T09:59:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85a9146700 1 -- 172.21.15.158:0/3647171405 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00bcc0 con 0x7f85a40cc1b0 2023-10-16T09:59:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 -- 172.21.15.158:0/3647171405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40cc1b0 msgr2=0x7f85a40cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/3647171405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40cc1b0 0x7f85a40cf470 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f859c009e50 tx=0x7f859c00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 -- 172.21.15.158:0/3647171405 shutdown_connections 2023-10-16T09:59:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/3647171405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40cc1b0 0x7f85a40cf470 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/3647171405 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40ce530 0x7f85a40b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 -- 172.21.15.158:0/3647171405 >> 172.21.15.158:0/3647171405 conn(0x7f85a406a8d0 msgr2=0x7f85a406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 -- 172.21.15.158:0/3647171405 shutdown_connections 2023-10-16T09:59:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.666+0000 7f85ac3ac700 1 -- 172.21.15.158:0/3647171405 wait complete. 2023-10-16T09:59:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 Processor -- start 2023-10-16T09:59:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 -- start start 2023-10-16T09:59:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40cc1b0 0x7f85a41530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40ce530 0x7f85a41f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415f0f0 0x7f85a40b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a40c4cd0 con 0x7f85a40ce530 2023-10-16T09:59:08.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a41f3870 con 0x7f85a415f0f0 2023-10-16T09:59:08.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.668+0000 7f85ac3ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a41f3b50 con 0x7f85a40cc1b0 2023-10-16T09:59:08.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40ce530 0x7f85a41f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40ce530 0x7f85a41f3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51708/0 (socket says 172.21.15.158:51708) 2023-10-16T09:59:08.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 -- 172.21.15.158:0/518525373 learned_addr learned my addr 172.21.15.158:0/518525373 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:08.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85aa148700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40cc1b0 0x7f85a41530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85aa949700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415f0f0 0x7f85a40b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:08.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 -- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40cc1b0 msgr2=0x7f85a41530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40cc1b0 0x7f85a41530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 -- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415f0f0 msgr2=0x7f85a40b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415f0f0 0x7f85a40b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 -- 172.21.15.158:0/518525373 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c009b30 con 0x7f85a40ce530 2023-10-16T09:59:08.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85a9947700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40ce530 0x7f85a41f3010 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f85940011e0 tx=0x7f859400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:08.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f859affd700 1 -- 172.21.15.158:0/518525373 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594004c80 con 0x7f85a40ce530 2023-10-16T09:59:08.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f859affd700 1 -- 172.21.15.158:0/518525373 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8594004e00 con 0x7f85a40ce530 2023-10-16T09:59:08.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f859affd700 1 -- 172.21.15.158:0/518525373 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594007310 con 0x7f85a40ce530 2023-10-16T09:59:08.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a41f3e30 con 0x7f85a40ce530 2023-10-16T09:59:08.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.669+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a41f3550 con 0x7f85a40ce530 2023-10-16T09:59:08.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.670+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a40c4e80 con 0x7f85a40ce530 2023-10-16T09:59:08.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.671+0000 7f859affd700 1 -- 172.21.15.158:0/518525373 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f8594007470 con 0x7f85a40ce530 2023-10-16T09:59:08.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.671+0000 7f859affd700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:08.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.671+0000 7f85aa148700 1 -- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 msgr2=0x7f858806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:08.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.671+0000 7f859affd700 1 -- 172.21.15.158:0/518525373 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f8594089820 con 0x7f85a40ce530 2023-10-16T09:59:08.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.671+0000 7f85aa148700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:08.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.675+0000 7f859affd700 1 -- 172.21.15.158:0/518525373 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85a40c4e80 con 0x7f85a40ce530 2023-10-16T09:59:08.725 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:08.724+0000 7f194ef5b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:59:08.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.872+0000 7f85aa148700 1 -- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 msgr2=0x7f858806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1799865685 2023-10-16T09:59:08.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.872+0000 7f85aa148700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:08.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.921+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85a40681b0 con 0x7f85a40ce530 2023-10-16T09:59:08.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.922+0000 7f859affd700 1 -- 172.21.15.158:0/518525373 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f85940543a0 con 0x7f85a40ce530 2023-10-16T09:59:08.923 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:08.923 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4264509745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4264509745},{"type":"v1","addr":"172.21.15.158:6835","nonce":4264509745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1091298830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1091298830},{"type":"v1","addr":"172.21.15.158:6837","nonce":1091298830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":35261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2630165052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2630165052},{"type":"v1","addr":"172.21.15.172:6834","nonce":2630165052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:57:03.005758+0000","modified":"2023-10-16T09:57:06.014112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35264":{"gid":35264,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1262687320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1262687320},{"type":"v1","addr":"172.21.15.172:6835","nonce":1262687320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T09:59:08.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 msgr2=0x7f858806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:08.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40ce530 msgr2=0x7f85a41f3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:08.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40ce530 0x7f85a41f3010 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f85940011e0 tx=0x7f859400ca30 comp rx=0 tx=0).stop 2023-10-16T09:59:08.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 shutdown_connections 2023-10-16T09:59:08.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:6832/1799865685,v1:172.21.15.158:6833/1799865685] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415f0f0 0x7f85a40b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a40cc1b0 0x7f85a41530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 --2- 172.21.15.158:0/518525373 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40ce530 0x7f85a41f3010 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:08.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 >> 172.21.15.158:0/518525373 conn(0x7f85a406a8d0 msgr2=0x7f85a40b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:08.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 shutdown_connections 2023-10-16T09:59:08.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:08.924+0000 7f85ac3ac700 1 -- 172.21.15.158:0/518525373 wait complete. 2023-10-16T09:59:08.928 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 25 2023-10-16T09:59:08.938 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:09.039 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.038+0000 7f68d0037200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:59:09.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.062+0000 7f4732147700 1 Processor -- start 2023-10-16T09:59:09.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.062+0000 7f4732147700 1 -- start start 2023-10-16T09:59:09.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.062+0000 7f4732147700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0d1210 0x7f472c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.062+0000 7f4732147700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f472c0ce950 0x7f472c0b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.062+0000 7f4732147700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0d3bd0 con 0x7f472c15d7c0 2023-10-16T09:59:09.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.062+0000 7f4732147700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0d4060 con 0x7f472c0ce950 2023-10-16T09:59:09.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.062+0000 7f4732147700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0cede0 con 0x7f472c0d1210 2023-10-16T09:59:09.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f472c0ce950 0x7f472c0b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0d1210 0x7f472c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f472c0ce950 0x7f472c0b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51722/0 (socket says 172.21.15.158:51722) 2023-10-16T09:59:09.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f472c15d7c0 0x7f472c0d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:55798/0 (socket says 172.21.15.158:55798) 2023-10-16T09:59:09.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 -- 172.21.15.158:0/3564177362 learned_addr learned my addr 172.21.15.158:0/3564177362 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:09.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 -- 172.21.15.158:0/3564177362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0d1210 msgr2=0x7f472c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:09.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 --2- 172.21.15.158:0/3564177362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0d1210 0x7f472c0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 -- 172.21.15.158:0/3564177362 >> v1:172.21.15.172:6789/0 conn(0x7f472c15d7c0 legacy=0x7f472c0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 -- 172.21.15.158:0/3564177362 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0d2fc0 con 0x7f472c0ce950 2023-10-16T09:59:09.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472bfff700 1 --2- 172.21.15.158:0/3564177362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c0ce950 0x7f472c0b6800 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f4724009e50 tx=0x7f472400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e173c3f36c3bde server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:09.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.063+0000 7f472a7fc700 1 -- 172.21.15.158:0/3564177362 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472400e070 con 0x7f472c0ce950 2023-10-16T09:59:09.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f472a7fc700 1 -- 172.21.15.158:0/3564177362 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472400b910 con 0x7f472c0ce950 2023-10-16T09:59:09.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f472a7fc700 1 -- 172.21.15.158:0/3564177362 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472400bca0 con 0x7f472c0ce950 2023-10-16T09:59:09.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f4732147700 1 -- 172.21.15.158:0/3564177362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c0ce950 msgr2=0x7f472c0b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f4732147700 1 --2- 172.21.15.158:0/3564177362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c0ce950 0x7f472c0b6800 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f4724009e50 tx=0x7f472400b330 comp rx=0 tx=0).stop 2023-10-16T09:59:09.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f4732147700 1 -- 172.21.15.158:0/3564177362 shutdown_connections 2023-10-16T09:59:09.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f4732147700 1 --2- 172.21.15.158:0/3564177362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c0ce950 0x7f472c0b6800 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f4732147700 1 --2- 172.21.15.158:0/3564177362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0d1210 0x7f472c0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.064+0000 7f4732147700 1 -- 172.21.15.158:0/3564177362 >> 172.21.15.158:0/3564177362 conn(0x7f472c0669f0 msgr2=0x7f472c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:09.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.065+0000 7f4732147700 1 -- 172.21.15.158:0/3564177362 shutdown_connections 2023-10-16T09:59:09.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.065+0000 7f4732147700 1 -- 172.21.15.158:0/3564177362 wait complete. 2023-10-16T09:59:09.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.066+0000 7f4732147700 1 Processor -- start 2023-10-16T09:59:09.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.067+0000 7f4732147700 1 -- start start 2023-10-16T09:59:09.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f4732147700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0ce950 0x7f472c152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f4732147700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f472c0d1210 0x7f472c155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f4732147700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c15d7c0 0x7f472c155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f4732147700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0c7170 con 0x7f472c15d7c0 2023-10-16T09:59:09.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f4732147700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0c7410 con 0x7f472c0d1210 2023-10-16T09:59:09.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f4732147700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0c76f0 con 0x7f472c0ce950 2023-10-16T09:59:09.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f472c0d1210 0x7f472c155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c15d7c0 0x7f472c155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f472c0d1210 0x7f472c155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:52912/0 (socket says 172.21.15.158:52912) 2023-10-16T09:59:09.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c15d7c0 0x7f472c155ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51734/0 (socket says 172.21.15.158:51734) 2023-10-16T09:59:09.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472affd700 1 -- 172.21.15.158:0/3618545000 learned_addr learned my addr 172.21.15.158:0/3618545000 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:09.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472b7fe700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0ce950 0x7f472c152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472bfff700 1 -- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0ce950 msgr2=0x7f472c152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472bfff700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0ce950 0x7f472c152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472bfff700 1 -- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f472c0d1210 msgr2=0x7f472c155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472bfff700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f472c0d1210 0x7f472c155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472bfff700 1 -- 172.21.15.158:0/3618545000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724009b30 con 0x7f472c15d7c0 2023-10-16T09:59:09.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.068+0000 7f472b7fe700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0ce950 0x7f472c152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:09.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.069+0000 7f472bfff700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c15d7c0 0x7f472c155ea0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4724015000 tx=0x7f4724004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:09.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.069+0000 7f4713fff700 1 -- 172.21.15.158:0/3618545000 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240092e0 con 0x7f472c15d7c0 2023-10-16T09:59:09.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.069+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c0c79d0 con 0x7f472c15d7c0 2023-10-16T09:59:09.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.069+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c153080 con 0x7f472c15d7c0 2023-10-16T09:59:09.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.069+0000 7f4713fff700 1 -- 172.21.15.158:0/3618545000 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4724004250 con 0x7f472c15d7c0 2023-10-16T09:59:09.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.069+0000 7f4713fff700 1 -- 172.21.15.158:0/3618545000 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240045c0 con 0x7f472c15d7c0 2023-10-16T09:59:09.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.070+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470c0052f0 con 0x7f472c15d7c0 2023-10-16T09:59:09.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.071+0000 7f4713fff700 1 -- 172.21.15.158:0/3618545000 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 29) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4724004720 con 0x7f472c15d7c0 2023-10-16T09:59:09.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.071+0000 7f4713fff700 1 -- 172.21.15.158:0/3618545000 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7199+0+0 (secure 0 0 0) 0x7f472408a8e0 con 0x7f472c15d7c0 2023-10-16T09:59:09.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.076+0000 7f4713fff700 1 -- 172.21.15.158:0/3618545000 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47240556a0 con 0x7f472c15d7c0 2023-10-16T09:59:09.136 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.134+0000 7f684b863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.136 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.134+0000 7f684b863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.136 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.134+0000 7f684b863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.134+0000 7f684b863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.134+0000 7f684b863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.135+0000 7f684881d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.135+0000 7f684881d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.135+0000 7f684881d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.135+0000 7f684881d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:09.135+0000 7f684881d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:09.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.329+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f470c005b20 con 0x7f472c15d7c0 2023-10-16T09:59:09.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.330+0000 7f4713fff700 1 -- 172.21.15.158:0/3618545000 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f4724020090 con 0x7f472c15d7c0 2023-10-16T09:59:09.332 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:09.332 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4264509745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4264509745},{"type":"v1","addr":"172.21.15.158:6835","nonce":4264509745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1091298830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1091298830},{"type":"v1","addr":"172.21.15.158:6837","nonce":1091298830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":35261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2630165052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2630165052},{"type":"v1","addr":"172.21.15.172:6834","nonce":2630165052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:57:03.005758+0000","modified":"2023-10-16T09:57:06.014112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35264":{"gid":35264,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1262687320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1262687320},{"type":"v1","addr":"172.21.15.172:6835","nonce":1262687320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T09:59:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c15d7c0 msgr2=0x7f472c155ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c15d7c0 0x7f472c155ea0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4724015000 tx=0x7f4724004b20 comp rx=0 tx=0).stop 2023-10-16T09:59:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 shutdown_connections 2023-10-16T09:59:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f472c15d7c0 0x7f472c155ea0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f472c0ce950 0x7f472c152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 --2- 172.21.15.158:0/3618545000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f472c0d1210 0x7f472c155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 >> 172.21.15.158:0/3618545000 conn(0x7f472c0669f0 msgr2=0x7f472c0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 shutdown_connections 2023-10-16T09:59:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.332+0000 7f4732147700 1 -- 172.21.15.158:0/3618545000 wait complete. 2023-10-16T09:59:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 25 2023-10-16T09:59:09.346 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:09.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.465+0000 7f89eae69700 1 Processor -- start 2023-10-16T09:59:09.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.465+0000 7f89eae69700 1 -- start start 2023-10-16T09:59:09.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.466+0000 7f89eae69700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e415d7c0 0x7f89e406b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.466+0000 7f89eae69700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f89e40cd810 0x7f89e40cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.466+0000 7f89eae69700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e40c4dd0 con 0x7f89e40ce1d0 2023-10-16T09:59:09.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.466+0000 7f89eae69700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e40af0a0 con 0x7f89e40cd810 2023-10-16T09:59:09.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.466+0000 7f89eae69700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40ce660 con 0x7f89e415d7c0 2023-10-16T09:59:09.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.466+0000 7f89e3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f89e40cd810 0x7f89e40cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.466+0000 7f89e8c05700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e415d7c0 0x7f89e406b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e8c05700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e415d7c0 0x7f89e406b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:34402/0 (socket says 172.21.15.158:34402) 2023-10-16T09:59:09.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f89e40cd810 0x7f89e40cdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:52980/0 (socket says 172.21.15.158:52980) 2023-10-16T09:59:09.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e8c05700 1 -- 172.21.15.158:0/1297403960 learned_addr learned my addr 172.21.15.158:0/1297403960 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:09.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e9406700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f89e40ce1d0 0x7f89e4161290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:55880/0 (socket says 172.21.15.158:55880) 2023-10-16T09:59:09.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e8c05700 1 -- 172.21.15.158:0/1297403960 >> v2:172.21.15.172:3300/0 conn(0x7f89e40cd810 msgr2=0x7f89e40cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:09.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e8c05700 1 --2- 172.21.15.158:0/1297403960 >> v2:172.21.15.172:3300/0 conn(0x7f89e40cd810 0x7f89e40cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e8c05700 1 -- 172.21.15.158:0/1297403960 >> v1:172.21.15.172:6789/0 conn(0x7f89e40ce1d0 legacy=0x7f89e4161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e8c05700 1 -- 172.21.15.158:0/1297403960 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e415dba0 con 0x7f89e415d7c0 2023-10-16T09:59:09.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e8c05700 1 --2- 172.21.15.158:0/1297403960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e415d7c0 0x7f89e406b2b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f89d80098f0 tx=0x7f89d8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baa484c6a383032a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:09.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e37fe700 1 -- 172.21.15.158:0/1297403960 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8011830 con 0x7f89e415d7c0 2023-10-16T09:59:09.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e37fe700 1 -- 172.21.15.158:0/1297403960 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d8011990 con 0x7f89e415d7c0 2023-10-16T09:59:09.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.467+0000 7f89e37fe700 1 -- 172.21.15.158:0/1297403960 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8011d20 con 0x7f89e415d7c0 2023-10-16T09:59:09.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 -- 172.21.15.158:0/1297403960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e415d7c0 msgr2=0x7f89e406b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 --2- 172.21.15.158:0/1297403960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e415d7c0 0x7f89e406b2b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f89d80098f0 tx=0x7f89d8009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:09.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 -- 172.21.15.158:0/1297403960 shutdown_connections 2023-10-16T09:59:09.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 --2- 172.21.15.158:0/1297403960 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e415d7c0 0x7f89e406b2b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 --2- 172.21.15.158:0/1297403960 >> v2:172.21.15.172:3300/0 conn(0x7f89e40cd810 0x7f89e40cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 -- 172.21.15.158:0/1297403960 >> 172.21.15.158:0/1297403960 conn(0x7f89e40cc4b0 msgr2=0x7f89e40cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:09.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 -- 172.21.15.158:0/1297403960 shutdown_connections 2023-10-16T09:59:09.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.468+0000 7f89eae69700 1 -- 172.21.15.158:0/1297403960 wait complete. 2023-10-16T09:59:09.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.469+0000 7f89eae69700 1 Processor -- start 2023-10-16T09:59:09.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.469+0000 7f89eae69700 1 -- start start 2023-10-16T09:59:09.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89eae69700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e40cd810 0x7f89e40c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89eae69700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89e40ce1d0 0x7f89e415b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89eae69700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89e415d7c0 0x7f89e40c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89eae69700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40c70e0 con 0x7f89e40ce1d0 2023-10-16T09:59:09.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89eae69700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40c73c0 con 0x7f89e415d7c0 2023-10-16T09:59:09.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89eae69700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40c76a0 con 0x7f89e40cd810 2023-10-16T09:59:09.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89e40ce1d0 0x7f89e415b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e8c05700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e40cd810 0x7f89e40c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89e40ce1d0 0x7f89e415b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51810/0 (socket says 172.21.15.158:51810) 2023-10-16T09:59:09.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 -- 172.21.15.158:0/2031725769 learned_addr learned my addr 172.21.15.158:0/2031725769 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:09.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e9406700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89e415d7c0 0x7f89e40c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 -- 172.21.15.158:0/2031725769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e40cd810 msgr2=0x7f89e40c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e40cd810 0x7f89e40c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 -- 172.21.15.158:0/2031725769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89e415d7c0 msgr2=0x7f89e40c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89e415d7c0 0x7f89e40c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 -- 172.21.15.158:0/2031725769 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d8009550 con 0x7f89e40ce1d0 2023-10-16T09:59:09.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.470+0000 7f89e3fff700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89e40ce1d0 0x7f89e415b620 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f89d000e870 tx=0x7f89d000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:09.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.471+0000 7f89e17fa700 1 -- 172.21.15.158:0/2031725769 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d0009590 con 0x7f89e40ce1d0 2023-10-16T09:59:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.471+0000 7f89e17fa700 1 -- 172.21.15.158:0/2031725769 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d0013680 con 0x7f89e40ce1d0 2023-10-16T09:59:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.471+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e40c7980 con 0x7f89e40ce1d0 2023-10-16T09:59:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.471+0000 7f89e17fa700 1 -- 172.21.15.158:0/2031725769 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d00139f0 con 0x7f89e40ce1d0 2023-10-16T09:59:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.471+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e415bb60 con 0x7f89e40ce1d0 2023-10-16T09:59:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.472+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e40ce9e0 con 0x7f89e40ce1d0 2023-10-16T09:59:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.472+0000 7f89e17fa700 1 -- 172.21.15.158:0/2031725769 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 29) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f89d0013b50 con 0x7f89e40ce1d0 2023-10-16T09:59:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.472+0000 7f89e17fa700 1 -- 172.21.15.158:0/2031725769 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7199+0+0 (secure 0 0 0) 0x7f89d0089570 con 0x7f89e40ce1d0 2023-10-16T09:59:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.477+0000 7f89e17fa700 1 -- 172.21.15.158:0/2031725769 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f89d00543b0 con 0x7f89e40ce1d0 2023-10-16T09:59:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.671+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89e40b2e80 con 0x7f89e40ce1d0 2023-10-16T09:59:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.672+0000 7f89e17fa700 1 -- 172.21.15.158:0/2031725769 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+18565 (secure 0 0 0) 0x7f89d00579d0 con 0x7f89e40ce1d0 2023-10-16T09:59:09.673 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:09.674 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":73,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:09.053596+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:57:01.416542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T09:57:01.962552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.28","mappings":[{"from":0,"to":6}]},{"pgid":"8.3d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2457697446":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/3433233254":"2023-10-17T09:59:09.053538+0000","172.21.15.158:6833/1799865685":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/2391232454":"2023-10-17T09:59:09.053538+0000","172.21.15.158:6832/1799865685":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/2552228936":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/3405184500":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3175913401":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/488409315":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2953589465":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3049701210":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/2437477336":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6833/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6832/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2185855121":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3427024404":"2023-10-17T09:57:16.707111+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:09.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89e40ce1d0 msgr2=0x7f89e415b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89e40ce1d0 0x7f89e415b620 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f89d000e870 tx=0x7f89d000eb80 comp rx=0 tx=0).stop 2023-10-16T09:59:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 shutdown_connections 2023-10-16T09:59:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89e415d7c0 0x7f89e40c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89e40cd810 0x7f89e40c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 --2- 172.21.15.158:0/2031725769 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89e40ce1d0 0x7f89e415b620 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 >> 172.21.15.158:0/2031725769 conn(0x7f89e40cc4b0 msgr2=0x7f89e40afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 shutdown_connections 2023-10-16T09:59:09.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.674+0000 7f89eae69700 1 -- 172.21.15.158:0/2031725769 wait complete. 2023-10-16T09:59:09.688 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T09:59:09.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.808+0000 7f1a22b0b700 1 Processor -- start 2023-10-16T09:59:09.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.808+0000 7f1a22b0b700 1 -- start start 2023-10-16T09:59:09.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.809+0000 7f1a22b0b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1a1c15b6e0 0x7f1a1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.809+0000 7f1a22b0b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1a1c15f4d0 0x7f1a1c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.809+0000 7f1a22b0b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c071af0 con 0x7f1a1c15bac0 2023-10-16T09:59:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.809+0000 7f1a22b0b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c0afb60 con 0x7f1a1c15b6e0 2023-10-16T09:59:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.809+0000 7f1a22b0b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c15fa40 con 0x7f1a1c15f4d0 2023-10-16T09:59:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.809+0000 7f1a208a7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1a1c15b6e0 0x7f1a1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.809+0000 7f1a208a7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1a1c15b6e0 0x7f1a1c06dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51812/0 (socket says 172.21.15.158:51812) 2023-10-16T09:59:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a208a7700 1 -- 172.21.15.158:0/3777424419 learned_addr learned my addr 172.21.15.158:0/3777424419 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a1bfff700 1 --2- 172.21.15.158:0/3777424419 >> v2:172.21.15.172:3300/0 conn(0x7f1a1c15f4d0 0x7f1a1c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a208a7700 1 -- 172.21.15.158:0/3777424419 >> v2:172.21.15.172:3300/0 conn(0x7f1a1c15f4d0 msgr2=0x7f1a1c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:09.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a208a7700 1 --2- 172.21.15.158:0/3777424419 >> v2:172.21.15.172:3300/0 conn(0x7f1a1c15f4d0 0x7f1a1c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a208a7700 1 -- 172.21.15.158:0/3777424419 >> v1:172.21.15.172:6789/0 conn(0x7f1a1c15bac0 legacy=0x7f1a1c071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a208a7700 1 -- 172.21.15.158:0/3777424419 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c15fd20 con 0x7f1a1c15b6e0 2023-10-16T09:59:09.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a208a7700 1 --2- 172.21.15.158:0/3777424419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c06dcd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c09e2d0 tx=0x7f1a10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31dfb6f6e54bfedb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:09.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a1b7fe700 1 -- 172.21.15.158:0/3777424419 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000b870 con 0x7f1a1c15b6e0 2023-10-16T09:59:09.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a1b7fe700 1 -- 172.21.15.158:0/3777424419 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1000b9d0 con 0x7f1a1c15b6e0 2023-10-16T09:59:09.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.810+0000 7f1a1b7fe700 1 -- 172.21.15.158:0/3777424419 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000bd60 con 0x7f1a1c15b6e0 2023-10-16T09:59:09.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 -- 172.21.15.158:0/3777424419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15b6e0 msgr2=0x7f1a1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/3777424419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c06dcd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c09e2d0 tx=0x7f1a10009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 -- 172.21.15.158:0/3777424419 shutdown_connections 2023-10-16T09:59:09.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/3777424419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c06dcd0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/3777424419 >> v2:172.21.15.172:3300/0 conn(0x7f1a1c15f4d0 0x7f1a1c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 -- 172.21.15.158:0/3777424419 >> 172.21.15.158:0/3777424419 conn(0x7f1a1c06b9e0 msgr2=0x7f1a1c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:09.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 -- 172.21.15.158:0/3777424419 shutdown_connections 2023-10-16T09:59:09.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.811+0000 7f1a22b0b700 1 -- 172.21.15.158:0/3777424419 wait complete. 2023-10-16T09:59:09.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.812+0000 7f1a22b0b700 1 Processor -- start 2023-10-16T09:59:09.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.812+0000 7f1a22b0b700 1 -- start start 2023-10-16T09:59:09.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.813+0000 7f1a22b0b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c1f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.813+0000 7f1a22b0b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15bac0 0x7f1a1c1f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.813+0000 7f1a22b0b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a1c15f4d0 0x7f1a1c0b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:09.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.813+0000 7f1a22b0b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c15ce10 con 0x7f1a1c15bac0 2023-10-16T09:59:09.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.813+0000 7f1a22b0b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c159450 con 0x7f1a1c15b6e0 2023-10-16T09:59:09.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.813+0000 7f1a22b0b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c1f2280 con 0x7f1a1c15f4d0 2023-10-16T09:59:09.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.813+0000 7f1a1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15bac0 0x7f1a1c1f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15bac0 0x7f1a1c1f1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51820/0 (socket says 172.21.15.158:51820) 2023-10-16T09:59:09.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 -- 172.21.15.158:0/4121161202 learned_addr learned my addr 172.21.15.158:0/4121161202 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:09.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a208a7700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c1f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a210a8700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a1c15f4d0 0x7f1a1c0b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:09.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 -- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a1c15f4d0 msgr2=0x7f1a1c0b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a1c15f4d0 0x7f1a1c0b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 -- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a1c15b6e0 msgr2=0x7f1a1c1f0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:09.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c1f0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:09.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 -- 172.21.15.158:0/4121161202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10009550 con 0x7f1a1c15bac0 2023-10-16T09:59:09.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a208a7700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c1f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:09.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a210a8700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a1c15f4d0 0x7f1a1c0b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:09.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a1bfff700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15bac0 0x7f1a1c1f1d40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800d800 tx=0x7f1a0800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:09.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a080125f0 con 0x7f1a1c15bac0 2023-10-16T09:59:09.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08012750 con 0x7f1a1c15bac0 2023-10-16T09:59:09.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c157da0 con 0x7f1a1c15bac0 2023-10-16T09:59:09.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a08012b00 con 0x7f1a1c15bac0 2023-10-16T09:59:09.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.814+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c158360 con 0x7f1a1c15bac0 2023-10-16T09:59:09.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.815+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c15cf50 con 0x7f1a1c15bac0 2023-10-16T09:59:09.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.816+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 29) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1a08012c60 con 0x7f1a1c15bac0 2023-10-16T09:59:09.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.816+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7199+0+0 (secure 0 0 0) 0x7f1a08089a10 con 0x7f1a1c15bac0 2023-10-16T09:59:09.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:09.820+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a08054940 con 0x7f1a1c15bac0 2023-10-16T09:59:10.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.057+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1a1c0681b0 con 0x7f1a1c15bac0 2023-10-16T09:59:10.058 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T09:59:10.057+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T09:59:10.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.065+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f1a080543f0 con 0x7f1a1c15bac0 2023-10-16T09:59:10.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.066+0000 7f1a197fa700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f1a00071250 0x7f1a00073640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.066+0000 7f1a208a7700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f1a00071250 0x7f1a00073640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.068+0000 7f1a208a7700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f1a00071250 0x7f1a00073640 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009130 tx=0x7f1a10008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:10.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.068+0000 7f1a197fa700 1 -- 172.21.15.158:0/4121161202 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1a08087a10 con 0x7f1a1c15bac0 2023-10-16T09:59:10.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f1a00071250 msgr2=0x7f1a00073640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:10.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f1a00071250 0x7f1a00073640 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009130 tx=0x7f1a10008030 comp rx=0 tx=0).stop 2023-10-16T09:59:10.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15bac0 msgr2=0x7f1a1c1f1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:10.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15bac0 0x7f1a1c1f1d40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800d800 tx=0x7f1a0800dbc0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 shutdown_connections 2023-10-16T09:59:10.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f1a00071250 0x7f1a00073640 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a1c15f4d0 0x7f1a1c0b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a1c15b6e0 0x7f1a1c1f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 --2- 172.21.15.158:0/4121161202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a1c15bac0 0x7f1a1c1f1d40 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 >> 172.21.15.158:0/4121161202 conn(0x7f1a1c06b9e0 msgr2=0x7f1a1c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:10.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 shutdown_connections 2023-10-16T09:59:10.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.071+0000 7f1a22b0b700 1 -- 172.21.15.158:0/4121161202 wait complete. 2023-10-16T09:59:10.076 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T09:59:10.085 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T09:59:10.110 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:59:10.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.208+0000 7efef36cf700 1 Processor -- start 2023-10-16T09:59:10.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.208+0000 7efef36cf700 1 -- start start 2023-10-16T09:59:10.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef36cf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec15da40 0x7efeec0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef36cf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efeec0cc590 0x7efeec06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef36cf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efeec06ae10 0x7efeec0ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef36cf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec06b280 con 0x7efeec0cc590 2023-10-16T09:59:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef36cf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0cbd30 con 0x7efeec06ae10 2023-10-16T09:59:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef36cf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0cc1c0 con 0x7efeec15da40 2023-10-16T09:59:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efeec0cc590 0x7efeec06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efeec0cc590 0x7efeec06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51848/0 (socket says 172.21.15.158:51848) 2023-10-16T09:59:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 -- 172.21.15.158:0/919575439 learned_addr learned my addr 172.21.15.158:0/919575439 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef146b700 1 --2- 172.21.15.158:0/919575439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec15da40 0x7efeec0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef1c6c700 1 --2- 172.21.15.158:0/919575439 >> v2:172.21.15.172:3300/0 conn(0x7efeec06ae10 0x7efeec0ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 -- 172.21.15.158:0/919575439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec15da40 msgr2=0x7efeec0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 --2- 172.21.15.158:0/919575439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec15da40 0x7efeec0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 -- 172.21.15.158:0/919575439 >> v2:172.21.15.172:3300/0 conn(0x7efeec06ae10 msgr2=0x7efeec0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 --2- 172.21.15.158:0/919575439 >> v2:172.21.15.172:3300/0 conn(0x7efeec06ae10 0x7efeec0ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 -- 172.21.15.158:0/919575439 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec15de20 con 0x7efeec0cc590 2023-10-16T09:59:10.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efef0c6a700 1 --2- 172.21.15.158:0/919575439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec0cc590 0x7efeec06a8d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7efeec0d1380 tx=0x7efedc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0ff85932b50cd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:10.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efee3fff700 1 -- 172.21.15.158:0/919575439 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc00b870 con 0x7efeec0cc590 2023-10-16T09:59:10.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efee3fff700 1 -- 172.21.15.158:0/919575439 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc00b9d0 con 0x7efeec0cc590 2023-10-16T09:59:10.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.209+0000 7efee3fff700 1 -- 172.21.15.158:0/919575439 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc00bd60 con 0x7efeec0cc590 2023-10-16T09:59:10.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 -- 172.21.15.158:0/919575439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec0cc590 msgr2=0x7efeec06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:10.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 --2- 172.21.15.158:0/919575439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec0cc590 0x7efeec06a8d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7efeec0d1380 tx=0x7efedc009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 -- 172.21.15.158:0/919575439 shutdown_connections 2023-10-16T09:59:10.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 --2- 172.21.15.158:0/919575439 >> v2:172.21.15.172:3300/0 conn(0x7efeec06ae10 0x7efeec0ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 --2- 172.21.15.158:0/919575439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec15da40 0x7efeec0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 --2- 172.21.15.158:0/919575439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec0cc590 0x7efeec06a8d0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 -- 172.21.15.158:0/919575439 >> 172.21.15.158:0/919575439 conn(0x7efeec099a10 msgr2=0x7efeec0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:10.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 -- 172.21.15.158:0/919575439 shutdown_connections 2023-10-16T09:59:10.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.210+0000 7efef36cf700 1 -- 172.21.15.158:0/919575439 wait complete. 2023-10-16T09:59:10.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.211+0000 7efef36cf700 1 Processor -- start 2023-10-16T09:59:10.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.211+0000 7efef36cf700 1 -- start start 2023-10-16T09:59:10.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef36cf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec06ae10 0x7efeec14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef36cf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeec0cc590 0x7efeec149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef36cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec15da40 0x7efeec1498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef36cf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec14a1e0 con 0x7efeec15da40 2023-10-16T09:59:10.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef36cf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec14a4c0 con 0x7efeec0cc590 2023-10-16T09:59:10.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef36cf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec14a7a0 con 0x7efeec06ae10 2023-10-16T09:59:10.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec15da40 0x7efeec1498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec15da40 0x7efeec1498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51858/0 (socket says 172.21.15.158:51858) 2023-10-16T09:59:10.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 -- 172.21.15.158:0/766702085 learned_addr learned my addr 172.21.15.158:0/766702085 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:10.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef146b700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec06ae10 0x7efeec14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef0c6a700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeec0cc590 0x7efeec149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 -- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec06ae10 msgr2=0x7efeec14b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:10.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec06ae10 0x7efeec14b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 -- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeec0cc590 msgr2=0x7efeec149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:10.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeec0cc590 0x7efeec149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:10.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.212+0000 7efef1c6c700 1 -- 172.21.15.158:0/766702085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc009550 con 0x7efeec15da40 2023-10-16T09:59:10.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.213+0000 7efef1c6c700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec15da40 0x7efeec1498b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7efee4007e60 tx=0x7efee400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:10.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.213+0000 7efee1ffb700 1 -- 172.21.15.158:0/766702085 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee400ed30 con 0x7efeec15da40 2023-10-16T09:59:10.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.213+0000 7efee1ffb700 1 -- 172.21.15.158:0/766702085 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee4014420 con 0x7efeec15da40 2023-10-16T09:59:10.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.213+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeec14aa80 con 0x7efeec15da40 2023-10-16T09:59:10.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.213+0000 7efee1ffb700 1 -- 172.21.15.158:0/766702085 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee40147d0 con 0x7efeec15da40 2023-10-16T09:59:10.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.213+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec149df0 con 0x7efeec15da40 2023-10-16T09:59:10.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.214+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec14afa0 con 0x7efeec15da40 2023-10-16T09:59:10.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.214+0000 7efee1ffb700 1 -- 172.21.15.158:0/766702085 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7efee4014930 con 0x7efeec15da40 2023-10-16T09:59:10.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.215+0000 7efee1ffb700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7efed006ad50 0x7efed006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:10.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.215+0000 7efee1ffb700 1 -- 172.21.15.158:0/766702085 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7285+0+0 (secure 0 0 0) 0x7efee4089e50 con 0x7efeec15da40 2023-10-16T09:59:10.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.215+0000 7efef146b700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7efed006ad50 0x7efed006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:10.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.215+0000 7efef146b700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7efed006ad50 0x7efed006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efeec15cfb0 tx=0x7efee8008040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:10.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.219+0000 7efee1ffb700 1 -- 172.21.15.158:0/766702085 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efeec14afa0 con 0x7efeec15da40 2023-10-16T09:59:10.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:10.459+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efeec14afa0 con 0x7efeec15da40 2023-10-16T09:59:11.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.073+0000 7efee1ffb700 1 -- 172.21.15.158:0/766702085 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7efeec14afa0 con 0x7efeec15da40 2023-10-16T09:59:11.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7efed006ad50 msgr2=0x7efed006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:11.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7efed006ad50 0x7efed006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efeec15cfb0 tx=0x7efee8008040 comp rx=0 tx=0).stop 2023-10-16T09:59:11.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec15da40 msgr2=0x7efeec1498b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:11.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec15da40 0x7efeec1498b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7efee4007e60 tx=0x7efee400e3c0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 shutdown_connections 2023-10-16T09:59:11.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7efed006ad50 0x7efed006d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efeec15da40 0x7efeec1498b0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeec06ae10 0x7efeec14b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 --2- 172.21.15.158:0/766702085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeec0cc590 0x7efeec149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 >> 172.21.15.158:0/766702085 conn(0x7efeec099a10 msgr2=0x7efeec0d0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:11.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 shutdown_connections 2023-10-16T09:59:11.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.075+0000 7efef36cf700 1 -- 172.21.15.158:0/766702085 wait complete. 2023-10-16T09:59:11.090 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T09:59:11.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.219+0000 7fd9cb7dc700 1 Processor -- start 2023-10-16T09:59:11.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.219+0000 7fd9cb7dc700 1 -- start start 2023-10-16T09:59:11.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.220+0000 7fd9cb7dc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:11.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.220+0000 7fd9cb7dc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd9c40cc590 0x7fd9c406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:11.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.220+0000 7fd9cb7dc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c40cbd30 con 0x7fd9c406ae10 2023-10-16T09:59:11.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.220+0000 7fd9cb7dc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c406b280 con 0x7fd9c40cc590 2023-10-16T09:59:11.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.220+0000 7fd9cb7dc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40cc1c0 con 0x7fd9c415da40 2023-10-16T09:59:11.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9578700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:11.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c8d77700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd9c40cc590 0x7fd9c406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:11.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c8d77700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd9c40cc590 0x7fd9c406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:53062/0 (socket says 172.21.15.158:53062) 2023-10-16T09:59:11.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9d79700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd9c406ae10 0x7fd9c40ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:55962/0 (socket says 172.21.15.158:55962) 2023-10-16T09:59:11.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9578700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:34482/0 (socket says 172.21.15.158:34482) 2023-10-16T09:59:11.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c8d77700 1 -- 172.21.15.158:0/2092287951 learned_addr learned my addr 172.21.15.158:0/2092287951 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:11.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9578700 1 -- 172.21.15.158:0/2092287951 >> v2:172.21.15.172:3300/0 conn(0x7fd9c40cc590 msgr2=0x7fd9c406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:11.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9578700 1 --2- 172.21.15.158:0/2092287951 >> v2:172.21.15.172:3300/0 conn(0x7fd9c40cc590 0x7fd9c406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9578700 1 -- 172.21.15.158:0/2092287951 >> v1:172.21.15.172:6789/0 conn(0x7fd9c406ae10 legacy=0x7fd9c40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:11.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9578700 1 -- 172.21.15.158:0/2092287951 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c415de20 con 0x7fd9c415da40 2023-10-16T09:59:11.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c8d77700 1 --2- 172.21.15.158:0/2092287951 >> v2:172.21.15.172:3300/0 conn(0x7fd9c40cc590 0x7fd9c406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:11.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9c9578700 1 --2- 172.21.15.158:0/2092287951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40caa20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009840 tx=0x7fd9b4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=987a138b8b4c70c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:11.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9bbfff700 1 -- 172.21.15.158:0/2092287951 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4011830 con 0x7fd9c415da40 2023-10-16T09:59:11.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9bbfff700 1 -- 172.21.15.158:0/2092287951 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b4011990 con 0x7fd9c415da40 2023-10-16T09:59:11.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.221+0000 7fd9bbfff700 1 -- 172.21.15.158:0/2092287951 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4011d20 con 0x7fd9c415da40 2023-10-16T09:59:11.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/2092287951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 msgr2=0x7fd9c40caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:11.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/2092287951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40caa20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009840 tx=0x7fd9b4009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:11.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/2092287951 shutdown_connections 2023-10-16T09:59:11.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/2092287951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40caa20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/2092287951 >> v2:172.21.15.172:3300/0 conn(0x7fd9c40cc590 0x7fd9c406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/2092287951 >> 172.21.15.158:0/2092287951 conn(0x7fd9c4099a10 msgr2=0x7fd9c40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:11.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/2092287951 shutdown_connections 2023-10-16T09:59:11.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.222+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/2092287951 wait complete. 2023-10-16T09:59:11.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.223+0000 7fd9cb7dc700 1 Processor -- start 2023-10-16T09:59:11.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.223+0000 7fd9cb7dc700 1 -- start start 2023-10-16T09:59:11.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9cb7dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c406ae10 0x7fd9c40d0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:11.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9cb7dc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9c40cc590 0x7fd9c40d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:11.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9cb7dc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:11.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9cb7dc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40c4520 con 0x7fd9c406ae10 2023-10-16T09:59:11.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9cb7dc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40c8440 con 0x7fd9c40cc590 2023-10-16T09:59:11.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9cb7dc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40c8160 con 0x7fd9c415da40 2023-10-16T09:59:11.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c406ae10 0x7fd9c40d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:11.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c406ae10 0x7fd9c40d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51886/0 (socket says 172.21.15.158:51886) 2023-10-16T09:59:11.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 -- 172.21.15.158:0/3831299102 learned_addr learned my addr 172.21.15.158:0/3831299102 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:11.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9d79700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:11.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c8d77700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9c40cc590 0x7fd9c40d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:11.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 -- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 msgr2=0x7fd9c40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:11.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 -- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9c40cc590 msgr2=0x7fd9c40d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:11.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9c40cc590 0x7fd9c40d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:11.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 -- 172.21.15.158:0/3831299102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0009550 con 0x7fd9c406ae10 2023-10-16T09:59:11.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.224+0000 7fd9c9578700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c406ae10 0x7fd9c40d0760 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4014040 tx=0x7fd9b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:11.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.225+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40036a0 con 0x7fd9c406ae10 2023-10-16T09:59:11.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.225+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b401f070 con 0x7fd9c406ae10 2023-10-16T09:59:11.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.225+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b40094a0 con 0x7fd9c406ae10 2023-10-16T09:59:11.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.225+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4003a00 con 0x7fd9c406ae10 2023-10-16T09:59:11.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.225+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c40c41b0 con 0x7fd9c406ae10 2023-10-16T09:59:11.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.226+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c40681b0 con 0x7fd9c406ae10 2023-10-16T09:59:11.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.226+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd9b4003b60 con 0x7fd9c406ae10 2023-10-16T09:59:11.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.227+0000 7fd9b9ffb700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd9a806ad50 0x7fd9a806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:11.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.227+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7285+0+0 (secure 0 0 0) 0x7fd9b4021070 con 0x7fd9c406ae10 2023-10-16T09:59:11.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.227+0000 7fd9c8d77700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd9a806ad50 0x7fd9a806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:11.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.227+0000 7fd9c8d77700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd9a806ad50 0x7fd9a806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40b3d70 tx=0x7fd9c00092c0 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:11.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.231+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9b4055df0 con 0x7fd9c406ae10 2023-10-16T09:59:11.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.423+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd9c40cd990 con 0x7fd9c406ae10 2023-10-16T09:59:11.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:11.634+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 31) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd9b40561c0 con 0x7fd9c406ae10 2023-10-16T09:59:12.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.078+0000 7fd9b9ffb700 1 -- 172.21.15.158:0/3831299102 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v75) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd9b408cd70 con 0x7fd9c406ae10 2023-10-16T09:59:12.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd9a806ad50 msgr2=0x7fd9a806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:12.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd9a806ad50 0x7fd9a806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40b3d70 tx=0x7fd9c00092c0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c406ae10 msgr2=0x7fd9c40d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c406ae10 0x7fd9c40d0760 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4014040 tx=0x7fd9b4004750 comp rx=0 tx=0).stop 2023-10-16T09:59:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 shutdown_connections 2023-10-16T09:59:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd9a806ad50 0x7fd9a806d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9c415da40 0x7fd9c40c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c406ae10 0x7fd9c40d0760 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 --2- 172.21.15.158:0/3831299102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9c40cc590 0x7fd9c40d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 >> 172.21.15.158:0/3831299102 conn(0x7fd9c4099a10 msgr2=0x7fd9c406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 shutdown_connections 2023-10-16T09:59:12.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.082+0000 7fd9cb7dc700 1 -- 172.21.15.158:0/3831299102 wait complete. 2023-10-16T09:59:12.086 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T09:59:12.098 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T09:59:12.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.227+0000 7f8b1dc97700 1 Processor -- start 2023-10-16T09:59:12.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.227+0000 7f8b1dc97700 1 -- start start 2023-10-16T09:59:12.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b1dc97700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b180b3630 0x7f8b18166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b1dc97700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b180b4420 con 0x7f8b180d0c60 2023-10-16T09:59:12.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b1dc97700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b180cbc20 con 0x7f8b1815b6b0 2023-10-16T09:59:12.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b1dc97700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1815ba90 con 0x7f8b180b3630 2023-10-16T09:59:12.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b16ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8b180d0c60 0x7f8b180b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52070/0 (socket says 172.21.15.158:52070) 2023-10-16T09:59:12.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b16ffd700 1 -- 172.21.15.158:0/750662512 learned_addr learned my addr 172.21.15.158:0/750662512 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:12.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523309802 0 0) 0x7f8b180b4420 con 0x7f8b180d0c60 2023-10-16T09:59:12.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b17fff700 1 --2- 172.21.15.158:0/750662512 >> v2:172.21.15.172:3300/0 conn(0x7f8b180b3630 0x7f8b18166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.228+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b040035b0 con 0x7f8b180d0c60 2023-10-16T09:59:12.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111384577 0 0) 0x7f8b040035b0 con 0x7f8b180d0c60 2023-10-16T09:59:12.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 >> v2:172.21.15.172:3300/0 conn(0x7f8b180b3630 msgr2=0x7f8b18166120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:12.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 --2- 172.21.15.158:0/750662512 >> v2:172.21.15.172:3300/0 conn(0x7f8b180b3630 0x7f8b18166120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 >> v1:172.21.15.172:6789/0 conn(0x7f8b1815b6b0 legacy=0x7f8b180cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:12.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b180cbf00 con 0x7f8b180d0c60 2023-10-16T09:59:12.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f8b00001670 con 0x7f8b180d0c60 2023-10-16T09:59:12.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 >> v1:172.21.15.158:6789/0 conn(0x7f8b180d0c60 legacy=0x7f8b180b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:12.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04003a00 0x7f8b04003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b04004850 0x7f8b04004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b04005210 0x7f8b0400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0400c180 con 0x7f8b04005210 2023-10-16T09:59:12.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0400c2f0 con 0x7f8b04003a00 2023-10-16T09:59:12.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0400c4b0 con 0x7f8b04004850 2023-10-16T09:59:12.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b04005210 0x7f8b0400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b177fe700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04003a00 0x7f8b04003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b16ffd700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b04004850 0x7f8b04004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 -- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b04004850 msgr2=0x7f8b04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b04004850 0x7f8b04004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 -- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04003a00 msgr2=0x7f8b04003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:12.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04003a00 0x7f8b04003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 -- 172.21.15.158:0/750662512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b040035b0 con 0x7f8b04005210 2023-10-16T09:59:12.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 -- 172.21.15.158:0/750662512 reap_dead start 2023-10-16T09:59:12.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b17fff700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b04005210 0x7f8b0400ba10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8b180c8520 tx=0x7f8b04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b4e958b7b8328 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:12.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0800e6c0 con 0x7f8b04005210 2023-10-16T09:59:12.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0800e820 con 0x7f8b04005210 2023-10-16T09:59:12.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.229+0000 7f8b167fc700 1 -- 172.21.15.158:0/750662512 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0800ebd0 con 0x7f8b04005210 2023-10-16T09:59:12.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.230+0000 7f8b1dc97700 1 -- 172.21.15.158:0/750662512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b04005210 msgr2=0x7f8b0400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:12.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.230+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b04005210 0x7f8b0400ba10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8b180c8520 tx=0x7f8b04005880 comp rx=0 tx=0).stop 2023-10-16T09:59:12.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.230+0000 7f8b1dc97700 1 -- 172.21.15.158:0/750662512 shutdown_connections 2023-10-16T09:59:12.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.230+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/750662512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b04005210 0x7f8b0400ba10 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.230+0000 7f8b1dc97700 1 -- 172.21.15.158:0/750662512 >> 172.21.15.158:0/750662512 conn(0x7f8b180cdf20 msgr2=0x7f8b180ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:12.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.230+0000 7f8b1dc97700 1 -- 172.21.15.158:0/750662512 shutdown_connections 2023-10-16T09:59:12.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.230+0000 7f8b1dc97700 1 -- 172.21.15.158:0/750662512 wait complete. 2023-10-16T09:59:12.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.231+0000 7f8b1dc97700 1 Processor -- start 2023-10-16T09:59:12.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.231+0000 7f8b1dc97700 1 -- start start 2023-10-16T09:59:12.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b1dc97700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04005210 0x7f8b1815d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b1dc97700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1806d730 0x7f8b181506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b1dc97700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b181547f0 0x7f8b18150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b1dc97700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181580d0 con 0x7f8b1806d730 2023-10-16T09:59:12.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b1dc97700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181551b0 con 0x7f8b04005210 2023-10-16T09:59:12.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b1dc97700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1806c700 con 0x7f8b181547f0 2023-10-16T09:59:12.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b16ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1806d730 0x7f8b181506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b16ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1806d730 0x7f8b181506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51908/0 (socket says 172.21.15.158:51908) 2023-10-16T09:59:12.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b16ffd700 1 -- 172.21.15.158:0/894692824 learned_addr learned my addr 172.21.15.158:0/894692824 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:12.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b17fff700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b181547f0 0x7f8b18150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.232+0000 7f8b177fe700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04005210 0x7f8b1815d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b16ffd700 1 -- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b181547f0 msgr2=0x7f8b18150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:12.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b16ffd700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b181547f0 0x7f8b18150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b16ffd700 1 -- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04005210 msgr2=0x7f8b1815d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:12.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b16ffd700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04005210 0x7f8b1815d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:12.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b16ffd700 1 -- 172.21.15.158:0/894692824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1806c9e0 con 0x7f8b1806d730 2023-10-16T09:59:12.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b177fe700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04005210 0x7f8b1815d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:12.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b16ffd700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1806d730 0x7f8b181506c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000eaf0 tx=0x7f8b0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:12.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b0ffff700 1 -- 172.21.15.158:0/894692824 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00005e00 con 0x7f8b1806d730 2023-10-16T09:59:12.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b0ffff700 1 -- 172.21.15.158:0/894692824 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0000c7b0 con 0x7f8b1806d730 2023-10-16T09:59:12.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1806ccc0 con 0x7f8b1806d730 2023-10-16T09:59:12.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b0ffff700 1 -- 172.21.15.158:0/894692824 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000cb60 con 0x7f8b1806d730 2023-10-16T09:59:12.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.233+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1806c3c0 con 0x7f8b1806d730 2023-10-16T09:59:12.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.234+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b181555e0 con 0x7f8b1806d730 2023-10-16T09:59:12.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.234+0000 7f8b0ffff700 1 -- 172.21.15.158:0/894692824 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 31) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8b0000ccc0 con 0x7f8b1806d730 2023-10-16T09:59:12.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.235+0000 7f8b0ffff700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8af806af90 0x7f8af806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:12.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.235+0000 7f8b0ffff700 1 -- 172.21.15.158:0/894692824 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f8b000897c0 con 0x7f8b1806d730 2023-10-16T09:59:12.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.235+0000 7f8b177fe700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8af806af90 0x7f8af806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:12.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.235+0000 7f8b177fe700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8af806af90 0x7f8af806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b10002230 tx=0x7f8b10009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:12.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.239+0000 7f8b0ffff700 1 -- 172.21.15.158:0/894692824 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b000544a0 con 0x7f8b1806d730 2023-10-16T09:59:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:12.430+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8b180681b0 con 0x7f8b1806d730 2023-10-16T09:59:13.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.080+0000 7f8b0ffff700 1 -- 172.21.15.158:0/894692824 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v76) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8b00057ac0 con 0x7f8b1806d730 2023-10-16T09:59:13.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.083+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8af806af90 msgr2=0x7f8af806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.083+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8af806af90 0x7f8af806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b10002230 tx=0x7f8b10009040 comp rx=0 tx=0).stop 2023-10-16T09:59:13.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.083+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1806d730 msgr2=0x7f8b181506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.083+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1806d730 0x7f8b181506c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000eaf0 tx=0x7f8b0000eeb0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.084+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 shutdown_connections 2023-10-16T09:59:13.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.084+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8af806af90 0x7f8af806d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.084+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b181547f0 0x7f8b18150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.084+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b04005210 0x7f8b1815d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.084+0000 7f8b1dc97700 1 --2- 172.21.15.158:0/894692824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1806d730 0x7f8b181506c0 secure :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000eaf0 tx=0x7f8b0000eeb0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.084+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 >> 172.21.15.158:0/894692824 conn(0x7f8b180cdf20 msgr2=0x7f8b1815aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:13.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.085+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 shutdown_connections 2023-10-16T09:59:13.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.085+0000 7f8b1dc97700 1 -- 172.21.15.158:0/894692824 wait complete. 2023-10-16T09:59:13.088 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T09:59:13.107 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T09:59:13.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.233+0000 7f9273743700 1 Processor -- start 2023-10-16T09:59:13.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.233+0000 7f9273743700 1 -- start start 2023-10-16T09:59:13.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9273743700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 0x7f926c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9273743700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c0cbd30 con 0x7f926c06afe0 2023-10-16T09:59:13.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9273743700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c0cc1c0 con 0x7f926c15da40 2023-10-16T09:59:13.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9273743700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c15de20 con 0x7f926c0cc590 2023-10-16T09:59:13.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9270cde700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f926c06afe0 0x7f926c0afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52090/0 (socket says 172.21.15.158:52090) 2023-10-16T09:59:13.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9270cde700 1 -- 172.21.15.158:0/2039371775 learned_addr learned my addr 172.21.15.158:0/2039371775 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:13.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9271ce0700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 0x7f926c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055888799 0 0) 0x7f926c0cbd30 con 0x7f926c06afe0 2023-10-16T09:59:13.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92580035b0 con 0x7f926c06afe0 2023-10-16T09:59:13.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3226321778 0 0) 0x7f92580035b0 con 0x7f926c06afe0 2023-10-16T09:59:13.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 msgr2=0x7f926c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:13.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9263fff700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 0x7f926c0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.234+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 >> v1:172.21.15.172:6789/0 conn(0x7f926c15da40 legacy=0x7f926c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0ce260 con 0x7f926c06afe0 2023-10-16T09:59:13.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f925c001670 con 0x7f926c06afe0 2023-10-16T09:59:13.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 >> v1:172.21.15.158:6789/0 conn(0x7f926c06afe0 legacy=0x7f926c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9258003a30 0x7f9258003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 0x7f9258004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9258004d90 0x7f9258005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c15de20 con 0x7f9258004d90 2023-10-16T09:59:13.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925800ba10 con 0x7f9258003a30 2023-10-16T09:59:13.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925800bbd0 con 0x7f926c0cc590 2023-10-16T09:59:13.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9271ce0700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9258004d90 0x7f9258005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f92714df700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9258003a30 0x7f9258003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9270cde700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 0x7f9258004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9271ce0700 1 -- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 msgr2=0x7f9258004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:13.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9271ce0700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 0x7f9258004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9271ce0700 1 -- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9258003a30 msgr2=0x7f9258003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:13.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9271ce0700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9258003a30 0x7f9258003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9271ce0700 1 -- 172.21.15.158:0/2039371775 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580035b0 con 0x7f9258004d90 2023-10-16T09:59:13.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9270cde700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0cc590 0x7f9258004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:13.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9271ce0700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9258004d90 0x7f9258005210 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9264009fc0 tx=0x7f9258005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b95daf3a85d8b13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926400ec30 con 0x7f9258004d90 2023-10-16T09:59:13.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926400f040 con 0x7f9258004d90 2023-10-16T09:59:13.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.235+0000 7f9263fff700 1 -- 172.21.15.158:0/2039371775 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264014480 con 0x7f9258004d90 2023-10-16T09:59:13.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.236+0000 7f9273743700 1 -- 172.21.15.158:0/2039371775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9258004d90 msgr2=0x7f9258005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.236+0000 7f9273743700 1 --2- 172.21.15.158:0/2039371775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9258004d90 0x7f9258005210 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9264009fc0 tx=0x7f9258005880 comp rx=0 tx=0).stop 2023-10-16T09:59:13.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.236+0000 7f9271ce0700 1 -- 172.21.15.158:0/2039371775 reap_dead start 2023-10-16T09:59:13.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.236+0000 7f9273743700 1 -- 172.21.15.158:0/2039371775 shutdown_connections 2023-10-16T09:59:13.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.236+0000 7f9273743700 1 -- 172.21.15.158:0/2039371775 >> 172.21.15.158:0/2039371775 conn(0x7f926c099a10 msgr2=0x7f926c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:13.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.236+0000 7f9273743700 1 -- 172.21.15.158:0/2039371775 shutdown_connections 2023-10-16T09:59:13.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.236+0000 7f9273743700 1 -- 172.21.15.158:0/2039371775 wait complete. 2023-10-16T09:59:13.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.237+0000 7f9273743700 1 Processor -- start 2023-10-16T09:59:13.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.237+0000 7f9273743700 1 -- start start 2023-10-16T09:59:13.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9273743700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0c9cc0 0x7f926c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9273743700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c0ca0a0 0x7f926c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9273743700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f926c0c4210 0x7f926c0c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9273743700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c5ac0 con 0x7f926c0ca0a0 2023-10-16T09:59:13.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9273743700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c4f40 con 0x7f926c0c4210 2023-10-16T09:59:13.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9273743700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c5220 con 0x7f926c0c9cc0 2023-10-16T09:59:13.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c0ca0a0 0x7f926c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c0ca0a0 0x7f926c15b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51926/0 (socket says 172.21.15.158:51926) 2023-10-16T09:59:13.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 -- 172.21.15.158:0/1220368974 learned_addr learned my addr 172.21.15.158:0/1220368974 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:13.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9271ce0700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f926c0c4210 0x7f926c0c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f92714df700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0c9cc0 0x7f926c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 -- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0c9cc0 msgr2=0x7f926c15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0c9cc0 0x7f926c15ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 -- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f926c0c4210 msgr2=0x7f926c0c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f926c0c4210 0x7f926c0c46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 -- 172.21.15.158:0/1220368974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0c5500 con 0x7f926c0ca0a0 2023-10-16T09:59:13.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9270cde700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c0ca0a0 0x7f926c15b2a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f925c00eaf0 tx=0x7f925c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9261ffb700 1 -- 172.21.15.158:0/1220368974 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c005e30 con 0x7f926c0ca0a0 2023-10-16T09:59:13.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.238+0000 7f9261ffb700 1 -- 172.21.15.158:0/1220368974 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c00c7e0 con 0x7f926c0ca0a0 2023-10-16T09:59:13.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.239+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c0c57e0 con 0x7f926c0ca0a0 2023-10-16T09:59:13.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.239+0000 7f9261ffb700 1 -- 172.21.15.158:0/1220368974 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c00cb70 con 0x7f926c0ca0a0 2023-10-16T09:59:13.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.239+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c14e4e0 con 0x7f926c0ca0a0 2023-10-16T09:59:13.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.240+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c0681b0 con 0x7f926c0ca0a0 2023-10-16T09:59:13.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.240+0000 7f9261ffb700 1 -- 172.21.15.158:0/1220368974 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 31) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f925c00ccd0 con 0x7f926c0ca0a0 2023-10-16T09:59:13.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.240+0000 7f9261ffb700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f925006af90 0x7f925006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.241+0000 7f9261ffb700 1 -- 172.21.15.158:0/1220368974 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f925c089650 con 0x7f926c0ca0a0 2023-10-16T09:59:13.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.241+0000 7f92714df700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f925006af90 0x7f925006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.241+0000 7f92714df700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f925006af90 0x7f925006d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9268000c00 tx=0x7f9268009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.245+0000 7f9261ffb700 1 -- 172.21.15.158:0/1220368974 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f925c054500 con 0x7f926c0ca0a0 2023-10-16T09:59:13.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.437+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f926c06aca0 con 0x7f926c0ca0a0 2023-10-16T09:59:13.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.440+0000 7f9261ffb700 1 -- 172.21.15.158:0/1220368974 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f925c057b20 con 0x7f926c0ca0a0 2023-10-16T09:59:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.442+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f925006af90 msgr2=0x7f925006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.442+0000 7f9273743700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f925006af90 0x7f925006d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9268000c00 tx=0x7f9268009040 comp rx=0 tx=0).stop 2023-10-16T09:59:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.442+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c0ca0a0 msgr2=0x7f926c15b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.442+0000 7f9273743700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c0ca0a0 0x7f926c15b2a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f925c00eaf0 tx=0x7f925c00ee00 comp rx=0 tx=0).stop 2023-10-16T09:59:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.442+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 shutdown_connections 2023-10-16T09:59:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.443+0000 7f9273743700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f925006af90 0x7f925006d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.443+0000 7f9273743700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f926c0c4210 0x7f926c0c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.443+0000 7f9273743700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f926c0c9cc0 0x7f926c15ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.443+0000 7f9273743700 1 --2- 172.21.15.158:0/1220368974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c0ca0a0 0x7f926c15b2a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.443+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 >> 172.21.15.158:0/1220368974 conn(0x7f926c099a10 msgr2=0x7f926c15b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.443+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 shutdown_connections 2023-10-16T09:59:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.443+0000 7f9273743700 1 -- 172.21.15.158:0/1220368974 wait complete. 2023-10-16T09:59:13.456 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T09:59:13.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.582+0000 7f6f2b0e1700 1 Processor -- start 2023-10-16T09:59:13.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.582+0000 7f6f2b0e1700 1 -- start start 2023-10-16T09:59:13.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f2b0e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2415dc40 0x7f6f2415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f2b0e1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6f2406d9c0 0x7f6f2406de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f2b0e1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2415f380 con 0x7f6f2406f180 2023-10-16T09:59:13.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f2b0e1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2406e380 con 0x7f6f2406d9c0 2023-10-16T09:59:13.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f2b0e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24071e70 con 0x7f6f2415dc40 2023-10-16T09:59:13.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f23fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6f2406d9c0 0x7f6f2406de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f2967e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6f2406f180 0x7f6f2406f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52112/0 (socket says 172.21.15.158:52112) 2023-10-16T09:59:13.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.583+0000 7f6f2967e700 1 -- 172.21.15.158:0/1631440360 learned_addr learned my addr 172.21.15.158:0/1631440360 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:13.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f23fff700 1 -- 172.21.15.158:0/1631440360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2415dc40 msgr2=0x7f6f2415ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:13.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f28e7d700 1 --2- 172.21.15.158:0/1631440360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2415dc40 0x7f6f2415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f237fe700 1 -- 172.21.15.158:0/1631440360 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028874026 0 0) 0x7f6f1c001490 con 0x7f6f2406f180 2023-10-16T09:59:13.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f23fff700 1 --2- 172.21.15.158:0/1631440360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2415dc40 0x7f6f2415ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f23fff700 1 -- 172.21.15.158:0/1631440360 >> v1:172.21.15.158:6789/0 conn(0x7f6f2406f180 legacy=0x7f6f2406f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f23fff700 1 -- 172.21.15.158:0/1631440360 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f24072150 con 0x7f6f2406d9c0 2023-10-16T09:59:13.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f23fff700 1 --2- 172.21.15.158:0/1631440360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2406d9c0 0x7f6f2406de40 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009840 tx=0x7f6f10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f4ef9bf96c7d7b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f237fe700 1 -- 172.21.15.158:0/1631440360 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10011830 con 0x7f6f2406d9c0 2023-10-16T09:59:13.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f237fe700 1 -- 172.21.15.158:0/1631440360 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f10011990 con 0x7f6f2406d9c0 2023-10-16T09:59:13.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f237fe700 1 -- 172.21.15.158:0/1631440360 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10011d20 con 0x7f6f2406d9c0 2023-10-16T09:59:13.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/1631440360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2406d9c0 msgr2=0x7f6f2406de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.584+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/1631440360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2406d9c0 0x7f6f2406de40 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009840 tx=0x7f6f10009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:13.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.585+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/1631440360 shutdown_connections 2023-10-16T09:59:13.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.585+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/1631440360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2415dc40 0x7f6f2415ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.585+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/1631440360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2406d9c0 0x7f6f2406de40 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.585+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/1631440360 >> 172.21.15.158:0/1631440360 conn(0x7f6f240d03b0 msgr2=0x7f6f240d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:13.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.585+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/1631440360 shutdown_connections 2023-10-16T09:59:13.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.585+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/1631440360 wait complete. 2023-10-16T09:59:13.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.586+0000 7f6f2b0e1700 1 Processor -- start 2023-10-16T09:59:13.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.586+0000 7f6f2b0e1700 1 -- start start 2023-10-16T09:59:13.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2406d9c0 0x7f6f24148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f2406f180 0x7f6f240b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2415dc40 0x7f6f240c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f240c8670 con 0x7f6f2415dc40 2023-10-16T09:59:13.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24148900 con 0x7f6f2406f180 2023-10-16T09:59:13.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f240c83a0 con 0x7f6f2406d9c0 2023-10-16T09:59:13.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2415dc40 0x7f6f240c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2415dc40 0x7f6f240c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51942/0 (socket says 172.21.15.158:51942) 2023-10-16T09:59:13.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 -- 172.21.15.158:0/584636214 learned_addr learned my addr 172.21.15.158:0/584636214 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:13.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 -- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2406d9c0 msgr2=0x7f6f24148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f28e7d700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2406d9c0 0x7f6f24148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2406d9c0 0x7f6f24148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f23fff700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f2406f180 0x7f6f240b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 -- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f2406f180 msgr2=0x7f6f240b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f2406f180 0x7f6f240b5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 -- 172.21.15.158:0/584636214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f100094a0 con 0x7f6f2415dc40 2023-10-16T09:59:13.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2967e700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2415dc40 0x7f6f240c7e60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00cca0 tx=0x7f6f1c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c00cea0 con 0x7f6f2415dc40 2023-10-16T09:59:13.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1c00ada0 con 0x7f6f2415dc40 2023-10-16T09:59:13.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f24147fd0 con 0x7f6f2415dc40 2023-10-16T09:59:13.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c007310 con 0x7f6f2415dc40 2023-10-16T09:59:13.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.587+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f24148590 con 0x7f6f2415dc40 2023-10-16T09:59:13.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.588+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f24148750 con 0x7f6f2415dc40 2023-10-16T09:59:13.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.589+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 31) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6f1c007470 con 0x7f6f2415dc40 2023-10-16T09:59:13.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.589+0000 7f6f217fa700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6f0806af90 0x7f6f0806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.589+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f6f1c0895f0 con 0x7f6f2415dc40 2023-10-16T09:59:13.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.589+0000 7f6f28e7d700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6f0806af90 0x7f6f0806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.590+0000 7f6f28e7d700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6f0806af90 0x7f6f0806d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f240cd150 tx=0x7f6f18008040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.594+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f1c054410 con 0x7f6f2415dc40 2023-10-16T09:59:13.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.649+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6f1c0547e0 con 0x7f6f2415dc40 2023-10-16T09:59:13.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.793+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6f2415f380 con 0x7f6f2415dc40 2023-10-16T09:59:13.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.796+0000 7f6f217fa700 1 -- 172.21.15.158:0/584636214 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6f2415f380 con 0x7f6f2415dc40 2023-10-16T09:59:13.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6f0806af90 msgr2=0x7f6f0806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6f0806af90 0x7f6f0806d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f240cd150 tx=0x7f6f18008040 comp rx=0 tx=0).stop 2023-10-16T09:59:13.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2415dc40 msgr2=0x7f6f240c7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2415dc40 0x7f6f240c7e60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00cca0 tx=0x7f6f1c00a340 comp rx=0 tx=0).stop 2023-10-16T09:59:13.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 shutdown_connections 2023-10-16T09:59:13.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6f0806af90 0x7f6f0806d450 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f2415dc40 0x7f6f240c7e60 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f2406d9c0 0x7f6f24148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 --2- 172.21.15.158:0/584636214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f2406f180 0x7f6f240b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 >> 172.21.15.158:0/584636214 conn(0x7f6f240d03b0 msgr2=0x7f6f240afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:13.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 shutdown_connections 2023-10-16T09:59:13.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.798+0000 7f6f2b0e1700 1 -- 172.21.15.158:0/584636214 wait complete. 2023-10-16T09:59:13.811 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T09:59:13.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.933+0000 7fc5e381f700 1 Processor -- start 2023-10-16T09:59:13.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.933+0000 7fc5e381f700 1 -- start start 2023-10-16T09:59:13.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5e381f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 0x7fc5dc071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5e381f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc06a8d0 con 0x7fc5dc0cc9e0 2023-10-16T09:59:13.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5e381f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc161e50 con 0x7fc5dc15d910 2023-10-16T09:59:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5e381f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc15dcf0 con 0x7fc5dc0ce680 2023-10-16T09:59:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5e1dbc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc5dc0cc9e0 0x7fc5dc071ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52126/0 (socket says 172.21.15.158:52126) 2023-10-16T09:59:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5e1dbc700 1 -- 172.21.15.158:0/3844151700 learned_addr learned my addr 172.21.15.158:0/3844151700 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5e0dba700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 0x7fc5dc071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355688859 0 0) 0x7fc5dc06a8d0 con 0x7fc5dc0cc9e0 2023-10-16T09:59:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c80035b0 con 0x7fc5dc0cc9e0 2023-10-16T09:59:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2189869645 0 0) 0x7fc5c80035b0 con 0x7fc5dc0cc9e0 2023-10-16T09:59:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 msgr2=0x7fc5dc071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 0x7fc5dc071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 >> v1:172.21.15.172:6789/0 conn(0x7fc5dc15d910 legacy=0x7fc5dc0cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc161520 con 0x7fc5dc0cc9e0 2023-10-16T09:59:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.934+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc5d40019f0 con 0x7fc5dc0cc9e0 2023-10-16T09:59:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 >> v1:172.21.15.158:6789/0 conn(0x7fc5dc0cc9e0 legacy=0x7fc5dc071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 0x7fc5c8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c8003f70 0x7fc5c8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5c8004d90 0x7fc5c8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc15dcf0 con 0x7fc5c8004d90 2023-10-16T09:59:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c800ba70 con 0x7fc5c8003f70 2023-10-16T09:59:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c800bc30 con 0x7fc5dc0ce680 2023-10-16T09:59:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e1dbc700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5c8004d90 0x7fc5c8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e0dba700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c8003f70 0x7fc5c8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e15bb700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 0x7fc5c8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e1dbc700 1 -- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 msgr2=0x7fc5c8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:13.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e1dbc700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ce680 0x7fc5c8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e1dbc700 1 -- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c8003f70 msgr2=0x7fc5c8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:13.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e1dbc700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c8003f70 0x7fc5c8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e1dbc700 1 -- 172.21.15.158:0/3844151700 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80035b0 con 0x7fc5c8004d90 2023-10-16T09:59:13.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5e1dbc700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5c8004d90 0x7fc5c8005210 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc06d4b0 tx=0x7fc5c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbeed7ed8b8bc11d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d400a2b0 con 0x7fc5c8004d90 2023-10-16T09:59:13.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d40072d0 con 0x7fc5c8004d90 2023-10-16T09:59:13.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.935+0000 7fc5d3fff700 1 -- 172.21.15.158:0/3844151700 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4007660 con 0x7fc5c8004d90 2023-10-16T09:59:13.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.936+0000 7fc5e381f700 1 -- 172.21.15.158:0/3844151700 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5c8004d90 msgr2=0x7fc5c8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.936+0000 7fc5e381f700 1 --2- 172.21.15.158:0/3844151700 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5c8004d90 0x7fc5c8005210 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc06d4b0 tx=0x7fc5c8005880 comp rx=0 tx=0).stop 2023-10-16T09:59:13.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.936+0000 7fc5e1dbc700 1 -- 172.21.15.158:0/3844151700 reap_dead start 2023-10-16T09:59:13.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.936+0000 7fc5e381f700 1 -- 172.21.15.158:0/3844151700 shutdown_connections 2023-10-16T09:59:13.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.936+0000 7fc5e381f700 1 -- 172.21.15.158:0/3844151700 >> 172.21.15.158:0/3844151700 conn(0x7fc5dc0b1e90 msgr2=0x7fc5dc0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:13.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.936+0000 7fc5e381f700 1 -- 172.21.15.158:0/3844151700 shutdown_connections 2023-10-16T09:59:13.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.936+0000 7fc5e381f700 1 -- 172.21.15.158:0/3844151700 wait complete. 2023-10-16T09:59:13.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.937+0000 7fc5e381f700 1 Processor -- start 2023-10-16T09:59:13.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.937+0000 7fc5e381f700 1 -- start start 2023-10-16T09:59:13.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e381f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5dc1f2a00 0x7fc5dc0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e381f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5dc1f2de0 0x7fc5dc06ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e381f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ca330 0x7fc5dc06b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e381f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc1606b0 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e381f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc0c71d0 con 0x7fc5dc1f2a00 2023-10-16T09:59:13.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e381f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc0c7610 con 0x7fc5dc0ca330 2023-10-16T09:59:13.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5dc1f2de0 0x7fc5dc06ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5dc1f2de0 0x7fc5dc06ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51960/0 (socket says 172.21.15.158:51960) 2023-10-16T09:59:13.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 -- 172.21.15.158:0/3244858275 learned_addr learned my addr 172.21.15.158:0/3244858275 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:13.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e15bb700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5dc1f2a00 0x7fc5dc0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e1dbc700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ca330 0x7fc5dc06b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 -- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ca330 msgr2=0x7fc5dc06b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ca330 0x7fc5dc06b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 -- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5dc1f2a00 msgr2=0x7fc5dc0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:13.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5dc1f2a00 0x7fc5dc0c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:13.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 -- 172.21.15.158:0/3244858275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc0c78f0 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e1dbc700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ca330 0x7fc5dc06b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:13.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.938+0000 7fc5e0dba700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5dc1f2de0 0x7fc5dc06ade0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc011aa0 tx=0x7fc5cc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.939+0000 7fc5d1ffb700 1 -- 172.21.15.158:0/3244858275 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00f980 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.939+0000 7fc5d1ffb700 1 -- 172.21.15.158:0/3244858275 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc00fae0 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.939+0000 7fc5d1ffb700 1 -- 172.21.15.158:0/3244858275 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00fe90 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.939+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc160950 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.939+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc0c90e0 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.940+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5dc0681b0 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.940+0000 7fc5d1ffb700 1 -- 172.21.15.158:0/3244858275 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc5cc014440 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.941+0000 7fc5d1ffb700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc5c006b020 0x7fc5c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:13.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.941+0000 7fc5d1ffb700 1 -- 172.21.15.158:0/3244858275 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fc5cc08a150 con 0x7fc5dc1f2de0 2023-10-16T09:59:13.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.941+0000 7fc5e15bb700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc5c006b020 0x7fc5c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:13.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.941+0000 7fc5e15bb700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc5c006b020 0x7fc5c006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8007fd0 tx=0x7fc5d8009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:13.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:13.945+0000 7fc5d1ffb700 1 -- 172.21.15.158:0/3244858275 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5cc054fa0 con 0x7fc5dc1f2de0 2023-10-16T09:59:14.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.141+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc5dc0ce430 con 0x7fc5dc1f2de0 2023-10-16T09:59:14.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.145+0000 7fc5d1ffb700 1 -- 172.21.15.158:0/3244858275 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc5cc0585c0 con 0x7fc5dc1f2de0 2023-10-16T09:59:14.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc5c006b020 msgr2=0x7fc5c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:14.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc5c006b020 0x7fc5c006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8007fd0 tx=0x7fc5d8009040 comp rx=0 tx=0).stop 2023-10-16T09:59:14.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5dc1f2de0 msgr2=0x7fc5dc06ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:14.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5dc1f2de0 0x7fc5dc06ade0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc011aa0 tx=0x7fc5cc011e60 comp rx=0 tx=0).stop 2023-10-16T09:59:14.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 shutdown_connections 2023-10-16T09:59:14.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc5c006b020 0x7fc5c006d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5dc0ca330 0x7fc5dc06b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5dc1f2a00 0x7fc5dc0c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 --2- 172.21.15.158:0/3244858275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5dc1f2de0 0x7fc5dc06ade0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 >> 172.21.15.158:0/3244858275 conn(0x7fc5dc0b1e90 msgr2=0x7fc5dc0afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:14.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 shutdown_connections 2023-10-16T09:59:14.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.147+0000 7fc5e381f700 1 -- 172.21.15.158:0/3244858275 wait complete. 2023-10-16T09:59:14.161 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T09:59:14.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.280+0000 7fc00a770700 1 Processor -- start 2023-10-16T09:59:14.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.280+0000 7fc00a770700 1 -- start start 2023-10-16T09:59:14.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc00a770700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc0040665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:14.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc00a770700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0040ce610 0x7fc0040ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:14.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc00a770700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0040cefe0 con 0x7fc0040cbf40 2023-10-16T09:59:14.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc00a770700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00415e480 con 0x7fc0040ce610 2023-10-16T09:59:14.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc00a770700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00415d800 con 0x7fc0040c65b0 2023-10-16T09:59:14.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc008d0d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0040ce610 0x7fc0040ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:14.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc008d0d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0040ce610 0x7fc0040ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51970/0 (socket says 172.21.15.158:51970) 2023-10-16T09:59:14.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.281+0000 7fc008d0d700 1 -- 172.21.15.158:0/1951033984 learned_addr learned my addr 172.21.15.158:0/1951033984 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:14.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc008d0d700 1 -- 172.21.15.158:0/1951033984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 msgr2=0x7fc0040665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:14.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc0037fe700 1 --2- 172.21.15.158:0/1951033984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc0040665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:14.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc008d0d700 1 --2- 172.21.15.158:0/1951033984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc0040665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc008d0d700 1 -- 172.21.15.158:0/1951033984 >> v1:172.21.15.172:6789/0 conn(0x7fc0040cbf40 legacy=0x7fc00406f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:14.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc008d0d700 1 -- 172.21.15.158:0/1951033984 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00415dae0 con 0x7fc0040ce610 2023-10-16T09:59:14.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc0037fe700 1 --2- 172.21.15.158:0/1951033984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc0040665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:14.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc008d0d700 1 --2- 172.21.15.158:0/1951033984 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040ceaa0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009e50 tx=0x7fbffc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a79f4e48f92145c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:14.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc002ffd700 1 -- 172.21.15.158:0/1951033984 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00e070 con 0x7fc0040ce610 2023-10-16T09:59:14.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc002ffd700 1 -- 172.21.15.158:0/1951033984 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc003a20 con 0x7fc0040ce610 2023-10-16T09:59:14.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc002ffd700 1 -- 172.21.15.158:0/1951033984 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc003dd0 con 0x7fc0040ce610 2023-10-16T09:59:14.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc00a770700 1 -- 172.21.15.158:0/1951033984 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 msgr2=0x7fc0040ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:14.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc00a770700 1 --2- 172.21.15.158:0/1951033984 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040ceaa0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009e50 tx=0x7fbffc00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:14.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc00a770700 1 -- 172.21.15.158:0/1951033984 shutdown_connections 2023-10-16T09:59:14.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc00a770700 1 --2- 172.21.15.158:0/1951033984 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040ceaa0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc00a770700 1 --2- 172.21.15.158:0/1951033984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc0040665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.282+0000 7fc00a770700 1 -- 172.21.15.158:0/1951033984 >> 172.21.15.158:0/1951033984 conn(0x7fc0040cc4b0 msgr2=0x7fc0040cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:14.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.283+0000 7fc00a770700 1 -- 172.21.15.158:0/1951033984 shutdown_connections 2023-10-16T09:59:14.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.283+0000 7fc00a770700 1 -- 172.21.15.158:0/1951033984 wait complete. 2023-10-16T09:59:14.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.284+0000 7fc00a770700 1 Processor -- start 2023-10-16T09:59:14.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.284+0000 7fc00a770700 1 -- start start 2023-10-16T09:59:14.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc00415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:14.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0040cbf40 0x7fc00413f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:14.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:14.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040b53d0 con 0x7fc0040ce610 2023-10-16T09:59:14.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040b47c0 con 0x7fc0040cbf40 2023-10-16T09:59:14.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040b4aa0 con 0x7fc0040c65b0 2023-10-16T09:59:14.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:14.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51976/0 (socket says 172.21.15.158:51976) 2023-10-16T09:59:14.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 -- 172.21.15.158:0/3350390977 learned_addr learned my addr 172.21.15.158:0/3350390977 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:14.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc0037fe700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0040cbf40 0x7fc00413f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:14.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc003fff700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc00415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:14.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 -- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 msgr2=0x7fc00415c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:14.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc00415c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 -- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0040cbf40 msgr2=0x7fc00413f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:14.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0040cbf40 0x7fc00413f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:14.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 -- 172.21.15.158:0/3350390977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc009b30 con 0x7fc0040ce610 2023-10-16T09:59:14.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc0037fe700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0040cbf40 0x7fc00413f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:14.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc008d0d700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040afa70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fbffc015040 tx=0x7fbffc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:14.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc000ff9700 1 -- 172.21.15.158:0/3350390977 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc0092e0 con 0x7fc0040ce610 2023-10-16T09:59:14.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc000ff9700 1 -- 172.21.15.158:0/3350390977 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc004240 con 0x7fc0040ce610 2023-10-16T09:59:14.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0040b4d80 con 0x7fc0040ce610 2023-10-16T09:59:14.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc000ff9700 1 -- 172.21.15.158:0/3350390977 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc0045f0 con 0x7fc0040ce610 2023-10-16T09:59:14.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.285+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0040b4450 con 0x7fc0040ce610 2023-10-16T09:59:14.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.286+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0040ac880 con 0x7fc0040ce610 2023-10-16T09:59:14.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.287+0000 7fc000ff9700 1 -- 172.21.15.158:0/3350390977 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbffc019070 con 0x7fc0040ce610 2023-10-16T09:59:14.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.287+0000 7fc000ff9700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fbff006b020 0x7fbff006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:14.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.287+0000 7fc000ff9700 1 -- 172.21.15.158:0/3350390977 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fbffc08aa10 con 0x7fc0040ce610 2023-10-16T09:59:14.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.287+0000 7fc003fff700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fbff006b020 0x7fbff006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:14.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.287+0000 7fc003fff700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fbff006b020 0x7fbff006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbff8007fd0 tx=0x7fbff8007ec0 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:14.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.291+0000 7fc000ff9700 1 -- 172.21.15.158:0/3350390977 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0040ac880 con 0x7fc0040ce610 2023-10-16T09:59:14.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:14.564+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fc0040ac880 con 0x7fc0040ce610 2023-10-16T09:59:15.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.082+0000 7fc000ff9700 1 -- 172.21.15.158:0/3350390977 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v572) v1 ==== 154+0+0 (secure 0 0 0) 0x7fc0040ac880 con 0x7fc0040ce610 2023-10-16T09:59:15.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fbff006b020 msgr2=0x7fbff006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fbff006b020 0x7fbff006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbff8007fd0 tx=0x7fbff8007ec0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 msgr2=0x7fc0040afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040afa70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fbffc015040 tx=0x7fbffc004ad0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 shutdown_connections 2023-10-16T09:59:15.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fbff006b020 0x7fbff006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0040ce610 0x7fc0040afa70 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0040c65b0 0x7fc00415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 --2- 172.21.15.158:0/3350390977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0040cbf40 0x7fc00413f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 >> 172.21.15.158:0/3350390977 conn(0x7fc0040cc4b0 msgr2=0x7fc0040b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:15.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 shutdown_connections 2023-10-16T09:59:15.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.085+0000 7fc00a770700 1 -- 172.21.15.158:0/3350390977 wait complete. 2023-10-16T09:59:15.099 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-10-16T09:59:15.100 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-10-16T09:59:15.100 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T09:59:15.101 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:15.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.224+0000 7f6e1eca3700 1 Processor -- start 2023-10-16T09:59:15.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.224+0000 7f6e1eca3700 1 -- start start 2023-10-16T09:59:15.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.224+0000 7f6e1eca3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e180cf4c0 0x7f6e18161270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.224+0000 7f6e1eca3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e180cf8a0 0x7f6e1806d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.224+0000 7f6e1eca3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1815de20 con 0x7f6e1806bd10 2023-10-16T09:59:15.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.224+0000 7f6e1eca3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e18162180 con 0x7f6e180cf8a0 2023-10-16T09:59:15.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.224+0000 7f6e1eca3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e181618e0 con 0x7f6e180cf4c0 2023-10-16T09:59:15.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e17fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e180cf8a0 0x7f6e1806d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e1d240700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6e1806bd10 0x7f6e1815e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52160/0 (socket says 172.21.15.158:52160) 2023-10-16T09:59:15.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e1d240700 1 -- 172.21.15.158:0/2742307190 learned_addr learned my addr 172.21.15.158:0/2742307190 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:15.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e1ca3f700 1 --2- 172.21.15.158:0/2742307190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e180cf4c0 0x7f6e18161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e17fff700 1 -- 172.21.15.158:0/2742307190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e180cf4c0 msgr2=0x7f6e18161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:15.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e17fff700 1 --2- 172.21.15.158:0/2742307190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e180cf4c0 0x7f6e18161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e17fff700 1 -- 172.21.15.158:0/2742307190 >> v1:172.21.15.158:6789/0 conn(0x7f6e1806bd10 legacy=0x7f6e1815e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e177fe700 1 -- 172.21.15.158:0/2742307190 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025199920 0 0) 0x7f6e1815de20 con 0x7f6e1806bd10 2023-10-16T09:59:15.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e17fff700 1 -- 172.21.15.158:0/2742307190 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18161bc0 con 0x7f6e180cf8a0 2023-10-16T09:59:15.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e1ca3f700 1 --2- 172.21.15.158:0/2742307190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e180cf4c0 0x7f6e18161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:15.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e17fff700 1 --2- 172.21.15.158:0/2742307190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf8a0 0x7f6e1806d460 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009840 tx=0x7f6e04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e225c418b246f2b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e177fe700 1 -- 172.21.15.158:0/2742307190 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04011830 con 0x7f6e180cf8a0 2023-10-16T09:59:15.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e177fe700 1 -- 172.21.15.158:0/2742307190 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e04011990 con 0x7f6e180cf8a0 2023-10-16T09:59:15.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.225+0000 7f6e177fe700 1 -- 172.21.15.158:0/2742307190 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04011d20 con 0x7f6e180cf8a0 2023-10-16T09:59:15.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 -- 172.21.15.158:0/2742307190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf8a0 msgr2=0x7f6e1806d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/2742307190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf8a0 0x7f6e1806d460 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009840 tx=0x7f6e04009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:15.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 -- 172.21.15.158:0/2742307190 shutdown_connections 2023-10-16T09:59:15.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/2742307190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e180cf4c0 0x7f6e18161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/2742307190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf8a0 0x7f6e1806d460 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 -- 172.21.15.158:0/2742307190 >> 172.21.15.158:0/2742307190 conn(0x7f6e1806a8d0 msgr2=0x7f6e1806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:15.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 -- 172.21.15.158:0/2742307190 shutdown_connections 2023-10-16T09:59:15.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.226+0000 7f6e1eca3700 1 -- 172.21.15.158:0/2742307190 wait complete. 2023-10-16T09:59:15.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.227+0000 7f6e1eca3700 1 Processor -- start 2023-10-16T09:59:15.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.227+0000 7f6e1eca3700 1 -- start start 2023-10-16T09:59:15.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1eca3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e1806bd10 0x7f6e181f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1eca3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf4c0 0x7f6e180c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1eca3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e180cf8a0 0x7f6e181606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1eca3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180c88c0 con 0x7f6e180cf4c0 2023-10-16T09:59:15.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1eca3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180c8130 con 0x7f6e180cf8a0 2023-10-16T09:59:15.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1eca3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e18160c20 con 0x7f6e1806bd10 2023-10-16T09:59:15.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf4c0 0x7f6e180c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf4c0 0x7f6e180c7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51988/0 (socket says 172.21.15.158:51988) 2023-10-16T09:59:15.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 -- 172.21.15.158:0/1268927121 learned_addr learned my addr 172.21.15.158:0/1268927121 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:15.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1ca3f700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e1806bd10 0x7f6e181f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e1d240700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e180cf8a0 0x7f6e181606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 -- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e1806bd10 msgr2=0x7f6e181f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e1806bd10 0x7f6e181f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 -- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e180cf8a0 msgr2=0x7f6e181606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e180cf8a0 0x7f6e181606e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 -- 172.21.15.158:0/1268927121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e040094a0 con 0x7f6e180cf4c0 2023-10-16T09:59:15.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.228+0000 7f6e17fff700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf4c0 0x7f6e180c7bf0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6e040090b0 tx=0x7f6e04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.229+0000 7f6e157fa700 1 -- 172.21.15.158:0/1268927121 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e040038a0 con 0x7f6e180cf4c0 2023-10-16T09:59:15.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.229+0000 7f6e157fa700 1 -- 172.21.15.158:0/1268927121 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0401f070 con 0x7f6e180cf4c0 2023-10-16T09:59:15.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.229+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e181f2fc0 con 0x7f6e180cf4c0 2023-10-16T09:59:15.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.229+0000 7f6e157fa700 1 -- 172.21.15.158:0/1268927121 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04003c60 con 0x7f6e180cf4c0 2023-10-16T09:59:15.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.229+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e181f3580 con 0x7f6e180cf4c0 2023-10-16T09:59:15.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.230+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1806d920 con 0x7f6e180cf4c0 2023-10-16T09:59:15.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.230+0000 7f6e157fa700 1 -- 172.21.15.158:0/1268927121 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e0401a430 con 0x7f6e180cf4c0 2023-10-16T09:59:15.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.230+0000 7f6e157fa700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6dfc06b020 0x7f6dfc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.230+0000 7f6e157fa700 1 -- 172.21.15.158:0/1268927121 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f6e04021070 con 0x7f6e180cf4c0 2023-10-16T09:59:15.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.230+0000 7f6e1ca3f700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6dfc06b020 0x7f6dfc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.231+0000 7f6e1ca3f700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6dfc06b020 0x7f6dfc06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c006fd0 tx=0x7f6e0c008040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.235+0000 7f6e157fa700 1 -- 172.21.15.158:0/1268927121 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e04056190 con 0x7f6e180cf4c0 2023-10-16T09:59:15.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.430+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e180681b0 con 0x7f6e180cf4c0 2023-10-16T09:59:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.431+0000 7f6e157fa700 1 -- 172.21.15.158:0/1268927121 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+14867 (secure 0 0 0) 0x7f6e040597b0 con 0x7f6e180cf4c0 2023-10-16T09:59:15.433 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:15.433 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":76,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:13.079199+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2457697446":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/3433233254":"2023-10-17T09:59:09.053538+0000","172.21.15.158:6833/1799865685":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/2391232454":"2023-10-17T09:59:09.053538+0000","172.21.15.158:6832/1799865685":"2023-10-17T09:59:09.053538+0000","172.21.15.172:6833/1262687320":"2023-10-17T09:59:10.058405+0000","172.21.15.158:0/2552228936":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/3405184500":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3175913401":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/488409315":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2953589465":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3049701210":"2023-10-17T09:59:09.053538+0000","172.21.15.158:0/2437477336":"2023-10-17T09:57:16.707111+0000","172.21.15.158:6833/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.172:6835/1262687320":"2023-10-17T09:59:10.058405+0000","172.21.15.158:6832/211595975":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/2185855121":"2023-10-17T09:57:16.707111+0000","172.21.15.158:0/3427024404":"2023-10-17T09:57:16.707111+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:15.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6dfc06b020 msgr2=0x7f6dfc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6dfc06b020 0x7f6dfc06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c006fd0 tx=0x7f6e0c008040 comp rx=0 tx=0).stop 2023-10-16T09:59:15.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf4c0 msgr2=0x7f6e180c7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf4c0 0x7f6e180c7bf0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6e040090b0 tx=0x7f6e04004750 comp rx=0 tx=0).stop 2023-10-16T09:59:15.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 shutdown_connections 2023-10-16T09:59:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6dfc06b020 0x7f6dfc06d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e180cf8a0 0x7f6e181606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e1806bd10 0x7f6e181f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 --2- 172.21.15.158:0/1268927121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180cf4c0 0x7f6e180c7bf0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 >> 172.21.15.158:0/1268927121 conn(0x7f6e1806a8d0 msgr2=0x7f6e1815ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 shutdown_connections 2023-10-16T09:59:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.433+0000 7f6e1eca3700 1 -- 172.21.15.158:0/1268927121 wait complete. 2023-10-16T09:59:15.447 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:15.448 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:15.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.569+0000 7fc48c763700 1 Processor -- start 2023-10-16T09:59:15.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.569+0000 7fc48c763700 1 -- start start 2023-10-16T09:59:15.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.570+0000 7fc48c763700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 0x7fc4840caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.570+0000 7fc48c763700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4840cbd30 con 0x7fc48415da40 2023-10-16T09:59:15.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.570+0000 7fc48c763700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4840cc1c0 con 0x7fc4840cc590 2023-10-16T09:59:15.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.570+0000 7fc48c763700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48415de20 con 0x7fc48406afe0 2023-10-16T09:59:15.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.570+0000 7fc48a4ff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc48415da40 0x7fc48406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52170/0 (socket says 172.21.15.158:52170) 2023-10-16T09:59:15.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.570+0000 7fc48a4ff700 1 -- 172.21.15.158:0/2002169690 learned_addr learned my addr 172.21.15.158:0/2002169690 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:15.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc489cfe700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 0x7fc4840caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760191026 0 0) 0x7fc4840cbd30 con 0x7fc48415da40 2023-10-16T09:59:15.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4700035b0 con 0x7fc48415da40 2023-10-16T09:59:15.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309727939 0 0) 0x7fc4840cc1c0 con 0x7fc4840cc590 2023-10-16T09:59:15.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4840cbd30 con 0x7fc4840cc590 2023-10-16T09:59:15.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291337522 0 0) 0x7fc4700035b0 con 0x7fc48415da40 2023-10-16T09:59:15.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 msgr2=0x7fc4840caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:15.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 0x7fc4840caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 >> v1:172.21.15.172:6789/0 conn(0x7fc4840cc590 legacy=0x7fc4840afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4840ce260 con 0x7fc48415da40 2023-10-16T09:59:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc474001670 con 0x7fc48415da40 2023-10-16T09:59:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 >> v1:172.21.15.158:6789/0 conn(0x7fc48415da40 legacy=0x7fc48406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470003bc0 0x7fc470003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc470004850 0x7fc470004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 0x7fc470005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48415de20 con 0x7fc470004850 2023-10-16T09:59:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47000baf0 con 0x7fc470003bc0 2023-10-16T09:59:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47000bcb0 con 0x7fc48406afe0 2023-10-16T09:59:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc489cfe700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc470004850 0x7fc470004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc48a4ff700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470003bc0 0x7fc470003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc48ad00700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 0x7fc470005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc489cfe700 1 -- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 msgr2=0x7fc470005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc489cfe700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48406afe0 0x7fc470005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.571+0000 7fc489cfe700 1 -- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470003bc0 msgr2=0x7fc470003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc489cfe700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470003bc0 0x7fc470003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc489cfe700 1 -- 172.21.15.158:0/2002169690 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700035b0 con 0x7fc470004850 2023-10-16T09:59:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc489cfe700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc470004850 0x7fc470004cd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc480009b30 tx=0x7fc470005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa7f412f874a564 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000e6c0 con 0x7fc470004850 2023-10-16T09:59:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48000e820 con 0x7fc470004850 2023-10-16T09:59:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc4894fd700 1 -- 172.21.15.158:0/2002169690 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000eb90 con 0x7fc470004850 2023-10-16T09:59:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc48c763700 1 -- 172.21.15.158:0/2002169690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc470004850 msgr2=0x7fc470004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc48c763700 1 --2- 172.21.15.158:0/2002169690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc470004850 0x7fc470004cd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc480009b30 tx=0x7fc470005880 comp rx=0 tx=0).stop 2023-10-16T09:59:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc48ad00700 1 -- 172.21.15.158:0/2002169690 reap_dead start 2023-10-16T09:59:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc48c763700 1 -- 172.21.15.158:0/2002169690 shutdown_connections 2023-10-16T09:59:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc48c763700 1 -- 172.21.15.158:0/2002169690 >> 172.21.15.158:0/2002169690 conn(0x7fc484099a10 msgr2=0x7fc4840d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc48c763700 1 -- 172.21.15.158:0/2002169690 shutdown_connections 2023-10-16T09:59:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.572+0000 7fc48c763700 1 -- 172.21.15.158:0/2002169690 wait complete. 2023-10-16T09:59:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 Processor -- start 2023-10-16T09:59:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 -- start start 2023-10-16T09:59:15.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470004850 0x7fc4840c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4840c9ea0 0x7fc4840c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4841558c0 0x7fc4841580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4840ca680 con 0x7fc4840c9ea0 2023-10-16T09:59:15.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4840c8fc0 con 0x7fc470004850 2023-10-16T09:59:15.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc48c763700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4840c7fe0 con 0x7fc4841558c0 2023-10-16T09:59:15.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.574+0000 7fc489cfe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4840c9ea0 0x7fc4840c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4840c9ea0 0x7fc4840c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52008/0 (socket says 172.21.15.158:52008) 2023-10-16T09:59:15.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 -- 172.21.15.158:0/3081131868 learned_addr learned my addr 172.21.15.158:0/3081131868 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:15.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc48a4ff700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470004850 0x7fc4840c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc48ad00700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4841558c0 0x7fc4841580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 -- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4841558c0 msgr2=0x7fc4841580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4841558c0 0x7fc4841580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 -- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470004850 msgr2=0x7fc4840c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470004850 0x7fc4840c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 -- 172.21.15.158:0/3081131868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4840ca3a0 con 0x7fc4840c9ea0 2023-10-16T09:59:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc489cfe700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4840c9ea0 0x7fc4840c96b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc47000bac0 tx=0x7fc4800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc47affd700 1 -- 172.21.15.158:0/3081131868 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480021070 con 0x7fc4840c9ea0 2023-10-16T09:59:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc47affd700 1 -- 172.21.15.158:0/3081131868 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc480004520 con 0x7fc4840c9ea0 2023-10-16T09:59:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc47affd700 1 -- 172.21.15.158:0/3081131868 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000e6c0 con 0x7fc4840c9ea0 2023-10-16T09:59:15.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4840c8280 con 0x7fc4840c9ea0 2023-10-16T09:59:15.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.575+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4840c9bf0 con 0x7fc4840c9ea0 2023-10-16T09:59:15.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.576+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4840ced90 con 0x7fc4840c9ea0 2023-10-16T09:59:15.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.577+0000 7fc47affd700 1 -- 172.21.15.158:0/3081131868 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc480025070 con 0x7fc4840c9ea0 2023-10-16T09:59:15.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.577+0000 7fc47affd700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc46806b020 0x7fc46806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.577+0000 7fc47affd700 1 -- 172.21.15.158:0/3081131868 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fc48008a960 con 0x7fc4840c9ea0 2023-10-16T09:59:15.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.577+0000 7fc48a4ff700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc46806b020 0x7fc46806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.578+0000 7fc48a4ff700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc46806b020 0x7fc46806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc474001050 tx=0x7fc474001e20 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.582+0000 7fc47affd700 1 -- 172.21.15.158:0/3081131868 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4800556f0 con 0x7fc4840c9ea0 2023-10-16T09:59:15.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.773+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4840c88d0 con 0x7fc4840c9ea0 2023-10-16T09:59:15.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.774+0000 7fc47affd700 1 -- 172.21.15.158:0/3081131868 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc48002c0a0 con 0x7fc4840c9ea0 2023-10-16T09:59:15.775 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:15.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc46806b020 msgr2=0x7fc46806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc46806b020 0x7fc46806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc474001050 tx=0x7fc474001e20 comp rx=0 tx=0).stop 2023-10-16T09:59:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4840c9ea0 msgr2=0x7fc4840c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4840c9ea0 0x7fc4840c96b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc47000bac0 tx=0x7fc4800047c0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 shutdown_connections 2023-10-16T09:59:15.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc46806b020 0x7fc46806d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4841558c0 0x7fc4841580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc470004850 0x7fc4840c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 --2- 172.21.15.158:0/3081131868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4840c9ea0 0x7fc4840c96b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 >> 172.21.15.158:0/3081131868 conn(0x7fc484099a10 msgr2=0x7fc4840c4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 shutdown_connections 2023-10-16T09:59:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.776+0000 7fc48c763700 1 -- 172.21.15.158:0/3081131868 wait complete. 2023-10-16T09:59:15.789 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T09:59:15.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.909+0000 7f2d396be700 1 Processor -- start 2023-10-16T09:59:15.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.909+0000 7f2d396be700 1 -- start start 2023-10-16T09:59:15.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.909+0000 7f2d396be700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d3415d7c0 0x7f2d340d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.909+0000 7f2d396be700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d340d1210 0x7f2d340ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.909+0000 7f2d396be700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340d3bd0 con 0x7f2d340ce950 2023-10-16T09:59:15.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.909+0000 7f2d396be700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d340d4060 con 0x7f2d340d1210 2023-10-16T09:59:15.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.909+0000 7f2d396be700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340cede0 con 0x7f2d3415d7c0 2023-10-16T09:59:15.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d327fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d340d1210 0x7f2d340ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d3415d7c0 0x7f2d340d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d337fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2d340ce950 0x7f2d340b50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:56096/0 (socket says 172.21.15.158:56096) 2023-10-16T09:59:15.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d3415d7c0 0x7f2d340d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:34616/0 (socket says 172.21.15.158:34616) 2023-10-16T09:59:15.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 -- 172.21.15.158:0/2092380614 learned_addr learned my addr 172.21.15.158:0/2092380614 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:15.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d327fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d340d1210 0x7f2d340ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:53198/0 (socket says 172.21.15.158:53198) 2023-10-16T09:59:15.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d31ffb700 1 -- 172.21.15.158:0/2092380614 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808836079 0 0) 0x7f2d340d3bd0 con 0x7f2d340ce950 2023-10-16T09:59:15.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d31ffb700 1 -- 172.21.15.158:0/2092380614 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d1c0035b0 con 0x7f2d340ce950 2023-10-16T09:59:15.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 -- 172.21.15.158:0/2092380614 >> v2:172.21.15.172:3300/0 conn(0x7f2d340d1210 msgr2=0x7f2d340ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:15.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 --2- 172.21.15.158:0/2092380614 >> v2:172.21.15.172:3300/0 conn(0x7f2d340d1210 0x7f2d340ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 -- 172.21.15.158:0/2092380614 >> v1:172.21.15.172:6789/0 conn(0x7f2d340ce950 legacy=0x7f2d340b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 -- 172.21.15.158:0/2092380614 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d340d2cf0 con 0x7f2d3415d7c0 2023-10-16T09:59:15.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d327fc700 1 --2- 172.21.15.158:0/2092380614 >> v2:172.21.15.172:3300/0 conn(0x7f2d340d1210 0x7f2d340ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:15.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d32ffd700 1 --2- 172.21.15.158:0/2092380614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d3415d7c0 0x7f2d340d27b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2d280098f0 tx=0x7f2d28009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f108d4cd317905d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d31ffb700 1 -- 172.21.15.158:0/2092380614 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28011830 con 0x7f2d3415d7c0 2023-10-16T09:59:15.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d31ffb700 1 -- 172.21.15.158:0/2092380614 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d28011990 con 0x7f2d3415d7c0 2023-10-16T09:59:15.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.910+0000 7f2d31ffb700 1 -- 172.21.15.158:0/2092380614 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28011d20 con 0x7f2d3415d7c0 2023-10-16T09:59:15.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 -- 172.21.15.158:0/2092380614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d3415d7c0 msgr2=0x7f2d340d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 --2- 172.21.15.158:0/2092380614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d3415d7c0 0x7f2d340d27b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2d280098f0 tx=0x7f2d28009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:15.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 -- 172.21.15.158:0/2092380614 shutdown_connections 2023-10-16T09:59:15.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 --2- 172.21.15.158:0/2092380614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d3415d7c0 0x7f2d340d27b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 --2- 172.21.15.158:0/2092380614 >> v2:172.21.15.172:3300/0 conn(0x7f2d340d1210 0x7f2d340ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 -- 172.21.15.158:0/2092380614 >> 172.21.15.158:0/2092380614 conn(0x7f2d340669f0 msgr2=0x7f2d34066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:15.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 -- 172.21.15.158:0/2092380614 shutdown_connections 2023-10-16T09:59:15.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.911+0000 7f2d396be700 1 -- 172.21.15.158:0/2092380614 wait complete. 2023-10-16T09:59:15.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.912+0000 7f2d396be700 1 Processor -- start 2023-10-16T09:59:15.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.912+0000 7f2d396be700 1 -- start start 2023-10-16T09:59:15.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d396be700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d340ce950 0x7f2d34159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d396be700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340d1210 0x7f2d34153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d396be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d3415d7c0 0x7f2d340c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d396be700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34158d60 con 0x7f2d3415d7c0 2023-10-16T09:59:15.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d396be700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341537c0 con 0x7f2d340ce950 2023-10-16T09:59:15.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d396be700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34158400 con 0x7f2d340d1210 2023-10-16T09:59:15.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d3415d7c0 0x7f2d340c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d3415d7c0 0x7f2d340c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52024/0 (socket says 172.21.15.158:52024) 2023-10-16T09:59:15.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d337fe700 1 -- 172.21.15.158:0/896868507 learned_addr learned my addr 172.21.15.158:0/896868507 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:15.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d32ffd700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d340ce950 0x7f2d34159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.913+0000 7f2d327fc700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340d1210 0x7f2d34153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d337fe700 1 -- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340d1210 msgr2=0x7f2d34153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d337fe700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340d1210 0x7f2d34153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d337fe700 1 -- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d340ce950 msgr2=0x7f2d34159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:15.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d337fe700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d340ce950 0x7f2d34159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:15.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d337fe700 1 -- 172.21.15.158:0/896868507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d240094a0 con 0x7f2d3415d7c0 2023-10-16T09:59:15.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d337fe700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d3415d7c0 0x7f2d340c62e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00cbc0 tx=0x7f2d2c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d1b7fe700 1 -- 172.21.15.158:0/896868507 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00a9e0 con 0x7f2d3415d7c0 2023-10-16T09:59:15.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d1b7fe700 1 -- 172.21.15.158:0/896868507 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c00ab40 con 0x7f2d3415d7c0 2023-10-16T09:59:15.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d28009550 con 0x7f2d3415d7c0 2023-10-16T09:59:15.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d1b7fe700 1 -- 172.21.15.158:0/896868507 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c012400 con 0x7f2d3415d7c0 2023-10-16T09:59:15.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.914+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d340c6820 con 0x7f2d3415d7c0 2023-10-16T09:59:15.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.915+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d34158ea0 con 0x7f2d3415d7c0 2023-10-16T09:59:15.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.915+0000 7f2d1b7fe700 1 -- 172.21.15.158:0/896868507 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d2c007070 con 0x7f2d3415d7c0 2023-10-16T09:59:15.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.916+0000 7f2d1b7fe700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2d2006b020 0x7f2d2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:15.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.916+0000 7f2d1b7fe700 1 -- 172.21.15.158:0/896868507 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f2d2c0895f0 con 0x7f2d3415d7c0 2023-10-16T09:59:15.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.916+0000 7f2d32ffd700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2d2006b020 0x7f2d2006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:15.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.916+0000 7f2d32ffd700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2d2006b020 0x7f2d2006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009c40 tx=0x7f2d28008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:15.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:15.920+0000 7f2d1b7fe700 1 -- 172.21.15.158:0/896868507 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d34158ea0 con 0x7f2d3415d7c0 2023-10-16T09:59:16.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:16.193+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f2d34158ea0 con 0x7f2d3415d7c0 2023-10-16T09:59:17.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.089+0000 7f2d1b7fe700 1 -- 172.21.15.158:0/896868507 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v574) v1 ==== 159+0+0 (secure 0 0 0) 0x7f2d34158ea0 con 0x7f2d3415d7c0 2023-10-16T09:59:17.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2d2006b020 msgr2=0x7f2d2006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2d2006b020 0x7f2d2006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009c40 tx=0x7f2d28008030 comp rx=0 tx=0).stop 2023-10-16T09:59:17.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d3415d7c0 msgr2=0x7f2d340c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d3415d7c0 0x7f2d340c62e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00cbc0 tx=0x7f2d2c00ced0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 shutdown_connections 2023-10-16T09:59:17.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2d2006b020 0x7f2d2006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d3415d7c0 0x7f2d340c62e0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d340ce950 0x7f2d34159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 --2- 172.21.15.158:0/896868507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340d1210 0x7f2d34153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 >> 172.21.15.158:0/896868507 conn(0x7f2d340669f0 msgr2=0x7f2d34073e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:17.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 shutdown_connections 2023-10-16T09:59:17.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.092+0000 7f2d396be700 1 -- 172.21.15.158:0/896868507 wait complete. 2023-10-16T09:59:17.108 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T09:59:17.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.236+0000 7f60c7989700 1 Processor -- start 2023-10-16T09:59:17.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.236+0000 7f60c7989700 1 -- start start 2023-10-16T09:59:17.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c7989700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60c006ed20 0x7f60c006b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c7989700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c015a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c7989700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c006c280 con 0x7f60c015a890 2023-10-16T09:59:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c7989700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c006de30 con 0x7f60c006ed20 2023-10-16T09:59:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c7989700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c00b3a50 con 0x7f60c006be20 2023-10-16T09:59:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c5725700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60c006ed20 0x7f60c006b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c5725700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60c006ed20 0x7f60c006b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52028/0 (socket says 172.21.15.158:52028) 2023-10-16T09:59:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.237+0000 7f60c5725700 1 -- 172.21.15.158:0/862304171 learned_addr learned my addr 172.21.15.158:0/862304171 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60c4f24700 1 --2- 172.21.15.158:0/862304171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60c5725700 1 -- 172.21.15.158:0/862304171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 msgr2=0x7f60c015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60c5725700 1 --2- 172.21.15.158:0/862304171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c015a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60c5725700 1 -- 172.21.15.158:0/862304171 >> v1:172.21.15.172:6789/0 conn(0x7f60c015a890 legacy=0x7f60c015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60c5725700 1 -- 172.21.15.158:0/862304171 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c015b0c0 con 0x7f60c006ed20 2023-10-16T09:59:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60c4f24700 1 --2- 172.21.15.158:0/862304171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60c5725700 1 --2- 172.21.15.158:0/862304171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006ed20 0x7f60c006b8e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f60b00098f0 tx=0x7f60b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d4225964e70cb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60b7fff700 1 -- 172.21.15.158:0/862304171 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0004230 con 0x7f60c006ed20 2023-10-16T09:59:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60b7fff700 1 -- 172.21.15.158:0/862304171 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b0004390 con 0x7f60c006ed20 2023-10-16T09:59:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.238+0000 7f60b7fff700 1 -- 172.21.15.158:0/862304171 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00036a0 con 0x7f60c006ed20 2023-10-16T09:59:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 -- 172.21.15.158:0/862304171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006ed20 msgr2=0x7f60c006b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 --2- 172.21.15.158:0/862304171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006ed20 0x7f60c006b8e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f60b00098f0 tx=0x7f60b0009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 -- 172.21.15.158:0/862304171 shutdown_connections 2023-10-16T09:59:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 --2- 172.21.15.158:0/862304171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006ed20 0x7f60c006b8e0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 --2- 172.21.15.158:0/862304171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 -- 172.21.15.158:0/862304171 >> 172.21.15.158:0/862304171 conn(0x7f60c0071110 msgr2=0x7f60c0071520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 -- 172.21.15.158:0/862304171 shutdown_connections 2023-10-16T09:59:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.239+0000 7f60c7989700 1 -- 172.21.15.158:0/862304171 wait complete. 2023-10-16T09:59:17.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.240+0000 7f60c7989700 1 Processor -- start 2023-10-16T09:59:17.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.240+0000 7f60c7989700 1 -- start start 2023-10-16T09:59:17.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c7989700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c0154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c7989700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60c006ed20 0x7f60c0147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c7989700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c015a890 0x7f60c0148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c7989700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c0148f00 con 0x7f60c015a890 2023-10-16T09:59:17.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c7989700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c014d1d0 con 0x7f60c006ed20 2023-10-16T09:59:17.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c7989700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c014c8a0 con 0x7f60c006be20 2023-10-16T09:59:17.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c015a890 0x7f60c0148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c015a890 0x7f60c0148520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52036/0 (socket says 172.21.15.158:52036) 2023-10-16T09:59:17.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 -- 172.21.15.158:0/3857003771 learned_addr learned my addr 172.21.15.158:0/3857003771 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:17.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c4f24700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60c006ed20 0x7f60c0147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5725700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c0154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 -- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 msgr2=0x7f60c0154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c0154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 -- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60c006ed20 msgr2=0x7f60c0147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60c006ed20 0x7f60c0147450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 -- 172.21.15.158:0/3857003771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0009550 con 0x7f60c015a890 2023-10-16T09:59:17.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c5f26700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c015a890 0x7f60c0148520 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f60b800cc10 tx=0x7f60b800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:17.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60b5ffb700 1 -- 172.21.15.158:0/3857003771 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b800aa60 con 0x7f60c015a890 2023-10-16T09:59:17.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60b5ffb700 1 -- 172.21.15.158:0/3857003771 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b800abc0 con 0x7f60c015a890 2023-10-16T09:59:17.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c014cb80 con 0x7f60c015a890 2023-10-16T09:59:17.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.241+0000 7f60b5ffb700 1 -- 172.21.15.158:0/3857003771 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b8012420 con 0x7f60c015a890 2023-10-16T09:59:17.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.242+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c0147990 con 0x7f60c015a890 2023-10-16T09:59:17.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.243+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c00b40d0 con 0x7f60c015a890 2023-10-16T09:59:17.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.243+0000 7f60b5ffb700 1 -- 172.21.15.158:0/3857003771 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f60b8012580 con 0x7f60c015a890 2023-10-16T09:59:17.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.243+0000 7f60b5ffb700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f60a406b020 0x7f60a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.243+0000 7f60b5ffb700 1 -- 172.21.15.158:0/3857003771 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f60b8089d00 con 0x7f60c015a890 2023-10-16T09:59:17.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.243+0000 7f60c5725700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f60a406b020 0x7f60a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.244+0000 7f60c5725700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f60a406b020 0x7f60a406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009130 tx=0x7f60b0008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:17.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.248+0000 7f60b5ffb700 1 -- 172.21.15.158:0/3857003771 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60b8054a00 con 0x7f60c015a890 2023-10-16T09:59:17.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.439+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f60c00cf560 con 0x7f60c015a890 2023-10-16T09:59:17.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.442+0000 7f60b5ffb700 1 -- 172.21.15.158:0/3857003771 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f60b8058020 con 0x7f60c015a890 2023-10-16T09:59:17.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f60a406b020 msgr2=0x7f60a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f60a406b020 0x7f60a406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009130 tx=0x7f60b0008030 comp rx=0 tx=0).stop 2023-10-16T09:59:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c015a890 msgr2=0x7f60c0148520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c015a890 0x7f60c0148520 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f60b800cc10 tx=0x7f60b800cf20 comp rx=0 tx=0).stop 2023-10-16T09:59:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 shutdown_connections 2023-10-16T09:59:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f60a406b020 0x7f60a406d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c015a890 0x7f60c0148520 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60c006be20 0x7f60c0154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 --2- 172.21.15.158:0/3857003771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60c006ed20 0x7f60c0147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 >> 172.21.15.158:0/3857003771 conn(0x7f60c0071110 msgr2=0x7f60c014fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:17.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 shutdown_connections 2023-10-16T09:59:17.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.444+0000 7f60c7989700 1 -- 172.21.15.158:0/3857003771 wait complete. 2023-10-16T09:59:17.457 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T09:59:17.458 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:17.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:17.554 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T09:59:17.555 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:17.556 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:17.577 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:59:17.577 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:17.577 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:17.597 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:17.598 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:17.598 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:17.634 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:59:17.634 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:17.635 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:17.656 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:17.656 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T09:59:17.657 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:17.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.793+0000 7fd83faae700 1 Processor -- start 2023-10-16T09:59:17.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.793+0000 7fd83faae700 1 -- start start 2023-10-16T09:59:17.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83faae700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd83806e5d0 0x7fd83806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83faae700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd838161270 con 0x7fd8380729b0 2023-10-16T09:59:17.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83faae700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd838161510 con 0x7fd83806ab40 2023-10-16T09:59:17.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83faae700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8381617f0 con 0x7fd83806e5d0 2023-10-16T09:59:17.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83d049700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd83806e5d0 0x7fd83806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83e04b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd8380729b0 0x7fd83815df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52218/0 (socket says 172.21.15.158:52218) 2023-10-16T09:59:17.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83e04b700 1 -- 172.21.15.158:0/2072638162 learned_addr learned my addr 172.21.15.158:0/2072638162 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:17.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83d049700 1 -- 172.21.15.158:0/2072638162 >> v1:172.21.15.172:6789/0 conn(0x7fd83806ab40 legacy=0x7fd8380722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:17.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.794+0000 7fd83d049700 1 -- 172.21.15.158:0/2072638162 >> v1:172.21.15.158:6789/0 conn(0x7fd8380729b0 legacy=0x7fd83815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83c848700 1 -- 172.21.15.158:0/2072638162 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50958084 0 0) 0x7fd838161270 con 0x7fd8380729b0 2023-10-16T09:59:17.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83d049700 1 -- 172.21.15.158:0/2072638162 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838161ad0 con 0x7fd83806e5d0 2023-10-16T09:59:17.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83d049700 1 --2- 172.21.15.158:0/2072638162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd83806d820 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd834009840 tx=0x7fd834009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fd0372fdea34d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:17.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83c848700 1 -- 172.21.15.158:0/2072638162 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834004230 con 0x7fd83806e5d0 2023-10-16T09:59:17.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83c848700 1 -- 172.21.15.158:0/2072638162 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd834004390 con 0x7fd83806e5d0 2023-10-16T09:59:17.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83c848700 1 -- 172.21.15.158:0/2072638162 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8340036a0 con 0x7fd83806e5d0 2023-10-16T09:59:17.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83faae700 1 -- 172.21.15.158:0/2072638162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 msgr2=0x7fd83806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83faae700 1 --2- 172.21.15.158:0/2072638162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd83806d820 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd834009840 tx=0x7fd834009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:17.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83faae700 1 -- 172.21.15.158:0/2072638162 shutdown_connections 2023-10-16T09:59:17.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83faae700 1 --2- 172.21.15.158:0/2072638162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd83806d820 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83faae700 1 -- 172.21.15.158:0/2072638162 >> 172.21.15.158:0/2072638162 conn(0x7fd8380d02b0 msgr2=0x7fd8380d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:17.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83faae700 1 -- 172.21.15.158:0/2072638162 shutdown_connections 2023-10-16T09:59:17.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.795+0000 7fd83faae700 1 -- 172.21.15.158:0/2072638162 wait complete. 2023-10-16T09:59:17.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 Processor -- start 2023-10-16T09:59:17.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 -- start start 2023-10-16T09:59:17.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd83806ab40 0x7fd8380b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd838156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8380729b0 0x7fd8380c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838157460 con 0x7fd83806e5d0 2023-10-16T09:59:17.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380b0550 con 0x7fd83806ab40 2023-10-16T09:59:17.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83faae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380c5c20 con 0x7fd8380729b0 2023-10-16T09:59:17.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83d049700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd838156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83d049700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd838156f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52048/0 (socket says 172.21.15.158:52048) 2023-10-16T09:59:17.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.797+0000 7fd83d049700 1 -- 172.21.15.158:0/3560741659 learned_addr learned my addr 172.21.15.158:0/3560741659 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:17.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83d84a700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd83806ab40 0x7fd8380b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83e04b700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8380729b0 0x7fd8380c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83d049700 1 -- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8380729b0 msgr2=0x7fd8380c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83d049700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8380729b0 0x7fd8380c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83d049700 1 -- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd83806ab40 msgr2=0x7fd8380b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:17.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83d049700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd83806ab40 0x7fd8380b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:17.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83d049700 1 -- 172.21.15.158:0/3560741659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340094a0 con 0x7fd83806e5d0 2023-10-16T09:59:17.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83e04b700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8380729b0 0x7fd8380c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:17.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83d049700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd838156f20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd834004000 tx=0x7fd834004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:17.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd82e7fc700 1 -- 172.21.15.158:0/3560741659 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834011630 con 0x7fd83806e5d0 2023-10-16T09:59:17.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd82e7fc700 1 -- 172.21.15.158:0/3560741659 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83401e070 con 0x7fd83806e5d0 2023-10-16T09:59:17.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8380c5ec0 con 0x7fd83806e5d0 2023-10-16T09:59:17.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd82e7fc700 1 -- 172.21.15.158:0/3560741659 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8340119b0 con 0x7fd83806e5d0 2023-10-16T09:59:17.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.798+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8380c6480 con 0x7fd83806e5d0 2023-10-16T09:59:17.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.799+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8380681b0 con 0x7fd83806e5d0 2023-10-16T09:59:17.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.800+0000 7fd82e7fc700 1 -- 172.21.15.158:0/3560741659 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd834011b10 con 0x7fd83806e5d0 2023-10-16T09:59:17.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.800+0000 7fd82e7fc700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd81c06b020 0x7fd81c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:17.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.800+0000 7fd82e7fc700 1 -- 172.21.15.158:0/3560741659 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fd83408ad40 con 0x7fd83806e5d0 2023-10-16T09:59:17.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.800+0000 7fd83d84a700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd81c06b020 0x7fd81c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:17.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.800+0000 7fd83d84a700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd81c06b020 0x7fd81c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd828001730 tx=0x7fd828009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:17.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:17.804+0000 7fd82e7fc700 1 -- 172.21.15.158:0/3560741659 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd834055ad0 con 0x7fd83806e5d0 2023-10-16T09:59:18.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.038+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8380681b0 con 0x7fd83806e5d0 2023-10-16T09:59:18.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.039+0000 7fd82e7fc700 1 -- 172.21.15.158:0/3560741659 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3712 (secure 0 0 0) 0x7fd834020020 con 0x7fd83806e5d0 2023-10-16T09:59:18.041 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:18.041 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2744363151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2744363151},{"type":"v1","addr":"172.21.15.172:6835","nonce":2744363151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4264509745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":4264509745},{"type":"v1","addr":"172.21.15.158:6835","nonce":4264509745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1091298830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1091298830},{"type":"v1","addr":"172.21.15.158:6837","nonce":1091298830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":35261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/2630165052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2630165052},{"type":"v1","addr":"172.21.15.172:6834","nonce":2630165052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2023-10-16T09:59:18.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.041+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd81c06b020 msgr2=0x7fd81c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:18.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.041+0000 7fd83faae700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd81c06b020 0x7fd81c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd828001730 tx=0x7fd828009040 comp rx=0 tx=0).stop 2023-10-16T09:59:18.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.041+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 msgr2=0x7fd838156f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:18.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.041+0000 7fd83faae700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd838156f20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd834004000 tx=0x7fd834004750 comp rx=0 tx=0).stop 2023-10-16T09:59:18.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.041+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 shutdown_connections 2023-10-16T09:59:18.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.041+0000 7fd83faae700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd81c06b020 0x7fd81c06d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:18.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.041+0000 7fd83faae700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8380729b0 0x7fd8380c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:18.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.042+0000 7fd83faae700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd83806ab40 0x7fd8380b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:18.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.042+0000 7fd83faae700 1 --2- 172.21.15.158:0/3560741659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd83806e5d0 0x7fd838156f20 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:18.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.042+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 >> 172.21.15.158:0/3560741659 conn(0x7fd8380d02b0 msgr2=0x7fd8380c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:18.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.042+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 shutdown_connections 2023-10-16T09:59:18.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.042+0000 7fd83faae700 1 -- 172.21.15.158:0/3560741659 wait complete. 2023-10-16T09:59:18.045 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 27 2023-10-16T09:59:18.055 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T09:59:18.056 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T09:59:18.056 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T09:59:18.056 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:18.057 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T09:59:18.057 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T09:59:18.057 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T09:59:18.057 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:18.058 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T09:59:18.058 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T09:59:18.058 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T09:59:18.058 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:18.059 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T09:59:18.059 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T09:59:18.059 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T09:59:18.060 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:18.064 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T09:59:18.065 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T09:59:18.066 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T09:59:18.066 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T09:59:18.070 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T09:59:18.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T09:59:18.071 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T09:59:18.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T09:59:18.108 INFO:tasks.ceph.mds.a:Started 2023-10-16T09:59:18.108 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:18.109 INFO:tasks.ceph.mds.c:Started 2023-10-16T09:59:18.110 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:18.112 INFO:tasks.ceph.mds.b:Started 2023-10-16T09:59:18.113 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:18.113 INFO:tasks.ceph.mds.d:Started 2023-10-16T09:59:18.114 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:18.114 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T09:59:18.187 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T09:59:18.189 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T09:59:18.200 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T09:59:18.201 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:59:18.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.240+0000 7f3778d52700 1 Processor -- start 2023-10-16T09:59:18.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.240+0000 7f3778d52700 1 -- start start 2023-10-16T09:59:18.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.241+0000 7f3778d52700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f377415d870 0x7f377406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:18.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.241+0000 7f3778d52700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37741621b0 con 0x7f37740b3290 2023-10-16T09:59:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.241+0000 7f3778d52700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377415dc50 con 0x7f37740cf7c0 2023-10-16T09:59:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.241+0000 7f3778d52700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37740701a0 con 0x7f377415d870 2023-10-16T09:59:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f377415d870 0x7f377406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f3772d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f37740b3290 0x7f3774161aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52258/0 (socket says 172.21.15.158:52258) 2023-10-16T09:59:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f3772d9d700 1 -- 172.21.15.158:0/2454693044 learned_addr learned my addr 172.21.15.158:0/2454693044 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377259c700 1 -- 172.21.15.158:0/2454693044 >> v1:172.21.15.172:6789/0 conn(0x7f37740cf7c0 legacy=0x7f3774161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377259c700 1 -- 172.21.15.158:0/2454693044 >> v1:172.21.15.158:6789/0 conn(0x7f37740b3290 legacy=0x7f3774161aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377259c700 1 -- 172.21.15.158:0/2454693044 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377406f100 con 0x7f377415d870 2023-10-16T09:59:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377159a700 1 -- 172.21.15.158:0/2454693044 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480422127 0 0) 0x7f37741621b0 con 0x7f37740b3290 2023-10-16T09:59:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377259c700 1 --2- 172.21.15.158:0/2454693044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377415d870 0x7f377406a8d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3768009840 tx=0x7f3768009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e0b277a24acf16 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:18.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377159a700 1 -- 172.21.15.158:0/2454693044 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768011830 con 0x7f377415d870 2023-10-16T09:59:18.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377159a700 1 -- 172.21.15.158:0/2454693044 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3768011990 con 0x7f377415d870 2023-10-16T09:59:18.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.242+0000 7f377159a700 1 -- 172.21.15.158:0/2454693044 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768011d20 con 0x7f377415d870 2023-10-16T09:59:18.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.243+0000 7f3778d52700 1 -- 172.21.15.158:0/2454693044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377415d870 msgr2=0x7f377406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:18.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.243+0000 7f3778d52700 1 --2- 172.21.15.158:0/2454693044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377415d870 0x7f377406a8d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3768009840 tx=0x7f3768009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:18.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.243+0000 7f3778d52700 1 -- 172.21.15.158:0/2454693044 shutdown_connections 2023-10-16T09:59:18.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.243+0000 7f3778d52700 1 --2- 172.21.15.158:0/2454693044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377415d870 0x7f377406a8d0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:18.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.243+0000 7f3778d52700 1 -- 172.21.15.158:0/2454693044 >> 172.21.15.158:0/2454693044 conn(0x7f377406da00 msgr2=0x7f377406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:18.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.243+0000 7f3778d52700 1 -- 172.21.15.158:0/2454693044 shutdown_connections 2023-10-16T09:59:18.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.243+0000 7f3778d52700 1 -- 172.21.15.158:0/2454693044 wait complete. 2023-10-16T09:59:18.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.244+0000 7f3778d52700 1 Processor -- start 2023-10-16T09:59:18.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.244+0000 7f3778d52700 1 -- start start 2023-10-16T09:59:18.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3778d52700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37740b3290 0x7f377413c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:18.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3778d52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37740cf7c0 0x7f37741f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:18.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3778d52700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f377415d870 0x7f37741f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:18.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3778d52700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3774160240 con 0x7f37740cf7c0 2023-10-16T09:59:18.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3778d52700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377413c9c0 con 0x7f37740b3290 2023-10-16T09:59:18.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3778d52700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3774151fa0 con 0x7f377415d870 2023-10-16T09:59:18.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37740cf7c0 0x7f37741f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:18.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37740cf7c0 0x7f37741f6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52084/0 (socket says 172.21.15.158:52084) 2023-10-16T09:59:18.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 -- 172.21.15.158:0/3963124500 learned_addr learned my addr 172.21.15.158:0/3963124500 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:18.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f377259c700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37740b3290 0x7f377413c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:18.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3772d9d700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f377415d870 0x7f37741f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:18.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 -- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f377415d870 msgr2=0x7f37741f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:18.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f377415d870 0x7f37741f6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 -- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37740b3290 msgr2=0x7f377413c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37740b3290 0x7f377413c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 -- 172.21.15.158:0/3963124500 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37680094a0 con 0x7f37740cf7c0 2023-10-16T09:59:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.245+0000 7f3771d9b700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37740cf7c0 0x7f37741f6540 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f376400c4b0 tx=0x7f376400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.246+0000 7f3762ffd700 1 -- 172.21.15.158:0/3963124500 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37640052d0 con 0x7f37740cf7c0 2023-10-16T09:59:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.246+0000 7f3762ffd700 1 -- 172.21.15.158:0/3963124500 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3764005430 con 0x7f37740cf7c0 2023-10-16T09:59:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.246+0000 7f3762ffd700 1 -- 172.21.15.158:0/3963124500 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37640038c0 con 0x7f37740cf7c0 2023-10-16T09:59:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.246+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377415f910 con 0x7f37740cf7c0 2023-10-16T09:59:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.246+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377415fed0 con 0x7f37740cf7c0 2023-10-16T09:59:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.247+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3774160090 con 0x7f37740cf7c0 2023-10-16T09:59:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.247+0000 7f3762ffd700 1 -- 172.21.15.158:0/3963124500 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3764003a20 con 0x7f37740cf7c0 2023-10-16T09:59:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.247+0000 7f3762ffd700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f375406b020 0x7f375406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.248+0000 7f3762ffd700 1 -- 172.21.15.158:0/3963124500 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f3764088d40 con 0x7f37740cf7c0 2023-10-16T09:59:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.248+0000 7f377259c700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f375406b020 0x7f375406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.248+0000 7f377259c700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f375406b020 0x7f375406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f37680090b0 tx=0x7f376801c040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:18.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.252+0000 7f3762ffd700 1 -- 172.21.15.158:0/3963124500 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3764053a40 con 0x7f37740cf7c0 2023-10-16T09:59:18.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:18.462+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f37740681b0 con 0x7f37740cf7c0 2023-10-16T09:59:19.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.583+0000 7f3762ffd700 1 -- 172.21.15.158:0/3963124500 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v81) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3764017080 con 0x7f37740cf7c0 2023-10-16T09:59:19.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f375406b020 msgr2=0x7f375406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:19.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f375406b020 0x7f375406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f37680090b0 tx=0x7f376801c040 comp rx=0 tx=0).stop 2023-10-16T09:59:19.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37740cf7c0 msgr2=0x7f37741f6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:19.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37740cf7c0 0x7f37741f6540 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f376400c4b0 tx=0x7f376400c7c0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 shutdown_connections 2023-10-16T09:59:19.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f375406b020 0x7f375406d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f377415d870 0x7f37741f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37740b3290 0x7f377413c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 --2- 172.21.15.158:0/3963124500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37740cf7c0 0x7f37741f6540 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.586+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 >> 172.21.15.158:0/3963124500 conn(0x7f377406da00 msgr2=0x7f377406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:19.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.587+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 shutdown_connections 2023-10-16T09:59:19.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.587+0000 7f3778d52700 1 -- 172.21.15.158:0/3963124500 wait complete. 2023-10-16T09:59:19.591 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T09:59:19.603 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T09:59:19.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.728+0000 7f206ab01700 1 Processor -- start 2023-10-16T09:59:19.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.728+0000 7f206ab01700 1 -- start start 2023-10-16T09:59:19.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.729+0000 7f206ab01700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f206406ab40 0x7f206406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.729+0000 7f206ab01700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2064161270 con 0x7f206406e5d0 2023-10-16T09:59:19.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.729+0000 7f206ab01700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2064161510 con 0x7f20640729b0 2023-10-16T09:59:19.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.729+0000 7f206ab01700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20641617f0 con 0x7f206406ab40 2023-10-16T09:59:19.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.729+0000 7f2063fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f206406e5d0 0x7f20640722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52272/0 (socket says 172.21.15.158:52272) 2023-10-16T09:59:19.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.729+0000 7f2063fff700 1 -- 172.21.15.158:0/4099309749 learned_addr learned my addr 172.21.15.158:0/4099309749 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:19.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958256849 0 0) 0x7f2064161270 con 0x7f206406e5d0 2023-10-16T09:59:19.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540035b0 con 0x7f206406e5d0 2023-10-16T09:59:19.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206889d700 1 --2- 172.21.15.158:0/4099309749 >> v2:172.21.15.172:3300/0 conn(0x7f206406ab40 0x7f206406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 795716644 0 0) 0x7f20540035b0 con 0x7f206406e5d0 2023-10-16T09:59:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 >> v2:172.21.15.172:3300/0 conn(0x7f206406ab40 msgr2=0x7f206406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 --2- 172.21.15.158:0/4099309749 >> v2:172.21.15.172:3300/0 conn(0x7f206406ab40 0x7f206406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 >> v1:172.21.15.172:6789/0 conn(0x7f20640729b0 legacy=0x7f206415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2064161ad0 con 0x7f206406e5d0 2023-10-16T09:59:19.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f2050001670 con 0x7f206406e5d0 2023-10-16T09:59:19.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 >> v1:172.21.15.158:6789/0 conn(0x7f206406e5d0 legacy=0x7f20640722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:19.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f2054003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2054004850 0x7f2054004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2054005210 0x7f205400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20641617f0 con 0x7f2054003a00 2023-10-16T09:59:19.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205400c0b0 con 0x7f2054005210 2023-10-16T09:59:19.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205400c270 con 0x7f2054004850 2023-10-16T09:59:19.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206889d700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f2054003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f2063fff700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2054004850 0x7f2054004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206909e700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2054005210 0x7f205400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206889d700 1 -- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2054004850 msgr2=0x7f2054004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:19.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206889d700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2054004850 0x7f2054004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206889d700 1 -- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2054005210 msgr2=0x7f205400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:19.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206889d700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2054005210 0x7f205400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206909e700 1 -- 172.21.15.158:0/4099309749 reap_dead start 2023-10-16T09:59:19.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.730+0000 7f206889d700 1 -- 172.21.15.158:0/4099309749 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20540035b0 con 0x7f2054003a00 2023-10-16T09:59:19.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206889d700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f2054003e20 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2058009b60 tx=0x7f2054005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bb02fb9a963933 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:19.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205800e6c0 con 0x7f2054003a00 2023-10-16T09:59:19.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205800e820 con 0x7f2054003a00 2023-10-16T09:59:19.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f20637fe700 1 -- 172.21.15.158:0/4099309749 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205800eb90 con 0x7f2054003a00 2023-10-16T09:59:19.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206ab01700 1 -- 172.21.15.158:0/4099309749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 msgr2=0x7f2054003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:19.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206ab01700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f2054003e20 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2058009b60 tx=0x7f2054005880 comp rx=0 tx=0).stop 2023-10-16T09:59:19.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206ab01700 1 -- 172.21.15.158:0/4099309749 shutdown_connections 2023-10-16T09:59:19.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206ab01700 1 --2- 172.21.15.158:0/4099309749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f2054003e20 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206ab01700 1 -- 172.21.15.158:0/4099309749 >> 172.21.15.158:0/4099309749 conn(0x7f20640d02b0 msgr2=0x7f20640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:19.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206ab01700 1 -- 172.21.15.158:0/4099309749 shutdown_connections 2023-10-16T09:59:19.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.731+0000 7f206ab01700 1 -- 172.21.15.158:0/4099309749 wait complete. 2023-10-16T09:59:19.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.732+0000 7f206ab01700 1 Processor -- start 2023-10-16T09:59:19.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206ab01700 1 -- start start 2023-10-16T09:59:19.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206ab01700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f20640c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206ab01700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20640c3d60 0x7f20640b5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206ab01700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20640b5a90 0x7f20640c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206ab01700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20640b3a60 con 0x7f2054003a00 2023-10-16T09:59:19.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206ab01700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20640b3d40 con 0x7f20640b5a90 2023-10-16T09:59:19.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206ab01700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20640c63e0 con 0x7f20640c3d60 2023-10-16T09:59:19.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.733+0000 7f206889d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f20640c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f20640c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52106/0 (socket says 172.21.15.158:52106) 2023-10-16T09:59:19.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 -- 172.21.15.158:0/3490827468 learned_addr learned my addr 172.21.15.158:0/3490827468 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:19.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206909e700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20640b5a90 0x7f20640c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f2063fff700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20640c3d60 0x7f20640b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 -- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20640c3d60 msgr2=0x7f20640b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:19.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20640c3d60 0x7f20640b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 -- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20640b5a90 msgr2=0x7f20640c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:19.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20640b5a90 0x7f20640c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:19.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 -- 172.21.15.158:0/3490827468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20640c4e80 con 0x7f2054003a00 2023-10-16T09:59:19.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206889d700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f20640c4940 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f205400c080 tx=0x7f20580047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:19.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f20617fa700 1 -- 172.21.15.158:0/3490827468 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205800e6c0 con 0x7f2054003a00 2023-10-16T09:59:19.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f20617fa700 1 -- 172.21.15.158:0/3490827468 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2058003ad0 con 0x7f2054003a00 2023-10-16T09:59:19.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20640b5f20 con 0x7f2054003a00 2023-10-16T09:59:19.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f20617fa700 1 -- 172.21.15.158:0/3490827468 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058003e80 con 0x7f2054003a00 2023-10-16T09:59:19.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.734+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20641562b0 con 0x7f2054003a00 2023-10-16T09:59:19.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.735+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20640b37d0 con 0x7f2054003a00 2023-10-16T09:59:19.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.736+0000 7f20617fa700 1 -- 172.21.15.158:0/3490827468 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2058013470 con 0x7f2054003a00 2023-10-16T09:59:19.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.736+0000 7f20617fa700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f204806b050 0x7f204806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:19.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.736+0000 7f20617fa700 1 -- 172.21.15.158:0/3490827468 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f205808ad90 con 0x7f2054003a00 2023-10-16T09:59:19.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.736+0000 7f2063fff700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f204806b050 0x7f204806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:19.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.736+0000 7f2063fff700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f204806b050 0x7f204806d510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f206406b560 tx=0x7f2050009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:19.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:19.740+0000 7f20617fa700 1 -- 172.21.15.158:0/3490827468 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2058055f40 con 0x7f2054003a00 2023-10-16T09:59:20.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.010+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f20640681b0 con 0x7f2054003a00 2023-10-16T09:59:20.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.011+0000 7f20617fa700 1 -- 172.21.15.158:0/3490827468 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2058017020 con 0x7f2054003a00 2023-10-16T09:59:20.012 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:20.013 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:59:20.013 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T09:59:20.013 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.013 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T09:59:20.013 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T09:59:20.014 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.014 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.014 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:20.014 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:20.014 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T09:59:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T09:59:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T09:59:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T09:59:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T09:59:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T09:59:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T09:59:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T09:59:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T09:59:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T09:59:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T09:59:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T09:59:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T09:59:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T09:59:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T09:59:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T09:59:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T09:59:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T09:59:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T09:59:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T09:59:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T09:59:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T09:59:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T09:59:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T09:59:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.036 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.036 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T09:59:20.036 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T09:59:20.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T09:59:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T09:59:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:59:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:59:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:59:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T09:59:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T09:59:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:59:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:59:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:59:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.041 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.041 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T09:59:20.041 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T09:59:20.041 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.042 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:20.042 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:59:20.042 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:59:20.042 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:59:20.042 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.042 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.043 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.043 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T09:59:20.043 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T09:59:20.043 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.043 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:59:20.044 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:20.044 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:20.044 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.044 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.045 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.045 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T09:59:20.045 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T09:59:20.045 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.046 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:59:20.046 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:59:20.046 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.046 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.046 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:20.047 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:20.047 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T09:59:20.047 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T09:59:20.047 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:20.047 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:59:20.048 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:59:20.048 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:20.048 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.048 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:20.048 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T09:59:20.049 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:59:20.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f204806b050 msgr2=0x7f204806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f204806b050 0x7f204806d510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f206406b560 tx=0x7f2050009040 comp rx=0 tx=0).stop 2023-10-16T09:59:20.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 msgr2=0x7f20640c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f20640c4940 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f205400c080 tx=0x7f20580047c0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 shutdown_connections 2023-10-16T09:59:20.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f204806b050 0x7f204806d510 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20640b5a90 0x7f20640c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2054003a00 0x7f20640c4940 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 --2- 172.21.15.158:0/3490827468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20640c3d60 0x7f20640b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 >> 172.21.15.158:0/3490827468 conn(0x7f20640d02b0 msgr2=0x7f20640b0740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 shutdown_connections 2023-10-16T09:59:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.013+0000 7f206ab01700 1 -- 172.21.15.158:0/3490827468 wait complete. 2023-10-16T09:59:20.053 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:20.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.176+0000 7fe73a281700 1 Processor -- start 2023-10-16T09:59:20.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.176+0000 7fe73a281700 1 -- start start 2023-10-16T09:59:20.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73a281700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73415dfb0 0x7fe73406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73a281700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe73406f340 0x7fe73406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73a281700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73406e770 con 0x7fe7340cc5d0 2023-10-16T09:59:20.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73a281700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73406fd30 con 0x7fe73406f340 2023-10-16T09:59:20.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73a281700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340cf270 con 0x7fe73415dfb0 2023-10-16T09:59:20.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73881e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe73406f340 0x7fe73406f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe733fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe7340cc5d0 0x7fe7340cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52290/0 (socket says 172.21.15.158:52290) 2023-10-16T09:59:20.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe733fff700 1 -- 172.21.15.158:0/1235193466 learned_addr learned my addr 172.21.15.158:0/1235193466 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:20.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe7337fe700 1 --2- 172.21.15.158:0/1235193466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73415dfb0 0x7fe73406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73881e700 1 -- 172.21.15.158:0/1235193466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73415dfb0 msgr2=0x7fe73406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:20.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73881e700 1 --2- 172.21.15.158:0/1235193466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73415dfb0 0x7fe73406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73881e700 1 -- 172.21.15.158:0/1235193466 >> v1:172.21.15.158:6789/0 conn(0x7fe7340cc5d0 legacy=0x7fe7340cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe732ffd700 1 -- 172.21.15.158:0/1235193466 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780658383 0 0) 0x7fe728001110 con 0x7fe7340cc5d0 2023-10-16T09:59:20.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73881e700 1 -- 172.21.15.158:0/1235193466 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7340cbe00 con 0x7fe73406f340 2023-10-16T09:59:20.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe73881e700 1 --2- 172.21.15.158:0/1235193466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe73406f340 0x7fe73406f7f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe72c009d90 tx=0x7fe72c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a125c51d68cb83de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe732ffd700 1 -- 172.21.15.158:0/1235193466 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00e070 con 0x7fe73406f340 2023-10-16T09:59:20.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe732ffd700 1 -- 172.21.15.158:0/1235193466 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c00bbd0 con 0x7fe73406f340 2023-10-16T09:59:20.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.177+0000 7fe732ffd700 1 -- 172.21.15.158:0/1235193466 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c012430 con 0x7fe73406f340 2023-10-16T09:59:20.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 -- 172.21.15.158:0/1235193466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe73406f340 msgr2=0x7fe73406f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 --2- 172.21.15.158:0/1235193466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe73406f340 0x7fe73406f7f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe72c009d90 tx=0x7fe72c00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:20.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 -- 172.21.15.158:0/1235193466 shutdown_connections 2023-10-16T09:59:20.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 --2- 172.21.15.158:0/1235193466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe73406f340 0x7fe73406f7f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 --2- 172.21.15.158:0/1235193466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73415dfb0 0x7fe73406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 -- 172.21.15.158:0/1235193466 >> 172.21.15.158:0/1235193466 conn(0x7fe7340d02b0 msgr2=0x7fe7340d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:20.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 -- 172.21.15.158:0/1235193466 shutdown_connections 2023-10-16T09:59:20.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.178+0000 7fe73a281700 1 -- 172.21.15.158:0/1235193466 wait complete. 2023-10-16T09:59:20.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.179+0000 7fe73a281700 1 Processor -- start 2023-10-16T09:59:20.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.179+0000 7fe73a281700 1 -- start start 2023-10-16T09:59:20.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe73a281700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73406f340 0x7fe734151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe73a281700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7340cc5d0 0x7fe7340ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe73a281700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73415dfb0 0x7fe73415c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe73a281700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340af380 con 0x7fe7340cc5d0 2023-10-16T09:59:20.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe73a281700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73415c0d0 con 0x7fe73415dfb0 2023-10-16T09:59:20.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe73a281700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe734151c70 con 0x7fe73406f340 2023-10-16T09:59:20.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7340cc5d0 0x7fe7340ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7340cc5d0 0x7fe7340ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52122/0 (socket says 172.21.15.158:52122) 2023-10-16T09:59:20.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 -- 172.21.15.158:0/631524699 learned_addr learned my addr 172.21.15.158:0/631524699 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:20.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe733fff700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73406f340 0x7fe734151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe73881e700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73415dfb0 0x7fe73415c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 -- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73406f340 msgr2=0x7fe734151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73406f340 0x7fe734151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 -- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73415dfb0 msgr2=0x7fe73415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73415dfb0 0x7fe73415c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.180+0000 7fe7337fe700 1 -- 172.21.15.158:0/631524699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c009a70 con 0x7fe7340cc5d0 2023-10-16T09:59:20.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.181+0000 7fe73881e700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73415dfb0 0x7fe73415c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:20.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.181+0000 7fe7337fe700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7340cc5d0 0x7fe7340ae570 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe72400e7e0 tx=0x7fe72400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.181+0000 7fe730ff9700 1 -- 172.21.15.158:0/631524699 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724004230 con 0x7fe7340cc5d0 2023-10-16T09:59:20.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.181+0000 7fe730ff9700 1 -- 172.21.15.158:0/631524699 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe724004390 con 0x7fe7340cc5d0 2023-10-16T09:59:20.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.181+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7340aedc0 con 0x7fe7340cc5d0 2023-10-16T09:59:20.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.181+0000 7fe730ff9700 1 -- 172.21.15.158:0/631524699 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240036a0 con 0x7fe7340cc5d0 2023-10-16T09:59:20.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.181+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73415bd60 con 0x7fe7340cc5d0 2023-10-16T09:59:20.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.182+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73415c280 con 0x7fe7340cc5d0 2023-10-16T09:59:20.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.182+0000 7fe730ff9700 1 -- 172.21.15.158:0/631524699 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe724003800 con 0x7fe7340cc5d0 2023-10-16T09:59:20.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.183+0000 7fe730ff9700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe72006b020 0x7fe72006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.183+0000 7fe730ff9700 1 -- 172.21.15.158:0/631524699 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5601+0+0 (secure 0 0 0) 0x7fe724089680 con 0x7fe7340cc5d0 2023-10-16T09:59:20.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.183+0000 7fe733fff700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe72006b020 0x7fe72006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.183+0000 7fe733fff700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe72006b020 0x7fe72006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe728000f60 tx=0x7fe728007c90 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.187+0000 7fe730ff9700 1 -- 172.21.15.158:0/631524699 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe73415c280 con 0x7fe7340cc5d0 2023-10-16T09:59:20.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.378+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe73415bbc0 con 0x7fe7340cc5d0 2023-10-16T09:59:20.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.379+0000 7fe730ff9700 1 -- 172.21.15.158:0/631524699 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+13719 (secure 0 0 0) 0x7fe73415bbc0 con 0x7fe7340cc5d0 2023-10-16T09:59:20.380 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:20.381 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":81,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:19.582608+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:20.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe72006b020 msgr2=0x7fe72006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe72006b020 0x7fe72006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe728000f60 tx=0x7fe728007c90 comp rx=0 tx=0).stop 2023-10-16T09:59:20.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7340cc5d0 msgr2=0x7fe7340ae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7340cc5d0 0x7fe7340ae570 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe72400e7e0 tx=0x7fe72400eba0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 shutdown_connections 2023-10-16T09:59:20.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe72006b020 0x7fe72006d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73415dfb0 0x7fe73415c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73406f340 0x7fe734151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 --2- 172.21.15.158:0/631524699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7340cc5d0 0x7fe7340ae570 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 >> 172.21.15.158:0/631524699 conn(0x7fe7340d02b0 msgr2=0x7fe7340b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:20.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.381+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 shutdown_connections 2023-10-16T09:59:20.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.382+0000 7fe73a281700 1 -- 172.21.15.158:0/631524699 wait complete. 2023-10-16T09:59:20.395 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:20.396 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:20.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.518+0000 7fc95312a700 1 Processor -- start 2023-10-16T09:59:20.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.518+0000 7fc95312a700 1 -- start start 2023-10-16T09:59:20.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.519+0000 7fc95312a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc94c15dad0 0x7fc94c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.519+0000 7fc95312a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c0d0ae0 con 0x7fc94c15deb0 2023-10-16T09:59:20.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.519+0000 7fc95312a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c0d12d0 con 0x7fc94c0ccc00 2023-10-16T09:59:20.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.519+0000 7fc95312a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c0cbdf0 con 0x7fc94c15dad0 2023-10-16T09:59:20.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.519+0000 7fc950ec6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc94c15dad0 0x7fc94c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.519+0000 7fc94bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc94c15deb0 0x7fc94c0ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52304/0 (socket says 172.21.15.158:52304) 2023-10-16T09:59:20.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.519+0000 7fc94bfff700 1 -- 172.21.15.158:0/4161338258 learned_addr learned my addr 172.21.15.158:0/4161338258 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:20.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc950ec6700 1 -- 172.21.15.158:0/4161338258 >> v1:172.21.15.172:6789/0 conn(0x7fc94c0ccc00 legacy=0x7fc94c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:20.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc950ec6700 1 -- 172.21.15.158:0/4161338258 >> v1:172.21.15.158:6789/0 conn(0x7fc94c15deb0 legacy=0x7fc94c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc950ec6700 1 -- 172.21.15.158:0/4161338258 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c0cc280 con 0x7fc94c15dad0 2023-10-16T09:59:20.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc94b7fe700 1 -- 172.21.15.158:0/4161338258 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784169907 0 0) 0x7fc94c0d0ae0 con 0x7fc94c15deb0 2023-10-16T09:59:20.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc950ec6700 1 --2- 172.21.15.158:0/4161338258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0cdfa0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc940009870 tx=0x7fc940009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d102631e236c678d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc94b7fe700 1 -- 172.21.15.158:0/4161338258 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940011830 con 0x7fc94c15dad0 2023-10-16T09:59:20.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc94b7fe700 1 -- 172.21.15.158:0/4161338258 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc940011990 con 0x7fc94c15dad0 2023-10-16T09:59:20.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.520+0000 7fc94b7fe700 1 -- 172.21.15.158:0/4161338258 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940011d20 con 0x7fc94c15dad0 2023-10-16T09:59:20.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.521+0000 7fc95312a700 1 -- 172.21.15.158:0/4161338258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 msgr2=0x7fc94c0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.521+0000 7fc95312a700 1 --2- 172.21.15.158:0/4161338258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0cdfa0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc940009870 tx=0x7fc940009b80 comp rx=0 tx=0).stop 2023-10-16T09:59:20.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.521+0000 7fc95312a700 1 -- 172.21.15.158:0/4161338258 shutdown_connections 2023-10-16T09:59:20.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.521+0000 7fc95312a700 1 --2- 172.21.15.158:0/4161338258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0cdfa0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.521+0000 7fc95312a700 1 -- 172.21.15.158:0/4161338258 >> 172.21.15.158:0/4161338258 conn(0x7fc94c099a10 msgr2=0x7fc94c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:20.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.521+0000 7fc95312a700 1 -- 172.21.15.158:0/4161338258 shutdown_connections 2023-10-16T09:59:20.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.521+0000 7fc95312a700 1 -- 172.21.15.158:0/4161338258 wait complete. 2023-10-16T09:59:20.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.522+0000 7fc95312a700 1 Processor -- start 2023-10-16T09:59:20.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.522+0000 7fc95312a700 1 -- start start 2023-10-16T09:59:20.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc94c0ccc00 0x7fc94c06ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc94c15deb0 0x7fc94c0b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c0c82f0 con 0x7fc94c15dad0 2023-10-16T09:59:20.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c0b5b40 con 0x7fc94c0ccc00 2023-10-16T09:59:20.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c0b5e20 con 0x7fc94c15deb0 2023-10-16T09:59:20.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52134/0 (socket says 172.21.15.158:52134) 2023-10-16T09:59:20.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 -- 172.21.15.158:0/4052193771 learned_addr learned my addr 172.21.15.158:0/4052193771 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:20.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc950ec6700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc94c0ccc00 0x7fc94c06ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc9516c7700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc94c15deb0 0x7fc94c0b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 -- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc94c15deb0 msgr2=0x7fc94c0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc94c15deb0 0x7fc94c0b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 -- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc94c0ccc00 msgr2=0x7fc94c06ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc94c0ccc00 0x7fc94c06ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 -- 172.21.15.158:0/4052193771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400094a0 con 0x7fc94c15dad0 2023-10-16T09:59:20.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc950ec6700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc94c0ccc00 0x7fc94c06ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:20.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc94bfff700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0b5140 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc93800c520 tx=0x7fc93800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc9497fa700 1 -- 172.21.15.158:0/4052193771 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938004c80 con 0x7fc94c15dad0 2023-10-16T09:59:20.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc9497fa700 1 -- 172.21.15.158:0/4052193771 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc938004de0 con 0x7fc94c15dad0 2023-10-16T09:59:20.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c0c7c30 con 0x7fc94c15dad0 2023-10-16T09:59:20.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc9497fa700 1 -- 172.21.15.158:0/4052193771 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938007310 con 0x7fc94c15dad0 2023-10-16T09:59:20.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.523+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c06b080 con 0x7fc94c15dad0 2023-10-16T09:59:20.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.524+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c074a90 con 0x7fc94c15dad0 2023-10-16T09:59:20.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.525+0000 7fc9497fa700 1 -- 172.21.15.158:0/4052193771 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc938007470 con 0x7fc94c15dad0 2023-10-16T09:59:20.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.525+0000 7fc9497fa700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc93006b020 0x7fc93006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.526+0000 7fc9497fa700 1 -- 172.21.15.158:0/4052193771 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5601+0+0 (secure 0 0 0) 0x7fc938089910 con 0x7fc94c15dad0 2023-10-16T09:59:20.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.526+0000 7fc950ec6700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc93006b020 0x7fc93006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.526+0000 7fc950ec6700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc93006b020 0x7fc93006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9400090b0 tx=0x7fc940008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.529+0000 7fc9497fa700 1 -- 172.21.15.158:0/4052193771 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc938054980 con 0x7fc94c15dad0 2023-10-16T09:59:20.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.720+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc94c0681b0 con 0x7fc94c15dad0 2023-10-16T09:59:20.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.721+0000 7fc9497fa700 1 -- 172.21.15.158:0/4052193771 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc938057fa0 con 0x7fc94c15dad0 2023-10-16T09:59:20.723 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:20.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc93006b020 msgr2=0x7fc93006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc93006b020 0x7fc93006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9400090b0 tx=0x7fc940008030 comp rx=0 tx=0).stop 2023-10-16T09:59:20.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 msgr2=0x7fc94c0b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0b5140 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc93800c520 tx=0x7fc93800c830 comp rx=0 tx=0).stop 2023-10-16T09:59:20.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 shutdown_connections 2023-10-16T09:59:20.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc93006b020 0x7fc93006d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc94c15deb0 0x7fc94c0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc94c0ccc00 0x7fc94c06ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 --2- 172.21.15.158:0/4052193771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc94c15dad0 0x7fc94c0b5140 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 >> 172.21.15.158:0/4052193771 conn(0x7fc94c099a10 msgr2=0x7fc94c06f170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:20.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 shutdown_connections 2023-10-16T09:59:20.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.723+0000 7fc95312a700 1 -- 172.21.15.158:0/4052193771 wait complete. 2023-10-16T09:59:20.737 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T09:59:20.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.856+0000 7f20baf9c700 1 Processor -- start 2023-10-16T09:59:20.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.856+0000 7f20baf9c700 1 -- start start 2023-10-16T09:59:20.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.857+0000 7f20baf9c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20b40ce640 0x7f20b40cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.857+0000 7f20baf9c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b40cef60 con 0x7f20b40c65b0 2023-10-16T09:59:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.857+0000 7f20baf9c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b415d800 con 0x7f20b40cae30 2023-10-16T09:59:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.857+0000 7f20baf9c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b415e480 con 0x7f20b40ce640 2023-10-16T09:59:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.857+0000 7f20b8d38700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20b40ce640 0x7f20b40cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.857+0000 7f20b3fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f20b40c65b0 0x7f20b406f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52318/0 (socket says 172.21.15.158:52318) 2023-10-16T09:59:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.857+0000 7f20b3fff700 1 -- 172.21.15.158:0/4278572815 learned_addr learned my addr 172.21.15.158:0/4278572815 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20b8d38700 1 -- 172.21.15.158:0/4278572815 >> v1:172.21.15.172:6789/0 conn(0x7f20b40cae30 legacy=0x7f20b40cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20b8d38700 1 -- 172.21.15.158:0/4278572815 >> v1:172.21.15.158:6789/0 conn(0x7f20b40c65b0 legacy=0x7f20b406f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20b8d38700 1 -- 172.21.15.158:0/4278572815 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b415dae0 con 0x7f20b40ce640 2023-10-16T09:59:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20b8d38700 1 --2- 172.21.15.158:0/4278572815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40ce640 0x7f20b40cea20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f20a800bae0 tx=0x7f20a800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29dc799cebf72c9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20b37fe700 1 -- 172.21.15.158:0/4278572815 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a800dbe0 con 0x7f20b40ce640 2023-10-16T09:59:20.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20b37fe700 1 -- 172.21.15.158:0/4278572815 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a800dd40 con 0x7f20b40ce640 2023-10-16T09:59:20.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20b37fe700 1 -- 172.21.15.158:0/4278572815 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a8004270 con 0x7f20b40ce640 2023-10-16T09:59:20.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20baf9c700 1 -- 172.21.15.158:0/4278572815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40ce640 msgr2=0x7f20b40cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.858+0000 7f20baf9c700 1 --2- 172.21.15.158:0/4278572815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40ce640 0x7f20b40cea20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f20a800bae0 tx=0x7f20a800bdf0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.859+0000 7f20baf9c700 1 -- 172.21.15.158:0/4278572815 shutdown_connections 2023-10-16T09:59:20.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.859+0000 7f20baf9c700 1 --2- 172.21.15.158:0/4278572815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40ce640 0x7f20b40cea20 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.859+0000 7f20baf9c700 1 -- 172.21.15.158:0/4278572815 >> 172.21.15.158:0/4278572815 conn(0x7f20b40cc4b0 msgr2=0x7f20b40cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:20.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.859+0000 7f20baf9c700 1 -- 172.21.15.158:0/4278572815 shutdown_connections 2023-10-16T09:59:20.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.859+0000 7f20baf9c700 1 -- 172.21.15.158:0/4278572815 wait complete. 2023-10-16T09:59:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.860+0000 7f20baf9c700 1 Processor -- start 2023-10-16T09:59:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.860+0000 7f20baf9c700 1 -- start start 2023-10-16T09:59:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20b40c65b0 0x7f20b413e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40cae30 0x7f20b413f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20b40ce640 0x7f20b413ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40c85e0 con 0x7f20b40cae30 2023-10-16T09:59:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b413fb50 con 0x7f20b40c65b0 2023-10-16T09:59:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40c8300 con 0x7f20b40ce640 2023-10-16T09:59:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40cae30 0x7f20b413f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40cae30 0x7f20b413f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52148/0 (socket says 172.21.15.158:52148) 2023-10-16T09:59:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 -- 172.21.15.158:0/2841765424 learned_addr learned my addr 172.21.15.158:0/2841765424 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b8d38700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20b40c65b0 0x7f20b413e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b9539700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20b40ce640 0x7f20b413ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 -- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20b40ce640 msgr2=0x7f20b413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20b40ce640 0x7f20b413ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 -- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20b40c65b0 msgr2=0x7f20b413e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20b40c65b0 0x7f20b413e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 -- 172.21.15.158:0/2841765424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a800b740 con 0x7f20b40cae30 2023-10-16T09:59:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b8d38700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20b40c65b0 0x7f20b413e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:20.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b3fff700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40cae30 0x7f20b413f610 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c430 tx=0x7f20a000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b17fa700 1 -- 172.21.15.158:0/2841765424 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0005d70 con 0x7f20b40cae30 2023-10-16T09:59:20.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b17fa700 1 -- 172.21.15.158:0/2841765424 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a00070d0 con 0x7f20b40cae30 2023-10-16T09:59:20.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b41e4150 con 0x7f20b40cae30 2023-10-16T09:59:20.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20b17fa700 1 -- 172.21.15.158:0/2841765424 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0007460 con 0x7f20b40cae30 2023-10-16T09:59:20.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.861+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b41e4710 con 0x7f20b40cae30 2023-10-16T09:59:20.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.862+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b41e4930 con 0x7f20b40cae30 2023-10-16T09:59:20.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.863+0000 7f20b17fa700 1 -- 172.21.15.158:0/2841765424 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f20a00075c0 con 0x7f20b40cae30 2023-10-16T09:59:20.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.863+0000 7f20b17fa700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f209806b020 0x7f209806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:20.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.863+0000 7f20b17fa700 1 -- 172.21.15.158:0/2841765424 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f20a0088ba0 con 0x7f20b40cae30 2023-10-16T09:59:20.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.863+0000 7f20b8d38700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f209806b020 0x7f209806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:20.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.864+0000 7f20b8d38700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f209806b020 0x7f209806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20a8019040 tx=0x7f20a8008040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:20.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:20.868+0000 7f20b17fa700 1 -- 172.21.15.158:0/2841765424 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f20a0053b20 con 0x7f20b40cae30 2023-10-16T09:59:21.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.064+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f20b40681b0 con 0x7f20b40cae30 2023-10-16T09:59:21.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.065+0000 7f20b17fa700 1 -- 172.21.15.158:0/2841765424 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f20a0057140 con 0x7f20b40cae30 2023-10-16T09:59:21.067 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:59:21.067 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:21.067 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T09:59:21.068 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T09:59:21.068 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:59:21.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f209806b020 msgr2=0x7f209806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:21.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f209806b020 0x7f209806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20a8019040 tx=0x7f20a8008040 comp rx=0 tx=0).stop 2023-10-16T09:59:21.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40cae30 msgr2=0x7f20b413f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40cae30 0x7f20b413f610 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c430 tx=0x7f20a000c7f0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 shutdown_connections 2023-10-16T09:59:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f209806b020 0x7f209806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20b40ce640 0x7f20b413ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20b40c65b0 0x7f20b413e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 --2- 172.21.15.158:0/2841765424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20b40cae30 0x7f20b413f610 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 >> 172.21.15.158:0/2841765424 conn(0x7f20b40cc4b0 msgr2=0x7f20b406f420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 shutdown_connections 2023-10-16T09:59:21.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.067+0000 7f20baf9c700 1 -- 172.21.15.158:0/2841765424 wait complete. 2023-10-16T09:59:21.081 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T09:59:21.081 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T09:59:21.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.205+0000 7f9a1bc39700 1 Processor -- start 2023-10-16T09:59:21.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.205+0000 7f9a1bc39700 1 -- start start 2023-10-16T09:59:21.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1bc39700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1415da40 0x7f9a140caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:21.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1bc39700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a1406afe0 0x7f9a140afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:21.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1bc39700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140cbd30 con 0x7f9a140cc590 2023-10-16T09:59:21.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1bc39700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a140cc1c0 con 0x7f9a1406afe0 2023-10-16T09:59:21.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1bc39700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1415de20 con 0x7f9a1415da40 2023-10-16T09:59:21.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1a1d6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a1406afe0 0x7f9a140afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:21.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1a1d6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a1406afe0 0x7f9a140afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52158/0 (socket says 172.21.15.158:52158) 2023-10-16T09:59:21.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.206+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/925024520 learned_addr learned my addr 172.21.15.158:0/925024520 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:21.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a199d5700 1 --2- 172.21.15.158:0/925024520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1415da40 0x7f9a140caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:21.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/925024520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1415da40 msgr2=0x7f9a140caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:21.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1a1d6700 1 --2- 172.21.15.158:0/925024520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1415da40 0x7f9a140caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/925024520 >> v1:172.21.15.172:6789/0 conn(0x7f9a140cc590 legacy=0x7f9a1406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:21.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/925024520 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a140ce260 con 0x7f9a1406afe0 2023-10-16T09:59:21.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a199d5700 1 --2- 172.21.15.158:0/925024520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1415da40 0x7f9a140caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:21.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1a1d6700 1 --2- 172.21.15.158:0/925024520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1406afe0 0x7f9a140afa40 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009e50 tx=0x7f9a0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02e2a840d1d9ea8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:21.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a189d3700 1 -- 172.21.15.158:0/925024520 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c00e070 con 0x7f9a1406afe0 2023-10-16T09:59:21.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a189d3700 1 -- 172.21.15.158:0/925024520 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c003a20 con 0x7f9a1406afe0 2023-10-16T09:59:21.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a189d3700 1 -- 172.21.15.158:0/925024520 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c003dd0 con 0x7f9a1406afe0 2023-10-16T09:59:21.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1bc39700 1 -- 172.21.15.158:0/925024520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1406afe0 msgr2=0x7f9a140afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:21.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/925024520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1406afe0 0x7f9a140afa40 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009e50 tx=0x7f9a0c00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:21.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1bc39700 1 -- 172.21.15.158:0/925024520 shutdown_connections 2023-10-16T09:59:21.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/925024520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1406afe0 0x7f9a140afa40 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/925024520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1415da40 0x7f9a140caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.207+0000 7f9a1bc39700 1 -- 172.21.15.158:0/925024520 >> 172.21.15.158:0/925024520 conn(0x7f9a14099a10 msgr2=0x7f9a140d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:21.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.208+0000 7f9a1bc39700 1 -- 172.21.15.158:0/925024520 shutdown_connections 2023-10-16T09:59:21.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.208+0000 7f9a1bc39700 1 -- 172.21.15.158:0/925024520 wait complete. 2023-10-16T09:59:21.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 Processor -- start 2023-10-16T09:59:21.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 -- start start 2023-10-16T09:59:21.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1406afe0 0x7f9a1409cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:21.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a140cc590 0x7f9a140c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:21.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1415da40 0x7f9a1409a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:21.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1409da30 con 0x7f9a1415da40 2023-10-16T09:59:21.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a140b49d0 con 0x7f9a140cc590 2023-10-16T09:59:21.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.209+0000 7f9a1bc39700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1409d190 con 0x7f9a1406afe0 2023-10-16T09:59:21.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1415da40 0x7f9a1409a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:21.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1415da40 0x7f9a1409a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52164/0 (socket says 172.21.15.158:52164) 2023-10-16T09:59:21.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/3999289845 learned_addr learned my addr 172.21.15.158:0/3999289845 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:21.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a199d5700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1406afe0 0x7f9a1409cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:21.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a191d4700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a140cc590 0x7f9a140c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:21.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1406afe0 msgr2=0x7f9a1409cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:21.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1406afe0 0x7f9a1409cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a140cc590 msgr2=0x7f9a140c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:21.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a140cc590 0x7f9a140c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:21.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 -- 172.21.15.158:0/3999289845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c009b30 con 0x7f9a1415da40 2023-10-16T09:59:21.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a191d4700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a140cc590 0x7f9a140c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:21.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a199d5700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1406afe0 0x7f9a1409cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:21.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1a1d6700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1415da40 0x7f9a1409a400 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9a140702f0 tx=0x7f9a0c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:21.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a0a7fc700 1 -- 172.21.15.158:0/3999289845 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c004240 con 0x7f9a1415da40 2023-10-16T09:59:21.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a0a7fc700 1 -- 172.21.15.158:0/3999289845 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c0043a0 con 0x7f9a1415da40 2023-10-16T09:59:21.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a0a7fc700 1 -- 172.21.15.158:0/3999289845 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c004750 con 0x7f9a1415da40 2023-10-16T09:59:21.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1409d470 con 0x7f9a1415da40 2023-10-16T09:59:21.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.210+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a140c8980 con 0x7f9a1415da40 2023-10-16T09:59:21.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.211+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a140681b0 con 0x7f9a1415da40 2023-10-16T09:59:21.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.212+0000 7f9a0a7fc700 1 -- 172.21.15.158:0/3999289845 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a0c019070 con 0x7f9a1415da40 2023-10-16T09:59:21.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.212+0000 7f9a0a7fc700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f99f806b020 0x7f99f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:21.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.212+0000 7f9a199d5700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f99f806b020 0x7f99f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:21.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.212+0000 7f9a0a7fc700 1 -- 172.21.15.158:0/3999289845 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f9a0c08a750 con 0x7f9a1415da40 2023-10-16T09:59:21.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.213+0000 7f9a199d5700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f99f806b020 0x7f99f806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a04006fd0 tx=0x7f9a04008040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:21.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.217+0000 7f9a0a7fc700 1 -- 172.21.15.158:0/3999289845 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a0c055650 con 0x7f9a1415da40 2023-10-16T09:59:21.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:21.409+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9a140b4b10 con 0x7f9a1415da40 2023-10-16T09:59:22.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.115+0000 7f9a0a7fc700 1 -- 172.21.15.158:0/3999289845 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9a0c058c70 con 0x7f9a1415da40 2023-10-16T09:59:22.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f99f806b020 msgr2=0x7f99f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:22.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f99f806b020 0x7f99f806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a04006fd0 tx=0x7f9a04008040 comp rx=0 tx=0).stop 2023-10-16T09:59:22.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1415da40 msgr2=0x7f9a1409a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:22.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1415da40 0x7f9a1409a400 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9a140702f0 tx=0x7f9a0c004ad0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 shutdown_connections 2023-10-16T09:59:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f99f806b020 0x7f99f806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1415da40 0x7f9a1409a400 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a1406afe0 0x7f9a1409cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 --2- 172.21.15.158:0/3999289845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a140cc590 0x7f9a140c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 >> 172.21.15.158:0/3999289845 conn(0x7f9a14099a10 msgr2=0x7f9a1415c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:22.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 shutdown_connections 2023-10-16T09:59:22.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.118+0000 7f9a1bc39700 1 -- 172.21.15.158:0/3999289845 wait complete. 2023-10-16T09:59:22.122 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T09:59:22.138 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T09:59:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.270+0000 7f2aa7694700 1 Processor -- start 2023-10-16T09:59:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.271+0000 7f2aa7694700 1 -- start start 2023-10-16T09:59:22.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.271+0000 7f2aa7694700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2aa00af5f0 0x7f2aa0161e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:22.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.271+0000 7f2aa7694700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2aa00b3b60 0x7f2aa00724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:22.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa7694700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa0071d70 con 0x7f2aa0072a20 2023-10-16T09:59:22.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa7694700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00af9d0 con 0x7f2aa00af5f0 2023-10-16T09:59:22.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa7694700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00b3970 con 0x7f2aa00b3b60 2023-10-16T09:59:22.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5430700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2aa00af5f0 0x7f2aa0161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:22.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5c31700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2aa0072a20 0x7f2aa006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52348/0 (socket says 172.21.15.158:52348) 2023-10-16T09:59:22.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5c31700 1 -- 172.21.15.158:0/2526780657 learned_addr learned my addr 172.21.15.158:0/2526780657 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:22.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa4c2f700 1 --2- 172.21.15.158:0/2526780657 >> v2:172.21.15.172:3300/0 conn(0x7f2aa00b3b60 0x7f2aa00724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:22.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5430700 1 -- 172.21.15.158:0/2526780657 >> v2:172.21.15.172:3300/0 conn(0x7f2aa00b3b60 msgr2=0x7f2aa00724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:22.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5430700 1 --2- 172.21.15.158:0/2526780657 >> v2:172.21.15.172:3300/0 conn(0x7f2aa00b3b60 0x7f2aa00724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5430700 1 -- 172.21.15.158:0/2526780657 >> v1:172.21.15.158:6789/0 conn(0x7f2aa0072a20 legacy=0x7f2aa006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:22.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2a93fff700 1 -- 172.21.15.158:0/2526780657 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252090068 0 0) 0x7f2aa0071d70 con 0x7f2aa0072a20 2023-10-16T09:59:22.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5430700 1 -- 172.21.15.158:0/2526780657 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0162370 con 0x7f2aa00af5f0 2023-10-16T09:59:22.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa4c2f700 1 --2- 172.21.15.158:0/2526780657 >> v2:172.21.15.172:3300/0 conn(0x7f2aa00b3b60 0x7f2aa00724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:22.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2aa5430700 1 --2- 172.21.15.158:0/2526780657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00af5f0 0x7f2aa0161e30 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2aa006c9c0 tx=0x7f2a94009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe39cbbb102f87d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:22.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2a93fff700 1 -- 172.21.15.158:0/2526780657 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94011830 con 0x7f2aa00af5f0 2023-10-16T09:59:22.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2a93fff700 1 -- 172.21.15.158:0/2526780657 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94011990 con 0x7f2aa00af5f0 2023-10-16T09:59:22.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.272+0000 7f2a93fff700 1 -- 172.21.15.158:0/2526780657 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94011d20 con 0x7f2aa00af5f0 2023-10-16T09:59:22.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 -- 172.21.15.158:0/2526780657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00af5f0 msgr2=0x7f2aa0161e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:22.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 --2- 172.21.15.158:0/2526780657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00af5f0 0x7f2aa0161e30 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2aa006c9c0 tx=0x7f2a94009b20 comp rx=0 tx=0).stop 2023-10-16T09:59:22.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 -- 172.21.15.158:0/2526780657 shutdown_connections 2023-10-16T09:59:22.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 --2- 172.21.15.158:0/2526780657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00af5f0 0x7f2aa0161e30 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 --2- 172.21.15.158:0/2526780657 >> v2:172.21.15.172:3300/0 conn(0x7f2aa00b3b60 0x7f2aa00724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 -- 172.21.15.158:0/2526780657 >> 172.21.15.158:0/2526780657 conn(0x7f2aa00ccd30 msgr2=0x7f2aa00cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:22.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 -- 172.21.15.158:0/2526780657 shutdown_connections 2023-10-16T09:59:22.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.273+0000 7f2aa7694700 1 -- 172.21.15.158:0/2526780657 wait complete. 2023-10-16T09:59:22.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.274+0000 7f2aa7694700 1 Processor -- start 2023-10-16T09:59:22.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.274+0000 7f2aa7694700 1 -- start start 2023-10-16T09:59:22.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa7694700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa0072a20 0x7f2aa0156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:22.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa7694700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa00af5f0 0x7f2aa00b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:22.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa7694700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00b3b60 0x7f2aa00b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:22.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa7694700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0157100 con 0x7f2aa00b3b60 2023-10-16T09:59:22.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa7694700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0156bf0 con 0x7f2aa00af5f0 2023-10-16T09:59:22.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa7694700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa01573a0 con 0x7f2aa0072a20 2023-10-16T09:59:22.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00b3b60 0x7f2aa00b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:22.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00b3b60 0x7f2aa00b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52178/0 (socket says 172.21.15.158:52178) 2023-10-16T09:59:22.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 -- 172.21.15.158:0/4112497348 learned_addr learned my addr 172.21.15.158:0/4112497348 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:22.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5430700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa0072a20 0x7f2aa0156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:22.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa4c2f700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa00af5f0 0x7f2aa00b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:22.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 -- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa0072a20 msgr2=0x7f2aa0156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:22.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa0072a20 0x7f2aa0156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 -- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa00af5f0 msgr2=0x7f2aa00b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:22.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa00af5f0 0x7f2aa00b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:22.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 -- 172.21.15.158:0/4112497348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940094a0 con 0x7f2aa00b3b60 2023-10-16T09:59:22.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.275+0000 7f2aa5c31700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00b3b60 0x7f2aa00b0560 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a98002230 tx=0x7f2a9800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:22.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.276+0000 7f2a91ffb700 1 -- 172.21.15.158:0/4112497348 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9800ac20 con 0x7f2aa00b3b60 2023-10-16T09:59:22.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.276+0000 7f2a91ffb700 1 -- 172.21.15.158:0/4112497348 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9800ada0 con 0x7f2aa00b3b60 2023-10-16T09:59:22.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.276+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa0157680 con 0x7f2aa00b3b60 2023-10-16T09:59:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.276+0000 7f2a91ffb700 1 -- 172.21.15.158:0/4112497348 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98007310 con 0x7f2aa00b3b60 2023-10-16T09:59:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.276+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa0156670 con 0x7f2aa00b3b60 2023-10-16T09:59:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.277+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa00681b0 con 0x7f2aa00b3b60 2023-10-16T09:59:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.277+0000 7f2a91ffb700 1 -- 172.21.15.158:0/4112497348 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2a98007470 con 0x7f2aa00b3b60 2023-10-16T09:59:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.278+0000 7f2a91ffb700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2a8406b020 0x7f2a8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.278+0000 7f2a91ffb700 1 -- 172.21.15.158:0/4112497348 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5971+0+0 (secure 0 0 0) 0x7f2a980897f0 con 0x7f2aa00b3b60 2023-10-16T09:59:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.278+0000 7f2aa5430700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2a8406b020 0x7f2a8406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.278+0000 7f2aa5430700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2a8406b020 0x7f2a8406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2a940090b0 tx=0x7f2a94008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.282+0000 7f2a91ffb700 1 -- 172.21.15.158:0/4112497348 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a980545f0 con 0x7f2aa00b3b60 2023-10-16T09:59:22.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:22.474+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2aa00cd960 con 0x7f2aa00b3b60 2023-10-16T09:59:23.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.119+0000 7f2a91ffb700 1 -- 172.21.15.158:0/4112497348 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2a98018070 con 0x7f2aa00b3b60 2023-10-16T09:59:23.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2a8406b020 msgr2=0x7f2a8406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:23.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2a8406b020 0x7f2a8406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2a940090b0 tx=0x7f2a94008030 comp rx=0 tx=0).stop 2023-10-16T09:59:23.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00b3b60 msgr2=0x7f2aa00b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:23.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00b3b60 0x7f2aa00b0560 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a98002230 tx=0x7f2a9800a060 comp rx=0 tx=0).stop 2023-10-16T09:59:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 shutdown_connections 2023-10-16T09:59:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f2a8406b020 0x7f2a8406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aa00b3b60 0x7f2aa00b0560 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa0072a20 0x7f2aa0156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 --2- 172.21.15.158:0/4112497348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa00af5f0 0x7f2aa00b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 >> 172.21.15.158:0/4112497348 conn(0x7f2aa00ccd30 msgr2=0x7f2aa00cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:23.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 shutdown_connections 2023-10-16T09:59:23.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.122+0000 7f2aa7694700 1 -- 172.21.15.158:0/4112497348 wait complete. 2023-10-16T09:59:23.131 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T09:59:23.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T09:59:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.268+0000 7f8b62cea700 1 Processor -- start 2023-10-16T09:59:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.268+0000 7f8b62cea700 1 -- start start 2023-10-16T09:59:23.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b62cea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c15da40 0x7f8b5c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:23.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b62cea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b5c0cc590 0x7f8b5c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b62cea700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b5c06ae10 0x7f8b5c0ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b62cea700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5c06b280 con 0x7f8b5c06ae10 2023-10-16T09:59:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b62cea700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5c0cbd30 con 0x7f8b5c0cc590 2023-10-16T09:59:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b62cea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c0cc1c0 con 0x7f8b5c15da40 2023-10-16T09:59:23.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b61287700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b5c06ae10 0x7f8b5c0ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:23.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b61287700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b5c06ae10 0x7f8b5c0ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52186/0 (socket says 172.21.15.158:52186) 2023-10-16T09:59:23.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.269+0000 7f8b61287700 1 -- 172.21.15.158:0/647064401 learned_addr learned my addr 172.21.15.158:0/647064401 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:23.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b5bfff700 1 --2- 172.21.15.158:0/647064401 >> v2:172.21.15.172:3300/0 conn(0x7f8b5c0cc590 0x7f8b5c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:23.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b61287700 1 -- 172.21.15.158:0/647064401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c15da40 msgr2=0x7f8b5c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:23.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b60a86700 1 --2- 172.21.15.158:0/647064401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c15da40 0x7f8b5c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:23.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b61287700 1 --2- 172.21.15.158:0/647064401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c15da40 0x7f8b5c0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b61287700 1 -- 172.21.15.158:0/647064401 >> v2:172.21.15.172:3300/0 conn(0x7f8b5c0cc590 msgr2=0x7f8b5c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:23.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b61287700 1 --2- 172.21.15.158:0/647064401 >> v2:172.21.15.172:3300/0 conn(0x7f8b5c0cc590 0x7f8b5c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b61287700 1 -- 172.21.15.158:0/647064401 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c15de20 con 0x7f8b5c06ae10 2023-10-16T09:59:23.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b60a86700 1 --2- 172.21.15.158:0/647064401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c15da40 0x7f8b5c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:23.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b61287700 1 --2- 172.21.15.158:0/647064401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c0ccd60 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c070430 tx=0x7f8b5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5d10d506859605 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:23.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b5b7fe700 1 -- 172.21.15.158:0/647064401 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400e070 con 0x7f8b5c06ae10 2023-10-16T09:59:23.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b5b7fe700 1 -- 172.21.15.158:0/647064401 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5400b9c0 con 0x7f8b5c06ae10 2023-10-16T09:59:23.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.270+0000 7f8b5b7fe700 1 -- 172.21.15.158:0/647064401 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400bd70 con 0x7f8b5c06ae10 2023-10-16T09:59:23.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 -- 172.21.15.158:0/647064401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 msgr2=0x7f8b5c0ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:23.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 --2- 172.21.15.158:0/647064401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c0ccd60 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c070430 tx=0x7f8b5400b330 comp rx=0 tx=0).stop 2023-10-16T09:59:23.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 -- 172.21.15.158:0/647064401 shutdown_connections 2023-10-16T09:59:23.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 --2- 172.21.15.158:0/647064401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c0ccd60 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 --2- 172.21.15.158:0/647064401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c15da40 0x7f8b5c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 --2- 172.21.15.158:0/647064401 >> v2:172.21.15.172:3300/0 conn(0x7f8b5c0cc590 0x7f8b5c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 -- 172.21.15.158:0/647064401 >> 172.21.15.158:0/647064401 conn(0x7f8b5c099a10 msgr2=0x7f8b5c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:23.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 -- 172.21.15.158:0/647064401 shutdown_connections 2023-10-16T09:59:23.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.271+0000 7f8b62cea700 1 -- 172.21.15.158:0/647064401 wait complete. 2023-10-16T09:59:23.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.272+0000 7f8b62cea700 1 Processor -- start 2023-10-16T09:59:23.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.272+0000 7f8b62cea700 1 -- start start 2023-10-16T09:59:23.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b62cea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c1e8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:23.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b62cea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cc590 0x7f8b5c147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:23.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b62cea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c15da40 0x7f8b5c147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:23.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b62cea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c147dc0 con 0x7f8b5c06ae10 2023-10-16T09:59:23.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b62cea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c0cf710 con 0x7f8b5c15da40 2023-10-16T09:59:23.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b62cea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c0cf920 con 0x7f8b5c0cc590 2023-10-16T09:59:23.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c1e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:23.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c1e8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52188/0 (socket says 172.21.15.158:52188) 2023-10-16T09:59:23.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 -- 172.21.15.158:0/3050531811 learned_addr learned my addr 172.21.15.158:0/3050531811 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:23.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b5bfff700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cc590 0x7f8b5c147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:23.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b61287700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c15da40 0x7f8b5c147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:23.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 -- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cc590 msgr2=0x7f8b5c147340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:23.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cc590 0x7f8b5c147340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 -- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c15da40 msgr2=0x7f8b5c147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:23.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c15da40 0x7f8b5c147880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:23.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 -- 172.21.15.158:0/3050531811 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b54009b30 con 0x7f8b5c06ae10 2023-10-16T09:59:23.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b60a86700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c1e8560 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c7f0 tx=0x7f8b5000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:23.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b597fa700 1 -- 172.21.15.158:0/3050531811 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50011800 con 0x7f8b5c06ae10 2023-10-16T09:59:23.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.273+0000 7f8b597fa700 1 -- 172.21.15.158:0/3050531811 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b50011960 con 0x7f8b5c06ae10 2023-10-16T09:59:23.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.274+0000 7f8b597fa700 1 -- 172.21.15.158:0/3050531811 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50011cf0 con 0x7f8b5c06ae10 2023-10-16T09:59:23.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.274+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5c0cfc00 con 0x7f8b5c06ae10 2023-10-16T09:59:23.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.274+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5c0d01c0 con 0x7f8b5c06ae10 2023-10-16T09:59:23.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.274+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5c0d0530 con 0x7f8b5c06ae10 2023-10-16T09:59:23.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.275+0000 7f8b597fa700 1 -- 172.21.15.158:0/3050531811 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b50018360 con 0x7f8b5c06ae10 2023-10-16T09:59:23.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.275+0000 7f8b597fa700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8b4006b020 0x7f8b4006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:23.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.275+0000 7f8b597fa700 1 -- 172.21.15.158:0/3050531811 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f8b50089950 con 0x7f8b5c06ae10 2023-10-16T09:59:23.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.275+0000 7f8b5bfff700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8b4006b020 0x7f8b4006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:23.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.276+0000 7f8b5bfff700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8b4006b020 0x7f8b4006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c1e8250 tx=0x7f8b48008040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:23.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.280+0000 7f8b597fa700 1 -- 172.21.15.158:0/3050531811 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b50054550 con 0x7f8b5c06ae10 2023-10-16T09:59:23.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:23.514+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8b5c0681b0 con 0x7f8b5c06ae10 2023-10-16T09:59:24.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.121+0000 7f8b597fa700 1 -- 172.21.15.158:0/3050531811 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v29) v1 ==== 173+0+0 (secure 0 0 0) 0x7f8b50057b70 con 0x7f8b5c06ae10 2023-10-16T09:59:24.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.123+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8b4006b020 msgr2=0x7f8b4006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.123+0000 7f8b62cea700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8b4006b020 0x7f8b4006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c1e8250 tx=0x7f8b48008040 comp rx=0 tx=0).stop 2023-10-16T09:59:24.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.123+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 msgr2=0x7f8b5c1e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.123+0000 7f8b62cea700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c1e8560 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c7f0 tx=0x7f8b5000cb00 comp rx=0 tx=0).stop 2023-10-16T09:59:24.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 shutdown_connections 2023-10-16T09:59:24.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f8b4006b020 0x7f8b4006d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c15da40 0x7f8b5c147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b5c06ae10 0x7f8b5c1e8560 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 --2- 172.21.15.158:0/3050531811 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cc590 0x7f8b5c147340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 >> 172.21.15.158:0/3050531811 conn(0x7f8b5c099a10 msgr2=0x7f8b5c0b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:24.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 shutdown_connections 2023-10-16T09:59:24.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.124+0000 7f8b62cea700 1 -- 172.21.15.158:0/3050531811 wait complete. 2023-10-16T09:59:24.128 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 9 and data pool 10 2023-10-16T09:59:24.142 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:24.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.262+0000 7f65932a7700 1 Processor -- start 2023-10-16T09:59:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.263+0000 7f65932a7700 1 -- start start 2023-10-16T09:59:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.263+0000 7f65932a7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f658c06ab40 0x7f658c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.263+0000 7f65932a7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c0729b0 0x7f658c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f65932a7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c071e50 con 0x7f658c06e5d0 2023-10-16T09:59:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f65932a7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658c161270 con 0x7f658c06ab40 2023-10-16T09:59:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f65932a7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c161510 con 0x7f658c0729b0 2023-10-16T09:59:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f658c06ab40 0x7f658c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f658c06ab40 0x7f658c06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52196/0 (socket says 172.21.15.158:52196) 2023-10-16T09:59:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 -- 172.21.15.158:0/1310309702 learned_addr learned my addr 172.21.15.158:0/1310309702 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591844700 1 --2- 172.21.15.158:0/1310309702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c0729b0 0x7f658c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 -- 172.21.15.158:0/1310309702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c0729b0 msgr2=0x7f658c071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 --2- 172.21.15.158:0/1310309702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c0729b0 0x7f658c071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 -- 172.21.15.158:0/1310309702 >> v1:172.21.15.172:6789/0 conn(0x7f658c06e5d0 legacy=0x7f658c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 -- 172.21.15.158:0/1310309702 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c1617f0 con 0x7f658c06ab40 2023-10-16T09:59:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591844700 1 --2- 172.21.15.158:0/1310309702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c0729b0 0x7f658c071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6591043700 1 --2- 172.21.15.158:0/1310309702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c06ab40 0x7f658c06d820 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f657c0098f0 tx=0x7f657c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c7e158521c13d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6583fff700 1 -- 172.21.15.158:0/1310309702 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c004230 con 0x7f658c06ab40 2023-10-16T09:59:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6583fff700 1 -- 172.21.15.158:0/1310309702 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c004390 con 0x7f658c06ab40 2023-10-16T09:59:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.264+0000 7f6583fff700 1 -- 172.21.15.158:0/1310309702 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c0036a0 con 0x7f658c06ab40 2023-10-16T09:59:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 -- 172.21.15.158:0/1310309702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c06ab40 msgr2=0x7f658c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 --2- 172.21.15.158:0/1310309702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c06ab40 0x7f658c06d820 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f657c0098f0 tx=0x7f657c009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 -- 172.21.15.158:0/1310309702 shutdown_connections 2023-10-16T09:59:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 --2- 172.21.15.158:0/1310309702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c0729b0 0x7f658c071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 --2- 172.21.15.158:0/1310309702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c06ab40 0x7f658c06d820 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 -- 172.21.15.158:0/1310309702 >> 172.21.15.158:0/1310309702 conn(0x7f658c0d02b0 msgr2=0x7f658c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 -- 172.21.15.158:0/1310309702 shutdown_connections 2023-10-16T09:59:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.265+0000 7f65932a7700 1 -- 172.21.15.158:0/1310309702 wait complete. 2023-10-16T09:59:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.266+0000 7f65932a7700 1 Processor -- start 2023-10-16T09:59:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.266+0000 7f65932a7700 1 -- start start 2023-10-16T09:59:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f65932a7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c06ab40 0x7f658c0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f65932a7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f658c06e5d0 0x7f658c0c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f65932a7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c0729b0 0x7f658c0b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f65932a7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0ac930 con 0x7f658c0729b0 2023-10-16T09:59:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f65932a7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0acc10 con 0x7f658c06e5d0 2023-10-16T09:59:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f65932a7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0acef0 con 0x7f658c06ab40 2023-10-16T09:59:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c0729b0 0x7f658c0b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c0729b0 0x7f658c0b0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52204/0 (socket says 172.21.15.158:52204) 2023-10-16T09:59:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 -- 172.21.15.158:0/4292679698 learned_addr learned my addr 172.21.15.158:0/4292679698 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591043700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c06ab40 0x7f658c0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6590842700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f658c06e5d0 0x7f658c0c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 -- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c06ab40 msgr2=0x7f658c0ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c06ab40 0x7f658c0ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 -- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f658c06e5d0 msgr2=0x7f658c0c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f658c06e5d0 0x7f658c0c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 -- 172.21.15.158:0/4292679698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c009550 con 0x7f658c0729b0 2023-10-16T09:59:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.267+0000 7f6591844700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c0729b0 0x7f658c0b0b50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f658400cd80 tx=0x7f658400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.268+0000 7f6581ffb700 1 -- 172.21.15.158:0/4292679698 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658400f480 con 0x7f658c0729b0 2023-10-16T09:59:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.268+0000 7f6581ffb700 1 -- 172.21.15.158:0/4292679698 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658400f600 con 0x7f658c0729b0 2023-10-16T09:59:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.268+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c0ad1d0 con 0x7f658c0729b0 2023-10-16T09:59:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.268+0000 7f6581ffb700 1 -- 172.21.15.158:0/4292679698 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658400f9b0 con 0x7f658c0729b0 2023-10-16T09:59:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.268+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c0c6b50 con 0x7f658c0729b0 2023-10-16T09:59:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.269+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c0734b0 con 0x7f658c0729b0 2023-10-16T09:59:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.269+0000 7f6581ffb700 1 -- 172.21.15.158:0/4292679698 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f658400b040 con 0x7f658c0729b0 2023-10-16T09:59:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.269+0000 7f6581ffb700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f657006b020 0x7f657006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.270+0000 7f6581ffb700 1 -- 172.21.15.158:0/4292679698 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f658408a270 con 0x7f658c0729b0 2023-10-16T09:59:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.270+0000 7f6591043700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f657006b020 0x7f657006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.270+0000 7f6591043700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f657006b020 0x7f657006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f657c009130 tx=0x7f657c008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.274+0000 7f6581ffb700 1 -- 172.21.15.158:0/4292679698 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f658c0734b0 con 0x7f658c0729b0 2023-10-16T09:59:24.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.467+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f658c0ad6f0 con 0x7f658c0729b0 2023-10-16T09:59:24.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.469+0000 7f6581ffb700 1 -- 172.21.15.158:0/4292679698 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+17386 (secure 0 0 0) 0x7f658c0ad6f0 con 0x7f658c0729b0 2023-10-16T09:59:24.470 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:24.473 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":84,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:23.515537+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:21.428579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:22.493630+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:24.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f657006b020 msgr2=0x7f657006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f657006b020 0x7f657006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f657c009130 tx=0x7f657c008030 comp rx=0 tx=0).stop 2023-10-16T09:59:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c0729b0 msgr2=0x7f658c0b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c0729b0 0x7f658c0b0b50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f658400cd80 tx=0x7f658400a560 comp rx=0 tx=0).stop 2023-10-16T09:59:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 shutdown_connections 2023-10-16T09:59:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f657006b020 0x7f657006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f658c0729b0 0x7f658c0b0b50 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658c06ab40 0x7f658c0ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 --2- 172.21.15.158:0/4292679698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f658c06e5d0 0x7f658c0c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 >> 172.21.15.158:0/4292679698 conn(0x7f658c0d02b0 msgr2=0x7f658c0c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 shutdown_connections 2023-10-16T09:59:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.471+0000 7f65932a7700 1 -- 172.21.15.158:0/4292679698 wait complete. 2023-10-16T09:59:24.485 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:24.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.605+0000 7f136ea9b700 1 Processor -- start 2023-10-16T09:59:24.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.605+0000 7f136ea9b700 1 -- start start 2023-10-16T09:59:24.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136ea9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f136806ab40 0x7f136806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136ea9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13680729b0 0x7f1368071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136ea9b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1368071e50 con 0x7f136806e5d0 2023-10-16T09:59:24.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136ea9b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1368161270 con 0x7f136806ab40 2023-10-16T09:59:24.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136ea9b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1368161510 con 0x7f13680729b0 2023-10-16T09:59:24.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f136806ab40 0x7f136806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f136806ab40 0x7f136806d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52208/0 (socket says 172.21.15.158:52208) 2023-10-16T09:59:24.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 -- 172.21.15.158:0/1633123345 learned_addr learned my addr 172.21.15.158:0/1633123345 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:24.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136d038700 1 --2- 172.21.15.158:0/1633123345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13680729b0 0x7f1368071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 -- 172.21.15.158:0/1633123345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13680729b0 msgr2=0x7f1368071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:24.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 --2- 172.21.15.158:0/1633123345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13680729b0 0x7f1368071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 -- 172.21.15.158:0/1633123345 >> v1:172.21.15.172:6789/0 conn(0x7f136806e5d0 legacy=0x7f13680722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 -- 172.21.15.158:0/1633123345 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13681617f0 con 0x7f136806ab40 2023-10-16T09:59:24.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f136c837700 1 --2- 172.21.15.158:0/1633123345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806ab40 0x7f136806d820 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f135c0098f0 tx=0x7f135c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb144132713a7d2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.606+0000 7f13677fe700 1 -- 172.21.15.158:0/1633123345 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c00b8a0 con 0x7f136806ab40 2023-10-16T09:59:24.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f13677fe700 1 -- 172.21.15.158:0/1633123345 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c00ba00 con 0x7f136806ab40 2023-10-16T09:59:24.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f13677fe700 1 -- 172.21.15.158:0/1633123345 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c00bd90 con 0x7f136806ab40 2023-10-16T09:59:24.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 -- 172.21.15.158:0/1633123345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806ab40 msgr2=0x7f136806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 --2- 172.21.15.158:0/1633123345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806ab40 0x7f136806d820 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f135c0098f0 tx=0x7f135c009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:24.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 -- 172.21.15.158:0/1633123345 shutdown_connections 2023-10-16T09:59:24.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 --2- 172.21.15.158:0/1633123345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13680729b0 0x7f1368071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 --2- 172.21.15.158:0/1633123345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806ab40 0x7f136806d820 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 -- 172.21.15.158:0/1633123345 >> 172.21.15.158:0/1633123345 conn(0x7f13680d02b0 msgr2=0x7f13680d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:24.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 -- 172.21.15.158:0/1633123345 shutdown_connections 2023-10-16T09:59:24.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.607+0000 7f136ea9b700 1 -- 172.21.15.158:0/1633123345 wait complete. 2023-10-16T09:59:24.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.608+0000 7f136ea9b700 1 Processor -- start 2023-10-16T09:59:24.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.608+0000 7f136ea9b700 1 -- start start 2023-10-16T09:59:24.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f136ea9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f136806ab40 0x7f13680b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f136ea9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806e5d0 0x7f13680b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f136ea9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13680729b0 0x7f13681463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f136ea9b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13680b5170 con 0x7f136806e5d0 2023-10-16T09:59:24.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f136ea9b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13680b2c70 con 0x7f13680729b0 2023-10-16T09:59:24.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f136ea9b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1368146930 con 0x7f136806ab40 2023-10-16T09:59:24.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f1367fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806e5d0 0x7f13680b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f1367fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806e5d0 0x7f13680b2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52216/0 (socket says 172.21.15.158:52216) 2023-10-16T09:59:24.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.609+0000 7f1367fff700 1 -- 172.21.15.158:0/766478828 learned_addr learned my addr 172.21.15.158:0/766478828 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:24.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f136d038700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13680729b0 0x7f13681463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f136c837700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f136806ab40 0x7f13680b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f1367fff700 1 -- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f136806ab40 msgr2=0x7f13680b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f1367fff700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f136806ab40 0x7f13680b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f1367fff700 1 -- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13680729b0 msgr2=0x7f13681463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f1367fff700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13680729b0 0x7f13681463f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f1367fff700 1 -- 172.21.15.158:0/766478828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c009550 con 0x7f136806e5d0 2023-10-16T09:59:24.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f1367fff700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806e5d0 0x7f13680b2730 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f135400c4b0 tx=0x7f135400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f13657fa700 1 -- 172.21.15.158:0/766478828 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13540052d0 con 0x7f136806e5d0 2023-10-16T09:59:24.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f13657fa700 1 -- 172.21.15.158:0/766478828 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354005430 con 0x7f136806e5d0 2023-10-16T09:59:24.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13680b4560 con 0x7f136806e5d0 2023-10-16T09:59:24.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f13657fa700 1 -- 172.21.15.158:0/766478828 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13540038c0 con 0x7f136806e5d0 2023-10-16T09:59:24.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.610+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13680b4b20 con 0x7f136806e5d0 2023-10-16T09:59:24.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.611+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13680681b0 con 0x7f136806e5d0 2023-10-16T09:59:24.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.611+0000 7f13657fa700 1 -- 172.21.15.158:0/766478828 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1354003a20 con 0x7f136806e5d0 2023-10-16T09:59:24.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.612+0000 7f13657fa700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f134c06b020 0x7f134c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.612+0000 7f13657fa700 1 -- 172.21.15.158:0/766478828 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f1354088e90 con 0x7f136806e5d0 2023-10-16T09:59:24.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.612+0000 7f136c837700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f134c06b020 0x7f134c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.612+0000 7f136c837700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f134c06b020 0x7f134c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f135c009c40 tx=0x7f135c008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.616+0000 7f13657fa700 1 -- 172.21.15.158:0/766478828 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1354053bd0 con 0x7f136806e5d0 2023-10-16T09:59:24.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.810+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13680b6230 con 0x7f136806e5d0 2023-10-16T09:59:24.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.811+0000 7f13657fa700 1 -- 172.21.15.158:0/766478828 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+17373 (secure 0 0 0) 0x7f1354017090 con 0x7f136806e5d0 2023-10-16T09:59:24.813 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:24.813 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:24.555713+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:21.428579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:22.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:24.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f134c06b020 msgr2=0x7f134c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f134c06b020 0x7f134c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f135c009c40 tx=0x7f135c008030 comp rx=0 tx=0).stop 2023-10-16T09:59:24.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806e5d0 msgr2=0x7f13680b2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806e5d0 0x7f13680b2730 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f135400c4b0 tx=0x7f135400c870 comp rx=0 tx=0).stop 2023-10-16T09:59:24.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 shutdown_connections 2023-10-16T09:59:24.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f134c06b020 0x7f134c06d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13680729b0 0x7f13681463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f136806ab40 0x7f13680b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 --2- 172.21.15.158:0/766478828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f136806e5d0 0x7f13680b2730 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 >> 172.21.15.158:0/766478828 conn(0x7f13680d02b0 msgr2=0x7f13680ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:24.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 shutdown_connections 2023-10-16T09:59:24.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.813+0000 7f136ea9b700 1 -- 172.21.15.158:0/766478828 wait complete. 2023-10-16T09:59:24.827 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T09:59:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.950+0000 7f6bdc29c700 1 Processor -- start 2023-10-16T09:59:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.950+0000 7f6bdc29c700 1 -- start start 2023-10-16T09:59:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bdc29c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd406b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bdc29c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40c52f0 con 0x7f6bd40c49b0 2023-10-16T09:59:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bdc29c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40ce110 con 0x7f6bd4160d40 2023-10-16T09:59:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bdc29c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40c61e0 con 0x7f6bd40c4e60 2023-10-16T09:59:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bd9837700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6bd40c49b0 0x7f6bd406abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52400/0 (socket says 172.21.15.158:52400) 2023-10-16T09:59:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bd9837700 1 -- 172.21.15.158:0/1543397480 learned_addr learned my addr 172.21.15.158:0/1543397480 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:24.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bda839700 1 --2- 172.21.15.158:0/1543397480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd406b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693152102 0 0) 0x7f6bd40c52f0 con 0x7f6bd40c49b0 2023-10-16T09:59:24.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc00035b0 con 0x7f6bd40c49b0 2023-10-16T09:59:24.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558075202 0 0) 0x7f6bd40ce110 con 0x7f6bd4160d40 2023-10-16T09:59:24.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd40c52f0 con 0x7f6bd4160d40 2023-10-16T09:59:24.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2889486653 0 0) 0x7f6bc00035b0 con 0x7f6bd40c49b0 2023-10-16T09:59:24.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bda839700 1 -- 172.21.15.158:0/1543397480 >> v1:172.21.15.172:6789/0 conn(0x7f6bd4160d40 legacy=0x7f6bd40ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bda839700 1 -- 172.21.15.158:0/1543397480 >> v1:172.21.15.158:6789/0 conn(0x7f6bd40c49b0 legacy=0x7f6bd406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.951+0000 7f6bda839700 1 -- 172.21.15.158:0/1543397480 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40ce3f0 con 0x7f6bd40c4e60 2023-10-16T09:59:24.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bda839700 1 --2- 172.21.15.158:0/1543397480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd406b2c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009dc0 tx=0x7f6bcc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7025367a75a42a30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00e070 con 0x7f6bd40c4e60 2023-10-16T09:59:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc003a20 con 0x7f6bd40c4e60 2023-10-16T09:59:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bd9036700 1 -- 172.21.15.158:0/1543397480 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0001670 con 0x7f6bd40c4e60 2023-10-16T09:59:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bdc29c700 1 -- 172.21.15.158:0/1543397480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 msgr2=0x7f6bd406b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/1543397480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd406b2c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009dc0 tx=0x7f6bcc00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bdc29c700 1 -- 172.21.15.158:0/1543397480 shutdown_connections 2023-10-16T09:59:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/1543397480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd406b2c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bdc29c700 1 -- 172.21.15.158:0/1543397480 >> 172.21.15.158:0/1543397480 conn(0x7f6bd40cc5b0 msgr2=0x7f6bd40cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bdc29c700 1 -- 172.21.15.158:0/1543397480 shutdown_connections 2023-10-16T09:59:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.952+0000 7f6bdc29c700 1 -- 172.21.15.158:0/1543397480 wait complete. 2023-10-16T09:59:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.953+0000 7f6bdc29c700 1 Processor -- start 2023-10-16T09:59:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bdc29c700 1 -- start start 2023-10-16T09:59:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bdc29c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bd40c49b0 0x7f6bd40c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bdc29c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd40b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bdc29c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bd4160d40 0x7f6bd4152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bdc29c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40b58b0 con 0x7f6bd4160d40 2023-10-16T09:59:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bdc29c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40b4f80 con 0x7f6bd40c49b0 2023-10-16T09:59:24.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bdc29c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40b5260 con 0x7f6bd40c4e60 2023-10-16T09:59:24.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bda839700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bd4160d40 0x7f6bd4152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bda839700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bd4160d40 0x7f6bd4152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52232/0 (socket says 172.21.15.158:52232) 2023-10-16T09:59:24.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bda839700 1 -- 172.21.15.158:0/3900522918 learned_addr learned my addr 172.21.15.158:0/3900522918 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:24.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bd9837700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd40b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.954+0000 7f6bda038700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bd40c49b0 0x7f6bd40c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bda839700 1 -- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 msgr2=0x7f6bd40b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bda839700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd40b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bda839700 1 -- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bd40c49b0 msgr2=0x7f6bd40c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bda839700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bd40c49b0 0x7f6bd40c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bda839700 1 -- 172.21.15.158:0/3900522918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc009a70 con 0x7f6bd4160d40 2023-10-16T09:59:24.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bda038700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bd40c49b0 0x7f6bd40c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:24.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bda839700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bd4160d40 0x7f6bd4152da0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009d90 tx=0x7f6bcc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bc6ffd700 1 -- 172.21.15.158:0/3900522918 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc004540 con 0x7f6bd4160d40 2023-10-16T09:59:24.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bc6ffd700 1 -- 172.21.15.158:0/3900522918 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc0046a0 con 0x7f6bd4160d40 2023-10-16T09:59:24.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bc6ffd700 1 -- 172.21.15.158:0/3900522918 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc01d470 con 0x7f6bd4160d40 2023-10-16T09:59:24.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd40b5540 con 0x7f6bd4160d40 2023-10-16T09:59:24.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.955+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd41532e0 con 0x7f6bd4160d40 2023-10-16T09:59:24.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.956+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd4066db0 con 0x7f6bd4160d40 2023-10-16T09:59:24.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.957+0000 7f6bc6ffd700 1 -- 172.21.15.158:0/3900522918 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6bcc01d5d0 con 0x7f6bd4160d40 2023-10-16T09:59:24.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.957+0000 7f6bc6ffd700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6bb806b020 0x7f6bb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:24.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.957+0000 7f6bc6ffd700 1 -- 172.21.15.158:0/3900522918 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f6bcc08b8a0 con 0x7f6bd4160d40 2023-10-16T09:59:24.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.957+0000 7f6bda038700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6bb806b020 0x7f6bb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:24.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.957+0000 7f6bda038700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6bb806b020 0x7f6bb806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6bc80011e0 tx=0x7f6bc8009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:24.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:24.961+0000 7f6bc6ffd700 1 -- 172.21.15.158:0/3900522918 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6bcc059ee0 con 0x7f6bd4160d40 2023-10-16T09:59:25.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:25.203+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6bd40681b0 con 0x7f6bd4160d40 2023-10-16T09:59:26.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.136+0000 7f6bc6ffd700 1 -- 172.21.15.158:0/3900522918 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6bcc059a70 con 0x7f6bd4160d40 2023-10-16T09:59:26.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6bb806b020 msgr2=0x7f6bb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:26.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6bb806b020 0x7f6bb806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6bc80011e0 tx=0x7f6bc8009040 comp rx=0 tx=0).stop 2023-10-16T09:59:26.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bd4160d40 msgr2=0x7f6bd4152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:26.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bd4160d40 0x7f6bd4152da0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009d90 tx=0x7f6bcc004ad0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 shutdown_connections 2023-10-16T09:59:26.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f6bb806b020 0x7f6bb806d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bd4160d40 0x7f6bd4152da0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bd40c49b0 0x7f6bd40c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 --2- 172.21.15.158:0/3900522918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bd40c4e60 0x7f6bd40b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 >> 172.21.15.158:0/3900522918 conn(0x7f6bd40cc5b0 msgr2=0x7f6bd40cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:26.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 shutdown_connections 2023-10-16T09:59:26.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.139+0000 7f6bdc29c700 1 -- 172.21.15.158:0/3900522918 wait complete. 2023-10-16T09:59:26.155 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T09:59:26.155 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T09:59:26.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.283+0000 7fd6460cf700 1 Processor -- start 2023-10-16T09:59:26.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.283+0000 7fd6460cf700 1 -- start start 2023-10-16T09:59:26.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.284+0000 7fd6460cf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd64015a5b0 0x7fd640071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:26.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.284+0000 7fd6460cf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400cb900 0x7fd640071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:26.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.284+0000 7fd6460cf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6400ce150 con 0x7fd6400726a0 2023-10-16T09:59:26.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.284+0000 7fd6460cf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6400cdf70 con 0x7fd64015a5b0 2023-10-16T09:59:26.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.284+0000 7fd6460cf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400ce430 con 0x7fd6400cb900 2023-10-16T09:59:26.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd64015a5b0 0x7fd640071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:26.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd64015a5b0 0x7fd640071960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52236/0 (socket says 172.21.15.158:52236) 2023-10-16T09:59:26.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 -- 172.21.15.158:0/1925091757 learned_addr learned my addr 172.21.15.158:0/1925091757 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:26.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 -- 172.21.15.158:0/1925091757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400cb900 msgr2=0x7fd640071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:26.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63effd700 1 --2- 172.21.15.158:0/1925091757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400cb900 0x7fd640071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:26.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 --2- 172.21.15.158:0/1925091757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400cb900 0x7fd640071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 -- 172.21.15.158:0/1925091757 >> v1:172.21.15.172:6789/0 conn(0x7fd6400726a0 legacy=0x7fd64006ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 -- 172.21.15.158:0/1925091757 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400ce710 con 0x7fd64015a5b0 2023-10-16T09:59:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63effd700 1 --2- 172.21.15.158:0/1925091757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400cb900 0x7fd640071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63f7fe700 1 --2- 172.21.15.158:0/1925091757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd64015a5b0 0x7fd640071960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd6340098f0 tx=0x7fd634009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e5bb05651ce9a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63e7fc700 1 -- 172.21.15.158:0/1925091757 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634004230 con 0x7fd64015a5b0 2023-10-16T09:59:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63e7fc700 1 -- 172.21.15.158:0/1925091757 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd634004390 con 0x7fd64015a5b0 2023-10-16T09:59:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.285+0000 7fd63e7fc700 1 -- 172.21.15.158:0/1925091757 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6340036a0 con 0x7fd64015a5b0 2023-10-16T09:59:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 -- 172.21.15.158:0/1925091757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd64015a5b0 msgr2=0x7fd640071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 --2- 172.21.15.158:0/1925091757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd64015a5b0 0x7fd640071960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd6340098f0 tx=0x7fd634009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 -- 172.21.15.158:0/1925091757 shutdown_connections 2023-10-16T09:59:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 --2- 172.21.15.158:0/1925091757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd64015a5b0 0x7fd640071960 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 --2- 172.21.15.158:0/1925091757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400cb900 0x7fd640071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 -- 172.21.15.158:0/1925091757 >> 172.21.15.158:0/1925091757 conn(0x7fd6400b1e90 msgr2=0x7fd6400b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 -- 172.21.15.158:0/1925091757 shutdown_connections 2023-10-16T09:59:26.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.286+0000 7fd6460cf700 1 -- 172.21.15.158:0/1925091757 wait complete. 2023-10-16T09:59:26.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.287+0000 7fd6460cf700 1 Processor -- start 2023-10-16T09:59:26.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.288+0000 7fd6460cf700 1 -- start start 2023-10-16T09:59:26.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.288+0000 7fd6460cf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400726a0 0x7fd640152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:26.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.288+0000 7fd6460cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6400cb900 0x7fd640150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:26.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.288+0000 7fd6460cf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd64015a5b0 0x7fd6401f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:26.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.288+0000 7fd6460cf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd640151460 con 0x7fd6400cb900 2023-10-16T09:59:26.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.288+0000 7fd6460cf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd640151960 con 0x7fd64015a5b0 2023-10-16T09:59:26.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.288+0000 7fd6460cf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd640151c40 con 0x7fd6400726a0 2023-10-16T09:59:26.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6400cb900 0x7fd640150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:26.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6400cb900 0x7fd640150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52244/0 (socket says 172.21.15.158:52244) 2023-10-16T09:59:26.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 -- 172.21.15.158:0/3226679472 learned_addr learned my addr 172.21.15.158:0/3226679472 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:26.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63f7fe700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400726a0 0x7fd640152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:26.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63ffff700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd64015a5b0 0x7fd6401f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:26.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 -- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400726a0 msgr2=0x7fd640152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:26.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400726a0 0x7fd640152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 -- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd64015a5b0 msgr2=0x7fd6401f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:26.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd64015a5b0 0x7fd6401f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:26.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 -- 172.21.15.158:0/3226679472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634009550 con 0x7fd6400cb900 2023-10-16T09:59:26.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63f7fe700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400726a0 0x7fd640152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:26.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd63effd700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6400cb900 0x7fd640150a80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fd63000c750 tx=0x7fd63000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:26.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd627fff700 1 -- 172.21.15.158:0/3226679472 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630011800 con 0x7fd6400cb900 2023-10-16T09:59:26.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd627fff700 1 -- 172.21.15.158:0/3226679472 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630011960 con 0x7fd6400cb900 2023-10-16T09:59:26.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd640151f20 con 0x7fd6400cb900 2023-10-16T09:59:26.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd627fff700 1 -- 172.21.15.158:0/3226679472 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630011cd0 con 0x7fd6400cb900 2023-10-16T09:59:26.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.289+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64014e3d0 con 0x7fd6400cb900 2023-10-16T09:59:26.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.290+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd640152440 con 0x7fd6400cb900 2023-10-16T09:59:26.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.291+0000 7fd627fff700 1 -- 172.21.15.158:0/3226679472 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd630021350 con 0x7fd6400cb900 2023-10-16T09:59:26.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.291+0000 7fd627fff700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd62c06b020 0x7fd62c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:26.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.291+0000 7fd627fff700 1 -- 172.21.15.158:0/3226679472 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fd630089ad0 con 0x7fd6400cb900 2023-10-16T09:59:26.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.291+0000 7fd63f7fe700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd62c06b020 0x7fd62c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:26.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.291+0000 7fd63f7fe700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd62c06b020 0x7fd62c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd634009130 tx=0x7fd634008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:26.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.295+0000 7fd627fff700 1 -- 172.21.15.158:0/3226679472 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd640152440 con 0x7fd6400cb900 2023-10-16T09:59:26.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:26.531+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd640152440 con 0x7fd6400cb900 2023-10-16T09:59:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.141+0000 7fd627fff700 1 -- 172.21.15.158:0/3226679472 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd640152440 con 0x7fd6400cb900 2023-10-16T09:59:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd62c06b020 msgr2=0x7fd62c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd62c06b020 0x7fd62c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd634009130 tx=0x7fd634008030 comp rx=0 tx=0).stop 2023-10-16T09:59:27.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6400cb900 msgr2=0x7fd640150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6400cb900 0x7fd640150a80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fd63000c750 tx=0x7fd63000cb10 comp rx=0 tx=0).stop 2023-10-16T09:59:27.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 shutdown_connections 2023-10-16T09:59:27.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fd62c06b020 0x7fd62c06d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd64015a5b0 0x7fd6401f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6400726a0 0x7fd640152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 --2- 172.21.15.158:0/3226679472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6400cb900 0x7fd640150a80 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 >> 172.21.15.158:0/3226679472 conn(0x7fd6400b1e90 msgr2=0x7fd6400b5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:27.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 shutdown_connections 2023-10-16T09:59:27.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.144+0000 7fd6460cf700 1 -- 172.21.15.158:0/3226679472 wait complete. 2023-10-16T09:59:27.159 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T09:59:27.160 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:27.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.280+0000 7fa8c06a8700 1 Processor -- start 2023-10-16T09:59:27.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.280+0000 7fa8c06a8700 1 -- start start 2023-10-16T09:59:27.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.281+0000 7fa8c06a8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8b815db30 0x7fa8b806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.281+0000 7fa8c06a8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa8b80cbfd0 0x7fa8b806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.281+0000 7fa8c06a8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b8161c80 con 0x7fa8b8162110 2023-10-16T09:59:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.281+0000 7fa8c06a8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b806e090 con 0x7fa8b815db30 2023-10-16T09:59:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.281+0000 7fa8c06a8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b806ddb0 con 0x7fa8b80cbfd0 2023-10-16T09:59:27.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8be444700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8b815db30 0x7fa8b806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bdc43700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa8b8162110 0x7fa8b80b1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52426/0 (socket says 172.21.15.158:52426) 2023-10-16T09:59:27.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bdc43700 1 -- 172.21.15.158:0/197706295 learned_addr learned my addr 172.21.15.158:0/197706295 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:27.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bec45700 1 --2- 172.21.15.158:0/197706295 >> v2:172.21.15.172:3300/0 conn(0x7fa8b80cbfd0 0x7fa8b806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8be444700 1 -- 172.21.15.158:0/197706295 >> v2:172.21.15.172:3300/0 conn(0x7fa8b80cbfd0 msgr2=0x7fa8b806ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:27.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8be444700 1 --2- 172.21.15.158:0/197706295 >> v2:172.21.15.172:3300/0 conn(0x7fa8b80cbfd0 0x7fa8b806ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8be444700 1 -- 172.21.15.158:0/197706295 >> v1:172.21.15.158:6789/0 conn(0x7fa8b8162110 legacy=0x7fa8b80b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8be444700 1 -- 172.21.15.158:0/197706295 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8161630 con 0x7fa8b815db30 2023-10-16T09:59:27.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bd442700 1 -- 172.21.15.158:0/197706295 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99427037 0 0) 0x7fa8b8161c80 con 0x7fa8b8162110 2023-10-16T09:59:27.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bec45700 1 --2- 172.21.15.158:0/197706295 >> v2:172.21.15.172:3300/0 conn(0x7fa8b80cbfd0 0x7fa8b806ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:27.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8be444700 1 --2- 172.21.15.158:0/197706295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b815db30 0x7fa8b806a8d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80b64a0 tx=0x7fa8a8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b4448ca53373de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:27.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bd442700 1 -- 172.21.15.158:0/197706295 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8011830 con 0x7fa8b815db30 2023-10-16T09:59:27.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bd442700 1 -- 172.21.15.158:0/197706295 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8011990 con 0x7fa8b815db30 2023-10-16T09:59:27.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.282+0000 7fa8bd442700 1 -- 172.21.15.158:0/197706295 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8011d20 con 0x7fa8b815db30 2023-10-16T09:59:27.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 -- 172.21.15.158:0/197706295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b815db30 msgr2=0x7fa8b806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/197706295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b815db30 0x7fa8b806a8d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80b64a0 tx=0x7fa8a8009b20 comp rx=0 tx=0).stop 2023-10-16T09:59:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 -- 172.21.15.158:0/197706295 shutdown_connections 2023-10-16T09:59:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/197706295 >> v2:172.21.15.172:3300/0 conn(0x7fa8b80cbfd0 0x7fa8b806ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/197706295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b815db30 0x7fa8b806a8d0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 -- 172.21.15.158:0/197706295 >> 172.21.15.158:0/197706295 conn(0x7fa8b80ccdb0 msgr2=0x7fa8b80cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:27.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 -- 172.21.15.158:0/197706295 shutdown_connections 2023-10-16T09:59:27.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.283+0000 7fa8c06a8700 1 -- 172.21.15.158:0/197706295 wait complete. 2023-10-16T09:59:27.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.284+0000 7fa8c06a8700 1 Processor -- start 2023-10-16T09:59:27.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.284+0000 7fa8c06a8700 1 -- start start 2023-10-16T09:59:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8c06a8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8b80cbfd0 0x7fa8b814e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8c06a8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8b815db30 0x7fa8b80b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8c06a8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b8162110 0x7fa8b80b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8c06a8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80b2eb0 con 0x7fa8b8162110 2023-10-16T09:59:27.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8c06a8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80c5290 con 0x7fa8b815db30 2023-10-16T09:59:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8c06a8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80b36c0 con 0x7fa8b80cbfd0 2023-10-16T09:59:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b8162110 0x7fa8b80b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b8162110 0x7fa8b80b53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52256/0 (socket says 172.21.15.158:52256) 2023-10-16T09:59:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 -- 172.21.15.158:0/2576613923 learned_addr learned my addr 172.21.15.158:0/2576613923 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:27.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bdc43700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8b815db30 0x7fa8b80b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8be444700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8b80cbfd0 0x7fa8b814e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 -- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8b80cbfd0 msgr2=0x7fa8b814e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8b80cbfd0 0x7fa8b814e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 -- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8b815db30 msgr2=0x7fa8b80b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8b815db30 0x7fa8b80b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 -- 172.21.15.158:0/2576613923 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a80094a0 con 0x7fa8b8162110 2023-10-16T09:59:27.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8bec45700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b8162110 0x7fa8b80b53b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0007ef0 tx=0x7fa8b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:27.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8aeffd700 1 -- 172.21.15.158:0/2576613923 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000de00 con 0x7fa8b8162110 2023-10-16T09:59:27.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8aeffd700 1 -- 172.21.15.158:0/2576613923 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b000f040 con 0x7fa8b8162110 2023-10-16T09:59:27.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b814ed60 con 0x7fa8b8162110 2023-10-16T09:59:27.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.285+0000 7fa8aeffd700 1 -- 172.21.15.158:0/2576613923 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0003c60 con 0x7fa8b8162110 2023-10-16T09:59:27.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.286+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b80b3100 con 0x7fa8b8162110 2023-10-16T09:59:27.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.287+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b80681b0 con 0x7fa8b8162110 2023-10-16T09:59:27.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.287+0000 7fa8aeffd700 1 -- 172.21.15.158:0/2576613923 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa8b0004030 con 0x7fa8b8162110 2023-10-16T09:59:27.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.287+0000 7fa8aeffd700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fa89c06b020 0x7fa89c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.287+0000 7fa8aeffd700 1 -- 172.21.15.158:0/2576613923 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fa8b0089c90 con 0x7fa8b8162110 2023-10-16T09:59:27.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.287+0000 7fa8be444700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fa89c06b020 0x7fa89c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.288+0000 7fa8be444700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fa89c06b020 0x7fa89c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8000c00 tx=0x7fa8a8008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:27.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.292+0000 7fa8aeffd700 1 -- 172.21.15.158:0/2576613923 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8b00548c0 con 0x7fa8b8162110 2023-10-16T09:59:27.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.526+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8b80c53d0 con 0x7fa8b8162110 2023-10-16T09:59:27.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.527+0000 7fa8aeffd700 1 -- 172.21.15.158:0/2576613923 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4926 (secure 0 0 0) 0x7fa8b001a090 con 0x7fa8b8162110 2023-10-16T09:59:27.529 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:27.529 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3413925509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3413925509},{"type":"v1","addr":"172.21.15.158:6835","nonce":3413925509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/655082434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":655082434},{"type":"v1","addr":"172.21.15.158:6837","nonce":655082434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2666705933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2666705933},{"type":"v1","addr":"172.21.15.172:6835","nonce":2666705933}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:23.515854+0000","modified":"2023-10-16T09:59:26.532765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35639":{"gid":35639,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3389815719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3389815719},{"type":"v1","addr":"172.21.15.172:6834","nonce":3389815719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T09:59:27.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fa89c06b020 msgr2=0x7fa89c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fa89c06b020 0x7fa89c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8000c00 tx=0x7fa8a8008030 comp rx=0 tx=0).stop 2023-10-16T09:59:27.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b8162110 msgr2=0x7fa8b80b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b8162110 0x7fa8b80b53b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0007ef0 tx=0x7fa8b000d3b0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 shutdown_connections 2023-10-16T09:59:27.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fa89c06b020 0x7fa89c06d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8b8162110 0x7fa8b80b53b0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8b80cbfd0 0x7fa8b814e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 --2- 172.21.15.158:0/2576613923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8b815db30 0x7fa8b80b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 >> 172.21.15.158:0/2576613923 conn(0x7fa8b80ccdb0 msgr2=0x7fa8b80cb850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:27.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 shutdown_connections 2023-10-16T09:59:27.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.529+0000 7fa8c06a8700 1 -- 172.21.15.158:0/2576613923 wait complete. 2023-10-16T09:59:27.533 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 33 2023-10-16T09:59:27.542 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:27.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.662+0000 7fedf8e05700 1 Processor -- start 2023-10-16T09:59:27.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.662+0000 7fedf8e05700 1 -- start start 2023-10-16T09:59:27.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf8e05700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cc0c0 0x7fedf40aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf8e05700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fedf40cf4c0 0x7fedf40cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf8e05700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf415b810 con 0x7fedf406bfb0 2023-10-16T09:59:27.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf8e05700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf415a990 con 0x7fedf40cf4c0 2023-10-16T09:59:27.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf8e05700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf415ac00 con 0x7fedf40cc0c0 2023-10-16T09:59:27.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf1d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fedf406bfb0 0x7fedf40cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52436/0 (socket says 172.21.15.158:52436) 2023-10-16T09:59:27.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf1d9b700 1 -- 172.21.15.158:0/1958087670 learned_addr learned my addr 172.21.15.158:0/1958087670 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:27.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf259c700 1 --2- 172.21.15.158:0/1958087670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cc0c0 0x7fedf40aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf2d9d700 1 --2- 172.21.15.158:0/1958087670 >> v2:172.21.15.172:3300/0 conn(0x7fedf40cf4c0 0x7fedf40cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf159a700 1 -- 172.21.15.158:0/1958087670 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709237263 0 0) 0x7fedf415b810 con 0x7fedf406bfb0 2023-10-16T09:59:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf159a700 1 -- 172.21.15.158:0/1958087670 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feddc0035b0 con 0x7fedf406bfb0 2023-10-16T09:59:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf159a700 1 -- 172.21.15.158:0/1958087670 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3209611025 0 0) 0x7feddc0035b0 con 0x7fedf406bfb0 2023-10-16T09:59:27.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf259c700 1 -- 172.21.15.158:0/1958087670 >> v2:172.21.15.172:3300/0 conn(0x7fedf40cf4c0 msgr2=0x7fedf40cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf259c700 1 --2- 172.21.15.158:0/1958087670 >> v2:172.21.15.172:3300/0 conn(0x7fedf40cf4c0 0x7fedf40cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf259c700 1 -- 172.21.15.158:0/1958087670 >> v1:172.21.15.158:6789/0 conn(0x7fedf406bfb0 legacy=0x7fedf40cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf259c700 1 -- 172.21.15.158:0/1958087670 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf415aee0 con 0x7fedf40cc0c0 2023-10-16T09:59:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.663+0000 7fedf2d9d700 1 --2- 172.21.15.158:0/1958087670 >> v2:172.21.15.172:3300/0 conn(0x7fedf40cf4c0 0x7fedf40cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:27.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf259c700 1 --2- 172.21.15.158:0/1958087670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cc0c0 0x7fedf40aeda0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fede4009840 tx=0x7fede4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe9f9c36c7172437 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf159a700 1 -- 172.21.15.158:0/1958087670 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8001670 con 0x7fedf40cc0c0 2023-10-16T09:59:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf159a700 1 -- 172.21.15.158:0/1958087670 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede4004030 con 0x7fedf40cc0c0 2023-10-16T09:59:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf159a700 1 -- 172.21.15.158:0/1958087670 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede40043a0 con 0x7fedf40cc0c0 2023-10-16T09:59:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 -- 172.21.15.158:0/1958087670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cc0c0 msgr2=0x7fedf40aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1958087670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cc0c0 0x7fedf40aeda0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fede4009840 tx=0x7fede4009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 -- 172.21.15.158:0/1958087670 shutdown_connections 2023-10-16T09:59:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1958087670 >> v2:172.21.15.172:3300/0 conn(0x7fedf40cf4c0 0x7fedf40cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1958087670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cc0c0 0x7fedf40aeda0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 -- 172.21.15.158:0/1958087670 >> 172.21.15.158:0/1958087670 conn(0x7fedf40cc4b0 msgr2=0x7fedf40cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:27.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 -- 172.21.15.158:0/1958087670 shutdown_connections 2023-10-16T09:59:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.664+0000 7fedf8e05700 1 -- 172.21.15.158:0/1958087670 wait complete. 2023-10-16T09:59:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 Processor -- start 2023-10-16T09:59:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 -- start start 2023-10-16T09:59:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fedf406bfb0 0x7fedf414b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cf4c0 0x7fedf40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fedf40b4ba0 0x7fedf40c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b5790 con 0x7fedf406bfb0 2023-10-16T09:59:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf414f600 con 0x7fedf40b4ba0 2023-10-16T09:59:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf8e05700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b51d0 con 0x7fedf40cf4c0 2023-10-16T09:59:27.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fedf406bfb0 0x7fedf414b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.666+0000 7fedf259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fedf406bfb0 0x7fedf414b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52266/0 (socket says 172.21.15.158:52266) 2023-10-16T09:59:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf259c700 1 -- 172.21.15.158:0/1094477783 learned_addr learned my addr 172.21.15.158:0/1094477783 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf1d9b700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cf4c0 0x7fedf40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf2d9d700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fedf40b4ba0 0x7fedf40c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf259c700 1 -- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cf4c0 msgr2=0x7fedf40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf259c700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cf4c0 0x7fedf40b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf259c700 1 -- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fedf40b4ba0 msgr2=0x7fedf40c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf259c700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fedf40b4ba0 0x7fedf40c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf259c700 1 -- 172.21.15.158:0/1094477783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec009a70 con 0x7fedf406bfb0 2023-10-16T09:59:27.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf1d9b700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cf4c0 0x7fedf40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:27.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf2d9d700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fedf40b4ba0 0x7fedf40c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:27.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf259c700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fedf406bfb0 0x7fedf414b680 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fede4009b90 tx=0x7fede4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:27.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fede2ffd700 1 -- 172.21.15.158:0/1094477783 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4004590 con 0x7fedf406bfb0 2023-10-16T09:59:27.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fede2ffd700 1 -- 172.21.15.158:0/1094477783 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede401f070 con 0x7fedf406bfb0 2023-10-16T09:59:27.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fede2ffd700 1 -- 172.21.15.158:0/1094477783 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4011870 con 0x7fedf406bfb0 2023-10-16T09:59:27.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede40094a0 con 0x7fedf406bfb0 2023-10-16T09:59:27.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.667+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf40b2a10 con 0x7fedf406bfb0 2023-10-16T09:59:27.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.669+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf40b2870 con 0x7fedf406bfb0 2023-10-16T09:59:27.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.669+0000 7fede2ffd700 1 -- 172.21.15.158:0/1094477783 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fede40119d0 con 0x7fedf406bfb0 2023-10-16T09:59:27.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.669+0000 7fede2ffd700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fedd406b020 0x7fedd406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:27.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.669+0000 7fede2ffd700 1 -- 172.21.15.158:0/1094477783 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fede4021070 con 0x7fedf406bfb0 2023-10-16T09:59:27.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.669+0000 7fedf1d9b700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fedd406b020 0x7fedd406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:27.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.670+0000 7fedf1d9b700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fedd406b020 0x7fedd406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fede80011e0 tx=0x7fede8007ae0 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:27.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.674+0000 7fede2ffd700 1 -- 172.21.15.158:0/1094477783 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fedf40b2870 con 0x7fedf406bfb0 2023-10-16T09:59:27.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.865+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedf40b2870 con 0x7fedf406bfb0 2023-10-16T09:59:27.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.866+0000 7fede2ffd700 1 -- 172.21.15.158:0/1094477783 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+17373 (secure 0 0 0) 0x7fedf40b2870 con 0x7fedf406bfb0 2023-10-16T09:59:27.868 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:27.868 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:24.555713+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:21.428579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:22.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:27.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fedd406b020 msgr2=0x7fedd406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fedd406b020 0x7fedd406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fede80011e0 tx=0x7fede8007ae0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fedf406bfb0 msgr2=0x7fedf414b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:27.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fedf406bfb0 0x7fedf414b680 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fede4009b90 tx=0x7fede4004750 comp rx=0 tx=0).stop 2023-10-16T09:59:27.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 shutdown_connections 2023-10-16T09:59:27.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fedd406b020 0x7fedd406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fedf40b4ba0 0x7fedf40c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fedf406bfb0 0x7fedf414b680 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 --2- 172.21.15.158:0/1094477783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fedf40cf4c0 0x7fedf40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:27.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 >> 172.21.15.158:0/1094477783 conn(0x7fedf40cc4b0 msgr2=0x7fedf40cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:27.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 shutdown_connections 2023-10-16T09:59:27.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:27.868+0000 7fedf8e05700 1 -- 172.21.15.158:0/1094477783 wait complete. 2023-10-16T09:59:27.882 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T09:59:27.883 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:59:28.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.003+0000 7fefb1c81700 1 Processor -- start 2023-10-16T09:59:28.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.003+0000 7fefb1c81700 1 -- start start 2023-10-16T09:59:28.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.003+0000 7fefb1c81700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fefac06e5d0 0x7fefac06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.003+0000 7fefb1c81700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac161270 con 0x7fefac06ab40 2023-10-16T09:59:28.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.003+0000 7fefb1c81700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac161510 con 0x7fefac0729b0 2023-10-16T09:59:28.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.003+0000 7fefb1c81700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefac1617f0 con 0x7fefac06e5d0 2023-10-16T09:59:28.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefab7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fefac06ab40 0x7fefac0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52450/0 (socket says 172.21.15.158:52450) 2023-10-16T09:59:28.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefab7fe700 1 -- 172.21.15.158:0/4093622094 learned_addr learned my addr 172.21.15.158:0/4093622094 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:28.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200604685 0 0) 0x7fefac161270 con 0x7fefac06ab40 2023-10-16T09:59:28.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef940035b0 con 0x7fefac06ab40 2023-10-16T09:59:28.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaaffd700 1 --2- 172.21.15.158:0/4093622094 >> v2:172.21.15.172:3300/0 conn(0x7fefac06e5d0 0x7fefac06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3501564326 0 0) 0x7fef940035b0 con 0x7fefac06ab40 2023-10-16T09:59:28.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 >> v2:172.21.15.172:3300/0 conn(0x7fefac06e5d0 msgr2=0x7fefac06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:28.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 --2- 172.21.15.158:0/4093622094 >> v2:172.21.15.172:3300/0 conn(0x7fefac06e5d0 0x7fefac06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 >> v1:172.21.15.172:6789/0 conn(0x7fefac0729b0 legacy=0x7fefac15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac161ad0 con 0x7fefac06ab40 2023-10-16T09:59:28.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fef9c001670 con 0x7fefac06ab40 2023-10-16T09:59:28.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 >> v1:172.21.15.158:6789/0 conn(0x7fefac06ab40 legacy=0x7fefac0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94003a00 0x7fef94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef94004850 0x7fef94004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef94005210 0x7fef9400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac1617f0 con 0x7fef94005210 2023-10-16T09:59:28.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9400c160 con 0x7fef94004850 2023-10-16T09:59:28.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9400c320 con 0x7fef94003a00 2023-10-16T09:59:28.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefabfff700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef94005210 0x7fef9400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefab7fe700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94003a00 0x7fef94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.004+0000 7fefaaffd700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef94004850 0x7fef94004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefabfff700 1 -- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94003a00 msgr2=0x7fef94003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:28.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefabfff700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94003a00 0x7fef94003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefabfff700 1 -- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef94004850 msgr2=0x7fef94004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:28.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefabfff700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef94004850 0x7fef94004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefabfff700 1 -- 172.21.15.158:0/4093622094 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef940035b0 con 0x7fef94005210 2023-10-16T09:59:28.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefabfff700 1 -- 172.21.15.158:0/4093622094 reap_dead start 2023-10-16T09:59:28.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefabfff700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef94005210 0x7fef9400ba10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c210 tx=0x7fef94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d290466231bbe6cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4001df0 con 0x7fef94005210 2023-10-16T09:59:28.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa400cd70 con 0x7fef94005210 2023-10-16T09:59:28.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefaa7fc700 1 -- 172.21.15.158:0/4093622094 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4003c40 con 0x7fef94005210 2023-10-16T09:59:28.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefb1c81700 1 -- 172.21.15.158:0/4093622094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef94005210 msgr2=0x7fef9400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefb1c81700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef94005210 0x7fef9400ba10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c210 tx=0x7fef94005880 comp rx=0 tx=0).stop 2023-10-16T09:59:28.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefb1c81700 1 -- 172.21.15.158:0/4093622094 shutdown_connections 2023-10-16T09:59:28.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefb1c81700 1 --2- 172.21.15.158:0/4093622094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef94005210 0x7fef9400ba10 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefb1c81700 1 -- 172.21.15.158:0/4093622094 >> 172.21.15.158:0/4093622094 conn(0x7fefac0d02b0 msgr2=0x7fefac0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:28.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.005+0000 7fefb1c81700 1 -- 172.21.15.158:0/4093622094 shutdown_connections 2023-10-16T09:59:28.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.006+0000 7fefb1c81700 1 -- 172.21.15.158:0/4093622094 wait complete. 2023-10-16T09:59:28.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.007+0000 7fefb1c81700 1 Processor -- start 2023-10-16T09:59:28.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.007+0000 7fefb1c81700 1 -- start start 2023-10-16T09:59:28.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94005210 0x7fefac14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fefac0c9600 0x7fefac0b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefac0b3f40 0x7fefac0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac0b43a0 con 0x7fefac0b3f40 2023-10-16T09:59:28.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac0c5a40 con 0x7fefac0c9600 2023-10-16T09:59:28.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefac0c5d20 con 0x7fef94005210 2023-10-16T09:59:28.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefac0b3f40 0x7fefac0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefac0b3f40 0x7fefac0c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52290/0 (socket says 172.21.15.158:52290) 2023-10-16T09:59:28.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 -- 172.21.15.158:0/1367071897 learned_addr learned my addr 172.21.15.158:0/1367071897 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:28.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefab7fe700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94005210 0x7fefac14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefaaffd700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fefac0c9600 0x7fefac0b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 -- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94005210 msgr2=0x7fefac14f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94005210 0x7fefac14f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 -- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fefac0c9600 msgr2=0x7fefac0b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fefac0c9600 0x7fefac0b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 -- 172.21.15.158:0/1367071897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac0c6000 con 0x7fefac0b3f40 2023-10-16T09:59:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefabfff700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefac0b3f40 0x7fefac0c3c80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fefac06eaf0 tx=0x7fefa40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fef93fff700 1 -- 172.21.15.158:0/1367071897 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4003e90 con 0x7fefac0b3f40 2023-10-16T09:59:28.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fef93fff700 1 -- 172.21.15.158:0/1367071897 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa4005070 con 0x7fefac0b3f40 2023-10-16T09:59:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac0c62e0 con 0x7fefac0b3f40 2023-10-16T09:59:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fef93fff700 1 -- 172.21.15.158:0/1367071897 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4005440 con 0x7fefac0b3f40 2023-10-16T09:59:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.008+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac0c9a80 con 0x7fefac0b3f40 2023-10-16T09:59:28.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.009+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac0c6800 con 0x7fefac0b3f40 2023-10-16T09:59:28.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.010+0000 7fef93fff700 1 -- 172.21.15.158:0/1367071897 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fefa40055a0 con 0x7fefac0b3f40 2023-10-16T09:59:28.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.010+0000 7fef93fff700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fef9806b020 0x7fef9806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.010+0000 7fef93fff700 1 -- 172.21.15.158:0/1367071897 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fefa408a730 con 0x7fefac0b3f40 2023-10-16T09:59:28.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.010+0000 7fefab7fe700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fef9806b020 0x7fef9806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.011+0000 7fefab7fe700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fef9806b020 0x7fef9806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef9c000c00 tx=0x7fef9c001c00 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.015+0000 7fef93fff700 1 -- 172.21.15.158:0/1367071897 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fefac0c6800 con 0x7fefac0b3f40 2023-10-16T09:59:28.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.204+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 --> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fefac0c5e90 con 0x7fef9806b020 2023-10-16T09:59:28.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.212+0000 7fef93fff700 1 -- 172.21.15.158:0/1367071897 <== mgr.35456 v2:172.21.15.158:6832/3717499045 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289552 (secure 0 0 0) 0x7fefac0c5e90 con 0x7fef9806b020 2023-10-16T09:59:28.217 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:28.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fef9806b020 msgr2=0x7fef9806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fef9806b020 0x7fef9806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef9c000c00 tx=0x7fef9c001c00 comp rx=0 tx=0).stop 2023-10-16T09:59:28.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefac0b3f40 msgr2=0x7fefac0c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefac0b3f40 0x7fefac0c3c80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fefac06eaf0 tx=0x7fefa40014a0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 shutdown_connections 2023-10-16T09:59:28.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fef9806b020 0x7fef9806d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefac0b3f40 0x7fefac0c3c80 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef94005210 0x7fefac14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 --2- 172.21.15.158:0/1367071897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fefac0c9600 0x7fefac0b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.215+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 >> 172.21.15.158:0/1367071897 conn(0x7fefac0d02b0 msgr2=0x7fefac0ca5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:28.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.216+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 shutdown_connections 2023-10-16T09:59:28.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.216+0000 7fefb1c81700 1 -- 172.21.15.158:0/1367071897 wait complete. 2023-10-16T09:59:28.223 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:59:28.229 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T09:59:27.075144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":325300,"kb_used_data":9216,"kb_used_omap":12,"kb_used_meta":315955,"kb_avail":754649420,"statfs":{"total":773094113280,"available":772761006080,"internally_reserved":0,"allocated":9437184,"data_stored":5376364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":323538487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.958561"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.558947+0000","last_change":"2023-10-16T09:59:23.574345+0000","last_active":"2023-10-16T09:59:24.558947+0000","last_peered":"2023-10-16T09:59:24.558947+0000","last_clean":"2023-10-16T09:59:24.558947+0000","last_became_active":"2023-10-16T09:59:23.574076+0000","last_became_peered":"2023-10-16T09:59:23.574076+0000","last_unstale":"2023-10-16T09:59:24.558947+0000","last_undegraded":"2023-10-16T09:59:24.558947+0000","last_fullsized":"2023-10-16T09:59:24.558947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:42:48.948770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559187+0000","last_change":"2023-10-16T09:59:23.565351+0000","last_active":"2023-10-16T09:59:24.559187+0000","last_peered":"2023-10-16T09:59:24.559187+0000","last_clean":"2023-10-16T09:59:24.559187+0000","last_became_active":"2023-10-16T09:59:23.565106+0000","last_became_peered":"2023-10-16T09:59:23.565106+0000","last_unstale":"2023-10-16T09:59:24.559187+0000","last_undegraded":"2023-10-16T09:59:24.559187+0000","last_fullsized":"2023-10-16T09:59:24.559187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:58.071562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729622+0000","last_change":"2023-10-16T09:59:23.567454+0000","last_active":"2023-10-16T09:59:24.729622+0000","last_peered":"2023-10-16T09:59:24.729622+0000","last_clean":"2023-10-16T09:59:24.729622+0000","last_became_active":"2023-10-16T09:59:23.567172+0000","last_became_peered":"2023-10-16T09:59:23.567172+0000","last_unstale":"2023-10-16T09:59:24.729622+0000","last_undegraded":"2023-10-16T09:59:24.729622+0000","last_fullsized":"2023-10-16T09:59:24.729622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:27.479652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560413+0000","last_change":"2023-10-16T09:59:23.575949+0000","last_active":"2023-10-16T09:59:24.560413+0000","last_peered":"2023-10-16T09:59:24.560413+0000","last_clean":"2023-10-16T09:59:24.560413+0000","last_became_active":"2023-10-16T09:59:23.575689+0000","last_became_peered":"2023-10-16T09:59:23.575689+0000","last_unstale":"2023-10-16T09:59:24.560413+0000","last_undegraded":"2023-10-16T09:59:24.560413+0000","last_fullsized":"2023-10-16T09:59:24.560413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:21.538488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589760+0000","last_change":"2023-10-16T09:59:23.586027+0000","last_active":"2023-10-16T09:59:23.589760+0000","last_peered":"2023-10-16T09:59:23.589760+0000","last_clean":"2023-10-16T09:59:23.589760+0000","last_became_active":"2023-10-16T09:59:23.585027+0000","last_became_peered":"2023-10-16T09:59:23.585027+0000","last_unstale":"2023-10-16T09:59:23.589760+0000","last_undegraded":"2023-10-16T09:59:23.589760+0000","last_fullsized":"2023-10-16T09:59:23.589760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:35.582399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.570466+0000","last_change":"2023-10-16T09:59:23.567329+0000","last_active":"2023-10-16T09:59:23.570466+0000","last_peered":"2023-10-16T09:59:23.570466+0000","last_clean":"2023-10-16T09:59:23.570466+0000","last_became_active":"2023-10-16T09:59:23.567070+0000","last_became_peered":"2023-10-16T09:59:23.567070+0000","last_unstale":"2023-10-16T09:59:23.570466+0000","last_undegraded":"2023-10-16T09:59:23.570466+0000","last_fullsized":"2023-10-16T09:59:23.570466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:04.914215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.568547+0000","last_change":"2023-10-16T09:59:23.568023+0000","last_active":"2023-10-16T09:59:23.568547+0000","last_peered":"2023-10-16T09:59:23.568547+0000","last_clean":"2023-10-16T09:59:23.568547+0000","last_became_active":"2023-10-16T09:59:23.567758+0000","last_became_peered":"2023-10-16T09:59:23.567758+0000","last_unstale":"2023-10-16T09:59:23.568547+0000","last_undegraded":"2023-10-16T09:59:23.568547+0000","last_fullsized":"2023-10-16T09:59:23.568547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:35:55.633336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.749944+0000","last_change":"2023-10-16T09:59:23.577776+0000","last_active":"2023-10-16T09:59:24.749944+0000","last_peered":"2023-10-16T09:59:24.749944+0000","last_clean":"2023-10-16T09:59:24.749944+0000","last_became_active":"2023-10-16T09:59:23.576971+0000","last_became_peered":"2023-10-16T09:59:23.576971+0000","last_unstale":"2023-10-16T09:59:24.749944+0000","last_undegraded":"2023-10-16T09:59:24.749944+0000","last_fullsized":"2023-10-16T09:59:24.749944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:42.588750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729779+0000","last_change":"2023-10-16T09:59:23.572964+0000","last_active":"2023-10-16T09:59:24.729779+0000","last_peered":"2023-10-16T09:59:24.729779+0000","last_clean":"2023-10-16T09:59:24.729779+0000","last_became_active":"2023-10-16T09:59:23.572676+0000","last_became_peered":"2023-10-16T09:59:23.572676+0000","last_unstale":"2023-10-16T09:59:24.729779+0000","last_undegraded":"2023-10-16T09:59:24.729779+0000","last_fullsized":"2023-10-16T09:59:24.729779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:13:09.310141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729631+0000","last_change":"2023-10-16T09:59:23.572333+0000","last_active":"2023-10-16T09:59:24.729631+0000","last_peered":"2023-10-16T09:59:24.729631+0000","last_clean":"2023-10-16T09:59:24.729631+0000","last_became_active":"2023-10-16T09:59:23.571837+0000","last_became_peered":"2023-10-16T09:59:23.571837+0000","last_unstale":"2023-10-16T09:59:24.729631+0000","last_undegraded":"2023-10-16T09:59:24.729631+0000","last_fullsized":"2023-10-16T09:59:24.729631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:24.280819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560499+0000","last_change":"2023-10-16T09:59:23.573333+0000","last_active":"2023-10-16T09:59:24.560499+0000","last_peered":"2023-10-16T09:59:24.560499+0000","last_clean":"2023-10-16T09:59:24.560499+0000","last_became_active":"2023-10-16T09:59:23.573111+0000","last_became_peered":"2023-10-16T09:59:23.573111+0000","last_unstale":"2023-10-16T09:59:24.560499+0000","last_undegraded":"2023-10-16T09:59:24.560499+0000","last_fullsized":"2023-10-16T09:59:24.560499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:47.211413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750217+0000","last_change":"2023-10-16T09:59:23.577980+0000","last_active":"2023-10-16T09:59:24.750217+0000","last_peered":"2023-10-16T09:59:24.750217+0000","last_clean":"2023-10-16T09:59:24.750217+0000","last_became_active":"2023-10-16T09:59:23.577710+0000","last_became_peered":"2023-10-16T09:59:23.577710+0000","last_unstale":"2023-10-16T09:59:24.750217+0000","last_undegraded":"2023-10-16T09:59:24.750217+0000","last_fullsized":"2023-10-16T09:59:24.750217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:12.930818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750107+0000","last_change":"2023-10-16T09:59:23.577698+0000","last_active":"2023-10-16T09:59:24.750107+0000","last_peered":"2023-10-16T09:59:24.750107+0000","last_clean":"2023-10-16T09:59:24.750107+0000","last_became_active":"2023-10-16T09:59:23.577403+0000","last_became_peered":"2023-10-16T09:59:23.577403+0000","last_unstale":"2023-10-16T09:59:24.750107+0000","last_undegraded":"2023-10-16T09:59:24.750107+0000","last_fullsized":"2023-10-16T09:59:24.750107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:51.453447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729675+0000","last_change":"2023-10-16T09:59:23.567578+0000","last_active":"2023-10-16T09:59:24.729675+0000","last_peered":"2023-10-16T09:59:24.729675+0000","last_clean":"2023-10-16T09:59:24.729675+0000","last_became_active":"2023-10-16T09:59:23.566964+0000","last_became_peered":"2023-10-16T09:59:23.566964+0000","last_unstale":"2023-10-16T09:59:24.729675+0000","last_undegraded":"2023-10-16T09:59:24.729675+0000","last_fullsized":"2023-10-16T09:59:24.729675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:24.950903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729710+0000","last_change":"2023-10-16T09:59:23.576194+0000","last_active":"2023-10-16T09:59:24.729710+0000","last_peered":"2023-10-16T09:59:24.729710+0000","last_clean":"2023-10-16T09:59:24.729710+0000","last_became_active":"2023-10-16T09:59:23.575601+0000","last_became_peered":"2023-10-16T09:59:23.575601+0000","last_unstale":"2023-10-16T09:59:24.729710+0000","last_undegraded":"2023-10-16T09:59:24.729710+0000","last_fullsized":"2023-10-16T09:59:24.729710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:26.719461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.575677+0000","last_change":"2023-10-16T09:59:23.574567+0000","last_active":"2023-10-16T09:59:23.575677+0000","last_peered":"2023-10-16T09:59:23.575677+0000","last_clean":"2023-10-16T09:59:23.575677+0000","last_became_active":"2023-10-16T09:59:23.574250+0000","last_became_peered":"2023-10-16T09:59:23.574250+0000","last_unstale":"2023-10-16T09:59:23.575677+0000","last_undegraded":"2023-10-16T09:59:23.575677+0000","last_fullsized":"2023-10-16T09:59:23.575677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:05:23.349438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559115+0000","last_change":"2023-10-16T09:59:23.572899+0000","last_active":"2023-10-16T09:59:24.559115+0000","last_peered":"2023-10-16T09:59:24.559115+0000","last_clean":"2023-10-16T09:59:24.559115+0000","last_became_active":"2023-10-16T09:59:23.572591+0000","last_became_peered":"2023-10-16T09:59:23.572591+0000","last_unstale":"2023-10-16T09:59:24.559115+0000","last_undegraded":"2023-10-16T09:59:24.559115+0000","last_fullsized":"2023-10-16T09:59:24.559115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:29.547767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560720+0000","last_change":"2023-10-16T09:59:23.571659+0000","last_active":"2023-10-16T09:59:24.560720+0000","last_peered":"2023-10-16T09:59:24.560720+0000","last_clean":"2023-10-16T09:59:24.560720+0000","last_became_active":"2023-10-16T09:59:23.571327+0000","last_became_peered":"2023-10-16T09:59:23.571327+0000","last_unstale":"2023-10-16T09:59:24.560720+0000","last_undegraded":"2023-10-16T09:59:24.560720+0000","last_fullsized":"2023-10-16T09:59:24.560720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:23.175763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589564+0000","last_change":"2023-10-16T09:59:23.584152+0000","last_active":"2023-10-16T09:59:23.589564+0000","last_peered":"2023-10-16T09:59:23.589564+0000","last_clean":"2023-10-16T09:59:23.589564+0000","last_became_active":"2023-10-16T09:59:23.583828+0000","last_became_peered":"2023-10-16T09:59:23.583828+0000","last_unstale":"2023-10-16T09:59:23.589564+0000","last_undegraded":"2023-10-16T09:59:23.589564+0000","last_fullsized":"2023-10-16T09:59:23.589564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:09.316950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560712+0000","last_change":"2023-10-16T09:59:23.585232+0000","last_active":"2023-10-16T09:59:24.560712+0000","last_peered":"2023-10-16T09:59:24.560712+0000","last_clean":"2023-10-16T09:59:24.560712+0000","last_became_active":"2023-10-16T09:59:23.585000+0000","last_became_peered":"2023-10-16T09:59:23.585000+0000","last_unstale":"2023-10-16T09:59:24.560712+0000","last_undegraded":"2023-10-16T09:59:24.560712+0000","last_fullsized":"2023-10-16T09:59:24.560712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:34.265425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750284+0000","last_change":"2023-10-16T09:59:23.570168+0000","last_active":"2023-10-16T09:59:24.750284+0000","last_peered":"2023-10-16T09:59:24.750284+0000","last_clean":"2023-10-16T09:59:24.750284+0000","last_became_active":"2023-10-16T09:59:23.569825+0000","last_became_peered":"2023-10-16T09:59:23.569825+0000","last_unstale":"2023-10-16T09:59:24.750284+0000","last_undegraded":"2023-10-16T09:59:24.750284+0000","last_fullsized":"2023-10-16T09:59:24.750284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:16:24.387486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560620+0000","last_change":"2023-10-16T09:59:23.572664+0000","last_active":"2023-10-16T09:59:24.560620+0000","last_peered":"2023-10-16T09:59:24.560620+0000","last_clean":"2023-10-16T09:59:24.560620+0000","last_became_active":"2023-10-16T09:59:23.572406+0000","last_became_peered":"2023-10-16T09:59:23.572406+0000","last_unstale":"2023-10-16T09:59:24.560620+0000","last_undegraded":"2023-10-16T09:59:24.560620+0000","last_fullsized":"2023-10-16T09:59:24.560620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:57:06.869600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560797+0000","last_change":"2023-10-16T09:59:23.572835+0000","last_active":"2023-10-16T09:59:24.560797+0000","last_peered":"2023-10-16T09:59:24.560797+0000","last_clean":"2023-10-16T09:59:24.560797+0000","last_became_active":"2023-10-16T09:59:23.572468+0000","last_became_peered":"2023-10-16T09:59:23.572468+0000","last_unstale":"2023-10-16T09:59:24.560797+0000","last_undegraded":"2023-10-16T09:59:24.560797+0000","last_fullsized":"2023-10-16T09:59:24.560797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:03.011626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729208+0000","last_change":"2023-10-16T09:59:23.569594+0000","last_active":"2023-10-16T09:59:24.729208+0000","last_peered":"2023-10-16T09:59:24.729208+0000","last_clean":"2023-10-16T09:59:24.729208+0000","last_became_active":"2023-10-16T09:59:23.568421+0000","last_became_peered":"2023-10-16T09:59:23.568421+0000","last_unstale":"2023-10-16T09:59:24.729208+0000","last_undegraded":"2023-10-16T09:59:24.729208+0000","last_fullsized":"2023-10-16T09:59:24.729208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:23.959799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750432+0000","last_change":"2023-10-16T09:59:23.572894+0000","last_active":"2023-10-16T09:59:24.750432+0000","last_peered":"2023-10-16T09:59:24.750432+0000","last_clean":"2023-10-16T09:59:24.750432+0000","last_became_active":"2023-10-16T09:59:23.572623+0000","last_became_peered":"2023-10-16T09:59:23.572623+0000","last_unstale":"2023-10-16T09:59:24.750432+0000","last_undegraded":"2023-10-16T09:59:24.750432+0000","last_fullsized":"2023-10-16T09:59:24.750432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:52.467502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750261+0000","last_change":"2023-10-16T09:59:23.573382+0000","last_active":"2023-10-16T09:59:24.750261+0000","last_peered":"2023-10-16T09:59:24.750261+0000","last_clean":"2023-10-16T09:59:24.750261+0000","last_became_active":"2023-10-16T09:59:23.570211+0000","last_became_peered":"2023-10-16T09:59:23.570211+0000","last_unstale":"2023-10-16T09:59:24.750261+0000","last_undegraded":"2023-10-16T09:59:24.750261+0000","last_fullsized":"2023-10-16T09:59:24.750261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:43.411176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560531+0000","last_change":"2023-10-16T09:59:23.572236+0000","last_active":"2023-10-16T09:59:24.560531+0000","last_peered":"2023-10-16T09:59:24.560531+0000","last_clean":"2023-10-16T09:59:24.560531+0000","last_became_active":"2023-10-16T09:59:23.571337+0000","last_became_peered":"2023-10-16T09:59:23.571337+0000","last_unstale":"2023-10-16T09:59:24.560531+0000","last_undegraded":"2023-10-16T09:59:24.560531+0000","last_fullsized":"2023-10-16T09:59:24.560531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:45.997463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559107+0000","last_change":"2023-10-16T09:59:23.584795+0000","last_active":"2023-10-16T09:59:24.559107+0000","last_peered":"2023-10-16T09:59:24.559107+0000","last_clean":"2023-10-16T09:59:24.559107+0000","last_became_active":"2023-10-16T09:59:23.584540+0000","last_became_peered":"2023-10-16T09:59:23.584540+0000","last_unstale":"2023-10-16T09:59:24.559107+0000","last_undegraded":"2023-10-16T09:59:24.559107+0000","last_fullsized":"2023-10-16T09:59:24.559107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:30.495916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729605+0000","last_change":"2023-10-16T09:59:23.572168+0000","last_active":"2023-10-16T09:59:24.729605+0000","last_peered":"2023-10-16T09:59:24.729605+0000","last_clean":"2023-10-16T09:59:24.729605+0000","last_became_active":"2023-10-16T09:59:23.571885+0000","last_became_peered":"2023-10-16T09:59:23.571885+0000","last_unstale":"2023-10-16T09:59:24.729605+0000","last_undegraded":"2023-10-16T09:59:24.729605+0000","last_fullsized":"2023-10-16T09:59:24.729605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:24.234610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750057+0000","last_change":"2023-10-16T09:59:23.578383+0000","last_active":"2023-10-16T09:59:24.750057+0000","last_peered":"2023-10-16T09:59:24.750057+0000","last_clean":"2023-10-16T09:59:24.750057+0000","last_became_active":"2023-10-16T09:59:23.578073+0000","last_became_peered":"2023-10-16T09:59:23.578073+0000","last_unstale":"2023-10-16T09:59:24.750057+0000","last_undegraded":"2023-10-16T09:59:24.750057+0000","last_fullsized":"2023-10-16T09:59:24.750057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:26.719461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560090+0000","last_change":"2023-10-16T09:59:23.570768+0000","last_active":"2023-10-16T09:59:24.560090+0000","last_peered":"2023-10-16T09:59:24.560090+0000","last_clean":"2023-10-16T09:59:24.560090+0000","last_became_active":"2023-10-16T09:59:23.570004+0000","last_became_peered":"2023-10-16T09:59:23.570004+0000","last_unstale":"2023-10-16T09:59:24.560090+0000","last_undegraded":"2023-10-16T09:59:24.560090+0000","last_fullsized":"2023-10-16T09:59:24.560090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:31.836130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.570515+0000","last_change":"2023-10-16T09:59:23.567140+0000","last_active":"2023-10-16T09:59:23.570515+0000","last_peered":"2023-10-16T09:59:23.570515+0000","last_clean":"2023-10-16T09:59:23.570515+0000","last_became_active":"2023-10-16T09:59:23.566912+0000","last_became_peered":"2023-10-16T09:59:23.566912+0000","last_unstale":"2023-10-16T09:59:23.570515+0000","last_undegraded":"2023-10-16T09:59:23.570515+0000","last_fullsized":"2023-10-16T09:59:23.570515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:01.740391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589620+0000","last_change":"2023-10-16T09:59:23.584662+0000","last_active":"2023-10-16T09:59:23.589620+0000","last_peered":"2023-10-16T09:59:23.589620+0000","last_clean":"2023-10-16T09:59:23.589620+0000","last_became_active":"2023-10-16T09:59:23.584353+0000","last_became_peered":"2023-10-16T09:59:23.584353+0000","last_unstale":"2023-10-16T09:59:23.589620+0000","last_undegraded":"2023-10-16T09:59:23.589620+0000","last_fullsized":"2023-10-16T09:59:23.589620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:03.715288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559495+0000","last_change":"2023-10-16T09:59:23.135009+0000","last_active":"2023-10-16T09:59:24.559495+0000","last_peered":"2023-10-16T09:59:24.559495+0000","last_clean":"2023-10-16T09:59:24.559495+0000","last_became_active":"2023-10-16T09:59:23.134763+0000","last_became_peered":"2023-10-16T09:59:23.134763+0000","last_unstale":"2023-10-16T09:59:24.559495+0000","last_undegraded":"2023-10-16T09:59:24.559495+0000","last_fullsized":"2023-10-16T09:59:24.559495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:48.255262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560657+0000","last_change":"2023-10-16T09:59:23.584704+0000","last_active":"2023-10-16T09:59:24.560657+0000","last_peered":"2023-10-16T09:59:24.560657+0000","last_clean":"2023-10-16T09:59:24.560657+0000","last_became_active":"2023-10-16T09:59:23.584416+0000","last_became_peered":"2023-10-16T09:59:23.584416+0000","last_unstale":"2023-10-16T09:59:24.560657+0000","last_undegraded":"2023-10-16T09:59:24.560657+0000","last_fullsized":"2023-10-16T09:59:24.560657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:43.443423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.136189+0000","last_change":"2023-10-16T09:59:23.133680+0000","last_active":"2023-10-16T09:59:24.136189+0000","last_peered":"2023-10-16T09:59:24.136189+0000","last_clean":"2023-10-16T09:59:24.136189+0000","last_became_active":"2023-10-16T09:59:23.133385+0000","last_became_peered":"2023-10-16T09:59:23.133385+0000","last_unstale":"2023-10-16T09:59:24.136189+0000","last_undegraded":"2023-10-16T09:59:24.136189+0000","last_fullsized":"2023-10-16T09:59:24.136189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:03.999236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.565879+0000","last_change":"2023-10-16T09:59:23.565182+0000","last_active":"2023-10-16T09:59:23.565879+0000","last_peered":"2023-10-16T09:59:23.565879+0000","last_clean":"2023-10-16T09:59:23.565879+0000","last_became_active":"2023-10-16T09:59:23.564915+0000","last_became_peered":"2023-10-16T09:59:23.564915+0000","last_unstale":"2023-10-16T09:59:23.565879+0000","last_undegraded":"2023-10-16T09:59:23.565879+0000","last_fullsized":"2023-10-16T09:59:23.565879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:23.856491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.134901+0000","last_change":"2023-10-16T09:59:23.139099+0000","last_active":"2023-10-16T09:59:24.134901+0000","last_peered":"2023-10-16T09:59:24.134901+0000","last_clean":"2023-10-16T09:59:24.134901+0000","last_became_active":"2023-10-16T09:59:23.138749+0000","last_became_peered":"2023-10-16T09:59:23.138749+0000","last_unstale":"2023-10-16T09:59:24.134901+0000","last_undegraded":"2023-10-16T09:59:24.134901+0000","last_fullsized":"2023-10-16T09:59:24.134901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:31.164001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729796+0000","last_change":"2023-10-16T09:59:23.569631+0000","last_active":"2023-10-16T09:59:24.729796+0000","last_peered":"2023-10-16T09:59:24.729796+0000","last_clean":"2023-10-16T09:59:24.729796+0000","last_became_active":"2023-10-16T09:59:23.569333+0000","last_became_peered":"2023-10-16T09:59:23.569333+0000","last_unstale":"2023-10-16T09:59:24.729796+0000","last_undegraded":"2023-10-16T09:59:24.729796+0000","last_fullsized":"2023-10-16T09:59:24.729796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:51.453447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.134161+0000","last_change":"2023-10-16T09:59:23.135150+0000","last_active":"2023-10-16T09:59:24.134161+0000","last_peered":"2023-10-16T09:59:24.134161+0000","last_clean":"2023-10-16T09:59:24.134161+0000","last_became_active":"2023-10-16T09:59:23.134903+0000","last_became_peered":"2023-10-16T09:59:23.134903+0000","last_unstale":"2023-10-16T09:59:24.134161+0000","last_undegraded":"2023-10-16T09:59:24.134161+0000","last_fullsized":"2023-10-16T09:59:24.134161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:25:32.755927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.569987+0000","last_change":"2023-10-16T09:59:23.569359+0000","last_active":"2023-10-16T09:59:23.569987+0000","last_peered":"2023-10-16T09:59:23.569987+0000","last_clean":"2023-10-16T09:59:23.569987+0000","last_became_active":"2023-10-16T09:59:23.569068+0000","last_became_peered":"2023-10-16T09:59:23.569068+0000","last_unstale":"2023-10-16T09:59:23.569987+0000","last_undegraded":"2023-10-16T09:59:23.569987+0000","last_fullsized":"2023-10-16T09:59:23.569987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:14.142714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560772+0000","last_change":"2023-10-16T09:59:23.140640+0000","last_active":"2023-10-16T09:59:24.560772+0000","last_peered":"2023-10-16T09:59:24.560772+0000","last_clean":"2023-10-16T09:59:24.560772+0000","last_became_active":"2023-10-16T09:59:23.140369+0000","last_became_peered":"2023-10-16T09:59:23.140369+0000","last_unstale":"2023-10-16T09:59:24.560772+0000","last_undegraded":"2023-10-16T09:59:24.560772+0000","last_fullsized":"2023-10-16T09:59:24.560772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:25.710865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559215+0000","last_change":"2023-10-16T09:59:23.571277+0000","last_active":"2023-10-16T09:59:24.559215+0000","last_peered":"2023-10-16T09:59:24.559215+0000","last_clean":"2023-10-16T09:59:24.559215+0000","last_became_active":"2023-10-16T09:59:23.570919+0000","last_became_peered":"2023-10-16T09:59:23.570919+0000","last_unstale":"2023-10-16T09:59:24.559215+0000","last_undegraded":"2023-10-16T09:59:24.559215+0000","last_fullsized":"2023-10-16T09:59:24.559215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:31.483542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560564+0000","last_change":"2023-10-16T09:59:23.143003+0000","last_active":"2023-10-16T09:59:24.560564+0000","last_peered":"2023-10-16T09:59:24.560564+0000","last_clean":"2023-10-16T09:59:24.560564+0000","last_became_active":"2023-10-16T09:59:23.142744+0000","last_became_peered":"2023-10-16T09:59:23.142744+0000","last_unstale":"2023-10-16T09:59:24.560564+0000","last_undegraded":"2023-10-16T09:59:24.560564+0000","last_fullsized":"2023-10-16T09:59:24.560564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:42.402581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.586758+0000","last_change":"2023-10-16T09:59:23.583814+0000","last_active":"2023-10-16T09:59:23.586758+0000","last_peered":"2023-10-16T09:59:23.586758+0000","last_clean":"2023-10-16T09:59:23.586758+0000","last_became_active":"2023-10-16T09:59:23.583440+0000","last_became_peered":"2023-10-16T09:59:23.583440+0000","last_unstale":"2023-10-16T09:59:23.586758+0000","last_undegraded":"2023-10-16T09:59:23.586758+0000","last_fullsized":"2023-10-16T09:59:23.586758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:50.691937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560593+0000","last_change":"2023-10-16T09:59:23.147098+0000","last_active":"2023-10-16T09:59:24.560593+0000","last_peered":"2023-10-16T09:59:24.560593+0000","last_clean":"2023-10-16T09:59:24.560593+0000","last_became_active":"2023-10-16T09:59:23.146814+0000","last_became_peered":"2023-10-16T09:59:23.146814+0000","last_unstale":"2023-10-16T09:59:24.560593+0000","last_undegraded":"2023-10-16T09:59:24.560593+0000","last_fullsized":"2023-10-16T09:59:24.560593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:23.272223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560715+0000","last_change":"2023-10-16T09:59:23.576763+0000","last_active":"2023-10-16T09:59:24.560715+0000","last_peered":"2023-10-16T09:59:24.560715+0000","last_clean":"2023-10-16T09:59:24.560715+0000","last_became_active":"2023-10-16T09:59:23.576408+0000","last_became_peered":"2023-10-16T09:59:23.576408+0000","last_unstale":"2023-10-16T09:59:24.560715+0000","last_undegraded":"2023-10-16T09:59:24.560715+0000","last_fullsized":"2023-10-16T09:59:24.560715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:37.036382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.569229+0000","last_change":"2023-10-16T09:59:23.147076+0000","last_active":"2023-10-16T09:59:23.569229+0000","last_peered":"2023-10-16T09:59:23.569229+0000","last_clean":"2023-10-16T09:59:23.569229+0000","last_became_active":"2023-10-16T09:59:23.146614+0000","last_became_peered":"2023-10-16T09:59:23.146614+0000","last_unstale":"2023-10-16T09:59:23.569229+0000","last_undegraded":"2023-10-16T09:59:23.569229+0000","last_fullsized":"2023-10-16T09:59:23.569229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:21.366241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559145+0000","last_change":"2023-10-16T09:59:23.563237+0000","last_active":"2023-10-16T09:59:24.559145+0000","last_peered":"2023-10-16T09:59:24.559145+0000","last_clean":"2023-10-16T09:59:24.559145+0000","last_became_active":"2023-10-16T09:59:23.562968+0000","last_became_peered":"2023-10-16T09:59:23.562968+0000","last_unstale":"2023-10-16T09:59:24.559145+0000","last_undegraded":"2023-10-16T09:59:24.559145+0000","last_fullsized":"2023-10-16T09:59:24.559145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:54:54.741956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729836+0000","last_change":"2023-10-16T09:59:23.133175+0000","last_active":"2023-10-16T09:59:24.729836+0000","last_peered":"2023-10-16T09:59:24.729836+0000","last_clean":"2023-10-16T09:59:24.729836+0000","last_became_active":"2023-10-16T09:59:23.132856+0000","last_became_peered":"2023-10-16T09:59:23.132856+0000","last_unstale":"2023-10-16T09:59:24.729836+0000","last_undegraded":"2023-10-16T09:59:24.729836+0000","last_fullsized":"2023-10-16T09:59:24.729836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:10:35.848074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560712+0000","last_change":"2023-10-16T09:59:23.571587+0000","last_active":"2023-10-16T09:59:24.560712+0000","last_peered":"2023-10-16T09:59:24.560712+0000","last_clean":"2023-10-16T09:59:24.560712+0000","last_became_active":"2023-10-16T09:59:23.571295+0000","last_became_peered":"2023-10-16T09:59:23.571295+0000","last_unstale":"2023-10-16T09:59:24.560712+0000","last_undegraded":"2023-10-16T09:59:24.560712+0000","last_fullsized":"2023-10-16T09:59:24.560712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:17.517871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560751+0000","last_change":"2023-10-16T09:59:23.136723+0000","last_active":"2023-10-16T09:59:24.560751+0000","last_peered":"2023-10-16T09:59:24.560751+0000","last_clean":"2023-10-16T09:59:24.560751+0000","last_became_active":"2023-10-16T09:59:23.136416+0000","last_became_peered":"2023-10-16T09:59:23.136416+0000","last_unstale":"2023-10-16T09:59:24.560751+0000","last_undegraded":"2023-10-16T09:59:24.560751+0000","last_fullsized":"2023-10-16T09:59:24.560751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:19.894024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589632+0000","last_change":"2023-10-16T09:59:23.583632+0000","last_active":"2023-10-16T09:59:23.589632+0000","last_peered":"2023-10-16T09:59:23.589632+0000","last_clean":"2023-10-16T09:59:23.589632+0000","last_became_active":"2023-10-16T09:59:23.583332+0000","last_became_peered":"2023-10-16T09:59:23.583332+0000","last_unstale":"2023-10-16T09:59:23.589632+0000","last_undegraded":"2023-10-16T09:59:23.589632+0000","last_fullsized":"2023-10-16T09:59:23.589632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:12.826867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729453+0000","last_change":"2023-10-16T09:59:23.132607+0000","last_active":"2023-10-16T09:59:24.729453+0000","last_peered":"2023-10-16T09:59:24.729453+0000","last_clean":"2023-10-16T09:59:24.729453+0000","last_became_active":"2023-10-16T09:59:23.132372+0000","last_became_peered":"2023-10-16T09:59:23.132372+0000","last_unstale":"2023-10-16T09:59:24.729453+0000","last_undegraded":"2023-10-16T09:59:24.729453+0000","last_fullsized":"2023-10-16T09:59:24.729453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:00.731796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560789+0000","last_change":"2023-10-16T09:59:23.571932+0000","last_active":"2023-10-16T09:59:24.560789+0000","last_peered":"2023-10-16T09:59:24.560789+0000","last_clean":"2023-10-16T09:59:24.560789+0000","last_became_active":"2023-10-16T09:59:23.571636+0000","last_became_peered":"2023-10-16T09:59:23.571636+0000","last_unstale":"2023-10-16T09:59:24.560789+0000","last_undegraded":"2023-10-16T09:59:24.560789+0000","last_fullsized":"2023-10-16T09:59:24.560789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:08.278546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559240+0000","last_change":"2023-10-16T09:59:23.140254+0000","last_active":"2023-10-16T09:59:24.559240+0000","last_peered":"2023-10-16T09:59:24.559240+0000","last_clean":"2023-10-16T09:59:24.559240+0000","last_became_active":"2023-10-16T09:59:23.139983+0000","last_became_peered":"2023-10-16T09:59:23.139983+0000","last_unstale":"2023-10-16T09:59:24.559240+0000","last_undegraded":"2023-10-16T09:59:24.559240+0000","last_fullsized":"2023-10-16T09:59:24.559240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:12.787645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560050+0000","last_change":"2023-10-16T09:59:23.579928+0000","last_active":"2023-10-16T09:59:24.560050+0000","last_peered":"2023-10-16T09:59:24.560050+0000","last_clean":"2023-10-16T09:59:24.560050+0000","last_became_active":"2023-10-16T09:59:23.579653+0000","last_became_peered":"2023-10-16T09:59:23.579653+0000","last_unstale":"2023-10-16T09:59:24.560050+0000","last_undegraded":"2023-10-16T09:59:24.560050+0000","last_fullsized":"2023-10-16T09:59:24.560050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:30.854975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729488+0000","last_change":"2023-10-16T09:59:23.147274+0000","last_active":"2023-10-16T09:59:24.729488+0000","last_peered":"2023-10-16T09:59:24.729488+0000","last_clean":"2023-10-16T09:59:24.729488+0000","last_became_active":"2023-10-16T09:59:23.147022+0000","last_became_peered":"2023-10-16T09:59:23.147022+0000","last_unstale":"2023-10-16T09:59:24.729488+0000","last_undegraded":"2023-10-16T09:59:24.729488+0000","last_fullsized":"2023-10-16T09:59:24.729488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:16.095624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.749957+0000","last_change":"2023-10-16T09:59:23.573241+0000","last_active":"2023-10-16T09:59:24.749957+0000","last_peered":"2023-10-16T09:59:24.749957+0000","last_clean":"2023-10-16T09:59:24.749957+0000","last_became_active":"2023-10-16T09:59:23.572426+0000","last_became_peered":"2023-10-16T09:59:23.572426+0000","last_unstale":"2023-10-16T09:59:24.749957+0000","last_undegraded":"2023-10-16T09:59:24.749957+0000","last_fullsized":"2023-10-16T09:59:24.749957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:24.950903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.749981+0000","last_change":"2023-10-16T09:59:23.140753+0000","last_active":"2023-10-16T09:59:24.749981+0000","last_peered":"2023-10-16T09:59:24.749981+0000","last_clean":"2023-10-16T09:59:24.749981+0000","last_became_active":"2023-10-16T09:59:23.140018+0000","last_became_peered":"2023-10-16T09:59:23.140018+0000","last_unstale":"2023-10-16T09:59:24.749981+0000","last_undegraded":"2023-10-16T09:59:24.749981+0000","last_fullsized":"2023-10-16T09:59:24.749981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:16.672596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.584766+0000","last_change":"2023-10-16T09:59:23.579552+0000","last_active":"2023-10-16T09:59:23.584766+0000","last_peered":"2023-10-16T09:59:23.584766+0000","last_clean":"2023-10-16T09:59:23.584766+0000","last_became_active":"2023-10-16T09:59:23.579090+0000","last_became_peered":"2023-10-16T09:59:23.579090+0000","last_unstale":"2023-10-16T09:59:23.584766+0000","last_undegraded":"2023-10-16T09:59:23.584766+0000","last_fullsized":"2023-10-16T09:59:23.584766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:14.367906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729397+0000","last_change":"2023-10-16T09:59:23.138860+0000","last_active":"2023-10-16T09:59:24.729397+0000","last_peered":"2023-10-16T09:59:24.729397+0000","last_clean":"2023-10-16T09:59:24.729397+0000","last_became_active":"2023-10-16T09:59:23.138509+0000","last_became_peered":"2023-10-16T09:59:23.138509+0000","last_unstale":"2023-10-16T09:59:24.729397+0000","last_undegraded":"2023-10-16T09:59:24.729397+0000","last_fullsized":"2023-10-16T09:59:24.729397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:19.604724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559270+0000","last_change":"2023-10-16T09:59:23.567670+0000","last_active":"2023-10-16T09:59:24.559270+0000","last_peered":"2023-10-16T09:59:24.559270+0000","last_clean":"2023-10-16T09:59:24.559270+0000","last_became_active":"2023-10-16T09:59:23.567413+0000","last_became_peered":"2023-10-16T09:59:23.567413+0000","last_unstale":"2023-10-16T09:59:24.559270+0000","last_undegraded":"2023-10-16T09:59:24.559270+0000","last_fullsized":"2023-10-16T09:59:24.559270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:28.806332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559996+0000","last_change":"2023-10-16T09:59:23.139821+0000","last_active":"2023-10-16T09:59:24.559996+0000","last_peered":"2023-10-16T09:59:24.559996+0000","last_clean":"2023-10-16T09:59:24.559996+0000","last_became_active":"2023-10-16T09:59:23.139564+0000","last_became_peered":"2023-10-16T09:59:23.139564+0000","last_unstale":"2023-10-16T09:59:24.559996+0000","last_undegraded":"2023-10-16T09:59:24.559996+0000","last_fullsized":"2023-10-16T09:59:24.559996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:47.749148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729661+0000","last_change":"2023-10-16T09:59:23.572841+0000","last_active":"2023-10-16T09:59:24.729661+0000","last_peered":"2023-10-16T09:59:24.729661+0000","last_clean":"2023-10-16T09:59:24.729661+0000","last_became_active":"2023-10-16T09:59:23.572527+0000","last_became_peered":"2023-10-16T09:59:23.572527+0000","last_unstale":"2023-10-16T09:59:24.729661+0000","last_undegraded":"2023-10-16T09:59:24.729661+0000","last_fullsized":"2023-10-16T09:59:24.729661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:42.588750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750449+0000","last_change":"2023-10-16T09:59:23.140854+0000","last_active":"2023-10-16T09:59:24.750449+0000","last_peered":"2023-10-16T09:59:24.750449+0000","last_clean":"2023-10-16T09:59:24.750449+0000","last_became_active":"2023-10-16T09:59:23.140537+0000","last_became_peered":"2023-10-16T09:59:23.140537+0000","last_unstale":"2023-10-16T09:59:24.750449+0000","last_undegraded":"2023-10-16T09:59:24.750449+0000","last_fullsized":"2023-10-16T09:59:24.750449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:23.226014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.586784+0000","last_change":"2023-10-16T09:59:23.583528+0000","last_active":"2023-10-16T09:59:23.586784+0000","last_peered":"2023-10-16T09:59:23.586784+0000","last_clean":"2023-10-16T09:59:23.586784+0000","last_became_active":"2023-10-16T09:59:23.583230+0000","last_became_peered":"2023-10-16T09:59:23.583230+0000","last_unstale":"2023-10-16T09:59:23.586784+0000","last_undegraded":"2023-10-16T09:59:23.586784+0000","last_fullsized":"2023-10-16T09:59:23.586784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:07.770406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729644+0000","last_change":"2023-10-16T09:59:23.142607+0000","last_active":"2023-10-16T09:59:24.729644+0000","last_peered":"2023-10-16T09:59:24.729644+0000","last_clean":"2023-10-16T09:59:24.729644+0000","last_became_active":"2023-10-16T09:59:23.142271+0000","last_became_peered":"2023-10-16T09:59:23.142271+0000","last_unstale":"2023-10-16T09:59:24.729644+0000","last_undegraded":"2023-10-16T09:59:24.729644+0000","last_fullsized":"2023-10-16T09:59:24.729644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:57.330886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560571+0000","last_change":"2023-10-16T09:59:23.586363+0000","last_active":"2023-10-16T09:59:24.560571+0000","last_peered":"2023-10-16T09:59:24.560571+0000","last_clean":"2023-10-16T09:59:24.560571+0000","last_became_active":"2023-10-16T09:59:23.586115+0000","last_became_peered":"2023-10-16T09:59:23.586115+0000","last_unstale":"2023-10-16T09:59:24.560571+0000","last_undegraded":"2023-10-16T09:59:24.560571+0000","last_fullsized":"2023-10-16T09:59:24.560571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:11.304253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729743+0000","last_change":"2023-10-16T09:59:23.132575+0000","last_active":"2023-10-16T09:59:24.729743+0000","last_peered":"2023-10-16T09:59:24.729743+0000","last_clean":"2023-10-16T09:59:24.729743+0000","last_became_active":"2023-10-16T09:59:23.132043+0000","last_became_peered":"2023-10-16T09:59:23.132043+0000","last_unstale":"2023-10-16T09:59:24.729743+0000","last_undegraded":"2023-10-16T09:59:24.729743+0000","last_fullsized":"2023-10-16T09:59:24.729743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:20.065856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589623+0000","last_change":"2023-10-16T09:59:23.584736+0000","last_active":"2023-10-16T09:59:23.589623+0000","last_peered":"2023-10-16T09:59:23.589623+0000","last_clean":"2023-10-16T09:59:23.589623+0000","last_became_active":"2023-10-16T09:59:23.584473+0000","last_became_peered":"2023-10-16T09:59:23.584473+0000","last_unstale":"2023-10-16T09:59:23.589623+0000","last_undegraded":"2023-10-16T09:59:23.589623+0000","last_fullsized":"2023-10-16T09:59:23.589623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:20.902619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560818+0000","last_change":"2023-10-16T09:59:23.139799+0000","last_active":"2023-10-16T09:59:24.560818+0000","last_peered":"2023-10-16T09:59:24.560818+0000","last_clean":"2023-10-16T09:59:24.560818+0000","last_became_active":"2023-10-16T09:59:23.139533+0000","last_became_peered":"2023-10-16T09:59:23.139533+0000","last_unstale":"2023-10-16T09:59:24.560818+0000","last_undegraded":"2023-10-16T09:59:24.560818+0000","last_fullsized":"2023-10-16T09:59:24.560818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:13:08.301545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.565655+0000","last_change":"2023-10-16T09:59:23.565028+0000","last_active":"2023-10-16T09:59:23.565655+0000","last_peered":"2023-10-16T09:59:23.565655+0000","last_clean":"2023-10-16T09:59:23.565655+0000","last_became_active":"2023-10-16T09:59:23.563007+0000","last_became_peered":"2023-10-16T09:59:23.563007+0000","last_unstale":"2023-10-16T09:59:23.565655+0000","last_undegraded":"2023-10-16T09:59:23.565655+0000","last_fullsized":"2023-10-16T09:59:23.565655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:57.217307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.136027+0000","last_change":"2023-10-16T09:59:23.147188+0000","last_active":"2023-10-16T09:59:24.136027+0000","last_peered":"2023-10-16T09:59:24.136027+0000","last_clean":"2023-10-16T09:59:24.136027+0000","last_became_active":"2023-10-16T09:59:23.146915+0000","last_became_peered":"2023-10-16T09:59:23.146915+0000","last_unstale":"2023-10-16T09:59:24.136027+0000","last_undegraded":"2023-10-16T09:59:24.136027+0000","last_fullsized":"2023-10-16T09:59:24.136027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:39.944738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750070+0000","last_change":"2023-10-16T09:59:23.570277+0000","last_active":"2023-10-16T09:59:24.750070+0000","last_peered":"2023-10-16T09:59:24.750070+0000","last_clean":"2023-10-16T09:59:24.750070+0000","last_became_active":"2023-10-16T09:59:23.569934+0000","last_became_peered":"2023-10-16T09:59:23.569934+0000","last_unstale":"2023-10-16T09:59:24.750070+0000","last_undegraded":"2023-10-16T09:59:24.750070+0000","last_fullsized":"2023-10-16T09:59:24.750070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:13:09.310141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750104+0000","last_change":"2023-10-16T09:59:23.140960+0000","last_active":"2023-10-16T09:59:24.750104+0000","last_peered":"2023-10-16T09:59:24.750104+0000","last_clean":"2023-10-16T09:59:24.750104+0000","last_became_active":"2023-10-16T09:59:23.140435+0000","last_became_peered":"2023-10-16T09:59:23.140435+0000","last_unstale":"2023-10-16T09:59:24.750104+0000","last_undegraded":"2023-10-16T09:59:24.750104+0000","last_fullsized":"2023-10-16T09:59:24.750104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:10:35.848074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589611+0000","last_change":"2023-10-16T09:59:23.584676+0000","last_active":"2023-10-16T09:59:23.589611+0000","last_peered":"2023-10-16T09:59:23.589611+0000","last_clean":"2023-10-16T09:59:23.589611+0000","last_became_active":"2023-10-16T09:59:23.584395+0000","last_became_peered":"2023-10-16T09:59:23.584395+0000","last_unstale":"2023-10-16T09:59:23.589611+0000","last_undegraded":"2023-10-16T09:59:23.589611+0000","last_fullsized":"2023-10-16T09:59:23.589611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:54.878354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.559234+0000","last_change":"2023-10-16T09:59:23.133674+0000","last_active":"2023-10-16T09:59:23.559234+0000","last_peered":"2023-10-16T09:59:23.559234+0000","last_clean":"2023-10-16T09:59:23.559234+0000","last_became_active":"2023-10-16T09:59:23.133407+0000","last_became_peered":"2023-10-16T09:59:23.133407+0000","last_unstale":"2023-10-16T09:59:23.559234+0000","last_undegraded":"2023-10-16T09:59:23.559234+0000","last_fullsized":"2023-10-16T09:59:23.559234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:55.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.574777+0000","last_change":"2023-10-16T09:59:23.572823+0000","last_active":"2023-10-16T09:59:23.574777+0000","last_peered":"2023-10-16T09:59:23.574777+0000","last_clean":"2023-10-16T09:59:23.574777+0000","last_became_active":"2023-10-16T09:59:23.572568+0000","last_became_peered":"2023-10-16T09:59:23.572568+0000","last_unstale":"2023-10-16T09:59:23.574777+0000","last_undegraded":"2023-10-16T09:59:23.574777+0000","last_fullsized":"2023-10-16T09:59:23.574777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:57.141048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750499+0000","last_change":"2023-10-16T09:59:23.140261+0000","last_active":"2023-10-16T09:59:24.750499+0000","last_peered":"2023-10-16T09:59:24.750499+0000","last_clean":"2023-10-16T09:59:24.750499+0000","last_became_active":"2023-10-16T09:59:23.139672+0000","last_became_peered":"2023-10-16T09:59:23.139672+0000","last_unstale":"2023-10-16T09:59:24.750499+0000","last_undegraded":"2023-10-16T09:59:24.750499+0000","last_fullsized":"2023-10-16T09:59:24.750499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:23.272223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.567451+0000","last_change":"2023-10-16T09:59:23.566239+0000","last_active":"2023-10-16T09:59:23.567451+0000","last_peered":"2023-10-16T09:59:23.567451+0000","last_clean":"2023-10-16T09:59:23.567451+0000","last_became_active":"2023-10-16T09:59:23.565855+0000","last_became_peered":"2023-10-16T09:59:23.565855+0000","last_unstale":"2023-10-16T09:59:23.567451+0000","last_undegraded":"2023-10-16T09:59:23.567451+0000","last_fullsized":"2023-10-16T09:59:23.567451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:22.591205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750378+0000","last_change":"2023-10-16T09:59:23.141475+0000","last_active":"2023-10-16T09:59:24.750378+0000","last_peered":"2023-10-16T09:59:24.750378+0000","last_clean":"2023-10-16T09:59:24.750378+0000","last_became_active":"2023-10-16T09:59:23.141196+0000","last_became_peered":"2023-10-16T09:59:23.141196+0000","last_unstale":"2023-10-16T09:59:24.750378+0000","last_undegraded":"2023-10-16T09:59:24.750378+0000","last_fullsized":"2023-10-16T09:59:24.750378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:22.951203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750428+0000","last_change":"2023-10-16T09:59:23.573584+0000","last_active":"2023-10-16T09:59:24.750428+0000","last_peered":"2023-10-16T09:59:24.750428+0000","last_clean":"2023-10-16T09:59:24.750428+0000","last_became_active":"2023-10-16T09:59:23.573084+0000","last_became_peered":"2023-10-16T09:59:23.573084+0000","last_unstale":"2023-10-16T09:59:24.750428+0000","last_undegraded":"2023-10-16T09:59:24.750428+0000","last_fullsized":"2023-10-16T09:59:24.750428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:30.767880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.559608+0000","last_change":"2023-10-16T09:59:23.138796+0000","last_active":"2023-10-16T09:59:23.559608+0000","last_peered":"2023-10-16T09:59:23.559608+0000","last_clean":"2023-10-16T09:59:23.559608+0000","last_became_active":"2023-10-16T09:59:23.138559+0000","last_became_peered":"2023-10-16T09:59:23.138559+0000","last_unstale":"2023-10-16T09:59:23.559608+0000","last_undegraded":"2023-10-16T09:59:23.559608+0000","last_fullsized":"2023-10-16T09:59:23.559608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:58.209416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589709+0000","last_change":"2023-10-16T09:59:23.585296+0000","last_active":"2023-10-16T09:59:23.589709+0000","last_peered":"2023-10-16T09:59:23.589709+0000","last_clean":"2023-10-16T09:59:23.589709+0000","last_became_active":"2023-10-16T09:59:23.584241+0000","last_became_peered":"2023-10-16T09:59:23.584241+0000","last_unstale":"2023-10-16T09:59:23.589709+0000","last_undegraded":"2023-10-16T09:59:23.589709+0000","last_fullsized":"2023-10-16T09:59:23.589709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:09.755265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559219+0000","last_change":"2023-10-16T09:59:23.138362+0000","last_active":"2023-10-16T09:59:24.559219+0000","last_peered":"2023-10-16T09:59:24.559219+0000","last_clean":"2023-10-16T09:59:24.559219+0000","last_became_active":"2023-10-16T09:59:23.138100+0000","last_became_peered":"2023-10-16T09:59:23.138100+0000","last_unstale":"2023-10-16T09:59:24.559219+0000","last_undegraded":"2023-10-16T09:59:24.559219+0000","last_fullsized":"2023-10-16T09:59:24.559219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:16.178113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589644+0000","last_change":"2023-10-16T09:59:23.584866+0000","last_active":"2023-10-16T09:59:23.589644+0000","last_peered":"2023-10-16T09:59:23.589644+0000","last_clean":"2023-10-16T09:59:23.589644+0000","last_became_active":"2023-10-16T09:59:23.583216+0000","last_became_peered":"2023-10-16T09:59:23.583216+0000","last_unstale":"2023-10-16T09:59:23.589644+0000","last_undegraded":"2023-10-16T09:59:23.589644+0000","last_fullsized":"2023-10-16T09:59:23.589644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:22.128625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729647+0000","last_change":"2023-10-16T09:59:23.142484+0000","last_active":"2023-10-16T09:59:24.729647+0000","last_peered":"2023-10-16T09:59:24.729647+0000","last_clean":"2023-10-16T09:59:24.729647+0000","last_became_active":"2023-10-16T09:59:23.142213+0000","last_became_peered":"2023-10-16T09:59:23.142213+0000","last_unstale":"2023-10-16T09:59:24.729647+0000","last_undegraded":"2023-10-16T09:59:24.729647+0000","last_fullsized":"2023-10-16T09:59:24.729647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:36.220534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560244+0000","last_change":"2023-10-16T09:59:23.575511+0000","last_active":"2023-10-16T09:59:24.560244+0000","last_peered":"2023-10-16T09:59:24.560244+0000","last_clean":"2023-10-16T09:59:24.560244+0000","last_became_active":"2023-10-16T09:59:23.575157+0000","last_became_peered":"2023-10-16T09:59:23.575157+0000","last_unstale":"2023-10-16T09:59:24.560244+0000","last_undegraded":"2023-10-16T09:59:24.560244+0000","last_fullsized":"2023-10-16T09:59:24.560244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:28.431724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560656+0000","last_change":"2023-10-16T09:59:23.141121+0000","last_active":"2023-10-16T09:59:24.560656+0000","last_peered":"2023-10-16T09:59:24.560656+0000","last_clean":"2023-10-16T09:59:24.560656+0000","last_became_active":"2023-10-16T09:59:23.140876+0000","last_became_peered":"2023-10-16T09:59:23.140876+0000","last_unstale":"2023-10-16T09:59:24.560656+0000","last_undegraded":"2023-10-16T09:59:24.560656+0000","last_fullsized":"2023-10-16T09:59:24.560656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:16:23.378890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559039+0000","last_change":"2023-10-16T09:59:23.571192+0000","last_active":"2023-10-16T09:59:24.559039+0000","last_peered":"2023-10-16T09:59:24.559039+0000","last_clean":"2023-10-16T09:59:24.559039+0000","last_became_active":"2023-10-16T09:59:23.570948+0000","last_became_peered":"2023-10-16T09:59:23.570948+0000","last_unstale":"2023-10-16T09:59:24.559039+0000","last_undegraded":"2023-10-16T09:59:24.559039+0000","last_fullsized":"2023-10-16T09:59:24.559039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:10.087841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560678+0000","last_change":"2023-10-16T09:59:23.147218+0000","last_active":"2023-10-16T09:59:24.560678+0000","last_peered":"2023-10-16T09:59:24.560678+0000","last_clean":"2023-10-16T09:59:24.560678+0000","last_became_active":"2023-10-16T09:59:23.146899+0000","last_became_peered":"2023-10-16T09:59:23.146899+0000","last_unstale":"2023-10-16T09:59:24.560678+0000","last_undegraded":"2023-10-16T09:59:24.560678+0000","last_fullsized":"2023-10-16T09:59:24.560678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:23.942308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.573247+0000","last_change":"2023-10-16T09:59:23.570493+0000","last_active":"2023-10-16T09:59:23.573247+0000","last_peered":"2023-10-16T09:59:23.573247+0000","last_clean":"2023-10-16T09:59:23.573247+0000","last_became_active":"2023-10-16T09:59:23.570000+0000","last_became_peered":"2023-10-16T09:59:23.570000+0000","last_unstale":"2023-10-16T09:59:23.573247+0000","last_undegraded":"2023-10-16T09:59:23.573247+0000","last_fullsized":"2023-10-16T09:59:23.573247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:18.527228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.558671+0000","last_change":"2023-10-16T09:59:23.147119+0000","last_active":"2023-10-16T09:59:24.558671+0000","last_peered":"2023-10-16T09:59:24.558671+0000","last_clean":"2023-10-16T09:59:24.558671+0000","last_became_active":"2023-10-16T09:59:23.146692+0000","last_became_peered":"2023-10-16T09:59:23.146692+0000","last_unstale":"2023-10-16T09:59:24.558671+0000","last_undegraded":"2023-10-16T09:59:24.558671+0000","last_fullsized":"2023-10-16T09:59:24.558671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:44.988868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589745+0000","last_change":"2023-10-16T09:59:23.585868+0000","last_active":"2023-10-16T09:59:23.589745+0000","last_peered":"2023-10-16T09:59:23.589745+0000","last_clean":"2023-10-16T09:59:23.589745+0000","last_became_active":"2023-10-16T09:59:23.584053+0000","last_became_peered":"2023-10-16T09:59:23.584053+0000","last_unstale":"2023-10-16T09:59:23.589745+0000","last_undegraded":"2023-10-16T09:59:23.589745+0000","last_fullsized":"2023-10-16T09:59:23.589745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:54:39.398124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560522+0000","last_change":"2023-10-16T09:59:23.136471+0000","last_active":"2023-10-16T09:59:24.560522+0000","last_peered":"2023-10-16T09:59:24.560522+0000","last_clean":"2023-10-16T09:59:24.560522+0000","last_became_active":"2023-10-16T09:59:23.136241+0000","last_became_peered":"2023-10-16T09:59:23.136241+0000","last_unstale":"2023-10-16T09:59:24.560522+0000","last_undegraded":"2023-10-16T09:59:24.560522+0000","last_fullsized":"2023-10-16T09:59:24.560522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:41.580154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":387,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729291+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:59:24.729291+0000","last_peered":"2023-10-16T09:59:24.729291+0000","last_clean":"2023-10-16T09:59:24.729291+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:59:24.729291+0000","last_undegraded":"2023-10-16T09:59:24.729291+0000","last_fullsized":"2023-10-16T09:59:24.729291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834575042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1024000,"data_stored":522124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57099999999999995,"15min":0.58599999999999997},"min":{"1min":0.46800000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.79400000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.73199999999999998,"5min":0.78800000000000003,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83699999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59299999999999997,"15min":0.59999999999999998},"min":{"1min":0.51900000000000002,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.72199999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.39300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71999999999999997,"5min":1.448,"15min":1.448},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.52700000000000002,"5min":0.35899999999999999,"15min":0.317},"max":{"1min":0.874,"5min":1.708,"15min":1.708},"last":0.65400000000000003}]},{"osd":3,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.38800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.81699999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.46600000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.69299999999999995,"5min":0.77100000000000002,"15min":0.82999999999999996},"last":0.626}]},{"osd":4,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45600000000000002,"15min":0.48899999999999999},"min":{"1min":0.32900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":0.77400000000000002,"15min":0.93200000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.49199999999999999,"15min":0.51600000000000001},"min":{"1min":0.38200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67300000000000004,"5min":0.75700000000000001,"15min":0.85399999999999998},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.495,"15min":0.50600000000000001},"min":{"1min":0.38300000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.73899999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.81299999999999994,"15min":0.89200000000000002},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54500000000000004,"15min":0.56100000000000005},"min":{"1min":0.40400000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85299999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78000000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.57299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46064,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325776,"statfs":{"total":96636764160,"available":96589594624,"internally_reserved":0,"allocated":1024000,"data_stored":522112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63400000000000001,"15min":0.61199999999999999},"min":{"1min":0.35299999999999998,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":0.84199999999999997,"5min":1.792,"15min":1.792},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63300000000000001,"15min":0.61899999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.85699999999999998,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.57899999999999996},"min":{"1min":0.46100000000000002,"5min":0.315,"15min":0.249},"max":{"1min":0.745,"5min":0.875,"15min":1.1279999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.42599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":1.0720000000000001},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.70299999999999996,"5min":1.496,"15min":1.496},"last":0.432},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.622,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.879,"5min":2.069,"15min":2.069},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76900000000000002,"5min":0.78000000000000003,"15min":0.84499999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61299999999999999,"15min":0.59499999999999997},"min":{"1min":0.46000000000000002,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.89100000000000001,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.63}]},{"osd":4,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47599999999999998,"15min":0.47799999999999998},"min":{"1min":0.34699999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64600000000000002,"5min":0.747,"15min":0.84199999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.41999999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.85399999999999998,"15min":0.86899999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48999999999999999,"15min":0.49299999999999999},"min":{"1min":0.35899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76000000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54100000000000004},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70799999999999996,"5min":0.82299999999999995,"15min":0.88200000000000001},"last":0.745}]},{"osd":7,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76700000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.44800000000000001,"5min":0.34100000000000003,"15min":0.315},"max":{"1min":0.83499999999999996,"5min":0.876,"15min":0.88500000000000001},"last":0.78700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54392,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317448,"statfs":{"total":96636764160,"available":96581066752,"internally_reserved":0,"allocated":1019904,"data_stored":522090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.38400000000000001},"min":{"1min":0.311,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.254},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.316,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54000000000000004,"5min":0.64300000000000002,"15min":0.67500000000000004},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.754},"last":0.312},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.29199999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.505,"5min":0.60299999999999998,"15min":0.624},"last":0.27200000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41699999999999998,"15min":0.41899999999999998},"min":{"1min":0.34000000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.47099999999999997,"15min":0.48199999999999998},"min":{"1min":0.379,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.373}]},{"osd":4,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.51100000000000001,"5min":0.36799999999999999,"15min":0.36299999999999999},"max":{"1min":0.81200000000000006,"5min":1.0309999999999999,"15min":1.0700000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.65800000000000003,"15min":0.65600000000000003},"min":{"1min":0.52400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82799999999999996,"5min":0.95099999999999996,"15min":1.071},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.65600000000000003,"15min":0.65400000000000003},"min":{"1min":0.54600000000000004,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":0.92900000000000005,"5min":1.103,"15min":1.103},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.58999999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85599999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.42999999999999999}]},{"osd":6,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48999999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67500000000000004,"15min":0.68500000000000005},"min":{"1min":0.57799999999999996,"5min":0.40799999999999997,"15min":0.39200000000000002},"max":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.441}]},{"osd":7,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68400000000000005,"15min":0.68600000000000005},"min":{"1min":0.56000000000000005,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88200000000000001,"5min":1.0069999999999999,"15min":1.107},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68200000000000005,"15min":0.67400000000000004},"min":{"1min":0.58299999999999996,"5min":0.371,"15min":0.371},"max":{"1min":0.95299999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.46800000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1613824,"data_stored":1112458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38900000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.58499999999999996,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.441,"15min":0.42999999999999999},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.245},"max":{"1min":0.70999999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.497}]},{"osd":2,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41699999999999998,"15min":0.41799999999999998},"min":{"1min":0.32900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.56899999999999995,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.38100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.27800000000000002,"15min":0.217},"max":{"1min":0.60299999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.36899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.622,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.52600000000000002,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.622,"15min":0.624},"min":{"1min":0.46000000000000002,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":0.74299999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.81100000000000005}]},{"osd":5,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.626,"15min":0.63600000000000001},"min":{"1min":0.52400000000000002,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61599999999999999,"15min":0.64000000000000001},"min":{"1min":0.53500000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.74199999999999999,"5min":0.88700000000000001,"15min":5.2240000000000002},"last":0.82599999999999996}]},{"osd":6,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66000000000000003},"min":{"1min":0.54300000000000004,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77100000000000002,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66600000000000004,"15min":0.67100000000000004},"min":{"1min":0.59699999999999998,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.85799999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":1.014,"5min":1.014,"15min":1.02},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.55600000000000005,"5min":0.41299999999999998,"15min":0.34899999999999998},"max":{"1min":0.77200000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.876}]}]},{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59148,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312692,"statfs":{"total":96636764160,"available":96576196608,"internally_reserved":0,"allocated":1630208,"data_stored":1113234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.56100000000000005},"min":{"1min":0.45800000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":0.68400000000000005,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57199999999999995,"15min":0.60799999999999998},"min":{"1min":0.42499999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.72299999999999998,"5min":1.9059999999999999,"15min":11.246},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.59199999999999997,"15min":0.61299999999999999},"min":{"1min":0.51700000000000002,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78000000000000003,"5min":0.79300000000000004,"15min":1.1439999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.55600000000000005,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.76900000000000002,"5min":1.1279999999999999,"15min":1.212},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.81200000000000006,"5min":1.607,"15min":1.607},"last":0.5},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.48699999999999999,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.88700000000000001,"5min":2.044,"15min":2.044},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.47299999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71599999999999997,"5min":1.524,"15min":1.524},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.504,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.81799999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.67400000000000004}]},{"osd":4,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66400000000000003,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.40699999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76200000000000001,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":1.056},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.33200000000000002,"15min":0.308},"max":{"1min":0.84299999999999997,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.752,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.58299999999999996},"min":{"1min":0.36299999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96399999999999997},"last":0.58999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":531510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.25800000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.53400000000000003,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.317},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44400000000000001,"15min":0.439},"min":{"1min":0.36699999999999999,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.60099999999999998,"5min":0.67900000000000005,"15min":1.593},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.40699999999999997},"min":{"1min":0.29099999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53900000000000003,"5min":4.923,"15min":4.923},"last":0.374},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.249,"15min":0.216},"max":{"1min":0.56799999999999995,"5min":1.712,"15min":1.712},"last":0.39800000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47599999999999998,"15min":0.45500000000000002},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.61699999999999999,"5min":1.984,"15min":1.984},"last":0.504},{"interface":"front","average":{"1min":0.44,"5min":0.438,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.65900000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63600000000000001,"15min":0.63100000000000001},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.81200000000000006,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68899999999999995,"15min":0.65800000000000003},"min":{"1min":0.51700000000000002,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.99199999999999999,"5min":5.0069999999999997,"15min":5.141},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.69799999999999995,"15min":0.67700000000000005},"min":{"1min":0.44900000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.91800000000000004,"5min":4.9749999999999996,"15min":5.0789999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.69399999999999995,"15min":0.67700000000000005},"min":{"1min":0.50700000000000001,"5min":0.40500000000000003,"15min":0.40300000000000002},"max":{"1min":0.93100000000000005,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.53400000000000003,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.95199999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69899999999999995,"15min":0.67000000000000004},"min":{"1min":0.55500000000000005,"5min":0.41599999999999998,"15min":0.371},"max":{"1min":0.84099999999999997,"5min":5.0419999999999998,"15min":5.1760000000000002},"last":0.47299999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.47899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.82399999999999995,"5min":1.744,"15min":1.744},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.432,"15min":0.38500000000000001},"max":{"1min":0.90200000000000002,"5min":0.91400000000000003,"15min":1.0329999999999999},"last":0.49199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36356,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335484,"statfs":{"total":96636764160,"available":96599535616,"internally_reserved":0,"allocated":1032192,"data_stored":529134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.378,"15min":0.371},"min":{"1min":0.29099999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.56000000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.27600000000000002,"5min":0.23100000000000001,"15min":0.20999999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.67000000000000004},"last":0.45100000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.39500000000000002},"min":{"1min":0.33400000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51600000000000001,"5min":0.65400000000000003,"15min":0.66100000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.434,"15min":0.42999999999999999},"min":{"1min":0.33800000000000002,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":0.60899999999999999,"5min":0.61899999999999999,"15min":0.67500000000000004},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.46999999999999997,"15min":0.46500000000000002},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":1.0760000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.33600000000000002,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.93799999999999994},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63100000000000001,"15min":0.623},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.90300000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.64900000000000002,"15min":0.64000000000000001},"min":{"1min":0.48299999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.64500000000000002},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65100000000000002,"15min":0.64300000000000002},"min":{"1min":0.45700000000000002,"5min":0.35999999999999999,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.90200000000000002,"15min":1.036},"last":0.57899999999999996}]},{"osd":6,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.66300000000000003,"15min":0.65000000000000002},"min":{"1min":0.51000000000000001,"5min":0.374,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.66100000000000003},"min":{"1min":0.53300000000000003,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.82399999999999995,"5min":0.94299999999999995,"15min":1.0589999999999999},"last":0.55700000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.90600000000000003,"5min":0.90900000000000003,"15min":1.0029999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.47099999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.877,"5min":0.92800000000000005,"15min":0.999},"last":0.52100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1032192,"data_stored":523702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70699999999999996,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.441},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.39400000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.71999999999999997,"5min":1.0860000000000001,"15min":1.3029999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.82599999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57999999999999996,"15min":0.59299999999999997},"min":{"1min":0.45400000000000001,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.73499999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58299999999999996,"15min":0.59799999999999998},"min":{"1min":0.45200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60999999999999999,"15min":0.627},"min":{"1min":0.44,"5min":0.379,"15min":0.318},"max":{"1min":0.76300000000000001,"5min":1.2270000000000001,"15min":1.8899999999999999},"last":0.60999999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58799999999999997,"15min":0.59899999999999998},"min":{"1min":0.47899999999999998,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.77700000000000002,"5min":0.86099999999999999,"15min":1.7909999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":1.1839999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.29999999999999999,"15min":0.28599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.88900000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.50800000000000001,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78500000000000003,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.624}]},{"osd":6,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51600000000000001,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.74299999999999999,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.57999999999999996},"min":{"1min":0.39200000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84899999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56399999999999995,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83099999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60499999999999998,"15min":0.61399999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80000000000000004,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:59:28.236 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T09:59:28.369 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:59:28.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:59:28.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.389+0000 7f287306f700 1 Processor -- start 2023-10-16T09:59:28.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.389+0000 7f287306f700 1 -- start start 2023-10-16T09:59:28.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.390+0000 7f287306f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f286c06bbf0 0x7f286c0ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.390+0000 7f287306f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c0cd600 con 0x7f286c15aa00 2023-10-16T09:59:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.390+0000 7f287306f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c15af30 con 0x7f286c0c6180 2023-10-16T09:59:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.390+0000 7f287306f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c15b210 con 0x7f286c06bbf0 2023-10-16T09:59:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.390+0000 7f287160c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f286c06bbf0 0x7f286c0ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.390+0000 7f286bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f286c15aa00 0x7f286c0cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52472/0 (socket says 172.21.15.158:52472) 2023-10-16T09:59:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.390+0000 7f286bfff700 1 -- 172.21.15.158:0/1059924108 learned_addr learned my addr 172.21.15.158:0/1059924108 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f287160c700 1 -- 172.21.15.158:0/1059924108 >> v1:172.21.15.172:6789/0 conn(0x7f286c0c6180 legacy=0x7f286c0cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f287160c700 1 -- 172.21.15.158:0/1059924108 >> v1:172.21.15.158:6789/0 conn(0x7f286c15aa00 legacy=0x7f286c0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f286b7fe700 1 -- 172.21.15.158:0/1059924108 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193264377 0 0) 0x7f286c0cd600 con 0x7f286c15aa00 2023-10-16T09:59:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f287160c700 1 -- 172.21.15.158:0/1059924108 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c15b4f0 con 0x7f286c06bbf0 2023-10-16T09:59:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f287160c700 1 --2- 172.21.15.158:0/1059924108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06bbf0 0x7f286c0ce1a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f2864009d90 tx=0x7f286400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d070a65a65b3cbf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f286b7fe700 1 -- 172.21.15.158:0/1059924108 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400e070 con 0x7f286c06bbf0 2023-10-16T09:59:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f286b7fe700 1 -- 172.21.15.158:0/1059924108 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286400bbd0 con 0x7f286c06bbf0 2023-10-16T09:59:28.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.391+0000 7f286b7fe700 1 -- 172.21.15.158:0/1059924108 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864012430 con 0x7f286c06bbf0 2023-10-16T09:59:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.392+0000 7f287306f700 1 -- 172.21.15.158:0/1059924108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06bbf0 msgr2=0x7f286c0ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.392+0000 7f287306f700 1 --2- 172.21.15.158:0/1059924108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06bbf0 0x7f286c0ce1a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f2864009d90 tx=0x7f286400b330 comp rx=0 tx=0).stop 2023-10-16T09:59:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.392+0000 7f287306f700 1 -- 172.21.15.158:0/1059924108 shutdown_connections 2023-10-16T09:59:28.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.392+0000 7f287306f700 1 --2- 172.21.15.158:0/1059924108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06bbf0 0x7f286c0ce1a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.392+0000 7f287306f700 1 -- 172.21.15.158:0/1059924108 >> 172.21.15.158:0/1059924108 conn(0x7f286c0cc4b0 msgr2=0x7f286c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:28.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.392+0000 7f287306f700 1 -- 172.21.15.158:0/1059924108 shutdown_connections 2023-10-16T09:59:28.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.392+0000 7f287306f700 1 -- 172.21.15.158:0/1059924108 wait complete. 2023-10-16T09:59:28.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.393+0000 7f287306f700 1 Processor -- start 2023-10-16T09:59:28.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.393+0000 7f287306f700 1 -- start start 2023-10-16T09:59:28.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f287306f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c06bbf0 0x7f286c1f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f287306f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0c6180 0x7f286c156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f287306f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c15aa00 0x7f286c1f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f287306f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1f1b70 con 0x7f286c0c6180 2023-10-16T09:59:28.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f287306f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1f50c0 con 0x7f286c06bbf0 2023-10-16T09:59:28.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f287306f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1f15b0 con 0x7f286c15aa00 2023-10-16T09:59:28.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f286bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0c6180 0x7f286c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f286bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0c6180 0x7f286c156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52302/0 (socket says 172.21.15.158:52302) 2023-10-16T09:59:28.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.394+0000 7f286bfff700 1 -- 172.21.15.158:0/54284260 learned_addr learned my addr 172.21.15.158:0/54284260 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:28.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f2870e0b700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c06bbf0 0x7f286c1f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f287160c700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c15aa00 0x7f286c1f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f286bfff700 1 -- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c15aa00 msgr2=0x7f286c1f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f286bfff700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c15aa00 0x7f286c1f0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f286bfff700 1 -- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c06bbf0 msgr2=0x7f286c1f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f286bfff700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c06bbf0 0x7f286c1f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f286bfff700 1 -- 172.21.15.158:0/54284260 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864009a70 con 0x7f286c0c6180 2023-10-16T09:59:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f286bfff700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0c6180 0x7f286c156070 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f286000c520 tx=0x7f286000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f28697fa700 1 -- 172.21.15.158:0/54284260 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860004c80 con 0x7f286c0c6180 2023-10-16T09:59:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f28697fa700 1 -- 172.21.15.158:0/54284260 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2860004de0 con 0x7f286c0c6180 2023-10-16T09:59:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c1f1890 con 0x7f286c0c6180 2023-10-16T09:59:28.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f28697fa700 1 -- 172.21.15.158:0/54284260 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860007310 con 0x7f286c0c6180 2023-10-16T09:59:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.395+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c1f4660 con 0x7f286c0c6180 2023-10-16T09:59:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.396+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c1f41e0 con 0x7f286c0c6180 2023-10-16T09:59:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.397+0000 7f28697fa700 1 -- 172.21.15.158:0/54284260 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2860007470 con 0x7f286c0c6180 2023-10-16T09:59:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.397+0000 7f28697fa700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f285806b020 0x7f285806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.397+0000 7f28697fa700 1 -- 172.21.15.158:0/54284260 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f28600895f0 con 0x7f286c0c6180 2023-10-16T09:59:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.397+0000 7f2870e0b700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f285806b020 0x7f285806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.398+0000 7f2870e0b700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f285806b020 0x7f285806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f285c000f60 tx=0x7f285c009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.402+0000 7f28697fa700 1 -- 172.21.15.158:0/54284260 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f286c1f41e0 con 0x7f286c0c6180 2023-10-16T09:59:28.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.593+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 --> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f286c0c6560 con 0x7f285806b020 2023-10-16T09:59:28.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.601+0000 7f28697fa700 1 -- 172.21.15.158:0/54284260 <== mgr.35456 v2:172.21.15.158:6832/3717499045 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289552 (secure 0 0 0) 0x7f285c0015c0 con 0x7f285806b020 2023-10-16T09:59:28.606 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:28.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f285806b020 msgr2=0x7f285806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f285806b020 0x7f285806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f285c000f60 tx=0x7f285c009040 comp rx=0 tx=0).stop 2023-10-16T09:59:28.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0c6180 msgr2=0x7f286c156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0c6180 0x7f286c156070 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f286000c520 tx=0x7f286000c830 comp rx=0 tx=0).stop 2023-10-16T09:59:28.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 shutdown_connections 2023-10-16T09:59:28.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f285806b020 0x7f285806d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c15aa00 0x7f286c1f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c06bbf0 0x7f286c1f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 --2- 172.21.15.158:0/54284260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0c6180 0x7f286c156070 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 >> 172.21.15.158:0/54284260 conn(0x7f286c0cc4b0 msgr2=0x7f286c15ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:28.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 shutdown_connections 2023-10-16T09:59:28.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.604+0000 7f287306f700 1 -- 172.21.15.158:0/54284260 wait complete. 2023-10-16T09:59:28.611 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T09:59:28.618 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T09:59:27.075144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":325300,"kb_used_data":9216,"kb_used_omap":12,"kb_used_meta":315955,"kb_avail":754649420,"statfs":{"total":773094113280,"available":772761006080,"internally_reserved":0,"allocated":9437184,"data_stored":5376364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":323538487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.958561"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.558947+0000","last_change":"2023-10-16T09:59:23.574345+0000","last_active":"2023-10-16T09:59:24.558947+0000","last_peered":"2023-10-16T09:59:24.558947+0000","last_clean":"2023-10-16T09:59:24.558947+0000","last_became_active":"2023-10-16T09:59:23.574076+0000","last_became_peered":"2023-10-16T09:59:23.574076+0000","last_unstale":"2023-10-16T09:59:24.558947+0000","last_undegraded":"2023-10-16T09:59:24.558947+0000","last_fullsized":"2023-10-16T09:59:24.558947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:42:48.948770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559187+0000","last_change":"2023-10-16T09:59:23.565351+0000","last_active":"2023-10-16T09:59:24.559187+0000","last_peered":"2023-10-16T09:59:24.559187+0000","last_clean":"2023-10-16T09:59:24.559187+0000","last_became_active":"2023-10-16T09:59:23.565106+0000","last_became_peered":"2023-10-16T09:59:23.565106+0000","last_unstale":"2023-10-16T09:59:24.559187+0000","last_undegraded":"2023-10-16T09:59:24.559187+0000","last_fullsized":"2023-10-16T09:59:24.559187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:58.071562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729622+0000","last_change":"2023-10-16T09:59:23.567454+0000","last_active":"2023-10-16T09:59:24.729622+0000","last_peered":"2023-10-16T09:59:24.729622+0000","last_clean":"2023-10-16T09:59:24.729622+0000","last_became_active":"2023-10-16T09:59:23.567172+0000","last_became_peered":"2023-10-16T09:59:23.567172+0000","last_unstale":"2023-10-16T09:59:24.729622+0000","last_undegraded":"2023-10-16T09:59:24.729622+0000","last_fullsized":"2023-10-16T09:59:24.729622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:27.479652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560413+0000","last_change":"2023-10-16T09:59:23.575949+0000","last_active":"2023-10-16T09:59:24.560413+0000","last_peered":"2023-10-16T09:59:24.560413+0000","last_clean":"2023-10-16T09:59:24.560413+0000","last_became_active":"2023-10-16T09:59:23.575689+0000","last_became_peered":"2023-10-16T09:59:23.575689+0000","last_unstale":"2023-10-16T09:59:24.560413+0000","last_undegraded":"2023-10-16T09:59:24.560413+0000","last_fullsized":"2023-10-16T09:59:24.560413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:04:21.538488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589760+0000","last_change":"2023-10-16T09:59:23.586027+0000","last_active":"2023-10-16T09:59:23.589760+0000","last_peered":"2023-10-16T09:59:23.589760+0000","last_clean":"2023-10-16T09:59:23.589760+0000","last_became_active":"2023-10-16T09:59:23.585027+0000","last_became_peered":"2023-10-16T09:59:23.585027+0000","last_unstale":"2023-10-16T09:59:23.589760+0000","last_undegraded":"2023-10-16T09:59:23.589760+0000","last_fullsized":"2023-10-16T09:59:23.589760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:35.582399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.570466+0000","last_change":"2023-10-16T09:59:23.567329+0000","last_active":"2023-10-16T09:59:23.570466+0000","last_peered":"2023-10-16T09:59:23.570466+0000","last_clean":"2023-10-16T09:59:23.570466+0000","last_became_active":"2023-10-16T09:59:23.567070+0000","last_became_peered":"2023-10-16T09:59:23.567070+0000","last_unstale":"2023-10-16T09:59:23.570466+0000","last_undegraded":"2023-10-16T09:59:23.570466+0000","last_fullsized":"2023-10-16T09:59:23.570466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:04.914215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.568547+0000","last_change":"2023-10-16T09:59:23.568023+0000","last_active":"2023-10-16T09:59:23.568547+0000","last_peered":"2023-10-16T09:59:23.568547+0000","last_clean":"2023-10-16T09:59:23.568547+0000","last_became_active":"2023-10-16T09:59:23.567758+0000","last_became_peered":"2023-10-16T09:59:23.567758+0000","last_unstale":"2023-10-16T09:59:23.568547+0000","last_undegraded":"2023-10-16T09:59:23.568547+0000","last_fullsized":"2023-10-16T09:59:23.568547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:35:55.633336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.749944+0000","last_change":"2023-10-16T09:59:23.577776+0000","last_active":"2023-10-16T09:59:24.749944+0000","last_peered":"2023-10-16T09:59:24.749944+0000","last_clean":"2023-10-16T09:59:24.749944+0000","last_became_active":"2023-10-16T09:59:23.576971+0000","last_became_peered":"2023-10-16T09:59:23.576971+0000","last_unstale":"2023-10-16T09:59:24.749944+0000","last_undegraded":"2023-10-16T09:59:24.749944+0000","last_fullsized":"2023-10-16T09:59:24.749944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:42.588750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729779+0000","last_change":"2023-10-16T09:59:23.572964+0000","last_active":"2023-10-16T09:59:24.729779+0000","last_peered":"2023-10-16T09:59:24.729779+0000","last_clean":"2023-10-16T09:59:24.729779+0000","last_became_active":"2023-10-16T09:59:23.572676+0000","last_became_peered":"2023-10-16T09:59:23.572676+0000","last_unstale":"2023-10-16T09:59:24.729779+0000","last_undegraded":"2023-10-16T09:59:24.729779+0000","last_fullsized":"2023-10-16T09:59:24.729779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:13:09.310141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729631+0000","last_change":"2023-10-16T09:59:23.572333+0000","last_active":"2023-10-16T09:59:24.729631+0000","last_peered":"2023-10-16T09:59:24.729631+0000","last_clean":"2023-10-16T09:59:24.729631+0000","last_became_active":"2023-10-16T09:59:23.571837+0000","last_became_peered":"2023-10-16T09:59:23.571837+0000","last_unstale":"2023-10-16T09:59:24.729631+0000","last_undegraded":"2023-10-16T09:59:24.729631+0000","last_fullsized":"2023-10-16T09:59:24.729631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:24.280819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560499+0000","last_change":"2023-10-16T09:59:23.573333+0000","last_active":"2023-10-16T09:59:24.560499+0000","last_peered":"2023-10-16T09:59:24.560499+0000","last_clean":"2023-10-16T09:59:24.560499+0000","last_became_active":"2023-10-16T09:59:23.573111+0000","last_became_peered":"2023-10-16T09:59:23.573111+0000","last_unstale":"2023-10-16T09:59:24.560499+0000","last_undegraded":"2023-10-16T09:59:24.560499+0000","last_fullsized":"2023-10-16T09:59:24.560499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:47.211413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750217+0000","last_change":"2023-10-16T09:59:23.577980+0000","last_active":"2023-10-16T09:59:24.750217+0000","last_peered":"2023-10-16T09:59:24.750217+0000","last_clean":"2023-10-16T09:59:24.750217+0000","last_became_active":"2023-10-16T09:59:23.577710+0000","last_became_peered":"2023-10-16T09:59:23.577710+0000","last_unstale":"2023-10-16T09:59:24.750217+0000","last_undegraded":"2023-10-16T09:59:24.750217+0000","last_fullsized":"2023-10-16T09:59:24.750217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:12.930818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750107+0000","last_change":"2023-10-16T09:59:23.577698+0000","last_active":"2023-10-16T09:59:24.750107+0000","last_peered":"2023-10-16T09:59:24.750107+0000","last_clean":"2023-10-16T09:59:24.750107+0000","last_became_active":"2023-10-16T09:59:23.577403+0000","last_became_peered":"2023-10-16T09:59:23.577403+0000","last_unstale":"2023-10-16T09:59:24.750107+0000","last_undegraded":"2023-10-16T09:59:24.750107+0000","last_fullsized":"2023-10-16T09:59:24.750107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:51.453447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729675+0000","last_change":"2023-10-16T09:59:23.567578+0000","last_active":"2023-10-16T09:59:24.729675+0000","last_peered":"2023-10-16T09:59:24.729675+0000","last_clean":"2023-10-16T09:59:24.729675+0000","last_became_active":"2023-10-16T09:59:23.566964+0000","last_became_peered":"2023-10-16T09:59:23.566964+0000","last_unstale":"2023-10-16T09:59:24.729675+0000","last_undegraded":"2023-10-16T09:59:24.729675+0000","last_fullsized":"2023-10-16T09:59:24.729675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:24.950903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729710+0000","last_change":"2023-10-16T09:59:23.576194+0000","last_active":"2023-10-16T09:59:24.729710+0000","last_peered":"2023-10-16T09:59:24.729710+0000","last_clean":"2023-10-16T09:59:24.729710+0000","last_became_active":"2023-10-16T09:59:23.575601+0000","last_became_peered":"2023-10-16T09:59:23.575601+0000","last_unstale":"2023-10-16T09:59:24.729710+0000","last_undegraded":"2023-10-16T09:59:24.729710+0000","last_fullsized":"2023-10-16T09:59:24.729710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:26.719461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.575677+0000","last_change":"2023-10-16T09:59:23.574567+0000","last_active":"2023-10-16T09:59:23.575677+0000","last_peered":"2023-10-16T09:59:23.575677+0000","last_clean":"2023-10-16T09:59:23.575677+0000","last_became_active":"2023-10-16T09:59:23.574250+0000","last_became_peered":"2023-10-16T09:59:23.574250+0000","last_unstale":"2023-10-16T09:59:23.575677+0000","last_undegraded":"2023-10-16T09:59:23.575677+0000","last_fullsized":"2023-10-16T09:59:23.575677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:05:23.349438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559115+0000","last_change":"2023-10-16T09:59:23.572899+0000","last_active":"2023-10-16T09:59:24.559115+0000","last_peered":"2023-10-16T09:59:24.559115+0000","last_clean":"2023-10-16T09:59:24.559115+0000","last_became_active":"2023-10-16T09:59:23.572591+0000","last_became_peered":"2023-10-16T09:59:23.572591+0000","last_unstale":"2023-10-16T09:59:24.559115+0000","last_undegraded":"2023-10-16T09:59:24.559115+0000","last_fullsized":"2023-10-16T09:59:24.559115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:29.547767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560720+0000","last_change":"2023-10-16T09:59:23.571659+0000","last_active":"2023-10-16T09:59:24.560720+0000","last_peered":"2023-10-16T09:59:24.560720+0000","last_clean":"2023-10-16T09:59:24.560720+0000","last_became_active":"2023-10-16T09:59:23.571327+0000","last_became_peered":"2023-10-16T09:59:23.571327+0000","last_unstale":"2023-10-16T09:59:24.560720+0000","last_undegraded":"2023-10-16T09:59:24.560720+0000","last_fullsized":"2023-10-16T09:59:24.560720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:13:23.175763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589564+0000","last_change":"2023-10-16T09:59:23.584152+0000","last_active":"2023-10-16T09:59:23.589564+0000","last_peered":"2023-10-16T09:59:23.589564+0000","last_clean":"2023-10-16T09:59:23.589564+0000","last_became_active":"2023-10-16T09:59:23.583828+0000","last_became_peered":"2023-10-16T09:59:23.583828+0000","last_unstale":"2023-10-16T09:59:23.589564+0000","last_undegraded":"2023-10-16T09:59:23.589564+0000","last_fullsized":"2023-10-16T09:59:23.589564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:09.316950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560712+0000","last_change":"2023-10-16T09:59:23.585232+0000","last_active":"2023-10-16T09:59:24.560712+0000","last_peered":"2023-10-16T09:59:24.560712+0000","last_clean":"2023-10-16T09:59:24.560712+0000","last_became_active":"2023-10-16T09:59:23.585000+0000","last_became_peered":"2023-10-16T09:59:23.585000+0000","last_unstale":"2023-10-16T09:59:24.560712+0000","last_undegraded":"2023-10-16T09:59:24.560712+0000","last_fullsized":"2023-10-16T09:59:24.560712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:34.265425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750284+0000","last_change":"2023-10-16T09:59:23.570168+0000","last_active":"2023-10-16T09:59:24.750284+0000","last_peered":"2023-10-16T09:59:24.750284+0000","last_clean":"2023-10-16T09:59:24.750284+0000","last_became_active":"2023-10-16T09:59:23.569825+0000","last_became_peered":"2023-10-16T09:59:23.569825+0000","last_unstale":"2023-10-16T09:59:24.750284+0000","last_undegraded":"2023-10-16T09:59:24.750284+0000","last_fullsized":"2023-10-16T09:59:24.750284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:16:24.387486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560620+0000","last_change":"2023-10-16T09:59:23.572664+0000","last_active":"2023-10-16T09:59:24.560620+0000","last_peered":"2023-10-16T09:59:24.560620+0000","last_clean":"2023-10-16T09:59:24.560620+0000","last_became_active":"2023-10-16T09:59:23.572406+0000","last_became_peered":"2023-10-16T09:59:23.572406+0000","last_unstale":"2023-10-16T09:59:24.560620+0000","last_undegraded":"2023-10-16T09:59:24.560620+0000","last_fullsized":"2023-10-16T09:59:24.560620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:57:06.869600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560797+0000","last_change":"2023-10-16T09:59:23.572835+0000","last_active":"2023-10-16T09:59:24.560797+0000","last_peered":"2023-10-16T09:59:24.560797+0000","last_clean":"2023-10-16T09:59:24.560797+0000","last_became_active":"2023-10-16T09:59:23.572468+0000","last_became_peered":"2023-10-16T09:59:23.572468+0000","last_unstale":"2023-10-16T09:59:24.560797+0000","last_undegraded":"2023-10-16T09:59:24.560797+0000","last_fullsized":"2023-10-16T09:59:24.560797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:03.011626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729208+0000","last_change":"2023-10-16T09:59:23.569594+0000","last_active":"2023-10-16T09:59:24.729208+0000","last_peered":"2023-10-16T09:59:24.729208+0000","last_clean":"2023-10-16T09:59:24.729208+0000","last_became_active":"2023-10-16T09:59:23.568421+0000","last_became_peered":"2023-10-16T09:59:23.568421+0000","last_unstale":"2023-10-16T09:59:24.729208+0000","last_undegraded":"2023-10-16T09:59:24.729208+0000","last_fullsized":"2023-10-16T09:59:24.729208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:23.959799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750432+0000","last_change":"2023-10-16T09:59:23.572894+0000","last_active":"2023-10-16T09:59:24.750432+0000","last_peered":"2023-10-16T09:59:24.750432+0000","last_clean":"2023-10-16T09:59:24.750432+0000","last_became_active":"2023-10-16T09:59:23.572623+0000","last_became_peered":"2023-10-16T09:59:23.572623+0000","last_unstale":"2023-10-16T09:59:24.750432+0000","last_undegraded":"2023-10-16T09:59:24.750432+0000","last_fullsized":"2023-10-16T09:59:24.750432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:52.467502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750261+0000","last_change":"2023-10-16T09:59:23.573382+0000","last_active":"2023-10-16T09:59:24.750261+0000","last_peered":"2023-10-16T09:59:24.750261+0000","last_clean":"2023-10-16T09:59:24.750261+0000","last_became_active":"2023-10-16T09:59:23.570211+0000","last_became_peered":"2023-10-16T09:59:23.570211+0000","last_unstale":"2023-10-16T09:59:24.750261+0000","last_undegraded":"2023-10-16T09:59:24.750261+0000","last_fullsized":"2023-10-16T09:59:24.750261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:43.411176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560531+0000","last_change":"2023-10-16T09:59:23.572236+0000","last_active":"2023-10-16T09:59:24.560531+0000","last_peered":"2023-10-16T09:59:24.560531+0000","last_clean":"2023-10-16T09:59:24.560531+0000","last_became_active":"2023-10-16T09:59:23.571337+0000","last_became_peered":"2023-10-16T09:59:23.571337+0000","last_unstale":"2023-10-16T09:59:24.560531+0000","last_undegraded":"2023-10-16T09:59:24.560531+0000","last_fullsized":"2023-10-16T09:59:24.560531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:45.997463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559107+0000","last_change":"2023-10-16T09:59:23.584795+0000","last_active":"2023-10-16T09:59:24.559107+0000","last_peered":"2023-10-16T09:59:24.559107+0000","last_clean":"2023-10-16T09:59:24.559107+0000","last_became_active":"2023-10-16T09:59:23.584540+0000","last_became_peered":"2023-10-16T09:59:23.584540+0000","last_unstale":"2023-10-16T09:59:24.559107+0000","last_undegraded":"2023-10-16T09:59:24.559107+0000","last_fullsized":"2023-10-16T09:59:24.559107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:30.495916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729605+0000","last_change":"2023-10-16T09:59:23.572168+0000","last_active":"2023-10-16T09:59:24.729605+0000","last_peered":"2023-10-16T09:59:24.729605+0000","last_clean":"2023-10-16T09:59:24.729605+0000","last_became_active":"2023-10-16T09:59:23.571885+0000","last_became_peered":"2023-10-16T09:59:23.571885+0000","last_unstale":"2023-10-16T09:59:24.729605+0000","last_undegraded":"2023-10-16T09:59:24.729605+0000","last_fullsized":"2023-10-16T09:59:24.729605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:24.234610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750057+0000","last_change":"2023-10-16T09:59:23.578383+0000","last_active":"2023-10-16T09:59:24.750057+0000","last_peered":"2023-10-16T09:59:24.750057+0000","last_clean":"2023-10-16T09:59:24.750057+0000","last_became_active":"2023-10-16T09:59:23.578073+0000","last_became_peered":"2023-10-16T09:59:23.578073+0000","last_unstale":"2023-10-16T09:59:24.750057+0000","last_undegraded":"2023-10-16T09:59:24.750057+0000","last_fullsized":"2023-10-16T09:59:24.750057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:26.719461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560090+0000","last_change":"2023-10-16T09:59:23.570768+0000","last_active":"2023-10-16T09:59:24.560090+0000","last_peered":"2023-10-16T09:59:24.560090+0000","last_clean":"2023-10-16T09:59:24.560090+0000","last_became_active":"2023-10-16T09:59:23.570004+0000","last_became_peered":"2023-10-16T09:59:23.570004+0000","last_unstale":"2023-10-16T09:59:24.560090+0000","last_undegraded":"2023-10-16T09:59:24.560090+0000","last_fullsized":"2023-10-16T09:59:24.560090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:31.836130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.570515+0000","last_change":"2023-10-16T09:59:23.567140+0000","last_active":"2023-10-16T09:59:23.570515+0000","last_peered":"2023-10-16T09:59:23.570515+0000","last_clean":"2023-10-16T09:59:23.570515+0000","last_became_active":"2023-10-16T09:59:23.566912+0000","last_became_peered":"2023-10-16T09:59:23.566912+0000","last_unstale":"2023-10-16T09:59:23.570515+0000","last_undegraded":"2023-10-16T09:59:23.570515+0000","last_fullsized":"2023-10-16T09:59:23.570515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:01.740391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589620+0000","last_change":"2023-10-16T09:59:23.584662+0000","last_active":"2023-10-16T09:59:23.589620+0000","last_peered":"2023-10-16T09:59:23.589620+0000","last_clean":"2023-10-16T09:59:23.589620+0000","last_became_active":"2023-10-16T09:59:23.584353+0000","last_became_peered":"2023-10-16T09:59:23.584353+0000","last_unstale":"2023-10-16T09:59:23.589620+0000","last_undegraded":"2023-10-16T09:59:23.589620+0000","last_fullsized":"2023-10-16T09:59:23.589620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:03.715288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559495+0000","last_change":"2023-10-16T09:59:23.135009+0000","last_active":"2023-10-16T09:59:24.559495+0000","last_peered":"2023-10-16T09:59:24.559495+0000","last_clean":"2023-10-16T09:59:24.559495+0000","last_became_active":"2023-10-16T09:59:23.134763+0000","last_became_peered":"2023-10-16T09:59:23.134763+0000","last_unstale":"2023-10-16T09:59:24.559495+0000","last_undegraded":"2023-10-16T09:59:24.559495+0000","last_fullsized":"2023-10-16T09:59:24.559495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:48.255262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560657+0000","last_change":"2023-10-16T09:59:23.584704+0000","last_active":"2023-10-16T09:59:24.560657+0000","last_peered":"2023-10-16T09:59:24.560657+0000","last_clean":"2023-10-16T09:59:24.560657+0000","last_became_active":"2023-10-16T09:59:23.584416+0000","last_became_peered":"2023-10-16T09:59:23.584416+0000","last_unstale":"2023-10-16T09:59:24.560657+0000","last_undegraded":"2023-10-16T09:59:24.560657+0000","last_fullsized":"2023-10-16T09:59:24.560657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:43.443423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.136189+0000","last_change":"2023-10-16T09:59:23.133680+0000","last_active":"2023-10-16T09:59:24.136189+0000","last_peered":"2023-10-16T09:59:24.136189+0000","last_clean":"2023-10-16T09:59:24.136189+0000","last_became_active":"2023-10-16T09:59:23.133385+0000","last_became_peered":"2023-10-16T09:59:23.133385+0000","last_unstale":"2023-10-16T09:59:24.136189+0000","last_undegraded":"2023-10-16T09:59:24.136189+0000","last_fullsized":"2023-10-16T09:59:24.136189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:03.999236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.565879+0000","last_change":"2023-10-16T09:59:23.565182+0000","last_active":"2023-10-16T09:59:23.565879+0000","last_peered":"2023-10-16T09:59:23.565879+0000","last_clean":"2023-10-16T09:59:23.565879+0000","last_became_active":"2023-10-16T09:59:23.564915+0000","last_became_peered":"2023-10-16T09:59:23.564915+0000","last_unstale":"2023-10-16T09:59:23.565879+0000","last_undegraded":"2023-10-16T09:59:23.565879+0000","last_fullsized":"2023-10-16T09:59:23.565879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:23.856491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.134901+0000","last_change":"2023-10-16T09:59:23.139099+0000","last_active":"2023-10-16T09:59:24.134901+0000","last_peered":"2023-10-16T09:59:24.134901+0000","last_clean":"2023-10-16T09:59:24.134901+0000","last_became_active":"2023-10-16T09:59:23.138749+0000","last_became_peered":"2023-10-16T09:59:23.138749+0000","last_unstale":"2023-10-16T09:59:24.134901+0000","last_undegraded":"2023-10-16T09:59:24.134901+0000","last_fullsized":"2023-10-16T09:59:24.134901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:31.164001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729796+0000","last_change":"2023-10-16T09:59:23.569631+0000","last_active":"2023-10-16T09:59:24.729796+0000","last_peered":"2023-10-16T09:59:24.729796+0000","last_clean":"2023-10-16T09:59:24.729796+0000","last_became_active":"2023-10-16T09:59:23.569333+0000","last_became_peered":"2023-10-16T09:59:23.569333+0000","last_unstale":"2023-10-16T09:59:24.729796+0000","last_undegraded":"2023-10-16T09:59:24.729796+0000","last_fullsized":"2023-10-16T09:59:24.729796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:51.453447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.134161+0000","last_change":"2023-10-16T09:59:23.135150+0000","last_active":"2023-10-16T09:59:24.134161+0000","last_peered":"2023-10-16T09:59:24.134161+0000","last_clean":"2023-10-16T09:59:24.134161+0000","last_became_active":"2023-10-16T09:59:23.134903+0000","last_became_peered":"2023-10-16T09:59:23.134903+0000","last_unstale":"2023-10-16T09:59:24.134161+0000","last_undegraded":"2023-10-16T09:59:24.134161+0000","last_fullsized":"2023-10-16T09:59:24.134161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:25:32.755927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.569987+0000","last_change":"2023-10-16T09:59:23.569359+0000","last_active":"2023-10-16T09:59:23.569987+0000","last_peered":"2023-10-16T09:59:23.569987+0000","last_clean":"2023-10-16T09:59:23.569987+0000","last_became_active":"2023-10-16T09:59:23.569068+0000","last_became_peered":"2023-10-16T09:59:23.569068+0000","last_unstale":"2023-10-16T09:59:23.569987+0000","last_undegraded":"2023-10-16T09:59:23.569987+0000","last_fullsized":"2023-10-16T09:59:23.569987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:14.142714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560772+0000","last_change":"2023-10-16T09:59:23.140640+0000","last_active":"2023-10-16T09:59:24.560772+0000","last_peered":"2023-10-16T09:59:24.560772+0000","last_clean":"2023-10-16T09:59:24.560772+0000","last_became_active":"2023-10-16T09:59:23.140369+0000","last_became_peered":"2023-10-16T09:59:23.140369+0000","last_unstale":"2023-10-16T09:59:24.560772+0000","last_undegraded":"2023-10-16T09:59:24.560772+0000","last_fullsized":"2023-10-16T09:59:24.560772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:25.710865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559215+0000","last_change":"2023-10-16T09:59:23.571277+0000","last_active":"2023-10-16T09:59:24.559215+0000","last_peered":"2023-10-16T09:59:24.559215+0000","last_clean":"2023-10-16T09:59:24.559215+0000","last_became_active":"2023-10-16T09:59:23.570919+0000","last_became_peered":"2023-10-16T09:59:23.570919+0000","last_unstale":"2023-10-16T09:59:24.559215+0000","last_undegraded":"2023-10-16T09:59:24.559215+0000","last_fullsized":"2023-10-16T09:59:24.559215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:31.483542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560564+0000","last_change":"2023-10-16T09:59:23.143003+0000","last_active":"2023-10-16T09:59:24.560564+0000","last_peered":"2023-10-16T09:59:24.560564+0000","last_clean":"2023-10-16T09:59:24.560564+0000","last_became_active":"2023-10-16T09:59:23.142744+0000","last_became_peered":"2023-10-16T09:59:23.142744+0000","last_unstale":"2023-10-16T09:59:24.560564+0000","last_undegraded":"2023-10-16T09:59:24.560564+0000","last_fullsized":"2023-10-16T09:59:24.560564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:42.402581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.586758+0000","last_change":"2023-10-16T09:59:23.583814+0000","last_active":"2023-10-16T09:59:23.586758+0000","last_peered":"2023-10-16T09:59:23.586758+0000","last_clean":"2023-10-16T09:59:23.586758+0000","last_became_active":"2023-10-16T09:59:23.583440+0000","last_became_peered":"2023-10-16T09:59:23.583440+0000","last_unstale":"2023-10-16T09:59:23.586758+0000","last_undegraded":"2023-10-16T09:59:23.586758+0000","last_fullsized":"2023-10-16T09:59:23.586758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:50.691937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560593+0000","last_change":"2023-10-16T09:59:23.147098+0000","last_active":"2023-10-16T09:59:24.560593+0000","last_peered":"2023-10-16T09:59:24.560593+0000","last_clean":"2023-10-16T09:59:24.560593+0000","last_became_active":"2023-10-16T09:59:23.146814+0000","last_became_peered":"2023-10-16T09:59:23.146814+0000","last_unstale":"2023-10-16T09:59:24.560593+0000","last_undegraded":"2023-10-16T09:59:24.560593+0000","last_fullsized":"2023-10-16T09:59:24.560593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:23.272223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560715+0000","last_change":"2023-10-16T09:59:23.576763+0000","last_active":"2023-10-16T09:59:24.560715+0000","last_peered":"2023-10-16T09:59:24.560715+0000","last_clean":"2023-10-16T09:59:24.560715+0000","last_became_active":"2023-10-16T09:59:23.576408+0000","last_became_peered":"2023-10-16T09:59:23.576408+0000","last_unstale":"2023-10-16T09:59:24.560715+0000","last_undegraded":"2023-10-16T09:59:24.560715+0000","last_fullsized":"2023-10-16T09:59:24.560715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:37.036382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.569229+0000","last_change":"2023-10-16T09:59:23.147076+0000","last_active":"2023-10-16T09:59:23.569229+0000","last_peered":"2023-10-16T09:59:23.569229+0000","last_clean":"2023-10-16T09:59:23.569229+0000","last_became_active":"2023-10-16T09:59:23.146614+0000","last_became_peered":"2023-10-16T09:59:23.146614+0000","last_unstale":"2023-10-16T09:59:23.569229+0000","last_undegraded":"2023-10-16T09:59:23.569229+0000","last_fullsized":"2023-10-16T09:59:23.569229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:21.366241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559145+0000","last_change":"2023-10-16T09:59:23.563237+0000","last_active":"2023-10-16T09:59:24.559145+0000","last_peered":"2023-10-16T09:59:24.559145+0000","last_clean":"2023-10-16T09:59:24.559145+0000","last_became_active":"2023-10-16T09:59:23.562968+0000","last_became_peered":"2023-10-16T09:59:23.562968+0000","last_unstale":"2023-10-16T09:59:24.559145+0000","last_undegraded":"2023-10-16T09:59:24.559145+0000","last_fullsized":"2023-10-16T09:59:24.559145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:54:54.741956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729836+0000","last_change":"2023-10-16T09:59:23.133175+0000","last_active":"2023-10-16T09:59:24.729836+0000","last_peered":"2023-10-16T09:59:24.729836+0000","last_clean":"2023-10-16T09:59:24.729836+0000","last_became_active":"2023-10-16T09:59:23.132856+0000","last_became_peered":"2023-10-16T09:59:23.132856+0000","last_unstale":"2023-10-16T09:59:24.729836+0000","last_undegraded":"2023-10-16T09:59:24.729836+0000","last_fullsized":"2023-10-16T09:59:24.729836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:10:35.848074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560712+0000","last_change":"2023-10-16T09:59:23.571587+0000","last_active":"2023-10-16T09:59:24.560712+0000","last_peered":"2023-10-16T09:59:24.560712+0000","last_clean":"2023-10-16T09:59:24.560712+0000","last_became_active":"2023-10-16T09:59:23.571295+0000","last_became_peered":"2023-10-16T09:59:23.571295+0000","last_unstale":"2023-10-16T09:59:24.560712+0000","last_undegraded":"2023-10-16T09:59:24.560712+0000","last_fullsized":"2023-10-16T09:59:24.560712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:17.517871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560751+0000","last_change":"2023-10-16T09:59:23.136723+0000","last_active":"2023-10-16T09:59:24.560751+0000","last_peered":"2023-10-16T09:59:24.560751+0000","last_clean":"2023-10-16T09:59:24.560751+0000","last_became_active":"2023-10-16T09:59:23.136416+0000","last_became_peered":"2023-10-16T09:59:23.136416+0000","last_unstale":"2023-10-16T09:59:24.560751+0000","last_undegraded":"2023-10-16T09:59:24.560751+0000","last_fullsized":"2023-10-16T09:59:24.560751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:19.894024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589632+0000","last_change":"2023-10-16T09:59:23.583632+0000","last_active":"2023-10-16T09:59:23.589632+0000","last_peered":"2023-10-16T09:59:23.589632+0000","last_clean":"2023-10-16T09:59:23.589632+0000","last_became_active":"2023-10-16T09:59:23.583332+0000","last_became_peered":"2023-10-16T09:59:23.583332+0000","last_unstale":"2023-10-16T09:59:23.589632+0000","last_undegraded":"2023-10-16T09:59:23.589632+0000","last_fullsized":"2023-10-16T09:59:23.589632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:12.826867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729453+0000","last_change":"2023-10-16T09:59:23.132607+0000","last_active":"2023-10-16T09:59:24.729453+0000","last_peered":"2023-10-16T09:59:24.729453+0000","last_clean":"2023-10-16T09:59:24.729453+0000","last_became_active":"2023-10-16T09:59:23.132372+0000","last_became_peered":"2023-10-16T09:59:23.132372+0000","last_unstale":"2023-10-16T09:59:24.729453+0000","last_undegraded":"2023-10-16T09:59:24.729453+0000","last_fullsized":"2023-10-16T09:59:24.729453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:00.731796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560789+0000","last_change":"2023-10-16T09:59:23.571932+0000","last_active":"2023-10-16T09:59:24.560789+0000","last_peered":"2023-10-16T09:59:24.560789+0000","last_clean":"2023-10-16T09:59:24.560789+0000","last_became_active":"2023-10-16T09:59:23.571636+0000","last_became_peered":"2023-10-16T09:59:23.571636+0000","last_unstale":"2023-10-16T09:59:24.560789+0000","last_undegraded":"2023-10-16T09:59:24.560789+0000","last_fullsized":"2023-10-16T09:59:24.560789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:08.278546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559240+0000","last_change":"2023-10-16T09:59:23.140254+0000","last_active":"2023-10-16T09:59:24.559240+0000","last_peered":"2023-10-16T09:59:24.559240+0000","last_clean":"2023-10-16T09:59:24.559240+0000","last_became_active":"2023-10-16T09:59:23.139983+0000","last_became_peered":"2023-10-16T09:59:23.139983+0000","last_unstale":"2023-10-16T09:59:24.559240+0000","last_undegraded":"2023-10-16T09:59:24.559240+0000","last_fullsized":"2023-10-16T09:59:24.559240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:12.787645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560050+0000","last_change":"2023-10-16T09:59:23.579928+0000","last_active":"2023-10-16T09:59:24.560050+0000","last_peered":"2023-10-16T09:59:24.560050+0000","last_clean":"2023-10-16T09:59:24.560050+0000","last_became_active":"2023-10-16T09:59:23.579653+0000","last_became_peered":"2023-10-16T09:59:23.579653+0000","last_unstale":"2023-10-16T09:59:24.560050+0000","last_undegraded":"2023-10-16T09:59:24.560050+0000","last_fullsized":"2023-10-16T09:59:24.560050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:30.854975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729488+0000","last_change":"2023-10-16T09:59:23.147274+0000","last_active":"2023-10-16T09:59:24.729488+0000","last_peered":"2023-10-16T09:59:24.729488+0000","last_clean":"2023-10-16T09:59:24.729488+0000","last_became_active":"2023-10-16T09:59:23.147022+0000","last_became_peered":"2023-10-16T09:59:23.147022+0000","last_unstale":"2023-10-16T09:59:24.729488+0000","last_undegraded":"2023-10-16T09:59:24.729488+0000","last_fullsized":"2023-10-16T09:59:24.729488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:16.095624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.749957+0000","last_change":"2023-10-16T09:59:23.573241+0000","last_active":"2023-10-16T09:59:24.749957+0000","last_peered":"2023-10-16T09:59:24.749957+0000","last_clean":"2023-10-16T09:59:24.749957+0000","last_became_active":"2023-10-16T09:59:23.572426+0000","last_became_peered":"2023-10-16T09:59:23.572426+0000","last_unstale":"2023-10-16T09:59:24.749957+0000","last_undegraded":"2023-10-16T09:59:24.749957+0000","last_fullsized":"2023-10-16T09:59:24.749957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:24.950903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.749981+0000","last_change":"2023-10-16T09:59:23.140753+0000","last_active":"2023-10-16T09:59:24.749981+0000","last_peered":"2023-10-16T09:59:24.749981+0000","last_clean":"2023-10-16T09:59:24.749981+0000","last_became_active":"2023-10-16T09:59:23.140018+0000","last_became_peered":"2023-10-16T09:59:23.140018+0000","last_unstale":"2023-10-16T09:59:24.749981+0000","last_undegraded":"2023-10-16T09:59:24.749981+0000","last_fullsized":"2023-10-16T09:59:24.749981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:16.672596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.584766+0000","last_change":"2023-10-16T09:59:23.579552+0000","last_active":"2023-10-16T09:59:23.584766+0000","last_peered":"2023-10-16T09:59:23.584766+0000","last_clean":"2023-10-16T09:59:23.584766+0000","last_became_active":"2023-10-16T09:59:23.579090+0000","last_became_peered":"2023-10-16T09:59:23.579090+0000","last_unstale":"2023-10-16T09:59:23.584766+0000","last_undegraded":"2023-10-16T09:59:23.584766+0000","last_fullsized":"2023-10-16T09:59:23.584766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:14.367906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729397+0000","last_change":"2023-10-16T09:59:23.138860+0000","last_active":"2023-10-16T09:59:24.729397+0000","last_peered":"2023-10-16T09:59:24.729397+0000","last_clean":"2023-10-16T09:59:24.729397+0000","last_became_active":"2023-10-16T09:59:23.138509+0000","last_became_peered":"2023-10-16T09:59:23.138509+0000","last_unstale":"2023-10-16T09:59:24.729397+0000","last_undegraded":"2023-10-16T09:59:24.729397+0000","last_fullsized":"2023-10-16T09:59:24.729397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:19.604724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559270+0000","last_change":"2023-10-16T09:59:23.567670+0000","last_active":"2023-10-16T09:59:24.559270+0000","last_peered":"2023-10-16T09:59:24.559270+0000","last_clean":"2023-10-16T09:59:24.559270+0000","last_became_active":"2023-10-16T09:59:23.567413+0000","last_became_peered":"2023-10-16T09:59:23.567413+0000","last_unstale":"2023-10-16T09:59:24.559270+0000","last_undegraded":"2023-10-16T09:59:24.559270+0000","last_fullsized":"2023-10-16T09:59:24.559270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:28.806332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559996+0000","last_change":"2023-10-16T09:59:23.139821+0000","last_active":"2023-10-16T09:59:24.559996+0000","last_peered":"2023-10-16T09:59:24.559996+0000","last_clean":"2023-10-16T09:59:24.559996+0000","last_became_active":"2023-10-16T09:59:23.139564+0000","last_became_peered":"2023-10-16T09:59:23.139564+0000","last_unstale":"2023-10-16T09:59:24.559996+0000","last_undegraded":"2023-10-16T09:59:24.559996+0000","last_fullsized":"2023-10-16T09:59:24.559996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:47.749148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729661+0000","last_change":"2023-10-16T09:59:23.572841+0000","last_active":"2023-10-16T09:59:24.729661+0000","last_peered":"2023-10-16T09:59:24.729661+0000","last_clean":"2023-10-16T09:59:24.729661+0000","last_became_active":"2023-10-16T09:59:23.572527+0000","last_became_peered":"2023-10-16T09:59:23.572527+0000","last_unstale":"2023-10-16T09:59:24.729661+0000","last_undegraded":"2023-10-16T09:59:24.729661+0000","last_fullsized":"2023-10-16T09:59:24.729661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:42.588750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750449+0000","last_change":"2023-10-16T09:59:23.140854+0000","last_active":"2023-10-16T09:59:24.750449+0000","last_peered":"2023-10-16T09:59:24.750449+0000","last_clean":"2023-10-16T09:59:24.750449+0000","last_became_active":"2023-10-16T09:59:23.140537+0000","last_became_peered":"2023-10-16T09:59:23.140537+0000","last_unstale":"2023-10-16T09:59:24.750449+0000","last_undegraded":"2023-10-16T09:59:24.750449+0000","last_fullsized":"2023-10-16T09:59:24.750449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:23.226014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.586784+0000","last_change":"2023-10-16T09:59:23.583528+0000","last_active":"2023-10-16T09:59:23.586784+0000","last_peered":"2023-10-16T09:59:23.586784+0000","last_clean":"2023-10-16T09:59:23.586784+0000","last_became_active":"2023-10-16T09:59:23.583230+0000","last_became_peered":"2023-10-16T09:59:23.583230+0000","last_unstale":"2023-10-16T09:59:23.586784+0000","last_undegraded":"2023-10-16T09:59:23.586784+0000","last_fullsized":"2023-10-16T09:59:23.586784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:07.770406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729644+0000","last_change":"2023-10-16T09:59:23.142607+0000","last_active":"2023-10-16T09:59:24.729644+0000","last_peered":"2023-10-16T09:59:24.729644+0000","last_clean":"2023-10-16T09:59:24.729644+0000","last_became_active":"2023-10-16T09:59:23.142271+0000","last_became_peered":"2023-10-16T09:59:23.142271+0000","last_unstale":"2023-10-16T09:59:24.729644+0000","last_undegraded":"2023-10-16T09:59:24.729644+0000","last_fullsized":"2023-10-16T09:59:24.729644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:57.330886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560571+0000","last_change":"2023-10-16T09:59:23.586363+0000","last_active":"2023-10-16T09:59:24.560571+0000","last_peered":"2023-10-16T09:59:24.560571+0000","last_clean":"2023-10-16T09:59:24.560571+0000","last_became_active":"2023-10-16T09:59:23.586115+0000","last_became_peered":"2023-10-16T09:59:23.586115+0000","last_unstale":"2023-10-16T09:59:24.560571+0000","last_undegraded":"2023-10-16T09:59:24.560571+0000","last_fullsized":"2023-10-16T09:59:24.560571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:11.304253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729743+0000","last_change":"2023-10-16T09:59:23.132575+0000","last_active":"2023-10-16T09:59:24.729743+0000","last_peered":"2023-10-16T09:59:24.729743+0000","last_clean":"2023-10-16T09:59:24.729743+0000","last_became_active":"2023-10-16T09:59:23.132043+0000","last_became_peered":"2023-10-16T09:59:23.132043+0000","last_unstale":"2023-10-16T09:59:24.729743+0000","last_undegraded":"2023-10-16T09:59:24.729743+0000","last_fullsized":"2023-10-16T09:59:24.729743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:20.065856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589623+0000","last_change":"2023-10-16T09:59:23.584736+0000","last_active":"2023-10-16T09:59:23.589623+0000","last_peered":"2023-10-16T09:59:23.589623+0000","last_clean":"2023-10-16T09:59:23.589623+0000","last_became_active":"2023-10-16T09:59:23.584473+0000","last_became_peered":"2023-10-16T09:59:23.584473+0000","last_unstale":"2023-10-16T09:59:23.589623+0000","last_undegraded":"2023-10-16T09:59:23.589623+0000","last_fullsized":"2023-10-16T09:59:23.589623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:20.902619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560818+0000","last_change":"2023-10-16T09:59:23.139799+0000","last_active":"2023-10-16T09:59:24.560818+0000","last_peered":"2023-10-16T09:59:24.560818+0000","last_clean":"2023-10-16T09:59:24.560818+0000","last_became_active":"2023-10-16T09:59:23.139533+0000","last_became_peered":"2023-10-16T09:59:23.139533+0000","last_unstale":"2023-10-16T09:59:24.560818+0000","last_undegraded":"2023-10-16T09:59:24.560818+0000","last_fullsized":"2023-10-16T09:59:24.560818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:13:08.301545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.565655+0000","last_change":"2023-10-16T09:59:23.565028+0000","last_active":"2023-10-16T09:59:23.565655+0000","last_peered":"2023-10-16T09:59:23.565655+0000","last_clean":"2023-10-16T09:59:23.565655+0000","last_became_active":"2023-10-16T09:59:23.563007+0000","last_became_peered":"2023-10-16T09:59:23.563007+0000","last_unstale":"2023-10-16T09:59:23.565655+0000","last_undegraded":"2023-10-16T09:59:23.565655+0000","last_fullsized":"2023-10-16T09:59:23.565655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:57.217307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.136027+0000","last_change":"2023-10-16T09:59:23.147188+0000","last_active":"2023-10-16T09:59:24.136027+0000","last_peered":"2023-10-16T09:59:24.136027+0000","last_clean":"2023-10-16T09:59:24.136027+0000","last_became_active":"2023-10-16T09:59:23.146915+0000","last_became_peered":"2023-10-16T09:59:23.146915+0000","last_unstale":"2023-10-16T09:59:24.136027+0000","last_undegraded":"2023-10-16T09:59:24.136027+0000","last_fullsized":"2023-10-16T09:59:24.136027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:39.944738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750070+0000","last_change":"2023-10-16T09:59:23.570277+0000","last_active":"2023-10-16T09:59:24.750070+0000","last_peered":"2023-10-16T09:59:24.750070+0000","last_clean":"2023-10-16T09:59:24.750070+0000","last_became_active":"2023-10-16T09:59:23.569934+0000","last_became_peered":"2023-10-16T09:59:23.569934+0000","last_unstale":"2023-10-16T09:59:24.750070+0000","last_undegraded":"2023-10-16T09:59:24.750070+0000","last_fullsized":"2023-10-16T09:59:24.750070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:13:09.310141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750104+0000","last_change":"2023-10-16T09:59:23.140960+0000","last_active":"2023-10-16T09:59:24.750104+0000","last_peered":"2023-10-16T09:59:24.750104+0000","last_clean":"2023-10-16T09:59:24.750104+0000","last_became_active":"2023-10-16T09:59:23.140435+0000","last_became_peered":"2023-10-16T09:59:23.140435+0000","last_unstale":"2023-10-16T09:59:24.750104+0000","last_undegraded":"2023-10-16T09:59:24.750104+0000","last_fullsized":"2023-10-16T09:59:24.750104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:10:35.848074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589611+0000","last_change":"2023-10-16T09:59:23.584676+0000","last_active":"2023-10-16T09:59:23.589611+0000","last_peered":"2023-10-16T09:59:23.589611+0000","last_clean":"2023-10-16T09:59:23.589611+0000","last_became_active":"2023-10-16T09:59:23.584395+0000","last_became_peered":"2023-10-16T09:59:23.584395+0000","last_unstale":"2023-10-16T09:59:23.589611+0000","last_undegraded":"2023-10-16T09:59:23.589611+0000","last_fullsized":"2023-10-16T09:59:23.589611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:54.878354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.559234+0000","last_change":"2023-10-16T09:59:23.133674+0000","last_active":"2023-10-16T09:59:23.559234+0000","last_peered":"2023-10-16T09:59:23.559234+0000","last_clean":"2023-10-16T09:59:23.559234+0000","last_became_active":"2023-10-16T09:59:23.133407+0000","last_became_peered":"2023-10-16T09:59:23.133407+0000","last_unstale":"2023-10-16T09:59:23.559234+0000","last_undegraded":"2023-10-16T09:59:23.559234+0000","last_fullsized":"2023-10-16T09:59:23.559234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:55.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.574777+0000","last_change":"2023-10-16T09:59:23.572823+0000","last_active":"2023-10-16T09:59:23.574777+0000","last_peered":"2023-10-16T09:59:23.574777+0000","last_clean":"2023-10-16T09:59:23.574777+0000","last_became_active":"2023-10-16T09:59:23.572568+0000","last_became_peered":"2023-10-16T09:59:23.572568+0000","last_unstale":"2023-10-16T09:59:23.574777+0000","last_undegraded":"2023-10-16T09:59:23.574777+0000","last_fullsized":"2023-10-16T09:59:23.574777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:57.141048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750499+0000","last_change":"2023-10-16T09:59:23.140261+0000","last_active":"2023-10-16T09:59:24.750499+0000","last_peered":"2023-10-16T09:59:24.750499+0000","last_clean":"2023-10-16T09:59:24.750499+0000","last_became_active":"2023-10-16T09:59:23.139672+0000","last_became_peered":"2023-10-16T09:59:23.139672+0000","last_unstale":"2023-10-16T09:59:24.750499+0000","last_undegraded":"2023-10-16T09:59:24.750499+0000","last_fullsized":"2023-10-16T09:59:24.750499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:23.272223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.567451+0000","last_change":"2023-10-16T09:59:23.566239+0000","last_active":"2023-10-16T09:59:23.567451+0000","last_peered":"2023-10-16T09:59:23.567451+0000","last_clean":"2023-10-16T09:59:23.567451+0000","last_became_active":"2023-10-16T09:59:23.565855+0000","last_became_peered":"2023-10-16T09:59:23.565855+0000","last_unstale":"2023-10-16T09:59:23.567451+0000","last_undegraded":"2023-10-16T09:59:23.567451+0000","last_fullsized":"2023-10-16T09:59:23.567451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:22.591205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750378+0000","last_change":"2023-10-16T09:59:23.141475+0000","last_active":"2023-10-16T09:59:24.750378+0000","last_peered":"2023-10-16T09:59:24.750378+0000","last_clean":"2023-10-16T09:59:24.750378+0000","last_became_active":"2023-10-16T09:59:23.141196+0000","last_became_peered":"2023-10-16T09:59:23.141196+0000","last_unstale":"2023-10-16T09:59:24.750378+0000","last_undegraded":"2023-10-16T09:59:24.750378+0000","last_fullsized":"2023-10-16T09:59:24.750378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:22.951203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.750428+0000","last_change":"2023-10-16T09:59:23.573584+0000","last_active":"2023-10-16T09:59:24.750428+0000","last_peered":"2023-10-16T09:59:24.750428+0000","last_clean":"2023-10-16T09:59:24.750428+0000","last_became_active":"2023-10-16T09:59:23.573084+0000","last_became_peered":"2023-10-16T09:59:23.573084+0000","last_unstale":"2023-10-16T09:59:24.750428+0000","last_undegraded":"2023-10-16T09:59:24.750428+0000","last_fullsized":"2023-10-16T09:59:24.750428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:30.767880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.559608+0000","last_change":"2023-10-16T09:59:23.138796+0000","last_active":"2023-10-16T09:59:23.559608+0000","last_peered":"2023-10-16T09:59:23.559608+0000","last_clean":"2023-10-16T09:59:23.559608+0000","last_became_active":"2023-10-16T09:59:23.138559+0000","last_became_peered":"2023-10-16T09:59:23.138559+0000","last_unstale":"2023-10-16T09:59:23.559608+0000","last_undegraded":"2023-10-16T09:59:23.559608+0000","last_fullsized":"2023-10-16T09:59:23.559608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:58.209416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589709+0000","last_change":"2023-10-16T09:59:23.585296+0000","last_active":"2023-10-16T09:59:23.589709+0000","last_peered":"2023-10-16T09:59:23.589709+0000","last_clean":"2023-10-16T09:59:23.589709+0000","last_became_active":"2023-10-16T09:59:23.584241+0000","last_became_peered":"2023-10-16T09:59:23.584241+0000","last_unstale":"2023-10-16T09:59:23.589709+0000","last_undegraded":"2023-10-16T09:59:23.589709+0000","last_fullsized":"2023-10-16T09:59:23.589709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:09.755265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559219+0000","last_change":"2023-10-16T09:59:23.138362+0000","last_active":"2023-10-16T09:59:24.559219+0000","last_peered":"2023-10-16T09:59:24.559219+0000","last_clean":"2023-10-16T09:59:24.559219+0000","last_became_active":"2023-10-16T09:59:23.138100+0000","last_became_peered":"2023-10-16T09:59:23.138100+0000","last_unstale":"2023-10-16T09:59:24.559219+0000","last_undegraded":"2023-10-16T09:59:24.559219+0000","last_fullsized":"2023-10-16T09:59:24.559219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:16.178113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589644+0000","last_change":"2023-10-16T09:59:23.584866+0000","last_active":"2023-10-16T09:59:23.589644+0000","last_peered":"2023-10-16T09:59:23.589644+0000","last_clean":"2023-10-16T09:59:23.589644+0000","last_became_active":"2023-10-16T09:59:23.583216+0000","last_became_peered":"2023-10-16T09:59:23.583216+0000","last_unstale":"2023-10-16T09:59:23.589644+0000","last_undegraded":"2023-10-16T09:59:23.589644+0000","last_fullsized":"2023-10-16T09:59:23.589644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:22.128625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729647+0000","last_change":"2023-10-16T09:59:23.142484+0000","last_active":"2023-10-16T09:59:24.729647+0000","last_peered":"2023-10-16T09:59:24.729647+0000","last_clean":"2023-10-16T09:59:24.729647+0000","last_became_active":"2023-10-16T09:59:23.142213+0000","last_became_peered":"2023-10-16T09:59:23.142213+0000","last_unstale":"2023-10-16T09:59:24.729647+0000","last_undegraded":"2023-10-16T09:59:24.729647+0000","last_fullsized":"2023-10-16T09:59:24.729647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:36.220534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560244+0000","last_change":"2023-10-16T09:59:23.575511+0000","last_active":"2023-10-16T09:59:24.560244+0000","last_peered":"2023-10-16T09:59:24.560244+0000","last_clean":"2023-10-16T09:59:24.560244+0000","last_became_active":"2023-10-16T09:59:23.575157+0000","last_became_peered":"2023-10-16T09:59:23.575157+0000","last_unstale":"2023-10-16T09:59:24.560244+0000","last_undegraded":"2023-10-16T09:59:24.560244+0000","last_fullsized":"2023-10-16T09:59:24.560244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:28.431724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560656+0000","last_change":"2023-10-16T09:59:23.141121+0000","last_active":"2023-10-16T09:59:24.560656+0000","last_peered":"2023-10-16T09:59:24.560656+0000","last_clean":"2023-10-16T09:59:24.560656+0000","last_became_active":"2023-10-16T09:59:23.140876+0000","last_became_peered":"2023-10-16T09:59:23.140876+0000","last_unstale":"2023-10-16T09:59:24.560656+0000","last_undegraded":"2023-10-16T09:59:24.560656+0000","last_fullsized":"2023-10-16T09:59:24.560656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:16:23.378890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.559039+0000","last_change":"2023-10-16T09:59:23.571192+0000","last_active":"2023-10-16T09:59:24.559039+0000","last_peered":"2023-10-16T09:59:24.559039+0000","last_clean":"2023-10-16T09:59:24.559039+0000","last_became_active":"2023-10-16T09:59:23.570948+0000","last_became_peered":"2023-10-16T09:59:23.570948+0000","last_unstale":"2023-10-16T09:59:24.559039+0000","last_undegraded":"2023-10-16T09:59:24.559039+0000","last_fullsized":"2023-10-16T09:59:24.559039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:10.087841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560678+0000","last_change":"2023-10-16T09:59:23.147218+0000","last_active":"2023-10-16T09:59:24.560678+0000","last_peered":"2023-10-16T09:59:24.560678+0000","last_clean":"2023-10-16T09:59:24.560678+0000","last_became_active":"2023-10-16T09:59:23.146899+0000","last_became_peered":"2023-10-16T09:59:23.146899+0000","last_unstale":"2023-10-16T09:59:24.560678+0000","last_undegraded":"2023-10-16T09:59:24.560678+0000","last_fullsized":"2023-10-16T09:59:24.560678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:23.942308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.573247+0000","last_change":"2023-10-16T09:59:23.570493+0000","last_active":"2023-10-16T09:59:23.573247+0000","last_peered":"2023-10-16T09:59:23.573247+0000","last_clean":"2023-10-16T09:59:23.573247+0000","last_became_active":"2023-10-16T09:59:23.570000+0000","last_became_peered":"2023-10-16T09:59:23.570000+0000","last_unstale":"2023-10-16T09:59:23.573247+0000","last_undegraded":"2023-10-16T09:59:23.573247+0000","last_fullsized":"2023-10-16T09:59:23.573247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:18.527228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.558671+0000","last_change":"2023-10-16T09:59:23.147119+0000","last_active":"2023-10-16T09:59:24.558671+0000","last_peered":"2023-10-16T09:59:24.558671+0000","last_clean":"2023-10-16T09:59:24.558671+0000","last_became_active":"2023-10-16T09:59:23.146692+0000","last_became_peered":"2023-10-16T09:59:23.146692+0000","last_unstale":"2023-10-16T09:59:24.558671+0000","last_undegraded":"2023-10-16T09:59:24.558671+0000","last_fullsized":"2023-10-16T09:59:24.558671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:44.988868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T09:59:23.589745+0000","last_change":"2023-10-16T09:59:23.585868+0000","last_active":"2023-10-16T09:59:23.589745+0000","last_peered":"2023-10-16T09:59:23.589745+0000","last_clean":"2023-10-16T09:59:23.589745+0000","last_became_active":"2023-10-16T09:59:23.584053+0000","last_became_peered":"2023-10-16T09:59:23.584053+0000","last_unstale":"2023-10-16T09:59:23.589745+0000","last_undegraded":"2023-10-16T09:59:23.589745+0000","last_fullsized":"2023-10-16T09:59:23.589745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:23.116271+0000","last_clean_scrub_stamp":"2023-10-16T09:59:23.116271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:54:39.398124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.560522+0000","last_change":"2023-10-16T09:59:23.136471+0000","last_active":"2023-10-16T09:59:24.560522+0000","last_peered":"2023-10-16T09:59:24.560522+0000","last_clean":"2023-10-16T09:59:24.560522+0000","last_became_active":"2023-10-16T09:59:23.136241+0000","last_became_peered":"2023-10-16T09:59:23.136241+0000","last_unstale":"2023-10-16T09:59:24.560522+0000","last_undegraded":"2023-10-16T09:59:24.560522+0000","last_fullsized":"2023-10-16T09:59:24.560522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:22.107676+0000","last_clean_scrub_stamp":"2023-10-16T09:59:22.107676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:41.580154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":387,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T09:59:24.729291+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:59:24.729291+0000","last_peered":"2023-10-16T09:59:24.729291+0000","last_clean":"2023-10-16T09:59:24.729291+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:59:24.729291+0000","last_undegraded":"2023-10-16T09:59:24.729291+0000","last_fullsized":"2023-10-16T09:59:24.729291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834575042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1024000,"data_stored":522124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57099999999999995,"15min":0.58599999999999997},"min":{"1min":0.46800000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.79400000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.73199999999999998,"5min":0.78800000000000003,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83699999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59299999999999997,"15min":0.59999999999999998},"min":{"1min":0.51900000000000002,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.72199999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.39300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71999999999999997,"5min":1.448,"15min":1.448},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.52700000000000002,"5min":0.35899999999999999,"15min":0.317},"max":{"1min":0.874,"5min":1.708,"15min":1.708},"last":0.65400000000000003}]},{"osd":3,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.38800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.81699999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.46600000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.69299999999999995,"5min":0.77100000000000002,"15min":0.82999999999999996},"last":0.626}]},{"osd":4,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45600000000000002,"15min":0.48899999999999999},"min":{"1min":0.32900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":0.77400000000000002,"15min":0.93200000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.49199999999999999,"15min":0.51600000000000001},"min":{"1min":0.38200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67300000000000004,"5min":0.75700000000000001,"15min":0.85399999999999998},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.495,"15min":0.50600000000000001},"min":{"1min":0.38300000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.73899999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.81299999999999994,"15min":0.89200000000000002},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Oct 16 09:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54500000000000004,"15min":0.56100000000000005},"min":{"1min":0.40400000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85299999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78000000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.57299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46064,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325776,"statfs":{"total":96636764160,"available":96589594624,"internally_reserved":0,"allocated":1024000,"data_stored":522112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63400000000000001,"15min":0.61199999999999999},"min":{"1min":0.35299999999999998,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":0.84199999999999997,"5min":1.792,"15min":1.792},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63300000000000001,"15min":0.61899999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.85699999999999998,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.57899999999999996},"min":{"1min":0.46100000000000002,"5min":0.315,"15min":0.249},"max":{"1min":0.745,"5min":0.875,"15min":1.1279999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.42599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":1.0720000000000001},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.70299999999999996,"5min":1.496,"15min":1.496},"last":0.432},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.622,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.879,"5min":2.069,"15min":2.069},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76900000000000002,"5min":0.78000000000000003,"15min":0.84499999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61299999999999999,"15min":0.59499999999999997},"min":{"1min":0.46000000000000002,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.89100000000000001,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.63}]},{"osd":4,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47599999999999998,"15min":0.47799999999999998},"min":{"1min":0.34699999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64600000000000002,"5min":0.747,"15min":0.84199999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.41999999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.85399999999999998,"15min":0.86899999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48999999999999999,"15min":0.49299999999999999},"min":{"1min":0.35899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76000000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54100000000000004},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70799999999999996,"5min":0.82299999999999995,"15min":0.88200000000000001},"last":0.745}]},{"osd":7,"last update":"Mon Oct 16 09:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76700000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.44800000000000001,"5min":0.34100000000000003,"15min":0.315},"max":{"1min":0.83499999999999996,"5min":0.876,"15min":0.88500000000000001},"last":0.78700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54392,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317448,"statfs":{"total":96636764160,"available":96581066752,"internally_reserved":0,"allocated":1019904,"data_stored":522090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.38400000000000001},"min":{"1min":0.311,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.254},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.316,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54000000000000004,"5min":0.64300000000000002,"15min":0.67500000000000004},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.754},"last":0.312},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.29199999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.505,"5min":0.60299999999999998,"15min":0.624},"last":0.27200000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41699999999999998,"15min":0.41899999999999998},"min":{"1min":0.34000000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.47099999999999997,"15min":0.48199999999999998},"min":{"1min":0.379,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.373}]},{"osd":4,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.51100000000000001,"5min":0.36799999999999999,"15min":0.36299999999999999},"max":{"1min":0.81200000000000006,"5min":1.0309999999999999,"15min":1.0700000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.65800000000000003,"15min":0.65600000000000003},"min":{"1min":0.52400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82799999999999996,"5min":0.95099999999999996,"15min":1.071},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.65600000000000003,"15min":0.65400000000000003},"min":{"1min":0.54600000000000004,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":0.92900000000000005,"5min":1.103,"15min":1.103},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.58999999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85599999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.42999999999999999}]},{"osd":6,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48999999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67500000000000004,"15min":0.68500000000000005},"min":{"1min":0.57799999999999996,"5min":0.40799999999999997,"15min":0.39200000000000002},"max":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.441}]},{"osd":7,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68400000000000005,"15min":0.68600000000000005},"min":{"1min":0.56000000000000005,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88200000000000001,"5min":1.0069999999999999,"15min":1.107},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68200000000000005,"15min":0.67400000000000004},"min":{"1min":0.58299999999999996,"5min":0.371,"15min":0.371},"max":{"1min":0.95299999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.46800000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1613824,"data_stored":1112458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38900000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.58499999999999996,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.441,"15min":0.42999999999999999},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.245},"max":{"1min":0.70999999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.497}]},{"osd":2,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41699999999999998,"15min":0.41799999999999998},"min":{"1min":0.32900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.56899999999999995,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.38100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.27800000000000002,"15min":0.217},"max":{"1min":0.60299999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.36899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.622,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.52600000000000002,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.622,"15min":0.624},"min":{"1min":0.46000000000000002,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":0.74299999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.81100000000000005}]},{"osd":5,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.626,"15min":0.63600000000000001},"min":{"1min":0.52400000000000002,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61599999999999999,"15min":0.64000000000000001},"min":{"1min":0.53500000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.74199999999999999,"5min":0.88700000000000001,"15min":5.2240000000000002},"last":0.82599999999999996}]},{"osd":6,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66000000000000003},"min":{"1min":0.54300000000000004,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77100000000000002,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66600000000000004,"15min":0.67100000000000004},"min":{"1min":0.59699999999999998,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.85799999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":1.014,"5min":1.014,"15min":1.02},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.55600000000000005,"5min":0.41299999999999998,"15min":0.34899999999999998},"max":{"1min":0.77200000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.876}]}]},{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59148,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312692,"statfs":{"total":96636764160,"available":96576196608,"internally_reserved":0,"allocated":1630208,"data_stored":1113234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.56100000000000005},"min":{"1min":0.45800000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":0.68400000000000005,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57199999999999995,"15min":0.60799999999999998},"min":{"1min":0.42499999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.72299999999999998,"5min":1.9059999999999999,"15min":11.246},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.59199999999999997,"15min":0.61299999999999999},"min":{"1min":0.51700000000000002,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78000000000000003,"5min":0.79300000000000004,"15min":1.1439999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.55600000000000005,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.76900000000000002,"5min":1.1279999999999999,"15min":1.212},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.81200000000000006,"5min":1.607,"15min":1.607},"last":0.5},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.48699999999999999,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.88700000000000001,"5min":2.044,"15min":2.044},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.47299999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71599999999999997,"5min":1.524,"15min":1.524},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.504,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.81799999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.67400000000000004}]},{"osd":4,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66400000000000003,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.40699999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76200000000000001,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":1.056},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.33200000000000002,"15min":0.308},"max":{"1min":0.84299999999999997,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.752,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.58299999999999996},"min":{"1min":0.36299999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96399999999999997},"last":0.58999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":531510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.25800000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.53400000000000003,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.317},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44400000000000001,"15min":0.439},"min":{"1min":0.36699999999999999,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.60099999999999998,"5min":0.67900000000000005,"15min":1.593},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.40699999999999997},"min":{"1min":0.29099999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53900000000000003,"5min":4.923,"15min":4.923},"last":0.374},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.249,"15min":0.216},"max":{"1min":0.56799999999999995,"5min":1.712,"15min":1.712},"last":0.39800000000000002}]},{"osd":3,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47599999999999998,"15min":0.45500000000000002},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.61699999999999999,"5min":1.984,"15min":1.984},"last":0.504},{"interface":"front","average":{"1min":0.44,"5min":0.438,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.65900000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63600000000000001,"15min":0.63100000000000001},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.81200000000000006,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68899999999999995,"15min":0.65800000000000003},"min":{"1min":0.51700000000000002,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.99199999999999999,"5min":5.0069999999999997,"15min":5.141},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.69799999999999995,"15min":0.67700000000000005},"min":{"1min":0.44900000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.91800000000000004,"5min":4.9749999999999996,"15min":5.0789999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.69399999999999995,"15min":0.67700000000000005},"min":{"1min":0.50700000000000001,"5min":0.40500000000000003,"15min":0.40300000000000002},"max":{"1min":0.93100000000000005,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.53400000000000003,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.95199999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69899999999999995,"15min":0.67000000000000004},"min":{"1min":0.55500000000000005,"5min":0.41599999999999998,"15min":0.371},"max":{"1min":0.84099999999999997,"5min":5.0419999999999998,"15min":5.1760000000000002},"last":0.47299999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:58:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.47899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.82399999999999995,"5min":1.744,"15min":1.744},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.432,"15min":0.38500000000000001},"max":{"1min":0.90200000000000002,"5min":0.91400000000000003,"15min":1.0329999999999999},"last":0.49199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36356,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335484,"statfs":{"total":96636764160,"available":96599535616,"internally_reserved":0,"allocated":1032192,"data_stored":529134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.378,"15min":0.371},"min":{"1min":0.29099999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.56000000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.27600000000000002,"5min":0.23100000000000001,"15min":0.20999999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.67000000000000004},"last":0.45100000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.39500000000000002},"min":{"1min":0.33400000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51600000000000001,"5min":0.65400000000000003,"15min":0.66100000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.434,"15min":0.42999999999999999},"min":{"1min":0.33800000000000002,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":0.60899999999999999,"5min":0.61899999999999999,"15min":0.67500000000000004},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.46999999999999997,"15min":0.46500000000000002},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":1.0760000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.33600000000000002,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.93799999999999994},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63100000000000001,"15min":0.623},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.90300000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.64900000000000002,"15min":0.64000000000000001},"min":{"1min":0.48299999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.64500000000000002},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65100000000000002,"15min":0.64300000000000002},"min":{"1min":0.45700000000000002,"5min":0.35999999999999999,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.90200000000000002,"15min":1.036},"last":0.57899999999999996}]},{"osd":6,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.66300000000000003,"15min":0.65000000000000002},"min":{"1min":0.51000000000000001,"5min":0.374,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.66100000000000003},"min":{"1min":0.53300000000000003,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.82399999999999995,"5min":0.94299999999999995,"15min":1.0589999999999999},"last":0.55700000000000005}]},{"osd":7,"last update":"Mon Oct 16 09:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.90600000000000003,"5min":0.90900000000000003,"15min":1.0029999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.47099999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.877,"5min":0.92800000000000005,"15min":0.999},"last":0.52100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1032192,"data_stored":523702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70699999999999996,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.441},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.39400000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.71999999999999997,"5min":1.0860000000000001,"15min":1.3029999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.82599999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57999999999999996,"15min":0.59299999999999997},"min":{"1min":0.45400000000000001,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.73499999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58299999999999996,"15min":0.59799999999999998},"min":{"1min":0.45200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60999999999999999,"15min":0.627},"min":{"1min":0.44,"5min":0.379,"15min":0.318},"max":{"1min":0.76300000000000001,"5min":1.2270000000000001,"15min":1.8899999999999999},"last":0.60999999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58799999999999997,"15min":0.59899999999999998},"min":{"1min":0.47899999999999998,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.77700000000000002,"5min":0.86099999999999999,"15min":1.7909999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":1.1839999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.29999999999999999,"15min":0.28599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.88900000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.50800000000000001,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78500000000000003,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.624}]},{"osd":6,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51600000000000001,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.74299999999999999,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.57999999999999996},"min":{"1min":0.39200000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84899999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56399999999999995,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83099999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60499999999999998,"15min":0.61399999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80000000000000004,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T09:59:28.624 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T09:59:28.625 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T09:59:28.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.785+0000 7f7df5ae4700 1 Processor -- start 2023-10-16T09:59:28.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.785+0000 7f7df5ae4700 1 -- start start 2023-10-16T09:59:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7df5ae4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7df00b30a0 0x7f7df00cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7df5ae4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 0x7f7df0151f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7df5ae4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df015f060 con 0x7f7df00b3480 2023-10-16T09:59:28.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7df5ae4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df015c2a0 con 0x7f7df00b30a0 2023-10-16T09:59:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7df5ae4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df015c580 con 0x7f7df015f820 2023-10-16T09:59:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7deeffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7df00b3480 0x7f7df015ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52486/0 (socket says 172.21.15.158:52486) 2023-10-16T09:59:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7deeffd700 1 -- 172.21.15.158:0/1489903262 learned_addr learned my addr 172.21.15.158:0/1489903262 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7def7fe700 1 --2- 172.21.15.158:0/1489903262 >> v2:172.21.15.172:3300/0 conn(0x7f7df00b30a0 0x7f7df00cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7deffff700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 0x7f7df0151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206969493 0 0) 0x7f7df015f060 con 0x7f7df00b3480 2023-10-16T09:59:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd80035b0 con 0x7f7df00b3480 2023-10-16T09:59:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3053651523 0 0) 0x7f7dd80035b0 con 0x7f7df00b3480 2023-10-16T09:59:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 msgr2=0x7f7df0151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7dee7fc700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 0x7f7df0151f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 >> v2:172.21.15.172:3300/0 conn(0x7f7df00b30a0 msgr2=0x7f7df00cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:28.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.786+0000 7f7dee7fc700 1 --2- 172.21.15.158:0/1489903262 >> v2:172.21.15.172:3300/0 conn(0x7f7df00b30a0 0x7f7df00cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df015c860 con 0x7f7df00b3480 2023-10-16T09:59:28.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7de0001670 con 0x7f7df00b3480 2023-10-16T09:59:28.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 >> v1:172.21.15.158:6789/0 conn(0x7f7df00b3480 legacy=0x7f7df015ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 0x7f7dd8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd8003f80 0x7f7dd8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dd8004d90 0x7f7dd8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df015c2a0 con 0x7f7dd8004d90 2023-10-16T09:59:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df015c580 con 0x7f7dd8003f80 2023-10-16T09:59:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd800ba10 con 0x7f7df015f820 2023-10-16T09:59:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deffff700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dd8004d90 0x7f7dd8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7def7fe700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 0x7f7dd8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deeffd700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd8003f80 0x7f7dd8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deffff700 1 -- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 msgr2=0x7f7dd8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deffff700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df015f820 0x7f7dd8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deffff700 1 -- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd8003f80 msgr2=0x7f7dd8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:28.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deffff700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd8003f80 0x7f7dd8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deffff700 1 -- 172.21.15.158:0/1489903262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80035b0 con 0x7f7dd8004d90 2023-10-16T09:59:28.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7deffff700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dd8004d90 0x7f7dd8005210 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009fc0 tx=0x7f7dd8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec81a6a6b5ee2c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de800ebd0 con 0x7f7dd8004d90 2023-10-16T09:59:28.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de800f040 con 0x7f7dd8004d90 2023-10-16T09:59:28.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.787+0000 7f7dee7fc700 1 -- 172.21.15.158:0/1489903262 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8014480 con 0x7f7dd8004d90 2023-10-16T09:59:28.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.788+0000 7f7df5ae4700 1 -- 172.21.15.158:0/1489903262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dd8004d90 msgr2=0x7f7dd8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.788+0000 7f7df5ae4700 1 --2- 172.21.15.158:0/1489903262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dd8004d90 0x7f7dd8005210 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009fc0 tx=0x7f7dd8005880 comp rx=0 tx=0).stop 2023-10-16T09:59:28.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.788+0000 7f7deffff700 1 -- 172.21.15.158:0/1489903262 reap_dead start 2023-10-16T09:59:28.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.788+0000 7f7df5ae4700 1 -- 172.21.15.158:0/1489903262 shutdown_connections 2023-10-16T09:59:28.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.788+0000 7f7df5ae4700 1 -- 172.21.15.158:0/1489903262 >> 172.21.15.158:0/1489903262 conn(0x7f7df006ec50 msgr2=0x7f7df006f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:28.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.788+0000 7f7df5ae4700 1 -- 172.21.15.158:0/1489903262 shutdown_connections 2023-10-16T09:59:28.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.788+0000 7f7df5ae4700 1 -- 172.21.15.158:0/1489903262 wait complete. 2023-10-16T09:59:28.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.789+0000 7f7df5ae4700 1 Processor -- start 2023-10-16T09:59:28.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.789+0000 7f7df5ae4700 1 -- start start 2023-10-16T09:59:28.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7df5ae4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df015a7b0 0x7f7df006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7df5ae4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7df00b48a0 0x7f7df006d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7df5ae4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df006d6a0 0x7f7df01f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7df5ae4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df015ab90 con 0x7f7df015a7b0 2023-10-16T09:59:28.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7df5ae4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df006c0c0 con 0x7f7df00b48a0 2023-10-16T09:59:28.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7df5ae4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df006c3a0 con 0x7f7df006d6a0 2023-10-16T09:59:28.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df015a7b0 0x7f7df006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df015a7b0 0x7f7df006b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52320/0 (socket says 172.21.15.158:52320) 2023-10-16T09:59:28.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 -- 172.21.15.158:0/3842511979 learned_addr learned my addr 172.21.15.158:0/3842511979 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:28.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 -- 172.21.15.158:0/3842511979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df006d6a0 msgr2=0x7f7df01f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df006d6a0 0x7f7df01f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 -- 172.21.15.158:0/3842511979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7df00b48a0 msgr2=0x7f7df006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:28.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7df00b48a0 0x7f7df006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:28.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.790+0000 7f7def7fe700 1 -- 172.21.15.158:0/3842511979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df006c680 con 0x7f7df015a7b0 2023-10-16T09:59:28.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.791+0000 7f7def7fe700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df015a7b0 0x7f7df006b9e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7de4011aa0 tx=0x7f7de4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.791+0000 7f7dd7fff700 1 -- 172.21.15.158:0/3842511979 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de400f960 con 0x7f7df015a7b0 2023-10-16T09:59:28.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.791+0000 7f7dd7fff700 1 -- 172.21.15.158:0/3842511979 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de400fac0 con 0x7f7df015a7b0 2023-10-16T09:59:28.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.791+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df006c960 con 0x7f7df015a7b0 2023-10-16T09:59:28.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.791+0000 7f7dd7fff700 1 -- 172.21.15.158:0/3842511979 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de400fe70 con 0x7f7df015a7b0 2023-10-16T09:59:28.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.791+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df00b7030 con 0x7f7df015a7b0 2023-10-16T09:59:28.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.792+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df00706b0 con 0x7f7df015a7b0 2023-10-16T09:59:28.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.792+0000 7f7dd7fff700 1 -- 172.21.15.158:0/3842511979 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7de4014440 con 0x7f7df015a7b0 2023-10-16T09:59:28.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.792+0000 7f7dd7fff700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f7ddc06b020 0x7f7ddc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:28.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.793+0000 7f7dd7fff700 1 -- 172.21.15.158:0/3842511979 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f7de408a190 con 0x7f7df015a7b0 2023-10-16T09:59:28.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.793+0000 7f7deeffd700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f7ddc06b020 0x7f7ddc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:28.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.793+0000 7f7deeffd700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f7ddc06b020 0x7f7ddc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7df00c4ad0 tx=0x7f7de0004870 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:28.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:28.797+0000 7f7dd7fff700 1 -- 172.21.15.158:0/3842511979 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7de4054ed0 con 0x7f7df015a7b0 2023-10-16T09:59:29.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.069+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7df00681b0 con 0x7f7df015a7b0 2023-10-16T09:59:29.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.071+0000 7f7dd7fff700 1 -- 172.21.15.158:0/3842511979 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7de40584f0 con 0x7f7df015a7b0 2023-10-16T09:59:29.073 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T09:59:29.073 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T09:59:29.073 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T09:59:29.073 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T09:59:29.073 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T09:59:29.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.073+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f7ddc06b020 msgr2=0x7f7ddc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:29.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.073+0000 7f7df5ae4700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f7ddc06b020 0x7f7ddc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7df00c4ad0 tx=0x7f7de0004870 comp rx=0 tx=0).stop 2023-10-16T09:59:29.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df015a7b0 msgr2=0x7f7df006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:29.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df015a7b0 0x7f7df006b9e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7de4011aa0 tx=0x7f7de4011e60 comp rx=0 tx=0).stop 2023-10-16T09:59:29.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 shutdown_connections 2023-10-16T09:59:29.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f7ddc06b020 0x7f7ddc06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7df006d6a0 0x7f7df01f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df015a7b0 0x7f7df006b9e0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 --2- 172.21.15.158:0/3842511979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7df00b48a0 0x7f7df006d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 >> 172.21.15.158:0/3842511979 conn(0x7f7df006ec50 msgr2=0x7f7df0073db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:29.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 shutdown_connections 2023-10-16T09:59:29.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.074+0000 7f7df5ae4700 1 -- 172.21.15.158:0/3842511979 wait complete. 2023-10-16T09:59:29.078 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T09:59:29.087 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:29.108 INFO:tasks.ceph.mds.d.smithi172.stderr:2023-10-16T09:59:29.107+0000 7fd6644af700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T09:59:29.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.207+0000 7f43278dd700 1 Processor -- start 2023-10-16T09:59:29.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.207+0000 7f43278dd700 1 -- start start 2023-10-16T09:59:29.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f43278dd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f43200cc5d0 0x7f432006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:29.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f43278dd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f432015dfb0 0x7f43200cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:29.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f43278dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 0x7f432006f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:29.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f43278dd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43200cf270 con 0x7f432015dfb0 2023-10-16T09:59:29.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f43278dd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432006fd10 con 0x7f43200cc5d0 2023-10-16T09:59:29.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f43278dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432006e770 con 0x7f432006f340 2023-10-16T09:59:29.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f4324e78700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f432015dfb0 0x7f43200cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:29.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f4324e78700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f432015dfb0 0x7f43200cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52330/0 (socket says 172.21.15.158:52330) 2023-10-16T09:59:29.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.208+0000 7f4324e78700 1 -- 172.21.15.158:0/2502903449 learned_addr learned my addr 172.21.15.158:0/2502903449 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:29.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4324e78700 1 -- 172.21.15.158:0/2502903449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 msgr2=0x7f432006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:29.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4324e78700 1 --2- 172.21.15.158:0/2502903449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 0x7f432006f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4324e78700 1 -- 172.21.15.158:0/2502903449 >> v2:172.21.15.172:3300/0 conn(0x7f43200cc5d0 msgr2=0x7f432006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:29.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4324e78700 1 --2- 172.21.15.158:0/2502903449 >> v2:172.21.15.172:3300/0 conn(0x7f43200cc5d0 0x7f432006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4324e78700 1 -- 172.21.15.158:0/2502903449 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432015e4e0 con 0x7f432015dfb0 2023-10-16T09:59:29.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4324e78700 1 --2- 172.21.15.158:0/2502903449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200cb6f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f43200d1390 tx=0x7f431c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb2fe81a1fd3b78 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:29.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4317fff700 1 -- 172.21.15.158:0/2502903449 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00b870 con 0x7f432015dfb0 2023-10-16T09:59:29.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4317fff700 1 -- 172.21.15.158:0/2502903449 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c00b9d0 con 0x7f432015dfb0 2023-10-16T09:59:29.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.209+0000 7f4317fff700 1 -- 172.21.15.158:0/2502903449 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00bd60 con 0x7f432015dfb0 2023-10-16T09:59:29.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 -- 172.21.15.158:0/2502903449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 msgr2=0x7f43200cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:29.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 --2- 172.21.15.158:0/2502903449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200cb6f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f43200d1390 tx=0x7f431c009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 -- 172.21.15.158:0/2502903449 shutdown_connections 2023-10-16T09:59:29.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 --2- 172.21.15.158:0/2502903449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 0x7f432006f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 --2- 172.21.15.158:0/2502903449 >> v2:172.21.15.172:3300/0 conn(0x7f43200cc5d0 0x7f432006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 --2- 172.21.15.158:0/2502903449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200cb6f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 -- 172.21.15.158:0/2502903449 >> 172.21.15.158:0/2502903449 conn(0x7f43200d02b0 msgr2=0x7f43200d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:29.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 -- 172.21.15.158:0/2502903449 shutdown_connections 2023-10-16T09:59:29.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.210+0000 7f43278dd700 1 -- 172.21.15.158:0/2502903449 wait complete. 2023-10-16T09:59:29.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.211+0000 7f43278dd700 1 Processor -- start 2023-10-16T09:59:29.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.211+0000 7f43278dd700 1 -- start start 2023-10-16T09:59:29.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f43278dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 0x7f43201f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:29.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f43278dd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43200cc5d0 0x7f43200b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:29.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f43278dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:29.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f43278dd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43200b5580 con 0x7f432015dfb0 2023-10-16T09:59:29.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f43278dd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43200b52a0 con 0x7f43200cc5d0 2023-10-16T09:59:29.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f43278dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43201f0e80 con 0x7f432006f340 2023-10-16T09:59:29.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:29.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200b4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52338/0 (socket says 172.21.15.158:52338) 2023-10-16T09:59:29.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 -- 172.21.15.158:0/3732088113 learned_addr learned my addr 172.21.15.158:0/3732088113 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:29.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4324e78700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43200cc5d0 0x7f43200b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:29.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 -- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 msgr2=0x7f43201f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:29.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325679700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 0x7f43201f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:29.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 0x7f43201f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 -- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43200cc5d0 msgr2=0x7f43200b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:29.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43200cc5d0 0x7f43200b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 -- 172.21.15.158:0/3732088113 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c009550 con 0x7f432015dfb0 2023-10-16T09:59:29.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4325e7a700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200b4c70 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f431800b970 tx=0x7f431800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:29.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4315ffb700 1 -- 172.21.15.158:0/3732088113 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431800c9b0 con 0x7f432015dfb0 2023-10-16T09:59:29.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4315ffb700 1 -- 172.21.15.158:0/3732088113 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431800cb30 con 0x7f432015dfb0 2023-10-16T09:59:29.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43201f1160 con 0x7f432015dfb0 2023-10-16T09:59:29.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.212+0000 7f4315ffb700 1 -- 172.21.15.158:0/3732088113 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318014440 con 0x7f432015dfb0 2023-10-16T09:59:29.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.213+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43201f1720 con 0x7f432015dfb0 2023-10-16T09:59:29.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.213+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43201f1910 con 0x7f432015dfb0 2023-10-16T09:59:29.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.214+0000 7f4315ffb700 1 -- 172.21.15.158:0/3732088113 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f431801d3b0 con 0x7f432015dfb0 2023-10-16T09:59:29.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.214+0000 7f4315ffb700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f430406b020 0x7f430406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:29.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.214+0000 7f4325679700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f430406b020 0x7f430406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:29.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.214+0000 7f4315ffb700 1 -- 172.21.15.158:0/3732088113 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f4318089830 con 0x7f432015dfb0 2023-10-16T09:59:29.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.215+0000 7f4325679700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f430406b020 0x7f430406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4320158f40 tx=0x7f4310008040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:29.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.219+0000 7f4315ffb700 1 -- 172.21.15.158:0/3732088113 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43180544e0 con 0x7f432015dfb0 2023-10-16T09:59:29.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.454+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43200681b0 con 0x7f432015dfb0 2023-10-16T09:59:29.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.455+0000 7f4315ffb700 1 -- 172.21.15.158:0/3732088113 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f4318019090 con 0x7f432015dfb0 2023-10-16T09:59:29.457 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:29.457 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3413925509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3413925509},{"type":"v1","addr":"172.21.15.158:6835","nonce":3413925509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/655082434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":655082434},{"type":"v1","addr":"172.21.15.158:6837","nonce":655082434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2666705933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2666705933},{"type":"v1","addr":"172.21.15.172:6835","nonce":2666705933}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:23.515854+0000","modified":"2023-10-16T09:59:26.532765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35639":{"gid":35639,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3389815719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3389815719},{"type":"v1","addr":"172.21.15.172:6834","nonce":3389815719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T09:59:29.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f430406b020 msgr2=0x7f430406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:29.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f430406b020 0x7f430406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4320158f40 tx=0x7f4310008040 comp rx=0 tx=0).stop 2023-10-16T09:59:29.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 msgr2=0x7f43200b4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:29.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200b4c70 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f431800b970 tx=0x7f431800bd30 comp rx=0 tx=0).stop 2023-10-16T09:59:29.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 shutdown_connections 2023-10-16T09:59:29.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f430406b020 0x7f430406d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f432015dfb0 0x7f43200b4c70 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f432006f340 0x7f43201f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 --2- 172.21.15.158:0/3732088113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43200cc5d0 0x7f43200b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:29.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 >> 172.21.15.158:0/3732088113 conn(0x7f43200d02b0 msgr2=0x7f43200d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:29.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 shutdown_connections 2023-10-16T09:59:29.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.457+0000 7f43278dd700 1 -- 172.21.15.158:0/3732088113 wait complete. 2023-10-16T09:59:29.461 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 33 2023-10-16T09:59:29.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T09:59:23.515854+0000', 'modified': '2023-10-16T09:59:26.532765+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 35639}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_35639': {'gid': 35639, 'name': 'd', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6834/3389815719', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6832', 'nonce': 3389815719}, {'type': 'v1', 'addr': '172.21.15.172:6834', 'nonce': 3389815719}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T09:59:29.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T09:59:29.873 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T09:59:29.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.997+0000 7f8941b75700 1 Processor -- start 2023-10-16T09:59:29.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.997+0000 7f8941b75700 1 -- start start 2023-10-16T09:59:30.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.997+0000 7f8941b75700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f893c0cfe30 0x7f893c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.997+0000 7f8941b75700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f893c0cd190 0x7f893c06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.997+0000 7f8941b75700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c15ec70 con 0x7f893c15a6d0 2023-10-16T09:59:30.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.997+0000 7f8941b75700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893c0afcd0 con 0x7f893c0cfe30 2023-10-16T09:59:30.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.997+0000 7f8941b75700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893c0cd620 con 0x7f893c0cd190 2023-10-16T09:59:30.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f893c0cfe30 0x7f893c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f893c0cfe30 0x7f893c0ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52344/0 (socket says 172.21.15.158:52344) 2023-10-16T09:59:30.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 -- 172.21.15.158:0/1501075620 learned_addr learned my addr 172.21.15.158:0/1501075620 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:30.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 -- 172.21.15.158:0/1501075620 >> v2:172.21.15.172:3300/0 conn(0x7f893c0cd190 msgr2=0x7f893c06ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 --2- 172.21.15.158:0/1501075620 >> v2:172.21.15.172:3300/0 conn(0x7f893c0cd190 0x7f893c06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 -- 172.21.15.158:0/1501075620 >> v1:172.21.15.172:6789/0 conn(0x7f893c15a6d0 legacy=0x7f893c0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 -- 172.21.15.158:0/1501075620 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c15f1b0 con 0x7f893c0cfe30 2023-10-16T09:59:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893affd700 1 --2- 172.21.15.158:0/1501075620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c0cfe30 0x7f893c0ccc50 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f893c0c7420 tx=0x7f892400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c6692f1f32db2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893a7fc700 1 -- 172.21.15.158:0/1501075620 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892400dcf0 con 0x7f893c0cfe30 2023-10-16T09:59:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893a7fc700 1 -- 172.21.15.158:0/1501075620 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892400de50 con 0x7f893c0cfe30 2023-10-16T09:59:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.998+0000 7f893a7fc700 1 -- 172.21.15.158:0/1501075620 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924004250 con 0x7f893c0cfe30 2023-10-16T09:59:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 -- 172.21.15.158:0/1501075620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c0cfe30 msgr2=0x7f893c0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 --2- 172.21.15.158:0/1501075620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c0cfe30 0x7f893c0ccc50 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f893c0c7420 tx=0x7f892400be70 comp rx=0 tx=0).stop 2023-10-16T09:59:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 -- 172.21.15.158:0/1501075620 shutdown_connections 2023-10-16T09:59:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 --2- 172.21.15.158:0/1501075620 >> v2:172.21.15.172:3300/0 conn(0x7f893c0cd190 0x7f893c06ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 --2- 172.21.15.158:0/1501075620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c0cfe30 0x7f893c0ccc50 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 -- 172.21.15.158:0/1501075620 >> 172.21.15.158:0/1501075620 conn(0x7f893c06b9e0 msgr2=0x7f893c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 -- 172.21.15.158:0/1501075620 shutdown_connections 2023-10-16T09:59:30.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:29.999+0000 7f8941b75700 1 -- 172.21.15.158:0/1501075620 wait complete. 2023-10-16T09:59:30.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.000+0000 7f8941b75700 1 Processor -- start 2023-10-16T09:59:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.000+0000 7f8941b75700 1 -- start start 2023-10-16T09:59:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.001+0000 7f8941b75700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f893c0cd190 0x7f893c1ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.001+0000 7f8941b75700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f893c0cfe30 0x7f893c15baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.001+0000 7f8941b75700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c15a6d0 0x7f893c0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.001+0000 7f8941b75700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0c74a0 con 0x7f893c15a6d0 2023-10-16T09:59:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.001+0000 7f8941b75700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0c7780 con 0x7f893c0cd190 2023-10-16T09:59:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.001+0000 7f8941b75700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0c7a60 con 0x7f893c0cfe30 2023-10-16T09:59:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c15a6d0 0x7f893c0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c15a6d0 0x7f893c0b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52350/0 (socket says 172.21.15.158:52350) 2023-10-16T09:59:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 -- 172.21.15.158:0/1430954421 learned_addr learned my addr 172.21.15.158:0/1430954421 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893b7fe700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f893c0cd190 0x7f893c1ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893affd700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f893c0cfe30 0x7f893c15baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 -- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f893c0cfe30 msgr2=0x7f893c15baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f893c0cfe30 0x7f893c15baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 -- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f893c0cd190 msgr2=0x7f893c1ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f893c0cd190 0x7f893c1ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 -- 172.21.15.158:0/1430954421 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892400b7f0 con 0x7f893c15a6d0 2023-10-16T09:59:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f893bfff700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c15a6d0 0x7f893c0b0820 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f893400b970 tx=0x7f893400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f8933fff700 1 -- 172.21.15.158:0/1430954421 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893400d860 con 0x7f893c15a6d0 2023-10-16T09:59:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f8933fff700 1 -- 172.21.15.158:0/1430954421 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893400d9e0 con 0x7f893c15a6d0 2023-10-16T09:59:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f8933fff700 1 -- 172.21.15.158:0/1430954421 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893400dd90 con 0x7f893c15a6d0 2023-10-16T09:59:30.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c0c7d40 con 0x7f893c15a6d0 2023-10-16T09:59:30.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.002+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c0c9dc0 con 0x7f893c15a6d0 2023-10-16T09:59:30.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.004+0000 7f8933fff700 1 -- 172.21.15.158:0/1430954421 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f893400e430 con 0x7f893c15a6d0 2023-10-16T09:59:30.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.004+0000 7f8933fff700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f891c06b020 0x7f891c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.004+0000 7f893b7fe700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f891c06b020 0x7f891c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.004+0000 7f8933fff700 1 -- 172.21.15.158:0/1430954421 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f8934089c90 con 0x7f893c15a6d0 2023-10-16T09:59:30.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.005+0000 7f893b7fe700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f891c06b020 0x7f891c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f892c007fd0 tx=0x7f892c009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.006+0000 7f894dd5fb80 1 Processor -- start 2023-10-16T09:59:30.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.006+0000 7f894dd5fb80 1 -- start start 2023-10-16T09:59:30.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a07b40 0x55aad3a07f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a07280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a2e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad397bc20 con 0x55aad3a2ef30 2023-10-16T09:59:30.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad3960c10 con 0x55aad3a02f00 2023-10-16T09:59:30.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aad395e530 con 0x55aad3a07b40 2023-10-16T09:59:30.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a07280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a07280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52358/0 (socket says 172.21.15.158:52358) 2023-10-16T09:59:30.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 -- 172.21.15.158:0/11104937 learned_addr learned my addr 172.21.15.158:0/11104937 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:30.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893b7fe700 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a07b40 0x55aad3a07f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893bfff700 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a2e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 -- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a07b40 msgr2=0x55aad3a07f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a07b40 0x55aad3a07f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 -- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 msgr2=0x55aad3a2e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a2e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 -- 172.21.15.158:0/11104937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aad39173a0 con 0x55aad3a2ef30 2023-10-16T09:59:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.007+0000 7f893affd700 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a07280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f892400beb0 tx=0x7f8924012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a2c3c923403d18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f89317fa700 1 -- 172.21.15.158:0/11104937 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924020040 con 0x55aad3a2ef30 2023-10-16T09:59:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f89317fa700 1 -- 172.21.15.158:0/11104937 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924004030 con 0x55aad3a2ef30 2023-10-16T09:59:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f89317fa700 1 -- 172.21.15.158:0/11104937 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89240036a0 con 0x55aad3a2ef30 2023-10-16T09:59:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 -- 172.21.15.158:0/11104937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 msgr2=0x55aad3a07280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a07280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f892400beb0 tx=0x7f8924012000 comp rx=0 tx=0).stop 2023-10-16T09:59:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 -- 172.21.15.158:0/11104937 shutdown_connections 2023-10-16T09:59:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a2e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a07b40 0x55aad3a07f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/11104937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a07280 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 -- 172.21.15.158:0/11104937 >> 172.21.15.158:0/11104937 conn(0x55aad3934d80 msgr2=0x55aad3a4fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 -- 172.21.15.158:0/11104937 shutdown_connections 2023-10-16T09:59:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.008+0000 7f894dd5fb80 1 -- 172.21.15.158:0/11104937 wait complete. 2023-10-16T09:59:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.009+0000 7f894dd5fb80 1 -- start start 2023-10-16T09:59:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.009+0000 7f894dd5fb80 1 Processor -- start 2023-10-16T09:59:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a02f00 0x55aad3a492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a34210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a2b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad3959e20 con 0x55aad3a2ef30 2023-10-16T09:59:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad38f66c0 con 0x55aad3a07b40 2023-10-16T09:59:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aad38f6360 con 0x55aad3a02f00 2023-10-16T09:59:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 10 client.0 _ll_register_callbacks cb 0x55aad37537b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:59:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a2b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a2b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52366/0 (socket says 172.21.15.158:52366) 2023-10-16T09:59:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 -- 172.21.15.158:0/3029074571 learned_addr learned my addr 172.21.15.158:0/3029074571 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893b7fe700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a02f00 0x55aad3a492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893affd700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a34210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 -- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a02f00 msgr2=0x55aad3a492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a02f00 0x55aad3a492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 -- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 msgr2=0x55aad3a34210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a34210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 -- 172.21.15.158:0/3029074571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c15f1b0 con 0x55aad3a2ef30 2023-10-16T09:59:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893affd700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a34210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f893bfff700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a2b760 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8934058460 tx=0x7f89340584d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868e54af3a7c5d26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f891a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:59:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f891a7fc700 1 -- 172.21.15.158:0/3029074571 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934058930 con 0x55aad3a2ef30 2023-10-16T09:59:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f891a7fc700 1 -- 172.21.15.158:0/3029074571 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f89340579c0 con 0x55aad3a2ef30 2023-10-16T09:59:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.010+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aad39fa5b0 con 0x55aad3a2ef30 2023-10-16T09:59:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f891a7fc700 1 -- 172.21.15.158:0/3029074571 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8934057b80 con 0x55aad3a2ef30 2023-10-16T09:59:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 10 client.6903 fetch_fsmap learned FSMap version 33 2023-10-16T09:59:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55aad3784d90 con 0x55aad3a2ef30 2023-10-16T09:59:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f891a7fc700 1 -- 172.21.15.158:0/3029074571 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55aad39fa5b0 con 0x55aad3a2ef30 2023-10-16T09:59:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f891a7fc700 1 -- 172.21.15.158:0/3029074571 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8934060340 con 0x55aad3a2ef30 2023-10-16T09:59:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 10 client.6903 fetch_fsmap finished waiting for FSMap version 33 2023-10-16T09:59:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 10 client.6903 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T09:59:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 20 client.6903 populate_metadata read hostname 'smithi158' 2023-10-16T09:59:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a35600 0x55aad3a47010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 4 client.6903 mds_command: new command op to 35639 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T09:59:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 --> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aad3952e00 con 0x55aad3a35600 2023-10-16T09:59:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.011+0000 7f893b7fe700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a35600 0x55aad3a47010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f891a7fc700 10 client.6903 ms_handle_connect on v2:172.21.15.172:6832/3389815719 2023-10-16T09:59:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f893b7fe700 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a35600 0x55aad3a47010 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1be643fb6559b778 server_cookie=5965a8575c88d217 in_seq=0 out_seq=0 2023-10-16T09:59:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f891a7fc700 1 -- 172.21.15.158:0/3029074571 <== mds.0 v2:172.21.15.172:6832/3389815719 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55aad3952e00 con 0x55aad3a35600 2023-10-16T09:59:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f891a7fc700 10 client.6903 handle_command_reply: tid=0 2023-10-16T09:59:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 client.6903 shutdown 2023-10-16T09:59:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 msgr2=0x55aad3a2b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a2b760 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8934058460 tx=0x7f89340584d0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 shutdown_connections 2023-10-16T09:59:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a35600 0x55aad3a47010 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a2ef30 0x55aad3a2b760 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a02f00 0x55aad3a492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f891a7fc700 0 client.6903 ms_handle_reset on v2:172.21.15.172:6832/3389815719 2023-10-16T09:59:30.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/3029074571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a34210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 >> 172.21.15.158:0/3029074571 conn(0x55aad3934d80 msgr2=0x55aad3a01f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 shutdown_connections 2023-10-16T09:59:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.012+0000 7f894dd5fb80 1 -- 172.21.15.158:0/3029074571 wait complete. 2023-10-16T09:59:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.013+0000 7f894dd5fb80 20 client.6903 trim_cache size 0 max 16384 2023-10-16T09:59:30.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.030+0000 7f894dd5fb80 1 Processor -- start 2023-10-16T09:59:30.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.030+0000 7f894dd5fb80 1 -- start start 2023-10-16T09:59:30.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a49490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a07b40 0x55aad3a341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a2b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad37da8d0 con 0x55aad3a07b40 2023-10-16T09:59:30.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad3a50950 con 0x55aad3a02f00 2023-10-16T09:59:30.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aad39fcd70 con 0x55aad3a2ef30 2023-10-16T09:59:30.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a07b40 0x55aad3a341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a07b40 0x55aad3a341f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52372/0 (socket says 172.21.15.158:52372) 2023-10-16T09:59:30.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 -- 172.21.15.158:0/248277184 learned_addr learned my addr 172.21.15.158:0/248277184 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:30.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893b7fe700 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a49490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893bfff700 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a2b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 -- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 msgr2=0x55aad3a2b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a2b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 -- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 msgr2=0x55aad3a49490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a49490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.031+0000 7f893affd700 1 -- 172.21.15.158:0/248277184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aad3a08cf0 con 0x55aad3a07b40 2023-10-16T09:59:30.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f893affd700 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a07b40 0x55aad3a341f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f892400beb0 tx=0x7f892400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a65dc9301749af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f8931ffb700 1 -- 172.21.15.158:0/248277184 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924020040 con 0x55aad3a07b40 2023-10-16T09:59:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f8931ffb700 1 -- 172.21.15.158:0/248277184 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924006bb0 con 0x55aad3a07b40 2023-10-16T09:59:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f8931ffb700 1 -- 172.21.15.158:0/248277184 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89240036a0 con 0x55aad3a07b40 2023-10-16T09:59:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 -- 172.21.15.158:0/248277184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a07b40 msgr2=0x55aad3a341f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a07b40 0x55aad3a341f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f892400beb0 tx=0x7f892400bf90 comp rx=0 tx=0).stop 2023-10-16T09:59:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 -- 172.21.15.158:0/248277184 shutdown_connections 2023-10-16T09:59:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a2b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a02f00 0x55aad3a49490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/248277184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a07b40 0x55aad3a341f0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 -- 172.21.15.158:0/248277184 >> 172.21.15.158:0/248277184 conn(0x55aad3934d80 msgr2=0x55aad3a03850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 -- 172.21.15.158:0/248277184 shutdown_connections 2023-10-16T09:59:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.032+0000 7f894dd5fb80 1 -- 172.21.15.158:0/248277184 wait complete. 2023-10-16T09:59:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.033+0000 7f894dd5fb80 1 -- start start 2023-10-16T09:59:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.033+0000 7f894dd5fb80 1 Processor -- start 2023-10-16T09:59:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a02f00 0x55aad3ab1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a6f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a70b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad3a6ffb0 con 0x55aad3a02f00 2023-10-16T09:59:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aad3ab1560 con 0x55aad3a07b40 2023-10-16T09:59:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aad3a083a0 con 0x55aad3a2ef30 2023-10-16T09:59:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 10 client.0 _ll_register_callbacks cb 0x7f8934057e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T09:59:30.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a02f00 0x55aad3ab1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a02f00 0x55aad3ab1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52376/0 (socket says 172.21.15.158:52376) 2023-10-16T09:59:30.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 -- 172.21.15.158:0/1649947419 learned_addr learned my addr 172.21.15.158:0/1649947419 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:30.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893affd700 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a6f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893bfff700 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a70b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 -- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 msgr2=0x55aad3a70b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a70b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 -- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 msgr2=0x55aad3a6f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a6f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 -- 172.21.15.158:0/1649947419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aad39173a0 con 0x55aad3a02f00 2023-10-16T09:59:30.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f893b7fe700 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a02f00 0x55aad3ab1020 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f892c00f4b0 tx=0x7f892c000fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e6c610b61153a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f891a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T09:59:30.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f891a7fc700 1 -- 172.21.15.158:0/1649947419 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c024a00 con 0x55aad3a02f00 2023-10-16T09:59:30.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f891a7fc700 1 -- 172.21.15.158:0/1649947419 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f892c01c040 con 0x55aad3a02f00 2023-10-16T09:59:30.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.034+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8924004030 con 0x55aad3a02f00 2023-10-16T09:59:30.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f891a7fc700 1 -- 172.21.15.158:0/1649947419 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f892c024b60 con 0x55aad3a02f00 2023-10-16T09:59:30.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 10 client.6909 fetch_fsmap learned FSMap version 33 2023-10-16T09:59:30.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55aad376dc10 con 0x55aad3a02f00 2023-10-16T09:59:30.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f891a7fc700 1 -- 172.21.15.158:0/1649947419 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924004030 con 0x55aad3a02f00 2023-10-16T09:59:30.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f891a7fc700 1 -- 172.21.15.158:0/1649947419 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f892c024cc0 con 0x55aad3a02f00 2023-10-16T09:59:30.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 10 client.6909 fetch_fsmap finished waiting for FSMap version 33 2023-10-16T09:59:30.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 10 client.6909 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T09:59:30.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 20 client.6909 populate_metadata read hostname 'smithi158' 2023-10-16T09:59:30.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a6d0c0 0x55aad3a6d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 4 client.6909 mds_command: new command op to 35639 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T09:59:30.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 --> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aad3952b20 con 0x55aad3a6d0c0 2023-10-16T09:59:30.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f893affd700 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a6d0c0 0x55aad3a6d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f893affd700 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a6d0c0 0x55aad3a6d4a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7377de16b74db656 server_cookie=7fcedc2c4c7835b5 in_seq=0 out_seq=0 2023-10-16T09:59:30.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.035+0000 7f891a7fc700 10 client.6909 ms_handle_connect on v2:172.21.15.172:6832/3389815719 2023-10-16T09:59:30.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f891a7fc700 1 -- 172.21.15.158:0/1649947419 <== mds.0 v2:172.21.15.172:6832/3389815719 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55aad3952b20 con 0x55aad3a6d0c0 2023-10-16T09:59:30.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f891a7fc700 10 client.6909 handle_command_reply: tid=0 2023-10-16T09:59:30.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 client.6909 shutdown 2023-10-16T09:59:30.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a02f00 msgr2=0x55aad3ab1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a02f00 0x55aad3ab1020 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f892c00f4b0 tx=0x7f892c000fb0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 shutdown_connections 2023-10-16T09:59:30.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:6832/3389815719,v1:172.21.15.172:6834/3389815719] conn(0x55aad3a6d0c0 0x55aad3a6d4a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55aad3a2ef30 0x55aad3a70b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f891a7fc700 0 client.6909 ms_handle_reset on v2:172.21.15.172:6832/3389815719 2023-10-16T09:59:30.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55aad3a02f00 0x55aad3ab1020 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 --2- 172.21.15.158:0/1649947419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aad3a07b40 0x55aad3a6f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 >> 172.21.15.158:0/1649947419 conn(0x55aad3934d80 msgr2=0x55aad3a49490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 shutdown_connections 2023-10-16T09:59:30.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 1 -- 172.21.15.158:0/1649947419 wait complete. 2023-10-16T09:59:30.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f894dd5fb80 20 client.6909 trim_cache size 0 max 16384 2023-10-16T09:59:30.057 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:59:30.058 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T09:59:30.058 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T09:59:30.058 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 35639, 2023-10-16T09:59:30.058 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T09:59:30.058 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T09:59:30.059 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T09:59:30.059 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9138121239999997, 2023-10-16T09:59:30.059 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 33, 2023-10-16T09:59:30.059 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 85, 2023-10-16T09:59:30.059 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 85, 2023-10-16T09:59:30.060 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 11.834761517 2023-10-16T09:59:30.060 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:59:30.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f891c06b020 msgr2=0x7f891c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f891c06b020 0x7f891c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f892c007fd0 tx=0x7f892c009040 comp rx=0 tx=0).stop 2023-10-16T09:59:30.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c15a6d0 msgr2=0x7f893c0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c15a6d0 0x7f893c0b0820 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f893400b970 tx=0x7f893400bc80 comp rx=0 tx=0).stop 2023-10-16T09:59:30.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 shutdown_connections 2023-10-16T09:59:30.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f891c06b020 0x7f891c06d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f893c15a6d0 0x7f893c0b0820 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f893c0cd190 0x7f893c1ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 --2- 172.21.15.158:0/1430954421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f893c0cfe30 0x7f893c15baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.036+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 >> 172.21.15.158:0/1430954421 conn(0x7f893c06b9e0 msgr2=0x7f893c06e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.037+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 shutdown_connections 2023-10-16T09:59:30.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.037+0000 7f8941b75700 1 -- 172.21.15.158:0/1430954421 wait complete. 2023-10-16T09:59:30.064 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T09:59:30.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.183+0000 7f65eb83b700 1 Processor -- start 2023-10-16T09:59:30.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.184+0000 7f65eb83b700 1 -- start start 2023-10-16T09:59:30.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.184+0000 7f65eb83b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 0x7f65e40b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.184+0000 7f65eb83b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e415b930 con 0x7f65e40cc1b0 2023-10-16T09:59:30.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.184+0000 7f65eb83b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e415f5f0 con 0x7f65e40ce530 2023-10-16T09:59:30.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.184+0000 7f65eb83b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40cf020 con 0x7f65e415f0f0 2023-10-16T09:59:30.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.184+0000 7f65e8dd6700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f65e40cc1b0 0x7f65e40cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52562/0 (socket says 172.21.15.158:52562) 2023-10-16T09:59:30.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.184+0000 7f65e8dd6700 1 -- 172.21.15.158:0/1746946314 learned_addr learned my addr 172.21.15.158:0/1746946314 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:30.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250821937 0 0) 0x7f65e415b930 con 0x7f65e40cc1b0 2023-10-16T09:59:30.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 0x7f65e40b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d00035b0 con 0x7f65e40cc1b0 2023-10-16T09:59:30.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4267568278 0 0) 0x7f65d00035b0 con 0x7f65e40cc1b0 2023-10-16T09:59:30.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 msgr2=0x7f65e40b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 0x7f65e40b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 >> v1:172.21.15.172:6789/0 conn(0x7f65e40ce530 legacy=0x7f65e40cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e415f8d0 con 0x7f65e40cc1b0 2023-10-16T09:59:30.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f65e0001670 con 0x7f65e40cc1b0 2023-10-16T09:59:30.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 >> v1:172.21.15.158:6789/0 conn(0x7f65e40cc1b0 legacy=0x7f65e40cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d0003a30 0x7f65d0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 0x7f65d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d0004d90 0x7f65d0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40cf020 con 0x7f65d0004d90 2023-10-16T09:59:30.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d000bac0 con 0x7f65d0003a30 2023-10-16T09:59:30.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d000bc80 con 0x7f65e415f0f0 2023-10-16T09:59:30.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d0004d90 0x7f65d0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e8dd6700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 0x7f65d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e95d7700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d0003a30 0x7f65d0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e9dd8700 1 -- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 msgr2=0x7f65d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e415f0f0 0x7f65d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e9dd8700 1 -- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d0003a30 msgr2=0x7f65d0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d0003a30 0x7f65d0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.185+0000 7f65e9dd8700 1 -- 172.21.15.158:0/1746946314 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00035b0 con 0x7f65d0004d90 2023-10-16T09:59:30.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d0004d90 0x7f65d0005210 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f65e4072910 tx=0x7f65d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43098510bfd933a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00eba0 con 0x7f65d0004d90 2023-10-16T09:59:30.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc00f040 con 0x7f65d0004d90 2023-10-16T09:59:30.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65dbfff700 1 -- 172.21.15.158:0/1746946314 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc014480 con 0x7f65d0004d90 2023-10-16T09:59:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65eb83b700 1 -- 172.21.15.158:0/1746946314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d0004d90 msgr2=0x7f65d0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65eb83b700 1 --2- 172.21.15.158:0/1746946314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65d0004d90 0x7f65d0005210 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f65e4072910 tx=0x7f65d0005880 comp rx=0 tx=0).stop 2023-10-16T09:59:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65e9dd8700 1 -- 172.21.15.158:0/1746946314 reap_dead start 2023-10-16T09:59:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65eb83b700 1 -- 172.21.15.158:0/1746946314 shutdown_connections 2023-10-16T09:59:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65eb83b700 1 -- 172.21.15.158:0/1746946314 >> 172.21.15.158:0/1746946314 conn(0x7f65e406a8d0 msgr2=0x7f65e406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65eb83b700 1 -- 172.21.15.158:0/1746946314 shutdown_connections 2023-10-16T09:59:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.186+0000 7f65eb83b700 1 -- 172.21.15.158:0/1746946314 wait complete. 2023-10-16T09:59:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 Processor -- start 2023-10-16T09:59:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 -- start start 2023-10-16T09:59:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c5d90 0x7f65e40ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c8520 0x7f65e40c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65e40c6170 0x7f65e40c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40c9db0 con 0x7f65e40c6170 2023-10-16T09:59:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40c8ee0 con 0x7f65e40c8520 2023-10-16T09:59:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65eb83b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40c91a0 con 0x7f65e40c5d90 2023-10-16T09:59:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65e9dd8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65e40c6170 0x7f65e40c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65e9dd8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65e40c6170 0x7f65e40c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52400/0 (socket says 172.21.15.158:52400) 2023-10-16T09:59:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.188+0000 7f65e9dd8700 1 -- 172.21.15.158:0/2438976944 learned_addr learned my addr 172.21.15.158:0/2438976944 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e95d7700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c5d90 0x7f65e40ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e8dd6700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c8520 0x7f65e40c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e9dd8700 1 -- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c5d90 msgr2=0x7f65e40ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c5d90 0x7f65e40ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e9dd8700 1 -- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c8520 msgr2=0x7f65e40c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c8520 0x7f65e40c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e9dd8700 1 -- 172.21.15.158:0/2438976944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e40c9480 con 0x7f65e40c6170 2023-10-16T09:59:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65e9dd8700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65e40c6170 0x7f65e40c6600 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f65dc015000 tx=0x7f65dc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65d9ffb700 1 -- 172.21.15.158:0/2438976944 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc007000 con 0x7f65e40c6170 2023-10-16T09:59:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65d9ffb700 1 -- 172.21.15.158:0/2438976944 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc014c70 con 0x7f65e40c6170 2023-10-16T09:59:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65d9ffb700 1 -- 172.21.15.158:0/2438976944 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc009410 con 0x7f65e40c6170 2023-10-16T09:59:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e40c9760 con 0x7f65e40c6170 2023-10-16T09:59:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.189+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e40c6b40 con 0x7f65e40c6170 2023-10-16T09:59:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.190+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e40c9f20 con 0x7f65e40c6170 2023-10-16T09:59:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.191+0000 7f65d9ffb700 1 -- 172.21.15.158:0/2438976944 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65dc01e400 con 0x7f65e40c6170 2023-10-16T09:59:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.191+0000 7f65d9ffb700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f65c806b020 0x7f65c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.191+0000 7f65d9ffb700 1 -- 172.21.15.158:0/2438976944 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f65dc08b790 con 0x7f65e40c6170 2023-10-16T09:59:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.191+0000 7f65e95d7700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f65c806b020 0x7f65c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:30.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.192+0000 7f65e95d7700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f65c806b020 0x7f65c806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f65d4000c00 tx=0x7f65d4009040 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:30.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.196+0000 7f65d9ffb700 1 -- 172.21.15.158:0/2438976944 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65dc05a0e0 con 0x7f65e40c6170 2023-10-16T09:59:30.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.433+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f65e40681b0 con 0x7f65e40c6170 2023-10-16T09:59:30.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.433+0000 7f65d9ffb700 1 -- 172.21.15.158:0/2438976944 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7f65dc059c70 con 0x7f65e40c6170 2023-10-16T09:59:30.435 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T09:59:30.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f65c806b020 msgr2=0x7f65c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f65c806b020 0x7f65c806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f65d4000c00 tx=0x7f65d4009040 comp rx=0 tx=0).stop 2023-10-16T09:59:30.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65e40c6170 msgr2=0x7f65e40c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:30.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65e40c6170 0x7f65e40c6600 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f65dc015000 tx=0x7f65dc009750 comp rx=0 tx=0).stop 2023-10-16T09:59:30.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 shutdown_connections 2023-10-16T09:59:30.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f65c806b020 0x7f65c806d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65e40c6170 0x7f65e40c6600 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c5d90 0x7f65e40ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 --2- 172.21.15.158:0/2438976944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c8520 0x7f65e40c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:30.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 >> 172.21.15.158:0/2438976944 conn(0x7f65e406a8d0 msgr2=0x7f65e40b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:30.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 shutdown_connections 2023-10-16T09:59:30.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:30.435+0000 7f65eb83b700 1 -- 172.21.15.158:0/2438976944 wait complete. 2023-10-16T09:59:30.449 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T09:59:30.449 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T09:59:30.449 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T09:59:30.450 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T09:59:30.450 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T09:59:30.450 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T09:59:30.451 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T09:59:30.451 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T09:59:30.451 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:59:30.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:30.452 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T09:59:30.470 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T09:59:30.470 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T09:59:30.470 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T09:59:30.470 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:59:30.471 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T09:59:30.471 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:59:30.471 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:59:30.471 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:59:30.472 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:59:30.472 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T09:59:30.472 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T09:59:30.472 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T09:59:30.472 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T09:59:30.473 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T09:59:30.473 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T09:59:30.473 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 41462sec preferred_lft 41462sec 2023-10-16T09:59:30.473 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T09:59:30.473 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:59:30.474 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T09:59:30.474 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T09:59:30.474 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T09:59:30.474 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:59:30.474 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T09:59:30.475 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:59:30.475 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:59:30.475 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T09:59:30.475 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T09:59:30.475 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:59:30.475 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T09:59:30.476 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T09:59:30.476 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T09:59:30.476 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T09:59:30.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:30.477 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T09:59:30.535 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T09:59:30.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:30.537 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T09:59:30.595 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T09:59:30.595 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T09:59:30.595 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T09:59:30.595 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T09:59:30.595 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T09:59:30.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:30.597 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T09:59:30.597 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T09:59:30.598 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T09:59:30.598 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T09:59:30.598 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T09:59:30.599 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T09:59:30.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:30.701 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T09:59:30.718 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T09:59:30.718 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T09:59:30.718 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:59:30.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:30.719 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T09:59:30.784 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:59:30.784 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T09:59:30.840 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T09:59:30.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T09:59:30.841 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:59:30.896 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T09:59:30.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:30.897 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:59:30.982 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:59:30.982 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:59:31.005 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:59:31.005 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:59:31.005 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:59:31.006 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:59:31.006 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:59:31.006 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:59:31.006 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:59:31.006 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:59:31.080 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T09:59:31.081 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T09:59:31.081 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T09:59:31.081 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T09:59:31.081 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T09:59:31.082 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T09:59:31.082 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T09:59:31.082 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T09:59:31.082 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T09:59:31.082 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T09:59:31.083 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T09:59:31.084 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T09:59:31.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:31.084 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T09:59:31.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:31.112 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T09:59:31.200 INFO:teuthology.orchestra.run.smithi158.stdout:client35723 2023-10-16T09:59:31.200 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:31.200 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:31.217 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T09:59:31.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.377+0000 7fc76fb29700 1 Processor -- start 2023-10-16T09:59:31.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.378+0000 7fc76fb29700 1 -- start start 2023-10-16T09:59:31.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.378+0000 7fc76fb29700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc76815d990 0x7fc7680b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.378+0000 7fc76fb29700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc76806e480 0x7fc7680b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.378+0000 7fc76fb29700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7680af600 con 0x7fc7680b3760 2023-10-16T09:59:31.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.378+0000 7fc76fb29700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680b2c90 con 0x7fc76806e480 2023-10-16T09:59:31.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.378+0000 7fc76fb29700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc76806e9b0 con 0x7fc76815d990 2023-10-16T09:59:31.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc76806e480 0x7fc7680b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc76806e480 0x7fc7680b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52412/0 (socket says 172.21.15.158:52412) 2023-10-16T09:59:31.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 -- 172.21.15.158:0/3035189450 learned_addr learned my addr 172.21.15.158:0/3035189450 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:31.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d8c5700 1 --2- 172.21.15.158:0/3035189450 >> v2:172.21.15.172:3300/0 conn(0x7fc76815d990 0x7fc7680b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 -- 172.21.15.158:0/3035189450 >> v2:172.21.15.172:3300/0 conn(0x7fc76815d990 msgr2=0x7fc7680b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:31.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 --2- 172.21.15.158:0/3035189450 >> v2:172.21.15.172:3300/0 conn(0x7fc76815d990 0x7fc7680b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 -- 172.21.15.158:0/3035189450 >> v1:172.21.15.172:6789/0 conn(0x7fc7680b3760 legacy=0x7fc7680cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:31.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 -- 172.21.15.158:0/3035189450 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680ce890 con 0x7fc76806e480 2023-10-16T09:59:31.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76d0c4700 1 --2- 172.21.15.158:0/3035189450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc7680b3220 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc7680b2fc0 tx=0x7fc764009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7663071b6544e221 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:31.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76c8c3700 1 -- 172.21.15.158:0/3035189450 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764004230 con 0x7fc76806e480 2023-10-16T09:59:31.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76c8c3700 1 -- 172.21.15.158:0/3035189450 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc764004390 con 0x7fc76806e480 2023-10-16T09:59:31.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.379+0000 7fc76c8c3700 1 -- 172.21.15.158:0/3035189450 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7640036a0 con 0x7fc76806e480 2023-10-16T09:59:31.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 -- 172.21.15.158:0/3035189450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 msgr2=0x7fc7680b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 --2- 172.21.15.158:0/3035189450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc7680b3220 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc7680b2fc0 tx=0x7fc764009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 -- 172.21.15.158:0/3035189450 shutdown_connections 2023-10-16T09:59:31.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 --2- 172.21.15.158:0/3035189450 >> v2:172.21.15.172:3300/0 conn(0x7fc76815d990 0x7fc7680b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 --2- 172.21.15.158:0/3035189450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc7680b3220 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 -- 172.21.15.158:0/3035189450 >> 172.21.15.158:0/3035189450 conn(0x7fc76806b9e0 msgr2=0x7fc76806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:31.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 -- 172.21.15.158:0/3035189450 shutdown_connections 2023-10-16T09:59:31.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.380+0000 7fc76fb29700 1 -- 172.21.15.158:0/3035189450 wait complete. 2023-10-16T09:59:31.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.381+0000 7fc76fb29700 1 Processor -- start 2023-10-16T09:59:31.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.381+0000 7fc76fb29700 1 -- start start 2023-10-16T09:59:31.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76fb29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc768160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76fb29700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7680b3760 0x7fc76814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76fb29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc76815d990 0x7fc7680c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76fb29700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc768160f40 con 0x7fc76806e480 2023-10-16T09:59:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76fb29700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76815ff40 con 0x7fc7680b3760 2023-10-16T09:59:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76fb29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680cd570 con 0x7fc76815d990 2023-10-16T09:59:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc768160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc768160a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52414/0 (socket says 172.21.15.158:52414) 2023-10-16T09:59:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 -- 172.21.15.158:0/50250545 learned_addr learned my addr 172.21.15.158:0/50250545 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d0c4700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7680b3760 0x7fc76814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76e0c6700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc76815d990 0x7fc7680c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 -- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc76815d990 msgr2=0x7fc7680c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc76815d990 0x7fc7680c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 -- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7680b3760 msgr2=0x7fc76814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7680b3760 0x7fc76814b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 -- 172.21.15.158:0/50250545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764009550 con 0x7fc76806e480 2023-10-16T09:59:31.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76d8c5700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc768160a00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc75800c7f0 tx=0x7fc75800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:31.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc75e7fc700 1 -- 172.21.15.158:0/50250545 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758011800 con 0x7fc76806e480 2023-10-16T09:59:31.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc75e7fc700 1 -- 172.21.15.158:0/50250545 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc758011960 con 0x7fc76806e480 2023-10-16T09:59:31.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.382+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7680cd850 con 0x7fc76806e480 2023-10-16T09:59:31.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.383+0000 7fc75e7fc700 1 -- 172.21.15.158:0/50250545 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758011cf0 con 0x7fc76806e480 2023-10-16T09:59:31.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.383+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7681604e0 con 0x7fc76806e480 2023-10-16T09:59:31.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.384+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc768160700 con 0x7fc76806e480 2023-10-16T09:59:31.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.384+0000 7fc75e7fc700 1 -- 172.21.15.158:0/50250545 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc758018360 con 0x7fc76806e480 2023-10-16T09:59:31.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.384+0000 7fc75e7fc700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.385+0000 7fc75e7fc700 1 -- 172.21.15.158:0/50250545 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fc758089ae0 con 0x7fc76806e480 2023-10-16T09:59:31.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.385+0000 7fc76d0c4700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.385+0000 7fc76d0c4700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc74c06b020 0x7fc74c06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc7680b2fc0 tx=0x7fc764008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:31.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.389+0000 7fc75e7fc700 1 -- 172.21.15.158:0/50250545 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7580545e0 con 0x7fc76806e480 2023-10-16T09:59:31.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.612+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 --> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7680cee50 con 0x7fc74c06b020 2023-10-16T09:59:31.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.613+0000 7fc75e7fc700 1 -- 172.21.15.158:0/50250545 <== mgr.35456 v2:172.21.15.158:6832/3717499045 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc764020070 con 0x7fc74c06b020 2023-10-16T09:59:31.615 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:59:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T09:59:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:31.615 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:59:31.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.615+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc74c06b020 msgr2=0x7fc74c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.615+0000 7fc76fb29700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc74c06b020 0x7fc74c06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc7680b2fc0 tx=0x7fc764008030 comp rx=0 tx=0).stop 2023-10-16T09:59:31.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.615+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 msgr2=0x7fc768160a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.615+0000 7fc76fb29700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc768160a00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc75800c7f0 tx=0x7fc75800cbb0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 shutdown_connections 2023-10-16T09:59:31.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc76815d990 0x7fc7680c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76806e480 0x7fc768160a00 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 --2- 172.21.15.158:0/50250545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7680b3760 0x7fc76814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 >> 172.21.15.158:0/50250545 conn(0x7fc76806b9e0 msgr2=0x7fc7680acfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:31.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 shutdown_connections 2023-10-16T09:59:31.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.616+0000 7fc76fb29700 1 -- 172.21.15.158:0/50250545 wait complete. 2023-10-16T09:59:31.629 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T09:59:31.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.750+0000 7f4efaa29700 1 Processor -- start 2023-10-16T09:59:31.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.750+0000 7f4efaa29700 1 -- start start 2023-10-16T09:59:31.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4efaa29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415b590 0x7f4ef40ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4efaa29700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ef415f1d0 0x7f4ef40b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4efaa29700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ef40cbaf0 0x7f4ef40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4efaa29700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40cc4c0 con 0x7f4ef415f1d0 2023-10-16T09:59:31.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4efaa29700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40cf4c0 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4efaa29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef415f970 con 0x7f4ef415b590 2023-10-16T09:59:31.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ef415f1d0 0x7f4ef40b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ef415f1d0 0x7f4ef40b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52424/0 (socket says 172.21.15.158:52424) 2023-10-16T09:59:31.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 -- 172.21.15.158:0/1373655863 learned_addr learned my addr 172.21.15.158:0/1373655863 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:31.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef8fc6700 1 --2- 172.21.15.158:0/1373655863 >> v2:172.21.15.172:3300/0 conn(0x7f4ef40cbaf0 0x7f4ef40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef3fff700 1 --2- 172.21.15.158:0/1373655863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415b590 0x7f4ef40ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 -- 172.21.15.158:0/1373655863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415b590 msgr2=0x7f4ef40ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:31.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 --2- 172.21.15.158:0/1373655863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415b590 0x7f4ef40ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 -- 172.21.15.158:0/1373655863 >> v2:172.21.15.172:3300/0 conn(0x7f4ef40cbaf0 msgr2=0x7f4ef40cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:31.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 --2- 172.21.15.158:0/1373655863 >> v2:172.21.15.172:3300/0 conn(0x7f4ef40cbaf0 0x7f4ef40cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 -- 172.21.15.158:0/1373655863 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef415fc50 con 0x7f4ef415f1d0 2023-10-16T09:59:31.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.751+0000 7f4ef37fe700 1 --2- 172.21.15.158:0/1373655863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef415f1d0 0x7f4ef40b42d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40b3640 tx=0x7f4ee8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5cfd980ca6457ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:31.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4ef2ffd700 1 -- 172.21.15.158:0/1373655863 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8011830 con 0x7f4ef415f1d0 2023-10-16T09:59:31.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4ef2ffd700 1 -- 172.21.15.158:0/1373655863 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee8011990 con 0x7f4ef415f1d0 2023-10-16T09:59:31.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4ef2ffd700 1 -- 172.21.15.158:0/1373655863 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8011d20 con 0x7f4ef415f1d0 2023-10-16T09:59:31.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 -- 172.21.15.158:0/1373655863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef415f1d0 msgr2=0x7f4ef40b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 --2- 172.21.15.158:0/1373655863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef415f1d0 0x7f4ef40b42d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40b3640 tx=0x7f4ee8009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 -- 172.21.15.158:0/1373655863 shutdown_connections 2023-10-16T09:59:31.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 --2- 172.21.15.158:0/1373655863 >> v2:172.21.15.172:3300/0 conn(0x7f4ef40cbaf0 0x7f4ef40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 --2- 172.21.15.158:0/1373655863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415b590 0x7f4ef40ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 --2- 172.21.15.158:0/1373655863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef415f1d0 0x7f4ef40b42d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 -- 172.21.15.158:0/1373655863 >> 172.21.15.158:0/1373655863 conn(0x7f4ef406a8d0 msgr2=0x7f4ef406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:31.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 -- 172.21.15.158:0/1373655863 shutdown_connections 2023-10-16T09:59:31.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.752+0000 7f4efaa29700 1 -- 172.21.15.158:0/1373655863 wait complete. 2023-10-16T09:59:31.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.753+0000 7f4efaa29700 1 Processor -- start 2023-10-16T09:59:31.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4efaa29700 1 -- start start 2023-10-16T09:59:31.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4efaa29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef40cbaf0 0x7f4ef415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4efaa29700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef415b590 0x7f4ef415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4efaa29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415f1d0 0x7f4ef415c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4efaa29700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40cb1d0 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4efaa29700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40ca5c0 con 0x7f4ef415b590 2023-10-16T09:59:31.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4efaa29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40ca8a0 con 0x7f4ef415f1d0 2023-10-16T09:59:31.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4ef3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef40cbaf0 0x7f4ef415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4ef3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef40cbaf0 0x7f4ef415d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52428/0 (socket says 172.21.15.158:52428) 2023-10-16T09:59:31.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.754+0000 7f4ef3fff700 1 -- 172.21.15.158:0/3060378062 learned_addr learned my addr 172.21.15.158:0/3060378062 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:31.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef3fff700 1 -- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415f1d0 msgr2=0x7f4ef415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef8fc6700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415f1d0 0x7f4ef415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef37fe700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef415b590 0x7f4ef415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef3fff700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415f1d0 0x7f4ef415c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef3fff700 1 -- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef415b590 msgr2=0x7f4ef415bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:31.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef3fff700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef415b590 0x7f4ef415bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:31.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef3fff700 1 -- 172.21.15.158:0/3060378062 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8009550 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef3fff700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef40cbaf0 0x7f4ef415d020 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e9b0 tx=0x7f4ee400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:31.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef0ff9700 1 -- 172.21.15.158:0/3060378062 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4009710 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef0ff9700 1 -- 172.21.15.158:0/3060378062 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee400c850 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef40cab80 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4ef0ff9700 1 -- 172.21.15.158:0/3060378062 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee400cbc0 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.755+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef40c79b0 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.756+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef40681b0 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.757+0000 7f4ef0ff9700 1 -- 172.21.15.158:0/3060378062 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4ee400cd20 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.757+0000 7f4ef0ff9700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f4ee006b020 0x7f4ee006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:31.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.757+0000 7f4ef0ff9700 1 -- 172.21.15.158:0/3060378062 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f4ee4089b40 con 0x7f4ef40cbaf0 2023-10-16T09:59:31.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.757+0000 7f4ef37fe700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f4ee006b020 0x7f4ee006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:31.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.757+0000 7f4ef37fe700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f4ee006b020 0x7f4ee006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ef414a370 tx=0x7f4ee8008030 comp rx=0 tx=0).ready entity=mgr.35456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:31.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:31.762+0000 7f4ef0ff9700 1 -- 172.21.15.158:0/3060378062 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ee40547f0 con 0x7f4ef40cbaf0 2023-10-16T09:59:32.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.006+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4ef40b07c0 con 0x7f4ef40cbaf0 2023-10-16T09:59:32.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.561+0000 7f4ef0ff9700 1 -- 172.21.15.158:0/3060378062 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v33) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4ee40210a0 con 0x7f4ef40cbaf0 2023-10-16T09:59:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.564+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f4ee006b020 msgr2=0x7f4ee006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.564+0000 7f4efaa29700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f4ee006b020 0x7f4ee006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ef414a370 tx=0x7f4ee8008030 comp rx=0 tx=0).stop 2023-10-16T09:59:32.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.564+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef40cbaf0 msgr2=0x7f4ef415d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:32.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.564+0000 7f4efaa29700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef40cbaf0 0x7f4ef415d020 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e9b0 tx=0x7f4ee400ed70 comp rx=0 tx=0).stop 2023-10-16T09:59:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.565+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 shutdown_connections 2023-10-16T09:59:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.565+0000 7f4efaa29700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f4ee006b020 0x7f4ee006d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.565+0000 7f4efaa29700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef415f1d0 0x7f4ef415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.565+0000 7f4efaa29700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef40cbaf0 0x7f4ef415d020 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.565+0000 7f4efaa29700 1 --2- 172.21.15.158:0/3060378062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef415b590 0x7f4ef415bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.565+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 >> 172.21.15.158:0/3060378062 conn(0x7f4ef406a8d0 msgr2=0x7f4ef40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.567+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 shutdown_connections 2023-10-16T09:59:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.567+0000 7f4efaa29700 1 -- 172.21.15.158:0/3060378062 wait complete. 2023-10-16T09:59:32.582 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T09:59:32.707 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:32.707+0000 7f15bab57200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:59:32.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.711+0000 7fe259f94700 1 Processor -- start 2023-10-16T09:59:32.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.711+0000 7fe259f94700 1 -- start start 2023-10-16T09:59:32.713 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:32.711+0000 7fea12f8b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:59:32.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.711+0000 7fe259f94700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25406e0d0 0x7fe25415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:32.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe259f94700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe2540cbcf0 0x7fe2540cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe259f94700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25415f380 con 0x7fe25415dc40 2023-10-16T09:59:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe259f94700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe254071e70 con 0x7fe2540cbcf0 2023-10-16T09:59:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe259f94700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe254072150 con 0x7fe25406e0d0 2023-10-16T09:59:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25406e0d0 0x7fe25415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe253fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe2540cbcf0 0x7fe2540cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25406e0d0 0x7fe25415ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:35066/0 (socket says 172.21.15.158:35066) 2023-10-16T09:59:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe2537fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe25415dc40 0x7fe25406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:56542/0 (socket says 172.21.15.158:56542) 2023-10-16T09:59:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 -- 172.21.15.158:0/2518725687 learned_addr learned my addr 172.21.15.158:0/2518725687 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 -- 172.21.15.158:0/2518725687 >> v2:172.21.15.172:3300/0 conn(0x7fe2540cbcf0 msgr2=0x7fe2540cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 --2- 172.21.15.158:0/2518725687 >> v2:172.21.15.172:3300/0 conn(0x7fe2540cbcf0 0x7fe2540cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 -- 172.21.15.158:0/2518725687 >> v1:172.21.15.172:6789/0 conn(0x7fe25415dc40 legacy=0x7fe25406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:32.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 -- 172.21.15.158:0/2518725687 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe254072430 con 0x7fe25406e0d0 2023-10-16T09:59:32.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe253fff700 1 --2- 172.21.15.158:0/2518725687 >> v2:172.21.15.172:3300/0 conn(0x7fe2540cbcf0 0x7fe2540cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:32.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe252ffd700 1 --2- 172.21.15.158:0/2518725687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25406e0d0 0x7fe25415ee40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe2480098f0 tx=0x7fe248009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=112f45b328551205 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:32.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.712+0000 7fe2527fc700 1 -- 172.21.15.158:0/2518725687 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248011830 con 0x7fe25406e0d0 2023-10-16T09:59:32.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe2527fc700 1 -- 172.21.15.158:0/2518725687 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe248011990 con 0x7fe25406e0d0 2023-10-16T09:59:32.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe2527fc700 1 -- 172.21.15.158:0/2518725687 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248011d20 con 0x7fe25406e0d0 2023-10-16T09:59:32.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 -- 172.21.15.158:0/2518725687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25406e0d0 msgr2=0x7fe25415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:32.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 --2- 172.21.15.158:0/2518725687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25406e0d0 0x7fe25415ee40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe2480098f0 tx=0x7fe248009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:32.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 -- 172.21.15.158:0/2518725687 shutdown_connections 2023-10-16T09:59:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 --2- 172.21.15.158:0/2518725687 >> v2:172.21.15.172:3300/0 conn(0x7fe2540cbcf0 0x7fe2540cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 --2- 172.21.15.158:0/2518725687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25406e0d0 0x7fe25415ee40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 -- 172.21.15.158:0/2518725687 >> 172.21.15.158:0/2518725687 conn(0x7fe2540d03b0 msgr2=0x7fe2540d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 -- 172.21.15.158:0/2518725687 shutdown_connections 2023-10-16T09:59:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.713+0000 7fe259f94700 1 -- 172.21.15.158:0/2518725687 wait complete. 2023-10-16T09:59:32.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.714+0000 7fe259f94700 1 Processor -- start 2023-10-16T09:59:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.715+0000 7fe259f94700 1 -- start start 2023-10-16T09:59:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.715+0000 7fe259f94700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe25406e0d0 0x7fe2540c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.715+0000 7fe259f94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2540cbcf0 0x7fe2540b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.715+0000 7fe259f94700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25415dc40 0x7fe254155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.715+0000 7fe259f94700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2540c6920 con 0x7fe2540cbcf0 2023-10-16T09:59:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.715+0000 7fe259f94700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2540c6c00 con 0x7fe25406e0d0 2023-10-16T09:59:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.715+0000 7fe259f94700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2540c3ee0 con 0x7fe25415dc40 2023-10-16T09:59:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe252ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2540cbcf0 0x7fe2540b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe253fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25415dc40 0x7fe254155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe2537fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe25406e0d0 0x7fe2540c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe253fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25415dc40 0x7fe254155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:35074/0 (socket says 172.21.15.158:35074) 2023-10-16T09:59:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe253fff700 1 -- 172.21.15.158:0/3726594459 learned_addr learned my addr 172.21.15.158:0/3726594459 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe252ffd700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2540cbcf0 0x7fe2540b53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52470/0 (socket says 172.21.15.158:52470) 2023-10-16T09:59:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe2537fe700 1 -- 172.21.15.158:0/3726594459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25415dc40 msgr2=0x7fe254155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe2537fe700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25415dc40 0x7fe254155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe2537fe700 1 -- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2540cbcf0 msgr2=0x7fe2540b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe2537fe700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2540cbcf0 0x7fe2540b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe2537fe700 1 -- 172.21.15.158:0/3726594459 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c009a70 con 0x7fe25406e0d0 2023-10-16T09:59:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe252ffd700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2540cbcf0 0x7fe2540b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe2537fe700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe25406e0d0 0x7fe2540c6260 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c3c0 tx=0x7fe24400c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe23bfff700 1 -- 172.21.15.158:0/3726594459 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244004a70 con 0x7fe25406e0d0 2023-10-16T09:59:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe23bfff700 1 -- 172.21.15.158:0/3726594459 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe244004bd0 con 0x7fe25406e0d0 2023-10-16T09:59:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe23bfff700 1 -- 172.21.15.158:0/3726594459 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2440070d0 con 0x7fe25406e0d0 2023-10-16T09:59:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe248009550 con 0x7fe25406e0d0 2023-10-16T09:59:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.716+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2540c44a0 con 0x7fe25406e0d0 2023-10-16T09:59:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.717+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2540681b0 con 0x7fe25406e0d0 2023-10-16T09:59:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.718+0000 7fe23bfff700 1 -- 172.21.15.158:0/3726594459 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 33) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe244007230 con 0x7fe25406e0d0 2023-10-16T09:59:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.718+0000 7fe23bfff700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 0x7fe24006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.718+0000 7fe252ffd700 1 -- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 msgr2=0x7fe24006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.718+0000 7fe23bfff700 1 -- 172.21.15.158:0/3726594459 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fe2440892e0 con 0x7fe25406e0d0 2023-10-16T09:59:32.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.718+0000 7fe252ffd700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 0x7fe24006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:32.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.726+0000 7fe23bfff700 1 -- 172.21.15.158:0/3726594459 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe244053f90 con 0x7fe25406e0d0 2023-10-16T09:59:32.777 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:32.776+0000 7f15bab57200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:32.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:32.780+0000 7fea12f8b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:32.849 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:32.849+0000 7f15bab57200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:59:32.855 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:32.854+0000 7fea12f8b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:59:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.919+0000 7fe252ffd700 1 -- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 msgr2=0x7fe24006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:32.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.919+0000 7fe252ffd700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 0x7fe24006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:32.926 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:32.926+0000 7f15bab57200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:59:32.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.927+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe25415f380 con 0x7fe25406e0d0 2023-10-16T09:59:32.934 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:32.933+0000 7fea12f8b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:59:32.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.934+0000 7fe23bfff700 1 -- 172.21.15.158:0/3726594459 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe25415f380 con 0x7fe25406e0d0 2023-10-16T09:59:32.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 msgr2=0x7fe24006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:32.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 0x7fe24006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe25406e0d0 msgr2=0x7fe2540c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:32.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe25406e0d0 0x7fe2540c6260 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c3c0 tx=0x7fe24400c6d0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 shutdown_connections 2023-10-16T09:59:32.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7fe24006b070 0x7fe24006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe25415dc40 0x7fe254155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe25406e0d0 0x7fe2540c6260 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 --2- 172.21.15.158:0/3726594459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2540cbcf0 0x7fe2540b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:32.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 >> 172.21.15.158:0/3726594459 conn(0x7fe2540d03b0 msgr2=0x7fe2540cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:32.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 shutdown_connections 2023-10-16T09:59:32.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:32.937+0000 7fe259f94700 1 -- 172.21.15.158:0/3726594459 wait complete. 2023-10-16T09:59:32.951 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T09:59:33.067 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:33.067+0000 7f15bab57200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:59:33.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.075+0000 7f0df688e700 1 Processor -- start 2023-10-16T09:59:33.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.075+0000 7f0df688e700 1 -- start start 2023-10-16T09:59:33.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df688e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df00b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df688e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0df00b3460 0x7f0df00b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df688e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df00b4420 con 0x7f0df015b6b0 2023-10-16T09:59:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df688e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df015ba90 con 0x7f0df00b3460 2023-10-16T09:59:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df688e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00b38f0 con 0x7f0df00d0c60 2023-10-16T09:59:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df4e2b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0df00b3460 0x7f0df00b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df4e2b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0df00b3460 0x7f0df00b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52526/0 (socket says 172.21.15.158:52526) 2023-10-16T09:59:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2446335039 learned_addr learned my addr 172.21.15.158:0/2446335039 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.076+0000 7f0def7fe700 1 --2- 172.21.15.158:0/2446335039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df00b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2446335039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 msgr2=0x7f0df00b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0deeffd700 1 -- 172.21.15.158:0/2446335039 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350368108 0 0) 0x7f0df00b4420 con 0x7f0df015b6b0 2023-10-16T09:59:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0df4e2b700 1 --2- 172.21.15.158:0/2446335039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df00b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2446335039 >> v1:172.21.15.172:6789/0 conn(0x7f0df015b6b0 legacy=0x7f0df00cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2446335039 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df00cbc20 con 0x7f0df00b3460 2023-10-16T09:59:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0df4e2b700 1 --2- 172.21.15.158:0/2446335039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df00b3460 0x7f0df00b48b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009e50 tx=0x7f0de800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c1fa67e535edee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0deeffd700 1 -- 172.21.15.158:0/2446335039 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de800e070 con 0x7f0df00b3460 2023-10-16T09:59:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0deeffd700 1 -- 172.21.15.158:0/2446335039 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de800b910 con 0x7f0df00b3460 2023-10-16T09:59:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.077+0000 7f0deeffd700 1 -- 172.21.15.158:0/2446335039 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de800bcc0 con 0x7f0df00b3460 2023-10-16T09:59:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 -- 172.21.15.158:0/2446335039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df00b3460 msgr2=0x7f0df00b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 --2- 172.21.15.158:0/2446335039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df00b3460 0x7f0df00b48b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009e50 tx=0x7f0de800b330 comp rx=0 tx=0).stop 2023-10-16T09:59:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 -- 172.21.15.158:0/2446335039 shutdown_connections 2023-10-16T09:59:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 --2- 172.21.15.158:0/2446335039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df00b3460 0x7f0df00b48b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 --2- 172.21.15.158:0/2446335039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df00b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 -- 172.21.15.158:0/2446335039 >> 172.21.15.158:0/2446335039 conn(0x7f0df00cdf20 msgr2=0x7f0df00ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 -- 172.21.15.158:0/2446335039 shutdown_connections 2023-10-16T09:59:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.078+0000 7f0df688e700 1 -- 172.21.15.158:0/2446335039 wait complete. 2023-10-16T09:59:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.079+0000 7f0df688e700 1 Processor -- start 2023-10-16T09:59:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.079+0000 7f0df688e700 1 -- start start 2023-10-16T09:59:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df688e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0df00b3460 0x7f0df00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df688e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df688e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df015b6b0 0x7f0df015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df688e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00c95c0 con 0x7f0df015b6b0 2023-10-16T09:59:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df688e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00c78e0 con 0x7f0df00b3460 2023-10-16T09:59:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df688e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df01f6a20 con 0x7f0df00d0c60 2023-10-16T09:59:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df015b6b0 0x7f0df015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df015b6b0 0x7f0df015c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52532/0 (socket says 172.21.15.158:52532) 2023-10-16T09:59:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2768302050 learned_addr learned my addr 172.21.15.158:0/2768302050 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0def7fe700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0deffff700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0df00b3460 0x7f0df00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 msgr2=0x7f0df015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0df00b3460 msgr2=0x7f0df00c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0df00b3460 0x7f0df00c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.080+0000 7f0df4e2b700 1 -- 172.21.15.158:0/2768302050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8009b30 con 0x7f0df015b6b0 2023-10-16T09:59:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.081+0000 7f0df4e2b700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df015b6b0 0x7f0df015c9b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f0de800b680 tx=0x7f0de8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.081+0000 7f0decff9700 1 -- 172.21.15.158:0/2768302050 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80048b0 con 0x7f0df015b6b0 2023-10-16T09:59:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.081+0000 7f0decff9700 1 -- 172.21.15.158:0/2768302050 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80099b0 con 0x7f0df015b6b0 2023-10-16T09:59:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.081+0000 7f0decff9700 1 -- 172.21.15.158:0/2768302050 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de801c680 con 0x7f0df015b6b0 2023-10-16T09:59:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.081+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df00c97d0 con 0x7f0df015b6b0 2023-10-16T09:59:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.081+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df00c9d90 con 0x7f0df015b6b0 2023-10-16T09:59:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.082+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df00681b0 con 0x7f0df015b6b0 2023-10-16T09:59:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.083+0000 7f0decff9700 1 -- 172.21.15.158:0/2768302050 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 33) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0de801c7e0 con 0x7f0df015b6b0 2023-10-16T09:59:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.083+0000 7f0decff9700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 0x7f0ddc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.083+0000 7f0deffff700 1 -- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 msgr2=0x7f0ddc06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.083+0000 7f0decff9700 1 -- 172.21.15.158:0/2768302050 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f0de808b790 con 0x7f0df015b6b0 2023-10-16T09:59:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.083+0000 7f0deffff700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 0x7f0ddc06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:33.093 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:33.084+0000 7fea12f8b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:59:33.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.087+0000 7f0decff9700 1 -- 172.21.15.158:0/2768302050 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0de80564a0 con 0x7f0df015b6b0 2023-10-16T09:59:33.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.284+0000 7f0deffff700 1 -- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 msgr2=0x7f0ddc06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:33.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.284+0000 7f0deffff700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 0x7f0ddc06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:33.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.292+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0df015acb0 con 0x7f0df015b6b0 2023-10-16T09:59:33.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.298+0000 7f0decff9700 1 -- 172.21.15.158:0/2768302050 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0de8059ac0 con 0x7f0df015b6b0 2023-10-16T09:59:33.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.300+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 msgr2=0x7f0ddc06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:33.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.300+0000 7f0df688e700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 0x7f0ddc06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.300+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df015b6b0 msgr2=0x7f0df015c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.300+0000 7f0df688e700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df015b6b0 0x7f0df015c9b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f0de800b680 tx=0x7f0de8004ad0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.300+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 shutdown_connections 2023-10-16T09:59:33.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.301+0000 7f0df688e700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f0ddc06b070 0x7f0ddc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.301+0000 7f0df688e700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0df015b6b0 0x7f0df015c9b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.301+0000 7f0df688e700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0df00b3460 0x7f0df00c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.301+0000 7f0df688e700 1 --2- 172.21.15.158:0/2768302050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0df00d0c60 0x7f0df015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.301+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 >> 172.21.15.158:0/2768302050 conn(0x7f0df00cdf20 msgr2=0x7f0df00ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:33.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.301+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 shutdown_connections 2023-10-16T09:59:33.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.301+0000 7f0df688e700 1 -- 172.21.15.158:0/2768302050 wait complete. 2023-10-16T09:59:33.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:33.315 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T09:59:33.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T09:59:33.393 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:33.393+0000 7f15bab57200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:59:33.429 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:33.428+0000 7fea12f8b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:59:33.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.500+0000 7f5368988700 1 Processor -- start 2023-10-16T09:59:33.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.500+0000 7f5368988700 1 -- start start 2023-10-16T09:59:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f5368988700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5364160d30 0x7f53640b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f5368988700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f53640b7590 0x7f53640d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f5368988700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53640b7a20 con 0x7f53640d0530 2023-10-16T09:59:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f5368988700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53640b8910 con 0x7f5364160d30 2023-10-16T09:59:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f5368988700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53640d0ef0 con 0x7f53640b7590 2023-10-16T09:59:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f536259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5364160d30 0x7f53640b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f536259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5364160d30 0x7f53640b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52582/0 (socket says 172.21.15.158:52582) 2023-10-16T09:59:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f536259c700 1 -- 172.21.15.158:0/3972769432 learned_addr learned my addr 172.21.15.158:0/3972769432 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f5362d9d700 1 --2- 172.21.15.158:0/3972769432 >> v2:172.21.15.172:3300/0 conn(0x7f53640b7590 0x7f53640d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f536259c700 1 -- 172.21.15.158:0/3972769432 >> v2:172.21.15.172:3300/0 conn(0x7f53640b7590 msgr2=0x7f53640d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:33.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f536259c700 1 --2- 172.21.15.158:0/3972769432 >> v2:172.21.15.172:3300/0 conn(0x7f53640b7590 0x7f53640d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f536259c700 1 -- 172.21.15.158:0/3972769432 >> v1:172.21.15.172:6789/0 conn(0x7f53640d0530 legacy=0x7f53640d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.501+0000 7f536259c700 1 -- 172.21.15.158:0/3972769432 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53640b7c90 con 0x7f5364160d30 2023-10-16T09:59:33.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.502+0000 7f536259c700 1 --2- 172.21.15.158:0/3972769432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5364160d30 0x7f53640b8f50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f53540098f0 tx=0x7f5354009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd41c799de5cd4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:33.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.502+0000 7f536159a700 1 -- 172.21.15.158:0/3972769432 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354011830 con 0x7f5364160d30 2023-10-16T09:59:33.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.502+0000 7f536159a700 1 -- 172.21.15.158:0/3972769432 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5354011990 con 0x7f5364160d30 2023-10-16T09:59:33.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.502+0000 7f536159a700 1 -- 172.21.15.158:0/3972769432 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354011d20 con 0x7f5364160d30 2023-10-16T09:59:33.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.502+0000 7f5368988700 1 -- 172.21.15.158:0/3972769432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5364160d30 msgr2=0x7f53640b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.502+0000 7f5368988700 1 --2- 172.21.15.158:0/3972769432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5364160d30 0x7f53640b8f50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f53540098f0 tx=0x7f5354009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:33.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.503+0000 7f5368988700 1 -- 172.21.15.158:0/3972769432 shutdown_connections 2023-10-16T09:59:33.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.503+0000 7f5368988700 1 --2- 172.21.15.158:0/3972769432 >> v2:172.21.15.172:3300/0 conn(0x7f53640b7590 0x7f53640d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.503+0000 7f5368988700 1 --2- 172.21.15.158:0/3972769432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5364160d30 0x7f53640b8f50 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.503+0000 7f5368988700 1 -- 172.21.15.158:0/3972769432 >> 172.21.15.158:0/3972769432 conn(0x7f53640669f0 msgr2=0x7f5364066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:33.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.503+0000 7f5368988700 1 -- 172.21.15.158:0/3972769432 shutdown_connections 2023-10-16T09:59:33.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.503+0000 7f5368988700 1 -- 172.21.15.158:0/3972769432 wait complete. 2023-10-16T09:59:33.509 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:33.507+0000 7f15bab57200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:59:33.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.504+0000 7f5368988700 1 Processor -- start 2023-10-16T09:59:33.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.504+0000 7f5368988700 1 -- start start 2023-10-16T09:59:33.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5368988700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53640b7590 0x7f53641f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5368988700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53640d0530 0x7f5364159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5368988700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5364160d30 0x7f5364159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5368988700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53641473e0 con 0x7f53640b7590 2023-10-16T09:59:33.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5368988700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536415d440 con 0x7f53640d0530 2023-10-16T09:59:33.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5368988700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5364159e20 con 0x7f5364160d30 2023-10-16T09:59:33.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53640b7590 0x7f53641f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53640b7590 0x7f53641f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52588/0 (socket says 172.21.15.158:52588) 2023-10-16T09:59:33.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 -- 172.21.15.158:0/3544460126 learned_addr learned my addr 172.21.15.158:0/3544460126 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:33.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5362d9d700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5364160d30 0x7f5364159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53640d0530 0x7f5364159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:33.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5364160d30 msgr2=0x7f5364159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5364160d30 0x7f5364159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53640d0530 msgr2=0x7f5364159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:33.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53640d0530 0x7f5364159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:33.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 -- 172.21.15.158:0/3544460126 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5354009550 con 0x7f53640b7590 2023-10-16T09:59:33.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.505+0000 7f536259c700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53640b7590 0x7f53641f2740 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f5354009c40 tx=0x7f5354004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:33.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.506+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354004510 con 0x7f53640b7590 2023-10-16T09:59:33.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.506+0000 7f5368988700 1 -- 172.21.15.158:0/3544460126 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536415a100 con 0x7f53640b7590 2023-10-16T09:59:33.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.506+0000 7f5368988700 1 -- 172.21.15.158:0/3544460126 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5364147910 con 0x7f53640b7590 2023-10-16T09:59:33.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.506+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535401e070 con 0x7f53640b7590 2023-10-16T09:59:33.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.506+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53540038a0 con 0x7f53640b7590 2023-10-16T09:59:33.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.507+0000 7f5368988700 1 -- 172.21.15.158:0/3544460126 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53480052f0 con 0x7f53640b7590 2023-10-16T09:59:33.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.507+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 33) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5354003a00 con 0x7f53640b7590 2023-10-16T09:59:33.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.507+0000 7f5352ffd700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 0x7f534406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:33.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.507+0000 7f5361d9b700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 msgr2=0x7f534406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:33.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.507+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f535408a9c0 con 0x7f53640b7590 2023-10-16T09:59:33.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.507+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 0x7f534406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:33.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.515+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53540556a0 con 0x7f53640b7590 2023-10-16T09:59:33.553 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:33.552+0000 7fea12f8b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:59:33.582 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:33.581+0000 7f15bab57200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:59:33.630 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:33.629+0000 7fea12f8b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:59:33.662 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:33.661+0000 7f15bab57200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:59:33.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.708+0000 7f5361d9b700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 msgr2=0x7f534406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:33.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.708+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 0x7f534406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:33.713 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:33.712+0000 7fea12f8b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:59:33.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:33.741+0000 7f5368988700 1 -- 172.21.15.158:0/3544460126 --> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5348000bc0 con 0x7f534406b070 2023-10-16T09:59:34.028 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:34.027+0000 7f15bab57200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:59:34.086 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:34.085+0000 7fea12f8b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:59:34.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:34.109+0000 7f5361d9b700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 msgr2=0x7f534406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:34.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:34.109+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 0x7f534406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T09:59:34.593 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:34.593+0000 7f15bab57200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:59:34.655 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:34.654+0000 7fea12f8b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:59:34.662 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:34.662+0000 7f15bab57200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:59:34.725 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:34.724+0000 7fea12f8b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:59:34.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:34.910+0000 7f5361d9b700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 msgr2=0x7f534406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:34.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:34.910+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 0x7f534406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T09:59:35.287 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:35.285+0000 7f15bab57200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:35.355 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:35.354+0000 7fea12f8b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:35.364 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:35.362+0000 7f15bab57200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:59:35.432 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:35.431+0000 7fea12f8b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:59:35.506 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:35.505+0000 7f15bab57200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:35.576 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:35.574+0000 7fea12f8b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:35.731 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:35.730+0000 7f15bab57200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:59:35.804 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:35.803+0000 7f15bab57200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:59:35.805 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:35.803+0000 7fea12f8b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:59:35.878 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:35.877+0000 7fea12f8b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:59:36.069 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:36.068+0000 7f15bab57200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:59:36.144 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:36.143+0000 7fea12f8b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:59:36.287 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:36.286+0000 7f15bab57200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:59:36.364 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:36.363+0000 7fea12f8b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:59:36.434 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:36.432+0000 7f15bab57200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:59:36.512 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:36.510+0000 7fea12f8b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:59:36.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:36.511+0000 7f5361d9b700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 msgr2=0x7f534406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3717499045 2023-10-16T09:59:36.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:36.511+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 0x7f534406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T09:59:36.546 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:36.545+0000 7f15bab57200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:59:36.625 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:36.624+0000 7fea12f8b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:59:36.699 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:36.697+0000 7f15bab57200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:59:36.779 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:36.778+0000 7fea12f8b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:59:36.959 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:36.958+0000 7f15bab57200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:37.040 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.039+0000 7fea12f8b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:37.116 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:37.114+0000 7f15bab57200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:59:37.200 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.198+0000 7fea12f8b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:59:37.632 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:37.631+0000 7f15bab57200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:59:37.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:37.701+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 34) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5354055a70 con 0x7f53640b7590 2023-10-16T09:59:37.730 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.729+0000 7fea12f8b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:59:37.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:37.749+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 35) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f5354088bc0 con 0x7f53640b7590 2023-10-16T09:59:37.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:37.749+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 msgr2=0x7f534406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:37.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:37.749+0000 7f5352ffd700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3717499045,v1:172.21.15.158:6833/3717499045] conn(0x7f534406b070 0x7f534406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:37.889 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe98876b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.889 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe98876b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.889 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe98876b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.889 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe98876b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.889 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe98876b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.890 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe985f66700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.890 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe985f66700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.890 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe985f66700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.890 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe985f66700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:37.890 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:37.887+0000 7fe985f66700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:38.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.755+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 36) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5354056340 con 0x7f53640b7590 2023-10-16T09:59:38.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.755+0000 7f5352ffd700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f53440279c0 0x7f5344027da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:38.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.755+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 --> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5348000bc0 con 0x7f53440279c0 2023-10-16T09:59:38.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.758+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f53440279c0 0x7f5344027da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:38.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.761+0000 7f5361d9b700 1 --2- 172.21.15.158:0/3544460126 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f53440279c0 0x7f5344027da0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5364157280 tx=0x7f5358007ec0 comp rx=0 tx=0).ready entity=mgr.6939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:38.769 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:38.768+0000 7fe98ffba700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T09:59:38.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.768+0000 7f5352ffd700 1 -- 172.21.15.158:0/3544460126 <== mgr.6939 v2:172.21.15.158:6832/3502835696 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f5348000bc0 con 0x7f53440279c0 2023-10-16T09:59:38.772 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T09:59:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T09:59:38.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:38.777 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T09:59:38.798 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T09:59:38.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.964+0000 7f01bdece700 1 Processor -- start 2023-10-16T09:59:38.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.964+0000 7f01bdece700 1 -- start start 2023-10-16T09:59:38.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.965+0000 7f01bdece700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b80cc200 0x7f01b815f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:38.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.965+0000 7f01bdece700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80cf4c0 con 0x7f01b80ce510 2023-10-16T09:59:38.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.965+0000 7f01bdece700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b815f710 con 0x7f01b815b590 2023-10-16T09:59:38.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.965+0000 7f01bdece700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80b49e0 con 0x7f01b80cc200 2023-10-16T09:59:38.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.965+0000 7f01b6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f01b80ce510 0x7f01b80cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52968/0 (socket says 172.21.15.158:52968) 2023-10-16T09:59:38.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.965+0000 7f01b6ffd700 1 -- 172.21.15.158:0/3256095841 learned_addr learned my addr 172.21.15.158:0/3256095841 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:38.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b7fff700 1 --2- 172.21.15.158:0/3256095841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b80cc200 0x7f01b815f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:38.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b67fc700 1 -- 172.21.15.158:0/3256095841 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672075504 0 0) 0x7f01b80cf4c0 con 0x7f01b80ce510 2023-10-16T09:59:38.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b67fc700 1 -- 172.21.15.158:0/3256095841 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a00035b0 con 0x7f01b80ce510 2023-10-16T09:59:38.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b67fc700 1 -- 172.21.15.158:0/3256095841 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659412329 0 0) 0x7f01b815f710 con 0x7f01b815b590 2023-10-16T09:59:38.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b7fff700 1 -- 172.21.15.158:0/3256095841 >> v1:172.21.15.172:6789/0 conn(0x7f01b815b590 legacy=0x7f01b80b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:38.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b7fff700 1 -- 172.21.15.158:0/3256095841 >> v1:172.21.15.158:6789/0 conn(0x7f01b80ce510 legacy=0x7f01b80cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:38.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b7fff700 1 -- 172.21.15.158:0/3256095841 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b815f970 con 0x7f01b80cc200 2023-10-16T09:59:38.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b7fff700 1 --2- 172.21.15.158:0/3256095841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b80cc200 0x7f01b815f1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009dc0 tx=0x7f01b000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fe48e6a2cf20ca7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:38.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b67fc700 1 -- 172.21.15.158:0/3256095841 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b000e070 con 0x7f01b80cc200 2023-10-16T09:59:38.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b67fc700 1 -- 172.21.15.158:0/3256095841 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b0003a20 con 0x7f01b80cc200 2023-10-16T09:59:38.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.966+0000 7f01b67fc700 1 -- 172.21.15.158:0/3256095841 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a80015f0 con 0x7f01b80cc200 2023-10-16T09:59:38.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.967+0000 7f01bdece700 1 -- 172.21.15.158:0/3256095841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b80cc200 msgr2=0x7f01b815f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:38.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.967+0000 7f01bdece700 1 --2- 172.21.15.158:0/3256095841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b80cc200 0x7f01b815f1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009dc0 tx=0x7f01b000b330 comp rx=0 tx=0).stop 2023-10-16T09:59:38.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.967+0000 7f01bdece700 1 -- 172.21.15.158:0/3256095841 shutdown_connections 2023-10-16T09:59:38.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.967+0000 7f01bdece700 1 --2- 172.21.15.158:0/3256095841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b80cc200 0x7f01b815f1d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:38.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.967+0000 7f01bdece700 1 -- 172.21.15.158:0/3256095841 >> 172.21.15.158:0/3256095841 conn(0x7f01b806a8d0 msgr2=0x7f01b806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:38.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.967+0000 7f01bdece700 1 -- 172.21.15.158:0/3256095841 shutdown_connections 2023-10-16T09:59:38.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.967+0000 7f01bdece700 1 -- 172.21.15.158:0/3256095841 wait complete. 2023-10-16T09:59:38.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.968+0000 7f01bdece700 1 Processor -- start 2023-10-16T09:59:38.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.968+0000 7f01bdece700 1 -- start start 2023-10-16T09:59:38.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01bdece700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01b80cc200 0x7f01b815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:38.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01bdece700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01b80ce510 0x7f01b8159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:38.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01bdece700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b815b590 0x7f01b806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:38.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01bdece700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b815a1e0 con 0x7f01b80cc200 2023-10-16T09:59:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01bdece700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b815c170 con 0x7f01b80ce510 2023-10-16T09:59:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01bdece700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b815c450 con 0x7f01b815b590 2023-10-16T09:59:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01b80cc200 0x7f01b815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01b80cc200 0x7f01b815a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52796/0 (socket says 172.21.15.158:52796) 2023-10-16T09:59:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 -- 172.21.15.158:0/3752674917 learned_addr learned my addr 172.21.15.158:0/3752674917 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b7fff700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b815b590 0x7f01b806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b6ffd700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01b80ce510 0x7f01b8159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 -- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b815b590 msgr2=0x7f01b806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b815b590 0x7f01b806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 -- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01b80ce510 msgr2=0x7f01b8159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01b80ce510 0x7f01b8159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 -- 172.21.15.158:0/3752674917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0009a70 con 0x7f01b80cc200 2023-10-16T09:59:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b7fff700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b815b590 0x7f01b806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.969+0000 7f01b77fe700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01b80cc200 0x7f01b815a370 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f01ac00c730 tx=0x7f01ac00caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.970+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac0054e0 con 0x7f01b80cc200 2023-10-16T09:59:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.970+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac003680 con 0x7f01b80cc200 2023-10-16T09:59:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.970+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b815c730 con 0x7f01b80cc200 2023-10-16T09:59:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.970+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac003a30 con 0x7f01b80cc200 2023-10-16T09:59:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.970+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b815ccf0 con 0x7f01b80cc200 2023-10-16T09:59:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.971+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b80681b0 con 0x7f01b80cc200 2023-10-16T09:59:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.971+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 36) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f01ac003b90 con 0x7f01b80cc200 2023-10-16T09:59:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.972+0000 7f019ffff700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f01a406ada0 0x7f01a406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.972+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f01ac088c50 con 0x7f01b80cc200 2023-10-16T09:59:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.972+0000 7f01b6ffd700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f01a406ada0 0x7f01a406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.972+0000 7f01b6ffd700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f01a406ada0 0x7f01a406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01a8001eb0 tx=0x7f01a80015d0 comp rx=0 tx=0).ready entity=mgr.6939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:38.976+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f01ac053c00 con 0x7f01b80cc200 2023-10-16T09:59:39.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.228+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f01b80cc7b0 con 0x7f01b80cc200 2023-10-16T09:59:39.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.757+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v37) v1 ==== 93+0+0 (secure 0 0 0) 0x7f01ac017020 con 0x7f01b80cc200 2023-10-16T09:59:39.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.762+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f01a406ada0 msgr2=0x7f01a406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:39.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.762+0000 7f01bdece700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f01a406ada0 0x7f01a406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01a8001eb0 tx=0x7f01a80015d0 comp rx=0 tx=0).stop 2023-10-16T09:59:39.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.762+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01b80cc200 msgr2=0x7f01b815a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:39.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01b80cc200 0x7f01b815a370 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f01ac00c730 tx=0x7f01ac00caf0 comp rx=0 tx=0).stop 2023-10-16T09:59:39.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f019ffff700 1 -- 172.21.15.158:0/3752674917 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 37) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f01ac0bbfe0 con 0x7f01b80cc200 2023-10-16T09:59:39.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 shutdown_connections 2023-10-16T09:59:39.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01b815b590 0x7f01b806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:39.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01b80cc200 0x7f01b815a370 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:39.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f01a406ada0 0x7f01a406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:39.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 --2- 172.21.15.158:0/3752674917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01b80ce510 0x7f01b8159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:39.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 >> 172.21.15.158:0/3752674917 conn(0x7f01b806a8d0 msgr2=0x7f01b80afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:39.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 shutdown_connections 2023-10-16T09:59:39.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:39.763+0000 7f01bdece700 1 -- 172.21.15.158:0/3752674917 wait complete. 2023-10-16T09:59:39.778 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T09:59:39.779 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:39.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:39.863 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T09:59:39.864 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:39.864 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:39.883 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:39.883 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:39.907 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:39.906+0000 7f5459781200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:59:39.908 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:39.906+0000 7fa46b5dc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T09:59:39.926 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:39.973 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:39.972+0000 7f5459781200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:39.979 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:39.978+0000 7fa46b5dc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:40.046 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:40.044+0000 7f5459781200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:59:40.056 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:40.055+0000 7fa46b5dc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T09:59:40.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.091+0000 7f4c7212d700 1 Processor -- start 2023-10-16T09:59:40.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.091+0000 7f4c7212d700 1 -- start start 2023-10-16T09:59:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c7212d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0b6e10 0x7f4c6c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c7212d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4c6c0ce2c0 0x7f4c6c1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c7212d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c15c5d0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c7212d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6c15d2c0 con 0x7f4c6c0ce2c0 2023-10-16T09:59:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c7212d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c152c20 con 0x7f4c6c0b6e10 2023-10-16T09:59:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c6b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4c6c0cdee0 0x7f4c6c15bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53050/0 (socket says 172.21.15.158:53050) 2023-10-16T09:59:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c6b7fe700 1 -- 172.21.15.158:0/2895550155 learned_addr learned my addr 172.21.15.158:0/2895550155 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c6affd700 1 --2- 172.21.15.158:0/2895550155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0b6e10 0x7f4c6c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c6bfff700 1 --2- 172.21.15.158:0/2895550155 >> v2:172.21.15.172:3300/0 conn(0x7f4c6c0ce2c0 0x7f4c6c1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.092+0000 7f4c6a7fc700 1 -- 172.21.15.158:0/2895550155 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875108238 0 0) 0x7f4c6c15c5d0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6a7fc700 1 -- 172.21.15.158:0/2895550155 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c580035b0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6affd700 1 -- 172.21.15.158:0/2895550155 >> v2:172.21.15.172:3300/0 conn(0x7f4c6c0ce2c0 msgr2=0x7f4c6c1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6affd700 1 --2- 172.21.15.158:0/2895550155 >> v2:172.21.15.172:3300/0 conn(0x7f4c6c0ce2c0 0x7f4c6c1526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6affd700 1 -- 172.21.15.158:0/2895550155 >> v1:172.21.15.158:6789/0 conn(0x7f4c6c0cdee0 legacy=0x7f4c6c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6affd700 1 -- 172.21.15.158:0/2895550155 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c15c8b0 con 0x7f4c6c0b6e10 2023-10-16T09:59:40.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6affd700 1 --2- 172.21.15.158:0/2895550155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0b6e10 0x7f4c6c1521a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009840 tx=0x7f4c54009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c95979c4fd44e4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:40.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6a7fc700 1 -- 172.21.15.158:0/2895550155 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54011830 con 0x7f4c6c0b6e10 2023-10-16T09:59:40.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6a7fc700 1 -- 172.21.15.158:0/2895550155 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54011990 con 0x7f4c6c0b6e10 2023-10-16T09:59:40.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.093+0000 7f4c6a7fc700 1 -- 172.21.15.158:0/2895550155 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54011d20 con 0x7f4c6c0b6e10 2023-10-16T09:59:40.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 -- 172.21.15.158:0/2895550155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0b6e10 msgr2=0x7f4c6c1521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 --2- 172.21.15.158:0/2895550155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0b6e10 0x7f4c6c1521a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009840 tx=0x7f4c54009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:40.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 -- 172.21.15.158:0/2895550155 shutdown_connections 2023-10-16T09:59:40.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 --2- 172.21.15.158:0/2895550155 >> v2:172.21.15.172:3300/0 conn(0x7f4c6c0ce2c0 0x7f4c6c1526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 --2- 172.21.15.158:0/2895550155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0b6e10 0x7f4c6c1521a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 -- 172.21.15.158:0/2895550155 >> 172.21.15.158:0/2895550155 conn(0x7f4c6c06a8d0 msgr2=0x7f4c6c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:40.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 -- 172.21.15.158:0/2895550155 shutdown_connections 2023-10-16T09:59:40.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.094+0000 7f4c7212d700 1 -- 172.21.15.158:0/2895550155 wait complete. 2023-10-16T09:59:40.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.095+0000 7f4c7212d700 1 Processor -- start 2023-10-16T09:59:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.095+0000 7f4c7212d700 1 -- start start 2023-10-16T09:59:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c7212d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c6c0b6e10 0x7f4c6c0c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c7212d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c6c0cdee0 0x7f4c6c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c7212d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0ce2c0 0x7f4c6c154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c7212d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c1538a0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c7212d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c153af0 con 0x7f4c6c0b6e10 2023-10-16T09:59:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c7212d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c153dd0 con 0x7f4c6c0ce2c0 2023-10-16T09:59:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c6c0cdee0 0x7f4c6c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0ce2c0 0x7f4c6c154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c6c0b6e10 0x7f4c6c0c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c6c0cdee0 0x7f4c6c0ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52882/0 (socket says 172.21.15.158:52882) 2023-10-16T09:59:40.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 -- 172.21.15.158:0/1050108469 learned_addr learned my addr 172.21.15.158:0/1050108469 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:40.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 -- 172.21.15.158:0/1050108469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0ce2c0 msgr2=0x7f4c6c154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0ce2c0 0x7f4c6c154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 -- 172.21.15.158:0/1050108469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c6c0b6e10 msgr2=0x7f4c6c0c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c6c0b6e10 0x7f4c6c0c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 -- 172.21.15.158:0/1050108469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540094a0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.096+0000 7f4c6affd700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c6c0cdee0 0x7f4c6c0ceee0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4c540090b0 tx=0x7f4c54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:40.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.097+0000 7f4c63fff700 1 -- 172.21.15.158:0/1050108469 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54004590 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.097+0000 7f4c63fff700 1 -- 172.21.15.158:0/1050108469 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5401f070 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.097+0000 7f4c63fff700 1 -- 172.21.15.158:0/1050108469 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c540038e0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.097+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c1540b0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.097+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c154670 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.097+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c15c350 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.098+0000 7f4c63fff700 1 -- 172.21.15.158:0/1050108469 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 37) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4c54003a40 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.098+0000 7f4c63fff700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 0x7f4c4c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.099+0000 7f4c6b7fe700 1 -- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 msgr2=0x7f4c4c06d240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:40.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.099+0000 7f4c63fff700 1 -- 172.21.15.158:0/1050108469 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f4c54021070 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.099+0000 7f4c6b7fe700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 0x7f4c4c06d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:40.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.103+0000 7f4c63fff700 1 -- 172.21.15.158:0/1050108469 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4c54090020 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.124 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:40.122+0000 7f5459781200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:59:40.140 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:40.139+0000 7fa46b5dc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T09:59:40.265 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:40.264+0000 7f5459781200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:59:40.284 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:40.283+0000 7fa46b5dc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T09:59:40.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.300+0000 7f4c6b7fe700 1 -- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 msgr2=0x7f4c4c06d240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:40.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.300+0000 7f4c6b7fe700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 0x7f4c4c06d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:40.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.304+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c6c0681b0 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.305+0000 7f4c63fff700 1 -- 172.21.15.158:0/1050108469 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17880 (secure 0 0 0) 0x7f4c54055d30 con 0x7f4c6c0cdee0 2023-10-16T09:59:40.307 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:40.307 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":86,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:37.743739+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:21.428579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:22.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3100922361":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6833/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6832/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/3345169280":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/4171724344":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/310788915":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/2329993263":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/234410797":"2023-10-17T09:59:37.743702+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:40.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 msgr2=0x7f4c4c06d240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:40.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 0x7f4c4c06d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c6c0cdee0 msgr2=0x7f4c6c0ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c6c0cdee0 0x7f4c6c0ceee0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4c540090b0 tx=0x7f4c54004750 comp rx=0 tx=0).stop 2023-10-16T09:59:40.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 shutdown_connections 2023-10-16T09:59:40.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c6c0ce2c0 0x7f4c6c154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4c4c06ad80 0x7f4c4c06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c6c0b6e10 0x7f4c6c0c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 --2- 172.21.15.158:0/1050108469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c6c0cdee0 0x7f4c6c0ceee0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 >> 172.21.15.158:0/1050108469 conn(0x7f4c6c06a8d0 msgr2=0x7f4c6c0b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:40.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 shutdown_connections 2023-10-16T09:59:40.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.308+0000 7f4c7212d700 1 -- 172.21.15.158:0/1050108469 wait complete. 2023-10-16T09:59:40.322 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:59:21.428579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:59:22.493630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:40.323 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:40.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.487+0000 7f9c25dbd700 1 Processor -- start 2023-10-16T09:59:40.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.487+0000 7f9c25dbd700 1 -- start start 2023-10-16T09:59:40.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c25dbd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c2015b6b0 0x7f9c200cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c25dbd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c200b4420 con 0x7f9c200d0c60 2023-10-16T09:59:40.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c25dbd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c200cbc20 con 0x7f9c200b3630 2023-10-16T09:59:40.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c25dbd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2015ba90 con 0x7f9c2015b6b0 2023-10-16T09:59:40.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c2015b6b0 0x7f9c200cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c2015b6b0 0x7f9c200cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52934/0 (socket says 172.21.15.158:52934) 2023-10-16T09:59:40.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1f7fe700 1 -- 172.21.15.158:0/3217136888 learned_addr learned my addr 172.21.15.158:0/3217136888 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:40.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1f7fe700 1 -- 172.21.15.158:0/3217136888 >> v1:172.21.15.172:6789/0 conn(0x7f9c200b3630 legacy=0x7f9c200b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1f7fe700 1 -- 172.21.15.158:0/3217136888 >> v1:172.21.15.158:6789/0 conn(0x7f9c200d0c60 legacy=0x7f9c200b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1f7fe700 1 -- 172.21.15.158:0/3217136888 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c200cbf00 con 0x7f9c2015b6b0 2023-10-16T09:59:40.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/3217136888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c2015b6b0 0x7f9c200cfdb0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9c100098f0 tx=0x7f9c10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28ba887f39ec196 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:40.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1effd700 1 -- 172.21.15.158:0/3217136888 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000b8a0 con 0x7f9c2015b6b0 2023-10-16T09:59:40.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1effd700 1 -- 172.21.15.158:0/3217136888 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1000ba00 con 0x7f9c2015b6b0 2023-10-16T09:59:40.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.488+0000 7f9c1effd700 1 -- 172.21.15.158:0/3217136888 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000bd90 con 0x7f9c2015b6b0 2023-10-16T09:59:40.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.489+0000 7f9c25dbd700 1 -- 172.21.15.158:0/3217136888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c2015b6b0 msgr2=0x7f9c200cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.489+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/3217136888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c2015b6b0 0x7f9c200cfdb0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9c100098f0 tx=0x7f9c10009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:40.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.489+0000 7f9c25dbd700 1 -- 172.21.15.158:0/3217136888 shutdown_connections 2023-10-16T09:59:40.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.489+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/3217136888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c2015b6b0 0x7f9c200cfdb0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.489+0000 7f9c25dbd700 1 -- 172.21.15.158:0/3217136888 >> 172.21.15.158:0/3217136888 conn(0x7f9c200cdf20 msgr2=0x7f9c200ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:40.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.489+0000 7f9c25dbd700 1 -- 172.21.15.158:0/3217136888 shutdown_connections 2023-10-16T09:59:40.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.489+0000 7f9c25dbd700 1 -- 172.21.15.158:0/3217136888 wait complete. 2023-10-16T09:59:40.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.490+0000 7f9c25dbd700 1 Processor -- start 2023-10-16T09:59:40.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.490+0000 7f9c25dbd700 1 -- start start 2023-10-16T09:59:40.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c25dbd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c200b3630 0x7f9c200ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c25dbd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d0c60 0x7f9c200c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c25dbd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2015b6b0 0x7f9c2015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c25dbd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200caa00 con 0x7f9c200d0c60 2023-10-16T09:59:40.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c25dbd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200c8ab0 con 0x7f9c200b3630 2023-10-16T09:59:40.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c25dbd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c201f3c10 con 0x7f9c2015b6b0 2023-10-16T09:59:40.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c17fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d0c60 0x7f9c200c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c17fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d0c60 0x7f9c200c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52942/0 (socket says 172.21.15.158:52942) 2023-10-16T09:59:40.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c17fff700 1 -- 172.21.15.158:0/1150945072 learned_addr learned my addr 172.21.15.158:0/1150945072 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:40.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c200b3630 0x7f9c200ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.491+0000 7f9c1ffff700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2015b6b0 0x7f9c2015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c17fff700 1 -- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2015b6b0 msgr2=0x7f9c2015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c17fff700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2015b6b0 0x7f9c2015cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c17fff700 1 -- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c200b3630 msgr2=0x7f9c200ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c17fff700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c200b3630 0x7f9c200ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c17fff700 1 -- 172.21.15.158:0/1150945072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10009550 con 0x7f9c200d0c60 2023-10-16T09:59:40.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c1ffff700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2015b6b0 0x7f9c2015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:40.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c17fff700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d0c60 0x7f9c200c8570 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c4b0 tx=0x7f9c0800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:40.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c1cff9700 1 -- 172.21.15.158:0/1150945072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080052d0 con 0x7f9c200d0c60 2023-10-16T09:59:40.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c1cff9700 1 -- 172.21.15.158:0/1150945072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08005430 con 0x7f9c200d0c60 2023-10-16T09:59:40.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c201f3090 con 0x7f9c200d0c60 2023-10-16T09:59:40.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c1cff9700 1 -- 172.21.15.158:0/1150945072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080038c0 con 0x7f9c200d0c60 2023-10-16T09:59:40.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.492+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c201f3650 con 0x7f9c200d0c60 2023-10-16T09:59:40.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.493+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2006ba50 con 0x7f9c200d0c60 2023-10-16T09:59:40.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.493+0000 7f9c1cff9700 1 -- 172.21.15.158:0/1150945072 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 37) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9c08003a20 con 0x7f9c200d0c60 2023-10-16T09:59:40.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.494+0000 7f9c1cff9700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 0x7f9c0006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.494+0000 7f9c1f7fe700 1 -- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 msgr2=0x7f9c0006d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:40.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.494+0000 7f9c1cff9700 1 -- 172.21.15.158:0/1150945072 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f9c08088a80 con 0x7f9c200d0c60 2023-10-16T09:59:40.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.494+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 0x7f9c0006d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:40.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.498+0000 7f9c1cff9700 1 -- 172.21.15.158:0/1150945072 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c2006ba50 con 0x7f9c200d0c60 2023-10-16T09:59:40.589 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:40.588+0000 7f5459781200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:59:40.626 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:40.624+0000 7fa46b5dc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T09:59:40.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.695+0000 7f9c1f7fe700 1 -- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 msgr2=0x7f9c0006d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:40.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.695+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 0x7f9c0006d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:40.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.697+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c2006ba50 con 0x7f9c200d0c60 2023-10-16T09:59:40.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.697+0000 7f9c1cff9700 1 -- 172.21.15.158:0/1150945072 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c08017020 con 0x7f9c200d0c60 2023-10-16T09:59:40.699 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:40.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 msgr2=0x7f9c0006d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:40.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 0x7f9c0006d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d0c60 msgr2=0x7f9c200c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d0c60 0x7f9c200c8570 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c4b0 tx=0x7f9c0800c7c0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 shutdown_connections 2023-10-16T09:59:40.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2015b6b0 0x7f9c2015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f9c0006ad50 0x7f9c0006d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c200b3630 0x7f9c200ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 --2- 172.21.15.158:0/1150945072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d0c60 0x7f9c200c8570 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 >> 172.21.15.158:0/1150945072 conn(0x7f9c200cdf20 msgr2=0x7f9c200b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:40.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 shutdown_connections 2023-10-16T09:59:40.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.700+0000 7f9c25dbd700 1 -- 172.21.15.158:0/1150945072 wait complete. 2023-10-16T09:59:40.704 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:40.702+0000 7f5459781200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:59:40.715 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:59:40.749 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:40.748+0000 7fa46b5dc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T09:59:40.779 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:40.778+0000 7f5459781200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:59:40.828 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:40.827+0000 7fa46b5dc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T09:59:40.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.844+0000 7fc7998e3700 1 Processor -- start 2023-10-16T09:59:40.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.844+0000 7fc7998e3700 1 -- start start 2023-10-16T09:59:40.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.844+0000 7fc7998e3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc7940d29b0 0x7fc7940d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.844+0000 7fc7998e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7940d0a70 0x7fc7940b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.844+0000 7fc7998e3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b7c90 con 0x7fc794160d30 2023-10-16T09:59:40.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.844+0000 7fc7998e3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7940b8910 con 0x7fc7940d29b0 2023-10-16T09:59:40.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.844+0000 7fc7998e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940d0f00 con 0x7fc7940d0a70 2023-10-16T09:59:40.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc792ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc794160d30 0x7fc7940b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53126/0 (socket says 172.21.15.158:53126) 2023-10-16T09:59:40.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc792ffd700 1 -- 172.21.15.158:0/2175489854 learned_addr learned my addr 172.21.15.158:0/2175489854 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:40.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7937fe700 1 --2- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7940d0a70 0x7fc7940b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7927fc700 1 --2- 172.21.15.158:0/2175489854 >> v2:172.21.15.172:3300/0 conn(0x7fc7940d29b0 0x7fc7940d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc791ffb700 1 -- 172.21.15.158:0/2175489854 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055374096 0 0) 0x7fc7940b7c90 con 0x7fc794160d30 2023-10-16T09:59:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc791ffb700 1 -- 172.21.15.158:0/2175489854 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc77c0035b0 con 0x7fc794160d30 2023-10-16T09:59:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7927fc700 1 -- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7940d0a70 msgr2=0x7fc7940b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7927fc700 1 --2- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7940d0a70 0x7fc7940b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7927fc700 1 -- 172.21.15.158:0/2175489854 >> v1:172.21.15.158:6789/0 conn(0x7fc794160d30 legacy=0x7fc7940b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7927fc700 1 -- 172.21.15.158:0/2175489854 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940b7f70 con 0x7fc7940d29b0 2023-10-16T09:59:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7937fe700 1 --2- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7940d0a70 0x7fc7940b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc7927fc700 1 --2- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d29b0 0x7fc7940d0530 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc788009840 tx=0x7fc788009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78703f7540ed3c40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc791ffb700 1 -- 172.21.15.158:0/2175489854 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788011830 con 0x7fc7940d29b0 2023-10-16T09:59:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc791ffb700 1 -- 172.21.15.158:0/2175489854 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc788011990 con 0x7fc7940d29b0 2023-10-16T09:59:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.845+0000 7fc791ffb700 1 -- 172.21.15.158:0/2175489854 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788011d20 con 0x7fc7940d29b0 2023-10-16T09:59:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 -- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d29b0 msgr2=0x7fc7940d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 --2- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d29b0 0x7fc7940d0530 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc788009840 tx=0x7fc788009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 -- 172.21.15.158:0/2175489854 shutdown_connections 2023-10-16T09:59:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 --2- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7940d0a70 0x7fc7940b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 --2- 172.21.15.158:0/2175489854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d29b0 0x7fc7940d0530 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 -- 172.21.15.158:0/2175489854 >> 172.21.15.158:0/2175489854 conn(0x7fc7940669f0 msgr2=0x7fc794066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 -- 172.21.15.158:0/2175489854 shutdown_connections 2023-10-16T09:59:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.846+0000 7fc7998e3700 1 -- 172.21.15.158:0/2175489854 wait complete. 2023-10-16T09:59:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.847+0000 7fc7998e3700 1 Processor -- start 2023-10-16T09:59:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.847+0000 7fc7998e3700 1 -- start start 2023-10-16T09:59:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7998e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d0a70 0x7fc794156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7998e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940d29b0 0x7fc7940ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7998e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc794160d30 0x7fc7940cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7998e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941f2e30 con 0x7fc7940d29b0 2023-10-16T09:59:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7998e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940c7f70 con 0x7fc7940d0a70 2023-10-16T09:59:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7998e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940c8250 con 0x7fc794160d30 2023-10-16T09:59:40.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7927fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940d29b0 0x7fc7940ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7927fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940d29b0 0x7fc7940ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:52958/0 (socket says 172.21.15.158:52958) 2023-10-16T09:59:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7927fc700 1 -- 172.21.15.158:0/1090154130 learned_addr learned my addr 172.21.15.158:0/1090154130 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc792ffd700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d0a70 0x7fc794156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.848+0000 7fc7937fe700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc794160d30 0x7fc7940cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7937fe700 1 -- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d0a70 msgr2=0x7fc794156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7937fe700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d0a70 0x7fc794156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7937fe700 1 -- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940d29b0 msgr2=0x7fc7940ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7937fe700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940d29b0 0x7fc7940ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7937fe700 1 -- 172.21.15.158:0/1090154130 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c009a70 con 0x7fc794160d30 2023-10-16T09:59:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7927fc700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940d29b0 0x7fc7940ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7937fe700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc794160d30 0x7fc7940cabd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00efc0 tx=0x7fc78c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc77bfff700 1 -- 172.21.15.158:0/1090154130 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c00ca90 con 0x7fc794160d30 2023-10-16T09:59:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc77bfff700 1 -- 172.21.15.158:0/1090154130 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c00f040 con 0x7fc794160d30 2023-10-16T09:59:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7880094a0 con 0x7fc794160d30 2023-10-16T09:59:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc77bfff700 1 -- 172.21.15.158:0/1090154130 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c00ce30 con 0x7fc794160d30 2023-10-16T09:59:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.849+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7940c8810 con 0x7fc794160d30 2023-10-16T09:59:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.850+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc794155090 con 0x7fc794160d30 2023-10-16T09:59:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.850+0000 7fc77bfff700 1 -- 172.21.15.158:0/1090154130 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc78c014480 con 0x7fc794160d30 2023-10-16T09:59:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.851+0000 7fc77bfff700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.851+0000 7fc792ffd700 1 -- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 msgr2=0x7fc78006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.851+0000 7fc77bfff700 1 -- 172.21.15.158:0/1090154130 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fc78c08a210 con 0x7fc794160d30 2023-10-16T09:59:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.851+0000 7fc792ffd700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:40.866 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:40.859+0000 7f5459781200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:59:40.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:40.856+0000 7fc77bfff700 1 -- 172.21.15.158:0/1090154130 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc78c054cf0 con 0x7fc794160d30 2023-10-16T09:59:40.914 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:40.913+0000 7fa46b5dc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T09:59:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.052+0000 7fc792ffd700 1 -- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 msgr2=0x7fc78006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.052+0000 7fc792ffd700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.052+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7940c8390 con 0x7fc794160d30 2023-10-16T09:59:41.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.052+0000 7fc77bfff700 1 -- 172.21.15.158:0/1090154130 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc7940c8390 con 0x7fc794160d30 2023-10-16T09:59:41.054 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:59:41.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 msgr2=0x7fc78006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:41.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc794160d30 msgr2=0x7fc7940cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc794160d30 0x7fc7940cabd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00efc0 tx=0x7fc78c00c3a0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 shutdown_connections 2023-10-16T09:59:41.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc794160d30 0x7fc7940cabd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7940d0a70 0x7fc794156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 --2- 172.21.15.158:0/1090154130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940d29b0 0x7fc7940ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 >> 172.21.15.158:0/1090154130 conn(0x7fc7940669f0 msgr2=0x7fc794154980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:41.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 shutdown_connections 2023-10-16T09:59:41.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.055+0000 7fc7998e3700 1 -- 172.21.15.158:0/1090154130 wait complete. 2023-10-16T09:59:41.068 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:59:41.230 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:41.229+0000 7f5459781200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:59:41.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.235+0000 7faa6f59b700 1 Processor -- start 2023-10-16T09:59:41.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.235+0000 7faa6f59b700 1 -- start start 2023-10-16T09:59:41.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6f59b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faa6815b6b0 0x7faa680cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6f59b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa680d0c60 0x7faa680b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6f59b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa680b4420 con 0x7faa680b3460 2023-10-16T09:59:41.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6f59b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6815ba90 con 0x7faa680d0c60 2023-10-16T09:59:41.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6f59b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa680b38d0 con 0x7faa6815b6b0 2023-10-16T09:59:41.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6cb36700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa680d0c60 0x7faa680b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6cb36700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa680d0c60 0x7faa680b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53024/0 (socket says 172.21.15.158:53024) 2023-10-16T09:59:41.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.236+0000 7faa6cb36700 1 -- 172.21.15.158:0/3380417017 learned_addr learned my addr 172.21.15.158:0/3380417017 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:41.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa6d337700 1 --2- 172.21.15.158:0/3380417017 >> v2:172.21.15.172:3300/0 conn(0x7faa6815b6b0 0x7faa680cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa6cb36700 1 -- 172.21.15.158:0/3380417017 >> v2:172.21.15.172:3300/0 conn(0x7faa6815b6b0 msgr2=0x7faa680cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:41.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa6cb36700 1 --2- 172.21.15.158:0/3380417017 >> v2:172.21.15.172:3300/0 conn(0x7faa6815b6b0 0x7faa680cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa6cb36700 1 -- 172.21.15.158:0/3380417017 >> v1:172.21.15.158:6789/0 conn(0x7faa680b3460 legacy=0x7faa68070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa6cb36700 1 -- 172.21.15.158:0/3380417017 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680cbc20 con 0x7faa680d0c60 2023-10-16T09:59:41.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa6cb36700 1 --2- 172.21.15.158:0/3380417017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa680d0c60 0x7faa680b2f20 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7faa6806ffb0 tx=0x7faa58009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c613bc0f5af678d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:41.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa5ffff700 1 -- 172.21.15.158:0/3380417017 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58011830 con 0x7faa680d0c60 2023-10-16T09:59:41.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa5ffff700 1 -- 172.21.15.158:0/3380417017 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa58011990 con 0x7faa680d0c60 2023-10-16T09:59:41.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.237+0000 7faa5ffff700 1 -- 172.21.15.158:0/3380417017 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58011d20 con 0x7faa680d0c60 2023-10-16T09:59:41.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 -- 172.21.15.158:0/3380417017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa680d0c60 msgr2=0x7faa680b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 --2- 172.21.15.158:0/3380417017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa680d0c60 0x7faa680b2f20 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7faa6806ffb0 tx=0x7faa58009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:41.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 -- 172.21.15.158:0/3380417017 shutdown_connections 2023-10-16T09:59:41.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 --2- 172.21.15.158:0/3380417017 >> v2:172.21.15.172:3300/0 conn(0x7faa6815b6b0 0x7faa680cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 --2- 172.21.15.158:0/3380417017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa680d0c60 0x7faa680b2f20 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 -- 172.21.15.158:0/3380417017 >> 172.21.15.158:0/3380417017 conn(0x7faa680cdf20 msgr2=0x7faa680ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:41.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 -- 172.21.15.158:0/3380417017 shutdown_connections 2023-10-16T09:59:41.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.238+0000 7faa6f59b700 1 -- 172.21.15.158:0/3380417017 wait complete. 2023-10-16T09:59:41.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.239+0000 7faa6f59b700 1 Processor -- start 2023-10-16T09:59:41.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.239+0000 7faa6f59b700 1 -- start start 2023-10-16T09:59:41.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6f59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa680b3460 0x7faa68150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6f59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa680d0c60 0x7faa680c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6f59b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa6815b6b0 0x7faa681f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6f59b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa68151e60 con 0x7faa6815b6b0 2023-10-16T09:59:41.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6f59b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa68151830 con 0x7faa680d0c60 2023-10-16T09:59:41.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6f59b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa68151070 con 0x7faa680b3460 2023-10-16T09:59:41.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa6815b6b0 0x7faa681f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa6815b6b0 0x7faa681f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53032/0 (socket says 172.21.15.158:53032) 2023-10-16T09:59:41.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 -- 172.21.15.158:0/2612616266 learned_addr learned my addr 172.21.15.158:0/2612616266 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:41.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6cb36700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa680d0c60 0x7faa680c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6d337700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa680b3460 0x7faa68150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 -- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa680b3460 msgr2=0x7faa68150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa680b3460 0x7faa68150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 -- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa680d0c60 msgr2=0x7faa680c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa680d0c60 0x7faa680c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 -- 172.21.15.158:0/2612616266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa58009550 con 0x7faa6815b6b0 2023-10-16T09:59:41.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6cb36700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa680d0c60 0x7faa680c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:41.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.240+0000 7faa6db38700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa6815b6b0 0x7faa681f2c90 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7faa6000cb50 tx=0x7faa6000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:41.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.241+0000 7faa5dffb700 1 -- 172.21.15.158:0/2612616266 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60003c00 con 0x7faa6815b6b0 2023-10-16T09:59:41.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.241+0000 7faa5dffb700 1 -- 172.21.15.158:0/2612616266 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60003d60 con 0x7faa6815b6b0 2023-10-16T09:59:41.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.241+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6806ca70 con 0x7faa6815b6b0 2023-10-16T09:59:41.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.241+0000 7faa5dffb700 1 -- 172.21.15.158:0/2612616266 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6000aa20 con 0x7faa6815b6b0 2023-10-16T09:59:41.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.241+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa68151a80 con 0x7faa6815b6b0 2023-10-16T09:59:41.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.242+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa680681b0 con 0x7faa6815b6b0 2023-10-16T09:59:41.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.242+0000 7faa5dffb700 1 -- 172.21.15.158:0/2612616266 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7faa6000ab80 con 0x7faa6815b6b0 2023-10-16T09:59:41.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.243+0000 7faa5dffb700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 0x7faa4c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.243+0000 7faa6d337700 1 -- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 msgr2=0x7faa4c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:41.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.243+0000 7faa5dffb700 1 -- 172.21.15.158:0/2612616266 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7faa60089b00 con 0x7faa6815b6b0 2023-10-16T09:59:41.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.243+0000 7faa6d337700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 0x7faa4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:41.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.248+0000 7faa5dffb700 1 -- 172.21.15.158:0/2612616266 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa600546f0 con 0x7faa6815b6b0 2023-10-16T09:59:41.302 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:41.301+0000 7fa46b5dc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T09:59:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.444+0000 7faa6d337700 1 -- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 msgr2=0x7faa4c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.444+0000 7faa6d337700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 0x7faa4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:41.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.447+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faa680cc1b0 con 0x7faa6815b6b0 2023-10-16T09:59:41.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.448+0000 7faa5dffb700 1 -- 172.21.15.158:0/2612616266 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7faa60057d10 con 0x7faa6815b6b0 2023-10-16T09:59:41.449 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:59:41.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 msgr2=0x7faa4c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:41.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 0x7faa4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa6815b6b0 msgr2=0x7faa681f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa6815b6b0 0x7faa681f2c90 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7faa6000cb50 tx=0x7faa6000ce60 comp rx=0 tx=0).stop 2023-10-16T09:59:41.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 shutdown_connections 2023-10-16T09:59:41.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa6815b6b0 0x7faa681f2c90 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa680b3460 0x7faa68150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7faa4c06b020 0x7faa4c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 --2- 172.21.15.158:0/2612616266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa680d0c60 0x7faa680c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 >> 172.21.15.158:0/2612616266 conn(0x7faa680cdf20 msgr2=0x7faa6815c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:41.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 shutdown_connections 2023-10-16T09:59:41.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.450+0000 7faa6f59b700 1 -- 172.21.15.158:0/2612616266 wait complete. 2023-10-16T09:59:41.464 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T09:59:41.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.587+0000 7f178014b700 1 Processor -- start 2023-10-16T09:59:41.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.587+0000 7f178014b700 1 -- start start 2023-10-16T09:59:41.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f178014b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f177815df60 0x7f177806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f178014b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f177815e3f0 0x7f177806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f178014b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1778071540 con 0x7f1778151400 2023-10-16T09:59:41.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f178014b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780c9e60 con 0x7f177815df60 2023-10-16T09:59:41.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f178014b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17781517e0 con 0x7f177815e3f0 2023-10-16T09:59:41.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f177815df60 0x7f177806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f177815df60 0x7f177806cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53040/0 (socket says 172.21.15.158:53040) 2023-10-16T09:59:41.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 -- 172.21.15.158:0/459235375 learned_addr learned my addr 172.21.15.158:0/459235375 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:41.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177e6e8700 1 --2- 172.21.15.158:0/459235375 >> v2:172.21.15.172:3300/0 conn(0x7f177815e3f0 0x7f177806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 -- 172.21.15.158:0/459235375 >> v2:172.21.15.172:3300/0 conn(0x7f177815e3f0 msgr2=0x7f177806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:41.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 --2- 172.21.15.158:0/459235375 >> v2:172.21.15.172:3300/0 conn(0x7f177815e3f0 0x7f177806d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 -- 172.21.15.158:0/459235375 >> v1:172.21.15.172:6789/0 conn(0x7f1778151400 legacy=0x7f17780c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 -- 172.21.15.158:0/459235375 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780717a0 con 0x7f177815df60 2023-10-16T09:59:41.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177d6e6700 1 --2- 172.21.15.158:0/459235375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815df60 0x7f177806cc90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f177806d900 tx=0x7f176800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da093e1f417983bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:41.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177cee5700 1 -- 172.21.15.158:0/459235375 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176800dd30 con 0x7f177815df60 2023-10-16T09:59:41.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177cee5700 1 -- 172.21.15.158:0/459235375 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176800de90 con 0x7f177815df60 2023-10-16T09:59:41.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.588+0000 7f177cee5700 1 -- 172.21.15.158:0/459235375 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768010690 con 0x7f177815df60 2023-10-16T09:59:41.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 -- 172.21.15.158:0/459235375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815df60 msgr2=0x7f177806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 --2- 172.21.15.158:0/459235375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815df60 0x7f177806cc90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f177806d900 tx=0x7f176800be70 comp rx=0 tx=0).stop 2023-10-16T09:59:41.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 -- 172.21.15.158:0/459235375 shutdown_connections 2023-10-16T09:59:41.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 --2- 172.21.15.158:0/459235375 >> v2:172.21.15.172:3300/0 conn(0x7f177815e3f0 0x7f177806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 --2- 172.21.15.158:0/459235375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815df60 0x7f177806cc90 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 -- 172.21.15.158:0/459235375 >> 172.21.15.158:0/459235375 conn(0x7f177806acc0 msgr2=0x7f177806b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:41.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 -- 172.21.15.158:0/459235375 shutdown_connections 2023-10-16T09:59:41.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.589+0000 7f178014b700 1 -- 172.21.15.158:0/459235375 wait complete. 2023-10-16T09:59:41.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.590+0000 7f178014b700 1 Processor -- start 2023-10-16T09:59:41.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.590+0000 7f178014b700 1 -- start start 2023-10-16T09:59:41.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f178014b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1778151400 0x7f177815ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f178014b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815df60 0x7f17780c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f178014b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815e3f0 0x7f17780c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f178014b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780b05f0 con 0x7f177815e3f0 2023-10-16T09:59:41.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f178014b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780b0840 con 0x7f1778151400 2023-10-16T09:59:41.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f178014b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780b0b20 con 0x7f177815df60 2023-10-16T09:59:41.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f177e6e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815e3f0 0x7f17780c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f177e6e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815e3f0 0x7f17780c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53048/0 (socket says 172.21.15.158:53048) 2023-10-16T09:59:41.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f177e6e8700 1 -- 172.21.15.158:0/2482817679 learned_addr learned my addr 172.21.15.158:0/2482817679 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:41.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f177dee7700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1778151400 0x7f177815ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.591+0000 7f177d6e6700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815df60 0x7f17780c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:41.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f177e6e8700 1 -- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815df60 msgr2=0x7f17780c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f177e6e8700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815df60 0x7f17780c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f177e6e8700 1 -- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1778151400 msgr2=0x7f177815ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f177e6e8700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1778151400 0x7f177815ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f177e6e8700 1 -- 172.21.15.158:0/2482817679 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176800b7f0 con 0x7f177815e3f0 2023-10-16T09:59:41.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f177e6e8700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815e3f0 0x7f17780c40a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1770007e40 tx=0x7f177000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:41.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f176e7fc700 1 -- 172.21.15.158:0/2482817679 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000ed30 con 0x7f177815e3f0 2023-10-16T09:59:41.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f176e7fc700 1 -- 172.21.15.158:0/2482817679 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770014420 con 0x7f177815e3f0 2023-10-16T09:59:41.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17780b0e00 con 0x7f177815e3f0 2023-10-16T09:59:41.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f176e7fc700 1 -- 172.21.15.158:0/2482817679 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17700147d0 con 0x7f177815e3f0 2023-10-16T09:59:41.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.592+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17780b13c0 con 0x7f177815e3f0 2023-10-16T09:59:41.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.593+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177804ec50 con 0x7f177815e3f0 2023-10-16T09:59:41.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.594+0000 7f176e7fc700 1 -- 172.21.15.158:0/2482817679 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1770014930 con 0x7f177815e3f0 2023-10-16T09:59:41.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.594+0000 7f176e7fc700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 0x7f175c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:41.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.594+0000 7f177dee7700 1 -- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 msgr2=0x7f175c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:41.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.594+0000 7f176e7fc700 1 -- 172.21.15.158:0/2482817679 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f177008a1b0 con 0x7f177815e3f0 2023-10-16T09:59:41.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.594+0000 7f177dee7700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 0x7f175c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:41.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.599+0000 7f176e7fc700 1 -- 172.21.15.158:0/2482817679 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1770054da0 con 0x7f177815e3f0 2023-10-16T09:59:41.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.795+0000 7f177dee7700 1 -- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 msgr2=0x7f175c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:41.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.795+0000 7f177dee7700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 0x7f175c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:41.800 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:41.798+0000 7f5459781200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:59:41.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.847+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f17780cda00 con 0x7f177815e3f0 2023-10-16T09:59:41.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.848+0000 7f176e7fc700 1 -- 172.21.15.158:0/2482817679 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v86) v1 ==== 80+0+476 (secure 0 0 0) 0x7f17700583c0 con 0x7f177815e3f0 2023-10-16T09:59:41.849 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3100922361 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.850 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/3717499045 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.850 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/3717499045 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.850 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3345169280 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.850 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/4171724344 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.851 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/310788915 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.851 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2329993263 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.851 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/234410797 2023-10-17T09:59:37.743702+0000 2023-10-16T09:59:41.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 msgr2=0x7f175c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:41.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 0x7f175c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815e3f0 msgr2=0x7f17780c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:41.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815e3f0 0x7f17780c40a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1770007e40 tx=0x7f177000e3c0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 shutdown_connections 2023-10-16T09:59:41.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f177815e3f0 0x7f17780c40a0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f175c06b020 0x7f175c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1778151400 0x7f177815ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 --2- 172.21.15.158:0/2482817679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815df60 0x7f17780c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:41.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 >> 172.21.15.158:0/2482817679 conn(0x7f177806acc0 msgr2=0x7f17780cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:41.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 shutdown_connections 2023-10-16T09:59:41.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:41.850+0000 7f178014b700 1 -- 172.21.15.158:0/2482817679 wait complete. 2023-10-16T09:59:41.855 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T09:59:41.866 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T09:59:41.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:41.867 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T09:59:41.869 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:41.868+0000 7f5459781200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:59:41.912 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T09:59:41.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T09:59:41.913 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T09:59:41.915 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:41.914+0000 7fa46b5dc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T09:59:41.930 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:41.931 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:41.947 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:41.948 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:41.948 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:41.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:59:41.988 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:41.988 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:41.990 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:41.989+0000 7fa46b5dc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T09:59:42.005 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:42.006 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:42.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.128+0000 7f7db557e700 1 Processor -- start 2023-10-16T09:59:42.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.128+0000 7f7db557e700 1 -- start start 2023-10-16T09:59:42.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7db557e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7db00cf610 0x7f7db00cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7db557e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db006a8d0 0x7f7db006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7db557e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00afb60 con 0x7f7db0160e20 2023-10-16T09:59:42.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7db557e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db006b280 con 0x7f7db00cf610 2023-10-16T09:59:42.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7db557e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cdc80 con 0x7f7db006a8d0 2023-10-16T09:59:42.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7dae7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7db00cf610 0x7f7db00cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7dae7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7db00cf610 0x7f7db00cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53056/0 (socket says 172.21.15.158:53056) 2023-10-16T09:59:42.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.129+0000 7f7dae7fc700 1 -- 172.21.15.158:0/3483399176 learned_addr learned my addr 172.21.15.158:0/3483399176 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:42.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7daf7fe700 1 --2- 172.21.15.158:0/3483399176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db006a8d0 0x7f7db006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dae7fc700 1 -- 172.21.15.158:0/3483399176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db006a8d0 msgr2=0x7f7db006ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:42.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dae7fc700 1 --2- 172.21.15.158:0/3483399176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db006a8d0 0x7f7db006ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dae7fc700 1 -- 172.21.15.158:0/3483399176 >> v1:172.21.15.158:6789/0 conn(0x7f7db0160e20 legacy=0x7f7db00ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dae7fc700 1 -- 172.21.15.158:0/3483399176 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db00cfa70 con 0x7f7db00cf610 2023-10-16T09:59:42.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7daf7fe700 1 --2- 172.21.15.158:0/3483399176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db006a8d0 0x7f7db006ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:42.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dae7fc700 1 --2- 172.21.15.158:0/3483399176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00cf610 0x7f7db00cd740 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7da00098f0 tx=0x7f7da0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac28f5c6b9d892e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:42.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dadffb700 1 -- 172.21.15.158:0/3483399176 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000b8a0 con 0x7f7db00cf610 2023-10-16T09:59:42.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dadffb700 1 -- 172.21.15.158:0/3483399176 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da000ba00 con 0x7f7db00cf610 2023-10-16T09:59:42.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.130+0000 7f7dadffb700 1 -- 172.21.15.158:0/3483399176 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000bd90 con 0x7f7db00cf610 2023-10-16T09:59:42.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 -- 172.21.15.158:0/3483399176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00cf610 msgr2=0x7f7db00cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 --2- 172.21.15.158:0/3483399176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00cf610 0x7f7db00cd740 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7da00098f0 tx=0x7f7da0009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:42.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 -- 172.21.15.158:0/3483399176 shutdown_connections 2023-10-16T09:59:42.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 --2- 172.21.15.158:0/3483399176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db006a8d0 0x7f7db006ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 --2- 172.21.15.158:0/3483399176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00cf610 0x7f7db00cd740 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 -- 172.21.15.158:0/3483399176 >> 172.21.15.158:0/3483399176 conn(0x7f7db00d03a0 msgr2=0x7f7db00d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:42.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 -- 172.21.15.158:0/3483399176 shutdown_connections 2023-10-16T09:59:42.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.131+0000 7f7db557e700 1 -- 172.21.15.158:0/3483399176 wait complete. 2023-10-16T09:59:42.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.132+0000 7f7db557e700 1 Processor -- start 2023-10-16T09:59:42.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.132+0000 7f7db557e700 1 -- start start 2023-10-16T09:59:42.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7db557e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db006a8d0 0x7f7db01f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7db557e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00cf610 0x7f7db00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7db557e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db0160e20 0x7f7db00b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7db557e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db0159d20 con 0x7f7db0160e20 2023-10-16T09:59:42.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7db557e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00b4af0 con 0x7f7db006a8d0 2023-10-16T09:59:42.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7db557e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00b4dd0 con 0x7f7db00cf610 2023-10-16T09:59:42.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7dae7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00cf610 0x7f7db00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7daeffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db006a8d0 0x7f7db01f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7daeffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db006a8d0 0x7f7db01f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:54240/0 (socket says 172.21.15.158:54240) 2023-10-16T09:59:42.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7daeffd700 1 -- 172.21.15.158:0/438478819 learned_addr learned my addr 172.21.15.158:0/438478819 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:42.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.133+0000 7f7daf7fe700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db0160e20 0x7f7db00b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.134+0000 7f7daeffd700 1 -- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00cf610 msgr2=0x7f7db00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.134+0000 7f7daeffd700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00cf610 0x7f7db00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.134+0000 7f7daeffd700 1 -- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db0160e20 msgr2=0x7f7db00b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.134+0000 7f7daeffd700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db0160e20 0x7f7db00b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.134+0000 7f7daeffd700 1 -- 172.21.15.158:0/438478819 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da0009550 con 0x7f7db006a8d0 2023-10-16T09:59:42.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.134+0000 7f7dae7fc700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00cf610 0x7f7db00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:42.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.134+0000 7f7daf7fe700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db0160e20 0x7f7db00b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:42.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.136+0000 7f7daeffd700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db006a8d0 0x7f7db01f25c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c430 tx=0x7f7da400c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:42.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.136+0000 7f7d977fe700 1 -- 172.21.15.158:0/438478819 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da40052d0 con 0x7f7db006a8d0 2023-10-16T09:59:42.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.136+0000 7f7d977fe700 1 -- 172.21.15.158:0/438478819 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da4005450 con 0x7f7db006a8d0 2023-10-16T09:59:42.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.136+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db00b50b0 con 0x7f7db006a8d0 2023-10-16T09:59:42.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.136+0000 7f7d977fe700 1 -- 172.21.15.158:0/438478819 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4006aa0 con 0x7f7db006a8d0 2023-10-16T09:59:42.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.136+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db00b5670 con 0x7f7db006a8d0 2023-10-16T09:59:42.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.137+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db00681b0 con 0x7f7db006a8d0 2023-10-16T09:59:42.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.138+0000 7f7d977fe700 1 -- 172.21.15.158:0/438478819 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7da4003680 con 0x7f7db006a8d0 2023-10-16T09:59:42.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.138+0000 7f7d977fe700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 0x7f7d9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.138+0000 7f7dae7fc700 1 -- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 msgr2=0x7f7d9c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:42.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.138+0000 7f7d977fe700 1 -- 172.21.15.158:0/438478819 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f7da4088c90 con 0x7f7db006a8d0 2023-10-16T09:59:42.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.138+0000 7f7dae7fc700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 0x7f7d9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:42.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.143+0000 7f7d977fe700 1 -- 172.21.15.158:0/438478819 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7da4053880 con 0x7f7db006a8d0 2023-10-16T09:59:42.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.339+0000 7f7dae7fc700 1 -- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 msgr2=0x7f7d9c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:42.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.339+0000 7f7dae7fc700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 0x7f7d9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:42.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.386+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7db00cef10 con 0x7f7db006a8d0 2023-10-16T09:59:42.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.387+0000 7f7d977fe700 1 -- 172.21.15.158:0/438478819 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f7da4056ea0 con 0x7f7db006a8d0 2023-10-16T09:59:42.389 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:42.389 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3413925509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3413925509},{"type":"v1","addr":"172.21.15.158:6835","nonce":3413925509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/655082434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":655082434},{"type":"v1","addr":"172.21.15.158:6837","nonce":655082434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2666705933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2666705933},{"type":"v1","addr":"172.21.15.172:6835","nonce":2666705933}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:23.515854+0000","modified":"2023-10-16T09:59:26.532765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35639":{"gid":35639,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3389815719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3389815719},{"type":"v1","addr":"172.21.15.172:6834","nonce":3389815719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T09:59:42.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 msgr2=0x7f7d9c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:42.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 0x7f7d9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db006a8d0 msgr2=0x7f7db01f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db006a8d0 0x7f7db01f25c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c430 tx=0x7f7da400c740 comp rx=0 tx=0).stop 2023-10-16T09:59:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 shutdown_connections 2023-10-16T09:59:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db0160e20 0x7f7db00b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f7d9c06b020 0x7f7d9c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7db006a8d0 0x7f7db01f25c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 --2- 172.21.15.158:0/438478819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7db00cf610 0x7f7db00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 >> 172.21.15.158:0/438478819 conn(0x7f7db00d03a0 msgr2=0x7f7db00adb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 shutdown_connections 2023-10-16T09:59:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.390+0000 7f7db557e700 1 -- 172.21.15.158:0/438478819 wait complete. 2023-10-16T09:59:42.394 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 33 2023-10-16T09:59:42.404 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:42.508 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:42.507+0000 7f5459781200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:42.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.528+0000 7ff613201700 1 Processor -- start 2023-10-16T09:59:42.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.528+0000 7ff613201700 1 -- start start 2023-10-16T09:59:42.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.528+0000 7ff613201700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff60c15a6c0 0x7ff60c0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff613201700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c0af590 0x7ff60c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff613201700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff60c06dec0 0x7ff60c15e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff613201700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c0ca7e0 con 0x7ff60c15a6c0 2023-10-16T09:59:42.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff613201700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c06e340 con 0x7ff60c06dec0 2023-10-16T09:59:42.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff613201700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c072910 con 0x7ff60c0af590 2023-10-16T09:59:42.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff60c15a6c0 0x7ff60c0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff60c15a6c0 0x7ff60c0723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53070/0 (socket says 172.21.15.158:53070) 2023-10-16T09:59:42.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 -- 172.21.15.158:0/3538953797 learned_addr learned my addr 172.21.15.158:0/3538953797 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:42.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff61179e700 1 --2- 172.21.15.158:0/3538953797 >> v2:172.21.15.172:3300/0 conn(0x7ff60c06dec0 0x7ff60c15e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff60bfff700 1 --2- 172.21.15.158:0/3538953797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c0af590 0x7ff60c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 -- 172.21.15.158:0/3538953797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c0af590 msgr2=0x7ff60c06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:42.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 --2- 172.21.15.158:0/3538953797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c0af590 0x7ff60c06d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 -- 172.21.15.158:0/3538953797 >> v2:172.21.15.172:3300/0 conn(0x7ff60c06dec0 msgr2=0x7ff60c15e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:42.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 --2- 172.21.15.158:0/3538953797 >> v2:172.21.15.172:3300/0 conn(0x7ff60c06dec0 0x7ff60c15e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 -- 172.21.15.158:0/3538953797 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0b1e40 con 0x7ff60c15a6c0 2023-10-16T09:59:42.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff610f9d700 1 --2- 172.21.15.158:0/3538953797 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c15a6c0 0x7ff60c0723d0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff60c09de90 tx=0x7ff600009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5974790e75e345 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:42.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff60b7fe700 1 -- 172.21.15.158:0/3538953797 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600005e30 con 0x7ff60c15a6c0 2023-10-16T09:59:42.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff60b7fe700 1 -- 172.21.15.158:0/3538953797 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff600004030 con 0x7ff60c15a6c0 2023-10-16T09:59:42.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.529+0000 7ff60b7fe700 1 -- 172.21.15.158:0/3538953797 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000043e0 con 0x7ff60c15a6c0 2023-10-16T09:59:42.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 -- 172.21.15.158:0/3538953797 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c15a6c0 msgr2=0x7ff60c0723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 --2- 172.21.15.158:0/3538953797 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c15a6c0 0x7ff60c0723d0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff60c09de90 tx=0x7ff600009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 -- 172.21.15.158:0/3538953797 shutdown_connections 2023-10-16T09:59:42.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 --2- 172.21.15.158:0/3538953797 >> v2:172.21.15.172:3300/0 conn(0x7ff60c06dec0 0x7ff60c15e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 --2- 172.21.15.158:0/3538953797 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c15a6c0 0x7ff60c0723d0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 --2- 172.21.15.158:0/3538953797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c0af590 0x7ff60c06d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 -- 172.21.15.158:0/3538953797 >> 172.21.15.158:0/3538953797 conn(0x7ff60c0ccda0 msgr2=0x7ff60c0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:42.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 -- 172.21.15.158:0/3538953797 shutdown_connections 2023-10-16T09:59:42.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.530+0000 7ff613201700 1 -- 172.21.15.158:0/3538953797 wait complete. 2023-10-16T09:59:42.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.532+0000 7ff613201700 1 Processor -- start 2023-10-16T09:59:42.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.532+0000 7ff613201700 1 -- start start 2023-10-16T09:59:42.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff613201700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c06dec0 0x7ff60c151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff613201700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff60c0af590 0x7ff60c1f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff613201700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c15a6c0 0x7ff60c1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff613201700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0b0800 con 0x7ff60c06dec0 2023-10-16T09:59:42.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff613201700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1f3ed0 con 0x7ff60c0af590 2023-10-16T09:59:42.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff613201700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0afbf0 con 0x7ff60c15a6c0 2023-10-16T09:59:42.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c06dec0 0x7ff60c151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c06dec0 0x7ff60c151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53076/0 (socket says 172.21.15.158:53076) 2023-10-16T09:59:42.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 -- 172.21.15.158:0/3659017223 learned_addr learned my addr 172.21.15.158:0/3659017223 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:42.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff61179e700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c15a6c0 0x7ff60c1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff60bfff700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff60c0af590 0x7ff60c1f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 -- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c15a6c0 msgr2=0x7ff60c1f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c15a6c0 0x7ff60c1f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 -- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff60c0af590 msgr2=0x7ff60c1f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff60c0af590 0x7ff60c1f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 -- 172.21.15.158:0/3659017223 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600009500 con 0x7ff60c06dec0 2023-10-16T09:59:42.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff610f9d700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c06dec0 0x7ff60c151420 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7ff600009c10 tx=0x7ff600004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:42.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.533+0000 7ff6097fa700 1 -- 172.21.15.158:0/3659017223 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600003bb0 con 0x7ff60c06dec0 2023-10-16T09:59:42.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.534+0000 7ff6097fa700 1 -- 172.21.15.158:0/3659017223 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff600003d10 con 0x7ff60c06dec0 2023-10-16T09:59:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.534+0000 7ff6097fa700 1 -- 172.21.15.158:0/3659017223 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60001c660 con 0x7ff60c06dec0 2023-10-16T09:59:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.534+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c0afed0 con 0x7ff60c06dec0 2023-10-16T09:59:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.534+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c0b0490 con 0x7ff60c06dec0 2023-10-16T09:59:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.535+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c0b0650 con 0x7ff60c06dec0 2023-10-16T09:59:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.535+0000 7ff6097fa700 1 -- 172.21.15.158:0/3659017223 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff60001c7c0 con 0x7ff60c06dec0 2023-10-16T09:59:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.536+0000 7ff6097fa700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 0x7ff5f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.536+0000 7ff60bfff700 1 -- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 msgr2=0x7ff5f006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:42.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.536+0000 7ff6097fa700 1 -- 172.21.15.158:0/3659017223 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7ff60008a9b0 con 0x7ff60c06dec0 2023-10-16T09:59:42.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.536+0000 7ff60bfff700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 0x7ff5f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:42.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.540+0000 7ff6097fa700 1 -- 172.21.15.158:0/3659017223 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6000555a0 con 0x7ff60c06dec0 2023-10-16T09:59:42.586 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:42.585+0000 7f5459781200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:59:42.660 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:42.658+0000 7fa46b5dc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:42.729 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:42.728+0000 7f5459781200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:42.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.737+0000 7ff60bfff700 1 -- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 msgr2=0x7ff5f006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:42.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.737+0000 7ff60bfff700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 0x7ff5f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:42.738 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:42.737+0000 7fa46b5dc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T09:59:42.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.740+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff60c0681b0 con 0x7ff60c06dec0 2023-10-16T09:59:42.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.741+0000 7ff6097fa700 1 -- 172.21.15.158:0/3659017223 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17880 (secure 0 0 0) 0x7ff600058bc0 con 0x7ff60c06dec0 2023-10-16T09:59:42.743 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:42.743 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":86,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:37.743739+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:21.428579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:22.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3100922361":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6833/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6832/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/3345169280":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/4171724344":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/310788915":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/2329993263":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/234410797":"2023-10-17T09:59:37.743702+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:42.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 msgr2=0x7ff5f006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:42.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 0x7ff5f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c06dec0 msgr2=0x7ff60c151420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c06dec0 0x7ff60c151420 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7ff600009c10 tx=0x7ff600004750 comp rx=0 tx=0).stop 2023-10-16T09:59:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 shutdown_connections 2023-10-16T09:59:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff60c15a6c0 0x7ff60c1f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff60c06dec0 0x7ff60c151420 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7ff5f006b020 0x7ff5f006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 --2- 172.21.15.158:0/3659017223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff60c0af590 0x7ff60c1f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 >> 172.21.15.158:0/3659017223 conn(0x7ff60c0ccda0 msgr2=0x7ff60c0cbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 shutdown_connections 2023-10-16T09:59:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.744+0000 7ff613201700 1 -- 172.21.15.158:0/3659017223 wait complete. 2023-10-16T09:59:42.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:59:21.428579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:59:22.493630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:42.759 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:42.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.890+0000 7f135f337700 1 Processor -- start 2023-10-16T09:59:42.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.891+0000 7f135f337700 1 -- start start 2023-10-16T09:59:42.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.891+0000 7f135f337700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815d6c0 0x7f13580cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.891+0000 7f135f337700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13580d1ee0 con 0x7f13580cf360 2023-10-16T09:59:42.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.891+0000 7f135f337700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13580d12d0 con 0x7f135815daa0 2023-10-16T09:59:42.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.891+0000 7f135f337700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13580d40a0 con 0x7f135815d6c0 2023-10-16T09:59:42.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.891+0000 7f135d8d4700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f13580cf360 0x7f135806b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53308/0 (socket says 172.21.15.158:53308) 2023-10-16T09:59:42.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f135d8d4700 1 -- 172.21.15.158:0/2150968515 learned_addr learned my addr 172.21.15.158:0/2150968515 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:42.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f135d0d3700 1 --2- 172.21.15.158:0/2150968515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815d6c0 0x7f13580cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f134ffff700 1 -- 172.21.15.158:0/2150968515 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186193501 0 0) 0x7f13580d1ee0 con 0x7f13580cf360 2023-10-16T09:59:42.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f134ffff700 1 -- 172.21.15.158:0/2150968515 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13440035b0 con 0x7f13580cf360 2023-10-16T09:59:42.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f134ffff700 1 -- 172.21.15.158:0/2150968515 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331100191 0 0) 0x7f13580d12d0 con 0x7f135815daa0 2023-10-16T09:59:42.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f134ffff700 1 -- 172.21.15.158:0/2150968515 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13580d1ee0 con 0x7f135815daa0 2023-10-16T09:59:42.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f135d0d3700 1 -- 172.21.15.158:0/2150968515 >> v1:172.21.15.172:6789/0 conn(0x7f135815daa0 legacy=0x7f13580cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f135d0d3700 1 -- 172.21.15.158:0/2150968515 >> v1:172.21.15.158:6789/0 conn(0x7f13580cf360 legacy=0x7f135806b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f135d0d3700 1 -- 172.21.15.158:0/2150968515 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580d15b0 con 0x7f135815d6c0 2023-10-16T09:59:42.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f135d0d3700 1 --2- 172.21.15.158:0/2150968515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815d6c0 0x7f13580cff30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1354009870 tx=0x7f1354009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3da635365f4707ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:42.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f134ffff700 1 -- 172.21.15.158:0/2150968515 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354011830 con 0x7f135815d6c0 2023-10-16T09:59:42.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f134ffff700 1 -- 172.21.15.158:0/2150968515 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354011990 con 0x7f135815d6c0 2023-10-16T09:59:42.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.892+0000 7f134ffff700 1 -- 172.21.15.158:0/2150968515 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354011d20 con 0x7f135815d6c0 2023-10-16T09:59:42.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.893+0000 7f135f337700 1 -- 172.21.15.158:0/2150968515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815d6c0 msgr2=0x7f13580cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.893+0000 7f135f337700 1 --2- 172.21.15.158:0/2150968515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815d6c0 0x7f13580cff30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1354009870 tx=0x7f1354009b80 comp rx=0 tx=0).stop 2023-10-16T09:59:42.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.893+0000 7f135f337700 1 -- 172.21.15.158:0/2150968515 shutdown_connections 2023-10-16T09:59:42.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.893+0000 7f135f337700 1 --2- 172.21.15.158:0/2150968515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815d6c0 0x7f13580cff30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.893+0000 7f135f337700 1 -- 172.21.15.158:0/2150968515 >> 172.21.15.158:0/2150968515 conn(0x7f135809c940 msgr2=0x7f13580d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:42.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.893+0000 7f135f337700 1 -- 172.21.15.158:0/2150968515 shutdown_connections 2023-10-16T09:59:42.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.893+0000 7f135f337700 1 -- 172.21.15.158:0/2150968515 wait complete. 2023-10-16T09:59:42.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.894+0000 7f135f337700 1 Processor -- start 2023-10-16T09:59:42.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.894+0000 7f135f337700 1 -- start start 2023-10-16T09:59:42.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135f337700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13580cf360 0x7f1358160b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135f337700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815d6c0 0x7f135815e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135f337700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815daa0 0x7f135806c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135f337700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13581606e0 con 0x7f13580cf360 2023-10-16T09:59:42.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135f337700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135815fe10 con 0x7f135815d6c0 2023-10-16T09:59:42.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135f337700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1358144050 con 0x7f135815daa0 2023-10-16T09:59:42.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13580cf360 0x7f1358160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13580cf360 0x7f1358160b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53134/0 (socket says 172.21.15.158:53134) 2023-10-16T09:59:42.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 -- 172.21.15.158:0/589087480 learned_addr learned my addr 172.21.15.158:0/589087480 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:42.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135c8d2700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815d6c0 0x7f135815e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d8d4700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815daa0 0x7f135806c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:42.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 -- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815daa0 msgr2=0x7f135806c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815daa0 0x7f135806c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 -- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815d6c0 msgr2=0x7f135815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815d6c0 0x7f135815e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.895+0000 7f135d0d3700 1 -- 172.21.15.158:0/589087480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13540094a0 con 0x7f13580cf360 2023-10-16T09:59:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.896+0000 7f135d0d3700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13580cf360 0x7f1358160b70 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f13540097c0 tx=0x7f1354004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.896+0000 7f134dffb700 1 -- 172.21.15.158:0/589087480 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135401a8f0 con 0x7f13580cf360 2023-10-16T09:59:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.896+0000 7f134dffb700 1 -- 172.21.15.158:0/589087480 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135401aa50 con 0x7f13580cf360 2023-10-16T09:59:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.896+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1358160090 con 0x7f13580cf360 2023-10-16T09:59:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.896+0000 7f134dffb700 1 -- 172.21.15.158:0/589087480 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135401adc0 con 0x7f13580cf360 2023-10-16T09:59:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.896+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135815e890 con 0x7f13580cf360 2023-10-16T09:59:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.897+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13581609c0 con 0x7f13580cf360 2023-10-16T09:59:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.897+0000 7f134dffb700 1 -- 172.21.15.158:0/589087480 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f13540243f0 con 0x7f13580cf360 2023-10-16T09:59:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.898+0000 7f134dffb700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:42.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.898+0000 7f135c8d2700 1 -- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 msgr2=0x7f133c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:42.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.898+0000 7f134dffb700 1 -- 172.21.15.158:0/589087480 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f135408b1c0 con 0x7f13580cf360 2023-10-16T09:59:42.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.898+0000 7f135c8d2700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:42.907 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:42.899+0000 7fa46b5dc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T09:59:42.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:42.902+0000 7f134dffb700 1 -- 172.21.15.158:0/589087480 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1354055ed0 con 0x7f13580cf360 2023-10-16T09:59:42.958 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:42.957+0000 7f5459781200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:59:43.031 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:43.030+0000 7f5459781200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:59:43.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.099+0000 7f135c8d2700 1 -- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 msgr2=0x7f133c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:43.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.099+0000 7f135c8d2700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:43.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.102+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13580681b0 con 0x7f13580cf360 2023-10-16T09:59:43.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.103+0000 7f134dffb700 1 -- 172.21.15.158:0/589087480 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13540594f0 con 0x7f13580cf360 2023-10-16T09:59:43.104 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:43.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 msgr2=0x7f133c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:43.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13580cf360 msgr2=0x7f1358160b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13580cf360 0x7f1358160b70 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f13540097c0 tx=0x7f1354004750 comp rx=0 tx=0).stop 2023-10-16T09:59:43.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 shutdown_connections 2023-10-16T09:59:43.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f135815daa0 0x7f135806c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13580cf360 0x7f1358160b70 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f133c06b020 0x7f133c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 --2- 172.21.15.158:0/589087480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f135815d6c0 0x7f135815e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 >> 172.21.15.158:0/589087480 conn(0x7f135809c940 msgr2=0x7f135815c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:43.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 shutdown_connections 2023-10-16T09:59:43.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.106+0000 7f135f337700 1 -- 172.21.15.158:0/589087480 wait complete. 2023-10-16T09:59:43.120 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T09:59:43.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:43.137+0000 7fa46b5dc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T09:59:43.217 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:43.216+0000 7fa46b5dc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T09:59:43.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.246+0000 7f5a7034d700 1 Processor -- start 2023-10-16T09:59:43.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.246+0000 7f5a7034d700 1 -- start start 2023-10-16T09:59:43.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.247+0000 7f5a7034d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5a6806f180 0x7f5a6806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.247+0000 7f5a7034d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6806fea0 0x7f5a6806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.247+0000 7f5a7034d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6815ee60 con 0x7f5a6815b540 2023-10-16T09:59:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.247+0000 7f5a7034d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6815b920 con 0x7f5a6806f180 2023-10-16T09:59:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.247+0000 7f5a7034d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6815e2e0 con 0x7f5a6806fea0 2023-10-16T09:59:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6806fea0 0x7f5a6806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6d8e8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5a6806f180 0x7f5a6806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6806fea0 0x7f5a6806ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:35750/0 (socket says 172.21.15.158:35750) 2023-10-16T09:59:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6d8e8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5a6806f180 0x7f5a6806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:54326/0 (socket says 172.21.15.158:54326) 2023-10-16T09:59:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 -- 172.21.15.158:0/1625306614 learned_addr learned my addr 172.21.15.158:0/1625306614 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 -- 172.21.15.158:0/1625306614 >> v2:172.21.15.172:3300/0 conn(0x7f5a6806f180 msgr2=0x7f5a6806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 --2- 172.21.15.158:0/1625306614 >> v2:172.21.15.172:3300/0 conn(0x7f5a6806f180 0x7f5a6806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 -- 172.21.15.158:0/1625306614 >> v1:172.21.15.172:6789/0 conn(0x7f5a6815b540 legacy=0x7f5a6806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 -- 172.21.15.158:0/1625306614 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6815e5c0 con 0x7f5a6806fea0 2023-10-16T09:59:43.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6d8e8700 1 --2- 172.21.15.158:0/1625306614 >> v2:172.21.15.172:3300/0 conn(0x7f5a6806f180 0x7f5a6806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:43.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6e8ea700 1 --2- 172.21.15.158:0/1625306614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6806fea0 0x7f5a6806ae10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009e50 tx=0x7f5a6000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb0704a4c76ae5cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:43.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6d0e7700 1 -- 172.21.15.158:0/1625306614 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000e070 con 0x7f5a6806fea0 2023-10-16T09:59:43.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6d0e7700 1 -- 172.21.15.158:0/1625306614 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000b910 con 0x7f5a6806fea0 2023-10-16T09:59:43.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.248+0000 7f5a6d0e7700 1 -- 172.21.15.158:0/1625306614 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000bcc0 con 0x7f5a6806fea0 2023-10-16T09:59:43.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 -- 172.21.15.158:0/1625306614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6806fea0 msgr2=0x7f5a6806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 --2- 172.21.15.158:0/1625306614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6806fea0 0x7f5a6806ae10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009e50 tx=0x7f5a6000b330 comp rx=0 tx=0).stop 2023-10-16T09:59:43.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 -- 172.21.15.158:0/1625306614 shutdown_connections 2023-10-16T09:59:43.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 --2- 172.21.15.158:0/1625306614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6806fea0 0x7f5a6806ae10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 --2- 172.21.15.158:0/1625306614 >> v2:172.21.15.172:3300/0 conn(0x7f5a6806f180 0x7f5a6806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 -- 172.21.15.158:0/1625306614 >> 172.21.15.158:0/1625306614 conn(0x7f5a680d03b0 msgr2=0x7f5a680d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:43.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 -- 172.21.15.158:0/1625306614 shutdown_connections 2023-10-16T09:59:43.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.249+0000 7f5a7034d700 1 -- 172.21.15.158:0/1625306614 wait complete. 2023-10-16T09:59:43.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.250+0000 7f5a7034d700 1 Processor -- start 2023-10-16T09:59:43.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.250+0000 7f5a7034d700 1 -- start start 2023-10-16T09:59:43.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a7034d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a6806f180 0x7f5a6814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a7034d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a6806fea0 0x7f5a681603e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a7034d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6815b540 0x7f5a680b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a7034d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680ad840 con 0x7f5a6806f180 2023-10-16T09:59:43.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a7034d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680acbc0 con 0x7f5a6806fea0 2023-10-16T09:59:43.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a7034d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680acea0 con 0x7f5a6815b540 2023-10-16T09:59:43.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a6e0e9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a6806f180 0x7f5a6814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a6e0e9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a6806f180 0x7f5a6814cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53150/0 (socket says 172.21.15.158:53150) 2023-10-16T09:59:43.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a6e0e9700 1 -- 172.21.15.158:0/193714133 learned_addr learned my addr 172.21.15.158:0/193714133 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:43.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a6d8e8700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a6806fea0 0x7f5a681603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.251+0000 7f5a6e8ea700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6815b540 0x7f5a680b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a6e0e9700 1 -- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6815b540 msgr2=0x7f5a680b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a6e0e9700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6815b540 0x7f5a680b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a6e0e9700 1 -- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a6806fea0 msgr2=0x7f5a681603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a6e0e9700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a6806fea0 0x7f5a681603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a6e0e9700 1 -- 172.21.15.158:0/193714133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a580094a0 con 0x7f5a6806f180 2023-10-16T09:59:43.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a6e0e9700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a6806f180 0x7f5a6814cb10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c450 tx=0x7f5a6400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:43.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a5effd700 1 -- 172.21.15.158:0/193714133 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a640052d0 con 0x7f5a6806f180 2023-10-16T09:59:43.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a5effd700 1 -- 172.21.15.158:0/193714133 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a64005430 con 0x7f5a6806f180 2023-10-16T09:59:43.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a60009b30 con 0x7f5a6806f180 2023-10-16T09:59:43.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a5effd700 1 -- 172.21.15.158:0/193714133 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a640038c0 con 0x7f5a6806f180 2023-10-16T09:59:43.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.252+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a680ad460 con 0x7f5a6806f180 2023-10-16T09:59:43.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.253+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a680681b0 con 0x7f5a6806f180 2023-10-16T09:59:43.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.253+0000 7f5a5effd700 1 -- 172.21.15.158:0/193714133 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f5a64003a20 con 0x7f5a6806f180 2023-10-16T09:59:43.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.254+0000 7f5a5effd700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 0x7f5a4c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.254+0000 7f5a6d8e8700 1 -- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 msgr2=0x7f5a4c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:43.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.254+0000 7f5a5effd700 1 -- 172.21.15.158:0/193714133 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f5a64088e30 con 0x7f5a6806f180 2023-10-16T09:59:43.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.254+0000 7f5a6d8e8700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 0x7f5a4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:43.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.258+0000 7f5a5effd700 1 -- 172.21.15.158:0/193714133 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a64053a20 con 0x7f5a6806f180 2023-10-16T09:59:43.298 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:43.297+0000 7f5459781200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:59:43.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.455+0000 7f5a6d8e8700 1 -- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 msgr2=0x7f5a4c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:43.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.455+0000 7f5a6d8e8700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 0x7f5a4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:43.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.469+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5a6815de70 con 0x7f5a6806f180 2023-10-16T09:59:43.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.470+0000 7f5a5effd700 1 -- 172.21.15.158:0/193714133 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5a64017020 con 0x7f5a6806f180 2023-10-16T09:59:43.471 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T09:59:43.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 msgr2=0x7f5a4c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:43.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 0x7f5a4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a6806f180 msgr2=0x7f5a6814cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a6806f180 0x7f5a6814cb10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c450 tx=0x7f5a6400c760 comp rx=0 tx=0).stop 2023-10-16T09:59:43.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 shutdown_connections 2023-10-16T09:59:43.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a6815b540 0x7f5a680b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a6806f180 0x7f5a6814cb10 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f5a4c06b020 0x7f5a4c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 --2- 172.21.15.158:0/193714133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a6806fea0 0x7f5a681603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 >> 172.21.15.158:0/193714133 conn(0x7f5a680d03b0 msgr2=0x7f5a680d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 shutdown_connections 2023-10-16T09:59:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.472+0000 7f5a7034d700 1 -- 172.21.15.158:0/193714133 wait complete. 2023-10-16T09:59:43.486 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T09:59:43.502 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:43.501+0000 7fa46b5dc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T09:59:43.516 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:43.515+0000 7f5459781200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:59:43.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.627+0000 7f311bf22700 1 Processor -- start 2023-10-16T09:59:43.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.627+0000 7f311bf22700 1 -- start start 2023-10-16T09:59:43.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f311bf22700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f311415dad0 0x7f31140cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f311bf22700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f311415deb0 0x7f31140ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f311bf22700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31140d0ae0 con 0x7f31140cea20 2023-10-16T09:59:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f311bf22700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31140cee50 con 0x7f311415dad0 2023-10-16T09:59:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f311bf22700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31140d12d0 con 0x7f311415deb0 2023-10-16T09:59:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f311415dad0 0x7f31140cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f311415dad0 0x7f31140cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53160/0 (socket says 172.21.15.158:53160) 2023-10-16T09:59:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 -- 172.21.15.158:0/2930111571 learned_addr learned my addr 172.21.15.158:0/2930111571 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f31194bd700 1 --2- 172.21.15.158:0/2930111571 >> v2:172.21.15.172:3300/0 conn(0x7f311415deb0 0x7f31140ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 -- 172.21.15.158:0/2930111571 >> v2:172.21.15.172:3300/0 conn(0x7f311415deb0 msgr2=0x7f31140ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 --2- 172.21.15.158:0/2930111571 >> v2:172.21.15.172:3300/0 conn(0x7f311415deb0 0x7f31140ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 -- 172.21.15.158:0/2930111571 >> v1:172.21.15.172:6789/0 conn(0x7f31140cea20 legacy=0x7f31140c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 -- 172.21.15.158:0/2930111571 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31140cbdf0 con 0x7f311415dad0 2023-10-16T09:59:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3119cbe700 1 --2- 172.21.15.158:0/2930111571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311415dad0 0x7f31140cdfa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f31100098f0 tx=0x7f3110009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7656d11ea258d1b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3118cbc700 1 -- 172.21.15.158:0/2930111571 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311000b860 con 0x7f311415dad0 2023-10-16T09:59:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3118cbc700 1 -- 172.21.15.158:0/2930111571 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311000b9c0 con 0x7f311415dad0 2023-10-16T09:59:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.628+0000 7f3118cbc700 1 -- 172.21.15.158:0/2930111571 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311000bd50 con 0x7f311415dad0 2023-10-16T09:59:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 -- 172.21.15.158:0/2930111571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311415dad0 msgr2=0x7f31140cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 --2- 172.21.15.158:0/2930111571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311415dad0 0x7f31140cdfa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f31100098f0 tx=0x7f3110009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 -- 172.21.15.158:0/2930111571 shutdown_connections 2023-10-16T09:59:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 --2- 172.21.15.158:0/2930111571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311415dad0 0x7f31140cdfa0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 --2- 172.21.15.158:0/2930111571 >> v2:172.21.15.172:3300/0 conn(0x7f311415deb0 0x7f31140ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 -- 172.21.15.158:0/2930111571 >> 172.21.15.158:0/2930111571 conn(0x7f3114099a10 msgr2=0x7f31140d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 -- 172.21.15.158:0/2930111571 shutdown_connections 2023-10-16T09:59:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.629+0000 7f311bf22700 1 -- 172.21.15.158:0/2930111571 wait complete. 2023-10-16T09:59:43.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.630+0000 7f311bf22700 1 Processor -- start 2023-10-16T09:59:43.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.630+0000 7f311bf22700 1 -- start start 2023-10-16T09:59:43.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f311bf22700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31140cea20 0x7f31141e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f311bf22700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311415dad0 0x7f31141e2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f311bf22700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311415deb0 0x7f31140b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f311bf22700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31140c4a40 con 0x7f31140cea20 2023-10-16T09:59:43.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f311bf22700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31140c4c90 con 0x7f311415deb0 2023-10-16T09:59:43.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f311bf22700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31140c4f70 con 0x7f311415dad0 2023-10-16T09:59:43.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f3119cbe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31140cea20 0x7f31141e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f3119cbe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31140cea20 0x7f31141e5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53166/0 (socket says 172.21.15.158:53166) 2023-10-16T09:59:43.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f3119cbe700 1 -- 172.21.15.158:0/775652707 learned_addr learned my addr 172.21.15.158:0/775652707 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:43.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f31194bd700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311415dad0 0x7f31141e2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.631+0000 7f311a4bf700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311415deb0 0x7f31140b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f3119cbe700 1 -- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311415dad0 msgr2=0x7f31141e2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f3119cbe700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311415dad0 0x7f31141e2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f3119cbe700 1 -- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311415deb0 msgr2=0x7f31140b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f3119cbe700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311415deb0 0x7f31140b54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f3119cbe700 1 -- 172.21.15.158:0/775652707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110009550 con 0x7f31140cea20 2023-10-16T09:59:43.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f31194bd700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311415dad0 0x7f31141e2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:43.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f3119cbe700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31140cea20 0x7f31141e5df0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3110009c40 tx=0x7f3110004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:43.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f310a7fc700 1 -- 172.21.15.158:0/775652707 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110004380 con 0x7f31140cea20 2023-10-16T09:59:43.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f310a7fc700 1 -- 172.21.15.158:0/775652707 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311001f070 con 0x7f31140cea20 2023-10-16T09:59:43.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f310a7fc700 1 -- 172.21.15.158:0/775652707 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31100036a0 con 0x7f31140cea20 2023-10-16T09:59:43.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31140c5250 con 0x7f31140cea20 2023-10-16T09:59:43.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.632+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31141e34e0 con 0x7f31140cea20 2023-10-16T09:59:43.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.633+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31141e36a0 con 0x7f31140cea20 2023-10-16T09:59:43.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.633+0000 7f310a7fc700 1 -- 172.21.15.158:0/775652707 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f3110021030 con 0x7f31140cea20 2023-10-16T09:59:43.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.634+0000 7f310a7fc700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 0x7f30f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.634+0000 7f31194bd700 1 -- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 msgr2=0x7f30f806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:43.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.634+0000 7f310a7fc700 1 -- 172.21.15.158:0/775652707 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f311008ae90 con 0x7f31140cea20 2023-10-16T09:59:43.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.634+0000 7f31194bd700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 0x7f30f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:43.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.638+0000 7f310a7fc700 1 -- 172.21.15.158:0/775652707 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3110055a80 con 0x7f31140cea20 2023-10-16T09:59:43.665 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:43.664+0000 7f5459781200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:59:43.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:43.734+0000 7fa46b5dc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T09:59:43.779 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:43.778+0000 7f5459781200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:59:43.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.835+0000 7f31194bd700 1 -- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 msgr2=0x7f30f806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:43.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.835+0000 7f31194bd700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 0x7f30f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:43.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.841+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f31140cc260 con 0x7f31140cea20 2023-10-16T09:59:43.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.841+0000 7f310a7fc700 1 -- 172.21.15.158:0/775652707 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f31140cc260 con 0x7f31140cea20 2023-10-16T09:59:43.843 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T09:59:43.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 msgr2=0x7f30f806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:43.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 0x7f30f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31140cea20 msgr2=0x7f31141e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31140cea20 0x7f31141e5df0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3110009c40 tx=0x7f3110004750 comp rx=0 tx=0).stop 2023-10-16T09:59:43.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 shutdown_connections 2023-10-16T09:59:43.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f30f806b020 0x7f30f806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311415deb0 0x7f31140b54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31140cea20 0x7f31141e5df0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 --2- 172.21.15.158:0/775652707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311415dad0 0x7f31141e2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 >> 172.21.15.158:0/775652707 conn(0x7f3114099a10 msgr2=0x7f31140ce080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:43.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 shutdown_connections 2023-10-16T09:59:43.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.843+0000 7f311bf22700 1 -- 172.21.15.158:0/775652707 wait complete. 2023-10-16T09:59:43.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:43.895 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:43.894+0000 7fa46b5dc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T09:59:43.933 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:43.932+0000 7f5459781200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:59:43.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.979+0000 7f4cef7bd700 1 Processor -- start 2023-10-16T09:59:43.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.980+0000 7f4cef7bd700 1 -- start start 2023-10-16T09:59:43.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cef7bd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ce80cc5d0 0x7f4ce806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cef7bd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce815dfb0 0x7f4ce80cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cef7bd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ce806f340 0x7f4ce806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cef7bd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce80cf270 con 0x7f4ce80cc5d0 2023-10-16T09:59:43.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cef7bd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce806fd10 con 0x7f4ce806f340 2023-10-16T09:59:43.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cef7bd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce806e770 con 0x7f4ce815dfb0 2023-10-16T09:59:43.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ce80cc5d0 0x7f4ce806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ce80cc5d0 0x7f4ce806e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53176/0 (socket says 172.21.15.158:53176) 2023-10-16T09:59:43.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 -- 172.21.15.158:0/3050856149 learned_addr learned my addr 172.21.15.158:0/3050856149 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:43.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cecd58700 1 --2- 172.21.15.158:0/3050856149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce815dfb0 0x7f4ce80cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cedd5a700 1 --2- 172.21.15.158:0/3050856149 >> v2:172.21.15.172:3300/0 conn(0x7f4ce806f340 0x7f4ce806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 -- 172.21.15.158:0/3050856149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce815dfb0 msgr2=0x7f4ce80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:43.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 --2- 172.21.15.158:0/3050856149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce815dfb0 0x7f4ce80cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 -- 172.21.15.158:0/3050856149 >> v2:172.21.15.172:3300/0 conn(0x7f4ce806f340 msgr2=0x7f4ce806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:43.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 --2- 172.21.15.158:0/3050856149 >> v2:172.21.15.172:3300/0 conn(0x7f4ce806f340 0x7f4ce806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 -- 172.21.15.158:0/3050856149 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce815e4e0 con 0x7f4ce80cc5d0 2023-10-16T09:59:43.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cecd58700 1 --2- 172.21.15.158:0/3050856149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce815dfb0 0x7f4ce80cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:43.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cedd5a700 1 --2- 172.21.15.158:0/3050856149 >> v2:172.21.15.172:3300/0 conn(0x7f4ce806f340 0x7f4ce806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:43.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4ced559700 1 --2- 172.21.15.158:0/3050856149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce806e230 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4ce815df10 tx=0x7f4ce4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e133c04d9460f80a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:43.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cdffff700 1 -- 172.21.15.158:0/3050856149 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400b870 con 0x7f4ce80cc5d0 2023-10-16T09:59:43.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cdffff700 1 -- 172.21.15.158:0/3050856149 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce400b9d0 con 0x7f4ce80cc5d0 2023-10-16T09:59:43.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.981+0000 7f4cdffff700 1 -- 172.21.15.158:0/3050856149 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400bd60 con 0x7f4ce80cc5d0 2023-10-16T09:59:43.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3050856149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce80cc5d0 msgr2=0x7f4ce806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3050856149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce806e230 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4ce815df10 tx=0x7f4ce4009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3050856149 shutdown_connections 2023-10-16T09:59:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3050856149 >> v2:172.21.15.172:3300/0 conn(0x7f4ce806f340 0x7f4ce806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3050856149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce806e230 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3050856149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce815dfb0 0x7f4ce80cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3050856149 >> 172.21.15.158:0/3050856149 conn(0x7f4ce80d02b0 msgr2=0x7f4ce80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3050856149 shutdown_connections 2023-10-16T09:59:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.982+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3050856149 wait complete. 2023-10-16T09:59:43.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.983+0000 7f4cef7bd700 1 Processor -- start 2023-10-16T09:59:43.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.983+0000 7f4cef7bd700 1 -- start start 2023-10-16T09:59:43.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cef7bd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce806f340 0x7f4ce8153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cef7bd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cef7bd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815dfb0 0x7f4ce80b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cef7bd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81f3110 con 0x7f4ce815dfb0 2023-10-16T09:59:43.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cef7bd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81f3320 con 0x7f4ce80cc5d0 2023-10-16T09:59:43.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cef7bd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81f3600 con 0x7f4ce806f340 2023-10-16T09:59:43.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cedd5a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815dfb0 0x7f4ce80b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cedd5a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815dfb0 0x7f4ce80b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53186/0 (socket says 172.21.15.158:53186) 2023-10-16T09:59:43.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4ced559700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce806f340 0x7f4ce8153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cecd58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:43.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.984+0000 7f4cedd5a700 1 -- 172.21.15.158:0/3035300663 learned_addr learned my addr 172.21.15.158:0/3035300663 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:43.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cedd5a700 1 -- 172.21.15.158:0/3035300663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce806f340 msgr2=0x7f4ce8153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cedd5a700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce806f340 0x7f4ce8153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cedd5a700 1 -- 172.21.15.158:0/3035300663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce80cc5d0 msgr2=0x7f4ce80b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:43.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cedd5a700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce80b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:43.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cedd5a700 1 -- 172.21.15.158:0/3035300663 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009550 con 0x7f4ce815dfb0 2023-10-16T09:59:43.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cecd58700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce80b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:43.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cedd5a700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815dfb0 0x7f4ce80b53f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0007e70 tx=0x7f4ce000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:43.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cddffb700 1 -- 172.21.15.158:0/3035300663 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce000dd20 con 0x7f4ce815dfb0 2023-10-16T09:59:43.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce81f38e0 con 0x7f4ce815dfb0 2023-10-16T09:59:43.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cddffb700 1 -- 172.21.15.158:0/3035300663 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce000f040 con 0x7f4ce815dfb0 2023-10-16T09:59:43.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cddffb700 1 -- 172.21.15.158:0/3035300663 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce000dea0 con 0x7f4ce815dfb0 2023-10-16T09:59:43.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.985+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce8153f50 con 0x7f4ce815dfb0 2023-10-16T09:59:43.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.986+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce80681b0 con 0x7f4ce815dfb0 2023-10-16T09:59:43.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.987+0000 7f4cddffb700 1 -- 172.21.15.158:0/3035300663 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f4ce0003c60 con 0x7f4ce815dfb0 2023-10-16T09:59:43.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.987+0000 7f4cddffb700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 0x7f4ccc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:43.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.987+0000 7f4ced559700 1 -- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 msgr2=0x7f4ccc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:43.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.988+0000 7f4cddffb700 1 -- 172.21.15.158:0/3035300663 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f4ce0089c50 con 0x7f4ce815dfb0 2023-10-16T09:59:43.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.988+0000 7f4ced559700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 0x7f4ccc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:43.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:43.996+0000 7f4cddffb700 1 -- 172.21.15.158:0/3035300663 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ce008f020 con 0x7f4ce815dfb0 2023-10-16T09:59:44.021 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:44.020+0000 7fa46b5dc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T09:59:44.180 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:44.179+0000 7fa46b5dc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T09:59:44.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.189+0000 7f4ced559700 1 -- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 msgr2=0x7f4ccc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:44.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.189+0000 7f4ced559700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 0x7f4ccc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:44.197 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:44.196+0000 7f5459781200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:44.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.239+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ce80af4f0 con 0x7f4ce815dfb0 2023-10-16T09:59:44.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.240+0000 7f4cddffb700 1 -- 172.21.15.158:0/3035300663 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f4ce00546b0 con 0x7f4ce815dfb0 2023-10-16T09:59:44.241 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:44.241 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3413925509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3413925509},{"type":"v1","addr":"172.21.15.158:6835","nonce":3413925509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/655082434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":655082434},{"type":"v1","addr":"172.21.15.158:6837","nonce":655082434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2666705933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2666705933},{"type":"v1","addr":"172.21.15.172:6835","nonce":2666705933}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:23.515854+0000","modified":"2023-10-16T09:59:26.532765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35639":{"gid":35639,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3389815719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3389815719},{"type":"v1","addr":"172.21.15.172:6834","nonce":3389815719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T09:59:44.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 msgr2=0x7f4ccc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:44.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 0x7f4ccc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815dfb0 msgr2=0x7f4ce80b53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815dfb0 0x7f4ce80b53f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0007e70 tx=0x7f4ce000d3b0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 shutdown_connections 2023-10-16T09:59:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815dfb0 0x7f4ce80b53f0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ce806f340 0x7f4ce8153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ccc06b020 0x7f4ccc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 --2- 172.21.15.158:0/3035300663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce80cc5d0 0x7f4ce80b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.243+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 >> 172.21.15.158:0/3035300663 conn(0x7f4ce80d02b0 msgr2=0x7f4ce815c980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.244+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 shutdown_connections 2023-10-16T09:59:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.244+0000 7f4cef7bd700 1 -- 172.21.15.158:0/3035300663 wait complete. 2023-10-16T09:59:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 33 2023-10-16T09:59:44.263 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:44.355 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:44.354+0000 7f5459781200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:59:44.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.426+0000 7f26683a5700 1 Processor -- start 2023-10-16T09:59:44.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.427+0000 7f26683a5700 1 -- start start 2023-10-16T09:59:44.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.427+0000 7f26683a5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f266015db90 0x7f266006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f26683a5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f26600cddb0 0x7f26600ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f26683a5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26600d0fb0 con 0x7f266015d7b0 2023-10-16T09:59:44.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f26683a5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266015e100 con 0x7f26600cddb0 2023-10-16T09:59:44.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f26683a5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266015e3e0 con 0x7f266015db90 2023-10-16T09:59:44.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f26600cddb0 0x7f26600ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f26600cddb0 0x7f26600ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53196/0 (socket says 172.21.15.158:53196) 2023-10-16T09:59:44.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 -- 172.21.15.158:0/3183933566 learned_addr learned my addr 172.21.15.158:0/3183933566 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:44.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2665940700 1 --2- 172.21.15.158:0/3183933566 >> v2:172.21.15.172:3300/0 conn(0x7f266015db90 0x7f266006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 -- 172.21.15.158:0/3183933566 >> v2:172.21.15.172:3300/0 conn(0x7f266015db90 msgr2=0x7f266006b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:44.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 --2- 172.21.15.158:0/3183933566 >> v2:172.21.15.172:3300/0 conn(0x7f266015db90 0x7f266006b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 -- 172.21.15.158:0/3183933566 >> v1:172.21.15.172:6789/0 conn(0x7f266015d7b0 legacy=0x7f26600b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 -- 172.21.15.158:0/3183933566 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266015e6c0 con 0x7f26600cddb0 2023-10-16T09:59:44.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f2666942700 1 --2- 172.21.15.158:0/3183933566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26600cddb0 0x7f26600ce1b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f26600c4d70 tx=0x7f265800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88bb086b3b99b0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:44.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f266513f700 1 -- 172.21.15.158:0/3183933566 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265800e070 con 0x7f26600cddb0 2023-10-16T09:59:44.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f266513f700 1 -- 172.21.15.158:0/3183933566 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265800b910 con 0x7f26600cddb0 2023-10-16T09:59:44.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.428+0000 7f266513f700 1 -- 172.21.15.158:0/3183933566 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265800bcc0 con 0x7f26600cddb0 2023-10-16T09:59:44.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 -- 172.21.15.158:0/3183933566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26600cddb0 msgr2=0x7f26600ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 --2- 172.21.15.158:0/3183933566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26600cddb0 0x7f26600ce1b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f26600c4d70 tx=0x7f265800b330 comp rx=0 tx=0).stop 2023-10-16T09:59:44.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 -- 172.21.15.158:0/3183933566 shutdown_connections 2023-10-16T09:59:44.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 --2- 172.21.15.158:0/3183933566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26600cddb0 0x7f26600ce1b0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 --2- 172.21.15.158:0/3183933566 >> v2:172.21.15.172:3300/0 conn(0x7f266015db90 0x7f266006b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 -- 172.21.15.158:0/3183933566 >> 172.21.15.158:0/3183933566 conn(0x7f2660071220 msgr2=0x7f266006a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:44.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 -- 172.21.15.158:0/3183933566 shutdown_connections 2023-10-16T09:59:44.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.429+0000 7f26683a5700 1 -- 172.21.15.158:0/3183933566 wait complete. 2023-10-16T09:59:44.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.430+0000 7f26683a5700 1 Processor -- start 2023-10-16T09:59:44.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.430+0000 7f26683a5700 1 -- start start 2023-10-16T09:59:44.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f26683a5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26600cddb0 0x7f2660155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f26683a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f266015d7b0 0x7f26601618c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f26683a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f266015db90 0x7f2660147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f26683a5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26601605d0 con 0x7f266015db90 2023-10-16T09:59:44.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f26683a5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2660160820 con 0x7f26600cddb0 2023-10-16T09:59:44.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f26683a5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2660160b00 con 0x7f266015d7b0 2023-10-16T09:59:44.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2666942700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f266015db90 0x7f2660147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2666942700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f266015db90 0x7f2660147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53202/0 (socket says 172.21.15.158:53202) 2023-10-16T09:59:44.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2666942700 1 -- 172.21.15.158:0/1542312290 learned_addr learned my addr 172.21.15.158:0/1542312290 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:44.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2666141700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26600cddb0 0x7f2660155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2665940700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f266015d7b0 0x7f26601618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2666942700 1 -- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f266015d7b0 msgr2=0x7f26601618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2666942700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f266015d7b0 0x7f26601618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.431+0000 7f2666942700 1 -- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26600cddb0 msgr2=0x7f2660155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f2666942700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26600cddb0 0x7f2660155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f2666942700 1 -- 172.21.15.158:0/1542312290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2658009b30 con 0x7f266015db90 2023-10-16T09:59:44.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f2666942700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f266015db90 0x7f2660147650 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f26600ceab0 tx=0x7f2658004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:44.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f2656ffd700 1 -- 172.21.15.158:0/1542312290 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658004880 con 0x7f266015db90 2023-10-16T09:59:44.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f2656ffd700 1 -- 172.21.15.158:0/1542312290 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26580099b0 con 0x7f266015db90 2023-10-16T09:59:44.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2660160de0 con 0x7f266015db90 2023-10-16T09:59:44.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f2656ffd700 1 -- 172.21.15.158:0/1542312290 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265801c680 con 0x7f266015db90 2023-10-16T09:59:44.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.432+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26601613a0 con 0x7f266015db90 2023-10-16T09:59:44.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.433+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266015e980 con 0x7f266015db90 2023-10-16T09:59:44.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.433+0000 7f2656ffd700 1 -- 172.21.15.158:0/1542312290 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f265801c7e0 con 0x7f266015db90 2023-10-16T09:59:44.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.434+0000 7f2656ffd700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 0x7f264406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.434+0000 7f2666141700 1 -- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 msgr2=0x7f264406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:44.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.434+0000 7f2656ffd700 1 -- 172.21.15.158:0/1542312290 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f265808b810 con 0x7f266015db90 2023-10-16T09:59:44.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.434+0000 7f2666141700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 0x7f264406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:44.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.438+0000 7f2656ffd700 1 -- 172.21.15.158:0/1542312290 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f266015e980 con 0x7f266015db90 2023-10-16T09:59:44.463 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:44.462+0000 7fa46b5dc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T09:59:44.629 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:44.628+0000 7fa46b5dc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T09:59:44.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.635+0000 7f2666141700 1 -- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 msgr2=0x7f264406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:44.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.635+0000 7f2666141700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 0x7f264406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:44.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.637+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f266015e980 con 0x7f266015db90 2023-10-16T09:59:44.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.639+0000 7f2656ffd700 1 -- 172.21.15.158:0/1542312290 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17880 (secure 0 0 0) 0x7f266015e980 con 0x7f266015db90 2023-10-16T09:59:44.640 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:44.641 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":86,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:37.743739+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:21.428579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:22.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3100922361":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6833/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6832/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/3345169280":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/4171724344":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/310788915":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/2329993263":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/234410797":"2023-10-17T09:59:37.743702+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 msgr2=0x7f264406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 0x7f264406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f266015db90 msgr2=0x7f2660147650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f266015db90 0x7f2660147650 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f26600ceab0 tx=0x7f2658004ad0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 shutdown_connections 2023-10-16T09:59:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f266015db90 0x7f2660147650 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f264406b020 0x7f264406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26600cddb0 0x7f2660155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 --2- 172.21.15.158:0/1542312290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f266015d7b0 0x7f26601618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 >> 172.21.15.158:0/1542312290 conn(0x7f2660071220 msgr2=0x7f26600b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 shutdown_connections 2023-10-16T09:59:44.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.641+0000 7f26683a5700 1 -- 172.21.15.158:0/1542312290 wait complete. 2023-10-16T09:59:44.655 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T09:59:44.655 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:44.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.777+0000 7f0ec540d700 1 Processor -- start 2023-10-16T09:59:44.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.777+0000 7f0ec540d700 1 -- start start 2023-10-16T09:59:44.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.778+0000 7f0ec540d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec00723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.778+0000 7f0ec540d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ec00af590 0x7f0ec006d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.778+0000 7f0ec540d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec0072910 con 0x7f0ec006dec0 2023-10-16T09:59:44.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.778+0000 7f0ec540d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00ca7e0 con 0x7f0ec00af590 2023-10-16T09:59:44.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.778+0000 7f0ec540d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec006e340 con 0x7f0ec015a6c0 2023-10-16T09:59:44.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ec00af590 0x7f0ec006d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ec00af590 0x7f0ec006d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53210/0 (socket says 172.21.15.158:53210) 2023-10-16T09:59:44.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 -- 172.21.15.158:0/2332435170 learned_addr learned my addr 172.21.15.158:0/2332435170 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:44.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebeffd700 1 --2- 172.21.15.158:0/2332435170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec00723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 -- 172.21.15.158:0/2332435170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 msgr2=0x7f0ec00723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:44.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 --2- 172.21.15.158:0/2332435170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec00723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 -- 172.21.15.158:0/2332435170 >> v1:172.21.15.172:6789/0 conn(0x7f0ec006dec0 legacy=0x7f0ec015e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 -- 172.21.15.158:0/2332435170 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00b1e40 con 0x7f0ec00af590 2023-10-16T09:59:44.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebe7fc700 1 --2- 172.21.15.158:0/2332435170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00af590 0x7f0ec006d980 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40098f0 tx=0x7f0eb4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e791584750339930 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:44.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebdffb700 1 -- 172.21.15.158:0/2332435170 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4011830 con 0x7f0ec00af590 2023-10-16T09:59:44.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebdffb700 1 -- 172.21.15.158:0/2332435170 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb4011990 con 0x7f0ec00af590 2023-10-16T09:59:44.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.779+0000 7f0ebdffb700 1 -- 172.21.15.158:0/2332435170 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4011d20 con 0x7f0ec00af590 2023-10-16T09:59:44.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 -- 172.21.15.158:0/2332435170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00af590 msgr2=0x7f0ec006d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 --2- 172.21.15.158:0/2332435170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00af590 0x7f0ec006d980 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40098f0 tx=0x7f0eb4009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:44.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 -- 172.21.15.158:0/2332435170 shutdown_connections 2023-10-16T09:59:44.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 --2- 172.21.15.158:0/2332435170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec00723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 --2- 172.21.15.158:0/2332435170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00af590 0x7f0ec006d980 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 -- 172.21.15.158:0/2332435170 >> 172.21.15.158:0/2332435170 conn(0x7f0ec00ccda0 msgr2=0x7f0ec00cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:44.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 -- 172.21.15.158:0/2332435170 shutdown_connections 2023-10-16T09:59:44.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.780+0000 7f0ec540d700 1 -- 172.21.15.158:0/2332435170 wait complete. 2023-10-16T09:59:44.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 Processor -- start 2023-10-16T09:59:44.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 -- start start 2023-10-16T09:59:44.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec006dec0 0x7f0ec0157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ec00af590 0x7f0ec01f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec01f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec015b530 con 0x7f0ec006dec0 2023-10-16T09:59:44.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00cc640 con 0x7f0ec00af590 2023-10-16T09:59:44.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ec540d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec0158490 con 0x7f0ec015a6c0 2023-10-16T09:59:44.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ebeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec006dec0 0x7f0ec0157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.782+0000 7f0ebeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec006dec0 0x7f0ec0157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53214/0 (socket says 172.21.15.158:53214) 2023-10-16T09:59:44.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebeffd700 1 -- 172.21.15.158:0/1084784088 learned_addr learned my addr 172.21.15.158:0/1084784088 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:44.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebf7fe700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec01f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebe7fc700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ec00af590 0x7f0ec01f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:44.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebeffd700 1 -- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 msgr2=0x7f0ec01f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebeffd700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec01f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebeffd700 1 -- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ec00af590 msgr2=0x7f0ec01f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:44.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebeffd700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ec00af590 0x7f0ec01f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:44.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebeffd700 1 -- 172.21.15.158:0/1084784088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb4009550 con 0x7f0ec006dec0 2023-10-16T09:59:44.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebf7fe700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec01f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:44.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ebeffd700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec006dec0 0x7f0ec0157f50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000e990 tx=0x7f0eb000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:44.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ea77fe700 1 -- 172.21.15.158:0/1084784088 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00096f0 con 0x7f0ec006dec0 2023-10-16T09:59:44.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ea77fe700 1 -- 172.21.15.158:0/1084784088 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb000c690 con 0x7f0ec006dec0 2023-10-16T09:59:44.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ea77fe700 1 -- 172.21.15.158:0/1084784088 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb000ca00 con 0x7f0ec006dec0 2023-10-16T09:59:44.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec00cc080 con 0x7f0ec006dec0 2023-10-16T09:59:44.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.783+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec00c61e0 con 0x7f0ec006dec0 2023-10-16T09:59:44.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.784+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec00c6b10 con 0x7f0ec006dec0 2023-10-16T09:59:44.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.785+0000 7f0ea77fe700 1 -- 172.21.15.158:0/1084784088 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0eb000cb60 con 0x7f0ec006dec0 2023-10-16T09:59:44.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.785+0000 7f0ea77fe700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 0x7f0eac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:44.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.785+0000 7f0ebe7fc700 1 -- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 msgr2=0x7f0eac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:44.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.785+0000 7f0ea77fe700 1 -- 172.21.15.158:0/1084784088 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f0eb0089f70 con 0x7f0ec006dec0 2023-10-16T09:59:44.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.785+0000 7f0ebe7fc700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 0x7f0eac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:44.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.790+0000 7f0ea77fe700 1 -- 172.21.15.158:0/1084784088 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0eb0054a40 con 0x7f0ec006dec0 2023-10-16T09:59:44.879 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T09:59:44.878+0000 7f5459781200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:59:44.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.986+0000 7f0ebe7fc700 1 -- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 msgr2=0x7f0eac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:44.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:44.986+0000 7f0ebe7fc700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 0x7f0eac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T09:59:45.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.033+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ec00681b0 con 0x7f0ec006dec0 2023-10-16T09:59:45.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.034+0000 7f0ea77fe700 1 -- 172.21.15.158:0/1084784088 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f0eb0058060 con 0x7f0ec006dec0 2023-10-16T09:59:45.035 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:45.035 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3413925509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3413925509},{"type":"v1","addr":"172.21.15.158:6835","nonce":3413925509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/655082434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":655082434},{"type":"v1","addr":"172.21.15.158:6837","nonce":655082434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2666705933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2666705933},{"type":"v1","addr":"172.21.15.172:6835","nonce":2666705933}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:23.515854+0000","modified":"2023-10-16T09:59:26.532765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35639":{"gid":35639,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3389815719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3389815719},{"type":"v1","addr":"172.21.15.172:6834","nonce":3389815719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T09:59:45.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 msgr2=0x7f0eac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:45.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 0x7f0eac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec006dec0 msgr2=0x7f0ec0157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec006dec0 0x7f0ec0157f50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000e990 tx=0x7f0eb000eca0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 shutdown_connections 2023-10-16T09:59:45.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ec015a6c0 0x7f0ec01f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec006dec0 0x7f0ec0157f50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f0eac06b020 0x7f0eac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 --2- 172.21.15.158:0/1084784088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ec00af590 0x7f0ec01f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 >> 172.21.15.158:0/1084784088 conn(0x7f0ec00ccda0 msgr2=0x7f0ec00b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:45.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 shutdown_connections 2023-10-16T09:59:45.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.036+0000 7f0ec540d700 1 -- 172.21.15.158:0/1084784088 wait complete. 2023-10-16T09:59:45.040 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 33 2023-10-16T09:59:45.050 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:45.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.171+0000 7f4ef7e5e700 1 Processor -- start 2023-10-16T09:59:45.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.172+0000 7f4ef7e5e700 1 -- start start 2023-10-16T09:59:45.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.172+0000 7f4ef7e5e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.172+0000 7f4ef7e5e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ef00d0bd0 0x7f4ef00d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.172+0000 7f4ef7e5e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00ce4d0 con 0x7f4ef00d3570 2023-10-16T09:59:45.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.172+0000 7f4ef7e5e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef006c1b0 con 0x7f4ef00d0bd0 2023-10-16T09:59:45.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.172+0000 7f4ef7e5e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef006c490 con 0x7f4ef015daa0 2023-10-16T09:59:45.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ef00d0bd0 0x7f4ef00d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ef00d0bd0 0x7f4ef00d1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53226/0 (socket says 172.21.15.158:53226) 2023-10-16T09:59:45.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 -- 172.21.15.158:0/3523632995 learned_addr learned my addr 172.21.15.158:0/3523632995 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:45.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef5bfa700 1 --2- 172.21.15.158:0/3523632995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 -- 172.21.15.158:0/3523632995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 msgr2=0x7f4ef006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:45.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 --2- 172.21.15.158:0/3523632995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef006b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 -- 172.21.15.158:0/3523632995 >> v1:172.21.15.172:6789/0 conn(0x7f4ef00d3570 legacy=0x7f4ef00d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 -- 172.21.15.158:0/3523632995 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef006c770 con 0x7f4ef00d0bd0 2023-10-16T09:59:45.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef53f9700 1 --2- 172.21.15.158:0/3523632995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d0bd0 0x7f4ef00d1510 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00098f0 tx=0x7f4ee0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44c8a7528fc9a8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef4bf8700 1 -- 172.21.15.158:0/3523632995 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0011830 con 0x7f4ef00d0bd0 2023-10-16T09:59:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef4bf8700 1 -- 172.21.15.158:0/3523632995 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee0011990 con 0x7f4ef00d0bd0 2023-10-16T09:59:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.173+0000 7f4ef4bf8700 1 -- 172.21.15.158:0/3523632995 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0011d20 con 0x7f4ef00d0bd0 2023-10-16T09:59:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.174+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/3523632995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d0bd0 msgr2=0x7f4ef00d1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.174+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/3523632995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d0bd0 0x7f4ef00d1510 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00098f0 tx=0x7f4ee0009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.174+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/3523632995 shutdown_connections 2023-10-16T09:59:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.174+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/3523632995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef006b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.174+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/3523632995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d0bd0 0x7f4ef00d1510 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.174+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/3523632995 >> 172.21.15.158:0/3523632995 conn(0x7f4ef008d040 msgr2=0x7f4ef009c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.175+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/3523632995 shutdown_connections 2023-10-16T09:59:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.175+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/3523632995 wait complete. 2023-10-16T09:59:45.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.176+0000 7f4ef7e5e700 1 Processor -- start 2023-10-16T09:59:45.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.176+0000 7f4ef7e5e700 1 -- start start 2023-10-16T09:59:45.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef7e5e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef00d0bd0 0x7f4ef01f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef7e5e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d3570 0x7f4ef0160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef7e5e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef01f6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef7e5e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01f3700 con 0x7f4ef00d3570 2023-10-16T09:59:45.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef7e5e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b92d0 con 0x7f4ef00d0bd0 2023-10-16T09:59:45.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef7e5e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01f3120 con 0x7f4ef015daa0 2023-10-16T09:59:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d3570 0x7f4ef0160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d3570 0x7f4ef0160440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53232/0 (socket says 172.21.15.158:53232) 2023-10-16T09:59:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 -- 172.21.15.158:0/943136874 learned_addr learned my addr 172.21.15.158:0/943136874 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef5bfa700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef00d0bd0 0x7f4ef01f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef63fb700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef01f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 -- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 msgr2=0x7f4ef01f6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef01f6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 -- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef00d0bd0 msgr2=0x7f4ef01f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef00d0bd0 0x7f4ef01f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 -- 172.21.15.158:0/943136874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009550 con 0x7f4ef00d3570 2023-10-16T09:59:45.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef53f9700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d3570 0x7f4ef0160440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009c40 tx=0x7f4ee0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:45.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0004400 con 0x7f4ef00d3570 2023-10-16T09:59:45.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee001f070 con 0x7f4ef00d3570 2023-10-16T09:59:45.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef01f71a0 con 0x7f4ef00d3570 2023-10-16T09:59:45.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.178+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00038c0 con 0x7f4ef00d3570 2023-10-16T09:59:45.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.179+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef01f7480 con 0x7f4ef00d3570 2023-10-16T09:59:45.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.179+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef00681b0 con 0x7f4ef00d3570 2023-10-16T09:59:45.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.180+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 38) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f4ee0021030 con 0x7f4ef00d3570 2023-10-16T09:59:45.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.180+0000 7f4ee67fc700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ed406b020 0x7f4ed406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.181+0000 7f4ef5bfa700 1 -- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ed406b020 msgr2=0x7f4ed406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3502835696 2023-10-16T09:59:45.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.181+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f4ee008b140 con 0x7f4ef00d3570 2023-10-16T09:59:45.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.181+0000 7f4ef5bfa700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ed406b020 0x7f4ed406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T09:59:45.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.185+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ee0055db0 con 0x7f4ef00d3570 2023-10-16T09:59:45.192 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.188+0000 7fa46b5dc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T09:59:45.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.210+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 39) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4ee0056180 con 0x7f4ef00d3570 2023-10-16T09:59:45.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.211+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ed406b020 msgr2=0x7f4ed406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T09:59:45.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.211+0000 7f4ee67fc700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ed406b020 0x7f4ed406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.273 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e6e08700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.273 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e6e08700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.273 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e6e08700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.274 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e6e08700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.274 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e6e08700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.274 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e3dc2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.274 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e3dc2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.275 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e3dc2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.275 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e3dc2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.275 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T09:59:45.271+0000 7fa3e3dc2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T09:59:45.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.440+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ef01f3840 con 0x7f4ef00d3570 2023-10-16T09:59:45.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.441+0000 7f4ee67fc700 1 -- 172.21.15.158:0/943136874 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f4ee005cb30 con 0x7f4ef00d3570 2023-10-16T09:59:45.443 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:45.443 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3413925509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3413925509},{"type":"v1","addr":"172.21.15.158:6835","nonce":3413925509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/655082434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":655082434},{"type":"v1","addr":"172.21.15.158:6837","nonce":655082434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2666705933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2666705933},{"type":"v1","addr":"172.21.15.172:6835","nonce":2666705933}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:23.515854+0000","modified":"2023-10-16T09:59:26.532765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35639":{"gid":35639,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3389815719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3389815719},{"type":"v1","addr":"172.21.15.172:6834","nonce":3389815719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T09:59:45.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d3570 msgr2=0x7f4ef0160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d3570 0x7f4ef0160440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009c40 tx=0x7f4ee0004750 comp rx=0 tx=0).stop 2023-10-16T09:59:45.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 shutdown_connections 2023-10-16T09:59:45.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ef015daa0 0x7f4ef01f6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:6832/3502835696,v1:172.21.15.158:6833/3502835696] conn(0x7f4ed406b020 0x7f4ed406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef00d0bd0 0x7f4ef01f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 --2- 172.21.15.158:0/943136874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ef00d3570 0x7f4ef0160440 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 >> 172.21.15.158:0/943136874 conn(0x7f4ef008d040 msgr2=0x7f4ef00d39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:45.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 shutdown_connections 2023-10-16T09:59:45.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.443+0000 7f4ef7e5e700 1 -- 172.21.15.158:0/943136874 wait complete. 2023-10-16T09:59:45.447 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 33 2023-10-16T09:59:45.456 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:45.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.574+0000 7f48afb15700 1 Processor -- start 2023-10-16T09:59:45.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.575+0000 7f48afb15700 1 -- start start 2023-10-16T09:59:45.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.575+0000 7f48afb15700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f48a80c65d0 0x7f48a806be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.575+0000 7f48afb15700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a806b970 con 0x7f48a80ce590 2023-10-16T09:59:45.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.575+0000 7f48afb15700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a806ad60 con 0x7f48a806a8d0 2023-10-16T09:59:45.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.575+0000 7f48afb15700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a806b040 con 0x7f48a80c65d0 2023-10-16T09:59:45.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ae0b2700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f48a80ce590 0x7f48a80c4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53488/0 (socket says 172.21.15.158:53488) 2023-10-16T09:59:45.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ae0b2700 1 -- 172.21.15.158:0/533462790 learned_addr learned my addr 172.21.15.158:0/533462790 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:45.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195195547 0 0) 0x7f48a806b970 con 0x7f48a80ce590 2023-10-16T09:59:45.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ad8b1700 1 --2- 172.21.15.158:0/533462790 >> v2:172.21.15.172:3300/0 conn(0x7f48a80c65d0 0x7f48a806be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48940035b0 con 0x7f48a80ce590 2023-10-16T09:59:45.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3918616614 0 0) 0x7f48940035b0 con 0x7f48a80ce590 2023-10-16T09:59:45.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 >> v2:172.21.15.172:3300/0 conn(0x7f48a80c65d0 msgr2=0x7f48a806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:45.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 --2- 172.21.15.158:0/533462790 >> v2:172.21.15.172:3300/0 conn(0x7f48a80c65d0 0x7f48a806be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 >> v1:172.21.15.172:6789/0 conn(0x7f48a806a8d0 legacy=0x7f48a80cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a806b320 con 0x7f48a80ce590 2023-10-16T09:59:45.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f48a00019f0 con 0x7f48a80ce590 2023-10-16T09:59:45.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 >> v1:172.21.15.158:6789/0 conn(0x7f48a80ce590 legacy=0x7f48a80c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894003a00 0x7f4894003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4894004850 0x7f4894004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4894005210 0x7f489400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a806b040 con 0x7f4894004850 2023-10-16T09:59:45.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489400c0b0 con 0x7f4894003a00 2023-10-16T09:59:45.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489400c270 con 0x7f4894005210 2023-10-16T09:59:45.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ad0b0700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4894004850 0x7f4894004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ad8b1700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894003a00 0x7f4894003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.576+0000 7f48ae0b2700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4894005210 0x7f489400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ad0b0700 1 -- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4894005210 msgr2=0x7f489400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:45.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ad0b0700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4894005210 0x7f489400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ad0b0700 1 -- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894003a00 msgr2=0x7f4894003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:45.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ad0b0700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894003a00 0x7f4894003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ad0b0700 1 -- 172.21.15.158:0/533462790 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940035b0 con 0x7f4894004850 2023-10-16T09:59:45.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ae0b2700 1 -- 172.21.15.158:0/533462790 reap_dead start 2023-10-16T09:59:45.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ad0b0700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4894004850 0x7f4894004cd0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f48a80cdae0 tx=0x7f4894005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b088d70b0db19a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:45.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40012f0 con 0x7f4894004850 2023-10-16T09:59:45.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a4005310 con 0x7f4894004850 2023-10-16T09:59:45.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48ac8af700 1 -- 172.21.15.158:0/533462790 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4006860 con 0x7f4894004850 2023-10-16T09:59:45.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48afb15700 1 -- 172.21.15.158:0/533462790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4894004850 msgr2=0x7f4894004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48afb15700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4894004850 0x7f4894004cd0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f48a80cdae0 tx=0x7f4894005880 comp rx=0 tx=0).stop 2023-10-16T09:59:45.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48afb15700 1 -- 172.21.15.158:0/533462790 shutdown_connections 2023-10-16T09:59:45.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48afb15700 1 --2- 172.21.15.158:0/533462790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4894004850 0x7f4894004cd0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48afb15700 1 -- 172.21.15.158:0/533462790 >> 172.21.15.158:0/533462790 conn(0x7f48a80cc4b0 msgr2=0x7f48a80cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:45.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.577+0000 7f48afb15700 1 -- 172.21.15.158:0/533462790 shutdown_connections 2023-10-16T09:59:45.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.578+0000 7f48afb15700 1 -- 172.21.15.158:0/533462790 wait complete. 2023-10-16T09:59:45.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 Processor -- start 2023-10-16T09:59:45.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 -- start start 2023-10-16T09:59:45.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894004850 0x7f48a81e5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f48a80b2750 0x7f48a815d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48a80c8690 0x7f48a80c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a80b3dd0 con 0x7f48a80c8690 2023-10-16T09:59:45.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a80c9050 con 0x7f4894004850 2023-10-16T09:59:45.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.579+0000 7f48afb15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a815dd70 con 0x7f48a80b2750 2023-10-16T09:59:45.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48a80c8690 0x7f48a80c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48a80c8690 0x7f48a80c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53324/0 (socket says 172.21.15.158:53324) 2023-10-16T09:59:45.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 -- 172.21.15.158:0/2115348546 learned_addr learned my addr 172.21.15.158:0/2115348546 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:45.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ad8b1700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894004850 0x7f48a81e5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ad0b0700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f48a80b2750 0x7f48a815d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 -- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f48a80b2750 msgr2=0x7f48a815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f48a80b2750 0x7f48a815d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 -- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894004850 msgr2=0x7f48a81e5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894004850 0x7f48a81e5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 -- 172.21.15.158:0/2115348546 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a80b2ef0 con 0x7f48a80c8690 2023-10-16T09:59:45.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48ae0b2700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48a80c8690 0x7f48a80c8b10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f48a0000f80 tx=0x7f48a000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:45.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f489e7fc700 1 -- 172.21.15.158:0/2115348546 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a00019e0 con 0x7f48a80c8690 2023-10-16T09:59:45.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f489e7fc700 1 -- 172.21.15.158:0/2115348546 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a000ad00 con 0x7f48a80c8690 2023-10-16T09:59:45.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a80b31d0 con 0x7f48a80c8690 2023-10-16T09:59:45.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f489e7fc700 1 -- 172.21.15.158:0/2115348546 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0013630 con 0x7f48a80c8690 2023-10-16T09:59:45.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.580+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a80b3790 con 0x7f48a80c8690 2023-10-16T09:59:45.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.581+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a80b3f10 con 0x7f48a80c8690 2023-10-16T09:59:45.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.582+0000 7f489e7fc700 1 -- 172.21.15.158:0/2115348546 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 39) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f48a0013790 con 0x7f48a80c8690 2023-10-16T09:59:45.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.583+0000 7f489e7fc700 1 -- 172.21.15.158:0/2115348546 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7141+0+0 (secure 0 0 0) 0x7f48a008a5c0 con 0x7f48a80c8690 2023-10-16T09:59:45.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.586+0000 7f489e7fc700 1 -- 172.21.15.158:0/2115348546 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48a00555c0 con 0x7f48a80c8690 2023-10-16T09:59:45.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.784+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48a80b3f10 con 0x7f48a80c8690 2023-10-16T09:59:45.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.785+0000 7f489e7fc700 1 -- 172.21.15.158:0/2115348546 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+18386 (secure 0 0 0) 0x7f48a80b3f10 con 0x7f48a80c8690 2023-10-16T09:59:45.787 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:45.787 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":87,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:45.203005+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:21.428579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:22.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/3502835696":"2023-10-17T09:59:45.202968+0000","172.21.15.158:6832/3502835696":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/400306926":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/2577742248":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/2992039003":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/23874899":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/183559494":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/1052952591":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/3100922361":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6833/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6832/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/3345169280":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/4171724344":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/310788915":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/2329993263":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/234410797":"2023-10-17T09:59:37.743702+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:45.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48a80c8690 msgr2=0x7f48a80c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48a80c8690 0x7f48a80c8b10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f48a0000f80 tx=0x7f48a000a5f0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 shutdown_connections 2023-10-16T09:59:45.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48a80c8690 0x7f48a80c8b10 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4894004850 0x7f48a81e5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 --2- 172.21.15.158:0/2115348546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f48a80b2750 0x7f48a815d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 >> 172.21.15.158:0/2115348546 conn(0x7f48a80cc4b0 msgr2=0x7f48a806db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:45.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 shutdown_connections 2023-10-16T09:59:45.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.787+0000 7f48afb15700 1 -- 172.21.15.158:0/2115348546 wait complete. 2023-10-16T09:59:45.801 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T09:59:45.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.921+0000 7fa6e6dc9700 1 Processor -- start 2023-10-16T09:59:45.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.921+0000 7fa6e6dc9700 1 -- start start 2023-10-16T09:59:45.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e6dc9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa6e015d950 0x7fa6e015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e6dc9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e00ce320 con 0x7fa6e00cec30 2023-10-16T09:59:45.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e6dc9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e015e270 con 0x7fa6e00ce7b0 2023-10-16T09:59:45.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e6dc9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e00725c0 con 0x7fa6e015d950 2023-10-16T09:59:45.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e4b65700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa6e015d950 0x7fa6e015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e4b65700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa6e015d950 0x7fa6e015dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53326/0 (socket says 172.21.15.158:53326) 2023-10-16T09:59:45.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e4b65700 1 -- 172.21.15.158:0/3896529588 learned_addr learned my addr 172.21.15.158:0/3896529588 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:45.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e4b65700 1 -- 172.21.15.158:0/3896529588 >> v1:172.21.15.172:6789/0 conn(0x7fa6e00ce7b0 legacy=0x7fa6e0071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:45.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e4b65700 1 -- 172.21.15.158:0/3896529588 >> v1:172.21.15.158:6789/0 conn(0x7fa6e00cec30 legacy=0x7fa6e006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e4b65700 1 -- 172.21.15.158:0/3896529588 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e00722f0 con 0x7fa6e015d950 2023-10-16T09:59:45.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6e4b65700 1 --2- 172.21.15.158:0/3896529588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e015d950 0x7fa6e015dd30 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40098f0 tx=0x7fa6d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f788e86e9d37fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:45.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6df7fe700 1 -- 172.21.15.158:0/3896529588 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4004230 con 0x7fa6e015d950 2023-10-16T09:59:45.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6df7fe700 1 -- 172.21.15.158:0/3896529588 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d4004390 con 0x7fa6e015d950 2023-10-16T09:59:45.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.922+0000 7fa6df7fe700 1 -- 172.21.15.158:0/3896529588 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d40036a0 con 0x7fa6e015d950 2023-10-16T09:59:45.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.923+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/3896529588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e015d950 msgr2=0x7fa6e015dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.923+0000 7fa6e6dc9700 1 --2- 172.21.15.158:0/3896529588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e015d950 0x7fa6e015dd30 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40098f0 tx=0x7fa6d4009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:45.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.923+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/3896529588 shutdown_connections 2023-10-16T09:59:45.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.923+0000 7fa6e6dc9700 1 --2- 172.21.15.158:0/3896529588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e015d950 0x7fa6e015dd30 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.923+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/3896529588 >> 172.21.15.158:0/3896529588 conn(0x7fa6e00cc700 msgr2=0x7fa6e00ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:45.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.923+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/3896529588 shutdown_connections 2023-10-16T09:59:45.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.923+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/3896529588 wait complete. 2023-10-16T09:59:45.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.924+0000 7fa6e6dc9700 1 Processor -- start 2023-10-16T09:59:45.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.924+0000 7fa6e6dc9700 1 -- start start 2023-10-16T09:59:45.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e6dc9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6e00ce7b0 0x7fa6e006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e6dc9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e00cec30 0x7fa6e006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e6dc9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6e015d950 0x7fa6e01f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:45.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e6dc9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00b2b90 con 0x7fa6e00cec30 2023-10-16T09:59:45.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e6dc9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00c95e0 con 0x7fa6e015d950 2023-10-16T09:59:45.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e6dc9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00c98c0 con 0x7fa6e00ce7b0 2023-10-16T09:59:45.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e00cec30 0x7fa6e006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e00cec30 0x7fa6e006ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53334/0 (socket says 172.21.15.158:53334) 2023-10-16T09:59:45.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6dffff700 1 -- 172.21.15.158:0/2731619464 learned_addr learned my addr 172.21.15.158:0/2731619464 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:45.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e4b65700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6e00ce7b0 0x7fa6e006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.925+0000 7fa6e5366700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6e015d950 0x7fa6e01f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:45.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dffff700 1 -- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6e00ce7b0 msgr2=0x7fa6e006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dffff700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6e00ce7b0 0x7fa6e006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dffff700 1 -- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6e015d950 msgr2=0x7fa6e01f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:45.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dffff700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6e015d950 0x7fa6e01f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:45.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dffff700 1 -- 172.21.15.158:0/2731619464 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d4009550 con 0x7fa6e00cec30 2023-10-16T09:59:45.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dffff700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e00cec30 0x7fa6e006ae10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00c4d0 tx=0x7fa6cc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:45.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dd7fa700 1 -- 172.21.15.158:0/2731619464 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc011790 con 0x7fa6e00cec30 2023-10-16T09:59:45.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dd7fa700 1 -- 172.21.15.158:0/2731619464 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc0118f0 con 0x7fa6e00cec30 2023-10-16T09:59:45.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e00c9ba0 con 0x7fa6e00cec30 2023-10-16T09:59:45.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6dd7fa700 1 -- 172.21.15.158:0/2731619464 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc011ca0 con 0x7fa6e00cec30 2023-10-16T09:59:45.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.926+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e01f3960 con 0x7fa6e00cec30 2023-10-16T09:59:45.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.927+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e00b2d00 con 0x7fa6e00cec30 2023-10-16T09:59:45.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.927+0000 7fa6dd7fa700 1 -- 172.21.15.158:0/2731619464 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 39) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa6cc01c460 con 0x7fa6e00cec30 2023-10-16T09:59:45.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.928+0000 7fa6dd7fa700 1 -- 172.21.15.158:0/2731619464 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7141+0+0 (secure 0 0 0) 0x7fa6cc089420 con 0x7fa6e00cec30 2023-10-16T09:59:45.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:45.932+0000 7fa6dd7fa700 1 -- 172.21.15.158:0/2731619464 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6cc054290 con 0x7fa6e00cec30 2023-10-16T09:59:46.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.168+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa6e00681b0 con 0x7fa6e00cec30 2023-10-16T09:59:46.169 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T09:59:46.168+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T09:59:46.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.170+0000 7fa6dd7fa700 1 -- 172.21.15.158:0/2731619464 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa6cc0578b0 con 0x7fa6e00cec30 2023-10-16T09:59:46.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e00cec30 msgr2=0x7fa6e006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:46.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e00cec30 0x7fa6e006ae10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00c4d0 tx=0x7fa6cc00c7e0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 shutdown_connections 2023-10-16T09:59:46.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6e015d950 0x7fa6e01f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6e00ce7b0 0x7fa6e006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 --2- 172.21.15.158:0/2731619464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6e00cec30 0x7fa6e006ae10 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 >> 172.21.15.158:0/2731619464 conn(0x7fa6e00cc700 msgr2=0x7fa6e015afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:46.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 shutdown_connections 2023-10-16T09:59:46.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.172+0000 7fa6e6dc9700 1 -- 172.21.15.158:0/2731619464 wait complete. 2023-10-16T09:59:46.176 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T09:59:46.177 INFO:tasks.ceph.mds.d.smithi172.stderr: -4678> 2023-10-16T09:59:29.107+0000 7fd6644af700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T09:59:46.186 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T09:59:46.209 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T09:59:46.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.312+0000 7fdbcf26e700 1 Processor -- start 2023-10-16T09:59:46.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.312+0000 7fdbcf26e700 1 -- start start 2023-10-16T09:59:46.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcf26e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc8160d30 0x7fdbc80b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:46.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcf26e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbc80b7590 0x7fdbc80d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:46.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcf26e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc80b7a20 con 0x7fdbc80d0530 2023-10-16T09:59:46.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcf26e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc80b8910 con 0x7fdbc80b7590 2023-10-16T09:59:46.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcf26e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80d0ef0 con 0x7fdbc8160d30 2023-10-16T09:59:46.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcd00a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc8160d30 0x7fdbc80b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:46.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcc809700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdbc80d0530 0x7fdbc80d19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:57440/0 (socket says 172.21.15.158:57440) 2023-10-16T09:59:46.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcc809700 1 -- 172.21.15.158:0/2071736415 learned_addr learned my addr 172.21.15.158:0/2071736415 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:46.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.313+0000 7fdbcd80b700 1 --2- 172.21.15.158:0/2071736415 >> v2:172.21.15.172:3300/0 conn(0x7fdbc80b7590 0x7fdbc80d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:46.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbbffff700 1 -- 172.21.15.158:0/2071736415 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044617613 0 0) 0x7fdbc80b7a20 con 0x7fdbc80d0530 2023-10-16T09:59:46.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbbffff700 1 -- 172.21.15.158:0/2071736415 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb40035b0 con 0x7fdbc80d0530 2023-10-16T09:59:46.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbcd80b700 1 -- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc8160d30 msgr2=0x7fdbc80b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:46.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbcd80b700 1 --2- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc8160d30 0x7fdbc80b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbcd80b700 1 -- 172.21.15.158:0/2071736415 >> v1:172.21.15.172:6789/0 conn(0x7fdbc80d0530 legacy=0x7fdbc80d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:46.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbcd80b700 1 -- 172.21.15.158:0/2071736415 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80b7c90 con 0x7fdbc80b7590 2023-10-16T09:59:46.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbcd00a700 1 --2- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc8160d30 0x7fdbc80b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:46.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbcd80b700 1 --2- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc80b7590 0x7fdbc80d09b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009d90 tx=0x7fdbc000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc0d68baf87b6943 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:46.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbbffff700 1 -- 172.21.15.158:0/2071736415 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc000e070 con 0x7fdbc80b7590 2023-10-16T09:59:46.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbbffff700 1 -- 172.21.15.158:0/2071736415 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc000bb20 con 0x7fdbc80b7590 2023-10-16T09:59:46.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.314+0000 7fdbbffff700 1 -- 172.21.15.158:0/2071736415 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0012430 con 0x7fdbc80b7590 2023-10-16T09:59:46.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 -- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc80b7590 msgr2=0x7fdbc80d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:46.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc80b7590 0x7fdbc80d09b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009d90 tx=0x7fdbc000b330 comp rx=0 tx=0).stop 2023-10-16T09:59:46.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 -- 172.21.15.158:0/2071736415 shutdown_connections 2023-10-16T09:59:46.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc80b7590 0x7fdbc80d09b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/2071736415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc8160d30 0x7fdbc80b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 -- 172.21.15.158:0/2071736415 >> 172.21.15.158:0/2071736415 conn(0x7fdbc80669f0 msgr2=0x7fdbc8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:46.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 -- 172.21.15.158:0/2071736415 shutdown_connections 2023-10-16T09:59:46.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.315+0000 7fdbcf26e700 1 -- 172.21.15.158:0/2071736415 wait complete. 2023-10-16T09:59:46.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.316+0000 7fdbcf26e700 1 Processor -- start 2023-10-16T09:59:46.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.316+0000 7fdbcf26e700 1 -- start start 2023-10-16T09:59:46.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcf26e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80b7590 0x7fdbc81552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:46.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcf26e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbc80d0530 0x7fdbc8153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:46.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcf26e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc8160d30 0x7fdbc81579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:46.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcf26e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc815ba40 con 0x7fdbc80d0530 2023-10-16T09:59:46.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcf26e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc81557f0 con 0x7fdbc8160d30 2023-10-16T09:59:46.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcf26e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc81f2f40 con 0x7fdbc80b7590 2023-10-16T09:59:46.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcc809700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbc80d0530 0x7fdbc8153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:46.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcc809700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbc80d0530 0x7fdbc8153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53368/0 (socket says 172.21.15.158:53368) 2023-10-16T09:59:46.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcc809700 1 -- 172.21.15.158:0/3872352924 learned_addr learned my addr 172.21.15.158:0/3872352924 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:46.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcd80b700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc8160d30 0x7fdbc81579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:46.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.317+0000 7fdbcd00a700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80b7590 0x7fdbc81552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:46.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcc809700 1 -- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80b7590 msgr2=0x7fdbc81552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:46.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcc809700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80b7590 0x7fdbc81552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcc809700 1 -- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc8160d30 msgr2=0x7fdbc81579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:46.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcc809700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc8160d30 0x7fdbc81579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:46.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcc809700 1 -- 172.21.15.158:0/3872352924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80094a0 con 0x7fdbc80d0530 2023-10-16T09:59:46.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcd80b700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc8160d30 0x7fdbc81579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:46.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcc809700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbc80d0530 0x7fdbc8153960 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c700 tx=0x7fdbc400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:46.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbbdffb700 1 -- 172.21.15.158:0/3872352924 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4005d70 con 0x7fdbc80d0530 2023-10-16T09:59:46.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbbdffb700 1 -- 172.21.15.158:0/3872352924 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc40070d0 con 0x7fdbc80d0530 2023-10-16T09:59:46.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc0009a70 con 0x7fdbc80d0530 2023-10-16T09:59:46.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbbdffb700 1 -- 172.21.15.158:0/3872352924 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4007460 con 0x7fdbc80d0530 2023-10-16T09:59:46.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.318+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc81f3500 con 0x7fdbc80d0530 2023-10-16T09:59:46.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.319+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc80681b0 con 0x7fdbc80d0530 2023-10-16T09:59:46.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.320+0000 7fdbbdffb700 1 -- 172.21.15.158:0/3872352924 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 40) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fdbc40075c0 con 0x7fdbc80d0530 2023-10-16T09:59:46.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.320+0000 7fdbbdffb700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbac06ad50 0x7fdbac06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:46.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.320+0000 7fdbbdffb700 1 -- 172.21.15.158:0/3872352924 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7227+0+0 (secure 0 0 0) 0x7fdbc4088c20 con 0x7fdbc80d0530 2023-10-16T09:59:46.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.320+0000 7fdbcd00a700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbac06ad50 0x7fdbac06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:46.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.321+0000 7fdbcd00a700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbac06ad50 0x7fdbac06d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009cd0 tx=0x7fdbb8009210 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:46.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.325+0000 7fdbbdffb700 1 -- 172.21.15.158:0/3872352924 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbc4053a20 con 0x7fdbc80d0530 2023-10-16T09:59:46.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:46.560+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdbc80ca350 con 0x7fdbc80d0530 2023-10-16T09:59:47.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.213+0000 7fdbbdffb700 1 -- 172.21.15.158:0/3872352924 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdbc4057040 con 0x7fdbc80d0530 2023-10-16T09:59:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbac06ad50 msgr2=0x7fdbac06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbac06ad50 0x7fdbac06d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009cd0 tx=0x7fdbb8009210 comp rx=0 tx=0).stop 2023-10-16T09:59:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbc80d0530 msgr2=0x7fdbc8153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:47.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbc80d0530 0x7fdbc8153960 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c700 tx=0x7fdbc400ca10 comp rx=0 tx=0).stop 2023-10-16T09:59:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 shutdown_connections 2023-10-16T09:59:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbac06ad50 0x7fdbac06d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc8160d30 0x7fdbc81579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80b7590 0x7fdbc81552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 --2- 172.21.15.158:0/3872352924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbc80d0530 0x7fdbc8153960 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 >> 172.21.15.158:0/3872352924 conn(0x7fdbc80669f0 msgr2=0x7fdbc80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:47.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 shutdown_connections 2023-10-16T09:59:47.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.216+0000 7fdbcf26e700 1 -- 172.21.15.158:0/3872352924 wait complete. 2023-10-16T09:59:47.231 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T09:59:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.362+0000 7f3a412aa700 1 Processor -- start 2023-10-16T09:59:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.362+0000 7f3a412aa700 1 -- start start 2023-10-16T09:59:47.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a412aa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a3c0b38f0 0x7f3a3c0d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a412aa700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3a3c0d2010 0x7f3a3c06bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a412aa700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c15e3e0 con 0x7f3a3c15d840 2023-10-16T09:59:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a412aa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c15ec80 con 0x7f3a3c0b38f0 2023-10-16T09:59:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a412aa700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c0d13b0 con 0x7f3a3c0d2010 2023-10-16T09:59:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a3c0b38f0 0x7f3a3c0d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a3c0b38f0 0x7f3a3c0d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53394/0 (socket says 172.21.15.158:53394) 2023-10-16T09:59:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1183950751 learned_addr learned my addr 172.21.15.158:0/1183950751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3b7fe700 1 --2- 172.21.15.158:0/1183950751 >> v2:172.21.15.172:3300/0 conn(0x7f3a3c0d2010 0x7f3a3c06bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1183950751 >> v2:172.21.15.172:3300/0 conn(0x7f3a3c0d2010 msgr2=0x7f3a3c06bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 --2- 172.21.15.158:0/1183950751 >> v2:172.21.15.172:3300/0 conn(0x7f3a3c0d2010 0x7f3a3c06bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1183950751 >> v1:172.21.15.172:6789/0 conn(0x7f3a3c15d840 legacy=0x7f3a3c15dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1183950751 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c15e6c0 con 0x7f3a3c0b38f0 2023-10-16T09:59:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a3a7fc700 1 --2- 172.21.15.158:0/1183950751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0b38f0 0x7f3a3c0d1ad0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0729f0 tx=0x7f3a30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1923d55b1a8998be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a39ffb700 1 -- 172.21.15.158:0/1183950751 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000b870 con 0x7f3a3c0b38f0 2023-10-16T09:59:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a39ffb700 1 -- 172.21.15.158:0/1183950751 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3000b9d0 con 0x7f3a3c0b38f0 2023-10-16T09:59:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.363+0000 7f3a39ffb700 1 -- 172.21.15.158:0/1183950751 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000bd60 con 0x7f3a3c0b38f0 2023-10-16T09:59:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 -- 172.21.15.158:0/1183950751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0b38f0 msgr2=0x7f3a3c0d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1183950751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0b38f0 0x7f3a3c0d1ad0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0729f0 tx=0x7f3a30009bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 -- 172.21.15.158:0/1183950751 shutdown_connections 2023-10-16T09:59:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1183950751 >> v2:172.21.15.172:3300/0 conn(0x7f3a3c0d2010 0x7f3a3c06bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1183950751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0b38f0 0x7f3a3c0d1ad0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 -- 172.21.15.158:0/1183950751 >> 172.21.15.158:0/1183950751 conn(0x7f3a3c06ec00 msgr2=0x7f3a3c06efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 -- 172.21.15.158:0/1183950751 shutdown_connections 2023-10-16T09:59:47.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.364+0000 7f3a412aa700 1 -- 172.21.15.158:0/1183950751 wait complete. 2023-10-16T09:59:47.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.365+0000 7f3a412aa700 1 Processor -- start 2023-10-16T09:59:47.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.365+0000 7f3a412aa700 1 -- start start 2023-10-16T09:59:47.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a412aa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a3c0b38f0 0x7f3a3c0c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:47.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a412aa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0d2010 0x7f3a3c1f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:47.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a412aa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a3c15d840 0x7f3a3c1f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:47.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a412aa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c1f3710 con 0x7f3a3c0d2010 2023-10-16T09:59:47.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a412aa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c1531e0 con 0x7f3a3c0b38f0 2023-10-16T09:59:47.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a412aa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c1f7110 con 0x7f3a3c15d840 2023-10-16T09:59:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0d2010 0x7f3a3c1f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0d2010 0x7f3a3c1f6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53400/0 (socket says 172.21.15.158:53400) 2023-10-16T09:59:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1968815671 learned_addr learned my addr 172.21.15.158:0/1968815671 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3b7fe700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a3c15d840 0x7f3a3c1f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3affd700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a3c0b38f0 0x7f3a3c0c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a3c15d840 msgr2=0x7f3a3c1f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a3c15d840 0x7f3a3c1f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a3c0b38f0 msgr2=0x7f3a3c0c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a3c0b38f0 0x7f3a3c0c7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.366+0000 7f3a3a7fc700 1 -- 172.21.15.158:0/1968815671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a30009550 con 0x7f3a3c0d2010 2023-10-16T09:59:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.367+0000 7f3a3a7fc700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0d2010 0x7f3a3c1f6810 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000c00 tx=0x7f3a30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.367+0000 7f3a237fe700 1 -- 172.21.15.158:0/1968815671 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30004540 con 0x7f3a3c0d2010 2023-10-16T09:59:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.367+0000 7f3a237fe700 1 -- 172.21.15.158:0/1968815671 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a300036a0 con 0x7f3a3c0d2010 2023-10-16T09:59:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.367+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c1f73f0 con 0x7f3a3c0d2010 2023-10-16T09:59:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.367+0000 7f3a237fe700 1 -- 172.21.15.158:0/1968815671 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30003a30 con 0x7f3a3c0d2010 2023-10-16T09:59:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.367+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c1f76d0 con 0x7f3a3c0d2010 2023-10-16T09:59:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.368+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3c1f3850 con 0x7f3a3c0d2010 2023-10-16T09:59:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.368+0000 7f3a237fe700 1 -- 172.21.15.158:0/1968815671 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 40) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3a30021070 con 0x7f3a3c0d2010 2023-10-16T09:59:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.368+0000 7f3a237fe700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f3a2806ad50 0x7f3a2806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.369+0000 7f3a237fe700 1 -- 172.21.15.158:0/1968815671 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7227+0+0 (secure 0 0 0) 0x7f3a3008af30 con 0x7f3a3c0d2010 2023-10-16T09:59:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.369+0000 7f3a3affd700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f3a2806ad50 0x7f3a2806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.369+0000 7f3a3affd700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f3a2806ad50 0x7f3a2806d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c007fd0 tx=0x7f3a2c009040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:47.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.373+0000 7f3a237fe700 1 -- 172.21.15.158:0/1968815671 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a3c1f3850 con 0x7f3a3c0d2010 2023-10-16T09:59:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:47.565+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3a3c1f3850 con 0x7f3a3c0d2010 2023-10-16T09:59:48.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.221+0000 7f3a237fe700 1 -- 172.21.15.158:0/1968815671 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3a3c1f3850 con 0x7f3a3c0d2010 2023-10-16T09:59:48.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.226+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f3a2806ad50 msgr2=0x7f3a2806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:48.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.226+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f3a2806ad50 0x7f3a2806d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c007fd0 tx=0x7f3a2c009040 comp rx=0 tx=0).stop 2023-10-16T09:59:48.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.226+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0d2010 msgr2=0x7f3a3c1f6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:48.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.226+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0d2010 0x7f3a3c1f6810 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000c00 tx=0x7f3a30004750 comp rx=0 tx=0).stop 2023-10-16T09:59:48.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 shutdown_connections 2023-10-16T09:59:48.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f3a2806ad50 0x7f3a2806d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a3c15d840 0x7f3a3c1f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a3c0b38f0 0x7f3a3c0c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 --2- 172.21.15.158:0/1968815671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3c0d2010 0x7f3a3c1f6810 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 >> 172.21.15.158:0/1968815671 conn(0x7f3a3c06ec00 msgr2=0x7f3a3c15dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:48.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 shutdown_connections 2023-10-16T09:59:48.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.230+0000 7f3a412aa700 1 -- 172.21.15.158:0/1968815671 wait complete. 2023-10-16T09:59:48.233 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T09:59:48.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T09:59:48.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.376+0000 7f9fc6c7a700 1 Processor -- start 2023-10-16T09:59:48.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.376+0000 7f9fc6c7a700 1 -- start start 2023-10-16T09:59:48.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc6c7a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc006e0d0 0x7f9fc015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:48.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc6c7a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9fc00cbcf0 0x7f9fc00cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:48.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc6c7a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc015f380 con 0x7f9fc015dc40 2023-10-16T09:59:48.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc6c7a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc0071e70 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc6c7a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc0072150 con 0x7f9fc006e0d0 2023-10-16T09:59:48.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc5217700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9fc00cbcf0 0x7f9fc00cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:48.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc4a16700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9fc015dc40 0x7f9fc006d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53582/0 (socket says 172.21.15.158:53582) 2023-10-16T09:59:48.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc5217700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9fc00cbcf0 0x7f9fc00cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53408/0 (socket says 172.21.15.158:53408) 2023-10-16T09:59:48.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc4a16700 1 -- 172.21.15.158:0/4128588498 learned_addr learned my addr 172.21.15.158:0/4128588498 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:48.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc5217700 1 -- 172.21.15.158:0/4128588498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc006e0d0 msgr2=0x7f9fc015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:48.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fbffff700 1 --2- 172.21.15.158:0/4128588498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc006e0d0 0x7f9fc015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:48.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fbf7fe700 1 -- 172.21.15.158:0/4128588498 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62452488 0 0) 0x7f9fb4001110 con 0x7f9fc015dc40 2023-10-16T09:59:48.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc5217700 1 --2- 172.21.15.158:0/4128588498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc006e0d0 0x7f9fc015ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc5217700 1 -- 172.21.15.158:0/4128588498 >> v1:172.21.15.158:6789/0 conn(0x7f9fc015dc40 legacy=0x7f9fc006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:48.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc5217700 1 -- 172.21.15.158:0/4128588498 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc0072430 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fbffff700 1 --2- 172.21.15.158:0/4128588498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc006e0d0 0x7f9fc015ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:48.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fc5217700 1 --2- 172.21.15.158:0/4128588498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc00cc1a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009d90 tx=0x7f9fb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef832348d6fa55eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:48.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.377+0000 7f9fbf7fe700 1 -- 172.21.15.158:0/4128588498 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb800e070 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fbf7fe700 1 -- 172.21.15.158:0/4128588498 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb800bbd0 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fbf7fe700 1 -- 172.21.15.158:0/4128588498 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8012430 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4128588498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 msgr2=0x7f9fc00cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:48.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4128588498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc00cc1a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009d90 tx=0x7f9fb800b330 comp rx=0 tx=0).stop 2023-10-16T09:59:48.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4128588498 shutdown_connections 2023-10-16T09:59:48.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4128588498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc00cc1a0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4128588498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc006e0d0 0x7f9fc015ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4128588498 >> 172.21.15.158:0/4128588498 conn(0x7f9fc00d03b0 msgr2=0x7f9fc00d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:48.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4128588498 shutdown_connections 2023-10-16T09:59:48.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.378+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4128588498 wait complete. 2023-10-16T09:59:48.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.379+0000 7f9fc6c7a700 1 Processor -- start 2023-10-16T09:59:48.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.379+0000 7f9fc6c7a700 1 -- start start 2023-10-16T09:59:48.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fc6c7a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc006e0d0 0x7f9fc014c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:48.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fc6c7a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc014b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:48.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fc6c7a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc015dc40 0x7f9fc00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:48.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fc6c7a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc014bdb0 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fc6c7a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00c6180 con 0x7f9fc006e0d0 2023-10-16T09:59:48.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fc6c7a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc0158f60 con 0x7f9fc015dc40 2023-10-16T09:59:48.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fbffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc014b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:48.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fbffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc014b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53414/0 (socket says 172.21.15.158:53414) 2023-10-16T09:59:48.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.380+0000 7f9fbffff700 1 -- 172.21.15.158:0/4193522401 learned_addr learned my addr 172.21.15.158:0/4193522401 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:48.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fc4a16700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc006e0d0 0x7f9fc014c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:48.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fc5217700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc015dc40 0x7f9fc00c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:48.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbffff700 1 -- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc015dc40 msgr2=0x7f9fc00c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:48.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbffff700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc015dc40 0x7f9fc00c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbffff700 1 -- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc006e0d0 msgr2=0x7f9fc014c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:48.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbffff700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc006e0d0 0x7f9fc014c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:48.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbffff700 1 -- 172.21.15.158:0/4193522401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb8009a70 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbffff700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc014b870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00c750 tx=0x7f9fac00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:48.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbd7fa700 1 -- 172.21.15.158:0/4193522401 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac011800 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbd7fa700 1 -- 172.21.15.158:0/4193522401 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fac011960 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc00c63d0 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fbd7fa700 1 -- 172.21.15.158:0/4193522401 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac011cd0 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.381+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc00c6990 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.382+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc015c280 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.382+0000 7f9fbd7fa700 1 -- 172.21.15.158:0/4193522401 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 41) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9fac021350 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.383+0000 7f9fbd7fa700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9fa406af90 0x7f9fa406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:48.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.383+0000 7f9fbd7fa700 1 -- 172.21.15.158:0/4193522401 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f9fac089a70 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.383+0000 7f9fc4a16700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9fa406af90 0x7f9fa406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:48.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.383+0000 7f9fc4a16700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9fa406af90 0x7f9fa406d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4000f60 tx=0x7f9fb4007c90 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:48.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.387+0000 7f9fbd7fa700 1 -- 172.21.15.158:0/4193522401 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9fc015c280 con 0x7f9fc00cbcf0 2023-10-16T09:59:48.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:48.578+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9fc015c280 con 0x7f9fc00cbcf0 2023-10-16T09:59:49.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.561+0000 7f9fbd7fa700 1 -- 172.21.15.158:0/4193522401 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9fc015c280 con 0x7f9fc00cbcf0 2023-10-16T09:59:49.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.564+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9fa406af90 msgr2=0x7f9fa406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.564+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9fa406af90 0x7f9fa406d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4000f60 tx=0x7f9fb4007c90 comp rx=0 tx=0).stop 2023-10-16T09:59:49.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.564+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 msgr2=0x7f9fc014b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.564+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc014b870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00c750 tx=0x7f9fac00cb10 comp rx=0 tx=0).stop 2023-10-16T09:59:49.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 shutdown_connections 2023-10-16T09:59:49.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9fa406af90 0x7f9fa406d450 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc015dc40 0x7f9fc00c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc006e0d0 0x7f9fc014c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 --2- 172.21.15.158:0/4193522401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc00cbcf0 0x7f9fc014b870 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 >> 172.21.15.158:0/4193522401 conn(0x7f9fc00d03b0 msgr2=0x7f9fc00cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:49.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 shutdown_connections 2023-10-16T09:59:49.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.565+0000 7f9fc6c7a700 1 -- 172.21.15.158:0/4193522401 wait complete. 2023-10-16T09:59:49.569 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T09:59:49.582 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T09:59:49.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.709+0000 7fdc1fb9d700 1 Processor -- start 2023-10-16T09:59:49.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.709+0000 7fdc1fb9d700 1 -- start start 2023-10-16T09:59:49.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1fb9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdc1806afe0 0x7fdc180caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1fb9d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc180cbd30 con 0x7fdc180cc590 2023-10-16T09:59:49.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1fb9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc180cc1c0 con 0x7fdc1815da40 2023-10-16T09:59:49.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1fb9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1815de20 con 0x7fdc1806afe0 2023-10-16T09:59:49.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1e13a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdc180cc590 0x7fdc180afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53600/0 (socket says 172.21.15.158:53600) 2023-10-16T09:59:49.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1e13a700 1 -- 172.21.15.158:0/3463721419 learned_addr learned my addr 172.21.15.158:0/3463721419 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:49.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1d138700 1 --2- 172.21.15.158:0/3463721419 >> v2:172.21.15.172:3300/0 conn(0x7fdc1806afe0 0x7fdc180caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610624138 0 0) 0x7fdc180cbd30 con 0x7fdc180cc590 2023-10-16T09:59:49.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc040035b0 con 0x7fdc180cc590 2023-10-16T09:59:49.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604469674 0 0) 0x7fdc040035b0 con 0x7fdc180cc590 2023-10-16T09:59:49.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 >> v2:172.21.15.172:3300/0 conn(0x7fdc1806afe0 msgr2=0x7fdc180caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:49.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 --2- 172.21.15.158:0/3463721419 >> v2:172.21.15.172:3300/0 conn(0x7fdc1806afe0 0x7fdc180caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 >> v1:172.21.15.172:6789/0 conn(0x7fdc1815da40 legacy=0x7fdc1806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180ce260 con 0x7fdc180cc590 2023-10-16T09:59:49.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.710+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fdc100019f0 con 0x7fdc180cc590 2023-10-16T09:59:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 >> v1:172.21.15.158:6789/0 conn(0x7fdc180cc590 legacy=0x7fdc180afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04003a00 0x7fdc04003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04004850 0x7fdc04004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc04005210 0x7fdc0400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1815de20 con 0x7fdc04003a00 2023-10-16T09:59:49.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0400c0b0 con 0x7fdc04005210 2023-10-16T09:59:49.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0400c270 con 0x7fdc04004850 2023-10-16T09:59:49.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d939700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04003a00 0x7fdc04003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1e13a700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc04005210 0x7fdc0400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d138700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04004850 0x7fdc04004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d939700 1 -- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04004850 msgr2=0x7fdc04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d939700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04004850 0x7fdc04004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d939700 1 -- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc04005210 msgr2=0x7fdc0400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d939700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc04005210 0x7fdc0400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d939700 1 -- 172.21.15.158:0/3463721419 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040035b0 con 0x7fdc04003a00 2023-10-16T09:59:49.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1e13a700 1 -- 172.21.15.158:0/3463721419 reap_dead start 2023-10-16T09:59:49.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1d939700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04003a00 0x7fdc04003e20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fdc140017c0 tx=0x7fdc04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7e75d386dcffff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:49.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc140052d0 con 0x7fdc04003a00 2023-10-16T09:59:49.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14005450 con 0x7fdc04003a00 2023-10-16T09:59:49.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.711+0000 7fdc1c937700 1 -- 172.21.15.158:0/3463721419 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc140038c0 con 0x7fdc04003a00 2023-10-16T09:59:49.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.712+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/3463721419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04003a00 msgr2=0x7fdc04003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.712+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04003a00 0x7fdc04003e20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fdc140017c0 tx=0x7fdc04005880 comp rx=0 tx=0).stop 2023-10-16T09:59:49.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.712+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/3463721419 shutdown_connections 2023-10-16T09:59:49.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.712+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/3463721419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04003a00 0x7fdc04003e20 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.712+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/3463721419 >> 172.21.15.158:0/3463721419 conn(0x7fdc18099a10 msgr2=0x7fdc180d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:49.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.712+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/3463721419 shutdown_connections 2023-10-16T09:59:49.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.712+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/3463721419 wait complete. 2023-10-16T09:59:49.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.713+0000 7fdc1fb9d700 1 Processor -- start 2023-10-16T09:59:49.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.713+0000 7fdc1fb9d700 1 -- start start 2023-10-16T09:59:49.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1fb9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04003a00 0x7fdc180c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1fb9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc18150c10 0x7fdc180ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1fb9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc180c74a0 0x7fdc1806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1fb9d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180ca780 con 0x7fdc18150c10 2023-10-16T09:59:49.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1fb9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180c7da0 con 0x7fdc180c74a0 2023-10-16T09:59:49.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1fb9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180c8080 con 0x7fdc04003a00 2023-10-16T09:59:49.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc18150c10 0x7fdc180ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc18150c10 0x7fdc180ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53434/0 (socket says 172.21.15.158:53434) 2023-10-16T09:59:49.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 -- 172.21.15.158:0/1519661507 learned_addr learned my addr 172.21.15.158:0/1519661507 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:49.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1e13a700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc180c74a0 0x7fdc1806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d939700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04003a00 0x7fdc180c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 -- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04003a00 msgr2=0x7fdc180c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04003a00 0x7fdc180c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 -- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc180c74a0 msgr2=0x7fdc1806eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc180c74a0 0x7fdc1806eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.714+0000 7fdc1d138700 1 -- 172.21.15.158:0/1519661507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180c8360 con 0x7fdc18150c10 2023-10-16T09:59:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.715+0000 7fdc1d138700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc18150c10 0x7fdc180ca240 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fdc080108c0 tx=0x7fdc08010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.715+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08016670 con 0x7fdc18150c10 2023-10-16T09:59:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.715+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc080167d0 con 0x7fdc18150c10 2023-10-16T09:59:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.715+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc180b4a10 con 0x7fdc18150c10 2023-10-16T09:59:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.715+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08016ba0 con 0x7fdc18150c10 2023-10-16T09:59:49.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.715+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc180c7950 con 0x7fdc18150c10 2023-10-16T09:59:49.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.717+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc180b4d80 con 0x7fdc18150c10 2023-10-16T09:59:49.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.717+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 41) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdc08016d00 con 0x7fdc18150c10 2023-10-16T09:59:49.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.717+0000 7fdc0e7fc700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbfc06af90 0x7fdbfc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:49.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.717+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fdc080899d0 con 0x7fdc18150c10 2023-10-16T09:59:49.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.717+0000 7fdc1d939700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbfc06af90 0x7fdbfc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:49.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.718+0000 7fdc1d939700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbfc06af90 0x7fdbfc06d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000c00 tx=0x7fdc14009030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:49.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.724+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc080547f0 con 0x7fdc18150c10 2023-10-16T09:59:49.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.786+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdc08054bc0 con 0x7fdc18150c10 2023-10-16T09:59:49.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.919+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fdc180b4810 con 0x7fdc18150c10 2023-10-16T09:59:49.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.922+0000 7fdc0e7fc700 1 -- 172.21.15.158:0/1519661507 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7fdc08085560 con 0x7fdc18150c10 2023-10-16T09:59:49.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbfc06af90 msgr2=0x7fdbfc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbfc06af90 0x7fdbfc06d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000c00 tx=0x7fdc14009030 comp rx=0 tx=0).stop 2023-10-16T09:59:49.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc18150c10 msgr2=0x7fdc180ca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:49.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc18150c10 0x7fdc180ca240 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fdc080108c0 tx=0x7fdc08010bd0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 shutdown_connections 2023-10-16T09:59:49.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fdbfc06af90 0x7fdbfc06d450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc180c74a0 0x7fdc1806eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc04003a00 0x7fdc180c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 --2- 172.21.15.158:0/1519661507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc18150c10 0x7fdc180ca240 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:49.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 >> 172.21.15.158:0/1519661507 conn(0x7fdc18099a10 msgr2=0x7fdc18160a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:49.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 shutdown_connections 2023-10-16T09:59:49.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:49.925+0000 7fdc1fb9d700 1 -- 172.21.15.158:0/1519661507 wait complete. 2023-10-16T09:59:49.939 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T09:59:50.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.060+0000 7ff0f4f60700 1 Processor -- start 2023-10-16T09:59:50.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.060+0000 7ff0f4f60700 1 -- start start 2023-10-16T09:59:50.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0f4f60700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 0x7ff0f006b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0f4f60700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff0f015e520 0x7ff0f006d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0f4f60700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f015e9f0 con 0x7ff0f006be20 2023-10-16T09:59:50.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0f4f60700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f00ce460 con 0x7ff0f015e520 2023-10-16T09:59:50.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0f4f60700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f006c2a0 con 0x7ff0f015da30 2023-10-16T09:59:50.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0edd9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff0f006be20 0x7ff0f015de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53616/0 (socket says 172.21.15.158:53616) 2023-10-16T09:59:50.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0edd9b700 1 -- 172.21.15.158:0/2396385624 learned_addr learned my addr 172.21.15.158:0/2396385624 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:50.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2396385624 >> v2:172.21.15.172:3300/0 conn(0x7ff0f015e520 0x7ff0f006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0ee59c700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 0x7ff0f006b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255153146 0 0) 0x7ff0f015e9f0 con 0x7ff0f006be20 2023-10-16T09:59:50.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.061+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d80035b0 con 0x7ff0f006be20 2023-10-16T09:59:50.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1251898905 0 0) 0x7ff0d80035b0 con 0x7ff0f006be20 2023-10-16T09:59:50.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 msgr2=0x7ff0f006b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:50.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 0x7ff0f006b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 >> v2:172.21.15.172:3300/0 conn(0x7ff0f015e520 msgr2=0x7ff0f006d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:50.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 --2- 172.21.15.158:0/2396385624 >> v2:172.21.15.172:3300/0 conn(0x7ff0f015e520 0x7ff0f006d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f015ecd0 con 0x7ff0f006be20 2023-10-16T09:59:50.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff0e0001670 con 0x7ff0f006be20 2023-10-16T09:59:50.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 >> v1:172.21.15.158:6789/0 conn(0x7ff0f006be20 legacy=0x7ff0f015de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 0x7ff0d8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0d8003f80 0x7ff0d8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0d8004d90 0x7ff0d8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00ce460 con 0x7ff0d8004d90 2023-10-16T09:59:50.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f006c2a0 con 0x7ff0d8003f80 2023-10-16T09:59:50.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d800ba10 con 0x7ff0f015da30 2023-10-16T09:59:50.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0d8004d90 0x7ff0d8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ee59c700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 0x7ff0d8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0edd9b700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0d8003f80 0x7ff0d8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 msgr2=0x7ff0d8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:50.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0f015da30 0x7ff0d8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0d8003f80 msgr2=0x7ff0d8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:50.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0d8003f80 0x7ff0d8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2396385624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d80035b0 con 0x7ff0d8004d90 2023-10-16T09:59:50.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0edd9b700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0d8003f80 0x7ff0d8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:50.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0d8004d90 0x7ff0d8005210 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0071d70 tx=0x7ff0d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f424d5496be940 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800eb70 con 0x7ff0d8004d90 2023-10-16T09:59:50.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e800f040 con 0x7ff0d8004d90 2023-10-16T09:59:50.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.062+0000 7ff0ed59a700 1 -- 172.21.15.158:0/2396385624 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800eed0 con 0x7ff0d8004d90 2023-10-16T09:59:50.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.063+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2396385624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0d8004d90 msgr2=0x7ff0d8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.063+0000 7ff0f4f60700 1 --2- 172.21.15.158:0/2396385624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0d8004d90 0x7ff0d8005210 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0071d70 tx=0x7ff0d8005880 comp rx=0 tx=0).stop 2023-10-16T09:59:50.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.063+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2396385624 reap_dead start 2023-10-16T09:59:50.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.063+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2396385624 shutdown_connections 2023-10-16T09:59:50.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.063+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2396385624 >> 172.21.15.158:0/2396385624 conn(0x7ff0f00710a0 msgr2=0x7ff0f00714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:50.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.063+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2396385624 shutdown_connections 2023-10-16T09:59:50.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.063+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2396385624 wait complete. 2023-10-16T09:59:50.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.064+0000 7ff0f4f60700 1 Processor -- start 2023-10-16T09:59:50.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.064+0000 7ff0f4f60700 1 -- start start 2023-10-16T09:59:50.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0f4f60700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0d8004d90 0x7ff0f00c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0f4f60700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0f00b4f60 0x7ff0f00c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0f4f60700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0f015b180 0x7ff0f00ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0f4f60700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c88c0 con 0x7ff0f015b180 2023-10-16T09:59:50.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0f4f60700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00b54b0 con 0x7ff0f00b4f60 2023-10-16T09:59:50.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0f4f60700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c7f90 con 0x7ff0d8004d90 2023-10-16T09:59:50.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0eed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0f015b180 0x7ff0f00ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0eed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0f015b180 0x7ff0f00ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53454/0 (socket says 172.21.15.158:53454) 2023-10-16T09:59:50.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2743029426 learned_addr learned my addr 172.21.15.158:0/2743029426 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:50.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0edd9b700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0f00b4f60 0x7ff0f00c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.065+0000 7ff0ee59c700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0d8004d90 0x7ff0f00c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0d8004d90 msgr2=0x7ff0f00c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0d8004d90 0x7ff0f00c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0f00b4f60 msgr2=0x7ff0f00c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0f00b4f60 0x7ff0f00c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0eed9d700 1 -- 172.21.15.158:0/2743029426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f00c8270 con 0x7ff0f015b180 2023-10-16T09:59:50.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0ee59c700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0d8004d90 0x7ff0f00c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:50.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0eed9d700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0f015b180 0x7ff0f00ca590 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800b010 tx=0x7ff0e8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0deffd700 1 -- 172.21.15.158:0/2743029426 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8007000 con 0x7ff0f015b180 2023-10-16T09:59:50.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0deffd700 1 -- 172.21.15.158:0/2743029426 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e8022070 con 0x7ff0f015b180 2023-10-16T09:59:50.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f00c8550 con 0x7ff0f015b180 2023-10-16T09:59:50.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0deffd700 1 -- 172.21.15.158:0/2743029426 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8029480 con 0x7ff0f015b180 2023-10-16T09:59:50.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.066+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f015b600 con 0x7ff0f015b180 2023-10-16T09:59:50.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.067+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f00681b0 con 0x7ff0f015b180 2023-10-16T09:59:50.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.067+0000 7ff0deffd700 1 -- 172.21.15.158:0/2743029426 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff0e801e400 con 0x7ff0f015b180 2023-10-16T09:59:50.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.068+0000 7ff0deffd700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff0d006b020 0x7ff0d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.068+0000 7ff0deffd700 1 -- 172.21.15.158:0/2743029426 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7ff0e8024070 con 0x7ff0f015b180 2023-10-16T09:59:50.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.068+0000 7ff0ee59c700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff0d006b020 0x7ff0d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.068+0000 7ff0ee59c700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff0d006b020 0x7ff0d006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009c60 tx=0x7ff0e4008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.072+0000 7ff0deffd700 1 -- 172.21.15.158:0/2743029426 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff0e8056610 con 0x7ff0f015b180 2023-10-16T09:59:50.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.276+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff0f00b5ab0 con 0x7ff0f015b180 2023-10-16T09:59:50.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.279+0000 7ff0deffd700 1 -- 172.21.15.158:0/2743029426 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff0e8059c30 con 0x7ff0f015b180 2023-10-16T09:59:50.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff0d006b020 msgr2=0x7ff0d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff0d006b020 0x7ff0d006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009c60 tx=0x7ff0e4008040 comp rx=0 tx=0).stop 2023-10-16T09:59:50.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0f015b180 msgr2=0x7ff0f00ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0f015b180 0x7ff0f00ca590 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800b010 tx=0x7ff0e8009750 comp rx=0 tx=0).stop 2023-10-16T09:59:50.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 shutdown_connections 2023-10-16T09:59:50.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff0d006b020 0x7ff0d006d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0f015b180 0x7ff0f00ca590 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0d8004d90 0x7ff0f00c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 --2- 172.21.15.158:0/2743029426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0f00b4f60 0x7ff0f00c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 >> 172.21.15.158:0/2743029426 conn(0x7ff0f00710a0 msgr2=0x7ff0f00af590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:50.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 shutdown_connections 2023-10-16T09:59:50.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.281+0000 7ff0f4f60700 1 -- 172.21.15.158:0/2743029426 wait complete. 2023-10-16T09:59:50.294 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T09:59:50.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.415+0000 7f7aa1f3a700 1 Processor -- start 2023-10-16T09:59:50.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.415+0000 7f7aa1f3a700 1 -- start start 2023-10-16T09:59:50.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7aa1f3a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15b590 0x7f7a9c0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7aa1f3a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a9c0cbaf0 0x7f7a9c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7aa1f3a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c0cf4c0 con 0x7f7a9c15f1d0 2023-10-16T09:59:50.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7aa1f3a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a9c0cc4c0 con 0x7f7a9c0cbaf0 2023-10-16T09:59:50.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7aa1f3a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c0b49e0 con 0x7f7a9c15b590 2023-10-16T09:59:50.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7a9bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a9c0cbaf0 0x7f7a9c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7a9affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7a9c15f1d0 0x7f7a9c0b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53634/0 (socket says 172.21.15.158:53634) 2023-10-16T09:59:50.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.416+0000 7f7a9affd700 1 -- 172.21.15.158:0/3176073950 learned_addr learned my addr 172.21.15.158:0/3176073950 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:50.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9b7fe700 1 --2- 172.21.15.158:0/3176073950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15b590 0x7f7a9c0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9bfff700 1 -- 172.21.15.158:0/3176073950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15b590 msgr2=0x7f7a9c0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:50.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9bfff700 1 --2- 172.21.15.158:0/3176073950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15b590 0x7f7a9c0ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9bfff700 1 -- 172.21.15.158:0/3176073950 >> v1:172.21.15.158:6789/0 conn(0x7f7a9c15f1d0 legacy=0x7f7a9c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9bfff700 1 -- 172.21.15.158:0/3176073950 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c15f970 con 0x7f7a9c0cbaf0 2023-10-16T09:59:50.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9b7fe700 1 --2- 172.21.15.158:0/3176073950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15b590 0x7f7a9c0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:50.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9bfff700 1 --2- 172.21.15.158:0/3176073950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c0cbaf0 0x7f7a9c0cbf80 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009dc0 tx=0x7f7a9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2aa36f15f7c8e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9a7fc700 1 -- 172.21.15.158:0/3176073950 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9400e070 con 0x7f7a9c0cbaf0 2023-10-16T09:59:50.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9a7fc700 1 -- 172.21.15.158:0/3176073950 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9400bb90 con 0x7f7a9c0cbaf0 2023-10-16T09:59:50.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7a9a7fc700 1 -- 172.21.15.158:0/3176073950 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94012430 con 0x7f7a9c0cbaf0 2023-10-16T09:59:50.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3176073950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c0cbaf0 msgr2=0x7f7a9c0cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3176073950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c0cbaf0 0x7f7a9c0cbf80 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009dc0 tx=0x7f7a9400b330 comp rx=0 tx=0).stop 2023-10-16T09:59:50.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3176073950 shutdown_connections 2023-10-16T09:59:50.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3176073950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c0cbaf0 0x7f7a9c0cbf80 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3176073950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15b590 0x7f7a9c0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.417+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3176073950 >> 172.21.15.158:0/3176073950 conn(0x7f7a9c06a8d0 msgr2=0x7f7a9c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:50.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.418+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3176073950 shutdown_connections 2023-10-16T09:59:50.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.418+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3176073950 wait complete. 2023-10-16T09:59:50.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.419+0000 7f7aa1f3a700 1 Processor -- start 2023-10-16T09:59:50.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.419+0000 7f7aa1f3a700 1 -- start start 2023-10-16T09:59:50.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c0cbaf0 0x7f7a9c0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c15b590 0x7f7a9c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c15f1d0 0x7f7a9c0a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1f2c80 con 0x7f7a9c15b590 2023-10-16T09:59:50.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c0cb940 con 0x7f7a9c15f1d0 2023-10-16T09:59:50.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1f2070 con 0x7f7a9c0cbaf0 2023-10-16T09:59:50.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c15b590 0x7f7a9c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c15b590 0x7f7a9c0cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53466/0 (socket says 172.21.15.158:53466) 2023-10-16T09:59:50.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 -- 172.21.15.158:0/3803987950 learned_addr learned my addr 172.21.15.158:0/3803987950 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:50.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9b7fe700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c0cbaf0 0x7f7a9c0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 -- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c0cbaf0 msgr2=0x7f7a9c0c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9bfff700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c15f1d0 0x7f7a9c0a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c0cbaf0 0x7f7a9c0c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 -- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c15f1d0 msgr2=0x7f7a9c0a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c15f1d0 0x7f7a9c0a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 -- 172.21.15.158:0/3803987950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94009a70 con 0x7f7a9c15b590 2023-10-16T09:59:50.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a9affd700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c15b590 0x7f7a9c0cb400 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00c550 tx=0x7f7a8c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a83fff700 1 -- 172.21.15.158:0/3803987950 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c005d70 con 0x7f7a9c15b590 2023-10-16T09:59:50.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a83fff700 1 -- 172.21.15.158:0/3803987950 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8c0070d0 con 0x7f7a9c15b590 2023-10-16T09:59:50.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c1f2350 con 0x7f7a9c15b590 2023-10-16T09:59:50.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7a83fff700 1 -- 172.21.15.158:0/3803987950 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c007440 con 0x7f7a9c15b590 2023-10-16T09:59:50.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.420+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c1f2910 con 0x7f7a9c15b590 2023-10-16T09:59:50.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.421+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9c0681b0 con 0x7f7a9c15b590 2023-10-16T09:59:50.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.422+0000 7f7a83fff700 1 -- 172.21.15.158:0/3803987950 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a8c0075a0 con 0x7f7a9c15b590 2023-10-16T09:59:50.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.422+0000 7f7a83fff700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7a8806b020 0x7f7a8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.422+0000 7f7a83fff700 1 -- 172.21.15.158:0/3803987950 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f7a8c088e00 con 0x7f7a9c15b590 2023-10-16T09:59:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.422+0000 7f7a9b7fe700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7a8806b020 0x7f7a8806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.423+0000 7f7a9b7fe700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7a8806b020 0x7f7a8806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006fd0 tx=0x7f7a90008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.427+0000 7f7a83fff700 1 -- 172.21.15.158:0/3803987950 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a8c053bb0 con 0x7f7a9c15b590 2023-10-16T09:59:50.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.621+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f7a9c0b07c0 con 0x7f7a9c15b590 2023-10-16T09:59:50.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.624+0000 7f7a83fff700 1 -- 172.21.15.158:0/3803987950 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7f7a8c0571d0 con 0x7f7a9c15b590 2023-10-16T09:59:50.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7a8806b020 msgr2=0x7f7a8806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7a8806b020 0x7f7a8806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006fd0 tx=0x7f7a90008040 comp rx=0 tx=0).stop 2023-10-16T09:59:50.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c15b590 msgr2=0x7f7a9c0cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c15b590 0x7f7a9c0cb400 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00c550 tx=0x7f7a8c00c910 comp rx=0 tx=0).stop 2023-10-16T09:59:50.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 shutdown_connections 2023-10-16T09:59:50.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7a8806b020 0x7f7a8806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c15f1d0 0x7f7a9c0a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c0cbaf0 0x7f7a9c0c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 --2- 172.21.15.158:0/3803987950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c15b590 0x7f7a9c0cb400 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 >> 172.21.15.158:0/3803987950 conn(0x7f7a9c06a8d0 msgr2=0x7f7a9c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:50.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 shutdown_connections 2023-10-16T09:59:50.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.626+0000 7f7aa1f3a700 1 -- 172.21.15.158:0/3803987950 wait complete. 2023-10-16T09:59:50.640 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T09:59:50.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.760+0000 7f16bd3ae700 1 Processor -- start 2023-10-16T09:59:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.760+0000 7f16bd3ae700 1 -- start start 2023-10-16T09:59:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.760+0000 7f16bd3ae700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f16b80ca7d0 0x7f16b8071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.760+0000 7f16bd3ae700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b815dc90 con 0x7f16b81599b0 2023-10-16T09:59:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.760+0000 7f16bd3ae700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b815df70 con 0x7f16b815d7b0 2023-10-16T09:59:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.760+0000 7f16bd3ae700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b815e250 con 0x7f16b80ca7d0 2023-10-16T09:59:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f16b80ca7d0 0x7f16b8071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f16b81599b0 0x7f16b80ca0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53648/0 (socket says 172.21.15.158:53648) 2023-10-16T09:59:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f16b80ca7d0 0x7f16b8071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53474/0 (socket says 172.21.15.158:53474) 2023-10-16T09:59:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b6ffd700 1 -- 172.21.15.158:0/3740607728 learned_addr learned my addr 172.21.15.158:0/3740607728 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b67fc700 1 -- 172.21.15.158:0/3740607728 >> v1:172.21.15.172:6789/0 conn(0x7f16b815d7b0 legacy=0x7f16b8071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b67fc700 1 -- 172.21.15.158:0/3740607728 >> v1:172.21.15.158:6789/0 conn(0x7f16b81599b0 legacy=0x7f16b80ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b67fc700 1 -- 172.21.15.158:0/3740607728 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b815e530 con 0x7f16b80ca7d0 2023-10-16T09:59:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b5ffb700 1 -- 172.21.15.158:0/3740607728 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343920097 0 0) 0x7f16b815dc90 con 0x7f16b81599b0 2023-10-16T09:59:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b67fc700 1 --2- 172.21.15.158:0/3740607728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b8071240 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00a9e0 tx=0x7f16ac00acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38dad9b26cde94b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b5ffb700 1 -- 172.21.15.158:0/3740607728 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac00cb30 con 0x7f16b80ca7d0 2023-10-16T09:59:50.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b5ffb700 1 -- 172.21.15.158:0/3740607728 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac00cc90 con 0x7f16b80ca7d0 2023-10-16T09:59:50.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.761+0000 7f16b5ffb700 1 -- 172.21.15.158:0/3740607728 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac004030 con 0x7f16b80ca7d0 2023-10-16T09:59:50.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.762+0000 7f16bd3ae700 1 -- 172.21.15.158:0/3740607728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 msgr2=0x7f16b8071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.762+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/3740607728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b8071240 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00a9e0 tx=0x7f16ac00acf0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.762+0000 7f16bd3ae700 1 -- 172.21.15.158:0/3740607728 shutdown_connections 2023-10-16T09:59:50.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.762+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/3740607728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b8071240 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.762+0000 7f16bd3ae700 1 -- 172.21.15.158:0/3740607728 >> 172.21.15.158:0/3740607728 conn(0x7f16b806af00 msgr2=0x7f16b80cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:50.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.762+0000 7f16bd3ae700 1 -- 172.21.15.158:0/3740607728 shutdown_connections 2023-10-16T09:59:50.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.762+0000 7f16bd3ae700 1 -- 172.21.15.158:0/3740607728 wait complete. 2023-10-16T09:59:50.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.763+0000 7f16bd3ae700 1 Processor -- start 2023-10-16T09:59:50.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.763+0000 7f16bd3ae700 1 -- start start 2023-10-16T09:59:50.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16bd3ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b80c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16bd3ae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16b81599b0 0x7f16b80c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16bd3ae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16b815d7b0 0x7f16b80b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16bd3ae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c73b0 con 0x7f16b80ca7d0 2023-10-16T09:59:50.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16bd3ae700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c7690 con 0x7f16b815d7b0 2023-10-16T09:59:50.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16bd3ae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c7970 con 0x7f16b81599b0 2023-10-16T09:59:50.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b80c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b80c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53478/0 (socket says 172.21.15.158:53478) 2023-10-16T09:59:50.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 -- 172.21.15.158:0/2692633647 learned_addr learned my addr 172.21.15.158:0/2692633647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:50.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b67fc700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16b81599b0 0x7f16b80c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b77fe700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16b815d7b0 0x7f16b80b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 -- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16b81599b0 msgr2=0x7f16b80c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16b81599b0 0x7f16b80c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 -- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16b815d7b0 msgr2=0x7f16b80b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:50.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16b815d7b0 0x7f16b80b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:50.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 -- 172.21.15.158:0/2692633647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac00a640 con 0x7f16b80ca7d0 2023-10-16T09:59:50.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b77fe700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16b815d7b0 0x7f16b80b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:50.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.764+0000 7f16b6ffd700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b80c8ee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f16a80011e0 tx=0x7f16a800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.765+0000 7f169f7fe700 1 -- 172.21.15.158:0/2692633647 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8004c80 con 0x7f16b80ca7d0 2023-10-16T09:59:50.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.765+0000 7f169f7fe700 1 -- 172.21.15.158:0/2692633647 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a8004e00 con 0x7f16b80ca7d0 2023-10-16T09:59:50.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.765+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b80c7c50 con 0x7f16b80ca7d0 2023-10-16T09:59:50.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.765+0000 7f169f7fe700 1 -- 172.21.15.158:0/2692633647 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8007310 con 0x7f16b80ca7d0 2023-10-16T09:59:50.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.765+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b80c70b0 con 0x7f16b80ca7d0 2023-10-16T09:59:50.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.765+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80ce5c0 con 0x7f16b80ca7d0 2023-10-16T09:59:50.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.766+0000 7f169f7fe700 1 -- 172.21.15.158:0/2692633647 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f16a8007470 con 0x7f16b80ca7d0 2023-10-16T09:59:50.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.766+0000 7f169f7fe700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f16a406b020 0x7f16a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:50.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.767+0000 7f16b67fc700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f16a406b020 0x7f16a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:50.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.767+0000 7f169f7fe700 1 -- 172.21.15.158:0/2692633647 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f16a8089740 con 0x7f16b80ca7d0 2023-10-16T09:59:50.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.767+0000 7f16b67fc700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f16a406b020 0x7f16a406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f16b80ccf20 tx=0x7f16ac008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:50.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:50.771+0000 7f169f7fe700 1 -- 172.21.15.158:0/2692633647 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16a80544d0 con 0x7f16b80ca7d0 2023-10-16T09:59:51.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.044+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f16b80681b0 con 0x7f16b80ca7d0 2023-10-16T09:59:51.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.564+0000 7f169f7fe700 1 -- 172.21.15.158:0/2692633647 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v603) v1 ==== 156+0+0 (secure 0 0 0) 0x7f16a8057af0 con 0x7f16b80ca7d0 2023-10-16T09:59:51.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f16a406b020 msgr2=0x7f16a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f16a406b020 0x7f16a406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f16b80ccf20 tx=0x7f16ac008040 comp rx=0 tx=0).stop 2023-10-16T09:59:51.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 msgr2=0x7f16b80c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b80c8ee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f16a80011e0 tx=0x7f16a800cae0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 shutdown_connections 2023-10-16T09:59:51.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f16a406b020 0x7f16a406d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16b815d7b0 0x7f16b80b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16b80ca7d0 0x7f16b80c8ee0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 --2- 172.21.15.158:0/2692633647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16b81599b0 0x7f16b80c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 >> 172.21.15.158:0/2692633647 conn(0x7f16b806af00 msgr2=0x7f16b8156400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:51.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 shutdown_connections 2023-10-16T09:59:51.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.566+0000 7f16bd3ae700 1 -- 172.21.15.158:0/2692633647 wait complete. 2023-10-16T09:59:51.582 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-10-16T09:59:51.582 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-10-16T09:59:51.582 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T09:59:51.583 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:51.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.708+0000 7f69784e1700 1 Processor -- start 2023-10-16T09:59:51.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.708+0000 7f69784e1700 1 -- start start 2023-10-16T09:59:51.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.709+0000 7f69784e1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69700729b0 0x7f697006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:51.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.709+0000 7f69784e1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6970161270 con 0x7f697006ab40 2023-10-16T09:59:51.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.709+0000 7f69784e1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6970161510 con 0x7f697006e5d0 2023-10-16T09:59:51.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.709+0000 7f69784e1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69701617f0 con 0x7f69700729b0 2023-10-16T09:59:51.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.709+0000 7f6976a7e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69700729b0 0x7f697006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:51.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.709+0000 7f697627d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f697006ab40 0x7f69700722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53662/0 (socket says 172.21.15.158:53662) 2023-10-16T09:59:51.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.709+0000 7f697627d700 1 -- 172.21.15.158:0/178567548 learned_addr learned my addr 172.21.15.158:0/178567548 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:51.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f6976a7e700 1 -- 172.21.15.158:0/178567548 >> v1:172.21.15.172:6789/0 conn(0x7f697006e5d0 legacy=0x7f697015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:51.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f6976a7e700 1 -- 172.21.15.158:0/178567548 >> v1:172.21.15.158:6789/0 conn(0x7f697006ab40 legacy=0x7f69700722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f697527b700 1 -- 172.21.15.158:0/178567548 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438209681 0 0) 0x7f6970161270 con 0x7f697006ab40 2023-10-16T09:59:51.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f6976a7e700 1 -- 172.21.15.158:0/178567548 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6970161ad0 con 0x7f69700729b0 2023-10-16T09:59:51.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f6976a7e700 1 --2- 172.21.15.158:0/178567548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700729b0 0x7f697006d820 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d90 tx=0x7f696800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fb0b983f837860 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:51.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f697527b700 1 -- 172.21.15.158:0/178567548 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696800e070 con 0x7f69700729b0 2023-10-16T09:59:51.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f697527b700 1 -- 172.21.15.158:0/178567548 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696800bbd0 con 0x7f69700729b0 2023-10-16T09:59:51.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f697527b700 1 -- 172.21.15.158:0/178567548 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968012430 con 0x7f69700729b0 2023-10-16T09:59:51.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f69784e1700 1 -- 172.21.15.158:0/178567548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700729b0 msgr2=0x7f697006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.710+0000 7f69784e1700 1 --2- 172.21.15.158:0/178567548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700729b0 0x7f697006d820 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d90 tx=0x7f696800b330 comp rx=0 tx=0).stop 2023-10-16T09:59:51.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.711+0000 7f69784e1700 1 -- 172.21.15.158:0/178567548 shutdown_connections 2023-10-16T09:59:51.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.711+0000 7f69784e1700 1 --2- 172.21.15.158:0/178567548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700729b0 0x7f697006d820 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.711+0000 7f69784e1700 1 -- 172.21.15.158:0/178567548 >> 172.21.15.158:0/178567548 conn(0x7f69700d02b0 msgr2=0x7f69700d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:51.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.711+0000 7f69784e1700 1 -- 172.21.15.158:0/178567548 shutdown_connections 2023-10-16T09:59:51.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.711+0000 7f69784e1700 1 -- 172.21.15.158:0/178567548 wait complete. 2023-10-16T09:59:51.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.712+0000 7f69784e1700 1 Processor -- start 2023-10-16T09:59:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.712+0000 7f69784e1700 1 -- start start 2023-10-16T09:59:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.712+0000 7f69784e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f697006ab40 0x7f69700aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.712+0000 7f69784e1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f697006e5d0 0x7f69700b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.712+0000 7f69784e1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700729b0 0x7f69700c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:51.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.712+0000 7f69784e1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700c5920 con 0x7f697006e5d0 2023-10-16T09:59:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f69784e1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700c5c00 con 0x7f69700729b0 2023-10-16T09:59:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f69784e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69700b2f40 con 0x7f697006ab40 2023-10-16T09:59:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f697006e5d0 0x7f69700b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f697006e5d0 0x7f69700b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53494/0 (socket says 172.21.15.158:53494) 2023-10-16T09:59:51.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 -- 172.21.15.158:0/276824428 learned_addr learned my addr 172.21.15.158:0/276824428 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:51.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6976a7e700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700729b0 0x7f69700c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:51.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f697627d700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f697006ab40 0x7f69700aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:51.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 -- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f697006ab40 msgr2=0x7f69700aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f697006ab40 0x7f69700aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 -- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700729b0 msgr2=0x7f69700c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700729b0 0x7f69700c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 -- 172.21.15.158:0/276824428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968009a70 con 0x7f697006e5d0 2023-10-16T09:59:51.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f697627d700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f697006ab40 0x7f69700aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:51.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6975a7c700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f697006e5d0 0x7f69700b0e40 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f696c00c500 tx=0x7f696c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:51.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6966ffd700 1 -- 172.21.15.158:0/276824428 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c0052d0 con 0x7f697006e5d0 2023-10-16T09:59:51.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6966ffd700 1 -- 172.21.15.158:0/276824428 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c005430 con 0x7f697006e5d0 2023-10-16T09:59:51.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69700b3220 con 0x7f697006e5d0 2023-10-16T09:59:51.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f6966ffd700 1 -- 172.21.15.158:0/276824428 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c0038c0 con 0x7f697006e5d0 2023-10-16T09:59:51.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.713+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69700af230 con 0x7f697006e5d0 2023-10-16T09:59:51.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.714+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69700b3740 con 0x7f697006e5d0 2023-10-16T09:59:51.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.715+0000 7f6966ffd700 1 -- 172.21.15.158:0/276824428 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f696c003a20 con 0x7f697006e5d0 2023-10-16T09:59:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.715+0000 7f6966ffd700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f695406b020 0x7f695406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.715+0000 7f6966ffd700 1 -- 172.21.15.158:0/276824428 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f696c088f90 con 0x7f697006e5d0 2023-10-16T09:59:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.715+0000 7f697627d700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f695406b020 0x7f695406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:51.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.716+0000 7f697627d700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f695406b020 0x7f695406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6960000f60 tx=0x7f6960009040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:51.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.720+0000 7f6966ffd700 1 -- 172.21.15.158:0/276824428 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69700b3740 con 0x7f697006e5d0 2023-10-16T09:59:51.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.913+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69700b3740 con 0x7f697006e5d0 2023-10-16T09:59:51.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.914+0000 7f6966ffd700 1 -- 172.21.15.158:0/276824428 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+14865 (secure 0 0 0) 0x7f69700b3740 con 0x7f697006e5d0 2023-10-16T09:59:51.916 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:51.916 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":90,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:49.560582+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6834/3389815719":"2023-10-17T09:59:46.169411+0000","172.21.15.172:6832/3389815719":"2023-10-17T09:59:46.169411+0000","172.21.15.158:6833/3502835696":"2023-10-17T09:59:45.202968+0000","172.21.15.158:6832/3502835696":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/400306926":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/2577742248":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/2992039003":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/23874899":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/183559494":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/1052952591":"2023-10-17T09:59:45.202968+0000","172.21.15.158:0/3100922361":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6833/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:6832/3717499045":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/3345169280":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/4171724344":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/310788915":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/2329993263":"2023-10-17T09:59:37.743702+0000","172.21.15.158:0/234410797":"2023-10-17T09:59:37.743702+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:51.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f695406b020 msgr2=0x7f695406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f695406b020 0x7f695406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6960000f60 tx=0x7f6960009040 comp rx=0 tx=0).stop 2023-10-16T09:59:51.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f697006e5d0 msgr2=0x7f69700b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:51.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f697006e5d0 0x7f69700b0e40 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f696c00c500 tx=0x7f696c00c8c0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 shutdown_connections 2023-10-16T09:59:51.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f695406b020 0x7f695406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700729b0 0x7f69700c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f697006ab40 0x7f69700aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 --2- 172.21.15.158:0/276824428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f697006e5d0 0x7f69700b0e40 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:51.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 >> 172.21.15.158:0/276824428 conn(0x7f69700d02b0 msgr2=0x7f697015d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:51.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 shutdown_connections 2023-10-16T09:59:51.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:51.916+0000 7f69784e1700 1 -- 172.21.15.158:0/276824428 wait complete. 2023-10-16T09:59:51.930 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:51.931 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:52.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.053+0000 7fc5d9124700 1 Processor -- start 2023-10-16T09:59:52.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.053+0000 7fc5d9124700 1 -- start start 2023-10-16T09:59:52.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d9124700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc5d415da40 0x7fc5d40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d9124700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d9124700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d40cbd30 con 0x7fc5d406ae10 2023-10-16T09:59:52.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d9124700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d406b2c0 con 0x7fc5d415da40 2023-10-16T09:59:52.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d9124700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40cc1c0 con 0x7fc5d40cc590 2023-10-16T09:59:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d2d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc5d415da40 0x7fc5d40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d359e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc5d406ae10 0x7fc5d40ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53678/0 (socket says 172.21.15.158:53678) 2023-10-16T09:59:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d359e700 1 -- 172.21.15.158:0/3922216253 learned_addr learned my addr 172.21.15.158:0/3922216253 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d259c700 1 --2- 172.21.15.158:0/3922216253 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d2d9d700 1 -- 172.21.15.158:0/3922216253 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 msgr2=0x7fc5d406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:52.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d2d9d700 1 --2- 172.21.15.158:0/3922216253 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d2d9d700 1 -- 172.21.15.158:0/3922216253 >> v1:172.21.15.158:6789/0 conn(0x7fc5d406ae10 legacy=0x7fc5d40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d2d9d700 1 -- 172.21.15.158:0/3922216253 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d415de20 con 0x7fc5d415da40 2023-10-16T09:59:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d1d9b700 1 -- 172.21.15.158:0/3922216253 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689636279 0 0) 0x7fc5d40cbd30 con 0x7fc5d406ae10 2023-10-16T09:59:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d259c700 1 --2- 172.21.15.158:0/3922216253 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d2d9d700 1 --2- 172.21.15.158:0/3922216253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d40caa20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009840 tx=0x7fc5c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5818f2a23b99db1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d1d9b700 1 -- 172.21.15.158:0/3922216253 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4011830 con 0x7fc5d415da40 2023-10-16T09:59:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d1d9b700 1 -- 172.21.15.158:0/3922216253 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4011990 con 0x7fc5d415da40 2023-10-16T09:59:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.054+0000 7fc5d1d9b700 1 -- 172.21.15.158:0/3922216253 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4011d20 con 0x7fc5d415da40 2023-10-16T09:59:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 -- 172.21.15.158:0/3922216253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 msgr2=0x7fc5d40caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 --2- 172.21.15.158:0/3922216253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d40caa20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009840 tx=0x7fc5c4009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 -- 172.21.15.158:0/3922216253 shutdown_connections 2023-10-16T09:59:52.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 --2- 172.21.15.158:0/3922216253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d40caa20 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 --2- 172.21.15.158:0/3922216253 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 -- 172.21.15.158:0/3922216253 >> 172.21.15.158:0/3922216253 conn(0x7fc5d4099a10 msgr2=0x7fc5d40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:52.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 -- 172.21.15.158:0/3922216253 shutdown_connections 2023-10-16T09:59:52.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.055+0000 7fc5d9124700 1 -- 172.21.15.158:0/3922216253 wait complete. 2023-10-16T09:59:52.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.056+0000 7fc5d9124700 1 Processor -- start 2023-10-16T09:59:52.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.056+0000 7fc5d9124700 1 -- start start 2023-10-16T09:59:52.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d9124700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d406ae10 0x7fc5d41e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d9124700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d40c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d9124700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d41f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d9124700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4159cc0 con 0x7fc5d415da40 2023-10-16T09:59:52.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d9124700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d41593d0 con 0x7fc5d406ae10 2023-10-16T09:59:52.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d9124700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4159670 con 0x7fc5d40cc590 2023-10-16T09:59:52.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d41f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d41f64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53508/0 (socket says 172.21.15.158:53508) 2023-10-16T09:59:52.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d359e700 1 -- 172.21.15.158:0/595762878 learned_addr learned my addr 172.21.15.158:0/595762878 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:52.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.057+0000 7fc5d2d9d700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d406ae10 0x7fc5d41e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d259c700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d40c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d359e700 1 -- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 msgr2=0x7fc5d40c6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d359e700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d40c6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d359e700 1 -- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d406ae10 msgr2=0x7fc5d41e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d359e700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d406ae10 0x7fc5d41e7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d359e700 1 -- 172.21.15.158:0/595762878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c40094a0 con 0x7fc5d415da40 2023-10-16T09:59:52.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d359e700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d41f64c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc002230 tx=0x7fc5cc00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:52.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5bb7fe700 1 -- 172.21.15.158:0/595762878 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00ab70 con 0x7fc5d415da40 2023-10-16T09:59:52.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5bb7fe700 1 -- 172.21.15.158:0/595762878 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc00acf0 con 0x7fc5d415da40 2023-10-16T09:59:52.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d4159950 con 0x7fc5d415da40 2023-10-16T09:59:52.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5bb7fe700 1 -- 172.21.15.158:0/595762878 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc007310 con 0x7fc5d415da40 2023-10-16T09:59:52.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.058+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d40c71b0 con 0x7fc5d415da40 2023-10-16T09:59:52.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.059+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d4159e70 con 0x7fc5d415da40 2023-10-16T09:59:52.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.059+0000 7fc5bb7fe700 1 -- 172.21.15.158:0/595762878 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc5cc007470 con 0x7fc5d415da40 2023-10-16T09:59:52.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.060+0000 7fc5bb7fe700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fc5c006b020 0x7fc5c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.060+0000 7fc5d2d9d700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fc5c006b020 0x7fc5c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.060+0000 7fc5bb7fe700 1 -- 172.21.15.158:0/595762878 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fc5cc0897a0 con 0x7fc5d415da40 2023-10-16T09:59:52.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.060+0000 7fc5d2d9d700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fc5c006b020 0x7fc5c006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40090b0 tx=0x7fc5c401c040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:52.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.064+0000 7fc5bb7fe700 1 -- 172.21.15.158:0/595762878 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5d4159e70 con 0x7fc5d415da40 2023-10-16T09:59:52.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.254+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5d4074a90 con 0x7fc5d415da40 2023-10-16T09:59:52.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.255+0000 7fc5bb7fe700 1 -- 172.21.15.158:0/595762878 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5cc018020 con 0x7fc5d415da40 2023-10-16T09:59:52.256 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fc5c006b020 msgr2=0x7fc5c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fc5c006b020 0x7fc5c006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40090b0 tx=0x7fc5c401c040 comp rx=0 tx=0).stop 2023-10-16T09:59:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 msgr2=0x7fc5d41f64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d41f64c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc002230 tx=0x7fc5cc00cf50 comp rx=0 tx=0).stop 2023-10-16T09:59:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 shutdown_connections 2023-10-16T09:59:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fc5c006b020 0x7fc5c006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5d415da40 0x7fc5d41f64c0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5d406ae10 0x7fc5d41e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 --2- 172.21.15.158:0/595762878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5d40cc590 0x7fc5d40c6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 >> 172.21.15.158:0/595762878 conn(0x7fc5d4099a10 msgr2=0x7fc5d40cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:52.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 shutdown_connections 2023-10-16T09:59:52.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.257+0000 7fc5d9124700 1 -- 172.21.15.158:0/595762878 wait complete. 2023-10-16T09:59:52.271 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T09:59:52.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.391+0000 7f8fe5c84700 1 Processor -- start 2023-10-16T09:59:52.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.391+0000 7f8fe5c84700 1 -- start start 2023-10-16T09:59:52.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fe5c84700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8fe006e5d0 0x7f8fe006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fe5c84700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8fe00729b0 0x7f8fe0071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fe5c84700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe0071e50 con 0x7f8fe006ab40 2023-10-16T09:59:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fe5c84700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe0161270 con 0x7f8fe00729b0 2023-10-16T09:59:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fe5c84700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe0161510 con 0x7f8fe006e5d0 2023-10-16T09:59:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8fe00729b0 0x7f8fe0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdf7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8fe006ab40 0x7f8fe00722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53688/0 (socket says 172.21.15.158:53688) 2023-10-16T09:59:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdf7fe700 1 -- 172.21.15.158:0/1212367384 learned_addr learned my addr 172.21.15.158:0/1212367384 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdffff700 1 -- 172.21.15.158:0/1212367384 >> v2:172.21.15.172:3300/0 conn(0x7f8fe006e5d0 msgr2=0x7f8fe006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdeffd700 1 --2- 172.21.15.158:0/1212367384 >> v2:172.21.15.172:3300/0 conn(0x7f8fe006e5d0 0x7f8fe006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdffff700 1 --2- 172.21.15.158:0/1212367384 >> v2:172.21.15.172:3300/0 conn(0x7f8fe006e5d0 0x7f8fe006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fde7fc700 1 -- 172.21.15.158:0/1212367384 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451370198 0 0) 0x7f8fe0071e50 con 0x7f8fe006ab40 2023-10-16T09:59:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdffff700 1 -- 172.21.15.158:0/1212367384 >> v1:172.21.15.158:6789/0 conn(0x7f8fe006ab40 legacy=0x7f8fe00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdffff700 1 -- 172.21.15.158:0/1212367384 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe01617f0 con 0x7f8fe00729b0 2023-10-16T09:59:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.392+0000 7f8fdeffd700 1 --2- 172.21.15.158:0/1212367384 >> v2:172.21.15.172:3300/0 conn(0x7f8fe006e5d0 0x7f8fe006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fdffff700 1 --2- 172.21.15.158:0/1212367384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe00729b0 0x7f8fe0071910 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00c3bf0 tx=0x7f8fd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e6a4b468f659da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fde7fc700 1 -- 172.21.15.158:0/1212367384 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd800e070 con 0x7f8fe00729b0 2023-10-16T09:59:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fde7fc700 1 -- 172.21.15.158:0/1212367384 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd800bbd0 con 0x7f8fe00729b0 2023-10-16T09:59:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fde7fc700 1 -- 172.21.15.158:0/1212367384 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8012430 con 0x7f8fe00729b0 2023-10-16T09:59:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 -- 172.21.15.158:0/1212367384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe00729b0 msgr2=0x7f8fe0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/1212367384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe00729b0 0x7f8fe0071910 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00c3bf0 tx=0x7f8fd800b330 comp rx=0 tx=0).stop 2023-10-16T09:59:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 -- 172.21.15.158:0/1212367384 shutdown_connections 2023-10-16T09:59:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/1212367384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe00729b0 0x7f8fe0071910 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/1212367384 >> v2:172.21.15.172:3300/0 conn(0x7f8fe006e5d0 0x7f8fe006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 -- 172.21.15.158:0/1212367384 >> 172.21.15.158:0/1212367384 conn(0x7f8fe00d02b0 msgr2=0x7f8fe00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 -- 172.21.15.158:0/1212367384 shutdown_connections 2023-10-16T09:59:52.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.393+0000 7f8fe5c84700 1 -- 172.21.15.158:0/1212367384 wait complete. 2023-10-16T09:59:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.394+0000 7f8fe5c84700 1 Processor -- start 2023-10-16T09:59:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.395+0000 7f8fe5c84700 1 -- start start 2023-10-16T09:59:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.395+0000 7f8fe5c84700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe006ab40 0x7f8fe00afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.395+0000 7f8fe5c84700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fe006e5d0 0x7f8fe00cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.395+0000 7f8fe5c84700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fe00729b0 0x7f8fe00b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.395+0000 7f8fe5c84700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe00b36e0 con 0x7f8fe006ab40 2023-10-16T09:59:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.395+0000 7f8fe5c84700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe00b39c0 con 0x7f8fe00729b0 2023-10-16T09:59:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.395+0000 7f8fe5c84700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe00b03b0 con 0x7f8fe006e5d0 2023-10-16T09:59:52.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe006ab40 0x7f8fe00afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe006ab40 0x7f8fe00afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53518/0 (socket says 172.21.15.158:53518) 2023-10-16T09:59:52.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 -- 172.21.15.158:0/2255854804 learned_addr learned my addr 172.21.15.158:0/2255854804 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:52.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdeffd700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fe006e5d0 0x7f8fe00cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdffff700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fe00729b0 0x7f8fe00b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 -- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fe006e5d0 msgr2=0x7f8fe00cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fe006e5d0 0x7f8fe00cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 -- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fe00729b0 msgr2=0x7f8fe00b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:52.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fe00729b0 0x7f8fe00b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:52.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 -- 172.21.15.158:0/2255854804 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd8009a70 con 0x7f8fe006ab40 2023-10-16T09:59:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdeffd700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fe006e5d0 0x7f8fe00cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdffff700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fe00729b0 0x7f8fe00b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fdf7fe700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe006ab40 0x7f8fe00afb60 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c580 tx=0x7f8fd400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fc7fff700 1 -- 172.21.15.158:0/2255854804 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4004c80 con 0x7f8fe006ab40 2023-10-16T09:59:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fc7fff700 1 -- 172.21.15.158:0/2255854804 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd4004e00 con 0x7f8fe006ab40 2023-10-16T09:59:52.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe00b0690 con 0x7f8fe006ab40 2023-10-16T09:59:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fc7fff700 1 -- 172.21.15.158:0/2255854804 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4007310 con 0x7f8fe006ab40 2023-10-16T09:59:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.396+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe00b00a0 con 0x7f8fe006ab40 2023-10-16T09:59:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.397+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe00681b0 con 0x7f8fe006ab40 2023-10-16T09:59:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.398+0000 7f8fc7fff700 1 -- 172.21.15.158:0/2255854804 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8fd4007470 con 0x7f8fe006ab40 2023-10-16T09:59:52.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.398+0000 7f8fc7fff700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8fcc06b020 0x7f8fcc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.398+0000 7f8fc7fff700 1 -- 172.21.15.158:0/2255854804 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f8fd40896d0 con 0x7f8fe006ab40 2023-10-16T09:59:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.398+0000 7f8fdeffd700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8fcc06b020 0x7f8fcc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.398+0000 7f8fdeffd700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8fcc06b020 0x7f8fcc06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0155240 tx=0x7f8fd0008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:52.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.402+0000 7f8fc7fff700 1 -- 172.21.15.158:0/2255854804 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fd40543d0 con 0x7f8fe006ab40 2023-10-16T09:59:52.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:52.674+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f8fe00b6230 con 0x7f8fe006ab40 2023-10-16T09:59:53.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.573+0000 7f8fc7fff700 1 -- 172.21.15.158:0/2255854804 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v605) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8fd40579f0 con 0x7f8fe006ab40 2023-10-16T09:59:53.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8fcc06b020 msgr2=0x7f8fcc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8fcc06b020 0x7f8fcc06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0155240 tx=0x7f8fd0008040 comp rx=0 tx=0).stop 2023-10-16T09:59:53.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe006ab40 msgr2=0x7f8fe00afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe006ab40 0x7f8fe00afb60 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c580 tx=0x7f8fd400c940 comp rx=0 tx=0).stop 2023-10-16T09:59:53.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 shutdown_connections 2023-10-16T09:59:53.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8fcc06b020 0x7f8fcc06d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fe00729b0 0x7f8fe00b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fe006ab40 0x7f8fe00afb60 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 --2- 172.21.15.158:0/2255854804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fe006e5d0 0x7f8fe00cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.576+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 >> 172.21.15.158:0/2255854804 conn(0x7f8fe00d02b0 msgr2=0x7f8fe00ae7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:53.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.577+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 shutdown_connections 2023-10-16T09:59:53.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.577+0000 7f8fe5c84700 1 -- 172.21.15.158:0/2255854804 wait complete. 2023-10-16T09:59:53.592 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T09:59:53.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.720+0000 7f4f6b732700 1 Processor -- start 2023-10-16T09:59:53.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.720+0000 7f4f6b732700 1 -- start start 2023-10-16T09:59:53.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.721+0000 7f4f6b732700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 0x7f4f64071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.721+0000 7f4f6b732700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6406fca0 con 0x7f4f640726a0 2023-10-16T09:59:53.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.721+0000 7f4f6b732700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f640ce150 con 0x7f4f640cb900 2023-10-16T09:59:53.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.721+0000 7f4f6b732700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640cdf70 con 0x7f4f6415a5b0 2023-10-16T09:59:53.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.721+0000 7f4f69ccf700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4f640726a0 0x7f4f6406f590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53702/0 (socket says 172.21.15.158:53702) 2023-10-16T09:59:53.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.721+0000 7f4f69ccf700 1 -- 172.21.15.158:0/16602695 learned_addr learned my addr 172.21.15.158:0/16602695 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:53.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115363580 0 0) 0x7f4f6406fca0 con 0x7f4f640726a0 2023-10-16T09:59:53.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f694ce700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 0x7f4f64071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f500057c0 con 0x7f4f640726a0 2023-10-16T09:59:53.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1080635424 0 0) 0x7f4f500057c0 con 0x7f4f640726a0 2023-10-16T09:59:53.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 msgr2=0x7f4f64071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:53.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 0x7f4f64071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 >> v1:172.21.15.172:6789/0 conn(0x7f4f640cb900 legacy=0x7f4f6406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640ce430 con 0x7f4f640726a0 2023-10-16T09:59:53.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4f5c0019f0 con 0x7f4f640726a0 2023-10-16T09:59:53.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 >> v1:172.21.15.158:6789/0 conn(0x7f4f640726a0 legacy=0x7f4f6406f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 0x7f4f50005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f50006180 0x7f4f50006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f50006fa0 0x7f4f50007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640cdf70 con 0x7f4f50006180 2023-10-16T09:59:53.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000dc80 con 0x7f4f50006fa0 2023-10-16T09:59:53.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000de40 con 0x7f4f6415a5b0 2023-10-16T09:59:53.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f68ccd700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f50006180 0x7f4f50006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f694ce700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 0x7f4f50005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f69ccf700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f50006fa0 0x7f4f50007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f68ccd700 1 -- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 msgr2=0x7f4f50005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:53.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f68ccd700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 0x7f4f50005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f68ccd700 1 -- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f50006fa0 msgr2=0x7f4f50007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:53.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f68ccd700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f50006fa0 0x7f4f50007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.722+0000 7f4f68ccd700 1 -- 172.21.15.158:0/16602695 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f500057c0 con 0x7f4f50006180 2023-10-16T09:59:53.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f694ce700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6415a5b0 0x7f4f50005c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:53.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f68ccd700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f50006180 0x7f4f50006a60 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f4f600018d0 tx=0x7f4f50007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e526cf751967663 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:53.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60012620 con 0x7f4f50006180 2023-10-16T09:59:53.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f60012780 con 0x7f4f50006180 2023-10-16T09:59:53.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f5bfff700 1 -- 172.21.15.158:0/16602695 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60012b30 con 0x7f4f50006180 2023-10-16T09:59:53.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f6b732700 1 -- 172.21.15.158:0/16602695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f50006180 msgr2=0x7f4f50006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f6b732700 1 --2- 172.21.15.158:0/16602695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f50006180 0x7f4f50006a60 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f4f600018d0 tx=0x7f4f50007a90 comp rx=0 tx=0).stop 2023-10-16T09:59:53.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f69ccf700 1 -- 172.21.15.158:0/16602695 reap_dead start 2023-10-16T09:59:53.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f6b732700 1 -- 172.21.15.158:0/16602695 shutdown_connections 2023-10-16T09:59:53.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.723+0000 7f4f6b732700 1 -- 172.21.15.158:0/16602695 >> 172.21.15.158:0/16602695 conn(0x7f4f640b1e90 msgr2=0x7f4f640b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:53.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.724+0000 7f4f6b732700 1 -- 172.21.15.158:0/16602695 shutdown_connections 2023-10-16T09:59:53.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.724+0000 7f4f6b732700 1 -- 172.21.15.158:0/16602695 wait complete. 2023-10-16T09:59:53.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.725+0000 7f4f6b732700 1 Processor -- start 2023-10-16T09:59:53.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.725+0000 7f4f6b732700 1 -- start start 2023-10-16T09:59:53.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f6b732700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f50006180 0x7f4f64148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f6b732700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f640c3e50 0x7f4f6415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f6b732700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f640c42d0 0x7f4f6415d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f6b732700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640b2720 con 0x7f4f640c42d0 2023-10-16T09:59:53.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f6b732700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64148d70 con 0x7f4f640c3e50 2023-10-16T09:59:53.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f6b732700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640c4df0 con 0x7f4f50006180 2023-10-16T09:59:53.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f640c42d0 0x7f4f6415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f640c42d0 0x7f4f6415d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53540/0 (socket says 172.21.15.158:53540) 2023-10-16T09:59:53.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 -- 172.21.15.158:0/2466873840 learned_addr learned my addr 172.21.15.158:0/2466873840 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:53.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f68ccd700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f640c3e50 0x7f4f6415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f694ce700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f50006180 0x7f4f64148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 -- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f50006180 msgr2=0x7f4f64148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f50006180 0x7f4f64148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 -- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f640c3e50 msgr2=0x7f4f6415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f640c3e50 0x7f4f6415cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 -- 172.21.15.158:0/2466873840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640c4830 con 0x7f4f640c42d0 2023-10-16T09:59:53.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f68ccd700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f640c3e50 0x7f4f6415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:53.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f694ce700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f50006180 0x7f4f64148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:53.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f69ccf700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f640c42d0 0x7f4f6415d260 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c002230 tx=0x7f4f5c00a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:53.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f59ffb700 1 -- 172.21.15.158:0/2466873840 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00adf0 con 0x7f4f640c42d0 2023-10-16T09:59:53.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f59ffb700 1 -- 172.21.15.158:0/2466873840 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c0133f0 con 0x7f4f640c42d0 2023-10-16T09:59:53.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f640c4b10 con 0x7f4f640c42d0 2023-10-16T09:59:53.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.726+0000 7f4f59ffb700 1 -- 172.21.15.158:0/2466873840 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0137a0 con 0x7f4f640c42d0 2023-10-16T09:59:53.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.727+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f64156f50 con 0x7f4f640c42d0 2023-10-16T09:59:53.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.727+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f64157110 con 0x7f4f640c42d0 2023-10-16T09:59:53.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.728+0000 7f4f59ffb700 1 -- 172.21.15.158:0/2466873840 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f5c013900 con 0x7f4f640c42d0 2023-10-16T09:59:53.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.728+0000 7f4f59ffb700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f4f4806b020 0x7f4f4806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:53.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.728+0000 7f4f59ffb700 1 -- 172.21.15.158:0/2466873840 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f4f5c08a160 con 0x7f4f640c42d0 2023-10-16T09:59:53.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.728+0000 7f4f694ce700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f4f4806b020 0x7f4f4806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:53.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.728+0000 7f4f694ce700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f4f4806b020 0x7f4f4806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4f640c5f60 tx=0x7f4f54008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:53.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.733+0000 7f4f59ffb700 1 -- 172.21.15.158:0/2466873840 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f5c054e60 con 0x7f4f640c42d0 2023-10-16T09:59:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.923+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4f640681b0 con 0x7f4f640c42d0 2023-10-16T09:59:53.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.927+0000 7f4f59ffb700 1 -- 172.21.15.158:0/2466873840 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4f5c058480 con 0x7f4f640c42d0 2023-10-16T09:59:53.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f4f4806b020 msgr2=0x7f4f4806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f4f4806b020 0x7f4f4806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4f640c5f60 tx=0x7f4f54008040 comp rx=0 tx=0).stop 2023-10-16T09:59:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f640c42d0 msgr2=0x7f4f6415d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f640c42d0 0x7f4f6415d260 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c002230 tx=0x7f4f5c00a640 comp rx=0 tx=0).stop 2023-10-16T09:59:53.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 shutdown_connections 2023-10-16T09:59:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f4f4806b020 0x7f4f4806d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f640c42d0 0x7f4f6415d260 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f50006180 0x7f4f64148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 --2- 172.21.15.158:0/2466873840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f640c3e50 0x7f4f6415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:53.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 >> 172.21.15.158:0/2466873840 conn(0x7f4f640b1e90 msgr2=0x7f4f640b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:53.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 shutdown_connections 2023-10-16T09:59:53.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:53.929+0000 7f4f6b732700 1 -- 172.21.15.158:0/2466873840 wait complete. 2023-10-16T09:59:53.942 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T09:59:53.943 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:53.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T09:59:54.038 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T09:59:54.038 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:54.039 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:54.056 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:59:54.056 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:54.057 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:54.076 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:54.076 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T09:59:54.077 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:54.112 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T09:59:54.113 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T09:59:54.113 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T09:59:54.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T09:59:54.135 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T09:59:54.135 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T09:59:54.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.272+0000 7ff9ad8ba700 1 Processor -- start 2023-10-16T09:59:54.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.272+0000 7ff9ad8ba700 1 -- start start 2023-10-16T09:59:54.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9ad8ba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80cdee0 0x7ff9a815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9ad8ba700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff9a80b6e10 0x7ff9a81521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9ad8ba700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff9a80ce2c0 0x7ff9a81526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9ad8ba700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a815d2c0 con 0x7ff9a80ce2c0 2023-10-16T09:59:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9ad8ba700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a8152c20 con 0x7ff9a80b6e10 2023-10-16T09:59:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9ad8ba700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a815c5d0 con 0x7ff9a80cdee0 2023-10-16T09:59:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9a77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff9a80ce2c0 0x7ff9a81526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9a77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff9a80ce2c0 0x7ff9a81526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53548/0 (socket says 172.21.15.158:53548) 2023-10-16T09:59:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.273+0000 7ff9a77fe700 1 -- 172.21.15.158:0/2372781519 learned_addr learned my addr 172.21.15.158:0/2372781519 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a67fc700 1 --2- 172.21.15.158:0/2372781519 >> v2:172.21.15.172:3300/0 conn(0x7ff9a80b6e10 0x7ff9a81521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a6ffd700 1 --2- 172.21.15.158:0/2372781519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80cdee0 0x7ff9a815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a77fe700 1 -- 172.21.15.158:0/2372781519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80cdee0 msgr2=0x7ff9a815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a77fe700 1 --2- 172.21.15.158:0/2372781519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80cdee0 0x7ff9a815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a77fe700 1 -- 172.21.15.158:0/2372781519 >> v2:172.21.15.172:3300/0 conn(0x7ff9a80b6e10 msgr2=0x7ff9a81521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a77fe700 1 --2- 172.21.15.158:0/2372781519 >> v2:172.21.15.172:3300/0 conn(0x7ff9a80b6e10 0x7ff9a81521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a77fe700 1 -- 172.21.15.158:0/2372781519 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a815c8b0 con 0x7ff9a80ce2c0 2023-10-16T09:59:54.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a67fc700 1 --2- 172.21.15.158:0/2372781519 >> v2:172.21.15.172:3300/0 conn(0x7ff9a80b6e10 0x7ff9a81521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:54.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a77fe700 1 --2- 172.21.15.158:0/2372781519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80ce2c0 0x7ff9a81526e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80cef50 tx=0x7ff9a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1e9429baa96739 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:54.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a5ffb700 1 -- 172.21.15.158:0/2372781519 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000e070 con 0x7ff9a80ce2c0 2023-10-16T09:59:54.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a5ffb700 1 -- 172.21.15.158:0/2372781519 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a000b980 con 0x7ff9a80ce2c0 2023-10-16T09:59:54.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9a5ffb700 1 -- 172.21.15.158:0/2372781519 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000bd30 con 0x7ff9a80ce2c0 2023-10-16T09:59:54.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2372781519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80ce2c0 msgr2=0x7ff9a81526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2372781519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80ce2c0 0x7ff9a81526e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80cef50 tx=0x7ff9a000b330 comp rx=0 tx=0).stop 2023-10-16T09:59:54.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2372781519 shutdown_connections 2023-10-16T09:59:54.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2372781519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80ce2c0 0x7ff9a81526e0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2372781519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80cdee0 0x7ff9a815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.274+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2372781519 >> v2:172.21.15.172:3300/0 conn(0x7ff9a80b6e10 0x7ff9a81521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.275+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2372781519 >> 172.21.15.158:0/2372781519 conn(0x7ff9a806a8d0 msgr2=0x7ff9a806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:54.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.275+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2372781519 shutdown_connections 2023-10-16T09:59:54.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.275+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2372781519 wait complete. 2023-10-16T09:59:54.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.276+0000 7ff9ad8ba700 1 Processor -- start 2023-10-16T09:59:54.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.276+0000 7ff9ad8ba700 1 -- start start 2023-10-16T09:59:54.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a80b6e10 0x7ff9a814ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80cdee0 0x7ff9a81517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80ce2c0 0x7ff9a8151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a8153020 con 0x7ff9a80cdee0 2023-10-16T09:59:54.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a814bf50 con 0x7ff9a80b6e10 2023-10-16T09:59:54.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a81508c0 con 0x7ff9a80ce2c0 2023-10-16T09:59:54.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80cdee0 0x7ff9a81517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80cdee0 0x7ff9a81517b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53552/0 (socket says 172.21.15.158:53552) 2023-10-16T09:59:54.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 -- 172.21.15.158:0/2448118622 learned_addr learned my addr 172.21.15.158:0/2448118622 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:54.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a77fe700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80ce2c0 0x7ff9a8151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a6ffd700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a80b6e10 0x7ff9a814ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 -- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80ce2c0 msgr2=0x7ff9a8151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80ce2c0 0x7ff9a8151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 -- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a80b6e10 msgr2=0x7ff9a814ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a80b6e10 0x7ff9a814ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 -- 172.21.15.158:0/2448118622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a0009b30 con 0x7ff9a80cdee0 2023-10-16T09:59:54.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9a67fc700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80cdee0 0x7ff9a81517b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff99800c7f0 tx=0x7ff99800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:54.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff98ffff700 1 -- 172.21.15.158:0/2448118622 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff998011800 con 0x7ff9a80cdee0 2023-10-16T09:59:54.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff98ffff700 1 -- 172.21.15.158:0/2448118622 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff998011960 con 0x7ff9a80cdee0 2023-10-16T09:59:54.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a8150ba0 con 0x7ff9a80cdee0 2023-10-16T09:59:54.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff98ffff700 1 -- 172.21.15.158:0/2448118622 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff998011d10 con 0x7ff9a80cdee0 2023-10-16T09:59:54.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.277+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a8151160 con 0x7ff9a80cdee0 2023-10-16T09:59:54.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.278+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a8152e80 con 0x7ff9a80cdee0 2023-10-16T09:59:54.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.279+0000 7ff98ffff700 1 -- 172.21.15.158:0/2448118622 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff998018360 con 0x7ff9a80cdee0 2023-10-16T09:59:54.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.279+0000 7ff98ffff700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff99406b020 0x7ff99406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.279+0000 7ff9a6ffd700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff99406b020 0x7ff99406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.279+0000 7ff98ffff700 1 -- 172.21.15.158:0/2448118622 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7ff998089870 con 0x7ff9a80cdee0 2023-10-16T09:59:54.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.280+0000 7ff9a6ffd700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff99406b020 0x7ff99406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff99c006b20 tx=0x7ff99c008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:54.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.284+0000 7ff98ffff700 1 -- 172.21.15.158:0/2448118622 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9a8152e80 con 0x7ff9a80cdee0 2023-10-16T09:59:54.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.518+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9a8152e80 con 0x7ff9a80cdee0 2023-10-16T09:59:54.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.518+0000 7ff98ffff700 1 -- 172.21.15.158:0/2448118622 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3709 (secure 0 0 0) 0x7ff99801fe90 con 0x7ff9a80cdee0 2023-10-16T09:59:54.520 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:54.520 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3413925509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3413925509},{"type":"v1","addr":"172.21.15.158:6835","nonce":3413925509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/655082434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":655082434},{"type":"v1","addr":"172.21.15.158:6837","nonce":655082434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2666705933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2666705933},{"type":"v1","addr":"172.21.15.172:6835","nonce":2666705933}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":35915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1864613480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1864613480},{"type":"v1","addr":"172.21.15.172:6834","nonce":1864613480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[]} 2023-10-16T09:59:54.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.520+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff99406b020 msgr2=0x7ff99406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.520+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff99406b020 0x7ff99406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff99c006b20 tx=0x7ff99c008040 comp rx=0 tx=0).stop 2023-10-16T09:59:54.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.520+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80cdee0 msgr2=0x7ff9a81517b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.520+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80cdee0 0x7ff9a81517b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff99800c7f0 tx=0x7ff99800cb00 comp rx=0 tx=0).stop 2023-10-16T09:59:54.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 shutdown_connections 2023-10-16T09:59:54.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7ff99406b020 0x7ff99406d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9a80ce2c0 0x7ff9a8151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a80b6e10 0x7ff9a814ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 --2- 172.21.15.158:0/2448118622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9a80cdee0 0x7ff9a81517b0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 >> 172.21.15.158:0/2448118622 conn(0x7ff9a806a8d0 msgr2=0x7ff9a814d740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:54.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 shutdown_connections 2023-10-16T09:59:54.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.521+0000 7ff9ad8ba700 1 -- 172.21.15.158:0/2448118622 wait complete. 2023-10-16T09:59:54.525 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 35 2023-10-16T09:59:54.534 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T09:59:54.535 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T09:59:54.535 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T09:59:54.535 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:54.535 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T09:59:54.536 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T09:59:54.536 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T09:59:54.536 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:54.537 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T09:59:54.537 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T09:59:54.537 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T09:59:54.537 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:54.538 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T09:59:54.538 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T09:59:54.538 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T09:59:54.539 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T09:59:54.543 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T09:59:54.544 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T09:59:54.545 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T09:59:54.545 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T09:59:54.548 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T09:59:54.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T09:59:54.550 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T09:59:54.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T09:59:54.586 INFO:tasks.ceph.mds.a:Started 2023-10-16T09:59:54.587 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:54.588 INFO:tasks.ceph.mds.c:Started 2023-10-16T09:59:54.588 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:54.591 INFO:tasks.ceph.mds.b:Started 2023-10-16T09:59:54.591 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:54.592 INFO:tasks.ceph.mds.d:Started 2023-10-16T09:59:54.592 DEBUG:teuthology.parallel:result is None 2023-10-16T09:59:54.593 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T09:59:54.666 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T09:59:54.666 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T09:59:54.678 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T09:59:54.678 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T09:59:54.679 INFO:tasks.ceph.mds.d.smithi172.stderr:2023-10-16T09:59:54.677+0000 7fae3e7b2700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-10-16T09:59:54.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.719+0000 7fb5fbb9b700 1 Processor -- start 2023-10-16T09:59:54.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.719+0000 7fb5fbb9b700 1 -- start start 2023-10-16T09:59:54.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5fbb9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb5f406e480 0x7fb5f40b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5fbb9b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f40af600 con 0x7fb5f415d990 2023-10-16T09:59:54.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5fbb9b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f406e9b0 con 0x7fb5f40cf650 2023-10-16T09:59:54.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5fbb9b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f40b2c90 con 0x7fb5f406e480 2023-10-16T09:59:54.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f9136700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb5f406e480 0x7fb5f40b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f9937700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb5f415d990 0x7fb5f40b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53758/0 (socket says 172.21.15.158:53758) 2023-10-16T09:59:54.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f9937700 1 -- 172.21.15.158:0/2551620558 learned_addr learned my addr 172.21.15.158:0/2551620558 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:54.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f9136700 1 -- 172.21.15.158:0/2551620558 >> v1:172.21.15.172:6789/0 conn(0x7fb5f40cf650 legacy=0x7fb5f406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f9136700 1 -- 172.21.15.158:0/2551620558 >> v1:172.21.15.158:6789/0 conn(0x7fb5f415d990 legacy=0x7fb5f40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f9136700 1 -- 172.21.15.158:0/2551620558 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f40ce890 con 0x7fb5f406e480 2023-10-16T09:59:54.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f8935700 1 -- 172.21.15.158:0/2551620558 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976829742 0 0) 0x7fb5f40af600 con 0x7fb5f415d990 2023-10-16T09:59:54.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.720+0000 7fb5f9136700 1 --2- 172.21.15.158:0/2551620558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40b2750 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009840 tx=0x7fb5e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806d2484f93eab13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:54.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5f8935700 1 -- 172.21.15.158:0/2551620558 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4011830 con 0x7fb5f406e480 2023-10-16T09:59:54.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5f8935700 1 -- 172.21.15.158:0/2551620558 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e4011990 con 0x7fb5f406e480 2023-10-16T09:59:54.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5f8935700 1 -- 172.21.15.158:0/2551620558 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4011d20 con 0x7fb5f406e480 2023-10-16T09:59:54.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2551620558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 msgr2=0x7fb5f40b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2551620558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40b2750 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009840 tx=0x7fb5e4009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:54.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2551620558 shutdown_connections 2023-10-16T09:59:54.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2551620558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40b2750 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2551620558 >> 172.21.15.158:0/2551620558 conn(0x7fb5f406b9e0 msgr2=0x7fb5f406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:54.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2551620558 shutdown_connections 2023-10-16T09:59:54.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.721+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2551620558 wait complete. 2023-10-16T09:59:54.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.723+0000 7fb5fbb9b700 1 Processor -- start 2023-10-16T09:59:54.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.723+0000 7fb5fbb9b700 1 -- start start 2023-10-16T09:59:54.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f40cf650 0x7fb5f41f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f415d990 0x7fb5f415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f4160120 con 0x7fb5f406e480 2023-10-16T09:59:54.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f41605b0 con 0x7fb5f40cf650 2023-10-16T09:59:54.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f415f510 con 0x7fb5f415d990 2023-10-16T09:59:54.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53588/0 (socket says 172.21.15.158:53588) 2023-10-16T09:59:54.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 -- 172.21.15.158:0/2772416959 learned_addr learned my addr 172.21.15.158:0/2772416959 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:54.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9136700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f40cf650 0x7fb5f41f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fa138700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f415d990 0x7fb5f415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 -- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f415d990 msgr2=0x7fb5f415dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f415d990 0x7fb5f415dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 -- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f40cf650 msgr2=0x7fb5f41f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:54.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f40cf650 0x7fb5f41f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:54.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 -- 172.21.15.158:0/2772416959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e40094a0 con 0x7fb5f406e480 2023-10-16T09:59:54.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5f9937700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40c6e40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000c520 tx=0x7fb5f000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:54.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5ea7fc700 1 -- 172.21.15.158:0/2772416959 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0004c80 con 0x7fb5f406e480 2023-10-16T09:59:54.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5ea7fc700 1 -- 172.21.15.158:0/2772416959 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f0004de0 con 0x7fb5f406e480 2023-10-16T09:59:54.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f415f7f0 con 0x7fb5f406e480 2023-10-16T09:59:54.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5ea7fc700 1 -- 172.21.15.158:0/2772416959 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0007310 con 0x7fb5f406e480 2023-10-16T09:59:54.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.724+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f415fdb0 con 0x7fb5f406e480 2023-10-16T09:59:54.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.725+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f4160400 con 0x7fb5f406e480 2023-10-16T09:59:54.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.726+0000 7fb5ea7fc700 1 -- 172.21.15.158:0/2772416959 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb5f0007470 con 0x7fb5f406e480 2023-10-16T09:59:54.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.726+0000 7fb5ea7fc700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fb5d806b020 0x7fb5d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:54.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.726+0000 7fb5ea7fc700 1 -- 172.21.15.158:0/2772416959 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fb5f0089570 con 0x7fb5f406e480 2023-10-16T09:59:54.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.726+0000 7fb5f9136700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fb5d806b020 0x7fb5d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:54.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.727+0000 7fb5f9136700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fb5d806b020 0x7fb5d806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009840 tx=0x7fb5e4008030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:54.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.731+0000 7fb5ea7fc700 1 -- 172.21.15.158:0/2772416959 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5f0054300 con 0x7fb5f406e480 2023-10-16T09:59:54.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:54.939+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb5f40681b0 con 0x7fb5f406e480 2023-10-16T09:59:55.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.585+0000 7fb5ea7fc700 1 -- 172.21.15.158:0/2772416959 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v92) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb5f0057920 con 0x7fb5f406e480 2023-10-16T09:59:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fb5d806b020 msgr2=0x7fb5d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fb5d806b020 0x7fb5d806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009840 tx=0x7fb5e4008030 comp rx=0 tx=0).stop 2023-10-16T09:59:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 msgr2=0x7fb5f40c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40c6e40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000c520 tx=0x7fb5f000c830 comp rx=0 tx=0).stop 2023-10-16T09:59:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 shutdown_connections 2023-10-16T09:59:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fb5d806b020 0x7fb5d806d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f415d990 0x7fb5f415dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f406e480 0x7fb5f40c6e40 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 --2- 172.21.15.158:0/2772416959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f40cf650 0x7fb5f41f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 >> 172.21.15.158:0/2772416959 conn(0x7fb5f406b9e0 msgr2=0x7fb5f415cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 shutdown_connections 2023-10-16T09:59:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.589+0000 7fb5fbb9b700 1 -- 172.21.15.158:0/2772416959 wait complete. 2023-10-16T09:59:55.594 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T09:59:55.605 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T09:59:55.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.726+0000 7f46ab492700 1 Processor -- start 2023-10-16T09:59:55.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.726+0000 7f46ab492700 1 -- start start 2023-10-16T09:59:55.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46ab492700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406ab40 0x7f46a406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46ab492700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46a406e5d0 0x7f46a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46ab492700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a4071e50 con 0x7f46a406af20 2023-10-16T09:59:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46ab492700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a4161270 con 0x7f46a406e5d0 2023-10-16T09:59:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46ab492700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a4161510 con 0x7f46a406ab40 2023-10-16T09:59:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46a406e5d0 0x7f46a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46a406e5d0 0x7f46a4071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53600/0 (socket says 172.21.15.158:53600) 2023-10-16T09:59:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 -- 172.21.15.158:0/4147695346 learned_addr learned my addr 172.21.15.158:0/4147695346 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a922e700 1 --2- 172.21.15.158:0/4147695346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406ab40 0x7f46a406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 -- 172.21.15.158:0/4147695346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406ab40 msgr2=0x7f46a406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 --2- 172.21.15.158:0/4147695346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406ab40 0x7f46a406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 -- 172.21.15.158:0/4147695346 >> v1:172.21.15.172:6789/0 conn(0x7f46a406af20 legacy=0x7f46a40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 -- 172.21.15.158:0/4147695346 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a41617f0 con 0x7f46a406e5d0 2023-10-16T09:59:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a922e700 1 --2- 172.21.15.158:0/4147695346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406ab40 0x7f46a406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f46a8a2d700 1 --2- 172.21.15.158:0/4147695346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406e5d0 0x7f46a4071910 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f46940098f0 tx=0x7f4694009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789dafd48e9054e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.727+0000 7f469bfff700 1 -- 172.21.15.158:0/4147695346 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694004230 con 0x7f46a406e5d0 2023-10-16T09:59:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f469bfff700 1 -- 172.21.15.158:0/4147695346 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694004390 con 0x7f46a406e5d0 2023-10-16T09:59:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f469bfff700 1 -- 172.21.15.158:0/4147695346 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46940036a0 con 0x7f46a406e5d0 2023-10-16T09:59:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 -- 172.21.15.158:0/4147695346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406e5d0 msgr2=0x7f46a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 --2- 172.21.15.158:0/4147695346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406e5d0 0x7f46a4071910 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f46940098f0 tx=0x7f4694009c00 comp rx=0 tx=0).stop 2023-10-16T09:59:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 -- 172.21.15.158:0/4147695346 shutdown_connections 2023-10-16T09:59:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 --2- 172.21.15.158:0/4147695346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406ab40 0x7f46a406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 --2- 172.21.15.158:0/4147695346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406e5d0 0x7f46a4071910 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 -- 172.21.15.158:0/4147695346 >> 172.21.15.158:0/4147695346 conn(0x7f46a40d02b0 msgr2=0x7f46a40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 -- 172.21.15.158:0/4147695346 shutdown_connections 2023-10-16T09:59:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.728+0000 7f46ab492700 1 -- 172.21.15.158:0/4147695346 wait complete. 2023-10-16T09:59:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.730+0000 7f46ab492700 1 Processor -- start 2023-10-16T09:59:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.730+0000 7f46ab492700 1 -- start start 2023-10-16T09:59:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46ab492700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406ab40 0x7f46a40cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46ab492700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46a406af20 0x7f46a41e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46ab492700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406e5d0 0x7f46a4155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46ab492700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41e7950 con 0x7f46a406ab40 2023-10-16T09:59:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46ab492700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41e7c30 con 0x7f46a406af20 2023-10-16T09:59:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46ab492700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41e7f10 con 0x7f46a406e5d0 2023-10-16T09:59:55.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406ab40 0x7f46a40cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:55.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406ab40 0x7f46a40cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53604/0 (socket says 172.21.15.158:53604) 2023-10-16T09:59:55.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 -- 172.21.15.158:0/2130158966 learned_addr learned my addr 172.21.15.158:0/2130158966 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:55.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a8a2d700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46a406af20 0x7f46a41e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:55.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a9a2f700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406e5d0 0x7f46a4155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:55.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 -- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406e5d0 msgr2=0x7f46a4155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:55.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406e5d0 0x7f46a4155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 -- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46a406af20 msgr2=0x7f46a41e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:55.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46a406af20 0x7f46a41e6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:55.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 -- 172.21.15.158:0/2130158966 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4694009550 con 0x7f46a406ab40 2023-10-16T09:59:55.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a9a2f700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406e5d0 0x7f46a4155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:55.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a8a2d700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46a406af20 0x7f46a41e6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:55.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46a922e700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406ab40 0x7f46a40cbe00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c750 tx=0x7f46a000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:55.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f4699ffb700 1 -- 172.21.15.158:0/2130158966 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0011800 con 0x7f46a406ab40 2023-10-16T09:59:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f4699ffb700 1 -- 172.21.15.158:0/2130158966 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a0011960 con 0x7f46a406ab40 2023-10-16T09:59:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a41e81f0 con 0x7f46a406ab40 2023-10-16T09:59:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.731+0000 7f4699ffb700 1 -- 172.21.15.158:0/2130158966 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0011cd0 con 0x7f46a406ab40 2023-10-16T09:59:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.732+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a40cc340 con 0x7f46a406ab40 2023-10-16T09:59:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.732+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a40b35a0 con 0x7f46a406ab40 2023-10-16T09:59:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.733+0000 7f4699ffb700 1 -- 172.21.15.158:0/2130158966 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f46a0021350 con 0x7f46a406ab40 2023-10-16T09:59:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.733+0000 7f4699ffb700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f468806b020 0x7f468806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.733+0000 7f4699ffb700 1 -- 172.21.15.158:0/2130158966 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f46a0089820 con 0x7f46a406ab40 2023-10-16T09:59:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.733+0000 7f46a8a2d700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f468806b020 0x7f468806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.734+0000 7f46a8a2d700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f468806b020 0x7f468806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46a40c3c10 tx=0x7f4694008030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:55.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:55.738+0000 7f4699ffb700 1 -- 172.21.15.158:0/2130158966 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46a40b35a0 con 0x7f46a406ab40 2023-10-16T09:59:56.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.007+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f46a40b35a0 con 0x7f46a406ab40 2023-10-16T09:59:56.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.008+0000 7f4699ffb700 1 -- 172.21.15.158:0/2130158966 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f46a40b35a0 con 0x7f46a406ab40 2023-10-16T09:59:56.010 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:56.010 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T09:59:56.010 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T09:59:56.010 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.010 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T09:59:56.011 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T09:59:56.011 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.011 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.011 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:56.011 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:56.012 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.012 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.012 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.012 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.012 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T09:59:56.013 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T09:59:56.013 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.013 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.013 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:56.013 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:56.014 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.014 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.014 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T09:59:56.015 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T09:59:56.015 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.015 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.015 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:56.015 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:56.016 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.016 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.016 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.016 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T09:59:56.016 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T09:59:56.017 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.017 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.017 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:56.017 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:56.017 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.018 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.018 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.018 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.018 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T09:59:56.018 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T09:59:56.019 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.019 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.019 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.019 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.020 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.020 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T09:59:56.020 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T09:59:56.020 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.020 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.021 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.021 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.021 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.022 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T09:59:56.022 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T09:59:56.022 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.022 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.022 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.023 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.023 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.023 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T09:59:56.024 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T09:59:56.024 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.024 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.024 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.024 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.025 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.025 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.025 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T09:59:56.025 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T09:59:56.026 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.026 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.026 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.026 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.026 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.027 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.027 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T09:59:56.027 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T09:59:56.027 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.028 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.028 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.028 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.028 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.028 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.028 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.029 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T09:59:56.029 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T09:59:56.029 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.029 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.029 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.030 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.030 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.030 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.030 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.031 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T09:59:56.031 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T09:59:56.031 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.031 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T09:59:56.031 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T09:59:56.032 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.032 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.032 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.032 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T09:59:56.033 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T09:59:56.033 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.033 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.033 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T09:59:56.033 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T09:59:56.034 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.034 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T09:59:56.034 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T09:59:56.035 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.035 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.035 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:59:56.035 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:59:56.035 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:59:56.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.036 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.036 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T09:59:56.036 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T09:59:56.037 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.037 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.037 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:59:56.037 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:59:56.037 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:59:56.037 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.038 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.038 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.038 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T09:59:56.038 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T09:59:56.038 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.039 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T09:59:56.039 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T09:59:56.039 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T09:59:56.039 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T09:59:56.039 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.040 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.040 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.040 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T09:59:56.040 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T09:59:56.040 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.041 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:59:56.041 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T09:59:56.041 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T09:59:56.041 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.042 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.042 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T09:59:56.042 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T09:59:56.042 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.042 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:59:56.043 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:59:56.044 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.044 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T09:59:56.044 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:56.044 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T09:59:56.045 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T09:59:56.045 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T09:59:56.045 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T09:59:56.045 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T09:59:56.045 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T09:59:56.045 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.046 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:56.046 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T09:59:56.046 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T09:59:56.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f468806b020 msgr2=0x7f468806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f468806b020 0x7f468806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46a40c3c10 tx=0x7f4694008030 comp rx=0 tx=0).stop 2023-10-16T09:59:56.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406ab40 msgr2=0x7f46a40cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406ab40 0x7f46a40cbe00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c750 tx=0x7f46a000cb10 comp rx=0 tx=0).stop 2023-10-16T09:59:56.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 shutdown_connections 2023-10-16T09:59:56.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f468806b020 0x7f468806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46a406e5d0 0x7f46a4155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46a406ab40 0x7f46a40cbe00 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 --2- 172.21.15.158:0/2130158966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46a406af20 0x7f46a41e6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 >> 172.21.15.158:0/2130158966 conn(0x7f46a40d02b0 msgr2=0x7f46a406d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:56.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 shutdown_connections 2023-10-16T09:59:56.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.010+0000 7f46ab492700 1 -- 172.21.15.158:0/2130158966 wait complete. 2023-10-16T09:59:56.051 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:56.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.172+0000 7f839e7ea700 1 Processor -- start 2023-10-16T09:59:56.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.172+0000 7f839e7ea700 1 -- start start 2023-10-16T09:59:56.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f839e7ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 0x7f839815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f839e7ea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83980b6e10 0x7f83981521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f839e7ea700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839815c5d0 con 0x7f83980ce2c0 2023-10-16T09:59:56.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f839e7ea700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839815d2c0 con 0x7f83980b6e10 2023-10-16T09:59:56.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f839e7ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839815c8b0 con 0x7f83980cdee0 2023-10-16T09:59:56.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f839cd87700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f83980ce2c0 0x7f83980cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53792/0 (socket says 172.21.15.158:53792) 2023-10-16T09:59:56.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f839cd87700 1 -- 172.21.15.158:0/3256387261 learned_addr learned my addr 172.21.15.158:0/3256387261 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:56.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8397fff700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 0x7f839815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620404551 0 0) 0x7f839815c5d0 con 0x7f83980ce2c0 2023-10-16T09:59:56.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f83977fe700 1 --2- 172.21.15.158:0/3256387261 >> v2:172.21.15.172:3300/0 conn(0x7f83980b6e10 0x7f83981521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83800035b0 con 0x7f83980ce2c0 2023-10-16T09:59:56.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1898644121 0 0) 0x7f83800035b0 con 0x7f83980ce2c0 2023-10-16T09:59:56.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 msgr2=0x7f839815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 0x7f839815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 >> v2:172.21.15.172:3300/0 conn(0x7f83980b6e10 msgr2=0x7f83981521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 --2- 172.21.15.158:0/3256387261 >> v2:172.21.15.172:3300/0 conn(0x7f83980b6e10 0x7f83981521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839815cb90 con 0x7f83980ce2c0 2023-10-16T09:59:56.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f83900019f0 con 0x7f83980ce2c0 2023-10-16T09:59:56.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.173+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 >> v1:172.21.15.158:6789/0 conn(0x7f83980ce2c0 legacy=0x7f83980cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 0x7f8380003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8380003f80 0x7f8380004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8380004d90 0x7f8380005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839815d2c0 con 0x7f8380004d90 2023-10-16T09:59:56.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839815c8b0 con 0x7f8380003f80 2023-10-16T09:59:56.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838000ba60 con 0x7f83980cdee0 2023-10-16T09:59:56.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f839cd87700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8380004d90 0x7f8380005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8397fff700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 0x7f8380003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f83977fe700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8380003f80 0x7f8380004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f839cd87700 1 -- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 msgr2=0x7f8380003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f839cd87700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980cdee0 0x7f8380003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f839cd87700 1 -- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8380003f80 msgr2=0x7f8380004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f839cd87700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8380003f80 0x7f8380004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f839cd87700 1 -- 172.21.15.158:0/3256387261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83800035b0 con 0x7f8380004d90 2023-10-16T09:59:56.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f839cd87700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8380004d90 0x7f8380005210 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f839000a900 tx=0x7f8380005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b1f8c25fc0cf1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839000a340 con 0x7f8380004d90 2023-10-16T09:59:56.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390012620 con 0x7f8380004d90 2023-10-16T09:59:56.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.174+0000 7f8396ffd700 1 -- 172.21.15.158:0/3256387261 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83900129d0 con 0x7f8380004d90 2023-10-16T09:59:56.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.175+0000 7f839e7ea700 1 -- 172.21.15.158:0/3256387261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8380004d90 msgr2=0x7f8380005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.175+0000 7f839e7ea700 1 --2- 172.21.15.158:0/3256387261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8380004d90 0x7f8380005210 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f839000a900 tx=0x7f8380005880 comp rx=0 tx=0).stop 2023-10-16T09:59:56.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.175+0000 7f839cd87700 1 -- 172.21.15.158:0/3256387261 reap_dead start 2023-10-16T09:59:56.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.175+0000 7f839e7ea700 1 -- 172.21.15.158:0/3256387261 shutdown_connections 2023-10-16T09:59:56.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.175+0000 7f839e7ea700 1 -- 172.21.15.158:0/3256387261 >> 172.21.15.158:0/3256387261 conn(0x7f839806a8d0 msgr2=0x7f839806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:56.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.175+0000 7f839e7ea700 1 -- 172.21.15.158:0/3256387261 shutdown_connections 2023-10-16T09:59:56.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.175+0000 7f839e7ea700 1 -- 172.21.15.158:0/3256387261 wait complete. 2023-10-16T09:59:56.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.176+0000 7f839e7ea700 1 Processor -- start 2023-10-16T09:59:56.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.176+0000 7f839e7ea700 1 -- start start 2023-10-16T09:59:56.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f839e7ea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f839814fea0 0x7f83980b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f839e7ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980b5460 0x7f83981f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f839e7ea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83981f3810 0x7f839806cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f839e7ea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83981f4d00 con 0x7f839814fea0 2023-10-16T09:59:56.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f839e7ea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83981f4060 con 0x7f83981f3810 2023-10-16T09:59:56.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f839e7ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83980b58c0 con 0x7f83980b5460 2023-10-16T09:59:56.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f8397fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f839814fea0 0x7f83980b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f8397fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f839814fea0 0x7f83980b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53624/0 (socket says 172.21.15.158:53624) 2023-10-16T09:59:56.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f8397fff700 1 -- 172.21.15.158:0/489729820 learned_addr learned my addr 172.21.15.158:0/489729820 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:56.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f83977fe700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980b5460 0x7f83981f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f8397fff700 1 -- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980b5460 msgr2=0x7f83981f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f839cd87700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83981f3810 0x7f839806cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f8397fff700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980b5460 0x7f83981f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.177+0000 7f8397fff700 1 -- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83981f3810 msgr2=0x7f839806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f8397fff700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83981f3810 0x7f839806cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f8397fff700 1 -- 172.21.15.158:0/489729820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8398150280 con 0x7f839814fea0 2023-10-16T09:59:56.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f83977fe700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980b5460 0x7f83981f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T09:59:56.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f8397fff700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f839814fea0 0x7f83980b4f20 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f838c011ad0 tx=0x7f838c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f8394ff9700 1 -- 172.21.15.158:0/489729820 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c00fa00 con 0x7f839814fea0 2023-10-16T09:59:56.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f8394ff9700 1 -- 172.21.15.158:0/489729820 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c00fb60 con 0x7f839814fea0 2023-10-16T09:59:56.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f8394ff9700 1 -- 172.21.15.158:0/489729820 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c014440 con 0x7f839814fea0 2023-10-16T09:59:56.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83981f4360 con 0x7f839814fea0 2023-10-16T09:59:56.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.178+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83981f4920 con 0x7f839814fea0 2023-10-16T09:59:56.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.179+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839815cec0 con 0x7f839814fea0 2023-10-16T09:59:56.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.179+0000 7f8394ff9700 1 -- 172.21.15.158:0/489729820 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f838c0145a0 con 0x7f839814fea0 2023-10-16T09:59:56.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.180+0000 7f8394ff9700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f838406b050 0x7f838406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.180+0000 7f8394ff9700 1 -- 172.21.15.158:0/489729820 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f838c089e10 con 0x7f839814fea0 2023-10-16T09:59:56.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.180+0000 7f83977fe700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f838406b050 0x7f838406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.180+0000 7f83977fe700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f838406b050 0x7f838406d510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8388009b50 tx=0x7f8388008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.184+0000 7f8394ff9700 1 -- 172.21.15.158:0/489729820 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f839815cec0 con 0x7f839814fea0 2023-10-16T09:59:56.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.383+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f839815cec0 con 0x7f839814fea0 2023-10-16T09:59:56.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.383+0000 7f8394ff9700 1 -- 172.21.15.158:0/489729820 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13720 (secure 0 0 0) 0x7f839815cec0 con 0x7f839814fea0 2023-10-16T09:59:56.385 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:56.385 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":94,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:55.593161+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:56.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f838406b050 msgr2=0x7f838406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f838406b050 0x7f838406d510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8388009b50 tx=0x7f8388008040 comp rx=0 tx=0).stop 2023-10-16T09:59:56.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f839814fea0 msgr2=0x7f83980b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f839814fea0 0x7f83980b4f20 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f838c011ad0 tx=0x7f838c011e90 comp rx=0 tx=0).stop 2023-10-16T09:59:56.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 shutdown_connections 2023-10-16T09:59:56.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f838406b050 0x7f838406d510 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83981f3810 0x7f839806cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f839814fea0 0x7f83980b4f20 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 --2- 172.21.15.158:0/489729820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83980b5460 0x7f83981f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 >> 172.21.15.158:0/489729820 conn(0x7f839806a8d0 msgr2=0x7f839815d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:56.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 shutdown_connections 2023-10-16T09:59:56.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.386+0000 7f839e7ea700 1 -- 172.21.15.158:0/489729820 wait complete. 2023-10-16T09:59:56.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T09:59:56.400 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T09:59:56.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.522+0000 7f69a9e15700 1 Processor -- start 2023-10-16T09:59:56.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.522+0000 7f69a9e15700 1 -- start start 2023-10-16T09:59:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a9e15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a415dbd0 0x7f69a406df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a9e15700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f69a415dfc0 0x7f69a40718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a9e15700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69a406a8d0 0x7f69a406ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a9e15700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a40ccbb0 con 0x7f69a406a8d0 2023-10-16T09:59:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a9e15700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a406b2c0 con 0x7f69a415dfc0 2023-10-16T09:59:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a9e15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a415e510 con 0x7f69a415dbd0 2023-10-16T09:59:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69a406a8d0 0x7f69a406ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69a406a8d0 0x7f69a406ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53636/0 (socket says 172.21.15.158:53636) 2023-10-16T09:59:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 -- 172.21.15.158:0/2118430395 learned_addr learned my addr 172.21.15.158:0/2118430395 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a2ffd700 1 --2- 172.21.15.158:0/2118430395 >> v2:172.21.15.172:3300/0 conn(0x7f69a415dfc0 0x7f69a40718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a37fe700 1 --2- 172.21.15.158:0/2118430395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a415dbd0 0x7f69a406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 -- 172.21.15.158:0/2118430395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a415dbd0 msgr2=0x7f69a406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 --2- 172.21.15.158:0/2118430395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a415dbd0 0x7f69a406df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 -- 172.21.15.158:0/2118430395 >> v2:172.21.15.172:3300/0 conn(0x7f69a415dfc0 msgr2=0x7f69a40718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 --2- 172.21.15.158:0/2118430395 >> v2:172.21.15.172:3300/0 conn(0x7f69a415dfc0 0x7f69a40718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a3fff700 1 -- 172.21.15.158:0/2118430395 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a415e7f0 con 0x7f69a406a8d0 2023-10-16T09:59:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.523+0000 7f69a2ffd700 1 --2- 172.21.15.158:0/2118430395 >> v2:172.21.15.172:3300/0 conn(0x7f69a415dfc0 0x7f69a40718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a3fff700 1 --2- 172.21.15.158:0/2118430395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a406a8d0 0x7f69a406ad80 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f69a40cdf50 tx=0x7f699c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75b9cf601d09e17 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a27fc700 1 -- 172.21.15.158:0/2118430395 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c00e070 con 0x7f69a406a8d0 2023-10-16T09:59:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a27fc700 1 -- 172.21.15.158:0/2118430395 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c00b9c0 con 0x7f69a406a8d0 2023-10-16T09:59:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a27fc700 1 -- 172.21.15.158:0/2118430395 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c00bd70 con 0x7f69a406a8d0 2023-10-16T09:59:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 -- 172.21.15.158:0/2118430395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a406a8d0 msgr2=0x7f69a406ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 --2- 172.21.15.158:0/2118430395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a406a8d0 0x7f69a406ad80 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f69a40cdf50 tx=0x7f699c00b330 comp rx=0 tx=0).stop 2023-10-16T09:59:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 -- 172.21.15.158:0/2118430395 shutdown_connections 2023-10-16T09:59:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 --2- 172.21.15.158:0/2118430395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a406a8d0 0x7f69a406ad80 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 --2- 172.21.15.158:0/2118430395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a415dbd0 0x7f69a406df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 --2- 172.21.15.158:0/2118430395 >> v2:172.21.15.172:3300/0 conn(0x7f69a415dfc0 0x7f69a40718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 -- 172.21.15.158:0/2118430395 >> 172.21.15.158:0/2118430395 conn(0x7f69a40b1f00 msgr2=0x7f69a40b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:56.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 -- 172.21.15.158:0/2118430395 shutdown_connections 2023-10-16T09:59:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.524+0000 7f69a9e15700 1 -- 172.21.15.158:0/2118430395 wait complete. 2023-10-16T09:59:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.525+0000 7f69a9e15700 1 Processor -- start 2023-10-16T09:59:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.526+0000 7f69a9e15700 1 -- start start 2023-10-16T09:59:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.526+0000 7f69a9e15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a406a8d0 0x7f69a414dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.526+0000 7f69a9e15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a415dfc0 0x7f69a41603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.526+0000 7f69a9e15700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4160910 0x7f69a40c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.526+0000 7f69a9e15700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a4160dc0 con 0x7f69a415dfc0 2023-10-16T09:59:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.526+0000 7f69a9e15700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a415f090 con 0x7f69a4160910 2023-10-16T09:59:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.526+0000 7f69a9e15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a415f370 con 0x7f69a406a8d0 2023-10-16T09:59:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a415dfc0 0x7f69a41603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a415dfc0 0x7f69a41603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53638/0 (socket says 172.21.15.158:53638) 2023-10-16T09:59:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 -- 172.21.15.158:0/3544455833 learned_addr learned my addr 172.21.15.158:0/3544455833 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a3fff700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4160910 0x7f69a40c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a37fe700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a406a8d0 0x7f69a414dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 -- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a406a8d0 msgr2=0x7f69a414dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a406a8d0 0x7f69a414dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 -- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4160910 msgr2=0x7f69a40c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4160910 0x7f69a40c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 -- 172.21.15.158:0/3544455833 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c009b30 con 0x7f69a415dfc0 2023-10-16T09:59:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a2ffd700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a415dfc0 0x7f69a41603d0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f699400e810 tx=0x7f699400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f698bfff700 1 -- 172.21.15.158:0/3544455833 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994013610 con 0x7f69a415dfc0 2023-10-16T09:59:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f698bfff700 1 -- 172.21.15.158:0/3544455833 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6994013770 con 0x7f69a415dfc0 2023-10-16T09:59:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a415f650 con 0x7f69a415dfc0 2023-10-16T09:59:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f698bfff700 1 -- 172.21.15.158:0/3544455833 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994013b20 con 0x7f69a415dfc0 2023-10-16T09:59:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.527+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a415fc10 con 0x7f69a415dfc0 2023-10-16T09:59:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.528+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a415fe30 con 0x7f69a415dfc0 2023-10-16T09:59:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.529+0000 7f698bfff700 1 -- 172.21.15.158:0/3544455833 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6994013c80 con 0x7f69a415dfc0 2023-10-16T09:59:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.529+0000 7f698bfff700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f699006b020 0x7f699006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.529+0000 7f698bfff700 1 -- 172.21.15.158:0/3544455833 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f6994089500 con 0x7f69a415dfc0 2023-10-16T09:59:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.529+0000 7f69a37fe700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f699006b020 0x7f699006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.529+0000 7f69a37fe700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f699006b020 0x7f699006d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f699800c900 tx=0x7f6998008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.533+0000 7f698bfff700 1 -- 172.21.15.158:0/3544455833 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69940543f0 con 0x7f69a415dfc0 2023-10-16T09:59:56.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.730+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69a40681b0 con 0x7f69a415dfc0 2023-10-16T09:59:56.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.730+0000 7f698bfff700 1 -- 172.21.15.158:0/3544455833 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6994057a10 con 0x7f69a415dfc0 2023-10-16T09:59:56.732 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T09:59:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f699006b020 msgr2=0x7f699006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f699006b020 0x7f699006d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f699800c900 tx=0x7f6998008040 comp rx=0 tx=0).stop 2023-10-16T09:59:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a415dfc0 msgr2=0x7f69a41603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a415dfc0 0x7f69a41603d0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f699400e810 tx=0x7f699400ebd0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 shutdown_connections 2023-10-16T09:59:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f699006b020 0x7f699006d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4160910 0x7f69a40c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a406a8d0 0x7f69a414dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 --2- 172.21.15.158:0/3544455833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a415dfc0 0x7f69a41603d0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.732+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 >> 172.21.15.158:0/3544455833 conn(0x7f69a40b1f00 msgr2=0x7f69a41614a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.733+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 shutdown_connections 2023-10-16T09:59:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.733+0000 7f69a9e15700 1 -- 172.21.15.158:0/3544455833 wait complete. 2023-10-16T09:59:56.746 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T09:59:56.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.866+0000 7f03142f2700 1 Processor -- start 2023-10-16T09:59:56.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.866+0000 7f03142f2700 1 -- start start 2023-10-16T09:59:56.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f03142f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 0x7f030c06f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f03142f2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f030c0cae30 0x7f030c0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f03142f2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c15d800 con 0x7f030c0ce640 2023-10-16T09:59:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f03142f2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c15e480 con 0x7f030c0cae30 2023-10-16T09:59:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f03142f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c15dae0 con 0x7f030c0c65b0 2023-10-16T09:59:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031208e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f030c0ce640 0x7f030c0cea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53824/0 (socket says 172.21.15.158:53824) 2023-10-16T09:59:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031208e700 1 -- 172.21.15.158:0/2201307557 learned_addr learned my addr 172.21.15.158:0/2201307557 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031288f700 1 --2- 172.21.15.158:0/2201307557 >> v2:172.21.15.172:3300/0 conn(0x7f030c0cae30 0x7f030c0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031188d700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 0x7f030c06f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873742670 0 0) 0x7f030c15d800 con 0x7f030c0ce640 2023-10-16T09:59:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f80035b0 con 0x7f030c0ce640 2023-10-16T09:59:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779475967 0 0) 0x7f02f80035b0 con 0x7f030c0ce640 2023-10-16T09:59:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 msgr2=0x7f030c06f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 0x7f030c06f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 >> v2:172.21.15.172:3300/0 conn(0x7f030c0cae30 msgr2=0x7f030c0cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 --2- 172.21.15.158:0/2201307557 >> v2:172.21.15.172:3300/0 conn(0x7f030c0cae30 0x7f030c0cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c15ddc0 con 0x7f030c0ce640 2023-10-16T09:59:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0300001670 con 0x7f030c0ce640 2023-10-16T09:59:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 >> v1:172.21.15.158:6789/0 conn(0x7f030c0ce640 legacy=0x7f030c0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 0x7f02f8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f8003fb0 0x7f02f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f8004d90 0x7f02f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c15e480 con 0x7f02f8004d90 2023-10-16T09:59:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c15dae0 con 0x7f02f8003fb0 2023-10-16T09:59:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f800ba10 con 0x7f030c0c65b0 2023-10-16T09:59:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.867+0000 7f031288f700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f8004d90 0x7f02f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031188d700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f8003fb0 0x7f02f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031208e700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 0x7f02f8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031288f700 1 -- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 msgr2=0x7f02f8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031288f700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f030c0c65b0 0x7f02f8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031288f700 1 -- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f8003fb0 msgr2=0x7f02f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031288f700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f8003fb0 0x7f02f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031288f700 1 -- 172.21.15.158:0/2201307557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f80035b0 con 0x7f02f8004d90 2023-10-16T09:59:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031288f700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f8004d90 0x7f02f8005210 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0304009fc0 tx=0x7f02f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bb77943e63a59a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030400ec30 con 0x7f02f8004d90 2023-10-16T09:59:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030400f040 con 0x7f02f8004d90 2023-10-16T09:59:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.868+0000 7f031108c700 1 -- 172.21.15.158:0/2201307557 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304014480 con 0x7f02f8004d90 2023-10-16T09:59:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.869+0000 7f03142f2700 1 -- 172.21.15.158:0/2201307557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f8004d90 msgr2=0x7f02f8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.869+0000 7f03142f2700 1 --2- 172.21.15.158:0/2201307557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f8004d90 0x7f02f8005210 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0304009fc0 tx=0x7f02f8005880 comp rx=0 tx=0).stop 2023-10-16T09:59:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.869+0000 7f031288f700 1 -- 172.21.15.158:0/2201307557 reap_dead start 2023-10-16T09:59:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.869+0000 7f03142f2700 1 -- 172.21.15.158:0/2201307557 shutdown_connections 2023-10-16T09:59:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.869+0000 7f03142f2700 1 -- 172.21.15.158:0/2201307557 >> 172.21.15.158:0/2201307557 conn(0x7f030c0cc4b0 msgr2=0x7f030c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.869+0000 7f03142f2700 1 -- 172.21.15.158:0/2201307557 shutdown_connections 2023-10-16T09:59:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.869+0000 7f03142f2700 1 -- 172.21.15.158:0/2201307557 wait complete. 2023-10-16T09:59:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.870+0000 7f03142f2700 1 Processor -- start 2023-10-16T09:59:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.870+0000 7f03142f2700 1 -- start start 2023-10-16T09:59:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8004d90 0x7f030c0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f030c15c9e0 0x7f030c151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f030c06b700 0x7f030c06bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c06c0f0 con 0x7f030c15c9e0 2023-10-16T09:59:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0c7680 con 0x7f030c06b700 2023-10-16T09:59:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c15ce60 con 0x7f02f8004d90 2023-10-16T09:59:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f030c15c9e0 0x7f030c151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f030c15c9e0 0x7f030c151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53662/0 (socket says 172.21.15.158:53662) 2023-10-16T09:59:56.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 -- 172.21.15.158:0/1376145770 learned_addr learned my addr 172.21.15.158:0/1376145770 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:56.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031208e700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8004d90 0x7f030c0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031288f700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f030c06b700 0x7f030c06bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 -- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8004d90 msgr2=0x7f030c0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8004d90 0x7f030c0b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 -- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f030c06b700 msgr2=0x7f030c06bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f030c06b700 0x7f030c06bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 -- 172.21.15.158:0/1376145770 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c15c630 con 0x7f030c15c9e0 2023-10-16T09:59:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031208e700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8004d90 0x7f030c0b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031288f700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f030c06b700 0x7f030c06bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f031188d700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f030c15c9e0 0x7f030c151240 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f0308011960 tx=0x7f0308011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f02feffd700 1 -- 172.21.15.158:0/1376145770 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030800f940 con 0x7f030c15c9e0 2023-10-16T09:59:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f02feffd700 1 -- 172.21.15.158:0/1376145770 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030800faa0 con 0x7f030c15c9e0 2023-10-16T09:59:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c151780 con 0x7f030c15c9e0 2023-10-16T09:59:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f02feffd700 1 -- 172.21.15.158:0/1376145770 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030800fe30 con 0x7f030c15c9e0 2023-10-16T09:59:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.871+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c0b2cd0 con 0x7f030c15c9e0 2023-10-16T09:59:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.872+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c0681b0 con 0x7f030c15c9e0 2023-10-16T09:59:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.873+0000 7f02feffd700 1 -- 172.21.15.158:0/1376145770 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0308014440 con 0x7f030c15c9e0 2023-10-16T09:59:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.873+0000 7f02feffd700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f02f006b020 0x7f02f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.873+0000 7f02feffd700 1 -- 172.21.15.158:0/1376145770 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f0308089e40 con 0x7f030c15c9e0 2023-10-16T09:59:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.873+0000 7f031208e700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f02f006b020 0x7f02f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.874+0000 7f031208e700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f02f006b020 0x7f02f006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0300001050 tx=0x7f0300009040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:56.878+0000 7f02feffd700 1 -- 172.21.15.158:0/1376145770 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0308054ef0 con 0x7f030c15c9e0 2023-10-16T09:59:57.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.073+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f030c0c7450 con 0x7f030c15c9e0 2023-10-16T09:59:57.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.074+0000 7f02feffd700 1 -- 172.21.15.158:0/1376145770 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0308058510 con 0x7f030c15c9e0 2023-10-16T09:59:57.075 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:57.075 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T09:59:57.076 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T09:59:57.076 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T09:59:57.076 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T09:59:57.076 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T09:59:57.076 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T09:59:57.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f02f006b020 msgr2=0x7f02f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f02f006b020 0x7f02f006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0300001050 tx=0x7f0300009040 comp rx=0 tx=0).stop 2023-10-16T09:59:57.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f030c15c9e0 msgr2=0x7f030c151240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f030c15c9e0 0x7f030c151240 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f0308011960 tx=0x7f0308011d20 comp rx=0 tx=0).stop 2023-10-16T09:59:57.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 shutdown_connections 2023-10-16T09:59:57.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f02f006b020 0x7f02f006d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f030c06b700 0x7f030c06bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8004d90 0x7f030c0b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 --2- 172.21.15.158:0/1376145770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f030c15c9e0 0x7f030c151240 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 >> 172.21.15.158:0/1376145770 conn(0x7f030c0cc4b0 msgr2=0x7f030c073c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:57.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 shutdown_connections 2023-10-16T09:59:57.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.076+0000 7f03142f2700 1 -- 172.21.15.158:0/1376145770 wait complete. 2023-10-16T09:59:57.089 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T09:59:57.090 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T09:59:57.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.213+0000 7f095e54e700 1 Processor -- start 2023-10-16T09:59:57.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.213+0000 7f095e54e700 1 -- start start 2023-10-16T09:59:57.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f095e54e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 0x7f095806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f095e54e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095806c210 con 0x7f0958144840 2023-10-16T09:59:57.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f095e54e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0958141d00 con 0x7f09580cde10 2023-10-16T09:59:57.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f095e54e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095806b970 con 0x7f0958141920 2023-10-16T09:59:57.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f09577fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0958144840 0x7f09580cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53844/0 (socket says 172.21.15.158:53844) 2023-10-16T09:59:57.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f09577fe700 1 -- 172.21.15.158:0/1110698596 learned_addr learned my addr 172.21.15.158:0/1110698596 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:57.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0957fff700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 0x7f095806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395641716 0 0) 0x7f095806c210 con 0x7f0958144840 2023-10-16T09:59:57.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09400035b0 con 0x7f0958144840 2023-10-16T09:59:57.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 202025096 0 0) 0x7f09400035b0 con 0x7f0958144840 2023-10-16T09:59:57.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 msgr2=0x7f095806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:57.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 0x7f095806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 >> v1:172.21.15.172:6789/0 conn(0x7f09580cde10 legacy=0x7f095806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095806bc50 con 0x7f0958144840 2023-10-16T09:59:57.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0948001670 con 0x7f0958144840 2023-10-16T09:59:57.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 >> v1:172.21.15.158:6789/0 conn(0x7f0958144840 legacy=0x7f09580cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 0x7f0940003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0940003fb0 0x7f0940004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0940004d90 0x7f0940005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095806b970 con 0x7f0940004d90 2023-10-16T09:59:57.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094000bba0 con 0x7f0940003fb0 2023-10-16T09:59:57.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.214+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094000bd60 con 0x7f0958141920 2023-10-16T09:59:57.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f095caeb700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0940004d90 0x7f0940005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f0957fff700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 0x7f0940003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f09577fe700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0940003fb0 0x7f0940004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f095caeb700 1 -- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 msgr2=0x7f0940003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:57.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f095caeb700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0958141920 0x7f0940003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f095caeb700 1 -- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0940003fb0 msgr2=0x7f0940004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:57.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f095caeb700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0940003fb0 0x7f0940004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f095caeb700 1 -- 172.21.15.158:0/1110698596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09400035b0 con 0x7f0940004d90 2023-10-16T09:59:57.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f09577fe700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0940003fb0 0x7f0940004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T09:59:57.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f095caeb700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0940004d90 0x7f0940005210 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0958066650 tx=0x7f0940005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102bd6151b38f34f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:57.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950001db0 con 0x7f0940004d90 2023-10-16T09:59:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095000cc70 con 0x7f0940004d90 2023-10-16T09:59:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.215+0000 7f0956ffd700 1 -- 172.21.15.158:0/1110698596 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950003a00 con 0x7f0940004d90 2023-10-16T09:59:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.216+0000 7f095e54e700 1 -- 172.21.15.158:0/1110698596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0940004d90 msgr2=0x7f0940005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.216+0000 7f095e54e700 1 --2- 172.21.15.158:0/1110698596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0940004d90 0x7f0940005210 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0958066650 tx=0x7f0940005880 comp rx=0 tx=0).stop 2023-10-16T09:59:57.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.216+0000 7f095caeb700 1 -- 172.21.15.158:0/1110698596 reap_dead start 2023-10-16T09:59:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.216+0000 7f095e54e700 1 -- 172.21.15.158:0/1110698596 shutdown_connections 2023-10-16T09:59:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.216+0000 7f095e54e700 1 -- 172.21.15.158:0/1110698596 >> 172.21.15.158:0/1110698596 conn(0x7f09580cc5b0 msgr2=0x7f09580cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.216+0000 7f095e54e700 1 -- 172.21.15.158:0/1110698596 shutdown_connections 2023-10-16T09:59:57.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.216+0000 7f095e54e700 1 -- 172.21.15.158:0/1110698596 wait complete. 2023-10-16T09:59:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.217+0000 7f095e54e700 1 Processor -- start 2023-10-16T09:59:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.217+0000 7f095e54e700 1 -- start start 2023-10-16T09:59:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f095e54e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f095815ae70 0x7f0958143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f095e54e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09580b12f0 0x7f0958143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f095e54e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09581f0640 0x7f09581f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f095e54e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580c6fc0 con 0x7f09580b12f0 2023-10-16T09:59:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f095e54e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580c72a0 con 0x7f09581f0640 2023-10-16T09:59:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f095e54e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09580c7580 con 0x7f095815ae70 2023-10-16T09:59:57.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09580b12f0 0x7f0958143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09580b12f0 0x7f0958143b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53682/0 (socket says 172.21.15.158:53682) 2023-10-16T09:59:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 -- 172.21.15.158:0/1925263927 learned_addr learned my addr 172.21.15.158:0/1925263927 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f095caeb700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09581f0640 0x7f09581f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f0957fff700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f095815ae70 0x7f0958143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 -- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f095815ae70 msgr2=0x7f0958143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f095815ae70 0x7f0958143640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 -- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09581f0640 msgr2=0x7f09581f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09581f0640 0x7f09581f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 -- 172.21.15.158:0/1925263927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09580c7860 con 0x7f09580b12f0 2023-10-16T09:59:57.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f09577fe700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09580b12f0 0x7f0958143b80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f094800eb20 tx=0x7f094800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:57.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.218+0000 7f0954ff9700 1 -- 172.21.15.158:0/1925263927 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948005e10 con 0x7f09580b12f0 2023-10-16T09:59:57.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.219+0000 7f0954ff9700 1 -- 172.21.15.158:0/1925263927 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094800c7e0 con 0x7f09580b12f0 2023-10-16T09:59:57.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.219+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09580c7b40 con 0x7f09580b12f0 2023-10-16T09:59:57.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.219+0000 7f0954ff9700 1 -- 172.21.15.158:0/1925263927 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800cb50 con 0x7f09580b12f0 2023-10-16T09:59:57.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.219+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09580c6c00 con 0x7f09580b12f0 2023-10-16T09:59:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.220+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09580c7d30 con 0x7f09580b12f0 2023-10-16T09:59:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.220+0000 7f0954ff9700 1 -- 172.21.15.158:0/1925263927 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f094800ccb0 con 0x7f09580b12f0 2023-10-16T09:59:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.220+0000 7f0954ff9700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f094406b020 0x7f094406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.220+0000 7f0954ff9700 1 -- 172.21.15.158:0/1925263927 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f0948089470 con 0x7f09580b12f0 2023-10-16T09:59:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.220+0000 7f0957fff700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f094406b020 0x7f094406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.221+0000 7f0957fff700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f094406b020 0x7f094406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f094c009c60 tx=0x7f094c009210 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:57.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.225+0000 7f0954ff9700 1 -- 172.21.15.158:0/1925263927 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0948054500 con 0x7f09580b12f0 2023-10-16T09:59:57.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.418+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f09580681b0 con 0x7f09580b12f0 2023-10-16T09:59:57.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.594+0000 7f0954ff9700 1 -- 172.21.15.158:0/1925263927 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0948057b20 con 0x7f09580b12f0 2023-10-16T09:59:57.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f094406b020 msgr2=0x7f094406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f094406b020 0x7f094406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f094c009c60 tx=0x7f094c009210 comp rx=0 tx=0).stop 2023-10-16T09:59:57.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09580b12f0 msgr2=0x7f0958143b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09580b12f0 0x7f0958143b80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f094800eb20 tx=0x7f094800ee30 comp rx=0 tx=0).stop 2023-10-16T09:59:57.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 shutdown_connections 2023-10-16T09:59:57.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f094406b020 0x7f094406d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09581f0640 0x7f09581f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f095815ae70 0x7f0958143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 --2- 172.21.15.158:0/1925263927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09580b12f0 0x7f0958143b80 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 >> 172.21.15.158:0/1925263927 conn(0x7f09580cc5b0 msgr2=0x7f09580af690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:57.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 shutdown_connections 2023-10-16T09:59:57.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.597+0000 7f095e54e700 1 -- 172.21.15.158:0/1925263927 wait complete. 2023-10-16T09:59:57.601 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T09:59:57.614 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T09:59:57.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.742+0000 7f8b38940700 1 Processor -- start 2023-10-16T09:59:57.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.742+0000 7f8b38940700 1 -- start start 2023-10-16T09:59:57.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b38940700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b340af5f0 0x7f8b34161e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b38940700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b34071d70 con 0x7f8b3406a8d0 2023-10-16T09:59:57.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b38940700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340af9d0 con 0x7f8b340b3b60 2023-10-16T09:59:57.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b38940700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b340b3970 con 0x7f8b340af5f0 2023-10-16T09:59:57.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b3259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b340af5f0 0x7f8b34161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b32d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8b3406a8d0 0x7f8b3406de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53866/0 (socket says 172.21.15.158:53866) 2023-10-16T09:59:57.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b32d9d700 1 -- 172.21.15.158:0/2185797959 learned_addr learned my addr 172.21.15.158:0/2185797959 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T09:59:57.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b3259c700 1 -- 172.21.15.158:0/2185797959 >> v1:172.21.15.172:6789/0 conn(0x7f8b340b3b60 legacy=0x7f8b340724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:57.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b3259c700 1 -- 172.21.15.158:0/2185797959 >> v1:172.21.15.158:6789/0 conn(0x7f8b3406a8d0 legacy=0x7f8b3406de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b3159a700 1 -- 172.21.15.158:0/2185797959 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248764457 0 0) 0x7f8b34071d70 con 0x7f8b3406a8d0 2023-10-16T09:59:57.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.743+0000 7f8b3259c700 1 -- 172.21.15.158:0/2185797959 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b34162370 con 0x7f8b340af5f0 2023-10-16T09:59:57.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b3259c700 1 --2- 172.21.15.158:0/2185797959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b34161e30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009840 tx=0x7f8b28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47267106bcf44ef5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:57.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b3159a700 1 -- 172.21.15.158:0/2185797959 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28004230 con 0x7f8b340af5f0 2023-10-16T09:59:57.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b3159a700 1 -- 172.21.15.158:0/2185797959 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28004390 con 0x7f8b340af5f0 2023-10-16T09:59:57.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b3159a700 1 -- 172.21.15.158:0/2185797959 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b280036a0 con 0x7f8b340af5f0 2023-10-16T09:59:57.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b38940700 1 -- 172.21.15.158:0/2185797959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 msgr2=0x7f8b34161e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b38940700 1 --2- 172.21.15.158:0/2185797959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b34161e30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009840 tx=0x7f8b28009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:57.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b38940700 1 -- 172.21.15.158:0/2185797959 shutdown_connections 2023-10-16T09:59:57.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b38940700 1 --2- 172.21.15.158:0/2185797959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b34161e30 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b38940700 1 -- 172.21.15.158:0/2185797959 >> 172.21.15.158:0/2185797959 conn(0x7f8b340ccd30 msgr2=0x7f8b340cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:57.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b38940700 1 -- 172.21.15.158:0/2185797959 shutdown_connections 2023-10-16T09:59:57.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.744+0000 7f8b38940700 1 -- 172.21.15.158:0/2185797959 wait complete. 2023-10-16T09:59:57.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 Processor -- start 2023-10-16T09:59:57.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 -- start start 2023-10-16T09:59:57.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3406a8d0 0x7f8b340b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b340c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b340b3b60 0x7f8b34158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341f2f80 con 0x7f8b340af5f0 2023-10-16T09:59:57.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340ca3e0 con 0x7f8b340b3b60 2023-10-16T09:59:57.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.746+0000 7f8b38940700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340b1520 con 0x7f8b3406a8d0 2023-10-16T09:59:57.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b340c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b340c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53694/0 (socket says 172.21.15.158:53694) 2023-10-16T09:59:57.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 -- 172.21.15.158:0/3237090667 learned_addr learned my addr 172.21.15.158:0/3237090667 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:57.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 -- 172.21.15.158:0/3237090667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3406a8d0 msgr2=0x7f8b340b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b32d9d700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b340b3b60 0x7f8b34158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3406a8d0 0x7f8b340b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 -- 172.21.15.158:0/3237090667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b340b3b60 msgr2=0x7f8b34158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:57.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b340b3b60 0x7f8b34158840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:57.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 -- 172.21.15.158:0/3237090667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b280094a0 con 0x7f8b340af5f0 2023-10-16T09:59:57.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b31d9b700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b340c9ea0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400c4a0 tx=0x7f8b2400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:57.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b22ffd700 1 -- 172.21.15.158:0/3237090667 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240054e0 con 0x7f8b340af5f0 2023-10-16T09:59:57.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b22ffd700 1 -- 172.21.15.158:0/3237090667 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b24003680 con 0x7f8b340af5f0 2023-10-16T09:59:57.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b22ffd700 1 -- 172.21.15.158:0/3237090667 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24003a30 con 0x7f8b340af5f0 2023-10-16T09:59:57.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b340b0f60 con 0x7f8b340af5f0 2023-10-16T09:59:57.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.747+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3415c6c0 con 0x7f8b340af5f0 2023-10-16T09:59:57.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.748+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b340681b0 con 0x7f8b340af5f0 2023-10-16T09:59:57.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.749+0000 7f8b22ffd700 1 -- 172.21.15.158:0/3237090667 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b24003b90 con 0x7f8b340af5f0 2023-10-16T09:59:57.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.749+0000 7f8b22ffd700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8b1406b020 0x7f8b1406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:57.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.749+0000 7f8b22ffd700 1 -- 172.21.15.158:0/3237090667 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5971+0+0 (secure 0 0 0) 0x7f8b24088850 con 0x7f8b340af5f0 2023-10-16T09:59:57.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.749+0000 7f8b3259c700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8b1406b020 0x7f8b1406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:57.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.750+0000 7f8b3259c700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8b1406b020 0x7f8b1406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8b28000c00 tx=0x7f8b28010000 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:57.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.754+0000 7f8b22ffd700 1 -- 172.21.15.158:0/3237090667 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b24053770 con 0x7f8b340af5f0 2023-10-16T09:59:57.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:57.946+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8b34071fb0 con 0x7f8b340af5f0 2023-10-16T09:59:58.442 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:59:58.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T09:59:58.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.602+0000 7f8b22ffd700 1 -- 172.21.15.158:0/3237090667 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v96) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8b24056d90 con 0x7f8b340af5f0 2023-10-16T09:59:58.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.605+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8b1406b020 msgr2=0x7f8b1406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:58.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.605+0000 7f8b38940700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8b1406b020 0x7f8b1406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8b28000c00 tx=0x7f8b28010000 comp rx=0 tx=0).stop 2023-10-16T09:59:58.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.605+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 msgr2=0x7f8b340c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:58.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.605+0000 7f8b38940700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b340c9ea0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400c4a0 tx=0x7f8b2400c860 comp rx=0 tx=0).stop 2023-10-16T09:59:58.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.606+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 shutdown_connections 2023-10-16T09:59:58.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.606+0000 7f8b38940700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8b1406b020 0x7f8b1406d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.606+0000 7f8b38940700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b340b3b60 0x7f8b34158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.606+0000 7f8b38940700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3406a8d0 0x7f8b340b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.606+0000 7f8b38940700 1 --2- 172.21.15.158:0/3237090667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b340af5f0 0x7f8b340c9ea0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.606+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 >> 172.21.15.158:0/3237090667 conn(0x7f8b340ccd30 msgr2=0x7f8b340724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:58.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.607+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 shutdown_connections 2023-10-16T09:59:58.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.607+0000 7f8b38940700 1 -- 172.21.15.158:0/3237090667 wait complete. 2023-10-16T09:59:58.619 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T09:59:58.632 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T09:59:58.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.756+0000 7f5fe1858700 1 Processor -- start 2023-10-16T09:59:58.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.756+0000 7f5fe1858700 1 -- start start 2023-10-16T09:59:58.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.756+0000 7f5fe1858700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fdc15b4a0 0x7f5fdc0cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:58.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.756+0000 7f5fe1858700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:58.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.756+0000 7f5fe1858700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc0cfc90 con 0x7f5fdc0706c0 2023-10-16T09:59:58.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.756+0000 7f5fe1858700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc0b2510 con 0x7f5fdc15b4a0 2023-10-16T09:59:58.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.756+0000 7f5fe1858700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc15f5c0 con 0x7f5fdc15f050 2023-10-16T09:59:58.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fda7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:58.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdaffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fdc15b4a0 0x7f5fdc0cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:58.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fda7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:36306/0 (socket says 172.21.15.158:36306) 2023-10-16T09:59:58.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdaffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fdc15b4a0 0x7f5fdc0cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:54882/0 (socket says 172.21.15.158:54882) 2023-10-16T09:59:58.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fda7fc700 1 -- 172.21.15.158:0/1849678517 learned_addr learned my addr 172.21.15.158:0/1849678517 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:58.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdb7fe700 1 --1- 172.21.15.158:0/1849678517 >> v1:172.21.15.172:6789/0 conn(0x7f5fdc0706c0 0x7f5fdc0c9e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:57786/0 (socket says 172.21.15.158:57786) 2023-10-16T09:59:58.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fd9ffb700 1 -- 172.21.15.158:0/1849678517 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442186148 0 0) 0x7f5fdc0cfc90 con 0x7f5fdc0706c0 2023-10-16T09:59:58.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fd9ffb700 1 -- 172.21.15.158:0/1849678517 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc40035b0 con 0x7f5fdc0706c0 2023-10-16T09:59:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdaffd700 1 -- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 msgr2=0x7f5fdc0b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdaffd700 1 --2- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdaffd700 1 -- 172.21.15.158:0/1849678517 >> v1:172.21.15.172:6789/0 conn(0x7f5fdc0706c0 legacy=0x7f5fdc0c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdaffd700 1 -- 172.21.15.158:0/1849678517 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc15f8a0 con 0x7f5fdc15b4a0 2023-10-16T09:59:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fda7fc700 1 --2- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T09:59:58.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fdaffd700 1 --2- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 0x7f5fdc0cb2d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009840 tx=0x7f5fcc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b8a0eed7b8d6ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:58.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.757+0000 7f5fd9ffb700 1 -- 172.21.15.158:0/1849678517 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc011830 con 0x7f5fdc15b4a0 2023-10-16T09:59:58.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fd9ffb700 1 -- 172.21.15.158:0/1849678517 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc011990 con 0x7f5fdc15b4a0 2023-10-16T09:59:58.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fd9ffb700 1 -- 172.21.15.158:0/1849678517 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc011d00 con 0x7f5fdc15b4a0 2023-10-16T09:59:58.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 -- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 msgr2=0x7f5fdc0cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 --2- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 0x7f5fdc0cb2d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009840 tx=0x7f5fcc009b50 comp rx=0 tx=0).stop 2023-10-16T09:59:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 -- 172.21.15.158:0/1849678517 shutdown_connections 2023-10-16T09:59:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 --2- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 0x7f5fdc0cb2d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 --2- 172.21.15.158:0/1849678517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 -- 172.21.15.158:0/1849678517 >> 172.21.15.158:0/1849678517 conn(0x7f5fdc06af00 msgr2=0x7f5fdc0cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:58.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 -- 172.21.15.158:0/1849678517 shutdown_connections 2023-10-16T09:59:58.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.758+0000 7f5fe1858700 1 -- 172.21.15.158:0/1849678517 wait complete. 2023-10-16T09:59:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.760+0000 7f5fe1858700 1 Processor -- start 2023-10-16T09:59:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.760+0000 7f5fe1858700 1 -- start start 2023-10-16T09:59:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0706c0 0x7f5fdc0c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 0x7f5fdc14dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0c8350 con 0x7f5fdc0706c0 2023-10-16T09:59:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc14e290 con 0x7f5fdc15b4a0 2023-10-16T09:59:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0c72b0 con 0x7f5fdc15f050 2023-10-16T09:59:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0706c0 0x7f5fdc0c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0706c0 0x7f5fdc0c6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53708/0 (socket says 172.21.15.158:53708) 2023-10-16T09:59:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 -- 172.21.15.158:0/609467888 learned_addr learned my addr 172.21.15.158:0/609467888 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fda7fc700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 0x7f5fdc14dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdb7fe700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 -- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 msgr2=0x7f5fdc0c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 -- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 msgr2=0x7f5fdc14dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:58.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 0x7f5fdc14dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:58.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 -- 172.21.15.158:0/609467888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0009550 con 0x7f5fdc0706c0 2023-10-16T09:59:58.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fdaffd700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0706c0 0x7f5fdc0c6bd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0090b0 tx=0x7f5fcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:58.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fc3fff700 1 -- 172.21.15.158:0/609467888 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0044d0 con 0x7f5fdc0706c0 2023-10-16T09:59:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fc3fff700 1 -- 172.21.15.158:0/609467888 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc01e070 con 0x7f5fdc0706c0 2023-10-16T09:59:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc0094a0 con 0x7f5fdc0706c0 2023-10-16T09:59:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fc3fff700 1 -- 172.21.15.158:0/609467888 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0038e0 con 0x7f5fdc0706c0 2023-10-16T09:59:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.761+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc0c7870 con 0x7f5fdc0706c0 2023-10-16T09:59:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.762+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc0681b0 con 0x7f5fdc0706c0 2023-10-16T09:59:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.763+0000 7f5fc3fff700 1 -- 172.21.15.158:0/609467888 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5fcc003a40 con 0x7f5fdc0706c0 2023-10-16T09:59:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.763+0000 7f5fc3fff700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5fc806b020 0x7f5fc806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.763+0000 7f5fc3fff700 1 -- 172.21.15.158:0/609467888 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f5fcc08a820 con 0x7f5fdc0706c0 2023-10-16T09:59:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.763+0000 7f5fda7fc700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5fc806b020 0x7f5fc806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.764+0000 7f5fda7fc700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5fc806b020 0x7f5fc806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0c24c0 tx=0x7f5fd00092c0 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:58.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:58.768+0000 7f5fc3fff700 1 -- 172.21.15.158:0/609467888 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5fcc055540 con 0x7f5fdc0706c0 2023-10-16T09:59:59.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.004+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5fdc0c9170 con 0x7f5fdc0706c0 2023-10-16T09:59:59.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.620+0000 7f5fc3fff700 1 -- 172.21.15.158:0/609467888 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5fcc058b60 con 0x7f5fdc0706c0 2023-10-16T09:59:59.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5fc806b020 msgr2=0x7f5fc806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5fc806b020 0x7f5fc806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0c24c0 tx=0x7f5fd00092c0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0706c0 msgr2=0x7f5fdc0c6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0706c0 0x7f5fdc0c6bd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0090b0 tx=0x7f5fcc004750 comp rx=0 tx=0).stop 2023-10-16T09:59:59.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 shutdown_connections 2023-10-16T09:59:59.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5fc806b020 0x7f5fc806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fdc15f050 0x7f5fdc0c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0706c0 0x7f5fdc0c6bd0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 --2- 172.21.15.158:0/609467888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fdc15b4a0 0x7f5fdc14dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 >> 172.21.15.158:0/609467888 conn(0x7f5fdc06af00 msgr2=0x7f5fdc0b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:59.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 shutdown_connections 2023-10-16T09:59:59.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.633+0000 7f5fe1858700 1 -- 172.21.15.158:0/609467888 wait complete. 2023-10-16T09:59:59.637 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 11 and data pool 12 2023-10-16T09:59:59.653 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T09:59:59.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.776+0000 7f6476839700 1 Processor -- start 2023-10-16T09:59:59.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6476839700 1 -- start start 2023-10-16T09:59:59.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6476839700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64700c46f0 0x7f6470166100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:59.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6476839700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f647006b9e0 0x7f647006be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:59.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6476839700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6470166640 con 0x7f64700cf020 2023-10-16T09:59:59.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6476839700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6470162320 con 0x7f647006b9e0 2023-10-16T09:59:59.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6476839700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700cf520 con 0x7f64700c46f0 2023-10-16T09:59:59.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6474dd6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f647006b9e0 0x7f647006be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:59.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6474dd6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f647006b9e0 0x7f647006be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53734/0 (socket says 172.21.15.158:53734) 2023-10-16T09:59:59.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.777+0000 7f6474dd6700 1 -- 172.21.15.158:0/39669487 learned_addr learned my addr 172.21.15.158:0/39669487 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:59.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f646ffff700 1 --2- 172.21.15.158:0/39669487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64700c46f0 0x7f6470166100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:59.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f6474dd6700 1 -- 172.21.15.158:0/39669487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64700c46f0 msgr2=0x7f6470166100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T09:59:59.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f6474dd6700 1 --2- 172.21.15.158:0/39669487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64700c46f0 0x7f6470166100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f6474dd6700 1 -- 172.21.15.158:0/39669487 >> v1:172.21.15.172:6789/0 conn(0x7f64700cf020 legacy=0x7f6470162ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f6474dd6700 1 -- 172.21.15.158:0/39669487 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700cf800 con 0x7f647006b9e0 2023-10-16T09:59:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f6474dd6700 1 --2- 172.21.15.158:0/39669487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647006b9e0 0x7f647006be30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6468009e50 tx=0x7f646800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdb7f1b1f7f97be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f646effd700 1 -- 172.21.15.158:0/39669487 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800e070 con 0x7f647006b9e0 2023-10-16T09:59:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f646effd700 1 -- 172.21.15.158:0/39669487 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646800b9c0 con 0x7f647006b9e0 2023-10-16T09:59:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.778+0000 7f646effd700 1 -- 172.21.15.158:0/39669487 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800bd70 con 0x7f647006b9e0 2023-10-16T09:59:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 -- 172.21.15.158:0/39669487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647006b9e0 msgr2=0x7f647006be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 --2- 172.21.15.158:0/39669487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647006b9e0 0x7f647006be30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6468009e50 tx=0x7f646800b330 comp rx=0 tx=0).stop 2023-10-16T09:59:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 -- 172.21.15.158:0/39669487 shutdown_connections 2023-10-16T09:59:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 --2- 172.21.15.158:0/39669487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647006b9e0 0x7f647006be30 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 --2- 172.21.15.158:0/39669487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64700c46f0 0x7f6470166100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 -- 172.21.15.158:0/39669487 >> 172.21.15.158:0/39669487 conn(0x7f647006a8d0 msgr2=0x7f647006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 -- 172.21.15.158:0/39669487 shutdown_connections 2023-10-16T09:59:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.779+0000 7f6476839700 1 -- 172.21.15.158:0/39669487 wait complete. 2023-10-16T09:59:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.780+0000 7f6476839700 1 Processor -- start 2023-10-16T09:59:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.780+0000 7f6476839700 1 -- start start 2023-10-16T09:59:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f647006b9e0 0x7f64701608f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:59.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64700c46f0 0x7f64700cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:59.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700cf020 0x7f64700cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:59.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700b6fe0 con 0x7f64700cf020 2023-10-16T09:59:59.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700b5f40 con 0x7f64700c46f0 2023-10-16T09:59:59.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700b6220 con 0x7f647006b9e0 2023-10-16T09:59:59.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700cf020 0x7f64700cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:59.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700cf020 0x7f64700cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53740/0 (socket says 172.21.15.158:53740) 2023-10-16T09:59:59.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 -- 172.21.15.158:0/4235513677 learned_addr learned my addr 172.21.15.158:0/4235513677 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T09:59:59.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f646f7fe700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64700c46f0 0x7f64700cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:59.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f646ffff700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f647006b9e0 0x7f64701608f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:59.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 -- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f647006b9e0 msgr2=0x7f64701608f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f647006b9e0 0x7f64701608f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 -- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64700c46f0 msgr2=0x7f64700cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64700c46f0 0x7f64700cac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 -- 172.21.15.158:0/4235513677 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468009b30 con 0x7f64700cf020 2023-10-16T09:59:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6474dd6700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700cf020 0x7f64700cb140 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f64700ceaf0 tx=0x7f6468004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f646cff9700 1 -- 172.21.15.158:0/4235513677 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468004800 con 0x7f64700cf020 2023-10-16T09:59:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f646cff9700 1 -- 172.21.15.158:0/4235513677 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468004960 con 0x7f64700cf020 2023-10-16T09:59:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64700b6500 con 0x7f64700cf020 2023-10-16T09:59:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f646cff9700 1 -- 172.21.15.158:0/4235513677 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646801c680 con 0x7f64700cf020 2023-10-16T09:59:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.781+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64700b6ac0 con 0x7f64700cf020 2023-10-16T09:59:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.782+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64700b6ce0 con 0x7f64700cf020 2023-10-16T09:59:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.783+0000 7f646cff9700 1 -- 172.21.15.158:0/4235513677 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f646801c7e0 con 0x7f64700cf020 2023-10-16T09:59:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.783+0000 7f646cff9700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f645006b020 0x7f645006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T09:59:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.783+0000 7f646cff9700 1 -- 172.21.15.158:0/4235513677 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f646808b750 con 0x7f64700cf020 2023-10-16T09:59:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.783+0000 7f646ffff700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f645006b020 0x7f645006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T09:59:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.784+0000 7f646ffff700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f645006b020 0x7f645006d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64700c4200 tx=0x7f6460008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T09:59:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.788+0000 7f646cff9700 1 -- 172.21.15.158:0/4235513677 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64680562e0 con 0x7f64700cf020 2023-10-16T09:59:59.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.981+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64700681b0 con 0x7f64700cf020 2023-10-16T09:59:59.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.982+0000 7f646cff9700 1 -- 172.21.15.158:0/4235513677 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+17447 (secure 0 0 0) 0x7f6468059900 con 0x7f64700cf020 2023-10-16T09:59:59.983 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T09:59:59.984 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":97,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:59.006156+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:57.438311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:57.966439+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T09:59:59.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f645006b020 msgr2=0x7f645006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f645006b020 0x7f645006d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64700c4200 tx=0x7f6460008040 comp rx=0 tx=0).stop 2023-10-16T09:59:59.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700cf020 msgr2=0x7f64700cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T09:59:59.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700cf020 0x7f64700cb140 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f64700ceaf0 tx=0x7f6468004ad0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 shutdown_connections 2023-10-16T09:59:59.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f645006b020 0x7f645006d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700cf020 0x7f64700cb140 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f647006b9e0 0x7f64701608f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 --2- 172.21.15.158:0/4235513677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64700c46f0 0x7f64700cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T09:59:59.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 >> 172.21.15.158:0/4235513677 conn(0x7f647006a8d0 msgr2=0x7f647006bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T09:59:59.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 shutdown_connections 2023-10-16T09:59:59.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T09:59:59.984+0000 7f6476839700 1 -- 172.21.15.158:0/4235513677 wait complete. 2023-10-16T09:59:59.998 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:00.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.119+0000 7f23b8796700 1 Processor -- start 2023-10-16T10:00:00.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.119+0000 7f23b8796700 1 -- start start 2023-10-16T10:00:00.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.120+0000 7f23b8796700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b00cc5d0 0x7f23b006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.120+0000 7f23b8796700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f23b015dfb0 0x7f23b00cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.120+0000 7f23b8796700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b006e770 con 0x7f23b006f340 2023-10-16T10:00:00.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.120+0000 7f23b8796700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b00cf270 con 0x7f23b015dfb0 2023-10-16T10:00:00.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.120+0000 7f23b8796700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b015e4e0 con 0x7f23b00cc5d0 2023-10-16T10:00:00.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f23b015dfb0 0x7f23b00cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f23b015dfb0 0x7f23b00cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53744/0 (socket says 172.21.15.158:53744) 2023-10-16T10:00:00.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b6d33700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f23b006f340 0x7f23b0071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53924/0 (socket says 172.21.15.158:53924) 2023-10-16T10:00:00.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 -- 172.21.15.158:0/3404922340 learned_addr learned my addr 172.21.15.158:0/3404922340 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:00.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 -- 172.21.15.158:0/3404922340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b00cc5d0 msgr2=0x7f23b006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:00.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b6532700 1 --2- 172.21.15.158:0/3404922340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b00cc5d0 0x7f23b006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 --2- 172.21.15.158:0/3404922340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b00cc5d0 0x7f23b006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5530700 1 -- 172.21.15.158:0/3404922340 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423711851 0 0) 0x7f23a8001490 con 0x7f23b006f340 2023-10-16T10:00:00.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 -- 172.21.15.158:0/3404922340 >> v1:172.21.15.158:6789/0 conn(0x7f23b006f340 legacy=0x7f23b0071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 -- 172.21.15.158:0/3404922340 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b015e7c0 con 0x7f23b015dfb0 2023-10-16T10:00:00.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b6532700 1 --2- 172.21.15.158:0/3404922340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b00cc5d0 0x7f23b006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:00.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5d31700 1 --2- 172.21.15.158:0/3404922340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b015dfb0 0x7f23b00cb6f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0098f0 tx=0x7f23ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ca22343df1d753 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:00.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5530700 1 -- 172.21.15.158:0/3404922340 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac00b8a0 con 0x7f23b015dfb0 2023-10-16T10:00:00.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5530700 1 -- 172.21.15.158:0/3404922340 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac00ba00 con 0x7f23b015dfb0 2023-10-16T10:00:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.121+0000 7f23b5530700 1 -- 172.21.15.158:0/3404922340 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac00bd90 con 0x7f23b015dfb0 2023-10-16T10:00:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 -- 172.21.15.158:0/3404922340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b015dfb0 msgr2=0x7f23b00cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 --2- 172.21.15.158:0/3404922340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b015dfb0 0x7f23b00cb6f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0098f0 tx=0x7f23ac009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 -- 172.21.15.158:0/3404922340 shutdown_connections 2023-10-16T10:00:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 --2- 172.21.15.158:0/3404922340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b00cc5d0 0x7f23b006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 --2- 172.21.15.158:0/3404922340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b015dfb0 0x7f23b00cb6f0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 -- 172.21.15.158:0/3404922340 >> 172.21.15.158:0/3404922340 conn(0x7f23b00d02b0 msgr2=0x7f23b00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 -- 172.21.15.158:0/3404922340 shutdown_connections 2023-10-16T10:00:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.122+0000 7f23b8796700 1 -- 172.21.15.158:0/3404922340 wait complete. 2023-10-16T10:00:00.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.123+0000 7f23b8796700 1 Processor -- start 2023-10-16T10:00:00.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.123+0000 7f23b8796700 1 -- start start 2023-10-16T10:00:00.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b8796700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b006f340 0x7f23b00c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b8796700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23b00cc5d0 0x7f23b00b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b8796700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b015dfb0 0x7f23b01558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b8796700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00b5d40 con 0x7f23b006f340 2023-10-16T10:00:00.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b8796700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0155de0 con 0x7f23b00cc5d0 2023-10-16T10:00:00.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b8796700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00c5910 con 0x7f23b015dfb0 2023-10-16T10:00:00.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b006f340 0x7f23b00c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b006f340 0x7f23b00c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53750/0 (socket says 172.21.15.158:53750) 2023-10-16T10:00:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 -- 172.21.15.158:0/1048348578 learned_addr learned my addr 172.21.15.158:0/1048348578 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6d33700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b015dfb0 0x7f23b01558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b5d31700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23b00cc5d0 0x7f23b00b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 -- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b015dfb0 msgr2=0x7f23b01558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b015dfb0 0x7f23b01558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 -- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23b00cc5d0 msgr2=0x7f23b00b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23b00cc5d0 0x7f23b00b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 -- 172.21.15.158:0/1048348578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac009550 con 0x7f23b006f340 2023-10-16T10:00:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23b6532700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b006f340 0x7f23b00c5250 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f23a000c750 tx=0x7f23a000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.124+0000 7f23a6ffd700 1 -- 172.21.15.158:0/1048348578 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0011800 con 0x7f23b006f340 2023-10-16T10:00:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.125+0000 7f23a6ffd700 1 -- 172.21.15.158:0/1048348578 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a0011960 con 0x7f23b006f340 2023-10-16T10:00:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.125+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b00c5bf0 con 0x7f23b006f340 2023-10-16T10:00:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.125+0000 7f23a6ffd700 1 -- 172.21.15.158:0/1048348578 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0011cd0 con 0x7f23b006f340 2023-10-16T10:00:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.125+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b00c61b0 con 0x7f23b006f340 2023-10-16T10:00:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.126+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b00681b0 con 0x7f23b006f340 2023-10-16T10:00:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.126+0000 7f23a6ffd700 1 -- 172.21.15.158:0/1048348578 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f23a0023480 con 0x7f23b006f340 2023-10-16T10:00:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.127+0000 7f23a6ffd700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f239406b020 0x7f239406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.127+0000 7f23a6ffd700 1 -- 172.21.15.158:0/1048348578 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f23a0089e90 con 0x7f23b006f340 2023-10-16T10:00:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.127+0000 7f23b5d31700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f239406b020 0x7f239406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.127+0000 7f23b5d31700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f239406b020 0x7f239406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f23b015e410 tx=0x7f23ac008030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:00.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.131+0000 7f23a6ffd700 1 -- 172.21.15.158:0/1048348578 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f23a0054a10 con 0x7f23b006f340 2023-10-16T10:00:00.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.326+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23b00b6230 con 0x7f23b006f340 2023-10-16T10:00:00.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.327+0000 7f23a6ffd700 1 -- 172.21.15.158:0/1048348578 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+17447 (secure 0 0 0) 0x7f23a001e090 con 0x7f23b006f340 2023-10-16T10:00:00.328 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:00.329 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":97,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T09:59:59.006156+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:57.438311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:57.966439+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:00.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f239406b020 msgr2=0x7f239406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f239406b020 0x7f239406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f23b015e410 tx=0x7f23ac008030 comp rx=0 tx=0).stop 2023-10-16T10:00:00.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b006f340 msgr2=0x7f23b00c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b006f340 0x7f23b00c5250 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f23a000c750 tx=0x7f23a000ca60 comp rx=0 tx=0).stop 2023-10-16T10:00:00.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 shutdown_connections 2023-10-16T10:00:00.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f239406b020 0x7f239406d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23b015dfb0 0x7f23b01558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23b006f340 0x7f23b00c5250 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 --2- 172.21.15.158:0/1048348578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23b00cc5d0 0x7f23b00b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 >> 172.21.15.158:0/1048348578 conn(0x7f23b00d02b0 msgr2=0x7f23b01586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:00.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 shutdown_connections 2023-10-16T10:00:00.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.329+0000 7f23b8796700 1 -- 172.21.15.158:0/1048348578 wait complete. 2023-10-16T10:00:00.343 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T10:00:00.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.465+0000 7f8f8d18b700 1 Processor -- start 2023-10-16T10:00:00.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.465+0000 7f8f8d18b700 1 -- start start 2023-10-16T10:00:00.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8d18b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8f8806afe0 0x7f8f880caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8d18b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f880cc590 0x7f8f880afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8d18b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f880cbd30 con 0x7f8f8815da40 2023-10-16T10:00:00.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8d18b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f880cc1c0 con 0x7f8f880cc590 2023-10-16T10:00:00.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8d18b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8815de20 con 0x7f8f8806afe0 2023-10-16T10:00:00.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8759e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f880cc590 0x7f8f880afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8759e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f880cc590 0x7f8f880afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53762/0 (socket says 172.21.15.158:53762) 2023-10-16T10:00:00.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.466+0000 7f8f8759e700 1 -- 172.21.15.158:0/2330545894 learned_addr learned my addr 172.21.15.158:0/2330545894 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:00.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f8659c700 1 --2- 172.21.15.158:0/2330545894 >> v2:172.21.15.172:3300/0 conn(0x7f8f8806afe0 0x7f8f880caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f8759e700 1 -- 172.21.15.158:0/2330545894 >> v2:172.21.15.172:3300/0 conn(0x7f8f8806afe0 msgr2=0x7f8f880caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:00.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f8759e700 1 --2- 172.21.15.158:0/2330545894 >> v2:172.21.15.172:3300/0 conn(0x7f8f8806afe0 0x7f8f880caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f8759e700 1 -- 172.21.15.158:0/2330545894 >> v1:172.21.15.172:6789/0 conn(0x7f8f8815da40 legacy=0x7f8f8806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f8759e700 1 -- 172.21.15.158:0/2330545894 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f880ce260 con 0x7f8f880cc590 2023-10-16T10:00:00.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f8759e700 1 --2- 172.21.15.158:0/2330545894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f880cc590 0x7f8f880afa40 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f8f880729e0 tx=0x7f8f8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ed11997c894c0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:00.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f85d9b700 1 -- 172.21.15.158:0/2330545894 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8000e070 con 0x7f8f880cc590 2023-10-16T10:00:00.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f85d9b700 1 -- 172.21.15.158:0/2330545894 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8000b9c0 con 0x7f8f880cc590 2023-10-16T10:00:00.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.467+0000 7f8f85d9b700 1 -- 172.21.15.158:0/2330545894 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8000bd70 con 0x7f8f880cc590 2023-10-16T10:00:00.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 -- 172.21.15.158:0/2330545894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f880cc590 msgr2=0x7f8f880afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/2330545894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f880cc590 0x7f8f880afa40 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f8f880729e0 tx=0x7f8f8000b330 comp rx=0 tx=0).stop 2023-10-16T10:00:00.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 -- 172.21.15.158:0/2330545894 shutdown_connections 2023-10-16T10:00:00.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/2330545894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f880cc590 0x7f8f880afa40 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/2330545894 >> v2:172.21.15.172:3300/0 conn(0x7f8f8806afe0 0x7f8f880caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 -- 172.21.15.158:0/2330545894 >> 172.21.15.158:0/2330545894 conn(0x7f8f88099a10 msgr2=0x7f8f880d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:00.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 -- 172.21.15.158:0/2330545894 shutdown_connections 2023-10-16T10:00:00.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.468+0000 7f8f8d18b700 1 -- 172.21.15.158:0/2330545894 wait complete. 2023-10-16T10:00:00.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.469+0000 7f8f8d18b700 1 Processor -- start 2023-10-16T10:00:00.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.469+0000 7f8f8d18b700 1 -- start start 2023-10-16T10:00:00.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8d18b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8806afe0 0x7f8f8814d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8d18b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f880cc590 0x7f8f881f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8d18b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8815da40 0x7f8f8806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8d18b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8814de10 con 0x7f8f8815da40 2023-10-16T10:00:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8d18b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881f72c0 con 0x7f8f880cc590 2023-10-16T10:00:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8d18b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8815d120 con 0x7f8f8806afe0 2023-10-16T10:00:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8815da40 0x7f8f8806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8815da40 0x7f8f8806da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53768/0 (socket says 172.21.15.158:53768) 2023-10-16T10:00:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 -- 172.21.15.158:0/1701025938 learned_addr learned my addr 172.21.15.158:0/1701025938 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8659c700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f880cc590 0x7f8f881f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f86d9d700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8806afe0 0x7f8f8814d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 -- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8806afe0 msgr2=0x7f8f8814d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8806afe0 0x7f8f8814d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 -- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f880cc590 msgr2=0x7f8f881f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f880cc590 0x7f8f881f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.470+0000 7f8f8759e700 1 -- 172.21.15.158:0/1701025938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80009b30 con 0x7f8f8815da40 2023-10-16T10:00:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.471+0000 7f8f8759e700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8815da40 0x7f8f8806da60 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8f880702f0 tx=0x7f8f80004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.471+0000 7f8f6f7fe700 1 -- 172.21.15.158:0/1701025938 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f80004880 con 0x7f8f8815da40 2023-10-16T10:00:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.471+0000 7f8f6f7fe700 1 -- 172.21.15.158:0/1701025938 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f800099b0 con 0x7f8f8815da40 2023-10-16T10:00:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.471+0000 7f8f6f7fe700 1 -- 172.21.15.158:0/1701025938 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8001c680 con 0x7f8f8815da40 2023-10-16T10:00:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.471+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f880c6c50 con 0x7f8f8815da40 2023-10-16T10:00:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.471+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f880c63b0 con 0x7f8f8815da40 2023-10-16T10:00:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.472+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f880c74a0 con 0x7f8f8815da40 2023-10-16T10:00:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.472+0000 7f8f6f7fe700 1 -- 172.21.15.158:0/1701025938 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8f8001c7e0 con 0x7f8f8815da40 2023-10-16T10:00:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.473+0000 7f8f6f7fe700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8f7406b020 0x7f8f7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.473+0000 7f8f6f7fe700 1 -- 172.21.15.158:0/1701025938 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f8f8008b3e0 con 0x7f8f8815da40 2023-10-16T10:00:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.473+0000 7f8f86d9d700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8f7406b020 0x7f8f7406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.473+0000 7f8f86d9d700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8f7406b020 0x7f8f7406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c007fd0 tx=0x7f8f7c009040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.477+0000 7f8f6f7fe700 1 -- 172.21.15.158:0/1701025938 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f880c74a0 con 0x7f8f8815da40 2023-10-16T10:00:00.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:00.717+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8f880c74a0 con 0x7f8f8815da40 2023-10-16T10:00:01.590 INFO:tasks.ceph.mds.a.smithi158.stderr:2023-10-16T10:00:01.589+0000 7fe93942c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:00:01.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.633+0000 7f8f6f7fe700 1 -- 172.21.15.158:0/1701025938 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8f880c74a0 con 0x7f8f8815da40 2023-10-16T10:00:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8f7406b020 msgr2=0x7f8f7406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8f7406b020 0x7f8f7406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c007fd0 tx=0x7f8f7c009040 comp rx=0 tx=0).stop 2023-10-16T10:00:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8815da40 msgr2=0x7f8f8806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8815da40 0x7f8f8806da60 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8f880702f0 tx=0x7f8f80004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 shutdown_connections 2023-10-16T10:00:01.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f8f7406b020 0x7f8f7406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8815da40 0x7f8f8806da60 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8806afe0 0x7f8f8814d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 --2- 172.21.15.158:0/1701025938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f880cc590 0x7f8f881f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 >> 172.21.15.158:0/1701025938 conn(0x7f8f88099a10 msgr2=0x7f8f880b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:01.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 shutdown_connections 2023-10-16T10:00:01.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.636+0000 7f8f8d18b700 1 -- 172.21.15.158:0/1701025938 wait complete. 2023-10-16T10:00:01.652 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T10:00:01.653 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T10:00:01.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.782+0000 7f85c73a3700 1 Processor -- start 2023-10-16T10:00:01.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.782+0000 7f85c73a3700 1 -- start start 2023-10-16T10:00:01.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.783+0000 7f85c73a3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00ce510 0x7f85c015f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.783+0000 7f85c73a3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f85c00cbaf0 0x7f85c00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.783+0000 7f85c73a3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00cf4c0 con 0x7f85c015b590 2023-10-16T10:00:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.783+0000 7f85c73a3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c00cc4c0 con 0x7f85c00cbaf0 2023-10-16T10:00:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.783+0000 7f85c73a3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00b49e0 con 0x7f85c00ce510 2023-10-16T10:00:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.783+0000 7f85c5940700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f85c00cbaf0 0x7f85c00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.783+0000 7f85c5940700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f85c00cbaf0 0x7f85c00cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:54956/0 (socket says 172.21.15.158:54956) 2023-10-16T10:00:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c5940700 1 -- 172.21.15.158:0/534174893 learned_addr learned my addr 172.21.15.158:0/534174893 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c513f700 1 --1- 172.21.15.158:0/534174893 >> v1:172.21.15.172:6789/0 conn(0x7f85c015b590 0x7f85c00b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:57852/0 (socket says 172.21.15.158:57852) 2023-10-16T10:00:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c493e700 1 --2- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00ce510 0x7f85c015f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c5940700 1 -- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00ce510 msgr2=0x7f85c015f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:01.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c5940700 1 --2- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00ce510 0x7f85c015f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c5940700 1 -- 172.21.15.158:0/534174893 >> v1:172.21.15.172:6789/0 conn(0x7f85c015b590 legacy=0x7f85c00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:01.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c5940700 1 -- 172.21.15.158:0/534174893 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c015f710 con 0x7f85c00cbaf0 2023-10-16T10:00:01.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85c5940700 1 --2- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00cbaf0 0x7f85c00cbf80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009e80 tx=0x7f85b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9642ec4aae45145b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:01.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85b7fff700 1 -- 172.21.15.158:0/534174893 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b800e070 con 0x7f85c00cbaf0 2023-10-16T10:00:01.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85b7fff700 1 -- 172.21.15.158:0/534174893 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b800b8d0 con 0x7f85c00cbaf0 2023-10-16T10:00:01.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.784+0000 7f85b7fff700 1 -- 172.21.15.158:0/534174893 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b800bc80 con 0x7f85c00cbaf0 2023-10-16T10:00:01.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 -- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00cbaf0 msgr2=0x7f85c00cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:01.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 --2- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00cbaf0 0x7f85c00cbf80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009e80 tx=0x7f85b800b330 comp rx=0 tx=0).stop 2023-10-16T10:00:01.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 -- 172.21.15.158:0/534174893 shutdown_connections 2023-10-16T10:00:01.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 --2- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00cbaf0 0x7f85c00cbf80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 --2- 172.21.15.158:0/534174893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00ce510 0x7f85c015f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 -- 172.21.15.158:0/534174893 >> 172.21.15.158:0/534174893 conn(0x7f85c006a8d0 msgr2=0x7f85c006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:01.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 -- 172.21.15.158:0/534174893 shutdown_connections 2023-10-16T10:00:01.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.785+0000 7f85c73a3700 1 -- 172.21.15.158:0/534174893 wait complete. 2023-10-16T10:00:01.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.786+0000 7f85c73a3700 1 Processor -- start 2023-10-16T10:00:01.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.786+0000 7f85c73a3700 1 -- start start 2023-10-16T10:00:01.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c73a3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00cbaf0 0x7f85c00ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:01.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c73a3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00ce510 0x7f85c00b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:01.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c73a3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c015b590 0x7f85c00b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:01.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c73a3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00cb980 con 0x7f85c015b590 2023-10-16T10:00:01.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c73a3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c01f1930 con 0x7f85c00ce510 2023-10-16T10:00:01.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c73a3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c01f15a0 con 0x7f85c00cbaf0 2023-10-16T10:00:01.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c015b590 0x7f85c00b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:01.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c015b590 0x7f85c00b5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53782/0 (socket says 172.21.15.158:53782) 2023-10-16T10:00:01.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c513f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00cbaf0 0x7f85c00ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:01.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 -- 172.21.15.158:0/1526600423 learned_addr learned my addr 172.21.15.158:0/1526600423 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:01.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c493e700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00ce510 0x7f85c00b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:01.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 -- 172.21.15.158:0/1526600423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00cbaf0 msgr2=0x7f85c00ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:01.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00cbaf0 0x7f85c00ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 -- 172.21.15.158:0/1526600423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00ce510 msgr2=0x7f85c00b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:01.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00ce510 0x7f85c00b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:01.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 -- 172.21.15.158:0/1526600423 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b8009b30 con 0x7f85c015b590 2023-10-16T10:00:01.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85c5940700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c015b590 0x7f85c00b5d70 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f85c00d25a0 tx=0x7f85b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:01.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.787+0000 7f85b5ffb700 1 -- 172.21.15.158:0/1526600423 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8004800 con 0x7f85c015b590 2023-10-16T10:00:01.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.788+0000 7f85b5ffb700 1 -- 172.21.15.158:0/1526600423 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8004960 con 0x7f85c015b590 2023-10-16T10:00:01.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.788+0000 7f85b5ffb700 1 -- 172.21.15.158:0/1526600423 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b801c680 con 0x7f85c015b590 2023-10-16T10:00:01.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.788+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c00cb0e0 con 0x7f85c015b590 2023-10-16T10:00:01.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.788+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c00cb6a0 con 0x7f85c015b590 2023-10-16T10:00:01.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.789+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c01f1a70 con 0x7f85c015b590 2023-10-16T10:00:01.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.789+0000 7f85b5ffb700 1 -- 172.21.15.158:0/1526600423 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f85b801c7e0 con 0x7f85c015b590 2023-10-16T10:00:01.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.790+0000 7f85b5ffb700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f85a406b020 0x7f85a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:01.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.790+0000 7f85b5ffb700 1 -- 172.21.15.158:0/1526600423 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f85b808b3f0 con 0x7f85c015b590 2023-10-16T10:00:01.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.790+0000 7f85c513f700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f85a406b020 0x7f85a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:01.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.790+0000 7f85c513f700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f85a406b020 0x7f85a406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000f60 tx=0x7f85b0007ed0 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:01.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:01.794+0000 7f85b5ffb700 1 -- 172.21.15.158:0/1526600423 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85b8056020 con 0x7f85c015b590 2023-10-16T10:00:02.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.033+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f85c00681b0 con 0x7f85c015b590 2023-10-16T10:00:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.635+0000 7f85b5ffb700 1 -- 172.21.15.158:0/1526600423 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f85b8059640 con 0x7f85c015b590 2023-10-16T10:00:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f85a406b020 msgr2=0x7f85a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f85a406b020 0x7f85a406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000f60 tx=0x7f85b0007ed0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c015b590 msgr2=0x7f85c00b5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c015b590 0x7f85c00b5d70 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f85c00d25a0 tx=0x7f85b8004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 shutdown_connections 2023-10-16T10:00:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f85a406b020 0x7f85a406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c015b590 0x7f85c00b5d70 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85c00cbaf0 0x7f85c00ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 --2- 172.21.15.158:0/1526600423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85c00ce510 0x7f85c00b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 >> 172.21.15.158:0/1526600423 conn(0x7f85c006a8d0 msgr2=0x7f85c00b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:02.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 shutdown_connections 2023-10-16T10:00:02.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.638+0000 7f85c73a3700 1 -- 172.21.15.158:0/1526600423 wait complete. 2023-10-16T10:00:02.654 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T10:00:02.654 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:02.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.779+0000 7f5b465b3700 1 Processor -- start 2023-10-16T10:00:02.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.779+0000 7f5b465b3700 1 -- start start 2023-10-16T10:00:02.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b465b3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b400cc5d0 0x7f5b4006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:02.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b465b3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b4006f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:02.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b465b3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4006e770 con 0x7f5b4015dfb0 2023-10-16T10:00:02.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b465b3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b400cf270 con 0x7f5b400cc5d0 2023-10-16T10:00:02.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b465b3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4006fd10 con 0x7f5b4006f340 2023-10-16T10:00:02.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b3ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b400cc5d0 0x7f5b4006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:02.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b44b50700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b4006f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:02.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b44b50700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b4006f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:36392/0 (socket says 172.21.15.158:36392) 2023-10-16T10:00:02.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b3ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b400cc5d0 0x7f5b4006e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:54966/0 (socket says 172.21.15.158:54966) 2023-10-16T10:00:02.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b3f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5b4015dfb0 0x7f5b400cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:57866/0 (socket says 172.21.15.158:57866) 2023-10-16T10:00:02.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.780+0000 7f5b44b50700 1 -- 172.21.15.158:0/4233637349 learned_addr learned my addr 172.21.15.158:0/4233637349 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:02.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3ffff700 1 -- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 msgr2=0x7f5b4006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:02.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3ffff700 1 --2- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b4006f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3ffff700 1 -- 172.21.15.158:0/4233637349 >> v1:172.21.15.172:6789/0 conn(0x7f5b4015dfb0 legacy=0x7f5b400cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:02.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3effd700 1 -- 172.21.15.158:0/4233637349 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907284446 0 0) 0x7f5b30001110 con 0x7f5b4015dfb0 2023-10-16T10:00:02.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3ffff700 1 -- 172.21.15.158:0/4233637349 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b400cbe00 con 0x7f5b400cc5d0 2023-10-16T10:00:02.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b44b50700 1 --2- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b4006f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:02.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3ffff700 1 --2- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b4006e230 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5b34009840 tx=0x7f5b34009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35d8c953ffb0e8d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:02.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3effd700 1 -- 172.21.15.158:0/4233637349 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b34011830 con 0x7f5b400cc5d0 2023-10-16T10:00:02.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3effd700 1 -- 172.21.15.158:0/4233637349 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b34011990 con 0x7f5b400cc5d0 2023-10-16T10:00:02.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b3effd700 1 -- 172.21.15.158:0/4233637349 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b34011d20 con 0x7f5b400cc5d0 2023-10-16T10:00:02.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b465b3700 1 -- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 msgr2=0x7f5b4006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:02.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.781+0000 7f5b465b3700 1 --2- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b4006e230 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5b34009840 tx=0x7f5b34009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:02.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.782+0000 7f5b465b3700 1 -- 172.21.15.158:0/4233637349 shutdown_connections 2023-10-16T10:00:02.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.782+0000 7f5b465b3700 1 --2- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b4006f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.782+0000 7f5b465b3700 1 --2- 172.21.15.158:0/4233637349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b4006e230 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.782+0000 7f5b465b3700 1 -- 172.21.15.158:0/4233637349 >> 172.21.15.158:0/4233637349 conn(0x7f5b400d02b0 msgr2=0x7f5b400d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:02.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.782+0000 7f5b465b3700 1 -- 172.21.15.158:0/4233637349 shutdown_connections 2023-10-16T10:00:02.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.782+0000 7f5b465b3700 1 -- 172.21.15.158:0/4233637349 wait complete. 2023-10-16T10:00:02.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.783+0000 7f5b465b3700 1 Processor -- start 2023-10-16T10:00:02.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.783+0000 7f5b465b3700 1 -- start start 2023-10-16T10:00:02.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b465b3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b400c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:02.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b465b3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b400c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:02.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b465b3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b4015dfb0 0x7f5b400c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:02.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b465b3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b40153020 con 0x7f5b4015dfb0 2023-10-16T10:00:02.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b465b3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b400b2f40 con 0x7f5b400cc5d0 2023-10-16T10:00:02.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b465b3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b400b31e0 con 0x7f5b4006f340 2023-10-16T10:00:02.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b4015dfb0 0x7f5b400c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:02.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b4015dfb0 0x7f5b400c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53796/0 (socket says 172.21.15.158:53796) 2023-10-16T10:00:02.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 -- 172.21.15.158:0/2417315069 learned_addr learned my addr 172.21.15.158:0/2417315069 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:02.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b3f7fe700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b400c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:02.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b3ffff700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b400c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:02.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 -- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 msgr2=0x7f5b400c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:02.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b400c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 -- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 msgr2=0x7f5b400c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:02.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b400c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:02.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 -- 172.21.15.158:0/2417315069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3800bd40 con 0x7f5b4015dfb0 2023-10-16T10:00:02.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b3f7fe700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b400c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:02.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b3ffff700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b400c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:02.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b44b50700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b4015dfb0 0x7f5b400c9100 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800e0f0 tx=0x7f5b3800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b3cff9700 1 -- 172.21.15.158:0/2417315069 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3800b110 con 0x7f5b4015dfb0 2023-10-16T10:00:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b340094a0 con 0x7f5b4015dfb0 2023-10-16T10:00:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b3cff9700 1 -- 172.21.15.158:0/2417315069 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38010030 con 0x7f5b4015dfb0 2023-10-16T10:00:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.784+0000 7f5b3cff9700 1 -- 172.21.15.158:0/2417315069 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3800b470 con 0x7f5b4015dfb0 2023-10-16T10:00:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.785+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b400b37a0 con 0x7f5b4015dfb0 2023-10-16T10:00:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.785+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b400681b0 con 0x7f5b4015dfb0 2023-10-16T10:00:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.786+0000 7f5b3cff9700 1 -- 172.21.15.158:0/2417315069 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5b38009600 con 0x7f5b4015dfb0 2023-10-16T10:00:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.786+0000 7f5b3cff9700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5b2c06b020 0x7f5b2c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.786+0000 7f5b3cff9700 1 -- 172.21.15.158:0/2417315069 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f5b3808c190 con 0x7f5b4015dfb0 2023-10-16T10:00:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.786+0000 7f5b3ffff700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5b2c06b020 0x7f5b2c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.787+0000 7f5b3ffff700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5b2c06b020 0x7f5b2c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5b400720a0 tx=0x7f5b34008030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:02.791+0000 7f5b3cff9700 1 -- 172.21.15.158:0/2417315069 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b38056d40 con 0x7f5b4015dfb0 2023-10-16T10:00:03.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.027+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b400ca6e0 con 0x7f5b4015dfb0 2023-10-16T10:00:03.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.028+0000 7f5b3cff9700 1 -- 172.21.15.158:0/2417315069 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f5b3805a360 con 0x7f5b4015dfb0 2023-10-16T10:00:03.029 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:03.030 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6676,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2569267295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2569267295},{"type":"v1","addr":"172.21.15.172:6835","nonce":2569267295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1485896118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1485896118},{"type":"v1","addr":"172.21.15.172:6834","nonce":1485896118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2664141151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2664141151},{"type":"v1","addr":"172.21.15.158:6836","nonce":2664141151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:59.006477+0000","modified":"2023-10-16T10:00:02.034482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7239":{"gid":7239,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/1006858952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1006858952},{"type":"v1","addr":"172.21.15.158:6837","nonce":1006858952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T10:00:03.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5b2c06b020 msgr2=0x7f5b2c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5b2c06b020 0x7f5b2c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5b400720a0 tx=0x7f5b34008030 comp rx=0 tx=0).stop 2023-10-16T10:00:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b4015dfb0 msgr2=0x7f5b400c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b4015dfb0 0x7f5b400c9100 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800e0f0 tx=0x7f5b3800e4b0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 shutdown_connections 2023-10-16T10:00:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f5b2c06b020 0x7f5b2c06d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b4015dfb0 0x7f5b400c9100 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b4006f340 0x7f5b400c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 --2- 172.21.15.158:0/2417315069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b400cc5d0 0x7f5b400c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 >> 172.21.15.158:0/2417315069 conn(0x7f5b400d02b0 msgr2=0x7f5b4006b160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 shutdown_connections 2023-10-16T10:00:03.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.030+0000 7f5b465b3700 1 -- 172.21.15.158:0/2417315069 wait complete. 2023-10-16T10:00:03.034 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 42 2023-10-16T10:00:03.043 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:03.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.162+0000 7fd23469e700 1 Processor -- start 2023-10-16T10:00:03.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.162+0000 7fd23469e700 1 -- start start 2023-10-16T10:00:03.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.163+0000 7fd23469e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd22c0b6e10 0x7fd22c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.163+0000 7fd23469e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd22c0ce2c0 0x7fd22c1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.163+0000 7fd23469e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c15c5d0 con 0x7fd22c0cdee0 2023-10-16T10:00:03.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.163+0000 7fd23469e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c15d2c0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.163+0000 7fd23469e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c152c20 con 0x7fd22c0b6e10 2023-10-16T10:00:03.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.163+0000 7fd232c3b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd22c0ce2c0 0x7fd22c1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd232c3b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd22c0ce2c0 0x7fd22c1526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53804/0 (socket says 172.21.15.158:53804) 2023-10-16T10:00:03.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd232c3b700 1 -- 172.21.15.158:0/4095029228 learned_addr learned my addr 172.21.15.158:0/4095029228 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:03.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd23243a700 1 --1- 172.21.15.158:0/4095029228 >> v1:172.21.15.158:6789/0 conn(0x7fd22c0cdee0 0x7fd22c15bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:53976/0 (socket says 172.21.15.158:53976) 2023-10-16T10:00:03.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd231c39700 1 --2- 172.21.15.158:0/4095029228 >> v2:172.21.15.172:3300/0 conn(0x7fd22c0b6e10 0x7fd22c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd232c3b700 1 -- 172.21.15.158:0/4095029228 >> v2:172.21.15.172:3300/0 conn(0x7fd22c0b6e10 msgr2=0x7fd22c1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:03.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd232c3b700 1 --2- 172.21.15.158:0/4095029228 >> v2:172.21.15.172:3300/0 conn(0x7fd22c0b6e10 0x7fd22c1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd232c3b700 1 -- 172.21.15.158:0/4095029228 >> v1:172.21.15.158:6789/0 conn(0x7fd22c0cdee0 legacy=0x7fd22c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd232c3b700 1 -- 172.21.15.158:0/4095029228 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c15c8b0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd231438700 1 -- 172.21.15.158:0/4095029228 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144807463 0 0) 0x7fd22c15c5d0 con 0x7fd22c0cdee0 2023-10-16T10:00:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd232c3b700 1 --2- 172.21.15.158:0/4095029228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c1526e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0b3b90 tx=0x7fd22400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3d4e2f48bcafea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd231438700 1 -- 172.21.15.158:0/4095029228 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22400e070 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd231438700 1 -- 172.21.15.158:0/4095029228 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22400b9c0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.164+0000 7fd231438700 1 -- 172.21.15.158:0/4095029228 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22400bd70 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 -- 172.21.15.158:0/4095029228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 msgr2=0x7fd22c1526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 --2- 172.21.15.158:0/4095029228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c1526e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0b3b90 tx=0x7fd22400b330 comp rx=0 tx=0).stop 2023-10-16T10:00:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 -- 172.21.15.158:0/4095029228 shutdown_connections 2023-10-16T10:00:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 --2- 172.21.15.158:0/4095029228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c1526e0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 --2- 172.21.15.158:0/4095029228 >> v2:172.21.15.172:3300/0 conn(0x7fd22c0b6e10 0x7fd22c1521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 -- 172.21.15.158:0/4095029228 >> 172.21.15.158:0/4095029228 conn(0x7fd22c06a8d0 msgr2=0x7fd22c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:03.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 -- 172.21.15.158:0/4095029228 shutdown_connections 2023-10-16T10:00:03.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.165+0000 7fd23469e700 1 -- 172.21.15.158:0/4095029228 wait complete. 2023-10-16T10:00:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.166+0000 7fd23469e700 1 Processor -- start 2023-10-16T10:00:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.166+0000 7fd23469e700 1 -- start start 2023-10-16T10:00:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd22c0b6e10 0x7fd22c0b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd22c0cdee0 0x7fd22c1f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0b3240 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c75d0 con 0x7fd22c0b6e10 2023-10-16T10:00:03.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0b2f60 con 0x7fd22c0cdee0 2023-10-16T10:00:03.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53810/0 (socket says 172.21.15.158:53810) 2023-10-16T10:00:03.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 -- 172.21.15.158:0/2049133659 learned_addr learned my addr 172.21.15.158:0/2049133659 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:03.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23243a700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd22c0b6e10 0x7fd22c0b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd231c39700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd22c0cdee0 0x7fd22c1f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 -- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd22c0cdee0 msgr2=0x7fd22c1f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd22c0cdee0 0x7fd22c1f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 -- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd22c0b6e10 msgr2=0x7fd22c0b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd22c0b6e10 0x7fd22c0b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 -- 172.21.15.158:0/2049133659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd224009b30 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23243a700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd22c0b6e10 0x7fd22c0b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:03.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd232c3b700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c0b20e0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0ce770 tx=0x7fd224004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd222ffd700 1 -- 172.21.15.158:0/2049133659 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240099b0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd222ffd700 1 -- 172.21.15.158:0/2049133659 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2240092e0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c0c7870 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd222ffd700 1 -- 172.21.15.158:0/2049133659 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22401b650 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.167+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c0c7b50 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.168+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c0681b0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.169+0000 7fd222ffd700 1 -- 172.21.15.158:0/2049133659 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd22401b7b0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.169+0000 7fd222ffd700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fd21006b020 0x7fd21006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.169+0000 7fd222ffd700 1 -- 172.21.15.158:0/2049133659 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fd22408b4f0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.169+0000 7fd23243a700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fd21006b020 0x7fd21006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.170+0000 7fd23243a700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fd21006b020 0x7fd21006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd21c000f60 tx=0x7fd21c007ca0 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.174+0000 7fd222ffd700 1 -- 172.21.15.158:0/2049133659 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd224056080 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.368+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd22c0b3bf0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.369+0000 7fd222ffd700 1 -- 172.21.15.158:0/2049133659 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17434 (secure 0 0 0) 0x7fd2240596a0 con 0x7fd22c0ce2c0 2023-10-16T10:00:03.370 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:03.371 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":98,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:00.604867+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:57.438311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:57.966439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:03.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fd21006b020 msgr2=0x7fd21006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fd21006b020 0x7fd21006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd21c000f60 tx=0x7fd21c007ca0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 msgr2=0x7fd22c0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c0b20e0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0ce770 tx=0x7fd224004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 shutdown_connections 2023-10-16T10:00:03.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fd21006b020 0x7fd21006d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd22c0ce2c0 0x7fd22c0b20e0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd22c0b6e10 0x7fd22c0b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 --2- 172.21.15.158:0/2049133659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd22c0cdee0 0x7fd22c1f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 >> 172.21.15.158:0/2049133659 conn(0x7fd22c06a8d0 msgr2=0x7fd22c0cbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:03.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 shutdown_connections 2023-10-16T10:00:03.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.371+0000 7fd23469e700 1 -- 172.21.15.158:0/2049133659 wait complete. 2023-10-16T10:00:03.386 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T10:00:03.386 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:00:03.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.506+0000 7fcf043e3700 1 Processor -- start 2023-10-16T10:00:03.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.506+0000 7fcf043e3700 1 -- start start 2023-10-16T10:00:03.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.507+0000 7fcf043e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 0x7fcefc161290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.507+0000 7fcf043e3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcefc0ca5b0 0x7fcefc1617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.507+0000 7fcf043e3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcefc06b300 0x7fcefc06b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.507+0000 7fcf043e3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0c4d10 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.507+0000 7fcf043e3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0c55b0 con 0x7fcefc06b300 2023-10-16T10:00:03.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.507+0000 7fcf043e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc15dba0 con 0x7fcefc15d7c0 2023-10-16T10:00:03.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcefc0ca5b0 0x7fcefc1617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcefc0ca5b0 0x7fcefc1617d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53816/0 (socket says 172.21.15.158:53816) 2023-10-16T10:00:03.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 -- 172.21.15.158:0/1032853112 learned_addr learned my addr 172.21.15.158:0/1032853112 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:03.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf02980700 1 --2- 172.21.15.158:0/1032853112 >> v2:172.21.15.172:3300/0 conn(0x7fcefc06b300 0x7fcefc06b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0217f700 1 --2- 172.21.15.158:0/1032853112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 0x7fcefc161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 -- 172.21.15.158:0/1032853112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 msgr2=0x7fcefc161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:03.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 --2- 172.21.15.158:0/1032853112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 0x7fcefc161290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 -- 172.21.15.158:0/1032853112 >> v2:172.21.15.172:3300/0 conn(0x7fcefc06b300 msgr2=0x7fcefc06b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:03.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 --2- 172.21.15.158:0/1032853112 >> v2:172.21.15.172:3300/0 conn(0x7fcefc06b300 0x7fcefc06b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 -- 172.21.15.158:0/1032853112 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc06c0c0 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0197e700 1 --2- 172.21.15.158:0/1032853112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc1617d0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fcefc06f720 tx=0x7fcef8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9824db7556f6f812 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0117d700 1 -- 172.21.15.158:0/1032853112 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8011830 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0117d700 1 -- 172.21.15.158:0/1032853112 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef8011990 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.508+0000 7fcf0117d700 1 -- 172.21.15.158:0/1032853112 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8011d20 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 -- 172.21.15.158:0/1032853112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 msgr2=0x7fcefc1617d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1032853112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc1617d0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fcefc06f720 tx=0x7fcef8009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 -- 172.21.15.158:0/1032853112 shutdown_connections 2023-10-16T10:00:03.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1032853112 >> v2:172.21.15.172:3300/0 conn(0x7fcefc06b300 0x7fcefc06b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1032853112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 0x7fcefc161290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1032853112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc1617d0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 -- 172.21.15.158:0/1032853112 >> 172.21.15.158:0/1032853112 conn(0x7fcefc0cc4b0 msgr2=0x7fcefc0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:03.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 -- 172.21.15.158:0/1032853112 shutdown_connections 2023-10-16T10:00:03.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.509+0000 7fcf043e3700 1 -- 172.21.15.158:0/1032853112 wait complete. 2023-10-16T10:00:03.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.510+0000 7fcf043e3700 1 Processor -- start 2023-10-16T10:00:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.511+0000 7fcf043e3700 1 -- start start 2023-10-16T10:00:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.511+0000 7fcf043e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcefc06b300 0x7fcefc1475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.511+0000 7fcf043e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.511+0000 7fcf043e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 0x7fcefc156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.511+0000 7fcf043e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc147b10 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.511+0000 7fcf043e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc1f3d60 con 0x7fcefc06b300 2023-10-16T10:00:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.511+0000 7fcf043e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc15f4a0 con 0x7fcefc15d7c0 2023-10-16T10:00:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53822/0 (socket says 172.21.15.158:53822) 2023-10-16T10:00:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 -- 172.21.15.158:0/1492235138 learned_addr learned my addr 172.21.15.158:0/1492235138 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:03.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 -- 172.21.15.158:0/1492235138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 msgr2=0x7fcefc156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 0x7fcefc156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0217f700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcefc06b300 0x7fcefc1475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 -- 172.21.15.158:0/1492235138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcefc06b300 msgr2=0x7fcefc1475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcefc06b300 0x7fcefc1475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 -- 172.21.15.158:0/1492235138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8009550 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf0197e700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc155df0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009c10 tx=0x7fcef8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcef2ffd700 1 -- 172.21.15.158:0/1492235138 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8004300 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcef2ffd700 1 -- 172.21.15.158:0/1492235138 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef801f070 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc15f170 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcef2ffd700 1 -- 172.21.15.158:0/1492235138 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef80036a0 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.512+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc156960 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.513+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc0681b0 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.513+0000 7fcef2ffd700 1 -- 172.21.15.158:0/1492235138 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcef8021030 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.514+0000 7fcef2ffd700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fcee006b020 0x7fcee006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.514+0000 7fcef2ffd700 1 -- 172.21.15.158:0/1492235138 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fcef808acd0 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.514+0000 7fcf0217f700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fcee006b020 0x7fcee006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.514+0000 7fcf0217f700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fcee006b020 0x7fcee006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fceec009a10 tx=0x7fceec008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.518+0000 7fcef2ffd700 1 -- 172.21.15.158:0/1492235138 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcef8055970 con 0x7fcefc0ca5b0 2023-10-16T10:00:03.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.711+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 --> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcefc15eff0 con 0x7fcee006b020 2023-10-16T10:00:03.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.719+0000 7fcef2ffd700 1 -- 172.21.15.158:0/1492235138 <== mgr.7005 v2:172.21.15.158:6832/3798429240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289533 (secure 0 0 0) 0x7fceec009660 con 0x7fcee006b020 2023-10-16T10:00:03.724 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:03.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fcee006b020 msgr2=0x7fcee006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fcee006b020 0x7fcee006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fceec009a10 tx=0x7fceec008040 comp rx=0 tx=0).stop 2023-10-16T10:00:03.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 msgr2=0x7fcefc155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc155df0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009c10 tx=0x7fcef8004750 comp rx=0 tx=0).stop 2023-10-16T10:00:03.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 shutdown_connections 2023-10-16T10:00:03.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fcee006b020 0x7fcee006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcefc15d7c0 0x7fcefc156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcefc06b300 0x7fcefc1475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 --2- 172.21.15.158:0/1492235138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcefc0ca5b0 0x7fcefc155df0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 >> 172.21.15.158:0/1492235138 conn(0x7fcefc0cc4b0 msgr2=0x7fcefc06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:03.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 shutdown_connections 2023-10-16T10:00:03.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.723+0000 7fcf043e3700 1 -- 172.21.15.158:0/1492235138 wait complete. 2023-10-16T10:00:03.729 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:00:03.736 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T10:00:03.222129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330780,"kb_used_data":10516,"kb_used_omap":12,"kb_used_meta":320115,"kb_avail":754643940,"statfs":{"total":773094113280,"available":772755394560,"internally_reserved":0,"allocated":10768384,"data_stored":6087096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":327798327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.626345"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613455+0000","last_change":"2023-10-16T09:59:59.626224+0000","last_active":"2023-10-16T10:00:00.613455+0000","last_peered":"2023-10-16T10:00:00.613455+0000","last_clean":"2023-10-16T10:00:00.613455+0000","last_became_active":"2023-10-16T09:59:59.625938+0000","last_became_peered":"2023-10-16T09:59:59.625938+0000","last_unstale":"2023-10-16T10:00:00.613455+0000","last_undegraded":"2023-10-16T10:00:00.613455+0000","last_fullsized":"2023-10-16T10:00:00.613455+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:26.228687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.637953+0000","last_change":"2023-10-16T09:59:59.636815+0000","last_active":"2023-10-16T09:59:59.637953+0000","last_peered":"2023-10-16T09:59:59.637953+0000","last_clean":"2023-10-16T09:59:59.637953+0000","last_became_active":"2023-10-16T09:59:59.636532+0000","last_became_peered":"2023-10-16T09:59:59.636532+0000","last_unstale":"2023-10-16T09:59:59.637953+0000","last_undegraded":"2023-10-16T09:59:59.637953+0000","last_fullsized":"2023-10-16T09:59:59.637953+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:27:44.943360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632844+0000","last_change":"2023-10-16T09:59:59.628243+0000","last_active":"2023-10-16T09:59:59.632844+0000","last_peered":"2023-10-16T09:59:59.632844+0000","last_clean":"2023-10-16T09:59:59.632844+0000","last_became_active":"2023-10-16T09:59:59.627440+0000","last_became_peered":"2023-10-16T09:59:59.627440+0000","last_unstale":"2023-10-16T09:59:59.632844+0000","last_undegraded":"2023-10-16T09:59:59.632844+0000","last_fullsized":"2023-10-16T09:59:59.632844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:21:11.435556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632839+0000","last_change":"2023-10-16T09:59:59.625959+0000","last_active":"2023-10-16T09:59:59.632839+0000","last_peered":"2023-10-16T09:59:59.632839+0000","last_clean":"2023-10-16T09:59:59.632839+0000","last_became_active":"2023-10-16T09:59:59.625662+0000","last_became_peered":"2023-10-16T09:59:59.625662+0000","last_unstale":"2023-10-16T09:59:59.632839+0000","last_undegraded":"2023-10-16T09:59:59.632839+0000","last_fullsized":"2023-10-16T09:59:59.632839+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:24.242736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.642533+0000","last_change":"2023-10-16T09:59:59.640781+0000","last_active":"2023-10-16T09:59:59.642533+0000","last_peered":"2023-10-16T09:59:59.642533+0000","last_clean":"2023-10-16T09:59:59.642533+0000","last_became_active":"2023-10-16T09:59:59.640466+0000","last_became_peered":"2023-10-16T09:59:59.640466+0000","last_unstale":"2023-10-16T09:59:59.642533+0000","last_undegraded":"2023-10-16T09:59:59.642533+0000","last_fullsized":"2023-10-16T09:59:59.642533+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:38.269611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613646+0000","last_change":"2023-10-16T09:59:59.626317+0000","last_active":"2023-10-16T10:00:00.613646+0000","last_peered":"2023-10-16T10:00:00.613646+0000","last_clean":"2023-10-16T10:00:00.613646+0000","last_became_active":"2023-10-16T09:59:59.625993+0000","last_became_peered":"2023-10-16T09:59:59.625993+0000","last_unstale":"2023-10-16T10:00:00.613646+0000","last_undegraded":"2023-10-16T10:00:00.613646+0000","last_fullsized":"2023-10-16T10:00:00.613646+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:06.124021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643508+0000","last_change":"2023-10-16T09:59:59.638874+0000","last_active":"2023-10-16T09:59:59.643508+0000","last_peered":"2023-10-16T09:59:59.643508+0000","last_clean":"2023-10-16T09:59:59.643508+0000","last_became_active":"2023-10-16T09:59:59.638614+0000","last_became_peered":"2023-10-16T09:59:59.638614+0000","last_unstale":"2023-10-16T09:59:59.643508+0000","last_undegraded":"2023-10-16T09:59:59.643508+0000","last_fullsized":"2023-10-16T09:59:59.643508+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:18.720561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635827+0000","last_change":"2023-10-16T09:59:59.623655+0000","last_active":"2023-10-16T09:59:59.635827+0000","last_peered":"2023-10-16T09:59:59.635827+0000","last_clean":"2023-10-16T09:59:59.635827+0000","last_became_active":"2023-10-16T09:59:59.623378+0000","last_became_peered":"2023-10-16T09:59:59.623378+0000","last_unstale":"2023-10-16T09:59:59.635827+0000","last_undegraded":"2023-10-16T09:59:59.635827+0000","last_fullsized":"2023-10-16T09:59:59.635827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:36:49.047683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613266+0000","last_change":"2023-10-16T09:59:59.625862+0000","last_active":"2023-10-16T10:00:00.613266+0000","last_peered":"2023-10-16T10:00:00.613266+0000","last_clean":"2023-10-16T10:00:00.613266+0000","last_became_active":"2023-10-16T09:59:59.625534+0000","last_became_peered":"2023-10-16T09:59:59.625534+0000","last_unstale":"2023-10-16T10:00:00.613266+0000","last_undegraded":"2023-10-16T10:00:00.613266+0000","last_fullsized":"2023-10-16T10:00:00.613266+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:21:11.435556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.631202+0000","last_change":"2023-10-16T09:59:59.623181+0000","last_active":"2023-10-16T09:59:59.631202+0000","last_peered":"2023-10-16T09:59:59.631202+0000","last_clean":"2023-10-16T09:59:59.631202+0000","last_became_active":"2023-10-16T09:59:59.622882+0000","last_became_peered":"2023-10-16T09:59:59.622882+0000","last_unstale":"2023-10-16T09:59:59.631202+0000","last_undegraded":"2023-10-16T09:59:59.631202+0000","last_fullsized":"2023-10-16T09:59:59.631202+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:04:54.785665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.637965+0000","last_change":"2023-10-16T09:59:59.636956+0000","last_active":"2023-10-16T09:59:59.637965+0000","last_peered":"2023-10-16T09:59:59.637965+0000","last_clean":"2023-10-16T09:59:59.637965+0000","last_became_active":"2023-10-16T09:59:59.636658+0000","last_became_peered":"2023-10-16T09:59:59.636658+0000","last_unstale":"2023-10-16T09:59:59.637965+0000","last_undegraded":"2023-10-16T09:59:59.637965+0000","last_fullsized":"2023-10-16T09:59:59.637965+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:54.407322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632858+0000","last_change":"2023-10-16T09:59:59.628664+0000","last_active":"2023-10-16T09:59:59.632858+0000","last_peered":"2023-10-16T09:59:59.632858+0000","last_clean":"2023-10-16T09:59:59.632858+0000","last_became_active":"2023-10-16T09:59:59.628375+0000","last_became_peered":"2023-10-16T09:59:59.628375+0000","last_unstale":"2023-10-16T09:59:59.632858+0000","last_undegraded":"2023-10-16T09:59:59.632858+0000","last_fullsized":"2023-10-16T09:59:59.632858+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:17.169481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.642554+0000","last_change":"2023-10-16T09:59:59.640996+0000","last_active":"2023-10-16T09:59:59.642554+0000","last_peered":"2023-10-16T09:59:59.642554+0000","last_clean":"2023-10-16T09:59:59.642554+0000","last_became_active":"2023-10-16T09:59:59.635548+0000","last_became_peered":"2023-10-16T09:59:59.635548+0000","last_unstale":"2023-10-16T09:59:59.642554+0000","last_undegraded":"2023-10-16T09:59:59.642554+0000","last_fullsized":"2023-10-16T09:59:59.642554+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:37:38.036445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613599+0000","last_change":"2023-10-16T09:59:59.629345+0000","last_active":"2023-10-16T10:00:00.613599+0000","last_peered":"2023-10-16T10:00:00.613599+0000","last_clean":"2023-10-16T10:00:00.613599+0000","last_became_active":"2023-10-16T09:59:59.627694+0000","last_became_peered":"2023-10-16T09:59:59.627694+0000","last_unstale":"2023-10-16T10:00:00.613599+0000","last_undegraded":"2023-10-16T10:00:00.613599+0000","last_fullsized":"2023-10-16T10:00:00.613599+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:13.600302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.628323+0000","last_change":"2023-10-16T09:59:59.622542+0000","last_active":"2023-10-16T09:59:59.628323+0000","last_peered":"2023-10-16T09:59:59.628323+0000","last_clean":"2023-10-16T09:59:59.628323+0000","last_became_active":"2023-10-16T09:59:59.622265+0000","last_became_peered":"2023-10-16T09:59:59.622265+0000","last_unstale":"2023-10-16T09:59:59.628323+0000","last_undegraded":"2023-10-16T09:59:59.628323+0000","last_fullsized":"2023-10-16T09:59:59.628323+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:06.247261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612452+0000","last_change":"2023-10-16T09:59:59.617173+0000","last_active":"2023-10-16T10:00:00.612452+0000","last_peered":"2023-10-16T10:00:00.612452+0000","last_clean":"2023-10-16T10:00:00.612452+0000","last_became_active":"2023-10-16T09:59:59.616865+0000","last_became_peered":"2023-10-16T09:59:59.616865+0000","last_unstale":"2023-10-16T10:00:00.612452+0000","last_undegraded":"2023-10-16T10:00:00.612452+0000","last_fullsized":"2023-10-16T10:00:00.612452+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:24.957452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.640647+0000","last_change":"2023-10-16T09:59:59.635993+0000","last_active":"2023-10-16T09:59:59.640647+0000","last_peered":"2023-10-16T09:59:59.640647+0000","last_clean":"2023-10-16T09:59:59.640647+0000","last_became_active":"2023-10-16T09:59:59.635702+0000","last_became_peered":"2023-10-16T09:59:59.635702+0000","last_unstale":"2023-10-16T09:59:59.640647+0000","last_undegraded":"2023-10-16T09:59:59.640647+0000","last_fullsized":"2023-10-16T09:59:59.640647+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:54.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632856+0000","last_change":"2023-10-16T09:59:59.628141+0000","last_active":"2023-10-16T09:59:59.632856+0000","last_peered":"2023-10-16T09:59:59.632856+0000","last_clean":"2023-10-16T09:59:59.632856+0000","last_became_active":"2023-10-16T09:59:59.627363+0000","last_became_peered":"2023-10-16T09:59:59.627363+0000","last_unstale":"2023-10-16T09:59:59.632856+0000","last_undegraded":"2023-10-16T09:59:59.632856+0000","last_fullsized":"2023-10-16T09:59:59.632856+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:48:02.115525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613711+0000","last_change":"2023-10-16T09:59:59.638835+0000","last_active":"2023-10-16T10:00:00.613711+0000","last_peered":"2023-10-16T10:00:00.613711+0000","last_clean":"2023-10-16T10:00:00.613711+0000","last_became_active":"2023-10-16T09:59:59.638586+0000","last_became_peered":"2023-10-16T09:59:59.638586+0000","last_unstale":"2023-10-16T10:00:00.613711+0000","last_undegraded":"2023-10-16T10:00:00.613711+0000","last_fullsized":"2023-10-16T10:00:00.613711+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:09.065430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.640001+0000","last_change":"2023-10-16T09:59:59.635957+0000","last_active":"2023-10-16T09:59:59.640001+0000","last_peered":"2023-10-16T09:59:59.640001+0000","last_clean":"2023-10-16T09:59:59.640001+0000","last_became_active":"2023-10-16T09:59:59.635666+0000","last_became_peered":"2023-10-16T09:59:59.635666+0000","last_unstale":"2023-10-16T09:59:59.640001+0000","last_undegraded":"2023-10-16T09:59:59.640001+0000","last_fullsized":"2023-10-16T09:59:59.640001+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:53.868228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612907+0000","last_change":"2023-10-16T09:59:59.625613+0000","last_active":"2023-10-16T10:00:00.612907+0000","last_peered":"2023-10-16T10:00:00.612907+0000","last_clean":"2023-10-16T10:00:00.612907+0000","last_became_active":"2023-10-16T09:59:59.625188+0000","last_became_peered":"2023-10-16T09:59:59.625188+0000","last_unstale":"2023-10-16T10:00:00.612907+0000","last_undegraded":"2023-10-16T10:00:00.612907+0000","last_fullsized":"2023-10-16T10:00:00.612907+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:48.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.639859+0000","last_change":"2023-10-16T09:59:59.635478+0000","last_active":"2023-10-16T09:59:59.639859+0000","last_peered":"2023-10-16T09:59:59.639859+0000","last_clean":"2023-10-16T09:59:59.639859+0000","last_became_active":"2023-10-16T09:59:59.635154+0000","last_became_peered":"2023-10-16T09:59:59.635154+0000","last_unstale":"2023-10-16T09:59:59.639859+0000","last_undegraded":"2023-10-16T09:59:59.639859+0000","last_fullsized":"2023-10-16T09:59:59.639859+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:55.077197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613388+0000","last_change":"2023-10-16T09:59:59.622128+0000","last_active":"2023-10-16T10:00:00.613388+0000","last_peered":"2023-10-16T10:00:00.613388+0000","last_clean":"2023-10-16T10:00:00.613388+0000","last_became_active":"2023-10-16T09:59:59.621868+0000","last_became_peered":"2023-10-16T09:59:59.621868+0000","last_unstale":"2023-10-16T10:00:00.613388+0000","last_undegraded":"2023-10-16T10:00:00.613388+0000","last_fullsized":"2023-10-16T10:00:00.613388+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:07:45.973592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626318+0000","last_change":"2023-10-16T09:59:59.620760+0000","last_active":"2023-10-16T09:59:59.626318+0000","last_peered":"2023-10-16T09:59:59.626318+0000","last_clean":"2023-10-16T09:59:59.626318+0000","last_became_active":"2023-10-16T09:59:59.620465+0000","last_became_peered":"2023-10-16T09:59:59.620465+0000","last_unstale":"2023-10-16T09:59:59.626318+0000","last_undegraded":"2023-10-16T09:59:59.626318+0000","last_fullsized":"2023-10-16T09:59:59.626318+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:27.946882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626526+0000","last_change":"2023-10-16T09:59:59.622906+0000","last_active":"2023-10-16T09:59:59.626526+0000","last_peered":"2023-10-16T09:59:59.626526+0000","last_clean":"2023-10-16T09:59:59.626526+0000","last_became_active":"2023-10-16T09:59:59.622623+0000","last_became_peered":"2023-10-16T09:59:59.622623+0000","last_unstale":"2023-10-16T09:59:59.626526+0000","last_undegraded":"2023-10-16T09:59:59.626526+0000","last_fullsized":"2023-10-16T09:59:59.626526+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:51.904350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.633250+0000","last_change":"2023-10-16T09:59:59.628714+0000","last_active":"2023-10-16T09:59:59.633250+0000","last_peered":"2023-10-16T09:59:59.633250+0000","last_clean":"2023-10-16T09:59:59.633250+0000","last_became_active":"2023-10-16T09:59:59.628424+0000","last_became_peered":"2023-10-16T09:59:59.628424+0000","last_unstale":"2023-10-16T09:59:59.633250+0000","last_undegraded":"2023-10-16T09:59:59.633250+0000","last_fullsized":"2023-10-16T09:59:59.633250+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:53.621117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613448+0000","last_change":"2023-10-16T09:59:59.619292+0000","last_active":"2023-10-16T10:00:00.613448+0000","last_peered":"2023-10-16T10:00:00.613448+0000","last_clean":"2023-10-16T10:00:00.613448+0000","last_became_active":"2023-10-16T09:59:59.618911+0000","last_became_peered":"2023-10-16T09:59:59.618911+0000","last_unstale":"2023-10-16T10:00:00.613448+0000","last_undegraded":"2023-10-16T10:00:00.613448+0000","last_fullsized":"2023-10-16T10:00:00.613448+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:13.600302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.638136+0000","last_change":"2023-10-16T09:59:59.616080+0000","last_active":"2023-10-16T09:59:59.638136+0000","last_peered":"2023-10-16T09:59:59.638136+0000","last_clean":"2023-10-16T09:59:59.638136+0000","last_became_active":"2023-10-16T09:59:59.615698+0000","last_became_peered":"2023-10-16T09:59:59.615698+0000","last_unstale":"2023-10-16T09:59:59.638136+0000","last_undegraded":"2023-10-16T09:59:59.638136+0000","last_fullsized":"2023-10-16T09:59:59.638136+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:41:21.351163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626438+0000","last_change":"2023-10-16T09:59:59.623109+0000","last_active":"2023-10-16T09:59:59.626438+0000","last_peered":"2023-10-16T09:59:59.626438+0000","last_clean":"2023-10-16T09:59:59.626438+0000","last_became_active":"2023-10-16T09:59:59.622891+0000","last_became_peered":"2023-10-16T09:59:59.622891+0000","last_unstale":"2023-10-16T09:59:59.626438+0000","last_undegraded":"2023-10-16T09:59:59.626438+0000","last_fullsized":"2023-10-16T09:59:59.626438+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:50.955432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613036+0000","last_change":"2023-10-16T09:59:59.623095+0000","last_active":"2023-10-16T10:00:00.613036+0000","last_peered":"2023-10-16T10:00:00.613036+0000","last_clean":"2023-10-16T10:00:00.613036+0000","last_became_active":"2023-10-16T09:59:59.621647+0000","last_became_peered":"2023-10-16T09:59:59.621647+0000","last_unstale":"2023-10-16T10:00:00.613036+0000","last_undegraded":"2023-10-16T10:00:00.613036+0000","last_fullsized":"2023-10-16T10:00:00.613036+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:44.249052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632920+0000","last_change":"2023-10-16T09:59:59.628025+0000","last_active":"2023-10-16T09:59:59.632920+0000","last_peered":"2023-10-16T09:59:59.632920+0000","last_clean":"2023-10-16T09:59:59.632920+0000","last_became_active":"2023-10-16T09:59:59.627380+0000","last_became_peered":"2023-10-16T09:59:59.627380+0000","last_unstale":"2023-10-16T09:59:59.632920+0000","last_undegraded":"2023-10-16T09:59:59.632920+0000","last_fullsized":"2023-10-16T09:59:59.632920+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:07:11.376044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612468+0000","last_change":"2023-10-16T09:59:59.627078+0000","last_active":"2023-10-16T10:00:00.612468+0000","last_peered":"2023-10-16T10:00:00.612468+0000","last_clean":"2023-10-16T10:00:00.612468+0000","last_became_active":"2023-10-16T09:59:59.626626+0000","last_became_peered":"2023-10-16T09:59:59.626626+0000","last_unstale":"2023-10-16T10:00:00.612468+0000","last_undegraded":"2023-10-16T10:00:00.612468+0000","last_fullsized":"2023-10-16T10:00:00.612468+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:17.660472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632107+0000","last_change":"2023-10-16T09:59:59.627011+0000","last_active":"2023-10-16T09:59:59.632107+0000","last_peered":"2023-10-16T09:59:59.632107+0000","last_clean":"2023-10-16T09:59:59.632107+0000","last_became_active":"2023-10-16T09:59:59.626677+0000","last_became_peered":"2023-10-16T09:59:59.626677+0000","last_unstale":"2023-10-16T09:59:59.632107+0000","last_undegraded":"2023-10-16T09:59:59.632107+0000","last_fullsized":"2023-10-16T09:59:59.632107+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:18.720561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"97'4","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.644209+0000","last_change":"2023-10-16T09:59:58.624762+0000","last_active":"2023-10-16T09:59:59.644209+0000","last_peered":"2023-10-16T09:59:59.644209+0000","last_clean":"2023-10-16T09:59:59.644209+0000","last_became_active":"2023-10-16T09:59:58.624504+0000","last_became_peered":"2023-10-16T09:59:58.624504+0000","last_unstale":"2023-10-16T09:59:59.644209+0000","last_undegraded":"2023-10-16T09:59:59.644209+0000","last_fullsized":"2023-10-16T09:59:59.644209+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:34.766524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.617970+0000","last_change":"2023-10-16T09:59:59.613077+0000","last_active":"2023-10-16T09:59:59.617970+0000","last_peered":"2023-10-16T09:59:59.617970+0000","last_clean":"2023-10-16T09:59:59.617970+0000","last_became_active":"2023-10-16T09:59:59.612786+0000","last_became_peered":"2023-10-16T09:59:59.612786+0000","last_unstale":"2023-10-16T09:59:59.617970+0000","last_undegraded":"2023-10-16T09:59:59.617970+0000","last_fullsized":"2023-10-16T09:59:59.617970+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:38.491006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"97'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613516+0000","last_change":"2023-10-16T09:59:58.620958+0000","last_active":"2023-10-16T10:00:00.613516+0000","last_peered":"2023-10-16T10:00:00.613516+0000","last_clean":"2023-10-16T10:00:00.613516+0000","last_became_active":"2023-10-16T09:59:58.620551+0000","last_became_peered":"2023-10-16T09:59:58.620551+0000","last_unstale":"2023-10-16T10:00:00.613516+0000","last_undegraded":"2023-10-16T10:00:00.613516+0000","last_fullsized":"2023-10-16T10:00:00.613516+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:16.162032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612948+0000","last_change":"2023-10-16T09:59:59.626738+0000","last_active":"2023-10-16T10:00:00.612948+0000","last_peered":"2023-10-16T10:00:00.612948+0000","last_clean":"2023-10-16T10:00:00.612948+0000","last_became_active":"2023-10-16T09:59:59.626411+0000","last_became_peered":"2023-10-16T09:59:59.626411+0000","last_unstale":"2023-10-16T10:00:00.612948+0000","last_undegraded":"2023-10-16T10:00:00.612948+0000","last_fullsized":"2023-10-16T10:00:00.612948+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:12.623356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"97'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613558+0000","last_change":"2023-10-16T09:59:58.617465+0000","last_active":"2023-10-16T10:00:00.613558+0000","last_peered":"2023-10-16T10:00:00.613558+0000","last_clean":"2023-10-16T10:00:00.613558+0000","last_became_active":"2023-10-16T09:59:58.617155+0000","last_became_peered":"2023-10-16T09:59:58.617155+0000","last_unstale":"2023-10-16T10:00:00.613558+0000","last_undegraded":"2023-10-16T10:00:00.613558+0000","last_fullsized":"2023-10-16T10:00:00.613558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:55.612848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613121+0000","last_change":"2023-10-16T09:59:59.628842+0000","last_active":"2023-10-16T10:00:00.613121+0000","last_peered":"2023-10-16T10:00:00.613121+0000","last_clean":"2023-10-16T10:00:00.613121+0000","last_became_active":"2023-10-16T09:59:59.626453+0000","last_became_peered":"2023-10-16T09:59:59.626453+0000","last_unstale":"2023-10-16T10:00:00.613121+0000","last_undegraded":"2023-10-16T10:00:00.613121+0000","last_fullsized":"2023-10-16T10:00:00.613121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:07:11.376044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"97'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635450+0000","last_change":"2023-10-16T09:59:58.618372+0000","last_active":"2023-10-16T09:59:59.635450+0000","last_peered":"2023-10-16T09:59:59.635450+0000","last_clean":"2023-10-16T09:59:59.635450+0000","last_became_active":"2023-10-16T09:59:58.618045+0000","last_became_peered":"2023-10-16T09:59:58.618045+0000","last_unstale":"2023-10-16T09:59:59.635450+0000","last_undegraded":"2023-10-16T09:59:59.635450+0000","last_fullsized":"2023-10-16T09:59:59.635450+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:16.530500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.645063+0000","last_change":"2023-10-16T09:59:59.644631+0000","last_active":"2023-10-16T09:59:59.645063+0000","last_peered":"2023-10-16T09:59:59.645063+0000","last_clean":"2023-10-16T09:59:59.645063+0000","last_became_active":"2023-10-16T09:59:59.644272+0000","last_became_peered":"2023-10-16T09:59:59.644272+0000","last_unstale":"2023-10-16T09:59:59.645063+0000","last_undegraded":"2023-10-16T09:59:59.645063+0000","last_fullsized":"2023-10-16T09:59:59.645063+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:09.744805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"97'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613016+0000","last_change":"2023-10-16T09:59:58.620864+0000","last_active":"2023-10-16T10:00:00.613016+0000","last_peered":"2023-10-16T10:00:00.613016+0000","last_clean":"2023-10-16T10:00:00.613016+0000","last_became_active":"2023-10-16T09:59:58.620368+0000","last_became_peered":"2023-10-16T09:59:58.620368+0000","last_unstale":"2023-10-16T10:00:00.613016+0000","last_undegraded":"2023-10-16T10:00:00.613016+0000","last_fullsized":"2023-10-16T10:00:00.613016+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:32.134734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626563+0000","last_change":"2023-10-16T09:59:59.617389+0000","last_active":"2023-10-16T09:59:59.626563+0000","last_peered":"2023-10-16T09:59:59.626563+0000","last_clean":"2023-10-16T09:59:59.626563+0000","last_became_active":"2023-10-16T09:59:59.617107+0000","last_became_peered":"2023-10-16T09:59:59.617107+0000","last_unstale":"2023-10-16T09:59:59.626563+0000","last_undegraded":"2023-10-16T09:59:59.626563+0000","last_fullsized":"2023-10-16T09:59:59.626563+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:31.032244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"97'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632834+0000","last_change":"2023-10-16T09:59:58.624889+0000","last_active":"2023-10-16T09:59:59.632834+0000","last_peered":"2023-10-16T09:59:59.632834+0000","last_clean":"2023-10-16T09:59:59.632834+0000","last_became_active":"2023-10-16T09:59:58.624602+0000","last_became_peered":"2023-10-16T09:59:58.624602+0000","last_unstale":"2023-10-16T09:59:59.632834+0000","last_undegraded":"2023-10-16T09:59:59.632834+0000","last_fullsized":"2023-10-16T09:59:59.632834+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:17.280350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613210+0000","last_change":"2023-10-16T09:59:59.626856+0000","last_active":"2023-10-16T10:00:00.613210+0000","last_peered":"2023-10-16T10:00:00.613210+0000","last_clean":"2023-10-16T10:00:00.613210+0000","last_became_active":"2023-10-16T09:59:59.625352+0000","last_became_peered":"2023-10-16T09:59:59.625352+0000","last_unstale":"2023-10-16T10:00:00.613210+0000","last_undegraded":"2023-10-16T10:00:00.613210+0000","last_fullsized":"2023-10-16T10:00:00.613210+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:00.811337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.607281+0000","last_change":"2023-10-16T09:59:58.617074+0000","last_active":"2023-10-16T09:59:59.607281+0000","last_peered":"2023-10-16T09:59:59.607281+0000","last_clean":"2023-10-16T09:59:59.607281+0000","last_became_active":"2023-10-16T09:59:58.616834+0000","last_became_peered":"2023-10-16T09:59:58.616834+0000","last_unstale":"2023-10-16T09:59:59.607281+0000","last_undegraded":"2023-10-16T09:59:59.607281+0000","last_fullsized":"2023-10-16T09:59:59.607281+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:51.989803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643528+0000","last_change":"2023-10-16T09:59:59.639061+0000","last_active":"2023-10-16T09:59:59.643528+0000","last_peered":"2023-10-16T09:59:59.643528+0000","last_clean":"2023-10-16T09:59:59.643528+0000","last_became_active":"2023-10-16T09:59:59.638829+0000","last_became_peered":"2023-10-16T09:59:59.638829+0000","last_unstale":"2023-10-16T09:59:59.643528+0000","last_undegraded":"2023-10-16T09:59:59.643528+0000","last_fullsized":"2023-10-16T09:59:59.643528+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:48.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.616184+0000","last_change":"2023-10-16T09:59:58.615808+0000","last_active":"2023-10-16T09:59:59.616184+0000","last_peered":"2023-10-16T09:59:59.616184+0000","last_clean":"2023-10-16T09:59:59.616184+0000","last_became_active":"2023-10-16T09:59:58.615576+0000","last_became_peered":"2023-10-16T09:59:58.615576+0000","last_unstale":"2023-10-16T09:59:59.616184+0000","last_undegraded":"2023-10-16T09:59:59.616184+0000","last_fullsized":"2023-10-16T09:59:59.616184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:12.963507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.621092+0000","last_change":"2023-10-16T09:59:59.615222+0000","last_active":"2023-10-16T09:59:59.621092+0000","last_peered":"2023-10-16T09:59:59.621092+0000","last_clean":"2023-10-16T09:59:59.621092+0000","last_became_active":"2023-10-16T09:59:59.614962+0000","last_became_peered":"2023-10-16T09:59:59.614962+0000","last_unstale":"2023-10-16T09:59:59.621092+0000","last_undegraded":"2023-10-16T09:59:59.621092+0000","last_fullsized":"2023-10-16T09:59:59.621092+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:16:59.866866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613632+0000","last_change":"2023-10-16T09:59:58.620712+0000","last_active":"2023-10-16T10:00:00.613632+0000","last_peered":"2023-10-16T10:00:00.613632+0000","last_clean":"2023-10-16T10:00:00.613632+0000","last_became_active":"2023-10-16T09:59:58.619281+0000","last_became_peered":"2023-10-16T09:59:58.619281+0000","last_unstale":"2023-10-16T10:00:00.613632+0000","last_undegraded":"2023-10-16T10:00:00.613632+0000","last_fullsized":"2023-10-16T10:00:00.613632+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:17.280350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.631210+0000","last_change":"2023-10-16T09:59:59.623052+0000","last_active":"2023-10-16T09:59:59.631210+0000","last_peered":"2023-10-16T09:59:59.631210+0000","last_clean":"2023-10-16T09:59:59.631210+0000","last_became_active":"2023-10-16T09:59:59.622509+0000","last_became_peered":"2023-10-16T09:59:59.622509+0000","last_unstale":"2023-10-16T09:59:59.631210+0000","last_undegraded":"2023-10-16T09:59:59.631210+0000","last_fullsized":"2023-10-16T09:59:59.631210+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:59.056218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.611360+0000","last_change":"2023-10-16T09:59:58.621839+0000","last_active":"2023-10-16T09:59:59.611360+0000","last_peered":"2023-10-16T09:59:59.611360+0000","last_clean":"2023-10-16T09:59:59.611360+0000","last_became_active":"2023-10-16T09:59:58.621580+0000","last_became_peered":"2023-10-16T09:59:58.621580+0000","last_unstale":"2023-10-16T09:59:59.611360+0000","last_undegraded":"2023-10-16T09:59:59.611360+0000","last_fullsized":"2023-10-16T09:59:59.611360+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:36.752829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613491+0000","last_change":"2023-10-16T09:59:59.622677+0000","last_active":"2023-10-16T10:00:00.613491+0000","last_peered":"2023-10-16T10:00:00.613491+0000","last_clean":"2023-10-16T10:00:00.613491+0000","last_became_active":"2023-10-16T09:59:59.622373+0000","last_became_peered":"2023-10-16T09:59:59.622373+0000","last_unstale":"2023-10-16T10:00:00.613491+0000","last_undegraded":"2023-10-16T10:00:00.613491+0000","last_fullsized":"2023-10-16T10:00:00.613491+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:25.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"97'3","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643204+0000","last_change":"2023-10-16T09:59:58.619676+0000","last_active":"2023-10-16T09:59:59.643204+0000","last_peered":"2023-10-16T09:59:59.643204+0000","last_clean":"2023-10-16T09:59:59.643204+0000","last_became_active":"2023-10-16T09:59:58.619420+0000","last_became_peered":"2023-10-16T09:59:58.619420+0000","last_unstale":"2023-10-16T09:59:59.643204+0000","last_undegraded":"2023-10-16T09:59:59.643204+0000","last_fullsized":"2023-10-16T09:59:59.643204+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:19:15.153943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632769+0000","last_change":"2023-10-16T09:59:59.627746+0000","last_active":"2023-10-16T09:59:59.632769+0000","last_peered":"2023-10-16T09:59:59.632769+0000","last_clean":"2023-10-16T09:59:59.632769+0000","last_became_active":"2023-10-16T09:59:59.627160+0000","last_became_peered":"2023-10-16T09:59:59.627160+0000","last_unstale":"2023-10-16T09:59:59.632769+0000","last_undegraded":"2023-10-16T09:59:59.632769+0000","last_fullsized":"2023-10-16T09:59:59.632769+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:07:45.973592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"97'6","reported_seq":17,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643588+0000","last_change":"2023-10-16T09:59:58.621867+0000","last_active":"2023-10-16T09:59:59.643588+0000","last_peered":"2023-10-16T09:59:59.643588+0000","last_clean":"2023-10-16T09:59:59.643588+0000","last_became_active":"2023-10-16T09:59:58.621519+0000","last_became_peered":"2023-10-16T09:59:58.621519+0000","last_unstale":"2023-10-16T09:59:59.643588+0000","last_undegraded":"2023-10-16T09:59:59.643588+0000","last_fullsized":"2023-10-16T09:59:59.643588+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:05.454987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632509+0000","last_change":"2023-10-16T09:59:59.623230+0000","last_active":"2023-10-16T09:59:59.632509+0000","last_peered":"2023-10-16T09:59:59.632509+0000","last_clean":"2023-10-16T09:59:59.632509+0000","last_became_active":"2023-10-16T09:59:59.622938+0000","last_became_peered":"2023-10-16T09:59:59.622938+0000","last_unstale":"2023-10-16T09:59:59.632509+0000","last_undegraded":"2023-10-16T09:59:59.632509+0000","last_fullsized":"2023-10-16T09:59:59.632509+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:19.035867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"97'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.636187+0000","last_change":"2023-10-16T09:59:58.614363+0000","last_active":"2023-10-16T09:59:59.636187+0000","last_peered":"2023-10-16T09:59:59.636187+0000","last_clean":"2023-10-16T09:59:59.636187+0000","last_became_active":"2023-10-16T09:59:58.614066+0000","last_became_peered":"2023-10-16T09:59:58.614066+0000","last_unstale":"2023-10-16T09:59:59.636187+0000","last_undegraded":"2023-10-16T09:59:59.636187+0000","last_fullsized":"2023-10-16T09:59:59.636187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:55.612848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635368+0000","last_change":"2023-10-16T09:59:59.629993+0000","last_active":"2023-10-16T09:59:59.635368+0000","last_peered":"2023-10-16T09:59:59.635368+0000","last_clean":"2023-10-16T09:59:59.635368+0000","last_became_active":"2023-10-16T09:59:59.629705+0000","last_became_peered":"2023-10-16T09:59:59.629705+0000","last_unstale":"2023-10-16T09:59:59.635368+0000","last_undegraded":"2023-10-16T09:59:59.635368+0000","last_fullsized":"2023-10-16T09:59:59.635368+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:43.179440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613128+0000","last_change":"2023-10-16T09:59:58.616338+0000","last_active":"2023-10-16T10:00:00.613128+0000","last_peered":"2023-10-16T10:00:00.613128+0000","last_clean":"2023-10-16T10:00:00.613128+0000","last_became_active":"2023-10-16T09:59:58.616065+0000","last_became_peered":"2023-10-16T09:59:58.616065+0000","last_unstale":"2023-10-16T10:00:00.613128+0000","last_undegraded":"2023-10-16T10:00:00.613128+0000","last_fullsized":"2023-10-16T10:00:00.613128+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:23.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635526+0000","last_change":"2023-10-16T09:59:59.627772+0000","last_active":"2023-10-16T09:59:59.635526+0000","last_peered":"2023-10-16T09:59:59.635526+0000","last_clean":"2023-10-16T09:59:59.635526+0000","last_became_active":"2023-10-16T09:59:59.627464+0000","last_became_peered":"2023-10-16T09:59:59.627464+0000","last_unstale":"2023-10-16T09:59:59.635526+0000","last_undegraded":"2023-10-16T09:59:59.635526+0000","last_fullsized":"2023-10-16T09:59:59.635526+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:52:31.337269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.615641+0000","last_change":"2023-10-16T09:59:58.622887+0000","last_active":"2023-10-16T09:59:59.615641+0000","last_peered":"2023-10-16T09:59:59.615641+0000","last_clean":"2023-10-16T09:59:59.615641+0000","last_became_active":"2023-10-16T09:59:58.622622+0000","last_became_peered":"2023-10-16T09:59:58.622622+0000","last_unstale":"2023-10-16T09:59:59.615641+0000","last_undegraded":"2023-10-16T09:59:59.615641+0000","last_fullsized":"2023-10-16T09:59:59.615641+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:48.271736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635465+0000","last_change":"2023-10-16T09:59:59.631991+0000","last_active":"2023-10-16T09:59:59.635465+0000","last_peered":"2023-10-16T09:59:59.635465+0000","last_clean":"2023-10-16T09:59:59.635465+0000","last_became_active":"2023-10-16T09:59:59.630142+0000","last_became_peered":"2023-10-16T09:59:59.630142+0000","last_unstale":"2023-10-16T09:59:59.635465+0000","last_undegraded":"2023-10-16T09:59:59.635465+0000","last_fullsized":"2023-10-16T09:59:59.635465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:54.625419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":19,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.637001+0000","last_change":"2023-10-16T09:59:58.618185+0000","last_active":"2023-10-16T09:59:59.637001+0000","last_peered":"2023-10-16T09:59:59.637001+0000","last_clean":"2023-10-16T09:59:59.637001+0000","last_became_active":"2023-10-16T09:59:58.617840+0000","last_became_peered":"2023-10-16T09:59:58.617840+0000","last_unstale":"2023-10-16T09:59:59.637001+0000","last_undegraded":"2023-10-16T09:59:59.637001+0000","last_fullsized":"2023-10-16T09:59:59.637001+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:09.477597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613178+0000","last_change":"2023-10-16T09:59:59.636714+0000","last_active":"2023-10-16T10:00:00.613178+0000","last_peered":"2023-10-16T10:00:00.613178+0000","last_clean":"2023-10-16T10:00:00.613178+0000","last_became_active":"2023-10-16T09:59:59.636445+0000","last_became_peered":"2023-10-16T09:59:59.636445+0000","last_unstale":"2023-10-16T10:00:00.613178+0000","last_undegraded":"2023-10-16T10:00:00.613178+0000","last_fullsized":"2023-10-16T10:00:00.613178+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:44.249052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"97'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613687+0000","last_change":"2023-10-16T09:59:58.617946+0000","last_active":"2023-10-16T10:00:00.613687+0000","last_peered":"2023-10-16T10:00:00.613687+0000","last_clean":"2023-10-16T10:00:00.613687+0000","last_became_active":"2023-10-16T09:59:58.617403+0000","last_became_peered":"2023-10-16T09:59:58.617403+0000","last_unstale":"2023-10-16T10:00:00.613687+0000","last_undegraded":"2023-10-16T10:00:00.613687+0000","last_fullsized":"2023-10-16T10:00:00.613687+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:04:53.778216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.640330+0000","last_change":"2023-10-16T09:59:59.635520+0000","last_active":"2023-10-16T09:59:59.640330+0000","last_peered":"2023-10-16T09:59:59.640330+0000","last_clean":"2023-10-16T09:59:59.640330+0000","last_became_active":"2023-10-16T09:59:59.635108+0000","last_became_peered":"2023-10-16T09:59:59.635108+0000","last_unstale":"2023-10-16T09:59:59.640330+0000","last_undegraded":"2023-10-16T09:59:59.640330+0000","last_fullsized":"2023-10-16T09:59:59.640330+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:47.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"97'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613357+0000","last_change":"2023-10-16T09:59:58.621599+0000","last_active":"2023-10-16T10:00:00.613357+0000","last_peered":"2023-10-16T10:00:00.613357+0000","last_clean":"2023-10-16T10:00:00.613357+0000","last_became_active":"2023-10-16T09:59:58.621142+0000","last_became_peered":"2023-10-16T09:59:58.621142+0000","last_unstale":"2023-10-16T10:00:00.613357+0000","last_undegraded":"2023-10-16T10:00:00.613357+0000","last_fullsized":"2023-10-16T10:00:00.613357+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:51:13.193671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635884+0000","last_change":"2023-10-16T09:59:59.624290+0000","last_active":"2023-10-16T09:59:59.635884+0000","last_peered":"2023-10-16T09:59:59.635884+0000","last_clean":"2023-10-16T09:59:59.635884+0000","last_became_active":"2023-10-16T09:59:59.616441+0000","last_became_peered":"2023-10-16T09:59:59.616441+0000","last_unstale":"2023-10-16T09:59:59.635884+0000","last_undegraded":"2023-10-16T09:59:59.635884+0000","last_fullsized":"2023-10-16T09:59:59.635884+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:05:28.238511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"97'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643155+0000","last_change":"2023-10-16T09:59:58.614640+0000","last_active":"2023-10-16T09:59:59.643155+0000","last_peered":"2023-10-16T09:59:59.643155+0000","last_clean":"2023-10-16T09:59:59.643155+0000","last_became_active":"2023-10-16T09:59:58.614352+0000","last_became_peered":"2023-10-16T09:59:58.614352+0000","last_unstale":"2023-10-16T09:59:59.643155+0000","last_undegraded":"2023-10-16T09:59:59.643155+0000","last_fullsized":"2023-10-16T09:59:59.643155+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:47.353949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613456+0000","last_change":"2023-10-16T09:59:59.641573+0000","last_active":"2023-10-16T10:00:00.613456+0000","last_peered":"2023-10-16T10:00:00.613456+0000","last_clean":"2023-10-16T10:00:00.613456+0000","last_became_active":"2023-10-16T09:59:59.641306+0000","last_became_peered":"2023-10-16T09:59:59.641306+0000","last_unstale":"2023-10-16T10:00:00.613456+0000","last_undegraded":"2023-10-16T10:00:00.613456+0000","last_fullsized":"2023-10-16T10:00:00.613456+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:26.228687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.610634+0000","last_change":"2023-10-16T09:59:58.618990+0000","last_active":"2023-10-16T09:59:59.610634+0000","last_peered":"2023-10-16T09:59:59.610634+0000","last_clean":"2023-10-16T09:59:59.610634+0000","last_became_active":"2023-10-16T09:59:58.618729+0000","last_became_peered":"2023-10-16T09:59:58.618729+0000","last_unstale":"2023-10-16T09:59:59.610634+0000","last_undegraded":"2023-10-16T09:59:59.610634+0000","last_fullsized":"2023-10-16T09:59:59.610634+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:11.615907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.621113+0000","last_change":"2023-10-16T09:59:59.613731+0000","last_active":"2023-10-16T09:59:59.621113+0000","last_peered":"2023-10-16T09:59:59.621113+0000","last_clean":"2023-10-16T09:59:59.621113+0000","last_became_active":"2023-10-16T09:59:59.613480+0000","last_became_peered":"2023-10-16T09:59:59.613480+0000","last_unstale":"2023-10-16T09:59:59.621113+0000","last_undegraded":"2023-10-16T09:59:59.621113+0000","last_fullsized":"2023-10-16T09:59:59.621113+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:57:42.348980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"97'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613474+0000","last_change":"2023-10-16T09:59:58.624490+0000","last_active":"2023-10-16T10:00:00.613474+0000","last_peered":"2023-10-16T10:00:00.613474+0000","last_clean":"2023-10-16T10:00:00.613474+0000","last_became_active":"2023-10-16T09:59:58.624231+0000","last_became_peered":"2023-10-16T09:59:58.624231+0000","last_unstale":"2023-10-16T10:00:00.613474+0000","last_undegraded":"2023-10-16T10:00:00.613474+0000","last_fullsized":"2023-10-16T10:00:00.613474+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:16.653024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.628275+0000","last_change":"2023-10-16T09:59:59.626030+0000","last_active":"2023-10-16T09:59:59.628275+0000","last_peered":"2023-10-16T09:59:59.628275+0000","last_clean":"2023-10-16T09:59:59.628275+0000","last_became_active":"2023-10-16T09:59:59.625737+0000","last_became_peered":"2023-10-16T09:59:59.625737+0000","last_unstale":"2023-10-16T09:59:59.628275+0000","last_undegraded":"2023-10-16T09:59:59.628275+0000","last_fullsized":"2023-10-16T09:59:59.628275+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:12.515762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613329+0000","last_change":"2023-10-16T09:59:58.626472+0000","last_active":"2023-10-16T10:00:00.613329+0000","last_peered":"2023-10-16T10:00:00.613329+0000","last_clean":"2023-10-16T10:00:00.613329+0000","last_became_active":"2023-10-16T09:59:58.626202+0000","last_became_peered":"2023-10-16T09:59:58.626202+0000","last_unstale":"2023-10-16T10:00:00.613329+0000","last_undegraded":"2023-10-16T10:00:00.613329+0000","last_fullsized":"2023-10-16T10:00:00.613329+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:23.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635348+0000","last_change":"2023-10-16T09:59:59.628375+0000","last_active":"2023-10-16T09:59:59.635348+0000","last_peered":"2023-10-16T09:59:59.635348+0000","last_clean":"2023-10-16T09:59:59.635348+0000","last_became_active":"2023-10-16T09:59:59.628039+0000","last_became_peered":"2023-10-16T09:59:59.628039+0000","last_unstale":"2023-10-16T09:59:59.635348+0000","last_undegraded":"2023-10-16T09:59:59.635348+0000","last_fullsized":"2023-10-16T09:59:59.635348+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:51:14.201119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613262+0000","last_change":"2023-10-16T09:59:58.625678+0000","last_active":"2023-10-16T10:00:00.613262+0000","last_peered":"2023-10-16T10:00:00.613262+0000","last_clean":"2023-10-16T10:00:00.613262+0000","last_became_active":"2023-10-16T09:59:58.625376+0000","last_became_peered":"2023-10-16T09:59:58.625376+0000","last_unstale":"2023-10-16T10:00:00.613262+0000","last_undegraded":"2023-10-16T10:00:00.613262+0000","last_fullsized":"2023-10-16T10:00:00.613262+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:26.838497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612999+0000","last_change":"2023-10-16T09:59:59.625835+0000","last_active":"2023-10-16T10:00:00.612999+0000","last_peered":"2023-10-16T10:00:00.612999+0000","last_clean":"2023-10-16T10:00:00.612999+0000","last_became_active":"2023-10-16T09:59:59.625462+0000","last_became_peered":"2023-10-16T09:59:59.625462+0000","last_unstale":"2023-10-16T10:00:00.612999+0000","last_undegraded":"2023-10-16T10:00:00.612999+0000","last_fullsized":"2023-10-16T10:00:00.612999+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:18.720561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"97'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.633790+0000","last_change":"2023-10-16T09:59:58.620387+0000","last_active":"2023-10-16T09:59:59.633790+0000","last_peered":"2023-10-16T09:59:59.633790+0000","last_clean":"2023-10-16T09:59:59.633790+0000","last_became_active":"2023-10-16T09:59:58.620087+0000","last_became_peered":"2023-10-16T09:59:58.620087+0000","last_unstale":"2023-10-16T09:59:59.633790+0000","last_undegraded":"2023-10-16T09:59:59.633790+0000","last_fullsized":"2023-10-16T09:59:59.633790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:18.871926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635395+0000","last_change":"2023-10-16T09:59:59.628652+0000","last_active":"2023-10-16T09:59:59.635395+0000","last_peered":"2023-10-16T09:59:59.635395+0000","last_clean":"2023-10-16T09:59:59.635395+0000","last_became_active":"2023-10-16T09:59:59.628287+0000","last_became_peered":"2023-10-16T09:59:59.628287+0000","last_unstale":"2023-10-16T09:59:59.635395+0000","last_undegraded":"2023-10-16T09:59:59.635395+0000","last_fullsized":"2023-10-16T09:59:59.635395+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:12.623356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.608121+0000","last_change":"2023-10-16T09:59:58.615206+0000","last_active":"2023-10-16T09:59:59.608121+0000","last_peered":"2023-10-16T09:59:59.608121+0000","last_clean":"2023-10-16T09:59:59.608121+0000","last_became_active":"2023-10-16T09:59:58.614936+0000","last_became_peered":"2023-10-16T09:59:58.614936+0000","last_unstale":"2023-10-16T09:59:59.608121+0000","last_undegraded":"2023-10-16T09:59:59.608121+0000","last_fullsized":"2023-10-16T09:59:59.608121+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:02.421634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613263+0000","last_change":"2023-10-16T09:59:59.627140+0000","last_active":"2023-10-16T10:00:00.613263+0000","last_peered":"2023-10-16T10:00:00.613263+0000","last_clean":"2023-10-16T10:00:00.613263+0000","last_became_active":"2023-10-16T09:59:59.626845+0000","last_became_peered":"2023-10-16T09:59:59.626845+0000","last_unstale":"2023-10-16T10:00:00.613263+0000","last_undegraded":"2023-10-16T10:00:00.613263+0000","last_fullsized":"2023-10-16T10:00:00.613263+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:37:39.047713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613580+0000","last_change":"2023-10-16T09:59:58.614417+0000","last_active":"2023-10-16T10:00:00.613580+0000","last_peered":"2023-10-16T10:00:00.613580+0000","last_clean":"2023-10-16T10:00:00.613580+0000","last_became_active":"2023-10-16T09:59:58.614064+0000","last_became_peered":"2023-10-16T09:59:58.614064+0000","last_unstale":"2023-10-16T10:00:00.613580+0000","last_undegraded":"2023-10-16T10:00:00.613580+0000","last_fullsized":"2023-10-16T10:00:00.613580+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:26.838497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.617548+0000","last_change":"2023-10-16T09:59:59.613088+0000","last_active":"2023-10-16T09:59:59.617548+0000","last_peered":"2023-10-16T09:59:59.617548+0000","last_clean":"2023-10-16T09:59:59.617548+0000","last_became_active":"2023-10-16T09:59:59.612231+0000","last_became_peered":"2023-10-16T09:59:59.612231+0000","last_unstale":"2023-10-16T09:59:59.617548+0000","last_undegraded":"2023-10-16T09:59:59.617548+0000","last_fullsized":"2023-10-16T09:59:59.617548+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:55.809034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613260+0000","last_change":"2023-10-16T09:59:58.620726+0000","last_active":"2023-10-16T10:00:00.613260+0000","last_peered":"2023-10-16T10:00:00.613260+0000","last_clean":"2023-10-16T10:00:00.613260+0000","last_became_active":"2023-10-16T09:59:58.619307+0000","last_became_peered":"2023-10-16T09:59:58.619307+0000","last_unstale":"2023-10-16T10:00:00.613260+0000","last_undegraded":"2023-10-16T10:00:00.613260+0000","last_fullsized":"2023-10-16T10:00:00.613260+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:52.613669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.634502+0000","last_change":"2023-10-16T09:59:59.626531+0000","last_active":"2023-10-16T09:59:59.634502+0000","last_peered":"2023-10-16T09:59:59.634502+0000","last_clean":"2023-10-16T09:59:59.634502+0000","last_became_active":"2023-10-16T09:59:59.626244+0000","last_became_peered":"2023-10-16T09:59:59.626244+0000","last_unstale":"2023-10-16T09:59:59.634502+0000","last_undegraded":"2023-10-16T09:59:59.634502+0000","last_fullsized":"2023-10-16T09:59:59.634502+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:33.142182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.617513+0000","last_change":"2023-10-16T09:59:58.619645+0000","last_active":"2023-10-16T09:59:59.617513+0000","last_peered":"2023-10-16T09:59:59.617513+0000","last_clean":"2023-10-16T09:59:59.617513+0000","last_became_active":"2023-10-16T09:59:58.619375+0000","last_became_peered":"2023-10-16T09:59:58.619375+0000","last_unstale":"2023-10-16T09:59:59.617513+0000","last_undegraded":"2023-10-16T09:59:59.617513+0000","last_fullsized":"2023-10-16T09:59:59.617513+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:52.988880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.649069+0000","last_change":"2023-10-16T09:59:59.643514+0000","last_active":"2023-10-16T09:59:59.649069+0000","last_peered":"2023-10-16T09:59:59.649069+0000","last_clean":"2023-10-16T09:59:59.649069+0000","last_became_active":"2023-10-16T09:59:59.643258+0000","last_became_peered":"2023-10-16T09:59:59.643258+0000","last_unstale":"2023-10-16T09:59:59.649069+0000","last_undegraded":"2023-10-16T09:59:59.649069+0000","last_fullsized":"2023-10-16T09:59:59.649069+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:06:45.905561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.624146+0000","last_change":"2023-10-16T09:59:58.626053+0000","last_active":"2023-10-16T09:59:59.624146+0000","last_peered":"2023-10-16T09:59:59.624146+0000","last_clean":"2023-10-16T09:59:59.624146+0000","last_became_active":"2023-10-16T09:59:58.625769+0000","last_became_peered":"2023-10-16T09:59:58.625769+0000","last_unstale":"2023-10-16T09:59:59.624146+0000","last_undegraded":"2023-10-16T09:59:59.624146+0000","last_fullsized":"2023-10-16T09:59:59.624146+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:32:53.820213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.622601+0000","last_change":"2023-10-16T09:59:59.621193+0000","last_active":"2023-10-16T09:59:59.622601+0000","last_peered":"2023-10-16T09:59:59.622601+0000","last_clean":"2023-10-16T09:59:59.622601+0000","last_became_active":"2023-10-16T09:59:59.620916+0000","last_became_peered":"2023-10-16T09:59:59.620916+0000","last_unstale":"2023-10-16T09:59:59.622601+0000","last_undegraded":"2023-10-16T09:59:59.622601+0000","last_fullsized":"2023-10-16T09:59:59.622601+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:37:39.047713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.622953+0000","last_change":"2023-10-16T09:59:58.624846+0000","last_active":"2023-10-16T09:59:59.622953+0000","last_peered":"2023-10-16T09:59:59.622953+0000","last_clean":"2023-10-16T09:59:59.622953+0000","last_became_active":"2023-10-16T09:59:58.624548+0000","last_became_peered":"2023-10-16T09:59:58.624548+0000","last_unstale":"2023-10-16T09:59:59.622953+0000","last_undegraded":"2023-10-16T09:59:59.622953+0000","last_fullsized":"2023-10-16T09:59:59.622953+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:07:10.368596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.622005+0000","last_change":"2023-10-16T09:59:59.620677+0000","last_active":"2023-10-16T09:59:59.622005+0000","last_peered":"2023-10-16T09:59:59.622005+0000","last_clean":"2023-10-16T09:59:59.622005+0000","last_became_active":"2023-10-16T09:59:59.620393+0000","last_became_peered":"2023-10-16T09:59:59.620393+0000","last_unstale":"2023-10-16T09:59:59.622005+0000","last_undegraded":"2023-10-16T09:59:59.622005+0000","last_fullsized":"2023-10-16T09:59:59.622005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:05.195299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.610991+0000","last_change":"2023-10-16T09:59:58.625330+0000","last_active":"2023-10-16T09:59:59.610991+0000","last_peered":"2023-10-16T09:59:59.610991+0000","last_clean":"2023-10-16T09:59:59.610991+0000","last_became_active":"2023-10-16T09:59:58.625068+0000","last_became_peered":"2023-10-16T09:59:58.625068+0000","last_unstale":"2023-10-16T09:59:59.610991+0000","last_undegraded":"2023-10-16T09:59:59.610991+0000","last_fullsized":"2023-10-16T09:59:59.610991+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:59.803889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613438+0000","last_change":"2023-10-16T09:59:59.622046+0000","last_active":"2023-10-16T10:00:00.613438+0000","last_peered":"2023-10-16T10:00:00.613438+0000","last_clean":"2023-10-16T10:00:00.613438+0000","last_became_active":"2023-10-16T09:59:59.621678+0000","last_became_peered":"2023-10-16T09:59:59.621678+0000","last_unstale":"2023-10-16T10:00:00.613438+0000","last_undegraded":"2023-10-16T10:00:00.613438+0000","last_fullsized":"2023-10-16T10:00:00.613438+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:25.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613408+0000","last_change":"2023-10-16T09:59:58.626614+0000","last_active":"2023-10-16T10:00:00.613408+0000","last_peered":"2023-10-16T10:00:00.613408+0000","last_clean":"2023-10-16T10:00:00.613408+0000","last_became_active":"2023-10-16T09:59:58.625764+0000","last_became_peered":"2023-10-16T09:59:58.625764+0000","last_unstale":"2023-10-16T10:00:00.613408+0000","last_undegraded":"2023-10-16T10:00:00.613408+0000","last_fullsized":"2023-10-16T10:00:00.613408+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:23.950004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'172","reported_seq":438,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.614269+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:59:59.614269+0000","last_peered":"2023-10-16T09:59:59.614269+0000","last_clean":"2023-10-16T09:59:59.614269+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:59:59.614269+0000","last_undegraded":"2023-10-16T09:59:59.614269+0000","last_fullsized":"2023-10-16T09:59:59.614269+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59292,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312548,"statfs":{"total":96636764160,"available":96576049152,"internally_reserved":0,"allocated":1785856,"data_stored":1202202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.56100000000000005},"min":{"1min":0.36499999999999999,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":0.73299999999999998,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.67700000000000005,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.442,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70399999999999996,"5min":0.79300000000000004,"15min":1.1439999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.46500000000000002,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.78100000000000003,"5min":1.1279999999999999,"15min":1.212},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.63100000000000001},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.81599999999999995,"5min":1.607,"15min":1.607},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61399999999999999,"15min":0.63400000000000001},"min":{"1min":0.432,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":2.044,"15min":2.044},"last":0.57499999999999996}]},{"osd":3,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58399999999999996,"15min":0.60099999999999998},"min":{"1min":0.42799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.755,"5min":1.524,"15min":1.524},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.50700000000000001,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.70399999999999996,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.497},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68600000000000005,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":0.54900000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73899999999999999,"5min":0.81100000000000005,"15min":0.875},"last":0.438},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.37,"5min":0.33200000000000002,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.76000000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.438,"5min":0.311,"15min":0.311},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.85299999999999998},"last":0.58899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1200128,"data_stored":613424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55600000000000005,"15min":0.58299999999999996},"min":{"1min":0.29699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.68200000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.498},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57299999999999995,"15min":0.59299999999999997},"min":{"1min":0.38100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.76900000000000002,"5min":0.78800000000000003,"15min":0.82099999999999995},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.36099999999999999,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.72199999999999998,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56399999999999995,"15min":0.58199999999999996},"min":{"1min":0.33100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.749,"5min":1.448,"15min":1.448},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":1.708,"15min":1.708},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.76000000000000001,"15min":0.81699999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.82999999999999996},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.83999999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.81100000000000005},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.253,"15min":0.253},"max":{"1min":0.81100000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53700000000000003,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.872},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.79100000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1196032,"data_stored":611924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.38800000000000001,"15min":0.39500000000000002},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":0.52200000000000002,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.51000000000000001,"5min":0.67900000000000005,"15min":0.70199999999999996},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.43099999999999999,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54000000000000004,"5min":4.923,"15min":4.923},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.216},"max":{"1min":0.58699999999999997,"5min":1.712,"15min":1.712},"last":0.26000000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46400000000000002,"15min":0.45600000000000002},"min":{"1min":0.32100000000000001,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.66500000000000004,"5min":1.984,"15min":1.984},"last":0.313},{"interface":"front","average":{"1min":0.37,"5min":0.41999999999999998,"15min":0.42499999999999999},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.56699999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61499999999999999,"15min":0.628},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.45200000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67700000000000005,"15min":0.66500000000000004},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.85899999999999999,"5min":4.9749999999999996,"15min":4.9749999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77400000000000002,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.54600000000000004}]},{"osd":6,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.80700000000000005,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.68100000000000005,"15min":0.65700000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.82899999999999996,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.45800000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.75900000000000001,"5min":1.744,"15min":1.744},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38500000000000001},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.93400000000000005},"last":0.61099999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575049,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":1202180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39200000000000002,"15min":0.38300000000000001},"min":{"1min":0.27100000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":0.55600000000000005,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.318}]},{"osd":2,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53000000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.34799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.63600000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.373}]},{"osd":3,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.224},"max":{"1min":0.57899999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46400000000000002},"min":{"1min":0.27700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59699999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.437,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.77300000000000002,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":0.81100000000000005,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.71999999999999997}]},{"osd":5,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61199999999999999,"15min":0.63500000000000001},"min":{"1min":0.39200000000000002,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.94999999999999996,"15min":0.998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.82599999999999996,"5min":0.88700000000000001,"15min":0.93000000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84999999999999998,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.747}]},{"osd":7,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":1.014,"15min":1.02},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58728,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313112,"statfs":{"total":96636764160,"available":96576626688,"internally_reserved":0,"allocated":1200128,"data_stored":612554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.51400000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.246},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.53800000000000003,"5min":0.64300000000000002,"15min":0.67500000000000004},"last":0.314}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.61899999999999999,"5min":0.68000000000000005,"15min":0.754},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40799999999999997,"15min":0.41999999999999998},"min":{"1min":0.29999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64800000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.35299999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.64500000000000002},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.103},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.65700000000000003},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.434}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63100000000000001,"15min":0.64900000000000002},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.046,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65700000000000003,"15min":0.68300000000000005},"min":{"1min":0.441,"5min":0.40799999999999997,"15min":0.40100000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65800000000000003,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.43099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.0069999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36520,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335320,"statfs":{"total":96636764160,"available":96599367680,"internally_reserved":0,"allocated":1204224,"data_stored":619438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.20999999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.34100000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.39400000000000002,"15min":0.39500000000000002},"min":{"1min":0.24099999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.65400000000000003},"last":0.255},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":0.56799999999999995,"5min":0.61899999999999999,"15min":0.67500000000000004},"last":0.309}]},{"osd":2,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":0.59999999999999998,"5min":0.70499999999999996,"15min":1.0760000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.64100000000000001,"5min":0.73999999999999999,"15min":0.93799999999999994},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.63500000000000001,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89000000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33200000000000002},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":1.036},"last":0.71899999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64100000000000001,"15min":0.65000000000000002},"min":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.374},"max":{"1min":0.83299999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":1.0589999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42599999999999999},"max":{"1min":0.872,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.42699999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":0.999},"last":0.79100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575048,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1208320,"data_stored":613528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.59199999999999997},"min":{"1min":0.34200000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.83099999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.38}]},{"osd":1,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.72299999999999998,"5min":0.81499999999999995,"15min":0.82599999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.56200000000000006,"15min":0.58999999999999997},"min":{"1min":0.33300000000000002,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.66800000000000004,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.499},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.622},"min":{"1min":0.438,"5min":0.379,"15min":0.32500000000000001},"max":{"1min":0.79600000000000004,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75800000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58299999999999996,"15min":0.62},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.84599999999999997},"last":0.626}]},{"osd":5,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.749,"5min":0.76400000000000001,"15min":0.84299999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.77500000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.34100000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.84699999999999998,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80100000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.48199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80800000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.66800000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46240,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325600,"statfs":{"total":96636764160,"available":96589414400,"internally_reserved":0,"allocated":1191936,"data_stored":611846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":0.80600000000000005,"5min":1.792,"15min":1.792},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45700000000000002,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.83099999999999996,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.439}]},{"osd":2,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.57299999999999995},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.496,"15min":1.496},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.72099999999999997,"5min":2.069,"15min":2.069},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46700000000000003,"15min":0.47799999999999998},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66100000000000003,"5min":0.71299999999999997,"15min":0.84199999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.86899999999999999},"last":0.374}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48699999999999999,"15min":0.48999999999999999},"min":{"1min":0.312,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.84099999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.88200000000000001},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80500000000000005,"5min":0.81799999999999995,"15min":0.90300000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.79500000000000004,"5min":0.84899999999999998,"15min":0.88500000000000001},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:00:03.743 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:00:03.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.896+0000 7f3060380700 1 Processor -- start 2023-10-16T10:00:03.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.896+0000 7f3060380700 1 -- start start 2023-10-16T10:00:03.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f3060380700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f305806ab40 0x7f305806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f3060380700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f305806e5d0 0x7f3058071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f3060380700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806af20 0x7f30580722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f3060380700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3058071e50 con 0x7f305806ab40 2023-10-16T10:00:03.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f3060380700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3058161270 con 0x7f305806e5d0 2023-10-16T10:00:03.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f3060380700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3058161510 con 0x7f305806af20 2023-10-16T10:00:03.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f305806ab40 0x7f305806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f305806ab40 0x7f305806d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53828/0 (socket says 172.21.15.158:53828) 2023-10-16T10:00:03.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 -- 172.21.15.158:0/1607365810 learned_addr learned my addr 172.21.15.158:0/1607365810 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:03.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305d91b700 1 --2- 172.21.15.158:0/1607365810 >> v2:172.21.15.172:3300/0 conn(0x7f305806e5d0 0x7f3058071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e91d700 1 --2- 172.21.15.158:0/1607365810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806af20 0x7f30580722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 -- 172.21.15.158:0/1607365810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806af20 msgr2=0x7f30580722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:03.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 --2- 172.21.15.158:0/1607365810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806af20 0x7f30580722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 -- 172.21.15.158:0/1607365810 >> v2:172.21.15.172:3300/0 conn(0x7f305806e5d0 msgr2=0x7f3058071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:03.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 --2- 172.21.15.158:0/1607365810 >> v2:172.21.15.172:3300/0 conn(0x7f305806e5d0 0x7f3058071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 -- 172.21.15.158:0/1607365810 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30581617f0 con 0x7f305806ab40 2023-10-16T10:00:03.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e91d700 1 --2- 172.21.15.158:0/1607365810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806af20 0x7f30580722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:03.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305e11c700 1 --2- 172.21.15.158:0/1607365810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806ab40 0x7f305806d820 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f30580c3bf0 tx=0x7f3054009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bdb17aeca73d439 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.897+0000 7f305d11a700 1 -- 172.21.15.158:0/1607365810 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400b870 con 0x7f305806ab40 2023-10-16T10:00:03.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f305d11a700 1 -- 172.21.15.158:0/1607365810 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305400b9d0 con 0x7f305806ab40 2023-10-16T10:00:03.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f305d11a700 1 -- 172.21.15.158:0/1607365810 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400bd60 con 0x7f305806ab40 2023-10-16T10:00:03.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 -- 172.21.15.158:0/1607365810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806ab40 msgr2=0x7f305806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 --2- 172.21.15.158:0/1607365810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806ab40 0x7f305806d820 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f30580c3bf0 tx=0x7f3054009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 -- 172.21.15.158:0/1607365810 shutdown_connections 2023-10-16T10:00:03.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 --2- 172.21.15.158:0/1607365810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806af20 0x7f30580722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 --2- 172.21.15.158:0/1607365810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806ab40 0x7f305806d820 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 --2- 172.21.15.158:0/1607365810 >> v2:172.21.15.172:3300/0 conn(0x7f305806e5d0 0x7f3058071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 -- 172.21.15.158:0/1607365810 >> 172.21.15.158:0/1607365810 conn(0x7f30580d02b0 msgr2=0x7f30580d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:03.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 -- 172.21.15.158:0/1607365810 shutdown_connections 2023-10-16T10:00:03.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.898+0000 7f3060380700 1 -- 172.21.15.158:0/1607365810 wait complete. 2023-10-16T10:00:03.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.899+0000 7f3060380700 1 Processor -- start 2023-10-16T10:00:03.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.899+0000 7f3060380700 1 -- start start 2023-10-16T10:00:03.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f3060380700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806ab40 0x7f3058145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f3060380700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f305806af20 0x7f30580c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f3060380700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806e5d0 0x7f30580b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f3060380700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058143270 con 0x7f305806e5d0 2023-10-16T10:00:03.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f3060380700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581541f0 con 0x7f305806af20 2023-10-16T10:00:03.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f3060380700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30580c64b0 con 0x7f305806ab40 2023-10-16T10:00:03.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806e5d0 0x7f30580b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806e5d0 0x7f30580b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53838/0 (socket says 172.21.15.158:53838) 2023-10-16T10:00:03.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 -- 172.21.15.158:0/1991708142 learned_addr learned my addr 172.21.15.158:0/1991708142 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:03.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e11c700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806ab40 0x7f3058145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305d91b700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f305806af20 0x7f30580c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 -- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806ab40 msgr2=0x7f3058145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806ab40 0x7f3058145ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 -- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f305806af20 msgr2=0x7f30580c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:03.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f305806af20 0x7f30580c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:03.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.900+0000 7f305e91d700 1 -- 172.21.15.158:0/1991708142 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3054009550 con 0x7f305806e5d0 2023-10-16T10:00:03.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.901+0000 7f305e91d700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806e5d0 0x7f30580b1b80 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f305000cef0 tx=0x7f305000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.901+0000 7f304effd700 1 -- 172.21.15.158:0/1991708142 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305000ad00 con 0x7f305806e5d0 2023-10-16T10:00:03.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.901+0000 7f304effd700 1 -- 172.21.15.158:0/1991708142 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305000ae60 con 0x7f305806e5d0 2023-10-16T10:00:03.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.901+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30580c6750 con 0x7f305806e5d0 2023-10-16T10:00:03.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.901+0000 7f304effd700 1 -- 172.21.15.158:0/1991708142 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050015680 con 0x7f305806e5d0 2023-10-16T10:00:03.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.901+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3058154440 con 0x7f305806e5d0 2023-10-16T10:00:03.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.902+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30580681b0 con 0x7f305806e5d0 2023-10-16T10:00:03.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.903+0000 7f304effd700 1 -- 172.21.15.158:0/1991708142 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f305000d430 con 0x7f305806e5d0 2023-10-16T10:00:03.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.903+0000 7f304effd700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f303c06b020 0x7f303c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:03.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.903+0000 7f304effd700 1 -- 172.21.15.158:0/1991708142 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f305008a3e0 con 0x7f305806e5d0 2023-10-16T10:00:03.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.903+0000 7f305e11c700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f303c06b020 0x7f303c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:03.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.904+0000 7f305e11c700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f303c06b020 0x7f303c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3054000c00 tx=0x7f3054008030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:03.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:03.907+0000 7f304effd700 1 -- 172.21.15.158:0/1991708142 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3050055070 con 0x7f305806e5d0 2023-10-16T10:00:04.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.095+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 --> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3058161db0 con 0x7f303c06b020 2023-10-16T10:00:04.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.103+0000 7f304effd700 1 -- 172.21.15.158:0/1991708142 <== mgr.7005 v2:172.21.15.158:6832/3798429240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289533 (secure 0 0 0) 0x7f305400bb40 con 0x7f303c06b020 2023-10-16T10:00:04.108 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:04.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f303c06b020 msgr2=0x7f303c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f303c06b020 0x7f303c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3054000c00 tx=0x7f3054008030 comp rx=0 tx=0).stop 2023-10-16T10:00:04.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806e5d0 msgr2=0x7f30580b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806e5d0 0x7f30580b1b80 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f305000cef0 tx=0x7f305000cf20 comp rx=0 tx=0).stop 2023-10-16T10:00:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 shutdown_connections 2023-10-16T10:00:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f303c06b020 0x7f303c06d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f305806e5d0 0x7f30580b1b80 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f305806ab40 0x7f3058145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 --2- 172.21.15.158:0/1991708142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f305806af20 0x7f30580c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 >> 172.21.15.158:0/1991708142 conn(0x7f30580d02b0 msgr2=0x7f3058156640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 shutdown_connections 2023-10-16T10:00:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.107+0000 7f3060380700 1 -- 172.21.15.158:0/1991708142 wait complete. 2023-10-16T10:00:04.113 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:00:04.120 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T10:00:03.222129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330780,"kb_used_data":10516,"kb_used_omap":12,"kb_used_meta":320115,"kb_avail":754643940,"statfs":{"total":773094113280,"available":772755394560,"internally_reserved":0,"allocated":10768384,"data_stored":6087096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":327798327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.626345"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613455+0000","last_change":"2023-10-16T09:59:59.626224+0000","last_active":"2023-10-16T10:00:00.613455+0000","last_peered":"2023-10-16T10:00:00.613455+0000","last_clean":"2023-10-16T10:00:00.613455+0000","last_became_active":"2023-10-16T09:59:59.625938+0000","last_became_peered":"2023-10-16T09:59:59.625938+0000","last_unstale":"2023-10-16T10:00:00.613455+0000","last_undegraded":"2023-10-16T10:00:00.613455+0000","last_fullsized":"2023-10-16T10:00:00.613455+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:26.228687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.637953+0000","last_change":"2023-10-16T09:59:59.636815+0000","last_active":"2023-10-16T09:59:59.637953+0000","last_peered":"2023-10-16T09:59:59.637953+0000","last_clean":"2023-10-16T09:59:59.637953+0000","last_became_active":"2023-10-16T09:59:59.636532+0000","last_became_peered":"2023-10-16T09:59:59.636532+0000","last_unstale":"2023-10-16T09:59:59.637953+0000","last_undegraded":"2023-10-16T09:59:59.637953+0000","last_fullsized":"2023-10-16T09:59:59.637953+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:27:44.943360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632844+0000","last_change":"2023-10-16T09:59:59.628243+0000","last_active":"2023-10-16T09:59:59.632844+0000","last_peered":"2023-10-16T09:59:59.632844+0000","last_clean":"2023-10-16T09:59:59.632844+0000","last_became_active":"2023-10-16T09:59:59.627440+0000","last_became_peered":"2023-10-16T09:59:59.627440+0000","last_unstale":"2023-10-16T09:59:59.632844+0000","last_undegraded":"2023-10-16T09:59:59.632844+0000","last_fullsized":"2023-10-16T09:59:59.632844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:21:11.435556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632839+0000","last_change":"2023-10-16T09:59:59.625959+0000","last_active":"2023-10-16T09:59:59.632839+0000","last_peered":"2023-10-16T09:59:59.632839+0000","last_clean":"2023-10-16T09:59:59.632839+0000","last_became_active":"2023-10-16T09:59:59.625662+0000","last_became_peered":"2023-10-16T09:59:59.625662+0000","last_unstale":"2023-10-16T09:59:59.632839+0000","last_undegraded":"2023-10-16T09:59:59.632839+0000","last_fullsized":"2023-10-16T09:59:59.632839+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:24.242736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.642533+0000","last_change":"2023-10-16T09:59:59.640781+0000","last_active":"2023-10-16T09:59:59.642533+0000","last_peered":"2023-10-16T09:59:59.642533+0000","last_clean":"2023-10-16T09:59:59.642533+0000","last_became_active":"2023-10-16T09:59:59.640466+0000","last_became_peered":"2023-10-16T09:59:59.640466+0000","last_unstale":"2023-10-16T09:59:59.642533+0000","last_undegraded":"2023-10-16T09:59:59.642533+0000","last_fullsized":"2023-10-16T09:59:59.642533+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:38.269611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613646+0000","last_change":"2023-10-16T09:59:59.626317+0000","last_active":"2023-10-16T10:00:00.613646+0000","last_peered":"2023-10-16T10:00:00.613646+0000","last_clean":"2023-10-16T10:00:00.613646+0000","last_became_active":"2023-10-16T09:59:59.625993+0000","last_became_peered":"2023-10-16T09:59:59.625993+0000","last_unstale":"2023-10-16T10:00:00.613646+0000","last_undegraded":"2023-10-16T10:00:00.613646+0000","last_fullsized":"2023-10-16T10:00:00.613646+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:06.124021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643508+0000","last_change":"2023-10-16T09:59:59.638874+0000","last_active":"2023-10-16T09:59:59.643508+0000","last_peered":"2023-10-16T09:59:59.643508+0000","last_clean":"2023-10-16T09:59:59.643508+0000","last_became_active":"2023-10-16T09:59:59.638614+0000","last_became_peered":"2023-10-16T09:59:59.638614+0000","last_unstale":"2023-10-16T09:59:59.643508+0000","last_undegraded":"2023-10-16T09:59:59.643508+0000","last_fullsized":"2023-10-16T09:59:59.643508+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:18.720561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635827+0000","last_change":"2023-10-16T09:59:59.623655+0000","last_active":"2023-10-16T09:59:59.635827+0000","last_peered":"2023-10-16T09:59:59.635827+0000","last_clean":"2023-10-16T09:59:59.635827+0000","last_became_active":"2023-10-16T09:59:59.623378+0000","last_became_peered":"2023-10-16T09:59:59.623378+0000","last_unstale":"2023-10-16T09:59:59.635827+0000","last_undegraded":"2023-10-16T09:59:59.635827+0000","last_fullsized":"2023-10-16T09:59:59.635827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:36:49.047683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613266+0000","last_change":"2023-10-16T09:59:59.625862+0000","last_active":"2023-10-16T10:00:00.613266+0000","last_peered":"2023-10-16T10:00:00.613266+0000","last_clean":"2023-10-16T10:00:00.613266+0000","last_became_active":"2023-10-16T09:59:59.625534+0000","last_became_peered":"2023-10-16T09:59:59.625534+0000","last_unstale":"2023-10-16T10:00:00.613266+0000","last_undegraded":"2023-10-16T10:00:00.613266+0000","last_fullsized":"2023-10-16T10:00:00.613266+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:21:11.435556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.631202+0000","last_change":"2023-10-16T09:59:59.623181+0000","last_active":"2023-10-16T09:59:59.631202+0000","last_peered":"2023-10-16T09:59:59.631202+0000","last_clean":"2023-10-16T09:59:59.631202+0000","last_became_active":"2023-10-16T09:59:59.622882+0000","last_became_peered":"2023-10-16T09:59:59.622882+0000","last_unstale":"2023-10-16T09:59:59.631202+0000","last_undegraded":"2023-10-16T09:59:59.631202+0000","last_fullsized":"2023-10-16T09:59:59.631202+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:04:54.785665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.637965+0000","last_change":"2023-10-16T09:59:59.636956+0000","last_active":"2023-10-16T09:59:59.637965+0000","last_peered":"2023-10-16T09:59:59.637965+0000","last_clean":"2023-10-16T09:59:59.637965+0000","last_became_active":"2023-10-16T09:59:59.636658+0000","last_became_peered":"2023-10-16T09:59:59.636658+0000","last_unstale":"2023-10-16T09:59:59.637965+0000","last_undegraded":"2023-10-16T09:59:59.637965+0000","last_fullsized":"2023-10-16T09:59:59.637965+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:54.407322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632858+0000","last_change":"2023-10-16T09:59:59.628664+0000","last_active":"2023-10-16T09:59:59.632858+0000","last_peered":"2023-10-16T09:59:59.632858+0000","last_clean":"2023-10-16T09:59:59.632858+0000","last_became_active":"2023-10-16T09:59:59.628375+0000","last_became_peered":"2023-10-16T09:59:59.628375+0000","last_unstale":"2023-10-16T09:59:59.632858+0000","last_undegraded":"2023-10-16T09:59:59.632858+0000","last_fullsized":"2023-10-16T09:59:59.632858+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:17.169481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.642554+0000","last_change":"2023-10-16T09:59:59.640996+0000","last_active":"2023-10-16T09:59:59.642554+0000","last_peered":"2023-10-16T09:59:59.642554+0000","last_clean":"2023-10-16T09:59:59.642554+0000","last_became_active":"2023-10-16T09:59:59.635548+0000","last_became_peered":"2023-10-16T09:59:59.635548+0000","last_unstale":"2023-10-16T09:59:59.642554+0000","last_undegraded":"2023-10-16T09:59:59.642554+0000","last_fullsized":"2023-10-16T09:59:59.642554+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:37:38.036445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613599+0000","last_change":"2023-10-16T09:59:59.629345+0000","last_active":"2023-10-16T10:00:00.613599+0000","last_peered":"2023-10-16T10:00:00.613599+0000","last_clean":"2023-10-16T10:00:00.613599+0000","last_became_active":"2023-10-16T09:59:59.627694+0000","last_became_peered":"2023-10-16T09:59:59.627694+0000","last_unstale":"2023-10-16T10:00:00.613599+0000","last_undegraded":"2023-10-16T10:00:00.613599+0000","last_fullsized":"2023-10-16T10:00:00.613599+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:13.600302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.628323+0000","last_change":"2023-10-16T09:59:59.622542+0000","last_active":"2023-10-16T09:59:59.628323+0000","last_peered":"2023-10-16T09:59:59.628323+0000","last_clean":"2023-10-16T09:59:59.628323+0000","last_became_active":"2023-10-16T09:59:59.622265+0000","last_became_peered":"2023-10-16T09:59:59.622265+0000","last_unstale":"2023-10-16T09:59:59.628323+0000","last_undegraded":"2023-10-16T09:59:59.628323+0000","last_fullsized":"2023-10-16T09:59:59.628323+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:06.247261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612452+0000","last_change":"2023-10-16T09:59:59.617173+0000","last_active":"2023-10-16T10:00:00.612452+0000","last_peered":"2023-10-16T10:00:00.612452+0000","last_clean":"2023-10-16T10:00:00.612452+0000","last_became_active":"2023-10-16T09:59:59.616865+0000","last_became_peered":"2023-10-16T09:59:59.616865+0000","last_unstale":"2023-10-16T10:00:00.612452+0000","last_undegraded":"2023-10-16T10:00:00.612452+0000","last_fullsized":"2023-10-16T10:00:00.612452+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:24.957452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.640647+0000","last_change":"2023-10-16T09:59:59.635993+0000","last_active":"2023-10-16T09:59:59.640647+0000","last_peered":"2023-10-16T09:59:59.640647+0000","last_clean":"2023-10-16T09:59:59.640647+0000","last_became_active":"2023-10-16T09:59:59.635702+0000","last_became_peered":"2023-10-16T09:59:59.635702+0000","last_unstale":"2023-10-16T09:59:59.640647+0000","last_undegraded":"2023-10-16T09:59:59.640647+0000","last_fullsized":"2023-10-16T09:59:59.640647+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:54.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632856+0000","last_change":"2023-10-16T09:59:59.628141+0000","last_active":"2023-10-16T09:59:59.632856+0000","last_peered":"2023-10-16T09:59:59.632856+0000","last_clean":"2023-10-16T09:59:59.632856+0000","last_became_active":"2023-10-16T09:59:59.627363+0000","last_became_peered":"2023-10-16T09:59:59.627363+0000","last_unstale":"2023-10-16T09:59:59.632856+0000","last_undegraded":"2023-10-16T09:59:59.632856+0000","last_fullsized":"2023-10-16T09:59:59.632856+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:48:02.115525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613711+0000","last_change":"2023-10-16T09:59:59.638835+0000","last_active":"2023-10-16T10:00:00.613711+0000","last_peered":"2023-10-16T10:00:00.613711+0000","last_clean":"2023-10-16T10:00:00.613711+0000","last_became_active":"2023-10-16T09:59:59.638586+0000","last_became_peered":"2023-10-16T09:59:59.638586+0000","last_unstale":"2023-10-16T10:00:00.613711+0000","last_undegraded":"2023-10-16T10:00:00.613711+0000","last_fullsized":"2023-10-16T10:00:00.613711+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:09.065430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.640001+0000","last_change":"2023-10-16T09:59:59.635957+0000","last_active":"2023-10-16T09:59:59.640001+0000","last_peered":"2023-10-16T09:59:59.640001+0000","last_clean":"2023-10-16T09:59:59.640001+0000","last_became_active":"2023-10-16T09:59:59.635666+0000","last_became_peered":"2023-10-16T09:59:59.635666+0000","last_unstale":"2023-10-16T09:59:59.640001+0000","last_undegraded":"2023-10-16T09:59:59.640001+0000","last_fullsized":"2023-10-16T09:59:59.640001+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:53.868228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612907+0000","last_change":"2023-10-16T09:59:59.625613+0000","last_active":"2023-10-16T10:00:00.612907+0000","last_peered":"2023-10-16T10:00:00.612907+0000","last_clean":"2023-10-16T10:00:00.612907+0000","last_became_active":"2023-10-16T09:59:59.625188+0000","last_became_peered":"2023-10-16T09:59:59.625188+0000","last_unstale":"2023-10-16T10:00:00.612907+0000","last_undegraded":"2023-10-16T10:00:00.612907+0000","last_fullsized":"2023-10-16T10:00:00.612907+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:48.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.639859+0000","last_change":"2023-10-16T09:59:59.635478+0000","last_active":"2023-10-16T09:59:59.639859+0000","last_peered":"2023-10-16T09:59:59.639859+0000","last_clean":"2023-10-16T09:59:59.639859+0000","last_became_active":"2023-10-16T09:59:59.635154+0000","last_became_peered":"2023-10-16T09:59:59.635154+0000","last_unstale":"2023-10-16T09:59:59.639859+0000","last_undegraded":"2023-10-16T09:59:59.639859+0000","last_fullsized":"2023-10-16T09:59:59.639859+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:55.077197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613388+0000","last_change":"2023-10-16T09:59:59.622128+0000","last_active":"2023-10-16T10:00:00.613388+0000","last_peered":"2023-10-16T10:00:00.613388+0000","last_clean":"2023-10-16T10:00:00.613388+0000","last_became_active":"2023-10-16T09:59:59.621868+0000","last_became_peered":"2023-10-16T09:59:59.621868+0000","last_unstale":"2023-10-16T10:00:00.613388+0000","last_undegraded":"2023-10-16T10:00:00.613388+0000","last_fullsized":"2023-10-16T10:00:00.613388+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:07:45.973592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626318+0000","last_change":"2023-10-16T09:59:59.620760+0000","last_active":"2023-10-16T09:59:59.626318+0000","last_peered":"2023-10-16T09:59:59.626318+0000","last_clean":"2023-10-16T09:59:59.626318+0000","last_became_active":"2023-10-16T09:59:59.620465+0000","last_became_peered":"2023-10-16T09:59:59.620465+0000","last_unstale":"2023-10-16T09:59:59.626318+0000","last_undegraded":"2023-10-16T09:59:59.626318+0000","last_fullsized":"2023-10-16T09:59:59.626318+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:27.946882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626526+0000","last_change":"2023-10-16T09:59:59.622906+0000","last_active":"2023-10-16T09:59:59.626526+0000","last_peered":"2023-10-16T09:59:59.626526+0000","last_clean":"2023-10-16T09:59:59.626526+0000","last_became_active":"2023-10-16T09:59:59.622623+0000","last_became_peered":"2023-10-16T09:59:59.622623+0000","last_unstale":"2023-10-16T09:59:59.626526+0000","last_undegraded":"2023-10-16T09:59:59.626526+0000","last_fullsized":"2023-10-16T09:59:59.626526+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:51.904350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.633250+0000","last_change":"2023-10-16T09:59:59.628714+0000","last_active":"2023-10-16T09:59:59.633250+0000","last_peered":"2023-10-16T09:59:59.633250+0000","last_clean":"2023-10-16T09:59:59.633250+0000","last_became_active":"2023-10-16T09:59:59.628424+0000","last_became_peered":"2023-10-16T09:59:59.628424+0000","last_unstale":"2023-10-16T09:59:59.633250+0000","last_undegraded":"2023-10-16T09:59:59.633250+0000","last_fullsized":"2023-10-16T09:59:59.633250+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:53.621117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613448+0000","last_change":"2023-10-16T09:59:59.619292+0000","last_active":"2023-10-16T10:00:00.613448+0000","last_peered":"2023-10-16T10:00:00.613448+0000","last_clean":"2023-10-16T10:00:00.613448+0000","last_became_active":"2023-10-16T09:59:59.618911+0000","last_became_peered":"2023-10-16T09:59:59.618911+0000","last_unstale":"2023-10-16T10:00:00.613448+0000","last_undegraded":"2023-10-16T10:00:00.613448+0000","last_fullsized":"2023-10-16T10:00:00.613448+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:13.600302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.638136+0000","last_change":"2023-10-16T09:59:59.616080+0000","last_active":"2023-10-16T09:59:59.638136+0000","last_peered":"2023-10-16T09:59:59.638136+0000","last_clean":"2023-10-16T09:59:59.638136+0000","last_became_active":"2023-10-16T09:59:59.615698+0000","last_became_peered":"2023-10-16T09:59:59.615698+0000","last_unstale":"2023-10-16T09:59:59.638136+0000","last_undegraded":"2023-10-16T09:59:59.638136+0000","last_fullsized":"2023-10-16T09:59:59.638136+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:41:21.351163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626438+0000","last_change":"2023-10-16T09:59:59.623109+0000","last_active":"2023-10-16T09:59:59.626438+0000","last_peered":"2023-10-16T09:59:59.626438+0000","last_clean":"2023-10-16T09:59:59.626438+0000","last_became_active":"2023-10-16T09:59:59.622891+0000","last_became_peered":"2023-10-16T09:59:59.622891+0000","last_unstale":"2023-10-16T09:59:59.626438+0000","last_undegraded":"2023-10-16T09:59:59.626438+0000","last_fullsized":"2023-10-16T09:59:59.626438+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:50.955432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613036+0000","last_change":"2023-10-16T09:59:59.623095+0000","last_active":"2023-10-16T10:00:00.613036+0000","last_peered":"2023-10-16T10:00:00.613036+0000","last_clean":"2023-10-16T10:00:00.613036+0000","last_became_active":"2023-10-16T09:59:59.621647+0000","last_became_peered":"2023-10-16T09:59:59.621647+0000","last_unstale":"2023-10-16T10:00:00.613036+0000","last_undegraded":"2023-10-16T10:00:00.613036+0000","last_fullsized":"2023-10-16T10:00:00.613036+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:44.249052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632920+0000","last_change":"2023-10-16T09:59:59.628025+0000","last_active":"2023-10-16T09:59:59.632920+0000","last_peered":"2023-10-16T09:59:59.632920+0000","last_clean":"2023-10-16T09:59:59.632920+0000","last_became_active":"2023-10-16T09:59:59.627380+0000","last_became_peered":"2023-10-16T09:59:59.627380+0000","last_unstale":"2023-10-16T09:59:59.632920+0000","last_undegraded":"2023-10-16T09:59:59.632920+0000","last_fullsized":"2023-10-16T09:59:59.632920+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:07:11.376044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612468+0000","last_change":"2023-10-16T09:59:59.627078+0000","last_active":"2023-10-16T10:00:00.612468+0000","last_peered":"2023-10-16T10:00:00.612468+0000","last_clean":"2023-10-16T10:00:00.612468+0000","last_became_active":"2023-10-16T09:59:59.626626+0000","last_became_peered":"2023-10-16T09:59:59.626626+0000","last_unstale":"2023-10-16T10:00:00.612468+0000","last_undegraded":"2023-10-16T10:00:00.612468+0000","last_fullsized":"2023-10-16T10:00:00.612468+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:17.660472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632107+0000","last_change":"2023-10-16T09:59:59.627011+0000","last_active":"2023-10-16T09:59:59.632107+0000","last_peered":"2023-10-16T09:59:59.632107+0000","last_clean":"2023-10-16T09:59:59.632107+0000","last_became_active":"2023-10-16T09:59:59.626677+0000","last_became_peered":"2023-10-16T09:59:59.626677+0000","last_unstale":"2023-10-16T09:59:59.632107+0000","last_undegraded":"2023-10-16T09:59:59.632107+0000","last_fullsized":"2023-10-16T09:59:59.632107+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:18.720561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"97'4","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.644209+0000","last_change":"2023-10-16T09:59:58.624762+0000","last_active":"2023-10-16T09:59:59.644209+0000","last_peered":"2023-10-16T09:59:59.644209+0000","last_clean":"2023-10-16T09:59:59.644209+0000","last_became_active":"2023-10-16T09:59:58.624504+0000","last_became_peered":"2023-10-16T09:59:58.624504+0000","last_unstale":"2023-10-16T09:59:59.644209+0000","last_undegraded":"2023-10-16T09:59:59.644209+0000","last_fullsized":"2023-10-16T09:59:59.644209+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:34.766524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.617970+0000","last_change":"2023-10-16T09:59:59.613077+0000","last_active":"2023-10-16T09:59:59.617970+0000","last_peered":"2023-10-16T09:59:59.617970+0000","last_clean":"2023-10-16T09:59:59.617970+0000","last_became_active":"2023-10-16T09:59:59.612786+0000","last_became_peered":"2023-10-16T09:59:59.612786+0000","last_unstale":"2023-10-16T09:59:59.617970+0000","last_undegraded":"2023-10-16T09:59:59.617970+0000","last_fullsized":"2023-10-16T09:59:59.617970+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:38.491006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"97'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613516+0000","last_change":"2023-10-16T09:59:58.620958+0000","last_active":"2023-10-16T10:00:00.613516+0000","last_peered":"2023-10-16T10:00:00.613516+0000","last_clean":"2023-10-16T10:00:00.613516+0000","last_became_active":"2023-10-16T09:59:58.620551+0000","last_became_peered":"2023-10-16T09:59:58.620551+0000","last_unstale":"2023-10-16T10:00:00.613516+0000","last_undegraded":"2023-10-16T10:00:00.613516+0000","last_fullsized":"2023-10-16T10:00:00.613516+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:16.162032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612948+0000","last_change":"2023-10-16T09:59:59.626738+0000","last_active":"2023-10-16T10:00:00.612948+0000","last_peered":"2023-10-16T10:00:00.612948+0000","last_clean":"2023-10-16T10:00:00.612948+0000","last_became_active":"2023-10-16T09:59:59.626411+0000","last_became_peered":"2023-10-16T09:59:59.626411+0000","last_unstale":"2023-10-16T10:00:00.612948+0000","last_undegraded":"2023-10-16T10:00:00.612948+0000","last_fullsized":"2023-10-16T10:00:00.612948+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:12.623356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"97'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613558+0000","last_change":"2023-10-16T09:59:58.617465+0000","last_active":"2023-10-16T10:00:00.613558+0000","last_peered":"2023-10-16T10:00:00.613558+0000","last_clean":"2023-10-16T10:00:00.613558+0000","last_became_active":"2023-10-16T09:59:58.617155+0000","last_became_peered":"2023-10-16T09:59:58.617155+0000","last_unstale":"2023-10-16T10:00:00.613558+0000","last_undegraded":"2023-10-16T10:00:00.613558+0000","last_fullsized":"2023-10-16T10:00:00.613558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:55.612848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613121+0000","last_change":"2023-10-16T09:59:59.628842+0000","last_active":"2023-10-16T10:00:00.613121+0000","last_peered":"2023-10-16T10:00:00.613121+0000","last_clean":"2023-10-16T10:00:00.613121+0000","last_became_active":"2023-10-16T09:59:59.626453+0000","last_became_peered":"2023-10-16T09:59:59.626453+0000","last_unstale":"2023-10-16T10:00:00.613121+0000","last_undegraded":"2023-10-16T10:00:00.613121+0000","last_fullsized":"2023-10-16T10:00:00.613121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:07:11.376044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"97'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635450+0000","last_change":"2023-10-16T09:59:58.618372+0000","last_active":"2023-10-16T09:59:59.635450+0000","last_peered":"2023-10-16T09:59:59.635450+0000","last_clean":"2023-10-16T09:59:59.635450+0000","last_became_active":"2023-10-16T09:59:58.618045+0000","last_became_peered":"2023-10-16T09:59:58.618045+0000","last_unstale":"2023-10-16T09:59:59.635450+0000","last_undegraded":"2023-10-16T09:59:59.635450+0000","last_fullsized":"2023-10-16T09:59:59.635450+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:16.530500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.645063+0000","last_change":"2023-10-16T09:59:59.644631+0000","last_active":"2023-10-16T09:59:59.645063+0000","last_peered":"2023-10-16T09:59:59.645063+0000","last_clean":"2023-10-16T09:59:59.645063+0000","last_became_active":"2023-10-16T09:59:59.644272+0000","last_became_peered":"2023-10-16T09:59:59.644272+0000","last_unstale":"2023-10-16T09:59:59.645063+0000","last_undegraded":"2023-10-16T09:59:59.645063+0000","last_fullsized":"2023-10-16T09:59:59.645063+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:09.744805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"97'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613016+0000","last_change":"2023-10-16T09:59:58.620864+0000","last_active":"2023-10-16T10:00:00.613016+0000","last_peered":"2023-10-16T10:00:00.613016+0000","last_clean":"2023-10-16T10:00:00.613016+0000","last_became_active":"2023-10-16T09:59:58.620368+0000","last_became_peered":"2023-10-16T09:59:58.620368+0000","last_unstale":"2023-10-16T10:00:00.613016+0000","last_undegraded":"2023-10-16T10:00:00.613016+0000","last_fullsized":"2023-10-16T10:00:00.613016+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:32.134734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.626563+0000","last_change":"2023-10-16T09:59:59.617389+0000","last_active":"2023-10-16T09:59:59.626563+0000","last_peered":"2023-10-16T09:59:59.626563+0000","last_clean":"2023-10-16T09:59:59.626563+0000","last_became_active":"2023-10-16T09:59:59.617107+0000","last_became_peered":"2023-10-16T09:59:59.617107+0000","last_unstale":"2023-10-16T09:59:59.626563+0000","last_undegraded":"2023-10-16T09:59:59.626563+0000","last_fullsized":"2023-10-16T09:59:59.626563+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:31.032244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"97'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632834+0000","last_change":"2023-10-16T09:59:58.624889+0000","last_active":"2023-10-16T09:59:59.632834+0000","last_peered":"2023-10-16T09:59:59.632834+0000","last_clean":"2023-10-16T09:59:59.632834+0000","last_became_active":"2023-10-16T09:59:58.624602+0000","last_became_peered":"2023-10-16T09:59:58.624602+0000","last_unstale":"2023-10-16T09:59:59.632834+0000","last_undegraded":"2023-10-16T09:59:59.632834+0000","last_fullsized":"2023-10-16T09:59:59.632834+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:17.280350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613210+0000","last_change":"2023-10-16T09:59:59.626856+0000","last_active":"2023-10-16T10:00:00.613210+0000","last_peered":"2023-10-16T10:00:00.613210+0000","last_clean":"2023-10-16T10:00:00.613210+0000","last_became_active":"2023-10-16T09:59:59.625352+0000","last_became_peered":"2023-10-16T09:59:59.625352+0000","last_unstale":"2023-10-16T10:00:00.613210+0000","last_undegraded":"2023-10-16T10:00:00.613210+0000","last_fullsized":"2023-10-16T10:00:00.613210+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:00.811337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.607281+0000","last_change":"2023-10-16T09:59:58.617074+0000","last_active":"2023-10-16T09:59:59.607281+0000","last_peered":"2023-10-16T09:59:59.607281+0000","last_clean":"2023-10-16T09:59:59.607281+0000","last_became_active":"2023-10-16T09:59:58.616834+0000","last_became_peered":"2023-10-16T09:59:58.616834+0000","last_unstale":"2023-10-16T09:59:59.607281+0000","last_undegraded":"2023-10-16T09:59:59.607281+0000","last_fullsized":"2023-10-16T09:59:59.607281+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:51.989803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643528+0000","last_change":"2023-10-16T09:59:59.639061+0000","last_active":"2023-10-16T09:59:59.643528+0000","last_peered":"2023-10-16T09:59:59.643528+0000","last_clean":"2023-10-16T09:59:59.643528+0000","last_became_active":"2023-10-16T09:59:59.638829+0000","last_became_peered":"2023-10-16T09:59:59.638829+0000","last_unstale":"2023-10-16T09:59:59.643528+0000","last_undegraded":"2023-10-16T09:59:59.643528+0000","last_fullsized":"2023-10-16T09:59:59.643528+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:48.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.616184+0000","last_change":"2023-10-16T09:59:58.615808+0000","last_active":"2023-10-16T09:59:59.616184+0000","last_peered":"2023-10-16T09:59:59.616184+0000","last_clean":"2023-10-16T09:59:59.616184+0000","last_became_active":"2023-10-16T09:59:58.615576+0000","last_became_peered":"2023-10-16T09:59:58.615576+0000","last_unstale":"2023-10-16T09:59:59.616184+0000","last_undegraded":"2023-10-16T09:59:59.616184+0000","last_fullsized":"2023-10-16T09:59:59.616184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:12.963507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.621092+0000","last_change":"2023-10-16T09:59:59.615222+0000","last_active":"2023-10-16T09:59:59.621092+0000","last_peered":"2023-10-16T09:59:59.621092+0000","last_clean":"2023-10-16T09:59:59.621092+0000","last_became_active":"2023-10-16T09:59:59.614962+0000","last_became_peered":"2023-10-16T09:59:59.614962+0000","last_unstale":"2023-10-16T09:59:59.621092+0000","last_undegraded":"2023-10-16T09:59:59.621092+0000","last_fullsized":"2023-10-16T09:59:59.621092+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:16:59.866866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613632+0000","last_change":"2023-10-16T09:59:58.620712+0000","last_active":"2023-10-16T10:00:00.613632+0000","last_peered":"2023-10-16T10:00:00.613632+0000","last_clean":"2023-10-16T10:00:00.613632+0000","last_became_active":"2023-10-16T09:59:58.619281+0000","last_became_peered":"2023-10-16T09:59:58.619281+0000","last_unstale":"2023-10-16T10:00:00.613632+0000","last_undegraded":"2023-10-16T10:00:00.613632+0000","last_fullsized":"2023-10-16T10:00:00.613632+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:17.280350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.631210+0000","last_change":"2023-10-16T09:59:59.623052+0000","last_active":"2023-10-16T09:59:59.631210+0000","last_peered":"2023-10-16T09:59:59.631210+0000","last_clean":"2023-10-16T09:59:59.631210+0000","last_became_active":"2023-10-16T09:59:59.622509+0000","last_became_peered":"2023-10-16T09:59:59.622509+0000","last_unstale":"2023-10-16T09:59:59.631210+0000","last_undegraded":"2023-10-16T09:59:59.631210+0000","last_fullsized":"2023-10-16T09:59:59.631210+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:59.056218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.611360+0000","last_change":"2023-10-16T09:59:58.621839+0000","last_active":"2023-10-16T09:59:59.611360+0000","last_peered":"2023-10-16T09:59:59.611360+0000","last_clean":"2023-10-16T09:59:59.611360+0000","last_became_active":"2023-10-16T09:59:58.621580+0000","last_became_peered":"2023-10-16T09:59:58.621580+0000","last_unstale":"2023-10-16T09:59:59.611360+0000","last_undegraded":"2023-10-16T09:59:59.611360+0000","last_fullsized":"2023-10-16T09:59:59.611360+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:36.752829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613491+0000","last_change":"2023-10-16T09:59:59.622677+0000","last_active":"2023-10-16T10:00:00.613491+0000","last_peered":"2023-10-16T10:00:00.613491+0000","last_clean":"2023-10-16T10:00:00.613491+0000","last_became_active":"2023-10-16T09:59:59.622373+0000","last_became_peered":"2023-10-16T09:59:59.622373+0000","last_unstale":"2023-10-16T10:00:00.613491+0000","last_undegraded":"2023-10-16T10:00:00.613491+0000","last_fullsized":"2023-10-16T10:00:00.613491+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:25.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"97'3","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643204+0000","last_change":"2023-10-16T09:59:58.619676+0000","last_active":"2023-10-16T09:59:59.643204+0000","last_peered":"2023-10-16T09:59:59.643204+0000","last_clean":"2023-10-16T09:59:59.643204+0000","last_became_active":"2023-10-16T09:59:58.619420+0000","last_became_peered":"2023-10-16T09:59:58.619420+0000","last_unstale":"2023-10-16T09:59:59.643204+0000","last_undegraded":"2023-10-16T09:59:59.643204+0000","last_fullsized":"2023-10-16T09:59:59.643204+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:19:15.153943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632769+0000","last_change":"2023-10-16T09:59:59.627746+0000","last_active":"2023-10-16T09:59:59.632769+0000","last_peered":"2023-10-16T09:59:59.632769+0000","last_clean":"2023-10-16T09:59:59.632769+0000","last_became_active":"2023-10-16T09:59:59.627160+0000","last_became_peered":"2023-10-16T09:59:59.627160+0000","last_unstale":"2023-10-16T09:59:59.632769+0000","last_undegraded":"2023-10-16T09:59:59.632769+0000","last_fullsized":"2023-10-16T09:59:59.632769+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:07:45.973592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"97'6","reported_seq":17,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643588+0000","last_change":"2023-10-16T09:59:58.621867+0000","last_active":"2023-10-16T09:59:59.643588+0000","last_peered":"2023-10-16T09:59:59.643588+0000","last_clean":"2023-10-16T09:59:59.643588+0000","last_became_active":"2023-10-16T09:59:58.621519+0000","last_became_peered":"2023-10-16T09:59:58.621519+0000","last_unstale":"2023-10-16T09:59:59.643588+0000","last_undegraded":"2023-10-16T09:59:59.643588+0000","last_fullsized":"2023-10-16T09:59:59.643588+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:05.454987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.632509+0000","last_change":"2023-10-16T09:59:59.623230+0000","last_active":"2023-10-16T09:59:59.632509+0000","last_peered":"2023-10-16T09:59:59.632509+0000","last_clean":"2023-10-16T09:59:59.632509+0000","last_became_active":"2023-10-16T09:59:59.622938+0000","last_became_peered":"2023-10-16T09:59:59.622938+0000","last_unstale":"2023-10-16T09:59:59.632509+0000","last_undegraded":"2023-10-16T09:59:59.632509+0000","last_fullsized":"2023-10-16T09:59:59.632509+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:19.035867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"97'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.636187+0000","last_change":"2023-10-16T09:59:58.614363+0000","last_active":"2023-10-16T09:59:59.636187+0000","last_peered":"2023-10-16T09:59:59.636187+0000","last_clean":"2023-10-16T09:59:59.636187+0000","last_became_active":"2023-10-16T09:59:58.614066+0000","last_became_peered":"2023-10-16T09:59:58.614066+0000","last_unstale":"2023-10-16T09:59:59.636187+0000","last_undegraded":"2023-10-16T09:59:59.636187+0000","last_fullsized":"2023-10-16T09:59:59.636187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:55.612848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635368+0000","last_change":"2023-10-16T09:59:59.629993+0000","last_active":"2023-10-16T09:59:59.635368+0000","last_peered":"2023-10-16T09:59:59.635368+0000","last_clean":"2023-10-16T09:59:59.635368+0000","last_became_active":"2023-10-16T09:59:59.629705+0000","last_became_peered":"2023-10-16T09:59:59.629705+0000","last_unstale":"2023-10-16T09:59:59.635368+0000","last_undegraded":"2023-10-16T09:59:59.635368+0000","last_fullsized":"2023-10-16T09:59:59.635368+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:43.179440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613128+0000","last_change":"2023-10-16T09:59:58.616338+0000","last_active":"2023-10-16T10:00:00.613128+0000","last_peered":"2023-10-16T10:00:00.613128+0000","last_clean":"2023-10-16T10:00:00.613128+0000","last_became_active":"2023-10-16T09:59:58.616065+0000","last_became_peered":"2023-10-16T09:59:58.616065+0000","last_unstale":"2023-10-16T10:00:00.613128+0000","last_undegraded":"2023-10-16T10:00:00.613128+0000","last_fullsized":"2023-10-16T10:00:00.613128+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:23.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635526+0000","last_change":"2023-10-16T09:59:59.627772+0000","last_active":"2023-10-16T09:59:59.635526+0000","last_peered":"2023-10-16T09:59:59.635526+0000","last_clean":"2023-10-16T09:59:59.635526+0000","last_became_active":"2023-10-16T09:59:59.627464+0000","last_became_peered":"2023-10-16T09:59:59.627464+0000","last_unstale":"2023-10-16T09:59:59.635526+0000","last_undegraded":"2023-10-16T09:59:59.635526+0000","last_fullsized":"2023-10-16T09:59:59.635526+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:52:31.337269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.615641+0000","last_change":"2023-10-16T09:59:58.622887+0000","last_active":"2023-10-16T09:59:59.615641+0000","last_peered":"2023-10-16T09:59:59.615641+0000","last_clean":"2023-10-16T09:59:59.615641+0000","last_became_active":"2023-10-16T09:59:58.622622+0000","last_became_peered":"2023-10-16T09:59:58.622622+0000","last_unstale":"2023-10-16T09:59:59.615641+0000","last_undegraded":"2023-10-16T09:59:59.615641+0000","last_fullsized":"2023-10-16T09:59:59.615641+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:48.271736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635465+0000","last_change":"2023-10-16T09:59:59.631991+0000","last_active":"2023-10-16T09:59:59.635465+0000","last_peered":"2023-10-16T09:59:59.635465+0000","last_clean":"2023-10-16T09:59:59.635465+0000","last_became_active":"2023-10-16T09:59:59.630142+0000","last_became_peered":"2023-10-16T09:59:59.630142+0000","last_unstale":"2023-10-16T09:59:59.635465+0000","last_undegraded":"2023-10-16T09:59:59.635465+0000","last_fullsized":"2023-10-16T09:59:59.635465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:54.625419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":19,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.637001+0000","last_change":"2023-10-16T09:59:58.618185+0000","last_active":"2023-10-16T09:59:59.637001+0000","last_peered":"2023-10-16T09:59:59.637001+0000","last_clean":"2023-10-16T09:59:59.637001+0000","last_became_active":"2023-10-16T09:59:58.617840+0000","last_became_peered":"2023-10-16T09:59:58.617840+0000","last_unstale":"2023-10-16T09:59:59.637001+0000","last_undegraded":"2023-10-16T09:59:59.637001+0000","last_fullsized":"2023-10-16T09:59:59.637001+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:09.477597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613178+0000","last_change":"2023-10-16T09:59:59.636714+0000","last_active":"2023-10-16T10:00:00.613178+0000","last_peered":"2023-10-16T10:00:00.613178+0000","last_clean":"2023-10-16T10:00:00.613178+0000","last_became_active":"2023-10-16T09:59:59.636445+0000","last_became_peered":"2023-10-16T09:59:59.636445+0000","last_unstale":"2023-10-16T10:00:00.613178+0000","last_undegraded":"2023-10-16T10:00:00.613178+0000","last_fullsized":"2023-10-16T10:00:00.613178+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:44.249052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"97'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613687+0000","last_change":"2023-10-16T09:59:58.617946+0000","last_active":"2023-10-16T10:00:00.613687+0000","last_peered":"2023-10-16T10:00:00.613687+0000","last_clean":"2023-10-16T10:00:00.613687+0000","last_became_active":"2023-10-16T09:59:58.617403+0000","last_became_peered":"2023-10-16T09:59:58.617403+0000","last_unstale":"2023-10-16T10:00:00.613687+0000","last_undegraded":"2023-10-16T10:00:00.613687+0000","last_fullsized":"2023-10-16T10:00:00.613687+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:04:53.778216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.640330+0000","last_change":"2023-10-16T09:59:59.635520+0000","last_active":"2023-10-16T09:59:59.640330+0000","last_peered":"2023-10-16T09:59:59.640330+0000","last_clean":"2023-10-16T09:59:59.640330+0000","last_became_active":"2023-10-16T09:59:59.635108+0000","last_became_peered":"2023-10-16T09:59:59.635108+0000","last_unstale":"2023-10-16T09:59:59.640330+0000","last_undegraded":"2023-10-16T09:59:59.640330+0000","last_fullsized":"2023-10-16T09:59:59.640330+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:47.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"97'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613357+0000","last_change":"2023-10-16T09:59:58.621599+0000","last_active":"2023-10-16T10:00:00.613357+0000","last_peered":"2023-10-16T10:00:00.613357+0000","last_clean":"2023-10-16T10:00:00.613357+0000","last_became_active":"2023-10-16T09:59:58.621142+0000","last_became_peered":"2023-10-16T09:59:58.621142+0000","last_unstale":"2023-10-16T10:00:00.613357+0000","last_undegraded":"2023-10-16T10:00:00.613357+0000","last_fullsized":"2023-10-16T10:00:00.613357+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:51:13.193671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635884+0000","last_change":"2023-10-16T09:59:59.624290+0000","last_active":"2023-10-16T09:59:59.635884+0000","last_peered":"2023-10-16T09:59:59.635884+0000","last_clean":"2023-10-16T09:59:59.635884+0000","last_became_active":"2023-10-16T09:59:59.616441+0000","last_became_peered":"2023-10-16T09:59:59.616441+0000","last_unstale":"2023-10-16T09:59:59.635884+0000","last_undegraded":"2023-10-16T09:59:59.635884+0000","last_fullsized":"2023-10-16T09:59:59.635884+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:05:28.238511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"97'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.643155+0000","last_change":"2023-10-16T09:59:58.614640+0000","last_active":"2023-10-16T09:59:59.643155+0000","last_peered":"2023-10-16T09:59:59.643155+0000","last_clean":"2023-10-16T09:59:59.643155+0000","last_became_active":"2023-10-16T09:59:58.614352+0000","last_became_peered":"2023-10-16T09:59:58.614352+0000","last_unstale":"2023-10-16T09:59:59.643155+0000","last_undegraded":"2023-10-16T09:59:59.643155+0000","last_fullsized":"2023-10-16T09:59:59.643155+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:47.353949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613456+0000","last_change":"2023-10-16T09:59:59.641573+0000","last_active":"2023-10-16T10:00:00.613456+0000","last_peered":"2023-10-16T10:00:00.613456+0000","last_clean":"2023-10-16T10:00:00.613456+0000","last_became_active":"2023-10-16T09:59:59.641306+0000","last_became_peered":"2023-10-16T09:59:59.641306+0000","last_unstale":"2023-10-16T10:00:00.613456+0000","last_undegraded":"2023-10-16T10:00:00.613456+0000","last_fullsized":"2023-10-16T10:00:00.613456+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:26.228687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.610634+0000","last_change":"2023-10-16T09:59:58.618990+0000","last_active":"2023-10-16T09:59:59.610634+0000","last_peered":"2023-10-16T09:59:59.610634+0000","last_clean":"2023-10-16T09:59:59.610634+0000","last_became_active":"2023-10-16T09:59:58.618729+0000","last_became_peered":"2023-10-16T09:59:58.618729+0000","last_unstale":"2023-10-16T09:59:59.610634+0000","last_undegraded":"2023-10-16T09:59:59.610634+0000","last_fullsized":"2023-10-16T09:59:59.610634+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:11.615907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.621113+0000","last_change":"2023-10-16T09:59:59.613731+0000","last_active":"2023-10-16T09:59:59.621113+0000","last_peered":"2023-10-16T09:59:59.621113+0000","last_clean":"2023-10-16T09:59:59.621113+0000","last_became_active":"2023-10-16T09:59:59.613480+0000","last_became_peered":"2023-10-16T09:59:59.613480+0000","last_unstale":"2023-10-16T09:59:59.621113+0000","last_undegraded":"2023-10-16T09:59:59.621113+0000","last_fullsized":"2023-10-16T09:59:59.621113+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:57:42.348980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"97'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613474+0000","last_change":"2023-10-16T09:59:58.624490+0000","last_active":"2023-10-16T10:00:00.613474+0000","last_peered":"2023-10-16T10:00:00.613474+0000","last_clean":"2023-10-16T10:00:00.613474+0000","last_became_active":"2023-10-16T09:59:58.624231+0000","last_became_peered":"2023-10-16T09:59:58.624231+0000","last_unstale":"2023-10-16T10:00:00.613474+0000","last_undegraded":"2023-10-16T10:00:00.613474+0000","last_fullsized":"2023-10-16T10:00:00.613474+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:16.653024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.628275+0000","last_change":"2023-10-16T09:59:59.626030+0000","last_active":"2023-10-16T09:59:59.628275+0000","last_peered":"2023-10-16T09:59:59.628275+0000","last_clean":"2023-10-16T09:59:59.628275+0000","last_became_active":"2023-10-16T09:59:59.625737+0000","last_became_peered":"2023-10-16T09:59:59.625737+0000","last_unstale":"2023-10-16T09:59:59.628275+0000","last_undegraded":"2023-10-16T09:59:59.628275+0000","last_fullsized":"2023-10-16T09:59:59.628275+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:12.515762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613329+0000","last_change":"2023-10-16T09:59:58.626472+0000","last_active":"2023-10-16T10:00:00.613329+0000","last_peered":"2023-10-16T10:00:00.613329+0000","last_clean":"2023-10-16T10:00:00.613329+0000","last_became_active":"2023-10-16T09:59:58.626202+0000","last_became_peered":"2023-10-16T09:59:58.626202+0000","last_unstale":"2023-10-16T10:00:00.613329+0000","last_undegraded":"2023-10-16T10:00:00.613329+0000","last_fullsized":"2023-10-16T10:00:00.613329+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:11:23.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635348+0000","last_change":"2023-10-16T09:59:59.628375+0000","last_active":"2023-10-16T09:59:59.635348+0000","last_peered":"2023-10-16T09:59:59.635348+0000","last_clean":"2023-10-16T09:59:59.635348+0000","last_became_active":"2023-10-16T09:59:59.628039+0000","last_became_peered":"2023-10-16T09:59:59.628039+0000","last_unstale":"2023-10-16T09:59:59.635348+0000","last_undegraded":"2023-10-16T09:59:59.635348+0000","last_fullsized":"2023-10-16T09:59:59.635348+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:51:14.201119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613262+0000","last_change":"2023-10-16T09:59:58.625678+0000","last_active":"2023-10-16T10:00:00.613262+0000","last_peered":"2023-10-16T10:00:00.613262+0000","last_clean":"2023-10-16T10:00:00.613262+0000","last_became_active":"2023-10-16T09:59:58.625376+0000","last_became_peered":"2023-10-16T09:59:58.625376+0000","last_unstale":"2023-10-16T10:00:00.613262+0000","last_undegraded":"2023-10-16T10:00:00.613262+0000","last_fullsized":"2023-10-16T10:00:00.613262+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:26.838497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.612999+0000","last_change":"2023-10-16T09:59:59.625835+0000","last_active":"2023-10-16T10:00:00.612999+0000","last_peered":"2023-10-16T10:00:00.612999+0000","last_clean":"2023-10-16T10:00:00.612999+0000","last_became_active":"2023-10-16T09:59:59.625462+0000","last_became_peered":"2023-10-16T09:59:59.625462+0000","last_unstale":"2023-10-16T10:00:00.612999+0000","last_undegraded":"2023-10-16T10:00:00.612999+0000","last_fullsized":"2023-10-16T10:00:00.612999+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:18.720561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"97'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.633790+0000","last_change":"2023-10-16T09:59:58.620387+0000","last_active":"2023-10-16T09:59:59.633790+0000","last_peered":"2023-10-16T09:59:59.633790+0000","last_clean":"2023-10-16T09:59:59.633790+0000","last_became_active":"2023-10-16T09:59:58.620087+0000","last_became_peered":"2023-10-16T09:59:58.620087+0000","last_unstale":"2023-10-16T09:59:59.633790+0000","last_undegraded":"2023-10-16T09:59:59.633790+0000","last_fullsized":"2023-10-16T09:59:59.633790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:18.871926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.635395+0000","last_change":"2023-10-16T09:59:59.628652+0000","last_active":"2023-10-16T09:59:59.635395+0000","last_peered":"2023-10-16T09:59:59.635395+0000","last_clean":"2023-10-16T09:59:59.635395+0000","last_became_active":"2023-10-16T09:59:59.628287+0000","last_became_peered":"2023-10-16T09:59:59.628287+0000","last_unstale":"2023-10-16T09:59:59.635395+0000","last_undegraded":"2023-10-16T09:59:59.635395+0000","last_fullsized":"2023-10-16T09:59:59.635395+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:12.623356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.608121+0000","last_change":"2023-10-16T09:59:58.615206+0000","last_active":"2023-10-16T09:59:59.608121+0000","last_peered":"2023-10-16T09:59:59.608121+0000","last_clean":"2023-10-16T09:59:59.608121+0000","last_became_active":"2023-10-16T09:59:58.614936+0000","last_became_peered":"2023-10-16T09:59:58.614936+0000","last_unstale":"2023-10-16T09:59:59.608121+0000","last_undegraded":"2023-10-16T09:59:59.608121+0000","last_fullsized":"2023-10-16T09:59:59.608121+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:02.421634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613263+0000","last_change":"2023-10-16T09:59:59.627140+0000","last_active":"2023-10-16T10:00:00.613263+0000","last_peered":"2023-10-16T10:00:00.613263+0000","last_clean":"2023-10-16T10:00:00.613263+0000","last_became_active":"2023-10-16T09:59:59.626845+0000","last_became_peered":"2023-10-16T09:59:59.626845+0000","last_unstale":"2023-10-16T10:00:00.613263+0000","last_undegraded":"2023-10-16T10:00:00.613263+0000","last_fullsized":"2023-10-16T10:00:00.613263+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:37:39.047713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613580+0000","last_change":"2023-10-16T09:59:58.614417+0000","last_active":"2023-10-16T10:00:00.613580+0000","last_peered":"2023-10-16T10:00:00.613580+0000","last_clean":"2023-10-16T10:00:00.613580+0000","last_became_active":"2023-10-16T09:59:58.614064+0000","last_became_peered":"2023-10-16T09:59:58.614064+0000","last_unstale":"2023-10-16T10:00:00.613580+0000","last_undegraded":"2023-10-16T10:00:00.613580+0000","last_fullsized":"2023-10-16T10:00:00.613580+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:26.838497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.617548+0000","last_change":"2023-10-16T09:59:59.613088+0000","last_active":"2023-10-16T09:59:59.617548+0000","last_peered":"2023-10-16T09:59:59.617548+0000","last_clean":"2023-10-16T09:59:59.617548+0000","last_became_active":"2023-10-16T09:59:59.612231+0000","last_became_peered":"2023-10-16T09:59:59.612231+0000","last_unstale":"2023-10-16T09:59:59.617548+0000","last_undegraded":"2023-10-16T09:59:59.617548+0000","last_fullsized":"2023-10-16T09:59:59.617548+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:55.809034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613260+0000","last_change":"2023-10-16T09:59:58.620726+0000","last_active":"2023-10-16T10:00:00.613260+0000","last_peered":"2023-10-16T10:00:00.613260+0000","last_clean":"2023-10-16T10:00:00.613260+0000","last_became_active":"2023-10-16T09:59:58.619307+0000","last_became_peered":"2023-10-16T09:59:58.619307+0000","last_unstale":"2023-10-16T10:00:00.613260+0000","last_undegraded":"2023-10-16T10:00:00.613260+0000","last_fullsized":"2023-10-16T10:00:00.613260+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:52.613669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.634502+0000","last_change":"2023-10-16T09:59:59.626531+0000","last_active":"2023-10-16T09:59:59.634502+0000","last_peered":"2023-10-16T09:59:59.634502+0000","last_clean":"2023-10-16T09:59:59.634502+0000","last_became_active":"2023-10-16T09:59:59.626244+0000","last_became_peered":"2023-10-16T09:59:59.626244+0000","last_unstale":"2023-10-16T09:59:59.634502+0000","last_undegraded":"2023-10-16T09:59:59.634502+0000","last_fullsized":"2023-10-16T09:59:59.634502+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:33.142182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.617513+0000","last_change":"2023-10-16T09:59:58.619645+0000","last_active":"2023-10-16T09:59:59.617513+0000","last_peered":"2023-10-16T09:59:59.617513+0000","last_clean":"2023-10-16T09:59:59.617513+0000","last_became_active":"2023-10-16T09:59:58.619375+0000","last_became_peered":"2023-10-16T09:59:58.619375+0000","last_unstale":"2023-10-16T09:59:59.617513+0000","last_undegraded":"2023-10-16T09:59:59.617513+0000","last_fullsized":"2023-10-16T09:59:59.617513+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:52.988880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.649069+0000","last_change":"2023-10-16T09:59:59.643514+0000","last_active":"2023-10-16T09:59:59.649069+0000","last_peered":"2023-10-16T09:59:59.649069+0000","last_clean":"2023-10-16T09:59:59.649069+0000","last_became_active":"2023-10-16T09:59:59.643258+0000","last_became_peered":"2023-10-16T09:59:59.643258+0000","last_unstale":"2023-10-16T09:59:59.649069+0000","last_undegraded":"2023-10-16T09:59:59.649069+0000","last_fullsized":"2023-10-16T09:59:59.649069+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:06:45.905561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.624146+0000","last_change":"2023-10-16T09:59:58.626053+0000","last_active":"2023-10-16T09:59:59.624146+0000","last_peered":"2023-10-16T09:59:59.624146+0000","last_clean":"2023-10-16T09:59:59.624146+0000","last_became_active":"2023-10-16T09:59:58.625769+0000","last_became_peered":"2023-10-16T09:59:58.625769+0000","last_unstale":"2023-10-16T09:59:59.624146+0000","last_undegraded":"2023-10-16T09:59:59.624146+0000","last_fullsized":"2023-10-16T09:59:59.624146+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:32:53.820213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.622601+0000","last_change":"2023-10-16T09:59:59.621193+0000","last_active":"2023-10-16T09:59:59.622601+0000","last_peered":"2023-10-16T09:59:59.622601+0000","last_clean":"2023-10-16T09:59:59.622601+0000","last_became_active":"2023-10-16T09:59:59.620916+0000","last_became_peered":"2023-10-16T09:59:59.620916+0000","last_unstale":"2023-10-16T09:59:59.622601+0000","last_undegraded":"2023-10-16T09:59:59.622601+0000","last_fullsized":"2023-10-16T09:59:59.622601+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:37:39.047713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.622953+0000","last_change":"2023-10-16T09:59:58.624846+0000","last_active":"2023-10-16T09:59:59.622953+0000","last_peered":"2023-10-16T09:59:59.622953+0000","last_clean":"2023-10-16T09:59:59.622953+0000","last_became_active":"2023-10-16T09:59:58.624548+0000","last_became_peered":"2023-10-16T09:59:58.624548+0000","last_unstale":"2023-10-16T09:59:59.622953+0000","last_undegraded":"2023-10-16T09:59:59.622953+0000","last_fullsized":"2023-10-16T09:59:59.622953+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:07:10.368596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.622005+0000","last_change":"2023-10-16T09:59:59.620677+0000","last_active":"2023-10-16T09:59:59.622005+0000","last_peered":"2023-10-16T09:59:59.622005+0000","last_clean":"2023-10-16T09:59:59.622005+0000","last_became_active":"2023-10-16T09:59:59.620393+0000","last_became_peered":"2023-10-16T09:59:59.620393+0000","last_unstale":"2023-10-16T09:59:59.622005+0000","last_undegraded":"2023-10-16T09:59:59.622005+0000","last_fullsized":"2023-10-16T09:59:59.622005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:05.195299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.610991+0000","last_change":"2023-10-16T09:59:58.625330+0000","last_active":"2023-10-16T09:59:59.610991+0000","last_peered":"2023-10-16T09:59:59.610991+0000","last_clean":"2023-10-16T09:59:59.610991+0000","last_became_active":"2023-10-16T09:59:58.625068+0000","last_became_peered":"2023-10-16T09:59:58.625068+0000","last_unstale":"2023-10-16T09:59:59.610991+0000","last_undegraded":"2023-10-16T09:59:59.610991+0000","last_fullsized":"2023-10-16T09:59:59.610991+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:59.803889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613438+0000","last_change":"2023-10-16T09:59:59.622046+0000","last_active":"2023-10-16T10:00:00.613438+0000","last_peered":"2023-10-16T10:00:00.613438+0000","last_clean":"2023-10-16T10:00:00.613438+0000","last_became_active":"2023-10-16T09:59:59.621678+0000","last_became_peered":"2023-10-16T09:59:59.621678+0000","last_unstale":"2023-10-16T10:00:00.613438+0000","last_undegraded":"2023-10-16T10:00:00.613438+0000","last_fullsized":"2023-10-16T10:00:00.613438+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:58.595651+0000","last_clean_scrub_stamp":"2023-10-16T09:59:58.595651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:25.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T10:00:00.613408+0000","last_change":"2023-10-16T09:59:58.626614+0000","last_active":"2023-10-16T10:00:00.613408+0000","last_peered":"2023-10-16T10:00:00.613408+0000","last_clean":"2023-10-16T10:00:00.613408+0000","last_became_active":"2023-10-16T09:59:58.625764+0000","last_became_peered":"2023-10-16T09:59:58.625764+0000","last_unstale":"2023-10-16T10:00:00.613408+0000","last_undegraded":"2023-10-16T10:00:00.613408+0000","last_fullsized":"2023-10-16T10:00:00.613408+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:59:57.588203+0000","last_clean_scrub_stamp":"2023-10-16T09:59:57.588203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:23.950004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'172","reported_seq":438,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T09:59:59.614269+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T09:59:59.614269+0000","last_peered":"2023-10-16T09:59:59.614269+0000","last_clean":"2023-10-16T09:59:59.614269+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T09:59:59.614269+0000","last_undegraded":"2023-10-16T09:59:59.614269+0000","last_fullsized":"2023-10-16T09:59:59.614269+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59292,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312548,"statfs":{"total":96636764160,"available":96576049152,"internally_reserved":0,"allocated":1785856,"data_stored":1202202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.56100000000000005},"min":{"1min":0.36499999999999999,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":0.73299999999999998,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.67700000000000005,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.442,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70399999999999996,"5min":0.79300000000000004,"15min":1.1439999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.46500000000000002,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.78100000000000003,"5min":1.1279999999999999,"15min":1.212},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.63100000000000001},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.81599999999999995,"5min":1.607,"15min":1.607},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61399999999999999,"15min":0.63400000000000001},"min":{"1min":0.432,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":2.044,"15min":2.044},"last":0.57499999999999996}]},{"osd":3,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58399999999999996,"15min":0.60099999999999998},"min":{"1min":0.42799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.755,"5min":1.524,"15min":1.524},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.50700000000000001,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.70399999999999996,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.497},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68600000000000005,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":0.54900000000000004}]},{"osd":5,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73899999999999999,"5min":0.81100000000000005,"15min":0.875},"last":0.438},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.37,"5min":0.33200000000000002,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.76000000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.438,"5min":0.311,"15min":0.311},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.85299999999999998},"last":0.58899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1200128,"data_stored":613424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55600000000000005,"15min":0.58299999999999996},"min":{"1min":0.29699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.68200000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.498},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57299999999999995,"15min":0.59299999999999997},"min":{"1min":0.38100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.76900000000000002,"5min":0.78800000000000003,"15min":0.82099999999999995},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.36099999999999999,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.72199999999999998,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56399999999999995,"15min":0.58199999999999996},"min":{"1min":0.33100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.749,"5min":1.448,"15min":1.448},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":1.708,"15min":1.708},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.76000000000000001,"15min":0.81699999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.82999999999999996},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.83999999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.81100000000000005},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.253,"15min":0.253},"max":{"1min":0.81100000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53700000000000003,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.872},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.79100000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1196032,"data_stored":611924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.38800000000000001,"15min":0.39500000000000002},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":0.52200000000000002,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.51000000000000001,"5min":0.67900000000000005,"15min":0.70199999999999996},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.43099999999999999,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54000000000000004,"5min":4.923,"15min":4.923},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.216},"max":{"1min":0.58699999999999997,"5min":1.712,"15min":1.712},"last":0.26000000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46400000000000002,"15min":0.45600000000000002},"min":{"1min":0.32100000000000001,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.66500000000000004,"5min":1.984,"15min":1.984},"last":0.313},{"interface":"front","average":{"1min":0.37,"5min":0.41999999999999998,"15min":0.42499999999999999},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.56699999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61499999999999999,"15min":0.628},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.45200000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67700000000000005,"15min":0.66500000000000004},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.85899999999999999,"5min":4.9749999999999996,"15min":4.9749999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77400000000000002,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.54600000000000004}]},{"osd":6,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.80700000000000005,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.68100000000000005,"15min":0.65700000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.82899999999999996,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.45800000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.75900000000000001,"5min":1.744,"15min":1.744},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38500000000000001},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.93400000000000005},"last":0.61099999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575049,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":1202180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39200000000000002,"15min":0.38300000000000001},"min":{"1min":0.27100000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":0.55600000000000005,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.318}]},{"osd":2,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53000000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.34799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.63600000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.373}]},{"osd":3,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.224},"max":{"1min":0.57899999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46400000000000002},"min":{"1min":0.27700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59699999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.437,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.77300000000000002,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":0.81100000000000005,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.71999999999999997}]},{"osd":5,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61199999999999999,"15min":0.63500000000000001},"min":{"1min":0.39200000000000002,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.94999999999999996,"15min":0.998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.82599999999999996,"5min":0.88700000000000001,"15min":0.93000000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84999999999999998,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.747}]},{"osd":7,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":1.014,"15min":1.02},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58728,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313112,"statfs":{"total":96636764160,"available":96576626688,"internally_reserved":0,"allocated":1200128,"data_stored":612554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.51400000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.246},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.53800000000000003,"5min":0.64300000000000002,"15min":0.67500000000000004},"last":0.314}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.61899999999999999,"5min":0.68000000000000005,"15min":0.754},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40799999999999997,"15min":0.41999999999999998},"min":{"1min":0.29999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64800000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.35299999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.64500000000000002},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.103},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.65700000000000003},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.434}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63100000000000001,"15min":0.64900000000000002},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.046,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65700000000000003,"15min":0.68300000000000005},"min":{"1min":0.441,"5min":0.40799999999999997,"15min":0.40100000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65800000000000003,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.43099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.0069999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36520,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335320,"statfs":{"total":96636764160,"available":96599367680,"internally_reserved":0,"allocated":1204224,"data_stored":619438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.20999999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.34100000000000003}]},{"osd":1,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.39400000000000002,"15min":0.39500000000000002},"min":{"1min":0.24099999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.65400000000000003},"last":0.255},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":0.56799999999999995,"5min":0.61899999999999999,"15min":0.67500000000000004},"last":0.309}]},{"osd":2,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":0.59999999999999998,"5min":0.70499999999999996,"15min":1.0760000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.64100000000000001,"5min":0.73999999999999999,"15min":0.93799999999999994},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.63500000000000001,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89000000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33200000000000002},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":1.036},"last":0.71899999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64100000000000001,"15min":0.65000000000000002},"min":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.374},"max":{"1min":0.83299999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":1.0589999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42599999999999999},"max":{"1min":0.872,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.42699999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":0.999},"last":0.79100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575048,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1208320,"data_stored":613528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.59199999999999997},"min":{"1min":0.34200000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.83099999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.38}]},{"osd":1,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.72299999999999998,"5min":0.81499999999999995,"15min":0.82599999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.56200000000000006,"15min":0.58999999999999997},"min":{"1min":0.33300000000000002,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.66800000000000004,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.499},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.622},"min":{"1min":0.438,"5min":0.379,"15min":0.32500000000000001},"max":{"1min":0.79600000000000004,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75800000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58299999999999996,"15min":0.62},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.84599999999999997},"last":0.626}]},{"osd":5,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.749,"5min":0.76400000000000001,"15min":0.84299999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.77500000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.34100000000000003}]},{"osd":6,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.84699999999999998,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80100000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.48199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80800000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.66800000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46240,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325600,"statfs":{"total":96636764160,"available":96589414400,"internally_reserved":0,"allocated":1191936,"data_stored":611846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":0.80600000000000005,"5min":1.792,"15min":1.792},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45700000000000002,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.83099999999999996,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.439}]},{"osd":2,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.57299999999999995},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.496,"15min":1.496},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.72099999999999997,"5min":2.069,"15min":2.069},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46700000000000003,"15min":0.47799999999999998},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66100000000000003,"5min":0.71299999999999997,"15min":0.84199999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.86899999999999999},"last":0.374}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48699999999999999,"15min":0.48999999999999999},"min":{"1min":0.312,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.84099999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.88200000000000001},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80500000000000005,"5min":0.81799999999999995,"15min":0.90300000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.79500000000000004,"5min":0.84899999999999998,"15min":0.88500000000000001},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:00:04.126 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T10:00:04.127 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T10:00:04.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.286+0000 7fee17593700 1 Processor -- start 2023-10-16T10:00:04.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.286+0000 7fee17593700 1 -- start start 2023-10-16T10:00:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee17593700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee1006b9e0 0x7fee1006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee17593700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee100d0560 0x7fee10070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee17593700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee10161440 con 0x7fee10070530 2023-10-16T10:00:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee17593700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1006c300 con 0x7fee1006b9e0 2023-10-16T10:00:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee17593700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee10161670 con 0x7fee100d0560 2023-10-16T10:00:04.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee1532f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee1006b9e0 0x7fee1006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee1532f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee1006b9e0 0x7fee1006bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53842/0 (socket says 172.21.15.158:53842) 2023-10-16T10:00:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.287+0000 7fee1532f700 1 -- 172.21.15.158:0/2113748106 learned_addr learned my addr 172.21.15.158:0/2113748106 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee15b30700 1 --2- 172.21.15.158:0/2113748106 >> v2:172.21.15.172:3300/0 conn(0x7fee100d0560 0x7fee10070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee1532f700 1 -- 172.21.15.158:0/2113748106 >> v2:172.21.15.172:3300/0 conn(0x7fee100d0560 msgr2=0x7fee10070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:04.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee1532f700 1 --2- 172.21.15.158:0/2113748106 >> v2:172.21.15.172:3300/0 conn(0x7fee100d0560 0x7fee10070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee1532f700 1 -- 172.21.15.158:0/2113748106 >> v1:172.21.15.172:6789/0 conn(0x7fee10070530 legacy=0x7fee1006ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee1532f700 1 -- 172.21.15.158:0/2113748106 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10070ed0 con 0x7fee1006b9e0 2023-10-16T10:00:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee15b30700 1 --2- 172.21.15.158:0/2113748106 >> v2:172.21.15.172:3300/0 conn(0x7fee100d0560 0x7fee10070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee1532f700 1 --2- 172.21.15.158:0/2113748106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee1006b9e0 0x7fee1006bdc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fee100739b0 tx=0x7fee0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd1723f4c03f248 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:04.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee0ffff700 1 -- 172.21.15.158:0/2113748106 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0000e070 con 0x7fee1006b9e0 2023-10-16T10:00:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee0ffff700 1 -- 172.21.15.158:0/2113748106 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee00003a20 con 0x7fee1006b9e0 2023-10-16T10:00:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee0ffff700 1 -- 172.21.15.158:0/2113748106 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00003dd0 con 0x7fee1006b9e0 2023-10-16T10:00:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee17593700 1 -- 172.21.15.158:0/2113748106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee1006b9e0 msgr2=0x7fee1006bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.288+0000 7fee17593700 1 --2- 172.21.15.158:0/2113748106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee1006b9e0 0x7fee1006bdc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fee100739b0 tx=0x7fee0000b330 comp rx=0 tx=0).stop 2023-10-16T10:00:04.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.289+0000 7fee17593700 1 -- 172.21.15.158:0/2113748106 shutdown_connections 2023-10-16T10:00:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.289+0000 7fee17593700 1 --2- 172.21.15.158:0/2113748106 >> v2:172.21.15.172:3300/0 conn(0x7fee100d0560 0x7fee10070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.289+0000 7fee17593700 1 --2- 172.21.15.158:0/2113748106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee1006b9e0 0x7fee1006bdc0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.289+0000 7fee17593700 1 -- 172.21.15.158:0/2113748106 >> 172.21.15.158:0/2113748106 conn(0x7fee100d1410 msgr2=0x7fee100d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.289+0000 7fee17593700 1 -- 172.21.15.158:0/2113748106 shutdown_connections 2023-10-16T10:00:04.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.289+0000 7fee17593700 1 -- 172.21.15.158:0/2113748106 wait complete. 2023-10-16T10:00:04.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.290+0000 7fee17593700 1 Processor -- start 2023-10-16T10:00:04.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.290+0000 7fee17593700 1 -- start start 2023-10-16T10:00:04.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee17593700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee1006b9e0 0x7fee100b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee17593700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee10070530 0x7fee10159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee17593700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee100d0560 0x7fee100c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee17593700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10159ec0 con 0x7fee100d0560 2023-10-16T10:00:04.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee17593700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee100c77d0 con 0x7fee10070530 2023-10-16T10:00:04.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee17593700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee101448f0 con 0x7fee1006b9e0 2023-10-16T10:00:04.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee15b30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee100d0560 0x7fee100c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee15b30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee100d0560 0x7fee100c4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53852/0 (socket says 172.21.15.158:53852) 2023-10-16T10:00:04.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee15b30700 1 -- 172.21.15.158:0/2399207448 learned_addr learned my addr 172.21.15.158:0/2399207448 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:04.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee1532f700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee1006b9e0 0x7fee100b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee14b2e700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee10070530 0x7fee10159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.291+0000 7fee15b30700 1 -- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee1006b9e0 msgr2=0x7fee100b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee15b30700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee1006b9e0 0x7fee100b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee15b30700 1 -- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee10070530 msgr2=0x7fee10159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee15b30700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee10070530 0x7fee10159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee15b30700 1 -- 172.21.15.158:0/2399207448 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee00009b30 con 0x7fee100d0560 2023-10-16T10:00:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee15b30700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee100d0560 0x7fee100c4420 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fee0800c820 tx=0x7fee0800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:04.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee0dffb700 1 -- 172.21.15.158:0/2399207448 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08004220 con 0x7fee100d0560 2023-10-16T10:00:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee0dffb700 1 -- 172.21.15.158:0/2399207448 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee08004380 con 0x7fee100d0560 2023-10-16T10:00:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee100c4960 con 0x7fee100d0560 2023-10-16T10:00:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee0dffb700 1 -- 172.21.15.158:0/2399207448 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee080036a0 con 0x7fee100d0560 2023-10-16T10:00:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.292+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee101442a0 con 0x7fee100d0560 2023-10-16T10:00:04.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.293+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10144100 con 0x7fee100d0560 2023-10-16T10:00:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.293+0000 7fee0dffb700 1 -- 172.21.15.158:0/2399207448 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee08003800 con 0x7fee100d0560 2023-10-16T10:00:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.294+0000 7fee0dffb700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fedfc06b020 0x7fedfc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.294+0000 7fee0dffb700 1 -- 172.21.15.158:0/2399207448 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fee080896f0 con 0x7fee100d0560 2023-10-16T10:00:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.294+0000 7fee1532f700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fedfc06b020 0x7fedfc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.294+0000 7fee1532f700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fedfc06b020 0x7fedfc06d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fee00015040 tx=0x7fee00009000 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:04.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.298+0000 7fee0dffb700 1 -- 172.21.15.158:0/2399207448 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee10144100 con 0x7fee100d0560 2023-10-16T10:00:04.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.568+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fee10144100 con 0x7fee100d0560 2023-10-16T10:00:04.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.571+0000 7fee0dffb700 1 -- 172.21.15.158:0/2399207448 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7fee10144100 con 0x7fee100d0560 2023-10-16T10:00:04.572 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T10:00:04.572 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T10:00:04.572 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T10:00:04.572 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T10:00:04.573 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T10:00:04.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fedfc06b020 msgr2=0x7fedfc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fedfc06b020 0x7fedfc06d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fee00015040 tx=0x7fee00009000 comp rx=0 tx=0).stop 2023-10-16T10:00:04.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee100d0560 msgr2=0x7fee100c4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee100d0560 0x7fee100c4420 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fee0800c820 tx=0x7fee0800cbe0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 shutdown_connections 2023-10-16T10:00:04.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7fedfc06b020 0x7fedfc06d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee100d0560 0x7fee100c4420 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee1006b9e0 0x7fee100b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 --2- 172.21.15.158:0/2399207448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee10070530 0x7fee10159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 >> 172.21.15.158:0/2399207448 conn(0x7fee100d1410 msgr2=0x7fee1006ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:04.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 shutdown_connections 2023-10-16T10:00:04.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.573+0000 7fee17593700 1 -- 172.21.15.158:0/2399207448 wait complete. 2023-10-16T10:00:04.577 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T10:00:04.586 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:04.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.708+0000 7f735d447700 1 Processor -- start 2023-10-16T10:00:04.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.708+0000 7f735d447700 1 -- start start 2023-10-16T10:00:04.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f735d447700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f735815d7c0 0x7f73580d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f735d447700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580ceb20 0x7f73580b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f735d447700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73580d3bd0 con 0x7f73580d1210 2023-10-16T10:00:04.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f735d447700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73580d4060 con 0x7f735815d7c0 2023-10-16T10:00:04.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f735d447700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73580d2cf0 con 0x7f73580ceb20 2023-10-16T10:00:04.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f7356ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f735815d7c0 0x7f73580d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f73577fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580ceb20 0x7f73580b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f7356ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f735815d7c0 0x7f73580d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:55036/0 (socket says 172.21.15.158:55036) 2023-10-16T10:00:04.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f7356ffd700 1 -- 172.21.15.158:0/1433378138 learned_addr learned my addr 172.21.15.158:0/1433378138 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:04.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f73567fc700 1 --1- 172.21.15.158:0/1433378138 >> v1:172.21.15.172:6789/0 conn(0x7f73580d1210 0x7f73580ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:57936/0 (socket says 172.21.15.158:57936) 2023-10-16T10:00:04.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f7355ffb700 1 -- 172.21.15.158:0/1433378138 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385325703 0 0) 0x7f73580d3bd0 con 0x7f73580d1210 2023-10-16T10:00:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f7355ffb700 1 -- 172.21.15.158:0/1433378138 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73400035b0 con 0x7f73580d1210 2023-10-16T10:00:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f73577fe700 1 -- 172.21.15.158:0/1433378138 >> v2:172.21.15.172:3300/0 conn(0x7f735815d7c0 msgr2=0x7f73580d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f73577fe700 1 --2- 172.21.15.158:0/1433378138 >> v2:172.21.15.172:3300/0 conn(0x7f735815d7c0 0x7f73580d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f73577fe700 1 -- 172.21.15.158:0/1433378138 >> v1:172.21.15.172:6789/0 conn(0x7f73580d1210 legacy=0x7f73580ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f73577fe700 1 -- 172.21.15.158:0/1433378138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73580d2fc0 con 0x7f73580ceb20 2023-10-16T10:00:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.709+0000 7f7356ffd700 1 --2- 172.21.15.158:0/1433378138 >> v2:172.21.15.172:3300/0 conn(0x7f735815d7c0 0x7f73580d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f73577fe700 1 --2- 172.21.15.158:0/1433378138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580ceb20 0x7f73580b6800 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7350009d90 tx=0x7f735000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e2a7c4ff5e53fb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f7355ffb700 1 -- 172.21.15.158:0/1433378138 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735000e070 con 0x7f73580ceb20 2023-10-16T10:00:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f7355ffb700 1 -- 172.21.15.158:0/1433378138 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735000bb20 con 0x7f73580ceb20 2023-10-16T10:00:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f7355ffb700 1 -- 172.21.15.158:0/1433378138 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350012430 con 0x7f73580ceb20 2023-10-16T10:00:04.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 -- 172.21.15.158:0/1433378138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580ceb20 msgr2=0x7f73580b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 --2- 172.21.15.158:0/1433378138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580ceb20 0x7f73580b6800 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7350009d90 tx=0x7f735000b330 comp rx=0 tx=0).stop 2023-10-16T10:00:04.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 -- 172.21.15.158:0/1433378138 shutdown_connections 2023-10-16T10:00:04.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 --2- 172.21.15.158:0/1433378138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580ceb20 0x7f73580b6800 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 --2- 172.21.15.158:0/1433378138 >> v2:172.21.15.172:3300/0 conn(0x7f735815d7c0 0x7f73580d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 -- 172.21.15.158:0/1433378138 >> 172.21.15.158:0/1433378138 conn(0x7f73580669f0 msgr2=0x7f7358066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:04.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 -- 172.21.15.158:0/1433378138 shutdown_connections 2023-10-16T10:00:04.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.710+0000 7f735d447700 1 -- 172.21.15.158:0/1433378138 wait complete. 2023-10-16T10:00:04.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 Processor -- start 2023-10-16T10:00:04.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 -- start start 2023-10-16T10:00:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73580ceb20 0x7f735815f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580d1210 0x7f735815ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f735815d7c0 0x7f73581f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73581f3e20 con 0x7f73580ceb20 2023-10-16T10:00:04.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73581f4100 con 0x7f735815d7c0 2023-10-16T10:00:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.712+0000 7f735d447700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73581f43e0 con 0x7f73580d1210 2023-10-16T10:00:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73580ceb20 0x7f735815f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73580ceb20 0x7f735815f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53862/0 (socket says 172.21.15.158:53862) 2023-10-16T10:00:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 -- 172.21.15.158:0/3128297347 learned_addr learned my addr 172.21.15.158:0/3128297347 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f73577fe700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f735815d7c0 0x7f73581f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f73567fc700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580d1210 0x7f735815ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 -- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580d1210 msgr2=0x7f735815ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580d1210 0x7f735815ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 -- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f735815d7c0 msgr2=0x7f73581f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f735815d7c0 0x7f73581f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 -- 172.21.15.158:0/3128297347 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7348009550 con 0x7f73580ceb20 2023-10-16T10:00:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f73567fc700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580d1210 0x7f735815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f73577fe700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f735815d7c0 0x7f73581f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f7356ffd700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73580ceb20 0x7f735815f5c0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f734800e9b0 tx=0x7f734800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:04.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f733f7fe700 1 -- 172.21.15.158:0/3128297347 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734800c850 con 0x7f73580ceb20 2023-10-16T10:00:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f733f7fe700 1 -- 172.21.15.158:0/3128297347 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734800c9b0 con 0x7f73580ceb20 2023-10-16T10:00:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f733f7fe700 1 -- 172.21.15.158:0/3128297347 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734800cd20 con 0x7f73580ceb20 2023-10-16T10:00:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7350009a70 con 0x7f73580ceb20 2023-10-16T10:00:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.713+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73581f49a0 con 0x7f73580ceb20 2023-10-16T10:00:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.714+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73581f4b60 con 0x7f73580ceb20 2023-10-16T10:00:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.715+0000 7f733f7fe700 1 -- 172.21.15.158:0/3128297347 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7348013470 con 0x7f73580ceb20 2023-10-16T10:00:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.715+0000 7f733f7fe700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.715+0000 7f733f7fe700 1 -- 172.21.15.158:0/3128297347 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f7348089b40 con 0x7f73580ceb20 2023-10-16T10:00:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.715+0000 7f73567fc700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.716+0000 7f73567fc700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f734406b020 0x7f734406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73580b63f0 tx=0x7f734c007ca0 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:04.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.720+0000 7f733f7fe700 1 -- 172.21.15.158:0/3128297347 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73480547f0 con 0x7f73580ceb20 2023-10-16T10:00:04.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.955+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73581f4d10 con 0x7f73580ceb20 2023-10-16T10:00:04.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.955+0000 7f733f7fe700 1 -- 172.21.15.158:0/3128297347 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f7348057e10 con 0x7f73580ceb20 2023-10-16T10:00:04.957 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:04.957 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6676,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2569267295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2569267295},{"type":"v1","addr":"172.21.15.172:6835","nonce":2569267295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1485896118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1485896118},{"type":"v1","addr":"172.21.15.172:6834","nonce":1485896118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2664141151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2664141151},{"type":"v1","addr":"172.21.15.158:6836","nonce":2664141151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:59.006477+0000","modified":"2023-10-16T10:00:02.034482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7239":{"gid":7239,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/1006858952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1006858952},{"type":"v1","addr":"172.21.15.158:6837","nonce":1006858952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T10:00:04.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f734406b020 msgr2=0x7f734406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f734406b020 0x7f734406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73580b63f0 tx=0x7f734c007ca0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73580ceb20 msgr2=0x7f735815f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:04.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73580ceb20 0x7f735815f5c0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f734800e9b0 tx=0x7f734800ed70 comp rx=0 tx=0).stop 2023-10-16T10:00:04.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 shutdown_connections 2023-10-16T10:00:04.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f735815d7c0 0x7f73581f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73580ceb20 0x7f735815f5c0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 --2- 172.21.15.158:0/3128297347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73580d1210 0x7f735815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:04.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 >> 172.21.15.158:0/3128297347 conn(0x7f73580669f0 msgr2=0x7f73580ca920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:04.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 shutdown_connections 2023-10-16T10:00:04.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:04.957+0000 7f735d447700 1 -- 172.21.15.158:0/3128297347 wait complete. 2023-10-16T10:00:04.961 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 42 2023-10-16T10:00:05.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T09:59:59.006477+0000', 'modified': '2023-10-16T10:00:02.034482+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7239}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7239': {'gid': 7239, 'name': 'a', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6837/1006858952', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6835', 'nonce': 1006858952}, {'type': 'v1', 'addr': '172.21.15.158:6837', 'nonce': 1006858952}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T10:00:05.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T10:00:05.374 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T10:00:05.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.495+0000 7f93dc874700 1 Processor -- start 2023-10-16T10:00:05.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.495+0000 7f93dc874700 1 -- start start 2023-10-16T10:00:05.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.496+0000 7f93dc874700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f93d815d5c0 0x7f93d806bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.496+0000 7f93dc874700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80cfc90 con 0x7f93d815d9a0 2023-10-16T10:00:05.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.496+0000 7f93dc874700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d8161620 con 0x7f93d80b3640 2023-10-16T10:00:05.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.496+0000 7f93dc874700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d8161900 con 0x7f93d815d5c0 2023-10-16T10:00:05.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.496+0000 7f93d659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f93d815d5c0 0x7f93d806bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.496+0000 7f93d5d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f93d815d9a0 0x7f93d80b2f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54050/0 (socket says 172.21.15.158:54050) 2023-10-16T10:00:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.496+0000 7f93d5d9b700 1 -- 172.21.15.158:0/1912810408 learned_addr learned my addr 172.21.15.158:0/1912810408 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d659c700 1 -- 172.21.15.158:0/1912810408 >> v1:172.21.15.172:6789/0 conn(0x7f93d80b3640 legacy=0x7f93d815e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d659c700 1 -- 172.21.15.158:0/1912810408 >> v1:172.21.15.158:6789/0 conn(0x7f93d815d9a0 legacy=0x7f93d80b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d659c700 1 -- 172.21.15.158:0/1912810408 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8161be0 con 0x7f93d815d5c0 2023-10-16T10:00:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d559a700 1 -- 172.21.15.158:0/1912810408 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923687584 0 0) 0x7f93d80cfc90 con 0x7f93d815d9a0 2023-10-16T10:00:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d659c700 1 --2- 172.21.15.158:0/1912810408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d806bdd0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009840 tx=0x7f93cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b4e49a37473735 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d559a700 1 -- 172.21.15.158:0/1912810408 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc004230 con 0x7f93d815d5c0 2023-10-16T10:00:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d559a700 1 -- 172.21.15.158:0/1912810408 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc004390 con 0x7f93d815d5c0 2023-10-16T10:00:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93d559a700 1 -- 172.21.15.158:0/1912810408 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0036a0 con 0x7f93d815d5c0 2023-10-16T10:00:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93dc874700 1 -- 172.21.15.158:0/1912810408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 msgr2=0x7f93d806bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.497+0000 7f93dc874700 1 --2- 172.21.15.158:0/1912810408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d806bdd0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009840 tx=0x7f93cc009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.498+0000 7f93dc874700 1 -- 172.21.15.158:0/1912810408 shutdown_connections 2023-10-16T10:00:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.498+0000 7f93dc874700 1 --2- 172.21.15.158:0/1912810408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d806bdd0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.498+0000 7f93dc874700 1 -- 172.21.15.158:0/1912810408 >> 172.21.15.158:0/1912810408 conn(0x7f93d80cdf30 msgr2=0x7f93d80ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.498+0000 7f93dc874700 1 -- 172.21.15.158:0/1912810408 shutdown_connections 2023-10-16T10:00:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.498+0000 7f93dc874700 1 -- 172.21.15.158:0/1912810408 wait complete. 2023-10-16T10:00:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.499+0000 7f93dc874700 1 Processor -- start 2023-10-16T10:00:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.499+0000 7f93dc874700 1 -- start start 2023-10-16T10:00:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93dc874700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93d80b3640 0x7f93d8158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93dc874700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d80cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93dc874700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93d815d9a0 0x7f93d80cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93dc874700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80cb150 con 0x7f93d815d5c0 2023-10-16T10:00:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93dc874700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80c9530 con 0x7f93d80b3640 2023-10-16T10:00:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93dc874700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80c9810 con 0x7f93d815d9a0 2023-10-16T10:00:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93d5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d80cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93d5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d80cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53878/0 (socket says 172.21.15.158:53878) 2023-10-16T10:00:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.500+0000 7f93d5d9b700 1 -- 172.21.15.158:0/265519867 learned_addr learned my addr 172.21.15.158:0/265519867 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d659c700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93d80b3640 0x7f93d8158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93d815d9a0 0x7f93d80cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d5d9b700 1 -- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93d815d9a0 msgr2=0x7f93d80cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93d815d9a0 0x7f93d80cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d5d9b700 1 -- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93d80b3640 msgr2=0x7f93d8158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93d80b3640 0x7f93d8158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d5d9b700 1 -- 172.21.15.158:0/265519867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc0094a0 con 0x7f93d815d5c0 2023-10-16T10:00:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d659c700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93d80b3640 0x7f93d8158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d80cd1a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c520 tx=0x7f93c800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93c6ffd700 1 -- 172.21.15.158:0/265519867 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8004c80 con 0x7f93d815d5c0 2023-10-16T10:00:05.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93c6ffd700 1 -- 172.21.15.158:0/265519867 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8004de0 con 0x7f93d815d5c0 2023-10-16T10:00:05.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d80c9af0 con 0x7f93d815d5c0 2023-10-16T10:00:05.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93c6ffd700 1 -- 172.21.15.158:0/265519867 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c80072d0 con 0x7f93d815d5c0 2023-10-16T10:00:05.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.501+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d80c9200 con 0x7f93d815d5c0 2023-10-16T10:00:05.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.502+0000 7f93c6ffd700 1 -- 172.21.15.158:0/265519867 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f93c8007430 con 0x7f93d815d5c0 2023-10-16T10:00:05.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.503+0000 7f93c6ffd700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f93b806b020 0x7f93b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.503+0000 7f93c6ffd700 1 -- 172.21.15.158:0/265519867 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f93c8089770 con 0x7f93d815d5c0 2023-10-16T10:00:05.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.503+0000 7f93d659c700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f93b806b020 0x7f93b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.503+0000 7f93d659c700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f93b806b020 0x7f93b806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93cc000c00 tx=0x7f93cc01c040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.505+0000 7f93e8a5eb80 1 Processor -- start 2023-10-16T10:00:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.505+0000 7f93e8a5eb80 1 -- start start 2023-10-16T10:00:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19c4fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c4ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c236c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b19b99c20 con 0x559b19c4d630 2023-10-16T10:00:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b19b7ec10 con 0x559b19c23010 2023-10-16T10:00:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b19b7c530 con 0x559b19c48220 2023-10-16T10:00:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93d5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c4ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93d5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c4ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53884/0 (socket says 172.21.15.158:53884) 2023-10-16T10:00:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.506+0000 7f93d5d9b700 1 -- 172.21.15.158:0/1988537289 learned_addr learned my addr 172.21.15.158:0/1988537289 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d659c700 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19c4fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c236c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d5d9b700 1 -- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 msgr2=0x559b19c4fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19c4fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d5d9b700 1 -- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 msgr2=0x559b19c236c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c236c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d5d9b700 1 -- 172.21.15.158:0/1988537289 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b19b353a0 con 0x559b19c4d630 2023-10-16T10:00:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c4ffb0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f93c8001eb0 tx=0x7f93c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16376f3fc9f440f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93b7fff700 1 -- 172.21.15.158:0/1988537289 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c80580e0 con 0x559b19c4d630 2023-10-16T10:00:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93b7fff700 1 -- 172.21.15.158:0/1988537289 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c805c230 con 0x559b19c4d630 2023-10-16T10:00:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.507+0000 7f93b7fff700 1 -- 172.21.15.158:0/1988537289 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c80603e0 con 0x559b19c4d630 2023-10-16T10:00:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1988537289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 msgr2=0x559b19c4ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c4ffb0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f93c8001eb0 tx=0x7f93c800cf90 comp rx=0 tx=0).stop 2023-10-16T10:00:05.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1988537289 shutdown_connections 2023-10-16T10:00:05.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c236c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19c4fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1988537289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c4ffb0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1988537289 >> 172.21.15.158:0/1988537289 conn(0x559b19b52d80 msgr2=0x559b19c6dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1988537289 shutdown_connections 2023-10-16T10:00:05.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1988537289 wait complete. 2023-10-16T10:00:05.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 -- start start 2023-10-16T10:00:05.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.508+0000 7f93e8a5eb80 1 Processor -- start 2023-10-16T10:00:05.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c52dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c48220 0x559b19c51a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c4d630 0x559b19c49080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b19b77e20 con 0x559b19c48220 2023-10-16T10:00:05.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b19b146c0 con 0x559b19c23010 2023-10-16T10:00:05.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b19b14360 con 0x559b19c4d630 2023-10-16T10:00:05.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93e8a5eb80 10 client.0 _ll_register_callbacks cb 0x559b199717b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:00:05.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93d5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c48220 0x559b19c51a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93d5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c48220 0x559b19c51a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53892/0 (socket says 172.21.15.158:53892) 2023-10-16T10:00:05.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.509+0000 7f93d5d9b700 1 -- 172.21.15.158:0/897757622 learned_addr learned my addr 172.21.15.158:0/897757622 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:05.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c4d630 0x559b19c49080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d659c700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c52dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d5d9b700 1 -- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c4d630 msgr2=0x559b19c49080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c4d630 0x559b19c49080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d5d9b700 1 -- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 msgr2=0x559b19c52dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c52dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d5d9b700 1 -- 172.21.15.158:0/897757622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d80c9200 con 0x559b19c48220 2023-10-16T10:00:05.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c4d630 0x559b19c49080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:05.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c48220 0x559b19c51a50 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f93c8087ca0 tx=0x7f93c800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6c2846e3e2ca0a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:00:05.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93b57fa700 1 -- 172.21.15.158:0/897757622 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c80580e0 con 0x559b19c48220 2023-10-16T10:00:05.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93b57fa700 1 -- 172.21.15.158:0/897757622 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f93c80603e0 con 0x559b19c48220 2023-10-16T10:00:05.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b19c185b0 con 0x559b19c48220 2023-10-16T10:00:05.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93b57fa700 1 -- 172.21.15.158:0/897757622 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93c80575d0 con 0x559b19c48220 2023-10-16T10:00:05.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93e8a5eb80 10 client.7377 fetch_fsmap learned FSMap version 42 2023-10-16T10:00:05.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x559b199a2d90 con 0x559b19c48220 2023-10-16T10:00:05.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93b57fa700 1 -- 172.21.15.158:0/897757622 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8073020 con 0x559b19c48220 2023-10-16T10:00:05.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93b57fa700 1 -- 172.21.15.158:0/897757622 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 42) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f93c805ff50 con 0x559b19c48220 2023-10-16T10:00:05.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93e8a5eb80 10 client.7377 fetch_fsmap finished waiting for FSMap version 42 2023-10-16T10:00:05.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.510+0000 7f93e8a5eb80 10 client.7377 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T10:00:05.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93e8a5eb80 20 client.7377 populate_metadata read hostname 'smithi158' 2023-10-16T10:00:05.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c23c10 0x559b19c23ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93e8a5eb80 4 client.7377 mds_command: new command op to 7239 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T10:00:05.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 --> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b19b70e00 con 0x559b19c23c10 2023-10-16T10:00:05.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93d659c700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c23c10 0x559b19c23ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93b57fa700 10 client.7377 ms_handle_connect on v2:172.21.15.158:6835/1006858952 2023-10-16T10:00:05.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93d659c700 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c23c10 0x559b19c23ff0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72f8750edfba4183 server_cookie=80b6d34242f626a6 in_seq=0 out_seq=0 2023-10-16T10:00:05.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93b57fa700 1 -- 172.21.15.158:0/897757622 <== mds.0 v2:172.21.15.158:6835/1006858952 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559b19b70e00 con 0x559b19c23c10 2023-10-16T10:00:05.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93b57fa700 10 client.7377 handle_command_reply: tid=0 2023-10-16T10:00:05.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.511+0000 7f93e8a5eb80 1 client.7377 shutdown 2023-10-16T10:00:05.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c48220 msgr2=0x559b19c51a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c48220 0x559b19c51a50 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f93c8087ca0 tx=0x7f93c800f000 comp rx=0 tx=0).stop 2023-10-16T10:00:05.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 shutdown_connections 2023-10-16T10:00:05.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c23c10 0x559b19c23ff0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c4d630 0x559b19c49080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93b57fa700 0 client.7377 ms_handle_reset on v2:172.21.15.158:6835/1006858952 2023-10-16T10:00:05.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c23010 0x559b19c52dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/897757622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c48220 0x559b19c51a50 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 >> 172.21.15.158:0/897757622 conn(0x559b19b52d80 msgr2=0x559b19c6df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 shutdown_connections 2023-10-16T10:00:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/897757622 wait complete. 2023-10-16T10:00:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.512+0000 7f93e8a5eb80 20 client.7377 trim_cache size 0 max 16384 2023-10-16T10:00:05.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.529+0000 7f93e8a5eb80 1 Processor -- start 2023-10-16T10:00:05.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.529+0000 7f93e8a5eb80 1 -- start start 2023-10-16T10:00:05.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c23010 0x559b19c6b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c48220 0x559b19c23ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c52dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b199f88d0 con 0x559b19c4d630 2023-10-16T10:00:05.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b19c84470 con 0x559b19c48220 2023-10-16T10:00:05.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b19c4a770 con 0x559b19c23010 2023-10-16T10:00:05.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c52dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c52dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53902/0 (socket says 172.21.15.158:53902) 2023-10-16T10:00:05.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 -- 172.21.15.158:0/1006636242 learned_addr learned my addr 172.21.15.158:0/1006636242 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:05.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d659c700 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c23010 0x559b19c6b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c48220 0x559b19c23ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 -- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c23010 msgr2=0x559b19c6b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c23010 0x559b19c6b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 -- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c48220 msgr2=0x559b19c23ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c48220 0x559b19c23ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 -- 172.21.15.158:0/1006636242 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b1998bc10 con 0x559b19c4d630 2023-10-16T10:00:05.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c48220 0x559b19c23ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:05.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.530+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c52dd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f93d00058c0 tx=0x7f93d0012540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d3765572aefca9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93c4ff9700 1 -- 172.21.15.158:0/1006636242 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0012aa0 con 0x559b19c4d630 2023-10-16T10:00:05.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93c4ff9700 1 -- 172.21.15.158:0/1006636242 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d0012c00 con 0x559b19c4d630 2023-10-16T10:00:05.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93c4ff9700 1 -- 172.21.15.158:0/1006636242 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0016440 con 0x559b19c4d630 2023-10-16T10:00:05.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1006636242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 msgr2=0x559b19c52dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c52dd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f93d00058c0 tx=0x7f93d0012540 comp rx=0 tx=0).stop 2023-10-16T10:00:05.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1006636242 shutdown_connections 2023-10-16T10:00:05.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c4d630 0x559b19c52dd0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c23010 0x559b19c6b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/1006636242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c48220 0x559b19c23ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1006636242 >> 172.21.15.158:0/1006636242 conn(0x559b19b52d80 msgr2=0x559b19c524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1006636242 shutdown_connections 2023-10-16T10:00:05.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.531+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/1006636242 wait complete. 2023-10-16T10:00:05.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.532+0000 7f93e8a5eb80 1 -- start start 2023-10-16T10:00:05.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.532+0000 7f93e8a5eb80 1 Processor -- start 2023-10-16T10:00:05.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c23010 0x559b19c23ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19cbad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c4d630 0x559b19c518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b19c89610 con 0x559b19c23010 2023-10-16T10:00:05.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b19c8ac40 con 0x559b19c4d630 2023-10-16T10:00:05.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b19c245f0 con 0x559b19c48220 2023-10-16T10:00:05.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 10 client.0 _ll_register_callbacks cb 0x559b199717b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:00:05.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c23010 0x559b19c23ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c23010 0x559b19c23ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53904/0 (socket says 172.21.15.158:53904) 2023-10-16T10:00:05.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 -- 172.21.15.158:0/459566060 learned_addr learned my addr 172.21.15.158:0/459566060 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:05.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19cbad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d6d9d700 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c4d630 0x559b19c518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 -- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 msgr2=0x559b19cbad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19cbad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 -- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c4d630 msgr2=0x559b19c518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c4d630 0x559b19c518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 -- 172.21.15.158:0/459566060 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d000f040 con 0x559b19c23010 2023-10-16T10:00:05.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93d659c700 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c23010 0x559b19c23ff0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f93cc004000 tx=0x7f93cc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e204ef7bdc01c3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:00:05.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93b57fa700 1 -- 172.21.15.158:0/459566060 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc003e40 con 0x559b19c23010 2023-10-16T10:00:05.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93b57fa700 1 -- 172.21.15.158:0/459566060 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f93cc03cda0 con 0x559b19c23010 2023-10-16T10:00:05.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.533+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93c805c230 con 0x559b19c23010 2023-10-16T10:00:05.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93b57fa700 1 -- 172.21.15.158:0/459566060 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93cc033a00 con 0x559b19c23010 2023-10-16T10:00:05.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 10 client.7383 fetch_fsmap learned FSMap version 42 2023-10-16T10:00:05.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x559b199c9200 con 0x559b19c23010 2023-10-16T10:00:05.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93b57fa700 1 -- 172.21.15.158:0/459566060 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c805c230 con 0x559b19c23010 2023-10-16T10:00:05.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93b57fa700 1 -- 172.21.15.158:0/459566060 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 42) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f93cc039020 con 0x559b19c23010 2023-10-16T10:00:05.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 10 client.7383 fetch_fsmap finished waiting for FSMap version 42 2023-10-16T10:00:05.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 10 client.7383 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T10:00:05.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 20 client.7383 populate_metadata read hostname 'smithi158' 2023-10-16T10:00:05.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c796e0 0x559b19c79ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 4 client.7383 mds_command: new command op to 7239 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T10:00:05.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 --> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559b19b70b20 con 0x559b19c796e0 2023-10-16T10:00:05.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c796e0 0x559b19c79ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93b57fa700 10 client.7383 ms_handle_connect on v2:172.21.15.158:6835/1006858952 2023-10-16T10:00:05.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93d5d9b700 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c796e0 0x559b19c79ac0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9631d8faec738f29 server_cookie=a8dc198af0b43d94 in_seq=0 out_seq=0 2023-10-16T10:00:05.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93b57fa700 1 -- 172.21.15.158:0/459566060 <== mds.0 v2:172.21.15.158:6835/1006858952 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559b19b70b20 con 0x559b19c796e0 2023-10-16T10:00:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93b57fa700 10 client.7383 handle_command_reply: tid=0 2023-10-16T10:00:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.534+0000 7f93e8a5eb80 1 client.7383 shutdown 2023-10-16T10:00:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c23010 msgr2=0x559b19c23ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c23010 0x559b19c23ff0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f93cc004000 tx=0x7f93cc0047c0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 shutdown_connections 2023-10-16T10:00:05.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:6835/1006858952,v1:172.21.15.158:6837/1006858952] conn(0x559b19c796e0 0x559b19c79ac0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559b19c4d630 0x559b19c518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x559b19c23010 0x559b19c23ff0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 --2- 172.21.15.158:0/459566060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x559b19c48220 0x559b19cbad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93b57fa700 0 client.7383 ms_handle_reset on v2:172.21.15.158:6835/1006858952 2023-10-16T10:00:05.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 >> 172.21.15.158:0/459566060 conn(0x559b19b52d80 msgr2=0x559b19c52450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 shutdown_connections 2023-10-16T10:00:05.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 1 -- 172.21.15.158:0/459566060 wait complete. 2023-10-16T10:00:05.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93e8a5eb80 20 client.7383 trim_cache size 0 max 16384 2023-10-16T10:00:05.561 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:00:05.562 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T10:00:05.562 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T10:00:05.562 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 7239, 2023-10-16T10:00:05.562 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T10:00:05.563 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T10:00:05.563 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T10:00:05.563 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9180705119999999, 2023-10-16T10:00:05.563 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 42, 2023-10-16T10:00:05.563 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 98, 2023-10-16T10:00:05.564 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 98, 2023-10-16T10:00:05.564 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.868496799000001 2023-10-16T10:00:05.564 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:00:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f93b806b020 msgr2=0x7f93b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f93b806b020 0x7f93b806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93cc000c00 tx=0x7f93cc01c040 comp rx=0 tx=0).stop 2023-10-16T10:00:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 msgr2=0x7f93d80cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d80cd1a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c520 tx=0x7f93c800c8e0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 shutdown_connections 2023-10-16T10:00:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f93b806b020 0x7f93b806d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93d815d9a0 0x7f93d80cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93d80b3640 0x7f93d8158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 --2- 172.21.15.158:0/265519867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93d815d5c0 0x7f93d80cd1a0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 >> 172.21.15.158:0/265519867 conn(0x7f93d80cdf30 msgr2=0x7f93d81607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 shutdown_connections 2023-10-16T10:00:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.535+0000 7f93dc874700 1 -- 172.21.15.158:0/265519867 wait complete. 2023-10-16T10:00:05.568 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T10:00:05.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.690+0000 7f9c2901e700 1 Processor -- start 2023-10-16T10:00:05.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.690+0000 7f9c2901e700 1 -- start start 2023-10-16T10:00:05.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2901e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406ab40 0x7f9c2406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2901e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c240729b0 0x7f9c24071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2901e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c24071e50 con 0x7f9c2406e5d0 2023-10-16T10:00:05.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2901e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c24161270 con 0x7f9c240729b0 2023-10-16T10:00:05.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2901e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c24161510 con 0x7f9c2406ab40 2023-10-16T10:00:05.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9c2406e5d0 0x7f9c240722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54088/0 (socket says 172.21.15.158:54088) 2023-10-16T10:00:05.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2259c700 1 -- 172.21.15.158:0/330879697 learned_addr learned my addr 172.21.15.158:0/330879697 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:05.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c2359e700 1 --2- 172.21.15.158:0/330879697 >> v2:172.21.15.172:3300/0 conn(0x7f9c240729b0 0x7f9c24071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.691+0000 7f9c22d9d700 1 --2- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406ab40 0x7f9c2406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c21d9b700 1 -- 172.21.15.158:0/330879697 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072788006 0 0) 0x7f9c24071e50 con 0x7f9c2406e5d0 2023-10-16T10:00:05.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c21d9b700 1 -- 172.21.15.158:0/330879697 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c0c0035b0 con 0x7f9c2406e5d0 2023-10-16T10:00:05.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c2359e700 1 -- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406ab40 msgr2=0x7f9c2406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:05.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c2359e700 1 --2- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406ab40 0x7f9c2406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c2359e700 1 -- 172.21.15.158:0/330879697 >> v1:172.21.15.158:6789/0 conn(0x7f9c2406e5d0 legacy=0x7f9c240722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c2359e700 1 -- 172.21.15.158:0/330879697 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c241617f0 con 0x7f9c240729b0 2023-10-16T10:00:05.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c22d9d700 1 --2- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406ab40 0x7f9c2406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:05.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c2359e700 1 --2- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c240729b0 0x7f9c24071910 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009d90 tx=0x7f9c1c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bfb9836d5a09071 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c21d9b700 1 -- 172.21.15.158:0/330879697 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c00e070 con 0x7f9c240729b0 2023-10-16T10:00:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c21d9b700 1 -- 172.21.15.158:0/330879697 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c00bb20 con 0x7f9c240729b0 2023-10-16T10:00:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.692+0000 7f9c21d9b700 1 -- 172.21.15.158:0/330879697 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c012430 con 0x7f9c240729b0 2023-10-16T10:00:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 -- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c240729b0 msgr2=0x7f9c24071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 --2- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c240729b0 0x7f9c24071910 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009d90 tx=0x7f9c1c00b330 comp rx=0 tx=0).stop 2023-10-16T10:00:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 -- 172.21.15.158:0/330879697 shutdown_connections 2023-10-16T10:00:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 --2- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c240729b0 0x7f9c24071910 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 --2- 172.21.15.158:0/330879697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406ab40 0x7f9c2406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 -- 172.21.15.158:0/330879697 >> 172.21.15.158:0/330879697 conn(0x7f9c240d02b0 msgr2=0x7f9c240d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 -- 172.21.15.158:0/330879697 shutdown_connections 2023-10-16T10:00:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.693+0000 7f9c2901e700 1 -- 172.21.15.158:0/330879697 wait complete. 2023-10-16T10:00:05.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.694+0000 7f9c2901e700 1 Processor -- start 2023-10-16T10:00:05.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.694+0000 7f9c2901e700 1 -- start start 2023-10-16T10:00:05.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2901e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2406ab40 0x7f9c240c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2901e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406e5d0 0x7f9c240c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2901e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c240729b0 0x7f9c240c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2901e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c240c7ae0 con 0x7f9c240729b0 2023-10-16T10:00:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2901e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c24145f50 con 0x7f9c2406ab40 2023-10-16T10:00:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2901e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c240b30d0 con 0x7f9c2406e5d0 2023-10-16T10:00:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c240729b0 0x7f9c240c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c240729b0 0x7f9c240c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53922/0 (socket says 172.21.15.158:53922) 2023-10-16T10:00:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 -- 172.21.15.158:0/2800049492 learned_addr learned my addr 172.21.15.158:0/2800049492 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2259c700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406e5d0 0x7f9c240c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c22d9d700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2406ab40 0x7f9c240c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 -- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406e5d0 msgr2=0x7f9c240c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406e5d0 0x7f9c240c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 -- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2406ab40 msgr2=0x7f9c240c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2406ab40 0x7f9c240c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.695+0000 7f9c2359e700 1 -- 172.21.15.158:0/2800049492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c180094a0 con 0x7f9c240729b0 2023-10-16T10:00:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.696+0000 7f9c2359e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c240729b0 0x7f9c240c5810 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c014040 tx=0x7f9c1c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.696+0000 7f9c0b7fe700 1 -- 172.21.15.158:0/2800049492 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0129f0 con 0x7f9c240729b0 2023-10-16T10:00:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.696+0000 7f9c0b7fe700 1 -- 172.21.15.158:0/2800049492 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c020070 con 0x7f9c240729b0 2023-10-16T10:00:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.696+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c009a70 con 0x7f9c240729b0 2023-10-16T10:00:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.696+0000 7f9c0b7fe700 1 -- 172.21.15.158:0/2800049492 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c012db0 con 0x7f9c240729b0 2023-10-16T10:00:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.696+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c240c7420 con 0x7f9c240729b0 2023-10-16T10:00:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.697+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c24162070 con 0x7f9c240729b0 2023-10-16T10:00:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.697+0000 7f9c0b7fe700 1 -- 172.21.15.158:0/2800049492 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9c1c004040 con 0x7f9c240729b0 2023-10-16T10:00:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.698+0000 7f9c0b7fe700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9c1006b020 0x7f9c1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.698+0000 7f9c0b7fe700 1 -- 172.21.15.158:0/2800049492 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9c1c022070 con 0x7f9c240729b0 2023-10-16T10:00:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.698+0000 7f9c22d9d700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9c1006b020 0x7f9c1006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.698+0000 7f9c22d9d700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9c1006b020 0x7f9c1006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009d00 tx=0x7f9c18008040 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:05.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.702+0000 7f9c0b7fe700 1 -- 172.21.15.158:0/2800049492 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c24162070 con 0x7f9c240729b0 2023-10-16T10:00:05.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.941+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9c24162070 con 0x7f9c240729b0 2023-10-16T10:00:05.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.942+0000 7f9c0b7fe700 1 -- 172.21.15.158:0/2800049492 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9c24162070 con 0x7f9c240729b0 2023-10-16T10:00:05.943 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T10:00:05.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9c1006b020 msgr2=0x7f9c1006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9c1006b020 0x7f9c1006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009d00 tx=0x7f9c18008040 comp rx=0 tx=0).stop 2023-10-16T10:00:05.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c240729b0 msgr2=0x7f9c240c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:05.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c240729b0 0x7f9c240c5810 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c014040 tx=0x7f9c1c004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 shutdown_connections 2023-10-16T10:00:05.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f9c1006b020 0x7f9c1006d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c240729b0 0x7f9c240c5810 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2406ab40 0x7f9c240c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 --2- 172.21.15.158:0/2800049492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c2406e5d0 0x7f9c240c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:05.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 >> 172.21.15.158:0/2800049492 conn(0x7f9c240d02b0 msgr2=0x7f9c240ad140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:05.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 shutdown_connections 2023-10-16T10:00:05.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:05.944+0000 7f9c2901e700 1 -- 172.21.15.158:0/2800049492 wait complete. 2023-10-16T10:00:05.957 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T10:00:05.958 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T10:00:05.958 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T10:00:05.958 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T10:00:05.959 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:05.959 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T10:00:05.959 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T10:00:05.960 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T10:00:05.960 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:00:05.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:05.961 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T10:00:05.978 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T10:00:05.978 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T10:00:05.979 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T10:00:05.979 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:05.979 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T10:00:05.979 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:05.979 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:00:05.980 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:05.980 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:00:05.980 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:05.980 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:00:05.980 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:05.981 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T10:00:05.981 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:05.981 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T10:00:05.981 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 41426sec preferred_lft 41426sec 2023-10-16T10:00:05.981 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T10:00:05.981 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:05.982 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T10:00:05.982 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:05.982 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T10:00:05.982 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:05.982 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T10:00:05.983 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:05.983 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:00:05.983 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T10:00:05.983 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T10:00:05.983 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:05.984 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:00:05.984 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T10:00:05.984 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T10:00:05.984 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:05.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:05.985 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T10:00:06.044 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T10:00:06.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:06.046 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T10:00:06.105 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T10:00:06.105 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T10:00:06.105 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T10:00:06.105 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T10:00:06.106 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T10:00:06.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:06.107 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T10:00:06.107 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T10:00:06.107 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T10:00:06.108 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T10:00:06.108 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T10:00:06.108 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T10:00:06.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:06.210 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T10:00:06.227 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T10:00:06.227 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T10:00:06.227 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:00:06.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:06.228 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T10:00:06.293 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:00:06.294 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:06.350 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:00:06.351 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:00:06.351 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:06.407 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T10:00:06.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:06.408 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:06.493 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:06.494 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:00:06.521 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:06.521 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:00:06.521 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:00:06.521 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:00:06.522 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:00:06.522 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:00:06.522 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:00:06.522 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:00:06.599 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:06.599 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:00:06.599 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:00:06.600 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:00:06.600 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:00:06.600 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:00:06.600 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:00:06.600 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:00:06.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T10:00:06.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T10:00:06.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T10:00:06.602 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T10:00:06.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:06.603 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:06.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:06.631 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T10:00:06.717 INFO:teuthology.orchestra.run.smithi158.stdout:client36077 2023-10-16T10:00:06.717 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:00:06.718 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:06.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T10:00:06.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.896+0000 7f56a9a20700 1 Processor -- start 2023-10-16T10:00:06.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.896+0000 7f56a9a20700 1 -- start start 2023-10-16T10:00:06.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a9a20700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56a415d990 0x7f56a40b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:06.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a9a20700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f56a40cf650 0x7f56a40cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:06.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a9a20700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40af600 con 0x7f56a406e480 2023-10-16T10:00:06.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a9a20700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a40d0020 con 0x7f56a415d990 2023-10-16T10:00:06.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a9a20700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a40b2c90 con 0x7f56a40cf650 2023-10-16T10:00:06.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56a415d990 0x7f56a40b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:06.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56a415d990 0x7f56a40b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53930/0 (socket says 172.21.15.158:53930) 2023-10-16T10:00:06.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a2ffd700 1 -- 172.21.15.158:0/2361299839 learned_addr learned my addr 172.21.15.158:0/2361299839 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:06.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.897+0000 7f56a37fe700 1 --2- 172.21.15.158:0/2361299839 >> v2:172.21.15.172:3300/0 conn(0x7f56a40cf650 0x7f56a40cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:06.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a2ffd700 1 -- 172.21.15.158:0/2361299839 >> v2:172.21.15.172:3300/0 conn(0x7f56a40cf650 msgr2=0x7f56a40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:06.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a2ffd700 1 --2- 172.21.15.158:0/2361299839 >> v2:172.21.15.172:3300/0 conn(0x7f56a40cf650 0x7f56a40cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:06.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a2ffd700 1 -- 172.21.15.158:0/2361299839 >> v1:172.21.15.172:6789/0 conn(0x7f56a406e480 legacy=0x7f56a40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:06.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a2ffd700 1 -- 172.21.15.158:0/2361299839 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a406e9b0 con 0x7f56a415d990 2023-10-16T10:00:06.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a2ffd700 1 --2- 172.21.15.158:0/2361299839 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40b2750 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f56a4070bb0 tx=0x7f5698009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92ab1c87f545c4b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:06.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a1ffb700 1 -- 172.21.15.158:0/2361299839 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698004230 con 0x7f56a415d990 2023-10-16T10:00:06.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a1ffb700 1 -- 172.21.15.158:0/2361299839 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5698004390 con 0x7f56a415d990 2023-10-16T10:00:06.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a1ffb700 1 -- 172.21.15.158:0/2361299839 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56980036a0 con 0x7f56a415d990 2023-10-16T10:00:06.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a9a20700 1 -- 172.21.15.158:0/2361299839 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 msgr2=0x7f56a40b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:06.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2361299839 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40b2750 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f56a4070bb0 tx=0x7f5698009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:06.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a9a20700 1 -- 172.21.15.158:0/2361299839 shutdown_connections 2023-10-16T10:00:06.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2361299839 >> v2:172.21.15.172:3300/0 conn(0x7f56a40cf650 0x7f56a40cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:06.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2361299839 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40b2750 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:06.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.898+0000 7f56a9a20700 1 -- 172.21.15.158:0/2361299839 >> 172.21.15.158:0/2361299839 conn(0x7f56a406b9e0 msgr2=0x7f56a406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:06.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.899+0000 7f56a9a20700 1 -- 172.21.15.158:0/2361299839 shutdown_connections 2023-10-16T10:00:06.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.899+0000 7f56a9a20700 1 -- 172.21.15.158:0/2361299839 wait complete. 2023-10-16T10:00:06.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.900+0000 7f56a9a20700 1 Processor -- start 2023-10-16T10:00:06.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.900+0000 7f56a9a20700 1 -- start start 2023-10-16T10:00:06.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a9a20700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56a406e480 0x7f56a4151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:06.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a9a20700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a40cf650 0x7f56a40ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:06.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a9a20700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:06.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a9a20700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a415b810 con 0x7f56a415d990 2023-10-16T10:00:06.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a9a20700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40ca720 con 0x7f56a40cf650 2023-10-16T10:00:06.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a9a20700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40c9680 con 0x7f56a406e480 2023-10-16T10:00:06.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:06.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40cd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53938/0 (socket says 172.21.15.158:53938) 2023-10-16T10:00:06.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a37fe700 1 -- 172.21.15.158:0/2866926200 learned_addr learned my addr 172.21.15.158:0/2866926200 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:06.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a27fc700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a40cf650 0x7f56a40ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:06.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.901+0000 7f56a2ffd700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56a406e480 0x7f56a4151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:06.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a37fe700 1 -- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56a406e480 msgr2=0x7f56a4151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:06.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a37fe700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56a406e480 0x7f56a4151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:06.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a37fe700 1 -- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a40cf650 msgr2=0x7f56a40ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:06.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a37fe700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a40cf650 0x7f56a40ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:06.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a37fe700 1 -- 172.21.15.158:0/2866926200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5698009550 con 0x7f56a415d990 2023-10-16T10:00:06.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a37fe700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40cd220 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f569c007e40 tx=0x7f569c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:06.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f568bfff700 1 -- 172.21.15.158:0/2866926200 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c00ed30 con 0x7f56a415d990 2023-10-16T10:00:06.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f568bfff700 1 -- 172.21.15.158:0/2866926200 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c014420 con 0x7f56a415d990 2023-10-16T10:00:06.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a40c9960 con 0x7f56a415d990 2023-10-16T10:00:06.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f568bfff700 1 -- 172.21.15.158:0/2866926200 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c0147d0 con 0x7f56a415d990 2023-10-16T10:00:06.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.902+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a40c9f20 con 0x7f56a415d990 2023-10-16T10:00:06.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.903+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a40681b0 con 0x7f56a415d990 2023-10-16T10:00:06.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.903+0000 7f568bfff700 1 -- 172.21.15.158:0/2866926200 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f569c014930 con 0x7f56a415d990 2023-10-16T10:00:06.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.904+0000 7f568bfff700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f569006b050 0x7f569006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:06.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.904+0000 7f56a2ffd700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f569006b050 0x7f569006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:06.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.904+0000 7f568bfff700 1 -- 172.21.15.158:0/2866926200 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f569c08a0d0 con 0x7f56a415d990 2023-10-16T10:00:06.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.904+0000 7f56a2ffd700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f569006b050 0x7f569006d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5698009130 tx=0x7f5698008030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:06.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:06.908+0000 7f568bfff700 1 -- 172.21.15.158:0/2866926200 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f569c054d80 con 0x7f56a415d990 2023-10-16T10:00:07.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.128+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 --> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f56a40ceb70 con 0x7f569006b050 2023-10-16T10:00:07.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.130+0000 7f568bfff700 1 -- 172.21.15.158:0/2866926200 <== mgr.7005 v2:172.21.15.158:6832/3798429240 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f56980041b0 con 0x7f569006b050 2023-10-16T10:00:07.131 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:00:07.131 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:07.132 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T10:00:07.132 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:07.132 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:00:07.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f569006b050 msgr2=0x7f569006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f569006b050 0x7f569006d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5698009130 tx=0x7f5698008030 comp rx=0 tx=0).stop 2023-10-16T10:00:07.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 msgr2=0x7f56a40cd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40cd220 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f569c007e40 tx=0x7f569c00e3c0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 shutdown_connections 2023-10-16T10:00:07.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f569006b050 0x7f569006d510 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56a415d990 0x7f56a40cd220 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56a406e480 0x7f56a4151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 --2- 172.21.15.158:0/2866926200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a40cf650 0x7f56a40ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 >> 172.21.15.158:0/2866926200 conn(0x7f56a406b9e0 msgr2=0x7f56a415cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:07.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 shutdown_connections 2023-10-16T10:00:07.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.132+0000 7f56a9a20700 1 -- 172.21.15.158:0/2866926200 wait complete. 2023-10-16T10:00:07.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T10:00:07.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.267+0000 7f7b1e606700 1 Processor -- start 2023-10-16T10:00:07.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.267+0000 7f7b1e606700 1 -- start start 2023-10-16T10:00:07.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.267+0000 7f7b1e606700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b1815da40 0x7f7b180caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b1e606700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b180cbd30 con 0x7f7b1806afe0 2023-10-16T10:00:07.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b1e606700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b180cc1c0 con 0x7f7b180cc590 2023-10-16T10:00:07.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b1e606700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b1815de20 con 0x7f7b1815da40 2023-10-16T10:00:07.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b1cba3700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7b1806afe0 0x7f7b180afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54122/0 (socket says 172.21.15.158:54122) 2023-10-16T10:00:07.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b1cba3700 1 -- 172.21.15.158:0/3870799156 learned_addr learned my addr 172.21.15.158:0/3870799156 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:07.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145595310 0 0) 0x7f7b180cbd30 con 0x7f7b1806afe0 2023-10-16T10:00:07.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b000035b0 con 0x7f7b1806afe0 2023-10-16T10:00:07.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b17fff700 1 --2- 172.21.15.158:0/3870799156 >> v2:172.21.15.172:3300/0 conn(0x7f7b1815da40 0x7f7b180caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2124212723 0 0) 0x7f7b000035b0 con 0x7f7b1806afe0 2023-10-16T10:00:07.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 >> v2:172.21.15.172:3300/0 conn(0x7f7b1815da40 msgr2=0x7f7b180caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:07.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 --2- 172.21.15.158:0/3870799156 >> v2:172.21.15.172:3300/0 conn(0x7f7b1815da40 0x7f7b180caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 >> v1:172.21.15.172:6789/0 conn(0x7f7b180cc590 legacy=0x7f7b1806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b180ce260 con 0x7f7b1806afe0 2023-10-16T10:00:07.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7b100019f0 con 0x7f7b1806afe0 2023-10-16T10:00:07.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 >> v1:172.21.15.158:6789/0 conn(0x7f7b1806afe0 legacy=0x7f7b180afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b00003a00 0x7f7b00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b00004850 0x7f7b00004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00005210 0x7f7b0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1815de20 con 0x7f7b00003a00 2023-10-16T10:00:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0000c0b0 con 0x7f7b00005210 2023-10-16T10:00:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0000c270 con 0x7f7b00004850 2023-10-16T10:00:07.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.268+0000 7f7b17fff700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b00003a00 0x7f7b00003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b1cba3700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00005210 0x7f7b0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b177fe700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b00004850 0x7f7b00004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b17fff700 1 -- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b00004850 msgr2=0x7f7b00004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b17fff700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b00004850 0x7f7b00004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b17fff700 1 -- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00005210 msgr2=0x7f7b0000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b17fff700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00005210 0x7f7b0000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b17fff700 1 -- 172.21.15.158:0/3870799156 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b000035b0 con 0x7f7b00003a00 2023-10-16T10:00:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b1cba3700 1 -- 172.21.15.158:0/3870799156 reap_dead start 2023-10-16T10:00:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b177fe700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b00004850 0x7f7b00004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b17fff700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b00003a00 0x7f7b00003e20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009b60 tx=0x7f7b00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3caf0cb81e1cd861 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0800e6c0 con 0x7f7b00003a00 2023-10-16T10:00:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0800e820 con 0x7f7b00003a00 2023-10-16T10:00:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b16ffd700 1 -- 172.21.15.158:0/3870799156 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0800eb90 con 0x7f7b00003a00 2023-10-16T10:00:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b1e606700 1 -- 172.21.15.158:0/3870799156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b00003a00 msgr2=0x7f7b00003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b1e606700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b00003a00 0x7f7b00003e20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009b60 tx=0x7f7b00005880 comp rx=0 tx=0).stop 2023-10-16T10:00:07.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b1e606700 1 -- 172.21.15.158:0/3870799156 shutdown_connections 2023-10-16T10:00:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b1e606700 1 --2- 172.21.15.158:0/3870799156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b00003a00 0x7f7b00003e20 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.269+0000 7f7b1e606700 1 -- 172.21.15.158:0/3870799156 >> 172.21.15.158:0/3870799156 conn(0x7f7b18099a10 msgr2=0x7f7b180d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.270+0000 7f7b1e606700 1 -- 172.21.15.158:0/3870799156 shutdown_connections 2023-10-16T10:00:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.270+0000 7f7b1e606700 1 -- 172.21.15.158:0/3870799156 wait complete. 2023-10-16T10:00:07.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.271+0000 7f7b1e606700 1 Processor -- start 2023-10-16T10:00:07.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.271+0000 7f7b1e606700 1 -- start start 2023-10-16T10:00:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00003a00 0x7f7b1814e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b180c4c50 0x7f7b1815fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b1815ff60 0x7f7b180c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b181603e0 con 0x7f7b1815ff60 2023-10-16T10:00:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b180c3ed0 con 0x7f7b00003a00 2023-10-16T10:00:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b180c4170 con 0x7f7b180c4c50 2023-10-16T10:00:07.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b1815ff60 0x7f7b180c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b1815ff60 0x7f7b180c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53958/0 (socket says 172.21.15.158:53958) 2023-10-16T10:00:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 -- 172.21.15.158:0/284338505 learned_addr learned my addr 172.21.15.158:0/284338505 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b177fe700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b180c4c50 0x7f7b1815fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b17fff700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00003a00 0x7f7b1814e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 -- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b180c4c50 msgr2=0x7f7b1815fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b180c4c50 0x7f7b1815fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 -- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00003a00 msgr2=0x7f7b1814e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00003a00 0x7f7b1814e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 -- 172.21.15.158:0/284338505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b180c4450 con 0x7f7b1815ff60 2023-10-16T10:00:07.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b17fff700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00003a00 0x7f7b1814e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:07.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b177fe700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b180c4c50 0x7f7b1815fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:07.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1cba3700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b1815ff60 0x7f7b180c9620 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7b10002230 tx=0x7f7b1000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:07.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b14ff9700 1 -- 172.21.15.158:0/284338505 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1000add0 con 0x7f7b1815ff60 2023-10-16T10:00:07.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b14ff9700 1 -- 172.21.15.158:0/284338505 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b100133f0 con 0x7f7b1815ff60 2023-10-16T10:00:07.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b14ff9700 1 -- 172.21.15.158:0/284338505 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b100137a0 con 0x7f7b1815ff60 2023-10-16T10:00:07.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b180c4730 con 0x7f7b1815ff60 2023-10-16T10:00:07.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.272+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1814ec90 con 0x7f7b1815ff60 2023-10-16T10:00:07.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.273+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b180c4920 con 0x7f7b1815ff60 2023-10-16T10:00:07.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.274+0000 7f7b14ff9700 1 -- 172.21.15.158:0/284338505 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 42) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b10013900 con 0x7f7b1815ff60 2023-10-16T10:00:07.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.274+0000 7f7b14ff9700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7b0406b020 0x7f7b0406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.275+0000 7f7b17fff700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7b0406b020 0x7f7b0406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.275+0000 7f7b14ff9700 1 -- 172.21.15.158:0/284338505 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f7b1008a6e0 con 0x7f7b1815ff60 2023-10-16T10:00:07.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.275+0000 7f7b17fff700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7b0406b020 0x7f7b0406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b080090b0 tx=0x7f7b08008030 comp rx=0 tx=0).ready entity=mgr.7005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:07.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.279+0000 7f7b14ff9700 1 -- 172.21.15.158:0/284338505 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b10055390 con 0x7f7b1815ff60 2023-10-16T10:00:07.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.531+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7b180cb6b0 con 0x7f7b1815ff60 2023-10-16T10:00:07.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.652+0000 7f7b14ff9700 1 -- 172.21.15.158:0/284338505 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v43) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7b100589b0 con 0x7f7b1815ff60 2023-10-16T10:00:07.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.656+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7b0406b020 msgr2=0x7f7b0406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.656+0000 7f7b1e606700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7b0406b020 0x7f7b0406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b080090b0 tx=0x7f7b08008030 comp rx=0 tx=0).stop 2023-10-16T10:00:07.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.656+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b1815ff60 msgr2=0x7f7b180c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.656+0000 7f7b1e606700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b1815ff60 0x7f7b180c9620 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7b10002230 tx=0x7f7b1000a620 comp rx=0 tx=0).stop 2023-10-16T10:00:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 shutdown_connections 2023-10-16T10:00:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f7b0406b020 0x7f7b0406d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b1815ff60 0x7f7b180c9620 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b00003a00 0x7f7b1814e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 --2- 172.21.15.158:0/284338505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b180c4c50 0x7f7b1815fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 >> 172.21.15.158:0/284338505 conn(0x7f7b18099a10 msgr2=0x7f7b1815f260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 shutdown_connections 2023-10-16T10:00:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.657+0000 7f7b1e606700 1 -- 172.21.15.158:0/284338505 wait complete. 2023-10-16T10:00:07.672 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T10:00:07.794 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:07.793+0000 7f839345c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:00:07.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.799+0000 7f9321a2f700 1 Processor -- start 2023-10-16T10:00:07.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.799+0000 7f9321a2f700 1 -- start start 2023-10-16T10:00:07.800 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:07.799+0000 7f144f8df200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:00:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f9321a2f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f931c160d30 0x7f931c0b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f9321a2f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c0d0530 0x7f931c0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f9321a2f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931c0b7c90 con 0x7f931c0d19a0 2023-10-16T10:00:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f9321a2f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c0b8910 con 0x7f931c160d30 2023-10-16T10:00:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f9321a2f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0d0ef0 con 0x7f931c0d0530 2023-10-16T10:00:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f931c160d30 0x7f931c0b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f931c0d19a0 0x7f931c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54168/0 (socket says 172.21.15.158:54168) 2023-10-16T10:00:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f931c160d30 0x7f931c0b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53988/0 (socket says 172.21.15.158:53988) 2023-10-16T10:00:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931b7fe700 1 -- 172.21.15.158:0/1076216554 learned_addr learned my addr 172.21.15.158:0/1076216554 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931a7fc700 1 --2- 172.21.15.158:0/1076216554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c0d0530 0x7f931c0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931affd700 1 -- 172.21.15.158:0/1076216554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c0d0530 msgr2=0x7f931c0d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931affd700 1 --2- 172.21.15.158:0/1076216554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c0d0530 0x7f931c0d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931affd700 1 -- 172.21.15.158:0/1076216554 >> v1:172.21.15.158:6789/0 conn(0x7f931c0d19a0 legacy=0x7f931c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931affd700 1 -- 172.21.15.158:0/1076216554 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0b7f70 con 0x7f931c160d30 2023-10-16T10:00:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f9319ffb700 1 -- 172.21.15.158:0/1076216554 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559805734 0 0) 0x7f931c0b7c90 con 0x7f931c0d19a0 2023-10-16T10:00:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.800+0000 7f931affd700 1 --2- 172.21.15.158:0/1076216554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c160d30 0x7f931c0b8f50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f930c009840 tx=0x7f930c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558ded705b8b649 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9319ffb700 1 -- 172.21.15.158:0/1076216554 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c011830 con 0x7f931c160d30 2023-10-16T10:00:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9319ffb700 1 -- 172.21.15.158:0/1076216554 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c011990 con 0x7f931c160d30 2023-10-16T10:00:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9319ffb700 1 -- 172.21.15.158:0/1076216554 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c011d20 con 0x7f931c160d30 2023-10-16T10:00:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 -- 172.21.15.158:0/1076216554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c160d30 msgr2=0x7f931c0b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 --2- 172.21.15.158:0/1076216554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c160d30 0x7f931c0b8f50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f930c009840 tx=0x7f930c009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 -- 172.21.15.158:0/1076216554 shutdown_connections 2023-10-16T10:00:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 --2- 172.21.15.158:0/1076216554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c160d30 0x7f931c0b8f50 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 --2- 172.21.15.158:0/1076216554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c0d0530 0x7f931c0d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 -- 172.21.15.158:0/1076216554 >> 172.21.15.158:0/1076216554 conn(0x7f931c0669f0 msgr2=0x7f931c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:07.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 -- 172.21.15.158:0/1076216554 shutdown_connections 2023-10-16T10:00:07.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.801+0000 7f9321a2f700 1 -- 172.21.15.158:0/1076216554 wait complete. 2023-10-16T10:00:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.802+0000 7f9321a2f700 1 Processor -- start 2023-10-16T10:00:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.803+0000 7f9321a2f700 1 -- start start 2023-10-16T10:00:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.803+0000 7f9321a2f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c0d0530 0x7f931c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.803+0000 7f9321a2f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931c0d19a0 0x7f931c0c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.803+0000 7f9321a2f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c160d30 0x7f931c0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.803+0000 7f9321a2f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0c9f80 con 0x7f931c0d0530 2023-10-16T10:00:07.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.803+0000 7f9321a2f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c147310 con 0x7f931c0d19a0 2023-10-16T10:00:07.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.803+0000 7f9321a2f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0cb790 con 0x7f931c160d30 2023-10-16T10:00:07.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c0d0530 0x7f931c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c0d0530 0x7f931c0c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:53994/0 (socket says 172.21.15.158:53994) 2023-10-16T10:00:07.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 -- 172.21.15.158:0/3228025362 learned_addr learned my addr 172.21.15.158:0/3228025362 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931a7fc700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931c0d19a0 0x7f931c0c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931b7fe700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c160d30 0x7f931c0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 -- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c160d30 msgr2=0x7f931c0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c160d30 0x7f931c0c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 -- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931c0d19a0 msgr2=0x7f931c0c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931c0d19a0 0x7f931c0c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 -- 172.21.15.158:0/3228025362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0094a0 con 0x7f931c0d0530 2023-10-16T10:00:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f931affd700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c0d0530 0x7f931c0c70f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f930c009b90 tx=0x7f930c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f9303fff700 1 -- 172.21.15.158:0/3228025362 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c0036a0 con 0x7f931c0d0530 2023-10-16T10:00:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f9303fff700 1 -- 172.21.15.158:0/3228025362 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c01e070 con 0x7f931c0d0530 2023-10-16T10:00:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f9303fff700 1 -- 172.21.15.158:0/3228025362 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c003a60 con 0x7f931c0d0530 2023-10-16T10:00:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c0cc310 con 0x7f931c0d0530 2023-10-16T10:00:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.804+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c0cbcc0 con 0x7f931c0d0530 2023-10-16T10:00:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.805+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c0681b0 con 0x7f931c0d0530 2023-10-16T10:00:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.806+0000 7f9303fff700 1 -- 172.21.15.158:0/3228025362 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 43) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f930c003bc0 con 0x7f931c0d0530 2023-10-16T10:00:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.806+0000 7f9303fff700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 0x7f930806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.806+0000 7f931a7fc700 1 -- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 msgr2=0x7f930806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.806+0000 7f9303fff700 1 -- 172.21.15.158:0/3228025362 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f930c08a8b0 con 0x7f931c0d0530 2023-10-16T10:00:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.806+0000 7f931a7fc700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 0x7f930806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:07.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:07.811+0000 7f9303fff700 1 -- 172.21.15.158:0/3228025362 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f930c0555f0 con 0x7f931c0d0530 2023-10-16T10:00:07.862 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:07.861+0000 7f839345c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:07.870 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:07.869+0000 7f144f8df200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:07.934 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:07.933+0000 7f839345c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:00:07.944 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:07.943+0000 7f144f8df200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:00:08.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.007+0000 7f931a7fc700 1 -- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 msgr2=0x7f930806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:08.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.007+0000 7f931a7fc700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 0x7f930806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:08.012 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:08.010+0000 7f839345c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:00:08.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.016+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f931c0b5bf0 con 0x7f931c0d0530 2023-10-16T10:00:08.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.020+0000 7f9303fff700 1 -- 172.21.15.158:0/3228025362 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f930c0200d0 con 0x7f931c0d0530 2023-10-16T10:00:08.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 msgr2=0x7f930806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:08.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 0x7f930806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c0d0530 msgr2=0x7f931c0c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c0d0530 0x7f931c0c70f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f930c009b90 tx=0x7f930c004750 comp rx=0 tx=0).stop 2023-10-16T10:00:08.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 shutdown_connections 2023-10-16T10:00:08.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f930806b070 0x7f930806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931c160d30 0x7f931c0c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f931c0d0530 0x7f931c0c70f0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 --2- 172.21.15.158:0/3228025362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931c0d19a0 0x7f931c0c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 >> 172.21.15.158:0/3228025362 conn(0x7f931c0669f0 msgr2=0x7f931c0d3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:08.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 shutdown_connections 2023-10-16T10:00:08.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.023+0000 7f9321a2f700 1 -- 172.21.15.158:0/3228025362 wait complete. 2023-10-16T10:00:08.028 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:08.025+0000 7f144f8df200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:00:08.037 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T10:00:08.154 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:08.153+0000 7f839345c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:00:08.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.162+0000 7f395101f700 1 Processor -- start 2023-10-16T10:00:08.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.162+0000 7f395101f700 1 -- start start 2023-10-16T10:00:08.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f395101f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f394c15b6b0 0x7f394c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f395101f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f394c0d0c60 0x7f394c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f395101f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f395101f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394c15ba90 con 0x7f394c15b6b0 2023-10-16T10:00:08.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f395101f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394c0b38f0 con 0x7f394c0d0c60 2023-10-16T10:00:08.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f395101f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c0b4420 con 0x7f394c0b3460 2023-10-16T10:00:08.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f394ad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f394c15b6b0 0x7f394c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f394ad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f394c15b6b0 0x7f394c0cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54024/0 (socket says 172.21.15.158:54024) 2023-10-16T10:00:08.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f394ad9d700 1 -- 172.21.15.158:0/1868409021 learned_addr learned my addr 172.21.15.158:0/1868409021 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:08.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f394b59e700 1 --2- 172.21.15.158:0/1868409021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.163+0000 7f394a59c700 1 --2- 172.21.15.158:0/1868409021 >> v2:172.21.15.172:3300/0 conn(0x7f394c0d0c60 0x7f394c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f394ad9d700 1 -- 172.21.15.158:0/1868409021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 msgr2=0x7f394c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:08.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f394ad9d700 1 --2- 172.21.15.158:0/1868409021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f394ad9d700 1 -- 172.21.15.158:0/1868409021 >> v2:172.21.15.172:3300/0 conn(0x7f394c0d0c60 msgr2=0x7f394c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:08.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f394ad9d700 1 --2- 172.21.15.158:0/1868409021 >> v2:172.21.15.172:3300/0 conn(0x7f394c0d0c60 0x7f394c0b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f394ad9d700 1 -- 172.21.15.158:0/1868409021 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c0cbc20 con 0x7f394c15b6b0 2023-10-16T10:00:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f394ad9d700 1 --2- 172.21.15.158:0/1868409021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0cfdb0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f394c0b7610 tx=0x7f393c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61de8d712cb9b13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f3949d9b700 1 -- 172.21.15.158:0/1868409021 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c011830 con 0x7f394c15b6b0 2023-10-16T10:00:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f3949d9b700 1 -- 172.21.15.158:0/1868409021 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c011990 con 0x7f394c15b6b0 2023-10-16T10:00:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.164+0000 7f3949d9b700 1 -- 172.21.15.158:0/1868409021 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c011d20 con 0x7f394c15b6b0 2023-10-16T10:00:08.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 -- 172.21.15.158:0/1868409021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 msgr2=0x7f394c0cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 --2- 172.21.15.158:0/1868409021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0cfdb0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f394c0b7610 tx=0x7f393c009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 -- 172.21.15.158:0/1868409021 shutdown_connections 2023-10-16T10:00:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 --2- 172.21.15.158:0/1868409021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 --2- 172.21.15.158:0/1868409021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0cfdb0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 --2- 172.21.15.158:0/1868409021 >> v2:172.21.15.172:3300/0 conn(0x7f394c0d0c60 0x7f394c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 -- 172.21.15.158:0/1868409021 >> 172.21.15.158:0/1868409021 conn(0x7f394c0cdf20 msgr2=0x7f394c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:08.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 -- 172.21.15.158:0/1868409021 shutdown_connections 2023-10-16T10:00:08.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.165+0000 7f395101f700 1 -- 172.21.15.158:0/1868409021 wait complete. 2023-10-16T10:00:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.167+0000 7f395101f700 1 Processor -- start 2023-10-16T10:00:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.167+0000 7f395101f700 1 -- start start 2023-10-16T10:00:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.167+0000 7f395101f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c0c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.167+0000 7f395101f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f394c0d0c60 0x7f394c1f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f395101f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f395101f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c0c9c30 con 0x7f394c15b6b0 2023-10-16T10:00:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f395101f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c0c7880 con 0x7f394c0d0c60 2023-10-16T10:00:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f395101f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c1f4d90 con 0x7f394c0b3460 2023-10-16T10:00:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c0c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f394c0d0c60 0x7f394c1f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54034/0 (socket says 172.21.15.158:54034) 2023-10-16T10:00:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c0c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:36632/0 (socket says 172.21.15.158:36632) 2023-10-16T10:00:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 -- 172.21.15.158:0/1656095739 learned_addr learned my addr 172.21.15.158:0/1656095739 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 -- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 msgr2=0x7f394c0c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c0c7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 -- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f394c0d0c60 msgr2=0x7f394c1f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f394c0d0c60 0x7f394c1f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 -- 172.21.15.158:0/1656095739 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c009550 con 0x7f394c15b6b0 2023-10-16T10:00:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394ad9d700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c0c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:08.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394a59c700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f394c0d0c60 0x7f394c1f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:08.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f394b59e700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0b50c0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f394400eed0 tx=0x7f394400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:08.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f39337fe700 1 -- 172.21.15.158:0/1656095739 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944010040 con 0x7f394c15b6b0 2023-10-16T10:00:08.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f39337fe700 1 -- 172.21.15.158:0/1656095739 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394400b9f0 con 0x7f394c15b6b0 2023-10-16T10:00:08.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c0c9020 con 0x7f394c15b6b0 2023-10-16T10:00:08.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f39337fe700 1 -- 172.21.15.158:0/1656095739 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394400bda0 con 0x7f394c15b6b0 2023-10-16T10:00:08.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.168+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c0c95e0 con 0x7f394c15b6b0 2023-10-16T10:00:08.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.169+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394c0681b0 con 0x7f394c15b6b0 2023-10-16T10:00:08.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.170+0000 7f39337fe700 1 -- 172.21.15.158:0/1656095739 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 43) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3944014400 con 0x7f394c15b6b0 2023-10-16T10:00:08.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.170+0000 7f39337fe700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 0x7f393806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.171+0000 7f394ad9d700 1 -- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 msgr2=0x7f393806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.171+0000 7f39337fe700 1 -- 172.21.15.158:0/1656095739 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f3944089e20 con 0x7f394c15b6b0 2023-10-16T10:00:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.171+0000 7f394ad9d700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 0x7f393806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:08.183 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:08.174+0000 7f144f8df200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:00:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.175+0000 7f39337fe700 1 -- 172.21.15.158:0/1656095739 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f39440549b0 con 0x7f394c15b6b0 2023-10-16T10:00:08.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.372+0000 7f394ad9d700 1 -- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 msgr2=0x7f393806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:08.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.372+0000 7f394ad9d700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 0x7f393806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:08.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.377+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f394c0c9d70 con 0x7f394c15b6b0 2023-10-16T10:00:08.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.381+0000 7f39337fe700 1 -- 172.21.15.158:0/1656095739 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f39440180c0 con 0x7f394c15b6b0 2023-10-16T10:00:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 msgr2=0x7f393806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 0x7f393806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 msgr2=0x7f394c0b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0b50c0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f394400eed0 tx=0x7f394400ef00 comp rx=0 tx=0).stop 2023-10-16T10:00:08.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 shutdown_connections 2023-10-16T10:00:08.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f393806b070 0x7f393806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f394c15b6b0 0x7f394c0b50c0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f394c0b3460 0x7f394c0c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 --2- 172.21.15.158:0/1656095739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f394c0d0c60 0x7f394c1f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 >> 172.21.15.158:0/1656095739 conn(0x7f394c0cdf20 msgr2=0x7f394c15dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:08.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 shutdown_connections 2023-10-16T10:00:08.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.383+0000 7f395101f700 1 -- 172.21.15.158:0/1656095739 wait complete. 2023-10-16T10:00:08.397 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T10:00:08.481 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:08.479+0000 7f839345c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:00:08.517 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:08.515+0000 7f144f8df200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:00:08.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.521+0000 7f29b3745700 1 Processor -- start 2023-10-16T10:00:08.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.521+0000 7f29b3745700 1 -- start start 2023-10-16T10:00:08.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b3745700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac15deb0 0x7f29ac0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b3745700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac0d0ae0 con 0x7f29ac15dad0 2023-10-16T10:00:08.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b3745700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac0d12d0 con 0x7f29ac0ccc00 2023-10-16T10:00:08.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b3745700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac0cbdf0 con 0x7f29ac15deb0 2023-10-16T10:00:08.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b14e1700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f29ac15dad0 0x7f29ac0cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54244/0 (socket says 172.21.15.158:54244) 2023-10-16T10:00:08.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b14e1700 1 -- 172.21.15.158:0/1658725683 learned_addr learned my addr 172.21.15.158:0/1658725683 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:08.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1658725683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac15deb0 0x7f29ac0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29a3fff700 1 -- 172.21.15.158:0/1658725683 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523343136 0 0) 0x7f29ac0d0ae0 con 0x7f29ac15dad0 2023-10-16T10:00:08.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29a3fff700 1 -- 172.21.15.158:0/1658725683 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29980035b0 con 0x7f29ac15dad0 2023-10-16T10:00:08.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29a3fff700 1 -- 172.21.15.158:0/1658725683 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725051699 0 0) 0x7f29ac0d12d0 con 0x7f29ac0ccc00 2023-10-16T10:00:08.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29a3fff700 1 -- 172.21.15.158:0/1658725683 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ac0d0ae0 con 0x7f29ac0ccc00 2023-10-16T10:00:08.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1658725683 >> v1:172.21.15.172:6789/0 conn(0x7f29ac0ccc00 legacy=0x7f29ac0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1658725683 >> v1:172.21.15.158:6789/0 conn(0x7f29ac15dad0 legacy=0x7f29ac0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.522+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1658725683 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac0cc280 con 0x7f29ac15deb0 2023-10-16T10:00:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1658725683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac15deb0 0x7f29ac0ce6b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f299c009870 tx=0x7f299c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf440c531c180a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29a3fff700 1 -- 172.21.15.158:0/1658725683 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c011830 con 0x7f29ac15deb0 2023-10-16T10:00:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29a3fff700 1 -- 172.21.15.158:0/1658725683 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c011990 con 0x7f29ac15deb0 2023-10-16T10:00:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29a3fff700 1 -- 172.21.15.158:0/1658725683 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c011d20 con 0x7f29ac15deb0 2023-10-16T10:00:08.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29b3745700 1 -- 172.21.15.158:0/1658725683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac15deb0 msgr2=0x7f29ac0ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29b3745700 1 --2- 172.21.15.158:0/1658725683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac15deb0 0x7f29ac0ce6b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f299c009870 tx=0x7f299c009b80 comp rx=0 tx=0).stop 2023-10-16T10:00:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29b3745700 1 -- 172.21.15.158:0/1658725683 shutdown_connections 2023-10-16T10:00:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29b3745700 1 --2- 172.21.15.158:0/1658725683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac15deb0 0x7f29ac0ce6b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.523+0000 7f29b3745700 1 -- 172.21.15.158:0/1658725683 >> 172.21.15.158:0/1658725683 conn(0x7f29ac099a10 msgr2=0x7f29ac0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:08.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.524+0000 7f29b3745700 1 -- 172.21.15.158:0/1658725683 shutdown_connections 2023-10-16T10:00:08.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.524+0000 7f29b3745700 1 -- 172.21.15.158:0/1658725683 wait complete. 2023-10-16T10:00:08.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.525+0000 7f29b3745700 1 Processor -- start 2023-10-16T10:00:08.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.525+0000 7f29b3745700 1 -- start start 2023-10-16T10:00:08.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b3745700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac0ccc00 0x7f29ac0c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b3745700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29ac15dad0 0x7f29ac06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b3745700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29ac15deb0 0x7f29ac0c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b3745700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac0c8360 con 0x7f29ac15deb0 2023-10-16T10:00:08.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b3745700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac0c8640 con 0x7f29ac15dad0 2023-10-16T10:00:08.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b3745700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac0c8920 con 0x7f29ac0ccc00 2023-10-16T10:00:08.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b1ce2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29ac15deb0 0x7f29ac0c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b14e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac0ccc00 0x7f29ac0c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b1ce2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29ac15deb0 0x7f29ac0c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54078/0 (socket says 172.21.15.158:54078) 2023-10-16T10:00:08.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b1ce2700 1 -- 172.21.15.158:0/1198117717 learned_addr learned my addr 172.21.15.158:0/1198117717 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:08.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29ac15dad0 0x7f29ac06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:08.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b14e1700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29ac15dad0 msgr2=0x7f29ac06ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b14e1700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29ac15dad0 0x7f29ac06ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b14e1700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29ac15deb0 msgr2=0x7f29ac0c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:08.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b14e1700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29ac15deb0 0x7f29ac0c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:08.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b14e1700 1 -- 172.21.15.158:0/1198117717 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c0094a0 con 0x7f29ac0ccc00 2023-10-16T10:00:08.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b1ce2700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29ac15deb0 0x7f29ac0c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:08.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b14e1700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29ac0ccc00 0x7f29ac0c9400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f29a800c4d0 tx=0x7f29a800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:08.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8005d70 con 0x7f29ac0ccc00 2023-10-16T10:00:08.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a80070d0 con 0x7f29ac0ccc00 2023-10-16T10:00:08.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29b3745700 1 -- 172.21.15.158:0/1198117717 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac0c8c00 con 0x7f29ac0ccc00 2023-10-16T10:00:08.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.526+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8007440 con 0x7f29ac0ccc00 2023-10-16T10:00:08.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.527+0000 7f29b3745700 1 -- 172.21.15.158:0/1198117717 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac0b5fb0 con 0x7f29ac0ccc00 2023-10-16T10:00:08.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.528+0000 7f29b3745700 1 -- 172.21.15.158:0/1198117717 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ac0b6140 con 0x7f29ac0ccc00 2023-10-16T10:00:08.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.528+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 43) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f29a80075a0 con 0x7f29ac0ccc00 2023-10-16T10:00:08.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.528+0000 7f29a1ffb700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 0x7f299006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:08.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.529+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 msgr2=0x7f299006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:08.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.529+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f29a8088d40 con 0x7f29ac0ccc00 2023-10-16T10:00:08.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.529+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 0x7f299006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:08.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.534+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29a8053b60 con 0x7f29ac0ccc00 2023-10-16T10:00:08.598 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:08.597+0000 7f839345c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:00:08.639 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:08.638+0000 7f144f8df200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:00:08.672 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:08.670+0000 7f839345c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:00:08.714 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:08.713+0000 7f144f8df200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:00:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.730+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 msgr2=0x7f299006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.730+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 0x7f299006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:08.751 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:08.750+0000 7f839345c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:00:08.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:08.765+0000 7f29b3745700 1 -- 172.21.15.158:0/1198117717 --> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f29ac0d0ff0 con 0x7f299006b070 2023-10-16T10:00:08.797 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:08.796+0000 7f144f8df200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:00:09.123 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:09.121+0000 7f839345c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:00:09.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:09.130+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 msgr2=0x7f299006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:09.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:09.130+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 0x7f299006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T10:00:09.166 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:09.164+0000 7f144f8df200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:00:09.693 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:09.692+0000 7f839345c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:00:09.734 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:09.732+0000 7f144f8df200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:00:09.762 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:09.762+0000 7f839345c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:00:09.803 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:09.801+0000 7f144f8df200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:00:09.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:09.931+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 msgr2=0x7f299006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:09.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:09.931+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 0x7f299006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T10:00:10.392 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:10.392+0000 7f839345c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:10.433 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:10.431+0000 7f144f8df200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:10.468 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:10.468+0000 7f839345c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:00:10.511 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:10.508+0000 7f144f8df200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:00:10.611 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:10.611+0000 7f839345c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:10.655 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:10.653+0000 7f144f8df200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:10.836 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:10.836+0000 7f839345c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:00:10.883 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:10.881+0000 7f144f8df200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:00:10.909 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:10.909+0000 7f839345c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:00:10.957 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:10.955+0000 7f144f8df200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:00:11.175 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:11.174+0000 7f839345c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:00:11.221 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:11.219+0000 7f144f8df200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:00:11.392 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:11.392+0000 7f839345c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:00:11.440 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:11.438+0000 7f144f8df200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:00:11.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:11.532+0000 7f29b0ce0700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 msgr2=0x7f299006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3798429240 2023-10-16T10:00:11.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:11.532+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 0x7f299006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T10:00:11.540 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:11.540+0000 7f839345c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:00:11.589 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:11.586+0000 7f144f8df200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:00:11.654 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:11.654+0000 7f839345c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:00:11.701 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:11.699+0000 7f144f8df200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:00:11.806 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:11.806+0000 7f839345c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:00:11.856 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:11.854+0000 7f144f8df200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:00:12.069 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:12.069+0000 7f839345c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:12.118 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.115+0000 7f144f8df200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:12.228 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:12.227+0000 7f839345c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:00:12.278 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.276+0000 7f144f8df200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:00:12.750 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:12.750+0000 7f839345c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:00:12.802 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.800+0000 7f144f8df200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:00:12.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:12.818+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mgrmap(e 44) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f29a8086f70 con 0x7f29ac0ccc00 2023-10-16T10:00:12.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:12.832+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 8 ==== mgrmap(e 45) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f29a8075ee0 con 0x7f29ac0ccc00 2023-10-16T10:00:12.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:12.833+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 msgr2=0x7f299006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:12.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:12.833+0000 7f29a1ffb700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/3798429240,v1:172.21.15.158:6833/3798429240] conn(0x7f299006b070 0x7f299006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:12.984 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c28ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.985 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c28ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.985 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c28ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.985 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c28ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.985 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c28ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.985 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c58c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.986 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c58c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.986 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c58c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.986 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c58c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:12.986 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:12.982+0000 7f13c58c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:13.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.837+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mon.2 v2:172.21.15.172:3301/0 9 ==== mgrmap(e 46) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f29a8054800 con 0x7f29ac0ccc00 2023-10-16T10:00:13.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.838+0000 7f29a1ffb700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f29900279c0 0x7f2990027da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:13.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.838+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 --> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2990071580 con 0x7f29900279c0 2023-10-16T10:00:13.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.838+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f29900279c0 0x7f2990027da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:13.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.841+0000 7f29b0ce0700 1 --2- 172.21.15.158:0/1198117717 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f29900279c0 0x7f2990027da0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f299c0090b0 tx=0x7f299c01c040 comp rx=0 tx=0).ready entity=mgr.7407 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:13.850 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:13.848+0000 7f13cc90e700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T10:00:13.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.848+0000 7f29a1ffb700 1 -- 172.21.15.158:0/1198117717 <== mgr.7407 v2:172.21.15.158:6832/334523996 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f2990071580 con 0x7f29900279c0 2023-10-16T10:00:13.852 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T10:00:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T10:00:13.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T10:00:13.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ead9ec700 1 Processor -- start 2023-10-16T10:00:13.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ead9ec700 1 -- start start 2023-10-16T10:00:13.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ead9ec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8ea8160d30 0x7f8ea80b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:13.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ead9ec700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea80d0530 0x7f8ea80d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:13.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ead9ec700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea80b7c90 con 0x7f8ea80d19a0 2023-10-16T10:00:13.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ead9ec700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea80b8910 con 0x7f8ea8160d30 2023-10-16T10:00:13.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ead9ec700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea80d0ef0 con 0x7f8ea80d0530 2023-10-16T10:00:13.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ea6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8ea8160d30 0x7f8ea80b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:13.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ea6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8ea8160d30 0x7f8ea80b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54242/0 (socket says 172.21.15.158:54242) 2023-10-16T10:00:13.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.983+0000 7f8ea6ffd700 1 -- 172.21.15.158:0/250378039 learned_addr learned my addr 172.21.15.158:0/250378039 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:13.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea67fc700 1 --2- 172.21.15.158:0/250378039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea80d0530 0x7f8ea80d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:13.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea6ffd700 1 -- 172.21.15.158:0/250378039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea80d0530 msgr2=0x7f8ea80d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:13.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea6ffd700 1 --2- 172.21.15.158:0/250378039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea80d0530 0x7f8ea80d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:13.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea6ffd700 1 -- 172.21.15.158:0/250378039 >> v1:172.21.15.172:6789/0 conn(0x7f8ea80d19a0 legacy=0x7f8ea80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:13.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea6ffd700 1 -- 172.21.15.158:0/250378039 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea80b7f70 con 0x7f8ea8160d30 2023-10-16T10:00:13.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea6ffd700 1 --2- 172.21.15.158:0/250378039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea8160d30 0x7f8ea80b8f50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0098f0 tx=0x7f8e9c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02334050dceee05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:13.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea5ffb700 1 -- 172.21.15.158:0/250378039 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c00b8a0 con 0x7f8ea8160d30 2023-10-16T10:00:13.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea5ffb700 1 -- 172.21.15.158:0/250378039 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c00ba00 con 0x7f8ea8160d30 2023-10-16T10:00:13.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.984+0000 7f8ea5ffb700 1 -- 172.21.15.158:0/250378039 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c00bd90 con 0x7f8ea8160d30 2023-10-16T10:00:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 -- 172.21.15.158:0/250378039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea8160d30 msgr2=0x7f8ea80b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/250378039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea8160d30 0x7f8ea80b8f50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0098f0 tx=0x7f8e9c009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 -- 172.21.15.158:0/250378039 shutdown_connections 2023-10-16T10:00:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/250378039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea8160d30 0x7f8ea80b8f50 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/250378039 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea80d0530 0x7f8ea80d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 -- 172.21.15.158:0/250378039 >> 172.21.15.158:0/250378039 conn(0x7f8ea80669f0 msgr2=0x7f8ea8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 -- 172.21.15.158:0/250378039 shutdown_connections 2023-10-16T10:00:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.985+0000 7f8ead9ec700 1 -- 172.21.15.158:0/250378039 wait complete. 2023-10-16T10:00:13.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.986+0000 7f8ead9ec700 1 Processor -- start 2023-10-16T10:00:13.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.986+0000 7f8ead9ec700 1 -- start start 2023-10-16T10:00:13.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ead9ec700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80d0530 0x7f8ea81f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:13.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ead9ec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea80d19a0 0x7f8ea8148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:13.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ead9ec700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea8160d30 0x7f8ea8156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:13.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ead9ec700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea80c9f20 con 0x7f8ea80d19a0 2023-10-16T10:00:13.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ead9ec700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea81588d0 con 0x7f8ea80d0530 2023-10-16T10:00:13.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ead9ec700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8158bb0 con 0x7f8ea8160d30 2023-10-16T10:00:13.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea80d19a0 0x7f8ea8148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:13.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea80d19a0 0x7f8ea8148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54250/0 (socket says 172.21.15.158:54250) 2023-10-16T10:00:13.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 -- 172.21.15.158:0/682781584 learned_addr learned my addr 172.21.15.158:0/682781584 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:13.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea6ffd700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80d0530 0x7f8ea81f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:13.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea77fe700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea8160d30 0x7f8ea8156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:13.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 -- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea8160d30 msgr2=0x7f8ea8156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:13.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea8160d30 0x7f8ea8156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:13.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 -- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80d0530 msgr2=0x7f8ea81f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:13.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80d0530 0x7f8ea81f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:13.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 -- 172.21.15.158:0/682781584 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c009550 con 0x7f8ea80d19a0 2023-10-16T10:00:13.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.987+0000 7f8ea67fc700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea80d19a0 0x7f8ea8148610 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800e810 tx=0x7f8e9800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:13.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.988+0000 7f8e8ffff700 1 -- 172.21.15.158:0/682781584 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9800c070 con 0x7f8ea80d19a0 2023-10-16T10:00:13.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.988+0000 7f8e8ffff700 1 -- 172.21.15.158:0/682781584 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98013610 con 0x7f8ea80d19a0 2023-10-16T10:00:13.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.988+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea8158e90 con 0x7f8ea80d19a0 2023-10-16T10:00:13.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.988+0000 7f8e8ffff700 1 -- 172.21.15.158:0/682781584 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e980139c0 con 0x7f8ea80d19a0 2023-10-16T10:00:13.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.988+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea8148b50 con 0x7f8ea80d19a0 2023-10-16T10:00:13.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.989+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea8148d70 con 0x7f8ea80d19a0 2023-10-16T10:00:13.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.989+0000 7f8e8ffff700 1 -- 172.21.15.158:0/682781584 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 46) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f8e98013b20 con 0x7f8ea80d19a0 2023-10-16T10:00:14.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.989+0000 7f8e8ffff700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f8e9406ada0 0x7f8e9406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:14.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.990+0000 7f8ea6ffd700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f8e9406ada0 0x7f8e9406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:14.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.990+0000 7f8e8ffff700 1 -- 172.21.15.158:0/682781584 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f8e98089430 con 0x7f8ea80d19a0 2023-10-16T10:00:14.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.990+0000 7f8ea6ffd700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f8e9406ada0 0x7f8e9406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c014040 tx=0x7f8e9c008030 comp rx=0 tx=0).ready entity=mgr.7407 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:14.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:13.994+0000 7f8e8ffff700 1 -- 172.21.15.158:0/682781584 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8ea8148d70 con 0x7f8ea80d19a0 2023-10-16T10:00:14.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.246+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f8ea8148d70 con 0x7f8ea80d19a0 2023-10-16T10:00:14.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.837+0000 7f8e8ffff700 1 -- 172.21.15.158:0/682781584 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v47) v1 ==== 93+0+0 (secure 0 0 0) 0x7f8ea8148d70 con 0x7f8ea80d19a0 2023-10-16T10:00:14.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.840+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f8e9406ada0 msgr2=0x7f8e9406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:14.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.840+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f8e9406ada0 0x7f8e9406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c014040 tx=0x7f8e9c008030 comp rx=0 tx=0).stop 2023-10-16T10:00:14.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.840+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea80d19a0 msgr2=0x7f8ea8148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:14.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.840+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea80d19a0 0x7f8ea8148610 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800e810 tx=0x7f8e9800eb20 comp rx=0 tx=0).stop 2023-10-16T10:00:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.841+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 shutdown_connections 2023-10-16T10:00:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.841+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f8e9406ada0 0x7f8e9406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.841+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ea8160d30 0x7f8ea8156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.841+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80d0530 0x7f8ea81f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.841+0000 7f8ead9ec700 1 --2- 172.21.15.158:0/682781584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea80d19a0 0x7f8ea8148610 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.841+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 >> 172.21.15.158:0/682781584 conn(0x7f8ea80669f0 msgr2=0x7f8ea8066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.842+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 shutdown_connections 2023-10-16T10:00:14.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:14.843+0000 7f8ead9ec700 1 -- 172.21.15.158:0/682781584 wait complete. 2023-10-16T10:00:14.861 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:00:14.862 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:00:14.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:00:14.953 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:00:14.953 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:00:14.954 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:14.971 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:00:14.972 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:14.991 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:14.991+0000 7fc5dee94200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:00:14.994 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:14.992+0000 7f2e09c28200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:00:15.011 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:15.057 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.056+0000 7fc5dee94200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:15.063 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:15.061+0000 7f2e09c28200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:15.129 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.129+0000 7fc5dee94200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:00:15.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:15.136+0000 7f2e09c28200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:00:15.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.173+0000 7fd76e3bd700 1 Processor -- start 2023-10-16T10:00:15.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.173+0000 7fd76e3bd700 1 -- start start 2023-10-16T10:00:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd76e3bd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7680d1210 0x7fd7680ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd76e3bd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd7680ce950 0x7fd7680b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd76e3bd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7680d3bd0 con 0x7fd76815d7c0 2023-10-16T10:00:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd76e3bd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7680d4060 con 0x7fd7680d1210 2023-10-16T10:00:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd76e3bd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7680d2fc0 con 0x7fd7680ce950 2023-10-16T10:00:15.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd7677fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7680d1210 0x7fd7680ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd767fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd76815d7c0 0x7fd7680d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54500/0 (socket says 172.21.15.158:54500) 2023-10-16T10:00:15.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd767fff700 1 -- 172.21.15.158:0/2440154560 learned_addr learned my addr 172.21.15.158:0/2440154560 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:15.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd76c95a700 1 --2- 172.21.15.158:0/2440154560 >> v2:172.21.15.172:3300/0 conn(0x7fd7680ce950 0x7fd7680b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd7677fe700 1 -- 172.21.15.158:0/2440154560 >> v2:172.21.15.172:3300/0 conn(0x7fd7680ce950 msgr2=0x7fd7680b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:15.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd7677fe700 1 --2- 172.21.15.158:0/2440154560 >> v2:172.21.15.172:3300/0 conn(0x7fd7680ce950 0x7fd7680b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd7677fe700 1 -- 172.21.15.158:0/2440154560 >> v1:172.21.15.158:6789/0 conn(0x7fd76815d7c0 legacy=0x7fd7680d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd7677fe700 1 -- 172.21.15.158:0/2440154560 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680d32a0 con 0x7fd7680d1210 2023-10-16T10:00:15.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd766ffd700 1 -- 172.21.15.158:0/2440154560 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439206560 0 0) 0x7fd7680d3bd0 con 0x7fd76815d7c0 2023-10-16T10:00:15.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.174+0000 7fd7677fe700 1 --2- 172.21.15.158:0/2440154560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680d1210 0x7fd7680ce410 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd7680c7120 tx=0x7fd75c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc7126085f36199 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:15.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd766ffd700 1 -- 172.21.15.158:0/2440154560 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c011830 con 0x7fd7680d1210 2023-10-16T10:00:15.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd766ffd700 1 -- 172.21.15.158:0/2440154560 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c011990 con 0x7fd7680d1210 2023-10-16T10:00:15.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd766ffd700 1 -- 172.21.15.158:0/2440154560 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c011d20 con 0x7fd7680d1210 2023-10-16T10:00:15.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 -- 172.21.15.158:0/2440154560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680d1210 msgr2=0x7fd7680ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/2440154560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680d1210 0x7fd7680ce410 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd7680c7120 tx=0x7fd75c009b20 comp rx=0 tx=0).stop 2023-10-16T10:00:15.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 -- 172.21.15.158:0/2440154560 shutdown_connections 2023-10-16T10:00:15.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/2440154560 >> v2:172.21.15.172:3300/0 conn(0x7fd7680ce950 0x7fd7680b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/2440154560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680d1210 0x7fd7680ce410 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 -- 172.21.15.158:0/2440154560 >> 172.21.15.158:0/2440154560 conn(0x7fd7680669f0 msgr2=0x7fd768066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:15.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 -- 172.21.15.158:0/2440154560 shutdown_connections 2023-10-16T10:00:15.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.175+0000 7fd76e3bd700 1 -- 172.21.15.158:0/2440154560 wait complete. 2023-10-16T10:00:15.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.176+0000 7fd76e3bd700 1 Processor -- start 2023-10-16T10:00:15.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76e3bd700 1 -- start start 2023-10-16T10:00:15.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76e3bd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680ce950 0x7fd7681590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76e3bd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7680d1210 0x7fd7680c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76e3bd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd76815d7c0 0x7fd7681f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76e3bd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680cc630 con 0x7fd7680ce950 2023-10-16T10:00:15.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76e3bd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680cc910 con 0x7fd76815d7c0 2023-10-16T10:00:15.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76e3bd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680ccbf0 con 0x7fd7680d1210 2023-10-16T10:00:15.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd767fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680ce950 0x7fd7681590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd767fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680ce950 0x7fd7681590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54330/0 (socket says 172.21.15.158:54330) 2023-10-16T10:00:15.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd767fff700 1 -- 172.21.15.158:0/3107872005 learned_addr learned my addr 172.21.15.158:0/3107872005 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:15.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd76c95a700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd76815d7c0 0x7fd7681f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.177+0000 7fd7677fe700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7680d1210 0x7fd7680c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76c95a700 1 -- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7680d1210 msgr2=0x7fd7680c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76c95a700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7680d1210 0x7fd7680c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76c95a700 1 -- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680ce950 msgr2=0x7fd7681590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76c95a700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680ce950 0x7fd7681590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76c95a700 1 -- 172.21.15.158:0/3107872005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c0094a0 con 0x7fd76815d7c0 2023-10-16T10:00:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd767fff700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680ce950 0x7fd7681590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd7677fe700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7680d1210 0x7fd7680c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76c95a700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd76815d7c0 0x7fd7681f6240 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd760007d90 tx=0x7fd76000e3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd764ff9700 1 -- 172.21.15.158:0/3107872005 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760018070 con 0x7fd76815d7c0 2023-10-16T10:00:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd764ff9700 1 -- 172.21.15.158:0/3107872005 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76000eb30 con 0x7fd76815d7c0 2023-10-16T10:00:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7680cced0 con 0x7fd76815d7c0 2023-10-16T10:00:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd764ff9700 1 -- 172.21.15.158:0/3107872005 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760003a70 con 0x7fd76815d7c0 2023-10-16T10:00:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.178+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7681595e0 con 0x7fd76815d7c0 2023-10-16T10:00:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.179+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7681542f0 con 0x7fd76815d7c0 2023-10-16T10:00:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.180+0000 7fd764ff9700 1 -- 172.21.15.158:0/3107872005 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd760014420 con 0x7fd76815d7c0 2023-10-16T10:00:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.180+0000 7fd764ff9700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 0x7fd75406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.180+0000 7fd767fff700 1 -- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 msgr2=0x7fd75406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.180+0000 7fd764ff9700 1 -- 172.21.15.158:0/3107872005 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fd760089a20 con 0x7fd76815d7c0 2023-10-16T10:00:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.180+0000 7fd767fff700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 0x7fd75406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.185+0000 7fd764ff9700 1 -- 172.21.15.158:0/3107872005 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd760054610 con 0x7fd76815d7c0 2023-10-16T10:00:15.208 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.208+0000 7fc5dee94200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:00:15.223 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:15.221+0000 7f2e09c28200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:00:15.350 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.349+0000 7fc5dee94200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:00:15.369 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:15.367+0000 7f2e09c28200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:00:15.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.381+0000 7fd767fff700 1 -- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 msgr2=0x7fd75406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:15.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.381+0000 7fd767fff700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 0x7fd75406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:15.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.386+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7680cca50 con 0x7fd76815d7c0 2023-10-16T10:00:15.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.387+0000 7fd764ff9700 1 -- 172.21.15.158:0/3107872005 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17941 (secure 0 0 0) 0x7fd7680cca50 con 0x7fd76815d7c0 2023-10-16T10:00:15.390 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:15.390 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":99,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:12.821319+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:57.438311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:57.966439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1178199719":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/873662109":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6833/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6832/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/1880882485":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/693060665":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/2133642547":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/4289198127":"2023-10-17T10:00:12.821288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:15.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 msgr2=0x7fd75406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:15.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 0x7fd75406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd76815d7c0 msgr2=0x7fd7681f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd76815d7c0 0x7fd7681f6240 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd760007d90 tx=0x7fd76000e3c0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 shutdown_connections 2023-10-16T10:00:15.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fd75406b020 0x7fd75406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd76815d7c0 0x7fd7681f6240 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7680ce950 0x7fd7681590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 --2- 172.21.15.158:0/3107872005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7680d1210 0x7fd7680c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.389+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 >> 172.21.15.158:0/3107872005 conn(0x7fd7680669f0 msgr2=0x7fd76815b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:15.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.390+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 shutdown_connections 2023-10-16T10:00:15.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.390+0000 7fd76e3bd700 1 -- 172.21.15.158:0/3107872005 wait complete. 2023-10-16T10:00:15.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:59:57.438311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:59:57.966439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:00:15.406 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:00:15.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.568+0000 7fb598f25700 1 Processor -- start 2023-10-16T10:00:15.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.568+0000 7fb598f25700 1 -- start start 2023-10-16T10:00:15.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.569+0000 7fb598f25700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb59415da40 0x7fb5940caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.569+0000 7fb598f25700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940cbd30 con 0x7fb59406afe0 2023-10-16T10:00:15.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.569+0000 7fb598f25700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940cc1c0 con 0x7fb5940cc590 2023-10-16T10:00:15.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.569+0000 7fb598f25700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59415de20 con 0x7fb59415da40 2023-10-16T10:00:15.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb592d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb59406afe0 0x7fb5940afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54564/0 (socket says 172.21.15.158:54564) 2023-10-16T10:00:15.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb592d9d700 1 -- 172.21.15.158:0/2819888903 learned_addr learned my addr 172.21.15.158:0/2819888903 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:15.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59259c700 1 --2- 172.21.15.158:0/2819888903 >> v2:172.21.15.172:3300/0 conn(0x7fb59415da40 0x7fb5940caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59159a700 1 -- 172.21.15.158:0/2819888903 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299163060 0 0) 0x7fb5940cbd30 con 0x7fb59406afe0 2023-10-16T10:00:15.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59159a700 1 -- 172.21.15.158:0/2819888903 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb57c0035b0 con 0x7fb59406afe0 2023-10-16T10:00:15.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59259c700 1 -- 172.21.15.158:0/2819888903 >> v1:172.21.15.172:6789/0 conn(0x7fb5940cc590 legacy=0x7fb59406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59259c700 1 -- 172.21.15.158:0/2819888903 >> v1:172.21.15.158:6789/0 conn(0x7fb59406afe0 legacy=0x7fb5940afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59259c700 1 -- 172.21.15.158:0/2819888903 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940ce260 con 0x7fb59415da40 2023-10-16T10:00:15.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59259c700 1 --2- 172.21.15.158:0/2819888903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59415da40 0x7fb5940caa20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb588009840 tx=0x7fb588009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdff1262f227d04a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:15.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59159a700 1 -- 172.21.15.158:0/2819888903 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588011830 con 0x7fb59415da40 2023-10-16T10:00:15.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59159a700 1 -- 172.21.15.158:0/2819888903 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb588011990 con 0x7fb59415da40 2023-10-16T10:00:15.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.570+0000 7fb59159a700 1 -- 172.21.15.158:0/2819888903 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588011d20 con 0x7fb59415da40 2023-10-16T10:00:15.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.571+0000 7fb598f25700 1 -- 172.21.15.158:0/2819888903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59415da40 msgr2=0x7fb5940caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.571+0000 7fb598f25700 1 --2- 172.21.15.158:0/2819888903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59415da40 0x7fb5940caa20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb588009840 tx=0x7fb588009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:15.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.571+0000 7fb598f25700 1 -- 172.21.15.158:0/2819888903 shutdown_connections 2023-10-16T10:00:15.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.571+0000 7fb598f25700 1 --2- 172.21.15.158:0/2819888903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59415da40 0x7fb5940caa20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.571+0000 7fb598f25700 1 -- 172.21.15.158:0/2819888903 >> 172.21.15.158:0/2819888903 conn(0x7fb594099a10 msgr2=0x7fb5940d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:15.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.571+0000 7fb598f25700 1 -- 172.21.15.158:0/2819888903 shutdown_connections 2023-10-16T10:00:15.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.571+0000 7fb598f25700 1 -- 172.21.15.158:0/2819888903 wait complete. 2023-10-16T10:00:15.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.573+0000 7fb598f25700 1 Processor -- start 2023-10-16T10:00:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.573+0000 7fb598f25700 1 -- start start 2023-10-16T10:00:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb598f25700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59406afe0 0x7fb59414a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb598f25700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5940cc590 0x7fb5940ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb598f25700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb59415da40 0x7fb5940b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb598f25700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940c9540 con 0x7fb59415da40 2023-10-16T10:00:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb598f25700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940c97c0 con 0x7fb59406afe0 2023-10-16T10:00:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb598f25700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59414a970 con 0x7fb5940cc590 2023-10-16T10:00:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb59415da40 0x7fb5940b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb59415da40 0x7fb5940b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54394/0 (socket says 172.21.15.158:54394) 2023-10-16T10:00:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb59259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59406afe0 0x7fb59414a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 -- 172.21.15.158:0/3948573534 learned_addr learned my addr 172.21.15.158:0/3948573534 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb591d9b700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5940cc590 0x7fb5940ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 -- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5940cc590 msgr2=0x7fb5940ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5940cc590 0x7fb5940ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 -- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59406afe0 msgr2=0x7fb59414a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59406afe0 0x7fb59414a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 -- 172.21.15.158:0/3948573534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880094a0 con 0x7fb59415da40 2023-10-16T10:00:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb59259c700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59406afe0 0x7fb59414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.574+0000 7fb592d9d700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb59415da40 0x7fb5940b4980 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00cbb0 tx=0x7fb58c00cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:15.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.575+0000 7fb582ffd700 1 -- 172.21.15.158:0/3948573534 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00a9e0 con 0x7fb59415da40 2023-10-16T10:00:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.575+0000 7fb582ffd700 1 -- 172.21.15.158:0/3948573534 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c00ab40 con 0x7fb59415da40 2023-10-16T10:00:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.575+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59414da60 con 0x7fb59415da40 2023-10-16T10:00:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.575+0000 7fb582ffd700 1 -- 172.21.15.158:0/3948573534 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c012400 con 0x7fb59415da40 2023-10-16T10:00:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.575+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5940c9d60 con 0x7fb59415da40 2023-10-16T10:00:15.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.576+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5940ca0d0 con 0x7fb59415da40 2023-10-16T10:00:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.576+0000 7fb582ffd700 1 -- 172.21.15.158:0/3948573534 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fb58c01b3b0 con 0x7fb59415da40 2023-10-16T10:00:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.576+0000 7fb582ffd700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 0x7fb57406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.577+0000 7fb59259c700 1 -- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 msgr2=0x7fb57406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.577+0000 7fb582ffd700 1 -- 172.21.15.158:0/3948573534 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fb58c089830 con 0x7fb59415da40 2023-10-16T10:00:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.577+0000 7fb59259c700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 0x7fb57406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:15.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.581+0000 7fb582ffd700 1 -- 172.21.15.158:0/3948573534 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb58c0543c0 con 0x7fb59415da40 2023-10-16T10:00:15.678 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.677+0000 7fc5dee94200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:00:15.723 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:15.721+0000 7f2e09c28200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:00:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.778+0000 7fb59259c700 1 -- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 msgr2=0x7fb57406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.778+0000 7fb59259c700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 0x7fb57406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:15.795 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.795+0000 7fc5dee94200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:00:15.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.795+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5940681b0 con 0x7fb59415da40 2023-10-16T10:00:15.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.795+0000 7fb582ffd700 1 -- 172.21.15.158:0/3948573534 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb58c05ce60 con 0x7fb59415da40 2023-10-16T10:00:15.798 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:00:15.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.797+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 msgr2=0x7fb57406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:15.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.797+0000 7fb598f25700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 0x7fb57406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.797+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb59415da40 msgr2=0x7fb5940b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.797+0000 7fb598f25700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb59415da40 0x7fb5940b4980 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00cbb0 tx=0x7fb58c00cec0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 shutdown_connections 2023-10-16T10:00:15.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fb57406b020 0x7fb57406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb59415da40 0x7fb5940b4980 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb59406afe0 0x7fb59414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 --2- 172.21.15.158:0/3948573534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5940cc590 0x7fb5940ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 >> 172.21.15.158:0/3948573534 conn(0x7fb594099a10 msgr2=0x7fb5940b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:15.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 shutdown_connections 2023-10-16T10:00:15.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.798+0000 7fb598f25700 1 -- 172.21.15.158:0/3948573534 wait complete. 2023-10-16T10:00:15.814 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:00:15.847 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:15.845+0000 7f2e09c28200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:00:15.869 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.869+0000 7fc5dee94200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:00:15.924 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:15.922+0000 7f2e09c28200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:00:15.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.939+0000 7fe42dcd9700 1 Processor -- start 2023-10-16T10:00:15.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.939+0000 7fe42dcd9700 1 -- start start 2023-10-16T10:00:15.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe42dcd9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42815b6b0 0x7fe4280cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe42dcd9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4280d0c60 0x7fe4280b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe42dcd9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4280b4420 con 0x7fe4280b3460 2023-10-16T10:00:15.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe42dcd9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4280b3910 con 0x7fe4280d0c60 2023-10-16T10:00:15.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe42dcd9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42815ba90 con 0x7fe42815b6b0 2023-10-16T10:00:15.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4280d0c60 0x7fe4280b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4280d0c60 0x7fe4280b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54402/0 (socket says 172.21.15.158:54402) 2023-10-16T10:00:15.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 -- 172.21.15.158:0/2852145947 learned_addr learned my addr 172.21.15.158:0/2852145947 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:15.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe4277fe700 1 --2- 172.21.15.158:0/2852145947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42815b6b0 0x7fe4280cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 -- 172.21.15.158:0/2852145947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42815b6b0 msgr2=0x7fe4280cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:15.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 --2- 172.21.15.158:0/2852145947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42815b6b0 0x7fe4280cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 -- 172.21.15.158:0/2852145947 >> v1:172.21.15.172:6789/0 conn(0x7fe4280b3460 legacy=0x7fe428070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 -- 172.21.15.158:0/2852145947 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4280cbc20 con 0x7fe4280d0c60 2023-10-16T10:00:15.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe426ffd700 1 --2- 172.21.15.158:0/2852145947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280b2f20 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0098f0 tx=0x7fe41c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b1203c1cb2b15b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:15.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.940+0000 7fe4267fc700 1 -- 172.21.15.158:0/2852145947 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c011830 con 0x7fe4280d0c60 2023-10-16T10:00:15.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe4267fc700 1 -- 172.21.15.158:0/2852145947 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c011990 con 0x7fe4280d0c60 2023-10-16T10:00:15.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe4267fc700 1 -- 172.21.15.158:0/2852145947 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c011d20 con 0x7fe4280d0c60 2023-10-16T10:00:15.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 -- 172.21.15.158:0/2852145947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 msgr2=0x7fe4280b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/2852145947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280b2f20 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0098f0 tx=0x7fe41c009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 -- 172.21.15.158:0/2852145947 shutdown_connections 2023-10-16T10:00:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/2852145947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42815b6b0 0x7fe4280cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/2852145947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280b2f20 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 -- 172.21.15.158:0/2852145947 >> 172.21.15.158:0/2852145947 conn(0x7fe4280cdf20 msgr2=0x7fe4280ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:15.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 -- 172.21.15.158:0/2852145947 shutdown_connections 2023-10-16T10:00:15.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.941+0000 7fe42dcd9700 1 -- 172.21.15.158:0/2852145947 wait complete. 2023-10-16T10:00:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.942+0000 7fe42dcd9700 1 Processor -- start 2023-10-16T10:00:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe42dcd9700 1 -- start start 2023-10-16T10:00:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe42dcd9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4280b3460 0x7fe4281529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe42dcd9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe42dcd9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42815b6b0 0x7fe4280cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe42dcd9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281f1800 con 0x7fe4280d0c60 2023-10-16T10:00:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe42dcd9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281f0c10 con 0x7fe42815b6b0 2023-10-16T10:00:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe42dcd9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281f0ed0 con 0x7fe4280b3460 2023-10-16T10:00:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe426ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe426ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54408/0 (socket says 172.21.15.158:54408) 2023-10-16T10:00:15.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe426ffd700 1 -- 172.21.15.158:0/4120021727 learned_addr learned my addr 172.21.15.158:0/4120021727 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:15.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe427fff700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42815b6b0 0x7fe4280cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.943+0000 7fe4277fe700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4280b3460 0x7fe4281529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:15.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe426ffd700 1 -- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4280b3460 msgr2=0x7fe4281529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe426ffd700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4280b3460 0x7fe4281529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe426ffd700 1 -- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42815b6b0 msgr2=0x7fe4280cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:15.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe426ffd700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42815b6b0 0x7fe4280cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:15.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe426ffd700 1 -- 172.21.15.158:0/4120021727 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c009550 con 0x7fe4280d0c60 2023-10-16T10:00:15.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe426ffd700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280ca8c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fe41c000c00 tx=0x7fe41c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:15.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe40ffff700 1 -- 172.21.15.158:0/4120021727 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c004400 con 0x7fe4280d0c60 2023-10-16T10:00:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe40ffff700 1 -- 172.21.15.158:0/4120021727 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c01f070 con 0x7fe4280d0c60 2023-10-16T10:00:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4281f11b0 con 0x7fe4280d0c60 2023-10-16T10:00:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe40ffff700 1 -- 172.21.15.158:0/4120021727 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0038e0 con 0x7fe4280d0c60 2023-10-16T10:00:15.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.944+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe428152f20 con 0x7fe4280d0c60 2023-10-16T10:00:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.945+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4281f1970 con 0x7fe4280d0c60 2023-10-16T10:00:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.945+0000 7fe40ffff700 1 -- 172.21.15.158:0/4120021727 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe41c021030 con 0x7fe4280d0c60 2023-10-16T10:00:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.946+0000 7fe40ffff700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 0x7fe41406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.946+0000 7fe4277fe700 1 -- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 msgr2=0x7fe41406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:15.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.946+0000 7fe40ffff700 1 -- 172.21.15.158:0/4120021727 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fe41c08ae80 con 0x7fe4280d0c60 2023-10-16T10:00:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.946+0000 7fe4277fe700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 0x7fe41406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:15.958 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:15.950+0000 7fc5dee94200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:00:15.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:15.954+0000 7fe40ffff700 1 -- 172.21.15.158:0/4120021727 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe41c055bc0 con 0x7fe4280d0c60 2023-10-16T10:00:16.013 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:16.011+0000 7f2e09c28200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:00:16.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.147+0000 7fe4277fe700 1 -- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 msgr2=0x7fe41406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:16.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.147+0000 7fe4277fe700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 0x7fe41406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:16.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.154+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4280681b0 con 0x7fe4280d0c60 2023-10-16T10:00:16.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.155+0000 7fe40ffff700 1 -- 172.21.15.158:0/4120021727 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe41c0150a0 con 0x7fe4280d0c60 2023-10-16T10:00:16.157 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:00:16.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 msgr2=0x7fe41406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:16.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 0x7fe41406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 msgr2=0x7fe4280ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280ca8c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fe41c000c00 tx=0x7fe41c004750 comp rx=0 tx=0).stop 2023-10-16T10:00:16.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 shutdown_connections 2023-10-16T10:00:16.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe41406b020 0x7fe41406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42815b6b0 0x7fe4280cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4280b3460 0x7fe4281529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 --2- 172.21.15.158:0/4120021727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4280d0c60 0x7fe4280ca8c0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 >> 172.21.15.158:0/4120021727 conn(0x7fe4280cdf20 msgr2=0x7fe4280d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:16.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 shutdown_connections 2023-10-16T10:00:16.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.157+0000 7fe42dcd9700 1 -- 172.21.15.158:0/4120021727 wait complete. 2023-10-16T10:00:16.172 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:00:16.320 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:16.319+0000 7fc5dee94200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:00:16.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.336+0000 7f9c6506a700 1 Processor -- start 2023-10-16T10:00:16.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.336+0000 7f9c6506a700 1 -- start start 2023-10-16T10:00:16.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c6506a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c6015d6c0 0x7f9c600cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c6506a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c600cf360 0x7f9c600d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c6506a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c600d1ee0 con 0x7f9c6015daa0 2023-10-16T10:00:16.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c6506a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c600d40a0 con 0x7f9c6015d6c0 2023-10-16T10:00:16.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c6506a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600d12d0 con 0x7f9c600cf360 2023-10-16T10:00:16.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c5ed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c6015d6c0 0x7f9c600cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c5f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c600cf360 0x7f9c600d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c5e59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9c6015daa0 0x7f9c600cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:58522/0 (socket says 172.21.15.158:58522) 2023-10-16T10:00:16.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c5e59c700 1 -- 172.21.15.158:0/299362049 learned_addr learned my addr 172.21.15.158:0/299362049 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:16.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c5ed9d700 1 --2- 172.21.15.158:0/299362049 >> v2:172.21.15.172:3300/0 conn(0x7f9c6015d6c0 0x7f9c600cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:55622/0 (socket says 172.21.15.158:55622) 2023-10-16T10:00:16.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.337+0000 7f9c5f59e700 1 --2- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c600cf360 0x7f9c600d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:37048/0 (socket says 172.21.15.158:37048) 2023-10-16T10:00:16.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5ed9d700 1 -- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c600cf360 msgr2=0x7f9c600d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:16.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5ed9d700 1 --2- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c600cf360 0x7f9c600d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5ed9d700 1 -- 172.21.15.158:0/299362049 >> v1:172.21.15.172:6789/0 conn(0x7f9c6015daa0 legacy=0x7f9c600cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5ed9d700 1 -- 172.21.15.158:0/299362049 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c600d15b0 con 0x7f9c6015d6c0 2023-10-16T10:00:16.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5dd9b700 1 -- 172.21.15.158:0/299362049 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157466630 0 0) 0x7f9c600d1ee0 con 0x7f9c6015daa0 2023-10-16T10:00:16.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5f59e700 1 --2- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c600cf360 0x7f9c600d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:16.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5ed9d700 1 --2- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c600cff30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009840 tx=0x7f9c50009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78b89482595b794c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:16.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5dd9b700 1 -- 172.21.15.158:0/299362049 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c50011830 con 0x7f9c6015d6c0 2023-10-16T10:00:16.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5dd9b700 1 -- 172.21.15.158:0/299362049 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c50011990 con 0x7f9c6015d6c0 2023-10-16T10:00:16.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c5dd9b700 1 -- 172.21.15.158:0/299362049 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c50011d20 con 0x7f9c6015d6c0 2023-10-16T10:00:16.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.338+0000 7f9c6506a700 1 -- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 msgr2=0x7f9c600cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.339+0000 7f9c6506a700 1 --2- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c600cff30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009840 tx=0x7f9c50009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:16.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.339+0000 7f9c6506a700 1 -- 172.21.15.158:0/299362049 shutdown_connections 2023-10-16T10:00:16.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.339+0000 7f9c6506a700 1 --2- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c600cf360 0x7f9c600d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.339+0000 7f9c6506a700 1 --2- 172.21.15.158:0/299362049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c600cff30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.339+0000 7f9c6506a700 1 -- 172.21.15.158:0/299362049 >> 172.21.15.158:0/299362049 conn(0x7f9c6009c940 msgr2=0x7f9c600d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:16.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.339+0000 7f9c6506a700 1 -- 172.21.15.158:0/299362049 shutdown_connections 2023-10-16T10:00:16.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.339+0000 7f9c6506a700 1 -- 172.21.15.158:0/299362049 wait complete. 2023-10-16T10:00:16.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.340+0000 7f9c6506a700 1 Processor -- start 2023-10-16T10:00:16.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.340+0000 7f9c6506a700 1 -- start start 2023-10-16T10:00:16.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c6506a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c600cf360 0x7f9c6015a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c6506a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c60160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c6506a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c6015daa0 0x7f9c600ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c6506a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6015abd0 con 0x7f9c600cf360 2023-10-16T10:00:16.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c6506a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600cb670 con 0x7f9c6015d6c0 2023-10-16T10:00:16.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c6506a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600cb950 con 0x7f9c6015daa0 2023-10-16T10:00:16.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c600cf360 0x7f9c6015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c6015daa0 0x7f9c600ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c60160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c600cf360 0x7f9c6015a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54448/0 (socket says 172.21.15.158:54448) 2023-10-16T10:00:16.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5ed9d700 1 -- 172.21.15.158:0/2000503647 learned_addr learned my addr 172.21.15.158:0/2000503647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:16.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c6015daa0 0x7f9c600ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:37054/0 (socket says 172.21.15.158:37054) 2023-10-16T10:00:16.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5e59c700 1 -- 172.21.15.158:0/2000503647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c6015daa0 msgr2=0x7f9c600ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5e59c700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c6015daa0 0x7f9c600ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5e59c700 1 -- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c600cf360 msgr2=0x7f9c6015a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5e59c700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c600cf360 0x7f9c6015a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.341+0000 7f9c5e59c700 1 -- 172.21.15.158:0/2000503647 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58009a70 con 0x7f9c6015d6c0 2023-10-16T10:00:16.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.342+0000 7f9c5e59c700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c60160250 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400c3e0 tx=0x7f9c5400c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:16.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.342+0000 7f9c477fe700 1 -- 172.21.15.158:0/2000503647 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54004a70 con 0x7f9c6015d6c0 2023-10-16T10:00:16.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.342+0000 7f9c477fe700 1 -- 172.21.15.158:0/2000503647 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c54004bd0 con 0x7f9c6015d6c0 2023-10-16T10:00:16.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.342+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c500094a0 con 0x7f9c6015d6c0 2023-10-16T10:00:16.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.342+0000 7f9c477fe700 1 -- 172.21.15.158:0/2000503647 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c540070d0 con 0x7f9c6015d6c0 2023-10-16T10:00:16.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.342+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c600cbf10 con 0x7f9c6015d6c0 2023-10-16T10:00:16.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.343+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c600d2350 con 0x7f9c6015d6c0 2023-10-16T10:00:16.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.343+0000 7f9c477fe700 1 -- 172.21.15.158:0/2000503647 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f9c54007440 con 0x7f9c6015d6c0 2023-10-16T10:00:16.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.343+0000 7f9c477fe700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 0x7f9c4c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.344+0000 7f9c5ed9d700 1 -- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 msgr2=0x7f9c4c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:16.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.344+0000 7f9c477fe700 1 -- 172.21.15.158:0/2000503647 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f9c540898a0 con 0x7f9c6015d6c0 2023-10-16T10:00:16.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.344+0000 7f9c5ed9d700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 0x7f9c4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:16.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.348+0000 7f9c477fe700 1 -- 172.21.15.158:0/2000503647 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c600d2350 con 0x7f9c6015d6c0 2023-10-16T10:00:16.403 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:16.401+0000 7f2e09c28200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:00:16.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.545+0000 7f9c5ed9d700 1 -- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 msgr2=0x7f9c4c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:16.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.545+0000 7f9c5ed9d700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 0x7f9c4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:16.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.547+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c600d2350 con 0x7f9c6015d6c0 2023-10-16T10:00:16.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.548+0000 7f9c477fe700 1 -- 172.21.15.158:0/2000503647 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9c600d2350 con 0x7f9c6015d6c0 2023-10-16T10:00:16.550 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:00:16.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 msgr2=0x7f9c4c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:16.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 0x7f9c4c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 msgr2=0x7f9c60160250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c60160250 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400c3e0 tx=0x7f9c5400c6f0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 shutdown_connections 2023-10-16T10:00:16.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9c4c06b020 0x7f9c4c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c6015daa0 0x7f9c600ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c600cf360 0x7f9c6015a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 --2- 172.21.15.158:0/2000503647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c6015d6c0 0x7f9c60160250 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 >> 172.21.15.158:0/2000503647 conn(0x7f9c6009c940 msgr2=0x7f9c600b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:16.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 shutdown_connections 2023-10-16T10:00:16.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.550+0000 7f9c6506a700 1 -- 172.21.15.158:0/2000503647 wait complete. 2023-10-16T10:00:16.565 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T10:00:16.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.687+0000 7f59e2a22700 1 Processor -- start 2023-10-16T10:00:16.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.687+0000 7f59e2a22700 1 -- start start 2023-10-16T10:00:16.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59e2a22700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59dc0d1900 0x7f59dc0d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59e2a22700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc0d22a0 0x7f59dc0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59e2a22700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59dc0d2730 con 0x7f59dc15d920 2023-10-16T10:00:16.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59e2a22700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59dc0d0040 con 0x7f59dc0d1900 2023-10-16T10:00:16.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59e2a22700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc0cf5a0 con 0x7f59dc0d22a0 2023-10-16T10:00:16.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59dc0d1900 0x7f59dc0d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59dc0d1900 0x7f59dc0d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54460/0 (socket says 172.21.15.158:54460) 2023-10-16T10:00:16.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 -- 172.21.15.158:0/3244126010 learned_addr learned my addr 172.21.15.158:0/3244126010 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:16.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59e0fbf700 1 --2- 172.21.15.158:0/3244126010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc0d22a0 0x7f59dc0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 -- 172.21.15.158:0/3244126010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc0d22a0 msgr2=0x7f59dc0c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:16.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 --2- 172.21.15.158:0/3244126010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc0d22a0 0x7f59dc0c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 -- 172.21.15.158:0/3244126010 >> v1:172.21.15.172:6789/0 conn(0x7f59dc15d920 legacy=0x7f59dc0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 -- 172.21.15.158:0/3244126010 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc0c73a0 con 0x7f59dc0d1900 2023-10-16T10:00:16.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.688+0000 7f59db7fe700 1 --2- 172.21.15.158:0/3244126010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc0d1d60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009920 tx=0x7f59d0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7175f8ac2a93cb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:16.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59daffd700 1 -- 172.21.15.158:0/3244126010 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0011830 con 0x7f59dc0d1900 2023-10-16T10:00:16.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59daffd700 1 -- 172.21.15.158:0/3244126010 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d0011990 con 0x7f59dc0d1900 2023-10-16T10:00:16.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59daffd700 1 -- 172.21.15.158:0/3244126010 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0011d20 con 0x7f59dc0d1900 2023-10-16T10:00:16.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 -- 172.21.15.158:0/3244126010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 msgr2=0x7f59dc0d1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3244126010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc0d1d60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009920 tx=0x7f59d0009c30 comp rx=0 tx=0).stop 2023-10-16T10:00:16.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 -- 172.21.15.158:0/3244126010 shutdown_connections 2023-10-16T10:00:16.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3244126010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc0d22a0 0x7f59dc0c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3244126010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc0d1d60 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 -- 172.21.15.158:0/3244126010 >> 172.21.15.158:0/3244126010 conn(0x7f59dc09ca30 msgr2=0x7f59dc0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:16.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 -- 172.21.15.158:0/3244126010 shutdown_connections 2023-10-16T10:00:16.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.689+0000 7f59e2a22700 1 -- 172.21.15.158:0/3244126010 wait complete. 2023-10-16T10:00:16.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.690+0000 7f59e2a22700 1 Processor -- start 2023-10-16T10:00:16.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.691+0000 7f59e2a22700 1 -- start start 2023-10-16T10:00:16.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.691+0000 7f59e2a22700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.691+0000 7f59e2a22700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59dc0d22a0 0x7f59dc0b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.691+0000 7f59e2a22700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc15d920 0x7f59dc06bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.691+0000 7f59e2a22700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc0cc9e0 con 0x7f59dc0d1900 2023-10-16T10:00:16.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.691+0000 7f59e2a22700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc0cce00 con 0x7f59dc0d22a0 2023-10-16T10:00:16.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.691+0000 7f59e2a22700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc0cbd60 con 0x7f59dc15d920 2023-10-16T10:00:16.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54464/0 (socket says 172.21.15.158:54464) 2023-10-16T10:00:16.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 -- 172.21.15.158:0/3082267384 learned_addr learned my addr 172.21.15.158:0/3082267384 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:16.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59db7fe700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59dc0d22a0 0x7f59dc0b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59e0fbf700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc15d920 0x7f59dc06bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:16.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 -- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc15d920 msgr2=0x7f59dc06bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc15d920 0x7f59dc06bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 -- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59dc0d22a0 msgr2=0x7f59dc0b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59dc0d22a0 0x7f59dc0b7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 -- 172.21.15.158:0/3082267384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0009550 con 0x7f59dc0d1900 2023-10-16T10:00:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59dbfff700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc157e40 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00c430 tx=0x7f59cc00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59d8ff9700 1 -- 172.21.15.158:0/3082267384 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc0052d0 con 0x7f59dc0d1900 2023-10-16T10:00:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59d8ff9700 1 -- 172.21.15.158:0/3082267384 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc005450 con 0x7f59dc0d1900 2023-10-16T10:00:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59dc0cc040 con 0x7f59dc0d1900 2023-10-16T10:00:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59d8ff9700 1 -- 172.21.15.158:0/3082267384 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc006af0 con 0x7f59dc0d1900 2023-10-16T10:00:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.692+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59dc0cc600 con 0x7f59dc0d1900 2023-10-16T10:00:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.693+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59dc0681b0 con 0x7f59dc0d1900 2023-10-16T10:00:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.694+0000 7f59d8ff9700 1 -- 172.21.15.158:0/3082267384 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f59cc0036d0 con 0x7f59dc0d1900 2023-10-16T10:00:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.694+0000 7f59d8ff9700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 0x7f59c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.694+0000 7f59db7fe700 1 -- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 msgr2=0x7f59c806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.695+0000 7f59d8ff9700 1 -- 172.21.15.158:0/3082267384 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f59cc088c10 con 0x7f59dc0d1900 2023-10-16T10:00:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.695+0000 7f59db7fe700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 0x7f59c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:16.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.699+0000 7f59d8ff9700 1 -- 172.21.15.158:0/3082267384 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59cc053800 con 0x7f59dc0d1900 2023-10-16T10:00:16.890 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:16.888+0000 7fc5dee94200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:00:16.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.896+0000 7f59db7fe700 1 -- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 msgr2=0x7f59c806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:16.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.896+0000 7f59db7fe700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 0x7f59c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:16.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.949+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f59dc0d3e50 con 0x7f59dc0d1900 2023-10-16T10:00:16.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.950+0000 7f59d8ff9700 1 -- 172.21.15.158:0/3082267384 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v99) v1 ==== 80+0+476 (secure 0 0 0) 0x7f59cc056e20 con 0x7f59dc0d1900 2023-10-16T10:00:16.952 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1178199719 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.952 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/873662109 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.953 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/3798429240 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.953 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/3798429240 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.953 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1880882485 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.953 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/693060665 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.954 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2133642547 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.954 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/4289198127 2023-10-17T10:00:12.821288+0000 2023-10-16T10:00:16.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 msgr2=0x7f59c806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:16.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 0x7f59c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 msgr2=0x7f59dc157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:16.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc157e40 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00c430 tx=0x7f59cc00c740 comp rx=0 tx=0).stop 2023-10-16T10:00:16.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 shutdown_connections 2023-10-16T10:00:16.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f59c806b020 0x7f59c806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59dc15d920 0x7f59dc06bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59dc0d1900 0x7f59dc157e40 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 --2- 172.21.15.158:0/3082267384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59dc0d22a0 0x7f59dc0b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:16.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 >> 172.21.15.158:0/3082267384 conn(0x7f59dc09ca30 msgr2=0x7f59dc0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:16.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 shutdown_connections 2023-10-16T10:00:16.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:16.952+0000 7f59e2a22700 1 -- 172.21.15.158:0/3082267384 wait complete. 2023-10-16T10:00:16.958 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T10:00:16.959 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:16.958+0000 7fc5dee94200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:00:16.968 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T10:00:16.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:16.969 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:17.004 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:17.002+0000 7f2e09c28200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:00:17.007 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T10:00:17.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:17.008 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T10:00:17.065 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:00:17.065 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:17.079 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:17.077+0000 7f2e09c28200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:00:17.083 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:00:17.084 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:00:17.084 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:17.102 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:00:17.102 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:00:17.102 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:17.140 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:00:17.140 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:17.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.262+0000 7f29c0957700 1 Processor -- start 2023-10-16T10:00:17.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.262+0000 7f29c0957700 1 -- start start 2023-10-16T10:00:17.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29c0957700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f29bc0d0bd0 0x7f29bc0d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29c0957700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f29bc0d3570 0x7f29bc0d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29c0957700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc0ce4d0 con 0x7f29bc15daa0 2023-10-16T10:00:17.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29c0957700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29bc0d3ec0 con 0x7f29bc0d3570 2023-10-16T10:00:17.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29c0957700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29bc06c1b0 con 0x7f29bc0d0bd0 2023-10-16T10:00:17.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29ba59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f29bc15daa0 0x7f29bc06b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54650/0 (socket says 172.21.15.158:54650) 2023-10-16T10:00:17.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29bad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f29bc0d3570 0x7f29bc0d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29ba59c700 1 -- 172.21.15.158:0/574098592 learned_addr learned my addr 172.21.15.158:0/574098592 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:17.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29b9d9b700 1 --2- 172.21.15.158:0/574098592 >> v2:172.21.15.172:3300/0 conn(0x7f29bc0d0bd0 0x7f29bc0d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29bad9d700 1 -- 172.21.15.158:0/574098592 >> v2:172.21.15.172:3300/0 conn(0x7f29bc0d0bd0 msgr2=0x7f29bc0d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:17.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29bad9d700 1 --2- 172.21.15.158:0/574098592 >> v2:172.21.15.172:3300/0 conn(0x7f29bc0d0bd0 0x7f29bc0d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29bad9d700 1 -- 172.21.15.158:0/574098592 >> v1:172.21.15.158:6789/0 conn(0x7f29bc15daa0 legacy=0x7f29bc06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29b959a700 1 -- 172.21.15.158:0/574098592 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211913958 0 0) 0x7f29bc0ce4d0 con 0x7f29bc15daa0 2023-10-16T10:00:17.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.263+0000 7f29bad9d700 1 -- 172.21.15.158:0/574098592 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc06c490 con 0x7f29bc0d3570 2023-10-16T10:00:17.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29bad9d700 1 --2- 172.21.15.158:0/574098592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d3570 0x7f29bc0d3980 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f29bc08ca90 tx=0x7f29b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361ad1badd140bda server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:17.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29b959a700 1 -- 172.21.15.158:0/574098592 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b400e070 con 0x7f29bc0d3570 2023-10-16T10:00:17.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29b959a700 1 -- 172.21.15.158:0/574098592 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b400bb20 con 0x7f29bc0d3570 2023-10-16T10:00:17.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29b959a700 1 -- 172.21.15.158:0/574098592 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4012430 con 0x7f29bc0d3570 2023-10-16T10:00:17.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 -- 172.21.15.158:0/574098592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d3570 msgr2=0x7f29bc0d3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 --2- 172.21.15.158:0/574098592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d3570 0x7f29bc0d3980 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f29bc08ca90 tx=0x7f29b400b330 comp rx=0 tx=0).stop 2023-10-16T10:00:17.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 -- 172.21.15.158:0/574098592 shutdown_connections 2023-10-16T10:00:17.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 --2- 172.21.15.158:0/574098592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d3570 0x7f29bc0d3980 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 --2- 172.21.15.158:0/574098592 >> v2:172.21.15.172:3300/0 conn(0x7f29bc0d0bd0 0x7f29bc0d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 -- 172.21.15.158:0/574098592 >> 172.21.15.158:0/574098592 conn(0x7f29bc08d040 msgr2=0x7f29bc09c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:17.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 -- 172.21.15.158:0/574098592 shutdown_connections 2023-10-16T10:00:17.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.264+0000 7f29c0957700 1 -- 172.21.15.158:0/574098592 wait complete. 2023-10-16T10:00:17.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.266+0000 7f29c0957700 1 Processor -- start 2023-10-16T10:00:17.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.266+0000 7f29c0957700 1 -- start start 2023-10-16T10:00:17.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.266+0000 7f29c0957700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d0bd0 0x7f29bc0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.266+0000 7f29c0957700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29bc0d3570 0x7f29bc152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29c0957700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29bc15daa0 0x7f29bc0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29c0957700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc0ca120 con 0x7f29bc0d0bd0 2023-10-16T10:00:17.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29c0957700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc0cad80 con 0x7f29bc0d3570 2023-10-16T10:00:17.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29c0957700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc153260 con 0x7f29bc15daa0 2023-10-16T10:00:17.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29ba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d0bd0 0x7f29bc0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29ba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d0bd0 0x7f29bc0ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54480/0 (socket says 172.21.15.158:54480) 2023-10-16T10:00:17.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29ba59c700 1 -- 172.21.15.158:0/624139654 learned_addr learned my addr 172.21.15.158:0/624139654 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:17.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29b9d9b700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29bc0d3570 0x7f29bc152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29bad9d700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29bc15daa0 0x7f29bc0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29b9d9b700 1 -- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29bc15daa0 msgr2=0x7f29bc0cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29b9d9b700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29bc15daa0 0x7f29bc0cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29b9d9b700 1 -- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d0bd0 msgr2=0x7f29bc0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29b9d9b700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d0bd0 0x7f29bc0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29b9d9b700 1 -- 172.21.15.158:0/624139654 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b4009a70 con 0x7f29bc0d3570 2023-10-16T10:00:17.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29ba59c700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d0bd0 0x7f29bc0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:17.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29b9d9b700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29bc0d3570 0x7f29bc152d20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f29b000e8a0 tx=0x7f29b000ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:17.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29aaffd700 1 -- 172.21.15.158:0/624139654 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0013680 con 0x7f29bc0d3570 2023-10-16T10:00:17.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29aaffd700 1 -- 172.21.15.158:0/624139654 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b00137e0 con 0x7f29bc0d3570 2023-10-16T10:00:17.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29bc0c9880 con 0x7f29bc0d3570 2023-10-16T10:00:17.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29aaffd700 1 -- 172.21.15.158:0/624139654 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0013b70 con 0x7f29bc0d3570 2023-10-16T10:00:17.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.267+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29bc0c9e40 con 0x7f29bc0d3570 2023-10-16T10:00:17.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.268+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29bc1e6790 con 0x7f29bc0d3570 2023-10-16T10:00:17.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.269+0000 7f29aaffd700 1 -- 172.21.15.158:0/624139654 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f29b0013cd0 con 0x7f29bc0d3570 2023-10-16T10:00:17.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.269+0000 7f29aaffd700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 0x7f299c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.269+0000 7f29ba59c700 1 -- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 msgr2=0x7f299c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:17.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.269+0000 7f29aaffd700 1 -- 172.21.15.158:0/624139654 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f29b0089900 con 0x7f29bc0d3570 2023-10-16T10:00:17.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.269+0000 7f29ba59c700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 0x7f299c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:17.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.274+0000 7f29aaffd700 1 -- 172.21.15.158:0/624139654 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29b0054460 con 0x7f29bc0d3570 2023-10-16T10:00:17.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.470+0000 7f29ba59c700 1 -- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 msgr2=0x7f299c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:17.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.470+0000 7f29ba59c700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 0x7f299c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.516+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29bc15de80 con 0x7f29bc0d3570 2023-10-16T10:00:17.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.517+0000 7f29aaffd700 1 -- 172.21.15.158:0/624139654 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f29b0057a80 con 0x7f29bc0d3570 2023-10-16T10:00:17.519 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:17.519 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6676,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2569267295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2569267295},{"type":"v1","addr":"172.21.15.172:6835","nonce":2569267295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1485896118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1485896118},{"type":"v1","addr":"172.21.15.172:6834","nonce":1485896118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2664141151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2664141151},{"type":"v1","addr":"172.21.15.158:6836","nonce":2664141151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:59.006477+0000","modified":"2023-10-16T10:00:02.034482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7239":{"gid":7239,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/1006858952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1006858952},{"type":"v1","addr":"172.21.15.158:6837","nonce":1006858952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T10:00:17.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 msgr2=0x7f299c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:17.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 0x7f299c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29bc0d3570 msgr2=0x7f29bc152d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29bc0d3570 0x7f29bc152d20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f29b000e8a0 tx=0x7f29b000ebb0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 shutdown_connections 2023-10-16T10:00:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f299c06b020 0x7f299c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29bc15daa0 0x7f29bc0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29bc0d0bd0 0x7f29bc0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 --2- 172.21.15.158:0/624139654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29bc0d3570 0x7f29bc152d20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 >> 172.21.15.158:0/624139654 conn(0x7f29bc08d040 msgr2=0x7f29bc0d2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 shutdown_connections 2023-10-16T10:00:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.519+0000 7f29c0957700 1 -- 172.21.15.158:0/624139654 wait complete. 2023-10-16T10:00:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 42 2023-10-16T10:00:17.535 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:17.596 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:17.595+0000 7fc5dee94200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:17.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.662+0000 7f5ebc5f6700 1 Processor -- start 2023-10-16T10:00:17.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.663+0000 7f5ebc5f6700 1 -- start start 2023-10-16T10:00:17.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.663+0000 7f5ebc5f6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5eb40cf570 0x7f5eb40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.663+0000 7f5ebc5f6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb40b6540 con 0x7f5eb4161de0 2023-10-16T10:00:17.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.663+0000 7f5ebc5f6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb40d1e20 con 0x7f5eb40b4f60 2023-10-16T10:00:17.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.663+0000 7f5ebc5f6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb40d1ad0 con 0x7f5eb40cf570 2023-10-16T10:00:17.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5eb9b91700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5eb4161de0 0x7f5eb40d1020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54666/0 (socket says 172.21.15.158:54666) 2023-10-16T10:00:17.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5eb9b91700 1 -- 172.21.15.158:0/1343811720 learned_addr learned my addr 172.21.15.158:0/1343811720 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:17.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5ebab93700 1 --2- 172.21.15.158:0/1343811720 >> v2:172.21.15.172:3300/0 conn(0x7f5eb40cf570 0x7f5eb40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5eb9390700 1 -- 172.21.15.158:0/1343811720 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320742004 0 0) 0x7f5eb40b6540 con 0x7f5eb4161de0 2023-10-16T10:00:17.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5eb9390700 1 -- 172.21.15.158:0/1343811720 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea00035b0 con 0x7f5eb4161de0 2023-10-16T10:00:17.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5ebab93700 1 -- 172.21.15.158:0/1343811720 >> v1:172.21.15.172:6789/0 conn(0x7f5eb40b4f60 legacy=0x7f5eb40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5ebab93700 1 -- 172.21.15.158:0/1343811720 >> v1:172.21.15.158:6789/0 conn(0x7f5eb4161de0 legacy=0x7f5eb40d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5ebab93700 1 -- 172.21.15.158:0/1343811720 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb40b8e60 con 0x7f5eb40cf570 2023-10-16T10:00:17.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5ebab93700 1 --2- 172.21.15.158:0/1343811720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40cf570 0x7f5eb40cdec0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009d90 tx=0x7f5eac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e47d00f783a4aa8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:17.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5eb9390700 1 -- 172.21.15.158:0/1343811720 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac00e070 con 0x7f5eb40cf570 2023-10-16T10:00:17.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5eb9390700 1 -- 172.21.15.158:0/1343811720 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eac00bb20 con 0x7f5eb40cf570 2023-10-16T10:00:17.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.664+0000 7f5eb9390700 1 -- 172.21.15.158:0/1343811720 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac012430 con 0x7f5eb40cf570 2023-10-16T10:00:17.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.665+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1343811720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40cf570 msgr2=0x7f5eb40cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.665+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1343811720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40cf570 0x7f5eb40cdec0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009d90 tx=0x7f5eac00b330 comp rx=0 tx=0).stop 2023-10-16T10:00:17.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.665+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1343811720 shutdown_connections 2023-10-16T10:00:17.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.665+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1343811720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40cf570 0x7f5eb40cdec0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.665+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1343811720 >> 172.21.15.158:0/1343811720 conn(0x7f5eb40d34e0 msgr2=0x7f5eb40d38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:17.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.665+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1343811720 shutdown_connections 2023-10-16T10:00:17.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.665+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1343811720 wait complete. 2023-10-16T10:00:17.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.667+0000 7f5ebc5f6700 1 Processor -- start 2023-10-16T10:00:17.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.667+0000 7f5ebc5f6700 1 -- start start 2023-10-16T10:00:17.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebc5f6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40b4f60 0x7f5eb41f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebc5f6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5eb40cf570 0x7f5eb40a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebc5f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5eb4161de0 0x7f5eb40aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebc5f6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41f3e60 con 0x7f5eb4161de0 2023-10-16T10:00:17.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebc5f6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40cd0c0 con 0x7f5eb40b4f60 2023-10-16T10:00:17.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebc5f6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40cc540 con 0x7f5eb40cf570 2023-10-16T10:00:17.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5eb4161de0 0x7f5eb40aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5eb4161de0 0x7f5eb40aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54498/0 (socket says 172.21.15.158:54498) 2023-10-16T10:00:17.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 -- 172.21.15.158:0/1846006757 learned_addr learned my addr 172.21.15.158:0/1846006757 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:17.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5eba392700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40b4f60 0x7f5eb41f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5eb9b91700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5eb40cf570 0x7f5eb40a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:17.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 -- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5eb40cf570 msgr2=0x7f5eb40a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5eb40cf570 0x7f5eb40a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 -- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40b4f60 msgr2=0x7f5eb41f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40b4f60 0x7f5eb41f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 -- 172.21.15.158:0/1846006757 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac009a70 con 0x7f5eb4161de0 2023-10-16T10:00:17.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebab93700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5eb4161de0 0x7f5eb40aa3b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f5eb406cb90 tx=0x7f5eac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:17.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1846006757 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac012430 con 0x7f5eb4161de0 2023-10-16T10:00:17.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1846006757 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eac020070 con 0x7f5eb4161de0 2023-10-16T10:00:17.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb40cc820 con 0x7f5eb4161de0 2023-10-16T10:00:17.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.668+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1846006757 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac004280 con 0x7f5eb4161de0 2023-10-16T10:00:17.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.669+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb40ccde0 con 0x7f5eb4161de0 2023-10-16T10:00:17.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.669+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb41f40a0 con 0x7f5eb4161de0 2023-10-16T10:00:17.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.670+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1846006757 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f5eac0043e0 con 0x7f5eb4161de0 2023-10-16T10:00:17.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.670+0000 7f5eaaffd700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 0x7f5e9806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:17.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.670+0000 7f5eba392700 1 -- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 msgr2=0x7f5e9806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:17.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.670+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1846006757 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f5eac022070 con 0x7f5eb4161de0 2023-10-16T10:00:17.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.670+0000 7f5eba392700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 0x7f5e9806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:17.682 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:17.673+0000 7fc5dee94200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:00:17.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.674+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1846006757 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5eac0567f0 con 0x7f5eb4161de0 2023-10-16T10:00:17.745 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:17.743+0000 7f2e09c28200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:17.819 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:17.818+0000 7fc5dee94200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:17.826 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:17.824+0000 7f2e09c28200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:00:17.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.872+0000 7f5eba392700 1 -- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 msgr2=0x7f5e9806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:17.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.872+0000 7f5eba392700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 0x7f5e9806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:17.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.880+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eb41f4060 con 0x7f5eb4161de0 2023-10-16T10:00:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.880+0000 7f5eaaffd700 1 -- 172.21.15.158:0/1846006757 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17941 (secure 0 0 0) 0x7f5eac059e10 con 0x7f5eb4161de0 2023-10-16T10:00:17.883 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:17.883 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":99,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:12.821319+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:57.438311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:57.966439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1178199719":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/873662109":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6833/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6832/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/1880882485":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/693060665":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/2133642547":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/4289198127":"2023-10-17T10:00:12.821288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 msgr2=0x7f5e9806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 0x7f5e9806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5eb4161de0 msgr2=0x7f5eb40aa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5eb4161de0 0x7f5eb40aa3b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f5eb406cb90 tx=0x7f5eac004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 shutdown_connections 2023-10-16T10:00:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f5e9806b020 0x7f5e9806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5eb4161de0 0x7f5eb40aa3b0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5eb40b4f60 0x7f5eb41f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 --2- 172.21.15.158:0/1846006757 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5eb40cf570 0x7f5eb40a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 >> 172.21.15.158:0/1846006757 conn(0x7f5eb40d34e0 msgr2=0x7f5eb415ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.882+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 shutdown_connections 2023-10-16T10:00:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:17.883+0000 7f5ebc5f6700 1 -- 172.21.15.158:0/1846006757 wait complete. 2023-10-16T10:00:17.897 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T09:59:57.438311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T09:59:57.966439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:00:17.898 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:00:17.984 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:17.981+0000 7f2e09c28200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:18.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.024+0000 7f5543762700 1 Processor -- start 2023-10-16T10:00:18.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.024+0000 7f5543762700 1 -- start start 2023-10-16T10:00:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5543762700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f553c0d0640 0x7f553c1686e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5543762700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0d1ee0 con 0x7f553c15d6c0 2023-10-16T10:00:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5543762700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0d12d0 con 0x7f553c15daa0 2023-10-16T10:00:18.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5543762700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553c0d40a0 con 0x7f553c0d0640 2023-10-16T10:00:18.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5541cff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f553c0d0640 0x7f553c1686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5541cff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f553c0d0640 0x7f553c1686e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54530/0 (socket says 172.21.15.158:54530) 2023-10-16T10:00:18.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5541cff700 1 -- 172.21.15.158:0/3948891844 learned_addr learned my addr 172.21.15.158:0/3948891844 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:18.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f55414fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f553c15d6c0 0x7f553c0cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54702/0 (socket says 172.21.15.158:54702) 2023-10-16T10:00:18.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5541cff700 1 -- 172.21.15.158:0/3948891844 >> v1:172.21.15.172:6789/0 conn(0x7f553c15daa0 legacy=0x7f553c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5541cff700 1 -- 172.21.15.158:0/3948891844 >> v1:172.21.15.158:6789/0 conn(0x7f553c15d6c0 legacy=0x7f553c0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5541cff700 1 -- 172.21.15.158:0/3948891844 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c0d15b0 con 0x7f553c0d0640 2023-10-16T10:00:18.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.025+0000 7f5533fff700 1 -- 172.21.15.158:0/3948891844 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744042855 0 0) 0x7f553c0d1ee0 con 0x7f553c15d6c0 2023-10-16T10:00:18.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5541cff700 1 --2- 172.21.15.158:0/3948891844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c0d0640 0x7f553c1686e0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f5534009e50 tx=0x7f553400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d1b908ea507741 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:18.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5533fff700 1 -- 172.21.15.158:0/3948891844 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553400e070 con 0x7f553c0d0640 2023-10-16T10:00:18.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5533fff700 1 -- 172.21.15.158:0/3948891844 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553400b9c0 con 0x7f553c0d0640 2023-10-16T10:00:18.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5533fff700 1 -- 172.21.15.158:0/3948891844 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553400bd70 con 0x7f553c0d0640 2023-10-16T10:00:18.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5543762700 1 -- 172.21.15.158:0/3948891844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c0d0640 msgr2=0x7f553c1686e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5543762700 1 --2- 172.21.15.158:0/3948891844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c0d0640 0x7f553c1686e0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f5534009e50 tx=0x7f553400b330 comp rx=0 tx=0).stop 2023-10-16T10:00:18.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5543762700 1 -- 172.21.15.158:0/3948891844 shutdown_connections 2023-10-16T10:00:18.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5543762700 1 --2- 172.21.15.158:0/3948891844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c0d0640 0x7f553c1686e0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.026+0000 7f5543762700 1 -- 172.21.15.158:0/3948891844 >> 172.21.15.158:0/3948891844 conn(0x7f553c09c940 msgr2=0x7f553c0d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:18.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.027+0000 7f5543762700 1 -- 172.21.15.158:0/3948891844 shutdown_connections 2023-10-16T10:00:18.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.027+0000 7f5543762700 1 -- 172.21.15.158:0/3948891844 wait complete. 2023-10-16T10:00:18.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.028+0000 7f5543762700 1 Processor -- start 2023-10-16T10:00:18.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.028+0000 7f5543762700 1 -- start start 2023-10-16T10:00:18.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5543762700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f553c0d0640 0x7f553c160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5543762700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f553c15d6c0 0x7f553c0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5543762700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c15daa0 0x7f553c0b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5543762700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c160e70 con 0x7f553c15daa0 2023-10-16T10:00:18.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5543762700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0b6390 con 0x7f553c15d6c0 2023-10-16T10:00:18.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5543762700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0b6670 con 0x7f553c0d0640 2023-10-16T10:00:18.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5541cff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c15daa0 0x7f553c0b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5541cff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c15daa0 0x7f553c0b4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54536/0 (socket says 172.21.15.158:54536) 2023-10-16T10:00:18.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5541cff700 1 -- 172.21.15.158:0/2521403189 learned_addr learned my addr 172.21.15.158:0/2521403189 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:18.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5540cfd700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f553c15d6c0 0x7f553c0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f55414fe700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f553c0d0640 0x7f553c160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.029+0000 7f5541cff700 1 -- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f553c0d0640 msgr2=0x7f553c160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5541cff700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f553c0d0640 0x7f553c160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5541cff700 1 -- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f553c15d6c0 msgr2=0x7f553c0c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5541cff700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f553c15d6c0 0x7f553c0c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5541cff700 1 -- 172.21.15.158:0/2521403189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5534009b30 con 0x7f553c15daa0 2023-10-16T10:00:18.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5541cff700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c15daa0 0x7f553c0b4e00 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f553c08d7d0 tx=0x7f5534004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:18.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5531ffb700 1 -- 172.21.15.158:0/2521403189 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553401c640 con 0x7f553c15daa0 2023-10-16T10:00:18.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5531ffb700 1 -- 172.21.15.158:0/2521403189 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553401c7c0 con 0x7f553c15daa0 2023-10-16T10:00:18.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c0b6950 con 0x7f553c15daa0 2023-10-16T10:00:18.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5531ffb700 1 -- 172.21.15.158:0/2521403189 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553401cb70 con 0x7f553c15daa0 2023-10-16T10:00:18.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.030+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c0b6f10 con 0x7f553c15daa0 2023-10-16T10:00:18.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.031+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553c0681b0 con 0x7f553c15daa0 2023-10-16T10:00:18.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.032+0000 7f5531ffb700 1 -- 172.21.15.158:0/2521403189 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f553401ccd0 con 0x7f553c15daa0 2023-10-16T10:00:18.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.032+0000 7f5531ffb700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 0x7f552006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.032+0000 7f55414fe700 1 -- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 msgr2=0x7f552006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:18.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.032+0000 7f5531ffb700 1 -- 172.21.15.158:0/2521403189 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f553408b580 con 0x7f553c15daa0 2023-10-16T10:00:18.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.032+0000 7f55414fe700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 0x7f552006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:18.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.037+0000 7f5531ffb700 1 -- 172.21.15.158:0/2521403189 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5534056060 con 0x7f553c15daa0 2023-10-16T10:00:18.048 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:18.047+0000 7fc5dee94200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:00:18.122 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:18.121+0000 7fc5dee94200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:00:18.224 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:18.222+0000 7f2e09c28200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:00:18.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.233+0000 7f55414fe700 1 -- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 msgr2=0x7f552006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:18.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.233+0000 7f55414fe700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 0x7f552006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:18.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.236+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f553c0d0d30 con 0x7f553c15daa0 2023-10-16T10:00:18.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.237+0000 7f5531ffb700 1 -- 172.21.15.158:0/2521403189 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5534059680 con 0x7f553c15daa0 2023-10-16T10:00:18.239 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:00:18.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 msgr2=0x7f552006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:18.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 0x7f552006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c15daa0 msgr2=0x7f553c0b4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c15daa0 0x7f553c0b4e00 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f553c08d7d0 tx=0x7f5534004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 shutdown_connections 2023-10-16T10:00:18.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f552006b020 0x7f552006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f553c15daa0 0x7f553c0b4e00 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f553c0d0640 0x7f553c160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 --2- 172.21.15.158:0/2521403189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f553c15d6c0 0x7f553c0c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 >> 172.21.15.158:0/2521403189 conn(0x7f553c09c940 msgr2=0x7f553c0b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:18.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 shutdown_connections 2023-10-16T10:00:18.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.239+0000 7f5543762700 1 -- 172.21.15.158:0/2521403189 wait complete. 2023-10-16T10:00:18.254 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:00:18.307 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:18.305+0000 7f2e09c28200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:00:18.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.380+0000 7f24109fd700 1 Processor -- start 2023-10-16T10:00:18.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.380+0000 7f24109fd700 1 -- start start 2023-10-16T10:00:18.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.381+0000 7f24109fd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f240c0ce510 0x7f240c15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.381+0000 7f24109fd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0cf4c0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.381+0000 7f24109fd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c15f710 con 0x7f240c15b590 2023-10-16T10:00:18.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.381+0000 7f24109fd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c0b49e0 con 0x7f240c0ce510 2023-10-16T10:00:18.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.381+0000 7f2409d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f240c0ce510 0x7f240c15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f240ad9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f240c0cbaf0 0x7f240c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54722/0 (socket says 172.21.15.158:54722) 2023-10-16T10:00:18.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f2409d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f240c0ce510 0x7f240c15f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54544/0 (socket says 172.21.15.158:54544) 2023-10-16T10:00:18.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f240ad9d700 1 -- 172.21.15.158:0/2992880198 learned_addr learned my addr 172.21.15.158:0/2992880198 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:18.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f2409d9b700 1 -- 172.21.15.158:0/2992880198 >> v1:172.21.15.172:6789/0 conn(0x7f240c15b590 legacy=0x7f240c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f2409d9b700 1 -- 172.21.15.158:0/2992880198 >> v1:172.21.15.158:6789/0 conn(0x7f240c0cbaf0 legacy=0x7f240c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f2409d9b700 1 -- 172.21.15.158:0/2992880198 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c15f970 con 0x7f240c0ce510 2023-10-16T10:00:18.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f240959a700 1 -- 172.21.15.158:0/2992880198 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292603209 0 0) 0x7f240c0cf4c0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f2409d9b700 1 --2- 172.21.15.158:0/2992880198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0ce510 0x7f240c15f1d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009840 tx=0x7f23fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d831668fc678bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:18.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f240959a700 1 -- 172.21.15.158:0/2992880198 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc011830 con 0x7f240c0ce510 2023-10-16T10:00:18.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f240959a700 1 -- 172.21.15.158:0/2992880198 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc011990 con 0x7f240c0ce510 2023-10-16T10:00:18.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.382+0000 7f240959a700 1 -- 172.21.15.158:0/2992880198 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc011d20 con 0x7f240c0ce510 2023-10-16T10:00:18.389 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:18.385+0000 7fc5dee94200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:00:18.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.383+0000 7f24109fd700 1 -- 172.21.15.158:0/2992880198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0ce510 msgr2=0x7f240c15f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.383+0000 7f24109fd700 1 --2- 172.21.15.158:0/2992880198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0ce510 0x7f240c15f1d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009840 tx=0x7f23fc009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:18.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.383+0000 7f24109fd700 1 -- 172.21.15.158:0/2992880198 shutdown_connections 2023-10-16T10:00:18.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.383+0000 7f24109fd700 1 --2- 172.21.15.158:0/2992880198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0ce510 0x7f240c15f1d0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.383+0000 7f24109fd700 1 -- 172.21.15.158:0/2992880198 >> 172.21.15.158:0/2992880198 conn(0x7f240c06a8d0 msgr2=0x7f240c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:18.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.383+0000 7f24109fd700 1 -- 172.21.15.158:0/2992880198 shutdown_connections 2023-10-16T10:00:18.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.383+0000 7f24109fd700 1 -- 172.21.15.158:0/2992880198 wait complete. 2023-10-16T10:00:18.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.385+0000 7f24109fd700 1 Processor -- start 2023-10-16T10:00:18.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.385+0000 7f24109fd700 1 -- start start 2023-10-16T10:00:18.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.386+0000 7f24109fd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0cbaf0 0x7f240c1491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.386+0000 7f24109fd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240c0ce510 0x7f240c0c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.386+0000 7f24109fd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240c15b590 0x7f240c0c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.386+0000 7f24109fd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c99b0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.386+0000 7f24109fd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c7a20 con 0x7f240c15b590 2023-10-16T10:00:18.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.386+0000 7f24109fd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c9c10 con 0x7f240c0ce510 2023-10-16T10:00:18.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0cbaf0 0x7f240c1491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0cbaf0 0x7f240c1491e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54548/0 (socket says 172.21.15.158:54548) 2023-10-16T10:00:18.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 -- 172.21.15.158:0/1599599620 learned_addr learned my addr 172.21.15.158:0/1599599620 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:18.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240ad9d700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240c15b590 0x7f240c0c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f2409d9b700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240c0ce510 0x7f240c0c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 -- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240c0ce510 msgr2=0x7f240c0c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240c0ce510 0x7f240c0c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 -- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240c15b590 msgr2=0x7f240c0c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240c15b590 0x7f240c0c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 -- 172.21.15.158:0/1599599620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0094a0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f240a59c700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0cbaf0 0x7f240c1491e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f240000c450 tx=0x7f240000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:18.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f23faffd700 1 -- 172.21.15.158:0/1599599620 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24000052d0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f23faffd700 1 -- 172.21.15.158:0/1599599620 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2400005430 con 0x7f240c0cbaf0 2023-10-16T10:00:18.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c15d2b0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f23faffd700 1 -- 172.21.15.158:0/1599599620 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24000038c0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.387+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c15d870 con 0x7f240c0cbaf0 2023-10-16T10:00:18.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.389+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c0681b0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.389+0000 7f23faffd700 1 -- 172.21.15.158:0/1599599620 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2400003a20 con 0x7f240c0cbaf0 2023-10-16T10:00:18.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.389+0000 7f23faffd700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 0x7f23ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.389+0000 7f2409d9b700 1 -- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 msgr2=0x7f23ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:18.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.389+0000 7f23faffd700 1 -- 172.21.15.158:0/1599599620 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f2400088eb0 con 0x7f240c0cbaf0 2023-10-16T10:00:18.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.389+0000 7f2409d9b700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 0x7f23ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:18.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.394+0000 7f23faffd700 1 -- 172.21.15.158:0/1599599620 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2400053a10 con 0x7f240c0cbaf0 2023-10-16T10:00:18.586 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:18.585+0000 7f2e09c28200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:00:18.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.591+0000 7f2409d9b700 1 -- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 msgr2=0x7f23ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:18.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.591+0000 7f2409d9b700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 0x7f23ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:18.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.592+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f240c157000 con 0x7f240c0cbaf0 2023-10-16T10:00:18.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.593+0000 7f23faffd700 1 -- 172.21.15.158:0/1599599620 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2400017020 con 0x7f240c0cbaf0 2023-10-16T10:00:18.594 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:00:18.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 msgr2=0x7f23ec06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:18.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 0x7f23ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0cbaf0 msgr2=0x7f240c1491e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0cbaf0 0x7f240c1491e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f240000c450 tx=0x7f240000c760 comp rx=0 tx=0).stop 2023-10-16T10:00:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 shutdown_connections 2023-10-16T10:00:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f23ec06b020 0x7f23ec06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f240c15b590 0x7f240c0c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f240c0cbaf0 0x7f240c1491e0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 --2- 172.21.15.158:0/1599599620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f240c0ce510 0x7f240c0c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 >> 172.21.15.158:0/1599599620 conn(0x7f240c06a8d0 msgr2=0x7f240c0b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 shutdown_connections 2023-10-16T10:00:18.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.595+0000 7f24109fd700 1 -- 172.21.15.158:0/1599599620 wait complete. 2023-10-16T10:00:18.609 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:00:18.610 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:18.608+0000 7fc5dee94200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:00:18.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.742+0000 7f1f5b111700 1 Processor -- start 2023-10-16T10:00:18.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.742+0000 7f1f5b111700 1 -- start start 2023-10-16T10:00:18.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.742+0000 7f1f5b111700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f540b3a60 0x7f1f540716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.742+0000 7f1f5b111700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5415b420 con 0x7f1f54071bf0 2023-10-16T10:00:18.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.742+0000 7f1f5b111700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5406d590 con 0x7f1f54143630 2023-10-16T10:00:18.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.742+0000 7f1f5b111700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f54072080 con 0x7f1f540b3a60 2023-10-16T10:00:18.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f596ae700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1f54071bf0 0x7f1f5406b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54738/0 (socket says 172.21.15.158:54738) 2023-10-16T10:00:18.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f596ae700 1 -- 172.21.15.158:0/929475954 learned_addr learned my addr 172.21.15.158:0/929475954 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:18.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f53fff700 1 --2- 172.21.15.158:0/929475954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f540b3a60 0x7f1f540716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819753068 0 0) 0x7f1f5415b420 con 0x7f1f54071bf0 2023-10-16T10:00:18.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f440035b0 con 0x7f1f54071bf0 2023-10-16T10:00:18.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326493532 0 0) 0x7f1f5406d590 con 0x7f1f54143630 2023-10-16T10:00:18.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f5415b420 con 0x7f1f54143630 2023-10-16T10:00:18.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f53fff700 1 -- 172.21.15.158:0/929475954 >> v1:172.21.15.172:6789/0 conn(0x7f1f54143630 legacy=0x7f1f540b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 532657642 0 0) 0x7f1f440035b0 con 0x7f1f54071bf0 2023-10-16T10:00:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f53fff700 1 -- 172.21.15.158:0/929475954 >> v1:172.21.15.158:6789/0 conn(0x7f1f54071bf0 legacy=0x7f1f5406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f53fff700 1 -- 172.21.15.158:0/929475954 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5406d870 con 0x7f1f540b3a60 2023-10-16T10:00:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f53fff700 1 --2- 172.21.15.158:0/929475954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f540b3a60 0x7f1f540716b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009870 tx=0x7f1f40009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=543f303f2ff5cddf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.743+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0019f0 con 0x7f1f540b3a60 2023-10-16T10:00:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f40004030 con 0x7f1f540b3a60 2023-10-16T10:00:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f537fe700 1 -- 172.21.15.158:0/929475954 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f400043c0 con 0x7f1f540b3a60 2023-10-16T10:00:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f5b111700 1 -- 172.21.15.158:0/929475954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f540b3a60 msgr2=0x7f1f540716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f5b111700 1 --2- 172.21.15.158:0/929475954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f540b3a60 0x7f1f540716b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009870 tx=0x7f1f40009b80 comp rx=0 tx=0).stop 2023-10-16T10:00:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f5b111700 1 -- 172.21.15.158:0/929475954 shutdown_connections 2023-10-16T10:00:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f5b111700 1 --2- 172.21.15.158:0/929475954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f540b3a60 0x7f1f540716b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f5b111700 1 -- 172.21.15.158:0/929475954 >> 172.21.15.158:0/929475954 conn(0x7f1f540b1bd0 msgr2=0x7f1f540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:18.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f5b111700 1 -- 172.21.15.158:0/929475954 shutdown_connections 2023-10-16T10:00:18.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.744+0000 7f1f5b111700 1 -- 172.21.15.158:0/929475954 wait complete. 2023-10-16T10:00:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.745+0000 7f1f5b111700 1 Processor -- start 2023-10-16T10:00:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.745+0000 7f1f5b111700 1 -- start start 2023-10-16T10:00:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f5b111700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f54071bf0 0x7f1f54139a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f5b111700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f540b3a60 0x7f1f5415cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f5b111700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f54143630 0x7f1f540ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f5b111700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f54139fb0 con 0x7f1f54071bf0 2023-10-16T10:00:18.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f5b111700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5413aeb0 con 0x7f1f540b3a60 2023-10-16T10:00:18.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f5b111700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5415d5f0 con 0x7f1f54143630 2023-10-16T10:00:18.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f58ead700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f54071bf0 0x7f1f54139a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f58ead700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f54071bf0 0x7f1f54139a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54564/0 (socket says 172.21.15.158:54564) 2023-10-16T10:00:18.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.746+0000 7f1f58ead700 1 -- 172.21.15.158:0/297754335 learned_addr learned my addr 172.21.15.158:0/297754335 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:18.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f596ae700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f54143630 0x7f1f540ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:18.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f58ead700 1 -- 172.21.15.158:0/297754335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f54143630 msgr2=0x7f1f540ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f58ead700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f54143630 0x7f1f540ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f58ead700 1 -- 172.21.15.158:0/297754335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f540b3a60 msgr2=0x7f1f5415cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T10:00:18.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f58ead700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f540b3a60 0x7f1f5415cba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f58ead700 1 -- 172.21.15.158:0/297754335 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f400094a0 con 0x7f1f54071bf0 2023-10-16T10:00:18.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f58ead700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f54071bf0 0x7f1f54139a70 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c500 tx=0x7f1f4800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f517fa700 1 -- 172.21.15.158:0/297754335 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f480052d0 con 0x7f1f54071bf0 2023-10-16T10:00:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f517fa700 1 -- 172.21.15.158:0/297754335 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f48005450 con 0x7f1f54071bf0 2023-10-16T10:00:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5413a580 con 0x7f1f54071bf0 2023-10-16T10:00:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f517fa700 1 -- 172.21.15.158:0/297754335 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f480038c0 con 0x7f1f54071bf0 2023-10-16T10:00:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.747+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5413ab40 con 0x7f1f54071bf0 2023-10-16T10:00:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.749+0000 7f1f517fa700 1 -- 172.21.15.158:0/297754335 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1f48003a20 con 0x7f1f54071bf0 2023-10-16T10:00:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.749+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f540681b0 con 0x7f1f54071bf0 2023-10-16T10:00:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.749+0000 7f1f517fa700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 0x7f1f3806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.749+0000 7f1f53fff700 1 -- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 msgr2=0x7f1f3806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.749+0000 7f1f517fa700 1 -- 172.21.15.158:0/297754335 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f1f48088fb0 con 0x7f1f54071bf0 2023-10-16T10:00:18.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.749+0000 7f1f53fff700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 0x7f1f3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:18.760 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:18.757+0000 7fc5dee94200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:00:18.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.754+0000 7f1f517fa700 1 -- 172.21.15.158:0/297754335 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f48053b00 con 0x7f1f54071bf0 2023-10-16T10:00:18.840 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:18.839+0000 7f2e09c28200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:00:18.874 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:18.872+0000 7fc5dee94200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:00:18.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.950+0000 7f1f53fff700 1 -- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 msgr2=0x7f1f3806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:18.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.950+0000 7f1f53fff700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 0x7f1f3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:18.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.962+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1f540b1fb0 con 0x7f1f54071bf0 2023-10-16T10:00:18.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.962+0000 7f1f517fa700 1 -- 172.21.15.158:0/297754335 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1f480170a0 con 0x7f1f54071bf0 2023-10-16T10:00:18.964 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:00:18.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 msgr2=0x7f1f3806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:18.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 0x7f1f3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f54071bf0 msgr2=0x7f1f54139a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:18.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f54071bf0 0x7f1f54139a70 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c500 tx=0x7f1f4800c810 comp rx=0 tx=0).stop 2023-10-16T10:00:18.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 shutdown_connections 2023-10-16T10:00:18.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f1f3806b020 0x7f1f3806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f54143630 0x7f1f540ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f54071bf0 0x7f1f54139a70 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 --2- 172.21.15.158:0/297754335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f540b3a60 0x7f1f5415cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:18.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.964+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 >> 172.21.15.158:0/297754335 conn(0x7f1f540b1bd0 msgr2=0x7f1f54073db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:18.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.965+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 shutdown_connections 2023-10-16T10:00:18.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:18.965+0000 7f1f5b111700 1 -- 172.21.15.158:0/297754335 wait complete. 2023-10-16T10:00:18.979 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:19.000 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:18.999+0000 7f2e09c28200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:00:19.029 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:19.027+0000 7fc5dee94200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:00:19.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.101+0000 7f9a84578700 1 Processor -- start 2023-10-16T10:00:19.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.101+0000 7f9a84578700 1 -- start start 2023-10-16T10:00:19.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.102+0000 7f9a84578700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a7c0d3570 0x7f9a7c0d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.102+0000 7f9a84578700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c0ce4d0 con 0x7f9a7c15daa0 2023-10-16T10:00:19.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.102+0000 7f9a84578700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c0d3ec0 con 0x7f9a7c0d0bd0 2023-10-16T10:00:19.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.102+0000 7f9a84578700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c0d1c20 con 0x7f9a7c0d3570 2023-10-16T10:00:19.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a82b15700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a7c0d3570 0x7f9a7c0d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a82b15700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a7c0d3570 0x7f9a7c0d3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54578/0 (socket says 172.21.15.158:54578) 2023-10-16T10:00:19.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a82b15700 1 -- 172.21.15.158:0/3382192371 learned_addr learned my addr 172.21.15.158:0/3382192371 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:19.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a82b15700 1 -- 172.21.15.158:0/3382192371 >> v1:172.21.15.172:6789/0 conn(0x7f9a7c0d0bd0 legacy=0x7f9a7c0d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a82b15700 1 -- 172.21.15.158:0/3382192371 >> v1:172.21.15.158:6789/0 conn(0x7f9a7c15daa0 legacy=0x7f9a7c06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a82b15700 1 -- 172.21.15.158:0/3382192371 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c06c1b0 con 0x7f9a7c0d3570 2023-10-16T10:00:19.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a81312700 1 -- 172.21.15.158:0/3382192371 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534861824 0 0) 0x7f9a7c0ce4d0 con 0x7f9a7c15daa0 2023-10-16T10:00:19.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a82b15700 1 --2- 172.21.15.158:0/3382192371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c0d3570 0x7f9a7c0d3980 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009e50 tx=0x7f9a7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7849c02402a5ba09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:19.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a81312700 1 -- 172.21.15.158:0/3382192371 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400e070 con 0x7f9a7c0d3570 2023-10-16T10:00:19.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a81312700 1 -- 172.21.15.158:0/3382192371 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7400b9c0 con 0x7f9a7c0d3570 2023-10-16T10:00:19.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.103+0000 7f9a81312700 1 -- 172.21.15.158:0/3382192371 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400bd70 con 0x7f9a7c0d3570 2023-10-16T10:00:19.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.104+0000 7f9a84578700 1 -- 172.21.15.158:0/3382192371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c0d3570 msgr2=0x7f9a7c0d3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.104+0000 7f9a84578700 1 --2- 172.21.15.158:0/3382192371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c0d3570 0x7f9a7c0d3980 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009e50 tx=0x7f9a7400b330 comp rx=0 tx=0).stop 2023-10-16T10:00:19.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.104+0000 7f9a84578700 1 -- 172.21.15.158:0/3382192371 shutdown_connections 2023-10-16T10:00:19.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.104+0000 7f9a84578700 1 --2- 172.21.15.158:0/3382192371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c0d3570 0x7f9a7c0d3980 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.104+0000 7f9a84578700 1 -- 172.21.15.158:0/3382192371 >> 172.21.15.158:0/3382192371 conn(0x7f9a7c08d040 msgr2=0x7f9a7c09c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:19.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.104+0000 7f9a84578700 1 -- 172.21.15.158:0/3382192371 shutdown_connections 2023-10-16T10:00:19.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.104+0000 7f9a84578700 1 -- 172.21.15.158:0/3382192371 wait complete. 2023-10-16T10:00:19.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.105+0000 7f9a84578700 1 Processor -- start 2023-10-16T10:00:19.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.105+0000 7f9a84578700 1 -- start start 2023-10-16T10:00:19.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a84578700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a7c0d0bd0 0x7f9a7c0b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a84578700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a7c0d3570 0x7f9a7c1f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a84578700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c15daa0 0x7f9a7c0b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a84578700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0c9410 con 0x7f9a7c15daa0 2023-10-16T10:00:19.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a84578700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0cabf0 con 0x7f9a7c0d3570 2023-10-16T10:00:19.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a84578700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c1f3100 con 0x7f9a7c0d0bd0 2023-10-16T10:00:19.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a82b15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c15daa0 0x7f9a7c0b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a82b15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c15daa0 0x7f9a7c0b8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54584/0 (socket says 172.21.15.158:54584) 2023-10-16T10:00:19.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a82b15700 1 -- 172.21.15.158:0/4183775730 learned_addr learned my addr 172.21.15.158:0/4183775730 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:19.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a81b13700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a7c0d3570 0x7f9a7c1f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.106+0000 7f9a82314700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a7c0d0bd0 0x7f9a7c0b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a82b15700 1 -- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a7c0d0bd0 msgr2=0x7f9a7c0b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a82b15700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a7c0d0bd0 0x7f9a7c0b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a82b15700 1 -- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a7c0d3570 msgr2=0x7f9a7c1f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a82b15700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a7c0d3570 0x7f9a7c1f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a82b15700 1 -- 172.21.15.158:0/4183775730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a74009b30 con 0x7f9a7c15daa0 2023-10-16T10:00:19.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a82b15700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c15daa0 0x7f9a7c0b8820 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9a74008dd0 tx=0x7f9a74004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:19.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a72ffd700 1 -- 172.21.15.158:0/4183775730 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7401c640 con 0x7f9a7c15daa0 2023-10-16T10:00:19.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a72ffd700 1 -- 172.21.15.158:0/4183775730 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7401c7c0 con 0x7f9a7c15daa0 2023-10-16T10:00:19.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c1f7670 con 0x7f9a7c15daa0 2023-10-16T10:00:19.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a72ffd700 1 -- 172.21.15.158:0/4183775730 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7401cb70 con 0x7f9a7c15daa0 2023-10-16T10:00:19.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.107+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c0c9130 con 0x7f9a7c15daa0 2023-10-16T10:00:19.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.108+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c0cad80 con 0x7f9a7c15daa0 2023-10-16T10:00:19.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.108+0000 7f9a72ffd700 1 -- 172.21.15.158:0/4183775730 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f9a7401ccd0 con 0x7f9a7c15daa0 2023-10-16T10:00:19.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.109+0000 7f9a72ffd700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 0x7f9a6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.109+0000 7f9a82314700 1 -- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 msgr2=0x7f9a6006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:19.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.109+0000 7f9a72ffd700 1 -- 172.21.15.158:0/4183775730 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f9a7408b550 con 0x7f9a7c15daa0 2023-10-16T10:00:19.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.109+0000 7f9a82314700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 0x7f9a6006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:19.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.113+0000 7f9a72ffd700 1 -- 172.21.15.158:0/4183775730 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a74056030 con 0x7f9a7c15daa0 2023-10-16T10:00:19.123 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:19.123+0000 7f2e09c28200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:00:19.286 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:19.285+0000 7f2e09c28200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:00:19.292 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:19.290+0000 7fc5dee94200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:19.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.310+0000 7f9a82314700 1 -- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 msgr2=0x7f9a6006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:19.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.310+0000 7f9a82314700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 0x7f9a6006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:19.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.359+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a7c0681b0 con 0x7f9a7c15daa0 2023-10-16T10:00:19.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.359+0000 7f9a72ffd700 1 -- 172.21.15.158:0/4183775730 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f9a74059650 con 0x7f9a7c15daa0 2023-10-16T10:00:19.361 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:19.361 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6676,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2569267295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2569267295},{"type":"v1","addr":"172.21.15.172:6835","nonce":2569267295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1485896118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1485896118},{"type":"v1","addr":"172.21.15.172:6834","nonce":1485896118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2664141151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2664141151},{"type":"v1","addr":"172.21.15.158:6836","nonce":2664141151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:59.006477+0000","modified":"2023-10-16T10:00:02.034482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7239":{"gid":7239,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/1006858952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1006858952},{"type":"v1","addr":"172.21.15.158:6837","nonce":1006858952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T10:00:19.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 msgr2=0x7f9a6006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:19.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 0x7f9a6006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c15daa0 msgr2=0x7f9a7c0b8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c15daa0 0x7f9a7c0b8820 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9a74008dd0 tx=0x7f9a74004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 shutdown_connections 2023-10-16T10:00:19.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f9a6006b020 0x7f9a6006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a7c15daa0 0x7f9a7c0b8820 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a7c0d0bd0 0x7f9a7c0b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 --2- 172.21.15.158:0/4183775730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a7c0d3570 0x7f9a7c1f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 >> 172.21.15.158:0/4183775730 conn(0x7f9a7c08d040 msgr2=0x7f9a7c0d1510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:19.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 shutdown_connections 2023-10-16T10:00:19.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.362+0000 7f9a84578700 1 -- 172.21.15.158:0/4183775730 wait complete. 2023-10-16T10:00:19.366 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 42 2023-10-16T10:00:19.377 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:19.450 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:19.449+0000 7fc5dee94200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:00:19.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.540+0000 7fe91cc1b700 1 Processor -- start 2023-10-16T10:00:19.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.540+0000 7fe91cc1b700 1 -- start start 2023-10-16T10:00:19.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91cc1b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180ce530 0x7fe9180b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91cc1b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe91815f0f0 0x7fe9180cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91cc1b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91815b930 con 0x7fe9180cbfe0 2023-10-16T10:00:19.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91cc1b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9180cc490 con 0x7fe91815f0f0 2023-10-16T10:00:19.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91cc1b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9180cf020 con 0x7fe9180ce530 2023-10-16T10:00:19.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe91815f0f0 0x7fe9180cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe91815f0f0 0x7fe9180cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54592/0 (socket says 172.21.15.158:54592) 2023-10-16T10:00:19.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 -- 172.21.15.158:0/639666259 learned_addr learned my addr 172.21.15.158:0/639666259 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:19.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe916d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe9180cbfe0 0x7fe9180d13e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54770/0 (socket says 172.21.15.158:54770) 2023-10-16T10:00:19.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91659c700 1 --2- 172.21.15.158:0/639666259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180ce530 0x7fe9180b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 -- 172.21.15.158:0/639666259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180ce530 msgr2=0x7fe9180b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:19.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 --2- 172.21.15.158:0/639666259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180ce530 0x7fe9180b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 -- 172.21.15.158:0/639666259 >> v1:172.21.15.158:6789/0 conn(0x7fe9180cbfe0 legacy=0x7fe9180d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91559a700 1 -- 172.21.15.158:0/639666259 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221976634 0 0) 0x7fe91815b930 con 0x7fe9180cbfe0 2023-10-16T10:00:19.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 -- 172.21.15.158:0/639666259 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91815f5f0 con 0x7fe91815f0f0 2023-10-16T10:00:19.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe915d9b700 1 --2- 172.21.15.158:0/639666259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe91815f0f0 0x7fe9180cbaa0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0098f0 tx=0x7fe90c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3995d9125365f537 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:19.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91559a700 1 -- 172.21.15.158:0/639666259 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c011830 con 0x7fe91815f0f0 2023-10-16T10:00:19.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91559a700 1 -- 172.21.15.158:0/639666259 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c011990 con 0x7fe91815f0f0 2023-10-16T10:00:19.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.542+0000 7fe91559a700 1 -- 172.21.15.158:0/639666259 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c011d20 con 0x7fe91815f0f0 2023-10-16T10:00:19.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 -- 172.21.15.158:0/639666259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe91815f0f0 msgr2=0x7fe9180cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/639666259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe91815f0f0 0x7fe9180cbaa0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0098f0 tx=0x7fe90c009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:19.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 -- 172.21.15.158:0/639666259 shutdown_connections 2023-10-16T10:00:19.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/639666259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180ce530 0x7fe9180b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/639666259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe91815f0f0 0x7fe9180cbaa0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 -- 172.21.15.158:0/639666259 >> 172.21.15.158:0/639666259 conn(0x7fe91806a8d0 msgr2=0x7fe91806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:19.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 -- 172.21.15.158:0/639666259 shutdown_connections 2023-10-16T10:00:19.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.543+0000 7fe91cc1b700 1 -- 172.21.15.158:0/639666259 wait complete. 2023-10-16T10:00:19.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.546+0000 7fe91cc1b700 1 Processor -- start 2023-10-16T10:00:19.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.546+0000 7fe91cc1b700 1 -- start start 2023-10-16T10:00:19.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe91cc1b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180cbfe0 0x7fe918152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe91cc1b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9180ce530 0x7fe9180ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe91cc1b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe91815f0f0 0x7fe9180cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe91cc1b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe918153240 con 0x7fe9180ce530 2023-10-16T10:00:19.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe91cc1b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91806b9e0 con 0x7fe91815f0f0 2023-10-16T10:00:19.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe91cc1b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91806bc30 con 0x7fe9180cbfe0 2023-10-16T10:00:19.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9180ce530 0x7fe9180ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9180ce530 0x7fe9180ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54596/0 (socket says 172.21.15.158:54596) 2023-10-16T10:00:19.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 -- 172.21.15.158:0/104004934 learned_addr learned my addr 172.21.15.158:0/104004934 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:19.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe91659c700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180cbfe0 0x7fe918152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe916d9d700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe91815f0f0 0x7fe9180cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 -- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180cbfe0 msgr2=0x7fe918152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180cbfe0 0x7fe918152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 -- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe91815f0f0 msgr2=0x7fe9180cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe91815f0f0 0x7fe9180cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.547+0000 7fe915d9b700 1 -- 172.21.15.158:0/104004934 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c009550 con 0x7fe9180ce530 2023-10-16T10:00:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.548+0000 7fe915d9b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9180ce530 0x7fe9180ca6f0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009c40 tx=0x7fe90c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:19.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.548+0000 7fe906ffd700 1 -- 172.21.15.158:0/104004934 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0045d0 con 0x7fe9180ce530 2023-10-16T10:00:19.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.548+0000 7fe906ffd700 1 -- 172.21.15.158:0/104004934 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c01f070 con 0x7fe9180ce530 2023-10-16T10:00:19.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.548+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91806bf10 con 0x7fe9180ce530 2023-10-16T10:00:19.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.548+0000 7fe906ffd700 1 -- 172.21.15.158:0/104004934 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0038a0 con 0x7fe9180ce530 2023-10-16T10:00:19.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.548+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91806c4d0 con 0x7fe9180ce530 2023-10-16T10:00:19.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.549+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9180681b0 con 0x7fe9180ce530 2023-10-16T10:00:19.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.549+0000 7fe906ffd700 1 -- 172.21.15.158:0/104004934 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe90c003a00 con 0x7fe9180ce530 2023-10-16T10:00:19.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.550+0000 7fe906ffd700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 0x7fe8f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.550+0000 7fe91659c700 1 -- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 msgr2=0x7fe8f806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:19.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.550+0000 7fe906ffd700 1 -- 172.21.15.158:0/104004934 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fe90c021070 con 0x7fe9180ce530 2023-10-16T10:00:19.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.550+0000 7fe91659c700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 0x7fe8f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:19.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.554+0000 7fe906ffd700 1 -- 172.21.15.158:0/104004934 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe90c056000 con 0x7fe9180ce530 2023-10-16T10:00:19.566 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:19.565+0000 7f2e09c28200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:19.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:19.734+0000 7f2e09c28200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:00:19.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.751+0000 7fe91659c700 1 -- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 msgr2=0x7fe8f806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:19.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.751+0000 7fe91659c700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 0x7fe8f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:19.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.758+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9180cd210 con 0x7fe9180ce530 2023-10-16T10:00:19.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.760+0000 7fe906ffd700 1 -- 172.21.15.158:0/104004934 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17941 (secure 0 0 0) 0x7fe90c003cb0 con 0x7fe9180ce530 2023-10-16T10:00:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:19.762 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":99,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:12.821319+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:57.438311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:57.966439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1178199719":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/873662109":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6833/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6832/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/1880882485":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/693060665":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/2133642547":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/4289198127":"2023-10-17T10:00:12.821288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:19.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 msgr2=0x7fe8f806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:19.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 0x7fe8f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9180ce530 msgr2=0x7fe9180ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9180ce530 0x7fe9180ca6f0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009c40 tx=0x7fe90c004750 comp rx=0 tx=0).stop 2023-10-16T10:00:19.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 shutdown_connections 2023-10-16T10:00:19.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7fe8f806b020 0x7fe8f806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe91815f0f0 0x7fe9180cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9180cbfe0 0x7fe918152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 --2- 172.21.15.158:0/104004934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9180ce530 0x7fe9180ca6f0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 >> 172.21.15.158:0/104004934 conn(0x7fe91806a8d0 msgr2=0x7fe9180cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:19.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 shutdown_connections 2023-10-16T10:00:19.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.762+0000 7fe91cc1b700 1 -- 172.21.15.158:0/104004934 wait complete. 2023-10-16T10:00:19.777 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T10:00:19.777 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:19.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.901+0000 7f82e1da3700 1 Processor -- start 2023-10-16T10:00:19.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.901+0000 7f82e1da3700 1 -- start start 2023-10-16T10:00:19.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82e1da3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82dc0d0bd0 0x7f82dc0d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82e1da3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82e1da3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc0ce4d0 con 0x7f82dc15daa0 2023-10-16T10:00:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82e1da3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82dc0d3f00 con 0x7f82dc0d0bd0 2023-10-16T10:00:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82e1da3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc06c1b0 con 0x7f82dc0d3570 2023-10-16T10:00:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82daffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82dc0d0bd0 0x7f82dc0d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82daffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82dc0d0bd0 0x7f82dc0d1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54608/0 (socket says 172.21.15.158:54608) 2023-10-16T10:00:19.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82daffd700 1 -- 172.21.15.158:0/1451432416 learned_addr learned my addr 172.21.15.158:0/1451432416 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:19.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.902+0000 7f82dbfff700 1 --2- 172.21.15.158:0/1451432416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82daffd700 1 -- 172.21.15.158:0/1451432416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 msgr2=0x7f82dc0d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:19.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82daffd700 1 --2- 172.21.15.158:0/1451432416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82daffd700 1 -- 172.21.15.158:0/1451432416 >> v1:172.21.15.172:6789/0 conn(0x7f82dc15daa0 legacy=0x7f82dc06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82daffd700 1 -- 172.21.15.158:0/1451432416 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc06c490 con 0x7f82dc0d0bd0 2023-10-16T10:00:19.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82daffd700 1 --2- 172.21.15.158:0/1451432416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc0d0bd0 0x7f82dc0d1510 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f82d00098f0 tx=0x7f82d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4440c31d5e9189fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:19.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82da7fc700 1 -- 172.21.15.158:0/1451432416 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0011830 con 0x7f82dc0d0bd0 2023-10-16T10:00:19.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82da7fc700 1 -- 172.21.15.158:0/1451432416 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d0011990 con 0x7f82dc0d0bd0 2023-10-16T10:00:19.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82da7fc700 1 -- 172.21.15.158:0/1451432416 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0011d20 con 0x7f82dc0d0bd0 2023-10-16T10:00:19.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82e1da3700 1 -- 172.21.15.158:0/1451432416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc0d0bd0 msgr2=0x7f82dc0d1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82e1da3700 1 --2- 172.21.15.158:0/1451432416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc0d0bd0 0x7f82dc0d1510 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f82d00098f0 tx=0x7f82d0009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:19.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82e1da3700 1 -- 172.21.15.158:0/1451432416 shutdown_connections 2023-10-16T10:00:19.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82e1da3700 1 --2- 172.21.15.158:0/1451432416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82e1da3700 1 --2- 172.21.15.158:0/1451432416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc0d0bd0 0x7f82dc0d1510 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.903+0000 7f82e1da3700 1 -- 172.21.15.158:0/1451432416 >> 172.21.15.158:0/1451432416 conn(0x7f82dc08d040 msgr2=0x7f82dc09c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:19.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.904+0000 7f82e1da3700 1 -- 172.21.15.158:0/1451432416 shutdown_connections 2023-10-16T10:00:19.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.904+0000 7f82e1da3700 1 -- 172.21.15.158:0/1451432416 wait complete. 2023-10-16T10:00:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.905+0000 7f82e1da3700 1 Processor -- start 2023-10-16T10:00:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.905+0000 7f82e1da3700 1 -- start start 2023-10-16T10:00:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82dc0d0bd0 0x7f82dc152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc15daa0 0x7f82dc0c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0b3ff0 con 0x7f82dc15daa0 2023-10-16T10:00:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc153310 con 0x7f82dc0d0bd0 2023-10-16T10:00:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0b3100 con 0x7f82dc0d3570 2023-10-16T10:00:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc15daa0 0x7f82dc0c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc15daa0 0x7f82dc0c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54616/0 (socket says 172.21.15.158:54616) 2023-10-16T10:00:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 -- 172.21.15.158:0/3796504609 learned_addr learned my addr 172.21.15.158:0/3796504609 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82db7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82dc0d0bd0 0x7f82dc152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82daffd700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 -- 172.21.15.158:0/3796504609 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 msgr2=0x7f82dc0cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 -- 172.21.15.158:0/3796504609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82dc0d0bd0 msgr2=0x7f82dc152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82dc0d0bd0 0x7f82dc152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 -- 172.21.15.158:0/3796504609 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0009550 con 0x7f82dc15daa0 2023-10-16T10:00:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82dbfff700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc15daa0 0x7f82dc0c8520 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f82d400efc0 tx=0x7f82d400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82c3fff700 1 -- 172.21.15.158:0/3796504609 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4009ea0 con 0x7f82dc15daa0 2023-10-16T10:00:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc0b33e0 con 0x7f82dc15daa0 2023-10-16T10:00:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82c3fff700 1 -- 172.21.15.158:0/3796504609 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d400f040 con 0x7f82dc15daa0 2023-10-16T10:00:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82c3fff700 1 -- 172.21.15.158:0/3796504609 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d400ced0 con 0x7f82dc15daa0 2023-10-16T10:00:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.906+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc0b39a0 con 0x7f82dc15daa0 2023-10-16T10:00:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.907+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82dc0681b0 con 0x7f82dc15daa0 2023-10-16T10:00:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.908+0000 7f82c3fff700 1 -- 172.21.15.158:0/3796504609 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f82d4014480 con 0x7f82dc15daa0 2023-10-16T10:00:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.908+0000 7f82c3fff700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.908+0000 7f82db7fe700 1 -- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 msgr2=0x7f82c806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.909+0000 7f82c3fff700 1 -- 172.21.15.158:0/3796504609 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f82d408ac70 con 0x7f82dc15daa0 2023-10-16T10:00:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.909+0000 7f82db7fe700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:19.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:19.912+0000 7f82c3fff700 1 -- 172.21.15.158:0/3796504609 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82d4055750 con 0x7f82dc15daa0 2023-10-16T10:00:19.977 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:19.976+0000 7fc5dee94200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:00:20.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.110+0000 7f82db7fe700 1 -- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 msgr2=0x7f82c806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:20.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.110+0000 7f82db7fe700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:20.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.158+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82dc0d4450 con 0x7f82dc15daa0 2023-10-16T10:00:20.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.158+0000 7f82c3fff700 1 -- 172.21.15.158:0/3796504609 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f82d4018020 con 0x7f82dc15daa0 2023-10-16T10:00:20.160 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:20.160 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6676,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2569267295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2569267295},{"type":"v1","addr":"172.21.15.172:6835","nonce":2569267295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1485896118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1485896118},{"type":"v1","addr":"172.21.15.172:6834","nonce":1485896118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2664141151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2664141151},{"type":"v1","addr":"172.21.15.158:6836","nonce":2664141151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:59.006477+0000","modified":"2023-10-16T10:00:02.034482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7239":{"gid":7239,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/1006858952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1006858952},{"type":"v1","addr":"172.21.15.158:6837","nonce":1006858952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T10:00:20.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 msgr2=0x7f82c806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:20.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc15daa0 msgr2=0x7f82dc0c8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc15daa0 0x7f82dc0c8520 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f82d400efc0 tx=0x7f82d400c3a0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 shutdown_connections 2023-10-16T10:00:20.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82dc15daa0 0x7f82dc0c8520 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82dc0d0bd0 0x7f82dc152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 --2- 172.21.15.158:0/3796504609 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82dc0d3570 0x7f82dc0cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.160+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 >> 172.21.15.158:0/3796504609 conn(0x7f82dc08d040 msgr2=0x7f82dc06cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:20.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.161+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 shutdown_connections 2023-10-16T10:00:20.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.161+0000 7f82e1da3700 1 -- 172.21.15.158:0/3796504609 wait complete. 2023-10-16T10:00:20.165 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 42 2023-10-16T10:00:20.174 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:20.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.296+0000 7f8299fa0700 1 Processor -- start 2023-10-16T10:00:20.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.296+0000 7f8299fa0700 1 -- start start 2023-10-16T10:00:20.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.296+0000 7f8299fa0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f829415a6c0 0x7f82940723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.296+0000 7f8299fa0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82940af590 0x7f829406d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.296+0000 7f8299fa0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f829406dec0 0x7f829415e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.296+0000 7f8299fa0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829406e370 con 0x7f82940af590 2023-10-16T10:00:20.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8299fa0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82940ca7e0 con 0x7f829415a6c0 2023-10-16T10:00:20.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8299fa0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8294072910 con 0x7f829406dec0 2023-10-16T10:00:20.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82940af590 0x7f829406d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82940af590 0x7f829406d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54624/0 (socket says 172.21.15.158:54624) 2023-10-16T10:00:20.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 -- 172.21.15.158:0/338993459 learned_addr learned my addr 172.21.15.158:0/338993459 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:20.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f82937fe700 1 --2- 172.21.15.158:0/338993459 >> v2:172.21.15.172:3300/0 conn(0x7f829415a6c0 0x7f82940723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8293fff700 1 --2- 172.21.15.158:0/338993459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f829406dec0 0x7f829415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 -- 172.21.15.158:0/338993459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f829406dec0 msgr2=0x7f829415e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:20.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 --2- 172.21.15.158:0/338993459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f829406dec0 0x7f829415e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 -- 172.21.15.158:0/338993459 >> v2:172.21.15.172:3300/0 conn(0x7f829415a6c0 msgr2=0x7f82940723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:20.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 --2- 172.21.15.158:0/338993459 >> v2:172.21.15.172:3300/0 conn(0x7f829415a6c0 0x7f82940723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 -- 172.21.15.158:0/338993459 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82940b1e40 con 0x7f82940af590 2023-10-16T10:00:20.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f82937fe700 1 --2- 172.21.15.158:0/338993459 >> v2:172.21.15.172:3300/0 conn(0x7f829415a6c0 0x7f82940723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:20.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f8292ffd700 1 --2- 172.21.15.158:0/338993459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940af590 0x7f829406d980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f82940737c0 tx=0x7f8288009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58649acf084e0e1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:20.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f82927fc700 1 -- 172.21.15.158:0/338993459 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828800b870 con 0x7f82940af590 2023-10-16T10:00:20.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f82927fc700 1 -- 172.21.15.158:0/338993459 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828800b9d0 con 0x7f82940af590 2023-10-16T10:00:20.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.297+0000 7f82927fc700 1 -- 172.21.15.158:0/338993459 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828800bd60 con 0x7f82940af590 2023-10-16T10:00:20.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 -- 172.21.15.158:0/338993459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940af590 msgr2=0x7f829406d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 --2- 172.21.15.158:0/338993459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940af590 0x7f829406d980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f82940737c0 tx=0x7f8288009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 -- 172.21.15.158:0/338993459 shutdown_connections 2023-10-16T10:00:20.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 --2- 172.21.15.158:0/338993459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f829406dec0 0x7f829415e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 --2- 172.21.15.158:0/338993459 >> v2:172.21.15.172:3300/0 conn(0x7f829415a6c0 0x7f82940723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 --2- 172.21.15.158:0/338993459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940af590 0x7f829406d980 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 -- 172.21.15.158:0/338993459 >> 172.21.15.158:0/338993459 conn(0x7f82940ccda0 msgr2=0x7f82940cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:20.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 -- 172.21.15.158:0/338993459 shutdown_connections 2023-10-16T10:00:20.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.298+0000 7f8299fa0700 1 -- 172.21.15.158:0/338993459 wait complete. 2023-10-16T10:00:20.307 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.299+0000 7f2e09c28200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:00:20.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.299+0000 7f8299fa0700 1 Processor -- start 2023-10-16T10:00:20.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.299+0000 7f8299fa0700 1 -- start start 2023-10-16T10:00:20.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8299fa0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829406dec0 0x7f8294159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8299fa0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82940af590 0x7f82940c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8299fa0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829415a6c0 0x7f82940c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8299fa0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82940ae740 con 0x7f829415a6c0 2023-10-16T10:00:20.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8299fa0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8294159b40 con 0x7f829406dec0 2023-10-16T10:00:20.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8299fa0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82940c6c60 con 0x7f82940af590 2023-10-16T10:00:20.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8293fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829415a6c0 0x7f82940c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8293fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829415a6c0 0x7f82940c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54632/0 (socket says 172.21.15.158:54632) 2023-10-16T10:00:20.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8293fff700 1 -- 172.21.15.158:0/2167360523 learned_addr learned my addr 172.21.15.158:0/2167360523 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:20.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f8292ffd700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82940af590 0x7f82940c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.300+0000 7f82937fe700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829406dec0 0x7f8294159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8293fff700 1 -- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82940af590 msgr2=0x7f82940c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8293fff700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82940af590 0x7f82940c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8293fff700 1 -- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829406dec0 msgr2=0x7f8294159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8293fff700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829406dec0 0x7f8294159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8293fff700 1 -- 172.21.15.158:0/2167360523 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8288009550 con 0x7f829415a6c0 2023-10-16T10:00:20.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8293fff700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829415a6c0 0x7f82940c8650 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f828c007e60 tx=0x7f828c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:20.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c00ed30 con 0x7f829415a6c0 2023-10-16T10:00:20.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c014420 con 0x7f829415a6c0 2023-10-16T10:00:20.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82940c3f80 con 0x7f829415a6c0 2023-10-16T10:00:20.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c0147d0 con 0x7f829415a6c0 2023-10-16T10:00:20.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.301+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82940c4540 con 0x7f829415a6c0 2023-10-16T10:00:20.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.302+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82940681b0 con 0x7f829415a6c0 2023-10-16T10:00:20.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.302+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f828c014930 con 0x7f829415a6c0 2023-10-16T10:00:20.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.303+0000 7f827bfff700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f828006b020 0x7f828006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.303+0000 7f82937fe700 1 -- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f828006b020 msgr2=0x7f828006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/334523996 2023-10-16T10:00:20.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.303+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f828c08a1b0 con 0x7f829415a6c0 2023-10-16T10:00:20.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.303+0000 7f82937fe700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f828006b020 0x7f828006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:20.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.307+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f828c054d20 con 0x7f829415a6c0 2023-10-16T10:00:20.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.321+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 48) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f828c0550f0 con 0x7f829415a6c0 2023-10-16T10:00:20.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.322+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f828006b020 msgr2=0x7f828006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:20.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.322+0000 7f827bfff700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f828006b020 0x7f828006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.384 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d85454700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.384 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d85454700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.386 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d85454700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.386 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d85454700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.386 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d85454700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.386 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d8240e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.387 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d8240e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.387 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d8240e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.387 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d8240e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.388 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:20.383+0000 7f2d8240e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:20.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.563+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82940b3e40 con 0x7f829415a6c0 2023-10-16T10:00:20.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.564+0000 7f827bfff700 1 -- 172.21.15.158:0/2167360523 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f828c074250 con 0x7f829415a6c0 2023-10-16T10:00:20.566 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:20.566 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6676,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2569267295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2569267295},{"type":"v1","addr":"172.21.15.172:6835","nonce":2569267295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1485896118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1485896118},{"type":"v1","addr":"172.21.15.172:6834","nonce":1485896118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2664141151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2664141151},{"type":"v1","addr":"172.21.15.158:6836","nonce":2664141151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T09:59:59.006477+0000","modified":"2023-10-16T10:00:02.034482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7239":{"gid":7239,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/1006858952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1006858952},{"type":"v1","addr":"172.21.15.158:6837","nonce":1006858952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T10:00:20.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829415a6c0 msgr2=0x7f82940c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829415a6c0 0x7f82940c8650 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f828c007e60 tx=0x7f828c00e3c0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 shutdown_connections 2023-10-16T10:00:20.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:6832/334523996,v1:172.21.15.158:6833/334523996] conn(0x7f828006b020 0x7f828006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829415a6c0 0x7f82940c8650 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829406dec0 0x7f8294159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 --2- 172.21.15.158:0/2167360523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82940af590 0x7f82940c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 >> 172.21.15.158:0/2167360523 conn(0x7f82940ccda0 msgr2=0x7f82940cbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:20.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 shutdown_connections 2023-10-16T10:00:20.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.566+0000 7f8299fa0700 1 -- 172.21.15.158:0/2167360523 wait complete. 2023-10-16T10:00:20.570 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 42 2023-10-16T10:00:20.580 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:20.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.699+0000 7f8af37a4700 1 Processor -- start 2023-10-16T10:00:20.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.700+0000 7f8af37a4700 1 -- start start 2023-10-16T10:00:20.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.700+0000 7f8af37a4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8aec06ab40 0x7f8aec06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af37a4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8aec0729b0 0x7f8aec071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af37a4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec071e50 con 0x7f8aec06e5d0 2023-10-16T10:00:20.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af37a4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec161270 con 0x7f8aec06ab40 2023-10-16T10:00:20.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af37a4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec161510 con 0x7f8aec0729b0 2023-10-16T10:00:20.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8aec06ab40 0x7f8aec06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8aec06ab40 0x7f8aec06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54706/0 (socket says 172.21.15.158:54706) 2023-10-16T10:00:20.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 -- 172.21.15.158:0/1016432104 learned_addr learned my addr 172.21.15.158:0/1016432104 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:20.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1d41700 1 --2- 172.21.15.158:0/1016432104 >> v2:172.21.15.172:3300/0 conn(0x7f8aec0729b0 0x7f8aec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 -- 172.21.15.158:0/1016432104 >> v2:172.21.15.172:3300/0 conn(0x7f8aec0729b0 msgr2=0x7f8aec071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:20.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 --2- 172.21.15.158:0/1016432104 >> v2:172.21.15.172:3300/0 conn(0x7f8aec0729b0 0x7f8aec071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 -- 172.21.15.158:0/1016432104 >> v1:172.21.15.172:6789/0 conn(0x7f8aec06e5d0 legacy=0x7f8aec0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 -- 172.21.15.158:0/1016432104 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec1617f0 con 0x7f8aec06ab40 2023-10-16T10:00:20.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8af1540700 1 --2- 172.21.15.158:0/1016432104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06ab40 0x7f8aec06d820 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8aec158240 tx=0x7f8adc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e207d1d056a3d01c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:20.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8ae3fff700 1 -- 172.21.15.158:0/1016432104 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc011830 con 0x7f8aec06ab40 2023-10-16T10:00:20.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8ae3fff700 1 -- 172.21.15.158:0/1016432104 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc011990 con 0x7f8aec06ab40 2023-10-16T10:00:20.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.701+0000 7f8ae3fff700 1 -- 172.21.15.158:0/1016432104 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc011d20 con 0x7f8aec06ab40 2023-10-16T10:00:20.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 -- 172.21.15.158:0/1016432104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06ab40 msgr2=0x7f8aec06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 --2- 172.21.15.158:0/1016432104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06ab40 0x7f8aec06d820 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8aec158240 tx=0x7f8adc009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 -- 172.21.15.158:0/1016432104 shutdown_connections 2023-10-16T10:00:20.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 --2- 172.21.15.158:0/1016432104 >> v2:172.21.15.172:3300/0 conn(0x7f8aec0729b0 0x7f8aec071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 --2- 172.21.15.158:0/1016432104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06ab40 0x7f8aec06d820 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 -- 172.21.15.158:0/1016432104 >> 172.21.15.158:0/1016432104 conn(0x7f8aec0d02b0 msgr2=0x7f8aec0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:20.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 -- 172.21.15.158:0/1016432104 shutdown_connections 2023-10-16T10:00:20.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.702+0000 7f8af37a4700 1 -- 172.21.15.158:0/1016432104 wait complete. 2023-10-16T10:00:20.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.703+0000 7f8af37a4700 1 Processor -- start 2023-10-16T10:00:20.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.703+0000 7f8af37a4700 1 -- start start 2023-10-16T10:00:20.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af37a4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec06ab40 0x7f8aec1f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af37a4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06e5d0 0x7f8aec0c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af37a4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec0729b0 0x7f8aec0c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:20.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af37a4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0b1120 con 0x7f8aec06e5d0 2023-10-16T10:00:20.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af37a4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0b1400 con 0x7f8aec0729b0 2023-10-16T10:00:20.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af37a4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0b16e0 con 0x7f8aec06ab40 2023-10-16T10:00:20.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06e5d0 0x7f8aec0c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06e5d0 0x7f8aec0c83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54710/0 (socket says 172.21.15.158:54710) 2023-10-16T10:00:20.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 -- 172.21.15.158:0/2502848295 learned_addr learned my addr 172.21.15.158:0/2502848295 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:20.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af1d41700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec0729b0 0x7f8aec0c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af1540700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec06ab40 0x7f8aec1f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:20.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 -- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec06ab40 msgr2=0x7f8aec1f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec06ab40 0x7f8aec1f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 -- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec0729b0 msgr2=0x7f8aec0c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec0729b0 0x7f8aec0c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 -- 172.21.15.158:0/2502848295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc009550 con 0x7f8aec06e5d0 2023-10-16T10:00:20.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.704+0000 7f8af0d3f700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06e5d0 0x7f8aec0c83d0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800c4b0 tx=0x7f8ae800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:20.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.705+0000 7f8ae1ffb700 1 -- 172.21.15.158:0/2502848295 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae80052d0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.705+0000 7f8ae1ffb700 1 -- 172.21.15.158:0/2502848295 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae8005430 con 0x7f8aec06e5d0 2023-10-16T10:00:20.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.705+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec0b19c0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.705+0000 7f8ae1ffb700 1 -- 172.21.15.158:0/2502848295 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae80038c0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.705+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec1f2bb0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.706+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec0b1ee0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.706+0000 7f8ae1ffb700 1 -- 172.21.15.158:0/2502848295 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 48) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f8ae8003a20 con 0x7f8aec06e5d0 2023-10-16T10:00:20.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.706+0000 7f8ae1ffb700 1 -- 172.21.15.158:0/2502848295 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 7141+0+0 (secure 0 0 0) 0x7f8ae8088990 con 0x7f8aec06e5d0 2023-10-16T10:00:20.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.711+0000 7f8ae1ffb700 1 -- 172.21.15.158:0/2502848295 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8aec0b1ee0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.903+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8aec0b1ee0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.904+0000 7f8ae1ffb700 1 -- 172.21.15.158:0/2502848295 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+18446 (secure 0 0 0) 0x7f8aec0b1ee0 con 0x7f8aec06e5d0 2023-10-16T10:00:20.906 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:20.906 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":100,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:20.313507+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T09:59:57.438311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T09:59:57.966439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1634845683":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/1729189331":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/331886015":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/1776913260":"2023-10-17T10:00:20.313472+0000","172.21.15.158:6833/334523996":"2023-10-17T10:00:20.313472+0000","172.21.15.158:6832/334523996":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/245591896":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/10645813":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/1178199719":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/873662109":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6833/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6832/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/1880882485":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/693060665":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/2133642547":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/4289198127":"2023-10-17T10:00:12.821288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:20.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06e5d0 msgr2=0x7f8aec0c83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:20.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06e5d0 0x7f8aec0c83d0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800c4b0 tx=0x7f8ae800c7c0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 shutdown_connections 2023-10-16T10:00:20.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec0729b0 0x7f8aec0c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec06ab40 0x7f8aec1f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 --2- 172.21.15.158:0/2502848295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec06e5d0 0x7f8aec0c83d0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:20.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 >> 172.21.15.158:0/2502848295 conn(0x7f8aec0d02b0 msgr2=0x7f8aec0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:20.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 shutdown_connections 2023-10-16T10:00:20.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:20.907+0000 7f8af37a4700 1 -- 172.21.15.158:0/2502848295 wait complete. 2023-10-16T10:00:20.920 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T10:00:21.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.042+0000 7f8b1a823700 1 Processor -- start 2023-10-16T10:00:21.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.042+0000 7f8b1a823700 1 -- start start 2023-10-16T10:00:21.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.042+0000 7f8b1a823700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b1415d950 0x7f8b1415dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.042+0000 7f8b1a823700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b140ce7b0 0x7f8b14071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.042+0000 7f8b1a823700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b140ce320 con 0x7f8b14072120 2023-10-16T10:00:21.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.042+0000 7f8b1a823700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140725c0 con 0x7f8b1415d950 2023-10-16T10:00:21.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.042+0000 7f8b1a823700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1415e270 con 0x7f8b140ce7b0 2023-10-16T10:00:21.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b1415d950 0x7f8b1415dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b1415d950 0x7f8b1415dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54716/0 (socket says 172.21.15.158:54716) 2023-10-16T10:00:21.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 -- 172.21.15.158:0/1334248975 learned_addr learned my addr 172.21.15.158:0/1334248975 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:21.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b137fe700 1 --2- 172.21.15.158:0/1334248975 >> v2:172.21.15.172:3300/0 conn(0x7f8b140ce7b0 0x7f8b14071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 -- 172.21.15.158:0/1334248975 >> v2:172.21.15.172:3300/0 conn(0x7f8b140ce7b0 msgr2=0x7f8b14071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:21.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 --2- 172.21.15.158:0/1334248975 >> v2:172.21.15.172:3300/0 conn(0x7f8b140ce7b0 0x7f8b14071be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 -- 172.21.15.158:0/1334248975 >> v1:172.21.15.172:6789/0 conn(0x7f8b14072120 legacy=0x7f8b1406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:21.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 -- 172.21.15.158:0/1334248975 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b140cd9f0 con 0x7f8b1415d950 2023-10-16T10:00:21.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b137fe700 1 --2- 172.21.15.158:0/1334248975 >> v2:172.21.15.172:3300/0 conn(0x7f8b140ce7b0 0x7f8b14071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:21.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b13fff700 1 --2- 172.21.15.158:0/1334248975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b1415dd30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f8b1406cce0 tx=0x7f8b04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c97201499484b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:21.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b12ffd700 1 -- 172.21.15.158:0/1334248975 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04004230 con 0x7f8b1415d950 2023-10-16T10:00:21.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b12ffd700 1 -- 172.21.15.158:0/1334248975 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b04004390 con 0x7f8b1415d950 2023-10-16T10:00:21.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.043+0000 7f8b12ffd700 1 -- 172.21.15.158:0/1334248975 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b040036a0 con 0x7f8b1415d950 2023-10-16T10:00:21.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 -- 172.21.15.158:0/1334248975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 msgr2=0x7f8b1415dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 --2- 172.21.15.158:0/1334248975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b1415dd30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f8b1406cce0 tx=0x7f8b04009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 -- 172.21.15.158:0/1334248975 shutdown_connections 2023-10-16T10:00:21.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 --2- 172.21.15.158:0/1334248975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b1415dd30 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 --2- 172.21.15.158:0/1334248975 >> v2:172.21.15.172:3300/0 conn(0x7f8b140ce7b0 0x7f8b14071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 -- 172.21.15.158:0/1334248975 >> 172.21.15.158:0/1334248975 conn(0x7f8b140cc700 msgr2=0x7f8b140ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:21.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 -- 172.21.15.158:0/1334248975 shutdown_connections 2023-10-16T10:00:21.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.044+0000 7f8b1a823700 1 -- 172.21.15.158:0/1334248975 wait complete. 2023-10-16T10:00:21.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.045+0000 7f8b1a823700 1 Processor -- start 2023-10-16T10:00:21.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.045+0000 7f8b1a823700 1 -- start start 2023-10-16T10:00:21.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b1a823700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b14072120 0x7f8b141498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b1a823700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b140ce7b0 0x7f8b14149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b1a823700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b140c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b1a823700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140c7080 con 0x7f8b1415d950 2023-10-16T10:00:21.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b1a823700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140c5a60 con 0x7f8b14072120 2023-10-16T10:00:21.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b1a823700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140c5cd0 con 0x7f8b140ce7b0 2023-10-16T10:00:21.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b140c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b140c95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54726/0 (socket says 172.21.15.158:54726) 2023-10-16T10:00:21.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 -- 172.21.15.158:0/3909276324 learned_addr learned my addr 172.21.15.158:0/3909276324 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:21.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b137fe700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b140ce7b0 0x7f8b14149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b13fff700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b14072120 0x7f8b141498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 -- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b140ce7b0 msgr2=0x7f8b14149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b140ce7b0 0x7f8b14149240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 -- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b14072120 msgr2=0x7f8b141498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b14072120 0x7f8b141498d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.046+0000 7f8b18dc0700 1 -- 172.21.15.158:0/3909276324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b04009550 con 0x7f8b1415d950 2023-10-16T10:00:21.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.047+0000 7f8b18dc0700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b140c95d0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00cc10 tx=0x7f8b0c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:21.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.047+0000 7f8b10ff9700 1 -- 172.21.15.158:0/3909276324 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c00aa60 con 0x7f8b1415d950 2023-10-16T10:00:21.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.047+0000 7f8b10ff9700 1 -- 172.21.15.158:0/3909276324 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c00abc0 con 0x7f8b1415d950 2023-10-16T10:00:21.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.047+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b140c5fb0 con 0x7f8b1415d950 2023-10-16T10:00:21.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.047+0000 7f8b10ff9700 1 -- 172.21.15.158:0/3909276324 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c012420 con 0x7f8b1415d950 2023-10-16T10:00:21.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.047+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b140c6570 con 0x7f8b1415d950 2023-10-16T10:00:21.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.048+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b141e4640 con 0x7f8b1415d950 2023-10-16T10:00:21.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.048+0000 7f8b10ff9700 1 -- 172.21.15.158:0/3909276324 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 48) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f8b0c012580 con 0x7f8b1415d950 2023-10-16T10:00:21.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.049+0000 7f8b10ff9700 1 -- 172.21.15.158:0/3909276324 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 7141+0+0 (secure 0 0 0) 0x7f8b0c089890 con 0x7f8b1415d950 2023-10-16T10:00:21.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.053+0000 7f8b10ff9700 1 -- 172.21.15.158:0/3909276324 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b141e4640 con 0x7f8b1415d950 2023-10-16T10:00:21.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.289+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8b141e4640 con 0x7f8b1415d950 2023-10-16T10:00:21.291 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T10:00:21.289+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T10:00:21.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.291+0000 7f8b10ff9700 1 -- 172.21.15.158:0/3909276324 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8b141e4640 con 0x7f8b1415d950 2023-10-16T10:00:21.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.294+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 msgr2=0x7f8b140c95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.294+0000 7f8b1a823700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b140c95d0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00cc10 tx=0x7f8b0c00cf20 comp rx=0 tx=0).stop 2023-10-16T10:00:21.296 INFO:tasks.ceph.mds.a.smithi158.stderr: -3783> 2023-10-16T10:00:01.589+0000 7fe93942c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:00:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.298+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 shutdown_connections 2023-10-16T10:00:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.298+0000 7f8b1a823700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b1415d950 0x7f8b140c95d0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.298+0000 7f8b1a823700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b14072120 0x7f8b141498d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.298+0000 7f8b1a823700 1 --2- 172.21.15.158:0/3909276324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b140ce7b0 0x7f8b14149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.298+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 >> 172.21.15.158:0/3909276324 conn(0x7f8b140cc700 msgr2=0x7f8b14159360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:21.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.298+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 shutdown_connections 2023-10-16T10:00:21.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.298+0000 7f8b1a823700 1 -- 172.21.15.158:0/3909276324 wait complete. 2023-10-16T10:00:21.300 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T10:00:21.312 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T10:00:21.331 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T10:00:21.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.438+0000 7f868d510700 1 Processor -- start 2023-10-16T10:00:21.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.438+0000 7f868d510700 1 -- start start 2023-10-16T10:00:21.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.439+0000 7f868d510700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f868815deb0 0x7f86880ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.439+0000 7f868d510700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86880ccc00 0x7f86880cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.439+0000 7f868d510700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86880d0ae0 con 0x7f868815dad0 2023-10-16T10:00:21.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.439+0000 7f868d510700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86880d12d0 con 0x7f868815deb0 2023-10-16T10:00:21.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.439+0000 7f868d510700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86880cbdf0 con 0x7f86880ccc00 2023-10-16T10:00:21.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86877fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86880ccc00 0x7f86880cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f868815deb0 0x7f86880ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f868815deb0 0x7f86880ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:55942/0 (socket says 172.21.15.158:55942) 2023-10-16T10:00:21.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f8686ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f868815dad0 0x7f86880cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:58840/0 (socket says 172.21.15.158:58840) 2023-10-16T10:00:21.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86877fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86880ccc00 0x7f86880cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:37366/0 (socket says 172.21.15.158:37366) 2023-10-16T10:00:21.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 -- 172.21.15.158:0/4153859663 learned_addr learned my addr 172.21.15.158:0/4153859663 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:21.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 -- 172.21.15.158:0/4153859663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86880ccc00 msgr2=0x7f86880cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:21.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 --2- 172.21.15.158:0/4153859663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86880ccc00 0x7f86880cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 -- 172.21.15.158:0/4153859663 >> v1:172.21.15.172:6789/0 conn(0x7f868815dad0 legacy=0x7f86880cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 -- 172.21.15.158:0/4153859663 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86880cc280 con 0x7f868815deb0 2023-10-16T10:00:21.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f86867fc700 1 --2- 172.21.15.158:0/4153859663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 0x7f86880ce6b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f86780098f0 tx=0x7f8678009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f99921ab02820e97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:21.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f8685ffb700 1 -- 172.21.15.158:0/4153859663 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678011830 con 0x7f868815deb0 2023-10-16T10:00:21.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f8685ffb700 1 -- 172.21.15.158:0/4153859663 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8678011990 con 0x7f868815deb0 2023-10-16T10:00:21.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.440+0000 7f8685ffb700 1 -- 172.21.15.158:0/4153859663 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678011d20 con 0x7f868815deb0 2023-10-16T10:00:21.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 -- 172.21.15.158:0/4153859663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 msgr2=0x7f86880ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 --2- 172.21.15.158:0/4153859663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 0x7f86880ce6b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f86780098f0 tx=0x7f8678009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:21.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 -- 172.21.15.158:0/4153859663 shutdown_connections 2023-10-16T10:00:21.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 --2- 172.21.15.158:0/4153859663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86880ccc00 0x7f86880cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 --2- 172.21.15.158:0/4153859663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 0x7f86880ce6b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 -- 172.21.15.158:0/4153859663 >> 172.21.15.158:0/4153859663 conn(0x7f8688099a10 msgr2=0x7f86880d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:21.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 -- 172.21.15.158:0/4153859663 shutdown_connections 2023-10-16T10:00:21.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.441+0000 7f868d510700 1 -- 172.21.15.158:0/4153859663 wait complete. 2023-10-16T10:00:21.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.442+0000 7f868d510700 1 Processor -- start 2023-10-16T10:00:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.442+0000 7f868d510700 1 -- start start 2023-10-16T10:00:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f868d510700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86880ccc00 0x7f86880b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f868d510700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f868815dad0 0x7f86881ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f868d510700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 0x7f86881eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f868d510700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880c74a0 con 0x7f86880ccc00 2023-10-16T10:00:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f868d510700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880b2490 con 0x7f868815deb0 2023-10-16T10:00:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f868d510700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86880b2770 con 0x7f868815dad0 2023-10-16T10:00:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86880ccc00 0x7f86880b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86880ccc00 0x7f86880b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54766/0 (socket says 172.21.15.158:54766) 2023-10-16T10:00:21.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 -- 172.21.15.158:0/2081393916 learned_addr learned my addr 172.21.15.158:0/2081393916 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:21.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f86867fc700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f868815dad0 0x7f86881ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f86877fe700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 0x7f86881eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 -- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f868815dad0 msgr2=0x7f86881ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f868815dad0 0x7f86881ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 -- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 msgr2=0x7f86881eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 0x7f86881eeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.443+0000 7f8686ffd700 1 -- 172.21.15.158:0/2081393916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8678009550 con 0x7f86880ccc00 2023-10-16T10:00:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.444+0000 7f8686ffd700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86880ccc00 0x7f86880b5860 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f867c00c520 tx=0x7f867c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.444+0000 7f866f7fe700 1 -- 172.21.15.158:0/2081393916 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c005cb0 con 0x7f86880ccc00 2023-10-16T10:00:21.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.444+0000 7f866f7fe700 1 -- 172.21.15.158:0/2081393916 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c005e10 con 0x7f86880ccc00 2023-10-16T10:00:21.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.444+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86880b2a50 con 0x7f86880ccc00 2023-10-16T10:00:21.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.444+0000 7f866f7fe700 1 -- 172.21.15.158:0/2081393916 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c0072d0 con 0x7f86880ccc00 2023-10-16T10:00:21.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.444+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86880b5da0 con 0x7f86880ccc00 2023-10-16T10:00:21.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.445+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86880681b0 con 0x7f86880ccc00 2023-10-16T10:00:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.445+0000 7f866f7fe700 1 -- 172.21.15.158:0/2081393916 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 49) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f867c007430 con 0x7f86880ccc00 2023-10-16T10:00:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.446+0000 7f866f7fe700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f867406ad50 0x7f867406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.446+0000 7f866f7fe700 1 -- 172.21.15.158:0/2081393916 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 7227+0+0 (secure 0 0 0) 0x7f867c088a80 con 0x7f86880ccc00 2023-10-16T10:00:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.446+0000 7f86867fc700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f867406ad50 0x7f867406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.446+0000 7f86867fc700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f867406ad50 0x7f867406d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8678009c40 tx=0x7f8678008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:21.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.450+0000 7f866f7fe700 1 -- 172.21.15.158:0/2081393916 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f867c053930 con 0x7f86880ccc00 2023-10-16T10:00:21.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:21.690+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f868815b870 con 0x7f86880ccc00 2023-10-16T10:00:22.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.327+0000 7f866f7fe700 1 -- 172.21.15.158:0/2081393916 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f867c056f50 con 0x7f86880ccc00 2023-10-16T10:00:22.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f867406ad50 msgr2=0x7f867406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:22.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f867406ad50 0x7f867406d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8678009c40 tx=0x7f8678008030 comp rx=0 tx=0).stop 2023-10-16T10:00:22.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86880ccc00 msgr2=0x7f86880b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:22.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86880ccc00 0x7f86880b5860 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f867c00c520 tx=0x7f867c00c830 comp rx=0 tx=0).stop 2023-10-16T10:00:22.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 shutdown_connections 2023-10-16T10:00:22.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f867406ad50 0x7f867406d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f868815deb0 0x7f86881eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86880ccc00 0x7f86880b5860 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 --2- 172.21.15.158:0/2081393916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f868815dad0 0x7f86881ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 >> 172.21.15.158:0/2081393916 conn(0x7f8688099a10 msgr2=0x7f868815a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:22.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 shutdown_connections 2023-10-16T10:00:22.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.330+0000 7f868d510700 1 -- 172.21.15.158:0/2081393916 wait complete. 2023-10-16T10:00:22.346 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T10:00:22.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.510+0000 7f757e776700 1 Processor -- start 2023-10-16T10:00:22.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.510+0000 7f757e776700 1 -- start start 2023-10-16T10:00:22.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.511+0000 7f757e776700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757815dbd0 0x7f757806df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.511+0000 7f757e776700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757815e510 con 0x7f757815dfc0 2023-10-16T10:00:22.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.511+0000 7f757e776700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757815e7f0 con 0x7f757806afe0 2023-10-16T10:00:22.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.511+0000 7f757e776700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75780ccbb0 con 0x7f757815dbd0 2023-10-16T10:00:22.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.511+0000 7f75777fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f757815dfc0 0x7f757806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54968/0 (socket says 172.21.15.158:54968) 2023-10-16T10:00:22.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.511+0000 7f75777fe700 1 -- 172.21.15.158:0/3325966194 learned_addr learned my addr 172.21.15.158:0/3325966194 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:22.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7577fff700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757815dbd0 0x7f757806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182015683 0 0) 0x7f757815e510 con 0x7f757815dfc0 2023-10-16T10:00:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75600035b0 con 0x7f757815dfc0 2023-10-16T10:00:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2881671666 0 0) 0x7f75600035b0 con 0x7f757815dfc0 2023-10-16T10:00:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757815dbd0 msgr2=0x7f757806df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757815dbd0 0x7f757806df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 >> v1:172.21.15.172:6789/0 conn(0x7f757806afe0 legacy=0x7f75780ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757815ead0 con 0x7f757815dfc0 2023-10-16T10:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7568001670 con 0x7f757815dfc0 2023-10-16T10:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 >> v1:172.21.15.158:6789/0 conn(0x7f757815dfc0 legacy=0x7f757806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7560003a30 0x7f7560003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7560004850 0x7f7560004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75600051d0 0x7f756000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75780ccbb0 con 0x7f75600051d0 2023-10-16T10:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756000bf50 con 0x7f7560003a30 2023-10-16T10:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756000c0e0 con 0x7f7560004850 2023-10-16T10:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f757cd13700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75600051d0 0x7f756000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f7577fff700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7560003a30 0x7f7560003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f75777fe700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7560004850 0x7f7560004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f757cd13700 1 -- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7560004850 msgr2=0x7f7560004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f757cd13700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7560004850 0x7f7560004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f757cd13700 1 -- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7560003a30 msgr2=0x7f7560003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f757cd13700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7560003a30 0x7f7560003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.512+0000 7f757cd13700 1 -- 172.21.15.158:0/3325966194 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75600035b0 con 0x7f75600051d0 2023-10-16T10:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757cd13700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75600051d0 0x7f756000ba10 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f75780ccaf0 tx=0x7f7560005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce17c95dbef76b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570001db0 con 0x7f75600051d0 2023-10-16T10:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757000cc70 con 0x7f75600051d0 2023-10-16T10:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f7576ffd700 1 -- 172.21.15.158:0/3325966194 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570003a00 con 0x7f75600051d0 2023-10-16T10:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757e776700 1 -- 172.21.15.158:0/3325966194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75600051d0 msgr2=0x7f756000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757e776700 1 --2- 172.21.15.158:0/3325966194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75600051d0 0x7f756000ba10 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f75780ccaf0 tx=0x7f7560005880 comp rx=0 tx=0).stop 2023-10-16T10:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757cd13700 1 -- 172.21.15.158:0/3325966194 reap_dead start 2023-10-16T10:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757e776700 1 -- 172.21.15.158:0/3325966194 shutdown_connections 2023-10-16T10:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757e776700 1 -- 172.21.15.158:0/3325966194 >> 172.21.15.158:0/3325966194 conn(0x7f75780b1f00 msgr2=0x7f75780b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757e776700 1 -- 172.21.15.158:0/3325966194 shutdown_connections 2023-10-16T10:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.513+0000 7f757e776700 1 -- 172.21.15.158:0/3325966194 wait complete. 2023-10-16T10:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 Processor -- start 2023-10-16T10:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 -- start start 2023-10-16T10:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75600051d0 0x7f757814d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75781f67c0 0x7f75780b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75781f6c40 0x7f75780ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75781f7740 con 0x7f75781f6c40 2023-10-16T10:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75780acda0 con 0x7f75600051d0 2023-10-16T10:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.515+0000 7f757e776700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75781f7180 con 0x7f75781f67c0 2023-10-16T10:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75781f6c40 0x7f75780ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75781f6c40 0x7f75780ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54808/0 (socket says 172.21.15.158:54808) 2023-10-16T10:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 -- 172.21.15.158:0/2776865055 learned_addr learned my addr 172.21.15.158:0/2776865055 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f7577fff700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75600051d0 0x7f757814d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f75777fe700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75781f67c0 0x7f75780b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 -- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75781f67c0 msgr2=0x7f75780b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:22.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75781f67c0 0x7f75780b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 -- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75600051d0 msgr2=0x7f757814d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:22.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75600051d0 0x7f757814d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:22.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 -- 172.21.15.158:0/2776865055 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75781f7460 con 0x7f75781f6c40 2023-10-16T10:00:22.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f75777fe700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75781f67c0 0x7f75780b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:22.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f7577fff700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75600051d0 0x7f757814d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:22.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757cd13700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75781f6c40 0x7f75780ad230 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7570005040 tx=0x7f75700014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:22.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570003c10 con 0x7f75781f6c40 2023-10-16T10:00:22.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7570003db0 con 0x7f75781f6c40 2023-10-16T10:00:22.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75780aca30 con 0x7f75781f6c40 2023-10-16T10:00:22.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570005310 con 0x7f75781f6c40 2023-10-16T10:00:22.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.516+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75780c9370 con 0x7f75781f6c40 2023-10-16T10:00:22.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.517+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75780ac890 con 0x7f75781f6c40 2023-10-16T10:00:22.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.518+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 49) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7570019070 con 0x7f75781f6c40 2023-10-16T10:00:22.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.518+0000 7f7574ff9700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f756406ad50 0x7f756406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:22.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.518+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 7227+0+0 (secure 0 0 0) 0x7f757008a420 con 0x7f75781f6c40 2023-10-16T10:00:22.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.518+0000 7f7577fff700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f756406ad50 0x7f756406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:22.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.518+0000 7f7577fff700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f756406ad50 0x7f756406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f756c009c60 tx=0x7f756c008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:22.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.522+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75780ac890 con 0x7f75781f6c40 2023-10-16T10:00:22.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:22.718+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f75780ac890 con 0x7f75781f6c40 2023-10-16T10:00:23.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.327+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 50) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f75700555f0 con 0x7f75781f6c40 2023-10-16T10:00:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.336+0000 7f7574ff9700 1 -- 172.21.15.158:0/2776865055 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) v1 ==== 181+0+0 (secure 0 0 0) 0x7f757005c400 con 0x7f75781f6c40 2023-10-16T10:00:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f756406ad50 msgr2=0x7f756406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f756406ad50 0x7f756406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f756c009c60 tx=0x7f756c008040 comp rx=0 tx=0).stop 2023-10-16T10:00:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75781f6c40 msgr2=0x7f75780ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75781f6c40 0x7f75780ad230 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7570005040 tx=0x7f75700014a0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 shutdown_connections 2023-10-16T10:00:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f756406ad50 0x7f756406d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75781f6c40 0x7f75780ad230 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75600051d0 0x7f757814d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 --2- 172.21.15.158:0/2776865055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75781f67c0 0x7f75780b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 >> 172.21.15.158:0/2776865055 conn(0x7f75780b1f00 msgr2=0x7f75780cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:23.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 shutdown_connections 2023-10-16T10:00:23.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.339+0000 7f757e776700 1 -- 172.21.15.158:0/2776865055 wait complete. 2023-10-16T10:00:23.344 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T10:00:23.354 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T10:00:23.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.481+0000 7feacbcef700 1 Processor -- start 2023-10-16T10:00:23.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.481+0000 7feacbcef700 1 -- start start 2023-10-16T10:00:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feacbcef700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feac40cf520 0x7feac40cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feacbcef700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac415dc20 con 0x7feac415d840 2023-10-16T10:00:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feacbcef700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac40cf990 con 0x7feac40cf070 2023-10-16T10:00:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feacbcef700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac406c0e0 con 0x7feac40cf520 2023-10-16T10:00:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feaca28c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feac40cf520 0x7feac40cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feaca28c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feac40cf520 0x7feac40cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54816/0 (socket says 172.21.15.158:54816) 2023-10-16T10:00:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feaca28c700 1 -- 172.21.15.158:0/1507623579 learned_addr learned my addr 172.21.15.158:0/1507623579 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.482+0000 7feac9a8b700 1 --1- 172.21.15.158:0/1507623579 >> v1:172.21.15.158:6789/0 conn(0x7feac415d840 0x7feac4161d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:54988/0 (socket says 172.21.15.158:54988) 2023-10-16T10:00:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feaca28c700 1 -- 172.21.15.158:0/1507623579 >> v1:172.21.15.172:6789/0 conn(0x7feac40cf070 legacy=0x7feac40cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feaca28c700 1 -- 172.21.15.158:0/1507623579 >> v1:172.21.15.158:6789/0 conn(0x7feac415d840 legacy=0x7feac4161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feaca28c700 1 -- 172.21.15.158:0/1507623579 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac406c3c0 con 0x7feac40cf520 2023-10-16T10:00:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feac8a89700 1 -- 172.21.15.158:0/1507623579 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439081816 0 0) 0x7feab4001110 con 0x7feac415d840 2023-10-16T10:00:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feaca28c700 1 --2- 172.21.15.158:0/1507623579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40cc4e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7feabc009e50 tx=0x7feabc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f63374ed80755e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feac8a89700 1 -- 172.21.15.158:0/1507623579 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc00e070 con 0x7feac40cf520 2023-10-16T10:00:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feac8a89700 1 -- 172.21.15.158:0/1507623579 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc00b9c0 con 0x7feac40cf520 2023-10-16T10:00:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feac8a89700 1 -- 172.21.15.158:0/1507623579 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc00bd70 con 0x7feac40cf520 2023-10-16T10:00:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feacbcef700 1 -- 172.21.15.158:0/1507623579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 msgr2=0x7feac40cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feacbcef700 1 --2- 172.21.15.158:0/1507623579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40cc4e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7feabc009e50 tx=0x7feabc00b330 comp rx=0 tx=0).stop 2023-10-16T10:00:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feacbcef700 1 -- 172.21.15.158:0/1507623579 shutdown_connections 2023-10-16T10:00:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feacbcef700 1 --2- 172.21.15.158:0/1507623579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40cc4e0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.483+0000 7feacbcef700 1 -- 172.21.15.158:0/1507623579 >> 172.21.15.158:0/1507623579 conn(0x7feac406a8d0 msgr2=0x7feac406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.484+0000 7feacbcef700 1 -- 172.21.15.158:0/1507623579 shutdown_connections 2023-10-16T10:00:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.484+0000 7feacbcef700 1 -- 172.21.15.158:0/1507623579 wait complete. 2023-10-16T10:00:23.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.485+0000 7feacbcef700 1 Processor -- start 2023-10-16T10:00:23.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.485+0000 7feacbcef700 1 -- start start 2023-10-16T10:00:23.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feacbcef700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feac40cf070 0x7feac40b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:23.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feacbcef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:23.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feacbcef700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feac415d840 0x7feac40cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feacbcef700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40c5040 con 0x7feac40cf520 2023-10-16T10:00:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feacbcef700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40b6310 con 0x7feac415d840 2023-10-16T10:00:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feacbcef700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac40b65f0 con 0x7feac40cf070 2023-10-16T10:00:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54820/0 (socket says 172.21.15.158:54820) 2023-10-16T10:00:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 -- 172.21.15.158:0/4063700840 learned_addr learned my addr 172.21.15.158:0/4063700840 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 -- 172.21.15.158:0/4063700840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feac40cf070 msgr2=0x7feac40b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feac40cf070 0x7feac40b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 -- 172.21.15.158:0/4063700840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feac415d840 msgr2=0x7feac40cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feac415d840 0x7feac40cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.486+0000 7feac928a700 1 -- 172.21.15.158:0/4063700840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc009b30 con 0x7feac40cf520 2023-10-16T10:00:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.487+0000 7feac928a700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40ca740 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7feac000c4e0 tx=0x7feac000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.487+0000 7feaba7fc700 1 -- 172.21.15.158:0/4063700840 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00052d0 con 0x7feac40cf520 2023-10-16T10:00:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.487+0000 7feaba7fc700 1 -- 172.21.15.158:0/4063700840 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac0005430 con 0x7feac40cf520 2023-10-16T10:00:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.487+0000 7feaba7fc700 1 -- 172.21.15.158:0/4063700840 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00038c0 con 0x7feac40cf520 2023-10-16T10:00:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.487+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac40b68d0 con 0x7feac40cf520 2023-10-16T10:00:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.487+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac40b6e90 con 0x7feac40cf520 2023-10-16T10:00:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.488+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac41e71a0 con 0x7feac40cf520 2023-10-16T10:00:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.488+0000 7feaba7fc700 1 -- 172.21.15.158:0/4063700840 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 50) v1 ==== 89275+0+0 (secure 0 0 0) 0x7feac0003a20 con 0x7feac40cf520 2023-10-16T10:00:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.488+0000 7feaba7fc700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7feaa806af90 0x7feaa806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.489+0000 7feaba7fc700 1 -- 172.21.15.158:0/4063700840 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6789+0+0 (secure 0 0 0) 0x7feac0088ff0 con 0x7feac40cf520 2023-10-16T10:00:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.489+0000 7feac9a8b700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7feaa806af90 0x7feaa806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.489+0000 7feac9a8b700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7feaa806af90 0x7feaa806d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feab4000c00 tx=0x7feab4007a50 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:23.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.493+0000 7feaba7fc700 1 -- 172.21.15.158:0/4063700840 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feac00bbfe0 con 0x7feac40cf520 2023-10-16T10:00:23.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:23.694+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feac40681b0 con 0x7feac40cf520 2023-10-16T10:00:24.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.569+0000 7feaba7fc700 1 -- 172.21.15.158:0/4063700840 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) v1 ==== 169+0+0 (secure 0 0 0) 0x7feac00170d0 con 0x7feac40cf520 2023-10-16T10:00:24.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7feaa806af90 msgr2=0x7feaa806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7feaa806af90 0x7feaa806d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feab4000c00 tx=0x7feab4007a50 comp rx=0 tx=0).stop 2023-10-16T10:00:24.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 msgr2=0x7feac40ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40ca740 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7feac000c4e0 tx=0x7feac000c8a0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 shutdown_connections 2023-10-16T10:00:24.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7feaa806af90 0x7feaa806d450 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feac415d840 0x7feac40cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feac40cf070 0x7feac40b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 --2- 172.21.15.158:0/4063700840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feac40cf520 0x7feac40ca740 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 >> 172.21.15.158:0/4063700840 conn(0x7feac406a8d0 msgr2=0x7feac40cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:24.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 shutdown_connections 2023-10-16T10:00:24.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.572+0000 7feacbcef700 1 -- 172.21.15.158:0/4063700840 wait complete. 2023-10-16T10:00:24.576 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T10:00:24.588 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T10:00:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.716+0000 7fa8ffb0c700 1 Processor -- start 2023-10-16T10:00:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.716+0000 7fa8ffb0c700 1 -- start start 2023-10-16T10:00:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8ffb0c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f815c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8ffb0c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8f80cbef0 0x7fa8f80cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8ffb0c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f8159660 con 0x7fa8f8159280 2023-10-16T10:00:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8ffb0c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f80cb710 con 0x7fa8f80cbef0 2023-10-16T10:00:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8ffb0c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f806b120 con 0x7fa8f80d0e10 2023-10-16T10:00:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fe0a9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8f80cbef0 0x7fa8f80cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fd8a8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa8f8159280 0x7fa8f806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55002/0 (socket says 172.21.15.158:55002) 2023-10-16T10:00:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fd8a8700 1 -- 172.21.15.158:0/1499787191 learned_addr learned my addr 172.21.15.158:0/1499787191 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fe0a9700 1 -- 172.21.15.158:0/1499787191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 msgr2=0x7fa8f815c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fd0a7700 1 --2- 172.21.15.158:0/1499787191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fe0a9700 1 --2- 172.21.15.158:0/1499787191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f815c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fe0a9700 1 -- 172.21.15.158:0/1499787191 >> v1:172.21.15.158:6789/0 conn(0x7fa8f8159280 legacy=0x7fa8f806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fe0a9700 1 -- 172.21.15.158:0/1499787191 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f806b400 con 0x7fa8f80cbef0 2023-10-16T10:00:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fc8a6700 1 -- 172.21.15.158:0/1499787191 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385148633 0 0) 0x7fa8f8159660 con 0x7fa8f8159280 2023-10-16T10:00:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fd0a7700 1 --2- 172.21.15.158:0/1499787191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f815c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fe0a9700 1 --2- 172.21.15.158:0/1499787191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80cc380 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0009d90 tx=0x7fa8f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac66e2c47bcad855 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fc8a6700 1 -- 172.21.15.158:0/1499787191 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f000e070 con 0x7fa8f80cbef0 2023-10-16T10:00:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fc8a6700 1 -- 172.21.15.158:0/1499787191 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f000bbd0 con 0x7fa8f80cbef0 2023-10-16T10:00:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.717+0000 7fa8fc8a6700 1 -- 172.21.15.158:0/1499787191 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0012430 con 0x7fa8f80cbef0 2023-10-16T10:00:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/1499787191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 msgr2=0x7fa8f80cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/1499787191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80cc380 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0009d90 tx=0x7fa8f000b330 comp rx=0 tx=0).stop 2023-10-16T10:00:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/1499787191 shutdown_connections 2023-10-16T10:00:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/1499787191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80cc380 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/1499787191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f815c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/1499787191 >> 172.21.15.158:0/1499787191 conn(0x7fa8f8099a10 msgr2=0x7fa8f80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/1499787191 shutdown_connections 2023-10-16T10:00:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.718+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/1499787191 wait complete. 2023-10-16T10:00:24.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.719+0000 7fa8ffb0c700 1 Processor -- start 2023-10-16T10:00:24.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.719+0000 7fa8ffb0c700 1 -- start start 2023-10-16T10:00:24.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8ffb0c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:24.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8ffb0c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:24.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8ffb0c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8f8159280 0x7fa8f80b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:24.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8ffb0c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80c8670 con 0x7fa8f80cbef0 2023-10-16T10:00:24.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8ffb0c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80c7a60 con 0x7fa8f8159280 2023-10-16T10:00:24.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8ffb0c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80c7d40 con 0x7fa8f80d0e10 2023-10-16T10:00:24.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:24.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54832/0 (socket says 172.21.15.158:54832) 2023-10-16T10:00:24.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 -- 172.21.15.158:0/716295411 learned_addr learned my addr 172.21.15.158:0/716295411 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:24.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 -- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 msgr2=0x7fa8f80b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd0a7700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:24.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fe0a9700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8f8159280 0x7fa8f80b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:24.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f80b5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 -- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8f8159280 msgr2=0x7fa8f80b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8f8159280 0x7fa8f80b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 -- 172.21.15.158:0/716295411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f0009a70 con 0x7fa8f80cbef0 2023-10-16T10:00:24.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd0a7700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f80b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:24.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fe0a9700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8f8159280 0x7fa8f80b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:24.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.720+0000 7fa8fd8a8700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80ca120 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400c550 tx=0x7fa8f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:24.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.721+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4004c80 con 0x7fa8f80cbef0 2023-10-16T10:00:24.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.721+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f4004e00 con 0x7fa8f80cbef0 2023-10-16T10:00:24.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.721+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f80c8020 con 0x7fa8f80cbef0 2023-10-16T10:00:24.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.721+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4007310 con 0x7fa8f80cbef0 2023-10-16T10:00:24.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.721+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f80ca660 con 0x7fa8f80cbef0 2023-10-16T10:00:24.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.722+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f80681b0 con 0x7fa8f80cbef0 2023-10-16T10:00:24.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.722+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 50) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa8f4007470 con 0x7fa8f80cbef0 2023-10-16T10:00:24.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.722+0000 7fa8ee7fc700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fa8dc06af90 0x7fa8dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:24.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.723+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fa8f4089710 con 0x7fa8f80cbef0 2023-10-16T10:00:24.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.723+0000 7fa8fd0a7700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fa8dc06af90 0x7fa8dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:24.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.723+0000 7fa8fd0a7700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fa8dc06af90 0x7fa8dc06d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8f81f3cd0 tx=0x7fa8e8008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:24.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.727+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8f4054430 con 0x7fa8f80cbef0 2023-10-16T10:00:24.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.858+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa8f4054800 con 0x7fa8f80cbef0 2023-10-16T10:00:24.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.926+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fa8f80cd4f0 con 0x7fa8f80cbef0 2023-10-16T10:00:24.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.930+0000 7fa8ee7fc700 1 -- 172.21.15.158:0/716295411 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa8f4074790 con 0x7fa8f80cbef0 2023-10-16T10:00:24.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fa8dc06af90 msgr2=0x7fa8dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fa8dc06af90 0x7fa8dc06d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8f81f3cd0 tx=0x7fa8e8008040 comp rx=0 tx=0).stop 2023-10-16T10:00:24.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 msgr2=0x7fa8f80ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:24.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80ca120 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400c550 tx=0x7fa8f400c910 comp rx=0 tx=0).stop 2023-10-16T10:00:24.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 shutdown_connections 2023-10-16T10:00:24.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fa8dc06af90 0x7fa8dc06d450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8f8159280 0x7fa8f80b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8f80cbef0 0x7fa8f80ca120 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 --2- 172.21.15.158:0/716295411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8f80d0e10 0x7fa8f80b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:24.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 >> 172.21.15.158:0/716295411 conn(0x7fa8f8099a10 msgr2=0x7fa8f80caa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:24.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 shutdown_connections 2023-10-16T10:00:24.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:24.932+0000 7fa8ffb0c700 1 -- 172.21.15.158:0/716295411 wait complete. 2023-10-16T10:00:24.946 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T10:00:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.075+0000 7fe0d2eb1700 1 Processor -- start 2023-10-16T10:00:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.075+0000 7fe0d2eb1700 1 -- start start 2023-10-16T10:00:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0d2eb1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0d2eb1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe0cc15deb0 0x7fe0cc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0d2eb1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc0d0ae0 con 0x7fe0cc0cea20 2023-10-16T10:00:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0d2eb1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0cc0d12d0 con 0x7fe0cc15deb0 2023-10-16T10:00:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0d2eb1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0cbdf0 con 0x7fe0cc15dad0 2023-10-16T10:00:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0cbfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe0cc15deb0 0x7fe0cc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0cbfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe0cc15deb0 0x7fe0cc0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54842/0 (socket says 172.21.15.158:54842) 2023-10-16T10:00:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0d144e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe0cc0cea20 0x7fe0cc0c3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55016/0 (socket says 172.21.15.158:55016) 2023-10-16T10:00:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0cbfff700 1 -- 172.21.15.158:0/3312197296 learned_addr learned my addr 172.21.15.158:0/3312197296 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.076+0000 7fe0d0c4d700 1 --2- 172.21.15.158:0/3312197296 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cbfff700 1 -- 172.21.15.158:0/3312197296 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 msgr2=0x7fe0cc0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cbfff700 1 --2- 172.21.15.158:0/3312197296 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cbfff700 1 -- 172.21.15.158:0/3312197296 >> v1:172.21.15.158:6789/0 conn(0x7fe0cc0cea20 legacy=0x7fe0cc0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cbfff700 1 -- 172.21.15.158:0/3312197296 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0cc280 con 0x7fe0cc15deb0 2023-10-16T10:00:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cb7fe700 1 -- 172.21.15.158:0/3312197296 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556732220 0 0) 0x7fe0cc0d0ae0 con 0x7fe0cc0cea20 2023-10-16T10:00:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cbfff700 1 --2- 172.21.15.158:0/3312197296 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc15deb0 0x7fe0cc0ce4e0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80098f0 tx=0x7fe0b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322d12b6119b4160 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cb7fe700 1 -- 172.21.15.158:0/3312197296 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b800b8a0 con 0x7fe0cc15deb0 2023-10-16T10:00:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cb7fe700 1 -- 172.21.15.158:0/3312197296 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b800ba00 con 0x7fe0cc15deb0 2023-10-16T10:00:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0cb7fe700 1 -- 172.21.15.158:0/3312197296 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b800bd90 con 0x7fe0cc15deb0 2023-10-16T10:00:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/3312197296 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc15deb0 msgr2=0x7fe0cc0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/3312197296 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc15deb0 0x7fe0cc0ce4e0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80098f0 tx=0x7fe0b8009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/3312197296 shutdown_connections 2023-10-16T10:00:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/3312197296 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/3312197296 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc15deb0 0x7fe0cc0ce4e0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/3312197296 >> 172.21.15.158:0/3312197296 conn(0x7fe0cc099a10 msgr2=0x7fe0cc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/3312197296 shutdown_connections 2023-10-16T10:00:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.077+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/3312197296 wait complete. 2023-10-16T10:00:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.079+0000 7fe0d2eb1700 1 Processor -- start 2023-10-16T10:00:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.079+0000 7fe0d2eb1700 1 -- start start 2023-10-16T10:00:25.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc0cea20 0x7fe0cc0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc1f73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0cc15deb0 0x7fe0cc06ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc1f7900 con 0x7fe0cc0cea20 2023-10-16T10:00:25.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0b2ff0 con 0x7fe0cc15deb0 2023-10-16T10:00:25.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0c7b40 con 0x7fe0cc15dad0 2023-10-16T10:00:25.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc0cea20 0x7fe0cc0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc0cea20 0x7fe0cc0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54846/0 (socket says 172.21.15.158:54846) 2023-10-16T10:00:25.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 -- 172.21.15.158:0/4104505823 learned_addr learned my addr 172.21.15.158:0/4104505823 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:25.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0cbfff700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc1f73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d144e700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0cc15deb0 0x7fe0cc06ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 -- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 msgr2=0x7fe0cc1f73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc1f73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 -- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0cc15deb0 msgr2=0x7fe0cc06ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0cc15deb0 0x7fe0cc06ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 -- 172.21.15.158:0/4104505823 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b8009550 con 0x7fe0cc0cea20 2023-10-16T10:00:25.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d0c4d700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc0cea20 0x7fe0cc0c74a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000e810 tx=0x7fe0c000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0c97fa700 1 -- 172.21.15.158:0/4104505823 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c0013610 con 0x7fe0cc0cea20 2023-10-16T10:00:25.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0c97fa700 1 -- 172.21.15.158:0/4104505823 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c0013770 con 0x7fe0cc0cea20 2023-10-16T10:00:25.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc0c7e20 con 0x7fe0cc0cea20 2023-10-16T10:00:25.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0c97fa700 1 -- 172.21.15.158:0/4104505823 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c0013b00 con 0x7fe0cc0cea20 2023-10-16T10:00:25.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.080+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc0c83e0 con 0x7fe0cc0cea20 2023-10-16T10:00:25.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.081+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc0681b0 con 0x7fe0cc0cea20 2023-10-16T10:00:25.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.082+0000 7fe0c97fa700 1 -- 172.21.15.158:0/4104505823 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe0c0013c60 con 0x7fe0cc0cea20 2023-10-16T10:00:25.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.082+0000 7fe0c97fa700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe0b006b020 0x7fe0b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.082+0000 7fe0cbfff700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe0b006b020 0x7fe0b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.082+0000 7fe0c97fa700 1 -- 172.21.15.158:0/4104505823 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fe0c00896c0 con 0x7fe0cc0cea20 2023-10-16T10:00:25.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.083+0000 7fe0cbfff700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe0b006b020 0x7fe0b006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc15e2b0 tx=0x7fe0b8008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.087+0000 7fe0c97fa700 1 -- 172.21.15.158:0/4104505823 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0c00542c0 con 0x7fe0cc0cea20 2023-10-16T10:00:25.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.278+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe0cc0cb730 con 0x7fe0cc0cea20 2023-10-16T10:00:25.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.281+0000 7fe0c97fa700 1 -- 172.21.15.158:0/4104505823 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe0c00578e0 con 0x7fe0cc0cea20 2023-10-16T10:00:25.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe0b006b020 msgr2=0x7fe0b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe0b006b020 0x7fe0b006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc15e2b0 tx=0x7fe0b8008030 comp rx=0 tx=0).stop 2023-10-16T10:00:25.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc0cea20 msgr2=0x7fe0cc0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc0cea20 0x7fe0cc0c74a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000e810 tx=0x7fe0c000eb20 comp rx=0 tx=0).stop 2023-10-16T10:00:25.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 shutdown_connections 2023-10-16T10:00:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe0b006b020 0x7fe0b006d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0cc15deb0 0x7fe0cc06ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0cc0cea20 0x7fe0cc0c74a0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 --2- 172.21.15.158:0/4104505823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0cc15dad0 0x7fe0cc1f73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 >> 172.21.15.158:0/4104505823 conn(0x7fe0cc099a10 msgr2=0x7fe0cc15c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 shutdown_connections 2023-10-16T10:00:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.283+0000 7fe0d2eb1700 1 -- 172.21.15.158:0/4104505823 wait complete. 2023-10-16T10:00:25.297 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T10:00:25.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.418+0000 7fc26dbad700 1 Processor -- start 2023-10-16T10:00:25.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.419+0000 7fc26dbad700 1 -- start start 2023-10-16T10:00:25.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.419+0000 7fc26dbad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26806d4d0 0x7fc2680cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.419+0000 7fc26dbad700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2681614e0 con 0x7fc268160cb0 2023-10-16T10:00:25.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.419+0000 7fc26dbad700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2681617c0 con 0x7fc2680cf780 2023-10-16T10:00:25.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.419+0000 7fc26dbad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc268161aa0 con 0x7fc26806d4d0 2023-10-16T10:00:25.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.419+0000 7fc2677fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc268160cb0 0x7fc2680cf070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55030/0 (socket says 172.21.15.158:55030) 2023-10-16T10:00:25.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.419+0000 7fc2677fe700 1 -- 172.21.15.158:0/158312107 learned_addr learned my addr 172.21.15.158:0/158312107 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:25.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc267fff700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26806d4d0 0x7fc2680cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160661094 0 0) 0x7fc2681614e0 con 0x7fc268160cb0 2023-10-16T10:00:25.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2500035b0 con 0x7fc268160cb0 2023-10-16T10:00:25.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270796345 0 0) 0x7fc2500035b0 con 0x7fc268160cb0 2023-10-16T10:00:25.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26806d4d0 msgr2=0x7fc2680cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:25.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26806d4d0 0x7fc2680cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 >> v1:172.21.15.172:6789/0 conn(0x7fc2680cf780 legacy=0x7fc26815e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268161d80 con 0x7fc268160cb0 2023-10-16T10:00:25.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc25c001670 con 0x7fc268160cb0 2023-10-16T10:00:25.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 >> v1:172.21.15.158:6789/0 conn(0x7fc268160cb0 legacy=0x7fc2680cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc250003a30 0x7fc250003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc250004390 0x7fc250004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc250005070 0x7fc25000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268161aa0 con 0x7fc250005070 2023-10-16T10:00:25.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25000bf80 con 0x7fc250004390 2023-10-16T10:00:25.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25000c140 con 0x7fc250003a30 2023-10-16T10:00:25.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc267fff700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26806d4d0 0x7fc2680cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:25.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc2677fe700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc250003a30 0x7fc250003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc267fff700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc250005070 0x7fc25000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.420+0000 7fc266ffd700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc250004390 0x7fc250004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc267fff700 1 -- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc250003a30 msgr2=0x7fc250003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:25.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc267fff700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc250003a30 0x7fc250003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc267fff700 1 -- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc250004390 msgr2=0x7fc250004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:25.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc267fff700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc250004390 0x7fc250004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc267fff700 1 -- 172.21.15.158:0/158312107 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2500035b0 con 0x7fc250005070 2023-10-16T10:00:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc267fff700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc250005070 0x7fc25000ba10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fc26806d8b0 tx=0x7fc250005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d07046a3b54053 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260003c20 con 0x7fc250005070 2023-10-16T10:00:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26000f040 con 0x7fc250005070 2023-10-16T10:00:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc2667fc700 1 -- 172.21.15.158:0/158312107 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26000ea30 con 0x7fc250005070 2023-10-16T10:00:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc26dbad700 1 -- 172.21.15.158:0/158312107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc250005070 msgr2=0x7fc25000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc26dbad700 1 --2- 172.21.15.158:0/158312107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc250005070 0x7fc25000ba10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fc26806d8b0 tx=0x7fc250005880 comp rx=0 tx=0).stop 2023-10-16T10:00:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc267fff700 1 -- 172.21.15.158:0/158312107 reap_dead start 2023-10-16T10:00:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc26dbad700 1 -- 172.21.15.158:0/158312107 shutdown_connections 2023-10-16T10:00:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc26dbad700 1 -- 172.21.15.158:0/158312107 >> 172.21.15.158:0/158312107 conn(0x7fc26806a8d0 msgr2=0x7fc26806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.421+0000 7fc26dbad700 1 -- 172.21.15.158:0/158312107 shutdown_connections 2023-10-16T10:00:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.422+0000 7fc26dbad700 1 -- 172.21.15.158:0/158312107 wait complete. 2023-10-16T10:00:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.423+0000 7fc26dbad700 1 Processor -- start 2023-10-16T10:00:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.423+0000 7fc26dbad700 1 -- start start 2023-10-16T10:00:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc26815cac0 0x7fc2680c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2680c5510 0x7fc26815fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc268160050 0x7fc2681f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268160500 con 0x7fc26815cac0 2023-10-16T10:00:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680c5990 con 0x7fc268160050 2023-10-16T10:00:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680c5c70 con 0x7fc2680c5510 2023-10-16T10:00:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc26815cac0 0x7fc2680c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc26815cac0 0x7fc2680c4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54864/0 (socket says 172.21.15.158:54864) 2023-10-16T10:00:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 -- 172.21.15.158:0/2230451819 learned_addr learned my addr 172.21.15.158:0/2230451819 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc267fff700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc268160050 0x7fc2681f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc266ffd700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2680c5510 0x7fc26815fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 -- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2680c5510 msgr2=0x7fc26815fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2680c5510 0x7fc26815fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 -- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc268160050 msgr2=0x7fc2681f6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc268160050 0x7fc2681f6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 -- 172.21.15.158:0/2230451819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2680c5f50 con 0x7fc26815cac0 2023-10-16T10:00:25.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc266ffd700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2680c5510 0x7fc26815fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:25.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc267fff700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc268160050 0x7fc2681f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:25.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc2677fe700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc26815cac0 0x7fc2680c4fd0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc25c007290 tx=0x7fc25c007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc24ffff700 1 -- 172.21.15.158:0/2230451819 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c00f8d0 con 0x7fc26815cac0 2023-10-16T10:00:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc24ffff700 1 -- 172.21.15.158:0/2230451819 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c00fa50 con 0x7fc26815cac0 2023-10-16T10:00:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2680aca30 con 0x7fc26815cac0 2023-10-16T10:00:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc24ffff700 1 -- 172.21.15.158:0/2230451819 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c00fe00 con 0x7fc26815cac0 2023-10-16T10:00:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.424+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2680c6e30 con 0x7fc26815cac0 2023-10-16T10:00:25.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.425+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2680c7020 con 0x7fc26815cac0 2023-10-16T10:00:25.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.426+0000 7fc24ffff700 1 -- 172.21.15.158:0/2230451819 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc25c01d460 con 0x7fc26815cac0 2023-10-16T10:00:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.427+0000 7fc24ffff700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc25406b020 0x7fc25406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.427+0000 7fc24ffff700 1 -- 172.21.15.158:0/2230451819 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fc25c08a3b0 con 0x7fc26815cac0 2023-10-16T10:00:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.427+0000 7fc266ffd700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc25406b020 0x7fc25406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.427+0000 7fc266ffd700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc25406b020 0x7fc25406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2680c4d20 tx=0x7fc258009040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.430+0000 7fc24ffff700 1 -- 172.21.15.158:0/2230451819 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc25c055190 con 0x7fc26815cac0 2023-10-16T10:00:25.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.627+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc2680ac890 con 0x7fc26815cac0 2023-10-16T10:00:25.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.630+0000 7fc24ffff700 1 -- 172.21.15.158:0/2230451819 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc2680ac890 con 0x7fc26815cac0 2023-10-16T10:00:25.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc25406b020 msgr2=0x7fc25406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc25406b020 0x7fc25406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2680c4d20 tx=0x7fc258009040 comp rx=0 tx=0).stop 2023-10-16T10:00:25.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc26815cac0 msgr2=0x7fc2680c4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc26815cac0 0x7fc2680c4fd0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc25c007290 tx=0x7fc25c007650 comp rx=0 tx=0).stop 2023-10-16T10:00:25.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 shutdown_connections 2023-10-16T10:00:25.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc25406b020 0x7fc25406d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc268160050 0x7fc2681f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc26815cac0 0x7fc2680c4fd0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 --2- 172.21.15.158:0/2230451819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2680c5510 0x7fc26815fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.632+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 >> 172.21.15.158:0/2230451819 conn(0x7fc26806a8d0 msgr2=0x7fc26806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.633+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 shutdown_connections 2023-10-16T10:00:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.633+0000 7fc26dbad700 1 -- 172.21.15.158:0/2230451819 wait complete. 2023-10-16T10:00:25.646 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T10:00:25.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.766+0000 7fbddf909700 1 Processor -- start 2023-10-16T10:00:25.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.767+0000 7fbddf909700 1 -- start start 2023-10-16T10:00:25.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.767+0000 7fbddf909700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd8160e20 0x7fbdd80cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.767+0000 7fbddf909700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbdd806afe0 0x7fbdd80cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.767+0000 7fbddf909700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd80afb60 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.767+0000 7fbddf909700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd80cdc80 con 0x7fbdd806afe0 2023-10-16T10:00:25.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.767+0000 7fbddf909700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd80cfcf0 con 0x7fbdd8160e20 2023-10-16T10:00:25.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbddd6a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd8160e20 0x7fbdd80cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbdddea6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbdd806afe0 0x7fbdd80cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbddd6a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd8160e20 0x7fbdd80cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:37474/0 (socket says 172.21.15.158:37474) 2023-10-16T10:00:25.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbdddea6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbdd806afe0 0x7fbdd80cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:56056/0 (socket says 172.21.15.158:56056) 2023-10-16T10:00:25.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbddd6a5700 1 -- 172.21.15.158:0/3572017598 learned_addr learned my addr 172.21.15.158:0/3572017598 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:25.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbddcea4700 1 --1- 172.21.15.158:0/3572017598 >> v1:172.21.15.172:6789/0 conn(0x7fbdd80ca9f0 0x7fbdd806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:58952/0 (socket says 172.21.15.158:58952) 2023-10-16T10:00:25.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbdddea6700 1 -- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd8160e20 msgr2=0x7fbdd80cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:25.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbdddea6700 1 --2- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd8160e20 0x7fbdd80cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbdddea6700 1 -- 172.21.15.158:0/3572017598 >> v1:172.21.15.172:6789/0 conn(0x7fbdd80ca9f0 legacy=0x7fbdd806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbdddea6700 1 -- 172.21.15.158:0/3572017598 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd80cffd0 con 0x7fbdd806afe0 2023-10-16T10:00:25.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.768+0000 7fbddd6a5700 1 --2- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd8160e20 0x7fbdd80cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:25.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbdddea6700 1 --2- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd806afe0 0x7fbdd80cd740 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0009d90 tx=0x7fbdd000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c868928f728f69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbdcffff700 1 -- 172.21.15.158:0/3572017598 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd000e070 con 0x7fbdd806afe0 2023-10-16T10:00:25.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbdcffff700 1 -- 172.21.15.158:0/3572017598 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd000bb20 con 0x7fbdd806afe0 2023-10-16T10:00:25.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbdcffff700 1 -- 172.21.15.158:0/3572017598 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0012430 con 0x7fbdd806afe0 2023-10-16T10:00:25.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 -- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd806afe0 msgr2=0x7fbdd80cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 --2- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd806afe0 0x7fbdd80cd740 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0009d90 tx=0x7fbdd000b330 comp rx=0 tx=0).stop 2023-10-16T10:00:25.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 -- 172.21.15.158:0/3572017598 shutdown_connections 2023-10-16T10:00:25.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 --2- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd806afe0 0x7fbdd80cd740 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 --2- 172.21.15.158:0/3572017598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd8160e20 0x7fbdd80cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 -- 172.21.15.158:0/3572017598 >> 172.21.15.158:0/3572017598 conn(0x7fbdd80d03a0 msgr2=0x7fbdd80d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:25.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 -- 172.21.15.158:0/3572017598 shutdown_connections 2023-10-16T10:00:25.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.769+0000 7fbddf909700 1 -- 172.21.15.158:0/3572017598 wait complete. 2023-10-16T10:00:25.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.770+0000 7fbddf909700 1 Processor -- start 2023-10-16T10:00:25.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddf909700 1 -- start start 2023-10-16T10:00:25.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddf909700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd806afe0 0x7fbdd806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddf909700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdd80ca9f0 0x7fbdd81f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddf909700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd8160e20 0x7fbdd815d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddf909700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd81e6d30 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddf909700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd80c6c50 con 0x7fbdd8160e20 2023-10-16T10:00:25.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddf909700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd80af020 con 0x7fbdd806afe0 2023-10-16T10:00:25.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddcea4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdd80ca9f0 0x7fbdd81f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddcea4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdd80ca9f0 0x7fbdd81f2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54880/0 (socket says 172.21.15.158:54880) 2023-10-16T10:00:25.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.771+0000 7fbddcea4700 1 -- 172.21.15.158:0/2052682587 learned_addr learned my addr 172.21.15.158:0/2052682587 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:25.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbdddea6700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd8160e20 0x7fbdd815d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddd6a5700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd806afe0 0x7fbdd806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddcea4700 1 -- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd806afe0 msgr2=0x7fbdd806f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddcea4700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd806afe0 0x7fbdd806f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddcea4700 1 -- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd8160e20 msgr2=0x7fbdd815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:25.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddcea4700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd8160e20 0x7fbdd815d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:25.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddcea4700 1 -- 172.21.15.158:0/2052682587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8009550 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddd6a5700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd806afe0 0x7fbdd806f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:25.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddcea4700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdd80ca9f0 0x7fbdd81f2ae0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400c520 tx=0x7fbdd400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbdcdffb700 1 -- 172.21.15.158:0/2052682587 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4005cb0 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd0009a70 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbdcdffb700 1 -- 172.21.15.158:0/2052682587 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd4005e10 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbdcdffb700 1 -- 172.21.15.158:0/2052682587 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40072d0 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.772+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd81512b0 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.773+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd80681b0 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.773+0000 7fbdcdffb700 1 -- 172.21.15.158:0/2052682587 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbdd4007430 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.774+0000 7fbdcdffb700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fbdbc06b020 0x7fbdbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:25.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.774+0000 7fbdcdffb700 1 -- 172.21.15.158:0/2052682587 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fbdd4088dc0 con 0x7fbdd80ca9f0 2023-10-16T10:00:25.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.774+0000 7fbddd6a5700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fbdbc06b020 0x7fbdbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:25.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.774+0000 7fbddd6a5700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fbdbc06b020 0x7fbdbc06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009ce0 tx=0x7fbdc8008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:25.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:25.778+0000 7fbdcdffb700 1 -- 172.21.15.158:0/2052682587 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbdd4053b80 con 0x7fbdd80ca9f0 2023-10-16T10:00:26.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.051+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fbdd80c3c80 con 0x7fbdd80ca9f0 2023-10-16T10:00:26.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.571+0000 7fbdcdffb700 1 -- 172.21.15.158:0/2052682587 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v634) v1 ==== 166+0+0 (secure 0 0 0) 0x7fbdd40571a0 con 0x7fbdd80ca9f0 2023-10-16T10:00:26.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.574+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fbdbc06b020 msgr2=0x7fbdbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:26.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.574+0000 7fbddf909700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fbdbc06b020 0x7fbdbc06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009ce0 tx=0x7fbdc8008040 comp rx=0 tx=0).stop 2023-10-16T10:00:26.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdd80ca9f0 msgr2=0x7fbdd81f2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:26.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdd80ca9f0 0x7fbdd81f2ae0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400c520 tx=0x7fbdd400c8e0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 shutdown_connections 2023-10-16T10:00:26.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fbdbc06b020 0x7fbdbc06d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbdd8160e20 0x7fbdd815d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbdd806afe0 0x7fbdd806f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 --2- 172.21.15.158:0/2052682587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdd80ca9f0 0x7fbdd81f2ae0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 >> 172.21.15.158:0/2052682587 conn(0x7fbdd80d03a0 msgr2=0x7fbdd81537f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:26.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 shutdown_connections 2023-10-16T10:00:26.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.575+0000 7fbddf909700 1 -- 172.21.15.158:0/2052682587 wait complete. 2023-10-16T10:00:26.591 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-10-16T10:00:26.592 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-10-16T10:00:26.592 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T10:00:26.593 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:26.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.716+0000 7fdd4bf83700 1 Processor -- start 2023-10-16T10:00:26.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.716+0000 7fdd4bf83700 1 -- start start 2023-10-16T10:00:26.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4bf83700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdd4406ab40 0x7fdd4406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:26.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4bf83700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdd4406e5d0 0x7fdd44071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:26.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4bf83700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406af20 0x7fdd440722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:26.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4bf83700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd44071e50 con 0x7fdd4406ab40 2023-10-16T10:00:26.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4bf83700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd44161270 con 0x7fdd4406e5d0 2023-10-16T10:00:26.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4bf83700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44161510 con 0x7fdd4406af20 2023-10-16T10:00:26.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdd4406ab40 0x7fdd4406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:26.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdd4406ab40 0x7fdd4406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54886/0 (socket says 172.21.15.158:54886) 2023-10-16T10:00:26.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 -- 172.21.15.158:0/1260838726 learned_addr learned my addr 172.21.15.158:0/1260838726 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:26.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4a520700 1 --2- 172.21.15.158:0/1260838726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406af20 0x7fdd440722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:26.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd4951e700 1 --2- 172.21.15.158:0/1260838726 >> v2:172.21.15.172:3300/0 conn(0x7fdd4406e5d0 0x7fdd44071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:26.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 -- 172.21.15.158:0/1260838726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406af20 msgr2=0x7fdd440722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:26.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 --2- 172.21.15.158:0/1260838726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406af20 0x7fdd440722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 -- 172.21.15.158:0/1260838726 >> v2:172.21.15.172:3300/0 conn(0x7fdd4406e5d0 msgr2=0x7fdd44071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:26.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 --2- 172.21.15.158:0/1260838726 >> v2:172.21.15.172:3300/0 conn(0x7fdd4406e5d0 0x7fdd44071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 -- 172.21.15.158:0/1260838726 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd441617f0 con 0x7fdd4406ab40 2023-10-16T10:00:26.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd49d1f700 1 --2- 172.21.15.158:0/1260838726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406ab40 0x7fdd4406d820 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fdd440c3bf0 tx=0x7fdd40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689b6605c22201a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:26.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd48d1d700 1 -- 172.21.15.158:0/1260838726 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000b870 con 0x7fdd4406ab40 2023-10-16T10:00:26.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd48d1d700 1 -- 172.21.15.158:0/1260838726 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4000b9d0 con 0x7fdd4406ab40 2023-10-16T10:00:26.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.717+0000 7fdd48d1d700 1 -- 172.21.15.158:0/1260838726 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000bd60 con 0x7fdd4406ab40 2023-10-16T10:00:26.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1260838726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406ab40 msgr2=0x7fdd4406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:26.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1260838726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406ab40 0x7fdd4406d820 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fdd440c3bf0 tx=0x7fdd40009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1260838726 shutdown_connections 2023-10-16T10:00:26.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1260838726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406af20 0x7fdd440722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1260838726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406ab40 0x7fdd4406d820 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1260838726 >> v2:172.21.15.172:3300/0 conn(0x7fdd4406e5d0 0x7fdd44071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1260838726 >> 172.21.15.158:0/1260838726 conn(0x7fdd440d02b0 msgr2=0x7fdd440d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:26.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1260838726 shutdown_connections 2023-10-16T10:00:26.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.718+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1260838726 wait complete. 2023-10-16T10:00:26.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.719+0000 7fdd4bf83700 1 Processor -- start 2023-10-16T10:00:26.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.719+0000 7fdd4bf83700 1 -- start start 2023-10-16T10:00:26.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4bf83700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406ab40 0x7fdd441466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:26.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4bf83700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd4406af20 0x7fdd441512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:26.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4bf83700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406e5d0 0x7fdd440cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:26.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4bf83700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440c40c0 con 0x7fdd4406e5d0 2023-10-16T10:00:26.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4bf83700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440c5490 con 0x7fdd4406af20 2023-10-16T10:00:26.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4bf83700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44146bf0 con 0x7fdd4406ab40 2023-10-16T10:00:26.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406e5d0 0x7fdd440cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:26.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406e5d0 0x7fdd440cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54896/0 (socket says 172.21.15.158:54896) 2023-10-16T10:00:26.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 -- 172.21.15.158:0/1734661069 learned_addr learned my addr 172.21.15.158:0/1734661069 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:26.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 -- 172.21.15.158:0/1734661069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406ab40 msgr2=0x7fdd441466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:26.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406ab40 0x7fdd441466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 -- 172.21.15.158:0/1734661069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd4406af20 msgr2=0x7fdd441512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:26.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4951e700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd4406af20 0x7fdd441512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:26.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd4406af20 0x7fdd441512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.720+0000 7fdd4a520700 1 -- 172.21.15.158:0/1734661069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd40009550 con 0x7fdd4406e5d0 2023-10-16T10:00:26.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.721+0000 7fdd4a520700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406e5d0 0x7fdd440cbfe0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00cd70 tx=0x7fdd3c00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:26.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.721+0000 7fdd3a7fc700 1 -- 172.21.15.158:0/1734661069 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c003a20 con 0x7fdd4406e5d0 2023-10-16T10:00:26.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.721+0000 7fdd3a7fc700 1 -- 172.21.15.158:0/1734661069 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c003b80 con 0x7fdd4406e5d0 2023-10-16T10:00:26.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.721+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd44151af0 con 0x7fdd4406e5d0 2023-10-16T10:00:26.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.721+0000 7fdd3a7fc700 1 -- 172.21.15.158:0/1734661069 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c004020 con 0x7fdd4406e5d0 2023-10-16T10:00:26.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.721+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd441520b0 con 0x7fdd4406e5d0 2023-10-16T10:00:26.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.722+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd440681b0 con 0x7fdd4406e5d0 2023-10-16T10:00:26.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.722+0000 7fdd3a7fc700 1 -- 172.21.15.158:0/1734661069 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd3c004180 con 0x7fdd4406e5d0 2023-10-16T10:00:26.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.722+0000 7fdd3a7fc700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fdd2806b020 0x7fdd2806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:26.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.723+0000 7fdd3a7fc700 1 -- 172.21.15.158:0/1734661069 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fdd3c012070 con 0x7fdd4406e5d0 2023-10-16T10:00:26.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.723+0000 7fdd49d1f700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fdd2806b020 0x7fdd2806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:26.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.723+0000 7fdd49d1f700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fdd2806b020 0x7fdd2806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009c10 tx=0x7fdd40010000 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:26.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.727+0000 7fdd3a7fc700 1 -- 172.21.15.158:0/1734661069 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd3c054ed0 con 0x7fdd4406e5d0 2023-10-16T10:00:26.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.924+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd440c5600 con 0x7fdd4406e5d0 2023-10-16T10:00:26.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.925+0000 7fdd3a7fc700 1 -- 172.21.15.158:0/1734661069 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+14864 (secure 0 0 0) 0x7fdd3c0584f0 con 0x7fdd4406e5d0 2023-10-16T10:00:26.926 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:26.926 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":103,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:24.568337+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6837/1006858952":"2023-10-17T10:00:21.290569+0000","172.21.15.158:6835/1006858952":"2023-10-17T10:00:21.290569+0000","172.21.15.158:0/1634845683":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/1729189331":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/331886015":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/1776913260":"2023-10-17T10:00:20.313472+0000","172.21.15.158:6833/334523996":"2023-10-17T10:00:20.313472+0000","172.21.15.158:6832/334523996":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/245591896":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/10645813":"2023-10-17T10:00:20.313472+0000","172.21.15.158:0/1178199719":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/873662109":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6833/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:6832/3798429240":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/1880882485":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/693060665":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/2133642547":"2023-10-17T10:00:12.821288+0000","172.21.15.158:0/4289198127":"2023-10-17T10:00:12.821288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:26.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fdd2806b020 msgr2=0x7fdd2806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:26.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fdd2806b020 0x7fdd2806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009c10 tx=0x7fdd40010000 comp rx=0 tx=0).stop 2023-10-16T10:00:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406e5d0 msgr2=0x7fdd440cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406e5d0 0x7fdd440cbfe0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00cd70 tx=0x7fdd3c00a570 comp rx=0 tx=0).stop 2023-10-16T10:00:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 shutdown_connections 2023-10-16T10:00:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fdd2806b020 0x7fdd2806d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd4406e5d0 0x7fdd440cbfe0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdd4406ab40 0x7fdd441466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 --2- 172.21.15.158:0/1734661069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd4406af20 0x7fdd441512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 >> 172.21.15.158:0/1734661069 conn(0x7fdd440d02b0 msgr2=0x7fdd440adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 shutdown_connections 2023-10-16T10:00:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:26.927+0000 7fdd4bf83700 1 -- 172.21.15.158:0/1734661069 wait complete. 2023-10-16T10:00:26.940 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:00:26.941 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:00:27.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.062+0000 7fcefed58700 1 Processor -- start 2023-10-16T10:00:27.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.062+0000 7fcefed58700 1 -- start start 2023-10-16T10:00:27.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefed58700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcef815d840 0x7fcef8161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefed58700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcef80cf070 0x7fcef80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefed58700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef815dc20 con 0x7fcef80d1430 2023-10-16T10:00:27.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefed58700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef80cfa40 con 0x7fcef815d840 2023-10-16T10:00:27.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefed58700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef806c0e0 con 0x7fcef80cf070 2023-10-16T10:00:27.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefcaf4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcef815d840 0x7fcef8161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefcaf4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcef815d840 0x7fcef8161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54900/0 (socket says 172.21.15.158:54900) 2023-10-16T10:00:27.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefd2f5700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcef80d1430 0x7fcef80d1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55078/0 (socket says 172.21.15.158:55078) 2023-10-16T10:00:27.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.063+0000 7fcefcaf4700 1 -- 172.21.15.158:0/224192309 learned_addr learned my addr 172.21.15.158:0/224192309 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:27.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcef7fff700 1 --2- 172.21.15.158:0/224192309 >> v2:172.21.15.172:3300/0 conn(0x7fcef80cf070 0x7fcef80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcefcaf4700 1 -- 172.21.15.158:0/224192309 >> v2:172.21.15.172:3300/0 conn(0x7fcef80cf070 msgr2=0x7fcef80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:27.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcefcaf4700 1 --2- 172.21.15.158:0/224192309 >> v2:172.21.15.172:3300/0 conn(0x7fcef80cf070 0x7fcef80cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcefcaf4700 1 -- 172.21.15.158:0/224192309 >> v1:172.21.15.158:6789/0 conn(0x7fcef80d1430 legacy=0x7fcef80d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcefcaf4700 1 -- 172.21.15.158:0/224192309 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef806c3c0 con 0x7fcef815d840 2023-10-16T10:00:27.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcef77fe700 1 -- 172.21.15.158:0/224192309 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476086994 0 0) 0x7fcef0001490 con 0x7fcef80d1430 2023-10-16T10:00:27.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcef7fff700 1 --2- 172.21.15.158:0/224192309 >> v2:172.21.15.172:3300/0 conn(0x7fcef80cf070 0x7fcef80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:27.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcefcaf4700 1 --2- 172.21.15.158:0/224192309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef815d840 0x7fcef8161d00 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fcef80b3a00 tx=0x7fceec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b0fb39adb96c1b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:27.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcef77fe700 1 -- 172.21.15.158:0/224192309 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec00b870 con 0x7fcef815d840 2023-10-16T10:00:27.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcef77fe700 1 -- 172.21.15.158:0/224192309 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec00b9d0 con 0x7fcef815d840 2023-10-16T10:00:27.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.064+0000 7fcef77fe700 1 -- 172.21.15.158:0/224192309 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec00bd60 con 0x7fcef815d840 2023-10-16T10:00:27.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 -- 172.21.15.158:0/224192309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef815d840 msgr2=0x7fcef8161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 --2- 172.21.15.158:0/224192309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef815d840 0x7fcef8161d00 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fcef80b3a00 tx=0x7fceec009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 -- 172.21.15.158:0/224192309 shutdown_connections 2023-10-16T10:00:27.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 --2- 172.21.15.158:0/224192309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef815d840 0x7fcef8161d00 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 --2- 172.21.15.158:0/224192309 >> v2:172.21.15.172:3300/0 conn(0x7fcef80cf070 0x7fcef80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 -- 172.21.15.158:0/224192309 >> 172.21.15.158:0/224192309 conn(0x7fcef806a8d0 msgr2=0x7fcef806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:27.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 -- 172.21.15.158:0/224192309 shutdown_connections 2023-10-16T10:00:27.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.065+0000 7fcefed58700 1 -- 172.21.15.158:0/224192309 wait complete. 2023-10-16T10:00:27.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.066+0000 7fcefed58700 1 Processor -- start 2023-10-16T10:00:27.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.066+0000 7fcefed58700 1 -- start start 2023-10-16T10:00:27.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefed58700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef80cf070 0x7fcef8157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefed58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcef80d1430 0x7fcef8160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefed58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcef815d840 0x7fcef8144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefed58700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef815f480 con 0x7fcef80cf070 2023-10-16T10:00:27.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefed58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef81602c0 con 0x7fcef80d1430 2023-10-16T10:00:27.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefed58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8158a20 con 0x7fcef815d840 2023-10-16T10:00:27.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefcaf4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef80cf070 0x7fcef8157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefcaf4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef80cf070 0x7fcef8157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54906/0 (socket says 172.21.15.158:54906) 2023-10-16T10:00:27.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.067+0000 7fcefcaf4700 1 -- 172.21.15.158:0/2308857489 learned_addr learned my addr 172.21.15.158:0/2308857489 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:27.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefcaf4700 1 -- 172.21.15.158:0/2308857489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcef815d840 msgr2=0x7fcef8144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefcaf4700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcef815d840 0x7fcef8144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefcaf4700 1 -- 172.21.15.158:0/2308857489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcef80d1430 msgr2=0x7fcef8160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefcaf4700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcef80d1430 0x7fcef8160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefcaf4700 1 -- 172.21.15.158:0/2308857489 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec009550 con 0x7fcef80cf070 2023-10-16T10:00:27.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefcaf4700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef80cf070 0x7fcef8157a60 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fceec014040 tx=0x7fceec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:27.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcef57fa700 1 -- 172.21.15.158:0/2308857489 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec004520 con 0x7fcef80cf070 2023-10-16T10:00:27.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcef57fa700 1 -- 172.21.15.158:0/2308857489 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec01e070 con 0x7fcef80cf070 2023-10-16T10:00:27.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcef57fa700 1 -- 172.21.15.158:0/2308857489 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec0038a0 con 0x7fcef80cf070 2023-10-16T10:00:27.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef8157fa0 con 0x7fcef80cf070 2023-10-16T10:00:27.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.068+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef815f990 con 0x7fcef80cf070 2023-10-16T10:00:27.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.069+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef81605a0 con 0x7fcef80cf070 2023-10-16T10:00:27.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.069+0000 7fcef57fa700 1 -- 172.21.15.158:0/2308857489 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fceec003a00 con 0x7fcef80cf070 2023-10-16T10:00:27.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.070+0000 7fcef57fa700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fcedc06b020 0x7fcedc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.070+0000 7fcef57fa700 1 -- 172.21.15.158:0/2308857489 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fceec020070 con 0x7fcef80cf070 2023-10-16T10:00:27.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.070+0000 7fcef7fff700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fcedc06b020 0x7fcedc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.070+0000 7fcef7fff700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fcedc06b020 0x7fcedc06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcee4006fd0 tx=0x7fcee4008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:27.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.074+0000 7fcef57fa700 1 -- 172.21.15.158:0/2308857489 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fceec055cd0 con 0x7fcef80cf070 2023-10-16T10:00:27.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.267+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcef80681b0 con 0x7fcef80cf070 2023-10-16T10:00:27.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.268+0000 7fcef57fa700 1 -- 172.21.15.158:0/2308857489 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7fceec02a0a0 con 0x7fcef80cf070 2023-10-16T10:00:27.269 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:00:27.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fcedc06b020 msgr2=0x7fcedc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fcedc06b020 0x7fcedc06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcee4006fd0 tx=0x7fcee4008040 comp rx=0 tx=0).stop 2023-10-16T10:00:27.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef80cf070 msgr2=0x7fcef8157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef80cf070 0x7fcef8157a60 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fceec014040 tx=0x7fceec004750 comp rx=0 tx=0).stop 2023-10-16T10:00:27.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 shutdown_connections 2023-10-16T10:00:27.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fcedc06b020 0x7fcedc06d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcef815d840 0x7fcef8144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcef80cf070 0x7fcef8157a60 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 --2- 172.21.15.158:0/2308857489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcef80d1430 0x7fcef8160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 >> 172.21.15.158:0/2308857489 conn(0x7fcef806a8d0 msgr2=0x7fcef80cdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:27.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 shutdown_connections 2023-10-16T10:00:27.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.270+0000 7fcefed58700 1 -- 172.21.15.158:0/2308857489 wait complete. 2023-10-16T10:00:27.284 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T10:00:27.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.404+0000 7f99bc4ab700 1 Processor -- start 2023-10-16T10:00:27.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.404+0000 7f99bc4ab700 1 -- start start 2023-10-16T10:00:27.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99bc4ab700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40cdee0 0x7f99b415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99bc4ab700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f99b40b6e10 0x7f99b41521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99bc4ab700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b415c5d0 con 0x7f99b40ce2c0 2023-10-16T10:00:27.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99bc4ab700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b415d2c0 con 0x7f99b40b6e10 2023-10-16T10:00:27.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99bc4ab700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b415c8b0 con 0x7f99b40cdee0 2023-10-16T10:00:27.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f99b40b6e10 0x7f99b41521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f99b40b6e10 0x7f99b41521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54916/0 (socket says 172.21.15.158:54916) 2023-10-16T10:00:27.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 -- 172.21.15.158:0/3500140569 learned_addr learned my addr 172.21.15.158:0/3500140569 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:27.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99ba247700 1 --2- 172.21.15.158:0/3500140569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40cdee0 0x7f99b415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 -- 172.21.15.158:0/3500140569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40cdee0 msgr2=0x7f99b415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:27.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 --2- 172.21.15.158:0/3500140569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40cdee0 0x7f99b415bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 -- 172.21.15.158:0/3500140569 >> v1:172.21.15.172:6789/0 conn(0x7f99b40ce2c0 legacy=0x7f99b40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:27.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 -- 172.21.15.158:0/3500140569 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b415cb90 con 0x7f99b40b6e10 2023-10-16T10:00:27.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.405+0000 7f99b9a46700 1 --2- 172.21.15.158:0/3500140569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40b6e10 0x7f99b41521a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098f0 tx=0x7f99a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23002ca734e70479 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:27.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99b9245700 1 -- 172.21.15.158:0/3500140569 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4004230 con 0x7f99b40b6e10 2023-10-16T10:00:27.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99b9245700 1 -- 172.21.15.158:0/3500140569 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a4004390 con 0x7f99b40b6e10 2023-10-16T10:00:27.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99b9245700 1 -- 172.21.15.158:0/3500140569 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40036a0 con 0x7f99b40b6e10 2023-10-16T10:00:27.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 -- 172.21.15.158:0/3500140569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40b6e10 msgr2=0x7f99b41521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/3500140569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40b6e10 0x7f99b41521a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098f0 tx=0x7f99a4009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:27.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 -- 172.21.15.158:0/3500140569 shutdown_connections 2023-10-16T10:00:27.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/3500140569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40cdee0 0x7f99b415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/3500140569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40b6e10 0x7f99b41521a0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 -- 172.21.15.158:0/3500140569 >> 172.21.15.158:0/3500140569 conn(0x7f99b406a8d0 msgr2=0x7f99b406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:27.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 -- 172.21.15.158:0/3500140569 shutdown_connections 2023-10-16T10:00:27.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.406+0000 7f99bc4ab700 1 -- 172.21.15.158:0/3500140569 wait complete. 2023-10-16T10:00:27.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.407+0000 7f99bc4ab700 1 Processor -- start 2023-10-16T10:00:27.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.407+0000 7f99bc4ab700 1 -- start start 2023-10-16T10:00:27.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99bc4ab700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40b6e10 0x7f99b40cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99bc4ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40cdee0 0x7f99b4155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99bc4ab700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b40ce2c0 0x7f99b40c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99bc4ab700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4155ed0 con 0x7f99b40cdee0 2023-10-16T10:00:27.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99bc4ab700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b40c60f0 con 0x7f99b40ce2c0 2023-10-16T10:00:27.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99bc4ab700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4155bf0 con 0x7f99b40b6e10 2023-10-16T10:00:27.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99b9a46700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40cdee0 0x7f99b4155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99b9a46700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40cdee0 0x7f99b4155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54922/0 (socket says 172.21.15.158:54922) 2023-10-16T10:00:27.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99b9a46700 1 -- 172.21.15.158:0/1693671860 learned_addr learned my addr 172.21.15.158:0/1693671860 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:27.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99ba247700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40b6e10 0x7f99b40cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99baa48700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b40ce2c0 0x7f99b40c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99b9a46700 1 -- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40b6e10 msgr2=0x7f99b40cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.408+0000 7f99b9a46700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40b6e10 0x7f99b40cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99b9a46700 1 -- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b40ce2c0 msgr2=0x7f99b40c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:27.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99b9a46700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b40ce2c0 0x7f99b40c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:27.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99b9a46700 1 -- 172.21.15.158:0/1693671860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4009550 con 0x7f99b40cdee0 2023-10-16T10:00:27.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99ba247700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40b6e10 0x7f99b40cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:27.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99baa48700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b40ce2c0 0x7f99b40c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:27.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99b9a46700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40cdee0 0x7f99b4155150 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009130 tx=0x7f99a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:27.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99aaffd700 1 -- 172.21.15.158:0/1693671860 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4003e30 con 0x7f99b40cdee0 2023-10-16T10:00:27.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99aaffd700 1 -- 172.21.15.158:0/1693671860 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a401f070 con 0x7f99b40cdee0 2023-10-16T10:00:27.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b40c6390 con 0x7f99b40cdee0 2023-10-16T10:00:27.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99aaffd700 1 -- 172.21.15.158:0/1693671860 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a400b8a0 con 0x7f99b40cdee0 2023-10-16T10:00:27.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.409+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b4152c40 con 0x7f99b40cdee0 2023-10-16T10:00:27.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.410+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b40c5f50 con 0x7f99b40cdee0 2023-10-16T10:00:27.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.410+0000 7f99aaffd700 1 -- 172.21.15.158:0/1693671860 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f99a4021030 con 0x7f99b40cdee0 2023-10-16T10:00:27.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.410+0000 7f99aaffd700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f999806b020 0x7f999806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:27.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.411+0000 7f99aaffd700 1 -- 172.21.15.158:0/1693671860 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f99a408b090 con 0x7f99b40cdee0 2023-10-16T10:00:27.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.411+0000 7f99ba247700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f999806b020 0x7f999806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:27.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.411+0000 7f99ba247700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f999806b020 0x7f999806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f99b0006fd0 tx=0x7f99b0008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:27.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.415+0000 7f99aaffd700 1 -- 172.21.15.158:0/1693671860 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99b40c5f50 con 0x7f99b40cdee0 2023-10-16T10:00:27.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:27.694+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f99b40c68b0 con 0x7f99b40cdee0 2023-10-16T10:00:28.547 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:00:28.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:00:28.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.577+0000 7f99aaffd700 1 -- 172.21.15.158:0/1693671860 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v636) v1 ==== 186+0+0 (secure 0 0 0) 0x7f99b40c68b0 con 0x7f99b40cdee0 2023-10-16T10:00:28.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f999806b020 msgr2=0x7f999806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f999806b020 0x7f999806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f99b0006fd0 tx=0x7f99b0008040 comp rx=0 tx=0).stop 2023-10-16T10:00:28.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40cdee0 msgr2=0x7f99b4155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40cdee0 0x7f99b4155150 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009130 tx=0x7f99a4004750 comp rx=0 tx=0).stop 2023-10-16T10:00:28.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 shutdown_connections 2023-10-16T10:00:28.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f999806b020 0x7f999806d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b40ce2c0 0x7f99b40c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99b40b6e10 0x7f99b40cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 --2- 172.21.15.158:0/1693671860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b40cdee0 0x7f99b4155150 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 >> 172.21.15.158:0/1693671860 conn(0x7f99b406a8d0 msgr2=0x7f99b415f100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:28.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 shutdown_connections 2023-10-16T10:00:28.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.581+0000 7f99bc4ab700 1 -- 172.21.15.158:0/1693671860 wait complete. 2023-10-16T10:00:28.596 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T10:00:28.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.763+0000 7f4860429700 1 Processor -- start 2023-10-16T10:00:28.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.763+0000 7f4860429700 1 -- start start 2023-10-16T10:00:28.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f4860429700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f485815da40 0x7f48580caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f4860429700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580cbd30 con 0x7f48580cc590 2023-10-16T10:00:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f4860429700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580cc1c0 con 0x7f485806afe0 2023-10-16T10:00:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f4860429700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485815de20 con 0x7f485815da40 2023-10-16T10:00:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485e1c5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f485815da40 0x7f48580caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485d9c4700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f48580cc590 0x7f485806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55104/0 (socket says 172.21.15.158:55104) 2023-10-16T10:00:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485d9c4700 1 -- 172.21.15.158:0/2498872610 learned_addr learned my addr 172.21.15.158:0/2498872610 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485e1c5700 1 -- 172.21.15.158:0/2498872610 >> v1:172.21.15.172:6789/0 conn(0x7f485806afe0 legacy=0x7f48580afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485e1c5700 1 -- 172.21.15.158:0/2498872610 >> v1:172.21.15.158:6789/0 conn(0x7f48580cc590 legacy=0x7f485806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485e1c5700 1 -- 172.21.15.158:0/2498872610 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48580ce260 con 0x7f485815da40 2023-10-16T10:00:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485e1c5700 1 --2- 172.21.15.158:0/2498872610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485815da40 0x7f48580caa20 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4854009840 tx=0x7f4854009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8123071dc0c67708 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:28.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485d1c3700 1 -- 172.21.15.158:0/2498872610 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854004230 con 0x7f485815da40 2023-10-16T10:00:28.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485d1c3700 1 -- 172.21.15.158:0/2498872610 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4854004390 con 0x7f485815da40 2023-10-16T10:00:28.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.764+0000 7f485d1c3700 1 -- 172.21.15.158:0/2498872610 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48540036a0 con 0x7f485815da40 2023-10-16T10:00:28.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.765+0000 7f4860429700 1 -- 172.21.15.158:0/2498872610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485815da40 msgr2=0x7f48580caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.765+0000 7f4860429700 1 --2- 172.21.15.158:0/2498872610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485815da40 0x7f48580caa20 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4854009840 tx=0x7f4854009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:28.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.765+0000 7f4860429700 1 -- 172.21.15.158:0/2498872610 shutdown_connections 2023-10-16T10:00:28.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.765+0000 7f4860429700 1 --2- 172.21.15.158:0/2498872610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485815da40 0x7f48580caa20 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.765+0000 7f4860429700 1 -- 172.21.15.158:0/2498872610 >> 172.21.15.158:0/2498872610 conn(0x7f4858099a10 msgr2=0x7f48580d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:28.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.765+0000 7f4860429700 1 -- 172.21.15.158:0/2498872610 shutdown_connections 2023-10-16T10:00:28.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.765+0000 7f4860429700 1 -- 172.21.15.158:0/2498872610 wait complete. 2023-10-16T10:00:28.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.766+0000 7f4860429700 1 Processor -- start 2023-10-16T10:00:28.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.766+0000 7f4860429700 1 -- start start 2023-10-16T10:00:28.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f4860429700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485806afe0 0x7f4858159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:28.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f4860429700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48580cc590 0x7f48581e8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:28.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f4860429700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485815da40 0x7f48581e8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:28.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f4860429700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48580c74a0 con 0x7f48580cc590 2023-10-16T10:00:28.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f4860429700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485806f770 con 0x7f485806afe0 2023-10-16T10:00:28.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f4860429700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48580c7fb0 con 0x7f485815da40 2023-10-16T10:00:28.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48580cc590 0x7f48581e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:28.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48580cc590 0x7f48581e8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54934/0 (socket says 172.21.15.158:54934) 2023-10-16T10:00:28.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 -- 172.21.15.158:0/101662713 learned_addr learned my addr 172.21.15.158:0/101662713 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:28.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485e1c5700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485806afe0 0x7f4858159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:28.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485e9c6700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485815da40 0x7f48581e8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:28.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 -- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485815da40 msgr2=0x7f48581e8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485815da40 0x7f48581e8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 -- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485806afe0 msgr2=0x7f4858159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485806afe0 0x7f4858159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.767+0000 7f485d9c4700 1 -- 172.21.15.158:0/101662713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48540094a0 con 0x7f48580cc590 2023-10-16T10:00:28.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.768+0000 7f485d9c4700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48580cc590 0x7f48581e8580 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f484800c500 tx=0x7f484800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:28.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.768+0000 7f484effd700 1 -- 172.21.15.158:0/101662713 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848005d70 con 0x7f48580cc590 2023-10-16T10:00:28.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.768+0000 7f484effd700 1 -- 172.21.15.158:0/101662713 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48480070d0 con 0x7f48580cc590 2023-10-16T10:00:28.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.768+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48580c7680 con 0x7f48580cc590 2023-10-16T10:00:28.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.768+0000 7f484effd700 1 -- 172.21.15.158:0/101662713 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848007460 con 0x7f48580cc590 2023-10-16T10:00:28.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.768+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48580c7c40 con 0x7f48580cc590 2023-10-16T10:00:28.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.769+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48580681b0 con 0x7f48580cc590 2023-10-16T10:00:28.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.769+0000 7f484effd700 1 -- 172.21.15.158:0/101662713 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f48480075c0 con 0x7f48580cc590 2023-10-16T10:00:28.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.769+0000 7f484effd700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f483c06b020 0x7f483c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:28.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.770+0000 7f484effd700 1 -- 172.21.15.158:0/101662713 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f4848088dc0 con 0x7f48580cc590 2023-10-16T10:00:28.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.770+0000 7f485e1c5700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f483c06b020 0x7f483c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:28.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.770+0000 7f485e1c5700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f483c06b020 0x7f483c06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f48540090b0 tx=0x7f4854008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:28.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.774+0000 7f484effd700 1 -- 172.21.15.158:0/101662713 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4848053b50 con 0x7f48580cc590 2023-10-16T10:00:28.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.967+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f48580cd7a0 con 0x7f48580cc590 2023-10-16T10:00:28.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.970+0000 7f484effd700 1 -- 172.21.15.158:0/101662713 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4848017020 con 0x7f48580cc590 2023-10-16T10:00:28.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f483c06b020 msgr2=0x7f483c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f483c06b020 0x7f483c06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f48540090b0 tx=0x7f4854008030 comp rx=0 tx=0).stop 2023-10-16T10:00:28.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48580cc590 msgr2=0x7f48581e8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:28.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48580cc590 0x7f48581e8580 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f484800c500 tx=0x7f484800c8c0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 shutdown_connections 2023-10-16T10:00:28.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f483c06b020 0x7f483c06d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485815da40 0x7f48581e8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485806afe0 0x7f4858159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 --2- 172.21.15.158:0/101662713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48580cc590 0x7f48581e8580 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:28.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 >> 172.21.15.158:0/101662713 conn(0x7f4858099a10 msgr2=0x7f485815c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:28.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 shutdown_connections 2023-10-16T10:00:28.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:28.972+0000 7f4860429700 1 -- 172.21.15.158:0/101662713 wait complete. 2023-10-16T10:00:28.986 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:00:28.986 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:00:29.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:00:29.075 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:00:29.076 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:00:29.077 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:29.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:00:29.098 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:00:29.099 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:29.120 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:00:29.121 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:00:29.122 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:29.154 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:00:29.154 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:00:29.155 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:29.179 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:00:29.179 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T10:00:29.180 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:29.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.314+0000 7f09dfa20700 1 Processor -- start 2023-10-16T10:00:29.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.314+0000 7f09dfa20700 1 -- start start 2023-10-16T10:00:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09dfa20700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09d815f0f0 0x7f09d80b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09dfa20700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d815b930 con 0x7f09d80ce530 2023-10-16T10:00:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09dfa20700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d815f5f0 con 0x7f09d80cc1b0 2023-10-16T10:00:29.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09dfa20700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d80cf020 con 0x7f09d815f0f0 2023-10-16T10:00:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09ddfbd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09d815f0f0 0x7f09d80b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09dd7bc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f09d80ce530 0x7f09d80cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55118/0 (socket says 172.21.15.158:55118) 2023-10-16T10:00:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09dd7bc700 1 -- 172.21.15.158:0/2333170083 learned_addr learned my addr 172.21.15.158:0/2333170083 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09ddfbd700 1 -- 172.21.15.158:0/2333170083 >> v1:172.21.15.172:6789/0 conn(0x7f09d80cc1b0 legacy=0x7f09d80cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:29.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09ddfbd700 1 -- 172.21.15.158:0/2333170083 >> v1:172.21.15.158:6789/0 conn(0x7f09d80ce530 legacy=0x7f09d80cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09cffff700 1 -- 172.21.15.158:0/2333170083 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787247670 0 0) 0x7f09d815b930 con 0x7f09d80ce530 2023-10-16T10:00:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.315+0000 7f09ddfbd700 1 -- 172.21.15.158:0/2333170083 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d815f8d0 con 0x7f09d815f0f0 2023-10-16T10:00:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09ddfbd700 1 --2- 172.21.15.158:0/2333170083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d815f0f0 0x7f09d80b3b20 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009d90 tx=0x7f09d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85edf60cfbb41bae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09cffff700 1 -- 172.21.15.158:0/2333170083 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d000e070 con 0x7f09d815f0f0 2023-10-16T10:00:29.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09cffff700 1 -- 172.21.15.158:0/2333170083 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d000bb90 con 0x7f09d815f0f0 2023-10-16T10:00:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09cffff700 1 -- 172.21.15.158:0/2333170083 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0003a20 con 0x7f09d815f0f0 2023-10-16T10:00:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09dfa20700 1 -- 172.21.15.158:0/2333170083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d815f0f0 msgr2=0x7f09d80b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09dfa20700 1 --2- 172.21.15.158:0/2333170083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d815f0f0 0x7f09d80b3b20 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009d90 tx=0x7f09d000b330 comp rx=0 tx=0).stop 2023-10-16T10:00:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09dfa20700 1 -- 172.21.15.158:0/2333170083 shutdown_connections 2023-10-16T10:00:29.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09dfa20700 1 --2- 172.21.15.158:0/2333170083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d815f0f0 0x7f09d80b3b20 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09dfa20700 1 -- 172.21.15.158:0/2333170083 >> 172.21.15.158:0/2333170083 conn(0x7f09d806a8d0 msgr2=0x7f09d806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09dfa20700 1 -- 172.21.15.158:0/2333170083 shutdown_connections 2023-10-16T10:00:29.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.316+0000 7f09dfa20700 1 -- 172.21.15.158:0/2333170083 wait complete. 2023-10-16T10:00:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 Processor -- start 2023-10-16T10:00:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 -- start start 2023-10-16T10:00:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09d80cc1b0 0x7f09d80c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d80ce530 0x7f09d81f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09d815f0f0 0x7f09d80c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d80c9da0 con 0x7f09d80ce530 2023-10-16T10:00:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d80ca080 con 0x7f09d815f0f0 2023-10-16T10:00:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dfa20700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d80ca360 con 0x7f09d80cc1b0 2023-10-16T10:00:29.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dcfbb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d80ce530 0x7f09d81f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dcfbb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d80ce530 0x7f09d81f3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54950/0 (socket says 172.21.15.158:54950) 2023-10-16T10:00:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.318+0000 7f09dcfbb700 1 -- 172.21.15.158:0/4272137129 learned_addr learned my addr 172.21.15.158:0/4272137129 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dcfbb700 1 -- 172.21.15.158:0/4272137129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09d80cc1b0 msgr2=0x7f09d80c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dcfbb700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09d80cc1b0 0x7f09d80c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dcfbb700 1 -- 172.21.15.158:0/4272137129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09d815f0f0 msgr2=0x7f09d80c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09ddfbd700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09d815f0f0 0x7f09d80c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dcfbb700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09d815f0f0 0x7f09d80c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dcfbb700 1 -- 172.21.15.158:0/4272137129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d0009a70 con 0x7f09d80ce530 2023-10-16T10:00:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dcfbb700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d80ce530 0x7f09d81f3870 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f09d400b4a0 tx=0x7f09d400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:29.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09cdffb700 1 -- 172.21.15.158:0/4272137129 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4011830 con 0x7f09d80ce530 2023-10-16T10:00:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09cdffb700 1 -- 172.21.15.158:0/4272137129 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d4011990 con 0x7f09d80ce530 2023-10-16T10:00:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d81f3070 con 0x7f09d80ce530 2023-10-16T10:00:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09cdffb700 1 -- 172.21.15.158:0/4272137129 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4011d40 con 0x7f09d80ce530 2023-10-16T10:00:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.319+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d81f3db0 con 0x7f09d80ce530 2023-10-16T10:00:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.320+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d80b7460 con 0x7f09d80ce530 2023-10-16T10:00:29.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.321+0000 7f09cdffb700 1 -- 172.21.15.158:0/4272137129 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f09d401a400 con 0x7f09d80ce530 2023-10-16T10:00:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.321+0000 7f09cdffb700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f09bc06b020 0x7f09bc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.321+0000 7f09cdffb700 1 -- 172.21.15.158:0/4272137129 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f09d4089710 con 0x7f09d80ce530 2023-10-16T10:00:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.321+0000 7f09dd7bc700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f09bc06b020 0x7f09bc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.322+0000 7f09dd7bc700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f09bc06b020 0x7f09bc06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f09c80011e0 tx=0x7f09c80079b0 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:29.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.326+0000 7f09cdffb700 1 -- 172.21.15.158:0/4272137129 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09d40544a0 con 0x7f09d80ce530 2023-10-16T10:00:29.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.564+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d80681b0 con 0x7f09d80ce530 2023-10-16T10:00:29.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.565+0000 7f09cdffb700 1 -- 172.21.15.158:0/4272137129 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f09d4015e90 con 0x7f09d80ce530 2023-10-16T10:00:29.566 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:29.567 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6676,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2569267295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2569267295},{"type":"v1","addr":"172.21.15.172:6835","nonce":2569267295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1485896118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1485896118},{"type":"v1","addr":"172.21.15.172:6834","nonce":1485896118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2664141151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2664141151},{"type":"v1","addr":"172.21.15.158:6836","nonce":2664141151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":7626,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/374222597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":374222597},{"type":"v1","addr":"172.21.15.158:6837","nonce":374222597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[]} 2023-10-16T10:00:29.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f09bc06b020 msgr2=0x7f09bc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f09bc06b020 0x7f09bc06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f09c80011e0 tx=0x7f09c80079b0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d80ce530 msgr2=0x7f09d81f3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d80ce530 0x7f09d81f3870 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f09d400b4a0 tx=0x7f09d400b860 comp rx=0 tx=0).stop 2023-10-16T10:00:29.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 shutdown_connections 2023-10-16T10:00:29.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f09bc06b020 0x7f09bc06d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09d815f0f0 0x7f09d80c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09d80cc1b0 0x7f09d80c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 --2- 172.21.15.158:0/4272137129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09d80ce530 0x7f09d81f3870 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 >> 172.21.15.158:0/4272137129 conn(0x7f09d806a8d0 msgr2=0x7f09d80cba10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:29.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 shutdown_connections 2023-10-16T10:00:29.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.567+0000 7f09dfa20700 1 -- 172.21.15.158:0/4272137129 wait complete. 2023-10-16T10:00:29.571 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 44 2023-10-16T10:00:29.580 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T10:00:29.581 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T10:00:29.581 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T10:00:29.581 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:00:29.581 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T10:00:29.582 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T10:00:29.582 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T10:00:29.582 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:00:29.584 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T10:00:29.584 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T10:00:29.585 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T10:00:29.585 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:00:29.585 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T10:00:29.585 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T10:00:29.586 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T10:00:29.586 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:00:29.591 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T10:00:29.591 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T10:00:29.592 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T10:00:29.592 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T10:00:29.595 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T10:00:29.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T10:00:29.596 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T10:00:29.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T10:00:29.633 INFO:tasks.ceph.mds.a:Started 2023-10-16T10:00:29.634 DEBUG:teuthology.parallel:result is None 2023-10-16T10:00:29.635 INFO:tasks.ceph.mds.c:Started 2023-10-16T10:00:29.635 DEBUG:teuthology.parallel:result is None 2023-10-16T10:00:29.637 INFO:tasks.ceph.mds.b:Started 2023-10-16T10:00:29.638 DEBUG:teuthology.parallel:result is None 2023-10-16T10:00:29.639 INFO:tasks.ceph.mds.d:Started 2023-10-16T10:00:29.639 DEBUG:teuthology.parallel:result is None 2023-10-16T10:00:29.639 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T10:00:29.714 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T10:00:29.714 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T10:00:29.723 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T10:00:29.724 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T10:00:29.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.766+0000 7f94ece4b700 1 Processor -- start 2023-10-16T10:00:29.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.766+0000 7f94ece4b700 1 -- start start 2023-10-16T10:00:29.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94ece4b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94e815d870 0x7f94e806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94ece4b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f94e80b3290 0x7f94e806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94ece4b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e815dc50 con 0x7f94e80cf7c0 2023-10-16T10:00:29.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94ece4b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e80701a0 con 0x7f94e80b3290 2023-10-16T10:00:29.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94ece4b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e806f100 con 0x7f94e815d870 2023-10-16T10:00:29.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94e5d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f94e80cf7c0 0x7f94e8161390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55164/0 (socket says 172.21.15.158:55164) 2023-10-16T10:00:29.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94e6d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f94e80b3290 0x7f94e806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.767+0000 7f94e5d9b700 1 -- 172.21.15.158:0/2860184560 learned_addr learned my addr 172.21.15.158:0/2860184560 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:29.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e659c700 1 --2- 172.21.15.158:0/2860184560 >> v2:172.21.15.172:3300/0 conn(0x7f94e815d870 0x7f94e806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e6d9d700 1 -- 172.21.15.158:0/2860184560 >> v2:172.21.15.172:3300/0 conn(0x7f94e815d870 msgr2=0x7f94e806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:29.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e559a700 1 -- 172.21.15.158:0/2860184560 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700526924 0 0) 0x7f94e815dc50 con 0x7f94e80cf7c0 2023-10-16T10:00:29.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e6d9d700 1 --2- 172.21.15.158:0/2860184560 >> v2:172.21.15.172:3300/0 conn(0x7f94e815d870 0x7f94e806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e6d9d700 1 -- 172.21.15.158:0/2860184560 >> v1:172.21.15.158:6789/0 conn(0x7f94e80cf7c0 legacy=0x7f94e8161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e6d9d700 1 -- 172.21.15.158:0/2860184560 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e806f3e0 con 0x7f94e80b3290 2023-10-16T10:00:29.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e6d9d700 1 --2- 172.21.15.158:0/2860184560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e806ae10 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f94e806d130 tx=0x7f94d8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1198b58a3d6e8b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:29.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e559a700 1 -- 172.21.15.158:0/2860184560 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d8011830 con 0x7f94e80b3290 2023-10-16T10:00:29.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e559a700 1 -- 172.21.15.158:0/2860184560 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d8011990 con 0x7f94e80b3290 2023-10-16T10:00:29.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94e559a700 1 -- 172.21.15.158:0/2860184560 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d8011d20 con 0x7f94e80b3290 2023-10-16T10:00:29.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94ece4b700 1 -- 172.21.15.158:0/2860184560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 msgr2=0x7f94e806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.768+0000 7f94ece4b700 1 --2- 172.21.15.158:0/2860184560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e806ae10 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f94e806d130 tx=0x7f94d8009b20 comp rx=0 tx=0).stop 2023-10-16T10:00:29.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.769+0000 7f94ece4b700 1 -- 172.21.15.158:0/2860184560 shutdown_connections 2023-10-16T10:00:29.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.769+0000 7f94ece4b700 1 --2- 172.21.15.158:0/2860184560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e806ae10 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.769+0000 7f94ece4b700 1 --2- 172.21.15.158:0/2860184560 >> v2:172.21.15.172:3300/0 conn(0x7f94e815d870 0x7f94e806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.769+0000 7f94ece4b700 1 -- 172.21.15.158:0/2860184560 >> 172.21.15.158:0/2860184560 conn(0x7f94e806da00 msgr2=0x7f94e806dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:29.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.769+0000 7f94ece4b700 1 -- 172.21.15.158:0/2860184560 shutdown_connections 2023-10-16T10:00:29.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.769+0000 7f94ece4b700 1 -- 172.21.15.158:0/2860184560 wait complete. 2023-10-16T10:00:29.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.770+0000 7f94ece4b700 1 Processor -- start 2023-10-16T10:00:29.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.770+0000 7f94ece4b700 1 -- start start 2023-10-16T10:00:29.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e814d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e80cf7c0 0x7f94e8150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94e815d870 0x7f94e8071e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8151c60 con 0x7f94e80b3290 2023-10-16T10:00:29.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8150fe0 con 0x7f94e80cf7c0 2023-10-16T10:00:29.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81512c0 con 0x7f94e815d870 2023-10-16T10:00:29.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e814d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e814d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:54992/0 (socket says 172.21.15.158:54992) 2023-10-16T10:00:29.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 -- 172.21.15.158:0/896756614 learned_addr learned my addr 172.21.15.158:0/896756614 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:29.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e6d9d700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94e815d870 0x7f94e8071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e5d9b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e80cf7c0 0x7f94e8150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 -- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94e815d870 msgr2=0x7f94e8071e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94e815d870 0x7f94e8071e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 -- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e80cf7c0 msgr2=0x7f94e8150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:29.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e80cf7c0 0x7f94e8150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:29.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 -- 172.21.15.158:0/896756614 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80094a0 con 0x7f94e80b3290 2023-10-16T10:00:29.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94e659c700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e814d770 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f94e0007e40 tx=0x7f94e000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:29.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94d6ffd700 1 -- 172.21.15.158:0/896756614 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e000ed30 con 0x7f94e80b3290 2023-10-16T10:00:29.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94d6ffd700 1 -- 172.21.15.158:0/896756614 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e000ee90 con 0x7f94e80b3290 2023-10-16T10:00:29.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e81515a0 con 0x7f94e80b3290 2023-10-16T10:00:29.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94d6ffd700 1 -- 172.21.15.158:0/896756614 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0014660 con 0x7f94e80b3290 2023-10-16T10:00:29.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.771+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e814e1f0 con 0x7f94e80b3290 2023-10-16T10:00:29.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.772+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e814e3b0 con 0x7f94e80b3290 2023-10-16T10:00:29.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.773+0000 7f94d6ffd700 1 -- 172.21.15.158:0/896756614 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f94e00147c0 con 0x7f94e80b3290 2023-10-16T10:00:29.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.773+0000 7f94d6ffd700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f94c806b020 0x7f94c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:29.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.773+0000 7f94d6ffd700 1 -- 172.21.15.158:0/896756614 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f94e0089f60 con 0x7f94e80b3290 2023-10-16T10:00:29.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.773+0000 7f94e5d9b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f94c806b020 0x7f94c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:29.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.774+0000 7f94e5d9b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f94c806b020 0x7f94c806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f94e81503c0 tx=0x7f94dc007de0 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:29.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.778+0000 7f94d6ffd700 1 -- 172.21.15.158:0/896756614 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94e0054cf0 con 0x7f94e80b3290 2023-10-16T10:00:29.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:29.982+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f94e80681b0 con 0x7f94e80b3290 2023-10-16T10:00:30.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.591+0000 7f94d6ffd700 1 -- 172.21.15.158:0/896756614 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v105) v1 ==== 97+0+0 (secure 0 0 0) 0x7f94e0058310 con 0x7f94e80b3290 2023-10-16T10:00:30.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f94c806b020 msgr2=0x7f94c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:30.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f94c806b020 0x7f94c806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f94e81503c0 tx=0x7f94dc007de0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 msgr2=0x7f94e814d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:30.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e814d770 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f94e0007e40 tx=0x7f94e000e3c0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 shutdown_connections 2023-10-16T10:00:30.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f94c806b020 0x7f94c806d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94e815d870 0x7f94e8071e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94e80b3290 0x7f94e814d770 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 --2- 172.21.15.158:0/896756614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94e80cf7c0 0x7f94e8150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.594+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 >> 172.21.15.158:0/896756614 conn(0x7f94e806da00 msgr2=0x7f94e80d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:30.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.595+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 shutdown_connections 2023-10-16T10:00:30.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.595+0000 7f94ece4b700 1 -- 172.21.15.158:0/896756614 wait complete. 2023-10-16T10:00:30.599 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T10:00:30.610 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T10:00:30.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.731+0000 7fb8f776a700 1 Processor -- start 2023-10-16T10:00:30.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f776a700 1 -- start start 2023-10-16T10:00:30.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f776a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8f00ce530 0x7fb8f00b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f776a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 0x7fb8f00cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f776a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f015b930 con 0x7fb8f015f0f0 2023-10-16T10:00:30.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f776a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f00cf020 con 0x7fb8f00ce530 2023-10-16T10:00:30.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f776a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f015f5f0 con 0x7fb8f00cc1b0 2023-10-16T10:00:30.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f4d05700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb8f015f0f0 0x7fb8f00cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55178/0 (socket says 172.21.15.158:55178) 2023-10-16T10:00:30.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.732+0000 7fb8f4d05700 1 -- 172.21.15.158:0/3968539225 learned_addr learned my addr 172.21.15.158:0/3968539225 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:30.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684309546 0 0) 0x7fb8f015b930 con 0x7fb8f015f0f0 2023-10-16T10:00:30.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8dc0035b0 con 0x7fb8f015f0f0 2023-10-16T10:00:30.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8f5d07700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 0x7fb8f00cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8f5506700 1 --2- 172.21.15.158:0/3968539225 >> v2:172.21.15.172:3300/0 conn(0x7fb8f00ce530 0x7fb8f00b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2378553011 0 0) 0x7fb8dc0035b0 con 0x7fb8f015f0f0 2023-10-16T10:00:30.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 msgr2=0x7fb8f00cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:30.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 0x7fb8f00cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 >> v2:172.21.15.172:3300/0 conn(0x7fb8f00ce530 msgr2=0x7fb8f00b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:30.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 --2- 172.21.15.158:0/3968539225 >> v2:172.21.15.172:3300/0 conn(0x7fb8f00ce530 0x7fb8f00b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f015f8d0 con 0x7fb8f015f0f0 2023-10-16T10:00:30.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fb8ec001670 con 0x7fb8f015f0f0 2023-10-16T10:00:30.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8f5d07700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 0x7fb8f00cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:30.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 >> v1:172.21.15.158:6789/0 conn(0x7fb8f015f0f0 legacy=0x7fb8f00cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:30.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 0x7fb8dc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc003ff0 0x7fb8dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc004d90 0x7fb8dc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00cf020 con 0x7fb8dc004d90 2023-10-16T10:00:30.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f015f5f0 con 0x7fb8dc003ff0 2023-10-16T10:00:30.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc00baf0 con 0x7fb8f00cc1b0 2023-10-16T10:00:30.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8f5506700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 0x7fb8dc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8f5d07700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc004d90 0x7fb8dc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.733+0000 7fb8f4d05700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc003ff0 0x7fb8dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8f5d07700 1 -- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 msgr2=0x7fb8dc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:30.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8f5d07700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f00cc1b0 0x7fb8dc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8f5d07700 1 -- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc003ff0 msgr2=0x7fb8dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:30.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8f5d07700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8dc003ff0 0x7fb8dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8f5d07700 1 -- 172.21.15.158:0/3968539225 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc0035b0 con 0x7fb8dc004d90 2023-10-16T10:00:30.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8f5d07700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc004d90 0x7fb8dc005210 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00d2550 tx=0x7fb8dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19915b58c5fe082 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:30.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8007e70 con 0x7fb8dc004d90 2023-10-16T10:00:30.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e8003a20 con 0x7fb8dc004d90 2023-10-16T10:00:30.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.734+0000 7fb8e7fff700 1 -- 172.21.15.158:0/3968539225 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8003db0 con 0x7fb8dc004d90 2023-10-16T10:00:30.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.735+0000 7fb8f776a700 1 -- 172.21.15.158:0/3968539225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc004d90 msgr2=0x7fb8dc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:30.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.735+0000 7fb8f776a700 1 --2- 172.21.15.158:0/3968539225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8dc004d90 0x7fb8dc005210 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00d2550 tx=0x7fb8dc005880 comp rx=0 tx=0).stop 2023-10-16T10:00:30.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.735+0000 7fb8f5d07700 1 -- 172.21.15.158:0/3968539225 reap_dead start 2023-10-16T10:00:30.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.735+0000 7fb8f776a700 1 -- 172.21.15.158:0/3968539225 shutdown_connections 2023-10-16T10:00:30.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.735+0000 7fb8f776a700 1 -- 172.21.15.158:0/3968539225 >> 172.21.15.158:0/3968539225 conn(0x7fb8f006a8d0 msgr2=0x7fb8f006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:30.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.736+0000 7fb8f776a700 1 -- 172.21.15.158:0/3968539225 shutdown_connections 2023-10-16T10:00:30.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.736+0000 7fb8f776a700 1 -- 172.21.15.158:0/3968539225 wait complete. 2023-10-16T10:00:30.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.737+0000 7fb8f776a700 1 Processor -- start 2023-10-16T10:00:30.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.737+0000 7fb8f776a700 1 -- start start 2023-10-16T10:00:30.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f776a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f0157f70 0x7fb8f00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f776a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8f00cb2e0 0x7fb8f015d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f776a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8f00b20a0 0x7fb8f015d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f776a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00b45f0 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f776a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00cb780 con 0x7fb8f00b20a0 2023-10-16T10:00:30.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f776a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00b3c50 con 0x7fb8f0157f70 2023-10-16T10:00:30.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8f00cb2e0 0x7fb8f015d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8f00cb2e0 0x7fb8f015d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55014/0 (socket says 172.21.15.158:55014) 2023-10-16T10:00:30.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 -- 172.21.15.158:0/1513782456 learned_addr learned my addr 172.21.15.158:0/1513782456 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f5506700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f0157f70 0x7fb8f00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f5d07700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8f00b20a0 0x7fb8f015d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 -- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f0157f70 msgr2=0x7fb8f00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f0157f70 0x7fb8f00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 -- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8f00b20a0 msgr2=0x7fb8f015d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8f00b20a0 0x7fb8f015d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 -- 172.21.15.158:0/1513782456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f00b3f30 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8f4d05700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8f00cb2e0 0x7fb8f015d280 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0048d0 tx=0x7fb8ec0075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.738+0000 7fb8e5ffb700 1 -- 172.21.15.158:0/1513782456 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec00f6e0 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.739+0000 7fb8e5ffb700 1 -- 172.21.15.158:0/1513782456 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec00f860 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.739+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f00b4210 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.739+0000 7fb8e5ffb700 1 -- 172.21.15.158:0/1513782456 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec00fc10 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.739+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f0158530 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.740+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f00681b0 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.740+0000 7fb8e5ffb700 1 -- 172.21.15.158:0/1513782456 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb8ec00fd70 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.740+0000 7fb8e5ffb700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fb8d406b020 0x7fb8d406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:30.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.741+0000 7fb8e5ffb700 1 -- 172.21.15.158:0/1513782456 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5601+0+0 (secure 0 0 0) 0x7fb8ec0896e0 con 0x7fb8f00cb2e0 2023-10-16T10:00:30.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.741+0000 7fb8f5506700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fb8d406b020 0x7fb8d406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:30.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.741+0000 7fb8f5506700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fb8d406b020 0x7fb8d406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009a80 tx=0x7fb8e0008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:30.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:30.745+0000 7fb8e5ffb700 1 -- 172.21.15.158:0/1513782456 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8ec054770 con 0x7fb8f00cb2e0 2023-10-16T10:00:31.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.015+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb8f00b28c0 con 0x7fb8f00cb2e0 2023-10-16T10:00:31.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.015+0000 7fb8e5ffb700 1 -- 172.21.15.158:0/1513782456 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb8ec057d90 con 0x7fb8f00cb2e0 2023-10-16T10:00:31.017 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:31.017 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:00:31.018 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T10:00:31.018 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.018 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T10:00:31.018 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T10:00:31.019 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.019 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.019 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:00:31.019 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:00:31.019 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T10:00:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T10:00:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:00:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:00:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.022 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.022 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T10:00:31.022 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T10:00:31.022 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.022 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:00:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:00:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.024 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T10:00:31.024 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T10:00:31.024 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.024 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.024 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:00:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:00:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.026 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T10:00:31.026 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T10:00:31.026 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.026 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.026 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T10:00:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T10:00:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.029 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.029 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T10:00:31.029 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T10:00:31.029 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.029 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.031 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T10:00:31.031 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T10:00:31.031 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.031 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.031 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T10:00:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T10:00:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T10:00:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T10:00:31.035 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.035 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.035 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.035 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.035 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T10:00:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T10:00:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.038 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T10:00:31.038 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T10:00:31.038 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.038 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:00:31.038 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:00:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T10:00:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T10:00:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T10:00:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T10:00:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T10:00:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T10:00:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:00:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:00:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:00:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T10:00:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T10:00:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:00:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:00:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:00:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T10:00:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T10:00:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:00:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:00:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:00:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:00:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T10:00:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T10:00:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:00:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:00:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:00:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T10:00:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T10:00:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:00:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:00:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:00:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T10:00:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T10:00:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:00:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:00:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:00:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:00:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T10:00:31.053 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:00:31.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fb8d406b020 msgr2=0x7fb8d406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fb8d406b020 0x7fb8d406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009a80 tx=0x7fb8e0008040 comp rx=0 tx=0).stop 2023-10-16T10:00:31.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8f00cb2e0 msgr2=0x7fb8f015d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8f00cb2e0 0x7fb8f015d280 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0048d0 tx=0x7fb8ec0075f0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 shutdown_connections 2023-10-16T10:00:31.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fb8d406b020 0x7fb8d406d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8f00b20a0 0x7fb8f015d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8f0157f70 0x7fb8f00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 --2- 172.21.15.158:0/1513782456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8f00cb2e0 0x7fb8f015d280 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 >> 172.21.15.158:0/1513782456 conn(0x7fb8f006a8d0 msgr2=0x7fb8f00cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:31.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 shutdown_connections 2023-10-16T10:00:31.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.018+0000 7fb8f776a700 1 -- 172.21.15.158:0/1513782456 wait complete. 2023-10-16T10:00:31.057 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:31.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.177+0000 7f57edf16700 1 Processor -- start 2023-10-16T10:00:31.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.177+0000 7f57edf16700 1 -- start start 2023-10-16T10:00:31.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57edf16700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f57e806ab40 0x7f57e806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57edf16700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57edf16700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e8071e50 con 0x7f57e806af20 2023-10-16T10:00:31.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57edf16700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e8161270 con 0x7f57e806ab40 2023-10-16T10:00:31.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57edf16700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e8161510 con 0x7f57e806e5d0 2023-10-16T10:00:31.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57e77fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f57e806ab40 0x7f57e806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57e7fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f57e806af20 0x7f57e80722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:59104/0 (socket says 172.21.15.158:59104) 2023-10-16T10:00:31.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57e6ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.178+0000 7f57e7fff700 1 -- 172.21.15.158:0/966006809 learned_addr learned my addr 172.21.15.158:0/966006809 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:31.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e77fe700 1 -- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 msgr2=0x7f57e8071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:31.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e77fe700 1 --2- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e8071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e77fe700 1 -- 172.21.15.158:0/966006809 >> v1:172.21.15.172:6789/0 conn(0x7f57e806af20 legacy=0x7f57e80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e77fe700 1 -- 172.21.15.158:0/966006809 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e81617f0 con 0x7f57e806ab40 2023-10-16T10:00:31.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e6ffd700 1 --2- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e8071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:31.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e77fe700 1 --2- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806ab40 0x7f57e806d820 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009840 tx=0x7f57dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e39e8567ea1a4208 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e67fc700 1 -- 172.21.15.158:0/966006809 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc011830 con 0x7f57e806ab40 2023-10-16T10:00:31.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e67fc700 1 -- 172.21.15.158:0/966006809 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc011990 con 0x7f57e806ab40 2023-10-16T10:00:31.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.179+0000 7f57e67fc700 1 -- 172.21.15.158:0/966006809 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc011d20 con 0x7f57e806ab40 2023-10-16T10:00:31.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 -- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806ab40 msgr2=0x7f57e806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 --2- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806ab40 0x7f57e806d820 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009840 tx=0x7f57dc009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:31.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 -- 172.21.15.158:0/966006809 shutdown_connections 2023-10-16T10:00:31.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 --2- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806ab40 0x7f57e806d820 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 --2- 172.21.15.158:0/966006809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e8071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 -- 172.21.15.158:0/966006809 >> 172.21.15.158:0/966006809 conn(0x7f57e80d02b0 msgr2=0x7f57e80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:31.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 -- 172.21.15.158:0/966006809 shutdown_connections 2023-10-16T10:00:31.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.180+0000 7f57edf16700 1 -- 172.21.15.158:0/966006809 wait complete. 2023-10-16T10:00:31.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.181+0000 7f57edf16700 1 Processor -- start 2023-10-16T10:00:31.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.181+0000 7f57edf16700 1 -- start start 2023-10-16T10:00:31.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57edf16700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806ab40 0x7f57e80c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57edf16700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806af20 0x7f57e80c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57edf16700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e81f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57edf16700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e81f3e90 con 0x7f57e806ab40 2023-10-16T10:00:31.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57edf16700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e80c9250 con 0x7f57e806af20 2023-10-16T10:00:31.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57edf16700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e80c94d0 con 0x7f57e806e5d0 2023-10-16T10:00:31.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806ab40 0x7f57e80c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806ab40 0x7f57e80c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55026/0 (socket says 172.21.15.158:55026) 2023-10-16T10:00:31.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 -- 172.21.15.158:0/56480801 learned_addr learned my addr 172.21.15.158:0/56480801 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:31.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e6ffd700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806af20 0x7f57e80c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e7fff700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e81f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 -- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 msgr2=0x7f57e81f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e81f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 -- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806af20 msgr2=0x7f57e80c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806af20 0x7f57e80c78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 -- 172.21.15.158:0/56480801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d80094a0 con 0x7f57e806ab40 2023-10-16T10:00:31.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e7fff700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e81f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:31.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.182+0000 7f57e77fe700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806ab40 0x7f57e80c7380 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009fd0 tx=0x7f57dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.183+0000 7f57cffff700 1 -- 172.21.15.158:0/56480801 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0036a0 con 0x7f57e806ab40 2023-10-16T10:00:31.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.183+0000 7f57cffff700 1 -- 172.21.15.158:0/56480801 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc01f070 con 0x7f57e806ab40 2023-10-16T10:00:31.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.183+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57dc0094a0 con 0x7f57e806ab40 2023-10-16T10:00:31.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.183+0000 7f57cffff700 1 -- 172.21.15.158:0/56480801 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc003a60 con 0x7f57e806ab40 2023-10-16T10:00:31.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.183+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e80c9a90 con 0x7f57e806ab40 2023-10-16T10:00:31.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.184+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e80681b0 con 0x7f57e806ab40 2023-10-16T10:00:31.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.185+0000 7f57cffff700 1 -- 172.21.15.158:0/56480801 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f57dc003bc0 con 0x7f57e806ab40 2023-10-16T10:00:31.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.185+0000 7f57cffff700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57d406b020 0x7f57d406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.185+0000 7f57cffff700 1 -- 172.21.15.158:0/56480801 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f57dc021070 con 0x7f57e806ab40 2023-10-16T10:00:31.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.185+0000 7f57e6ffd700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57d406b020 0x7f57d406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.185+0000 7f57e6ffd700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57d406b020 0x7f57d406d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57e8152e20 tx=0x7f57d8008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.189+0000 7f57cffff700 1 -- 172.21.15.158:0/56480801 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f57dc056470 con 0x7f57e806ab40 2023-10-16T10:00:31.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.383+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57e80b5a70 con 0x7f57e806ab40 2023-10-16T10:00:31.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.384+0000 7f57cffff700 1 -- 172.21.15.158:0/56480801 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+13721 (secure 0 0 0) 0x7f57dc059a90 con 0x7f57e806ab40 2023-10-16T10:00:31.385 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:31.386 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":107,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:30.598810+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:31.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57d406b020 msgr2=0x7f57d406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57d406b020 0x7f57d406d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57e8152e20 tx=0x7f57d8008040 comp rx=0 tx=0).stop 2023-10-16T10:00:31.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806ab40 msgr2=0x7f57e80c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806ab40 0x7f57e80c7380 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009fd0 tx=0x7f57dc004750 comp rx=0 tx=0).stop 2023-10-16T10:00:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 shutdown_connections 2023-10-16T10:00:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57d406b020 0x7f57d406d4e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806e5d0 0x7f57e81f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806ab40 0x7f57e80c7380 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 --2- 172.21.15.158:0/56480801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e806af20 0x7f57e80c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 >> 172.21.15.158:0/56480801 conn(0x7f57e80d02b0 msgr2=0x7f57e8072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 shutdown_connections 2023-10-16T10:00:31.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.386+0000 7f57edf16700 1 -- 172.21.15.158:0/56480801 wait complete. 2023-10-16T10:00:31.400 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:00:31.401 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:00:31.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.523+0000 7f565f419700 1 Processor -- start 2023-10-16T10:00:31.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.523+0000 7f565f419700 1 -- start start 2023-10-16T10:00:31.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565f419700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815b590 0x7f56580ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565f419700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f565815f1d0 0x7f56580b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565f419700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580cf4c0 con 0x7f56580cbaf0 2023-10-16T10:00:31.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565f419700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565815f970 con 0x7f565815f1d0 2023-10-16T10:00:31.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565f419700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565815fc50 con 0x7f565815b590 2023-10-16T10:00:31.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f565815f1d0 0x7f56580b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f565815f1d0 0x7f56580b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55034/0 (socket says 172.21.15.158:55034) 2023-10-16T10:00:31.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 -- 172.21.15.158:0/1909404347 learned_addr learned my addr 172.21.15.158:0/1909404347 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:31.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565d1b5700 1 --2- 172.21.15.158:0/1909404347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815b590 0x7f56580ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 -- 172.21.15.158:0/1909404347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815b590 msgr2=0x7f56580ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:31.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 --2- 172.21.15.158:0/1909404347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815b590 0x7f56580ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 -- 172.21.15.158:0/1909404347 >> v1:172.21.15.172:6789/0 conn(0x7f56580cbaf0 legacy=0x7f56580cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 -- 172.21.15.158:0/1909404347 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565815ff30 con 0x7f565815f1d0 2023-10-16T10:00:31.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565d1b5700 1 --2- 172.21.15.158:0/1909404347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815b590 0x7f56580ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:31.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f565c9b4700 1 --2- 172.21.15.158:0/1909404347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815f1d0 0x7f56580b42d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f56480098f0 tx=0x7f5648009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed6ce52d5ad84df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f564ffff700 1 -- 172.21.15.158:0/1909404347 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564800b8a0 con 0x7f565815f1d0 2023-10-16T10:00:31.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.524+0000 7f564ffff700 1 -- 172.21.15.158:0/1909404347 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564800ba00 con 0x7f565815f1d0 2023-10-16T10:00:31.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f564ffff700 1 -- 172.21.15.158:0/1909404347 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564800bd90 con 0x7f565815f1d0 2023-10-16T10:00:31.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 -- 172.21.15.158:0/1909404347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815f1d0 msgr2=0x7f56580b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 --2- 172.21.15.158:0/1909404347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815f1d0 0x7f56580b42d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f56480098f0 tx=0x7f5648009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:31.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 -- 172.21.15.158:0/1909404347 shutdown_connections 2023-10-16T10:00:31.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 --2- 172.21.15.158:0/1909404347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815b590 0x7f56580ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 --2- 172.21.15.158:0/1909404347 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815f1d0 0x7f56580b42d0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 -- 172.21.15.158:0/1909404347 >> 172.21.15.158:0/1909404347 conn(0x7f565806a8d0 msgr2=0x7f565806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:31.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 -- 172.21.15.158:0/1909404347 shutdown_connections 2023-10-16T10:00:31.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.525+0000 7f565f419700 1 -- 172.21.15.158:0/1909404347 wait complete. 2023-10-16T10:00:31.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.526+0000 7f565f419700 1 Processor -- start 2023-10-16T10:00:31.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.526+0000 7f565f419700 1 -- start start 2023-10-16T10:00:31.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565f419700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56580cbaf0 0x7f56581f4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565f419700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815b590 0x7f565815cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565f419700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815f1d0 0x7f565815d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565f419700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565815e2c0 con 0x7f565815b590 2023-10-16T10:00:31.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565f419700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565815d6b0 con 0x7f56580cbaf0 2023-10-16T10:00:31.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565f419700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565815d990 con 0x7f565815f1d0 2023-10-16T10:00:31.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815b590 0x7f565815cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815b590 0x7f565815cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55038/0 (socket says 172.21.15.158:55038) 2023-10-16T10:00:31.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 -- 172.21.15.158:0/1130209416 learned_addr learned my addr 172.21.15.158:0/1130209416 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:31.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 -- 172.21.15.158:0/1130209416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815f1d0 msgr2=0x7f565815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815f1d0 0x7f565815d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565d1b5700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56580cbaf0 0x7f56581f4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 -- 172.21.15.158:0/1130209416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56580cbaf0 msgr2=0x7f56581f4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56580cbaf0 0x7f56581f4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.527+0000 7f565c9b4700 1 -- 172.21.15.158:0/1130209416 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5648009550 con 0x7f565815b590 2023-10-16T10:00:31.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.528+0000 7f565c9b4700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815b590 0x7f565815cb40 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f5648009c40 tx=0x7f5648004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.528+0000 7f564dffb700 1 -- 172.21.15.158:0/1130209416 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56480036a0 con 0x7f565815b590 2023-10-16T10:00:31.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.528+0000 7f564dffb700 1 -- 172.21.15.158:0/1130209416 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564801e070 con 0x7f565815b590 2023-10-16T10:00:31.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.528+0000 7f564dffb700 1 -- 172.21.15.158:0/1130209416 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648003a60 con 0x7f565815b590 2023-10-16T10:00:31.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.528+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565815dc70 con 0x7f565815b590 2023-10-16T10:00:31.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.528+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5658157f90 con 0x7f565815b590 2023-10-16T10:00:31.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.529+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565815e470 con 0x7f565815b590 2023-10-16T10:00:31.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.529+0000 7f564dffb700 1 -- 172.21.15.158:0/1130209416 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5648003bc0 con 0x7f565815b590 2023-10-16T10:00:31.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.530+0000 7f564dffb700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f563c06b020 0x7f563c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.530+0000 7f564dffb700 1 -- 172.21.15.158:0/1130209416 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f5648020070 con 0x7f565815b590 2023-10-16T10:00:31.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.530+0000 7f565d1b5700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f563c06b020 0x7f563c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.530+0000 7f565d1b5700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f563c06b020 0x7f563c06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5654009930 tx=0x7f5654008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.534+0000 7f564dffb700 1 -- 172.21.15.158:0/1130209416 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5648055b60 con 0x7f565815b590 2023-10-16T10:00:31.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.732+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f56580681b0 con 0x7f565815b590 2023-10-16T10:00:31.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.732+0000 7f564dffb700 1 -- 172.21.15.158:0/1130209416 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f564802a0a0 con 0x7f565815b590 2023-10-16T10:00:31.734 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:00:31.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.734+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f563c06b020 msgr2=0x7f563c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.734+0000 7f565f419700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f563c06b020 0x7f563c06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5654009930 tx=0x7f5654008040 comp rx=0 tx=0).stop 2023-10-16T10:00:31.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815b590 msgr2=0x7f565815cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815b590 0x7f565815cb40 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f5648009c40 tx=0x7f5648004750 comp rx=0 tx=0).stop 2023-10-16T10:00:31.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 shutdown_connections 2023-10-16T10:00:31.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f563c06b020 0x7f563c06d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f565815f1d0 0x7f565815d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56580cbaf0 0x7f56581f4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 --2- 172.21.15.158:0/1130209416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f565815b590 0x7f565815cb40 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 >> 172.21.15.158:0/1130209416 conn(0x7f565806a8d0 msgr2=0x7f56580d1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:31.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 shutdown_connections 2023-10-16T10:00:31.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.735+0000 7f565f419700 1 -- 172.21.15.158:0/1130209416 wait complete. 2023-10-16T10:00:31.749 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T10:00:31.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.868+0000 7fc074c82700 1 Processor -- start 2023-10-16T10:00:31.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.868+0000 7fc074c82700 1 -- start start 2023-10-16T10:00:31.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc074c82700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc07006ab40 0x7fc07006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc074c82700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc070161270 con 0x7fc07006e5d0 2023-10-16T10:00:31.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc074c82700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc070161510 con 0x7fc0700729b0 2023-10-16T10:00:31.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc074c82700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0701617f0 con 0x7fc07006ab40 2023-10-16T10:00:31.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06dd9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc07006e5d0 0x7fc0700722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55226/0 (socket says 172.21.15.158:55226) 2023-10-16T10:00:31.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06dd9b700 1 -- 172.21.15.158:0/3531876865 learned_addr learned my addr 172.21.15.158:0/3531876865 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:31.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06e59c700 1 --2- 172.21.15.158:0/3531876865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc07006ab40 0x7fc07006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06d59a700 1 -- 172.21.15.158:0/3531876865 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531617890 0 0) 0x7fc070161270 con 0x7fc07006e5d0 2023-10-16T10:00:31.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06d59a700 1 -- 172.21.15.158:0/3531876865 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0580035b0 con 0x7fc07006e5d0 2023-10-16T10:00:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06d59a700 1 -- 172.21.15.158:0/3531876865 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505026412 0 0) 0x7fc070161510 con 0x7fc0700729b0 2023-10-16T10:00:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06d59a700 1 -- 172.21.15.158:0/3531876865 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc070161270 con 0x7fc0700729b0 2023-10-16T10:00:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06e59c700 1 -- 172.21.15.158:0/3531876865 >> v1:172.21.15.172:6789/0 conn(0x7fc0700729b0 legacy=0x7fc07015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06e59c700 1 -- 172.21.15.158:0/3531876865 >> v1:172.21.15.158:6789/0 conn(0x7fc07006e5d0 legacy=0x7fc0700722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.869+0000 7fc06e59c700 1 -- 172.21.15.158:0/3531876865 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc070161ad0 con 0x7fc07006ab40 2023-10-16T10:00:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc06e59c700 1 --2- 172.21.15.158:0/3531876865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc07006ab40 0x7fc07006d820 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc060009870 tx=0x7fc060009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a37c5bc1e725f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc06d59a700 1 -- 172.21.15.158:0/3531876865 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060011830 con 0x7fc07006ab40 2023-10-16T10:00:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc06d59a700 1 -- 172.21.15.158:0/3531876865 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc060011990 con 0x7fc07006ab40 2023-10-16T10:00:31.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc06d59a700 1 -- 172.21.15.158:0/3531876865 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060011d00 con 0x7fc07006ab40 2023-10-16T10:00:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc074c82700 1 -- 172.21.15.158:0/3531876865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc07006ab40 msgr2=0x7fc07006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc074c82700 1 --2- 172.21.15.158:0/3531876865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc07006ab40 0x7fc07006d820 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc060009870 tx=0x7fc060009b80 comp rx=0 tx=0).stop 2023-10-16T10:00:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc074c82700 1 -- 172.21.15.158:0/3531876865 shutdown_connections 2023-10-16T10:00:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc074c82700 1 --2- 172.21.15.158:0/3531876865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc07006ab40 0x7fc07006d820 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc074c82700 1 -- 172.21.15.158:0/3531876865 >> 172.21.15.158:0/3531876865 conn(0x7fc0700d02b0 msgr2=0x7fc0700d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc074c82700 1 -- 172.21.15.158:0/3531876865 shutdown_connections 2023-10-16T10:00:31.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.870+0000 7fc074c82700 1 -- 172.21.15.158:0/3531876865 wait complete. 2023-10-16T10:00:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 Processor -- start 2023-10-16T10:00:31.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 -- start start 2023-10-16T10:00:31.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc07006ab40 0x7fc0701f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc07006e5d0 0x7fc0701f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0700729b0 0x7fc0701f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0701f6910 con 0x7fc07006e5d0 2023-10-16T10:00:31.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0701f62e0 con 0x7fc07006ab40 2023-10-16T10:00:31.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.872+0000 7fc074c82700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0701f3820 con 0x7fc0700729b0 2023-10-16T10:00:31.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc07006e5d0 0x7fc0701f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc07006e5d0 0x7fc0701f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55056/0 (socket says 172.21.15.158:55056) 2023-10-16T10:00:31.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 -- 172.21.15.158:0/2142924420 learned_addr learned my addr 172.21.15.158:0/2142924420 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:31.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06ed9d700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0700729b0 0x7fc0701f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06e59c700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc07006ab40 0x7fc0701f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 -- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0700729b0 msgr2=0x7fc0701f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0700729b0 0x7fc0701f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 -- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc07006ab40 msgr2=0x7fc0701f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc07006ab40 0x7fc0701f6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 -- 172.21.15.158:0/2142924420 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0600094a0 con 0x7fc07006e5d0 2023-10-16T10:00:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06ed9d700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0700729b0 0x7fc0701f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc06dd9b700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc07006e5d0 0x7fc0701f2c90 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc064005910 tx=0x7fc06400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc05effd700 1 -- 172.21.15.158:0/2142924420 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064005d70 con 0x7fc07006e5d0 2023-10-16T10:00:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc05effd700 1 -- 172.21.15.158:0/2142924420 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0640070d0 con 0x7fc07006e5d0 2023-10-16T10:00:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0701f3b00 con 0x7fc07006e5d0 2023-10-16T10:00:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc05effd700 1 -- 172.21.15.158:0/2142924420 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064007480 con 0x7fc07006e5d0 2023-10-16T10:00:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.873+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0700ca020 con 0x7fc07006e5d0 2023-10-16T10:00:31.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.874+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0700681b0 con 0x7fc07006e5d0 2023-10-16T10:00:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.875+0000 7fc05effd700 1 -- 172.21.15.158:0/2142924420 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc0640075e0 con 0x7fc07006e5d0 2023-10-16T10:00:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.875+0000 7fc05effd700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc05006b020 0x7fc05006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.875+0000 7fc05effd700 1 -- 172.21.15.158:0/2142924420 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5601+0+0 (secure 0 0 0) 0x7fc064088cb0 con 0x7fc07006e5d0 2023-10-16T10:00:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.875+0000 7fc06e59c700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc05006b020 0x7fc05006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:31.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.876+0000 7fc06e59c700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc05006b020 0x7fc05006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc070073f60 tx=0x7fc060008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:31.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:31.880+0000 7fc05effd700 1 -- 172.21.15.158:0/2142924420 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc064053cb0 con 0x7fc07006e5d0 2023-10-16T10:00:32.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.076+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc0700aee90 con 0x7fc07006e5d0 2023-10-16T10:00:32.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.076+0000 7fc05effd700 1 -- 172.21.15.158:0/2142924420 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc064017090 con 0x7fc07006e5d0 2023-10-16T10:00:32.078 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:32.078 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:00:32.078 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:32.079 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T10:00:32.079 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T10:00:32.079 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:32.079 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:00:32.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc05006b020 msgr2=0x7fc05006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc05006b020 0x7fc05006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc070073f60 tx=0x7fc060008030 comp rx=0 tx=0).stop 2023-10-16T10:00:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc07006e5d0 msgr2=0x7fc0701f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc07006e5d0 0x7fc0701f2c90 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc064005910 tx=0x7fc06400ca90 comp rx=0 tx=0).stop 2023-10-16T10:00:32.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 shutdown_connections 2023-10-16T10:00:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc05006b020 0x7fc05006d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0700729b0 0x7fc0701f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc07006ab40 0x7fc0701f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 --2- 172.21.15.158:0/2142924420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc07006e5d0 0x7fc0701f2c90 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 >> 172.21.15.158:0/2142924420 conn(0x7fc0700d02b0 msgr2=0x7fc0700b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:32.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 shutdown_connections 2023-10-16T10:00:32.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.078+0000 7fc074c82700 1 -- 172.21.15.158:0/2142924420 wait complete. 2023-10-16T10:00:32.092 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T10:00:32.092 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T10:00:32.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.216+0000 7fe4b6364700 1 Processor -- start 2023-10-16T10:00:32.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.216+0000 7fe4b6364700 1 -- start start 2023-10-16T10:00:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4b6364700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe4b015da40 0x7fe4b00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4b6364700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b00cbd30 con 0x7fe4b006afe0 2023-10-16T10:00:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4b6364700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b00cc1c0 con 0x7fe4b00cc590 2023-10-16T10:00:32.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4b6364700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b015de20 con 0x7fe4b015da40 2023-10-16T10:00:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4b4901700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe4b006afe0 0x7fe4b00afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55242/0 (socket says 172.21.15.158:55242) 2023-10-16T10:00:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4b4901700 1 -- 172.21.15.158:0/105036831 learned_addr learned my addr 172.21.15.158:0/105036831 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4affff700 1 --2- 172.21.15.158:0/105036831 >> v2:172.21.15.172:3300/0 conn(0x7fe4b015da40 0x7fe4b00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4aeffd700 1 -- 172.21.15.158:0/105036831 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8035123 0 0) 0x7fe4b00cbd30 con 0x7fe4b006afe0 2023-10-16T10:00:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4aeffd700 1 -- 172.21.15.158:0/105036831 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4980035b0 con 0x7fe4b006afe0 2023-10-16T10:00:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4affff700 1 -- 172.21.15.158:0/105036831 >> v1:172.21.15.172:6789/0 conn(0x7fe4b00cc590 legacy=0x7fe4b006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4affff700 1 -- 172.21.15.158:0/105036831 >> v1:172.21.15.158:6789/0 conn(0x7fe4b006afe0 legacy=0x7fe4b00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4affff700 1 -- 172.21.15.158:0/105036831 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b00ce260 con 0x7fe4b015da40 2023-10-16T10:00:32.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.217+0000 7fe4affff700 1 --2- 172.21.15.158:0/105036831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b015da40 0x7fe4b00caa20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009840 tx=0x7fe4a0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caeef5f270f1903f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4aeffd700 1 -- 172.21.15.158:0/105036831 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0011830 con 0x7fe4b015da40 2023-10-16T10:00:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4aeffd700 1 -- 172.21.15.158:0/105036831 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a0011990 con 0x7fe4b015da40 2023-10-16T10:00:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4aeffd700 1 -- 172.21.15.158:0/105036831 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0011d20 con 0x7fe4b015da40 2023-10-16T10:00:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4b6364700 1 -- 172.21.15.158:0/105036831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b015da40 msgr2=0x7fe4b00caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4b6364700 1 --2- 172.21.15.158:0/105036831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b015da40 0x7fe4b00caa20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009840 tx=0x7fe4a0009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:32.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4b6364700 1 -- 172.21.15.158:0/105036831 shutdown_connections 2023-10-16T10:00:32.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4b6364700 1 --2- 172.21.15.158:0/105036831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b015da40 0x7fe4b00caa20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4b6364700 1 -- 172.21.15.158:0/105036831 >> 172.21.15.158:0/105036831 conn(0x7fe4b0099a10 msgr2=0x7fe4b00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:32.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4b6364700 1 -- 172.21.15.158:0/105036831 shutdown_connections 2023-10-16T10:00:32.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.218+0000 7fe4b6364700 1 -- 172.21.15.158:0/105036831 wait complete. 2023-10-16T10:00:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.219+0000 7fe4b6364700 1 Processor -- start 2023-10-16T10:00:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b6364700 1 -- start start 2023-10-16T10:00:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b6364700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4b006afe0 0x7fe4b014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b6364700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b00cc590 0x7fe4b00ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b6364700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b015da40 0x7fe4b00b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b6364700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b00c9540 con 0x7fe4b015da40 2023-10-16T10:00:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b6364700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b00c97c0 con 0x7fe4b00cc590 2023-10-16T10:00:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b6364700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b014a970 con 0x7fe4b006afe0 2023-10-16T10:00:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b4901700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b015da40 0x7fe4b00b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b4901700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b015da40 0x7fe4b00b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55072/0 (socket says 172.21.15.158:55072) 2023-10-16T10:00:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.220+0000 7fe4b4901700 1 -- 172.21.15.158:0/4191084385 learned_addr learned my addr 172.21.15.158:0/4191084385 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4affff700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4b006afe0 0x7fe4b014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4af7fe700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b00cc590 0x7fe4b00ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b4901700 1 -- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4b006afe0 msgr2=0x7fe4b014a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b4901700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4b006afe0 0x7fe4b014a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b4901700 1 -- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b00cc590 msgr2=0x7fe4b00ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b4901700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b00cc590 0x7fe4b00ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b4901700 1 -- 172.21.15.158:0/4191084385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a00094a0 con 0x7fe4b015da40 2023-10-16T10:00:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4af7fe700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b00cc590 0x7fe4b00ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b4901700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b015da40 0x7fe4b00b4980 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe4a800cbb0 tx=0x7fe4a800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4acff9700 1 -- 172.21.15.158:0/4191084385 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a800ab50 con 0x7fe4b015da40 2023-10-16T10:00:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4acff9700 1 -- 172.21.15.158:0/4191084385 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a800acb0 con 0x7fe4b015da40 2023-10-16T10:00:32.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b014da60 con 0x7fe4b015da40 2023-10-16T10:00:32.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4acff9700 1 -- 172.21.15.158:0/4191084385 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8012600 con 0x7fe4b015da40 2023-10-16T10:00:32.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.221+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b00c9d60 con 0x7fe4b015da40 2023-10-16T10:00:32.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.222+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b00ca0d0 con 0x7fe4b015da40 2023-10-16T10:00:32.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.223+0000 7fe4acff9700 1 -- 172.21.15.158:0/4191084385 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe4a8012760 con 0x7fe4b015da40 2023-10-16T10:00:32.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.223+0000 7fe4acff9700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe49c06b020 0x7fe49c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.223+0000 7fe4acff9700 1 -- 172.21.15.158:0/4191084385 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5601+0+0 (secure 0 0 0) 0x7fe4a80893c0 con 0x7fe4b015da40 2023-10-16T10:00:32.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.223+0000 7fe4affff700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe49c06b020 0x7fe49c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.223+0000 7fe4affff700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe49c06b020 0x7fe49c06d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00090b0 tx=0x7fe4a001c040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:32.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.227+0000 7fe4acff9700 1 -- 172.21.15.158:0/4191084385 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4a8054470 con 0x7fe4b015da40 2023-10-16T10:00:32.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.423+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe4b00681b0 con 0x7fe4b015da40 2023-10-16T10:00:32.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.593+0000 7fe4acff9700 1 -- 172.21.15.158:0/4191084385 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe4a8057a90 con 0x7fe4b015da40 2023-10-16T10:00:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe49c06b020 msgr2=0x7fe49c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe49c06b020 0x7fe49c06d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00090b0 tx=0x7fe4a001c040 comp rx=0 tx=0).stop 2023-10-16T10:00:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b015da40 msgr2=0x7fe4b00b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b015da40 0x7fe4b00b4980 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe4a800cbb0 tx=0x7fe4a800cf70 comp rx=0 tx=0).stop 2023-10-16T10:00:32.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 shutdown_connections 2023-10-16T10:00:32.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fe49c06b020 0x7fe49c06d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b015da40 0x7fe4b00b4980 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4b006afe0 0x7fe4b014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 --2- 172.21.15.158:0/4191084385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4b00cc590 0x7fe4b00ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.595+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 >> 172.21.15.158:0/4191084385 conn(0x7fe4b0099a10 msgr2=0x7fe4b00b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.596+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 shutdown_connections 2023-10-16T10:00:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.596+0000 7fe4b6364700 1 -- 172.21.15.158:0/4191084385 wait complete. 2023-10-16T10:00:32.600 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T10:00:32.612 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T10:00:32.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.739+0000 7fc834796700 1 Processor -- start 2023-10-16T10:00:32.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.739+0000 7fc834796700 1 -- start start 2023-10-16T10:00:32.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc834796700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc82c15a780 0x7fc82c0b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc834796700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c072b60 0x7fc82c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc834796700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c06dde0 con 0x7fc82c06d960 2023-10-16T10:00:32.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc834796700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc82c0b45a0 con 0x7fc82c15a780 2023-10-16T10:00:32.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc834796700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c15e190 con 0x7fc82c072b60 2023-10-16T10:00:32.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc82c15a780 0x7fc82c0b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc82c15a780 0x7fc82c0b4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55076/0 (socket says 172.21.15.158:55076) 2023-10-16T10:00:32.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 -- 172.21.15.158:0/2707211078 learned_addr learned my addr 172.21.15.158:0/2707211078 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 -- 172.21.15.158:0/2707211078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c072b60 msgr2=0x7fc82c06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc831d31700 1 --2- 172.21.15.158:0/2707211078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c072b60 0x7fc82c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 --2- 172.21.15.158:0/2707211078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c072b60 0x7fc82c06d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 -- 172.21.15.158:0/2707211078 >> v1:172.21.15.172:6789/0 conn(0x7fc82c06d960 legacy=0x7fc82c0718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:32.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 -- 172.21.15.158:0/2707211078 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c15e470 con 0x7fc82c15a780 2023-10-16T10:00:32.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc832532700 1 --2- 172.21.15.158:0/2707211078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c15a780 0x7fc82c0b4930 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fc8280098f0 tx=0x7fc828009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f12feb653db1a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:32.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.740+0000 7fc831530700 1 -- 172.21.15.158:0/2707211078 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc828004230 con 0x7fc82c15a780 2023-10-16T10:00:32.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc831530700 1 -- 172.21.15.158:0/2707211078 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc828004390 con 0x7fc82c15a780 2023-10-16T10:00:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc831530700 1 -- 172.21.15.158:0/2707211078 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8280036a0 con 0x7fc82c15a780 2023-10-16T10:00:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 -- 172.21.15.158:0/2707211078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c15a780 msgr2=0x7fc82c0b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 --2- 172.21.15.158:0/2707211078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c15a780 0x7fc82c0b4930 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fc8280098f0 tx=0x7fc828009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 -- 172.21.15.158:0/2707211078 shutdown_connections 2023-10-16T10:00:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 --2- 172.21.15.158:0/2707211078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c15a780 0x7fc82c0b4930 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 --2- 172.21.15.158:0/2707211078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c072b60 0x7fc82c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 -- 172.21.15.158:0/2707211078 >> 172.21.15.158:0/2707211078 conn(0x7fc82c0ccd30 msgr2=0x7fc82c0cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 -- 172.21.15.158:0/2707211078 shutdown_connections 2023-10-16T10:00:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.741+0000 7fc834796700 1 -- 172.21.15.158:0/2707211078 wait complete. 2023-10-16T10:00:32.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.742+0000 7fc834796700 1 Processor -- start 2023-10-16T10:00:32.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.742+0000 7fc834796700 1 -- start start 2023-10-16T10:00:32.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc834796700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc82c06d960 0x7fc82c0b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc834796700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c072b60 0x7fc82c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc834796700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c15a780 0x7fc82c099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc834796700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0b1540 con 0x7fc82c072b60 2023-10-16T10:00:32.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc834796700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0b0690 con 0x7fc82c06d960 2023-10-16T10:00:32.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc834796700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0b0930 con 0x7fc82c15a780 2023-10-16T10:00:32.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc831d31700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c072b60 0x7fc82c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc831d31700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c072b60 0x7fc82c159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55084/0 (socket says 172.21.15.158:55084) 2023-10-16T10:00:32.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc831d31700 1 -- 172.21.15.158:0/3763396534 learned_addr learned my addr 172.21.15.158:0/3763396534 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:32.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc832532700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc82c06d960 0x7fc82c0b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc832d33700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c15a780 0x7fc82c099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc831d31700 1 -- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c15a780 msgr2=0x7fc82c099ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.743+0000 7fc831d31700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c15a780 0x7fc82c099ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc831d31700 1 -- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc82c06d960 msgr2=0x7fc82c0b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:32.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc831d31700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc82c06d960 0x7fc82c0b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:32.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc831d31700 1 -- 172.21.15.158:0/3763396534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828009550 con 0x7fc82c072b60 2023-10-16T10:00:32.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc832532700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc82c06d960 0x7fc82c0b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:32.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc832d33700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c15a780 0x7fc82c099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:32.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc831d31700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c072b60 0x7fc82c159120 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00c7f0 tx=0x7fc81c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:32.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc822ffd700 1 -- 172.21.15.158:0/3763396534 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c004220 con 0x7fc82c072b60 2023-10-16T10:00:32.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc822ffd700 1 -- 172.21.15.158:0/3763396534 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81c004380 con 0x7fc82c072b60 2023-10-16T10:00:32.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c0b0c10 con 0x7fc82c072b60 2023-10-16T10:00:32.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc822ffd700 1 -- 172.21.15.158:0/3763396534 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c0036a0 con 0x7fc82c072b60 2023-10-16T10:00:32.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.744+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c0b11d0 con 0x7fc82c072b60 2023-10-16T10:00:32.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.745+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82c0b16f0 con 0x7fc82c072b60 2023-10-16T10:00:32.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.745+0000 7fc822ffd700 1 -- 172.21.15.158:0/3763396534 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc81c003800 con 0x7fc82c072b60 2023-10-16T10:00:32.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.746+0000 7fc822ffd700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc81006b020 0x7fc81006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:32.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.746+0000 7fc822ffd700 1 -- 172.21.15.158:0/3763396534 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5971+0+0 (secure 0 0 0) 0x7fc81c0895a0 con 0x7fc82c072b60 2023-10-16T10:00:32.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.746+0000 7fc832532700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc81006b020 0x7fc81006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:32.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.746+0000 7fc832532700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc81006b020 0x7fc81006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc828009840 tx=0x7fc828008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:32.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.751+0000 7fc822ffd700 1 -- 172.21.15.158:0/3763396534 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc82c0b16f0 con 0x7fc82c072b60 2023-10-16T10:00:32.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:32.942+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc82c0681b0 con 0x7fc82c072b60 2023-10-16T10:00:33.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.595+0000 7fc822ffd700 1 -- 172.21.15.158:0/3763396534 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc81c054330 con 0x7fc82c072b60 2023-10-16T10:00:33.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.598+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc81006b020 msgr2=0x7fc81006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:33.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.598+0000 7fc834796700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc81006b020 0x7fc81006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc828009840 tx=0x7fc828008030 comp rx=0 tx=0).stop 2023-10-16T10:00:33.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.598+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c072b60 msgr2=0x7fc82c159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:33.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.598+0000 7fc834796700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c072b60 0x7fc82c159120 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00c7f0 tx=0x7fc81c00cbb0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.599+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 shutdown_connections 2023-10-16T10:00:33.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.599+0000 7fc834796700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc81006b020 0x7fc81006d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.599+0000 7fc834796700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc82c15a780 0x7fc82c099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.599+0000 7fc834796700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc82c06d960 0x7fc82c0b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.599+0000 7fc834796700 1 --2- 172.21.15.158:0/3763396534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc82c072b60 0x7fc82c159120 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.599+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 >> 172.21.15.158:0/3763396534 conn(0x7fc82c0ccd30 msgr2=0x7fc82c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:33.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.600+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 shutdown_connections 2023-10-16T10:00:33.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.600+0000 7fc834796700 1 -- 172.21.15.158:0/3763396534 wait complete. 2023-10-16T10:00:33.605 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T10:00:33.628 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T10:00:33.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.756+0000 7f4bd0f25700 1 Processor -- start 2023-10-16T10:00:33.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.756+0000 7f4bd0f25700 1 -- start start 2023-10-16T10:00:33.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bd0f25700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:33.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bd0f25700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4bcc0cbef0 0x7f4bcc0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:33.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bd0f25700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bcc159660 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bd0f25700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bcc0cb710 con 0x7f4bcc0cbef0 2023-10-16T10:00:33.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bd0f25700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc06b120 con 0x7f4bcc159280 2023-10-16T10:00:33.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bca59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:33.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bcad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4bcc0cbef0 0x7f4bcc0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:33.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bcad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4bcc0cbef0 0x7f4bcc0cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:56274/0 (socket says 172.21.15.158:56274) 2023-10-16T10:00:33.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.757+0000 7f4bcad9d700 1 -- 172.21.15.158:0/4145843302 learned_addr learned my addr 172.21.15.158:0/4145843302 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:33.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bcad9d700 1 -- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 msgr2=0x7f4bcc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:33.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bcad9d700 1 --2- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bcad9d700 1 -- 172.21.15.158:0/4145843302 >> v1:172.21.15.172:6789/0 conn(0x7f4bcc0d0e10 legacy=0x7f4bcc15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:33.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bcad9d700 1 -- 172.21.15.158:0/4145843302 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc06b400 con 0x7f4bcc0cbef0 2023-10-16T10:00:33.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bca59c700 1 --2- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:33.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bcad9d700 1 --2- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 0x7f4bcc0cc380 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009e50 tx=0x7f4bc400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cf686ffe6595b8b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:33.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bc959a700 1 -- 172.21.15.158:0/4145843302 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400e070 con 0x7f4bcc0cbef0 2023-10-16T10:00:33.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bc959a700 1 -- 172.21.15.158:0/4145843302 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc400b910 con 0x7f4bcc0cbef0 2023-10-16T10:00:33.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.758+0000 7f4bc959a700 1 -- 172.21.15.158:0/4145843302 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400bcc0 con 0x7f4bcc0cbef0 2023-10-16T10:00:33.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 -- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 msgr2=0x7f4bcc0cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:33.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 0x7f4bcc0cc380 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009e50 tx=0x7f4bc400b330 comp rx=0 tx=0).stop 2023-10-16T10:00:33.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 -- 172.21.15.158:0/4145843302 shutdown_connections 2023-10-16T10:00:33.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 0x7f4bcc0cc380 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/4145843302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 -- 172.21.15.158:0/4145843302 >> 172.21.15.158:0/4145843302 conn(0x7f4bcc099a10 msgr2=0x7f4bcc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:33.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 -- 172.21.15.158:0/4145843302 shutdown_connections 2023-10-16T10:00:33.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.759+0000 7f4bd0f25700 1 -- 172.21.15.158:0/4145843302 wait complete. 2023-10-16T10:00:33.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.760+0000 7f4bd0f25700 1 Processor -- start 2023-10-16T10:00:33.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.760+0000 7f4bd0f25700 1 -- start start 2023-10-16T10:00:33.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bd0f25700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 0x7f4bcc154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:33.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bd0f25700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bcc0d0e10 0x7f4bcc0c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:33.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bd0f25700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc0c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:33.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bd0f25700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc154ff0 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bd0f25700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc0b0010 con 0x7f4bcc0cbef0 2023-10-16T10:00:33.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bd0f25700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc0c6ce0 con 0x7f4bcc159280 2023-10-16T10:00:33.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bc9d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bcc0d0e10 0x7f4bcc0c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:33.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bc9d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bcc0d0e10 0x7f4bcc0c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55098/0 (socket says 172.21.15.158:55098) 2023-10-16T10:00:33.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bc9d9b700 1 -- 172.21.15.158:0/1002058399 learned_addr learned my addr 172.21.15.158:0/1002058399 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:33.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bcad9d700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc0c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:33.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.761+0000 7f4bca59c700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 0x7f4bcc154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:33.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bc9d9b700 1 -- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 msgr2=0x7f4bcc0c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:33.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bc9d9b700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc0c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bc9d9b700 1 -- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 msgr2=0x7f4bcc154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:33.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bc9d9b700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 0x7f4bcc154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:33.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bc9d9b700 1 -- 172.21.15.158:0/1002058399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc00094a0 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bc9d9b700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bcc0d0e10 0x7f4bcc0c3f00 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c450 tx=0x7f4bbc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:33.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bbaffd700 1 -- 172.21.15.158:0/1002058399 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0052d0 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bbaffd700 1 -- 172.21.15.158:0/1002058399 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc005430 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc4009b30 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bbaffd700 1 -- 172.21.15.158:0/1002058399 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0038c0 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.762+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc0af770 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.763+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc0681b0 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.764+0000 7f4bbaffd700 1 -- 172.21.15.158:0/1002058399 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4bbc003a20 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.764+0000 7f4bbaffd700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f4bac06b020 0x7f4bac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:33.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.764+0000 7f4bbaffd700 1 -- 172.21.15.158:0/1002058399 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f4bbc088d60 con 0x7f4bcc0d0e10 2023-10-16T10:00:33.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.764+0000 7f4bca59c700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f4bac06b020 0x7f4bac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:33.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.764+0000 7f4bca59c700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f4bac06b020 0x7f4bac06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009d00 tx=0x7f4bc0008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:33.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:33.769+0000 7f4bbaffd700 1 -- 172.21.15.158:0/1002058399 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bbc053b00 con 0x7f4bcc0d0e10 2023-10-16T10:00:34.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.012+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4bcc074270 con 0x7f4bcc0d0e10 2023-10-16T10:00:34.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.630+0000 7f4bbaffd700 1 -- 172.21.15.158:0/1002058399 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v47) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4bbc0170c0 con 0x7f4bcc0d0e10 2023-10-16T10:00:34.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.637+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f4bac06b020 msgr2=0x7f4bac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:34.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.637+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f4bac06b020 0x7f4bac06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009d00 tx=0x7f4bc0008040 comp rx=0 tx=0).stop 2023-10-16T10:00:34.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.637+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bcc0d0e10 msgr2=0x7f4bcc0c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:34.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.637+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bcc0d0e10 0x7f4bcc0c3f00 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c450 tx=0x7f4bbc00c810 comp rx=0 tx=0).stop 2023-10-16T10:00:34.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.640+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 shutdown_connections 2023-10-16T10:00:34.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.640+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f4bac06b020 0x7f4bac06d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.640+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bcc159280 0x7f4bcc0c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.640+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bcc0cbef0 0x7f4bcc154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.640+0000 7f4bd0f25700 1 --2- 172.21.15.158:0/1002058399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bcc0d0e10 0x7f4bcc0c3f00 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.640+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 >> 172.21.15.158:0/1002058399 conn(0x7f4bcc099a10 msgr2=0x7f4bcc15a830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:34.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.643+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 shutdown_connections 2023-10-16T10:00:34.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.643+0000 7f4bd0f25700 1 -- 172.21.15.158:0/1002058399 wait complete. 2023-10-16T10:00:34.650 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 13 and data pool 14 2023-10-16T10:00:34.668 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:34.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.792+0000 7ff2e3697700 1 Processor -- start 2023-10-16T10:00:34.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.792+0000 7ff2e3697700 1 -- start start 2023-10-16T10:00:34.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e3697700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff2dc15b350 0x7ff2dc0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:34.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e3697700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc06af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e3697700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc0ca780 con 0x7ff2dc15b730 2023-10-16T10:00:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e3697700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2dc06aa90 con 0x7ff2dc15b350 2023-10-16T10:00:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e3697700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0cf820 con 0x7ff2dc0ce630 2023-10-16T10:00:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e1433700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff2dc15b350 0x7ff2dc0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e0c32700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff2dc15b730 0x7ff2dc0cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55280/0 (socket says 172.21.15.158:55280) 2023-10-16T10:00:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e0c32700 1 -- 172.21.15.158:0/2236377921 learned_addr learned my addr 172.21.15.158:0/2236377921 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.793+0000 7ff2e1c34700 1 --2- 172.21.15.158:0/2236377921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2e1433700 1 -- 172.21.15.158:0/2236377921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 msgr2=0x7ff2dc06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2e1433700 1 --2- 172.21.15.158:0/2236377921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2e1433700 1 -- 172.21.15.158:0/2236377921 >> v1:172.21.15.158:6789/0 conn(0x7ff2dc15b730 legacy=0x7ff2dc0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2e1433700 1 -- 172.21.15.158:0/2236377921 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc0cf260 con 0x7ff2dc15b350 2023-10-16T10:00:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2d3fff700 1 -- 172.21.15.158:0/2236377921 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973444248 0 0) 0x7ff2dc0ca780 con 0x7ff2dc15b730 2023-10-16T10:00:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2e1433700 1 --2- 172.21.15.158:0/2236377921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc0ca080 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009840 tx=0x7ff2cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39072ba227678245 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:34.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2d3fff700 1 -- 172.21.15.158:0/2236377921 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc011830 con 0x7ff2dc15b350 2023-10-16T10:00:34.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2d3fff700 1 -- 172.21.15.158:0/2236377921 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc011990 con 0x7ff2dc15b350 2023-10-16T10:00:34.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2d3fff700 1 -- 172.21.15.158:0/2236377921 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc011d20 con 0x7ff2dc15b350 2023-10-16T10:00:34.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2e3697700 1 -- 172.21.15.158:0/2236377921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 msgr2=0x7ff2dc0ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:34.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.794+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2236377921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc0ca080 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009840 tx=0x7ff2cc009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:34.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.795+0000 7ff2e3697700 1 -- 172.21.15.158:0/2236377921 shutdown_connections 2023-10-16T10:00:34.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.795+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2236377921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc0ca080 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.795+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2236377921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.795+0000 7ff2e3697700 1 -- 172.21.15.158:0/2236377921 >> 172.21.15.158:0/2236377921 conn(0x7ff2dc0cc4b0 msgr2=0x7ff2dc0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:34.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.795+0000 7ff2e3697700 1 -- 172.21.15.158:0/2236377921 shutdown_connections 2023-10-16T10:00:34.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.795+0000 7ff2e3697700 1 -- 172.21.15.158:0/2236377921 wait complete. 2023-10-16T10:00:34.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.796+0000 7ff2e3697700 1 Processor -- start 2023-10-16T10:00:34.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.796+0000 7ff2e3697700 1 -- start start 2023-10-16T10:00:34.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.796+0000 7ff2e3697700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc0c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:34.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.796+0000 7ff2e3697700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:34.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e3697700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2dc15b730 0x7ff2dc0c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:34.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e3697700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0c8f80 con 0x7ff2dc15b350 2023-10-16T10:00:34.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e3697700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0c8370 con 0x7ff2dc15b730 2023-10-16T10:00:34.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e3697700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0c8650 con 0x7ff2dc0ce630 2023-10-16T10:00:34.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:34.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc14bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55110/0 (socket says 172.21.15.158:55110) 2023-10-16T10:00:34.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 -- 172.21.15.158:0/2749310796 learned_addr learned my addr 172.21.15.158:0/2749310796 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:34.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e1c34700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2dc15b730 0x7ff2dc0c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:34.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e1433700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc0c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:34.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 -- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 msgr2=0x7ff2dc0c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:34.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc0c52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 -- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2dc15b730 msgr2=0x7ff2dc0c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:34.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2dc15b730 0x7ff2dc0c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:34.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 -- 172.21.15.158:0/2749310796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc0094a0 con 0x7ff2dc15b350 2023-10-16T10:00:34.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e1433700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc0c52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:34.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e1c34700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2dc15b730 0x7ff2dc0c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:34.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e0c32700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc14bc30 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800c520 tx=0x7ff2d800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:34.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2d1ffb700 1 -- 172.21.15.158:0/2749310796 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8004c80 con 0x7ff2dc15b350 2023-10-16T10:00:34.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2d1ffb700 1 -- 172.21.15.158:0/2749310796 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d8004de0 con 0x7ff2dc15b350 2023-10-16T10:00:34.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc0c8930 con 0x7ff2dc15b350 2023-10-16T10:00:34.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.797+0000 7ff2d1ffb700 1 -- 172.21.15.158:0/2749310796 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d80072d0 con 0x7ff2dc15b350 2023-10-16T10:00:34.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.798+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2dc14c170 con 0x7ff2dc15b350 2023-10-16T10:00:34.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.798+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc0c9130 con 0x7ff2dc15b350 2023-10-16T10:00:34.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.799+0000 7ff2d1ffb700 1 -- 172.21.15.158:0/2749310796 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff2d8007430 con 0x7ff2dc15b350 2023-10-16T10:00:34.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.799+0000 7ff2d1ffb700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7ff2c006b020 0x7ff2c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:34.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.799+0000 7ff2d1ffb700 1 -- 172.21.15.158:0/2749310796 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6453+0+0 (secure 0 0 0) 0x7ff2d8089810 con 0x7ff2dc15b350 2023-10-16T10:00:34.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.799+0000 7ff2e1433700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7ff2c006b020 0x7ff2c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:34.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.800+0000 7ff2e1433700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7ff2c006b020 0x7ff2c006d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc000c00 tx=0x7ff2cc01c040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:34.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.804+0000 7ff2d1ffb700 1 -- 172.21.15.158:0/2749310796 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2dc0c9130 con 0x7ff2dc15b350 2023-10-16T10:00:35.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:34.999+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2dc0681b0 con 0x7ff2dc15b350 2023-10-16T10:00:35.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.000+0000 7ff2d1ffb700 1 -- 172.21.15.158:0/2749310796 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+17398 (secure 0 0 0) 0x7ff2d8054420 con 0x7ff2dc15b350 2023-10-16T10:00:35.002 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:35.002 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":110,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:34.013375+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:00:32.442785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T10:00:32.961620+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:35.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7ff2c006b020 msgr2=0x7ff2c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7ff2c006b020 0x7ff2c006d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc000c00 tx=0x7ff2cc01c040 comp rx=0 tx=0).stop 2023-10-16T10:00:35.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 msgr2=0x7ff2dc14bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc14bc30 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800c520 tx=0x7ff2d800c8e0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 shutdown_connections 2023-10-16T10:00:35.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7ff2c006b020 0x7ff2c006d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2dc15b730 0x7ff2dc0c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2dc0ce630 0x7ff2dc0c52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 --2- 172.21.15.158:0/2749310796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2dc15b350 0x7ff2dc14bc30 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 >> 172.21.15.158:0/2749310796 conn(0x7ff2dc0cc4b0 msgr2=0x7ff2dc06b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:35.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 shutdown_connections 2023-10-16T10:00:35.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.002+0000 7ff2e3697700 1 -- 172.21.15.158:0/2749310796 wait complete. 2023-10-16T10:00:35.016 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:35.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.135+0000 7f361cf2a700 1 Processor -- start 2023-10-16T10:00:35.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.135+0000 7f361cf2a700 1 -- start start 2023-10-16T10:00:35.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361cf2a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36180cc1e0 0x7f361815be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361cf2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f361815c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361cf2a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36180cfe80 con 0x7f36181524e0 2023-10-16T10:00:35.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361cf2a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361815c8a0 con 0x7f36180cc1e0 2023-10-16T10:00:35.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361cf2a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36180cf270 con 0x7f36180d0310 2023-10-16T10:00:35.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3615d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36180cc1e0 0x7f361815be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361659c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f36181524e0 0x7f36180cbad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55294/0 (socket says 172.21.15.158:55294) 2023-10-16T10:00:35.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361659c700 1 -- 172.21.15.158:0/4119238814 learned_addr learned my addr 172.21.15.158:0/4119238814 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:35.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3616d9d700 1 --2- 172.21.15.158:0/4119238814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f361815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3615d9b700 1 -- 172.21.15.158:0/4119238814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 msgr2=0x7f361815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:35.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3615d9b700 1 --2- 172.21.15.158:0/4119238814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f361815c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3615d9b700 1 -- 172.21.15.158:0/4119238814 >> v1:172.21.15.158:6789/0 conn(0x7f36181524e0 legacy=0x7f36180cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3615d9b700 1 -- 172.21.15.158:0/4119238814 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36180cf550 con 0x7f36180cc1e0 2023-10-16T10:00:35.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f361559a700 1 -- 172.21.15.158:0/4119238814 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177518696 0 0) 0x7f36180cfe80 con 0x7f36181524e0 2023-10-16T10:00:35.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3616d9d700 1 --2- 172.21.15.158:0/4119238814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f361815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:35.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.136+0000 7f3615d9b700 1 --2- 172.21.15.158:0/4119238814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f361815be20 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f360c009840 tx=0x7f360c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcacc1fcf13758db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:35.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361559a700 1 -- 172.21.15.158:0/4119238814 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c011830 con 0x7f36180cc1e0 2023-10-16T10:00:35.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361559a700 1 -- 172.21.15.158:0/4119238814 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c011990 con 0x7f36180cc1e0 2023-10-16T10:00:35.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361559a700 1 -- 172.21.15.158:0/4119238814 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c011d20 con 0x7f36180cc1e0 2023-10-16T10:00:35.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 -- 172.21.15.158:0/4119238814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 msgr2=0x7f361815be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 --2- 172.21.15.158:0/4119238814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f361815be20 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f360c009840 tx=0x7f360c009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:35.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 -- 172.21.15.158:0/4119238814 shutdown_connections 2023-10-16T10:00:35.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 --2- 172.21.15.158:0/4119238814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f361815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 --2- 172.21.15.158:0/4119238814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f361815be20 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 -- 172.21.15.158:0/4119238814 >> 172.21.15.158:0/4119238814 conn(0x7f361806a8d0 msgr2=0x7f361806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:35.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 -- 172.21.15.158:0/4119238814 shutdown_connections 2023-10-16T10:00:35.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.137+0000 7f361cf2a700 1 -- 172.21.15.158:0/4119238814 wait complete. 2023-10-16T10:00:35.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.138+0000 7f361cf2a700 1 Processor -- start 2023-10-16T10:00:35.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.138+0000 7f361cf2a700 1 -- start start 2023-10-16T10:00:35.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361cf2a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f3618158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361cf2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f36181f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361cf2a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36181524e0 0x7f36181f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361cf2a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36181501d0 con 0x7f36180cc1e0 2023-10-16T10:00:35.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361cf2a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36180b3e80 con 0x7f36181524e0 2023-10-16T10:00:35.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361cf2a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36180b4160 con 0x7f36180d0310 2023-10-16T10:00:35.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f3618158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f3618158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55124/0 (socket says 172.21.15.158:55124) 2023-10-16T10:00:35.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.139+0000 7f361659c700 1 -- 172.21.15.158:0/3735646395 learned_addr learned my addr 172.21.15.158:0/3735646395 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:35.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f3615d9b700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f36181f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361659c700 1 -- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 msgr2=0x7f36181f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361659c700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f36181f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f3616d9d700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36181524e0 0x7f36181f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361659c700 1 -- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36181524e0 msgr2=0x7f36181f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361659c700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36181524e0 0x7f36181f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361659c700 1 -- 172.21.15.158:0/3735646395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c0094a0 con 0x7f36180cc1e0 2023-10-16T10:00:35.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361659c700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f3618158690 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f360800c580 tx=0x7f360800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:35.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f3606ffd700 1 -- 172.21.15.158:0/3735646395 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608004c80 con 0x7f36180cc1e0 2023-10-16T10:00:35.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f3606ffd700 1 -- 172.21.15.158:0/3735646395 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608004e00 con 0x7f36180cc1e0 2023-10-16T10:00:35.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3618158bd0 con 0x7f36180cc1e0 2023-10-16T10:00:35.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f3606ffd700 1 -- 172.21.15.158:0/3735646395 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608007310 con 0x7f36180cc1e0 2023-10-16T10:00:35.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.140+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36180ca7e0 con 0x7f36180cc1e0 2023-10-16T10:00:35.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.141+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36180681b0 con 0x7f36180cc1e0 2023-10-16T10:00:35.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.141+0000 7f3606ffd700 1 -- 172.21.15.158:0/3735646395 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3608007470 con 0x7f36180cc1e0 2023-10-16T10:00:35.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.142+0000 7f3606ffd700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f35f806b020 0x7f35f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.142+0000 7f3606ffd700 1 -- 172.21.15.158:0/3735646395 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f3608089690 con 0x7f36180cc1e0 2023-10-16T10:00:35.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.142+0000 7f3615d9b700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f35f806b020 0x7f35f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.142+0000 7f3615d9b700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f35f806b020 0x7f35f806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36180b6420 tx=0x7f360c008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:35.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.146+0000 7f3606ffd700 1 -- 172.21.15.158:0/3735646395 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f36080543d0 con 0x7f36180cc1e0 2023-10-16T10:00:35.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.339+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36180cf050 con 0x7f36180cc1e0 2023-10-16T10:00:35.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.340+0000 7f3606ffd700 1 -- 172.21.15.158:0/3735646395 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+17398 (secure 0 0 0) 0x7f36080579f0 con 0x7f36180cc1e0 2023-10-16T10:00:35.342 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:35.342 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":110,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:34.013375+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:00:32.442785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T10:00:32.961620+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:35.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f35f806b020 msgr2=0x7f35f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f35f806b020 0x7f35f806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36180b6420 tx=0x7f360c008030 comp rx=0 tx=0).stop 2023-10-16T10:00:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 msgr2=0x7f3618158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f3618158690 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f360800c580 tx=0x7f360800c940 comp rx=0 tx=0).stop 2023-10-16T10:00:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 shutdown_connections 2023-10-16T10:00:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f35f806b020 0x7f35f806d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36181524e0 0x7f36181f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36180cc1e0 0x7f3618158690 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 --2- 172.21.15.158:0/3735646395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36180d0310 0x7f36181f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 >> 172.21.15.158:0/3735646395 conn(0x7f361806a8d0 msgr2=0x7f36180d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 shutdown_connections 2023-10-16T10:00:35.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.343+0000 7f361cf2a700 1 -- 172.21.15.158:0/3735646395 wait complete. 2023-10-16T10:00:35.357 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T10:00:35.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.479+0000 7f84f3507700 1 Processor -- start 2023-10-16T10:00:35.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.479+0000 7f84f3507700 1 -- start start 2023-10-16T10:00:35.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f3507700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f84ec15da40 0x7f84ec0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f3507700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f3507700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ec0cbd30 con 0x7f84ec0cc590 2023-10-16T10:00:35.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f3507700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec0cc1c0 con 0x7f84ec15da40 2023-10-16T10:00:35.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f3507700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec15de20 con 0x7f84ec06afe0 2023-10-16T10:00:35.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f12a3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f84ec15da40 0x7f84ec0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f0aa2700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f84ec0cc590 0x7f84ec06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55308/0 (socket says 172.21.15.158:55308) 2023-10-16T10:00:35.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f0aa2700 1 -- 172.21.15.158:0/1694497448 learned_addr learned my addr 172.21.15.158:0/1694497448 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:35.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f1aa4700 1 --2- 172.21.15.158:0/1694497448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f12a3700 1 -- 172.21.15.158:0/1694497448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 msgr2=0x7f84ec0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:35.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f12a3700 1 --2- 172.21.15.158:0/1694497448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f12a3700 1 -- 172.21.15.158:0/1694497448 >> v1:172.21.15.158:6789/0 conn(0x7f84ec0cc590 legacy=0x7f84ec06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84f12a3700 1 -- 172.21.15.158:0/1694497448 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec0ce260 con 0x7f84ec15da40 2023-10-16T10:00:35.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.480+0000 7f84e3fff700 1 -- 172.21.15.158:0/1694497448 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958902394 0 0) 0x7f84ec0cbd30 con 0x7f84ec0cc590 2023-10-16T10:00:35.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f12a3700 1 --2- 172.21.15.158:0/1694497448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec0caa20 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f84dc009840 tx=0x7f84dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1885a45079b70b93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:35.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84e3fff700 1 -- 172.21.15.158:0/1694497448 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc011830 con 0x7f84ec15da40 2023-10-16T10:00:35.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84e3fff700 1 -- 172.21.15.158:0/1694497448 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc011990 con 0x7f84ec15da40 2023-10-16T10:00:35.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84e3fff700 1 -- 172.21.15.158:0/1694497448 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc011d20 con 0x7f84ec15da40 2023-10-16T10:00:35.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 -- 172.21.15.158:0/1694497448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 msgr2=0x7f84ec0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 --2- 172.21.15.158:0/1694497448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec0caa20 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f84dc009840 tx=0x7f84dc009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:35.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 -- 172.21.15.158:0/1694497448 shutdown_connections 2023-10-16T10:00:35.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 --2- 172.21.15.158:0/1694497448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec0caa20 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 --2- 172.21.15.158:0/1694497448 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 -- 172.21.15.158:0/1694497448 >> 172.21.15.158:0/1694497448 conn(0x7f84ec099a10 msgr2=0x7f84ec0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:35.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 -- 172.21.15.158:0/1694497448 shutdown_connections 2023-10-16T10:00:35.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.481+0000 7f84f3507700 1 -- 172.21.15.158:0/1694497448 wait complete. 2023-10-16T10:00:35.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 Processor -- start 2023-10-16T10:00:35.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 -- start start 2023-10-16T10:00:35.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec1e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84ec0cc590 0x7f84ec1e6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec14e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec15acd0 con 0x7f84ec15da40 2023-10-16T10:00:35.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1e5560 con 0x7f84ec0cc590 2023-10-16T10:00:35.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f3507700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1e5840 con 0x7f84ec06afe0 2023-10-16T10:00:35.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f1aa4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec14e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f1aa4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec14e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55142/0 (socket says 172.21.15.158:55142) 2023-10-16T10:00:35.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.483+0000 7f84f1aa4700 1 -- 172.21.15.158:0/1762271750 learned_addr learned my addr 172.21.15.158:0/1762271750 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:35.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f0aa2700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84ec0cc590 0x7f84ec1e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f12a3700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec1e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f1aa4700 1 -- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 msgr2=0x7f84ec1e4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f1aa4700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec1e4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f1aa4700 1 -- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84ec0cc590 msgr2=0x7f84ec1e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:35.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f1aa4700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84ec0cc590 0x7f84ec1e6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:35.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f1aa4700 1 -- 172.21.15.158:0/1762271750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc0094a0 con 0x7f84ec15da40 2023-10-16T10:00:35.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f1aa4700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec14e690 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f84e4007d40 tx=0x7f84e400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:35.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84e1ffb700 1 -- 172.21.15.158:0/1762271750 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e400ede0 con 0x7f84ec15da40 2023-10-16T10:00:35.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84e1ffb700 1 -- 172.21.15.158:0/1762271750 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e4014420 con 0x7f84ec15da40 2023-10-16T10:00:35.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec1e5b20 con 0x7f84ec15da40 2023-10-16T10:00:35.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84e1ffb700 1 -- 172.21.15.158:0/1762271750 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e40147d0 con 0x7f84ec15da40 2023-10-16T10:00:35.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.484+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec1e60e0 con 0x7f84ec15da40 2023-10-16T10:00:35.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.485+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ec0681b0 con 0x7f84ec15da40 2023-10-16T10:00:35.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.485+0000 7f84e1ffb700 1 -- 172.21.15.158:0/1762271750 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f84e4014930 con 0x7f84ec15da40 2023-10-16T10:00:35.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.486+0000 7f84e1ffb700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f84d006b020 0x7f84d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:35.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.486+0000 7f84f12a3700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f84d006b020 0x7f84d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:35.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.486+0000 7f84e1ffb700 1 -- 172.21.15.158:0/1762271750 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f84e408a170 con 0x7f84ec15da40 2023-10-16T10:00:35.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.486+0000 7f84f12a3700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f84d006b020 0x7f84d006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84dc000c00 tx=0x7f84dc008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:35.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.490+0000 7f84e1ffb700 1 -- 172.21.15.158:0/1762271750 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84e4054e20 con 0x7f84ec15da40 2023-10-16T10:00:35.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:35.731+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f84ec1e62a0 con 0x7f84ec15da40 2023-10-16T10:00:36.597 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T10:00:36.596+0000 7fdcab5c2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:00:36.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.649+0000 7f84e1ffb700 1 -- 172.21.15.158:0/1762271750 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7f84e4058440 con 0x7f84ec15da40 2023-10-16T10:00:36.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f84d006b020 msgr2=0x7f84d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:36.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f84d006b020 0x7f84d006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84dc000c00 tx=0x7f84dc008030 comp rx=0 tx=0).stop 2023-10-16T10:00:36.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 msgr2=0x7f84ec14e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec14e690 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f84e4007d40 tx=0x7f84e400e3c0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 shutdown_connections 2023-10-16T10:00:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f84d006b020 0x7f84d006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84ec15da40 0x7f84ec14e690 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84ec06afe0 0x7f84ec1e4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 --2- 172.21.15.158:0/1762271750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84ec0cc590 0x7f84ec1e6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 >> 172.21.15.158:0/1762271750 conn(0x7f84ec099a10 msgr2=0x7f84ec15c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:36.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.652+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 shutdown_connections 2023-10-16T10:00:36.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.653+0000 7f84f3507700 1 -- 172.21.15.158:0/1762271750 wait complete. 2023-10-16T10:00:36.669 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T10:00:36.669 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T10:00:36.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.798+0000 7f04b7dcd700 1 Processor -- start 2023-10-16T10:00:36.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.798+0000 7f04b7dcd700 1 -- start start 2023-10-16T10:00:36.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.799+0000 7f04b7dcd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f04b015da40 0x7f04b00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.799+0000 7f04b7dcd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b00cbd30 con 0x7f04b00cc590 2023-10-16T10:00:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.799+0000 7f04b7dcd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b00cc1c0 con 0x7f04b006afe0 2023-10-16T10:00:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.799+0000 7f04b7dcd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b015de20 con 0x7f04b015da40 2023-10-16T10:00:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.799+0000 7f04b5368700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f04b00cc590 0x7f04b006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55322/0 (socket says 172.21.15.158:55322) 2023-10-16T10:00:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.799+0000 7f04b5368700 1 -- 172.21.15.158:0/3562100834 learned_addr learned my addr 172.21.15.158:0/3562100834 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927943058 0 0) 0x7f04b00cbd30 con 0x7f04b00cc590 2023-10-16T10:00:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c0035b0 con 0x7f04b00cc590 2023-10-16T10:00:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3562100834 >> v2:172.21.15.172:3300/0 conn(0x7f04b015da40 0x7f04b00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 768909538 0 0) 0x7f049c0035b0 con 0x7f04b00cc590 2023-10-16T10:00:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 >> v2:172.21.15.172:3300/0 conn(0x7f04b015da40 msgr2=0x7f04b00caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 --2- 172.21.15.158:0/3562100834 >> v2:172.21.15.172:3300/0 conn(0x7f04b015da40 0x7f04b00caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 >> v1:172.21.15.172:6789/0 conn(0x7f04b006afe0 legacy=0x7f04b00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b00ce260 con 0x7f04b00cc590 2023-10-16T10:00:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f04ac001670 con 0x7f04b00cc590 2023-10-16T10:00:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 >> v1:172.21.15.158:6789/0 conn(0x7f04b00cc590 legacy=0x7f04b006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c003a00 0x7f049c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c004850 0x7f049c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c005210 0x7f049c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b015de20 con 0x7f049c003a00 2023-10-16T10:00:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c00c0b0 con 0x7f049c005210 2023-10-16T10:00:36.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c00c270 con 0x7f049c004850 2023-10-16T10:00:36.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c003a00 0x7f049c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5368700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c004850 0x7f049c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b636a700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c005210 0x7f049c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5b69700 1 -- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c004850 msgr2=0x7f049c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:36.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c004850 0x7f049c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5b69700 1 -- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c005210 msgr2=0x7f049c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:36.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c005210 0x7f049c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b5b69700 1 -- 172.21.15.158:0/3562100834 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c0035b0 con 0x7f049c003a00 2023-10-16T10:00:36.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.800+0000 7f04b636a700 1 -- 172.21.15.158:0/3562100834 reap_dead start 2023-10-16T10:00:36.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c003a00 0x7f049c003e20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009b60 tx=0x7f049c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104e0561a34ec50f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:36.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a000e6c0 con 0x7f049c003a00 2023-10-16T10:00:36.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a000e820 con 0x7f049c003a00 2023-10-16T10:00:36.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b4b67700 1 -- 172.21.15.158:0/3562100834 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a000eb90 con 0x7f049c003a00 2023-10-16T10:00:36.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3562100834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c003a00 msgr2=0x7f049c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:36.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c003a00 0x7f049c003e20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009b60 tx=0x7f049c005880 comp rx=0 tx=0).stop 2023-10-16T10:00:36.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3562100834 shutdown_connections 2023-10-16T10:00:36.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3562100834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c003a00 0x7f049c003e20 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3562100834 >> 172.21.15.158:0/3562100834 conn(0x7f04b0099a10 msgr2=0x7f04b00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:36.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3562100834 shutdown_connections 2023-10-16T10:00:36.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.801+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3562100834 wait complete. 2023-10-16T10:00:36.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.803+0000 7f04b7dcd700 1 Processor -- start 2023-10-16T10:00:36.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.803+0000 7f04b7dcd700 1 -- start start 2023-10-16T10:00:36.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b7dcd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c003a00 0x7f04b014d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b7dcd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b00c7770 0x7f04b00ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b7dcd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b00b1e90 0x7f04b00b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b7dcd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00b2880 con 0x7f04b00c7770 2023-10-16T10:00:36.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b7dcd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b014de10 con 0x7f04b00b1e90 2023-10-16T10:00:36.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b7dcd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b015efe0 con 0x7f049c003a00 2023-10-16T10:00:36.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b00c7770 0x7f04b00ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b00c7770 0x7f04b00ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55158/0 (socket says 172.21.15.158:55158) 2023-10-16T10:00:36.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 -- 172.21.15.158:0/3838180475 learned_addr learned my addr 172.21.15.158:0/3838180475 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:36.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b636a700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b00b1e90 0x7f04b00b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c003a00 0x7f04b014d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 -- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c003a00 msgr2=0x7f04b014d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:36.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c003a00 0x7f04b014d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 -- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b00b1e90 msgr2=0x7f04b00b2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:36.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b00b1e90 0x7f04b00b2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:36.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 -- 172.21.15.158:0/3838180475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b00c8230 con 0x7f04b00c7770 2023-10-16T10:00:36.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.804+0000 7f04b5368700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b00c7770 0x7f04b00ca6d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00eaf0 tx=0x7f04ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:36.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.805+0000 7f04a67fc700 1 -- 172.21.15.158:0/3838180475 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac005e00 con 0x7f04b00c7770 2023-10-16T10:00:36.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.805+0000 7f04a67fc700 1 -- 172.21.15.158:0/3838180475 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac00c750 con 0x7f04b00c7770 2023-10-16T10:00:36.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.805+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b00c7c70 con 0x7f04b00c7770 2023-10-16T10:00:36.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.805+0000 7f04a67fc700 1 -- 172.21.15.158:0/3838180475 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00cb00 con 0x7f04b00c7770 2023-10-16T10:00:36.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.805+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b015ec70 con 0x7f04b00c7770 2023-10-16T10:00:36.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.806+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b015ee30 con 0x7f04b00c7770 2023-10-16T10:00:36.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.806+0000 7f04a67fc700 1 -- 172.21.15.158:0/3838180475 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f04ac00cc60 con 0x7f04b00c7770 2023-10-16T10:00:36.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.806+0000 7f04a67fc700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f049406b020 0x7f049406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:36.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.806+0000 7f04a67fc700 1 -- 172.21.15.158:0/3838180475 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f04ac0897c0 con 0x7f04b00c7770 2023-10-16T10:00:36.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.806+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f049406b020 0x7f049406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:36.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.807+0000 7f04b5b69700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f049406b020 0x7f049406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f04a00090b0 tx=0x7f04a0008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:36.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:36.811+0000 7f04a67fc700 1 -- 172.21.15.158:0/3838180475 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f04ac054470 con 0x7f04b00c7770 2023-10-16T10:00:37.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.051+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f04b015f150 con 0x7f04b00c7770 2023-10-16T10:00:37.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.651+0000 7f04a67fc700 1 -- 172.21.15.158:0/3838180475 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f04ac001510 con 0x7f04b00c7770 2023-10-16T10:00:37.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f049406b020 msgr2=0x7f049406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:37.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f049406b020 0x7f049406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f04a00090b0 tx=0x7f04a0008030 comp rx=0 tx=0).stop 2023-10-16T10:00:37.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b00c7770 msgr2=0x7f04b00ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:37.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b00c7770 0x7f04b00ca6d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00eaf0 tx=0x7f04ac00eeb0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 shutdown_connections 2023-10-16T10:00:37.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f049406b020 0x7f049406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b00b1e90 0x7f04b00b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c003a00 0x7f04b014d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 --2- 172.21.15.158:0/3838180475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b00c7770 0x7f04b00ca6d0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 >> 172.21.15.158:0/3838180475 conn(0x7f04b0099a10 msgr2=0x7f04b01e7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:37.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 shutdown_connections 2023-10-16T10:00:37.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.654+0000 7f04b7dcd700 1 -- 172.21.15.158:0/3838180475 wait complete. 2023-10-16T10:00:37.669 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T10:00:37.670 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:37.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.796+0000 7f965c9c7700 1 Processor -- start 2023-10-16T10:00:37.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.796+0000 7f965c9c7700 1 -- start start 2023-10-16T10:00:37.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f965c9c7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f96580cdee0 0x7f965815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:37.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f965c9c7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f96580b6e10 0x7f96581521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:37.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f965c9c7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965815c5d0 con 0x7f96580ce2c0 2023-10-16T10:00:37.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f965c9c7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965815d2c0 con 0x7f96580b6e10 2023-10-16T10:00:37.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f965c9c7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965815c8b0 con 0x7f96580cdee0 2023-10-16T10:00:37.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f9655d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f96580b6e10 0x7f96581521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:37.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f9656d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f96580ce2c0 0x7f96580cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55344/0 (socket says 172.21.15.158:55344) 2023-10-16T10:00:37.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f9656d9d700 1 -- 172.21.15.158:0/3530205068 learned_addr learned my addr 172.21.15.158:0/3530205068 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:37.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f965659c700 1 --2- 172.21.15.158:0/3530205068 >> v2:172.21.15.172:3300/0 conn(0x7f96580cdee0 0x7f965815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:37.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f9655d9b700 1 -- 172.21.15.158:0/3530205068 >> v2:172.21.15.172:3300/0 conn(0x7f96580cdee0 msgr2=0x7f965815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:37.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f9655d9b700 1 --2- 172.21.15.158:0/3530205068 >> v2:172.21.15.172:3300/0 conn(0x7f96580cdee0 0x7f965815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f9655d9b700 1 -- 172.21.15.158:0/3530205068 >> v1:172.21.15.158:6789/0 conn(0x7f96580ce2c0 legacy=0x7f96580cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:37.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f9655d9b700 1 -- 172.21.15.158:0/3530205068 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965815cb90 con 0x7f96580b6e10 2023-10-16T10:00:37.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.797+0000 7f965659c700 1 --2- 172.21.15.158:0/3530205068 >> v2:172.21.15.172:3300/0 conn(0x7f96580cdee0 0x7f965815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:37.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f9655d9b700 1 --2- 172.21.15.158:0/3530205068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580b6e10 0x7f96581521a0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f96580d2470 tx=0x7f9648009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915797d856080eb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:37.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965559a700 1 -- 172.21.15.158:0/3530205068 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9648011830 con 0x7f96580b6e10 2023-10-16T10:00:37.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965559a700 1 -- 172.21.15.158:0/3530205068 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9648011990 con 0x7f96580b6e10 2023-10-16T10:00:37.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965559a700 1 -- 172.21.15.158:0/3530205068 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9648011d20 con 0x7f96580b6e10 2023-10-16T10:00:37.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 -- 172.21.15.158:0/3530205068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580b6e10 msgr2=0x7f96581521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:37.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3530205068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580b6e10 0x7f96581521a0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f96580d2470 tx=0x7f9648009b20 comp rx=0 tx=0).stop 2023-10-16T10:00:37.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 -- 172.21.15.158:0/3530205068 shutdown_connections 2023-10-16T10:00:37.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3530205068 >> v2:172.21.15.172:3300/0 conn(0x7f96580cdee0 0x7f965815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3530205068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580b6e10 0x7f96581521a0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 -- 172.21.15.158:0/3530205068 >> 172.21.15.158:0/3530205068 conn(0x7f965806a8d0 msgr2=0x7f965806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:37.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 -- 172.21.15.158:0/3530205068 shutdown_connections 2023-10-16T10:00:37.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.798+0000 7f965c9c7700 1 -- 172.21.15.158:0/3530205068 wait complete. 2023-10-16T10:00:37.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 Processor -- start 2023-10-16T10:00:37.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 -- start start 2023-10-16T10:00:37.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96580b6e10 0x7f96580cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:37.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580cdee0 0x7f96580d1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:37.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96580ce2c0 0x7f96580d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:37.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96580d1070 con 0x7f96580cdee0 2023-10-16T10:00:37.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96580d0460 con 0x7f96580b6e10 2023-10-16T10:00:37.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f965c9c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96580d0740 con 0x7f96580ce2c0 2023-10-16T10:00:37.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.800+0000 7f9655d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580cdee0 0x7f96580d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:37.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580cdee0 0x7f96580d1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55172/0 (socket says 172.21.15.158:55172) 2023-10-16T10:00:37.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 -- 172.21.15.158:0/3240597157 learned_addr learned my addr 172.21.15.158:0/3240597157 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:37.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f965659c700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96580b6e10 0x7f96580cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:37.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 -- 172.21.15.158:0/3240597157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96580ce2c0 msgr2=0x7f96580d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:37.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96580ce2c0 0x7f96580d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 -- 172.21.15.158:0/3240597157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96580b6e10 msgr2=0x7f96580cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:37.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96580b6e10 0x7f96580cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:37.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 -- 172.21.15.158:0/3240597157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480094a0 con 0x7f96580cdee0 2023-10-16T10:00:37.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9655d9b700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580cdee0 0x7f96580d1500 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f96480090b0 tx=0x7f9648004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:37.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9646ffd700 1 -- 172.21.15.158:0/3240597157 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96480038a0 con 0x7f96580cdee0 2023-10-16T10:00:37.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9646ffd700 1 -- 172.21.15.158:0/3240597157 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964801f070 con 0x7f96580cdee0 2023-10-16T10:00:37.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f9646ffd700 1 -- 172.21.15.158:0/3240597157 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9648003c60 con 0x7f96580cdee0 2023-10-16T10:00:37.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96580d0a20 con 0x7f96580cdee0 2023-10-16T10:00:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.801+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96580cfe30 con 0x7f96580cdee0 2023-10-16T10:00:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.802+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96580d1220 con 0x7f96580cdee0 2023-10-16T10:00:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.802+0000 7f9646ffd700 1 -- 172.21.15.158:0/3240597157 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f964801a430 con 0x7f96580cdee0 2023-10-16T10:00:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.803+0000 7f9646ffd700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f963806b020 0x7f963806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.803+0000 7f965659c700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f963806b020 0x7f963806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.803+0000 7f9646ffd700 1 -- 172.21.15.158:0/3240597157 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9648021070 con 0x7f96580cdee0 2023-10-16T10:00:37.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.803+0000 7f965659c700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f963806b020 0x7f963806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f964c006fd0 tx=0x7f964c008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:37.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:37.808+0000 7f9646ffd700 1 -- 172.21.15.158:0/3240597157 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96580d1220 con 0x7f96580cdee0 2023-10-16T10:00:38.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.047+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96580b35d0 con 0x7f96580cdee0 2023-10-16T10:00:38.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.048+0000 7f9646ffd700 1 -- 172.21.15.158:0/3240597157 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f96480562a0 con 0x7f96580cdee0 2023-10-16T10:00:38.049 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:38.050 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2674019415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2674019415},{"type":"v1","addr":"172.21.15.158:6836","nonce":2674019415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2425279024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2425279024},{"type":"v1","addr":"172.21.15.158:6837","nonce":2425279024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3002995342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3002995342},{"type":"v1","addr":"172.21.15.172:6834","nonce":3002995342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:00:34.013744+0000","modified":"2023-10-16T10:00:37.052969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7740":{"gid":7740,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1596548674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1596548674},{"type":"v1","addr":"172.21.15.172:6835","nonce":1596548674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T10:00:38.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f963806b020 msgr2=0x7f963806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f963806b020 0x7f963806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f964c006fd0 tx=0x7f964c008040 comp rx=0 tx=0).stop 2023-10-16T10:00:38.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580cdee0 msgr2=0x7f96580d1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580cdee0 0x7f96580d1500 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f96480090b0 tx=0x7f9648004750 comp rx=0 tx=0).stop 2023-10-16T10:00:38.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 shutdown_connections 2023-10-16T10:00:38.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f963806b020 0x7f963806d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96580ce2c0 0x7f96580d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96580b6e10 0x7f96580cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 --2- 172.21.15.158:0/3240597157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96580cdee0 0x7f96580d1500 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 >> 172.21.15.158:0/3240597157 conn(0x7f965806a8d0 msgr2=0x7f96580b5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:38.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 shutdown_connections 2023-10-16T10:00:38.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.050+0000 7f965c9c7700 1 -- 172.21.15.158:0/3240597157 wait complete. 2023-10-16T10:00:38.054 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 51 2023-10-16T10:00:38.063 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:00:38.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.184+0000 7f28730ac700 1 Processor -- start 2023-10-16T10:00:38.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.184+0000 7f28730ac700 1 -- start start 2023-10-16T10:00:38.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f28730ac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f286c06e5d0 0x7f286c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f28730ac700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f286c0729b0 0x7f286c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f28730ac700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c071e50 con 0x7f286c06ab40 2023-10-16T10:00:38.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f28730ac700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c161270 con 0x7f286c0729b0 2023-10-16T10:00:38.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f28730ac700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c161510 con 0x7f286c06e5d0 2023-10-16T10:00:38.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f2871649700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f286c0729b0 0x7f286c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f2870e48700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f286c06ab40 0x7f286c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55354/0 (socket says 172.21.15.158:55354) 2023-10-16T10:00:38.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.185+0000 7f2870e48700 1 -- 172.21.15.158:0/1362885100 learned_addr learned my addr 172.21.15.158:0/1362885100 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:38.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f286bfff700 1 --2- 172.21.15.158:0/1362885100 >> v2:172.21.15.172:3300/0 conn(0x7f286c06e5d0 0x7f286c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f2871649700 1 -- 172.21.15.158:0/1362885100 >> v2:172.21.15.172:3300/0 conn(0x7f286c06e5d0 msgr2=0x7f286c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:38.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f2871649700 1 --2- 172.21.15.158:0/1362885100 >> v2:172.21.15.172:3300/0 conn(0x7f286c06e5d0 0x7f286c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f2871649700 1 -- 172.21.15.158:0/1362885100 >> v1:172.21.15.158:6789/0 conn(0x7f286c06ab40 legacy=0x7f286c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f2871649700 1 -- 172.21.15.158:0/1362885100 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c1617f0 con 0x7f286c0729b0 2023-10-16T10:00:38.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f286b7fe700 1 -- 172.21.15.158:0/1362885100 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447655412 0 0) 0x7f286c071e50 con 0x7f286c06ab40 2023-10-16T10:00:38.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f286bfff700 1 --2- 172.21.15.158:0/1362885100 >> v2:172.21.15.172:3300/0 conn(0x7f286c06e5d0 0x7f286c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:38.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f2871649700 1 --2- 172.21.15.158:0/1362885100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0729b0 0x7f286c071910 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f286c0c3bf0 tx=0x7f286400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd0fd86102789ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f286b7fe700 1 -- 172.21.15.158:0/1362885100 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400e070 con 0x7f286c0729b0 2023-10-16T10:00:38.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f286b7fe700 1 -- 172.21.15.158:0/1362885100 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286400bbd0 con 0x7f286c0729b0 2023-10-16T10:00:38.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f286b7fe700 1 -- 172.21.15.158:0/1362885100 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864012430 con 0x7f286c0729b0 2023-10-16T10:00:38.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f28730ac700 1 -- 172.21.15.158:0/1362885100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0729b0 msgr2=0x7f286c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.186+0000 7f28730ac700 1 --2- 172.21.15.158:0/1362885100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0729b0 0x7f286c071910 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f286c0c3bf0 tx=0x7f286400b330 comp rx=0 tx=0).stop 2023-10-16T10:00:38.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.187+0000 7f28730ac700 1 -- 172.21.15.158:0/1362885100 shutdown_connections 2023-10-16T10:00:38.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.187+0000 7f28730ac700 1 --2- 172.21.15.158:0/1362885100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c0729b0 0x7f286c071910 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.187+0000 7f28730ac700 1 --2- 172.21.15.158:0/1362885100 >> v2:172.21.15.172:3300/0 conn(0x7f286c06e5d0 0x7f286c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.187+0000 7f28730ac700 1 -- 172.21.15.158:0/1362885100 >> 172.21.15.158:0/1362885100 conn(0x7f286c0d02b0 msgr2=0x7f286c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:38.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.187+0000 7f28730ac700 1 -- 172.21.15.158:0/1362885100 shutdown_connections 2023-10-16T10:00:38.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.187+0000 7f28730ac700 1 -- 172.21.15.158:0/1362885100 wait complete. 2023-10-16T10:00:38.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.188+0000 7f28730ac700 1 Processor -- start 2023-10-16T10:00:38.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.188+0000 7f28730ac700 1 -- start start 2023-10-16T10:00:38.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06ab40 0x7f286c160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c06e5d0 0x7f286c146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c0729b0 0x7f286c0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c146510 con 0x7f286c06ab40 2023-10-16T10:00:38.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0b3420 con 0x7f286c0729b0 2023-10-16T10:00:38.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0b2ea0 con 0x7f286c06e5d0 2023-10-16T10:00:38.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06ab40 0x7f286c160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06ab40 0x7f286c160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55184/0 (socket says 172.21.15.158:55184) 2023-10-16T10:00:38.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 -- 172.21.15.158:0/1265081417 learned_addr learned my addr 172.21.15.158:0/1265081417 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:38.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f286bfff700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c06e5d0 0x7f286c146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2871649700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c0729b0 0x7f286c0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 -- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c06e5d0 msgr2=0x7f286c146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c06e5d0 0x7f286c146760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 -- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c0729b0 msgr2=0x7f286c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c0729b0 0x7f286c0b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 -- 172.21.15.158:0/1265081417 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864009a70 con 0x7f286c06ab40 2023-10-16T10:00:38.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f2870e48700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06ab40 0x7f286c160af0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f286000c550 tx=0x7f286000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28697fa700 1 -- 172.21.15.158:0/1265081417 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860004c80 con 0x7f286c06ab40 2023-10-16T10:00:38.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28697fa700 1 -- 172.21.15.158:0/1265081417 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2860004e00 con 0x7f286c06ab40 2023-10-16T10:00:38.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c0b3140 con 0x7f286c06ab40 2023-10-16T10:00:38.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28697fa700 1 -- 172.21.15.158:0/1265081417 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860007310 con 0x7f286c06ab40 2023-10-16T10:00:38.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.189+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c0b5de0 con 0x7f286c06ab40 2023-10-16T10:00:38.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.190+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c146d00 con 0x7f286c06ab40 2023-10-16T10:00:38.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.191+0000 7f28697fa700 1 -- 172.21.15.158:0/1265081417 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2860007470 con 0x7f286c06ab40 2023-10-16T10:00:38.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.191+0000 7f28697fa700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f285006b020 0x7f285006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.191+0000 7f28697fa700 1 -- 172.21.15.158:0/1265081417 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f2860089760 con 0x7f286c06ab40 2023-10-16T10:00:38.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.191+0000 7f286bfff700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f285006b020 0x7f285006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.192+0000 7f286bfff700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f285006b020 0x7f285006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f286c14ce70 tx=0x7f2858008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.196+0000 7f28697fa700 1 -- 172.21.15.158:0/1265081417 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2860054410 con 0x7f286c06ab40 2023-10-16T10:00:38.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.390+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f286c0681b0 con 0x7f286c06ab40 2023-10-16T10:00:38.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.392+0000 7f28697fa700 1 -- 172.21.15.158:0/1265081417 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+17385 (secure 0 0 0) 0x7f2860057a30 con 0x7f286c06ab40 2023-10-16T10:00:38.393 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:38.393 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":111,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:00:35.629910+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:00:32.442785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T10:00:32.961620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:00:38.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f285006b020 msgr2=0x7f285006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f285006b020 0x7f285006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f286c14ce70 tx=0x7f2858008040 comp rx=0 tx=0).stop 2023-10-16T10:00:38.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06ab40 msgr2=0x7f286c160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06ab40 0x7f286c160af0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f286000c550 tx=0x7f286000c910 comp rx=0 tx=0).stop 2023-10-16T10:00:38.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 shutdown_connections 2023-10-16T10:00:38.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f285006b020 0x7f285006d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f286c0729b0 0x7f286c0b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f286c06ab40 0x7f286c160af0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 --2- 172.21.15.158:0/1265081417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f286c06e5d0 0x7f286c146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 >> 172.21.15.158:0/1265081417 conn(0x7f286c0d02b0 msgr2=0x7f286c0ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:38.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 shutdown_connections 2023-10-16T10:00:38.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.394+0000 7f28730ac700 1 -- 172.21.15.158:0/1265081417 wait complete. 2023-10-16T10:00:38.407 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T10:00:38.407 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:00:38.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.527+0000 7f8af1845700 1 Processor -- start 2023-10-16T10:00:38.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.527+0000 7f8af1845700 1 -- start start 2023-10-16T10:00:38.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8af1845700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8aec15d7c0 0x7f8aec0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8af1845700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec0b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8af1845700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec0d3bd0 con 0x7f8aec0d1210 2023-10-16T10:00:38.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8af1845700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec0d4060 con 0x7f8aec15d7c0 2023-10-16T10:00:38.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8af1845700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0d2cf0 con 0x7f8aec0ceb20 2023-10-16T10:00:38.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aeaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8aec15d7c0 0x7f8aec0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aea7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8aec0d1210 0x7f8aec0ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55370/0 (socket says 172.21.15.158:55370) 2023-10-16T10:00:38.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aea7fc700 1 -- 172.21.15.158:0/3587614213 learned_addr learned my addr 172.21.15.158:0/3587614213 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:38.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aeb7fe700 1 --2- 172.21.15.158:0/3587614213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec0b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aeaffd700 1 -- 172.21.15.158:0/3587614213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 msgr2=0x7f8aec0b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:38.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aeaffd700 1 --2- 172.21.15.158:0/3587614213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec0b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aeaffd700 1 -- 172.21.15.158:0/3587614213 >> v1:172.21.15.158:6789/0 conn(0x7f8aec0d1210 legacy=0x7f8aec0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aeaffd700 1 -- 172.21.15.158:0/3587614213 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec0d2fc0 con 0x7f8aec15d7c0 2023-10-16T10:00:38.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.528+0000 7f8aeaffd700 1 --2- 172.21.15.158:0/3587614213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec15d7c0 0x7f8aec0d27b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009840 tx=0x7f8ae0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0412771aa0d962 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8ae9ffb700 1 -- 172.21.15.158:0/3587614213 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0004230 con 0x7f8aec15d7c0 2023-10-16T10:00:38.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8ae9ffb700 1 -- 172.21.15.158:0/3587614213 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae0004390 con 0x7f8aec15d7c0 2023-10-16T10:00:38.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8ae9ffb700 1 -- 172.21.15.158:0/3587614213 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae00036a0 con 0x7f8aec15d7c0 2023-10-16T10:00:38.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 -- 172.21.15.158:0/3587614213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec15d7c0 msgr2=0x7f8aec0d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 --2- 172.21.15.158:0/3587614213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec15d7c0 0x7f8aec0d27b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009840 tx=0x7f8ae0009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:38.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 -- 172.21.15.158:0/3587614213 shutdown_connections 2023-10-16T10:00:38.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 --2- 172.21.15.158:0/3587614213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec15d7c0 0x7f8aec0d27b0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 --2- 172.21.15.158:0/3587614213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec0b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 -- 172.21.15.158:0/3587614213 >> 172.21.15.158:0/3587614213 conn(0x7f8aec0669f0 msgr2=0x7f8aec066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:38.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 -- 172.21.15.158:0/3587614213 shutdown_connections 2023-10-16T10:00:38.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.529+0000 7f8af1845700 1 -- 172.21.15.158:0/3587614213 wait complete. 2023-10-16T10:00:38.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.531+0000 7f8af1845700 1 Processor -- start 2023-10-16T10:00:38.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.531+0000 7f8af1845700 1 -- start start 2023-10-16T10:00:38.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8af1845700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec1605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8af1845700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec0d1210 0x7f8aec0b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8af1845700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec15d7c0 0x7f8aec0b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8af1845700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec158a10 con 0x7f8aec0d1210 2023-10-16T10:00:38.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8af1845700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec160af0 con 0x7f8aec15d7c0 2023-10-16T10:00:38.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8af1845700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec15dba0 con 0x7f8aec0ceb20 2023-10-16T10:00:38.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec0d1210 0x7f8aec0b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec0d1210 0x7f8aec0b5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55200/0 (socket says 172.21.15.158:55200) 2023-10-16T10:00:38.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 -- 172.21.15.158:0/2210994802 learned_addr learned my addr 172.21.15.158:0/2210994802 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:38.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aeaffd700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec1605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aeb7fe700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec15d7c0 0x7f8aec0b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 -- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 msgr2=0x7f8aec1605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec1605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 -- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec15d7c0 msgr2=0x7f8aec0b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec15d7c0 0x7f8aec0b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.532+0000 7f8aea7fc700 1 -- 172.21.15.158:0/2210994802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae00094a0 con 0x7f8aec0d1210 2023-10-16T10:00:38.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.533+0000 7f8aea7fc700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec0d1210 0x7f8aec0b5090 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00c7b0 tx=0x7f8adc00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.533+0000 7f8ad3fff700 1 -- 172.21.15.158:0/2210994802 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc004c80 con 0x7f8aec0d1210 2023-10-16T10:00:38.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.533+0000 7f8ad3fff700 1 -- 172.21.15.158:0/2210994802 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc004de0 con 0x7f8aec0d1210 2023-10-16T10:00:38.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.533+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec0b5f30 con 0x7f8aec0d1210 2023-10-16T10:00:38.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.533+0000 7f8ad3fff700 1 -- 172.21.15.158:0/2210994802 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc007310 con 0x7f8aec0d1210 2023-10-16T10:00:38.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.533+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec1583c0 con 0x7f8aec0d1210 2023-10-16T10:00:38.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.534+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec0681b0 con 0x7f8aec0d1210 2023-10-16T10:00:38.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.534+0000 7f8ad3fff700 1 -- 172.21.15.158:0/2210994802 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8adc007470 con 0x7f8aec0d1210 2023-10-16T10:00:38.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.534+0000 7f8ad3fff700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f8ad806b020 0x7f8ad806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.535+0000 7f8ad3fff700 1 -- 172.21.15.158:0/2210994802 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f8adc0897f0 con 0x7f8aec0d1210 2023-10-16T10:00:38.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.535+0000 7f8aeaffd700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f8ad806b020 0x7f8ad806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.535+0000 7f8aeaffd700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f8ad806b020 0x7f8ad806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0000c00 tx=0x7f8ae0008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.539+0000 7f8ad3fff700 1 -- 172.21.15.158:0/2210994802 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8adc0544a0 con 0x7f8aec0d1210 2023-10-16T10:00:38.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.727+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 --> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8aec0d3580 con 0x7f8ad806b020 2023-10-16T10:00:38.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.735+0000 7f8ad3fff700 1 -- 172.21.15.158:0/2210994802 <== mgr.6895 v2:172.21.15.158:6832/1430981267 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289993 (secure 0 0 0) 0x7f8ae0006c60 con 0x7f8ad806b020 2023-10-16T10:00:38.740 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:38.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f8ad806b020 msgr2=0x7f8ad806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f8ad806b020 0x7f8ad806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0000c00 tx=0x7f8ae0008030 comp rx=0 tx=0).stop 2023-10-16T10:00:38.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec0d1210 msgr2=0x7f8aec0b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec0d1210 0x7f8aec0b5090 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00c7b0 tx=0x7f8adc00cac0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 shutdown_connections 2023-10-16T10:00:38.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f8ad806b020 0x7f8ad806d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8aec15d7c0 0x7f8aec0b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8aec0ceb20 0x7f8aec1605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 --2- 172.21.15.158:0/2210994802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8aec0d1210 0x7f8aec0b5090 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 >> 172.21.15.158:0/2210994802 conn(0x7f8aec0669f0 msgr2=0x7f8aec15caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:38.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 shutdown_connections 2023-10-16T10:00:38.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.739+0000 7f8af1845700 1 -- 172.21.15.158:0/2210994802 wait complete. 2023-10-16T10:00:38.745 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:00:38.752 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T10:00:38.332352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336260,"kb_used_data":11852,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754638460,"statfs":{"total":773094113280,"available":772749783040,"internally_reserved":0,"allocated":12136448,"data_stored":6818960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":332058167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.737801"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634490+0000","last_change":"2023-10-16T10:00:34.651969+0000","last_active":"2023-10-16T10:00:35.634490+0000","last_peered":"2023-10-16T10:00:35.634490+0000","last_clean":"2023-10-16T10:00:35.634490+0000","last_became_active":"2023-10-16T10:00:34.651692+0000","last_became_peered":"2023-10-16T10:00:34.651692+0000","last_unstale":"2023-10-16T10:00:35.634490+0000","last_undegraded":"2023-10-16T10:00:35.634490+0000","last_fullsized":"2023-10-16T10:00:35.634490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:32.443386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860161+0000","last_change":"2023-10-16T10:00:34.653193+0000","last_active":"2023-10-16T10:00:35.860161+0000","last_peered":"2023-10-16T10:00:35.860161+0000","last_clean":"2023-10-16T10:00:35.860161+0000","last_became_active":"2023-10-16T10:00:34.652878+0000","last_became_peered":"2023-10-16T10:00:34.652878+0000","last_unstale":"2023-10-16T10:00:35.860161+0000","last_undegraded":"2023-10-16T10:00:35.860161+0000","last_fullsized":"2023-10-16T10:00:35.860161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:42:17.438735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634280+0000","last_change":"2023-10-16T10:00:34.647330+0000","last_active":"2023-10-16T10:00:35.634280+0000","last_peered":"2023-10-16T10:00:35.634280+0000","last_clean":"2023-10-16T10:00:35.634280+0000","last_became_active":"2023-10-16T10:00:34.640824+0000","last_became_peered":"2023-10-16T10:00:34.640824+0000","last_unstale":"2023-10-16T10:00:35.634280+0000","last_undegraded":"2023-10-16T10:00:35.634280+0000","last_fullsized":"2023-10-16T10:00:35.634280+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:36.973289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.645600+0000","last_change":"2023-10-16T10:00:34.642790+0000","last_active":"2023-10-16T10:00:34.645600+0000","last_peered":"2023-10-16T10:00:34.645600+0000","last_clean":"2023-10-16T10:00:34.645600+0000","last_became_active":"2023-10-16T10:00:34.642492+0000","last_became_peered":"2023-10-16T10:00:34.642492+0000","last_unstale":"2023-10-16T10:00:34.645600+0000","last_undegraded":"2023-10-16T10:00:34.645600+0000","last_fullsized":"2023-10-16T10:00:34.645600+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:34.758584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635724+0000","last_change":"2023-10-16T10:00:34.641813+0000","last_active":"2023-10-16T10:00:35.635724+0000","last_peered":"2023-10-16T10:00:35.635724+0000","last_clean":"2023-10-16T10:00:35.635724+0000","last_became_active":"2023-10-16T10:00:34.641535+0000","last_became_peered":"2023-10-16T10:00:34.641535+0000","last_unstale":"2023-10-16T10:00:35.635724+0000","last_undegraded":"2023-10-16T10:00:35.635724+0000","last_fullsized":"2023-10-16T10:00:35.635724+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:44.743191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322064+0000","last_change":"2023-10-16T10:00:34.641617+0000","last_active":"2023-10-16T10:00:36.322064+0000","last_peered":"2023-10-16T10:00:36.322064+0000","last_clean":"2023-10-16T10:00:36.322064+0000","last_became_active":"2023-10-16T10:00:34.639579+0000","last_became_peered":"2023-10-16T10:00:34.639579+0000","last_unstale":"2023-10-16T10:00:36.322064+0000","last_undegraded":"2023-10-16T10:00:36.322064+0000","last_fullsized":"2023-10-16T10:00:36.322064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:36.973289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633175+0000","last_change":"2023-10-16T10:00:34.654804+0000","last_active":"2023-10-16T10:00:35.633175+0000","last_peered":"2023-10-16T10:00:35.633175+0000","last_clean":"2023-10-16T10:00:35.633175+0000","last_became_active":"2023-10-16T10:00:34.654501+0000","last_became_peered":"2023-10-16T10:00:34.654501+0000","last_unstale":"2023-10-16T10:00:35.633175+0000","last_undegraded":"2023-10-16T10:00:35.633175+0000","last_fullsized":"2023-10-16T10:00:35.633175+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.637087+0000","last_change":"2023-10-16T10:00:34.634553+0000","last_active":"2023-10-16T10:00:34.637087+0000","last_peered":"2023-10-16T10:00:34.637087+0000","last_clean":"2023-10-16T10:00:34.637087+0000","last_became_active":"2023-10-16T10:00:34.634264+0000","last_became_peered":"2023-10-16T10:00:34.634264+0000","last_unstale":"2023-10-16T10:00:34.637087+0000","last_undegraded":"2023-10-16T10:00:34.637087+0000","last_fullsized":"2023-10-16T10:00:34.637087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:34.712375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635331+0000","last_change":"2023-10-16T10:00:34.639615+0000","last_active":"2023-10-16T10:00:35.635331+0000","last_peered":"2023-10-16T10:00:35.635331+0000","last_clean":"2023-10-16T10:00:35.635331+0000","last_became_active":"2023-10-16T10:00:34.639313+0000","last_became_peered":"2023-10-16T10:00:34.639313+0000","last_unstale":"2023-10-16T10:00:35.635331+0000","last_undegraded":"2023-10-16T10:00:35.635331+0000","last_fullsized":"2023-10-16T10:00:35.635331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:17:34.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.644650+0000","last_change":"2023-10-16T10:00:34.640739+0000","last_active":"2023-10-16T10:00:34.644650+0000","last_peered":"2023-10-16T10:00:34.644650+0000","last_clean":"2023-10-16T10:00:34.644650+0000","last_became_active":"2023-10-16T10:00:34.640473+0000","last_became_peered":"2023-10-16T10:00:34.640473+0000","last_unstale":"2023-10-16T10:00:34.644650+0000","last_undegraded":"2023-10-16T10:00:34.644650+0000","last_fullsized":"2023-10-16T10:00:34.644650+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:53.921189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635685+0000","last_change":"2023-10-16T10:00:34.642226+0000","last_active":"2023-10-16T10:00:35.635685+0000","last_peered":"2023-10-16T10:00:35.635685+0000","last_clean":"2023-10-16T10:00:35.635685+0000","last_became_active":"2023-10-16T10:00:34.641968+0000","last_became_peered":"2023-10-16T10:00:34.641968+0000","last_unstale":"2023-10-16T10:00:35.635685+0000","last_undegraded":"2023-10-16T10:00:35.635685+0000","last_fullsized":"2023-10-16T10:00:35.635685+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:17.347366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.658350+0000","last_change":"2023-10-16T10:00:34.656817+0000","last_active":"2023-10-16T10:00:34.658350+0000","last_peered":"2023-10-16T10:00:34.658350+0000","last_clean":"2023-10-16T10:00:34.658350+0000","last_became_active":"2023-10-16T10:00:34.656528+0000","last_became_peered":"2023-10-16T10:00:34.656528+0000","last_unstale":"2023-10-16T10:00:34.658350+0000","last_undegraded":"2023-10-16T10:00:34.658350+0000","last_fullsized":"2023-10-16T10:00:34.658350+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:38.909489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.636891+0000","last_change":"2023-10-16T10:00:34.634375+0000","last_active":"2023-10-16T10:00:34.636891+0000","last_peered":"2023-10-16T10:00:34.636891+0000","last_clean":"2023-10-16T10:00:34.636891+0000","last_became_active":"2023-10-16T10:00:34.634105+0000","last_became_peered":"2023-10-16T10:00:34.634105+0000","last_unstale":"2023-10-16T10:00:34.636891+0000","last_undegraded":"2023-10-16T10:00:34.636891+0000","last_fullsized":"2023-10-16T10:00:34.636891+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:34.437565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634891+0000","last_change":"2023-10-16T10:00:34.638358+0000","last_active":"2023-10-16T10:00:35.634891+0000","last_peered":"2023-10-16T10:00:35.634891+0000","last_clean":"2023-10-16T10:00:35.634891+0000","last_became_active":"2023-10-16T10:00:34.638078+0000","last_became_peered":"2023-10-16T10:00:34.638078+0000","last_unstale":"2023-10-16T10:00:35.634891+0000","last_undegraded":"2023-10-16T10:00:35.634891+0000","last_fullsized":"2023-10-16T10:00:35.634891+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:14:19.787907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.658256+0000","last_change":"2023-10-16T10:00:34.657614+0000","last_active":"2023-10-16T10:00:34.658256+0000","last_peered":"2023-10-16T10:00:34.658256+0000","last_clean":"2023-10-16T10:00:34.658256+0000","last_became_active":"2023-10-16T10:00:34.657341+0000","last_became_peered":"2023-10-16T10:00:34.657341+0000","last_unstale":"2023-10-16T10:00:34.658256+0000","last_undegraded":"2023-10-16T10:00:34.658256+0000","last_fullsized":"2023-10-16T10:00:34.658256+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:47.706895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322268+0000","last_change":"2023-10-16T10:00:34.659947+0000","last_active":"2023-10-16T10:00:36.322268+0000","last_peered":"2023-10-16T10:00:36.322268+0000","last_clean":"2023-10-16T10:00:36.322268+0000","last_became_active":"2023-10-16T10:00:34.659515+0000","last_became_peered":"2023-10-16T10:00:34.659515+0000","last_unstale":"2023-10-16T10:00:36.322268+0000","last_undegraded":"2023-10-16T10:00:36.322268+0000","last_fullsized":"2023-10-16T10:00:36.322268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:08.396720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322452+0000","last_change":"2023-10-16T10:00:34.642125+0000","last_active":"2023-10-16T10:00:36.322452+0000","last_peered":"2023-10-16T10:00:36.322452+0000","last_clean":"2023-10-16T10:00:36.322452+0000","last_became_active":"2023-10-16T10:00:34.641816+0000","last_became_peered":"2023-10-16T10:00:34.641816+0000","last_unstale":"2023-10-16T10:00:36.322452+0000","last_undegraded":"2023-10-16T10:00:36.322452+0000","last_fullsized":"2023-10-16T10:00:36.322452+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:14:36.640328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634484+0000","last_change":"2023-10-16T10:00:34.652848+0000","last_active":"2023-10-16T10:00:35.634484+0000","last_peered":"2023-10-16T10:00:35.634484+0000","last_clean":"2023-10-16T10:00:35.634484+0000","last_became_active":"2023-10-16T10:00:34.652328+0000","last_became_peered":"2023-10-16T10:00:34.652328+0000","last_unstale":"2023-10-16T10:00:35.634484+0000","last_undegraded":"2023-10-16T10:00:35.634484+0000","last_fullsized":"2023-10-16T10:00:35.634484+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635340+0000","last_change":"2023-10-16T10:00:34.641331+0000","last_active":"2023-10-16T10:00:35.635340+0000","last_peered":"2023-10-16T10:00:35.635340+0000","last_clean":"2023-10-16T10:00:35.635340+0000","last_became_active":"2023-10-16T10:00:34.641060+0000","last_became_peered":"2023-10-16T10:00:34.641060+0000","last_unstale":"2023-10-16T10:00:35.635340+0000","last_undegraded":"2023-10-16T10:00:35.635340+0000","last_fullsized":"2023-10-16T10:00:35.635340+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:41.245646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.656768+0000","last_change":"2023-10-16T10:00:34.654314+0000","last_active":"2023-10-16T10:00:34.656768+0000","last_peered":"2023-10-16T10:00:34.656768+0000","last_clean":"2023-10-16T10:00:34.656768+0000","last_became_active":"2023-10-16T10:00:34.653734+0000","last_became_peered":"2023-10-16T10:00:34.653734+0000","last_unstale":"2023-10-16T10:00:34.656768+0000","last_undegraded":"2023-10-16T10:00:34.656768+0000","last_fullsized":"2023-10-16T10:00:34.656768+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:35.428669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635839+0000","last_change":"2023-10-16T10:00:34.639677+0000","last_active":"2023-10-16T10:00:35.635839+0000","last_peered":"2023-10-16T10:00:35.635839+0000","last_clean":"2023-10-16T10:00:35.635839+0000","last_became_active":"2023-10-16T10:00:34.639378+0000","last_became_peered":"2023-10-16T10:00:34.639378+0000","last_unstale":"2023-10-16T10:00:35.635839+0000","last_undegraded":"2023-10-16T10:00:35.635839+0000","last_fullsized":"2023-10-16T10:00:35.635839+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:47.514148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634909+0000","last_change":"2023-10-16T10:00:34.641232+0000","last_active":"2023-10-16T10:00:35.634909+0000","last_peered":"2023-10-16T10:00:35.634909+0000","last_clean":"2023-10-16T10:00:35.634909+0000","last_became_active":"2023-10-16T10:00:34.640902+0000","last_became_peered":"2023-10-16T10:00:34.640902+0000","last_unstale":"2023-10-16T10:00:35.634909+0000","last_undegraded":"2023-10-16T10:00:35.634909+0000","last_fullsized":"2023-10-16T10:00:35.634909+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:01.931212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.651389+0000","last_change":"2023-10-16T10:00:34.640573+0000","last_active":"2023-10-16T10:00:34.651389+0000","last_peered":"2023-10-16T10:00:34.651389+0000","last_clean":"2023-10-16T10:00:34.651389+0000","last_became_active":"2023-10-16T10:00:34.640264+0000","last_became_peered":"2023-10-16T10:00:34.640264+0000","last_unstale":"2023-10-16T10:00:34.651389+0000","last_undegraded":"2023-10-16T10:00:34.651389+0000","last_fullsized":"2023-10-16T10:00:34.651389+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:47.706895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634150+0000","last_change":"2023-10-16T10:00:34.656232+0000","last_active":"2023-10-16T10:00:35.634150+0000","last_peered":"2023-10-16T10:00:35.634150+0000","last_clean":"2023-10-16T10:00:35.634150+0000","last_became_active":"2023-10-16T10:00:34.655948+0000","last_became_peered":"2023-10-16T10:00:34.655948+0000","last_unstale":"2023-10-16T10:00:35.634150+0000","last_undegraded":"2023-10-16T10:00:35.634150+0000","last_fullsized":"2023-10-16T10:00:35.634150+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:33.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860382+0000","last_change":"2023-10-16T10:00:34.646343+0000","last_active":"2023-10-16T10:00:35.860382+0000","last_peered":"2023-10-16T10:00:35.860382+0000","last_clean":"2023-10-16T10:00:35.860382+0000","last_became_active":"2023-10-16T10:00:34.646035+0000","last_became_peered":"2023-10-16T10:00:34.646035+0000","last_unstale":"2023-10-16T10:00:35.860382+0000","last_undegraded":"2023-10-16T10:00:35.860382+0000","last_fullsized":"2023-10-16T10:00:35.860382+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635629+0000","last_change":"2023-10-16T10:00:34.641499+0000","last_active":"2023-10-16T10:00:35.635629+0000","last_peered":"2023-10-16T10:00:35.635629+0000","last_clean":"2023-10-16T10:00:35.635629+0000","last_became_active":"2023-10-16T10:00:34.640873+0000","last_became_peered":"2023-10-16T10:00:34.640873+0000","last_unstale":"2023-10-16T10:00:35.635629+0000","last_undegraded":"2023-10-16T10:00:35.635629+0000","last_fullsized":"2023-10-16T10:00:35.635629+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:13.489391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.643970+0000","last_change":"2023-10-16T10:00:34.639323+0000","last_active":"2023-10-16T10:00:34.643970+0000","last_peered":"2023-10-16T10:00:34.643970+0000","last_clean":"2023-10-16T10:00:34.643970+0000","last_became_active":"2023-10-16T10:00:34.639040+0000","last_became_peered":"2023-10-16T10:00:34.639040+0000","last_unstale":"2023-10-16T10:00:34.643970+0000","last_undegraded":"2023-10-16T10:00:34.643970+0000","last_fullsized":"2023-10-16T10:00:34.643970+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:05:32.016253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.644984+0000","last_change":"2023-10-16T10:00:34.641690+0000","last_active":"2023-10-16T10:00:34.644984+0000","last_peered":"2023-10-16T10:00:34.644984+0000","last_clean":"2023-10-16T10:00:34.644984+0000","last_became_active":"2023-10-16T10:00:34.641407+0000","last_became_peered":"2023-10-16T10:00:34.641407+0000","last_unstale":"2023-10-16T10:00:34.644984+0000","last_undegraded":"2023-10-16T10:00:34.644984+0000","last_fullsized":"2023-10-16T10:00:34.644984+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:37.957418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633214+0000","last_change":"2023-10-16T10:00:34.634138+0000","last_active":"2023-10-16T10:00:35.633214+0000","last_peered":"2023-10-16T10:00:35.633214+0000","last_clean":"2023-10-16T10:00:35.633214+0000","last_became_active":"2023-10-16T10:00:34.633857+0000","last_became_peered":"2023-10-16T10:00:34.633857+0000","last_unstale":"2023-10-16T10:00:35.633214+0000","last_undegraded":"2023-10-16T10:00:35.633214+0000","last_fullsized":"2023-10-16T10:00:35.633214+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:32.443386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635072+0000","last_change":"2023-10-16T10:00:34.637077+0000","last_active":"2023-10-16T10:00:35.635072+0000","last_peered":"2023-10-16T10:00:35.635072+0000","last_clean":"2023-10-16T10:00:35.635072+0000","last_became_active":"2023-10-16T10:00:34.636579+0000","last_became_peered":"2023-10-16T10:00:34.636579+0000","last_unstale":"2023-10-16T10:00:35.635072+0000","last_undegraded":"2023-10-16T10:00:35.635072+0000","last_fullsized":"2023-10-16T10:00:35.635072+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:23.408584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.859956+0000","last_change":"2023-10-16T10:00:34.643324+0000","last_active":"2023-10-16T10:00:35.859956+0000","last_peered":"2023-10-16T10:00:35.859956+0000","last_clean":"2023-10-16T10:00:35.859956+0000","last_became_active":"2023-10-16T10:00:34.643016+0000","last_became_peered":"2023-10-16T10:00:34.643016+0000","last_unstale":"2023-10-16T10:00:35.859956+0000","last_undegraded":"2023-10-16T10:00:35.859956+0000","last_fullsized":"2023-10-16T10:00:35.859956+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:31.785355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322091+0000","last_change":"2023-10-16T10:00:34.659774+0000","last_active":"2023-10-16T10:00:36.322091+0000","last_peered":"2023-10-16T10:00:36.322091+0000","last_clean":"2023-10-16T10:00:36.322091+0000","last_became_active":"2023-10-16T10:00:34.659164+0000","last_became_peered":"2023-10-16T10:00:34.659164+0000","last_unstale":"2023-10-16T10:00:36.322091+0000","last_undegraded":"2023-10-16T10:00:36.322091+0000","last_fullsized":"2023-10-16T10:00:36.322091+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:42:17.438735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322221+0000","last_change":"2023-10-16T10:00:34.641759+0000","last_active":"2023-10-16T10:00:36.322221+0000","last_peered":"2023-10-16T10:00:36.322221+0000","last_clean":"2023-10-16T10:00:36.322221+0000","last_became_active":"2023-10-16T10:00:34.641480+0000","last_became_peered":"2023-10-16T10:00:34.641480+0000","last_unstale":"2023-10-16T10:00:36.322221+0000","last_undegraded":"2023-10-16T10:00:36.322221+0000","last_fullsized":"2023-10-16T10:00:36.322221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"110'4","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.662107+0000","last_change":"2023-10-16T10:00:33.621706+0000","last_active":"2023-10-16T10:00:34.662107+0000","last_peered":"2023-10-16T10:00:34.662107+0000","last_clean":"2023-10-16T10:00:34.662107+0000","last_became_active":"2023-10-16T10:00:33.621410+0000","last_became_peered":"2023-10-16T10:00:33.621410+0000","last_unstale":"2023-10-16T10:00:34.662107+0000","last_undegraded":"2023-10-16T10:00:34.662107+0000","last_fullsized":"2023-10-16T10:00:34.662107+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:33.708051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322381+0000","last_change":"2023-10-16T10:00:34.640248+0000","last_active":"2023-10-16T10:00:36.322381+0000","last_peered":"2023-10-16T10:00:36.322381+0000","last_clean":"2023-10-16T10:00:36.322381+0000","last_became_active":"2023-10-16T10:00:34.639121+0000","last_became_peered":"2023-10-16T10:00:34.639121+0000","last_unstale":"2023-10-16T10:00:36.322381+0000","last_undegraded":"2023-10-16T10:00:36.322381+0000","last_fullsized":"2023-10-16T10:00:36.322381+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:20:29.845160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635577+0000","last_change":"2023-10-16T10:00:33.616391+0000","last_active":"2023-10-16T10:00:35.635577+0000","last_peered":"2023-10-16T10:00:35.635577+0000","last_clean":"2023-10-16T10:00:35.635577+0000","last_became_active":"2023-10-16T10:00:33.616089+0000","last_became_peered":"2023-10-16T10:00:33.616089+0000","last_unstale":"2023-10-16T10:00:35.635577+0000","last_undegraded":"2023-10-16T10:00:35.635577+0000","last_fullsized":"2023-10-16T10:00:35.635577+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:20.593335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633315+0000","last_change":"2023-10-16T10:00:34.639078+0000","last_active":"2023-10-16T10:00:35.633315+0000","last_peered":"2023-10-16T10:00:35.633315+0000","last_clean":"2023-10-16T10:00:35.633315+0000","last_became_active":"2023-10-16T10:00:34.638801+0000","last_became_peered":"2023-10-16T10:00:34.638801+0000","last_unstale":"2023-10-16T10:00:35.633315+0000","last_undegraded":"2023-10-16T10:00:35.633315+0000","last_fullsized":"2023-10-16T10:00:35.633315+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:08.396720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633201+0000","last_change":"2023-10-16T10:00:33.618957+0000","last_active":"2023-10-16T10:00:35.633201+0000","last_peered":"2023-10-16T10:00:35.633201+0000","last_clean":"2023-10-16T10:00:35.633201+0000","last_became_active":"2023-10-16T10:00:33.618682+0000","last_became_peered":"2023-10-16T10:00:33.618682+0000","last_unstale":"2023-10-16T10:00:35.633201+0000","last_undegraded":"2023-10-16T10:00:35.633201+0000","last_fullsized":"2023-10-16T10:00:35.633201+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:32.231011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322475+0000","last_change":"2023-10-16T10:00:34.649617+0000","last_active":"2023-10-16T10:00:36.322475+0000","last_peered":"2023-10-16T10:00:36.322475+0000","last_clean":"2023-10-16T10:00:36.322475+0000","last_became_active":"2023-10-16T10:00:34.649284+0000","last_became_peered":"2023-10-16T10:00:34.649284+0000","last_unstale":"2023-10-16T10:00:36.322475+0000","last_undegraded":"2023-10-16T10:00:36.322475+0000","last_fullsized":"2023-10-16T10:00:36.322475+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:36:47.105291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"110'2","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.657141+0000","last_change":"2023-10-16T10:00:33.619365+0000","last_active":"2023-10-16T10:00:34.657141+0000","last_peered":"2023-10-16T10:00:34.657141+0000","last_clean":"2023-10-16T10:00:34.657141+0000","last_became_active":"2023-10-16T10:00:33.618925+0000","last_became_peered":"2023-10-16T10:00:33.618925+0000","last_unstale":"2023-10-16T10:00:34.657141+0000","last_undegraded":"2023-10-16T10:00:34.657141+0000","last_fullsized":"2023-10-16T10:00:34.657141+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:11:46.330111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.659144+0000","last_change":"2023-10-16T10:00:34.658172+0000","last_active":"2023-10-16T10:00:34.659144+0000","last_peered":"2023-10-16T10:00:34.659144+0000","last_clean":"2023-10-16T10:00:34.659144+0000","last_became_active":"2023-10-16T10:00:34.657885+0000","last_became_peered":"2023-10-16T10:00:34.657885+0000","last_unstale":"2023-10-16T10:00:34.659144+0000","last_undegraded":"2023-10-16T10:00:34.659144+0000","last_fullsized":"2023-10-16T10:00:34.659144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:11:47.334435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.636082+0000","last_change":"2023-10-16T10:00:33.617380+0000","last_active":"2023-10-16T10:00:35.636082+0000","last_peered":"2023-10-16T10:00:35.636082+0000","last_clean":"2023-10-16T10:00:35.636082+0000","last_became_active":"2023-10-16T10:00:33.617089+0000","last_became_peered":"2023-10-16T10:00:33.617089+0000","last_unstale":"2023-10-16T10:00:35.636082+0000","last_undegraded":"2023-10-16T10:00:35.636082+0000","last_fullsized":"2023-10-16T10:00:35.636082+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:36.953094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.651822+0000","last_change":"2023-10-16T10:00:34.648773+0000","last_active":"2023-10-16T10:00:34.651822+0000","last_peered":"2023-10-16T10:00:34.651822+0000","last_clean":"2023-10-16T10:00:34.651822+0000","last_became_active":"2023-10-16T10:00:34.648501+0000","last_became_peered":"2023-10-16T10:00:34.648501+0000","last_unstale":"2023-10-16T10:00:34.651822+0000","last_undegraded":"2023-10-16T10:00:34.651822+0000","last_fullsized":"2023-10-16T10:00:34.651822+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.817247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"110'1","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.656593+0000","last_change":"2023-10-16T10:00:33.612826+0000","last_active":"2023-10-16T10:00:34.656593+0000","last_peered":"2023-10-16T10:00:34.656593+0000","last_clean":"2023-10-16T10:00:34.656593+0000","last_became_active":"2023-10-16T10:00:33.612535+0000","last_became_peered":"2023-10-16T10:00:33.612535+0000","last_unstale":"2023-10-16T10:00:34.656593+0000","last_undegraded":"2023-10-16T10:00:34.656593+0000","last_fullsized":"2023-10-16T10:00:34.656593+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:44.114445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322069+0000","last_change":"2023-10-16T10:00:34.639361+0000","last_active":"2023-10-16T10:00:36.322069+0000","last_peered":"2023-10-16T10:00:36.322069+0000","last_clean":"2023-10-16T10:00:36.322069+0000","last_became_active":"2023-10-16T10:00:34.639056+0000","last_became_peered":"2023-10-16T10:00:34.639056+0000","last_unstale":"2023-10-16T10:00:36.322069+0000","last_undegraded":"2023-10-16T10:00:36.322069+0000","last_fullsized":"2023-10-16T10:00:36.322069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:33.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634338+0000","last_change":"2023-10-16T10:00:33.609835+0000","last_active":"2023-10-16T10:00:35.634338+0000","last_peered":"2023-10-16T10:00:35.634338+0000","last_clean":"2023-10-16T10:00:35.634338+0000","last_became_active":"2023-10-16T10:00:33.609534+0000","last_became_peered":"2023-10-16T10:00:33.609534+0000","last_unstale":"2023-10-16T10:00:35.634338+0000","last_undegraded":"2023-10-16T10:00:35.634338+0000","last_fullsized":"2023-10-16T10:00:35.634338+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:30.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.640604+0000","last_change":"2023-10-16T10:00:34.639323+0000","last_active":"2023-10-16T10:00:34.640604+0000","last_peered":"2023-10-16T10:00:34.640604+0000","last_clean":"2023-10-16T10:00:34.640604+0000","last_became_active":"2023-10-16T10:00:34.639040+0000","last_became_peered":"2023-10-16T10:00:34.639040+0000","last_unstale":"2023-10-16T10:00:34.640604+0000","last_undegraded":"2023-10-16T10:00:34.640604+0000","last_fullsized":"2023-10-16T10:00:34.640604+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:21.597660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633544+0000","last_change":"2023-10-16T10:00:33.617489+0000","last_active":"2023-10-16T10:00:35.633544+0000","last_peered":"2023-10-16T10:00:35.633544+0000","last_clean":"2023-10-16T10:00:35.633544+0000","last_became_active":"2023-10-16T10:00:33.617187+0000","last_became_peered":"2023-10-16T10:00:33.617187+0000","last_unstale":"2023-10-16T10:00:35.633544+0000","last_undegraded":"2023-10-16T10:00:35.633544+0000","last_fullsized":"2023-10-16T10:00:35.633544+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:11.654572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635188+0000","last_change":"2023-10-16T10:00:34.651964+0000","last_active":"2023-10-16T10:00:35.635188+0000","last_peered":"2023-10-16T10:00:35.635188+0000","last_clean":"2023-10-16T10:00:35.635188+0000","last_became_active":"2023-10-16T10:00:34.651432+0000","last_became_peered":"2023-10-16T10:00:34.651432+0000","last_unstale":"2023-10-16T10:00:35.635188+0000","last_undegraded":"2023-10-16T10:00:35.635188+0000","last_fullsized":"2023-10-16T10:00:35.635188+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:53.888942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860429+0000","last_change":"2023-10-16T10:00:33.621520+0000","last_active":"2023-10-16T10:00:35.860429+0000","last_peered":"2023-10-16T10:00:35.860429+0000","last_clean":"2023-10-16T10:00:35.860429+0000","last_became_active":"2023-10-16T10:00:33.621238+0000","last_became_peered":"2023-10-16T10:00:33.621238+0000","last_unstale":"2023-10-16T10:00:35.860429+0000","last_undegraded":"2023-10-16T10:00:35.860429+0000","last_fullsized":"2023-10-16T10:00:35.860429+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:21:05.422515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633209+0000","last_change":"2023-10-16T10:00:34.637725+0000","last_active":"2023-10-16T10:00:35.633209+0000","last_peered":"2023-10-16T10:00:35.633209+0000","last_clean":"2023-10-16T10:00:35.633209+0000","last_became_active":"2023-10-16T10:00:34.637429+0000","last_became_peered":"2023-10-16T10:00:34.637429+0000","last_unstale":"2023-10-16T10:00:35.633209+0000","last_undegraded":"2023-10-16T10:00:35.633209+0000","last_fullsized":"2023-10-16T10:00:35.633209+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634568+0000","last_change":"2023-10-16T10:00:33.616300+0000","last_active":"2023-10-16T10:00:35.634568+0000","last_peered":"2023-10-16T10:00:35.634568+0000","last_clean":"2023-10-16T10:00:35.634568+0000","last_became_active":"2023-10-16T10:00:33.615871+0000","last_became_peered":"2023-10-16T10:00:33.615871+0000","last_unstale":"2023-10-16T10:00:35.634568+0000","last_undegraded":"2023-10-16T10:00:35.634568+0000","last_fullsized":"2023-10-16T10:00:35.634568+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634475+0000","last_change":"2023-10-16T10:00:34.652588+0000","last_active":"2023-10-16T10:00:35.634475+0000","last_peered":"2023-10-16T10:00:35.634475+0000","last_clean":"2023-10-16T10:00:35.634475+0000","last_became_active":"2023-10-16T10:00:34.652285+0000","last_became_peered":"2023-10-16T10:00:34.652285+0000","last_unstale":"2023-10-16T10:00:35.634475+0000","last_undegraded":"2023-10-16T10:00:35.634475+0000","last_fullsized":"2023-10-16T10:00:35.634475+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"110'3","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635439+0000","last_change":"2023-10-16T10:00:33.619846+0000","last_active":"2023-10-16T10:00:35.635439+0000","last_peered":"2023-10-16T10:00:35.635439+0000","last_clean":"2023-10-16T10:00:35.635439+0000","last_became_active":"2023-10-16T10:00:33.619562+0000","last_became_peered":"2023-10-16T10:00:33.619562+0000","last_unstale":"2023-10-16T10:00:35.635439+0000","last_undegraded":"2023-10-16T10:00:35.635439+0000","last_fullsized":"2023-10-16T10:00:35.635439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:52.062191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.636725+0000","last_change":"2023-10-16T10:00:34.633749+0000","last_active":"2023-10-16T10:00:34.636725+0000","last_peered":"2023-10-16T10:00:34.636725+0000","last_clean":"2023-10-16T10:00:34.636725+0000","last_became_active":"2023-10-16T10:00:34.633473+0000","last_became_peered":"2023-10-16T10:00:34.633473+0000","last_unstale":"2023-10-16T10:00:34.636725+0000","last_undegraded":"2023-10-16T10:00:34.636725+0000","last_fullsized":"2023-10-16T10:00:34.636725+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:37.197226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"110'6","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322336+0000","last_change":"2023-10-16T10:00:33.612790+0000","last_active":"2023-10-16T10:00:36.322336+0000","last_peered":"2023-10-16T10:00:36.322336+0000","last_clean":"2023-10-16T10:00:36.322336+0000","last_became_active":"2023-10-16T10:00:33.612503+0000","last_became_peered":"2023-10-16T10:00:33.612503+0000","last_unstale":"2023-10-16T10:00:36.322336+0000","last_undegraded":"2023-10-16T10:00:36.322336+0000","last_fullsized":"2023-10-16T10:00:36.322336+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:16.904124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.636068+0000","last_change":"2023-10-16T10:00:34.631460+0000","last_active":"2023-10-16T10:00:34.636068+0000","last_peered":"2023-10-16T10:00:34.636068+0000","last_clean":"2023-10-16T10:00:34.636068+0000","last_became_active":"2023-10-16T10:00:34.631163+0000","last_became_peered":"2023-10-16T10:00:34.631163+0000","last_unstale":"2023-10-16T10:00:34.636068+0000","last_undegraded":"2023-10-16T10:00:34.636068+0000","last_fullsized":"2023-10-16T10:00:34.636068+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.817247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633398+0000","last_change":"2023-10-16T10:00:33.617695+0000","last_active":"2023-10-16T10:00:35.633398+0000","last_peered":"2023-10-16T10:00:35.633398+0000","last_clean":"2023-10-16T10:00:35.633398+0000","last_became_active":"2023-10-16T10:00:33.617418+0000","last_became_peered":"2023-10-16T10:00:33.617418+0000","last_unstale":"2023-10-16T10:00:35.633398+0000","last_undegraded":"2023-10-16T10:00:35.633398+0000","last_fullsized":"2023-10-16T10:00:35.633398+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:07.844585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634884+0000","last_change":"2023-10-16T10:00:34.641842+0000","last_active":"2023-10-16T10:00:35.634884+0000","last_peered":"2023-10-16T10:00:35.634884+0000","last_clean":"2023-10-16T10:00:35.634884+0000","last_became_active":"2023-10-16T10:00:34.641514+0000","last_became_peered":"2023-10-16T10:00:34.641514+0000","last_unstale":"2023-10-16T10:00:35.634884+0000","last_undegraded":"2023-10-16T10:00:35.634884+0000","last_fullsized":"2023-10-16T10:00:35.634884+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:37.197226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322175+0000","last_change":"2023-10-16T10:00:33.613435+0000","last_active":"2023-10-16T10:00:36.322175+0000","last_peered":"2023-10-16T10:00:36.322175+0000","last_clean":"2023-10-16T10:00:36.322175+0000","last_became_active":"2023-10-16T10:00:33.613110+0000","last_became_peered":"2023-10-16T10:00:33.613110+0000","last_unstale":"2023-10-16T10:00:36.322175+0000","last_undegraded":"2023-10-16T10:00:36.322175+0000","last_fullsized":"2023-10-16T10:00:36.322175+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:07.844585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.633719+0000","last_change":"2023-10-16T10:00:34.632468+0000","last_active":"2023-10-16T10:00:34.633719+0000","last_peered":"2023-10-16T10:00:34.633719+0000","last_clean":"2023-10-16T10:00:34.633719+0000","last_became_active":"2023-10-16T10:00:34.632145+0000","last_became_peered":"2023-10-16T10:00:34.632145+0000","last_unstale":"2023-10-16T10:00:34.633719+0000","last_undegraded":"2023-10-16T10:00:34.633719+0000","last_fullsized":"2023-10-16T10:00:34.633719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:56.475229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.627196+0000","last_change":"2023-10-16T10:00:33.612790+0000","last_active":"2023-10-16T10:00:34.627196+0000","last_peered":"2023-10-16T10:00:34.627196+0000","last_clean":"2023-10-16T10:00:34.627196+0000","last_became_active":"2023-10-16T10:00:33.612516+0000","last_became_peered":"2023-10-16T10:00:33.612516+0000","last_unstale":"2023-10-16T10:00:34.627196+0000","last_undegraded":"2023-10-16T10:00:34.627196+0000","last_fullsized":"2023-10-16T10:00:34.627196+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:27.154632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.633496+0000","last_change":"2023-10-16T10:00:34.631459+0000","last_active":"2023-10-16T10:00:34.633496+0000","last_peered":"2023-10-16T10:00:34.633496+0000","last_clean":"2023-10-16T10:00:34.633496+0000","last_became_active":"2023-10-16T10:00:34.631133+0000","last_became_peered":"2023-10-16T10:00:34.631133+0000","last_unstale":"2023-10-16T10:00:34.633496+0000","last_undegraded":"2023-10-16T10:00:34.633496+0000","last_fullsized":"2023-10-16T10:00:34.633496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:42.313895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"110'8","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.658010+0000","last_change":"2023-10-16T10:00:33.615584+0000","last_active":"2023-10-16T10:00:34.658010+0000","last_peered":"2023-10-16T10:00:34.658010+0000","last_clean":"2023-10-16T10:00:34.658010+0000","last_became_active":"2023-10-16T10:00:33.615281+0000","last_became_peered":"2023-10-16T10:00:33.615281+0000","last_unstale":"2023-10-16T10:00:34.658010+0000","last_undegraded":"2023-10-16T10:00:34.658010+0000","last_fullsized":"2023-10-16T10:00:34.658010+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:52.062191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322224+0000","last_change":"2023-10-16T10:00:34.642445+0000","last_active":"2023-10-16T10:00:36.322224+0000","last_peered":"2023-10-16T10:00:36.322224+0000","last_clean":"2023-10-16T10:00:36.322224+0000","last_became_active":"2023-10-16T10:00:34.642149+0000","last_became_peered":"2023-10-16T10:00:34.642149+0000","last_unstale":"2023-10-16T10:00:36.322224+0000","last_undegraded":"2023-10-16T10:00:36.322224+0000","last_fullsized":"2023-10-16T10:00:36.322224+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"110'1","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.655516+0000","last_change":"2023-10-16T10:00:33.620935+0000","last_active":"2023-10-16T10:00:34.655516+0000","last_peered":"2023-10-16T10:00:34.655516+0000","last_clean":"2023-10-16T10:00:34.655516+0000","last_became_active":"2023-10-16T10:00:33.620663+0000","last_became_peered":"2023-10-16T10:00:33.620663+0000","last_unstale":"2023-10-16T10:00:34.655516+0000","last_undegraded":"2023-10-16T10:00:34.655516+0000","last_fullsized":"2023-10-16T10:00:34.655516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:33.754260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.653673+0000","last_change":"2023-10-16T10:00:34.652357+0000","last_active":"2023-10-16T10:00:34.653673+0000","last_peered":"2023-10-16T10:00:34.653673+0000","last_clean":"2023-10-16T10:00:34.653673+0000","last_became_active":"2023-10-16T10:00:34.651972+0000","last_became_peered":"2023-10-16T10:00:34.651972+0000","last_unstale":"2023-10-16T10:00:34.653673+0000","last_undegraded":"2023-10-16T10:00:34.653673+0000","last_fullsized":"2023-10-16T10:00:34.653673+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:34.437565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633422+0000","last_change":"2023-10-16T10:00:33.619445+0000","last_active":"2023-10-16T10:00:35.633422+0000","last_peered":"2023-10-16T10:00:35.633422+0000","last_clean":"2023-10-16T10:00:35.633422+0000","last_became_active":"2023-10-16T10:00:33.619172+0000","last_became_peered":"2023-10-16T10:00:33.619172+0000","last_unstale":"2023-10-16T10:00:35.633422+0000","last_undegraded":"2023-10-16T10:00:35.633422+0000","last_fullsized":"2023-10-16T10:00:35.633422+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635398+0000","last_change":"2023-10-16T10:00:34.640882+0000","last_active":"2023-10-16T10:00:35.635398+0000","last_peered":"2023-10-16T10:00:35.635398+0000","last_clean":"2023-10-16T10:00:35.635398+0000","last_became_active":"2023-10-16T10:00:34.640590+0000","last_became_peered":"2023-10-16T10:00:34.640590+0000","last_unstale":"2023-10-16T10:00:35.635398+0000","last_undegraded":"2023-10-16T10:00:35.635398+0000","last_fullsized":"2023-10-16T10:00:35.635398+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:02.945267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635172+0000","last_change":"2023-10-16T10:00:33.618426+0000","last_active":"2023-10-16T10:00:35.635172+0000","last_peered":"2023-10-16T10:00:35.635172+0000","last_clean":"2023-10-16T10:00:35.635172+0000","last_became_active":"2023-10-16T10:00:33.617984+0000","last_became_peered":"2023-10-16T10:00:33.617984+0000","last_unstale":"2023-10-16T10:00:35.635172+0000","last_undegraded":"2023-10-16T10:00:35.635172+0000","last_fullsized":"2023-10-16T10:00:35.635172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:33.754260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322262+0000","last_change":"2023-10-16T10:00:34.641853+0000","last_active":"2023-10-16T10:00:36.322262+0000","last_peered":"2023-10-16T10:00:36.322262+0000","last_clean":"2023-10-16T10:00:36.322262+0000","last_became_active":"2023-10-16T10:00:34.640494+0000","last_became_peered":"2023-10-16T10:00:34.640494+0000","last_unstale":"2023-10-16T10:00:36.322262+0000","last_undegraded":"2023-10-16T10:00:36.322262+0000","last_fullsized":"2023-10-16T10:00:36.322262+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:32.443386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860102+0000","last_change":"2023-10-16T10:00:33.619469+0000","last_active":"2023-10-16T10:00:35.860102+0000","last_peered":"2023-10-16T10:00:35.860102+0000","last_clean":"2023-10-16T10:00:35.860102+0000","last_became_active":"2023-10-16T10:00:33.619102+0000","last_became_peered":"2023-10-16T10:00:33.619102+0000","last_unstale":"2023-10-16T10:00:35.860102+0000","last_undegraded":"2023-10-16T10:00:35.860102+0000","last_fullsized":"2023-10-16T10:00:35.860102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:22.006766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860003+0000","last_change":"2023-10-16T10:00:34.657621+0000","last_active":"2023-10-16T10:00:35.860003+0000","last_peered":"2023-10-16T10:00:35.860003+0000","last_clean":"2023-10-16T10:00:35.860003+0000","last_became_active":"2023-10-16T10:00:34.657316+0000","last_became_peered":"2023-10-16T10:00:34.657316+0000","last_unstale":"2023-10-16T10:00:35.860003+0000","last_undegraded":"2023-10-16T10:00:35.860003+0000","last_fullsized":"2023-10-16T10:00:35.860003+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:33.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.626243+0000","last_change":"2023-10-16T10:00:33.614479+0000","last_active":"2023-10-16T10:00:34.626243+0000","last_peered":"2023-10-16T10:00:34.626243+0000","last_clean":"2023-10-16T10:00:34.626243+0000","last_became_active":"2023-10-16T10:00:33.614267+0000","last_became_peered":"2023-10-16T10:00:33.614267+0000","last_unstale":"2023-10-16T10:00:34.626243+0000","last_undegraded":"2023-10-16T10:00:34.626243+0000","last_fullsized":"2023-10-16T10:00:34.626243+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:20.777695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860091+0000","last_change":"2023-10-16T10:00:34.640131+0000","last_active":"2023-10-16T10:00:35.860091+0000","last_peered":"2023-10-16T10:00:35.860091+0000","last_clean":"2023-10-16T10:00:35.860091+0000","last_became_active":"2023-10-16T10:00:34.639455+0000","last_became_peered":"2023-10-16T10:00:34.639455+0000","last_unstale":"2023-10-16T10:00:35.860091+0000","last_undegraded":"2023-10-16T10:00:35.860091+0000","last_fullsized":"2023-10-16T10:00:35.860091+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:36.973289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860177+0000","last_change":"2023-10-16T10:00:33.618623+0000","last_active":"2023-10-16T10:00:35.860177+0000","last_peered":"2023-10-16T10:00:35.860177+0000","last_clean":"2023-10-16T10:00:35.860177+0000","last_became_active":"2023-10-16T10:00:33.618227+0000","last_became_peered":"2023-10-16T10:00:33.618227+0000","last_unstale":"2023-10-16T10:00:35.860177+0000","last_undegraded":"2023-10-16T10:00:35.860177+0000","last_fullsized":"2023-10-16T10:00:35.860177+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:26.257889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322408+0000","last_change":"2023-10-16T10:00:34.640393+0000","last_active":"2023-10-16T10:00:36.322408+0000","last_peered":"2023-10-16T10:00:36.322408+0000","last_clean":"2023-10-16T10:00:36.322408+0000","last_became_active":"2023-10-16T10:00:34.639742+0000","last_became_peered":"2023-10-16T10:00:34.639742+0000","last_unstale":"2023-10-16T10:00:36.322408+0000","last_undegraded":"2023-10-16T10:00:36.322408+0000","last_fullsized":"2023-10-16T10:00:36.322408+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:19.030816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.625707+0000","last_change":"2023-10-16T10:00:33.620235+0000","last_active":"2023-10-16T10:00:34.625707+0000","last_peered":"2023-10-16T10:00:34.625707+0000","last_clean":"2023-10-16T10:00:34.625707+0000","last_became_active":"2023-10-16T10:00:33.619956+0000","last_became_peered":"2023-10-16T10:00:33.619956+0000","last_unstale":"2023-10-16T10:00:34.625707+0000","last_undegraded":"2023-10-16T10:00:34.625707+0000","last_fullsized":"2023-10-16T10:00:34.625707+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:33:56.684855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.652683+0000","last_change":"2023-10-16T10:00:34.645655+0000","last_active":"2023-10-16T10:00:34.652683+0000","last_peered":"2023-10-16T10:00:34.652683+0000","last_clean":"2023-10-16T10:00:34.652683+0000","last_became_active":"2023-10-16T10:00:34.645392+0000","last_became_peered":"2023-10-16T10:00:34.645392+0000","last_unstale":"2023-10-16T10:00:34.652683+0000","last_undegraded":"2023-10-16T10:00:34.652683+0000","last_fullsized":"2023-10-16T10:00:34.652683+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:45.118769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634062+0000","last_change":"2023-10-16T10:00:33.622451+0000","last_active":"2023-10-16T10:00:35.634062+0000","last_peered":"2023-10-16T10:00:35.634062+0000","last_clean":"2023-10-16T10:00:35.634062+0000","last_became_active":"2023-10-16T10:00:33.622173+0000","last_became_peered":"2023-10-16T10:00:33.622173+0000","last_unstale":"2023-10-16T10:00:35.634062+0000","last_undegraded":"2023-10-16T10:00:35.634062+0000","last_fullsized":"2023-10-16T10:00:35.634062+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:30.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322490+0000","last_change":"2023-10-16T10:00:34.640516+0000","last_active":"2023-10-16T10:00:36.322490+0000","last_peered":"2023-10-16T10:00:36.322490+0000","last_clean":"2023-10-16T10:00:36.322490+0000","last_became_active":"2023-10-16T10:00:34.639964+0000","last_became_peered":"2023-10-16T10:00:34.639964+0000","last_unstale":"2023-10-16T10:00:36.322490+0000","last_undegraded":"2023-10-16T10:00:36.322490+0000","last_fullsized":"2023-10-16T10:00:36.322490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:16.261934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635483+0000","last_change":"2023-10-16T10:00:33.620773+0000","last_active":"2023-10-16T10:00:35.635483+0000","last_peered":"2023-10-16T10:00:35.635483+0000","last_clean":"2023-10-16T10:00:35.635483+0000","last_became_active":"2023-10-16T10:00:33.620474+0000","last_became_peered":"2023-10-16T10:00:33.620474+0000","last_unstale":"2023-10-16T10:00:35.635483+0000","last_undegraded":"2023-10-16T10:00:35.635483+0000","last_fullsized":"2023-10-16T10:00:35.635483+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:46.702571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322360+0000","last_change":"2023-10-16T10:00:34.641261+0000","last_active":"2023-10-16T10:00:36.322360+0000","last_peered":"2023-10-16T10:00:36.322360+0000","last_clean":"2023-10-16T10:00:36.322360+0000","last_became_active":"2023-10-16T10:00:34.640291+0000","last_became_peered":"2023-10-16T10:00:34.640291+0000","last_unstale":"2023-10-16T10:00:36.322360+0000","last_undegraded":"2023-10-16T10:00:36.322360+0000","last_fullsized":"2023-10-16T10:00:36.322360+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:45:58.597170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635534+0000","last_change":"2023-10-16T10:00:33.618247+0000","last_active":"2023-10-16T10:00:35.635534+0000","last_peered":"2023-10-16T10:00:35.635534+0000","last_clean":"2023-10-16T10:00:35.635534+0000","last_became_active":"2023-10-16T10:00:33.617621+0000","last_became_peered":"2023-10-16T10:00:33.617621+0000","last_unstale":"2023-10-16T10:00:35.635534+0000","last_undegraded":"2023-10-16T10:00:35.635534+0000","last_fullsized":"2023-10-16T10:00:35.635534+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:34.424345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860375+0000","last_change":"2023-10-16T10:00:34.639499+0000","last_active":"2023-10-16T10:00:35.860375+0000","last_peered":"2023-10-16T10:00:35.860375+0000","last_clean":"2023-10-16T10:00:35.860375+0000","last_became_active":"2023-10-16T10:00:34.639131+0000","last_became_peered":"2023-10-16T10:00:34.639131+0000","last_unstale":"2023-10-16T10:00:35.860375+0000","last_undegraded":"2023-10-16T10:00:35.860375+0000","last_fullsized":"2023-10-16T10:00:35.860375+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.635932+0000","last_change":"2023-10-16T10:00:33.618243+0000","last_active":"2023-10-16T10:00:34.635932+0000","last_peered":"2023-10-16T10:00:34.635932+0000","last_clean":"2023-10-16T10:00:34.635932+0000","last_became_active":"2023-10-16T10:00:33.617950+0000","last_became_peered":"2023-10-16T10:00:33.617950+0000","last_unstale":"2023-10-16T10:00:34.635932+0000","last_undegraded":"2023-10-16T10:00:34.635932+0000","last_fullsized":"2023-10-16T10:00:34.635932+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:27.154632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.644617+0000","last_change":"2023-10-16T10:00:34.642974+0000","last_active":"2023-10-16T10:00:34.644617+0000","last_peered":"2023-10-16T10:00:34.644617+0000","last_clean":"2023-10-16T10:00:34.644617+0000","last_became_active":"2023-10-16T10:00:34.642662+0000","last_became_peered":"2023-10-16T10:00:34.642662+0000","last_unstale":"2023-10-16T10:00:34.644617+0000","last_undegraded":"2023-10-16T10:00:34.644617+0000","last_fullsized":"2023-10-16T10:00:34.644617+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:41.332741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322446+0000","last_change":"2023-10-16T10:00:33.620943+0000","last_active":"2023-10-16T10:00:36.322446+0000","last_peered":"2023-10-16T10:00:36.322446+0000","last_clean":"2023-10-16T10:00:36.322446+0000","last_became_active":"2023-10-16T10:00:33.620634+0000","last_became_peered":"2023-10-16T10:00:33.620634+0000","last_unstale":"2023-10-16T10:00:36.322446+0000","last_undegraded":"2023-10-16T10:00:36.322446+0000","last_fullsized":"2023-10-16T10:00:36.322446+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634303+0000","last_change":"2023-10-16T10:00:34.653328+0000","last_active":"2023-10-16T10:00:35.634303+0000","last_peered":"2023-10-16T10:00:35.634303+0000","last_clean":"2023-10-16T10:00:35.634303+0000","last_became_active":"2023-10-16T10:00:34.653072+0000","last_became_peered":"2023-10-16T10:00:34.653072+0000","last_unstale":"2023-10-16T10:00:35.634303+0000","last_undegraded":"2023-10-16T10:00:35.634303+0000","last_fullsized":"2023-10-16T10:00:35.634303+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:42:17.438735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860077+0000","last_change":"2023-10-16T10:00:33.621129+0000","last_active":"2023-10-16T10:00:35.860077+0000","last_peered":"2023-10-16T10:00:35.860077+0000","last_clean":"2023-10-16T10:00:35.860077+0000","last_became_active":"2023-10-16T10:00:33.619822+0000","last_became_peered":"2023-10-16T10:00:33.619822+0000","last_unstale":"2023-10-16T10:00:35.860077+0000","last_undegraded":"2023-10-16T10:00:35.860077+0000","last_fullsized":"2023-10-16T10:00:35.860077+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.645665+0000","last_change":"2023-10-16T10:00:34.638976+0000","last_active":"2023-10-16T10:00:34.645665+0000","last_peered":"2023-10-16T10:00:34.645665+0000","last_clean":"2023-10-16T10:00:34.645665+0000","last_became_active":"2023-10-16T10:00:34.638729+0000","last_became_peered":"2023-10-16T10:00:34.638729+0000","last_unstale":"2023-10-16T10:00:34.645665+0000","last_undegraded":"2023-10-16T10:00:34.645665+0000","last_fullsized":"2023-10-16T10:00:34.645665+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:27.581985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.635645+0000","last_change":"2023-10-16T10:00:33.618355+0000","last_active":"2023-10-16T10:00:34.635645+0000","last_peered":"2023-10-16T10:00:34.635645+0000","last_clean":"2023-10-16T10:00:34.635645+0000","last_became_active":"2023-10-16T10:00:33.617839+0000","last_became_peered":"2023-10-16T10:00:33.617839+0000","last_unstale":"2023-10-16T10:00:34.635645+0000","last_undegraded":"2023-10-16T10:00:34.635645+0000","last_fullsized":"2023-10-16T10:00:34.635645+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:20.593335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322341+0000","last_change":"2023-10-16T10:00:34.656620+0000","last_active":"2023-10-16T10:00:36.322341+0000","last_peered":"2023-10-16T10:00:36.322341+0000","last_clean":"2023-10-16T10:00:36.322341+0000","last_became_active":"2023-10-16T10:00:34.656363+0000","last_became_peered":"2023-10-16T10:00:34.656363+0000","last_unstale":"2023-10-16T10:00:36.322341+0000","last_undegraded":"2023-10-16T10:00:36.322341+0000","last_fullsized":"2023-10-16T10:00:36.322341+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:14.159457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633652+0000","last_change":"2023-10-16T10:00:33.616467+0000","last_active":"2023-10-16T10:00:35.633652+0000","last_peered":"2023-10-16T10:00:35.633652+0000","last_clean":"2023-10-16T10:00:35.633652+0000","last_became_active":"2023-10-16T10:00:33.616180+0000","last_became_peered":"2023-10-16T10:00:33.616180+0000","last_unstale":"2023-10-16T10:00:35.633652+0000","last_undegraded":"2023-10-16T10:00:35.633652+0000","last_fullsized":"2023-10-16T10:00:35.633652+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:22.006766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633971+0000","last_change":"2023-10-16T10:00:34.653674+0000","last_active":"2023-10-16T10:00:35.633971+0000","last_peered":"2023-10-16T10:00:35.633971+0000","last_clean":"2023-10-16T10:00:35.633971+0000","last_became_active":"2023-10-16T10:00:34.653381+0000","last_became_peered":"2023-10-16T10:00:34.653381+0000","last_unstale":"2023-10-16T10:00:35.633971+0000","last_undegraded":"2023-10-16T10:00:35.633971+0000","last_fullsized":"2023-10-16T10:00:35.633971+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:14.159457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.628900+0000","last_change":"2023-10-16T10:00:33.619788+0000","last_active":"2023-10-16T10:00:34.628900+0000","last_peered":"2023-10-16T10:00:34.628900+0000","last_clean":"2023-10-16T10:00:34.628900+0000","last_became_active":"2023-10-16T10:00:33.619491+0000","last_became_peered":"2023-10-16T10:00:33.619491+0000","last_unstale":"2023-10-16T10:00:34.628900+0000","last_undegraded":"2023-10-16T10:00:34.628900+0000","last_fullsized":"2023-10-16T10:00:34.628900+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:14:18.783582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"101'183","reported_seq":489,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.624567+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:00:34.624567+0000","last_peered":"2023-10-16T10:00:34.624567+0000","last_clean":"2023-10-16T10:00:34.624567+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:00:34.624567+0000","last_undegraded":"2023-10-16T10:00:34.624567+0000","last_fullsized":"2023-10-16T10:00:34.624567+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575054,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59472,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312368,"statfs":{"total":96636764160,"available":96575864832,"internally_reserved":0,"allocated":1970176,"data_stored":1299002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.56100000000000005},"min":{"1min":0.36499999999999999,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":0.73299999999999998,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.67700000000000005,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.911}]},{"osd":1,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.442,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70399999999999996,"5min":0.79300000000000004,"15min":1.1439999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.46500000000000002,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.78100000000000003,"5min":1.1279999999999999,"15min":1.212},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.63100000000000001},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.81599999999999995,"5min":1.607,"15min":1.607},"last":3.3820000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61399999999999999,"15min":0.63400000000000001},"min":{"1min":0.432,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":2.044,"15min":2.044},"last":3.1640000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58399999999999996,"15min":0.60099999999999998},"min":{"1min":0.42799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.755,"5min":1.524,"15min":1.524},"last":2.0979999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.50700000000000001,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.70399999999999996,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":2.2029999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.8220000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68600000000000005,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":3.8149999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73899999999999999,"5min":0.81100000000000005,"15min":0.875},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.37,"5min":0.33200000000000002,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.6619999999999999}]},{"osd":6,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.76000000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.438,"5min":0.311,"15min":0.311},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.85299999999999998},"last":0.98699999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575054,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1376256,"data_stored":703750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55600000000000005,"15min":0.58299999999999996},"min":{"1min":0.29699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.68200000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57299999999999995,"15min":0.59299999999999997},"min":{"1min":0.38100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.76900000000000002,"5min":0.78800000000000003,"15min":0.82099999999999995},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.36099999999999999,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.72199999999999998,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56399999999999995,"15min":0.58199999999999996},"min":{"1min":0.33100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.749,"5min":1.448,"15min":1.448},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":1.708,"15min":1.708},"last":0.63}]},{"osd":3,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.76000000000000001,"15min":0.81699999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.82999999999999996},"last":0.60199999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.83999999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.81100000000000005},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.253,"15min":0.253},"max":{"1min":0.81100000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.496},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53700000000000003,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.872},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.79100000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575055,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":701604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.38800000000000001,"15min":0.39500000000000002},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":0.52200000000000002,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.51000000000000001,"5min":0.67900000000000005,"15min":0.70199999999999996},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.43099999999999999,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54000000000000004,"5min":4.923,"15min":4.923},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.216},"max":{"1min":0.58699999999999997,"5min":1.712,"15min":1.712},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46400000000000002,"15min":0.45600000000000002},"min":{"1min":0.32100000000000001,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.66500000000000004,"5min":1.984,"15min":1.984},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.41999999999999998,"15min":0.42499999999999999},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.56699999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.379}]},{"osd":4,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61499999999999999,"15min":0.628},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67700000000000005,"15min":0.66500000000000004},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.85899999999999999,"5min":4.9749999999999996,"15min":4.9749999999999996},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77400000000000002,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.80700000000000005,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.68100000000000005,"15min":0.65700000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.82899999999999996,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.45800000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.75900000000000001,"5min":1.744,"15min":1.744},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38500000000000001},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.93400000000000005},"last":0.505}]}]},{"osd":0,"up_from":13,"seq":55834575055,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":1974272,"data_stored":1294220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39200000000000002,"15min":0.38300000000000001},"min":{"1min":0.27100000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.442},{"interface":"front","average":{"1min":0.376,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":0.55600000000000005,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53000000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.34799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.63600000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.5}]},{"osd":3,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.224},"max":{"1min":0.57899999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46400000000000002},"min":{"1min":0.27700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59699999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.437,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.77300000000000002,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":0.81100000000000005,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61199999999999999,"15min":0.63500000000000001},"min":{"1min":0.39200000000000002,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.94999999999999996,"15min":0.998},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.82599999999999996,"5min":0.88700000000000001,"15min":0.93000000000000005},"last":0.626}]},{"osd":6,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84999999999999998,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":1.014,"15min":1.02},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.66000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575054,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58880,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312960,"statfs":{"total":96636764160,"available":96576471040,"internally_reserved":0,"allocated":1355776,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.51400000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.53800000000000003,"5min":0.64300000000000002,"15min":0.67500000000000004},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.61899999999999999,"5min":0.68000000000000005,"15min":0.754},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.25900000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40799999999999997,"15min":0.41999999999999998},"min":{"1min":0.29999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64800000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.374}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.64500000000000002},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.103},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.65700000000000003},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63100000000000001,"15min":0.64900000000000002},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.046,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65700000000000003,"15min":0.68300000000000005},"min":{"1min":0.441,"5min":0.40799999999999997,"15min":0.40100000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65800000000000003,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.43099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.0069999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.58999999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575054,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36676,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335164,"statfs":{"total":96636764160,"available":96599207936,"internally_reserved":0,"allocated":1368064,"data_stored":708578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.20999999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.308}]},{"osd":1,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.39400000000000002,"15min":0.39500000000000002},"min":{"1min":0.24099999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.65400000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":0.56799999999999995,"5min":0.61899999999999999,"15min":0.67500000000000004},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":0.59999999999999998,"5min":0.70499999999999996,"15min":1.0760000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.64100000000000001,"5min":0.73999999999999999,"15min":0.93799999999999994},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.504}]},{"osd":5,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.63500000000000001,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89000000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33200000000000002},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":1.036},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64100000000000001,"15min":0.65000000000000002},"min":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.374},"max":{"1min":0.83299999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":1.0589999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42599999999999999},"max":{"1min":0.872,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.42699999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":0.999},"last":0.443}]}]},{"osd":4,"up_from":13,"seq":55834575054,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":701694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.59199999999999997},"min":{"1min":0.34200000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.83099999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.44500000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.72299999999999998,"5min":0.81499999999999995,"15min":0.82599999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.498,"5min":0.56200000000000006,"15min":0.58999999999999997},"min":{"1min":0.33300000000000002,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.66800000000000004,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.622},"min":{"1min":0.438,"5min":0.379,"15min":0.32500000000000001},"max":{"1min":0.79600000000000004,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.499}]},{"osd":3,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75800000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58299999999999996,"15min":0.62},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.84599999999999997},"last":0.66000000000000003}]},{"osd":5,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.749,"5min":0.76400000000000001,"15min":0.84299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.77500000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.84699999999999998,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80100000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.48199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80800000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.72799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575054,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50572,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321268,"statfs":{"total":96636764160,"available":96584978432,"internally_reserved":0,"allocated":1368064,"data_stored":708578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":0.80600000000000005,"5min":1.792,"15min":1.792},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45700000000000002,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.83099999999999996,"15min":0.875},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.57299999999999995},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.496,"15min":1.496},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.72099999999999997,"5min":2.069,"15min":2.069},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46700000000000003,"15min":0.47799999999999998},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66100000000000003,"5min":0.71299999999999997,"15min":0.84199999999999997},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.86899999999999999},"last":0.72099999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48699999999999999,"15min":0.48999999999999999},"min":{"1min":0.312,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.84099999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.88200000000000001},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80500000000000005,"5min":0.81799999999999995,"15min":0.90300000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.79500000000000004,"5min":0.84899999999999998,"15min":0.88500000000000001},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:00:38.759 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:00:38.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.912+0000 7f87e469f700 1 Processor -- start 2023-10-16T10:00:38.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.912+0000 7f87e469f700 1 -- start start 2023-10-16T10:00:38.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.913+0000 7f87e469f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f87dc15dbc0 0x7f87dc0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.913+0000 7f87e469f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 0x7f87dc0cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.913+0000 7f87e469f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87dc0d21e0 con 0x7f87dc0d1d50 2023-10-16T10:00:38.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.913+0000 7f87e469f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87dc162550 con 0x7f87dc15dbc0 2023-10-16T10:00:38.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.913+0000 7f87e469f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc15f320 con 0x7f87dc06ba90 2023-10-16T10:00:38.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1c3a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f87dc0d1d50 0x7f87dc06beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55384/0 (socket says 172.21.15.158:55384) 2023-10-16T10:00:38.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1c3a700 1 -- 172.21.15.158:0/881483897 learned_addr learned my addr 172.21.15.158:0/881483897 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:38.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e243b700 1 --2- 172.21.15.158:0/881483897 >> v2:172.21.15.172:3300/0 conn(0x7f87dc15dbc0 0x7f87dc0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e2c3c700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 0x7f87dc0cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927256765 0 0) 0x7f87dc0d21e0 con 0x7f87dc0d1d50 2023-10-16T10:00:38.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c40035b0 con 0x7f87dc0d1d50 2023-10-16T10:00:38.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1958252905 0 0) 0x7f87c40035b0 con 0x7f87dc0d1d50 2023-10-16T10:00:38.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 msgr2=0x7f87dc0cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:38.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 0x7f87dc0cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 >> v2:172.21.15.172:3300/0 conn(0x7f87dc15dbc0 msgr2=0x7f87dc0cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:38.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 --2- 172.21.15.158:0/881483897 >> v2:172.21.15.172:3300/0 conn(0x7f87dc15dbc0 0x7f87dc0cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc15ef10 con 0x7f87dc0d1d50 2023-10-16T10:00:38.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f87d0001670 con 0x7f87dc0d1d50 2023-10-16T10:00:38.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 >> v1:172.21.15.158:6789/0 conn(0x7f87dc0d1d50 legacy=0x7f87dc06beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 0x7f87c4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4003f80 0x7f87c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87c4004d90 0x7f87c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc162550 con 0x7f87c4004d90 2023-10-16T10:00:38.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c400ba10 con 0x7f87c4003f80 2023-10-16T10:00:38.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c400bba0 con 0x7f87dc06ba90 2023-10-16T10:00:38.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e2c3c700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87c4004d90 0x7f87c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e1c3a700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4003f80 0x7f87c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.914+0000 7f87e243b700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 0x7f87c4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e2c3c700 1 -- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 msgr2=0x7f87c4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:38.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e2c3c700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc06ba90 0x7f87c4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e2c3c700 1 -- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4003f80 msgr2=0x7f87c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:38.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e2c3c700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4003f80 0x7f87c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e2c3c700 1 -- 172.21.15.158:0/881483897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c40035b0 con 0x7f87c4004d90 2023-10-16T10:00:38.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e2c3c700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87c4004d90 0x7f87c4005210 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009fc0 tx=0x7f87c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad28036f075589e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc00ec30 con 0x7f87c4004d90 2023-10-16T10:00:38.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc00f040 con 0x7f87c4004d90 2023-10-16T10:00:38.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e1439700 1 -- 172.21.15.158:0/881483897 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc014480 con 0x7f87c4004d90 2023-10-16T10:00:38.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e469f700 1 -- 172.21.15.158:0/881483897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87c4004d90 msgr2=0x7f87c4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e469f700 1 --2- 172.21.15.158:0/881483897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87c4004d90 0x7f87c4005210 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009fc0 tx=0x7f87c4005880 comp rx=0 tx=0).stop 2023-10-16T10:00:38.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.915+0000 7f87e2c3c700 1 -- 172.21.15.158:0/881483897 reap_dead start 2023-10-16T10:00:38.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.916+0000 7f87e469f700 1 -- 172.21.15.158:0/881483897 shutdown_connections 2023-10-16T10:00:38.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.916+0000 7f87e469f700 1 -- 172.21.15.158:0/881483897 >> 172.21.15.158:0/881483897 conn(0x7f87dc06a8d0 msgr2=0x7f87dc06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:38.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.916+0000 7f87e469f700 1 -- 172.21.15.158:0/881483897 shutdown_connections 2023-10-16T10:00:38.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.916+0000 7f87e469f700 1 -- 172.21.15.158:0/881483897 wait complete. 2023-10-16T10:00:38.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.917+0000 7f87e469f700 1 Processor -- start 2023-10-16T10:00:38.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.917+0000 7f87e469f700 1 -- start start 2023-10-16T10:00:38.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4004d90 0x7f87dc1f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87dc14e7c0 0x7f87dc14e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc0cc0d0 0x7f87dc0cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0cca90 con 0x7f87dc14e7c0 2023-10-16T10:00:38.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1f6fc0 con 0x7f87c4004d90 2023-10-16T10:00:38.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1f4070 con 0x7f87dc0cc0d0 2023-10-16T10:00:38.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87dc14e7c0 0x7f87dc14e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87dc14e7c0 0x7f87dc14e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55220/0 (socket says 172.21.15.158:55220) 2023-10-16T10:00:38.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 -- 172.21.15.158:0/1054860802 learned_addr learned my addr 172.21.15.158:0/1054860802 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:38.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e243b700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4004d90 0x7f87dc1f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 -- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc0cc0d0 msgr2=0x7f87dc0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e2c3c700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc0cc0d0 0x7f87dc0cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc0cc0d0 0x7f87dc0cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 -- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4004d90 msgr2=0x7f87dc1f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:38.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4004d90 0x7f87dc1f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:38.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 -- 172.21.15.158:0/1054860802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc1f4350 con 0x7f87dc14e7c0 2023-10-16T10:00:38.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e243b700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4004d90 0x7f87dc1f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:38.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e2c3c700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc0cc0d0 0x7f87dc0cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:38.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e1c3a700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87dc14e7c0 0x7f87dc14e1c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007340 tx=0x7f87d0007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87daffd700 1 -- 172.21.15.158:0/1054860802 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d000f6c0 con 0x7f87dc14e7c0 2023-10-16T10:00:38.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87daffd700 1 -- 172.21.15.158:0/1054860802 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d000f840 con 0x7f87dc14e7c0 2023-10-16T10:00:38.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc1f4630 con 0x7f87dc14e7c0 2023-10-16T10:00:38.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87daffd700 1 -- 172.21.15.158:0/1054860802 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d000fbf0 con 0x7f87dc14e7c0 2023-10-16T10:00:38.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.918+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc0c71b0 con 0x7f87dc14e7c0 2023-10-16T10:00:38.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.919+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc0681b0 con 0x7f87dc14e7c0 2023-10-16T10:00:38.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.920+0000 7f87daffd700 1 -- 172.21.15.158:0/1054860802 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f87d000fd50 con 0x7f87dc14e7c0 2023-10-16T10:00:38.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.920+0000 7f87daffd700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f87c806b020 0x7f87c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:38.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.920+0000 7f87e243b700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f87c806b020 0x7f87c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:38.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.920+0000 7f87daffd700 1 -- 172.21.15.158:0/1054860802 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f87d0089a80 con 0x7f87dc14e7c0 2023-10-16T10:00:38.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.921+0000 7f87e243b700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f87c806b020 0x7f87c806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009bd0 tx=0x7f87d4008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:38.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:38.925+0000 7f87daffd700 1 -- 172.21.15.158:0/1054860802 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87d00547c0 con 0x7f87dc14e7c0 2023-10-16T10:00:39.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.111+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 --> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87dc15e430 con 0x7f87c806b020 2023-10-16T10:00:39.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.119+0000 7f87daffd700 1 -- 172.21.15.158:0/1054860802 <== mgr.6895 v2:172.21.15.158:6832/1430981267 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289993 (secure 0 0 0) 0x7f87d4004030 con 0x7f87c806b020 2023-10-16T10:00:39.123 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:39.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f87c806b020 msgr2=0x7f87c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f87c806b020 0x7f87c806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009bd0 tx=0x7f87d4008040 comp rx=0 tx=0).stop 2023-10-16T10:00:39.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87dc14e7c0 msgr2=0x7f87dc14e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87dc14e7c0 0x7f87dc14e1c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007340 tx=0x7f87d0007650 comp rx=0 tx=0).stop 2023-10-16T10:00:39.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 shutdown_connections 2023-10-16T10:00:39.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f87c806b020 0x7f87c806d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87dc0cc0d0 0x7f87dc0cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87c4004d90 0x7f87dc1f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 --2- 172.21.15.158:0/1054860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87dc14e7c0 0x7f87dc14e1c0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 >> 172.21.15.158:0/1054860802 conn(0x7f87dc06a8d0 msgr2=0x7f87dc0ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:39.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 shutdown_connections 2023-10-16T10:00:39.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.122+0000 7f87e469f700 1 -- 172.21.15.158:0/1054860802 wait complete. 2023-10-16T10:00:39.128 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:00:39.136 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T10:00:38.332352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336260,"kb_used_data":11852,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754638460,"statfs":{"total":773094113280,"available":772749783040,"internally_reserved":0,"allocated":12136448,"data_stored":6818960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":332058167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.737801"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634490+0000","last_change":"2023-10-16T10:00:34.651969+0000","last_active":"2023-10-16T10:00:35.634490+0000","last_peered":"2023-10-16T10:00:35.634490+0000","last_clean":"2023-10-16T10:00:35.634490+0000","last_became_active":"2023-10-16T10:00:34.651692+0000","last_became_peered":"2023-10-16T10:00:34.651692+0000","last_unstale":"2023-10-16T10:00:35.634490+0000","last_undegraded":"2023-10-16T10:00:35.634490+0000","last_fullsized":"2023-10-16T10:00:35.634490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:32.443386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860161+0000","last_change":"2023-10-16T10:00:34.653193+0000","last_active":"2023-10-16T10:00:35.860161+0000","last_peered":"2023-10-16T10:00:35.860161+0000","last_clean":"2023-10-16T10:00:35.860161+0000","last_became_active":"2023-10-16T10:00:34.652878+0000","last_became_peered":"2023-10-16T10:00:34.652878+0000","last_unstale":"2023-10-16T10:00:35.860161+0000","last_undegraded":"2023-10-16T10:00:35.860161+0000","last_fullsized":"2023-10-16T10:00:35.860161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:42:17.438735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634280+0000","last_change":"2023-10-16T10:00:34.647330+0000","last_active":"2023-10-16T10:00:35.634280+0000","last_peered":"2023-10-16T10:00:35.634280+0000","last_clean":"2023-10-16T10:00:35.634280+0000","last_became_active":"2023-10-16T10:00:34.640824+0000","last_became_peered":"2023-10-16T10:00:34.640824+0000","last_unstale":"2023-10-16T10:00:35.634280+0000","last_undegraded":"2023-10-16T10:00:35.634280+0000","last_fullsized":"2023-10-16T10:00:35.634280+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:36.973289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.645600+0000","last_change":"2023-10-16T10:00:34.642790+0000","last_active":"2023-10-16T10:00:34.645600+0000","last_peered":"2023-10-16T10:00:34.645600+0000","last_clean":"2023-10-16T10:00:34.645600+0000","last_became_active":"2023-10-16T10:00:34.642492+0000","last_became_peered":"2023-10-16T10:00:34.642492+0000","last_unstale":"2023-10-16T10:00:34.645600+0000","last_undegraded":"2023-10-16T10:00:34.645600+0000","last_fullsized":"2023-10-16T10:00:34.645600+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:34.758584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635724+0000","last_change":"2023-10-16T10:00:34.641813+0000","last_active":"2023-10-16T10:00:35.635724+0000","last_peered":"2023-10-16T10:00:35.635724+0000","last_clean":"2023-10-16T10:00:35.635724+0000","last_became_active":"2023-10-16T10:00:34.641535+0000","last_became_peered":"2023-10-16T10:00:34.641535+0000","last_unstale":"2023-10-16T10:00:35.635724+0000","last_undegraded":"2023-10-16T10:00:35.635724+0000","last_fullsized":"2023-10-16T10:00:35.635724+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:44.743191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322064+0000","last_change":"2023-10-16T10:00:34.641617+0000","last_active":"2023-10-16T10:00:36.322064+0000","last_peered":"2023-10-16T10:00:36.322064+0000","last_clean":"2023-10-16T10:00:36.322064+0000","last_became_active":"2023-10-16T10:00:34.639579+0000","last_became_peered":"2023-10-16T10:00:34.639579+0000","last_unstale":"2023-10-16T10:00:36.322064+0000","last_undegraded":"2023-10-16T10:00:36.322064+0000","last_fullsized":"2023-10-16T10:00:36.322064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:36.973289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633175+0000","last_change":"2023-10-16T10:00:34.654804+0000","last_active":"2023-10-16T10:00:35.633175+0000","last_peered":"2023-10-16T10:00:35.633175+0000","last_clean":"2023-10-16T10:00:35.633175+0000","last_became_active":"2023-10-16T10:00:34.654501+0000","last_became_peered":"2023-10-16T10:00:34.654501+0000","last_unstale":"2023-10-16T10:00:35.633175+0000","last_undegraded":"2023-10-16T10:00:35.633175+0000","last_fullsized":"2023-10-16T10:00:35.633175+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.637087+0000","last_change":"2023-10-16T10:00:34.634553+0000","last_active":"2023-10-16T10:00:34.637087+0000","last_peered":"2023-10-16T10:00:34.637087+0000","last_clean":"2023-10-16T10:00:34.637087+0000","last_became_active":"2023-10-16T10:00:34.634264+0000","last_became_peered":"2023-10-16T10:00:34.634264+0000","last_unstale":"2023-10-16T10:00:34.637087+0000","last_undegraded":"2023-10-16T10:00:34.637087+0000","last_fullsized":"2023-10-16T10:00:34.637087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:34.712375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635331+0000","last_change":"2023-10-16T10:00:34.639615+0000","last_active":"2023-10-16T10:00:35.635331+0000","last_peered":"2023-10-16T10:00:35.635331+0000","last_clean":"2023-10-16T10:00:35.635331+0000","last_became_active":"2023-10-16T10:00:34.639313+0000","last_became_peered":"2023-10-16T10:00:34.639313+0000","last_unstale":"2023-10-16T10:00:35.635331+0000","last_undegraded":"2023-10-16T10:00:35.635331+0000","last_fullsized":"2023-10-16T10:00:35.635331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:17:34.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.644650+0000","last_change":"2023-10-16T10:00:34.640739+0000","last_active":"2023-10-16T10:00:34.644650+0000","last_peered":"2023-10-16T10:00:34.644650+0000","last_clean":"2023-10-16T10:00:34.644650+0000","last_became_active":"2023-10-16T10:00:34.640473+0000","last_became_peered":"2023-10-16T10:00:34.640473+0000","last_unstale":"2023-10-16T10:00:34.644650+0000","last_undegraded":"2023-10-16T10:00:34.644650+0000","last_fullsized":"2023-10-16T10:00:34.644650+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:53.921189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635685+0000","last_change":"2023-10-16T10:00:34.642226+0000","last_active":"2023-10-16T10:00:35.635685+0000","last_peered":"2023-10-16T10:00:35.635685+0000","last_clean":"2023-10-16T10:00:35.635685+0000","last_became_active":"2023-10-16T10:00:34.641968+0000","last_became_peered":"2023-10-16T10:00:34.641968+0000","last_unstale":"2023-10-16T10:00:35.635685+0000","last_undegraded":"2023-10-16T10:00:35.635685+0000","last_fullsized":"2023-10-16T10:00:35.635685+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:17.347366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.658350+0000","last_change":"2023-10-16T10:00:34.656817+0000","last_active":"2023-10-16T10:00:34.658350+0000","last_peered":"2023-10-16T10:00:34.658350+0000","last_clean":"2023-10-16T10:00:34.658350+0000","last_became_active":"2023-10-16T10:00:34.656528+0000","last_became_peered":"2023-10-16T10:00:34.656528+0000","last_unstale":"2023-10-16T10:00:34.658350+0000","last_undegraded":"2023-10-16T10:00:34.658350+0000","last_fullsized":"2023-10-16T10:00:34.658350+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:38.909489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.636891+0000","last_change":"2023-10-16T10:00:34.634375+0000","last_active":"2023-10-16T10:00:34.636891+0000","last_peered":"2023-10-16T10:00:34.636891+0000","last_clean":"2023-10-16T10:00:34.636891+0000","last_became_active":"2023-10-16T10:00:34.634105+0000","last_became_peered":"2023-10-16T10:00:34.634105+0000","last_unstale":"2023-10-16T10:00:34.636891+0000","last_undegraded":"2023-10-16T10:00:34.636891+0000","last_fullsized":"2023-10-16T10:00:34.636891+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:34.437565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634891+0000","last_change":"2023-10-16T10:00:34.638358+0000","last_active":"2023-10-16T10:00:35.634891+0000","last_peered":"2023-10-16T10:00:35.634891+0000","last_clean":"2023-10-16T10:00:35.634891+0000","last_became_active":"2023-10-16T10:00:34.638078+0000","last_became_peered":"2023-10-16T10:00:34.638078+0000","last_unstale":"2023-10-16T10:00:35.634891+0000","last_undegraded":"2023-10-16T10:00:35.634891+0000","last_fullsized":"2023-10-16T10:00:35.634891+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:14:19.787907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.658256+0000","last_change":"2023-10-16T10:00:34.657614+0000","last_active":"2023-10-16T10:00:34.658256+0000","last_peered":"2023-10-16T10:00:34.658256+0000","last_clean":"2023-10-16T10:00:34.658256+0000","last_became_active":"2023-10-16T10:00:34.657341+0000","last_became_peered":"2023-10-16T10:00:34.657341+0000","last_unstale":"2023-10-16T10:00:34.658256+0000","last_undegraded":"2023-10-16T10:00:34.658256+0000","last_fullsized":"2023-10-16T10:00:34.658256+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:47.706895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322268+0000","last_change":"2023-10-16T10:00:34.659947+0000","last_active":"2023-10-16T10:00:36.322268+0000","last_peered":"2023-10-16T10:00:36.322268+0000","last_clean":"2023-10-16T10:00:36.322268+0000","last_became_active":"2023-10-16T10:00:34.659515+0000","last_became_peered":"2023-10-16T10:00:34.659515+0000","last_unstale":"2023-10-16T10:00:36.322268+0000","last_undegraded":"2023-10-16T10:00:36.322268+0000","last_fullsized":"2023-10-16T10:00:36.322268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:08.396720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322452+0000","last_change":"2023-10-16T10:00:34.642125+0000","last_active":"2023-10-16T10:00:36.322452+0000","last_peered":"2023-10-16T10:00:36.322452+0000","last_clean":"2023-10-16T10:00:36.322452+0000","last_became_active":"2023-10-16T10:00:34.641816+0000","last_became_peered":"2023-10-16T10:00:34.641816+0000","last_unstale":"2023-10-16T10:00:36.322452+0000","last_undegraded":"2023-10-16T10:00:36.322452+0000","last_fullsized":"2023-10-16T10:00:36.322452+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:14:36.640328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634484+0000","last_change":"2023-10-16T10:00:34.652848+0000","last_active":"2023-10-16T10:00:35.634484+0000","last_peered":"2023-10-16T10:00:35.634484+0000","last_clean":"2023-10-16T10:00:35.634484+0000","last_became_active":"2023-10-16T10:00:34.652328+0000","last_became_peered":"2023-10-16T10:00:34.652328+0000","last_unstale":"2023-10-16T10:00:35.634484+0000","last_undegraded":"2023-10-16T10:00:35.634484+0000","last_fullsized":"2023-10-16T10:00:35.634484+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635340+0000","last_change":"2023-10-16T10:00:34.641331+0000","last_active":"2023-10-16T10:00:35.635340+0000","last_peered":"2023-10-16T10:00:35.635340+0000","last_clean":"2023-10-16T10:00:35.635340+0000","last_became_active":"2023-10-16T10:00:34.641060+0000","last_became_peered":"2023-10-16T10:00:34.641060+0000","last_unstale":"2023-10-16T10:00:35.635340+0000","last_undegraded":"2023-10-16T10:00:35.635340+0000","last_fullsized":"2023-10-16T10:00:35.635340+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:41.245646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.656768+0000","last_change":"2023-10-16T10:00:34.654314+0000","last_active":"2023-10-16T10:00:34.656768+0000","last_peered":"2023-10-16T10:00:34.656768+0000","last_clean":"2023-10-16T10:00:34.656768+0000","last_became_active":"2023-10-16T10:00:34.653734+0000","last_became_peered":"2023-10-16T10:00:34.653734+0000","last_unstale":"2023-10-16T10:00:34.656768+0000","last_undegraded":"2023-10-16T10:00:34.656768+0000","last_fullsized":"2023-10-16T10:00:34.656768+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:35.428669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635839+0000","last_change":"2023-10-16T10:00:34.639677+0000","last_active":"2023-10-16T10:00:35.635839+0000","last_peered":"2023-10-16T10:00:35.635839+0000","last_clean":"2023-10-16T10:00:35.635839+0000","last_became_active":"2023-10-16T10:00:34.639378+0000","last_became_peered":"2023-10-16T10:00:34.639378+0000","last_unstale":"2023-10-16T10:00:35.635839+0000","last_undegraded":"2023-10-16T10:00:35.635839+0000","last_fullsized":"2023-10-16T10:00:35.635839+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:47.514148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634909+0000","last_change":"2023-10-16T10:00:34.641232+0000","last_active":"2023-10-16T10:00:35.634909+0000","last_peered":"2023-10-16T10:00:35.634909+0000","last_clean":"2023-10-16T10:00:35.634909+0000","last_became_active":"2023-10-16T10:00:34.640902+0000","last_became_peered":"2023-10-16T10:00:34.640902+0000","last_unstale":"2023-10-16T10:00:35.634909+0000","last_undegraded":"2023-10-16T10:00:35.634909+0000","last_fullsized":"2023-10-16T10:00:35.634909+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:01.931212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.651389+0000","last_change":"2023-10-16T10:00:34.640573+0000","last_active":"2023-10-16T10:00:34.651389+0000","last_peered":"2023-10-16T10:00:34.651389+0000","last_clean":"2023-10-16T10:00:34.651389+0000","last_became_active":"2023-10-16T10:00:34.640264+0000","last_became_peered":"2023-10-16T10:00:34.640264+0000","last_unstale":"2023-10-16T10:00:34.651389+0000","last_undegraded":"2023-10-16T10:00:34.651389+0000","last_fullsized":"2023-10-16T10:00:34.651389+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:47.706895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634150+0000","last_change":"2023-10-16T10:00:34.656232+0000","last_active":"2023-10-16T10:00:35.634150+0000","last_peered":"2023-10-16T10:00:35.634150+0000","last_clean":"2023-10-16T10:00:35.634150+0000","last_became_active":"2023-10-16T10:00:34.655948+0000","last_became_peered":"2023-10-16T10:00:34.655948+0000","last_unstale":"2023-10-16T10:00:35.634150+0000","last_undegraded":"2023-10-16T10:00:35.634150+0000","last_fullsized":"2023-10-16T10:00:35.634150+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:33.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860382+0000","last_change":"2023-10-16T10:00:34.646343+0000","last_active":"2023-10-16T10:00:35.860382+0000","last_peered":"2023-10-16T10:00:35.860382+0000","last_clean":"2023-10-16T10:00:35.860382+0000","last_became_active":"2023-10-16T10:00:34.646035+0000","last_became_peered":"2023-10-16T10:00:34.646035+0000","last_unstale":"2023-10-16T10:00:35.860382+0000","last_undegraded":"2023-10-16T10:00:35.860382+0000","last_fullsized":"2023-10-16T10:00:35.860382+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635629+0000","last_change":"2023-10-16T10:00:34.641499+0000","last_active":"2023-10-16T10:00:35.635629+0000","last_peered":"2023-10-16T10:00:35.635629+0000","last_clean":"2023-10-16T10:00:35.635629+0000","last_became_active":"2023-10-16T10:00:34.640873+0000","last_became_peered":"2023-10-16T10:00:34.640873+0000","last_unstale":"2023-10-16T10:00:35.635629+0000","last_undegraded":"2023-10-16T10:00:35.635629+0000","last_fullsized":"2023-10-16T10:00:35.635629+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:13.489391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.643970+0000","last_change":"2023-10-16T10:00:34.639323+0000","last_active":"2023-10-16T10:00:34.643970+0000","last_peered":"2023-10-16T10:00:34.643970+0000","last_clean":"2023-10-16T10:00:34.643970+0000","last_became_active":"2023-10-16T10:00:34.639040+0000","last_became_peered":"2023-10-16T10:00:34.639040+0000","last_unstale":"2023-10-16T10:00:34.643970+0000","last_undegraded":"2023-10-16T10:00:34.643970+0000","last_fullsized":"2023-10-16T10:00:34.643970+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:05:32.016253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.644984+0000","last_change":"2023-10-16T10:00:34.641690+0000","last_active":"2023-10-16T10:00:34.644984+0000","last_peered":"2023-10-16T10:00:34.644984+0000","last_clean":"2023-10-16T10:00:34.644984+0000","last_became_active":"2023-10-16T10:00:34.641407+0000","last_became_peered":"2023-10-16T10:00:34.641407+0000","last_unstale":"2023-10-16T10:00:34.644984+0000","last_undegraded":"2023-10-16T10:00:34.644984+0000","last_fullsized":"2023-10-16T10:00:34.644984+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:37.957418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633214+0000","last_change":"2023-10-16T10:00:34.634138+0000","last_active":"2023-10-16T10:00:35.633214+0000","last_peered":"2023-10-16T10:00:35.633214+0000","last_clean":"2023-10-16T10:00:35.633214+0000","last_became_active":"2023-10-16T10:00:34.633857+0000","last_became_peered":"2023-10-16T10:00:34.633857+0000","last_unstale":"2023-10-16T10:00:35.633214+0000","last_undegraded":"2023-10-16T10:00:35.633214+0000","last_fullsized":"2023-10-16T10:00:35.633214+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:32.443386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635072+0000","last_change":"2023-10-16T10:00:34.637077+0000","last_active":"2023-10-16T10:00:35.635072+0000","last_peered":"2023-10-16T10:00:35.635072+0000","last_clean":"2023-10-16T10:00:35.635072+0000","last_became_active":"2023-10-16T10:00:34.636579+0000","last_became_peered":"2023-10-16T10:00:34.636579+0000","last_unstale":"2023-10-16T10:00:35.635072+0000","last_undegraded":"2023-10-16T10:00:35.635072+0000","last_fullsized":"2023-10-16T10:00:35.635072+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:23.408584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.859956+0000","last_change":"2023-10-16T10:00:34.643324+0000","last_active":"2023-10-16T10:00:35.859956+0000","last_peered":"2023-10-16T10:00:35.859956+0000","last_clean":"2023-10-16T10:00:35.859956+0000","last_became_active":"2023-10-16T10:00:34.643016+0000","last_became_peered":"2023-10-16T10:00:34.643016+0000","last_unstale":"2023-10-16T10:00:35.859956+0000","last_undegraded":"2023-10-16T10:00:35.859956+0000","last_fullsized":"2023-10-16T10:00:35.859956+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:31.785355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322091+0000","last_change":"2023-10-16T10:00:34.659774+0000","last_active":"2023-10-16T10:00:36.322091+0000","last_peered":"2023-10-16T10:00:36.322091+0000","last_clean":"2023-10-16T10:00:36.322091+0000","last_became_active":"2023-10-16T10:00:34.659164+0000","last_became_peered":"2023-10-16T10:00:34.659164+0000","last_unstale":"2023-10-16T10:00:36.322091+0000","last_undegraded":"2023-10-16T10:00:36.322091+0000","last_fullsized":"2023-10-16T10:00:36.322091+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:42:17.438735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322221+0000","last_change":"2023-10-16T10:00:34.641759+0000","last_active":"2023-10-16T10:00:36.322221+0000","last_peered":"2023-10-16T10:00:36.322221+0000","last_clean":"2023-10-16T10:00:36.322221+0000","last_became_active":"2023-10-16T10:00:34.641480+0000","last_became_peered":"2023-10-16T10:00:34.641480+0000","last_unstale":"2023-10-16T10:00:36.322221+0000","last_undegraded":"2023-10-16T10:00:36.322221+0000","last_fullsized":"2023-10-16T10:00:36.322221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"110'4","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.662107+0000","last_change":"2023-10-16T10:00:33.621706+0000","last_active":"2023-10-16T10:00:34.662107+0000","last_peered":"2023-10-16T10:00:34.662107+0000","last_clean":"2023-10-16T10:00:34.662107+0000","last_became_active":"2023-10-16T10:00:33.621410+0000","last_became_peered":"2023-10-16T10:00:33.621410+0000","last_unstale":"2023-10-16T10:00:34.662107+0000","last_undegraded":"2023-10-16T10:00:34.662107+0000","last_fullsized":"2023-10-16T10:00:34.662107+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:33.708051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322381+0000","last_change":"2023-10-16T10:00:34.640248+0000","last_active":"2023-10-16T10:00:36.322381+0000","last_peered":"2023-10-16T10:00:36.322381+0000","last_clean":"2023-10-16T10:00:36.322381+0000","last_became_active":"2023-10-16T10:00:34.639121+0000","last_became_peered":"2023-10-16T10:00:34.639121+0000","last_unstale":"2023-10-16T10:00:36.322381+0000","last_undegraded":"2023-10-16T10:00:36.322381+0000","last_fullsized":"2023-10-16T10:00:36.322381+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:20:29.845160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635577+0000","last_change":"2023-10-16T10:00:33.616391+0000","last_active":"2023-10-16T10:00:35.635577+0000","last_peered":"2023-10-16T10:00:35.635577+0000","last_clean":"2023-10-16T10:00:35.635577+0000","last_became_active":"2023-10-16T10:00:33.616089+0000","last_became_peered":"2023-10-16T10:00:33.616089+0000","last_unstale":"2023-10-16T10:00:35.635577+0000","last_undegraded":"2023-10-16T10:00:35.635577+0000","last_fullsized":"2023-10-16T10:00:35.635577+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:20.593335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633315+0000","last_change":"2023-10-16T10:00:34.639078+0000","last_active":"2023-10-16T10:00:35.633315+0000","last_peered":"2023-10-16T10:00:35.633315+0000","last_clean":"2023-10-16T10:00:35.633315+0000","last_became_active":"2023-10-16T10:00:34.638801+0000","last_became_peered":"2023-10-16T10:00:34.638801+0000","last_unstale":"2023-10-16T10:00:35.633315+0000","last_undegraded":"2023-10-16T10:00:35.633315+0000","last_fullsized":"2023-10-16T10:00:35.633315+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:08.396720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633201+0000","last_change":"2023-10-16T10:00:33.618957+0000","last_active":"2023-10-16T10:00:35.633201+0000","last_peered":"2023-10-16T10:00:35.633201+0000","last_clean":"2023-10-16T10:00:35.633201+0000","last_became_active":"2023-10-16T10:00:33.618682+0000","last_became_peered":"2023-10-16T10:00:33.618682+0000","last_unstale":"2023-10-16T10:00:35.633201+0000","last_undegraded":"2023-10-16T10:00:35.633201+0000","last_fullsized":"2023-10-16T10:00:35.633201+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:32.231011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322475+0000","last_change":"2023-10-16T10:00:34.649617+0000","last_active":"2023-10-16T10:00:36.322475+0000","last_peered":"2023-10-16T10:00:36.322475+0000","last_clean":"2023-10-16T10:00:36.322475+0000","last_became_active":"2023-10-16T10:00:34.649284+0000","last_became_peered":"2023-10-16T10:00:34.649284+0000","last_unstale":"2023-10-16T10:00:36.322475+0000","last_undegraded":"2023-10-16T10:00:36.322475+0000","last_fullsized":"2023-10-16T10:00:36.322475+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:36:47.105291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"110'2","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.657141+0000","last_change":"2023-10-16T10:00:33.619365+0000","last_active":"2023-10-16T10:00:34.657141+0000","last_peered":"2023-10-16T10:00:34.657141+0000","last_clean":"2023-10-16T10:00:34.657141+0000","last_became_active":"2023-10-16T10:00:33.618925+0000","last_became_peered":"2023-10-16T10:00:33.618925+0000","last_unstale":"2023-10-16T10:00:34.657141+0000","last_undegraded":"2023-10-16T10:00:34.657141+0000","last_fullsized":"2023-10-16T10:00:34.657141+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:11:46.330111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.659144+0000","last_change":"2023-10-16T10:00:34.658172+0000","last_active":"2023-10-16T10:00:34.659144+0000","last_peered":"2023-10-16T10:00:34.659144+0000","last_clean":"2023-10-16T10:00:34.659144+0000","last_became_active":"2023-10-16T10:00:34.657885+0000","last_became_peered":"2023-10-16T10:00:34.657885+0000","last_unstale":"2023-10-16T10:00:34.659144+0000","last_undegraded":"2023-10-16T10:00:34.659144+0000","last_fullsized":"2023-10-16T10:00:34.659144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:11:47.334435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.636082+0000","last_change":"2023-10-16T10:00:33.617380+0000","last_active":"2023-10-16T10:00:35.636082+0000","last_peered":"2023-10-16T10:00:35.636082+0000","last_clean":"2023-10-16T10:00:35.636082+0000","last_became_active":"2023-10-16T10:00:33.617089+0000","last_became_peered":"2023-10-16T10:00:33.617089+0000","last_unstale":"2023-10-16T10:00:35.636082+0000","last_undegraded":"2023-10-16T10:00:35.636082+0000","last_fullsized":"2023-10-16T10:00:35.636082+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:36.953094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.651822+0000","last_change":"2023-10-16T10:00:34.648773+0000","last_active":"2023-10-16T10:00:34.651822+0000","last_peered":"2023-10-16T10:00:34.651822+0000","last_clean":"2023-10-16T10:00:34.651822+0000","last_became_active":"2023-10-16T10:00:34.648501+0000","last_became_peered":"2023-10-16T10:00:34.648501+0000","last_unstale":"2023-10-16T10:00:34.651822+0000","last_undegraded":"2023-10-16T10:00:34.651822+0000","last_fullsized":"2023-10-16T10:00:34.651822+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.817247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"110'1","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.656593+0000","last_change":"2023-10-16T10:00:33.612826+0000","last_active":"2023-10-16T10:00:34.656593+0000","last_peered":"2023-10-16T10:00:34.656593+0000","last_clean":"2023-10-16T10:00:34.656593+0000","last_became_active":"2023-10-16T10:00:33.612535+0000","last_became_peered":"2023-10-16T10:00:33.612535+0000","last_unstale":"2023-10-16T10:00:34.656593+0000","last_undegraded":"2023-10-16T10:00:34.656593+0000","last_fullsized":"2023-10-16T10:00:34.656593+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:44.114445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322069+0000","last_change":"2023-10-16T10:00:34.639361+0000","last_active":"2023-10-16T10:00:36.322069+0000","last_peered":"2023-10-16T10:00:36.322069+0000","last_clean":"2023-10-16T10:00:36.322069+0000","last_became_active":"2023-10-16T10:00:34.639056+0000","last_became_peered":"2023-10-16T10:00:34.639056+0000","last_unstale":"2023-10-16T10:00:36.322069+0000","last_undegraded":"2023-10-16T10:00:36.322069+0000","last_fullsized":"2023-10-16T10:00:36.322069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:33.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634338+0000","last_change":"2023-10-16T10:00:33.609835+0000","last_active":"2023-10-16T10:00:35.634338+0000","last_peered":"2023-10-16T10:00:35.634338+0000","last_clean":"2023-10-16T10:00:35.634338+0000","last_became_active":"2023-10-16T10:00:33.609534+0000","last_became_peered":"2023-10-16T10:00:33.609534+0000","last_unstale":"2023-10-16T10:00:35.634338+0000","last_undegraded":"2023-10-16T10:00:35.634338+0000","last_fullsized":"2023-10-16T10:00:35.634338+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:30.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.640604+0000","last_change":"2023-10-16T10:00:34.639323+0000","last_active":"2023-10-16T10:00:34.640604+0000","last_peered":"2023-10-16T10:00:34.640604+0000","last_clean":"2023-10-16T10:00:34.640604+0000","last_became_active":"2023-10-16T10:00:34.639040+0000","last_became_peered":"2023-10-16T10:00:34.639040+0000","last_unstale":"2023-10-16T10:00:34.640604+0000","last_undegraded":"2023-10-16T10:00:34.640604+0000","last_fullsized":"2023-10-16T10:00:34.640604+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:21.597660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633544+0000","last_change":"2023-10-16T10:00:33.617489+0000","last_active":"2023-10-16T10:00:35.633544+0000","last_peered":"2023-10-16T10:00:35.633544+0000","last_clean":"2023-10-16T10:00:35.633544+0000","last_became_active":"2023-10-16T10:00:33.617187+0000","last_became_peered":"2023-10-16T10:00:33.617187+0000","last_unstale":"2023-10-16T10:00:35.633544+0000","last_undegraded":"2023-10-16T10:00:35.633544+0000","last_fullsized":"2023-10-16T10:00:35.633544+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:11.654572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635188+0000","last_change":"2023-10-16T10:00:34.651964+0000","last_active":"2023-10-16T10:00:35.635188+0000","last_peered":"2023-10-16T10:00:35.635188+0000","last_clean":"2023-10-16T10:00:35.635188+0000","last_became_active":"2023-10-16T10:00:34.651432+0000","last_became_peered":"2023-10-16T10:00:34.651432+0000","last_unstale":"2023-10-16T10:00:35.635188+0000","last_undegraded":"2023-10-16T10:00:35.635188+0000","last_fullsized":"2023-10-16T10:00:35.635188+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:53.888942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860429+0000","last_change":"2023-10-16T10:00:33.621520+0000","last_active":"2023-10-16T10:00:35.860429+0000","last_peered":"2023-10-16T10:00:35.860429+0000","last_clean":"2023-10-16T10:00:35.860429+0000","last_became_active":"2023-10-16T10:00:33.621238+0000","last_became_peered":"2023-10-16T10:00:33.621238+0000","last_unstale":"2023-10-16T10:00:35.860429+0000","last_undegraded":"2023-10-16T10:00:35.860429+0000","last_fullsized":"2023-10-16T10:00:35.860429+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:21:05.422515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633209+0000","last_change":"2023-10-16T10:00:34.637725+0000","last_active":"2023-10-16T10:00:35.633209+0000","last_peered":"2023-10-16T10:00:35.633209+0000","last_clean":"2023-10-16T10:00:35.633209+0000","last_became_active":"2023-10-16T10:00:34.637429+0000","last_became_peered":"2023-10-16T10:00:34.637429+0000","last_unstale":"2023-10-16T10:00:35.633209+0000","last_undegraded":"2023-10-16T10:00:35.633209+0000","last_fullsized":"2023-10-16T10:00:35.633209+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634568+0000","last_change":"2023-10-16T10:00:33.616300+0000","last_active":"2023-10-16T10:00:35.634568+0000","last_peered":"2023-10-16T10:00:35.634568+0000","last_clean":"2023-10-16T10:00:35.634568+0000","last_became_active":"2023-10-16T10:00:33.615871+0000","last_became_peered":"2023-10-16T10:00:33.615871+0000","last_unstale":"2023-10-16T10:00:35.634568+0000","last_undegraded":"2023-10-16T10:00:35.634568+0000","last_fullsized":"2023-10-16T10:00:35.634568+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634475+0000","last_change":"2023-10-16T10:00:34.652588+0000","last_active":"2023-10-16T10:00:35.634475+0000","last_peered":"2023-10-16T10:00:35.634475+0000","last_clean":"2023-10-16T10:00:35.634475+0000","last_became_active":"2023-10-16T10:00:34.652285+0000","last_became_peered":"2023-10-16T10:00:34.652285+0000","last_unstale":"2023-10-16T10:00:35.634475+0000","last_undegraded":"2023-10-16T10:00:35.634475+0000","last_fullsized":"2023-10-16T10:00:35.634475+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:00.868370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"110'3","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635439+0000","last_change":"2023-10-16T10:00:33.619846+0000","last_active":"2023-10-16T10:00:35.635439+0000","last_peered":"2023-10-16T10:00:35.635439+0000","last_clean":"2023-10-16T10:00:35.635439+0000","last_became_active":"2023-10-16T10:00:33.619562+0000","last_became_peered":"2023-10-16T10:00:33.619562+0000","last_unstale":"2023-10-16T10:00:35.635439+0000","last_undegraded":"2023-10-16T10:00:35.635439+0000","last_fullsized":"2023-10-16T10:00:35.635439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:52.062191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.636725+0000","last_change":"2023-10-16T10:00:34.633749+0000","last_active":"2023-10-16T10:00:34.636725+0000","last_peered":"2023-10-16T10:00:34.636725+0000","last_clean":"2023-10-16T10:00:34.636725+0000","last_became_active":"2023-10-16T10:00:34.633473+0000","last_became_peered":"2023-10-16T10:00:34.633473+0000","last_unstale":"2023-10-16T10:00:34.636725+0000","last_undegraded":"2023-10-16T10:00:34.636725+0000","last_fullsized":"2023-10-16T10:00:34.636725+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:37.197226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"110'6","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322336+0000","last_change":"2023-10-16T10:00:33.612790+0000","last_active":"2023-10-16T10:00:36.322336+0000","last_peered":"2023-10-16T10:00:36.322336+0000","last_clean":"2023-10-16T10:00:36.322336+0000","last_became_active":"2023-10-16T10:00:33.612503+0000","last_became_peered":"2023-10-16T10:00:33.612503+0000","last_unstale":"2023-10-16T10:00:36.322336+0000","last_undegraded":"2023-10-16T10:00:36.322336+0000","last_fullsized":"2023-10-16T10:00:36.322336+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:16.904124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.636068+0000","last_change":"2023-10-16T10:00:34.631460+0000","last_active":"2023-10-16T10:00:34.636068+0000","last_peered":"2023-10-16T10:00:34.636068+0000","last_clean":"2023-10-16T10:00:34.636068+0000","last_became_active":"2023-10-16T10:00:34.631163+0000","last_became_peered":"2023-10-16T10:00:34.631163+0000","last_unstale":"2023-10-16T10:00:34.636068+0000","last_undegraded":"2023-10-16T10:00:34.636068+0000","last_fullsized":"2023-10-16T10:00:34.636068+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.817247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633398+0000","last_change":"2023-10-16T10:00:33.617695+0000","last_active":"2023-10-16T10:00:35.633398+0000","last_peered":"2023-10-16T10:00:35.633398+0000","last_clean":"2023-10-16T10:00:35.633398+0000","last_became_active":"2023-10-16T10:00:33.617418+0000","last_became_peered":"2023-10-16T10:00:33.617418+0000","last_unstale":"2023-10-16T10:00:35.633398+0000","last_undegraded":"2023-10-16T10:00:35.633398+0000","last_fullsized":"2023-10-16T10:00:35.633398+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:07.844585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634884+0000","last_change":"2023-10-16T10:00:34.641842+0000","last_active":"2023-10-16T10:00:35.634884+0000","last_peered":"2023-10-16T10:00:35.634884+0000","last_clean":"2023-10-16T10:00:35.634884+0000","last_became_active":"2023-10-16T10:00:34.641514+0000","last_became_peered":"2023-10-16T10:00:34.641514+0000","last_unstale":"2023-10-16T10:00:35.634884+0000","last_undegraded":"2023-10-16T10:00:35.634884+0000","last_fullsized":"2023-10-16T10:00:35.634884+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:37.197226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322175+0000","last_change":"2023-10-16T10:00:33.613435+0000","last_active":"2023-10-16T10:00:36.322175+0000","last_peered":"2023-10-16T10:00:36.322175+0000","last_clean":"2023-10-16T10:00:36.322175+0000","last_became_active":"2023-10-16T10:00:33.613110+0000","last_became_peered":"2023-10-16T10:00:33.613110+0000","last_unstale":"2023-10-16T10:00:36.322175+0000","last_undegraded":"2023-10-16T10:00:36.322175+0000","last_fullsized":"2023-10-16T10:00:36.322175+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:07.844585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.633719+0000","last_change":"2023-10-16T10:00:34.632468+0000","last_active":"2023-10-16T10:00:34.633719+0000","last_peered":"2023-10-16T10:00:34.633719+0000","last_clean":"2023-10-16T10:00:34.633719+0000","last_became_active":"2023-10-16T10:00:34.632145+0000","last_became_peered":"2023-10-16T10:00:34.632145+0000","last_unstale":"2023-10-16T10:00:34.633719+0000","last_undegraded":"2023-10-16T10:00:34.633719+0000","last_fullsized":"2023-10-16T10:00:34.633719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:56.475229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.627196+0000","last_change":"2023-10-16T10:00:33.612790+0000","last_active":"2023-10-16T10:00:34.627196+0000","last_peered":"2023-10-16T10:00:34.627196+0000","last_clean":"2023-10-16T10:00:34.627196+0000","last_became_active":"2023-10-16T10:00:33.612516+0000","last_became_peered":"2023-10-16T10:00:33.612516+0000","last_unstale":"2023-10-16T10:00:34.627196+0000","last_undegraded":"2023-10-16T10:00:34.627196+0000","last_fullsized":"2023-10-16T10:00:34.627196+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:27.154632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.633496+0000","last_change":"2023-10-16T10:00:34.631459+0000","last_active":"2023-10-16T10:00:34.633496+0000","last_peered":"2023-10-16T10:00:34.633496+0000","last_clean":"2023-10-16T10:00:34.633496+0000","last_became_active":"2023-10-16T10:00:34.631133+0000","last_became_peered":"2023-10-16T10:00:34.631133+0000","last_unstale":"2023-10-16T10:00:34.633496+0000","last_undegraded":"2023-10-16T10:00:34.633496+0000","last_fullsized":"2023-10-16T10:00:34.633496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:42.313895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"110'8","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.658010+0000","last_change":"2023-10-16T10:00:33.615584+0000","last_active":"2023-10-16T10:00:34.658010+0000","last_peered":"2023-10-16T10:00:34.658010+0000","last_clean":"2023-10-16T10:00:34.658010+0000","last_became_active":"2023-10-16T10:00:33.615281+0000","last_became_peered":"2023-10-16T10:00:33.615281+0000","last_unstale":"2023-10-16T10:00:34.658010+0000","last_undegraded":"2023-10-16T10:00:34.658010+0000","last_fullsized":"2023-10-16T10:00:34.658010+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:52.062191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322224+0000","last_change":"2023-10-16T10:00:34.642445+0000","last_active":"2023-10-16T10:00:36.322224+0000","last_peered":"2023-10-16T10:00:36.322224+0000","last_clean":"2023-10-16T10:00:36.322224+0000","last_became_active":"2023-10-16T10:00:34.642149+0000","last_became_peered":"2023-10-16T10:00:34.642149+0000","last_unstale":"2023-10-16T10:00:36.322224+0000","last_undegraded":"2023-10-16T10:00:36.322224+0000","last_fullsized":"2023-10-16T10:00:36.322224+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"110'1","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.655516+0000","last_change":"2023-10-16T10:00:33.620935+0000","last_active":"2023-10-16T10:00:34.655516+0000","last_peered":"2023-10-16T10:00:34.655516+0000","last_clean":"2023-10-16T10:00:34.655516+0000","last_became_active":"2023-10-16T10:00:33.620663+0000","last_became_peered":"2023-10-16T10:00:33.620663+0000","last_unstale":"2023-10-16T10:00:34.655516+0000","last_undegraded":"2023-10-16T10:00:34.655516+0000","last_fullsized":"2023-10-16T10:00:34.655516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:33.754260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.653673+0000","last_change":"2023-10-16T10:00:34.652357+0000","last_active":"2023-10-16T10:00:34.653673+0000","last_peered":"2023-10-16T10:00:34.653673+0000","last_clean":"2023-10-16T10:00:34.653673+0000","last_became_active":"2023-10-16T10:00:34.651972+0000","last_became_peered":"2023-10-16T10:00:34.651972+0000","last_unstale":"2023-10-16T10:00:34.653673+0000","last_undegraded":"2023-10-16T10:00:34.653673+0000","last_fullsized":"2023-10-16T10:00:34.653673+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:34.437565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633422+0000","last_change":"2023-10-16T10:00:33.619445+0000","last_active":"2023-10-16T10:00:35.633422+0000","last_peered":"2023-10-16T10:00:35.633422+0000","last_clean":"2023-10-16T10:00:35.633422+0000","last_became_active":"2023-10-16T10:00:33.619172+0000","last_became_peered":"2023-10-16T10:00:33.619172+0000","last_unstale":"2023-10-16T10:00:35.633422+0000","last_undegraded":"2023-10-16T10:00:35.633422+0000","last_fullsized":"2023-10-16T10:00:35.633422+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635398+0000","last_change":"2023-10-16T10:00:34.640882+0000","last_active":"2023-10-16T10:00:35.635398+0000","last_peered":"2023-10-16T10:00:35.635398+0000","last_clean":"2023-10-16T10:00:35.635398+0000","last_became_active":"2023-10-16T10:00:34.640590+0000","last_became_peered":"2023-10-16T10:00:34.640590+0000","last_unstale":"2023-10-16T10:00:35.635398+0000","last_undegraded":"2023-10-16T10:00:35.635398+0000","last_fullsized":"2023-10-16T10:00:35.635398+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:02.945267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635172+0000","last_change":"2023-10-16T10:00:33.618426+0000","last_active":"2023-10-16T10:00:35.635172+0000","last_peered":"2023-10-16T10:00:35.635172+0000","last_clean":"2023-10-16T10:00:35.635172+0000","last_became_active":"2023-10-16T10:00:33.617984+0000","last_became_peered":"2023-10-16T10:00:33.617984+0000","last_unstale":"2023-10-16T10:00:35.635172+0000","last_undegraded":"2023-10-16T10:00:35.635172+0000","last_fullsized":"2023-10-16T10:00:35.635172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:33.754260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322262+0000","last_change":"2023-10-16T10:00:34.641853+0000","last_active":"2023-10-16T10:00:36.322262+0000","last_peered":"2023-10-16T10:00:36.322262+0000","last_clean":"2023-10-16T10:00:36.322262+0000","last_became_active":"2023-10-16T10:00:34.640494+0000","last_became_peered":"2023-10-16T10:00:34.640494+0000","last_unstale":"2023-10-16T10:00:36.322262+0000","last_undegraded":"2023-10-16T10:00:36.322262+0000","last_fullsized":"2023-10-16T10:00:36.322262+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:32.443386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860102+0000","last_change":"2023-10-16T10:00:33.619469+0000","last_active":"2023-10-16T10:00:35.860102+0000","last_peered":"2023-10-16T10:00:35.860102+0000","last_clean":"2023-10-16T10:00:35.860102+0000","last_became_active":"2023-10-16T10:00:33.619102+0000","last_became_peered":"2023-10-16T10:00:33.619102+0000","last_unstale":"2023-10-16T10:00:35.860102+0000","last_undegraded":"2023-10-16T10:00:35.860102+0000","last_fullsized":"2023-10-16T10:00:35.860102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:22.006766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860003+0000","last_change":"2023-10-16T10:00:34.657621+0000","last_active":"2023-10-16T10:00:35.860003+0000","last_peered":"2023-10-16T10:00:35.860003+0000","last_clean":"2023-10-16T10:00:35.860003+0000","last_became_active":"2023-10-16T10:00:34.657316+0000","last_became_peered":"2023-10-16T10:00:34.657316+0000","last_unstale":"2023-10-16T10:00:35.860003+0000","last_undegraded":"2023-10-16T10:00:35.860003+0000","last_fullsized":"2023-10-16T10:00:35.860003+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:00:33.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.626243+0000","last_change":"2023-10-16T10:00:33.614479+0000","last_active":"2023-10-16T10:00:34.626243+0000","last_peered":"2023-10-16T10:00:34.626243+0000","last_clean":"2023-10-16T10:00:34.626243+0000","last_became_active":"2023-10-16T10:00:33.614267+0000","last_became_peered":"2023-10-16T10:00:33.614267+0000","last_unstale":"2023-10-16T10:00:34.626243+0000","last_undegraded":"2023-10-16T10:00:34.626243+0000","last_fullsized":"2023-10-16T10:00:34.626243+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:20.777695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860091+0000","last_change":"2023-10-16T10:00:34.640131+0000","last_active":"2023-10-16T10:00:35.860091+0000","last_peered":"2023-10-16T10:00:35.860091+0000","last_clean":"2023-10-16T10:00:35.860091+0000","last_became_active":"2023-10-16T10:00:34.639455+0000","last_became_peered":"2023-10-16T10:00:34.639455+0000","last_unstale":"2023-10-16T10:00:35.860091+0000","last_undegraded":"2023-10-16T10:00:35.860091+0000","last_fullsized":"2023-10-16T10:00:35.860091+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:36.973289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860177+0000","last_change":"2023-10-16T10:00:33.618623+0000","last_active":"2023-10-16T10:00:35.860177+0000","last_peered":"2023-10-16T10:00:35.860177+0000","last_clean":"2023-10-16T10:00:35.860177+0000","last_became_active":"2023-10-16T10:00:33.618227+0000","last_became_peered":"2023-10-16T10:00:33.618227+0000","last_unstale":"2023-10-16T10:00:35.860177+0000","last_undegraded":"2023-10-16T10:00:35.860177+0000","last_fullsized":"2023-10-16T10:00:35.860177+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:26.257889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322408+0000","last_change":"2023-10-16T10:00:34.640393+0000","last_active":"2023-10-16T10:00:36.322408+0000","last_peered":"2023-10-16T10:00:36.322408+0000","last_clean":"2023-10-16T10:00:36.322408+0000","last_became_active":"2023-10-16T10:00:34.639742+0000","last_became_peered":"2023-10-16T10:00:34.639742+0000","last_unstale":"2023-10-16T10:00:36.322408+0000","last_undegraded":"2023-10-16T10:00:36.322408+0000","last_fullsized":"2023-10-16T10:00:36.322408+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:19.030816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.625707+0000","last_change":"2023-10-16T10:00:33.620235+0000","last_active":"2023-10-16T10:00:34.625707+0000","last_peered":"2023-10-16T10:00:34.625707+0000","last_clean":"2023-10-16T10:00:34.625707+0000","last_became_active":"2023-10-16T10:00:33.619956+0000","last_became_peered":"2023-10-16T10:00:33.619956+0000","last_unstale":"2023-10-16T10:00:34.625707+0000","last_undegraded":"2023-10-16T10:00:34.625707+0000","last_fullsized":"2023-10-16T10:00:34.625707+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:33:56.684855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.652683+0000","last_change":"2023-10-16T10:00:34.645655+0000","last_active":"2023-10-16T10:00:34.652683+0000","last_peered":"2023-10-16T10:00:34.652683+0000","last_clean":"2023-10-16T10:00:34.652683+0000","last_became_active":"2023-10-16T10:00:34.645392+0000","last_became_peered":"2023-10-16T10:00:34.645392+0000","last_unstale":"2023-10-16T10:00:34.652683+0000","last_undegraded":"2023-10-16T10:00:34.652683+0000","last_fullsized":"2023-10-16T10:00:34.652683+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:45.118769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634062+0000","last_change":"2023-10-16T10:00:33.622451+0000","last_active":"2023-10-16T10:00:35.634062+0000","last_peered":"2023-10-16T10:00:35.634062+0000","last_clean":"2023-10-16T10:00:35.634062+0000","last_became_active":"2023-10-16T10:00:33.622173+0000","last_became_peered":"2023-10-16T10:00:33.622173+0000","last_unstale":"2023-10-16T10:00:35.634062+0000","last_undegraded":"2023-10-16T10:00:35.634062+0000","last_fullsized":"2023-10-16T10:00:35.634062+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:30.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322490+0000","last_change":"2023-10-16T10:00:34.640516+0000","last_active":"2023-10-16T10:00:36.322490+0000","last_peered":"2023-10-16T10:00:36.322490+0000","last_clean":"2023-10-16T10:00:36.322490+0000","last_became_active":"2023-10-16T10:00:34.639964+0000","last_became_peered":"2023-10-16T10:00:34.639964+0000","last_unstale":"2023-10-16T10:00:36.322490+0000","last_undegraded":"2023-10-16T10:00:36.322490+0000","last_fullsized":"2023-10-16T10:00:36.322490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:16.261934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635483+0000","last_change":"2023-10-16T10:00:33.620773+0000","last_active":"2023-10-16T10:00:35.635483+0000","last_peered":"2023-10-16T10:00:35.635483+0000","last_clean":"2023-10-16T10:00:35.635483+0000","last_became_active":"2023-10-16T10:00:33.620474+0000","last_became_peered":"2023-10-16T10:00:33.620474+0000","last_unstale":"2023-10-16T10:00:35.635483+0000","last_undegraded":"2023-10-16T10:00:35.635483+0000","last_fullsized":"2023-10-16T10:00:35.635483+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:46.702571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322360+0000","last_change":"2023-10-16T10:00:34.641261+0000","last_active":"2023-10-16T10:00:36.322360+0000","last_peered":"2023-10-16T10:00:36.322360+0000","last_clean":"2023-10-16T10:00:36.322360+0000","last_became_active":"2023-10-16T10:00:34.640291+0000","last_became_peered":"2023-10-16T10:00:34.640291+0000","last_unstale":"2023-10-16T10:00:36.322360+0000","last_undegraded":"2023-10-16T10:00:36.322360+0000","last_fullsized":"2023-10-16T10:00:36.322360+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:45:58.597170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.635534+0000","last_change":"2023-10-16T10:00:33.618247+0000","last_active":"2023-10-16T10:00:35.635534+0000","last_peered":"2023-10-16T10:00:35.635534+0000","last_clean":"2023-10-16T10:00:35.635534+0000","last_became_active":"2023-10-16T10:00:33.617621+0000","last_became_peered":"2023-10-16T10:00:33.617621+0000","last_unstale":"2023-10-16T10:00:35.635534+0000","last_undegraded":"2023-10-16T10:00:35.635534+0000","last_fullsized":"2023-10-16T10:00:35.635534+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:34.424345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860375+0000","last_change":"2023-10-16T10:00:34.639499+0000","last_active":"2023-10-16T10:00:35.860375+0000","last_peered":"2023-10-16T10:00:35.860375+0000","last_clean":"2023-10-16T10:00:35.860375+0000","last_became_active":"2023-10-16T10:00:34.639131+0000","last_became_peered":"2023-10-16T10:00:34.639131+0000","last_unstale":"2023-10-16T10:00:35.860375+0000","last_undegraded":"2023-10-16T10:00:35.860375+0000","last_fullsized":"2023-10-16T10:00:35.860375+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:03.689699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.635932+0000","last_change":"2023-10-16T10:00:33.618243+0000","last_active":"2023-10-16T10:00:34.635932+0000","last_peered":"2023-10-16T10:00:34.635932+0000","last_clean":"2023-10-16T10:00:34.635932+0000","last_became_active":"2023-10-16T10:00:33.617950+0000","last_became_peered":"2023-10-16T10:00:33.617950+0000","last_unstale":"2023-10-16T10:00:34.635932+0000","last_undegraded":"2023-10-16T10:00:34.635932+0000","last_fullsized":"2023-10-16T10:00:34.635932+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:27.154632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.644617+0000","last_change":"2023-10-16T10:00:34.642974+0000","last_active":"2023-10-16T10:00:34.644617+0000","last_peered":"2023-10-16T10:00:34.644617+0000","last_clean":"2023-10-16T10:00:34.644617+0000","last_became_active":"2023-10-16T10:00:34.642662+0000","last_became_peered":"2023-10-16T10:00:34.642662+0000","last_unstale":"2023-10-16T10:00:34.644617+0000","last_undegraded":"2023-10-16T10:00:34.644617+0000","last_fullsized":"2023-10-16T10:00:34.644617+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:41.332741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322446+0000","last_change":"2023-10-16T10:00:33.620943+0000","last_active":"2023-10-16T10:00:36.322446+0000","last_peered":"2023-10-16T10:00:36.322446+0000","last_clean":"2023-10-16T10:00:36.322446+0000","last_became_active":"2023-10-16T10:00:33.620634+0000","last_became_peered":"2023-10-16T10:00:33.620634+0000","last_unstale":"2023-10-16T10:00:36.322446+0000","last_undegraded":"2023-10-16T10:00:36.322446+0000","last_fullsized":"2023-10-16T10:00:36.322446+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.634303+0000","last_change":"2023-10-16T10:00:34.653328+0000","last_active":"2023-10-16T10:00:35.634303+0000","last_peered":"2023-10-16T10:00:35.634303+0000","last_clean":"2023-10-16T10:00:35.634303+0000","last_became_active":"2023-10-16T10:00:34.653072+0000","last_became_peered":"2023-10-16T10:00:34.653072+0000","last_unstale":"2023-10-16T10:00:35.634303+0000","last_undegraded":"2023-10-16T10:00:35.634303+0000","last_fullsized":"2023-10-16T10:00:35.634303+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:42:17.438735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.860077+0000","last_change":"2023-10-16T10:00:33.621129+0000","last_active":"2023-10-16T10:00:35.860077+0000","last_peered":"2023-10-16T10:00:35.860077+0000","last_clean":"2023-10-16T10:00:35.860077+0000","last_became_active":"2023-10-16T10:00:33.619822+0000","last_became_peered":"2023-10-16T10:00:33.619822+0000","last_unstale":"2023-10-16T10:00:35.860077+0000","last_undegraded":"2023-10-16T10:00:35.860077+0000","last_fullsized":"2023-10-16T10:00:35.860077+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:05.869210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.645665+0000","last_change":"2023-10-16T10:00:34.638976+0000","last_active":"2023-10-16T10:00:34.645665+0000","last_peered":"2023-10-16T10:00:34.645665+0000","last_clean":"2023-10-16T10:00:34.645665+0000","last_became_active":"2023-10-16T10:00:34.638729+0000","last_became_peered":"2023-10-16T10:00:34.638729+0000","last_unstale":"2023-10-16T10:00:34.645665+0000","last_undegraded":"2023-10-16T10:00:34.645665+0000","last_fullsized":"2023-10-16T10:00:34.645665+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:27.581985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.635645+0000","last_change":"2023-10-16T10:00:33.618355+0000","last_active":"2023-10-16T10:00:34.635645+0000","last_peered":"2023-10-16T10:00:34.635645+0000","last_clean":"2023-10-16T10:00:34.635645+0000","last_became_active":"2023-10-16T10:00:33.617839+0000","last_became_peered":"2023-10-16T10:00:33.617839+0000","last_unstale":"2023-10-16T10:00:34.635645+0000","last_undegraded":"2023-10-16T10:00:34.635645+0000","last_fullsized":"2023-10-16T10:00:34.635645+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:20.593335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:36.322341+0000","last_change":"2023-10-16T10:00:34.656620+0000","last_active":"2023-10-16T10:00:36.322341+0000","last_peered":"2023-10-16T10:00:36.322341+0000","last_clean":"2023-10-16T10:00:36.322341+0000","last_became_active":"2023-10-16T10:00:34.656363+0000","last_became_peered":"2023-10-16T10:00:34.656363+0000","last_unstale":"2023-10-16T10:00:36.322341+0000","last_undegraded":"2023-10-16T10:00:36.322341+0000","last_fullsized":"2023-10-16T10:00:36.322341+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:14.159457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633652+0000","last_change":"2023-10-16T10:00:33.616467+0000","last_active":"2023-10-16T10:00:35.633652+0000","last_peered":"2023-10-16T10:00:35.633652+0000","last_clean":"2023-10-16T10:00:35.633652+0000","last_became_active":"2023-10-16T10:00:33.616180+0000","last_became_peered":"2023-10-16T10:00:33.616180+0000","last_unstale":"2023-10-16T10:00:35.633652+0000","last_undegraded":"2023-10-16T10:00:35.633652+0000","last_fullsized":"2023-10-16T10:00:35.633652+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:22.006766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T10:00:35.633971+0000","last_change":"2023-10-16T10:00:34.653674+0000","last_active":"2023-10-16T10:00:35.633971+0000","last_peered":"2023-10-16T10:00:35.633971+0000","last_clean":"2023-10-16T10:00:35.633971+0000","last_became_active":"2023-10-16T10:00:34.653381+0000","last_became_peered":"2023-10-16T10:00:34.653381+0000","last_unstale":"2023-10-16T10:00:35.633971+0000","last_undegraded":"2023-10-16T10:00:35.633971+0000","last_fullsized":"2023-10-16T10:00:35.633971+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:33.594037+0000","last_clean_scrub_stamp":"2023-10-16T10:00:33.594037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:14.159457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.628900+0000","last_change":"2023-10-16T10:00:33.619788+0000","last_active":"2023-10-16T10:00:34.628900+0000","last_peered":"2023-10-16T10:00:34.628900+0000","last_clean":"2023-10-16T10:00:34.628900+0000","last_became_active":"2023-10-16T10:00:33.619491+0000","last_became_peered":"2023-10-16T10:00:33.619491+0000","last_unstale":"2023-10-16T10:00:34.628900+0000","last_undegraded":"2023-10-16T10:00:34.628900+0000","last_fullsized":"2023-10-16T10:00:34.628900+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:00:32.589713+0000","last_clean_scrub_stamp":"2023-10-16T10:00:32.589713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:14:18.783582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"101'183","reported_seq":489,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T10:00:34.624567+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:00:34.624567+0000","last_peered":"2023-10-16T10:00:34.624567+0000","last_clean":"2023-10-16T10:00:34.624567+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:00:34.624567+0000","last_undegraded":"2023-10-16T10:00:34.624567+0000","last_fullsized":"2023-10-16T10:00:34.624567+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575054,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59472,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312368,"statfs":{"total":96636764160,"available":96575864832,"internally_reserved":0,"allocated":1970176,"data_stored":1299002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.56100000000000005},"min":{"1min":0.36499999999999999,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":0.73299999999999998,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.67700000000000005,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.911}]},{"osd":1,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.442,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70399999999999996,"5min":0.79300000000000004,"15min":1.1439999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.46500000000000002,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.78100000000000003,"5min":1.1279999999999999,"15min":1.212},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.63100000000000001},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.27700000000000002},"max":{"1min":0.81599999999999995,"5min":1.607,"15min":1.607},"last":3.3820000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61399999999999999,"15min":0.63400000000000001},"min":{"1min":0.432,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":2.044,"15min":2.044},"last":3.1640000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58399999999999996,"15min":0.60099999999999998},"min":{"1min":0.42799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.755,"5min":1.524,"15min":1.524},"last":2.0979999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.50700000000000001,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.70399999999999996,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":2.2029999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.8220000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68600000000000005,"5min":0.82699999999999996,"15min":0.88900000000000001},"last":3.8149999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73899999999999999,"5min":0.81100000000000005,"15min":0.875},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.37,"5min":0.33200000000000002,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.6619999999999999}]},{"osd":6,"last update":"Mon Oct 16 09:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.76000000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.438,"5min":0.311,"15min":0.311},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.85299999999999998},"last":0.98699999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575054,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1376256,"data_stored":703750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55600000000000005,"15min":0.58299999999999996},"min":{"1min":0.29699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.68200000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57299999999999995,"15min":0.59299999999999997},"min":{"1min":0.38100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.76900000000000002,"5min":0.78800000000000003,"15min":0.82099999999999995},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.36099999999999999,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.72199999999999998,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56399999999999995,"15min":0.58199999999999996},"min":{"1min":0.33100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.749,"5min":1.448,"15min":1.448},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":1.708,"15min":1.708},"last":0.63}]},{"osd":3,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.76000000000000001,"15min":0.81699999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.82999999999999996},"last":0.60199999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.83999999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.81100000000000005},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.253,"15min":0.253},"max":{"1min":0.81100000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.496},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53700000000000003,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.872},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Oct 16 09:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.79100000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575055,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":701604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.38800000000000001,"15min":0.39500000000000002},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":0.52200000000000002,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.51000000000000001,"5min":0.67900000000000005,"15min":0.70199999999999996},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.43099999999999999,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54000000000000004,"5min":4.923,"15min":4.923},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.216},"max":{"1min":0.58699999999999997,"5min":1.712,"15min":1.712},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46400000000000002,"15min":0.45600000000000002},"min":{"1min":0.32100000000000001,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.66500000000000004,"5min":1.984,"15min":1.984},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.41999999999999998,"15min":0.42499999999999999},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.56699999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.379}]},{"osd":4,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61499999999999999,"15min":0.628},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67700000000000005,"15min":0.66500000000000004},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.36599999999999999},"max":{"1min":0.85899999999999999,"5min":4.9749999999999996,"15min":4.9749999999999996},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77400000000000002,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.41399999999999998,"15min":0.38200000000000001},"max":{"1min":0.80700000000000005,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.68100000000000005,"15min":0.65700000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.82899999999999996,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Oct 16 09:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.45800000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.75900000000000001,"5min":1.744,"15min":1.744},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38500000000000001},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.93400000000000005},"last":0.505}]}]},{"osd":0,"up_from":13,"seq":55834575055,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":1974272,"data_stored":1294220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39200000000000002,"15min":0.38300000000000001},"min":{"1min":0.27100000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.442},{"interface":"front","average":{"1min":0.376,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":0.55600000000000005,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53000000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.34799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.63600000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.5}]},{"osd":3,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.224},"max":{"1min":0.57899999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46400000000000002},"min":{"1min":0.27700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59699999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.437,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.77300000000000002,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.32100000000000001},"max":{"1min":0.81100000000000005,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61199999999999999,"15min":0.63500000000000001},"min":{"1min":0.39200000000000002,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.94999999999999996,"15min":0.998},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.82599999999999996,"5min":0.88700000000000001,"15min":0.93000000000000005},"last":0.626}]},{"osd":6,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84999999999999998,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Oct 16 09:59:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":1.014,"15min":1.02},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.66000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575054,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58880,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312960,"statfs":{"total":96636764160,"available":96576471040,"internally_reserved":0,"allocated":1355776,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.221},"max":{"1min":0.51400000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.53800000000000003,"5min":0.64300000000000002,"15min":0.67500000000000004},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.61899999999999999,"5min":0.68000000000000005,"15min":0.754},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.25900000000000001}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40799999999999997,"15min":0.41999999999999998},"min":{"1min":0.29999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64800000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.374}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.64500000000000002},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.33000000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.103},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.65700000000000003},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63100000000000001,"15min":0.64900000000000002},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.046,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65700000000000003,"15min":0.68300000000000005},"min":{"1min":0.441,"5min":0.40799999999999997,"15min":0.40100000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65800000000000003,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.43099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.0069999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.58999999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575054,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36676,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335164,"statfs":{"total":96636764160,"available":96599207936,"internally_reserved":0,"allocated":1368064,"data_stored":708578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.20999999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.308}]},{"osd":1,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.39400000000000002,"15min":0.39500000000000002},"min":{"1min":0.24099999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.65400000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":0.56799999999999995,"5min":0.61899999999999999,"15min":0.67500000000000004},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":0.59999999999999998,"5min":0.70499999999999996,"15min":1.0760000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":0.64100000000000001,"5min":0.73999999999999999,"15min":0.93799999999999994},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.504}]},{"osd":5,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.63500000000000001,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89000000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33200000000000002},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":1.036},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64100000000000001,"15min":0.65000000000000002},"min":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.374},"max":{"1min":0.83299999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63500000000000001,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":1.0589999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Oct 16 09:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42599999999999999},"max":{"1min":0.872,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.42699999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":0.999},"last":0.443}]}]},{"osd":4,"up_from":13,"seq":55834575054,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":701694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.59199999999999997},"min":{"1min":0.34200000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.83099999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.44500000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.72299999999999998,"5min":0.81499999999999995,"15min":0.82599999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.498,"5min":0.56200000000000006,"15min":0.58999999999999997},"min":{"1min":0.33300000000000002,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.66800000000000004,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.622},"min":{"1min":0.438,"5min":0.379,"15min":0.32500000000000001},"max":{"1min":0.79600000000000004,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.499}]},{"osd":3,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75800000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58299999999999996,"15min":0.62},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.84599999999999997},"last":0.66000000000000003}]},{"osd":5,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.749,"5min":0.76400000000000001,"15min":0.84299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.77500000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.84699999999999998,"5min":0.94099999999999995,"15min":0.98999999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Oct 16 09:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80100000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.48199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80800000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.72799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575054,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50572,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321268,"statfs":{"total":96636764160,"available":96584978432,"internally_reserved":0,"allocated":1368064,"data_stored":708578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":0.80600000000000005,"5min":1.792,"15min":1.792},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45700000000000002,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.83099999999999996,"15min":0.875},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.57299999999999995},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.496,"15min":1.496},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.72099999999999997,"5min":2.069,"15min":2.069},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46700000000000003,"15min":0.47799999999999998},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66100000000000003,"5min":0.71299999999999997,"15min":0.84199999999999997},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.86899999999999999},"last":0.72099999999999997}]},{"osd":6,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48699999999999999,"15min":0.48999999999999999},"min":{"1min":0.312,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.84099999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.88200000000000001},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Oct 16 09:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80500000000000005,"5min":0.81799999999999995,"15min":0.90300000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.79500000000000004,"5min":0.84899999999999998,"15min":0.88500000000000001},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:00:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T10:00:39.142 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T10:00:39.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.303+0000 7f2c1c829700 1 Processor -- start 2023-10-16T10:00:39.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.303+0000 7f2c1c829700 1 -- start start 2023-10-16T10:00:39.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c1c829700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c1815e980 0x7f2c1806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c1c829700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180ce620 0x7f2c1806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c1c829700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1815ee10 con 0x7f2c1815d840 2023-10-16T10:00:39.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c1c829700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c181623c0 con 0x7f2c1815e980 2023-10-16T10:00:39.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c1c829700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18161600 con 0x7f2c180ce620 2023-10-16T10:00:39.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c15d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c1815e980 0x7f2c1806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c16d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180ce620 0x7f2c1806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c1659c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2c1815d840 0x7f2c180cdf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:59306/0 (socket says 172.21.15.158:59306) 2023-10-16T10:00:39.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.304+0000 7f2c1659c700 1 -- 172.21.15.158:0/3593350628 learned_addr learned my addr 172.21.15.158:0/3593350628 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:39.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c15d9b700 1 -- 172.21.15.158:0/3593350628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180ce620 msgr2=0x7f2c1806ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:39.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c15d9b700 1 --2- 172.21.15.158:0/3593350628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180ce620 0x7f2c1806ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c15d9b700 1 -- 172.21.15.158:0/3593350628 >> v1:172.21.15.172:6789/0 conn(0x7f2c1815d840 legacy=0x7f2c180cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c15d9b700 1 -- 172.21.15.158:0/3593350628 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c181618e0 con 0x7f2c1815e980 2023-10-16T10:00:39.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c15d9b700 1 --2- 172.21.15.158:0/3593350628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815e980 0x7f2c1806a8d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009840 tx=0x7f2c08009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b22bb557de9417a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:39.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c1559a700 1 -- 172.21.15.158:0/3593350628 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08011830 con 0x7f2c1815e980 2023-10-16T10:00:39.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c1559a700 1 -- 172.21.15.158:0/3593350628 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c08011990 con 0x7f2c1815e980 2023-10-16T10:00:39.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.305+0000 7f2c1559a700 1 -- 172.21.15.158:0/3593350628 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08011d20 con 0x7f2c1815e980 2023-10-16T10:00:39.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 -- 172.21.15.158:0/3593350628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815e980 msgr2=0x7f2c1806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3593350628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815e980 0x7f2c1806a8d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009840 tx=0x7f2c08009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:39.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 -- 172.21.15.158:0/3593350628 shutdown_connections 2023-10-16T10:00:39.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3593350628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180ce620 0x7f2c1806ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3593350628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815e980 0x7f2c1806a8d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 -- 172.21.15.158:0/3593350628 >> 172.21.15.158:0/3593350628 conn(0x7f2c180b1e30 msgr2=0x7f2c180b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:39.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 -- 172.21.15.158:0/3593350628 shutdown_connections 2023-10-16T10:00:39.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.306+0000 7f2c1c829700 1 -- 172.21.15.158:0/3593350628 wait complete. 2023-10-16T10:00:39.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.307+0000 7f2c1c829700 1 Processor -- start 2023-10-16T10:00:39.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.307+0000 7f2c1c829700 1 -- start start 2023-10-16T10:00:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c1c829700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c180ce620 0x7f2c180c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c1c829700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815d840 0x7f2c18157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c1c829700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c180ad780 0x7f2c180ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c1c829700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180c9010 con 0x7f2c180ad780 2023-10-16T10:00:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c1c829700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180adcb0 con 0x7f2c180ce620 2023-10-16T10:00:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c1c829700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18158450 con 0x7f2c1815d840 2023-10-16T10:00:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c180ad780 0x7f2c180ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c180ad780 0x7f2c180ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55234/0 (socket says 172.21.15.158:55234) 2023-10-16T10:00:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 -- 172.21.15.158:0/3737540091 learned_addr learned my addr 172.21.15.158:0/3737540091 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c1659c700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c180ce620 0x7f2c180c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c15d9b700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815d840 0x7f2c18157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 -- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815d840 msgr2=0x7f2c18157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815d840 0x7f2c18157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 -- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c180ce620 msgr2=0x7f2c180c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c180ce620 0x7f2c180c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.308+0000 7f2c16d9d700 1 -- 172.21.15.158:0/3737540091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080094a0 con 0x7f2c180ad780 2023-10-16T10:00:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.309+0000 7f2c16d9d700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c180ad780 0x7f2c180ae9e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000efc0 tx=0x7f2c1000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.309+0000 7f2c06ffd700 1 -- 172.21.15.158:0/3737540091 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1000cc90 con 0x7f2c180ad780 2023-10-16T10:00:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.309+0000 7f2c06ffd700 1 -- 172.21.15.158:0/3737540091 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1000f040 con 0x7f2c180ad780 2023-10-16T10:00:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.309+0000 7f2c06ffd700 1 -- 172.21.15.158:0/3737540091 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100146c0 con 0x7f2c180ad780 2023-10-16T10:00:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.309+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c18158730 con 0x7f2c180ad780 2023-10-16T10:00:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.309+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18158cf0 con 0x7f2c180ad780 2023-10-16T10:00:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.310+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c18158f10 con 0x7f2c180ad780 2023-10-16T10:00:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.310+0000 7f2c06ffd700 1 -- 172.21.15.158:0/3737540091 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2c10014820 con 0x7f2c180ad780 2023-10-16T10:00:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.311+0000 7f2c06ffd700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f2bf806b020 0x7f2bf806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.311+0000 7f2c1659c700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f2bf806b020 0x7f2bf806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.311+0000 7f2c06ffd700 1 -- 172.21.15.158:0/3737540091 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f2c1008a6d0 con 0x7f2c180ad780 2023-10-16T10:00:39.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.311+0000 7f2c1659c700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f2bf806b020 0x7f2bf806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c005910 tx=0x7f2c0c009040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:39.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.315+0000 7f2c06ffd700 1 -- 172.21.15.158:0/3737540091 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2c10055270 con 0x7f2c180ad780 2023-10-16T10:00:39.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.584+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2c180681b0 con 0x7f2c180ad780 2023-10-16T10:00:39.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.586+0000 7f2c06ffd700 1 -- 172.21.15.158:0/3737540091 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2c10009ca0 con 0x7f2c180ad780 2023-10-16T10:00:39.587 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T10:00:39.588 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T10:00:39.588 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T10:00:39.588 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T10:00:39.588 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T10:00:39.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f2bf806b020 msgr2=0x7f2bf806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f2bf806b020 0x7f2bf806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c005910 tx=0x7f2c0c009040 comp rx=0 tx=0).stop 2023-10-16T10:00:39.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c180ad780 msgr2=0x7f2c180ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c180ad780 0x7f2c180ae9e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000efc0 tx=0x7f2c1000c3a0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 shutdown_connections 2023-10-16T10:00:39.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f2bf806b020 0x7f2bf806d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c180ad780 0x7f2c180ae9e0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c180ce620 0x7f2c180c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 --2- 172.21.15.158:0/3737540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815d840 0x7f2c18157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 >> 172.21.15.158:0/3737540091 conn(0x7f2c180b1e30 msgr2=0x7f2c180caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:39.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 shutdown_connections 2023-10-16T10:00:39.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.589+0000 7f2c1c829700 1 -- 172.21.15.158:0/3737540091 wait complete. 2023-10-16T10:00:39.593 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T10:00:39.602 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:00:39.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.723+0000 7f74cd83a700 1 Processor -- start 2023-10-16T10:00:39.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.723+0000 7f74cd83a700 1 -- start start 2023-10-16T10:00:39.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.723+0000 7f74cd83a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f74c815db90 0x7f74c806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.723+0000 7f74cd83a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c80d0fb0 con 0x7f74c80cddb0 2023-10-16T10:00:39.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.723+0000 7f74cd83a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c815e100 con 0x7f74c815d7b0 2023-10-16T10:00:39.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.723+0000 7f74cd83a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c815e3e0 con 0x7f74c815db90 2023-10-16T10:00:39.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.723+0000 7f74c67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f74c815db90 0x7f74c806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f74c80cddb0 0x7f74c80ce1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55420/0 (socket says 172.21.15.158:55420) 2023-10-16T10:00:39.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c77fe700 1 -- 172.21.15.158:0/2426769661 learned_addr learned my addr 172.21.15.158:0/2426769661 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:39.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c67fc700 1 -- 172.21.15.158:0/2426769661 >> v1:172.21.15.172:6789/0 conn(0x7f74c815d7b0 legacy=0x7f74c80b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:39.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c67fc700 1 -- 172.21.15.158:0/2426769661 >> v1:172.21.15.158:6789/0 conn(0x7f74c80cddb0 legacy=0x7f74c80ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c67fc700 1 -- 172.21.15.158:0/2426769661 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c815e6c0 con 0x7f74c815db90 2023-10-16T10:00:39.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c5ffb700 1 -- 172.21.15.158:0/2426769661 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302711117 0 0) 0x7f74c80d0fb0 con 0x7f74c80cddb0 2023-10-16T10:00:39.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c67fc700 1 --2- 172.21.15.158:0/2426769661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c806b9e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009840 tx=0x7f74b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d7f0ebf8b6d191 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:39.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c5ffb700 1 -- 172.21.15.158:0/2426769661 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8004230 con 0x7f74c815db90 2023-10-16T10:00:39.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c5ffb700 1 -- 172.21.15.158:0/2426769661 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b8004390 con 0x7f74c815db90 2023-10-16T10:00:39.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.724+0000 7f74c5ffb700 1 -- 172.21.15.158:0/2426769661 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b80036a0 con 0x7f74c815db90 2023-10-16T10:00:39.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.725+0000 7f74cd83a700 1 -- 172.21.15.158:0/2426769661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 msgr2=0x7f74c806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.725+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2426769661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c806b9e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009840 tx=0x7f74b8009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:39.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.725+0000 7f74cd83a700 1 -- 172.21.15.158:0/2426769661 shutdown_connections 2023-10-16T10:00:39.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.725+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2426769661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c806b9e0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.725+0000 7f74cd83a700 1 -- 172.21.15.158:0/2426769661 >> 172.21.15.158:0/2426769661 conn(0x7f74c8071220 msgr2=0x7f74c806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:39.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.725+0000 7f74cd83a700 1 -- 172.21.15.158:0/2426769661 shutdown_connections 2023-10-16T10:00:39.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.725+0000 7f74cd83a700 1 -- 172.21.15.158:0/2426769661 wait complete. 2023-10-16T10:00:39.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.726+0000 7f74cd83a700 1 Processor -- start 2023-10-16T10:00:39.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.726+0000 7f74cd83a700 1 -- start start 2023-10-16T10:00:39.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74cd83a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74c80cddb0 0x7f74c8161a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74cd83a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74c815d7b0 0x7f74c81514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74cd83a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c8152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74cd83a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8151a30 con 0x7f74c815db90 2023-10-16T10:00:39.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74cd83a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8153fd0 con 0x7f74c80cddb0 2023-10-16T10:00:39.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74cd83a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8153120 con 0x7f74c815d7b0 2023-10-16T10:00:39.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c8152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c8152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55250/0 (socket says 172.21.15.158:55250) 2023-10-16T10:00:39.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 -- 172.21.15.158:0/2848037045 learned_addr learned my addr 172.21.15.158:0/2848037045 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:39.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c6ffd700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74c80cddb0 0x7f74c8161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c67fc700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74c815d7b0 0x7f74c81514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 -- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74c815d7b0 msgr2=0x7f74c81514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74c815d7b0 0x7f74c81514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 -- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74c80cddb0 msgr2=0x7f74c8161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74c80cddb0 0x7f74c8161a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 -- 172.21.15.158:0/2848037045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b80094a0 con 0x7f74c815db90 2023-10-16T10:00:39.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74c77fe700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c8152be0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f74c0002230 tx=0x7f74c000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:39.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74bffff700 1 -- 172.21.15.158:0/2848037045 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c000aac0 con 0x7f74c815db90 2023-10-16T10:00:39.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.727+0000 7f74bffff700 1 -- 172.21.15.158:0/2848037045 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c000ac40 con 0x7f74c815db90 2023-10-16T10:00:39.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.728+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c81533c0 con 0x7f74c815db90 2023-10-16T10:00:39.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.728+0000 7f74bffff700 1 -- 172.21.15.158:0/2848037045 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c00070d0 con 0x7f74c815db90 2023-10-16T10:00:39.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.728+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c8153980 con 0x7f74c815db90 2023-10-16T10:00:39.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.728+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c80d1470 con 0x7f74c815db90 2023-10-16T10:00:39.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.729+0000 7f74bffff700 1 -- 172.21.15.158:0/2848037045 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f74c0007250 con 0x7f74c815db90 2023-10-16T10:00:39.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.729+0000 7f74bffff700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f74a806b020 0x7f74a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:39.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.729+0000 7f74c6ffd700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f74a806b020 0x7f74a806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:39.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.729+0000 7f74bffff700 1 -- 172.21.15.158:0/2848037045 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f74c0089770 con 0x7f74c815db90 2023-10-16T10:00:39.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.730+0000 7f74c6ffd700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f74a806b020 0x7f74a806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74b0007fd0 tx=0x7f74b0007ec0 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:39.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.734+0000 7f74bffff700 1 -- 172.21.15.158:0/2848037045 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74c80d1470 con 0x7f74c815db90 2023-10-16T10:00:39.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.969+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c80d1470 con 0x7f74c815db90 2023-10-16T10:00:39.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.969+0000 7f74bffff700 1 -- 172.21.15.158:0/2848037045 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f74c80d1470 con 0x7f74c815db90 2023-10-16T10:00:39.971 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:39.971 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2674019415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2674019415},{"type":"v1","addr":"172.21.15.158:6836","nonce":2674019415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2425279024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2425279024},{"type":"v1","addr":"172.21.15.158:6837","nonce":2425279024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3002995342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3002995342},{"type":"v1","addr":"172.21.15.172:6834","nonce":3002995342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:00:34.013744+0000","modified":"2023-10-16T10:00:37.052969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7740":{"gid":7740,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1596548674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1596548674},{"type":"v1","addr":"172.21.15.172:6835","nonce":1596548674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T10:00:39.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f74a806b020 msgr2=0x7f74a806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f74a806b020 0x7f74a806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74b0007fd0 tx=0x7f74b0007ec0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 msgr2=0x7f74c8152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:39.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c8152be0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f74c0002230 tx=0x7f74c000cf50 comp rx=0 tx=0).stop 2023-10-16T10:00:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 shutdown_connections 2023-10-16T10:00:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f74a806b020 0x7f74a806d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74c815db90 0x7f74c8152be0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74c80cddb0 0x7f74c8161a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 --2- 172.21.15.158:0/2848037045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74c815d7b0 0x7f74c81514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:39.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 >> 172.21.15.158:0/2848037045 conn(0x7f74c8071220 msgr2=0x7f74c815b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:39.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 shutdown_connections 2023-10-16T10:00:39.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:39.971+0000 7f74cd83a700 1 -- 172.21.15.158:0/2848037045 wait complete. 2023-10-16T10:00:39.975 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 51 2023-10-16T10:00:40.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T10:00:34.013744+0000', 'modified': '2023-10-16T10:00:37.052969+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7740}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7740': {'gid': 7740, 'name': 'b', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/1596548674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 1596548674}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 1596548674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T10:00:40.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T10:00:40.389 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T10:00:40.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.512+0000 7f9fcbe8d700 1 Processor -- start 2023-10-16T10:00:40.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.512+0000 7f9fcbe8d700 1 -- start start 2023-10-16T10:00:40.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fcbe8d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc415d990 0x7f9fc40b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fcbe8d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9fc40cf650 0x7f9fc40cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fcbe8d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc40af600 con 0x7f9fc406e480 2023-10-16T10:00:40.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fcbe8d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc40b2c90 con 0x7f9fc40cf650 2023-10-16T10:00:40.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fcbe8d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc406e9b0 con 0x7f9fc415d990 2023-10-16T10:00:40.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fc9428700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9fc406e480 0x7f9fc40b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55430/0 (socket says 172.21.15.158:55430) 2023-10-16T10:00:40.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fc9428700 1 -- 172.21.15.158:0/3376362931 learned_addr learned my addr 172.21.15.158:0/3376362931 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:40.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fca42a700 1 --2- 172.21.15.158:0/3376362931 >> v2:172.21.15.158:3300/0 conn(0x7f9fc40cf650 0x7f9fc40cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/3376362931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc415d990 0x7f9fc40b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fc8c27700 1 -- 172.21.15.158:0/3376362931 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237943529 0 0) 0x7f9fc40af600 con 0x7f9fc406e480 2023-10-16T10:00:40.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.513+0000 7f9fc8c27700 1 -- 172.21.15.158:0/3376362931 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb00035b0 con 0x7f9fc406e480 2023-10-16T10:00:40.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fca42a700 1 -- 172.21.15.158:0/3376362931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc415d990 msgr2=0x7f9fc40b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fca42a700 1 --2- 172.21.15.158:0/3376362931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc415d990 0x7f9fc40b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fca42a700 1 -- 172.21.15.158:0/3376362931 >> v1:172.21.15.158:6789/0 conn(0x7f9fc406e480 legacy=0x7f9fc40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fca42a700 1 -- 172.21.15.158:0/3376362931 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc40ce890 con 0x7f9fc40cf650 2023-10-16T10:00:40.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fca42a700 1 --2- 172.21.15.158:0/3376362931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc40cf650 0x7f9fc40cfb00 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009d90 tx=0x7f9fbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f836c75dc8445b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fc8c27700 1 -- 172.21.15.158:0/3376362931 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc00e070 con 0x7f9fc40cf650 2023-10-16T10:00:40.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fc8c27700 1 -- 172.21.15.158:0/3376362931 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc00bb20 con 0x7f9fc40cf650 2023-10-16T10:00:40.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fc8c27700 1 -- 172.21.15.158:0/3376362931 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc012430 con 0x7f9fc40cf650 2023-10-16T10:00:40.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/3376362931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc40cf650 msgr2=0x7f9fc40cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/3376362931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc40cf650 0x7f9fc40cfb00 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009d90 tx=0x7f9fbc00b330 comp rx=0 tx=0).stop 2023-10-16T10:00:40.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/3376362931 shutdown_connections 2023-10-16T10:00:40.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.514+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/3376362931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc40cf650 0x7f9fc40cfb00 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.515+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/3376362931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc415d990 0x7f9fc40b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.515+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/3376362931 >> 172.21.15.158:0/3376362931 conn(0x7f9fc406b9e0 msgr2=0x7f9fc406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.515+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/3376362931 shutdown_connections 2023-10-16T10:00:40.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.515+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/3376362931 wait complete. 2023-10-16T10:00:40.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.516+0000 7f9fcbe8d700 1 Processor -- start 2023-10-16T10:00:40.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.516+0000 7f9fcbe8d700 1 -- start start 2023-10-16T10:00:40.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.516+0000 7f9fcbe8d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc406e480 0x7f9fc41526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.516+0000 7f9fcbe8d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc40cf650 0x7f9fc415e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fcbe8d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc415d990 0x7f9fc41e7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fcbe8d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc41600e0 con 0x7f9fc415d990 2023-10-16T10:00:40.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fcbe8d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc415dd70 con 0x7f9fc40cf650 2023-10-16T10:00:40.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fcbe8d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc415ed00 con 0x7f9fc406e480 2023-10-16T10:00:40.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc415d990 0x7f9fc41e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc415d990 0x7f9fc41e7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55262/0 (socket says 172.21.15.158:55262) 2023-10-16T10:00:40.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 -- 172.21.15.158:0/2798601674 learned_addr learned my addr 172.21.15.158:0/2798601674 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:40.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc406e480 0x7f9fc41526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fc9428700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc40cf650 0x7f9fc415e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 -- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc406e480 msgr2=0x7f9fc41526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc406e480 0x7f9fc41526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 -- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc40cf650 msgr2=0x7f9fc415e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc40cf650 0x7f9fc415e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 -- 172.21.15.158:0/2798601674 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc009a70 con 0x7f9fc415d990 2023-10-16T10:00:40.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc406e480 0x7f9fc41526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:40.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fc9428700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc40cf650 0x7f9fc415e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:40.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fca42a700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc415d990 0x7f9fc41e7310 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc014040 tx=0x7f9fbc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fba7fc700 1 -- 172.21.15.158:0/2798601674 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc012430 con 0x7f9fc415d990 2023-10-16T10:00:40.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fba7fc700 1 -- 172.21.15.158:0/2798601674 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc020070 con 0x7f9fc415d990 2023-10-16T10:00:40.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc41e7dd0 con 0x7f9fc415d990 2023-10-16T10:00:40.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fba7fc700 1 -- 172.21.15.158:0/2798601674 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc003ea0 con 0x7f9fc415d990 2023-10-16T10:00:40.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.517+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc41e8160 con 0x7f9fc415d990 2023-10-16T10:00:40.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.519+0000 7f9fba7fc700 1 -- 172.21.15.158:0/2798601674 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9fbc004280 con 0x7f9fc415d990 2023-10-16T10:00:40.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.519+0000 7f9fba7fc700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9fa806b020 0x7f9fa806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.519+0000 7f9fba7fc700 1 -- 172.21.15.158:0/2798601674 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9fbc022070 con 0x7f9fc415d990 2023-10-16T10:00:40.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.519+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9fa806b020 0x7f9fa806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.519+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9fa806b020 0x7f9fa806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009820 tx=0x7f9fb4008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.521+0000 7f9fd8077b80 1 Processor -- start 2023-10-16T10:00:40.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.521+0000 7f9fd8077b80 1 -- start start 2023-10-16T10:00:40.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8ff5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8ff3630 0x564eb8ff5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fc9010 0x564eb8fc96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb8f3fc20 con 0x564eb8ff3630 2023-10-16T10:00:40.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb8f24c10 con 0x564eb8fee220 2023-10-16T10:00:40.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564eb8f22530 con 0x564eb8fc9010 2023-10-16T10:00:40.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8ff3630 0x564eb8ff5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8ff3630 0x564eb8ff5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55268/0 (socket says 172.21.15.158:55268) 2023-10-16T10:00:40.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 -- 172.21.15.158:0/3600608848 learned_addr learned my addr 172.21.15.158:0/3600608848 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:40.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8ff5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fca42a700 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fc9010 0x564eb8fc96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 -- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fc9010 msgr2=0x564eb8fc96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fc9010 0x564eb8fc96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 -- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 msgr2=0x564eb8ff5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8ff5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 -- 172.21.15.158:0/3600608848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564eb8edb3a0 con 0x564eb8ff3630 2023-10-16T10:00:40.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8ff5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:40.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fc9428700 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8ff3630 0x564eb8ff5fb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00011e0 tx=0x7f9fc0001960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5afbcba7262c4f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fa77fe700 1 -- 172.21.15.158:0/3600608848 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0005d70 con 0x564eb8ff3630 2023-10-16T10:00:40.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fa77fe700 1 -- 172.21.15.158:0/3600608848 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc000d050 con 0x564eb8ff3630 2023-10-16T10:00:40.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.522+0000 7f9fa77fe700 1 -- 172.21.15.158:0/3600608848 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0007310 con 0x564eb8ff3630 2023-10-16T10:00:40.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 -- 172.21.15.158:0/3600608848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8ff3630 msgr2=0x564eb8ff5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8ff3630 0x564eb8ff5fb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00011e0 tx=0x7f9fc0001960 comp rx=0 tx=0).stop 2023-10-16T10:00:40.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 -- 172.21.15.158:0/3600608848 shutdown_connections 2023-10-16T10:00:40.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fc9010 0x564eb8fc96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8ff5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/3600608848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8ff3630 0x564eb8ff5fb0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 -- 172.21.15.158:0/3600608848 >> 172.21.15.158:0/3600608848 conn(0x564eb8ef8d80 msgr2=0x564eb9013ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 -- 172.21.15.158:0/3600608848 shutdown_connections 2023-10-16T10:00:40.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.523+0000 7f9fd8077b80 1 -- 172.21.15.158:0/3600608848 wait complete. 2023-10-16T10:00:40.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.524+0000 7f9fd8077b80 1 -- start start 2023-10-16T10:00:40.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.524+0000 7f9fd8077b80 1 Processor -- start 2023-10-16T10:00:40.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8ff8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fee220 0x564eb8fc99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8ff3630 0x564eb8fc9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb8f1de20 con 0x564eb8fc9010 2023-10-16T10:00:40.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb8eba6c0 con 0x564eb8ff3630 2023-10-16T10:00:40.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564eb8eba360 con 0x564eb8fee220 2023-10-16T10:00:40.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fd8077b80 10 client.0 _ll_register_callbacks cb 0x564eb8d177b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:00:40.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8ff8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8ff8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55272/0 (socket says 172.21.15.158:55272) 2023-10-16T10:00:40.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 -- 172.21.15.158:0/2486132139 learned_addr learned my addr 172.21.15.158:0/2486132139 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:40.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9428700 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fee220 0x564eb8fc99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fca42a700 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8ff3630 0x564eb8fc9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 -- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fee220 msgr2=0x564eb8fc99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fee220 0x564eb8fc99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 -- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8ff3630 msgr2=0x564eb8fc9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8ff3630 0x564eb8fc9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 -- 172.21.15.158:0/2486132139 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb00035b0 con 0x564eb8fc9010 2023-10-16T10:00:40.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8ff8e70 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4015e90 tx=0x7f9fb40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586911bfd275c2c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fa4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:00:40.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.525+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/2486132139 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4015520 con 0x564eb8fc9010 2023-10-16T10:00:40.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/2486132139 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9fb4013d70 con 0x564eb8fc9010 2023-10-16T10:00:40.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564eb8fbe5b0 con 0x564eb8fc9010 2023-10-16T10:00:40.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/2486132139 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fb4021420 con 0x564eb8fc9010 2023-10-16T10:00:40.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 10 client.7878 fetch_fsmap learned FSMap version 51 2023-10-16T10:00:40.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x564eb8d48d90 con 0x564eb8fc9010 2023-10-16T10:00:40.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/2486132139 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564eb8fbe5b0 con 0x564eb8fc9010 2023-10-16T10:00:40.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/2486132139 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9fb4029350 con 0x564eb8fc9010 2023-10-16T10:00:40.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 10 client.7878 fetch_fsmap finished waiting for FSMap version 51 2023-10-16T10:00:40.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 10 client.7878 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T10:00:40.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 20 client.7878 populate_metadata read hostname 'smithi158' 2023-10-16T10:00:40.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb902d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 4 client.7878 mds_command: new command op to 7740 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T10:00:40.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 --> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564eb8f16e00 con 0x564eb8ff8370 2023-10-16T10:00:40.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.526+0000 7f9fc9428700 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb902d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fa4ff9700 10 client.7878 ms_handle_connect on v2:172.21.15.172:6833/1596548674 2023-10-16T10:00:40.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fc9428700 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb902d460 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb3b466dd4306939 server_cookie=791c079659fdf97a in_seq=0 out_seq=0 2023-10-16T10:00:40.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/2486132139 <== mds.0 v2:172.21.15.172:6833/1596548674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564eb8f16e00 con 0x564eb8ff8370 2023-10-16T10:00:40.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fa4ff9700 10 client.7878 handle_command_reply: tid=0 2023-10-16T10:00:40.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 client.7878 shutdown 2023-10-16T10:00:40.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 msgr2=0x564eb8ff8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8ff8e70 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4015e90 tx=0x7f9fb40047c0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 shutdown_connections 2023-10-16T10:00:40.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb902d460 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8ff3630 0x564eb8fc9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8ff8e70 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fa4ff9700 0 client.7878 ms_handle_reset on v2:172.21.15.172:6833/1596548674 2023-10-16T10:00:40.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/2486132139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8fee220 0x564eb8fc99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.527+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 >> 172.21.15.158:0/2486132139 conn(0x564eb8ef8d80 msgr2=0x564eb8fc07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.528+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 shutdown_connections 2023-10-16T10:00:40.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.528+0000 7f9fd8077b80 1 -- 172.21.15.158:0/2486132139 wait complete. 2023-10-16T10:00:40.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.528+0000 7f9fd8077b80 20 client.7878 trim_cache size 0 max 16384 2023-10-16T10:00:40.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.545+0000 7f9fd8077b80 1 Processor -- start 2023-10-16T10:00:40.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.545+0000 7f9fd8077b80 1 -- start start 2023-10-16T10:00:40.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8fc5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fcac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb8d9e8d0 con 0x564eb8fc9010 2023-10-16T10:00:40.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb8ff8900 con 0x564eb8fee220 2023-10-16T10:00:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564eb8feff00 con 0x564eb8ff3630 2023-10-16T10:00:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8fc5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8fc5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55280/0 (socket says 172.21.15.158:55280) 2023-10-16T10:00:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 -- 172.21.15.158:0/708771876 learned_addr learned my addr 172.21.15.158:0/708771876 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fca42a700 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fcac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9428700 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 -- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 msgr2=0x564eb8fcac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fcac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 -- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 msgr2=0x564eb8fca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9c29700 1 -- 172.21.15.158:0/708771876 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564eb8d31c10 con 0x564eb8fc9010 2023-10-16T10:00:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fca42a700 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fcac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.546+0000 7f9fc9428700 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:00:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8fc5c80 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40139e0 tx=0x7f9fb401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811ccb780e7388ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fa7fff700 1 -- 172.21.15.158:0/708771876 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4029490 con 0x564eb8fc9010 2023-10-16T10:00:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fa7fff700 1 -- 172.21.15.158:0/708771876 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb4013d70 con 0x564eb8fc9010 2023-10-16T10:00:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fa7fff700 1 -- 172.21.15.158:0/708771876 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4013ee0 con 0x564eb8fc9010 2023-10-16T10:00:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 -- 172.21.15.158:0/708771876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 msgr2=0x564eb8fc5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8fc5c80 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40139e0 tx=0x7f9fb401f000 comp rx=0 tx=0).stop 2023-10-16T10:00:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 -- 172.21.15.158:0/708771876 shutdown_connections 2023-10-16T10:00:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fcac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb8fc5c80 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/708771876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 -- 172.21.15.158:0/708771876 >> 172.21.15.158:0/708771876 conn(0x564eb8ef8d80 msgr2=0x564eb8fc7850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 -- 172.21.15.158:0/708771876 shutdown_connections 2023-10-16T10:00:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.547+0000 7f9fd8077b80 1 -- 172.21.15.158:0/708771876 wait complete. 2023-10-16T10:00:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.548+0000 7f9fd8077b80 1 -- start start 2023-10-16T10:00:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.548+0000 7f9fd8077b80 1 Processor -- start 2023-10-16T10:00:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb900cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fc5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fc61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb900ece0 con 0x564eb8fc9010 2023-10-16T10:00:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eb8feb6b0 con 0x564eb8fee220 2023-10-16T10:00:40.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564eb8fec760 con 0x564eb8ff3630 2023-10-16T10:00:40.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 10 client.0 _ll_register_callbacks cb 0x7f9fb4015990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:00:40.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb900cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb900cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55284/0 (socket says 172.21.15.158:55284) 2023-10-16T10:00:40.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 -- 172.21.15.158:0/416882860 learned_addr learned my addr 172.21.15.158:0/416882860 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:40.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fca42a700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fc61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9428700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fc5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 -- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 msgr2=0x564eb8fc61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fc61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 -- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 msgr2=0x564eb8fc5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fc5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 -- 172.21.15.158:0/416882860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564eb8d48d90 con 0x564eb8fc9010 2023-10-16T10:00:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9428700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fc5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fc9c29700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb900cfe0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009f70 tx=0x7f9fb4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d91c553e43b465 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fa4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:00:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/416882860 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40042f0 con 0x564eb8fc9010 2023-10-16T10:00:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/416882860 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9fb4004450 con 0x564eb8fc9010 2023-10-16T10:00:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.549+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9fc000d050 con 0x564eb8fc9010 2023-10-16T10:00:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/416882860 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fb40297c0 con 0x564eb8fc9010 2023-10-16T10:00:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 10 client.7884 fetch_fsmap learned FSMap version 51 2023-10-16T10:00:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x564eb8d6f200 con 0x564eb8fc9010 2023-10-16T10:00:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/416882860 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc000d050 con 0x564eb8fc9010 2023-10-16T10:00:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/416882860 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9fb4038070 con 0x564eb8fc9010 2023-10-16T10:00:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 10 client.7884 fetch_fsmap finished waiting for FSMap version 51 2023-10-16T10:00:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 10 client.7884 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T10:00:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 20 client.7884 populate_metadata read hostname 'smithi158' 2023-10-16T10:00:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb901eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 4 client.7884 mds_command: new command op to 7740 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T10:00:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 --> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564eb8f16b20 con 0x564eb8ff8370 2023-10-16T10:00:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fc9428700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb901eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fa4ff9700 10 client.7884 ms_handle_connect on v2:172.21.15.172:6833/1596548674 2023-10-16T10:00:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.550+0000 7f9fc9428700 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb901eaa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96e2d31faf9ccd91 server_cookie=67bf0bff405ea75f in_seq=0 out_seq=0 2023-10-16T10:00:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fa4ff9700 1 -- 172.21.15.158:0/416882860 <== mds.0 v2:172.21.15.172:6833/1596548674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x564eb8f16b20 con 0x564eb8ff8370 2023-10-16T10:00:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fa4ff9700 10 client.7884 handle_command_reply: tid=0 2023-10-16T10:00:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 client.7884 shutdown 2023-10-16T10:00:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 msgr2=0x564eb900cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb900cfe0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009f70 tx=0x7f9fb4013000 comp rx=0 tx=0).stop 2023-10-16T10:00:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 shutdown_connections 2023-10-16T10:00:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:6833/1596548674,v1:172.21.15.172:6835/1596548674] conn(0x564eb8ff8370 0x564eb901eaa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564eb8ff3630 0x564eb8fc61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fa4ff9700 0 client.7884 ms_handle_reset on v2:172.21.15.172:6833/1596548674 2023-10-16T10:00:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564eb8fc9010 0x564eb900cfe0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 --2- 172.21.15.158:0/416882860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564eb8fee220 0x564eb8fc5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 >> 172.21.15.158:0/416882860 conn(0x564eb8ef8d80 msgr2=0x564eb9010ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 shutdown_connections 2023-10-16T10:00:40.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 1 -- 172.21.15.158:0/416882860 wait complete. 2023-10-16T10:00:40.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fd8077b80 20 client.7884 trim_cache size 0 max 16384 2023-10-16T10:00:40.579 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:00:40.580 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T10:00:40.580 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T10:00:40.580 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 7740, 2023-10-16T10:00:40.580 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T10:00:40.580 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T10:00:40.581 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T10:00:40.581 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9186976019999999, 2023-10-16T10:00:40.581 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 51, 2023-10-16T10:00:40.581 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 111, 2023-10-16T10:00:40.581 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 111, 2023-10-16T10:00:40.581 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.826443489000001 2023-10-16T10:00:40.582 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:00:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9fa806b020 msgr2=0x7f9fa806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9fa806b020 0x7f9fa806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009820 tx=0x7f9fb4008040 comp rx=0 tx=0).stop 2023-10-16T10:00:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc415d990 msgr2=0x7f9fc41e7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc415d990 0x7f9fc41e7310 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc014040 tx=0x7f9fbc004ad0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 shutdown_connections 2023-10-16T10:00:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9fa806b020 0x7f9fa806d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fc415d990 0x7f9fc41e7310 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9fc406e480 0x7f9fc41526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.551+0000 7f9fcbe8d700 1 --2- 172.21.15.158:0/2798601674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9fc40cf650 0x7f9fc415e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.552+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 >> 172.21.15.158:0/2798601674 conn(0x7f9fc406b9e0 msgr2=0x7f9fc406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.552+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 shutdown_connections 2023-10-16T10:00:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.552+0000 7f9fcbe8d700 1 -- 172.21.15.158:0/2798601674 wait complete. 2023-10-16T10:00:40.586 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T10:00:40.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.707+0000 7f28e2ecc700 1 Processor -- start 2023-10-16T10:00:40.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.707+0000 7f28e2ecc700 1 -- start start 2023-10-16T10:00:40.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e2ecc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28dc06ab40 0x7f28dc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e2ecc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06e5d0 0x7f28dc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e2ecc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28dc071e50 con 0x7f28dc06af20 2023-10-16T10:00:40.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e2ecc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc161270 con 0x7f28dc06ab40 2023-10-16T10:00:40.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e2ecc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc161510 con 0x7f28dc06e5d0 2023-10-16T10:00:40.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28dc06ab40 0x7f28dc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28dc06ab40 0x7f28dc06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55292/0 (socket says 172.21.15.158:55292) 2023-10-16T10:00:40.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 -- 172.21.15.158:0/2331738002 learned_addr learned my addr 172.21.15.158:0/2331738002 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:40.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 -- 172.21.15.158:0/2331738002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06e5d0 msgr2=0x7f28dc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28dbfff700 1 --2- 172.21.15.158:0/2331738002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06e5d0 0x7f28dc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 --2- 172.21.15.158:0/2331738002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06e5d0 0x7f28dc071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 -- 172.21.15.158:0/2331738002 >> v1:172.21.15.172:6789/0 conn(0x7f28dc06af20 legacy=0x7f28dc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:40.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 -- 172.21.15.158:0/2331738002 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc1617f0 con 0x7f28dc06ab40 2023-10-16T10:00:40.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28e0c68700 1 --2- 172.21.15.158:0/2331738002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06ab40 0x7f28dc06d820 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f28d00098f0 tx=0x7f28d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905ca140214cbc1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.708+0000 7f28db7fe700 1 -- 172.21.15.158:0/2331738002 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0004230 con 0x7f28dc06ab40 2023-10-16T10:00:40.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28db7fe700 1 -- 172.21.15.158:0/2331738002 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d0004390 con 0x7f28dc06ab40 2023-10-16T10:00:40.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28db7fe700 1 -- 172.21.15.158:0/2331738002 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00036a0 con 0x7f28dc06ab40 2023-10-16T10:00:40.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2331738002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06ab40 msgr2=0x7f28dc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2331738002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06ab40 0x7f28dc06d820 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f28d00098f0 tx=0x7f28d0009c00 comp rx=0 tx=0).stop 2023-10-16T10:00:40.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2331738002 shutdown_connections 2023-10-16T10:00:40.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2331738002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06ab40 0x7f28dc06d820 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2331738002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06e5d0 0x7f28dc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2331738002 >> 172.21.15.158:0/2331738002 conn(0x7f28dc0d02b0 msgr2=0x7f28dc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2331738002 shutdown_connections 2023-10-16T10:00:40.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.709+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2331738002 wait complete. 2023-10-16T10:00:40.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.710+0000 7f28e2ecc700 1 Processor -- start 2023-10-16T10:00:40.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.710+0000 7f28e2ecc700 1 -- start start 2023-10-16T10:00:40.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e2ecc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28dc06ab40 0x7f28dc15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e2ecc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06af20 0x7f28dc0c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e2ecc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06e5d0 0x7f28dc160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e2ecc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0b0ac0 con 0x7f28dc06e5d0 2023-10-16T10:00:40.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e2ecc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0b0da0 con 0x7f28dc06ab40 2023-10-16T10:00:40.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e2ecc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0b1080 con 0x7f28dc06af20 2023-10-16T10:00:40.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e1469700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06e5d0 0x7f28dc160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e1469700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06e5d0 0x7f28dc160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55302/0 (socket says 172.21.15.158:55302) 2023-10-16T10:00:40.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.711+0000 7f28e1469700 1 -- 172.21.15.158:0/2070076512 learned_addr learned my addr 172.21.15.158:0/2070076512 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:40.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28dbfff700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06af20 0x7f28dc0c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e0c68700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28dc06ab40 0x7f28dc15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e1469700 1 -- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06af20 msgr2=0x7f28dc0c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e1469700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06af20 0x7f28dc0c5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e1469700 1 -- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28dc06ab40 msgr2=0x7f28dc15f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e1469700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28dc06ab40 0x7f28dc15f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e1469700 1 -- 172.21.15.158:0/2070076512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0009550 con 0x7f28dc06e5d0 2023-10-16T10:00:40.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28dbfff700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06af20 0x7f28dc0c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:40.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e1469700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06e5d0 0x7f28dc160160 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f28d400c7a0 tx=0x7f28d400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28d97fa700 1 -- 172.21.15.158:0/2070076512 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4003e10 con 0x7f28dc06e5d0 2023-10-16T10:00:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28d97fa700 1 -- 172.21.15.158:0/2070076512 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d4005070 con 0x7f28dc06e5d0 2023-10-16T10:00:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc0b1360 con 0x7f28dc06e5d0 2023-10-16T10:00:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28d97fa700 1 -- 172.21.15.158:0/2070076512 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4005420 con 0x7f28dc06e5d0 2023-10-16T10:00:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.712+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc0c6080 con 0x7f28dc06e5d0 2023-10-16T10:00:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.713+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc0b0c00 con 0x7f28dc06e5d0 2023-10-16T10:00:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.713+0000 7f28d97fa700 1 -- 172.21.15.158:0/2070076512 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f28d4005580 con 0x7f28dc06e5d0 2023-10-16T10:00:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.714+0000 7f28d97fa700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f28c006b020 0x7f28c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.714+0000 7f28d97fa700 1 -- 172.21.15.158:0/2070076512 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f28d4089170 con 0x7f28dc06e5d0 2023-10-16T10:00:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.714+0000 7f28e0c68700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f28c006b020 0x7f28c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:40.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.714+0000 7f28e0c68700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f28c006b020 0x7f28c006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009130 tx=0x7f28d0008030 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:40.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.718+0000 7f28d97fa700 1 -- 172.21.15.158:0/2070076512 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f28dc0b0c00 con 0x7f28dc06e5d0 2023-10-16T10:00:40.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.953+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f28dc0b0c00 con 0x7f28dc06e5d0 2023-10-16T10:00:40.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.954+0000 7f28d97fa700 1 -- 172.21.15.158:0/2070076512 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7f28dc0b0c00 con 0x7f28dc06e5d0 2023-10-16T10:00:40.955 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T10:00:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f28c006b020 msgr2=0x7f28c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f28c006b020 0x7f28c006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009130 tx=0x7f28d0008030 comp rx=0 tx=0).stop 2023-10-16T10:00:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06e5d0 msgr2=0x7f28dc160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06e5d0 0x7f28dc160160 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f28d400c7a0 tx=0x7f28d400cb60 comp rx=0 tx=0).stop 2023-10-16T10:00:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 shutdown_connections 2023-10-16T10:00:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f28c006b020 0x7f28c006d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28dc06e5d0 0x7f28dc160160 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28dc06ab40 0x7f28dc15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 --2- 172.21.15.158:0/2070076512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28dc06af20 0x7f28dc0c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 >> 172.21.15.158:0/2070076512 conn(0x7f28dc0d02b0 msgr2=0x7f28dc0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 shutdown_connections 2023-10-16T10:00:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:40.956+0000 7f28e2ecc700 1 -- 172.21.15.158:0/2070076512 wait complete. 2023-10-16T10:00:40.969 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T10:00:40.969 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T10:00:40.970 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T10:00:40.970 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T10:00:40.970 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:40.971 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T10:00:40.971 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T10:00:40.971 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T10:00:40.972 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:00:40.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:40.972 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T10:00:40.990 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T10:00:40.990 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T10:00:40.990 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T10:00:40.990 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:40.991 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T10:00:40.991 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:40.991 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:00:40.991 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:40.991 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:00:40.992 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:40.992 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:00:40.992 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:40.992 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T10:00:40.992 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T10:00:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 41392sec preferred_lft 41392sec 2023-10-16T10:00:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T10:00:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:40.993 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T10:00:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T10:00:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T10:00:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T10:00:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:40.995 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:00:40.995 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T10:00:40.995 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T10:00:40.995 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:40.995 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:00:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T10:00:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T10:00:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:00:40.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:40.997 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T10:00:41.055 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T10:00:41.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:41.056 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T10:00:41.115 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T10:00:41.115 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T10:00:41.115 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T10:00:41.116 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T10:00:41.116 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T10:00:41.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:41.117 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T10:00:41.117 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T10:00:41.117 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T10:00:41.118 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T10:00:41.118 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T10:00:41.119 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T10:00:41.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:41.220 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T10:00:41.237 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T10:00:41.237 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T10:00:41.237 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:00:41.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:41.238 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T10:00:41.304 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:00:41.304 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:41.361 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:00:41.361 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:00:41.362 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:41.417 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T10:00:41.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:41.418 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:41.502 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:41.502 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:00:41.525 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:41.526 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:00:41.526 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:00:41.526 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:00:41.526 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:00:41.527 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:00:41.527 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:00:41.527 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:00:41.553 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:00:41.553 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:00:41.553 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:00:41.553 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:00:41.554 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:00:41.554 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:00:41.554 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:00:41.554 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:00:41.554 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T10:00:41.555 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T10:00:41.555 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T10:00:41.557 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T10:00:41.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:41.558 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:00:41.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:00:41.587 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T10:00:41.675 INFO:teuthology.orchestra.run.smithi158.stdout:client7893 2023-10-16T10:00:41.675 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:00:41.676 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:00:41.693 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T10:00:41.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.853+0000 7fd36deb9700 1 Processor -- start 2023-10-16T10:00:41.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.853+0000 7fd36deb9700 1 -- start start 2023-10-16T10:00:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd36deb9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd36815d990 0x7fd3680b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd36deb9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3680cf650 0x7fd3680cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd36deb9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3680af600 con 0x7fd36806e480 2023-10-16T10:00:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd36deb9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680d0020 con 0x7fd36815d990 2023-10-16T10:00:41.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd36deb9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680b2c90 con 0x7fd3680cf650 2023-10-16T10:00:41.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd3677fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd36815d990 0x7fd3680b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:41.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd366ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd36806e480 0x7fd3680b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55488/0 (socket says 172.21.15.158:55488) 2023-10-16T10:00:41.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd366ffd700 1 -- 172.21.15.158:0/2791955617 learned_addr learned my addr 172.21.15.158:0/2791955617 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:41.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.854+0000 7fd367fff700 1 --2- 172.21.15.158:0/2791955617 >> v2:172.21.15.172:3300/0 conn(0x7fd3680cf650 0x7fd3680cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:41.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3677fe700 1 -- 172.21.15.158:0/2791955617 >> v2:172.21.15.172:3300/0 conn(0x7fd3680cf650 msgr2=0x7fd3680cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:41.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3677fe700 1 --2- 172.21.15.158:0/2791955617 >> v2:172.21.15.172:3300/0 conn(0x7fd3680cf650 0x7fd3680cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:41.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3677fe700 1 -- 172.21.15.158:0/2791955617 >> v1:172.21.15.158:6789/0 conn(0x7fd36806e480 legacy=0x7fd3680b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:41.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3677fe700 1 -- 172.21.15.158:0/2791955617 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36806e9b0 con 0x7fd36815d990 2023-10-16T10:00:41.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3667fc700 1 -- 172.21.15.158:0/2791955617 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069431308 0 0) 0x7fd3680af600 con 0x7fd36806e480 2023-10-16T10:00:41.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3677fe700 1 --2- 172.21.15.158:0/2791955617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36815d990 0x7fd3680b2750 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd368070bb0 tx=0x7fd35c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1c618bf02249c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:41.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3667fc700 1 -- 172.21.15.158:0/2791955617 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c011830 con 0x7fd36815d990 2023-10-16T10:00:41.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3667fc700 1 -- 172.21.15.158:0/2791955617 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c011990 con 0x7fd36815d990 2023-10-16T10:00:41.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd3667fc700 1 -- 172.21.15.158:0/2791955617 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c011d20 con 0x7fd36815d990 2023-10-16T10:00:41.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd36deb9700 1 -- 172.21.15.158:0/2791955617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36815d990 msgr2=0x7fd3680b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:41.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2791955617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36815d990 0x7fd3680b2750 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd368070bb0 tx=0x7fd35c009b20 comp rx=0 tx=0).stop 2023-10-16T10:00:41.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd36deb9700 1 -- 172.21.15.158:0/2791955617 shutdown_connections 2023-10-16T10:00:41.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2791955617 >> v2:172.21.15.172:3300/0 conn(0x7fd3680cf650 0x7fd3680cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:41.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2791955617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36815d990 0x7fd3680b2750 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:41.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd36deb9700 1 -- 172.21.15.158:0/2791955617 >> 172.21.15.158:0/2791955617 conn(0x7fd36806b9e0 msgr2=0x7fd36806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:41.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.855+0000 7fd36deb9700 1 -- 172.21.15.158:0/2791955617 shutdown_connections 2023-10-16T10:00:41.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.856+0000 7fd36deb9700 1 -- 172.21.15.158:0/2791955617 wait complete. 2023-10-16T10:00:41.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.857+0000 7fd36deb9700 1 Processor -- start 2023-10-16T10:00:41.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.857+0000 7fd36deb9700 1 -- start start 2023-10-16T10:00:41.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36806e480 0x7fd3681f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:41.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3680cf650 0x7fd3680cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:41.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd36815d990 0x7fd3680718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:41.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680ccdd0 con 0x7fd36806e480 2023-10-16T10:00:41.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680cb470 con 0x7fd3680cf650 2023-10-16T10:00:41.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680cab80 con 0x7fd36815d990 2023-10-16T10:00:41.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36806e480 0x7fd3681f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:41.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36806e480 0x7fd3681f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55316/0 (socket says 172.21.15.158:55316) 2023-10-16T10:00:41.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 -- 172.21.15.158:0/2203977822 learned_addr learned my addr 172.21.15.158:0/2203977822 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:41.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd366ffd700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3680cf650 0x7fd3680cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:41.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 -- 172.21.15.158:0/2203977822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd36815d990 msgr2=0x7fd3680718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:41.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd36815d990 0x7fd3680718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:41.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 -- 172.21.15.158:0/2203977822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3680cf650 msgr2=0x7fd3680cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:41.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3680cf650 0x7fd3680cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:41.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 -- 172.21.15.158:0/2203977822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c0094a0 con 0x7fd36806e480 2023-10-16T10:00:41.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd3677fe700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36806e480 0x7fd3681f2840 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0090b0 tx=0x7fd35c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:41.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd34ffff700 1 -- 172.21.15.158:0/2203977822 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c0038a0 con 0x7fd36806e480 2023-10-16T10:00:41.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd34ffff700 1 -- 172.21.15.158:0/2203977822 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c01e070 con 0x7fd36806e480 2023-10-16T10:00:41.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd34ffff700 1 -- 172.21.15.158:0/2203977822 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c003c60 con 0x7fd36806e480 2023-10-16T10:00:41.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3680cae20 con 0x7fd36806e480 2023-10-16T10:00:41.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.858+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3681f2d80 con 0x7fd36806e480 2023-10-16T10:00:41.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.859+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3680681b0 con 0x7fd36806e480 2023-10-16T10:00:41.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.860+0000 7fd34ffff700 1 -- 172.21.15.158:0/2203977822 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd35c01a420 con 0x7fd36806e480 2023-10-16T10:00:41.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.860+0000 7fd34ffff700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fd35406b020 0x7fd35406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:41.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.860+0000 7fd34ffff700 1 -- 172.21.15.158:0/2203977822 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fd35c08aac0 con 0x7fd36806e480 2023-10-16T10:00:41.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.860+0000 7fd366ffd700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fd35406b020 0x7fd35406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:41.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.861+0000 7fd366ffd700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fd35406b020 0x7fd35406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3680c6970 tx=0x7fd358009040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:41.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:41.865+0000 7fd34ffff700 1 -- 172.21.15.158:0/2203977822 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd35c0556f0 con 0x7fd36806e480 2023-10-16T10:00:42.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.085+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 --> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3680ceb70 con 0x7fd35406b020 2023-10-16T10:00:42.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.087+0000 7fd34ffff700 1 -- 172.21.15.158:0/2203977822 <== mgr.6895 v2:172.21.15.158:6832/1430981267 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd3580013d0 con 0x7fd35406b020 2023-10-16T10:00:42.088 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:00:42.088 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:00:42.089 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T10:00:42.089 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:00:42.089 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:00:42.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fd35406b020 msgr2=0x7fd35406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fd35406b020 0x7fd35406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3680c6970 tx=0x7fd358009040 comp rx=0 tx=0).stop 2023-10-16T10:00:42.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36806e480 msgr2=0x7fd3681f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36806e480 0x7fd3681f2840 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0090b0 tx=0x7fd35c004750 comp rx=0 tx=0).stop 2023-10-16T10:00:42.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 shutdown_connections 2023-10-16T10:00:42.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fd35406b020 0x7fd35406d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd36815d990 0x7fd3680718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd36806e480 0x7fd3681f2840 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 --2- 172.21.15.158:0/2203977822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3680cf650 0x7fd3680cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.089+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 >> 172.21.15.158:0/2203977822 conn(0x7fd36806b9e0 msgr2=0x7fd3680acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:42.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.090+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 shutdown_connections 2023-10-16T10:00:42.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.090+0000 7fd36deb9700 1 -- 172.21.15.158:0/2203977822 wait complete. 2023-10-16T10:00:42.103 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T10:00:42.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.224+0000 7f57d0bb1700 1 Processor -- start 2023-10-16T10:00:42.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.224+0000 7f57d0bb1700 1 -- start start 2023-10-16T10:00:42.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57d0bb1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 0x7f57cc15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57d0bb1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc0cf4c0 con 0x7f57cc15b590 2023-10-16T10:00:42.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57d0bb1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc15f710 con 0x7f57cc0cbaf0 2023-10-16T10:00:42.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57d0bb1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc0b49e0 con 0x7f57cc0ce510 2023-10-16T10:00:42.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57ca59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f57cc15b590 0x7f57cc0b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55500/0 (socket says 172.21.15.158:55500) 2023-10-16T10:00:42.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57ca59c700 1 -- 172.21.15.158:0/429564733 learned_addr learned my addr 172.21.15.158:0/429564733 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:42.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c9d9b700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 0x7f57cc15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302996525 0 0) 0x7f57cc0cf4c0 con 0x7f57cc15b590 2023-10-16T10:00:42.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b40035b0 con 0x7f57cc15b590 2023-10-16T10:00:42.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180466854 0 0) 0x7f57b40035b0 con 0x7f57cc15b590 2023-10-16T10:00:42.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 msgr2=0x7f57cc15f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:42.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 0x7f57cc15f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 >> v1:172.21.15.172:6789/0 conn(0x7f57cc0cbaf0 legacy=0x7f57cc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc15f970 con 0x7f57cc15b590 2023-10-16T10:00:42.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f57bc001670 con 0x7f57cc15b590 2023-10-16T10:00:42.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.225+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 >> v1:172.21.15.158:6789/0 conn(0x7f57cc15b590 legacy=0x7f57cc0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57b40043e0 0x7f57b40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 0x7f57b40051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc0b49e0 con 0x7f57b4003a30 2023-10-16T10:00:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b400bac0 con 0x7f57b40043e0 2023-10-16T10:00:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b400bc80 con 0x7f57cc0ce510 2023-10-16T10:00:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57ca59c700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c9d9b700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57b40043e0 0x7f57b40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57cad9d700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 0x7f57b40051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57ca59c700 1 -- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 msgr2=0x7f57b40051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57ca59c700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 0x7f57b40051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57ca59c700 1 -- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57b40043e0 msgr2=0x7f57b40048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57ca59c700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57b40043e0 0x7f57b40048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57ca59c700 1 -- 172.21.15.158:0/429564733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40035b0 con 0x7f57b4003a30 2023-10-16T10:00:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57cad9d700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc0ce510 0x7f57b40051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:00:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57ca59c700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57b4003e10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c240 tx=0x7f57b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0f8a5b0e4e56a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc00cde0 con 0x7f57b4003a30 2023-10-16T10:00:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc007ba0 con 0x7f57b4003a30 2023-10-16T10:00:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.226+0000 7f57c959a700 1 -- 172.21.15.158:0/429564733 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc005b80 con 0x7f57b4003a30 2023-10-16T10:00:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.227+0000 7f57d0bb1700 1 -- 172.21.15.158:0/429564733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 msgr2=0x7f57b4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.227+0000 7f57d0bb1700 1 --2- 172.21.15.158:0/429564733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57b4003e10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c240 tx=0x7f57b4005880 comp rx=0 tx=0).stop 2023-10-16T10:00:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.227+0000 7f57cad9d700 1 -- 172.21.15.158:0/429564733 reap_dead start 2023-10-16T10:00:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.227+0000 7f57d0bb1700 1 -- 172.21.15.158:0/429564733 shutdown_connections 2023-10-16T10:00:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.227+0000 7f57d0bb1700 1 -- 172.21.15.158:0/429564733 >> 172.21.15.158:0/429564733 conn(0x7f57cc06a8d0 msgr2=0x7f57cc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:42.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.227+0000 7f57d0bb1700 1 -- 172.21.15.158:0/429564733 shutdown_connections 2023-10-16T10:00:42.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.227+0000 7f57d0bb1700 1 -- 172.21.15.158:0/429564733 wait complete. 2023-10-16T10:00:42.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.228+0000 7f57d0bb1700 1 Processor -- start 2023-10-16T10:00:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.228+0000 7f57d0bb1700 1 -- start start 2023-10-16T10:00:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57d0bb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57cc1f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57d0bb1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc1f3aa0 0x7f57cc0c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57d0bb1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57cc0ca030 0x7f57cc0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57d0bb1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc15bff0 con 0x7f57b4003a30 2023-10-16T10:00:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57d0bb1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc15c2d0 con 0x7f57cc0ca030 2023-10-16T10:00:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57d0bb1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc15c5b0 con 0x7f57cc1f3aa0 2023-10-16T10:00:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57cc1f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57cc1f1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55336/0 (socket says 172.21.15.158:55336) 2023-10-16T10:00:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 -- 172.21.15.158:0/4136129103 learned_addr learned my addr 172.21.15.158:0/4136129103 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57c9d9b700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc1f3aa0 0x7f57cc0c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57cad9d700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57cc0ca030 0x7f57cc0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 -- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc1f3aa0 msgr2=0x7f57cc0c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc1f3aa0 0x7f57cc0c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 -- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57cc0ca030 msgr2=0x7f57cc0b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57cc0ca030 0x7f57cc0b60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 -- 172.21.15.158:0/4136129103 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc15c890 con 0x7f57b4003a30 2023-10-16T10:00:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.229+0000 7f57ca59c700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57cc1f1e00 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f57bc001930 tx=0x7f57cc0cf430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.230+0000 7f57baffd700 1 -- 172.21.15.158:0/4136129103 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0072d0 con 0x7f57b4003a30 2023-10-16T10:00:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.230+0000 7f57baffd700 1 -- 172.21.15.158:0/4136129103 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc007450 con 0x7f57b4003a30 2023-10-16T10:00:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.230+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc15cb70 con 0x7f57b4003a30 2023-10-16T10:00:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.230+0000 7f57baffd700 1 -- 172.21.15.158:0/4136129103 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc007800 con 0x7f57b4003a30 2023-10-16T10:00:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.230+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc15bc50 con 0x7f57b4003a30 2023-10-16T10:00:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.231+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc0681b0 con 0x7f57b4003a30 2023-10-16T10:00:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.231+0000 7f57baffd700 1 -- 172.21.15.158:0/4136129103 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f57bc019030 con 0x7f57b4003a30 2023-10-16T10:00:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.232+0000 7f57baffd700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57ac06b020 0x7f57ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.232+0000 7f57baffd700 1 -- 172.21.15.158:0/4136129103 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f57bc08b0b0 con 0x7f57b4003a30 2023-10-16T10:00:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.232+0000 7f57c9d9b700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57ac06b020 0x7f57ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.232+0000 7f57c9d9b700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57ac06b020 0x7f57ac06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f57cc15f470 tx=0x7f57c0008040 comp rx=0 tx=0).ready entity=mgr.6895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.236+0000 7f57baffd700 1 -- 172.21.15.158:0/4136129103 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f57bc055f30 con 0x7f57b4003a30 2023-10-16T10:00:42.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.487+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f57cc0b07c0 con 0x7f57b4003a30 2023-10-16T10:00:42.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.675+0000 7f57baffd700 1 -- 172.21.15.158:0/4136129103 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v52) v1 ==== 92+0+0 (secure 0 0 0) 0x7f57bc01ae60 con 0x7f57b4003a30 2023-10-16T10:00:42.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57ac06b020 msgr2=0x7f57ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57ac06b020 0x7f57ac06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f57cc15f470 tx=0x7f57c0008040 comp rx=0 tx=0).stop 2023-10-16T10:00:42.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 msgr2=0x7f57cc1f1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57cc1f1e00 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f57bc001930 tx=0x7f57cc0cf430 comp rx=0 tx=0).stop 2023-10-16T10:00:42.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 shutdown_connections 2023-10-16T10:00:42.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f57ac06b020 0x7f57ac06d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57cc0ca030 0x7f57cc0b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57b4003a30 0x7f57cc1f1e00 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 --2- 172.21.15.158:0/4136129103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57cc1f3aa0 0x7f57cc0c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 >> 172.21.15.158:0/4136129103 conn(0x7f57cc06a8d0 msgr2=0x7f57cc159570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:42.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 shutdown_connections 2023-10-16T10:00:42.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.680+0000 7f57d0bb1700 1 -- 172.21.15.158:0/4136129103 wait complete. 2023-10-16T10:00:42.696 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T10:00:42.824 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:42.822+0000 7fdf9112e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:00:42.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.823+0000 7f10ed82d700 1 Processor -- start 2023-10-16T10:00:42.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.823+0000 7f10ed82d700 1 -- start start 2023-10-16T10:00:42.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10ed82d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10e815dc40 0x7f10e815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10ed82d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e806d9c0 0x7f10e806de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10ed82d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e815f380 con 0x7f10e806f180 2023-10-16T10:00:42.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10ed82d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e806e380 con 0x7f10e815dc40 2023-10-16T10:00:42.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10ed82d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e8071e70 con 0x7f10e806d9c0 2023-10-16T10:00:42.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10e6ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10e815dc40 0x7f10e815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10e67fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e806d9c0 0x7f10e806de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10e67fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e806d9c0 0x7f10e806de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:37974/0 (socket says 172.21.15.158:37974) 2023-10-16T10:00:42.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10e77fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f10e806f180 0x7f10e806f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:59454/0 (socket says 172.21.15.158:59454) 2023-10-16T10:00:42.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10e6ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10e815dc40 0x7f10e815ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:56550/0 (socket says 172.21.15.158:56550) 2023-10-16T10:00:42.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.824+0000 7f10e77fe700 1 -- 172.21.15.158:0/235339160 learned_addr learned my addr 172.21.15.158:0/235339160 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:42.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e6ffd700 1 -- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e806d9c0 msgr2=0x7f10e806de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:42.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e6ffd700 1 --2- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e806d9c0 0x7f10e806de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e6ffd700 1 -- 172.21.15.158:0/235339160 >> v1:172.21.15.172:6789/0 conn(0x7f10e806f180 legacy=0x7f10e806f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e6ffd700 1 -- 172.21.15.158:0/235339160 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e8072150 con 0x7f10e815dc40 2023-10-16T10:00:42.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e67fc700 1 --2- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e806d9c0 0x7f10e806de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:42.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e5ffb700 1 -- 172.21.15.158:0/235339160 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102005782 0 0) 0x7f10e0001490 con 0x7f10e806f180 2023-10-16T10:00:42.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e6ffd700 1 --2- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e815dc40 0x7f10e815ee40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009840 tx=0x7f10d8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41eb716c091a576f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:42.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e5ffb700 1 -- 172.21.15.158:0/235339160 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8011830 con 0x7f10e815dc40 2023-10-16T10:00:42.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e5ffb700 1 -- 172.21.15.158:0/235339160 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d8011990 con 0x7f10e815dc40 2023-10-16T10:00:42.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.825+0000 7f10e5ffb700 1 -- 172.21.15.158:0/235339160 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8011d20 con 0x7f10e815dc40 2023-10-16T10:00:42.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 -- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e815dc40 msgr2=0x7f10e815ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 --2- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e815dc40 0x7f10e815ee40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009840 tx=0x7f10d8009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 -- 172.21.15.158:0/235339160 shutdown_connections 2023-10-16T10:00:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 --2- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e815dc40 0x7f10e815ee40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 --2- 172.21.15.158:0/235339160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e806d9c0 0x7f10e806de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 -- 172.21.15.158:0/235339160 >> 172.21.15.158:0/235339160 conn(0x7f10e80d03b0 msgr2=0x7f10e80d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 -- 172.21.15.158:0/235339160 shutdown_connections 2023-10-16T10:00:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.826+0000 7f10ed82d700 1 -- 172.21.15.158:0/235339160 wait complete. 2023-10-16T10:00:42.833 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:42.827+0000 7f6a63629200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:00:42.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.827+0000 7f10ed82d700 1 Processor -- start 2023-10-16T10:00:42.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.827+0000 7f10ed82d700 1 -- start start 2023-10-16T10:00:42.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10ed82d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e806d9c0 0x7f10e814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10ed82d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e806f180 0x7f10e80b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10ed82d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e815dc40 0x7f10e80b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10ed82d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e814f040 con 0x7f10e806d9c0 2023-10-16T10:00:42.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10ed82d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e814de60 con 0x7f10e806f180 2023-10-16T10:00:42.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10ed82d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e814e430 con 0x7f10e815dc40 2023-10-16T10:00:42.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e806d9c0 0x7f10e814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e67fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e806f180 0x7f10e80b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e806d9c0 0x7f10e814d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55376/0 (socket says 172.21.15.158:55376) 2023-10-16T10:00:42.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e6ffd700 1 -- 172.21.15.158:0/560182406 learned_addr learned my addr 172.21.15.158:0/560182406 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:42.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e77fe700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e815dc40 0x7f10e80b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:42.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e67fc700 1 -- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e815dc40 msgr2=0x7f10e80b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e67fc700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e815dc40 0x7f10e80b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e67fc700 1 -- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e806d9c0 msgr2=0x7f10e814d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:42.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e67fc700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e806d9c0 0x7f10e814d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:42.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e67fc700 1 -- 172.21.15.158:0/560182406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc0094a0 con 0x7f10e806f180 2023-10-16T10:00:42.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e6ffd700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e806d9c0 0x7f10e814d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:42.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.828+0000 7f10e67fc700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e806f180 0x7f10e80b0710 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00e9b0 tx=0x7f10dc00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:42.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.829+0000 7f10cffff700 1 -- 172.21.15.158:0/560182406 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc00ee50 con 0x7f10e806f180 2023-10-16T10:00:42.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.829+0000 7f10cffff700 1 -- 172.21.15.158:0/560182406 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc00c850 con 0x7f10e806f180 2023-10-16T10:00:42.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.829+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d80094a0 con 0x7f10e806f180 2023-10-16T10:00:42.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.829+0000 7f10cffff700 1 -- 172.21.15.158:0/560182406 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc00cc00 con 0x7f10e806f180 2023-10-16T10:00:42.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.829+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e814e9f0 con 0x7f10e806f180 2023-10-16T10:00:42.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.830+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e814f1b0 con 0x7f10e806f180 2023-10-16T10:00:42.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.830+0000 7f10cffff700 1 -- 172.21.15.158:0/560182406 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 52) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f10dc013470 con 0x7f10e806f180 2023-10-16T10:00:42.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.830+0000 7f10cffff700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 0x7f10d406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:42.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.830+0000 7f10e6ffd700 1 -- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 msgr2=0x7f10d406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:42.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.830+0000 7f10cffff700 1 -- 172.21.15.158:0/560182406 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f10dc089a40 con 0x7f10e806f180 2023-10-16T10:00:42.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.830+0000 7f10e6ffd700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 0x7f10d406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:42.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:42.835+0000 7f10cffff700 1 -- 172.21.15.158:0/560182406 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10dc08be60 con 0x7f10e806f180 2023-10-16T10:00:42.890 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:42.889+0000 7fdf9112e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:42.897 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:42.896+0000 7f6a63629200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:42.963 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:42.962+0000 7fdf9112e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:00:42.971 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:42.970+0000 7f6a63629200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:00:43.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.031+0000 7f10e6ffd700 1 -- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 msgr2=0x7f10d406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:43.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.031+0000 7f10e6ffd700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 0x7f10d406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:43.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.036+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f10e80681b0 con 0x7f10e806f180 2023-10-16T10:00:43.040 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:43.039+0000 7fdf9112e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:00:43.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.041+0000 7f10cffff700 1 -- 172.21.15.158:0/560182406 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f10dc0170d0 con 0x7f10e806f180 2023-10-16T10:00:43.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 msgr2=0x7f10d406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:43.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 0x7f10d406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e806f180 msgr2=0x7f10e80b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e806f180 0x7f10e80b0710 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00e9b0 tx=0x7f10dc00ecc0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 shutdown_connections 2023-10-16T10:00:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f10d406b070 0x7f10d406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10e815dc40 0x7f10e80b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e806d9c0 0x7f10e814d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 --2- 172.21.15.158:0/560182406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e806f180 0x7f10e80b0710 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 >> 172.21.15.158:0/560182406 conn(0x7f10e80d03b0 msgr2=0x7f10e80b1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 shutdown_connections 2023-10-16T10:00:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.043+0000 7f10ed82d700 1 -- 172.21.15.158:0/560182406 wait complete. 2023-10-16T10:00:43.051 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:43.050+0000 7f6a63629200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:00:43.057 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T10:00:43.181 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:43.179+0000 7fdf9112e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:00:43.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.182+0000 7fc3c21ce700 1 Processor -- start 2023-10-16T10:00:43.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.182+0000 7fc3c21ce700 1 -- start start 2023-10-16T10:00:43.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3c21ce700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3bc154b70 0x7fc3bc15eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3c21ce700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc3bc15df00 0x7fc3bc15e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3c21ce700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc15e890 con 0x7fc3bc0cd6a0 2023-10-16T10:00:43.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3c21ce700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3bc06d600 con 0x7fc3bc154b70 2023-10-16T10:00:43.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3c21ce700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3bc0cc000 con 0x7fc3bc15df00 2023-10-16T10:00:43.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3bc154b70 0x7fc3bc15eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3bc154b70 0x7fc3bc15eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55402/0 (socket says 172.21.15.158:55402) 2023-10-16T10:00:43.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bb7fe700 1 -- 172.21.15.158:0/259323095 learned_addr learned my addr 172.21.15.158:0/259323095 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:43.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3baffd700 1 --1- 172.21.15.158:0/259323095 >> v1:172.21.15.158:6789/0 conn(0x7fc3bc0cd6a0 0x7fc3bc0cb8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55580/0 (socket says 172.21.15.158:55580) 2023-10-16T10:00:43.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bbfff700 1 --2- 172.21.15.158:0/259323095 >> v2:172.21.15.172:3300/0 conn(0x7fc3bc15df00 0x7fc3bc15e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bb7fe700 1 -- 172.21.15.158:0/259323095 >> v2:172.21.15.172:3300/0 conn(0x7fc3bc15df00 msgr2=0x7fc3bc15e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:43.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bb7fe700 1 --2- 172.21.15.158:0/259323095 >> v2:172.21.15.172:3300/0 conn(0x7fc3bc15df00 0x7fc3bc15e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bb7fe700 1 -- 172.21.15.158:0/259323095 >> v1:172.21.15.158:6789/0 conn(0x7fc3bc0cd6a0 legacy=0x7fc3bc0cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3ba7fc700 1 -- 172.21.15.158:0/259323095 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204330866 0 0) 0x7fc3b0001110 con 0x7fc3bc0cd6a0 2023-10-16T10:00:43.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.183+0000 7fc3bb7fe700 1 -- 172.21.15.158:0/259323095 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc0cc270 con 0x7fc3bc154b70 2023-10-16T10:00:43.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3bb7fe700 1 --2- 172.21.15.158:0/259323095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc15eaa0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc1584e0 tx=0x7fc3ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f38293c2394413 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:43.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3ba7fc700 1 -- 172.21.15.158:0/259323095 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac011830 con 0x7fc3bc154b70 2023-10-16T10:00:43.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3ba7fc700 1 -- 172.21.15.158:0/259323095 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac011990 con 0x7fc3bc154b70 2023-10-16T10:00:43.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3ba7fc700 1 -- 172.21.15.158:0/259323095 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac011d20 con 0x7fc3bc154b70 2023-10-16T10:00:43.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3c21ce700 1 -- 172.21.15.158:0/259323095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 msgr2=0x7fc3bc15eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/259323095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc15eaa0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc1584e0 tx=0x7fc3ac009bd0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3c21ce700 1 -- 172.21.15.158:0/259323095 shutdown_connections 2023-10-16T10:00:43.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/259323095 >> v2:172.21.15.172:3300/0 conn(0x7fc3bc15df00 0x7fc3bc15e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/259323095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc15eaa0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.184+0000 7fc3c21ce700 1 -- 172.21.15.158:0/259323095 >> 172.21.15.158:0/259323095 conn(0x7fc3bc0b15c0 msgr2=0x7fc3bc0b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:43.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.185+0000 7fc3c21ce700 1 -- 172.21.15.158:0/259323095 shutdown_connections 2023-10-16T10:00:43.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.185+0000 7fc3c21ce700 1 -- 172.21.15.158:0/259323095 wait complete. 2023-10-16T10:00:43.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.186+0000 7fc3c21ce700 1 Processor -- start 2023-10-16T10:00:43.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.186+0000 7fc3c21ce700 1 -- start start 2023-10-16T10:00:43.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3c21ce700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3bc0cd6a0 0x7fc3bc14d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3c21ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc0c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3c21ce700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3bc15df00 0x7fc3bc0c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3c21ce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc15c3f0 con 0x7fc3bc154b70 2023-10-16T10:00:43.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3c21ce700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc155ee0 con 0x7fc3bc15df00 2023-10-16T10:00:43.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3c21ce700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc1561c0 con 0x7fc3bc0cd6a0 2023-10-16T10:00:43.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc0c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3baffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc0c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55410/0 (socket says 172.21.15.158:55410) 2023-10-16T10:00:43.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3baffd700 1 -- 172.21.15.158:0/3040854845 learned_addr learned my addr 172.21.15.158:0/3040854845 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:43.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3bbfff700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3bc15df00 0x7fc3bc0c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.187+0000 7fc3bb7fe700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3bc0cd6a0 0x7fc3bc14d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3baffd700 1 -- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3bc0cd6a0 msgr2=0x7fc3bc14d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3baffd700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3bc0cd6a0 0x7fc3bc14d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3baffd700 1 -- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3bc15df00 msgr2=0x7fc3bc0c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3baffd700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3bc15df00 0x7fc3bc0c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3baffd700 1 -- 172.21.15.158:0/3040854845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac009550 con 0x7fc3bc154b70 2023-10-16T10:00:43.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3baffd700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc0c59f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c550 tx=0x7fc3b000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:43.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3a3fff700 1 -- 172.21.15.158:0/3040854845 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0004c80 con 0x7fc3bc154b70 2023-10-16T10:00:43.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3bc1564a0 con 0x7fc3bc154b70 2023-10-16T10:00:43.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3a3fff700 1 -- 172.21.15.158:0/3040854845 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b0004e00 con 0x7fc3bc154b70 2023-10-16T10:00:43.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3a3fff700 1 -- 172.21.15.158:0/3040854845 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0007310 con 0x7fc3bc154b70 2023-10-16T10:00:43.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.188+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc14cc80 con 0x7fc3bc154b70 2023-10-16T10:00:43.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.189+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3bc0681b0 con 0x7fc3bc154b70 2023-10-16T10:00:43.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.189+0000 7fc3a3fff700 1 -- 172.21.15.158:0/3040854845 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 52) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc3b0007470 con 0x7fc3bc154b70 2023-10-16T10:00:43.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.190+0000 7fc3a3fff700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 0x7fc3a806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.190+0000 7fc3bb7fe700 1 -- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 msgr2=0x7fc3a806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:43.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.190+0000 7fc3a3fff700 1 -- 172.21.15.158:0/3040854845 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fc3b0089550 con 0x7fc3bc154b70 2023-10-16T10:00:43.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.190+0000 7fc3bb7fe700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 0x7fc3a806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:43.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.194+0000 7fc3a3fff700 1 -- 172.21.15.158:0/3040854845 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3b0054210 con 0x7fc3bc154b70 2023-10-16T10:00:43.201 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:43.200+0000 7f6a63629200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:00:43.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.391+0000 7fc3bb7fe700 1 -- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 msgr2=0x7fc3a806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:43.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.391+0000 7fc3bb7fe700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 0x7fc3a806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.396+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fc3bc15e890 con 0x7fc3bc154b70 2023-10-16T10:00:43.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.399+0000 7fc3a3fff700 1 -- 172.21.15.158:0/3040854845 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc3bc15e890 con 0x7fc3bc154b70 2023-10-16T10:00:43.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 msgr2=0x7fc3a806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:43.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 0x7fc3a806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 msgr2=0x7fc3bc0c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc0c59f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c550 tx=0x7fc3b000c860 comp rx=0 tx=0).stop 2023-10-16T10:00:43.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 shutdown_connections 2023-10-16T10:00:43.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7fc3a806b070 0x7fc3a806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3bc15df00 0x7fc3bc0c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3bc0cd6a0 0x7fc3bc14d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 --2- 172.21.15.158:0/3040854845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3bc154b70 0x7fc3bc0c59f0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 >> 172.21.15.158:0/3040854845 conn(0x7fc3bc0b15c0 msgr2=0x7fc3bc0cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:43.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 shutdown_connections 2023-10-16T10:00:43.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.402+0000 7fc3c21ce700 1 -- 172.21.15.158:0/3040854845 wait complete. 2023-10-16T10:00:43.416 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2023-10-16T10:00:43.505 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:43.504+0000 7fdf9112e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:00:43.541 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:43.540+0000 7f6a63629200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:00:43.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.541+0000 7f9a32b40700 1 Processor -- start 2023-10-16T10:00:43.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.541+0000 7f9a32b40700 1 -- start start 2023-10-16T10:00:43.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a32b40700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a2c0aefa0 0x7f9a2c161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a32b40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c071cb0 0x7f9a2c06ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a32b40700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c0b3dc0 con 0x7f9a2c15eae0 2023-10-16T10:00:43.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a32b40700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2c15ddf0 con 0x7f9a2c0aefa0 2023-10-16T10:00:43.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a32b40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c1622f0 con 0x7f9a2c071cb0 2023-10-16T10:00:43.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a308dc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a2c0aefa0 0x7f9a2c161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a310dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c071cb0 0x7f9a2c06ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a2bfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9a2c15eae0 0x7f9a2c0715a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:59526/0 (socket says 172.21.15.158:59526) 2023-10-16T10:00:43.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a308dc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a2c0aefa0 0x7f9a2c161db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:56626/0 (socket says 172.21.15.158:56626) 2023-10-16T10:00:43.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a310dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c071cb0 0x7f9a2c06ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:38052/0 (socket says 172.21.15.158:38052) 2023-10-16T10:00:43.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a2bfff700 1 -- 172.21.15.158:0/2816854945 learned_addr learned my addr 172.21.15.158:0/2816854945 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:43.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a308dc700 1 -- 172.21.15.158:0/2816854945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c071cb0 msgr2=0x7f9a2c06ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:43.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a308dc700 1 --2- 172.21.15.158:0/2816854945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c071cb0 0x7f9a2c06ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.542+0000 7f9a308dc700 1 -- 172.21.15.158:0/2816854945 >> v1:172.21.15.172:6789/0 conn(0x7f9a2c15eae0 legacy=0x7f9a2c0715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a308dc700 1 -- 172.21.15.158:0/2816854945 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c0b3ae0 con 0x7f9a2c0aefa0 2023-10-16T10:00:43.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a2b7fe700 1 -- 172.21.15.158:0/2816854945 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220830972 0 0) 0x7f9a2c0b3dc0 con 0x7f9a2c15eae0 2023-10-16T10:00:43.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a308dc700 1 --2- 172.21.15.158:0/2816854945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c0aefa0 0x7f9a2c161db0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009840 tx=0x7f9a20009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce8f48268759f2ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:43.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a2b7fe700 1 -- 172.21.15.158:0/2816854945 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20011830 con 0x7f9a2c0aefa0 2023-10-16T10:00:43.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a2b7fe700 1 -- 172.21.15.158:0/2816854945 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a20011990 con 0x7f9a2c0aefa0 2023-10-16T10:00:43.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a2b7fe700 1 -- 172.21.15.158:0/2816854945 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20011d20 con 0x7f9a2c0aefa0 2023-10-16T10:00:43.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a32b40700 1 -- 172.21.15.158:0/2816854945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c0aefa0 msgr2=0x7f9a2c161db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.543+0000 7f9a32b40700 1 --2- 172.21.15.158:0/2816854945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c0aefa0 0x7f9a2c161db0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009840 tx=0x7f9a20009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:43.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.544+0000 7f9a32b40700 1 -- 172.21.15.158:0/2816854945 shutdown_connections 2023-10-16T10:00:43.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.544+0000 7f9a32b40700 1 --2- 172.21.15.158:0/2816854945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c071cb0 0x7f9a2c06ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.544+0000 7f9a32b40700 1 --2- 172.21.15.158:0/2816854945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c0aefa0 0x7f9a2c161db0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.544+0000 7f9a32b40700 1 -- 172.21.15.158:0/2816854945 >> 172.21.15.158:0/2816854945 conn(0x7f9a2c0b1bd0 msgr2=0x7f9a2c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:43.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.544+0000 7f9a32b40700 1 -- 172.21.15.158:0/2816854945 shutdown_connections 2023-10-16T10:00:43.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.544+0000 7f9a32b40700 1 -- 172.21.15.158:0/2816854945 wait complete. 2023-10-16T10:00:43.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.545+0000 7f9a32b40700 1 Processor -- start 2023-10-16T10:00:43.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.545+0000 7f9a32b40700 1 -- start start 2023-10-16T10:00:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a32b40700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c071cb0 0x7f9a2c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a32b40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c0aefa0 0x7f9a2c15ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a32b40700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a2c15eae0 0x7f9a2c0ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a32b40700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c15bbb0 con 0x7f9a2c15eae0 2023-10-16T10:00:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a32b40700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c15bde0 con 0x7f9a2c071cb0 2023-10-16T10:00:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a32b40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c15c0c0 con 0x7f9a2c0aefa0 2023-10-16T10:00:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a2c15eae0 0x7f9a2c0ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a2c15eae0 0x7f9a2c0ad1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55456/0 (socket says 172.21.15.158:55456) 2023-10-16T10:00:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 -- 172.21.15.158:0/4207602881 learned_addr learned my addr 172.21.15.158:0/4207602881 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c071cb0 0x7f9a2c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a2bfff700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c0aefa0 0x7f9a2c15ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c0aefa0 msgr2=0x7f9a2c15ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a2c0aefa0 0x7f9a2c15ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c071cb0 msgr2=0x7f9a2c149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:43.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c071cb0 0x7f9a2c149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:43.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 -- 172.21.15.158:0/4207602881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200094a0 con 0x7f9a2c15eae0 2023-10-16T10:00:43.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a2c071cb0 0x7f9a2c149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:00:43.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a310dd700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a2c15eae0 0x7f9a2c0ad1e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400efc0 tx=0x7f9a24009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:43.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24009b20 con 0x7f9a2c15eae0 2023-10-16T10:00:43.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2400f040 con 0x7f9a2c15eae0 2023-10-16T10:00:43.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c15c3a0 con 0x7f9a2c15eae0 2023-10-16T10:00:43.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.546+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2400cc90 con 0x7f9a2c15eae0 2023-10-16T10:00:43.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.547+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c15c960 con 0x7f9a2c15eae0 2023-10-16T10:00:43.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.547+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c0681b0 con 0x7f9a2c15eae0 2023-10-16T10:00:43.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.548+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 52) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9a24014480 con 0x7f9a2c15eae0 2023-10-16T10:00:43.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.548+0000 7f9a297fa700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 0x7f9a1006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:43.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.548+0000 7f9a308dc700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 msgr2=0x7f9a1006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:43.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.548+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9a2408a610 con 0x7f9a2c15eae0 2023-10-16T10:00:43.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.548+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 0x7f9a1006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:00:43.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.553+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a240551c0 con 0x7f9a2c15eae0 2023-10-16T10:00:43.629 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:43.628+0000 7fdf9112e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:00:43.664 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:43.664+0000 7f6a63629200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:00:43.705 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:43.703+0000 7fdf9112e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:00:43.740 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:43.739+0000 7f6a63629200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:00:43.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.749+0000 7f9a308dc700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 msgr2=0x7f9a1006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:43.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.749+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 0x7f9a1006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:00:43.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:43.780+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 --> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a2c15c230 con 0x7f9a1006b070 2023-10-16T10:00:43.785 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:43.784+0000 7fdf9112e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:00:43.822 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:43.821+0000 7f6a63629200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:00:44.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:44.150+0000 7f9a308dc700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 msgr2=0x7f9a1006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:44.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:44.150+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 0x7f9a1006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T10:00:44.153 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:44.152+0000 7fdf9112e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:00:44.192 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:44.191+0000 7f6a63629200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:00:44.722 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:44.721+0000 7fdf9112e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:00:44.761 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:44.760+0000 7f6a63629200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:00:44.792 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:44.791+0000 7fdf9112e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:00:44.830 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:44.829+0000 7f6a63629200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:00:44.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:44.951+0000 7f9a308dc700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 msgr2=0x7f9a1006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:44.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:44.951+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 0x7f9a1006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T10:00:45.417 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:45.415+0000 7fdf9112e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:45.460 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:45.458+0000 7f6a63629200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:45.493 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:45.492+0000 7fdf9112e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:00:45.537 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:45.536+0000 7f6a63629200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:00:45.636 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:45.635+0000 7fdf9112e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:45.682 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:45.681+0000 7f6a63629200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:00:45.862 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:45.861+0000 7fdf9112e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:00:45.910 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:45.909+0000 7f6a63629200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:00:45.936 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:45.934+0000 7fdf9112e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:00:45.984 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:45.983+0000 7f6a63629200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:00:46.213 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:46.212+0000 7fdf9112e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:00:46.259 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:46.258+0000 7f6a63629200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:00:46.433 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:46.431+0000 7fdf9112e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:00:46.483 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:46.482+0000 7f6a63629200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:00:46.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:46.552+0000 7f9a308dc700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 msgr2=0x7f9a1006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1430981267 2023-10-16T10:00:46.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:46.552+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 0x7f9a1006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T10:00:46.582 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:46.580+0000 7fdf9112e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:00:46.634 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:46.633+0000 7f6a63629200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:00:46.694 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:46.693+0000 7fdf9112e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:00:46.747 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:46.746+0000 7f6a63629200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:00:46.848 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:46.847+0000 7fdf9112e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:00:46.904 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:46.902+0000 7f6a63629200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:00:47.109 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:47.108+0000 7fdf9112e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:47.166 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:47.165+0000 7f6a63629200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:00:47.268 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:47.267+0000 7fdf9112e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:00:47.329 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:47.328+0000 7f6a63629200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:00:47.786 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:00:47.785+0000 7fdf9112e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:00:47.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:47.856+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 53) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9a24088840 con 0x7f9a2c15eae0 2023-10-16T10:00:47.861 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:47.860+0000 7f6a63629200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:00:47.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:47.880+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 54) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f9a24012dc0 con 0x7f9a2c15eae0 2023-10-16T10:00:47.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:47.880+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 msgr2=0x7f9a1006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:00:47.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:47.880+0000 7f9a297fa700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1430981267,v1:172.21.15.158:6833/1430981267] conn(0x7f9a1006b070 0x7f9a1006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:48.035 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d6604700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.035 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d6604700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.035 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d6604700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.035 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d6604700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.036 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d6604700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.036 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d960a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.036 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d960a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.036 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d960a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.036 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d960a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.037 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:00:48.033+0000 7f69d960a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:00:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.885+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 55) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9a24055e60 con 0x7f9a2c15eae0 2023-10-16T10:00:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.886+0000 7f9a297fa700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f9a100279c0 0x7f9a10027da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.886+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 --> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a2c15c230 con 0x7f9a100279c0 2023-10-16T10:00:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.886+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f9a100279c0 0x7f9a10027da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.889+0000 7f9a308dc700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f9a100279c0 0x7f9a10027da0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a200090b0 tx=0x7f9a20008030 comp rx=0 tx=0).ready entity=mgr.7920 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:48.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.953+0000 7f9a297fa700 1 -- 172.21.15.158:0/4207602881 <== mgr.7920 v2:172.21.15.158:6832/1516256968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f9a2c15c230 con 0x7f9a100279c0 2023-10-16T10:00:48.954 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /bad-path 2023-10-16T10:00:48.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f9a100279c0 msgr2=0x7f9a10027da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:48.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f9a100279c0 0x7f9a10027da0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a200090b0 tx=0x7f9a20008030 comp rx=0 tx=0).stop 2023-10-16T10:00:48.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a2c15eae0 msgr2=0x7f9a2c0ad1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:48.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 --2- 172.21.15.158:0/4207602881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a2c15eae0 0x7f9a2c0ad1e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400efc0 tx=0x7f9a24009f60 comp rx=0 tx=0).stop 2023-10-16T10:00:48.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a310dd700 1 -- 172.21.15.158:0/4207602881 reap_dead start 2023-10-16T10:00:48.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 shutdown_connections 2023-10-16T10:00:48.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 >> 172.21.15.158:0/4207602881 conn(0x7f9a2c0b1bd0 msgr2=0x7f9a2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:48.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 shutdown_connections 2023-10-16T10:00:48.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:48.955+0000 7f9a32b40700 1 -- 172.21.15.158:0/4207602881 wait complete. 2023-10-16T10:00:48.969 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-10-16T10:00:48.970 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T10:00:49.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.093+0000 7f38a078e700 1 Processor -- start 2023-10-16T10:00:49.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.093+0000 7f38a078e700 1 -- start start 2023-10-16T10:00:49.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f38a078e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f389806afe0 0x7f38980caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:49.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f38a078e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980cbd30 con 0x7f38980cc590 2023-10-16T10:00:49.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f38a078e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980cc1c0 con 0x7f389815da40 2023-10-16T10:00:49.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f38a078e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389815de20 con 0x7f389806afe0 2023-10-16T10:00:49.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389dd29700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f389806afe0 0x7f38980caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:49.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389ed2b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f38980cc590 0x7f38980afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55804/0 (socket says 172.21.15.158:55804) 2023-10-16T10:00:49.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389ed2b700 1 -- 172.21.15.158:0/2155893102 learned_addr learned my addr 172.21.15.158:0/2155893102 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:00:49.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389dd29700 1 -- 172.21.15.158:0/2155893102 >> v1:172.21.15.172:6789/0 conn(0x7f389815da40 legacy=0x7f389806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:00:49.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389dd29700 1 -- 172.21.15.158:0/2155893102 >> v1:172.21.15.158:6789/0 conn(0x7f38980cc590 legacy=0x7f38980afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:49.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389d528700 1 -- 172.21.15.158:0/2155893102 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023443875 0 0) 0x7f38980cbd30 con 0x7f38980cc590 2023-10-16T10:00:49.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389dd29700 1 -- 172.21.15.158:0/2155893102 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980ce260 con 0x7f389806afe0 2023-10-16T10:00:49.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389dd29700 1 --2- 172.21.15.158:0/2155893102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f389806afe0 0x7f38980caa20 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3888009840 tx=0x7f3888009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e632ef4c9b3b51d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:49.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389d528700 1 -- 172.21.15.158:0/2155893102 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888004230 con 0x7f389806afe0 2023-10-16T10:00:49.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.094+0000 7f389d528700 1 -- 172.21.15.158:0/2155893102 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3888004390 con 0x7f389806afe0 2023-10-16T10:00:49.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f389d528700 1 -- 172.21.15.158:0/2155893102 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38880036a0 con 0x7f389806afe0 2023-10-16T10:00:49.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f38a078e700 1 -- 172.21.15.158:0/2155893102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f389806afe0 msgr2=0x7f38980caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:49.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f38a078e700 1 --2- 172.21.15.158:0/2155893102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f389806afe0 0x7f38980caa20 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3888009840 tx=0x7f3888009b50 comp rx=0 tx=0).stop 2023-10-16T10:00:49.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f38a078e700 1 -- 172.21.15.158:0/2155893102 shutdown_connections 2023-10-16T10:00:49.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f38a078e700 1 --2- 172.21.15.158:0/2155893102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f389806afe0 0x7f38980caa20 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f38a078e700 1 -- 172.21.15.158:0/2155893102 >> 172.21.15.158:0/2155893102 conn(0x7f3898099a10 msgr2=0x7f38980d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:49.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f38a078e700 1 -- 172.21.15.158:0/2155893102 shutdown_connections 2023-10-16T10:00:49.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.095+0000 7f38a078e700 1 -- 172.21.15.158:0/2155893102 wait complete. 2023-10-16T10:00:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.096+0000 7f38a078e700 1 Processor -- start 2023-10-16T10:00:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.096+0000 7f38a078e700 1 -- start start 2023-10-16T10:00:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f38a078e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f389806afe0 0x7f3898159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f38a078e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38980cc590 0x7f38980c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f38a078e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f389815da40 0x7f38980c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f38a078e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389815ab30 con 0x7f38980cc590 2023-10-16T10:00:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f38a078e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980c5260 con 0x7f389815da40 2023-10-16T10:00:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f38a078e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3898159fb0 con 0x7f389806afe0 2023-10-16T10:00:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38980cc590 0x7f38980c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:49.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38980cc590 0x7f38980c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55632/0 (socket says 172.21.15.158:55632) 2023-10-16T10:00:49.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 -- 172.21.15.158:0/3624831425 learned_addr learned my addr 172.21.15.158:0/3624831425 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:00:49.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389ed2b700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f389815da40 0x7f38980c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:49.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389e52a700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f389806afe0 0x7f3898159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:49.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 -- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f389806afe0 msgr2=0x7f3898159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:49.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f389806afe0 0x7f3898159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 -- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f389815da40 msgr2=0x7f38980c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:49.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f389815da40 0x7f38980c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.097+0000 7f389dd29700 1 -- 172.21.15.158:0/3624831425 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38880094a0 con 0x7f38980cc590 2023-10-16T10:00:49.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.098+0000 7f389dd29700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38980cc590 0x7f38980c94b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f38880090b0 tx=0x7f3888004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:49.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.098+0000 7f388effd700 1 -- 172.21.15.158:0/3624831425 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888003ea0 con 0x7f38980cc590 2023-10-16T10:00:49.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.098+0000 7f388effd700 1 -- 172.21.15.158:0/3624831425 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388801e070 con 0x7f38980cc590 2023-10-16T10:00:49.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.098+0000 7f38a078e700 1 -- 172.21.15.158:0/3624831425 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389815a290 con 0x7f38980cc590 2023-10-16T10:00:49.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.098+0000 7f388effd700 1 -- 172.21.15.158:0/3624831425 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888011850 con 0x7f38980cc590 2023-10-16T10:00:49.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.098+0000 7f38a078e700 1 -- 172.21.15.158:0/3624831425 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389815a850 con 0x7f38980cc590 2023-10-16T10:00:49.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.099+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38740052f0 con 0x7f38980cc590 2023-10-16T10:00:49.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.099+0000 7f388effd700 1 -- 172.21.15.158:0/3624831425 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 55) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f38880119b0 con 0x7f38980cc590 2023-10-16T10:00:49.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.099+0000 7f388effd700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f387c06add0 0x7f387c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:00:49.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.100+0000 7f388effd700 1 -- 172.21.15.158:0/3624831425 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f388808a900 con 0x7f38980cc590 2023-10-16T10:00:49.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.100+0000 7f389e52a700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f387c06add0 0x7f387c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:00:49.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.100+0000 7f389e52a700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f387c06add0 0x7f387c06d290 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3894009960 tx=0x7f38940090e0 comp rx=0 tx=0).ready entity=mgr.7920 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:00:49.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.104+0000 7f388effd700 1 -- 172.21.15.158:0/3624831425 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38880558b0 con 0x7f38980cc590 2023-10-16T10:00:49.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.320+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 --> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3874000bc0 con 0x7f387c06add0 2023-10-16T10:00:49.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.330+0000 7f388effd700 1 -- 172.21.15.158:0/3624831425 <== mgr.7920 v2:172.21.15.158:6832/1516256968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7f3874000bc0 con 0x7f387c06add0 2023-10-16T10:00:49.331 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:00:49.331 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T10:00:48", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T10:00:49.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f387c06add0 msgr2=0x7f387c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:49.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f387c06add0 0x7f387c06d290 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3894009960 tx=0x7f38940090e0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38980cc590 msgr2=0x7f38980c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:00:49.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38980cc590 0x7f38980c94b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f38880090b0 tx=0x7f3888004750 comp rx=0 tx=0).stop 2023-10-16T10:00:49.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 shutdown_connections 2023-10-16T10:00:49.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f387c06add0 0x7f387c06d290 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f389815da40 0x7f38980c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f389806afe0 0x7f3898159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 --2- 172.21.15.158:0/3624831425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38980cc590 0x7f38980c94b0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:00:49.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 >> 172.21.15.158:0/3624831425 conn(0x7f3898099a10 msgr2=0x7f389815c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:00:49.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 shutdown_connections 2023-10-16T10:00:49.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:00:49.332+0000 7f388cff9700 1 -- 172.21.15.158:0/3624831425 wait complete. 2023-10-16T10:00:49.345 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T10:00:48", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T10:00:58.699 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:00:58.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:01:00.964 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:00.963+0000 7f69c8de9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:00.965 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:00.963+0000 7f69c8de9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:00.965 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:00.963+0000 7f69c8de9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:00.965 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:00.963+0000 7f69c8de9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:00.965 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:00.963+0000 7f69c8de9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:28.766 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:01:28.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:01:49.347 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T10:01:49.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.483+0000 7f6585aa8700 1 Processor -- start 2023-10-16T10:01:49.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.484+0000 7f6585aa8700 1 -- start start 2023-10-16T10:01:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.484+0000 7f6585aa8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 0x7f658015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.484+0000 7f6585aa8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658015f380 con 0x7f658006d9c0 2023-10-16T10:01:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.484+0000 7f6585aa8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658006ffa0 con 0x7f658006de40 2023-10-16T10:01:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.484+0000 7f6585aa8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658006e2d0 con 0x7f658015dc40 2023-10-16T10:01:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.484+0000 7f657effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f658006d9c0 0x7f658006f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55846/0 (socket says 172.21.15.158:55846) 2023-10-16T10:01:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.484+0000 7f657effd700 1 -- 172.21.15.158:0/1716931340 learned_addr learned my addr 172.21.15.158:0/1716931340 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657f7fe700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 0x7f658015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28919455 0 0) 0x7f6570001110 con 0x7f658006d9c0 2023-10-16T10:01:49.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65680035b0 con 0x7f658006d9c0 2023-10-16T10:01:49.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854048167 0 0) 0x7f65680035b0 con 0x7f658006d9c0 2023-10-16T10:01:49.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 msgr2=0x7f658015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:49.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 0x7f658015ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 >> v1:172.21.15.172:6789/0 conn(0x7f658006de40 legacy=0x7f658006f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6580071e70 con 0x7f658006d9c0 2023-10-16T10:01:49.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f6570001800 con 0x7f658006d9c0 2023-10-16T10:01:49.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 >> v1:172.21.15.158:6789/0 conn(0x7f658006d9c0 legacy=0x7f658006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568003a30 0x7f6568003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6568004850 0x7f6568004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 0x7f6568005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658006e2d0 con 0x7f6568004850 2023-10-16T10:01:49.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656800bac0 con 0x7f6568003a30 2023-10-16T10:01:49.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656800bc80 con 0x7f658015dc40 2023-10-16T10:01:49.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657effd700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6568004850 0x7f6568004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657f7fe700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568003a30 0x7f6568003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657ffff700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 0x7f6568005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657effd700 1 -- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 msgr2=0x7f6568005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657effd700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f658015dc40 0x7f6568005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657effd700 1 -- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568003a30 msgr2=0x7f6568003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.485+0000 7f657effd700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568003a30 0x7f6568003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f657effd700 1 -- 172.21.15.158:0/1716931340 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680035b0 con 0x7f6568004850 2023-10-16T10:01:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f657effd700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6568004850 0x7f6568004cd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f657000c210 tx=0x7f6568005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90044c69d498813 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570007db0 con 0x7f6568004850 2023-10-16T10:01:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657000cdd0 con 0x7f6568004850 2023-10-16T10:01:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f657e7fc700 1 -- 172.21.15.158:0/1716931340 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570005ba0 con 0x7f6568004850 2023-10-16T10:01:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f6585aa8700 1 -- 172.21.15.158:0/1716931340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6568004850 msgr2=0x7f6568004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f6585aa8700 1 --2- 172.21.15.158:0/1716931340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6568004850 0x7f6568004cd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f657000c210 tx=0x7f6568005880 comp rx=0 tx=0).stop 2023-10-16T10:01:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f657ffff700 1 -- 172.21.15.158:0/1716931340 reap_dead start 2023-10-16T10:01:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f6585aa8700 1 -- 172.21.15.158:0/1716931340 shutdown_connections 2023-10-16T10:01:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f6585aa8700 1 -- 172.21.15.158:0/1716931340 >> 172.21.15.158:0/1716931340 conn(0x7f65800d03b0 msgr2=0x7f65800d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f6585aa8700 1 -- 172.21.15.158:0/1716931340 shutdown_connections 2023-10-16T10:01:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.486+0000 7f6585aa8700 1 -- 172.21.15.158:0/1716931340 wait complete. 2023-10-16T10:01:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.488+0000 7f6585aa8700 1 Processor -- start 2023-10-16T10:01:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.488+0000 7f6585aa8700 1 -- start start 2023-10-16T10:01:49.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568004850 0x7f6580150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6580150bf0 0x7f65800c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65800c8b40 0x7f65800cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65800acca0 con 0x7f65800c8b40 2023-10-16T10:01:49.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6580150940 con 0x7f6568004850 2023-10-16T10:01:49.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65800c8fe0 con 0x7f6580150bf0 2023-10-16T10:01:49.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65800c8b40 0x7f65800cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65800c8b40 0x7f65800cf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55682/0 (socket says 172.21.15.158:55682) 2023-10-16T10:01:49.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 -- 172.21.15.158:0/2099668261 learned_addr learned my addr 172.21.15.158:0/2099668261 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:49.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657effd700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6580150bf0 0x7f65800c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 -- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6580150bf0 msgr2=0x7f65800c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657f7fe700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568004850 0x7f6580150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6580150bf0 0x7f65800c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 -- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568004850 msgr2=0x7f6580150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568004850 0x7f6580150400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 -- 172.21.15.158:0/2099668261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65800c92c0 con 0x7f65800c8b40 2023-10-16T10:01:49.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f657ffff700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65800c8b40 0x7f65800cf540 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6578000f80 tx=0x7f657800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:49.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6567fff700 1 -- 172.21.15.158:0/2099668261 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578003c10 con 0x7f65800c8b40 2023-10-16T10:01:49.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6567fff700 1 -- 172.21.15.158:0/2099668261 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6578003d90 con 0x7f65800c8b40 2023-10-16T10:01:49.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6567fff700 1 -- 172.21.15.158:0/2099668261 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578005310 con 0x7f65800c8b40 2023-10-16T10:01:49.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65800c95a0 con 0x7f65800c8b40 2023-10-16T10:01:49.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.489+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65800cfa80 con 0x7f65800c8b40 2023-10-16T10:01:49.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.490+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65800ace50 con 0x7f65800c8b40 2023-10-16T10:01:49.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.491+0000 7f6567fff700 1 -- 172.21.15.158:0/2099668261 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 58) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6578005470 con 0x7f65800c8b40 2023-10-16T10:01:49.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.491+0000 7f6567fff700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f656c06b190 0x7f656c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.491+0000 7f657f7fe700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f656c06b190 0x7f656c06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.491+0000 7f6567fff700 1 -- 172.21.15.158:0/2099668261 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f6578089f60 con 0x7f65800c8b40 2023-10-16T10:01:49.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.492+0000 7f657f7fe700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f656c06b190 0x7f656c06d650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6574009aa0 tx=0x7f6574008040 comp rx=0 tx=0).ready entity=mgr.7920 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:49.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.496+0000 7f6567fff700 1 -- 172.21.15.158:0/2099668261 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65800ace50 con 0x7f65800c8b40 2023-10-16T10:01:49.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.715+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 --> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6580072430 con 0x7f656c06b190 2023-10-16T10:01:49.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.724+0000 7f6567fff700 1 -- 172.21.15.158:0/2099668261 <== mgr.7920 v2:172.21.15.158:6832/1516256968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7f65740137d0 con 0x7f656c06b190 2023-10-16T10:01:49.726 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:49.726 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T10:00:48", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T10:01:49.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.726+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f656c06b190 msgr2=0x7f656c06d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f656c06b190 0x7f656c06d650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6574009aa0 tx=0x7f6574008040 comp rx=0 tx=0).stop 2023-10-16T10:01:49.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65800c8b40 msgr2=0x7f65800cf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65800c8b40 0x7f65800cf540 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6578000f80 tx=0x7f657800ee80 comp rx=0 tx=0).stop 2023-10-16T10:01:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 shutdown_connections 2023-10-16T10:01:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f656c06b190 0x7f656c06d650 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65800c8b40 0x7f65800cf540 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6568004850 0x7f6580150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 --2- 172.21.15.158:0/2099668261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6580150bf0 0x7f65800c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 >> 172.21.15.158:0/2099668261 conn(0x7f65800d03b0 msgr2=0x7f65800ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 shutdown_connections 2023-10-16T10:01:49.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.727+0000 7f6585aa8700 1 -- 172.21.15.158:0/2099668261 wait complete. 2023-10-16T10:01:49.740 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T10:00:48", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T10:01:49.740 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2023-10-16T10:01:49.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.862+0000 7f37a9a47700 1 Processor -- start 2023-10-16T10:01:49.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.862+0000 7f37a9a47700 1 -- start start 2023-10-16T10:01:49.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.862+0000 7f37a9a47700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f37a415d840 0x7f37a415dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.862+0000 7f37a9a47700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37a406bbf0 0x7f37a406c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.862+0000 7f37a9a47700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a415e160 con 0x7f37a40b38f0 2023-10-16T10:01:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.862+0000 7f37a9a47700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a415ec80 con 0x7f37a415d840 2023-10-16T10:01:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.862+0000 7f37a9a47700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a40d13b0 con 0x7f37a406bbf0 2023-10-16T10:01:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f37a415d840 0x7f37a415dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f37a415d840 0x7f37a415dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55688/0 (socket says 172.21.15.158:55688) 2023-10-16T10:01:49.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 -- 172.21.15.158:0/2105382571 learned_addr learned my addr 172.21.15.158:0/2105382571 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:49.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a37fe700 1 --2- 172.21.15.158:0/2105382571 >> v2:172.21.15.172:3300/0 conn(0x7f37a406bbf0 0x7f37a406c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 -- 172.21.15.158:0/2105382571 >> v2:172.21.15.172:3300/0 conn(0x7f37a406bbf0 msgr2=0x7f37a406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:49.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 --2- 172.21.15.158:0/2105382571 >> v2:172.21.15.172:3300/0 conn(0x7f37a406bbf0 0x7f37a406c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 -- 172.21.15.158:0/2105382571 >> v1:172.21.15.172:6789/0 conn(0x7f37a40b38f0 legacy=0x7f37a40d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:49.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 -- 172.21.15.158:0/2105382571 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a415e3e0 con 0x7f37a415d840 2023-10-16T10:01:49.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a2ffd700 1 --2- 172.21.15.158:0/2105382571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a415dc20 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f37a40729f0 tx=0x7f3798009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31486d204efd7b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:49.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a1ffb700 1 -- 172.21.15.158:0/2105382571 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798004230 con 0x7f37a415d840 2023-10-16T10:01:49.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a1ffb700 1 -- 172.21.15.158:0/2105382571 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3798004390 con 0x7f37a415d840 2023-10-16T10:01:49.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.863+0000 7f37a1ffb700 1 -- 172.21.15.158:0/2105382571 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37980036a0 con 0x7f37a415d840 2023-10-16T10:01:49.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 -- 172.21.15.158:0/2105382571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 msgr2=0x7f37a415dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 --2- 172.21.15.158:0/2105382571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a415dc20 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f37a40729f0 tx=0x7f3798009bd0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 -- 172.21.15.158:0/2105382571 shutdown_connections 2023-10-16T10:01:49.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 --2- 172.21.15.158:0/2105382571 >> v2:172.21.15.172:3300/0 conn(0x7f37a406bbf0 0x7f37a406c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 --2- 172.21.15.158:0/2105382571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a415dc20 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 -- 172.21.15.158:0/2105382571 >> 172.21.15.158:0/2105382571 conn(0x7f37a406ec00 msgr2=0x7f37a406efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:49.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 -- 172.21.15.158:0/2105382571 shutdown_connections 2023-10-16T10:01:49.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.864+0000 7f37a9a47700 1 -- 172.21.15.158:0/2105382571 wait complete. 2023-10-16T10:01:49.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.865+0000 7f37a9a47700 1 Processor -- start 2023-10-16T10:01:49.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.866+0000 7f37a9a47700 1 -- start start 2023-10-16T10:01:49.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.866+0000 7f37a9a47700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37a406bbf0 0x7f37a40b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.866+0000 7f37a9a47700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37a40b38f0 0x7f37a40b5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.866+0000 7f37a9a47700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a41f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.866+0000 7f37a9a47700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40b45e0 con 0x7f37a415d840 2023-10-16T10:01:49.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a9a47700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40c53d0 con 0x7f37a406bbf0 2023-10-16T10:01:49.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a9a47700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40b47b0 con 0x7f37a40b38f0 2023-10-16T10:01:49.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a41f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a41f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55696/0 (socket says 172.21.15.158:55696) 2023-10-16T10:01:49.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 -- 172.21.15.158:0/1224457782 learned_addr learned my addr 172.21.15.158:0/1224457782 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:49.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a2ffd700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37a406bbf0 0x7f37a40b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a27fc700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37a40b38f0 0x7f37a40b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 -- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37a40b38f0 msgr2=0x7f37a40b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37a40b38f0 0x7f37a40b5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 -- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37a406bbf0 msgr2=0x7f37a40b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:49.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37a406bbf0 0x7f37a40b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:49.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 -- 172.21.15.158:0/1224457782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3798009550 con 0x7f37a415d840 2023-10-16T10:01:49.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a27fc700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37a40b38f0 0x7f37a40b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:01:49.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a37fe700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a41f6680 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f379c007e40 tx=0x7f379c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:49.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f378bfff700 1 -- 172.21.15.158:0/1224457782 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c00ede0 con 0x7f37a415d840 2023-10-16T10:01:49.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f378bfff700 1 -- 172.21.15.158:0/1224457782 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c014420 con 0x7f37a415d840 2023-10-16T10:01:49.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a40b4a90 con 0x7f37a415d840 2023-10-16T10:01:49.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f378bfff700 1 -- 172.21.15.158:0/1224457782 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c0147d0 con 0x7f37a415d840 2023-10-16T10:01:49.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.867+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a40b5050 con 0x7f37a415d840 2023-10-16T10:01:49.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.868+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a40681b0 con 0x7f37a415d840 2023-10-16T10:01:49.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.869+0000 7f378bfff700 1 -- 172.21.15.158:0/1224457782 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 58) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f379c014930 con 0x7f37a415d840 2023-10-16T10:01:49.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.869+0000 7f378bfff700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f379006b190 0x7f379006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:49.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.869+0000 7f378bfff700 1 -- 172.21.15.158:0/1224457782 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f379c08a510 con 0x7f37a415d840 2023-10-16T10:01:49.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.869+0000 7f37a2ffd700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f379006b190 0x7f379006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:49.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.870+0000 7f37a2ffd700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f379006b190 0x7f379006d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3798009130 tx=0x7f3798008030 comp rx=0 tx=0).ready entity=mgr.7920 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:49.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:49.874+0000 7f378bfff700 1 -- 172.21.15.158:0/1224457782 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f379c054f00 con 0x7f37a415d840 2023-10-16T10:01:50.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.092+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 --> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7f37a415e9a0 con 0x7f379006b190 2023-10-16T10:01:50.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.163+0000 7f378bfff700 1 -- 172.21.15.158:0/1224457782 <== mgr.7920 v2:172.21.15.158:6832/1516256968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7f37980041b0 con 0x7f379006b190 2023-10-16T10:01:50.165 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /bad-path 2023-10-16T10:01:50.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f379006b190 msgr2=0x7f379006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f379006b190 0x7f379006d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3798009130 tx=0x7f3798008030 comp rx=0 tx=0).stop 2023-10-16T10:01:50.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 msgr2=0x7f37a41f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a41f6680 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f379c007e40 tx=0x7f379c00e3c0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 shutdown_connections 2023-10-16T10:01:50.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f379006b190 0x7f379006d650 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37a415d840 0x7f37a41f6680 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37a406bbf0 0x7f37a40b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.165+0000 7f37a9a47700 1 --2- 172.21.15.158:0/1224457782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37a40b38f0 0x7f37a40b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.166+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 >> 172.21.15.158:0/1224457782 conn(0x7f37a406ec00 msgr2=0x7f37a415ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:50.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.166+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 shutdown_connections 2023-10-16T10:01:50.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.166+0000 7f37a9a47700 1 -- 172.21.15.158:0/1224457782 wait complete. 2023-10-16T10:01:50.179 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-10-16T10:01:50.179 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T10:01:50.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.301+0000 7f6ff94dd700 1 Processor -- start 2023-10-16T10:01:50.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.301+0000 7f6ff94dd700 1 -- start start 2023-10-16T10:01:50.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.302+0000 7f6ff94dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff415da40 0x7f6ff40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:50.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.302+0000 7f6ff94dd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6ff40cc590 0x7f6ff406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:50.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.302+0000 7f6ff94dd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40cbd30 con 0x7f6ff406ae10 2023-10-16T10:01:50.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.302+0000 7f6ff94dd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff406b280 con 0x7f6ff40cc590 2023-10-16T10:01:50.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.302+0000 7f6ff94dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40cc1c0 con 0x7f6ff415da40 2023-10-16T10:01:50.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff415da40 0x7f6ff40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:50.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff27fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6ff40cc590 0x7f6ff406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:50.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff415da40 0x7f6ff40caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:38302/0 (socket says 172.21.15.158:38302) 2023-10-16T10:01:50.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff37fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6ff406ae10 0x7f6ff40ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:59784/0 (socket says 172.21.15.158:59784) 2023-10-16T10:01:50.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff27fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6ff40cc590 0x7f6ff406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:56882/0 (socket says 172.21.15.158:56882) 2023-10-16T10:01:50.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/1067973640 learned_addr learned my addr 172.21.15.158:0/1067973640 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:50.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/1067973640 >> v2:172.21.15.172:3300/0 conn(0x7f6ff40cc590 msgr2=0x7f6ff406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:50.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 --2- 172.21.15.158:0/1067973640 >> v2:172.21.15.172:3300/0 conn(0x7f6ff40cc590 0x7f6ff406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/1067973640 >> v1:172.21.15.172:6789/0 conn(0x7f6ff406ae10 legacy=0x7f6ff40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/1067973640 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff415de20 con 0x7f6ff415da40 2023-10-16T10:01:50.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff27fc700 1 --2- 172.21.15.158:0/1067973640 >> v2:172.21.15.172:3300/0 conn(0x7f6ff40cc590 0x7f6ff406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:50.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff2ffd700 1 --2- 172.21.15.158:0/1067973640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff415da40 0x7f6ff40caa20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40098f0 tx=0x7f6fe4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c631b1f2b87b6d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:50.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff1ffb700 1 -- 172.21.15.158:0/1067973640 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4011830 con 0x7f6ff415da40 2023-10-16T10:01:50.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff1ffb700 1 -- 172.21.15.158:0/1067973640 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4011990 con 0x7f6ff415da40 2023-10-16T10:01:50.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.303+0000 7f6ff1ffb700 1 -- 172.21.15.158:0/1067973640 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4011d20 con 0x7f6ff415da40 2023-10-16T10:01:50.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 -- 172.21.15.158:0/1067973640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff415da40 msgr2=0x7f6ff40caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/1067973640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff415da40 0x7f6ff40caa20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40098f0 tx=0x7f6fe4009c00 comp rx=0 tx=0).stop 2023-10-16T10:01:50.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 -- 172.21.15.158:0/1067973640 shutdown_connections 2023-10-16T10:01:50.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/1067973640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff415da40 0x7f6ff40caa20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/1067973640 >> v2:172.21.15.172:3300/0 conn(0x7f6ff40cc590 0x7f6ff406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 -- 172.21.15.158:0/1067973640 >> 172.21.15.158:0/1067973640 conn(0x7f6ff4099a10 msgr2=0x7f6ff40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 -- 172.21.15.158:0/1067973640 shutdown_connections 2023-10-16T10:01:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.304+0000 7f6ff94dd700 1 -- 172.21.15.158:0/1067973640 wait complete. 2023-10-16T10:01:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.305+0000 7f6ff94dd700 1 Processor -- start 2023-10-16T10:01:50.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.305+0000 7f6ff94dd700 1 -- start start 2023-10-16T10:01:50.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff94dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406ae10 0x7f6ff415be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:50.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff94dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff40cc590 0x7f6ff40c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:50.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff94dd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ff415da40 0x7f6ff40c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:50.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff94dd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40c77d0 con 0x7f6ff406ae10 2023-10-16T10:01:50.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff94dd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41e6c10 con 0x7f6ff415da40 2023-10-16T10:01:50.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff94dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41e6ef0 con 0x7f6ff40cc590 2023-10-16T10:01:50.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406ae10 0x7f6ff415be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:50.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406ae10 0x7f6ff415be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55706/0 (socket says 172.21.15.158:55706) 2023-10-16T10:01:50.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/599258117 learned_addr learned my addr 172.21.15.158:0/599258117 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:50.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff27fc700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff40cc590 0x7f6ff40c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:50.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.306+0000 7f6ff37fe700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ff415da40 0x7f6ff40c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:50.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff40cc590 msgr2=0x7f6ff40c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff2ffd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff40cc590 0x7f6ff40c6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ff415da40 msgr2=0x7f6ff40c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff2ffd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ff415da40 0x7f6ff40c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff2ffd700 1 -- 172.21.15.158:0/599258117 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe80094a0 con 0x7f6ff406ae10 2023-10-16T10:01:50.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff37fe700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ff415da40 0x7f6ff40c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:01:50.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff2ffd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406ae10 0x7f6ff415be00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000c00 tx=0x7f6fe4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:50.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004570 con 0x7f6ff406ae10 2023-10-16T10:01:50.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe401f070 con 0x7f6ff406ae10 2023-10-16T10:01:50.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe4009550 con 0x7f6ff406ae10 2023-10-16T10:01:50.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40038c0 con 0x7f6ff406ae10 2023-10-16T10:01:50.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.307+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff41e74b0 con 0x7f6ff406ae10 2023-10-16T10:01:50.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.308+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff40c8e00 con 0x7f6ff406ae10 2023-10-16T10:01:50.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.308+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 58) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6fe4021030 con 0x7f6ff406ae10 2023-10-16T10:01:50.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.309+0000 7f6fdb7fe700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6fe006b190 0x7f6fe006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:50.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.309+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f6fe408b370 con 0x7f6ff406ae10 2023-10-16T10:01:50.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.309+0000 7f6ff27fc700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6fe006b190 0x7f6fe006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:50.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.309+0000 7f6ff27fc700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6fe006b190 0x7f6fe006d650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40cdc40 tx=0x7f6fe8008040 comp rx=0 tx=0).ready entity=mgr.7920 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:50.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.313+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fe4055c90 con 0x7f6ff406ae10 2023-10-16T10:01:50.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.561+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f6ff40681b0 con 0x7f6ff406ae10 2023-10-16T10:01:50.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.676+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v59) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6fe40592b0 con 0x7f6ff406ae10 2023-10-16T10:01:50.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.682+0000 7f6fdb7fe700 1 -- 172.21.15.158:0/599258117 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f6fe4056060 con 0x7f6ff406ae10 2023-10-16T10:01:50.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.682+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6fe006b190 msgr2=0x7f6fe006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.682+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6fe006b190 0x7f6fe006d650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40cdc40 tx=0x7f6fe8008040 comp rx=0 tx=0).stop 2023-10-16T10:01:50.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.682+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406ae10 msgr2=0x7f6ff415be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:50.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.682+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406ae10 0x7f6ff415be00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000c00 tx=0x7f6fe4004750 comp rx=0 tx=0).stop 2023-10-16T10:01:50.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 shutdown_connections 2023-10-16T10:01:50.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6fe006b190 0x7f6fe006d650 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ff415da40 0x7f6ff40c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406ae10 0x7f6ff415be00 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 --2- 172.21.15.158:0/599258117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ff40cc590 0x7f6ff40c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:50.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 >> 172.21.15.158:0/599258117 conn(0x7f6ff4099a10 msgr2=0x7f6ff406e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:50.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 shutdown_connections 2023-10-16T10:01:50.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:50.683+0000 7f6ff94dd700 1 -- 172.21.15.158:0/599258117 wait complete. 2023-10-16T10:01:50.697 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:01:50.698 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:01:50.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:01:50.776 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:01:50.777 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:01:50.777 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:01:50.799 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:01:50.799 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:01:50.827 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:50.826+0000 7f627c619200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:01:50.828 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:50.826+0000 7fd0111c9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:01:50.843 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:01:50.893 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:50.892+0000 7fd0111c9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:01:50.895 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:50.894+0000 7f627c619200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:01:50.966 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:50.965+0000 7fd0111c9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:01:50.970 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:50.969+0000 7f627c619200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:01:51.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.005+0000 7f78486e3700 1 Processor -- start 2023-10-16T10:01:51.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.005+0000 7f78486e3700 1 -- start start 2023-10-16T10:01:51.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.006+0000 7f78486e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.006+0000 7f78486e3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784006e770 con 0x7f78400cc5d0 2023-10-16T10:01:51.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.006+0000 7f78486e3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78400cbe00 con 0x7f784006f340 2023-10-16T10:01:51.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.006+0000 7f78486e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78400cf270 con 0x7f784015dfb0 2023-10-16T10:01:51.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.006+0000 7f784647f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f78400cc5d0 0x7f78400cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:55950/0 (socket says 172.21.15.158:55950) 2023-10-16T10:01:51.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.006+0000 7f784647f700 1 -- 172.21.15.158:0/528573660 learned_addr learned my addr 172.21.15.158:0/528573660 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:51.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.006+0000 7f7845c7e700 1 --2- 172.21.15.158:0/528573660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f784547d700 1 -- 172.21.15.158:0/528573660 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462461207 0 0) 0x7f783c001110 con 0x7f78400cc5d0 2023-10-16T10:01:51.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f784547d700 1 -- 172.21.15.158:0/528573660 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f782c0035b0 con 0x7f78400cc5d0 2023-10-16T10:01:51.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f784547d700 1 -- 172.21.15.158:0/528573660 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201913346 0 0) 0x7f78400cbe00 con 0x7f784006f340 2023-10-16T10:01:51.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f784547d700 1 -- 172.21.15.158:0/528573660 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f783c001110 con 0x7f784006f340 2023-10-16T10:01:51.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f7845c7e700 1 -- 172.21.15.158:0/528573660 >> v1:172.21.15.172:6789/0 conn(0x7f784006f340 legacy=0x7f7840071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f7845c7e700 1 -- 172.21.15.158:0/528573660 >> v1:172.21.15.158:6789/0 conn(0x7f78400cc5d0 legacy=0x7f78400cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f7845c7e700 1 -- 172.21.15.158:0/528573660 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784015e4e0 con 0x7f784015dfb0 2023-10-16T10:01:51.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f7845c7e700 1 --2- 172.21.15.158:0/528573660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784006e230 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7830009870 tx=0x7f7830009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5fdb33b0e105ba2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:51.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f784547d700 1 -- 172.21.15.158:0/528573660 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7830011830 con 0x7f784015dfb0 2023-10-16T10:01:51.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f784547d700 1 -- 172.21.15.158:0/528573660 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7830011990 con 0x7f784015dfb0 2023-10-16T10:01:51.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.007+0000 7f784547d700 1 -- 172.21.15.158:0/528573660 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7830011d20 con 0x7f784015dfb0 2023-10-16T10:01:51.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.008+0000 7f78486e3700 1 -- 172.21.15.158:0/528573660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 msgr2=0x7f784006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.008+0000 7f78486e3700 1 --2- 172.21.15.158:0/528573660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784006e230 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7830009870 tx=0x7f7830009b80 comp rx=0 tx=0).stop 2023-10-16T10:01:51.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.008+0000 7f78486e3700 1 -- 172.21.15.158:0/528573660 shutdown_connections 2023-10-16T10:01:51.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.008+0000 7f78486e3700 1 --2- 172.21.15.158:0/528573660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784006e230 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.008+0000 7f78486e3700 1 -- 172.21.15.158:0/528573660 >> 172.21.15.158:0/528573660 conn(0x7f78400d02b0 msgr2=0x7f78400d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:51.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.008+0000 7f78486e3700 1 -- 172.21.15.158:0/528573660 shutdown_connections 2023-10-16T10:01:51.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.008+0000 7f78486e3700 1 -- 172.21.15.158:0/528573660 wait complete. 2023-10-16T10:01:51.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.009+0000 7f78486e3700 1 Processor -- start 2023-10-16T10:01:51.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.009+0000 7f78486e3700 1 -- start start 2023-10-16T10:01:51.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f78486e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f784006f340 0x7f78400aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f78486e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78400cc5d0 0x7f784014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f78486e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784014a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f78486e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784014f7c0 con 0x7f784006f340 2023-10-16T10:01:51.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f78486e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784014a510 con 0x7f78400cc5d0 2023-10-16T10:01:51.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f78486e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784014c360 con 0x7f784015dfb0 2023-10-16T10:01:51.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f784006f340 0x7f78400aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f784006f340 0x7f78400aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55780/0 (socket says 172.21.15.158:55780) 2023-10-16T10:01:51.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 -- 172.21.15.158:0/336934404 learned_addr learned my addr 172.21.15.158:0/336934404 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:51.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f7845c7e700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78400cc5d0 0x7f784014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f7846c80700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784014a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 -- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 msgr2=0x7f784014a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784014a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 -- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78400cc5d0 msgr2=0x7f784014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78400cc5d0 0x7f784014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.010+0000 7f784647f700 1 -- 172.21.15.158:0/336934404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78300094a0 con 0x7f784006f340 2023-10-16T10:01:51.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.011+0000 7f784647f700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f784006f340 0x7f78400aeff0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f783c00c700 tx=0x7f783c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:51.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.011+0000 7f7836ffd700 1 -- 172.21.15.158:0/336934404 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c005d70 con 0x7f784006f340 2023-10-16T10:01:51.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.011+0000 7f7836ffd700 1 -- 172.21.15.158:0/336934404 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783c0070d0 con 0x7f784006f340 2023-10-16T10:01:51.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.011+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7840149900 con 0x7f784006f340 2023-10-16T10:01:51.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.011+0000 7f7836ffd700 1 -- 172.21.15.158:0/336934404 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c007440 con 0x7f784006f340 2023-10-16T10:01:51.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.011+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7840149ec0 con 0x7f784006f340 2023-10-16T10:01:51.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.012+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78400681b0 con 0x7f784006f340 2023-10-16T10:01:51.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.012+0000 7f7836ffd700 1 -- 172.21.15.158:0/336934404 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f783c0075a0 con 0x7f784006f340 2023-10-16T10:01:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.013+0000 7f7836ffd700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 0x7f782406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.013+0000 7f7845c7e700 1 -- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 msgr2=0x7f782406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.013+0000 7f7836ffd700 1 -- 172.21.15.158:0/336934404 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f783c089160 con 0x7f784006f340 2023-10-16T10:01:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.013+0000 7f7845c7e700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 0x7f782406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:51.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.017+0000 7f7836ffd700 1 -- 172.21.15.158:0/336934404 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f783c053a10 con 0x7f784006f340 2023-10-16T10:01:51.045 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:51.043+0000 7fd0111c9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:01:51.054 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:51.053+0000 7f627c619200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:01:51.184 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:51.183+0000 7fd0111c9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:01:51.200 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:51.199+0000 7f627c619200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:01:51.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.214+0000 7f7845c7e700 1 -- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 msgr2=0x7f782406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:51.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.214+0000 7f7845c7e700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 0x7f782406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:51.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.220+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f784006e770 con 0x7f784006f340 2023-10-16T10:01:51.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.221+0000 7f7836ffd700 1 -- 172.21.15.158:0/336934404 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+18086 (secure 0 0 0) 0x7f784006e770 con 0x7f784006f340 2023-10-16T10:01:51.222 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:51.223 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":114,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:01:49.664583+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:00:32.442785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T10:00:32.961620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.17","mappings":[{"from":1,"to":7}]},{"pgid":"14.2c","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/934459119":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/1239292381":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6833/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6832/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2498623628":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2203508586":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/521051078":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2930980190":"2023-10-17T10:00:47.874811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:01:51.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 msgr2=0x7f782406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:51.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 0x7f782406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f784006f340 msgr2=0x7f78400aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f784006f340 0x7f78400aeff0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f783c00c700 tx=0x7f783c00ca10 comp rx=0 tx=0).stop 2023-10-16T10:01:51.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 shutdown_connections 2023-10-16T10:01:51.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f782406b140 0x7f782406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784015dfb0 0x7f784014a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f784006f340 0x7f78400aeff0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 --2- 172.21.15.158:0/336934404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78400cc5d0 0x7f784014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 >> 172.21.15.158:0/336934404 conn(0x7f78400d02b0 msgr2=0x7f78400d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:51.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 shutdown_connections 2023-10-16T10:01:51.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.223+0000 7f78486e3700 1 -- 172.21.15.158:0/336934404 wait complete. 2023-10-16T10:01:51.237 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:00:32.442785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:00:32.961620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:01:51.238 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:01:51.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.402+0000 7f552a9d5700 1 Processor -- start 2023-10-16T10:01:51.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.402+0000 7f552a9d5700 1 -- start start 2023-10-16T10:01:51.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f552a9d5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f55240d29b0 0x7f55240d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f552a9d5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55240b7c90 con 0x7f5524160d30 2023-10-16T10:01:51.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f552a9d5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55240b7f70 con 0x7f55240d0a70 2023-10-16T10:01:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f552a9d5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55240b8910 con 0x7f55240d29b0 2023-10-16T10:01:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f5523fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5524160d30 0x7f55240b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56010/0 (socket says 172.21.15.158:56010) 2023-10-16T10:01:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f5523fff700 1 -- 172.21.15.158:0/1747178604 learned_addr learned my addr 172.21.15.158:0/1747178604 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f55237fe700 1 --2- 172.21.15.158:0/1747178604 >> v2:172.21.15.172:3300/0 conn(0x7f55240d29b0 0x7f55240d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f5522ffd700 1 -- 172.21.15.158:0/1747178604 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500099860 0 0) 0x7f55240b7c90 con 0x7f5524160d30 2023-10-16T10:01:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.403+0000 7f5522ffd700 1 -- 172.21.15.158:0/1747178604 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550c0035b0 con 0x7f5524160d30 2023-10-16T10:01:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f5522ffd700 1 -- 172.21.15.158:0/1747178604 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828669760 0 0) 0x7f55240b7f70 con 0x7f55240d0a70 2023-10-16T10:01:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f5522ffd700 1 -- 172.21.15.158:0/1747178604 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55240b7c90 con 0x7f55240d0a70 2023-10-16T10:01:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f55237fe700 1 -- 172.21.15.158:0/1747178604 >> v1:172.21.15.172:6789/0 conn(0x7f55240d0a70 legacy=0x7f55240b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f55237fe700 1 -- 172.21.15.158:0/1747178604 >> v1:172.21.15.158:6789/0 conn(0x7f5524160d30 legacy=0x7f55240b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f55237fe700 1 -- 172.21.15.158:0/1747178604 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55240b8250 con 0x7f55240d29b0 2023-10-16T10:01:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f55237fe700 1 --2- 172.21.15.158:0/1747178604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d29b0 0x7f55240d0530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5518009870 tx=0x7f5518009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d266858447f8ce1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f5522ffd700 1 -- 172.21.15.158:0/1747178604 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518011830 con 0x7f55240d29b0 2023-10-16T10:01:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f5522ffd700 1 -- 172.21.15.158:0/1747178604 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5518011990 con 0x7f55240d29b0 2023-10-16T10:01:51.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.404+0000 7f5522ffd700 1 -- 172.21.15.158:0/1747178604 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518011d20 con 0x7f55240d29b0 2023-10-16T10:01:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.405+0000 7f552a9d5700 1 -- 172.21.15.158:0/1747178604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d29b0 msgr2=0x7f55240d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.405+0000 7f552a9d5700 1 --2- 172.21.15.158:0/1747178604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d29b0 0x7f55240d0530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5518009870 tx=0x7f5518009b80 comp rx=0 tx=0).stop 2023-10-16T10:01:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.405+0000 7f552a9d5700 1 -- 172.21.15.158:0/1747178604 shutdown_connections 2023-10-16T10:01:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.405+0000 7f552a9d5700 1 --2- 172.21.15.158:0/1747178604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d29b0 0x7f55240d0530 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.405+0000 7f552a9d5700 1 -- 172.21.15.158:0/1747178604 >> 172.21.15.158:0/1747178604 conn(0x7f55240669f0 msgr2=0x7f5524066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.405+0000 7f552a9d5700 1 -- 172.21.15.158:0/1747178604 shutdown_connections 2023-10-16T10:01:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.405+0000 7f552a9d5700 1 -- 172.21.15.158:0/1747178604 wait complete. 2023-10-16T10:01:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.406+0000 7f552a9d5700 1 Processor -- start 2023-10-16T10:01:51.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.406+0000 7f552a9d5700 1 -- start start 2023-10-16T10:01:51.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f552a9d5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d0a70 0x7f552415e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f552a9d5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55240d29b0 0x7f552414fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f552a9d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5524160d30 0x7f5524156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f552a9d5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552415eac0 con 0x7f5524160d30 2023-10-16T10:01:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f552a9d5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5524155e40 con 0x7f55240d0a70 2023-10-16T10:01:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f552a9d5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5524156120 con 0x7f55240d29b0 2023-10-16T10:01:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5524160d30 0x7f5524156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5524160d30 0x7f5524156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55844/0 (socket says 172.21.15.158:55844) 2023-10-16T10:01:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 -- 172.21.15.158:0/3594046972 learned_addr learned my addr 172.21.15.158:0/3594046972 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5523fff700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d0a70 0x7f552415e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f55237fe700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55240d29b0 0x7f552414fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 -- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55240d29b0 msgr2=0x7f552414fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55240d29b0 0x7f552414fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 -- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d0a70 msgr2=0x7f552415e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d0a70 0x7f552415e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.407+0000 7f5528f72700 1 -- 172.21.15.158:0/3594046972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55180094a0 con 0x7f5524160d30 2023-10-16T10:01:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.408+0000 7f5528f72700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5524160d30 0x7f5524156da0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f551c00cc20 tx=0x7f551c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:51.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.408+0000 7f5520ff9700 1 -- 172.21.15.158:0/3594046972 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c003c00 con 0x7f5524160d30 2023-10-16T10:01:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.408+0000 7f5520ff9700 1 -- 172.21.15.158:0/3594046972 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c003d60 con 0x7f5524160d30 2023-10-16T10:01:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.408+0000 7f5520ff9700 1 -- 172.21.15.158:0/3594046972 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c00aae0 con 0x7f5524160d30 2023-10-16T10:01:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.408+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5524156400 con 0x7f5524160d30 2023-10-16T10:01:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.408+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5524150380 con 0x7f5524160d30 2023-10-16T10:01:51.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.409+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55241565f0 con 0x7f5524160d30 2023-10-16T10:01:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.409+0000 7f5520ff9700 1 -- 172.21.15.158:0/3594046972 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f551c00ac40 con 0x7f5524160d30 2023-10-16T10:01:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.410+0000 7f5520ff9700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 0x7f551006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.410+0000 7f5523fff700 1 -- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 msgr2=0x7f551006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.410+0000 7f5520ff9700 1 -- 172.21.15.158:0/3594046972 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f551c089f00 con 0x7f5524160d30 2023-10-16T10:01:51.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.410+0000 7f5523fff700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 0x7f551006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.414+0000 7f5520ff9700 1 -- 172.21.15.158:0/3594046972 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f551c054790 con 0x7f5524160d30 2023-10-16T10:01:51.510 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:51.509+0000 7fd0111c9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:01:51.553 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:51.552+0000 7f627c619200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:01:51.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.611+0000 7f5523fff700 1 -- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 msgr2=0x7f551006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:51.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.611+0000 7f5523fff700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 0x7f551006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:51.624 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:51.623+0000 7fd0111c9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:01:51.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.632+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55240681b0 con 0x7f5524160d30 2023-10-16T10:01:51.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.633+0000 7f5520ff9700 1 -- 172.21.15.158:0/3594046972 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f551c057db0 con 0x7f5524160d30 2023-10-16T10:01:51.634 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:01:51.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 msgr2=0x7f551006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:51.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 0x7f551006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5524160d30 msgr2=0x7f5524156da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5524160d30 0x7f5524156da0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f551c00cc20 tx=0x7f551c00cf30 comp rx=0 tx=0).stop 2023-10-16T10:01:51.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 shutdown_connections 2023-10-16T10:01:51.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f551006b140 0x7f551006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5524160d30 0x7f5524156da0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55240d0a70 0x7f552415e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 --2- 172.21.15.158:0/3594046972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55240d29b0 0x7f552414fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 >> 172.21.15.158:0/3594046972 conn(0x7f55240669f0 msgr2=0x7f5524066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:51.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.635+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 shutdown_connections 2023-10-16T10:01:51.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.636+0000 7f552a9d5700 1 -- 172.21.15.158:0/3594046972 wait complete. 2023-10-16T10:01:51.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:01:51.676 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:51.675+0000 7f627c619200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:01:51.698 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:51.697+0000 7fd0111c9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:01:51.754 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:51.753+0000 7f627c619200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:01:51.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.776+0000 7f82fabd2700 1 Processor -- start 2023-10-16T10:01:51.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.776+0000 7f82fabd2700 1 -- start start 2023-10-16T10:01:51.779 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:51.776+0000 7fd0111c9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:01:51.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82fabd2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82f415dc40 0x7f82f415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82fabd2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f406de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82fabd2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f415f380 con 0x7f82f406f180 2023-10-16T10:01:51.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82fabd2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f406e380 con 0x7f82f415dc40 2023-10-16T10:01:51.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82fabd2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4071e70 con 0x7f82f406d9c0 2023-10-16T10:01:51.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82f896e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82f415dc40 0x7f82f415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82f896e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82f415dc40 0x7f82f415ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55850/0 (socket says 172.21.15.158:55850) 2023-10-16T10:01:51.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82f896e700 1 -- 172.21.15.158:0/1495600055 learned_addr learned my addr 172.21.15.158:0/1495600055 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:51.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.777+0000 7f82f3fff700 1 --2- 172.21.15.158:0/1495600055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f406de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f37fe700 1 -- 172.21.15.158:0/1495600055 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746842555 0 0) 0x7f82ec001490 con 0x7f82f406f180 2023-10-16T10:01:51.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f896e700 1 -- 172.21.15.158:0/1495600055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 msgr2=0x7f82f406de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:51.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f896e700 1 --2- 172.21.15.158:0/1495600055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f406de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f896e700 1 -- 172.21.15.158:0/1495600055 >> v1:172.21.15.172:6789/0 conn(0x7f82f406f180 legacy=0x7f82f406f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f896e700 1 -- 172.21.15.158:0/1495600055 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f4072150 con 0x7f82f415dc40 2023-10-16T10:01:51.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f3fff700 1 --2- 172.21.15.158:0/1495600055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f406de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:51.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f896e700 1 --2- 172.21.15.158:0/1495600055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f415ee40 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f82e800bb90 tx=0x7f82e800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd2d3928d37dd49 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:51.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f37fe700 1 -- 172.21.15.158:0/1495600055 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e800dd30 con 0x7f82f415dc40 2023-10-16T10:01:51.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f37fe700 1 -- 172.21.15.158:0/1495600055 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e800de90 con 0x7f82f415dc40 2023-10-16T10:01:51.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.778+0000 7f82f37fe700 1 -- 172.21.15.158:0/1495600055 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8010690 con 0x7f82f415dc40 2023-10-16T10:01:51.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 -- 172.21.15.158:0/1495600055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 msgr2=0x7f82f415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1495600055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f415ee40 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f82e800bb90 tx=0x7f82e800bea0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 -- 172.21.15.158:0/1495600055 shutdown_connections 2023-10-16T10:01:51.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1495600055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f415ee40 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1495600055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f406de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 -- 172.21.15.158:0/1495600055 >> 172.21.15.158:0/1495600055 conn(0x7f82f40d03b0 msgr2=0x7f82f40d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:51.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 -- 172.21.15.158:0/1495600055 shutdown_connections 2023-10-16T10:01:51.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.779+0000 7f82fabd2700 1 -- 172.21.15.158:0/1495600055 wait complete. 2023-10-16T10:01:51.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.780+0000 7f82fabd2700 1 Processor -- start 2023-10-16T10:01:51.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.780+0000 7f82fabd2700 1 -- start start 2023-10-16T10:01:51.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82fabd2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f40b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82fabd2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82f406f180 0x7f82f40b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82fabd2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f40b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82fabd2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4153330 con 0x7f82f415dc40 2023-10-16T10:01:51.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82fabd2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b3b10 con 0x7f82f406f180 2023-10-16T10:01:51.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82fabd2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b3df0 con 0x7f82f406d9c0 2023-10-16T10:01:51.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f916f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f40b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82f406f180 0x7f82f40b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f896e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f40b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:51.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82f406f180 0x7f82f40b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:57038/0 (socket says 172.21.15.158:57038) 2023-10-16T10:01:51.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f896e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f40b56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:38458/0 (socket says 172.21.15.158:38458) 2023-10-16T10:01:51.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f916f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f40b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55860/0 (socket says 172.21.15.158:55860) 2023-10-16T10:01:51.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 -- 172.21.15.158:0/1095869224 learned_addr learned my addr 172.21.15.158:0/1095869224 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:51.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 -- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 msgr2=0x7f82f40b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f40b56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 -- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 msgr2=0x7f82f40b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f40b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 -- 172.21.15.158:0/1095869224 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00094a0 con 0x7f82f406f180 2023-10-16T10:01:51.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f916f700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f40b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:51.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f896e700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f40b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:51.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.781+0000 7f82f3fff700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82f406f180 0x7f82f40b2f30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f82e000ebe0 tx=0x7f82e000eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:51.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.782+0000 7f82f17fa700 1 -- 172.21.15.158:0/1095869224 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e000c850 con 0x7f82f406f180 2023-10-16T10:01:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.782+0000 7f82f17fa700 1 -- 172.21.15.158:0/1095869224 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e000c9b0 con 0x7f82f406f180 2023-10-16T10:01:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.782+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e800b7f0 con 0x7f82f406f180 2023-10-16T10:01:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.782+0000 7f82f17fa700 1 -- 172.21.15.158:0/1095869224 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e000cd40 con 0x7f82f406f180 2023-10-16T10:01:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.782+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40b43b0 con 0x7f82f406f180 2023-10-16T10:01:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.783+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f40681b0 con 0x7f82f406f180 2023-10-16T10:01:51.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.783+0000 7f82f17fa700 1 -- 172.21.15.158:0/1095869224 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f82e0013470 con 0x7f82f406f180 2023-10-16T10:01:51.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.783+0000 7f82f17fa700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 0x7f82d806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:51.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.783+0000 7f82f896e700 1 -- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 msgr2=0x7f82d806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:51.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.783+0000 7f82f17fa700 1 -- 172.21.15.158:0/1095869224 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f82e0089ff0 con 0x7f82f406f180 2023-10-16T10:01:51.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.783+0000 7f82f896e700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 0x7f82d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:51.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.788+0000 7f82f17fa700 1 -- 172.21.15.158:0/1095869224 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82e0054800 con 0x7f82f406f180 2023-10-16T10:01:51.839 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:51.838+0000 7f627c619200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:01:51.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.984+0000 7f82f896e700 1 -- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 msgr2=0x7f82d806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:51.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.984+0000 7f82f896e700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 0x7f82d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:51.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.986+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82f406e920 con 0x7f82f406f180 2023-10-16T10:01:51.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.987+0000 7f82f17fa700 1 -- 172.21.15.158:0/1095869224 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f82e000cea0 con 0x7f82f406f180 2023-10-16T10:01:51.988 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:01:51.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 msgr2=0x7f82d806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:51.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 0x7f82d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82f406f180 msgr2=0x7f82f40b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:51.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82f406f180 0x7f82f40b2f30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f82e000ebe0 tx=0x7f82e000eef0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 shutdown_connections 2023-10-16T10:01:51.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f82d806b140 0x7f82d806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f415dc40 0x7f82f40b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82f406d9c0 0x7f82f40b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 --2- 172.21.15.158:0/1095869224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82f406f180 0x7f82f40b2f30 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:51.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 >> 172.21.15.158:0/1095869224 conn(0x7f82f40d03b0 msgr2=0x7f82f40cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:51.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 shutdown_connections 2023-10-16T10:01:51.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:51.989+0000 7f82fabd2700 1 -- 172.21.15.158:0/1095869224 wait complete. 2023-10-16T10:01:52.003 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:01:52.149 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:52.148+0000 7fd0111c9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:01:52.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.168+0000 7f778d0e1700 1 Processor -- start 2023-10-16T10:01:52.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.168+0000 7f778d0e1700 1 -- start start 2023-10-16T10:01:52.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.169+0000 7f778d0e1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77880d29b0 0x7f77880d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.169+0000 7f778d0e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.169+0000 7f778d0e1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77880b7c90 con 0x7f7788160d30 2023-10-16T10:01:52.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.169+0000 7f778d0e1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77880b8910 con 0x7f77880d29b0 2023-10-16T10:01:52.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.169+0000 7f778d0e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77880d0f00 con 0x7f77880d0a70 2023-10-16T10:01:52.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.169+0000 7f7786d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7788160d30 0x7f77880b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56086/0 (socket says 172.21.15.158:56086) 2023-10-16T10:01:52.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.169+0000 7f7786d9d700 1 -- 172.21.15.158:0/1504432837 learned_addr learned my addr 172.21.15.158:0/1504432837 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:52.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778659c700 1 --2- 172.21.15.158:0/1504432837 >> v2:172.21.15.172:3300/0 conn(0x7f77880d29b0 0x7f77880d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778759e700 1 --2- 172.21.15.158:0/1504432837 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f7785d9b700 1 -- 172.21.15.158:0/1504432837 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271747469 0 0) 0x7f77880b7c90 con 0x7f7788160d30 2023-10-16T10:01:52.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f7785d9b700 1 -- 172.21.15.158:0/1504432837 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77700035b0 con 0x7f7788160d30 2023-10-16T10:01:52.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778759e700 1 -- 172.21.15.158:0/1504432837 >> v2:172.21.15.172:3300/0 conn(0x7f77880d29b0 msgr2=0x7f77880d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:52.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778759e700 1 --2- 172.21.15.158:0/1504432837 >> v2:172.21.15.172:3300/0 conn(0x7f77880d29b0 0x7f77880d0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778759e700 1 -- 172.21.15.158:0/1504432837 >> v1:172.21.15.158:6789/0 conn(0x7f7788160d30 legacy=0x7f77880b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778759e700 1 -- 172.21.15.158:0/1504432837 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880b7f70 con 0x7f77880d0a70 2023-10-16T10:01:52.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778659c700 1 --2- 172.21.15.158:0/1504432837 >> v2:172.21.15.172:3300/0 conn(0x7f77880d29b0 0x7f77880d0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:52.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f778759e700 1 --2- 172.21.15.158:0/1504432837 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880b6580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7780009d90 tx=0x7f778000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f03c50ec4edf464 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:52.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f7785d9b700 1 -- 172.21.15.158:0/1504432837 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778000e070 con 0x7f77880d0a70 2023-10-16T10:01:52.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f7785d9b700 1 -- 172.21.15.158:0/1504432837 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778000bb20 con 0x7f77880d0a70 2023-10-16T10:01:52.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.170+0000 7f7785d9b700 1 -- 172.21.15.158:0/1504432837 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780012430 con 0x7f77880d0a70 2023-10-16T10:01:52.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 -- 172.21.15.158:0/1504432837 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 msgr2=0x7f77880b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 --2- 172.21.15.158:0/1504432837 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880b6580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7780009d90 tx=0x7f778000b330 comp rx=0 tx=0).stop 2023-10-16T10:01:52.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 -- 172.21.15.158:0/1504432837 shutdown_connections 2023-10-16T10:01:52.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 --2- 172.21.15.158:0/1504432837 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880b6580 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 --2- 172.21.15.158:0/1504432837 >> v2:172.21.15.172:3300/0 conn(0x7f77880d29b0 0x7f77880d0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 -- 172.21.15.158:0/1504432837 >> 172.21.15.158:0/1504432837 conn(0x7f77880669f0 msgr2=0x7f7788066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:52.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 -- 172.21.15.158:0/1504432837 shutdown_connections 2023-10-16T10:01:52.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.171+0000 7f778d0e1700 1 -- 172.21.15.158:0/1504432837 wait complete. 2023-10-16T10:01:52.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.174+0000 7f778d0e1700 1 Processor -- start 2023-10-16T10:01:52.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.174+0000 7f778d0e1700 1 -- start start 2023-10-16T10:01:52.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778d0e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778d0e1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77880d29b0 0x7f77880ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778d0e1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7788160d30 0x7f77881f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778d0e1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77881f3f10 con 0x7f77880d29b0 2023-10-16T10:01:52.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778d0e1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77881573c0 con 0x7f7788160d30 2023-10-16T10:01:52.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778d0e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77880c9440 con 0x7f77880d0a70 2023-10-16T10:01:52.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77880d29b0 0x7f77880ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77880d29b0 0x7f77880ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55918/0 (socket says 172.21.15.158:55918) 2023-10-16T10:01:52.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 -- 172.21.15.158:0/403926810 learned_addr learned my addr 172.21.15.158:0/403926810 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:52.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778759e700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7788160d30 0x7f77881f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f7786d9d700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 -- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 msgr2=0x7f77880ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880ce2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 -- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7788160d30 msgr2=0x7f77881f2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7788160d30 0x7f77881f2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 -- 172.21.15.158:0/403926810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0094a0 con 0x7f77880d29b0 2023-10-16T10:01:52.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.175+0000 7f778659c700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77880d29b0 0x7f77880ce820 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f777c00ec10 tx=0x7f777c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:52.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.176+0000 7f776f7fe700 1 -- 172.21.15.158:0/403926810 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c00c850 con 0x7f77880d29b0 2023-10-16T10:01:52.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.176+0000 7f776f7fe700 1 -- 172.21.15.158:0/403926810 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c00c9b0 con 0x7f77880d29b0 2023-10-16T10:01:52.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.176+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7780009a70 con 0x7f77880d29b0 2023-10-16T10:01:52.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.176+0000 7f776f7fe700 1 -- 172.21.15.158:0/403926810 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c00cd20 con 0x7f77880d29b0 2023-10-16T10:01:52.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.176+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77881f3c30 con 0x7f77880d29b0 2023-10-16T10:01:52.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.177+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77880681b0 con 0x7f77880d29b0 2023-10-16T10:01:52.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.177+0000 7f776f7fe700 1 -- 172.21.15.158:0/403926810 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f777c013470 con 0x7f77880d29b0 2023-10-16T10:01:52.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.178+0000 7f776f7fe700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 0x7f777406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.178+0000 7f7786d9d700 1 -- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 msgr2=0x7f777406d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:52.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.178+0000 7f776f7fe700 1 -- 172.21.15.158:0/403926810 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f777c089f10 con 0x7f77880d29b0 2023-10-16T10:01:52.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.178+0000 7f7786d9d700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 0x7f777406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:52.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.182+0000 7f776f7fe700 1 -- 172.21.15.158:0/403926810 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f777c0547a0 con 0x7f77880d29b0 2023-10-16T10:01:52.230 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:52.229+0000 7f627c619200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:01:52.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.379+0000 7f7786d9d700 1 -- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 msgr2=0x7f777406d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:52.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.379+0000 7f7786d9d700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 0x7f777406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:52.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.382+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7788156570 con 0x7f77880d29b0 2023-10-16T10:01:52.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.382+0000 7f776f7fe700 1 -- 172.21.15.158:0/403926810 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f777c0210a0 con 0x7f77880d29b0 2023-10-16T10:01:52.384 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:01:52.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 msgr2=0x7f777406d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:52.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 0x7f777406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77880d29b0 msgr2=0x7f77880ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77880d29b0 0x7f77880ce820 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f777c00ec10 tx=0x7f777c00ef20 comp rx=0 tx=0).stop 2023-10-16T10:01:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 shutdown_connections 2023-10-16T10:01:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f777406b170 0x7f777406d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7788160d30 0x7f77881f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77880d0a70 0x7f77880ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 --2- 172.21.15.158:0/403926810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77880d29b0 0x7f77880ce820 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 >> 172.21.15.158:0/403926810 conn(0x7f77880669f0 msgr2=0x7f77881549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:52.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 shutdown_connections 2023-10-16T10:01:52.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.384+0000 7f778d0e1700 1 -- 172.21.15.158:0/403926810 wait complete. 2023-10-16T10:01:52.398 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T10:01:52.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.520+0000 7ff37c826700 1 Processor -- start 2023-10-16T10:01:52.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.520+0000 7ff37c826700 1 -- start start 2023-10-16T10:01:52.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff37c826700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff37815b650 0x7ff37815fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff37c826700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3780b4320 0x7ff3780b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff37c826700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3780cf550 con 0x7ff3780b47d0 2023-10-16T10:01:52.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff37c826700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3780cc6d0 con 0x7ff3780b4320 2023-10-16T10:01:52.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff37c826700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3780cf2b0 con 0x7ff37815b650 2023-10-16T10:01:52.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff375d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3780b4320 0x7ff3780b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff376d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff3780b47d0 0x7ff3780cf9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56106/0 (socket says 172.21.15.158:56106) 2023-10-16T10:01:52.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.521+0000 7ff376d9d700 1 -- 172.21.15.158:0/2530835716 learned_addr learned my addr 172.21.15.158:0/2530835716 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:52.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff37659c700 1 --2- 172.21.15.158:0/2530835716 >> v2:172.21.15.172:3300/0 conn(0x7ff37815b650 0x7ff37815fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff375d9b700 1 -- 172.21.15.158:0/2530835716 >> v2:172.21.15.172:3300/0 conn(0x7ff37815b650 msgr2=0x7ff37815fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:52.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff375d9b700 1 --2- 172.21.15.158:0/2530835716 >> v2:172.21.15.172:3300/0 conn(0x7ff37815b650 0x7ff37815fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff375d9b700 1 -- 172.21.15.158:0/2530835716 >> v1:172.21.15.158:6789/0 conn(0x7ff3780b47d0 legacy=0x7ff3780cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff375d9b700 1 -- 172.21.15.158:0/2530835716 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3781600f0 con 0x7ff3780b4320 2023-10-16T10:01:52.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff37559a700 1 -- 172.21.15.158:0/2530835716 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742123536 0 0) 0x7ff3780cf550 con 0x7ff3780b47d0 2023-10-16T10:01:52.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff375d9b700 1 --2- 172.21.15.158:0/2530835716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3780b4320 0x7ff3780b2f50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7ff3780d14a0 tx=0x7ff36c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed94729ea3940326 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:52.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff37559a700 1 -- 172.21.15.158:0/2530835716 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c011830 con 0x7ff3780b4320 2023-10-16T10:01:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff37559a700 1 -- 172.21.15.158:0/2530835716 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c011990 con 0x7ff3780b4320 2023-10-16T10:01:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.522+0000 7ff37559a700 1 -- 172.21.15.158:0/2530835716 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c011d20 con 0x7ff3780b4320 2023-10-16T10:01:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 -- 172.21.15.158:0/2530835716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3780b4320 msgr2=0x7ff3780b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 --2- 172.21.15.158:0/2530835716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3780b4320 0x7ff3780b2f50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7ff3780d14a0 tx=0x7ff36c009b20 comp rx=0 tx=0).stop 2023-10-16T10:01:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 -- 172.21.15.158:0/2530835716 shutdown_connections 2023-10-16T10:01:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 --2- 172.21.15.158:0/2530835716 >> v2:172.21.15.172:3300/0 conn(0x7ff37815b650 0x7ff37815fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 --2- 172.21.15.158:0/2530835716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3780b4320 0x7ff3780b2f50 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 -- 172.21.15.158:0/2530835716 >> 172.21.15.158:0/2530835716 conn(0x7ff3780cdf50 msgr2=0x7ff3780ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 -- 172.21.15.158:0/2530835716 shutdown_connections 2023-10-16T10:01:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.523+0000 7ff37c826700 1 -- 172.21.15.158:0/2530835716 wait complete. 2023-10-16T10:01:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.524+0000 7ff37c826700 1 Processor -- start 2023-10-16T10:01:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.524+0000 7ff37c826700 1 -- start start 2023-10-16T10:01:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff37c826700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3780b4320 0x7ff3780c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff37c826700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3780b47d0 0x7ff3780c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff37c826700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff37815b650 0x7ff3780a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff37c826700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37814d230 con 0x7ff37815b650 2023-10-16T10:01:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff37c826700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37814c620 con 0x7ff3780b4320 2023-10-16T10:01:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff37c826700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37814c900 con 0x7ff3780b47d0 2023-10-16T10:01:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff376d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff37815b650 0x7ff3780a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff376d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff37815b650 0x7ff3780a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55936/0 (socket says 172.21.15.158:55936) 2023-10-16T10:01:52.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff376d9d700 1 -- 172.21.15.158:0/2313367483 learned_addr learned my addr 172.21.15.158:0/2313367483 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:52.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.525+0000 7ff375d9b700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3780b47d0 0x7ff3780c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff37659c700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3780b4320 0x7ff3780c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:52.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff376d9d700 1 -- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3780b47d0 msgr2=0x7ff3780c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff376d9d700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3780b47d0 0x7ff3780c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff376d9d700 1 -- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3780b4320 msgr2=0x7ff3780c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff376d9d700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3780b4320 0x7ff3780c47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff376d9d700 1 -- 172.21.15.158:0/2313367483 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c0094a0 con 0x7ff37815b650 2023-10-16T10:01:52.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff376d9d700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff37815b650 0x7ff3780a84f0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7ff370002230 tx=0x7ff37000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:52.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff366ffd700 1 -- 172.21.15.158:0/2313367483 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37000ab70 con 0x7ff37815b650 2023-10-16T10:01:52.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff366ffd700 1 -- 172.21.15.158:0/2313367483 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37000acf0 con 0x7ff37815b650 2023-10-16T10:01:52.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37814cbe0 con 0x7ff37815b650 2023-10-16T10:01:52.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff366ffd700 1 -- 172.21.15.158:0/2313367483 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370007310 con 0x7ff37815b650 2023-10-16T10:01:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.526+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3780c4d00 con 0x7ff37815b650 2023-10-16T10:01:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.527+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3780681b0 con 0x7ff37815b650 2023-10-16T10:01:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.527+0000 7ff366ffd700 1 -- 172.21.15.158:0/2313367483 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff370007470 con 0x7ff37815b650 2023-10-16T10:01:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.528+0000 7ff366ffd700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 0x7ff35806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.528+0000 7ff37659c700 1 -- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 msgr2=0x7ff35806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.528+0000 7ff366ffd700 1 -- 172.21.15.158:0/2313367483 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7ff3700899d0 con 0x7ff37815b650 2023-10-16T10:01:52.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.528+0000 7ff37659c700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 0x7ff35806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:52.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.533+0000 7ff366ffd700 1 -- 172.21.15.158:0/2313367483 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff370054260 con 0x7ff37815b650 2023-10-16T10:01:52.722 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:52.721+0000 7fd0111c9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:01:52.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.729+0000 7ff37659c700 1 -- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 msgr2=0x7ff35806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:52.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.729+0000 7ff37659c700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 0x7ff35806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:52.792 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:52.791+0000 7fd0111c9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:01:52.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.795+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff37814d370 con 0x7ff37815b650 2023-10-16T10:01:52.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.795+0000 7ff366ffd700 1 -- 172.21.15.158:0/2313367483 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v114) v1 ==== 80+0+476 (secure 0 0 0) 0x7ff3700181f0 con 0x7ff37815b650 2023-10-16T10:01:52.797 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/934459119 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.797 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1239292381 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.797 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/1430981267 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.797 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/1430981267 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.797 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2498623628 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.798 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2203508586 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.798 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/521051078 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.798 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2930980190 2023-10-17T10:00:47.874811+0000 2023-10-16T10:01:52.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.797+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 msgr2=0x7ff35806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:52.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.797+0000 7ff37c826700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 0x7ff35806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.797+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff37815b650 msgr2=0x7ff3780a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:52.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.797+0000 7ff37c826700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff37815b650 0x7ff3780a84f0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7ff370002230 tx=0x7ff37000cf50 comp rx=0 tx=0).stop 2023-10-16T10:01:52.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 shutdown_connections 2023-10-16T10:01:52.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7ff35806b140 0x7ff35806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff37815b650 0x7ff3780a84f0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3780b4320 0x7ff3780c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 --2- 172.21.15.158:0/2313367483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3780b47d0 0x7ff3780c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:52.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 >> 172.21.15.158:0/2313367483 conn(0x7ff3780cdf50 msgr2=0x7ff3780cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:52.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 shutdown_connections 2023-10-16T10:01:52.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:52.798+0000 7ff37c826700 1 -- 172.21.15.158:0/2313367483 wait complete. 2023-10-16T10:01:52.802 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T10:01:52.814 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T10:01:52.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:01:52.815 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T10:01:52.831 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:52.830+0000 7f627c619200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:01:52.858 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T10:01:52.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:01:52.859 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T10:01:52.908 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:52.907+0000 7f627c619200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:01:52.915 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:01:52.915 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:01:52.932 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:01:52.933 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:01:52.933 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:01:52.972 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:01:52.973 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:01:52.973 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:01:52.991 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:01:52.991 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:01:53.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.134+0000 7f673b59e700 1 Processor -- start 2023-10-16T10:01:53.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.134+0000 7f673b59e700 1 -- start start 2023-10-16T10:01:53.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c0cedc0 0x7f673c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673b59e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0cbee0 con 0x7f673c15edd0 2023-10-16T10:01:53.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673b59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c15e0a0 con 0x7f673c06ea30 2023-10-16T10:01:53.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673b59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c15b8e0 con 0x7f673c0cedc0 2023-10-16T10:01:53.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f6739d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f673c15edd0 0x7f673c15ab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56118/0 (socket says 172.21.15.158:56118) 2023-10-16T10:01:53.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f6739d9b700 1 -- 172.21.15.158:0/555155912 learned_addr learned my addr 172.21.15.158:0/555155912 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:53.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673ad9d700 1 --2- 172.21.15.158:0/555155912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c0cedc0 0x7f673c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673959a700 1 -- 172.21.15.158:0/555155912 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569732916 0 0) 0x7f6730001110 con 0x7f673c15edd0 2023-10-16T10:01:53.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673959a700 1 -- 172.21.15.158:0/555155912 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67240035b0 con 0x7f673c15edd0 2023-10-16T10:01:53.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673959a700 1 -- 172.21.15.158:0/555155912 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280310178 0 0) 0x7f673c15e0a0 con 0x7f673c06ea30 2023-10-16T10:01:53.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673959a700 1 -- 172.21.15.158:0/555155912 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6730001110 con 0x7f673c06ea30 2023-10-16T10:01:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673ad9d700 1 -- 172.21.15.158:0/555155912 >> v1:172.21.15.172:6789/0 conn(0x7f673c06ea30 legacy=0x7f673c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673ad9d700 1 -- 172.21.15.158:0/555155912 >> v1:172.21.15.158:6789/0 conn(0x7f673c15edd0 legacy=0x7f673c15ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673ad9d700 1 -- 172.21.15.158:0/555155912 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c15e2f0 con 0x7f673c0cedc0 2023-10-16T10:01:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.136+0000 7f673ad9d700 1 --2- 172.21.15.158:0/555155912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c0cedc0 0x7f673c15b2a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6734009df0 tx=0x7f673400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f925eb3e6c5acd44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673959a700 1 -- 172.21.15.158:0/555155912 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673400e070 con 0x7f673c0cedc0 2023-10-16T10:01:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673959a700 1 -- 172.21.15.158:0/555155912 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673400bae0 con 0x7f673c0cedc0 2023-10-16T10:01:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673959a700 1 -- 172.21.15.158:0/555155912 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673400be90 con 0x7f673c0cedc0 2023-10-16T10:01:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673b59e700 1 -- 172.21.15.158:0/555155912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c0cedc0 msgr2=0x7f673c15b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673b59e700 1 --2- 172.21.15.158:0/555155912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c0cedc0 0x7f673c15b2a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6734009df0 tx=0x7f673400b330 comp rx=0 tx=0).stop 2023-10-16T10:01:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673b59e700 1 -- 172.21.15.158:0/555155912 shutdown_connections 2023-10-16T10:01:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673b59e700 1 --2- 172.21.15.158:0/555155912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c0cedc0 0x7f673c15b2a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673b59e700 1 -- 172.21.15.158:0/555155912 >> 172.21.15.158:0/555155912 conn(0x7f673c0cfe00 msgr2=0x7f673c0d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673b59e700 1 -- 172.21.15.158:0/555155912 shutdown_connections 2023-10-16T10:01:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.137+0000 7f673b59e700 1 -- 172.21.15.158:0/555155912 wait complete. 2023-10-16T10:01:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.139+0000 7f673b59e700 1 Processor -- start 2023-10-16T10:01:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.139+0000 7f673b59e700 1 -- start start 2023-10-16T10:01:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f673c06ea30 0x7f673c1471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f673c0cedc0 0x7f673c0c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c15edd0 0x7f673c0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673b59e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0c5510 con 0x7f673c0cedc0 2023-10-16T10:01:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673b59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0af3d0 con 0x7f673c06ea30 2023-10-16T10:01:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673b59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0c6ff0 con 0x7f673c15edd0 2023-10-16T10:01:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f673c0cedc0 0x7f673c0c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f673c0cedc0 0x7f673c0c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55946/0 (socket says 172.21.15.158:55946) 2023-10-16T10:01:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 -- 172.21.15.158:0/4149729392 learned_addr learned my addr 172.21.15.158:0/4149729392 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673a59c700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f673c06ea30 0x7f673c1471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f673ad9d700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c15edd0 0x7f673c0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 -- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c15edd0 msgr2=0x7f673c0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c15edd0 0x7f673c0c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 -- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f673c06ea30 msgr2=0x7f673c1471a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f673c06ea30 0x7f673c1471a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 -- 172.21.15.158:0/4149729392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734009a70 con 0x7f673c0cedc0 2023-10-16T10:01:53.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.140+0000 7f6739d9b700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f673c0cedc0 0x7f673c0c6ab0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f673000c700 tx=0x7f673000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:53.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.141+0000 7f672affd700 1 -- 172.21.15.158:0/4149729392 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6730005d70 con 0x7f673c0cedc0 2023-10-16T10:01:53.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.141+0000 7f672affd700 1 -- 172.21.15.158:0/4149729392 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67300070d0 con 0x7f673c0cedc0 2023-10-16T10:01:53.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.141+0000 7f672affd700 1 -- 172.21.15.158:0/4149729392 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6730007440 con 0x7f673c0cedc0 2023-10-16T10:01:53.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.141+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c14d970 con 0x7f673c0cedc0 2023-10-16T10:01:53.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.141+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c0af620 con 0x7f673c0cedc0 2023-10-16T10:01:53.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.142+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c0c56c0 con 0x7f673c0cedc0 2023-10-16T10:01:53.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.142+0000 7f672affd700 1 -- 172.21.15.158:0/4149729392 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f67300075a0 con 0x7f673c0cedc0 2023-10-16T10:01:53.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.143+0000 7f672affd700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 0x7f671c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.143+0000 7f673a59c700 1 -- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 msgr2=0x7f671c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:53.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.143+0000 7f672affd700 1 -- 172.21.15.158:0/4149729392 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f67300890d0 con 0x7f673c0cedc0 2023-10-16T10:01:53.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.143+0000 7f673a59c700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 0x7f671c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:53.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.147+0000 7f672affd700 1 -- 172.21.15.158:0/4149729392 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f673c0c56c0 con 0x7f673c0cedc0 2023-10-16T10:01:53.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.344+0000 7f673a59c700 1 -- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 msgr2=0x7f671c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:53.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.344+0000 7f673a59c700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 0x7f671c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:53.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.394+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f673c0cbee0 con 0x7f673c0cedc0 2023-10-16T10:01:53.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.395+0000 7f672affd700 1 -- 172.21.15.158:0/4149729392 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f673c0cbee0 con 0x7f673c0cedc0 2023-10-16T10:01:53.396 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:53.396 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2674019415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2674019415},{"type":"v1","addr":"172.21.15.158:6836","nonce":2674019415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2425279024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2425279024},{"type":"v1","addr":"172.21.15.158:6837","nonce":2425279024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3002995342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3002995342},{"type":"v1","addr":"172.21.15.172:6834","nonce":3002995342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:00:34.013744+0000","modified":"2023-10-16T10:00:37.052969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7740":{"gid":7740,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1596548674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1596548674},{"type":"v1","addr":"172.21.15.172:6835","nonce":1596548674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T10:01:53.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 msgr2=0x7f671c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:53.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 0x7f671c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f673c0cedc0 msgr2=0x7f673c0c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f673c0cedc0 0x7f673c0c6ab0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f673000c700 tx=0x7f673000ca10 comp rx=0 tx=0).stop 2023-10-16T10:01:53.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 shutdown_connections 2023-10-16T10:01:53.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f671c06b140 0x7f671c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f673c15edd0 0x7f673c0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f673c06ea30 0x7f673c1471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 --2- 172.21.15.158:0/4149729392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f673c0cedc0 0x7f673c0c6ab0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 >> 172.21.15.158:0/4149729392 conn(0x7f673c0cfe00 msgr2=0x7f673c158a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:53.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 shutdown_connections 2023-10-16T10:01:53.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.397+0000 7f673b59e700 1 -- 172.21.15.158:0/4149729392 wait complete. 2023-10-16T10:01:53.401 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 51 2023-10-16T10:01:53.411 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:01:53.434 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:53.432+0000 7fd0111c9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:01:53.511 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:53.510+0000 7fd0111c9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:01:53.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.538+0000 7febfbe0c700 1 Processor -- start 2023-10-16T10:01:53.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.539+0000 7febfbe0c700 1 -- start start 2023-10-16T10:01:53.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.539+0000 7febfbe0c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febf406afe0 0x7febf4072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.539+0000 7febfbe0c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf406f850 con 0x7febf406f470 2023-10-16T10:01:53.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.539+0000 7febfbe0c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf40b4260 con 0x7febf4072360 2023-10-16T10:01:53.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.539+0000 7febfbe0c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf4072d50 con 0x7febf406afe0 2023-10-16T10:01:53.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febfa3a9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febf406afe0 0x7febf4072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febfa3a9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febf406afe0 0x7febf4072810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55958/0 (socket says 172.21.15.158:55958) 2023-10-16T10:01:53.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febfa3a9700 1 -- 172.21.15.158:0/972680159 learned_addr learned my addr 172.21.15.158:0/972680159 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:53.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febf9ba8700 1 --1- 172.21.15.158:0/972680159 >> v1:172.21.15.158:6789/0 conn(0x7febf406f470 0x7febf406e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56130/0 (socket says 172.21.15.158:56130) 2023-10-16T10:01:53.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febfa3a9700 1 -- 172.21.15.158:0/972680159 >> v1:172.21.15.172:6789/0 conn(0x7febf4072360 legacy=0x7febf406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febfa3a9700 1 -- 172.21.15.158:0/972680159 >> v1:172.21.15.158:6789/0 conn(0x7febf406f470 legacy=0x7febf406e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febfa3a9700 1 -- 172.21.15.158:0/972680159 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40b4540 con 0x7febf406afe0 2023-10-16T10:01:53.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febf8ba6700 1 -- 172.21.15.158:0/972680159 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060632681 0 0) 0x7febf406f850 con 0x7febf406f470 2023-10-16T10:01:53.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febfa3a9700 1 --2- 172.21.15.158:0/972680159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf4072810 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7febec009e50 tx=0x7febec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb647de4979067d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:53.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febf8ba6700 1 -- 172.21.15.158:0/972680159 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec00e070 con 0x7febf406afe0 2023-10-16T10:01:53.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febf8ba6700 1 -- 172.21.15.158:0/972680159 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec00b9c0 con 0x7febf406afe0 2023-10-16T10:01:53.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.540+0000 7febf8ba6700 1 -- 172.21.15.158:0/972680159 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec00bd70 con 0x7febf406afe0 2023-10-16T10:01:53.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.541+0000 7febfbe0c700 1 -- 172.21.15.158:0/972680159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 msgr2=0x7febf4072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.541+0000 7febfbe0c700 1 --2- 172.21.15.158:0/972680159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf4072810 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7febec009e50 tx=0x7febec00b330 comp rx=0 tx=0).stop 2023-10-16T10:01:53.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.541+0000 7febfbe0c700 1 -- 172.21.15.158:0/972680159 shutdown_connections 2023-10-16T10:01:53.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.541+0000 7febfbe0c700 1 --2- 172.21.15.158:0/972680159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf4072810 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.541+0000 7febfbe0c700 1 -- 172.21.15.158:0/972680159 >> 172.21.15.158:0/972680159 conn(0x7febf40ccda0 msgr2=0x7febf40cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:53.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.541+0000 7febfbe0c700 1 -- 172.21.15.158:0/972680159 shutdown_connections 2023-10-16T10:01:53.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.541+0000 7febfbe0c700 1 -- 172.21.15.158:0/972680159 wait complete. 2023-10-16T10:01:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.543+0000 7febfbe0c700 1 Processor -- start 2023-10-16T10:01:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.543+0000 7febfbe0c700 1 -- start start 2023-10-16T10:01:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf41543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf406f470 0x7febf41f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf4072360 0x7febf41f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf41f7600 con 0x7febf406afe0 2023-10-16T10:01:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf40c77d0 con 0x7febf4072360 2023-10-16T10:01:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf40c7ab0 con 0x7febf406f470 2023-10-16T10:01:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf41543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf41543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:55960/0 (socket says 172.21.15.158:55960) 2023-10-16T10:01:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 -- 172.21.15.158:0/1283675090 learned_addr learned my addr 172.21.15.158:0/1283675090 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfa3a9700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf4072360 0x7febf41f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf93a7700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf406f470 0x7febf41f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 -- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf406f470 msgr2=0x7febf41f70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf406f470 0x7febf41f70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 -- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf4072360 msgr2=0x7febf41f30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf4072360 0x7febf41f30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 -- 172.21.15.158:0/1283675090 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec009b30 con 0x7febf406afe0 2023-10-16T10:01:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf93a7700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf406f470 0x7febf41f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:01:53.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febf9ba8700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf41543b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7febf000c500 tx=0x7febf000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febea7fc700 1 -- 172.21.15.158:0/1283675090 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf00074e0 con 0x7febf406afe0 2023-10-16T10:01:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febea7fc700 1 -- 172.21.15.158:0/1283675090 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf0007640 con 0x7febf406afe0 2023-10-16T10:01:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf40c7d90 con 0x7febf406afe0 2023-10-16T10:01:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febea7fc700 1 -- 172.21.15.158:0/1283675090 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0005a70 con 0x7febf406afe0 2023-10-16T10:01:53.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.544+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf40c8350 con 0x7febf406afe0 2023-10-16T10:01:53.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.545+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf40681b0 con 0x7febf406afe0 2023-10-16T10:01:53.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.546+0000 7febea7fc700 1 -- 172.21.15.158:0/1283675090 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7febf0005bd0 con 0x7febf406afe0 2023-10-16T10:01:53.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.546+0000 7febea7fc700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 0x7febd806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.546+0000 7febf93a7700 1 -- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 msgr2=0x7febd806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:53.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.546+0000 7febea7fc700 1 -- 172.21.15.158:0/1283675090 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7febf00896e0 con 0x7febf406afe0 2023-10-16T10:01:53.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.546+0000 7febf93a7700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 0x7febd806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:53.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.550+0000 7febea7fc700 1 -- 172.21.15.158:0/1283675090 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febf0054050 con 0x7febf406afe0 2023-10-16T10:01:53.580 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:53.579+0000 7f627c619200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:01:53.661 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:53.659+0000 7fd0111c9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:01:53.662 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:53.661+0000 7f627c619200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:01:53.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.747+0000 7febf93a7700 1 -- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 msgr2=0x7febd806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:53.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.747+0000 7febf93a7700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 0x7febd806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:53.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.759+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febf40ca6e0 con 0x7febf406afe0 2023-10-16T10:01:53.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.760+0000 7febea7fc700 1 -- 172.21.15.158:0/1283675090 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+18086 (secure 0 0 0) 0x7febf0057670 con 0x7febf406afe0 2023-10-16T10:01:53.761 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:53.762 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":114,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:01:49.664583+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:00:32.442785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T10:00:32.961620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.17","mappings":[{"from":1,"to":7}]},{"pgid":"14.2c","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/934459119":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/1239292381":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6833/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6832/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2498623628":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2203508586":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/521051078":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2930980190":"2023-10-17T10:00:47.874811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:01:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 msgr2=0x7febd806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 0x7febd806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 msgr2=0x7febf41543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf41543b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7febf000c500 tx=0x7febf000c810 comp rx=0 tx=0).stop 2023-10-16T10:01:53.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 shutdown_connections 2023-10-16T10:01:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7febd806b140 0x7febd806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf4072360 0x7febf41f30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febf406afe0 0x7febf41543b0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 --2- 172.21.15.158:0/1283675090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf406f470 0x7febf41f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 >> 172.21.15.158:0/1283675090 conn(0x7febf40ccda0 msgr2=0x7febf40ad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 shutdown_connections 2023-10-16T10:01:53.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.762+0000 7febfbe0c700 1 -- 172.21.15.158:0/1283675090 wait complete. 2023-10-16T10:01:53.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:00:32.442785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:00:32.961620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:01:53.777 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:01:53.816 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:53.815+0000 7f627c619200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:01:53.891 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:53.889+0000 7fd0111c9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:01:53.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.904+0000 7fb898929700 1 Processor -- start 2023-10-16T10:01:53.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.904+0000 7fb898929700 1 -- start start 2023-10-16T10:01:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb898929700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb8940b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb898929700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8940b3460 0x7fb8940b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb898929700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8940b4420 con 0x7fb89415b6b0 2023-10-16T10:01:53.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb898929700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89415ba90 con 0x7fb8940b3460 2023-10-16T10:01:53.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb898929700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940b38f0 con 0x7fb8940d0c60 2023-10-16T10:01:53.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb89415b6b0 0x7fb8940cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56180/0 (socket says 172.21.15.158:56180) 2023-10-16T10:01:53.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89259c700 1 -- 172.21.15.158:0/1419274402 learned_addr learned my addr 172.21.15.158:0/1419274402 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb892d9d700 1 --2- 172.21.15.158:0/1419274402 >> v2:172.21.15.172:3300/0 conn(0x7fb8940b3460 0x7fb8940b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb891d9b700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb8940b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221006110 0 0) 0x7fb8940b4420 con 0x7fb89415b6b0 2023-10-16T10:01:53.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb87c0035b0 con 0x7fb89415b6b0 2023-10-16T10:01:53.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826296431 0 0) 0x7fb87c0035b0 con 0x7fb89415b6b0 2023-10-16T10:01:53.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 msgr2=0x7fb8940b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:53.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb8940b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 >> v2:172.21.15.172:3300/0 conn(0x7fb8940b3460 msgr2=0x7fb8940b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:53.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 --2- 172.21.15.158:0/1419274402 >> v2:172.21.15.172:3300/0 conn(0x7fb8940b3460 0x7fb8940b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8940cbc20 con 0x7fb89415b6b0 2023-10-16T10:01:53.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fb884001670 con 0x7fb89415b6b0 2023-10-16T10:01:53.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb891d9b700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb8940b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:53.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb892d9d700 1 --2- 172.21.15.158:0/1419274402 >> v2:172.21.15.172:3300/0 conn(0x7fb8940b3460 0x7fb8940b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:53.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 >> v1:172.21.15.158:6789/0 conn(0x7fb89415b6b0 legacy=0x7fb8940cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb87c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c003f90 0x7fb87c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c004d90 0x7fb87c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940b38f0 con 0x7fb87c004d90 2023-10-16T10:01:53.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c00ba10 con 0x7fb87c003f90 2023-10-16T10:01:53.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c00bba0 con 0x7fb8940d0c60 2023-10-16T10:01:53.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb892d9d700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c004d90 0x7fb87c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.905+0000 7fb89259c700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb87c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb891d9b700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c003f90 0x7fb87c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb892d9d700 1 -- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 msgr2=0x7fb87c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:53.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb892d9d700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb87c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb892d9d700 1 -- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c003f90 msgr2=0x7fb87c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:53.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb892d9d700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c003f90 0x7fb87c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb892d9d700 1 -- 172.21.15.158:0/1419274402 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8880094a0 con 0x7fb87c004d90 2023-10-16T10:01:53.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb89259c700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8940d0c60 0x7fb87c003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:53.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb891d9b700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c003f90 0x7fb87c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:53.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb892d9d700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c004d90 0x7fb87c005210 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e350 tx=0x7fb87c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97687a263aebb9d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:53.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00edf0 con 0x7fb87c004d90 2023-10-16T10:01:53.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c00f040 con 0x7fb87c004d90 2023-10-16T10:01:53.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.906+0000 7fb89159a700 1 -- 172.21.15.158:0/1419274402 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0146a0 con 0x7fb87c004d90 2023-10-16T10:01:53.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.907+0000 7fb898929700 1 -- 172.21.15.158:0/1419274402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c004d90 msgr2=0x7fb87c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.907+0000 7fb898929700 1 --2- 172.21.15.158:0/1419274402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c004d90 0x7fb87c005210 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e350 tx=0x7fb87c005880 comp rx=0 tx=0).stop 2023-10-16T10:01:53.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.907+0000 7fb892d9d700 1 -- 172.21.15.158:0/1419274402 reap_dead start 2023-10-16T10:01:53.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.907+0000 7fb898929700 1 -- 172.21.15.158:0/1419274402 shutdown_connections 2023-10-16T10:01:53.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.907+0000 7fb898929700 1 -- 172.21.15.158:0/1419274402 >> 172.21.15.158:0/1419274402 conn(0x7fb8940cdf20 msgr2=0x7fb8940ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:53.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.907+0000 7fb898929700 1 -- 172.21.15.158:0/1419274402 shutdown_connections 2023-10-16T10:01:53.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.907+0000 7fb898929700 1 -- 172.21.15.158:0/1419274402 wait complete. 2023-10-16T10:01:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.908+0000 7fb898929700 1 Processor -- start 2023-10-16T10:01:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.908+0000 7fb898929700 1 -- start start 2023-10-16T10:01:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb898929700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c004d90 0x7fb8940ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb898929700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89414c6e0 0x7fb8940cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb898929700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb89406b9e0 0x7fb89406be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb898929700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940acfc0 con 0x7fb89406b9e0 2023-10-16T10:01:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb898929700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940acdd0 con 0x7fb87c004d90 2023-10-16T10:01:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb898929700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89414c450 con 0x7fb89414c6e0 2023-10-16T10:01:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb89406b9e0 0x7fb89406be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb89406b9e0 0x7fb89406be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56020/0 (socket says 172.21.15.158:56020) 2023-10-16T10:01:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 -- 172.21.15.158:0/60873658 learned_addr learned my addr 172.21.15.158:0/60873658 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb89259c700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c004d90 0x7fb8940ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb891d9b700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89414c6e0 0x7fb8940cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:53.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 -- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89414c6e0 msgr2=0x7fb8940cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89414c6e0 0x7fb8940cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 -- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c004d90 msgr2=0x7fb8940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c004d90 0x7fb8940ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:53.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb892d9d700 1 -- 172.21.15.158:0/60873658 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8940cb3d0 con 0x7fb89406b9e0 2023-10-16T10:01:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.909+0000 7fb89259c700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c004d90 0x7fb8940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.910+0000 7fb892d9d700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb89406b9e0 0x7fb89406be90 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb88c016000 tx=0x7fb88c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.910+0000 7fb882ffd700 1 -- 172.21.15.158:0/60873658 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c007000 con 0x7fb89406b9e0 2023-10-16T10:01:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.910+0000 7fb882ffd700 1 -- 172.21.15.158:0/60873658 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c009410 con 0x7fb89406b9e0 2023-10-16T10:01:53.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.910+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8940ae470 con 0x7fb89406b9e0 2023-10-16T10:01:53.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.910+0000 7fb882ffd700 1 -- 172.21.15.158:0/60873658 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c01d420 con 0x7fb89406b9e0 2023-10-16T10:01:53.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.910+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8940ae9f0 con 0x7fb89406b9e0 2023-10-16T10:01:53.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.911+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8940681b0 con 0x7fb89406b9e0 2023-10-16T10:01:53.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.911+0000 7fb882ffd700 1 -- 172.21.15.158:0/60873658 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb88c01d580 con 0x7fb89406b9e0 2023-10-16T10:01:53.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.912+0000 7fb882ffd700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 0x7fb87406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:53.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.912+0000 7fb89259c700 1 -- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 msgr2=0x7fb87406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:53.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.912+0000 7fb882ffd700 1 -- 172.21.15.158:0/60873658 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fb88c08b330 con 0x7fb89406b9e0 2023-10-16T10:01:53.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.912+0000 7fb89259c700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 0x7fb87406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:53.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:53.916+0000 7fb882ffd700 1 -- 172.21.15.158:0/60873658 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb88c055b40 con 0x7fb89406b9e0 2023-10-16T10:01:53.966 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:53.964+0000 7fd0111c9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:01:54.054 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:54.053+0000 7f627c619200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:01:54.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.113+0000 7fb89259c700 1 -- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 msgr2=0x7fb87406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:54.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.113+0000 7fb89259c700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 0x7fb87406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:54.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.116+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8940b4a90 con 0x7fb89406b9e0 2023-10-16T10:01:54.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.116+0000 7fb882ffd700 1 -- 172.21.15.158:0/60873658 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb88c059160 con 0x7fb89406b9e0 2023-10-16T10:01:54.118 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:01:54.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 msgr2=0x7fb87406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:54.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 0x7fb87406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb89406b9e0 msgr2=0x7fb89406be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb89406b9e0 0x7fb89406be90 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb88c016000 tx=0x7fb88c009750 comp rx=0 tx=0).stop 2023-10-16T10:01:54.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 shutdown_connections 2023-10-16T10:01:54.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb87406b140 0x7fb87406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb89406b9e0 0x7fb89406be90 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb87c004d90 0x7fb8940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 --2- 172.21.15.158:0/60873658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89414c6e0 0x7fb8940cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 >> 172.21.15.158:0/60873658 conn(0x7fb8940cdf20 msgr2=0x7fb8940b4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:54.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 shutdown_connections 2023-10-16T10:01:54.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.119+0000 7fb898929700 1 -- 172.21.15.158:0/60873658 wait complete. 2023-10-16T10:01:54.133 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:01:54.134 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:54.132+0000 7f627c619200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:01:54.230 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:54.229+0000 7fd0111c9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:01:54.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.262+0000 7f6b9ad96700 1 Processor -- start 2023-10-16T10:01:54.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.263+0000 7f6b9ad96700 1 -- start start 2023-10-16T10:01:54.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.263+0000 7f6b9ad96700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b9415dc40 0x7f6b9415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.263+0000 7f6b9ad96700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6b9406de70 0x7f6b9406f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.263+0000 7f6b9ad96700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9415f380 con 0x7f6b9406d9c0 2023-10-16T10:01:54.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.263+0000 7f6b9ad96700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9406e2e0 con 0x7f6b9415dc40 2023-10-16T10:01:54.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.263+0000 7f6b9ad96700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b94071e70 con 0x7f6b9406de70 2023-10-16T10:01:54.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b9415dc40 0x7f6b9415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b9415dc40 0x7f6b9415ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56026/0 (socket says 172.21.15.158:56026) 2023-10-16T10:01:54.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 -- 172.21.15.158:0/1371364982 learned_addr learned my addr 172.21.15.158:0/1371364982 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:54.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b93fff700 1 --1- 172.21.15.158:0/1371364982 >> v1:172.21.15.158:6789/0 conn(0x7f6b9406d9c0 0x7f6b9406f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56202/0 (socket says 172.21.15.158:56202) 2023-10-16T10:01:54.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b99333700 1 --2- 172.21.15.158:0/1371364982 >> v2:172.21.15.172:3300/0 conn(0x7f6b9406de70 0x7f6b9406f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 -- 172.21.15.158:0/1371364982 >> v2:172.21.15.172:3300/0 conn(0x7f6b9406de70 msgr2=0x7f6b9406f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:54.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 --2- 172.21.15.158:0/1371364982 >> v2:172.21.15.172:3300/0 conn(0x7f6b9406de70 0x7f6b9406f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 -- 172.21.15.158:0/1371364982 >> v1:172.21.15.158:6789/0 conn(0x7f6b9406d9c0 legacy=0x7f6b9406f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 -- 172.21.15.158:0/1371364982 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94072150 con 0x7f6b9415dc40 2023-10-16T10:01:54.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b98b32700 1 --2- 172.21.15.158:0/1371364982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b9415ee40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f6b940afe20 tx=0x7f6b88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbcac0d5894bcc08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:54.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b937fe700 1 -- 172.21.15.158:0/1371364982 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88011830 con 0x7f6b9415dc40 2023-10-16T10:01:54.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b937fe700 1 -- 172.21.15.158:0/1371364982 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b88011990 con 0x7f6b9415dc40 2023-10-16T10:01:54.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.264+0000 7f6b937fe700 1 -- 172.21.15.158:0/1371364982 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88011d20 con 0x7f6b9415dc40 2023-10-16T10:01:54.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 -- 172.21.15.158:0/1371364982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 msgr2=0x7f6b9415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/1371364982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b9415ee40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f6b940afe20 tx=0x7f6b88009bd0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 -- 172.21.15.158:0/1371364982 shutdown_connections 2023-10-16T10:01:54.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/1371364982 >> v2:172.21.15.172:3300/0 conn(0x7f6b9406de70 0x7f6b9406f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/1371364982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b9415ee40 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 -- 172.21.15.158:0/1371364982 >> 172.21.15.158:0/1371364982 conn(0x7f6b940d03b0 msgr2=0x7f6b940d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:54.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 -- 172.21.15.158:0/1371364982 shutdown_connections 2023-10-16T10:01:54.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.265+0000 7f6b9ad96700 1 -- 172.21.15.158:0/1371364982 wait complete. 2023-10-16T10:01:54.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.266+0000 7f6b9ad96700 1 Processor -- start 2023-10-16T10:01:54.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.266+0000 7f6b9ad96700 1 -- start start 2023-10-16T10:01:54.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b9ad96700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b9406d9c0 0x7f6b940cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b9ad96700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b9406de70 0x7f6b940c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b9ad96700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b940c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b9ad96700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b940c6380 con 0x7f6b9415dc40 2023-10-16T10:01:54.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b9ad96700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b94147da0 con 0x7f6b9406de70 2023-10-16T10:01:54.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b9ad96700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b94148080 con 0x7f6b9406d9c0 2023-10-16T10:01:54.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b99333700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b940c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b99333700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b940c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56036/0 (socket says 172.21.15.158:56036) 2023-10-16T10:01:54.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b99333700 1 -- 172.21.15.158:0/2326776119 learned_addr learned my addr 172.21.15.158:0/2326776119 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:54.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b93fff700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b9406de70 0x7f6b940c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.267+0000 7f6b98b32700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b9406d9c0 0x7f6b940cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b99333700 1 -- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b9406d9c0 msgr2=0x7f6b940cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b99333700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b9406d9c0 0x7f6b940cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b99333700 1 -- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b9406de70 msgr2=0x7f6b940c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b99333700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b9406de70 0x7f6b940c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b99333700 1 -- 172.21.15.158:0/2326776119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88009550 con 0x7f6b9415dc40 2023-10-16T10:01:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b99333700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b940c79c0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00efc0 tx=0x7f6b8c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:54.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b917fa700 1 -- 172.21.15.158:0/2326776119 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c009ea0 con 0x7f6b9415dc40 2023-10-16T10:01:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b917fa700 1 -- 172.21.15.158:0/2326776119 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c00f040 con 0x7f6b9415dc40 2023-10-16T10:01:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b94148360 con 0x7f6b9415dc40 2023-10-16T10:01:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b917fa700 1 -- 172.21.15.158:0/2326776119 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00ced0 con 0x7f6b9415dc40 2023-10-16T10:01:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.268+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b940c5fe0 con 0x7f6b9415dc40 2023-10-16T10:01:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.269+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b940681b0 con 0x7f6b9415dc40 2023-10-16T10:01:54.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.270+0000 7f6b917fa700 1 -- 172.21.15.158:0/2326776119 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f6b8c014480 con 0x7f6b9415dc40 2023-10-16T10:01:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.271+0000 7f6b917fa700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 0x7f6b7806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.271+0000 7f6b98b32700 1 -- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 msgr2=0x7f6b7806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.271+0000 7f6b917fa700 1 -- 172.21.15.158:0/2326776119 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f6b8c08ab70 con 0x7f6b9415dc40 2023-10-16T10:01:54.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.271+0000 7f6b98b32700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 0x7f6b7806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:54.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.275+0000 7f6b917fa700 1 -- 172.21.15.158:0/2326776119 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b8c055250 con 0x7f6b9415dc40 2023-10-16T10:01:54.412 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:54.411+0000 7f627c619200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:01:54.453 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:54.451+0000 7fd0111c9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:01:54.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.472+0000 7f6b98b32700 1 -- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 msgr2=0x7f6b7806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:54.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.472+0000 7f6b98b32700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 0x7f6b7806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:54.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.474+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b9406b760 con 0x7f6b9415dc40 2023-10-16T10:01:54.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.474+0000 7f6b917fa700 1 -- 172.21.15.158:0/2326776119 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b8c058870 con 0x7f6b9415dc40 2023-10-16T10:01:54.476 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:01:54.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 msgr2=0x7f6b7806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:54.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 0x7f6b7806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 msgr2=0x7f6b940c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b940c79c0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00efc0 tx=0x7f6b8c00c3a0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 shutdown_connections 2023-10-16T10:01:54.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f6b7806b140 0x7f6b7806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9415dc40 0x7f6b940c79c0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b9406d9c0 0x7f6b940cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 --2- 172.21.15.158:0/2326776119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b9406de70 0x7f6b940c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.476+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 >> 172.21.15.158:0/2326776119 conn(0x7f6b940d03b0 msgr2=0x7f6b940cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:54.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.477+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 shutdown_connections 2023-10-16T10:01:54.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.477+0000 7f6b9ad96700 1 -- 172.21.15.158:0/2326776119 wait complete. 2023-10-16T10:01:54.490 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:01:54.601 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:54.600+0000 7fd0111c9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:01:54.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.615+0000 7f132c5e8700 1 Processor -- start 2023-10-16T10:01:54.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.616+0000 7f132c5e8700 1 -- start start 2023-10-16T10:01:54.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.616+0000 7f132c5e8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f132415b590 0x7f13240ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.616+0000 7f132c5e8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13240cf4c0 con 0x7f13240cbaf0 2023-10-16T10:01:54.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132c5e8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132415f970 con 0x7f132415f1d0 2023-10-16T10:01:54.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132c5e8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13240b49e0 con 0x7f132415b590 2023-10-16T10:01:54.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132a384700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f132415b590 0x7f13240ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132a384700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f132415b590 0x7f13240ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56042/0 (socket says 172.21.15.158:56042) 2023-10-16T10:01:54.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132a384700 1 -- 172.21.15.158:0/4271516083 learned_addr learned my addr 172.21.15.158:0/4271516083 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:54.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132a384700 1 -- 172.21.15.158:0/4271516083 >> v1:172.21.15.172:6789/0 conn(0x7f132415f1d0 legacy=0x7f13240b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132a384700 1 -- 172.21.15.158:0/4271516083 >> v1:172.21.15.158:6789/0 conn(0x7f13240cbaf0 legacy=0x7f13240cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132a384700 1 -- 172.21.15.158:0/4271516083 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132415fc50 con 0x7f132415b590 2023-10-16T10:01:54.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f132a384700 1 --2- 172.21.15.158:0/4271516083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415b590 0x7f13240ce510 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f13140098f0 tx=0x7f1314009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebb56d9df9b6970 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f1329382700 1 -- 172.21.15.158:0/4271516083 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131400b8a0 con 0x7f132415b590 2023-10-16T10:01:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f1329382700 1 -- 172.21.15.158:0/4271516083 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131400ba00 con 0x7f132415b590 2023-10-16T10:01:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.617+0000 7f1329382700 1 -- 172.21.15.158:0/4271516083 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131400bd90 con 0x7f132415b590 2023-10-16T10:01:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.618+0000 7f132c5e8700 1 -- 172.21.15.158:0/4271516083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415b590 msgr2=0x7f13240ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.618+0000 7f132c5e8700 1 --2- 172.21.15.158:0/4271516083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415b590 0x7f13240ce510 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f13140098f0 tx=0x7f1314009c00 comp rx=0 tx=0).stop 2023-10-16T10:01:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.618+0000 7f132c5e8700 1 -- 172.21.15.158:0/4271516083 shutdown_connections 2023-10-16T10:01:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.618+0000 7f132c5e8700 1 --2- 172.21.15.158:0/4271516083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415b590 0x7f13240ce510 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.618+0000 7f132c5e8700 1 -- 172.21.15.158:0/4271516083 >> 172.21.15.158:0/4271516083 conn(0x7f132406a8d0 msgr2=0x7f132406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.618+0000 7f132c5e8700 1 -- 172.21.15.158:0/4271516083 shutdown_connections 2023-10-16T10:01:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.618+0000 7f132c5e8700 1 -- 172.21.15.158:0/4271516083 wait complete. 2023-10-16T10:01:54.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.619+0000 7f132c5e8700 1 Processor -- start 2023-10-16T10:01:54.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.619+0000 7f132c5e8700 1 -- start start 2023-10-16T10:01:54.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132c5e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13240cbaf0 0x7f13240c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132c5e8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f132415b590 0x7f13240b5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132c5e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415f1d0 0x7f13240b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132c5e8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132406b9e0 con 0x7f132415f1d0 2023-10-16T10:01:54.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132c5e8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240c6430 con 0x7f132415b590 2023-10-16T10:01:54.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132c5e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13241f3e80 con 0x7f13240cbaf0 2023-10-16T10:01:54.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415f1d0 0x7f13240b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415f1d0 0x7f13240b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56052/0 (socket says 172.21.15.158:56052) 2023-10-16T10:01:54.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 -- 172.21.15.158:0/2435375315 learned_addr learned my addr 172.21.15.158:0/2435375315 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:54.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f1329b83700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f132415b590 0x7f13240b5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132a384700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13240cbaf0 0x7f13240c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 -- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13240cbaf0 msgr2=0x7f13240c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13240cbaf0 0x7f13240c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 -- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f132415b590 msgr2=0x7f13240b5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f132415b590 0x7f13240b5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.620+0000 7f132ab85700 1 -- 172.21.15.158:0/2435375315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1314009550 con 0x7f132415f1d0 2023-10-16T10:01:54.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.621+0000 7f132ab85700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415f1d0 0x7f13240b61e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f131c00cb50 tx=0x7f131c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:54.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.621+0000 7f131affd700 1 -- 172.21.15.158:0/2435375315 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c003c00 con 0x7f132415f1d0 2023-10-16T10:01:54.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.621+0000 7f131affd700 1 -- 172.21.15.158:0/2435375315 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c003d60 con 0x7f132415f1d0 2023-10-16T10:01:54.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.621+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132406bbf0 con 0x7f132415f1d0 2023-10-16T10:01:54.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.621+0000 7f131affd700 1 -- 172.21.15.158:0/2435375315 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c00aa20 con 0x7f132415f1d0 2023-10-16T10:01:54.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.621+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132406c1b0 con 0x7f132415f1d0 2023-10-16T10:01:54.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.622+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13240681b0 con 0x7f132415f1d0 2023-10-16T10:01:54.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.623+0000 7f131affd700 1 -- 172.21.15.158:0/2435375315 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f131c00ab80 con 0x7f132415f1d0 2023-10-16T10:01:54.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.623+0000 7f131affd700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 0x7f130806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.623+0000 7f132a384700 1 -- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 msgr2=0x7f130806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:54.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.623+0000 7f131affd700 1 -- 172.21.15.158:0/2435375315 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f131c089db0 con 0x7f132415f1d0 2023-10-16T10:01:54.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.623+0000 7f132a384700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 0x7f130806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:54.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.627+0000 7f131affd700 1 -- 172.21.15.158:0/2435375315 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f131c054710 con 0x7f132415f1d0 2023-10-16T10:01:54.652 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:54.651+0000 7f627c619200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:01:54.720 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:54.719+0000 7fd0111c9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:01:54.810 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:54.809+0000 7f627c619200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:01:54.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.824+0000 7f132a384700 1 -- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 msgr2=0x7f130806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:54.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.824+0000 7f132a384700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 0x7f130806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:54.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.831+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13240b07c0 con 0x7f132415f1d0 2023-10-16T10:01:54.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.832+0000 7f131affd700 1 -- 172.21.15.158:0/2435375315 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f131c057d30 con 0x7f132415f1d0 2023-10-16T10:01:54.833 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:01:54.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 msgr2=0x7f130806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:54.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 0x7f130806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415f1d0 msgr2=0x7f13240b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415f1d0 0x7f13240b61e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f131c00cb50 tx=0x7f131c00ce60 comp rx=0 tx=0).stop 2023-10-16T10:01:54.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 shutdown_connections 2023-10-16T10:01:54.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7f130806b140 0x7f130806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f132415f1d0 0x7f13240b61e0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13240cbaf0 0x7f13240c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 --2- 172.21.15.158:0/2435375315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f132415b590 0x7f13240b5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 >> 172.21.15.158:0/2435375315 conn(0x7f132406a8d0 msgr2=0x7f13240cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:54.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 shutdown_connections 2023-10-16T10:01:54.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.834+0000 7f132c5e8700 1 -- 172.21.15.158:0/2435375315 wait complete. 2023-10-16T10:01:54.848 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:01:54.876 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:54.874+0000 7fd0111c9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:01:54.932 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:54.931+0000 7f627c619200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:01:54.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.970+0000 7fb80039d700 1 Processor -- start 2023-10-16T10:01:54.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.970+0000 7fb80039d700 1 -- start start 2023-10-16T10:01:54.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb80039d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb7f815dfb0 0x7fb7f806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb80039d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f806e770 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb80039d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f80cbe00 con 0x7fb7f806f340 2023-10-16T10:01:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb80039d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f80cf270 con 0x7fb7f815dfb0 2023-10-16T10:01:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb7fe139700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb7f80cc5d0 0x7fb7f80cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56234/0 (socket says 172.21.15.158:56234) 2023-10-16T10:01:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb7fe139700 1 -- 172.21.15.158:0/3755293328 learned_addr learned my addr 172.21.15.158:0/3755293328 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:54.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb7fd938700 1 --2- 172.21.15.158:0/3755293328 >> v2:172.21.15.172:3300/0 conn(0x7fb7f815dfb0 0x7fb7f806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734188519 0 0) 0x7fb7f4001110 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.971+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e40035b0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976439170 0 0) 0x7fb7f80cbe00 con 0x7fb7f806f340 2023-10-16T10:01:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f4001110 con 0x7fb7f806f340 2023-10-16T10:01:54.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd938700 1 -- 172.21.15.158:0/3755293328 >> v1:172.21.15.172:6789/0 conn(0x7fb7f806f340 legacy=0x7fb7f8071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 947810034 0 0) 0x7fb7e40035b0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd938700 1 -- 172.21.15.158:0/3755293328 >> v1:172.21.15.158:6789/0 conn(0x7fb7f80cc5d0 legacy=0x7fb7f80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd938700 1 -- 172.21.15.158:0/3755293328 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f815e4e0 con 0x7fb7f815dfb0 2023-10-16T10:01:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd938700 1 --2- 172.21.15.158:0/3755293328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f815dfb0 0x7fb7f806e230 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009840 tx=0x7fb7e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da56eac81747e7e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:54.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4001800 con 0x7fb7f815dfb0 2023-10-16T10:01:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e8004030 con 0x7fb7f815dfb0 2023-10-16T10:01:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.972+0000 7fb7fd137700 1 -- 172.21.15.158:0/3755293328 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e80043c0 con 0x7fb7f815dfb0 2023-10-16T10:01:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.973+0000 7fb80039d700 1 -- 172.21.15.158:0/3755293328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f815dfb0 msgr2=0x7fb7f806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.973+0000 7fb80039d700 1 --2- 172.21.15.158:0/3755293328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f815dfb0 0x7fb7f806e230 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009840 tx=0x7fb7e8009b50 comp rx=0 tx=0).stop 2023-10-16T10:01:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.973+0000 7fb80039d700 1 -- 172.21.15.158:0/3755293328 shutdown_connections 2023-10-16T10:01:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.973+0000 7fb80039d700 1 --2- 172.21.15.158:0/3755293328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f815dfb0 0x7fb7f806e230 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.973+0000 7fb80039d700 1 -- 172.21.15.158:0/3755293328 >> 172.21.15.158:0/3755293328 conn(0x7fb7f80d02b0 msgr2=0x7fb7f80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.973+0000 7fb80039d700 1 -- 172.21.15.158:0/3755293328 shutdown_connections 2023-10-16T10:01:54.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.973+0000 7fb80039d700 1 -- 172.21.15.158:0/3755293328 wait complete. 2023-10-16T10:01:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.974+0000 7fb80039d700 1 Processor -- start 2023-10-16T10:01:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.974+0000 7fb80039d700 1 -- start start 2023-10-16T10:01:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb80039d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f806f340 0x7fb7f80a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb80039d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7f80cc5d0 0x7fb7f81f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb80039d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7f815dfb0 0x7fb7f80b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb80039d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80a8dc0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb80039d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80a9010 con 0x7fb7f806f340 2023-10-16T10:01:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb80039d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80a92f0 con 0x7fb7f815dfb0 2023-10-16T10:01:54.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7f80cc5d0 0x7fb7f81f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7f80cc5d0 0x7fb7f81f2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56066/0 (socket says 172.21.15.158:56066) 2023-10-16T10:01:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 -- 172.21.15.158:0/3664076888 learned_addr learned my addr 172.21.15.158:0/3664076888 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fe139700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f806f340 0x7fb7f80a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fe93a700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7f815dfb0 0x7fb7f80b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 -- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7f815dfb0 msgr2=0x7fb7f80b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7f815dfb0 0x7fb7f80b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 -- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f806f340 msgr2=0x7fb7f80a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f806f340 0x7fb7f80a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 -- 172.21.15.158:0/3664076888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80094a0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7fd938700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7f80cc5d0 0x7fb7f81f2670 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800a010 tx=0x7fb7e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:54.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7eeffd700 1 -- 172.21.15.158:0/3664076888 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8003e60 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb7eeffd700 1 -- 172.21.15.158:0/3664076888 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e801f070 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.975+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f80a95d0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.976+0000 7fb7eeffd700 1 -- 172.21.15.158:0/3664076888 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8011870 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.976+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f81f2bb0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.976+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f80681b0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.977+0000 7fb7eeffd700 1 -- 172.21.15.158:0/3664076888 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb7e80119d0 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.977+0000 7fb7eeffd700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 0x7fb7dc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:54.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.977+0000 7fb7fe139700 1 -- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 msgr2=0x7fb7dc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.977+0000 7fb7eeffd700 1 -- 172.21.15.158:0/3664076888 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fb7e8021070 con 0x7fb7f80cc5d0 2023-10-16T10:01:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.977+0000 7fb7fe139700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 0x7fb7dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:54.982+0000 7fb7eeffd700 1 -- 172.21.15.158:0/3664076888 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb7e8055fd0 con 0x7fb7f80cc5d0 2023-10-16T10:01:55.097 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:55.096+0000 7f627c619200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:01:55.140 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:55.138+0000 7fd0111c9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:01:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.178+0000 7fb7fe139700 1 -- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 msgr2=0x7fb7dc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.178+0000 7fb7fe139700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 0x7fb7dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:55.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.228+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7f80b6230 con 0x7fb7f80cc5d0 2023-10-16T10:01:55.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.228+0000 7fb7eeffd700 1 -- 172.21.15.158:0/3664076888 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4927 (secure 0 0 0) 0x7fb7e80595f0 con 0x7fb7f80cc5d0 2023-10-16T10:01:55.230 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:55.230 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2674019415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2674019415},{"type":"v1","addr":"172.21.15.158:6836","nonce":2674019415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2425279024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2425279024},{"type":"v1","addr":"172.21.15.158:6837","nonce":2425279024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3002995342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3002995342},{"type":"v1","addr":"172.21.15.172:6834","nonce":3002995342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:00:34.013744+0000","modified":"2023-10-16T10:00:37.052969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7740":{"gid":7740,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1596548674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1596548674},{"type":"v1","addr":"172.21.15.172:6835","nonce":1596548674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T10:01:55.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 msgr2=0x7fb7dc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:55.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 0x7fb7dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7f80cc5d0 msgr2=0x7fb7f81f2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7f80cc5d0 0x7fb7f81f2670 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800a010 tx=0x7fb7e8004750 comp rx=0 tx=0).stop 2023-10-16T10:01:55.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 shutdown_connections 2023-10-16T10:01:55.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fb7dc06b140 0x7fb7dc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7f815dfb0 0x7fb7f80b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7f806f340 0x7fb7f80a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 --2- 172.21.15.158:0/3664076888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7f80cc5d0 0x7fb7f81f2670 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 >> 172.21.15.158:0/3664076888 conn(0x7fb7f80d02b0 msgr2=0x7fb7f815cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:55.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 shutdown_connections 2023-10-16T10:01:55.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.231+0000 7fb80039d700 1 -- 172.21.15.158:0/3664076888 wait complete. 2023-10-16T10:01:55.235 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 51 2023-10-16T10:01:55.246 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:01:55.300 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:55.298+0000 7fd0111c9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:01:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.370+0000 7fe065f4a700 1 Processor -- start 2023-10-16T10:01:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe065f4a700 1 -- start start 2023-10-16T10:01:55.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe065f4a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe06015daa0 0x7fe06006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe065f4a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d0bd0 0x7fe0600d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe065f4a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe0600d3570 0x7fe0600d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe065f4a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0600ce4d0 con 0x7fe06015daa0 2023-10-16T10:01:55.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe065f4a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0600d3f00 con 0x7fe0600d3570 2023-10-16T10:01:55.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe065f4a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06006c1b0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe05f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe06015daa0 0x7fe06006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.371+0000 7fe05f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe06015daa0 0x7fe06006b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56074/0 (socket says 172.21.15.158:56074) 2023-10-16T10:01:55.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05f7fe700 1 -- 172.21.15.158:0/354674187 learned_addr learned my addr 172.21.15.158:0/354674187 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:55.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05effd700 1 --2- 172.21.15.158:0/354674187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d0bd0 0x7fe0600d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05ffff700 1 --2- 172.21.15.158:0/354674187 >> v2:172.21.15.172:3300/0 conn(0x7fe0600d3570 0x7fe0600d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05f7fe700 1 -- 172.21.15.158:0/354674187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d0bd0 msgr2=0x7fe0600d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:55.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05f7fe700 1 --2- 172.21.15.158:0/354674187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d0bd0 0x7fe0600d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05f7fe700 1 -- 172.21.15.158:0/354674187 >> v2:172.21.15.172:3300/0 conn(0x7fe0600d3570 msgr2=0x7fe0600d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:55.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05f7fe700 1 --2- 172.21.15.158:0/354674187 >> v2:172.21.15.172:3300/0 conn(0x7fe0600d3570 0x7fe0600d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05f7fe700 1 -- 172.21.15.158:0/354674187 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06006c490 con 0x7fe06015daa0 2023-10-16T10:01:55.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05f7fe700 1 --2- 172.21.15.158:0/354674187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe06006b8d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fe06008ca90 tx=0x7fe05000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33e3ed20a265357 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:55.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05e7fc700 1 -- 172.21.15.158:0/354674187 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05000e070 con 0x7fe06015daa0 2023-10-16T10:01:55.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05e7fc700 1 -- 172.21.15.158:0/354674187 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05000b910 con 0x7fe06015daa0 2023-10-16T10:01:55.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.372+0000 7fe05e7fc700 1 -- 172.21.15.158:0/354674187 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05000bcc0 con 0x7fe06015daa0 2023-10-16T10:01:55.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 -- 172.21.15.158:0/354674187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 msgr2=0x7fe06006b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 --2- 172.21.15.158:0/354674187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe06006b8d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fe06008ca90 tx=0x7fe05000b330 comp rx=0 tx=0).stop 2023-10-16T10:01:55.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 -- 172.21.15.158:0/354674187 shutdown_connections 2023-10-16T10:01:55.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 --2- 172.21.15.158:0/354674187 >> v2:172.21.15.172:3300/0 conn(0x7fe0600d3570 0x7fe0600d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 --2- 172.21.15.158:0/354674187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe06006b8d0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 --2- 172.21.15.158:0/354674187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d0bd0 0x7fe0600d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 -- 172.21.15.158:0/354674187 >> 172.21.15.158:0/354674187 conn(0x7fe06008d040 msgr2=0x7fe06009c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:55.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 -- 172.21.15.158:0/354674187 shutdown_connections 2023-10-16T10:01:55.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.373+0000 7fe065f4a700 1 -- 172.21.15.158:0/354674187 wait complete. 2023-10-16T10:01:55.381 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:55.374+0000 7f627c619200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:01:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.375+0000 7fe065f4a700 1 Processor -- start 2023-10-16T10:01:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.375+0000 7fe065f4a700 1 -- start start 2023-10-16T10:01:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe065f4a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0600d0bd0 0x7fe0600c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe065f4a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d3570 0x7fe060160850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe065f4a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe0600b8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe065f4a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600b8100 con 0x7fe06015daa0 2023-10-16T10:01:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe065f4a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600b77d0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe065f4a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600b7ab0 con 0x7fe0600d3570 2023-10-16T10:01:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe05ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe0600b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe05ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe0600b8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56082/0 (socket says 172.21.15.158:56082) 2023-10-16T10:01:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe05ffff700 1 -- 172.21.15.158:0/3359458696 learned_addr learned my addr 172.21.15.158:0/3359458696 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe05f7fe700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0600d0bd0 0x7fe0600c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.376+0000 7fe05effd700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d3570 0x7fe060160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe05f7fe700 1 -- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d3570 msgr2=0x7fe060160850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe05f7fe700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d3570 0x7fe060160850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe05f7fe700 1 -- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 msgr2=0x7fe0600b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe05f7fe700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe0600b8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe05f7fe700 1 -- 172.21.15.158:0/3359458696 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe050009b30 con 0x7fe0600d0bd0 2023-10-16T10:01:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe05ffff700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe0600b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe05f7fe700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0600d0bd0 0x7fe0600c8ed0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe050015000 tx=0x7fe050004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe057fff700 1 -- 172.21.15.158:0/3359458696 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0500099b0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe057fff700 1 -- 172.21.15.158:0/3359458696 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05000ce80 con 0x7fe0600d0bd0 2023-10-16T10:01:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe057fff700 1 -- 172.21.15.158:0/3359458696 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050004470 con 0x7fe0600d0bd0 2023-10-16T10:01:55.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0600b7d90 con 0x7fe0600d0bd0 2023-10-16T10:01:55.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.377+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0600b8ad0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.378+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0600b8cc0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.378+0000 7fe057fff700 1 -- 172.21.15.158:0/3359458696 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe0500045d0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.379+0000 7fe057fff700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 0x7fe04006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.379+0000 7fe05effd700 1 -- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 msgr2=0x7fe04006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.379+0000 7fe057fff700 1 -- 172.21.15.158:0/3359458696 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fe05008b3d0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.379+0000 7fe05effd700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 0x7fe04006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:55.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.383+0000 7fe057fff700 1 -- 172.21.15.158:0/3359458696 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe050055bf0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.543 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:55.542+0000 7f627c619200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:01:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.580+0000 7fe05effd700 1 -- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 msgr2=0x7fe04006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.580+0000 7fe05effd700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 0x7fe04006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:55.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.586+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe06006bde0 con 0x7fe0600d0bd0 2023-10-16T10:01:55.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.588+0000 7fe057fff700 1 -- 172.21.15.158:0/3359458696 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+18086 (secure 0 0 0) 0x7fe050020090 con 0x7fe0600d0bd0 2023-10-16T10:01:55.590 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:55.590 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":114,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:01:49.664583+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:00:32.442785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T10:00:32.961620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.17","mappings":[{"from":1,"to":7}]},{"pgid":"14.2c","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/934459119":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/1239292381":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6833/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6832/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2498623628":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2203508586":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/521051078":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2930980190":"2023-10-17T10:00:47.874811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:01:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 msgr2=0x7fe04006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 0x7fe04006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0600d0bd0 msgr2=0x7fe0600c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0600d0bd0 0x7fe0600c8ed0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe050015000 tx=0x7fe050004ad0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 shutdown_connections 2023-10-16T10:01:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7fe04006b140 0x7fe04006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe06015daa0 0x7fe0600b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0600d0bd0 0x7fe0600c8ed0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 --2- 172.21.15.158:0/3359458696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0600d3570 0x7fe060160850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 >> 172.21.15.158:0/3359458696 conn(0x7fe06008d040 msgr2=0x7fe0600b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 shutdown_connections 2023-10-16T10:01:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.590+0000 7fe065f4a700 1 -- 172.21.15.158:0/3359458696 wait complete. 2023-10-16T10:01:55.605 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T10:01:55.605 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:01:55.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.727+0000 7feb08fbf700 1 Processor -- start 2023-10-16T10:01:55.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.727+0000 7feb08fbf700 1 -- start start 2023-10-16T10:01:55.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.728+0000 7feb08fbf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb040729b0 0x7feb0406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.728+0000 7feb08fbf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb04161270 con 0x7feb0406e5d0 2023-10-16T10:01:55.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.728+0000 7feb08fbf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb04161510 con 0x7feb0406ab40 2023-10-16T10:01:55.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.728+0000 7feb08fbf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb041617f0 con 0x7feb040729b0 2023-10-16T10:01:55.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.728+0000 7feb01d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7feb0406e5d0 0x7feb0415df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56266/0 (socket says 172.21.15.158:56266) 2023-10-16T10:01:55.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.728+0000 7feb01d9b700 1 -- 172.21.15.158:0/1650690792 learned_addr learned my addr 172.21.15.158:0/1650690792 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:55.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.728+0000 7feb02d9d700 1 --2- 172.21.15.158:0/1650690792 >> v2:172.21.15.172:3300/0 conn(0x7feb040729b0 0x7feb0406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250029197 0 0) 0x7feb04161270 con 0x7feb0406e5d0 2023-10-16T10:01:55.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec0035b0 con 0x7feb0406e5d0 2023-10-16T10:01:55.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188020773 0 0) 0x7feb04161510 con 0x7feb0406ab40 2023-10-16T10:01:55.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb04161270 con 0x7feb0406ab40 2023-10-16T10:01:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 457399044 0 0) 0x7feaec0035b0 con 0x7feb0406e5d0 2023-10-16T10:01:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 >> v2:172.21.15.172:3300/0 conn(0x7feb040729b0 msgr2=0x7feb0406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 --2- 172.21.15.158:0/1650690792 >> v2:172.21.15.172:3300/0 conn(0x7feb040729b0 0x7feb0406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 >> v1:172.21.15.172:6789/0 conn(0x7feb0406ab40 legacy=0x7feb040722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb04161ad0 con 0x7feb0406e5d0 2023-10-16T10:01:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7feaf8001670 con 0x7feb0406e5d0 2023-10-16T10:01:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 >> v1:172.21.15.158:6789/0 conn(0x7feb0406e5d0 legacy=0x7feb0415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feaec003b90 0x7feaec003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec004850 0x7feaec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feaec005210 0x7feaec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb041617f0 con 0x7feaec004850 2023-10-16T10:01:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaec00c090 con 0x7feaec003b90 2023-10-16T10:01:55.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaec00c250 con 0x7feaec005210 2023-10-16T10:01:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0259c700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feaec003b90 0x7feaec003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb01d9b700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec004850 0x7feaec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb02d9d700 1 --2- 172.21.15.158:0/1650690792 >> v2:172.21.15.172:3300/0 conn(0x7feb040729b0 0x7feb0406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0259c700 1 -- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feaec005210 msgr2=0x7feaec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0259c700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feaec005210 0x7feaec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0259c700 1 -- 172.21.15.158:0/1650690792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec004850 msgr2=0x7feaec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0259c700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec004850 0x7feaec004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb0259c700 1 -- 172.21.15.158:0/1650690792 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec0035b0 con 0x7feaec003b90 2023-10-16T10:01:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.729+0000 7feb02d9d700 1 -- 172.21.15.158:0/1650690792 reap_dead start 2023-10-16T10:01:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb0259c700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feaec003b90 0x7feaec003f90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feaf4001970 tx=0x7feaec005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e1b9fc496f145c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf40052d0 con 0x7feaec003b90 2023-10-16T10:01:55.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf4005470 con 0x7feaec003b90 2023-10-16T10:01:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb0159a700 1 -- 172.21.15.158:0/1650690792 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4006aa0 con 0x7feaec003b90 2023-10-16T10:01:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb08fbf700 1 -- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feaec003b90 msgr2=0x7feaec003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb08fbf700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feaec003b90 0x7feaec003f90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feaf4001970 tx=0x7feaec005880 comp rx=0 tx=0).stop 2023-10-16T10:01:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb08fbf700 1 -- 172.21.15.158:0/1650690792 shutdown_connections 2023-10-16T10:01:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb08fbf700 1 --2- 172.21.15.158:0/1650690792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feaec003b90 0x7feaec003f90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.730+0000 7feb08fbf700 1 -- 172.21.15.158:0/1650690792 >> 172.21.15.158:0/1650690792 conn(0x7feb040d02b0 msgr2=0x7feb040d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.731+0000 7feb08fbf700 1 -- 172.21.15.158:0/1650690792 shutdown_connections 2023-10-16T10:01:55.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.731+0000 7feb08fbf700 1 -- 172.21.15.158:0/1650690792 wait complete. 2023-10-16T10:01:55.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.732+0000 7feb08fbf700 1 Processor -- start 2023-10-16T10:01:55.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.732+0000 7feb08fbf700 1 -- start start 2023-10-16T10:01:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb08fbf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec003b90 0x7feb040b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb08fbf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04160850 0x7feb04154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb08fbf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040aea40 0x7feb040aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb08fbf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb040af400 con 0x7feaec003b90 2023-10-16T10:01:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb08fbf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb04160cd0 con 0x7feb040aea40 2023-10-16T10:01:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb08fbf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb040b46c0 con 0x7feb04160850 2023-10-16T10:01:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec003b90 0x7feb040b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec003b90 0x7feb040b5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56100/0 (socket says 172.21.15.158:56100) 2023-10-16T10:01:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 -- 172.21.15.158:0/2168982474 learned_addr learned my addr 172.21.15.158:0/2168982474 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:55.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb01d9b700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04160850 0x7feb04154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb02d9d700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040aea40 0x7feb040aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:55.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 -- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04160850 msgr2=0x7feb04154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04160850 0x7feb04154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 -- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040aea40 msgr2=0x7feb040aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040aea40 0x7feb040aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 -- 172.21.15.158:0/2168982474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb040b49a0 con 0x7feaec003b90 2023-10-16T10:01:55.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb01d9b700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04160850 0x7feb04154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:55.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb0259c700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec003b90 0x7feb040b5760 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7feaec00c220 tx=0x7feaf4001310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:55.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feaf2ffd700 1 -- 172.21.15.158:0/2168982474 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4001110 con 0x7feaec003b90 2023-10-16T10:01:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feaf2ffd700 1 -- 172.21.15.158:0/2168982474 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf400c610 con 0x7feaec003b90 2023-10-16T10:01:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb040b4c80 con 0x7feaec003b90 2023-10-16T10:01:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.733+0000 7feaf2ffd700 1 -- 172.21.15.158:0/2168982474 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf400c9c0 con 0x7feaec003b90 2023-10-16T10:01:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.734+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb040b5240 con 0x7feaec003b90 2023-10-16T10:01:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.734+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb040b5400 con 0x7feaec003b90 2023-10-16T10:01:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.735+0000 7feaf2ffd700 1 -- 172.21.15.158:0/2168982474 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 59) v1 ==== 89445+0+0 (secure 0 0 0) 0x7feaf400cb20 con 0x7feaec003b90 2023-10-16T10:01:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.735+0000 7feaf2ffd700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 0x7feae406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.735+0000 7feb01d9b700 1 -- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 msgr2=0x7feae406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.735+0000 7feaf2ffd700 1 -- 172.21.15.158:0/2168982474 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6913+0+0 (secure 0 0 0) 0x7feaf408ab10 con 0x7feaec003b90 2023-10-16T10:01:55.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.735+0000 7feb01d9b700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 0x7feae406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:01:55.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.740+0000 7feaf2ffd700 1 -- 172.21.15.158:0/2168982474 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feaf4055500 con 0x7feaec003b90 2023-10-16T10:01:55.828 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:01:55.827+0000 7fd0111c9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:01:55.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.936+0000 7feb01d9b700 1 -- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 msgr2=0x7feae406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1516256968 2023-10-16T10:01:55.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.936+0000 7feb01d9b700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 0x7feae406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:01:55.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.984+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb040b55b0 con 0x7feaec003b90 2023-10-16T10:01:55.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.984+0000 7feaf2ffd700 1 -- 172.21.15.158:0/2168982474 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4927 (secure 0 0 0) 0x7feaf4058b20 con 0x7feaec003b90 2023-10-16T10:01:55.986 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:55.986 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2674019415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2674019415},{"type":"v1","addr":"172.21.15.158:6836","nonce":2674019415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2425279024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2425279024},{"type":"v1","addr":"172.21.15.158:6837","nonce":2425279024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3002995342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3002995342},{"type":"v1","addr":"172.21.15.172:6834","nonce":3002995342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:00:34.013744+0000","modified":"2023-10-16T10:00:37.052969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7740":{"gid":7740,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1596548674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1596548674},{"type":"v1","addr":"172.21.15.172:6835","nonce":1596548674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T10:01:55.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.986+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 msgr2=0x7feae406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:01:55.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 0x7feae406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec003b90 msgr2=0x7feb040b5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:55.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec003b90 0x7feb040b5760 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7feaec00c220 tx=0x7feaf4001310 comp rx=0 tx=0).stop 2023-10-16T10:01:55.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 shutdown_connections 2023-10-16T10:01:55.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:6832/1516256968,v1:172.21.15.158:6833/1516256968] conn(0x7feae406b140 0x7feae406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040aea40 0x7feb040aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feaec003b90 0x7feb040b5760 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 --2- 172.21.15.158:0/2168982474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04160850 0x7feb04154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:55.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 >> 172.21.15.158:0/2168982474 conn(0x7feb040d02b0 msgr2=0x7feb040b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:55.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 shutdown_connections 2023-10-16T10:01:55.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:55.987+0000 7feb08fbf700 1 -- 172.21.15.158:0/2168982474 wait complete. 2023-10-16T10:01:55.991 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 51 2023-10-16T10:01:56.001 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:01:56.104 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.103+0000 7f627c619200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:01:56.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.124+0000 7ff5100e7700 1 Processor -- start 2023-10-16T10:01:56.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.124+0000 7ff5100e7700 1 -- start start 2023-10-16T10:01:56.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.125+0000 7ff5100e7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff5080caf00 0x7ff50815dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.125+0000 7ff5100e7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50815b9b0 con 0x7ff50806c550 2023-10-16T10:01:56.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.125+0000 7ff5100e7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50815e4e0 con 0x7ff5080ce710 2023-10-16T10:01:56.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.125+0000 7ff5100e7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50815e7b0 con 0x7ff5080caf00 2023-10-16T10:01:56.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.125+0000 7ff50e684700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff50806c550 0x7ff5080cccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56290/0 (socket says 172.21.15.158:56290) 2023-10-16T10:01:56.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50e684700 1 -- 172.21.15.158:0/2237074201 learned_addr learned my addr 172.21.15.158:0/2237074201 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:56.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50de83700 1 --2- 172.21.15.158:0/2237074201 >> v2:172.21.15.172:3300/0 conn(0x7ff5080caf00 0x7ff50815dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50ce81700 1 -- 172.21.15.158:0/2237074201 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879002362 0 0) 0x7ff50815e4e0 con 0x7ff5080ce710 2023-10-16T10:01:56.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50ce81700 1 -- 172.21.15.158:0/2237074201 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f40035b0 con 0x7ff5080ce710 2023-10-16T10:01:56.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50de83700 1 -- 172.21.15.158:0/2237074201 >> v1:172.21.15.172:6789/0 conn(0x7ff5080ce710 legacy=0x7ff50815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50de83700 1 -- 172.21.15.158:0/2237074201 >> v1:172.21.15.158:6789/0 conn(0x7ff50806c550 legacy=0x7ff5080cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50de83700 1 -- 172.21.15.158:0/2237074201 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50815ea90 con 0x7ff5080caf00 2023-10-16T10:01:56.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.126+0000 7ff50de83700 1 --2- 172.21.15.158:0/2237074201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff50815dfa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff500009d90 tx=0x7ff50000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=285b27804f9097b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:56.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.127+0000 7ff50ce81700 1 -- 172.21.15.158:0/2237074201 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50000e070 con 0x7ff5080caf00 2023-10-16T10:01:56.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.127+0000 7ff50ce81700 1 -- 172.21.15.158:0/2237074201 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50000bb20 con 0x7ff5080caf00 2023-10-16T10:01:56.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.127+0000 7ff50ce81700 1 -- 172.21.15.158:0/2237074201 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500012430 con 0x7ff5080caf00 2023-10-16T10:01:56.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.127+0000 7ff5100e7700 1 -- 172.21.15.158:0/2237074201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 msgr2=0x7ff50815dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.127+0000 7ff5100e7700 1 --2- 172.21.15.158:0/2237074201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff50815dfa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff500009d90 tx=0x7ff50000b330 comp rx=0 tx=0).stop 2023-10-16T10:01:56.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.128+0000 7ff5100e7700 1 -- 172.21.15.158:0/2237074201 shutdown_connections 2023-10-16T10:01:56.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.128+0000 7ff5100e7700 1 --2- 172.21.15.158:0/2237074201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff50815dfa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.128+0000 7ff5100e7700 1 -- 172.21.15.158:0/2237074201 >> 172.21.15.158:0/2237074201 conn(0x7ff50806af00 msgr2=0x7ff5080cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:56.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.128+0000 7ff5100e7700 1 -- 172.21.15.158:0/2237074201 shutdown_connections 2023-10-16T10:01:56.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.129+0000 7ff5100e7700 1 -- 172.21.15.158:0/2237074201 wait complete. 2023-10-16T10:01:56.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.130+0000 7ff5100e7700 1 Processor -- start 2023-10-16T10:01:56.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.130+0000 7ff5100e7700 1 -- start start 2023-10-16T10:01:56.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff5100e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50806c550 0x7ff508147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff5100e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff5080c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff5100e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5080ce710 0x7ff5080c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff5100e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080c54d0 con 0x7ff5080ce710 2023-10-16T10:01:56.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff5100e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080c4ba0 con 0x7ff5080caf00 2023-10-16T10:01:56.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff5100e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080c4e80 con 0x7ff50806c550 2023-10-16T10:01:56.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff50e684700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5080ce710 0x7ff5080c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff50de83700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50806c550 0x7ff508147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff50d682700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff5080c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff50e684700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5080ce710 0x7ff5080c2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56128/0 (socket says 172.21.15.158:56128) 2023-10-16T10:01:56.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.131+0000 7ff50e684700 1 -- 172.21.15.158:0/1953331548 learned_addr learned my addr 172.21.15.158:0/1953331548 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:56.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff50e684700 1 -- 172.21.15.158:0/1953331548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50806c550 msgr2=0x7ff508147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff50e684700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50806c550 0x7ff508147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff50e684700 1 -- 172.21.15.158:0/1953331548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 msgr2=0x7ff5080c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff50e684700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff5080c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff50e684700 1 -- 172.21.15.158:0/1953331548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff500009a70 con 0x7ff5080ce710 2023-10-16T10:01:56.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff50d682700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff5080c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:01:56.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff50e684700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5080ce710 0x7ff5080c2c00 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c400 tx=0x7ff50400c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:56.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff4fe7fc700 1 -- 172.21.15.158:0/1953331548 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504005d70 con 0x7ff5080ce710 2023-10-16T10:01:56.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff4fe7fc700 1 -- 172.21.15.158:0/1953331548 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50400ce10 con 0x7ff5080ce710 2023-10-16T10:01:56.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff4fe7fc700 1 -- 172.21.15.158:0/1953331548 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5040072d0 con 0x7ff5080ce710 2023-10-16T10:01:56.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5080c5160 con 0x7ff5080ce710 2023-10-16T10:01:56.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.132+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080c3140 con 0x7ff5080ce710 2023-10-16T10:01:56.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.133+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5080681b0 con 0x7ff5080ce710 2023-10-16T10:01:56.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.134+0000 7ff4fe7fc700 1 -- 172.21.15.158:0/1953331548 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 60) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff504007430 con 0x7ff5080ce710 2023-10-16T10:01:56.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.134+0000 7ff4fe7fc700 1 -- 172.21.15.158:0/1953331548 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 7343+0+0 (secure 0 0 0) 0x7ff504088850 con 0x7ff5080ce710 2023-10-16T10:01:56.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.139+0000 7ff4fe7fc700 1 -- 172.21.15.158:0/1953331548 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5040537a0 con 0x7ff5080ce710 2023-10-16T10:01:56.189 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.187+0000 7f61f7e05700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.189 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.187+0000 7f61f7e05700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.189 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.187+0000 7f61f7e05700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.189 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.187+0000 7f61f7e05700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.189 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.187+0000 7f61f7e05700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.190 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.188+0000 7f61f4dff700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.190 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.188+0000 7f61f4dff700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.190 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.188+0000 7f61f4dff700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.190 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.188+0000 7f61f4dff700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.191 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:01:56.188+0000 7f61f4dff700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:01:56.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.390+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5080c5610 con 0x7ff5080ce710 2023-10-16T10:01:56.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.390+0000 7ff4fe7fc700 1 -- 172.21.15.158:0/1953331548 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4927 (secure 0 0 0) 0x7ff504056dc0 con 0x7ff5080ce710 2023-10-16T10:01:56.392 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:56.392 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2674019415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2674019415},{"type":"v1","addr":"172.21.15.158:6836","nonce":2674019415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2425279024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2425279024},{"type":"v1","addr":"172.21.15.158:6837","nonce":2425279024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3002995342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3002995342},{"type":"v1","addr":"172.21.15.172:6834","nonce":3002995342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:00:34.013744+0000","modified":"2023-10-16T10:00:37.052969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7740":{"gid":7740,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1596548674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1596548674},{"type":"v1","addr":"172.21.15.172:6835","nonce":1596548674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T10:01:56.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5080ce710 msgr2=0x7ff5080c2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5080ce710 0x7ff5080c2c00 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c400 tx=0x7ff50400c710 comp rx=0 tx=0).stop 2023-10-16T10:01:56.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 shutdown_connections 2023-10-16T10:01:56.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5080ce710 0x7ff5080c2c00 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff50806c550 0x7ff508147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 --2- 172.21.15.158:0/1953331548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5080caf00 0x7ff5080c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 >> 172.21.15.158:0/1953331548 conn(0x7ff50806af00 msgr2=0x7ff508072150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:56.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 shutdown_connections 2023-10-16T10:01:56.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.393+0000 7ff5100e7700 1 -- 172.21.15.158:0/1953331548 wait complete. 2023-10-16T10:01:56.396 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 51 2023-10-16T10:01:56.406 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:01:56.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.526+0000 7f6c0e667700 1 Processor -- start 2023-10-16T10:01:56.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.526+0000 7f6c0e667700 1 -- start start 2023-10-16T10:01:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c0e667700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c0806b0a0 0x7f6c0815f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c0e667700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c080cb010 0x7f6c0815bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c0e667700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c08072700 con 0x7f6c0806d820 2023-10-16T10:01:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c0e667700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c08071910 con 0x7f6c0806b0a0 2023-10-16T10:01:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c0e667700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0806b480 con 0x7f6c080cb010 2023-10-16T10:01:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c07fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c0806b0a0 0x7f6c0815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c077fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6c0806d820 0x7f6c080ca900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56374/0 (socket says 172.21.15.158:56374) 2023-10-16T10:01:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c077fe700 1 -- 172.21.15.158:0/4166880256 learned_addr learned my addr 172.21.15.158:0/4166880256 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c0cc04700 1 --2- 172.21.15.158:0/4166880256 >> v2:172.21.15.172:3300/0 conn(0x7f6c080cb010 0x7f6c0815bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c07fff700 1 -- 172.21.15.158:0/4166880256 >> v2:172.21.15.172:3300/0 conn(0x7f6c080cb010 msgr2=0x7f6c0815bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c07fff700 1 --2- 172.21.15.158:0/4166880256 >> v2:172.21.15.172:3300/0 conn(0x7f6c080cb010 0x7f6c0815bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c07fff700 1 -- 172.21.15.158:0/4166880256 >> v1:172.21.15.158:6789/0 conn(0x7f6c0806d820 legacy=0x7f6c080ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c06ffd700 1 -- 172.21.15.158:0/4166880256 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377001455 0 0) 0x7f6c08072700 con 0x7f6c0806d820 2023-10-16T10:01:56.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c07fff700 1 -- 172.21.15.158:0/4166880256 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c080cca90 con 0x7f6c0806b0a0 2023-10-16T10:01:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.527+0000 7f6c07fff700 1 --2- 172.21.15.158:0/4166880256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806b0a0 0x7f6c0815f010 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f6c08074dd0 tx=0x7f6bfc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2de13b7984c0254 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c06ffd700 1 -- 172.21.15.158:0/4166880256 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc011830 con 0x7f6c0806b0a0 2023-10-16T10:01:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c06ffd700 1 -- 172.21.15.158:0/4166880256 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc011990 con 0x7f6c0806b0a0 2023-10-16T10:01:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c06ffd700 1 -- 172.21.15.158:0/4166880256 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc011d20 con 0x7f6c0806b0a0 2023-10-16T10:01:56.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 -- 172.21.15.158:0/4166880256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806b0a0 msgr2=0x7f6c0815f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 --2- 172.21.15.158:0/4166880256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806b0a0 0x7f6c0815f010 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f6c08074dd0 tx=0x7f6bfc009b20 comp rx=0 tx=0).stop 2023-10-16T10:01:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 -- 172.21.15.158:0/4166880256 shutdown_connections 2023-10-16T10:01:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 --2- 172.21.15.158:0/4166880256 >> v2:172.21.15.172:3300/0 conn(0x7f6c080cb010 0x7f6c0815bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 --2- 172.21.15.158:0/4166880256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806b0a0 0x7f6c0815f010 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 -- 172.21.15.158:0/4166880256 >> 172.21.15.158:0/4166880256 conn(0x7f6c080d02b0 msgr2=0x7f6c080d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 -- 172.21.15.158:0/4166880256 shutdown_connections 2023-10-16T10:01:56.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.528+0000 7f6c0e667700 1 -- 172.21.15.158:0/4166880256 wait complete. 2023-10-16T10:01:56.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.529+0000 7f6c0e667700 1 Processor -- start 2023-10-16T10:01:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c0e667700 1 -- start start 2023-10-16T10:01:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c0e667700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c0806b0a0 0x7f6c080c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c0e667700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806d820 0x7f6c081f0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c0e667700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c080cb010 0x7f6c081f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c0e667700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0815e650 con 0x7f6c0806d820 2023-10-16T10:01:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c0e667700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0815e8c0 con 0x7f6c0806b0a0 2023-10-16T10:01:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c0e667700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0815eba0 con 0x7f6c080cb010 2023-10-16T10:01:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806d820 0x7f6c081f0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806d820 0x7f6c081f0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56202/0 (socket says 172.21.15.158:56202) 2023-10-16T10:01:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.530+0000 7f6c077fe700 1 -- 172.21.15.158:0/2348843064 learned_addr learned my addr 172.21.15.158:0/2348843064 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c0cc04700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c080cb010 0x7f6c081f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c07fff700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c0806b0a0 0x7f6c080c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c077fe700 1 -- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c080cb010 msgr2=0x7f6c081f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c077fe700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c080cb010 0x7f6c081f0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c077fe700 1 -- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c0806b0a0 msgr2=0x7f6c080c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c077fe700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c0806b0a0 0x7f6c080c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c077fe700 1 -- 172.21.15.158:0/2348843064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc0094a0 con 0x7f6c0806d820 2023-10-16T10:01:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c07fff700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c0806b0a0 0x7f6c080c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:01:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c077fe700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806d820 0x7f6c081f0150 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800c540 tx=0x7f6bf800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c04ff9700 1 -- 172.21.15.158:0/2348843064 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8004c80 con 0x7f6c0806d820 2023-10-16T10:01:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c04ff9700 1 -- 172.21.15.158:0/2348843064 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf8004de0 con 0x7f6c0806d820 2023-10-16T10:01:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0815ee80 con 0x7f6c0806d820 2023-10-16T10:01:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c04ff9700 1 -- 172.21.15.158:0/2348843064 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8007310 con 0x7f6c0806d820 2023-10-16T10:01:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.531+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c080c6c80 con 0x7f6c0806d820 2023-10-16T10:01:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.532+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0815f070 con 0x7f6c0806d820 2023-10-16T10:01:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.532+0000 7f6c04ff9700 1 -- 172.21.15.158:0/2348843064 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 60) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f6bf8007470 con 0x7f6c0806d820 2023-10-16T10:01:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.533+0000 7f6c04ff9700 1 -- 172.21.15.158:0/2348843064 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 7343+0+0 (secure 0 0 0) 0x7f6bf8089460 con 0x7f6c0806d820 2023-10-16T10:01:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.537+0000 7f6c04ff9700 1 -- 172.21.15.158:0/2348843064 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6bf8054300 con 0x7f6c0806d820 2023-10-16T10:01:56.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.731+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c080681b0 con 0x7f6c0806d820 2023-10-16T10:01:56.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.733+0000 7f6c04ff9700 1 -- 172.21.15.158:0/2348843064 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+18722 (secure 0 0 0) 0x7f6bf8057920 con 0x7f6c0806d820 2023-10-16T10:01:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:01:56.734 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":115,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:01:56.116887+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:00:32.442785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T10:00:32.961620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.17","mappings":[{"from":1,"to":7}]},{"pgid":"14.2c","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/1516256968":"2023-10-17T10:01:56.116841+0000","172.21.15.158:6832/1516256968":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3809361082":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3884887886":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/1766729069":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/2379821640":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3839669547":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3538867905":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3020719129":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/934459119":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/1239292381":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6833/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6832/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2498623628":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2203508586":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2678240587":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/521051078":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2930980190":"2023-10-17T10:00:47.874811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:01:56.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806d820 msgr2=0x7f6c081f0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806d820 0x7f6c081f0150 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800c540 tx=0x7f6bf800c900 comp rx=0 tx=0).stop 2023-10-16T10:01:56.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 shutdown_connections 2023-10-16T10:01:56.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c080cb010 0x7f6c081f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c0806b0a0 0x7f6c080c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 --2- 172.21.15.158:0/2348843064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0806d820 0x7f6c081f0150 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 >> 172.21.15.158:0/2348843064 conn(0x7f6c080d02b0 msgr2=0x7f6c080b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:56.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 shutdown_connections 2023-10-16T10:01:56.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.735+0000 7f6c0e667700 1 -- 172.21.15.158:0/2348843064 wait complete. 2023-10-16T10:01:56.749 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T10:01:56.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.869+0000 7fae08bcb700 1 Processor -- start 2023-10-16T10:01:56.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.869+0000 7fae08bcb700 1 -- start start 2023-10-16T10:01:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae08bcb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fae0406a8d0 0x7fae0406acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae08bcb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae040afac0 0x7fae04161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae08bcb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0406b1f0 con 0x7fae040cfeb0 2023-10-16T10:01:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae08bcb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0415dd30 con 0x7fae040afac0 2023-10-16T10:01:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae08bcb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae040b3f20 con 0x7fae0406a8d0 2023-10-16T10:01:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae01d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae040afac0 0x7fae04161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae02d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fae040cfeb0 0x7fae04071a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56388/0 (socket says 172.21.15.158:56388) 2023-10-16T10:01:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.870+0000 7fae02d9d700 1 -- 172.21.15.158:0/732193154 learned_addr learned my addr 172.21.15.158:0/732193154 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae0259c700 1 --2- 172.21.15.158:0/732193154 >> v2:172.21.15.172:3300/0 conn(0x7fae0406a8d0 0x7fae0406acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae01d9b700 1 -- 172.21.15.158:0/732193154 >> v2:172.21.15.172:3300/0 conn(0x7fae0406a8d0 msgr2=0x7fae0406acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae01d9b700 1 --2- 172.21.15.158:0/732193154 >> v2:172.21.15.172:3300/0 conn(0x7fae0406a8d0 0x7fae0406acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae01d9b700 1 -- 172.21.15.158:0/732193154 >> v1:172.21.15.158:6789/0 conn(0x7fae040cfeb0 legacy=0x7fae04071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae0159a700 1 -- 172.21.15.158:0/732193154 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698791200 0 0) 0x7fae0406b1f0 con 0x7fae040cfeb0 2023-10-16T10:01:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae01d9b700 1 -- 172.21.15.158:0/732193154 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae040b4200 con 0x7fae040afac0 2023-10-16T10:01:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae01d9b700 1 --2- 172.21.15.158:0/732193154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040afac0 0x7fae04161d00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fae040690d0 tx=0x7fadec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f82861366596d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae0159a700 1 -- 172.21.15.158:0/732193154 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec011830 con 0x7fae040afac0 2023-10-16T10:01:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae0159a700 1 -- 172.21.15.158:0/732193154 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec011990 con 0x7fae040afac0 2023-10-16T10:01:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.871+0000 7fae0159a700 1 -- 172.21.15.158:0/732193154 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec011d20 con 0x7fae040afac0 2023-10-16T10:01:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 -- 172.21.15.158:0/732193154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040afac0 msgr2=0x7fae04161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 --2- 172.21.15.158:0/732193154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040afac0 0x7fae04161d00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fae040690d0 tx=0x7fadec009b20 comp rx=0 tx=0).stop 2023-10-16T10:01:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 -- 172.21.15.158:0/732193154 shutdown_connections 2023-10-16T10:01:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 --2- 172.21.15.158:0/732193154 >> v2:172.21.15.172:3300/0 conn(0x7fae0406a8d0 0x7fae0406acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 --2- 172.21.15.158:0/732193154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040afac0 0x7fae04161d00 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 -- 172.21.15.158:0/732193154 >> 172.21.15.158:0/732193154 conn(0x7fae0406da40 msgr2=0x7fae0406de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 -- 172.21.15.158:0/732193154 shutdown_connections 2023-10-16T10:01:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.872+0000 7fae08bcb700 1 -- 172.21.15.158:0/732193154 wait complete. 2023-10-16T10:01:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.873+0000 7fae08bcb700 1 Processor -- start 2023-10-16T10:01:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.873+0000 7fae08bcb700 1 -- start start 2023-10-16T10:01:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae08bcb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae0406a8d0 0x7fae041f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae08bcb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae040afac0 0x7fae04160550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae08bcb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040cfeb0 0x7fae0415a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae08bcb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae041f4d10 con 0x7fae040cfeb0 2023-10-16T10:01:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae08bcb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae041f4ff0 con 0x7fae040afac0 2023-10-16T10:01:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae08bcb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae041f52d0 con 0x7fae0406a8d0 2023-10-16T10:01:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040cfeb0 0x7fae0415a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040cfeb0 0x7fae0415a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56218/0 (socket says 172.21.15.158:56218) 2023-10-16T10:01:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 -- 172.21.15.158:0/604650349 learned_addr learned my addr 172.21.15.158:0/604650349 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae01d9b700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae040afac0 0x7fae04160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae0259c700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae0406a8d0 0x7fae041f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 -- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae0406a8d0 msgr2=0x7fae041f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae0406a8d0 0x7fae041f5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 -- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae040afac0 msgr2=0x7fae04160550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae040afac0 0x7fae04160550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 -- 172.21.15.158:0/604650349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0094a0 con 0x7fae040cfeb0 2023-10-16T10:01:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae02d9d700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040cfeb0 0x7fae0415a500 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00cba0 tx=0x7fadfc00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fadfaffd700 1 -- 172.21.15.158:0/604650349 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc00aaa0 con 0x7fae040cfeb0 2023-10-16T10:01:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fadfaffd700 1 -- 172.21.15.158:0/604650349 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc00ac20 con 0x7fae040cfeb0 2023-10-16T10:01:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae041f55b0 con 0x7fae040cfeb0 2023-10-16T10:01:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.874+0000 7fadfaffd700 1 -- 172.21.15.158:0/604650349 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0070d0 con 0x7fae040cfeb0 2023-10-16T10:01:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.875+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae04160a90 con 0x7fae040cfeb0 2023-10-16T10:01:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.875+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae041f5ad0 con 0x7fae040cfeb0 2023-10-16T10:01:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.876+0000 7fadfaffd700 1 -- 172.21.15.158:0/604650349 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 60) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fadfc007230 con 0x7fae040cfeb0 2023-10-16T10:01:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.876+0000 7fadfaffd700 1 -- 172.21.15.158:0/604650349 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 7343+0+0 (secure 0 0 0) 0x7fadfc0894d0 con 0x7fae040cfeb0 2023-10-16T10:01:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:56.881+0000 7fadfaffd700 1 -- 172.21.15.158:0/604650349 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae041f5ad0 con 0x7fae040cfeb0 2023-10-16T10:01:57.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.118+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fae041f5ad0 con 0x7fae040cfeb0 2023-10-16T10:01:57.119 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T10:01:57.118+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T10:01:57.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.120+0000 7fadfaffd700 1 -- 172.21.15.158:0/604650349 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) v1 ==== 161+0+0 (secure 0 0 0) 0x7fae041f5ad0 con 0x7fae040cfeb0 2023-10-16T10:01:57.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040cfeb0 msgr2=0x7fae0415a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040cfeb0 0x7fae0415a500 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00cba0 tx=0x7fadfc00ceb0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 shutdown_connections 2023-10-16T10:01:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae040cfeb0 0x7fae0415a500 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae0406a8d0 0x7fae041f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 --2- 172.21.15.158:0/604650349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae040afac0 0x7fae04160550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 >> 172.21.15.158:0/604650349 conn(0x7fae0406da40 msgr2=0x7fae040b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 shutdown_connections 2023-10-16T10:01:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.123+0000 7fae08bcb700 1 -- 172.21.15.158:0/604650349 wait complete. 2023-10-16T10:01:57.127 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T10:01:57.129 INFO:tasks.ceph.mds.b.smithi172.stderr: -8572> 2023-10-16T10:00:36.596+0000 7fdcab5c2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:01:57.140 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T10:01:57.168 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T10:01:57.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.262+0000 7fe452c27700 1 Processor -- start 2023-10-16T10:01:57.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.262+0000 7fe452c27700 1 -- start start 2023-10-16T10:01:57.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe452c27700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:57.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe452c27700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe44c15deb0 0x7fe44c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:57.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe452c27700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44c0d0ae0 con 0x7fe44c0cea20 2023-10-16T10:01:57.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe452c27700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44c0d12d0 con 0x7fe44c15deb0 2023-10-16T10:01:57.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe452c27700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c0cbdf0 con 0x7fe44c15dad0 2023-10-16T10:01:57.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe44bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe44c15deb0 0x7fe44c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:57.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe44bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe44c15deb0 0x7fe44c0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56246/0 (socket says 172.21.15.158:56246) 2023-10-16T10:01:57.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe44bfff700 1 -- 172.21.15.158:0/2851867893 learned_addr learned my addr 172.21.15.158:0/2851867893 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:57.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.263+0000 7fe4509c3700 1 --2- 172.21.15.158:0/2851867893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:57.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44bfff700 1 -- 172.21.15.158:0/2851867893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 msgr2=0x7fe44c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:57.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44bfff700 1 --2- 172.21.15.158:0/2851867893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44bfff700 1 -- 172.21.15.158:0/2851867893 >> v1:172.21.15.172:6789/0 conn(0x7fe44c0cea20 legacy=0x7fe44c0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:57.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44bfff700 1 -- 172.21.15.158:0/2851867893 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c0cc280 con 0x7fe44c15deb0 2023-10-16T10:01:57.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44b7fe700 1 -- 172.21.15.158:0/2851867893 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239241103 0 0) 0x7fe44c0d0ae0 con 0x7fe44c0cea20 2023-10-16T10:01:57.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44bfff700 1 --2- 172.21.15.158:0/2851867893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c15deb0 0x7fe44c0ce4e0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fe4380098f0 tx=0x7fe438009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1915a0485145c9d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44b7fe700 1 -- 172.21.15.158:0/2851867893 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe438011830 con 0x7fe44c15deb0 2023-10-16T10:01:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44b7fe700 1 -- 172.21.15.158:0/2851867893 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe438011990 con 0x7fe44c15deb0 2023-10-16T10:01:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe44b7fe700 1 -- 172.21.15.158:0/2851867893 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe438011d20 con 0x7fe44c15deb0 2023-10-16T10:01:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe452c27700 1 -- 172.21.15.158:0/2851867893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c15deb0 msgr2=0x7fe44c0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe452c27700 1 --2- 172.21.15.158:0/2851867893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c15deb0 0x7fe44c0ce4e0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fe4380098f0 tx=0x7fe438009c00 comp rx=0 tx=0).stop 2023-10-16T10:01:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe452c27700 1 -- 172.21.15.158:0/2851867893 shutdown_connections 2023-10-16T10:01:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe452c27700 1 --2- 172.21.15.158:0/2851867893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe452c27700 1 --2- 172.21.15.158:0/2851867893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c15deb0 0x7fe44c0ce4e0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.264+0000 7fe452c27700 1 -- 172.21.15.158:0/2851867893 >> 172.21.15.158:0/2851867893 conn(0x7fe44c099a10 msgr2=0x7fe44c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.265+0000 7fe452c27700 1 -- 172.21.15.158:0/2851867893 shutdown_connections 2023-10-16T10:01:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.265+0000 7fe452c27700 1 -- 172.21.15.158:0/2851867893 wait complete. 2023-10-16T10:01:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.266+0000 7fe452c27700 1 Processor -- start 2023-10-16T10:01:57.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.266+0000 7fe452c27700 1 -- start start 2023-10-16T10:01:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c0cea20 0x7fe44c146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c15deb0 0x7fe44c0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c0c3610 con 0x7fe44c0cea20 2023-10-16T10:01:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c0b4410 con 0x7fe44c15deb0 2023-10-16T10:01:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c1468c0 con 0x7fe44c15dad0 2023-10-16T10:01:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c0cea20 0x7fe44c146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c0cea20 0x7fe44c146380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56248/0 (socket says 172.21.15.158:56248) 2023-10-16T10:01:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 -- 172.21.15.158:0/1892189887 learned_addr learned my addr 172.21.15.158:0/1892189887 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4511c4700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c15deb0 0x7fe44c0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe44bfff700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 -- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 msgr2=0x7fe44c0c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 -- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c15deb0 msgr2=0x7fe44c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c15deb0 0x7fe44c0cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 -- 172.21.15.158:0/1892189887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe438009550 con 0x7fe44c0cea20 2023-10-16T10:01:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4509c3700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c0cea20 0x7fe44c146380 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e9b0 tx=0x7fe44000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4497fa700 1 -- 172.21.15.158:0/1892189887 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440009710 con 0x7fe44c0cea20 2023-10-16T10:01:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4497fa700 1 -- 172.21.15.158:0/1892189887 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44000ee50 con 0x7fe44c0cea20 2023-10-16T10:01:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe4497fa700 1 -- 172.21.15.158:0/1892189887 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44000ca50 con 0x7fe44c0cea20 2023-10-16T10:01:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44c0c2800 con 0x7fe44c0cea20 2023-10-16T10:01:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.267+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44c0c1f60 con 0x7fe44c0cea20 2023-10-16T10:01:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.268+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44c0681b0 con 0x7fe44c0cea20 2023-10-16T10:01:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.269+0000 7fe4497fa700 1 -- 172.21.15.158:0/1892189887 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 61) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe44000cbb0 con 0x7fe44c0cea20 2023-10-16T10:01:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.269+0000 7fe4497fa700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe43006ad50 0x7fe43006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.269+0000 7fe4497fa700 1 -- 172.21.15.158:0/1892189887 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7429+0+0 (secure 0 0 0) 0x7fe440089870 con 0x7fe44c0cea20 2023-10-16T10:01:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.269+0000 7fe44bfff700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe43006ad50 0x7fe43006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:57.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.270+0000 7fe44bfff700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe43006ad50 0x7fe43006d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe44c15e2b0 tx=0x7fe438008030 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:57.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.274+0000 7fe4497fa700 1 -- 172.21.15.158:0/1892189887 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4400545a0 con 0x7fe44c0cea20 2023-10-16T10:01:57.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:57.509+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe44c0ccb90 con 0x7fe44c0cea20 2023-10-16T10:01:58.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.140+0000 7fe4497fa700 1 -- 172.21.15.158:0/1892189887 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe44000ce60 con 0x7fe44c0cea20 2023-10-16T10:01:58.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe43006ad50 msgr2=0x7fe43006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:58.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe43006ad50 0x7fe43006d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe44c15e2b0 tx=0x7fe438008030 comp rx=0 tx=0).stop 2023-10-16T10:01:58.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c0cea20 msgr2=0x7fe44c146380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:58.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c0cea20 0x7fe44c146380 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e9b0 tx=0x7fe44000ecc0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 shutdown_connections 2023-10-16T10:01:58.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c15deb0 0x7fe44c0cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe43006ad50 0x7fe43006d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe44c0cea20 0x7fe44c146380 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 --2- 172.21.15.158:0/1892189887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe44c15dad0 0x7fe44c0c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 >> 172.21.15.158:0/1892189887 conn(0x7fe44c099a10 msgr2=0x7fe44c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:58.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 shutdown_connections 2023-10-16T10:01:58.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.143+0000 7fe452c27700 1 -- 172.21.15.158:0/1892189887 wait complete. 2023-10-16T10:01:58.158 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T10:01:58.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.286+0000 7f45f49ae700 1 Processor -- start 2023-10-16T10:01:58.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.286+0000 7f45f49ae700 1 -- start start 2023-10-16T10:01:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45f49ae700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f45f015da40 0x7f45f00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45f49ae700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00cbd30 con 0x7f45f006afe0 2023-10-16T10:01:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45f49ae700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00cc1c0 con 0x7f45f00cc590 2023-10-16T10:01:58.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45f49ae700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f015de20 con 0x7f45f015da40 2023-10-16T10:01:58.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45eed9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f45f006afe0 0x7f45f00afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56454/0 (socket says 172.21.15.158:56454) 2023-10-16T10:01:58.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45eed9d700 1 -- 172.21.15.158:0/2883134256 learned_addr learned my addr 172.21.15.158:0/2883134256 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:01:58.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45ee59c700 1 --2- 172.21.15.158:0/2883134256 >> v2:172.21.15.172:3300/0 conn(0x7f45f015da40 0x7f45f00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:58.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45ed59a700 1 -- 172.21.15.158:0/2883134256 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986009377 0 0) 0x7f45f00cbd30 con 0x7f45f006afe0 2023-10-16T10:01:58.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.287+0000 7f45ed59a700 1 -- 172.21.15.158:0/2883134256 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d80035b0 con 0x7f45f006afe0 2023-10-16T10:01:58.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ee59c700 1 -- 172.21.15.158:0/2883134256 >> v1:172.21.15.172:6789/0 conn(0x7f45f00cc590 legacy=0x7f45f006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:58.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ee59c700 1 -- 172.21.15.158:0/2883134256 >> v1:172.21.15.158:6789/0 conn(0x7f45f006afe0 legacy=0x7f45f00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:58.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ed59a700 1 -- 172.21.15.158:0/2883134256 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2571189258 0 0) 0x7f45d80035b0 con 0x7f45f006afe0 2023-10-16T10:01:58.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ee59c700 1 -- 172.21.15.158:0/2883134256 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f00ce260 con 0x7f45f015da40 2023-10-16T10:01:58.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ee59c700 1 --2- 172.21.15.158:0/2883134256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f015da40 0x7f45f00caa20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009840 tx=0x7f45e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a5ba05e818d1381 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:58.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ed59a700 1 -- 172.21.15.158:0/2883134256 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e80019f0 con 0x7f45f015da40 2023-10-16T10:01:58.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ed59a700 1 -- 172.21.15.158:0/2883134256 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e0004030 con 0x7f45f015da40 2023-10-16T10:01:58.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45ed59a700 1 -- 172.21.15.158:0/2883134256 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e00043a0 con 0x7f45f015da40 2023-10-16T10:01:58.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45f49ae700 1 -- 172.21.15.158:0/2883134256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f015da40 msgr2=0x7f45f00caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:58.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.288+0000 7f45f49ae700 1 --2- 172.21.15.158:0/2883134256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f015da40 0x7f45f00caa20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009840 tx=0x7f45e0009b50 comp rx=0 tx=0).stop 2023-10-16T10:01:58.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.289+0000 7f45f49ae700 1 -- 172.21.15.158:0/2883134256 shutdown_connections 2023-10-16T10:01:58.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.289+0000 7f45f49ae700 1 --2- 172.21.15.158:0/2883134256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f015da40 0x7f45f00caa20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.289+0000 7f45f49ae700 1 -- 172.21.15.158:0/2883134256 >> 172.21.15.158:0/2883134256 conn(0x7f45f0099a10 msgr2=0x7f45f00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:58.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.289+0000 7f45f49ae700 1 -- 172.21.15.158:0/2883134256 shutdown_connections 2023-10-16T10:01:58.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.289+0000 7f45f49ae700 1 -- 172.21.15.158:0/2883134256 wait complete. 2023-10-16T10:01:58.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.290+0000 7f45f49ae700 1 Processor -- start 2023-10-16T10:01:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.290+0000 7f45f49ae700 1 -- start start 2023-10-16T10:01:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45f49ae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f006afe0 0x7f45f0157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45f49ae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00cc590 0x7f45f00c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45f49ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f015da40 0x7f45f01e4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45f49ae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f01577a0 con 0x7f45f015da40 2023-10-16T10:01:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45f49ae700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00c79e0 con 0x7f45f00cc590 2023-10-16T10:01:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45f49ae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00c7c80 con 0x7f45f006afe0 2023-10-16T10:01:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f015da40 0x7f45f01e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f015da40 0x7f45f01e4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56284/0 (socket says 172.21.15.158:56284) 2023-10-16T10:01:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 -- 172.21.15.158:0/4140647437 learned_addr learned my addr 172.21.15.158:0/4140647437 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:58.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45ee59c700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f006afe0 0x7f45f0157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:58.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45edd9b700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00cc590 0x7f45f00c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:58.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 -- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f006afe0 msgr2=0x7f45f0157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:58.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f006afe0 0x7f45f0157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 -- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00cc590 msgr2=0x7f45f00c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:58.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00cc590 0x7f45f00c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:58.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 -- 172.21.15.158:0/4140647437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e00094a0 con 0x7f45f015da40 2023-10-16T10:01:58.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45eed9d700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f015da40 0x7f45f01e4100 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f45e8001560 tx=0x7f45e800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:58.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e800aaa0 con 0x7f45f015da40 2023-10-16T10:01:58.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e800ac00 con 0x7f45f015da40 2023-10-16T10:01:58.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e80070d0 con 0x7f45f015da40 2023-10-16T10:01:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.291+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f00c7f60 con 0x7f45f015da40 2023-10-16T10:01:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.292+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f00c8520 con 0x7f45f015da40 2023-10-16T10:01:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.293+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f00681b0 con 0x7f45f015da40 2023-10-16T10:01:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.293+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 61) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f45e8007230 con 0x7f45f015da40 2023-10-16T10:01:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.293+0000 7f45deffd700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f45d006ad50 0x7f45d006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.293+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7429+0+0 (secure 0 0 0) 0x7f45e80892b0 con 0x7f45f015da40 2023-10-16T10:01:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.293+0000 7f45ee59c700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f45d006ad50 0x7f45d006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.294+0000 7f45ee59c700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f45d006ad50 0x7f45d006d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45e00090b0 tx=0x7f45e0008030 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:58.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.298+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f45e8053fe0 con 0x7f45f015da40 2023-10-16T10:01:58.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.489+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f45f00cb9b0 con 0x7f45f015da40 2023-10-16T10:01:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:58.697+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 62) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f45e80543b0 con 0x7f45f015da40 2023-10-16T10:01:58.828 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:01:58.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:01:59.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.144+0000 7f45deffd700 1 -- 172.21.15.158:0/4140647437 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v117) v1 ==== 181+0+0 (secure 0 0 0) 0x7f45e8074340 con 0x7f45f015da40 2023-10-16T10:01:59.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f45d006ad50 msgr2=0x7f45d006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:59.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f45d006ad50 0x7f45d006d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45e00090b0 tx=0x7f45e0008030 comp rx=0 tx=0).stop 2023-10-16T10:01:59.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f015da40 msgr2=0x7f45f01e4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:59.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f015da40 0x7f45f01e4100 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f45e8001560 tx=0x7f45e800cf20 comp rx=0 tx=0).stop 2023-10-16T10:01:59.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 shutdown_connections 2023-10-16T10:01:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f45d006ad50 0x7f45d006d210 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45e00090b0 tx=0x7f45e0008030 comp rx=0 tx=0).stop 2023-10-16T10:01:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f015da40 0x7f45f01e4100 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f006afe0 0x7f45f0157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 --2- 172.21.15.158:0/4140647437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00cc590 0x7f45f00c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 >> 172.21.15.158:0/4140647437 conn(0x7f45f0099a10 msgr2=0x7f45f015cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:59.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 shutdown_connections 2023-10-16T10:01:59.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.148+0000 7f45f49ae700 1 -- 172.21.15.158:0/4140647437 wait complete. 2023-10-16T10:01:59.152 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T10:01:59.163 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T10:01:59.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.289+0000 7f5d2a0da700 1 Processor -- start 2023-10-16T10:01:59.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.289+0000 7f5d2a0da700 1 -- start start 2023-10-16T10:01:59.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d2a0da700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5d24143790 0x7f5d2406bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:59.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d2a0da700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5d240c50a0 0x7f5d240c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:59.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d2a0da700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2406aa70 con 0x7f5d2406e540 2023-10-16T10:01:59.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d2a0da700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d24143b70 con 0x7f5d24143790 2023-10-16T10:01:59.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d2a0da700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d240c5a60 con 0x7f5d240c50a0 2023-10-16T10:01:59.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d237fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5d24143790 0x7f5d2406bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:59.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d237fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5d24143790 0x7f5d2406bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56288/0 (socket says 172.21.15.158:56288) 2023-10-16T10:01:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.290+0000 7f5d237fe700 1 -- 172.21.15.158:0/2246624843 learned_addr learned my addr 172.21.15.158:0/2246624843 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d237fe700 1 -- 172.21.15.158:0/2246624843 >> v2:172.21.15.172:3300/0 conn(0x7f5d240c50a0 msgr2=0x7f5d240c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d237fe700 1 --2- 172.21.15.158:0/2246624843 >> v2:172.21.15.172:3300/0 conn(0x7f5d240c50a0 0x7f5d240c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d237fe700 1 -- 172.21.15.158:0/2246624843 >> v1:172.21.15.172:6789/0 conn(0x7f5d2406e540 legacy=0x7f5d2406e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:01:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d237fe700 1 -- 172.21.15.158:0/2246624843 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2406ad50 con 0x7f5d24143790 2023-10-16T10:01:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d237fe700 1 --2- 172.21.15.158:0/2246624843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d24143790 0x7f5d2406bb10 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5d24070ca0 tx=0x7f5d14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba13f66080db6574 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d22ffd700 1 -- 172.21.15.158:0/2246624843 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d14004230 con 0x7f5d24143790 2023-10-16T10:01:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d22ffd700 1 -- 172.21.15.158:0/2246624843 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d14004390 con 0x7f5d24143790 2023-10-16T10:01:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d22ffd700 1 -- 172.21.15.158:0/2246624843 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d140036a0 con 0x7f5d24143790 2023-10-16T10:01:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2246624843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d24143790 msgr2=0x7f5d2406bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2246624843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d24143790 0x7f5d2406bb10 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5d24070ca0 tx=0x7f5d14009bd0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2246624843 shutdown_connections 2023-10-16T10:01:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2246624843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d24143790 0x7f5d2406bb10 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2246624843 >> v2:172.21.15.172:3300/0 conn(0x7f5d240c50a0 0x7f5d240c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.291+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2246624843 >> 172.21.15.158:0/2246624843 conn(0x7f5d240cc5b0 msgr2=0x7f5d240cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:01:59.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.292+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2246624843 shutdown_connections 2023-10-16T10:01:59.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.292+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2246624843 wait complete. 2023-10-16T10:01:59.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.293+0000 7f5d2a0da700 1 Processor -- start 2023-10-16T10:01:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.293+0000 7f5d2a0da700 1 -- start start 2023-10-16T10:01:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d2406e540 0x7f5d241f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d240c50a0 0x7f5d24158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d24143790 0x7f5d24158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240c7740 con 0x7f5d2406e540 2023-10-16T10:01:59.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24140c90 con 0x7f5d24143790 2023-10-16T10:01:59.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24140f70 con 0x7f5d240c50a0 2023-10-16T10:01:59.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d2406e540 0x7f5d241f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:59.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d2406e540 0x7f5d241f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56294/0 (socket says 172.21.15.158:56294) 2023-10-16T10:01:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 -- 172.21.15.158:0/2222049430 learned_addr learned my addr 172.21.15.158:0/2222049430 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:01:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d23fff700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d24143790 0x7f5d24158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d1affd700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d240c50a0 0x7f5d24158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 -- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d240c50a0 msgr2=0x7f5d24158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d240c50a0 0x7f5d24158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 -- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d24143790 msgr2=0x7f5d24158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:01:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d24143790 0x7f5d24158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:01:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 -- 172.21.15.158:0/2222049430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d14009550 con 0x7f5d2406e540 2023-10-16T10:01:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d237fe700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d2406e540 0x7f5d241f31a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009130 tx=0x7f5d14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d20ff9700 1 -- 172.21.15.158:0/2222049430 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d14011630 con 0x7f5d2406e540 2023-10-16T10:01:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d20ff9700 1 -- 172.21.15.158:0/2222049430 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d14011790 con 0x7f5d2406e540 2023-10-16T10:01:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d24141250 con 0x7f5d2406e540 2023-10-16T10:01:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d20ff9700 1 -- 172.21.15.158:0/2222049430 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d14011b40 con 0x7f5d2406e540 2023-10-16T10:01:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.294+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240caf20 con 0x7f5d2406e540 2023-10-16T10:01:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.295+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d240681b0 con 0x7f5d2406e540 2023-10-16T10:01:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.296+0000 7f5d20ff9700 1 -- 172.21.15.158:0/2222049430 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 62) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5d14011ca0 con 0x7f5d2406e540 2023-10-16T10:01:59.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.296+0000 7f5d20ff9700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5d0406af90 0x7f5d0406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:01:59.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.296+0000 7f5d20ff9700 1 -- 172.21.15.158:0/2222049430 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6991+0+0 (secure 0 0 0) 0x7f5d1408b070 con 0x7f5d2406e540 2023-10-16T10:01:59.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.296+0000 7f5d1affd700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5d0406af90 0x7f5d0406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:01:59.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.297+0000 7f5d1affd700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5d0406af90 0x7f5d0406d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c006fd0 tx=0x7f5d0c008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:01:59.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.301+0000 7f5d20ff9700 1 -- 172.21.15.158:0/2222049430 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d14055cc0 con 0x7f5d2406e540 2023-10-16T10:01:59.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:01:59.496+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5d240c5d90 con 0x7f5d2406e540 2023-10-16T10:02:00.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.146+0000 7f5d20ff9700 1 -- 172.21.15.158:0/2222049430 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v118) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5d140592e0 con 0x7f5d2406e540 2023-10-16T10:02:00.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5d0406af90 msgr2=0x7f5d0406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5d0406af90 0x7f5d0406d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c006fd0 tx=0x7f5d0c008040 comp rx=0 tx=0).stop 2023-10-16T10:02:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d2406e540 msgr2=0x7f5d241f31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d2406e540 0x7f5d241f31a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009130 tx=0x7f5d14004750 comp rx=0 tx=0).stop 2023-10-16T10:02:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 shutdown_connections 2023-10-16T10:02:00.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d24143790 0x7f5d24158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5d0406af90 0x7f5d0406d450 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d2406e540 0x7f5d241f31a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 --2- 172.21.15.158:0/2222049430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d240c50a0 0x7f5d24158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 >> 172.21.15.158:0/2222049430 conn(0x7f5d240cc5b0 msgr2=0x7f5d24159790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 shutdown_connections 2023-10-16T10:02:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.150+0000 7f5d2a0da700 1 -- 172.21.15.158:0/2222049430 wait complete. 2023-10-16T10:02:00.154 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T10:02:00.168 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T10:02:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.292+0000 7f05eb58c700 1 Processor -- start 2023-10-16T10:02:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.292+0000 7f05eb58c700 1 -- start start 2023-10-16T10:02:00.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05eb58c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 0x7f05e40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05eb58c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e40cbd30 con 0x7f05e406afe0 2023-10-16T10:02:00.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05eb58c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e40cc1c0 con 0x7f05e415da40 2023-10-16T10:02:00.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05eb58c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e415de20 con 0x7f05e40cc590 2023-10-16T10:02:00.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05e8b27700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f05e406afe0 0x7f05e40afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56478/0 (socket says 172.21.15.158:56478) 2023-10-16T10:02:00.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05e8b27700 1 -- 172.21.15.158:0/3182106466 learned_addr learned my addr 172.21.15.158:0/3182106466 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05e9b29700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 0x7f05e40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808522339 0 0) 0x7f05e40cbd30 con 0x7f05e406afe0 2023-10-16T10:02:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.293+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d00035b0 con 0x7f05e406afe0 2023-10-16T10:02:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3245451400 0 0) 0x7f05d00035b0 con 0x7f05e406afe0 2023-10-16T10:02:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 msgr2=0x7f05e40caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 0x7f05e40caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 >> v1:172.21.15.172:6789/0 conn(0x7f05e415da40 legacy=0x7f05e406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e40ce260 con 0x7f05e406afe0 2023-10-16T10:02:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f05d4001670 con 0x7f05e406afe0 2023-10-16T10:02:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 >> v1:172.21.15.158:6789/0 conn(0x7f05e406afe0 legacy=0x7f05e40afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 0x7f05d0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0003f70 0x7f05d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05d0004d90 0x7f05d0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e415de20 con 0x7f05d0004d90 2023-10-16T10:02:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d000ba70 con 0x7f05d0003f70 2023-10-16T10:02:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d000bc30 con 0x7f05e40cc590 2023-10-16T10:02:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9b29700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05d0004d90 0x7f05d0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9328700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 0x7f05d0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e8b27700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0003f70 0x7f05d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9b29700 1 -- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 msgr2=0x7f05d0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:00.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9b29700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40cc590 0x7f05d0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9b29700 1 -- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0003f70 msgr2=0x7f05d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:00.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9b29700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0003f70 0x7f05d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9b29700 1 -- 172.21.15.158:0/3182106466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d00035b0 con 0x7f05d0004d90 2023-10-16T10:02:00.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.294+0000 7f05e9b29700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05d0004d90 0x7f05d0005210 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f05dc009fc0 tx=0x7f05d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb927600aaf49c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:00.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc00ec30 con 0x7f05d0004d90 2023-10-16T10:02:00.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc00f040 con 0x7f05d0004d90 2023-10-16T10:02:00.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05dbfff700 1 -- 172.21.15.158:0/3182106466 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc014480 con 0x7f05d0004d90 2023-10-16T10:02:00.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05eb58c700 1 -- 172.21.15.158:0/3182106466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05d0004d90 msgr2=0x7f05d0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05eb58c700 1 --2- 172.21.15.158:0/3182106466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05d0004d90 0x7f05d0005210 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f05dc009fc0 tx=0x7f05d0005880 comp rx=0 tx=0).stop 2023-10-16T10:02:00.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05e9b29700 1 -- 172.21.15.158:0/3182106466 reap_dead start 2023-10-16T10:02:00.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05eb58c700 1 -- 172.21.15.158:0/3182106466 shutdown_connections 2023-10-16T10:02:00.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05eb58c700 1 -- 172.21.15.158:0/3182106466 >> 172.21.15.158:0/3182106466 conn(0x7f05e4099a10 msgr2=0x7f05e40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:00.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05eb58c700 1 -- 172.21.15.158:0/3182106466 shutdown_connections 2023-10-16T10:02:00.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.295+0000 7f05eb58c700 1 -- 172.21.15.158:0/3182106466 wait complete. 2023-10-16T10:02:00.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.296+0000 7f05eb58c700 1 Processor -- start 2023-10-16T10:02:00.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.296+0000 7f05eb58c700 1 -- start start 2023-10-16T10:02:00.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.297+0000 7f05eb58c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0004d90 0x7f05e40c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.297+0000 7f05eb58c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40c8850 0x7f05e4158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.297+0000 7f05eb58c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05e4159fb0 0x7f05e415a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.297+0000 7f05eb58c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e415a970 con 0x7f05e4159fb0 2023-10-16T10:02:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.297+0000 7f05eb58c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e40c6d10 con 0x7f05d0004d90 2023-10-16T10:02:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.297+0000 7f05eb58c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e40c8e10 con 0x7f05e40c8850 2023-10-16T10:02:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.297+0000 7f05e9b29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05e4159fb0 0x7f05e415a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05e4159fb0 0x7f05e415a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56318/0 (socket says 172.21.15.158:56318) 2023-10-16T10:02:00.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 -- 172.21.15.158:0/4176373207 learned_addr learned my addr 172.21.15.158:0/4176373207 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:00.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9328700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0004d90 0x7f05e40c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e8b27700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40c8850 0x7f05e4158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 -- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40c8850 msgr2=0x7f05e4158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40c8850 0x7f05e4158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 -- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0004d90 msgr2=0x7f05e40c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0004d90 0x7f05e40c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 -- 172.21.15.158:0/4176373207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e40c90f0 con 0x7f05e4159fb0 2023-10-16T10:02:00.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e8b27700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40c8850 0x7f05e4158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:00.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05e9b29700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05e4159fb0 0x7f05e415a430 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f05dc016000 tx=0x7f05dc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:00.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05d9ffb700 1 -- 172.21.15.158:0/4176373207 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc007000 con 0x7f05e4159fb0 2023-10-16T10:02:00.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05d9ffb700 1 -- 172.21.15.158:0/4176373207 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc022070 con 0x7f05e4159fb0 2023-10-16T10:02:00.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e40c93d0 con 0x7f05e4159fb0 2023-10-16T10:02:00.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05d9ffb700 1 -- 172.21.15.158:0/4176373207 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc029480 con 0x7f05e4159fb0 2023-10-16T10:02:00.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.298+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e4158c70 con 0x7f05e4159fb0 2023-10-16T10:02:00.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.299+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e40681b0 con 0x7f05e4159fb0 2023-10-16T10:02:00.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.300+0000 7f05d9ffb700 1 -- 172.21.15.158:0/4176373207 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 62) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f05dc01e430 con 0x7f05e4159fb0 2023-10-16T10:02:00.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.300+0000 7f05d9ffb700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f05c806afc0 0x7f05c806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.300+0000 7f05d9ffb700 1 -- 172.21.15.158:0/4176373207 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f05dc024070 con 0x7f05e4159fb0 2023-10-16T10:02:00.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.300+0000 7f05e9328700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f05c806afc0 0x7f05c806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.300+0000 7f05e9328700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f05c806afc0 0x7f05c806d480 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05e0007fd0 tx=0x7f05e0009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:00.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.304+0000 7f05d9ffb700 1 -- 172.21.15.158:0/4176373207 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f05dc056850 con 0x7f05e4159fb0 2023-10-16T10:02:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.496+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f05e415e940 con 0x7f05e4159fb0 2023-10-16T10:02:00.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.499+0000 7f05d9ffb700 1 -- 172.21.15.158:0/4176373207 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7f05dc059e70 con 0x7f05e4159fb0 2023-10-16T10:02:00.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f05c806afc0 msgr2=0x7f05c806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f05c806afc0 0x7f05c806d480 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05e0007fd0 tx=0x7f05e0009040 comp rx=0 tx=0).stop 2023-10-16T10:02:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05e4159fb0 msgr2=0x7f05e415a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05e4159fb0 0x7f05e415a430 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f05dc016000 tx=0x7f05dc009750 comp rx=0 tx=0).stop 2023-10-16T10:02:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 shutdown_connections 2023-10-16T10:02:00.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f05c806afc0 0x7f05c806d480 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05e4159fb0 0x7f05e415a430 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05d0004d90 0x7f05e40c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 --2- 172.21.15.158:0/4176373207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05e40c8850 0x7f05e4158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 >> 172.21.15.158:0/4176373207 conn(0x7f05e4099a10 msgr2=0x7f05e415ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 shutdown_connections 2023-10-16T10:02:00.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.502+0000 7f05eb58c700 1 -- 172.21.15.158:0/4176373207 wait complete. 2023-10-16T10:02:00.516 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T10:02:00.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.637+0000 7f4f74213700 1 Processor -- start 2023-10-16T10:02:00.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.637+0000 7f4f74213700 1 -- start start 2023-10-16T10:02:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f74213700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4f6c0c49b0 0x7f4f6c0c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f74213700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4f6c06abb0 0x7f4f6c06b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f74213700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0ce110 con 0x7f4f6c160d40 2023-10-16T10:02:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f74213700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6c0c53a0 con 0x7f4f6c06abb0 2023-10-16T10:02:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f74213700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6c0c61e0 con 0x7f4f6c0c49b0 2023-10-16T10:02:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f727b0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4f6c06abb0 0x7f4f6c06b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f727b0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4f6c06abb0 0x7f4f6c06b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56326/0 (socket says 172.21.15.158:56326) 2023-10-16T10:02:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.638+0000 7f4f727b0700 1 -- 172.21.15.158:0/3751239135 learned_addr learned my addr 172.21.15.158:0/3751239135 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f717ae700 1 --2- 172.21.15.158:0/3751239135 >> v2:172.21.15.172:3300/0 conn(0x7f4f6c0c49b0 0x7f4f6c0c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f727b0700 1 -- 172.21.15.158:0/3751239135 >> v2:172.21.15.172:3300/0 conn(0x7f4f6c0c49b0 msgr2=0x7f4f6c0c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f727b0700 1 --2- 172.21.15.158:0/3751239135 >> v2:172.21.15.172:3300/0 conn(0x7f4f6c0c49b0 0x7f4f6c0c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f727b0700 1 -- 172.21.15.158:0/3751239135 >> v1:172.21.15.172:6789/0 conn(0x7f4f6c160d40 legacy=0x7f4f6c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f727b0700 1 -- 172.21.15.158:0/3751239135 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c0ce3f0 con 0x7f4f6c06abb0 2023-10-16T10:02:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f727b0700 1 --2- 172.21.15.158:0/3751239135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c06b000 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0c6b30 tx=0x7f4f6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f405434b3b6a3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f70fad700 1 -- 172.21.15.158:0/3751239135 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400e070 con 0x7f4f6c06abb0 2023-10-16T10:02:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f70fad700 1 -- 172.21.15.158:0/3751239135 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f64003a20 con 0x7f4f6c06abb0 2023-10-16T10:02:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.639+0000 7f4f70fad700 1 -- 172.21.15.158:0/3751239135 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64003dd0 con 0x7f4f6c06abb0 2023-10-16T10:02:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 -- 172.21.15.158:0/3751239135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 msgr2=0x7f4f6c06b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 --2- 172.21.15.158:0/3751239135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c06b000 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0c6b30 tx=0x7f4f6400b330 comp rx=0 tx=0).stop 2023-10-16T10:02:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 -- 172.21.15.158:0/3751239135 shutdown_connections 2023-10-16T10:02:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 --2- 172.21.15.158:0/3751239135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c06b000 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 --2- 172.21.15.158:0/3751239135 >> v2:172.21.15.172:3300/0 conn(0x7f4f6c0c49b0 0x7f4f6c0c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 -- 172.21.15.158:0/3751239135 >> 172.21.15.158:0/3751239135 conn(0x7f4f6c0cc5b0 msgr2=0x7f4f6c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:00.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 -- 172.21.15.158:0/3751239135 shutdown_connections 2023-10-16T10:02:00.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.640+0000 7f4f74213700 1 -- 172.21.15.158:0/3751239135 wait complete. 2023-10-16T10:02:00.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.641+0000 7f4f74213700 1 Processor -- start 2023-10-16T10:02:00.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.641+0000 7f4f74213700 1 -- start start 2023-10-16T10:02:00.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c0ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6c0c49b0 0x7f4f6c14b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f6c160d40 0x7f4f6c0af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c14bb50 con 0x7f4f6c06abb0 2023-10-16T10:02:00.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0aca00 con 0x7f4f6c160d40 2023-10-16T10:02:00.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0acce0 con 0x7f4f6c0c49b0 2023-10-16T10:02:00.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c0ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c0ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56328/0 (socket says 172.21.15.158:56328) 2023-10-16T10:02:00.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 -- 172.21.15.158:0/3599696971 learned_addr learned my addr 172.21.15.158:0/3599696971 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:00.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 -- 172.21.15.158:0/3599696971 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6c0c49b0 msgr2=0x7f4f6c14b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6c0c49b0 0x7f4f6c14b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 -- 172.21.15.158:0/3599696971 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f6c160d40 msgr2=0x7f4f6c0af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f6c160d40 0x7f4f6c0af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 -- 172.21.15.158:0/3599696971 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f64009b30 con 0x7f4f6c06abb0 2023-10-16T10:02:00.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f71faf700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c0ad7c0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00b4a0 tx=0x7f4f5c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:00.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c011830 con 0x7f4f6c06abb0 2023-10-16T10:02:00.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c011990 con 0x7f4f6c06abb0 2023-10-16T10:02:00.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c0acfc0 con 0x7f4f6c06abb0 2023-10-16T10:02:00.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c011d40 con 0x7f4f6c06abb0 2023-10-16T10:02:00.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.642+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c0af910 con 0x7f4f6c06abb0 2023-10-16T10:02:00.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.643+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6c066db0 con 0x7f4f6c06abb0 2023-10-16T10:02:00.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.644+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 62) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4f5c01a400 con 0x7f4f6c06abb0 2023-10-16T10:02:00.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.644+0000 7f4f627fc700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4f5006af90 0x7f4f5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.644+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f4f5c089630 con 0x7f4f6c06abb0 2023-10-16T10:02:00.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.644+0000 7f4f717ae700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4f5006af90 0x7f4f5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.645+0000 7f4f717ae700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4f5006af90 0x7f4f5006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0c51a0 tx=0x7f4f68008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:00.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.649+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f5c054400 con 0x7f4f6c06abb0 2023-10-16T10:02:00.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.704+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f5c0547d0 con 0x7f4f6c06abb0 2023-10-16T10:02:00.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.847+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4f6c0681b0 con 0x7f4f6c06abb0 2023-10-16T10:02:00.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.850+0000 7f4f627fc700 1 -- 172.21.15.158:0/3599696971 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4f5c075000 con 0x7f4f6c06abb0 2023-10-16T10:02:00.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4f5006af90 msgr2=0x7f4f5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4f5006af90 0x7f4f5006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0c51a0 tx=0x7f4f68008040 comp rx=0 tx=0).stop 2023-10-16T10:02:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 msgr2=0x7f4f6c0ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c0ad7c0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00b4a0 tx=0x7f4f5c00b860 comp rx=0 tx=0).stop 2023-10-16T10:02:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 shutdown_connections 2023-10-16T10:02:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f6c160d40 0x7f4f6c0af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4f5006af90 0x7f4f5006d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f6c06abb0 0x7f4f6c0ad7c0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 --2- 172.21.15.158:0/3599696971 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f6c0c49b0 0x7f4f6c14b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 >> 172.21.15.158:0/3599696971 conn(0x7f4f6c0cc5b0 msgr2=0x7f4f6c0cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 shutdown_connections 2023-10-16T10:02:00.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.853+0000 7f4f74213700 1 -- 172.21.15.158:0/3599696971 wait complete. 2023-10-16T10:02:00.866 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T10:02:00.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.987+0000 7f39eed8c700 1 Processor -- start 2023-10-16T10:02:00.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.987+0000 7f39eed8c700 1 -- start start 2023-10-16T10:02:00.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39eed8c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f39e8159280 0x7f39e806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39eed8c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e80cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:00.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39eed8c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e8159660 con 0x7f39e80d0e10 2023-10-16T10:02:00.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39eed8c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e80cb710 con 0x7f39e8159280 2023-10-16T10:02:00.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39eed8c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e806b120 con 0x7f39e80cbef0 2023-10-16T10:02:00.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39ecb28700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f39e8159280 0x7f39e806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39ecb28700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f39e8159280 0x7f39e806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56336/0 (socket says 172.21.15.158:56336) 2023-10-16T10:02:00.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39ecb28700 1 -- 172.21.15.158:0/54289295 learned_addr learned my addr 172.21.15.158:0/54289295 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:00.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.988+0000 7f39e7fff700 1 --1- 172.21.15.158:0/54289295 >> v1:172.21.15.158:6789/0 conn(0x7f39e80d0e10 0x7f39e815c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56512/0 (socket says 172.21.15.158:56512) 2023-10-16T10:02:00.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39ecb28700 1 -- 172.21.15.158:0/54289295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 msgr2=0x7f39e80cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:00.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39ed329700 1 --2- 172.21.15.158:0/54289295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e80cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:00.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39ecb28700 1 --2- 172.21.15.158:0/54289295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e80cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39e77fe700 1 -- 172.21.15.158:0/54289295 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232071420 0 0) 0x7f39e8159660 con 0x7f39e80d0e10 2023-10-16T10:02:00.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39ecb28700 1 -- 172.21.15.158:0/54289295 >> v1:172.21.15.158:6789/0 conn(0x7f39e80d0e10 legacy=0x7f39e815c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39ecb28700 1 -- 172.21.15.158:0/54289295 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e806b400 con 0x7f39e8159280 2023-10-16T10:02:00.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39ed329700 1 --2- 172.21.15.158:0/54289295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e80cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:00.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39ecb28700 1 --2- 172.21.15.158:0/54289295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e8159280 0x7f39e806a8d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098f0 tx=0x7f39dc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf81d9476dd4272 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:00.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39e77fe700 1 -- 172.21.15.158:0/54289295 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc00b8a0 con 0x7f39e8159280 2023-10-16T10:02:00.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39e77fe700 1 -- 172.21.15.158:0/54289295 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc00ba00 con 0x7f39e8159280 2023-10-16T10:02:00.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39e77fe700 1 -- 172.21.15.158:0/54289295 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc00bd90 con 0x7f39e8159280 2023-10-16T10:02:00.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39eed8c700 1 -- 172.21.15.158:0/54289295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e8159280 msgr2=0x7f39e806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:00.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.989+0000 7f39eed8c700 1 --2- 172.21.15.158:0/54289295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e8159280 0x7f39e806a8d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098f0 tx=0x7f39dc009c00 comp rx=0 tx=0).stop 2023-10-16T10:02:00.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.990+0000 7f39eed8c700 1 -- 172.21.15.158:0/54289295 shutdown_connections 2023-10-16T10:02:00.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.990+0000 7f39eed8c700 1 --2- 172.21.15.158:0/54289295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e8159280 0x7f39e806a8d0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.990+0000 7f39eed8c700 1 --2- 172.21.15.158:0/54289295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e80cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:00.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.990+0000 7f39eed8c700 1 -- 172.21.15.158:0/54289295 >> 172.21.15.158:0/54289295 conn(0x7f39e8099a10 msgr2=0x7f39e80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:00.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.990+0000 7f39eed8c700 1 -- 172.21.15.158:0/54289295 shutdown_connections 2023-10-16T10:02:00.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.990+0000 7f39eed8c700 1 -- 172.21.15.158:0/54289295 wait complete. 2023-10-16T10:02:01.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.991+0000 7f39eed8c700 1 Processor -- start 2023-10-16T10:02:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.991+0000 7f39eed8c700 1 -- start start 2023-10-16T10:02:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e81f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e80d0e10 0x7f39e80c4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39e8159280 0x7f39e815eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e815cc90 con 0x7f39e80d0e10 2023-10-16T10:02:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e815f020 con 0x7f39e8159280 2023-10-16T10:02:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e815f230 con 0x7f39e80cbef0 2023-10-16T10:02:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e80d0e10 0x7f39e80c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e80d0e10 0x7f39e80c4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56340/0 (socket says 172.21.15.158:56340) 2023-10-16T10:02:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 -- 172.21.15.158:0/3713011181 learned_addr learned my addr 172.21.15.158:0/3713011181 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 -- 172.21.15.158:0/3713011181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 msgr2=0x7f39e81f4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e81f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 -- 172.21.15.158:0/3713011181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39e8159280 msgr2=0x7f39e815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39e8159280 0x7f39e815eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 -- 172.21.15.158:0/3713011181 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc009550 con 0x7f39e80d0e10 2023-10-16T10:02:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e7fff700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e80d0e10 0x7f39e80c4930 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c550 tx=0x7f39d400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e57fa700 1 -- 172.21.15.158:0/3713011181 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4004c80 con 0x7f39e80d0e10 2023-10-16T10:02:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e57fa700 1 -- 172.21.15.158:0/3713011181 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d4004e00 con 0x7f39e80d0e10 2023-10-16T10:02:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e815c5d0 con 0x7f39e80d0e10 2023-10-16T10:02:01.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39e57fa700 1 -- 172.21.15.158:0/3713011181 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4007310 con 0x7f39e80d0e10 2023-10-16T10:02:01.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.992+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e80b4bf0 con 0x7f39e80d0e10 2023-10-16T10:02:01.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.993+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e815cdd0 con 0x7f39e80d0e10 2023-10-16T10:02:01.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.994+0000 7f39e57fa700 1 -- 172.21.15.158:0/3713011181 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f39d4007470 con 0x7f39e80d0e10 2023-10-16T10:02:01.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.994+0000 7f39e57fa700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f39cc06b020 0x7f39cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.994+0000 7f39e57fa700 1 -- 172.21.15.158:0/3713011181 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f39d4089730 con 0x7f39e80d0e10 2023-10-16T10:02:01.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.994+0000 7f39ecb28700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f39cc06b020 0x7f39cc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.995+0000 7f39ecb28700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f39cc06b020 0x7f39cc06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39dc000c00 tx=0x7f39dc010000 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:01.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:00.999+0000 7f39e57fa700 1 -- 172.21.15.158:0/3713011181 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f39e815cdd0 con 0x7f39e80d0e10 2023-10-16T10:02:01.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.191+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f39e815cdd0 con 0x7f39e80d0e10 2023-10-16T10:02:01.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.195+0000 7f39e57fa700 1 -- 172.21.15.158:0/3713011181 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7f39e815cdd0 con 0x7f39e80d0e10 2023-10-16T10:02:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f39cc06b020 msgr2=0x7f39cc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f39cc06b020 0x7f39cc06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39dc000c00 tx=0x7f39dc010000 comp rx=0 tx=0).stop 2023-10-16T10:02:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e80d0e10 msgr2=0x7f39e80c4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:01.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e80d0e10 0x7f39e80c4930 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c550 tx=0x7f39d400c910 comp rx=0 tx=0).stop 2023-10-16T10:02:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 shutdown_connections 2023-10-16T10:02:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39e8159280 0x7f39e815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39e80cbef0 0x7f39e81f4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f39cc06b020 0x7f39cc06d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 --2- 172.21.15.158:0/3713011181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39e80d0e10 0x7f39e80c4930 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 >> 172.21.15.158:0/3713011181 conn(0x7f39e8099a10 msgr2=0x7f39e815b290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.197+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 shutdown_connections 2023-10-16T10:02:01.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.198+0000 7f39eed8c700 1 -- 172.21.15.158:0/3713011181 wait complete. 2023-10-16T10:02:01.211 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T10:02:01.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.332+0000 7f68ad072700 1 Processor -- start 2023-10-16T10:02:01.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.332+0000 7f68ad072700 1 -- start start 2023-10-16T10:02:01.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68ad072700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68a80b1250 0x7f68a80b1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68ad072700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a806afe0 0x7f68a80ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68ad072700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80cd5a0 con 0x7f68a815b7c0 2023-10-16T10:02:01.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68ad072700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a80cc9f0 con 0x7f68a80b1250 2023-10-16T10:02:01.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68ad072700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80b1b70 con 0x7f68a806afe0 2023-10-16T10:02:01.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68a80b1250 0x7f68a80b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68a80b1250 0x7f68a80b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56346/0 (socket says 172.21.15.158:56346) 2023-10-16T10:02:01.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 -- 172.21.15.158:0/1699504370 learned_addr learned my addr 172.21.15.158:0/1699504370 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:01.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 -- 172.21.15.158:0/1699504370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a806afe0 msgr2=0x7f68a80ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:01.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a759e700 1 --2- 172.21.15.158:0/1699504370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a806afe0 0x7f68a80ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 --2- 172.21.15.158:0/1699504370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a806afe0 0x7f68a80ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 -- 172.21.15.158:0/1699504370 >> v1:172.21.15.172:6789/0 conn(0x7f68a815b7c0 legacy=0x7f68a806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 -- 172.21.15.158:0/1699504370 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80ccc70 con 0x7f68a80b1250 2023-10-16T10:02:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a6d9d700 1 --2- 172.21.15.158:0/1699504370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a80b1250 0x7f68a80b1630 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f68980098f0 tx=0x7f6898009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3956e29e091519c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a5d9b700 1 -- 172.21.15.158:0/1699504370 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898004230 con 0x7f68a80b1250 2023-10-16T10:02:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a5d9b700 1 -- 172.21.15.158:0/1699504370 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6898004390 con 0x7f68a80b1250 2023-10-16T10:02:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.333+0000 7f68a5d9b700 1 -- 172.21.15.158:0/1699504370 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68980036a0 con 0x7f68a80b1250 2023-10-16T10:02:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 -- 172.21.15.158:0/1699504370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a80b1250 msgr2=0x7f68a80b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 --2- 172.21.15.158:0/1699504370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a80b1250 0x7f68a80b1630 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f68980098f0 tx=0x7f6898009c00 comp rx=0 tx=0).stop 2023-10-16T10:02:01.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 -- 172.21.15.158:0/1699504370 shutdown_connections 2023-10-16T10:02:01.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 --2- 172.21.15.158:0/1699504370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a806afe0 0x7f68a80ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 --2- 172.21.15.158:0/1699504370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a80b1250 0x7f68a80b1630 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 -- 172.21.15.158:0/1699504370 >> 172.21.15.158:0/1699504370 conn(0x7f68a80d03a0 msgr2=0x7f68a80d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:01.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 -- 172.21.15.158:0/1699504370 shutdown_connections 2023-10-16T10:02:01.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.334+0000 7f68ad072700 1 -- 172.21.15.158:0/1699504370 wait complete. 2023-10-16T10:02:01.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.335+0000 7f68ad072700 1 Processor -- start 2023-10-16T10:02:01.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68ad072700 1 -- start start 2023-10-16T10:02:01.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68ad072700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68a806afe0 0x7f68a80cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68ad072700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a80b1250 0x7f68a80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68ad072700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a815b7c0 0x7f68a80b3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68ad072700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80cf7e0 con 0x7f68a815b7c0 2023-10-16T10:02:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68ad072700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80c6d60 con 0x7f68a806afe0 2023-10-16T10:02:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68ad072700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80c6a80 con 0x7f68a80b1250 2023-10-16T10:02:01.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68a759e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a815b7c0 0x7f68a80b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68a759e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a815b7c0 0x7f68a80b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56352/0 (socket says 172.21.15.158:56352) 2023-10-16T10:02:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.336+0000 7f68a759e700 1 -- 172.21.15.158:0/1190793263 learned_addr learned my addr 172.21.15.158:0/1190793263 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a6d9d700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68a806afe0 0x7f68a80cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a659c700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a80b1250 0x7f68a80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a759e700 1 -- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a80b1250 msgr2=0x7f68a80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a759e700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a80b1250 0x7f68a80b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a759e700 1 -- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68a806afe0 msgr2=0x7f68a80cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a759e700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68a806afe0 0x7f68a80cfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:01.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a759e700 1 -- 172.21.15.158:0/1190793263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6898009550 con 0x7f68a815b7c0 2023-10-16T10:02:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68a759e700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a815b7c0 0x7f68a80b3800 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f68a000cd70 tx=0x7f68a000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f688f7fe700 1 -- 172.21.15.158:0/1190793263 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0003a20 con 0x7f68a815b7c0 2023-10-16T10:02:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f688f7fe700 1 -- 172.21.15.158:0/1190793263 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0003b80 con 0x7f68a815b7c0 2023-10-16T10:02:01.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a80cf120 con 0x7f68a815b7c0 2023-10-16T10:02:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f688f7fe700 1 -- 172.21.15.158:0/1190793263 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0004020 con 0x7f68a815b7c0 2023-10-16T10:02:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.337+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a80c6710 con 0x7f68a815b7c0 2023-10-16T10:02:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.338+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a80cda10 con 0x7f68a815b7c0 2023-10-16T10:02:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.338+0000 7f688f7fe700 1 -- 172.21.15.158:0/1190793263 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f68a0004180 con 0x7f68a815b7c0 2023-10-16T10:02:01.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.339+0000 7f688f7fe700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f689406b020 0x7f689406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.339+0000 7f688f7fe700 1 -- 172.21.15.158:0/1190793263 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f68a0012070 con 0x7f68a815b7c0 2023-10-16T10:02:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.339+0000 7f68a6d9d700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f689406b020 0x7f689406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:01.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.339+0000 7f68a6d9d700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f689406b020 0x7f689406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6898009130 tx=0x7f6898008030 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:01.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.343+0000 7f688f7fe700 1 -- 172.21.15.158:0/1190793263 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f68a80cda10 con 0x7f68a815b7c0 2023-10-16T10:02:01.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:01.626+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f68a80cf920 con 0x7f68a815b7c0 2023-10-16T10:02:02.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.149+0000 7f688f7fe700 1 -- 172.21.15.158:0/1190793263 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v699) v1 ==== 183+0+0 (secure 0 0 0) 0x7f68a80cf920 con 0x7f68a815b7c0 2023-10-16T10:02:02.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f689406b020 msgr2=0x7f689406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f689406b020 0x7f689406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6898009130 tx=0x7f6898008030 comp rx=0 tx=0).stop 2023-10-16T10:02:02.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a815b7c0 msgr2=0x7f68a80b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a815b7c0 0x7f68a80b3800 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f68a000cd70 tx=0x7f68a000a570 comp rx=0 tx=0).stop 2023-10-16T10:02:02.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 shutdown_connections 2023-10-16T10:02:02.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f689406b020 0x7f689406d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68a815b7c0 0x7f68a80b3800 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68a806afe0 0x7f68a80cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 --2- 172.21.15.158:0/1190793263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68a80b1250 0x7f68a80b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 >> 172.21.15.158:0/1190793263 conn(0x7f68a80d03a0 msgr2=0x7f68a80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:02.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 shutdown_connections 2023-10-16T10:02:02.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.151+0000 7f68ad072700 1 -- 172.21.15.158:0/1190793263 wait complete. 2023-10-16T10:02:02.166 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-10-16T10:02:02.166 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-10-16T10:02:02.166 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T10:02:02.167 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:02:02.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.288+0000 7f107377c700 1 Processor -- start 2023-10-16T10:02:02.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.288+0000 7f107377c700 1 -- start start 2023-10-16T10:02:02.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f107377c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f107377c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f106c0ce2c0 0x7f106c1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f107377c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c15c5d0 con 0x7f106c0cdee0 2023-10-16T10:02:02.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f107377c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c15d2c0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f107377c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c152c20 con 0x7f106c0b6e10 2023-10-16T10:02:02.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1071d19700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f106c0ce2c0 0x7f106c1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1071d19700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f106c0ce2c0 0x7f106c1526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56362/0 (socket says 172.21.15.158:56362) 2023-10-16T10:02:02.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1071d19700 1 -- 172.21.15.158:0/2507641061 learned_addr learned my addr 172.21.15.158:0/2507641061 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:02.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1070d17700 1 --2- 172.21.15.158:0/2507641061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1071d19700 1 -- 172.21.15.158:0/2507641061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 msgr2=0x7f106c1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:02.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1071d19700 1 --2- 172.21.15.158:0/2507641061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1071d19700 1 -- 172.21.15.158:0/2507641061 >> v1:172.21.15.172:6789/0 conn(0x7f106c0cdee0 legacy=0x7f106c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:02.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1071d19700 1 -- 172.21.15.158:0/2507641061 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c15c8b0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.289+0000 7f1070d17700 1 --2- 172.21.15.158:0/2507641061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c1521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:02.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f1071d19700 1 --2- 172.21.15.158:0/2507641061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c1526e0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f1064009e50 tx=0x7f106400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85a4923fd8a614a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:02.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f1063fff700 1 -- 172.21.15.158:0/2507641061 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106400e070 con 0x7f106c0ce2c0 2023-10-16T10:02:02.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f1063fff700 1 -- 172.21.15.158:0/2507641061 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064003a20 con 0x7f106c0ce2c0 2023-10-16T10:02:02.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f1063fff700 1 -- 172.21.15.158:0/2507641061 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064003dd0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 -- 172.21.15.158:0/2507641061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 msgr2=0x7f106c1526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 --2- 172.21.15.158:0/2507641061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c1526e0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f1064009e50 tx=0x7f106400b330 comp rx=0 tx=0).stop 2023-10-16T10:02:02.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 -- 172.21.15.158:0/2507641061 shutdown_connections 2023-10-16T10:02:02.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 --2- 172.21.15.158:0/2507641061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c1526e0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 --2- 172.21.15.158:0/2507641061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c1521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 -- 172.21.15.158:0/2507641061 >> 172.21.15.158:0/2507641061 conn(0x7f106c06a8d0 msgr2=0x7f106c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:02.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 -- 172.21.15.158:0/2507641061 shutdown_connections 2023-10-16T10:02:02.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.290+0000 7f107377c700 1 -- 172.21.15.158:0/2507641061 wait complete. 2023-10-16T10:02:02.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.292+0000 7f107377c700 1 Processor -- start 2023-10-16T10:02:02.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.292+0000 7f107377c700 1 -- start start 2023-10-16T10:02:02.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f107377c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f107377c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106c0cdee0 0x7f106c14fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f107377c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f107377c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c150270 con 0x7f106c0ce2c0 2023-10-16T10:02:02.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f107377c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c06ca50 con 0x7f106c0cdee0 2023-10-16T10:02:02.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f107377c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c15adc0 con 0x7f106c0b6e10 2023-10-16T10:02:02.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c06c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56368/0 (socket says 172.21.15.158:56368) 2023-10-16T10:02:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 -- 172.21.15.158:0/380744445 learned_addr learned my addr 172.21.15.158:0/380744445 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1070d17700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106c0cdee0 0x7f106c14fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071518700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 -- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 msgr2=0x7f106c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c15a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 -- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106c0cdee0 msgr2=0x7f106c14fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106c0cdee0 0x7f106c14fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1071d19700 1 -- 172.21.15.158:0/380744445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064009b30 con 0x7f106c0ce2c0 2023-10-16T10:02:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.293+0000 7f1070d17700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106c0cdee0 0x7f106c14fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.294+0000 7f1071d19700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c06c510 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f106c0ced40 tx=0x7f1064004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.294+0000 7f1061ffb700 1 -- 172.21.15.158:0/380744445 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064004440 con 0x7f106c0ce2c0 2023-10-16T10:02:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.294+0000 7f1061ffb700 1 -- 172.21.15.158:0/380744445 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10640045a0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.294+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c0c5ff0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.294+0000 7f1061ffb700 1 -- 172.21.15.158:0/380744445 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064004950 con 0x7f106c0ce2c0 2023-10-16T10:02:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.294+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c0c65b0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.295+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c0681b0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.295+0000 7f1061ffb700 1 -- 172.21.15.158:0/380744445 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1064012430 con 0x7f106c0ce2c0 2023-10-16T10:02:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.296+0000 7f1061ffb700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f105006b020 0x7f105006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.296+0000 7f1061ffb700 1 -- 172.21.15.158:0/380744445 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f106408b140 con 0x7f106c0ce2c0 2023-10-16T10:02:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.296+0000 7f1071518700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f105006b020 0x7f105006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.296+0000 7f1071518700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f105006b020 0x7f105006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f105c004f30 tx=0x7f105c009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:02.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.300+0000 7f1061ffb700 1 -- 172.21.15.158:0/380744445 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1064055cf0 con 0x7f106c0ce2c0 2023-10-16T10:02:02.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.491+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f106c0b4b80 con 0x7f106c0ce2c0 2023-10-16T10:02:02.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.492+0000 7f1061ffb700 1 -- 172.21.15.158:0/380744445 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+15004 (secure 0 0 0) 0x7f1064059310 con 0x7f106c0ce2c0 2023-10-16T10:02:02.494 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:02.494 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":118,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:00.145832+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6835/1596548674":"2023-10-17T10:01:57.119460+0000","172.21.15.172:6833/1596548674":"2023-10-17T10:01:57.119460+0000","172.21.15.158:6833/1516256968":"2023-10-17T10:01:56.116841+0000","172.21.15.158:6832/1516256968":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3809361082":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3884887886":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/1766729069":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/2379821640":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3839669547":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3538867905":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/3020719129":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/934459119":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/1239292381":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6833/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:6832/1430981267":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2498623628":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2203508586":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2678240587":"2023-10-17T10:01:56.116841+0000","172.21.15.158:0/521051078":"2023-10-17T10:00:47.874811+0000","172.21.15.158:0/2930980190":"2023-10-17T10:00:47.874811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:02:02.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.494+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f105006b020 msgr2=0x7f105006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.494+0000 7f107377c700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f105006b020 0x7f105006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f105c004f30 tx=0x7f105c009040 comp rx=0 tx=0).stop 2023-10-16T10:02:02.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 msgr2=0x7f106c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c06c510 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f106c0ced40 tx=0x7f1064004ad0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 shutdown_connections 2023-10-16T10:02:02.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f105006b020 0x7f105006d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f106c0ce2c0 0x7f106c06c510 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f106c0b6e10 0x7f106c15a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 --2- 172.21.15.158:0/380744445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f106c0cdee0 0x7f106c14fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 >> 172.21.15.158:0/380744445 conn(0x7f106c06a8d0 msgr2=0x7f106c150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:02.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 shutdown_connections 2023-10-16T10:02:02.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.495+0000 7f107377c700 1 -- 172.21.15.158:0/380744445 wait complete. 2023-10-16T10:02:02.508 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:02:02.509 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:02:02.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.632+0000 7f5ffec98700 1 Processor -- start 2023-10-16T10:02:02.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.632+0000 7f5ffec98700 1 -- start start 2023-10-16T10:02:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.633+0000 7f5ffec98700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ff806afe0 0x7f5ff80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.633+0000 7f5ffec98700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff80cbd30 con 0x7f5ff80cc590 2023-10-16T10:02:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.633+0000 7f5ffec98700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff80cc1c0 con 0x7f5ff815da40 2023-10-16T10:02:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.633+0000 7f5ffec98700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff815de20 con 0x7f5ff806afe0 2023-10-16T10:02:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ff806afe0 0x7f5ff80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ffd235700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5ff80cc590 0x7f5ff80afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56548/0 (socket says 172.21.15.158:56548) 2023-10-16T10:02:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ff806afe0 0x7f5ff80caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56372/0 (socket says 172.21.15.158:56372) 2023-10-16T10:02:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ffd235700 1 -- 172.21.15.158:0/412188177 learned_addr learned my addr 172.21.15.158:0/412188177 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff7fff700 1 -- 172.21.15.158:0/412188177 >> v1:172.21.15.172:6789/0 conn(0x7f5ff815da40 legacy=0x7f5ff806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff7fff700 1 -- 172.21.15.158:0/412188177 >> v1:172.21.15.158:6789/0 conn(0x7f5ff80cc590 legacy=0x7f5ff80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff77fe700 1 -- 172.21.15.158:0/412188177 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361834092 0 0) 0x7f5ff80cbd30 con 0x7f5ff80cc590 2023-10-16T10:02:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff7fff700 1 -- 172.21.15.158:0/412188177 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff80ce260 con 0x7f5ff806afe0 2023-10-16T10:02:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff7fff700 1 --2- 172.21.15.158:0/412188177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff80caa20 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009840 tx=0x7f5fe4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aaa77eab0a4aa68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff77fe700 1 -- 172.21.15.158:0/412188177 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4011830 con 0x7f5ff806afe0 2023-10-16T10:02:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff77fe700 1 -- 172.21.15.158:0/412188177 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe4011990 con 0x7f5ff806afe0 2023-10-16T10:02:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.634+0000 7f5ff77fe700 1 -- 172.21.15.158:0/412188177 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4011d20 con 0x7f5ff806afe0 2023-10-16T10:02:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.635+0000 7f5ffec98700 1 -- 172.21.15.158:0/412188177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 msgr2=0x7f5ff80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.635+0000 7f5ffec98700 1 --2- 172.21.15.158:0/412188177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff80caa20 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009840 tx=0x7f5fe4009b50 comp rx=0 tx=0).stop 2023-10-16T10:02:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.635+0000 7f5ffec98700 1 -- 172.21.15.158:0/412188177 shutdown_connections 2023-10-16T10:02:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.635+0000 7f5ffec98700 1 --2- 172.21.15.158:0/412188177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff80caa20 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.635+0000 7f5ffec98700 1 -- 172.21.15.158:0/412188177 >> 172.21.15.158:0/412188177 conn(0x7f5ff8099a10 msgr2=0x7f5ff80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.635+0000 7f5ffec98700 1 -- 172.21.15.158:0/412188177 shutdown_connections 2023-10-16T10:02:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.635+0000 7f5ffec98700 1 -- 172.21.15.158:0/412188177 wait complete. 2023-10-16T10:02:02.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.636+0000 7f5ffec98700 1 Processor -- start 2023-10-16T10:02:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.636+0000 7f5ffec98700 1 -- start start 2023-10-16T10:02:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.637+0000 7f5ffec98700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff81575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.637+0000 7f5ffec98700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ff80cc590 0x7f5ff80c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.637+0000 7f5ffec98700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ff815da40 0x7f5ff8159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.637+0000 7f5ffec98700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff80c8a20 con 0x7f5ff806afe0 2023-10-16T10:02:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.637+0000 7f5ffec98700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff80c7da0 con 0x7f5ff815da40 2023-10-16T10:02:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.637+0000 7f5ffec98700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff80c8080 con 0x7f5ff80cc590 2023-10-16T10:02:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff81575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff81575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56374/0 (socket says 172.21.15.158:56374) 2023-10-16T10:02:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 -- 172.21.15.158:0/3240642539 learned_addr learned my addr 172.21.15.158:0/3240642539 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ff7fff700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ff80cc590 0x7f5ff80c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffd235700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ff815da40 0x7f5ff8159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 -- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ff80cc590 msgr2=0x7f5ff80c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ff80cc590 0x7f5ff80c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 -- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ff815da40 msgr2=0x7f5ff8159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ff815da40 0x7f5ff8159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 -- 172.21.15.158:0/3240642539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40094a0 con 0x7f5ff806afe0 2023-10-16T10:02:02.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ff7fff700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ff80cc590 0x7f5ff80c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:02.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffca34700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff81575b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00c450 tx=0x7f5fec00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:02.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ff57fa700 1 -- 172.21.15.158:0/3240642539 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0052d0 con 0x7f5ff806afe0 2023-10-16T10:02:02.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ff57fa700 1 -- 172.21.15.158:0/3240642539 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec005430 con 0x7f5ff806afe0 2023-10-16T10:02:02.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff80c8360 con 0x7f5ff806afe0 2023-10-16T10:02:02.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ff57fa700 1 -- 172.21.15.158:0/3240642539 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0038c0 con 0x7f5ff806afe0 2023-10-16T10:02:02.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.638+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff80c79e0 con 0x7f5ff806afe0 2023-10-16T10:02:02.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.639+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff80681b0 con 0x7f5ff806afe0 2023-10-16T10:02:02.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.640+0000 7f5ff57fa700 1 -- 172.21.15.158:0/3240642539 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5fec003a20 con 0x7f5ff806afe0 2023-10-16T10:02:02.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.640+0000 7f5ff57fa700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5fdc06b020 0x7f5fdc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:02.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.640+0000 7f5ff57fa700 1 -- 172.21.15.158:0/3240642539 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f5fec088da0 con 0x7f5ff806afe0 2023-10-16T10:02:02.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.640+0000 7f5ff7fff700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5fdc06b020 0x7f5fdc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:02.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.641+0000 7f5ff7fff700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5fdc06b020 0x7f5fdc06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ff806dd10 tx=0x7f5fe401c040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:02.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.645+0000 7f5ff57fa700 1 -- 172.21.15.158:0/3240642539 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5fec053a50 con 0x7f5ff806afe0 2023-10-16T10:02:02.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.838+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ff806dd40 con 0x7f5ff806afe0 2023-10-16T10:02:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.839+0000 7f5ff57fa700 1 -- 172.21.15.158:0/3240642539 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fec057070 con 0x7f5ff806afe0 2023-10-16T10:02:02.872 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:02:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5fdc06b020 msgr2=0x7f5fdc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5fdc06b020 0x7f5fdc06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ff806dd10 tx=0x7f5fe401c040 comp rx=0 tx=0).stop 2023-10-16T10:02:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 msgr2=0x7f5ff81575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff81575b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00c450 tx=0x7f5fec00c810 comp rx=0 tx=0).stop 2023-10-16T10:02:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 shutdown_connections 2023-10-16T10:02:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ff815da40 0x7f5ff8159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5fdc06b020 0x7f5fdc06d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ff806afe0 0x7f5ff81575b0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 --2- 172.21.15.158:0/3240642539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ff80cc590 0x7f5ff80c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:02.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 >> 172.21.15.158:0/3240642539 conn(0x7f5ff8099a10 msgr2=0x7f5ff815cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:02.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 shutdown_connections 2023-10-16T10:02:02.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.841+0000 7f5ffec98700 1 -- 172.21.15.158:0/3240642539 wait complete. 2023-10-16T10:02:02.876 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T10:02:02.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.997+0000 7f31b31f7700 1 Processor -- start 2023-10-16T10:02:02.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.997+0000 7f31b31f7700 1 -- start start 2023-10-16T10:02:03.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.997+0000 7f31b31f7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f31ac0cdee0 0x7f31ac15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:03.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.997+0000 7f31b31f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0b6e10 0x7f31ac1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:03.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.997+0000 7f31b31f7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ac15c5d0 con 0x7f31ac0ce2c0 2023-10-16T10:02:03.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.997+0000 7f31b31f7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac15d2c0 con 0x7f31ac0cdee0 2023-10-16T10:02:03.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.997+0000 7f31b31f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac15c8b0 con 0x7f31ac0b6e10 2023-10-16T10:02:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0b6e10 0x7f31ac1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31b0f93700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f31ac0cdee0 0x7f31ac15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0b6e10 0x7f31ac1521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:38984/0 (socket says 172.21.15.158:38984) 2023-10-16T10:02:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 -- 172.21.15.158:0/2202499430 learned_addr learned my addr 172.21.15.158:0/2202499430 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 -- 172.21.15.158:0/2202499430 >> v2:172.21.15.172:3300/0 conn(0x7f31ac0cdee0 msgr2=0x7f31ac15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 --2- 172.21.15.158:0/2202499430 >> v2:172.21.15.172:3300/0 conn(0x7f31ac0cdee0 0x7f31ac15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 -- 172.21.15.158:0/2202499430 >> v1:172.21.15.172:6789/0 conn(0x7f31ac0ce2c0 legacy=0x7f31ac0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 -- 172.21.15.158:0/2202499430 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac15cb90 con 0x7f31ac0b6e10 2023-10-16T10:02:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31b0f93700 1 --2- 172.21.15.158:0/2202499430 >> v2:172.21.15.172:3300/0 conn(0x7f31ac0cdee0 0x7f31ac15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:02:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.998+0000 7f31abfff700 1 --2- 172.21.15.158:0/2202499430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0b6e10 0x7f31ac1521a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f31980098f0 tx=0x7f3198009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43a0488a919de9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:03.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31ab7fe700 1 -- 172.21.15.158:0/2202499430 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198011830 con 0x7f31ac0b6e10 2023-10-16T10:02:03.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31ab7fe700 1 -- 172.21.15.158:0/2202499430 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198011990 con 0x7f31ac0b6e10 2023-10-16T10:02:03.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31ab7fe700 1 -- 172.21.15.158:0/2202499430 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198011d20 con 0x7f31ac0b6e10 2023-10-16T10:02:03.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 -- 172.21.15.158:0/2202499430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0b6e10 msgr2=0x7f31ac1521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:03.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 --2- 172.21.15.158:0/2202499430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0b6e10 0x7f31ac1521a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f31980098f0 tx=0x7f3198009c00 comp rx=0 tx=0).stop 2023-10-16T10:02:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 -- 172.21.15.158:0/2202499430 shutdown_connections 2023-10-16T10:02:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 --2- 172.21.15.158:0/2202499430 >> v2:172.21.15.172:3300/0 conn(0x7f31ac0cdee0 0x7f31ac15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 --2- 172.21.15.158:0/2202499430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0b6e10 0x7f31ac1521a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 -- 172.21.15.158:0/2202499430 >> 172.21.15.158:0/2202499430 conn(0x7f31ac06a8d0 msgr2=0x7f31ac06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 -- 172.21.15.158:0/2202499430 shutdown_connections 2023-10-16T10:02:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:02.999+0000 7f31b31f7700 1 -- 172.21.15.158:0/2202499430 wait complete. 2023-10-16T10:02:03.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.000+0000 7f31b31f7700 1 Processor -- start 2023-10-16T10:02:03.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.000+0000 7f31b31f7700 1 -- start start 2023-10-16T10:02:03.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31b31f7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31ac0b6e10 0x7f31ac14b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:03.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31b31f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31ac0cdee0 0x7f31ac06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:03.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31b31f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0ce2c0 0x7f31ac15a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:03.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31b31f7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac14bee0 con 0x7f31ac0cdee0 2023-10-16T10:02:03.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31b31f7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac06c2e0 con 0x7f31ac0b6e10 2023-10-16T10:02:03.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31b31f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac06c5c0 con 0x7f31ac0ce2c0 2023-10-16T10:02:03.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31abfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31ac0cdee0 0x7f31ac06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:03.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31abfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31ac0cdee0 0x7f31ac06b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56388/0 (socket says 172.21.15.158:56388) 2023-10-16T10:02:03.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.001+0000 7f31abfff700 1 -- 172.21.15.158:0/1945394925 learned_addr learned my addr 172.21.15.158:0/1945394925 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:03.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31b0f93700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31ac0b6e10 0x7f31ac14b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:03.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31b1794700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0ce2c0 0x7f31ac15a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:03.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31abfff700 1 -- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0ce2c0 msgr2=0x7f31ac15a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:03.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31abfff700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0ce2c0 0x7f31ac15a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:03.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31abfff700 1 -- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31ac0b6e10 msgr2=0x7f31ac14b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:03.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31abfff700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31ac0b6e10 0x7f31ac14b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:03.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31abfff700 1 -- 172.21.15.158:0/1945394925 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00094a0 con 0x7f31ac0cdee0 2023-10-16T10:02:03.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31abfff700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31ac0cdee0 0x7f31ac06b9e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f3198000c00 tx=0x7f3198004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:03.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31a97fa700 1 -- 172.21.15.158:0/1945394925 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198004570 con 0x7f31ac0cdee0 2023-10-16T10:02:03.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31a97fa700 1 -- 172.21.15.158:0/1945394925 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319801f070 con 0x7f31ac0cdee0 2023-10-16T10:02:03.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31a97fa700 1 -- 172.21.15.158:0/1945394925 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980038c0 con 0x7f31ac0cdee0 2023-10-16T10:02:03.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3198009550 con 0x7f31ac0cdee0 2023-10-16T10:02:03.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.002+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac06cb80 con 0x7f31ac0cdee0 2023-10-16T10:02:03.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.003+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ac0681b0 con 0x7f31ac0cdee0 2023-10-16T10:02:03.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.003+0000 7f31a97fa700 1 -- 172.21.15.158:0/1945394925 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3198021030 con 0x7f31ac0cdee0 2023-10-16T10:02:03.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.004+0000 7f31a97fa700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f319006b020 0x7f319006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:03.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.004+0000 7f31a97fa700 1 -- 172.21.15.158:0/1945394925 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f319808aea0 con 0x7f31ac0cdee0 2023-10-16T10:02:03.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.004+0000 7f31b0f93700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f319006b020 0x7f319006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:03.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.004+0000 7f31b0f93700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f319006b020 0x7f319006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009c20 tx=0x7f31a0008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:03.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.008+0000 7f31a97fa700 1 -- 172.21.15.158:0/1945394925 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3198055d00 con 0x7f31ac0cdee0 2023-10-16T10:02:03.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:03.283+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f31ac0b3bf0 con 0x7f31ac0cdee0 2023-10-16T10:02:04.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.151+0000 7f31a97fa700 1 -- 172.21.15.158:0/1945394925 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v701) v1 ==== 151+0+0 (secure 0 0 0) 0x7f3198015020 con 0x7f31ac0cdee0 2023-10-16T10:02:04.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.155+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f319006b020 msgr2=0x7f319006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.155+0000 7f31b31f7700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f319006b020 0x7f319006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009c20 tx=0x7f31a0008040 comp rx=0 tx=0).stop 2023-10-16T10:02:04.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.155+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31ac0cdee0 msgr2=0x7f31ac06b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.155+0000 7f31b31f7700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31ac0cdee0 0x7f31ac06b9e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f3198000c00 tx=0x7f3198004750 comp rx=0 tx=0).stop 2023-10-16T10:02:04.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 shutdown_connections 2023-10-16T10:02:04.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31ac0ce2c0 0x7f31ac15a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31ac0b6e10 0x7f31ac14b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f319006b020 0x7f319006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 --2- 172.21.15.158:0/1945394925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31ac0cdee0 0x7f31ac06b9e0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 >> 172.21.15.158:0/1945394925 conn(0x7f31ac06a8d0 msgr2=0x7f31ac150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:04.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 shutdown_connections 2023-10-16T10:02:04.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.156+0000 7f31b31f7700 1 -- 172.21.15.158:0/1945394925 wait complete. 2023-10-16T10:02:04.173 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T10:02:04.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.305+0000 7efcd8a8c700 1 Processor -- start 2023-10-16T10:02:04.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.305+0000 7efcd8a8c700 1 -- start start 2023-10-16T10:02:04.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd8a8c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efcd415deb0 0x7efcd40ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd8a8c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd40d0ae0 con 0x7efcd415dad0 2023-10-16T10:02:04.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd8a8c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd40d12d0 con 0x7efcd40ccc00 2023-10-16T10:02:04.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd8a8c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd40cbdf0 con 0x7efcd415deb0 2023-10-16T10:02:04.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efcd415deb0 0x7efcd40ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efcd415dad0 0x7efcd40cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56572/0 (socket says 172.21.15.158:56572) 2023-10-16T10:02:04.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd259c700 1 -- 172.21.15.158:0/3897275608 learned_addr learned my addr 172.21.15.158:0/3897275608 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:04.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd1d9b700 1 -- 172.21.15.158:0/3897275608 >> v1:172.21.15.172:6789/0 conn(0x7efcd40ccc00 legacy=0x7efcd40cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:04.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd1d9b700 1 -- 172.21.15.158:0/3897275608 >> v1:172.21.15.158:6789/0 conn(0x7efcd415dad0 legacy=0x7efcd40cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd1d9b700 1 -- 172.21.15.158:0/3897275608 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40cc280 con 0x7efcd415deb0 2023-10-16T10:02:04.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd159a700 1 -- 172.21.15.158:0/3897275608 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789746442 0 0) 0x7efcd40d0ae0 con 0x7efcd415dad0 2023-10-16T10:02:04.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.306+0000 7efcd1d9b700 1 --2- 172.21.15.158:0/3897275608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415deb0 0x7efcd40ce6b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009870 tx=0x7efcc4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88756a3e0750c770 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:04.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd159a700 1 -- 172.21.15.158:0/3897275608 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4004230 con 0x7efcd415deb0 2023-10-16T10:02:04.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd159a700 1 -- 172.21.15.158:0/3897275608 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4004390 con 0x7efcd415deb0 2023-10-16T10:02:04.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd159a700 1 -- 172.21.15.158:0/3897275608 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc40036a0 con 0x7efcd415deb0 2023-10-16T10:02:04.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd8a8c700 1 -- 172.21.15.158:0/3897275608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415deb0 msgr2=0x7efcd40ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/3897275608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415deb0 0x7efcd40ce6b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009870 tx=0x7efcc4009b80 comp rx=0 tx=0).stop 2023-10-16T10:02:04.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd8a8c700 1 -- 172.21.15.158:0/3897275608 shutdown_connections 2023-10-16T10:02:04.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/3897275608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415deb0 0x7efcd40ce6b0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd8a8c700 1 -- 172.21.15.158:0/3897275608 >> 172.21.15.158:0/3897275608 conn(0x7efcd4099a10 msgr2=0x7efcd40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:04.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd8a8c700 1 -- 172.21.15.158:0/3897275608 shutdown_connections 2023-10-16T10:02:04.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.307+0000 7efcd8a8c700 1 -- 172.21.15.158:0/3897275608 wait complete. 2023-10-16T10:02:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 Processor -- start 2023-10-16T10:02:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 -- start start 2023-10-16T10:02:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd40ccc00 0x7efcd4155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415dad0 0x7efcd406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd415deb0 0x7efcd406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415a440 con 0x7efcd415dad0 2023-10-16T10:02:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40b4150 con 0x7efcd415deb0 2023-10-16T10:02:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd8a8c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40b4430 con 0x7efcd40ccc00 2023-10-16T10:02:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.309+0000 7efcd1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415dad0 0x7efcd406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415dad0 0x7efcd406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56406/0 (socket says 172.21.15.158:56406) 2023-10-16T10:02:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 -- 172.21.15.158:0/2430101491 learned_addr learned my addr 172.21.15.158:0/2430101491 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd2d9d700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd415deb0 0x7efcd406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd259c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd40ccc00 0x7efcd4155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 -- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd40ccc00 msgr2=0x7efcd4155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd40ccc00 0x7efcd4155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 -- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd415deb0 msgr2=0x7efcd406ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd415deb0 0x7efcd406ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 -- 172.21.15.158:0/2430101491 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc40094a0 con 0x7efcd415dad0 2023-10-16T10:02:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd259c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd40ccc00 0x7efcd4155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd1d9b700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415dad0 0x7efcd406a8d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009840 tx=0x7efcc40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcc2ffd700 1 -- 172.21.15.158:0/2430101491 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4011830 con 0x7efcd415dad0 2023-10-16T10:02:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcc2ffd700 1 -- 172.21.15.158:0/2430101491 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4011990 con 0x7efcd415dad0 2023-10-16T10:02:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd415a690 con 0x7efcd415dad0 2023-10-16T10:02:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcc2ffd700 1 -- 172.21.15.158:0/2430101491 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4011d20 con 0x7efcd415dad0 2023-10-16T10:02:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.310+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd40b3460 con 0x7efcd415dad0 2023-10-16T10:02:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.311+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd415abb0 con 0x7efcd415dad0 2023-10-16T10:02:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.312+0000 7efcc2ffd700 1 -- 172.21.15.158:0/2430101491 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efcc401b440 con 0x7efcd415dad0 2023-10-16T10:02:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.312+0000 7efcc2ffd700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7efcb406b020 0x7efcb406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.312+0000 7efcc2ffd700 1 -- 172.21.15.158:0/2430101491 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7efcc408b310 con 0x7efcd415dad0 2023-10-16T10:02:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.312+0000 7efcd259c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7efcb406b020 0x7efcb406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.313+0000 7efcd259c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7efcb406b020 0x7efcb406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efcc8000f60 tx=0x7efcc8007ca0 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.316+0000 7efcc2ffd700 1 -- 172.21.15.158:0/2430101491 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efcc4056030 con 0x7efcd415dad0 2023-10-16T10:02:04.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.509+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efcd415abb0 con 0x7efcd415dad0 2023-10-16T10:02:04.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.512+0000 7efcc2ffd700 1 -- 172.21.15.158:0/2430101491 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7efcd415abb0 con 0x7efcd415dad0 2023-10-16T10:02:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.514+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7efcb406b020 msgr2=0x7efcb406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.514+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7efcb406b020 0x7efcb406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efcc8000f60 tx=0x7efcc8007ca0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.514+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415dad0 msgr2=0x7efcd406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.514+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415dad0 0x7efcd406a8d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009840 tx=0x7efcc40047c0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.514+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 shutdown_connections 2023-10-16T10:02:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.515+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd415deb0 0x7efcd406ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.515+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efcd40ccc00 0x7efcd4155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.515+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7efcb406b020 0x7efcb406d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.515+0000 7efcd8a8c700 1 --2- 172.21.15.158:0/2430101491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcd415dad0 0x7efcd406a8d0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.515+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 >> 172.21.15.158:0/2430101491 conn(0x7efcd4099a10 msgr2=0x7efcd4152a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:04.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.515+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 shutdown_connections 2023-10-16T10:02:04.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.515+0000 7efcd8a8c700 1 -- 172.21.15.158:0/2430101491 wait complete. 2023-10-16T10:02:04.528 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:02:04.528 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:02:04.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:02:04.618 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:02:04.618 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:02:04.619 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:02:04.636 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:02:04.637 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:02:04.637 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:02:04.657 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:02:04.657 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:02:04.657 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:02:04.694 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:02:04.694 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:02:04.695 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:02:04.716 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:02:04.716 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T10:02:04.717 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:02:04.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.853+0000 7fdf615ba700 1 Processor -- start 2023-10-16T10:02:04.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.853+0000 7fdf615ba700 1 -- start start 2023-10-16T10:02:04.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.854+0000 7fdf615ba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06ab40 0x7fdf5c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.854+0000 7fdf615ba700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c161270 con 0x7fdf5c0729b0 2023-10-16T10:02:04.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.854+0000 7fdf615ba700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c161510 con 0x7fdf5c06e5d0 2023-10-16T10:02:04.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.854+0000 7fdf615ba700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c1617f0 con 0x7fdf5c06ab40 2023-10-16T10:02:04.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.854+0000 7fdf5b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdf5c0729b0 0x7fdf5c15df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56588/0 (socket says 172.21.15.158:56588) 2023-10-16T10:02:04.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.854+0000 7fdf5b7fe700 1 -- 172.21.15.158:0/4064280360 learned_addr learned my addr 172.21.15.158:0/4064280360 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:04.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf5affd700 1 --2- 172.21.15.158:0/4064280360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06ab40 0x7fdf5c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf59ffb700 1 -- 172.21.15.158:0/4064280360 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607837868 0 0) 0x7fdf5c161270 con 0x7fdf5c0729b0 2023-10-16T10:02:04.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf59ffb700 1 -- 172.21.15.158:0/4064280360 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf440035b0 con 0x7fdf5c0729b0 2023-10-16T10:02:04.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf5affd700 1 -- 172.21.15.158:0/4064280360 >> v1:172.21.15.172:6789/0 conn(0x7fdf5c06e5d0 legacy=0x7fdf5c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf59ffb700 1 -- 172.21.15.158:0/4064280360 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408067680 0 0) 0x7fdf5c161510 con 0x7fdf5c06e5d0 2023-10-16T10:02:04.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf5affd700 1 -- 172.21.15.158:0/4064280360 >> v1:172.21.15.158:6789/0 conn(0x7fdf5c0729b0 legacy=0x7fdf5c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf5affd700 1 -- 172.21.15.158:0/4064280360 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c161ad0 con 0x7fdf5c06ab40 2023-10-16T10:02:04.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf5affd700 1 --2- 172.21.15.158:0/4064280360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06ab40 0x7fdf5c06d820 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009870 tx=0x7fdf50009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3e4c58831390abd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:04.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf59ffb700 1 -- 172.21.15.158:0/4064280360 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50011830 con 0x7fdf5c06ab40 2023-10-16T10:02:04.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf59ffb700 1 -- 172.21.15.158:0/4064280360 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50011990 con 0x7fdf5c06ab40 2023-10-16T10:02:04.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.855+0000 7fdf59ffb700 1 -- 172.21.15.158:0/4064280360 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50011d00 con 0x7fdf5c06ab40 2023-10-16T10:02:04.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.856+0000 7fdf615ba700 1 -- 172.21.15.158:0/4064280360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06ab40 msgr2=0x7fdf5c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.856+0000 7fdf615ba700 1 --2- 172.21.15.158:0/4064280360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06ab40 0x7fdf5c06d820 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009870 tx=0x7fdf50009b80 comp rx=0 tx=0).stop 2023-10-16T10:02:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.856+0000 7fdf615ba700 1 -- 172.21.15.158:0/4064280360 shutdown_connections 2023-10-16T10:02:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.856+0000 7fdf615ba700 1 --2- 172.21.15.158:0/4064280360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06ab40 0x7fdf5c06d820 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.856+0000 7fdf615ba700 1 -- 172.21.15.158:0/4064280360 >> 172.21.15.158:0/4064280360 conn(0x7fdf5c0d02b0 msgr2=0x7fdf5c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.856+0000 7fdf615ba700 1 -- 172.21.15.158:0/4064280360 shutdown_connections 2023-10-16T10:02:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.856+0000 7fdf615ba700 1 -- 172.21.15.158:0/4064280360 wait complete. 2023-10-16T10:02:04.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.857+0000 7fdf615ba700 1 Processor -- start 2023-10-16T10:02:04.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.857+0000 7fdf615ba700 1 -- start start 2023-10-16T10:02:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf615ba700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf5c06ab40 0x7fdf5c0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf615ba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06e5d0 0x7fdf5c0c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf615ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5c0729b0 0x7fdf5c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf615ba700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0cbdf0 con 0x7fdf5c0729b0 2023-10-16T10:02:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf615ba700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c1f2c50 con 0x7fdf5c06ab40 2023-10-16T10:02:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf615ba700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c1f2f30 con 0x7fdf5c06e5d0 2023-10-16T10:02:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5c0729b0 0x7fdf5c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5c0729b0 0x7fdf5c0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56418/0 (socket says 172.21.15.158:56418) 2023-10-16T10:02:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 -- 172.21.15.158:0/1330384128 learned_addr learned my addr 172.21.15.158:0/1330384128 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5a7fc700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06e5d0 0x7fdf5c0c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5affd700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf5c06ab40 0x7fdf5c0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 -- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06e5d0 msgr2=0x7fdf5c0c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06e5d0 0x7fdf5c0c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 -- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf5c06ab40 msgr2=0x7fdf5c0c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:04.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf5c06ab40 0x7fdf5c0c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:04.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 -- 172.21.15.158:0/1330384128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf500094a0 con 0x7fdf5c0729b0 2023-10-16T10:02:04.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.858+0000 7fdf5b7fe700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5c0729b0 0x7fdf5c0c9a90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400ca70 tx=0x7fdf5400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:04.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.859+0000 7fdf437fe700 1 -- 172.21.15.158:0/1330384128 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5400a9e0 con 0x7fdf5c0729b0 2023-10-16T10:02:04.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.859+0000 7fdf437fe700 1 -- 172.21.15.158:0/1330384128 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5400ab40 con 0x7fdf5c0729b0 2023-10-16T10:02:04.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.859+0000 7fdf437fe700 1 -- 172.21.15.158:0/1330384128 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54012400 con 0x7fdf5c0729b0 2023-10-16T10:02:04.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.859+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c1f3210 con 0x7fdf5c0729b0 2023-10-16T10:02:04.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.859+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c1f28c0 con 0x7fdf5c0729b0 2023-10-16T10:02:04.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.860+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c1f3430 con 0x7fdf5c0729b0 2023-10-16T10:02:04.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.860+0000 7fdf437fe700 1 -- 172.21.15.158:0/1330384128 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf5401b3b0 con 0x7fdf5c0729b0 2023-10-16T10:02:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.860+0000 7fdf437fe700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdf4806b020 0x7fdf4806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.861+0000 7fdf437fe700 1 -- 172.21.15.158:0/1330384128 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fdf54089450 con 0x7fdf5c0729b0 2023-10-16T10:02:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.861+0000 7fdf5affd700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdf4806b020 0x7fdf4806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.861+0000 7fdf5affd700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdf4806b020 0x7fdf4806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf500097c0 tx=0x7fdf50008030 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:04.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:04.865+0000 7fdf437fe700 1 -- 172.21.15.158:0/1330384128 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf54054240 con 0x7fdf5c0729b0 2023-10-16T10:02:05.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.102+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf5c0681b0 con 0x7fdf5c0729b0 2023-10-16T10:02:05.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.102+0000 7fdf437fe700 1 -- 172.21.15.158:0/1330384128 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+3711 (secure 0 0 0) 0x7fdf54057860 con 0x7fdf5c0729b0 2023-10-16T10:02:05.104 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:05.104 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3713945529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3713945529},{"type":"v1","addr":"172.21.15.172:6835","nonce":3713945529}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2674019415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2674019415},{"type":"v1","addr":"172.21.15.158:6836","nonce":2674019415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2425279024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2425279024},{"type":"v1","addr":"172.21.15.158:6837","nonce":2425279024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3002995342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3002995342},{"type":"v1","addr":"172.21.15.172:6834","nonce":3002995342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[]} 2023-10-16T10:02:05.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.104+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdf4806b020 msgr2=0x7fdf4806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:05.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.104+0000 7fdf615ba700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdf4806b020 0x7fdf4806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf500097c0 tx=0x7fdf50008030 comp rx=0 tx=0).stop 2023-10-16T10:02:05.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.104+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5c0729b0 msgr2=0x7fdf5c0c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:05.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.104+0000 7fdf615ba700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5c0729b0 0x7fdf5c0c9a90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400ca70 tx=0x7fdf5400cd80 comp rx=0 tx=0).stop 2023-10-16T10:02:05.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.104+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 shutdown_connections 2023-10-16T10:02:05.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.104+0000 7fdf615ba700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdf4806b020 0x7fdf4806d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.104+0000 7fdf615ba700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5c0729b0 0x7fdf5c0c9a90 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.105+0000 7fdf615ba700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf5c06ab40 0x7fdf5c0c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.105+0000 7fdf615ba700 1 --2- 172.21.15.158:0/1330384128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf5c06e5d0 0x7fdf5c0c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.105+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 >> 172.21.15.158:0/1330384128 conn(0x7fdf5c0d02b0 msgr2=0x7fdf5c0af160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:05.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.105+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 shutdown_connections 2023-10-16T10:02:05.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.105+0000 7fdf615ba700 1 -- 172.21.15.158:0/1330384128 wait complete. 2023-10-16T10:02:05.109 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 53 2023-10-16T10:02:05.118 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T10:02:05.118 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T10:02:05.119 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T10:02:05.119 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:02:05.119 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T10:02:05.120 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T10:02:05.120 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T10:02:05.120 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:02:05.120 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T10:02:05.121 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T10:02:05.121 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T10:02:05.121 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:02:05.122 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T10:02:05.122 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T10:02:05.122 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T10:02:05.122 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:02:05.127 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T10:02:05.128 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T10:02:05.129 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T10:02:05.129 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T10:02:05.133 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T10:02:05.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T10:02:05.134 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T10:02:05.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T10:02:05.171 INFO:tasks.ceph.mds.a:Started 2023-10-16T10:02:05.171 DEBUG:teuthology.parallel:result is None 2023-10-16T10:02:05.172 INFO:tasks.ceph.mds.c:Started 2023-10-16T10:02:05.173 DEBUG:teuthology.parallel:result is None 2023-10-16T10:02:05.175 INFO:tasks.ceph.mds.b:Started 2023-10-16T10:02:05.176 DEBUG:teuthology.parallel:result is None 2023-10-16T10:02:05.177 INFO:tasks.ceph.mds.d:Started 2023-10-16T10:02:05.177 DEBUG:teuthology.parallel:result is None 2023-10-16T10:02:05.177 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T10:02:05.249 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T10:02:05.250 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T10:02:05.263 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T10:02:05.264 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T10:02:05.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.306+0000 7f48619f5700 1 Processor -- start 2023-10-16T10:02:05.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.306+0000 7f48619f5700 1 -- start start 2023-10-16T10:02:05.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.306+0000 7f48619f5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f485c15a500 0x7f485c0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:05.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.306+0000 7f48619f5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f485c0cfa70 0x7f485c0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:05.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.306+0000 7f48619f5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0b4240 con 0x7f485c0b3c70 2023-10-16T10:02:05.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.306+0000 7f48619f5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c0cd7a0 con 0x7f485c0cfa70 2023-10-16T10:02:05.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.306+0000 7f48619f5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c071b20 con 0x7f485c15a500 2023-10-16T10:02:05.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f485c0cfa70 0x7f485c0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:05.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f485c0cfa70 0x7f485c0b4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56448/0 (socket says 172.21.15.158:56448) 2023-10-16T10:02:05.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f485b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f485c0b3c70 0x7f485c15e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56626/0 (socket says 172.21.15.158:56626) 2023-10-16T10:02:05.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 -- 172.21.15.158:0/2830593170 learned_addr learned my addr 172.21.15.158:0/2830593170 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:05.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f485affd700 1 --2- 172.21.15.158:0/2830593170 >> v2:172.21.15.172:3300/0 conn(0x7f485c15a500 0x7f485c0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:05.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 -- 172.21.15.158:0/2830593170 >> v2:172.21.15.172:3300/0 conn(0x7f485c15a500 msgr2=0x7f485c0b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:05.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 --2- 172.21.15.158:0/2830593170 >> v2:172.21.15.172:3300/0 conn(0x7f485c15a500 0x7f485c0b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 -- 172.21.15.158:0/2830593170 >> v1:172.21.15.158:6789/0 conn(0x7f485c0b3c70 legacy=0x7f485c15e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:05.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 -- 172.21.15.158:0/2830593170 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c071dc0 con 0x7f485c0cfa70 2023-10-16T10:02:05.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f485a7fc700 1 -- 172.21.15.158:0/2830593170 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604002615 0 0) 0x7f485c0b4240 con 0x7f485c0b3c70 2023-10-16T10:02:05.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f48527fc700 1 --2- 172.21.15.158:0/2830593170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0cfa70 0x7f485c0b4f40 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f485c0719a0 tx=0x7f4844009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de057f5cfdc6dc1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:05.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f485a7fc700 1 -- 172.21.15.158:0/2830593170 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844011830 con 0x7f485c0cfa70 2023-10-16T10:02:05.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f485a7fc700 1 -- 172.21.15.158:0/2830593170 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4844011990 con 0x7f485c0cfa70 2023-10-16T10:02:05.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.307+0000 7f485a7fc700 1 -- 172.21.15.158:0/2830593170 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844011d20 con 0x7f485c0cfa70 2023-10-16T10:02:05.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 -- 172.21.15.158:0/2830593170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0cfa70 msgr2=0x7f485c0b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:05.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 --2- 172.21.15.158:0/2830593170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0cfa70 0x7f485c0b4f40 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f485c0719a0 tx=0x7f4844009bd0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 -- 172.21.15.158:0/2830593170 shutdown_connections 2023-10-16T10:02:05.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 --2- 172.21.15.158:0/2830593170 >> v2:172.21.15.172:3300/0 conn(0x7f485c15a500 0x7f485c0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 --2- 172.21.15.158:0/2830593170 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0cfa70 0x7f485c0b4f40 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 -- 172.21.15.158:0/2830593170 >> 172.21.15.158:0/2830593170 conn(0x7f485c06da00 msgr2=0x7f485c06dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:05.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 -- 172.21.15.158:0/2830593170 shutdown_connections 2023-10-16T10:02:05.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.308+0000 7f48619f5700 1 -- 172.21.15.158:0/2830593170 wait complete. 2023-10-16T10:02:05.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.309+0000 7f48619f5700 1 Processor -- start 2023-10-16T10:02:05.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.309+0000 7f48619f5700 1 -- start start 2023-10-16T10:02:05.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f48619f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0b3c70 0x7f485c148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:05.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f48619f5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cfa70 0x7f485c158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:05.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f48619f5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c15a500 0x7f485c1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:05.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f48619f5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c15cf50 con 0x7f485c0b3c70 2023-10-16T10:02:05.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f48619f5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c15d230 con 0x7f485c0cfa70 2023-10-16T10:02:05.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f48619f5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c15d510 con 0x7f485c15a500 2023-10-16T10:02:05.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0b3c70 0x7f485c148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:05.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0b3c70 0x7f485c148350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56452/0 (socket says 172.21.15.158:56452) 2023-10-16T10:02:05.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 -- 172.21.15.158:0/2000752987 learned_addr learned my addr 172.21.15.158:0/2000752987 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:05.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f48527fc700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cfa70 0x7f485c158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:05.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485b7fe700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c15a500 0x7f485c1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:05.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 -- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c15a500 msgr2=0x7f485c1594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:05.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c15a500 0x7f485c1594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 -- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cfa70 msgr2=0x7f485c158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:05.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cfa70 0x7f485c158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:05.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 -- 172.21.15.158:0/2000752987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4844009550 con 0x7f485c0b3c70 2023-10-16T10:02:05.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.310+0000 7f485affd700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0b3c70 0x7f485c148350 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f484c00e870 tx=0x7f484c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:05.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.311+0000 7f4858ff9700 1 -- 172.21.15.158:0/2000752987 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c009590 con 0x7f485c0b3c70 2023-10-16T10:02:05.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.311+0000 7f4858ff9700 1 -- 172.21.15.158:0/2000752987 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c013680 con 0x7f485c0b3c70 2023-10-16T10:02:05.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.311+0000 7f4858ff9700 1 -- 172.21.15.158:0/2000752987 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0139f0 con 0x7f485c0b3c70 2023-10-16T10:02:05.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.311+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c15d7f0 con 0x7f485c0b3c70 2023-10-16T10:02:05.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.311+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c14e380 con 0x7f485c0b3c70 2023-10-16T10:02:05.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.312+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c0b4870 con 0x7f485c0b3c70 2023-10-16T10:02:05.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.312+0000 7f4858ff9700 1 -- 172.21.15.158:0/2000752987 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f484c013b50 con 0x7f485c0b3c70 2023-10-16T10:02:05.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.313+0000 7f4858ff9700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f483c06b020 0x7f483c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:05.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.313+0000 7f48527fc700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f483c06b020 0x7f483c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:05.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.313+0000 7f4858ff9700 1 -- 172.21.15.158:0/2000752987 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f484c089810 con 0x7f485c0b3c70 2023-10-16T10:02:05.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.313+0000 7f48527fc700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f483c06b020 0x7f483c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f485c0719a0 tx=0x7f4844008030 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:05.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.318+0000 7f4858ff9700 1 -- 172.21.15.158:0/2000752987 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f484c054550 con 0x7f485c0b3c70 2023-10-16T10:02:05.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:05.526+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f485c15d090 con 0x7f485c0b3c70 2023-10-16T10:02:06.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.164+0000 7f4858ff9700 1 -- 172.21.15.158:0/2000752987 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v120) v1 ==== 97+0+0 (secure 0 0 0) 0x7f485c15d090 con 0x7f485c0b3c70 2023-10-16T10:02:06.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f483c06b020 msgr2=0x7f483c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f483c06b020 0x7f483c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f485c0719a0 tx=0x7f4844008030 comp rx=0 tx=0).stop 2023-10-16T10:02:06.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0b3c70 msgr2=0x7f485c148350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0b3c70 0x7f485c148350 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f484c00e870 tx=0x7f484c00eb80 comp rx=0 tx=0).stop 2023-10-16T10:02:06.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 shutdown_connections 2023-10-16T10:02:06.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c15a500 0x7f485c1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f483c06b020 0x7f483c06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f485c0b3c70 0x7f485c148350 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 --2- 172.21.15.158:0/2000752987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cfa70 0x7f485c158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 >> 172.21.15.158:0/2000752987 conn(0x7f485c06da00 msgr2=0x7f485c165a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:06.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 shutdown_connections 2023-10-16T10:02:06.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.168+0000 7f48619f5700 1 -- 172.21.15.158:0/2000752987 wait complete. 2023-10-16T10:02:06.172 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T10:02:06.185 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T10:02:06.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.311+0000 7f1f99411700 1 Processor -- start 2023-10-16T10:02:06.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.311+0000 7f1f99411700 1 -- start start 2023-10-16T10:02:06.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f99411700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f940cc0c0 0x7f1f940aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f99411700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f9406bfb0 0x7f1f940cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f99411700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940cf2f0 0x7f1f9415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f99411700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f940cf7a0 con 0x7f1f940cc0c0 2023-10-16T10:02:06.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f99411700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9415b810 con 0x7f1f9406bfb0 2023-10-16T10:02:06.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f99411700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9415a990 con 0x7f1f940cf2f0 2023-10-16T10:02:06.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f92ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f940cc0c0 0x7f1f940aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f92ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f940cc0c0 0x7f1f940aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56462/0 (socket says 172.21.15.158:56462) 2023-10-16T10:02:06.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.312+0000 7f1f92ffd700 1 -- 172.21.15.158:0/1857432164 learned_addr learned my addr 172.21.15.158:0/1857432164 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:06.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f92ffd700 1 -- 172.21.15.158:0/1857432164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940cf2f0 msgr2=0x7f1f9415dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:06.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f927fc700 1 --2- 172.21.15.158:0/1857432164 >> v2:172.21.15.172:3300/0 conn(0x7f1f9406bfb0 0x7f1f940cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f937fe700 1 --2- 172.21.15.158:0/1857432164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940cf2f0 0x7f1f9415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/1857432164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940cf2f0 0x7f1f9415dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f92ffd700 1 -- 172.21.15.158:0/1857432164 >> v2:172.21.15.172:3300/0 conn(0x7f1f9406bfb0 msgr2=0x7f1f940cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:06.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/1857432164 >> v2:172.21.15.172:3300/0 conn(0x7f1f9406bfb0 0x7f1f940cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f92ffd700 1 -- 172.21.15.158:0/1857432164 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9415ac00 con 0x7f1f940cc0c0 2023-10-16T10:02:06.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f927fc700 1 --2- 172.21.15.158:0/1857432164 >> v2:172.21.15.172:3300/0 conn(0x7f1f9406bfb0 0x7f1f940cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:06.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f937fe700 1 --2- 172.21.15.158:0/1857432164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940cf2f0 0x7f1f9415dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:06.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/1857432164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f940cc0c0 0x7f1f940aeda0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f1f940710a0 tx=0x7f1f84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32177eb380a13111 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:06.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f91ffb700 1 -- 172.21.15.158:0/1857432164 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400b870 con 0x7f1f940cc0c0 2023-10-16T10:02:06.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f91ffb700 1 -- 172.21.15.158:0/1857432164 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8400b9d0 con 0x7f1f940cc0c0 2023-10-16T10:02:06.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.313+0000 7f1f91ffb700 1 -- 172.21.15.158:0/1857432164 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400bd60 con 0x7f1f940cc0c0 2023-10-16T10:02:06.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 -- 172.21.15.158:0/1857432164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f940cc0c0 msgr2=0x7f1f940aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 --2- 172.21.15.158:0/1857432164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f940cc0c0 0x7f1f940aeda0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f1f940710a0 tx=0x7f1f84009bd0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 -- 172.21.15.158:0/1857432164 shutdown_connections 2023-10-16T10:02:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 --2- 172.21.15.158:0/1857432164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f940cf2f0 0x7f1f9415dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 --2- 172.21.15.158:0/1857432164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f940cc0c0 0x7f1f940aeda0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 --2- 172.21.15.158:0/1857432164 >> v2:172.21.15.172:3300/0 conn(0x7f1f9406bfb0 0x7f1f940cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 -- 172.21.15.158:0/1857432164 >> 172.21.15.158:0/1857432164 conn(0x7f1f940cc4b0 msgr2=0x7f1f940cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 -- 172.21.15.158:0/1857432164 shutdown_connections 2023-10-16T10:02:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.314+0000 7f1f99411700 1 -- 172.21.15.158:0/1857432164 wait complete. 2023-10-16T10:02:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.315+0000 7f1f99411700 1 Processor -- start 2023-10-16T10:02:06.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.315+0000 7f1f99411700 1 -- start start 2023-10-16T10:02:06.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f99411700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9406bfb0 0x7f1f94140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f99411700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940cf2f0 0x7f1f9414e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f99411700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9414e880 0x7f1f94141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f99411700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f94140d50 con 0x7f1f9406bfb0 2023-10-16T10:02:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f99411700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9414f8f0 con 0x7f1f940cf2f0 2023-10-16T10:02:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f99411700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9414ed70 con 0x7f1f9414e880 2023-10-16T10:02:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9406bfb0 0x7f1f94140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9406bfb0 0x7f1f94140810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56468/0 (socket says 172.21.15.158:56468) 2023-10-16T10:02:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 -- 172.21.15.158:0/3657763107 learned_addr learned my addr 172.21.15.158:0/3657763107 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f927fc700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940cf2f0 0x7f1f9414e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f937fe700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9414e880 0x7f1f94141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 -- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9414e880 msgr2=0x7f1f94141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9414e880 0x7f1f94141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 -- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940cf2f0 msgr2=0x7f1f9414e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940cf2f0 0x7f1f9414e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 -- 172.21.15.158:0/3657763107 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84009550 con 0x7f1f9406bfb0 2023-10-16T10:02:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f937fe700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9414e880 0x7f1f94141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f927fc700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940cf2f0 0x7f1f9414e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f92ffd700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9406bfb0 0x7f1f94140810 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f1f940cd160 tx=0x7f1f84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f7b7fe700 1 -- 172.21.15.158:0/3657763107 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84004310 con 0x7f1f9406bfb0 2023-10-16T10:02:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f7b7fe700 1 -- 172.21.15.158:0/3657763107 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8401f070 con 0x7f1f9406bfb0 2023-10-16T10:02:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9414f050 con 0x7f1f9406bfb0 2023-10-16T10:02:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.316+0000 7f1f7b7fe700 1 -- 172.21.15.158:0/3657763107 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f840036a0 con 0x7f1f9406bfb0 2023-10-16T10:02:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.317+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9414f610 con 0x7f1f9406bfb0 2023-10-16T10:02:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.317+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f94141290 con 0x7f1f9406bfb0 2023-10-16T10:02:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.318+0000 7f1f7b7fe700 1 -- 172.21.15.158:0/3657763107 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f84021030 con 0x7f1f9406bfb0 2023-10-16T10:02:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.318+0000 7f1f7b7fe700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.318+0000 7f1f7b7fe700 1 -- 172.21.15.158:0/3657763107 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f1f8408ab70 con 0x7f1f9406bfb0 2023-10-16T10:02:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.318+0000 7f1f927fc700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.319+0000 7f1f927fc700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1f8006b020 0x7f1f8006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f9414c2c0 tx=0x7f1f88008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.323+0000 7f1f7b7fe700 1 -- 172.21.15.158:0/3657763107 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f94141290 con 0x7f1f9406bfb0 2023-10-16T10:02:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.595+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1f94141290 con 0x7f1f9406bfb0 2023-10-16T10:02:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.596+0000 7f1f7b7fe700 1 -- 172.21.15.158:0/3657763107 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1f94141290 con 0x7f1f9406bfb0 2023-10-16T10:02:06.597 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:06.598 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:02:06.598 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T10:02:06.598 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.598 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T10:02:06.599 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T10:02:06.599 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.599 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.599 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:02:06.599 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:02:06.600 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.600 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.600 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.600 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.600 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T10:02:06.601 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T10:02:06.601 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.601 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.602 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:02:06.602 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:02:06.602 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.602 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.602 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.603 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.603 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T10:02:06.603 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T10:02:06.603 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.603 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.604 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:02:06.604 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:02:06.604 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.604 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.604 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.605 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.605 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T10:02:06.605 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T10:02:06.605 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.605 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.605 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:02:06.606 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:02:06.606 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.606 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.606 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.606 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.607 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T10:02:06.607 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T10:02:06.607 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.607 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.607 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.608 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.608 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.608 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.608 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T10:02:06.609 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T10:02:06.609 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.609 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.609 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.609 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.610 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.610 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.610 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.610 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T10:02:06.610 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T10:02:06.611 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.611 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.611 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.612 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.612 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.612 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.612 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.612 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T10:02:06.613 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T10:02:06.613 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.613 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.613 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.613 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.614 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.614 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.614 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.614 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T10:02:06.614 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T10:02:06.615 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.615 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.615 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.615 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.615 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.616 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.616 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.616 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T10:02:06.616 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T10:02:06.616 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.617 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.617 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.617 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.617 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.618 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.618 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T10:02:06.618 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T10:02:06.618 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.618 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.619 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.619 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.619 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.619 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.619 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.620 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T10:02:06.620 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T10:02:06.620 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.620 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:02:06.620 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:02:06.621 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.621 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.621 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.621 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.621 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T10:02:06.622 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T10:02:06.622 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.622 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.622 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T10:02:06.622 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T10:02:06.623 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.623 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.623 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.623 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T10:02:06.623 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T10:02:06.624 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.624 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.624 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:02:06.624 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:02:06.624 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:02:06.625 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.625 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.625 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.625 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T10:02:06.625 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T10:02:06.626 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.626 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.626 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:02:06.626 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:02:06.626 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:02:06.627 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.627 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.627 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.627 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T10:02:06.627 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T10:02:06.628 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.628 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:02:06.628 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:02:06.628 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:02:06.628 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:02:06.628 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.629 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.629 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.629 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T10:02:06.629 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T10:02:06.629 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.630 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:02:06.630 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:02:06.630 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:02:06.630 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.630 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.631 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.631 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.631 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T10:02:06.631 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T10:02:06.631 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.632 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:02:06.632 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:02:06.632 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.632 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.632 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:02:06.633 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:06.633 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T10:02:06.633 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T10:02:06.633 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:02:06.633 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:02:06.634 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:02:06.634 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:02:06.634 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.634 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:06.634 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T10:02:06.635 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:02:06.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1f8006b020 msgr2=0x7f1f8006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1f8006b020 0x7f1f8006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f9414c2c0 tx=0x7f1f88008040 comp rx=0 tx=0).stop 2023-10-16T10:02:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9406bfb0 msgr2=0x7f1f94140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9406bfb0 0x7f1f94140810 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f1f940cd160 tx=0x7f1f84004750 comp rx=0 tx=0).stop 2023-10-16T10:02:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 shutdown_connections 2023-10-16T10:02:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9414e880 0x7f1f94141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1f8006b020 0x7f1f8006d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9406bfb0 0x7f1f94140810 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 --2- 172.21.15.158:0/3657763107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f940cf2f0 0x7f1f9414e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 >> 172.21.15.158:0/3657763107 conn(0x7f1f940cc4b0 msgr2=0x7f1f9415bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 shutdown_connections 2023-10-16T10:02:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.598+0000 7f1f99411700 1 -- 172.21.15.158:0/3657763107 wait complete. 2023-10-16T10:02:06.639 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:02:06.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.759+0000 7f71fb199700 1 Processor -- start 2023-10-16T10:02:06.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.759+0000 7f71fb199700 1 -- start start 2023-10-16T10:02:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.759+0000 7f71fb199700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f415dfb0 0x7f71f406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.759+0000 7f71fb199700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71f406f340 0x7f71f406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71fb199700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f406e770 con 0x7f71f40cc5d0 2023-10-16T10:02:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71fb199700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f406fd30 con 0x7f71f406f340 2023-10-16T10:02:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71fb199700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40cf270 con 0x7f71f415dfb0 2023-10-16T10:02:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71f406f340 0x7f71f406f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71f406f340 0x7f71f406f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56478/0 (socket says 172.21.15.158:56478) 2023-10-16T10:02:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 -- 172.21.15.158:0/1201286331 learned_addr learned my addr 172.21.15.158:0/1201286331 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f3fff700 1 --2- 172.21.15.158:0/1201286331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f415dfb0 0x7f71f406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 -- 172.21.15.158:0/1201286331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f415dfb0 msgr2=0x7f71f406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 --2- 172.21.15.158:0/1201286331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f415dfb0 0x7f71f406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 -- 172.21.15.158:0/1201286331 >> v1:172.21.15.172:6789/0 conn(0x7f71f40cc5d0 legacy=0x7f71f40cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 -- 172.21.15.158:0/1201286331 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f40cbe00 con 0x7f71f406f340 2023-10-16T10:02:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f9736700 1 --2- 172.21.15.158:0/1201286331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f406f340 0x7f71f406f7f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00d150 tx=0x7f71ec00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae168443d839ef0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f37fe700 1 -- 172.21.15.158:0/1201286331 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec010070 con 0x7f71f406f340 2023-10-16T10:02:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f37fe700 1 -- 172.21.15.158:0/1201286331 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec00ddc0 con 0x7f71f406f340 2023-10-16T10:02:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.760+0000 7f71f37fe700 1 -- 172.21.15.158:0/1201286331 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec00b310 con 0x7f71f406f340 2023-10-16T10:02:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 -- 172.21.15.158:0/1201286331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f406f340 msgr2=0x7f71f406f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 --2- 172.21.15.158:0/1201286331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f406f340 0x7f71f406f7f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00d150 tx=0x7f71ec00d460 comp rx=0 tx=0).stop 2023-10-16T10:02:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 -- 172.21.15.158:0/1201286331 shutdown_connections 2023-10-16T10:02:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 --2- 172.21.15.158:0/1201286331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f406f340 0x7f71f406f7f0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 --2- 172.21.15.158:0/1201286331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f415dfb0 0x7f71f406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 -- 172.21.15.158:0/1201286331 >> 172.21.15.158:0/1201286331 conn(0x7f71f40d02b0 msgr2=0x7f71f40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 -- 172.21.15.158:0/1201286331 shutdown_connections 2023-10-16T10:02:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.761+0000 7f71fb199700 1 -- 172.21.15.158:0/1201286331 wait complete. 2023-10-16T10:02:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.762+0000 7f71fb199700 1 Processor -- start 2023-10-16T10:02:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.762+0000 7f71fb199700 1 -- start start 2023-10-16T10:02:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71fb199700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f406f340 0x7f71f41f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71fb199700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f40cc5d0 0x7f71f41f1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71fb199700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f415dfb0 0x7f71f415cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71fb199700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f41f2d10 con 0x7f71f415dfb0 2023-10-16T10:02:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71fb199700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40ca0c0 con 0x7f71f406f340 2023-10-16T10:02:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71fb199700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40c7560 con 0x7f71f40cc5d0 2023-10-16T10:02:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f415dfb0 0x7f71f415cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f415dfb0 0x7f71f415cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56486/0 (socket says 172.21.15.158:56486) 2023-10-16T10:02:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 -- 172.21.15.158:0/3495440049 learned_addr learned my addr 172.21.15.158:0/3495440049 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f8f35700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f406f340 0x7f71f41f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f3fff700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f40cc5d0 0x7f71f41f1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 -- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f40cc5d0 msgr2=0x7f71f41f1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f40cc5d0 0x7f71f41f1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 -- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f406f340 msgr2=0x7f71f41f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f406f340 0x7f71f41f27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.763+0000 7f71f9736700 1 -- 172.21.15.158:0/3495440049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec00bd40 con 0x7f71f415dfb0 2023-10-16T10:02:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.764+0000 7f71f9736700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f415dfb0 0x7f71f415cc10 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f71f4074ec0 tx=0x7f71ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.764+0000 7f71f17fa700 1 -- 172.21.15.158:0/3495440049 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec00bbc0 con 0x7f71f415dfb0 2023-10-16T10:02:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.764+0000 7f71f17fa700 1 -- 172.21.15.158:0/3495440049 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec00b4c0 con 0x7f71f415dfb0 2023-10-16T10:02:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.764+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f40c7840 con 0x7f71f415dfb0 2023-10-16T10:02:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.764+0000 7f71f17fa700 1 -- 172.21.15.158:0/3495440049 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec004040 con 0x7f71f415dfb0 2023-10-16T10:02:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.764+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f41f1aa0 con 0x7f71f415dfb0 2023-10-16T10:02:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.765+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f40c73c0 con 0x7f71f415dfb0 2023-10-16T10:02:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.765+0000 7f71f17fa700 1 -- 172.21.15.158:0/3495440049 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f71ec0041a0 con 0x7f71f415dfb0 2023-10-16T10:02:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.765+0000 7f71f17fa700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f71d806b020 0x7f71d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.766+0000 7f71f17fa700 1 -- 172.21.15.158:0/3495440049 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f71ec08cb40 con 0x7f71f415dfb0 2023-10-16T10:02:06.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.766+0000 7f71f8f35700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f71d806b020 0x7f71d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:06.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.766+0000 7f71f8f35700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f71d806b020 0x7f71d806d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f71e8001800 tx=0x7f71e8009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:06.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.770+0000 7f71f17fa700 1 -- 172.21.15.158:0/3495440049 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f71f40c73c0 con 0x7f71f415dfb0 2023-10-16T10:02:06.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.963+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71f40c73c0 con 0x7f71f415dfb0 2023-10-16T10:02:06.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.963+0000 7f71f17fa700 1 -- 172.21.15.158:0/3495440049 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+13729 (secure 0 0 0) 0x7f71ec084e60 con 0x7f71f415dfb0 2023-10-16T10:02:06.965 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:06.966 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":122,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:06.173046+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:02:06.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f71d806b020 msgr2=0x7f71d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f71d806b020 0x7f71d806d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f71e8001800 tx=0x7f71e8009040 comp rx=0 tx=0).stop 2023-10-16T10:02:06.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f415dfb0 msgr2=0x7f71f415cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:06.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f415dfb0 0x7f71f415cc10 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f71f4074ec0 tx=0x7f71ec004ad0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 shutdown_connections 2023-10-16T10:02:06.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f71d806b020 0x7f71d806d4e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f415dfb0 0x7f71f415cc10 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f406f340 0x7f71f41f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 --2- 172.21.15.158:0/3495440049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71f40cc5d0 0x7f71f41f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:06.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 >> 172.21.15.158:0/3495440049 conn(0x7f71f40d02b0 msgr2=0x7f71f40afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:06.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 shutdown_connections 2023-10-16T10:02:06.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:06.966+0000 7f71fb199700 1 -- 172.21.15.158:0/3495440049 wait complete. 2023-10-16T10:02:06.979 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:02:06.980 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:02:07.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.102+0000 7fcc125c7700 1 Processor -- start 2023-10-16T10:02:07.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.102+0000 7fcc125c7700 1 -- start start 2023-10-16T10:02:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.102+0000 7fcc125c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c0cc590 0x7fcc0c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.102+0000 7fcc125c7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c0cbd30 con 0x7fcc0c06afe0 2023-10-16T10:02:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.102+0000 7fcc125c7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c0cc1c0 con 0x7fcc0c15da40 2023-10-16T10:02:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.102+0000 7fcc125c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c15de20 con 0x7fcc0c0cc590 2023-10-16T10:02:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.102+0000 7fcc0b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcc0c06afe0 0x7fcc0c0afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56666/0 (socket says 172.21.15.158:56666) 2023-10-16T10:02:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0b7fe700 1 -- 172.21.15.158:0/4164914206 learned_addr learned my addr 172.21.15.158:0/4164914206 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc10b64700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c0cc590 0x7fcc0c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123493359 0 0) 0x7fcc0c0cbd30 con 0x7fcc0c06afe0 2023-10-16T10:02:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf40035b0 con 0x7fcc0c06afe0 2023-10-16T10:02:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580791605 0 0) 0x7fcc0c0cc1c0 con 0x7fcc0c15da40 2023-10-16T10:02:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c0cbd30 con 0x7fcc0c15da40 2023-10-16T10:02:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2665275619 0 0) 0x7fcbf40035b0 con 0x7fcc0c06afe0 2023-10-16T10:02:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c0cc590 msgr2=0x7fcc0c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c0cc590 0x7fcc0c0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 >> v1:172.21.15.172:6789/0 conn(0x7fcc0c15da40 legacy=0x7fcc0c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c0ce260 con 0x7fcc0c06afe0 2023-10-16T10:02:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcc00001670 con 0x7fcc0c06afe0 2023-10-16T10:02:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 >> v1:172.21.15.158:6789/0 conn(0x7fcc0c06afe0 legacy=0x7fcc0c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbf4003bc0 0x7fcbf4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf4004850 0x7fcbf4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbf4005210 0x7fcbf400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c15de20 con 0x7fcbf4005210 2023-10-16T10:02:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf400c030 con 0x7fcbf4004850 2023-10-16T10:02:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf400c1f0 con 0x7fcbf4003bc0 2023-10-16T10:02:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc10b64700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c0cc590 0x7fcc0c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:02:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0bfff700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbf4003bc0 0x7fcbf4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc0b7fe700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf4004850 0x7fcbf4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.103+0000 7fcc10b64700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbf4005210 0x7fcbf400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc10b64700 1 -- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbf4003bc0 msgr2=0x7fcbf4003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc10b64700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbf4003bc0 0x7fcbf4003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc10b64700 1 -- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf4004850 msgr2=0x7fcbf4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc10b64700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf4004850 0x7fcbf4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc10b64700 1 -- 172.21.15.158:0/4164914206 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40035b0 con 0x7fcbf4005210 2023-10-16T10:02:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc10b64700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbf4005210 0x7fcbf400ba10 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0cc970 tx=0x7fcbf4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1a23f04bf385fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04003c20 con 0x7fcbf4005210 2023-10-16T10:02:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0400f040 con 0x7fcbf4005210 2023-10-16T10:02:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc0affd700 1 -- 172.21.15.158:0/4164914206 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400ea60 con 0x7fcbf4005210 2023-10-16T10:02:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc125c7700 1 -- 172.21.15.158:0/4164914206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbf4005210 msgr2=0x7fcbf400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc125c7700 1 --2- 172.21.15.158:0/4164914206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcbf4005210 0x7fcbf400ba10 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0cc970 tx=0x7fcbf4005880 comp rx=0 tx=0).stop 2023-10-16T10:02:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc10b64700 1 -- 172.21.15.158:0/4164914206 reap_dead start 2023-10-16T10:02:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc125c7700 1 -- 172.21.15.158:0/4164914206 shutdown_connections 2023-10-16T10:02:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc125c7700 1 -- 172.21.15.158:0/4164914206 >> 172.21.15.158:0/4164914206 conn(0x7fcc0c099a10 msgr2=0x7fcc0c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc125c7700 1 -- 172.21.15.158:0/4164914206 shutdown_connections 2023-10-16T10:02:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.104+0000 7fcc125c7700 1 -- 172.21.15.158:0/4164914206 wait complete. 2023-10-16T10:02:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.106+0000 7fcc125c7700 1 Processor -- start 2023-10-16T10:02:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.106+0000 7fcc125c7700 1 -- start start 2023-10-16T10:02:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc125c7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc0c0ca860 0x7fcc0c0b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc125c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c150ba0 0x7fcc0c06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc125c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc0c06f000 0x7fcc0c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc125c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c0b5a90 con 0x7fcc0c06f000 2023-10-16T10:02:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc125c7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c0b21e0 con 0x7fcc0c0ca860 2023-10-16T10:02:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc125c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c06f480 con 0x7fcc0c150ba0 2023-10-16T10:02:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc10b64700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc0c06f000 0x7fcc0c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc10b64700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc0c06f000 0x7fcc0c0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56506/0 (socket says 172.21.15.158:56506) 2023-10-16T10:02:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc10b64700 1 -- 172.21.15.158:0/2636342080 learned_addr learned my addr 172.21.15.158:0/2636342080 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc0b7fe700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c150ba0 0x7fcc0c06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.107+0000 7fcc0bfff700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc0c0ca860 0x7fcc0c0b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc10b64700 1 -- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c150ba0 msgr2=0x7fcc0c06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc10b64700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c150ba0 0x7fcc0c06eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc10b64700 1 -- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc0c0ca860 msgr2=0x7fcc0c0b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc10b64700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc0c0ca860 0x7fcc0c0b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc10b64700 1 -- 172.21.15.158:0/2636342080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c0b2d80 con 0x7fcc0c06f000 2023-10-16T10:02:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc0b7fe700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c150ba0 0x7fcc0c06eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc0bfff700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc0c0ca860 0x7fcc0c0b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc10b64700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc0c06f000 0x7fcc0c0c74a0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fcc04015000 tx=0x7fcc04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc08ff9700 1 -- 172.21.15.158:0/2636342080 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400ee50 con 0x7fcc0c06f000 2023-10-16T10:02:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc08ff9700 1 -- 172.21.15.158:0/2636342080 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc04003a20 con 0x7fcc0c06f000 2023-10-16T10:02:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc08ff9700 1 -- 172.21.15.158:0/2636342080 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04003dd0 con 0x7fcc0c06f000 2023-10-16T10:02:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c0b2fb0 con 0x7fcc0c06f000 2023-10-16T10:02:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.108+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c0b53d0 con 0x7fcc0c06f000 2023-10-16T10:02:07.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.109+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c0b5bd0 con 0x7fcc0c06f000 2023-10-16T10:02:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.109+0000 7fcc08ff9700 1 -- 172.21.15.158:0/2636342080 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc04024070 con 0x7fcc0c06f000 2023-10-16T10:02:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.110+0000 7fcc08ff9700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fcbf806b020 0x7fcbf806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.110+0000 7fcc08ff9700 1 -- 172.21.15.158:0/2636342080 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5601+0+0 (secure 0 0 0) 0x7fcc0408aa50 con 0x7fcc0c06f000 2023-10-16T10:02:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.110+0000 7fcc0bfff700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fcbf806b020 0x7fcbf806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.110+0000 7fcc0bfff700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fcbf806b020 0x7fcbf806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc007fd0 tx=0x7fcbfc009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.114+0000 7fcc08ff9700 1 -- 172.21.15.158:0/2636342080 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc0c0b5bd0 con 0x7fcc0c06f000 2023-10-16T10:02:07.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.314+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc0c0681b0 con 0x7fcc0c06f000 2023-10-16T10:02:07.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.314+0000 7fcc08ff9700 1 -- 172.21.15.158:0/2636342080 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc04055ae0 con 0x7fcc0c06f000 2023-10-16T10:02:07.316 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:02:07.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fcbf806b020 msgr2=0x7fcbf806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fcbf806b020 0x7fcbf806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc007fd0 tx=0x7fcbfc009040 comp rx=0 tx=0).stop 2023-10-16T10:02:07.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc0c06f000 msgr2=0x7fcc0c0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc0c06f000 0x7fcc0c0c74a0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fcc04015000 tx=0x7fcc04009750 comp rx=0 tx=0).stop 2023-10-16T10:02:07.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 shutdown_connections 2023-10-16T10:02:07.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fcbf806b020 0x7fcbf806d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc0c06f000 0x7fcc0c0c74a0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc0c0ca860 0x7fcc0c0b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 --2- 172.21.15.158:0/2636342080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc0c150ba0 0x7fcc0c06eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 >> 172.21.15.158:0/2636342080 conn(0x7fcc0c099a10 msgr2=0x7fcc0c1e7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:07.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 shutdown_connections 2023-10-16T10:02:07.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.316+0000 7fcc125c7700 1 -- 172.21.15.158:0/2636342080 wait complete. 2023-10-16T10:02:07.331 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T10:02:07.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.450+0000 7f1d12e0a700 1 Processor -- start 2023-10-16T10:02:07.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.450+0000 7f1d12e0a700 1 -- start start 2023-10-16T10:02:07.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d12e0a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 0x7f1d0c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d12e0a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c15c5d0 con 0x7f1d0c0ce2c0 2023-10-16T10:02:07.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d12e0a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c15c8b0 con 0x7f1d0c0cdee0 2023-10-16T10:02:07.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d12e0a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c15d2c0 con 0x7f1d0c0b6e10 2023-10-16T10:02:07.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d113a7700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1d0c0ce2c0 0x7f1d0c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56692/0 (socket says 172.21.15.158:56692) 2023-10-16T10:02:07.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d113a7700 1 -- 172.21.15.158:0/3559418935 learned_addr learned my addr 172.21.15.158:0/3559418935 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:07.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865806468 0 0) 0x7f1d0c15c5d0 con 0x7f1d0c0ce2c0 2023-10-16T10:02:07.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cfc0035b0 con 0x7f1d0c0ce2c0 2023-10-16T10:02:07.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0bfff700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 0x7f1d0c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2320168486 0 0) 0x7f1cfc0035b0 con 0x7f1d0c0ce2c0 2023-10-16T10:02:07.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 msgr2=0x7f1d0c1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 0x7f1d0c1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 >> v1:172.21.15.172:6789/0 conn(0x7f1d0c0cdee0 legacy=0x7f1d0c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c15cb90 con 0x7f1d0c0ce2c0 2023-10-16T10:02:07.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1d040019f0 con 0x7f1d0c0ce2c0 2023-10-16T10:02:07.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 >> v1:172.21.15.158:6789/0 conn(0x7f1d0c0ce2c0 legacy=0x7f1d0c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.451+0000 7f1d0b7fe700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 0x7f1cfc003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cfc004060 0x7f1cfc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c15d2c0 con 0x7f1cfc004d90 2023-10-16T10:02:07.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc00bc50 con 0x7f1cfc004060 2023-10-16T10:02:07.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc00be10 con 0x7f1d0c0b6e10 2023-10-16T10:02:07.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d113a7700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0bfff700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cfc004060 0x7f1cfc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d10ba6700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 0x7f1cfc003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d113a7700 1 -- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 msgr2=0x7f1cfc003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d113a7700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0b6e10 0x7f1cfc003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d113a7700 1 -- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cfc004060 msgr2=0x7f1cfc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d113a7700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cfc004060 0x7f1cfc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d113a7700 1 -- 172.21.15.158:0/3559418935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc0035b0 con 0x7f1cfc004d90 2023-10-16T10:02:07.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d113a7700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400a830 tx=0x7f1cfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764bdfc3057138f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400a2f0 con 0x7f1cfc004d90 2023-10-16T10:02:07.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d04012620 con 0x7f1cfc004d90 2023-10-16T10:02:07.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.452+0000 7f1d0b7fe700 1 -- 172.21.15.158:0/3559418935 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d040129d0 con 0x7f1cfc004d90 2023-10-16T10:02:07.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.453+0000 7f1d12e0a700 1 -- 172.21.15.158:0/3559418935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 msgr2=0x7f1cfc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.453+0000 7f1d12e0a700 1 --2- 172.21.15.158:0/3559418935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400a830 tx=0x7f1cfc005880 comp rx=0 tx=0).stop 2023-10-16T10:02:07.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.453+0000 7f1d113a7700 1 -- 172.21.15.158:0/3559418935 reap_dead start 2023-10-16T10:02:07.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.453+0000 7f1d12e0a700 1 -- 172.21.15.158:0/3559418935 shutdown_connections 2023-10-16T10:02:07.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.453+0000 7f1d12e0a700 1 -- 172.21.15.158:0/3559418935 >> 172.21.15.158:0/3559418935 conn(0x7f1d0c06a8d0 msgr2=0x7f1d0c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:07.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.453+0000 7f1d12e0a700 1 -- 172.21.15.158:0/3559418935 shutdown_connections 2023-10-16T10:02:07.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.453+0000 7f1d12e0a700 1 -- 172.21.15.158:0/3559418935 wait complete. 2023-10-16T10:02:07.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.454+0000 7f1d12e0a700 1 Processor -- start 2023-10-16T10:02:07.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.454+0000 7f1d12e0a700 1 -- start start 2023-10-16T10:02:07.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d12e0a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1d0c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d12e0a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0d1bd0 0x7f1d0c1f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d12e0a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d0c1f01d0 0x7f1d0c1f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d12e0a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0d0e10 con 0x7f1cfc004d90 2023-10-16T10:02:07.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d12e0a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0d10f0 con 0x7f1d0c1f01d0 2023-10-16T10:02:07.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d12e0a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0d13d0 con 0x7f1d0c0d1bd0 2023-10-16T10:02:07.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1d0c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1d0c0ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56524/0 (socket says 172.21.15.158:56524) 2023-10-16T10:02:07.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 -- 172.21.15.158:0/398048436 learned_addr learned my addr 172.21.15.158:0/398048436 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:07.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 -- 172.21.15.158:0/398048436 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0d1bd0 msgr2=0x7f1d0c1f6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0d1bd0 0x7f1d0c1f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 -- 172.21.15.158:0/398048436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d0c1f01d0 msgr2=0x7f1d0c1f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d0c1f01d0 0x7f1d0c1f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.455+0000 7f1d10ba6700 1 -- 172.21.15.158:0/398048436 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c0d16b0 con 0x7f1cfc004d90 2023-10-16T10:02:07.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.456+0000 7f1d10ba6700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1d0c0ceee0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000f7f0 tx=0x7f1d0000fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.456+0000 7f1d097fa700 1 -- 172.21.15.158:0/398048436 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d000052e0 con 0x7f1cfc004d90 2023-10-16T10:02:07.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.456+0000 7f1d097fa700 1 -- 172.21.15.158:0/398048436 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d00005440 con 0x7f1cfc004d90 2023-10-16T10:02:07.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.456+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c0cf420 con 0x7f1cfc004d90 2023-10-16T10:02:07.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.456+0000 7f1d097fa700 1 -- 172.21.15.158:0/398048436 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d000038c0 con 0x7f1cfc004d90 2023-10-16T10:02:07.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.456+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c1f7220 con 0x7f1cfc004d90 2023-10-16T10:02:07.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.457+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0c0d1a20 con 0x7f1cfc004d90 2023-10-16T10:02:07.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.457+0000 7f1d097fa700 1 -- 172.21.15.158:0/398048436 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1d00003a20 con 0x7f1cfc004d90 2023-10-16T10:02:07.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.457+0000 7f1d097fa700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1cf006b020 0x7f1cf006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.458+0000 7f1d097fa700 1 -- 172.21.15.158:0/398048436 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f1d00089450 con 0x7f1cfc004d90 2023-10-16T10:02:07.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.458+0000 7f1d0bfff700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1cf006b020 0x7f1cf006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.458+0000 7f1d0bfff700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1cf006b020 0x7f1cf006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006b20 tx=0x7f1cf8006ab0 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.462+0000 7f1d097fa700 1 -- 172.21.15.158:0/398048436 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1d000544e0 con 0x7f1cfc004d90 2023-10-16T10:02:07.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.654+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1d0c0d0f50 con 0x7f1cfc004d90 2023-10-16T10:02:07.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.655+0000 7f1d097fa700 1 -- 172.21.15.158:0/398048436 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v122) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1d0c0d0f50 con 0x7f1cfc004d90 2023-10-16T10:02:07.656 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:07.656 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:02:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T10:02:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T10:02:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:07.657 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:02:07.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1cf006b020 msgr2=0x7f1cf006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1cf006b020 0x7f1cf006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006b20 tx=0x7f1cf8006ab0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 msgr2=0x7f1d0c0ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1d0c0ceee0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000f7f0 tx=0x7f1d0000fbb0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 shutdown_connections 2023-10-16T10:02:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d0c1f01d0 0x7f1d0c1f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f1cf006b020 0x7f1cf006d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cfc004d90 0x7f1d0c0ceee0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 --2- 172.21.15.158:0/398048436 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d0c0d1bd0 0x7f1d0c1f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 >> 172.21.15.158:0/398048436 conn(0x7f1d0c06a8d0 msgr2=0x7f1d0c0b5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:07.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 shutdown_connections 2023-10-16T10:02:07.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.657+0000 7f1d12e0a700 1 -- 172.21.15.158:0/398048436 wait complete. 2023-10-16T10:02:07.671 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T10:02:07.671 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T10:02:07.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.796+0000 7f328e05f700 1 Processor -- start 2023-10-16T10:02:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.796+0000 7f328e05f700 1 -- start start 2023-10-16T10:02:07.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f328e05f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f328815d6c0 0x7f32880cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f328e05f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 0x7f32880d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f328e05f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32880d1ee0 con 0x7f32880d2370 2023-10-16T10:02:07.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f328e05f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32880d40a0 con 0x7f328815d6c0 2023-10-16T10:02:07.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f328e05f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32880d12d0 con 0x7f328815daa0 2023-10-16T10:02:07.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f3287fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f32880d2370 0x7f32880cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56712/0 (socket says 172.21.15.158:56712) 2023-10-16T10:02:07.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f3287fff700 1 -- 172.21.15.158:0/3578459269 learned_addr learned my addr 172.21.15.158:0/3578459269 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32877fe700 1 --2- 172.21.15.158:0/3578459269 >> v2:172.21.15.172:3300/0 conn(0x7f328815d6c0 0x7f32880cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f3286ffd700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 0x7f32880d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085152620 0 0) 0x7f32880d1ee0 con 0x7f32880d2370 2023-10-16T10:02:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32740035b0 con 0x7f32880d2370 2023-10-16T10:02:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 496314515 0 0) 0x7f32740035b0 con 0x7f32880d2370 2023-10-16T10:02:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 msgr2=0x7f32880d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 0x7f32880d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 >> v2:172.21.15.172:3300/0 conn(0x7f328815d6c0 msgr2=0x7f32880cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 --2- 172.21.15.158:0/3578459269 >> v2:172.21.15.172:3300/0 conn(0x7f328815d6c0 0x7f32880cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880d15b0 con 0x7f32880d2370 2023-10-16T10:02:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f32800019f0 con 0x7f32880d2370 2023-10-16T10:02:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 >> v1:172.21.15.158:6789/0 conn(0x7f32880d2370 legacy=0x7f32880cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3274003a00 0x7f3274003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 0x7f3274004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.797+0000 7f32867fc700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274004d90 0x7f3274005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32880d12d0 con 0x7f3274004d90 2023-10-16T10:02:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327400ba10 con 0x7f3274003a00 2023-10-16T10:02:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327400bb50 con 0x7f328815daa0 2023-10-16T10:02:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3287fff700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274004d90 0x7f3274005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3286ffd700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 0x7f3274004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f32877fe700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3274003a00 0x7f3274003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3287fff700 1 -- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 msgr2=0x7f3274004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3287fff700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f328815daa0 0x7f3274004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3287fff700 1 -- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3274003a00 msgr2=0x7f3274003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3287fff700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3274003a00 0x7f3274003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3287fff700 1 -- 172.21.15.158:0/3578459269 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32740035b0 con 0x7f3274004d90 2023-10-16T10:02:07.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f3287fff700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274004d90 0x7f3274005210 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f328000a8d0 tx=0x7f3274005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be2670d6a2e10a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328000a340 con 0x7f3274004d90 2023-10-16T10:02:07.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3280012600 con 0x7f3274004d90 2023-10-16T10:02:07.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.798+0000 7f32867fc700 1 -- 172.21.15.158:0/3578459269 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32800129b0 con 0x7f3274004d90 2023-10-16T10:02:07.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.799+0000 7f328e05f700 1 -- 172.21.15.158:0/3578459269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274004d90 msgr2=0x7f3274005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.799+0000 7f328e05f700 1 --2- 172.21.15.158:0/3578459269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3274004d90 0x7f3274005210 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f328000a8d0 tx=0x7f3274005880 comp rx=0 tx=0).stop 2023-10-16T10:02:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.799+0000 7f3287fff700 1 -- 172.21.15.158:0/3578459269 reap_dead start 2023-10-16T10:02:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.799+0000 7f328e05f700 1 -- 172.21.15.158:0/3578459269 shutdown_connections 2023-10-16T10:02:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.799+0000 7f328e05f700 1 -- 172.21.15.158:0/3578459269 >> 172.21.15.158:0/3578459269 conn(0x7f328809c940 msgr2=0x7f32880d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.799+0000 7f328e05f700 1 -- 172.21.15.158:0/3578459269 shutdown_connections 2023-10-16T10:02:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.799+0000 7f328e05f700 1 -- 172.21.15.158:0/3578459269 wait complete. 2023-10-16T10:02:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.800+0000 7f328e05f700 1 Processor -- start 2023-10-16T10:02:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.800+0000 7f328e05f700 1 -- start start 2023-10-16T10:02:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f328e05f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32880c9380 0x7f32881f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f328e05f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3288160360 0x7f32881f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f328e05f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32880c9760 0x7f3288152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f328e05f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32881f7410 con 0x7f32880c9380 2023-10-16T10:02:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f328e05f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32881f7160 con 0x7f3288160360 2023-10-16T10:02:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f328e05f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328815f5a0 con 0x7f32880c9760 2023-10-16T10:02:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f32877fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32880c9380 0x7f32881f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f32877fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32880c9380 0x7f32881f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56546/0 (socket says 172.21.15.158:56546) 2023-10-16T10:02:07.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f32877fe700 1 -- 172.21.15.158:0/1342844774 learned_addr learned my addr 172.21.15.158:0/1342844774 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f3286ffd700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3288160360 0x7f32881f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f3287fff700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32880c9760 0x7f3288152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f32877fe700 1 -- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32880c9760 msgr2=0x7f3288152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f32877fe700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32880c9760 0x7f3288152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f32877fe700 1 -- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3288160360 msgr2=0x7f32881f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.801+0000 7f32877fe700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3288160360 0x7f32881f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f32877fe700 1 -- 172.21.15.158:0/1342844774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328815f880 con 0x7f32880c9380 2023-10-16T10:02:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f3287fff700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32880c9760 0x7f3288152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f32877fe700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32880c9380 0x7f32881f66e0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f3278011aa0 tx=0x7f3278011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f327ffff700 1 -- 172.21.15.158:0/1342844774 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327800f980 con 0x7f32880c9380 2023-10-16T10:02:07.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f327ffff700 1 -- 172.21.15.158:0/1342844774 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327800fb00 con 0x7f32880c9380 2023-10-16T10:02:07.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328815fb60 con 0x7f32880c9380 2023-10-16T10:02:07.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f327ffff700 1 -- 172.21.15.158:0/1342844774 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327800feb0 con 0x7f32880c9380 2023-10-16T10:02:07.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.802+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32881526b0 con 0x7f32880c9380 2023-10-16T10:02:07.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.803+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32880681b0 con 0x7f32880c9380 2023-10-16T10:02:07.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.803+0000 7f327ffff700 1 -- 172.21.15.158:0/1342844774 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f327801d400 con 0x7f32880c9380 2023-10-16T10:02:07.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.804+0000 7f327ffff700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f326806b020 0x7f326806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:07.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.804+0000 7f327ffff700 1 -- 172.21.15.158:0/1342844774 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5601+0+0 (secure 0 0 0) 0x7f3278089e60 con 0x7f32880c9380 2023-10-16T10:02:07.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.804+0000 7f3286ffd700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f326806b020 0x7f326806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:07.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.804+0000 7f3286ffd700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f326806b020 0x7f326806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f328806c610 tx=0x7f3270008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:07.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:07.808+0000 7f327ffff700 1 -- 172.21.15.158:0/1342844774 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3278054f20 con 0x7f32880c9380 2023-10-16T10:02:08.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.002+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f32880d0d30 con 0x7f32880c9380 2023-10-16T10:02:08.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.169+0000 7f327ffff700 1 -- 172.21.15.158:0/1342844774 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v123) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3278058540 con 0x7f32880c9380 2023-10-16T10:02:08.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f326806b020 msgr2=0x7f326806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:08.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f326806b020 0x7f326806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f328806c610 tx=0x7f3270008040 comp rx=0 tx=0).stop 2023-10-16T10:02:08.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32880c9380 msgr2=0x7f32881f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32880c9380 0x7f32881f66e0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f3278011aa0 tx=0x7f3278011e60 comp rx=0 tx=0).stop 2023-10-16T10:02:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 shutdown_connections 2023-10-16T10:02:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32880c9760 0x7f3288152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f326806b020 0x7f326806d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32880c9380 0x7f32881f66e0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 --2- 172.21.15.158:0/1342844774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3288160360 0x7f32881f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 >> 172.21.15.158:0/1342844774 conn(0x7f328809c940 msgr2=0x7f328815ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:08.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 shutdown_connections 2023-10-16T10:02:08.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.173+0000 7f328e05f700 1 -- 172.21.15.158:0/1342844774 wait complete. 2023-10-16T10:02:08.180 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T10:02:08.190 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T10:02:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.316+0000 7fdbe0f0f700 1 Processor -- start 2023-10-16T10:02:08.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.316+0000 7fdbe0f0f700 1 -- start start 2023-10-16T10:02:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbe0f0f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbdc06f610 0x7fdbdc0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbe0f0f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdbdc06bff0 0x7fdbdc150c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbe0f0f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc15e090 con 0x7fdbdc072170 2023-10-16T10:02:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbe0f0f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc15ed10 con 0x7fdbdc06bff0 2023-10-16T10:02:08.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbe0f0f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc15e370 con 0x7fdbdc06f610 2023-10-16T10:02:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbdad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdbdc06bff0 0x7fdbdc150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbdad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdbdc06bff0 0x7fdbdc150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56554/0 (socket says 172.21.15.158:56554) 2023-10-16T10:02:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbdad9d700 1 -- 172.21.15.158:0/2956543218 learned_addr learned my addr 172.21.15.158:0/2956543218 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.317+0000 7fdbda59c700 1 --2- 172.21.15.158:0/2956543218 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc06f610 0x7fdbdc0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:08.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbdad9d700 1 -- 172.21.15.158:0/2956543218 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc06f610 msgr2=0x7fdbdc0aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:08.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbdad9d700 1 --2- 172.21.15.158:0/2956543218 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc06f610 0x7fdbdc0aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbdad9d700 1 -- 172.21.15.158:0/2956543218 >> v1:172.21.15.172:6789/0 conn(0x7fdbdc072170 legacy=0x7fdbdc06b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:08.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbdad9d700 1 -- 172.21.15.158:0/2956543218 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc15e650 con 0x7fdbdc06bff0 2023-10-16T10:02:08.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbda59c700 1 --2- 172.21.15.158:0/2956543218 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc06f610 0x7fdbdc0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:08.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbdad9d700 1 --2- 172.21.15.158:0/2956543218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc06bff0 0x7fdbdc150c20 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc0672b0 tx=0x7fdbd400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181e2b60e8eb876e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:08.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbd959a700 1 -- 172.21.15.158:0/2956543218 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd400e070 con 0x7fdbdc06bff0 2023-10-16T10:02:08.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbd959a700 1 -- 172.21.15.158:0/2956543218 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd400b9c0 con 0x7fdbdc06bff0 2023-10-16T10:02:08.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbd959a700 1 -- 172.21.15.158:0/2956543218 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd400bd70 con 0x7fdbdc06bff0 2023-10-16T10:02:08.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/2956543218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc06bff0 msgr2=0x7fdbdc150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/2956543218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc06bff0 0x7fdbdc150c20 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc0672b0 tx=0x7fdbd400b330 comp rx=0 tx=0).stop 2023-10-16T10:02:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/2956543218 shutdown_connections 2023-10-16T10:02:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/2956543218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc06bff0 0x7fdbdc150c20 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/2956543218 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc06f610 0x7fdbdc0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.318+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/2956543218 >> 172.21.15.158:0/2956543218 conn(0x7fdbdc0710f0 msgr2=0x7fdbdc071500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:08.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.319+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/2956543218 shutdown_connections 2023-10-16T10:02:08.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.319+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/2956543218 wait complete. 2023-10-16T10:02:08.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.320+0000 7fdbe0f0f700 1 Processor -- start 2023-10-16T10:02:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.320+0000 7fdbe0f0f700 1 -- start start 2023-10-16T10:02:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbe0f0f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc06bff0 0x7fdbdc1593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbe0f0f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc06f610 0x7fdbdc15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbe0f0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc072170 0x7fdbdc14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:08.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbe0f0f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc13ace0 con 0x7fdbdc072170 2023-10-16T10:02:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbe0f0f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc13a060 con 0x7fdbdc06bff0 2023-10-16T10:02:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbe0f0f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc13a340 con 0x7fdbdc06f610 2023-10-16T10:02:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbdad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc072170 0x7fdbdc14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbdad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc072170 0x7fdbdc14de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56560/0 (socket says 172.21.15.158:56560) 2023-10-16T10:02:08.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbdad9d700 1 -- 172.21.15.158:0/3332972407 learned_addr learned my addr 172.21.15.158:0/3332972407 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:08.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbda59c700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc06bff0 0x7fdbdc1593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:08.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.321+0000 7fdbd9d9b700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc06f610 0x7fdbdc15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:08.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbdad9d700 1 -- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc06f610 msgr2=0x7fdbdc15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:08.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbdad9d700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc06f610 0x7fdbdc15cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbdad9d700 1 -- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc06bff0 msgr2=0x7fdbdc1593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:08.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbdad9d700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc06bff0 0x7fdbdc1593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:08.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbdad9d700 1 -- 172.21.15.158:0/3332972407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd4009b30 con 0x7fdbdc072170 2023-10-16T10:02:08.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbdad9d700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc072170 0x7fdbdc14de10 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc073660 tx=0x7fdbd4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:08.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbcaffd700 1 -- 172.21.15.158:0/3332972407 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4004930 con 0x7fdbdc072170 2023-10-16T10:02:08.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbcaffd700 1 -- 172.21.15.158:0/3332972407 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd40099b0 con 0x7fdbdc072170 2023-10-16T10:02:08.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbcaffd700 1 -- 172.21.15.158:0/3332972407 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd401b670 con 0x7fdbdc072170 2023-10-16T10:02:08.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc13a620 con 0x7fdbdc072170 2023-10-16T10:02:08.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.322+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc139cc0 con 0x7fdbdc072170 2023-10-16T10:02:08.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.323+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc066db0 con 0x7fdbdc072170 2023-10-16T10:02:08.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.324+0000 7fdbcaffd700 1 -- 172.21.15.158:0/3332972407 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdbd401b7d0 con 0x7fdbdc072170 2023-10-16T10:02:08.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.324+0000 7fdbcaffd700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdbbc06b020 0x7fdbbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:08.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.324+0000 7fdbcaffd700 1 -- 172.21.15.158:0/3332972407 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5971+0+0 (secure 0 0 0) 0x7fdbd408ae10 con 0x7fdbdc072170 2023-10-16T10:02:08.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.324+0000 7fdbda59c700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdbbc06b020 0x7fdbbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:08.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.324+0000 7fdbda59c700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdbbc06b020 0x7fdbbc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c940 tx=0x7fdbcc008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:08.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.329+0000 7fdbcaffd700 1 -- 172.21.15.158:0/3332972407 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbd4055c20 con 0x7fdbdc072170 2023-10-16T10:02:08.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:08.520+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdbdc0681b0 con 0x7fdbdc072170 2023-10-16T10:02:09.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.172+0000 7fdbcaffd700 1 -- 172.21.15.158:0/3332972407 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v124) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdbd4059240 con 0x7fdbdc072170 2023-10-16T10:02:09.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.175+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdbbc06b020 msgr2=0x7fdbbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:09.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.175+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdbbc06b020 0x7fdbbc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c940 tx=0x7fdbcc008040 comp rx=0 tx=0).stop 2023-10-16T10:02:09.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.175+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc072170 msgr2=0x7fdbdc14de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:09.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.175+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc072170 0x7fdbdc14de10 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc073660 tx=0x7fdbd4004ad0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.177+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 shutdown_connections 2023-10-16T10:02:09.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.177+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fdbbc06b020 0x7fdbbc06d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.177+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdbdc072170 0x7fdbdc14de10 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.177+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc06bff0 0x7fdbdc1593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.177+0000 7fdbe0f0f700 1 --2- 172.21.15.158:0/3332972407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc06f610 0x7fdbdc15cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.177+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 >> 172.21.15.158:0/3332972407 conn(0x7fdbdc0710f0 msgr2=0x7fdbdc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:09.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.177+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 shutdown_connections 2023-10-16T10:02:09.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.178+0000 7fdbe0f0f700 1 -- 172.21.15.158:0/3332972407 wait complete. 2023-10-16T10:02:09.185 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T10:02:09.204 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T10:02:09.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.329+0000 7f795b384700 1 Processor -- start 2023-10-16T10:02:09.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.329+0000 7f795b384700 1 -- start start 2023-10-16T10:02:09.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.330+0000 7f795b384700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7954160d30 0x7f79540b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.330+0000 7f795b384700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f79540d0530 0x7f79540d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.330+0000 7f795b384700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79540b7c90 con 0x7f79540d19a0 2023-10-16T10:02:09.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.330+0000 7f795b384700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79540b8910 con 0x7f79540d0530 2023-10-16T10:02:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.330+0000 7f795b384700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79540d0ef0 con 0x7f7954160d30 2023-10-16T10:02:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f795891f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f79540d0530 0x7f79540d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f7959921700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f79540d19a0 0x7f79540d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56744/0 (socket says 172.21.15.158:56744) 2023-10-16T10:02:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f7959921700 1 -- 172.21.15.158:0/2448643433 learned_addr learned my addr 172.21.15.158:0/2448643433 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f7959120700 1 --2- 172.21.15.158:0/2448643433 >> v2:172.21.15.172:3300/0 conn(0x7f7954160d30 0x7f79540b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f795891f700 1 -- 172.21.15.158:0/2448643433 >> v2:172.21.15.172:3300/0 conn(0x7f7954160d30 msgr2=0x7f79540b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f795891f700 1 --2- 172.21.15.158:0/2448643433 >> v2:172.21.15.172:3300/0 conn(0x7f7954160d30 0x7f79540b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f795891f700 1 -- 172.21.15.158:0/2448643433 >> v1:172.21.15.158:6789/0 conn(0x7f79540d19a0 legacy=0x7f79540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f7953fff700 1 -- 172.21.15.158:0/2448643433 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762136975 0 0) 0x7f79540b7c90 con 0x7f79540d19a0 2023-10-16T10:02:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f795891f700 1 -- 172.21.15.158:0/2448643433 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79540b7f70 con 0x7f79540d0530 2023-10-16T10:02:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f795891f700 1 --2- 172.21.15.158:0/2448643433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79540d0530 0x7f79540d09b0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f7948009840 tx=0x7f7948009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4049143e7a7cdc75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f7953fff700 1 -- 172.21.15.158:0/2448643433 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948011830 con 0x7f79540d0530 2023-10-16T10:02:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f7953fff700 1 -- 172.21.15.158:0/2448643433 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7948011990 con 0x7f79540d0530 2023-10-16T10:02:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.331+0000 7f7953fff700 1 -- 172.21.15.158:0/2448643433 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948011d20 con 0x7f79540d0530 2023-10-16T10:02:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 -- 172.21.15.158:0/2448643433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79540d0530 msgr2=0x7f79540d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 --2- 172.21.15.158:0/2448643433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79540d0530 0x7f79540d09b0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f7948009840 tx=0x7f7948009b50 comp rx=0 tx=0).stop 2023-10-16T10:02:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 -- 172.21.15.158:0/2448643433 shutdown_connections 2023-10-16T10:02:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 --2- 172.21.15.158:0/2448643433 >> v2:172.21.15.172:3300/0 conn(0x7f7954160d30 0x7f79540b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 --2- 172.21.15.158:0/2448643433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79540d0530 0x7f79540d09b0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 -- 172.21.15.158:0/2448643433 >> 172.21.15.158:0/2448643433 conn(0x7f79540669f0 msgr2=0x7f7954066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 -- 172.21.15.158:0/2448643433 shutdown_connections 2023-10-16T10:02:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.332+0000 7f795b384700 1 -- 172.21.15.158:0/2448643433 wait complete. 2023-10-16T10:02:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.333+0000 7f795b384700 1 Processor -- start 2023-10-16T10:02:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.333+0000 7f795b384700 1 -- start start 2023-10-16T10:02:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f795b384700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79540d0530 0x7f79540cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f795b384700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79540d19a0 0x7f795415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f795b384700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7954160d30 0x7f79540cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f795b384700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7954152e70 con 0x7f7954160d30 2023-10-16T10:02:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f795b384700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79540ccba0 con 0x7f79540d0530 2023-10-16T10:02:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f795b384700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79540cc300 con 0x7f79540d19a0 2023-10-16T10:02:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f7959921700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7954160d30 0x7f79540cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f7959921700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7954160d30 0x7f79540cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56574/0 (socket says 172.21.15.158:56574) 2023-10-16T10:02:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.334+0000 7f7959921700 1 -- 172.21.15.158:0/1238376586 learned_addr learned my addr 172.21.15.158:0/1238376586 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7959120700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79540d0530 0x7f79540cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f795891f700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79540d19a0 0x7f795415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7959921700 1 -- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79540d19a0 msgr2=0x7f795415e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7959921700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79540d19a0 0x7f795415e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7959921700 1 -- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79540d0530 msgr2=0x7f79540cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7959921700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79540d0530 0x7f79540cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7959921700 1 -- 172.21.15.158:0/1238376586 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480094a0 con 0x7f7954160d30 2023-10-16T10:02:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7959921700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7954160d30 0x7f79540cb420 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f7944002230 tx=0x7f794400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7951ffb700 1 -- 172.21.15.158:0/1238376586 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794400ab70 con 0x7f7954160d30 2023-10-16T10:02:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7951ffb700 1 -- 172.21.15.158:0/1238376586 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794400acf0 con 0x7f7954160d30 2023-10-16T10:02:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f7951ffb700 1 -- 172.21.15.158:0/1238376586 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944007310 con 0x7f7954160d30 2023-10-16T10:02:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79540cc5e0 con 0x7f7954160d30 2023-10-16T10:02:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.335+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79540cb960 con 0x7f7954160d30 2023-10-16T10:02:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.336+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7954152fe0 con 0x7f7954160d30 2023-10-16T10:02:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.337+0000 7f7951ffb700 1 -- 172.21.15.158:0/1238376586 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7944007470 con 0x7f7954160d30 2023-10-16T10:02:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.337+0000 7f7951ffb700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f794006b020 0x7f794006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.337+0000 7f7951ffb700 1 -- 172.21.15.158:0/1238376586 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f79440894e0 con 0x7f7954160d30 2023-10-16T10:02:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.337+0000 7f7959120700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f794006b020 0x7f794006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.337+0000 7f7959120700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f794006b020 0x7f794006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f794c006fd0 tx=0x7f794c008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.342+0000 7f7951ffb700 1 -- 172.21.15.158:0/1238376586 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79440541f0 con 0x7f7954160d30 2023-10-16T10:02:09.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:09.579+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f79540b8500 con 0x7f7954160d30 2023-10-16T10:02:10.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.197+0000 7f7951ffb700 1 -- 172.21.15.158:0/1238376586 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v56) v1 ==== 174+0+0 (secure 0 0 0) 0x7f794400ae60 con 0x7f7954160d30 2023-10-16T10:02:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.209+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f794006b020 msgr2=0x7f794006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.209+0000 7f795b384700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f794006b020 0x7f794006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f794c006fd0 tx=0x7f794c008040 comp rx=0 tx=0).stop 2023-10-16T10:02:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.209+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7954160d30 msgr2=0x7f79540cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.209+0000 7f795b384700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7954160d30 0x7f79540cb420 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f7944002230 tx=0x7f794400cf50 comp rx=0 tx=0).stop 2023-10-16T10:02:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.211+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 shutdown_connections 2023-10-16T10:02:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.211+0000 7f795b384700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f794006b020 0x7f794006d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.211+0000 7f795b384700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7954160d30 0x7f79540cb420 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.211+0000 7f795b384700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79540d0530 0x7f79540cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.211+0000 7f795b384700 1 --2- 172.21.15.158:0/1238376586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79540d19a0 0x7f795415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.211+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 >> 172.21.15.158:0/1238376586 conn(0x7f79540669f0 msgr2=0x7f795415b410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.211+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 shutdown_connections 2023-10-16T10:02:10.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.212+0000 7f795b384700 1 -- 172.21.15.158:0/1238376586 wait complete. 2023-10-16T10:02:10.216 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 15 and data pool 16 2023-10-16T10:02:10.229 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:02:10.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.349+0000 7f79d6ae1700 1 Processor -- start 2023-10-16T10:02:10.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.349+0000 7f79d6ae1700 1 -- start start 2023-10-16T10:02:10.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79d6ae1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 0x7f79d01521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79d6ae1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d015c5d0 con 0x7f79d00ce2c0 2023-10-16T10:02:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79d6ae1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d015c8b0 con 0x7f79d00cdee0 2023-10-16T10:02:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79d6ae1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d015d2c0 con 0x7f79d00b6e10 2023-10-16T10:02:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79d507e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f79d00ce2c0 0x7f79d00cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56772/0 (socket says 172.21.15.158:56772) 2023-10-16T10:02:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79d507e700 1 -- 172.21.15.158:0/512838711 learned_addr learned my addr 172.21.15.158:0/512838711 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cffff700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 0x7f79d01521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142791389 0 0) 0x7f79d015c5d0 con 0x7f79d00ce2c0 2023-10-16T10:02:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c00035b0 con 0x7f79d00ce2c0 2023-10-16T10:02:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439240919 0 0) 0x7f79d015c8b0 con 0x7f79d00cdee0 2023-10-16T10:02:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d015c5d0 con 0x7f79d00cdee0 2023-10-16T10:02:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850697616 0 0) 0x7f79c00035b0 con 0x7f79d00ce2c0 2023-10-16T10:02:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 msgr2=0x7f79d01521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 0x7f79d01521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 >> v1:172.21.15.172:6789/0 conn(0x7f79d00cdee0 legacy=0x7f79d015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d015cb90 con 0x7f79d00ce2c0 2023-10-16T10:02:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f79c80019f0 con 0x7f79d00ce2c0 2023-10-16T10:02:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 >> v1:172.21.15.158:6789/0 conn(0x7f79d00ce2c0 legacy=0x7f79d00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 0x7f79c0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.350+0000 7f79cf7fe700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004170 0x7f79c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cf7fe700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004d90 0x7f79c0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d015d2c0 con 0x7f79c0004170 2023-10-16T10:02:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c000bbd0 con 0x7f79c0004d90 2023-10-16T10:02:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c000bd90 con 0x7f79d00b6e10 2023-10-16T10:02:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cffff700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004170 0x7f79c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79d487d700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 0x7f79c0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79d507e700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004d90 0x7f79c0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cffff700 1 -- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 msgr2=0x7f79c0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:10.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cffff700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00b6e10 0x7f79c0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cffff700 1 -- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004d90 msgr2=0x7f79c0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:10.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cffff700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004d90 0x7f79c0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cffff700 1 -- 172.21.15.158:0/512838711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00035b0 con 0x7f79c0004170 2023-10-16T10:02:10.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79d507e700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004d90 0x7f79c0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:10.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cffff700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004170 0x7f79c0004850 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009b60 tx=0x7f79c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f070135ede2136 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:10.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00e6c0 con 0x7f79c0004170 2023-10-16T10:02:10.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79bc00e820 con 0x7f79c0004170 2023-10-16T10:02:10.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.351+0000 7f79cf7fe700 1 -- 172.21.15.158:0/512838711 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00eb90 con 0x7f79c0004170 2023-10-16T10:02:10.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.352+0000 7f79d6ae1700 1 -- 172.21.15.158:0/512838711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004170 msgr2=0x7f79c0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.352+0000 7f79d6ae1700 1 --2- 172.21.15.158:0/512838711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004170 0x7f79c0004850 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009b60 tx=0x7f79c0005880 comp rx=0 tx=0).stop 2023-10-16T10:02:10.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.352+0000 7f79d507e700 1 -- 172.21.15.158:0/512838711 reap_dead start 2023-10-16T10:02:10.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.352+0000 7f79d6ae1700 1 -- 172.21.15.158:0/512838711 shutdown_connections 2023-10-16T10:02:10.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.352+0000 7f79d6ae1700 1 -- 172.21.15.158:0/512838711 >> 172.21.15.158:0/512838711 conn(0x7f79d006a8d0 msgr2=0x7f79d006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:10.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.352+0000 7f79d6ae1700 1 -- 172.21.15.158:0/512838711 shutdown_connections 2023-10-16T10:02:10.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.352+0000 7f79d6ae1700 1 -- 172.21.15.158:0/512838711 wait complete. 2023-10-16T10:02:10.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.353+0000 7f79d6ae1700 1 Processor -- start 2023-10-16T10:02:10.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.353+0000 7f79d6ae1700 1 -- start start 2023-10-16T10:02:10.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d6ae1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004170 0x7f79d014d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d6ae1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79d00c9c10 0x7f79d0149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d6ae1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00d1820 0x7f79d014c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d6ae1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d014b3c0 con 0x7f79d00c9c10 2023-10-16T10:02:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d6ae1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00ca090 con 0x7f79c0004170 2023-10-16T10:02:10.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d6ae1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d014a7b0 con 0x7f79d00d1820 2023-10-16T10:02:10.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79d00c9c10 0x7f79d0149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79d00c9c10 0x7f79d0149f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56606/0 (socket says 172.21.15.158:56606) 2023-10-16T10:02:10.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 -- 172.21.15.158:0/4230146513 learned_addr learned my addr 172.21.15.158:0/4230146513 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:10.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d487d700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004170 0x7f79d014d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d507e700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00d1820 0x7f79d014c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 -- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00d1820 msgr2=0x7f79d014c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00d1820 0x7f79d014c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 -- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004170 msgr2=0x7f79d014d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004170 0x7f79d014d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 -- 172.21.15.158:0/4230146513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d014aa90 con 0x7f79d00c9c10 2023-10-16T10:02:10.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79d507e700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00d1820 0x7f79d014c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:10.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.354+0000 7f79cffff700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79d00c9c10 0x7f79d0149f10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f79c000bba0 tx=0x7f79bc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:10.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.355+0000 7f79cd7fa700 1 -- 172.21.15.158:0/4230146513 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00e6c0 con 0x7f79d00c9c10 2023-10-16T10:02:10.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.355+0000 7f79cd7fa700 1 -- 172.21.15.158:0/4230146513 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79bc003ab0 con 0x7f79d00c9c10 2023-10-16T10:02:10.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.355+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d014ad70 con 0x7f79d00c9c10 2023-10-16T10:02:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.355+0000 7f79cd7fa700 1 -- 172.21.15.158:0/4230146513 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79bc003e60 con 0x7f79d00c9c10 2023-10-16T10:02:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.355+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d014a450 con 0x7f79d00c9c10 2023-10-16T10:02:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.356+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d014b530 con 0x7f79d00c9c10 2023-10-16T10:02:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.356+0000 7f79cd7fa700 1 -- 172.21.15.158:0/4230146513 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79bc013470 con 0x7f79d00c9c10 2023-10-16T10:02:10.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.357+0000 7f79cd7fa700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f79b406b020 0x7f79b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.357+0000 7f79cd7fa700 1 -- 172.21.15.158:0/4230146513 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f79bc08b250 con 0x7f79d00c9c10 2023-10-16T10:02:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.357+0000 7f79d487d700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f79b406b020 0x7f79b406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.357+0000 7f79d487d700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f79b406b020 0x7f79b406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f79d0159860 tx=0x7f79c4009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:10.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.361+0000 7f79cd7fa700 1 -- 172.21.15.158:0/4230146513 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79bc0560b0 con 0x7f79d00c9c10 2023-10-16T10:02:10.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.557+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79d00681b0 con 0x7f79d00c9c10 2023-10-16T10:02:10.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.558+0000 7f79cd7fa700 1 -- 172.21.15.158:0/4230146513 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+17386 (secure 0 0 0) 0x7f79bc0596d0 con 0x7f79d00c9c10 2023-10-16T10:02:10.560 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:10.560 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":125,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:09.580443+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:02:08.021999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T10:02:08.539662+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:02:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f79b406b020 msgr2=0x7f79b406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f79b406b020 0x7f79b406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f79d0159860 tx=0x7f79c4009040 comp rx=0 tx=0).stop 2023-10-16T10:02:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79d00c9c10 msgr2=0x7f79d0149f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79d00c9c10 0x7f79d0149f10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f79c000bba0 tx=0x7f79bc0047c0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 shutdown_connections 2023-10-16T10:02:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79d00d1820 0x7f79d014c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c0004170 0x7f79d014d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f79b406b020 0x7f79b406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 --2- 172.21.15.158:0/4230146513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79d00c9c10 0x7f79d0149f10 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 >> 172.21.15.158:0/4230146513 conn(0x7f79d006a8d0 msgr2=0x7f79d006ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 shutdown_connections 2023-10-16T10:02:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.560+0000 7f79d6ae1700 1 -- 172.21.15.158:0/4230146513 wait complete. 2023-10-16T10:02:10.574 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:02:10.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.693+0000 7f68955e9700 1 Processor -- start 2023-10-16T10:02:10.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.693+0000 7f68955e9700 1 -- start start 2023-10-16T10:02:10.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f68955e9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68900b7810 0x7f68901683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f68955e9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68900d3bd0 con 0x7f68900d1210 2023-10-16T10:02:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f68955e9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68900d2fc0 con 0x7f689015d7c0 2023-10-16T10:02:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f68955e9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68900d4060 con 0x7f68900b7810 2023-10-16T10:02:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68900b7810 0x7f68901683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68900b7810 0x7f68901683f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56616/0 (socket says 172.21.15.158:56616) 2023-10-16T10:02:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688e7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f68900d1210 0x7f68900ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56790/0 (socket says 172.21.15.158:56790) 2023-10-16T10:02:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688f7fe700 1 -- 172.21.15.158:0/3089132973 learned_addr learned my addr 172.21.15.158:0/3089132973 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688f7fe700 1 -- 172.21.15.158:0/3089132973 >> v1:172.21.15.172:6789/0 conn(0x7f689015d7c0 legacy=0x7f68900d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688f7fe700 1 -- 172.21.15.158:0/3089132973 >> v1:172.21.15.158:6789/0 conn(0x7f68900d1210 legacy=0x7f68900ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688f7fe700 1 -- 172.21.15.158:0/3089132973 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68900d32a0 con 0x7f68900b7810 2023-10-16T10:02:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688dffb700 1 -- 172.21.15.158:0/3089132973 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201187615 0 0) 0x7f68900d3bd0 con 0x7f68900d1210 2023-10-16T10:02:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688f7fe700 1 --2- 172.21.15.158:0/3089132973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f68901683f0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6888009e50 tx=0x7f688800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b66f3cedab763a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688dffb700 1 -- 172.21.15.158:0/3089132973 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688800e070 con 0x7f68900b7810 2023-10-16T10:02:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688dffb700 1 -- 172.21.15.158:0/3089132973 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688800b9c0 con 0x7f68900b7810 2023-10-16T10:02:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.694+0000 7f688dffb700 1 -- 172.21.15.158:0/3089132973 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688800bd70 con 0x7f68900b7810 2023-10-16T10:02:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.695+0000 7f68955e9700 1 -- 172.21.15.158:0/3089132973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 msgr2=0x7f68901683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.695+0000 7f68955e9700 1 --2- 172.21.15.158:0/3089132973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f68901683f0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6888009e50 tx=0x7f688800b330 comp rx=0 tx=0).stop 2023-10-16T10:02:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.695+0000 7f68955e9700 1 -- 172.21.15.158:0/3089132973 shutdown_connections 2023-10-16T10:02:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.695+0000 7f68955e9700 1 --2- 172.21.15.158:0/3089132973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f68901683f0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.695+0000 7f68955e9700 1 -- 172.21.15.158:0/3089132973 >> 172.21.15.158:0/3089132973 conn(0x7f68900669f0 msgr2=0x7f6890066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:10.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.695+0000 7f68955e9700 1 -- 172.21.15.158:0/3089132973 shutdown_connections 2023-10-16T10:02:10.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.695+0000 7f68955e9700 1 -- 172.21.15.158:0/3089132973 wait complete. 2023-10-16T10:02:10.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.696+0000 7f68955e9700 1 Processor -- start 2023-10-16T10:02:10.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.696+0000 7f68955e9700 1 -- start start 2023-10-16T10:02:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f68955e9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f689015a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f68955e9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68900d1210 0x7f689015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f68955e9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f689015d7c0 0x7f68901f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f68955e9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900c6e00 con 0x7f68900b7810 2023-10-16T10:02:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f68955e9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6890157ad0 con 0x7f68900d1210 2023-10-16T10:02:10.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f68955e9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68900c70a0 con 0x7f689015d7c0 2023-10-16T10:02:10.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f689015a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f689015a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56618/0 (socket says 172.21.15.158:56618) 2023-10-16T10:02:10.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 -- 172.21.15.158:0/1837720386 learned_addr learned my addr 172.21.15.158:0/1837720386 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:10.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 -- 172.21.15.158:0/1837720386 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f689015d7c0 msgr2=0x7f68901f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f689015d7c0 0x7f68901f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688e7fc700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68900d1210 0x7f689015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 -- 172.21.15.158:0/1837720386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68900d1210 msgr2=0x7f689015a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68900d1210 0x7f689015a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 -- 172.21.15.158:0/1837720386 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6888009b30 con 0x7f68900b7810 2023-10-16T10:02:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.697+0000 7f688effd700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f689015a480 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f688000c4e0 tx=0x7f688000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.698+0000 7f68777fe700 1 -- 172.21.15.158:0/1837720386 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68800052d0 con 0x7f68900b7810 2023-10-16T10:02:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.698+0000 7f68777fe700 1 -- 172.21.15.158:0/1837720386 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6880005430 con 0x7f68900b7810 2023-10-16T10:02:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.698+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68900c7380 con 0x7f68900b7810 2023-10-16T10:02:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.698+0000 7f68777fe700 1 -- 172.21.15.158:0/1837720386 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68800038c0 con 0x7f68900b7810 2023-10-16T10:02:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.698+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689015a0c0 con 0x7f68900b7810 2023-10-16T10:02:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.699+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6890157cd0 con 0x7f68900b7810 2023-10-16T10:02:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.699+0000 7f68777fe700 1 -- 172.21.15.158:0/1837720386 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6880003a20 con 0x7f68900b7810 2023-10-16T10:02:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.699+0000 7f68777fe700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f687c06b020 0x7f687c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.699+0000 7f68777fe700 1 -- 172.21.15.158:0/1837720386 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f6880088e90 con 0x7f68900b7810 2023-10-16T10:02:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.699+0000 7f688e7fc700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f687c06b020 0x7f687c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.700+0000 7f688e7fc700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f687c06b020 0x7f687c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68900b29b0 tx=0x7f6884009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.704+0000 7f68777fe700 1 -- 172.21.15.158:0/1837720386 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6890157cd0 con 0x7f68900b7810 2023-10-16T10:02:10.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.900+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f689015a250 con 0x7f68900b7810 2023-10-16T10:02:10.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.902+0000 7f68777fe700 1 -- 172.21.15.158:0/1837720386 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+17386 (secure 0 0 0) 0x7f6880017090 con 0x7f68900b7810 2023-10-16T10:02:10.903 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:10.904 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":125,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:09.580443+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:02:08.021999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T10:02:08.539662+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:02:10.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f687c06b020 msgr2=0x7f687c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f687c06b020 0x7f687c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68900b29b0 tx=0x7f6884009040 comp rx=0 tx=0).stop 2023-10-16T10:02:10.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 msgr2=0x7f689015a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:10.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f689015a480 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f688000c4e0 tx=0x7f688000c8a0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 shutdown_connections 2023-10-16T10:02:10.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f689015d7c0 0x7f68901f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f687c06b020 0x7f687c06d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68900b7810 0x7f689015a480 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 --2- 172.21.15.158:0/1837720386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68900d1210 0x7f689015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:10.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 >> 172.21.15.158:0/1837720386 conn(0x7f68900669f0 msgr2=0x7f689015c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:10.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 shutdown_connections 2023-10-16T10:02:10.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:10.904+0000 7f68955e9700 1 -- 172.21.15.158:0/1837720386 wait complete. 2023-10-16T10:02:10.918 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T10:02:11.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.041+0000 7f8a1e5ae700 1 Processor -- start 2023-10-16T10:02:11.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.041+0000 7f8a1e5ae700 1 -- start start 2023-10-16T10:02:11.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.041+0000 7f8a1e5ae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a1815da40 0x7f8a180caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.041+0000 7f8a1e5ae700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a1806afe0 0x7f8a180afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.041+0000 7f8a1e5ae700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180cbd30 con 0x7f8a180cc590 2023-10-16T10:02:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.041+0000 7f8a1e5ae700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a180cc1c0 con 0x7f8a1806afe0 2023-10-16T10:02:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.041+0000 7f8a1e5ae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1815de20 con 0x7f8a1815da40 2023-10-16T10:02:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a1806afe0 0x7f8a180afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a1806afe0 0x7f8a180afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56628/0 (socket says 172.21.15.158:56628) 2023-10-16T10:02:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/2808662590 learned_addr learned my addr 172.21.15.158:0/2808662590 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a17fff700 1 --2- 172.21.15.158:0/2808662590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a1815da40 0x7f8a180caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/2808662590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a1815da40 msgr2=0x7f8a180caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 --2- 172.21.15.158:0/2808662590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a1815da40 0x7f8a180caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/2808662590 >> v1:172.21.15.172:6789/0 conn(0x7f8a180cc590 legacy=0x7f8a1806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/2808662590 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180ce260 con 0x7f8a1806afe0 2023-10-16T10:02:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a1cb4b700 1 --2- 172.21.15.158:0/2808662590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1806afe0 0x7f8a180afa40 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009e50 tx=0x7f8a1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b74f2d06810983d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a16ffd700 1 -- 172.21.15.158:0/2808662590 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1000e070 con 0x7f8a1806afe0 2023-10-16T10:02:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a16ffd700 1 -- 172.21.15.158:0/2808662590 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a10003a20 con 0x7f8a1806afe0 2023-10-16T10:02:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.042+0000 7f8a16ffd700 1 -- 172.21.15.158:0/2808662590 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10003dd0 con 0x7f8a1806afe0 2023-10-16T10:02:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/2808662590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1806afe0 msgr2=0x7f8a180afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/2808662590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1806afe0 0x7f8a180afa40 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009e50 tx=0x7f8a1000b330 comp rx=0 tx=0).stop 2023-10-16T10:02:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/2808662590 shutdown_connections 2023-10-16T10:02:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/2808662590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1806afe0 0x7f8a180afa40 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/2808662590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a1815da40 0x7f8a180caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/2808662590 >> 172.21.15.158:0/2808662590 conn(0x7f8a18099a10 msgr2=0x7f8a180d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/2808662590 shutdown_connections 2023-10-16T10:02:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.043+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/2808662590 wait complete. 2023-10-16T10:02:11.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.044+0000 7f8a1e5ae700 1 Processor -- start 2023-10-16T10:02:11.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.044+0000 7f8a1e5ae700 1 -- start start 2023-10-16T10:02:11.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1e5ae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a1806afe0 0x7f8a180b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:11.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1e5ae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a180cc590 0x7f8a1809a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1e5ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1815da40 0x7f8a1809d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1e5ae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180b5850 con 0x7f8a1815da40 2023-10-16T10:02:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1e5ae700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180b61b0 con 0x7f8a1806afe0 2023-10-16T10:02:11.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1e5ae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180b4bd0 con 0x7f8a180cc590 2023-10-16T10:02:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1815da40 0x7f8a1809d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1815da40 0x7f8a1809d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56634/0 (socket says 172.21.15.158:56634) 2023-10-16T10:02:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/1223068670 learned_addr learned my addr 172.21.15.158:0/1223068670 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a177fe700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a180cc590 0x7f8a1809a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:11.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a17fff700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a1806afe0 0x7f8a180b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:11.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a180cc590 msgr2=0x7f8a1809a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:11.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a180cc590 0x7f8a1809a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:11.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a1806afe0 msgr2=0x7f8a180b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:11.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a1806afe0 0x7f8a180b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:11.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 -- 172.21.15.158:0/1223068670 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10009b30 con 0x7f8a1815da40 2023-10-16T10:02:11.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a17fff700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a1806afe0 0x7f8a180b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:11.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.045+0000 7f8a1cb4b700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1815da40 0x7f8a1809d320 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f8a10004000 tx=0x7f8a10004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:11.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.046+0000 7f8a14ff9700 1 -- 172.21.15.158:0/1223068670 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100094d0 con 0x7f8a1815da40 2023-10-16T10:02:11.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.046+0000 7f8a14ff9700 1 -- 172.21.15.158:0/1223068670 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a10004240 con 0x7f8a1815da40 2023-10-16T10:02:11.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.046+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a180b4eb0 con 0x7f8a1815da40 2023-10-16T10:02:11.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.046+0000 7f8a14ff9700 1 -- 172.21.15.158:0/1223068670 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100045f0 con 0x7f8a1815da40 2023-10-16T10:02:11.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.046+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a180b5470 con 0x7f8a1815da40 2023-10-16T10:02:11.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.047+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a180b5990 con 0x7f8a1815da40 2023-10-16T10:02:11.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.047+0000 7f8a14ff9700 1 -- 172.21.15.158:0/1223068670 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a10019070 con 0x7f8a1815da40 2023-10-16T10:02:11.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.047+0000 7f8a14ff9700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8a0406b020 0x7f8a0406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:11.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.047+0000 7f8a14ff9700 1 -- 172.21.15.158:0/1223068670 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f8a1008aa00 con 0x7f8a1815da40 2023-10-16T10:02:11.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.048+0000 7f8a17fff700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8a0406b020 0x7f8a0406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:11.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.048+0000 7f8a17fff700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8a0406b020 0x7f8a0406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009820 tx=0x7f8a08008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:11.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.052+0000 7f8a14ff9700 1 -- 172.21.15.158:0/1223068670 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a180b5990 con 0x7f8a1815da40 2023-10-16T10:02:11.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:11.294+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8a180b5990 con 0x7f8a1815da40 2023-10-16T10:02:12.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.215+0000 7f8a14ff9700 1 -- 172.21.15.158:0/1223068670 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v59) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8a180b5990 con 0x7f8a1815da40 2023-10-16T10:02:12.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.218+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8a0406b020 msgr2=0x7f8a0406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:12.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.218+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8a0406b020 0x7f8a0406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009820 tx=0x7f8a08008040 comp rx=0 tx=0).stop 2023-10-16T10:02:12.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.218+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1815da40 msgr2=0x7f8a1809d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:12.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.218+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1815da40 0x7f8a1809d320 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f8a10004000 tx=0x7f8a10004ad0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 shutdown_connections 2023-10-16T10:02:12.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8a0406b020 0x7f8a0406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a1815da40 0x7f8a1809d320 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a1806afe0 0x7f8a180b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 --2- 172.21.15.158:0/1223068670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a180cc590 0x7f8a1809a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 >> 172.21.15.158:0/1223068670 conn(0x7f8a18099a10 msgr2=0x7f8a1815c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:12.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 shutdown_connections 2023-10-16T10:02:12.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.219+0000 7f8a1e5ae700 1 -- 172.21.15.158:0/1223068670 wait complete. 2023-10-16T10:02:12.235 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T10:02:12.235 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T10:02:12.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.366+0000 7f4aa4c15700 1 Processor -- start 2023-10-16T10:02:12.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.366+0000 7f4aa4c15700 1 -- start start 2023-10-16T10:02:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4aa4c15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 0x7f4aa006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4aa4c15700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa006c210 con 0x7f4aa0141920 2023-10-16T10:02:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4aa4c15700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa0141d00 con 0x7f4aa00cde10 2023-10-16T10:02:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4aa4c15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa006b970 con 0x7f4aa0144840 2023-10-16T10:02:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4a9e59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4aa0141920 0x7f4aa00cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56814/0 (socket says 172.21.15.158:56814) 2023-10-16T10:02:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4a9e59c700 1 -- 172.21.15.158:0/3775829058 learned_addr learned my addr 172.21.15.158:0/3775829058 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4a9ed9d700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 0x7f4aa006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.367+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423259556 0 0) 0x7f4aa006c210 con 0x7f4aa0141920 2023-10-16T10:02:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a880035b0 con 0x7f4aa0141920 2023-10-16T10:02:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559546575 0 0) 0x7f4a880035b0 con 0x7f4aa0141920 2023-10-16T10:02:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 msgr2=0x7f4aa006b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 0x7f4aa006b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 >> v1:172.21.15.172:6789/0 conn(0x7f4aa00cde10 legacy=0x7f4aa006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa006bc50 con 0x7f4aa0141920 2023-10-16T10:02:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4a94001670 con 0x7f4aa0141920 2023-10-16T10:02:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 >> v1:172.21.15.158:6789/0 conn(0x7f4aa0141920 legacy=0x7f4aa00cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 0x7f4a88003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a88003f70 0x7f4a88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a88004d90 0x7f4a88005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa006b970 con 0x7f4a88003f70 2023-10-16T10:02:12.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8800bb20 con 0x7f4a88004d90 2023-10-16T10:02:12.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8800bce0 con 0x7f4aa0144840 2023-10-16T10:02:12.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9dd9b700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a88003f70 0x7f4a88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:12.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9e59c700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 0x7f4a88003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:12.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9ed9d700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a88004d90 0x7f4a88005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:12.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9dd9b700 1 -- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 msgr2=0x7f4a88003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:12.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9dd9b700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4aa0144840 0x7f4a88003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9dd9b700 1 -- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a88004d90 msgr2=0x7f4a88005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9dd9b700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a88004d90 0x7f4a88005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9dd9b700 1 -- 172.21.15.158:0/3775829058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a880035b0 con 0x7f4a88003f70 2023-10-16T10:02:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.368+0000 7f4a9dd9b700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a88003f70 0x7f4a88004850 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f4a900018d0 tx=0x7f4a88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b049a58ea5544a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000c690 con 0x7f4a88003f70 2023-10-16T10:02:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9000c7f0 con 0x7f4a88003f70 2023-10-16T10:02:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4a9d59a700 1 -- 172.21.15.158:0/3775829058 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000cba0 con 0x7f4a88003f70 2023-10-16T10:02:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4aa4c15700 1 -- 172.21.15.158:0/3775829058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a88003f70 msgr2=0x7f4a88004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4aa4c15700 1 --2- 172.21.15.158:0/3775829058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a88003f70 0x7f4a88004850 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f4a900018d0 tx=0x7f4a88005880 comp rx=0 tx=0).stop 2023-10-16T10:02:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4a9ed9d700 1 -- 172.21.15.158:0/3775829058 reap_dead start 2023-10-16T10:02:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4aa4c15700 1 -- 172.21.15.158:0/3775829058 shutdown_connections 2023-10-16T10:02:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4aa4c15700 1 -- 172.21.15.158:0/3775829058 >> 172.21.15.158:0/3775829058 conn(0x7f4aa00cc5b0 msgr2=0x7f4aa00cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4aa4c15700 1 -- 172.21.15.158:0/3775829058 shutdown_connections 2023-10-16T10:02:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.369+0000 7f4aa4c15700 1 -- 172.21.15.158:0/3775829058 wait complete. 2023-10-16T10:02:12.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 Processor -- start 2023-10-16T10:02:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 -- start start 2023-10-16T10:02:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a88003f70 0x7f4aa015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4aa013d6c0 0x7f4aa00c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4aa0149480 0x7f4aa00c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00c9f20 con 0x7f4aa013d6c0 2023-10-16T10:02:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00c83d0 con 0x7f4aa0149480 2023-10-16T10:02:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4aa4c15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00c7aa0 con 0x7f4a88003f70 2023-10-16T10:02:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4a9dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4aa013d6c0 0x7f4aa00c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4a9dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4aa013d6c0 0x7f4aa00c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56650/0 (socket says 172.21.15.158:56650) 2023-10-16T10:02:12.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.371+0000 7f4a9dd9b700 1 -- 172.21.15.158:0/1410129766 learned_addr learned my addr 172.21.15.158:0/1410129766 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:12.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a9ed9d700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4aa0149480 0x7f4aa00c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:12.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a9dd9b700 1 -- 172.21.15.158:0/1410129766 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a88003f70 msgr2=0x7f4aa015a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:12.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a9dd9b700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a88003f70 0x7f4aa015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a9dd9b700 1 -- 172.21.15.158:0/1410129766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4aa0149480 msgr2=0x7f4aa00c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:12.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a9dd9b700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4aa0149480 0x7f4aa00c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:12.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a9dd9b700 1 -- 172.21.15.158:0/1410129766 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa00c7d80 con 0x7f4aa013d6c0 2023-10-16T10:02:12.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a9dd9b700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4aa013d6c0 0x7f4aa00c9890 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800baf0 tx=0x7f4a90001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:12.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a8effd700 1 -- 172.21.15.158:0/1410129766 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90000ed0 con 0x7f4aa013d6c0 2023-10-16T10:02:12.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a8effd700 1 -- 172.21.15.158:0/1410129766 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a90021070 con 0x7f4aa013d6c0 2023-10-16T10:02:12.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4a8effd700 1 -- 172.21.15.158:0/1410129766 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000cba0 con 0x7f4aa013d6c0 2023-10-16T10:02:12.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa00c8060 con 0x7f4aa013d6c0 2023-10-16T10:02:12.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.372+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa0149900 con 0x7f4aa013d6c0 2023-10-16T10:02:12.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.373+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa00c86b0 con 0x7f4aa013d6c0 2023-10-16T10:02:12.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.374+0000 7f4a8effd700 1 -- 172.21.15.158:0/1410129766 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4a90023030 con 0x7f4aa013d6c0 2023-10-16T10:02:12.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.374+0000 7f4a8effd700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4a8006b020 0x7f4a8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:12.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.374+0000 7f4a8effd700 1 -- 172.21.15.158:0/1410129766 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f4a9008abd0 con 0x7f4aa013d6c0 2023-10-16T10:02:12.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.374+0000 7f4a9e59c700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4a8006b020 0x7f4a8006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:12.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.374+0000 7f4a9e59c700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4a8006b020 0x7f4a8006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a94001b60 tx=0x7f4a94001bf0 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:12.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.378+0000 7f4a8effd700 1 -- 172.21.15.158:0/1410129766 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a90055910 con 0x7f4aa013d6c0 2023-10-16T10:02:12.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:12.618+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4aa00c8510 con 0x7f4aa013d6c0 2023-10-16T10:02:13.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.217+0000 7f4a8effd700 1 -- 172.21.15.158:0/1410129766 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v60) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4aa00c8510 con 0x7f4aa013d6c0 2023-10-16T10:02:13.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4a8006b020 msgr2=0x7f4a8006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4a8006b020 0x7f4a8006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a94001b60 tx=0x7f4a94001bf0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4aa013d6c0 msgr2=0x7f4aa00c9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4aa013d6c0 0x7f4aa00c9890 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800baf0 tx=0x7f4a90001230 comp rx=0 tx=0).stop 2023-10-16T10:02:13.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 shutdown_connections 2023-10-16T10:02:13.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4aa0149480 0x7f4aa00c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a88003f70 0x7f4aa015a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f4a8006b020 0x7f4a8006d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 --2- 172.21.15.158:0/1410129766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4aa013d6c0 0x7f4aa00c9890 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 >> 172.21.15.158:0/1410129766 conn(0x7f4aa00cc5b0 msgr2=0x7f4aa00cab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 shutdown_connections 2023-10-16T10:02:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.220+0000 7f4aa4c15700 1 -- 172.21.15.158:0/1410129766 wait complete. 2023-10-16T10:02:13.235 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T10:02:13.235 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:02:13.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.359+0000 7fc142d57700 1 Processor -- start 2023-10-16T10:02:13.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.359+0000 7fc142d57700 1 -- start start 2023-10-16T10:02:13.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc142d57700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c1619a0 0x7fc13c06b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc142d57700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc13c0ca620 0x7fc13c06b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc142d57700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c0c4d10 con 0x7fc13c160b20 2023-10-16T10:02:13.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc142d57700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c0c5620 con 0x7fc13c0ca620 2023-10-16T10:02:13.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc142d57700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c06c0a0 con 0x7fc13c1619a0 2023-10-16T10:02:13.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc1412f4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc13c0ca620 0x7fc13c06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc140af3700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc13c160b20 0x7fc13c161290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56834/0 (socket says 172.21.15.158:56834) 2023-10-16T10:02:13.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc1412f4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc13c0ca620 0x7fc13c06b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56660/0 (socket says 172.21.15.158:56660) 2023-10-16T10:02:13.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.360+0000 7fc140af3700 1 -- 172.21.15.158:0/3805204366 learned_addr learned my addr 172.21.15.158:0/3805204366 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:13.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc13bfff700 1 --2- 172.21.15.158:0/3805204366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c1619a0 0x7fc13c06b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc1412f4700 1 -- 172.21.15.158:0/3805204366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c1619a0 msgr2=0x7fc13c06b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:13.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc1412f4700 1 --2- 172.21.15.158:0/3805204366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c1619a0 0x7fc13c06b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc1412f4700 1 -- 172.21.15.158:0/3805204366 >> v1:172.21.15.158:6789/0 conn(0x7fc13c160b20 legacy=0x7fc13c161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc1412f4700 1 -- 172.21.15.158:0/3805204366 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0c4f60 con 0x7fc13c0ca620 2023-10-16T10:02:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc1412f4700 1 --2- 172.21.15.158:0/3805204366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c0ca620 0x7fc13c06b840 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fc134009d90 tx=0x7fc13400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e099060fba59677f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc13b7fe700 1 -- 172.21.15.158:0/3805204366 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13400e070 con 0x7fc13c0ca620 2023-10-16T10:02:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc13b7fe700 1 -- 172.21.15.158:0/3805204366 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13400bbd0 con 0x7fc13c0ca620 2023-10-16T10:02:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc13b7fe700 1 -- 172.21.15.158:0/3805204366 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134012430 con 0x7fc13c0ca620 2023-10-16T10:02:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc142d57700 1 -- 172.21.15.158:0/3805204366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c0ca620 msgr2=0x7fc13c06b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc142d57700 1 --2- 172.21.15.158:0/3805204366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c0ca620 0x7fc13c06b840 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fc134009d90 tx=0x7fc13400b330 comp rx=0 tx=0).stop 2023-10-16T10:02:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc142d57700 1 -- 172.21.15.158:0/3805204366 shutdown_connections 2023-10-16T10:02:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc142d57700 1 --2- 172.21.15.158:0/3805204366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c0ca620 0x7fc13c06b840 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc142d57700 1 --2- 172.21.15.158:0/3805204366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c1619a0 0x7fc13c06b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.361+0000 7fc142d57700 1 -- 172.21.15.158:0/3805204366 >> 172.21.15.158:0/3805204366 conn(0x7fc13c0cc4b0 msgr2=0x7fc13c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.362+0000 7fc142d57700 1 -- 172.21.15.158:0/3805204366 shutdown_connections 2023-10-16T10:02:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.362+0000 7fc142d57700 1 -- 172.21.15.158:0/3805204366 wait complete. 2023-10-16T10:02:13.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.363+0000 7fc142d57700 1 Processor -- start 2023-10-16T10:02:13.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.363+0000 7fc142d57700 1 -- start start 2023-10-16T10:02:13.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13c0ca620 0x7fc13c154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c160b20 0x7fc13c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c1619a0 0x7fc13c0b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1f6850 con 0x7fc13c1619a0 2023-10-16T10:02:13.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1f6aa0 con 0x7fc13c0ca620 2023-10-16T10:02:13.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1f6d80 con 0x7fc13c160b20 2023-10-16T10:02:13.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c1619a0 0x7fc13c0b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c1619a0 0x7fc13c0b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56668/0 (socket says 172.21.15.158:56668) 2023-10-16T10:02:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 -- 172.21.15.158:0/1455389611 learned_addr learned my addr 172.21.15.158:0/1455389611 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc140af3700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13c0ca620 0x7fc13c154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc13bfff700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c160b20 0x7fc13c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 -- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c160b20 msgr2=0x7fc13c0b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c160b20 0x7fc13c0b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 -- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13c0ca620 msgr2=0x7fc13c154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13c0ca620 0x7fc13c154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 -- 172.21.15.158:0/1455389611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc134009a70 con 0x7fc13c1619a0 2023-10-16T10:02:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1412f4700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c1619a0 0x7fc13c0b0e80 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc134014040 tx=0x7fc134004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1397fa700 1 -- 172.21.15.158:0/1455389611 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134012430 con 0x7fc13c1619a0 2023-10-16T10:02:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1397fa700 1 -- 172.21.15.158:0/1455389611 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc134020070 con 0x7fc13c1619a0 2023-10-16T10:02:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c1f7060 con 0x7fc13c1619a0 2023-10-16T10:02:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc1397fa700 1 -- 172.21.15.158:0/1455389611 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134003ea0 con 0x7fc13c1619a0 2023-10-16T10:02:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.364+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c0ac880 con 0x7fc13c1619a0 2023-10-16T10:02:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.365+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c1f7580 con 0x7fc13c1619a0 2023-10-16T10:02:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.366+0000 7fc1397fa700 1 -- 172.21.15.158:0/1455389611 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc134004040 con 0x7fc13c1619a0 2023-10-16T10:02:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.366+0000 7fc1397fa700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc12006b020 0x7fc12006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.366+0000 7fc1397fa700 1 -- 172.21.15.158:0/1455389611 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fc134022070 con 0x7fc13c1619a0 2023-10-16T10:02:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.366+0000 7fc140af3700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc12006b020 0x7fc12006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.367+0000 7fc140af3700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc12006b020 0x7fc12006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc130005910 tx=0x7fc130009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:13.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.371+0000 7fc1397fa700 1 -- 172.21.15.158:0/1455389611 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc13c1f7580 con 0x7fc13c1619a0 2023-10-16T10:02:13.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.611+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc13c1f7580 con 0x7fc13c1619a0 2023-10-16T10:02:13.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.611+0000 7fc1397fa700 1 -- 172.21.15.158:0/1455389611 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fc13c1f7580 con 0x7fc13c1619a0 2023-10-16T10:02:13.613 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:13.613 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/94308149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":94308149},{"type":"v1","addr":"172.21.15.172:6834","nonce":94308149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/4112063716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4112063716},{"type":"v1","addr":"172.21.15.172:6835","nonce":4112063716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8274,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/658160836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":658160836},{"type":"v1","addr":"172.21.15.158:6836","nonce":658160836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:02:09.580772+0000","modified":"2023-10-16T10:02:12.619480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8277":{"gid":8277,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/3693875115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":3693875115},{"type":"v1","addr":"172.21.15.158:6837","nonce":3693875115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T10:02:13.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.613+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc12006b020 msgr2=0x7fc12006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.613+0000 7fc142d57700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc12006b020 0x7fc12006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc130005910 tx=0x7fc130009040 comp rx=0 tx=0).stop 2023-10-16T10:02:13.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.613+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c1619a0 msgr2=0x7fc13c0b0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.613+0000 7fc142d57700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c1619a0 0x7fc13c0b0e80 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc134014040 tx=0x7fc134004ad0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 shutdown_connections 2023-10-16T10:02:13.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc12006b020 0x7fc12006d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc13c1619a0 0x7fc13c0b0e80 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc13c0ca620 0x7fc13c154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 --2- 172.21.15.158:0/1455389611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc13c160b20 0x7fc13c0b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 >> 172.21.15.158:0/1455389611 conn(0x7fc13c0cc4b0 msgr2=0x7fc13c0cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:13.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 shutdown_connections 2023-10-16T10:02:13.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.614+0000 7fc142d57700 1 -- 172.21.15.158:0/1455389611 wait complete. 2023-10-16T10:02:13.618 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 60 2023-10-16T10:02:13.627 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:02:13.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.746+0000 7fa401c7e700 1 Processor -- start 2023-10-16T10:02:13.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.746+0000 7fa401c7e700 1 -- start start 2023-10-16T10:02:13.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.746+0000 7fa401c7e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa3fc15b350 0x7fa3fc0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa401c7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc0ce630 0x7fa3fc06af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa401c7e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc0ca780 con 0x7fa3fc15b730 2023-10-16T10:02:13.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa401c7e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc06aa90 con 0x7fa3fc15b350 2023-10-16T10:02:13.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa401c7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0cf820 con 0x7fa3fc0ce630 2023-10-16T10:02:13.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa3fc15b350 0x7fa3fc0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa3fc15b350 0x7fa3fc0ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56672/0 (socket says 172.21.15.158:56672) 2023-10-16T10:02:13.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3faffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa3fc15b730 0x7fa3fc0cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56848/0 (socket says 172.21.15.158:56848) 2023-10-16T10:02:13.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 -- 172.21.15.158:0/3452860026 learned_addr learned my addr 172.21.15.158:0/3452860026 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:13.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fbfff700 1 --2- 172.21.15.158:0/3452860026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc0ce630 0x7fa3fc06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 -- 172.21.15.158:0/3452860026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc0ce630 msgr2=0x7fa3fc06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:13.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 --2- 172.21.15.158:0/3452860026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc0ce630 0x7fa3fc06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 -- 172.21.15.158:0/3452860026 >> v1:172.21.15.158:6789/0 conn(0x7fa3fc15b730 legacy=0x7fa3fc0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 -- 172.21.15.158:0/3452860026 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0cf260 con 0x7fa3fc15b350 2023-10-16T10:02:13.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fa7fc700 1 -- 172.21.15.158:0/3452860026 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659863855 0 0) 0x7fa3fc0ca780 con 0x7fa3fc15b730 2023-10-16T10:02:13.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fb7fe700 1 --2- 172.21.15.158:0/3452860026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b350 0x7fa3fc0ca080 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0098f0 tx=0x7fa3ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9861691964f2d5b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:13.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fa7fc700 1 -- 172.21.15.158:0/3452860026 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00b8a0 con 0x7fa3fc15b350 2023-10-16T10:02:13.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fa7fc700 1 -- 172.21.15.158:0/3452860026 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec00ba00 con 0x7fa3fc15b350 2023-10-16T10:02:13.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.747+0000 7fa3fa7fc700 1 -- 172.21.15.158:0/3452860026 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00bd90 con 0x7fa3fc15b350 2023-10-16T10:02:13.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 -- 172.21.15.158:0/3452860026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b350 msgr2=0x7fa3fc0ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3452860026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b350 0x7fa3fc0ca080 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0098f0 tx=0x7fa3ec009c00 comp rx=0 tx=0).stop 2023-10-16T10:02:13.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 -- 172.21.15.158:0/3452860026 shutdown_connections 2023-10-16T10:02:13.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3452860026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b350 0x7fa3fc0ca080 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3452860026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc0ce630 0x7fa3fc06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 -- 172.21.15.158:0/3452860026 >> 172.21.15.158:0/3452860026 conn(0x7fa3fc0cc4b0 msgr2=0x7fa3fc0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:13.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 -- 172.21.15.158:0/3452860026 shutdown_connections 2023-10-16T10:02:13.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.748+0000 7fa401c7e700 1 -- 172.21.15.158:0/3452860026 wait complete. 2023-10-16T10:02:13.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.749+0000 7fa401c7e700 1 Processor -- start 2023-10-16T10:02:13.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.750+0000 7fa401c7e700 1 -- start start 2023-10-16T10:02:13.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3fc0ce630 0x7fa3fc0b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc15b350 0x7fa3fc15a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b730 0x7fa3fc0c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc150560 con 0x7fa3fc15b730 2023-10-16T10:02:13.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c7a80 con 0x7fa3fc0ce630 2023-10-16T10:02:13.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c7d60 con 0x7fa3fc15b350 2023-10-16T10:02:13.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b730 0x7fa3fc0c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b730 0x7fa3fc0c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56680/0 (socket says 172.21.15.158:56680) 2023-10-16T10:02:13.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 -- 172.21.15.158:0/3207395953 learned_addr learned my addr 172.21.15.158:0/3207395953 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:13.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fb7fe700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3fc0ce630 0x7fa3fc0b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3faffd700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc15b350 0x7fa3fc15a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 -- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc15b350 msgr2=0x7fa3fc15a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc15b350 0x7fa3fc15a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 -- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3fc0ce630 msgr2=0x7fa3fc0b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3fc0ce630 0x7fa3fc0b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 -- 172.21.15.158:0/3207395953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec009550 con 0x7fa3fc15b730 2023-10-16T10:02:13.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3faffd700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc15b350 0x7fa3fc15a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:13.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3fbfff700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b730 0x7fa3fc0c8a90 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4007e60 tx=0x7fa3f400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:13.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3e3fff700 1 -- 172.21.15.158:0/3207395953 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f400ede0 con 0x7fa3fc15b730 2023-10-16T10:02:13.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3e3fff700 1 -- 172.21.15.158:0/3207395953 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4014420 con 0x7fa3fc15b730 2023-10-16T10:02:13.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc14fea0 con 0x7fa3fc15b730 2023-10-16T10:02:13.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa3e3fff700 1 -- 172.21.15.158:0/3207395953 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40147d0 con 0x7fa3fc15b730 2023-10-16T10:02:13.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.751+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc15ab20 con 0x7fa3fc15b730 2023-10-16T10:02:13.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.752+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc1506a0 con 0x7fa3fc15b730 2023-10-16T10:02:13.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.753+0000 7fa3e3fff700 1 -- 172.21.15.158:0/3207395953 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa3f4014930 con 0x7fa3fc15b730 2023-10-16T10:02:13.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.753+0000 7fa3e3fff700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fa3e806b020 0x7fa3e806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:13.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.753+0000 7fa3e3fff700 1 -- 172.21.15.158:0/3207395953 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fa3f408a090 con 0x7fa3fc15b730 2023-10-16T10:02:13.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.753+0000 7fa3fb7fe700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fa3e806b020 0x7fa3e806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:13.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.754+0000 7fa3fb7fe700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fa3e806b020 0x7fa3e806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec014040 tx=0x7fa3ec008030 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:13.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.758+0000 7fa3e3fff700 1 -- 172.21.15.158:0/3207395953 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3fc1506a0 con 0x7fa3fc15b730 2023-10-16T10:02:13.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.951+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3fc1506a0 con 0x7fa3fc15b730 2023-10-16T10:02:13.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.952+0000 7fa3e3fff700 1 -- 172.21.15.158:0/3207395953 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17373 (secure 0 0 0) 0x7fa3f40873f0 con 0x7fa3fc15b730 2023-10-16T10:02:13.954 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:13.954 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":126,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:11.184264+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:02:08.021999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T10:02:08.539662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:02:13.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fa3e806b020 msgr2=0x7fa3e806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fa3e806b020 0x7fa3e806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec014040 tx=0x7fa3ec008030 comp rx=0 tx=0).stop 2023-10-16T10:02:13.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b730 msgr2=0x7fa3fc0c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:13.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b730 0x7fa3fc0c8a90 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4007e60 tx=0x7fa3f400e3c0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 shutdown_connections 2023-10-16T10:02:13.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fa3e806b020 0x7fa3e806d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3fc15b730 0x7fa3fc0c8a90 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3fc0ce630 0x7fa3fc0b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 --2- 172.21.15.158:0/3207395953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3fc15b350 0x7fa3fc15a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:13.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 >> 172.21.15.158:0/3207395953 conn(0x7fa3fc0cc4b0 msgr2=0x7fa3fc0b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:13.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 shutdown_connections 2023-10-16T10:02:13.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:13.954+0000 7fa401c7e700 1 -- 172.21.15.158:0/3207395953 wait complete. 2023-10-16T10:02:13.968 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T10:02:13.969 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:02:14.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.088+0000 7f8c249eb700 1 Processor -- start 2023-10-16T10:02:14.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.088+0000 7f8c249eb700 1 -- start start 2023-10-16T10:02:14.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c249eb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200cc1e0 0x7f8c2015be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c249eb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8c200d0310 0x7f8c2015c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c249eb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200cfe80 con 0x7f8c201524e0 2023-10-16T10:02:14.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c249eb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2015c8a0 con 0x7f8c200d0310 2023-10-16T10:02:14.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c249eb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200cf270 con 0x7f8c200cc1e0 2023-10-16T10:02:14.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1ed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8c200d0310 0x7f8c2015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1e59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8c201524e0 0x7f8c200cbad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:60764/0 (socket says 172.21.15.158:60764) 2023-10-16T10:02:14.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200cc1e0 0x7f8c2015be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1e59c700 1 -- 172.21.15.158:0/3941246347 learned_addr learned my addr 172.21.15.158:0/3941246347 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:14.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1d59a700 1 -- 172.21.15.158:0/3941246347 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746788466 0 0) 0x7f8c200cfe80 con 0x7f8c201524e0 2023-10-16T10:02:14.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1d59a700 1 -- 172.21.15.158:0/3941246347 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c080035b0 con 0x7f8c201524e0 2023-10-16T10:02:14.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1ed9d700 1 -- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200cc1e0 msgr2=0x7f8c2015be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:14.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1ed9d700 1 --2- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200cc1e0 0x7f8c2015be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1ed9d700 1 -- 172.21.15.158:0/3941246347 >> v1:172.21.15.172:6789/0 conn(0x7f8c201524e0 legacy=0x7f8c200cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1ed9d700 1 -- 172.21.15.158:0/3941246347 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c200cf550 con 0x7f8c200d0310 2023-10-16T10:02:14.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.089+0000 7f8c1dd9b700 1 --2- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200cc1e0 0x7f8c2015be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:02:14.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c1ed9d700 1 --2- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c200d0310 0x7f8c2015c360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009d90 tx=0x7f8c1800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70b6610ce863709e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c1d59a700 1 -- 172.21.15.158:0/3941246347 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800e070 con 0x7f8c200d0310 2023-10-16T10:02:14.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c1d59a700 1 -- 172.21.15.158:0/3941246347 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1800bb20 con 0x7f8c200d0310 2023-10-16T10:02:14.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c1d59a700 1 -- 172.21.15.158:0/3941246347 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18012430 con 0x7f8c200d0310 2023-10-16T10:02:14.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c249eb700 1 -- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c200d0310 msgr2=0x7f8c2015c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c249eb700 1 --2- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c200d0310 0x7f8c2015c360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009d90 tx=0x7f8c1800b330 comp rx=0 tx=0).stop 2023-10-16T10:02:14.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c249eb700 1 -- 172.21.15.158:0/3941246347 shutdown_connections 2023-10-16T10:02:14.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c249eb700 1 --2- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c200d0310 0x7f8c2015c360 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c249eb700 1 --2- 172.21.15.158:0/3941246347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200cc1e0 0x7f8c2015be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c249eb700 1 -- 172.21.15.158:0/3941246347 >> 172.21.15.158:0/3941246347 conn(0x7f8c2006a8d0 msgr2=0x7f8c2006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:14.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.090+0000 7f8c249eb700 1 -- 172.21.15.158:0/3941246347 shutdown_connections 2023-10-16T10:02:14.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.091+0000 7f8c249eb700 1 -- 172.21.15.158:0/3941246347 wait complete. 2023-10-16T10:02:14.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.092+0000 7f8c249eb700 1 Processor -- start 2023-10-16T10:02:14.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.092+0000 7f8c249eb700 1 -- start start 2023-10-16T10:02:14.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.092+0000 7f8c249eb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c200cc1e0 0x7f8c20154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c249eb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200d0310 0x7f8c20157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c249eb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c201524e0 0x7f8c20152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c249eb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c201f1a80 con 0x7f8c200cc1e0 2023-10-16T10:02:14.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c249eb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c20156cd0 con 0x7f8c201524e0 2023-10-16T10:02:14.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c249eb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c201f0e00 con 0x7f8c200d0310 2023-10-16T10:02:14.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c200cc1e0 0x7f8c20154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c200cc1e0 0x7f8c20154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56690/0 (socket says 172.21.15.158:56690) 2023-10-16T10:02:14.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 -- 172.21.15.158:0/1305920803 learned_addr learned my addr 172.21.15.158:0/1305920803 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:14.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1ed9d700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c201524e0 0x7f8c20152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1dd9b700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200d0310 0x7f8c20157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 -- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200d0310 msgr2=0x7f8c20157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200d0310 0x7f8c20157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 -- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c201524e0 msgr2=0x7f8c20152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c201524e0 0x7f8c20152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 -- 172.21.15.158:0/1305920803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140094a0 con 0x7f8c200cc1e0 2023-10-16T10:02:14.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1ed9d700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c201524e0 0x7f8c20152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:14.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c1e59c700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c200cc1e0 0x7f8c20154ea0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000c420 tx=0x7f8c1000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c0effd700 1 -- 172.21.15.158:0/1305920803 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10005d70 con 0x7f8c200cc1e0 2023-10-16T10:02:14.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c0effd700 1 -- 172.21.15.158:0/1305920803 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c100070d0 con 0x7f8c200cc1e0 2023-10-16T10:02:14.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c18009a70 con 0x7f8c200cc1e0 2023-10-16T10:02:14.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c0effd700 1 -- 172.21.15.158:0/1305920803 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10007440 con 0x7f8c200cc1e0 2023-10-16T10:02:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.093+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c201539a0 con 0x7f8c200cc1e0 2023-10-16T10:02:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.094+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c200681b0 con 0x7f8c200cc1e0 2023-10-16T10:02:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.095+0000 7f8c0effd700 1 -- 172.21.15.158:0/1305920803 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8c100075a0 con 0x7f8c200cc1e0 2023-10-16T10:02:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.095+0000 7f8c0effd700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8c0006b020 0x7f8c0006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.095+0000 7f8c0effd700 1 -- 172.21.15.158:0/1305920803 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f8c10088e30 con 0x7f8c200cc1e0 2023-10-16T10:02:14.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.095+0000 7f8c1dd9b700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8c0006b020 0x7f8c0006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.096+0000 7f8c1dd9b700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8c0006b020 0x7f8c0006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8c201f50b0 tx=0x7f8c14008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.100+0000 7f8c0effd700 1 -- 172.21.15.158:0/1305920803 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c10053af0 con 0x7f8c200cc1e0 2023-10-16T10:02:14.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.294+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 --> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c200cfb10 con 0x7f8c0006b020 2023-10-16T10:02:14.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.302+0000 7f8c0effd700 1 -- 172.21.15.158:0/1305920803 <== mgr.8022 v2:172.21.15.158:6832/1080332904 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289996 (secure 0 0 0) 0x7f8c140107c0 con 0x7f8c0006b020 2023-10-16T10:02:14.307 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:14.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8c0006b020 msgr2=0x7f8c0006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8c0006b020 0x7f8c0006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8c201f50b0 tx=0x7f8c14008040 comp rx=0 tx=0).stop 2023-10-16T10:02:14.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c200cc1e0 msgr2=0x7f8c20154ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c200cc1e0 0x7f8c20154ea0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000c420 tx=0x7f8c1000c7e0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 shutdown_connections 2023-10-16T10:02:14.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c201524e0 0x7f8c20152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f8c0006b020 0x7f8c0006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c200cc1e0 0x7f8c20154ea0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 --2- 172.21.15.158:0/1305920803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c200d0310 0x7f8c20157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 >> 172.21.15.158:0/1305920803 conn(0x7f8c2006a8d0 msgr2=0x7f8c200b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:14.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 shutdown_connections 2023-10-16T10:02:14.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.306+0000 7f8c249eb700 1 -- 172.21.15.158:0/1305920803 wait complete. 2023-10-16T10:02:14.312 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:02:14.319 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-16T10:02:14.137380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":346120,"kb_used_data":13380,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754628600,"statfs":{"total":773094113280,"available":772739686400,"internally_reserved":0,"allocated":13701120,"data_stored":7677480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":340577847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.965144"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352538+0000","last_change":"2023-10-16T10:02:10.196386+0000","last_active":"2023-10-16T10:02:11.352538+0000","last_peered":"2023-10-16T10:02:11.352538+0000","last_clean":"2023-10-16T10:02:11.352538+0000","last_became_active":"2023-10-16T10:02:10.195959+0000","last_became_peered":"2023-10-16T10:02:10.195959+0000","last_unstale":"2023-10-16T10:02:11.352538+0000","last_undegraded":"2023-10-16T10:02:11.352538+0000","last_fullsized":"2023-10-16T10:02:11.352538+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:44.278653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193035+0000","last_change":"2023-10-16T10:02:10.206257+0000","last_active":"2023-10-16T10:02:11.193035+0000","last_peered":"2023-10-16T10:02:11.193035+0000","last_clean":"2023-10-16T10:02:11.193035+0000","last_became_active":"2023-10-16T10:02:10.205900+0000","last_became_peered":"2023-10-16T10:02:10.205900+0000","last_unstale":"2023-10-16T10:02:11.193035+0000","last_undegraded":"2023-10-16T10:02:11.193035+0000","last_fullsized":"2023-10-16T10:02:11.193035+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:37.304152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351844+0000","last_change":"2023-10-16T10:02:10.211560+0000","last_active":"2023-10-16T10:02:11.351844+0000","last_peered":"2023-10-16T10:02:11.351844+0000","last_clean":"2023-10-16T10:02:11.351844+0000","last_became_active":"2023-10-16T10:02:10.211244+0000","last_became_peered":"2023-10-16T10:02:10.211244+0000","last_unstale":"2023-10-16T10:02:11.351844+0000","last_undegraded":"2023-10-16T10:02:11.351844+0000","last_fullsized":"2023-10-16T10:02:11.351844+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:01.574752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353099+0000","last_change":"2023-10-16T10:02:10.197281+0000","last_active":"2023-10-16T10:02:11.353099+0000","last_peered":"2023-10-16T10:02:11.353099+0000","last_clean":"2023-10-16T10:02:11.353099+0000","last_became_active":"2023-10-16T10:02:10.197020+0000","last_became_peered":"2023-10-16T10:02:10.197020+0000","last_unstale":"2023-10-16T10:02:11.353099+0000","last_undegraded":"2023-10-16T10:02:11.353099+0000","last_fullsized":"2023-10-16T10:02:11.353099+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:20.124547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353470+0000","last_change":"2023-10-16T10:02:10.200305+0000","last_active":"2023-10-16T10:02:11.353470+0000","last_peered":"2023-10-16T10:02:11.353470+0000","last_clean":"2023-10-16T10:02:11.353470+0000","last_became_active":"2023-10-16T10:02:10.198547+0000","last_became_peered":"2023-10-16T10:02:10.198547+0000","last_unstale":"2023-10-16T10:02:11.353470+0000","last_undegraded":"2023-10-16T10:02:11.353470+0000","last_fullsized":"2023-10-16T10:02:11.353470+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:42:02.838978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192660+0000","last_change":"2023-10-16T10:02:10.208604+0000","last_active":"2023-10-16T10:02:11.192660+0000","last_peered":"2023-10-16T10:02:11.192660+0000","last_clean":"2023-10-16T10:02:11.192660+0000","last_became_active":"2023-10-16T10:02:10.208175+0000","last_became_peered":"2023-10-16T10:02:10.208175+0000","last_unstale":"2023-10-16T10:02:11.192660+0000","last_undegraded":"2023-10-16T10:02:11.192660+0000","last_fullsized":"2023-10-16T10:02:11.192660+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:24.573022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192538+0000","last_change":"2023-10-16T10:02:10.206361+0000","last_active":"2023-10-16T10:02:11.192538+0000","last_peered":"2023-10-16T10:02:11.192538+0000","last_clean":"2023-10-16T10:02:11.192538+0000","last_became_active":"2023-10-16T10:02:10.205750+0000","last_became_peered":"2023-10-16T10:02:10.205750+0000","last_unstale":"2023-10-16T10:02:11.192538+0000","last_undegraded":"2023-10-16T10:02:11.192538+0000","last_fullsized":"2023-10-16T10:02:11.192538+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:44.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352284+0000","last_change":"2023-10-16T10:02:10.203015+0000","last_active":"2023-10-16T10:02:11.352284+0000","last_peered":"2023-10-16T10:02:11.352284+0000","last_clean":"2023-10-16T10:02:11.352284+0000","last_became_active":"2023-10-16T10:02:10.202724+0000","last_became_peered":"2023-10-16T10:02:10.202724+0000","last_unstale":"2023-10-16T10:02:11.352284+0000","last_undegraded":"2023-10-16T10:02:11.352284+0000","last_fullsized":"2023-10-16T10:02:11.352284+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:37.507977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193050+0000","last_change":"2023-10-16T10:02:10.200246+0000","last_active":"2023-10-16T10:02:11.193050+0000","last_peered":"2023-10-16T10:02:11.193050+0000","last_clean":"2023-10-16T10:02:11.193050+0000","last_became_active":"2023-10-16T10:02:10.199979+0000","last_became_peered":"2023-10-16T10:02:10.199979+0000","last_unstale":"2023-10-16T10:02:11.193050+0000","last_undegraded":"2023-10-16T10:02:11.193050+0000","last_fullsized":"2023-10-16T10:02:11.193050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.235661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353139+0000","last_change":"2023-10-16T10:02:10.207624+0000","last_active":"2023-10-16T10:02:11.353139+0000","last_peered":"2023-10-16T10:02:11.353139+0000","last_clean":"2023-10-16T10:02:11.353139+0000","last_became_active":"2023-10-16T10:02:10.207321+0000","last_became_peered":"2023-10-16T10:02:10.207321+0000","last_unstale":"2023-10-16T10:02:11.353139+0000","last_undegraded":"2023-10-16T10:02:11.353139+0000","last_fullsized":"2023-10-16T10:02:11.353139+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352124+0000","last_change":"2023-10-16T10:02:10.208921+0000","last_active":"2023-10-16T10:02:11.352124+0000","last_peered":"2023-10-16T10:02:11.352124+0000","last_clean":"2023-10-16T10:02:11.352124+0000","last_became_active":"2023-10-16T10:02:10.208674+0000","last_became_peered":"2023-10-16T10:02:10.208674+0000","last_unstale":"2023-10-16T10:02:11.352124+0000","last_undegraded":"2023-10-16T10:02:11.352124+0000","last_fullsized":"2023-10-16T10:02:11.352124+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:30.773531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193753+0000","last_change":"2023-10-16T10:02:10.197261+0000","last_active":"2023-10-16T10:02:11.193753+0000","last_peered":"2023-10-16T10:02:11.193753+0000","last_clean":"2023-10-16T10:02:11.193753+0000","last_became_active":"2023-10-16T10:02:10.196997+0000","last_became_peered":"2023-10-16T10:02:10.196997+0000","last_unstale":"2023-10-16T10:02:11.193753+0000","last_undegraded":"2023-10-16T10:02:11.193753+0000","last_fullsized":"2023-10-16T10:02:11.193753+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:42:02.838978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352681+0000","last_change":"2023-10-16T10:02:10.207066+0000","last_active":"2023-10-16T10:02:11.352681+0000","last_peered":"2023-10-16T10:02:11.352681+0000","last_clean":"2023-10-16T10:02:11.352681+0000","last_became_active":"2023-10-16T10:02:10.206591+0000","last_became_peered":"2023-10-16T10:02:10.206591+0000","last_unstale":"2023-10-16T10:02:11.352681+0000","last_undegraded":"2023-10-16T10:02:11.352681+0000","last_fullsized":"2023-10-16T10:02:11.352681+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:56:22.099166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191800+0000","last_change":"2023-10-16T10:02:10.197986+0000","last_active":"2023-10-16T10:02:11.191800+0000","last_peered":"2023-10-16T10:02:11.191800+0000","last_clean":"2023-10-16T10:02:11.191800+0000","last_became_active":"2023-10-16T10:02:10.197744+0000","last_became_peered":"2023-10-16T10:02:10.197744+0000","last_unstale":"2023-10-16T10:02:11.191800+0000","last_undegraded":"2023-10-16T10:02:11.191800+0000","last_fullsized":"2023-10-16T10:02:11.191800+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:30.773531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191641+0000","last_change":"2023-10-16T10:02:10.203745+0000","last_active":"2023-10-16T10:02:11.191641+0000","last_peered":"2023-10-16T10:02:11.191641+0000","last_clean":"2023-10-16T10:02:11.191641+0000","last_became_active":"2023-10-16T10:02:10.203467+0000","last_became_peered":"2023-10-16T10:02:10.203467+0000","last_unstale":"2023-10-16T10:02:11.191641+0000","last_undegraded":"2023-10-16T10:02:11.191641+0000","last_fullsized":"2023-10-16T10:02:11.191641+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:56:22.099166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191511+0000","last_change":"2023-10-16T10:02:10.195395+0000","last_active":"2023-10-16T10:02:11.191511+0000","last_peered":"2023-10-16T10:02:11.191511+0000","last_clean":"2023-10-16T10:02:11.191511+0000","last_became_active":"2023-10-16T10:02:10.195117+0000","last_became_peered":"2023-10-16T10:02:10.195117+0000","last_unstale":"2023-10-16T10:02:11.191511+0000","last_undegraded":"2023-10-16T10:02:11.191511+0000","last_fullsized":"2023-10-16T10:02:11.191511+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:22.363722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352554+0000","last_change":"2023-10-16T10:02:10.204013+0000","last_active":"2023-10-16T10:02:11.352554+0000","last_peered":"2023-10-16T10:02:11.352554+0000","last_clean":"2023-10-16T10:02:11.352554+0000","last_became_active":"2023-10-16T10:02:10.203735+0000","last_became_peered":"2023-10-16T10:02:10.203735+0000","last_unstale":"2023-10-16T10:02:11.352554+0000","last_undegraded":"2023-10-16T10:02:11.352554+0000","last_fullsized":"2023-10-16T10:02:11.352554+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:43.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193038+0000","last_change":"2023-10-16T10:02:10.202592+0000","last_active":"2023-10-16T10:02:11.193038+0000","last_peered":"2023-10-16T10:02:11.193038+0000","last_clean":"2023-10-16T10:02:11.193038+0000","last_became_active":"2023-10-16T10:02:10.202314+0000","last_became_peered":"2023-10-16T10:02:10.202314+0000","last_unstale":"2023-10-16T10:02:11.193038+0000","last_undegraded":"2023-10-16T10:02:11.193038+0000","last_fullsized":"2023-10-16T10:02:11.193038+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352517+0000","last_change":"2023-10-16T10:02:10.206325+0000","last_active":"2023-10-16T10:02:11.352517+0000","last_peered":"2023-10-16T10:02:11.352517+0000","last_clean":"2023-10-16T10:02:11.352517+0000","last_became_active":"2023-10-16T10:02:10.205812+0000","last_became_peered":"2023-10-16T10:02:10.205812+0000","last_unstale":"2023-10-16T10:02:11.352517+0000","last_undegraded":"2023-10-16T10:02:11.352517+0000","last_fullsized":"2023-10-16T10:02:11.352517+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:22:04.943383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193764+0000","last_change":"2023-10-16T10:02:10.193072+0000","last_active":"2023-10-16T10:02:11.193764+0000","last_peered":"2023-10-16T10:02:11.193764+0000","last_clean":"2023-10-16T10:02:11.193764+0000","last_became_active":"2023-10-16T10:02:10.192797+0000","last_became_peered":"2023-10-16T10:02:10.192797+0000","last_unstale":"2023-10-16T10:02:11.193764+0000","last_undegraded":"2023-10-16T10:02:11.193764+0000","last_fullsized":"2023-10-16T10:02:11.193764+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:44.425674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352294+0000","last_change":"2023-10-16T10:02:10.202285+0000","last_active":"2023-10-16T10:02:11.352294+0000","last_peered":"2023-10-16T10:02:11.352294+0000","last_clean":"2023-10-16T10:02:11.352294+0000","last_became_active":"2023-10-16T10:02:10.202004+0000","last_became_peered":"2023-10-16T10:02:10.202004+0000","last_unstale":"2023-10-16T10:02:11.352294+0000","last_undegraded":"2023-10-16T10:02:11.352294+0000","last_fullsized":"2023-10-16T10:02:11.352294+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:58.985349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353359+0000","last_change":"2023-10-16T10:02:10.208486+0000","last_active":"2023-10-16T10:02:11.353359+0000","last_peered":"2023-10-16T10:02:11.353359+0000","last_clean":"2023-10-16T10:02:11.353359+0000","last_became_active":"2023-10-16T10:02:10.208225+0000","last_became_peered":"2023-10-16T10:02:10.208225+0000","last_unstale":"2023-10-16T10:02:11.353359+0000","last_undegraded":"2023-10-16T10:02:11.353359+0000","last_fullsized":"2023-10-16T10:02:11.353359+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.587855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351826+0000","last_change":"2023-10-16T10:02:10.203660+0000","last_active":"2023-10-16T10:02:11.351826+0000","last_peered":"2023-10-16T10:02:11.351826+0000","last_clean":"2023-10-16T10:02:11.351826+0000","last_became_active":"2023-10-16T10:02:10.203383+0000","last_became_peered":"2023-10-16T10:02:10.203383+0000","last_unstale":"2023-10-16T10:02:11.351826+0000","last_undegraded":"2023-10-16T10:02:11.351826+0000","last_fullsized":"2023-10-16T10:02:11.351826+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:42.615762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353178+0000","last_change":"2023-10-16T10:02:10.200856+0000","last_active":"2023-10-16T10:02:11.353178+0000","last_peered":"2023-10-16T10:02:11.353178+0000","last_clean":"2023-10-16T10:02:11.353178+0000","last_became_active":"2023-10-16T10:02:10.198790+0000","last_became_peered":"2023-10-16T10:02:10.198790+0000","last_unstale":"2023-10-16T10:02:11.353178+0000","last_undegraded":"2023-10-16T10:02:11.353178+0000","last_fullsized":"2023-10-16T10:02:11.353178+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.235661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193651+0000","last_change":"2023-10-16T10:02:10.205971+0000","last_active":"2023-10-16T10:02:11.193651+0000","last_peered":"2023-10-16T10:02:11.193651+0000","last_clean":"2023-10-16T10:02:11.193651+0000","last_became_active":"2023-10-16T10:02:10.204772+0000","last_became_peered":"2023-10-16T10:02:10.204772+0000","last_unstale":"2023-10-16T10:02:11.193651+0000","last_undegraded":"2023-10-16T10:02:11.193651+0000","last_fullsized":"2023-10-16T10:02:11.193651+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.587855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193761+0000","last_change":"2023-10-16T10:02:10.210279+0000","last_active":"2023-10-16T10:02:11.193761+0000","last_peered":"2023-10-16T10:02:11.193761+0000","last_clean":"2023-10-16T10:02:11.193761+0000","last_became_active":"2023-10-16T10:02:10.209956+0000","last_became_peered":"2023-10-16T10:02:10.209956+0000","last_unstale":"2023-10-16T10:02:11.193761+0000","last_undegraded":"2023-10-16T10:02:11.193761+0000","last_fullsized":"2023-10-16T10:02:11.193761+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:24.234939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352244+0000","last_change":"2023-10-16T10:02:10.206875+0000","last_active":"2023-10-16T10:02:11.352244+0000","last_peered":"2023-10-16T10:02:11.352244+0000","last_clean":"2023-10-16T10:02:11.352244+0000","last_became_active":"2023-10-16T10:02:10.206594+0000","last_became_peered":"2023-10-16T10:02:10.206594+0000","last_unstale":"2023-10-16T10:02:11.352244+0000","last_undegraded":"2023-10-16T10:02:11.352244+0000","last_fullsized":"2023-10-16T10:02:11.352244+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:36.493038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192344+0000","last_change":"2023-10-16T10:02:10.208786+0000","last_active":"2023-10-16T10:02:11.192344+0000","last_peered":"2023-10-16T10:02:11.192344+0000","last_clean":"2023-10-16T10:02:11.192344+0000","last_became_active":"2023-10-16T10:02:10.208572+0000","last_became_peered":"2023-10-16T10:02:10.208572+0000","last_unstale":"2023-10-16T10:02:11.192344+0000","last_undegraded":"2023-10-16T10:02:11.192344+0000","last_fullsized":"2023-10-16T10:02:11.192344+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:19.739611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352226+0000","last_change":"2023-10-16T10:02:10.200008+0000","last_active":"2023-10-16T10:02:11.352226+0000","last_peered":"2023-10-16T10:02:11.352226+0000","last_clean":"2023-10-16T10:02:11.352226+0000","last_became_active":"2023-10-16T10:02:10.199736+0000","last_became_peered":"2023-10-16T10:02:10.199736+0000","last_unstale":"2023-10-16T10:02:11.352226+0000","last_undegraded":"2023-10-16T10:02:11.352226+0000","last_fullsized":"2023-10-16T10:02:11.352226+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:15:55.364671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193841+0000","last_change":"2023-10-16T10:02:10.205048+0000","last_active":"2023-10-16T10:02:11.193841+0000","last_peered":"2023-10-16T10:02:11.193841+0000","last_clean":"2023-10-16T10:02:11.193841+0000","last_became_active":"2023-10-16T10:02:10.204731+0000","last_became_peered":"2023-10-16T10:02:10.204731+0000","last_unstale":"2023-10-16T10:02:11.193841+0000","last_undegraded":"2023-10-16T10:02:11.193841+0000","last_fullsized":"2023-10-16T10:02:11.193841+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:22:42.003604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192495+0000","last_change":"2023-10-16T10:02:10.199486+0000","last_active":"2023-10-16T10:02:11.192495+0000","last_peered":"2023-10-16T10:02:11.192495+0000","last_clean":"2023-10-16T10:02:11.192495+0000","last_became_active":"2023-10-16T10:02:10.199160+0000","last_became_peered":"2023-10-16T10:02:10.199160+0000","last_unstale":"2023-10-16T10:02:11.192495+0000","last_undegraded":"2023-10-16T10:02:11.192495+0000","last_fullsized":"2023-10-16T10:02:11.192495+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:52.749649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352359+0000","last_change":"2023-10-16T10:02:10.193839+0000","last_active":"2023-10-16T10:02:11.352359+0000","last_peered":"2023-10-16T10:02:11.352359+0000","last_clean":"2023-10-16T10:02:11.352359+0000","last_became_active":"2023-10-16T10:02:10.193445+0000","last_became_peered":"2023-10-16T10:02:10.193445+0000","last_unstale":"2023-10-16T10:02:11.352359+0000","last_undegraded":"2023-10-16T10:02:11.352359+0000","last_fullsized":"2023-10-16T10:02:11.352359+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:34:29.465706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352071+0000","last_change":"2023-10-16T10:02:10.207878+0000","last_active":"2023-10-16T10:02:11.352071+0000","last_peered":"2023-10-16T10:02:11.352071+0000","last_clean":"2023-10-16T10:02:11.352071+0000","last_became_active":"2023-10-16T10:02:10.206185+0000","last_became_peered":"2023-10-16T10:02:10.206185+0000","last_unstale":"2023-10-16T10:02:11.352071+0000","last_undegraded":"2023-10-16T10:02:11.352071+0000","last_fullsized":"2023-10-16T10:02:11.352071+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:07.878862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"125'4","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352903+0000","last_change":"2023-10-16T10:02:09.196805+0000","last_active":"2023-10-16T10:02:11.352903+0000","last_peered":"2023-10-16T10:02:11.352903+0000","last_clean":"2023-10-16T10:02:11.352903+0000","last_became_active":"2023-10-16T10:02:09.196548+0000","last_became_peered":"2023-10-16T10:02:09.196548+0000","last_unstale":"2023-10-16T10:02:11.352903+0000","last_undegraded":"2023-10-16T10:02:11.352903+0000","last_fullsized":"2023-10-16T10:02:11.352903+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:35.363470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352215+0000","last_change":"2023-10-16T10:02:10.202490+0000","last_active":"2023-10-16T10:02:11.352215+0000","last_peered":"2023-10-16T10:02:11.352215+0000","last_clean":"2023-10-16T10:02:11.352215+0000","last_became_active":"2023-10-16T10:02:10.202234+0000","last_became_peered":"2023-10-16T10:02:10.202234+0000","last_unstale":"2023-10-16T10:02:11.352215+0000","last_undegraded":"2023-10-16T10:02:11.352215+0000","last_fullsized":"2023-10-16T10:02:11.352215+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:13.534183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353059+0000","last_change":"2023-10-16T10:02:09.190703+0000","last_active":"2023-10-16T10:02:11.353059+0000","last_peered":"2023-10-16T10:02:11.353059+0000","last_clean":"2023-10-16T10:02:11.353059+0000","last_became_active":"2023-10-16T10:02:09.190435+0000","last_became_peered":"2023-10-16T10:02:09.190435+0000","last_unstale":"2023-10-16T10:02:11.353059+0000","last_undegraded":"2023-10-16T10:02:11.353059+0000","last_fullsized":"2023-10-16T10:02:11.353059+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:42.191292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193581+0000","last_change":"2023-10-16T10:02:10.201930+0000","last_active":"2023-10-16T10:02:11.193581+0000","last_peered":"2023-10-16T10:02:11.193581+0000","last_clean":"2023-10-16T10:02:11.193581+0000","last_became_active":"2023-10-16T10:02:10.201619+0000","last_became_peered":"2023-10-16T10:02:10.201619+0000","last_unstale":"2023-10-16T10:02:11.193581+0000","last_undegraded":"2023-10-16T10:02:11.193581+0000","last_fullsized":"2023-10-16T10:02:11.193581+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.235661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193528+0000","last_change":"2023-10-16T10:02:09.191920+0000","last_active":"2023-10-16T10:02:11.193528+0000","last_peered":"2023-10-16T10:02:11.193528+0000","last_clean":"2023-10-16T10:02:11.193528+0000","last_became_active":"2023-10-16T10:02:09.191606+0000","last_became_peered":"2023-10-16T10:02:09.191606+0000","last_unstale":"2023-10-16T10:02:11.193528+0000","last_undegraded":"2023-10-16T10:02:11.193528+0000","last_fullsized":"2023-10-16T10:02:11.193528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:13.037875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351929+0000","last_change":"2023-10-16T10:02:10.200643+0000","last_active":"2023-10-16T10:02:11.351929+0000","last_peered":"2023-10-16T10:02:11.351929+0000","last_clean":"2023-10-16T10:02:11.351929+0000","last_became_active":"2023-10-16T10:02:10.200326+0000","last_became_peered":"2023-10-16T10:02:10.200326+0000","last_unstale":"2023-10-16T10:02:11.351929+0000","last_undegraded":"2023-10-16T10:02:11.351929+0000","last_fullsized":"2023-10-16T10:02:11.351929+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:56:22.099166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352862+0000","last_change":"2023-10-16T10:02:09.191107+0000","last_active":"2023-10-16T10:02:11.352862+0000","last_peered":"2023-10-16T10:02:11.352862+0000","last_clean":"2023-10-16T10:02:11.352862+0000","last_became_active":"2023-10-16T10:02:09.190681+0000","last_became_peered":"2023-10-16T10:02:09.190681+0000","last_unstale":"2023-10-16T10:02:11.352862+0000","last_undegraded":"2023-10-16T10:02:11.352862+0000","last_fullsized":"2023-10-16T10:02:11.352862+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:05.397655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353173+0000","last_change":"2023-10-16T10:02:10.205939+0000","last_active":"2023-10-16T10:02:11.353173+0000","last_peered":"2023-10-16T10:02:11.353173+0000","last_clean":"2023-10-16T10:02:11.353173+0000","last_became_active":"2023-10-16T10:02:10.205656+0000","last_became_peered":"2023-10-16T10:02:10.205656+0000","last_unstale":"2023-10-16T10:02:11.353173+0000","last_undegraded":"2023-10-16T10:02:11.353173+0000","last_fullsized":"2023-10-16T10:02:11.353173+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:40.039367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191741+0000","last_change":"2023-10-16T10:02:09.193730+0000","last_active":"2023-10-16T10:02:11.191741+0000","last_peered":"2023-10-16T10:02:11.191741+0000","last_clean":"2023-10-16T10:02:11.191741+0000","last_became_active":"2023-10-16T10:02:09.193409+0000","last_became_peered":"2023-10-16T10:02:09.193409+0000","last_unstale":"2023-10-16T10:02:11.191741+0000","last_undegraded":"2023-10-16T10:02:11.191741+0000","last_fullsized":"2023-10-16T10:02:11.191741+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:57.060618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351977+0000","last_change":"2023-10-16T10:02:10.210109+0000","last_active":"2023-10-16T10:02:11.351977+0000","last_peered":"2023-10-16T10:02:11.351977+0000","last_clean":"2023-10-16T10:02:11.351977+0000","last_became_active":"2023-10-16T10:02:10.209805+0000","last_became_peered":"2023-10-16T10:02:10.209805+0000","last_unstale":"2023-10-16T10:02:11.351977+0000","last_undegraded":"2023-10-16T10:02:11.351977+0000","last_fullsized":"2023-10-16T10:02:11.351977+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:40.142426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351830+0000","last_change":"2023-10-16T10:02:09.190793+0000","last_active":"2023-10-16T10:02:11.351830+0000","last_peered":"2023-10-16T10:02:11.351830+0000","last_clean":"2023-10-16T10:02:11.351830+0000","last_became_active":"2023-10-16T10:02:09.190531+0000","last_became_peered":"2023-10-16T10:02:09.190531+0000","last_unstale":"2023-10-16T10:02:11.351830+0000","last_undegraded":"2023-10-16T10:02:11.351830+0000","last_fullsized":"2023-10-16T10:02:11.351830+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:53.236384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193206+0000","last_change":"2023-10-16T10:02:10.206010+0000","last_active":"2023-10-16T10:02:11.193206+0000","last_peered":"2023-10-16T10:02:11.193206+0000","last_clean":"2023-10-16T10:02:11.193206+0000","last_became_active":"2023-10-16T10:02:10.205533+0000","last_became_peered":"2023-10-16T10:02:10.205533+0000","last_unstale":"2023-10-16T10:02:11.193206+0000","last_undegraded":"2023-10-16T10:02:11.193206+0000","last_fullsized":"2023-10-16T10:02:11.193206+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.587855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193431+0000","last_change":"2023-10-16T10:02:09.192785+0000","last_active":"2023-10-16T10:02:11.193431+0000","last_peered":"2023-10-16T10:02:11.193431+0000","last_clean":"2023-10-16T10:02:11.193431+0000","last_became_active":"2023-10-16T10:02:09.192367+0000","last_became_peered":"2023-10-16T10:02:09.192367+0000","last_unstale":"2023-10-16T10:02:11.193431+0000","last_undegraded":"2023-10-16T10:02:11.193431+0000","last_fullsized":"2023-10-16T10:02:11.193431+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:51.745363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352902+0000","last_change":"2023-10-16T10:02:10.193903+0000","last_active":"2023-10-16T10:02:11.352902+0000","last_peered":"2023-10-16T10:02:11.352902+0000","last_clean":"2023-10-16T10:02:11.352902+0000","last_became_active":"2023-10-16T10:02:10.193626+0000","last_became_peered":"2023-10-16T10:02:10.193626+0000","last_unstale":"2023-10-16T10:02:11.352902+0000","last_undegraded":"2023-10-16T10:02:11.352902+0000","last_fullsized":"2023-10-16T10:02:11.352902+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:36.493038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352647+0000","last_change":"2023-10-16T10:02:09.195512+0000","last_active":"2023-10-16T10:02:11.352647+0000","last_peered":"2023-10-16T10:02:11.352647+0000","last_clean":"2023-10-16T10:02:11.352647+0000","last_became_active":"2023-10-16T10:02:09.195327+0000","last_became_peered":"2023-10-16T10:02:09.195327+0000","last_unstale":"2023-10-16T10:02:11.352647+0000","last_undegraded":"2023-10-16T10:02:11.352647+0000","last_fullsized":"2023-10-16T10:02:11.352647+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:37:19.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351765+0000","last_change":"2023-10-16T10:02:10.206240+0000","last_active":"2023-10-16T10:02:11.351765+0000","last_peered":"2023-10-16T10:02:11.351765+0000","last_clean":"2023-10-16T10:02:11.351765+0000","last_became_active":"2023-10-16T10:02:10.205981+0000","last_became_peered":"2023-10-16T10:02:10.205981+0000","last_unstale":"2023-10-16T10:02:11.351765+0000","last_undegraded":"2023-10-16T10:02:11.351765+0000","last_fullsized":"2023-10-16T10:02:11.351765+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:22.643343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193734+0000","last_change":"2023-10-16T10:02:09.191695+0000","last_active":"2023-10-16T10:02:11.193734+0000","last_peered":"2023-10-16T10:02:11.193734+0000","last_clean":"2023-10-16T10:02:11.193734+0000","last_became_active":"2023-10-16T10:02:09.191321+0000","last_became_peered":"2023-10-16T10:02:09.191321+0000","last_unstale":"2023-10-16T10:02:11.193734+0000","last_undegraded":"2023-10-16T10:02:11.193734+0000","last_fullsized":"2023-10-16T10:02:11.193734+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:05.397655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352396+0000","last_change":"2023-10-16T10:02:10.200463+0000","last_active":"2023-10-16T10:02:11.352396+0000","last_peered":"2023-10-16T10:02:11.352396+0000","last_clean":"2023-10-16T10:02:11.352396+0000","last_became_active":"2023-10-16T10:02:10.200171+0000","last_became_peered":"2023-10-16T10:02:10.200171+0000","last_unstale":"2023-10-16T10:02:11.352396+0000","last_undegraded":"2023-10-16T10:02:11.352396+0000","last_fullsized":"2023-10-16T10:02:11.352396+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:24.573022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351797+0000","last_change":"2023-10-16T10:02:09.192472+0000","last_active":"2023-10-16T10:02:11.351797+0000","last_peered":"2023-10-16T10:02:11.351797+0000","last_clean":"2023-10-16T10:02:11.351797+0000","last_became_active":"2023-10-16T10:02:09.192223+0000","last_became_peered":"2023-10-16T10:02:09.192223+0000","last_unstale":"2023-10-16T10:02:11.351797+0000","last_undegraded":"2023-10-16T10:02:11.351797+0000","last_fullsized":"2023-10-16T10:02:11.351797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:17.532553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352066+0000","last_change":"2023-10-16T10:02:10.207736+0000","last_active":"2023-10-16T10:02:11.352066+0000","last_peered":"2023-10-16T10:02:11.352066+0000","last_clean":"2023-10-16T10:02:11.352066+0000","last_became_active":"2023-10-16T10:02:10.207459+0000","last_became_peered":"2023-10-16T10:02:10.207459+0000","last_unstale":"2023-10-16T10:02:11.352066+0000","last_undegraded":"2023-10-16T10:02:11.352066+0000","last_fullsized":"2023-10-16T10:02:11.352066+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:11.005434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"125'3","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191845+0000","last_change":"2023-10-16T10:02:09.184622+0000","last_active":"2023-10-16T10:02:11.191845+0000","last_peered":"2023-10-16T10:02:11.191845+0000","last_clean":"2023-10-16T10:02:11.191845+0000","last_became_active":"2023-10-16T10:02:09.184348+0000","last_became_peered":"2023-10-16T10:02:09.184348+0000","last_unstale":"2023-10-16T10:02:11.191845+0000","last_undegraded":"2023-10-16T10:02:11.191845+0000","last_fullsized":"2023-10-16T10:02:11.191845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:27:10.568486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352988+0000","last_change":"2023-10-16T10:02:10.198334+0000","last_active":"2023-10-16T10:02:11.352988+0000","last_peered":"2023-10-16T10:02:11.352988+0000","last_clean":"2023-10-16T10:02:11.352988+0000","last_became_active":"2023-10-16T10:02:10.197929+0000","last_became_peered":"2023-10-16T10:02:10.197929+0000","last_unstale":"2023-10-16T10:02:11.352988+0000","last_undegraded":"2023-10-16T10:02:11.352988+0000","last_fullsized":"2023-10-16T10:02:11.352988+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:37.304152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"125'6","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192456+0000","last_change":"2023-10-16T10:02:09.193617+0000","last_active":"2023-10-16T10:02:11.192456+0000","last_peered":"2023-10-16T10:02:11.192456+0000","last_clean":"2023-10-16T10:02:11.192456+0000","last_became_active":"2023-10-16T10:02:09.193144+0000","last_became_peered":"2023-10-16T10:02:09.193144+0000","last_unstale":"2023-10-16T10:02:11.192456+0000","last_undegraded":"2023-10-16T10:02:11.192456+0000","last_fullsized":"2023-10-16T10:02:11.192456+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:29.614976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351898+0000","last_change":"2023-10-16T10:02:10.206952+0000","last_active":"2023-10-16T10:02:11.351898+0000","last_peered":"2023-10-16T10:02:11.351898+0000","last_clean":"2023-10-16T10:02:11.351898+0000","last_became_active":"2023-10-16T10:02:10.206612+0000","last_became_peered":"2023-10-16T10:02:10.206612+0000","last_unstale":"2023-10-16T10:02:11.351898+0000","last_undegraded":"2023-10-16T10:02:11.351898+0000","last_fullsized":"2023-10-16T10:02:11.351898+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:22.363722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353088+0000","last_change":"2023-10-16T10:02:09.194325+0000","last_active":"2023-10-16T10:02:11.353088+0000","last_peered":"2023-10-16T10:02:11.353088+0000","last_clean":"2023-10-16T10:02:11.353088+0000","last_became_active":"2023-10-16T10:02:09.193600+0000","last_became_peered":"2023-10-16T10:02:09.193600+0000","last_unstale":"2023-10-16T10:02:11.353088+0000","last_undegraded":"2023-10-16T10:02:11.353088+0000","last_fullsized":"2023-10-16T10:02:11.353088+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:17:15.859650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192787+0000","last_change":"2023-10-16T10:02:10.206644+0000","last_active":"2023-10-16T10:02:11.192787+0000","last_peered":"2023-10-16T10:02:11.192787+0000","last_clean":"2023-10-16T10:02:11.192787+0000","last_became_active":"2023-10-16T10:02:10.205464+0000","last_became_peered":"2023-10-16T10:02:10.205464+0000","last_unstale":"2023-10-16T10:02:11.192787+0000","last_undegraded":"2023-10-16T10:02:11.192787+0000","last_fullsized":"2023-10-16T10:02:11.192787+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.210215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352881+0000","last_change":"2023-10-16T10:02:09.193291+0000","last_active":"2023-10-16T10:02:11.352881+0000","last_peered":"2023-10-16T10:02:11.352881+0000","last_clean":"2023-10-16T10:02:11.352881+0000","last_became_active":"2023-10-16T10:02:09.192984+0000","last_became_peered":"2023-10-16T10:02:09.192984+0000","last_unstale":"2023-10-16T10:02:11.352881+0000","last_undegraded":"2023-10-16T10:02:11.352881+0000","last_fullsized":"2023-10-16T10:02:11.352881+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:41.612597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351904+0000","last_change":"2023-10-16T10:02:10.193385+0000","last_active":"2023-10-16T10:02:11.351904+0000","last_peered":"2023-10-16T10:02:11.351904+0000","last_clean":"2023-10-16T10:02:11.351904+0000","last_became_active":"2023-10-16T10:02:10.193114+0000","last_became_peered":"2023-10-16T10:02:10.193114+0000","last_unstale":"2023-10-16T10:02:11.351904+0000","last_undegraded":"2023-10-16T10:02:11.351904+0000","last_fullsized":"2023-10-16T10:02:11.351904+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:10.335349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352187+0000","last_change":"2023-10-16T10:02:09.191473+0000","last_active":"2023-10-16T10:02:11.352187+0000","last_peered":"2023-10-16T10:02:11.352187+0000","last_clean":"2023-10-16T10:02:11.352187+0000","last_became_active":"2023-10-16T10:02:09.191193+0000","last_became_peered":"2023-10-16T10:02:09.191193+0000","last_unstale":"2023-10-16T10:02:11.352187+0000","last_undegraded":"2023-10-16T10:02:11.352187+0000","last_fullsized":"2023-10-16T10:02:11.352187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:02.731436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352663+0000","last_change":"2023-10-16T10:02:10.203916+0000","last_active":"2023-10-16T10:02:11.352663+0000","last_peered":"2023-10-16T10:02:11.352663+0000","last_clean":"2023-10-16T10:02:11.352663+0000","last_became_active":"2023-10-16T10:02:10.203649+0000","last_became_peered":"2023-10-16T10:02:10.203649+0000","last_unstale":"2023-10-16T10:02:11.352663+0000","last_undegraded":"2023-10-16T10:02:11.352663+0000","last_fullsized":"2023-10-16T10:02:11.352663+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:04.547044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"125'8","reported_seq":20,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193921+0000","last_change":"2023-10-16T10:02:09.194873+0000","last_active":"2023-10-16T10:02:11.193921+0000","last_peered":"2023-10-16T10:02:11.193921+0000","last_clean":"2023-10-16T10:02:11.193921+0000","last_became_active":"2023-10-16T10:02:09.194584+0000","last_became_peered":"2023-10-16T10:02:09.194584+0000","last_unstale":"2023-10-16T10:02:11.193921+0000","last_undegraded":"2023-10-16T10:02:11.193921+0000","last_fullsized":"2023-10-16T10:02:11.193921+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:18.735326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352430+0000","last_change":"2023-10-16T10:02:10.199635+0000","last_active":"2023-10-16T10:02:11.352430+0000","last_peered":"2023-10-16T10:02:11.352430+0000","last_clean":"2023-10-16T10:02:11.352430+0000","last_became_active":"2023-10-16T10:02:10.199324+0000","last_became_peered":"2023-10-16T10:02:10.199324+0000","last_unstale":"2023-10-16T10:02:11.352430+0000","last_undegraded":"2023-10-16T10:02:11.352430+0000","last_fullsized":"2023-10-16T10:02:11.352430+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:19:10.442016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193158+0000","last_change":"2023-10-16T10:02:09.194317+0000","last_active":"2023-10-16T10:02:11.193158+0000","last_peered":"2023-10-16T10:02:11.193158+0000","last_clean":"2023-10-16T10:02:11.193158+0000","last_became_active":"2023-10-16T10:02:09.194035+0000","last_became_peered":"2023-10-16T10:02:09.194035+0000","last_unstale":"2023-10-16T10:02:11.193158+0000","last_undegraded":"2023-10-16T10:02:11.193158+0000","last_fullsized":"2023-10-16T10:02:11.193158+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:13.037875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192515+0000","last_change":"2023-10-16T10:02:10.200913+0000","last_active":"2023-10-16T10:02:11.192515+0000","last_peered":"2023-10-16T10:02:11.192515+0000","last_clean":"2023-10-16T10:02:11.192515+0000","last_became_active":"2023-10-16T10:02:10.200502+0000","last_became_peered":"2023-10-16T10:02:10.200502+0000","last_unstale":"2023-10-16T10:02:11.192515+0000","last_undegraded":"2023-10-16T10:02:11.192515+0000","last_fullsized":"2023-10-16T10:02:11.192515+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:17:16.863936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192042+0000","last_change":"2023-10-16T10:02:09.187874+0000","last_active":"2023-10-16T10:02:11.192042+0000","last_peered":"2023-10-16T10:02:11.192042+0000","last_clean":"2023-10-16T10:02:11.192042+0000","last_became_active":"2023-10-16T10:02:09.187542+0000","last_became_peered":"2023-10-16T10:02:09.187542+0000","last_unstale":"2023-10-16T10:02:11.192042+0000","last_undegraded":"2023-10-16T10:02:11.192042+0000","last_fullsized":"2023-10-16T10:02:11.192042+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:17.532553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352836+0000","last_change":"2023-10-16T10:02:10.200953+0000","last_active":"2023-10-16T10:02:11.352836+0000","last_peered":"2023-10-16T10:02:11.352836+0000","last_clean":"2023-10-16T10:02:11.352836+0000","last_became_active":"2023-10-16T10:02:10.200539+0000","last_became_peered":"2023-10-16T10:02:10.200539+0000","last_unstale":"2023-10-16T10:02:11.352836+0000","last_undegraded":"2023-10-16T10:02:11.352836+0000","last_fullsized":"2023-10-16T10:02:11.352836+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.210215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351929+0000","last_change":"2023-10-16T10:02:09.196786+0000","last_active":"2023-10-16T10:02:11.351929+0000","last_peered":"2023-10-16T10:02:11.351929+0000","last_clean":"2023-10-16T10:02:11.351929+0000","last_became_active":"2023-10-16T10:02:09.196485+0000","last_became_peered":"2023-10-16T10:02:09.196485+0000","last_unstale":"2023-10-16T10:02:11.351929+0000","last_undegraded":"2023-10-16T10:02:11.351929+0000","last_fullsized":"2023-10-16T10:02:11.351929+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:17.532553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351876+0000","last_change":"2023-10-16T10:02:10.196692+0000","last_active":"2023-10-16T10:02:11.351876+0000","last_peered":"2023-10-16T10:02:11.351876+0000","last_clean":"2023-10-16T10:02:11.351876+0000","last_became_active":"2023-10-16T10:02:10.196247+0000","last_became_peered":"2023-10-16T10:02:10.196247+0000","last_unstale":"2023-10-16T10:02:11.351876+0000","last_undegraded":"2023-10-16T10:02:11.351876+0000","last_fullsized":"2023-10-16T10:02:11.351876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:00.793954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352281+0000","last_change":"2023-10-16T10:02:09.189535+0000","last_active":"2023-10-16T10:02:11.352281+0000","last_peered":"2023-10-16T10:02:11.352281+0000","last_clean":"2023-10-16T10:02:11.352281+0000","last_became_active":"2023-10-16T10:02:09.189267+0000","last_became_peered":"2023-10-16T10:02:09.189267+0000","last_unstale":"2023-10-16T10:02:11.352281+0000","last_undegraded":"2023-10-16T10:02:11.352281+0000","last_fullsized":"2023-10-16T10:02:11.352281+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:35:06.124697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352110+0000","last_change":"2023-10-16T10:02:10.198667+0000","last_active":"2023-10-16T10:02:11.352110+0000","last_peered":"2023-10-16T10:02:11.352110+0000","last_clean":"2023-10-16T10:02:11.352110+0000","last_became_active":"2023-10-16T10:02:10.198400+0000","last_became_peered":"2023-10-16T10:02:10.198400+0000","last_unstale":"2023-10-16T10:02:11.352110+0000","last_undegraded":"2023-10-16T10:02:11.352110+0000","last_fullsized":"2023-10-16T10:02:11.352110+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.789284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352123+0000","last_change":"2023-10-16T10:02:09.195780+0000","last_active":"2023-10-16T10:02:11.352123+0000","last_peered":"2023-10-16T10:02:11.352123+0000","last_clean":"2023-10-16T10:02:11.352123+0000","last_became_active":"2023-10-16T10:02:09.195524+0000","last_became_peered":"2023-10-16T10:02:09.195524+0000","last_unstale":"2023-10-16T10:02:11.352123+0000","last_undegraded":"2023-10-16T10:02:11.352123+0000","last_fullsized":"2023-10-16T10:02:11.352123+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:22.279374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353097+0000","last_change":"2023-10-16T10:02:10.211311+0000","last_active":"2023-10-16T10:02:11.353097+0000","last_peered":"2023-10-16T10:02:11.353097+0000","last_clean":"2023-10-16T10:02:11.353097+0000","last_became_active":"2023-10-16T10:02:10.211029+0000","last_became_peered":"2023-10-16T10:02:10.211029+0000","last_unstale":"2023-10-16T10:02:11.353097+0000","last_undegraded":"2023-10-16T10:02:11.353097+0000","last_fullsized":"2023-10-16T10:02:11.353097+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:35.557520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192394+0000","last_change":"2023-10-16T10:02:09.196617+0000","last_active":"2023-10-16T10:02:11.192394+0000","last_peered":"2023-10-16T10:02:11.192394+0000","last_clean":"2023-10-16T10:02:11.192394+0000","last_became_active":"2023-10-16T10:02:09.196338+0000","last_became_peered":"2023-10-16T10:02:09.196338+0000","last_unstale":"2023-10-16T10:02:11.192394+0000","last_undegraded":"2023-10-16T10:02:11.192394+0000","last_fullsized":"2023-10-16T10:02:11.192394+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:25.426647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353367+0000","last_change":"2023-10-16T10:02:10.202288+0000","last_active":"2023-10-16T10:02:11.353367+0000","last_peered":"2023-10-16T10:02:11.353367+0000","last_clean":"2023-10-16T10:02:11.353367+0000","last_became_active":"2023-10-16T10:02:10.201981+0000","last_became_peered":"2023-10-16T10:02:10.201981+0000","last_unstale":"2023-10-16T10:02:11.353367+0000","last_undegraded":"2023-10-16T10:02:11.353367+0000","last_fullsized":"2023-10-16T10:02:11.353367+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:16.822411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352512+0000","last_change":"2023-10-16T10:02:09.196428+0000","last_active":"2023-10-16T10:02:11.352512+0000","last_peered":"2023-10-16T10:02:11.352512+0000","last_clean":"2023-10-16T10:02:11.352512+0000","last_became_active":"2023-10-16T10:02:09.196167+0000","last_became_peered":"2023-10-16T10:02:09.196167+0000","last_unstale":"2023-10-16T10:02:11.352512+0000","last_undegraded":"2023-10-16T10:02:11.352512+0000","last_fullsized":"2023-10-16T10:02:11.352512+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:05.663564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351720+0000","last_change":"2023-10-16T10:02:10.196276+0000","last_active":"2023-10-16T10:02:11.351720+0000","last_peered":"2023-10-16T10:02:11.351720+0000","last_clean":"2023-10-16T10:02:11.351720+0000","last_became_active":"2023-10-16T10:02:10.195993+0000","last_became_peered":"2023-10-16T10:02:10.195993+0000","last_unstale":"2023-10-16T10:02:11.351720+0000","last_undegraded":"2023-10-16T10:02:11.351720+0000","last_fullsized":"2023-10-16T10:02:11.351720+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.649854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352940+0000","last_change":"2023-10-16T10:02:09.192226+0000","last_active":"2023-10-16T10:02:11.352940+0000","last_peered":"2023-10-16T10:02:11.352940+0000","last_clean":"2023-10-16T10:02:11.352940+0000","last_became_active":"2023-10-16T10:02:09.191924+0000","last_became_peered":"2023-10-16T10:02:09.191924+0000","last_unstale":"2023-10-16T10:02:11.352940+0000","last_undegraded":"2023-10-16T10:02:11.352940+0000","last_fullsized":"2023-10-16T10:02:11.352940+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:18.636295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193258+0000","last_change":"2023-10-16T10:02:10.196683+0000","last_active":"2023-10-16T10:02:11.193258+0000","last_peered":"2023-10-16T10:02:11.193258+0000","last_clean":"2023-10-16T10:02:11.193258+0000","last_became_active":"2023-10-16T10:02:10.196401+0000","last_became_peered":"2023-10-16T10:02:10.196401+0000","last_unstale":"2023-10-16T10:02:11.193258+0000","last_undegraded":"2023-10-16T10:02:11.193258+0000","last_fullsized":"2023-10-16T10:02:11.193258+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351882+0000","last_change":"2023-10-16T10:02:09.191233+0000","last_active":"2023-10-16T10:02:11.351882+0000","last_peered":"2023-10-16T10:02:11.351882+0000","last_clean":"2023-10-16T10:02:11.351882+0000","last_became_active":"2023-10-16T10:02:09.190963+0000","last_became_peered":"2023-10-16T10:02:09.190963+0000","last_unstale":"2023-10-16T10:02:11.351882+0000","last_undegraded":"2023-10-16T10:02:11.351882+0000","last_fullsized":"2023-10-16T10:02:11.351882+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:29.769245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352735+0000","last_change":"2023-10-16T10:02:10.207400+0000","last_active":"2023-10-16T10:02:11.352735+0000","last_peered":"2023-10-16T10:02:11.352735+0000","last_clean":"2023-10-16T10:02:11.352735+0000","last_became_active":"2023-10-16T10:02:10.206418+0000","last_became_peered":"2023-10-16T10:02:10.206418+0000","last_unstale":"2023-10-16T10:02:11.352735+0000","last_undegraded":"2023-10-16T10:02:11.352735+0000","last_fullsized":"2023-10-16T10:02:11.352735+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:30.773531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192728+0000","last_change":"2023-10-16T10:02:09.196020+0000","last_active":"2023-10-16T10:02:11.192728+0000","last_peered":"2023-10-16T10:02:11.192728+0000","last_clean":"2023-10-16T10:02:11.192728+0000","last_became_active":"2023-10-16T10:02:09.195679+0000","last_became_peered":"2023-10-16T10:02:09.195679+0000","last_unstale":"2023-10-16T10:02:11.192728+0000","last_undegraded":"2023-10-16T10:02:11.192728+0000","last_fullsized":"2023-10-16T10:02:11.192728+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:46.209694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351898+0000","last_change":"2023-10-16T10:02:10.202892+0000","last_active":"2023-10-16T10:02:11.351898+0000","last_peered":"2023-10-16T10:02:11.351898+0000","last_clean":"2023-10-16T10:02:11.351898+0000","last_became_active":"2023-10-16T10:02:10.202548+0000","last_became_peered":"2023-10-16T10:02:10.202548+0000","last_unstale":"2023-10-16T10:02:11.351898+0000","last_undegraded":"2023-10-16T10:02:11.351898+0000","last_fullsized":"2023-10-16T10:02:11.351898+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:37:20.487393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352933+0000","last_change":"2023-10-16T10:02:09.192658+0000","last_active":"2023-10-16T10:02:11.352933+0000","last_peered":"2023-10-16T10:02:11.352933+0000","last_clean":"2023-10-16T10:02:11.352933+0000","last_became_active":"2023-10-16T10:02:09.192381+0000","last_became_peered":"2023-10-16T10:02:09.192381+0000","last_unstale":"2023-10-16T10:02:11.352933+0000","last_undegraded":"2023-10-16T10:02:11.352933+0000","last_fullsized":"2023-10-16T10:02:11.352933+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:18.735326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352062+0000","last_change":"2023-10-16T10:02:10.207705+0000","last_active":"2023-10-16T10:02:11.352062+0000","last_peered":"2023-10-16T10:02:11.352062+0000","last_clean":"2023-10-16T10:02:11.352062+0000","last_became_active":"2023-10-16T10:02:10.207425+0000","last_became_peered":"2023-10-16T10:02:10.207425+0000","last_unstale":"2023-10-16T10:02:11.352062+0000","last_undegraded":"2023-10-16T10:02:11.352062+0000","last_fullsized":"2023-10-16T10:02:11.352062+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:28.643280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193587+0000","last_change":"2023-10-16T10:02:09.193643+0000","last_active":"2023-10-16T10:02:11.193587+0000","last_peered":"2023-10-16T10:02:11.193587+0000","last_clean":"2023-10-16T10:02:11.193587+0000","last_became_active":"2023-10-16T10:02:09.193334+0000","last_became_peered":"2023-10-16T10:02:09.193334+0000","last_unstale":"2023-10-16T10:02:11.193587+0000","last_undegraded":"2023-10-16T10:02:11.193587+0000","last_fullsized":"2023-10-16T10:02:11.193587+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:52.205929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352155+0000","last_change":"2023-10-16T10:02:10.204803+0000","last_active":"2023-10-16T10:02:11.352155+0000","last_peered":"2023-10-16T10:02:11.352155+0000","last_clean":"2023-10-16T10:02:11.352155+0000","last_became_active":"2023-10-16T10:02:10.204521+0000","last_became_peered":"2023-10-16T10:02:10.204521+0000","last_unstale":"2023-10-16T10:02:11.352155+0000","last_undegraded":"2023-10-16T10:02:11.352155+0000","last_fullsized":"2023-10-16T10:02:11.352155+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:12.773991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352137+0000","last_change":"2023-10-16T10:02:09.192983+0000","last_active":"2023-10-16T10:02:11.352137+0000","last_peered":"2023-10-16T10:02:11.352137+0000","last_clean":"2023-10-16T10:02:11.352137+0000","last_became_active":"2023-10-16T10:02:09.191246+0000","last_became_peered":"2023-10-16T10:02:09.191246+0000","last_unstale":"2023-10-16T10:02:11.352137+0000","last_undegraded":"2023-10-16T10:02:11.352137+0000","last_fullsized":"2023-10-16T10:02:11.352137+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:56.170139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191954+0000","last_change":"2023-10-16T10:02:10.204350+0000","last_active":"2023-10-16T10:02:11.191954+0000","last_peered":"2023-10-16T10:02:11.191954+0000","last_clean":"2023-10-16T10:02:11.191954+0000","last_became_active":"2023-10-16T10:02:10.204025+0000","last_became_peered":"2023-10-16T10:02:10.204025+0000","last_unstale":"2023-10-16T10:02:11.191954+0000","last_undegraded":"2023-10-16T10:02:11.191954+0000","last_fullsized":"2023-10-16T10:02:11.191954+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:36.493038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192601+0000","last_change":"2023-10-16T10:02:09.196040+0000","last_active":"2023-10-16T10:02:11.192601+0000","last_peered":"2023-10-16T10:02:11.192601+0000","last_clean":"2023-10-16T10:02:11.192601+0000","last_became_active":"2023-10-16T10:02:09.195604+0000","last_became_peered":"2023-10-16T10:02:09.195604+0000","last_unstale":"2023-10-16T10:02:11.192601+0000","last_undegraded":"2023-10-16T10:02:11.192601+0000","last_fullsized":"2023-10-16T10:02:11.192601+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:05.397655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352340+0000","last_change":"2023-10-16T10:02:10.201729+0000","last_active":"2023-10-16T10:02:11.352340+0000","last_peered":"2023-10-16T10:02:11.352340+0000","last_clean":"2023-10-16T10:02:11.352340+0000","last_became_active":"2023-10-16T10:02:10.201463+0000","last_became_peered":"2023-10-16T10:02:10.201463+0000","last_unstale":"2023-10-16T10:02:11.352340+0000","last_undegraded":"2023-10-16T10:02:11.352340+0000","last_fullsized":"2023-10-16T10:02:11.352340+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:35.557520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352587+0000","last_change":"2023-10-16T10:02:09.190819+0000","last_active":"2023-10-16T10:02:11.352587+0000","last_peered":"2023-10-16T10:02:11.352587+0000","last_clean":"2023-10-16T10:02:11.352587+0000","last_became_active":"2023-10-16T10:02:09.190349+0000","last_became_peered":"2023-10-16T10:02:09.190349+0000","last_unstale":"2023-10-16T10:02:11.352587+0000","last_undegraded":"2023-10-16T10:02:11.352587+0000","last_fullsized":"2023-10-16T10:02:11.352587+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:19.691248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351799+0000","last_change":"2023-10-16T10:02:10.203732+0000","last_active":"2023-10-16T10:02:11.351799+0000","last_peered":"2023-10-16T10:02:11.351799+0000","last_clean":"2023-10-16T10:02:11.351799+0000","last_became_active":"2023-10-16T10:02:10.203427+0000","last_became_peered":"2023-10-16T10:02:10.203427+0000","last_unstale":"2023-10-16T10:02:11.351799+0000","last_undegraded":"2023-10-16T10:02:11.351799+0000","last_fullsized":"2023-10-16T10:02:11.351799+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.616882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353066+0000","last_change":"2023-10-16T10:02:09.193873+0000","last_active":"2023-10-16T10:02:11.353066+0000","last_peered":"2023-10-16T10:02:11.353066+0000","last_clean":"2023-10-16T10:02:11.353066+0000","last_became_active":"2023-10-16T10:02:09.193576+0000","last_became_peered":"2023-10-16T10:02:09.193576+0000","last_unstale":"2023-10-16T10:02:11.353066+0000","last_undegraded":"2023-10-16T10:02:11.353066+0000","last_fullsized":"2023-10-16T10:02:11.353066+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:28:18.814767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"116'205","reported_seq":583,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-16T10:02:10.190584+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:02:10.190584+0000","last_peered":"2023-10-16T10:02:10.190584+0000","last_clean":"2023-10-16T10:02:10.190584+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:02:10.190584+0000","last_undegraded":"2023-10-16T10:02:10.190584+0000","last_fullsized":"2023-10-16T10:02:10.190584+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59668,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312172,"statfs":{"total":96636764160,"available":96575664128,"internally_reserved":0,"allocated":2162688,"data_stored":1404660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54300000000000004,"15min":0.55700000000000005},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.219},"max":{"1min":0.753,"5min":0.84399999999999997,"15min":1.2150000000000001},"last":0.747},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.254},"max":{"1min":0.76900000000000002,"5min":1.911,"15min":1.911},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58199999999999996,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79200000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.80500000000000005,"5min":0.80800000000000005,"15min":1.1279999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.45300000000000001,"5min":0.30299999999999999,"15min":0.27700000000000002},"max":{"1min":0.78300000000000003,"5min":3.3820000000000001,"15min":3.3820000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.54400000000000004,"5min":0.35899999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":3.1640000000000001,"15min":3.1640000000000001},"last":0.80400000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.46500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82199999999999995,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.49399999999999999,"5min":0.41099999999999998,"15min":0.34999999999999998},"max":{"1min":0.84799999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.81899999999999995}]},{"osd":4,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.499,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65000000000000002,"5min":1.8220000000000001,"15min":1.9550000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55800000000000005,"15min":0.52300000000000002},"min":{"1min":0.39800000000000002,"5min":0.315,"15min":0.27200000000000002},"max":{"1min":0.66800000000000004,"5min":3.8149999999999999,"15min":3.8149999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.40000000000000002,"5min":0.33200000000000002,"15min":0.28199999999999997},"max":{"1min":0.68600000000000005,"5min":0.78600000000000003,"15min":0.84099999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.308},"max":{"1min":0.76200000000000001,"5min":1.6619999999999999,"15min":2.0150000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":1.7110000000000001,"15min":1.9870000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.44,"5min":0.31,"15min":0.31},"max":{"1min":0.751,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.76500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1581056,"data_stored":816348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56299999999999994,"15min":0.58399999999999996},"min":{"1min":0.45200000000000001,"5min":0.29699999999999999,"15min":0.223},"max":{"1min":0.89100000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.249},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.82099999999999995},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.59099999999999997},"min":{"1min":0.32500000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77100000000000002,"5min":0.83699999999999997,"15min":0.90000000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.57299999999999995,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.33000000000000002,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.39200000000000002,"5min":0.28799999999999998,"15min":0.27100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":1.448},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.60599999999999998},"min":{"1min":0.47799999999999998,"5min":0.35899999999999999,"15min":0.35799999999999998},"max":{"1min":0.72599999999999998,"5min":0.874,"15min":1.708},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57299999999999995,"15min":0.58999999999999997},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79900000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.59199999999999997},"min":{"1min":0.436,"5min":0.36899999999999999,"15min":0.315},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65800000000000003}]},{"osd":4,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.47999999999999998},"min":{"1min":0.33400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.76200000000000001,"5min":0.77400000000000002,"15min":0.80600000000000005},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70999999999999996}]},{"osd":5,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.504,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.69099999999999995,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39600000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56699999999999995,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.86499999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.57199999999999995},"min":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.94599999999999995},"last":0.79200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36860,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334980,"statfs":{"total":96636764160,"available":96599019520,"internally_reserved":0,"allocated":1552384,"data_stored":807122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38100000000000001,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.247,"15min":0.188},"max":{"1min":0.54000000000000004,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.23200000000000001},"max":{"1min":0.56399999999999995,"5min":0.67900000000000005,"15min":0.70199999999999996},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42799999999999999,"15min":0.40999999999999998},"min":{"1min":0.26300000000000001,"5min":0.25900000000000001,"15min":0.20799999999999999},"max":{"1min":0.61699999999999999,"5min":4.923,"15min":4.923},"last":0.252},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.432,"15min":0.435},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65800000000000003,"5min":1.712,"15min":1.712},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59399999999999997,"5min":1.984,"15min":1.984},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.42499999999999999,"15min":0.432},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63700000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59799999999999998,"15min":0.628},"min":{"1min":0.499,"5min":0.39900000000000002,"15min":0.377},"max":{"1min":0.87,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.34399999999999997},"max":{"1min":1.014,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.41799999999999998,"5min":0.39300000000000002,"15min":0.36599999999999999},"max":{"1min":0.96299999999999997,"5min":4.9749999999999996,"15min":4.9749999999999996},"last":0.504},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65700000000000003,"15min":0.67400000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40500000000000003},"max":{"1min":1.054,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.61799999999999999,"15min":0.64100000000000001},"min":{"1min":0.48599999999999999,"5min":0.42299999999999999,"15min":0.38200000000000001},"max":{"1min":0.91100000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.44},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68100000000000005,"15min":0.66000000000000003},"min":{"1min":0.47599999999999998,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":1.0449999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.53400000000000003}]},{"osd":7,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63600000000000001,"15min":0.66100000000000003},"min":{"1min":0.46700000000000003,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96599999999999997,"5min":1.744,"15min":1.744},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.624,"15min":0.65000000000000002},"min":{"1min":0.53300000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91400000000000003},"last":0.48199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575073,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37448,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334392,"statfs":{"total":96636764160,"available":96598417408,"internally_reserved":0,"allocated":2158592,"data_stored":1399158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.38300000000000001},"min":{"1min":0.254,"5min":0.222,"15min":0.192},"max":{"1min":0.40799999999999997,"5min":0.66800000000000004,"15min":1.2110000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.26500000000000001},"max":{"1min":0.63,"5min":0.70999999999999996,"15min":1.3169999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.42099999999999999},"min":{"1min":0.314,"5min":0.251,"15min":0.20599999999999999},"max":{"1min":0.498,"5min":0.57899999999999996,"15min":0.70099999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.44,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":0.54900000000000004,"5min":0.78100000000000003,"15min":0.79900000000000004},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39900000000000002,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":0.53800000000000003,"5min":0.60299999999999998,"15min":0.83399999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44800000000000001,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53100000000000003,"5min":0.68899999999999995,"15min":1.1859999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.621},"min":{"1min":0.505,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.81799999999999995,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.40999999999999998,"15min":0.32100000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":1.1659999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.53000000000000003,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.81599999999999995,"5min":0.94999999999999996,"15min":0.998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.59699999999999998,"15min":0.63},"min":{"1min":0.436,"5min":0.40400000000000003,"15min":0.311},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.93000000000000005},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63300000000000001,"15min":0.66300000000000003},"min":{"1min":0.51000000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.2410000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63800000000000001,"15min":0.67000000000000004},"min":{"1min":0.46600000000000003,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.877,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.69599999999999995}]},{"osd":7,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.316},"max":{"1min":0.78800000000000003,"5min":1.014,"15min":1.02},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.61399999999999999,"15min":0.64400000000000002},"min":{"1min":0.47799999999999998,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.79800000000000004,"5min":0.876,"15min":0.90900000000000003},"last":0.67800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575072,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1564672,"data_stored":814166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.379},"min":{"1min":0.251,"5min":0.217,"15min":0.217},"max":{"1min":0.57099999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.316},{"interface":"front","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.67500000000000004},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.309,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56000000000000005,"5min":0.68000000000000005,"15min":0.754},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.38,"15min":0.39800000000000002},"min":{"1min":0.28100000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59599999999999997,"5min":0.624,"15min":0.624},"last":0.379}]},{"osd":3,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40500000000000003,"15min":0.41799999999999998},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.23499999999999999},"max":{"1min":0.54100000000000004,"5min":0.57599999999999996,"15min":0.63500000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.45400000000000001,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65000000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.63900000000000001},"min":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84199999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65100000000000002},"min":{"1min":0.48899999999999999,"5min":0.38600000000000001,"15min":0.33400000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63300000000000001}]},{"osd":5,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84999999999999998,"5min":1.0249999999999999,"15min":1.103},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.624,"15min":0.65200000000000002},"min":{"1min":0.44400000000000001,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.46000000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82899999999999996,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.67600000000000005},"min":{"1min":0.42599999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91600000000000004,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.65500000000000003,"15min":0.68000000000000005},"min":{"1min":0.52100000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64700000000000002,"15min":0.66700000000000004},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.371},"max":{"1min":0.95899999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575072,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":807670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.36899999999999999},"min":{"1min":0.25600000000000001,"5min":0.20999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.37,"15min":0.39200000000000002},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.50800000000000001,"5min":0.53600000000000003,"15min":0.65400000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.42699999999999999},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.50900000000000001,"5min":0.60899999999999999,"15min":0.67500000000000004},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44,"15min":0.46500000000000002},"min":{"1min":0.36299999999999999,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":0.96099999999999997,"15min":1.0760000000000001},"last":0.251},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47099999999999997,"15min":0.48799999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.28699999999999998},"max":{"1min":0.61399999999999999,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.33000000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58699999999999997,"15min":0.621},"min":{"1min":0.498,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.84599999999999997,"5min":0.90300000000000002,"15min":0.96499999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.63600000000000001},"min":{"1min":0.46999999999999997,"5min":0.38900000000000001,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.432}]},{"osd":5,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.63700000000000001},"min":{"1min":0.49199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.51800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80200000000000005,"5min":0.873,"15min":1.036},"last":0.41699999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60799999999999998,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.39600000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.88100000000000001,"15min":0.98799999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.60199999999999998,"15min":0.65200000000000002},"min":{"1min":0.45400000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78400000000000003,"5min":0.92600000000000005,"15min":1.0589999999999999},"last":0.46899999999999997}]},{"osd":7,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.66900000000000004},"min":{"1min":0.53500000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84999999999999998,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60699999999999998,"15min":0.65100000000000002},"min":{"1min":0.53000000000000003,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.79200000000000004,"5min":0.99399999999999999,"15min":0.999},"last":0.48699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575072,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1568768,"data_stored":814224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.82099999999999995,"15min":1.0269999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.59099999999999997},"min":{"1min":0.498,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.77000000000000002,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.73899999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55100000000000005,"15min":0.58699999999999997},"min":{"1min":0.45300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.71899999999999997,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57299999999999995,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.307,"15min":0.28399999999999997},"max":{"1min":0.753,"5min":0.85199999999999998,"15min":0.88300000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.61699999999999999},"min":{"1min":0.38600000000000001,"5min":0.379,"15min":0.32500000000000001},"max":{"1min":0.745,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.47899999999999998}]},{"osd":3,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.433,"5min":0.313,"15min":0.313},"max":{"1min":0.73199999999999998,"5min":0.77700000000000002,"15min":0.86099999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.52900000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71199999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.42999999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71399999999999997,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.44500000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72899999999999998,"5min":0.79300000000000004,"15min":0.96599999999999997},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.79100000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.47299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78100000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.44600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.83099999999999996,"15min":0.91100000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.50800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81599999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575072,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50756,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321084,"statfs":{"total":96636764160,"available":96584790016,"internally_reserved":0,"allocated":1560576,"data_stored":814132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58799999999999997,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.28599999999999998},"max":{"1min":0.78000000000000003,"5min":0.96199999999999997,"15min":1.792},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.41399999999999998,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.7090000000000001},"last":0.41399999999999998}]},{"osd":1,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55400000000000005,"15min":0.57899999999999996},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.749,"5min":0.83099999999999996,"15min":0.875},"last":0.437},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55500000000000005,"15min":0.57799999999999996},"min":{"1min":0.39000000000000001,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":0.79300000000000004,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.38900000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.25},"max":{"1min":0.80800000000000005,"5min":1.496,"15min":1.496},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57899999999999996,"15min":0.59799999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":1.5209999999999999,"15min":2.069},"last":0.497}]},{"osd":3,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55700000000000005,"15min":0.57299999999999995},"min":{"1min":0.46000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.89100000000000001,"15min":1.7689999999999999},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.28399999999999997,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.73799999999999999,"15min":0.84199999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.318,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":0.70999999999999996,"5min":0.749,"15min":0.86899999999999999},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.374,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.39500000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.68400000000000005,"5min":0.76900000000000002,"15min":0.88200000000000001},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.25900000000000001},"max":{"1min":0.746,"5min":0.82399999999999995,"15min":0.90300000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.46500000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.88500000000000001},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:02:14.325 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:02:14.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.480+0000 7fc419667700 1 Processor -- start 2023-10-16T10:02:14.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.480+0000 7fc419667700 1 -- start start 2023-10-16T10:02:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc419667700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc41406a8d0 0x7fc41406e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc419667700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4140718d0 0x7fc41415e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc419667700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41415dcb0 con 0x7fc4141612a0 2023-10-16T10:02:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc419667700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc414161740 con 0x7fc4140718d0 2023-10-16T10:02:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc419667700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc414161a20 con 0x7fc41406a8d0 2023-10-16T10:02:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4140718d0 0x7fc41415e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4140718d0 0x7fc41415e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56702/0 (socket says 172.21.15.158:56702) 2023-10-16T10:02:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 -- 172.21.15.158:0/4137099819 learned_addr learned my addr 172.21.15.158:0/4137099819 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 -- 172.21.15.158:0/4137099819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc41406a8d0 msgr2=0x7fc41406e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 --2- 172.21.15.158:0/4137099819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc41406a8d0 0x7fc41406e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 -- 172.21.15.158:0/4137099819 >> v1:172.21.15.172:6789/0 conn(0x7fc4141612a0 legacy=0x7fc41406acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 -- 172.21.15.158:0/4137099819 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414161d00 con 0x7fc4140718d0 2023-10-16T10:02:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc4137fe700 1 --2- 172.21.15.158:0/4137099819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4140718d0 0x7fc41415e140 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009e50 tx=0x7fc40c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01127052bf09ed4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc411ffb700 1 -- 172.21.15.158:0/4137099819 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c00e070 con 0x7fc4140718d0 2023-10-16T10:02:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc411ffb700 1 -- 172.21.15.158:0/4137099819 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c003a20 con 0x7fc4140718d0 2023-10-16T10:02:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.481+0000 7fc411ffb700 1 -- 172.21.15.158:0/4137099819 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c003dd0 con 0x7fc4140718d0 2023-10-16T10:02:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 -- 172.21.15.158:0/4137099819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4140718d0 msgr2=0x7fc41415e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 --2- 172.21.15.158:0/4137099819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4140718d0 0x7fc41415e140 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009e50 tx=0x7fc40c00b330 comp rx=0 tx=0).stop 2023-10-16T10:02:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 -- 172.21.15.158:0/4137099819 shutdown_connections 2023-10-16T10:02:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 --2- 172.21.15.158:0/4137099819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4140718d0 0x7fc41415e140 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 --2- 172.21.15.158:0/4137099819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc41406a8d0 0x7fc41406e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 -- 172.21.15.158:0/4137099819 >> 172.21.15.158:0/4137099819 conn(0x7fc4140d02a0 msgr2=0x7fc4140d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 -- 172.21.15.158:0/4137099819 shutdown_connections 2023-10-16T10:02:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.482+0000 7fc419667700 1 -- 172.21.15.158:0/4137099819 wait complete. 2023-10-16T10:02:14.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.483+0000 7fc419667700 1 Processor -- start 2023-10-16T10:02:14.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.483+0000 7fc419667700 1 -- start start 2023-10-16T10:02:14.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc419667700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc41406a8d0 0x7fc414158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc419667700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4140718d0 0x7fc4141e6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc419667700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4141612a0 0x7fc4141536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc419667700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc414159510 con 0x7fc41406a8d0 2023-10-16T10:02:14.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc419667700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc414158900 con 0x7fc4141612a0 2023-10-16T10:02:14.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc419667700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc414158be0 con 0x7fc4140718d0 2023-10-16T10:02:14.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc41406a8d0 0x7fc414158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc41406a8d0 0x7fc414158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56704/0 (socket says 172.21.15.158:56704) 2023-10-16T10:02:14.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 -- 172.21.15.158:0/3433447307 learned_addr learned my addr 172.21.15.158:0/3433447307 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:14.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 -- 172.21.15.158:0/3433447307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4140718d0 msgr2=0x7fc4141e6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4140718d0 0x7fc4141e6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 -- 172.21.15.158:0/3433447307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4141612a0 msgr2=0x7fc4141536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4141612a0 0x7fc4141536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.484+0000 7fc412ffd700 1 -- 172.21.15.158:0/3433447307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c009b30 con 0x7fc41406a8d0 2023-10-16T10:02:14.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.485+0000 7fc412ffd700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc41406a8d0 0x7fc414158320 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fc40400b530 tx=0x7fc40400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.485+0000 7fc3fb7fe700 1 -- 172.21.15.158:0/3433447307 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404011830 con 0x7fc41406a8d0 2023-10-16T10:02:14.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.485+0000 7fc3fb7fe700 1 -- 172.21.15.158:0/3433447307 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4040119b0 con 0x7fc41406a8d0 2023-10-16T10:02:14.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.485+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc414158ec0 con 0x7fc41406a8d0 2023-10-16T10:02:14.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.485+0000 7fc3fb7fe700 1 -- 172.21.15.158:0/3433447307 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404011d60 con 0x7fc41406a8d0 2023-10-16T10:02:14.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.485+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc414153c10 con 0x7fc41406a8d0 2023-10-16T10:02:14.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.486+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4140681b0 con 0x7fc41406a8d0 2023-10-16T10:02:14.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.486+0000 7fc3fb7fe700 1 -- 172.21.15.158:0/3433447307 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc40401a400 con 0x7fc41406a8d0 2023-10-16T10:02:14.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.487+0000 7fc3fb7fe700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc40006b020 0x7fc40006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.487+0000 7fc3fb7fe700 1 -- 172.21.15.158:0/3433447307 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fc404089ac0 con 0x7fc41406a8d0 2023-10-16T10:02:14.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.487+0000 7fc4127fc700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc40006b020 0x7fc40006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.487+0000 7fc4127fc700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc40006b020 0x7fc40006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4141f6740 tx=0x7fc408006c50 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.491+0000 7fc3fb7fe700 1 -- 172.21.15.158:0/3433447307 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc404054770 con 0x7fc41406a8d0 2023-10-16T10:02:14.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.679+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 --> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc414161fe0 con 0x7fc40006b020 2023-10-16T10:02:14.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.687+0000 7fc3fb7fe700 1 -- 172.21.15.158:0/3433447307 <== mgr.8022 v2:172.21.15.158:6832/1080332904 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289996 (secure 0 0 0) 0x7fc408009390 con 0x7fc40006b020 2023-10-16T10:02:14.691 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:14.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc40006b020 msgr2=0x7fc40006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc40006b020 0x7fc40006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4141f6740 tx=0x7fc408006c50 comp rx=0 tx=0).stop 2023-10-16T10:02:14.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc41406a8d0 msgr2=0x7fc414158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc41406a8d0 0x7fc414158320 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fc40400b530 tx=0x7fc40400b8f0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 shutdown_connections 2023-10-16T10:02:14.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4141612a0 0x7fc4141536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fc40006b020 0x7fc40006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc41406a8d0 0x7fc414158320 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 --2- 172.21.15.158:0/3433447307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4140718d0 0x7fc4141e6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 >> 172.21.15.158:0/3433447307 conn(0x7fc4140d02a0 msgr2=0x7fc4140cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:14.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 shutdown_connections 2023-10-16T10:02:14.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.690+0000 7fc419667700 1 -- 172.21.15.158:0/3433447307 wait complete. 2023-10-16T10:02:14.696 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:02:14.703 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-16T10:02:14.137380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":346120,"kb_used_data":13380,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754628600,"statfs":{"total":773094113280,"available":772739686400,"internally_reserved":0,"allocated":13701120,"data_stored":7677480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":340577847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.965144"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352538+0000","last_change":"2023-10-16T10:02:10.196386+0000","last_active":"2023-10-16T10:02:11.352538+0000","last_peered":"2023-10-16T10:02:11.352538+0000","last_clean":"2023-10-16T10:02:11.352538+0000","last_became_active":"2023-10-16T10:02:10.195959+0000","last_became_peered":"2023-10-16T10:02:10.195959+0000","last_unstale":"2023-10-16T10:02:11.352538+0000","last_undegraded":"2023-10-16T10:02:11.352538+0000","last_fullsized":"2023-10-16T10:02:11.352538+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:44.278653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193035+0000","last_change":"2023-10-16T10:02:10.206257+0000","last_active":"2023-10-16T10:02:11.193035+0000","last_peered":"2023-10-16T10:02:11.193035+0000","last_clean":"2023-10-16T10:02:11.193035+0000","last_became_active":"2023-10-16T10:02:10.205900+0000","last_became_peered":"2023-10-16T10:02:10.205900+0000","last_unstale":"2023-10-16T10:02:11.193035+0000","last_undegraded":"2023-10-16T10:02:11.193035+0000","last_fullsized":"2023-10-16T10:02:11.193035+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:37.304152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351844+0000","last_change":"2023-10-16T10:02:10.211560+0000","last_active":"2023-10-16T10:02:11.351844+0000","last_peered":"2023-10-16T10:02:11.351844+0000","last_clean":"2023-10-16T10:02:11.351844+0000","last_became_active":"2023-10-16T10:02:10.211244+0000","last_became_peered":"2023-10-16T10:02:10.211244+0000","last_unstale":"2023-10-16T10:02:11.351844+0000","last_undegraded":"2023-10-16T10:02:11.351844+0000","last_fullsized":"2023-10-16T10:02:11.351844+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:01.574752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353099+0000","last_change":"2023-10-16T10:02:10.197281+0000","last_active":"2023-10-16T10:02:11.353099+0000","last_peered":"2023-10-16T10:02:11.353099+0000","last_clean":"2023-10-16T10:02:11.353099+0000","last_became_active":"2023-10-16T10:02:10.197020+0000","last_became_peered":"2023-10-16T10:02:10.197020+0000","last_unstale":"2023-10-16T10:02:11.353099+0000","last_undegraded":"2023-10-16T10:02:11.353099+0000","last_fullsized":"2023-10-16T10:02:11.353099+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:20.124547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353470+0000","last_change":"2023-10-16T10:02:10.200305+0000","last_active":"2023-10-16T10:02:11.353470+0000","last_peered":"2023-10-16T10:02:11.353470+0000","last_clean":"2023-10-16T10:02:11.353470+0000","last_became_active":"2023-10-16T10:02:10.198547+0000","last_became_peered":"2023-10-16T10:02:10.198547+0000","last_unstale":"2023-10-16T10:02:11.353470+0000","last_undegraded":"2023-10-16T10:02:11.353470+0000","last_fullsized":"2023-10-16T10:02:11.353470+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:42:02.838978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192660+0000","last_change":"2023-10-16T10:02:10.208604+0000","last_active":"2023-10-16T10:02:11.192660+0000","last_peered":"2023-10-16T10:02:11.192660+0000","last_clean":"2023-10-16T10:02:11.192660+0000","last_became_active":"2023-10-16T10:02:10.208175+0000","last_became_peered":"2023-10-16T10:02:10.208175+0000","last_unstale":"2023-10-16T10:02:11.192660+0000","last_undegraded":"2023-10-16T10:02:11.192660+0000","last_fullsized":"2023-10-16T10:02:11.192660+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:24.573022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192538+0000","last_change":"2023-10-16T10:02:10.206361+0000","last_active":"2023-10-16T10:02:11.192538+0000","last_peered":"2023-10-16T10:02:11.192538+0000","last_clean":"2023-10-16T10:02:11.192538+0000","last_became_active":"2023-10-16T10:02:10.205750+0000","last_became_peered":"2023-10-16T10:02:10.205750+0000","last_unstale":"2023-10-16T10:02:11.192538+0000","last_undegraded":"2023-10-16T10:02:11.192538+0000","last_fullsized":"2023-10-16T10:02:11.192538+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:44.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352284+0000","last_change":"2023-10-16T10:02:10.203015+0000","last_active":"2023-10-16T10:02:11.352284+0000","last_peered":"2023-10-16T10:02:11.352284+0000","last_clean":"2023-10-16T10:02:11.352284+0000","last_became_active":"2023-10-16T10:02:10.202724+0000","last_became_peered":"2023-10-16T10:02:10.202724+0000","last_unstale":"2023-10-16T10:02:11.352284+0000","last_undegraded":"2023-10-16T10:02:11.352284+0000","last_fullsized":"2023-10-16T10:02:11.352284+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:37.507977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193050+0000","last_change":"2023-10-16T10:02:10.200246+0000","last_active":"2023-10-16T10:02:11.193050+0000","last_peered":"2023-10-16T10:02:11.193050+0000","last_clean":"2023-10-16T10:02:11.193050+0000","last_became_active":"2023-10-16T10:02:10.199979+0000","last_became_peered":"2023-10-16T10:02:10.199979+0000","last_unstale":"2023-10-16T10:02:11.193050+0000","last_undegraded":"2023-10-16T10:02:11.193050+0000","last_fullsized":"2023-10-16T10:02:11.193050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.235661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353139+0000","last_change":"2023-10-16T10:02:10.207624+0000","last_active":"2023-10-16T10:02:11.353139+0000","last_peered":"2023-10-16T10:02:11.353139+0000","last_clean":"2023-10-16T10:02:11.353139+0000","last_became_active":"2023-10-16T10:02:10.207321+0000","last_became_peered":"2023-10-16T10:02:10.207321+0000","last_unstale":"2023-10-16T10:02:11.353139+0000","last_undegraded":"2023-10-16T10:02:11.353139+0000","last_fullsized":"2023-10-16T10:02:11.353139+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352124+0000","last_change":"2023-10-16T10:02:10.208921+0000","last_active":"2023-10-16T10:02:11.352124+0000","last_peered":"2023-10-16T10:02:11.352124+0000","last_clean":"2023-10-16T10:02:11.352124+0000","last_became_active":"2023-10-16T10:02:10.208674+0000","last_became_peered":"2023-10-16T10:02:10.208674+0000","last_unstale":"2023-10-16T10:02:11.352124+0000","last_undegraded":"2023-10-16T10:02:11.352124+0000","last_fullsized":"2023-10-16T10:02:11.352124+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:30.773531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193753+0000","last_change":"2023-10-16T10:02:10.197261+0000","last_active":"2023-10-16T10:02:11.193753+0000","last_peered":"2023-10-16T10:02:11.193753+0000","last_clean":"2023-10-16T10:02:11.193753+0000","last_became_active":"2023-10-16T10:02:10.196997+0000","last_became_peered":"2023-10-16T10:02:10.196997+0000","last_unstale":"2023-10-16T10:02:11.193753+0000","last_undegraded":"2023-10-16T10:02:11.193753+0000","last_fullsized":"2023-10-16T10:02:11.193753+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:42:02.838978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352681+0000","last_change":"2023-10-16T10:02:10.207066+0000","last_active":"2023-10-16T10:02:11.352681+0000","last_peered":"2023-10-16T10:02:11.352681+0000","last_clean":"2023-10-16T10:02:11.352681+0000","last_became_active":"2023-10-16T10:02:10.206591+0000","last_became_peered":"2023-10-16T10:02:10.206591+0000","last_unstale":"2023-10-16T10:02:11.352681+0000","last_undegraded":"2023-10-16T10:02:11.352681+0000","last_fullsized":"2023-10-16T10:02:11.352681+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:56:22.099166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191800+0000","last_change":"2023-10-16T10:02:10.197986+0000","last_active":"2023-10-16T10:02:11.191800+0000","last_peered":"2023-10-16T10:02:11.191800+0000","last_clean":"2023-10-16T10:02:11.191800+0000","last_became_active":"2023-10-16T10:02:10.197744+0000","last_became_peered":"2023-10-16T10:02:10.197744+0000","last_unstale":"2023-10-16T10:02:11.191800+0000","last_undegraded":"2023-10-16T10:02:11.191800+0000","last_fullsized":"2023-10-16T10:02:11.191800+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:30.773531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191641+0000","last_change":"2023-10-16T10:02:10.203745+0000","last_active":"2023-10-16T10:02:11.191641+0000","last_peered":"2023-10-16T10:02:11.191641+0000","last_clean":"2023-10-16T10:02:11.191641+0000","last_became_active":"2023-10-16T10:02:10.203467+0000","last_became_peered":"2023-10-16T10:02:10.203467+0000","last_unstale":"2023-10-16T10:02:11.191641+0000","last_undegraded":"2023-10-16T10:02:11.191641+0000","last_fullsized":"2023-10-16T10:02:11.191641+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:56:22.099166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191511+0000","last_change":"2023-10-16T10:02:10.195395+0000","last_active":"2023-10-16T10:02:11.191511+0000","last_peered":"2023-10-16T10:02:11.191511+0000","last_clean":"2023-10-16T10:02:11.191511+0000","last_became_active":"2023-10-16T10:02:10.195117+0000","last_became_peered":"2023-10-16T10:02:10.195117+0000","last_unstale":"2023-10-16T10:02:11.191511+0000","last_undegraded":"2023-10-16T10:02:11.191511+0000","last_fullsized":"2023-10-16T10:02:11.191511+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:22.363722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352554+0000","last_change":"2023-10-16T10:02:10.204013+0000","last_active":"2023-10-16T10:02:11.352554+0000","last_peered":"2023-10-16T10:02:11.352554+0000","last_clean":"2023-10-16T10:02:11.352554+0000","last_became_active":"2023-10-16T10:02:10.203735+0000","last_became_peered":"2023-10-16T10:02:10.203735+0000","last_unstale":"2023-10-16T10:02:11.352554+0000","last_undegraded":"2023-10-16T10:02:11.352554+0000","last_fullsized":"2023-10-16T10:02:11.352554+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:43.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193038+0000","last_change":"2023-10-16T10:02:10.202592+0000","last_active":"2023-10-16T10:02:11.193038+0000","last_peered":"2023-10-16T10:02:11.193038+0000","last_clean":"2023-10-16T10:02:11.193038+0000","last_became_active":"2023-10-16T10:02:10.202314+0000","last_became_peered":"2023-10-16T10:02:10.202314+0000","last_unstale":"2023-10-16T10:02:11.193038+0000","last_undegraded":"2023-10-16T10:02:11.193038+0000","last_fullsized":"2023-10-16T10:02:11.193038+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352517+0000","last_change":"2023-10-16T10:02:10.206325+0000","last_active":"2023-10-16T10:02:11.352517+0000","last_peered":"2023-10-16T10:02:11.352517+0000","last_clean":"2023-10-16T10:02:11.352517+0000","last_became_active":"2023-10-16T10:02:10.205812+0000","last_became_peered":"2023-10-16T10:02:10.205812+0000","last_unstale":"2023-10-16T10:02:11.352517+0000","last_undegraded":"2023-10-16T10:02:11.352517+0000","last_fullsized":"2023-10-16T10:02:11.352517+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:22:04.943383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193764+0000","last_change":"2023-10-16T10:02:10.193072+0000","last_active":"2023-10-16T10:02:11.193764+0000","last_peered":"2023-10-16T10:02:11.193764+0000","last_clean":"2023-10-16T10:02:11.193764+0000","last_became_active":"2023-10-16T10:02:10.192797+0000","last_became_peered":"2023-10-16T10:02:10.192797+0000","last_unstale":"2023-10-16T10:02:11.193764+0000","last_undegraded":"2023-10-16T10:02:11.193764+0000","last_fullsized":"2023-10-16T10:02:11.193764+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:44.425674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352294+0000","last_change":"2023-10-16T10:02:10.202285+0000","last_active":"2023-10-16T10:02:11.352294+0000","last_peered":"2023-10-16T10:02:11.352294+0000","last_clean":"2023-10-16T10:02:11.352294+0000","last_became_active":"2023-10-16T10:02:10.202004+0000","last_became_peered":"2023-10-16T10:02:10.202004+0000","last_unstale":"2023-10-16T10:02:11.352294+0000","last_undegraded":"2023-10-16T10:02:11.352294+0000","last_fullsized":"2023-10-16T10:02:11.352294+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:58.985349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353359+0000","last_change":"2023-10-16T10:02:10.208486+0000","last_active":"2023-10-16T10:02:11.353359+0000","last_peered":"2023-10-16T10:02:11.353359+0000","last_clean":"2023-10-16T10:02:11.353359+0000","last_became_active":"2023-10-16T10:02:10.208225+0000","last_became_peered":"2023-10-16T10:02:10.208225+0000","last_unstale":"2023-10-16T10:02:11.353359+0000","last_undegraded":"2023-10-16T10:02:11.353359+0000","last_fullsized":"2023-10-16T10:02:11.353359+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.587855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351826+0000","last_change":"2023-10-16T10:02:10.203660+0000","last_active":"2023-10-16T10:02:11.351826+0000","last_peered":"2023-10-16T10:02:11.351826+0000","last_clean":"2023-10-16T10:02:11.351826+0000","last_became_active":"2023-10-16T10:02:10.203383+0000","last_became_peered":"2023-10-16T10:02:10.203383+0000","last_unstale":"2023-10-16T10:02:11.351826+0000","last_undegraded":"2023-10-16T10:02:11.351826+0000","last_fullsized":"2023-10-16T10:02:11.351826+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:42.615762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353178+0000","last_change":"2023-10-16T10:02:10.200856+0000","last_active":"2023-10-16T10:02:11.353178+0000","last_peered":"2023-10-16T10:02:11.353178+0000","last_clean":"2023-10-16T10:02:11.353178+0000","last_became_active":"2023-10-16T10:02:10.198790+0000","last_became_peered":"2023-10-16T10:02:10.198790+0000","last_unstale":"2023-10-16T10:02:11.353178+0000","last_undegraded":"2023-10-16T10:02:11.353178+0000","last_fullsized":"2023-10-16T10:02:11.353178+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.235661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193651+0000","last_change":"2023-10-16T10:02:10.205971+0000","last_active":"2023-10-16T10:02:11.193651+0000","last_peered":"2023-10-16T10:02:11.193651+0000","last_clean":"2023-10-16T10:02:11.193651+0000","last_became_active":"2023-10-16T10:02:10.204772+0000","last_became_peered":"2023-10-16T10:02:10.204772+0000","last_unstale":"2023-10-16T10:02:11.193651+0000","last_undegraded":"2023-10-16T10:02:11.193651+0000","last_fullsized":"2023-10-16T10:02:11.193651+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.587855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193761+0000","last_change":"2023-10-16T10:02:10.210279+0000","last_active":"2023-10-16T10:02:11.193761+0000","last_peered":"2023-10-16T10:02:11.193761+0000","last_clean":"2023-10-16T10:02:11.193761+0000","last_became_active":"2023-10-16T10:02:10.209956+0000","last_became_peered":"2023-10-16T10:02:10.209956+0000","last_unstale":"2023-10-16T10:02:11.193761+0000","last_undegraded":"2023-10-16T10:02:11.193761+0000","last_fullsized":"2023-10-16T10:02:11.193761+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:24.234939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352244+0000","last_change":"2023-10-16T10:02:10.206875+0000","last_active":"2023-10-16T10:02:11.352244+0000","last_peered":"2023-10-16T10:02:11.352244+0000","last_clean":"2023-10-16T10:02:11.352244+0000","last_became_active":"2023-10-16T10:02:10.206594+0000","last_became_peered":"2023-10-16T10:02:10.206594+0000","last_unstale":"2023-10-16T10:02:11.352244+0000","last_undegraded":"2023-10-16T10:02:11.352244+0000","last_fullsized":"2023-10-16T10:02:11.352244+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:36.493038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192344+0000","last_change":"2023-10-16T10:02:10.208786+0000","last_active":"2023-10-16T10:02:11.192344+0000","last_peered":"2023-10-16T10:02:11.192344+0000","last_clean":"2023-10-16T10:02:11.192344+0000","last_became_active":"2023-10-16T10:02:10.208572+0000","last_became_peered":"2023-10-16T10:02:10.208572+0000","last_unstale":"2023-10-16T10:02:11.192344+0000","last_undegraded":"2023-10-16T10:02:11.192344+0000","last_fullsized":"2023-10-16T10:02:11.192344+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:19.739611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352226+0000","last_change":"2023-10-16T10:02:10.200008+0000","last_active":"2023-10-16T10:02:11.352226+0000","last_peered":"2023-10-16T10:02:11.352226+0000","last_clean":"2023-10-16T10:02:11.352226+0000","last_became_active":"2023-10-16T10:02:10.199736+0000","last_became_peered":"2023-10-16T10:02:10.199736+0000","last_unstale":"2023-10-16T10:02:11.352226+0000","last_undegraded":"2023-10-16T10:02:11.352226+0000","last_fullsized":"2023-10-16T10:02:11.352226+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:15:55.364671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193841+0000","last_change":"2023-10-16T10:02:10.205048+0000","last_active":"2023-10-16T10:02:11.193841+0000","last_peered":"2023-10-16T10:02:11.193841+0000","last_clean":"2023-10-16T10:02:11.193841+0000","last_became_active":"2023-10-16T10:02:10.204731+0000","last_became_peered":"2023-10-16T10:02:10.204731+0000","last_unstale":"2023-10-16T10:02:11.193841+0000","last_undegraded":"2023-10-16T10:02:11.193841+0000","last_fullsized":"2023-10-16T10:02:11.193841+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:22:42.003604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192495+0000","last_change":"2023-10-16T10:02:10.199486+0000","last_active":"2023-10-16T10:02:11.192495+0000","last_peered":"2023-10-16T10:02:11.192495+0000","last_clean":"2023-10-16T10:02:11.192495+0000","last_became_active":"2023-10-16T10:02:10.199160+0000","last_became_peered":"2023-10-16T10:02:10.199160+0000","last_unstale":"2023-10-16T10:02:11.192495+0000","last_undegraded":"2023-10-16T10:02:11.192495+0000","last_fullsized":"2023-10-16T10:02:11.192495+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:52.749649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352359+0000","last_change":"2023-10-16T10:02:10.193839+0000","last_active":"2023-10-16T10:02:11.352359+0000","last_peered":"2023-10-16T10:02:11.352359+0000","last_clean":"2023-10-16T10:02:11.352359+0000","last_became_active":"2023-10-16T10:02:10.193445+0000","last_became_peered":"2023-10-16T10:02:10.193445+0000","last_unstale":"2023-10-16T10:02:11.352359+0000","last_undegraded":"2023-10-16T10:02:11.352359+0000","last_fullsized":"2023-10-16T10:02:11.352359+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:34:29.465706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352071+0000","last_change":"2023-10-16T10:02:10.207878+0000","last_active":"2023-10-16T10:02:11.352071+0000","last_peered":"2023-10-16T10:02:11.352071+0000","last_clean":"2023-10-16T10:02:11.352071+0000","last_became_active":"2023-10-16T10:02:10.206185+0000","last_became_peered":"2023-10-16T10:02:10.206185+0000","last_unstale":"2023-10-16T10:02:11.352071+0000","last_undegraded":"2023-10-16T10:02:11.352071+0000","last_fullsized":"2023-10-16T10:02:11.352071+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:07.878862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"125'4","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352903+0000","last_change":"2023-10-16T10:02:09.196805+0000","last_active":"2023-10-16T10:02:11.352903+0000","last_peered":"2023-10-16T10:02:11.352903+0000","last_clean":"2023-10-16T10:02:11.352903+0000","last_became_active":"2023-10-16T10:02:09.196548+0000","last_became_peered":"2023-10-16T10:02:09.196548+0000","last_unstale":"2023-10-16T10:02:11.352903+0000","last_undegraded":"2023-10-16T10:02:11.352903+0000","last_fullsized":"2023-10-16T10:02:11.352903+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:35.363470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352215+0000","last_change":"2023-10-16T10:02:10.202490+0000","last_active":"2023-10-16T10:02:11.352215+0000","last_peered":"2023-10-16T10:02:11.352215+0000","last_clean":"2023-10-16T10:02:11.352215+0000","last_became_active":"2023-10-16T10:02:10.202234+0000","last_became_peered":"2023-10-16T10:02:10.202234+0000","last_unstale":"2023-10-16T10:02:11.352215+0000","last_undegraded":"2023-10-16T10:02:11.352215+0000","last_fullsized":"2023-10-16T10:02:11.352215+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:13.534183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353059+0000","last_change":"2023-10-16T10:02:09.190703+0000","last_active":"2023-10-16T10:02:11.353059+0000","last_peered":"2023-10-16T10:02:11.353059+0000","last_clean":"2023-10-16T10:02:11.353059+0000","last_became_active":"2023-10-16T10:02:09.190435+0000","last_became_peered":"2023-10-16T10:02:09.190435+0000","last_unstale":"2023-10-16T10:02:11.353059+0000","last_undegraded":"2023-10-16T10:02:11.353059+0000","last_fullsized":"2023-10-16T10:02:11.353059+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:42.191292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193581+0000","last_change":"2023-10-16T10:02:10.201930+0000","last_active":"2023-10-16T10:02:11.193581+0000","last_peered":"2023-10-16T10:02:11.193581+0000","last_clean":"2023-10-16T10:02:11.193581+0000","last_became_active":"2023-10-16T10:02:10.201619+0000","last_became_peered":"2023-10-16T10:02:10.201619+0000","last_unstale":"2023-10-16T10:02:11.193581+0000","last_undegraded":"2023-10-16T10:02:11.193581+0000","last_fullsized":"2023-10-16T10:02:11.193581+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.235661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193528+0000","last_change":"2023-10-16T10:02:09.191920+0000","last_active":"2023-10-16T10:02:11.193528+0000","last_peered":"2023-10-16T10:02:11.193528+0000","last_clean":"2023-10-16T10:02:11.193528+0000","last_became_active":"2023-10-16T10:02:09.191606+0000","last_became_peered":"2023-10-16T10:02:09.191606+0000","last_unstale":"2023-10-16T10:02:11.193528+0000","last_undegraded":"2023-10-16T10:02:11.193528+0000","last_fullsized":"2023-10-16T10:02:11.193528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:13.037875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351929+0000","last_change":"2023-10-16T10:02:10.200643+0000","last_active":"2023-10-16T10:02:11.351929+0000","last_peered":"2023-10-16T10:02:11.351929+0000","last_clean":"2023-10-16T10:02:11.351929+0000","last_became_active":"2023-10-16T10:02:10.200326+0000","last_became_peered":"2023-10-16T10:02:10.200326+0000","last_unstale":"2023-10-16T10:02:11.351929+0000","last_undegraded":"2023-10-16T10:02:11.351929+0000","last_fullsized":"2023-10-16T10:02:11.351929+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:56:22.099166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352862+0000","last_change":"2023-10-16T10:02:09.191107+0000","last_active":"2023-10-16T10:02:11.352862+0000","last_peered":"2023-10-16T10:02:11.352862+0000","last_clean":"2023-10-16T10:02:11.352862+0000","last_became_active":"2023-10-16T10:02:09.190681+0000","last_became_peered":"2023-10-16T10:02:09.190681+0000","last_unstale":"2023-10-16T10:02:11.352862+0000","last_undegraded":"2023-10-16T10:02:11.352862+0000","last_fullsized":"2023-10-16T10:02:11.352862+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:05.397655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353173+0000","last_change":"2023-10-16T10:02:10.205939+0000","last_active":"2023-10-16T10:02:11.353173+0000","last_peered":"2023-10-16T10:02:11.353173+0000","last_clean":"2023-10-16T10:02:11.353173+0000","last_became_active":"2023-10-16T10:02:10.205656+0000","last_became_peered":"2023-10-16T10:02:10.205656+0000","last_unstale":"2023-10-16T10:02:11.353173+0000","last_undegraded":"2023-10-16T10:02:11.353173+0000","last_fullsized":"2023-10-16T10:02:11.353173+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:40.039367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191741+0000","last_change":"2023-10-16T10:02:09.193730+0000","last_active":"2023-10-16T10:02:11.191741+0000","last_peered":"2023-10-16T10:02:11.191741+0000","last_clean":"2023-10-16T10:02:11.191741+0000","last_became_active":"2023-10-16T10:02:09.193409+0000","last_became_peered":"2023-10-16T10:02:09.193409+0000","last_unstale":"2023-10-16T10:02:11.191741+0000","last_undegraded":"2023-10-16T10:02:11.191741+0000","last_fullsized":"2023-10-16T10:02:11.191741+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:57.060618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351977+0000","last_change":"2023-10-16T10:02:10.210109+0000","last_active":"2023-10-16T10:02:11.351977+0000","last_peered":"2023-10-16T10:02:11.351977+0000","last_clean":"2023-10-16T10:02:11.351977+0000","last_became_active":"2023-10-16T10:02:10.209805+0000","last_became_peered":"2023-10-16T10:02:10.209805+0000","last_unstale":"2023-10-16T10:02:11.351977+0000","last_undegraded":"2023-10-16T10:02:11.351977+0000","last_fullsized":"2023-10-16T10:02:11.351977+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:40.142426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351830+0000","last_change":"2023-10-16T10:02:09.190793+0000","last_active":"2023-10-16T10:02:11.351830+0000","last_peered":"2023-10-16T10:02:11.351830+0000","last_clean":"2023-10-16T10:02:11.351830+0000","last_became_active":"2023-10-16T10:02:09.190531+0000","last_became_peered":"2023-10-16T10:02:09.190531+0000","last_unstale":"2023-10-16T10:02:11.351830+0000","last_undegraded":"2023-10-16T10:02:11.351830+0000","last_fullsized":"2023-10-16T10:02:11.351830+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:53.236384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193206+0000","last_change":"2023-10-16T10:02:10.206010+0000","last_active":"2023-10-16T10:02:11.193206+0000","last_peered":"2023-10-16T10:02:11.193206+0000","last_clean":"2023-10-16T10:02:11.193206+0000","last_became_active":"2023-10-16T10:02:10.205533+0000","last_became_peered":"2023-10-16T10:02:10.205533+0000","last_unstale":"2023-10-16T10:02:11.193206+0000","last_undegraded":"2023-10-16T10:02:11.193206+0000","last_fullsized":"2023-10-16T10:02:11.193206+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.587855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193431+0000","last_change":"2023-10-16T10:02:09.192785+0000","last_active":"2023-10-16T10:02:11.193431+0000","last_peered":"2023-10-16T10:02:11.193431+0000","last_clean":"2023-10-16T10:02:11.193431+0000","last_became_active":"2023-10-16T10:02:09.192367+0000","last_became_peered":"2023-10-16T10:02:09.192367+0000","last_unstale":"2023-10-16T10:02:11.193431+0000","last_undegraded":"2023-10-16T10:02:11.193431+0000","last_fullsized":"2023-10-16T10:02:11.193431+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:51.745363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352902+0000","last_change":"2023-10-16T10:02:10.193903+0000","last_active":"2023-10-16T10:02:11.352902+0000","last_peered":"2023-10-16T10:02:11.352902+0000","last_clean":"2023-10-16T10:02:11.352902+0000","last_became_active":"2023-10-16T10:02:10.193626+0000","last_became_peered":"2023-10-16T10:02:10.193626+0000","last_unstale":"2023-10-16T10:02:11.352902+0000","last_undegraded":"2023-10-16T10:02:11.352902+0000","last_fullsized":"2023-10-16T10:02:11.352902+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:36.493038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352647+0000","last_change":"2023-10-16T10:02:09.195512+0000","last_active":"2023-10-16T10:02:11.352647+0000","last_peered":"2023-10-16T10:02:11.352647+0000","last_clean":"2023-10-16T10:02:11.352647+0000","last_became_active":"2023-10-16T10:02:09.195327+0000","last_became_peered":"2023-10-16T10:02:09.195327+0000","last_unstale":"2023-10-16T10:02:11.352647+0000","last_undegraded":"2023-10-16T10:02:11.352647+0000","last_fullsized":"2023-10-16T10:02:11.352647+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:37:19.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351765+0000","last_change":"2023-10-16T10:02:10.206240+0000","last_active":"2023-10-16T10:02:11.351765+0000","last_peered":"2023-10-16T10:02:11.351765+0000","last_clean":"2023-10-16T10:02:11.351765+0000","last_became_active":"2023-10-16T10:02:10.205981+0000","last_became_peered":"2023-10-16T10:02:10.205981+0000","last_unstale":"2023-10-16T10:02:11.351765+0000","last_undegraded":"2023-10-16T10:02:11.351765+0000","last_fullsized":"2023-10-16T10:02:11.351765+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:22.643343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193734+0000","last_change":"2023-10-16T10:02:09.191695+0000","last_active":"2023-10-16T10:02:11.193734+0000","last_peered":"2023-10-16T10:02:11.193734+0000","last_clean":"2023-10-16T10:02:11.193734+0000","last_became_active":"2023-10-16T10:02:09.191321+0000","last_became_peered":"2023-10-16T10:02:09.191321+0000","last_unstale":"2023-10-16T10:02:11.193734+0000","last_undegraded":"2023-10-16T10:02:11.193734+0000","last_fullsized":"2023-10-16T10:02:11.193734+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:05.397655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352396+0000","last_change":"2023-10-16T10:02:10.200463+0000","last_active":"2023-10-16T10:02:11.352396+0000","last_peered":"2023-10-16T10:02:11.352396+0000","last_clean":"2023-10-16T10:02:11.352396+0000","last_became_active":"2023-10-16T10:02:10.200171+0000","last_became_peered":"2023-10-16T10:02:10.200171+0000","last_unstale":"2023-10-16T10:02:11.352396+0000","last_undegraded":"2023-10-16T10:02:11.352396+0000","last_fullsized":"2023-10-16T10:02:11.352396+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:24.573022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351797+0000","last_change":"2023-10-16T10:02:09.192472+0000","last_active":"2023-10-16T10:02:11.351797+0000","last_peered":"2023-10-16T10:02:11.351797+0000","last_clean":"2023-10-16T10:02:11.351797+0000","last_became_active":"2023-10-16T10:02:09.192223+0000","last_became_peered":"2023-10-16T10:02:09.192223+0000","last_unstale":"2023-10-16T10:02:11.351797+0000","last_undegraded":"2023-10-16T10:02:11.351797+0000","last_fullsized":"2023-10-16T10:02:11.351797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:17.532553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352066+0000","last_change":"2023-10-16T10:02:10.207736+0000","last_active":"2023-10-16T10:02:11.352066+0000","last_peered":"2023-10-16T10:02:11.352066+0000","last_clean":"2023-10-16T10:02:11.352066+0000","last_became_active":"2023-10-16T10:02:10.207459+0000","last_became_peered":"2023-10-16T10:02:10.207459+0000","last_unstale":"2023-10-16T10:02:11.352066+0000","last_undegraded":"2023-10-16T10:02:11.352066+0000","last_fullsized":"2023-10-16T10:02:11.352066+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:11.005434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"125'3","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191845+0000","last_change":"2023-10-16T10:02:09.184622+0000","last_active":"2023-10-16T10:02:11.191845+0000","last_peered":"2023-10-16T10:02:11.191845+0000","last_clean":"2023-10-16T10:02:11.191845+0000","last_became_active":"2023-10-16T10:02:09.184348+0000","last_became_peered":"2023-10-16T10:02:09.184348+0000","last_unstale":"2023-10-16T10:02:11.191845+0000","last_undegraded":"2023-10-16T10:02:11.191845+0000","last_fullsized":"2023-10-16T10:02:11.191845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:27:10.568486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352988+0000","last_change":"2023-10-16T10:02:10.198334+0000","last_active":"2023-10-16T10:02:11.352988+0000","last_peered":"2023-10-16T10:02:11.352988+0000","last_clean":"2023-10-16T10:02:11.352988+0000","last_became_active":"2023-10-16T10:02:10.197929+0000","last_became_peered":"2023-10-16T10:02:10.197929+0000","last_unstale":"2023-10-16T10:02:11.352988+0000","last_undegraded":"2023-10-16T10:02:11.352988+0000","last_fullsized":"2023-10-16T10:02:11.352988+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:37.304152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"125'6","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192456+0000","last_change":"2023-10-16T10:02:09.193617+0000","last_active":"2023-10-16T10:02:11.192456+0000","last_peered":"2023-10-16T10:02:11.192456+0000","last_clean":"2023-10-16T10:02:11.192456+0000","last_became_active":"2023-10-16T10:02:09.193144+0000","last_became_peered":"2023-10-16T10:02:09.193144+0000","last_unstale":"2023-10-16T10:02:11.192456+0000","last_undegraded":"2023-10-16T10:02:11.192456+0000","last_fullsized":"2023-10-16T10:02:11.192456+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:29.614976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351898+0000","last_change":"2023-10-16T10:02:10.206952+0000","last_active":"2023-10-16T10:02:11.351898+0000","last_peered":"2023-10-16T10:02:11.351898+0000","last_clean":"2023-10-16T10:02:11.351898+0000","last_became_active":"2023-10-16T10:02:10.206612+0000","last_became_peered":"2023-10-16T10:02:10.206612+0000","last_unstale":"2023-10-16T10:02:11.351898+0000","last_undegraded":"2023-10-16T10:02:11.351898+0000","last_fullsized":"2023-10-16T10:02:11.351898+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:22.363722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353088+0000","last_change":"2023-10-16T10:02:09.194325+0000","last_active":"2023-10-16T10:02:11.353088+0000","last_peered":"2023-10-16T10:02:11.353088+0000","last_clean":"2023-10-16T10:02:11.353088+0000","last_became_active":"2023-10-16T10:02:09.193600+0000","last_became_peered":"2023-10-16T10:02:09.193600+0000","last_unstale":"2023-10-16T10:02:11.353088+0000","last_undegraded":"2023-10-16T10:02:11.353088+0000","last_fullsized":"2023-10-16T10:02:11.353088+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:17:15.859650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192787+0000","last_change":"2023-10-16T10:02:10.206644+0000","last_active":"2023-10-16T10:02:11.192787+0000","last_peered":"2023-10-16T10:02:11.192787+0000","last_clean":"2023-10-16T10:02:11.192787+0000","last_became_active":"2023-10-16T10:02:10.205464+0000","last_became_peered":"2023-10-16T10:02:10.205464+0000","last_unstale":"2023-10-16T10:02:11.192787+0000","last_undegraded":"2023-10-16T10:02:11.192787+0000","last_fullsized":"2023-10-16T10:02:11.192787+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.210215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352881+0000","last_change":"2023-10-16T10:02:09.193291+0000","last_active":"2023-10-16T10:02:11.352881+0000","last_peered":"2023-10-16T10:02:11.352881+0000","last_clean":"2023-10-16T10:02:11.352881+0000","last_became_active":"2023-10-16T10:02:09.192984+0000","last_became_peered":"2023-10-16T10:02:09.192984+0000","last_unstale":"2023-10-16T10:02:11.352881+0000","last_undegraded":"2023-10-16T10:02:11.352881+0000","last_fullsized":"2023-10-16T10:02:11.352881+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:41.612597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351904+0000","last_change":"2023-10-16T10:02:10.193385+0000","last_active":"2023-10-16T10:02:11.351904+0000","last_peered":"2023-10-16T10:02:11.351904+0000","last_clean":"2023-10-16T10:02:11.351904+0000","last_became_active":"2023-10-16T10:02:10.193114+0000","last_became_peered":"2023-10-16T10:02:10.193114+0000","last_unstale":"2023-10-16T10:02:11.351904+0000","last_undegraded":"2023-10-16T10:02:11.351904+0000","last_fullsized":"2023-10-16T10:02:11.351904+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:10.335349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352187+0000","last_change":"2023-10-16T10:02:09.191473+0000","last_active":"2023-10-16T10:02:11.352187+0000","last_peered":"2023-10-16T10:02:11.352187+0000","last_clean":"2023-10-16T10:02:11.352187+0000","last_became_active":"2023-10-16T10:02:09.191193+0000","last_became_peered":"2023-10-16T10:02:09.191193+0000","last_unstale":"2023-10-16T10:02:11.352187+0000","last_undegraded":"2023-10-16T10:02:11.352187+0000","last_fullsized":"2023-10-16T10:02:11.352187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:02.731436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352663+0000","last_change":"2023-10-16T10:02:10.203916+0000","last_active":"2023-10-16T10:02:11.352663+0000","last_peered":"2023-10-16T10:02:11.352663+0000","last_clean":"2023-10-16T10:02:11.352663+0000","last_became_active":"2023-10-16T10:02:10.203649+0000","last_became_peered":"2023-10-16T10:02:10.203649+0000","last_unstale":"2023-10-16T10:02:11.352663+0000","last_undegraded":"2023-10-16T10:02:11.352663+0000","last_fullsized":"2023-10-16T10:02:11.352663+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:04.547044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"125'8","reported_seq":20,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193921+0000","last_change":"2023-10-16T10:02:09.194873+0000","last_active":"2023-10-16T10:02:11.193921+0000","last_peered":"2023-10-16T10:02:11.193921+0000","last_clean":"2023-10-16T10:02:11.193921+0000","last_became_active":"2023-10-16T10:02:09.194584+0000","last_became_peered":"2023-10-16T10:02:09.194584+0000","last_unstale":"2023-10-16T10:02:11.193921+0000","last_undegraded":"2023-10-16T10:02:11.193921+0000","last_fullsized":"2023-10-16T10:02:11.193921+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:18.735326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352430+0000","last_change":"2023-10-16T10:02:10.199635+0000","last_active":"2023-10-16T10:02:11.352430+0000","last_peered":"2023-10-16T10:02:11.352430+0000","last_clean":"2023-10-16T10:02:11.352430+0000","last_became_active":"2023-10-16T10:02:10.199324+0000","last_became_peered":"2023-10-16T10:02:10.199324+0000","last_unstale":"2023-10-16T10:02:11.352430+0000","last_undegraded":"2023-10-16T10:02:11.352430+0000","last_fullsized":"2023-10-16T10:02:11.352430+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:19:10.442016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193158+0000","last_change":"2023-10-16T10:02:09.194317+0000","last_active":"2023-10-16T10:02:11.193158+0000","last_peered":"2023-10-16T10:02:11.193158+0000","last_clean":"2023-10-16T10:02:11.193158+0000","last_became_active":"2023-10-16T10:02:09.194035+0000","last_became_peered":"2023-10-16T10:02:09.194035+0000","last_unstale":"2023-10-16T10:02:11.193158+0000","last_undegraded":"2023-10-16T10:02:11.193158+0000","last_fullsized":"2023-10-16T10:02:11.193158+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:13.037875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192515+0000","last_change":"2023-10-16T10:02:10.200913+0000","last_active":"2023-10-16T10:02:11.192515+0000","last_peered":"2023-10-16T10:02:11.192515+0000","last_clean":"2023-10-16T10:02:11.192515+0000","last_became_active":"2023-10-16T10:02:10.200502+0000","last_became_peered":"2023-10-16T10:02:10.200502+0000","last_unstale":"2023-10-16T10:02:11.192515+0000","last_undegraded":"2023-10-16T10:02:11.192515+0000","last_fullsized":"2023-10-16T10:02:11.192515+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:17:16.863936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"125'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192042+0000","last_change":"2023-10-16T10:02:09.187874+0000","last_active":"2023-10-16T10:02:11.192042+0000","last_peered":"2023-10-16T10:02:11.192042+0000","last_clean":"2023-10-16T10:02:11.192042+0000","last_became_active":"2023-10-16T10:02:09.187542+0000","last_became_peered":"2023-10-16T10:02:09.187542+0000","last_unstale":"2023-10-16T10:02:11.192042+0000","last_undegraded":"2023-10-16T10:02:11.192042+0000","last_fullsized":"2023-10-16T10:02:11.192042+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:17.532553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352836+0000","last_change":"2023-10-16T10:02:10.200953+0000","last_active":"2023-10-16T10:02:11.352836+0000","last_peered":"2023-10-16T10:02:11.352836+0000","last_clean":"2023-10-16T10:02:11.352836+0000","last_became_active":"2023-10-16T10:02:10.200539+0000","last_became_peered":"2023-10-16T10:02:10.200539+0000","last_unstale":"2023-10-16T10:02:11.352836+0000","last_undegraded":"2023-10-16T10:02:11.352836+0000","last_fullsized":"2023-10-16T10:02:11.352836+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.210215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351929+0000","last_change":"2023-10-16T10:02:09.196786+0000","last_active":"2023-10-16T10:02:11.351929+0000","last_peered":"2023-10-16T10:02:11.351929+0000","last_clean":"2023-10-16T10:02:11.351929+0000","last_became_active":"2023-10-16T10:02:09.196485+0000","last_became_peered":"2023-10-16T10:02:09.196485+0000","last_unstale":"2023-10-16T10:02:11.351929+0000","last_undegraded":"2023-10-16T10:02:11.351929+0000","last_fullsized":"2023-10-16T10:02:11.351929+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:17.532553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351876+0000","last_change":"2023-10-16T10:02:10.196692+0000","last_active":"2023-10-16T10:02:11.351876+0000","last_peered":"2023-10-16T10:02:11.351876+0000","last_clean":"2023-10-16T10:02:11.351876+0000","last_became_active":"2023-10-16T10:02:10.196247+0000","last_became_peered":"2023-10-16T10:02:10.196247+0000","last_unstale":"2023-10-16T10:02:11.351876+0000","last_undegraded":"2023-10-16T10:02:11.351876+0000","last_fullsized":"2023-10-16T10:02:11.351876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:29:00.793954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352281+0000","last_change":"2023-10-16T10:02:09.189535+0000","last_active":"2023-10-16T10:02:11.352281+0000","last_peered":"2023-10-16T10:02:11.352281+0000","last_clean":"2023-10-16T10:02:11.352281+0000","last_became_active":"2023-10-16T10:02:09.189267+0000","last_became_peered":"2023-10-16T10:02:09.189267+0000","last_unstale":"2023-10-16T10:02:11.352281+0000","last_undegraded":"2023-10-16T10:02:11.352281+0000","last_fullsized":"2023-10-16T10:02:11.352281+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:35:06.124697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352110+0000","last_change":"2023-10-16T10:02:10.198667+0000","last_active":"2023-10-16T10:02:11.352110+0000","last_peered":"2023-10-16T10:02:11.352110+0000","last_clean":"2023-10-16T10:02:11.352110+0000","last_became_active":"2023-10-16T10:02:10.198400+0000","last_became_peered":"2023-10-16T10:02:10.198400+0000","last_unstale":"2023-10-16T10:02:11.352110+0000","last_undegraded":"2023-10-16T10:02:11.352110+0000","last_fullsized":"2023-10-16T10:02:11.352110+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.789284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352123+0000","last_change":"2023-10-16T10:02:09.195780+0000","last_active":"2023-10-16T10:02:11.352123+0000","last_peered":"2023-10-16T10:02:11.352123+0000","last_clean":"2023-10-16T10:02:11.352123+0000","last_became_active":"2023-10-16T10:02:09.195524+0000","last_became_peered":"2023-10-16T10:02:09.195524+0000","last_unstale":"2023-10-16T10:02:11.352123+0000","last_undegraded":"2023-10-16T10:02:11.352123+0000","last_fullsized":"2023-10-16T10:02:11.352123+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:22.279374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353097+0000","last_change":"2023-10-16T10:02:10.211311+0000","last_active":"2023-10-16T10:02:11.353097+0000","last_peered":"2023-10-16T10:02:11.353097+0000","last_clean":"2023-10-16T10:02:11.353097+0000","last_became_active":"2023-10-16T10:02:10.211029+0000","last_became_peered":"2023-10-16T10:02:10.211029+0000","last_unstale":"2023-10-16T10:02:11.353097+0000","last_undegraded":"2023-10-16T10:02:11.353097+0000","last_fullsized":"2023-10-16T10:02:11.353097+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:35.557520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192394+0000","last_change":"2023-10-16T10:02:09.196617+0000","last_active":"2023-10-16T10:02:11.192394+0000","last_peered":"2023-10-16T10:02:11.192394+0000","last_clean":"2023-10-16T10:02:11.192394+0000","last_became_active":"2023-10-16T10:02:09.196338+0000","last_became_peered":"2023-10-16T10:02:09.196338+0000","last_unstale":"2023-10-16T10:02:11.192394+0000","last_undegraded":"2023-10-16T10:02:11.192394+0000","last_fullsized":"2023-10-16T10:02:11.192394+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:25.426647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353367+0000","last_change":"2023-10-16T10:02:10.202288+0000","last_active":"2023-10-16T10:02:11.353367+0000","last_peered":"2023-10-16T10:02:11.353367+0000","last_clean":"2023-10-16T10:02:11.353367+0000","last_became_active":"2023-10-16T10:02:10.201981+0000","last_became_peered":"2023-10-16T10:02:10.201981+0000","last_unstale":"2023-10-16T10:02:11.353367+0000","last_undegraded":"2023-10-16T10:02:11.353367+0000","last_fullsized":"2023-10-16T10:02:11.353367+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:16.822411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352512+0000","last_change":"2023-10-16T10:02:09.196428+0000","last_active":"2023-10-16T10:02:11.352512+0000","last_peered":"2023-10-16T10:02:11.352512+0000","last_clean":"2023-10-16T10:02:11.352512+0000","last_became_active":"2023-10-16T10:02:09.196167+0000","last_became_peered":"2023-10-16T10:02:09.196167+0000","last_unstale":"2023-10-16T10:02:11.352512+0000","last_undegraded":"2023-10-16T10:02:11.352512+0000","last_fullsized":"2023-10-16T10:02:11.352512+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:05.663564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351720+0000","last_change":"2023-10-16T10:02:10.196276+0000","last_active":"2023-10-16T10:02:11.351720+0000","last_peered":"2023-10-16T10:02:11.351720+0000","last_clean":"2023-10-16T10:02:11.351720+0000","last_became_active":"2023-10-16T10:02:10.195993+0000","last_became_peered":"2023-10-16T10:02:10.195993+0000","last_unstale":"2023-10-16T10:02:11.351720+0000","last_undegraded":"2023-10-16T10:02:11.351720+0000","last_fullsized":"2023-10-16T10:02:11.351720+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:58.649854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352940+0000","last_change":"2023-10-16T10:02:09.192226+0000","last_active":"2023-10-16T10:02:11.352940+0000","last_peered":"2023-10-16T10:02:11.352940+0000","last_clean":"2023-10-16T10:02:11.352940+0000","last_became_active":"2023-10-16T10:02:09.191924+0000","last_became_peered":"2023-10-16T10:02:09.191924+0000","last_unstale":"2023-10-16T10:02:11.352940+0000","last_undegraded":"2023-10-16T10:02:11.352940+0000","last_fullsized":"2023-10-16T10:02:11.352940+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:18.636295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193258+0000","last_change":"2023-10-16T10:02:10.196683+0000","last_active":"2023-10-16T10:02:11.193258+0000","last_peered":"2023-10-16T10:02:11.193258+0000","last_clean":"2023-10-16T10:02:11.193258+0000","last_became_active":"2023-10-16T10:02:10.196401+0000","last_became_peered":"2023-10-16T10:02:10.196401+0000","last_unstale":"2023-10-16T10:02:11.193258+0000","last_undegraded":"2023-10-16T10:02:11.193258+0000","last_fullsized":"2023-10-16T10:02:11.193258+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351882+0000","last_change":"2023-10-16T10:02:09.191233+0000","last_active":"2023-10-16T10:02:11.351882+0000","last_peered":"2023-10-16T10:02:11.351882+0000","last_clean":"2023-10-16T10:02:11.351882+0000","last_became_active":"2023-10-16T10:02:09.190963+0000","last_became_peered":"2023-10-16T10:02:09.190963+0000","last_unstale":"2023-10-16T10:02:11.351882+0000","last_undegraded":"2023-10-16T10:02:11.351882+0000","last_fullsized":"2023-10-16T10:02:11.351882+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:29.769245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352735+0000","last_change":"2023-10-16T10:02:10.207400+0000","last_active":"2023-10-16T10:02:11.352735+0000","last_peered":"2023-10-16T10:02:11.352735+0000","last_clean":"2023-10-16T10:02:11.352735+0000","last_became_active":"2023-10-16T10:02:10.206418+0000","last_became_peered":"2023-10-16T10:02:10.206418+0000","last_unstale":"2023-10-16T10:02:11.352735+0000","last_undegraded":"2023-10-16T10:02:11.352735+0000","last_fullsized":"2023-10-16T10:02:11.352735+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:30.773531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192728+0000","last_change":"2023-10-16T10:02:09.196020+0000","last_active":"2023-10-16T10:02:11.192728+0000","last_peered":"2023-10-16T10:02:11.192728+0000","last_clean":"2023-10-16T10:02:11.192728+0000","last_became_active":"2023-10-16T10:02:09.195679+0000","last_became_peered":"2023-10-16T10:02:09.195679+0000","last_unstale":"2023-10-16T10:02:11.192728+0000","last_undegraded":"2023-10-16T10:02:11.192728+0000","last_fullsized":"2023-10-16T10:02:11.192728+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:46.209694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351898+0000","last_change":"2023-10-16T10:02:10.202892+0000","last_active":"2023-10-16T10:02:11.351898+0000","last_peered":"2023-10-16T10:02:11.351898+0000","last_clean":"2023-10-16T10:02:11.351898+0000","last_became_active":"2023-10-16T10:02:10.202548+0000","last_became_peered":"2023-10-16T10:02:10.202548+0000","last_unstale":"2023-10-16T10:02:11.351898+0000","last_undegraded":"2023-10-16T10:02:11.351898+0000","last_fullsized":"2023-10-16T10:02:11.351898+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:37:20.487393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352933+0000","last_change":"2023-10-16T10:02:09.192658+0000","last_active":"2023-10-16T10:02:11.352933+0000","last_peered":"2023-10-16T10:02:11.352933+0000","last_clean":"2023-10-16T10:02:11.352933+0000","last_became_active":"2023-10-16T10:02:09.192381+0000","last_became_peered":"2023-10-16T10:02:09.192381+0000","last_unstale":"2023-10-16T10:02:11.352933+0000","last_undegraded":"2023-10-16T10:02:11.352933+0000","last_fullsized":"2023-10-16T10:02:11.352933+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:18.735326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352062+0000","last_change":"2023-10-16T10:02:10.207705+0000","last_active":"2023-10-16T10:02:11.352062+0000","last_peered":"2023-10-16T10:02:11.352062+0000","last_clean":"2023-10-16T10:02:11.352062+0000","last_became_active":"2023-10-16T10:02:10.207425+0000","last_became_peered":"2023-10-16T10:02:10.207425+0000","last_unstale":"2023-10-16T10:02:11.352062+0000","last_undegraded":"2023-10-16T10:02:11.352062+0000","last_fullsized":"2023-10-16T10:02:11.352062+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:28.643280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.193587+0000","last_change":"2023-10-16T10:02:09.193643+0000","last_active":"2023-10-16T10:02:11.193587+0000","last_peered":"2023-10-16T10:02:11.193587+0000","last_clean":"2023-10-16T10:02:11.193587+0000","last_became_active":"2023-10-16T10:02:09.193334+0000","last_became_peered":"2023-10-16T10:02:09.193334+0000","last_unstale":"2023-10-16T10:02:11.193587+0000","last_undegraded":"2023-10-16T10:02:11.193587+0000","last_fullsized":"2023-10-16T10:02:11.193587+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:52.205929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352155+0000","last_change":"2023-10-16T10:02:10.204803+0000","last_active":"2023-10-16T10:02:11.352155+0000","last_peered":"2023-10-16T10:02:11.352155+0000","last_clean":"2023-10-16T10:02:11.352155+0000","last_became_active":"2023-10-16T10:02:10.204521+0000","last_became_peered":"2023-10-16T10:02:10.204521+0000","last_unstale":"2023-10-16T10:02:11.352155+0000","last_undegraded":"2023-10-16T10:02:11.352155+0000","last_fullsized":"2023-10-16T10:02:11.352155+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:12.773991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352137+0000","last_change":"2023-10-16T10:02:09.192983+0000","last_active":"2023-10-16T10:02:11.352137+0000","last_peered":"2023-10-16T10:02:11.352137+0000","last_clean":"2023-10-16T10:02:11.352137+0000","last_became_active":"2023-10-16T10:02:09.191246+0000","last_became_peered":"2023-10-16T10:02:09.191246+0000","last_unstale":"2023-10-16T10:02:11.352137+0000","last_undegraded":"2023-10-16T10:02:11.352137+0000","last_fullsized":"2023-10-16T10:02:11.352137+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:56.170139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.191954+0000","last_change":"2023-10-16T10:02:10.204350+0000","last_active":"2023-10-16T10:02:11.191954+0000","last_peered":"2023-10-16T10:02:11.191954+0000","last_clean":"2023-10-16T10:02:11.191954+0000","last_became_active":"2023-10-16T10:02:10.204025+0000","last_became_peered":"2023-10-16T10:02:10.204025+0000","last_unstale":"2023-10-16T10:02:11.191954+0000","last_undegraded":"2023-10-16T10:02:11.191954+0000","last_fullsized":"2023-10-16T10:02:11.191954+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:36.493038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.192601+0000","last_change":"2023-10-16T10:02:09.196040+0000","last_active":"2023-10-16T10:02:11.192601+0000","last_peered":"2023-10-16T10:02:11.192601+0000","last_clean":"2023-10-16T10:02:11.192601+0000","last_became_active":"2023-10-16T10:02:09.195604+0000","last_became_peered":"2023-10-16T10:02:09.195604+0000","last_unstale":"2023-10-16T10:02:11.192601+0000","last_undegraded":"2023-10-16T10:02:11.192601+0000","last_fullsized":"2023-10-16T10:02:11.192601+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:05.397655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352340+0000","last_change":"2023-10-16T10:02:10.201729+0000","last_active":"2023-10-16T10:02:11.352340+0000","last_peered":"2023-10-16T10:02:11.352340+0000","last_clean":"2023-10-16T10:02:11.352340+0000","last_became_active":"2023-10-16T10:02:10.201463+0000","last_became_peered":"2023-10-16T10:02:10.201463+0000","last_unstale":"2023-10-16T10:02:11.352340+0000","last_undegraded":"2023-10-16T10:02:11.352340+0000","last_fullsized":"2023-10-16T10:02:11.352340+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:35.557520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.352587+0000","last_change":"2023-10-16T10:02:09.190819+0000","last_active":"2023-10-16T10:02:11.352587+0000","last_peered":"2023-10-16T10:02:11.352587+0000","last_clean":"2023-10-16T10:02:11.352587+0000","last_became_active":"2023-10-16T10:02:09.190349+0000","last_became_peered":"2023-10-16T10:02:09.190349+0000","last_unstale":"2023-10-16T10:02:11.352587+0000","last_undegraded":"2023-10-16T10:02:11.352587+0000","last_fullsized":"2023-10-16T10:02:11.352587+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:19.691248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.351799+0000","last_change":"2023-10-16T10:02:10.203732+0000","last_active":"2023-10-16T10:02:11.351799+0000","last_peered":"2023-10-16T10:02:11.351799+0000","last_clean":"2023-10-16T10:02:11.351799+0000","last_became_active":"2023-10-16T10:02:10.203427+0000","last_became_peered":"2023-10-16T10:02:10.203427+0000","last_unstale":"2023-10-16T10:02:11.351799+0000","last_undegraded":"2023-10-16T10:02:11.351799+0000","last_fullsized":"2023-10-16T10:02:11.351799+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:09.170802+0000","last_clean_scrub_stamp":"2023-10-16T10:02:09.170802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.616882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"125'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T10:02:11.353066+0000","last_change":"2023-10-16T10:02:09.193873+0000","last_active":"2023-10-16T10:02:11.353066+0000","last_peered":"2023-10-16T10:02:11.353066+0000","last_clean":"2023-10-16T10:02:11.353066+0000","last_became_active":"2023-10-16T10:02:09.193576+0000","last_became_peered":"2023-10-16T10:02:09.193576+0000","last_unstale":"2023-10-16T10:02:11.353066+0000","last_undegraded":"2023-10-16T10:02:11.353066+0000","last_fullsized":"2023-10-16T10:02:11.353066+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:02:08.166516+0000","last_clean_scrub_stamp":"2023-10-16T10:02:08.166516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:28:18.814767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"116'205","reported_seq":583,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-16T10:02:10.190584+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:02:10.190584+0000","last_peered":"2023-10-16T10:02:10.190584+0000","last_clean":"2023-10-16T10:02:10.190584+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:02:10.190584+0000","last_undegraded":"2023-10-16T10:02:10.190584+0000","last_fullsized":"2023-10-16T10:02:10.190584+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59668,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312172,"statfs":{"total":96636764160,"available":96575664128,"internally_reserved":0,"allocated":2162688,"data_stored":1404660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54300000000000004,"15min":0.55700000000000005},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.219},"max":{"1min":0.753,"5min":0.84399999999999997,"15min":1.2150000000000001},"last":0.747},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.254},"max":{"1min":0.76900000000000002,"5min":1.911,"15min":1.911},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58199999999999996,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79200000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.80500000000000005,"5min":0.80800000000000005,"15min":1.1279999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.45300000000000001,"5min":0.30299999999999999,"15min":0.27700000000000002},"max":{"1min":0.78300000000000003,"5min":3.3820000000000001,"15min":3.3820000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.54400000000000004,"5min":0.35899999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":3.1640000000000001,"15min":3.1640000000000001},"last":0.80400000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.46500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82199999999999995,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.49399999999999999,"5min":0.41099999999999998,"15min":0.34999999999999998},"max":{"1min":0.84799999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.81899999999999995}]},{"osd":4,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.499,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65000000000000002,"5min":1.8220000000000001,"15min":1.9550000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55800000000000005,"15min":0.52300000000000002},"min":{"1min":0.39800000000000002,"5min":0.315,"15min":0.27200000000000002},"max":{"1min":0.66800000000000004,"5min":3.8149999999999999,"15min":3.8149999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.40000000000000002,"5min":0.33200000000000002,"15min":0.28199999999999997},"max":{"1min":0.68600000000000005,"5min":0.78600000000000003,"15min":0.84099999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.308},"max":{"1min":0.76200000000000001,"5min":1.6619999999999999,"15min":2.0150000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":1.7110000000000001,"15min":1.9870000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.44,"5min":0.31,"15min":0.31},"max":{"1min":0.751,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.76500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1581056,"data_stored":816348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56299999999999994,"15min":0.58399999999999996},"min":{"1min":0.45200000000000001,"5min":0.29699999999999999,"15min":0.223},"max":{"1min":0.89100000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.249},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.82099999999999995},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.59099999999999997},"min":{"1min":0.32500000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77100000000000002,"5min":0.83699999999999997,"15min":0.90000000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.57299999999999995,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.33000000000000002,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.39200000000000002,"5min":0.28799999999999998,"15min":0.27100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":1.448},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.60599999999999998},"min":{"1min":0.47799999999999998,"5min":0.35899999999999999,"15min":0.35799999999999998},"max":{"1min":0.72599999999999998,"5min":0.874,"15min":1.708},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57299999999999995,"15min":0.58999999999999997},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79900000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.59199999999999997},"min":{"1min":0.436,"5min":0.36899999999999999,"15min":0.315},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65800000000000003}]},{"osd":4,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.47999999999999998},"min":{"1min":0.33400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.76200000000000001,"5min":0.77400000000000002,"15min":0.80600000000000005},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70999999999999996}]},{"osd":5,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.504,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.69099999999999995,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39600000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Oct 16 10:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56699999999999995,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.86499999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.57199999999999995},"min":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.94599999999999995},"last":0.79200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36860,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334980,"statfs":{"total":96636764160,"available":96599019520,"internally_reserved":0,"allocated":1552384,"data_stored":807122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38100000000000001,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.247,"15min":0.188},"max":{"1min":0.54000000000000004,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.23200000000000001},"max":{"1min":0.56399999999999995,"5min":0.67900000000000005,"15min":0.70199999999999996},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42799999999999999,"15min":0.40999999999999998},"min":{"1min":0.26300000000000001,"5min":0.25900000000000001,"15min":0.20799999999999999},"max":{"1min":0.61699999999999999,"5min":4.923,"15min":4.923},"last":0.252},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.432,"15min":0.435},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65800000000000003,"5min":1.712,"15min":1.712},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59399999999999997,"5min":1.984,"15min":1.984},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.42499999999999999,"15min":0.432},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63700000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59799999999999998,"15min":0.628},"min":{"1min":0.499,"5min":0.39900000000000002,"15min":0.377},"max":{"1min":0.87,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.34399999999999997},"max":{"1min":1.014,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.41799999999999998,"5min":0.39300000000000002,"15min":0.36599999999999999},"max":{"1min":0.96299999999999997,"5min":4.9749999999999996,"15min":4.9749999999999996},"last":0.504},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65700000000000003,"15min":0.67400000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40500000000000003},"max":{"1min":1.054,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.61799999999999999,"15min":0.64100000000000001},"min":{"1min":0.48599999999999999,"5min":0.42299999999999999,"15min":0.38200000000000001},"max":{"1min":0.91100000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.44},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68100000000000005,"15min":0.66000000000000003},"min":{"1min":0.47599999999999998,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":1.0449999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.53400000000000003}]},{"osd":7,"last update":"Mon Oct 16 10:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63600000000000001,"15min":0.66100000000000003},"min":{"1min":0.46700000000000003,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96599999999999997,"5min":1.744,"15min":1.744},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.624,"15min":0.65000000000000002},"min":{"1min":0.53300000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91400000000000003},"last":0.48199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575073,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37448,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334392,"statfs":{"total":96636764160,"available":96598417408,"internally_reserved":0,"allocated":2158592,"data_stored":1399158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.38300000000000001},"min":{"1min":0.254,"5min":0.222,"15min":0.192},"max":{"1min":0.40799999999999997,"5min":0.66800000000000004,"15min":1.2110000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.26500000000000001},"max":{"1min":0.63,"5min":0.70999999999999996,"15min":1.3169999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.42099999999999999},"min":{"1min":0.314,"5min":0.251,"15min":0.20599999999999999},"max":{"1min":0.498,"5min":0.57899999999999996,"15min":0.70099999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.44,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":0.54900000000000004,"5min":0.78100000000000003,"15min":0.79900000000000004},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39900000000000002,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":0.53800000000000003,"5min":0.60299999999999998,"15min":0.83399999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44800000000000001,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53100000000000003,"5min":0.68899999999999995,"15min":1.1859999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.621},"min":{"1min":0.505,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.81799999999999995,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.40999999999999998,"15min":0.32100000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":1.1659999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.53000000000000003,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":0.81599999999999995,"5min":0.94999999999999996,"15min":0.998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.59699999999999998,"15min":0.63},"min":{"1min":0.436,"5min":0.40400000000000003,"15min":0.311},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.93000000000000005},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63300000000000001,"15min":0.66300000000000003},"min":{"1min":0.51000000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.2410000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63800000000000001,"15min":0.67000000000000004},"min":{"1min":0.46600000000000003,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.877,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.69599999999999995}]},{"osd":7,"last update":"Mon Oct 16 10:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.316},"max":{"1min":0.78800000000000003,"5min":1.014,"15min":1.02},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.61399999999999999,"15min":0.64400000000000002},"min":{"1min":0.47799999999999998,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.79800000000000004,"5min":0.876,"15min":0.90900000000000003},"last":0.67800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575072,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1564672,"data_stored":814166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.379},"min":{"1min":0.251,"5min":0.217,"15min":0.217},"max":{"1min":0.57099999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.316},{"interface":"front","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.67500000000000004},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.309,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56000000000000005,"5min":0.68000000000000005,"15min":0.754},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.38,"15min":0.39800000000000002},"min":{"1min":0.28100000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59599999999999997,"5min":0.624,"15min":0.624},"last":0.379}]},{"osd":3,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40500000000000003,"15min":0.41799999999999998},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.23499999999999999},"max":{"1min":0.54100000000000004,"5min":0.57599999999999996,"15min":0.63500000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.45400000000000001,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65000000000000002,"5min":0.71599999999999997,"15min":0.79800000000000004},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.63900000000000001},"min":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84199999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65100000000000002},"min":{"1min":0.48899999999999999,"5min":0.38600000000000001,"15min":0.33400000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63300000000000001}]},{"osd":5,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84999999999999998,"5min":1.0249999999999999,"15min":1.103},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.624,"15min":0.65200000000000002},"min":{"1min":0.44400000000000001,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.46000000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82899999999999996,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.67600000000000005},"min":{"1min":0.42599999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91600000000000004,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.65500000000000003,"15min":0.68000000000000005},"min":{"1min":0.52100000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64700000000000002,"15min":0.66700000000000004},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.371},"max":{"1min":0.95899999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575072,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":807670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.36899999999999999},"min":{"1min":0.25600000000000001,"5min":0.20999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.37,"15min":0.39200000000000002},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.50800000000000001,"5min":0.53600000000000003,"15min":0.65400000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.42699999999999999},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.50900000000000001,"5min":0.60899999999999999,"15min":0.67500000000000004},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44,"15min":0.46500000000000002},"min":{"1min":0.36299999999999999,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":0.96099999999999997,"15min":1.0760000000000001},"last":0.251},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47099999999999997,"15min":0.48799999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.28699999999999998},"max":{"1min":0.61399999999999999,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.33000000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58699999999999997,"15min":0.621},"min":{"1min":0.498,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.84599999999999997,"5min":0.90300000000000002,"15min":0.96499999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.63600000000000001},"min":{"1min":0.46999999999999997,"5min":0.38900000000000001,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.432}]},{"osd":5,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.63700000000000001},"min":{"1min":0.49199999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.51800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80200000000000005,"5min":0.873,"15min":1.036},"last":0.41699999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60799999999999998,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.39600000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.88100000000000001,"15min":0.98799999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.60199999999999998,"15min":0.65200000000000002},"min":{"1min":0.45400000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78400000000000003,"5min":0.92600000000000005,"15min":1.0589999999999999},"last":0.46899999999999997}]},{"osd":7,"last update":"Mon Oct 16 10:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.66900000000000004},"min":{"1min":0.53500000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84999999999999998,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60699999999999998,"15min":0.65100000000000002},"min":{"1min":0.53000000000000003,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.79200000000000004,"5min":0.99399999999999999,"15min":0.999},"last":0.48699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575072,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1568768,"data_stored":814224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.82099999999999995,"15min":1.0269999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.59099999999999997},"min":{"1min":0.498,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.77000000000000002,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.73899999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55100000000000005,"15min":0.58699999999999997},"min":{"1min":0.45300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.71899999999999997,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57299999999999995,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.307,"15min":0.28399999999999997},"max":{"1min":0.753,"5min":0.85199999999999998,"15min":0.88300000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.61699999999999999},"min":{"1min":0.38600000000000001,"5min":0.379,"15min":0.32500000000000001},"max":{"1min":0.745,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.47899999999999998}]},{"osd":3,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.433,"5min":0.313,"15min":0.313},"max":{"1min":0.73199999999999998,"5min":0.77700000000000002,"15min":0.86099999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.52900000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71199999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.42999999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71399999999999997,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.44500000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72899999999999998,"5min":0.79300000000000004,"15min":0.96599999999999997},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.79100000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.47299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78100000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.44600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.83099999999999996,"15min":0.91100000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.50800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81599999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575072,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50756,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321084,"statfs":{"total":96636764160,"available":96584790016,"internally_reserved":0,"allocated":1560576,"data_stored":814132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58799999999999997,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.28599999999999998},"max":{"1min":0.78000000000000003,"5min":0.96199999999999997,"15min":1.792},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.41399999999999998,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.7090000000000001},"last":0.41399999999999998}]},{"osd":1,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55400000000000005,"15min":0.57899999999999996},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.749,"5min":0.83099999999999996,"15min":0.875},"last":0.437},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55500000000000005,"15min":0.57799999999999996},"min":{"1min":0.39000000000000001,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":0.79300000000000004,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.38900000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.25},"max":{"1min":0.80800000000000005,"5min":1.496,"15min":1.496},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57899999999999996,"15min":0.59799999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":1.5209999999999999,"15min":2.069},"last":0.497}]},{"osd":3,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55700000000000005,"15min":0.57299999999999995},"min":{"1min":0.46000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.89100000000000001,"15min":1.7689999999999999},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.28399999999999997,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.73799999999999999,"15min":0.84199999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.318,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":0.70999999999999996,"5min":0.749,"15min":0.86899999999999999},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.374,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.39500000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.68400000000000005,"5min":0.76900000000000002,"15min":0.88200000000000001},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Oct 16 10:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.25900000000000001},"max":{"1min":0.746,"5min":0.82399999999999995,"15min":0.90300000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.46500000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.88500000000000001},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:02:14.710 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T10:02:14.710 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T10:02:14.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.870+0000 7f5044412700 1 Processor -- start 2023-10-16T10:02:14.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.870+0000 7f5044412700 1 -- start start 2023-10-16T10:02:14.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f5044412700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f503c06b8e0 0x7f503c06bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f5044412700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c160110 con 0x7f503c15b4b0 2023-10-16T10:02:14.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f5044412700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c15f570 con 0x7f503c072150 2023-10-16T10:02:14.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f5044412700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503c06c2b0 con 0x7f503c06b8e0 2023-10-16T10:02:14.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f50429af700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f503c06b8e0 0x7f503c06bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f50421ae700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f503c15b4b0 0x7f503c15fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56888/0 (socket says 172.21.15.158:56888) 2023-10-16T10:02:14.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f50429af700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f503c06b8e0 0x7f503c06bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56716/0 (socket says 172.21.15.158:56716) 2023-10-16T10:02:14.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.871+0000 7f50421ae700 1 -- 172.21.15.158:0/3416944933 learned_addr learned my addr 172.21.15.158:0/3416944933 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:14.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50429af700 1 -- 172.21.15.158:0/3416944933 >> v1:172.21.15.172:6789/0 conn(0x7f503c072150 legacy=0x7f503c06dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:14.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50429af700 1 -- 172.21.15.158:0/3416944933 >> v1:172.21.15.158:6789/0 conn(0x7f503c15b4b0 legacy=0x7f503c15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50429af700 1 -- 172.21.15.158:0/3416944933 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c06e3d0 con 0x7f503c06b8e0 2023-10-16T10:02:14.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50411ac700 1 -- 172.21.15.158:0/3416944933 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742402858 0 0) 0x7f5038001110 con 0x7f503c15b4b0 2023-10-16T10:02:14.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50429af700 1 --2- 172.21.15.158:0/3416944933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c06bd70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f5034009d90 tx=0x7f503400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1105697db08c09e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50411ac700 1 -- 172.21.15.158:0/3416944933 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503400e070 con 0x7f503c06b8e0 2023-10-16T10:02:14.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50411ac700 1 -- 172.21.15.158:0/3416944933 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503400bbd0 con 0x7f503c06b8e0 2023-10-16T10:02:14.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f50411ac700 1 -- 172.21.15.158:0/3416944933 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034012430 con 0x7f503c06b8e0 2023-10-16T10:02:14.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f5044412700 1 -- 172.21.15.158:0/3416944933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 msgr2=0x7f503c06bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.872+0000 7f5044412700 1 --2- 172.21.15.158:0/3416944933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c06bd70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f5034009d90 tx=0x7f503400b330 comp rx=0 tx=0).stop 2023-10-16T10:02:14.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.873+0000 7f5044412700 1 -- 172.21.15.158:0/3416944933 shutdown_connections 2023-10-16T10:02:14.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.873+0000 7f5044412700 1 --2- 172.21.15.158:0/3416944933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c06bd70 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.873+0000 7f5044412700 1 -- 172.21.15.158:0/3416944933 >> 172.21.15.158:0/3416944933 conn(0x7f503c0710f0 msgr2=0x7f503c071500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:14.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.873+0000 7f5044412700 1 -- 172.21.15.158:0/3416944933 shutdown_connections 2023-10-16T10:02:14.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.873+0000 7f5044412700 1 -- 172.21.15.158:0/3416944933 wait complete. 2023-10-16T10:02:14.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.874+0000 7f5044412700 1 Processor -- start 2023-10-16T10:02:14.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.874+0000 7f5044412700 1 -- start start 2023-10-16T10:02:14.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.874+0000 7f5044412700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c0c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5044412700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f503c072150 0x7f503c1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5044412700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f503c15b4b0 0x7f503c15b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5044412700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0c8060 con 0x7f503c06b8e0 2023-10-16T10:02:14.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5044412700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0c8340 con 0x7f503c15b4b0 2023-10-16T10:02:14.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5044412700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0c8620 con 0x7f503c072150 2023-10-16T10:02:14.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c0c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c0c52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56718/0 (socket says 172.21.15.158:56718) 2023-10-16T10:02:14.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 -- 172.21.15.158:0/1573920764 learned_addr learned my addr 172.21.15.158:0/1573920764 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:14.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50429af700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f503c15b4b0 0x7f503c15b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50419ad700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f503c072150 0x7f503c1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 -- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f503c072150 msgr2=0x7f503c1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f503c072150 0x7f503c1566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 -- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f503c15b4b0 msgr2=0x7f503c15b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:14.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f503c15b4b0 0x7f503c15b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:14.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 -- 172.21.15.158:0/1573920764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5034009a70 con 0x7f503c06b8e0 2023-10-16T10:02:14.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f50421ae700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c0c52e0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f503800c520 tx=0x7f503800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5032ffd700 1 -- 172.21.15.158:0/1573920764 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038004c80 con 0x7f503c06b8e0 2023-10-16T10:02:14.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5032ffd700 1 -- 172.21.15.158:0/1573920764 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5038004e20 con 0x7f503c06b8e0 2023-10-16T10:02:14.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5032ffd700 1 -- 172.21.15.158:0/1573920764 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038007310 con 0x7f503c06b8e0 2023-10-16T10:02:14.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c0c8900 con 0x7f503c06b8e0 2023-10-16T10:02:14.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.875+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c15bdd0 con 0x7f503c06b8e0 2023-10-16T10:02:14.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.876+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c0681b0 con 0x7f503c06b8e0 2023-10-16T10:02:14.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.877+0000 7f5032ffd700 1 -- 172.21.15.158:0/1573920764 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5038007470 con 0x7f503c06b8e0 2023-10-16T10:02:14.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.877+0000 7f5032ffd700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f502006b020 0x7f502006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:14.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.877+0000 7f50419ad700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f502006b020 0x7f502006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:14.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.877+0000 7f5032ffd700 1 -- 172.21.15.158:0/1573920764 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f5038089600 con 0x7f503c06b8e0 2023-10-16T10:02:14.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.878+0000 7f50419ad700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f502006b020 0x7f502006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f503c1f2290 tx=0x7f502c009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:14.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:14.882+0000 7f5032ffd700 1 -- 172.21.15.158:0/1573920764 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5038054340 con 0x7f503c06b8e0 2023-10-16T10:02:15.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.152+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f503c1600e0 con 0x7f503c06b8e0 2023-10-16T10:02:15.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.154+0000 7f5032ffd700 1 -- 172.21.15.158:0/1573920764 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5038007dd0 con 0x7f503c06b8e0 2023-10-16T10:02:15.156 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T10:02:15.156 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T10:02:15.156 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T10:02:15.156 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T10:02:15.157 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T10:02:15.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f502006b020 msgr2=0x7f502006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f502006b020 0x7f502006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f503c1f2290 tx=0x7f502c009040 comp rx=0 tx=0).stop 2023-10-16T10:02:15.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 msgr2=0x7f503c0c52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c0c52e0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f503800c520 tx=0x7f503800c8e0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 shutdown_connections 2023-10-16T10:02:15.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f503c15b4b0 0x7f503c15b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f502006b020 0x7f502006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f503c06b8e0 0x7f503c0c52e0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 --2- 172.21.15.158:0/1573920764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f503c072150 0x7f503c1566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 >> 172.21.15.158:0/1573920764 conn(0x7f503c0710f0 msgr2=0x7f503c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:15.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 shutdown_connections 2023-10-16T10:02:15.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.157+0000 7f5044412700 1 -- 172.21.15.158:0/1573920764 wait complete. 2023-10-16T10:02:15.161 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T10:02:15.170 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:02:15.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.291+0000 7fe029ae2700 1 Processor -- start 2023-10-16T10:02:15.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.291+0000 7fe029ae2700 1 -- start start 2023-10-16T10:02:15.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.291+0000 7fe029ae2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe02415a5d0 0x7fe0240ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:15.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe029ae2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe02415e860 0x7fe0240b2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:15.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe029ae2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02415edd0 con 0x7fe0240b1db0 2023-10-16T10:02:15.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe029ae2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240b2770 con 0x7fe02415e860 2023-10-16T10:02:15.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe029ae2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240ce080 con 0x7fe02415a5d0 2023-10-16T10:02:15.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe0237fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe02415a5d0 0x7fe0240ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:15.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe023fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe02415e860 0x7fe0240b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:15.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe022ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe0240b1db0 0x7fe02415e150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:60808/0 (socket says 172.21.15.158:60808) 2023-10-16T10:02:15.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe022ffd700 1 -- 172.21.15.158:0/3842128739 learned_addr learned my addr 172.21.15.158:0/3842128739 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:15.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe0227fc700 1 -- 172.21.15.158:0/3842128739 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481424246 0 0) 0x7fe02415edd0 con 0x7fe0240b1db0 2023-10-16T10:02:15.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe0227fc700 1 -- 172.21.15.158:0/3842128739 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe00c0035b0 con 0x7fe0240b1db0 2023-10-16T10:02:15.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe0237fe700 1 -- 172.21.15.158:0/3842128739 >> v2:172.21.15.172:3300/0 conn(0x7fe02415e860 msgr2=0x7fe0240b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:15.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe0237fe700 1 --2- 172.21.15.158:0/3842128739 >> v2:172.21.15.172:3300/0 conn(0x7fe02415e860 0x7fe0240b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe0237fe700 1 -- 172.21.15.158:0/3842128739 >> v1:172.21.15.172:6789/0 conn(0x7fe0240b1db0 legacy=0x7fe02415e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe0237fe700 1 -- 172.21.15.158:0/3842128739 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0240ce360 con 0x7fe02415a5d0 2023-10-16T10:02:15.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.292+0000 7fe023fff700 1 --2- 172.21.15.158:0/3842128739 >> v2:172.21.15.172:3300/0 conn(0x7fe02415e860 0x7fe0240b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:02:15.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe0237fe700 1 --2- 172.21.15.158:0/3842128739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe02415a5d0 0x7fe0240ceb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe014009840 tx=0x7fe014009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fadab4566d86453 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:15.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe0227fc700 1 -- 172.21.15.158:0/3842128739 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe014011830 con 0x7fe02415a5d0 2023-10-16T10:02:15.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe0227fc700 1 -- 172.21.15.158:0/3842128739 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe014011990 con 0x7fe02415a5d0 2023-10-16T10:02:15.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe0227fc700 1 -- 172.21.15.158:0/3842128739 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe014011d20 con 0x7fe02415a5d0 2023-10-16T10:02:15.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 -- 172.21.15.158:0/3842128739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe02415a5d0 msgr2=0x7fe0240ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 --2- 172.21.15.158:0/3842128739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe02415a5d0 0x7fe0240ceb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe014009840 tx=0x7fe014009b50 comp rx=0 tx=0).stop 2023-10-16T10:02:15.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 -- 172.21.15.158:0/3842128739 shutdown_connections 2023-10-16T10:02:15.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 --2- 172.21.15.158:0/3842128739 >> v2:172.21.15.172:3300/0 conn(0x7fe02415e860 0x7fe0240b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 --2- 172.21.15.158:0/3842128739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe02415a5d0 0x7fe0240ceb60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 -- 172.21.15.158:0/3842128739 >> 172.21.15.158:0/3842128739 conn(0x7fe0240ccdb0 msgr2=0x7fe0240cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:15.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 -- 172.21.15.158:0/3842128739 shutdown_connections 2023-10-16T10:02:15.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.293+0000 7fe029ae2700 1 -- 172.21.15.158:0/3842128739 wait complete. 2023-10-16T10:02:15.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.294+0000 7fe029ae2700 1 Processor -- start 2023-10-16T10:02:15.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe029ae2700 1 -- start start 2023-10-16T10:02:15.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe029ae2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0240b1db0 0x7fe024157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:15.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe029ae2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe02415a5d0 0x7fe0240ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:15.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe029ae2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe02415e860 0x7fe0240b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:15.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe029ae2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240ad5d0 con 0x7fe02415a5d0 2023-10-16T10:02:15.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe029ae2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240ad8b0 con 0x7fe02415e860 2023-10-16T10:02:15.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe029ae2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240adb90 con 0x7fe0240b1db0 2023-10-16T10:02:15.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.295+0000 7fe022ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe02415a5d0 0x7fe0240ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:15.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe02415a5d0 0x7fe0240ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56736/0 (socket says 172.21.15.158:56736) 2023-10-16T10:02:15.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 -- 172.21.15.158:0/934656144 learned_addr learned my addr 172.21.15.158:0/934656144 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:15.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe0237fe700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0240b1db0 0x7fe024157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:15.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe023fff700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe02415e860 0x7fe0240b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:15.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 -- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0240b1db0 msgr2=0x7fe024157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0240b1db0 0x7fe024157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 -- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe02415e860 msgr2=0x7fe0240b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe02415e860 0x7fe0240b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 -- 172.21.15.158:0/934656144 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c009a70 con 0x7fe02415a5d0 2023-10-16T10:02:15.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe022ffd700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe02415a5d0 0x7fe0240ae650 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c700 tx=0x7fe01800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:15.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe00bfff700 1 -- 172.21.15.158:0/934656144 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018005d70 con 0x7fe02415a5d0 2023-10-16T10:02:15.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe00bfff700 1 -- 172.21.15.158:0/934656144 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0180070d0 con 0x7fe02415a5d0 2023-10-16T10:02:15.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0140094a0 con 0x7fe02415a5d0 2023-10-16T10:02:15.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe00bfff700 1 -- 172.21.15.158:0/934656144 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018007460 con 0x7fe02415a5d0 2023-10-16T10:02:15.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.296+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0240ae150 con 0x7fe02415a5d0 2023-10-16T10:02:15.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.297+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0240681b0 con 0x7fe02415a5d0 2023-10-16T10:02:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.298+0000 7fe00bfff700 1 -- 172.21.15.158:0/934656144 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe0180075c0 con 0x7fe02415a5d0 2023-10-16T10:02:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.298+0000 7fe00bfff700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe01006b020 0x7fe01006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.298+0000 7fe00bfff700 1 -- 172.21.15.158:0/934656144 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fe018088e70 con 0x7fe02415a5d0 2023-10-16T10:02:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.298+0000 7fe0237fe700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe01006b020 0x7fe01006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.298+0000 7fe0237fe700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe01006b020 0x7fe01006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe014000c00 tx=0x7fe014008030 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:15.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.302+0000 7fe00bfff700 1 -- 172.21.15.158:0/934656144 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe018053b20 con 0x7fe02415a5d0 2023-10-16T10:02:15.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.538+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0240aeb90 con 0x7fe02415a5d0 2023-10-16T10:02:15.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.539+0000 7fe00bfff700 1 -- 172.21.15.158:0/934656144 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fe018017090 con 0x7fe02415a5d0 2023-10-16T10:02:15.541 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:15.541 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/94308149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":94308149},{"type":"v1","addr":"172.21.15.172:6834","nonce":94308149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/4112063716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4112063716},{"type":"v1","addr":"172.21.15.172:6835","nonce":4112063716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8274,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/658160836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":658160836},{"type":"v1","addr":"172.21.15.158:6836","nonce":658160836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:02:09.580772+0000","modified":"2023-10-16T10:02:12.619480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8277":{"gid":8277,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/3693875115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":3693875115},{"type":"v1","addr":"172.21.15.158:6837","nonce":3693875115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T10:02:15.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe01006b020 msgr2=0x7fe01006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe01006b020 0x7fe01006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe014000c00 tx=0x7fe014008030 comp rx=0 tx=0).stop 2023-10-16T10:02:15.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe02415a5d0 msgr2=0x7fe0240ae650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:15.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe02415a5d0 0x7fe0240ae650 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c700 tx=0x7fe01800ca10 comp rx=0 tx=0).stop 2023-10-16T10:02:15.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 shutdown_connections 2023-10-16T10:02:15.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe02415e860 0x7fe0240b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0240b1db0 0x7fe024157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7fe01006b020 0x7fe01006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 --2- 172.21.15.158:0/934656144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe02415a5d0 0x7fe0240ae650 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:15.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 >> 172.21.15.158:0/934656144 conn(0x7fe0240ccdb0 msgr2=0x7fe0240cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:15.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 shutdown_connections 2023-10-16T10:02:15.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:15.541+0000 7fe029ae2700 1 -- 172.21.15.158:0/934656144 wait complete. 2023-10-16T10:02:15.545 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 60 2023-10-16T10:02:15.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T10:02:09.580772+0000', 'modified': '2023-10-16T10:02:12.619480+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8277}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8277': {'gid': 8277, 'name': 'c', 'rank': 0, 'incarnation': 56, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6837/3693875115', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6835', 'nonce': 3693875115}, {'type': 'v1', 'addr': '172.21.15.158:6837', 'nonce': 3693875115}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T10:02:15.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T10:02:15.958 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T10:02:16.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.081+0000 7f448a9b7700 1 Processor -- start 2023-10-16T10:02:16.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.081+0000 7f448a9b7700 1 -- start start 2023-10-16T10:02:16.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.081+0000 7f448a9b7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f448415a6d0 0x7f44840ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.081+0000 7f448a9b7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44840cd360 0x7f448406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.081+0000 7f448a9b7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448415ec70 con 0x7f44840cfe30 2023-10-16T10:02:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.081+0000 7f448a9b7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44840afcd0 con 0x7f44840cd360 2023-10-16T10:02:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.081+0000 7f448a9b7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448415f1b0 con 0x7f448415a6d0 2023-10-16T10:02:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44840cd360 0x7f448406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44840cd360 0x7f448406ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56746/0 (socket says 172.21.15.158:56746) 2023-10-16T10:02:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 -- 172.21.15.158:0/1737057158 learned_addr learned my addr 172.21.15.158:0/1737057158 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4483fff700 1 --2- 172.21.15.158:0/1737057158 >> v2:172.21.15.172:3300/0 conn(0x7f448415a6d0 0x7f44840ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 -- 172.21.15.158:0/1737057158 >> v2:172.21.15.172:3300/0 conn(0x7f448415a6d0 msgr2=0x7f44840ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 --2- 172.21.15.158:0/1737057158 >> v2:172.21.15.172:3300/0 conn(0x7f448415a6d0 0x7f44840ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 -- 172.21.15.158:0/1737057158 >> v1:172.21.15.172:6789/0 conn(0x7f44840cfe30 legacy=0x7f44840ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 -- 172.21.15.158:0/1737057158 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44840cab50 con 0x7f44840cd360 2023-10-16T10:02:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4483fff700 1 --2- 172.21.15.158:0/1737057158 >> v2:172.21.15.172:3300/0 conn(0x7f448415a6d0 0x7f44840ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4488f54700 1 --2- 172.21.15.158:0/1737057158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f448406ebb0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f448406c770 tx=0x7f447c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71a2a33312c0aee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4482ffd700 1 -- 172.21.15.158:0/1737057158 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c010070 con 0x7f44840cd360 2023-10-16T10:02:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4482ffd700 1 -- 172.21.15.158:0/1737057158 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c00dd90 con 0x7f44840cd360 2023-10-16T10:02:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.082+0000 7f4482ffd700 1 -- 172.21.15.158:0/1737057158 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c00b310 con 0x7f44840cd360 2023-10-16T10:02:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 -- 172.21.15.158:0/1737057158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 msgr2=0x7f448406ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 --2- 172.21.15.158:0/1737057158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f448406ebb0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f448406c770 tx=0x7f447c00d430 comp rx=0 tx=0).stop 2023-10-16T10:02:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 -- 172.21.15.158:0/1737057158 shutdown_connections 2023-10-16T10:02:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 --2- 172.21.15.158:0/1737057158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f448406ebb0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 --2- 172.21.15.158:0/1737057158 >> v2:172.21.15.172:3300/0 conn(0x7f448415a6d0 0x7f44840ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 -- 172.21.15.158:0/1737057158 >> 172.21.15.158:0/1737057158 conn(0x7f448406b9e0 msgr2=0x7f448406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 -- 172.21.15.158:0/1737057158 shutdown_connections 2023-10-16T10:02:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.083+0000 7f448a9b7700 1 -- 172.21.15.158:0/1737057158 wait complete. 2023-10-16T10:02:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.084+0000 7f448a9b7700 1 Processor -- start 2023-10-16T10:02:16.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.084+0000 7f448a9b7700 1 -- start start 2023-10-16T10:02:16.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f448a9b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f44840c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f448a9b7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44840cfe30 0x7f44841ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f448a9b7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448415a6d0 0x7f448415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f448a9b7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44840c9ee0 con 0x7f44840cd360 2023-10-16T10:02:16.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f448a9b7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44840b2a10 con 0x7f44840cfe30 2023-10-16T10:02:16.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f448a9b7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44840b1e00 con 0x7f448415a6d0 2023-10-16T10:02:16.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f44840c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f44840c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56748/0 (socket says 172.21.15.158:56748) 2023-10-16T10:02:16.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 -- 172.21.15.158:0/590478072 learned_addr learned my addr 172.21.15.158:0/590478072 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:16.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4488f54700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448415a6d0 0x7f448415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 -- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448415a6d0 msgr2=0x7f448415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448415a6d0 0x7f448415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f44837fe700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44840cfe30 0x7f44841ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 -- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44840cfe30 msgr2=0x7f44841ef3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44840cfe30 0x7f44841ef3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4483fff700 1 -- 172.21.15.158:0/590478072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c00bd40 con 0x7f44840cd360 2023-10-16T10:02:16.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.085+0000 7f4488f54700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448415a6d0 0x7f448415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:16.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.086+0000 7f44837fe700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44840cfe30 0x7f44841ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:02:16.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.086+0000 7f4483fff700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f44840c99a0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f447400e760 tx=0x7f447400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.086+0000 7f4480ff9700 1 -- 172.21.15.158:0/590478072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474013610 con 0x7f44840cd360 2023-10-16T10:02:16.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.086+0000 7f4480ff9700 1 -- 172.21.15.158:0/590478072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4474013770 con 0x7f44840cd360 2023-10-16T10:02:16.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.086+0000 7f4480ff9700 1 -- 172.21.15.158:0/590478072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474013b20 con 0x7f44840cd360 2023-10-16T10:02:16.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.086+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44840b20e0 con 0x7f44840cd360 2023-10-16T10:02:16.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.086+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44840b26a0 con 0x7f44840cd360 2023-10-16T10:02:16.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.087+0000 7f4480ff9700 1 -- 172.21.15.158:0/590478072 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4474013c80 con 0x7f44840cd360 2023-10-16T10:02:16.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.088+0000 7f4480ff9700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f447006b020 0x7f447006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.088+0000 7f4480ff9700 1 -- 172.21.15.158:0/590478072 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f4474089760 con 0x7f44840cd360 2023-10-16T10:02:16.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.088+0000 7f44837fe700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f447006b020 0x7f447006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.088+0000 7f44837fe700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f447006b020 0x7f447006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f448415c510 tx=0x7f4478009040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.089+0000 7f4496ba1b80 1 Processor -- start 2023-10-16T10:02:16.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.089+0000 7f4496ba1b80 1 -- start start 2023-10-16T10:02:16.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 0x56133a3f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 0x56133a3f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a33ec20 con 0x56133a3c8010 2023-10-16T10:02:16.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a323c10 con 0x56133a3f2630 2023-10-16T10:02:16.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x56133a321530 con 0x56133a3ed220 2023-10-16T10:02:16.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4488f54700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4488f54700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56760/0 (socket says 172.21.15.158:56760) 2023-10-16T10:02:16.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4488f54700 1 -- 172.21.15.158:0/2755956134 learned_addr learned my addr 172.21.15.158:0/2755956134 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:16.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f4483fff700 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 0x56133a3f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.090+0000 7f44837fe700 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 0x56133a3f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4488f54700 1 -- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 msgr2=0x56133a3f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4488f54700 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 0x56133a3f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4488f54700 1 -- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 msgr2=0x56133a3f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4488f54700 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 0x56133a3f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4488f54700 1 -- 172.21.15.158:0/2755956134 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56133a2da3a0 con 0x56133a3c8010 2023-10-16T10:02:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4488f54700 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3c86c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f447c014000 tx=0x7f447c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25232971ba8a8c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4469ffb700 1 -- 172.21.15.158:0/2755956134 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c00b2d0 con 0x56133a3c8010 2023-10-16T10:02:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4469ffb700 1 -- 172.21.15.158:0/2755956134 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c003a20 con 0x56133a3c8010 2023-10-16T10:02:16.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4469ffb700 1 -- 172.21.15.158:0/2755956134 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c003dd0 con 0x56133a3c8010 2023-10-16T10:02:16.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2755956134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 msgr2=0x56133a3c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.091+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3c86c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f447c014000 tx=0x7f447c012000 comp rx=0 tx=0).stop 2023-10-16T10:02:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2755956134 shutdown_connections 2023-10-16T10:02:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3c86c0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 0x56133a3f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2755956134 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 0x56133a3f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2755956134 >> 172.21.15.158:0/2755956134 conn(0x56133a2f7d80 msgr2=0x56133a412ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2755956134 shutdown_connections 2023-10-16T10:02:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2755956134 wait complete. 2023-10-16T10:02:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 -- start start 2023-10-16T10:02:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.092+0000 7f4496ba1b80 1 Processor -- start 2023-10-16T10:02:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3f2630 0x56133a41ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a31ce20 con 0x56133a3c8010 2023-10-16T10:02:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a2b96c0 con 0x56133a3ed220 2023-10-16T10:02:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x56133a2b9360 con 0x56133a3f2630 2023-10-16T10:02:16.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4496ba1b80 10 client.0 _ll_register_callbacks cb 0x56133a1167b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:02:16.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4483fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4483fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3f83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56762/0 (socket says 172.21.15.158:56762) 2023-10-16T10:02:16.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4483fff700 1 -- 172.21.15.158:0/2309054998 learned_addr learned my addr 172.21.15.158:0/2309054998 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:16.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4483fff700 1 -- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3f2630 msgr2=0x56133a41ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.093+0000 7f4488f54700 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3f2630 0x56133a41ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4483fff700 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3f2630 0x56133a41ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f44837fe700 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4483fff700 1 -- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 msgr2=0x56133a3f0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4483fff700 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4483fff700 1 -- 172.21.15.158:0/2309054998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44840cab50 con 0x56133a3c8010 2023-10-16T10:02:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4483fff700 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3f83e0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f44740544c0 tx=0x7f447405c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e82e7c4ae71e36e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4462ffd700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:02:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4462ffd700 1 -- 172.21.15.158:0/2309054998 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44740584f0 con 0x56133a3c8010 2023-10-16T10:02:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4462ffd700 1 -- 172.21.15.158:0/2309054998 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f44740576b0 con 0x56133a3c8010 2023-10-16T10:02:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56133a3bd5b0 con 0x56133a3c8010 2023-10-16T10:02:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4462ffd700 1 -- 172.21.15.158:0/2309054998 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f447405fe30 con 0x56133a3c8010 2023-10-16T10:02:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 10 client.8427 fetch_fsmap learned FSMap version 60 2023-10-16T10:02:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x56133a147d90 con 0x56133a3c8010 2023-10-16T10:02:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4462ffd700 1 -- 172.21.15.158:0/2309054998 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56133a3bd5b0 con 0x56133a3c8010 2023-10-16T10:02:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4462ffd700 1 -- 172.21.15.158:0/2309054998 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 60) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f447406fae0 con 0x56133a3c8010 2023-10-16T10:02:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 10 client.8427 fetch_fsmap finished waiting for FSMap version 60 2023-10-16T10:02:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 10 client.8427 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T10:02:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 20 client.8427 populate_metadata read hostname 'smithi158' 2023-10-16T10:02:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a3eeb40 0x56133a42c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 4 client.8427 mds_command: new command op to 8277 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T10:02:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 --> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56133a315e00 con 0x56133a3eeb40 2023-10-16T10:02:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.094+0000 7f44837fe700 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a3eeb40 0x56133a42c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4462ffd700 10 client.8427 ms_handle_connect on v2:172.21.15.158:6835/3693875115 2023-10-16T10:02:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f44837fe700 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a3eeb40 0x56133a42c4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7999d3f8de2b28e server_cookie=e9c98fd438e0d139 in_seq=0 out_seq=0 2023-10-16T10:02:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4462ffd700 1 -- 172.21.15.158:0/2309054998 <== mds.0 v2:172.21.15.158:6835/3693875115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56133a315e00 con 0x56133a3eeb40 2023-10-16T10:02:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4462ffd700 10 client.8427 handle_command_reply: tid=0 2023-10-16T10:02:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 client.8427 shutdown 2023-10-16T10:02:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 msgr2=0x56133a3f83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3f83e0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f44740544c0 tx=0x7f447405c110 comp rx=0 tx=0).stop 2023-10-16T10:02:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 shutdown_connections 2023-10-16T10:02:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a3eeb40 0x56133a42c4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3f2630 0x56133a41ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3f83e0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4462ffd700 0 client.8427 ms_handle_reset on v2:172.21.15.158:6835/3693875115 2023-10-16T10:02:16.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.095+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/2309054998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.096+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 >> 172.21.15.158:0/2309054998 conn(0x56133a2f7d80 msgr2=0x56133a4130b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.096+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 shutdown_connections 2023-10-16T10:02:16.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.096+0000 7f4496ba1b80 1 -- 172.21.15.158:0/2309054998 wait complete. 2023-10-16T10:02:16.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.096+0000 7f4496ba1b80 20 client.8427 trim_cache size 0 max 16384 2023-10-16T10:02:16.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.113+0000 7f4496ba1b80 1 Processor -- start 2023-10-16T10:02:16.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.113+0000 7f4496ba1b80 1 -- start start 2023-10-16T10:02:16.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 0x56133a41ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 0x56133a3f5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a19d8d0 con 0x56133a3c8010 2023-10-16T10:02:16.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a3bd340 con 0x56133a3f2630 2023-10-16T10:02:16.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x56133a430fa0 con 0x56133a3ed220 2023-10-16T10:02:16.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3eb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56770/0 (socket says 172.21.15.158:56770) 2023-10-16T10:02:16.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 -- 172.21.15.158:0/469506854 learned_addr learned my addr 172.21.15.158:0/469506854 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:16.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 -- 172.21.15.158:0/469506854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 msgr2=0x56133a41ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 --2- 172.21.15.158:0/469506854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 0x56133a41ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 -- 172.21.15.158:0/469506854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 msgr2=0x56133a3f5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 --2- 172.21.15.158:0/469506854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 0x56133a3f5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 -- 172.21.15.158:0/469506854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56133a130c10 con 0x56133a3c8010 2023-10-16T10:02:16.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f4483fff700 1 --2- 172.21.15.158:0/469506854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3eb8f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4474089a40 tx=0x7f4474015050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57bf315833b9fc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f446a7fc700 1 -- 172.21.15.158:0/469506854 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44740584f0 con 0x56133a3c8010 2023-10-16T10:02:16.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f446a7fc700 1 -- 172.21.15.158:0/469506854 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4474054300 con 0x56133a3c8010 2023-10-16T10:02:16.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.114+0000 7f446a7fc700 1 -- 172.21.15.158:0/469506854 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44740576f0 con 0x56133a3c8010 2023-10-16T10:02:16.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 -- 172.21.15.158:0/469506854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 msgr2=0x56133a3eb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/469506854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3eb8f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4474089a40 tx=0x7f4474015050 comp rx=0 tx=0).stop 2023-10-16T10:02:16.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 -- 172.21.15.158:0/469506854 shutdown_connections 2023-10-16T10:02:16.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/469506854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3f2630 0x56133a3f5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/469506854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3c8010 0x56133a3eb8f0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/469506854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3ed220 0x56133a41ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 -- 172.21.15.158:0/469506854 >> 172.21.15.158:0/469506854 conn(0x56133a2f7d80 msgr2=0x56133a3ec630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 -- 172.21.15.158:0/469506854 shutdown_connections 2023-10-16T10:02:16.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 -- 172.21.15.158:0/469506854 wait complete. 2023-10-16T10:02:16.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.115+0000 7f4496ba1b80 1 -- start start 2023-10-16T10:02:16.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.116+0000 7f4496ba1b80 1 Processor -- start 2023-10-16T10:02:16.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.116+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3c8010 0x56133a4283a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.116+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.116+0000 7f4496ba1b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3f2630 0x56133a3f5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.116+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a3c5820 con 0x56133a3f2630 2023-10-16T10:02:16.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.116+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133a410820 con 0x56133a3ed220 2023-10-16T10:02:16.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.116+0000 7f4496ba1b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x56133a3c4b30 con 0x56133a3c8010 2023-10-16T10:02:16.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4496ba1b80 10 client.0 _ll_register_callbacks cb 0x7f44740586c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:02:16.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3f2630 0x56133a3f5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3f2630 0x56133a3f5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56780/0 (socket says 172.21.15.158:56780) 2023-10-16T10:02:16.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 -- 172.21.15.158:0/238310385 learned_addr learned my addr 172.21.15.158:0/238310385 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:16.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4483fff700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3c8010 0x56133a4283a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f44837fe700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 -- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3c8010 msgr2=0x56133a4283a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3c8010 0x56133a4283a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 -- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 msgr2=0x56133a3f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 -- 172.21.15.158:0/238310385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56133a147d90 con 0x56133a3f2630 2023-10-16T10:02:16.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f44837fe700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:16.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4488f54700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3f2630 0x56133a3f5580 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x56133a3cbfa0 tx=0x7f447c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fd0ece81ba4f80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4462ffd700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:02:16.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4462ffd700 1 -- 172.21.15.158:0/238310385 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c01ec00 con 0x56133a3f2630 2023-10-16T10:02:16.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4462ffd700 1 -- 172.21.15.158:0/238310385 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f447c02c070 con 0x56133a3f2630 2023-10-16T10:02:16.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f447c003a20 con 0x56133a3f2630 2023-10-16T10:02:16.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4462ffd700 1 -- 172.21.15.158:0/238310385 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f447c01ed60 con 0x56133a3f2630 2023-10-16T10:02:16.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4496ba1b80 10 client.8433 fetch_fsmap learned FSMap version 60 2023-10-16T10:02:16.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x56133a16e200 con 0x56133a3f2630 2023-10-16T10:02:16.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4462ffd700 1 -- 172.21.15.158:0/238310385 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c003a20 con 0x56133a3f2630 2023-10-16T10:02:16.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.117+0000 7f4462ffd700 1 -- 172.21.15.158:0/238310385 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 60) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f447c024030 con 0x56133a3f2630 2023-10-16T10:02:16.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 10 client.8433 fetch_fsmap finished waiting for FSMap version 60 2023-10-16T10:02:16.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 10 client.8433 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T10:02:16.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 20 client.8433 populate_metadata read hostname 'smithi158' 2023-10-16T10:02:16.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a4304f0 0x56133a3f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 4 client.8433 mds_command: new command op to 8277 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T10:02:16.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 --> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56133a315b20 con 0x56133a4304f0 2023-10-16T10:02:16.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4483fff700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a4304f0 0x56133a3f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4462ffd700 10 client.8433 ms_handle_connect on v2:172.21.15.158:6835/3693875115 2023-10-16T10:02:16.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4483fff700 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a4304f0 0x56133a3f5ac0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=304c494bff8fc8cf server_cookie=831b1e03c634c989 in_seq=0 out_seq=0 2023-10-16T10:02:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4462ffd700 1 -- 172.21.15.158:0/238310385 <== mds.0 v2:172.21.15.158:6835/3693875115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56133a315b20 con 0x56133a4304f0 2023-10-16T10:02:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4462ffd700 10 client.8433 handle_command_reply: tid=0 2023-10-16T10:02:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 client.8433 shutdown 2023-10-16T10:02:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3f2630 msgr2=0x56133a3f5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3f2630 0x56133a3f5580 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x56133a3cbfa0 tx=0x7f447c004b40 comp rx=0 tx=0).stop 2023-10-16T10:02:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 shutdown_connections 2023-10-16T10:02:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:6835/3693875115,v1:172.21.15.158:6837/3693875115] conn(0x56133a4304f0 0x56133a3f5ac0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56133a3f2630 0x56133a3f5580 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x56133a3c8010 0x56133a4283a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4462ffd700 0 client.8433 ms_handle_reset on v2:172.21.15.158:6835/3693875115 2023-10-16T10:02:16.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 --2- 172.21.15.158:0/238310385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56133a3ed220 0x56133a3f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 >> 172.21.15.158:0/238310385 conn(0x56133a2f7d80 msgr2=0x56133a3c9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.118+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 shutdown_connections 2023-10-16T10:02:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f4496ba1b80 1 -- 172.21.15.158:0/238310385 wait complete. 2023-10-16T10:02:16.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f4496ba1b80 20 client.8433 trim_cache size 0 max 16384 2023-10-16T10:02:16.153 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:02:16.153 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T10:02:16.153 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T10:02:16.153 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 8277, 2023-10-16T10:02:16.153 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T10:02:16.154 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T10:02:16.154 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T10:02:16.154 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9204807639999997, 2023-10-16T10:02:16.154 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 60, 2023-10-16T10:02:16.154 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 126, 2023-10-16T10:02:16.155 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 126, 2023-10-16T10:02:16.155 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.868169216 2023-10-16T10:02:16.155 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:02:16.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f447006b020 msgr2=0x7f447006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f447006b020 0x7f447006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f448415c510 tx=0x7f4478009040 comp rx=0 tx=0).stop 2023-10-16T10:02:16.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 msgr2=0x7f44840c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f44840c99a0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f447400e760 tx=0x7f447400eb20 comp rx=0 tx=0).stop 2023-10-16T10:02:16.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 shutdown_connections 2023-10-16T10:02:16.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448415a6d0 0x7f448415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f447006b020 0x7f447006d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44840cd360 0x7f44840c99a0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 --2- 172.21.15.158:0/590478072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44840cfe30 0x7f44841ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 >> 172.21.15.158:0/590478072 conn(0x7f448406b9e0 msgr2=0x7f44840d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 shutdown_connections 2023-10-16T10:02:16.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.119+0000 7f448a9b7700 1 -- 172.21.15.158:0/590478072 wait complete. 2023-10-16T10:02:16.160 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T10:02:16.168 INFO:tasks.ceph.mds.c.smithi158.stderr:2023-10-16T10:02:16.167+0000 7f71db195700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:02:16.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.280+0000 7f71261a6700 1 Processor -- start 2023-10-16T10:02:16.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.280+0000 7f71261a6700 1 -- start start 2023-10-16T10:02:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f71261a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006e5d0 0x7f712006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f71261a6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71200729b0 0x7f7120071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f71261a6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7120071e50 con 0x7f712006ab40 2023-10-16T10:02:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f71261a6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7120161270 con 0x7f71200729b0 2023-10-16T10:02:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f71261a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7120161510 con 0x7f712006e5d0 2023-10-16T10:02:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71200729b0 0x7f7120071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f712006ab40 0x7f71200722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56960/0 (socket says 172.21.15.158:56960) 2023-10-16T10:02:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711f7fe700 1 -- 172.21.15.158:0/1139660119 learned_addr learned my addr 172.21.15.158:0/1139660119 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711effd700 1 --2- 172.21.15.158:0/1139660119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006e5d0 0x7f712006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711ffff700 1 -- 172.21.15.158:0/1139660119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006e5d0 msgr2=0x7f712006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711ffff700 1 --2- 172.21.15.158:0/1139660119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006e5d0 0x7f712006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711ffff700 1 -- 172.21.15.158:0/1139660119 >> v1:172.21.15.158:6789/0 conn(0x7f712006ab40 legacy=0x7f71200722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f711ffff700 1 -- 172.21.15.158:0/1139660119 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71201617f0 con 0x7f71200729b0 2023-10-16T10:02:16.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.281+0000 7f711e7fc700 1 -- 172.21.15.158:0/1139660119 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747571567 0 0) 0x7f7120071e50 con 0x7f712006ab40 2023-10-16T10:02:16.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f711ffff700 1 --2- 172.21.15.158:0/1139660119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71200729b0 0x7f7120071910 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f7118009840 tx=0x7f7118009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3596a547141fd5ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f711e7fc700 1 -- 172.21.15.158:0/1139660119 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118011830 con 0x7f71200729b0 2023-10-16T10:02:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f711e7fc700 1 -- 172.21.15.158:0/1139660119 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7118011990 con 0x7f71200729b0 2023-10-16T10:02:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f711e7fc700 1 -- 172.21.15.158:0/1139660119 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118011d20 con 0x7f71200729b0 2023-10-16T10:02:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 -- 172.21.15.158:0/1139660119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71200729b0 msgr2=0x7f7120071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 --2- 172.21.15.158:0/1139660119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71200729b0 0x7f7120071910 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f7118009840 tx=0x7f7118009b50 comp rx=0 tx=0).stop 2023-10-16T10:02:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 -- 172.21.15.158:0/1139660119 shutdown_connections 2023-10-16T10:02:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 --2- 172.21.15.158:0/1139660119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71200729b0 0x7f7120071910 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 --2- 172.21.15.158:0/1139660119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006e5d0 0x7f712006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 -- 172.21.15.158:0/1139660119 >> 172.21.15.158:0/1139660119 conn(0x7f71200d02b0 msgr2=0x7f71200d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 -- 172.21.15.158:0/1139660119 shutdown_connections 2023-10-16T10:02:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.282+0000 7f71261a6700 1 -- 172.21.15.158:0/1139660119 wait complete. 2023-10-16T10:02:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.284+0000 7f71261a6700 1 Processor -- start 2023-10-16T10:02:16.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.284+0000 7f71261a6700 1 -- start start 2023-10-16T10:02:16.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.284+0000 7f71261a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006ab40 0x7f712015e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f71261a6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f712006e5d0 0x7f7120160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f71261a6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71200729b0 0x7f71200aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f71261a6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7120160730 con 0x7f712006e5d0 2023-10-16T10:02:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f71261a6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7120146210 con 0x7f71200729b0 2023-10-16T10:02:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f71261a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712015e990 con 0x7f712006ab40 2023-10-16T10:02:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f712006e5d0 0x7f7120160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f712006e5d0 0x7f7120160b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56792/0 (socket says 172.21.15.158:56792) 2023-10-16T10:02:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 -- 172.21.15.158:0/4077243096 learned_addr learned my addr 172.21.15.158:0/4077243096 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711ffff700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71200729b0 0x7f71200aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711f7fe700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006ab40 0x7f712015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 -- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006ab40 msgr2=0x7f712015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006ab40 0x7f712015e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 -- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71200729b0 msgr2=0x7f71200aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71200729b0 0x7f71200aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 -- 172.21.15.158:0/4077243096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71180094a0 con 0x7f712006e5d0 2023-10-16T10:02:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711f7fe700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006ab40 0x7f712015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f711effd700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f712006e5d0 0x7f7120160b50 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f710800c7f0 tx=0x7f710800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f7117fff700 1 -- 172.21.15.158:0/4077243096 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108004230 con 0x7f712006e5d0 2023-10-16T10:02:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f7117fff700 1 -- 172.21.15.158:0/4077243096 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7108004390 con 0x7f712006e5d0 2023-10-16T10:02:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7120161090 con 0x7f712006e5d0 2023-10-16T10:02:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f7117fff700 1 -- 172.21.15.158:0/4077243096 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71080036a0 con 0x7f712006e5d0 2023-10-16T10:02:16.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.285+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7120149260 con 0x7f712006e5d0 2023-10-16T10:02:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.286+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7120160870 con 0x7f712006e5d0 2023-10-16T10:02:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.287+0000 7f7117fff700 1 -- 172.21.15.158:0/4077243096 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7108003800 con 0x7f712006e5d0 2023-10-16T10:02:16.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.287+0000 7f7117fff700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f710006b020 0x7f710006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:16.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.287+0000 7f7117fff700 1 -- 172.21.15.158:0/4077243096 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f7108089880 con 0x7f712006e5d0 2023-10-16T10:02:16.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.287+0000 7f711f7fe700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f710006b020 0x7f710006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:16.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.287+0000 7f711f7fe700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f710006b020 0x7f710006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f71100012e0 tx=0x7f7110001a30 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:16.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.291+0000 7f7117fff700 1 -- 172.21.15.158:0/4077243096 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7120160870 con 0x7f712006e5d0 2023-10-16T10:02:16.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.528+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7120160870 con 0x7f712006e5d0 2023-10-16T10:02:16.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.528+0000 7f7117fff700 1 -- 172.21.15.158:0/4077243096 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v60) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7120160870 con 0x7f712006e5d0 2023-10-16T10:02:16.530 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T10:02:16.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f710006b020 msgr2=0x7f710006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f710006b020 0x7f710006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f71100012e0 tx=0x7f7110001a30 comp rx=0 tx=0).stop 2023-10-16T10:02:16.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f712006e5d0 msgr2=0x7f7120160b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:16.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f712006e5d0 0x7f7120160b50 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f710800c7f0 tx=0x7f710800cb00 comp rx=0 tx=0).stop 2023-10-16T10:02:16.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 shutdown_connections 2023-10-16T10:02:16.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71200729b0 0x7f71200aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f712006ab40 0x7f712015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f710006b020 0x7f710006d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 --2- 172.21.15.158:0/4077243096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f712006e5d0 0x7f7120160b50 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:16.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 >> 172.21.15.158:0/4077243096 conn(0x7f71200d02b0 msgr2=0x7f71200ad370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:16.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 shutdown_connections 2023-10-16T10:02:16.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:16.531+0000 7f71261a6700 1 -- 172.21.15.158:0/4077243096 wait complete. 2023-10-16T10:02:16.544 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T10:02:16.544 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T10:02:16.545 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T10:02:16.545 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T10:02:16.545 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T10:02:16.546 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T10:02:16.546 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T10:02:16.546 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T10:02:16.547 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:02:16.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:16.548 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T10:02:16.601 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T10:02:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T10:02:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T10:02:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:02:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T10:02:16.603 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:02:16.603 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:02:16.603 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:02:16.603 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:02:16.603 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:02:16.603 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:02:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T10:02:16.604 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T10:02:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T10:02:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T10:02:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 41296sec preferred_lft 41296sec 2023-10-16T10:02:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T10:02:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:02:16.605 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T10:02:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T10:02:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T10:02:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:02:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T10:02:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:02:16.606 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:02:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T10:02:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T10:02:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:02:16.607 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:02:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T10:02:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T10:02:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:02:16.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:16.608 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T10:02:16.667 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T10:02:16.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:16.669 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T10:02:16.727 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T10:02:16.727 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T10:02:16.728 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T10:02:16.728 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T10:02:16.728 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T10:02:16.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:16.729 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T10:02:16.729 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T10:02:16.730 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T10:02:16.730 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T10:02:16.730 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T10:02:16.731 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T10:02:16.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:16.833 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T10:02:16.849 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T10:02:16.849 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T10:02:16.850 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:02:16.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:16.851 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T10:02:16.916 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:02:16.916 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T10:02:16.972 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:02:16.973 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:02:16.973 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:02:17.028 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T10:02:17.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:17.029 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:02:17.114 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:02:17.114 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:02:17.138 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:02:17.138 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:02:17.138 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:02:17.139 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:02:17.139 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:02:17.139 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:02:17.139 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:02:17.139 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:02:17.166 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:02:17.166 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:02:17.166 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:02:17.166 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:02:17.167 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:02:17.167 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:02:17.167 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:02:17.167 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:02:17.167 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T10:02:17.168 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T10:02:17.168 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T10:02:17.169 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T10:02:17.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:17.170 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:02:17.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:17.201 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T10:02:17.283 INFO:teuthology.orchestra.run.smithi158.stdout:client8445 2023-10-16T10:02:17.283 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:02:17.284 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:02:17.340 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T10:02:17.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.500+0000 7f6f0a447700 1 Processor -- start 2023-10-16T10:02:17.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.500+0000 7f6f0a447700 1 -- start start 2023-10-16T10:02:17.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.501+0000 7f6f0a447700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0415a4d0 0x7f6f04071810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.501+0000 7f6f0a447700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6f040cce80 0x7f6f040b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.501+0000 7f6f0a447700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f04072bb0 con 0x7f6f040b1270 2023-10-16T10:02:17.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.501+0000 7f6f0a447700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f04071d50 con 0x7f6f040cce80 2023-10-16T10:02:17.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.501+0000 7f6f0a447700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040b1700 con 0x7f6f0415a4d0 2023-10-16T10:02:17.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.501+0000 7f6f03fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0415a4d0 0x7f6f04071810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f037fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6f040cce80 0x7f6f040b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f089e4700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6f040b1270 0x7f6f0415e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:60882/0 (socket says 172.21.15.158:60882) 2023-10-16T10:02:17.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f03fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0415a4d0 0x7f6f04071810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:39404/0 (socket says 172.21.15.158:39404) 2023-10-16T10:02:17.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f089e4700 1 -- 172.21.15.158:0/3409322707 learned_addr learned my addr 172.21.15.158:0/3409322707 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:17.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f03fff700 1 -- 172.21.15.158:0/3409322707 >> v2:172.21.15.172:3300/0 conn(0x7f6f040cce80 msgr2=0x7f6f040b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:17.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f03fff700 1 --2- 172.21.15.158:0/3409322707 >> v2:172.21.15.172:3300/0 conn(0x7f6f040cce80 0x7f6f040b0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f03fff700 1 -- 172.21.15.158:0/3409322707 >> v1:172.21.15.172:6789/0 conn(0x7f6f040b1270 legacy=0x7f6f0415e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f03fff700 1 -- 172.21.15.158:0/3409322707 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0415a8b0 con 0x7f6f0415a4d0 2023-10-16T10:02:17.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f03fff700 1 --2- 172.21.15.158:0/3409322707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0415a4d0 0x7f6f04071810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009840 tx=0x7f6ef8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf30b82a727dad7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:17.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f02ffd700 1 -- 172.21.15.158:0/3409322707 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8011830 con 0x7f6f0415a4d0 2023-10-16T10:02:17.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f02ffd700 1 -- 172.21.15.158:0/3409322707 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef8011990 con 0x7f6f0415a4d0 2023-10-16T10:02:17.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.502+0000 7f6f02ffd700 1 -- 172.21.15.158:0/3409322707 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8011d20 con 0x7f6f0415a4d0 2023-10-16T10:02:17.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 -- 172.21.15.158:0/3409322707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0415a4d0 msgr2=0x7f6f04071810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3409322707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0415a4d0 0x7f6f04071810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009840 tx=0x7f6ef8009b50 comp rx=0 tx=0).stop 2023-10-16T10:02:17.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 -- 172.21.15.158:0/3409322707 shutdown_connections 2023-10-16T10:02:17.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3409322707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0415a4d0 0x7f6f04071810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3409322707 >> v2:172.21.15.172:3300/0 conn(0x7f6f040cce80 0x7f6f040b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 -- 172.21.15.158:0/3409322707 >> 172.21.15.158:0/3409322707 conn(0x7f6f040cbd30 msgr2=0x7f6f040cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:17.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 -- 172.21.15.158:0/3409322707 shutdown_connections 2023-10-16T10:02:17.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.503+0000 7f6f0a447700 1 -- 172.21.15.158:0/3409322707 wait complete. 2023-10-16T10:02:17.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.504+0000 7f6f0a447700 1 Processor -- start 2023-10-16T10:02:17.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.504+0000 7f6f0a447700 1 -- start start 2023-10-16T10:02:17.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f0a447700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f040b1270 0x7f6f040c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f0a447700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f040cce80 0x7f6f040c5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f0a447700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0415a4d0 0x7f6f0414e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f0a447700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0415c180 con 0x7f6f040b1270 2023-10-16T10:02:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f0a447700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041f1df0 con 0x7f6f0415a4d0 2023-10-16T10:02:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f0a447700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0415c3d0 con 0x7f6f040cce80 2023-10-16T10:02:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f040b1270 0x7f6f040c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f040b1270 0x7f6f040c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56806/0 (socket says 172.21.15.158:56806) 2023-10-16T10:02:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 -- 172.21.15.158:0/3150694768 learned_addr learned my addr 172.21.15.158:0/3150694768 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:17.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f089e4700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0415a4d0 0x7f6f0414e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f037fe700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f040cce80 0x7f6f040c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 -- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f040cce80 msgr2=0x7f6f040c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f040cce80 0x7f6f040c5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 -- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0415a4d0 msgr2=0x7f6f0414e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0415a4d0 0x7f6f0414e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.505+0000 7f6f03fff700 1 -- 172.21.15.158:0/3150694768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef80094a0 con 0x7f6f040b1270 2023-10-16T10:02:17.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.506+0000 7f6f089e4700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0415a4d0 0x7f6f0414e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:17.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.506+0000 7f6f03fff700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f040b1270 0x7f6f040c6380 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80090b0 tx=0x7f6ef8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:17.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.506+0000 7f6f00ff9700 1 -- 172.21.15.158:0/3150694768 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80038a0 con 0x7f6f040b1270 2023-10-16T10:02:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.506+0000 7f6f00ff9700 1 -- 172.21.15.158:0/3150694768 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef801e070 con 0x7f6f040b1270 2023-10-16T10:02:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.506+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0415c6b0 con 0x7f6f040b1270 2023-10-16T10:02:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.506+0000 7f6f00ff9700 1 -- 172.21.15.158:0/3150694768 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8003c60 con 0x7f6f040b1270 2023-10-16T10:02:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.506+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f040c68c0 con 0x7f6f040b1270 2023-10-16T10:02:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.507+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f040681b0 con 0x7f6f040b1270 2023-10-16T10:02:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.507+0000 7f6f00ff9700 1 -- 172.21.15.158:0/3150694768 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ef802d450 con 0x7f6f040b1270 2023-10-16T10:02:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.508+0000 7f6f00ff9700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f6ef006b020 0x7f6ef006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.508+0000 7f6f00ff9700 1 -- 172.21.15.158:0/3150694768 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f6ef808a9c0 con 0x7f6f040b1270 2023-10-16T10:02:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.508+0000 7f6f037fe700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f6ef006b020 0x7f6ef006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.508+0000 7f6f037fe700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f6ef006b020 0x7f6ef006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f040c82c0 tx=0x7f6ef4008040 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:17.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.512+0000 7f6f00ff9700 1 -- 172.21.15.158:0/3150694768 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ef8055700 con 0x7f6f040b1270 2023-10-16T10:02:17.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.736+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 --> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f04072210 con 0x7f6ef006b020 2023-10-16T10:02:17.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.737+0000 7f6f00ff9700 1 -- 172.21.15.158:0/3150694768 <== mgr.8022 v2:172.21.15.158:6832/1080332904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6ef40098a0 con 0x7f6ef006b020 2023-10-16T10:02:17.739 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:02:17.739 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:02:17.739 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T10:02:17.739 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:02:17.739 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:02:17.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f6ef006b020 msgr2=0x7f6ef006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f6ef006b020 0x7f6ef006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f040c82c0 tx=0x7f6ef4008040 comp rx=0 tx=0).stop 2023-10-16T10:02:17.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f040b1270 msgr2=0x7f6f040c6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f040b1270 0x7f6f040c6380 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80090b0 tx=0x7f6ef8004750 comp rx=0 tx=0).stop 2023-10-16T10:02:17.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 shutdown_connections 2023-10-16T10:02:17.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0415a4d0 0x7f6f0414e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f6ef006b020 0x7f6ef006d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f040b1270 0x7f6f040c6380 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 --2- 172.21.15.158:0/3150694768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f040cce80 0x7f6f040c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 >> 172.21.15.158:0/3150694768 conn(0x7f6f040cbd30 msgr2=0x7f6f040af260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:17.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 shutdown_connections 2023-10-16T10:02:17.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.740+0000 7f6f0a447700 1 -- 172.21.15.158:0/3150694768 wait complete. 2023-10-16T10:02:17.753 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T10:02:17.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.878+0000 7f7a4b0ad700 1 Processor -- start 2023-10-16T10:02:17.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.878+0000 7f7a4b0ad700 1 -- start start 2023-10-16T10:02:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.879+0000 7f7a4b0ad700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a440cc590 0x7f7a440caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.879+0000 7f7a4b0ad700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a440cbd30 con 0x7f7a4406afe0 2023-10-16T10:02:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.879+0000 7f7a4b0ad700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a440cc1c0 con 0x7f7a4415da40 2023-10-16T10:02:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.879+0000 7f7a4b0ad700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4415de20 con 0x7f7a440cc590 2023-10-16T10:02:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.879+0000 7f7a43fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7a4406afe0 0x7f7a440afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:56992/0 (socket says 172.21.15.158:56992) 2023-10-16T10:02:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.879+0000 7f7a43fff700 1 -- 172.21.15.158:0/392921411 learned_addr learned my addr 172.21.15.158:0/392921411 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.879+0000 7f7a4964a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a440cc590 0x7f7a440caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a4964a700 1 -- 172.21.15.158:0/392921411 >> v1:172.21.15.172:6789/0 conn(0x7f7a4415da40 legacy=0x7f7a4406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a4964a700 1 -- 172.21.15.158:0/392921411 >> v1:172.21.15.158:6789/0 conn(0x7f7a4406afe0 legacy=0x7f7a440afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a4964a700 1 -- 172.21.15.158:0/392921411 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a440ce260 con 0x7f7a440cc590 2023-10-16T10:02:17.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a437fe700 1 -- 172.21.15.158:0/392921411 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135256348 0 0) 0x7f7a440cbd30 con 0x7f7a4406afe0 2023-10-16T10:02:17.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a4964a700 1 --2- 172.21.15.158:0/392921411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a440cc590 0x7f7a440caa20 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009d90 tx=0x7f7a3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c11d7ce7ec94c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:17.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a437fe700 1 -- 172.21.15.158:0/392921411 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c00e070 con 0x7f7a440cc590 2023-10-16T10:02:17.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a437fe700 1 -- 172.21.15.158:0/392921411 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c00bbd0 con 0x7f7a440cc590 2023-10-16T10:02:17.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a437fe700 1 -- 172.21.15.158:0/392921411 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c012430 con 0x7f7a440cc590 2023-10-16T10:02:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/392921411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a440cc590 msgr2=0x7f7a440caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/392921411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a440cc590 0x7f7a440caa20 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009d90 tx=0x7f7a3c00b330 comp rx=0 tx=0).stop 2023-10-16T10:02:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.880+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/392921411 shutdown_connections 2023-10-16T10:02:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.881+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/392921411 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a440cc590 0x7f7a440caa20 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.881+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/392921411 >> 172.21.15.158:0/392921411 conn(0x7f7a44099a10 msgr2=0x7f7a440d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.881+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/392921411 shutdown_connections 2023-10-16T10:02:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.881+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/392921411 wait complete. 2023-10-16T10:02:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.882+0000 7f7a4b0ad700 1 Processor -- start 2023-10-16T10:02:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.882+0000 7f7a4b0ad700 1 -- start start 2023-10-16T10:02:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4406afe0 0x7f7a441f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a440cc590 0x7f7a440b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a4415da40 0x7f7a440b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a440c4520 con 0x7f7a4406afe0 2023-10-16T10:02:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a440c3ed0 con 0x7f7a4415da40 2023-10-16T10:02:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a441e7d80 con 0x7f7a440cc590 2023-10-16T10:02:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4406afe0 0x7f7a441f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4406afe0 0x7f7a441f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56820/0 (socket says 172.21.15.158:56820) 2023-10-16T10:02:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 -- 172.21.15.158:0/1289299255 learned_addr learned my addr 172.21.15.158:0/1289299255 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4964a700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a4415da40 0x7f7a440b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a43fff700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a440cc590 0x7f7a440b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 -- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a440cc590 msgr2=0x7f7a440b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a440cc590 0x7f7a440b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 -- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a4415da40 msgr2=0x7f7a440b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:17.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a4415da40 0x7f7a440b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:17.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 -- 172.21.15.158:0/1289299255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c009a70 con 0x7f7a4406afe0 2023-10-16T10:02:17.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a43fff700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a440cc590 0x7f7a440b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:02:17.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a48e49700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4406afe0 0x7f7a441f2b80 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c480 tx=0x7f7a3800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:17.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a417fa700 1 -- 172.21.15.158:0/1289299255 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380052d0 con 0x7f7a4406afe0 2023-10-16T10:02:17.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a417fa700 1 -- 172.21.15.158:0/1289299255 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a38005430 con 0x7f7a4406afe0 2023-10-16T10:02:17.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a440c4140 con 0x7f7a4406afe0 2023-10-16T10:02:17.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a417fa700 1 -- 172.21.15.158:0/1289299255 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380038c0 con 0x7f7a4406afe0 2023-10-16T10:02:17.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.883+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a441e7090 con 0x7f7a4406afe0 2023-10-16T10:02:17.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.884+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a440681b0 con 0x7f7a4406afe0 2023-10-16T10:02:17.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.885+0000 7f7a417fa700 1 -- 172.21.15.158:0/1289299255 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 63) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a38003a20 con 0x7f7a4406afe0 2023-10-16T10:02:17.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.885+0000 7f7a417fa700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f7a2806b020 0x7f7a2806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:17.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.885+0000 7f7a417fa700 1 -- 172.21.15.158:0/1289299255 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f7a38088d00 con 0x7f7a4406afe0 2023-10-16T10:02:17.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.885+0000 7f7a43fff700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f7a2806b020 0x7f7a2806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:17.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.886+0000 7f7a43fff700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f7a2806b020 0x7f7a2806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a440b5290 tx=0x7f7a30007ce0 comp rx=0 tx=0).ready entity=mgr.8022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:17.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:17.890+0000 7f7a417fa700 1 -- 172.21.15.158:0/1289299255 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a38053a40 con 0x7f7a4406afe0 2023-10-16T10:02:18.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.136+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7a440b5550 con 0x7f7a4406afe0 2023-10-16T10:02:18.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.610+0000 7f7a417fa700 1 -- 172.21.15.158:0/1289299255 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v64) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7a380170a0 con 0x7f7a4406afe0 2023-10-16T10:02:18.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.613+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f7a2806b020 msgr2=0x7f7a2806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.613+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f7a2806b020 0x7f7a2806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a440b5290 tx=0x7f7a30007ce0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.613+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4406afe0 msgr2=0x7f7a441f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.613+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4406afe0 0x7f7a441f2b80 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c480 tx=0x7f7a3800c790 comp rx=0 tx=0).stop 2023-10-16T10:02:18.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.614+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 shutdown_connections 2023-10-16T10:02:18.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.614+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a4415da40 0x7f7a440b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.614+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f7a2806b020 0x7f7a2806d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.614+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4406afe0 0x7f7a441f2b80 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.614+0000 7f7a4b0ad700 1 --2- 172.21.15.158:0/1289299255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a440cc590 0x7f7a440b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.614+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 >> 172.21.15.158:0/1289299255 conn(0x7f7a44099a10 msgr2=0x7f7a4415ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:18.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.615+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 shutdown_connections 2023-10-16T10:02:18.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.615+0000 7f7a4b0ad700 1 -- 172.21.15.158:0/1289299255 wait complete. 2023-10-16T10:02:18.633 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T10:02:18.759 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:18.759+0000 7f86fd714200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:02:18.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.765+0000 7f04a1257700 1 Processor -- start 2023-10-16T10:02:18.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.765+0000 7f04a1257700 1 -- start start 2023-10-16T10:02:18.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f04a1257700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f049c157680 0x7f049c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:18.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f04a1257700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c15ac90 con 0x7f049c0b0c00 2023-10-16T10:02:18.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f04a1257700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c0c9a70 con 0x7f049c15a840 2023-10-16T10:02:18.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f04a1257700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0cace0 con 0x7f049c157680 2023-10-16T10:02:18.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f049a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f049c0b0c00 0x7f049c15a130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57032/0 (socket says 172.21.15.158:57032) 2023-10-16T10:02:18.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f049a59c700 1 -- 172.21.15.158:0/999129633 learned_addr learned my addr 172.21.15.158:0/999129633 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:18.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f049ad9d700 1 --2- 172.21.15.158:0/999129633 >> v2:172.21.15.172:3300/0 conn(0x7f049c157680 0x7f049c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:18.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f0499d9b700 1 -- 172.21.15.158:0/999129633 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009235776 0 0) 0x7f0490001110 con 0x7f049c0b0c00 2023-10-16T10:02:18.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f0499d9b700 1 -- 172.21.15.158:0/999129633 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04840035b0 con 0x7f049c0b0c00 2023-10-16T10:02:18.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f049ad9d700 1 -- 172.21.15.158:0/999129633 >> v1:172.21.15.172:6789/0 conn(0x7f049c15a840 legacy=0x7f049c0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f0499d9b700 1 -- 172.21.15.158:0/999129633 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908573016 0 0) 0x7f049c0c9a70 con 0x7f049c15a840 2023-10-16T10:02:18.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f049ad9d700 1 -- 172.21.15.158:0/999129633 >> v1:172.21.15.158:6789/0 conn(0x7f049c0b0c00 legacy=0x7f049c15a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.766+0000 7f049ad9d700 1 -- 172.21.15.158:0/999129633 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c15aed0 con 0x7f049c157680 2023-10-16T10:02:18.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f049ad9d700 1 --2- 172.21.15.158:0/999129633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0ca7a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f048c00a9e0 tx=0x7f048c00acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfb28fcbc935c6cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:18.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f0499d9b700 1 -- 172.21.15.158:0/999129633 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c00cb70 con 0x7f049c157680 2023-10-16T10:02:18.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f0499d9b700 1 -- 172.21.15.158:0/999129633 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c00ccd0 con 0x7f049c157680 2023-10-16T10:02:18.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f0499d9b700 1 -- 172.21.15.158:0/999129633 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c00ae80 con 0x7f049c157680 2023-10-16T10:02:18.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f04a1257700 1 -- 172.21.15.158:0/999129633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 msgr2=0x7f049c0ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f04a1257700 1 --2- 172.21.15.158:0/999129633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0ca7a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f048c00a9e0 tx=0x7f048c00acf0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f04a1257700 1 -- 172.21.15.158:0/999129633 shutdown_connections 2023-10-16T10:02:18.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f04a1257700 1 --2- 172.21.15.158:0/999129633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0ca7a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f04a1257700 1 -- 172.21.15.158:0/999129633 >> 172.21.15.158:0/999129633 conn(0x7f049c06af00 msgr2=0x7f049c0cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:18.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f04a1257700 1 -- 172.21.15.158:0/999129633 shutdown_connections 2023-10-16T10:02:18.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.767+0000 7f04a1257700 1 -- 172.21.15.158:0/999129633 wait complete. 2023-10-16T10:02:18.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.769+0000 7f04a1257700 1 Processor -- start 2023-10-16T10:02:18.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.769+0000 7f04a1257700 1 -- start start 2023-10-16T10:02:18.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:18.769+0000 7fd5fec7f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:02:18.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f04a1257700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c0b0c00 0x7f049c0ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:18.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f04a1257700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:18.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f04a1257700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c15a840 0x7f049c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:18.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f04a1257700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0adf60 con 0x7f049c0b0c00 2023-10-16T10:02:18.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f04a1257700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0ad2e0 con 0x7f049c157680 2023-10-16T10:02:18.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f04a1257700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0ad5c0 con 0x7f049c15a840 2023-10-16T10:02:18.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c0b0c00 0x7f049c0ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:18.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:18.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c0b0c00 0x7f049c0ae380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56860/0 (socket says 172.21.15.158:56860) 2023-10-16T10:02:18.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049ad9d700 1 -- 172.21.15.158:0/3129146539 learned_addr learned my addr 172.21.15.158:0/3129146539 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:18.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049b59e700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c15a840 0x7f049c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:18.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049a59c700 1 -- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c15a840 msgr2=0x7f049c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049a59c700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c15a840 0x7f049c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049a59c700 1 -- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c0b0c00 msgr2=0x7f049c0ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049a59c700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c0b0c00 0x7f049c0ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049a59c700 1 -- 172.21.15.158:0/3129146539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c00a640 con 0x7f049c157680 2023-10-16T10:02:18.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049ad9d700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c0b0c00 0x7f049c0ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:02:18.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.770+0000 7f049a59c700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0c8640 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f049000c700 tx=0x7f049000ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:18.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.771+0000 7f04837fe700 1 -- 172.21.15.158:0/3129146539 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490005d70 con 0x7f049c157680 2023-10-16T10:02:18.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.771+0000 7f04837fe700 1 -- 172.21.15.158:0/3129146539 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0490001d30 con 0x7f049c157680 2023-10-16T10:02:18.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.771+0000 7f04837fe700 1 -- 172.21.15.158:0/3129146539 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04900072f0 con 0x7f049c157680 2023-10-16T10:02:18.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.771+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c0ad8a0 con 0x7f049c157680 2023-10-16T10:02:18.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.771+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c0ae8c0 con 0x7f049c157680 2023-10-16T10:02:18.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.772+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c0ae0a0 con 0x7f049c157680 2023-10-16T10:02:18.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.772+0000 7f04837fe700 1 -- 172.21.15.158:0/3129146539 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 64) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0490007450 con 0x7f049c157680 2023-10-16T10:02:18.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.773+0000 7f04837fe700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 0x7f048806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:18.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.773+0000 7f049ad9d700 1 -- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 msgr2=0x7f048806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:18.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.773+0000 7f04837fe700 1 -- 172.21.15.158:0/3129146539 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f0490088c90 con 0x7f049c157680 2023-10-16T10:02:18.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.773+0000 7f049ad9d700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 0x7f048806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:02:18.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.777+0000 7f04837fe700 1 -- 172.21.15.158:0/3129146539 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f049c0ae0a0 con 0x7f049c157680 2023-10-16T10:02:18.827 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:18.825+0000 7f86fd714200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:02:18.840 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:18.838+0000 7fd5fec7f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:02:18.899 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:18.897+0000 7f86fd714200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:02:18.914 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:18.913+0000 7fd5fec7f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:02:18.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.974+0000 7f049ad9d700 1 -- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 msgr2=0x7f048806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:18.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.974+0000 7f049ad9d700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 0x7f048806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:02:18.977 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:18.975+0000 7f86fd714200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:02:18.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.981+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f049c15ac90 con 0x7f049c157680 2023-10-16T10:02:18.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.986+0000 7f04837fe700 1 -- 172.21.15.158:0/3129146539 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f049c15ac90 con 0x7f049c157680 2023-10-16T10:02:18.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 msgr2=0x7f048806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:02:18.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 0x7f048806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 msgr2=0x7f049c0c8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0c8640 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f049000c700 tx=0x7f049000ca10 comp rx=0 tx=0).stop 2023-10-16T10:02:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 shutdown_connections 2023-10-16T10:02:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f049c15a840 0x7f049c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f048806b070 0x7f048806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f049c0b0c00 0x7f049c0ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 --2- 172.21.15.158:0/3129146539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f049c157680 0x7f049c0c8640 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:18.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 >> 172.21.15.158:0/3129146539 conn(0x7f049c06af00 msgr2=0x7f049c15c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:18.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 shutdown_connections 2023-10-16T10:02:18.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:18.988+0000 7f04a1257700 1 -- 172.21.15.158:0/3129146539 wait complete. 2023-10-16T10:02:18.994 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:18.993+0000 7fd5fec7f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:02:19.002 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T10:02:19.118 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:19.116+0000 7f86fd714200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:02:19.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.129+0000 7f21b322f700 1 Processor -- start 2023-10-16T10:02:19.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.129+0000 7f21b322f700 1 -- start start 2023-10-16T10:02:19.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b322f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21ac15b540 0x7f21ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b322f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21ac06f180 0x7f21ac06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b322f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac15ee60 con 0x7f21ac06fea0 2023-10-16T10:02:19.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b322f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac15b920 con 0x7f21ac15b540 2023-10-16T10:02:19.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b322f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac15e2e0 con 0x7f21ac06f180 2023-10-16T10:02:19.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b0fcb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21ac15b540 0x7f21ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b0fcb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21ac15b540 0x7f21ac06d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56914/0 (socket says 172.21.15.158:56914) 2023-10-16T10:02:19.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b0fcb700 1 -- 172.21.15.158:0/3473405491 learned_addr learned my addr 172.21.15.158:0/3473405491 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:19.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.130+0000 7f21b17cc700 1 --1- 172.21.15.158:0/3473405491 >> v1:172.21.15.158:6789/0 conn(0x7f21ac06fea0 0x7f21ac161860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57094/0 (socket says 172.21.15.158:57094) 2023-10-16T10:02:19.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21abfff700 1 --2- 172.21.15.158:0/3473405491 >> v2:172.21.15.172:3300/0 conn(0x7f21ac06f180 0x7f21ac06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21b0fcb700 1 -- 172.21.15.158:0/3473405491 >> v2:172.21.15.172:3300/0 conn(0x7f21ac06f180 msgr2=0x7f21ac06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:19.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21b0fcb700 1 --2- 172.21.15.158:0/3473405491 >> v2:172.21.15.172:3300/0 conn(0x7f21ac06f180 0x7f21ac06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21b0fcb700 1 -- 172.21.15.158:0/3473405491 >> v1:172.21.15.158:6789/0 conn(0x7f21ac06fea0 legacy=0x7f21ac161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21ab7fe700 1 -- 172.21.15.158:0/3473405491 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537085611 0 0) 0x7f21ac15ee60 con 0x7f21ac06fea0 2023-10-16T10:02:19.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21b0fcb700 1 -- 172.21.15.158:0/3473405491 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac15e5c0 con 0x7f21ac15b540 2023-10-16T10:02:19.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21b0fcb700 1 --2- 172.21.15.158:0/3473405491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac15b540 0x7f21ac06d9c0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f21ac15ba60 tx=0x7f21a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dc1ad6955bda80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:19.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21ab7fe700 1 -- 172.21.15.158:0/3473405491 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0004230 con 0x7f21ac15b540 2023-10-16T10:02:19.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21ab7fe700 1 -- 172.21.15.158:0/3473405491 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0004390 con 0x7f21ac15b540 2023-10-16T10:02:19.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.131+0000 7f21ab7fe700 1 -- 172.21.15.158:0/3473405491 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a00036a0 con 0x7f21ac15b540 2023-10-16T10:02:19.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 -- 172.21.15.158:0/3473405491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac15b540 msgr2=0x7f21ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 --2- 172.21.15.158:0/3473405491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac15b540 0x7f21ac06d9c0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f21ac15ba60 tx=0x7f21a0009bd0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 -- 172.21.15.158:0/3473405491 shutdown_connections 2023-10-16T10:02:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 --2- 172.21.15.158:0/3473405491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac15b540 0x7f21ac06d9c0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 --2- 172.21.15.158:0/3473405491 >> v2:172.21.15.172:3300/0 conn(0x7f21ac06f180 0x7f21ac06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 -- 172.21.15.158:0/3473405491 >> 172.21.15.158:0/3473405491 conn(0x7f21ac0d03b0 msgr2=0x7f21ac0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:19.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 -- 172.21.15.158:0/3473405491 shutdown_connections 2023-10-16T10:02:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.132+0000 7f21b322f700 1 -- 172.21.15.158:0/3473405491 wait complete. 2023-10-16T10:02:19.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.133+0000 7f21b322f700 1 Processor -- start 2023-10-16T10:02:19.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.133+0000 7f21b322f700 1 -- start start 2023-10-16T10:02:19.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b322f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21ac06f180 0x7f21ac0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b322f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac06fea0 0x7f21ac0b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b322f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21ac15b540 0x7f21ac1f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b322f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0acbe0 con 0x7f21ac06fea0 2023-10-16T10:02:19.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b322f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0acec0 con 0x7f21ac15b540 2023-10-16T10:02:19.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b322f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0ad1a0 con 0x7f21ac06f180 2023-10-16T10:02:19.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac06fea0 0x7f21ac0b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac06fea0 0x7f21ac0b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56922/0 (socket says 172.21.15.158:56922) 2023-10-16T10:02:19.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 -- 172.21.15.158:0/2305943719 learned_addr learned my addr 172.21.15.158:0/2305943719 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b17cc700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21ac15b540 0x7f21ac1f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21b0fcb700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21ac06f180 0x7f21ac0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 -- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21ac06f180 msgr2=0x7f21ac0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21ac06f180 0x7f21ac0b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 -- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21ac15b540 msgr2=0x7f21ac1f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21ac15b540 0x7f21ac1f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.134+0000 7f21abfff700 1 -- 172.21.15.158:0/2305943719 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009550 con 0x7f21ac06fea0 2023-10-16T10:02:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.135+0000 7f21abfff700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac06fea0 0x7f21ac0b3c70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f219800e9b0 tx=0x7f219800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.135+0000 7f21a97fa700 1 -- 172.21.15.158:0/2305943719 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198009710 con 0x7f21ac06fea0 2023-10-16T10:02:19.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.135+0000 7f21a97fa700 1 -- 172.21.15.158:0/2305943719 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800ee50 con 0x7f21ac06fea0 2023-10-16T10:02:19.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.135+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac0ad480 con 0x7f21ac06fea0 2023-10-16T10:02:19.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.135+0000 7f21a97fa700 1 -- 172.21.15.158:0/2305943719 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800ca50 con 0x7f21ac06fea0 2023-10-16T10:02:19.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.135+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac0ac890 con 0x7f21ac06fea0 2023-10-16T10:02:19.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.136+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac1612a0 con 0x7f21ac06fea0 2023-10-16T10:02:19.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.136+0000 7f21a97fa700 1 -- 172.21.15.158:0/2305943719 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 64) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f219800cbb0 con 0x7f21ac06fea0 2023-10-16T10:02:19.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.137+0000 7f21a97fa700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 0x7f219006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.137+0000 7f21b0fcb700 1 -- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 msgr2=0x7f219006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:19.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.137+0000 7f21a97fa700 1 -- 172.21.15.158:0/2305943719 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f2198089a60 con 0x7f21ac06fea0 2023-10-16T10:02:19.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.137+0000 7f21b0fcb700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 0x7f219006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:02:19.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.142+0000 7f21a97fa700 1 -- 172.21.15.158:0/2305943719 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2198054710 con 0x7f21ac06fea0 2023-10-16T10:02:19.148 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:19.146+0000 7fd5fec7f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:02:19.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.338+0000 7f21b0fcb700 1 -- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 msgr2=0x7f219006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:19.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.338+0000 7f21b0fcb700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 0x7f219006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:02:19.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.344+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f21ac0ce6e0 con 0x7f21ac06fea0 2023-10-16T10:02:19.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.347+0000 7f21a97fa700 1 -- 172.21.15.158:0/2305943719 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f21980210c0 con 0x7f21ac06fea0 2023-10-16T10:02:19.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 msgr2=0x7f219006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:02:19.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 0x7f219006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac06fea0 msgr2=0x7f21ac0b3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac06fea0 0x7f21ac0b3c70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f219800e9b0 tx=0x7f219800ecc0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 shutdown_connections 2023-10-16T10:02:19.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21ac15b540 0x7f21ac1f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21ac06f180 0x7f21ac0b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f219006b070 0x7f219006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 --2- 172.21.15.158:0/2305943719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21ac06fea0 0x7f21ac0b3c70 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 >> 172.21.15.158:0/2305943719 conn(0x7f21ac0d03b0 msgr2=0x7f21ac0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:19.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 shutdown_connections 2023-10-16T10:02:19.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.350+0000 7f21b322f700 1 -- 172.21.15.158:0/2305943719 wait complete. 2023-10-16T10:02:19.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:19.364 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T10:02:19.386 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T10:02:19.444 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:19.443+0000 7f86fd714200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:02:19.493 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:19.492+0000 7fd5fec7f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:02:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.554+0000 7f5eae443700 1 Processor -- start 2023-10-16T10:02:19.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.555+0000 7f5eae443700 1 -- start start 2023-10-16T10:02:19.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.555+0000 7f5eae443700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea815da40 0x7f5ea80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.555+0000 7f5eae443700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea80cbd30 con 0x7f5ea80cc590 2023-10-16T10:02:19.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.555+0000 7f5eae443700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea80cc1c0 con 0x7f5ea806afe0 2023-10-16T10:02:19.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.555+0000 7f5eae443700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea815de20 con 0x7f5ea815da40 2023-10-16T10:02:19.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5ea80cc590 0x7f5ea806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57152/0 (socket says 172.21.15.158:57152) 2023-10-16T10:02:19.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea77fe700 1 -- 172.21.15.158:0/3410727640 learned_addr learned my addr 172.21.15.158:0/3410727640 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:02:19.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/3410727640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea815da40 0x7f5ea80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea6ffd700 1 -- 172.21.15.158:0/3410727640 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585986392 0 0) 0x7f5ea80cbd30 con 0x7f5ea80cc590 2023-10-16T10:02:19.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea6ffd700 1 -- 172.21.15.158:0/3410727640 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e940035b0 con 0x7f5ea80cc590 2023-10-16T10:02:19.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea6ffd700 1 -- 172.21.15.158:0/3410727640 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654480910 0 0) 0x7f5ea80cc1c0 con 0x7f5ea806afe0 2023-10-16T10:02:19.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea6ffd700 1 -- 172.21.15.158:0/3410727640 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea80cbd30 con 0x7f5ea806afe0 2023-10-16T10:02:19.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea7fff700 1 -- 172.21.15.158:0/3410727640 >> v1:172.21.15.172:6789/0 conn(0x7f5ea806afe0 legacy=0x7f5ea80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea7fff700 1 -- 172.21.15.158:0/3410727640 >> v1:172.21.15.158:6789/0 conn(0x7f5ea80cc590 legacy=0x7f5ea806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea7fff700 1 -- 172.21.15.158:0/3410727640 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80ce260 con 0x7f5ea815da40 2023-10-16T10:02:19.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/3410727640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea815da40 0x7f5ea80caa20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009870 tx=0x7f5e98009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c34579b6ffdb3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:19.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea6ffd700 1 -- 172.21.15.158:0/3410727640 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98011830 con 0x7f5ea815da40 2023-10-16T10:02:19.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea6ffd700 1 -- 172.21.15.158:0/3410727640 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98011990 con 0x7f5ea815da40 2023-10-16T10:02:19.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.556+0000 7f5ea6ffd700 1 -- 172.21.15.158:0/3410727640 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98011d20 con 0x7f5ea815da40 2023-10-16T10:02:19.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.557+0000 7f5eae443700 1 -- 172.21.15.158:0/3410727640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea815da40 msgr2=0x7f5ea80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.557+0000 7f5eae443700 1 --2- 172.21.15.158:0/3410727640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea815da40 0x7f5ea80caa20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009870 tx=0x7f5e98009b80 comp rx=0 tx=0).stop 2023-10-16T10:02:19.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.557+0000 7f5eae443700 1 -- 172.21.15.158:0/3410727640 shutdown_connections 2023-10-16T10:02:19.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.557+0000 7f5eae443700 1 --2- 172.21.15.158:0/3410727640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea815da40 0x7f5ea80caa20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.557+0000 7f5eae443700 1 -- 172.21.15.158:0/3410727640 >> 172.21.15.158:0/3410727640 conn(0x7f5ea8099a10 msgr2=0x7f5ea80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:19.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.557+0000 7f5eae443700 1 -- 172.21.15.158:0/3410727640 shutdown_connections 2023-10-16T10:02:19.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.557+0000 7f5eae443700 1 -- 172.21.15.158:0/3410727640 wait complete. 2023-10-16T10:02:19.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.558+0000 7f5eae443700 1 Processor -- start 2023-10-16T10:02:19.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eae443700 1 -- start start 2023-10-16T10:02:19.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eae443700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea806afe0 0x7f5ea815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eae443700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ea80cc590 0x7f5ea81e6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eae443700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea815da40 0x7f5ea815b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eae443700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81e6e40 con 0x7f5ea80cc590 2023-10-16T10:02:19.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eae443700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80c3fd0 con 0x7f5ea815da40 2023-10-16T10:02:19.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eae443700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80c42b0 con 0x7f5ea806afe0 2023-10-16T10:02:19.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5ea77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ea80cc590 0x7f5ea81e6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5ea77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ea80cc590 0x7f5ea81e6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56980/0 (socket says 172.21.15.158:56980) 2023-10-16T10:02:19.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5ea77fe700 1 -- 172.21.15.158:0/2688692787 learned_addr learned my addr 172.21.15.158:0/2688692787 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:19.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.559+0000 7f5eac9e0700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea815da40 0x7f5ea815b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea806afe0 0x7f5ea815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:19.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea77fe700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea806afe0 msgr2=0x7f5ea815a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea77fe700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea806afe0 0x7f5ea815a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea77fe700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea815da40 msgr2=0x7f5ea815b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:19.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea77fe700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea815da40 0x7f5ea815b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:19.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea77fe700 1 -- 172.21.15.158:0/2688692787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e980094a0 con 0x7f5ea80cc590 2023-10-16T10:02:19.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5eac9e0700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea815da40 0x7f5ea815b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:02:19.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea77fe700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ea80cc590 0x7f5ea81e6610 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c700 tx=0x7f5e9000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:19.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90005d70 con 0x7f5ea80cc590 2023-10-16T10:02:19.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e900070d0 con 0x7f5ea80cc590 2023-10-16T10:02:19.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea80c4590 con 0x7f5ea80cc590 2023-10-16T10:02:19.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90007440 con 0x7f5ea80cc590 2023-10-16T10:02:19.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.560+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea81e6b50 con 0x7f5ea80cc590 2023-10-16T10:02:19.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.561+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea80c4ab0 con 0x7f5ea80cc590 2023-10-16T10:02:19.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.562+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 64) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5e900075a0 con 0x7f5ea80cc590 2023-10-16T10:02:19.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.562+0000 7f5ea4ff9700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 0x7f5e8806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:19.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.562+0000 7f5ea7fff700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 msgr2=0x7f5e8806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:19.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.562+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f5e90088da0 con 0x7f5ea80cc590 2023-10-16T10:02:19.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.562+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 0x7f5e8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:02:19.579 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:19.563+0000 7f86fd714200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:02:19.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.567+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ea80c4ab0 con 0x7f5ea80cc590 2023-10-16T10:02:19.615 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:19.614+0000 7fd5fec7f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:02:19.640 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:19.638+0000 7f86fd714200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:02:19.692 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:19.689+0000 7fd5fec7f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:02:19.720 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:19.719+0000 7f86fd714200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:02:19.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.763+0000 7f5ea7fff700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 msgr2=0x7f5e8806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:19.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.763+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 0x7f5e8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:02:19.772 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:19.771+0000 7fd5fec7f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:02:19.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:19.794+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 --> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ea80cb400 con 0x7f5e8806b070 2023-10-16T10:02:20.090 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:20.088+0000 7f86fd714200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:02:20.145 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:20.144+0000 7fd5fec7f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:02:20.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:20.164+0000 7f5ea7fff700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 msgr2=0x7f5e8806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:20.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:20.164+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 0x7f5e8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T10:02:20.659 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:20.657+0000 7f86fd714200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:02:20.722 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:20.721+0000 7fd5fec7f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:02:20.728 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:20.726+0000 7f86fd714200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:02:20.793 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:20.792+0000 7fd5fec7f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:02:20.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:20.965+0000 7f5ea7fff700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 msgr2=0x7f5e8806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:20.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:20.965+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 0x7f5e8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T10:02:21.360 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:21.359+0000 7f86fd714200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:02:21.432 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:21.431+0000 7fd5fec7f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:02:21.438 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:21.437+0000 7f86fd714200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:02:21.510 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:21.509+0000 7fd5fec7f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:02:21.582 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:21.580+0000 7f86fd714200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:02:21.655 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:21.653+0000 7fd5fec7f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:02:21.810 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:21.809+0000 7f86fd714200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:02:21.883 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:21.882+0000 7f86fd714200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:02:21.884 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:21.882+0000 7fd5fec7f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:02:21.958 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:21.957+0000 7fd5fec7f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:02:22.148 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:22.147+0000 7f86fd714200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:02:22.226 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:22.225+0000 7fd5fec7f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:02:22.370 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:22.368+0000 7f86fd714200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:02:22.449 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:22.448+0000 7fd5fec7f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:02:22.519 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:22.518+0000 7f86fd714200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:02:22.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:22.566+0000 7f5ea7fff700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 msgr2=0x7f5e8806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1080332904 2023-10-16T10:02:22.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:22.566+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 0x7f5e8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T10:02:22.601 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:22.599+0000 7fd5fec7f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:02:22.632 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:22.630+0000 7f86fd714200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:02:22.714 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:22.713+0000 7fd5fec7f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:02:22.785 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:22.784+0000 7f86fd714200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:02:22.871 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:22.870+0000 7fd5fec7f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:02:23.047 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:23.046+0000 7f86fd714200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:02:23.134 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.133+0000 7fd5fec7f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:02:23.206 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:23.204+0000 7f86fd714200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:02:23.297 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.296+0000 7fd5fec7f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:02:23.730 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:02:23.728+0000 7f86fd714200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:02:23.829 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.828+0000 7fd5fec7f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:02:23.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:23.851+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 65) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f5e90086fd0 con 0x7f5ea80cc590 2023-10-16T10:02:23.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:23.852+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 msgr2=0x7f5e8806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:02:23.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:23.852+0000 7f5ea4ff9700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1080332904,v1:172.21.15.158:6833/1080332904] conn(0x7f5e8806b070 0x7f5e8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:23.996 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.993+0000 7fd574c60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.996 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.993+0000 7fd574c60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.996 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.993+0000 7fd574c60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.997 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.993+0000 7fd574c60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.997 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.993+0000 7fd574c60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.997 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.994+0000 7fd571c5a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.997 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.994+0000 7fd571c5a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.997 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.994+0000 7fd571c5a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.998 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.994+0000 7fd571c5a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:23.998 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:02:23.994+0000 7fd571c5a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:02:24.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.858+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 66) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5e90053e50 con 0x7f5ea80cc590 2023-10-16T10:02:24.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.858+0000 7f5ea4ff9700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f5e88035910 0x7f5e88075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:24.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.858+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 --> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e88071580 con 0x7f5e88035910 2023-10-16T10:02:24.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.859+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f5e88035910 0x7f5e88075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:24.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.861+0000 7f5ea7fff700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f5e88035910 0x7f5e88075a40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5e980090b0 tx=0x7f5e9801c040 comp rx=0 tx=0).ready entity=mgr.8475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:24.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.925+0000 7f5ea4ff9700 1 -- 172.21.15.158:0/2688692787 <== mgr.8475 v2:172.21.15.158:6832/1079935681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5e88071580 con 0x7f5e88035910 2023-10-16T10:02:24.927 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1 2023-10-16T10:02:24.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f5e88035910 msgr2=0x7f5e88075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:24.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f5e88035910 0x7f5e88075a40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5e980090b0 tx=0x7f5e9801c040 comp rx=0 tx=0).stop 2023-10-16T10:02:24.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ea80cc590 msgr2=0x7f5ea81e6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:24.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 --2- 172.21.15.158:0/2688692787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ea80cc590 0x7f5ea81e6610 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c700 tx=0x7f5e9000ca10 comp rx=0 tx=0).stop 2023-10-16T10:02:24.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eac9e0700 1 -- 172.21.15.158:0/2688692787 reap_dead start 2023-10-16T10:02:24.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 shutdown_connections 2023-10-16T10:02:24.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 >> 172.21.15.158:0/2688692787 conn(0x7f5ea8099a10 msgr2=0x7f5ea80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:24.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 shutdown_connections 2023-10-16T10:02:24.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:24.928+0000 7f5eae443700 1 -- 172.21.15.158:0/2688692787 wait complete. 2023-10-16T10:02:24.942 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T10:02:24.943 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-10_03_00 in ~36s... 2023-10-16T10:02:24.943 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T10:02:24.944 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T10:02:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.067+0000 7f09b404b700 1 Processor -- start 2023-10-16T10:02:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.068+0000 7f09b404b700 1 -- start start 2023-10-16T10:02:25.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b404b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f09ac15dad0 0x7f09ac0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:25.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b404b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09ac15deb0 0x7f09ac0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:25.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b404b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac0d0ae0 con 0x7f09ac0cea20 2023-10-16T10:02:25.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b404b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0cee50 con 0x7f09ac15deb0 2023-10-16T10:02:25.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b404b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0d12d0 con 0x7f09ac15dad0 2023-10-16T10:02:25.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09ac15deb0 0x7f09ac0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:25.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09ac15deb0 0x7f09ac0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57194/0 (socket says 172.21.15.158:57194) 2023-10-16T10:02:25.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 -- 172.21.15.158:0/1261306397 learned_addr learned my addr 172.21.15.158:0/1261306397 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:25.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b1de7700 1 --2- 172.21.15.158:0/1261306397 >> v2:172.21.15.172:3300/0 conn(0x7f09ac15dad0 0x7f09ac0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:25.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 -- 172.21.15.158:0/1261306397 >> v2:172.21.15.172:3300/0 conn(0x7f09ac15dad0 msgr2=0x7f09ac0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:02:25.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 --2- 172.21.15.158:0/1261306397 >> v2:172.21.15.172:3300/0 conn(0x7f09ac15dad0 0x7f09ac0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 -- 172.21.15.158:0/1261306397 >> v1:172.21.15.172:6789/0 conn(0x7f09ac0cea20 legacy=0x7f09ac0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:25.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 -- 172.21.15.158:0/1261306397 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0cbdf0 con 0x7f09ac15deb0 2023-10-16T10:02:25.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b15e6700 1 --2- 172.21.15.158:0/1261306397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac0ce4e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098f0 tx=0x7f09a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88335a87262d9746 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:25.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b0de5700 1 -- 172.21.15.158:0/1261306397 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8011830 con 0x7f09ac15deb0 2023-10-16T10:02:25.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b0de5700 1 -- 172.21.15.158:0/1261306397 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8011990 con 0x7f09ac15deb0 2023-10-16T10:02:25.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.069+0000 7f09b0de5700 1 -- 172.21.15.158:0/1261306397 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8011d20 con 0x7f09ac15deb0 2023-10-16T10:02:25.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 -- 172.21.15.158:0/1261306397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 msgr2=0x7f09ac0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:25.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 --2- 172.21.15.158:0/1261306397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac0ce4e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098f0 tx=0x7f09a8009c00 comp rx=0 tx=0).stop 2023-10-16T10:02:25.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 -- 172.21.15.158:0/1261306397 shutdown_connections 2023-10-16T10:02:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 --2- 172.21.15.158:0/1261306397 >> v2:172.21.15.172:3300/0 conn(0x7f09ac15dad0 0x7f09ac0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 --2- 172.21.15.158:0/1261306397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac0ce4e0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 -- 172.21.15.158:0/1261306397 >> 172.21.15.158:0/1261306397 conn(0x7f09ac099a10 msgr2=0x7f09ac0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 -- 172.21.15.158:0/1261306397 shutdown_connections 2023-10-16T10:02:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.070+0000 7f09b404b700 1 -- 172.21.15.158:0/1261306397 wait complete. 2023-10-16T10:02:25.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.071+0000 7f09b404b700 1 Processor -- start 2023-10-16T10:02:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.071+0000 7f09b404b700 1 -- start start 2023-10-16T10:02:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b404b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ac0cea20 0x7f09ac157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b404b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ac15dad0 0x7f09ac0c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:25.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b404b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac09d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b404b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0ca060 con 0x7f09ac15deb0 2023-10-16T10:02:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b404b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac157f40 con 0x7f09ac15dad0 2023-10-16T10:02:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b404b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0c5ce0 con 0x7f09ac0cea20 2023-10-16T10:02:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b25e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac09d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:25.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b25e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac09d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57202/0 (socket says 172.21.15.158:57202) 2023-10-16T10:02:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b25e8700 1 -- 172.21.15.158:0/2861123371 learned_addr learned my addr 172.21.15.158:0/2861123371 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:02:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b15e6700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ac15dad0 0x7f09ac0c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.072+0000 7f09b1de7700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ac0cea20 0x7f09ac157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b25e8700 1 -- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ac0cea20 msgr2=0x7f09ac157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:25.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b25e8700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ac0cea20 0x7f09ac157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b25e8700 1 -- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ac15dad0 msgr2=0x7f09ac0c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b25e8700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ac15dad0 0x7f09ac0c53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b25e8700 1 -- 172.21.15.158:0/2861123371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8009550 con 0x7f09ac15deb0 2023-10-16T10:02:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b25e8700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac09d690 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f09a400cc50 tx=0x7f09a400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:25.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09a27fc700 1 -- 172.21.15.158:0/2861123371 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a400ad80 con 0x7f09ac15deb0 2023-10-16T10:02:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09a27fc700 1 -- 172.21.15.158:0/2861123371 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a400ce50 con 0x7f09ac15deb0 2023-10-16T10:02:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09a27fc700 1 -- 172.21.15.158:0/2861123371 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4003c40 con 0x7f09ac15deb0 2023-10-16T10:02:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac0c99a0 con 0x7f09ac15deb0 2023-10-16T10:02:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.073+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac0c9650 con 0x7f09ac15deb0 2023-10-16T10:02:25.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.074+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac0ca1a0 con 0x7f09ac15deb0 2023-10-16T10:02:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.074+0000 7f09a27fc700 1 -- 172.21.15.158:0/2861123371 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 66) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f09a40050d0 con 0x7f09ac15deb0 2023-10-16T10:02:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.075+0000 7f09a27fc700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f099006ada0 0x7f099006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:02:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.075+0000 7f09a27fc700 1 -- 172.21.15.158:0/2861123371 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f09a4089800 con 0x7f09ac15deb0 2023-10-16T10:02:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.075+0000 7f09b1de7700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f099006ada0 0x7f099006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:02:25.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.075+0000 7f09b1de7700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f099006ada0 0x7f099006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f099c009b70 tx=0x7f099c008040 comp rx=0 tx=0).ready entity=mgr.8475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:02:25.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.079+0000 7f09a27fc700 1 -- 172.21.15.158:0/2861123371 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09ac0ca1a0 con 0x7f09ac15deb0 2023-10-16T10:02:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.302+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 --> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09ac0d0d10 con 0x7f099006ada0 2023-10-16T10:02:25.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.311+0000 7f09a27fc700 1 -- 172.21.15.158:0/2861123371 <== mgr.8475 v2:172.21.15.158:6832/1079935681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f099c0098a0 con 0x7f099006ada0 2023-10-16T10:02:25.313 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:02:25.313 INFO:teuthology.orchestra.run.smithi158.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T10:02:25.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f099006ada0 msgr2=0x7f099006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:25.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f099006ada0 0x7f099006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f099c009b70 tx=0x7f099c008040 comp rx=0 tx=0).stop 2023-10-16T10:02:25.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 msgr2=0x7f09ac09d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:02:25.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac09d690 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f09a400cc50 tx=0x7f09a400a340 comp rx=0 tx=0).stop 2023-10-16T10:02:25.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 shutdown_connections 2023-10-16T10:02:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f099006ada0 0x7f099006d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09ac15deb0 0x7f09ac09d690 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09ac0cea20 0x7f09ac157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 --2- 172.21.15.158:0/2861123371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09ac15dad0 0x7f09ac0c53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:02:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 >> 172.21.15.158:0/2861123371 conn(0x7f09ac099a10 msgr2=0x7f09ac0b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:02:25.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 shutdown_connections 2023-10-16T10:02:25.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:02:25.314+0000 7f09b404b700 1 -- 172.21.15.158:0/2861123371 wait complete. 2023-10-16T10:02:25.330 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T10:02:25.331 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2023-10-16T10:02:25.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:25.332 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:25.350 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:25.350 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:25.351 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:26.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:26.354 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:26.374 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:26.375 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:26.375 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:27.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:27.377 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:27.398 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:27.399 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:27.399 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:28.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:28.401 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:28.424 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:28.425 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:28.425 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:28.929 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:02:28.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:02:29.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:29.427 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:29.449 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:29.449 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:29.449 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:30.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:30.451 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:30.473 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:30.473 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:30.473 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:31.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:31.475 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:31.496 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:31.497 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:31.497 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:32.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:32.499 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:32.521 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:32.521 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:32.521 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:33.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:33.527 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:33.548 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:33.549 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:33.549 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:34.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:34.550 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:34.572 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:34.572 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:34.573 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:35.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:35.574 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:35.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:35.597 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:35.597 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:36.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:36.599 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:36.621 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:36.621 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:36.622 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:37.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:37.624 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:37.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:37.646 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:37.647 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:38.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:38.649 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:38.670 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:38.671 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:38.671 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:39.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:39.672 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:39.694 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:39.695 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:39.695 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:40.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:40.697 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:40.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:40.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:40.719 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:41.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:41.721 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:41.742 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:41.743 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:41.743 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:42.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:42.745 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:42.767 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:42.767 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:42.767 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:43.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:43.769 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:43.790 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:43.791 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:43.791 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:44.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:44.793 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:44.814 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:44.815 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:44.815 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:45.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:45.817 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:45.839 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:45.839 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:45.840 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:46.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:46.841 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:46.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:46.863 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:46.864 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:47.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:47.865 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:47.887 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:47.888 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:47.888 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:48.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:48.890 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:48.912 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:48.912 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:48.913 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:49.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:49.915 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:49.937 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:49.937 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:49.938 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:50.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:50.940 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:50.961 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:50.961 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:50.962 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:51.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:51.964 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:51.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:51.987 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:51.987 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:52.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:52.990 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:53.012 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:53.012 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:53.012 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:54.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:54.014 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:54.037 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:54.037 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:54.037 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:55.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:55.039 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:55.062 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:55.063 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:55.063 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:56.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:56.064 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:56.086 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:56.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:56.087 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:57.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:57.089 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:57.111 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:57.111 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:57.111 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:58.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:58.113 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:58.135 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:58.135 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:58.135 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:02:59.014 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:02:59.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:02:59.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:02:59.137 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:02:59.158 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:02:59.158 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:02:59.159 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:03:00.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:00.161 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:03:00.182 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T10:03:00.183 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T10:03:00.183 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:03:00.937 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:00.936+0000 7fd56443f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:00.938 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:00.936+0000 7fd56443f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:00.938 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:00.936+0000 7fd56443f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:00.938 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:00.936+0000 7fd56443f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:00.938 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:00.936+0000 7fd56443f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:01.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:01.185 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:03:01.205 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_03_00_UTC 2023-10-16T10:03:01.206 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-10_03_00_UTC'} 2023-10-16T10:03:01.206 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-10_03_00_UTC'} 2023-10-16T10:03:01.206 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T10:03:01.207 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-10_03_00_UTC'] 2023-10-16T10:03:01.207 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697450544.9432056, now = 1697450581.207516, timo = 36 2023-10-16T10:03:02.209 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T10:03:02.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.345+0000 7f0ffdce6700 1 Processor -- start 2023-10-16T10:03:02.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.345+0000 7f0ffdce6700 1 -- start start 2023-10-16T10:03:02.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ffdce6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0ff815da40 0x7f0ff80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ffdce6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff80cbd30 con 0x7f0ff80cc590 2023-10-16T10:03:02.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ffdce6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff80cc1c0 con 0x7f0ff806afe0 2023-10-16T10:03:02.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ffdce6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff815de20 con 0x7f0ff815da40 2023-10-16T10:03:02.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0ff80cc590 0x7f0ff806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57412/0 (socket says 172.21.15.158:57412) 2023-10-16T10:03:02.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff6ffd700 1 -- 172.21.15.158:0/2907906203 learned_addr learned my addr 172.21.15.158:0/2907906203 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:02.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352505848 0 0) 0x7f0ff80cbd30 con 0x7f0ff80cc590 2023-10-16T10:03:02.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff77fe700 1 --2- 172.21.15.158:0/2907906203 >> v2:172.21.15.172:3300/0 conn(0x7f0ff815da40 0x7f0ff80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe00035b0 con 0x7f0ff80cc590 2023-10-16T10:03:02.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3763162885 0 0) 0x7f0fe00035b0 con 0x7f0ff80cc590 2023-10-16T10:03:02.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 >> v2:172.21.15.172:3300/0 conn(0x7f0ff815da40 msgr2=0x7f0ff80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:02.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 --2- 172.21.15.158:0/2907906203 >> v2:172.21.15.172:3300/0 conn(0x7f0ff815da40 0x7f0ff80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 >> v1:172.21.15.172:6789/0 conn(0x7f0ff806afe0 legacy=0x7f0ff80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80ce260 con 0x7f0ff80cc590 2023-10-16T10:03:02.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0fec001670 con 0x7f0ff80cc590 2023-10-16T10:03:02.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 >> v1:172.21.15.158:6789/0 conn(0x7f0ff80cc590 legacy=0x7f0ff806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0fe0003a00 0x7f0fe0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fe0004850 0x7f0fe0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0fe000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff815de20 con 0x7f0fe0005210 2023-10-16T10:03:02.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe000c0b0 con 0x7f0fe0004850 2023-10-16T10:03:02.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.346+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe000c270 con 0x7f0fe0003a00 2023-10-16T10:03:02.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0fe000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff6ffd700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fe0004850 0x7f0fe0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff77fe700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0fe0003a00 0x7f0fe0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 -- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0fe0003a00 msgr2=0x7f0fe0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:02.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0fe0003a00 0x7f0fe0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 -- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fe0004850 msgr2=0x7f0fe0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:02.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fe0004850 0x7f0fe0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 -- 172.21.15.158:0/2907906203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe00035b0 con 0x7f0fe0005210 2023-10-16T10:03:02.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 -- 172.21.15.158:0/2907906203 reap_dead start 2023-10-16T10:03:02.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff7fff700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0fe000ba10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0ff000c160 tx=0x7f0fe0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c893b72392041 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:02.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0001db0 con 0x7f0fe0005210 2023-10-16T10:03:02.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff000ccc0 con 0x7f0fe0005210 2023-10-16T10:03:02.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.347+0000 7f0ff67fc700 1 -- 172.21.15.158:0/2907906203 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0003c40 con 0x7f0fe0005210 2023-10-16T10:03:02.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.348+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2907906203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 msgr2=0x7f0fe000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.348+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0fe000ba10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0ff000c160 tx=0x7f0fe0005880 comp rx=0 tx=0).stop 2023-10-16T10:03:02.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.348+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2907906203 shutdown_connections 2023-10-16T10:03:02.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.348+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2907906203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0fe000ba10 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.348+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2907906203 >> 172.21.15.158:0/2907906203 conn(0x7f0ff8099a10 msgr2=0x7f0ff80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:02.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.348+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2907906203 shutdown_connections 2023-10-16T10:03:02.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.348+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2907906203 wait complete. 2023-10-16T10:03:02.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.349+0000 7f0ffdce6700 1 Processor -- start 2023-10-16T10:03:02.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.349+0000 7f0ffdce6700 1 -- start start 2023-10-16T10:03:02.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ffdce6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0ff81e7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ffdce6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ff81e61d0 0x7f0ff80c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ffdce6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ff8147560 0x7f0ff81479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ffdce6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff81472d0 con 0x7f0fe0005210 2023-10-16T10:03:02.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ffdce6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8159ef0 con 0x7f0ff8147560 2023-10-16T10:03:02.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ffdce6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff81e73a0 con 0x7f0ff81e61d0 2023-10-16T10:03:02.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0ff81e7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0ff81e7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57246/0 (socket says 172.21.15.158:57246) 2023-10-16T10:03:02.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 -- 172.21.15.158:0/2403232109 learned_addr learned my addr 172.21.15.158:0/2403232109 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:02.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff6ffd700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ff81e61d0 0x7f0ff80c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff7fff700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ff8147560 0x7f0ff81479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 -- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ff81e61d0 msgr2=0x7f0ff80c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ff81e61d0 0x7f0ff80c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 -- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ff8147560 msgr2=0x7f0ff81479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ff8147560 0x7f0ff81479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff77fe700 1 -- 172.21.15.158:0/2403232109 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff81e6790 con 0x7f0fe0005210 2023-10-16T10:03:02.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.350+0000 7f0ff7fff700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ff8147560 0x7f0ff81479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:02.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.351+0000 7f0ff77fe700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0ff81e7830 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f0fe80119a0 tx=0x7f0fe8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:02.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.351+0000 7f0fdffff700 1 -- 172.21.15.158:0/2403232109 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800f940 con 0x7f0fe0005210 2023-10-16T10:03:02.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.351+0000 7f0fdffff700 1 -- 172.21.15.158:0/2403232109 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe800faa0 con 0x7f0fe0005210 2023-10-16T10:03:02.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.351+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff81e6a70 con 0x7f0fe0005210 2023-10-16T10:03:02.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.351+0000 7f0fdffff700 1 -- 172.21.15.158:0/2403232109 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800fe30 con 0x7f0fe0005210 2023-10-16T10:03:02.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.351+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff81e7030 con 0x7f0fe0005210 2023-10-16T10:03:02.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.352+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff80681b0 con 0x7f0fe0005210 2023-10-16T10:03:02.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.352+0000 7f0fdffff700 1 -- 172.21.15.158:0/2403232109 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 68) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f0fe8014440 con 0x7f0fe0005210 2023-10-16T10:03:02.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.353+0000 7f0fdffff700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0fe406b100 0x7f0fe406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.353+0000 7f0ff6ffd700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0fe406b100 0x7f0fe406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.353+0000 7f0fdffff700 1 -- 172.21.15.158:0/2403232109 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f0fe808a500 con 0x7f0fe0005210 2023-10-16T10:03:02.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.353+0000 7f0ff6ffd700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0fe406b100 0x7f0fe406d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80c3f50 tx=0x7f0fec009040 comp rx=0 tx=0).ready entity=mgr.8475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:02.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.357+0000 7f0fdffff700 1 -- 172.21.15.158:0/2403232109 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fe8054ef0 con 0x7f0fe0005210 2023-10-16T10:03:02.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.575+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 --> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ff80cb400 con 0x7f0fe406b100 2023-10-16T10:03:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.640+0000 7f0fdffff700 1 -- 172.21.15.158:0/2403232109 <== mgr.8475 v2:172.21.15.158:6832/1079935681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0fec007e60 con 0x7f0fe406b100 2023-10-16T10:03:02.642 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path snap_test_dir1 2023-10-16T10:03:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0fe406b100 msgr2=0x7f0fe406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0fe406b100 0x7f0fe406d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80c3f50 tx=0x7f0fec009040 comp rx=0 tx=0).stop 2023-10-16T10:03:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 msgr2=0x7f0ff81e7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0ff81e7830 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f0fe80119a0 tx=0x7f0fe8011d60 comp rx=0 tx=0).stop 2023-10-16T10:03:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 shutdown_connections 2023-10-16T10:03:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ff8147560 0x7f0ff81479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0fe406b100 0x7f0fe406d5c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fe0005210 0x7f0ff81e7830 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 --2- 172.21.15.158:0/2403232109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ff81e61d0 0x7f0ff80c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 >> 172.21.15.158:0/2403232109 conn(0x7f0ff8099a10 msgr2=0x7f0ff80b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.642+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 shutdown_connections 2023-10-16T10:03:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.643+0000 7f0ffdce6700 1 -- 172.21.15.158:0/2403232109 wait complete. 2023-10-16T10:03:02.659 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T10:03:02.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:02.660 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T10:03:02.677 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_03_00_UTC 2023-10-16T10:03:02.677 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_03_00_UTC 2023-10-16T10:03:02.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:02.678 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_03_00_UTC) 2023-10-16T10:03:02.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:02.738 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T10:03:02.797 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T10:03:02.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.958+0000 7f728ed87700 1 Processor -- start 2023-10-16T10:03:02.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.958+0000 7f728ed87700 1 -- start start 2023-10-16T10:03:02.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f728ed87700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f728815ed80 0x7f728815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f728ed87700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728815e6e0 0x7f72880cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f728ed87700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728815eaf0 con 0x7f728806abd0 2023-10-16T10:03:02.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f728ed87700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728806b410 con 0x7f728815ed80 2023-10-16T10:03:02.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f728ed87700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728806b6f0 con 0x7f728815e6e0 2023-10-16T10:03:02.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f728815ed80 0x7f728815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f728815ed80 0x7f728815e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57256/0 (socket says 172.21.15.158:57256) 2023-10-16T10:03:02.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 -- 172.21.15.158:0/942040180 learned_addr learned my addr 172.21.15.158:0/942040180 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:02.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f728d324700 1 --2- 172.21.15.158:0/942040180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728815e6e0 0x7f72880cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 -- 172.21.15.158:0/942040180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728815e6e0 msgr2=0x7f72880cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:02.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 --2- 172.21.15.158:0/942040180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728815e6e0 0x7f72880cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 -- 172.21.15.158:0/942040180 >> v1:172.21.15.172:6789/0 conn(0x7f728806abd0 legacy=0x7f7288143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:02.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 -- 172.21.15.158:0/942040180 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728806b9d0 con 0x7f728815ed80 2023-10-16T10:03:02.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f728d324700 1 --2- 172.21.15.158:0/942040180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728815e6e0 0x7f72880cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:02.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f7287fff700 1 --2- 172.21.15.158:0/942040180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728815e1a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f727400bb90 tx=0x7f727400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b49e229f2508d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:02.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f72877fe700 1 -- 172.21.15.158:0/942040180 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727400dd10 con 0x7f728815ed80 2023-10-16T10:03:02.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f72877fe700 1 -- 172.21.15.158:0/942040180 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727400de70 con 0x7f728815ed80 2023-10-16T10:03:02.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.959+0000 7f72877fe700 1 -- 172.21.15.158:0/942040180 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274004230 con 0x7f728815ed80 2023-10-16T10:03:02.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 -- 172.21.15.158:0/942040180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 msgr2=0x7f728815e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 --2- 172.21.15.158:0/942040180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728815e1a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f727400bb90 tx=0x7f727400bea0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 -- 172.21.15.158:0/942040180 shutdown_connections 2023-10-16T10:03:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 --2- 172.21.15.158:0/942040180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728815e6e0 0x7f72880cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 --2- 172.21.15.158:0/942040180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728815e1a0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 -- 172.21.15.158:0/942040180 >> 172.21.15.158:0/942040180 conn(0x7f72880cc690 msgr2=0x7f72880cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 -- 172.21.15.158:0/942040180 shutdown_connections 2023-10-16T10:03:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.960+0000 7f728ed87700 1 -- 172.21.15.158:0/942040180 wait complete. 2023-10-16T10:03:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.961+0000 7f728ed87700 1 Processor -- start 2023-10-16T10:03:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.961+0000 7f728ed87700 1 -- start start 2023-10-16T10:03:02.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728ed87700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728806abd0 0x7f72880c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728ed87700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f728815e6e0 0x7f72880c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728ed87700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728ed87700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728815b4d0 con 0x7f728815ed80 2023-10-16T10:03:02.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728ed87700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728815b130 con 0x7f728815e6e0 2023-10-16T10:03:02.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728ed87700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72880c9700 con 0x7f728806abd0 2023-10-16T10:03:02.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728d324700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728d324700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728806d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57264/0 (socket says 172.21.15.158:57264) 2023-10-16T10:03:02.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728d324700 1 -- 172.21.15.158:0/849302446 learned_addr learned my addr 172.21.15.158:0/849302446 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:02.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f728cb23700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728806abd0 0x7f72880c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.962+0000 7f7287fff700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f728815e6e0 0x7f72880c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728d324700 1 -- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728806abd0 msgr2=0x7f72880c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728d324700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728806abd0 0x7f72880c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728d324700 1 -- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f728815e6e0 msgr2=0x7f72880c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:02.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728d324700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f728815e6e0 0x7f72880c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:02.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728d324700 1 -- 172.21.15.158:0/849302446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727400b7f0 con 0x7f728815ed80 2023-10-16T10:03:02.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728cb23700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728806abd0 0x7f72880c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:02.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f7287fff700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f728815e6e0 0x7f72880c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:02.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728d324700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728806d8e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f728000cef0 tx=0x7f728000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:02.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f72857fa700 1 -- 172.21.15.158:0/849302446 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728000ae50 con 0x7f728815ed80 2023-10-16T10:03:02.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f72857fa700 1 -- 172.21.15.158:0/849302446 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7280015440 con 0x7f728815ed80 2023-10-16T10:03:02.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f72857fa700 1 -- 172.21.15.158:0/849302446 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72800157f0 con 0x7f728815ed80 2023-10-16T10:03:02.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72880c99e0 con 0x7f728815ed80 2023-10-16T10:03:02.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.963+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7288151c20 con 0x7f728815ed80 2023-10-16T10:03:02.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.964+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7288066db0 con 0x7f728815ed80 2023-10-16T10:03:02.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.964+0000 7f72857fa700 1 -- 172.21.15.158:0/849302446 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 69) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f728000d430 con 0x7f728815ed80 2023-10-16T10:03:02.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.965+0000 7f72857fa700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f726c06b190 0x7f726c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:02.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.965+0000 7f72857fa700 1 -- 172.21.15.158:0/849302446 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f728008a420 con 0x7f728815ed80 2023-10-16T10:03:02.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.965+0000 7f728cb23700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f726c06b190 0x7f726c06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:02.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.965+0000 7f728cb23700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f726c06b190 0x7f726c06d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f727c003fc0 tx=0x7f727c009040 comp rx=0 tx=0).ready entity=mgr.8475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:02.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:02.969+0000 7f72857fa700 1 -- 172.21.15.158:0/849302446 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7280054dc0 con 0x7f728815ed80 2023-10-16T10:03:03.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.215+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f72880681b0 con 0x7f728815ed80 2023-10-16T10:03:03.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.657+0000 7f72857fa700 1 -- 172.21.15.158:0/849302446 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v70) v1 ==== 93+0+0 (secure 0 0 0) 0x7f72800583e0 con 0x7f728815ed80 2023-10-16T10:03:03.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.660+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f726c06b190 msgr2=0x7f726c06d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:03.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.660+0000 7f728ed87700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f726c06b190 0x7f726c06d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f727c003fc0 tx=0x7f727c009040 comp rx=0 tx=0).stop 2023-10-16T10:03:03.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.660+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 msgr2=0x7f728806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:03.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.660+0000 7f728ed87700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728806d8e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f728000cef0 tx=0x7f728000a340 comp rx=0 tx=0).stop 2023-10-16T10:03:03.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.662+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 shutdown_connections 2023-10-16T10:03:03.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.662+0000 7f728ed87700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f726c06b190 0x7f726c06d650 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:03.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.662+0000 7f728ed87700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f728815ed80 0x7f728806d8e0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:03.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.662+0000 7f728ed87700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f728806abd0 0x7f72880c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:03.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.662+0000 7f728ed87700 1 --2- 172.21.15.158:0/849302446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f728815e6e0 0x7f72880c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:03.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.662+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 >> 172.21.15.158:0/849302446 conn(0x7f72880cc690 msgr2=0x7f72880c6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:03.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.662+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 shutdown_connections 2023-10-16T10:03:03.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.663+0000 7f728ed87700 1 -- 172.21.15.158:0/849302446 wait complete. 2023-10-16T10:03:03.677 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:03:03.677 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:03.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:03.763 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:03:03.763 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:03.764 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:03.781 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:03.781 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:03.805 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:03.804+0000 7f448077a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:03:03.811 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:03.810+0000 7ff67f63e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:03:03.822 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:03.874 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:03.873+0000 7f448077a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:03.877 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:03.875+0000 7ff67f63e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:03.949 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:03.948+0000 7f448077a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:03:03.950 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:03.948+0000 7ff67f63e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:03:03.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.986+0000 7f0b2d7e8700 1 Processor -- start 2023-10-16T10:03:03.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.986+0000 7f0b2d7e8700 1 -- start start 2023-10-16T10:03:03.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b2d7e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 0x7f0b280b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:03.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b2d7e8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2815b930 con 0x7f0b280ce530 2023-10-16T10:03:03.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b2d7e8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2815f5f0 con 0x7f0b2815f0f0 2023-10-16T10:03:03.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b2d7e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280cf020 con 0x7f0b280cc1b0 2023-10-16T10:03:03.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b26ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0b280ce530 0x7f0b280cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57498/0 (socket says 172.21.15.158:57498) 2023-10-16T10:03:03.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b26ffd700 1 -- 172.21.15.158:0/2525537287 learned_addr learned my addr 172.21.15.158:0/2525537287 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:03.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b267fc700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 0x7f0b280b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:03.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334101819 0 0) 0x7f0b2815b930 con 0x7f0b280ce530 2023-10-16T10:03:03.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b100035b0 con 0x7f0b280ce530 2023-10-16T10:03:03.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522877686 0 0) 0x7f0b2815f5f0 con 0x7f0b2815f0f0 2023-10-16T10:03:03.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2815b930 con 0x7f0b2815f0f0 2023-10-16T10:03:03.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1798767926 0 0) 0x7f0b100035b0 con 0x7f0b280ce530 2023-10-16T10:03:03.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 msgr2=0x7f0b280b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:03.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 0x7f0b280b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:03.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 >> v1:172.21.15.172:6789/0 conn(0x7f0b2815f0f0 legacy=0x7f0b280cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:03.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2815f8d0 con 0x7f0b280ce530 2023-10-16T10:03:03.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0b18001670 con 0x7f0b280ce530 2023-10-16T10:03:03.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b267fc700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 0x7f0b280b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:03.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 >> v1:172.21.15.158:6789/0 conn(0x7f0b280ce530 legacy=0x7f0b280cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:03.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b10003bc0 0x7f0b10003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:03.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:03.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 0x7f0b10005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:03.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280cf020 con 0x7f0b10003bc0 2023-10-16T10:03:03.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1000ba40 con 0x7f0b10004850 2023-10-16T10:03:03.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.988+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1000bc00 con 0x7f0b280cc1b0 2023-10-16T10:03:03.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b26ffd700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b10003bc0 0x7f0b10003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:03.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b267fc700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:03.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b277fe700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 0x7f0b10005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:03.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b26ffd700 1 -- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 msgr2=0x7f0b10005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:03.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b26ffd700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b280cc1b0 0x7f0b10005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:03.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b26ffd700 1 -- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10004850 msgr2=0x7f0b10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:03.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b26ffd700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:03.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b26ffd700 1 -- 172.21.15.158:0/2525537287 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c0094a0 con 0x7f0b10003bc0 2023-10-16T10:03:03.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b26ffd700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b10003bc0 0x7f0b10003fc0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001eb0 tx=0x7f0b10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cca4960f6068a48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:03.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1800cd80 con 0x7f0b10003bc0 2023-10-16T10:03:03.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b18001af0 con 0x7f0b10003bc0 2023-10-16T10:03:03.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.989+0000 7f0b25ffb700 1 -- 172.21.15.158:0/2525537287 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18005ba0 con 0x7f0b10003bc0 2023-10-16T10:03:03.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.990+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2525537287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b10003bc0 msgr2=0x7f0b10003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:03.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.990+0000 7f0b2d7e8700 1 --2- 172.21.15.158:0/2525537287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b10003bc0 0x7f0b10003fc0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001eb0 tx=0x7f0b10005880 comp rx=0 tx=0).stop 2023-10-16T10:03:04.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.990+0000 7f0b277fe700 1 -- 172.21.15.158:0/2525537287 reap_dead start 2023-10-16T10:03:04.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.990+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2525537287 shutdown_connections 2023-10-16T10:03:04.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.990+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2525537287 >> 172.21.15.158:0/2525537287 conn(0x7f0b2806a8d0 msgr2=0x7f0b2806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:04.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.990+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2525537287 shutdown_connections 2023-10-16T10:03:04.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.990+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2525537287 wait complete. 2023-10-16T10:03:04.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.991+0000 7f0b2d7e8700 1 Processor -- start 2023-10-16T10:03:04.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.991+0000 7f0b2d7e8700 1 -- start start 2023-10-16T10:03:04.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b2d7e8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10003bc0 0x7f0b281ee540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b2d7e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b281f1d50 0x7f0b280b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b2d7e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b280b5c90 0x7f0b280b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b2d7e8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280b6680 con 0x7f0b280b5c90 2023-10-16T10:03:04.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b2d7e8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281eea80 con 0x7f0b10003bc0 2023-10-16T10:03:04.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b2d7e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280b4490 con 0x7f0b281f1d50 2023-10-16T10:03:04.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b280b5c90 0x7f0b280b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b280b5c90 0x7f0b280b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57338/0 (socket says 172.21.15.158:57338) 2023-10-16T10:03:04.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 -- 172.21.15.158:0/2512486820 learned_addr learned my addr 172.21.15.158:0/2512486820 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:04.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b26ffd700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10003bc0 0x7f0b281ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b267fc700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b281f1d50 0x7f0b280b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 -- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b281f1d50 msgr2=0x7f0b280b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b281f1d50 0x7f0b280b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 -- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10003bc0 msgr2=0x7f0b281ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10003bc0 0x7f0b281ee540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.992+0000 7f0b277fe700 1 -- 172.21.15.158:0/2512486820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b280b4770 con 0x7f0b280b5c90 2023-10-16T10:03:04.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.993+0000 7f0b277fe700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b280b5c90 0x7f0b280b6140 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f0b28072910 tx=0x7f0b2000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:04.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.993+0000 7f0b0f7fe700 1 -- 172.21.15.158:0/2512486820 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20003c10 con 0x7f0b280b5c90 2023-10-16T10:03:04.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.993+0000 7f0b0f7fe700 1 -- 172.21.15.158:0/2512486820 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b20003d70 con 0x7f0b280b5c90 2023-10-16T10:03:04.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.993+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b280b4a50 con 0x7f0b280b5c90 2023-10-16T10:03:04.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.993+0000 7f0b0f7fe700 1 -- 172.21.15.158:0/2512486820 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2000db50 con 0x7f0b280b5c90 2023-10-16T10:03:04.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.993+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b280b5010 con 0x7f0b280b5c90 2023-10-16T10:03:04.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.994+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b280681b0 con 0x7f0b280b5c90 2023-10-16T10:03:04.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.994+0000 7f0b0f7fe700 1 -- 172.21.15.158:0/2512486820 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f0b2000dcb0 con 0x7f0b280b5c90 2023-10-16T10:03:04.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.994+0000 7f0b0f7fe700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 0x7f0b1406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.994+0000 7f0b26ffd700 1 -- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 msgr2=0x7f0b1406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:04.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.995+0000 7f0b0f7fe700 1 -- 172.21.15.158:0/2512486820 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f0b20089fe0 con 0x7f0b280b5c90 2023-10-16T10:03:04.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.995+0000 7f0b26ffd700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 0x7f0b1406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:04.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:03.999+0000 7f0b0f7fe700 1 -- 172.21.15.158:0/2512486820 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b20054980 con 0x7f0b280b5c90 2023-10-16T10:03:04.029 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:04.028+0000 7ff67f63e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:03:04.034 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:04.033+0000 7f448077a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:03:04.171 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:04.170+0000 7ff67f63e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:03:04.179 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:04.178+0000 7f448077a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:03:04.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.196+0000 7f0b26ffd700 1 -- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 msgr2=0x7f0b1406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:04.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.196+0000 7f0b26ffd700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 0x7f0b1406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:04.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.200+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b28144f40 con 0x7f0b280b5c90 2023-10-16T10:03:04.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.201+0000 7f0b0f7fe700 1 -- 172.21.15.158:0/2512486820 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+17882 (secure 0 0 0) 0x7f0b20057fa0 con 0x7f0b280b5c90 2023-10-16T10:03:04.203 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:04.203 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":127,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:23.842920+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:02:08.021999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T10:02:08.539662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1769168841":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3578502848":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2674552982":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2360787591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6833/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6832/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3568191591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/1741897775":"2023-10-17T10:02:23.842872+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:04.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 msgr2=0x7f0b1406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:04.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 0x7f0b1406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b280b5c90 msgr2=0x7f0b280b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b280b5c90 0x7f0b280b6140 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f0b28072910 tx=0x7f0b2000d430 comp rx=0 tx=0).stop 2023-10-16T10:03:04.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 shutdown_connections 2023-10-16T10:03:04.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0b1406b140 0x7f0b1406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b280b5c90 0x7f0b280b6140 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b10003bc0 0x7f0b281ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 --2- 172.21.15.158:0/2512486820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b281f1d50 0x7f0b280b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 >> 172.21.15.158:0/2512486820 conn(0x7f0b2806a8d0 msgr2=0x7f0b280cb930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:04.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 shutdown_connections 2023-10-16T10:03:04.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.204+0000 7f0b2d7e8700 1 -- 172.21.15.158:0/2512486820 wait complete. 2023-10-16T10:03:04.218 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:02:08.021999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:02:08.539662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:03:04.219 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:03:04.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.382+0000 7fabe1895700 1 Processor -- start 2023-10-16T10:03:04.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.382+0000 7fabe1895700 1 -- start start 2023-10-16T10:03:04.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabe1895700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabdc15b540 0x7fabdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabe1895700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc15ee60 con 0x7fabdc06f180 2023-10-16T10:03:04.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabe1895700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc15e2e0 con 0x7fabdc06afe0 2023-10-16T10:03:04.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabe1895700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabdc15b920 con 0x7fabdc15b540 2023-10-16T10:03:04.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabdaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabdc15b540 0x7fabdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabda7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fabdc06f180 0x7fabdc06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57548/0 (socket says 172.21.15.158:57548) 2023-10-16T10:03:04.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabda7fc700 1 -- 172.21.15.158:0/2532096114 learned_addr learned my addr 172.21.15.158:0/2532096114 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:04.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabdaffd700 1 -- 172.21.15.158:0/2532096114 >> v1:172.21.15.172:6789/0 conn(0x7fabdc06afe0 legacy=0x7fabdc161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabdaffd700 1 -- 172.21.15.158:0/2532096114 >> v1:172.21.15.158:6789/0 conn(0x7fabdc06f180 legacy=0x7fabdc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabdaffd700 1 -- 172.21.15.158:0/2532096114 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc15e5c0 con 0x7fabdc15b540 2023-10-16T10:03:04.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.383+0000 7fabd9ffb700 1 -- 172.21.15.158:0/2532096114 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699151720 0 0) 0x7fabdc15ee60 con 0x7fabdc06f180 2023-10-16T10:03:04.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabdaffd700 1 --2- 172.21.15.158:0/2532096114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc15b540 0x7fabdc06d9c0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009840 tx=0x7fabcc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae491dcdcb8fd5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:04.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabd9ffb700 1 -- 172.21.15.158:0/2532096114 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc011830 con 0x7fabdc15b540 2023-10-16T10:03:04.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabd9ffb700 1 -- 172.21.15.158:0/2532096114 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc011990 con 0x7fabdc15b540 2023-10-16T10:03:04.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabd9ffb700 1 -- 172.21.15.158:0/2532096114 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc011d20 con 0x7fabdc15b540 2023-10-16T10:03:04.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabe1895700 1 -- 172.21.15.158:0/2532096114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc15b540 msgr2=0x7fabdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabe1895700 1 --2- 172.21.15.158:0/2532096114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc15b540 0x7fabdc06d9c0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009840 tx=0x7fabcc009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:04.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabe1895700 1 -- 172.21.15.158:0/2532096114 shutdown_connections 2023-10-16T10:03:04.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabe1895700 1 --2- 172.21.15.158:0/2532096114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc15b540 0x7fabdc06d9c0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabe1895700 1 -- 172.21.15.158:0/2532096114 >> 172.21.15.158:0/2532096114 conn(0x7fabdc0d03b0 msgr2=0x7fabdc0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:04.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabe1895700 1 -- 172.21.15.158:0/2532096114 shutdown_connections 2023-10-16T10:03:04.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.384+0000 7fabe1895700 1 -- 172.21.15.158:0/2532096114 wait complete. 2023-10-16T10:03:04.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.386+0000 7fabe1895700 1 Processor -- start 2023-10-16T10:03:04.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.387+0000 7fabe1895700 1 -- start start 2023-10-16T10:03:04.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabe1895700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabdc06afe0 0x7fabdc0b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabe1895700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc06f180 0x7fabdc159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabe1895700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabdc15b540 0x7fabdc1597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabe1895700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc0c79f0 con 0x7fabdc06f180 2023-10-16T10:03:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabe1895700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc0c7cd0 con 0x7fabdc06afe0 2023-10-16T10:03:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabe1895700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc0c7fb0 con 0x7fabdc15b540 2023-10-16T10:03:04.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc06f180 0x7fabdc159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc06f180 0x7fabdc159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57380/0 (socket says 172.21.15.158:57380) 2023-10-16T10:03:04.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 -- 172.21.15.158:0/2837396433 learned_addr learned my addr 172.21.15.158:0/2837396433 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:04.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabdaffd700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabdc06afe0 0x7fabdc0b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabdb7fe700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabdc15b540 0x7fabdc1597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 -- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabdc15b540 msgr2=0x7fabdc1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabdc15b540 0x7fabdc1597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 -- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabdc06afe0 msgr2=0x7fabdc0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabdc06afe0 0x7fabdc0b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 -- 172.21.15.158:0/2837396433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0094a0 con 0x7fabdc06f180 2023-10-16T10:03:04.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabdb7fe700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabdc15b540 0x7fabdc1597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:04.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabda7fc700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc06f180 0x7fabdc159290 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fabd000c520 tx=0x7fabd000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:04.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabc3fff700 1 -- 172.21.15.158:0/2837396433 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0004c80 con 0x7fabdc06f180 2023-10-16T10:03:04.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabc3fff700 1 -- 172.21.15.158:0/2837396433 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd0004de0 con 0x7fabdc06f180 2023-10-16T10:03:04.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabc3fff700 1 -- 172.21.15.158:0/2837396433 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0007310 con 0x7fabdc06f180 2023-10-16T10:03:04.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.388+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc0c8290 con 0x7fabdc06f180 2023-10-16T10:03:04.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.389+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc0cbd30 con 0x7fabdc06f180 2023-10-16T10:03:04.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.389+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc0c3d10 con 0x7fabdc06f180 2023-10-16T10:03:04.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.390+0000 7fabc3fff700 1 -- 172.21.15.158:0/2837396433 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fabd0007470 con 0x7fabdc06f180 2023-10-16T10:03:04.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.391+0000 7fabc3fff700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 0x7fabc806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.391+0000 7fabdaffd700 1 -- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 msgr2=0x7fabc806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:04.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.391+0000 7fabc3fff700 1 -- 172.21.15.158:0/2837396433 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fabd0089b30 con 0x7fabdc06f180 2023-10-16T10:03:04.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.391+0000 7fabdaffd700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 0x7fabc806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:04.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.395+0000 7fabc3fff700 1 -- 172.21.15.158:0/2837396433 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fabd0054440 con 0x7fabdc06f180 2023-10-16T10:03:04.497 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:04.495+0000 7ff67f63e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:03:04.534 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:04.533+0000 7f448077a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:03:04.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.592+0000 7fabdaffd700 1 -- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 msgr2=0x7fabc806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:04.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.592+0000 7fabdaffd700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 0x7fabc806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:04.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.608+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fabdc0681b0 con 0x7fabdc06f180 2023-10-16T10:03:04.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.608+0000 7fabc3fff700 1 -- 172.21.15.158:0/2837396433 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7fabd0057a60 con 0x7fabdc06f180 2023-10-16T10:03:04.609 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:03:04.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 msgr2=0x7fabc806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:04.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 0x7fabc806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc06f180 msgr2=0x7fabdc159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc06f180 0x7fabdc159290 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fabd000c520 tx=0x7fabd000c830 comp rx=0 tx=0).stop 2023-10-16T10:03:04.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 shutdown_connections 2023-10-16T10:03:04.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabdc15b540 0x7fabdc1597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabdc06afe0 0x7fabdc0b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fabc806b140 0x7fabc806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 --2- 172.21.15.158:0/2837396433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc06f180 0x7fabdc159290 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.610+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 >> 172.21.15.158:0/2837396433 conn(0x7fabdc0d03b0 msgr2=0x7fabdc0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:04.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.611+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 shutdown_connections 2023-10-16T10:03:04.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.611+0000 7fabe1895700 1 -- 172.21.15.158:0/2837396433 wait complete. 2023-10-16T10:03:04.615 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:04.610+0000 7ff67f63e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:03:04.626 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:03:04.658 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:04.657+0000 7f448077a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:03:04.686 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:04.684+0000 7ff67f63e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:03:04.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:04.734+0000 7f448077a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:03:04.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.751+0000 7fd5242c1700 1 Processor -- start 2023-10-16T10:03:04.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.751+0000 7fd5242c1700 1 -- start start 2023-10-16T10:03:04.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd5242c1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd51c160d40 0x7fd51c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd5242c1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd51c0cf420 0x7fd51c0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd5242c1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0c5320 con 0x7fd51c0c4ef0 2023-10-16T10:03:04.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd5242c1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd51c0c61e0 con 0x7fd51c0cf420 2023-10-16T10:03:04.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd5242c1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd51c0ce110 con 0x7fd51c160d40 2023-10-16T10:03:04.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd52185c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd51c0cf420 0x7fd51c0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd52285e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd51c0c4ef0 0x7fd51c06d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57568/0 (socket says 172.21.15.158:57568) 2023-10-16T10:03:04.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd52285e700 1 -- 172.21.15.158:0/2663462242 learned_addr learned my addr 172.21.15.158:0/2663462242 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:04.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.752+0000 7fd52205d700 1 --2- 172.21.15.158:0/2663462242 >> v2:172.21.15.172:3300/0 conn(0x7fd51c160d40 0x7fd51c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52185c700 1 -- 172.21.15.158:0/2663462242 >> v2:172.21.15.172:3300/0 conn(0x7fd51c160d40 msgr2=0x7fd51c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:04.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52185c700 1 --2- 172.21.15.158:0/2663462242 >> v2:172.21.15.172:3300/0 conn(0x7fd51c160d40 0x7fd51c0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52185c700 1 -- 172.21.15.158:0/2663462242 >> v1:172.21.15.158:6789/0 conn(0x7fd51c0c4ef0 legacy=0x7fd51c06d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52105b700 1 -- 172.21.15.158:0/2663462242 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179420206 0 0) 0x7fd51c0c5320 con 0x7fd51c0c4ef0 2023-10-16T10:03:04.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52185c700 1 -- 172.21.15.158:0/2663462242 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c0ce3f0 con 0x7fd51c0cf420 2023-10-16T10:03:04.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52185c700 1 --2- 172.21.15.158:0/2663462242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c0cf420 0x7fd51c0c49b0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0d0b80 tx=0x7fd518009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d28171a7182fff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:04.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52105b700 1 -- 172.21.15.158:0/2663462242 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518011830 con 0x7fd51c0cf420 2023-10-16T10:03:04.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52105b700 1 -- 172.21.15.158:0/2663462242 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518011990 con 0x7fd51c0cf420 2023-10-16T10:03:04.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd52105b700 1 -- 172.21.15.158:0/2663462242 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518011d20 con 0x7fd51c0cf420 2023-10-16T10:03:04.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd5242c1700 1 -- 172.21.15.158:0/2663462242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c0cf420 msgr2=0x7fd51c0c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.753+0000 7fd5242c1700 1 --2- 172.21.15.158:0/2663462242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c0cf420 0x7fd51c0c49b0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0d0b80 tx=0x7fd518009b20 comp rx=0 tx=0).stop 2023-10-16T10:03:04.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.754+0000 7fd5242c1700 1 -- 172.21.15.158:0/2663462242 shutdown_connections 2023-10-16T10:03:04.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.754+0000 7fd5242c1700 1 --2- 172.21.15.158:0/2663462242 >> v2:172.21.15.172:3300/0 conn(0x7fd51c160d40 0x7fd51c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.754+0000 7fd5242c1700 1 --2- 172.21.15.158:0/2663462242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c0cf420 0x7fd51c0c49b0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.754+0000 7fd5242c1700 1 -- 172.21.15.158:0/2663462242 >> 172.21.15.158:0/2663462242 conn(0x7fd51c0cc5b0 msgr2=0x7fd51c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:04.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.754+0000 7fd5242c1700 1 -- 172.21.15.158:0/2663462242 shutdown_connections 2023-10-16T10:03:04.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.754+0000 7fd5242c1700 1 -- 172.21.15.158:0/2663462242 wait complete. 2023-10-16T10:03:04.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.755+0000 7fd5242c1700 1 Processor -- start 2023-10-16T10:03:04.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.755+0000 7fd5242c1700 1 -- start start 2023-10-16T10:03:04.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.755+0000 7fd5242c1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd51c0c4ef0 0x7fd51c15a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.755+0000 7fd5242c1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd51c0cf420 0x7fd51c0b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.755+0000 7fd5242c1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c160d40 0x7fd51c1e73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.755+0000 7fd5242c1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c0b0470 con 0x7fd51c160d40 2023-10-16T10:03:04.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd5242c1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c0b1500 con 0x7fd51c0cf420 2023-10-16T10:03:04.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd5242c1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c0af860 con 0x7fd51c0c4ef0 2023-10-16T10:03:04.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52285e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c160d40 0x7fd51c1e73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52285e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c160d40 0x7fd51c1e73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57398/0 (socket says 172.21.15.158:57398) 2023-10-16T10:03:04.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52285e700 1 -- 172.21.15.158:0/4043022076 learned_addr learned my addr 172.21.15.158:0/4043022076 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:04.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52185c700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd51c0cf420 0x7fd51c0b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52205d700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd51c0c4ef0 0x7fd51c15a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:04.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52185c700 1 -- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd51c0c4ef0 msgr2=0x7fd51c15a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52185c700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd51c0c4ef0 0x7fd51c15a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52185c700 1 -- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c160d40 msgr2=0x7fd51c1e73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52185c700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c160d40 0x7fd51c1e73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52185c700 1 -- 172.21.15.158:0/4043022076 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5180094a0 con 0x7fd51c0cf420 2023-10-16T10:03:04.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52285e700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c160d40 0x7fd51c1e73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:04.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd52185c700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd51c0cf420 0x7fd51c0b0fc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd518016000 tx=0x7fd518004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:04.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd512ffd700 1 -- 172.21.15.158:0/4043022076 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180036a0 con 0x7fd51c0cf420 2023-10-16T10:03:04.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd512ffd700 1 -- 172.21.15.158:0/4043022076 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51801f070 con 0x7fd51c0cf420 2023-10-16T10:03:04.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51c0afb40 con 0x7fd51c0cf420 2023-10-16T10:03:04.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd512ffd700 1 -- 172.21.15.158:0/4043022076 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518003a40 con 0x7fd51c0cf420 2023-10-16T10:03:04.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.756+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c0b0100 con 0x7fd51c0cf420 2023-10-16T10:03:04.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.757+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51c0b0290 con 0x7fd51c0cf420 2023-10-16T10:03:04.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.758+0000 7fd512ffd700 1 -- 172.21.15.158:0/4043022076 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd518003ba0 con 0x7fd51c0cf420 2023-10-16T10:03:04.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.758+0000 7fd512ffd700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 0x7fd50006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:04.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.758+0000 7fd52205d700 1 -- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 msgr2=0x7fd50006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:04.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.758+0000 7fd512ffd700 1 -- 172.21.15.158:0/4043022076 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fd518021070 con 0x7fd51c0cf420 2023-10-16T10:03:04.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.758+0000 7fd52205d700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 0x7fd50006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:04.769 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:04.765+0000 7ff67f63e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:03:04.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.763+0000 7fd512ffd700 1 -- 172.21.15.158:0/4043022076 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd518055f20 con 0x7fd51c0cf420 2023-10-16T10:03:04.821 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:04.820+0000 7f448077a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:03:04.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.959+0000 7fd52205d700 1 -- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 msgr2=0x7fd50006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:04.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.959+0000 7fd52205d700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 0x7fd50006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:04.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.959+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd51c0c6be0 con 0x7fd51c0cf420 2023-10-16T10:03:04.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.960+0000 7fd512ffd700 1 -- 172.21.15.158:0/4043022076 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd518059540 con 0x7fd51c0cf420 2023-10-16T10:03:04.962 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:03:04.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 msgr2=0x7fd50006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:04.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 0x7fd50006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd51c0cf420 msgr2=0x7fd51c0b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:04.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd51c0cf420 0x7fd51c0b0fc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd518016000 tx=0x7fd518004750 comp rx=0 tx=0).stop 2023-10-16T10:03:04.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 shutdown_connections 2023-10-16T10:03:04.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fd50006b140 0x7fd50006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd51c160d40 0x7fd51c1e73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd51c0c4ef0 0x7fd51c15a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 --2- 172.21.15.158:0/4043022076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd51c0cf420 0x7fd51c0b0fc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:04.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 >> 172.21.15.158:0/4043022076 conn(0x7fd51c0cc5b0 msgr2=0x7fd51c0cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:04.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 shutdown_connections 2023-10-16T10:03:04.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:04.962+0000 7fd5242c1700 1 -- 172.21.15.158:0/4043022076 wait complete. 2023-10-16T10:03:04.976 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:03:05.136 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:05.134+0000 7ff67f63e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:03:05.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.143+0000 7f8438433700 1 Processor -- start 2023-10-16T10:03:05.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.143+0000 7f8438433700 1 -- start start 2023-10-16T10:03:05.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f8438433700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84300d0c60 0x7f84300b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f8438433700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f84300b3460 0x7f84300b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f8438433700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84300b4420 con 0x7f843015b6b0 2023-10-16T10:03:05.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f8438433700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843015ba90 con 0x7f84300b3460 2023-10-16T10:03:05.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f8438433700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84300b38f0 con 0x7f84300d0c60 2023-10-16T10:03:05.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f84361cf700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f843015b6b0 0x7f84300cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57608/0 (socket says 172.21.15.158:57608) 2023-10-16T10:03:05.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f84361cf700 1 -- 172.21.15.158:0/984132178 learned_addr learned my addr 172.21.15.158:0/984132178 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:05.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f84359ce700 1 --2- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84300d0c60 0x7f84300b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f84369d0700 1 --2- 172.21.15.158:0/984132178 >> v2:172.21.15.172:3300/0 conn(0x7f84300b3460 0x7f84300b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f84351cd700 1 -- 172.21.15.158:0/984132178 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561237545 0 0) 0x7f84300b4420 con 0x7f843015b6b0 2023-10-16T10:03:05.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.144+0000 7f84351cd700 1 -- 172.21.15.158:0/984132178 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f841c0035b0 con 0x7f843015b6b0 2023-10-16T10:03:05.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84369d0700 1 -- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84300d0c60 msgr2=0x7f84300b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:05.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84369d0700 1 --2- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84300d0c60 0x7f84300b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84369d0700 1 -- 172.21.15.158:0/984132178 >> v1:172.21.15.158:6789/0 conn(0x7f843015b6b0 legacy=0x7f84300cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84369d0700 1 -- 172.21.15.158:0/984132178 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84300cbc20 con 0x7f84300b3460 2023-10-16T10:03:05.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84359ce700 1 --2- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84300d0c60 0x7f84300b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:05.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84369d0700 1 --2- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300b3460 0x7f84300b48b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8428009d90 tx=0x7f842800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab511366025f8a67 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:05.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84351cd700 1 -- 172.21.15.158:0/984132178 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842800e070 con 0x7f84300b3460 2023-10-16T10:03:05.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84351cd700 1 -- 172.21.15.158:0/984132178 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842800bb20 con 0x7f84300b3460 2023-10-16T10:03:05.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.145+0000 7f84351cd700 1 -- 172.21.15.158:0/984132178 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428012430 con 0x7f84300b3460 2023-10-16T10:03:05.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 -- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300b3460 msgr2=0x7f84300b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 --2- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300b3460 0x7f84300b48b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8428009d90 tx=0x7f842800b330 comp rx=0 tx=0).stop 2023-10-16T10:03:05.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 -- 172.21.15.158:0/984132178 shutdown_connections 2023-10-16T10:03:05.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 --2- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300b3460 0x7f84300b48b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 --2- 172.21.15.158:0/984132178 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84300d0c60 0x7f84300b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 -- 172.21.15.158:0/984132178 >> 172.21.15.158:0/984132178 conn(0x7f84300cdf20 msgr2=0x7f84300ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:05.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 -- 172.21.15.158:0/984132178 shutdown_connections 2023-10-16T10:03:05.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.146+0000 7f8438433700 1 -- 172.21.15.158:0/984132178 wait complete. 2023-10-16T10:03:05.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.147+0000 7f8438433700 1 Processor -- start 2023-10-16T10:03:05.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.147+0000 7f8438433700 1 -- start start 2023-10-16T10:03:05.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f8438433700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84300b3460 0x7f8430157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f8438433700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300d0c60 0x7f84300ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f8438433700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f843015b6b0 0x7f84300a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f8438433700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84300a8f40 con 0x7f84300b3460 2023-10-16T10:03:05.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f8438433700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84300a9220 con 0x7f84300d0c60 2023-10-16T10:03:05.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f8438433700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84300a9500 con 0x7f843015b6b0 2023-10-16T10:03:05.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f84361cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84300b3460 0x7f8430157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f84361cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84300b3460 0x7f8430157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57438/0 (socket says 172.21.15.158:57438) 2023-10-16T10:03:05.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.148+0000 7f84361cf700 1 -- 172.21.15.158:0/1946818181 learned_addr learned my addr 172.21.15.158:0/1946818181 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:05.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84359ce700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300d0c60 0x7f84300ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84369d0700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f843015b6b0 0x7f84300a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84359ce700 1 -- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f843015b6b0 msgr2=0x7f84300a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84359ce700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f843015b6b0 0x7f84300a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84359ce700 1 -- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84300b3460 msgr2=0x7f8430157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84359ce700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84300b3460 0x7f8430157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84359ce700 1 -- 172.21.15.158:0/1946818181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200094a0 con 0x7f84300d0c60 2023-10-16T10:03:05.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f84359ce700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300d0c60 0x7f84300ad7a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f842000e980 tx=0x7f842000ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:05.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f8426ffd700 1 -- 172.21.15.158:0/1946818181 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842000c690 con 0x7f84300d0c60 2023-10-16T10:03:05.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f8426ffd700 1 -- 172.21.15.158:0/1946818181 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842000c7f0 con 0x7f84300d0c60 2023-10-16T10:03:05.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8428009a70 con 0x7f84300d0c60 2023-10-16T10:03:05.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f8426ffd700 1 -- 172.21.15.158:0/1946818181 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842000cba0 con 0x7f84300d0c60 2023-10-16T10:03:05.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.149+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84300a9ac0 con 0x7f84300d0c60 2023-10-16T10:03:05.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.150+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84300681b0 con 0x7f84300d0c60 2023-10-16T10:03:05.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.151+0000 7f8426ffd700 1 -- 172.21.15.158:0/1946818181 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f842000cd00 con 0x7f84300d0c60 2023-10-16T10:03:05.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.151+0000 7f8426ffd700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 0x7f841406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.151+0000 7f84361cf700 1 -- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 msgr2=0x7f841406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:05.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.151+0000 7f8426ffd700 1 -- 172.21.15.158:0/1946818181 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f8420089d80 con 0x7f84300d0c60 2023-10-16T10:03:05.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.151+0000 7f84361cf700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 0x7f841406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:05.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.156+0000 7f8426ffd700 1 -- 172.21.15.158:0/1946818181 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8420054720 con 0x7f84300d0c60 2023-10-16T10:03:05.211 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:05.210+0000 7f448077a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:03:05.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.352+0000 7f84361cf700 1 -- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 msgr2=0x7f841406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:05.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.352+0000 7f84361cf700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 0x7f841406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:05.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.357+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84300cb590 con 0x7f84300d0c60 2023-10-16T10:03:05.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.358+0000 7f8426ffd700 1 -- 172.21.15.158:0/1946818181 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84200210a0 con 0x7f84300d0c60 2023-10-16T10:03:05.359 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:03:05.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 msgr2=0x7f841406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 0x7f841406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300d0c60 msgr2=0x7f84300ad7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300d0c60 0x7f84300ad7a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f842000e980 tx=0x7f842000ec90 comp rx=0 tx=0).stop 2023-10-16T10:03:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 shutdown_connections 2023-10-16T10:03:05.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f843015b6b0 0x7f84300a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f841406b140 0x7f841406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84300b3460 0x7f8430157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 --2- 172.21.15.158:0/1946818181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84300d0c60 0x7f84300ad7a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 >> 172.21.15.158:0/1946818181 conn(0x7f84300cdf20 msgr2=0x7f84300ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:05.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 shutdown_connections 2023-10-16T10:03:05.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.360+0000 7f8438433700 1 -- 172.21.15.158:0/1946818181 wait complete. 2023-10-16T10:03:05.374 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T10:03:05.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.497+0000 7f1ffcd86700 1 Processor -- start 2023-10-16T10:03:05.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.497+0000 7f1ffcd86700 1 -- start start 2023-10-16T10:03:05.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ffcd86700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ff80caaa0 0x7f1ff80cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ffcd86700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff806d740 0x7f1ff806dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ffcd86700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff80af510 con 0x7f1ff806bbc0 2023-10-16T10:03:05.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ffcd86700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff806a8d0 con 0x7f1ff80caaa0 2023-10-16T10:03:05.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ffcd86700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80cb3c0 con 0x7f1ff806d740 2023-10-16T10:03:05.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ff80caaa0 0x7f1ff80cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ff80caaa0 0x7f1ff80cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57448/0 (socket says 172.21.15.158:57448) 2023-10-16T10:03:05.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 -- 172.21.15.158:0/4239083193 learned_addr learned my addr 172.21.15.158:0/4239083193 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:05.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff6d9d700 1 --2- 172.21.15.158:0/4239083193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff806d740 0x7f1ff806dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 -- 172.21.15.158:0/4239083193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff806d740 msgr2=0x7f1ff806dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 --2- 172.21.15.158:0/4239083193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff806d740 0x7f1ff806dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 -- 172.21.15.158:0/4239083193 >> v1:172.21.15.172:6789/0 conn(0x7f1ff806bbc0 legacy=0x7f1ff80cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 -- 172.21.15.158:0/4239083193 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff806ab20 con 0x7f1ff80caaa0 2023-10-16T10:03:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff659c700 1 --2- 172.21.15.158:0/4239083193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff80caaa0 0x7f1ff80cae80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80098f0 tx=0x7f1fe8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad38fd5bbc21cdd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.498+0000 7f1ff559a700 1 -- 172.21.15.158:0/4239083193 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8011830 con 0x7f1ff80caaa0 2023-10-16T10:03:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ff559a700 1 -- 172.21.15.158:0/4239083193 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe8011990 con 0x7f1ff80caaa0 2023-10-16T10:03:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ff559a700 1 -- 172.21.15.158:0/4239083193 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8011d20 con 0x7f1ff80caaa0 2023-10-16T10:03:05.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 -- 172.21.15.158:0/4239083193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff80caaa0 msgr2=0x7f1ff80cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/4239083193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff80caaa0 0x7f1ff80cae80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80098f0 tx=0x7f1fe8009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 -- 172.21.15.158:0/4239083193 shutdown_connections 2023-10-16T10:03:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/4239083193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff806d740 0x7f1ff806dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/4239083193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff80caaa0 0x7f1ff80cae80 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 -- 172.21.15.158:0/4239083193 >> 172.21.15.158:0/4239083193 conn(0x7f1ff80cc650 msgr2=0x7f1ff80cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 -- 172.21.15.158:0/4239083193 shutdown_connections 2023-10-16T10:03:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.499+0000 7f1ffcd86700 1 -- 172.21.15.158:0/4239083193 wait complete. 2023-10-16T10:03:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 Processor -- start 2023-10-16T10:03:05.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 -- start start 2023-10-16T10:03:05.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff806bbc0 0x7f1ff8071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff806d740 0x7f1ff81f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff80caaa0 0x7f1ff80c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff81f7250 con 0x7f1ff806d740 2023-10-16T10:03:05.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff81f7530 con 0x7f1ff806bbc0 2023-10-16T10:03:05.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ffcd86700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80c80d0 con 0x7f1ff80caaa0 2023-10-16T10:03:05.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.501+0000 7f1ff5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff806d740 0x7f1ff81f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff806d740 0x7f1ff81f67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57456/0 (socket says 172.21.15.158:57456) 2023-10-16T10:03:05.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 -- 172.21.15.158:0/3894402857 learned_addr learned my addr 172.21.15.158:0/3894402857 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:05.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff6d9d700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff80caaa0 0x7f1ff80c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff659c700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff806bbc0 0x7f1ff8071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 -- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff80caaa0 msgr2=0x7f1ff80c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff80caaa0 0x7f1ff80c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 -- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff806bbc0 msgr2=0x7f1ff8071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff806bbc0 0x7f1ff8071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 -- 172.21.15.158:0/3894402857 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8009550 con 0x7f1ff806d740 2023-10-16T10:03:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ff5d9b700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff806d740 0x7f1ff81f67b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c4e0 tx=0x7f1fec00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1fe6ffd700 1 -- 172.21.15.158:0/3894402857 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0052d0 con 0x7f1ff806d740 2023-10-16T10:03:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1fe6ffd700 1 -- 172.21.15.158:0/3894402857 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec005430 con 0x7f1ff806d740 2023-10-16T10:03:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff80c83b0 con 0x7f1ff806d740 2023-10-16T10:03:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1fe6ffd700 1 -- 172.21.15.158:0/3894402857 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0038c0 con 0x7f1ff806d740 2023-10-16T10:03:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.502+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff81f3420 con 0x7f1ff806d740 2023-10-16T10:03:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.504+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff80c85a0 con 0x7f1ff806d740 2023-10-16T10:03:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.504+0000 7f1fe6ffd700 1 -- 172.21.15.158:0/3894402857 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f1fec003a20 con 0x7f1ff806d740 2023-10-16T10:03:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.504+0000 7f1fe6ffd700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 0x7f1fd806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.504+0000 7f1ff659c700 1 -- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 msgr2=0x7f1fd806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.504+0000 7f1fe6ffd700 1 -- 172.21.15.158:0/3894402857 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f1fec088f40 con 0x7f1ff806d740 2023-10-16T10:03:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.504+0000 7f1ff659c700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 0x7f1fd806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.508+0000 7f1fe6ffd700 1 -- 172.21.15.158:0/3894402857 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1fec053930 con 0x7f1ff806d740 2023-10-16T10:03:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.705+0000 7f1ff659c700 1 -- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 msgr2=0x7f1fd806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.705+0000 7f1ff659c700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 0x7f1fd806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:05.710 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:05.709+0000 7ff67f63e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:03:05.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.775+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1ff80af510 con 0x7f1ff806d740 2023-10-16T10:03:05.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.776+0000 7f1fe6ffd700 1 -- 172.21.15.158:0/3894402857 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v127) v1 ==== 80+0+478 (secure 0 0 0) 0x7f1ff80af510 con 0x7f1ff806d740 2023-10-16T10:03:05.777 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1769168841 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.778 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3578502848 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.778 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2674552982 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.778 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2360787591 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.778 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/1080332904 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.779 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/1080332904 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.779 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3568191591 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.779 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1741897775 2023-10-17T10:02:23.842872+0000 2023-10-16T10:03:05.780 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:05.778+0000 7ff67f63e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:03:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 msgr2=0x7f1fd806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 0x7f1fd806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff806d740 msgr2=0x7f1ff81f67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:05.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff806d740 0x7f1ff81f67b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c4e0 tx=0x7f1fec00c7f0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 shutdown_connections 2023-10-16T10:03:05.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ff80caaa0 0x7f1ff80c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ff806bbc0 0x7f1ff8071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f1fd806b140 0x7f1fd806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 --2- 172.21.15.158:0/3894402857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ff806d740 0x7f1ff81f67b0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:05.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 >> 172.21.15.158:0/3894402857 conn(0x7f1ff80cc650 msgr2=0x7f1ff806dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:05.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 shutdown_connections 2023-10-16T10:03:05.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:05.778+0000 7f1ffcd86700 1 -- 172.21.15.158:0/3894402857 wait complete. 2023-10-16T10:03:05.784 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T10:03:05.793 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T10:03:05.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:05.794 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T10:03:05.819 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:05.818+0000 7f448077a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:03:05.830 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T10:03:05.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:05.831 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T10:03:05.888 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:05.888 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:05.897 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:05.896+0000 7f448077a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:03:05.905 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:05.906 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:05.906 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:05.923 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:03:05.923 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:05.924 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:05.963 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:05.963 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:06.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.083+0000 7fa3af87e700 1 Processor -- start 2023-10-16T10:03:06.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.084+0000 7fa3af87e700 1 -- start start 2023-10-16T10:03:06.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.084+0000 7fa3af87e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a81683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3af87e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a80d3bd0 con 0x7fa3a80d1210 2023-10-16T10:03:06.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3af87e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a80d2fc0 con 0x7fa3a815d7c0 2023-10-16T10:03:06.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3af87e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80d4060 con 0x7fa3a80b7810 2023-10-16T10:03:06.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3ace19700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa3a80d1210 0x7fa3a80ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57642/0 (socket says 172.21.15.158:57642) 2023-10-16T10:03:06.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3ace19700 1 -- 172.21.15.158:0/270898033 learned_addr learned my addr 172.21.15.158:0/270898033 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:06.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3ade1b700 1 --2- 172.21.15.158:0/270898033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a81683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa39ffff700 1 -- 172.21.15.158:0/270898033 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617034689 0 0) 0x7fa3a80d3bd0 con 0x7fa3a80d1210 2023-10-16T10:03:06.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa39ffff700 1 -- 172.21.15.158:0/270898033 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3940035b0 con 0x7fa3a80d1210 2023-10-16T10:03:06.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa39ffff700 1 -- 172.21.15.158:0/270898033 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511832426 0 0) 0x7fa3a80d2fc0 con 0x7fa3a815d7c0 2023-10-16T10:03:06.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa39ffff700 1 -- 172.21.15.158:0/270898033 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a80d3bd0 con 0x7fa3a815d7c0 2023-10-16T10:03:06.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3ade1b700 1 -- 172.21.15.158:0/270898033 >> v1:172.21.15.172:6789/0 conn(0x7fa3a815d7c0 legacy=0x7fa3a80d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3ade1b700 1 -- 172.21.15.158:0/270898033 >> v1:172.21.15.158:6789/0 conn(0x7fa3a80d1210 legacy=0x7fa3a80ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3ade1b700 1 -- 172.21.15.158:0/270898033 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a80d32a0 con 0x7fa3a80b7810 2023-10-16T10:03:06.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa3ade1b700 1 --2- 172.21.15.158:0/270898033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a81683f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009dc0 tx=0x7fa3a000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0988a7f9b33fe4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:06.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa39ffff700 1 -- 172.21.15.158:0/270898033 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000e070 con 0x7fa3a80b7810 2023-10-16T10:03:06.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa39ffff700 1 -- 172.21.15.158:0/270898033 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a000bb20 con 0x7fa3a80b7810 2023-10-16T10:03:06.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.085+0000 7fa39ffff700 1 -- 172.21.15.158:0/270898033 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0012430 con 0x7fa3a80b7810 2023-10-16T10:03:06.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.086+0000 7fa3af87e700 1 -- 172.21.15.158:0/270898033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 msgr2=0x7fa3a81683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.086+0000 7fa3af87e700 1 --2- 172.21.15.158:0/270898033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a81683f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009dc0 tx=0x7fa3a000b330 comp rx=0 tx=0).stop 2023-10-16T10:03:06.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.086+0000 7fa3af87e700 1 -- 172.21.15.158:0/270898033 shutdown_connections 2023-10-16T10:03:06.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.086+0000 7fa3af87e700 1 --2- 172.21.15.158:0/270898033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a81683f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.086+0000 7fa3af87e700 1 -- 172.21.15.158:0/270898033 >> 172.21.15.158:0/270898033 conn(0x7fa3a80669f0 msgr2=0x7fa3a8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:06.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.086+0000 7fa3af87e700 1 -- 172.21.15.158:0/270898033 shutdown_connections 2023-10-16T10:03:06.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.086+0000 7fa3af87e700 1 -- 172.21.15.158:0/270898033 wait complete. 2023-10-16T10:03:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.087+0000 7fa3af87e700 1 Processor -- start 2023-10-16T10:03:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.087+0000 7fa3af87e700 1 -- start start 2023-10-16T10:03:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3af87e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a80c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3af87e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3a80d1210 0x7fa3a81f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3af87e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a815d7c0 0x7fa3a81f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3af87e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80c9ec0 con 0x7fa3a815d7c0 2023-10-16T10:03:06.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3af87e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80c8ff0 con 0x7fa3a80d1210 2023-10-16T10:03:06.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3af87e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a81f73c0 con 0x7fa3a80b7810 2023-10-16T10:03:06.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3ade1b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a815d7c0 0x7fa3a81f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3ade1b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a815d7c0 0x7fa3a81f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57474/0 (socket says 172.21.15.158:57474) 2023-10-16T10:03:06.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3ade1b700 1 -- 172.21.15.158:0/2138367799 learned_addr learned my addr 172.21.15.158:0/2138367799 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3ad61a700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a80c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.088+0000 7fa3ace19700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3a80d1210 0x7fa3a81f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3ade1b700 1 -- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 msgr2=0x7fa3a80c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3ade1b700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a80c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3ade1b700 1 -- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3a80d1210 msgr2=0x7fa3a81f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3ade1b700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3a80d1210 0x7fa3a81f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3ade1b700 1 -- 172.21.15.158:0/2138367799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0009a70 con 0x7fa3a815d7c0 2023-10-16T10:03:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3ade1b700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a815d7c0 0x7fa3a81f2710 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0014040 tx=0x7fa3a0004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa39dffb700 1 -- 172.21.15.158:0/2138367799 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0012430 con 0x7fa3a815d7c0 2023-10-16T10:03:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa39dffb700 1 -- 172.21.15.158:0/2138367799 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0020070 con 0x7fa3a815d7c0 2023-10-16T10:03:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a80c9240 con 0x7fa3a815d7c0 2023-10-16T10:03:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa39dffb700 1 -- 172.21.15.158:0/2138367799 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a001ba90 con 0x7fa3a815d7c0 2023-10-16T10:03:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.089+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a80c9800 con 0x7fa3a815d7c0 2023-10-16T10:03:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.090+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a80ca000 con 0x7fa3a815d7c0 2023-10-16T10:03:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.090+0000 7fa39dffb700 1 -- 172.21.15.158:0/2138367799 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa3a001bbf0 con 0x7fa3a815d7c0 2023-10-16T10:03:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.091+0000 7fa39dffb700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 0x7fa38c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.091+0000 7fa3ad61a700 1 -- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 msgr2=0x7fa38c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.091+0000 7fa39dffb700 1 -- 172.21.15.158:0/2138367799 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fa3a0022070 con 0x7fa3a815d7c0 2023-10-16T10:03:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.091+0000 7fa3ad61a700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 0x7fa38c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.095+0000 7fa39dffb700 1 -- 172.21.15.158:0/2138367799 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3a80ca000 con 0x7fa3a815d7c0 2023-10-16T10:03:06.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.292+0000 7fa3ad61a700 1 -- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 msgr2=0x7fa38c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:06.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.292+0000 7fa3ad61a700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 0x7fa38c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.344+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3a80ca1e0 con 0x7fa3a815d7c0 2023-10-16T10:03:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.344+0000 7fa39dffb700 1 -- 172.21.15.158:0/2138367799 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fa3a80ca1e0 con 0x7fa3a815d7c0 2023-10-16T10:03:06.346 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:06.346 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/94308149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":94308149},{"type":"v1","addr":"172.21.15.172:6834","nonce":94308149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/4112063716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4112063716},{"type":"v1","addr":"172.21.15.172:6835","nonce":4112063716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8274,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/658160836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":658160836},{"type":"v1","addr":"172.21.15.158:6836","nonce":658160836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:02:09.580772+0000","modified":"2023-10-16T10:02:12.619480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8277":{"gid":8277,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/3693875115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":3693875115},{"type":"v1","addr":"172.21.15.158:6837","nonce":3693875115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T10:03:06.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 msgr2=0x7fa38c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:06.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 0x7fa38c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a815d7c0 msgr2=0x7fa3a81f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a815d7c0 0x7fa3a81f2710 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0014040 tx=0x7fa3a0004b20 comp rx=0 tx=0).stop 2023-10-16T10:03:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 shutdown_connections 2023-10-16T10:03:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fa38c06b140 0x7fa38c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a815d7c0 0x7fa3a81f2710 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3a80b7810 0x7fa3a80c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 --2- 172.21.15.158:0/2138367799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3a80d1210 0x7fa3a81f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 >> 172.21.15.158:0/2138367799 conn(0x7fa3a80669f0 msgr2=0x7fa3a80b6800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 shutdown_connections 2023-10-16T10:03:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.347+0000 7fa3af87e700 1 -- 172.21.15.158:0/2138367799 wait complete. 2023-10-16T10:03:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 60 2023-10-16T10:03:06.361 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:06.423 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:06.422+0000 7ff67f63e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:06.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.487+0000 7fe128251700 1 Processor -- start 2023-10-16T10:03:06.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.487+0000 7fe128251700 1 -- start start 2023-10-16T10:03:06.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe128251700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006ab40 0x7fe12006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe128251700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe12006e5d0 0x7fe120071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe128251700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe12006af20 0x7fe1200722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe128251700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe120071e50 con 0x7fe12006e5d0 2023-10-16T10:03:06.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe128251700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe120161270 con 0x7fe12006af20 2023-10-16T10:03:06.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe128251700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe120161510 con 0x7fe12006ab40 2023-10-16T10:03:06.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe12006e5d0 0x7fe120071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe12006e5d0 0x7fe120071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57480/0 (socket says 172.21.15.158:57480) 2023-10-16T10:03:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 -- 172.21.15.158:0/2459281947 learned_addr learned my addr 172.21.15.158:0/2459281947 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe125fed700 1 --2- 172.21.15.158:0/2459281947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006ab40 0x7fe12006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1267ee700 1 --2- 172.21.15.158:0/2459281947 >> v2:172.21.15.172:3300/0 conn(0x7fe12006af20 0x7fe1200722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 -- 172.21.15.158:0/2459281947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006ab40 msgr2=0x7fe12006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 --2- 172.21.15.158:0/2459281947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006ab40 0x7fe12006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 -- 172.21.15.158:0/2459281947 >> v2:172.21.15.172:3300/0 conn(0x7fe12006af20 msgr2=0x7fe1200722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 --2- 172.21.15.158:0/2459281947 >> v2:172.21.15.172:3300/0 conn(0x7fe12006af20 0x7fe1200722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 -- 172.21.15.158:0/2459281947 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1201617f0 con 0x7fe12006e5d0 2023-10-16T10:03:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.488+0000 7fe1257ec700 1 --2- 172.21.15.158:0/2459281947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006e5d0 0x7fe120071910 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fe12006eaf0 tx=0x7fe110009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af42154179fd6a65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe124feb700 1 -- 172.21.15.158:0/2459281947 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110011830 con 0x7fe12006e5d0 2023-10-16T10:03:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe124feb700 1 -- 172.21.15.158:0/2459281947 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe110011990 con 0x7fe12006e5d0 2023-10-16T10:03:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe124feb700 1 -- 172.21.15.158:0/2459281947 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110011d20 con 0x7fe12006e5d0 2023-10-16T10:03:06.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 -- 172.21.15.158:0/2459281947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006e5d0 msgr2=0x7fe120071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 --2- 172.21.15.158:0/2459281947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006e5d0 0x7fe120071910 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fe12006eaf0 tx=0x7fe110009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 -- 172.21.15.158:0/2459281947 shutdown_connections 2023-10-16T10:03:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 --2- 172.21.15.158:0/2459281947 >> v2:172.21.15.172:3300/0 conn(0x7fe12006af20 0x7fe1200722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 --2- 172.21.15.158:0/2459281947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006ab40 0x7fe12006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 --2- 172.21.15.158:0/2459281947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006e5d0 0x7fe120071910 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 -- 172.21.15.158:0/2459281947 >> 172.21.15.158:0/2459281947 conn(0x7fe1200d02b0 msgr2=0x7fe1200d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 -- 172.21.15.158:0/2459281947 shutdown_connections 2023-10-16T10:03:06.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.489+0000 7fe128251700 1 -- 172.21.15.158:0/2459281947 wait complete. 2023-10-16T10:03:06.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.490+0000 7fe128251700 1 Processor -- start 2023-10-16T10:03:06.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe128251700 1 -- start start 2023-10-16T10:03:06.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe128251700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe12006ab40 0x7fe1200b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe128251700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006af20 0x7fe1200b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe128251700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006e5d0 0x7fe120149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe128251700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b5d40 con 0x7fe12006af20 2023-10-16T10:03:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe128251700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12014a3a0 con 0x7fe12006ab40 2023-10-16T10:03:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe128251700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b1240 con 0x7fe12006e5d0 2023-10-16T10:03:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe1257ec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006af20 0x7fe1200b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe1257ec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006af20 0x7fe1200b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57488/0 (socket says 172.21.15.158:57488) 2023-10-16T10:03:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe1257ec700 1 -- 172.21.15.158:0/1836691852 learned_addr learned my addr 172.21.15.158:0/1836691852 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe1267ee700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006e5d0 0x7fe120149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.491+0000 7fe125fed700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe12006ab40 0x7fe1200b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1257ec700 1 -- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006e5d0 msgr2=0x7fe120149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1257ec700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006e5d0 0x7fe120149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1257ec700 1 -- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe12006ab40 msgr2=0x7fe1200b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1257ec700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe12006ab40 0x7fe1200b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1257ec700 1 -- 172.21.15.158:0/1836691852 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110009550 con 0x7fe12006af20 2023-10-16T10:03:06.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1257ec700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006af20 0x7fe1200b5800 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fe110009c10 tx=0x7fe110004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1167fc700 1 -- 172.21.15.158:0/1836691852 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110004450 con 0x7fe12006af20 2023-10-16T10:03:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1167fc700 1 -- 172.21.15.158:0/1836691852 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11001f070 con 0x7fe12006af20 2023-10-16T10:03:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200b1520 con 0x7fe12006af20 2023-10-16T10:03:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe1167fc700 1 -- 172.21.15.158:0/1836691852 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1100038e0 con 0x7fe12006af20 2023-10-16T10:03:06.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.492+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200b1ae0 con 0x7fe12006af20 2023-10-16T10:03:06.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.493+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1200681b0 con 0x7fe12006af20 2023-10-16T10:03:06.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.494+0000 7fe1167fc700 1 -- 172.21.15.158:0/1836691852 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe110003a40 con 0x7fe12006af20 2023-10-16T10:03:06.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.494+0000 7fe1167fc700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 0x7fe10406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.494+0000 7fe125fed700 1 -- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 msgr2=0x7fe10406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:06.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.494+0000 7fe1167fc700 1 -- 172.21.15.158:0/1836691852 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fe110021070 con 0x7fe12006af20 2023-10-16T10:03:06.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.494+0000 7fe125fed700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 0x7fe10406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:06.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.499+0000 7fe1167fc700 1 -- 172.21.15.158:0/1836691852 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe110055cc0 con 0x7fe12006af20 2023-10-16T10:03:06.508 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:06.499+0000 7ff67f63e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:03:06.568 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:06.566+0000 7f448077a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:06.645 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:06.644+0000 7ff67f63e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:06.648 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:06.647+0000 7f448077a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:03:06.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.695+0000 7fe125fed700 1 -- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 msgr2=0x7fe10406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:06.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.695+0000 7fe125fed700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 0x7fe10406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:06.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.700+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1200b6230 con 0x7fe12006af20 2023-10-16T10:03:06.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.702+0000 7fe1167fc700 1 -- 172.21.15.158:0/1836691852 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+17882 (secure 0 0 0) 0x7fe110015090 con 0x7fe12006af20 2023-10-16T10:03:06.703 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:06.703 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":127,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:23.842920+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:02:08.021999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T10:02:08.539662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1769168841":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3578502848":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2674552982":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2360787591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6833/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6832/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3568191591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/1741897775":"2023-10-17T10:02:23.842872+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:06.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 msgr2=0x7fe10406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:06.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 0x7fe10406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006af20 msgr2=0x7fe1200b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006af20 0x7fe1200b5800 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fe110009c10 tx=0x7fe110004750 comp rx=0 tx=0).stop 2023-10-16T10:03:06.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 shutdown_connections 2023-10-16T10:03:06.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe12006e5d0 0x7fe120149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe12006ab40 0x7fe1200b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe10406b140 0x7fe10406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 --2- 172.21.15.158:0/1836691852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe12006af20 0x7fe1200b5800 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 >> 172.21.15.158:0/1836691852 conn(0x7fe1200d02b0 msgr2=0x7fe120072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:06.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 shutdown_connections 2023-10-16T10:03:06.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.704+0000 7fe128251700 1 -- 172.21.15.158:0/1836691852 wait complete. 2023-10-16T10:03:06.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:02:08.021999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:02:08.539662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:03:06.719 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:03:06.804 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:06.802+0000 7f448077a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.844+0000 7f204700a700 1 Processor -- start 2023-10-16T10:03:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.844+0000 7f204700a700 1 -- start start 2023-10-16T10:03:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f204700a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2040160d30 0x7f20400b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f204700a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20400d0530 0x7f20400d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f204700a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20400b7c90 con 0x7f20400d19a0 2023-10-16T10:03:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f204700a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20400b8910 con 0x7f20400d0530 2023-10-16T10:03:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f204700a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20400d0ef0 con 0x7f2040160d30 2023-10-16T10:03:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20400d0530 0x7f20400d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20400d0530 0x7f20400d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57514/0 (socket says 172.21.15.158:57514) 2023-10-16T10:03:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 -- 172.21.15.158:0/160906551 learned_addr learned my addr 172.21.15.158:0/160906551 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f2044da6700 1 --2- 172.21.15.158:0/160906551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2040160d30 0x7f20400b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 -- 172.21.15.158:0/160906551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2040160d30 msgr2=0x7f20400b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 --2- 172.21.15.158:0/160906551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2040160d30 0x7f20400b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203f7fe700 1 -- 172.21.15.158:0/160906551 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457644352 0 0) 0x7f20400b7c90 con 0x7f20400d19a0 2023-10-16T10:03:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 -- 172.21.15.158:0/160906551 >> v1:172.21.15.172:6789/0 conn(0x7f20400d19a0 legacy=0x7f20400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 -- 172.21.15.158:0/160906551 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20400b7f70 con 0x7f20400d0530 2023-10-16T10:03:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.845+0000 7f203ffff700 1 --2- 172.21.15.158:0/160906551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d0530 0x7f20400d09b0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f202c0098f0 tx=0x7f202c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0af6012d11b494 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f203f7fe700 1 -- 172.21.15.158:0/160906551 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c011830 con 0x7f20400d0530 2023-10-16T10:03:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f203f7fe700 1 -- 172.21.15.158:0/160906551 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c011990 con 0x7f20400d0530 2023-10-16T10:03:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f203f7fe700 1 -- 172.21.15.158:0/160906551 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c011d20 con 0x7f20400d0530 2023-10-16T10:03:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 -- 172.21.15.158:0/160906551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d0530 msgr2=0x7f20400d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 --2- 172.21.15.158:0/160906551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d0530 0x7f20400d09b0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f202c0098f0 tx=0x7f202c009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 -- 172.21.15.158:0/160906551 shutdown_connections 2023-10-16T10:03:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 --2- 172.21.15.158:0/160906551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2040160d30 0x7f20400b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 --2- 172.21.15.158:0/160906551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d0530 0x7f20400d09b0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 -- 172.21.15.158:0/160906551 >> 172.21.15.158:0/160906551 conn(0x7f20400669f0 msgr2=0x7f2040066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 -- 172.21.15.158:0/160906551 shutdown_connections 2023-10-16T10:03:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.846+0000 7f204700a700 1 -- 172.21.15.158:0/160906551 wait complete. 2023-10-16T10:03:06.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.848+0000 7f204700a700 1 Processor -- start 2023-10-16T10:03:06.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.848+0000 7f204700a700 1 -- start start 2023-10-16T10:03:06.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f204700a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20400d0530 0x7f20401f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f204700a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d19a0 0x7f204014a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f204700a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2040160d30 0x7f20400cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f204700a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20401f7090 con 0x7f20400d19a0 2023-10-16T10:03:06.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f204700a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20401f7370 con 0x7f2040160d30 2023-10-16T10:03:06.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f204700a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20401f7650 con 0x7f20400d0530 2023-10-16T10:03:06.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d19a0 0x7f204014a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d19a0 0x7f204014a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57520/0 (socket says 172.21.15.158:57520) 2023-10-16T10:03:06.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 -- 172.21.15.158:0/3245274391 learned_addr learned my addr 172.21.15.158:0/3245274391 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:06.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f2044da6700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20400d0530 0x7f20401f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f20455a7700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2040160d30 0x7f20400cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:06.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 -- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20400d0530 msgr2=0x7f20401f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20400d0530 0x7f20401f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 -- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2040160d30 msgr2=0x7f20400cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:06.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2040160d30 0x7f20400cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:06.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.849+0000 7f203ffff700 1 -- 172.21.15.158:0/3245274391 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c009550 con 0x7f20400d19a0 2023-10-16T10:03:06.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.850+0000 7f203ffff700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d19a0 0x7f204014a940 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f202c009c40 tx=0x7f202c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:06.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.850+0000 7f203d7fa700 1 -- 172.21.15.158:0/3245274391 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c004340 con 0x7f20400d19a0 2023-10-16T10:03:06.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.850+0000 7f203d7fa700 1 -- 172.21.15.158:0/3245274391 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c01f070 con 0x7f20400d19a0 2023-10-16T10:03:06.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.850+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20400c7190 con 0x7f20400d19a0 2023-10-16T10:03:06.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.850+0000 7f203d7fa700 1 -- 172.21.15.158:0/3245274391 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0036a0 con 0x7f20400d19a0 2023-10-16T10:03:06.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.850+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20401f6cd0 con 0x7f20400d19a0 2023-10-16T10:03:06.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.851+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20401f6e60 con 0x7f20400d19a0 2023-10-16T10:03:06.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.851+0000 7f203d7fa700 1 -- 172.21.15.158:0/3245274391 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f202c021030 con 0x7f20400d19a0 2023-10-16T10:03:06.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.852+0000 7f203d7fa700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 0x7f202406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:06.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.852+0000 7f2044da6700 1 -- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 msgr2=0x7f202406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:06.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.852+0000 7f203d7fa700 1 -- 172.21.15.158:0/3245274391 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f202c08b090 con 0x7f20400d19a0 2023-10-16T10:03:06.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.852+0000 7f2044da6700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 0x7f202406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:06.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:06.856+0000 7f203d7fa700 1 -- 172.21.15.158:0/3245274391 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f202c055a80 con 0x7f20400d19a0 2023-10-16T10:03:06.874 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:06.873+0000 7ff67f63e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:03:06.947 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:06.945+0000 7ff67f63e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:03:07.044 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:07.042+0000 7f448077a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:03:07.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.053+0000 7f2044da6700 1 -- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 msgr2=0x7f202406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:07.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.053+0000 7f2044da6700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 0x7f202406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:07.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.057+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20400c76b0 con 0x7f20400d19a0 2023-10-16T10:03:07.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.057+0000 7f203d7fa700 1 -- 172.21.15.158:0/3245274391 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7f202c0590a0 con 0x7f20400d19a0 2023-10-16T10:03:07.058 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:03:07.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 msgr2=0x7f202406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:07.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 0x7f202406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d19a0 msgr2=0x7f204014a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d19a0 0x7f204014a940 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f202c009c40 tx=0x7f202c004750 comp rx=0 tx=0).stop 2023-10-16T10:03:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 shutdown_connections 2023-10-16T10:03:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2040160d30 0x7f20400cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20400d0530 0x7f20401f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f202406b140 0x7f202406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 --2- 172.21.15.158:0/3245274391 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20400d19a0 0x7f204014a940 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 >> 172.21.15.158:0/3245274391 conn(0x7f20400669f0 msgr2=0x7f20400b5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 shutdown_connections 2023-10-16T10:03:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.059+0000 7f204700a700 1 -- 172.21.15.158:0/3245274391 wait complete. 2023-10-16T10:03:07.073 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:03:07.134 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:07.133+0000 7f448077a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:03:07.210 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:07.209+0000 7ff67f63e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:03:07.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.210+0000 7f0c5f77a700 1 Processor -- start 2023-10-16T10:03:07.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.210+0000 7f0c5f77a700 1 -- start start 2023-10-16T10:03:07.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5f77a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0c5815d6c0 0x7f0c580cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5f77a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c580d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5f77a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c580d1ee0 con 0x7f0c5815daa0 2023-10-16T10:03:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5f77a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c580d40a0 con 0x7f0c5815d6c0 2023-10-16T10:03:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5f77a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580d12d0 con 0x7f0c580cf360 2023-10-16T10:03:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5d516700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0c5815d6c0 0x7f0c580cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5dd17700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c580d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5cd15700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0c5815daa0 0x7f0c580cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:33378/0 (socket says 172.21.15.158:33378) 2023-10-16T10:03:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.211+0000 7f0c5cd15700 1 -- 172.21.15.158:0/2366876067 learned_addr learned my addr 172.21.15.158:0/2366876067 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c5d516700 1 -- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 msgr2=0x7f0c580d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c5d516700 1 --2- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c580d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c5d516700 1 -- 172.21.15.158:0/2366876067 >> v1:172.21.15.172:6789/0 conn(0x7f0c5815daa0 legacy=0x7f0c580cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c5d516700 1 -- 172.21.15.158:0/2366876067 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580d15b0 con 0x7f0c5815d6c0 2023-10-16T10:03:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c5dd17700 1 --2- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c580d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c5d516700 1 --2- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815d6c0 0x7f0c580cff30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009840 tx=0x7f0c54009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e9f8c377ee05633 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c4ffff700 1 -- 172.21.15.158:0/2366876067 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54011830 con 0x7f0c5815d6c0 2023-10-16T10:03:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c4ffff700 1 -- 172.21.15.158:0/2366876067 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54011990 con 0x7f0c5815d6c0 2023-10-16T10:03:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.212+0000 7f0c4ffff700 1 -- 172.21.15.158:0/2366876067 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54011d20 con 0x7f0c5815d6c0 2023-10-16T10:03:07.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 -- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815d6c0 msgr2=0x7f0c580cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815d6c0 0x7f0c580cff30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009840 tx=0x7f0c54009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:07.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 -- 172.21.15.158:0/2366876067 shutdown_connections 2023-10-16T10:03:07.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c580d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/2366876067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815d6c0 0x7f0c580cff30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 -- 172.21.15.158:0/2366876067 >> 172.21.15.158:0/2366876067 conn(0x7f0c5809c940 msgr2=0x7f0c580d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 -- 172.21.15.158:0/2366876067 shutdown_connections 2023-10-16T10:03:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.213+0000 7f0c5f77a700 1 -- 172.21.15.158:0/2366876067 wait complete. 2023-10-16T10:03:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.214+0000 7f0c5f77a700 1 Processor -- start 2023-10-16T10:03:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.214+0000 7f0c5f77a700 1 -- start start 2023-10-16T10:03:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5f77a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c5815e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5f77a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c5815d6c0 0x7f0c580b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5f77a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815daa0 0x7f0c581f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5f77a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b4a60 con 0x7f0c5815d6c0 2023-10-16T10:03:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5f77a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b4d40 con 0x7f0c5815daa0 2023-10-16T10:03:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5f77a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b5020 con 0x7f0c580cf360 2023-10-16T10:03:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5cd15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c5815d6c0 0x7f0c580b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5dd17700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815daa0 0x7f0c581f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5cd15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c5815d6c0 0x7f0c580b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57536/0 (socket says 172.21.15.158:57536) 2023-10-16T10:03:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5d516700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c5815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.215+0000 7f0c5cd15700 1 -- 172.21.15.158:0/4154161280 learned_addr learned my addr 172.21.15.158:0/4154161280 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5dd17700 1 -- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 msgr2=0x7f0c5815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5dd17700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c5815e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5dd17700 1 -- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c5815d6c0 msgr2=0x7f0c580b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5dd17700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c5815d6c0 0x7f0c580b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5dd17700 1 -- 172.21.15.158:0/4154161280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c50009a70 con 0x7f0c5815daa0 2023-10-16T10:03:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5d516700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c5815e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5dd17700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815daa0 0x7f0c581f59e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000efc0 tx=0x7f0c5000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c4dffb700 1 -- 172.21.15.158:0/4154161280 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000ca90 con 0x7f0c5815daa0 2023-10-16T10:03:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c4dffb700 1 -- 172.21.15.158:0/4154161280 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5000f040 con 0x7f0c5815daa0 2023-10-16T10:03:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c540094a0 con 0x7f0c5815daa0 2023-10-16T10:03:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c4dffb700 1 -- 172.21.15.158:0/4154161280 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000ce30 con 0x7f0c5815daa0 2023-10-16T10:03:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.216+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c580b55e0 con 0x7f0c5815daa0 2023-10-16T10:03:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.217+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c580d2350 con 0x7f0c5815daa0 2023-10-16T10:03:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.217+0000 7f0c4dffb700 1 -- 172.21.15.158:0/4154161280 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f0c50014480 con 0x7f0c5815daa0 2023-10-16T10:03:07.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.218+0000 7f0c4dffb700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 0x7f0c3c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.218+0000 7f0c5d516700 1 -- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 msgr2=0x7f0c3c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:07.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.218+0000 7f0c4dffb700 1 -- 172.21.15.158:0/4154161280 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f0c5008a4d0 con 0x7f0c5815daa0 2023-10-16T10:03:07.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.218+0000 7f0c5d516700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 0x7f0c3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:07.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.226+0000 7f0c4dffb700 1 -- 172.21.15.158:0/4154161280 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0c580d2350 con 0x7f0c5815daa0 2023-10-16T10:03:07.413 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:07.412+0000 7f448077a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:03:07.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.419+0000 7f0c5d516700 1 -- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 msgr2=0x7f0c3c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:07.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.419+0000 7f0c5d516700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 0x7f0c3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:07.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.425+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c580d2350 con 0x7f0c5815daa0 2023-10-16T10:03:07.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.426+0000 7f0c4dffb700 1 -- 172.21.15.158:0/4154161280 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c580d2350 con 0x7f0c5815daa0 2023-10-16T10:03:07.427 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:03:07.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 msgr2=0x7f0c3c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:07.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 0x7f0c3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815daa0 msgr2=0x7f0c581f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815daa0 0x7f0c581f59e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000efc0 tx=0x7f0c5000c3a0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 shutdown_connections 2023-10-16T10:03:07.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c5815daa0 0x7f0c581f59e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c580cf360 0x7f0c5815e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f0c3c06b140 0x7f0c3c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 --2- 172.21.15.158:0/4154161280 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c5815d6c0 0x7f0c580b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 >> 172.21.15.158:0/4154161280 conn(0x7f0c5809c940 msgr2=0x7f0c580b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:07.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 shutdown_connections 2023-10-16T10:03:07.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.428+0000 7f0c5f77a700 1 -- 172.21.15.158:0/4154161280 wait complete. 2023-10-16T10:03:07.433 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:07.432+0000 7ff67f63e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:03:07.446 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:03:07.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.570+0000 7f4bd6571700 1 Processor -- start 2023-10-16T10:03:07.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.570+0000 7f4bd6571700 1 -- start start 2023-10-16T10:03:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bd6571700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4bd00d29b0 0x7f4bd00d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bd6571700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4bd00d0a70 0x7f4bd00b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bd6571700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd00b7c90 con 0x7f4bd0160d30 2023-10-16T10:03:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bd6571700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd00b8910 con 0x7f4bd00d29b0 2023-10-16T10:03:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bd6571700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd00d0f00 con 0x7f4bd00d0a70 2023-10-16T10:03:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bcf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4bd00d29b0 0x7f4bd00d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bcf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4bd00d29b0 0x7f4bd00d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57546/0 (socket says 172.21.15.158:57546) 2023-10-16T10:03:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bcf7fe700 1 -- 172.21.15.158:0/1866118610 learned_addr learned my addr 172.21.15.158:0/1866118610 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bd4b0e700 1 --2- 172.21.15.158:0/1866118610 >> v2:172.21.15.172:3300/0 conn(0x7f4bd00d0a70 0x7f4bd00b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bcf7fe700 1 -- 172.21.15.158:0/1866118610 >> v2:172.21.15.172:3300/0 conn(0x7f4bd00d0a70 msgr2=0x7f4bd00b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bcf7fe700 1 --2- 172.21.15.158:0/1866118610 >> v2:172.21.15.172:3300/0 conn(0x7f4bd00d0a70 0x7f4bd00b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bcf7fe700 1 -- 172.21.15.158:0/1866118610 >> v1:172.21.15.172:6789/0 conn(0x7f4bd0160d30 legacy=0x7f4bd00b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.571+0000 7f4bcf7fe700 1 -- 172.21.15.158:0/1866118610 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd00b7f70 con 0x7f4bd00d29b0 2023-10-16T10:03:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bcf7fe700 1 --2- 172.21.15.158:0/1866118610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd00d29b0 0x7f4bd00d0530 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40098f0 tx=0x7f4bc4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530e7422e79cd6e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bceffd700 1 -- 172.21.15.158:0/1866118610 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400b860 con 0x7f4bd00d29b0 2023-10-16T10:03:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bceffd700 1 -- 172.21.15.158:0/1866118610 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc400b9c0 con 0x7f4bd00d29b0 2023-10-16T10:03:07.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bceffd700 1 -- 172.21.15.158:0/1866118610 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400bd50 con 0x7f4bd00d29b0 2023-10-16T10:03:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 -- 172.21.15.158:0/1866118610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd00d29b0 msgr2=0x7f4bd00d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 --2- 172.21.15.158:0/1866118610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd00d29b0 0x7f4bd00d0530 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40098f0 tx=0x7f4bc4009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 -- 172.21.15.158:0/1866118610 shutdown_connections 2023-10-16T10:03:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 --2- 172.21.15.158:0/1866118610 >> v2:172.21.15.172:3300/0 conn(0x7f4bd00d0a70 0x7f4bd00b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 --2- 172.21.15.158:0/1866118610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd00d29b0 0x7f4bd00d0530 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 -- 172.21.15.158:0/1866118610 >> 172.21.15.158:0/1866118610 conn(0x7f4bd00669f0 msgr2=0x7f4bd0066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 -- 172.21.15.158:0/1866118610 shutdown_connections 2023-10-16T10:03:07.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.572+0000 7f4bd6571700 1 -- 172.21.15.158:0/1866118610 wait complete. 2023-10-16T10:03:07.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 Processor -- start 2023-10-16T10:03:07.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 -- start start 2023-10-16T10:03:07.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bd00d0a70 0x7f4bd00c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bd00d29b0 0x7f4bd015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd0160d30 0x7f4bd0149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00c9f50 con 0x7f4bd0160d30 2023-10-16T10:03:07.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd015eaa0 con 0x7f4bd00d0a70 2023-10-16T10:03:07.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.574+0000 7f4bd6571700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00c9c70 con 0x7f4bd00d29b0 2023-10-16T10:03:07.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd0160d30 0x7f4bd0149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd0160d30 0x7f4bd0149890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57554/0 (socket says 172.21.15.158:57554) 2023-10-16T10:03:07.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 -- 172.21.15.158:0/15342339 learned_addr learned my addr 172.21.15.158:0/15342339 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:07.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bcffff700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bd00d0a70 0x7f4bd00c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bcf7fe700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bd00d29b0 0x7f4bd015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 -- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bd00d29b0 msgr2=0x7f4bd015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bd00d29b0 0x7f4bd015bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 -- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bd00d0a70 msgr2=0x7f4bd00c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bd00d0a70 0x7f4bd00c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 -- 172.21.15.158:0/15342339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4009550 con 0x7f4bd0160d30 2023-10-16T10:03:07.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd4b0e700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd0160d30 0x7f4bd0149890 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8007e40 tx=0x7f4bc800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:07.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bccff9700 1 -- 172.21.15.158:0/15342339 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800ed30 con 0x7f4bd0160d30 2023-10-16T10:03:07.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bccff9700 1 -- 172.21.15.158:0/15342339 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc8014420 con 0x7f4bd0160d30 2023-10-16T10:03:07.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd015ed40 con 0x7f4bd0160d30 2023-10-16T10:03:07.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bccff9700 1 -- 172.21.15.158:0/15342339 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80147d0 con 0x7f4bd0160d30 2023-10-16T10:03:07.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.575+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd01f6e90 con 0x7f4bd0160d30 2023-10-16T10:03:07.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.576+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd00681b0 con 0x7f4bd0160d30 2023-10-16T10:03:07.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.577+0000 7f4bccff9700 1 -- 172.21.15.158:0/15342339 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f4bc8014930 con 0x7f4bd0160d30 2023-10-16T10:03:07.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.577+0000 7f4bccff9700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 0x7f4bbc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.577+0000 7f4bcffff700 1 -- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 msgr2=0x7f4bbc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:07.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.577+0000 7f4bccff9700 1 -- 172.21.15.158:0/15342339 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f4bc808a3a0 con 0x7f4bd0160d30 2023-10-16T10:03:07.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.577+0000 7f4bcffff700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 0x7f4bbc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:07.588 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:07.580+0000 7ff67f63e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:03:07.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.582+0000 7f4bccff9700 1 -- 172.21.15.158:0/15342339 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bc8054e10 con 0x7f4bd0160d30 2023-10-16T10:03:07.656 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:07.653+0000 7f448077a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:03:07.696 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:07.694+0000 7ff67f63e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:03:07.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.777+0000 7f4bcffff700 1 -- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 msgr2=0x7f4bbc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:07.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.777+0000 7f4bcffff700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 0x7f4bbc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:07.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.781+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4bd00b5710 con 0x7f4bd0160d30 2023-10-16T10:03:07.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.781+0000 7f4bccff9700 1 -- 172.21.15.158:0/15342339 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4bc8058430 con 0x7f4bd0160d30 2023-10-16T10:03:07.784 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:03:07.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 msgr2=0x7f4bbc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:07.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 0x7f4bbc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd0160d30 msgr2=0x7f4bd0149890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd0160d30 0x7f4bd0149890 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8007e40 tx=0x7f4bc800e3c0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 shutdown_connections 2023-10-16T10:03:07.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f4bbc06b140 0x7f4bbc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bd0160d30 0x7f4bd0149890 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bd00d0a70 0x7f4bd00c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 --2- 172.21.15.158:0/15342339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bd00d29b0 0x7f4bd015bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 >> 172.21.15.158:0/15342339 conn(0x7f4bd00669f0 msgr2=0x7f4bd0066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:07.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 shutdown_connections 2023-10-16T10:03:07.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.784+0000 7f4bd6571700 1 -- 172.21.15.158:0/15342339 wait complete. 2023-10-16T10:03:07.799 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:07.817 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:07.815+0000 7f448077a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:03:07.849 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:07.848+0000 7ff67f63e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:03:07.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.922+0000 7f9319ea9700 1 Processor -- start 2023-10-16T10:03:07.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.922+0000 7f9319ea9700 1 -- start start 2023-10-16T10:03:07.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.923+0000 7f9319ea9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9314072360 0x7f9314072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.923+0000 7f9319ea9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931406f850 con 0x7f931406f470 2023-10-16T10:03:07.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.923+0000 7f9319ea9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93140b4260 con 0x7f931406a8d0 2023-10-16T10:03:07.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.923+0000 7f9319ea9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9314072d50 con 0x7f9314072360 2023-10-16T10:03:07.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.923+0000 7f93137fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f931406f470 0x7f931406e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57736/0 (socket says 172.21.15.158:57736) 2023-10-16T10:03:07.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.923+0000 7f93137fe700 1 -- 172.21.15.158:0/797448056 learned_addr learned my addr 172.21.15.158:0/797448056 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f9312ffd700 1 --2- 172.21.15.158:0/797448056 >> v2:172.21.15.172:3300/0 conn(0x7f9314072360 0x7f9314072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995693517 0 0) 0x7f931406f850 con 0x7f931406f470 2023-10-16T10:03:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93000035b0 con 0x7f931406f470 2023-10-16T10:03:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328367564 0 0) 0x7f93140b4260 con 0x7f931406a8d0 2023-10-16T10:03:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f931406f850 con 0x7f931406a8d0 2023-10-16T10:03:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f9312ffd700 1 -- 172.21.15.158:0/797448056 >> v1:172.21.15.172:6789/0 conn(0x7f931406a8d0 legacy=0x7f931406ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f9312ffd700 1 -- 172.21.15.158:0/797448056 >> v1:172.21.15.158:6789/0 conn(0x7f931406f470 legacy=0x7f931406e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 478682692 0 0) 0x7f93000035b0 con 0x7f931406f470 2023-10-16T10:03:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f9312ffd700 1 -- 172.21.15.158:0/797448056 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93140b4540 con 0x7f9314072360 2023-10-16T10:03:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f9312ffd700 1 --2- 172.21.15.158:0/797448056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9314072360 0x7f9314072810 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009840 tx=0x7f92fc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f39b48e5e1349080 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304001b70 con 0x7f9314072360 2023-10-16T10:03:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc004030 con 0x7f9314072360 2023-10-16T10:03:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.924+0000 7f93127fc700 1 -- 172.21.15.158:0/797448056 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0043c0 con 0x7f9314072360 2023-10-16T10:03:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.925+0000 7f9319ea9700 1 -- 172.21.15.158:0/797448056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9314072360 msgr2=0x7f9314072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.925+0000 7f9319ea9700 1 --2- 172.21.15.158:0/797448056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9314072360 0x7f9314072810 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009840 tx=0x7f92fc009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.925+0000 7f9319ea9700 1 -- 172.21.15.158:0/797448056 shutdown_connections 2023-10-16T10:03:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.925+0000 7f9319ea9700 1 --2- 172.21.15.158:0/797448056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9314072360 0x7f9314072810 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.925+0000 7f9319ea9700 1 -- 172.21.15.158:0/797448056 >> 172.21.15.158:0/797448056 conn(0x7f93140ccda0 msgr2=0x7f93140cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.925+0000 7f9319ea9700 1 -- 172.21.15.158:0/797448056 shutdown_connections 2023-10-16T10:03:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.925+0000 7f9319ea9700 1 -- 172.21.15.158:0/797448056 wait complete. 2023-10-16T10:03:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.926+0000 7f9319ea9700 1 Processor -- start 2023-10-16T10:03:07.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.926+0000 7f9319ea9700 1 -- start start 2023-10-16T10:03:07.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9319ea9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931406a8d0 0x7f9314148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9319ea9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931406f470 0x7f931415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9319ea9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9314072360 0x7f93140b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9319ea9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931415ccc0 con 0x7f9314072360 2023-10-16T10:03:07.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9319ea9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931415c390 con 0x7f931406a8d0 2023-10-16T10:03:07.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9319ea9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931415c670 con 0x7f931406f470 2023-10-16T10:03:07.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9313fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9314072360 0x7f93140b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9313fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9314072360 0x7f93140b1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57570/0 (socket says 172.21.15.158:57570) 2023-10-16T10:03:07.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9313fff700 1 -- 172.21.15.158:0/3145663495 learned_addr learned my addr 172.21.15.158:0/3145663495 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:07.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f93137fe700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931406a8d0 0x7f9314148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.927+0000 7f9312ffd700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931406f470 0x7f931415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:07.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9313fff700 1 -- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931406f470 msgr2=0x7f931415a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9313fff700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931406f470 0x7f931415a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9313fff700 1 -- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931406a8d0 msgr2=0x7f9314148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:07.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9313fff700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931406a8d0 0x7f9314148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:07.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9313fff700 1 -- 172.21.15.158:0/3145663495 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc0094a0 con 0x7f9314072360 2023-10-16T10:03:07.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9313fff700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9314072360 0x7f93140b1910 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f930c00c490 tx=0x7f930c00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:07.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f930bfff700 1 -- 172.21.15.158:0/3145663495 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c00cea0 con 0x7f9314072360 2023-10-16T10:03:07.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f930bfff700 1 -- 172.21.15.158:0/3145663495 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c0052d0 con 0x7f9314072360 2023-10-16T10:03:07.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931415c950 con 0x7f9314072360 2023-10-16T10:03:07.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f930bfff700 1 -- 172.21.15.158:0/3145663495 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c003680 con 0x7f9314072360 2023-10-16T10:03:07.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.928+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931415bff0 con 0x7f9314072360 2023-10-16T10:03:07.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.929+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931415ce30 con 0x7f9314072360 2023-10-16T10:03:07.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.929+0000 7f930bfff700 1 -- 172.21.15.158:0/3145663495 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f930c0037e0 con 0x7f9314072360 2023-10-16T10:03:07.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.930+0000 7f930bfff700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 0x7f92f406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:07.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.930+0000 7f93137fe700 1 -- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 msgr2=0x7f92f406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:07.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.930+0000 7f930bfff700 1 -- 172.21.15.158:0/3145663495 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f930c089180 con 0x7f9314072360 2023-10-16T10:03:07.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.930+0000 7f93137fe700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 0x7f92f406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:07.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:07.934+0000 7f930bfff700 1 -- 172.21.15.158:0/3145663495 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f930c053ad0 con 0x7f9314072360 2023-10-16T10:03:07.944 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:07.940+0000 7f448077a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:03:08.103 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:08.101+0000 7f448077a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:03:08.111 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:08.110+0000 7ff67f63e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:08.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.131+0000 7f93137fe700 1 -- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 msgr2=0x7f92f406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:08.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.131+0000 7f93137fe700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 0x7f92f406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.176+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93140681b0 con 0x7f9314072360 2023-10-16T10:03:08.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.176+0000 7f930bfff700 1 -- 172.21.15.158:0/3145663495 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f930c0570f0 con 0x7f9314072360 2023-10-16T10:03:08.179 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:08.179 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/94308149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":94308149},{"type":"v1","addr":"172.21.15.172:6834","nonce":94308149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/4112063716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4112063716},{"type":"v1","addr":"172.21.15.172:6835","nonce":4112063716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8274,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/658160836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":658160836},{"type":"v1","addr":"172.21.15.158:6836","nonce":658160836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:02:09.580772+0000","modified":"2023-10-16T10:02:12.619480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8277":{"gid":8277,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/3693875115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":3693875115},{"type":"v1","addr":"172.21.15.158:6837","nonce":3693875115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T10:03:08.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 msgr2=0x7f92f406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 0x7f92f406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9314072360 msgr2=0x7f93140b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9314072360 0x7f93140b1910 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f930c00c490 tx=0x7f930c00c7a0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 shutdown_connections 2023-10-16T10:03:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f92f406b140 0x7f92f406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9314072360 0x7f93140b1910 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f931406a8d0 0x7f9314148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 --2- 172.21.15.158:0/3145663495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f931406f470 0x7f931415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 >> 172.21.15.158:0/3145663495 conn(0x7f93140ccda0 msgr2=0x7f93140b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 shutdown_connections 2023-10-16T10:03:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.179+0000 7f9319ea9700 1 -- 172.21.15.158:0/3145663495 wait complete. 2023-10-16T10:03:08.184 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 60 2023-10-16T10:03:08.195 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:08.270 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:08.269+0000 7ff67f63e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:03:08.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.359+0000 7fe8402dd700 1 Processor -- start 2023-10-16T10:03:08.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.359+0000 7fe8402dd700 1 -- start start 2023-10-16T10:03:08.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe8402dd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe83815a6c0 0x7fe8380723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe8402dd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe83806e090 0x7fe8380ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe8402dd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe838072910 con 0x7fe8380af590 2023-10-16T10:03:08.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe8402dd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8380ca7e0 con 0x7fe83815a6c0 2023-10-16T10:03:08.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe8402dd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8380b1e40 con 0x7fe83806e090 2023-10-16T10:03:08.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83e079700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe83815a6c0 0x7fe8380723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83d878700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe8380af590 0x7fe83806d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57754/0 (socket says 172.21.15.158:57754) 2023-10-16T10:03:08.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83d878700 1 -- 172.21.15.158:0/2816505392 learned_addr learned my addr 172.21.15.158:0/2816505392 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:08.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83e87a700 1 --2- 172.21.15.158:0/2816505392 >> v2:172.21.15.172:3300/0 conn(0x7fe83806e090 0x7fe8380ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83e079700 1 -- 172.21.15.158:0/2816505392 >> v2:172.21.15.172:3300/0 conn(0x7fe83806e090 msgr2=0x7fe8380ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:08.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83e079700 1 --2- 172.21.15.158:0/2816505392 >> v2:172.21.15.172:3300/0 conn(0x7fe83806e090 0x7fe8380ad430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83e079700 1 -- 172.21.15.158:0/2816505392 >> v1:172.21.15.158:6789/0 conn(0x7fe8380af590 legacy=0x7fe83806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83d077700 1 -- 172.21.15.158:0/2816505392 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224594268 0 0) 0x7fe838072910 con 0x7fe8380af590 2023-10-16T10:03:08.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.360+0000 7fe83e079700 1 -- 172.21.15.158:0/2816505392 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8380b2120 con 0x7fe83815a6c0 2023-10-16T10:03:08.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe83e079700 1 --2- 172.21.15.158:0/2816505392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8380723d0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fe8380b29f0 tx=0x7fe834009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18220324619974f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:08.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe83d077700 1 -- 172.21.15.158:0/2816505392 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe834005e30 con 0x7fe83815a6c0 2023-10-16T10:03:08.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe83d077700 1 -- 172.21.15.158:0/2816505392 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe834006030 con 0x7fe83815a6c0 2023-10-16T10:03:08.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe83d077700 1 -- 172.21.15.158:0/2816505392 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8340063c0 con 0x7fe83815a6c0 2023-10-16T10:03:08.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 -- 172.21.15.158:0/2816505392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 msgr2=0x7fe8380723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 --2- 172.21.15.158:0/2816505392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8380723d0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fe8380b29f0 tx=0x7fe834009b00 comp rx=0 tx=0).stop 2023-10-16T10:03:08.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 -- 172.21.15.158:0/2816505392 shutdown_connections 2023-10-16T10:03:08.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 --2- 172.21.15.158:0/2816505392 >> v2:172.21.15.172:3300/0 conn(0x7fe83806e090 0x7fe8380ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 --2- 172.21.15.158:0/2816505392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8380723d0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 -- 172.21.15.158:0/2816505392 >> 172.21.15.158:0/2816505392 conn(0x7fe8380ccda0 msgr2=0x7fe8380cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:08.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 -- 172.21.15.158:0/2816505392 shutdown_connections 2023-10-16T10:03:08.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.361+0000 7fe8402dd700 1 -- 172.21.15.158:0/2816505392 wait complete. 2023-10-16T10:03:08.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 Processor -- start 2023-10-16T10:03:08.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 -- start start 2023-10-16T10:03:08.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83806e090 0x7fe8381f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8380af590 0x7fe8380c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8381587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381f4110 con 0x7fe83815a6c0 2023-10-16T10:03:08.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381f3110 con 0x7fe8380af590 2023-10-16T10:03:08.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.363+0000 7fe8402dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8380c7c40 con 0x7fe83806e090 2023-10-16T10:03:08.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8381587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8381587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57586/0 (socket says 172.21.15.158:57586) 2023-10-16T10:03:08.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 -- 172.21.15.158:0/1926230785 learned_addr learned my addr 172.21.15.158:0/1926230785 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:08.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e079700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83806e090 0x7fe8381f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83d878700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8380af590 0x7fe8380c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 -- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83806e090 msgr2=0x7fe8381f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83806e090 0x7fe8381f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 -- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8380af590 msgr2=0x7fe8380c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8380af590 0x7fe8380c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 -- 172.21.15.158:0/1926230785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834009430 con 0x7fe83815a6c0 2023-10-16T10:03:08.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe83e87a700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8381587d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fe830007e30 tx=0x7fe83000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:08.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe82effd700 1 -- 172.21.15.158:0/1926230785 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83000ed30 con 0x7fe83815a6c0 2023-10-16T10:03:08.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe82effd700 1 -- 172.21.15.158:0/1926230785 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83000ee90 con 0x7fe83815a6c0 2023-10-16T10:03:08.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8380c7f20 con 0x7fe83815a6c0 2023-10-16T10:03:08.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe82effd700 1 -- 172.21.15.158:0/1926230785 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830014660 con 0x7fe83815a6c0 2023-10-16T10:03:08.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.364+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8381f36b0 con 0x7fe83815a6c0 2023-10-16T10:03:08.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.365+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8380681b0 con 0x7fe83815a6c0 2023-10-16T10:03:08.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.366+0000 7fe82effd700 1 -- 172.21.15.158:0/1926230785 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe8300147c0 con 0x7fe83815a6c0 2023-10-16T10:03:08.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.367+0000 7fe82effd700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 0x7fe81c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.367+0000 7fe83e079700 1 -- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 msgr2=0x7fe81c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:08.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.367+0000 7fe82effd700 1 -- 172.21.15.158:0/1926230785 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fe83008ae80 con 0x7fe83815a6c0 2023-10-16T10:03:08.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.367+0000 7fe83e079700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 0x7fe81c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:08.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.374+0000 7fe82effd700 1 -- 172.21.15.158:0/1926230785 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8300558d0 con 0x7fe83815a6c0 2023-10-16T10:03:08.383 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:08.381+0000 7f448077a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:08.551 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:08.549+0000 7f448077a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:03:08.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.568+0000 7fe83e079700 1 -- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 msgr2=0x7fe81c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:08.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.568+0000 7fe83e079700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 0x7fe81c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:08.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.575+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8380b26b0 con 0x7fe83815a6c0 2023-10-16T10:03:08.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.576+0000 7fe82effd700 1 -- 172.21.15.158:0/1926230785 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+17882 (secure 0 0 0) 0x7fe830058ef0 con 0x7fe83815a6c0 2023-10-16T10:03:08.579 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:08.579 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":127,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:02:23.842920+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:02:08.021999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T10:02:08.539662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1769168841":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3578502848":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2674552982":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2360787591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6833/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6832/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3568191591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/1741897775":"2023-10-17T10:02:23.842872+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:08.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 msgr2=0x7fe81c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:08.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 0x7fe81c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 msgr2=0x7fe8381587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8381587d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fe830007e30 tx=0x7fe83000e3c0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 shutdown_connections 2023-10-16T10:03:08.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fe81c06b140 0x7fe81c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815a6c0 0x7fe8381587d0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83806e090 0x7fe8381f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.578+0000 7fe8402dd700 1 --2- 172.21.15.158:0/1926230785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8380af590 0x7fe8380c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.579+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 >> 172.21.15.158:0/1926230785 conn(0x7fe8380ccda0 msgr2=0x7fe8380b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:08.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.579+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 shutdown_connections 2023-10-16T10:03:08.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.579+0000 7fe8402dd700 1 -- 172.21.15.158:0/1926230785 wait complete. 2023-10-16T10:03:08.593 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T10:03:08.594 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:08.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.715+0000 7f9b3db78700 1 Processor -- start 2023-10-16T10:03:08.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.715+0000 7f9b3db78700 1 -- start start 2023-10-16T10:03:08.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b3db78700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b3db78700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3815b930 con 0x7f9b380cc1b0 2023-10-16T10:03:08.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b3db78700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3815f5f0 con 0x7f9b380ce530 2023-10-16T10:03:08.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b3db78700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380cf020 con 0x7f9b3815f0f0 2023-10-16T10:03:08.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b36ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9b380cc1b0 0x7f9b380cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57770/0 (socket says 172.21.15.158:57770) 2023-10-16T10:03:08.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b36ffd700 1 -- 172.21.15.158:0/1039812404 learned_addr learned my addr 172.21.15.158:0/1039812404 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:08.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b37fff700 1 --2- 172.21.15.158:0/1039812404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b367fc700 1 -- 172.21.15.158:0/1039812404 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841042981 0 0) 0x7f9b3815b930 con 0x7f9b380cc1b0 2023-10-16T10:03:08.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b367fc700 1 -- 172.21.15.158:0/1039812404 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b240035b0 con 0x7f9b380cc1b0 2023-10-16T10:03:08.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b367fc700 1 -- 172.21.15.158:0/1039812404 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814756836 0 0) 0x7f9b3815f5f0 con 0x7f9b380ce530 2023-10-16T10:03:08.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b367fc700 1 -- 172.21.15.158:0/1039812404 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3815b930 con 0x7f9b380ce530 2023-10-16T10:03:08.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b37fff700 1 -- 172.21.15.158:0/1039812404 >> v1:172.21.15.172:6789/0 conn(0x7f9b380ce530 legacy=0x7f9b380cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b37fff700 1 -- 172.21.15.158:0/1039812404 >> v1:172.21.15.158:6789/0 conn(0x7f9b380cc1b0 legacy=0x7f9b380cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b37fff700 1 -- 172.21.15.158:0/1039812404 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3815f8d0 con 0x7f9b3815f0f0 2023-10-16T10:03:08.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b37fff700 1 --2- 172.21.15.158:0/1039812404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380b3b20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009870 tx=0x7f9b30009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f2396259645e32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:08.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.717+0000 7f9b367fc700 1 -- 172.21.15.158:0/1039812404 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30011830 con 0x7f9b3815f0f0 2023-10-16T10:03:08.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b367fc700 1 -- 172.21.15.158:0/1039812404 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b30011990 con 0x7f9b3815f0f0 2023-10-16T10:03:08.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b367fc700 1 -- 172.21.15.158:0/1039812404 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30011d20 con 0x7f9b3815f0f0 2023-10-16T10:03:08.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b3db78700 1 -- 172.21.15.158:0/1039812404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 msgr2=0x7f9b380b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b3db78700 1 --2- 172.21.15.158:0/1039812404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380b3b20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009870 tx=0x7f9b30009b80 comp rx=0 tx=0).stop 2023-10-16T10:03:08.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b3db78700 1 -- 172.21.15.158:0/1039812404 shutdown_connections 2023-10-16T10:03:08.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b3db78700 1 --2- 172.21.15.158:0/1039812404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380b3b20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b3db78700 1 -- 172.21.15.158:0/1039812404 >> 172.21.15.158:0/1039812404 conn(0x7f9b3806a8d0 msgr2=0x7f9b3806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:08.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b3db78700 1 -- 172.21.15.158:0/1039812404 shutdown_connections 2023-10-16T10:03:08.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.718+0000 7f9b3db78700 1 -- 172.21.15.158:0/1039812404 wait complete. 2023-10-16T10:03:08.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.720+0000 7f9b3db78700 1 Processor -- start 2023-10-16T10:03:08.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.720+0000 7f9b3db78700 1 -- start start 2023-10-16T10:03:08.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b3db78700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b380cc1b0 0x7f9b380afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b3db78700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b380ce530 0x7f9b3815a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b3db78700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b3db78700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3815af20 con 0x7f9b380ce530 2023-10-16T10:03:08.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b3db78700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380ca950 con 0x7f9b380cc1b0 2023-10-16T10:03:08.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b3db78700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380b0340 con 0x7f9b3815f0f0 2023-10-16T10:03:08.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b380ce530 0x7f9b3815a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b380ce530 0x7f9b3815a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57600/0 (socket says 172.21.15.158:57600) 2023-10-16T10:03:08.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 -- 172.21.15.158:0/2245294672 learned_addr learned my addr 172.21.15.158:0/2245294672 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:08.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b377fe700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b380cc1b0 0x7f9b380afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b37fff700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:08.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 -- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 msgr2=0x7f9b380cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 -- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b380cc1b0 msgr2=0x7f9b380afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b380cc1b0 0x7f9b380afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 -- 172.21.15.158:0/2245294672 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300094a0 con 0x7f9b380ce530 2023-10-16T10:03:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b36ffd700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b380ce530 0x7f9b3815a9e0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f9b20001eb0 tx=0x7f9b2000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.721+0000 7f9b2ffff700 1 -- 172.21.15.158:0/2245294672 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20005d70 con 0x7f9b380ce530 2023-10-16T10:03:08.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.722+0000 7f9b2ffff700 1 -- 172.21.15.158:0/2245294672 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b200070d0 con 0x7f9b380ce530 2023-10-16T10:03:08.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.722+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b380ca020 con 0x7f9b380ce530 2023-10-16T10:03:08.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.722+0000 7f9b2ffff700 1 -- 172.21.15.158:0/2245294672 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20007480 con 0x7f9b380ce530 2023-10-16T10:03:08.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.722+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b380ca5e0 con 0x7f9b380ce530 2023-10-16T10:03:08.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.723+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b380caac0 con 0x7f9b380ce530 2023-10-16T10:03:08.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.723+0000 7f9b2ffff700 1 -- 172.21.15.158:0/2245294672 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f9b200075e0 con 0x7f9b380ce530 2023-10-16T10:03:08.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.723+0000 7f9b2ffff700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 0x7f9b1806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:08.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.723+0000 7f9b377fe700 1 -- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 msgr2=0x7f9b1806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:08.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.724+0000 7f9b2ffff700 1 -- 172.21.15.158:0/2245294672 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f9b20089230 con 0x7f9b380ce530 2023-10-16T10:03:08.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.724+0000 7f9b377fe700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 0x7f9b1806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:08.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.728+0000 7f9b2ffff700 1 -- 172.21.15.158:0/2245294672 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b20053b40 con 0x7f9b380ce530 2023-10-16T10:03:08.802 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:08.800+0000 7ff67f63e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:03:08.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.925+0000 7f9b377fe700 1 -- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 msgr2=0x7f9b1806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:08.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.925+0000 7f9b377fe700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 0x7f9b1806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:08.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.984+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b380ca7a0 con 0x7f9b380ce530 2023-10-16T10:03:08.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.985+0000 7f9b2ffff700 1 -- 172.21.15.158:0/2245294672 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f9b20057160 con 0x7f9b380ce530 2023-10-16T10:03:08.987 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:08.987 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/94308149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":94308149},{"type":"v1","addr":"172.21.15.172:6834","nonce":94308149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/4112063716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4112063716},{"type":"v1","addr":"172.21.15.172:6835","nonce":4112063716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8274,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/658160836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":658160836},{"type":"v1","addr":"172.21.15.158:6836","nonce":658160836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:02:09.580772+0000","modified":"2023-10-16T10:02:12.619480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8277":{"gid":8277,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/3693875115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":3693875115},{"type":"v1","addr":"172.21.15.158:6837","nonce":3693875115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T10:03:08.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 msgr2=0x7f9b1806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:08.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 0x7f9b1806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b380ce530 msgr2=0x7f9b3815a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:08.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b380ce530 0x7f9b3815a9e0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f9b20001eb0 tx=0x7f9b2000c7b0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 shutdown_connections 2023-10-16T10:03:08.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b3815f0f0 0x7f9b380cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b380cc1b0 0x7f9b380afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7f9b1806b140 0x7f9b1806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 --2- 172.21.15.158:0/2245294672 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b380ce530 0x7f9b3815a9e0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:08.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 >> 172.21.15.158:0/2245294672 conn(0x7f9b3806a8d0 msgr2=0x7f9b380b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:08.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 shutdown_connections 2023-10-16T10:03:08.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:08.987+0000 7f9b3db78700 1 -- 172.21.15.158:0/2245294672 wait complete. 2023-10-16T10:03:08.993 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 60 2023-10-16T10:03:09.002 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:09.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.123+0000 7fbf9830f700 1 Processor -- start 2023-10-16T10:03:09.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.123+0000 7fbf9830f700 1 -- start start 2023-10-16T10:03:09.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf9830f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf900b3480 0x7fbf900cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf9830f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf9006a8d0 0x7fbf9006ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf9830f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900b3910 con 0x7fbf9015ed60 2023-10-16T10:03:09.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf9830f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9006b2a0 con 0x7fbf9006a8d0 2023-10-16T10:03:09.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf9830f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900b3f30 con 0x7fbf900b3480 2023-10-16T10:03:09.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf958aa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf900b3480 0x7fbf900cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf960ab700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbf9015ed60 0x7fbf900b43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:33456/0 (socket says 172.21.15.158:33456) 2023-10-16T10:03:09.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf968ac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf9006a8d0 0x7fbf9006ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.124+0000 7fbf960ab700 1 -- 172.21.15.158:0/3142590062 learned_addr learned my addr 172.21.15.158:0/3142590062 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:09.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf950a9700 1 -- 172.21.15.158:0/3142590062 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199148363 0 0) 0x7fbf8c001160 con 0x7fbf9015ed60 2023-10-16T10:03:09.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf950a9700 1 -- 172.21.15.158:0/3142590062 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf7c0035b0 con 0x7fbf9015ed60 2023-10-16T10:03:09.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf968ac700 1 -- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf900b3480 msgr2=0x7fbf900cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:09.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf968ac700 1 --2- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf900b3480 0x7fbf900cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf968ac700 1 -- 172.21.15.158:0/3142590062 >> v1:172.21.15.172:6789/0 conn(0x7fbf9015ed60 legacy=0x7fbf900b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf968ac700 1 -- 172.21.15.158:0/3142590062 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900b4ad0 con 0x7fbf9006a8d0 2023-10-16T10:03:09.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf958aa700 1 --2- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf900b3480 0x7fbf900cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:09.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf968ac700 1 --2- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf9006ad60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009d90 tx=0x7fbf8800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b41515dbd84aab81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:09.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf950a9700 1 -- 172.21.15.158:0/3142590062 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8800e070 con 0x7fbf9006a8d0 2023-10-16T10:03:09.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf950a9700 1 -- 172.21.15.158:0/3142590062 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8800bb20 con 0x7fbf9006a8d0 2023-10-16T10:03:09.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.125+0000 7fbf950a9700 1 -- 172.21.15.158:0/3142590062 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8800be90 con 0x7fbf9006a8d0 2023-10-16T10:03:09.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 -- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 msgr2=0x7fbf9006ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 --2- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf9006ad60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009d90 tx=0x7fbf8800b330 comp rx=0 tx=0).stop 2023-10-16T10:03:09.132 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.126+0000 7f448077a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:03:09.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 -- 172.21.15.158:0/3142590062 shutdown_connections 2023-10-16T10:03:09.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 --2- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf9006ad60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 --2- 172.21.15.158:0/3142590062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf900b3480 0x7fbf900cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 -- 172.21.15.158:0/3142590062 >> 172.21.15.158:0/3142590062 conn(0x7fbf900ccda0 msgr2=0x7fbf900cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:09.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 -- 172.21.15.158:0/3142590062 shutdown_connections 2023-10-16T10:03:09.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.126+0000 7fbf9830f700 1 -- 172.21.15.158:0/3142590062 wait complete. 2023-10-16T10:03:09.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.127+0000 7fbf9830f700 1 Processor -- start 2023-10-16T10:03:09.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.127+0000 7fbf9830f700 1 -- start start 2023-10-16T10:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf9830f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf90150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf9830f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf9015ed60 0x7fbf90150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf9830f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf900cf7e0 0x7fbf900cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf9830f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900cf350 con 0x7fbf900cf7e0 2023-10-16T10:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf9830f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90150a60 con 0x7fbf9006a8d0 2023-10-16T10:03:09.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf9830f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900ce740 con 0x7fbf9015ed60 2023-10-16T10:03:09.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf900cf7e0 0x7fbf900cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf960ab700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf90150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf900cf7e0 0x7fbf900cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57618/0 (socket says 172.21.15.158:57618) 2023-10-16T10:03:09.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 -- 172.21.15.158:0/1650053915 learned_addr learned my addr 172.21.15.158:0/1650053915 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:09.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf958aa700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf9015ed60 0x7fbf90150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 -- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf9015ed60 msgr2=0x7fbf90150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf9015ed60 0x7fbf90150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 -- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 msgr2=0x7fbf90150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf90150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 -- 172.21.15.158:0/1650053915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf800094a0 con 0x7fbf900cf7e0 2023-10-16T10:03:09.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf960ab700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf90150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:09.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf968ac700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf900cf7e0 0x7fbf900cdef0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fbf88014040 tx=0x7fbf88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:09.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88004800 con 0x7fbf900cf7e0 2023-10-16T10:03:09.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf88020070 con 0x7fbf900cf7e0 2023-10-16T10:03:09.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf880276c0 con 0x7fbf900cf7e0 2023-10-16T10:03:09.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.128+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf88009a70 con 0x7fbf900cf7e0 2023-10-16T10:03:09.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.129+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf900ced00 con 0x7fbf900cf7e0 2023-10-16T10:03:09.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.129+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf900cf4c0 con 0x7fbf900cf7e0 2023-10-16T10:03:09.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.130+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fbf88012430 con 0x7fbf900cf7e0 2023-10-16T10:03:09.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.130+0000 7fbf86ffd700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fbf7406b170 0x7fbf7406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.130+0000 7fbf960ab700 1 -- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fbf7406b170 msgr2=0x7fbf7406d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1079935681 2023-10-16T10:03:09.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.130+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fbf88022070 con 0x7fbf900cf7e0 2023-10-16T10:03:09.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.130+0000 7fbf960ab700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fbf7406b170 0x7fbf7406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:09.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.135+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf88056260 con 0x7fbf900cf7e0 2023-10-16T10:03:09.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.150+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 71) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbf88056630 con 0x7fbf900cf7e0 2023-10-16T10:03:09.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.150+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fbf7406b170 msgr2=0x7fbf7406d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:09.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.150+0000 7fbf86ffd700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fbf7406b170 0x7fbf7406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.212 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43fb765700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.212 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43fb765700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.213 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43fb765700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.213 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43fb765700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.213 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43fb765700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.213 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43f8f60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.213 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43f8f60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.214 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43f8f60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.214 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43f8f60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.214 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:09.210+0000 7f43f8f60700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:09.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.392+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf900681b0 con 0x7fbf900cf7e0 2023-10-16T10:03:09.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.392+0000 7fbf86ffd700 1 -- 172.21.15.158:0/1650053915 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fbf8805c3a0 con 0x7fbf900cf7e0 2023-10-16T10:03:09.395 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:09.395 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/94308149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":94308149},{"type":"v1","addr":"172.21.15.172:6834","nonce":94308149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/4112063716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4112063716},{"type":"v1","addr":"172.21.15.172:6835","nonce":4112063716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8274,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/658160836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":658160836},{"type":"v1","addr":"172.21.15.158:6836","nonce":658160836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:02:09.580772+0000","modified":"2023-10-16T10:02:12.619480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8277":{"gid":8277,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/3693875115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":3693875115},{"type":"v1","addr":"172.21.15.158:6837","nonce":3693875115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T10:03:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf900cf7e0 msgr2=0x7fbf900cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf900cf7e0 0x7fbf900cdef0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fbf88014040 tx=0x7fbf88004ad0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 shutdown_connections 2023-10-16T10:03:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:6832/1079935681,v1:172.21.15.158:6833/1079935681] conn(0x7fbf7406b170 0x7fbf7406d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf900cf7e0 0x7fbf900cdef0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf9006a8d0 0x7fbf90150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 --2- 172.21.15.158:0/1650053915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf9015ed60 0x7fbf90150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 >> 172.21.15.158:0/1650053915 conn(0x7fbf900ccda0 msgr2=0x7fbf9015c230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 shutdown_connections 2023-10-16T10:03:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.395+0000 7fbf9830f700 1 -- 172.21.15.158:0/1650053915 wait complete. 2023-10-16T10:03:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 60 2023-10-16T10:03:09.409 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:09.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.527+0000 7f57ee5a5700 1 Processor -- start 2023-10-16T10:03:09.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.527+0000 7f57ee5a5700 1 -- start start 2023-10-16T10:03:09.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ee5a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806afe0 0x7f57e815df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ee5a5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f57e8162110 0x7f57e815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ee5a5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e8161c80 con 0x7f57e815db30 2023-10-16T10:03:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ee5a5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e806e090 con 0x7f57e8162110 2023-10-16T10:03:09.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ee5a5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e806ddb0 con 0x7f57e806afe0 2023-10-16T10:03:09.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57e77fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806afe0 0x7f57e815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ecb42700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f57e8162110 0x7f57e815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57e7fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f57e815db30 0x7f57e806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:33542/0 (socket says 172.21.15.158:33542) 2023-10-16T10:03:09.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57e7fff700 1 -- 172.21.15.158:0/1389714620 learned_addr learned my addr 172.21.15.158:0/1389714620 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:09.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ecb42700 1 -- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806afe0 msgr2=0x7f57e815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:09.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ecb42700 1 --2- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806afe0 0x7f57e815df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ecb42700 1 -- 172.21.15.158:0/1389714620 >> v1:172.21.15.172:6789/0 conn(0x7f57e815db30 legacy=0x7f57e806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57ecb42700 1 -- 172.21.15.158:0/1389714620 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8161630 con 0x7f57e8162110 2023-10-16T10:03:09.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57e6ffd700 1 -- 172.21.15.158:0/1389714620 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415196117 0 0) 0x7f57e8161c80 con 0x7f57e815db30 2023-10-16T10:03:09.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.528+0000 7f57e77fe700 1 --2- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806afe0 0x7f57e815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:09.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ecb42700 1 --2- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e8162110 0x7f57e815e450 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009d90 tx=0x7f57e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe614ac5519a51b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:09.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57e6ffd700 1 -- 172.21.15.158:0/1389714620 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e000e070 con 0x7f57e8162110 2023-10-16T10:03:09.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57e6ffd700 1 -- 172.21.15.158:0/1389714620 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e000bb20 con 0x7f57e8162110 2023-10-16T10:03:09.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57e6ffd700 1 -- 172.21.15.158:0/1389714620 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0012430 con 0x7f57e8162110 2023-10-16T10:03:09.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 -- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e8162110 msgr2=0x7f57e815e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 --2- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e8162110 0x7f57e815e450 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009d90 tx=0x7f57e000b330 comp rx=0 tx=0).stop 2023-10-16T10:03:09.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 -- 172.21.15.158:0/1389714620 shutdown_connections 2023-10-16T10:03:09.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 --2- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e8162110 0x7f57e815e450 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 --2- 172.21.15.158:0/1389714620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e806afe0 0x7f57e815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 -- 172.21.15.158:0/1389714620 >> 172.21.15.158:0/1389714620 conn(0x7f57e80ccdb0 msgr2=0x7f57e80cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 -- 172.21.15.158:0/1389714620 shutdown_connections 2023-10-16T10:03:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.529+0000 7f57ee5a5700 1 -- 172.21.15.158:0/1389714620 wait complete. 2023-10-16T10:03:09.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.530+0000 7f57ee5a5700 1 Processor -- start 2023-10-16T10:03:09.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57ee5a5700 1 -- start start 2023-10-16T10:03:09.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57ee5a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806afe0 0x7f57e80c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57ee5a5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e815db30 0x7f57e81f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57ee5a5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e8162110 0x7f57e8146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57ee5a5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e80c5f30 con 0x7f57e806afe0 2023-10-16T10:03:09.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57ee5a5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e80b4260 con 0x7f57e815db30 2023-10-16T10:03:09.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57ee5a5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e80b4500 con 0x7f57e8162110 2023-10-16T10:03:09.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.531+0000 7f57e7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806afe0 0x7f57e80c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806afe0 0x7f57e80c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57700/0 (socket says 172.21.15.158:57700) 2023-10-16T10:03:09.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 -- 172.21.15.158:0/3491322344 learned_addr learned my addr 172.21.15.158:0/3491322344 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:09.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57ecb42700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e8162110 0x7f57e8146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e77fe700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e815db30 0x7f57e81f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 -- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e8162110 msgr2=0x7f57e8146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e8162110 0x7f57e8146f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 -- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e815db30 msgr2=0x7f57e81f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e815db30 0x7f57e81f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 -- 172.21.15.158:0/3491322344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d80094a0 con 0x7f57e806afe0 2023-10-16T10:03:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57ecb42700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e8162110 0x7f57e8146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:09.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e77fe700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e815db30 0x7f57e81f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e7fff700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806afe0 0x7f57e80c59f0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0011e0 tx=0x7f57dc00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e4ff9700 1 -- 172.21.15.158:0/3491322344 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004c80 con 0x7f57e806afe0 2023-10-16T10:03:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e4ff9700 1 -- 172.21.15.158:0/3491322344 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc004e00 con 0x7f57e806afe0 2023-10-16T10:03:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e0009a70 con 0x7f57e806afe0 2023-10-16T10:03:09.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57e4ff9700 1 -- 172.21.15.158:0/3491322344 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc007310 con 0x7f57e806afe0 2023-10-16T10:03:09.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.532+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e80b4ac0 con 0x7f57e806afe0 2023-10-16T10:03:09.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.533+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e80681b0 con 0x7f57e806afe0 2023-10-16T10:03:09.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.534+0000 7f57e4ff9700 1 -- 172.21.15.158:0/3491322344 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 71) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f57dc007470 con 0x7f57e806afe0 2023-10-16T10:03:09.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.534+0000 7f57e4ff9700 1 -- 172.21.15.158:0/3491322344 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7227+0+0 (secure 0 0 0) 0x7f57dc089460 con 0x7f57e806afe0 2023-10-16T10:03:09.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.538+0000 7f57e4ff9700 1 -- 172.21.15.158:0/3491322344 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f57dc054370 con 0x7f57e806afe0 2023-10-16T10:03:09.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.737+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57e806da60 con 0x7f57e806afe0 2023-10-16T10:03:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.739+0000 7f57e4ff9700 1 -- 172.21.15.158:0/3491322344 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+18515 (secure 0 0 0) 0x7f57dc057990 con 0x7f57e806afe0 2023-10-16T10:03:09.741 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:09.741 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":128,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:09.141977+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:02:08.021999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T10:02:08.539662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3944050656":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/59698897":"2023-10-17T10:03:09.141935+0000","172.21.15.158:6833/1079935681":"2023-10-17T10:03:09.141935+0000","172.21.15.158:6832/1079935681":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/1997480059":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/3901996147":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/3589635435":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/303455253":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/4078321936":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/1769168841":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/1113811588":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/3578502848":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2674552982":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2360787591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6833/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6832/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3568191591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/1741897775":"2023-10-17T10:02:23.842872+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:09.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806afe0 msgr2=0x7f57e80c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806afe0 0x7f57e80c59f0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0011e0 tx=0x7f57dc00cae0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 shutdown_connections 2023-10-16T10:03:09.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57e8162110 0x7f57e8146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57e806afe0 0x7f57e80c59f0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 --2- 172.21.15.158:0/3491322344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57e815db30 0x7f57e81f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 >> 172.21.15.158:0/3491322344 conn(0x7f57e80ccdb0 msgr2=0x7f57e80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:09.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 shutdown_connections 2023-10-16T10:03:09.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.741+0000 7f57ee5a5700 1 -- 172.21.15.158:0/3491322344 wait complete. 2023-10-16T10:03:09.755 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T10:03:09.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.875+0000 7fb9fc76d700 1 Processor -- start 2023-10-16T10:03:09.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.875+0000 7fb9fc76d700 1 -- start start 2023-10-16T10:03:09.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9fc76d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb9f40cfd60 0x7fb9f40b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9fc76d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40af600 con 0x7fb9f415d990 2023-10-16T10:03:09.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9fc76d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f406e9b0 con 0x7fb9f406e480 2023-10-16T10:03:09.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9fc76d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f40b2c90 con 0x7fb9f40cfd60 2023-10-16T10:03:09.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9fa509700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb9f415d990 0x7fb9f40b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57882/0 (socket says 172.21.15.158:57882) 2023-10-16T10:03:09.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9fa509700 1 -- 172.21.15.158:0/1375466967 learned_addr learned my addr 172.21.15.158:0/1375466967 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:09.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9fad0a700 1 --2- 172.21.15.158:0/1375466967 >> v2:172.21.15.172:3300/0 conn(0x7fb9f40cfd60 0x7fb9f40b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060354982 0 0) 0x7fb9f40af600 con 0x7fb9f415d990 2023-10-16T10:03:09.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e00035b0 con 0x7fb9f415d990 2023-10-16T10:03:09.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201827460 0 0) 0x7fb9e00035b0 con 0x7fb9f415d990 2023-10-16T10:03:09.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 >> v2:172.21.15.172:3300/0 conn(0x7fb9f40cfd60 msgr2=0x7fb9f40b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:09.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 --2- 172.21.15.158:0/1375466967 >> v2:172.21.15.172:3300/0 conn(0x7fb9f40cfd60 0x7fb9f40b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 >> v1:172.21.15.172:6789/0 conn(0x7fb9f406e480 legacy=0x7fb9f40cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40ce890 con 0x7fb9f415d990 2023-10-16T10:03:09.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.876+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fb9f0001670 con 0x7fb9f415d990 2023-10-16T10:03:09.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 >> v1:172.21.15.158:6789/0 conn(0x7fb9f415d990 legacy=0x7fb9f40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0003a00 0x7fb9e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9e0004850 0x7fb9e0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9e0005210 0x7fb9e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40b2c90 con 0x7fb9e0004850 2023-10-16T10:03:09.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e000c0c0 con 0x7fb9e0005210 2023-10-16T10:03:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e000c280 con 0x7fb9e0003a00 2023-10-16T10:03:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9d08700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9e0004850 0x7fb9e0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9fa509700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0003a00 0x7fb9e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9fad0a700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9e0005210 0x7fb9e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9d08700 1 -- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0003a00 msgr2=0x7fb9e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9d08700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0003a00 0x7fb9e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9d08700 1 -- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9e0005210 msgr2=0x7fb9e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9d08700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9e0005210 0x7fb9e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9d08700 1 -- 172.21.15.158:0/1375466967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e00035b0 con 0x7fb9e0004850 2023-10-16T10:03:09.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9fa509700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0003a00 0x7fb9e0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:09.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9fad0a700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9e0005210 0x7fb9e000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:09.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9fad0a700 1 -- 172.21.15.158:0/1375466967 reap_dead start 2023-10-16T10:03:09.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9d08700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9e0004850 0x7fb9e0004cd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb9f406e8e0 tx=0x7fb9e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9cfaab79a669f0a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:09.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e40052d0 con 0x7fb9e0004850 2023-10-16T10:03:09.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e4005430 con 0x7fb9e0004850 2023-10-16T10:03:09.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.877+0000 7fb9f9507700 1 -- 172.21.15.158:0/1375466967 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e40038c0 con 0x7fb9e0004850 2023-10-16T10:03:09.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.878+0000 7fb9fc76d700 1 -- 172.21.15.158:0/1375466967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9e0004850 msgr2=0x7fb9e0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.878+0000 7fb9fc76d700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9e0004850 0x7fb9e0004cd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb9f406e8e0 tx=0x7fb9e0005880 comp rx=0 tx=0).stop 2023-10-16T10:03:09.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.878+0000 7fb9fc76d700 1 -- 172.21.15.158:0/1375466967 shutdown_connections 2023-10-16T10:03:09.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.878+0000 7fb9fc76d700 1 --2- 172.21.15.158:0/1375466967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9e0004850 0x7fb9e0004cd0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.878+0000 7fb9fc76d700 1 -- 172.21.15.158:0/1375466967 >> 172.21.15.158:0/1375466967 conn(0x7fb9f406b9e0 msgr2=0x7fb9f406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:09.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.878+0000 7fb9fc76d700 1 -- 172.21.15.158:0/1375466967 shutdown_connections 2023-10-16T10:03:09.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.878+0000 7fb9fc76d700 1 -- 172.21.15.158:0/1375466967 wait complete. 2023-10-16T10:03:09.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.879+0000 7fb9fc76d700 1 Processor -- start 2023-10-16T10:03:09.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.879+0000 7fb9fc76d700 1 -- start start 2023-10-16T10:03:09.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fc76d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0004850 0x7fb9f41f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fc76d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9f415a040 0x7fb9f41f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fc76d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9f41526e0 0x7fb9f41e84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:09.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fc76d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f41f4020 con 0x7fb9f41526e0 2023-10-16T10:03:09.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fc76d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f415a4c0 con 0x7fb9f415a040 2023-10-16T10:03:09.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fc76d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f41f7520 con 0x7fb9e0004850 2023-10-16T10:03:09.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9f41526e0 0x7fb9f41e84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9f41526e0 0x7fb9f41e84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57722/0 (socket says 172.21.15.158:57722) 2023-10-16T10:03:09.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 -- 172.21.15.158:0/2069491133 learned_addr learned my addr 172.21.15.158:0/2069491133 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:09.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9f9d08700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9f415a040 0x7fb9f41f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fa509700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0004850 0x7fb9f41f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:09.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 -- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0004850 msgr2=0x7fb9f41f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0004850 0x7fb9f41f6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 -- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9f415a040 msgr2=0x7fb9f41f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:09.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9f415a040 0x7fb9f41f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:09.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 -- 172.21.15.158:0/2069491133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f41f6c80 con 0x7fb9f41526e0 2023-10-16T10:03:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.880+0000 7fb9fad0a700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9f41526e0 0x7fb9f41e84e0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40af310 tx=0x7fb9ec0104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.881+0000 7fb9eaffd700 1 -- 172.21.15.158:0/2069491133 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec01b070 con 0x7fb9f41526e0 2023-10-16T10:03:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.881+0000 7fb9eaffd700 1 -- 172.21.15.158:0/2069491133 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec00f040 con 0x7fb9f41526e0 2023-10-16T10:03:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.881+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f41f6f60 con 0x7fb9f41526e0 2023-10-16T10:03:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.881+0000 7fb9eaffd700 1 -- 172.21.15.158:0/2069491133 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec00e050 con 0x7fb9f41526e0 2023-10-16T10:03:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.881+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f415de80 con 0x7fb9f41526e0 2023-10-16T10:03:09.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.882+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f40cfbd0 con 0x7fb9f41526e0 2023-10-16T10:03:09.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.882+0000 7fb9eaffd700 1 -- 172.21.15.158:0/2069491133 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 71) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb9ec015440 con 0x7fb9f41526e0 2023-10-16T10:03:09.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.883+0000 7fb9eaffd700 1 -- 172.21.15.158:0/2069491133 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7227+0+0 (secure 0 0 0) 0x7fb9ec089820 con 0x7fb9f41526e0 2023-10-16T10:03:09.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:09.887+0000 7fb9eaffd700 1 -- 172.21.15.158:0/2069491133 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9ec054730 con 0x7fb9f41526e0 2023-10-16T10:03:10.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.122+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb9f40681b0 con 0x7fb9f41526e0 2023-10-16T10:03:10.125 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T10:03:10.123+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T10:03:10.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.126+0000 7fb9eaffd700 1 -- 172.21.15.158:0/2069491133 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v61) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb9ec010df0 con 0x7fb9f41526e0 2023-10-16T10:03:10.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9f41526e0 msgr2=0x7fb9f41e84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:10.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9f41526e0 0x7fb9f41e84e0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40af310 tx=0x7fb9ec0104a0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 shutdown_connections 2023-10-16T10:03:10.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9f41526e0 0x7fb9f41e84e0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9e0004850 0x7fb9f41f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 --2- 172.21.15.158:0/2069491133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9f415a040 0x7fb9f41f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 >> 172.21.15.158:0/2069491133 conn(0x7fb9f406b9e0 msgr2=0x7fb9f40ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:10.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 shutdown_connections 2023-10-16T10:03:10.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.129+0000 7fb9fc76d700 1 -- 172.21.15.158:0/2069491133 wait complete. 2023-10-16T10:03:10.133 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T10:03:10.134 INFO:tasks.ceph.mds.c.smithi158.stderr: -9678> 2023-10-16T10:02:16.167+0000 7f71db195700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:03:10.146 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T10:03:10.173 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T10:03:10.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.273+0000 7f5cfc2fc700 1 Processor -- start 2023-10-16T10:03:10.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.273+0000 7f5cfc2fc700 1 -- start start 2023-10-16T10:03:10.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfc2fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5cf415dad0 0x7f5cf40cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:10.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfc2fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:10.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfc2fc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40d0ae0 con 0x7f5cf415deb0 2023-10-16T10:03:10.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfc2fc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf40d12d0 con 0x7f5cf415dad0 2023-10-16T10:03:10.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfc2fc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40cbdf0 con 0x7f5cf40ccc00 2023-10-16T10:03:10.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfa098700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5cf415dad0 0x7f5cf40cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:10.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cf9897700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5cf415deb0 0x7f5cf40ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57934/0 (socket says 172.21.15.158:57934) 2023-10-16T10:03:10.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfa098700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5cf415dad0 0x7f5cf40cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57756/0 (socket says 172.21.15.158:57756) 2023-10-16T10:03:10.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cf9897700 1 -- 172.21.15.158:0/317675851 learned_addr learned my addr 172.21.15.158:0/317675851 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:10.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.274+0000 7f5cfa899700 1 --2- 172.21.15.158:0/317675851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:10.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfa098700 1 -- 172.21.15.158:0/317675851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 msgr2=0x7f5cf40cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:10.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfa098700 1 --2- 172.21.15.158:0/317675851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf40cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfa098700 1 -- 172.21.15.158:0/317675851 >> v1:172.21.15.158:6789/0 conn(0x7f5cf415deb0 legacy=0x7f5cf40ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:10.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfa098700 1 -- 172.21.15.158:0/317675851 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf40cc280 con 0x7f5cf415dad0 2023-10-16T10:03:10.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cf9096700 1 -- 172.21.15.158:0/317675851 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954841946 0 0) 0x7f5cf40d0ae0 con 0x7f5cf415deb0 2023-10-16T10:03:10.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfa098700 1 --2- 172.21.15.158:0/317675851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415dad0 0x7f5cf40cdfa0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4009840 tx=0x7f5ce4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997d14dfdfd911bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:10.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cf9096700 1 -- 172.21.15.158:0/317675851 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4011830 con 0x7f5cf415dad0 2023-10-16T10:03:10.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cf9096700 1 -- 172.21.15.158:0/317675851 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce4011990 con 0x7f5cf415dad0 2023-10-16T10:03:10.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cf9096700 1 -- 172.21.15.158:0/317675851 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4011d20 con 0x7f5cf415dad0 2023-10-16T10:03:10.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/317675851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415dad0 msgr2=0x7f5cf40cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:10.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/317675851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415dad0 0x7f5cf40cdfa0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4009840 tx=0x7f5ce4009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:10.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/317675851 shutdown_connections 2023-10-16T10:03:10.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/317675851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415dad0 0x7f5cf40cdfa0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/317675851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf40cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.275+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/317675851 >> 172.21.15.158:0/317675851 conn(0x7f5cf4099a10 msgr2=0x7f5cf40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:10.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.276+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/317675851 shutdown_connections 2023-10-16T10:03:10.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.276+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/317675851 wait complete. 2023-10-16T10:03:10.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 Processor -- start 2023-10-16T10:03:10.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 -- start start 2023-10-16T10:03:10.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf41f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:10.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cf415dad0 0x7f5cf4146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:10.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415deb0 0x7f5cf40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:10.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf406ac20 con 0x7f5cf415deb0 2023-10-16T10:03:10.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf41f2f80 con 0x7f5cf415dad0 2023-10-16T10:03:10.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfc2fc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf4147190 con 0x7f5cf40ccc00 2023-10-16T10:03:10.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.277+0000 7f5cfa899700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415deb0 0x7f5cf40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:10.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415deb0 0x7f5cf40c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57766/0 (socket says 172.21.15.158:57766) 2023-10-16T10:03:10.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 -- 172.21.15.158:0/4279572087 learned_addr learned my addr 172.21.15.158:0/4279572087 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:10.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa098700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf41f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:10.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cf9897700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cf415dad0 0x7f5cf4146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:10.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 -- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 msgr2=0x7f5cf41f2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:10.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf41f2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 -- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cf415dad0 msgr2=0x7f5cf4146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:10.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cf415dad0 0x7f5cf4146c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:10.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 -- 172.21.15.158:0/4279572087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40094a0 con 0x7f5cf415deb0 2023-10-16T10:03:10.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cf9897700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cf415dad0 0x7f5cf4146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:10.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfa899700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415deb0 0x7f5cf40c74a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f5cec007e40 tx=0x7f5cec00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:10.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5ceaffd700 1 -- 172.21.15.158:0/4279572087 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec00ede0 con 0x7f5cf415deb0 2023-10-16T10:03:10.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5ceaffd700 1 -- 172.21.15.158:0/4279572087 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec014420 con 0x7f5cf415deb0 2023-10-16T10:03:10.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5ceaffd700 1 -- 172.21.15.158:0/4279572087 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec0147d0 con 0x7f5cf415deb0 2023-10-16T10:03:10.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf40c9a10 con 0x7f5cf415deb0 2023-10-16T10:03:10.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.278+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf40c9fd0 con 0x7f5cf415deb0 2023-10-16T10:03:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.279+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf41441f0 con 0x7f5cf415deb0 2023-10-16T10:03:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.279+0000 7f5ceaffd700 1 -- 172.21.15.158:0/4279572087 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 72) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f5cec014930 con 0x7f5cf415deb0 2023-10-16T10:03:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.280+0000 7f5ceaffd700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f5cd806ad50 0x7f5cd806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.280+0000 7f5ceaffd700 1 -- 172.21.15.158:0/4279572087 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f5cec08a030 con 0x7f5cf415deb0 2023-10-16T10:03:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.280+0000 7f5cfa098700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f5cd806ad50 0x7f5cd806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.280+0000 7f5cfa098700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f5cd806ad50 0x7f5cd806d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40090b0 tx=0x7f5ce4008030 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.284+0000 7f5ceaffd700 1 -- 172.21.15.158:0/4279572087 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5cf41441f0 con 0x7f5cf415deb0 2023-10-16T10:03:10.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:10.527+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5cf41441f0 con 0x7f5cf415deb0 2023-10-16T10:03:11.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.151+0000 7f5ceaffd700 1 -- 172.21.15.158:0/4279572087 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5cf41441f0 con 0x7f5cf415deb0 2023-10-16T10:03:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f5cd806ad50 msgr2=0x7f5cd806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f5cd806ad50 0x7f5cd806d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40090b0 tx=0x7f5ce4008030 comp rx=0 tx=0).stop 2023-10-16T10:03:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415deb0 msgr2=0x7f5cf40c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415deb0 0x7f5cf40c74a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f5cec007e40 tx=0x7f5cec00e3c0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 shutdown_connections 2023-10-16T10:03:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f5cd806ad50 0x7f5cd806d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf415deb0 0x7f5cf40c74a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cf40ccc00 0x7f5cf41f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.154+0000 7f5cfc2fc700 1 --2- 172.21.15.158:0/4279572087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cf415dad0 0x7f5cf4146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.155+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 >> 172.21.15.158:0/4279572087 conn(0x7f5cf4099a10 msgr2=0x7f5cf406f000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.155+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 shutdown_connections 2023-10-16T10:03:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.155+0000 7f5cfc2fc700 1 -- 172.21.15.158:0/4279572087 wait complete. 2023-10-16T10:03:11.170 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T10:03:11.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.333+0000 7f7712db7700 1 Processor -- start 2023-10-16T10:03:11.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.333+0000 7f7712db7700 1 -- start start 2023-10-16T10:03:11.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7712db7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f770c0c4e60 0x7f770c06b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:11.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7712db7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770c0c52b0 con 0x7f770c0c49b0 2023-10-16T10:03:11.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7712db7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770c0ce110 con 0x7f770c160d40 2023-10-16T10:03:11.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7712db7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770c0c61e0 con 0x7f770c0c4e60 2023-10-16T10:03:11.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f770bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f770c0c49b0 0x7f770c06abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:57968/0 (socket says 172.21.15.158:57968) 2023-10-16T10:03:11.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f770bfff700 1 -- 172.21.15.158:0/2921816043 learned_addr learned my addr 172.21.15.158:0/2921816043 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:11.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7711354700 1 --2- 172.21.15.158:0/2921816043 >> v2:172.21.15.172:3300/0 conn(0x7f770c0c4e60 0x7f770c06b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:11.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f770b7fe700 1 -- 172.21.15.158:0/2921816043 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734506052 0 0) 0x7f770c0c52b0 con 0x7f770c0c49b0 2023-10-16T10:03:11.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f770b7fe700 1 -- 172.21.15.158:0/2921816043 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76fc0035b0 con 0x7f770c0c49b0 2023-10-16T10:03:11.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7711354700 1 -- 172.21.15.158:0/2921816043 >> v1:172.21.15.172:6789/0 conn(0x7f770c160d40 legacy=0x7f770c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:11.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7711354700 1 -- 172.21.15.158:0/2921816043 >> v1:172.21.15.158:6789/0 conn(0x7f770c0c49b0 legacy=0x7f770c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:11.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.334+0000 7f7711354700 1 -- 172.21.15.158:0/2921816043 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c0ce3f0 con 0x7f770c0c4e60 2023-10-16T10:03:11.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7711354700 1 --2- 172.21.15.158:0/2921816043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c4e60 0x7f770c06b2c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7704009d90 tx=0x7f770400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=593c946c4b77ceff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:11.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f770b7fe700 1 -- 172.21.15.158:0/2921816043 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770400e070 con 0x7f770c0c4e60 2023-10-16T10:03:11.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f770b7fe700 1 -- 172.21.15.158:0/2921816043 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770400bb20 con 0x7f770c0c4e60 2023-10-16T10:03:11.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f770b7fe700 1 -- 172.21.15.158:0/2921816043 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770400be90 con 0x7f770c0c4e60 2023-10-16T10:03:11.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7712db7700 1 -- 172.21.15.158:0/2921816043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c4e60 msgr2=0x7f770c06b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:11.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7712db7700 1 --2- 172.21.15.158:0/2921816043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c4e60 0x7f770c06b2c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7704009d90 tx=0x7f770400b330 comp rx=0 tx=0).stop 2023-10-16T10:03:11.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7712db7700 1 -- 172.21.15.158:0/2921816043 shutdown_connections 2023-10-16T10:03:11.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7712db7700 1 --2- 172.21.15.158:0/2921816043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c4e60 0x7f770c06b2c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7712db7700 1 -- 172.21.15.158:0/2921816043 >> 172.21.15.158:0/2921816043 conn(0x7f770c0cc5b0 msgr2=0x7f770c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:11.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7712db7700 1 -- 172.21.15.158:0/2921816043 shutdown_connections 2023-10-16T10:03:11.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.335+0000 7f7712db7700 1 -- 172.21.15.158:0/2921816043 wait complete. 2023-10-16T10:03:11.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.336+0000 7f7712db7700 1 Processor -- start 2023-10-16T10:03:11.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f7712db7700 1 -- start start 2023-10-16T10:03:11.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f7712db7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c49b0 0x7f770c0c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:11.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f7712db7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f770c0c4e60 0x7f770c0af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:11.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f7712db7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770c160d40 0x7f770c155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:11.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f7712db7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c0ad210 con 0x7f770c0c4e60 2023-10-16T10:03:11.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f7712db7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c0ad460 con 0x7f770c0c49b0 2023-10-16T10:03:11.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f7712db7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770c0ad740 con 0x7f770c160d40 2023-10-16T10:03:11.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f770bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f770c0c4e60 0x7f770c0af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:11.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f770bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f770c0c4e60 0x7f770c0af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57798/0 (socket says 172.21.15.158:57798) 2023-10-16T10:03:11.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.337+0000 7f770bfff700 1 -- 172.21.15.158:0/2789186324 learned_addr learned my addr 172.21.15.158:0/2789186324 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:11.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f7710b53700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c49b0 0x7f770c0c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:11.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f7711354700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770c160d40 0x7f770c155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:11.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f770bfff700 1 -- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770c160d40 msgr2=0x7f770c155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:11.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f770bfff700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770c160d40 0x7f770c155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f770bfff700 1 -- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c49b0 msgr2=0x7f770c0c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:11.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f770bfff700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c49b0 0x7f770c0c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:11.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f770bfff700 1 -- 172.21.15.158:0/2789186324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7704009a70 con 0x7f770c0c4e60 2023-10-16T10:03:11.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f770bfff700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f770c0c4e60 0x7f770c0af3d0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f76f800c700 tx=0x7f76f800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:11.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f77097fa700 1 -- 172.21.15.158:0/2789186324 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8005d70 con 0x7f770c0c4e60 2023-10-16T10:03:11.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f77097fa700 1 -- 172.21.15.158:0/2789186324 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f80070d0 con 0x7f770c0c4e60 2023-10-16T10:03:11.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770c0ada20 con 0x7f770c0c4e60 2023-10-16T10:03:11.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f77097fa700 1 -- 172.21.15.158:0/2789186324 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8007460 con 0x7f770c0c4e60 2023-10-16T10:03:11.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.338+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770c1563b0 con 0x7f770c0c4e60 2023-10-16T10:03:11.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.339+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770c066db0 con 0x7f770c0c4e60 2023-10-16T10:03:11.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.339+0000 7f77097fa700 1 -- 172.21.15.158:0/2789186324 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 72) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f76f80075c0 con 0x7f770c0c4e60 2023-10-16T10:03:11.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.340+0000 7f77097fa700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f76f006ad50 0x7f76f006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:11.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.340+0000 7f77097fa700 1 -- 172.21.15.158:0/2789186324 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f76f8089080 con 0x7f770c0c4e60 2023-10-16T10:03:11.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.340+0000 7f7710b53700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f76f006ad50 0x7f76f006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:11.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.340+0000 7f7710b53700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f76f006ad50 0x7f76f006d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7700001620 tx=0x7f7700009040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:11.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.344+0000 7f77097fa700 1 -- 172.21.15.158:0/2789186324 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76f80bbfe0 con 0x7f770c0c4e60 2023-10-16T10:03:11.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:11.537+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f770c0681b0 con 0x7f770c0c4e60 2023-10-16T10:03:12.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.156+0000 7f77097fa700 1 -- 172.21.15.158:0/2789186324 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 181+0+0 (secure 0 0 0) 0x7f76f8053ca0 con 0x7f770c0c4e60 2023-10-16T10:03:12.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f76f006ad50 msgr2=0x7f76f006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:12.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f76f006ad50 0x7f76f006d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7700001620 tx=0x7f7700009040 comp rx=0 tx=0).stop 2023-10-16T10:03:12.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f770c0c4e60 msgr2=0x7f770c0af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:12.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f770c0c4e60 0x7f770c0af3d0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f76f800c700 tx=0x7f76f800cac0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 shutdown_connections 2023-10-16T10:03:12.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f76f006ad50 0x7f76f006d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770c160d40 0x7f770c155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f770c0c49b0 0x7f770c0c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 --2- 172.21.15.158:0/2789186324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f770c0c4e60 0x7f770c0af3d0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 >> 172.21.15.158:0/2789186324 conn(0x7f770c0cc5b0 msgr2=0x7f770c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:12.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.162+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 shutdown_connections 2023-10-16T10:03:12.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.163+0000 7f7712db7700 1 -- 172.21.15.158:0/2789186324 wait complete. 2023-10-16T10:03:12.167 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T10:03:12.179 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T10:03:12.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.305+0000 7f45fdb77700 1 Processor -- start 2023-10-16T10:03:12.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.305+0000 7f45fdb77700 1 -- start start 2023-10-16T10:03:12.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.306+0000 7f45fdb77700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f815da40 0x7f45f80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:12.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.306+0000 7f45fdb77700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f45f80cc590 0x7f45f806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:12.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.306+0000 7f45fdb77700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f80cbd30 con 0x7f45f806ae10 2023-10-16T10:03:12.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.306+0000 7f45fdb77700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f806b280 con 0x7f45f80cc590 2023-10-16T10:03:12.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.306+0000 7f45fdb77700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80cc1c0 con 0x7f45f815da40 2023-10-16T10:03:12.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.306+0000 7f45f77fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f815da40 0x7f45f80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:12.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.306+0000 7f45f6ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f45f80cc590 0x7f45f806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:12.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f7fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f45f806ae10 0x7f45f80ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:33656/0 (socket says 172.21.15.158:33656) 2023-10-16T10:03:12.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f7fff700 1 -- 172.21.15.158:0/2524051791 learned_addr learned my addr 172.21.15.158:0/2524051791 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:12.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2524051791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f815da40 msgr2=0x7f45f80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:12.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f6ffd700 1 --2- 172.21.15.158:0/2524051791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f815da40 0x7f45f80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2524051791 >> v1:172.21.15.172:6789/0 conn(0x7f45f806ae10 legacy=0x7f45f80ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f6ffd700 1 -- 172.21.15.158:0/2524051791 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f815de20 con 0x7f45f80cc590 2023-10-16T10:03:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f6ffd700 1 --2- 172.21.15.158:0/2524051791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80cc590 0x7f45f806a8d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009840 tx=0x7f45ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ece704f4b2545150 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f67fc700 1 -- 172.21.15.158:0/2524051791 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec011830 con 0x7f45f80cc590 2023-10-16T10:03:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f67fc700 1 -- 172.21.15.158:0/2524051791 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec011990 con 0x7f45f80cc590 2023-10-16T10:03:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.307+0000 7f45f67fc700 1 -- 172.21.15.158:0/2524051791 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec011d20 con 0x7f45f80cc590 2023-10-16T10:03:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 -- 172.21.15.158:0/2524051791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80cc590 msgr2=0x7f45f806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 --2- 172.21.15.158:0/2524051791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80cc590 0x7f45f806a8d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009840 tx=0x7f45ec009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 -- 172.21.15.158:0/2524051791 shutdown_connections 2023-10-16T10:03:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 --2- 172.21.15.158:0/2524051791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f815da40 0x7f45f80caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 --2- 172.21.15.158:0/2524051791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f80cc590 0x7f45f806a8d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 -- 172.21.15.158:0/2524051791 >> 172.21.15.158:0/2524051791 conn(0x7f45f8099a10 msgr2=0x7f45f80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 -- 172.21.15.158:0/2524051791 shutdown_connections 2023-10-16T10:03:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.308+0000 7f45fdb77700 1 -- 172.21.15.158:0/2524051791 wait complete. 2023-10-16T10:03:12.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.309+0000 7f45fdb77700 1 Processor -- start 2023-10-16T10:03:12.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.309+0000 7f45fdb77700 1 -- start start 2023-10-16T10:03:12.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45fdb77700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f806ae10 0x7f45f8147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:12.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45fdb77700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f815da40 0x7f45f80c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:12.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45fdb77700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b36d0 0x7f45f80b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:12.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45fdb77700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80b40c0 con 0x7f45f806ae10 2023-10-16T10:03:12.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45fdb77700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80c9f20 con 0x7f45f815da40 2023-10-16T10:03:12.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45fdb77700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80ca3b0 con 0x7f45f80b36d0 2023-10-16T10:03:12.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f806ae10 0x7f45f8147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f806ae10 0x7f45f8147160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57810/0 (socket says 172.21.15.158:57810) 2023-10-16T10:03:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 -- 172.21.15.158:0/996104119 learned_addr learned my addr 172.21.15.158:0/996104119 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f6ffd700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f815da40 0x7f45f80c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f7fff700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b36d0 0x7f45f80b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 -- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b36d0 msgr2=0x7f45f80b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b36d0 0x7f45f80b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 -- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f815da40 msgr2=0x7f45f80c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f815da40 0x7f45f80c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 -- 172.21.15.158:0/996104119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec0094a0 con 0x7f45f806ae10 2023-10-16T10:03:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.310+0000 7f45f77fe700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f806ae10 0x7f45f8147160 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f45e800e8a0 tx=0x7f45e800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.311+0000 7f45dffff700 1 -- 172.21.15.158:0/996104119 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8009590 con 0x7f45f806ae10 2023-10-16T10:03:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.311+0000 7f45dffff700 1 -- 172.21.15.158:0/996104119 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e8013680 con 0x7f45f806ae10 2023-10-16T10:03:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.311+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f80c95f0 con 0x7f45f806ae10 2023-10-16T10:03:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.311+0000 7f45dffff700 1 -- 172.21.15.158:0/996104119 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e80139f0 con 0x7f45f806ae10 2023-10-16T10:03:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.311+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f80c9bb0 con 0x7f45f806ae10 2023-10-16T10:03:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.312+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f80681b0 con 0x7f45f806ae10 2023-10-16T10:03:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.312+0000 7f45dffff700 1 -- 172.21.15.158:0/996104119 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 73) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f45e8013b50 con 0x7f45f806ae10 2023-10-16T10:03:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.313+0000 7f45dffff700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f45e406afc0 0x7f45e406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.313+0000 7f45f6ffd700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f45e406afc0 0x7f45e406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.313+0000 7f45dffff700 1 -- 172.21.15.158:0/996104119 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6903+0+0 (secure 0 0 0) 0x7f45e8089910 con 0x7f45f806ae10 2023-10-16T10:03:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.313+0000 7f45f6ffd700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f45e406afc0 0x7f45e406d480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f45f80cdec0 tx=0x7f45ec008030 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:12.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.317+0000 7f45dffff700 1 -- 172.21.15.158:0/996104119 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f45e8054520 con 0x7f45f806ae10 2023-10-16T10:03:12.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:12.510+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f45f80cb920 con 0x7f45f806ae10 2023-10-16T10:03:13.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.157+0000 7f45dffff700 1 -- 172.21.15.158:0/996104119 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 169+0+0 (secure 0 0 0) 0x7f45e8057b40 con 0x7f45f806ae10 2023-10-16T10:03:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f45e406afc0 msgr2=0x7f45e406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f45e406afc0 0x7f45e406d480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f45f80cdec0 tx=0x7f45ec008030 comp rx=0 tx=0).stop 2023-10-16T10:03:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f806ae10 msgr2=0x7f45f8147160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f806ae10 0x7f45f8147160 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f45e800e8a0 tx=0x7f45e800ebb0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 shutdown_connections 2023-10-16T10:03:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f45e406afc0 0x7f45e406d480 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f80b36d0 0x7f45f80b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45f806ae10 0x7f45f8147160 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 --2- 172.21.15.158:0/996104119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f815da40 0x7f45f80c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 >> 172.21.15.158:0/996104119 conn(0x7f45f8099a10 msgr2=0x7f45f80b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:13.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 shutdown_connections 2023-10-16T10:03:13.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.160+0000 7f45fdb77700 1 -- 172.21.15.158:0/996104119 wait complete. 2023-10-16T10:03:13.165 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T10:03:13.180 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T10:03:13.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.306+0000 7f4d794fb700 1 Processor -- start 2023-10-16T10:03:13.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.306+0000 7f4d794fb700 1 -- start start 2023-10-16T10:03:13.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d794fb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4d7415d640 0x7f4d740cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d794fb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4d7415da20 0x7f4d740cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d794fb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d7406bd90 con 0x7f4d7406d4d0 2023-10-16T10:03:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d794fb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d740b3f70 con 0x7f4d7415d640 2023-10-16T10:03:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d794fb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d7406c070 con 0x7f4d7415da20 2023-10-16T10:03:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d72ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4d7415d640 0x7f4d740cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d72ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4d7415d640 0x7f4d740cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57818/0 (socket says 172.21.15.158:57818) 2023-10-16T10:03:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.307+0000 7f4d72ffd700 1 -- 172.21.15.158:0/403575710 learned_addr learned my addr 172.21.15.158:0/403575710 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d72ffd700 1 -- 172.21.15.158:0/403575710 >> v2:172.21.15.172:3300/0 conn(0x7f4d7415da20 msgr2=0x7f4d740cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d727fc700 1 --2- 172.21.15.158:0/403575710 >> v2:172.21.15.172:3300/0 conn(0x7f4d7415da20 0x7f4d740cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d72ffd700 1 --2- 172.21.15.158:0/403575710 >> v2:172.21.15.172:3300/0 conn(0x7f4d7415da20 0x7f4d740cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d72ffd700 1 -- 172.21.15.158:0/403575710 >> v1:172.21.15.172:6789/0 conn(0x7f4d7406d4d0 legacy=0x7f4d740d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d72ffd700 1 -- 172.21.15.158:0/403575710 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7406c350 con 0x7f4d7415d640 2023-10-16T10:03:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d727fc700 1 --2- 172.21.15.158:0/403575710 >> v2:172.21.15.172:3300/0 conn(0x7f4d7415da20 0x7f4d740cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d72ffd700 1 --2- 172.21.15.158:0/403575710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740cf070 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f4d74162480 tx=0x7f4d68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a500c2e215f4b1c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d71ffb700 1 -- 172.21.15.158:0/403575710 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68004230 con 0x7f4d7415d640 2023-10-16T10:03:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d71ffb700 1 -- 172.21.15.158:0/403575710 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d68004390 con 0x7f4d7415d640 2023-10-16T10:03:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.308+0000 7f4d71ffb700 1 -- 172.21.15.158:0/403575710 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d680036a0 con 0x7f4d7415d640 2023-10-16T10:03:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 -- 172.21.15.158:0/403575710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 msgr2=0x7f4d740cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 --2- 172.21.15.158:0/403575710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740cf070 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f4d74162480 tx=0x7f4d68009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 -- 172.21.15.158:0/403575710 shutdown_connections 2023-10-16T10:03:13.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 --2- 172.21.15.158:0/403575710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740cf070 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 --2- 172.21.15.158:0/403575710 >> v2:172.21.15.172:3300/0 conn(0x7f4d7415da20 0x7f4d740cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 -- 172.21.15.158:0/403575710 >> 172.21.15.158:0/403575710 conn(0x7f4d7406a8d0 msgr2=0x7f4d7406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:13.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 -- 172.21.15.158:0/403575710 shutdown_connections 2023-10-16T10:03:13.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.309+0000 7f4d794fb700 1 -- 172.21.15.158:0/403575710 wait complete. 2023-10-16T10:03:13.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.310+0000 7f4d794fb700 1 Processor -- start 2023-10-16T10:03:13.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.310+0000 7f4d794fb700 1 -- start start 2023-10-16T10:03:13.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d794fb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d7406d4d0 0x7f4d741512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d794fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d794fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d740b4600 0x7f4d740b5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d794fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d740b4a90 con 0x7f4d7415d640 2023-10-16T10:03:13.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d794fb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7415eb70 con 0x7f4d7406d4d0 2023-10-16T10:03:13.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d794fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d740b64f0 con 0x7f4d740b4600 2023-10-16T10:03:13.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57826/0 (socket says 172.21.15.158:57826) 2023-10-16T10:03:13.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 -- 172.21.15.158:0/1947233662 learned_addr learned my addr 172.21.15.158:0/1947233662 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:13.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d72ffd700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d7406d4d0 0x7f4d741512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d737fe700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d740b4600 0x7f4d740b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 -- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d740b4600 msgr2=0x7f4d740b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d740b4600 0x7f4d740b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 -- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d7406d4d0 msgr2=0x7f4d741512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d7406d4d0 0x7f4d741512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 -- 172.21.15.158:0/1947233662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d68009550 con 0x7f4d7415d640 2023-10-16T10:03:13.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d727fc700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740b40c0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400c7f0 tx=0x7f4d6400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:13.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d5b7fe700 1 -- 172.21.15.158:0/1947233662 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64011800 con 0x7f4d7415d640 2023-10-16T10:03:13.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.311+0000 7f4d5b7fe700 1 -- 172.21.15.158:0/1947233662 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d64011960 con 0x7f4d7415d640 2023-10-16T10:03:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.312+0000 7f4d5b7fe700 1 -- 172.21.15.158:0/1947233662 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64011d10 con 0x7f4d7415d640 2023-10-16T10:03:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.312+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d740b67d0 con 0x7f4d7415d640 2023-10-16T10:03:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.312+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d740c5180 con 0x7f4d7415d640 2023-10-16T10:03:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.312+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d740681b0 con 0x7f4d7415d640 2023-10-16T10:03:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.313+0000 7f4d5b7fe700 1 -- 172.21.15.158:0/1947233662 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 73) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4d64018360 con 0x7f4d7415d640 2023-10-16T10:03:13.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.313+0000 7f4d5b7fe700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d6006af90 0x7f4d6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.314+0000 7f4d72ffd700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d6006af90 0x7f4d6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.314+0000 7f4d5b7fe700 1 -- 172.21.15.158:0/1947233662 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f4d64089380 con 0x7f4d7415d640 2023-10-16T10:03:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.314+0000 7f4d72ffd700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d6006af90 0x7f4d6006d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009130 tx=0x7f4d68008030 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:13.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.318+0000 7f4d5b7fe700 1 -- 172.21.15.158:0/1947233662 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d640540b0 con 0x7f4d7415d640 2023-10-16T10:03:13.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.513+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f4d74151e40 con 0x7f4d7415d640 2023-10-16T10:03:13.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.517+0000 7f4d5b7fe700 1 -- 172.21.15.158:0/1947233662 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4d640576d0 con 0x7f4d7415d640 2023-10-16T10:03:13.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d6006af90 msgr2=0x7f4d6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d6006af90 0x7f4d6006d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009130 tx=0x7f4d68008030 comp rx=0 tx=0).stop 2023-10-16T10:03:13.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 msgr2=0x7f4d740b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740b40c0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400c7f0 tx=0x7f4d6400cb00 comp rx=0 tx=0).stop 2023-10-16T10:03:13.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 shutdown_connections 2023-10-16T10:03:13.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d6006af90 0x7f4d6006d450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d740b4600 0x7f4d740b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d7406d4d0 0x7f4d741512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 --2- 172.21.15.158:0/1947233662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d7415d640 0x7f4d740b40c0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 >> 172.21.15.158:0/1947233662 conn(0x7f4d7406a8d0 msgr2=0x7f4d7415b400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:13.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 shutdown_connections 2023-10-16T10:03:13.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.519+0000 7f4d794fb700 1 -- 172.21.15.158:0/1947233662 wait complete. 2023-10-16T10:03:13.534 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T10:03:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.655+0000 7fe5ee844700 1 Processor -- start 2023-10-16T10:03:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.655+0000 7fe5ee844700 1 -- start start 2023-10-16T10:03:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5ee844700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e8143790 0x7fe5e806bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5ee844700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5e80c50a0 0x7fe5e80c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5ee844700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe5e806e540 0x7fe5e806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5ee844700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e8143b70 con 0x7fe5e80c50a0 2023-10-16T10:03:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5ee844700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e80c5a60 con 0x7fe5e806e540 2023-10-16T10:03:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5ee844700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e806ef10 con 0x7fe5e8143790 2023-10-16T10:03:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5e77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5e80c50a0 0x7fe5e80c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5e77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5e80c50a0 0x7fe5e80c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57832/0 (socket says 172.21.15.158:57832) 2023-10-16T10:03:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.656+0000 7fe5e77fe700 1 -- 172.21.15.158:0/4178897562 learned_addr learned my addr 172.21.15.158:0/4178897562 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5ecde1700 1 --2- 172.21.15.158:0/4178897562 >> v2:172.21.15.172:3300/0 conn(0x7fe5e806e540 0x7fe5e806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e7fff700 1 --2- 172.21.15.158:0/4178897562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e8143790 0x7fe5e806bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e77fe700 1 -- 172.21.15.158:0/4178897562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e8143790 msgr2=0x7fe5e806bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e77fe700 1 --2- 172.21.15.158:0/4178897562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e8143790 0x7fe5e806bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e77fe700 1 -- 172.21.15.158:0/4178897562 >> v2:172.21.15.172:3300/0 conn(0x7fe5e806e540 msgr2=0x7fe5e806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:13.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e77fe700 1 --2- 172.21.15.158:0/4178897562 >> v2:172.21.15.172:3300/0 conn(0x7fe5e806e540 0x7fe5e806e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e77fe700 1 -- 172.21.15.158:0/4178897562 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e806aa70 con 0x7fe5e80c50a0 2023-10-16T10:03:13.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e77fe700 1 --2- 172.21.15.158:0/4178897562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e80c50a0 0x7fe5e80c5520 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80cd250 tx=0x7fe5d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca7ffb960908 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:13.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e6ffd700 1 -- 172.21.15.158:0/4178897562 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8014e90 con 0x7fe5e80c50a0 2023-10-16T10:03:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e6ffd700 1 -- 172.21.15.158:0/4178897562 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d801c440 con 0x7fe5e80c50a0 2023-10-16T10:03:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.657+0000 7fe5e6ffd700 1 -- 172.21.15.158:0/4178897562 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d801c7d0 con 0x7fe5e80c50a0 2023-10-16T10:03:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 -- 172.21.15.158:0/4178897562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e80c50a0 msgr2=0x7fe5e80c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 --2- 172.21.15.158:0/4178897562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e80c50a0 0x7fe5e80c5520 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80cd250 tx=0x7fe5d8009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 -- 172.21.15.158:0/4178897562 shutdown_connections 2023-10-16T10:03:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 --2- 172.21.15.158:0/4178897562 >> v2:172.21.15.172:3300/0 conn(0x7fe5e806e540 0x7fe5e806e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 --2- 172.21.15.158:0/4178897562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e8143790 0x7fe5e806bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 --2- 172.21.15.158:0/4178897562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e80c50a0 0x7fe5e80c5520 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 -- 172.21.15.158:0/4178897562 >> 172.21.15.158:0/4178897562 conn(0x7fe5e80cc5b0 msgr2=0x7fe5e80cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:13.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 -- 172.21.15.158:0/4178897562 shutdown_connections 2023-10-16T10:03:13.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.658+0000 7fe5ee844700 1 -- 172.21.15.158:0/4178897562 wait complete. 2023-10-16T10:03:13.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.659+0000 7fe5ee844700 1 Processor -- start 2023-10-16T10:03:13.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.659+0000 7fe5ee844700 1 -- start start 2023-10-16T10:03:13.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ee844700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e806e540 0x7fe5e81f1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ee844700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e80c50a0 0x7fe5e815a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ee844700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e8143790 0x7fe5e815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ee844700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8142ea0 con 0x7fe5e8143790 2023-10-16T10:03:13.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ee844700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80cbad0 con 0x7fe5e80c50a0 2023-10-16T10:03:13.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ee844700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80cad10 con 0x7fe5e806e540 2023-10-16T10:03:13.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e8143790 0x7fe5e815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e8143790 0x7fe5e815ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57842/0 (socket says 172.21.15.158:57842) 2023-10-16T10:03:13.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 -- 172.21.15.158:0/1048472129 learned_addr learned my addr 172.21.15.158:0/1048472129 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:13.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5e7fff700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e806e540 0x7fe5e81f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5e77fe700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e80c50a0 0x7fe5e815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 -- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e806e540 msgr2=0x7fe5e81f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e806e540 0x7fe5e81f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 -- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e80c50a0 msgr2=0x7fe5e815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e80c50a0 0x7fe5e815a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 -- 172.21.15.158:0/1048472129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8009550 con 0x7fe5e8143790 2023-10-16T10:03:13.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ecde1700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e8143790 0x7fe5e815ad70 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009ea0 tx=0x7fe5e00084e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:13.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5e4ff9700 1 -- 172.21.15.158:0/1048472129 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0010070 con 0x7fe5e8143790 2023-10-16T10:03:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5e4ff9700 1 -- 172.21.15.158:0/1048472129 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e000be80 con 0x7fe5e8143790 2023-10-16T10:03:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e80caff0 con 0x7fe5e8143790 2023-10-16T10:03:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.660+0000 7fe5e4ff9700 1 -- 172.21.15.158:0/1048472129 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0008c80 con 0x7fe5e8143790 2023-10-16T10:03:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.661+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e80cb5b0 con 0x7fe5e8143790 2023-10-16T10:03:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.662+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e80cb7a0 con 0x7fe5e8143790 2023-10-16T10:03:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.662+0000 7fe5e4ff9700 1 -- 172.21.15.158:0/1048472129 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 73) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe5e0014030 con 0x7fe5e8143790 2023-10-16T10:03:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.662+0000 7fe5e4ff9700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe5d406af90 0x7fe5d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.662+0000 7fe5e4ff9700 1 -- 172.21.15.158:0/1048472129 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fe5e0089600 con 0x7fe5e8143790 2023-10-16T10:03:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.662+0000 7fe5e7fff700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe5d406af90 0x7fe5d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.663+0000 7fe5e7fff700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe5d406af90 0x7fe5d406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006b20 tx=0x7fe5dc00f040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:13.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.667+0000 7fe5e4ff9700 1 -- 172.21.15.158:0/1048472129 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5e00541a0 con 0x7fe5e8143790 2023-10-16T10:03:13.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.858+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe5e80681b0 con 0x7fe5e8143790 2023-10-16T10:03:13.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.861+0000 7fe5e4ff9700 1 -- 172.21.15.158:0/1048472129 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe5e00577c0 con 0x7fe5e8143790 2023-10-16T10:03:13.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe5d406af90 msgr2=0x7fe5d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe5d406af90 0x7fe5d406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006b20 tx=0x7fe5dc00f040 comp rx=0 tx=0).stop 2023-10-16T10:03:13.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e8143790 msgr2=0x7fe5e815ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:13.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e8143790 0x7fe5e815ad70 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009ea0 tx=0x7fe5e00084e0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 shutdown_connections 2023-10-16T10:03:13.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe5d406af90 0x7fe5d406d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e8143790 0x7fe5e815ad70 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e806e540 0x7fe5e81f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 --2- 172.21.15.158:0/1048472129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e80c50a0 0x7fe5e815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:13.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 >> 172.21.15.158:0/1048472129 conn(0x7fe5e80cc5b0 msgr2=0x7fe5e8159670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:13.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 shutdown_connections 2023-10-16T10:03:13.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.864+0000 7fe5ee844700 1 -- 172.21.15.158:0/1048472129 wait complete. 2023-10-16T10:03:13.878 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T10:03:14.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.999+0000 7fdaf9b93700 1 Processor -- start 2023-10-16T10:03:14.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:13.999+0000 7fdaf9b93700 1 -- start start 2023-10-16T10:03:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf9b93700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdaf40cfd90 0x7fdaf40cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf9b93700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf40caa80 con 0x7fdaf415db50 2023-10-16T10:03:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf9b93700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf40cad60 con 0x7fdaf415d770 2023-10-16T10:03:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf9b93700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf40cb040 con 0x7fdaf40cfd90 2023-10-16T10:03:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf2ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdaf415db50 0x7fdaf40cf680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58024/0 (socket says 172.21.15.158:58024) 2023-10-16T10:03:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf2ffd700 1 -- 172.21.15.158:0/1702048399 learned_addr learned my addr 172.21.15.158:0/1702048399 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf3fff700 1 --2- 172.21.15.158:0/1702048399 >> v2:172.21.15.172:3300/0 conn(0x7fdaf40cfd90 0x7fdaf40cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf27fc700 1 -- 172.21.15.158:0/1702048399 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724076896 0 0) 0x7fdaf40caa80 con 0x7fdaf415db50 2023-10-16T10:03:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf27fc700 1 -- 172.21.15.158:0/1702048399 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdadc0035b0 con 0x7fdaf415db50 2023-10-16T10:03:14.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf3fff700 1 -- 172.21.15.158:0/1702048399 >> v1:172.21.15.172:6789/0 conn(0x7fdaf415d770 legacy=0x7fdaf40cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf3fff700 1 -- 172.21.15.158:0/1702048399 >> v1:172.21.15.158:6789/0 conn(0x7fdaf415db50 legacy=0x7fdaf40cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.000+0000 7fdaf3fff700 1 -- 172.21.15.158:0/1702048399 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf40cb320 con 0x7fdaf40cfd90 2023-10-16T10:03:14.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf3fff700 1 --2- 172.21.15.158:0/1702048399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf40cfd90 0x7fdaf40cbe00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009d90 tx=0x7fdaec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b0090ecf8547f0a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf27fc700 1 -- 172.21.15.158:0/1702048399 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec00e070 con 0x7fdaf40cfd90 2023-10-16T10:03:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf27fc700 1 -- 172.21.15.158:0/1702048399 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec00bb20 con 0x7fdaf40cfd90 2023-10-16T10:03:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf27fc700 1 -- 172.21.15.158:0/1702048399 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec012430 con 0x7fdaf40cfd90 2023-10-16T10:03:14.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf9b93700 1 -- 172.21.15.158:0/1702048399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf40cfd90 msgr2=0x7fdaf40cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/1702048399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf40cfd90 0x7fdaf40cbe00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009d90 tx=0x7fdaec00b330 comp rx=0 tx=0).stop 2023-10-16T10:03:14.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf9b93700 1 -- 172.21.15.158:0/1702048399 shutdown_connections 2023-10-16T10:03:14.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/1702048399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf40cfd90 0x7fdaf40cbe00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf9b93700 1 -- 172.21.15.158:0/1702048399 >> 172.21.15.158:0/1702048399 conn(0x7fdaf4099a10 msgr2=0x7fdaf40d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:14.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf9b93700 1 -- 172.21.15.158:0/1702048399 shutdown_connections 2023-10-16T10:03:14.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.001+0000 7fdaf9b93700 1 -- 172.21.15.158:0/1702048399 wait complete. 2023-10-16T10:03:14.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.002+0000 7fdaf9b93700 1 Processor -- start 2023-10-16T10:03:14.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.003+0000 7fdaf9b93700 1 -- start start 2023-10-16T10:03:14.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.003+0000 7fdaf9b93700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf40cfd90 0x7fdaf41f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.003+0000 7fdaf9b93700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf415d770 0x7fdaf41465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.003+0000 7fdaf9b93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf415db50 0x7fdaf4160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.003+0000 7fdaf9b93700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40c8550 con 0x7fdaf415db50 2023-10-16T10:03:14.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.003+0000 7fdaf9b93700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf41460d0 con 0x7fdaf415d770 2023-10-16T10:03:14.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.003+0000 7fdaf9b93700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf4146b00 con 0x7fdaf40cfd90 2023-10-16T10:03:14.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf415db50 0x7fdaf4160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf415db50 0x7fdaf4160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57856/0 (socket says 172.21.15.158:57856) 2023-10-16T10:03:14.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 -- 172.21.15.158:0/2876470513 learned_addr learned my addr 172.21.15.158:0/2876470513 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:14.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 -- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf40cfd90 msgr2=0x7fdaf41f72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf37fe700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf40cfd90 0x7fdaf41f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf2ffd700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf415d770 0x7fdaf41465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf40cfd90 0x7fdaf41f72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 -- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf415d770 msgr2=0x7fdaf41465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf415d770 0x7fdaf41465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 -- 172.21.15.158:0/2876470513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec009a70 con 0x7fdaf415db50 2023-10-16T10:03:14.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf3fff700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf415db50 0x7fdaf4160830 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00b680 tx=0x7fdaec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:14.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec012430 con 0x7fdaf415db50 2023-10-16T10:03:14.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec020070 con 0x7fdaf415db50 2023-10-16T10:03:14.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf4156460 con 0x7fdaf415db50 2023-10-16T10:03:14.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec004280 con 0x7fdaf415db50 2023-10-16T10:03:14.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.004+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf40c8a80 con 0x7fdaf415db50 2023-10-16T10:03:14.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.005+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf40681b0 con 0x7fdaf415db50 2023-10-16T10:03:14.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.006+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 73) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdaec0043e0 con 0x7fdaf415db50 2023-10-16T10:03:14.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.006+0000 7fdadbfff700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fdae006af90 0x7fdae006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.006+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fdaec022070 con 0x7fdaf415db50 2023-10-16T10:03:14.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.006+0000 7fdaf37fe700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fdae006af90 0x7fdae006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.007+0000 7fdaf37fe700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fdae006af90 0x7fdae006d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdae8007fd0 tx=0x7fdae8009040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:14.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.011+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdaec056440 con 0x7fdaf415db50 2023-10-16T10:03:14.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.166+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdaec056810 con 0x7fdaf415db50 2023-10-16T10:03:14.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.210+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fdaf40ce420 con 0x7fdaf415db50 2023-10-16T10:03:14.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.213+0000 7fdadbfff700 1 -- 172.21.15.158:0/2876470513 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdaec029d30 con 0x7fdaf415db50 2023-10-16T10:03:14.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fdae006af90 msgr2=0x7fdae006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fdae006af90 0x7fdae006d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdae8007fd0 tx=0x7fdae8009040 comp rx=0 tx=0).stop 2023-10-16T10:03:14.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf415db50 msgr2=0x7fdaf4160830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf415db50 0x7fdaf4160830 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00b680 tx=0x7fdaec004ad0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 shutdown_connections 2023-10-16T10:03:14.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fdae006af90 0x7fdae006d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf415db50 0x7fdaf4160830 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf40cfd90 0x7fdaf41f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 --2- 172.21.15.158:0/2876470513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf415d770 0x7fdaf41465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 >> 172.21.15.158:0/2876470513 conn(0x7fdaf4099a10 msgr2=0x7fdaf40d0290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 shutdown_connections 2023-10-16T10:03:14.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.215+0000 7fdaf9b93700 1 -- 172.21.15.158:0/2876470513 wait complete. 2023-10-16T10:03:14.230 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T10:03:14.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.349+0000 7f8cdbd77700 1 Processor -- start 2023-10-16T10:03:14.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.349+0000 7f8cdbd77700 1 -- start start 2023-10-16T10:03:14.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.350+0000 7f8cdbd77700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cd415dbc0 0x7f8cd40cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.350+0000 7f8cdbd77700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8cd406ba90 0x7f8cd40cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.350+0000 7f8cdbd77700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40d21e0 con 0x7f8cd40d1d50 2023-10-16T10:03:14.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.350+0000 7f8cdbd77700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd4162550 con 0x7f8cd406ba90 2023-10-16T10:03:14.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.350+0000 7f8cdbd77700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd415f320 con 0x7f8cd415dbc0 2023-10-16T10:03:14.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.350+0000 7f8cd9312700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8cd40d1d50 0x7f8cd406beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58036/0 (socket says 172.21.15.158:58036) 2023-10-16T10:03:14.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.350+0000 7f8cd9312700 1 -- 172.21.15.158:0/2206883573 learned_addr learned my addr 172.21.15.158:0/2206883573 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:14.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cda314700 1 --2- 172.21.15.158:0/2206883573 >> v2:172.21.15.172:3300/0 conn(0x7f8cd406ba90 0x7f8cd40cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9b13700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cd415dbc0 0x7f8cd40cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304621930 0 0) 0x7f8cd40d21e0 con 0x7f8cd40d1d50 2023-10-16T10:03:14.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc00035b0 con 0x7f8cd40d1d50 2023-10-16T10:03:14.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 904989510 0 0) 0x7f8cc00035b0 con 0x7f8cd40d1d50 2023-10-16T10:03:14.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cd415dbc0 msgr2=0x7f8cd40cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:14.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cd415dbc0 0x7f8cd40cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 >> v2:172.21.15.172:3300/0 conn(0x7f8cd406ba90 msgr2=0x7f8cd40cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:14.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 --2- 172.21.15.158:0/2206883573 >> v2:172.21.15.172:3300/0 conn(0x7f8cd406ba90 0x7f8cd40cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd415ef10 con 0x7f8cd40d1d50 2023-10-16T10:03:14.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f8cd0001670 con 0x7f8cd40d1d50 2023-10-16T10:03:14.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 >> v1:172.21.15.158:6789/0 conn(0x7f8cd40d1d50 legacy=0x7f8cd406beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cc0003a00 0x7f8cc0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc0004850 0x7f8cc0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0005210 0x7f8cc000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd4162550 con 0x7f8cc0004850 2023-10-16T10:03:14.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc0005680 con 0x7f8cc0003a00 2023-10-16T10:03:14.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc000bf50 con 0x7f8cc0005210 2023-10-16T10:03:14.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9312700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc0004850 0x7f8cc0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9b13700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cc0003a00 0x7f8cc0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cda314700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0005210 0x7f8cc000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9312700 1 -- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0005210 msgr2=0x7f8cc000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:14.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9312700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0005210 0x7f8cc000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9312700 1 -- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cc0003a00 msgr2=0x7f8cc0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:14.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9312700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cc0003a00 0x7f8cc0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.351+0000 7f8cd9312700 1 -- 172.21.15.158:0/2206883573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc00035b0 con 0x7f8cc0004850 2023-10-16T10:03:14.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cd9312700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc0004850 0x7f8cc0004cd0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f8cd40c4e70 tx=0x7f8cc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d20243afdfea51 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:14.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0007c40 con 0x7f8cc0004850 2023-10-16T10:03:14.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd000ce00 con 0x7f8cc0004850 2023-10-16T10:03:14.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cd8b11700 1 -- 172.21.15.158:0/2206883573 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0005ba0 con 0x7f8cc0004850 2023-10-16T10:03:14.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cdbd77700 1 -- 172.21.15.158:0/2206883573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc0004850 msgr2=0x7f8cc0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cdbd77700 1 --2- 172.21.15.158:0/2206883573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc0004850 0x7f8cc0004cd0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f8cd40c4e70 tx=0x7f8cc0005880 comp rx=0 tx=0).stop 2023-10-16T10:03:14.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cda314700 1 -- 172.21.15.158:0/2206883573 reap_dead start 2023-10-16T10:03:14.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cdbd77700 1 -- 172.21.15.158:0/2206883573 shutdown_connections 2023-10-16T10:03:14.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cdbd77700 1 -- 172.21.15.158:0/2206883573 >> 172.21.15.158:0/2206883573 conn(0x7f8cd406a8d0 msgr2=0x7f8cd406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:14.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cdbd77700 1 -- 172.21.15.158:0/2206883573 shutdown_connections 2023-10-16T10:03:14.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.352+0000 7f8cdbd77700 1 -- 172.21.15.158:0/2206883573 wait complete. 2023-10-16T10:03:14.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 Processor -- start 2023-10-16T10:03:14.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 -- start start 2023-10-16T10:03:14.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0004850 0x7f8cd4160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd40c59c0 0x7f8cd40c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cd40cbd20 0x7f8cd4150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd40c5530 con 0x7f8cd40cbd20 2023-10-16T10:03:14.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd40c7800 con 0x7f8cd40c59c0 2023-10-16T10:03:14.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cdbd77700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd40c7ae0 con 0x7f8cc0004850 2023-10-16T10:03:14.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cda314700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cd40cbd20 0x7f8cd4150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cda314700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cd40cbd20 0x7f8cd4150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57876/0 (socket says 172.21.15.158:57876) 2023-10-16T10:03:14.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.354+0000 7f8cda314700 1 -- 172.21.15.158:0/4105843972 learned_addr learned my addr 172.21.15.158:0/4105843972 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:14.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cd9b13700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0004850 0x7f8cd4160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cd9312700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd40c59c0 0x7f8cd40c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cda314700 1 -- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0004850 msgr2=0x7f8cd4160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cda314700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0004850 0x7f8cd4160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cda314700 1 -- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd40c59c0 msgr2=0x7f8cd40c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cda314700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd40c59c0 0x7f8cd40c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cda314700 1 -- 172.21.15.158:0/4105843972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40c4ee0 con 0x7f8cd40cbd20 2023-10-16T10:03:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cda314700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cd40cbd20 0x7f8cd4150780 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f8cd40ce9f0 tx=0x7f8ccc011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cca7fc700 1 -- 172.21.15.158:0/4105843972 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00e040 con 0x7f8cd40cbd20 2023-10-16T10:03:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cca7fc700 1 -- 172.21.15.158:0/4105843972 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc00f040 con 0x7f8cd40cbd20 2023-10-16T10:03:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd40c51c0 con 0x7f8cd40cbd20 2023-10-16T10:03:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cca7fc700 1 -- 172.21.15.158:0/4105843972 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00fc40 con 0x7f8cd40cbd20 2023-10-16T10:03:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.355+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd40cc1b0 con 0x7f8cd40cbd20 2023-10-16T10:03:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.356+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd40c56e0 con 0x7f8cd40cbd20 2023-10-16T10:03:14.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.357+0000 7f8cca7fc700 1 -- 172.21.15.158:0/4105843972 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8ccc022460 con 0x7f8cd40cbd20 2023-10-16T10:03:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.357+0000 7f8cca7fc700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f8cb806b020 0x7f8cb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.357+0000 7f8cca7fc700 1 -- 172.21.15.158:0/4105843972 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f8ccc08a010 con 0x7f8cd40cbd20 2023-10-16T10:03:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.357+0000 7f8cd9b13700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f8cb806b020 0x7f8cb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.357+0000 7f8cd9b13700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f8cb806b020 0x7f8cb806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8cd40cc690 tx=0x7f8cc4009210 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:14.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.361+0000 7f8cca7fc700 1 -- 172.21.15.158:0/4105843972 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8cd40c56e0 con 0x7f8cd40cbd20 2023-10-16T10:03:14.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:14.636+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f8cd40cc430 con 0x7f8cd40cbd20 2023-10-16T10:03:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.171+0000 7f8cca7fc700 1 -- 172.21.15.158:0/4105843972 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v749) v1 ==== 148+0+0 (secure 0 0 0) 0x7f8ccc054b90 con 0x7f8cd40cbd20 2023-10-16T10:03:15.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.174+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f8cb806b020 msgr2=0x7f8cb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.174+0000 7f8cdbd77700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f8cb806b020 0x7f8cb806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8cd40cc690 tx=0x7f8cc4009210 comp rx=0 tx=0).stop 2023-10-16T10:03:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.174+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cd40cbd20 msgr2=0x7f8cd4150780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.174+0000 7f8cdbd77700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cd40cbd20 0x7f8cd4150780 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f8cd40ce9f0 tx=0x7f8ccc011ed0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 shutdown_connections 2023-10-16T10:03:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f8cb806b020 0x7f8cb806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cd40cbd20 0x7f8cd4150780 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cc0004850 0x7f8cd4160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 --2- 172.21.15.158:0/4105843972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd40c59c0 0x7f8cd40c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 >> 172.21.15.158:0/4105843972 conn(0x7f8cd406a8d0 msgr2=0x7f8cd40cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:15.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 shutdown_connections 2023-10-16T10:03:15.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.175+0000 7f8cdbd77700 1 -- 172.21.15.158:0/4105843972 wait complete. 2023-10-16T10:03:15.192 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-10-16T10:03:15.192 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T10:03:15.193 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:15.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b887ca700 1 Processor -- start 2023-10-16T10:03:15.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b887ca700 1 -- start start 2023-10-16T10:03:15.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b887ca700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9b800cf8a0 0x7f9b80161270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b887ca700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8015de20 con 0x7f9b800cf4c0 2023-10-16T10:03:15.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b887ca700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b80162180 con 0x7f9b8006bd10 2023-10-16T10:03:15.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b887ca700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b801618e0 con 0x7f9b800cf8a0 2023-10-16T10:03:15.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b86566700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9b800cf4c0 0x7f9b8006d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58056/0 (socket says 172.21.15.158:58056) 2023-10-16T10:03:15.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b86566700 1 -- 172.21.15.158:0/3562260626 learned_addr learned my addr 172.21.15.158:0/3562260626 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:15.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85d65700 1 --2- 172.21.15.158:0/3562260626 >> v2:172.21.15.172:3300/0 conn(0x7f9b800cf8a0 0x7f9b80161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453597716 0 0) 0x7f9b8015de20 con 0x7f9b800cf4c0 2023-10-16T10:03:15.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b6c0035b0 con 0x7f9b800cf4c0 2023-10-16T10:03:15.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952029860 0 0) 0x7f9b6c0035b0 con 0x7f9b800cf4c0 2023-10-16T10:03:15.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 >> v2:172.21.15.172:3300/0 conn(0x7f9b800cf8a0 msgr2=0x7f9b80161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:15.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 --2- 172.21.15.158:0/3562260626 >> v2:172.21.15.172:3300/0 conn(0x7f9b800cf8a0 0x7f9b80161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 >> v1:172.21.15.172:6789/0 conn(0x7f9b8006bd10 legacy=0x7f9b8015e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b80161bc0 con 0x7f9b800cf4c0 2023-10-16T10:03:15.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f9b7c001670 con 0x7f9b800cf4c0 2023-10-16T10:03:15.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 >> v1:172.21.15.158:6789/0 conn(0x7f9b800cf4c0 legacy=0x7f9b8006d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b6c003a00 0x7f9b6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b6c004850 0x7f9b6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c005210 0x7f9b6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801618e0 con 0x7f9b6c004850 2023-10-16T10:03:15.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c00c1b0 con 0x7f9b6c003a00 2023-10-16T10:03:15.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.320+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c00c370 con 0x7f9b6c005210 2023-10-16T10:03:15.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85d65700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b6c004850 0x7f9b6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b86566700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b6c003a00 0x7f9b6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b86d67700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c005210 0x7f9b6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85d65700 1 -- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c005210 msgr2=0x7f9b6c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:15.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85d65700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c005210 0x7f9b6c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85d65700 1 -- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b6c003a00 msgr2=0x7f9b6c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:15.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85d65700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b6c003a00 0x7f9b6c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85d65700 1 -- 172.21.15.158:0/3562260626 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c0035b0 con 0x7f9b6c004850 2023-10-16T10:03:15.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b86d67700 1 -- 172.21.15.158:0/3562260626 reap_dead start 2023-10-16T10:03:15.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85d65700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b6c004850 0x7f9b6c004cd0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009b60 tx=0x7f9b6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15172cabf48f4501 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:15.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7000e6c0 con 0x7f9b6c004850 2023-10-16T10:03:15.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7000e820 con 0x7f9b6c004850 2023-10-16T10:03:15.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.321+0000 7f9b85564700 1 -- 172.21.15.158:0/3562260626 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7000ebb0 con 0x7f9b6c004850 2023-10-16T10:03:15.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.322+0000 7f9b887ca700 1 -- 172.21.15.158:0/3562260626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b6c004850 msgr2=0x7f9b6c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.322+0000 7f9b887ca700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b6c004850 0x7f9b6c004cd0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009b60 tx=0x7f9b6c005880 comp rx=0 tx=0).stop 2023-10-16T10:03:15.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.322+0000 7f9b887ca700 1 -- 172.21.15.158:0/3562260626 shutdown_connections 2023-10-16T10:03:15.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.322+0000 7f9b887ca700 1 --2- 172.21.15.158:0/3562260626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b6c004850 0x7f9b6c004cd0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.322+0000 7f9b887ca700 1 -- 172.21.15.158:0/3562260626 >> 172.21.15.158:0/3562260626 conn(0x7f9b8006a8d0 msgr2=0x7f9b8006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:15.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.322+0000 7f9b887ca700 1 -- 172.21.15.158:0/3562260626 shutdown_connections 2023-10-16T10:03:15.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.322+0000 7f9b887ca700 1 -- 172.21.15.158:0/3562260626 wait complete. 2023-10-16T10:03:15.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.323+0000 7f9b887ca700 1 Processor -- start 2023-10-16T10:03:15.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.323+0000 7f9b887ca700 1 -- start start 2023-10-16T10:03:15.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c004850 0x7f9b80155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b800c7500 0x7f9b800c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b800b49a0 0x7f9b800b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801f32f0 con 0x7f9b800c7500 2023-10-16T10:03:15.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b800c7ec0 con 0x7f9b800b49a0 2023-10-16T10:03:15.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b800b56f0 con 0x7f9b6c004850 2023-10-16T10:03:15.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b800c7500 0x7f9b800c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b800c7500 0x7f9b800c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57894/0 (socket says 172.21.15.158:57894) 2023-10-16T10:03:15.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 -- 172.21.15.158:0/1636755143 learned_addr learned my addr 172.21.15.158:0/1636755143 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:15.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b86566700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c004850 0x7f9b80155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b86d67700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b800b49a0 0x7f9b800b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 -- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c004850 msgr2=0x7f9b80155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c004850 0x7f9b80155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 -- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b800b49a0 msgr2=0x7f9b800b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b800b49a0 0x7f9b800b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 -- 172.21.15.158:0/1636755143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800b59d0 con 0x7f9b800c7500 2023-10-16T10:03:15.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b85d65700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b800c7500 0x7f9b800c7980 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f9b700090b0 tx=0x7f9b700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:15.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b76ffd700 1 -- 172.21.15.158:0/1636755143 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7000e6c0 con 0x7f9b800c7500 2023-10-16T10:03:15.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b76ffd700 1 -- 172.21.15.158:0/1636755143 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b700038c0 con 0x7f9b800c7500 2023-10-16T10:03:15.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b76ffd700 1 -- 172.21.15.158:0/1636755143 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70003c70 con 0x7f9b800c7500 2023-10-16T10:03:15.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b800b5cb0 con 0x7f9b800c7500 2023-10-16T10:03:15.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.324+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b800c4f80 con 0x7f9b800c7500 2023-10-16T10:03:15.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.325+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b800c52f0 con 0x7f9b800c7500 2023-10-16T10:03:15.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.326+0000 7f9b76ffd700 1 -- 172.21.15.158:0/1636755143 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9b70013470 con 0x7f9b800c7500 2023-10-16T10:03:15.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.326+0000 7f9b76ffd700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9b6406b020 0x7f9b6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.326+0000 7f9b76ffd700 1 -- 172.21.15.158:0/1636755143 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f9b7008b1e0 con 0x7f9b800c7500 2023-10-16T10:03:15.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.326+0000 7f9b86566700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9b6406b020 0x7f9b6406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.326+0000 7f9b86566700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9b6406b020 0x7f9b6406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0017a0 tx=0x7f9b7c001690 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:15.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.330+0000 7f9b76ffd700 1 -- 172.21.15.158:0/1636755143 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b70056020 con 0x7f9b800c7500 2023-10-16T10:03:15.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.524+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b800c5110 con 0x7f9b800c7500 2023-10-16T10:03:15.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.525+0000 7f9b76ffd700 1 -- 172.21.15.158:0/1636755143 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+15048 (secure 0 0 0) 0x7f9b70059640 con 0x7f9b800c7500 2023-10-16T10:03:15.528 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:15.528 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":131,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:13.158354+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6837/3693875115":"2023-10-17T10:03:10.124753+0000","172.21.15.158:6835/3693875115":"2023-10-17T10:03:10.124753+0000","172.21.15.158:0/3944050656":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/59698897":"2023-10-17T10:03:09.141935+0000","172.21.15.158:6833/1079935681":"2023-10-17T10:03:09.141935+0000","172.21.15.158:6832/1079935681":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/1997480059":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/3901996147":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/3589635435":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/303455253":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/4078321936":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/1769168841":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/1113811588":"2023-10-17T10:03:09.141935+0000","172.21.15.158:0/3578502848":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2674552982":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/2360787591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6833/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:6832/1080332904":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/3568191591":"2023-10-17T10:02:23.842872+0000","172.21.15.158:0/1741897775":"2023-10-17T10:02:23.842872+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:15.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9b6406b020 msgr2=0x7f9b6406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9b6406b020 0x7f9b6406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0017a0 tx=0x7f9b7c001690 comp rx=0 tx=0).stop 2023-10-16T10:03:15.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b800c7500 msgr2=0x7f9b800c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b800c7500 0x7f9b800c7980 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f9b700090b0 tx=0x7f9b700047c0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 shutdown_connections 2023-10-16T10:03:15.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9b6406b020 0x7f9b6406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b800b49a0 0x7f9b800b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b6c004850 0x7f9b80155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 --2- 172.21.15.158:0/1636755143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b800c7500 0x7f9b800c7980 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 >> 172.21.15.158:0/1636755143 conn(0x7f9b8006a8d0 msgr2=0x7f9b8015c100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:15.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 shutdown_connections 2023-10-16T10:03:15.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.528+0000 7f9b887ca700 1 -- 172.21.15.158:0/1636755143 wait complete. 2023-10-16T10:03:15.542 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:03:15.542 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:03:15.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.664+0000 7ffa73e85700 1 Processor -- start 2023-10-16T10:03:15.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.664+0000 7ffa73e85700 1 -- start start 2023-10-16T10:03:15.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.664+0000 7ffa73e85700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffa6c0cbc30 0x7ffa6c0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa73e85700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffa6c15f0a0 0x7ffa6c0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa73e85700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cc010 0x7ffa6c15bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa73e85700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6c0d0320 con 0x7ffa6c0cbc30 2023-10-16T10:03:15.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa73e85700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6c15bab0 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa73e85700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c15f5d0 con 0x7ffa6c0cc010 2023-10-16T10:03:15.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffa6c0cbc30 0x7ffa6c0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffa6c0cbc30 0x7ffa6c0d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57900/0 (socket says 172.21.15.158:57900) 2023-10-16T10:03:15.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 -- 172.21.15.158:0/3641497485 learned_addr learned my addr 172.21.15.158:0/3641497485 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:15.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71420700 1 --2- 172.21.15.158:0/3641497485 >> v2:172.21.15.172:3300/0 conn(0x7ffa6c15f0a0 0x7ffa6c0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa72422700 1 --2- 172.21.15.158:0/3641497485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cc010 0x7ffa6c15bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 -- 172.21.15.158:0/3641497485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cc010 msgr2=0x7ffa6c15bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:15.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 --2- 172.21.15.158:0/3641497485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cc010 0x7ffa6c15bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 -- 172.21.15.158:0/3641497485 >> v2:172.21.15.172:3300/0 conn(0x7ffa6c15f0a0 msgr2=0x7ffa6c0cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:15.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 --2- 172.21.15.158:0/3641497485 >> v2:172.21.15.172:3300/0 conn(0x7ffa6c15f0a0 0x7ffa6c0cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 -- 172.21.15.158:0/3641497485 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c15f8b0 con 0x7ffa6c0cbc30 2023-10-16T10:03:15.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71420700 1 --2- 172.21.15.158:0/3641497485 >> v2:172.21.15.172:3300/0 conn(0x7ffa6c15f0a0 0x7ffa6c0cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:15.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa72422700 1 --2- 172.21.15.158:0/3641497485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cc010 0x7ffa6c15bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:15.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa71c21700 1 --2- 172.21.15.158:0/3641497485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c0cbc30 0x7ffa6c0d0b50 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c06f9c0 tx=0x7ffa5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d16cb20f2ad53c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:15.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa70c1f700 1 -- 172.21.15.158:0/3641497485 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00b870 con 0x7ffa6c0cbc30 2023-10-16T10:03:15.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa70c1f700 1 -- 172.21.15.158:0/3641497485 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c00b9d0 con 0x7ffa6c0cbc30 2023-10-16T10:03:15.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.665+0000 7ffa70c1f700 1 -- 172.21.15.158:0/3641497485 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00bd60 con 0x7ffa6c0cbc30 2023-10-16T10:03:15.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 -- 172.21.15.158:0/3641497485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c0cbc30 msgr2=0x7ffa6c0d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 --2- 172.21.15.158:0/3641497485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c0cbc30 0x7ffa6c0d0b50 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c06f9c0 tx=0x7ffa5c009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 -- 172.21.15.158:0/3641497485 shutdown_connections 2023-10-16T10:03:15.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 --2- 172.21.15.158:0/3641497485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cc010 0x7ffa6c15bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 --2- 172.21.15.158:0/3641497485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c0cbc30 0x7ffa6c0d0b50 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 --2- 172.21.15.158:0/3641497485 >> v2:172.21.15.172:3300/0 conn(0x7ffa6c15f0a0 0x7ffa6c0cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 -- 172.21.15.158:0/3641497485 >> 172.21.15.158:0/3641497485 conn(0x7ffa6c0cdfd0 msgr2=0x7ffa6c0ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:15.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 -- 172.21.15.158:0/3641497485 shutdown_connections 2023-10-16T10:03:15.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.666+0000 7ffa73e85700 1 -- 172.21.15.158:0/3641497485 wait complete. 2023-10-16T10:03:15.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.667+0000 7ffa73e85700 1 Processor -- start 2023-10-16T10:03:15.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.667+0000 7ffa73e85700 1 -- start start 2023-10-16T10:03:15.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa73e85700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cbc30 0x7ffa6c0cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa73e85700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c0cc010 0x7ffa6c0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa73e85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c15f0a0 0x7ffa6c0b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa73e85700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0b5890 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa73e85700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0b5000 con 0x7ffa6c0cc010 2023-10-16T10:03:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa73e85700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0b52d0 con 0x7ffa6c0cbc30 2023-10-16T10:03:15.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c15f0a0 0x7ffa6c0b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c15f0a0 0x7ffa6c0b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57910/0 (socket says 172.21.15.158:57910) 2023-10-16T10:03:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 -- 172.21.15.158:0/2488959782 learned_addr learned my addr 172.21.15.158:0/2488959782 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa71c21700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cbc30 0x7ffa6c0cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa71420700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c0cc010 0x7ffa6c0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 -- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cbc30 msgr2=0x7ffa6c0cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cbc30 0x7ffa6c0cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 -- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c0cc010 msgr2=0x7ffa6c0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c0cc010 0x7ffa6c0b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa72422700 1 -- 172.21.15.158:0/2488959782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c009550 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.668+0000 7ffa71420700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c0cc010 0x7ffa6c0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.669+0000 7ffa72422700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c15f0a0 0x7ffa6c0b6380 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ffa64007e10 tx=0x7ffa6400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.669+0000 7ffa627fc700 1 -- 172.21.15.158:0/2488959782 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6400de90 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.669+0000 7ffa627fc700 1 -- 172.21.15.158:0/2488959782 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6400f040 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.669+0000 7ffa627fc700 1 -- 172.21.15.158:0/2488959782 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64003c60 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.669+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c0b55b0 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.669+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c1529b0 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.670+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c152bd0 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.670+0000 7ffa627fc700 1 -- 172.21.15.158:0/2488959782 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffa64004030 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.670+0000 7ffa627fc700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7ffa5006b020 0x7ffa5006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.671+0000 7ffa627fc700 1 -- 172.21.15.158:0/2488959782 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7ffa64089c20 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.671+0000 7ffa71c21700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7ffa5006b020 0x7ffa5006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.671+0000 7ffa71c21700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7ffa5006b020 0x7ffa5006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c014040 tx=0x7ffa5c008030 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:15.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.675+0000 7ffa627fc700 1 -- 172.21.15.158:0/2488959782 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa64054830 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.865+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa6c04ec20 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.865+0000 7ffa627fc700 1 -- 172.21.15.158:0/2488959782 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa6401a0a0 con 0x7ffa6c15f0a0 2023-10-16T10:03:15.867 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:03:15.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7ffa5006b020 msgr2=0x7ffa5006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7ffa5006b020 0x7ffa5006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c014040 tx=0x7ffa5c008030 comp rx=0 tx=0).stop 2023-10-16T10:03:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c15f0a0 msgr2=0x7ffa6c0b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c15f0a0 0x7ffa6c0b6380 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ffa64007e10 tx=0x7ffa6400d3b0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 shutdown_connections 2023-10-16T10:03:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7ffa5006b020 0x7ffa5006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa6c15f0a0 0x7ffa6c0b6380 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa6c0cbc30 0x7ffa6c0cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 --2- 172.21.15.158:0/2488959782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c0cc010 0x7ffa6c0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 >> 172.21.15.158:0/2488959782 conn(0x7ffa6c0cdfd0 msgr2=0x7ffa6c0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 shutdown_connections 2023-10-16T10:03:15.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:15.868+0000 7ffa73e85700 1 -- 172.21.15.158:0/2488959782 wait complete. 2023-10-16T10:03:15.881 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T10:03:16.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.001+0000 7f79fa276700 1 Processor -- start 2023-10-16T10:03:16.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.001+0000 7f79fa276700 1 -- start start 2023-10-16T10:03:16.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f40cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:16.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f79f40c65b0 0x7f79f406f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:16.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f40cef60 con 0x7f79f40cbf40 2023-10-16T10:03:16.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f415e480 con 0x7f79f40c65b0 2023-10-16T10:03:16.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4066cb0 con 0x7f79f40ce640 2023-10-16T10:03:16.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f37fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f79f40c65b0 0x7f79f406f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:16.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f8813700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f79f40cbf40 0x7f79f40665a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58092/0 (socket says 172.21.15.158:58092) 2023-10-16T10:03:16.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f8813700 1 -- 172.21.15.158:0/3811674236 learned_addr learned my addr 172.21.15.158:0/3811674236 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:16.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f3fff700 1 --2- 172.21.15.158:0/3811674236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f40cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:16.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f37fe700 1 -- 172.21.15.158:0/3811674236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 msgr2=0x7f79f40cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:16.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f37fe700 1 --2- 172.21.15.158:0/3811674236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f40cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:16.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f37fe700 1 -- 172.21.15.158:0/3811674236 >> v1:172.21.15.158:6789/0 conn(0x7f79f40cbf40 legacy=0x7f79f40665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:16.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f37fe700 1 -- 172.21.15.158:0/3811674236 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f415d800 con 0x7f79f40c65b0 2023-10-16T10:03:16.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f2ffd700 1 -- 172.21.15.158:0/3811674236 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831893069 0 0) 0x7f79f40cef60 con 0x7f79f40cbf40 2023-10-16T10:03:16.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f37fe700 1 --2- 172.21.15.158:0/3811674236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f406f460 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009870 tx=0x7f79e4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89816a6a78df9ae0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:16.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f2ffd700 1 -- 172.21.15.158:0/3811674236 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4011830 con 0x7f79f40c65b0 2023-10-16T10:03:16.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f2ffd700 1 -- 172.21.15.158:0/3811674236 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e4011990 con 0x7f79f40c65b0 2023-10-16T10:03:16.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79f2ffd700 1 -- 172.21.15.158:0/3811674236 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4011d20 con 0x7f79f40c65b0 2023-10-16T10:03:16.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- 172.21.15.158:0/3811674236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 msgr2=0x7f79f406f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:16.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 --2- 172.21.15.158:0/3811674236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f406f460 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009870 tx=0x7f79e4009b80 comp rx=0 tx=0).stop 2023-10-16T10:03:16.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- 172.21.15.158:0/3811674236 shutdown_connections 2023-10-16T10:03:16.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 --2- 172.21.15.158:0/3811674236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f40cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:16.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 --2- 172.21.15.158:0/3811674236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f406f460 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:16.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- 172.21.15.158:0/3811674236 >> 172.21.15.158:0/3811674236 conn(0x7f79f40cc4b0 msgr2=0x7f79f40cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:16.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- 172.21.15.158:0/3811674236 shutdown_connections 2023-10-16T10:03:16.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.002+0000 7f79fa276700 1 -- 172.21.15.158:0/3811674236 wait complete. 2023-10-16T10:03:16.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.003+0000 7f79fa276700 1 Processor -- start 2023-10-16T10:03:16.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79fa276700 1 -- start start 2023-10-16T10:03:16.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79fa276700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f4159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:16.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79fa276700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f40cbf40 0x7f79f4142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:16.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79fa276700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f4142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:16.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79fa276700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4155de0 con 0x7f79f40c65b0 2023-10-16T10:03:16.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79fa276700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4159cb0 con 0x7f79f40cbf40 2023-10-16T10:03:16.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79fa276700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f41586f0 con 0x7f79f40ce640 2023-10-16T10:03:16.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79f3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:16.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79f3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f4159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57918/0 (socket says 172.21.15.158:57918) 2023-10-16T10:03:16.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.004+0000 7f79f3fff700 1 -- 172.21.15.158:0/1368703371 learned_addr learned my addr 172.21.15.158:0/1368703371 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:16.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f37fe700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f40cbf40 0x7f79f4142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:16.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f8813700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f4142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:16.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f3fff700 1 -- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 msgr2=0x7f79f4142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:16.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f3fff700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f4142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:16.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f3fff700 1 -- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f40cbf40 msgr2=0x7f79f4142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:16.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f3fff700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f40cbf40 0x7f79f4142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:16.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f3fff700 1 -- 172.21.15.158:0/1368703371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40094a0 con 0x7f79f40c65b0 2023-10-16T10:03:16.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f3fff700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f4159770 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f79e800e9b0 tx=0x7f79e800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:16.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f0ff9700 1 -- 172.21.15.158:0/1368703371 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e8009710 con 0x7f79f40c65b0 2023-10-16T10:03:16.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f0ff9700 1 -- 172.21.15.158:0/1368703371 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e800c850 con 0x7f79f40c65b0 2023-10-16T10:03:16.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f41589d0 con 0x7f79f40c65b0 2023-10-16T10:03:16.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79f0ff9700 1 -- 172.21.15.158:0/1368703371 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e800cbc0 con 0x7f79f40c65b0 2023-10-16T10:03:16.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.005+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f4159270 con 0x7f79f40c65b0 2023-10-16T10:03:16.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.006+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f41595e0 con 0x7f79f40c65b0 2023-10-16T10:03:16.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.006+0000 7f79f0ff9700 1 -- 172.21.15.158:0/1368703371 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79e800cd20 con 0x7f79f40c65b0 2023-10-16T10:03:16.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.007+0000 7f79f0ff9700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f79e006b020 0x7f79e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:16.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.007+0000 7f79f0ff9700 1 -- 172.21.15.158:0/1368703371 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f79e8089bc0 con 0x7f79f40c65b0 2023-10-16T10:03:16.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.007+0000 7f79f37fe700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f79e006b020 0x7f79e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:16.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.007+0000 7f79f37fe700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f79e006b020 0x7f79e006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f79f4142d50 tx=0x7f79e401c040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:16.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.012+0000 7f79f0ff9700 1 -- 172.21.15.158:0/1368703371 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79e8054850 con 0x7f79f40c65b0 2023-10-16T10:03:16.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:16.288+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f79f40681b0 con 0x7f79f40c65b0 2023-10-16T10:03:17.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.177+0000 7f79f0ff9700 1 -- 172.21.15.158:0/1368703371 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v751) v1 ==== 163+0+0 (secure 0 0 0) 0x7f79e80210e0 con 0x7f79f40c65b0 2023-10-16T10:03:17.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.180+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f79e006b020 msgr2=0x7f79e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.180+0000 7f79fa276700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f79e006b020 0x7f79e006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f79f4142d50 tx=0x7f79e401c040 comp rx=0 tx=0).stop 2023-10-16T10:03:17.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.180+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 msgr2=0x7f79f4159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.180+0000 7f79fa276700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f4159770 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f79e800e9b0 tx=0x7f79e800ed70 comp rx=0 tx=0).stop 2023-10-16T10:03:17.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 shutdown_connections 2023-10-16T10:03:17.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f79e006b020 0x7f79e006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79f40ce640 0x7f79f4142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79f40c65b0 0x7f79f4159770 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 --2- 172.21.15.158:0/1368703371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79f40cbf40 0x7f79f4142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 >> 172.21.15.158:0/1368703371 conn(0x7f79f40cc4b0 msgr2=0x7f79f40af970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:17.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 shutdown_connections 2023-10-16T10:03:17.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.181+0000 7f79fa276700 1 -- 172.21.15.158:0/1368703371 wait complete. 2023-10-16T10:03:17.197 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T10:03:17.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.325+0000 7f08c6b0d700 1 Processor -- start 2023-10-16T10:03:17.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.325+0000 7f08c6b0d700 1 -- start start 2023-10-16T10:03:17.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c6b0d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08c015b590 0x7f08c00ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c6b0d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f08c015f1d0 0x7f08c00b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c6b0d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c00cf4c0 con 0x7f08c00cbaf0 2023-10-16T10:03:17.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c6b0d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c015f970 con 0x7f08c015b590 2023-10-16T10:03:17.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c6b0d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c015fc50 con 0x7f08c015f1d0 2023-10-16T10:03:17.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08c015b590 0x7f08c00ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08c015b590 0x7f08c00ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57926/0 (socket says 172.21.15.158:57926) 2023-10-16T10:03:17.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 -- 172.21.15.158:0/3712270157 learned_addr learned my addr 172.21.15.158:0/3712270157 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:17.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08bffff700 1 --2- 172.21.15.158:0/3712270157 >> v2:172.21.15.172:3300/0 conn(0x7f08c015f1d0 0x7f08c00b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 -- 172.21.15.158:0/3712270157 >> v2:172.21.15.172:3300/0 conn(0x7f08c015f1d0 msgr2=0x7f08c00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:17.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 --2- 172.21.15.158:0/3712270157 >> v2:172.21.15.172:3300/0 conn(0x7f08c015f1d0 0x7f08c00b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 -- 172.21.15.158:0/3712270157 >> v1:172.21.15.172:6789/0 conn(0x7f08c00cbaf0 legacy=0x7f08c00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:17.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 -- 172.21.15.158:0/3712270157 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c015ff30 con 0x7f08c015b590 2023-10-16T10:03:17.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08c48a9700 1 --2- 172.21.15.158:0/3712270157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015b590 0x7f08c00ce510 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f08c00d1360 tx=0x7f08b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdf6f462c87b0a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:17.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08bf7fe700 1 -- 172.21.15.158:0/3712270157 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4004230 con 0x7f08c015b590 2023-10-16T10:03:17.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08bf7fe700 1 -- 172.21.15.158:0/3712270157 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b4004390 con 0x7f08c015b590 2023-10-16T10:03:17.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.326+0000 7f08bf7fe700 1 -- 172.21.15.158:0/3712270157 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b40036a0 con 0x7f08c015b590 2023-10-16T10:03:17.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 -- 172.21.15.158:0/3712270157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015b590 msgr2=0x7f08c00ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/3712270157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015b590 0x7f08c00ce510 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f08c00d1360 tx=0x7f08b4009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 -- 172.21.15.158:0/3712270157 shutdown_connections 2023-10-16T10:03:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/3712270157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015b590 0x7f08c00ce510 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/3712270157 >> v2:172.21.15.172:3300/0 conn(0x7f08c015f1d0 0x7f08c00b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 -- 172.21.15.158:0/3712270157 >> 172.21.15.158:0/3712270157 conn(0x7f08c006a8d0 msgr2=0x7f08c006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 -- 172.21.15.158:0/3712270157 shutdown_connections 2023-10-16T10:03:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.327+0000 7f08c6b0d700 1 -- 172.21.15.158:0/3712270157 wait complete. 2023-10-16T10:03:17.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.328+0000 7f08c6b0d700 1 Processor -- start 2023-10-16T10:03:17.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.328+0000 7f08c6b0d700 1 -- start start 2023-10-16T10:03:17.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c6b0d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c00cbaf0 0x7f08c01f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c6b0d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015f1d0 0x7f08c00b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c6b0d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c015cc70 0x7f08c014d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c6b0d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c01f5530 con 0x7f08c015f1d0 2023-10-16T10:03:17.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c6b0d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c015d1e0 con 0x7f08c015cc70 2023-10-16T10:03:17.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c6b0d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c015d4c0 con 0x7f08c00cbaf0 2023-10-16T10:03:17.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015f1d0 0x7f08c00b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015f1d0 0x7f08c00b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57934/0 (socket says 172.21.15.158:57934) 2023-10-16T10:03:17.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08bffff700 1 -- 172.21.15.158:0/1177366303 learned_addr learned my addr 172.21.15.158:0/1177366303 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:17.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c48a9700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c00cbaf0 0x7f08c01f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.329+0000 7f08c50aa700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c015cc70 0x7f08c014d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bffff700 1 -- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c00cbaf0 msgr2=0x7f08c01f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bffff700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c00cbaf0 0x7f08c01f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bffff700 1 -- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c015cc70 msgr2=0x7f08c014d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bffff700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c015cc70 0x7f08c014d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bffff700 1 -- 172.21.15.158:0/1177366303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4009550 con 0x7f08c015f1d0 2023-10-16T10:03:17.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08c48a9700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c00cbaf0 0x7f08c01f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:17.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bffff700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015f1d0 0x7f08c00b4de0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d800 tx=0x7f08ac00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:17.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bd7fa700 1 -- 172.21.15.158:0/1177366303 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0125f0 con 0x7f08c015f1d0 2023-10-16T10:03:17.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bd7fa700 1 -- 172.21.15.158:0/1177366303 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac012750 con 0x7f08c015f1d0 2023-10-16T10:03:17.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08bd7fa700 1 -- 172.21.15.158:0/1177366303 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac012b00 con 0x7f08c015f1d0 2023-10-16T10:03:17.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c015d7a0 con 0x7f08c015f1d0 2023-10-16T10:03:17.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.330+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c00b5320 con 0x7f08c015f1d0 2023-10-16T10:03:17.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.331+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c015dcc0 con 0x7f08c015f1d0 2023-10-16T10:03:17.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.331+0000 7f08bd7fa700 1 -- 172.21.15.158:0/1177366303 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08ac012c60 con 0x7f08c015f1d0 2023-10-16T10:03:17.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.332+0000 7f08bd7fa700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f08a406b020 0x7f08a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.332+0000 7f08bd7fa700 1 -- 172.21.15.158:0/1177366303 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f08ac089d30 con 0x7f08c015f1d0 2023-10-16T10:03:17.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.332+0000 7f08c48a9700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f08a406b020 0x7f08a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.332+0000 7f08c48a9700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f08a406b020 0x7f08a406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08c00cee80 tx=0x7f08b4008030 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:17.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.336+0000 7f08bd7fa700 1 -- 172.21.15.158:0/1177366303 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08c015dcc0 con 0x7f08c015f1d0 2023-10-16T10:03:17.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.529+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f08c015dcc0 con 0x7f08c015f1d0 2023-10-16T10:03:17.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.532+0000 7f08bd7fa700 1 -- 172.21.15.158:0/1177366303 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f08c015dcc0 con 0x7f08c015f1d0 2023-10-16T10:03:17.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f08a406b020 msgr2=0x7f08a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f08a406b020 0x7f08a406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08c00cee80 tx=0x7f08b4008030 comp rx=0 tx=0).stop 2023-10-16T10:03:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015f1d0 msgr2=0x7f08c00b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015f1d0 0x7f08c00b4de0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d800 tx=0x7f08ac00db10 comp rx=0 tx=0).stop 2023-10-16T10:03:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 shutdown_connections 2023-10-16T10:03:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f08a406b020 0x7f08a406d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c015cc70 0x7f08c014d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c00cbaf0 0x7f08c01f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 --2- 172.21.15.158:0/1177366303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08c015f1d0 0x7f08c00b4de0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 >> 172.21.15.158:0/1177366303 conn(0x7f08c006a8d0 msgr2=0x7f08c00cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:17.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 shutdown_connections 2023-10-16T10:03:17.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.535+0000 7f08c6b0d700 1 -- 172.21.15.158:0/1177366303 wait complete. 2023-10-16T10:03:17.548 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:03:17.549 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:17.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:17.640 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:03:17.640 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:17.641 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:17.659 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:03:17.659 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:17.660 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:17.679 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:17.680 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:17.680 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:17.716 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:03:17.716 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:17.717 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:17.740 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:17.740 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T10:03:17.741 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:17.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.875+0000 7f42b9afc700 1 Processor -- start 2023-10-16T10:03:17.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.875+0000 7f42b9afc700 1 -- start start 2023-10-16T10:03:17.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.875+0000 7f42b9afc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40ce640 0x7f42b40cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.875+0000 7f42b9afc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42b40c65b0 0x7f42b406f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.875+0000 7f42b9afc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b40cef60 con 0x7f42b40cd600 2023-10-16T10:03:17.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.875+0000 7f42b9afc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b415e480 con 0x7f42b40c65b0 2023-10-16T10:03:17.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.875+0000 7f42b9afc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b415d800 con 0x7f42b40ce640 2023-10-16T10:03:17.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42b40c65b0 0x7f42b406f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42b40c65b0 0x7f42b406f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57942/0 (socket says 172.21.15.158:57942) 2023-10-16T10:03:17.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 -- 172.21.15.158:0/2111946884 learned_addr learned my addr 172.21.15.158:0/2111946884 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:17.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b37fe700 1 --2- 172.21.15.158:0/2111946884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40ce640 0x7f42b40cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 -- 172.21.15.158:0/2111946884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40ce640 msgr2=0x7f42b40cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:17.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 --2- 172.21.15.158:0/2111946884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40ce640 0x7f42b40cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 -- 172.21.15.158:0/2111946884 >> v1:172.21.15.172:6789/0 conn(0x7f42b40cd600 legacy=0x7f42b40cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:17.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 -- 172.21.15.158:0/2111946884 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b415dae0 con 0x7f42b40c65b0 2023-10-16T10:03:17.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b2ffd700 1 --2- 172.21.15.158:0/2111946884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40c65b0 0x7f42b406f460 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f429c0098f0 tx=0x7f429c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d73687711d2bf80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b27fc700 1 -- 172.21.15.158:0/2111946884 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c004230 con 0x7f42b40c65b0 2023-10-16T10:03:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b27fc700 1 -- 172.21.15.158:0/2111946884 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c004390 con 0x7f42b40c65b0 2023-10-16T10:03:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.876+0000 7f42b27fc700 1 -- 172.21.15.158:0/2111946884 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0036a0 con 0x7f42b40c65b0 2023-10-16T10:03:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 -- 172.21.15.158:0/2111946884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40c65b0 msgr2=0x7f42b406f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 --2- 172.21.15.158:0/2111946884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40c65b0 0x7f42b406f460 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f429c0098f0 tx=0x7f429c009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 -- 172.21.15.158:0/2111946884 shutdown_connections 2023-10-16T10:03:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 --2- 172.21.15.158:0/2111946884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40ce640 0x7f42b40cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 --2- 172.21.15.158:0/2111946884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40c65b0 0x7f42b406f460 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 -- 172.21.15.158:0/2111946884 >> 172.21.15.158:0/2111946884 conn(0x7f42b40cc4b0 msgr2=0x7f42b40cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 -- 172.21.15.158:0/2111946884 shutdown_connections 2023-10-16T10:03:17.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.877+0000 7f42b9afc700 1 -- 172.21.15.158:0/2111946884 wait complete. 2023-10-16T10:03:17.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.878+0000 7f42b9afc700 1 Processor -- start 2023-10-16T10:03:17.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.878+0000 7f42b9afc700 1 -- start start 2023-10-16T10:03:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b9afc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40c65b0 0x7f42b4164100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b9afc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40cd600 0x7f42b4165e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b9afc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ce640 0x7f42b41427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b9afc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4162e50 con 0x7f42b40cd600 2023-10-16T10:03:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b9afc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4163060 con 0x7f42b40ce640 2023-10-16T10:03:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b9afc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4163340 con 0x7f42b40c65b0 2023-10-16T10:03:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40cd600 0x7f42b4165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40cd600 0x7f42b4165e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57948/0 (socket says 172.21.15.158:57948) 2023-10-16T10:03:17.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 -- 172.21.15.158:0/1023921281 learned_addr learned my addr 172.21.15.158:0/1023921281 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b37fe700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40c65b0 0x7f42b4164100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b3fff700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ce640 0x7f42b41427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 -- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40c65b0 msgr2=0x7f42b4164100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40c65b0 0x7f42b4164100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 -- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ce640 msgr2=0x7f42b41427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:17.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ce640 0x7f42b41427d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.879+0000 7f42b2ffd700 1 -- 172.21.15.158:0/1023921281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c009550 con 0x7f42b40cd600 2023-10-16T10:03:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.880+0000 7f42b2ffd700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40cd600 0x7f42b4165e70 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f429c009130 tx=0x7f429c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.880+0000 7f42abfff700 1 -- 172.21.15.158:0/1023921281 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c00b660 con 0x7f42b40cd600 2023-10-16T10:03:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.880+0000 7f42abfff700 1 -- 172.21.15.158:0/1023921281 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c00b7c0 con 0x7f42b40cd600 2023-10-16T10:03:17.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.880+0000 7f42abfff700 1 -- 172.21.15.158:0/1023921281 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c00bb70 con 0x7f42b40cd600 2023-10-16T10:03:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.880+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b4163620 con 0x7f42b40cd600 2023-10-16T10:03:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.880+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b4163be0 con 0x7f42b40cd600 2023-10-16T10:03:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.881+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b40681b0 con 0x7f42b40cd600 2023-10-16T10:03:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.881+0000 7f42abfff700 1 -- 172.21.15.158:0/1023921281 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f429c00bcd0 con 0x7f42b40cd600 2023-10-16T10:03:17.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.882+0000 7f42abfff700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f429406b020 0x7f429406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.882+0000 7f42b37fe700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f429406b020 0x7f429406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.882+0000 7f42abfff700 1 -- 172.21.15.158:0/1023921281 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f429c08b2b0 con 0x7f42b40cd600 2023-10-16T10:03:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.882+0000 7f42b37fe700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f429406b020 0x7f429406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42a400ab60 tx=0x7f42a4008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:17.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:17.886+0000 7f42abfff700 1 -- 172.21.15.158:0/1023921281 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f429c055fd0 con 0x7f42b40cd600 2023-10-16T10:03:18.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.123+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42b40c5300 con 0x7f42b40cd600 2023-10-16T10:03:18.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.124+0000 7f42abfff700 1 -- 172.21.15.158:0/1023921281 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f429c029e90 con 0x7f42b40cd600 2023-10-16T10:03:18.125 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:18.125 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":62,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/94308149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":94308149},{"type":"v1","addr":"172.21.15.172:6834","nonce":94308149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/4112063716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4112063716},{"type":"v1","addr":"172.21.15.172:6835","nonce":4112063716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8274,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/658160836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":658160836},{"type":"v1","addr":"172.21.15.158:6836","nonce":658160836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112338889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112338889},{"type":"v1","addr":"172.21.15.158:6837","nonce":112338889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[]} 2023-10-16T10:03:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f429406b020 msgr2=0x7f429406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:18.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f429406b020 0x7f429406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42a400ab60 tx=0x7f42a4008040 comp rx=0 tx=0).stop 2023-10-16T10:03:18.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40cd600 msgr2=0x7f42b4165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:18.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40cd600 0x7f42b4165e70 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f429c009130 tx=0x7f429c004750 comp rx=0 tx=0).stop 2023-10-16T10:03:18.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 shutdown_connections 2023-10-16T10:03:18.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f429406b020 0x7f429406d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ce640 0x7f42b41427d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40c65b0 0x7f42b4164100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 --2- 172.21.15.158:0/1023921281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b40cd600 0x7f42b4165e70 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 >> 172.21.15.158:0/1023921281 conn(0x7f42b40cc4b0 msgr2=0x7f42b40cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:18.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 shutdown_connections 2023-10-16T10:03:18.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.126+0000 7f42b9afc700 1 -- 172.21.15.158:0/1023921281 wait complete. 2023-10-16T10:03:18.130 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 62 2023-10-16T10:03:18.140 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T10:03:18.140 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T10:03:18.140 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T10:03:18.140 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:18.141 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T10:03:18.141 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T10:03:18.141 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T10:03:18.141 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:18.142 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T10:03:18.142 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T10:03:18.142 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T10:03:18.143 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:18.143 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T10:03:18.143 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T10:03:18.143 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T10:03:18.144 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:18.149 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T10:03:18.150 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T10:03:18.151 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T10:03:18.151 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T10:03:18.153 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T10:03:18.154 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T10:03:18.155 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T10:03:18.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T10:03:18.192 INFO:tasks.ceph.mds.a:Started 2023-10-16T10:03:18.193 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:18.194 INFO:tasks.ceph.mds.c:Started 2023-10-16T10:03:18.194 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:18.196 INFO:tasks.ceph.mds.b:Started 2023-10-16T10:03:18.197 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:18.197 INFO:tasks.ceph.mds.d:Started 2023-10-16T10:03:18.198 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:18.198 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T10:03:18.272 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T10:03:18.272 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T10:03:18.282 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T10:03:18.283 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T10:03:18.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.326+0000 7fc688e9e700 1 Processor -- start 2023-10-16T10:03:18.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.326+0000 7fc688e9e700 1 -- start start 2023-10-16T10:03:18.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc688e9e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc6840cbb00 0x7fc68415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:18.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc688e9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:18.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc688e9e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68415d0c0 con 0x7fc6840d0bd0 2023-10-16T10:03:18.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc688e9e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6840cf320 con 0x7fc6840cbb00 2023-10-16T10:03:18.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc688e9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6840cc420 con 0x7fc68415cc60 2023-10-16T10:03:18.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc68259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc6840cbb00 0x7fc68415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:18.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc68259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc6840cbb00 0x7fc68415baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57978/0 (socket says 172.21.15.158:57978) 2023-10-16T10:03:18.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc68259c700 1 -- 172.21.15.158:0/2926820151 learned_addr learned my addr 172.21.15.158:0/2926820151 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:18.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc681d9b700 1 --2- 172.21.15.158:0/2926820151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:18.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc68259c700 1 -- 172.21.15.158:0/2926820151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 msgr2=0x7fc6840cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:18.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc68259c700 1 --2- 172.21.15.158:0/2926820151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc68259c700 1 -- 172.21.15.158:0/2926820151 >> v1:172.21.15.172:6789/0 conn(0x7fc6840d0bd0 legacy=0x7fc6840cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:18.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc68259c700 1 -- 172.21.15.158:0/2926820151 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68415c080 con 0x7fc6840cbb00 2023-10-16T10:03:18.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.327+0000 7fc681d9b700 1 --2- 172.21.15.158:0/2926820151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:18.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc68259c700 1 --2- 172.21.15.158:0/2926820151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68415baa0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc67c0098f0 tx=0x7fc67c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f7937da80be0c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:18.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc68159a700 1 -- 172.21.15.158:0/2926820151 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c00b8a0 con 0x7fc6840cbb00 2023-10-16T10:03:18.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc68159a700 1 -- 172.21.15.158:0/2926820151 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67c00ba00 con 0x7fc6840cbb00 2023-10-16T10:03:18.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc68159a700 1 -- 172.21.15.158:0/2926820151 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c00bd90 con 0x7fc6840cbb00 2023-10-16T10:03:18.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 -- 172.21.15.158:0/2926820151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 msgr2=0x7fc68415baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:18.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 --2- 172.21.15.158:0/2926820151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68415baa0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc67c0098f0 tx=0x7fc67c009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:18.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 -- 172.21.15.158:0/2926820151 shutdown_connections 2023-10-16T10:03:18.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 --2- 172.21.15.158:0/2926820151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68415baa0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 --2- 172.21.15.158:0/2926820151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 -- 172.21.15.158:0/2926820151 >> 172.21.15.158:0/2926820151 conn(0x7fc6840cdf30 msgr2=0x7fc6840ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:18.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 -- 172.21.15.158:0/2926820151 shutdown_connections 2023-10-16T10:03:18.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.328+0000 7fc688e9e700 1 -- 172.21.15.158:0/2926820151 wait complete. 2023-10-16T10:03:18.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.330+0000 7fc688e9e700 1 Processor -- start 2023-10-16T10:03:18.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.330+0000 7fc688e9e700 1 -- start start 2023-10-16T10:03:18.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68414ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:18.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6840d0bd0 0x7fc6840c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:18.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:18.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6840b0850 con 0x7fc6840cbb00 2023-10-16T10:03:18.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68414f280 con 0x7fc6840d0bd0 2023-10-16T10:03:18.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6840cacc0 con 0x7fc68415cc60 2023-10-16T10:03:18.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68414ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:18.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68414ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57984/0 (socket says 172.21.15.158:57984) 2023-10-16T10:03:18.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 -- 172.21.15.158:0/1914722749 learned_addr learned my addr 172.21.15.158:0/1914722749 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:18.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc681d9b700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6840d0bd0 0x7fc6840c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:18.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc682d9d700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:18.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 -- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 msgr2=0x7fc6840c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:18.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840c7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 -- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6840d0bd0 msgr2=0x7fc6840c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:18.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6840d0bd0 0x7fc6840c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:18.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 -- 172.21.15.158:0/1914722749 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67c009550 con 0x7fc6840cbb00 2023-10-16T10:03:18.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc68259c700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68414ed40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc67c000c00 tx=0x7fc67c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:18.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc672ffd700 1 -- 172.21.15.158:0/1914722749 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c004300 con 0x7fc6840cbb00 2023-10-16T10:03:18.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc672ffd700 1 -- 172.21.15.158:0/1914722749 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67c01f070 con 0x7fc6840cbb00 2023-10-16T10:03:18.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6840cafa0 con 0x7fc6840cbb00 2023-10-16T10:03:18.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc672ffd700 1 -- 172.21.15.158:0/1914722749 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c0036a0 con 0x7fc6840cbb00 2023-10-16T10:03:18.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.331+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6840cb560 con 0x7fc6840cbb00 2023-10-16T10:03:18.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.332+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6840b09c0 con 0x7fc6840cbb00 2023-10-16T10:03:18.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.333+0000 7fc672ffd700 1 -- 172.21.15.158:0/1914722749 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc67c021030 con 0x7fc6840cbb00 2023-10-16T10:03:18.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.333+0000 7fc672ffd700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fc66406b020 0x7fc66406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:18.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.333+0000 7fc672ffd700 1 -- 172.21.15.158:0/1914722749 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fc67c08ade0 con 0x7fc6840cbb00 2023-10-16T10:03:18.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.333+0000 7fc681d9b700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fc66406b020 0x7fc66406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:18.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.334+0000 7fc681d9b700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fc66406b020 0x7fc66406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc68415b980 tx=0x7fc678008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:18.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.338+0000 7fc672ffd700 1 -- 172.21.15.158:0/1914722749 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc67c055a80 con 0x7fc6840cbb00 2023-10-16T10:03:18.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:18.543+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc6840681b0 con 0x7fc6840cbb00 2023-10-16T10:03:19.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.638+0000 7fc672ffd700 1 -- 172.21.15.158:0/1914722749 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc67c0590a0 con 0x7fc6840cbb00 2023-10-16T10:03:19.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fc66406b020 msgr2=0x7fc66406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:19.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fc66406b020 0x7fc66406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc68415b980 tx=0x7fc678008040 comp rx=0 tx=0).stop 2023-10-16T10:03:19.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 msgr2=0x7fc68414ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:19.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68414ed40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc67c000c00 tx=0x7fc67c004750 comp rx=0 tx=0).stop 2023-10-16T10:03:19.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 shutdown_connections 2023-10-16T10:03:19.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fc66406b020 0x7fc66406d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc68415cc60 0x7fc6840c7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6840cbb00 0x7fc68414ed40 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 --2- 172.21.15.158:0/1914722749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6840d0bd0 0x7fc6840c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 >> 172.21.15.158:0/1914722749 conn(0x7fc6840cdf30 msgr2=0x7fc68415d250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:19.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 shutdown_connections 2023-10-16T10:03:19.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.642+0000 7fc688e9e700 1 -- 172.21.15.158:0/1914722749 wait complete. 2023-10-16T10:03:19.647 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T10:03:19.658 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T10:03:19.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.783+0000 7f7d78e10700 1 Processor -- start 2023-10-16T10:03:19.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.783+0000 7f7d78e10700 1 -- start start 2023-10-16T10:03:19.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d78e10700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:19.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d78e10700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d7415dfb0 0x7f7d740cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:19.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d78e10700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7406e770 con 0x7f7d7406f340 2023-10-16T10:03:19.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d78e10700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d740cf270 con 0x7f7d7415dfb0 2023-10-16T10:03:19.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d78e10700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7415e4e0 con 0x7f7d740cc5d0 2023-10-16T10:03:19.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d71d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d7415dfb0 0x7f7d740cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:19.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d71d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d7415dfb0 0x7f7d740cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57996/0 (socket says 172.21.15.158:57996) 2023-10-16T10:03:19.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d72d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7d7406f340 0x7f7d74071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58176/0 (socket says 172.21.15.158:58176) 2023-10-16T10:03:19.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d71d9b700 1 -- 172.21.15.158:0/3804693666 learned_addr learned my addr 172.21.15.158:0/3804693666 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:19.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d7259c700 1 --2- 172.21.15.158:0/3804693666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:19.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d71d9b700 1 -- 172.21.15.158:0/3804693666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 msgr2=0x7f7d7406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:19.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d71d9b700 1 --2- 172.21.15.158:0/3804693666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d71d9b700 1 -- 172.21.15.158:0/3804693666 >> v1:172.21.15.158:6789/0 conn(0x7f7d7406f340 legacy=0x7f7d74071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:19.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d71d9b700 1 -- 172.21.15.158:0/3804693666 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7415e7c0 con 0x7f7d7415dfb0 2023-10-16T10:03:19.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.784+0000 7f7d7159a700 1 -- 172.21.15.158:0/3804693666 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680668165 0 0) 0x7f7d6c001490 con 0x7f7d7406f340 2023-10-16T10:03:19.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d71d9b700 1 --2- 172.21.15.158:0/3804693666 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d740cb6f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7d640098f0 tx=0x7f7d64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4f3fe3c89a90f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:19.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d7159a700 1 -- 172.21.15.158:0/3804693666 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400b8a0 con 0x7f7d7415dfb0 2023-10-16T10:03:19.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d7159a700 1 -- 172.21.15.158:0/3804693666 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6400ba00 con 0x7f7d7415dfb0 2023-10-16T10:03:19.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d7159a700 1 -- 172.21.15.158:0/3804693666 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400bd90 con 0x7f7d7415dfb0 2023-10-16T10:03:19.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 -- 172.21.15.158:0/3804693666 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 msgr2=0x7f7d740cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:19.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 --2- 172.21.15.158:0/3804693666 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d740cb6f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7d640098f0 tx=0x7f7d64009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:19.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 -- 172.21.15.158:0/3804693666 shutdown_connections 2023-10-16T10:03:19.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 --2- 172.21.15.158:0/3804693666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 --2- 172.21.15.158:0/3804693666 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d740cb6f0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 -- 172.21.15.158:0/3804693666 >> 172.21.15.158:0/3804693666 conn(0x7f7d740d02b0 msgr2=0x7f7d740d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:19.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 -- 172.21.15.158:0/3804693666 shutdown_connections 2023-10-16T10:03:19.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.785+0000 7f7d78e10700 1 -- 172.21.15.158:0/3804693666 wait complete. 2023-10-16T10:03:19.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.787+0000 7f7d78e10700 1 Processor -- start 2023-10-16T10:03:19.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.787+0000 7f7d78e10700 1 -- start start 2023-10-16T10:03:19.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.787+0000 7f7d78e10700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d7406f340 0x7f7d740b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:19.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.787+0000 7f7d78e10700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7414d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:19.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d78e10700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d7414f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:19.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d78e10700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7414def0 con 0x7f7d7415dfb0 2023-10-16T10:03:19.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d78e10700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74167170 con 0x7f7d7406f340 2023-10-16T10:03:19.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d78e10700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740b11a0 con 0x7f7d740cc5d0 2023-10-16T10:03:19.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d7414f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:19.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d7414f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58006/0 (socket says 172.21.15.158:58006) 2023-10-16T10:03:19.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 -- 172.21.15.158:0/2853814317 learned_addr learned my addr 172.21.15.158:0/2853814317 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:19.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d71d9b700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7414d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:19.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d7259c700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d7406f340 0x7f7d740b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:19.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 -- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 msgr2=0x7f7d7414d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:19.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7414d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 -- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d7406f340 msgr2=0x7f7d740b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:19.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d7406f340 0x7f7d740b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:19.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 -- 172.21.15.158:0/2853814317 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64009550 con 0x7f7d7415dfb0 2023-10-16T10:03:19.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d72d9d700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d7414f8c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00cb60 tx=0x7f7d6c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:19.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d62ffd700 1 -- 172.21.15.158:0/2853814317 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c00ab50 con 0x7f7d7415dfb0 2023-10-16T10:03:19.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d62ffd700 1 -- 172.21.15.158:0/2853814317 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c00acb0 con 0x7f7d7415dfb0 2023-10-16T10:03:19.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d740b1480 con 0x7f7d7415dfb0 2023-10-16T10:03:19.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d62ffd700 1 -- 172.21.15.158:0/2853814317 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c007310 con 0x7f7d7415dfb0 2023-10-16T10:03:19.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.788+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d740b1a40 con 0x7f7d7415dfb0 2023-10-16T10:03:19.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.789+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d74166f40 con 0x7f7d7415dfb0 2023-10-16T10:03:19.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.790+0000 7f7d62ffd700 1 -- 172.21.15.158:0/2853814317 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7d6c007470 con 0x7f7d7415dfb0 2023-10-16T10:03:19.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.790+0000 7f7d62ffd700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f7d5406b020 0x7f7d5406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:19.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.790+0000 7f7d62ffd700 1 -- 172.21.15.158:0/2853814317 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f7d6c088f60 con 0x7f7d7415dfb0 2023-10-16T10:03:19.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.790+0000 7f7d7259c700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f7d5406b020 0x7f7d5406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:19.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.791+0000 7f7d7259c700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f7d5406b020 0x7f7d5406d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b9b0 tx=0x7f7d68008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:19.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:19.795+0000 7f7d62ffd700 1 -- 172.21.15.158:0/2853814317 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d74166f40 con 0x7f7d7415dfb0 2023-10-16T10:03:20.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.062+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7d7406e770 con 0x7f7d7415dfb0 2023-10-16T10:03:20.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.062+0000 7f7d62ffd700 1 -- 172.21.15.158:0/2853814317 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7d7406e770 con 0x7f7d7415dfb0 2023-10-16T10:03:20.106 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:20.106 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:03:20.107 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T10:03:20.107 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.107 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T10:03:20.107 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T10:03:20.108 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.108 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.108 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:03:20.109 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:03:20.109 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.109 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.109 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.110 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.110 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T10:03:20.110 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T10:03:20.110 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.111 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.111 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:03:20.111 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:03:20.111 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.111 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.112 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.112 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.112 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T10:03:20.112 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T10:03:20.112 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.113 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.113 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:03:20.113 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:03:20.113 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.113 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.114 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.114 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.114 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T10:03:20.114 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T10:03:20.114 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.115 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.115 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:03:20.115 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:03:20.115 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.115 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.115 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.116 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.116 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T10:03:20.116 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T10:03:20.116 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.116 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.117 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.117 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.117 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.117 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.117 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.118 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T10:03:20.118 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T10:03:20.118 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.118 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.118 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.119 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.119 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.119 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.119 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.119 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T10:03:20.120 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T10:03:20.120 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.120 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.120 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.120 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.121 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.121 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T10:03:20.121 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T10:03:20.122 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.122 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.122 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.122 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.122 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.123 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.123 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.123 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T10:03:20.123 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T10:03:20.123 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.123 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.124 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.124 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.124 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.124 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.124 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.125 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T10:03:20.125 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T10:03:20.125 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.125 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.125 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.126 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.126 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.126 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.126 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T10:03:20.127 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T10:03:20.127 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.127 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.127 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.127 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.128 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.128 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.128 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.128 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T10:03:20.128 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T10:03:20.128 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.129 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:03:20.129 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:03:20.129 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.129 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.129 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.130 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.130 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T10:03:20.130 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T10:03:20.130 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.130 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.131 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T10:03:20.131 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T10:03:20.131 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.131 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.132 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.132 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T10:03:20.132 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T10:03:20.132 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:03:20.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:03:20.133 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:03:20.133 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.134 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.134 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T10:03:20.134 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T10:03:20.135 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:03:20.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:03:20.135 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:03:20.136 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.136 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.137 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T10:03:20.137 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T10:03:20.137 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.137 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:03:20.137 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:03:20.138 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:03:20.138 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:03:20.138 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.138 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.139 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.139 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T10:03:20.139 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T10:03:20.139 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.140 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:03:20.140 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:03:20.140 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:03:20.140 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.141 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.141 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.141 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.142 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T10:03:20.142 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T10:03:20.142 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.142 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:03:20.143 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:03:20.143 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.143 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:03:20.144 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:20.144 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T10:03:20.144 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T10:03:20.145 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:03:20.145 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:03:20.145 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:03:20.145 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:03:20.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.146 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:20.146 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T10:03:20.146 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:03:20.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.064+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f7d5406b020 msgr2=0x7f7d5406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.064+0000 7f7d78e10700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f7d5406b020 0x7f7d5406d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b9b0 tx=0x7f7d68008040 comp rx=0 tx=0).stop 2023-10-16T10:03:20.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.064+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 msgr2=0x7f7d7414f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.064+0000 7f7d78e10700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d7414f8c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00cb60 tx=0x7f7d6c00cf20 comp rx=0 tx=0).stop 2023-10-16T10:03:20.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 shutdown_connections 2023-10-16T10:03:20.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f7d5406b020 0x7f7d5406d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d7415dfb0 0x7f7d7414f8c0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d7406f340 0x7f7d740b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 --2- 172.21.15.158:0/2853814317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d740cc5d0 0x7f7d7414d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 >> 172.21.15.158:0/2853814317 conn(0x7f7d740d02b0 msgr2=0x7f7d740ce990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:20.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 shutdown_connections 2023-10-16T10:03:20.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.065+0000 7f7d78e10700 1 -- 172.21.15.158:0/2853814317 wait complete. 2023-10-16T10:03:20.150 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:20.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.271+0000 7fbc135d0700 1 Processor -- start 2023-10-16T10:03:20.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.271+0000 7fbc135d0700 1 -- start start 2023-10-16T10:03:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc135d0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbc0c15dfb0 0x7fbc0c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc135d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c06f340 0x7fbc0c06f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc135d0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c06e770 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc135d0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc0c06fd30 con 0x7fbc0c15dfb0 2023-10-16T10:03:20.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc135d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c0cf270 con 0x7fbc0c06f340 2023-10-16T10:03:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc10b6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbc0c15dfb0 0x7fbc0c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc1136c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbc0c0cc5d0 0x7fbc0c0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58186/0 (socket says 172.21.15.158:58186) 2023-10-16T10:03:20.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc10b6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbc0c15dfb0 0x7fbc0c06e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58012/0 (socket says 172.21.15.158:58012) 2023-10-16T10:03:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc1136c700 1 -- 172.21.15.158:0/4291352191 learned_addr learned my addr 172.21.15.158:0/4291352191 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc11b6d700 1 --2- 172.21.15.158:0/4291352191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c06f340 0x7fbc0c06f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc10b6b700 1 -- 172.21.15.158:0/4291352191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c06f340 msgr2=0x7fbc0c06f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc10b6b700 1 --2- 172.21.15.158:0/4291352191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c06f340 0x7fbc0c06f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc10b6b700 1 -- 172.21.15.158:0/4291352191 >> v1:172.21.15.158:6789/0 conn(0x7fbc0c0cc5d0 legacy=0x7fbc0c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc03fff700 1 -- 172.21.15.158:0/4291352191 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268638915 0 0) 0x7fbc08001110 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc10b6b700 1 -- 172.21.15.158:0/4291352191 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0cbe00 con 0x7fbc0c15dfb0 2023-10-16T10:03:20.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.272+0000 7fbc10b6b700 1 --2- 172.21.15.158:0/4291352191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c15dfb0 0x7fbc0c06e230 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009840 tx=0x7fbbfc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fe8b6bd2eef598 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc03fff700 1 -- 172.21.15.158:0/4291352191 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc011830 con 0x7fbc0c15dfb0 2023-10-16T10:03:20.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc03fff700 1 -- 172.21.15.158:0/4291352191 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc011990 con 0x7fbc0c15dfb0 2023-10-16T10:03:20.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc03fff700 1 -- 172.21.15.158:0/4291352191 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc011d20 con 0x7fbc0c15dfb0 2023-10-16T10:03:20.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 -- 172.21.15.158:0/4291352191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c15dfb0 msgr2=0x7fbc0c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 --2- 172.21.15.158:0/4291352191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c15dfb0 0x7fbc0c06e230 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009840 tx=0x7fbbfc009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:20.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 -- 172.21.15.158:0/4291352191 shutdown_connections 2023-10-16T10:03:20.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 --2- 172.21.15.158:0/4291352191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c06f340 0x7fbc0c06f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 --2- 172.21.15.158:0/4291352191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c15dfb0 0x7fbc0c06e230 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 -- 172.21.15.158:0/4291352191 >> 172.21.15.158:0/4291352191 conn(0x7fbc0c0d02b0 msgr2=0x7fbc0c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:20.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 -- 172.21.15.158:0/4291352191 shutdown_connections 2023-10-16T10:03:20.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.273+0000 7fbc135d0700 1 -- 172.21.15.158:0/4291352191 wait complete. 2023-10-16T10:03:20.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.274+0000 7fbc135d0700 1 Processor -- start 2023-10-16T10:03:20.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc135d0700 1 -- start start 2023-10-16T10:03:20.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc135d0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc0c06f340 0x7fbc0c157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc135d0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c0cc5d0 0x7fbc0c0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc135d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c15dfb0 0x7fbc0c15c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc135d0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c0c4ff0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc135d0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c0c52d0 con 0x7fbc0c06f340 2023-10-16T10:03:20.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc135d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c0c55b0 con 0x7fbc0c15dfb0 2023-10-16T10:03:20.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc10b6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c0cc5d0 0x7fbc0c0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc10b6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c0cc5d0 0x7fbc0c0c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58016/0 (socket says 172.21.15.158:58016) 2023-10-16T10:03:20.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.275+0000 7fbc10b6b700 1 -- 172.21.15.158:0/1907124810 learned_addr learned my addr 172.21.15.158:0/1907124810 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:20.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc10b6b700 1 -- 172.21.15.158:0/1907124810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c15dfb0 msgr2=0x7fbc0c15c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc10b6b700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c15dfb0 0x7fbc0c15c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc10b6b700 1 -- 172.21.15.158:0/1907124810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc0c06f340 msgr2=0x7fbc0c157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc10b6b700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc0c06f340 0x7fbc0c157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc10b6b700 1 -- 172.21.15.158:0/1907124810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc0094a0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc10b6b700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c0cc5d0 0x7fbc0c0c7380 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0090b0 tx=0x7fbbfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc01ffb700 1 -- 172.21.15.158:0/1907124810 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0036a0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc01ffb700 1 -- 172.21.15.158:0/1907124810 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc01f070 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c0c5890 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc01ffb700 1 -- 172.21.15.158:0/1907124810 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc003a60 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.276+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c157660 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.277+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c0c5db0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.277+0000 7fbc01ffb700 1 -- 172.21.15.158:0/1907124810 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbbfc003bc0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.278+0000 7fbc01ffb700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbbf006b020 0x7fbbf006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.278+0000 7fbc1136c700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbbf006b020 0x7fbbf006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.278+0000 7fbc01ffb700 1 -- 172.21.15.158:0/1907124810 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5629+0+0 (secure 0 0 0) 0x7fbbfc021070 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.278+0000 7fbc1136c700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbbf006b020 0x7fbbf006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbc08001610 tx=0x7fbc080079f0 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.282+0000 7fbc01ffb700 1 -- 172.21.15.158:0/1907124810 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbc0c0c5db0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.475+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc0c0c5db0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.476+0000 7fbc01ffb700 1 -- 172.21.15.158:0/1907124810 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+13774 (secure 0 0 0) 0x7fbc0c0c5db0 con 0x7fbc0c0cc5d0 2023-10-16T10:03:20.477 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:20.477 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":136,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:19.634478+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:20.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbbf006b020 msgr2=0x7fbbf006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbbf006b020 0x7fbbf006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbc08001610 tx=0x7fbc080079f0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c0cc5d0 msgr2=0x7fbc0c0c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c0cc5d0 0x7fbc0c0c7380 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0090b0 tx=0x7fbbfc004750 comp rx=0 tx=0).stop 2023-10-16T10:03:20.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 shutdown_connections 2023-10-16T10:03:20.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbbf006b020 0x7fbbf006d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbc0c15dfb0 0x7fbc0c15c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc0c06f340 0x7fbc0c157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 --2- 172.21.15.158:0/1907124810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc0c0cc5d0 0x7fbc0c0c7380 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 >> 172.21.15.158:0/1907124810 conn(0x7fbc0c0d02b0 msgr2=0x7fbc0c15ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:20.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 shutdown_connections 2023-10-16T10:03:20.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.478+0000 7fbc135d0700 1 -- 172.21.15.158:0/1907124810 wait complete. 2023-10-16T10:03:20.492 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:03:20.492 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:03:20.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.614+0000 7f4cc3846700 1 Processor -- start 2023-10-16T10:03:20.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.615+0000 7f4cc3846700 1 -- start start 2023-10-16T10:03:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.615+0000 7f4cc3846700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4cbc15dad0 0x7f4cbc0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.615+0000 7f4cc3846700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.615+0000 7f4cc3846700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc0d0ae0 con 0x7f4cbc15deb0 2023-10-16T10:03:20.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.615+0000 7f4cc3846700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc0d12d0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.615+0000 7f4cc3846700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0cbdf0 con 0x7f4cbc0ccc00 2023-10-16T10:03:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4cbc15dad0 0x7f4cbc0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4cbc15dad0 0x7f4cbc0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58024/0 (socket says 172.21.15.158:58024) 2023-10-16T10:03:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc0de1700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4cbc15deb0 0x7f4cbc0ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58202/0 (socket says 172.21.15.158:58202) 2023-10-16T10:03:20.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 -- 172.21.15.158:0/4137932488 learned_addr learned my addr 172.21.15.158:0/4137932488 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:20.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc1de3700 1 --2- 172.21.15.158:0/4137932488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 -- 172.21.15.158:0/4137932488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 msgr2=0x7f4cbc0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:20.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 --2- 172.21.15.158:0/4137932488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 -- 172.21.15.158:0/4137932488 >> v1:172.21.15.158:6789/0 conn(0x7f4cbc15deb0 legacy=0x7f4cbc0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 -- 172.21.15.158:0/4137932488 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0cc280 con 0x7f4cbc15dad0 2023-10-16T10:03:20.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc1de3700 1 --2- 172.21.15.158:0/4137932488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:20.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cc15e2700 1 --2- 172.21.15.158:0/4137932488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0cdfa0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0098f0 tx=0x7f4cac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661873ea86643241 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cb3fff700 1 -- 172.21.15.158:0/4137932488 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac00b8a0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cb3fff700 1 -- 172.21.15.158:0/4137932488 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac00ba00 con 0x7f4cbc15dad0 2023-10-16T10:03:20.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.616+0000 7f4cb3fff700 1 -- 172.21.15.158:0/4137932488 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac00bd90 con 0x7f4cbc15dad0 2023-10-16T10:03:20.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 -- 172.21.15.158:0/4137932488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 msgr2=0x7f4cbc0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 --2- 172.21.15.158:0/4137932488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0cdfa0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0098f0 tx=0x7f4cac009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:20.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 -- 172.21.15.158:0/4137932488 shutdown_connections 2023-10-16T10:03:20.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 --2- 172.21.15.158:0/4137932488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0cdfa0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 --2- 172.21.15.158:0/4137932488 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 -- 172.21.15.158:0/4137932488 >> 172.21.15.158:0/4137932488 conn(0x7f4cbc099a10 msgr2=0x7f4cbc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:20.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 -- 172.21.15.158:0/4137932488 shutdown_connections 2023-10-16T10:03:20.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.617+0000 7f4cc3846700 1 -- 172.21.15.158:0/4137932488 wait complete. 2023-10-16T10:03:20.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.618+0000 7f4cc3846700 1 Processor -- start 2023-10-16T10:03:20.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.618+0000 7f4cc3846700 1 -- start start 2023-10-16T10:03:20.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc3846700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc3846700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc3846700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc15deb0 0x7f4cbc15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc3846700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0b2fc0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc3846700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0b3210 con 0x7f4cbc15deb0 2023-10-16T10:03:20.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc3846700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0b34f0 con 0x7f4cbc0ccc00 2023-10-16T10:03:20.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58028/0 (socket says 172.21.15.158:58028) 2023-10-16T10:03:20.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 -- 172.21.15.158:0/1084214932 learned_addr learned my addr 172.21.15.158:0/1084214932 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:20.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc15e2700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc1de3700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc15deb0 0x7f4cbc15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 -- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 msgr2=0x7f4cbc0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 -- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc15deb0 msgr2=0x7f4cbc15b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc15deb0 0x7f4cbc15b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 -- 172.21.15.158:0/1084214932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac009550 con 0x7f4cbc15dad0 2023-10-16T10:03:20.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc15e2700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:20.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.619+0000 7f4cc0de1700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0c9be0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c500 tx=0x7f4cb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.620+0000 7f4cb1ffb700 1 -- 172.21.15.158:0/1084214932 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8005cb0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.620+0000 7f4cb1ffb700 1 -- 172.21.15.158:0/1084214932 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb8005e10 con 0x7f4cbc15dad0 2023-10-16T10:03:20.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.620+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc0b37d0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.620+0000 7f4cb1ffb700 1 -- 172.21.15.158:0/1084214932 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80072d0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.620+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc0b4510 con 0x7f4cbc15dad0 2023-10-16T10:03:20.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.621+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc0681b0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.621+0000 7f4cb1ffb700 1 -- 172.21.15.158:0/1084214932 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4cb8007430 con 0x7f4cbc15dad0 2023-10-16T10:03:20.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.621+0000 7f4cb1ffb700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4ca006b020 0x7f4ca006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.621+0000 7f4cb1ffb700 1 -- 172.21.15.158:0/1084214932 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f4cb8088a40 con 0x7f4cbc15dad0 2023-10-16T10:03:20.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.621+0000 7f4cc15e2700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4ca006b020 0x7f4ca006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.622+0000 7f4cc15e2700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4ca006b020 0x7f4ca006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009c40 tx=0x7f4cac008030 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.626+0000 7f4cb1ffb700 1 -- 172.21.15.158:0/1084214932 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4cb808ae60 con 0x7f4cbc15dad0 2023-10-16T10:03:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.816+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cbc0ce0e0 con 0x7f4cbc15dad0 2023-10-16T10:03:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.816+0000 7f4cb1ffb700 1 -- 172.21.15.158:0/1084214932 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v136) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4cb8017020 con 0x7f4cbc15dad0 2023-10-16T10:03:20.818 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:03:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4ca006b020 msgr2=0x7f4ca006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4ca006b020 0x7f4ca006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009c40 tx=0x7f4cac008030 comp rx=0 tx=0).stop 2023-10-16T10:03:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 msgr2=0x7f4cbc0c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0c9be0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c500 tx=0x7f4cb800c8c0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 shutdown_connections 2023-10-16T10:03:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4ca006b020 0x7f4ca006d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc15deb0 0x7f4cbc15b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cbc0ccc00 0x7f4cbc0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 --2- 172.21.15.158:0/1084214932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cbc15dad0 0x7f4cbc0c9be0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 >> 172.21.15.158:0/1084214932 conn(0x7f4cbc099a10 msgr2=0x7f4cbc15c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 shutdown_connections 2023-10-16T10:03:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.819+0000 7f4cc3846700 1 -- 172.21.15.158:0/1084214932 wait complete. 2023-10-16T10:03:20.834 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T10:03:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.953+0000 7fe7ecbb9700 1 Processor -- start 2023-10-16T10:03:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.953+0000 7fe7ecbb9700 1 -- start start 2023-10-16T10:03:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.954+0000 7fe7ecbb9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe7e80cc0c0 0x7fe7e80aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.954+0000 7fe7ecbb9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe7e806bfb0 0x7fe7e80cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.954+0000 7fe7ecbb9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e80cf2f0 0x7fe7e815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.954+0000 7fe7ecbb9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e80cf7a0 con 0x7fe7e806bfb0 2023-10-16T10:03:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.954+0000 7fe7ecbb9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e815b810 con 0x7fe7e80cc0c0 2023-10-16T10:03:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.954+0000 7fe7ecbb9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e815a990 con 0x7fe7e80cf2f0 2023-10-16T10:03:20.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.954+0000 7fe7e5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe7e806bfb0 0x7fe7e80cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe7e806bfb0 0x7fe7e80cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58036/0 (socket says 172.21.15.158:58036) 2023-10-16T10:03:20.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 -- 172.21.15.158:0/553658335 learned_addr learned my addr 172.21.15.158:0/553658335 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:20.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e6d9d700 1 --2- 172.21.15.158:0/553658335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e80cf2f0 0x7fe7e815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e659c700 1 --2- 172.21.15.158:0/553658335 >> v2:172.21.15.172:3300/0 conn(0x7fe7e80cc0c0 0x7fe7e80aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 -- 172.21.15.158:0/553658335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e80cf2f0 msgr2=0x7fe7e815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:20.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 --2- 172.21.15.158:0/553658335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e80cf2f0 0x7fe7e815dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 -- 172.21.15.158:0/553658335 >> v2:172.21.15.172:3300/0 conn(0x7fe7e80cc0c0 msgr2=0x7fe7e80aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:20.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 --2- 172.21.15.158:0/553658335 >> v2:172.21.15.172:3300/0 conn(0x7fe7e80cc0c0 0x7fe7e80aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 -- 172.21.15.158:0/553658335 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e815ac00 con 0x7fe7e806bfb0 2023-10-16T10:03:20.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e659c700 1 --2- 172.21.15.158:0/553658335 >> v2:172.21.15.172:3300/0 conn(0x7fe7e80cc0c0 0x7fe7e80aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:20.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e5d9b700 1 --2- 172.21.15.158:0/553658335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e806bfb0 0x7fe7e80cedb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80cf990 tx=0x7fe7d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8743c513c772486d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e559a700 1 -- 172.21.15.158:0/553658335 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d800b870 con 0x7fe7e806bfb0 2023-10-16T10:03:20.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e559a700 1 -- 172.21.15.158:0/553658335 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d800b9d0 con 0x7fe7e806bfb0 2023-10-16T10:03:20.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.955+0000 7fe7e559a700 1 -- 172.21.15.158:0/553658335 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d800bd60 con 0x7fe7e806bfb0 2023-10-16T10:03:20.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/553658335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e806bfb0 msgr2=0x7fe7e80cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/553658335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e806bfb0 0x7fe7e80cedb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80cf990 tx=0x7fe7d8009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/553658335 shutdown_connections 2023-10-16T10:03:20.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/553658335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e80cf2f0 0x7fe7e815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/553658335 >> v2:172.21.15.172:3300/0 conn(0x7fe7e80cc0c0 0x7fe7e80aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/553658335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e806bfb0 0x7fe7e80cedb0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/553658335 >> 172.21.15.158:0/553658335 conn(0x7fe7e80cc4b0 msgr2=0x7fe7e80cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:20.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/553658335 shutdown_connections 2023-10-16T10:03:20.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.956+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/553658335 wait complete. 2023-10-16T10:03:20.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.957+0000 7fe7ecbb9700 1 Processor -- start 2023-10-16T10:03:20.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.957+0000 7fe7ecbb9700 1 -- start start 2023-10-16T10:03:20.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7ecbb9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e806bfb0 0x7fe7e80ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7ecbb9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e80cf2f0 0x7fe7e81f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7ecbb9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e8148450 0x7fe7e80c4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7ecbb9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f53c0 con 0x7fe7e8148450 2023-10-16T10:03:20.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7ecbb9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f4210 con 0x7fe7e80cf2f0 2023-10-16T10:03:20.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7ecbb9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f47b0 con 0x7fe7e806bfb0 2023-10-16T10:03:20.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7e6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e8148450 0x7fe7e80c4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7e6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e8148450 0x7fe7e80c4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58046/0 (socket says 172.21.15.158:58046) 2023-10-16T10:03:20.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7e6d9d700 1 -- 172.21.15.158:0/3968625642 learned_addr learned my addr 172.21.15.158:0/3968625642 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:20.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7e5d9b700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e80cf2f0 0x7fe7e81f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.958+0000 7fe7e659c700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e806bfb0 0x7fe7e80ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7e6d9d700 1 -- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e806bfb0 msgr2=0x7fe7e80ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7e6d9d700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e806bfb0 0x7fe7e80ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7e6d9d700 1 -- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e80cf2f0 msgr2=0x7fe7e81f3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:20.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7e6d9d700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e80cf2f0 0x7fe7e81f3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:20.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7e6d9d700 1 -- 172.21.15.158:0/3968625642 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d8009550 con 0x7fe7e8148450 2023-10-16T10:03:20.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7e6d9d700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e8148450 0x7fe7e80c4e30 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0007e60 tx=0x7fe7e000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7d6ffd700 1 -- 172.21.15.158:0/3968625642 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e000ed30 con 0x7fe7e8148450 2023-10-16T10:03:20.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7d6ffd700 1 -- 172.21.15.158:0/3968625642 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e0014420 con 0x7fe7e8148450 2023-10-16T10:03:20.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e81f4a90 con 0x7fe7e8148450 2023-10-16T10:03:20.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7d6ffd700 1 -- 172.21.15.158:0/3968625642 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00147d0 con 0x7fe7e8148450 2023-10-16T10:03:20.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.959+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e81f5050 con 0x7fe7e8148450 2023-10-16T10:03:20.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.960+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e80681b0 con 0x7fe7e8148450 2023-10-16T10:03:20.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.960+0000 7fe7d6ffd700 1 -- 172.21.15.158:0/3968625642 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe7e0014930 con 0x7fe7e8148450 2023-10-16T10:03:20.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.961+0000 7fe7d6ffd700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe7c806b020 0x7fe7c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:20.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.961+0000 7fe7d6ffd700 1 -- 172.21.15.158:0/3968625642 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5629+0+0 (secure 0 0 0) 0x7fe7e0089d20 con 0x7fe7e8148450 2023-10-16T10:03:20.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.961+0000 7fe7e659c700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe7c806b020 0x7fe7c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:20.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.961+0000 7fe7e659c700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe7c806b020 0x7fe7c806d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0045f0 tx=0x7fe7dc008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:20.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:20.965+0000 7fe7d6ffd700 1 -- 172.21.15.158:0/3968625642 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7e0054dc0 con 0x7fe7e8148450 2023-10-16T10:03:21.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.159+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe7e806bc90 con 0x7fe7e8148450 2023-10-16T10:03:21.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.159+0000 7fe7d6ffd700 1 -- 172.21.15.158:0/3968625642 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v136) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe7e00583e0 con 0x7fe7e8148450 2023-10-16T10:03:21.161 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:21.162 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:03:21.162 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:21.162 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T10:03:21.162 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T10:03:21.162 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:21.163 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:03:21.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.161+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe7c806b020 msgr2=0x7fe7c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:21.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.161+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe7c806b020 0x7fe7c806d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0045f0 tx=0x7fe7dc008040 comp rx=0 tx=0).stop 2023-10-16T10:03:21.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e8148450 msgr2=0x7fe7e80c4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:21.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e8148450 0x7fe7e80c4e30 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0007e60 tx=0x7fe7e000e3c0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 shutdown_connections 2023-10-16T10:03:21.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe7c806b020 0x7fe7c806d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7e8148450 0x7fe7e80c4e30 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e806bfb0 0x7fe7e80ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 --2- 172.21.15.158:0/3968625642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e80cf2f0 0x7fe7e81f3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 >> 172.21.15.158:0/3968625642 conn(0x7fe7e80cc4b0 msgr2=0x7fe7e815d040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:21.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 shutdown_connections 2023-10-16T10:03:21.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.162+0000 7fe7ecbb9700 1 -- 172.21.15.158:0/3968625642 wait complete. 2023-10-16T10:03:21.175 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T10:03:21.176 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T10:03:21.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.299+0000 7f1114607700 1 Processor -- start 2023-10-16T10:03:21.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.299+0000 7f1114607700 1 -- start start 2023-10-16T10:03:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1114607700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c15dbd0 0x7f110c06df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1114607700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f110c15dfc0 0x7f110c0718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1114607700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110c15e510 con 0x7f110c06a8d0 2023-10-16T10:03:21.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1114607700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110c0ccbb0 con 0x7f110c15dfc0 2023-10-16T10:03:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1114607700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c15e7f0 con 0x7f110c15dbd0 2023-10-16T10:03:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1111ba2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f110c15dfc0 0x7f110c0718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1112ba4700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f110c06a8d0 0x7f110c0ccfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58230/0 (socket says 172.21.15.158:58230) 2023-10-16T10:03:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1112ba4700 1 -- 172.21.15.158:0/3671722067 learned_addr learned my addr 172.21.15.158:0/3671722067 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:21.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f11123a3700 1 --2- 172.21.15.158:0/3671722067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c15dbd0 0x7f110c06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1111ba2700 1 -- 172.21.15.158:0/3671722067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c15dbd0 msgr2=0x7f110c06df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1111ba2700 1 --2- 172.21.15.158:0/3671722067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c15dbd0 0x7f110c06df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1111ba2700 1 -- 172.21.15.158:0/3671722067 >> v1:172.21.15.158:6789/0 conn(0x7f110c06a8d0 legacy=0x7f110c0ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1111ba2700 1 -- 172.21.15.158:0/3671722067 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c15ead0 con 0x7f110c15dfc0 2023-10-16T10:03:21.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f11113a1700 1 -- 172.21.15.158:0/3671722067 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151623229 0 0) 0x7f110c15e510 con 0x7f110c06a8d0 2023-10-16T10:03:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f1111ba2700 1 --2- 172.21.15.158:0/3671722067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c15dfc0 0x7f110c0718f0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009840 tx=0x7f10fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfebb4ab1798ce21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f11113a1700 1 -- 172.21.15.158:0/3671722067 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc011830 con 0x7f110c15dfc0 2023-10-16T10:03:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f11113a1700 1 -- 172.21.15.158:0/3671722067 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc011990 con 0x7f110c15dfc0 2023-10-16T10:03:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.300+0000 7f11113a1700 1 -- 172.21.15.158:0/3671722067 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc011d20 con 0x7f110c15dfc0 2023-10-16T10:03:21.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 -- 172.21.15.158:0/3671722067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c15dfc0 msgr2=0x7f110c0718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 --2- 172.21.15.158:0/3671722067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c15dfc0 0x7f110c0718f0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009840 tx=0x7f10fc009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 -- 172.21.15.158:0/3671722067 shutdown_connections 2023-10-16T10:03:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 --2- 172.21.15.158:0/3671722067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c15dbd0 0x7f110c06df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 --2- 172.21.15.158:0/3671722067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c15dfc0 0x7f110c0718f0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 -- 172.21.15.158:0/3671722067 >> 172.21.15.158:0/3671722067 conn(0x7f110c0b1f00 msgr2=0x7f110c0b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:21.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 -- 172.21.15.158:0/3671722067 shutdown_connections 2023-10-16T10:03:21.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.301+0000 7f1114607700 1 -- 172.21.15.158:0/3671722067 wait complete. 2023-10-16T10:03:21.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.302+0000 7f1114607700 1 Processor -- start 2023-10-16T10:03:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.302+0000 7f1114607700 1 -- start start 2023-10-16T10:03:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1114607700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c06a8d0 0x7f110c159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1114607700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c15dfc0 0x7f110c151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1114607700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c1f3480 0x7f110c0b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1114607700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c1592a0 con 0x7f110c1f3480 2023-10-16T10:03:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1114607700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c1f3ed0 con 0x7f110c15dfc0 2023-10-16T10:03:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1114607700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c1f41b0 con 0x7f110c06a8d0 2023-10-16T10:03:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c1f3480 0x7f110c0b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c1f3480 0x7f110c0b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58060/0 (socket says 172.21.15.158:58060) 2023-10-16T10:03:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 -- 172.21.15.158:0/1251098191 learned_addr learned my addr 172.21.15.158:0/1251098191 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:21.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f11123a3700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c06a8d0 0x7f110c159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:21.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1111ba2700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c15dfc0 0x7f110c151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:21.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 -- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c06a8d0 msgr2=0x7f110c159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:21.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c06a8d0 0x7f110c159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 -- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c15dfc0 msgr2=0x7f110c151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:21.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c15dfc0 0x7f110c151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:21.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.303+0000 7f1112ba4700 1 -- 172.21.15.158:0/1251098191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc0094a0 con 0x7f110c1f3480 2023-10-16T10:03:21.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.304+0000 7f1112ba4700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c1f3480 0x7f110c0b0ec0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f1104002230 tx=0x7f110400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:21.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.304+0000 7f1102ffd700 1 -- 172.21.15.158:0/1251098191 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110400ac20 con 0x7f110c1f3480 2023-10-16T10:03:21.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.304+0000 7f1102ffd700 1 -- 172.21.15.158:0/1251098191 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110400ada0 con 0x7f110c1f3480 2023-10-16T10:03:21.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.304+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110c159480 con 0x7f110c1f3480 2023-10-16T10:03:21.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.304+0000 7f1102ffd700 1 -- 172.21.15.158:0/1251098191 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104007310 con 0x7f110c1f3480 2023-10-16T10:03:21.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.304+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110c15a1c0 con 0x7f110c1f3480 2023-10-16T10:03:21.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.305+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110c1599a0 con 0x7f110c1f3480 2023-10-16T10:03:21.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.305+0000 7f1102ffd700 1 -- 172.21.15.158:0/1251098191 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1104007470 con 0x7f110c1f3480 2023-10-16T10:03:21.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.305+0000 7f1102ffd700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f10f006b020 0x7f10f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:21.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.306+0000 7f11123a3700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f10f006b020 0x7f10f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:21.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.306+0000 7f1102ffd700 1 -- 172.21.15.158:0/1251098191 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f1104089570 con 0x7f110c1f3480 2023-10-16T10:03:21.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.306+0000 7f11123a3700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f10f006b020 0x7f10f006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1108009930 tx=0x7f1108008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:21.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.310+0000 7f1102ffd700 1 -- 172.21.15.158:0/1251098191 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f110c1599a0 con 0x7f110c1f3480 2023-10-16T10:03:21.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:21.501+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f110c1599a0 con 0x7f110c1f3480 2023-10-16T10:03:22.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.195+0000 7f1102ffd700 1 -- 172.21.15.158:0/1251098191 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v137) v1 ==== 138+0+0 (secure 0 0 0) 0x7f110c1599a0 con 0x7f110c1f3480 2023-10-16T10:03:22.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f10f006b020 msgr2=0x7f10f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:22.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f10f006b020 0x7f10f006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1108009930 tx=0x7f1108008040 comp rx=0 tx=0).stop 2023-10-16T10:03:22.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c1f3480 msgr2=0x7f110c0b0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:22.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c1f3480 0x7f110c0b0ec0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f1104002230 tx=0x7f110400a060 comp rx=0 tx=0).stop 2023-10-16T10:03:22.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 shutdown_connections 2023-10-16T10:03:22.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f10f006b020 0x7f10f006d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f110c1f3480 0x7f110c0b0ec0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f110c06a8d0 0x7f110c159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 --2- 172.21.15.158:0/1251098191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c15dfc0 0x7f110c151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.198+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 >> 172.21.15.158:0/1251098191 conn(0x7f110c0b1f00 msgr2=0x7f110c0cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:22.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.199+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 shutdown_connections 2023-10-16T10:03:22.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.199+0000 7f1114607700 1 -- 172.21.15.158:0/1251098191 wait complete. 2023-10-16T10:03:22.202 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T10:03:22.218 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T10:03:22.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.349+0000 7fb3d3e05700 1 Processor -- start 2023-10-16T10:03:22.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.349+0000 7fb3d3e05700 1 -- start start 2023-10-16T10:03:22.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d3e05700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3cc15b4b0 0x7fb3cc15fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:22.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d3e05700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:22.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d3e05700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3cc15f570 con 0x7fb3cc06b8e0 2023-10-16T10:03:22.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d3e05700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3cc06d9a0 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d3e05700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc06d3e0 con 0x7fb3cc072150 2023-10-16T10:03:22.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d13a0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:22.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d1ba1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3cc15b4b0 0x7fb3cc15fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:22.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d13a0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc06dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:40666/0 (socket says 172.21.15.158:40666) 2023-10-16T10:03:22.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d1ba1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3cc15b4b0 0x7fb3cc15fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:59244/0 (socket says 172.21.15.158:59244) 2023-10-16T10:03:22.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d13a0700 1 -- 172.21.15.158:0/2073855998 learned_addr learned my addr 172.21.15.158:0/2073855998 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:22.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d23a2700 1 --1- 172.21.15.158:0/2073855998 >> v1:172.21.15.172:6789/0 conn(0x7fb3cc06b8e0 0x7fb3cc0b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:33912/0 (socket says 172.21.15.158:33912) 2023-10-16T10:03:22.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d13a0700 1 -- 172.21.15.158:0/2073855998 >> v2:172.21.15.172:3300/0 conn(0x7fb3cc15b4b0 msgr2=0x7fb3cc15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:22.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d13a0700 1 --2- 172.21.15.158:0/2073855998 >> v2:172.21.15.172:3300/0 conn(0x7fb3cc15b4b0 0x7fb3cc15fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d13a0700 1 -- 172.21.15.158:0/2073855998 >> v1:172.21.15.172:6789/0 conn(0x7fb3cc06b8e0 legacy=0x7fb3cc0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:22.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d13a0700 1 -- 172.21.15.158:0/2073855998 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc06d6c0 con 0x7fb3cc072150 2023-10-16T10:03:22.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.350+0000 7fb3d1ba1700 1 --2- 172.21.15.158:0/2073855998 >> v2:172.21.15.172:3300/0 conn(0x7fb3cc15b4b0 0x7fb3cc15fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:22.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d13a0700 1 --2- 172.21.15.158:0/2073855998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc06dcc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0098f0 tx=0x7fb3bc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153474aaf8dd91d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:22.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d0b9f700 1 -- 172.21.15.158:0/2073855998 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc011830 con 0x7fb3cc072150 2023-10-16T10:03:22.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d0b9f700 1 -- 172.21.15.158:0/2073855998 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc011990 con 0x7fb3cc072150 2023-10-16T10:03:22.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d0b9f700 1 -- 172.21.15.158:0/2073855998 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc011d20 con 0x7fb3cc072150 2023-10-16T10:03:22.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2073855998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 msgr2=0x7fb3cc06dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:22.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2073855998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc06dcc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0098f0 tx=0x7fb3bc009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:22.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2073855998 shutdown_connections 2023-10-16T10:03:22.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2073855998 >> v2:172.21.15.172:3300/0 conn(0x7fb3cc15b4b0 0x7fb3cc15fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2073855998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc06dcc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.351+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2073855998 >> 172.21.15.158:0/2073855998 conn(0x7fb3cc0710f0 msgr2=0x7fb3cc071500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:22.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.352+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2073855998 shutdown_connections 2023-10-16T10:03:22.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.352+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2073855998 wait complete. 2023-10-16T10:03:22.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.353+0000 7fb3d3e05700 1 Processor -- start 2023-10-16T10:03:22.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.353+0000 7fb3d3e05700 1 -- start start 2023-10-16T10:03:22.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d3e05700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc06b8e0 0x7fb3cc140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:22.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d3e05700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:22.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d3e05700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3cc15b4b0 0x7fb3cc156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:22.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d3e05700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc157e40 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d3e05700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc1571c0 con 0x7fb3cc06b8e0 2023-10-16T10:03:22.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d3e05700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc1574a0 con 0x7fb3cc072150 2023-10-16T10:03:22.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d23a2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3cc15b4b0 0x7fb3cc156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:22.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d23a2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3cc15b4b0 0x7fb3cc156bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58072/0 (socket says 172.21.15.158:58072) 2023-10-16T10:03:22.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d23a2700 1 -- 172.21.15.158:0/2241289050 learned_addr learned my addr 172.21.15.158:0/2241289050 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:22.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d13a0700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:22.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.354+0000 7fb3d1ba1700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc06b8e0 0x7fb3cc140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:22.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d23a2700 1 -- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 msgr2=0x7fb3cc156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:22.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d23a2700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d23a2700 1 -- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc06b8e0 msgr2=0x7fb3cc140a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:22.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d23a2700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc06b8e0 0x7fb3cc140a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:22.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d23a2700 1 -- 172.21.15.158:0/2241289050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c80094a0 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d1ba1700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc06b8e0 0x7fb3cc140a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d13a0700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d23a2700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3cc15b4b0 0x7fb3cc156bd0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400cbd0 tx=0x7fb3c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3c27fc700 1 -- 172.21.15.158:0/2241289050 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c400abb0 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3c27fc700 1 -- 172.21.15.158:0/2241289050 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c400ad10 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc009550 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3c27fc700 1 -- 172.21.15.158:0/2241289050 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4007270 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.355+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3cc157a60 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.356+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3cc157f80 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.356+0000 7fb3c27fc700 1 -- 172.21.15.158:0/2241289050 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3c40073d0 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.357+0000 7fb3c27fc700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb3b006b020 0x7fb3b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.357+0000 7fb3d1ba1700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb3b006b020 0x7fb3b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.357+0000 7fb3c27fc700 1 -- 172.21.15.158:0/2241289050 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5999+0+0 (secure 0 0 0) 0x7fb3c4089880 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.357+0000 7fb3d1ba1700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb3b006b020 0x7fb3b006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009d20 tx=0x7fb3c8008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.361+0000 7fb3c27fc700 1 -- 172.21.15.158:0/2241289050 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3c4054810 con 0x7fb3cc15b4b0 2023-10-16T10:03:22.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:22.556+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb3cc157f80 con 0x7fb3cc15b4b0 2023-10-16T10:03:23.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.202+0000 7fb3c27fc700 1 -- 172.21.15.158:0/2241289050 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v138) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb3cc157f80 con 0x7fb3cc15b4b0 2023-10-16T10:03:23.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb3b006b020 msgr2=0x7fb3b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:23.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb3b006b020 0x7fb3b006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009d20 tx=0x7fb3c8008040 comp rx=0 tx=0).stop 2023-10-16T10:03:23.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3cc15b4b0 msgr2=0x7fb3cc156bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:23.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3cc15b4b0 0x7fb3cc156bd0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400cbd0 tx=0x7fb3c400cf90 comp rx=0 tx=0).stop 2023-10-16T10:03:23.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 shutdown_connections 2023-10-16T10:03:23.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb3b006b020 0x7fb3b006d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3cc15b4b0 0x7fb3cc156bd0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc06b8e0 0x7fb3cc140a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 --2- 172.21.15.158:0/2241289050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc072150 0x7fb3cc156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.205+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 >> 172.21.15.158:0/2241289050 conn(0x7fb3cc0710f0 msgr2=0x7fb3cc0cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:23.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.208+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 shutdown_connections 2023-10-16T10:03:23.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.208+0000 7fb3d3e05700 1 -- 172.21.15.158:0/2241289050 wait complete. 2023-10-16T10:03:23.213 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T10:03:23.233 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T10:03:23.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.359+0000 7fe67bed7700 1 Processor -- start 2023-10-16T10:03:23.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.359+0000 7fe67bed7700 1 -- start start 2023-10-16T10:03:23.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.359+0000 7fe67bed7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe67415b590 0x7fe6740ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:23.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.359+0000 7fe67bed7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe67415f1d0 0x7fe6740b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:23.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe67bed7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740cf4c0 con 0x7fe6740cbaf0 2023-10-16T10:03:23.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe67bed7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67415f970 con 0x7fe67415f1d0 2023-10-16T10:03:23.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe67bed7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67415fc50 con 0x7fe67415b590 2023-10-16T10:03:23.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe67415f1d0 0x7fe6740b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:23.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe67415f1d0 0x7fe6740b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58076/0 (socket says 172.21.15.158:58076) 2023-10-16T10:03:23.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 -- 172.21.15.158:0/1902366948 learned_addr learned my addr 172.21.15.158:0/1902366948 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:23.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679c73700 1 --2- 172.21.15.158:0/1902366948 >> v2:172.21.15.172:3300/0 conn(0x7fe67415b590 0x7fe6740ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:23.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 -- 172.21.15.158:0/1902366948 >> v2:172.21.15.172:3300/0 conn(0x7fe67415b590 msgr2=0x7fe6740ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:23.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 --2- 172.21.15.158:0/1902366948 >> v2:172.21.15.172:3300/0 conn(0x7fe67415b590 0x7fe6740ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 -- 172.21.15.158:0/1902366948 >> v1:172.21.15.172:6789/0 conn(0x7fe6740cbaf0 legacy=0x7fe6740cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:23.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 -- 172.21.15.158:0/1902366948 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67415ff30 con 0x7fe67415f1d0 2023-10-16T10:03:23.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe679472700 1 --2- 172.21.15.158:0/1902366948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740b42d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe6740b3520 tx=0x7fe670009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1205c17227e1cb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:23.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe678c71700 1 -- 172.21.15.158:0/1902366948 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000b870 con 0x7fe67415f1d0 2023-10-16T10:03:23.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe678c71700 1 -- 172.21.15.158:0/1902366948 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000b9d0 con 0x7fe67415f1d0 2023-10-16T10:03:23.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.360+0000 7fe678c71700 1 -- 172.21.15.158:0/1902366948 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000bd60 con 0x7fe67415f1d0 2023-10-16T10:03:23.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 -- 172.21.15.158:0/1902366948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 msgr2=0x7fe6740b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:23.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 --2- 172.21.15.158:0/1902366948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740b42d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe6740b3520 tx=0x7fe670009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 -- 172.21.15.158:0/1902366948 shutdown_connections 2023-10-16T10:03:23.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 --2- 172.21.15.158:0/1902366948 >> v2:172.21.15.172:3300/0 conn(0x7fe67415b590 0x7fe6740ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 --2- 172.21.15.158:0/1902366948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740b42d0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 -- 172.21.15.158:0/1902366948 >> 172.21.15.158:0/1902366948 conn(0x7fe67406a8d0 msgr2=0x7fe67406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:23.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 -- 172.21.15.158:0/1902366948 shutdown_connections 2023-10-16T10:03:23.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.361+0000 7fe67bed7700 1 -- 172.21.15.158:0/1902366948 wait complete. 2023-10-16T10:03:23.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.362+0000 7fe67bed7700 1 Processor -- start 2023-10-16T10:03:23.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.362+0000 7fe67bed7700 1 -- start start 2023-10-16T10:03:23.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67bed7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6740cbaf0 0x7fe674157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:23.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67bed7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe67415b590 0x7fe6740c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:23.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67bed7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:23.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67bed7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67415d420 con 0x7fe67415f1d0 2023-10-16T10:03:23.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67bed7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67415ce60 con 0x7fe6740cbaf0 2023-10-16T10:03:23.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67bed7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67415d140 con 0x7fe67415b590 2023-10-16T10:03:23.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:23.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58082/0 (socket says 172.21.15.158:58082) 2023-10-16T10:03:23.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 -- 172.21.15.158:0/2358468576 learned_addr learned my addr 172.21.15.158:0/2358468576 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:23.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe679472700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe67415b590 0x7fe6740c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:23.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe679c73700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6740cbaf0 0x7fe674157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:23.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 -- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe67415b590 msgr2=0x7fe6740c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:23.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe67415b590 0x7fe6740c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 -- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6740cbaf0 msgr2=0x7fe674157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:23.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6740cbaf0 0x7fe674157f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:23.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe67a474700 1 -- 172.21.15.158:0/2358468576 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670009550 con 0x7fe67415f1d0 2023-10-16T10:03:23.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.363+0000 7fe679c73700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6740cbaf0 0x7fe674157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:23.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.364+0000 7fe67a474700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740ca0a0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00cc50 tx=0x7fe66c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:23.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.364+0000 7fe66a7fc700 1 -- 172.21.15.158:0/2358468576 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c003c00 con 0x7fe67415f1d0 2023-10-16T10:03:23.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.364+0000 7fe66a7fc700 1 -- 172.21.15.158:0/2358468576 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c003d60 con 0x7fe67415f1d0 2023-10-16T10:03:23.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.364+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67415bc20 con 0x7fe67415f1d0 2023-10-16T10:03:23.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.364+0000 7fe66a7fc700 1 -- 172.21.15.158:0/2358468576 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c00ab60 con 0x7fe67415f1d0 2023-10-16T10:03:23.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.364+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67406cb10 con 0x7fe67415f1d0 2023-10-16T10:03:23.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.365+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67415c140 con 0x7fe67415f1d0 2023-10-16T10:03:23.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.365+0000 7fe66a7fc700 1 -- 172.21.15.158:0/2358468576 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe66c00acc0 con 0x7fe67415f1d0 2023-10-16T10:03:23.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.366+0000 7fe66a7fc700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe65806b020 0x7fe65806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:23.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.366+0000 7fe679c73700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe65806b020 0x7fe65806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:23.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.366+0000 7fe66a7fc700 1 -- 172.21.15.158:0/2358468576 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6381+0+0 (secure 0 0 0) 0x7fe66c0899d0 con 0x7fe67415f1d0 2023-10-16T10:03:23.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.366+0000 7fe679c73700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe65806b020 0x7fe65806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6640099f0 tx=0x7fe664008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:23.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.371+0000 7fe66a7fc700 1 -- 172.21.15.158:0/2358468576 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe67415c140 con 0x7fe67415f1d0 2023-10-16T10:03:23.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:23.607+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe67415c140 con 0x7fe67415f1d0 2023-10-16T10:03:24.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.203+0000 7fe66a7fc700 1 -- 172.21.15.158:0/2358468576 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v64) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe67415c140 con 0x7fe67415f1d0 2023-10-16T10:03:24.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe65806b020 msgr2=0x7fe65806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe65806b020 0x7fe65806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6640099f0 tx=0x7fe664008040 comp rx=0 tx=0).stop 2023-10-16T10:03:24.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 msgr2=0x7fe6740ca0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740ca0a0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00cc50 tx=0x7fe66c00cf60 comp rx=0 tx=0).stop 2023-10-16T10:03:24.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 shutdown_connections 2023-10-16T10:03:24.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fe65806b020 0x7fe65806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe67415f1d0 0x7fe6740ca0a0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6740cbaf0 0x7fe674157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 --2- 172.21.15.158:0/2358468576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe67415b590 0x7fe6740c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 >> 172.21.15.158:0/2358468576 conn(0x7fe67406a8d0 msgr2=0x7fe6740b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:24.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 shutdown_connections 2023-10-16T10:03:24.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.207+0000 7fe67bed7700 1 -- 172.21.15.158:0/2358468576 wait complete. 2023-10-16T10:03:24.216 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 17 and data pool 18 2023-10-16T10:03:24.228 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:24.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.355+0000 7fb578428700 1 Processor -- start 2023-10-16T10:03:24.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.355+0000 7fb578428700 1 -- start start 2023-10-16T10:03:24.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb578428700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb57006e5d0 0x7fb57006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb578428700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5700729b0 0x7fb570071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb578428700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb570071e50 con 0x7fb57006ab40 2023-10-16T10:03:24.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb578428700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb570161270 con 0x7fb57006e5d0 2023-10-16T10:03:24.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb578428700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb570161510 con 0x7fb5700729b0 2023-10-16T10:03:24.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb5759c3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb57006e5d0 0x7fb57006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb5761c4700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb57006ab40 0x7fb5700722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58266/0 (socket says 172.21.15.158:58266) 2023-10-16T10:03:24.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb5761c4700 1 -- 172.21.15.158:0/4147873116 learned_addr learned my addr 172.21.15.158:0/4147873116 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:24.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.356+0000 7fb5769c5700 1 --2- 172.21.15.158:0/4147873116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5700729b0 0x7fb570071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5759c3700 1 -- 172.21.15.158:0/4147873116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5700729b0 msgr2=0x7fb570071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:24.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5759c3700 1 --2- 172.21.15.158:0/4147873116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5700729b0 0x7fb570071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5759c3700 1 -- 172.21.15.158:0/4147873116 >> v1:172.21.15.158:6789/0 conn(0x7fb57006ab40 legacy=0x7fb5700722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5759c3700 1 -- 172.21.15.158:0/4147873116 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5701617f0 con 0x7fb57006e5d0 2023-10-16T10:03:24.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5751c2700 1 -- 172.21.15.158:0/4147873116 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887004738 0 0) 0x7fb570071e50 con 0x7fb57006ab40 2023-10-16T10:03:24.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5759c3700 1 --2- 172.21.15.158:0/4147873116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb57006e5d0 0x7fb57006d820 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009840 tx=0x7fb56c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d3424653fcabf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:24.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5751c2700 1 -- 172.21.15.158:0/4147873116 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c011830 con 0x7fb57006e5d0 2023-10-16T10:03:24.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5751c2700 1 -- 172.21.15.158:0/4147873116 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c011990 con 0x7fb57006e5d0 2023-10-16T10:03:24.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb5751c2700 1 -- 172.21.15.158:0/4147873116 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c011d20 con 0x7fb57006e5d0 2023-10-16T10:03:24.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb578428700 1 -- 172.21.15.158:0/4147873116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb57006e5d0 msgr2=0x7fb57006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb578428700 1 --2- 172.21.15.158:0/4147873116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb57006e5d0 0x7fb57006d820 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009840 tx=0x7fb56c009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:24.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb578428700 1 -- 172.21.15.158:0/4147873116 shutdown_connections 2023-10-16T10:03:24.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb578428700 1 --2- 172.21.15.158:0/4147873116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5700729b0 0x7fb570071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb578428700 1 --2- 172.21.15.158:0/4147873116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb57006e5d0 0x7fb57006d820 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.357+0000 7fb578428700 1 -- 172.21.15.158:0/4147873116 >> 172.21.15.158:0/4147873116 conn(0x7fb5700d02b0 msgr2=0x7fb5700d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:24.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.358+0000 7fb578428700 1 -- 172.21.15.158:0/4147873116 shutdown_connections 2023-10-16T10:03:24.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.358+0000 7fb578428700 1 -- 172.21.15.158:0/4147873116 wait complete. 2023-10-16T10:03:24.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 Processor -- start 2023-10-16T10:03:24.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 -- start start 2023-10-16T10:03:24.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb57006ab40 0x7fb5700c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb57006e5d0 0x7fb5700c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5700729b0 0x7fb5700ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700c8670 con 0x7fb5700729b0 2023-10-16T10:03:24.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700c9f20 con 0x7fb57006e5d0 2023-10-16T10:03:24.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.359+0000 7fb578428700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700c79f0 con 0x7fb57006ab40 2023-10-16T10:03:24.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5700729b0 0x7fb5700ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5700729b0 0x7fb5700ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58100/0 (socket says 172.21.15.158:58100) 2023-10-16T10:03:24.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 -- 172.21.15.158:0/2468771427 learned_addr learned my addr 172.21.15.158:0/2468771427 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:24.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5761c4700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb57006ab40 0x7fb5700c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 -- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb57006ab40 msgr2=0x7fb5700c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb57006ab40 0x7fb5700c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 -- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb57006e5d0 msgr2=0x7fb5700c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5759c3700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb57006e5d0 0x7fb5700c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb57006e5d0 0x7fb5700c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 -- 172.21.15.158:0/2468771427 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0094a0 con 0x7fb5700729b0 2023-10-16T10:03:24.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5761c4700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb57006ab40 0x7fb5700c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:24.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5759c3700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb57006e5d0 0x7fb5700c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:24.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb5769c5700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5700729b0 0x7fb5700ad5b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb568009fd0 tx=0x7fb56800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:24.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb566ffd700 1 -- 172.21.15.158:0/2468771427 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56800ede0 con 0x7fb5700729b0 2023-10-16T10:03:24.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb566ffd700 1 -- 172.21.15.158:0/2468771427 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb568014420 con 0x7fb5700729b0 2023-10-16T10:03:24.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5700c7cd0 con 0x7fb5700729b0 2023-10-16T10:03:24.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb566ffd700 1 -- 172.21.15.158:0/2468771427 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5680147d0 con 0x7fb5700729b0 2023-10-16T10:03:24.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.360+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5700c8290 con 0x7fb5700729b0 2023-10-16T10:03:24.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.361+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5700c87b0 con 0x7fb5700729b0 2023-10-16T10:03:24.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.362+0000 7fb566ffd700 1 -- 172.21.15.158:0/2468771427 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb568014930 con 0x7fb5700729b0 2023-10-16T10:03:24.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.362+0000 7fb566ffd700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb55406b020 0x7fb55406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.362+0000 7fb566ffd700 1 -- 172.21.15.158:0/2468771427 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fb56808a1b0 con 0x7fb5700729b0 2023-10-16T10:03:24.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.362+0000 7fb5761c4700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb55406b020 0x7fb55406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.363+0000 7fb5761c4700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb55406b020 0x7fb55406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb560000f60 tx=0x7fb560009040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:24.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.367+0000 7fb566ffd700 1 -- 172.21.15.158:0/2468771427 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5700c87b0 con 0x7fb5700729b0 2023-10-16T10:03:24.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.563+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5700c87b0 con 0x7fb5700729b0 2023-10-16T10:03:24.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.564+0000 7fb566ffd700 1 -- 172.21.15.158:0/2468771427 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+17499 (secure 0 0 0) 0x7fb5700c87b0 con 0x7fb5700729b0 2023-10-16T10:03:24.565 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:24.566 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":139,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:23.609224+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:03:21.520809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T10:03:22.576358+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:24.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb55406b020 msgr2=0x7fb55406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb55406b020 0x7fb55406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb560000f60 tx=0x7fb560009040 comp rx=0 tx=0).stop 2023-10-16T10:03:24.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5700729b0 msgr2=0x7fb5700ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5700729b0 0x7fb5700ad5b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb568009fd0 tx=0x7fb56800e3c0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 shutdown_connections 2023-10-16T10:03:24.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb55406b020 0x7fb55406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5700729b0 0x7fb5700ad5b0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb57006ab40 0x7fb5700c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 --2- 172.21.15.158:0/2468771427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb57006e5d0 0x7fb5700c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 >> 172.21.15.158:0/2468771427 conn(0x7fb5700d02b0 msgr2=0x7fb5701576b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:24.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 shutdown_connections 2023-10-16T10:03:24.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.566+0000 7fb578428700 1 -- 172.21.15.158:0/2468771427 wait complete. 2023-10-16T10:03:24.580 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:24.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.700+0000 7f4213eac700 1 Processor -- start 2023-10-16T10:03:24.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.700+0000 7f4213eac700 1 -- start start 2023-10-16T10:03:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4213eac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f420c0b3770 0x7f420c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4213eac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 0x7f420c0cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4213eac700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c15db90 con 0x7f420c15d7b0 2023-10-16T10:03:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4213eac700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c0cfa10 con 0x7f420c0b3770 2023-10-16T10:03:24.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4213eac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c06bea0 con 0x7f420c0cf590 2023-10-16T10:03:24.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4211c48700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f420c15d7b0 0x7f420c161ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58280/0 (socket says 172.21.15.158:58280) 2023-10-16T10:03:24.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4211c48700 1 -- 172.21.15.158:0/2568688980 learned_addr learned my addr 172.21.15.158:0/2568688980 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:24.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4212449700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 0x7f420c0cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4211447700 1 --2- 172.21.15.158:0/2568688980 >> v2:172.21.15.172:3300/0 conn(0x7f420c0b3770 0x7f420c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898050413 0 0) 0x7f420c15db90 con 0x7f420c15d7b0 2023-10-16T10:03:24.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.701+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f80035b0 con 0x7f420c15d7b0 2023-10-16T10:03:24.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3288853696 0 0) 0x7f41f80035b0 con 0x7f420c15d7b0 2023-10-16T10:03:24.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 msgr2=0x7f420c0cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:24.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 0x7f420c0cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 >> v2:172.21.15.172:3300/0 conn(0x7f420c0b3770 msgr2=0x7f420c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:24.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 --2- 172.21.15.158:0/2568688980 >> v2:172.21.15.172:3300/0 conn(0x7f420c0b3770 0x7f420c0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c06c140 con 0x7f420c15d7b0 2023-10-16T10:03:24.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f41fc001670 con 0x7f420c15d7b0 2023-10-16T10:03:24.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4212449700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 0x7f420c0cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:24.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 --2- 172.21.15.158:0/2568688980 >> v2:172.21.15.172:3300/0 conn(0x7f420c0b3770 0x7f420c0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:24.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 >> v1:172.21.15.158:6789/0 conn(0x7f420c15d7b0 legacy=0x7f420c161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8003a00 0x7f41f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41f8004850 0x7f41f8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 0x7f41f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0cfa10 con 0x7f41f8004850 2023-10-16T10:03:24.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c06bea0 con 0x7f41f8003a00 2023-10-16T10:03:24.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f800ba90 con 0x7f420c0cf590 2023-10-16T10:03:24.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41f8004850 0x7f41f8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211c48700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8003a00 0x7f41f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4212449700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 0x7f41f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 -- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 msgr2=0x7f41f8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:24.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c0cf590 0x7f41f8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 -- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8003a00 msgr2=0x7f41f8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:24.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8003a00 0x7f41f8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 -- 172.21.15.158:0/2568688980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4204009a70 con 0x7f41f8004850 2023-10-16T10:03:24.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4211447700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41f8004850 0x7f41f8004cd0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f4208009d00 tx=0x7f41f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2ad0f1c34da011 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:24.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.702+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208009e50 con 0x7f41f8004850 2023-10-16T10:03:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420800e9a0 con 0x7f41f8004850 2023-10-16T10:03:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4210c46700 1 -- 172.21.15.158:0/2568688980 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420800ed50 con 0x7f41f8004850 2023-10-16T10:03:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4213eac700 1 -- 172.21.15.158:0/2568688980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41f8004850 msgr2=0x7f41f8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4213eac700 1 --2- 172.21.15.158:0/2568688980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41f8004850 0x7f41f8004cd0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f4208009d00 tx=0x7f41f8005880 comp rx=0 tx=0).stop 2023-10-16T10:03:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4212449700 1 -- 172.21.15.158:0/2568688980 reap_dead start 2023-10-16T10:03:24.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4213eac700 1 -- 172.21.15.158:0/2568688980 shutdown_connections 2023-10-16T10:03:24.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4213eac700 1 -- 172.21.15.158:0/2568688980 >> 172.21.15.158:0/2568688980 conn(0x7f420c06a8d0 msgr2=0x7f420c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:24.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4213eac700 1 -- 172.21.15.158:0/2568688980 shutdown_connections 2023-10-16T10:03:24.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.703+0000 7f4213eac700 1 -- 172.21.15.158:0/2568688980 wait complete. 2023-10-16T10:03:24.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.704+0000 7f4213eac700 1 Processor -- start 2023-10-16T10:03:24.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.704+0000 7f4213eac700 1 -- start start 2023-10-16T10:03:24.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4213eac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8004850 0x7f420c155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4213eac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c1e7ca0 0x7f420c1e8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4213eac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c15db90 0x7f420c15e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4213eac700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c1e8660 con 0x7f420c1e7ca0 2023-10-16T10:03:24.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4213eac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0c9000 con 0x7f41f8004850 2023-10-16T10:03:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4213eac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0ca120 con 0x7f420c15db90 2023-10-16T10:03:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4211447700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c1e7ca0 0x7f420c1e8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4211447700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c1e7ca0 0x7f420c1e8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58118/0 (socket says 172.21.15.158:58118) 2023-10-16T10:03:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4211447700 1 -- 172.21.15.158:0/3833780553 learned_addr learned my addr 172.21.15.158:0/3833780553 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:24.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4211c48700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8004850 0x7f420c155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.705+0000 7f4212449700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c15db90 0x7f420c15e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4211447700 1 -- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c15db90 msgr2=0x7f420c15e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4211447700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c15db90 0x7f420c15e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4211447700 1 -- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8004850 msgr2=0x7f420c155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4211447700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8004850 0x7f420c155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4211447700 1 -- 172.21.15.158:0/3833780553 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c0b5b30 con 0x7f420c1e7ca0 2023-10-16T10:03:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4211447700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c1e7ca0 0x7f420c1e8120 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f420800e790 tx=0x7f42080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f42027fc700 1 -- 172.21.15.158:0/3833780553 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208020070 con 0x7f420c1e7ca0 2023-10-16T10:03:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f42027fc700 1 -- 172.21.15.158:0/3833780553 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42080045e0 con 0x7f420c1e7ca0 2023-10-16T10:03:24.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420c0c86d0 con 0x7f420c1e7ca0 2023-10-16T10:03:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f42027fc700 1 -- 172.21.15.158:0/3833780553 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208006e60 con 0x7f420c1e7ca0 2023-10-16T10:03:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.706+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c0c8c90 con 0x7f420c1e7ca0 2023-10-16T10:03:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.707+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420c0c8e50 con 0x7f420c1e7ca0 2023-10-16T10:03:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.707+0000 7f42027fc700 1 -- 172.21.15.158:0/3833780553 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4208003af0 con 0x7f420c1e7ca0 2023-10-16T10:03:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.708+0000 7f42027fc700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f41f006b020 0x7f41f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.708+0000 7f42027fc700 1 -- 172.21.15.158:0/3833780553 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f420808ae70 con 0x7f420c1e7ca0 2023-10-16T10:03:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.708+0000 7f4211c48700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f41f006b020 0x7f41f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.708+0000 7f4211c48700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f41f006b020 0x7f41f006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f420c0c9bd0 tx=0x7f41fc007980 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.712+0000 7f42027fc700 1 -- 172.21.15.158:0/3833780553 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4208055b90 con 0x7f420c1e7ca0 2023-10-16T10:03:24.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.907+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f420c0681b0 con 0x7f420c1e7ca0 2023-10-16T10:03:24.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.908+0000 7f42027fc700 1 -- 172.21.15.158:0/3833780553 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17486 (secure 0 0 0) 0x7f42080591b0 con 0x7f420c1e7ca0 2023-10-16T10:03:24.911 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:24.911 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":140,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:24.614282+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:03:21.520809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T10:03:22.576358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:24.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f41f006b020 msgr2=0x7f41f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f41f006b020 0x7f41f006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f420c0c9bd0 tx=0x7f41fc007980 comp rx=0 tx=0).stop 2023-10-16T10:03:24.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c1e7ca0 msgr2=0x7f420c1e8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:24.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c1e7ca0 0x7f420c1e8120 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f420800e790 tx=0x7f42080047c0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 shutdown_connections 2023-10-16T10:03:24.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f41f006b020 0x7f41f006d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c15db90 0x7f420c15e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8004850 0x7f420c155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 --2- 172.21.15.158:0/3833780553 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c1e7ca0 0x7f420c1e8120 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:24.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 >> 172.21.15.158:0/3833780553 conn(0x7f420c06a8d0 msgr2=0x7f420c0c78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:24.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 shutdown_connections 2023-10-16T10:03:24.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:24.911+0000 7f4213eac700 1 -- 172.21.15.158:0/3833780553 wait complete. 2023-10-16T10:03:24.925 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T10:03:25.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.048+0000 7f404c062700 1 Processor -- start 2023-10-16T10:03:25.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.048+0000 7f404c062700 1 -- start start 2023-10-16T10:03:25.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f404c062700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 0x7f40440cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f404c062700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f40440cc010 0x7f404415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f404c062700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40440cd200 con 0x7f404415c2b0 2023-10-16T10:03:25.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f404c062700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40440cc490 con 0x7f40440cc010 2023-10-16T10:03:25.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f404c062700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40440cfca0 con 0x7f40440ce5b0 2023-10-16T10:03:25.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f404a5ff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f404415c2b0 0x7f404415f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58304/0 (socket says 172.21.15.158:58304) 2023-10-16T10:03:25.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f404a5ff700 1 -- 172.21.15.158:0/1286962193 learned_addr learned my addr 172.21.15.158:0/1286962193 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:25.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f40495fd700 1 --2- 172.21.15.158:0/1286962193 >> v2:172.21.15.172:3300/0 conn(0x7f40440cc010 0x7f404415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 0x7f40440cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384851092 0 0) 0x7f40440cd200 con 0x7f404415c2b0 2023-10-16T10:03:25.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.049+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40300035b0 con 0x7f404415c2b0 2023-10-16T10:03:25.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766228165 0 0) 0x7f40300035b0 con 0x7f404415c2b0 2023-10-16T10:03:25.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 msgr2=0x7f40440cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:25.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 0x7f40440cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:25.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 >> v2:172.21.15.172:3300/0 conn(0x7f40440cc010 msgr2=0x7f404415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:25.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 --2- 172.21.15.158:0/1286962193 >> v2:172.21.15.172:3300/0 conn(0x7f40440cc010 0x7f404415bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:25.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404415caa0 con 0x7f404415c2b0 2023-10-16T10:03:25.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f403c0019f0 con 0x7f404415c2b0 2023-10-16T10:03:25.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f40495fd700 1 --2- 172.21.15.158:0/1286962193 >> v2:172.21.15.172:3300/0 conn(0x7f40440cc010 0x7f404415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:25.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 >> v1:172.21.15.158:6789/0 conn(0x7f404415c2b0 legacy=0x7f404415f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:25.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f4030003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 0x7f4030004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4030004d90 0x7f4030005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40440cc490 con 0x7f4030003a00 2023-10-16T10:03:25.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40440cfca0 con 0x7f4030004d90 2023-10-16T10:03:25.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403000ba10 con 0x7f40440ce5b0 2023-10-16T10:03:25.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f4030003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f40495fd700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 0x7f4030004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f404a5ff700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4030004d90 0x7f4030005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4049dfe700 1 -- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 msgr2=0x7f4030004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:25.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440ce5b0 0x7f4030004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:25.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4049dfe700 1 -- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4030004d90 msgr2=0x7f4030005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:25.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4030004d90 0x7f4030005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:25.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4049dfe700 1 -- 172.21.15.158:0/1286962193 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40300035b0 con 0x7f4030003a00 2023-10-16T10:03:25.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f4030003e20 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f4034009b60 tx=0x7f4030005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbe433c1568dc28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:25.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403400e6c0 con 0x7f4030003a00 2023-10-16T10:03:25.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403400e820 con 0x7f4030003a00 2023-10-16T10:03:25.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.050+0000 7f4048dfc700 1 -- 172.21.15.158:0/1286962193 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403400eb90 con 0x7f4030003a00 2023-10-16T10:03:25.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.051+0000 7f404c062700 1 -- 172.21.15.158:0/1286962193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 msgr2=0x7f4030003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:25.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.051+0000 7f404c062700 1 --2- 172.21.15.158:0/1286962193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f4030003e20 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f4034009b60 tx=0x7f4030005880 comp rx=0 tx=0).stop 2023-10-16T10:03:25.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.051+0000 7f404a5ff700 1 -- 172.21.15.158:0/1286962193 reap_dead start 2023-10-16T10:03:25.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.051+0000 7f404c062700 1 -- 172.21.15.158:0/1286962193 shutdown_connections 2023-10-16T10:03:25.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.051+0000 7f404c062700 1 -- 172.21.15.158:0/1286962193 >> 172.21.15.158:0/1286962193 conn(0x7f404406ec00 msgr2=0x7f404406efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:25.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.051+0000 7f404c062700 1 -- 172.21.15.158:0/1286962193 shutdown_connections 2023-10-16T10:03:25.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.051+0000 7f404c062700 1 -- 172.21.15.158:0/1286962193 wait complete. 2023-10-16T10:03:25.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.052+0000 7f404c062700 1 Processor -- start 2023-10-16T10:03:25.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.052+0000 7f404c062700 1 -- start start 2023-10-16T10:03:25.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f404c062700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f40440c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f404c062700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440d0ac0 0x7f40441f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f404c062700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40440d0f40 0x7f40441f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f404c062700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40440b54c0 con 0x7f4030003a00 2023-10-16T10:03:25.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f404c062700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40441f0c60 con 0x7f40440d0f40 2023-10-16T10:03:25.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f404c062700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40440b4560 con 0x7f40440d0ac0 2023-10-16T10:03:25.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f40440c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f40440c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58136/0 (socket says 172.21.15.158:58136) 2023-10-16T10:03:25.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 -- 172.21.15.158:0/1490662121 learned_addr learned my addr 172.21.15.158:0/1490662121 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:25.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f40495fd700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440d0ac0 0x7f40441f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f404a5ff700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40440d0f40 0x7f40441f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 -- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440d0ac0 msgr2=0x7f40441f01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440d0ac0 0x7f40441f01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 -- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40440d0f40 msgr2=0x7f40441f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40440d0f40 0x7f40441f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.053+0000 7f4049dfe700 1 -- 172.21.15.158:0/1490662121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40440b4840 con 0x7f4030003a00 2023-10-16T10:03:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.054+0000 7f4049dfe700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f40440c58e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f4034000c00 tx=0x7f40340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.054+0000 7f403a7fc700 1 -- 172.21.15.158:0/1490662121 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034021070 con 0x7f4030003a00 2023-10-16T10:03:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.054+0000 7f403a7fc700 1 -- 172.21.15.158:0/1490662121 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40340036a0 con 0x7f4030003a00 2023-10-16T10:03:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.054+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40440b4b20 con 0x7f4030003a00 2023-10-16T10:03:25.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.054+0000 7f403a7fc700 1 -- 172.21.15.158:0/1490662121 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403400e6c0 con 0x7f4030003a00 2023-10-16T10:03:25.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.054+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40440b50e0 con 0x7f4030003a00 2023-10-16T10:03:25.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.055+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40440b57e0 con 0x7f4030003a00 2023-10-16T10:03:25.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.055+0000 7f403a7fc700 1 -- 172.21.15.158:0/1490662121 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4034025070 con 0x7f4030003a00 2023-10-16T10:03:25.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.056+0000 7f403a7fc700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f402806b020 0x7f402806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:25.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.056+0000 7f403a7fc700 1 -- 172.21.15.158:0/1490662121 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f403408ada0 con 0x7f4030003a00 2023-10-16T10:03:25.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.056+0000 7f40495fd700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f402806b020 0x7f402806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:25.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.056+0000 7f40495fd700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f402806b020 0x7f402806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40440d1d00 tx=0x7f4040008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:25.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.060+0000 7f403a7fc700 1 -- 172.21.15.158:0/1490662121 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4034055be0 con 0x7f4030003a00 2023-10-16T10:03:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:25.304+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f40440b5600 con 0x7f4030003a00 2023-10-16T10:03:26.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.233+0000 7f403a7fc700 1 -- 172.21.15.158:0/1490662121 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v67) v1 ==== 118+0+0 (secure 0 0 0) 0x7f40440b5600 con 0x7f4030003a00 2023-10-16T10:03:26.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f402806b020 msgr2=0x7f402806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:26.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f402806b020 0x7f402806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40440d1d00 tx=0x7f4040008040 comp rx=0 tx=0).stop 2023-10-16T10:03:26.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 msgr2=0x7f40440c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:26.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f40440c58e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f4034000c00 tx=0x7f40340047c0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 shutdown_connections 2023-10-16T10:03:26.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f402806b020 0x7f402806d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40440d0f40 0x7f40441f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.236+0000 7f404c062700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4030003a00 0x7f40440c58e0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.237+0000 7f404c062700 1 --2- 172.21.15.158:0/1490662121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40440d0ac0 0x7f40441f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.237+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 >> 172.21.15.158:0/1490662121 conn(0x7f404406ec00 msgr2=0x7f40440b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.237+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 shutdown_connections 2023-10-16T10:03:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.237+0000 7f404c062700 1 -- 172.21.15.158:0/1490662121 wait complete. 2023-10-16T10:03:26.254 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T10:03:26.254 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T10:03:26.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.384+0000 7fafca214700 1 Processor -- start 2023-10-16T10:03:26.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.384+0000 7fafca214700 1 -- start start 2023-10-16T10:03:26.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.384+0000 7fafca214700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fafc415b540 0x7fafc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.384+0000 7fafca214700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc415ee60 con 0x7fafc406f180 2023-10-16T10:03:26.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.384+0000 7fafca214700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc415e2e0 con 0x7fafc406afe0 2023-10-16T10:03:26.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.384+0000 7fafca214700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc415b920 con 0x7fafc415b540 2023-10-16T10:03:26.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc2ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fafc406f180 0x7fafc406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58320/0 (socket says 172.21.15.158:58320) 2023-10-16T10:03:26.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc2ffd700 1 -- 172.21.15.158:0/2601274597 learned_addr learned my addr 172.21.15.158:0/2601274597 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:26.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775026407 0 0) 0x7fafc415ee60 con 0x7fafc406f180 2023-10-16T10:03:26.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac0035b0 con 0x7fafc406f180 2023-10-16T10:03:26.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc37fe700 1 --2- 172.21.15.158:0/2601274597 >> v2:172.21.15.172:3300/0 conn(0x7fafc415b540 0x7fafc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3634540872 0 0) 0x7fafac0035b0 con 0x7fafc406f180 2023-10-16T10:03:26.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 >> v2:172.21.15.172:3300/0 conn(0x7fafc415b540 msgr2=0x7fafc406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:26.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 --2- 172.21.15.158:0/2601274597 >> v2:172.21.15.172:3300/0 conn(0x7fafc415b540 0x7fafc406d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 >> v1:172.21.15.172:6789/0 conn(0x7fafc406afe0 legacy=0x7fafc4161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:26.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc415e5c0 con 0x7fafc406f180 2023-10-16T10:03:26.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fafb8001670 con 0x7fafc406f180 2023-10-16T10:03:26.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 >> v1:172.21.15.158:6789/0 conn(0x7fafc406f180 legacy=0x7fafc406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:26.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac003a00 0x7fafac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafac004850 0x7fafac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafac005330 0x7fafac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc415b920 con 0x7fafac004850 2023-10-16T10:03:26.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac00c200 con 0x7fafac003a00 2023-10-16T10:03:26.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac00c3c0 con 0x7fafac005330 2023-10-16T10:03:26.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc2ffd700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafac004850 0x7fafac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc37fe700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac003a00 0x7fafac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.385+0000 7fafc3fff700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafac005330 0x7fafac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc2ffd700 1 -- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafac005330 msgr2=0x7fafac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:26.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc2ffd700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafac005330 0x7fafac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc2ffd700 1 -- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac003a00 msgr2=0x7fafac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:26.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc2ffd700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac003a00 0x7fafac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc3fff700 1 -- 172.21.15.158:0/2601274597 reap_dead start 2023-10-16T10:03:26.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc2ffd700 1 -- 172.21.15.158:0/2601274597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0035b0 con 0x7fafac004850 2023-10-16T10:03:26.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc2ffd700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafac004850 0x7fafac004cd0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c210 tx=0x7fafac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d43b145d948d260 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:26.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8007c10 con 0x7fafac004850 2023-10-16T10:03:26.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb800cdd0 con 0x7fafac004850 2023-10-16T10:03:26.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafc27fc700 1 -- 172.21.15.158:0/2601274597 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8005ba0 con 0x7fafac004850 2023-10-16T10:03:26.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafca214700 1 -- 172.21.15.158:0/2601274597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafac004850 msgr2=0x7fafac004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:26.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafca214700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafac004850 0x7fafac004cd0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c210 tx=0x7fafac005880 comp rx=0 tx=0).stop 2023-10-16T10:03:26.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafca214700 1 -- 172.21.15.158:0/2601274597 shutdown_connections 2023-10-16T10:03:26.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafca214700 1 --2- 172.21.15.158:0/2601274597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafac004850 0x7fafac004cd0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafca214700 1 -- 172.21.15.158:0/2601274597 >> 172.21.15.158:0/2601274597 conn(0x7fafc40d03b0 msgr2=0x7fafc40d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:26.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.386+0000 7fafca214700 1 -- 172.21.15.158:0/2601274597 shutdown_connections 2023-10-16T10:03:26.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.387+0000 7fafca214700 1 -- 172.21.15.158:0/2601274597 wait complete. 2023-10-16T10:03:26.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.388+0000 7fafca214700 1 Processor -- start 2023-10-16T10:03:26.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.388+0000 7fafca214700 1 -- start start 2023-10-16T10:03:26.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafca214700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac004850 0x7fafc414fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafca214700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc41589e0 0x7fafc40b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafca214700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafc40b0ec0 0x7fafc4156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafca214700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40aea10 con 0x7fafc41589e0 2023-10-16T10:03:26.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafca214700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40b1340 con 0x7fafac004850 2023-10-16T10:03:26.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafca214700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40adb40 con 0x7fafc40b0ec0 2023-10-16T10:03:26.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafc2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc41589e0 0x7fafc40b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafc2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc41589e0 0x7fafc40b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58158/0 (socket says 172.21.15.158:58158) 2023-10-16T10:03:26.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.389+0000 7fafc2ffd700 1 -- 172.21.15.158:0/4067881776 learned_addr learned my addr 172.21.15.158:0/4067881776 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:26.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc37fe700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac004850 0x7fafc414fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc2ffd700 1 -- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafc40b0ec0 msgr2=0x7fafc4156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:26.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc3fff700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafc40b0ec0 0x7fafc4156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc2ffd700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafc40b0ec0 0x7fafc4156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc2ffd700 1 -- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac004850 msgr2=0x7fafc414fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:26.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc2ffd700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac004850 0x7fafc414fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc2ffd700 1 -- 172.21.15.158:0/4067881776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40add90 con 0x7fafc41589e0 2023-10-16T10:03:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc37fe700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac004850 0x7fafc414fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc3fff700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafc40b0ec0 0x7fafc4156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafc2ffd700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc41589e0 0x7fafc40b0980 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c800 tx=0x7fafb8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafabfff700 1 -- 172.21.15.158:0/4067881776 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8001630 con 0x7fafc41589e0 2023-10-16T10:03:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafabfff700 1 -- 172.21.15.158:0/4067881776 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb80070d0 con 0x7fafc41589e0 2023-10-16T10:03:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc40ae070 con 0x7fafc41589e0 2023-10-16T10:03:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafabfff700 1 -- 172.21.15.158:0/4067881776 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8007440 con 0x7fafc41589e0 2023-10-16T10:03:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.390+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc40ae630 con 0x7fafc41589e0 2023-10-16T10:03:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.391+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc40ae7f0 con 0x7fafc41589e0 2023-10-16T10:03:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.392+0000 7fafabfff700 1 -- 172.21.15.158:0/4067881776 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fafb80075a0 con 0x7fafc41589e0 2023-10-16T10:03:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.392+0000 7fafabfff700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fafb006b020 0x7fafb006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.392+0000 7fafabfff700 1 -- 172.21.15.158:0/4067881776 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fafb808ae90 con 0x7fafc41589e0 2023-10-16T10:03:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.392+0000 7fafc37fe700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fafb006b020 0x7fafb006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.392+0000 7fafc37fe700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fafb006b020 0x7fafb006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fafb40099c0 tx=0x7fafb4008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.396+0000 7fafabfff700 1 -- 172.21.15.158:0/4067881776 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fafb8055b10 con 0x7fafc41589e0 2023-10-16T10:03:26.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:26.632+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fafc40681b0 con 0x7fafc41589e0 2023-10-16T10:03:27.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.241+0000 7fafabfff700 1 -- 172.21.15.158:0/4067881776 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v68) v1 ==== 122+0+0 (secure 0 0 0) 0x7fafb8059130 con 0x7fafc41589e0 2023-10-16T10:03:27.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fafb006b020 msgr2=0x7fafb006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fafb006b020 0x7fafb006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fafb40099c0 tx=0x7fafb4008040 comp rx=0 tx=0).stop 2023-10-16T10:03:27.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc41589e0 msgr2=0x7fafc40b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc41589e0 0x7fafc40b0980 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c800 tx=0x7fafb8001a00 comp rx=0 tx=0).stop 2023-10-16T10:03:27.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 shutdown_connections 2023-10-16T10:03:27.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fafb006b020 0x7fafb006d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafc40b0ec0 0x7fafc4156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafac004850 0x7fafc414fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 --2- 172.21.15.158:0/4067881776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafc41589e0 0x7fafc40b0980 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 >> 172.21.15.158:0/4067881776 conn(0x7fafc40d03b0 msgr2=0x7fafc406e010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:27.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 shutdown_connections 2023-10-16T10:03:27.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.244+0000 7fafca214700 1 -- 172.21.15.158:0/4067881776 wait complete. 2023-10-16T10:03:27.261 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T10:03:27.261 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:27.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.387+0000 7fd06b0d6700 1 Processor -- start 2023-10-16T10:03:27.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.387+0000 7fd06b0d6700 1 -- start start 2023-10-16T10:03:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd06b0d6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 0x7fd0640723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd06b0d6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd0640af590 0x7fd06406d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd06b0d6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd064072910 con 0x7fd06406dec0 2023-10-16T10:03:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd06b0d6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0640ca7e0 con 0x7fd0640af590 2023-10-16T10:03:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd06b0d6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06406e340 con 0x7fd06415a6c0 2023-10-16T10:03:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd0640af590 0x7fd06406d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd0640af590 0x7fd06406d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58164/0 (socket says 172.21.15.158:58164) 2023-10-16T10:03:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 -- 172.21.15.158:0/2831548694 learned_addr learned my addr 172.21.15.158:0/2831548694 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 -- 172.21.15.158:0/2831548694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 msgr2=0x7fd0640723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 --2- 172.21.15.158:0/2831548694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 0x7fd0640723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 -- 172.21.15.158:0/2831548694 >> v1:172.21.15.172:6789/0 conn(0x7fd06406dec0 legacy=0x7fd06415e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 -- 172.21.15.158:0/2831548694 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0640b1e40 con 0x7fd0640af590 2023-10-16T10:03:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd063fff700 1 --2- 172.21.15.158:0/2831548694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd06406d980 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd0500098f0 tx=0x7fd050009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0c8c40bd683860 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd0637fe700 1 -- 172.21.15.158:0/2831548694 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050004230 con 0x7fd0640af590 2023-10-16T10:03:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd0637fe700 1 -- 172.21.15.158:0/2831548694 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd050004390 con 0x7fd0640af590 2023-10-16T10:03:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.388+0000 7fd0637fe700 1 -- 172.21.15.158:0/2831548694 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500036a0 con 0x7fd0640af590 2023-10-16T10:03:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 -- 172.21.15.158:0/2831548694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 msgr2=0x7fd06406d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/2831548694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd06406d980 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd0500098f0 tx=0x7fd050009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 -- 172.21.15.158:0/2831548694 shutdown_connections 2023-10-16T10:03:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/2831548694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 0x7fd0640723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/2831548694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd06406d980 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 -- 172.21.15.158:0/2831548694 >> 172.21.15.158:0/2831548694 conn(0x7fd0640ccda0 msgr2=0x7fd0640cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 -- 172.21.15.158:0/2831548694 shutdown_connections 2023-10-16T10:03:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.389+0000 7fd06b0d6700 1 -- 172.21.15.158:0/2831548694 wait complete. 2023-10-16T10:03:27.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.390+0000 7fd06b0d6700 1 Processor -- start 2023-10-16T10:03:27.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.390+0000 7fd06b0d6700 1 -- start start 2023-10-16T10:03:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd06b0d6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd06406dec0 0x7fd064154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd06b0d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd064156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd06b0d6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 0x7fd0640c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd06b0d6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0640c3e50 con 0x7fd0640af590 2023-10-16T10:03:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd06b0d6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0640ce570 con 0x7fd06406dec0 2023-10-16T10:03:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd06b0d6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0640ce850 con 0x7fd06415a6c0 2023-10-16T10:03:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd064156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd064156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58172/0 (socket says 172.21.15.158:58172) 2023-10-16T10:03:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 -- 172.21.15.158:0/1947157180 learned_addr learned my addr 172.21.15.158:0/1947157180 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd068e72700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd06406dec0 0x7fd064154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd069673700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 0x7fd0640c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 -- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 msgr2=0x7fd0640c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 0x7fd0640c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 -- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd06406dec0 msgr2=0x7fd064154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd06406dec0 0x7fd064154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd063fff700 1 -- 172.21.15.158:0/1947157180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd050009550 con 0x7fd0640af590 2023-10-16T10:03:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.391+0000 7fd068e72700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd06406dec0 0x7fd064154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.392+0000 7fd063fff700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd064156d00 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fd050004000 tx=0x7fd050004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.392+0000 7fd0617fa700 1 -- 172.21.15.158:0/1947157180 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05001f070 con 0x7fd0640af590 2023-10-16T10:03:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.392+0000 7fd0617fa700 1 -- 172.21.15.158:0/1947157180 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd050003e00 con 0x7fd0640af590 2023-10-16T10:03:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.392+0000 7fd0617fa700 1 -- 172.21.15.158:0/1947157180 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05000b880 con 0x7fd0640af590 2023-10-16T10:03:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.392+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0640ceb30 con 0x7fd0640af590 2023-10-16T10:03:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.392+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd064154e40 con 0x7fd0640af590 2023-10-16T10:03:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.393+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0640681b0 con 0x7fd0640af590 2023-10-16T10:03:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.393+0000 7fd0617fa700 1 -- 172.21.15.158:0/1947157180 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd05000b9e0 con 0x7fd0640af590 2023-10-16T10:03:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.394+0000 7fd0617fa700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fd04806b050 0x7fd04806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.394+0000 7fd0617fa700 1 -- 172.21.15.158:0/1947157180 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fd05008c790 con 0x7fd0640af590 2023-10-16T10:03:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.394+0000 7fd068e72700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fd04806b050 0x7fd04806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.394+0000 7fd068e72700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fd04806b050 0x7fd04806d510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd0580099b0 tx=0x7fd058008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:27.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.398+0000 7fd0617fa700 1 -- 172.21.15.158:0/1947157180 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd050057430 con 0x7fd0640af590 2023-10-16T10:03:27.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.632+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0640b26b0 con 0x7fd0640af590 2023-10-16T10:03:27.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.633+0000 7fd0617fa700 1 -- 172.21.15.158:0/1947157180 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fd05005aa50 con 0x7fd0640af590 2023-10-16T10:03:27.634 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:27.635 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4144904682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4144904682},{"type":"v1","addr":"172.21.15.172:6834","nonce":4144904682}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3562807630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3562807630},{"type":"v1","addr":"172.21.15.158:6836","nonce":3562807630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8805,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/614819185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":614819185},{"type":"v1","addr":"172.21.15.158:6837","nonce":614819185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:03:23.609567+0000","modified":"2023-10-16T10:03:26.633779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/4142092863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4142092863},{"type":"v1","addr":"172.21.15.172:6835","nonce":4142092863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T10:03:27.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fd04806b050 msgr2=0x7fd04806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fd04806b050 0x7fd04806d510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd0580099b0 tx=0x7fd058008040 comp rx=0 tx=0).stop 2023-10-16T10:03:27.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 msgr2=0x7fd064156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd064156d00 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fd050004000 tx=0x7fd050004750 comp rx=0 tx=0).stop 2023-10-16T10:03:27.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 shutdown_connections 2023-10-16T10:03:27.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fd04806b050 0x7fd04806d510 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd06415a6c0 0x7fd0640c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd06406dec0 0x7fd064154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 --2- 172.21.15.158:0/1947157180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0640af590 0x7fd064156d00 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 >> 172.21.15.158:0/1947157180 conn(0x7fd0640ccda0 msgr2=0x7fd064159900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:27.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 shutdown_connections 2023-10-16T10:03:27.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.635+0000 7fd06b0d6700 1 -- 172.21.15.158:0/1947157180 wait complete. 2023-10-16T10:03:27.639 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 68 2023-10-16T10:03:27.648 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:27.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.768+0000 7fb497f0c700 1 Processor -- start 2023-10-16T10:03:27.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.768+0000 7fb497f0c700 1 -- start start 2023-10-16T10:03:27.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb497f0c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb49006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb497f0c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49006e770 con 0x7fb49015dfb0 2023-10-16T10:03:27.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb497f0c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4900cbe00 con 0x7fb4900cc5d0 2023-10-16T10:03:27.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb497f0c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900cf270 con 0x7fb49006fa50 2023-10-16T10:03:27.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb4954a7700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb49015dfb0 0x7fb49006f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58356/0 (socket says 172.21.15.158:58356) 2023-10-16T10:03:27.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb4954a7700 1 -- 172.21.15.158:0/3170091109 learned_addr learned my addr 172.21.15.158:0/3170091109 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:27.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb4964a9700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb49006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253782393 0 0) 0x7fb48c001110 con 0x7fb49015dfb0 2023-10-16T10:03:27.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.769+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47c0035b0 con 0x7fb49015dfb0 2023-10-16T10:03:27.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878850366 0 0) 0x7fb4900cbe00 con 0x7fb4900cc5d0 2023-10-16T10:03:27.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c001110 con 0x7fb4900cc5d0 2023-10-16T10:03:27.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 919131525 0 0) 0x7fb47c0035b0 con 0x7fb49015dfb0 2023-10-16T10:03:27.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 msgr2=0x7fb49006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:27.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb49006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 >> v1:172.21.15.172:6789/0 conn(0x7fb4900cc5d0 legacy=0x7fb4900cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49015e4e0 con 0x7fb49015dfb0 2023-10-16T10:03:27.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fb48c001800 con 0x7fb49015dfb0 2023-10-16T10:03:27.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4964a9700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb49006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:27.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 >> v1:172.21.15.158:6789/0 conn(0x7fb49015dfb0 legacy=0x7fb49006f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb47c003bc0 0x7fb47c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb47c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb47c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900cf270 con 0x7fb47c004850 2023-10-16T10:03:27.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c00bac0 con 0x7fb47c003bc0 2023-10-16T10:03:27.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c00bc80 con 0x7fb49006fa50 2023-10-16T10:03:27.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4954a7700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb47c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb495ca8700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb47c003bc0 0x7fb47c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4964a9700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb47c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4954a7700 1 -- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 msgr2=0x7fb47c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:27.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4954a7700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb47c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4954a7700 1 -- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb47c003bc0 msgr2=0x7fb47c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:27.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4954a7700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb47c003bc0 0x7fb47c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4954a7700 1 -- 172.21.15.158:0/3170091109 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488009a70 con 0x7fb47c004850 2023-10-16T10:03:27.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4964a9700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49006fa50 0x7fb47c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:27.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb4954a7700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb47c004cd0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fb4900cee90 tx=0x7fb47c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec85483092a3757 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:27.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c00cc20 con 0x7fb47c004850 2023-10-16T10:03:27.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c00cd80 con 0x7fb47c004850 2023-10-16T10:03:27.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.770+0000 7fb494ca6700 1 -- 172.21.15.158:0/3170091109 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c005ba0 con 0x7fb47c004850 2023-10-16T10:03:27.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.771+0000 7fb497f0c700 1 -- 172.21.15.158:0/3170091109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 msgr2=0x7fb47c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.771+0000 7fb497f0c700 1 --2- 172.21.15.158:0/3170091109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb47c004cd0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fb4900cee90 tx=0x7fb47c005880 comp rx=0 tx=0).stop 2023-10-16T10:03:27.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.771+0000 7fb4964a9700 1 -- 172.21.15.158:0/3170091109 reap_dead start 2023-10-16T10:03:27.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.771+0000 7fb497f0c700 1 -- 172.21.15.158:0/3170091109 shutdown_connections 2023-10-16T10:03:27.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.771+0000 7fb497f0c700 1 -- 172.21.15.158:0/3170091109 >> 172.21.15.158:0/3170091109 conn(0x7fb4900d02b0 msgr2=0x7fb4900d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:27.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.771+0000 7fb497f0c700 1 -- 172.21.15.158:0/3170091109 shutdown_connections 2023-10-16T10:03:27.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.771+0000 7fb497f0c700 1 -- 172.21.15.158:0/3170091109 wait complete. 2023-10-16T10:03:27.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.772+0000 7fb497f0c700 1 Processor -- start 2023-10-16T10:03:27.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.772+0000 7fb497f0c700 1 -- start start 2023-10-16T10:03:27.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb497f0c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb49015a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb497f0c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49015ca20 0x7fb490152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb497f0c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb49014a410 0x7fb49014a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb497f0c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49014add0 con 0x7fb47c004850 2023-10-16T10:03:27.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb497f0c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49015c590 con 0x7fb49014a410 2023-10-16T10:03:27.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb497f0c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49015cea0 con 0x7fb49015ca20 2023-10-16T10:03:27.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb49015a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb49015a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58190/0 (socket says 172.21.15.158:58190) 2023-10-16T10:03:27.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 -- 172.21.15.158:0/4055540282 learned_addr learned my addr 172.21.15.158:0/4055540282 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:27.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb4954a7700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49015ca20 0x7fb490152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb4964a9700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb49014a410 0x7fb49014a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 -- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49015ca20 msgr2=0x7fb490152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49015ca20 0x7fb490152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 -- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb49014a410 msgr2=0x7fb49014a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb49014a410 0x7fb49014a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.773+0000 7fb495ca8700 1 -- 172.21.15.158:0/4055540282 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49015bc60 con 0x7fb47c004850 2023-10-16T10:03:27.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.774+0000 7fb495ca8700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb49015a190 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fb48000f840 tx=0x7fb48000fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:27.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.774+0000 7fb4867fc700 1 -- 172.21.15.158:0/4055540282 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480005320 con 0x7fb47c004850 2023-10-16T10:03:27.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.774+0000 7fb4867fc700 1 -- 172.21.15.158:0/4055540282 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4800054a0 con 0x7fb47c004850 2023-10-16T10:03:27.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.774+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49015bf40 con 0x7fb47c004850 2023-10-16T10:03:27.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.774+0000 7fb4867fc700 1 -- 172.21.15.158:0/4055540282 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480006aa0 con 0x7fb47c004850 2023-10-16T10:03:27.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.774+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49014d7a0 con 0x7fb47c004850 2023-10-16T10:03:27.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.775+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49015c870 con 0x7fb47c004850 2023-10-16T10:03:27.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.776+0000 7fb4867fc700 1 -- 172.21.15.158:0/4055540282 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb480003680 con 0x7fb47c004850 2023-10-16T10:03:27.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.776+0000 7fb4867fc700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb47406b020 0x7fb47406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:27.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.776+0000 7fb4954a7700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb47406b020 0x7fb47406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:27.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.776+0000 7fb4867fc700 1 -- 172.21.15.158:0/4055540282 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fb480089580 con 0x7fb47c004850 2023-10-16T10:03:27.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.777+0000 7fb4954a7700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb47406b020 0x7fb47406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4900ca4a0 tx=0x7fb48c007000 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:27.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.780+0000 7fb4867fc700 1 -- 172.21.15.158:0/4055540282 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4800542f0 con 0x7fb47c004850 2023-10-16T10:03:27.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.973+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4900681b0 con 0x7fb47c004850 2023-10-16T10:03:27.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.974+0000 7fb4867fc700 1 -- 172.21.15.158:0/4055540282 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17486 (secure 0 0 0) 0x7fb480057910 con 0x7fb47c004850 2023-10-16T10:03:27.976 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:27.976 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":140,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:24.614282+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:03:21.520809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T10:03:22.576358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:27.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb47406b020 msgr2=0x7fb47406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb47406b020 0x7fb47406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4900ca4a0 tx=0x7fb48c007000 comp rx=0 tx=0).stop 2023-10-16T10:03:27.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 msgr2=0x7fb49015a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:27.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb49015a190 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fb48000f840 tx=0x7fb48000fb50 comp rx=0 tx=0).stop 2023-10-16T10:03:27.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 shutdown_connections 2023-10-16T10:03:27.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fb47406b020 0x7fb47406d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb49014a410 0x7fb49014a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb47c004850 0x7fb49015a190 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 --2- 172.21.15.158:0/4055540282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb49015ca20 0x7fb490152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:27.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 >> 172.21.15.158:0/4055540282 conn(0x7fb4900d02b0 msgr2=0x7fb49015aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:27.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 shutdown_connections 2023-10-16T10:03:27.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:27.976+0000 7fb497f0c700 1 -- 172.21.15.158:0/4055540282 wait complete. 2023-10-16T10:03:27.990 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T10:03:27.990 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:03:28.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.110+0000 7fee406f7700 1 Processor -- start 2023-10-16T10:03:28.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.110+0000 7fee406f7700 1 -- start start 2023-10-16T10:03:28.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee406f7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee380c65b0 0x7fee3806f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee406f7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3815d800 con 0x7fee380cae30 2023-10-16T10:03:28.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee406f7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3815dae0 con 0x7fee380ce640 2023-10-16T10:03:28.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee406f7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3815e480 con 0x7fee380c65b0 2023-10-16T10:03:28.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee3dc92700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee380c65b0 0x7fee3806f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee3ec94700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fee380cae30 0x7fee380cb2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58378/0 (socket says 172.21.15.158:58378) 2023-10-16T10:03:28.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee3ec94700 1 -- 172.21.15.158:0/2416898679 learned_addr learned my addr 172.21.15.158:0/2416898679 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:28.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee3dc92700 1 -- 172.21.15.158:0/2416898679 >> v1:172.21.15.172:6789/0 conn(0x7fee380ce640 legacy=0x7fee380cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:28.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee3dc92700 1 -- 172.21.15.158:0/2416898679 >> v1:172.21.15.158:6789/0 conn(0x7fee380cae30 legacy=0x7fee380cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee3dc92700 1 -- 172.21.15.158:0/2416898679 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3815ddc0 con 0x7fee380c65b0 2023-10-16T10:03:28.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.111+0000 7fee3d491700 1 -- 172.21.15.158:0/2416898679 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247931928 0 0) 0x7fee3815d800 con 0x7fee380cae30 2023-10-16T10:03:28.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee3dc92700 1 --2- 172.21.15.158:0/2416898679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee3806f460 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fee34009840 tx=0x7fee34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf5279f5c26e2a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee3d491700 1 -- 172.21.15.158:0/2416898679 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee34004230 con 0x7fee380c65b0 2023-10-16T10:03:28.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee3d491700 1 -- 172.21.15.158:0/2416898679 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee34004390 con 0x7fee380c65b0 2023-10-16T10:03:28.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee3d491700 1 -- 172.21.15.158:0/2416898679 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee340036a0 con 0x7fee380c65b0 2023-10-16T10:03:28.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee406f7700 1 -- 172.21.15.158:0/2416898679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 msgr2=0x7fee3806f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee406f7700 1 --2- 172.21.15.158:0/2416898679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee3806f460 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fee34009840 tx=0x7fee34009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:28.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee406f7700 1 -- 172.21.15.158:0/2416898679 shutdown_connections 2023-10-16T10:03:28.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee406f7700 1 --2- 172.21.15.158:0/2416898679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee3806f460 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee406f7700 1 -- 172.21.15.158:0/2416898679 >> 172.21.15.158:0/2416898679 conn(0x7fee380cc4b0 msgr2=0x7fee380cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:28.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee406f7700 1 -- 172.21.15.158:0/2416898679 shutdown_connections 2023-10-16T10:03:28.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.112+0000 7fee406f7700 1 -- 172.21.15.158:0/2416898679 wait complete. 2023-10-16T10:03:28.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 Processor -- start 2023-10-16T10:03:28.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 -- start start 2023-10-16T10:03:28.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee38139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee380cae30 0x7fee38159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee380ce640 0x7fee381598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee381399c0 con 0x7fee380c65b0 2023-10-16T10:03:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee380b2750 con 0x7fee380cae30 2023-10-16T10:03:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee406f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee380b2a30 con 0x7fee380ce640 2023-10-16T10:03:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee3e493700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee38139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee3e493700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee38139480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58204/0 (socket says 172.21.15.158:58204) 2023-10-16T10:03:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.114+0000 7fee3e493700 1 -- 172.21.15.158:0/1079093280 learned_addr learned my addr 172.21.15.158:0/1079093280 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3ec94700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee380ce640 0x7fee381598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3dc92700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee380cae30 0x7fee38159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3e493700 1 -- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee380ce640 msgr2=0x7fee381598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3e493700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee380ce640 0x7fee381598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3e493700 1 -- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee380cae30 msgr2=0x7fee38159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3e493700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee380cae30 0x7fee38159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3e493700 1 -- 172.21.15.158:0/1079093280 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee340094a0 con 0x7fee380c65b0 2023-10-16T10:03:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3dc92700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee380cae30 0x7fee38159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee3e493700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee38139480 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fee2800c4d0 tx=0x7fee2800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee2effd700 1 -- 172.21.15.158:0/1079093280 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28011790 con 0x7fee380c65b0 2023-10-16T10:03:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee2effd700 1 -- 172.21.15.158:0/1079093280 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee280118f0 con 0x7fee380c65b0 2023-10-16T10:03:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee2effd700 1 -- 172.21.15.158:0/1079093280 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28011ca0 con 0x7fee380c65b0 2023-10-16T10:03:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee380b2d10 con 0x7fee380c65b0 2023-10-16T10:03:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.115+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee380b32d0 con 0x7fee380c65b0 2023-10-16T10:03:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.116+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee380681b0 con 0x7fee380c65b0 2023-10-16T10:03:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.116+0000 7fee2effd700 1 -- 172.21.15.158:0/1079093280 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee2801c460 con 0x7fee380c65b0 2023-10-16T10:03:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.117+0000 7fee2effd700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fee1c06b020 0x7fee1c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.117+0000 7fee2effd700 1 -- 172.21.15.158:0/1079093280 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fee28089910 con 0x7fee380c65b0 2023-10-16T10:03:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.117+0000 7fee3dc92700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fee1c06b020 0x7fee1c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.117+0000 7fee3dc92700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fee1c06b020 0x7fee1c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee34009840 tx=0x7fee34013040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.121+0000 7fee2effd700 1 -- 172.21.15.158:0/1079093280 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee28054630 con 0x7fee380c65b0 2023-10-16T10:03:28.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.311+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 --> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee3815e0a0 con 0x7fee1c06b020 2023-10-16T10:03:28.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.320+0000 7fee2effd700 1 -- 172.21.15.158:0/1079093280 <== mgr.8574 v2:172.21.15.158:6832/2342578387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289771 (secure 0 0 0) 0x7fee34004130 con 0x7fee1c06b020 2023-10-16T10:03:28.324 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:28.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fee1c06b020 msgr2=0x7fee1c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fee1c06b020 0x7fee1c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee34009840 tx=0x7fee34013040 comp rx=0 tx=0).stop 2023-10-16T10:03:28.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 msgr2=0x7fee38139480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee38139480 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fee2800c4d0 tx=0x7fee2800c890 comp rx=0 tx=0).stop 2023-10-16T10:03:28.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 shutdown_connections 2023-10-16T10:03:28.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fee1c06b020 0x7fee1c06d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee380ce640 0x7fee381598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee380c65b0 0x7fee38139480 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 --2- 172.21.15.158:0/1079093280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee380cae30 0x7fee38159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 >> 172.21.15.158:0/1079093280 conn(0x7fee380cc4b0 msgr2=0x7fee38139c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 shutdown_connections 2023-10-16T10:03:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.323+0000 7fee406f7700 1 -- 172.21.15.158:0/1079093280 wait complete. 2023-10-16T10:03:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:03:28.336 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T10:03:27.161506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":347464,"kb_used_data":14724,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754627256,"statfs":{"total":773094113280,"available":772738310144,"internally_reserved":0,"allocated":15077376,"data_stored":8444108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":340577847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.964343"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621495+0000","last_change":"2023-10-16T10:03:23.626314+0000","last_active":"2023-10-16T10:03:24.621495+0000","last_peered":"2023-10-16T10:03:24.621495+0000","last_clean":"2023-10-16T10:03:24.621495+0000","last_became_active":"2023-10-16T10:03:23.626013+0000","last_became_peered":"2023-10-16T10:03:23.626013+0000","last_unstale":"2023-10-16T10:03:24.621495+0000","last_undegraded":"2023-10-16T10:03:24.621495+0000","last_fullsized":"2023-10-16T10:03:24.621495+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:32.563072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621113+0000","last_change":"2023-10-16T10:03:23.632322+0000","last_active":"2023-10-16T10:03:24.621113+0000","last_peered":"2023-10-16T10:03:24.621113+0000","last_clean":"2023-10-16T10:03:24.621113+0000","last_became_active":"2023-10-16T10:03:23.632067+0000","last_became_peered":"2023-10-16T10:03:23.632067+0000","last_unstale":"2023-10-16T10:03:24.621113+0000","last_undegraded":"2023-10-16T10:03:24.621113+0000","last_fullsized":"2023-10-16T10:03:24.621113+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:40.726019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621129+0000","last_change":"2023-10-16T10:03:23.636914+0000","last_active":"2023-10-16T10:03:24.621129+0000","last_peered":"2023-10-16T10:03:24.621129+0000","last_clean":"2023-10-16T10:03:24.621129+0000","last_became_active":"2023-10-16T10:03:23.636515+0000","last_became_peered":"2023-10-16T10:03:23.636515+0000","last_unstale":"2023-10-16T10:03:24.621129+0000","last_undegraded":"2023-10-16T10:03:24.621129+0000","last_fullsized":"2023-10-16T10:03:24.621129+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:50.918218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620787+0000","last_change":"2023-10-16T10:03:23.626396+0000","last_active":"2023-10-16T10:03:24.620787+0000","last_peered":"2023-10-16T10:03:24.620787+0000","last_clean":"2023-10-16T10:03:24.620787+0000","last_became_active":"2023-10-16T10:03:23.626109+0000","last_became_peered":"2023-10-16T10:03:23.626109+0000","last_unstale":"2023-10-16T10:03:24.620787+0000","last_undegraded":"2023-10-16T10:03:24.620787+0000","last_fullsized":"2023-10-16T10:03:24.620787+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:09.163635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621457+0000","last_change":"2023-10-16T10:03:23.628137+0000","last_active":"2023-10-16T10:03:24.621457+0000","last_peered":"2023-10-16T10:03:24.621457+0000","last_clean":"2023-10-16T10:03:24.621457+0000","last_became_active":"2023-10-16T10:03:23.627803+0000","last_became_peered":"2023-10-16T10:03:23.627803+0000","last_unstale":"2023-10-16T10:03:24.621457+0000","last_undegraded":"2023-10-16T10:03:24.621457+0000","last_fullsized":"2023-10-16T10:03:24.621457+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:12.091137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634241+0000","last_change":"2023-10-16T10:03:23.631906+0000","last_active":"2023-10-16T10:03:23.634241+0000","last_peered":"2023-10-16T10:03:23.634241+0000","last_clean":"2023-10-16T10:03:23.634241+0000","last_became_active":"2023-10-16T10:03:23.631623+0000","last_became_peered":"2023-10-16T10:03:23.631623+0000","last_unstale":"2023-10-16T10:03:23.634241+0000","last_undegraded":"2023-10-16T10:03:23.634241+0000","last_fullsized":"2023-10-16T10:03:23.634241+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:23.123139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628826+0000","last_change":"2023-10-16T10:03:23.622121+0000","last_active":"2023-10-16T10:03:23.628826+0000","last_peered":"2023-10-16T10:03:23.628826+0000","last_clean":"2023-10-16T10:03:23.628826+0000","last_became_active":"2023-10-16T10:03:23.621461+0000","last_became_peered":"2023-10-16T10:03:23.621461+0000","last_unstale":"2023-10-16T10:03:23.628826+0000","last_undegraded":"2023-10-16T10:03:23.628826+0000","last_fullsized":"2023-10-16T10:03:23.628826+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:47:10.394652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620513+0000","last_change":"2023-10-16T10:03:23.636649+0000","last_active":"2023-10-16T10:03:24.620513+0000","last_peered":"2023-10-16T10:03:24.620513+0000","last_clean":"2023-10-16T10:03:24.620513+0000","last_became_active":"2023-10-16T10:03:23.636381+0000","last_became_peered":"2023-10-16T10:03:23.636381+0000","last_unstale":"2023-10-16T10:03:24.620513+0000","last_undegraded":"2023-10-16T10:03:24.620513+0000","last_fullsized":"2023-10-16T10:03:24.620513+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:56.100229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620980+0000","last_change":"2023-10-16T10:03:23.632272+0000","last_active":"2023-10-16T10:03:24.620980+0000","last_peered":"2023-10-16T10:03:24.620980+0000","last_clean":"2023-10-16T10:03:24.620980+0000","last_became_active":"2023-10-16T10:03:23.631887+0000","last_became_peered":"2023-10-16T10:03:23.631887+0000","last_unstale":"2023-10-16T10:03:24.620980+0000","last_undegraded":"2023-10-16T10:03:24.620980+0000","last_fullsized":"2023-10-16T10:03:24.620980+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:43.636028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.630233+0000","last_change":"2023-10-16T10:03:23.629557+0000","last_active":"2023-10-16T10:03:23.630233+0000","last_peered":"2023-10-16T10:03:23.630233+0000","last_clean":"2023-10-16T10:03:23.630233+0000","last_became_active":"2023-10-16T10:03:23.629276+0000","last_became_peered":"2023-10-16T10:03:23.629276+0000","last_unstale":"2023-10-16T10:03:23.630233+0000","last_undegraded":"2023-10-16T10:03:23.630233+0000","last_fullsized":"2023-10-16T10:03:23.630233+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:01.347666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621654+0000","last_change":"2023-10-16T10:03:23.635932+0000","last_active":"2023-10-16T10:03:24.621654+0000","last_peered":"2023-10-16T10:03:24.621654+0000","last_clean":"2023-10-16T10:03:24.621654+0000","last_became_active":"2023-10-16T10:03:23.635305+0000","last_became_peered":"2023-10-16T10:03:23.635305+0000","last_unstale":"2023-10-16T10:03:24.621654+0000","last_undegraded":"2023-10-16T10:03:24.621654+0000","last_fullsized":"2023-10-16T10:03:24.621654+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:50.393990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.629011+0000","last_change":"2023-10-16T10:03:23.627578+0000","last_active":"2023-10-16T10:03:23.629011+0000","last_peered":"2023-10-16T10:03:23.629011+0000","last_clean":"2023-10-16T10:03:23.629011+0000","last_became_active":"2023-10-16T10:03:23.627331+0000","last_became_peered":"2023-10-16T10:03:23.627331+0000","last_unstale":"2023-10-16T10:03:23.629011+0000","last_undegraded":"2023-10-16T10:03:23.629011+0000","last_fullsized":"2023-10-16T10:03:23.629011+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:30.812138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.635991+0000","last_change":"2023-10-16T10:03:23.635528+0000","last_active":"2023-10-16T10:03:23.635991+0000","last_peered":"2023-10-16T10:03:23.635991+0000","last_clean":"2023-10-16T10:03:23.635991+0000","last_became_active":"2023-10-16T10:03:23.635253+0000","last_became_peered":"2023-10-16T10:03:23.635253+0000","last_unstale":"2023-10-16T10:03:23.635991+0000","last_undegraded":"2023-10-16T10:03:23.635991+0000","last_fullsized":"2023-10-16T10:03:23.635991+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:44.041713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620651+0000","last_change":"2023-10-16T10:03:23.639551+0000","last_active":"2023-10-16T10:03:24.620651+0000","last_peered":"2023-10-16T10:03:24.620651+0000","last_clean":"2023-10-16T10:03:24.620651+0000","last_became_active":"2023-10-16T10:03:23.639367+0000","last_became_peered":"2023-10-16T10:03:23.639367+0000","last_unstale":"2023-10-16T10:03:24.620651+0000","last_undegraded":"2023-10-16T10:03:24.620651+0000","last_fullsized":"2023-10-16T10:03:24.620651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:19.518196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621198+0000","last_change":"2023-10-16T10:03:23.635736+0000","last_active":"2023-10-16T10:03:24.621198+0000","last_peered":"2023-10-16T10:03:24.621198+0000","last_clean":"2023-10-16T10:03:24.621198+0000","last_became_active":"2023-10-16T10:03:23.635340+0000","last_became_peered":"2023-10-16T10:03:23.635340+0000","last_unstale":"2023-10-16T10:03:24.621198+0000","last_undegraded":"2023-10-16T10:03:24.621198+0000","last_fullsized":"2023-10-16T10:03:24.621198+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:19.051819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628782+0000","last_change":"2023-10-16T10:03:23.625147+0000","last_active":"2023-10-16T10:03:23.628782+0000","last_peered":"2023-10-16T10:03:23.628782+0000","last_clean":"2023-10-16T10:03:23.628782+0000","last_became_active":"2023-10-16T10:03:23.624440+0000","last_became_peered":"2023-10-16T10:03:23.624440+0000","last_unstale":"2023-10-16T10:03:23.628782+0000","last_undegraded":"2023-10-16T10:03:23.628782+0000","last_fullsized":"2023-10-16T10:03:23.628782+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:18.495717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.637635+0000","last_change":"2023-10-16T10:03:23.636735+0000","last_active":"2023-10-16T10:03:23.637635+0000","last_peered":"2023-10-16T10:03:23.637635+0000","last_clean":"2023-10-16T10:03:23.637635+0000","last_became_active":"2023-10-16T10:03:23.636479+0000","last_became_peered":"2023-10-16T10:03:23.636479+0000","last_unstale":"2023-10-16T10:03:23.637635+0000","last_undegraded":"2023-10-16T10:03:23.637635+0000","last_fullsized":"2023-10-16T10:03:23.637635+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:29.501165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621113+0000","last_change":"2023-10-16T10:03:23.632457+0000","last_active":"2023-10-16T10:03:24.621113+0000","last_peered":"2023-10-16T10:03:24.621113+0000","last_clean":"2023-10-16T10:03:24.621113+0000","last_became_active":"2023-10-16T10:03:23.632191+0000","last_became_peered":"2023-10-16T10:03:23.632191+0000","last_unstale":"2023-10-16T10:03:24.621113+0000","last_undegraded":"2023-10-16T10:03:24.621113+0000","last_fullsized":"2023-10-16T10:03:24.621113+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:31.156072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638543+0000","last_change":"2023-10-16T10:03:23.634582+0000","last_active":"2023-10-16T10:03:23.638543+0000","last_peered":"2023-10-16T10:03:23.638543+0000","last_clean":"2023-10-16T10:03:23.638543+0000","last_became_active":"2023-10-16T10:03:23.634316+0000","last_became_peered":"2023-10-16T10:03:23.634316+0000","last_unstale":"2023-10-16T10:03:23.638543+0000","last_undegraded":"2023-10-16T10:03:23.638543+0000","last_fullsized":"2023-10-16T10:03:23.638543+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:29.501165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.625997+0000","last_change":"2023-10-16T10:03:23.624811+0000","last_active":"2023-10-16T10:03:23.625997+0000","last_peered":"2023-10-16T10:03:23.625997+0000","last_clean":"2023-10-16T10:03:23.625997+0000","last_became_active":"2023-10-16T10:03:23.624560+0000","last_became_peered":"2023-10-16T10:03:23.624560+0000","last_unstale":"2023-10-16T10:03:23.625997+0000","last_undegraded":"2023-10-16T10:03:23.625997+0000","last_fullsized":"2023-10-16T10:03:23.625997+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:30.725405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620316+0000","last_change":"2023-10-16T10:03:23.623080+0000","last_active":"2023-10-16T10:03:24.620316+0000","last_peered":"2023-10-16T10:03:24.620316+0000","last_clean":"2023-10-16T10:03:24.620316+0000","last_became_active":"2023-10-16T10:03:23.622799+0000","last_became_peered":"2023-10-16T10:03:23.622799+0000","last_unstale":"2023-10-16T10:03:24.620316+0000","last_undegraded":"2023-10-16T10:03:24.620316+0000","last_fullsized":"2023-10-16T10:03:24.620316+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:32.761688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621541+0000","last_change":"2023-10-16T10:03:23.632108+0000","last_active":"2023-10-16T10:03:24.621541+0000","last_peered":"2023-10-16T10:03:24.621541+0000","last_clean":"2023-10-16T10:03:24.621541+0000","last_became_active":"2023-10-16T10:03:23.631675+0000","last_became_peered":"2023-10-16T10:03:23.631675+0000","last_unstale":"2023-10-16T10:03:24.621541+0000","last_undegraded":"2023-10-16T10:03:24.621541+0000","last_fullsized":"2023-10-16T10:03:24.621541+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:08.266903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620270+0000","last_change":"2023-10-16T10:03:23.632409+0000","last_active":"2023-10-16T10:03:24.620270+0000","last_peered":"2023-10-16T10:03:24.620270+0000","last_clean":"2023-10-16T10:03:24.620270+0000","last_became_active":"2023-10-16T10:03:23.631812+0000","last_became_peered":"2023-10-16T10:03:23.631812+0000","last_unstale":"2023-10-16T10:03:24.620270+0000","last_undegraded":"2023-10-16T10:03:24.620270+0000","last_fullsized":"2023-10-16T10:03:24.620270+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:46:13.229655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.635449+0000","last_change":"2023-10-16T10:03:23.633052+0000","last_active":"2023-10-16T10:03:23.635449+0000","last_peered":"2023-10-16T10:03:23.635449+0000","last_clean":"2023-10-16T10:03:23.635449+0000","last_became_active":"2023-10-16T10:03:23.632701+0000","last_became_peered":"2023-10-16T10:03:23.632701+0000","last_unstale":"2023-10-16T10:03:23.635449+0000","last_undegraded":"2023-10-16T10:03:23.635449+0000","last_fullsized":"2023-10-16T10:03:23.635449+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:28.782584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.640615+0000","last_change":"2023-10-16T10:03:23.638315+0000","last_active":"2023-10-16T10:03:23.640615+0000","last_peered":"2023-10-16T10:03:23.640615+0000","last_clean":"2023-10-16T10:03:23.640615+0000","last_became_active":"2023-10-16T10:03:23.638028+0000","last_became_peered":"2023-10-16T10:03:23.638028+0000","last_unstale":"2023-10-16T10:03:23.640615+0000","last_undegraded":"2023-10-16T10:03:23.640615+0000","last_fullsized":"2023-10-16T10:03:23.640615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:17.739964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.624439+0000","last_change":"2023-10-16T10:03:23.623825+0000","last_active":"2023-10-16T10:03:23.624439+0000","last_peered":"2023-10-16T10:03:23.624439+0000","last_clean":"2023-10-16T10:03:23.624439+0000","last_became_active":"2023-10-16T10:03:23.623498+0000","last_became_peered":"2023-10-16T10:03:23.623498+0000","last_unstale":"2023-10-16T10:03:23.624439+0000","last_undegraded":"2023-10-16T10:03:23.624439+0000","last_fullsized":"2023-10-16T10:03:23.624439+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:33.666814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638561+0000","last_change":"2023-10-16T10:03:23.635272+0000","last_active":"2023-10-16T10:03:23.638561+0000","last_peered":"2023-10-16T10:03:23.638561+0000","last_clean":"2023-10-16T10:03:23.638561+0000","last_became_active":"2023-10-16T10:03:23.635004+0000","last_became_peered":"2023-10-16T10:03:23.635004+0000","last_unstale":"2023-10-16T10:03:23.638561+0000","last_undegraded":"2023-10-16T10:03:23.638561+0000","last_fullsized":"2023-10-16T10:03:23.638561+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:47.998013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.637094+0000","last_change":"2023-10-16T10:03:23.635932+0000","last_active":"2023-10-16T10:03:23.637094+0000","last_peered":"2023-10-16T10:03:23.637094+0000","last_clean":"2023-10-16T10:03:23.637094+0000","last_became_active":"2023-10-16T10:03:23.635667+0000","last_became_peered":"2023-10-16T10:03:23.635667+0000","last_unstale":"2023-10-16T10:03:23.637094+0000","last_undegraded":"2023-10-16T10:03:23.637094+0000","last_fullsized":"2023-10-16T10:03:23.637094+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:40.726019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.632923+0000","last_change":"2023-10-16T10:03:23.632332+0000","last_active":"2023-10-16T10:03:23.632923+0000","last_peered":"2023-10-16T10:03:23.632923+0000","last_clean":"2023-10-16T10:03:23.632923+0000","last_became_active":"2023-10-16T10:03:23.632060+0000","last_became_peered":"2023-10-16T10:03:23.632060+0000","last_unstale":"2023-10-16T10:03:23.632923+0000","last_undegraded":"2023-10-16T10:03:23.632923+0000","last_fullsized":"2023-10-16T10:03:23.632923+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:44.645495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620773+0000","last_change":"2023-10-16T10:03:23.638038+0000","last_active":"2023-10-16T10:03:24.620773+0000","last_peered":"2023-10-16T10:03:24.620773+0000","last_clean":"2023-10-16T10:03:24.620773+0000","last_became_active":"2023-10-16T10:03:23.637786+0000","last_became_peered":"2023-10-16T10:03:23.637786+0000","last_unstale":"2023-10-16T10:03:24.620773+0000","last_undegraded":"2023-10-16T10:03:24.620773+0000","last_fullsized":"2023-10-16T10:03:24.620773+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:03.461277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634520+0000","last_change":"2023-10-16T10:03:23.628135+0000","last_active":"2023-10-16T10:03:23.634520+0000","last_peered":"2023-10-16T10:03:23.634520+0000","last_clean":"2023-10-16T10:03:23.634520+0000","last_became_active":"2023-10-16T10:03:23.627456+0000","last_became_peered":"2023-10-16T10:03:23.627456+0000","last_unstale":"2023-10-16T10:03:23.634520+0000","last_undegraded":"2023-10-16T10:03:23.634520+0000","last_fullsized":"2023-10-16T10:03:23.634520+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:26.591160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638404+0000","last_change":"2023-10-16T10:03:23.634845+0000","last_active":"2023-10-16T10:03:23.638404+0000","last_peered":"2023-10-16T10:03:23.638404+0000","last_clean":"2023-10-16T10:03:23.638404+0000","last_became_active":"2023-10-16T10:03:23.634366+0000","last_became_peered":"2023-10-16T10:03:23.634366+0000","last_unstale":"2023-10-16T10:03:23.638404+0000","last_undegraded":"2023-10-16T10:03:23.638404+0000","last_fullsized":"2023-10-16T10:03:23.638404+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:26.681599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.630270+0000","last_change":"2023-10-16T10:03:23.629617+0000","last_active":"2023-10-16T10:03:23.630270+0000","last_peered":"2023-10-16T10:03:23.630270+0000","last_clean":"2023-10-16T10:03:23.630270+0000","last_became_active":"2023-10-16T10:03:23.629320+0000","last_became_peered":"2023-10-16T10:03:23.629320+0000","last_unstale":"2023-10-16T10:03:23.630270+0000","last_undegraded":"2023-10-16T10:03:23.630270+0000","last_fullsized":"2023-10-16T10:03:23.630270+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:22.800676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"139'4","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215525+0000","last_change":"2023-10-16T10:03:23.222284+0000","last_active":"2023-10-16T10:03:24.215525+0000","last_peered":"2023-10-16T10:03:24.215525+0000","last_clean":"2023-10-16T10:03:24.215525+0000","last_became_active":"2023-10-16T10:03:23.221978+0000","last_became_peered":"2023-10-16T10:03:23.221978+0000","last_unstale":"2023-10-16T10:03:24.215525+0000","last_undegraded":"2023-10-16T10:03:24.215525+0000","last_fullsized":"2023-10-16T10:03:24.215525+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:43.036173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.633935+0000","last_change":"2023-10-16T10:03:23.631704+0000","last_active":"2023-10-16T10:03:23.633935+0000","last_peered":"2023-10-16T10:03:23.633935+0000","last_clean":"2023-10-16T10:03:23.633935+0000","last_became_active":"2023-10-16T10:03:23.631429+0000","last_became_peered":"2023-10-16T10:03:23.631429+0000","last_unstale":"2023-10-16T10:03:23.633935+0000","last_undegraded":"2023-10-16T10:03:23.633935+0000","last_fullsized":"2023-10-16T10:03:23.633935+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:56.100229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.216475+0000","last_change":"2023-10-16T10:03:23.218482+0000","last_active":"2023-10-16T10:03:24.216475+0000","last_peered":"2023-10-16T10:03:24.216475+0000","last_clean":"2023-10-16T10:03:24.216475+0000","last_became_active":"2023-10-16T10:03:23.218249+0000","last_became_peered":"2023-10-16T10:03:23.218249+0000","last_unstale":"2023-10-16T10:03:24.216475+0000","last_undegraded":"2023-10-16T10:03:24.216475+0000","last_fullsized":"2023-10-16T10:03:24.216475+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:19.718159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638500+0000","last_change":"2023-10-16T10:03:23.635344+0000","last_active":"2023-10-16T10:03:23.638500+0000","last_peered":"2023-10-16T10:03:23.638500+0000","last_clean":"2023-10-16T10:03:23.638500+0000","last_became_active":"2023-10-16T10:03:23.634536+0000","last_became_peered":"2023-10-16T10:03:23.634536+0000","last_unstale":"2023-10-16T10:03:23.638500+0000","last_undegraded":"2023-10-16T10:03:23.638500+0000","last_fullsized":"2023-10-16T10:03:23.638500+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:51.573768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"139'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620553+0000","last_change":"2023-10-16T10:03:23.220018+0000","last_active":"2023-10-16T10:03:24.620553+0000","last_peered":"2023-10-16T10:03:24.620553+0000","last_clean":"2023-10-16T10:03:24.620553+0000","last_became_active":"2023-10-16T10:03:23.219746+0000","last_became_peered":"2023-10-16T10:03:23.219746+0000","last_unstale":"2023-10-16T10:03:24.620553+0000","last_undegraded":"2023-10-16T10:03:24.620553+0000","last_fullsized":"2023-10-16T10:03:24.620553+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:27.777043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621127+0000","last_change":"2023-10-16T10:03:23.629254+0000","last_active":"2023-10-16T10:03:24.621127+0000","last_peered":"2023-10-16T10:03:24.621127+0000","last_clean":"2023-10-16T10:03:24.621127+0000","last_became_active":"2023-10-16T10:03:23.628934+0000","last_became_peered":"2023-10-16T10:03:23.628934+0000","last_unstale":"2023-10-16T10:03:24.621127+0000","last_undegraded":"2023-10-16T10:03:24.621127+0000","last_fullsized":"2023-10-16T10:03:24.621127+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:58.758523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.219623+0000","last_change":"2023-10-16T10:03:23.218558+0000","last_active":"2023-10-16T10:03:24.219623+0000","last_peered":"2023-10-16T10:03:24.219623+0000","last_clean":"2023-10-16T10:03:24.219623+0000","last_became_active":"2023-10-16T10:03:23.218285+0000","last_became_peered":"2023-10-16T10:03:23.218285+0000","last_unstale":"2023-10-16T10:03:24.219623+0000","last_undegraded":"2023-10-16T10:03:24.219623+0000","last_fullsized":"2023-10-16T10:03:24.219623+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:38:39.156686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634175+0000","last_change":"2023-10-16T10:03:23.633313+0000","last_active":"2023-10-16T10:03:23.634175+0000","last_peered":"2023-10-16T10:03:23.634175+0000","last_clean":"2023-10-16T10:03:23.634175+0000","last_became_active":"2023-10-16T10:03:23.633027+0000","last_became_peered":"2023-10-16T10:03:23.633027+0000","last_unstale":"2023-10-16T10:03:23.634175+0000","last_undegraded":"2023-10-16T10:03:23.634175+0000","last_fullsized":"2023-10-16T10:03:23.634175+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:03.156368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621561+0000","last_change":"2023-10-16T10:03:23.209306+0000","last_active":"2023-10-16T10:03:24.621561+0000","last_peered":"2023-10-16T10:03:24.621561+0000","last_clean":"2023-10-16T10:03:24.621561+0000","last_became_active":"2023-10-16T10:03:23.209002+0000","last_became_peered":"2023-10-16T10:03:23.209002+0000","last_unstale":"2023-10-16T10:03:24.621561+0000","last_undegraded":"2023-10-16T10:03:24.621561+0000","last_fullsized":"2023-10-16T10:03:24.621561+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:43.036173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621339+0000","last_change":"2023-10-16T10:03:23.628173+0000","last_active":"2023-10-16T10:03:24.621339+0000","last_peered":"2023-10-16T10:03:24.621339+0000","last_clean":"2023-10-16T10:03:24.621339+0000","last_became_active":"2023-10-16T10:03:23.627904+0000","last_became_peered":"2023-10-16T10:03:23.627904+0000","last_unstale":"2023-10-16T10:03:24.621339+0000","last_undegraded":"2023-10-16T10:03:24.621339+0000","last_fullsized":"2023-10-16T10:03:24.621339+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:28:25.599006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.219460+0000","last_change":"2023-10-16T10:03:23.225572+0000","last_active":"2023-10-16T10:03:24.219460+0000","last_peered":"2023-10-16T10:03:24.219460+0000","last_clean":"2023-10-16T10:03:24.219460+0000","last_became_active":"2023-10-16T10:03:23.225284+0000","last_became_peered":"2023-10-16T10:03:23.225284+0000","last_unstale":"2023-10-16T10:03:24.219460+0000","last_undegraded":"2023-10-16T10:03:24.219460+0000","last_fullsized":"2023-10-16T10:03:24.219460+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:27.703147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620879+0000","last_change":"2023-10-16T10:03:23.631008+0000","last_active":"2023-10-16T10:03:24.620879+0000","last_peered":"2023-10-16T10:03:24.620879+0000","last_clean":"2023-10-16T10:03:24.620879+0000","last_became_active":"2023-10-16T10:03:23.630751+0000","last_became_peered":"2023-10-16T10:03:23.630751+0000","last_unstale":"2023-10-16T10:03:24.620879+0000","last_undegraded":"2023-10-16T10:03:24.620879+0000","last_fullsized":"2023-10-16T10:03:24.620879+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:30.812138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.623955+0000","last_change":"2023-10-16T10:03:23.222406+0000","last_active":"2023-10-16T10:03:23.623955+0000","last_peered":"2023-10-16T10:03:23.623955+0000","last_clean":"2023-10-16T10:03:23.623955+0000","last_became_active":"2023-10-16T10:03:23.222092+0000","last_became_peered":"2023-10-16T10:03:23.222092+0000","last_unstale":"2023-10-16T10:03:23.623955+0000","last_undegraded":"2023-10-16T10:03:23.623955+0000","last_fullsized":"2023-10-16T10:03:23.623955+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:07.965428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.633822+0000","last_change":"2023-10-16T10:03:23.631172+0000","last_active":"2023-10-16T10:03:23.633822+0000","last_peered":"2023-10-16T10:03:23.633822+0000","last_clean":"2023-10-16T10:03:23.633822+0000","last_became_active":"2023-10-16T10:03:23.630944+0000","last_became_peered":"2023-10-16T10:03:23.630944+0000","last_unstale":"2023-10-16T10:03:23.633822+0000","last_undegraded":"2023-10-16T10:03:23.633822+0000","last_fullsized":"2023-10-16T10:03:23.633822+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:59.596827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.618857+0000","last_change":"2023-10-16T10:03:23.225774+0000","last_active":"2023-10-16T10:03:23.618857+0000","last_peered":"2023-10-16T10:03:23.618857+0000","last_clean":"2023-10-16T10:03:23.618857+0000","last_became_active":"2023-10-16T10:03:23.225392+0000","last_became_peered":"2023-10-16T10:03:23.225392+0000","last_unstale":"2023-10-16T10:03:23.618857+0000","last_undegraded":"2023-10-16T10:03:23.618857+0000","last_fullsized":"2023-10-16T10:03:23.618857+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:30.150532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.635891+0000","last_change":"2023-10-16T10:03:23.629426+0000","last_active":"2023-10-16T10:03:23.635891+0000","last_peered":"2023-10-16T10:03:23.635891+0000","last_clean":"2023-10-16T10:03:23.635891+0000","last_became_active":"2023-10-16T10:03:23.629160+0000","last_became_peered":"2023-10-16T10:03:23.629160+0000","last_unstale":"2023-10-16T10:03:23.635891+0000","last_undegraded":"2023-10-16T10:03:23.635891+0000","last_fullsized":"2023-10-16T10:03:23.635891+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:20.723699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.629981+0000","last_change":"2023-10-16T10:03:23.229644+0000","last_active":"2023-10-16T10:03:23.629981+0000","last_peered":"2023-10-16T10:03:23.629981+0000","last_clean":"2023-10-16T10:03:23.629981+0000","last_became_active":"2023-10-16T10:03:23.229342+0000","last_became_peered":"2023-10-16T10:03:23.229342+0000","last_unstale":"2023-10-16T10:03:23.629981+0000","last_undegraded":"2023-10-16T10:03:23.629981+0000","last_fullsized":"2023-10-16T10:03:23.629981+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:21.795135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638901+0000","last_change":"2023-10-16T10:03:23.636750+0000","last_active":"2023-10-16T10:03:23.638901+0000","last_peered":"2023-10-16T10:03:23.638901+0000","last_clean":"2023-10-16T10:03:23.638901+0000","last_became_active":"2023-10-16T10:03:23.635328+0000","last_became_peered":"2023-10-16T10:03:23.635328+0000","last_unstale":"2023-10-16T10:03:23.638901+0000","last_undegraded":"2023-10-16T10:03:23.638901+0000","last_fullsized":"2023-10-16T10:03:23.638901+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:32.761688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.623752+0000","last_change":"2023-10-16T10:03:23.220505+0000","last_active":"2023-10-16T10:03:23.623752+0000","last_peered":"2023-10-16T10:03:23.623752+0000","last_clean":"2023-10-16T10:03:23.623752+0000","last_became_active":"2023-10-16T10:03:23.219944+0000","last_became_peered":"2023-10-16T10:03:23.219944+0000","last_unstale":"2023-10-16T10:03:23.623752+0000","last_undegraded":"2023-10-16T10:03:23.623752+0000","last_fullsized":"2023-10-16T10:03:23.623752+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:28:17.749101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621198+0000","last_change":"2023-10-16T10:03:23.639066+0000","last_active":"2023-10-16T10:03:24.621198+0000","last_peered":"2023-10-16T10:03:24.621198+0000","last_clean":"2023-10-16T10:03:24.621198+0000","last_became_active":"2023-10-16T10:03:23.638888+0000","last_became_peered":"2023-10-16T10:03:23.638888+0000","last_unstale":"2023-10-16T10:03:24.621198+0000","last_undegraded":"2023-10-16T10:03:24.621198+0000","last_fullsized":"2023-10-16T10:03:24.621198+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:47:10.394652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"139'3","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215141+0000","last_change":"2023-10-16T10:03:23.220514+0000","last_active":"2023-10-16T10:03:24.215141+0000","last_peered":"2023-10-16T10:03:24.215141+0000","last_clean":"2023-10-16T10:03:24.215141+0000","last_became_active":"2023-10-16T10:03:23.220232+0000","last_became_peered":"2023-10-16T10:03:23.220232+0000","last_unstale":"2023-10-16T10:03:24.215141+0000","last_undegraded":"2023-10-16T10:03:24.215141+0000","last_fullsized":"2023-10-16T10:03:24.215141+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:08.562450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638982+0000","last_change":"2023-10-16T10:03:23.636470+0000","last_active":"2023-10-16T10:03:23.638982+0000","last_peered":"2023-10-16T10:03:23.638982+0000","last_clean":"2023-10-16T10:03:23.638982+0000","last_became_active":"2023-10-16T10:03:23.635901+0000","last_became_peered":"2023-10-16T10:03:23.635901+0000","last_unstale":"2023-10-16T10:03:23.638982+0000","last_undegraded":"2023-10-16T10:03:23.638982+0000","last_fullsized":"2023-10-16T10:03:23.638982+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:03.245724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"139'6","reported_seq":18,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620953+0000","last_change":"2023-10-16T10:03:23.217728+0000","last_active":"2023-10-16T10:03:24.620953+0000","last_peered":"2023-10-16T10:03:24.620953+0000","last_clean":"2023-10-16T10:03:24.620953+0000","last_became_active":"2023-10-16T10:03:23.217471+0000","last_became_peered":"2023-10-16T10:03:23.217471+0000","last_unstale":"2023-10-16T10:03:24.620953+0000","last_undegraded":"2023-10-16T10:03:24.620953+0000","last_fullsized":"2023-10-16T10:03:24.620953+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:57.752982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638491+0000","last_change":"2023-10-16T10:03:23.632706+0000","last_active":"2023-10-16T10:03:23.638491+0000","last_peered":"2023-10-16T10:03:23.638491+0000","last_clean":"2023-10-16T10:03:23.638491+0000","last_became_active":"2023-10-16T10:03:23.632430+0000","last_became_peered":"2023-10-16T10:03:23.632430+0000","last_unstale":"2023-10-16T10:03:23.638491+0000","last_undegraded":"2023-10-16T10:03:23.638491+0000","last_fullsized":"2023-10-16T10:03:23.638491+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:45.192365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"139'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620720+0000","last_change":"2023-10-16T10:03:23.218565+0000","last_active":"2023-10-16T10:03:24.620720+0000","last_peered":"2023-10-16T10:03:24.620720+0000","last_clean":"2023-10-16T10:03:24.620720+0000","last_became_active":"2023-10-16T10:03:23.218301+0000","last_became_peered":"2023-10-16T10:03:23.218301+0000","last_unstale":"2023-10-16T10:03:24.620720+0000","last_undegraded":"2023-10-16T10:03:24.620720+0000","last_fullsized":"2023-10-16T10:03:24.620720+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:42.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.621426+0000","last_change":"2023-10-16T10:03:23.621038+0000","last_active":"2023-10-16T10:03:23.621426+0000","last_peered":"2023-10-16T10:03:23.621426+0000","last_clean":"2023-10-16T10:03:23.621426+0000","last_became_active":"2023-10-16T10:03:23.620775+0000","last_became_peered":"2023-10-16T10:03:23.620775+0000","last_unstale":"2023-10-16T10:03:23.621426+0000","last_undegraded":"2023-10-16T10:03:23.621426+0000","last_fullsized":"2023-10-16T10:03:23.621426+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:26.121933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621015+0000","last_change":"2023-10-16T10:03:23.223444+0000","last_active":"2023-10-16T10:03:24.621015+0000","last_peered":"2023-10-16T10:03:24.621015+0000","last_clean":"2023-10-16T10:03:24.621015+0000","last_became_active":"2023-10-16T10:03:23.223166+0000","last_became_peered":"2023-10-16T10:03:23.223166+0000","last_unstale":"2023-10-16T10:03:24.621015+0000","last_undegraded":"2023-10-16T10:03:24.621015+0000","last_fullsized":"2023-10-16T10:03:24.621015+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:08.158094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628166+0000","last_change":"2023-10-16T10:03:23.626943+0000","last_active":"2023-10-16T10:03:23.628166+0000","last_peered":"2023-10-16T10:03:23.628166+0000","last_clean":"2023-10-16T10:03:23.628166+0000","last_became_active":"2023-10-16T10:03:23.626689+0000","last_became_peered":"2023-10-16T10:03:23.626689+0000","last_unstale":"2023-10-16T10:03:23.628166+0000","last_undegraded":"2023-10-16T10:03:23.628166+0000","last_fullsized":"2023-10-16T10:03:23.628166+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:38.201686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620625+0000","last_change":"2023-10-16T10:03:23.215986+0000","last_active":"2023-10-16T10:03:24.620625+0000","last_peered":"2023-10-16T10:03:24.620625+0000","last_clean":"2023-10-16T10:03:24.620625+0000","last_became_active":"2023-10-16T10:03:23.215714+0000","last_became_peered":"2023-10-16T10:03:23.215714+0000","last_unstale":"2023-10-16T10:03:24.620625+0000","last_undegraded":"2023-10-16T10:03:24.620625+0000","last_fullsized":"2023-10-16T10:03:24.620625+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:42.630487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638379+0000","last_change":"2023-10-16T10:03:23.635325+0000","last_active":"2023-10-16T10:03:23.638379+0000","last_peered":"2023-10-16T10:03:23.638379+0000","last_clean":"2023-10-16T10:03:23.638379+0000","last_became_active":"2023-10-16T10:03:23.635037+0000","last_became_peered":"2023-10-16T10:03:23.635037+0000","last_unstale":"2023-10-16T10:03:23.638379+0000","last_undegraded":"2023-10-16T10:03:23.638379+0000","last_fullsized":"2023-10-16T10:03:23.638379+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:46:13.229655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.216649+0000","last_change":"2023-10-16T10:03:23.226350+0000","last_active":"2023-10-16T10:03:24.216649+0000","last_peered":"2023-10-16T10:03:24.216649+0000","last_clean":"2023-10-16T10:03:24.216649+0000","last_became_active":"2023-10-16T10:03:23.226078+0000","last_became_peered":"2023-10-16T10:03:23.226078+0000","last_unstale":"2023-10-16T10:03:24.216649+0000","last_undegraded":"2023-10-16T10:03:24.216649+0000","last_fullsized":"2023-10-16T10:03:24.216649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:44.186824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621810+0000","last_change":"2023-10-16T10:03:23.631928+0000","last_active":"2023-10-16T10:03:24.621810+0000","last_peered":"2023-10-16T10:03:24.621810+0000","last_clean":"2023-10-16T10:03:24.621810+0000","last_became_active":"2023-10-16T10:03:23.631643+0000","last_became_peered":"2023-10-16T10:03:23.631643+0000","last_unstale":"2023-10-16T10:03:24.621810+0000","last_undegraded":"2023-10-16T10:03:24.621810+0000","last_fullsized":"2023-10-16T10:03:24.621810+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:56.643116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.216208+0000","last_change":"2023-10-16T10:03:23.226061+0000","last_active":"2023-10-16T10:03:24.216208+0000","last_peered":"2023-10-16T10:03:24.216208+0000","last_clean":"2023-10-16T10:03:24.216208+0000","last_became_active":"2023-10-16T10:03:23.225788+0000","last_became_peered":"2023-10-16T10:03:23.225788+0000","last_unstale":"2023-10-16T10:03:24.216208+0000","last_undegraded":"2023-10-16T10:03:24.216208+0000","last_fullsized":"2023-10-16T10:03:24.216208+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:48.345279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628975+0000","last_change":"2023-10-16T10:03:23.628619+0000","last_active":"2023-10-16T10:03:23.628975+0000","last_peered":"2023-10-16T10:03:23.628975+0000","last_clean":"2023-10-16T10:03:23.628975+0000","last_became_active":"2023-10-16T10:03:23.628347+0000","last_became_peered":"2023-10-16T10:03:23.628347+0000","last_unstale":"2023-10-16T10:03:23.628975+0000","last_undegraded":"2023-10-16T10:03:23.628975+0000","last_fullsized":"2023-10-16T10:03:23.628975+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:19.518196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620633+0000","last_change":"2023-10-16T10:03:23.223362+0000","last_active":"2023-10-16T10:03:24.620633+0000","last_peered":"2023-10-16T10:03:24.620633+0000","last_clean":"2023-10-16T10:03:24.620633+0000","last_became_active":"2023-10-16T10:03:23.223083+0000","last_became_peered":"2023-10-16T10:03:23.223083+0000","last_unstale":"2023-10-16T10:03:24.620633+0000","last_undegraded":"2023-10-16T10:03:24.620633+0000","last_fullsized":"2023-10-16T10:03:24.620633+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:25.676058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638848+0000","last_change":"2023-10-16T10:03:23.636289+0000","last_active":"2023-10-16T10:03:23.638848+0000","last_peered":"2023-10-16T10:03:23.638848+0000","last_clean":"2023-10-16T10:03:23.638848+0000","last_became_active":"2023-10-16T10:03:23.635602+0000","last_became_peered":"2023-10-16T10:03:23.635602+0000","last_unstale":"2023-10-16T10:03:23.638848+0000","last_undegraded":"2023-10-16T10:03:23.638848+0000","last_fullsized":"2023-10-16T10:03:23.638848+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:22.363742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620898+0000","last_change":"2023-10-16T10:03:23.229780+0000","last_active":"2023-10-16T10:03:24.620898+0000","last_peered":"2023-10-16T10:03:24.620898+0000","last_clean":"2023-10-16T10:03:24.620898+0000","last_became_active":"2023-10-16T10:03:23.229479+0000","last_became_peered":"2023-10-16T10:03:23.229479+0000","last_unstale":"2023-10-16T10:03:24.620898+0000","last_undegraded":"2023-10-16T10:03:24.620898+0000","last_fullsized":"2023-10-16T10:03:24.620898+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:50.568227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634256+0000","last_change":"2023-10-16T10:03:23.632424+0000","last_active":"2023-10-16T10:03:23.634256+0000","last_peered":"2023-10-16T10:03:23.634256+0000","last_clean":"2023-10-16T10:03:23.634256+0000","last_became_active":"2023-10-16T10:03:23.631635+0000","last_became_peered":"2023-10-16T10:03:23.631635+0000","last_unstale":"2023-10-16T10:03:23.634256+0000","last_undegraded":"2023-10-16T10:03:23.634256+0000","last_fullsized":"2023-10-16T10:03:23.634256+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:03.461277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215829+0000","last_change":"2023-10-16T10:03:23.225699+0000","last_active":"2023-10-16T10:03:24.215829+0000","last_peered":"2023-10-16T10:03:24.215829+0000","last_clean":"2023-10-16T10:03:24.215829+0000","last_became_active":"2023-10-16T10:03:23.225238+0000","last_became_peered":"2023-10-16T10:03:23.225238+0000","last_unstale":"2023-10-16T10:03:24.215829+0000","last_undegraded":"2023-10-16T10:03:24.215829+0000","last_fullsized":"2023-10-16T10:03:24.215829+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:29.577019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620531+0000","last_change":"2023-10-16T10:03:23.635488+0000","last_active":"2023-10-16T10:03:24.620531+0000","last_peered":"2023-10-16T10:03:24.620531+0000","last_clean":"2023-10-16T10:03:24.620531+0000","last_became_active":"2023-10-16T10:03:23.635206+0000","last_became_peered":"2023-10-16T10:03:23.635206+0000","last_unstale":"2023-10-16T10:03:24.620531+0000","last_undegraded":"2023-10-16T10:03:24.620531+0000","last_fullsized":"2023-10-16T10:03:24.620531+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:15.675607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620476+0000","last_change":"2023-10-16T10:03:23.216885+0000","last_active":"2023-10-16T10:03:24.620476+0000","last_peered":"2023-10-16T10:03:24.620476+0000","last_clean":"2023-10-16T10:03:24.620476+0000","last_became_active":"2023-10-16T10:03:23.216614+0000","last_became_peered":"2023-10-16T10:03:23.216614+0000","last_unstale":"2023-10-16T10:03:24.620476+0000","last_undegraded":"2023-10-16T10:03:24.620476+0000","last_fullsized":"2023-10-16T10:03:24.620476+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:14.158582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620914+0000","last_change":"2023-10-16T10:03:23.637515+0000","last_active":"2023-10-16T10:03:24.620914+0000","last_peered":"2023-10-16T10:03:24.620914+0000","last_clean":"2023-10-16T10:03:24.620914+0000","last_became_active":"2023-10-16T10:03:23.637232+0000","last_became_peered":"2023-10-16T10:03:23.637232+0000","last_unstale":"2023-10-16T10:03:24.620914+0000","last_undegraded":"2023-10-16T10:03:24.620914+0000","last_fullsized":"2023-10-16T10:03:24.620914+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:18.495717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620665+0000","last_change":"2023-10-16T10:03:23.229519+0000","last_active":"2023-10-16T10:03:24.620665+0000","last_peered":"2023-10-16T10:03:24.620665+0000","last_clean":"2023-10-16T10:03:24.620665+0000","last_became_active":"2023-10-16T10:03:23.229227+0000","last_became_peered":"2023-10-16T10:03:23.229227+0000","last_unstale":"2023-10-16T10:03:24.620665+0000","last_undegraded":"2023-10-16T10:03:24.620665+0000","last_fullsized":"2023-10-16T10:03:24.620665+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:29.577019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638350+0000","last_change":"2023-10-16T10:03:23.635189+0000","last_active":"2023-10-16T10:03:23.638350+0000","last_peered":"2023-10-16T10:03:23.638350+0000","last_clean":"2023-10-16T10:03:23.638350+0000","last_became_active":"2023-10-16T10:03:23.634919+0000","last_became_peered":"2023-10-16T10:03:23.634919+0000","last_unstale":"2023-10-16T10:03:23.638350+0000","last_undegraded":"2023-10-16T10:03:23.638350+0000","last_fullsized":"2023-10-16T10:03:23.638350+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:23.556655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.619913+0000","last_change":"2023-10-16T10:03:23.220613+0000","last_active":"2023-10-16T10:03:24.619913+0000","last_peered":"2023-10-16T10:03:24.619913+0000","last_clean":"2023-10-16T10:03:24.619913+0000","last_became_active":"2023-10-16T10:03:23.220321+0000","last_became_peered":"2023-10-16T10:03:23.220321+0000","last_unstale":"2023-10-16T10:03:24.619913+0000","last_undegraded":"2023-10-16T10:03:24.619913+0000","last_fullsized":"2023-10-16T10:03:24.619913+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:22.535429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628992+0000","last_change":"2023-10-16T10:03:23.625806+0000","last_active":"2023-10-16T10:03:23.628992+0000","last_peered":"2023-10-16T10:03:23.628992+0000","last_clean":"2023-10-16T10:03:23.628992+0000","last_became_active":"2023-10-16T10:03:23.625558+0000","last_became_peered":"2023-10-16T10:03:23.625558+0000","last_unstale":"2023-10-16T10:03:23.628992+0000","last_undegraded":"2023-10-16T10:03:23.628992+0000","last_fullsized":"2023-10-16T10:03:23.628992+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:50.918218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215833+0000","last_change":"2023-10-16T10:03:23.225473+0000","last_active":"2023-10-16T10:03:24.215833+0000","last_peered":"2023-10-16T10:03:24.215833+0000","last_clean":"2023-10-16T10:03:24.215833+0000","last_became_active":"2023-10-16T10:03:23.225074+0000","last_became_peered":"2023-10-16T10:03:23.225074+0000","last_unstale":"2023-10-16T10:03:24.215833+0000","last_undegraded":"2023-10-16T10:03:24.215833+0000","last_fullsized":"2023-10-16T10:03:24.215833+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:14.670066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.630685+0000","last_change":"2023-10-16T10:03:23.628852+0000","last_active":"2023-10-16T10:03:23.630685+0000","last_peered":"2023-10-16T10:03:23.630685+0000","last_clean":"2023-10-16T10:03:23.630685+0000","last_became_active":"2023-10-16T10:03:23.628584+0000","last_became_peered":"2023-10-16T10:03:23.628584+0000","last_unstale":"2023-10-16T10:03:23.630685+0000","last_undegraded":"2023-10-16T10:03:23.630685+0000","last_fullsized":"2023-10-16T10:03:23.630685+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:21.082401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.631040+0000","last_change":"2023-10-16T10:03:23.229601+0000","last_active":"2023-10-16T10:03:23.631040+0000","last_peered":"2023-10-16T10:03:23.631040+0000","last_clean":"2023-10-16T10:03:23.631040+0000","last_became_active":"2023-10-16T10:03:23.229326+0000","last_became_peered":"2023-10-16T10:03:23.229326+0000","last_unstale":"2023-10-16T10:03:23.631040+0000","last_undegraded":"2023-10-16T10:03:23.631040+0000","last_fullsized":"2023-10-16T10:03:23.631040+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:38:39.156686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620904+0000","last_change":"2023-10-16T10:03:23.625596+0000","last_active":"2023-10-16T10:03:24.620904+0000","last_peered":"2023-10-16T10:03:24.620904+0000","last_clean":"2023-10-16T10:03:24.620904+0000","last_became_active":"2023-10-16T10:03:23.625320+0000","last_became_peered":"2023-10-16T10:03:23.625320+0000","last_unstale":"2023-10-16T10:03:24.620904+0000","last_undegraded":"2023-10-16T10:03:24.620904+0000","last_fullsized":"2023-10-16T10:03:24.620904+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:43.611093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628649+0000","last_change":"2023-10-16T10:03:23.229738+0000","last_active":"2023-10-16T10:03:23.628649+0000","last_peered":"2023-10-16T10:03:23.628649+0000","last_clean":"2023-10-16T10:03:23.628649+0000","last_became_active":"2023-10-16T10:03:23.229413+0000","last_became_peered":"2023-10-16T10:03:23.229413+0000","last_unstale":"2023-10-16T10:03:23.628649+0000","last_undegraded":"2023-10-16T10:03:23.628649+0000","last_fullsized":"2023-10-16T10:03:23.628649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:58.591286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.637772+0000","last_change":"2023-10-16T10:03:23.636879+0000","last_active":"2023-10-16T10:03:23.637772+0000","last_peered":"2023-10-16T10:03:23.637772+0000","last_clean":"2023-10-16T10:03:23.637772+0000","last_became_active":"2023-10-16T10:03:23.636630+0000","last_became_peered":"2023-10-16T10:03:23.636630+0000","last_unstale":"2023-10-16T10:03:23.637772+0000","last_undegraded":"2023-10-16T10:03:23.637772+0000","last_fullsized":"2023-10-16T10:03:23.637772+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:07:28.068394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.619969+0000","last_change":"2023-10-16T10:03:23.225477+0000","last_active":"2023-10-16T10:03:23.619969+0000","last_peered":"2023-10-16T10:03:23.619969+0000","last_clean":"2023-10-16T10:03:23.619969+0000","last_became_active":"2023-10-16T10:03:23.225189+0000","last_became_peered":"2023-10-16T10:03:23.225189+0000","last_unstale":"2023-10-16T10:03:23.619969+0000","last_undegraded":"2023-10-16T10:03:23.619969+0000","last_fullsized":"2023-10-16T10:03:23.619969+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:53.306782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628803+0000","last_change":"2023-10-16T10:03:23.627010+0000","last_active":"2023-10-16T10:03:23.628803+0000","last_peered":"2023-10-16T10:03:23.628803+0000","last_clean":"2023-10-16T10:03:23.628803+0000","last_became_active":"2023-10-16T10:03:23.626759+0000","last_became_peered":"2023-10-16T10:03:23.626759+0000","last_unstale":"2023-10-16T10:03:23.628803+0000","last_undegraded":"2023-10-16T10:03:23.628803+0000","last_fullsized":"2023-10-16T10:03:23.628803+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:43.940745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621650+0000","last_change":"2023-10-16T10:03:23.219621+0000","last_active":"2023-10-16T10:03:24.621650+0000","last_peered":"2023-10-16T10:03:24.621650+0000","last_clean":"2023-10-16T10:03:24.621650+0000","last_became_active":"2023-10-16T10:03:23.219193+0000","last_became_peered":"2023-10-16T10:03:23.219193+0000","last_unstale":"2023-10-16T10:03:24.621650+0000","last_undegraded":"2023-10-16T10:03:24.621650+0000","last_fullsized":"2023-10-16T10:03:24.621650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:21.795135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620580+0000","last_change":"2023-10-16T10:03:23.636431+0000","last_active":"2023-10-16T10:03:24.620580+0000","last_peered":"2023-10-16T10:03:24.620580+0000","last_clean":"2023-10-16T10:03:24.620580+0000","last_became_active":"2023-10-16T10:03:23.636170+0000","last_became_peered":"2023-10-16T10:03:23.636170+0000","last_unstale":"2023-10-16T10:03:24.620580+0000","last_undegraded":"2023-10-16T10:03:24.620580+0000","last_fullsized":"2023-10-16T10:03:24.620580+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:43.940745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621640+0000","last_change":"2023-10-16T10:03:23.214188+0000","last_active":"2023-10-16T10:03:24.621640+0000","last_peered":"2023-10-16T10:03:24.621640+0000","last_clean":"2023-10-16T10:03:24.621640+0000","last_became_active":"2023-10-16T10:03:23.213934+0000","last_became_peered":"2023-10-16T10:03:23.213934+0000","last_unstale":"2023-10-16T10:03:24.621640+0000","last_undegraded":"2023-10-16T10:03:24.621640+0000","last_fullsized":"2023-10-16T10:03:24.621640+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:38:39.156686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620417+0000","last_change":"2023-10-16T10:03:23.623294+0000","last_active":"2023-10-16T10:03:24.620417+0000","last_peered":"2023-10-16T10:03:24.620417+0000","last_clean":"2023-10-16T10:03:24.620417+0000","last_became_active":"2023-10-16T10:03:23.623038+0000","last_became_peered":"2023-10-16T10:03:23.623038+0000","last_unstale":"2023-10-16T10:03:24.620417+0000","last_undegraded":"2023-10-16T10:03:24.620417+0000","last_fullsized":"2023-10-16T10:03:24.620417+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:52.774556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.619972+0000","last_change":"2023-10-16T10:03:23.224056+0000","last_active":"2023-10-16T10:03:23.619972+0000","last_peered":"2023-10-16T10:03:23.619972+0000","last_clean":"2023-10-16T10:03:23.619972+0000","last_became_active":"2023-10-16T10:03:23.223772+0000","last_became_peered":"2023-10-16T10:03:23.223772+0000","last_unstale":"2023-10-16T10:03:23.619972+0000","last_undegraded":"2023-10-16T10:03:23.619972+0000","last_fullsized":"2023-10-16T10:03:23.619972+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:37.765844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.624703+0000","last_change":"2023-10-16T10:03:23.621851+0000","last_active":"2023-10-16T10:03:23.624703+0000","last_peered":"2023-10-16T10:03:23.624703+0000","last_clean":"2023-10-16T10:03:23.624703+0000","last_became_active":"2023-10-16T10:03:23.621575+0000","last_became_peered":"2023-10-16T10:03:23.621575+0000","last_unstale":"2023-10-16T10:03:23.624703+0000","last_undegraded":"2023-10-16T10:03:23.624703+0000","last_fullsized":"2023-10-16T10:03:23.624703+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:47.998013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.619913+0000","last_change":"2023-10-16T10:03:23.222546+0000","last_active":"2023-10-16T10:03:24.619913+0000","last_peered":"2023-10-16T10:03:24.619913+0000","last_clean":"2023-10-16T10:03:24.619913+0000","last_became_active":"2023-10-16T10:03:23.222265+0000","last_became_peered":"2023-10-16T10:03:23.222265+0000","last_unstale":"2023-10-16T10:03:24.619913+0000","last_undegraded":"2023-10-16T10:03:24.619913+0000","last_fullsized":"2023-10-16T10:03:24.619913+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:49:51.860147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620343+0000","last_change":"2023-10-16T10:03:23.623268+0000","last_active":"2023-10-16T10:03:24.620343+0000","last_peered":"2023-10-16T10:03:24.620343+0000","last_clean":"2023-10-16T10:03:24.620343+0000","last_became_active":"2023-10-16T10:03:23.622806+0000","last_became_peered":"2023-10-16T10:03:23.622806+0000","last_unstale":"2023-10-16T10:03:24.620343+0000","last_undegraded":"2023-10-16T10:03:24.620343+0000","last_fullsized":"2023-10-16T10:03:24.620343+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:09.567991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"139'8","reported_seq":20,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620528+0000","last_change":"2023-10-16T10:03:23.218690+0000","last_active":"2023-10-16T10:03:24.620528+0000","last_peered":"2023-10-16T10:03:24.620528+0000","last_clean":"2023-10-16T10:03:24.620528+0000","last_became_active":"2023-10-16T10:03:23.218391+0000","last_became_peered":"2023-10-16T10:03:23.218391+0000","last_unstale":"2023-10-16T10:03:24.620528+0000","last_undegraded":"2023-10-16T10:03:24.620528+0000","last_fullsized":"2023-10-16T10:03:24.620528+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:51.769015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"129'227","reported_seq":673,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.617552+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:03:23.617552+0000","last_peered":"2023-10-16T10:03:23.617552+0000","last_clean":"2023-10-16T10:03:23.617552+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:03:23.617552+0000","last_undegraded":"2023-10-16T10:03:23.617552+0000","last_fullsized":"2023-10-16T10:03:23.617552+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59824,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312016,"statfs":{"total":96636764160,"available":96575504384,"internally_reserved":0,"allocated":2322432,"data_stored":1496900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.25900000000000001},"max":{"1min":0.747,"5min":0.76500000000000001,"15min":1.2150000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.378,"5min":0.33500000000000002,"15min":0.254},"max":{"1min":0.745,"5min":1.911,"15min":1.911},"last":0.66600000000000004}]},{"osd":1,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.60099999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.29199999999999998},"max":{"1min":0.76300000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.41499999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78200000000000003,"5min":0.80500000000000005,"15min":1.1279999999999999},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.624,"15min":0.63100000000000001},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78800000000000003,"5min":3.3820000000000001,"15min":3.3820000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80400000000000005,"5min":3.1640000000000001,"15min":3.1640000000000001},"last":0.622}]},{"osd":3,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.373,"5min":0.33800000000000002,"15min":0.32300000000000001},"max":{"1min":0.81499999999999995,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83099999999999996,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32800000000000001,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.752,"5min":1.8220000000000001,"15min":1.9550000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.55300000000000005,"15min":0.52300000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27200000000000002},"max":{"1min":0.69099999999999995,"5min":3.8149999999999999,"15min":3.8149999999999999},"last":0.37}]},{"osd":5,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.84099999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.37,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":1.6619999999999999,"15min":2.0150000000000001},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84599999999999997,"5min":1.7110000000000001,"15min":1.9870000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.42299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575085,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1732608,"data_stored":907114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55600000000000005,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.29699999999999999,"15min":0.223},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.0920000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.43099999999999999,"5min":0.30599999999999999,"15min":0.249},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.82099999999999995},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71799999999999997,"5min":0.83699999999999997,"15min":0.90000000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.78200000000000003,"15min":1.448},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.60299999999999998},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":0.874,"15min":1.708},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58299999999999996,"15min":0.58799999999999997},"min":{"1min":0.39000000000000001,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":0.71799999999999997,"5min":0.79700000000000004,"15min":0.79900000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56599999999999995,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.36899999999999999,"15min":0.315},"max":{"1min":0.78600000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.47899999999999998},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.249},"max":{"1min":0.754,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.51300000000000001},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.27500000000000002},"max":{"1min":0.80800000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.505,"15min":0.503},"min":{"1min":0.33900000000000002,"5min":0.308,"15min":0.253},"max":{"1min":0.73199999999999998,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.311},"max":{"1min":0.79400000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.37,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.91900000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.42799999999999999,"5min":0.35799999999999998,"15min":0.33800000000000002},"max":{"1min":0.79200000000000004,"5min":0.80500000000000005,"15min":0.94599999999999995},"last":0.52400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1732608,"data_stored":906554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36099999999999999,"15min":0.39100000000000001},"min":{"1min":0.28000000000000003,"5min":0.251,"15min":0.188},"max":{"1min":0.504,"5min":0.54000000000000004,"15min":1.6000000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41399999999999998,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.23200000000000001},"max":{"1min":0.61599999999999999,"5min":0.62,"15min":0.70199999999999996},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.376,"15min":0.40799999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.20799999999999999},"max":{"1min":0.496,"5min":0.98699999999999999,"15min":4.923},"last":0.312},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54000000000000004,"5min":0.65800000000000003,"15min":1.712},"last":0.28000000000000003}]},{"osd":3,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.438,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65700000000000003,"5min":0.68999999999999995,"15min":1.984},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41699999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.26500000000000001,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.623},"min":{"1min":0.42899999999999999,"5min":0.39900000000000002,"15min":0.377},"max":{"1min":0.78200000000000003,"5min":0.87,"15min":0.93100000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.34399999999999997},"max":{"1min":0.73899999999999999,"5min":1.0840000000000001,"15min":5.0069999999999997},"last":0.502}]},{"osd":5,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.623,"15min":0.65800000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84799999999999998,"5min":1.04,"15min":4.9749999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63200000000000001,"15min":0.66700000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.751,"5min":1.054,"15min":1.7769999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61099999999999999,"15min":0.63700000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.90800000000000003,"5min":0.95199999999999996,"15min":1.0009999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.879,"5min":1.3029999999999999,"15min":5.0419999999999998},"last":0.42199999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61099999999999999,"15min":0.65500000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84699999999999998,"5min":0.96599999999999997,"15min":1.744},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.622,"15min":0.64600000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.83599999999999997,"5min":0.91100000000000003,"15min":0.91400000000000003},"last":0.47199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37616,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334224,"statfs":{"total":96636764160,"available":96598245376,"internally_reserved":0,"allocated":2326528,"data_stored":1496922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.38300000000000001},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.192},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":1.2110000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.42999999999999999},"min":{"1min":0.28399999999999997,"5min":0.27600000000000002,"15min":0.26500000000000001},"max":{"1min":0.55900000000000005,"5min":0.70999999999999996,"15min":1.3169999999999999},"last":0.439}]},{"osd":2,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40699999999999997,"15min":0.41699999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.70099999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.44,"15min":0.46000000000000002},"min":{"1min":0.32700000000000001,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":0.61199999999999999,"5min":0.78100000000000003,"15min":0.79900000000000004},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.40000000000000002,"15min":0.41599999999999998},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.83399999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.44500000000000001,"15min":0.46400000000000002},"min":{"1min":0.29399999999999998,"5min":0.27700000000000002,"15min":0.23300000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":1.1859999999999999},"last":0.499}]},{"osd":4,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60899999999999999,"15min":0.621},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.33700000000000002},"max":{"1min":0.83999999999999997,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60199999999999998,"15min":0.629},"min":{"1min":0.378,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":1.1659999999999999},"last":0.86499999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.39200000000000002,"15min":0.33300000000000002},"max":{"1min":0.86499999999999999,"5min":0.94999999999999996,"15min":0.99299999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.629},"min":{"1min":0.438,"5min":0.41799999999999998,"15min":0.311},"max":{"1min":0.85799999999999998,"5min":0.89100000000000001,"15min":0.91000000000000003},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.40899999999999997},"max":{"1min":0.88,"5min":0.90900000000000003,"15min":1.2410000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.66700000000000004},"min":{"1min":0.45000000000000001,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.86599999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.71699999999999997}]},{"osd":7,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.316},"max":{"1min":0.89200000000000002,"5min":1.014,"15min":1.02},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.64300000000000002},"min":{"1min":0.41299999999999998,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.90900000000000003},"last":0.79300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575085,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59236,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312604,"statfs":{"total":96636764160,"available":96576106496,"internally_reserved":0,"allocated":1728512,"data_stored":906532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.35899999999999999,"15min":0.374},"min":{"1min":0.25,"5min":0.217,"15min":0.217},"max":{"1min":0.51100000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.255},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.41199999999999998},"min":{"1min":0.26500000000000001,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.60099999999999998,"5min":0.61499999999999999,"15min":0.65400000000000003},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41299999999999998,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.754},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.38300000000000001,"15min":0.39400000000000002},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.22700000000000001},"max":{"1min":0.57099999999999995,"5min":0.624,"15min":0.624},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40200000000000002,"15min":0.41399999999999998},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.23499999999999999},"max":{"1min":0.50900000000000001,"5min":0.57599999999999996,"15min":0.63500000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.47699999999999998},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.25600000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.79800000000000004},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60699999999999998,"15min":0.63600000000000001},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36799999999999999},"max":{"1min":0.86699999999999999,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.33400000000000002},"max":{"1min":0.83199999999999996,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.41599999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83999999999999997,"5min":1.0249999999999999,"15min":1.103},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.622,"15min":0.64700000000000002},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.38400000000000001},"max":{"1min":0.75900000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.42099999999999999,"5min":0.39900000000000002,"15min":0.39800000000000002},"max":{"1min":0.83899999999999997,"5min":1.046,"15min":1.1699999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.63400000000000001,"15min":0.66600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76200000000000001,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65400000000000003,"15min":0.67700000000000005},"min":{"1min":0.51500000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79700000000000004,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.42599999999999999,"15min":0.371},"max":{"1min":0.79500000000000004,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.56899999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575085,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1744896,"data_stored":908178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34999999999999998,"15min":0.36899999999999999},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.17999999999999999},"max":{"1min":0.5,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.375,"15min":0.38800000000000001},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.45800000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.311}]},{"osd":1,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.39100000000000001},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.65400000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51100000000000001,"5min":0.60899999999999999,"15min":0.67500000000000004},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44,"15min":0.46200000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.53400000000000003,"5min":0.96099999999999997,"15min":1.0760000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.48699999999999999},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.56100000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.77900000000000003,"5min":0.90300000000000002,"15min":0.96499999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60499999999999998,"15min":0.63300000000000001},"min":{"1min":0.432,"5min":0.39600000000000002,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.63100000000000001},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.875,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.498},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.623,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.436}]},{"osd":6,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60299999999999998,"15min":0.64000000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.374},"max":{"1min":0.75900000000000001,"5min":0.88100000000000001,"15min":0.98799999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60699999999999998,"15min":0.64800000000000002},"min":{"1min":0.438,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84299999999999997,"5min":0.92600000000000005,"15min":1.0589999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63800000000000001,"15min":0.66400000000000003},"min":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.878,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.734,"5min":0.83799999999999997,"15min":0.72399999999999998},"min":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.36899999999999999},"max":{"1min":24.861000000000001,"5min":24.861000000000001,"15min":24.861000000000001},"last":0.56299999999999994}]}]},{"osd":4,"up_from":13,"seq":55834575085,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1753088,"data_stored":915216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":1.0269999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57399999999999995,"15min":0.58999999999999997},"min":{"1min":0.433,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":1.0860000000000001},"last":0.626}]},{"osd":1,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.58699999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.312},"max":{"1min":0.753,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56200000000000006,"15min":0.58499999999999996},"min":{"1min":0.44600000000000001,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":1.6359999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.85199999999999998,"15min":0.88300000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59399999999999997,"15min":0.61399999999999999},"min":{"1min":0.46899999999999997,"5min":0.38600000000000001,"15min":0.32500000000000001},"max":{"1min":0.85099999999999998,"5min":0.90500000000000003,"15min":1.2270000000000001},"last":0.503}]},{"osd":3,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.68300000000000005,"5min":0.77700000000000002,"15min":0.86099999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59799999999999998,"15min":0.61199999999999999},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.34200000000000003},"max":{"1min":0.83999999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.49299999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.39900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.69699999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.31},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51400000000000001,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.26400000000000001,"15min":0.26300000000000001},"max":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.96599999999999997},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53000000000000003},"min":{"1min":0.44500000000000001,"5min":0.29599999999999999,"15min":0.24199999999999999},"max":{"1min":0.69799999999999995,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.45000000000000001,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":0.79200000000000004,"5min":0.84899999999999998,"15min":0.92400000000000004},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.82399999999999995,"5min":0.83099999999999996,"15min":0.91100000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.625,"15min":0.61499999999999999},"min":{"1min":0.38900000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78900000000000003,"5min":0.89400000000000002,"15min":1.0429999999999999},"last":0.38700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50932,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320908,"statfs":{"total":96636764160,"available":96584609792,"internally_reserved":0,"allocated":1736704,"data_stored":906692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28599999999999998},"max":{"1min":0.81000000000000005,"5min":0.84199999999999997,"15min":1.792},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.34599999999999997,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.79600000000000004,"5min":0.85799999999999998,"15min":1.7090000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":0.70099999999999996,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.59999999999999998}]},{"osd":2,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25},"max":{"1min":0.753,"5min":0.80800000000000005,"15min":1.496},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.59599999999999997},"min":{"1min":0.497,"5min":0.39200000000000002,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.879,"15min":2.069},"last":0.63100000000000001}]},{"osd":3,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1,"5min":1,"15min":1},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.33000000000000002},"max":{"1min":0.79800000000000004,"5min":0.89100000000000001,"15min":1.7689999999999999},"last":0.65100000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.27600000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.84199999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.50700000000000001},"min":{"1min":0.40000000000000002,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.86899999999999999},"last":0.38100000000000001}]},{"osd":6,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.33300000000000002,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":0.83199999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.88200000000000001},"last":0.49099999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.25900000000000001},"max":{"1min":0.81599999999999995,"5min":0.82399999999999995,"15min":0.90300000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.624,"15min":0.61599999999999999},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.88,"5min":0.88,"15min":0.88500000000000001},"last":0.47299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:03:28.343 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:03:28.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.497+0000 7f4d516fb700 1 Processor -- start 2023-10-16T10:03:28.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.497+0000 7f4d516fb700 1 -- start start 2023-10-16T10:03:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d516fb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4d4c0b6e10 0x7f4d4c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d516fb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c15c5d0 con 0x7f4d4c0cdee0 2023-10-16T10:03:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d516fb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c15c8b0 con 0x7f4d4c0ce2c0 2023-10-16T10:03:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d516fb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4c15d2c0 con 0x7f4d4c0b6e10 2023-10-16T10:03:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d4affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4d4c0cdee0 0x7f4d4c15bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58388/0 (socket says 172.21.15.158:58388) 2023-10-16T10:03:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d4affd700 1 -- 172.21.15.158:0/4186441976 learned_addr learned my addr 172.21.15.158:0/4186441976 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d4a7fc700 1 --2- 172.21.15.158:0/4186441976 >> v2:172.21.15.172:3300/0 conn(0x7f4d4c0b6e10 0x7f4d4c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.498+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151980026 0 0) 0x7f4d4c15c5d0 con 0x7f4d4c0cdee0 2023-10-16T10:03:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d340035b0 con 0x7f4d4c0cdee0 2023-10-16T10:03:28.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066328276 0 0) 0x7f4d340035b0 con 0x7f4d4c0cdee0 2023-10-16T10:03:28.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 >> v2:172.21.15.172:3300/0 conn(0x7f4d4c0b6e10 msgr2=0x7f4d4c1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:28.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 --2- 172.21.15.158:0/4186441976 >> v2:172.21.15.172:3300/0 conn(0x7f4d4c0b6e10 0x7f4d4c1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 >> v1:172.21.15.172:6789/0 conn(0x7f4d4c0ce2c0 legacy=0x7f4d4c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c15cb90 con 0x7f4d4c0cdee0 2023-10-16T10:03:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4d40001670 con 0x7f4d4c0cdee0 2023-10-16T10:03:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 >> v1:172.21.15.158:6789/0 conn(0x7f4d4c0cdee0 legacy=0x7f4d4c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34003a00 0x7f4d34003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d34004850 0x7f4d34004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d34005210 0x7f4d3400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c15d2c0 con 0x7f4d34005210 2023-10-16T10:03:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3400c1b0 con 0x7f4d34003a00 2023-10-16T10:03:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3400c370 con 0x7f4d34004850 2023-10-16T10:03:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d34005210 0x7f4d3400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4affd700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34003a00 0x7f4d34003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4a7fc700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d34004850 0x7f4d34004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 -- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d34004850 msgr2=0x7f4d34004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d34004850 0x7f4d34004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 -- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34003a00 msgr2=0x7f4d34003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34003a00 0x7f4d34003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 -- 172.21.15.158:0/4186441976 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d340035b0 con 0x7f4d34005210 2023-10-16T10:03:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 -- 172.21.15.158:0/4186441976 reap_dead start 2023-10-16T10:03:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.499+0000 7f4d4b7fe700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d34005210 0x7f4d3400ba10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c190 tx=0x7f4d34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2d93d8a1d6e270 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44001db0 con 0x7f4d34005210 2023-10-16T10:03:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4400ccd0 con 0x7f4d34005210 2023-10-16T10:03:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d49ffb700 1 -- 172.21.15.158:0/4186441976 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44003c40 con 0x7f4d34005210 2023-10-16T10:03:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d516fb700 1 -- 172.21.15.158:0/4186441976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d34005210 msgr2=0x7f4d3400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d516fb700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d34005210 0x7f4d3400ba10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c190 tx=0x7f4d34005880 comp rx=0 tx=0).stop 2023-10-16T10:03:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d516fb700 1 -- 172.21.15.158:0/4186441976 shutdown_connections 2023-10-16T10:03:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d516fb700 1 --2- 172.21.15.158:0/4186441976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d34005210 0x7f4d3400ba10 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d516fb700 1 -- 172.21.15.158:0/4186441976 >> 172.21.15.158:0/4186441976 conn(0x7f4d4c06a8d0 msgr2=0x7f4d4c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d516fb700 1 -- 172.21.15.158:0/4186441976 shutdown_connections 2023-10-16T10:03:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.500+0000 7f4d516fb700 1 -- 172.21.15.158:0/4186441976 wait complete. 2023-10-16T10:03:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 Processor -- start 2023-10-16T10:03:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 -- start start 2023-10-16T10:03:28.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34005210 0x7f4d4c14f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d4c0b26a0 0x7f4d4c0b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d4c0d08f0 0x7f4d4c0d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0b3040 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0c5ad0 con 0x7f4d34005210 2023-10-16T10:03:28.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d516fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0c4ec0 con 0x7f4d4c0d08f0 2023-10-16T10:03:28.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d4a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d4c0b26a0 0x7f4d4c0b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.502+0000 7f4d4a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d4c0b26a0 0x7f4d4c0b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58226/0 (socket says 172.21.15.158:58226) 2023-10-16T10:03:28.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4a7fc700 1 -- 172.21.15.158:0/1749411092 learned_addr learned my addr 172.21.15.158:0/1749411092 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:28.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4affd700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34005210 0x7f4d4c14f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4b7fe700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d4c0d08f0 0x7f4d4c0d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4a7fc700 1 -- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d4c0d08f0 msgr2=0x7f4d4c0d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4a7fc700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d4c0d08f0 0x7f4d4c0d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4a7fc700 1 -- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34005210 msgr2=0x7f4d4c14f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4a7fc700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34005210 0x7f4d4c14f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4a7fc700 1 -- 172.21.15.158:0/1749411092 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c0c51a0 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d4a7fc700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d4c0b26a0 0x7f4d4c0b2b00 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c0119a0 tx=0x7f4d3c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d337fe700 1 -- 172.21.15.158:0/1749411092 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00f940 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d337fe700 1 -- 172.21.15.158:0/1749411092 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c00faa0 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c0c5480 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d337fe700 1 -- 172.21.15.158:0/1749411092 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00fe30 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.503+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c14faf0 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.504+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c0c5c80 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.505+0000 7f4d337fe700 1 -- 172.21.15.158:0/1749411092 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4d3c014440 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.505+0000 7f4d337fe700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d3806b020 0x7f4d3806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.505+0000 7f4d337fe700 1 -- 172.21.15.158:0/1749411092 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f4d3c08a180 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.505+0000 7f4d4affd700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d3806b020 0x7f4d3806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.505+0000 7f4d4affd700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d3806b020 0x7f4d3806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4d40007cd0 tx=0x7f4d40009040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.509+0000 7f4d337fe700 1 -- 172.21.15.158:0/1749411092 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d4c0c5c80 con 0x7f4d4c0b26a0 2023-10-16T10:03:28.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.700+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 --> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d4c15cee0 con 0x7f4d3806b020 2023-10-16T10:03:28.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.708+0000 7f4d337fe700 1 -- 172.21.15.158:0/1749411092 <== mgr.8574 v2:172.21.15.158:6832/2342578387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289771 (secure 0 0 0) 0x7f4d40000ed0 con 0x7f4d3806b020 2023-10-16T10:03:28.713 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:28.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d3806b020 msgr2=0x7f4d3806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d3806b020 0x7f4d3806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4d40007cd0 tx=0x7f4d40009040 comp rx=0 tx=0).stop 2023-10-16T10:03:28.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d4c0b26a0 msgr2=0x7f4d4c0b2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d4c0b26a0 0x7f4d4c0b2b00 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c0119a0 tx=0x7f4d3c011d60 comp rx=0 tx=0).stop 2023-10-16T10:03:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 shutdown_connections 2023-10-16T10:03:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4d3806b020 0x7f4d3806d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d4c0d08f0 0x7f4d4c0d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d34005210 0x7f4d4c14f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 --2- 172.21.15.158:0/1749411092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d4c0b26a0 0x7f4d4c0b2b00 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.711+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 >> 172.21.15.158:0/1749411092 conn(0x7f4d4c06a8d0 msgr2=0x7f4d4c0b3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:28.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.712+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 shutdown_connections 2023-10-16T10:03:28.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.712+0000 7f4d516fb700 1 -- 172.21.15.158:0/1749411092 wait complete. 2023-10-16T10:03:28.718 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:03:28.725 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T10:03:27.161506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":347464,"kb_used_data":14724,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754627256,"statfs":{"total":773094113280,"available":772738310144,"internally_reserved":0,"allocated":15077376,"data_stored":8444108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":340577847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.964343"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621495+0000","last_change":"2023-10-16T10:03:23.626314+0000","last_active":"2023-10-16T10:03:24.621495+0000","last_peered":"2023-10-16T10:03:24.621495+0000","last_clean":"2023-10-16T10:03:24.621495+0000","last_became_active":"2023-10-16T10:03:23.626013+0000","last_became_peered":"2023-10-16T10:03:23.626013+0000","last_unstale":"2023-10-16T10:03:24.621495+0000","last_undegraded":"2023-10-16T10:03:24.621495+0000","last_fullsized":"2023-10-16T10:03:24.621495+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:32.563072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621113+0000","last_change":"2023-10-16T10:03:23.632322+0000","last_active":"2023-10-16T10:03:24.621113+0000","last_peered":"2023-10-16T10:03:24.621113+0000","last_clean":"2023-10-16T10:03:24.621113+0000","last_became_active":"2023-10-16T10:03:23.632067+0000","last_became_peered":"2023-10-16T10:03:23.632067+0000","last_unstale":"2023-10-16T10:03:24.621113+0000","last_undegraded":"2023-10-16T10:03:24.621113+0000","last_fullsized":"2023-10-16T10:03:24.621113+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:40.726019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621129+0000","last_change":"2023-10-16T10:03:23.636914+0000","last_active":"2023-10-16T10:03:24.621129+0000","last_peered":"2023-10-16T10:03:24.621129+0000","last_clean":"2023-10-16T10:03:24.621129+0000","last_became_active":"2023-10-16T10:03:23.636515+0000","last_became_peered":"2023-10-16T10:03:23.636515+0000","last_unstale":"2023-10-16T10:03:24.621129+0000","last_undegraded":"2023-10-16T10:03:24.621129+0000","last_fullsized":"2023-10-16T10:03:24.621129+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:50.918218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620787+0000","last_change":"2023-10-16T10:03:23.626396+0000","last_active":"2023-10-16T10:03:24.620787+0000","last_peered":"2023-10-16T10:03:24.620787+0000","last_clean":"2023-10-16T10:03:24.620787+0000","last_became_active":"2023-10-16T10:03:23.626109+0000","last_became_peered":"2023-10-16T10:03:23.626109+0000","last_unstale":"2023-10-16T10:03:24.620787+0000","last_undegraded":"2023-10-16T10:03:24.620787+0000","last_fullsized":"2023-10-16T10:03:24.620787+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:09.163635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621457+0000","last_change":"2023-10-16T10:03:23.628137+0000","last_active":"2023-10-16T10:03:24.621457+0000","last_peered":"2023-10-16T10:03:24.621457+0000","last_clean":"2023-10-16T10:03:24.621457+0000","last_became_active":"2023-10-16T10:03:23.627803+0000","last_became_peered":"2023-10-16T10:03:23.627803+0000","last_unstale":"2023-10-16T10:03:24.621457+0000","last_undegraded":"2023-10-16T10:03:24.621457+0000","last_fullsized":"2023-10-16T10:03:24.621457+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:12.091137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634241+0000","last_change":"2023-10-16T10:03:23.631906+0000","last_active":"2023-10-16T10:03:23.634241+0000","last_peered":"2023-10-16T10:03:23.634241+0000","last_clean":"2023-10-16T10:03:23.634241+0000","last_became_active":"2023-10-16T10:03:23.631623+0000","last_became_peered":"2023-10-16T10:03:23.631623+0000","last_unstale":"2023-10-16T10:03:23.634241+0000","last_undegraded":"2023-10-16T10:03:23.634241+0000","last_fullsized":"2023-10-16T10:03:23.634241+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:23.123139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628826+0000","last_change":"2023-10-16T10:03:23.622121+0000","last_active":"2023-10-16T10:03:23.628826+0000","last_peered":"2023-10-16T10:03:23.628826+0000","last_clean":"2023-10-16T10:03:23.628826+0000","last_became_active":"2023-10-16T10:03:23.621461+0000","last_became_peered":"2023-10-16T10:03:23.621461+0000","last_unstale":"2023-10-16T10:03:23.628826+0000","last_undegraded":"2023-10-16T10:03:23.628826+0000","last_fullsized":"2023-10-16T10:03:23.628826+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:47:10.394652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620513+0000","last_change":"2023-10-16T10:03:23.636649+0000","last_active":"2023-10-16T10:03:24.620513+0000","last_peered":"2023-10-16T10:03:24.620513+0000","last_clean":"2023-10-16T10:03:24.620513+0000","last_became_active":"2023-10-16T10:03:23.636381+0000","last_became_peered":"2023-10-16T10:03:23.636381+0000","last_unstale":"2023-10-16T10:03:24.620513+0000","last_undegraded":"2023-10-16T10:03:24.620513+0000","last_fullsized":"2023-10-16T10:03:24.620513+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:56.100229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620980+0000","last_change":"2023-10-16T10:03:23.632272+0000","last_active":"2023-10-16T10:03:24.620980+0000","last_peered":"2023-10-16T10:03:24.620980+0000","last_clean":"2023-10-16T10:03:24.620980+0000","last_became_active":"2023-10-16T10:03:23.631887+0000","last_became_peered":"2023-10-16T10:03:23.631887+0000","last_unstale":"2023-10-16T10:03:24.620980+0000","last_undegraded":"2023-10-16T10:03:24.620980+0000","last_fullsized":"2023-10-16T10:03:24.620980+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:43.636028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.630233+0000","last_change":"2023-10-16T10:03:23.629557+0000","last_active":"2023-10-16T10:03:23.630233+0000","last_peered":"2023-10-16T10:03:23.630233+0000","last_clean":"2023-10-16T10:03:23.630233+0000","last_became_active":"2023-10-16T10:03:23.629276+0000","last_became_peered":"2023-10-16T10:03:23.629276+0000","last_unstale":"2023-10-16T10:03:23.630233+0000","last_undegraded":"2023-10-16T10:03:23.630233+0000","last_fullsized":"2023-10-16T10:03:23.630233+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:01.347666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621654+0000","last_change":"2023-10-16T10:03:23.635932+0000","last_active":"2023-10-16T10:03:24.621654+0000","last_peered":"2023-10-16T10:03:24.621654+0000","last_clean":"2023-10-16T10:03:24.621654+0000","last_became_active":"2023-10-16T10:03:23.635305+0000","last_became_peered":"2023-10-16T10:03:23.635305+0000","last_unstale":"2023-10-16T10:03:24.621654+0000","last_undegraded":"2023-10-16T10:03:24.621654+0000","last_fullsized":"2023-10-16T10:03:24.621654+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:50.393990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.629011+0000","last_change":"2023-10-16T10:03:23.627578+0000","last_active":"2023-10-16T10:03:23.629011+0000","last_peered":"2023-10-16T10:03:23.629011+0000","last_clean":"2023-10-16T10:03:23.629011+0000","last_became_active":"2023-10-16T10:03:23.627331+0000","last_became_peered":"2023-10-16T10:03:23.627331+0000","last_unstale":"2023-10-16T10:03:23.629011+0000","last_undegraded":"2023-10-16T10:03:23.629011+0000","last_fullsized":"2023-10-16T10:03:23.629011+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:30.812138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.635991+0000","last_change":"2023-10-16T10:03:23.635528+0000","last_active":"2023-10-16T10:03:23.635991+0000","last_peered":"2023-10-16T10:03:23.635991+0000","last_clean":"2023-10-16T10:03:23.635991+0000","last_became_active":"2023-10-16T10:03:23.635253+0000","last_became_peered":"2023-10-16T10:03:23.635253+0000","last_unstale":"2023-10-16T10:03:23.635991+0000","last_undegraded":"2023-10-16T10:03:23.635991+0000","last_fullsized":"2023-10-16T10:03:23.635991+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:44.041713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620651+0000","last_change":"2023-10-16T10:03:23.639551+0000","last_active":"2023-10-16T10:03:24.620651+0000","last_peered":"2023-10-16T10:03:24.620651+0000","last_clean":"2023-10-16T10:03:24.620651+0000","last_became_active":"2023-10-16T10:03:23.639367+0000","last_became_peered":"2023-10-16T10:03:23.639367+0000","last_unstale":"2023-10-16T10:03:24.620651+0000","last_undegraded":"2023-10-16T10:03:24.620651+0000","last_fullsized":"2023-10-16T10:03:24.620651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:19.518196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621198+0000","last_change":"2023-10-16T10:03:23.635736+0000","last_active":"2023-10-16T10:03:24.621198+0000","last_peered":"2023-10-16T10:03:24.621198+0000","last_clean":"2023-10-16T10:03:24.621198+0000","last_became_active":"2023-10-16T10:03:23.635340+0000","last_became_peered":"2023-10-16T10:03:23.635340+0000","last_unstale":"2023-10-16T10:03:24.621198+0000","last_undegraded":"2023-10-16T10:03:24.621198+0000","last_fullsized":"2023-10-16T10:03:24.621198+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:19.051819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628782+0000","last_change":"2023-10-16T10:03:23.625147+0000","last_active":"2023-10-16T10:03:23.628782+0000","last_peered":"2023-10-16T10:03:23.628782+0000","last_clean":"2023-10-16T10:03:23.628782+0000","last_became_active":"2023-10-16T10:03:23.624440+0000","last_became_peered":"2023-10-16T10:03:23.624440+0000","last_unstale":"2023-10-16T10:03:23.628782+0000","last_undegraded":"2023-10-16T10:03:23.628782+0000","last_fullsized":"2023-10-16T10:03:23.628782+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:18.495717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.637635+0000","last_change":"2023-10-16T10:03:23.636735+0000","last_active":"2023-10-16T10:03:23.637635+0000","last_peered":"2023-10-16T10:03:23.637635+0000","last_clean":"2023-10-16T10:03:23.637635+0000","last_became_active":"2023-10-16T10:03:23.636479+0000","last_became_peered":"2023-10-16T10:03:23.636479+0000","last_unstale":"2023-10-16T10:03:23.637635+0000","last_undegraded":"2023-10-16T10:03:23.637635+0000","last_fullsized":"2023-10-16T10:03:23.637635+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:29.501165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621113+0000","last_change":"2023-10-16T10:03:23.632457+0000","last_active":"2023-10-16T10:03:24.621113+0000","last_peered":"2023-10-16T10:03:24.621113+0000","last_clean":"2023-10-16T10:03:24.621113+0000","last_became_active":"2023-10-16T10:03:23.632191+0000","last_became_peered":"2023-10-16T10:03:23.632191+0000","last_unstale":"2023-10-16T10:03:24.621113+0000","last_undegraded":"2023-10-16T10:03:24.621113+0000","last_fullsized":"2023-10-16T10:03:24.621113+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:31.156072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638543+0000","last_change":"2023-10-16T10:03:23.634582+0000","last_active":"2023-10-16T10:03:23.638543+0000","last_peered":"2023-10-16T10:03:23.638543+0000","last_clean":"2023-10-16T10:03:23.638543+0000","last_became_active":"2023-10-16T10:03:23.634316+0000","last_became_peered":"2023-10-16T10:03:23.634316+0000","last_unstale":"2023-10-16T10:03:23.638543+0000","last_undegraded":"2023-10-16T10:03:23.638543+0000","last_fullsized":"2023-10-16T10:03:23.638543+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:29.501165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.625997+0000","last_change":"2023-10-16T10:03:23.624811+0000","last_active":"2023-10-16T10:03:23.625997+0000","last_peered":"2023-10-16T10:03:23.625997+0000","last_clean":"2023-10-16T10:03:23.625997+0000","last_became_active":"2023-10-16T10:03:23.624560+0000","last_became_peered":"2023-10-16T10:03:23.624560+0000","last_unstale":"2023-10-16T10:03:23.625997+0000","last_undegraded":"2023-10-16T10:03:23.625997+0000","last_fullsized":"2023-10-16T10:03:23.625997+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:30.725405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620316+0000","last_change":"2023-10-16T10:03:23.623080+0000","last_active":"2023-10-16T10:03:24.620316+0000","last_peered":"2023-10-16T10:03:24.620316+0000","last_clean":"2023-10-16T10:03:24.620316+0000","last_became_active":"2023-10-16T10:03:23.622799+0000","last_became_peered":"2023-10-16T10:03:23.622799+0000","last_unstale":"2023-10-16T10:03:24.620316+0000","last_undegraded":"2023-10-16T10:03:24.620316+0000","last_fullsized":"2023-10-16T10:03:24.620316+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:32.761688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621541+0000","last_change":"2023-10-16T10:03:23.632108+0000","last_active":"2023-10-16T10:03:24.621541+0000","last_peered":"2023-10-16T10:03:24.621541+0000","last_clean":"2023-10-16T10:03:24.621541+0000","last_became_active":"2023-10-16T10:03:23.631675+0000","last_became_peered":"2023-10-16T10:03:23.631675+0000","last_unstale":"2023-10-16T10:03:24.621541+0000","last_undegraded":"2023-10-16T10:03:24.621541+0000","last_fullsized":"2023-10-16T10:03:24.621541+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:08.266903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620270+0000","last_change":"2023-10-16T10:03:23.632409+0000","last_active":"2023-10-16T10:03:24.620270+0000","last_peered":"2023-10-16T10:03:24.620270+0000","last_clean":"2023-10-16T10:03:24.620270+0000","last_became_active":"2023-10-16T10:03:23.631812+0000","last_became_peered":"2023-10-16T10:03:23.631812+0000","last_unstale":"2023-10-16T10:03:24.620270+0000","last_undegraded":"2023-10-16T10:03:24.620270+0000","last_fullsized":"2023-10-16T10:03:24.620270+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:46:13.229655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.635449+0000","last_change":"2023-10-16T10:03:23.633052+0000","last_active":"2023-10-16T10:03:23.635449+0000","last_peered":"2023-10-16T10:03:23.635449+0000","last_clean":"2023-10-16T10:03:23.635449+0000","last_became_active":"2023-10-16T10:03:23.632701+0000","last_became_peered":"2023-10-16T10:03:23.632701+0000","last_unstale":"2023-10-16T10:03:23.635449+0000","last_undegraded":"2023-10-16T10:03:23.635449+0000","last_fullsized":"2023-10-16T10:03:23.635449+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:28.782584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.640615+0000","last_change":"2023-10-16T10:03:23.638315+0000","last_active":"2023-10-16T10:03:23.640615+0000","last_peered":"2023-10-16T10:03:23.640615+0000","last_clean":"2023-10-16T10:03:23.640615+0000","last_became_active":"2023-10-16T10:03:23.638028+0000","last_became_peered":"2023-10-16T10:03:23.638028+0000","last_unstale":"2023-10-16T10:03:23.640615+0000","last_undegraded":"2023-10-16T10:03:23.640615+0000","last_fullsized":"2023-10-16T10:03:23.640615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:17.739964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.624439+0000","last_change":"2023-10-16T10:03:23.623825+0000","last_active":"2023-10-16T10:03:23.624439+0000","last_peered":"2023-10-16T10:03:23.624439+0000","last_clean":"2023-10-16T10:03:23.624439+0000","last_became_active":"2023-10-16T10:03:23.623498+0000","last_became_peered":"2023-10-16T10:03:23.623498+0000","last_unstale":"2023-10-16T10:03:23.624439+0000","last_undegraded":"2023-10-16T10:03:23.624439+0000","last_fullsized":"2023-10-16T10:03:23.624439+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:33.666814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638561+0000","last_change":"2023-10-16T10:03:23.635272+0000","last_active":"2023-10-16T10:03:23.638561+0000","last_peered":"2023-10-16T10:03:23.638561+0000","last_clean":"2023-10-16T10:03:23.638561+0000","last_became_active":"2023-10-16T10:03:23.635004+0000","last_became_peered":"2023-10-16T10:03:23.635004+0000","last_unstale":"2023-10-16T10:03:23.638561+0000","last_undegraded":"2023-10-16T10:03:23.638561+0000","last_fullsized":"2023-10-16T10:03:23.638561+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:47.998013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.637094+0000","last_change":"2023-10-16T10:03:23.635932+0000","last_active":"2023-10-16T10:03:23.637094+0000","last_peered":"2023-10-16T10:03:23.637094+0000","last_clean":"2023-10-16T10:03:23.637094+0000","last_became_active":"2023-10-16T10:03:23.635667+0000","last_became_peered":"2023-10-16T10:03:23.635667+0000","last_unstale":"2023-10-16T10:03:23.637094+0000","last_undegraded":"2023-10-16T10:03:23.637094+0000","last_fullsized":"2023-10-16T10:03:23.637094+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:40.726019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.632923+0000","last_change":"2023-10-16T10:03:23.632332+0000","last_active":"2023-10-16T10:03:23.632923+0000","last_peered":"2023-10-16T10:03:23.632923+0000","last_clean":"2023-10-16T10:03:23.632923+0000","last_became_active":"2023-10-16T10:03:23.632060+0000","last_became_peered":"2023-10-16T10:03:23.632060+0000","last_unstale":"2023-10-16T10:03:23.632923+0000","last_undegraded":"2023-10-16T10:03:23.632923+0000","last_fullsized":"2023-10-16T10:03:23.632923+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:44.645495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620773+0000","last_change":"2023-10-16T10:03:23.638038+0000","last_active":"2023-10-16T10:03:24.620773+0000","last_peered":"2023-10-16T10:03:24.620773+0000","last_clean":"2023-10-16T10:03:24.620773+0000","last_became_active":"2023-10-16T10:03:23.637786+0000","last_became_peered":"2023-10-16T10:03:23.637786+0000","last_unstale":"2023-10-16T10:03:24.620773+0000","last_undegraded":"2023-10-16T10:03:24.620773+0000","last_fullsized":"2023-10-16T10:03:24.620773+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:03.461277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634520+0000","last_change":"2023-10-16T10:03:23.628135+0000","last_active":"2023-10-16T10:03:23.634520+0000","last_peered":"2023-10-16T10:03:23.634520+0000","last_clean":"2023-10-16T10:03:23.634520+0000","last_became_active":"2023-10-16T10:03:23.627456+0000","last_became_peered":"2023-10-16T10:03:23.627456+0000","last_unstale":"2023-10-16T10:03:23.634520+0000","last_undegraded":"2023-10-16T10:03:23.634520+0000","last_fullsized":"2023-10-16T10:03:23.634520+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:26.591160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638404+0000","last_change":"2023-10-16T10:03:23.634845+0000","last_active":"2023-10-16T10:03:23.638404+0000","last_peered":"2023-10-16T10:03:23.638404+0000","last_clean":"2023-10-16T10:03:23.638404+0000","last_became_active":"2023-10-16T10:03:23.634366+0000","last_became_peered":"2023-10-16T10:03:23.634366+0000","last_unstale":"2023-10-16T10:03:23.638404+0000","last_undegraded":"2023-10-16T10:03:23.638404+0000","last_fullsized":"2023-10-16T10:03:23.638404+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:26.681599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.630270+0000","last_change":"2023-10-16T10:03:23.629617+0000","last_active":"2023-10-16T10:03:23.630270+0000","last_peered":"2023-10-16T10:03:23.630270+0000","last_clean":"2023-10-16T10:03:23.630270+0000","last_became_active":"2023-10-16T10:03:23.629320+0000","last_became_peered":"2023-10-16T10:03:23.629320+0000","last_unstale":"2023-10-16T10:03:23.630270+0000","last_undegraded":"2023-10-16T10:03:23.630270+0000","last_fullsized":"2023-10-16T10:03:23.630270+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:22.800676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"139'4","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215525+0000","last_change":"2023-10-16T10:03:23.222284+0000","last_active":"2023-10-16T10:03:24.215525+0000","last_peered":"2023-10-16T10:03:24.215525+0000","last_clean":"2023-10-16T10:03:24.215525+0000","last_became_active":"2023-10-16T10:03:23.221978+0000","last_became_peered":"2023-10-16T10:03:23.221978+0000","last_unstale":"2023-10-16T10:03:24.215525+0000","last_undegraded":"2023-10-16T10:03:24.215525+0000","last_fullsized":"2023-10-16T10:03:24.215525+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:43.036173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.633935+0000","last_change":"2023-10-16T10:03:23.631704+0000","last_active":"2023-10-16T10:03:23.633935+0000","last_peered":"2023-10-16T10:03:23.633935+0000","last_clean":"2023-10-16T10:03:23.633935+0000","last_became_active":"2023-10-16T10:03:23.631429+0000","last_became_peered":"2023-10-16T10:03:23.631429+0000","last_unstale":"2023-10-16T10:03:23.633935+0000","last_undegraded":"2023-10-16T10:03:23.633935+0000","last_fullsized":"2023-10-16T10:03:23.633935+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:56.100229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.216475+0000","last_change":"2023-10-16T10:03:23.218482+0000","last_active":"2023-10-16T10:03:24.216475+0000","last_peered":"2023-10-16T10:03:24.216475+0000","last_clean":"2023-10-16T10:03:24.216475+0000","last_became_active":"2023-10-16T10:03:23.218249+0000","last_became_peered":"2023-10-16T10:03:23.218249+0000","last_unstale":"2023-10-16T10:03:24.216475+0000","last_undegraded":"2023-10-16T10:03:24.216475+0000","last_fullsized":"2023-10-16T10:03:24.216475+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:19.718159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638500+0000","last_change":"2023-10-16T10:03:23.635344+0000","last_active":"2023-10-16T10:03:23.638500+0000","last_peered":"2023-10-16T10:03:23.638500+0000","last_clean":"2023-10-16T10:03:23.638500+0000","last_became_active":"2023-10-16T10:03:23.634536+0000","last_became_peered":"2023-10-16T10:03:23.634536+0000","last_unstale":"2023-10-16T10:03:23.638500+0000","last_undegraded":"2023-10-16T10:03:23.638500+0000","last_fullsized":"2023-10-16T10:03:23.638500+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:51.573768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"139'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620553+0000","last_change":"2023-10-16T10:03:23.220018+0000","last_active":"2023-10-16T10:03:24.620553+0000","last_peered":"2023-10-16T10:03:24.620553+0000","last_clean":"2023-10-16T10:03:24.620553+0000","last_became_active":"2023-10-16T10:03:23.219746+0000","last_became_peered":"2023-10-16T10:03:23.219746+0000","last_unstale":"2023-10-16T10:03:24.620553+0000","last_undegraded":"2023-10-16T10:03:24.620553+0000","last_fullsized":"2023-10-16T10:03:24.620553+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:27.777043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621127+0000","last_change":"2023-10-16T10:03:23.629254+0000","last_active":"2023-10-16T10:03:24.621127+0000","last_peered":"2023-10-16T10:03:24.621127+0000","last_clean":"2023-10-16T10:03:24.621127+0000","last_became_active":"2023-10-16T10:03:23.628934+0000","last_became_peered":"2023-10-16T10:03:23.628934+0000","last_unstale":"2023-10-16T10:03:24.621127+0000","last_undegraded":"2023-10-16T10:03:24.621127+0000","last_fullsized":"2023-10-16T10:03:24.621127+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:58.758523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.219623+0000","last_change":"2023-10-16T10:03:23.218558+0000","last_active":"2023-10-16T10:03:24.219623+0000","last_peered":"2023-10-16T10:03:24.219623+0000","last_clean":"2023-10-16T10:03:24.219623+0000","last_became_active":"2023-10-16T10:03:23.218285+0000","last_became_peered":"2023-10-16T10:03:23.218285+0000","last_unstale":"2023-10-16T10:03:24.219623+0000","last_undegraded":"2023-10-16T10:03:24.219623+0000","last_fullsized":"2023-10-16T10:03:24.219623+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:38:39.156686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634175+0000","last_change":"2023-10-16T10:03:23.633313+0000","last_active":"2023-10-16T10:03:23.634175+0000","last_peered":"2023-10-16T10:03:23.634175+0000","last_clean":"2023-10-16T10:03:23.634175+0000","last_became_active":"2023-10-16T10:03:23.633027+0000","last_became_peered":"2023-10-16T10:03:23.633027+0000","last_unstale":"2023-10-16T10:03:23.634175+0000","last_undegraded":"2023-10-16T10:03:23.634175+0000","last_fullsized":"2023-10-16T10:03:23.634175+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:03.156368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621561+0000","last_change":"2023-10-16T10:03:23.209306+0000","last_active":"2023-10-16T10:03:24.621561+0000","last_peered":"2023-10-16T10:03:24.621561+0000","last_clean":"2023-10-16T10:03:24.621561+0000","last_became_active":"2023-10-16T10:03:23.209002+0000","last_became_peered":"2023-10-16T10:03:23.209002+0000","last_unstale":"2023-10-16T10:03:24.621561+0000","last_undegraded":"2023-10-16T10:03:24.621561+0000","last_fullsized":"2023-10-16T10:03:24.621561+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:43.036173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621339+0000","last_change":"2023-10-16T10:03:23.628173+0000","last_active":"2023-10-16T10:03:24.621339+0000","last_peered":"2023-10-16T10:03:24.621339+0000","last_clean":"2023-10-16T10:03:24.621339+0000","last_became_active":"2023-10-16T10:03:23.627904+0000","last_became_peered":"2023-10-16T10:03:23.627904+0000","last_unstale":"2023-10-16T10:03:24.621339+0000","last_undegraded":"2023-10-16T10:03:24.621339+0000","last_fullsized":"2023-10-16T10:03:24.621339+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:28:25.599006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.219460+0000","last_change":"2023-10-16T10:03:23.225572+0000","last_active":"2023-10-16T10:03:24.219460+0000","last_peered":"2023-10-16T10:03:24.219460+0000","last_clean":"2023-10-16T10:03:24.219460+0000","last_became_active":"2023-10-16T10:03:23.225284+0000","last_became_peered":"2023-10-16T10:03:23.225284+0000","last_unstale":"2023-10-16T10:03:24.219460+0000","last_undegraded":"2023-10-16T10:03:24.219460+0000","last_fullsized":"2023-10-16T10:03:24.219460+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:27.703147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620879+0000","last_change":"2023-10-16T10:03:23.631008+0000","last_active":"2023-10-16T10:03:24.620879+0000","last_peered":"2023-10-16T10:03:24.620879+0000","last_clean":"2023-10-16T10:03:24.620879+0000","last_became_active":"2023-10-16T10:03:23.630751+0000","last_became_peered":"2023-10-16T10:03:23.630751+0000","last_unstale":"2023-10-16T10:03:24.620879+0000","last_undegraded":"2023-10-16T10:03:24.620879+0000","last_fullsized":"2023-10-16T10:03:24.620879+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:30.812138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.623955+0000","last_change":"2023-10-16T10:03:23.222406+0000","last_active":"2023-10-16T10:03:23.623955+0000","last_peered":"2023-10-16T10:03:23.623955+0000","last_clean":"2023-10-16T10:03:23.623955+0000","last_became_active":"2023-10-16T10:03:23.222092+0000","last_became_peered":"2023-10-16T10:03:23.222092+0000","last_unstale":"2023-10-16T10:03:23.623955+0000","last_undegraded":"2023-10-16T10:03:23.623955+0000","last_fullsized":"2023-10-16T10:03:23.623955+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:07.965428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.633822+0000","last_change":"2023-10-16T10:03:23.631172+0000","last_active":"2023-10-16T10:03:23.633822+0000","last_peered":"2023-10-16T10:03:23.633822+0000","last_clean":"2023-10-16T10:03:23.633822+0000","last_became_active":"2023-10-16T10:03:23.630944+0000","last_became_peered":"2023-10-16T10:03:23.630944+0000","last_unstale":"2023-10-16T10:03:23.633822+0000","last_undegraded":"2023-10-16T10:03:23.633822+0000","last_fullsized":"2023-10-16T10:03:23.633822+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:59.596827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.618857+0000","last_change":"2023-10-16T10:03:23.225774+0000","last_active":"2023-10-16T10:03:23.618857+0000","last_peered":"2023-10-16T10:03:23.618857+0000","last_clean":"2023-10-16T10:03:23.618857+0000","last_became_active":"2023-10-16T10:03:23.225392+0000","last_became_peered":"2023-10-16T10:03:23.225392+0000","last_unstale":"2023-10-16T10:03:23.618857+0000","last_undegraded":"2023-10-16T10:03:23.618857+0000","last_fullsized":"2023-10-16T10:03:23.618857+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:30.150532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.635891+0000","last_change":"2023-10-16T10:03:23.629426+0000","last_active":"2023-10-16T10:03:23.635891+0000","last_peered":"2023-10-16T10:03:23.635891+0000","last_clean":"2023-10-16T10:03:23.635891+0000","last_became_active":"2023-10-16T10:03:23.629160+0000","last_became_peered":"2023-10-16T10:03:23.629160+0000","last_unstale":"2023-10-16T10:03:23.635891+0000","last_undegraded":"2023-10-16T10:03:23.635891+0000","last_fullsized":"2023-10-16T10:03:23.635891+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:20.723699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.629981+0000","last_change":"2023-10-16T10:03:23.229644+0000","last_active":"2023-10-16T10:03:23.629981+0000","last_peered":"2023-10-16T10:03:23.629981+0000","last_clean":"2023-10-16T10:03:23.629981+0000","last_became_active":"2023-10-16T10:03:23.229342+0000","last_became_peered":"2023-10-16T10:03:23.229342+0000","last_unstale":"2023-10-16T10:03:23.629981+0000","last_undegraded":"2023-10-16T10:03:23.629981+0000","last_fullsized":"2023-10-16T10:03:23.629981+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:21.795135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638901+0000","last_change":"2023-10-16T10:03:23.636750+0000","last_active":"2023-10-16T10:03:23.638901+0000","last_peered":"2023-10-16T10:03:23.638901+0000","last_clean":"2023-10-16T10:03:23.638901+0000","last_became_active":"2023-10-16T10:03:23.635328+0000","last_became_peered":"2023-10-16T10:03:23.635328+0000","last_unstale":"2023-10-16T10:03:23.638901+0000","last_undegraded":"2023-10-16T10:03:23.638901+0000","last_fullsized":"2023-10-16T10:03:23.638901+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:32.761688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.623752+0000","last_change":"2023-10-16T10:03:23.220505+0000","last_active":"2023-10-16T10:03:23.623752+0000","last_peered":"2023-10-16T10:03:23.623752+0000","last_clean":"2023-10-16T10:03:23.623752+0000","last_became_active":"2023-10-16T10:03:23.219944+0000","last_became_peered":"2023-10-16T10:03:23.219944+0000","last_unstale":"2023-10-16T10:03:23.623752+0000","last_undegraded":"2023-10-16T10:03:23.623752+0000","last_fullsized":"2023-10-16T10:03:23.623752+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:28:17.749101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621198+0000","last_change":"2023-10-16T10:03:23.639066+0000","last_active":"2023-10-16T10:03:24.621198+0000","last_peered":"2023-10-16T10:03:24.621198+0000","last_clean":"2023-10-16T10:03:24.621198+0000","last_became_active":"2023-10-16T10:03:23.638888+0000","last_became_peered":"2023-10-16T10:03:23.638888+0000","last_unstale":"2023-10-16T10:03:24.621198+0000","last_undegraded":"2023-10-16T10:03:24.621198+0000","last_fullsized":"2023-10-16T10:03:24.621198+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:47:10.394652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"139'3","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215141+0000","last_change":"2023-10-16T10:03:23.220514+0000","last_active":"2023-10-16T10:03:24.215141+0000","last_peered":"2023-10-16T10:03:24.215141+0000","last_clean":"2023-10-16T10:03:24.215141+0000","last_became_active":"2023-10-16T10:03:23.220232+0000","last_became_peered":"2023-10-16T10:03:23.220232+0000","last_unstale":"2023-10-16T10:03:24.215141+0000","last_undegraded":"2023-10-16T10:03:24.215141+0000","last_fullsized":"2023-10-16T10:03:24.215141+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:08.562450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638982+0000","last_change":"2023-10-16T10:03:23.636470+0000","last_active":"2023-10-16T10:03:23.638982+0000","last_peered":"2023-10-16T10:03:23.638982+0000","last_clean":"2023-10-16T10:03:23.638982+0000","last_became_active":"2023-10-16T10:03:23.635901+0000","last_became_peered":"2023-10-16T10:03:23.635901+0000","last_unstale":"2023-10-16T10:03:23.638982+0000","last_undegraded":"2023-10-16T10:03:23.638982+0000","last_fullsized":"2023-10-16T10:03:23.638982+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:09:03.245724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"139'6","reported_seq":18,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620953+0000","last_change":"2023-10-16T10:03:23.217728+0000","last_active":"2023-10-16T10:03:24.620953+0000","last_peered":"2023-10-16T10:03:24.620953+0000","last_clean":"2023-10-16T10:03:24.620953+0000","last_became_active":"2023-10-16T10:03:23.217471+0000","last_became_peered":"2023-10-16T10:03:23.217471+0000","last_unstale":"2023-10-16T10:03:24.620953+0000","last_undegraded":"2023-10-16T10:03:24.620953+0000","last_fullsized":"2023-10-16T10:03:24.620953+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:57.752982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638491+0000","last_change":"2023-10-16T10:03:23.632706+0000","last_active":"2023-10-16T10:03:23.638491+0000","last_peered":"2023-10-16T10:03:23.638491+0000","last_clean":"2023-10-16T10:03:23.638491+0000","last_became_active":"2023-10-16T10:03:23.632430+0000","last_became_peered":"2023-10-16T10:03:23.632430+0000","last_unstale":"2023-10-16T10:03:23.638491+0000","last_undegraded":"2023-10-16T10:03:23.638491+0000","last_fullsized":"2023-10-16T10:03:23.638491+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:45.192365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"139'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620720+0000","last_change":"2023-10-16T10:03:23.218565+0000","last_active":"2023-10-16T10:03:24.620720+0000","last_peered":"2023-10-16T10:03:24.620720+0000","last_clean":"2023-10-16T10:03:24.620720+0000","last_became_active":"2023-10-16T10:03:23.218301+0000","last_became_peered":"2023-10-16T10:03:23.218301+0000","last_unstale":"2023-10-16T10:03:24.620720+0000","last_undegraded":"2023-10-16T10:03:24.620720+0000","last_fullsized":"2023-10-16T10:03:24.620720+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:42.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.621426+0000","last_change":"2023-10-16T10:03:23.621038+0000","last_active":"2023-10-16T10:03:23.621426+0000","last_peered":"2023-10-16T10:03:23.621426+0000","last_clean":"2023-10-16T10:03:23.621426+0000","last_became_active":"2023-10-16T10:03:23.620775+0000","last_became_peered":"2023-10-16T10:03:23.620775+0000","last_unstale":"2023-10-16T10:03:23.621426+0000","last_undegraded":"2023-10-16T10:03:23.621426+0000","last_fullsized":"2023-10-16T10:03:23.621426+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:26.121933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621015+0000","last_change":"2023-10-16T10:03:23.223444+0000","last_active":"2023-10-16T10:03:24.621015+0000","last_peered":"2023-10-16T10:03:24.621015+0000","last_clean":"2023-10-16T10:03:24.621015+0000","last_became_active":"2023-10-16T10:03:23.223166+0000","last_became_peered":"2023-10-16T10:03:23.223166+0000","last_unstale":"2023-10-16T10:03:24.621015+0000","last_undegraded":"2023-10-16T10:03:24.621015+0000","last_fullsized":"2023-10-16T10:03:24.621015+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:48:08.158094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628166+0000","last_change":"2023-10-16T10:03:23.626943+0000","last_active":"2023-10-16T10:03:23.628166+0000","last_peered":"2023-10-16T10:03:23.628166+0000","last_clean":"2023-10-16T10:03:23.628166+0000","last_became_active":"2023-10-16T10:03:23.626689+0000","last_became_peered":"2023-10-16T10:03:23.626689+0000","last_unstale":"2023-10-16T10:03:23.628166+0000","last_undegraded":"2023-10-16T10:03:23.628166+0000","last_fullsized":"2023-10-16T10:03:23.628166+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:38.201686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620625+0000","last_change":"2023-10-16T10:03:23.215986+0000","last_active":"2023-10-16T10:03:24.620625+0000","last_peered":"2023-10-16T10:03:24.620625+0000","last_clean":"2023-10-16T10:03:24.620625+0000","last_became_active":"2023-10-16T10:03:23.215714+0000","last_became_peered":"2023-10-16T10:03:23.215714+0000","last_unstale":"2023-10-16T10:03:24.620625+0000","last_undegraded":"2023-10-16T10:03:24.620625+0000","last_fullsized":"2023-10-16T10:03:24.620625+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:42.630487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638379+0000","last_change":"2023-10-16T10:03:23.635325+0000","last_active":"2023-10-16T10:03:23.638379+0000","last_peered":"2023-10-16T10:03:23.638379+0000","last_clean":"2023-10-16T10:03:23.638379+0000","last_became_active":"2023-10-16T10:03:23.635037+0000","last_became_peered":"2023-10-16T10:03:23.635037+0000","last_unstale":"2023-10-16T10:03:23.638379+0000","last_undegraded":"2023-10-16T10:03:23.638379+0000","last_fullsized":"2023-10-16T10:03:23.638379+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:46:13.229655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.216649+0000","last_change":"2023-10-16T10:03:23.226350+0000","last_active":"2023-10-16T10:03:24.216649+0000","last_peered":"2023-10-16T10:03:24.216649+0000","last_clean":"2023-10-16T10:03:24.216649+0000","last_became_active":"2023-10-16T10:03:23.226078+0000","last_became_peered":"2023-10-16T10:03:23.226078+0000","last_unstale":"2023-10-16T10:03:24.216649+0000","last_undegraded":"2023-10-16T10:03:24.216649+0000","last_fullsized":"2023-10-16T10:03:24.216649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:44.186824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621810+0000","last_change":"2023-10-16T10:03:23.631928+0000","last_active":"2023-10-16T10:03:24.621810+0000","last_peered":"2023-10-16T10:03:24.621810+0000","last_clean":"2023-10-16T10:03:24.621810+0000","last_became_active":"2023-10-16T10:03:23.631643+0000","last_became_peered":"2023-10-16T10:03:23.631643+0000","last_unstale":"2023-10-16T10:03:24.621810+0000","last_undegraded":"2023-10-16T10:03:24.621810+0000","last_fullsized":"2023-10-16T10:03:24.621810+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:56.643116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.216208+0000","last_change":"2023-10-16T10:03:23.226061+0000","last_active":"2023-10-16T10:03:24.216208+0000","last_peered":"2023-10-16T10:03:24.216208+0000","last_clean":"2023-10-16T10:03:24.216208+0000","last_became_active":"2023-10-16T10:03:23.225788+0000","last_became_peered":"2023-10-16T10:03:23.225788+0000","last_unstale":"2023-10-16T10:03:24.216208+0000","last_undegraded":"2023-10-16T10:03:24.216208+0000","last_fullsized":"2023-10-16T10:03:24.216208+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:48.345279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628975+0000","last_change":"2023-10-16T10:03:23.628619+0000","last_active":"2023-10-16T10:03:23.628975+0000","last_peered":"2023-10-16T10:03:23.628975+0000","last_clean":"2023-10-16T10:03:23.628975+0000","last_became_active":"2023-10-16T10:03:23.628347+0000","last_became_peered":"2023-10-16T10:03:23.628347+0000","last_unstale":"2023-10-16T10:03:23.628975+0000","last_undegraded":"2023-10-16T10:03:23.628975+0000","last_fullsized":"2023-10-16T10:03:23.628975+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:19.518196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620633+0000","last_change":"2023-10-16T10:03:23.223362+0000","last_active":"2023-10-16T10:03:24.620633+0000","last_peered":"2023-10-16T10:03:24.620633+0000","last_clean":"2023-10-16T10:03:24.620633+0000","last_became_active":"2023-10-16T10:03:23.223083+0000","last_became_peered":"2023-10-16T10:03:23.223083+0000","last_unstale":"2023-10-16T10:03:24.620633+0000","last_undegraded":"2023-10-16T10:03:24.620633+0000","last_fullsized":"2023-10-16T10:03:24.620633+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:25.676058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638848+0000","last_change":"2023-10-16T10:03:23.636289+0000","last_active":"2023-10-16T10:03:23.638848+0000","last_peered":"2023-10-16T10:03:23.638848+0000","last_clean":"2023-10-16T10:03:23.638848+0000","last_became_active":"2023-10-16T10:03:23.635602+0000","last_became_peered":"2023-10-16T10:03:23.635602+0000","last_unstale":"2023-10-16T10:03:23.638848+0000","last_undegraded":"2023-10-16T10:03:23.638848+0000","last_fullsized":"2023-10-16T10:03:23.638848+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:22.363742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620898+0000","last_change":"2023-10-16T10:03:23.229780+0000","last_active":"2023-10-16T10:03:24.620898+0000","last_peered":"2023-10-16T10:03:24.620898+0000","last_clean":"2023-10-16T10:03:24.620898+0000","last_became_active":"2023-10-16T10:03:23.229479+0000","last_became_peered":"2023-10-16T10:03:23.229479+0000","last_unstale":"2023-10-16T10:03:24.620898+0000","last_undegraded":"2023-10-16T10:03:24.620898+0000","last_fullsized":"2023-10-16T10:03:24.620898+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:50.568227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.634256+0000","last_change":"2023-10-16T10:03:23.632424+0000","last_active":"2023-10-16T10:03:23.634256+0000","last_peered":"2023-10-16T10:03:23.634256+0000","last_clean":"2023-10-16T10:03:23.634256+0000","last_became_active":"2023-10-16T10:03:23.631635+0000","last_became_peered":"2023-10-16T10:03:23.631635+0000","last_unstale":"2023-10-16T10:03:23.634256+0000","last_undegraded":"2023-10-16T10:03:23.634256+0000","last_fullsized":"2023-10-16T10:03:23.634256+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:03.461277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215829+0000","last_change":"2023-10-16T10:03:23.225699+0000","last_active":"2023-10-16T10:03:24.215829+0000","last_peered":"2023-10-16T10:03:24.215829+0000","last_clean":"2023-10-16T10:03:24.215829+0000","last_became_active":"2023-10-16T10:03:23.225238+0000","last_became_peered":"2023-10-16T10:03:23.225238+0000","last_unstale":"2023-10-16T10:03:24.215829+0000","last_undegraded":"2023-10-16T10:03:24.215829+0000","last_fullsized":"2023-10-16T10:03:24.215829+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:29.577019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620531+0000","last_change":"2023-10-16T10:03:23.635488+0000","last_active":"2023-10-16T10:03:24.620531+0000","last_peered":"2023-10-16T10:03:24.620531+0000","last_clean":"2023-10-16T10:03:24.620531+0000","last_became_active":"2023-10-16T10:03:23.635206+0000","last_became_peered":"2023-10-16T10:03:23.635206+0000","last_unstale":"2023-10-16T10:03:24.620531+0000","last_undegraded":"2023-10-16T10:03:24.620531+0000","last_fullsized":"2023-10-16T10:03:24.620531+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:15.675607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620476+0000","last_change":"2023-10-16T10:03:23.216885+0000","last_active":"2023-10-16T10:03:24.620476+0000","last_peered":"2023-10-16T10:03:24.620476+0000","last_clean":"2023-10-16T10:03:24.620476+0000","last_became_active":"2023-10-16T10:03:23.216614+0000","last_became_peered":"2023-10-16T10:03:23.216614+0000","last_unstale":"2023-10-16T10:03:24.620476+0000","last_undegraded":"2023-10-16T10:03:24.620476+0000","last_fullsized":"2023-10-16T10:03:24.620476+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:14.158582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620914+0000","last_change":"2023-10-16T10:03:23.637515+0000","last_active":"2023-10-16T10:03:24.620914+0000","last_peered":"2023-10-16T10:03:24.620914+0000","last_clean":"2023-10-16T10:03:24.620914+0000","last_became_active":"2023-10-16T10:03:23.637232+0000","last_became_peered":"2023-10-16T10:03:23.637232+0000","last_unstale":"2023-10-16T10:03:24.620914+0000","last_undegraded":"2023-10-16T10:03:24.620914+0000","last_fullsized":"2023-10-16T10:03:24.620914+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:18.495717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620665+0000","last_change":"2023-10-16T10:03:23.229519+0000","last_active":"2023-10-16T10:03:24.620665+0000","last_peered":"2023-10-16T10:03:24.620665+0000","last_clean":"2023-10-16T10:03:24.620665+0000","last_became_active":"2023-10-16T10:03:23.229227+0000","last_became_peered":"2023-10-16T10:03:23.229227+0000","last_unstale":"2023-10-16T10:03:24.620665+0000","last_undegraded":"2023-10-16T10:03:24.620665+0000","last_fullsized":"2023-10-16T10:03:24.620665+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:29.577019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.638350+0000","last_change":"2023-10-16T10:03:23.635189+0000","last_active":"2023-10-16T10:03:23.638350+0000","last_peered":"2023-10-16T10:03:23.638350+0000","last_clean":"2023-10-16T10:03:23.638350+0000","last_became_active":"2023-10-16T10:03:23.634919+0000","last_became_peered":"2023-10-16T10:03:23.634919+0000","last_unstale":"2023-10-16T10:03:23.638350+0000","last_undegraded":"2023-10-16T10:03:23.638350+0000","last_fullsized":"2023-10-16T10:03:23.638350+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:23.556655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.619913+0000","last_change":"2023-10-16T10:03:23.220613+0000","last_active":"2023-10-16T10:03:24.619913+0000","last_peered":"2023-10-16T10:03:24.619913+0000","last_clean":"2023-10-16T10:03:24.619913+0000","last_became_active":"2023-10-16T10:03:23.220321+0000","last_became_peered":"2023-10-16T10:03:23.220321+0000","last_unstale":"2023-10-16T10:03:24.619913+0000","last_undegraded":"2023-10-16T10:03:24.619913+0000","last_fullsized":"2023-10-16T10:03:24.619913+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:22.535429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628992+0000","last_change":"2023-10-16T10:03:23.625806+0000","last_active":"2023-10-16T10:03:23.628992+0000","last_peered":"2023-10-16T10:03:23.628992+0000","last_clean":"2023-10-16T10:03:23.628992+0000","last_became_active":"2023-10-16T10:03:23.625558+0000","last_became_peered":"2023-10-16T10:03:23.625558+0000","last_unstale":"2023-10-16T10:03:23.628992+0000","last_undegraded":"2023-10-16T10:03:23.628992+0000","last_fullsized":"2023-10-16T10:03:23.628992+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:50.918218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.215833+0000","last_change":"2023-10-16T10:03:23.225473+0000","last_active":"2023-10-16T10:03:24.215833+0000","last_peered":"2023-10-16T10:03:24.215833+0000","last_clean":"2023-10-16T10:03:24.215833+0000","last_became_active":"2023-10-16T10:03:23.225074+0000","last_became_peered":"2023-10-16T10:03:23.225074+0000","last_unstale":"2023-10-16T10:03:24.215833+0000","last_undegraded":"2023-10-16T10:03:24.215833+0000","last_fullsized":"2023-10-16T10:03:24.215833+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:14.670066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.630685+0000","last_change":"2023-10-16T10:03:23.628852+0000","last_active":"2023-10-16T10:03:23.630685+0000","last_peered":"2023-10-16T10:03:23.630685+0000","last_clean":"2023-10-16T10:03:23.630685+0000","last_became_active":"2023-10-16T10:03:23.628584+0000","last_became_peered":"2023-10-16T10:03:23.628584+0000","last_unstale":"2023-10-16T10:03:23.630685+0000","last_undegraded":"2023-10-16T10:03:23.630685+0000","last_fullsized":"2023-10-16T10:03:23.630685+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:21.082401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.631040+0000","last_change":"2023-10-16T10:03:23.229601+0000","last_active":"2023-10-16T10:03:23.631040+0000","last_peered":"2023-10-16T10:03:23.631040+0000","last_clean":"2023-10-16T10:03:23.631040+0000","last_became_active":"2023-10-16T10:03:23.229326+0000","last_became_peered":"2023-10-16T10:03:23.229326+0000","last_unstale":"2023-10-16T10:03:23.631040+0000","last_undegraded":"2023-10-16T10:03:23.631040+0000","last_fullsized":"2023-10-16T10:03:23.631040+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:38:39.156686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620904+0000","last_change":"2023-10-16T10:03:23.625596+0000","last_active":"2023-10-16T10:03:24.620904+0000","last_peered":"2023-10-16T10:03:24.620904+0000","last_clean":"2023-10-16T10:03:24.620904+0000","last_became_active":"2023-10-16T10:03:23.625320+0000","last_became_peered":"2023-10-16T10:03:23.625320+0000","last_unstale":"2023-10-16T10:03:24.620904+0000","last_undegraded":"2023-10-16T10:03:24.620904+0000","last_fullsized":"2023-10-16T10:03:24.620904+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:43.611093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628649+0000","last_change":"2023-10-16T10:03:23.229738+0000","last_active":"2023-10-16T10:03:23.628649+0000","last_peered":"2023-10-16T10:03:23.628649+0000","last_clean":"2023-10-16T10:03:23.628649+0000","last_became_active":"2023-10-16T10:03:23.229413+0000","last_became_peered":"2023-10-16T10:03:23.229413+0000","last_unstale":"2023-10-16T10:03:23.628649+0000","last_undegraded":"2023-10-16T10:03:23.628649+0000","last_fullsized":"2023-10-16T10:03:23.628649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:58.591286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.637772+0000","last_change":"2023-10-16T10:03:23.636879+0000","last_active":"2023-10-16T10:03:23.637772+0000","last_peered":"2023-10-16T10:03:23.637772+0000","last_clean":"2023-10-16T10:03:23.637772+0000","last_became_active":"2023-10-16T10:03:23.636630+0000","last_became_peered":"2023-10-16T10:03:23.636630+0000","last_unstale":"2023-10-16T10:03:23.637772+0000","last_undegraded":"2023-10-16T10:03:23.637772+0000","last_fullsized":"2023-10-16T10:03:23.637772+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:07:28.068394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.619969+0000","last_change":"2023-10-16T10:03:23.225477+0000","last_active":"2023-10-16T10:03:23.619969+0000","last_peered":"2023-10-16T10:03:23.619969+0000","last_clean":"2023-10-16T10:03:23.619969+0000","last_became_active":"2023-10-16T10:03:23.225189+0000","last_became_peered":"2023-10-16T10:03:23.225189+0000","last_unstale":"2023-10-16T10:03:23.619969+0000","last_undegraded":"2023-10-16T10:03:23.619969+0000","last_fullsized":"2023-10-16T10:03:23.619969+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:53.306782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.628803+0000","last_change":"2023-10-16T10:03:23.627010+0000","last_active":"2023-10-16T10:03:23.628803+0000","last_peered":"2023-10-16T10:03:23.628803+0000","last_clean":"2023-10-16T10:03:23.628803+0000","last_became_active":"2023-10-16T10:03:23.626759+0000","last_became_peered":"2023-10-16T10:03:23.626759+0000","last_unstale":"2023-10-16T10:03:23.628803+0000","last_undegraded":"2023-10-16T10:03:23.628803+0000","last_fullsized":"2023-10-16T10:03:23.628803+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:43.940745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621650+0000","last_change":"2023-10-16T10:03:23.219621+0000","last_active":"2023-10-16T10:03:24.621650+0000","last_peered":"2023-10-16T10:03:24.621650+0000","last_clean":"2023-10-16T10:03:24.621650+0000","last_became_active":"2023-10-16T10:03:23.219193+0000","last_became_peered":"2023-10-16T10:03:23.219193+0000","last_unstale":"2023-10-16T10:03:24.621650+0000","last_undegraded":"2023-10-16T10:03:24.621650+0000","last_fullsized":"2023-10-16T10:03:24.621650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:21.795135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620580+0000","last_change":"2023-10-16T10:03:23.636431+0000","last_active":"2023-10-16T10:03:24.620580+0000","last_peered":"2023-10-16T10:03:24.620580+0000","last_clean":"2023-10-16T10:03:24.620580+0000","last_became_active":"2023-10-16T10:03:23.636170+0000","last_became_peered":"2023-10-16T10:03:23.636170+0000","last_unstale":"2023-10-16T10:03:24.620580+0000","last_undegraded":"2023-10-16T10:03:24.620580+0000","last_fullsized":"2023-10-16T10:03:24.620580+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:43.940745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.621640+0000","last_change":"2023-10-16T10:03:23.214188+0000","last_active":"2023-10-16T10:03:24.621640+0000","last_peered":"2023-10-16T10:03:24.621640+0000","last_clean":"2023-10-16T10:03:24.621640+0000","last_became_active":"2023-10-16T10:03:23.213934+0000","last_became_peered":"2023-10-16T10:03:23.213934+0000","last_unstale":"2023-10-16T10:03:24.621640+0000","last_undegraded":"2023-10-16T10:03:24.621640+0000","last_fullsized":"2023-10-16T10:03:24.621640+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:38:39.156686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620417+0000","last_change":"2023-10-16T10:03:23.623294+0000","last_active":"2023-10-16T10:03:24.620417+0000","last_peered":"2023-10-16T10:03:24.620417+0000","last_clean":"2023-10-16T10:03:24.620417+0000","last_became_active":"2023-10-16T10:03:23.623038+0000","last_became_peered":"2023-10-16T10:03:23.623038+0000","last_unstale":"2023-10-16T10:03:24.620417+0000","last_undegraded":"2023-10-16T10:03:24.620417+0000","last_fullsized":"2023-10-16T10:03:24.620417+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:52.774556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.619972+0000","last_change":"2023-10-16T10:03:23.224056+0000","last_active":"2023-10-16T10:03:23.619972+0000","last_peered":"2023-10-16T10:03:23.619972+0000","last_clean":"2023-10-16T10:03:23.619972+0000","last_became_active":"2023-10-16T10:03:23.223772+0000","last_became_peered":"2023-10-16T10:03:23.223772+0000","last_unstale":"2023-10-16T10:03:23.619972+0000","last_undegraded":"2023-10-16T10:03:23.619972+0000","last_fullsized":"2023-10-16T10:03:23.619972+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:37.765844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.624703+0000","last_change":"2023-10-16T10:03:23.621851+0000","last_active":"2023-10-16T10:03:23.624703+0000","last_peered":"2023-10-16T10:03:23.624703+0000","last_clean":"2023-10-16T10:03:23.624703+0000","last_became_active":"2023-10-16T10:03:23.621575+0000","last_became_peered":"2023-10-16T10:03:23.621575+0000","last_unstale":"2023-10-16T10:03:23.624703+0000","last_undegraded":"2023-10-16T10:03:23.624703+0000","last_fullsized":"2023-10-16T10:03:23.624703+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:47.998013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.619913+0000","last_change":"2023-10-16T10:03:23.222546+0000","last_active":"2023-10-16T10:03:24.619913+0000","last_peered":"2023-10-16T10:03:24.619913+0000","last_clean":"2023-10-16T10:03:24.619913+0000","last_became_active":"2023-10-16T10:03:23.222265+0000","last_became_peered":"2023-10-16T10:03:23.222265+0000","last_unstale":"2023-10-16T10:03:24.619913+0000","last_undegraded":"2023-10-16T10:03:24.619913+0000","last_fullsized":"2023-10-16T10:03:24.619913+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:49:51.860147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620343+0000","last_change":"2023-10-16T10:03:23.623268+0000","last_active":"2023-10-16T10:03:24.620343+0000","last_peered":"2023-10-16T10:03:24.620343+0000","last_clean":"2023-10-16T10:03:24.620343+0000","last_became_active":"2023-10-16T10:03:23.622806+0000","last_became_peered":"2023-10-16T10:03:23.622806+0000","last_unstale":"2023-10-16T10:03:24.620343+0000","last_undegraded":"2023-10-16T10:03:24.620343+0000","last_fullsized":"2023-10-16T10:03:24.620343+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:23.197035+0000","last_clean_scrub_stamp":"2023-10-16T10:03:23.197035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:09.567991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"139'8","reported_seq":20,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-16T10:03:24.620528+0000","last_change":"2023-10-16T10:03:23.218690+0000","last_active":"2023-10-16T10:03:24.620528+0000","last_peered":"2023-10-16T10:03:24.620528+0000","last_clean":"2023-10-16T10:03:24.620528+0000","last_became_active":"2023-10-16T10:03:23.218391+0000","last_became_peered":"2023-10-16T10:03:23.218391+0000","last_unstale":"2023-10-16T10:03:24.620528+0000","last_undegraded":"2023-10-16T10:03:24.620528+0000","last_fullsized":"2023-10-16T10:03:24.620528+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:03:22.191495+0000","last_clean_scrub_stamp":"2023-10-16T10:03:22.191495+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:51.769015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"129'227","reported_seq":673,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-16T10:03:23.617552+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:03:23.617552+0000","last_peered":"2023-10-16T10:03:23.617552+0000","last_clean":"2023-10-16T10:03:23.617552+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:03:23.617552+0000","last_undegraded":"2023-10-16T10:03:23.617552+0000","last_fullsized":"2023-10-16T10:03:23.617552+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59824,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312016,"statfs":{"total":96636764160,"available":96575504384,"internally_reserved":0,"allocated":2322432,"data_stored":1496900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.25900000000000001},"max":{"1min":0.747,"5min":0.76500000000000001,"15min":1.2150000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.378,"5min":0.33500000000000002,"15min":0.254},"max":{"1min":0.745,"5min":1.911,"15min":1.911},"last":0.66600000000000004}]},{"osd":1,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.60099999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.29199999999999998},"max":{"1min":0.76300000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.41499999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78200000000000003,"5min":0.80500000000000005,"15min":1.1279999999999999},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.624,"15min":0.63100000000000001},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78800000000000003,"5min":3.3820000000000001,"15min":3.3820000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80400000000000005,"5min":3.1640000000000001,"15min":3.1640000000000001},"last":0.622}]},{"osd":3,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.373,"5min":0.33800000000000002,"15min":0.32300000000000001},"max":{"1min":0.81499999999999995,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83099999999999996,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32800000000000001,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.752,"5min":1.8220000000000001,"15min":1.9550000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.55300000000000005,"15min":0.52300000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27200000000000002},"max":{"1min":0.69099999999999995,"5min":3.8149999999999999,"15min":3.8149999999999999},"last":0.37}]},{"osd":5,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.84099999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.37,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":1.6619999999999999,"15min":2.0150000000000001},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84599999999999997,"5min":1.7110000000000001,"15min":1.9870000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.42299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575085,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1732608,"data_stored":907114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55600000000000005,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.29699999999999999,"15min":0.223},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.0920000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.43099999999999999,"5min":0.30599999999999999,"15min":0.249},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.82099999999999995},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71799999999999997,"5min":0.83699999999999997,"15min":0.90000000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.83199999999999996},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.78200000000000003,"15min":1.448},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.60299999999999998},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":0.874,"15min":1.708},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58299999999999996,"15min":0.58799999999999997},"min":{"1min":0.39000000000000001,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":0.71799999999999997,"5min":0.79700000000000004,"15min":0.79900000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56599999999999995,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.36899999999999999,"15min":0.315},"max":{"1min":0.78600000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.47899999999999998},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.249},"max":{"1min":0.754,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.51300000000000001},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.27500000000000002},"max":{"1min":0.80800000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.505,"15min":0.503},"min":{"1min":0.33900000000000002,"5min":0.308,"15min":0.253},"max":{"1min":0.73199999999999998,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.311},"max":{"1min":0.79400000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Oct 16 10:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.37,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.91900000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.42799999999999999,"5min":0.35799999999999998,"15min":0.33800000000000002},"max":{"1min":0.79200000000000004,"5min":0.80500000000000005,"15min":0.94599999999999995},"last":0.52400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1732608,"data_stored":906554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36099999999999999,"15min":0.39100000000000001},"min":{"1min":0.28000000000000003,"5min":0.251,"15min":0.188},"max":{"1min":0.504,"5min":0.54000000000000004,"15min":1.6000000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41399999999999998,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.23200000000000001},"max":{"1min":0.61599999999999999,"5min":0.62,"15min":0.70199999999999996},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.376,"15min":0.40799999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.20799999999999999},"max":{"1min":0.496,"5min":0.98699999999999999,"15min":4.923},"last":0.312},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54000000000000004,"5min":0.65800000000000003,"15min":1.712},"last":0.28000000000000003}]},{"osd":3,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.438,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65700000000000003,"5min":0.68999999999999995,"15min":1.984},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41699999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.26500000000000001,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.623},"min":{"1min":0.42899999999999999,"5min":0.39900000000000002,"15min":0.377},"max":{"1min":0.78200000000000003,"5min":0.87,"15min":0.93100000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.34399999999999997},"max":{"1min":0.73899999999999999,"5min":1.0840000000000001,"15min":5.0069999999999997},"last":0.502}]},{"osd":5,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.623,"15min":0.65800000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84799999999999998,"5min":1.04,"15min":4.9749999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63200000000000001,"15min":0.66700000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.751,"5min":1.054,"15min":1.7769999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61099999999999999,"15min":0.63700000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.90800000000000003,"5min":0.95199999999999996,"15min":1.0009999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.879,"5min":1.3029999999999999,"15min":5.0419999999999998},"last":0.42199999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61099999999999999,"15min":0.65500000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84699999999999998,"5min":0.96599999999999997,"15min":1.744},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.622,"15min":0.64600000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.83599999999999997,"5min":0.91100000000000003,"15min":0.91400000000000003},"last":0.47199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37616,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334224,"statfs":{"total":96636764160,"available":96598245376,"internally_reserved":0,"allocated":2326528,"data_stored":1496922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.38300000000000001},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.192},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":1.2110000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.42999999999999999},"min":{"1min":0.28399999999999997,"5min":0.27600000000000002,"15min":0.26500000000000001},"max":{"1min":0.55900000000000005,"5min":0.70999999999999996,"15min":1.3169999999999999},"last":0.439}]},{"osd":2,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40699999999999997,"15min":0.41699999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.70099999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.44,"15min":0.46000000000000002},"min":{"1min":0.32700000000000001,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":0.61199999999999999,"5min":0.78100000000000003,"15min":0.79900000000000004},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.40000000000000002,"15min":0.41599999999999998},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.83399999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.44500000000000001,"15min":0.46400000000000002},"min":{"1min":0.29399999999999998,"5min":0.27700000000000002,"15min":0.23300000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":1.1859999999999999},"last":0.499}]},{"osd":4,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60899999999999999,"15min":0.621},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.33700000000000002},"max":{"1min":0.83999999999999997,"5min":0.89400000000000002,"15min":0.93700000000000006},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60199999999999998,"15min":0.629},"min":{"1min":0.378,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":1.1659999999999999},"last":0.86499999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.39200000000000002,"15min":0.33300000000000002},"max":{"1min":0.86499999999999999,"5min":0.94999999999999996,"15min":0.99299999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.629},"min":{"1min":0.438,"5min":0.41799999999999998,"15min":0.311},"max":{"1min":0.85799999999999998,"5min":0.89100000000000001,"15min":0.91000000000000003},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.40899999999999997},"max":{"1min":0.88,"5min":0.90900000000000003,"15min":1.2410000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.66700000000000004},"min":{"1min":0.45000000000000001,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.86599999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.71699999999999997}]},{"osd":7,"last update":"Mon Oct 16 10:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.316},"max":{"1min":0.89200000000000002,"5min":1.014,"15min":1.02},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.64300000000000002},"min":{"1min":0.41299999999999998,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.90900000000000003},"last":0.79300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575085,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59236,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312604,"statfs":{"total":96636764160,"available":96576106496,"internally_reserved":0,"allocated":1728512,"data_stored":906532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.35899999999999999,"15min":0.374},"min":{"1min":0.25,"5min":0.217,"15min":0.217},"max":{"1min":0.51100000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.255},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.41199999999999998},"min":{"1min":0.26500000000000001,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.60099999999999998,"5min":0.61499999999999999,"15min":0.65400000000000003},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41299999999999998,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.754},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.38300000000000001,"15min":0.39400000000000002},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.22700000000000001},"max":{"1min":0.57099999999999995,"5min":0.624,"15min":0.624},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40200000000000002,"15min":0.41399999999999998},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.23499999999999999},"max":{"1min":0.50900000000000001,"5min":0.57599999999999996,"15min":0.63500000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.47699999999999998},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.25600000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.79800000000000004},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60699999999999998,"15min":0.63600000000000001},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36799999999999999},"max":{"1min":0.86699999999999999,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.33400000000000002},"max":{"1min":0.83199999999999996,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.41599999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83999999999999997,"5min":1.0249999999999999,"15min":1.103},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.622,"15min":0.64700000000000002},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.38400000000000001},"max":{"1min":0.75900000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.42099999999999999,"5min":0.39900000000000002,"15min":0.39800000000000002},"max":{"1min":0.83899999999999997,"5min":1.046,"15min":1.1699999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.63400000000000001,"15min":0.66600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76200000000000001,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 10:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65400000000000003,"15min":0.67700000000000005},"min":{"1min":0.51500000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79700000000000004,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.42599999999999999,"15min":0.371},"max":{"1min":0.79500000000000004,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.56899999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575085,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1744896,"data_stored":908178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34999999999999998,"15min":0.36899999999999999},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.17999999999999999},"max":{"1min":0.5,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.375,"15min":0.38800000000000001},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.45800000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.311}]},{"osd":1,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.39100000000000001},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.65400000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51100000000000001,"5min":0.60899999999999999,"15min":0.67500000000000004},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44,"15min":0.46200000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.53400000000000003,"5min":0.96099999999999997,"15min":1.0760000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.48699999999999999},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.56100000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.77900000000000003,"5min":0.90300000000000002,"15min":0.96499999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60499999999999998,"15min":0.63300000000000001},"min":{"1min":0.432,"5min":0.39600000000000002,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.63100000000000001},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.875,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.498},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.623,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.436}]},{"osd":6,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60299999999999998,"15min":0.64000000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.374},"max":{"1min":0.75900000000000001,"5min":0.88100000000000001,"15min":0.98799999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60699999999999998,"15min":0.64800000000000002},"min":{"1min":0.438,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84299999999999997,"5min":0.92600000000000005,"15min":1.0589999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63800000000000001,"15min":0.66400000000000003},"min":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.878,"5min":0.90600000000000003,"15min":1.0029999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.734,"5min":0.83799999999999997,"15min":0.72399999999999998},"min":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.36899999999999999},"max":{"1min":24.861000000000001,"5min":24.861000000000001,"15min":24.861000000000001},"last":0.56299999999999994}]}]},{"osd":4,"up_from":13,"seq":55834575085,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1753088,"data_stored":915216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":1.0269999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57399999999999995,"15min":0.58999999999999997},"min":{"1min":0.433,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":1.0860000000000001},"last":0.626}]},{"osd":1,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.58699999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.312},"max":{"1min":0.753,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56200000000000006,"15min":0.58499999999999996},"min":{"1min":0.44600000000000001,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":1.6359999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.85199999999999998,"15min":0.88300000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59399999999999997,"15min":0.61399999999999999},"min":{"1min":0.46899999999999997,"5min":0.38600000000000001,"15min":0.32500000000000001},"max":{"1min":0.85099999999999998,"5min":0.90500000000000003,"15min":1.2270000000000001},"last":0.503}]},{"osd":3,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.68300000000000005,"5min":0.77700000000000002,"15min":0.86099999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59799999999999998,"15min":0.61199999999999999},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.34200000000000003},"max":{"1min":0.83999999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.49299999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.39900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.69699999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.31},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51400000000000001,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.26400000000000001,"15min":0.26300000000000001},"max":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.96599999999999997},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53000000000000003},"min":{"1min":0.44500000000000001,"5min":0.29599999999999999,"15min":0.24199999999999999},"max":{"1min":0.69799999999999995,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.45000000000000001,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":0.79200000000000004,"5min":0.84899999999999998,"15min":0.92400000000000004},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.82399999999999995,"5min":0.83099999999999996,"15min":0.91100000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.625,"15min":0.61499999999999999},"min":{"1min":0.38900000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78900000000000003,"5min":0.89400000000000002,"15min":1.0429999999999999},"last":0.38700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50932,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320908,"statfs":{"total":96636764160,"available":96584609792,"internally_reserved":0,"allocated":1736704,"data_stored":906692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28599999999999998},"max":{"1min":0.81000000000000005,"5min":0.84199999999999997,"15min":1.792},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.34599999999999997,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":0.79600000000000004,"5min":0.85799999999999998,"15min":1.7090000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":0.70099999999999996,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.59999999999999998}]},{"osd":2,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25},"max":{"1min":0.753,"5min":0.80800000000000005,"15min":1.496},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.59599999999999997},"min":{"1min":0.497,"5min":0.39200000000000002,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.879,"15min":2.069},"last":0.63100000000000001}]},{"osd":3,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1,"5min":1,"15min":1},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.33000000000000002},"max":{"1min":0.79800000000000004,"5min":0.89100000000000001,"15min":1.7689999999999999},"last":0.65100000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.27600000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.84199999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.50700000000000001},"min":{"1min":0.40000000000000002,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.86899999999999999},"last":0.38100000000000001}]},{"osd":6,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.33300000000000002,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":0.83199999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.88200000000000001},"last":0.49099999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.25900000000000001},"max":{"1min":0.81599999999999995,"5min":0.82399999999999995,"15min":0.90300000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.624,"15min":0.61599999999999999},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.88,"5min":0.88,"15min":0.88500000000000001},"last":0.47299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:03:28.731 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T10:03:28.732 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T10:03:28.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.892+0000 7f9034e7c700 1 Processor -- start 2023-10-16T10:03:28.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.892+0000 7f9034e7c700 1 -- start start 2023-10-16T10:03:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f9034e7c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90300af5f0 0x7f9030161e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f9034e7c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90300b3b60 0x7f90300724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f9034e7c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9030072a20 0x7f903006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f9034e7c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9030071d70 con 0x7f90300b3b60 2023-10-16T10:03:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f9034e7c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90300af9d0 con 0x7f90300af5f0 2023-10-16T10:03:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f9034e7c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90300b3970 con 0x7f9030072a20 2023-10-16T10:03:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90300b3b60 0x7f90300724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90300b3b60 0x7f90300724e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58232/0 (socket says 172.21.15.158:58232) 2023-10-16T10:03:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 -- 172.21.15.158:0/141432140 learned_addr learned my addr 172.21.15.158:0/141432140 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 -- 172.21.15.158:0/141432140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9030072a20 msgr2=0x7f903006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 --2- 172.21.15.158:0/141432140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9030072a20 0x7f903006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 -- 172.21.15.158:0/141432140 >> v2:172.21.15.172:3300/0 conn(0x7f90300af5f0 msgr2=0x7f9030161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902e59c700 1 --2- 172.21.15.158:0/141432140 >> v2:172.21.15.172:3300/0 conn(0x7f90300af5f0 0x7f9030161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 --2- 172.21.15.158:0/141432140 >> v2:172.21.15.172:3300/0 conn(0x7f90300af5f0 0x7f9030161e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 -- 172.21.15.158:0/141432140 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030162370 con 0x7f90300b3b60 2023-10-16T10:03:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.893+0000 7f902dd9b700 1 --2- 172.21.15.158:0/141432140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300724e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f903006cd50 tx=0x7f9020009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51370f2fa6b10c22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f902d59a700 1 -- 172.21.15.158:0/141432140 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000b870 con 0x7f90300b3b60 2023-10-16T10:03:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f902d59a700 1 -- 172.21.15.158:0/141432140 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902000b9d0 con 0x7f90300b3b60 2023-10-16T10:03:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f902d59a700 1 -- 172.21.15.158:0/141432140 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000bd60 con 0x7f90300b3b60 2023-10-16T10:03:28.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 -- 172.21.15.158:0/141432140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 msgr2=0x7f90300724e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 --2- 172.21.15.158:0/141432140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300724e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f903006cd50 tx=0x7f9020009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 -- 172.21.15.158:0/141432140 shutdown_connections 2023-10-16T10:03:28.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 --2- 172.21.15.158:0/141432140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9030072a20 0x7f903006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 --2- 172.21.15.158:0/141432140 >> v2:172.21.15.172:3300/0 conn(0x7f90300af5f0 0x7f9030161e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 --2- 172.21.15.158:0/141432140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300724e0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 -- 172.21.15.158:0/141432140 >> 172.21.15.158:0/141432140 conn(0x7f90300ccd30 msgr2=0x7f90300cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:28.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 -- 172.21.15.158:0/141432140 shutdown_connections 2023-10-16T10:03:28.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.894+0000 7f9034e7c700 1 -- 172.21.15.158:0/141432140 wait complete. 2023-10-16T10:03:28.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.896+0000 7f9034e7c700 1 Processor -- start 2023-10-16T10:03:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.896+0000 7f9034e7c700 1 -- start start 2023-10-16T10:03:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f9034e7c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9030072a20 0x7f903015fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f9034e7c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90300af5f0 0x7f90301f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f9034e7c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f9034e7c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9030160010 con 0x7f90300b3b60 2023-10-16T10:03:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f9034e7c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90301f3690 con 0x7f9030072a20 2023-10-16T10:03:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f9034e7c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90300c8960 con 0x7f90300af5f0 2023-10-16T10:03:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58240/0 (socket says 172.21.15.158:58240) 2023-10-16T10:03:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 -- 172.21.15.158:0/108125453 learned_addr learned my addr 172.21.15.158:0/108125453 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902e59c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9030072a20 0x7f903015fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902dd9b700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90300af5f0 0x7f90301f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 -- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90300af5f0 msgr2=0x7f90301f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90300af5f0 0x7f90301f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 -- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9030072a20 msgr2=0x7f903015fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9030072a20 0x7f903015fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.897+0000 7f902ed9d700 1 -- 172.21.15.158:0/108125453 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9020009550 con 0x7f90300b3b60 2023-10-16T10:03:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.898+0000 7f902ed9d700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300b66e0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f902800b9a0 tx=0x7f902800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.898+0000 7f901effd700 1 -- 172.21.15.158:0/108125453 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800d850 con 0x7f90300b3b60 2023-10-16T10:03:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.898+0000 7f901effd700 1 -- 172.21.15.158:0/108125453 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902800d9d0 con 0x7f90300b3b60 2023-10-16T10:03:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.898+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90300b6c20 con 0x7f90300b3b60 2023-10-16T10:03:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.898+0000 7f901effd700 1 -- 172.21.15.158:0/108125453 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800dd80 con 0x7f90300b3b60 2023-10-16T10:03:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.898+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90301f3b90 con 0x7f90300b3b60 2023-10-16T10:03:28.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.899+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90300681b0 con 0x7f90300b3b60 2023-10-16T10:03:28.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.899+0000 7f901effd700 1 -- 172.21.15.158:0/108125453 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f902800e420 con 0x7f90300b3b60 2023-10-16T10:03:28.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.900+0000 7f901effd700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f901006b020 0x7f901006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:28.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.900+0000 7f901effd700 1 -- 172.21.15.158:0/108125453 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f9028006e40 con 0x7f90300b3b60 2023-10-16T10:03:28.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.900+0000 7f902e59c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f901006b020 0x7f901006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:28.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.900+0000 7f902e59c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f901006b020 0x7f901006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90300c6df0 tx=0x7f9024008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:28.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:28.904+0000 7f901effd700 1 -- 172.21.15.158:0/108125453 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9028054830 con 0x7f90300b3b60 2023-10-16T10:03:29.085 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:03:29.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:03:29.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.177+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f90300cd960 con 0x7f90300b3b60 2023-10-16T10:03:29.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.180+0000 7f901effd700 1 -- 172.21.15.158:0/108125453 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9028057e50 con 0x7f90300b3b60 2023-10-16T10:03:29.181 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T10:03:29.181 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T10:03:29.181 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T10:03:29.182 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T10:03:29.182 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T10:03:29.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f901006b020 msgr2=0x7f901006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f901006b020 0x7f901006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90300c6df0 tx=0x7f9024008040 comp rx=0 tx=0).stop 2023-10-16T10:03:29.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 msgr2=0x7f90300b66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300b66e0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f902800b9a0 tx=0x7f902800bcb0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 shutdown_connections 2023-10-16T10:03:29.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f901006b020 0x7f901006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90300b3b60 0x7f90300b66e0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9030072a20 0x7f903015fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 --2- 172.21.15.158:0/108125453 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90300af5f0 0x7f90301f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 >> 172.21.15.158:0/108125453 conn(0x7f90300ccd30 msgr2=0x7f90300ad4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:29.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 shutdown_connections 2023-10-16T10:03:29.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.182+0000 7f9034e7c700 1 -- 172.21.15.158:0/108125453 wait complete. 2023-10-16T10:03:29.187 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T10:03:29.191 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T10:03:29.190+0000 7f483bbe3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:03:29.196 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:29.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.357+0000 7fbbc1aec700 1 Processor -- start 2023-10-16T10:03:29.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.357+0000 7fbbc1aec700 1 -- start start 2023-10-16T10:03:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbc1aec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbbbc15dfb0 0x7fbbbc06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbc1aec700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbbc06e770 con 0x7fbbbc06f340 2023-10-16T10:03:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbc1aec700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbbc0cbe00 con 0x7fbbbc0cc5d0 2023-10-16T10:03:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbc1aec700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbbc0cf270 con 0x7fbbbc15dfb0 2023-10-16T10:03:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbb3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbbbc15dfb0 0x7fbbbc06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbb3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbbbc15dfb0 0x7fbbbc06e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58248/0 (socket says 172.21.15.158:58248) 2023-10-16T10:03:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbb3fff700 1 -- 172.21.15.158:0/2877246381 learned_addr learned my addr 172.21.15.158:0/2877246381 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbbbfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbbbc06f340 0x7fbbbc071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58426/0 (socket says 172.21.15.158:58426) 2023-10-16T10:03:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbb3fff700 1 -- 172.21.15.158:0/2877246381 >> v1:172.21.15.172:6789/0 conn(0x7fbbbc0cc5d0 legacy=0x7fbbbc0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbb3fff700 1 -- 172.21.15.158:0/2877246381 >> v1:172.21.15.158:6789/0 conn(0x7fbbbc06f340 legacy=0x7fbbbc071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbb3fff700 1 -- 172.21.15.158:0/2877246381 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc15e4e0 con 0x7fbbbc15dfb0 2023-10-16T10:03:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbbaffd700 1 -- 172.21.15.158:0/2877246381 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014256178 0 0) 0x7fbbb4001490 con 0x7fbbbc06f340 2023-10-16T10:03:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.358+0000 7fbbb3fff700 1 --2- 172.21.15.158:0/2877246381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc15dfb0 0x7fbbbc06e230 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fbba40098f0 tx=0x7fbba4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38fd4d1817e8507 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbbaffd700 1 -- 172.21.15.158:0/2877246381 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba4004230 con 0x7fbbbc15dfb0 2023-10-16T10:03:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbbaffd700 1 -- 172.21.15.158:0/2877246381 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba4004390 con 0x7fbbbc15dfb0 2023-10-16T10:03:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbbaffd700 1 -- 172.21.15.158:0/2877246381 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40036a0 con 0x7fbbbc15dfb0 2023-10-16T10:03:29.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbc1aec700 1 -- 172.21.15.158:0/2877246381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc15dfb0 msgr2=0x7fbbbc06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/2877246381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc15dfb0 0x7fbbbc06e230 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fbba40098f0 tx=0x7fbba4009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:29.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbc1aec700 1 -- 172.21.15.158:0/2877246381 shutdown_connections 2023-10-16T10:03:29.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/2877246381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc15dfb0 0x7fbbbc06e230 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbc1aec700 1 -- 172.21.15.158:0/2877246381 >> 172.21.15.158:0/2877246381 conn(0x7fbbbc0d02b0 msgr2=0x7fbbbc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:29.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbc1aec700 1 -- 172.21.15.158:0/2877246381 shutdown_connections 2023-10-16T10:03:29.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.359+0000 7fbbc1aec700 1 -- 172.21.15.158:0/2877246381 wait complete. 2023-10-16T10:03:29.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.361+0000 7fbbc1aec700 1 Processor -- start 2023-10-16T10:03:29.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.361+0000 7fbbc1aec700 1 -- start start 2023-10-16T10:03:29.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbc1aec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc06f340 0x7fbbbc143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:29.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbc1aec700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbbc0cc5d0 0x7fbbbc1f0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbc1aec700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbbc15dfb0 0x7fbbbc0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbc1aec700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc144200 con 0x7fbbbc06f340 2023-10-16T10:03:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbc1aec700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc1f48c0 con 0x7fbbbc15dfb0 2023-10-16T10:03:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbc1aec700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc1f4ba0 con 0x7fbbbc0cc5d0 2023-10-16T10:03:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc06f340 0x7fbbbc143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc06f340 0x7fbbbc143cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58252/0 (socket says 172.21.15.158:58252) 2023-10-16T10:03:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 -- 172.21.15.158:0/512792654 learned_addr learned my addr 172.21.15.158:0/512792654 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbb3fff700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbbc0cc5d0 0x7fbbbc1f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbbfff700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbbc15dfb0 0x7fbbbc0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:29.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 -- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbbc0cc5d0 msgr2=0x7fbbbc1f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbbc0cc5d0 0x7fbbbc1f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 -- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbbc15dfb0 msgr2=0x7fbbbc0c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbbc15dfb0 0x7fbbbc0c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 -- 172.21.15.158:0/512792654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4009550 con 0x7fbbbc06f340 2023-10-16T10:03:29.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbb3fff700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbbc0cc5d0 0x7fbbbc1f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:29.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbbb7fe700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc06f340 0x7fbbbc143cc0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00b4a0 tx=0x7fbbac00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:29.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.362+0000 7fbbb8ff9700 1 -- 172.21.15.158:0/512792654 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac011830 con 0x7fbbbc06f340 2023-10-16T10:03:29.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.363+0000 7fbbb8ff9700 1 -- 172.21.15.158:0/512792654 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac011990 con 0x7fbbbc06f340 2023-10-16T10:03:29.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.363+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc1f4e80 con 0x7fbbbc06f340 2023-10-16T10:03:29.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.363+0000 7fbbb8ff9700 1 -- 172.21.15.158:0/512792654 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac011d40 con 0x7fbbbc06f340 2023-10-16T10:03:29.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.363+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc1f0ba0 con 0x7fbbbc06f340 2023-10-16T10:03:29.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.363+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc0681b0 con 0x7fbbbc06f340 2023-10-16T10:03:29.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.364+0000 7fbbb8ff9700 1 -- 172.21.15.158:0/512792654 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbbac01a400 con 0x7fbbbc06f340 2023-10-16T10:03:29.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.364+0000 7fbbb8ff9700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbb9c06b020 0x7fbb9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:29.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.364+0000 7fbbb8ff9700 1 -- 172.21.15.158:0/512792654 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fbbac0896b0 con 0x7fbbbc06f340 2023-10-16T10:03:29.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.365+0000 7fbbb3fff700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbb9c06b020 0x7fbb9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:29.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.365+0000 7fbbb3fff700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbb9c06b020 0x7fbb9c06d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbba40098f0 tx=0x7fbba4008030 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:29.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.369+0000 7fbbb8ff9700 1 -- 172.21.15.158:0/512792654 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbac054480 con 0x7fbbbc06f340 2023-10-16T10:03:29.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.606+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbbc0b6230 con 0x7fbbbc06f340 2023-10-16T10:03:29.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.607+0000 7fbbb8ff9700 1 -- 172.21.15.158:0/512792654 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fbbac057aa0 con 0x7fbbbc06f340 2023-10-16T10:03:29.608 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:29.608 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4144904682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4144904682},{"type":"v1","addr":"172.21.15.172:6834","nonce":4144904682}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3562807630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3562807630},{"type":"v1","addr":"172.21.15.158:6836","nonce":3562807630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8805,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/614819185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":614819185},{"type":"v1","addr":"172.21.15.158:6837","nonce":614819185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:03:23.609567+0000","modified":"2023-10-16T10:03:26.633779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/4142092863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4142092863},{"type":"v1","addr":"172.21.15.172:6835","nonce":4142092863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T10:03:29.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbb9c06b020 msgr2=0x7fbb9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbb9c06b020 0x7fbb9c06d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbba40098f0 tx=0x7fbba4008030 comp rx=0 tx=0).stop 2023-10-16T10:03:29.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc06f340 msgr2=0x7fbbbc143cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:29.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc06f340 0x7fbbbc143cc0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00b4a0 tx=0x7fbbac00b860 comp rx=0 tx=0).stop 2023-10-16T10:03:29.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 shutdown_connections 2023-10-16T10:03:29.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbb9c06b020 0x7fbb9c06d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbbc15dfb0 0x7fbbbc0c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbbc06f340 0x7fbbbc143cc0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 --2- 172.21.15.158:0/512792654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbbbc0cc5d0 0x7fbbbc1f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:29.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 >> 172.21.15.158:0/512792654 conn(0x7fbbbc0d02b0 msgr2=0x7fbbbc06f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 shutdown_connections 2023-10-16T10:03:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:29.609+0000 7fbbc1aec700 1 -- 172.21.15.158:0/512792654 wait complete. 2023-10-16T10:03:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 68 2023-10-16T10:03:30.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T10:03:23.609567+0000', 'modified': '2023-10-16T10:03:26.633779+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8817}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8817': {'gid': 8817, 'name': 'b', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/4142092863', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 4142092863}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 4142092863}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T10:03:30.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T10:03:30.027 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T10:03:30.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.148+0000 7fbf434a4700 1 Processor -- start 2023-10-16T10:03:30.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.148+0000 7fbf434a4700 1 -- start start 2023-10-16T10:03:30.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf434a4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf3c0cfad0 0x7fbf3c0cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf434a4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c15f990 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf434a4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c15fc70 con 0x7fbf3c15eb10 2023-10-16T10:03:30.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf434a4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c0cdef0 con 0x7fbf3c0cfad0 2023-10-16T10:03:30.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf41a41700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf3c0cfad0 0x7fbf3c0cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf41a41700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf3c0cfad0 0x7fbf3c0cff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58264/0 (socket says 172.21.15.158:58264) 2023-10-16T10:03:30.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf40a3f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbf3c0d0ae0 0x7fbf3c0b4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58438/0 (socket says 172.21.15.158:58438) 2023-10-16T10:03:30.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf41a41700 1 -- 172.21.15.158:0/3900855213 learned_addr learned my addr 172.21.15.158:0/3900855213 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf41a41700 1 -- 172.21.15.158:0/3900855213 >> v1:172.21.15.172:6789/0 conn(0x7fbf3c15eb10 legacy=0x7fbf3c15f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf41a41700 1 -- 172.21.15.158:0/3900855213 >> v1:172.21.15.158:6789/0 conn(0x7fbf3c0d0ae0 legacy=0x7fbf3c0b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf33fff700 1 -- 172.21.15.158:0/3900855213 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678176422 0 0) 0x7fbf3c15f990 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.149+0000 7fbf41a41700 1 -- 172.21.15.158:0/3900855213 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c15ff50 con 0x7fbf3c0cfad0 2023-10-16T10:03:30.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf41a41700 1 --2- 172.21.15.158:0/3900855213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0cfad0 0x7fbf3c0cff60 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009e50 tx=0x7fbf3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ad6b4a960a8a29 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf33fff700 1 -- 172.21.15.158:0/3900855213 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3400e070 con 0x7fbf3c0cfad0 2023-10-16T10:03:30.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf33fff700 1 -- 172.21.15.158:0/3900855213 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34003a20 con 0x7fbf3c0cfad0 2023-10-16T10:03:30.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf33fff700 1 -- 172.21.15.158:0/3900855213 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34003dd0 con 0x7fbf3c0cfad0 2023-10-16T10:03:30.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf434a4700 1 -- 172.21.15.158:0/3900855213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0cfad0 msgr2=0x7fbf3c0cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3900855213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0cfad0 0x7fbf3c0cff60 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009e50 tx=0x7fbf3400b330 comp rx=0 tx=0).stop 2023-10-16T10:03:30.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf434a4700 1 -- 172.21.15.158:0/3900855213 shutdown_connections 2023-10-16T10:03:30.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3900855213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0cfad0 0x7fbf3c0cff60 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf434a4700 1 -- 172.21.15.158:0/3900855213 >> 172.21.15.158:0/3900855213 conn(0x7fbf3c0ce040 msgr2=0x7fbf3c0ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf434a4700 1 -- 172.21.15.158:0/3900855213 shutdown_connections 2023-10-16T10:03:30.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.150+0000 7fbf434a4700 1 -- 172.21.15.158:0/3900855213 wait complete. 2023-10-16T10:03:30.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.151+0000 7fbf434a4700 1 Processor -- start 2023-10-16T10:03:30.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf434a4700 1 -- start start 2023-10-16T10:03:30.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf434a4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf3c0cfad0 0x7fbf3c0cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf434a4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0d0ae0 0x7fbf3c1f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf434a4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf3c15eb10 0x7fbf3c1f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf434a4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c157900 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf434a4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c157be0 con 0x7fbf3c15eb10 2023-10-16T10:03:30.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf434a4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c157ec0 con 0x7fbf3c0cfad0 2023-10-16T10:03:30.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf40a3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0d0ae0 0x7fbf3c1f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf40a3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0d0ae0 0x7fbf3c1f0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58268/0 (socket says 172.21.15.158:58268) 2023-10-16T10:03:30.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.152+0000 7fbf40a3f700 1 -- 172.21.15.158:0/3840181813 learned_addr learned my addr 172.21.15.158:0/3840181813 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf40a3f700 1 -- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf3c0cfad0 msgr2=0x7fbf3c0cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf41240700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf3c0cfad0 0x7fbf3c0cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf41a41700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf3c15eb10 0x7fbf3c1f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf3c0cfad0 0x7fbf3c0cbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf40a3f700 1 -- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf3c15eb10 msgr2=0x7fbf3c1f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf3c15eb10 0x7fbf3c1f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf40a3f700 1 -- 172.21.15.158:0/3840181813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf34009b30 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf41a41700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf3c15eb10 0x7fbf3c1f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:30.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0d0ae0 0x7fbf3c1f0ca0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c540 tx=0x7fbf3800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf31ffb700 1 -- 172.21.15.158:0/3840181813 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38004c80 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf31ffb700 1 -- 172.21.15.158:0/3840181813 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38004de0 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c1581a0 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf31ffb700 1 -- 172.21.15.158:0/3840181813 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38007310 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.153+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c0cc2d0 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.155+0000 7fbf31ffb700 1 -- 172.21.15.158:0/3840181813 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf38007470 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.155+0000 7fbf31ffb700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbf2006b020 0x7fbf2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.155+0000 7fbf31ffb700 1 -- 172.21.15.158:0/3840181813 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fbf38089770 con 0x7fbf3c0d0ae0 2023-10-16T10:03:30.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.155+0000 7fbf41240700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbf2006b020 0x7fbf2006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.156+0000 7fbf41240700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbf2006b020 0x7fbf2006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c003e60 tx=0x7fbf2c009040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.157+0000 7fbf4f68eb80 1 Processor -- start 2023-10-16T10:03:30.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.157+0000 7fbf4f68eb80 1 -- start start 2023-10-16T10:03:30.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e48a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e46630 0x55b8c0e48fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e1c010 0x55b8c0e1c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0d92c20 con 0x55b8c0e46630 2023-10-16T10:03:30.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0d77c10 con 0x55b8c0e41220 2023-10-16T10:03:30.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0d75530 con 0x55b8c0e1c010 2023-10-16T10:03:30.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf40a3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e46630 0x55b8c0e48fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf40a3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e46630 0x55b8c0e48fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58276/0 (socket says 172.21.15.158:58276) 2023-10-16T10:03:30.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf40a3f700 1 -- 172.21.15.158:0/2728366210 learned_addr learned my addr 172.21.15.158:0/2728366210 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf41240700 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e48a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.158+0000 7fbf41a41700 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e1c010 0x55b8c0e1c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf40a3f700 1 -- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e1c010 msgr2=0x55b8c0e1c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e1c010 0x55b8c0e1c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf40a3f700 1 -- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 msgr2=0x55b8c0e48a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e48a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf40a3f700 1 -- 172.21.15.158:0/2728366210 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8c0d2e3a0 con 0x55b8c0e46630 2023-10-16T10:03:30.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e46630 0x55b8c0e48fb0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800cfd0 tx=0x7fbf38057dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f115175dfb11a499 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf26ffd700 1 -- 172.21.15.158:0/2728366210 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3806bea0 con 0x55b8c0e46630 2023-10-16T10:03:30.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf26ffd700 1 -- 172.21.15.158:0/2728366210 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3805c2a0 con 0x55b8c0e46630 2023-10-16T10:03:30.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf26ffd700 1 -- 172.21.15.158:0/2728366210 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38060450 con 0x55b8c0e46630 2023-10-16T10:03:30.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2728366210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e46630 msgr2=0x55b8c0e48fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.159+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e46630 0x55b8c0e48fb0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800cfd0 tx=0x7fbf38057dc0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2728366210 shutdown_connections 2023-10-16T10:03:30.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e1c010 0x55b8c0e1c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e48a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2728366210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e46630 0x55b8c0e48fb0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2728366210 >> 172.21.15.158:0/2728366210 conn(0x55b8c0d4bd80 msgr2=0x55b8c0e66ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2728366210 shutdown_connections 2023-10-16T10:03:30.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2728366210 wait complete. 2023-10-16T10:03:30.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 -- start start 2023-10-16T10:03:30.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.160+0000 7fbf4f68eb80 1 Processor -- start 2023-10-16T10:03:30.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e664e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e66a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e43a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0d70e20 con 0x55b8c0e1c010 2023-10-16T10:03:30.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0d0d6c0 con 0x55b8c0e41220 2023-10-16T10:03:30.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0d0d360 con 0x55b8c0e46630 2023-10-16T10:03:30.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 10 client.0 _ll_register_callbacks cb 0x55b8c0b6a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:03:30.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e664e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58280/0 (socket says 172.21.15.158:58280) 2023-10-16T10:03:30.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 -- 172.21.15.158:0/3520713026 learned_addr learned my addr 172.21.15.158:0/3520713026 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41a41700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e43a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e66a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 -- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 msgr2=0x55b8c0e43a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e43a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 -- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 msgr2=0x55b8c0e66a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e66a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 -- 172.21.15.158:0/3520713026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c0cc2d0 con 0x55b8c0e1c010 2023-10-16T10:03:30.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41a41700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e43a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:30.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e66a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:30.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf41240700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e664e0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c014f80 tx=0x7fbf2c0037f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c0154bf46f674d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:03:30.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf1bfff700 1 -- 172.21.15.158:0/3520713026 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c0203f0 con 0x55b8c0e1c010 2023-10-16T10:03:30.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b8c0e115b0 con 0x55b8c0e1c010 2023-10-16T10:03:30.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.162+0000 7fbf1bfff700 1 -- 172.21.15.158:0/3520713026 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fbf2c020550 con 0x55b8c0e1c010 2023-10-16T10:03:30.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf1bfff700 1 -- 172.21.15.158:0/3520713026 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf2c020710 con 0x55b8c0e1c010 2023-10-16T10:03:30.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 10 client.8958 fetch_fsmap learned FSMap version 68 2023-10-16T10:03:30.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x55b8c0b9bd90 con 0x55b8c0e1c010 2023-10-16T10:03:30.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf1bfff700 1 -- 172.21.15.158:0/3520713026 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b8c0e115b0 con 0x55b8c0e1c010 2023-10-16T10:03:30.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf1bfff700 1 -- 172.21.15.158:0/3520713026 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 68) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbf2c01f070 con 0x55b8c0e1c010 2023-10-16T10:03:30.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 10 client.8958 fetch_fsmap finished waiting for FSMap version 68 2023-10-16T10:03:30.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 10 client.8958 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T10:03:30.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 20 client.8958 populate_metadata read hostname 'smithi158' 2023-10-16T10:03:30.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e44250 0x55b8c0e44630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 4 client.8958 mds_command: new command op to 8817 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T10:03:30.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.163+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 --> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8c0d69e00 con 0x55b8c0e44250 2023-10-16T10:03:30.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.164+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e44250 0x55b8c0e44630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.164+0000 7fbf1bfff700 10 client.8958 ms_handle_connect on v2:172.21.15.172:6833/4142092863 2023-10-16T10:03:30.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.164+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e44250 0x55b8c0e44630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84e1f093876d58fc server_cookie=8f50b06cb79ce369 in_seq=0 out_seq=0 2023-10-16T10:03:30.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf1bfff700 1 -- 172.21.15.158:0/3520713026 <== mds.0 v2:172.21.15.172:6833/4142092863 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b8c0d69e00 con 0x55b8c0e44250 2023-10-16T10:03:30.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf1bfff700 10 client.8958 handle_command_reply: tid=0 2023-10-16T10:03:30.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 client.8958 shutdown 2023-10-16T10:03:30.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 msgr2=0x55b8c0e664e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e664e0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c014f80 tx=0x7fbf2c0037f0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 shutdown_connections 2023-10-16T10:03:30.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e43a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e664e0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e44250 0x55b8c0e44630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/3520713026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e41220 0x55b8c0e66a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf1bfff700 0 client.8958 ms_handle_reset on v2:172.21.15.172:6833/4142092863 2023-10-16T10:03:30.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 >> 172.21.15.158:0/3520713026 conn(0x55b8c0d4bd80 msgr2=0x55b8c0e13640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 shutdown_connections 2023-10-16T10:03:30.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/3520713026 wait complete. 2023-10-16T10:03:30.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.165+0000 7fbf4f68eb80 20 client.8958 trim_cache size 0 max 16384 2023-10-16T10:03:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.183+0000 7fbf4f68eb80 1 Processor -- start 2023-10-16T10:03:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.183+0000 7fbf4f68eb80 1 -- start start 2023-10-16T10:03:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e1c010 0x55b8c0e67150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e41220 0x55b8c0e67690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e12f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0bf18d0 con 0x55b8c0e41220 2023-10-16T10:03:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0e5d840 con 0x55b8c0e1c010 2023-10-16T10:03:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0e72000 con 0x55b8c0e46630 2023-10-16T10:03:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e41220 0x55b8c0e67690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e41220 0x55b8c0e67690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58290/0 (socket says 172.21.15.158:58290) 2023-10-16T10:03:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 -- 172.21.15.158:0/1270344263 learned_addr learned my addr 172.21.15.158:0/1270344263 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf41240700 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e1c010 0x55b8c0e67150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf41a41700 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e12f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 -- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 msgr2=0x55b8c0e12f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e12f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 -- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e1c010 msgr2=0x55b8c0e67150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e1c010 0x55b8c0e67150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 -- 172.21.15.158:0/1270344263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8c0b84c10 con 0x55b8c0e41220 2023-10-16T10:03:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf41240700 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e1c010 0x55b8c0e67150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e41220 0x55b8c0e67690 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fbf38087c80 tx=0x7fbf380545c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0716497a77e266 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf277fe700 1 -- 172.21.15.158:0/1270344263 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38054860 con 0x55b8c0e41220 2023-10-16T10:03:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf277fe700 1 -- 172.21.15.158:0/1270344263 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf380549e0 con 0x55b8c0e41220 2023-10-16T10:03:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.184+0000 7fbf277fe700 1 -- 172.21.15.158:0/1270344263 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3800e4c0 con 0x55b8c0e41220 2023-10-16T10:03:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/1270344263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e41220 msgr2=0x55b8c0e67690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e41220 0x55b8c0e67690 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fbf38087c80 tx=0x7fbf380545c0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/1270344263 shutdown_connections 2023-10-16T10:03:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e46630 0x55b8c0e12f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e1c010 0x55b8c0e67150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/1270344263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e41220 0x55b8c0e67690 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/1270344263 >> 172.21.15.158:0/1270344263 conn(0x55b8c0d4bd80 msgr2=0x55b8c0e49310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/1270344263 shutdown_connections 2023-10-16T10:03:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.185+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/1270344263 wait complete. 2023-10-16T10:03:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.186+0000 7fbf4f68eb80 1 -- start start 2023-10-16T10:03:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.186+0000 7fbf4f68eb80 1 Processor -- start 2023-10-16T10:03:30.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e41220 0x55b8c0eca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf4f68eb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e46630 0x55b8c0e41cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0ecb3d0 con 0x55b8c0e1c010 2023-10-16T10:03:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0e75330 con 0x55b8c0e46630 2023-10-16T10:03:30.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf4f68eb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8c0e41830 con 0x55b8c0e41220 2023-10-16T10:03:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf4f68eb80 10 client.0 _ll_register_callbacks cb 0x7fbf2c014840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:03:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e673b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58294/0 (socket says 172.21.15.158:58294) 2023-10-16T10:03:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 -- 172.21.15.158:0/2978821193 learned_addr learned my addr 172.21.15.158:0/2978821193 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e41220 0x55b8c0eca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41a41700 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e46630 0x55b8c0e41cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 -- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e41220 msgr2=0x55b8c0eca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e41220 0x55b8c0eca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 -- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e46630 msgr2=0x55b8c0e41cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e46630 0x55b8c0e41cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.187+0000 7fbf41240700 1 -- 172.21.15.158:0/2978821193 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8c0d2e3a0 con 0x55b8c0e1c010 2023-10-16T10:03:30.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf41240700 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e673b0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c000c00 tx=0x7fbf2c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b90a2affb243a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:03:30.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf1bfff700 1 -- 172.21.15.158:0/2978821193 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c029800 con 0x55b8c0e1c010 2023-10-16T10:03:30.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf3805c2a0 con 0x55b8c0e1c010 2023-10-16T10:03:30.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf1bfff700 1 -- 172.21.15.158:0/2978821193 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fbf2c029960 con 0x55b8c0e1c010 2023-10-16T10:03:30.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf1bfff700 1 -- 172.21.15.158:0/2978821193 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf2c012af0 con 0x55b8c0e1c010 2023-10-16T10:03:30.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 10 client.8964 fetch_fsmap learned FSMap version 68 2023-10-16T10:03:30.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x55b8c0bc2200 con 0x55b8c0e1c010 2023-10-16T10:03:30.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf1bfff700 1 -- 172.21.15.158:0/2978821193 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3805c2a0 con 0x55b8c0e1c010 2023-10-16T10:03:30.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf1bfff700 1 -- 172.21.15.158:0/2978821193 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 68) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbf2c03b3f0 con 0x55b8c0e1c010 2023-10-16T10:03:30.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 10 client.8964 fetch_fsmap finished waiting for FSMap version 68 2023-10-16T10:03:30.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 10 client.8964 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T10:03:30.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 20 client.8964 populate_metadata read hostname 'smithi158' 2023-10-16T10:03:30.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e3f660 0x55b8c0e3fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 4 client.8964 mds_command: new command op to 8817 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T10:03:30.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 --> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b8c0d69b20 con 0x55b8c0e3f660 2023-10-16T10:03:30.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.188+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e3f660 0x55b8c0e3fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf1bfff700 10 client.8964 ms_handle_connect on v2:172.21.15.172:6833/4142092863 2023-10-16T10:03:30.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf40a3f700 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e3f660 0x55b8c0e3fa40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f91bfb2153ad7ab server_cookie=3e4afa26dfaf0778 in_seq=0 out_seq=0 2023-10-16T10:03:30.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf1bfff700 1 -- 172.21.15.158:0/2978821193 <== mds.0 v2:172.21.15.172:6833/4142092863 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55b8c0d69b20 con 0x55b8c0e3f660 2023-10-16T10:03:30.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf1bfff700 10 client.8964 handle_command_reply: tid=0 2023-10-16T10:03:30.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 client.8964 shutdown 2023-10-16T10:03:30.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 msgr2=0x55b8c0e673b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e673b0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c000c00 tx=0x7fbf2c014000 comp rx=0 tx=0).stop 2023-10-16T10:03:30.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 shutdown_connections 2023-10-16T10:03:30.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:6833/4142092863,v1:172.21.15.172:6835/4142092863] conn(0x55b8c0e3f660 0x55b8c0e3fa40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b8c0e46630 0x55b8c0e41cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b8c0e1c010 0x55b8c0e673b0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 --2- 172.21.15.158:0/2978821193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b8c0e41220 0x55b8c0eca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf1bfff700 0 client.8964 ms_handle_reset on v2:172.21.15.172:6833/4142092863 2023-10-16T10:03:30.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 >> 172.21.15.158:0/2978821193 conn(0x55b8c0d4bd80 msgr2=0x55b8c0e49260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 shutdown_connections 2023-10-16T10:03:30.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 1 -- 172.21.15.158:0/2978821193 wait complete. 2023-10-16T10:03:30.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.189+0000 7fbf4f68eb80 20 client.8964 trim_cache size 0 max 16384 2023-10-16T10:03:30.211 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:03:30.212 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T10:03:30.212 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T10:03:30.212 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 8817, 2023-10-16T10:03:30.212 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T10:03:30.212 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T10:03:30.212 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T10:03:30.213 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9850828470000002, 2023-10-16T10:03:30.213 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 68, 2023-10-16T10:03:30.213 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 140, 2023-10-16T10:03:30.213 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 140, 2023-10-16T10:03:30.213 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 11.906413674 2023-10-16T10:03:30.214 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:03:30.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbf2006b020 msgr2=0x7fbf2006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbf2006b020 0x7fbf2006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c003e60 tx=0x7fbf2c009040 comp rx=0 tx=0).stop 2023-10-16T10:03:30.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0d0ae0 msgr2=0x7fbf3c1f0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0d0ae0 0x7fbf3c1f0ca0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c540 tx=0x7fbf3800c900 comp rx=0 tx=0).stop 2023-10-16T10:03:30.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 shutdown_connections 2023-10-16T10:03:30.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7fbf2006b020 0x7fbf2006d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf3c15eb10 0x7fbf3c1f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf3c0cfad0 0x7fbf3c0cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 --2- 172.21.15.158:0/3840181813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf3c0d0ae0 0x7fbf3c1f0ca0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 >> 172.21.15.158:0/3840181813 conn(0x7fbf3c0ce040 msgr2=0x7fbf3c0b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 shutdown_connections 2023-10-16T10:03:30.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.190+0000 7fbf434a4700 1 -- 172.21.15.158:0/3840181813 wait complete. 2023-10-16T10:03:30.218 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T10:03:30.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.338+0000 7f63f4627700 1 Processor -- start 2023-10-16T10:03:30.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.338+0000 7f63f4627700 1 -- start start 2023-10-16T10:03:30.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f4627700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f4627700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63ec15dfb0 0x7f63ec0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f4627700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63ec06f340 0x7f63ec06f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f4627700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ec0cf270 con 0x7f63ec15dfb0 2023-10-16T10:03:30.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f4627700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ec06fd10 con 0x7f63ec06f340 2023-10-16T10:03:30.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f4627700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec06e770 con 0x7f63ec0cc5d0 2023-10-16T10:03:30.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63ec15dfb0 0x7f63ec0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63ec15dfb0 0x7f63ec0cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58304/0 (socket says 172.21.15.158:58304) 2023-10-16T10:03:30.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 -- 172.21.15.158:0/2945923551 learned_addr learned my addr 172.21.15.158:0/2945923551 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 -- 172.21.15.158:0/2945923551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 msgr2=0x7f63ec06e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f23c3700 1 --2- 172.21.15.158:0/2945923551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f2bc4700 1 --2- 172.21.15.158:0/2945923551 >> v2:172.21.15.172:3300/0 conn(0x7f63ec06f340 0x7f63ec06f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 --2- 172.21.15.158:0/2945923551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec06e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 -- 172.21.15.158:0/2945923551 >> v2:172.21.15.172:3300/0 conn(0x7f63ec06f340 msgr2=0x7f63ec06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 --2- 172.21.15.158:0/2945923551 >> v2:172.21.15.172:3300/0 conn(0x7f63ec06f340 0x7f63ec06f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 -- 172.21.15.158:0/2945923551 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec15e4e0 con 0x7f63ec15dfb0 2023-10-16T10:03:30.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f1bc2700 1 --2- 172.21.15.158:0/2945923551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0cb6f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f63ec15df10 tx=0x7f63e800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2da43ade3d1248 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f13c1700 1 -- 172.21.15.158:0/2945923551 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e800dd30 con 0x7f63ec15dfb0 2023-10-16T10:03:30.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f13c1700 1 -- 172.21.15.158:0/2945923551 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e800de90 con 0x7f63ec15dfb0 2023-10-16T10:03:30.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.339+0000 7f63f13c1700 1 -- 172.21.15.158:0/2945923551 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8010690 con 0x7f63ec15dfb0 2023-10-16T10:03:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 -- 172.21.15.158:0/2945923551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 msgr2=0x7f63ec0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 --2- 172.21.15.158:0/2945923551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0cb6f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f63ec15df10 tx=0x7f63e800be70 comp rx=0 tx=0).stop 2023-10-16T10:03:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 -- 172.21.15.158:0/2945923551 shutdown_connections 2023-10-16T10:03:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 --2- 172.21.15.158:0/2945923551 >> v2:172.21.15.172:3300/0 conn(0x7f63ec06f340 0x7f63ec06f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 --2- 172.21.15.158:0/2945923551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec06e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 --2- 172.21.15.158:0/2945923551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0cb6f0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 -- 172.21.15.158:0/2945923551 >> 172.21.15.158:0/2945923551 conn(0x7f63ec0d02b0 msgr2=0x7f63ec0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 -- 172.21.15.158:0/2945923551 shutdown_connections 2023-10-16T10:03:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.340+0000 7f63f4627700 1 -- 172.21.15.158:0/2945923551 wait complete. 2023-10-16T10:03:30.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.341+0000 7f63f4627700 1 Processor -- start 2023-10-16T10:03:30.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.341+0000 7f63f4627700 1 -- start start 2023-10-16T10:03:30.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f4627700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63ec06f340 0x7f63ec0c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f4627700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec1545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f4627700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f4627700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0c3f40 con 0x7f63ec15dfb0 2023-10-16T10:03:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f4627700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0b4000 con 0x7f63ec06f340 2023-10-16T10:03:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f4627700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec06b1a0 con 0x7f63ec0cc5d0 2023-10-16T10:03:30.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f2bc4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f2bc4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58312/0 (socket says 172.21.15.158:58312) 2023-10-16T10:03:30.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f2bc4700 1 -- 172.21.15.158:0/1392590393 learned_addr learned my addr 172.21.15.158:0/1392590393 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:30.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f23c3700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63ec06f340 0x7f63ec0c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f1bc2700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec1545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f2bc4700 1 -- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 msgr2=0x7f63ec1545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f2bc4700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec1545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.342+0000 7f63f2bc4700 1 -- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63ec06f340 msgr2=0x7f63ec0c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63f2bc4700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63ec06f340 0x7f63ec0c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63f2bc4700 1 -- 172.21.15.158:0/1392590393 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e800b7f0 con 0x7f63ec15dfb0 2023-10-16T10:03:30.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63f23c3700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63ec06f340 0x7f63ec0c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:30.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63f2bc4700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0b41e0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f63e400cc30 tx=0x7f63e400a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63e2ffd700 1 -- 172.21.15.158:0/1392590393 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e400ab70 con 0x7f63ec15dfb0 2023-10-16T10:03:30.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63e2ffd700 1 -- 172.21.15.158:0/1392590393 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e400acd0 con 0x7f63ec15dfb0 2023-10-16T10:03:30.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec0b3d20 con 0x7f63ec15dfb0 2023-10-16T10:03:30.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63e2ffd700 1 -- 172.21.15.158:0/1392590393 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e4003c40 con 0x7f63ec15dfb0 2023-10-16T10:03:30.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.343+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ec0c4430 con 0x7f63ec15dfb0 2023-10-16T10:03:30.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.344+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ec0681b0 con 0x7f63ec15dfb0 2023-10-16T10:03:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.344+0000 7f63e2ffd700 1 -- 172.21.15.158:0/1392590393 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f63e400f3f0 con 0x7f63ec15dfb0 2023-10-16T10:03:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.345+0000 7f63e2ffd700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f63d006b020 0x7f63d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.345+0000 7f63e2ffd700 1 -- 172.21.15.158:0/1392590393 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f63e4089d90 con 0x7f63ec15dfb0 2023-10-16T10:03:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.345+0000 7f63f23c3700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f63d006b020 0x7f63d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:30.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.345+0000 7f63f23c3700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f63d006b020 0x7f63d006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63dc006fd0 tx=0x7f63dc008040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:30.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.349+0000 7f63e2ffd700 1 -- 172.21.15.158:0/1392590393 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63e4054910 con 0x7f63ec15dfb0 2023-10-16T10:03:30.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.585+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f63ec0b6230 con 0x7f63ec15dfb0 2023-10-16T10:03:30.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.585+0000 7f63e2ffd700 1 -- 172.21.15.158:0/1392590393 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) v1 ==== 53+0+73 (secure 0 0 0) 0x7f63e401d070 con 0x7f63ec15dfb0 2023-10-16T10:03:30.587 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T10:03:30.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f63d006b020 msgr2=0x7f63d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f63d006b020 0x7f63d006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63dc006fd0 tx=0x7f63dc008040 comp rx=0 tx=0).stop 2023-10-16T10:03:30.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 msgr2=0x7f63ec0b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:30.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0b41e0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f63e400cc30 tx=0x7f63e400a1f0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 shutdown_connections 2023-10-16T10:03:30.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f63d006b020 0x7f63d006d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63ec15dfb0 0x7f63ec0b41e0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63ec06f340 0x7f63ec0c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 --2- 172.21.15.158:0/1392590393 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63ec0cc5d0 0x7f63ec1545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:30.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 >> 172.21.15.158:0/1392590393 conn(0x7f63ec0d02b0 msgr2=0x7f63ec071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:30.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.587+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 shutdown_connections 2023-10-16T10:03:30.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:30.588+0000 7f63f4627700 1 -- 172.21.15.158:0/1392590393 wait complete. 2023-10-16T10:03:30.601 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T10:03:30.601 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T10:03:30.601 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T10:03:30.602 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T10:03:30.602 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T10:03:30.602 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T10:03:30.603 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T10:03:30.603 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T10:03:30.603 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:03:30.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:30.604 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T10:03:30.622 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T10:03:30.622 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T10:03:30.622 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T10:03:30.623 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:03:30.623 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T10:03:30.623 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:03:30.623 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:03:30.623 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:03:30.624 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:03:30.624 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:03:30.624 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:03:30.624 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T10:03:30.624 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T10:03:30.625 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T10:03:30.625 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T10:03:30.625 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 41221sec preferred_lft 41221sec 2023-10-16T10:03:30.625 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T10:03:30.625 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:03:30.626 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T10:03:30.626 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T10:03:30.626 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T10:03:30.626 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:03:30.626 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T10:03:30.627 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:03:30.627 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:03:30.627 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T10:03:30.627 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T10:03:30.627 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:03:30.627 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:03:30.628 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T10:03:30.628 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T10:03:30.628 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:03:30.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:30.629 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T10:03:30.688 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T10:03:30.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:30.690 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T10:03:30.749 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T10:03:30.749 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T10:03:30.750 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T10:03:30.750 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T10:03:30.750 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T10:03:30.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:30.751 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T10:03:30.751 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T10:03:30.752 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T10:03:30.752 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T10:03:30.752 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T10:03:30.753 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T10:03:30.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:30.855 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T10:03:30.871 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T10:03:30.872 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T10:03:30.872 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:03:30.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:30.873 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T10:03:30.938 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:03:30.938 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T10:03:30.994 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:03:30.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:03:30.995 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:03:31.051 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T10:03:31.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:31.052 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:03:31.135 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:03:31.135 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:03:31.162 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:03:31.163 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:03:31.163 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:03:31.163 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:03:31.163 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:03:31.164 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:03:31.164 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:03:31.164 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:03:31.240 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:03:31.241 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:03:31.241 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:03:31.241 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:03:31.241 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:03:31.241 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:03:31.242 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:03:31.242 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:03:31.242 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T10:03:31.242 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T10:03:31.243 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T10:03:31.244 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T10:03:31.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:31.244 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:03:31.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:31.272 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T10:03:31.359 INFO:teuthology.orchestra.run.smithi158.stdout:client37070 2023-10-16T10:03:31.360 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:31.360 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:31.377 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T10:03:31.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.537+0000 7f9c8caac700 1 Processor -- start 2023-10-16T10:03:31.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.537+0000 7f9c8caac700 1 -- start start 2023-10-16T10:03:31.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c8caac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c8806e480 0x7f9c880b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c8caac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c880cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c8caac700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c880af600 con 0x7f9c8815d990 2023-10-16T10:03:31.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c8caac700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c880b2c90 con 0x7f9c8806e480 2023-10-16T10:03:31.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c8caac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8806e9b0 con 0x7f9c880cf650 2023-10-16T10:03:31.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c86d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c880cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c85d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c8806e480 0x7f9c880b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c86d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c880cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:40926/0 (socket says 172.21.15.158:40926) 2023-10-16T10:03:31.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c8659c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9c8815d990 0x7f9c880b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:34168/0 (socket says 172.21.15.158:34168) 2023-10-16T10:03:31.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c85d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c8806e480 0x7f9c880b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:59502/0 (socket says 172.21.15.158:59502) 2023-10-16T10:03:31.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.538+0000 7f9c86d9d700 1 -- 172.21.15.158:0/743967272 learned_addr learned my addr 172.21.15.158:0/743967272 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:31.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c86d9d700 1 -- 172.21.15.158:0/743967272 >> v2:172.21.15.172:3300/0 conn(0x7f9c8806e480 msgr2=0x7f9c880b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:31.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c86d9d700 1 --2- 172.21.15.158:0/743967272 >> v2:172.21.15.172:3300/0 conn(0x7f9c8806e480 0x7f9c880b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c86d9d700 1 -- 172.21.15.158:0/743967272 >> v1:172.21.15.172:6789/0 conn(0x7f9c8815d990 legacy=0x7f9c880b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c86d9d700 1 -- 172.21.15.158:0/743967272 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c880ce890 con 0x7f9c880cf650 2023-10-16T10:03:31.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c85d9b700 1 --2- 172.21.15.158:0/743967272 >> v2:172.21.15.172:3300/0 conn(0x7f9c8806e480 0x7f9c880b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:31.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c86d9d700 1 --2- 172.21.15.158:0/743967272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c880cfb00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009e50 tx=0x7f9c8000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5825fb1cea7bce66 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:31.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c8559a700 1 -- 172.21.15.158:0/743967272 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8000e070 con 0x7f9c880cf650 2023-10-16T10:03:31.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c8559a700 1 -- 172.21.15.158:0/743967272 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8000b910 con 0x7f9c880cf650 2023-10-16T10:03:31.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.539+0000 7f9c8559a700 1 -- 172.21.15.158:0/743967272 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8000bcc0 con 0x7f9c880cf650 2023-10-16T10:03:31.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 -- 172.21.15.158:0/743967272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 msgr2=0x7f9c880cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 --2- 172.21.15.158:0/743967272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c880cfb00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009e50 tx=0x7f9c8000b330 comp rx=0 tx=0).stop 2023-10-16T10:03:31.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 -- 172.21.15.158:0/743967272 shutdown_connections 2023-10-16T10:03:31.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 --2- 172.21.15.158:0/743967272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c880cfb00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 --2- 172.21.15.158:0/743967272 >> v2:172.21.15.172:3300/0 conn(0x7f9c8806e480 0x7f9c880b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 -- 172.21.15.158:0/743967272 >> 172.21.15.158:0/743967272 conn(0x7f9c8806b9e0 msgr2=0x7f9c8806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:31.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 -- 172.21.15.158:0/743967272 shutdown_connections 2023-10-16T10:03:31.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.540+0000 7f9c8caac700 1 -- 172.21.15.158:0/743967272 wait complete. 2023-10-16T10:03:31.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.541+0000 7f9c8caac700 1 Processor -- start 2023-10-16T10:03:31.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.541+0000 7f9c8caac700 1 -- start start 2023-10-16T10:03:31.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c8caac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8806e480 0x7f9c880c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c8caac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c881f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c8caac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c8815d990 0x7f9c880ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c8caac700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881f7510 con 0x7f9c8815d990 2023-10-16T10:03:31.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c8caac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8815f020 con 0x7f9c8806e480 2023-10-16T10:03:31.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c8caac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8815f300 con 0x7f9c880cf650 2023-10-16T10:03:31.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c8815d990 0x7f9c880ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c8815d990 0x7f9c880ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58330/0 (socket says 172.21.15.158:58330) 2023-10-16T10:03:31.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 -- 172.21.15.158:0/3235215478 learned_addr learned my addr 172.21.15.158:0/3235215478 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:31.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c8659c700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8806e480 0x7f9c880c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c85d9b700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c881f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 -- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 msgr2=0x7f9c881f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c881f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 -- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8806e480 msgr2=0x7f9c880c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8806e480 0x7f9c880c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.542+0000 7f9c86d9d700 1 -- 172.21.15.158:0/3235215478 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c780094a0 con 0x7f9c8815d990 2023-10-16T10:03:31.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.543+0000 7f9c86d9d700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c8815d990 0x7f9c880ca9e0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f9c8806eaf0 tx=0x7f9c80004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:31.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.543+0000 7f9c76ffd700 1 -- 172.21.15.158:0/3235215478 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80003ea0 con 0x7f9c8815d990 2023-10-16T10:03:31.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.543+0000 7f9c76ffd700 1 -- 172.21.15.158:0/3235215478 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c800099b0 con 0x7f9c8815d990 2023-10-16T10:03:31.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.543+0000 7f9c76ffd700 1 -- 172.21.15.158:0/3235215478 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8001c680 con 0x7f9c8815d990 2023-10-16T10:03:31.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.543+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c80009b30 con 0x7f9c8815d990 2023-10-16T10:03:31.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.543+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8815f8c0 con 0x7f9c8815d990 2023-10-16T10:03:31.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.544+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c880681b0 con 0x7f9c8815d990 2023-10-16T10:03:31.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.544+0000 7f9c76ffd700 1 -- 172.21.15.158:0/3235215478 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9c8001c7e0 con 0x7f9c8815d990 2023-10-16T10:03:31.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.544+0000 7f9c76ffd700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9c6806b020 0x7f9c6806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.545+0000 7f9c76ffd700 1 -- 172.21.15.158:0/3235215478 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f9c8008b830 con 0x7f9c8815d990 2023-10-16T10:03:31.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.545+0000 7f9c8659c700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9c6806b020 0x7f9c6806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.545+0000 7f9c8659c700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9c6806b020 0x7f9c6806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c881f6580 tx=0x7f9c7c007d70 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:31.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.549+0000 7f9c76ffd700 1 -- 172.21.15.158:0/3235215478 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c80056440 con 0x7f9c8815d990 2023-10-16T10:03:31.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.774+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 --> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c880cee50 con 0x7f9c6806b020 2023-10-16T10:03:31.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.776+0000 7f9c76ffd700 1 -- 172.21.15.158:0/3235215478 <== mgr.8574 v2:172.21.15.158:6832/2342578387 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9c7c004870 con 0x7f9c6806b020 2023-10-16T10:03:31.777 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:03:31.777 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:03:31.777 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T10:03:31.778 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:03:31.778 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:03:31.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9c6806b020 msgr2=0x7f9c6806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9c6806b020 0x7f9c6806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c881f6580 tx=0x7f9c7c007d70 comp rx=0 tx=0).stop 2023-10-16T10:03:31.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c8815d990 msgr2=0x7f9c880ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c8815d990 0x7f9c880ca9e0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f9c8806eaf0 tx=0x7f9c80004ad0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 shutdown_connections 2023-10-16T10:03:31.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f9c6806b020 0x7f9c6806d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c8815d990 0x7f9c880ca9e0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8806e480 0x7f9c880c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 --2- 172.21.15.158:0/3235215478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c880cf650 0x7f9c881f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 >> 172.21.15.158:0/3235215478 conn(0x7f9c8806b9e0 msgr2=0x7f9c880d0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:31.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 shutdown_connections 2023-10-16T10:03:31.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.778+0000 7f9c8caac700 1 -- 172.21.15.158:0/3235215478 wait complete. 2023-10-16T10:03:31.791 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T10:03:31.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.912+0000 7f9255294700 1 Processor -- start 2023-10-16T10:03:31.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.912+0000 7f9255294700 1 -- start start 2023-10-16T10:03:31.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.913+0000 7f9255294700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9250163380 0x7f9250165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.913+0000 7f9255294700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92500d0ae0 con 0x7f925015dad0 2023-10-16T10:03:31.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.913+0000 7f9255294700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92500d12d0 con 0x7f925015deb0 2023-10-16T10:03:31.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.913+0000 7f9255294700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92500cbdf0 con 0x7f9250163380 2023-10-16T10:03:31.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f925015dad0 0x7f92500cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58510/0 (socket says 172.21.15.158:58510) 2023-10-16T10:03:31.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924effd700 1 -- 172.21.15.158:0/3882624996 learned_addr learned my addr 172.21.15.158:0/3882624996 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:31.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924f7fe700 1 --2- 172.21.15.158:0/3882624996 >> v2:172.21.15.172:3300/0 conn(0x7f9250163380 0x7f9250165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647331286 0 0) 0x7f92500d0ae0 con 0x7f925015dad0 2023-10-16T10:03:31.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92380035b0 con 0x7f925015dad0 2023-10-16T10:03:31.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2507743326 0 0) 0x7f92380035b0 con 0x7f925015dad0 2023-10-16T10:03:31.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 >> v2:172.21.15.172:3300/0 conn(0x7f9250163380 msgr2=0x7f9250165770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:31.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 --2- 172.21.15.158:0/3882624996 >> v2:172.21.15.172:3300/0 conn(0x7f9250163380 0x7f9250165770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 >> v1:172.21.15.172:6789/0 conn(0x7f925015deb0 legacy=0x7f92500ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500cc280 con 0x7f925015dad0 2023-10-16T10:03:31.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f9244001670 con 0x7f925015dad0 2023-10-16T10:03:31.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 >> v1:172.21.15.158:6789/0 conn(0x7f925015dad0 legacy=0x7f92500cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238003a00 0x7f9238003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9238004850 0x7f9238004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9238005330 0x7f923800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500cbdf0 con 0x7f9238005330 2023-10-16T10:03:31.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923800c200 con 0x7f9238003a00 2023-10-16T10:03:31.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923800c3c0 con 0x7f9238004850 2023-10-16T10:03:31.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924f7fe700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9238005330 0x7f923800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924effd700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238003a00 0x7f9238003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.914+0000 7f924e7fc700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9238004850 0x7f9238004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924f7fe700 1 -- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9238004850 msgr2=0x7f9238004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:31.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924f7fe700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9238004850 0x7f9238004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924f7fe700 1 -- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238003a00 msgr2=0x7f9238003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:31.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924f7fe700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238003a00 0x7f9238003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924f7fe700 1 -- 172.21.15.158:0/3882624996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92380035b0 con 0x7f9238005330 2023-10-16T10:03:31.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924e7fc700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9238004850 0x7f9238004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:31.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924f7fe700 1 -- 172.21.15.158:0/3882624996 reap_dead start 2023-10-16T10:03:31.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924f7fe700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9238005330 0x7f923800ba10 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f9248009bd0 tx=0x7f9238005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc1f48bf037b6fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:31.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248003c20 con 0x7f9238005330 2023-10-16T10:03:31.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924800f040 con 0x7f9238005330 2023-10-16T10:03:31.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f924dffb700 1 -- 172.21.15.158:0/3882624996 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924800e9d0 con 0x7f9238005330 2023-10-16T10:03:31.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f9255294700 1 -- 172.21.15.158:0/3882624996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9238005330 msgr2=0x7f923800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f9255294700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9238005330 0x7f923800ba10 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f9248009bd0 tx=0x7f9238005880 comp rx=0 tx=0).stop 2023-10-16T10:03:31.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f9255294700 1 -- 172.21.15.158:0/3882624996 shutdown_connections 2023-10-16T10:03:31.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f9255294700 1 --2- 172.21.15.158:0/3882624996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9238005330 0x7f923800ba10 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.915+0000 7f9255294700 1 -- 172.21.15.158:0/3882624996 >> 172.21.15.158:0/3882624996 conn(0x7f9250099a10 msgr2=0x7f92500d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:31.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.916+0000 7f9255294700 1 -- 172.21.15.158:0/3882624996 shutdown_connections 2023-10-16T10:03:31.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.916+0000 7f9255294700 1 -- 172.21.15.158:0/3882624996 wait complete. 2023-10-16T10:03:31.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.917+0000 7f9255294700 1 Processor -- start 2023-10-16T10:03:31.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.917+0000 7f9255294700 1 -- start start 2023-10-16T10:03:31.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238005330 0x7f925014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92500c4f50 0x7f92500c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92500b3380 0x7f92500b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925014da50 con 0x7f92500c4f50 2023-10-16T10:03:31.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500c8220 con 0x7f9238005330 2023-10-16T10:03:31.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92500c8500 con 0x7f92500b3380 2023-10-16T10:03:31.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92500c4f50 0x7f92500c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92500c4f50 0x7f92500c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58348/0 (socket says 172.21.15.158:58348) 2023-10-16T10:03:31.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 -- 172.21.15.158:0/1569788567 learned_addr learned my addr 172.21.15.158:0/1569788567 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:31.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924effd700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238005330 0x7f925014d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924f7fe700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92500b3380 0x7f92500b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 -- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92500b3380 msgr2=0x7f92500b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92500b3380 0x7f92500b3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 -- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238005330 msgr2=0x7f925014d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:31.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238005330 0x7f925014d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:31.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 -- 172.21.15.158:0/1569788567 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500c87e0 con 0x7f92500c4f50 2023-10-16T10:03:31.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f924e7fc700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92500c4f50 0x7f92500c53d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f924000e550 tx=0x7f924000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:31.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f92377fe700 1 -- 172.21.15.158:0/1569788567 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92400052e0 con 0x7f92500c4f50 2023-10-16T10:03:31.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f92377fe700 1 -- 172.21.15.158:0/1569788567 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240005440 con 0x7f92500c4f50 2023-10-16T10:03:31.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f92377fe700 1 -- 172.21.15.158:0/1569788567 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92400038a0 con 0x7f92500c4f50 2023-10-16T10:03:31.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92500c8ac0 con 0x7f92500c4f50 2023-10-16T10:03:31.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.918+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92500c7eb0 con 0x7f92500c4f50 2023-10-16T10:03:31.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.919+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92500c9110 con 0x7f92500c4f50 2023-10-16T10:03:31.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.920+0000 7f92377fe700 1 -- 172.21.15.158:0/1569788567 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9240003a00 con 0x7f92500c4f50 2023-10-16T10:03:31.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.920+0000 7f92377fe700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f923c06b020 0x7f923c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:31.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.920+0000 7f92377fe700 1 -- 172.21.15.158:0/1569788567 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f92400891d0 con 0x7f92500c4f50 2023-10-16T10:03:31.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.920+0000 7f924effd700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f923c06b020 0x7f923c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:31.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.921+0000 7f924effd700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f923c06b020 0x7f923c06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9244007cd0 tx=0x7f9244009040 comp rx=0 tx=0).ready entity=mgr.8574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:31.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:31.925+0000 7f92377fe700 1 -- 172.21.15.158:0/1569788567 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9240053f30 con 0x7f92500c4f50 2023-10-16T10:03:32.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.171+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f92500681b0 con 0x7f92500c4f50 2023-10-16T10:03:32.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.629+0000 7f92377fe700 1 -- 172.21.15.158:0/1569788567 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v75) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9240003cb0 con 0x7f92500c4f50 2023-10-16T10:03:32.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.632+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f923c06b020 msgr2=0x7f923c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:32.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.632+0000 7f9255294700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f923c06b020 0x7f923c06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9244007cd0 tx=0x7f9244009040 comp rx=0 tx=0).stop 2023-10-16T10:03:32.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.632+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92500c4f50 msgr2=0x7f92500c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:32.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.632+0000 7f9255294700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92500c4f50 0x7f92500c53d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f924000e550 tx=0x7f924000e910 comp rx=0 tx=0).stop 2023-10-16T10:03:32.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.633+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 shutdown_connections 2023-10-16T10:03:32.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.633+0000 7f9255294700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f923c06b020 0x7f923c06d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:32.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.633+0000 7f9255294700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92500b3380 0x7f92500b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:32.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.633+0000 7f9255294700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9238005330 0x7f925014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:32.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.633+0000 7f9255294700 1 --2- 172.21.15.158:0/1569788567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92500c4f50 0x7f92500c53d0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:32.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.633+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 >> 172.21.15.158:0/1569788567 conn(0x7f9250099a10 msgr2=0x7f92500cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:32.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.635+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 shutdown_connections 2023-10-16T10:03:32.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.635+0000 7f9255294700 1 -- 172.21.15.158:0/1569788567 wait complete. 2023-10-16T10:03:32.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T10:03:32.777 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:32.776+0000 7f3b42f8d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:03:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.778+0000 7f22da682700 1 Processor -- start 2023-10-16T10:03:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.778+0000 7f22da682700 1 -- start start 2023-10-16T10:03:32.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22da682700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d406d5d0 0x7f22d4070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:32.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22da682700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d415dd70 con 0x7f22d40cbaf0 2023-10-16T10:03:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22da682700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d4070a00 con 0x7f22d415d990 2023-10-16T10:03:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22da682700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d41616b0 con 0x7f22d406d5d0 2023-10-16T10:03:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d8c1f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f22d40cbaf0 0x7f22d40b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58564/0 (socket says 172.21.15.158:58564) 2023-10-16T10:03:32.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f64d54c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:03:32.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d8c1f700 1 -- 172.21.15.158:0/3541072404 learned_addr learned my addr 172.21.15.158:0/3541072404 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d37fe700 1 --2- 172.21.15.158:0/3541072404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d406d5d0 0x7f22d4070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527251131 0 0) 0x7f22cc001490 con 0x7f22d40cbaf0 2023-10-16T10:03:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22bc0035b0 con 0x7f22d40cbaf0 2023-10-16T10:03:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356643206 0 0) 0x7f22d4070a00 con 0x7f22d415d990 2023-10-16T10:03:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc001490 con 0x7f22d415d990 2023-10-16T10:03:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d37fe700 1 -- 172.21.15.158:0/3541072404 >> v1:172.21.15.172:6789/0 conn(0x7f22d415d990 legacy=0x7f22d4161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d37fe700 1 -- 172.21.15.158:0/3541072404 >> v1:172.21.15.158:6789/0 conn(0x7f22d40cbaf0 legacy=0x7f22d40b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2769877628 0 0) 0x7f22bc0035b0 con 0x7f22d40cbaf0 2023-10-16T10:03:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.779+0000 7f22d37fe700 1 -- 172.21.15.158:0/3541072404 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4070ce0 con 0x7f22d406d5d0 2023-10-16T10:03:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22d37fe700 1 --2- 172.21.15.158:0/3541072404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d406d5d0 0x7f22d4070300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009870 tx=0x7f22c8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e5ed9f510f464be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc001b80 con 0x7f22d406d5d0 2023-10-16T10:03:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c8004030 con 0x7f22d406d5d0 2023-10-16T10:03:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22d2ffd700 1 -- 172.21.15.158:0/3541072404 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c80043a0 con 0x7f22d406d5d0 2023-10-16T10:03:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22da682700 1 -- 172.21.15.158:0/3541072404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d406d5d0 msgr2=0x7f22d4070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22da682700 1 --2- 172.21.15.158:0/3541072404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d406d5d0 0x7f22d4070300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009870 tx=0x7f22c8009b80 comp rx=0 tx=0).stop 2023-10-16T10:03:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22da682700 1 -- 172.21.15.158:0/3541072404 shutdown_connections 2023-10-16T10:03:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22da682700 1 --2- 172.21.15.158:0/3541072404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d406d5d0 0x7f22d4070300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22da682700 1 -- 172.21.15.158:0/3541072404 >> 172.21.15.158:0/3541072404 conn(0x7f22d40cdf20 msgr2=0x7f22d40ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22da682700 1 -- 172.21.15.158:0/3541072404 shutdown_connections 2023-10-16T10:03:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.780+0000 7f22da682700 1 -- 172.21.15.158:0/3541072404 wait complete. 2023-10-16T10:03:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.782+0000 7f22da682700 1 Processor -- start 2023-10-16T10:03:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.782+0000 7f22da682700 1 -- start start 2023-10-16T10:03:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.782+0000 7f22da682700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d406d5d0 0x7f22d40b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.782+0000 7f22da682700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d40cbaf0 0x7f22d40b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22da682700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d415d990 0x7f22d40cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22da682700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d40c7ca0 con 0x7f22d406d5d0 2023-10-16T10:03:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22da682700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d40b4be0 con 0x7f22d415d990 2023-10-16T10:03:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22da682700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d40b4ec0 con 0x7f22d40cbaf0 2023-10-16T10:03:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d406d5d0 0x7f22d40b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d406d5d0 0x7f22d40b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58390/0 (socket says 172.21.15.158:58390) 2023-10-16T10:03:32.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 -- 172.21.15.158:0/3019965537 learned_addr learned my addr 172.21.15.158:0/3019965537 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:32.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d37fe700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d40cbaf0 0x7f22d40b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:32.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d8c1f700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d415d990 0x7f22d40cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:32.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 -- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d40cbaf0 msgr2=0x7f22d40b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:32.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d40cbaf0 0x7f22d40b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:32.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 -- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d415d990 msgr2=0x7f22d40cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:32.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d415d990 0x7f22d40cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:32.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 -- 172.21.15.158:0/3019965537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80094a0 con 0x7f22d406d5d0 2023-10-16T10:03:32.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d3fff700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d406d5d0 0x7f22d40b4070 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f22c400c520 tx=0x7f22c400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:32.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d0ff9700 1 -- 172.21.15.158:0/3019965537 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c40052d0 con 0x7f22d406d5d0 2023-10-16T10:03:32.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d0ff9700 1 -- 172.21.15.158:0/3019965537 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c4005430 con 0x7f22d406d5d0 2023-10-16T10:03:32.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d40b58a0 con 0x7f22d406d5d0 2023-10-16T10:03:32.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22d0ff9700 1 -- 172.21.15.158:0/3019965537 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c40038c0 con 0x7f22d406d5d0 2023-10-16T10:03:32.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.783+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d41f35c0 con 0x7f22d406d5d0 2023-10-16T10:03:32.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.784+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d40b5dc0 con 0x7f22d406d5d0 2023-10-16T10:03:32.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.785+0000 7f22d0ff9700 1 -- 172.21.15.158:0/3019965537 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 75) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f22c4003a20 con 0x7f22d406d5d0 2023-10-16T10:03:32.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.785+0000 7f22d0ff9700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 0x7f22c006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:32.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.785+0000 7f22d37fe700 1 -- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 msgr2=0x7f22c006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:32.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.785+0000 7f22d0ff9700 1 -- 172.21.15.158:0/3019965537 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f22c4088f10 con 0x7f22d406d5d0 2023-10-16T10:03:32.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.785+0000 7f22d37fe700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 0x7f22c006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:32.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.790+0000 7f22d0ff9700 1 -- 172.21.15.158:0/3019965537 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22d40b5dc0 con 0x7f22d406d5d0 2023-10-16T10:03:32.844 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:32.843+0000 7f3b42f8d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:32.849 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:32.848+0000 7f64d54c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:32.916 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:32.915+0000 7f3b42f8d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:03:32.922 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:32.921+0000 7f64d54c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:03:32.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.986+0000 7f22d37fe700 1 -- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 msgr2=0x7f22c006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:32.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.986+0000 7f22d37fe700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 0x7f22c006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:32.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.991+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f22d40b5dc0 con 0x7f22d406d5d0 2023-10-16T10:03:32.993 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:32.992+0000 7f3b42f8d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:03:32.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.997+0000 7f22d0ff9700 1 -- 172.21.15.158:0/3019965537 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f22d40b5dc0 con 0x7f22d406d5d0 2023-10-16T10:03:33.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 msgr2=0x7f22c006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:33.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 0x7f22c006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d406d5d0 msgr2=0x7f22d40b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d406d5d0 0x7f22d40b4070 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f22c400c520 tx=0x7f22c400c830 comp rx=0 tx=0).stop 2023-10-16T10:03:33.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 shutdown_connections 2023-10-16T10:03:33.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f22c006b070 0x7f22c006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22d415d990 0x7f22d40cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22d406d5d0 0x7f22d40b4070 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 --2- 172.21.15.158:0/3019965537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22d40cbaf0 0x7f22d40b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 >> 172.21.15.158:0/3019965537 conn(0x7f22d40cdf20 msgr2=0x7f22d40ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:33.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 shutdown_connections 2023-10-16T10:03:33.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:32.999+0000 7f22da682700 1 -- 172.21.15.158:0/3019965537 wait complete. 2023-10-16T10:03:33.004 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:33.003+0000 7f64d54c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:03:33.013 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T10:03:33.135 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:33.134+0000 7f3b42f8d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:03:33.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.138+0000 7f4486597700 1 Processor -- start 2023-10-16T10:03:33.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.139+0000 7f4486597700 1 -- start start 2023-10-16T10:03:33.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f4486597700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448015b6b0 0x7f44800cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f4486597700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44800d0c60 0x7f44800b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f4486597700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f44800b3460 0x7f4480070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f4486597700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44800b3910 con 0x7f44800d0c60 2023-10-16T10:03:33.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f4486597700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448015ba90 con 0x7f44800b3460 2023-10-16T10:03:33.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f4486597700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44800b4420 con 0x7f448015b6b0 2023-10-16T10:03:33.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44800d0c60 0x7f44800b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44800d0c60 0x7f44800b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58446/0 (socket says 172.21.15.158:58446) 2023-10-16T10:03:33.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 -- 172.21.15.158:0/2860658875 learned_addr learned my addr 172.21.15.158:0/2860658875 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:33.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447ffff700 1 --2- 172.21.15.158:0/2860658875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448015b6b0 0x7f44800cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f4484b34700 1 --2- 172.21.15.158:0/2860658875 >> v2:172.21.15.172:3300/0 conn(0x7f44800b3460 0x7f4480070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 -- 172.21.15.158:0/2860658875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448015b6b0 msgr2=0x7f44800cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:33.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 --2- 172.21.15.158:0/2860658875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448015b6b0 0x7f44800cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 -- 172.21.15.158:0/2860658875 >> v2:172.21.15.172:3300/0 conn(0x7f44800b3460 msgr2=0x7f4480070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:33.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 --2- 172.21.15.158:0/2860658875 >> v2:172.21.15.172:3300/0 conn(0x7f44800b3460 0x7f4480070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.140+0000 7f447f7fe700 1 -- 172.21.15.158:0/2860658875 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44800cbc20 con 0x7f44800d0c60 2023-10-16T10:03:33.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f447f7fe700 1 --2- 172.21.15.158:0/2860658875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44800d0c60 0x7f44800b2f20 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f44800cd250 tx=0x7f4474009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d330d0d70720ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:33.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f447effd700 1 -- 172.21.15.158:0/2860658875 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474011830 con 0x7f44800d0c60 2023-10-16T10:03:33.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f447effd700 1 -- 172.21.15.158:0/2860658875 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4474011990 con 0x7f44800d0c60 2023-10-16T10:03:33.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f447effd700 1 -- 172.21.15.158:0/2860658875 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474011d20 con 0x7f44800d0c60 2023-10-16T10:03:33.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 -- 172.21.15.158:0/2860658875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44800d0c60 msgr2=0x7f44800b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 --2- 172.21.15.158:0/2860658875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44800d0c60 0x7f44800b2f20 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f44800cd250 tx=0x7f4474009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 -- 172.21.15.158:0/2860658875 shutdown_connections 2023-10-16T10:03:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 --2- 172.21.15.158:0/2860658875 >> v2:172.21.15.172:3300/0 conn(0x7f44800b3460 0x7f4480070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 --2- 172.21.15.158:0/2860658875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f448015b6b0 0x7f44800cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 --2- 172.21.15.158:0/2860658875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44800d0c60 0x7f44800b2f20 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 -- 172.21.15.158:0/2860658875 >> 172.21.15.158:0/2860658875 conn(0x7f44800cdf20 msgr2=0x7f44800ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:33.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.141+0000 7f4486597700 1 -- 172.21.15.158:0/2860658875 shutdown_connections 2023-10-16T10:03:33.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.142+0000 7f4486597700 1 -- 172.21.15.158:0/2860658875 wait complete. 2023-10-16T10:03:33.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.143+0000 7f4486597700 1 Processor -- start 2023-10-16T10:03:33.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.144+0000 7f4486597700 1 -- start start 2023-10-16T10:03:33.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4486597700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44800b3460 0x7f4480150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4486597700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44800d0c60 0x7f448006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4486597700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f448015b6b0 0x7f4480154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4486597700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448006d070 con 0x7f448015b6b0 2023-10-16T10:03:33.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4486597700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448006c460 con 0x7f44800d0c60 2023-10-16T10:03:33.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4486597700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448006c740 con 0x7f44800b3460 2023-10-16T10:03:33.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f448015b6b0 0x7f4480154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f447ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44800b3460 0x7f4480150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f448015b6b0 0x7f4480154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58452/0 (socket says 172.21.15.158:58452) 2023-10-16T10:03:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 -- 172.21.15.158:0/3533080967 learned_addr learned my addr 172.21.15.158:0/3533080967 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f447f7fe700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44800d0c60 0x7f448006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 -- 172.21.15.158:0/3533080967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44800b3460 msgr2=0x7f4480150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44800b3460 0x7f4480150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 -- 172.21.15.158:0/3533080967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44800d0c60 msgr2=0x7f448006b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44800d0c60 0x7f448006b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 -- 172.21.15.158:0/3533080967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4474009550 con 0x7f448015b6b0 2023-10-16T10:03:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.145+0000 7f4484b34700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f448015b6b0 0x7f4480154f50 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f447800eed0 tx=0x7f447800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.146+0000 7f447cff9700 1 -- 172.21.15.158:0/3533080967 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4478010040 con 0x7f448015b6b0 2023-10-16T10:03:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.146+0000 7f447cff9700 1 -- 172.21.15.158:0/3533080967 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447800b9f0 con 0x7f448015b6b0 2023-10-16T10:03:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.146+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448006ca20 con 0x7f448015b6b0 2023-10-16T10:03:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.146+0000 7f447cff9700 1 -- 172.21.15.158:0/3533080967 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447800bda0 con 0x7f448015b6b0 2023-10-16T10:03:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.146+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4480150e70 con 0x7f448015b6b0 2023-10-16T10:03:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.147+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448006d1e0 con 0x7f448015b6b0 2023-10-16T10:03:33.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.148+0000 7f447cff9700 1 -- 172.21.15.158:0/3533080967 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 75) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4478026420 con 0x7f448015b6b0 2023-10-16T10:03:33.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.149+0000 7f447cff9700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 0x7f446c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.149+0000 7f447ffff700 1 -- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 msgr2=0x7f446c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:33.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.149+0000 7f447cff9700 1 -- 172.21.15.158:0/3533080967 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f447808a5a0 con 0x7f448015b6b0 2023-10-16T10:03:33.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.149+0000 7f447ffff700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 0x7f446c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:33.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.153+0000 7f447cff9700 1 -- 172.21.15.158:0/3533080967 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4478055200 con 0x7f448015b6b0 2023-10-16T10:03:33.158 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:33.155+0000 7f64d54c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:03:33.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.350+0000 7f447ffff700 1 -- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 msgr2=0x7f446c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:33.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.350+0000 7f447ffff700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 0x7f446c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:33.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.355+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f44800cc1b0 con 0x7f448015b6b0 2023-10-16T10:03:33.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.358+0000 7f447cff9700 1 -- 172.21.15.158:0/3533080967 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4478058820 con 0x7f448015b6b0 2023-10-16T10:03:33.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.360+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 msgr2=0x7f446c06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:33.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.360+0000 7f4486597700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 0x7f446c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.360+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f448015b6b0 msgr2=0x7f4480154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.360+0000 7f4486597700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f448015b6b0 0x7f4480154f50 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f447800eed0 tx=0x7f447800ef00 comp rx=0 tx=0).stop 2023-10-16T10:03:33.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 shutdown_connections 2023-10-16T10:03:33.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f446c06b070 0x7f446c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f448015b6b0 0x7f4480154f50 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44800b3460 0x7f4480150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 --2- 172.21.15.158:0/3533080967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44800d0c60 0x7f448006b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 >> 172.21.15.158:0/3533080967 conn(0x7f44800cdf20 msgr2=0x7f448015bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:33.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 shutdown_connections 2023-10-16T10:03:33.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.361+0000 7f4486597700 1 -- 172.21.15.158:0/3533080967 wait complete. 2023-10-16T10:03:33.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:33.375 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2023-10-16T10:03:33.396 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2023-10-16T10:03:33.462 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:33.460+0000 7f3b42f8d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:03:33.497 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:33.496+0000 7f64d54c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:03:33.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.560+0000 7f4a23da9700 1 Processor -- start 2023-10-16T10:03:33.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.561+0000 7f4a23da9700 1 -- start start 2023-10-16T10:03:33.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a23da9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4a1c163380 0x7f4a1c165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a23da9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c0d0ae0 con 0x7f4a1c15deb0 2023-10-16T10:03:33.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a23da9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c0d12d0 con 0x7f4a1c15dad0 2023-10-16T10:03:33.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a23da9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c0cbdf0 con 0x7f4a1c163380 2023-10-16T10:03:33.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a22346700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4a1c163380 0x7f4a1c165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a21344700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4a1c15deb0 0x7f4a1c0ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58670/0 (socket says 172.21.15.158:58670) 2023-10-16T10:03:33.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a21344700 1 -- 172.21.15.158:0/992937816 learned_addr learned my addr 172.21.15.158:0/992937816 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:33.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a22346700 1 -- 172.21.15.158:0/992937816 >> v1:172.21.15.172:6789/0 conn(0x7f4a1c15dad0 legacy=0x7f4a1c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a22346700 1 -- 172.21.15.158:0/992937816 >> v1:172.21.15.158:6789/0 conn(0x7f4a1c15deb0 legacy=0x7f4a1c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a22346700 1 -- 172.21.15.158:0/992937816 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c0cc280 con 0x7f4a1c163380 2023-10-16T10:03:33.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a20b43700 1 -- 172.21.15.158:0/992937816 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154290177 0 0) 0x7f4a1c0d0ae0 con 0x7f4a1c15deb0 2023-10-16T10:03:33.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a22346700 1 --2- 172.21.15.158:0/992937816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c163380 0x7f4a1c165770 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009d90 tx=0x7f4a1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5affe82241c4256a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:33.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a20b43700 1 -- 172.21.15.158:0/992937816 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1400e070 con 0x7f4a1c163380 2023-10-16T10:03:33.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a20b43700 1 -- 172.21.15.158:0/992937816 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1400bbd0 con 0x7f4a1c163380 2023-10-16T10:03:33.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.562+0000 7f4a20b43700 1 -- 172.21.15.158:0/992937816 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14012430 con 0x7f4a1c163380 2023-10-16T10:03:33.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.563+0000 7f4a23da9700 1 -- 172.21.15.158:0/992937816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c163380 msgr2=0x7f4a1c165770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.563+0000 7f4a23da9700 1 --2- 172.21.15.158:0/992937816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c163380 0x7f4a1c165770 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009d90 tx=0x7f4a1400b330 comp rx=0 tx=0).stop 2023-10-16T10:03:33.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.563+0000 7f4a23da9700 1 -- 172.21.15.158:0/992937816 shutdown_connections 2023-10-16T10:03:33.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.563+0000 7f4a23da9700 1 --2- 172.21.15.158:0/992937816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c163380 0x7f4a1c165770 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.563+0000 7f4a23da9700 1 -- 172.21.15.158:0/992937816 >> 172.21.15.158:0/992937816 conn(0x7f4a1c099a10 msgr2=0x7f4a1c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:33.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.563+0000 7f4a23da9700 1 -- 172.21.15.158:0/992937816 shutdown_connections 2023-10-16T10:03:33.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.563+0000 7f4a23da9700 1 -- 172.21.15.158:0/992937816 wait complete. 2023-10-16T10:03:33.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.564+0000 7f4a23da9700 1 Processor -- start 2023-10-16T10:03:33.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.564+0000 7f4a23da9700 1 -- start start 2023-10-16T10:03:33.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a23da9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a1c15dad0 0x7f4a1c0b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a23da9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c15deb0 0x7f4a1c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a23da9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a1c163380 0x7f4a1c1626a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a23da9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0c3f00 con 0x7f4a1c15deb0 2023-10-16T10:03:33.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a23da9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0b3910 con 0x7f4a1c15dad0 2023-10-16T10:03:33.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a23da9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0b3bf0 con 0x7f4a1c163380 2023-10-16T10:03:33.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21344700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c15deb0 0x7f4a1c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21344700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c15deb0 0x7f4a1c0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58502/0 (socket says 172.21.15.158:58502) 2023-10-16T10:03:33.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21344700 1 -- 172.21.15.158:0/2569039725 learned_addr learned my addr 172.21.15.158:0/2569039725 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:33.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a1c15dad0 0x7f4a1c0b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a22346700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a1c163380 0x7f4a1c1626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:33.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21344700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a1c163380 msgr2=0x7f4a1c1626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21344700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a1c163380 0x7f4a1c1626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21344700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a1c15dad0 msgr2=0x7f4a1c0b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:33.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.565+0000 7f4a21344700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a1c15dad0 0x7f4a1c0b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:33.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.566+0000 7f4a21344700 1 -- 172.21.15.158:0/2569039725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14009a70 con 0x7f4a1c15deb0 2023-10-16T10:03:33.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.566+0000 7f4a21344700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c15deb0 0x7f4a1c0b2fc0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00c520 tx=0x7f4a0c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:33.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.566+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c004c80 con 0x7f4a1c15deb0 2023-10-16T10:03:33.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.566+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c004e00 con 0x7f4a1c15deb0 2023-10-16T10:03:33.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.566+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c0c4150 con 0x7f4a1c15deb0 2023-10-16T10:03:33.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.566+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c007310 con 0x7f4a1c15deb0 2023-10-16T10:03:33.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.566+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c0b5d70 con 0x7f4a1c15deb0 2023-10-16T10:03:33.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.567+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c0b3db0 con 0x7f4a1c15deb0 2023-10-16T10:03:33.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.568+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 75) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4a0c007470 con 0x7f4a1c15deb0 2023-10-16T10:03:33.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.568+0000 7f4a127fc700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 0x7f4a0006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:33.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.568+0000 7f4a21b45700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 msgr2=0x7f4a0006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:33.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.568+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f4a0c089660 con 0x7f4a1c15deb0 2023-10-16T10:03:33.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.568+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 0x7f4a0006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:33.578 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:33.576+0000 7f3b42f8d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:03:33.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.573+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a0c054380 con 0x7f4a1c15deb0 2023-10-16T10:03:33.619 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:33.618+0000 7f64d54c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:03:33.652 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:33.651+0000 7f3b42f8d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:03:33.695 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:33.694+0000 7f64d54c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:03:33.732 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:33.731+0000 7f3b42f8d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:03:33.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.769+0000 7f4a21b45700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 msgr2=0x7f4a0006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:33.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.769+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 0x7f4a0006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:33.776 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:33.775+0000 7f64d54c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:03:33.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:33.802+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 --> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a1c0d0d10 con 0x7f4a0006b070 2023-10-16T10:03:34.101 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:34.099+0000 7f3b42f8d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:03:34.145 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:34.144+0000 7f64d54c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:03:34.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:34.170+0000 7f4a21b45700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 msgr2=0x7f4a0006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:34.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:34.170+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 0x7f4a0006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T10:03:34.671 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:34.670+0000 7f3b42f8d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:03:34.718 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:34.717+0000 7f64d54c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:03:34.739 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:34.739+0000 7f3b42f8d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:03:34.787 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:34.786+0000 7f64d54c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:03:34.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:34.971+0000 7f4a21b45700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 msgr2=0x7f4a0006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:34.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:34.971+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 0x7f4a0006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T10:03:35.366 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:35.366+0000 7f3b42f8d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:35.426 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:35.425+0000 7f64d54c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:35.443 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:35.443+0000 7f3b42f8d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:03:35.503 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:35.502+0000 7f64d54c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:03:35.586 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:35.586+0000 7f3b42f8d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:35.648 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:35.647+0000 7f64d54c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:35.812 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:35.812+0000 7f3b42f8d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:03:35.876 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:35.875+0000 7f64d54c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:03:35.885 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:35.885+0000 7f3b42f8d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:03:35.950 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:35.949+0000 7f64d54c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:03:36.148 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:36.147+0000 7f3b42f8d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:03:36.215 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:36.214+0000 7f64d54c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:03:36.367 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:36.367+0000 7f3b42f8d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:03:36.438 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:36.437+0000 7f64d54c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:03:36.515 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:36.514+0000 7f3b42f8d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:03:36.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:36.572+0000 7f4a21b45700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 msgr2=0x7f4a0006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2342578387 2023-10-16T10:03:36.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:36.572+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 0x7f4a0006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T10:03:36.588 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:36.587+0000 7f64d54c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:03:36.627 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:36.626+0000 7f3b42f8d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:03:36.703 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:36.701+0000 7f64d54c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:03:36.781 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:36.781+0000 7f3b42f8d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:03:36.860 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:36.859+0000 7f64d54c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:03:37.044 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:37.043+0000 7f3b42f8d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:37.124 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.123+0000 7f64d54c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:37.205 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:37.204+0000 7f3b42f8d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:03:37.287 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.286+0000 7f64d54c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:03:37.726 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:37.725+0000 7f3b42f8d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:03:37.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:37.795+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 76) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4a0c054750 con 0x7f4a1c15deb0 2023-10-16T10:03:37.816 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.815+0000 7f64d54c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:03:37.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:37.834+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 77) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f4a0c087840 con 0x7f4a1c15deb0 2023-10-16T10:03:37.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:37.834+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 msgr2=0x7f4a0006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:37.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:37.834+0000 7f4a127fc700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/2342578387,v1:172.21.15.158:6833/2342578387] conn(0x7f4a0006b070 0x7f4a0006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:37.976 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644b4a4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.977 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644b4a4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.977 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644b4a4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.977 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644b4a4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.977 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644b4a4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.977 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644849e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.978 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644849e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.978 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644849e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.978 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644849e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:37.978 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:37.975+0000 7f644849e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:38.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.840+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f4a0c0813a0 con 0x7f4a1c15deb0 2023-10-16T10:03:38.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.841+0000 7f4a127fc700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4a000279c0 0x7f4a00027da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:38.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.841+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a00071580 con 0x7f4a000279c0 2023-10-16T10:03:38.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.842+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4a000279c0 0x7f4a00027da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:38.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.844+0000 7f4a21b45700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4a000279c0 0x7f4a00027da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a18007fd0 tx=0x7f4a18007ec0 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:38.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.906+0000 7f4a127fc700 1 -- 172.21.15.158:0/2569039725 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f4a00071580 con 0x7f4a000279c0 2023-10-16T10:03:38.908 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1/minutes 2023-10-16T10:03:38.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.908+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4a000279c0 msgr2=0x7f4a00027da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:38.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.908+0000 7f4a23da9700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4a000279c0 0x7f4a00027da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a18007fd0 tx=0x7f4a18007ec0 comp rx=0 tx=0).stop 2023-10-16T10:03:38.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.908+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c15deb0 msgr2=0x7f4a1c0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:38.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.908+0000 7f4a23da9700 1 --2- 172.21.15.158:0/2569039725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a1c15deb0 0x7f4a1c0b2fc0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00c520 tx=0x7f4a0c00c830 comp rx=0 tx=0).stop 2023-10-16T10:03:38.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.908+0000 7f4a22346700 1 -- 172.21.15.158:0/2569039725 reap_dead start 2023-10-16T10:03:38.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.909+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 shutdown_connections 2023-10-16T10:03:38.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.909+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 >> 172.21.15.158:0/2569039725 conn(0x7f4a1c099a10 msgr2=0x7f4a1c0c6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:38.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.909+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 shutdown_connections 2023-10-16T10:03:38.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:38.909+0000 7f4a23da9700 1 -- 172.21.15.158:0/2569039725 wait complete. 2023-10-16T10:03:38.923 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2023-10-16T10:03:38.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:38.924 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2023-10-16T10:03:38.943 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2023-10-16T10:03:39.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.107+0000 7fe487437700 1 Processor -- start 2023-10-16T10:03:39.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.107+0000 7fe487437700 1 -- start start 2023-10-16T10:03:39.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe487437700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe48006afe0 0x7fe4800caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe487437700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe4800cc590 0x7fe4800afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe487437700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4800cbd30 con 0x7fe48015da40 2023-10-16T10:03:39.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe487437700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4800cc1c0 con 0x7fe48006afe0 2023-10-16T10:03:39.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe487437700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48015de20 con 0x7fe4800cc590 2023-10-16T10:03:39.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe4849d2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe48006afe0 0x7fe4800caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe4851d3700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe48015da40 0x7fe48006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58878/0 (socket says 172.21.15.158:58878) 2023-10-16T10:03:39.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe4851d3700 1 -- 172.21.15.158:0/3971915225 learned_addr learned my addr 172.21.15.158:0/3971915225 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:39.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe4849d2700 1 -- 172.21.15.158:0/3971915225 >> v2:172.21.15.172:3300/0 conn(0x7fe4800cc590 msgr2=0x7fe4800afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:39.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe4849d2700 1 --2- 172.21.15.158:0/3971915225 >> v2:172.21.15.172:3300/0 conn(0x7fe4800cc590 0x7fe4800afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe4849d2700 1 -- 172.21.15.158:0/3971915225 >> v1:172.21.15.158:6789/0 conn(0x7fe48015da40 legacy=0x7fe48006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe477fff700 1 -- 172.21.15.158:0/3971915225 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578338668 0 0) 0x7fe4800cbd30 con 0x7fe48015da40 2023-10-16T10:03:39.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.108+0000 7fe4849d2700 1 -- 172.21.15.158:0/3971915225 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4800ce260 con 0x7fe48006afe0 2023-10-16T10:03:39.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe4849d2700 1 --2- 172.21.15.158:0/3971915225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48006afe0 0x7fe4800caa20 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fe4800cdd70 tx=0x7fe47c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd6293016fcce50 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:39.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe477fff700 1 -- 172.21.15.158:0/3971915225 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011830 con 0x7fe48006afe0 2023-10-16T10:03:39.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe477fff700 1 -- 172.21.15.158:0/3971915225 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c011990 con 0x7fe48006afe0 2023-10-16T10:03:39.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe477fff700 1 -- 172.21.15.158:0/3971915225 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011d20 con 0x7fe48006afe0 2023-10-16T10:03:39.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 -- 172.21.15.158:0/3971915225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48006afe0 msgr2=0x7fe4800caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 --2- 172.21.15.158:0/3971915225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48006afe0 0x7fe4800caa20 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fe4800cdd70 tx=0x7fe47c009b20 comp rx=0 tx=0).stop 2023-10-16T10:03:39.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 -- 172.21.15.158:0/3971915225 shutdown_connections 2023-10-16T10:03:39.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 --2- 172.21.15.158:0/3971915225 >> v2:172.21.15.172:3300/0 conn(0x7fe4800cc590 0x7fe4800afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 --2- 172.21.15.158:0/3971915225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48006afe0 0x7fe4800caa20 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 -- 172.21.15.158:0/3971915225 >> 172.21.15.158:0/3971915225 conn(0x7fe480099a10 msgr2=0x7fe4800d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:39.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 -- 172.21.15.158:0/3971915225 shutdown_connections 2023-10-16T10:03:39.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.109+0000 7fe487437700 1 -- 172.21.15.158:0/3971915225 wait complete. 2023-10-16T10:03:39.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 Processor -- start 2023-10-16T10:03:39.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 -- start start 2023-10-16T10:03:39.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48006afe0 0x7fe4800c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4800cc590 0x7fe48014c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48015da40 0x7fe48006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4800c63b0 con 0x7fe48015da40 2023-10-16T10:03:39.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4800c6690 con 0x7fe4800cc590 2023-10-16T10:03:39.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.111+0000 7fe487437700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4800c6970 con 0x7fe48006afe0 2023-10-16T10:03:39.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48015da40 0x7fe48006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48015da40 0x7fe48006e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58712/0 (socket says 172.21.15.158:58712) 2023-10-16T10:03:39.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 -- 172.21.15.158:0/2684824932 learned_addr learned my addr 172.21.15.158:0/2684824932 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:39.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4851d3700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48006afe0 0x7fe4800c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4849d2700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4800cc590 0x7fe48014c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 -- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48006afe0 msgr2=0x7fe4800c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48006afe0 0x7fe4800c5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 -- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4800cc590 msgr2=0x7fe48014c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4800cc590 0x7fe48014c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 -- 172.21.15.158:0/2684824932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0094a0 con 0x7fe48015da40 2023-10-16T10:03:39.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4849d2700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4800cc590 0x7fe48014c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:39.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe4859d4700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48015da40 0x7fe48006e860 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fe47800b970 tx=0x7fe47800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:39.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe475ffb700 1 -- 172.21.15.158:0/2684824932 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47800c9b0 con 0x7fe48015da40 2023-10-16T10:03:39.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe475ffb700 1 -- 172.21.15.158:0/2684824932 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47800cb30 con 0x7fe48015da40 2023-10-16T10:03:39.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4800c6c50 con 0x7fe48015da40 2023-10-16T10:03:39.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe475ffb700 1 -- 172.21.15.158:0/2684824932 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478014440 con 0x7fe48015da40 2023-10-16T10:03:39.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.112+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48006ef80 con 0x7fe48015da40 2023-10-16T10:03:39.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.113+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4801e2c70 con 0x7fe48015da40 2023-10-16T10:03:39.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.114+0000 7fe475ffb700 1 -- 172.21.15.158:0/2684824932 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe47801d3b0 con 0x7fe48015da40 2023-10-16T10:03:39.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.114+0000 7fe475ffb700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe46406ada0 0x7fe46406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.114+0000 7fe475ffb700 1 -- 172.21.15.158:0/2684824932 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fe478006e40 con 0x7fe48015da40 2023-10-16T10:03:39.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.114+0000 7fe4851d3700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe46406ada0 0x7fe46406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.114+0000 7fe4851d3700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe46406ada0 0x7fe46406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe470000f60 tx=0x7fe470007c90 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:39.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.119+0000 7fe475ffb700 1 -- 172.21.15.158:0/2684824932 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe478054290 con 0x7fe48015da40 2023-10-16T10:03:39.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.340+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4800c6800 con 0x7fe46406ada0 2023-10-16T10:03:39.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.396+0000 7fe475ffb700 1 -- 172.21.15.158:0/2684824932 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fe4800c6800 con 0x7fe46406ada0 2023-10-16T10:03:39.398 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1/hourly 2023-10-16T10:03:39.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe46406ada0 msgr2=0x7fe46406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe46406ada0 0x7fe46406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe470000f60 tx=0x7fe470007c90 comp rx=0 tx=0).stop 2023-10-16T10:03:39.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48015da40 msgr2=0x7fe48006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48015da40 0x7fe48006e860 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fe47800b970 tx=0x7fe47800bd30 comp rx=0 tx=0).stop 2023-10-16T10:03:39.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 shutdown_connections 2023-10-16T10:03:39.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe46406ada0 0x7fe46406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48015da40 0x7fe48006e860 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48006afe0 0x7fe4800c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 --2- 172.21.15.158:0/2684824932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4800cc590 0x7fe48014c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 >> 172.21.15.158:0/2684824932 conn(0x7fe480099a10 msgr2=0x7fe48015c720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:39.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 shutdown_connections 2023-10-16T10:03:39.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.399+0000 7fe487437700 1 -- 172.21.15.158:0/2684824932 wait complete. 2023-10-16T10:03:39.412 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2023-10-16T10:03:39.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:39.413 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2023-10-16T10:03:39.431 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2023-10-16T10:03:39.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d25d61700 1 Processor -- start 2023-10-16T10:03:39.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d25d61700 1 -- start start 2023-10-16T10:03:39.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d25d61700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 0x7f7d200cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d25d61700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200caa80 con 0x7f7d2015d770 2023-10-16T10:03:39.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d25d61700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200cad60 con 0x7f7d2015db50 2023-10-16T10:03:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d25d61700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d200cb040 con 0x7f7d200cfd90 2023-10-16T10:03:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d1f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7d2015d770 0x7f7d200cd7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58892/0 (socket says 172.21.15.158:58892) 2023-10-16T10:03:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.594+0000 7f7d1f7fe700 1 -- 172.21.15.158:0/2222754006 learned_addr learned my addr 172.21.15.158:0/2222754006 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1ffff700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 0x7f7d200cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611066998 0 0) 0x7f7d200caa80 con 0x7f7d2015d770 2023-10-16T10:03:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d080035b0 con 0x7f7d2015d770 2023-10-16T10:03:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1970148202 0 0) 0x7f7d080035b0 con 0x7f7d2015d770 2023-10-16T10:03:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 msgr2=0x7f7d200cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 0x7f7d200cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 >> v1:172.21.15.172:6789/0 conn(0x7f7d2015db50 legacy=0x7f7d200cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200cb320 con 0x7f7d2015d770 2023-10-16T10:03:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7d14001670 con 0x7f7d2015d770 2023-10-16T10:03:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 >> v1:172.21.15.158:6789/0 conn(0x7f7d2015d770 legacy=0x7f7d200cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 0x7f7d08003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d08003f70 0x7f7d08004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08004d90 0x7f7d08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d200cb040 con 0x7f7d08003f70 2023-10-16T10:03:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0800ba70 con 0x7f7d08004d90 2023-10-16T10:03:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0800bc30 con 0x7f7d200cfd90 2023-10-16T10:03:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1effd700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d08003f70 0x7f7d08004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1f7fe700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 0x7f7d08003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1ffff700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08004d90 0x7f7d08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1effd700 1 -- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 msgr2=0x7f7d08003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1effd700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d200cfd90 0x7f7d08003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1effd700 1 -- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08004d90 msgr2=0x7f7d08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1effd700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08004d90 0x7f7d08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.595+0000 7f7d1effd700 1 -- 172.21.15.158:0/2222754006 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d080035b0 con 0x7f7d08003f70 2023-10-16T10:03:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d1effd700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d08003f70 0x7f7d08004850 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f7d100018d0 tx=0x7f7d08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8ac6ab57c7910e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10012620 con 0x7f7d08003f70 2023-10-16T10:03:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d10012780 con 0x7f7d08003f70 2023-10-16T10:03:39.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d1e7fc700 1 -- 172.21.15.158:0/2222754006 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10012b30 con 0x7f7d08003f70 2023-10-16T10:03:39.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d25d61700 1 -- 172.21.15.158:0/2222754006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d08003f70 msgr2=0x7f7d08004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d25d61700 1 --2- 172.21.15.158:0/2222754006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d08003f70 0x7f7d08004850 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f7d100018d0 tx=0x7f7d08005880 comp rx=0 tx=0).stop 2023-10-16T10:03:39.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d1ffff700 1 -- 172.21.15.158:0/2222754006 reap_dead start 2023-10-16T10:03:39.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d25d61700 1 -- 172.21.15.158:0/2222754006 shutdown_connections 2023-10-16T10:03:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d25d61700 1 -- 172.21.15.158:0/2222754006 >> 172.21.15.158:0/2222754006 conn(0x7f7d20099a10 msgr2=0x7f7d200d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d25d61700 1 -- 172.21.15.158:0/2222754006 shutdown_connections 2023-10-16T10:03:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.596+0000 7f7d25d61700 1 -- 172.21.15.158:0/2222754006 wait complete. 2023-10-16T10:03:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.597+0000 7f7d25d61700 1 Processor -- start 2023-10-16T10:03:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.598+0000 7f7d25d61700 1 -- start start 2023-10-16T10:03:39.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.598+0000 7f7d25d61700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08003f70 0x7f7d20151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.598+0000 7f7d25d61700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20157d10 0x7f7d20145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.598+0000 7f7d25d61700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d201566e0 0x7f7d200c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.598+0000 7f7d25d61700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201578f0 con 0x7f7d20157d10 2023-10-16T10:03:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.598+0000 7f7d25d61700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d20156c70 con 0x7f7d08003f70 2023-10-16T10:03:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.598+0000 7f7d25d61700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d20156f50 con 0x7f7d201566e0 2023-10-16T10:03:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20157d10 0x7f7d20145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20157d10 0x7f7d20145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58730/0 (socket says 172.21.15.158:58730) 2023-10-16T10:03:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 -- 172.21.15.158:0/1234145043 learned_addr learned my addr 172.21.15.158:0/1234145043 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1f7fe700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08003f70 0x7f7d20151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1ffff700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d201566e0 0x7f7d200c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 -- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d201566e0 msgr2=0x7f7d200c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d201566e0 0x7f7d200c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 -- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08003f70 msgr2=0x7f7d20151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08003f70 0x7f7d20151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 -- 172.21.15.158:0/1234145043 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20157230 con 0x7f7d20157d10 2023-10-16T10:03:39.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d1effd700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20157d10 0x7f7d20145820 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000c0e0 tx=0x7f7d10001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:39.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d07fff700 1 -- 172.21.15.158:0/1234145043 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10012af0 con 0x7f7d20157d10 2023-10-16T10:03:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d07fff700 1 -- 172.21.15.158:0/1234145043 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d10021070 con 0x7f7d20157d10 2023-10-16T10:03:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d20157510 con 0x7f7d20157d10 2023-10-16T10:03:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d07fff700 1 -- 172.21.15.158:0/1234145043 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10012e90 con 0x7f7d20157d10 2023-10-16T10:03:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.599+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d200c4650 con 0x7f7d20157d10 2023-10-16T10:03:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.600+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d20157a30 con 0x7f7d20157d10 2023-10-16T10:03:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.601+0000 7f7d07fff700 1 -- 172.21.15.158:0/1234145043 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f7d10023030 con 0x7f7d20157d10 2023-10-16T10:03:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.601+0000 7f7d07fff700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7d0c06add0 0x7f7d0c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.601+0000 7f7d07fff700 1 -- 172.21.15.158:0/1234145043 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f7d1008a5b0 con 0x7f7d20157d10 2023-10-16T10:03:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.601+0000 7f7d1f7fe700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7d0c06add0 0x7f7d0c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.602+0000 7f7d1f7fe700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7d0c06add0 0x7f7d0c06d290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7d14001cb0 tx=0x7f7d14001c40 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.606+0000 7f7d07fff700 1 -- 172.21.15.158:0/1234145043 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d20157a30 con 0x7f7d20157d10 2023-10-16T10:03:39.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.831+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d201570c0 con 0x7f7d0c06add0 2023-10-16T10:03:39.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.889+0000 7f7d07fff700 1 -- 172.21.15.158:0/1234145043 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7d201570c0 con 0x7f7d0c06add0 2023-10-16T10:03:39.891 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1/daily 2023-10-16T10:03:39.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7d0c06add0 msgr2=0x7f7d0c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7d0c06add0 0x7f7d0c06d290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7d14001cb0 tx=0x7f7d14001c40 comp rx=0 tx=0).stop 2023-10-16T10:03:39.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20157d10 msgr2=0x7f7d20145820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:39.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20157d10 0x7f7d20145820 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000c0e0 tx=0x7f7d10001230 comp rx=0 tx=0).stop 2023-10-16T10:03:39.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 shutdown_connections 2023-10-16T10:03:39.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7d0c06add0 0x7f7d0c06d290 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d201566e0 0x7f7d200c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d08003f70 0x7f7d20151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 --2- 172.21.15.158:0/1234145043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d20157d10 0x7f7d20145820 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:39.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 >> 172.21.15.158:0/1234145043 conn(0x7f7d20099a10 msgr2=0x7f7d200d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:39.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 shutdown_connections 2023-10-16T10:03:39.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:39.892+0000 7f7d25d61700 1 -- 172.21.15.158:0/1234145043 wait complete. 2023-10-16T10:03:39.907 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2023-10-16T10:03:39.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:39.908 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2023-10-16T10:03:39.926 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2023-10-16T10:03:40.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.088+0000 7f005f23b700 1 Processor -- start 2023-10-16T10:03:40.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.088+0000 7f005f23b700 1 -- start start 2023-10-16T10:03:40.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005f23b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151220 0x7f00580665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005f23b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0058151600 0x7f00580c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005f23b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00580cf930 con 0x7f00580cf190 2023-10-16T10:03:40.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005f23b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00580cf650 con 0x7f0058151600 2023-10-16T10:03:40.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005f23b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00580c51b0 con 0x7f0058151220 2023-10-16T10:03:40.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005d7d8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0058151600 0x7f00580c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f0057fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f00580cf190 0x7f005806d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:34600/0 (socket says 172.21.15.158:34600) 2023-10-16T10:03:40.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f0057fff700 1 -- 172.21.15.158:0/3913378752 learned_addr learned my addr 172.21.15.158:0/3913378752 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:40.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005cfd7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151220 0x7f00580665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005d7d8700 1 -- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151220 msgr2=0x7f00580665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:40.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005d7d8700 1 --2- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151220 0x7f00580665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005d7d8700 1 -- 172.21.15.158:0/3913378752 >> v1:172.21.15.172:6789/0 conn(0x7f00580cf190 legacy=0x7f005806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005d7d8700 1 -- 172.21.15.158:0/3913378752 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580c5490 con 0x7f0058151600 2023-10-16T10:03:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f00577fe700 1 -- 172.21.15.158:0/3913378752 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283308044 0 0) 0x7f00580cf930 con 0x7f00580cf190 2023-10-16T10:03:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.089+0000 7f005cfd7700 1 --2- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151220 0x7f00580665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.090+0000 7f005d7d8700 1 --2- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151600 0x7f00580c5c90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0050009d90 tx=0x7f005000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73da89fa51ca5af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:40.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.090+0000 7f00577fe700 1 -- 172.21.15.158:0/3913378752 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005000e070 con 0x7f0058151600 2023-10-16T10:03:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.090+0000 7f00577fe700 1 -- 172.21.15.158:0/3913378752 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005000bb20 con 0x7f0058151600 2023-10-16T10:03:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.090+0000 7f00577fe700 1 -- 172.21.15.158:0/3913378752 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050012430 con 0x7f0058151600 2023-10-16T10:03:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.090+0000 7f005f23b700 1 -- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151600 msgr2=0x7f00580c5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.090+0000 7f005f23b700 1 --2- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151600 0x7f00580c5c90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0050009d90 tx=0x7f005000b330 comp rx=0 tx=0).stop 2023-10-16T10:03:40.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.091+0000 7f005f23b700 1 -- 172.21.15.158:0/3913378752 shutdown_connections 2023-10-16T10:03:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.091+0000 7f005f23b700 1 --2- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151600 0x7f00580c5c90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.091+0000 7f005f23b700 1 --2- 172.21.15.158:0/3913378752 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151220 0x7f00580665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.091+0000 7f005f23b700 1 -- 172.21.15.158:0/3913378752 >> 172.21.15.158:0/3913378752 conn(0x7f00580cc5b0 msgr2=0x7f00580cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.091+0000 7f005f23b700 1 -- 172.21.15.158:0/3913378752 shutdown_connections 2023-10-16T10:03:40.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.091+0000 7f005f23b700 1 -- 172.21.15.158:0/3913378752 wait complete. 2023-10-16T10:03:40.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.092+0000 7f005f23b700 1 Processor -- start 2023-10-16T10:03:40.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.092+0000 7f005f23b700 1 -- start start 2023-10-16T10:03:40.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005f23b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00580cf190 0x7f00581405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005f23b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151220 0x7f005815e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005f23b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151600 0x7f005815c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005f23b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058140b10 con 0x7f00580cf190 2023-10-16T10:03:40.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005f23b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580b3f90 con 0x7f0058151220 2023-10-16T10:03:40.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005f23b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00580b3380 con 0x7f0058151600 2023-10-16T10:03:40.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00580cf190 0x7f00581405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00580cf190 0x7f00581405d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58758/0 (socket says 172.21.15.158:58758) 2023-10-16T10:03:40.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 -- 172.21.15.158:0/3669820138 learned_addr learned my addr 172.21.15.158:0/3669820138 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:40.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f0057fff700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151220 0x7f005815e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005d7d8700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151600 0x7f005815c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 -- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151600 msgr2=0x7f005815c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151600 0x7f005815c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 -- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151220 msgr2=0x7f005815e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151220 0x7f005815e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 -- 172.21.15.158:0/3669820138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c0094a0 con 0x7f00580cf190 2023-10-16T10:03:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005cfd7700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00580cf190 0x7f00581405d0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f004c00ebe0 tx=0x7f004c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f00557fa700 1 -- 172.21.15.158:0/3669820138 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c00c850 con 0x7f00580cf190 2023-10-16T10:03:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f00557fa700 1 -- 172.21.15.158:0/3669820138 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c00c9b0 con 0x7f00580cf190 2023-10-16T10:03:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0050009a70 con 0x7f00580cf190 2023-10-16T10:03:40.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.093+0000 7f00557fa700 1 -- 172.21.15.158:0/3669820138 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c00cd40 con 0x7f00580cf190 2023-10-16T10:03:40.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.094+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00580b3940 con 0x7f00580cf190 2023-10-16T10:03:40.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.094+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00580b4270 con 0x7f00580cf190 2023-10-16T10:03:40.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.095+0000 7f00557fa700 1 -- 172.21.15.158:0/3669820138 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f004c013470 con 0x7f00580cf190 2023-10-16T10:03:40.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.095+0000 7f00557fa700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f003c06ada0 0x7f003c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.095+0000 7f00557fa700 1 -- 172.21.15.158:0/3669820138 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f004c0897b0 con 0x7f00580cf190 2023-10-16T10:03:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.095+0000 7f0057fff700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f003c06ada0 0x7f003c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.096+0000 7f0057fff700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f003c06ada0 0x7f003c06d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00580b5670 tx=0x7f0044007e70 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:40.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.100+0000 7f00557fa700 1 -- 172.21.15.158:0/3669820138 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f004c054740 con 0x7f00580cf190 2023-10-16T10:03:40.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.318+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7f00580c5770 con 0x7f003c06ada0 2023-10-16T10:03:40.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.378+0000 7f00557fa700 1 -- 172.21.15.158:0/3669820138 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f0044004ab0 con 0x7f003c06ada0 2023-10-16T10:03:40.379 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1/weekly 2023-10-16T10:03:40.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.380+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f003c06ada0 msgr2=0x7f003c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.380+0000 7f005f23b700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f003c06ada0 0x7f003c06d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00580b5670 tx=0x7f0044007e70 comp rx=0 tx=0).stop 2023-10-16T10:03:40.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.380+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00580cf190 msgr2=0x7f00581405d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.380+0000 7f005f23b700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00580cf190 0x7f00581405d0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f004c00ebe0 tx=0x7f004c00eef0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 shutdown_connections 2023-10-16T10:03:40.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f003c06ada0 0x7f003c06d260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0058151600 0x7f005815c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00580cf190 0x7f00581405d0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 --2- 172.21.15.158:0/3669820138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0058151220 0x7f005815e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 >> 172.21.15.158:0/3669820138 conn(0x7f00580cc5b0 msgr2=0x7f00580cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:40.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 shutdown_connections 2023-10-16T10:03:40.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.381+0000 7f005f23b700 1 -- 172.21.15.158:0/3669820138 wait complete. 2023-10-16T10:03:40.394 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2023-10-16T10:03:40.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:40.396 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2023-10-16T10:03:40.415 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2023-10-16T10:03:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.577+0000 7f264652a700 1 Processor -- start 2023-10-16T10:03:40.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.577+0000 7f264652a700 1 -- start start 2023-10-16T10:03:40.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f264652a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f264006afe0 0x7f26400caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f264652a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400cbd30 con 0x7f26400cc590 2023-10-16T10:03:40.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f264652a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400cc1c0 con 0x7f264015da40 2023-10-16T10:03:40.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f264652a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264015de20 con 0x7f264006afe0 2023-10-16T10:03:40.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f263f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f264006afe0 0x7f26400caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f263f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f264006afe0 0x7f26400caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58768/0 (socket says 172.21.15.158:58768) 2023-10-16T10:03:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f263f7fe700 1 -- 172.21.15.158:0/3546657770 learned_addr learned my addr 172.21.15.158:0/3546657770 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f263f7fe700 1 -- 172.21.15.158:0/3546657770 >> v1:172.21.15.172:6789/0 conn(0x7f264015da40 legacy=0x7f264006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f263f7fe700 1 -- 172.21.15.158:0/3546657770 >> v1:172.21.15.158:6789/0 conn(0x7f26400cc590 legacy=0x7f26400afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f263f7fe700 1 -- 172.21.15.158:0/3546657770 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400ce260 con 0x7f264006afe0 2023-10-16T10:03:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.578+0000 7f263f7fe700 1 --2- 172.21.15.158:0/3546657770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f264006afe0 0x7f26400caa20 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f26300098f0 tx=0x7f2630009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1bcd690e29e1ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f263effd700 1 -- 172.21.15.158:0/3546657770 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630004230 con 0x7f264006afe0 2023-10-16T10:03:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f263effd700 1 -- 172.21.15.158:0/3546657770 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2630004390 con 0x7f264006afe0 2023-10-16T10:03:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f263effd700 1 -- 172.21.15.158:0/3546657770 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300036a0 con 0x7f264006afe0 2023-10-16T10:03:40.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f264652a700 1 -- 172.21.15.158:0/3546657770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f264006afe0 msgr2=0x7f26400caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f264652a700 1 --2- 172.21.15.158:0/3546657770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f264006afe0 0x7f26400caa20 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f26300098f0 tx=0x7f2630009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f264652a700 1 -- 172.21.15.158:0/3546657770 shutdown_connections 2023-10-16T10:03:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f264652a700 1 --2- 172.21.15.158:0/3546657770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f264006afe0 0x7f26400caa20 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f264652a700 1 -- 172.21.15.158:0/3546657770 >> 172.21.15.158:0/3546657770 conn(0x7f2640099a10 msgr2=0x7f26400d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:40.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f264652a700 1 -- 172.21.15.158:0/3546657770 shutdown_connections 2023-10-16T10:03:40.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.579+0000 7f264652a700 1 -- 172.21.15.158:0/3546657770 wait complete. 2023-10-16T10:03:40.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.580+0000 7f264652a700 1 Processor -- start 2023-10-16T10:03:40.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f264652a700 1 -- start start 2023-10-16T10:03:40.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f264652a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f264006afe0 0x7f26401e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f264652a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26400cc590 0x7f26401f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f264652a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264015da40 0x7f264006e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f264652a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26401e68a0 con 0x7f26400cc590 2023-10-16T10:03:40.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f264652a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26401e5c90 con 0x7f264015da40 2023-10-16T10:03:40.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f264652a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26401e5f70 con 0x7f264006afe0 2023-10-16T10:03:40.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f263f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26400cc590 0x7f26401f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f263f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26400cc590 0x7f26401f6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58774/0 (socket says 172.21.15.158:58774) 2023-10-16T10:03:40.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.581+0000 7f263f7fe700 1 -- 172.21.15.158:0/3757134497 learned_addr learned my addr 172.21.15.158:0/3757134497 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:40.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f2644ac7700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264015da40 0x7f264006e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263ffff700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f264006afe0 0x7f26401e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263f7fe700 1 -- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f264006afe0 msgr2=0x7f26401e6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263f7fe700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f264006afe0 0x7f26401e6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263f7fe700 1 -- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264015da40 msgr2=0x7f264006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263f7fe700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264015da40 0x7f264006e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263f7fe700 1 -- 172.21.15.158:0/3757134497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2630009550 con 0x7f26400cc590 2023-10-16T10:03:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263f7fe700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26400cc590 0x7f26401f6ef0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f26400702f0 tx=0x7f2630004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630003dc0 con 0x7f26400cc590 2023-10-16T10:03:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263001f070 con 0x7f26400cc590 2023-10-16T10:03:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263000b8a0 con 0x7f26400cc590 2023-10-16T10:03:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26401e6250 con 0x7f26400cc590 2023-10-16T10:03:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.582+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26401f7430 con 0x7f26400cc590 2023-10-16T10:03:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.583+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26400681b0 con 0x7f26400cc590 2023-10-16T10:03:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.584+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2630021030 con 0x7f26400cc590 2023-10-16T10:03:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.584+0000 7f263cff9700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f262c06add0 0x7f262c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.584+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f263008ae40 con 0x7f26400cc590 2023-10-16T10:03:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.584+0000 7f263ffff700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f262c06add0 0x7f262c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.584+0000 7f263ffff700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f262c06add0 0x7f262c06d290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f263400a7d0 tx=0x7f2634008040 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:40.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.589+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2630055dd0 con 0x7f26400cc590 2023-10-16T10:03:40.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.807+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f26400cb400 con 0x7f262c06add0 2023-10-16T10:03:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.849+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f26300561a0 con 0x7f26400cc590 2023-10-16T10:03:40.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.872+0000 7f263cff9700 1 -- 172.21.15.158:0/3757134497 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f263400a420 con 0x7f262c06add0 2023-10-16T10:03:40.874 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1/monthly 2023-10-16T10:03:40.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f262c06add0 msgr2=0x7f262c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f262c06add0 0x7f262c06d290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f263400a7d0 tx=0x7f2634008040 comp rx=0 tx=0).stop 2023-10-16T10:03:40.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26400cc590 msgr2=0x7f26401f6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:40.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26400cc590 0x7f26401f6ef0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f26400702f0 tx=0x7f2630004750 comp rx=0 tx=0).stop 2023-10-16T10:03:40.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 shutdown_connections 2023-10-16T10:03:40.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f262c06add0 0x7f262c06d290 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264015da40 0x7f264006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f264006afe0 0x7f26401e6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 --2- 172.21.15.158:0/3757134497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26400cc590 0x7f26401f6ef0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:40.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 >> 172.21.15.158:0/3757134497 conn(0x7f2640099a10 msgr2=0x7f26400b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:40.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 shutdown_connections 2023-10-16T10:03:40.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:40.874+0000 7f264652a700 1 -- 172.21.15.158:0/3757134497 wait complete. 2023-10-16T10:03:40.888 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2023-10-16T10:03:40.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:40.889 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2023-10-16T10:03:40.909 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2023-10-16T10:03:41.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.077+0000 7f540900c700 1 Processor -- start 2023-10-16T10:03:41.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.077+0000 7f540900c700 1 -- start start 2023-10-16T10:03:41.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.077+0000 7f540900c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5404159280 0x7f540406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.077+0000 7f540900c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f54040cbef0 0x7f54040cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.077+0000 7f540900c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5404159660 con 0x7f54040d0e10 2023-10-16T10:03:41.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.077+0000 7f540900c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54040cb710 con 0x7f54040cbef0 2023-10-16T10:03:41.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.077+0000 7f540900c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540406b120 con 0x7f5404159280 2023-10-16T10:03:41.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f54040cbef0 0x7f54040cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f54040cbef0 0x7f54040cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58784/0 (socket says 172.21.15.158:58784) 2023-10-16T10:03:41.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 -- 172.21.15.158:0/4272321208 learned_addr learned my addr 172.21.15.158:0/4272321208 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:41.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 -- 172.21.15.158:0/4272321208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5404159280 msgr2=0x7f540406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:41.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f5402d9d700 1 --2- 172.21.15.158:0/4272321208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5404159280 0x7f540406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 --2- 172.21.15.158:0/4272321208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5404159280 0x7f540406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 -- 172.21.15.158:0/4272321208 >> v1:172.21.15.158:6789/0 conn(0x7f54040d0e10 legacy=0x7f540415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 -- 172.21.15.158:0/4272321208 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540406b400 con 0x7f54040cbef0 2023-10-16T10:03:41.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f5402d9d700 1 --2- 172.21.15.158:0/4272321208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5404159280 0x7f540406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:41.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f540359e700 1 --2- 172.21.15.158:0/4272321208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040cbef0 0x7f54040cc380 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009e50 tx=0x7f53fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b29c2f0f8d46ca6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f5401d9b700 1 -- 172.21.15.158:0/4272321208 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc00e070 con 0x7f54040cbef0 2023-10-16T10:03:41.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f5401d9b700 1 -- 172.21.15.158:0/4272321208 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc00b9c0 con 0x7f54040cbef0 2023-10-16T10:03:41.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.078+0000 7f5401d9b700 1 -- 172.21.15.158:0/4272321208 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc00bd70 con 0x7f54040cbef0 2023-10-16T10:03:41.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 -- 172.21.15.158:0/4272321208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040cbef0 msgr2=0x7f54040cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 --2- 172.21.15.158:0/4272321208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040cbef0 0x7f54040cc380 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009e50 tx=0x7f53fc00b330 comp rx=0 tx=0).stop 2023-10-16T10:03:41.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 -- 172.21.15.158:0/4272321208 shutdown_connections 2023-10-16T10:03:41.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 --2- 172.21.15.158:0/4272321208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040cbef0 0x7f54040cc380 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 --2- 172.21.15.158:0/4272321208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5404159280 0x7f540406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 -- 172.21.15.158:0/4272321208 >> 172.21.15.158:0/4272321208 conn(0x7f5404099a10 msgr2=0x7f54040d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:41.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 -- 172.21.15.158:0/4272321208 shutdown_connections 2023-10-16T10:03:41.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.079+0000 7f540900c700 1 -- 172.21.15.158:0/4272321208 wait complete. 2023-10-16T10:03:41.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.080+0000 7f540900c700 1 Processor -- start 2023-10-16T10:03:41.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.080+0000 7f540900c700 1 -- start start 2023-10-16T10:03:41.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540900c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54040cbef0 0x7f54040c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540900c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040d0e10 0x7f54040c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540900c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5404159280 0x7f54040c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540900c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540415a900 con 0x7f54040d0e10 2023-10-16T10:03:41.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540900c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54040c67a0 con 0x7f5404159280 2023-10-16T10:03:41.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540900c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54040c6a80 con 0x7f54040cbef0 2023-10-16T10:03:41.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040d0e10 0x7f54040c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040d0e10 0x7f54040c5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58788/0 (socket says 172.21.15.158:58788) 2023-10-16T10:03:41.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540259c700 1 -- 172.21.15.158:0/1293378536 learned_addr learned my addr 172.21.15.158:0/1293378536 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:41.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f5402d9d700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54040cbef0 0x7f54040c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.081+0000 7f540359e700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5404159280 0x7f54040c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540259c700 1 -- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54040cbef0 msgr2=0x7f54040c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540259c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54040cbef0 0x7f54040c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540259c700 1 -- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5404159280 msgr2=0x7f54040c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540259c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5404159280 0x7f54040c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540259c700 1 -- 172.21.15.158:0/1293378536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc009b30 con 0x7f54040d0e10 2023-10-16T10:03:41.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540259c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040d0e10 0x7f54040c5d90 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c450 tx=0x7f53f400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f53fb7fe700 1 -- 172.21.15.158:0/1293378536 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40052d0 con 0x7f54040d0e10 2023-10-16T10:03:41.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f53fb7fe700 1 -- 172.21.15.158:0/1293378536 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f4005430 con 0x7f54040d0e10 2023-10-16T10:03:41.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f53fb7fe700 1 -- 172.21.15.158:0/1293378536 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40038c0 con 0x7f54040d0e10 2023-10-16T10:03:41.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540415c590 con 0x7f54040d0e10 2023-10-16T10:03:41.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.082+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540415cb50 con 0x7f54040d0e10 2023-10-16T10:03:41.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.083+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54040681b0 con 0x7f54040d0e10 2023-10-16T10:03:41.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.083+0000 7f53fb7fe700 1 -- 172.21.15.158:0/1293378536 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f53f4003a20 con 0x7f54040d0e10 2023-10-16T10:03:41.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.084+0000 7f53fb7fe700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f53e406b070 0x7f53e406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.084+0000 7f53fb7fe700 1 -- 172.21.15.158:0/1293378536 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f53f4088f70 con 0x7f54040d0e10 2023-10-16T10:03:41.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.084+0000 7f5402d9d700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f53e406b070 0x7f53e406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.084+0000 7f5402d9d700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f53e406b070 0x7f53e406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f53ec006fd0 tx=0x7f53ec008040 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.088+0000 7f53fb7fe700 1 -- 172.21.15.158:0/1293378536 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53f40bbfe0 con 0x7f54040d0e10 2023-10-16T10:03:41.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.306+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7f540415bd90 con 0x7f53e406b070 2023-10-16T10:03:41.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.368+0000 7f53fb7fe700 1 -- 172.21.15.158:0/1293378536 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f53ec00c050 con 0x7f53e406b070 2023-10-16T10:03:41.370 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1/yearly 2023-10-16T10:03:41.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.370+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f53e406b070 msgr2=0x7f53e406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.370+0000 7f540900c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f53e406b070 0x7f53e406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f53ec006fd0 tx=0x7f53ec008040 comp rx=0 tx=0).stop 2023-10-16T10:03:41.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.370+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040d0e10 msgr2=0x7f54040c5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.370+0000 7f540900c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040d0e10 0x7f54040c5d90 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c450 tx=0x7f53f400c810 comp rx=0 tx=0).stop 2023-10-16T10:03:41.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.370+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 shutdown_connections 2023-10-16T10:03:41.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.371+0000 7f540900c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f53e406b070 0x7f53e406d530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.371+0000 7f540900c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5404159280 0x7f54040c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.371+0000 7f540900c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54040cbef0 0x7f54040c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.371+0000 7f540900c700 1 --2- 172.21.15.158:0/1293378536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54040d0e10 0x7f54040c5d90 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.371+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 >> 172.21.15.158:0/1293378536 conn(0x7f5404099a10 msgr2=0x7f5404159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:41.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.371+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 shutdown_connections 2023-10-16T10:03:41.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.371+0000 7f540900c700 1 -- 172.21.15.158:0/1293378536 wait complete. 2023-10-16T10:03:41.384 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2023-10-16T10:03:41.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:41.385 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2023-10-16T10:03:41.403 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2023-10-16T10:03:41.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.566+0000 7f99210c7700 1 Processor -- start 2023-10-16T10:03:41.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.566+0000 7f99210c7700 1 -- start start 2023-10-16T10:03:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f99210c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 0x7f991c06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f99210c7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c15e3e0 con 0x7f991c0b38f0 2023-10-16T10:03:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f99210c7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c15e6c0 con 0x7f991c15d840 2023-10-16T10:03:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f99210c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c15ec80 con 0x7f991c06bbf0 2023-10-16T10:03:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f991a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f991c0b38f0 0x7f991c0d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58972/0 (socket says 172.21.15.158:58972) 2023-10-16T10:03:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f991a59c700 1 -- 172.21.15.158:0/899628349 learned_addr learned my addr 172.21.15.158:0/899628349 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920042226 0 0) 0x7f991c15e3e0 con 0x7f991c0b38f0 2023-10-16T10:03:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99080035b0 con 0x7f991c0b38f0 2023-10-16T10:03:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f991b59e700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 0x7f991c06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 164889045 0 0) 0x7f99080035b0 con 0x7f991c0b38f0 2023-10-16T10:03:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 msgr2=0x7f991c06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f9919d9b700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 0x7f991c06c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 >> v1:172.21.15.172:6789/0 conn(0x7f991c15d840 legacy=0x7f991c15dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.567+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c15e9a0 con 0x7f991c0b38f0 2023-10-16T10:03:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f9904001670 con 0x7f991c0b38f0 2023-10-16T10:03:41.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 >> v1:172.21.15.158:6789/0 conn(0x7f991c0b38f0 legacy=0x7f991c0d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 0x7f9908003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f9908004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9908004d90 0x7f9908005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c15ec80 con 0x7f9908003fb0 2023-10-16T10:03:41.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990800baf0 con 0x7f9908004d90 2023-10-16T10:03:41.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990800bcb0 con 0x7f991c06bbf0 2023-10-16T10:03:41.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991a59c700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f9908004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991b59e700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9908004d90 0x7f9908005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991ad9d700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 0x7f9908003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991a59c700 1 -- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 msgr2=0x7f9908003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:41.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991a59c700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c06bbf0 0x7f9908003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991a59c700 1 -- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9908004d90 msgr2=0x7f9908005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:41.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991a59c700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9908004d90 0x7f9908005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991a59c700 1 -- 172.21.15.158:0/899628349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080035b0 con 0x7f9908003fb0 2023-10-16T10:03:41.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f991a59c700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f9908004850 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f990400c240 tx=0x7f9908005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6779e4fac96a124b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904007c10 con 0x7f9908003fb0 2023-10-16T10:03:41.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990400cde0 con 0x7f9908003fb0 2023-10-16T10:03:41.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.568+0000 7f9919d9b700 1 -- 172.21.15.158:0/899628349 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904005ba0 con 0x7f9908003fb0 2023-10-16T10:03:41.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.569+0000 7f99210c7700 1 -- 172.21.15.158:0/899628349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 msgr2=0x7f9908004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.569+0000 7f99210c7700 1 --2- 172.21.15.158:0/899628349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f9908004850 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f990400c240 tx=0x7f9908005880 comp rx=0 tx=0).stop 2023-10-16T10:03:41.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.569+0000 7f991b59e700 1 -- 172.21.15.158:0/899628349 reap_dead start 2023-10-16T10:03:41.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.569+0000 7f99210c7700 1 -- 172.21.15.158:0/899628349 shutdown_connections 2023-10-16T10:03:41.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.569+0000 7f99210c7700 1 -- 172.21.15.158:0/899628349 >> 172.21.15.158:0/899628349 conn(0x7f991c06ec00 msgr2=0x7f991c06efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:41.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.569+0000 7f99210c7700 1 -- 172.21.15.158:0/899628349 shutdown_connections 2023-10-16T10:03:41.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.569+0000 7f99210c7700 1 -- 172.21.15.158:0/899628349 wait complete. 2023-10-16T10:03:41.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.570+0000 7f99210c7700 1 Processor -- start 2023-10-16T10:03:41.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.570+0000 7f99210c7700 1 -- start start 2023-10-16T10:03:41.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f99210c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f991c0c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f99210c7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f991c158fe0 0x7f991c0c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f99210c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c0c8070 0x7f991c15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f99210c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c6880 con 0x7f9908003fb0 2023-10-16T10:03:41.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f99210c7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c52a0 con 0x7f991c158fe0 2023-10-16T10:03:41.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f99210c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c5580 con 0x7f991c0c8070 2023-10-16T10:03:41.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f991ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f991c0c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f991ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f991c0c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58806/0 (socket says 172.21.15.158:58806) 2023-10-16T10:03:41.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f991ad9d700 1 -- 172.21.15.158:0/408606560 learned_addr learned my addr 172.21.15.158:0/408606560 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:41.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f991b59e700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c0c8070 0x7f991c15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.571+0000 7f991a59c700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f991c158fe0 0x7f991c0c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f991ad9d700 1 -- 172.21.15.158:0/408606560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c0c8070 msgr2=0x7f991c15f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f991ad9d700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f991c0c8070 0x7f991c15f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f991ad9d700 1 -- 172.21.15.158:0/408606560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f991c158fe0 msgr2=0x7f991c0c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f991ad9d700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f991c158fe0 0x7f991c0c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f991ad9d700 1 -- 172.21.15.158:0/408606560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c0c5860 con 0x7f9908003fb0 2023-10-16T10:03:41.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f991a59c700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f991c158fe0 0x7f991c0c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:41.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f991ad9d700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9908003fb0 0x7f991c0c6340 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f991400f820 tx=0x7f991400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f99137fe700 1 -- 172.21.15.158:0/408606560 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140052e0 con 0x7f9908003fb0 2023-10-16T10:03:41.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f99137fe700 1 -- 172.21.15.158:0/408606560 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9914005440 con 0x7f9908003fb0 2023-10-16T10:03:41.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f99210c7700 1 -- 172.21.15.158:0/408606560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c0c5b40 con 0x7f9908003fb0 2023-10-16T10:03:41.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f99137fe700 1 -- 172.21.15.158:0/408606560 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140038c0 con 0x7f9908003fb0 2023-10-16T10:03:41.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.572+0000 7f99210c7700 1 -- 172.21.15.158:0/408606560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c0c4f70 con 0x7f9908003fb0 2023-10-16T10:03:41.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.573+0000 7f99210c7700 1 -- 172.21.15.158:0/408606560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c0c6060 con 0x7f9908003fb0 2023-10-16T10:03:41.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.573+0000 7f99137fe700 1 -- 172.21.15.158:0/408606560 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9914003a20 con 0x7f9908003fb0 2023-10-16T10:03:41.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.574+0000 7f99137fe700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f98fc06b0a0 0x7f98fc06d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.574+0000 7f99137fe700 1 -- 172.21.15.158:0/408606560 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f9914089a00 con 0x7f9908003fb0 2023-10-16T10:03:41.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.574+0000 7f991a59c700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f98fc06b0a0 0x7f98fc06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.574+0000 7f991a59c700 1 --2- 172.21.15.158:0/408606560 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f98fc06b0a0 0x7f98fc06d560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9904000c00 tx=0x7f9904007000 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.578+0000 7f99137fe700 1 -- 172.21.15.158:0/408606560 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f991c0c6060 con 0x7f9908003fb0 2023-10-16T10:03:41.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.794+0000 7f99210c7700 1 -- 172.21.15.158:0/408606560 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f991c0b3da0 con 0x7f98fc06b0a0 2023-10-16T10:03:41.796 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:41.795+0000 7f64524f2700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T10:03:41.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.795+0000 7f99137fe700 1 -- 172.21.15.158:0/408606560 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f9904007b30 con 0x7f98fc06b0a0 2023-10-16T10:03:41.798 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T10:03:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T10:03:41.803 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2023-10-16T10:03:41.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.929+0000 7f296de9b700 1 Processor -- start 2023-10-16T10:03:41.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.929+0000 7f296de9b700 1 -- start start 2023-10-16T10:03:41.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f296de9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 0x7f29680cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f296de9b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29680d0ae0 con 0x7f29680ccc00 2023-10-16T10:03:41.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f296de9b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29680d12d0 con 0x7f296815deb0 2023-10-16T10:03:41.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f296de9b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29680cbdf0 con 0x7f296815dad0 2023-10-16T10:03:41.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f2967fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f29680ccc00 0x7f29680cd0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:58992/0 (socket says 172.21.15.158:58992) 2023-10-16T10:03:41.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f2967fff700 1 -- 172.21.15.158:0/3861313632 learned_addr learned my addr 172.21.15.158:0/3861313632 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:41.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427187002 0 0) 0x7f29680d0ae0 con 0x7f29680ccc00 2023-10-16T10:03:41.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29500035b0 con 0x7f29680ccc00 2023-10-16T10:03:41.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.930+0000 7f29677fe700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 0x7f29680cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228504861 0 0) 0x7f29500035b0 con 0x7f29680ccc00 2023-10-16T10:03:41.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 msgr2=0x7f29680cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:41.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 0x7f29680cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 >> v1:172.21.15.172:6789/0 conn(0x7f296815deb0 legacy=0x7f29680ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29680cc280 con 0x7f29680ccc00 2023-10-16T10:03:41.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f29600019f0 con 0x7f29680ccc00 2023-10-16T10:03:41.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 >> v1:172.21.15.158:6789/0 conn(0x7f29680ccc00 legacy=0x7f29680cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2950003a30 0x7f2950003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 0x7f2950004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950004d90 0x7f2950005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29680cbdf0 con 0x7f2950003a30 2023-10-16T10:03:41.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295000bb40 con 0x7f2950004d90 2023-10-16T10:03:41.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295000bd00 con 0x7f296815dad0 2023-10-16T10:03:41.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29677fe700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2950003a30 0x7f2950003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f2966ffd700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 0x7f2950004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f2967fff700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950004d90 0x7f2950005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29677fe700 1 -- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 msgr2=0x7f2950004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29677fe700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f296815dad0 0x7f2950004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29677fe700 1 -- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950004d90 msgr2=0x7f2950005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29677fe700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950004d90 0x7f2950005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29677fe700 1 -- 172.21.15.158:0/3861313632 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29500035b0 con 0x7f2950003a30 2023-10-16T10:03:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f2967fff700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950004d90 0x7f2950005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29677fe700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2950003a30 0x7f2950003e50 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f295c009b60 tx=0x7f2950005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a1737a66c67039 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c00e6c0 con 0x7f2950003a30 2023-10-16T10:03:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c00e820 con 0x7f2950003a30 2023-10-16T10:03:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.931+0000 7f29667fc700 1 -- 172.21.15.158:0/3861313632 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c00eb90 con 0x7f2950003a30 2023-10-16T10:03:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.932+0000 7f296de9b700 1 -- 172.21.15.158:0/3861313632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2950003a30 msgr2=0x7f2950003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.932+0000 7f296de9b700 1 --2- 172.21.15.158:0/3861313632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2950003a30 0x7f2950003e50 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f295c009b60 tx=0x7f2950005880 comp rx=0 tx=0).stop 2023-10-16T10:03:41.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.932+0000 7f2967fff700 1 -- 172.21.15.158:0/3861313632 reap_dead start 2023-10-16T10:03:41.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.932+0000 7f296de9b700 1 -- 172.21.15.158:0/3861313632 shutdown_connections 2023-10-16T10:03:41.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.932+0000 7f296de9b700 1 -- 172.21.15.158:0/3861313632 >> 172.21.15.158:0/3861313632 conn(0x7f2968099a10 msgr2=0x7f29680d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:41.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.932+0000 7f296de9b700 1 -- 172.21.15.158:0/3861313632 shutdown_connections 2023-10-16T10:03:41.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.932+0000 7f296de9b700 1 -- 172.21.15.158:0/3861313632 wait complete. 2023-10-16T10:03:41.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.933+0000 7f296de9b700 1 Processor -- start 2023-10-16T10:03:41.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.933+0000 7f296de9b700 1 -- start start 2023-10-16T10:03:41.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f296de9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950003a30 0x7f296806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f296de9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29680b5650 0x7f29680b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f296de9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29680c5810 0x7f29680c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f296de9b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29680c61d0 con 0x7f29680c5810 2023-10-16T10:03:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f296de9b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29681f1380 con 0x7f2950003a30 2023-10-16T10:03:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f296de9b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29681f1660 con 0x7f29680b5650 2023-10-16T10:03:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f2967fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29680c5810 0x7f29680c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f2967fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29680c5810 0x7f29680c5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58826/0 (socket says 172.21.15.158:58826) 2023-10-16T10:03:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f2967fff700 1 -- 172.21.15.158:0/3584910711 learned_addr learned my addr 172.21.15.158:0/3584910711 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f2966ffd700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29680b5650 0x7f29680b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.934+0000 7f29677fe700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950003a30 0x7f296806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f2967fff700 1 -- 172.21.15.158:0/3584910711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29680b5650 msgr2=0x7f29680b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f2967fff700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29680b5650 0x7f29680b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f2967fff700 1 -- 172.21.15.158:0/3584910711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950003a30 msgr2=0x7f296806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:41.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f2967fff700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2950003a30 0x7f296806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:41.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f2967fff700 1 -- 172.21.15.158:0/3584910711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29681f1940 con 0x7f29680c5810 2023-10-16T10:03:41.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f2967fff700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29680c5810 0x7f29680c5c90 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f2950003a00 tx=0x7f296000e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f294ffff700 1 -- 172.21.15.158:0/3584910711 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296000ed80 con 0x7f29680c5810 2023-10-16T10:03:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f294ffff700 1 -- 172.21.15.158:0/3584910711 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2960013420 con 0x7f29680c5810 2023-10-16T10:03:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f296de9b700 1 -- 172.21.15.158:0/3584910711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29681f1c20 con 0x7f29680c5810 2023-10-16T10:03:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f294ffff700 1 -- 172.21.15.158:0/3584910711 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296000eee0 con 0x7f29680c5810 2023-10-16T10:03:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.935+0000 7f296de9b700 1 -- 172.21.15.158:0/3584910711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29680b4e70 con 0x7f29680c5810 2023-10-16T10:03:41.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.936+0000 7f296de9b700 1 -- 172.21.15.158:0/3584910711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29680681b0 con 0x7f29680c5810 2023-10-16T10:03:41.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.936+0000 7f294ffff700 1 -- 172.21.15.158:0/3584910711 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f29600137d0 con 0x7f29680c5810 2023-10-16T10:03:41.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.937+0000 7f294ffff700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f295406b070 0x7f295406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:41.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.937+0000 7f29677fe700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f295406b070 0x7f295406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:41.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.937+0000 7f294ffff700 1 -- 172.21.15.158:0/3584910711 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f296008a2b0 con 0x7f29680c5810 2023-10-16T10:03:41.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.937+0000 7f29677fe700 1 --2- 172.21.15.158:0/3584910711 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f295406b070 0x7f295406d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f295c0090b0 tx=0x7f295c008030 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:41.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:41.941+0000 7f294ffff700 1 -- 172.21.15.158:0/3584910711 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2960054df0 con 0x7f29680c5810 2023-10-16T10:03:42.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.163+0000 7f296de9b700 1 -- 172.21.15.158:0/3584910711 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f29681f17d0 con 0x7f295406b070 2023-10-16T10:03:42.165 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:42.164+0000 7f64524f2700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:42.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.164+0000 7f294ffff700 1 -- 172.21.15.158:0/3584910711 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f29681f17d0 con 0x7f295406b070 2023-10-16T10:03:42.167 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T10:03:42.171 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2023-10-16T10:03:42.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.294+0000 7ff2303f6700 1 Processor -- start 2023-10-16T10:03:42.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.294+0000 7ff2303f6700 1 -- start start 2023-10-16T10:03:42.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff2303f6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 0x7ff2280ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff2303f6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280cf4c0 con 0x7ff2280cbaf0 2023-10-16T10:03:42.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff2303f6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22815f970 con 0x7ff22815f1d0 2023-10-16T10:03:42.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff2303f6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280b49e0 con 0x7ff22815b590 2023-10-16T10:03:42.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22e993700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff2280cbaf0 0x7ff2280cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59010/0 (socket says 172.21.15.158:59010) 2023-10-16T10:03:42.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22e993700 1 -- 172.21.15.158:0/1748837536 learned_addr learned my addr 172.21.15.158:0/1748837536 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:42.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22e192700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 0x7ff2280ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995060664 0 0) 0x7ff2280cf4c0 con 0x7ff2280cbaf0 2023-10-16T10:03:42.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2140035b0 con 0x7ff2280cbaf0 2023-10-16T10:03:42.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047609623 0 0) 0x7ff22815f970 con 0x7ff22815f1d0 2023-10-16T10:03:42.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2280cf4c0 con 0x7ff22815f1d0 2023-10-16T10:03:42.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521291764 0 0) 0x7ff2140035b0 con 0x7ff2280cbaf0 2023-10-16T10:03:42.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 msgr2=0x7ff2280ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:42.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 0x7ff2280ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 >> v1:172.21.15.172:6789/0 conn(0x7ff22815f1d0 legacy=0x7ff2280b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22815fc50 con 0x7ff2280cbaf0 2023-10-16T10:03:42.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff2200019f0 con 0x7ff2280cbaf0 2023-10-16T10:03:42.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 >> v1:172.21.15.158:6789/0 conn(0x7ff2280cbaf0 legacy=0x7ff2280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 0x7ff214003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff214004170 0x7ff214004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff214005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280b49e0 con 0x7ff214004d90 2023-10-16T10:03:42.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21400bbd0 con 0x7ff214004170 2023-10-16T10:03:42.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.295+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21400bd90 con 0x7ff22815b590 2023-10-16T10:03:42.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e993700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff214005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e192700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 0x7ff214003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22d991700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff214004170 0x7ff214004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e993700 1 -- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 msgr2=0x7ff214003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:42.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e993700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 0x7ff214003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e993700 1 -- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff214004170 msgr2=0x7ff214004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:42.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e993700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff214004170 0x7ff214004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e993700 1 -- 172.21.15.158:0/1748837536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140035b0 con 0x7ff214004d90 2023-10-16T10:03:42.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22d991700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff214004170 0x7ff214004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:42.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e192700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22815b590 0x7ff214003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:42.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22e993700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff214005210 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7ff22000a710 tx=0x7ff214005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2b269bad3c123a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:42.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2200072d0 con 0x7ff214004d90 2023-10-16T10:03:42.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff220007450 con 0x7ff214004d90 2023-10-16T10:03:42.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.296+0000 7ff22d190700 1 -- 172.21.15.158:0/1748837536 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220007800 con 0x7ff214004d90 2023-10-16T10:03:42.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.297+0000 7ff2303f6700 1 -- 172.21.15.158:0/1748837536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 msgr2=0x7ff214005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.297+0000 7ff2303f6700 1 --2- 172.21.15.158:0/1748837536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff214005210 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7ff22000a710 tx=0x7ff214005880 comp rx=0 tx=0).stop 2023-10-16T10:03:42.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.297+0000 7ff22e993700 1 -- 172.21.15.158:0/1748837536 reap_dead start 2023-10-16T10:03:42.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.297+0000 7ff2303f6700 1 -- 172.21.15.158:0/1748837536 shutdown_connections 2023-10-16T10:03:42.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.297+0000 7ff2303f6700 1 -- 172.21.15.158:0/1748837536 >> 172.21.15.158:0/1748837536 conn(0x7ff22806a8d0 msgr2=0x7ff22806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:42.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.297+0000 7ff2303f6700 1 -- 172.21.15.158:0/1748837536 shutdown_connections 2023-10-16T10:03:42.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.297+0000 7ff2303f6700 1 -- 172.21.15.158:0/1748837536 wait complete. 2023-10-16T10:03:42.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.298+0000 7ff2303f6700 1 Processor -- start 2023-10-16T10:03:42.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.298+0000 7ff2303f6700 1 -- start start 2023-10-16T10:03:42.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff2280c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22806d780 0x7ff2281f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2281f49b0 0x7ff22815e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22815deb0 con 0x7ff214004d90 2023-10-16T10:03:42.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22815d000 con 0x7ff2281f49b0 2023-10-16T10:03:42.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280c8340 con 0x7ff22806d780 2023-10-16T10:03:42.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff2280c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff2280c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58842/0 (socket says 172.21.15.158:58842) 2023-10-16T10:03:42.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 -- 172.21.15.158:0/740532627 learned_addr learned my addr 172.21.15.158:0/740532627 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:42.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22d991700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22806d780 0x7ff2281f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 -- 172.21.15.158:0/740532627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22806d780 msgr2=0x7ff2281f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e993700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2281f49b0 0x7ff22815e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22806d780 0x7ff2281f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 -- 172.21.15.158:0/740532627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2281f49b0 msgr2=0x7ff22815e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2281f49b0 0x7ff22815e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 -- 172.21.15.158:0/740532627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22815d230 con 0x7ff214004d90 2023-10-16T10:03:42.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff22e192700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff214004d90 0x7ff2280c7e00 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7ff2240118d0 tx=0x7ff224011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:42.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff21effd700 1 -- 172.21.15.158:0/740532627 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224004220 con 0x7ff214004d90 2023-10-16T10:03:42.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff21effd700 1 -- 172.21.15.158:0/740532627 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff224004380 con 0x7ff214004d90 2023-10-16T10:03:42.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 -- 172.21.15.158:0/740532627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22815d510 con 0x7ff214004d90 2023-10-16T10:03:42.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff21effd700 1 -- 172.21.15.158:0/740532627 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2240036a0 con 0x7ff214004d90 2023-10-16T10:03:42.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.299+0000 7ff2303f6700 1 -- 172.21.15.158:0/740532627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22815dad0 con 0x7ff214004d90 2023-10-16T10:03:42.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.300+0000 7ff2303f6700 1 -- 172.21.15.158:0/740532627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22815e120 con 0x7ff214004d90 2023-10-16T10:03:42.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.301+0000 7ff21effd700 1 -- 172.21.15.158:0/740532627 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7ff224003800 con 0x7ff214004d90 2023-10-16T10:03:42.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.302+0000 7ff21effd700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff20c06b070 0x7ff20c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.302+0000 7ff21effd700 1 -- 172.21.15.158:0/740532627 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7ff22408a3c0 con 0x7ff214004d90 2023-10-16T10:03:42.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.302+0000 7ff22d991700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff20c06b070 0x7ff20c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.302+0000 7ff22d991700 1 --2- 172.21.15.158:0/740532627 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff20c06b070 0x7ff20c06d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff2280cfa70 tx=0x7ff218009040 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:42.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.305+0000 7ff21effd700 1 -- 172.21.15.158:0/740532627 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff224055070 con 0x7ff214004d90 2023-10-16T10:03:42.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.524+0000 7ff2303f6700 1 -- 172.21.15.158:0/740532627 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff22815f650 con 0x7ff20c06b070 2023-10-16T10:03:42.526 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:42.525+0000 7f64524f2700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:42.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.525+0000 7ff21effd700 1 -- 172.21.15.158:0/740532627 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff21800d050 con 0x7ff20c06b070 2023-10-16T10:03:42.528 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T10:03:42.532 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2023-10-16T10:03:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.654+0000 7f9f613be700 1 Processor -- start 2023-10-16T10:03:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.655+0000 7f9f613be700 1 -- start start 2023-10-16T10:03:42.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.655+0000 7f9f613be700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f5c0cc590 0x7f9f5c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.655+0000 7f9f613be700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c0cbd30 con 0x7f9f5c06afe0 2023-10-16T10:03:42.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.655+0000 7f9f613be700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c0cc1c0 con 0x7f9f5c15da40 2023-10-16T10:03:42.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.655+0000 7f9f613be700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5c15de20 con 0x7f9f5c0cc590 2023-10-16T10:03:42.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.655+0000 7f9f5a7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9f5c06afe0 0x7f9f5c0afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59026/0 (socket says 172.21.15.158:59026) 2023-10-16T10:03:42.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.655+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/1367625377 learned_addr learned my addr 172.21.15.158:0/1367625377 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:42.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5b7fe700 1 --2- 172.21.15.158:0/1367625377 >> v2:172.21.15.172:3300/0 conn(0x7f9f5c0cc590 0x7f9f5c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643813381 0 0) 0x7f9f5c0cbd30 con 0x7f9f5c06afe0 2023-10-16T10:03:42.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f440035b0 con 0x7f9f5c06afe0 2023-10-16T10:03:42.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44173457 0 0) 0x7f9f440035b0 con 0x7f9f5c06afe0 2023-10-16T10:03:42.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 >> v2:172.21.15.172:3300/0 conn(0x7f9f5c0cc590 msgr2=0x7f9f5c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:42.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 --2- 172.21.15.158:0/1367625377 >> v2:172.21.15.172:3300/0 conn(0x7f9f5c0cc590 0x7f9f5c0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 >> v1:172.21.15.172:6789/0 conn(0x7f9f5c15da40 legacy=0x7f9f5c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c0ce260 con 0x7f9f5c06afe0 2023-10-16T10:03:42.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f9f4c001670 con 0x7f9f5c06afe0 2023-10-16T10:03:42.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5b7fe700 1 --2- 172.21.15.158:0/1367625377 >> v2:172.21.15.172:3300/0 conn(0x7f9f5c0cc590 0x7f9f5c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:42.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 >> v1:172.21.15.158:6789/0 conn(0x7f9f5c06afe0 legacy=0x7f9f5c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f44003a00 0x7f9f44003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f44004400 0x7f9f440048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f440050d0 0x7f9f4400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c15de20 con 0x7f9f44004400 2023-10-16T10:03:42.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4400c030 con 0x7f9f44003a00 2023-10-16T10:03:42.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4400c1f0 con 0x7f9f440050d0 2023-10-16T10:03:42.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5a7fc700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f44004400 0x7f9f440048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5affd700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f44003a00 0x7f9f44003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5b7fe700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f440050d0 0x7f9f4400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f440050d0 msgr2=0x7f9f4400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:42.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5a7fc700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f440050d0 0x7f9f4400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f44003a00 msgr2=0x7f9f44003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:42.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5a7fc700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f44003a00 0x7f9f44003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/1367625377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f440035b0 con 0x7f9f44004400 2023-10-16T10:03:42.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.656+0000 7f9f5b7fe700 1 -- 172.21.15.158:0/1367625377 reap_dead start 2023-10-16T10:03:42.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f5a7fc700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f44004400 0x7f9f440048d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c2f0 tx=0x7f9f44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a884578f6f127807 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:42.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c00cd70 con 0x7f9f44004400 2023-10-16T10:03:42.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c005960 con 0x7f9f44004400 2023-10-16T10:03:42.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f59ffb700 1 -- 172.21.15.158:0/1367625377 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c005d10 con 0x7f9f44004400 2023-10-16T10:03:42.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f613be700 1 -- 172.21.15.158:0/1367625377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f44004400 msgr2=0x7f9f440048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f613be700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f44004400 0x7f9f440048d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c2f0 tx=0x7f9f44005880 comp rx=0 tx=0).stop 2023-10-16T10:03:42.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f613be700 1 -- 172.21.15.158:0/1367625377 shutdown_connections 2023-10-16T10:03:42.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f613be700 1 --2- 172.21.15.158:0/1367625377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f44004400 0x7f9f440048d0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f613be700 1 -- 172.21.15.158:0/1367625377 >> 172.21.15.158:0/1367625377 conn(0x7f9f5c099a10 msgr2=0x7f9f5c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:42.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f613be700 1 -- 172.21.15.158:0/1367625377 shutdown_connections 2023-10-16T10:03:42.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.657+0000 7f9f613be700 1 -- 172.21.15.158:0/1367625377 wait complete. 2023-10-16T10:03:42.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 Processor -- start 2023-10-16T10:03:42.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 -- start start 2023-10-16T10:03:42.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f44004400 0x7f9f5c0b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c1607d0 0x7f9f5c1fb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f5c15aba0 0x7f9f5c15f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c06f940 con 0x7f9f5c1607d0 2023-10-16T10:03:42.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c1fb590 con 0x7f9f5c15aba0 2023-10-16T10:03:42.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.659+0000 7f9f613be700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c0b01a0 con 0x7f9f44004400 2023-10-16T10:03:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c1607d0 0x7f9f5c1fb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c1607d0 0x7f9f5c1fb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58864/0 (socket says 172.21.15.158:58864) 2023-10-16T10:03:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/2783372789 learned_addr learned my addr 172.21.15.158:0/2783372789 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5b7fe700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f5c15aba0 0x7f9f5c15f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5affd700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f44004400 0x7f9f5c0b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/2783372789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f44004400 msgr2=0x7f9f5c0b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f44004400 0x7f9f5c0b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/2783372789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f5c15aba0 msgr2=0x7f9f5c15f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:42.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f5c15aba0 0x7f9f5c15f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:42.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 -- 172.21.15.158:0/2783372789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c0af590 con 0x7f9f5c1607d0 2023-10-16T10:03:42.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5affd700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f44004400 0x7f9f5c0b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:42.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f5a7fc700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c1607d0 0x7f9f5c1fb050 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c005d80 tx=0x7f9f4c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:42.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c004a80 con 0x7f9f5c1607d0 2023-10-16T10:03:42.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c004be0 con 0x7f9f5c1607d0 2023-10-16T10:03:42.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f613be700 1 -- 172.21.15.158:0/2783372789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c0af870 con 0x7f9f5c1607d0 2023-10-16T10:03:42.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c0070d0 con 0x7f9f5c1607d0 2023-10-16T10:03:42.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.660+0000 7f9f613be700 1 -- 172.21.15.158:0/2783372789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c0afe30 con 0x7f9f5c1607d0 2023-10-16T10:03:42.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.661+0000 7f9f613be700 1 -- 172.21.15.158:0/2783372789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c0681b0 con 0x7f9f5c1607d0 2023-10-16T10:03:42.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.662+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9f4c007230 con 0x7f9f5c1607d0 2023-10-16T10:03:42.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.662+0000 7f9f437fe700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f9f4806b070 0x7f9f4806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:42.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.662+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f9f4c08a570 con 0x7f9f5c1607d0 2023-10-16T10:03:42.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.662+0000 7f9f5affd700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f9f4806b070 0x7f9f4806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:42.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.662+0000 7f9f5affd700 1 --2- 172.21.15.158:0/2783372789 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f9f4806b070 0x7f9f4806d530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9f50007fd0 tx=0x7f9f50009040 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:42.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.666+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f4c055200 con 0x7f9f5c1607d0 2023-10-16T10:03:42.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.860+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f9f4c0555d0 con 0x7f9f5c1607d0 2023-10-16T10:03:42.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.890+0000 7f9f613be700 1 -- 172.21.15.158:0/2783372789 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f5c0cb400 con 0x7f9f4806b070 2023-10-16T10:03:42.892 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:42.891+0000 7f64524f2700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:42.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:42.891+0000 7f9f437fe700 1 -- 172.21.15.158:0/2783372789 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9f50005320 con 0x7f9f4806b070 2023-10-16T10:03:42.894 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T10:03:42.898 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2023-10-16T10:03:43.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.021+0000 7fdaa05b5700 1 Processor -- start 2023-10-16T10:03:43.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.021+0000 7fdaa05b5700 1 -- start start 2023-10-16T10:03:43.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fdaa05b5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 0x7fda980ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fdaa05b5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda980cbaf0 0x7fda980cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fdaa05b5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda980cf4c0 con 0x7fda9815f1d0 2023-10-16T10:03:43.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fdaa05b5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda980cc4c0 con 0x7fda980cbaf0 2023-10-16T10:03:43.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fdaa05b5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980b49e0 con 0x7fda9815b590 2023-10-16T10:03:43.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9db50700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fda9815f1d0 0x7fda980b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59046/0 (socket says 172.21.15.158:59046) 2023-10-16T10:03:43.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9db50700 1 -- 172.21.15.158:0/777595042 learned_addr learned my addr 172.21.15.158:0/777595042 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:43.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605639252 0 0) 0x7fda980cf4c0 con 0x7fda9815f1d0 2023-10-16T10:03:43.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9e351700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 0x7fda980ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda840035b0 con 0x7fda9815f1d0 2023-10-16T10:03:43.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9eb52700 1 --2- 172.21.15.158:0/777595042 >> v2:172.21.15.172:3300/0 conn(0x7fda980cbaf0 0x7fda980cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335006281 0 0) 0x7fda840035b0 con 0x7fda9815f1d0 2023-10-16T10:03:43.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 msgr2=0x7fda980ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:43.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 0x7fda980ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 >> v2:172.21.15.172:3300/0 conn(0x7fda980cbaf0 msgr2=0x7fda980cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:43.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 --2- 172.21.15.158:0/777595042 >> v2:172.21.15.172:3300/0 conn(0x7fda980cbaf0 0x7fda980cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9815f970 con 0x7fda9815f1d0 2023-10-16T10:03:43.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fda88001670 con 0x7fda9815f1d0 2023-10-16T10:03:43.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9eb52700 1 --2- 172.21.15.158:0/777595042 >> v2:172.21.15.172:3300/0 conn(0x7fda980cbaf0 0x7fda980cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:43.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 >> v1:172.21.15.158:6789/0 conn(0x7fda9815f1d0 legacy=0x7fda980b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda84003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.022+0000 7fda9d34f700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 0x7fda84004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9d34f700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda84004d90 0x7fda84005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980cc4c0 con 0x7fda84003a30 2023-10-16T10:03:43.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980b49e0 con 0x7fda84004d90 2023-10-16T10:03:43.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8400bae0 con 0x7fda9815b590 2023-10-16T10:03:43.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9e351700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda84003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9db50700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 0x7fda84004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9eb52700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda84004d90 0x7fda84005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9e351700 1 -- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 msgr2=0x7fda84004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:43.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9e351700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 0x7fda84004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9e351700 1 -- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda84004d90 msgr2=0x7fda84005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:43.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9e351700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda84004d90 0x7fda84005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9e351700 1 -- 172.21.15.158:0/777595042 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840035b0 con 0x7fda84003a30 2023-10-16T10:03:43.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9db50700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda9815b590 0x7fda84004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:43.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9e351700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda84003e50 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fda94009b60 tx=0x7fda84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d350509dae7b4f45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:43.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9400e6c0 con 0x7fda84003a30 2023-10-16T10:03:43.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9400e820 con 0x7fda84003a30 2023-10-16T10:03:43.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.023+0000 7fda9d34f700 1 -- 172.21.15.158:0/777595042 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9400eb90 con 0x7fda84003a30 2023-10-16T10:03:43.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.024+0000 7fdaa05b5700 1 -- 172.21.15.158:0/777595042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 msgr2=0x7fda84003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.024+0000 7fdaa05b5700 1 --2- 172.21.15.158:0/777595042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda84003e50 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fda94009b60 tx=0x7fda84005880 comp rx=0 tx=0).stop 2023-10-16T10:03:43.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.024+0000 7fda9eb52700 1 -- 172.21.15.158:0/777595042 reap_dead start 2023-10-16T10:03:43.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.024+0000 7fdaa05b5700 1 -- 172.21.15.158:0/777595042 shutdown_connections 2023-10-16T10:03:43.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.024+0000 7fdaa05b5700 1 -- 172.21.15.158:0/777595042 >> 172.21.15.158:0/777595042 conn(0x7fda9806a8d0 msgr2=0x7fda9806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:43.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.024+0000 7fdaa05b5700 1 -- 172.21.15.158:0/777595042 shutdown_connections 2023-10-16T10:03:43.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.024+0000 7fdaa05b5700 1 -- 172.21.15.158:0/777595042 wait complete. 2023-10-16T10:03:43.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.025+0000 7fdaa05b5700 1 Processor -- start 2023-10-16T10:03:43.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.025+0000 7fdaa05b5700 1 -- start start 2023-10-16T10:03:43.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda981f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda980c8570 0x7fda980b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda981f1760 0x7fda981f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981f0fc0 con 0x7fda84003a30 2023-10-16T10:03:43.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980c8d50 con 0x7fda980c8570 2023-10-16T10:03:43.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980c9030 con 0x7fda981f1760 2023-10-16T10:03:43.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda981f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda981f5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58880/0 (socket says 172.21.15.158:58880) 2023-10-16T10:03:43.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 -- 172.21.15.158:0/4065183060 learned_addr learned my addr 172.21.15.158:0/4065183060 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:43.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9eb52700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda981f1760 0x7fda981f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9db50700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda980c8570 0x7fda980b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 -- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda981f1760 msgr2=0x7fda981f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda981f1760 0x7fda981f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 -- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda980c8570 msgr2=0x7fda980b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda980c8570 0x7fda980b5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 -- 172.21.15.158:0/4065183060 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980c9310 con 0x7fda84003a30 2023-10-16T10:03:43.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9eb52700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda981f1760 0x7fda981f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:43.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9db50700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda980c8570 0x7fda980b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:43.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda9e351700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda84003a30 0x7fda981f5150 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fda94000c00 tx=0x7fda940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:43.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda8effd700 1 -- 172.21.15.158:0/4065183060 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9400e6c0 con 0x7fda84003a30 2023-10-16T10:03:43.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda8effd700 1 -- 172.21.15.158:0/4065183060 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda94009ed0 con 0x7fda84003a30 2023-10-16T10:03:43.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 -- 172.21.15.158:0/4065183060 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda981f1240 con 0x7fda84003a30 2023-10-16T10:03:43.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fda8effd700 1 -- 172.21.15.158:0/4065183060 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94003cf0 con 0x7fda84003a30 2023-10-16T10:03:43.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.026+0000 7fdaa05b5700 1 -- 172.21.15.158:0/4065183060 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda980b6360 con 0x7fda84003a30 2023-10-16T10:03:43.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.027+0000 7fdaa05b5700 1 -- 172.21.15.158:0/4065183060 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda980b6550 con 0x7fda84003a30 2023-10-16T10:03:43.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.028+0000 7fda8effd700 1 -- 172.21.15.158:0/4065183060 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fda94013470 con 0x7fda84003a30 2023-10-16T10:03:43.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.028+0000 7fda8effd700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fda7c06b100 0x7fda7c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.028+0000 7fda8effd700 1 -- 172.21.15.158:0/4065183060 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fda9408b9d0 con 0x7fda84003a30 2023-10-16T10:03:43.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.028+0000 7fda9db50700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fda7c06b100 0x7fda7c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.029+0000 7fda9db50700 1 --2- 172.21.15.158:0/4065183060 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fda7c06b100 0x7fda7c06d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fda88001eb0 tx=0x7fda88009040 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:43.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.033+0000 7fda8effd700 1 -- 172.21.15.158:0/4065183060 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda94056390 con 0x7fda84003a30 2023-10-16T10:03:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.249+0000 7fdaa05b5700 1 -- 172.21.15.158:0/4065183060 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fda9815ff30 con 0x7fda7c06b100 2023-10-16T10:03:43.251 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:43.250+0000 7f64524f2700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T10:03:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.250+0000 7fda8effd700 1 -- 172.21.15.158:0/4065183060 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7fda880107b0 con 0x7fda7c06b100 2023-10-16T10:03:43.253 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T10:03:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T10:03:43.257 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2023-10-16T10:03:43.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.381+0000 7f4ade25e700 1 Processor -- start 2023-10-16T10:03:43.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.382+0000 7f4ade25e700 1 -- start start 2023-10-16T10:03:43.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.382+0000 7f4ade25e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 0x7f4ad80cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.382+0000 7f4ade25e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ad815daa0 0x7f4ad80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.382+0000 7f4ade25e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad80d1ee0 con 0x7f4ad80d2370 2023-10-16T10:03:43.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.382+0000 7f4ade25e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad80d40a0 con 0x7f4ad815daa0 2023-10-16T10:03:43.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.382+0000 7f4ade25e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80d12d0 con 0x7f4ad815d6c0 2023-10-16T10:03:43.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ad815daa0 0x7f4ad80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ad815daa0 0x7f4ad80d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58890/0 (socket says 172.21.15.158:58890) 2023-10-16T10:03:43.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/944096162 learned_addr learned my addr 172.21.15.158:0/944096162 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:43.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/944096162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 msgr2=0x7f4ad80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:43.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/944096162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 0x7f4ad80cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/944096162 >> v1:172.21.15.158:6789/0 conn(0x7f4ad80d2370 legacy=0x7f4ad80cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/944096162 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80d15b0 con 0x7f4ad815daa0 2023-10-16T10:03:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/944096162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 0x7f4ad80d0470 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0098f0 tx=0x7f4acc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbfe767a0e4e759 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad67fc700 1 -- 172.21.15.158:0/944096162 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc00b8a0 con 0x7f4ad815daa0 2023-10-16T10:03:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad67fc700 1 -- 172.21.15.158:0/944096162 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc00ba00 con 0x7f4ad815daa0 2023-10-16T10:03:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.383+0000 7f4ad67fc700 1 -- 172.21.15.158:0/944096162 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc00bd90 con 0x7f4ad815daa0 2023-10-16T10:03:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 -- 172.21.15.158:0/944096162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 msgr2=0x7f4ad80d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 --2- 172.21.15.158:0/944096162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 0x7f4ad80d0470 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0098f0 tx=0x7f4acc009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 -- 172.21.15.158:0/944096162 shutdown_connections 2023-10-16T10:03:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 --2- 172.21.15.158:0/944096162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 0x7f4ad80cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 --2- 172.21.15.158:0/944096162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 0x7f4ad80d0470 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 -- 172.21.15.158:0/944096162 >> 172.21.15.158:0/944096162 conn(0x7f4ad809c940 msgr2=0x7f4ad80d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 -- 172.21.15.158:0/944096162 shutdown_connections 2023-10-16T10:03:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.384+0000 7f4ade25e700 1 -- 172.21.15.158:0/944096162 wait complete. 2023-10-16T10:03:43.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.385+0000 7f4ade25e700 1 Processor -- start 2023-10-16T10:03:43.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.385+0000 7f4ade25e700 1 -- start start 2023-10-16T10:03:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ade25e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad80d2370 0x7f4ad80cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ade25e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 0x7f4ad80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ade25e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 0x7f4ad80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ade25e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80c7a60 con 0x7f4ad815daa0 2023-10-16T10:03:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ade25e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80b90c0 con 0x7f4ad80d2370 2023-10-16T10:03:43.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ade25e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80cabe0 con 0x7f4ad815d6c0 2023-10-16T10:03:43.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 0x7f4ad80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 0x7f4ad80ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58898/0 (socket says 172.21.15.158:58898) 2023-10-16T10:03:43.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 -- 172.21.15.158:0/1821571014 learned_addr learned my addr 172.21.15.158:0/1821571014 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:43.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad80d2370 0x7f4ad80cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 0x7f4ad80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 -- 172.21.15.158:0/1821571014 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 msgr2=0x7f4ad80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad815d6c0 0x7f4ad80b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 -- 172.21.15.158:0/1821571014 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad80d2370 msgr2=0x7f4ad80cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:43.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad80d2370 0x7f4ad80cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 -- 172.21.15.158:0/1821571014 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc009550 con 0x7f4ad815daa0 2023-10-16T10:03:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.386+0000 7f4ad7fff700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad815daa0 0x7f4ad80ca6a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000cc50 tx=0x7f4ad000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.387+0000 7f4abffff700 1 -- 172.21.15.158:0/1821571014 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000ad80 con 0x7f4ad815daa0 2023-10-16T10:03:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.387+0000 7f4abffff700 1 -- 172.21.15.158:0/1821571014 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad000ce50 con 0x7f4ad815daa0 2023-10-16T10:03:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.387+0000 7f4ade25e700 1 -- 172.21.15.158:0/1821571014 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad80c70e0 con 0x7f4ad815daa0 2023-10-16T10:03:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.387+0000 7f4abffff700 1 -- 172.21.15.158:0/1821571014 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0003c40 con 0x7f4ad815daa0 2023-10-16T10:03:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.387+0000 7f4ade25e700 1 -- 172.21.15.158:0/1821571014 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad80c73a0 con 0x7f4ad815daa0 2023-10-16T10:03:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.388+0000 7f4ade25e700 1 -- 172.21.15.158:0/1821571014 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad80c7ba0 con 0x7f4ad815daa0 2023-10-16T10:03:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.389+0000 7f4abffff700 1 -- 172.21.15.158:0/1821571014 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f4ad00050d0 con 0x7f4ad815daa0 2023-10-16T10:03:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.389+0000 7f4abffff700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4ac406b100 0x7f4ac406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:43.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.389+0000 7f4abffff700 1 -- 172.21.15.158:0/1821571014 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f4ad008a770 con 0x7f4ad815daa0 2023-10-16T10:03:43.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.389+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4ac406b100 0x7f4ac406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:43.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.390+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/1821571014 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f4ac406b100 0x7f4ac406d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006fd0 tx=0x7f4ac8008040 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:43.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.393+0000 7f4abffff700 1 -- 172.21.15.158:0/1821571014 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ad00551b0 con 0x7f4ad815daa0 2023-10-16T10:03:43.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.613+0000 7f4ade25e700 1 -- 172.21.15.158:0/1821571014 --> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ad80d1b70 con 0x7f4ac406b100 2023-10-16T10:03:43.615 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:43.614+0000 7f64524f2700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:43.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:43.614+0000 7f4abffff700 1 -- 172.21.15.158:0/1821571014 <== mgr.8994 v2:172.21.15.158:6832/781614815 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4ac8009df0 con 0x7f4ac406b100 2023-10-16T10:03:43.617 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T10:03:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T10:03:43.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:43.622 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2023-10-16T10:03:43.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:43.641 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2023-10-16T10:03:43.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:43.700 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2023-10-16T10:03:43.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:43.758 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2023-10-16T10:03:43.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:43.816 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2023-10-16T10:03:43.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:43.875 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2023-10-16T10:03:43.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:43.934 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2023-10-16T10:03:43.992 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T10:03:44.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.153+0000 7f67c6338700 1 Processor -- start 2023-10-16T10:03:44.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.153+0000 7f67c6338700 1 -- start start 2023-10-16T10:03:44.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67c6338700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 0x7f67c00ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67c6338700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c00ce110 con 0x7f67c00cf420 2023-10-16T10:03:44.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67c6338700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c00ce3f0 con 0x7f67c00c50c0 2023-10-16T10:03:44.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67c6338700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c00c61e0 con 0x7f67c0160d40 2023-10-16T10:03:44.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67bf7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f67c00cf420 0x7f67c00c49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59078/0 (socket says 172.21.15.158:59078) 2023-10-16T10:03:44.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67bf7fe700 1 -- 172.21.15.158:0/873781767 learned_addr learned my addr 172.21.15.158:0/873781767 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:44.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694755882 0 0) 0x7f67c00ce110 con 0x7f67c00cf420 2023-10-16T10:03:44.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a80035b0 con 0x7f67c00cf420 2023-10-16T10:03:44.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.154+0000 7f67bffff700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 0x7f67c00ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:44.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3056377068 0 0) 0x7f67a80035b0 con 0x7f67c00cf420 2023-10-16T10:03:44.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 msgr2=0x7f67c00ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:44.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 0x7f67c00ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 >> v1:172.21.15.172:6789/0 conn(0x7f67c00c50c0 legacy=0x7f67c006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:44.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c00ce6d0 con 0x7f67c00cf420 2023-10-16T10:03:44.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f67b4001670 con 0x7f67c00cf420 2023-10-16T10:03:44.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 >> v1:172.21.15.158:6789/0 conn(0x7f67c00cf420 legacy=0x7f67c00c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:44.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 0x7f67a8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67a8003fb0 0x7f67a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67a8004d90 0x7f67a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a800bac0 con 0x7f67a8004d90 2023-10-16T10:03:44.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a800bc30 con 0x7f67a8003fb0 2023-10-16T10:03:44.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a800bdf0 con 0x7f67c0160d40 2023-10-16T10:03:44.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67c48d5700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67a8004d90 0x7f67a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:44.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67bffff700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 0x7f67a8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:44.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67bf7fe700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67a8003fb0 0x7f67a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:44.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67c48d5700 1 -- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 msgr2=0x7f67a8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:44.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67c48d5700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c0160d40 0x7f67a8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67c48d5700 1 -- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67a8003fb0 msgr2=0x7f67a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:44.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67c48d5700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67a8003fb0 0x7f67a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67c48d5700 1 -- 172.21.15.158:0/873781767 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a80035b0 con 0x7f67a8004d90 2023-10-16T10:03:44.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67c48d5700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67a8004d90 0x7f67a8005210 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f67c00d0a40 tx=0x7f67a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8335dac67cb1316 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:44.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.155+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8001db0 con 0x7f67a8004d90 2023-10-16T10:03:44.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b800cc40 con 0x7f67a8004d90 2023-10-16T10:03:44.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67beffd700 1 -- 172.21.15.158:0/873781767 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8003a00 con 0x7f67a8004d90 2023-10-16T10:03:44.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67c6338700 1 -- 172.21.15.158:0/873781767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67a8004d90 msgr2=0x7f67a8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:44.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67c6338700 1 --2- 172.21.15.158:0/873781767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67a8004d90 0x7f67a8005210 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f67c00d0a40 tx=0x7f67a8005880 comp rx=0 tx=0).stop 2023-10-16T10:03:44.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67c48d5700 1 -- 172.21.15.158:0/873781767 reap_dead start 2023-10-16T10:03:44.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67c6338700 1 -- 172.21.15.158:0/873781767 shutdown_connections 2023-10-16T10:03:44.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67c6338700 1 -- 172.21.15.158:0/873781767 >> 172.21.15.158:0/873781767 conn(0x7f67c00cc5b0 msgr2=0x7f67c00cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:44.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67c6338700 1 -- 172.21.15.158:0/873781767 shutdown_connections 2023-10-16T10:03:44.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.156+0000 7f67c6338700 1 -- 172.21.15.158:0/873781767 wait complete. 2023-10-16T10:03:44.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.157+0000 7f67c6338700 1 Processor -- start 2023-10-16T10:03:44.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67c6338700 1 -- start start 2023-10-16T10:03:44.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67c6338700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c00c8900 0x7f67c00c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67c6338700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c00c8ce0 0x7f67c00c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67c6338700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c00af460 0x7f67c00af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67c6338700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c00b4580 con 0x7f67c00c8900 2023-10-16T10:03:44.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67c6338700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c00b3900 con 0x7f67c00c8ce0 2023-10-16T10:03:44.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67c6338700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c00b3be0 con 0x7f67c00af460 2023-10-16T10:03:44.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c00c8900 0x7f67c00c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:44.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c00c8900 0x7f67c00c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58914/0 (socket says 172.21.15.158:58914) 2023-10-16T10:03:44.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.158+0000 7f67bffff700 1 -- 172.21.15.158:0/3826241531 learned_addr learned my addr 172.21.15.158:0/3826241531 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:44.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bf7fe700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c00c8ce0 0x7f67c00c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:44.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bffff700 1 -- 172.21.15.158:0/3826241531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c00af460 msgr2=0x7f67c00af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:44.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bffff700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c00af460 0x7f67c00af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bffff700 1 -- 172.21.15.158:0/3826241531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c00c8ce0 msgr2=0x7f67c00c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:44.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bffff700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c00c8ce0 0x7f67c00c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bffff700 1 -- 172.21.15.158:0/3826241531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c01612a0 con 0x7f67c00c8900 2023-10-16T10:03:44.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bffff700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c00c8900 0x7f67c00c6c00 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f67b0011aa0 tx=0x7f67b0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:44.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bcff9700 1 -- 172.21.15.158:0/3826241531 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b000f900 con 0x7f67c00c8900 2023-10-16T10:03:44.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bcff9700 1 -- 172.21.15.158:0/3826241531 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b000fa60 con 0x7f67c00c8900 2023-10-16T10:03:44.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c00b3ec0 con 0x7f67c00c8900 2023-10-16T10:03:44.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67bcff9700 1 -- 172.21.15.158:0/3826241531 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b000fe10 con 0x7f67c00c8900 2023-10-16T10:03:44.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.159+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c00b55a0 con 0x7f67c00c8900 2023-10-16T10:03:44.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.160+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c0066db0 con 0x7f67c00c8900 2023-10-16T10:03:44.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.160+0000 7f67bcff9700 1 -- 172.21.15.158:0/3826241531 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f67b0014440 con 0x7f67c00c8900 2023-10-16T10:03:44.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.161+0000 7f67bcff9700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f67ac06b100 0x7f67ac06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:44.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.161+0000 7f67bcff9700 1 -- 172.21.15.158:0/3826241531 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f67b008a520 con 0x7f67c00c8900 2023-10-16T10:03:44.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.161+0000 7f67bf7fe700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f67ac06b100 0x7f67ac06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:44.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.161+0000 7f67bf7fe700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f67ac06b100 0x7f67ac06d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f67b4001050 tx=0x7f67b4009040 comp rx=0 tx=0).ready entity=mgr.8994 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:44.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.165+0000 7f67bcff9700 1 -- 172.21.15.158:0/3826241531 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f67b0054fa0 con 0x7f67c00c8900 2023-10-16T10:03:44.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.417+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f67c0072e90 con 0x7f67c00c8900 2023-10-16T10:03:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.877+0000 7f67bcff9700 1 -- 172.21.15.158:0/3826241531 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v81) v1 ==== 93+0+0 (secure 0 0 0) 0x7f67b00585c0 con 0x7f67c00c8900 2023-10-16T10:03:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.880+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f67ac06b100 msgr2=0x7f67ac06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.880+0000 7f67c6338700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f67ac06b100 0x7f67ac06d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f67b4001050 tx=0x7f67b4009040 comp rx=0 tx=0).stop 2023-10-16T10:03:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.880+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c00c8900 msgr2=0x7f67c00c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.880+0000 7f67c6338700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c00c8900 0x7f67c00c6c00 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f67b0011aa0 tx=0x7f67b0011e60 comp rx=0 tx=0).stop 2023-10-16T10:03:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.881+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 shutdown_connections 2023-10-16T10:03:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.881+0000 7f67c6338700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f67ac06b100 0x7f67ac06d5c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.881+0000 7f67c6338700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c00af460 0x7f67c00af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.881+0000 7f67c6338700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c00c8900 0x7f67c00c6c00 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.881+0000 7f67c6338700 1 --2- 172.21.15.158:0/3826241531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c00c8ce0 0x7f67c00c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:44.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.881+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 >> 172.21.15.158:0/3826241531 conn(0x7f67c00cc5b0 msgr2=0x7f67c00cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:44.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.882+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 shutdown_connections 2023-10-16T10:03:44.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:44.882+0000 7f67c6338700 1 -- 172.21.15.158:0/3826241531 wait complete. 2023-10-16T10:03:44.902 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:03:44.903 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:44.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:44.988 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:03:44.988 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:44.989 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:45.006 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:45.007 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:45.025 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.023+0000 7fc1bbd99200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:03:45.034 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.033+0000 7f88cafcf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:03:45.054 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:45.090 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.089+0000 7fc1bbd99200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:45.103 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.102+0000 7f88cafcf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:45.161 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.160+0000 7fc1bbd99200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:03:45.179 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.178+0000 7f88cafcf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:03:45.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.213+0000 7fe81db29700 1 Processor -- start 2023-10-16T10:03:45.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.213+0000 7fe81db29700 1 -- start start 2023-10-16T10:03:45.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe81db29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe81815d7c0 0x7fe8180d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe81db29700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe8180d1210 0x7fe8180ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe81db29700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180d3bd0 con 0x7fe8180ce950 2023-10-16T10:03:45.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe81db29700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8180d4060 con 0x7fe8180d1210 2023-10-16T10:03:45.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe81db29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8180cede0 con 0x7fe81815d7c0 2023-10-16T10:03:45.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe817fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe8180ce950 0x7fe8180b50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59152/0 (socket says 172.21.15.158:59152) 2023-10-16T10:03:45.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe817fff700 1 -- 172.21.15.158:0/1322213567 learned_addr learned my addr 172.21.15.158:0/1322213567 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:45.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe8177fe700 1 --2- 172.21.15.158:0/1322213567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe81815d7c0 0x7fe8180d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.214+0000 7fe816ffd700 1 --2- 172.21.15.158:0/1322213567 >> v2:172.21.15.172:3300/0 conn(0x7fe8180d1210 0x7fe8180ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8167fc700 1 -- 172.21.15.158:0/1322213567 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376892095 0 0) 0x7fe8180d3bd0 con 0x7fe8180ce950 2023-10-16T10:03:45.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8167fc700 1 -- 172.21.15.158:0/1322213567 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8040035b0 con 0x7fe8180ce950 2023-10-16T10:03:45.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8177fe700 1 -- 172.21.15.158:0/1322213567 >> v2:172.21.15.172:3300/0 conn(0x7fe8180d1210 msgr2=0x7fe8180ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:45.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8177fe700 1 --2- 172.21.15.158:0/1322213567 >> v2:172.21.15.172:3300/0 conn(0x7fe8180d1210 0x7fe8180ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8177fe700 1 -- 172.21.15.158:0/1322213567 >> v1:172.21.15.158:6789/0 conn(0x7fe8180ce950 legacy=0x7fe8180b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8177fe700 1 -- 172.21.15.158:0/1322213567 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8180d2cf0 con 0x7fe81815d7c0 2023-10-16T10:03:45.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe816ffd700 1 --2- 172.21.15.158:0/1322213567 >> v2:172.21.15.172:3300/0 conn(0x7fe8180d1210 0x7fe8180ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:45.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8177fe700 1 --2- 172.21.15.158:0/1322213567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe81815d7c0 0x7fe8180d27b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe808009d90 tx=0x7fe80800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2c066119fdc40b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:45.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8167fc700 1 -- 172.21.15.158:0/1322213567 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80800e070 con 0x7fe81815d7c0 2023-10-16T10:03:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8167fc700 1 -- 172.21.15.158:0/1322213567 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80800bb20 con 0x7fe81815d7c0 2023-10-16T10:03:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.215+0000 7fe8167fc700 1 -- 172.21.15.158:0/1322213567 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808012430 con 0x7fe81815d7c0 2023-10-16T10:03:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 -- 172.21.15.158:0/1322213567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe81815d7c0 msgr2=0x7fe8180d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 --2- 172.21.15.158:0/1322213567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe81815d7c0 0x7fe8180d27b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe808009d90 tx=0x7fe80800b330 comp rx=0 tx=0).stop 2023-10-16T10:03:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 -- 172.21.15.158:0/1322213567 shutdown_connections 2023-10-16T10:03:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 --2- 172.21.15.158:0/1322213567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe81815d7c0 0x7fe8180d27b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 --2- 172.21.15.158:0/1322213567 >> v2:172.21.15.172:3300/0 conn(0x7fe8180d1210 0x7fe8180ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 -- 172.21.15.158:0/1322213567 >> 172.21.15.158:0/1322213567 conn(0x7fe8180669f0 msgr2=0x7fe818066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 -- 172.21.15.158:0/1322213567 shutdown_connections 2023-10-16T10:03:45.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.216+0000 7fe81db29700 1 -- 172.21.15.158:0/1322213567 wait complete. 2023-10-16T10:03:45.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.217+0000 7fe81db29700 1 Processor -- start 2023-10-16T10:03:45.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.217+0000 7fe81db29700 1 -- start start 2023-10-16T10:03:45.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe81db29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8180ce950 0x7fe818159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe81db29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8180d1210 0x7fe818153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe81db29700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe81815d7c0 0x7fe8180c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe81db29700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe818158d60 con 0x7fe8180ce950 2023-10-16T10:03:45.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe81db29700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8181537c0 con 0x7fe81815d7c0 2023-10-16T10:03:45.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe81db29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe818158400 con 0x7fe8180d1210 2023-10-16T10:03:45.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8180ce950 0x7fe818159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8180ce950 0x7fe818159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58978/0 (socket says 172.21.15.158:58978) 2023-10-16T10:03:45.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 -- 172.21.15.158:0/1353149748 learned_addr learned my addr 172.21.15.158:0/1353149748 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:45.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe816ffd700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8180d1210 0x7fe818153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe817fff700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe81815d7c0 0x7fe8180c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 -- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8180d1210 msgr2=0x7fe818153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8180d1210 0x7fe818153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 -- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe81815d7c0 msgr2=0x7fe8180c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe81815d7c0 0x7fe8180c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.218+0000 7fe8177fe700 1 -- 172.21.15.158:0/1353149748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000094a0 con 0x7fe8180ce950 2023-10-16T10:03:45.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.219+0000 7fe8177fe700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8180ce950 0x7fe818159180 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fe808014040 tx=0x7fe808004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:45.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.219+0000 7fe80ffff700 1 -- 172.21.15.158:0/1353149748 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808012430 con 0x7fe8180ce950 2023-10-16T10:03:45.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.219+0000 7fe80ffff700 1 -- 172.21.15.158:0/1353149748 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe808020070 con 0x7fe8180ce950 2023-10-16T10:03:45.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.219+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe808009a70 con 0x7fe8180ce950 2023-10-16T10:03:45.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.219+0000 7fe80ffff700 1 -- 172.21.15.158:0/1353149748 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808004280 con 0x7fe8180ce950 2023-10-16T10:03:45.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.219+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8180c6820 con 0x7fe8180ce950 2023-10-16T10:03:45.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.220+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8180681b0 con 0x7fe8180ce950 2023-10-16T10:03:45.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.221+0000 7fe80ffff700 1 -- 172.21.15.158:0/1353149748 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fe8080043e0 con 0x7fe8180ce950 2023-10-16T10:03:45.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.221+0000 7fe80ffff700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 0x7fe7f806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.221+0000 7fe816ffd700 1 -- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 msgr2=0x7fe7f806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:45.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.221+0000 7fe80ffff700 1 -- 172.21.15.158:0/1353149748 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fe808022070 con 0x7fe8180ce950 2023-10-16T10:03:45.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.221+0000 7fe816ffd700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 0x7fe7f806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:45.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.225+0000 7fe80ffff700 1 -- 172.21.15.158:0/1353149748 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8080566f0 con 0x7fe8180ce950 2023-10-16T10:03:45.239 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.238+0000 7fc1bbd99200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:03:45.263 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.262+0000 7f88cafcf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:03:45.380 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.378+0000 7fc1bbd99200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:03:45.408 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.407+0000 7f88cafcf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:03:45.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.422+0000 7fe816ffd700 1 -- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 msgr2=0x7fe7f806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:45.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.422+0000 7fe816ffd700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 0x7fe7f806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:45.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.426+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe81815ce70 con 0x7fe8180ce950 2023-10-16T10:03:45.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.428+0000 7fe80ffff700 1 -- 172.21.15.158:0/1353149748 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17995 (secure 0 0 0) 0x7fe808059d10 con 0x7fe8180ce950 2023-10-16T10:03:45.429 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:45.429 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":141,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:37.828969+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:03:21.520809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T10:03:22.576358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:6832/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/3946986955":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/2041489640":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4016680871":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4198985108":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1741373068":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1608841554":"2023-10-17T10:03:37.828937+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:45.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 msgr2=0x7fe7f806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:45.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 0x7fe7f806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8180ce950 msgr2=0x7fe818159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8180ce950 0x7fe818159180 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fe808014040 tx=0x7fe808004ad0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 shutdown_connections 2023-10-16T10:03:45.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe7f806b0b0 0x7fe7f806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe81815d7c0 0x7fe8180c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8180ce950 0x7fe818159180 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 --2- 172.21.15.158:0/1353149748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8180d1210 0x7fe818153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 >> 172.21.15.158:0/1353149748 conn(0x7fe8180669f0 msgr2=0x7fe818073e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:45.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 shutdown_connections 2023-10-16T10:03:45.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.430+0000 7fe81db29700 1 -- 172.21.15.158:0/1353149748 wait complete. 2023-10-16T10:03:45.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:03:21.520809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:03:22.576358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:03:45.445 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:03:45.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.611+0000 7f234dfd5700 1 Processor -- start 2023-10-16T10:03:45.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.611+0000 7f234dfd5700 1 -- start start 2023-10-16T10:03:45.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f234dfd5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f234815deb0 0x7f23480ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f234dfd5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f23480ccc00 0x7f23480cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f234dfd5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23480d0ae0 con 0x7f234815dad0 2023-10-16T10:03:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f234dfd5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23480d12d0 con 0x7f23480ccc00 2023-10-16T10:03:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f234dfd5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23480cbdf0 con 0x7f234815deb0 2023-10-16T10:03:45.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f23477fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f234815dad0 0x7f23480cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59180/0 (socket says 172.21.15.158:59180) 2023-10-16T10:03:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f23477fe700 1 -- 172.21.15.158:0/3685486166 learned_addr learned my addr 172.21.15.158:0/3685486166 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.612+0000 7f2347fff700 1 --2- 172.21.15.158:0/3685486166 >> v2:172.21.15.158:3300/0 conn(0x7f23480ccc00 0x7f23480cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f2346ffd700 1 --2- 172.21.15.158:0/3685486166 >> v2:172.21.15.172:3300/0 conn(0x7f234815deb0 0x7f23480ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f2347fff700 1 -- 172.21.15.158:0/3685486166 >> v2:172.21.15.172:3300/0 conn(0x7f234815deb0 msgr2=0x7f23480ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f2347fff700 1 --2- 172.21.15.158:0/3685486166 >> v2:172.21.15.172:3300/0 conn(0x7f234815deb0 0x7f23480ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f2347fff700 1 -- 172.21.15.158:0/3685486166 >> v1:172.21.15.158:6789/0 conn(0x7f234815dad0 legacy=0x7f23480cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f2347fff700 1 -- 172.21.15.158:0/3685486166 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480cc280 con 0x7f23480ccc00 2023-10-16T10:03:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f23467fc700 1 -- 172.21.15.158:0/3685486166 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832660721 0 0) 0x7f23480d0ae0 con 0x7f234815dad0 2023-10-16T10:03:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f2347fff700 1 --2- 172.21.15.158:0/3685486166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23480ccc00 0x7f23480cd030 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f2340009d90 tx=0x7f234000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c038e068ab6186b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:45.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f23467fc700 1 -- 172.21.15.158:0/3685486166 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234000e070 con 0x7f23480ccc00 2023-10-16T10:03:45.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f23467fc700 1 -- 172.21.15.158:0/3685486166 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234000bae0 con 0x7f23480ccc00 2023-10-16T10:03:45.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.613+0000 7f23467fc700 1 -- 172.21.15.158:0/3685486166 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234000be90 con 0x7f23480ccc00 2023-10-16T10:03:45.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 -- 172.21.15.158:0/3685486166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23480ccc00 msgr2=0x7f23480cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 --2- 172.21.15.158:0/3685486166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23480ccc00 0x7f23480cd030 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f2340009d90 tx=0x7f234000b330 comp rx=0 tx=0).stop 2023-10-16T10:03:45.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 -- 172.21.15.158:0/3685486166 shutdown_connections 2023-10-16T10:03:45.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 --2- 172.21.15.158:0/3685486166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23480ccc00 0x7f23480cd030 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 --2- 172.21.15.158:0/3685486166 >> v2:172.21.15.172:3300/0 conn(0x7f234815deb0 0x7f23480ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 -- 172.21.15.158:0/3685486166 >> 172.21.15.158:0/3685486166 conn(0x7f2348099a10 msgr2=0x7f23480d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:45.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 -- 172.21.15.158:0/3685486166 shutdown_connections 2023-10-16T10:03:45.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.614+0000 7f234dfd5700 1 -- 172.21.15.158:0/3685486166 wait complete. 2023-10-16T10:03:45.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.615+0000 7f234dfd5700 1 Processor -- start 2023-10-16T10:03:45.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.615+0000 7f234dfd5700 1 -- start start 2023-10-16T10:03:45.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f234dfd5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23480ccc00 0x7f23481429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f234dfd5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f234815dad0 0x7f2348154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f234dfd5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f234815deb0 0x7f23480b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f234dfd5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2348156250 con 0x7f234815dad0 2023-10-16T10:03:45.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f234dfd5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2348155310 con 0x7f23480ccc00 2023-10-16T10:03:45.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f234dfd5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23481555d0 con 0x7f234815deb0 2023-10-16T10:03:45.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f2346ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f234815dad0 0x7f2348154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f2346ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f234815dad0 0x7f2348154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59012/0 (socket says 172.21.15.158:59012) 2023-10-16T10:03:45.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f2346ffd700 1 -- 172.21.15.158:0/1848822659 learned_addr learned my addr 172.21.15.158:0/1848822659 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:45.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f23477fe700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23480ccc00 0x7f23481429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.616+0000 7f2347fff700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f234815deb0 0x7f23480b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f2346ffd700 1 -- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f234815deb0 msgr2=0x7f23480b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f2346ffd700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f234815deb0 0x7f23480b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f2346ffd700 1 -- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23480ccc00 msgr2=0x7f23481429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f2346ffd700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23480ccc00 0x7f23481429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f2346ffd700 1 -- 172.21.15.158:0/1848822659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2340009a70 con 0x7f234815dad0 2023-10-16T10:03:45.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f23477fe700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23480ccc00 0x7f23481429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:45.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f2347fff700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f234815deb0 0x7f23480b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:45.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f2346ffd700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f234815dad0 0x7f2348154dd0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f233c00e870 tx=0x7f233c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:45.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f232ffff700 1 -- 172.21.15.158:0/1848822659 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c009590 con 0x7f234815dad0 2023-10-16T10:03:45.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f232ffff700 1 -- 172.21.15.158:0/1848822659 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233c013680 con 0x7f234815dad0 2023-10-16T10:03:45.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23481558b0 con 0x7f234815dad0 2023-10-16T10:03:45.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f232ffff700 1 -- 172.21.15.158:0/1848822659 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c0139f0 con 0x7f234815dad0 2023-10-16T10:03:45.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.617+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2348155e70 con 0x7f234815dad0 2023-10-16T10:03:45.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.618+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23480681b0 con 0x7f234815dad0 2023-10-16T10:03:45.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.619+0000 7f232ffff700 1 -- 172.21.15.158:0/1848822659 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f233c013b50 con 0x7f234815dad0 2023-10-16T10:03:45.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.619+0000 7f232ffff700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 0x7f233406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.619+0000 7f23477fe700 1 -- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 msgr2=0x7f233406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:45.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.619+0000 7f232ffff700 1 -- 172.21.15.158:0/1848822659 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f233c089b20 con 0x7f234815dad0 2023-10-16T10:03:45.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.619+0000 7f23477fe700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 0x7f233406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:45.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.623+0000 7f232ffff700 1 -- 172.21.15.158:0/1848822659 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f233c054580 con 0x7f234815dad0 2023-10-16T10:03:45.703 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.702+0000 7fc1bbd99200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:03:45.754 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.753+0000 7f88cafcf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:03:45.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.820+0000 7f23477fe700 1 -- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 msgr2=0x7f233406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:45.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.820+0000 7f23477fe700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 0x7f233406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:45.821 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.820+0000 7fc1bbd99200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:03:45.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.823+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23480ca360 con 0x7f234815dad0 2023-10-16T10:03:45.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.823+0000 7f232ffff700 1 -- 172.21.15.158:0/1848822659 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f233c0210a0 con 0x7f234815dad0 2023-10-16T10:03:45.826 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:03:45.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.825+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 msgr2=0x7f233406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:45.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.825+0000 7f234dfd5700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 0x7f233406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.825+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f234815dad0 msgr2=0x7f2348154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.825+0000 7f234dfd5700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f234815dad0 0x7f2348154dd0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f233c00e870 tx=0x7f233c00eb80 comp rx=0 tx=0).stop 2023-10-16T10:03:45.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 shutdown_connections 2023-10-16T10:03:45.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f233406b0b0 0x7f233406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f234815deb0 0x7f23480b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23480ccc00 0x7f23481429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 --2- 172.21.15.158:0/1848822659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f234815dad0 0x7f2348154dd0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 >> 172.21.15.158:0/1848822659 conn(0x7f2348099a10 msgr2=0x7f23481527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:45.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 shutdown_connections 2023-10-16T10:03:45.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.826+0000 7f234dfd5700 1 -- 172.21.15.158:0/1848822659 wait complete. 2023-10-16T10:03:45.841 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:03:45.876 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.875+0000 7f88cafcf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:03:45.895 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.894+0000 7fc1bbd99200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:03:45.953 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:45.952+0000 7f88cafcf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:03:45.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.969+0000 7fce5f26e700 1 Processor -- start 2023-10-16T10:03:45.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.969+0000 7fce5f26e700 1 -- start start 2023-10-16T10:03:45.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5f26e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fce5815b540 0x7fce5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5f26e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce5806afe0 0x7fce58161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5f26e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5815ee60 con 0x7fce5806f180 2023-10-16T10:03:45.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5f26e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5815b920 con 0x7fce5806afe0 2023-10-16T10:03:45.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5f26e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5815e2e0 con 0x7fce5815b540 2023-10-16T10:03:45.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce5806afe0 0x7fce58161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce5806afe0 0x7fce58161860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59024/0 (socket says 172.21.15.158:59024) 2023-10-16T10:03:45.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5c809700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fce5806f180 0x7fce5806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59196/0 (socket says 172.21.15.158:59196) 2023-10-16T10:03:45.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 -- 172.21.15.158:0/3217240021 learned_addr learned my addr 172.21.15.158:0/3217240021 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:45.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d00a700 1 --2- 172.21.15.158:0/3217240021 >> v2:172.21.15.172:3300/0 conn(0x7fce5815b540 0x7fce5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 -- 172.21.15.158:0/3217240021 >> v2:172.21.15.172:3300/0 conn(0x7fce5815b540 msgr2=0x7fce5806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:45.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 --2- 172.21.15.158:0/3217240021 >> v2:172.21.15.172:3300/0 conn(0x7fce5815b540 0x7fce5806d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 -- 172.21.15.158:0/3217240021 >> v1:172.21.15.158:6789/0 conn(0x7fce5806f180 legacy=0x7fce5806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce4ffff700 1 -- 172.21.15.158:0/3217240021 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123779640 0 0) 0x7fce5815ee60 con 0x7fce5806f180 2023-10-16T10:03:45.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 -- 172.21.15.158:0/3217240021 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5815e5c0 con 0x7fce5806afe0 2023-10-16T10:03:45.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce5d80b700 1 --2- 172.21.15.158:0/3217240021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806afe0 0x7fce58161860 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fce5815ba60 tx=0x7fce5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcee9324b244c37 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:45.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.970+0000 7fce4ffff700 1 -- 172.21.15.158:0/3217240021 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5000e070 con 0x7fce5806afe0 2023-10-16T10:03:45.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce4ffff700 1 -- 172.21.15.158:0/3217240021 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce50003a20 con 0x7fce5806afe0 2023-10-16T10:03:45.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce4ffff700 1 -- 172.21.15.158:0/3217240021 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50003dd0 con 0x7fce5806afe0 2023-10-16T10:03:45.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 -- 172.21.15.158:0/3217240021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806afe0 msgr2=0x7fce58161860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 --2- 172.21.15.158:0/3217240021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806afe0 0x7fce58161860 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fce5815ba60 tx=0x7fce5000b330 comp rx=0 tx=0).stop 2023-10-16T10:03:45.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 -- 172.21.15.158:0/3217240021 shutdown_connections 2023-10-16T10:03:45.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 --2- 172.21.15.158:0/3217240021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806afe0 0x7fce58161860 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 --2- 172.21.15.158:0/3217240021 >> v2:172.21.15.172:3300/0 conn(0x7fce5815b540 0x7fce5806d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 -- 172.21.15.158:0/3217240021 >> 172.21.15.158:0/3217240021 conn(0x7fce580d03b0 msgr2=0x7fce580d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:45.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 -- 172.21.15.158:0/3217240021 shutdown_connections 2023-10-16T10:03:45.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.971+0000 7fce5f26e700 1 -- 172.21.15.158:0/3217240021 wait complete. 2023-10-16T10:03:45.978 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:45.975+0000 7fc1bbd99200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:03:45.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.972+0000 7fce5f26e700 1 Processor -- start 2023-10-16T10:03:45.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.972+0000 7fce5f26e700 1 -- start start 2023-10-16T10:03:45.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5f26e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce5806afe0 0x7fce58147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5f26e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806f180 0x7fce58161760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5f26e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce5815b540 0x7fce58161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5f26e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580af3c0 con 0x7fce5806f180 2023-10-16T10:03:45.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5f26e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580ae7b0 con 0x7fce5815b540 2023-10-16T10:03:45.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5f26e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce580aea90 con 0x7fce5806afe0 2023-10-16T10:03:45.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5c809700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806f180 0x7fce58161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5c809700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806f180 0x7fce58161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59028/0 (socket says 172.21.15.158:59028) 2023-10-16T10:03:45.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5c809700 1 -- 172.21.15.158:0/290446781 learned_addr learned my addr 172.21.15.158:0/290446781 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:45.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5d00a700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce5806afe0 0x7fce58147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.973+0000 7fce5d80b700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce5815b540 0x7fce58161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:45.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5c809700 1 -- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce5806afe0 msgr2=0x7fce58147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5c809700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce5806afe0 0x7fce58147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5c809700 1 -- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce5815b540 msgr2=0x7fce58161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:45.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5c809700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce5815b540 0x7fce58161ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:45.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5c809700 1 -- 172.21.15.158:0/290446781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50009b30 con 0x7fce5806f180 2023-10-16T10:03:45.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5d80b700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce5815b540 0x7fce58161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:45.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5c809700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806f180 0x7fce58161760 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fce480011e0 tx=0x7fce4800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:45.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce4dffb700 1 -- 172.21.15.158:0/290446781 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48004c80 con 0x7fce5806f180 2023-10-16T10:03:45.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce4dffb700 1 -- 172.21.15.158:0/290446781 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce48004de0 con 0x7fce5806f180 2023-10-16T10:03:45.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce4dffb700 1 -- 172.21.15.158:0/290446781 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48007310 con 0x7fce5806f180 2023-10-16T10:03:45.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580aed70 con 0x7fce5806f180 2023-10-16T10:03:45.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.974+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce58147c90 con 0x7fce5806f180 2023-10-16T10:03:45.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.975+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce58156110 con 0x7fce5806f180 2023-10-16T10:03:45.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.975+0000 7fce4dffb700 1 -- 172.21.15.158:0/290446781 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fce48007470 con 0x7fce5806f180 2023-10-16T10:03:45.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.976+0000 7fce4dffb700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 0x7fce3c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:45.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.976+0000 7fce5d00a700 1 -- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 msgr2=0x7fce3c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:45.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.976+0000 7fce4dffb700 1 -- 172.21.15.158:0/290446781 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fce48089ab0 con 0x7fce5806f180 2023-10-16T10:03:45.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.976+0000 7fce5d00a700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 0x7fce3c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:45.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:45.980+0000 7fce4dffb700 1 -- 172.21.15.158:0/290446781 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fce48054480 con 0x7fce5806f180 2023-10-16T10:03:46.039 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:46.038+0000 7f88cafcf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:03:46.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.177+0000 7fce5d00a700 1 -- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 msgr2=0x7fce3c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:46.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.177+0000 7fce5d00a700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 0x7fce3c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:46.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.181+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce580681b0 con 0x7fce5806f180 2023-10-16T10:03:46.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.182+0000 7fce4dffb700 1 -- 172.21.15.158:0/290446781 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7fce48057aa0 con 0x7fce5806f180 2023-10-16T10:03:46.183 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:03:46.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 msgr2=0x7fce3c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:46.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 0x7fce3c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806f180 msgr2=0x7fce58161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806f180 0x7fce58161760 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fce480011e0 tx=0x7fce4800ca30 comp rx=0 tx=0).stop 2023-10-16T10:03:46.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 shutdown_connections 2023-10-16T10:03:46.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fce3c06b0b0 0x7fce3c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce5815b540 0x7fce58161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce5806afe0 0x7fce58147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 --2- 172.21.15.158:0/290446781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce5806f180 0x7fce58161760 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 >> 172.21.15.158:0/290446781 conn(0x7fce580d03b0 msgr2=0x7fce5806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:46.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 shutdown_connections 2023-10-16T10:03:46.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.184+0000 7fce5f26e700 1 -- 172.21.15.158:0/290446781 wait complete. 2023-10-16T10:03:46.198 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:03:46.345 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:46.343+0000 7fc1bbd99200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:03:46.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.364+0000 7fdc81424700 1 Processor -- start 2023-10-16T10:03:46.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.364+0000 7fdc81424700 1 -- start start 2023-10-16T10:03:46.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc81424700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdc7c0d0c60 0x7fdc7c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc81424700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc7c0b3460 0x7fdc7c0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc81424700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0b4420 con 0x7fdc7c15b6b0 2023-10-16T10:03:46.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc81424700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c15ba90 con 0x7fdc7c0b3460 2023-10-16T10:03:46.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc81424700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0b38d0 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc7c0b3460 0x7fdc7c0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc7c0b3460 0x7fdc7c0b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59066/0 (socket says 172.21.15.158:59066) 2023-10-16T10:03:46.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7b7fe700 1 -- 172.21.15.158:0/2047502354 learned_addr learned my addr 172.21.15.158:0/2047502354 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:46.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7a7fc700 1 --2- 172.21.15.158:0/2047502354 >> v2:172.21.15.172:3300/0 conn(0x7fdc7c0d0c60 0x7fdc7c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7b7fe700 1 -- 172.21.15.158:0/2047502354 >> v2:172.21.15.172:3300/0 conn(0x7fdc7c0d0c60 msgr2=0x7fdc7c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:46.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7b7fe700 1 --2- 172.21.15.158:0/2047502354 >> v2:172.21.15.172:3300/0 conn(0x7fdc7c0d0c60 0x7fdc7c0b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7b7fe700 1 -- 172.21.15.158:0/2047502354 >> v1:172.21.15.172:6789/0 conn(0x7fdc7c15b6b0 legacy=0x7fdc7c0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.365+0000 7fdc7b7fe700 1 -- 172.21.15.158:0/2047502354 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0cbc20 con 0x7fdc7c0b3460 2023-10-16T10:03:46.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.366+0000 7fdc7b7fe700 1 --2- 172.21.15.158:0/2047502354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0b3460 0x7fdc7c0b48b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0b7610 tx=0x7fdc7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ef3c3ed8fec00f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:46.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.366+0000 7fdc79ffb700 1 -- 172.21.15.158:0/2047502354 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7400e070 con 0x7fdc7c0b3460 2023-10-16T10:03:46.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.366+0000 7fdc79ffb700 1 -- 172.21.15.158:0/2047502354 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7400b8d0 con 0x7fdc7c0b3460 2023-10-16T10:03:46.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.366+0000 7fdc79ffb700 1 -- 172.21.15.158:0/2047502354 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7400bc80 con 0x7fdc7c0b3460 2023-10-16T10:03:46.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 -- 172.21.15.158:0/2047502354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0b3460 msgr2=0x7fdc7c0b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 --2- 172.21.15.158:0/2047502354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0b3460 0x7fdc7c0b48b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0b7610 tx=0x7fdc7400b330 comp rx=0 tx=0).stop 2023-10-16T10:03:46.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 -- 172.21.15.158:0/2047502354 shutdown_connections 2023-10-16T10:03:46.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 --2- 172.21.15.158:0/2047502354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0b3460 0x7fdc7c0b48b0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 --2- 172.21.15.158:0/2047502354 >> v2:172.21.15.172:3300/0 conn(0x7fdc7c0d0c60 0x7fdc7c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 -- 172.21.15.158:0/2047502354 >> 172.21.15.158:0/2047502354 conn(0x7fdc7c0cdf20 msgr2=0x7fdc7c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:46.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 -- 172.21.15.158:0/2047502354 shutdown_connections 2023-10-16T10:03:46.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.367+0000 7fdc81424700 1 -- 172.21.15.158:0/2047502354 wait complete. 2023-10-16T10:03:46.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.368+0000 7fdc81424700 1 Processor -- start 2023-10-16T10:03:46.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.368+0000 7fdc81424700 1 -- start start 2023-10-16T10:03:46.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc81424700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c0b3460 0x7fdc7c151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc81424700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0d0c60 0x7fdc7c06c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc81424700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c15b6b0 0x7fdc7c0cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc81424700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c06d390 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc81424700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c151e60 con 0x7fdc7c15b6b0 2023-10-16T10:03:46.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc81424700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0c8810 con 0x7fdc7c0b3460 2023-10-16T10:03:46.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0d0c60 0x7fdc7c06c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0d0c60 0x7fdc7c06c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59070/0 (socket says 172.21.15.158:59070) 2023-10-16T10:03:46.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 -- 172.21.15.158:0/2814888219 learned_addr learned my addr 172.21.15.158:0/2814888219 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:46.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7b7fe700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c15b6b0 0x7fdc7c0cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7affd700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c0b3460 0x7fdc7c151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 -- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c0b3460 msgr2=0x7fdc7c151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c0b3460 0x7fdc7c151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 -- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c15b6b0 msgr2=0x7fdc7c0cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c15b6b0 0x7fdc7c0cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.369+0000 7fdc7a7fc700 1 -- 172.21.15.158:0/2814888219 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc74009b30 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.370+0000 7fdc7a7fc700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0d0c60 0x7fdc7c06c3b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e870 tx=0x7fdc7000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:46.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.370+0000 7fdc637fe700 1 -- 172.21.15.158:0/2814888219 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70009590 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.370+0000 7fdc637fe700 1 -- 172.21.15.158:0/2814888219 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc70013680 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.370+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c06caf0 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.370+0000 7fdc637fe700 1 -- 172.21.15.158:0/2814888219 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70013a30 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.370+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c06d0b0 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.371+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c0c8550 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.371+0000 7fdc637fe700 1 -- 172.21.15.158:0/2814888219 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fdc70013b90 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.372+0000 7fdc637fe700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 0x7fdc6806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.372+0000 7fdc7affd700 1 -- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 msgr2=0x7fdc6806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:46.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.372+0000 7fdc637fe700 1 -- 172.21.15.158:0/2814888219 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fdc70089b10 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.372+0000 7fdc7affd700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 0x7fdc6806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:46.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.376+0000 7fdc637fe700 1 -- 172.21.15.158:0/2814888219 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc700544f0 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.430 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:46.429+0000 7f88cafcf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:03:46.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.573+0000 7fdc7affd700 1 -- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 msgr2=0x7fdc6806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:46.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.573+0000 7fdc7affd700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 0x7fdc6806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:46.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.574+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdc7c0681b0 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.574+0000 7fdc637fe700 1 -- 172.21.15.158:0/2814888219 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdc70057b10 con 0x7fdc7c0d0c60 2023-10-16T10:03:46.575 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:03:46.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 msgr2=0x7fdc6806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:46.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 0x7fdc6806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0d0c60 msgr2=0x7fdc7c06c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0d0c60 0x7fdc7c06c3b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e870 tx=0x7fdc7000eb80 comp rx=0 tx=0).stop 2023-10-16T10:03:46.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 shutdown_connections 2023-10-16T10:03:46.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fdc6806b0b0 0x7fdc6806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c15b6b0 0x7fdc7c0cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c0b3460 0x7fdc7c151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 --2- 172.21.15.158:0/2814888219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c0d0c60 0x7fdc7c06c3b0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 >> 172.21.15.158:0/2814888219 conn(0x7fdc7c0cdf20 msgr2=0x7fdc7c0b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:46.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 shutdown_connections 2023-10-16T10:03:46.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.576+0000 7fdc81424700 1 -- 172.21.15.158:0/2814888219 wait complete. 2023-10-16T10:03:46.590 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T10:03:46.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.713+0000 7f7cc89f4700 1 Processor -- start 2023-10-16T10:03:46.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.713+0000 7f7cc89f4700 1 -- start start 2023-10-16T10:03:46.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc89f4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7cc406afe0 0x7f7cc415bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc89f4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc406ff10 con 0x7f7cc4072b40 2023-10-16T10:03:46.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc89f4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc406f060 con 0x7f7cc4144e80 2023-10-16T10:03:46.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc89f4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc4145260 con 0x7f7cc406afe0 2023-10-16T10:03:46.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7cc406afe0 0x7f7cc415bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc2d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7cc4072b40 0x7f7cc40cf420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59258/0 (socket says 172.21.15.158:59258) 2023-10-16T10:03:46.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc2d9d700 1 -- 172.21.15.158:0/912256949 learned_addr learned my addr 172.21.15.158:0/912256949 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:46.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc1d9b700 1 -- 172.21.15.158:0/912256949 >> v1:172.21.15.172:6789/0 conn(0x7f7cc4144e80 legacy=0x7f7cc406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc1d9b700 1 -- 172.21.15.158:0/912256949 >> v1:172.21.15.158:6789/0 conn(0x7f7cc4072b40 legacy=0x7f7cc40cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc1d9b700 1 -- 172.21.15.158:0/912256949 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc406f300 con 0x7f7cc406afe0 2023-10-16T10:03:46.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc159a700 1 -- 172.21.15.158:0/912256949 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002149418 0 0) 0x7f7cc406ff10 con 0x7f7cc4072b40 2023-10-16T10:03:46.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc1d9b700 1 --2- 172.21.15.158:0/912256949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc406afe0 0x7f7cc415bc40 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009840 tx=0x7f7cb8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd33b476c64e30ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:46.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc159a700 1 -- 172.21.15.158:0/912256949 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8011830 con 0x7f7cc406afe0 2023-10-16T10:03:46.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc159a700 1 -- 172.21.15.158:0/912256949 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb8011990 con 0x7f7cc406afe0 2023-10-16T10:03:46.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.714+0000 7f7cc159a700 1 -- 172.21.15.158:0/912256949 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8011d20 con 0x7f7cc406afe0 2023-10-16T10:03:46.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.715+0000 7f7cc89f4700 1 -- 172.21.15.158:0/912256949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc406afe0 msgr2=0x7f7cc415bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.715+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/912256949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc406afe0 0x7f7cc415bc40 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009840 tx=0x7f7cb8009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:46.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.715+0000 7f7cc89f4700 1 -- 172.21.15.158:0/912256949 shutdown_connections 2023-10-16T10:03:46.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.715+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/912256949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc406afe0 0x7f7cc415bc40 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.715+0000 7f7cc89f4700 1 -- 172.21.15.158:0/912256949 >> 172.21.15.158:0/912256949 conn(0x7f7cc406da00 msgr2=0x7f7cc406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:46.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.715+0000 7f7cc89f4700 1 -- 172.21.15.158:0/912256949 shutdown_connections 2023-10-16T10:03:46.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.715+0000 7f7cc89f4700 1 -- 172.21.15.158:0/912256949 wait complete. 2023-10-16T10:03:46.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 Processor -- start 2023-10-16T10:03:46.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 -- start start 2023-10-16T10:03:46.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cc406afe0 0x7f7cc4142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cc4072b40 0x7f7cc41e4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc4144e80 0x7f7cc41e5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc41e3e80 con 0x7f7cc4144e80 2023-10-16T10:03:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc4158840 con 0x7f7cc406afe0 2023-10-16T10:03:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc89f4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc41e4400 con 0x7f7cc4072b40 2023-10-16T10:03:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc4144e80 0x7f7cc41e5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc4144e80 0x7f7cc41e5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59086/0 (socket says 172.21.15.158:59086) 2023-10-16T10:03:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.717+0000 7f7cc2d9d700 1 -- 172.21.15.158:0/1207196018 learned_addr learned my addr 172.21.15.158:0/1207196018 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc259c700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cc406afe0 0x7f7cc4142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc1d9b700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cc4072b40 0x7f7cc41e4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc2d9d700 1 -- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cc4072b40 msgr2=0x7f7cc41e4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc2d9d700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cc4072b40 0x7f7cc41e4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc2d9d700 1 -- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cc406afe0 msgr2=0x7f7cc4142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc2d9d700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cc406afe0 0x7f7cc4142df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc2d9d700 1 -- 172.21.15.158:0/1207196018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80094a0 con 0x7f7cc4144e80 2023-10-16T10:03:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc1d9b700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cc4072b40 0x7f7cc41e4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc2d9d700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc4144e80 0x7f7cc41e5280 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00cc10 tx=0x7f7cbc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cb2ffd700 1 -- 172.21.15.158:0/1207196018 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc00ab50 con 0x7f7cc4144e80 2023-10-16T10:03:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cb2ffd700 1 -- 172.21.15.158:0/1207196018 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc00acb0 con 0x7f7cc4144e80 2023-10-16T10:03:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cb2ffd700 1 -- 172.21.15.158:0/1207196018 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc007310 con 0x7f7cc4144e80 2023-10-16T10:03:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc41e5b40 con 0x7f7cc4144e80 2023-10-16T10:03:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.718+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc4143330 con 0x7f7cc4144e80 2023-10-16T10:03:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.719+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc41e4570 con 0x7f7cc4144e80 2023-10-16T10:03:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.720+0000 7f7cb2ffd700 1 -- 172.21.15.158:0/1207196018 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f7cbc007470 con 0x7f7cc4144e80 2023-10-16T10:03:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.721+0000 7f7cb2ffd700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 0x7f7ca406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.721+0000 7f7cc259c700 1 -- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 msgr2=0x7f7ca406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.721+0000 7f7cb2ffd700 1 -- 172.21.15.158:0/1207196018 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f7cbc089ca0 con 0x7f7cc4144e80 2023-10-16T10:03:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.721+0000 7f7cc259c700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 0x7f7ca406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.724+0000 7f7cb2ffd700 1 -- 172.21.15.158:0/1207196018 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7cbc054750 con 0x7f7cc4144e80 2023-10-16T10:03:46.913 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:46.912+0000 7fc1bbd99200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:03:46.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.922+0000 7f7cc259c700 1 -- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 msgr2=0x7f7ca406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:46.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.922+0000 7f7cc259c700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 0x7f7ca406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:46.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.972+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7cc40681b0 con 0x7f7cc4144e80 2023-10-16T10:03:46.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.973+0000 7f7cb2ffd700 1 -- 172.21.15.158:0/1207196018 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v141) v1 ==== 80+0+478 (secure 0 0 0) 0x7f7cbc018030 con 0x7f7cc4144e80 2023-10-16T10:03:46.975 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/2342578387 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.975 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/2342578387 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.975 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3946986955 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.975 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2041489640 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.976 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/4016680871 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.976 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/4198985108 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.976 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1741373068 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.976 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1608841554 2023-10-17T10:03:37.828937+0000 2023-10-16T10:03:46.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.975+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 msgr2=0x7f7ca406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:46.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.975+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 0x7f7ca406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.975+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc4144e80 msgr2=0x7f7cc41e5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:46.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.975+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc4144e80 0x7f7cc41e5280 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00cc10 tx=0x7f7cbc00a060 comp rx=0 tx=0).stop 2023-10-16T10:03:46.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 shutdown_connections 2023-10-16T10:03:46.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f7ca406b0b0 0x7f7ca406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cc4144e80 0x7f7cc41e5280 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cc406afe0 0x7f7cc4142df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 --2- 172.21.15.158:0/1207196018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cc4072b40 0x7f7cc41e4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:46.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 >> 172.21.15.158:0/1207196018 conn(0x7f7cc406da00 msgr2=0x7f7cc406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:46.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 shutdown_connections 2023-10-16T10:03:46.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:46.976+0000 7f7cc89f4700 1 -- 172.21.15.158:0/1207196018 wait complete. 2023-10-16T10:03:46.980 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T10:03:46.982 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:46.981+0000 7fc1bbd99200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:03:46.990 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T10:03:46.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:46.991 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T10:03:47.030 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:47.029+0000 7f88cafcf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:03:47.042 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T10:03:47.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:03:47.043 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T10:03:47.099 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:47.100 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:47.106 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:47.105+0000 7f88cafcf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:03:47.118 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:47.118 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:47.119 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:47.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:03:47.137 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:47.137 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:47.175 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:47.175 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:47.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.302+0000 7fe66a48b700 1 Processor -- start 2023-10-16T10:03:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.302+0000 7fe66a48b700 1 -- start start 2023-10-16T10:03:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe66a48b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe6640cf4c0 0x7fe6640ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe66a48b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe6640cb660 0x7fe6640cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe66a48b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6640ceb10 con 0x7fe66415a340 2023-10-16T10:03:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe66a48b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6640cd600 con 0x7fe6640cb660 2023-10-16T10:03:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe66a48b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6640ce140 con 0x7fe6640cf4c0 2023-10-16T10:03:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe668a28700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe6640cb660 0x7fe6640cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe663fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe66415a340 0x7fe6640cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59270/0 (socket says 172.21.15.158:59270) 2023-10-16T10:03:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe663fff700 1 -- 172.21.15.158:0/884820401 learned_addr learned my addr 172.21.15.158:0/884820401 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe668a28700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe6640cb660 0x7fe6640cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59098/0 (socket says 172.21.15.158:59098) 2023-10-16T10:03:47.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe6637fe700 1 --2- 172.21.15.158:0/884820401 >> v2:172.21.15.172:3300/0 conn(0x7fe6640cf4c0 0x7fe6640ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe668a28700 1 -- 172.21.15.158:0/884820401 >> v2:172.21.15.172:3300/0 conn(0x7fe6640cf4c0 msgr2=0x7fe6640ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:47.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe668a28700 1 --2- 172.21.15.158:0/884820401 >> v2:172.21.15.172:3300/0 conn(0x7fe6640cf4c0 0x7fe6640ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe668a28700 1 -- 172.21.15.158:0/884820401 >> v1:172.21.15.158:6789/0 conn(0x7fe66415a340 legacy=0x7fe6640cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe668a28700 1 -- 172.21.15.158:0/884820401 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6640cd810 con 0x7fe6640cb660 2023-10-16T10:03:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.303+0000 7fe662ffd700 1 -- 172.21.15.158:0/884820401 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204456977 0 0) 0x7fe654001110 con 0x7fe66415a340 2023-10-16T10:03:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe668a28700 1 --2- 172.21.15.158:0/884820401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6640cb660 0x7fe6640cbad0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fe6640b1610 tx=0x7fe65c00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fed43998468226 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe662ffd700 1 -- 172.21.15.158:0/884820401 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c010070 con 0x7fe6640cb660 2023-10-16T10:03:47.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe662ffd700 1 -- 172.21.15.158:0/884820401 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c00b2e0 con 0x7fe6640cb660 2023-10-16T10:03:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe662ffd700 1 -- 172.21.15.158:0/884820401 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00dc10 con 0x7fe6640cb660 2023-10-16T10:03:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 -- 172.21.15.158:0/884820401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6640cb660 msgr2=0x7fe6640cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 --2- 172.21.15.158:0/884820401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6640cb660 0x7fe6640cbad0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fe6640b1610 tx=0x7fe65c00d330 comp rx=0 tx=0).stop 2023-10-16T10:03:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 -- 172.21.15.158:0/884820401 shutdown_connections 2023-10-16T10:03:47.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 --2- 172.21.15.158:0/884820401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6640cb660 0x7fe6640cbad0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 --2- 172.21.15.158:0/884820401 >> v2:172.21.15.172:3300/0 conn(0x7fe6640cf4c0 0x7fe6640ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 -- 172.21.15.158:0/884820401 >> 172.21.15.158:0/884820401 conn(0x7fe6640cc4b0 msgr2=0x7fe6640cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 -- 172.21.15.158:0/884820401 shutdown_connections 2023-10-16T10:03:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.304+0000 7fe66a48b700 1 -- 172.21.15.158:0/884820401 wait complete. 2023-10-16T10:03:47.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.306+0000 7fe66a48b700 1 Processor -- start 2023-10-16T10:03:47.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.306+0000 7fe66a48b700 1 -- start start 2023-10-16T10:03:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.306+0000 7fe66a48b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6640cb660 0x7fe664142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.306+0000 7fe66a48b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6640cf4c0 0x7fe6640af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe66a48b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe66415a340 0x7fe6640afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe66a48b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe664142c50 con 0x7fe66415a340 2023-10-16T10:03:47.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe66a48b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6641efa20 con 0x7fe6640cb660 2023-10-16T10:03:47.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe66a48b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6641f2150 con 0x7fe6640cf4c0 2023-10-16T10:03:47.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe66415a340 0x7fe6640afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe66415a340 0x7fe6640afac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59104/0 (socket says 172.21.15.158:59104) 2023-10-16T10:03:47.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 -- 172.21.15.158:0/1821632365 learned_addr learned my addr 172.21.15.158:0/1821632365 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:47.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe663fff700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6640cb660 0x7fe664142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe6637fe700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6640cf4c0 0x7fe6640af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 -- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6640cf4c0 msgr2=0x7fe6640af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6640cf4c0 0x7fe6640af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 -- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6640cb660 msgr2=0x7fe664142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6640cb660 0x7fe664142710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 -- 172.21.15.158:0/1821632365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c00bc80 con 0x7fe66415a340 2023-10-16T10:03:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe663fff700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6640cb660 0x7fe664142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:47.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe668a28700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe66415a340 0x7fe6640afac0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00d680 tx=0x7fe65c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe660ff9700 1 -- 172.21.15.158:0/1821632365 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00b450 con 0x7fe66415a340 2023-10-16T10:03:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe660ff9700 1 -- 172.21.15.158:0/1821632365 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c004250 con 0x7fe66415a340 2023-10-16T10:03:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6641f2430 con 0x7fe66415a340 2023-10-16T10:03:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe660ff9700 1 -- 172.21.15.158:0/1821632365 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0045e0 con 0x7fe66415a340 2023-10-16T10:03:47.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.307+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6641f29f0 con 0x7fe66415a340 2023-10-16T10:03:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.308+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6640681b0 con 0x7fe66415a340 2023-10-16T10:03:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.309+0000 7fe660ff9700 1 -- 172.21.15.158:0/1821632365 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fe65c004740 con 0x7fe66415a340 2023-10-16T10:03:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.309+0000 7fe660ff9700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 0x7fe65006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.309+0000 7fe663fff700 1 -- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 msgr2=0x7fe65006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:47.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.310+0000 7fe660ff9700 1 -- 172.21.15.158:0/1821632365 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fe65c08d480 con 0x7fe66415a340 2023-10-16T10:03:47.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.310+0000 7fe663fff700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 0x7fe65006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:47.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.314+0000 7fe660ff9700 1 -- 172.21.15.158:0/1821632365 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe65c057dd0 con 0x7fe66415a340 2023-10-16T10:03:47.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.511+0000 7fe663fff700 1 -- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 msgr2=0x7fe65006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:47.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.511+0000 7fe663fff700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 0x7fe65006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:47.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.558+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6640c5380 con 0x7fe66415a340 2023-10-16T10:03:47.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.559+0000 7fe660ff9700 1 -- 172.21.15.158:0/1821632365 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fe65c05b3f0 con 0x7fe66415a340 2023-10-16T10:03:47.560 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:47.561 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4144904682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4144904682},{"type":"v1","addr":"172.21.15.172:6834","nonce":4144904682}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3562807630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3562807630},{"type":"v1","addr":"172.21.15.158:6836","nonce":3562807630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8805,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/614819185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":614819185},{"type":"v1","addr":"172.21.15.158:6837","nonce":614819185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:03:23.609567+0000","modified":"2023-10-16T10:03:26.633779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/4142092863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4142092863},{"type":"v1","addr":"172.21.15.172:6835","nonce":4142092863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T10:03:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 msgr2=0x7fe65006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 0x7fe65006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe66415a340 msgr2=0x7fe6640afac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe66415a340 0x7fe6640afac0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00d680 tx=0x7fe65c004ad0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 shutdown_connections 2023-10-16T10:03:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fe65006b0b0 0x7fe65006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe66415a340 0x7fe6640afac0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6640cb660 0x7fe664142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 --2- 172.21.15.158:0/1821632365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6640cf4c0 0x7fe6640af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 >> 172.21.15.158:0/1821632365 conn(0x7fe6640cc4b0 msgr2=0x7fe6640c48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 shutdown_connections 2023-10-16T10:03:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.561+0000 7fe66a48b700 1 -- 172.21.15.158:0/1821632365 wait complete. 2023-10-16T10:03:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 68 2023-10-16T10:03:47.576 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:47.622 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:47.621+0000 7fc1bbd99200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:47.702 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:47.701+0000 7fc1bbd99200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:03:47.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.706+0000 7f01f396c700 1 Processor -- start 2023-10-16T10:03:47.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.706+0000 7f01f396c700 1 -- start start 2023-10-16T10:03:47.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f396c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f01ec0ce530 0x7f01ec0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f396c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f01ec15f0f0 0x7f01ec0cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f396c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec15b930 con 0x7f01ec0cbfe0 2023-10-16T10:03:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f396c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0cf020 con 0x7f01ec0ce530 2023-10-16T10:03:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f396c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0cc450 con 0x7f01ec15f0f0 2023-10-16T10:03:47.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f01ec0ce530 0x7f01ec0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f01ec0ce530 0x7f01ec0b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59110/0 (socket says 172.21.15.158:59110) 2023-10-16T10:03:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 -- 172.21.15.158:0/2069974309 learned_addr learned my addr 172.21.15.158:0/2069974309 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f0f07700 1 --2- 172.21.15.158:0/2069974309 >> v2:172.21.15.172:3300/0 conn(0x7f01ec15f0f0 0x7f01ec0cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 -- 172.21.15.158:0/2069974309 >> v2:172.21.15.172:3300/0 conn(0x7f01ec15f0f0 msgr2=0x7f01ec0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:47.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 --2- 172.21.15.158:0/2069974309 >> v2:172.21.15.172:3300/0 conn(0x7f01ec15f0f0 0x7f01ec0cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 -- 172.21.15.158:0/2069974309 >> v1:172.21.15.172:6789/0 conn(0x7f01ec0cbfe0 legacy=0x7f01ec0d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 -- 172.21.15.158:0/2069974309 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec15f5f0 con 0x7f01ec0ce530 2023-10-16T10:03:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.707+0000 7f01f1708700 1 --2- 172.21.15.158:0/2069974309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec0ce530 0x7f01ec0b3b20 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0cef50 tx=0x7f01e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2ec0034fc8b085 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01e3fff700 1 -- 172.21.15.158:0/2069974309 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e8011830 con 0x7f01ec0ce530 2023-10-16T10:03:47.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01e3fff700 1 -- 172.21.15.158:0/2069974309 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e8011990 con 0x7f01ec0ce530 2023-10-16T10:03:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01e3fff700 1 -- 172.21.15.158:0/2069974309 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e8011d20 con 0x7f01ec0ce530 2023-10-16T10:03:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 -- 172.21.15.158:0/2069974309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec0ce530 msgr2=0x7f01ec0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 --2- 172.21.15.158:0/2069974309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec0ce530 0x7f01ec0b3b20 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0cef50 tx=0x7f01e8009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 -- 172.21.15.158:0/2069974309 shutdown_connections 2023-10-16T10:03:47.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 --2- 172.21.15.158:0/2069974309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec0ce530 0x7f01ec0b3b20 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 --2- 172.21.15.158:0/2069974309 >> v2:172.21.15.172:3300/0 conn(0x7f01ec15f0f0 0x7f01ec0cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 -- 172.21.15.158:0/2069974309 >> 172.21.15.158:0/2069974309 conn(0x7f01ec06a8d0 msgr2=0x7f01ec06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 -- 172.21.15.158:0/2069974309 shutdown_connections 2023-10-16T10:03:47.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.708+0000 7f01f396c700 1 -- 172.21.15.158:0/2069974309 wait complete. 2023-10-16T10:03:47.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.709+0000 7f01f396c700 1 Processor -- start 2023-10-16T10:03:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f396c700 1 -- start start 2023-10-16T10:03:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f396c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01ec0cbfe0 0x7f01ec0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f396c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01ec0ce530 0x7f01ec0c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f396c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec15f0f0 0x7f01ec1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f396c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0c6120 con 0x7f01ec15f0f0 2023-10-16T10:03:47.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f396c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0c7e10 con 0x7f01ec0cbfe0 2023-10-16T10:03:47.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f396c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec158cf0 con 0x7f01ec0ce530 2023-10-16T10:03:47.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f1f09700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec15f0f0 0x7f01ec1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f1f09700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec15f0f0 0x7f01ec1597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59120/0 (socket says 172.21.15.158:59120) 2023-10-16T10:03:47.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f1f09700 1 -- 172.21.15.158:0/2429519217 learned_addr learned my addr 172.21.15.158:0/2429519217 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:47.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f1708700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01ec0cbfe0 0x7f01ec0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.710+0000 7f01f0f07700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01ec0ce530 0x7f01ec0c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:47.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f1f09700 1 -- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01ec0ce530 msgr2=0x7f01ec0c3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f1f09700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01ec0ce530 0x7f01ec0c3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f1f09700 1 -- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01ec0cbfe0 msgr2=0x7f01ec0c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f1f09700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01ec0cbfe0 0x7f01ec0c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f1f09700 1 -- 172.21.15.158:0/2429519217 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8009550 con 0x7f01ec15f0f0 2023-10-16T10:03:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f1f09700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec15f0f0 0x7f01ec1597a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f01e400cb50 tx=0x7f01e400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:47.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01e1ffb700 1 -- 172.21.15.158:0/2429519217 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4003c00 con 0x7f01ec15f0f0 2023-10-16T10:03:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01e1ffb700 1 -- 172.21.15.158:0/2429519217 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e4003d60 con 0x7f01ec15f0f0 2023-10-16T10:03:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01e1ffb700 1 -- 172.21.15.158:0/2429519217 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e400aa20 con 0x7f01ec15f0f0 2023-10-16T10:03:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec0c5780 con 0x7f01ec15f0f0 2023-10-16T10:03:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.711+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec0c5d40 con 0x7f01ec15f0f0 2023-10-16T10:03:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.712+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec0c6260 con 0x7f01ec15f0f0 2023-10-16T10:03:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.712+0000 7f01e1ffb700 1 -- 172.21.15.158:0/2429519217 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f01e400ab80 con 0x7f01ec15f0f0 2023-10-16T10:03:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.713+0000 7f01e1ffb700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 0x7f01d006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.713+0000 7f01f1708700 1 -- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 msgr2=0x7f01d006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.713+0000 7f01e1ffb700 1 -- 172.21.15.158:0/2429519217 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f01e4089d20 con 0x7f01ec15f0f0 2023-10-16T10:03:47.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.713+0000 7f01f1708700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 0x7f01d006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:47.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.717+0000 7f01e1ffb700 1 -- 172.21.15.158:0/2429519217 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f01ec0c6260 con 0x7f01ec15f0f0 2023-10-16T10:03:47.786 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:47.785+0000 7f88cafcf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:47.847 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:47.846+0000 7fc1bbd99200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:47.867 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:47.866+0000 7f88cafcf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:03:47.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.914+0000 7f01f1708700 1 -- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 msgr2=0x7f01d006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:47.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.914+0000 7f01f1708700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 0x7f01d006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:47.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.918+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01ec0c6260 con 0x7f01ec15f0f0 2023-10-16T10:03:47.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.920+0000 7f01e1ffb700 1 -- 172.21.15.158:0/2429519217 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17995 (secure 0 0 0) 0x7f01ec0c6260 con 0x7f01ec15f0f0 2023-10-16T10:03:47.921 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:47.922 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":141,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:37.828969+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:03:21.520809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T10:03:22.576358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:6832/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/3946986955":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/2041489640":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4016680871":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4198985108":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1741373068":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1608841554":"2023-10-17T10:03:37.828937+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:47.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 msgr2=0x7f01d006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:47.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 0x7f01d006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec15f0f0 msgr2=0x7f01ec1597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:47.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec15f0f0 0x7f01ec1597a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f01e400cb50 tx=0x7f01e400ce60 comp rx=0 tx=0).stop 2023-10-16T10:03:47.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 shutdown_connections 2023-10-16T10:03:47.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f01d006b0b0 0x7f01d006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01ec15f0f0 0x7f01ec1597a0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01ec0cbfe0 0x7f01ec0c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 --2- 172.21.15.158:0/2429519217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01ec0ce530 0x7f01ec0c3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:47.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 >> 172.21.15.158:0/2429519217 conn(0x7f01ec06a8d0 msgr2=0x7f01ec0b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:47.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 shutdown_connections 2023-10-16T10:03:47.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:47.922+0000 7f01f396c700 1 -- 172.21.15.158:0/2429519217 wait complete. 2023-10-16T10:03:47.937 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:03:21.520809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:03:22.576358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:03:47.938 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:03:48.022 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:48.021+0000 7f88cafcf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:03:48.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.063+0000 7f7a13131700 1 Processor -- start 2023-10-16T10:03:48.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.063+0000 7f7a13131700 1 -- start start 2023-10-16T10:03:48.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a13131700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a0c15b6b0 0x7f7a0c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a13131700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7a0c0d0c60 0x7f7a0c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a13131700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c0b4420 con 0x7f7a0c0b3460 2023-10-16T10:03:48.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a13131700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0c15ba90 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a13131700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0c0b38d0 con 0x7f7a0c0d0c60 2023-10-16T10:03:48.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a10ecd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a0c15b6b0 0x7f7a0c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a116ce700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7a0c0b3460 0x7f7a0c070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59324/0 (socket says 172.21.15.158:59324) 2023-10-16T10:03:48.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a10ecd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a0c15b6b0 0x7f7a0c0cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59144/0 (socket says 172.21.15.158:59144) 2023-10-16T10:03:48.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a116ce700 1 -- 172.21.15.158:0/3333176838 learned_addr learned my addr 172.21.15.158:0/3333176838 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:48.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a0bfff700 1 --2- 172.21.15.158:0/3333176838 >> v2:172.21.15.172:3300/0 conn(0x7f7a0c0d0c60 0x7f7a0c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a10ecd700 1 -- 172.21.15.158:0/3333176838 >> v2:172.21.15.172:3300/0 conn(0x7f7a0c0d0c60 msgr2=0x7f7a0c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a10ecd700 1 --2- 172.21.15.158:0/3333176838 >> v2:172.21.15.172:3300/0 conn(0x7f7a0c0d0c60 0x7f7a0c0b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a10ecd700 1 -- 172.21.15.158:0/3333176838 >> v1:172.21.15.158:6789/0 conn(0x7f7a0c0b3460 legacy=0x7f7a0c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a10ecd700 1 -- 172.21.15.158:0/3333176838 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c0cbc20 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.064+0000 7f7a0b7fe700 1 -- 172.21.15.158:0/3333176838 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429854162 0 0) 0x7f7a0c0b4420 con 0x7f7a0c0b3460 2023-10-16T10:03:48.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a10ecd700 1 --2- 172.21.15.158:0/3333176838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c0cfdb0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0b7610 tx=0x7f7a00009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8996cbc1f0152a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:48.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a0b7fe700 1 -- 172.21.15.158:0/3333176838 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00011830 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a0b7fe700 1 -- 172.21.15.158:0/3333176838 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a00011990 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a0b7fe700 1 -- 172.21.15.158:0/3333176838 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00011d20 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 -- 172.21.15.158:0/3333176838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 msgr2=0x7f7a0c0cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 --2- 172.21.15.158:0/3333176838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c0cfdb0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0b7610 tx=0x7f7a00009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:48.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 -- 172.21.15.158:0/3333176838 shutdown_connections 2023-10-16T10:03:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 --2- 172.21.15.158:0/3333176838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c0cfdb0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 --2- 172.21.15.158:0/3333176838 >> v2:172.21.15.172:3300/0 conn(0x7f7a0c0d0c60 0x7f7a0c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 -- 172.21.15.158:0/3333176838 >> 172.21.15.158:0/3333176838 conn(0x7f7a0c0cdf20 msgr2=0x7f7a0c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 -- 172.21.15.158:0/3333176838 shutdown_connections 2023-10-16T10:03:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.065+0000 7f7a13131700 1 -- 172.21.15.158:0/3333176838 wait complete. 2023-10-16T10:03:48.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 Processor -- start 2023-10-16T10:03:48.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 -- start start 2023-10-16T10:03:48.076 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:48.074+0000 7fc1bbd99200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:03:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0b3460 0x7f7a0c1f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0d0c60 0x7f7a0c152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c06d480 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c15cc80 con 0x7f7a0c0b3460 2023-10-16T10:03:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.067+0000 7f7a13131700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c15cf60 con 0x7f7a0c0d0c60 2023-10-16T10:03:48.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59154/0 (socket says 172.21.15.158:59154) 2023-10-16T10:03:48.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 -- 172.21.15.158:0/640709327 learned_addr learned my addr 172.21.15.158:0/640709327 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:48.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a10ecd700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0b3460 0x7f7a0c1f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a0bfff700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0d0c60 0x7f7a0c152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 -- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0d0c60 msgr2=0x7f7a0c152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0d0c60 0x7f7a0c152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 -- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0b3460 msgr2=0x7f7a0c1f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0b3460 0x7f7a0c1f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 -- 172.21.15.158:0/640709327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a000094a0 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a10ecd700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0b3460 0x7f7a0c1f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:48.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a116ce700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c152660 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400cb60 tx=0x7f7a0400ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:48.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a097fa700 1 -- 172.21.15.158:0/640709327 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0400a9e0 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a097fa700 1 -- 172.21.15.158:0/640709327 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0400ab40 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0c15d240 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a097fa700 1 -- 172.21.15.158:0/640709327 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a040070d0 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.068+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0c1f2930 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.069+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0c15d760 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.070+0000 7f7a097fa700 1 -- 172.21.15.158:0/640709327 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f7a04007230 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.071+0000 7f7a097fa700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 0x7f79f006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.071+0000 7f7a10ecd700 1 -- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 msgr2=0x7f79f006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.071+0000 7f7a097fa700 1 -- 172.21.15.158:0/640709327 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f7a040896f0 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.071+0000 7f7a10ecd700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 0x7f79f006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:48.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.075+0000 7f7a097fa700 1 -- 172.21.15.158:0/640709327 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a04054150 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.149 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:48.148+0000 7fc1bbd99200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:03:48.260 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:48.259+0000 7f88cafcf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:03:48.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.272+0000 7f7a10ecd700 1 -- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 msgr2=0x7f79f006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:48.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.272+0000 7f7a10ecd700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 0x7f79f006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:48.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.273+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a0c0681b0 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.273+0000 7f7a097fa700 1 -- 172.21.15.158:0/640709327 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a04018020 con 0x7f7a0c15b6b0 2023-10-16T10:03:48.275 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:03:48.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 msgr2=0x7f79f006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:48.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 0x7f79f006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 msgr2=0x7f7a0c152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c152660 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400cb60 tx=0x7f7a0400ce70 comp rx=0 tx=0).stop 2023-10-16T10:03:48.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 shutdown_connections 2023-10-16T10:03:48.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f79f006b0b0 0x7f79f006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a0c15b6b0 0x7f7a0c152660 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0b3460 0x7f7a0c1f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 --2- 172.21.15.158:0/640709327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0d0c60 0x7f7a0c152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 >> 172.21.15.158:0/640709327 conn(0x7f7a0c0cdf20 msgr2=0x7f7a0c0cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:48.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 shutdown_connections 2023-10-16T10:03:48.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.276+0000 7f7a13131700 1 -- 172.21.15.158:0/640709327 wait complete. 2023-10-16T10:03:48.290 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T10:03:48.352 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:48.351+0000 7f88cafcf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:03:48.414 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:48.413+0000 7fc1bbd99200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:03:48.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.429+0000 7f3114c47700 1 Processor -- start 2023-10-16T10:03:48.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.429+0000 7f3114c47700 1 -- start start 2023-10-16T10:03:48.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f3114c47700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f3114c47700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f311006de40 0x7f311006f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f3114c47700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311015f380 con 0x7f311006d9c0 2023-10-16T10:03:48.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f3114c47700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311006e2d0 con 0x7f311006de40 2023-10-16T10:03:48.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f3114c47700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3110071e70 con 0x7f311015dc40 2023-10-16T10:03:48.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f310dd9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f311006d9c0 0x7f311006f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59336/0 (socket says 172.21.15.158:59336) 2023-10-16T10:03:48.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f310dd9b700 1 -- 172.21.15.158:0/3823799603 learned_addr learned my addr 172.21.15.158:0/3823799603 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:48.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f310ed9d700 1 --2- 172.21.15.158:0/3823799603 >> v2:172.21.15.172:3300/0 conn(0x7f311006de40 0x7f311006f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f310e59c700 1 --2- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f310d59a700 1 -- 172.21.15.158:0/3823799603 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301704341 0 0) 0x7f3104001110 con 0x7f311006d9c0 2023-10-16T10:03:48.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.430+0000 7f310d59a700 1 -- 172.21.15.158:0/3823799603 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f80035b0 con 0x7f311006d9c0 2023-10-16T10:03:48.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310ed9d700 1 -- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 msgr2=0x7f311015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:48.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310ed9d700 1 --2- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311015ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310ed9d700 1 -- 172.21.15.158:0/3823799603 >> v1:172.21.15.158:6789/0 conn(0x7f311006d9c0 legacy=0x7f311006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310ed9d700 1 -- 172.21.15.158:0/3823799603 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110072150 con 0x7f311006de40 2023-10-16T10:03:48.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310e59c700 1 --2- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311015ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:48.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310ed9d700 1 --2- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006f890 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3108009d90 tx=0x7f310800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39767638babe15b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:48.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310d59a700 1 -- 172.21.15.158:0/3823799603 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310800e070 con 0x7f311006de40 2023-10-16T10:03:48.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310d59a700 1 -- 172.21.15.158:0/3823799603 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310800bb20 con 0x7f311006de40 2023-10-16T10:03:48.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.431+0000 7f310d59a700 1 -- 172.21.15.158:0/3823799603 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108012430 con 0x7f311006de40 2023-10-16T10:03:48.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 -- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 msgr2=0x7f311006f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 --2- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006f890 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3108009d90 tx=0x7f310800b330 comp rx=0 tx=0).stop 2023-10-16T10:03:48.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 -- 172.21.15.158:0/3823799603 shutdown_connections 2023-10-16T10:03:48.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 --2- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006f890 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 --2- 172.21.15.158:0/3823799603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311015ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 -- 172.21.15.158:0/3823799603 >> 172.21.15.158:0/3823799603 conn(0x7f31100d03b0 msgr2=0x7f31100d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:48.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 -- 172.21.15.158:0/3823799603 shutdown_connections 2023-10-16T10:03:48.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.432+0000 7f3114c47700 1 -- 172.21.15.158:0/3823799603 wait complete. 2023-10-16T10:03:48.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.433+0000 7f3114c47700 1 Processor -- start 2023-10-16T10:03:48.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.433+0000 7f3114c47700 1 -- start start 2023-10-16T10:03:48.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f3114c47700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311006d9c0 0x7f31100ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f3114c47700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f3114c47700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f3114c47700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31101513c0 con 0x7f311006d9c0 2023-10-16T10:03:48.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f3114c47700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100ad910 con 0x7f311006de40 2023-10-16T10:03:48.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f3114c47700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31100acd90 con 0x7f311015dc40 2023-10-16T10:03:48.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311006d9c0 0x7f31100ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311006d9c0 0x7f31100ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59166/0 (socket says 172.21.15.158:59166) 2023-10-16T10:03:48.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310e59c700 1 -- 172.21.15.158:0/2971652693 learned_addr learned my addr 172.21.15.158:0/2971652693 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:48.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310dd9b700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310ed9d700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310dd9b700 1 -- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 msgr2=0x7f311006ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310dd9b700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311006ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310dd9b700 1 -- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311006d9c0 msgr2=0x7f31100ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310dd9b700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311006d9c0 0x7f31100ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310dd9b700 1 -- 172.21.15.158:0/2971652693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31000094a0 con 0x7f311006de40 2023-10-16T10:03:48.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.434+0000 7f310e59c700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311006d9c0 0x7f31100ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:48.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.435+0000 7f310dd9b700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006a8d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f310400c730 tx=0x7f310400ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:48.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.435+0000 7f30feffd700 1 -- 172.21.15.158:0/2971652693 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104005d70 con 0x7f311006de40 2023-10-16T10:03:48.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.435+0000 7f30feffd700 1 -- 172.21.15.158:0/2971652693 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31040070d0 con 0x7f311006de40 2023-10-16T10:03:48.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.435+0000 7f30feffd700 1 -- 172.21.15.158:0/2971652693 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104007460 con 0x7f311006de40 2023-10-16T10:03:48.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.435+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3108009a70 con 0x7f311006de40 2023-10-16T10:03:48.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.435+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100ad350 con 0x7f311006de40 2023-10-16T10:03:48.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.436+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31100681b0 con 0x7f311006de40 2023-10-16T10:03:48.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.436+0000 7f30feffd700 1 -- 172.21.15.158:0/2971652693 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f31040075c0 con 0x7f311006de40 2023-10-16T10:03:48.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.437+0000 7f30feffd700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 0x7f30f006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.437+0000 7f310e59c700 1 -- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 msgr2=0x7f30f006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:48.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.437+0000 7f30feffd700 1 -- 172.21.15.158:0/2971652693 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f3104088bf0 con 0x7f311006de40 2023-10-16T10:03:48.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.437+0000 7f310e59c700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 0x7f30f006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:48.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.441+0000 7f30feffd700 1 -- 172.21.15.158:0/2971652693 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31040535b0 con 0x7f311006de40 2023-10-16T10:03:48.630 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:48.628+0000 7f88cafcf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:03:48.638 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:48.637+0000 7fc1bbd99200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:03:48.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.637+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f311015f380 con 0x7f311006de40 2023-10-16T10:03:48.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.638+0000 7f310e59c700 1 -- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 msgr2=0x7f30f006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:48.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.638+0000 7f310e59c700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 0x7f30f006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:48.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.638+0000 7f30feffd700 1 -- 172.21.15.158:0/2971652693 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7f311015f380 con 0x7f311006de40 2023-10-16T10:03:48.640 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T10:03:48.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 msgr2=0x7f30f006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:48.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 0x7f30f006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 msgr2=0x7f311006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006a8d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f310400c730 tx=0x7f310400ca40 comp rx=0 tx=0).stop 2023-10-16T10:03:48.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 shutdown_connections 2023-10-16T10:03:48.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f30f006b0b0 0x7f30f006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f311015dc40 0x7f311006ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f311006d9c0 0x7f31100ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 --2- 172.21.15.158:0/2971652693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f311006de40 0x7f311006a8d0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 >> 172.21.15.158:0/2971652693 conn(0x7f31100d03b0 msgr2=0x7f31100afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:48.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 shutdown_connections 2023-10-16T10:03:48.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.640+0000 7f3114c47700 1 -- 172.21.15.158:0/2971652693 wait complete. 2023-10-16T10:03:48.654 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T10:03:48.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.781+0000 7ff82a4e6700 1 Processor -- start 2023-10-16T10:03:48.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.781+0000 7ff82a4e6700 1 -- start start 2023-10-16T10:03:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.781+0000 7ff82a4e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff82415b590 0x7ff8240ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.781+0000 7ff82a4e6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff82415f1d0 0x7ff8240b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.781+0000 7ff82a4e6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240cf4c0 con 0x7ff8240cbaf0 2023-10-16T10:03:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.781+0000 7ff82a4e6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82415f970 con 0x7ff82415b590 2023-10-16T10:03:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.781+0000 7ff82a4e6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82415fc50 con 0x7ff82415f1d0 2023-10-16T10:03:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff82415b590 0x7ff8240ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff82415b590 0x7ff8240ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59176/0 (socket says 172.21.15.158:59176) 2023-10-16T10:03:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 -- 172.21.15.158:0/1714406247 learned_addr learned my addr 172.21.15.158:0/1714406247 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff8237fe700 1 --2- 172.21.15.158:0/1714406247 >> v2:172.21.15.172:3300/0 conn(0x7ff82415f1d0 0x7ff8240b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 -- 172.21.15.158:0/1714406247 >> v2:172.21.15.172:3300/0 conn(0x7ff82415f1d0 msgr2=0x7ff8240b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 --2- 172.21.15.158:0/1714406247 >> v2:172.21.15.172:3300/0 conn(0x7ff82415f1d0 0x7ff8240b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 -- 172.21.15.158:0/1714406247 >> v1:172.21.15.172:6789/0 conn(0x7ff8240cbaf0 legacy=0x7ff8240cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 -- 172.21.15.158:0/1714406247 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82415ff30 con 0x7ff82415b590 2023-10-16T10:03:48.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff823fff700 1 --2- 172.21.15.158:0/1714406247 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240ce510 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7ff8240d1360 tx=0x7ff814009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c582542dc8c2f8bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:48.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff822ffd700 1 -- 172.21.15.158:0/1714406247 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff814011830 con 0x7ff82415b590 2023-10-16T10:03:48.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff822ffd700 1 -- 172.21.15.158:0/1714406247 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff814011990 con 0x7ff82415b590 2023-10-16T10:03:48.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.782+0000 7ff822ffd700 1 -- 172.21.15.158:0/1714406247 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff814011d20 con 0x7ff82415b590 2023-10-16T10:03:48.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 -- 172.21.15.158:0/1714406247 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 msgr2=0x7ff8240ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/1714406247 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240ce510 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7ff8240d1360 tx=0x7ff814009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 -- 172.21.15.158:0/1714406247 shutdown_connections 2023-10-16T10:03:48.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/1714406247 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240ce510 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/1714406247 >> v2:172.21.15.172:3300/0 conn(0x7ff82415f1d0 0x7ff8240b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 -- 172.21.15.158:0/1714406247 >> 172.21.15.158:0/1714406247 conn(0x7ff82406a8d0 msgr2=0x7ff82406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:48.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 -- 172.21.15.158:0/1714406247 shutdown_connections 2023-10-16T10:03:48.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.783+0000 7ff82a4e6700 1 -- 172.21.15.158:0/1714406247 wait complete. 2023-10-16T10:03:48.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.784+0000 7ff82a4e6700 1 Processor -- start 2023-10-16T10:03:48.791 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:48.785+0000 7fc1bbd99200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:03:48.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.784+0000 7ff82a4e6700 1 -- start start 2023-10-16T10:03:48.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff82a4e6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8240cbaf0 0x7ff8240c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff82a4e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff82a4e6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff82415f1d0 0x7ff82415db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff82a4e6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8240c6020 con 0x7ff82415b590 2023-10-16T10:03:48.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff82a4e6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8240c8b90 con 0x7ff82415f1d0 2023-10-16T10:03:48.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff82a4e6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8241f2e50 con 0x7ff8240cbaf0 2023-10-16T10:03:48.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59182/0 (socket says 172.21.15.158:59182) 2023-10-16T10:03:48.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 -- 172.21.15.158:0/2895614258 learned_addr learned my addr 172.21.15.158:0/2895614258 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:48.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff823fff700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8240cbaf0 0x7ff8240c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff828a83700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff82415f1d0 0x7ff82415db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:48.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 -- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8240cbaf0 msgr2=0x7ff8240c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8240cbaf0 0x7ff8240c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 -- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff82415f1d0 msgr2=0x7ff82415db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff82415f1d0 0x7ff82415db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.785+0000 7ff8237fe700 1 -- 172.21.15.158:0/2895614258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff814009550 con 0x7ff82415b590 2023-10-16T10:03:48.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.786+0000 7ff8237fe700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240c8650 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e870 tx=0x7ff81800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:48.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.786+0000 7ff820ff9700 1 -- 172.21.15.158:0/2895614258 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818009590 con 0x7ff82415b590 2023-10-16T10:03:48.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.786+0000 7ff820ff9700 1 -- 172.21.15.158:0/2895614258 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff818013680 con 0x7ff82415b590 2023-10-16T10:03:48.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.786+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8240c6270 con 0x7ff82415b590 2023-10-16T10:03:48.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.786+0000 7ff820ff9700 1 -- 172.21.15.158:0/2895614258 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8180139f0 con 0x7ff82415b590 2023-10-16T10:03:48.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.786+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8240c6830 con 0x7ff82415b590 2023-10-16T10:03:48.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.787+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8240681b0 con 0x7ff82415b590 2023-10-16T10:03:48.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.787+0000 7ff820ff9700 1 -- 172.21.15.158:0/2895614258 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7ff818013b50 con 0x7ff82415b590 2023-10-16T10:03:48.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.788+0000 7ff820ff9700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 0x7ff81006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:48.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.788+0000 7ff823fff700 1 -- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 msgr2=0x7ff81006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:48.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.788+0000 7ff820ff9700 1 -- 172.21.15.158:0/2895614258 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7ff818089c40 con 0x7ff82415b590 2023-10-16T10:03:48.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.788+0000 7ff823fff700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 0x7ff81006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:48.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.792+0000 7ff820ff9700 1 -- 172.21.15.158:0/2895614258 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff818054620 con 0x7ff82415b590 2023-10-16T10:03:48.871 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:48.870+0000 7f88cafcf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:03:48.902 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:48.900+0000 7fc1bbd99200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:03:48.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.989+0000 7ff823fff700 1 -- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 msgr2=0x7ff81006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:48.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.989+0000 7ff823fff700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 0x7ff81006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:48.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.992+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff8240c7ae0 con 0x7ff82415b590 2023-10-16T10:03:48.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.992+0000 7ff820ff9700 1 -- 172.21.15.158:0/2895614258 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff81805e070 con 0x7ff82415b590 2023-10-16T10:03:48.994 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T10:03:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 msgr2=0x7ff81006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 0x7ff81006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 msgr2=0x7ff8240c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:48.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240c8650 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e870 tx=0x7ff81800eb80 comp rx=0 tx=0).stop 2023-10-16T10:03:48.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 shutdown_connections 2023-10-16T10:03:48.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7ff81006b0b0 0x7ff81006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff82415f1d0 0x7ff82415db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8240cbaf0 0x7ff8240c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 --2- 172.21.15.158:0/2895614258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff82415b590 0x7ff8240c8650 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.994+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 >> 172.21.15.158:0/2895614258 conn(0x7ff82406a8d0 msgr2=0x7ff8240b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.995+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 shutdown_connections 2023-10-16T10:03:48.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:48.995+0000 7ff82a4e6700 1 -- 172.21.15.158:0/2895614258 wait complete. 2023-10-16T10:03:49.009 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:49.030 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:49.029+0000 7f88cafcf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:03:49.057 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:49.055+0000 7fc1bbd99200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:03:49.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.132+0000 7f8f96d28700 1 Processor -- start 2023-10-16T10:03:49.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.132+0000 7f8f96d28700 1 -- start start 2023-10-16T10:03:49.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f96d28700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9015daa0 0x7f8f9006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f96d28700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f900d0bd0 0x7f8f900d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f96d28700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8f900d3570 0x7f8f900d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f96d28700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f900ce4d0 con 0x7f8f900d0bd0 2023-10-16T10:03:49.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f96d28700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f900d3f00 con 0x7f8f900d3570 2023-10-16T10:03:49.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f96d28700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9006c1b0 con 0x7f8f9015daa0 2023-10-16T10:03:49.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f8ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f900d0bd0 0x7f8f900d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f8ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f900d0bd0 0x7f8f900d1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59192/0 (socket says 172.21.15.158:59192) 2023-10-16T10:03:49.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.133+0000 7f8f8ffff700 1 -- 172.21.15.158:0/1172671300 learned_addr learned my addr 172.21.15.158:0/1172671300 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:49.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f94ac4700 1 --2- 172.21.15.158:0/1172671300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9015daa0 0x7f8f9006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f952c5700 1 --2- 172.21.15.158:0/1172671300 >> v2:172.21.15.172:3300/0 conn(0x7f8f900d3570 0x7f8f900d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8ffff700 1 -- 172.21.15.158:0/1172671300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9015daa0 msgr2=0x7f8f9006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:49.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8ffff700 1 --2- 172.21.15.158:0/1172671300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9015daa0 0x7f8f9006b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8ffff700 1 -- 172.21.15.158:0/1172671300 >> v2:172.21.15.172:3300/0 conn(0x7f8f900d3570 msgr2=0x7f8f900d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:49.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8ffff700 1 --2- 172.21.15.158:0/1172671300 >> v2:172.21.15.172:3300/0 conn(0x7f8f900d3570 0x7f8f900d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8ffff700 1 -- 172.21.15.158:0/1172671300 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9006c490 con 0x7f8f900d0bd0 2023-10-16T10:03:49.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8ffff700 1 --2- 172.21.15.158:0/1172671300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900d1510 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f8f9008ca90 tx=0x7f8f7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfcf39567c74b72 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:49.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8f7fe700 1 -- 172.21.15.158:0/1172671300 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c011830 con 0x7f8f900d0bd0 2023-10-16T10:03:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8f7fe700 1 -- 172.21.15.158:0/1172671300 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c011990 con 0x7f8f900d0bd0 2023-10-16T10:03:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.134+0000 7f8f8f7fe700 1 -- 172.21.15.158:0/1172671300 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c011d20 con 0x7f8f900d0bd0 2023-10-16T10:03:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 -- 172.21.15.158:0/1172671300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 msgr2=0x7f8f900d1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1172671300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900d1510 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f8f9008ca90 tx=0x7f8f7c009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 -- 172.21.15.158:0/1172671300 shutdown_connections 2023-10-16T10:03:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1172671300 >> v2:172.21.15.172:3300/0 conn(0x7f8f900d3570 0x7f8f900d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1172671300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9015daa0 0x7f8f9006b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1172671300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900d1510 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 -- 172.21.15.158:0/1172671300 >> 172.21.15.158:0/1172671300 conn(0x7f8f9008d040 msgr2=0x7f8f9009c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 -- 172.21.15.158:0/1172671300 shutdown_connections 2023-10-16T10:03:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.135+0000 7f8f96d28700 1 -- 172.21.15.158:0/1172671300 wait complete. 2023-10-16T10:03:49.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.137+0000 7f8f96d28700 1 Processor -- start 2023-10-16T10:03:49.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.137+0000 7f8f96d28700 1 -- start start 2023-10-16T10:03:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f96d28700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900b8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f96d28700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f900d3570 0x7f8f901f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f96d28700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f9015daa0 0x7f8f9015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f96d28700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901f31b0 con 0x7f8f900d0bd0 2023-10-16T10:03:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f96d28700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9015f180 con 0x7f8f9015daa0 2023-10-16T10:03:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f96d28700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9015e570 con 0x7f8f900d3570 2023-10-16T10:03:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f94ac4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f94ac4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900b8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59196/0 (socket says 172.21.15.158:59196) 2023-10-16T10:03:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f94ac4700 1 -- 172.21.15.158:0/1524204758 learned_addr learned my addr 172.21.15.158:0/1524204758 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f952c5700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f9015daa0 0x7f8f9015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.138+0000 7f8f8ffff700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f900d3570 0x7f8f901f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f94ac4700 1 -- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f900d3570 msgr2=0x7f8f901f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f94ac4700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f900d3570 0x7f8f901f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f94ac4700 1 -- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f9015daa0 msgr2=0x7f8f9015b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f94ac4700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f9015daa0 0x7f8f9015b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f94ac4700 1 -- 172.21.15.158:0/1524204758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c009550 con 0x7f8f900d0bd0 2023-10-16T10:03:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f94ac4700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900b8700 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400e900 tx=0x7f8f8400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f8d7fa700 1 -- 172.21.15.158:0/1524204758 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84013680 con 0x7f8f900d0bd0 2023-10-16T10:03:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f8d7fa700 1 -- 172.21.15.158:0/1524204758 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f840137e0 con 0x7f8f900d0bd0 2023-10-16T10:03:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9015e850 con 0x7f8f900d0bd0 2023-10-16T10:03:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f8d7fa700 1 -- 172.21.15.158:0/1524204758 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84013b50 con 0x7f8f900d0bd0 2023-10-16T10:03:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.139+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9015ee10 con 0x7f8f900d0bd0 2023-10-16T10:03:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.140+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f900b2960 con 0x7f8f900d0bd0 2023-10-16T10:03:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.140+0000 7f8f8d7fa700 1 -- 172.21.15.158:0/1524204758 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f8f84013cb0 con 0x7f8f900d0bd0 2023-10-16T10:03:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.141+0000 7f8f8d7fa700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 0x7f8f7406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.141+0000 7f8f8ffff700 1 -- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 msgr2=0x7f8f7406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.141+0000 7f8f8d7fa700 1 -- 172.21.15.158:0/1524204758 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f8f8408a070 con 0x7f8f900d0bd0 2023-10-16T10:03:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.141+0000 7f8f8ffff700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 0x7f8f7406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:49.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.145+0000 7f8f8d7fa700 1 -- 172.21.15.158:0/1524204758 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f84054a40 con 0x7f8f900d0bd0 2023-10-16T10:03:49.161 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:49.160+0000 7f88cafcf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:03:49.319 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:49.318+0000 7fc1bbd99200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:49.329 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:49.328+0000 7f88cafcf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:03:49.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.342+0000 7f8f8ffff700 1 -- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 msgr2=0x7f8f7406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:49.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.342+0000 7f8f8ffff700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 0x7f8f7406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:49.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.407+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f9015efd0 con 0x7f8f900d0bd0 2023-10-16T10:03:49.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.407+0000 7f8f8d7fa700 1 -- 172.21.15.158:0/1524204758 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f8f84021090 con 0x7f8f900d0bd0 2023-10-16T10:03:49.409 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:49.409 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4144904682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4144904682},{"type":"v1","addr":"172.21.15.172:6834","nonce":4144904682}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3562807630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3562807630},{"type":"v1","addr":"172.21.15.158:6836","nonce":3562807630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8805,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/614819185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":614819185},{"type":"v1","addr":"172.21.15.158:6837","nonce":614819185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:03:23.609567+0000","modified":"2023-10-16T10:03:26.633779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/4142092863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4142092863},{"type":"v1","addr":"172.21.15.172:6835","nonce":4142092863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T10:03:49.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 msgr2=0x7f8f7406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:49.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 0x7f8f7406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 msgr2=0x7f8f900b8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900b8700 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400e900 tx=0x7f8f8400ec10 comp rx=0 tx=0).stop 2023-10-16T10:03:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 shutdown_connections 2023-10-16T10:03:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7f8f7406b0b0 0x7f8f7406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f9015daa0 0x7f8f9015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f900d0bd0 0x7f8f900b8700 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 --2- 172.21.15.158:0/1524204758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f900d3570 0x7f8f901f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 >> 172.21.15.158:0/1524204758 conn(0x7f8f9008d040 msgr2=0x7f8f900b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 shutdown_connections 2023-10-16T10:03:49.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.410+0000 7f8f96d28700 1 -- 172.21.15.158:0/1524204758 wait complete. 2023-10-16T10:03:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 68 2023-10-16T10:03:49.425 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:49.479 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:49.478+0000 7fc1bbd99200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:03:49.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.590+0000 7fa2564e0700 1 Processor -- start 2023-10-16T10:03:49.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.590+0000 7fa2564e0700 1 -- start start 2023-10-16T10:03:49.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.590+0000 7fa2564e0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2500cc5d0 0x7fa25006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.590+0000 7fa2564e0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa25015dfb0 0x7fa2500cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa2564e0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25006e770 con 0x7fa25006f340 2023-10-16T10:03:49.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa2564e0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2500cf270 con 0x7fa25015dfb0 2023-10-16T10:03:49.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa2564e0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25015e4e0 con 0x7fa2500cc5d0 2023-10-16T10:03:49.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2500cc5d0 0x7fa25006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24f7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa25015dfb0 0x7fa2500cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2500cc5d0 0x7fa25006e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:41808/0 (socket says 172.21.15.158:41808) 2023-10-16T10:03:49.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24ffff700 1 -- 172.21.15.158:0/1404493118 learned_addr learned my addr 172.21.15.158:0/1404493118 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:49.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24f7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa25015dfb0 0x7fa2500cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:60388/0 (socket says 172.21.15.158:60388) 2023-10-16T10:03:49.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa254a7d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa25006f340 0x7fa250071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:35058/0 (socket says 172.21.15.158:35058) 2023-10-16T10:03:49.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24f7fe700 1 -- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2500cc5d0 msgr2=0x7fa25006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:49.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24f7fe700 1 --2- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2500cc5d0 0x7fa25006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24f7fe700 1 -- 172.21.15.158:0/1404493118 >> v1:172.21.15.172:6789/0 conn(0x7fa25006f340 legacy=0x7fa250071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24f7fe700 1 -- 172.21.15.158:0/1404493118 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25015e7c0 con 0x7fa25015dfb0 2023-10-16T10:03:49.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24effd700 1 -- 172.21.15.158:0/1404493118 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344189105 0 0) 0x7fa248001490 con 0x7fa25006f340 2023-10-16T10:03:49.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24ffff700 1 --2- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2500cc5d0 0x7fa25006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:49.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24f7fe700 1 --2- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa2500cb6f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa244009840 tx=0x7fa244009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=272f52ae0afd9bca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:49.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.591+0000 7fa24effd700 1 -- 172.21.15.158:0/1404493118 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244011830 con 0x7fa25015dfb0 2023-10-16T10:03:49.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa24effd700 1 -- 172.21.15.158:0/1404493118 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244011990 con 0x7fa25015dfb0 2023-10-16T10:03:49.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa24effd700 1 -- 172.21.15.158:0/1404493118 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244011d20 con 0x7fa25015dfb0 2023-10-16T10:03:49.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 -- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 msgr2=0x7fa2500cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 --2- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa2500cb6f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa244009840 tx=0x7fa244009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:49.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 -- 172.21.15.158:0/1404493118 shutdown_connections 2023-10-16T10:03:49.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 --2- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2500cc5d0 0x7fa25006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 --2- 172.21.15.158:0/1404493118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa2500cb6f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 -- 172.21.15.158:0/1404493118 >> 172.21.15.158:0/1404493118 conn(0x7fa2500d02b0 msgr2=0x7fa2500d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:49.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 -- 172.21.15.158:0/1404493118 shutdown_connections 2023-10-16T10:03:49.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.592+0000 7fa2564e0700 1 -- 172.21.15.158:0/1404493118 wait complete. 2023-10-16T10:03:49.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.593+0000 7fa2564e0700 1 Processor -- start 2023-10-16T10:03:49.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa2564e0700 1 -- start start 2023-10-16T10:03:49.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa2564e0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa25006f340 0x7fa2500ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa2564e0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2500cc5d0 0x7fa2501524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa2564e0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa25006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa2564e0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa250155090 con 0x7fa2500cc5d0 2023-10-16T10:03:49.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa2564e0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa250154c00 con 0x7fa25015dfb0 2023-10-16T10:03:49.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa2564e0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa250159400 con 0x7fa25006f340 2023-10-16T10:03:49.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa24f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2500cc5d0 0x7fa2501524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa254a7d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa25006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa24f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2500cc5d0 0x7fa2501524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59214/0 (socket says 172.21.15.158:59214) 2023-10-16T10:03:49.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa24f7fe700 1 -- 172.21.15.158:0/3035392736 learned_addr learned my addr 172.21.15.158:0/3035392736 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:49.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.594+0000 7fa24ffff700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa25006f340 0x7fa2500ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24f7fe700 1 -- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa25006f340 msgr2=0x7fa2500ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24f7fe700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa25006f340 0x7fa2500ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24f7fe700 1 -- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 msgr2=0x7fa25006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24f7fe700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa25006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24f7fe700 1 -- 172.21.15.158:0/3035392736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240009550 con 0x7fa2500cc5d0 2023-10-16T10:03:49.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa254a7d700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa25006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:49.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24f7fe700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2500cc5d0 0x7fa2501524b0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa244009fd0 tx=0x7fa244004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:49.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24cff9700 1 -- 172.21.15.158:0/3035392736 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2440036a0 con 0x7fa2500cc5d0 2023-10-16T10:03:49.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24cff9700 1 -- 172.21.15.158:0/3035392736 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24401f070 con 0x7fa2500cc5d0 2023-10-16T10:03:49.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2440094a0 con 0x7fa2500cc5d0 2023-10-16T10:03:49.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa24cff9700 1 -- 172.21.15.158:0/3035392736 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244003a60 con 0x7fa2500cc5d0 2023-10-16T10:03:49.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.595+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2501545b0 con 0x7fa2500cc5d0 2023-10-16T10:03:49.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.596+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa250154d70 con 0x7fa2500cc5d0 2023-10-16T10:03:49.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.597+0000 7fa24cff9700 1 -- 172.21.15.158:0/3035392736 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fa244003bc0 con 0x7fa2500cc5d0 2023-10-16T10:03:49.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.597+0000 7fa24cff9700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 0x7fa23c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.597+0000 7fa24ffff700 1 -- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 msgr2=0x7fa23c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:49.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.597+0000 7fa24cff9700 1 -- 172.21.15.158:0/3035392736 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fa244021070 con 0x7fa2500cc5d0 2023-10-16T10:03:49.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.597+0000 7fa24ffff700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 0x7fa23c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:49.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.601+0000 7fa24cff9700 1 -- 172.21.15.158:0/3035392736 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa244056120 con 0x7fa2500cc5d0 2023-10-16T10:03:49.610 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:49.609+0000 7f88cafcf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:03:49.778 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:49.777+0000 7f88cafcf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:03:49.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.799+0000 7fa24ffff700 1 -- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 msgr2=0x7fa23c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:49.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.799+0000 7fa24ffff700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 0x7fa23c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:49.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.800+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2500681b0 con 0x7fa2500cc5d0 2023-10-16T10:03:49.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.801+0000 7fa24cff9700 1 -- 172.21.15.158:0/3035392736 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17995 (secure 0 0 0) 0x7fa244015090 con 0x7fa2500cc5d0 2023-10-16T10:03:49.803 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:49.803 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":141,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:37.828969+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:03:21.520809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T10:03:22.576358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:6832/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/3946986955":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/2041489640":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4016680871":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4198985108":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1741373068":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1608841554":"2023-10-17T10:03:37.828937+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:49.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.803+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 msgr2=0x7fa23c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:49.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.803+0000 7fa2564e0700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 0x7fa23c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.803+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2500cc5d0 msgr2=0x7fa2501524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.803+0000 7fa2564e0700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2500cc5d0 0x7fa2501524b0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa244009fd0 tx=0x7fa244004750 comp rx=0 tx=0).stop 2023-10-16T10:03:49.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 shutdown_connections 2023-10-16T10:03:49.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fa23c06b0b0 0x7fa23c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa25015dfb0 0x7fa25006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa25006f340 0x7fa2500ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 --2- 172.21.15.158:0/3035392736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2500cc5d0 0x7fa2501524b0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 >> 172.21.15.158:0/3035392736 conn(0x7fa2500d02b0 msgr2=0x7fa2500d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:49.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 shutdown_connections 2023-10-16T10:03:49.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.804+0000 7fa2564e0700 1 -- 172.21.15.158:0/3035392736 wait complete. 2023-10-16T10:03:49.818 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T10:03:49.818 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:49.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.940+0000 7fcde3e50700 1 Processor -- start 2023-10-16T10:03:49.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.940+0000 7fcde3e50700 1 -- start start 2023-10-16T10:03:49.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde3e50700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcddc15ed60 0x7fcddc0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde3e50700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcddc06a8d0 0x7fcddc06ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde3e50700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcddc0b3910 con 0x7fcddc0b3480 2023-10-16T10:03:49.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde3e50700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0b3f30 con 0x7fcddc15ed60 2023-10-16T10:03:49.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde3e50700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0b4ad0 con 0x7fcddc06a8d0 2023-10-16T10:03:49.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde1bec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcddc15ed60 0x7fcddc0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde1bec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcddc15ed60 0x7fcddc0cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59222/0 (socket says 172.21.15.158:59222) 2023-10-16T10:03:49.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde1bec700 1 -- 172.21.15.158:0/3352098423 learned_addr learned my addr 172.21.15.158:0/3352098423 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:49.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.941+0000 7fcde23ed700 1 --2- 172.21.15.158:0/3352098423 >> v2:172.21.15.172:3300/0 conn(0x7fcddc06a8d0 0x7fcddc06ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde1bec700 1 -- 172.21.15.158:0/3352098423 >> v2:172.21.15.172:3300/0 conn(0x7fcddc06a8d0 msgr2=0x7fcddc06ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:49.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde1bec700 1 --2- 172.21.15.158:0/3352098423 >> v2:172.21.15.172:3300/0 conn(0x7fcddc06a8d0 0x7fcddc06ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde1bec700 1 -- 172.21.15.158:0/3352098423 >> v1:172.21.15.172:6789/0 conn(0x7fcddc0b3480 legacy=0x7fcddc0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde1bec700 1 -- 172.21.15.158:0/3352098423 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc06b240 con 0x7fcddc15ed60 2023-10-16T10:03:49.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde23ed700 1 --2- 172.21.15.158:0/3352098423 >> v2:172.21.15.172:3300/0 conn(0x7fcddc06a8d0 0x7fcddc06ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:49.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde1bec700 1 --2- 172.21.15.158:0/3352098423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cd890 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcddc09e1c0 tx=0x7fcdd8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e60af4821c0854 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:49.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde0bea700 1 -- 172.21.15.158:0/3352098423 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8005e30 con 0x7fcddc15ed60 2023-10-16T10:03:49.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde0bea700 1 -- 172.21.15.158:0/3352098423 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd8004030 con 0x7fcddc15ed60 2023-10-16T10:03:49.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.942+0000 7fcde0bea700 1 -- 172.21.15.158:0/3352098423 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80043e0 con 0x7fcddc15ed60 2023-10-16T10:03:49.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 -- 172.21.15.158:0/3352098423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 msgr2=0x7fcddc0cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3352098423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cd890 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcddc09e1c0 tx=0x7fcdd8009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 -- 172.21.15.158:0/3352098423 shutdown_connections 2023-10-16T10:03:49.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3352098423 >> v2:172.21.15.172:3300/0 conn(0x7fcddc06a8d0 0x7fcddc06ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3352098423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cd890 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 -- 172.21.15.158:0/3352098423 >> 172.21.15.158:0/3352098423 conn(0x7fcddc0ccda0 msgr2=0x7fcddc0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:49.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 -- 172.21.15.158:0/3352098423 shutdown_connections 2023-10-16T10:03:49.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.943+0000 7fcde3e50700 1 -- 172.21.15.158:0/3352098423 wait complete. 2023-10-16T10:03:49.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.944+0000 7fcde3e50700 1 Processor -- start 2023-10-16T10:03:49.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.944+0000 7fcde3e50700 1 -- start start 2023-10-16T10:03:49.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde3e50700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcddc06a8d0 0x7fcddc0cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde3e50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde3e50700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcddc0c7f70 0x7fcddc0ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde3e50700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0b18a0 con 0x7fcddc15ed60 2023-10-16T10:03:49.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde3e50700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0cf7b0 con 0x7fcddc06a8d0 2023-10-16T10:03:49.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde3e50700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0b0f00 con 0x7fcddc0c7f70 2023-10-16T10:03:49.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde1bec700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcddc06a8d0 0x7fcddc0cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59228/0 (socket says 172.21.15.158:59228) 2023-10-16T10:03:49.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 -- 172.21.15.158:0/3497647930 learned_addr learned my addr 172.21.15.158:0/3497647930 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:49.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde1bec700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcddc06a8d0 0x7fcddc0cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:60406/0 (socket says 172.21.15.158:60406) 2023-10-16T10:03:49.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde23ed700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcddc0c7f70 0x7fcddc0ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:49.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 -- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcddc0c7f70 msgr2=0x7fcddc0ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcddc0c7f70 0x7fcddc0ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 -- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcddc06a8d0 msgr2=0x7fcddc0cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:49.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcddc06a8d0 0x7fcddc0cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:49.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 -- 172.21.15.158:0/3497647930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd4009a70 con 0x7fcddc15ed60 2023-10-16T10:03:49.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde1bec700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcddc06a8d0 0x7fcddc0cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:49.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcde13eb700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cf180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c4e0 tx=0x7fcdcc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:49.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.945+0000 7fcdd27fc700 1 -- 172.21.15.158:0/3497647930 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0052d0 con 0x7fcddc15ed60 2023-10-16T10:03:49.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.946+0000 7fcdd27fc700 1 -- 172.21.15.158:0/3497647930 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc005430 con 0x7fcddc15ed60 2023-10-16T10:03:49.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.946+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd8009500 con 0x7fcddc15ed60 2023-10-16T10:03:49.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.946+0000 7fcdd27fc700 1 -- 172.21.15.158:0/3497647930 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0038c0 con 0x7fcddc15ed60 2023-10-16T10:03:49.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.946+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc0b14c0 con 0x7fcddc15ed60 2023-10-16T10:03:49.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.947+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc0b19e0 con 0x7fcddc15ed60 2023-10-16T10:03:49.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.947+0000 7fcdd27fc700 1 -- 172.21.15.158:0/3497647930 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fcdcc003a20 con 0x7fcddc15ed60 2023-10-16T10:03:49.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.947+0000 7fcdd27fc700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 0x7fcdc006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:49.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.948+0000 7fcde1bec700 1 -- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 msgr2=0x7fcdc006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:49.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.948+0000 7fcdd27fc700 1 -- 172.21.15.158:0/3497647930 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fcdcc088ea0 con 0x7fcddc15ed60 2023-10-16T10:03:49.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.948+0000 7fcde1bec700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 0x7fcdc006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:49.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:49.952+0000 7fcdd27fc700 1 -- 172.21.15.158:0/3497647930 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcddc0b19e0 con 0x7fcddc15ed60 2023-10-16T10:03:50.006 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:03:50.005+0000 7fc1bbd99200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:03:50.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.149+0000 7fcde1bec700 1 -- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 msgr2=0x7fcdc006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:50.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.149+0000 7fcde1bec700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 0x7fcdc006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:03:50.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.195+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcddc0b3910 con 0x7fcddc15ed60 2023-10-16T10:03:50.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.195+0000 7fcdd27fc700 1 -- 172.21.15.158:0/3497647930 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fcddc0b3910 con 0x7fcddc15ed60 2023-10-16T10:03:50.197 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:50.197 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4144904682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4144904682},{"type":"v1","addr":"172.21.15.172:6834","nonce":4144904682}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3562807630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3562807630},{"type":"v1","addr":"172.21.15.158:6836","nonce":3562807630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8805,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/614819185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":614819185},{"type":"v1","addr":"172.21.15.158:6837","nonce":614819185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:03:23.609567+0000","modified":"2023-10-16T10:03:26.633779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/4142092863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4142092863},{"type":"v1","addr":"172.21.15.172:6835","nonce":4142092863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T10:03:50.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.197+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 msgr2=0x7fcdc006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:50.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.197+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 0x7fcdc006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 msgr2=0x7fcddc0cf180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cf180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c4e0 tx=0x7fcdcc00c7f0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 shutdown_connections 2023-10-16T10:03:50.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fcdc006b0b0 0x7fcdc006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcddc0c7f70 0x7fcddc0ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcddc06a8d0 0x7fcddc0cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 --2- 172.21.15.158:0/3497647930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcddc15ed60 0x7fcddc0cf180 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 >> 172.21.15.158:0/3497647930 conn(0x7fcddc0ccda0 msgr2=0x7fcddc0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:50.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 shutdown_connections 2023-10-16T10:03:50.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.198+0000 7fcde3e50700 1 -- 172.21.15.158:0/3497647930 wait complete. 2023-10-16T10:03:50.202 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 68 2023-10-16T10:03:50.212 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:50.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.336+0000 7fc8daded700 1 Processor -- start 2023-10-16T10:03:50.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.336+0000 7fc8daded700 1 -- start start 2023-10-16T10:03:50.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.336+0000 7fc8daded700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d415d2f0 0x7fc8d415d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.336+0000 7fc8daded700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8d415ee50 0x7fc8d415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8daded700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d41613b0 con 0x7fc8d406d590 2023-10-16T10:03:50.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8daded700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d4161690 con 0x7fc8d415ee50 2023-10-16T10:03:50.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8daded700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d4161970 con 0x7fc8d415d2f0 2023-10-16T10:03:50.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8d415ee50 0x7fc8d415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8d415ee50 0x7fc8d415dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59238/0 (socket says 172.21.15.158:59238) 2023-10-16T10:03:50.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 -- 172.21.15.158:0/3991259187 learned_addr learned my addr 172.21.15.158:0/3991259187 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:50.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d8b89700 1 --2- 172.21.15.158:0/3991259187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d415d2f0 0x7fc8d415d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 -- 172.21.15.158:0/3991259187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d415d2f0 msgr2=0x7fc8d415d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:50.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 --2- 172.21.15.158:0/3991259187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d415d2f0 0x7fc8d415d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 -- 172.21.15.158:0/3991259187 >> v1:172.21.15.172:6789/0 conn(0x7fc8d406d590 legacy=0x7fc8d40cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d37fe700 1 -- 172.21.15.158:0/3991259187 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776802167 0 0) 0x7fc8d41613b0 con 0x7fc8d406d590 2023-10-16T10:03:50.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 -- 172.21.15.158:0/3991259187 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d4161c50 con 0x7fc8d415ee50 2023-10-16T10:03:50.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d3fff700 1 --2- 172.21.15.158:0/3991259187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d415dc10 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00098f0 tx=0x7fc8c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cee748e22946516 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:50.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d37fe700 1 -- 172.21.15.158:0/3991259187 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0011830 con 0x7fc8d415ee50 2023-10-16T10:03:50.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d37fe700 1 -- 172.21.15.158:0/3991259187 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c0011990 con 0x7fc8d415ee50 2023-10-16T10:03:50.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.337+0000 7fc8d37fe700 1 -- 172.21.15.158:0/3991259187 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0011d20 con 0x7fc8d415ee50 2023-10-16T10:03:50.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 -- 172.21.15.158:0/3991259187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 msgr2=0x7fc8d415dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 --2- 172.21.15.158:0/3991259187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d415dc10 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00098f0 tx=0x7fc8c0009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:50.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 -- 172.21.15.158:0/3991259187 shutdown_connections 2023-10-16T10:03:50.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 --2- 172.21.15.158:0/3991259187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d415d2f0 0x7fc8d415d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 --2- 172.21.15.158:0/3991259187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d415dc10 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 -- 172.21.15.158:0/3991259187 >> 172.21.15.158:0/3991259187 conn(0x7fc8d406a8d0 msgr2=0x7fc8d406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:50.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 -- 172.21.15.158:0/3991259187 shutdown_connections 2023-10-16T10:03:50.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.338+0000 7fc8daded700 1 -- 172.21.15.158:0/3991259187 wait complete. 2023-10-16T10:03:50.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.340+0000 7fc8daded700 1 Processor -- start 2023-10-16T10:03:50.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.340+0000 7fc8daded700 1 -- start start 2023-10-16T10:03:50.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8daded700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d406d590 0x7fc8d41598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8daded700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d415d2f0 0x7fc8d4155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8daded700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d4157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8daded700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d415a510 con 0x7fc8d415ee50 2023-10-16T10:03:50.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8daded700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d4159f50 con 0x7fc8d415d2f0 2023-10-16T10:03:50.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8daded700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d415a230 con 0x7fc8d406d590 2023-10-16T10:03:50.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d938a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d4157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d938a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d4157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59246/0 (socket says 172.21.15.158:59246) 2023-10-16T10:03:50.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d938a700 1 -- 172.21.15.158:0/4010073997 learned_addr learned my addr 172.21.15.158:0/4010073997 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:50.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d8b89700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d406d590 0x7fc8d41598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d3fff700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d415d2f0 0x7fc8d4155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d8b89700 1 -- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d415d2f0 msgr2=0x7fc8d4155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d8b89700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d415d2f0 0x7fc8d4155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d8b89700 1 -- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 msgr2=0x7fc8d4157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d8b89700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d4157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d8b89700 1 -- 172.21.15.158:0/4010073997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0009550 con 0x7fc8d406d590 2023-10-16T10:03:50.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d938a700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d4157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:50.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.341+0000 7fc8d3fff700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d415d2f0 0x7fc8d4155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:50.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.342+0000 7fc8d8b89700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d406d590 0x7fc8d41598c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800e900 tx=0x7fc8c800ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:50.351 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.342+0000 7f88cafcf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:03:50.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.342+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8013680 con 0x7fc8d406d590 2023-10-16T10:03:50.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.342+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c80137e0 con 0x7fc8d406d590 2023-10-16T10:03:50.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.342+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d40c5960 con 0x7fc8d406d590 2023-10-16T10:03:50.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.342+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8013b50 con 0x7fc8d406d590 2023-10-16T10:03:50.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.342+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d41564d0 con 0x7fc8d406d590 2023-10-16T10:03:50.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.343+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d40c5fb0 con 0x7fc8d406d590 2023-10-16T10:03:50.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.343+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fc8c8013cb0 con 0x7fc8d406d590 2023-10-16T10:03:50.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.344+0000 7fc8d17fa700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fc8b806b0b0 0x7fc8b806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.344+0000 7fc8d3fff700 1 -- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fc8b806b0b0 msgr2=0x7fc8b806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/781614815 2023-10-16T10:03:50.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.344+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fc8c8089cf0 con 0x7fc8d406d590 2023-10-16T10:03:50.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.344+0000 7fc8d3fff700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fc8b806b0b0 0x7fc8b806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:03:50.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.348+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc8c808e070 con 0x7fc8d406d590 2023-10-16T10:03:50.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.363+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mgrmap(e 82) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc8c8054aa0 con 0x7fc8d406d590 2023-10-16T10:03:50.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.363+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fc8b806b0b0 msgr2=0x7fc8b806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:03:50.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.363+0000 7fc8d17fa700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fc8b806b0b0 0x7fc8b806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.427 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88467bb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.427 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88467bb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.427 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88467bb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.428 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88467bb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.428 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88467bb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.428 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88437b5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.428 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88437b5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.428 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88437b5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.429 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88437b5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.429 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:03:50.425+0000 7f88437b5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:03:50.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.611+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8d40c57c0 con 0x7fc8d406d590 2023-10-16T10:03:50.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.612+0000 7fc8d17fa700 1 -- 172.21.15.158:0/4010073997 <== mon.2 v2:172.21.15.172:3301/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fc8d40c57c0 con 0x7fc8d406d590 2023-10-16T10:03:50.613 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:50.614 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4144904682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4144904682},{"type":"v1","addr":"172.21.15.172:6834","nonce":4144904682}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3562807630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3562807630},{"type":"v1","addr":"172.21.15.158:6836","nonce":3562807630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8805,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/614819185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":614819185},{"type":"v1","addr":"172.21.15.158:6837","nonce":614819185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:03:23.609567+0000","modified":"2023-10-16T10:03:26.633779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/4142092863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":4142092863},{"type":"v1","addr":"172.21.15.172:6835","nonce":4142092863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T10:03:50.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d406d590 msgr2=0x7fc8d41598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d406d590 0x7fc8d41598c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800e900 tx=0x7fc8c800ec10 comp rx=0 tx=0).stop 2023-10-16T10:03:50.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 shutdown_connections 2023-10-16T10:03:50.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:6832/781614815,v1:172.21.15.158:6833/781614815] conn(0x7fc8b806b0b0 0x7fc8b806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8d415ee50 0x7fc8d4157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d406d590 0x7fc8d41598c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 --2- 172.21.15.158:0/4010073997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d415d2f0 0x7fc8d4155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 >> 172.21.15.158:0/4010073997 conn(0x7fc8d406a8d0 msgr2=0x7fc8d415f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:50.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 shutdown_connections 2023-10-16T10:03:50.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.614+0000 7fc8daded700 1 -- 172.21.15.158:0/4010073997 wait complete. 2023-10-16T10:03:50.618 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 68 2023-10-16T10:03:50.628 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:50.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.746+0000 7f9c5df6d700 1 Processor -- start 2023-10-16T10:03:50.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.747+0000 7f9c5df6d700 1 -- start start 2023-10-16T10:03:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.747+0000 7f9c5df6d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580cbef0 0x7f9c5815ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.747+0000 7f9c5df6d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c5806f9b0 0x7f9c580cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.747+0000 7f9c5df6d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5806f520 con 0x7f9c580ced40 2023-10-16T10:03:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.747+0000 7f9c5df6d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5815ef70 con 0x7f9c5806f9b0 2023-10-16T10:03:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.747+0000 7f9c5df6d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580cf120 con 0x7f9c580cbef0 2023-10-16T10:03:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c5806f9b0 0x7f9c580cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c5806f9b0 0x7f9c580cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59326/0 (socket says 172.21.15.158:59326) 2023-10-16T10:03:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 -- 172.21.15.158:0/837126281 learned_addr learned my addr 172.21.15.158:0/837126281 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/837126281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580cbef0 0x7f9c5815ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 -- 172.21.15.158:0/837126281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580cbef0 msgr2=0x7f9c5815ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 --2- 172.21.15.158:0/837126281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580cbef0 0x7f9c5815ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 -- 172.21.15.158:0/837126281 >> v1:172.21.15.172:6789/0 conn(0x7f9c580ced40 legacy=0x7f9c580cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 -- 172.21.15.158:0/837126281 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5806c1d0 con 0x7f9c5806f9b0 2023-10-16T10:03:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/837126281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580cbef0 0x7f9c5815ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c57fff700 1 --2- 172.21.15.158:0/837126281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5806f9b0 0x7f9c580cd6f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009e50 tx=0x7f9c5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d771a1036e926c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c567fc700 1 -- 172.21.15.158:0/837126281 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5000e070 con 0x7f9c5806f9b0 2023-10-16T10:03:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c567fc700 1 -- 172.21.15.158:0/837126281 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5000b9c0 con 0x7f9c5806f9b0 2023-10-16T10:03:50.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.748+0000 7f9c567fc700 1 -- 172.21.15.158:0/837126281 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5000bd70 con 0x7f9c5806f9b0 2023-10-16T10:03:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 -- 172.21.15.158:0/837126281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5806f9b0 msgr2=0x7f9c580cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 --2- 172.21.15.158:0/837126281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5806f9b0 0x7f9c580cd6f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009e50 tx=0x7f9c5000b330 comp rx=0 tx=0).stop 2023-10-16T10:03:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 -- 172.21.15.158:0/837126281 shutdown_connections 2023-10-16T10:03:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 --2- 172.21.15.158:0/837126281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5806f9b0 0x7f9c580cd6f0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 --2- 172.21.15.158:0/837126281 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580cbef0 0x7f9c5815ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 -- 172.21.15.158:0/837126281 >> 172.21.15.158:0/837126281 conn(0x7f9c580cc5a0 msgr2=0x7f9c580cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:50.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 -- 172.21.15.158:0/837126281 shutdown_connections 2023-10-16T10:03:50.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.749+0000 7f9c5df6d700 1 -- 172.21.15.158:0/837126281 wait complete. 2023-10-16T10:03:50.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.750+0000 7f9c5df6d700 1 Processor -- start 2023-10-16T10:03:50.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.750+0000 7f9c5df6d700 1 -- start start 2023-10-16T10:03:50.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c5df6d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c5806f9b0 0x7f9c581f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c5df6d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c580cbef0 0x7f9c580c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c5df6d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580ced40 0x7f9c580c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:50.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c5df6d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580c2c80 con 0x7f9c580cbef0 2023-10-16T10:03:50.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c5df6d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c581f0590 con 0x7f9c5806f9b0 2023-10-16T10:03:50.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c5df6d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580c6bf0 con 0x7f9c580ced40 2023-10-16T10:03:50.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c580cbef0 0x7f9c580c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c57fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580ced40 0x7f9c580c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c577fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c5806f9b0 0x7f9c581f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:50.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c580cbef0 0x7f9c580c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59330/0 (socket says 172.21.15.158:59330) 2023-10-16T10:03:50.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 -- 172.21.15.158:0/1062807520 learned_addr learned my addr 172.21.15.158:0/1062807520 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:50.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 -- 172.21.15.158:0/1062807520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580ced40 msgr2=0x7f9c580c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/1062807520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580ced40 0x7f9c580c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 -- 172.21.15.158:0/1062807520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c5806f9b0 msgr2=0x7f9c581f0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/1062807520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c5806f9b0 0x7f9c581f0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 -- 172.21.15.158:0/1062807520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c50009b30 con 0x7f9c580cbef0 2023-10-16T10:03:50.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.751+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/1062807520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c580cbef0 0x7f9c580c9420 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000e810 tx=0x7f9c4000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:50.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.752+0000 7f9c4ffff700 1 -- 172.21.15.158:0/1062807520 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4000c070 con 0x7f9c580cbef0 2023-10-16T10:03:50.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.752+0000 7f9c4ffff700 1 -- 172.21.15.158:0/1062807520 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40004230 con 0x7f9c580cbef0 2023-10-16T10:03:50.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.752+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c580c6ed0 con 0x7f9c580cbef0 2023-10-16T10:03:50.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.752+0000 7f9c4ffff700 1 -- 172.21.15.158:0/1062807520 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c400036a0 con 0x7f9c580cbef0 2023-10-16T10:03:50.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.752+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c580c7490 con 0x7f9c580cbef0 2023-10-16T10:03:50.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.753+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c580c7ae0 con 0x7f9c580cbef0 2023-10-16T10:03:50.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.753+0000 7f9c4ffff700 1 -- 172.21.15.158:0/1062807520 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 82) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9c40003800 con 0x7f9c580cbef0 2023-10-16T10:03:50.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.754+0000 7f9c4ffff700 1 -- 172.21.15.158:0/1062807520 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 7212+0+0 (secure 0 0 0) 0x7f9c400892c0 con 0x7f9c580cbef0 2023-10-16T10:03:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.758+0000 7f9c4ffff700 1 -- 172.21.15.158:0/1062807520 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c400541e0 con 0x7f9c580cbef0 2023-10-16T10:03:50.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.953+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c580681b0 con 0x7f9c580cbef0 2023-10-16T10:03:50.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.955+0000 7f9c4ffff700 1 -- 172.21.15.158:0/1062807520 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+18564 (secure 0 0 0) 0x7f9c40057800 con 0x7f9c580cbef0 2023-10-16T10:03:50.956 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:50.956 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":142,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:50.355973+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:03:21.520809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T10:03:22.576358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/781614815":"2023-10-17T10:03:50.355931+0000","172.21.15.158:6832/781614815":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/4236549529":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/735244434":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/636538481":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/3740707264":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/3067380233":"2023-10-17T10:03:50.355931+0000","172.21.15.158:6833/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:6832/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/3946986955":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1853119573":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/2041489640":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4016680871":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4198985108":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/3100609660":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/1741373068":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1608841554":"2023-10-17T10:03:37.828937+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:50.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c580cbef0 msgr2=0x7f9c580c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:50.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 --2- 172.21.15.158:0/1062807520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c580cbef0 0x7f9c580c9420 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000e810 tx=0x7f9c4000eb20 comp rx=0 tx=0).stop 2023-10-16T10:03:50.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 shutdown_connections 2023-10-16T10:03:50.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 --2- 172.21.15.158:0/1062807520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c580ced40 0x7f9c580c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 --2- 172.21.15.158:0/1062807520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c5806f9b0 0x7f9c581f0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 --2- 172.21.15.158:0/1062807520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c580cbef0 0x7f9c580c9420 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:50.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 >> 172.21.15.158:0/1062807520 conn(0x7f9c580cc5a0 msgr2=0x7f9c580cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:50.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 shutdown_connections 2023-10-16T10:03:50.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:50.957+0000 7f9c5df6d700 1 -- 172.21.15.158:0/1062807520 wait complete. 2023-10-16T10:03:50.971 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T10:03:51.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.090+0000 7f07fa88a700 1 Processor -- start 2023-10-16T10:03:51.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.090+0000 7f07fa88a700 1 -- start start 2023-10-16T10:03:51.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07fa88a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 0x7f07f40b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07fa88a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f415f990 con 0x7f07f40b4950 2023-10-16T10:03:51.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07fa88a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f415fc70 con 0x7f07f415eb10 2023-10-16T10:03:51.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07fa88a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f40cdef0 con 0x7f07f40d0ae0 2023-10-16T10:03:51.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f8e27700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f07f40b4950 0x7f07f40cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59512/0 (socket says 172.21.15.158:59512) 2023-10-16T10:03:51.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f8e27700 1 -- 172.21.15.158:0/2344074946 learned_addr learned my addr 172.21.15.158:0/2344074946 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:51.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f37fe700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 0x7f07f40b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263690824 0 0) 0x7f07f415f990 con 0x7f07f40b4950 2023-10-16T10:03:51.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07dc0035b0 con 0x7f07f40b4950 2023-10-16T10:03:51.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441583229 0 0) 0x7f07dc0035b0 con 0x7f07f40b4950 2023-10-16T10:03:51.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 msgr2=0x7f07f40b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:51.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 0x7f07f40b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 >> v1:172.21.15.172:6789/0 conn(0x7f07f415eb10 legacy=0x7f07f415f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f415ff50 con 0x7f07f40b4950 2023-10-16T10:03:51.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f07ec0019f0 con 0x7f07f40b4950 2023-10-16T10:03:51.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 >> v1:172.21.15.158:6789/0 conn(0x7f07f40b4950 legacy=0x7f07f40cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07dc003a30 0x7f07dc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 0x7f07dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.091+0000 7f07f2ffd700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07dc004d90 0x7f07dc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc00ba10 con 0x7f07dc003a30 2023-10-16T10:03:51.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc00bb50 con 0x7f07dc004d90 2023-10-16T10:03:51.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc00bd10 con 0x7f07f40d0ae0 2023-10-16T10:03:51.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f3fff700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07dc003a30 0x7f07dc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f37fe700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 0x7f07dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f8e27700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07dc004d90 0x7f07dc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f3fff700 1 -- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 msgr2=0x7f07dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:51.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f3fff700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07f40d0ae0 0x7f07dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f3fff700 1 -- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07dc004d90 msgr2=0x7f07dc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:51.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f3fff700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07dc004d90 0x7f07dc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f3fff700 1 -- 172.21.15.158:0/2344074946 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc0035b0 con 0x7f07dc003a30 2023-10-16T10:03:51.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f3fff700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07dc003a30 0x7f07dc003e10 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f07e80018d0 tx=0x7f07dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183b05c1350208f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:51.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8012620 con 0x7f07dc003a30 2023-10-16T10:03:51.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e80127a0 con 0x7f07dc003a30 2023-10-16T10:03:51.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.092+0000 7f07f2ffd700 1 -- 172.21.15.158:0/2344074946 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8012b50 con 0x7f07dc003a30 2023-10-16T10:03:51.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.093+0000 7f07fa88a700 1 -- 172.21.15.158:0/2344074946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07dc003a30 msgr2=0x7f07dc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.093+0000 7f07fa88a700 1 --2- 172.21.15.158:0/2344074946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07dc003a30 0x7f07dc003e10 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f07e80018d0 tx=0x7f07dc005880 comp rx=0 tx=0).stop 2023-10-16T10:03:51.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.093+0000 7f07f8e27700 1 -- 172.21.15.158:0/2344074946 reap_dead start 2023-10-16T10:03:51.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.093+0000 7f07fa88a700 1 -- 172.21.15.158:0/2344074946 shutdown_connections 2023-10-16T10:03:51.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.093+0000 7f07fa88a700 1 -- 172.21.15.158:0/2344074946 >> 172.21.15.158:0/2344074946 conn(0x7f07f40ce040 msgr2=0x7f07f40ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:51.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.093+0000 7f07fa88a700 1 -- 172.21.15.158:0/2344074946 shutdown_connections 2023-10-16T10:03:51.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.093+0000 7f07fa88a700 1 -- 172.21.15.158:0/2344074946 wait complete. 2023-10-16T10:03:51.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.094+0000 7f07fa88a700 1 Processor -- start 2023-10-16T10:03:51.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.094+0000 7f07fa88a700 1 -- start start 2023-10-16T10:03:51.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07fa88a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07dc003a30 0x7f07f40c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07fa88a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07f40ca7d0 0x7f07f4159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07fa88a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07f40cbe00 0x7f07f41437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07fa88a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f40cb9e0 con 0x7f07f40ca7d0 2023-10-16T10:03:51.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07fa88a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f40c8060 con 0x7f07f40cbe00 2023-10-16T10:03:51.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07fa88a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f40c9310 con 0x7f07dc003a30 2023-10-16T10:03:51.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07f40ca7d0 0x7f07f4159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07f40ca7d0 0x7f07f4159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59350/0 (socket says 172.21.15.158:59350) 2023-10-16T10:03:51.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 -- 172.21.15.158:0/2814115121 learned_addr learned my addr 172.21.15.158:0/2814115121 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:51.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f3fff700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07dc003a30 0x7f07f40c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f8e27700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07f40cbe00 0x7f07f41437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 -- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07dc003a30 msgr2=0x7f07f40c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07dc003a30 0x7f07f40c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 -- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07f40cbe00 msgr2=0x7f07f41437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07f40cbe00 0x7f07f41437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 -- 172.21.15.158:0/2814115121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40cad60 con 0x7f07f40ca7d0 2023-10-16T10:03:51.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f37fe700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07f40ca7d0 0x7f07f4159740 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f07e4011aa0 tx=0x7f07e4011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:51.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f0ff9700 1 -- 172.21.15.158:0/2814115121 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e4011ee0 con 0x7f07f40ca7d0 2023-10-16T10:03:51.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07f0ff9700 1 -- 172.21.15.158:0/2814115121 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e400f8e0 con 0x7f07f40ca7d0 2023-10-16T10:03:51.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.095+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f40cb040 con 0x7f07f40ca7d0 2023-10-16T10:03:51.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.096+0000 7f07f0ff9700 1 -- 172.21.15.158:0/2814115121 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e400fc90 con 0x7f07f40ca7d0 2023-10-16T10:03:51.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.096+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f40cb600 con 0x7f07f40ca7d0 2023-10-16T10:03:51.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.096+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f40681b0 con 0x7f07f40ca7d0 2023-10-16T10:03:51.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.097+0000 7f07f0ff9700 1 -- 172.21.15.158:0/2814115121 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 82) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f07e4014440 con 0x7f07f40ca7d0 2023-10-16T10:03:51.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.097+0000 7f07f0ff9700 1 -- 172.21.15.158:0/2814115121 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 7212+0+0 (secure 0 0 0) 0x7f07e4089a80 con 0x7f07f40ca7d0 2023-10-16T10:03:51.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.102+0000 7f07f0ff9700 1 -- 172.21.15.158:0/2814115121 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07e4054a60 con 0x7f07f40ca7d0 2023-10-16T10:03:51.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.335+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f07f415d530 con 0x7f07f40ca7d0 2023-10-16T10:03:51.337 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T10:03:51.336+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T10:03:51.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.337+0000 7f07f0ff9700 1 -- 172.21.15.158:0/2814115121 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) v1 ==== 161+0+0 (secure 0 0 0) 0x7f07e4022090 con 0x7f07f40ca7d0 2023-10-16T10:03:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.339+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07f40ca7d0 msgr2=0x7f07f4159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.339+0000 7f07fa88a700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07f40ca7d0 0x7f07f4159740 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f07e4011aa0 tx=0x7f07e4011db0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.339+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 shutdown_connections 2023-10-16T10:03:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.340+0000 7f07fa88a700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07f40cbe00 0x7f07f41437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.340+0000 7f07fa88a700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07dc003a30 0x7f07f40c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.340+0000 7f07fa88a700 1 --2- 172.21.15.158:0/2814115121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07f40ca7d0 0x7f07f4159740 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.340+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 >> 172.21.15.158:0/2814115121 conn(0x7f07f40ce040 msgr2=0x7f07f4152c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.340+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 shutdown_connections 2023-10-16T10:03:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.340+0000 7f07fa88a700 1 -- 172.21.15.158:0/2814115121 wait complete. 2023-10-16T10:03:51.343 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T10:03:51.345 INFO:tasks.ceph.mds.b.smithi172.stderr: -9342> 2023-10-16T10:03:29.190+0000 7f483bbe3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:03:51.353 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T10:03:51.385 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T10:03:51.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.479+0000 7fda82975700 1 Processor -- start 2023-10-16T10:03:51.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.479+0000 7fda82975700 1 -- start start 2023-10-16T10:03:51.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda82975700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fda7c15deb0 0x7fda7c0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda82975700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda7c0ccc00 0x7fda7c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda82975700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0d0ae0 con 0x7fda7c15dad0 2023-10-16T10:03:51.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda82975700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7c0d12d0 con 0x7fda7c15deb0 2023-10-16T10:03:51.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda82975700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7c0cbdf0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda73fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fda7c15deb0 0x7fda7c0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda7bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fda7c15dad0 0x7fda7c0cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59550/0 (socket says 172.21.15.158:59550) 2023-10-16T10:03:51.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.480+0000 7fda7bfff700 1 -- 172.21.15.158:0/1662909242 learned_addr learned my addr 172.21.15.158:0/1662909242 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:51.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda7b7fe700 1 -- 172.21.15.158:0/1662909242 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57455349 0 0) 0x7fda7c0d0ae0 con 0x7fda7c15dad0 2023-10-16T10:03:51.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda80f12700 1 --2- 172.21.15.158:0/1662909242 >> v2:172.21.15.172:3300/0 conn(0x7fda7c0ccc00 0x7fda7c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda7b7fe700 1 -- 172.21.15.158:0/1662909242 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda680035b0 con 0x7fda7c15dad0 2023-10-16T10:03:51.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda73fff700 1 -- 172.21.15.158:0/1662909242 >> v2:172.21.15.172:3300/0 conn(0x7fda7c0ccc00 msgr2=0x7fda7c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:51.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda73fff700 1 --2- 172.21.15.158:0/1662909242 >> v2:172.21.15.172:3300/0 conn(0x7fda7c0ccc00 0x7fda7c0cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda73fff700 1 -- 172.21.15.158:0/1662909242 >> v1:172.21.15.158:6789/0 conn(0x7fda7c15dad0 legacy=0x7fda7c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda73fff700 1 -- 172.21.15.158:0/1662909242 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0cc280 con 0x7fda7c15deb0 2023-10-16T10:03:51.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda73fff700 1 --2- 172.21.15.158:0/1662909242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c15deb0 0x7fda7c0ce6b0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fda64009840 tx=0x7fda64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66aa1726b705162e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:51.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda7b7fe700 1 -- 172.21.15.158:0/1662909242 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64011830 con 0x7fda7c15deb0 2023-10-16T10:03:51.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda7b7fe700 1 -- 172.21.15.158:0/1662909242 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda64011990 con 0x7fda7c15deb0 2023-10-16T10:03:51.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda7b7fe700 1 -- 172.21.15.158:0/1662909242 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64011d20 con 0x7fda7c15deb0 2023-10-16T10:03:51.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda82975700 1 -- 172.21.15.158:0/1662909242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c15deb0 msgr2=0x7fda7c0ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.481+0000 7fda82975700 1 --2- 172.21.15.158:0/1662909242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c15deb0 0x7fda7c0ce6b0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fda64009840 tx=0x7fda64009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:51.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.482+0000 7fda82975700 1 -- 172.21.15.158:0/1662909242 shutdown_connections 2023-10-16T10:03:51.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.482+0000 7fda82975700 1 --2- 172.21.15.158:0/1662909242 >> v2:172.21.15.172:3300/0 conn(0x7fda7c0ccc00 0x7fda7c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.482+0000 7fda82975700 1 --2- 172.21.15.158:0/1662909242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c15deb0 0x7fda7c0ce6b0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.482+0000 7fda82975700 1 -- 172.21.15.158:0/1662909242 >> 172.21.15.158:0/1662909242 conn(0x7fda7c099a10 msgr2=0x7fda7c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:51.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.482+0000 7fda82975700 1 -- 172.21.15.158:0/1662909242 shutdown_connections 2023-10-16T10:03:51.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.482+0000 7fda82975700 1 -- 172.21.15.158:0/1662909242 wait complete. 2023-10-16T10:03:51.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.483+0000 7fda82975700 1 Processor -- start 2023-10-16T10:03:51.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.483+0000 7fda82975700 1 -- start start 2023-10-16T10:03:51.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.483+0000 7fda82975700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c0ccc00 0x7fda7c1e51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.483+0000 7fda82975700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda7c15dad0 0x7fda7c1e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.483+0000 7fda82975700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda7c15deb0 0x7fda7c0cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.483+0000 7fda82975700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c0c74a0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda82975700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c1572a0 con 0x7fda7c15deb0 2023-10-16T10:03:51.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda82975700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c1e5dc0 con 0x7fda7c15dad0 2023-10-16T10:03:51.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c0ccc00 0x7fda7c1e51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c0ccc00 0x7fda7c1e51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59380/0 (socket says 172.21.15.158:59380) 2023-10-16T10:03:51.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 -- 172.21.15.158:0/3826945957 learned_addr learned my addr 172.21.15.158:0/3826945957 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:51.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda80f12700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda7c15deb0 0x7fda7c0cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda73fff700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda7c15dad0 0x7fda7c1e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 -- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda7c15dad0 msgr2=0x7fda7c1e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda7c15dad0 0x7fda7c1e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 -- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda7c15deb0 msgr2=0x7fda7c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:51.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda7c15deb0 0x7fda7c0cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:51.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 -- 172.21.15.158:0/3826945957 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640094a0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda7bfff700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c0ccc00 0x7fda7c1e51e0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00c700 tx=0x7fda6c00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:51.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda797fa700 1 -- 172.21.15.158:0/3826945957 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c005d70 con 0x7fda7c0ccc00 2023-10-16T10:03:51.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda797fa700 1 -- 172.21.15.158:0/3826945957 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6c0070d0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7c0c76f0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda797fa700 1 -- 172.21.15.158:0/3826945957 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c007460 con 0x7fda7c0ccc00 2023-10-16T10:03:51.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.484+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7c0c7cb0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.485+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7c0681b0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.486+0000 7fda797fa700 1 -- 172.21.15.158:0/3826945957 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 83) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fda6c0075c0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.486+0000 7fda797fa700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fda5c06ad50 0x7fda5c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:51.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.486+0000 7fda797fa700 1 -- 172.21.15.158:0/3826945957 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 7298+0+0 (secure 0 0 0) 0x7fda6c088ea0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.486+0000 7fda73fff700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fda5c06ad50 0x7fda5c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:51.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.487+0000 7fda73fff700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fda5c06ad50 0x7fda5c06d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda640090b0 tx=0x7fda6401c040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:51.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.491+0000 7fda797fa700 1 -- 172.21.15.158:0/3826945957 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda6c053bd0 con 0x7fda7c0ccc00 2023-10-16T10:03:51.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:51.726+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fda7c0669e0 con 0x7fda7c0ccc00 2023-10-16T10:03:52.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.370+0000 7fda797fa700 1 -- 172.21.15.158:0/3826945957 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v70) v1 ==== 104+0+0 (secure 0 0 0) 0x7fda6c0571f0 con 0x7fda7c0ccc00 2023-10-16T10:03:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fda5c06ad50 msgr2=0x7fda5c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:52.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fda5c06ad50 0x7fda5c06d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda640090b0 tx=0x7fda6401c040 comp rx=0 tx=0).stop 2023-10-16T10:03:52.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c0ccc00 msgr2=0x7fda7c1e51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:52.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c0ccc00 0x7fda7c1e51e0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00c700 tx=0x7fda6c00cac0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 shutdown_connections 2023-10-16T10:03:52.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda7c15deb0 0x7fda7c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda7c0ccc00 0x7fda7c1e51e0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fda5c06ad50 0x7fda5c06d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 --2- 172.21.15.158:0/3826945957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda7c15dad0 0x7fda7c1e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 >> 172.21.15.158:0/3826945957 conn(0x7fda7c099a10 msgr2=0x7fda7c15c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 shutdown_connections 2023-10-16T10:03:52.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.374+0000 7fda82975700 1 -- 172.21.15.158:0/3826945957 wait complete. 2023-10-16T10:03:52.390 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T10:03:52.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.523+0000 7fcb2beb8700 1 Processor -- start 2023-10-16T10:03:52.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.523+0000 7fcb2beb8700 1 -- start start 2023-10-16T10:03:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb2beb8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415dad0 0x7fcb240cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb2beb8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb240ccc00 0x7fcb240cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:52.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb2beb8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb240d0ae0 con 0x7fcb2415deb0 2023-10-16T10:03:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb2beb8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb240d12d0 con 0x7fcb240ccc00 2023-10-16T10:03:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb2beb8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240cbdf0 con 0x7fcb2415dad0 2023-10-16T10:03:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb2a455700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb240ccc00 0x7fcb240cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb29453700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcb2415deb0 0x7fcb240ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59584/0 (socket says 172.21.15.158:59584) 2023-10-16T10:03:52.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb29453700 1 -- 172.21.15.158:0/302613634 learned_addr learned my addr 172.21.15.158:0/302613634 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:52.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.524+0000 7fcb2a455700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb240ccc00 0x7fcb240cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59410/0 (socket says 172.21.15.158:59410) 2023-10-16T10:03:52.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb29c54700 1 --2- 172.21.15.158:0/302613634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415dad0 0x7fcb240cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:52.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb2a455700 1 -- 172.21.15.158:0/302613634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415dad0 msgr2=0x7fcb240cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:52.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb2a455700 1 --2- 172.21.15.158:0/302613634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415dad0 0x7fcb240cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb2a455700 1 -- 172.21.15.158:0/302613634 >> v1:172.21.15.158:6789/0 conn(0x7fcb2415deb0 legacy=0x7fcb240ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:52.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb28c52700 1 -- 172.21.15.158:0/302613634 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222696999 0 0) 0x7fcb240d0ae0 con 0x7fcb2415deb0 2023-10-16T10:03:52.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb2a455700 1 -- 172.21.15.158:0/302613634 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb240cc280 con 0x7fcb240ccc00 2023-10-16T10:03:52.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb29c54700 1 --2- 172.21.15.158:0/302613634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415dad0 0x7fcb240cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:52.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb2a455700 1 --2- 172.21.15.158:0/302613634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb240ccc00 0x7fcb240cd0b0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00bfa0 tx=0x7fcb1c00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da642502600ca06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:52.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb28c52700 1 -- 172.21.15.158:0/302613634 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c010070 con 0x7fcb240ccc00 2023-10-16T10:03:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb28c52700 1 -- 172.21.15.158:0/302613634 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c00b2e0 con 0x7fcb240ccc00 2023-10-16T10:03:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.525+0000 7fcb28c52700 1 -- 172.21.15.158:0/302613634 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c00dcf0 con 0x7fcb240ccc00 2023-10-16T10:03:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 -- 172.21.15.158:0/302613634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb240ccc00 msgr2=0x7fcb240cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/302613634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb240ccc00 0x7fcb240cd0b0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00bfa0 tx=0x7fcb1c00d330 comp rx=0 tx=0).stop 2023-10-16T10:03:52.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 -- 172.21.15.158:0/302613634 shutdown_connections 2023-10-16T10:03:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/302613634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb240ccc00 0x7fcb240cd0b0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/302613634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415dad0 0x7fcb240cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 -- 172.21.15.158:0/302613634 >> 172.21.15.158:0/302613634 conn(0x7fcb24099a10 msgr2=0x7fcb240d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 -- 172.21.15.158:0/302613634 shutdown_connections 2023-10-16T10:03:52.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.526+0000 7fcb2beb8700 1 -- 172.21.15.158:0/302613634 wait complete. 2023-10-16T10:03:52.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.527+0000 7fcb2beb8700 1 Processor -- start 2023-10-16T10:03:52.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.527+0000 7fcb2beb8700 1 -- start start 2023-10-16T10:03:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.527+0000 7fcb2beb8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb240ccc00 0x7fcb2414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.527+0000 7fcb2beb8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb2415dad0 0x7fcb240b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb2beb8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415deb0 0x7fcb2415c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb2beb8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240c74a0 con 0x7fcb2415dad0 2023-10-16T10:03:52.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb2beb8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240b4ad0 con 0x7fcb240ccc00 2023-10-16T10:03:52.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb2beb8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240b4db0 con 0x7fcb2415deb0 2023-10-16T10:03:52.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb2415dad0 0x7fcb240b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:52.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb2415dad0 0x7fcb240b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59414/0 (socket says 172.21.15.158:59414) 2023-10-16T10:03:52.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 -- 172.21.15.158:0/1328716776 learned_addr learned my addr 172.21.15.158:0/1328716776 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:52.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29c54700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb240ccc00 0x7fcb2414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:52.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb2a455700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415deb0 0x7fcb2415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:52.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 -- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415deb0 msgr2=0x7fcb2415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:52.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415deb0 0x7fcb2415c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 -- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb240ccc00 msgr2=0x7fcb2414b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:52.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb240ccc00 0x7fcb2414b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:52.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 -- 172.21.15.158:0/1328716776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c00bc80 con 0x7fcb2415dad0 2023-10-16T10:03:52.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb29453700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb2415dad0 0x7fcb240b4010 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000c520 tx=0x7fcb2000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:52.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb1a7fc700 1 -- 172.21.15.158:0/1328716776 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20004c80 con 0x7fcb2415dad0 2023-10-16T10:03:52.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb1a7fc700 1 -- 172.21.15.158:0/1328716776 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb20004de0 con 0x7fcb2415dad0 2023-10-16T10:03:52.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb240c76f0 con 0x7fcb2415dad0 2023-10-16T10:03:52.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb1a7fc700 1 -- 172.21.15.158:0/1328716776 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200072d0 con 0x7fcb2415dad0 2023-10-16T10:03:52.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.528+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb240b5de0 con 0x7fcb2415dad0 2023-10-16T10:03:52.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.529+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb240681b0 con 0x7fcb2415dad0 2023-10-16T10:03:52.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.530+0000 7fcb1a7fc700 1 -- 172.21.15.158:0/1328716776 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 83) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fcb20007430 con 0x7fcb2415dad0 2023-10-16T10:03:52.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.530+0000 7fcb1a7fc700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fcb0806ad50 0x7fcb0806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:52.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.530+0000 7fcb1a7fc700 1 -- 172.21.15.158:0/1328716776 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 7298+0+0 (secure 0 0 0) 0x7fcb20089560 con 0x7fcb2415dad0 2023-10-16T10:03:52.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.530+0000 7fcb29c54700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fcb0806ad50 0x7fcb0806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:52.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.530+0000 7fcb29c54700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fcb0806ad50 0x7fcb0806d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009930 tx=0x7fcb14008040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:52.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.534+0000 7fcb1a7fc700 1 -- 172.21.15.158:0/1328716776 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb20054310 con 0x7fcb2415dad0 2023-10-16T10:03:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:52.725+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcb240cb630 con 0x7fcb2415dad0 2023-10-16T10:03:53.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.377+0000 7fcb1a7fc700 1 -- 172.21.15.158:0/1328716776 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v144) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcb2005ce60 con 0x7fcb2415dad0 2023-10-16T10:03:53.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.381+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fcb0806ad50 msgr2=0x7fcb0806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:53.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.381+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fcb0806ad50 0x7fcb0806d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009930 tx=0x7fcb14008040 comp rx=0 tx=0).stop 2023-10-16T10:03:53.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.381+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb2415dad0 msgr2=0x7fcb240b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:53.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.381+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb2415dad0 0x7fcb240b4010 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000c520 tx=0x7fcb2000c8e0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.385+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 shutdown_connections 2023-10-16T10:03:53.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.385+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fcb0806ad50 0x7fcb0806d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.385+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb2415deb0 0x7fcb2415c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.385+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb240ccc00 0x7fcb2414b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.385+0000 7fcb2beb8700 1 --2- 172.21.15.158:0/1328716776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb2415dad0 0x7fcb240b4010 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.385+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 >> 172.21.15.158:0/1328716776 conn(0x7fcb24099a10 msgr2=0x7fcb2415a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:53.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.385+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 shutdown_connections 2023-10-16T10:03:53.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.386+0000 7fcb2beb8700 1 -- 172.21.15.158:0/1328716776 wait complete. 2023-10-16T10:03:53.388 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T10:03:53.400 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T10:03:53.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.525+0000 7fc0e0ea7700 1 Processor -- start 2023-10-16T10:03:53.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.525+0000 7fc0e0ea7700 1 -- start start 2023-10-16T10:03:53.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0e0ea7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0dc15deb0 0x7fc0dc0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:53.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0e0ea7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0dc0d0ae0 con 0x7fc0dc0ccc00 2023-10-16T10:03:53.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0e0ea7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0dc0d12d0 con 0x7fc0dc15dad0 2023-10-16T10:03:53.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0e0ea7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0dc0cbdf0 con 0x7fc0dc15deb0 2023-10-16T10:03:53.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d9d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0dc15deb0 0x7fc0dc0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:53.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0dad9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc0dc0ccc00 0x7fc0dc0cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59600/0 (socket says 172.21.15.158:59600) 2023-10-16T10:03:53.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0dad9d700 1 -- 172.21.15.158:0/3637207022 learned_addr learned my addr 172.21.15.158:0/3637207022 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:53.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d9d9b700 1 -- 172.21.15.158:0/3637207022 >> v1:172.21.15.172:6789/0 conn(0x7fc0dc15dad0 legacy=0x7fc0dc0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:53.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d9d9b700 1 -- 172.21.15.158:0/3637207022 >> v1:172.21.15.158:6789/0 conn(0x7fc0dc0ccc00 legacy=0x7fc0dc0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:53.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d9d9b700 1 -- 172.21.15.158:0/3637207022 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc0cc280 con 0x7fc0dc15deb0 2023-10-16T10:03:53.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d9d9b700 1 --2- 172.21.15.158:0/3637207022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc0ce6b0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009870 tx=0x7fc0cc009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e79a2164d1b2163 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:53.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d959a700 1 -- 172.21.15.158:0/3637207022 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc004230 con 0x7fc0dc15deb0 2023-10-16T10:03:53.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d959a700 1 -- 172.21.15.158:0/3637207022 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0cc004390 con 0x7fc0dc15deb0 2023-10-16T10:03:53.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.526+0000 7fc0d959a700 1 -- 172.21.15.158:0/3637207022 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc0036a0 con 0x7fc0dc15deb0 2023-10-16T10:03:53.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.527+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3637207022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 msgr2=0x7fc0dc0ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:53.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.527+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3637207022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc0ce6b0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009870 tx=0x7fc0cc009b80 comp rx=0 tx=0).stop 2023-10-16T10:03:53.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.527+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3637207022 shutdown_connections 2023-10-16T10:03:53.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.527+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3637207022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc0ce6b0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.527+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3637207022 >> 172.21.15.158:0/3637207022 conn(0x7fc0dc099a10 msgr2=0x7fc0dc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:53.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.527+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3637207022 shutdown_connections 2023-10-16T10:03:53.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.527+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3637207022 wait complete. 2023-10-16T10:03:53.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.528+0000 7fc0e0ea7700 1 Processor -- start 2023-10-16T10:03:53.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.528+0000 7fc0e0ea7700 1 -- start start 2023-10-16T10:03:53.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0e0ea7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0ccc00 0x7fc0dc0c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:53.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0e0ea7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0dc15dad0 0x7fc0dc0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:53.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0e0ea7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc15a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:53.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0e0ea7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0b49e0 con 0x7fc0dc15deb0 2023-10-16T10:03:53.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0e0ea7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0b4c80 con 0x7fc0dc0ccc00 2023-10-16T10:03:53.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0e0ea7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0b4f60 con 0x7fc0dc15dad0 2023-10-16T10:03:53.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0dad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc15a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:53.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0dad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc15a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59430/0 (socket says 172.21.15.158:59430) 2023-10-16T10:03:53.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0dad9d700 1 -- 172.21.15.158:0/3133414980 learned_addr learned my addr 172.21.15.158:0/3133414980 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:53.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0d9d9b700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0dc15dad0 0x7fc0dc0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:53.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.529+0000 7fc0da59c700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0ccc00 0x7fc0dc0c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:53.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0dad9d700 1 -- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0dc15dad0 msgr2=0x7fc0dc0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:53.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0dad9d700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0dc15dad0 0x7fc0dc0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0dad9d700 1 -- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0ccc00 msgr2=0x7fc0dc0c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:53.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0dad9d700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0ccc00 0x7fc0dc0c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:53.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0dad9d700 1 -- 172.21.15.158:0/3133414980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc0094a0 con 0x7fc0dc15deb0 2023-10-16T10:03:53.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0d9d9b700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0dc15dad0 0x7fc0dc0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:53.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0dad9d700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc15a420 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400ee60 tx=0x7fc0d400c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:53.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0caffd700 1 -- 172.21.15.158:0/3133414980 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4011670 con 0x7fc0dc15deb0 2023-10-16T10:03:53.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0caffd700 1 -- 172.21.15.158:0/3133414980 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d40117d0 con 0x7fc0dc15deb0 2023-10-16T10:03:53.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0dc0b5240 con 0x7fc0dc15deb0 2023-10-16T10:03:53.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0caffd700 1 -- 172.21.15.158:0/3133414980 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4011b40 con 0x7fc0dc15deb0 2023-10-16T10:03:53.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.530+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0dc0b5800 con 0x7fc0dc15deb0 2023-10-16T10:03:53.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.531+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0dc0681b0 con 0x7fc0dc15deb0 2023-10-16T10:03:53.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.531+0000 7fc0caffd700 1 -- 172.21.15.158:0/3133414980 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 84) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc0d4011ca0 con 0x7fc0dc15deb0 2023-10-16T10:03:53.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.532+0000 7fc0caffd700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fc0bc06af90 0x7fc0bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:53.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.532+0000 7fc0caffd700 1 -- 172.21.15.158:0/3133414980 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6860+0+0 (secure 0 0 0) 0x7fc0d408c840 con 0x7fc0dc15deb0 2023-10-16T10:03:53.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.532+0000 7fc0da59c700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fc0bc06af90 0x7fc0bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:53.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.532+0000 7fc0da59c700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fc0bc06af90 0x7fc0bc06d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0004f30 tx=0x7fc0d0009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:53.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.536+0000 7fc0caffd700 1 -- 172.21.15.158:0/3133414980 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0d4057370 con 0x7fc0dc15deb0 2023-10-16T10:03:53.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:53.731+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc0dc0ce480 con 0x7fc0dc15deb0 2023-10-16T10:03:54.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.616+0000 7fc0caffd700 1 -- 172.21.15.158:0/3133414980 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v145) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc0d405a990 con 0x7fc0dc15deb0 2023-10-16T10:03:54.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.620+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fc0bc06af90 msgr2=0x7fc0bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.620+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fc0bc06af90 0x7fc0bc06d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0004f30 tx=0x7fc0d0009040 comp rx=0 tx=0).stop 2023-10-16T10:03:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.620+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 msgr2=0x7fc0dc15a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.620+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc15a420 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400ee60 tx=0x7fc0d400c550 comp rx=0 tx=0).stop 2023-10-16T10:03:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 shutdown_connections 2023-10-16T10:03:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0dc15deb0 0x7fc0dc15a420 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0ccc00 0x7fc0dc0c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fc0bc06af90 0x7fc0bc06d450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 --2- 172.21.15.158:0/3133414980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0dc15dad0 0x7fc0dc0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 >> 172.21.15.158:0/3133414980 conn(0x7fc0dc099a10 msgr2=0x7fc0dc0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:54.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 shutdown_connections 2023-10-16T10:03:54.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.621+0000 7fc0e0ea7700 1 -- 172.21.15.158:0/3133414980 wait complete. 2023-10-16T10:03:54.625 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T10:03:54.638 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T10:03:54.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.777+0000 7f5a3591a700 1 Processor -- start 2023-10-16T10:03:54.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.777+0000 7f5a3591a700 1 -- start start 2023-10-16T10:03:54.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.777+0000 7f5a3591a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5a300cd480 0x7f5a300b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:54.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.777+0000 7f5a3591a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5a30160fd0 0x7f5a300cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:54.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.777+0000 7f5a3591a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300ce9d0 con 0x7f5a30160bf0 2023-10-16T10:03:54.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.777+0000 7f5a3591a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a300c9c40 con 0x7f5a300cd480 2023-10-16T10:03:54.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.777+0000 7f5a3591a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a300c9f20 con 0x7f5a30160fd0 2023-10-16T10:03:54.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5a300cd480 0x7f5a300b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:54.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5a30160bf0 0x7f5a300ce2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59610/0 (socket says 172.21.15.158:59610) 2023-10-16T10:03:54.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2effd700 1 -- 172.21.15.158:0/2766874543 learned_addr learned my addr 172.21.15.158:0/2766874543 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:54.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2f7fe700 1 --2- 172.21.15.158:0/2766874543 >> v2:172.21.15.172:3300/0 conn(0x7f5a30160fd0 0x7f5a300cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:54.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2e7fc700 1 -- 172.21.15.158:0/2766874543 >> v2:172.21.15.172:3300/0 conn(0x7f5a30160fd0 msgr2=0x7f5a300cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:54.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2e7fc700 1 --2- 172.21.15.158:0/2766874543 >> v2:172.21.15.172:3300/0 conn(0x7f5a30160fd0 0x7f5a300cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2e7fc700 1 -- 172.21.15.158:0/2766874543 >> v1:172.21.15.158:6789/0 conn(0x7f5a30160bf0 legacy=0x7f5a300ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2dffb700 1 -- 172.21.15.158:0/2766874543 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315041409 0 0) 0x7f5a300ce9d0 con 0x7f5a30160bf0 2023-10-16T10:03:54.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2e7fc700 1 -- 172.21.15.158:0/2766874543 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300ca200 con 0x7f5a300cd480 2023-10-16T10:03:54.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2f7fe700 1 --2- 172.21.15.158:0/2766874543 >> v2:172.21.15.172:3300/0 conn(0x7f5a30160fd0 0x7f5a300cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:54.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2e7fc700 1 --2- 172.21.15.158:0/2766874543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300b2fc0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f5a300b1f50 tx=0x7f5a24009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81960422b97d29f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:54.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2dffb700 1 -- 172.21.15.158:0/2766874543 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a24011830 con 0x7f5a300cd480 2023-10-16T10:03:54.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2dffb700 1 -- 172.21.15.158:0/2766874543 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a24011990 con 0x7f5a300cd480 2023-10-16T10:03:54.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.778+0000 7f5a2dffb700 1 -- 172.21.15.158:0/2766874543 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a24011d20 con 0x7f5a300cd480 2023-10-16T10:03:54.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 -- 172.21.15.158:0/2766874543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 msgr2=0x7f5a300b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 --2- 172.21.15.158:0/2766874543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300b2fc0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f5a300b1f50 tx=0x7f5a24009b20 comp rx=0 tx=0).stop 2023-10-16T10:03:54.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 -- 172.21.15.158:0/2766874543 shutdown_connections 2023-10-16T10:03:54.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 --2- 172.21.15.158:0/2766874543 >> v2:172.21.15.172:3300/0 conn(0x7f5a30160fd0 0x7f5a300cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 --2- 172.21.15.158:0/2766874543 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300b2fc0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 -- 172.21.15.158:0/2766874543 >> 172.21.15.158:0/2766874543 conn(0x7f5a3006a8d0 msgr2=0x7f5a3006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:54.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 -- 172.21.15.158:0/2766874543 shutdown_connections 2023-10-16T10:03:54.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.779+0000 7f5a3591a700 1 -- 172.21.15.158:0/2766874543 wait complete. 2023-10-16T10:03:54.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.780+0000 7f5a3591a700 1 Processor -- start 2023-10-16T10:03:54.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.780+0000 7f5a3591a700 1 -- start start 2023-10-16T10:03:54.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a3591a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:54.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a3591a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30160bf0 0x7f5a301f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:54.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a3591a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a30160fd0 0x7f5a300c5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:54.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a3591a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300c40a0 con 0x7f5a300cd480 2023-10-16T10:03:54.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a3591a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301f3b90 con 0x7f5a30160fd0 2023-10-16T10:03:54.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a3591a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300c3490 con 0x7f5a30160bf0 2023-10-16T10:03:54.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:54.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59440/0 (socket says 172.21.15.158:59440) 2023-10-16T10:03:54.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 -- 172.21.15.158:0/3107548787 learned_addr learned my addr 172.21.15.158:0/3107548787 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:54.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 -- 172.21.15.158:0/3107548787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30160bf0 msgr2=0x7f5a301f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30160bf0 0x7f5a301f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2f7fe700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a30160fd0 0x7f5a300c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:54.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 -- 172.21.15.158:0/3107548787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a30160fd0 msgr2=0x7f5a300c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a30160fd0 0x7f5a300c5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 -- 172.21.15.158:0/3107548787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240094a0 con 0x7f5a300cd480 2023-10-16T10:03:54.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.781+0000 7f5a2effd700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300c82e0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f5a200011e0 tx=0x7f5a2000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:54.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.782+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20004c80 con 0x7f5a300cd480 2023-10-16T10:03:54.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.782+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a20004e00 con 0x7f5a300cd480 2023-10-16T10:03:54.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.782+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a300c3770 con 0x7f5a300cd480 2023-10-16T10:03:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.782+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20007310 con 0x7f5a300cd480 2023-10-16T10:03:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.782+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a300c3d30 con 0x7f5a300cd480 2023-10-16T10:03:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.783+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a300c4430 con 0x7f5a300cd480 2023-10-16T10:03:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.783+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 84) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5a20007470 con 0x7f5a300cd480 2023-10-16T10:03:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.783+0000 7f5a17fff700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a1c06af90 0x7f5a1c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:54.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.784+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7f5a20089760 con 0x7f5a300cd480 2023-10-16T10:03:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.784+0000 7f5a2e7fc700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a1c06af90 0x7f5a1c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.784+0000 7f5a2e7fc700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a1c06af90 0x7f5a1c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a300b1f50 tx=0x7f5a24010000 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:54.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.788+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a200544d0 con 0x7f5a300cd480 2023-10-16T10:03:54.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.898+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a20087990 con 0x7f5a300cd480 2023-10-16T10:03:54.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.985+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5a300681b0 con 0x7f5a300cd480 2023-10-16T10:03:54.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.988+0000 7f5a17fff700 1 -- 172.21.15.158:0/3107548787 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5a20073ba0 con 0x7f5a300cd480 2023-10-16T10:03:54.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a1c06af90 msgr2=0x7f5a1c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a1c06af90 0x7f5a1c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a300b1f50 tx=0x7f5a24010000 comp rx=0 tx=0).stop 2023-10-16T10:03:54.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 msgr2=0x7f5a300c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:54.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300c82e0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f5a200011e0 tx=0x7f5a2000cae0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 shutdown_connections 2023-10-16T10:03:54.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a30160fd0 0x7f5a300c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a300cd480 0x7f5a300c82e0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a1c06af90 0x7f5a1c06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 --2- 172.21.15.158:0/3107548787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30160bf0 0x7f5a301f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:54.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 >> 172.21.15.158:0/3107548787 conn(0x7f5a3006a8d0 msgr2=0x7f5a30156d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:54.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 shutdown_connections 2023-10-16T10:03:54.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:54.991+0000 7f5a3591a700 1 -- 172.21.15.158:0/3107548787 wait complete. 2023-10-16T10:03:55.004 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T10:03:55.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.125+0000 7f9790fdc700 1 Processor -- start 2023-10-16T10:03:55.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.125+0000 7f9790fdc700 1 -- start start 2023-10-16T10:03:55.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.126+0000 7f9790fdc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f978c160d30 0x7f978c0b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.126+0000 7f9790fdc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c0d0530 0x7f978c0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.126+0000 7f9790fdc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f978c0d19a0 0x7f978c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.126+0000 7f9790fdc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c0b8910 con 0x7f978c0d19a0 2023-10-16T10:03:55.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.126+0000 7f9790fdc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c0d0ef0 con 0x7f978c160d30 2023-10-16T10:03:55.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.126+0000 7f9790fdc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b7c90 con 0x7f978c0d0530 2023-10-16T10:03:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.126+0000 7f978ad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f978c0d19a0 0x7f978c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f978c0d19a0 0x7f978c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59452/0 (socket says 172.21.15.158:59452) 2023-10-16T10:03:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 -- 172.21.15.158:0/4140315586 learned_addr learned my addr 172.21.15.158:0/4140315586 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f9789d9b700 1 --2- 172.21.15.158:0/4140315586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c0d0530 0x7f978c0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978a59c700 1 --2- 172.21.15.158:0/4140315586 >> v2:172.21.15.172:3300/0 conn(0x7f978c160d30 0x7f978c0b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 -- 172.21.15.158:0/4140315586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c0d0530 msgr2=0x7f978c0d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 --2- 172.21.15.158:0/4140315586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c0d0530 0x7f978c0d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 -- 172.21.15.158:0/4140315586 >> v2:172.21.15.172:3300/0 conn(0x7f978c160d30 msgr2=0x7f978c0b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 --2- 172.21.15.158:0/4140315586 >> v2:172.21.15.172:3300/0 conn(0x7f978c160d30 0x7f978c0b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 -- 172.21.15.158:0/4140315586 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0b7f70 con 0x7f978c0d19a0 2023-10-16T10:03:55.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978ad9d700 1 --2- 172.21.15.158:0/4140315586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d19a0 0x7f978c0d1e30 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f978c0d4690 tx=0x7f978400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af973efa26f4e391 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978959a700 1 -- 172.21.15.158:0/4140315586 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978400e070 con 0x7f978c0d19a0 2023-10-16T10:03:55.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978959a700 1 -- 172.21.15.158:0/4140315586 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978400b9c0 con 0x7f978c0d19a0 2023-10-16T10:03:55.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.127+0000 7f978959a700 1 -- 172.21.15.158:0/4140315586 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978400bd70 con 0x7f978c0d19a0 2023-10-16T10:03:55.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 -- 172.21.15.158:0/4140315586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d19a0 msgr2=0x7f978c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 --2- 172.21.15.158:0/4140315586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d19a0 0x7f978c0d1e30 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f978c0d4690 tx=0x7f978400b330 comp rx=0 tx=0).stop 2023-10-16T10:03:55.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 -- 172.21.15.158:0/4140315586 shutdown_connections 2023-10-16T10:03:55.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 --2- 172.21.15.158:0/4140315586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d19a0 0x7f978c0d1e30 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 --2- 172.21.15.158:0/4140315586 >> v2:172.21.15.172:3300/0 conn(0x7f978c160d30 0x7f978c0b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 --2- 172.21.15.158:0/4140315586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c0d0530 0x7f978c0d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 -- 172.21.15.158:0/4140315586 >> 172.21.15.158:0/4140315586 conn(0x7f978c0669f0 msgr2=0x7f978c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:55.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 -- 172.21.15.158:0/4140315586 shutdown_connections 2023-10-16T10:03:55.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.128+0000 7f9790fdc700 1 -- 172.21.15.158:0/4140315586 wait complete. 2023-10-16T10:03:55.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.129+0000 7f9790fdc700 1 Processor -- start 2023-10-16T10:03:55.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.129+0000 7f9790fdc700 1 -- start start 2023-10-16T10:03:55.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f9790fdc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d0530 0x7f978c0cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f9790fdc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f978c0d19a0 0x7f978c0c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f9790fdc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c160d30 0x7f978c1572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f9790fdc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0c9fb0 con 0x7f978c0d0530 2023-10-16T10:03:55.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f9790fdc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0c99f0 con 0x7f978c0d19a0 2023-10-16T10:03:55.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f9790fdc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0c9cd0 con 0x7f978c160d30 2023-10-16T10:03:55.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d0530 0x7f978c0cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d0530 0x7f978c0cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59454/0 (socket says 172.21.15.158:59454) 2023-10-16T10:03:55.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 -- 172.21.15.158:0/79433840 learned_addr learned my addr 172.21.15.158:0/79433840 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:55.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f9789d9b700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f978c0d19a0 0x7f978c0c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978ad9d700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c160d30 0x7f978c1572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 -- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c160d30 msgr2=0x7f978c1572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c160d30 0x7f978c1572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 -- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f978c0d19a0 msgr2=0x7f978c0c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f978c0d19a0 0x7f978c0c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.130+0000 7f978a59c700 1 -- 172.21.15.158:0/79433840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784009b30 con 0x7f978c0d0530 2023-10-16T10:03:55.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.131+0000 7f978a59c700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d0530 0x7f978c0cbd50 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f977400d800 tx=0x7f977400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.131+0000 7f9782ffd700 1 -- 172.21.15.158:0/79433840 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740125f0 con 0x7f978c0d0530 2023-10-16T10:03:55.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.131+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c0c7290 con 0x7f978c0d0530 2023-10-16T10:03:55.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.131+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c0c9700 con 0x7f978c0d0530 2023-10-16T10:03:55.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.131+0000 7f9782ffd700 1 -- 172.21.15.158:0/79433840 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9774012750 con 0x7f978c0d0530 2023-10-16T10:03:55.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.131+0000 7f9782ffd700 1 -- 172.21.15.158:0/79433840 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774012ac0 con 0x7f978c0d0530 2023-10-16T10:03:55.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.132+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c0c78e0 con 0x7f978c0d0530 2023-10-16T10:03:55.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.132+0000 7f9782ffd700 1 -- 172.21.15.158:0/79433840 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9774012c20 con 0x7f978c0d0530 2023-10-16T10:03:55.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.133+0000 7f9782ffd700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f976c06b020 0x7f976c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.133+0000 7f9782ffd700 1 -- 172.21.15.158:0/79433840 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7f9774089a60 con 0x7f978c0d0530 2023-10-16T10:03:55.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.133+0000 7f9789d9b700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f976c06b020 0x7f976c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.133+0000 7f9789d9b700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f976c06b020 0x7f976c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f978c1542d0 tx=0x7f977c008040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.137+0000 7f9782ffd700 1 -- 172.21.15.158:0/79433840 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97740546a0 con 0x7f978c0d0530 2023-10-16T10:03:55.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.331+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f978c0c70f0 con 0x7f978c0d0530 2023-10-16T10:03:55.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.334+0000 7f9782ffd700 1 -- 172.21.15.158:0/79433840 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f978c0c70f0 con 0x7f978c0d0530 2023-10-16T10:03:55.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.336+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f976c06b020 msgr2=0x7f976c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.336+0000 7f9790fdc700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f976c06b020 0x7f976c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f978c1542d0 tx=0x7f977c008040 comp rx=0 tx=0).stop 2023-10-16T10:03:55.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.336+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d0530 msgr2=0x7f978c0cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.336+0000 7f9790fdc700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d0530 0x7f978c0cbd50 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f977400d800 tx=0x7f977400db10 comp rx=0 tx=0).stop 2023-10-16T10:03:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 shutdown_connections 2023-10-16T10:03:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f976c06b020 0x7f976c06d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f978c160d30 0x7f978c1572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f978c0d0530 0x7f978c0cbd50 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 --2- 172.21.15.158:0/79433840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f978c0d19a0 0x7f978c0c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 >> 172.21.15.158:0/79433840 conn(0x7f978c0669f0 msgr2=0x7f978c15f330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 shutdown_connections 2023-10-16T10:03:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.337+0000 7f9790fdc700 1 -- 172.21.15.158:0/79433840 wait complete. 2023-10-16T10:03:55.350 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T10:03:55.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.471+0000 7fdee5221700 1 Processor -- start 2023-10-16T10:03:55.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.471+0000 7fdee5221700 1 -- start start 2023-10-16T10:03:55.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdee5221700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdee006afe0 0x7fdee00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdee5221700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee00cc590 0x7fdee00afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdee5221700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00cbd30 con 0x7fdee015da40 2023-10-16T10:03:55.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdee5221700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee00cc1c0 con 0x7fdee006afe0 2023-10-16T10:03:55.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdee5221700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee015de20 con 0x7fdee00cc590 2023-10-16T10:03:55.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdede59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdee006afe0 0x7fdee00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdeded9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdee015da40 0x7fdee006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59638/0 (socket says 172.21.15.158:59638) 2023-10-16T10:03:55.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdeded9d700 1 -- 172.21.15.158:0/4198952846 learned_addr learned my addr 172.21.15.158:0/4198952846 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:55.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdede59c700 1 -- 172.21.15.158:0/4198952846 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee00cc590 msgr2=0x7fdee00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:55.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdede59c700 1 --2- 172.21.15.158:0/4198952846 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee00cc590 0x7fdee00afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdede59c700 1 -- 172.21.15.158:0/4198952846 >> v1:172.21.15.158:6789/0 conn(0x7fdee015da40 legacy=0x7fdee006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdede59c700 1 -- 172.21.15.158:0/4198952846 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00ce260 con 0x7fdee006afe0 2023-10-16T10:03:55.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdeddd9b700 1 -- 172.21.15.158:0/4198952846 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609762657 0 0) 0x7fdee00cbd30 con 0x7fdee015da40 2023-10-16T10:03:55.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.472+0000 7fdede59c700 1 --2- 172.21.15.158:0/4198952846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee006afe0 0x7fdee00caa20 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fded4009840 tx=0x7fded4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80da40e8ee500c6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdeddd9b700 1 -- 172.21.15.158:0/4198952846 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4011830 con 0x7fdee006afe0 2023-10-16T10:03:55.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdeddd9b700 1 -- 172.21.15.158:0/4198952846 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded4011990 con 0x7fdee006afe0 2023-10-16T10:03:55.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdeddd9b700 1 -- 172.21.15.158:0/4198952846 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4011d20 con 0x7fdee006afe0 2023-10-16T10:03:55.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 -- 172.21.15.158:0/4198952846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee006afe0 msgr2=0x7fdee00caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 --2- 172.21.15.158:0/4198952846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee006afe0 0x7fdee00caa20 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fded4009840 tx=0x7fded4009b50 comp rx=0 tx=0).stop 2023-10-16T10:03:55.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 -- 172.21.15.158:0/4198952846 shutdown_connections 2023-10-16T10:03:55.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 --2- 172.21.15.158:0/4198952846 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee00cc590 0x7fdee00afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 --2- 172.21.15.158:0/4198952846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee006afe0 0x7fdee00caa20 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 -- 172.21.15.158:0/4198952846 >> 172.21.15.158:0/4198952846 conn(0x7fdee0099a10 msgr2=0x7fdee00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:55.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 -- 172.21.15.158:0/4198952846 shutdown_connections 2023-10-16T10:03:55.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.473+0000 7fdee5221700 1 -- 172.21.15.158:0/4198952846 wait complete. 2023-10-16T10:03:55.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 Processor -- start 2023-10-16T10:03:55.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 -- start start 2023-10-16T10:03:55.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdee006afe0 0x7fdee015a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee00cc590 0x7fdee00c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee015da40 0x7fdee01e70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c4060 con 0x7fdee00cc590 2023-10-16T10:03:55.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee015aa90 con 0x7fdee006afe0 2023-10-16T10:03:55.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdee5221700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c5920 con 0x7fdee015da40 2023-10-16T10:03:55.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdede59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee00cc590 0x7fdee00c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdede59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee00cc590 0x7fdee00c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59470/0 (socket says 172.21.15.158:59470) 2023-10-16T10:03:55.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.475+0000 7fdede59c700 1 -- 172.21.15.158:0/2182526021 learned_addr learned my addr 172.21.15.158:0/2182526021 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:55.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdedf59e700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee015da40 0x7fdee01e70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdeded9d700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdee006afe0 0x7fdee015a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdede59c700 1 -- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee015da40 msgr2=0x7fdee01e70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdede59c700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee015da40 0x7fdee01e70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdede59c700 1 -- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdee006afe0 msgr2=0x7fdee015a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdede59c700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdee006afe0 0x7fdee015a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdede59c700 1 -- 172.21.15.158:0/2182526021 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded40094a0 con 0x7fdee00cc590 2023-10-16T10:03:55.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdede59c700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee00cc590 0x7fdee00c7a20 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fded4014040 tx=0x7fded4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdec77fe700 1 -- 172.21.15.158:0/2182526021 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40045f0 con 0x7fdee00cc590 2023-10-16T10:03:55.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdec77fe700 1 -- 172.21.15.158:0/2182526021 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded401e070 con 0x7fdee00cc590 2023-10-16T10:03:55.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee00c6410 con 0x7fdee00cc590 2023-10-16T10:03:55.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdec77fe700 1 -- 172.21.15.158:0/2182526021 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4003b00 con 0x7fdee00cc590 2023-10-16T10:03:55.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.476+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee00c7f60 con 0x7fdee00cc590 2023-10-16T10:03:55.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.477+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee00681b0 con 0x7fdee00cc590 2023-10-16T10:03:55.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.477+0000 7fdec77fe700 1 -- 172.21.15.158:0/2182526021 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fded4003c60 con 0x7fdee00cc590 2023-10-16T10:03:55.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.478+0000 7fdec77fe700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fdecc06b020 0x7fdecc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.478+0000 7fdec77fe700 1 -- 172.21.15.158:0/2182526021 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7fded408ab80 con 0x7fdee00cc590 2023-10-16T10:03:55.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.478+0000 7fdeded9d700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fdecc06b020 0x7fdecc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.478+0000 7fdeded9d700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fdecc06b020 0x7fdecc06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fded0000f60 tx=0x7fded0007be0 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.482+0000 7fdec77fe700 1 -- 172.21.15.158:0/2182526021 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fded40558d0 con 0x7fdee00cc590 2023-10-16T10:03:55.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.678+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fdee00cb9b0 con 0x7fdee00cc590 2023-10-16T10:03:55.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.681+0000 7fdec77fe700 1 -- 172.21.15.158:0/2182526021 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7fded40150b0 con 0x7fdee00cc590 2023-10-16T10:03:55.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.683+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fdecc06b020 msgr2=0x7fdecc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.683+0000 7fdee5221700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fdecc06b020 0x7fdecc06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fded0000f60 tx=0x7fded0007be0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.683+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee00cc590 msgr2=0x7fdee00c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.683+0000 7fdee5221700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee00cc590 0x7fdee00c7a20 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fded4014040 tx=0x7fded4004750 comp rx=0 tx=0).stop 2023-10-16T10:03:55.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 shutdown_connections 2023-10-16T10:03:55.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fdecc06b020 0x7fdecc06d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdee015da40 0x7fdee01e70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdee006afe0 0x7fdee015a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 --2- 172.21.15.158:0/2182526021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdee00cc590 0x7fdee00c7a20 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 >> 172.21.15.158:0/2182526021 conn(0x7fdee0099a10 msgr2=0x7fdee006ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:55.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 shutdown_connections 2023-10-16T10:03:55.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.684+0000 7fdee5221700 1 -- 172.21.15.158:0/2182526021 wait complete. 2023-10-16T10:03:55.698 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T10:03:55.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.817+0000 7fb75c6da700 1 Processor -- start 2023-10-16T10:03:55.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.817+0000 7fb75c6da700 1 -- start start 2023-10-16T10:03:55.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb75c6da700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb75406ab40 0x7fb75406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb75c6da700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb754161270 con 0x7fb75406e5d0 2023-10-16T10:03:55.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb75c6da700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb754161510 con 0x7fb7540729b0 2023-10-16T10:03:55.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb75c6da700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7541617f0 con 0x7fb75406ab40 2023-10-16T10:03:55.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759c75700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb75406e5d0 0x7fb7540722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59654/0 (socket says 172.21.15.158:59654) 2023-10-16T10:03:55.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759c75700 1 -- 172.21.15.158:0/3939980002 learned_addr learned my addr 172.21.15.158:0/3939980002 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:55.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb75a476700 1 --2- 172.21.15.158:0/3939980002 >> v2:172.21.15.172:3300/0 conn(0x7fb75406ab40 0x7fb75406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530706216 0 0) 0x7fb754161270 con 0x7fb75406e5d0 2023-10-16T10:03:55.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7400035b0 con 0x7fb75406e5d0 2023-10-16T10:03:55.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899695669 0 0) 0x7fb7400035b0 con 0x7fb75406e5d0 2023-10-16T10:03:55.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 >> v2:172.21.15.172:3300/0 conn(0x7fb75406ab40 msgr2=0x7fb75406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:55.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 --2- 172.21.15.158:0/3939980002 >> v2:172.21.15.172:3300/0 conn(0x7fb75406ab40 0x7fb75406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 >> v1:172.21.15.172:6789/0 conn(0x7fb7540729b0 legacy=0x7fb75415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb754161ad0 con 0x7fb75406e5d0 2023-10-16T10:03:55.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fb750001670 con 0x7fb75406e5d0 2023-10-16T10:03:55.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 >> v1:172.21.15.158:6789/0 conn(0x7fb75406e5d0 legacy=0x7fb7540722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb740003a00 0x7fb740003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740004850 0x7fb740004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb740005210 0x7fb74000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7541617f0 con 0x7fb740005210 2023-10-16T10:03:55.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74000c0b0 con 0x7fb740004850 2023-10-16T10:03:55.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74000c270 con 0x7fb740003a00 2023-10-16T10:03:55.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.819+0000 7fb75ac77700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb740005210 0x7fb74000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75a476700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb740003a00 0x7fb740003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb759c75700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740004850 0x7fb740004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75ac77700 1 -- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb740003a00 msgr2=0x7fb740003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:55.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75ac77700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb740003a00 0x7fb740003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75ac77700 1 -- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740004850 msgr2=0x7fb740004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:55.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75ac77700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740004850 0x7fb740004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75ac77700 1 -- 172.21.15.158:0/3939980002 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400035b0 con 0x7fb740005210 2023-10-16T10:03:55.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75a476700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb740003a00 0x7fb740003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:55.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75ac77700 1 -- 172.21.15.158:0/3939980002 reap_dead start 2023-10-16T10:03:55.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb75ac77700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb740005210 0x7fb74000ba10 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c210 tx=0x7fb740005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6874d0d3fa0d6a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c001df0 con 0x7fb740005210 2023-10-16T10:03:55.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c00cd70 con 0x7fb740005210 2023-10-16T10:03:55.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.820+0000 7fb759474700 1 -- 172.21.15.158:0/3939980002 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c003c40 con 0x7fb740005210 2023-10-16T10:03:55.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.821+0000 7fb75c6da700 1 -- 172.21.15.158:0/3939980002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb740005210 msgr2=0x7fb74000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.821+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb740005210 0x7fb74000ba10 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c210 tx=0x7fb740005880 comp rx=0 tx=0).stop 2023-10-16T10:03:55.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.821+0000 7fb75c6da700 1 -- 172.21.15.158:0/3939980002 shutdown_connections 2023-10-16T10:03:55.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.821+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3939980002 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb740005210 0x7fb74000ba10 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.821+0000 7fb75c6da700 1 -- 172.21.15.158:0/3939980002 >> 172.21.15.158:0/3939980002 conn(0x7fb7540d02b0 msgr2=0x7fb7540d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:55.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.821+0000 7fb75c6da700 1 -- 172.21.15.158:0/3939980002 shutdown_connections 2023-10-16T10:03:55.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.821+0000 7fb75c6da700 1 -- 172.21.15.158:0/3939980002 wait complete. 2023-10-16T10:03:55.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.822+0000 7fb75c6da700 1 Processor -- start 2023-10-16T10:03:55.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.822+0000 7fb75c6da700 1 -- start start 2023-10-16T10:03:55.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.822+0000 7fb75c6da700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740005210 0x7fb754156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75c6da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7541567b0 0x7fb7540c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75c6da700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7540b2ed0 0x7fb7540b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75c6da700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540b38c0 con 0x7fb7541567b0 2023-10-16T10:03:55.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75c6da700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540b3d30 con 0x7fb740005210 2023-10-16T10:03:55.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75c6da700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540c5300 con 0x7fb7540b2ed0 2023-10-16T10:03:55.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7541567b0 0x7fb7540c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7541567b0 0x7fb7540c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59490/0 (socket says 172.21.15.158:59490) 2023-10-16T10:03:55.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 -- 172.21.15.158:0/3358579398 learned_addr learned my addr 172.21.15.158:0/3358579398 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:55.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75a476700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740005210 0x7fb754156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75ac77700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7540b2ed0 0x7fb7540b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 -- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7540b2ed0 msgr2=0x7fb7540b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7540b2ed0 0x7fb7540b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 -- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740005210 msgr2=0x7fb754156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:55.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740005210 0x7fb754156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:55.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 -- 172.21.15.158:0/3358579398 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540c6d00 con 0x7fb7541567b0 2023-10-16T10:03:55.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb759c75700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7541567b0 0x7fb7540c67c0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb7500072f0 tx=0x7fb750007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb74affd700 1 -- 172.21.15.158:0/3358579398 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75000f5e0 con 0x7fb7541567b0 2023-10-16T10:03:55.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb74affd700 1 -- 172.21.15.158:0/3358579398 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75000f740 con 0x7fb7541567b0 2023-10-16T10:03:55.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb74affd700 1 -- 172.21.15.158:0/3358579398 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75000fad0 con 0x7fb7541567b0 2023-10-16T10:03:55.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7540c4730 con 0x7fb7541567b0 2023-10-16T10:03:55.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.823+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7540c49d0 con 0x7fb7541567b0 2023-10-16T10:03:55.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.824+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7540c5470 con 0x7fb7541567b0 2023-10-16T10:03:55.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.825+0000 7fb74affd700 1 -- 172.21.15.158:0/3358579398 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb75000fc30 con 0x7fb7541567b0 2023-10-16T10:03:55.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.825+0000 7fb74affd700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb73806b020 0x7fb73806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:55.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.825+0000 7fb74affd700 1 -- 172.21.15.158:0/3358579398 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7fb7500898d0 con 0x7fb7541567b0 2023-10-16T10:03:55.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.825+0000 7fb75a476700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb73806b020 0x7fb73806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:55.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.826+0000 7fb75a476700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb73806b020 0x7fb73806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb75406dea0 tx=0x7fb744008040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:55.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:55.830+0000 7fb74affd700 1 -- 172.21.15.158:0/3358579398 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb750054590 con 0x7fb7541567b0 2023-10-16T10:03:56.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.103+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fb7540681b0 con 0x7fb7541567b0 2023-10-16T10:03:56.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.619+0000 7fb74affd700 1 -- 172.21.15.158:0/3358579398 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v785) v1 ==== 160+0+0 (secure 0 0 0) 0x7fb750057bb0 con 0x7fb7541567b0 2023-10-16T10:03:56.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb73806b020 msgr2=0x7fb73806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb73806b020 0x7fb73806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb75406dea0 tx=0x7fb744008040 comp rx=0 tx=0).stop 2023-10-16T10:03:56.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7541567b0 msgr2=0x7fb7540c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7541567b0 0x7fb7540c67c0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb7500072f0 tx=0x7fb750007600 comp rx=0 tx=0).stop 2023-10-16T10:03:56.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 shutdown_connections 2023-10-16T10:03:56.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb73806b020 0x7fb73806d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7540b2ed0 0x7fb7540b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb740005210 0x7fb754156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 --2- 172.21.15.158:0/3358579398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7541567b0 0x7fb7540c67c0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.621+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 >> 172.21.15.158:0/3358579398 conn(0x7fb7540d02b0 msgr2=0x7fb7540afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:56.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.622+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 shutdown_connections 2023-10-16T10:03:56.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.622+0000 7fb75c6da700 1 -- 172.21.15.158:0/3358579398 wait complete. 2023-10-16T10:03:56.635 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-10-16T10:03:56.636 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-10-16T10:03:56.636 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T10:03:56.637 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:03:56.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.757+0000 7f456a3af700 1 Processor -- start 2023-10-16T10:03:56.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.757+0000 7f456a3af700 1 -- start start 2023-10-16T10:03:56.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f456a3af700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f456415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:56.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f456a3af700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f45640b6e10 0x7f45641521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:56.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f456a3af700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456415c5d0 con 0x7f45640ce2c0 2023-10-16T10:03:56.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f456a3af700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456415d2c0 con 0x7f45640b6e10 2023-10-16T10:03:56.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f456a3af700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456415c8b0 con 0x7f45640cdee0 2023-10-16T10:03:56.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f45637fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f45640b6e10 0x7f45641521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:56.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f45637fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f45640b6e10 0x7f45641521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59496/0 (socket says 172.21.15.158:59496) 2023-10-16T10:03:56.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.758+0000 7f45637fe700 1 -- 172.21.15.158:0/1905508080 learned_addr learned my addr 172.21.15.158:0/1905508080 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:56.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f4563fff700 1 --2- 172.21.15.158:0/1905508080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f456415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:56.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f45637fe700 1 -- 172.21.15.158:0/1905508080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 msgr2=0x7f456415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:56.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f45637fe700 1 --2- 172.21.15.158:0/1905508080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f456415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f45637fe700 1 -- 172.21.15.158:0/1905508080 >> v1:172.21.15.172:6789/0 conn(0x7f45640ce2c0 legacy=0x7f45640cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f45637fe700 1 -- 172.21.15.158:0/1905508080 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456415cb90 con 0x7f45640b6e10 2023-10-16T10:03:56.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f4563fff700 1 --2- 172.21.15.158:0/1905508080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f456415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:56.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f45637fe700 1 --2- 172.21.15.158:0/1905508080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f45641521a0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f45540098f0 tx=0x7f4554009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3162cef7d9389a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:56.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f4562ffd700 1 -- 172.21.15.158:0/1905508080 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554004230 con 0x7f45640b6e10 2023-10-16T10:03:56.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f4562ffd700 1 -- 172.21.15.158:0/1905508080 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4554004390 con 0x7f45640b6e10 2023-10-16T10:03:56.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f4562ffd700 1 -- 172.21.15.158:0/1905508080 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45540036a0 con 0x7f45640b6e10 2023-10-16T10:03:56.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f456a3af700 1 -- 172.21.15.158:0/1905508080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 msgr2=0x7f45641521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f456a3af700 1 --2- 172.21.15.158:0/1905508080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f45641521a0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f45540098f0 tx=0x7f4554009c00 comp rx=0 tx=0).stop 2023-10-16T10:03:56.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f456a3af700 1 -- 172.21.15.158:0/1905508080 shutdown_connections 2023-10-16T10:03:56.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f456a3af700 1 --2- 172.21.15.158:0/1905508080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f456415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f456a3af700 1 --2- 172.21.15.158:0/1905508080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f45641521a0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.759+0000 7f456a3af700 1 -- 172.21.15.158:0/1905508080 >> 172.21.15.158:0/1905508080 conn(0x7f456406a8d0 msgr2=0x7f456406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:56.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.760+0000 7f456a3af700 1 -- 172.21.15.158:0/1905508080 shutdown_connections 2023-10-16T10:03:56.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.760+0000 7f456a3af700 1 -- 172.21.15.158:0/1905508080 wait complete. 2023-10-16T10:03:56.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.761+0000 7f456a3af700 1 Processor -- start 2023-10-16T10:03:56.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.761+0000 7f456a3af700 1 -- start start 2023-10-16T10:03:56.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f4564155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:56.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f45640c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:56.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45640ce2c0 0x7f45641f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:56.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45641f5470 con 0x7f45640b6e10 2023-10-16T10:03:56.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45640c9900 con 0x7f45640ce2c0 2023-10-16T10:03:56.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45640c89e0 con 0x7f45640cdee0 2023-10-16T10:03:56.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f4564155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:56.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f4564155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59502/0 (socket says 172.21.15.158:59502) 2023-10-16T10:03:56.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 -- 172.21.15.158:0/858047114 learned_addr learned my addr 172.21.15.158:0/858047114 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:56.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456894c700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45640ce2c0 0x7f45641f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:56.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f45637fe700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f45640c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:56.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 -- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 msgr2=0x7f45640c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f45640c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 -- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45640ce2c0 msgr2=0x7f45641f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45640ce2c0 0x7f45641f4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 -- 172.21.15.158:0/858047114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554009550 con 0x7f45640b6e10 2023-10-16T10:03:56.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f45637fe700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f45640c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:56.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4563fff700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f4564155780 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f455800c750 tx=0x7f455800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:56.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4560ff9700 1 -- 172.21.15.158:0/858047114 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558011800 con 0x7f45640b6e10 2023-10-16T10:03:56.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4560ff9700 1 -- 172.21.15.158:0/858047114 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4558011960 con 0x7f45640b6e10 2023-10-16T10:03:56.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4564152140 con 0x7f45640b6e10 2023-10-16T10:03:56.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f4560ff9700 1 -- 172.21.15.158:0/858047114 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558011cd0 con 0x7f45640b6e10 2023-10-16T10:03:56.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.762+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45640d0c20 con 0x7f45640b6e10 2023-10-16T10:03:56.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.763+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45640681b0 con 0x7f45640b6e10 2023-10-16T10:03:56.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.764+0000 7f4560ff9700 1 -- 172.21.15.158:0/858047114 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4558021350 con 0x7f45640b6e10 2023-10-16T10:03:56.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.764+0000 7f4560ff9700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f455006b020 0x7f455006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:56.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.764+0000 7f4560ff9700 1 -- 172.21.15.158:0/858047114 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7f45580899b0 con 0x7f45640b6e10 2023-10-16T10:03:56.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.764+0000 7f45637fe700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f455006b020 0x7f455006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:56.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.765+0000 7f45637fe700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f455006b020 0x7f455006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4564154a50 tx=0x7f4554008030 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:56.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.769+0000 7f4560ff9700 1 -- 172.21.15.158:0/858047114 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f45580545f0 con 0x7f45640b6e10 2023-10-16T10:03:56.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.962+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45640b3bf0 con 0x7f45640b6e10 2023-10-16T10:03:56.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.962+0000 7f4560ff9700 1 -- 172.21.15.158:0/858047114 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+14984 (secure 0 0 0) 0x7f455801e090 con 0x7f45640b6e10 2023-10-16T10:03:56.964 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:56.964 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":145,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:03:54.615864+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6833/4142092863":"2023-10-17T10:03:51.336622+0000","172.21.15.158:6833/781614815":"2023-10-17T10:03:50.355931+0000","172.21.15.172:6835/4142092863":"2023-10-17T10:03:51.336622+0000","172.21.15.158:6832/781614815":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/4236549529":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/735244434":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/636538481":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/3740707264":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/3067380233":"2023-10-17T10:03:50.355931+0000","172.21.15.158:6833/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:6832/2342578387":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/3946986955":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1853119573":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/2041489640":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4016680871":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/4198985108":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/3100609660":"2023-10-17T10:03:50.355931+0000","172.21.15.158:0/1741373068":"2023-10-17T10:03:37.828937+0000","172.21.15.158:0/1608841554":"2023-10-17T10:03:37.828937+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:03:56.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f455006b020 msgr2=0x7f455006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f455006b020 0x7f455006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4564154a50 tx=0x7f4554008030 comp rx=0 tx=0).stop 2023-10-16T10:03:56.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 msgr2=0x7f4564155780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:56.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f4564155780 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f455800c750 tx=0x7f455800cb10 comp rx=0 tx=0).stop 2023-10-16T10:03:56.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 shutdown_connections 2023-10-16T10:03:56.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45640ce2c0 0x7f45641f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45640b6e10 0x7f4564155780 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f455006b020 0x7f455006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.964+0000 7f456a3af700 1 --2- 172.21.15.158:0/858047114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45640cdee0 0x7f45640c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:56.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.965+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 >> 172.21.15.158:0/858047114 conn(0x7f456406a8d0 msgr2=0x7f45640c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:56.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.965+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 shutdown_connections 2023-10-16T10:03:56.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:56.965+0000 7f456a3af700 1 -- 172.21.15.158:0/858047114 wait complete. 2023-10-16T10:03:56.978 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:03:56.979 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:03:57.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.101+0000 7f5efbeb0700 1 Processor -- start 2023-10-16T10:03:57.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.101+0000 7f5efbeb0700 1 -- start start 2023-10-16T10:03:57.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5efbeb0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ef415b590 0x7f5ef40ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5efbeb0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef40cbaf0 0x7f5ef40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5efbeb0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef40cf4c0 con 0x7f5ef415f1d0 2023-10-16T10:03:57.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5efbeb0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef40cc4c0 con 0x7f5ef415b590 2023-10-16T10:03:57.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5efbeb0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40b49e0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5efa44d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef40cbaf0 0x7f5ef40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5ef9c4c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ef415b590 0x7f5ef40ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5ef944b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5ef415f1d0 0x7f5ef40b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:35358/0 (socket says 172.21.15.158:35358) 2023-10-16T10:03:57.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.102+0000 7f5ef944b700 1 -- 172.21.15.158:0/3920496160 learned_addr learned my addr 172.21.15.158:0/3920496160 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:57.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef9c4c700 1 -- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef40cbaf0 msgr2=0x7f5ef40cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:57.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef9c4c700 1 --2- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef40cbaf0 0x7f5ef40cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef9c4c700 1 -- 172.21.15.158:0/3920496160 >> v1:172.21.15.172:6789/0 conn(0x7f5ef415f1d0 legacy=0x7f5ef40b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef8c4a700 1 -- 172.21.15.158:0/3920496160 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878175853 0 0) 0x7f5ef40cf4c0 con 0x7f5ef415f1d0 2023-10-16T10:03:57.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef9c4c700 1 -- 172.21.15.158:0/3920496160 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef415f970 con 0x7f5ef415b590 2023-10-16T10:03:57.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5efa44d700 1 --2- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef40cbaf0 0x7f5ef40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:03:57.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef9c4c700 1 --2- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415b590 0x7f5ef40ce510 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009870 tx=0x7f5ee4009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc00d2f7e435456f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:57.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef8c4a700 1 -- 172.21.15.158:0/3920496160 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4011830 con 0x7f5ef415b590 2023-10-16T10:03:57.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef8c4a700 1 -- 172.21.15.158:0/3920496160 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee4011990 con 0x7f5ef415b590 2023-10-16T10:03:57.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.103+0000 7f5ef8c4a700 1 -- 172.21.15.158:0/3920496160 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4011d20 con 0x7f5ef415b590 2023-10-16T10:03:57.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 -- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415b590 msgr2=0x7f5ef40ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415b590 0x7f5ef40ce510 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009870 tx=0x7f5ee4009b80 comp rx=0 tx=0).stop 2023-10-16T10:03:57.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 -- 172.21.15.158:0/3920496160 shutdown_connections 2023-10-16T10:03:57.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef40cbaf0 0x7f5ef40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/3920496160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415b590 0x7f5ef40ce510 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 -- 172.21.15.158:0/3920496160 >> 172.21.15.158:0/3920496160 conn(0x7f5ef406a8d0 msgr2=0x7f5ef406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:57.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 -- 172.21.15.158:0/3920496160 shutdown_connections 2023-10-16T10:03:57.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.104+0000 7f5efbeb0700 1 -- 172.21.15.158:0/3920496160 wait complete. 2023-10-16T10:03:57.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.105+0000 7f5efbeb0700 1 Processor -- start 2023-10-16T10:03:57.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.105+0000 7f5efbeb0700 1 -- start start 2023-10-16T10:03:57.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efbeb0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef40cbaf0 0x7f5ef40affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efbeb0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef415b590 0x7f5ef4152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efbeb0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415f1d0 0x7f5ef4157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efbeb0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40c95f0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efbeb0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40c8a70 con 0x7f5ef415f1d0 2023-10-16T10:03:57.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efbeb0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40c8d50 con 0x7f5ef415b590 2023-10-16T10:03:57.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef40cbaf0 0x7f5ef40affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef40cbaf0 0x7f5ef40affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59516/0 (socket says 172.21.15.158:59516) 2023-10-16T10:03:57.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 -- 172.21.15.158:0/1877705096 learned_addr learned my addr 172.21.15.158:0/1877705096 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:57.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef944b700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef415b590 0x7f5ef4152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efa44d700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415f1d0 0x7f5ef4157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 -- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef415b590 msgr2=0x7f5ef4152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef415b590 0x7f5ef4152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 -- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415f1d0 msgr2=0x7f5ef4157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415f1d0 0x7f5ef4157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef9c4c700 1 -- 172.21.15.158:0/1877705096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec009a70 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5ef944b700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef415b590 0x7f5ef4152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:57.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.106+0000 7f5efa44d700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415f1d0 0x7f5ef4157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:57.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.107+0000 7f5ef9c4c700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef40cbaf0 0x7f5ef40affc0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4000c00 tx=0x7f5ee4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:57.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.107+0000 7f5eea7fc700 1 -- 172.21.15.158:0/1877705096 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40036a0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.107+0000 7f5eea7fc700 1 -- 172.21.15.158:0/1877705096 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee401f070 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.107+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee40094a0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.107+0000 7f5eea7fc700 1 -- 172.21.15.158:0/1877705096 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4003a00 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.107+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef40c9310 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.108+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef406b9e0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.108+0000 7f5eea7fc700 1 -- 172.21.15.158:0/1877705096 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5ee4003b60 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.108+0000 7f5eea7fc700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5ed806b020 0x7f5ed806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.109+0000 7f5eea7fc700 1 -- 172.21.15.158:0/1877705096 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7f5ee4021070 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.109+0000 7f5ef944b700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5ed806b020 0x7f5ed806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.109+0000 7f5ef944b700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5ed806b020 0x7f5ed806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4158c30 tx=0x7f5ef0007ca0 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:57.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.113+0000 7f5eea7fc700 1 -- 172.21.15.158:0/1877705096 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ef406b9e0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.301+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ef406b9e0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.302+0000 7f5eea7fc700 1 -- 172.21.15.158:0/1877705096 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ef406b9e0 con 0x7f5ef40cbaf0 2023-10-16T10:03:57.303 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:03:57.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5ed806b020 msgr2=0x7f5ed806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5ed806b020 0x7f5ed806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4158c30 tx=0x7f5ef0007ca0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef40cbaf0 msgr2=0x7f5ef40affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef40cbaf0 0x7f5ef40affc0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4000c00 tx=0x7f5ee4004750 comp rx=0 tx=0).stop 2023-10-16T10:03:57.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 shutdown_connections 2023-10-16T10:03:57.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef415f1d0 0x7f5ef4157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef40cbaf0 0x7f5ef40affc0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5ed806b020 0x7f5ed806d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 --2- 172.21.15.158:0/1877705096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef415b590 0x7f5ef4152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 >> 172.21.15.158:0/1877705096 conn(0x7f5ef406a8d0 msgr2=0x7f5ef40b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:57.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 shutdown_connections 2023-10-16T10:03:57.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.304+0000 7f5efbeb0700 1 -- 172.21.15.158:0/1877705096 wait complete. 2023-10-16T10:03:57.317 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T10:03:57.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.437+0000 7f5698a7f700 1 Processor -- start 2023-10-16T10:03:57.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.437+0000 7f5698a7f700 1 -- start start 2023-10-16T10:03:57.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.438+0000 7f5698a7f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f569415a6c0 0x7f56940723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.438+0000 7f5698a7f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5694072910 con 0x7f56940af590 2023-10-16T10:03:57.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.438+0000 7f5698a7f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56940b1e40 con 0x7f569406e090 2023-10-16T10:03:57.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.438+0000 7f5698a7f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56940ca7e0 con 0x7f569415a6c0 2023-10-16T10:03:57.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f5691d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f56940af590 0x7f569406d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59700/0 (socket says 172.21.15.158:59700) 2023-10-16T10:03:57.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f569415a6c0 0x7f56940723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f5691d9b700 1 -- 172.21.15.158:0/155746077 learned_addr learned my addr 172.21.15.158:0/155746077 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:57.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569259c700 1 -- 172.21.15.158:0/155746077 >> v1:172.21.15.172:6789/0 conn(0x7f569406e090 legacy=0x7f56940ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:57.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569259c700 1 -- 172.21.15.158:0/155746077 >> v1:172.21.15.158:6789/0 conn(0x7f56940af590 legacy=0x7f569406d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569259c700 1 -- 172.21.15.158:0/155746077 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940b2120 con 0x7f569415a6c0 2023-10-16T10:03:57.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569159a700 1 -- 172.21.15.158:0/155746077 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702415146 0 0) 0x7f5694072910 con 0x7f56940af590 2023-10-16T10:03:57.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569259c700 1 --2- 172.21.15.158:0/155746077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569415a6c0 0x7f56940723d0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5688009820 tx=0x7f5688009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c7371b90c9bcf6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:57.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569159a700 1 -- 172.21.15.158:0/155746077 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688005e30 con 0x7f569415a6c0 2023-10-16T10:03:57.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569159a700 1 -- 172.21.15.158:0/155746077 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688004030 con 0x7f569415a6c0 2023-10-16T10:03:57.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.439+0000 7f569159a700 1 -- 172.21.15.158:0/155746077 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56880043c0 con 0x7f569415a6c0 2023-10-16T10:03:57.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.440+0000 7f5698a7f700 1 -- 172.21.15.158:0/155746077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569415a6c0 msgr2=0x7f56940723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.440+0000 7f5698a7f700 1 --2- 172.21.15.158:0/155746077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569415a6c0 0x7f56940723d0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5688009820 tx=0x7f5688009b30 comp rx=0 tx=0).stop 2023-10-16T10:03:57.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.440+0000 7f5698a7f700 1 -- 172.21.15.158:0/155746077 shutdown_connections 2023-10-16T10:03:57.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.440+0000 7f5698a7f700 1 --2- 172.21.15.158:0/155746077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569415a6c0 0x7f56940723d0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.440+0000 7f5698a7f700 1 -- 172.21.15.158:0/155746077 >> 172.21.15.158:0/155746077 conn(0x7f56940ccda0 msgr2=0x7f56940cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:57.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.440+0000 7f5698a7f700 1 -- 172.21.15.158:0/155746077 shutdown_connections 2023-10-16T10:03:57.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.440+0000 7f5698a7f700 1 -- 172.21.15.158:0/155746077 wait complete. 2023-10-16T10:03:57.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.441+0000 7f5698a7f700 1 Processor -- start 2023-10-16T10:03:57.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.441+0000 7f5698a7f700 1 -- start start 2023-10-16T10:03:57.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5698a7f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569406e090 0x7f56941efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5698a7f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56940af590 0x7f56940b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5698a7f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f569415a6c0 0x7f56940b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5698a7f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56940c73f0 con 0x7f569406e090 2023-10-16T10:03:57.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5698a7f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56940c7640 con 0x7f569415a6c0 2023-10-16T10:03:57.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5698a7f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56940c7920 con 0x7f56940af590 2023-10-16T10:03:57.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569406e090 0x7f56941efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569406e090 0x7f56941efbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59528/0 (socket says 172.21.15.158:59528) 2023-10-16T10:03:57.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 -- 172.21.15.158:0/340516868 learned_addr learned my addr 172.21.15.158:0/340516868 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:57.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5691d9b700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56940af590 0x7f56940b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f5692d9d700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f569415a6c0 0x7f56940b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 -- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56940af590 msgr2=0x7f56940b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56940af590 0x7f56940b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 -- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f569415a6c0 msgr2=0x7f56940b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:57.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f569415a6c0 0x7f56940b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:57.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 -- 172.21.15.158:0/340516868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5688009430 con 0x7f569406e090 2023-10-16T10:03:57.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.442+0000 7f569259c700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569406e090 0x7f56941efbc0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.443+0000 7f5682ffd700 1 -- 172.21.15.158:0/340516868 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688020070 con 0x7f569406e090 2023-10-16T10:03:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.443+0000 7f5682ffd700 1 -- 172.21.15.158:0/340516868 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688003dc0 con 0x7f569406e090 2023-10-16T10:03:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.443+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56940c7c00 con 0x7f569406e090 2023-10-16T10:03:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.443+0000 7f5682ffd700 1 -- 172.21.15.158:0/340516868 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56880196d0 con 0x7f569406e090 2023-10-16T10:03:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.443+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56940b50c0 con 0x7f569406e090 2023-10-16T10:03:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.444+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56940c8120 con 0x7f569406e090 2023-10-16T10:03:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.444+0000 7f5682ffd700 1 -- 172.21.15.158:0/340516868 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5688019830 con 0x7f569406e090 2023-10-16T10:03:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.445+0000 7f5682ffd700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f567406b020 0x7f567406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.445+0000 7f5691d9b700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f567406b020 0x7f567406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.445+0000 7f5682ffd700 1 -- 172.21.15.158:0/340516868 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7f568808c1e0 con 0x7f569406e090 2023-10-16T10:03:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.445+0000 7f5691d9b700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f567406b020 0x7f567406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f56940cf160 tx=0x7f5684009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:57.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.449+0000 7f5682ffd700 1 -- 172.21.15.158:0/340516868 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56940c8120 con 0x7f569406e090 2023-10-16T10:03:57.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:57.719+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f56940c8120 con 0x7f569406e090 2023-10-16T10:03:58.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.625+0000 7f5682ffd700 1 -- 172.21.15.158:0/340516868 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v787) v1 ==== 181+0+0 (secure 0 0 0) 0x7f56940c8120 con 0x7f569406e090 2023-10-16T10:03:58.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f567406b020 msgr2=0x7f567406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f567406b020 0x7f567406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f56940cf160 tx=0x7f5684009040 comp rx=0 tx=0).stop 2023-10-16T10:03:58.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569406e090 msgr2=0x7f56941efbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569406e090 0x7f56941efbc0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688004750 comp rx=0 tx=0).stop 2023-10-16T10:03:58.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 shutdown_connections 2023-10-16T10:03:58.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f569415a6c0 0x7f56940b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f569406e090 0x7f56941efbc0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f567406b020 0x7f567406d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 --2- 172.21.15.158:0/340516868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56940af590 0x7f56940b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 >> 172.21.15.158:0/340516868 conn(0x7f56940ccda0 msgr2=0x7f5694158880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:58.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 shutdown_connections 2023-10-16T10:03:58.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.628+0000 7f5698a7f700 1 -- 172.21.15.158:0/340516868 wait complete. 2023-10-16T10:03:58.644 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T10:03:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.773+0000 7f6770ab1700 1 Processor -- start 2023-10-16T10:03:58.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.773+0000 7f6770ab1700 1 -- start start 2023-10-16T10:03:58.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6770ab1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f676c15da40 0x7f676c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:58.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6770ab1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f676c0cc590 0x7f676c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:58.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6770ab1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676c0cbd30 con 0x7f676c06ae10 2023-10-16T10:03:58.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6770ab1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0cc1c0 con 0x7f676c0cc590 2023-10-16T10:03:58.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6770ab1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c06b240 con 0x7f676c15da40 2023-10-16T10:03:58.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f676c0cc590 0x7f676c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:58.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f676c0cc590 0x7f676c06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59538/0 (socket says 172.21.15.158:59538) 2023-10-16T10:03:58.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 -- 172.21.15.158:0/3170325538 learned_addr learned my addr 172.21.15.158:0/3170325538 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:58.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f676a59c700 1 --2- 172.21.15.158:0/3170325538 >> v2:172.21.15.172:3300/0 conn(0x7f676c15da40 0x7f676c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:58.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 -- 172.21.15.158:0/3170325538 >> v2:172.21.15.172:3300/0 conn(0x7f676c15da40 msgr2=0x7f676c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:58.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 --2- 172.21.15.158:0/3170325538 >> v2:172.21.15.172:3300/0 conn(0x7f676c15da40 0x7f676c0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 -- 172.21.15.158:0/3170325538 >> v1:172.21.15.172:6789/0 conn(0x7f676c06ae10 legacy=0x7f676c0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 -- 172.21.15.158:0/3170325538 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c15de20 con 0x7f676c0cc590 2023-10-16T10:03:58.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f676a59c700 1 --2- 172.21.15.158:0/3170325538 >> v2:172.21.15.172:3300/0 conn(0x7f676c15da40 0x7f676c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:58.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.774+0000 7f6769d9b700 1 --2- 172.21.15.158:0/3170325538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c0cc590 0x7f676c06a8d0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f676c06c9c0 tx=0x7f675c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90c90ec6bcdb4de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:58.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f676959a700 1 -- 172.21.15.158:0/3170325538 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00b870 con 0x7f676c0cc590 2023-10-16T10:03:58.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f676959a700 1 -- 172.21.15.158:0/3170325538 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c00b9d0 con 0x7f676c0cc590 2023-10-16T10:03:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f676959a700 1 -- 172.21.15.158:0/3170325538 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00bd60 con 0x7f676c0cc590 2023-10-16T10:03:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 -- 172.21.15.158:0/3170325538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c0cc590 msgr2=0x7f676c06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3170325538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c0cc590 0x7f676c06a8d0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f676c06c9c0 tx=0x7f675c009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 -- 172.21.15.158:0/3170325538 shutdown_connections 2023-10-16T10:03:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3170325538 >> v2:172.21.15.172:3300/0 conn(0x7f676c15da40 0x7f676c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3170325538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c0cc590 0x7f676c06a8d0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 -- 172.21.15.158:0/3170325538 >> 172.21.15.158:0/3170325538 conn(0x7f676c099a10 msgr2=0x7f676c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:58.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 -- 172.21.15.158:0/3170325538 shutdown_connections 2023-10-16T10:03:58.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.775+0000 7f6770ab1700 1 -- 172.21.15.158:0/3170325538 wait complete. 2023-10-16T10:03:58.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.776+0000 7f6770ab1700 1 Processor -- start 2023-10-16T10:03:58.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f6770ab1700 1 -- start start 2023-10-16T10:03:58.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f6770ab1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f676c06ae10 0x7f676c15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:58.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f6770ab1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c0cc590 0x7f676c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:58.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f6770ab1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c15da40 0x7f676c0c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f6770ab1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0ca400 con 0x7f676c15da40 2023-10-16T10:03:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f6770ab1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0c57d0 con 0x7f676c0cc590 2023-10-16T10:03:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f6770ab1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0c4730 con 0x7f676c06ae10 2023-10-16T10:03:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f676ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c15da40 0x7f676c0c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f676ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c15da40 0x7f676c0c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59544/0 (socket says 172.21.15.158:59544) 2023-10-16T10:03:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.777+0000 7f676ad9d700 1 -- 172.21.15.158:0/3347474928 learned_addr learned my addr 172.21.15.158:0/3347474928 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:58.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f6769d9b700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c0cc590 0x7f676c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:58.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f676a59c700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f676c06ae10 0x7f676c15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:58.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f676ad9d700 1 -- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f676c06ae10 msgr2=0x7f676c15b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f676ad9d700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f676c06ae10 0x7f676c15b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f676ad9d700 1 -- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c0cc590 msgr2=0x7f676c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f676ad9d700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c0cc590 0x7f676c0c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f676ad9d700 1 -- 172.21.15.158:0/3347474928 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c009550 con 0x7f676c15da40 2023-10-16T10:03:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f6769d9b700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c0cc590 0x7f676c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f676ad9d700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c15da40 0x7f676c0c9790 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f676400cc50 tx=0x7f676400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f675affd700 1 -- 172.21.15.158:0/3347474928 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400ad80 con 0x7f676c15da40 2023-10-16T10:03:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f675affd700 1 -- 172.21.15.158:0/3347474928 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676400ce50 con 0x7f676c15da40 2023-10-16T10:03:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c0c4a10 con 0x7f676c15da40 2023-10-16T10:03:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f675affd700 1 -- 172.21.15.158:0/3347474928 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764003c40 con 0x7f676c15da40 2023-10-16T10:03:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.778+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c0c4fd0 con 0x7f676c15da40 2023-10-16T10:03:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.779+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c0681b0 con 0x7f676c15da40 2023-10-16T10:03:58.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.779+0000 7f675affd700 1 -- 172.21.15.158:0/3347474928 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f67640050d0 con 0x7f676c15da40 2023-10-16T10:03:58.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.780+0000 7f675affd700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f674c06b020 0x7f674c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:58.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.780+0000 7f675affd700 1 -- 172.21.15.158:0/3347474928 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7f6764089be0 con 0x7f676c15da40 2023-10-16T10:03:58.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.780+0000 7f676a59c700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f674c06b020 0x7f674c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:58.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.780+0000 7f676a59c700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f674c06b020 0x7f674c06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f676000c900 tx=0x7f6760008040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:58.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.784+0000 7f675affd700 1 -- 172.21.15.158:0/3347474928 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6764054810 con 0x7f676c15da40 2023-10-16T10:03:58.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.980+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f676c0d0a10 con 0x7f676c15da40 2023-10-16T10:03:58.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.983+0000 7f675affd700 1 -- 172.21.15.158:0/3347474928 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6764057e30 con 0x7f676c15da40 2023-10-16T10:03:58.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.985+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f674c06b020 msgr2=0x7f674c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.985+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f674c06b020 0x7f674c06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f676000c900 tx=0x7f6760008040 comp rx=0 tx=0).stop 2023-10-16T10:03:58.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.985+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c15da40 msgr2=0x7f676c0c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.985+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c15da40 0x7f676c0c9790 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f676400cc50 tx=0x7f676400a340 comp rx=0 tx=0).stop 2023-10-16T10:03:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 shutdown_connections 2023-10-16T10:03:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f676c15da40 0x7f676c0c9790 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f674c06b020 0x7f674c06d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f676c06ae10 0x7f676c15b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 --2- 172.21.15.158:0/3347474928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c0cc590 0x7f676c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 >> 172.21.15.158:0/3347474928 conn(0x7f676c099a10 msgr2=0x7f676c0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 shutdown_connections 2023-10-16T10:03:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:58.986+0000 7f6770ab1700 1 -- 172.21.15.158:0/3347474928 wait complete. 2023-10-16T10:03:58.999 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T10:03:58.999 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:59.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T10:03:59.084 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T10:03:59.085 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:59.085 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:59.102 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:03:59.103 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:59.103 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:59.143 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:59.144 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:03:59.144 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:59.157 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:03:59.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:03:59.162 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T10:03:59.163 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T10:03:59.163 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:03:59.203 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T10:03:59.204 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T10:03:59.204 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:03:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.324+0000 7fa1de2eb700 1 Processor -- start 2023-10-16T10:03:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.324+0000 7fa1de2eb700 1 -- start start 2023-10-16T10:03:59.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.324+0000 7fa1de2eb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa1d815d7e0 0x7fa1d815dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.324+0000 7fa1de2eb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d806e7a0 con 0x7fa1d8150b80 2023-10-16T10:03:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.324+0000 7fa1de2eb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d806ea80 con 0x7fa1d80ce600 2023-10-16T10:03:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.324+0000 7fa1de2eb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1d806f420 con 0x7fa1d815d7e0 2023-10-16T10:03:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa1d8150b80 0x7fa1d806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59724/0 (socket says 172.21.15.158:59724) 2023-10-16T10:03:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d7fff700 1 -- 172.21.15.158:0/144553830 learned_addr learned my addr 172.21.15.158:0/144553830 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:03:59.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d77fe700 1 --2- 172.21.15.158:0/144553830 >> v2:172.21.15.172:3300/0 conn(0x7fa1d815d7e0 0x7fa1d815dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781949534 0 0) 0x7fa1d806e7a0 con 0x7fa1d8150b80 2023-10-16T10:03:59.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c00035b0 con 0x7fa1d8150b80 2023-10-16T10:03:59.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332512728 0 0) 0x7fa1d806ea80 con 0x7fa1d80ce600 2023-10-16T10:03:59.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d806e7a0 con 0x7fa1d80ce600 2023-10-16T10:03:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949771959 0 0) 0x7fa1c00035b0 con 0x7fa1d8150b80 2023-10-16T10:03:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 >> v2:172.21.15.172:3300/0 conn(0x7fa1d815d7e0 msgr2=0x7fa1d815dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 --2- 172.21.15.158:0/144553830 >> v2:172.21.15.172:3300/0 conn(0x7fa1d815d7e0 0x7fa1d815dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 >> v1:172.21.15.172:6789/0 conn(0x7fa1d80ce600 legacy=0x7fa1d80cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d806ed60 con 0x7fa1d8150b80 2023-10-16T10:03:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fa1cc001670 con 0x7fa1d8150b80 2023-10-16T10:03:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 >> v1:172.21.15.158:6789/0 conn(0x7fa1d8150b80 legacy=0x7fa1d806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1c0003be0 0x7fa1c0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1c0004850 0x7fa1c0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0005260 0x7fa1c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d806f420 con 0x7fa1c0003be0 2023-10-16T10:03:59.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c000c190 con 0x7fa1c0005260 2023-10-16T10:03:59.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c000c350 con 0x7fa1c0004850 2023-10-16T10:03:59.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1c0003be0 0x7fa1c0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1dc888700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0005260 0x7fa1c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d77fe700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1c0004850 0x7fa1c0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.325+0000 7fa1d7fff700 1 -- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1c0004850 msgr2=0x7fa1c0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1c0004850 0x7fa1c0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d7fff700 1 -- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0005260 msgr2=0x7fa1c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0005260 0x7fa1c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d7fff700 1 -- 172.21.15.158:0/144553830 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c00035b0 con 0x7fa1c0003be0 2023-10-16T10:03:59.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1dc888700 1 -- 172.21.15.158:0/144553830 reap_dead start 2023-10-16T10:03:59.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d77fe700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1c0004850 0x7fa1c0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:59.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1c0003be0 0x7fa1c0003fe0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00c160 tx=0x7fa1c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46fd4c46237db2b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:59.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc00cca0 con 0x7fa1c0003be0 2023-10-16T10:03:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc00ce20 con 0x7fa1c0003be0 2023-10-16T10:03:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1d6ffd700 1 -- 172.21.15.158:0/144553830 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc005ba0 con 0x7fa1c0003be0 2023-10-16T10:03:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1de2eb700 1 -- 172.21.15.158:0/144553830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1c0003be0 msgr2=0x7fa1c0003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1c0003be0 0x7fa1c0003fe0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00c160 tx=0x7fa1c0005880 comp rx=0 tx=0).stop 2023-10-16T10:03:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1de2eb700 1 -- 172.21.15.158:0/144553830 shutdown_connections 2023-10-16T10:03:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/144553830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1c0003be0 0x7fa1c0003fe0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1de2eb700 1 -- 172.21.15.158:0/144553830 >> 172.21.15.158:0/144553830 conn(0x7fa1d80cc5a0 msgr2=0x7fa1d80cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1de2eb700 1 -- 172.21.15.158:0/144553830 shutdown_connections 2023-10-16T10:03:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.326+0000 7fa1de2eb700 1 -- 172.21.15.158:0/144553830 wait complete. 2023-10-16T10:03:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 Processor -- start 2023-10-16T10:03:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 -- start start 2023-10-16T10:03:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0003be0 0x7fa1d81e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1d8151cb0 0x7fa1d81539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d815cb40 0x7fa1d80c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d815e1d0 con 0x7fa1d815cb40 2023-10-16T10:03:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d81521b0 con 0x7fa1c0003be0 2023-10-16T10:03:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.328+0000 7fa1de2eb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d81e4170 con 0x7fa1d8151cb0 2023-10-16T10:03:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d815cb40 0x7fa1d80c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d815cb40 0x7fa1d80c1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59562/0 (socket says 172.21.15.158:59562) 2023-10-16T10:03:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 -- 172.21.15.158:0/1579738018 learned_addr learned my addr 172.21.15.158:0/1579738018 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0003be0 0x7fa1d81e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1d77fe700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1d8151cb0 0x7fa1d81539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 -- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1d8151cb0 msgr2=0x7fa1d81539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1d8151cb0 0x7fa1d81539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 -- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0003be0 msgr2=0x7fa1d81e4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0003be0 0x7fa1d81e4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 -- 172.21.15.158:0/1579738018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d81e4450 con 0x7fa1d815cb40 2023-10-16T10:03:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0003be0 0x7fa1d81e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:03:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1dc888700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d815cb40 0x7fa1d80c1ce0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80cd670 tx=0x7fa1d000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1d4ff9700 1 -- 172.21.15.158:0/1579738018 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0003cd0 con 0x7fa1d815cb40 2023-10-16T10:03:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1d4ff9700 1 -- 172.21.15.158:0/1579738018 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d0003e30 con 0x7fa1d815cb40 2023-10-16T10:03:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1d4ff9700 1 -- 172.21.15.158:0/1579738018 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d000dbd0 con 0x7fa1d815cb40 2023-10-16T10:03:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d81e4730 con 0x7fa1d815cb40 2023-10-16T10:03:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.329+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d8153ef0 con 0x7fa1d815cb40 2023-10-16T10:03:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.330+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d80681b0 con 0x7fa1d815cb40 2023-10-16T10:03:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.330+0000 7fa1d4ff9700 1 -- 172.21.15.158:0/1579738018 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa1d000dd30 con 0x7fa1d815cb40 2023-10-16T10:03:59.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.331+0000 7fa1d4ff9700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa1c406b020 0x7fa1c406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.331+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa1c406b020 0x7fa1c406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.331+0000 7fa1d4ff9700 1 -- 172.21.15.158:0/1579738018 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7fa1d0089db0 con 0x7fa1d815cb40 2023-10-16T10:03:59.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.331+0000 7fa1d7fff700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa1c406b020 0x7fa1c406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc015040 tx=0x7fa1cc007000 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:59.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.336+0000 7fa1d4ff9700 1 -- 172.21.15.158:0/1579738018 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1d0054bb0 con 0x7fa1d815cb40 2023-10-16T10:03:59.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.576+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1d80ca680 con 0x7fa1d815cb40 2023-10-16T10:03:59.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.576+0000 7fa1d4ff9700 1 -- 172.21.15.158:0/1579738018 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fa1d00581d0 con 0x7fa1d815cb40 2023-10-16T10:03:59.578 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:03:59.578 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":70,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4144904682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4144904682},{"type":"v1","addr":"172.21.15.172:6834","nonce":4144904682}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/757153181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":757153181},{"type":"v1","addr":"172.21.15.172:6835","nonce":757153181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3562807630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3562807630},{"type":"v1","addr":"172.21.15.158:6836","nonce":3562807630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8805,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/614819185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":614819185},{"type":"v1","addr":"172.21.15.158:6837","nonce":614819185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[]} 2023-10-16T10:03:59.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa1c406b020 msgr2=0x7fa1c406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa1c406b020 0x7fa1c406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc015040 tx=0x7fa1cc007000 comp rx=0 tx=0).stop 2023-10-16T10:03:59.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d815cb40 msgr2=0x7fa1d80c1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d815cb40 0x7fa1d80c1ce0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80cd670 tx=0x7fa1d000d430 comp rx=0 tx=0).stop 2023-10-16T10:03:59.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 shutdown_connections 2023-10-16T10:03:59.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d815cb40 0x7fa1d80c1ce0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1c0003be0 0x7fa1d81e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa1c406b020 0x7fa1c406d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 --2- 172.21.15.158:0/1579738018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1d8151cb0 0x7fa1d81539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 >> 172.21.15.158:0/1579738018 conn(0x7fa1d80cc5a0 msgr2=0x7fa1d806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:59.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 shutdown_connections 2023-10-16T10:03:59.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.579+0000 7fa1de2eb700 1 -- 172.21.15.158:0/1579738018 wait complete. 2023-10-16T10:03:59.583 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 70 2023-10-16T10:03:59.592 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T10:03:59.593 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T10:03:59.593 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T10:03:59.593 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:59.594 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T10:03:59.594 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T10:03:59.594 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T10:03:59.594 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:59.595 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T10:03:59.595 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T10:03:59.595 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T10:03:59.596 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:59.596 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T10:03:59.597 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T10:03:59.597 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T10:03:59.597 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T10:03:59.601 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T10:03:59.602 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T10:03:59.603 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T10:03:59.604 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T10:03:59.607 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T10:03:59.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T10:03:59.608 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T10:03:59.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T10:03:59.645 INFO:tasks.ceph.mds.a:Started 2023-10-16T10:03:59.645 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:59.646 INFO:tasks.ceph.mds.c:Started 2023-10-16T10:03:59.647 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:59.650 INFO:tasks.ceph.mds.b:Started 2023-10-16T10:03:59.651 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:59.652 INFO:tasks.ceph.mds.d:Started 2023-10-16T10:03:59.652 DEBUG:teuthology.parallel:result is None 2023-10-16T10:03:59.653 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T10:03:59.724 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T10:03:59.724 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T10:03:59.725 INFO:tasks.ceph.mds.a.smithi158.stderr:2023-10-16T10:03:59.723+0000 7f316b030700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-10-16T10:03:59.736 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T10:03:59.736 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T10:03:59.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 Processor -- start 2023-10-16T10:03:59.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 -- start start 2023-10-16T10:03:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f471815d990 0x7f47180b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f471806e480 0x7f47180b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47180b3760 0x7f47180cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47180b2c90 con 0x7f471815d990 2023-10-16T10:03:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47180af600 con 0x7f471806e480 2023-10-16T10:03:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.779+0000 7f471eb96700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471806e9b0 con 0x7f47180b3760 2023-10-16T10:03:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f471815d990 0x7f47180b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f471815d990 0x7f47180b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59590/0 (socket says 172.21.15.158:59590) 2023-10-16T10:03:59.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 -- 172.21.15.158:0/3981472087 learned_addr learned my addr 172.21.15.158:0/3981472087 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471d133700 1 --2- 172.21.15.158:0/3981472087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47180b3760 0x7f47180cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f4717fff700 1 --2- 172.21.15.158:0/3981472087 >> v2:172.21.15.172:3300/0 conn(0x7f471806e480 0x7f47180b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 -- 172.21.15.158:0/3981472087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47180b3760 msgr2=0x7f47180cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 --2- 172.21.15.158:0/3981472087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47180b3760 0x7f47180cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 -- 172.21.15.158:0/3981472087 >> v2:172.21.15.172:3300/0 conn(0x7f471806e480 msgr2=0x7f47180b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:03:59.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 --2- 172.21.15.158:0/3981472087 >> v2:172.21.15.172:3300/0 conn(0x7f471806e480 0x7f47180b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 -- 172.21.15.158:0/3981472087 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47180ce890 con 0x7f471815d990 2023-10-16T10:03:59.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f4717fff700 1 --2- 172.21.15.158:0/3981472087 >> v2:172.21.15.172:3300/0 conn(0x7f471806e480 0x7f47180b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:59.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f471c932700 1 --2- 172.21.15.158:0/3981472087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f47180b2750 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f471809de90 tx=0x7f4700009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4f13e67a0c7def server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:59.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f47177fe700 1 -- 172.21.15.158:0/3981472087 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000b870 con 0x7f471815d990 2023-10-16T10:03:59.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f47177fe700 1 -- 172.21.15.158:0/3981472087 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470000b9d0 con 0x7f471815d990 2023-10-16T10:03:59.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.780+0000 7f47177fe700 1 -- 172.21.15.158:0/3981472087 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000bd60 con 0x7f471815d990 2023-10-16T10:03:59.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 -- 172.21.15.158:0/3981472087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 msgr2=0x7f47180b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 --2- 172.21.15.158:0/3981472087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f47180b2750 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f471809de90 tx=0x7f4700009bd0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 -- 172.21.15.158:0/3981472087 shutdown_connections 2023-10-16T10:03:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 --2- 172.21.15.158:0/3981472087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47180b3760 0x7f47180cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 --2- 172.21.15.158:0/3981472087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f47180b2750 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 --2- 172.21.15.158:0/3981472087 >> v2:172.21.15.172:3300/0 conn(0x7f471806e480 0x7f47180b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 -- 172.21.15.158:0/3981472087 >> 172.21.15.158:0/3981472087 conn(0x7f471806b9e0 msgr2=0x7f471806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:03:59.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 -- 172.21.15.158:0/3981472087 shutdown_connections 2023-10-16T10:03:59.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.781+0000 7f471eb96700 1 -- 172.21.15.158:0/3981472087 wait complete. 2023-10-16T10:03:59.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.782+0000 7f471eb96700 1 Processor -- start 2023-10-16T10:03:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.782+0000 7f471eb96700 1 -- start start 2023-10-16T10:03:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.783+0000 7f471eb96700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f471806e480 0x7f4718152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.783+0000 7f471eb96700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47180b3760 0x7f471815a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.783+0000 7f471eb96700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f471815a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471eb96700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471815f770 con 0x7f471815d990 2023-10-16T10:03:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471eb96700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47180c5500 con 0x7f47180b3760 2023-10-16T10:03:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471eb96700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47180c4220 con 0x7f471806e480 2023-10-16T10:03:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f471815a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f471815a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59598/0 (socket says 172.21.15.158:59598) 2023-10-16T10:03:59.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 -- 172.21.15.158:0/135986806 learned_addr learned my addr 172.21.15.158:0/135986806 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:03:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f4717fff700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47180b3760 0x7f471815a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471c932700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f471806e480 0x7f4718152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 -- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f471806e480 msgr2=0x7f4718152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f471806e480 0x7f4718152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 -- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47180b3760 msgr2=0x7f471815a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:03:59.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47180b3760 0x7f471815a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:03:59.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 -- 172.21.15.158:0/135986806 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700009550 con 0x7f471815d990 2023-10-16T10:03:59.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f4717fff700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47180b3760 0x7f471815a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:03:59.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471d133700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f471815a6c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f4710007e30 tx=0x7f471000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:59.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f47157fa700 1 -- 172.21.15.158:0/135986806 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471000ede0 con 0x7f471815d990 2023-10-16T10:03:59.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f47157fa700 1 -- 172.21.15.158:0/135986806 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4710014420 con 0x7f471815d990 2023-10-16T10:03:59.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4718152fe0 con 0x7f471815d990 2023-10-16T10:03:59.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f47157fa700 1 -- 172.21.15.158:0/135986806 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47100147d0 con 0x7f471815d990 2023-10-16T10:03:59.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.784+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471815f490 con 0x7f471815d990 2023-10-16T10:03:59.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.785+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47180681b0 con 0x7f471815d990 2023-10-16T10:03:59.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.786+0000 7f47157fa700 1 -- 172.21.15.158:0/135986806 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4710014930 con 0x7f471815d990 2023-10-16T10:03:59.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.786+0000 7f47157fa700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f46f806b020 0x7f46f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:03:59.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.786+0000 7f47157fa700 1 -- 172.21.15.158:0/135986806 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6446+0+0 (secure 0 0 0) 0x7f471008a070 con 0x7f471815d990 2023-10-16T10:03:59.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.786+0000 7f471c932700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f46f806b020 0x7f46f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:03:59.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.786+0000 7f471c932700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f46f806b020 0x7f46f806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4700014040 tx=0x7f4700008030 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:03:59.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.790+0000 7f47157fa700 1 -- 172.21.15.158:0/135986806 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4710054d30 con 0x7f471815d990 2023-10-16T10:03:59.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:03:59.997+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f47180cd730 con 0x7f471815d990 2023-10-16T10:04:00.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.638+0000 7f47157fa700 1 -- 172.21.15.158:0/135986806 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v147) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4710058350 con 0x7f471815d990 2023-10-16T10:04:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f46f806b020 msgr2=0x7f46f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f46f806b020 0x7f46f806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4700014040 tx=0x7f4700008030 comp rx=0 tx=0).stop 2023-10-16T10:04:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 msgr2=0x7f471815a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:00.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f471815a6c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f4710007e30 tx=0x7f471000e3c0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 shutdown_connections 2023-10-16T10:04:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f471815d990 0x7f471815a6c0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f46f806b020 0x7f46f806d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f471806e480 0x7f4718152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 --2- 172.21.15.158:0/135986806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47180b3760 0x7f471815a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 >> 172.21.15.158:0/135986806 conn(0x7f471806b9e0 msgr2=0x7f47180718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.641+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 shutdown_connections 2023-10-16T10:04:00.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.642+0000 7f471eb96700 1 -- 172.21.15.158:0/135986806 wait complete. 2023-10-16T10:04:00.645 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T10:04:00.658 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T10:04:00.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.782+0000 7fbf60052700 1 Processor -- start 2023-10-16T10:04:00.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.782+0000 7fbf60052700 1 -- start start 2023-10-16T10:04:00.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf60052700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf5806ab40 0x7fbf5806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:00.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf60052700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf58161270 con 0x7fbf5806e5d0 2023-10-16T10:04:00.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf60052700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf58161510 con 0x7fbf580729b0 2023-10-16T10:04:00.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf60052700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf581617f0 con 0x7fbf5806ab40 2023-10-16T10:04:00.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5ddee700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf5806ab40 0x7fbf5806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:00.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5d5ed700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbf5806e5d0 0x7fbf580722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59782/0 (socket says 172.21.15.158:59782) 2023-10-16T10:04:00.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5d5ed700 1 -- 172.21.15.158:0/19079028 learned_addr learned my addr 172.21.15.158:0/19079028 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:00.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5ddee700 1 -- 172.21.15.158:0/19079028 >> v1:172.21.15.172:6789/0 conn(0x7fbf580729b0 legacy=0x7fbf5815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:00.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5ddee700 1 -- 172.21.15.158:0/19079028 >> v1:172.21.15.158:6789/0 conn(0x7fbf5806e5d0 legacy=0x7fbf580722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:00.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5ddee700 1 -- 172.21.15.158:0/19079028 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58161ad0 con 0x7fbf5806ab40 2023-10-16T10:04:00.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5cdec700 1 -- 172.21.15.158:0/19079028 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942888682 0 0) 0x7fbf58161270 con 0x7fbf5806e5d0 2023-10-16T10:04:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.783+0000 7fbf5ddee700 1 --2- 172.21.15.158:0/19079028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf5806d820 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009840 tx=0x7fbf54009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2646c9bd49eb1ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf5cdec700 1 -- 172.21.15.158:0/19079028 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54011830 con 0x7fbf5806ab40 2023-10-16T10:04:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf5cdec700 1 -- 172.21.15.158:0/19079028 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf54011990 con 0x7fbf5806ab40 2023-10-16T10:04:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf5cdec700 1 -- 172.21.15.158:0/19079028 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54011d20 con 0x7fbf5806ab40 2023-10-16T10:04:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf60052700 1 -- 172.21.15.158:0/19079028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 msgr2=0x7fbf5806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf60052700 1 --2- 172.21.15.158:0/19079028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf5806d820 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009840 tx=0x7fbf54009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf60052700 1 -- 172.21.15.158:0/19079028 shutdown_connections 2023-10-16T10:04:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf60052700 1 --2- 172.21.15.158:0/19079028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf5806d820 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf60052700 1 -- 172.21.15.158:0/19079028 >> 172.21.15.158:0/19079028 conn(0x7fbf580d02b0 msgr2=0x7fbf580d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf60052700 1 -- 172.21.15.158:0/19079028 shutdown_connections 2023-10-16T10:04:00.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.784+0000 7fbf60052700 1 -- 172.21.15.158:0/19079028 wait complete. 2023-10-16T10:04:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.785+0000 7fbf60052700 1 Processor -- start 2023-10-16T10:04:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.785+0000 7fbf60052700 1 -- start start 2023-10-16T10:04:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf60052700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf581f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf60052700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf5806e5d0 0x7fbf581f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf60052700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf580729b0 0x7fbf5815e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf60052700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf58160800 con 0x7fbf5806ab40 2023-10-16T10:04:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf60052700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580b1c70 con 0x7fbf580729b0 2023-10-16T10:04:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf60052700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf581f7300 con 0x7fbf5806e5d0 2023-10-16T10:04:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf5ddee700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf581f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf5ddee700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf581f6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59612/0 (socket says 172.21.15.158:59612) 2023-10-16T10:04:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf5ddee700 1 -- 172.21.15.158:0/3409983741 learned_addr learned my addr 172.21.15.158:0/3409983741 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.786+0000 7fbf5ddee700 1 -- 172.21.15.158:0/3409983741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf5806e5d0 msgr2=0x7fbf581f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf5ddee700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf5806e5d0 0x7fbf581f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf5ddee700 1 -- 172.21.15.158:0/3409983741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf580729b0 msgr2=0x7fbf5815e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:00.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf5ddee700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf580729b0 0x7fbf5815e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:00.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf5ddee700 1 -- 172.21.15.158:0/3409983741 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540094a0 con 0x7fbf5806ab40 2023-10-16T10:04:00.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf5ddee700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf581f6dc0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009f50 tx=0x7fbf54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:00.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf4e7fc700 1 -- 172.21.15.158:0/3409983741 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540036a0 con 0x7fbf5806ab40 2023-10-16T10:04:00.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf4e7fc700 1 -- 172.21.15.158:0/3409983741 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5401f070 con 0x7fbf5806ab40 2023-10-16T10:04:00.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf581f2cd0 con 0x7fbf5806ab40 2023-10-16T10:04:00.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf4e7fc700 1 -- 172.21.15.158:0/3409983741 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54003a60 con 0x7fbf5806ab40 2023-10-16T10:04:00.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.787+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf58160a50 con 0x7fbf5806ab40 2023-10-16T10:04:00.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.788+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf580b14c0 con 0x7fbf5806ab40 2023-10-16T10:04:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.788+0000 7fbf4e7fc700 1 -- 172.21.15.158:0/3409983741 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf54003bc0 con 0x7fbf5806ab40 2023-10-16T10:04:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.789+0000 7fbf4e7fc700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fbf3c06b070 0x7fbf3c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.789+0000 7fbf4e7fc700 1 -- 172.21.15.158:0/3409983741 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5629+0+0 (secure 0 0 0) 0x7fbf54021070 con 0x7fbf5806ab40 2023-10-16T10:04:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.789+0000 7fbf5d5ed700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fbf3c06b070 0x7fbf3c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.789+0000 7fbf5d5ed700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fbf3c06b070 0x7fbf3c06d530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000c00 tx=0x7fbf48007990 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:00.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:00.793+0000 7fbf4e7fc700 1 -- 172.21.15.158:0/3409983741 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf580b14c0 con 0x7fbf5806ab40 2023-10-16T10:04:01.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.061+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbf580b14c0 con 0x7fbf5806ab40 2023-10-16T10:04:01.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.062+0000 7fbf4e7fc700 1 -- 172.21.15.158:0/3409983741 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbf580b14c0 con 0x7fbf5806ab40 2023-10-16T10:04:01.063 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:01.064 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:04:01.064 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T10:04:01.064 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.064 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T10:04:01.065 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T10:04:01.065 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.065 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.065 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:04:01.065 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:04:01.066 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.066 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.066 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.066 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.066 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T10:04:01.067 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T10:04:01.067 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.067 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.067 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:04:01.067 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:04:01.068 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.068 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.068 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.068 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.068 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T10:04:01.068 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T10:04:01.069 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.069 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.069 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:04:01.069 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:04:01.069 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.069 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.070 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.070 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.070 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T10:04:01.070 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T10:04:01.070 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.071 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.071 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:04:01.071 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:04:01.071 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.071 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.071 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.072 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.072 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T10:04:01.072 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T10:04:01.072 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.072 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.073 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.073 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.073 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.073 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.073 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.073 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T10:04:01.074 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T10:04:01.074 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.074 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.074 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.075 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.075 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.075 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.075 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.075 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T10:04:01.076 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T10:04:01.076 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.076 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.076 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.076 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.076 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.077 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.077 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.077 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T10:04:01.077 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T10:04:01.077 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.078 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.078 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.078 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.078 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.078 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.079 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.079 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T10:04:01.079 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T10:04:01.079 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.079 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.079 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.080 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.080 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.080 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.080 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.080 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T10:04:01.081 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T10:04:01.081 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.081 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.081 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.081 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.082 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.082 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.082 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.082 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T10:04:01.082 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T10:04:01.083 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.083 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.083 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.083 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.083 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.084 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.084 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.084 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T10:04:01.084 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T10:04:01.084 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.085 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T10:04:01.085 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T10:04:01.085 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.085 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.085 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.086 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.086 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T10:04:01.086 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T10:04:01.086 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.086 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.087 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T10:04:01.087 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T10:04:01.087 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.087 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.087 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.088 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T10:04:01.088 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T10:04:01.088 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.088 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.088 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:04:01.089 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:04:01.089 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:04:01.089 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.089 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.089 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.089 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T10:04:01.090 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T10:04:01.090 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.090 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.090 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:04:01.090 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:04:01.091 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:04:01.091 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.091 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.091 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.091 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T10:04:01.091 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T10:04:01.092 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.092 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T10:04:01.092 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T10:04:01.092 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T10:04:01.092 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T10:04:01.093 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.093 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.093 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.093 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T10:04:01.093 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T10:04:01.093 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.094 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:04:01.094 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T10:04:01.094 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T10:04:01.094 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.095 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.095 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.095 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.095 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T10:04:01.095 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T10:04:01.096 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.096 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:04:01.096 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:04:01.096 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.096 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.097 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T10:04:01.097 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:01.097 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T10:04:01.097 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T10:04:01.097 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T10:04:01.097 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T10:04:01.098 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T10:04:01.098 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T10:04:01.098 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.098 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:01.098 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T10:04:01.099 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:04:01.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fbf3c06b070 msgr2=0x7fbf3c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fbf3c06b070 0x7fbf3c06d530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000c00 tx=0x7fbf48007990 comp rx=0 tx=0).stop 2023-10-16T10:04:01.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 msgr2=0x7fbf581f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf581f6dc0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009f50 tx=0x7fbf54004750 comp rx=0 tx=0).stop 2023-10-16T10:04:01.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 shutdown_connections 2023-10-16T10:04:01.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf580729b0 0x7fbf5815e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5806ab40 0x7fbf581f6dc0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fbf3c06b070 0x7fbf3c06d530 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 --2- 172.21.15.158:0/3409983741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf5806e5d0 0x7fbf581f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 >> 172.21.15.158:0/3409983741 conn(0x7fbf580d02b0 msgr2=0x7fbf5815d250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:01.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 shutdown_connections 2023-10-16T10:04:01.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.064+0000 7fbf60052700 1 -- 172.21.15.158:0/3409983741 wait complete. 2023-10-16T10:04:01.103 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:04:01.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.223+0000 7f342eb43700 1 Processor -- start 2023-10-16T10:04:01.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.223+0000 7f342eb43700 1 -- start start 2023-10-16T10:04:01.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.224+0000 7f342eb43700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f34280ce640 0x7f34280cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.224+0000 7f342eb43700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280cae30 0x7f34280cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.224+0000 7f342eb43700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34280cef60 con 0x7f34280c65b0 2023-10-16T10:04:01.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.224+0000 7f342eb43700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342815e480 con 0x7f34280ce640 2023-10-16T10:04:01.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.224+0000 7f342eb43700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342815d800 con 0x7f34280cae30 2023-10-16T10:04:01.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.224+0000 7f342c8df700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f34280ce640 0x7f34280cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.224+0000 7f342d0e0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280cae30 0x7f34280cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342c8df700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f34280ce640 0x7f34280cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:60800/0 (socket says 172.21.15.158:60800) 2023-10-16T10:04:01.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342d0e0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280cae30 0x7f34280cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:42226/0 (socket says 172.21.15.158:42226) 2023-10-16T10:04:01.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342c8df700 1 -- 172.21.15.158:0/1800312515 learned_addr learned my addr 172.21.15.158:0/1800312515 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:01.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342c8df700 1 -- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280cae30 msgr2=0x7f34280cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:01.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342c8df700 1 --2- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280cae30 0x7f34280cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342c8df700 1 -- 172.21.15.158:0/1800312515 >> v1:172.21.15.172:6789/0 conn(0x7f34280c65b0 legacy=0x7f342806f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342c8df700 1 -- 172.21.15.158:0/1800312515 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342815dae0 con 0x7f34280ce640 2023-10-16T10:04:01.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342d0e0700 1 --2- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280cae30 0x7f34280cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:04:01.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f342c8df700 1 --2- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280ce640 0x7f34280cea20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f341c00aa90 tx=0x7f341c00ada0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca76a860be796d8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f34277fe700 1 -- 172.21.15.158:0/1800312515 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c00cb70 con 0x7f34280ce640 2023-10-16T10:04:01.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f34277fe700 1 -- 172.21.15.158:0/1800312515 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c00ccd0 con 0x7f34280ce640 2023-10-16T10:04:01.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.225+0000 7f34277fe700 1 -- 172.21.15.158:0/1800312515 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c018620 con 0x7f34280ce640 2023-10-16T10:04:01.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 -- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280ce640 msgr2=0x7f34280cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 --2- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280ce640 0x7f34280cea20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f341c00aa90 tx=0x7f341c00ada0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 -- 172.21.15.158:0/1800312515 shutdown_connections 2023-10-16T10:04:01.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 --2- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280cae30 0x7f34280cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 --2- 172.21.15.158:0/1800312515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280ce640 0x7f34280cea20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 -- 172.21.15.158:0/1800312515 >> 172.21.15.158:0/1800312515 conn(0x7f34280cc4b0 msgr2=0x7f34280cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:01.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 -- 172.21.15.158:0/1800312515 shutdown_connections 2023-10-16T10:04:01.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.226+0000 7f342eb43700 1 -- 172.21.15.158:0/1800312515 wait complete. 2023-10-16T10:04:01.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.227+0000 7f342eb43700 1 Processor -- start 2023-10-16T10:04:01.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.228+0000 7f342eb43700 1 -- start start 2023-10-16T10:04:01.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34280c65b0 0x7f34280c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280cae30 0x7f34280c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280ce640 0x7f34281f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342812d230 con 0x7f34280c65b0 2023-10-16T10:04:01.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342812cf90 con 0x7f34280cae30 2023-10-16T10:04:01.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342812d4d0 con 0x7f34280ce640 2023-10-16T10:04:01.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34280c65b0 0x7f34280c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34280c65b0 0x7f34280c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59626/0 (socket says 172.21.15.158:59626) 2023-10-16T10:04:01.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 -- 172.21.15.158:0/2932891003 learned_addr learned my addr 172.21.15.158:0/2932891003 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:01.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342d0e0700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280ce640 0x7f34281f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f3427fff700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280cae30 0x7f34280c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 -- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280ce640 msgr2=0x7f34281f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280ce640 0x7f34281f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 -- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280cae30 msgr2=0x7f34280c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280cae30 0x7f34280c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 -- 172.21.15.158:0/2932891003 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420009a70 con 0x7f34280c65b0 2023-10-16T10:04:01.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f3427fff700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280cae30 0x7f34280c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:01.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342d0e0700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280ce640 0x7f34281f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:01.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342c8df700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34280c65b0 0x7f34280c8d60 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f34280b2720 tx=0x7f341c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f34257fa700 1 -- 172.21.15.158:0/2932891003 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c003ab0 con 0x7f34280c65b0 2023-10-16T10:04:01.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f34257fa700 1 -- 172.21.15.158:0/2932891003 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c003c30 con 0x7f34280c65b0 2023-10-16T10:04:01.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f34257fa700 1 -- 172.21.15.158:0/2932891003 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c020400 con 0x7f34280c65b0 2023-10-16T10:04:01.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c00a6f0 con 0x7f34280c65b0 2023-10-16T10:04:01.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.229+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34280c9650 con 0x7f34280c65b0 2023-10-16T10:04:01.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.230+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34280681b0 con 0x7f34280c65b0 2023-10-16T10:04:01.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.231+0000 7f34257fa700 1 -- 172.21.15.158:0/2932891003 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f341c018400 con 0x7f34280c65b0 2023-10-16T10:04:01.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.231+0000 7f34257fa700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f340c06b050 0x7f340c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.231+0000 7f34257fa700 1 -- 172.21.15.158:0/2932891003 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f341c08acc0 con 0x7f34280c65b0 2023-10-16T10:04:01.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.231+0000 7f3427fff700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f340c06b050 0x7f340c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.232+0000 7f3427fff700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f340c06b050 0x7f340c06d510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34280c92a0 tx=0x7f3414009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.236+0000 7f34257fa700 1 -- 172.21.15.158:0/2932891003 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f341c055c30 con 0x7f34280c65b0 2023-10-16T10:04:01.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.430+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34280c5300 con 0x7f34280c65b0 2023-10-16T10:04:01.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.431+0000 7f34257fa700 1 -- 172.21.15.158:0/2932891003 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+13774 (secure 0 0 0) 0x7f341c025090 con 0x7f34280c65b0 2023-10-16T10:04:01.432 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:01.433 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":149,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:04:00.651492+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:04:01.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f340c06b050 msgr2=0x7f340c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f340c06b050 0x7f340c06d510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34280c92a0 tx=0x7f3414009040 comp rx=0 tx=0).stop 2023-10-16T10:04:01.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34280c65b0 msgr2=0x7f34280c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34280c65b0 0x7f34280c8d60 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f34280b2720 tx=0x7f341c0047c0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 shutdown_connections 2023-10-16T10:04:01.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f340c06b050 0x7f340c06d510 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34280ce640 0x7f34281f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34280c65b0 0x7f34280c8d60 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 --2- 172.21.15.158:0/2932891003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34280cae30 0x7f34280c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 >> 172.21.15.158:0/2932891003 conn(0x7f34280cc4b0 msgr2=0x7f34280af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:01.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 shutdown_connections 2023-10-16T10:04:01.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.433+0000 7f342eb43700 1 -- 172.21.15.158:0/2932891003 wait complete. 2023-10-16T10:04:01.447 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T10:04:01.447 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T10:04:01.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.570+0000 7fb5144de700 1 Processor -- start 2023-10-16T10:04:01.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.570+0000 7fb5144de700 1 -- start start 2023-10-16T10:04:01.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb5144de700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb50c15da40 0x7fb50c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb5144de700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb50c0cc590 0x7fb50c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb5144de700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c06ae10 0x7fb50c0ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb5144de700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50c06b280 con 0x7fb50c15da40 2023-10-16T10:04:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb5144de700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50c0cbd30 con 0x7fb50c0cc590 2023-10-16T10:04:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb5144de700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c0cc1c0 con 0x7fb50c06ae10 2023-10-16T10:04:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb51227a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb50c15da40 0x7fb50c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb51227a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb50c15da40 0x7fb50c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59634/0 (socket says 172.21.15.158:59634) 2023-10-16T10:04:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.571+0000 7fb51227a700 1 -- 172.21.15.158:0/2875239500 learned_addr learned my addr 172.21.15.158:0/2875239500 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb511a79700 1 --2- 172.21.15.158:0/2875239500 >> v2:172.21.15.172:3300/0 conn(0x7fb50c0cc590 0x7fb50c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb512a7b700 1 --2- 172.21.15.158:0/2875239500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c06ae10 0x7fb50c0ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb51227a700 1 -- 172.21.15.158:0/2875239500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c06ae10 msgr2=0x7fb50c0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb51227a700 1 --2- 172.21.15.158:0/2875239500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c06ae10 0x7fb50c0ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb51227a700 1 -- 172.21.15.158:0/2875239500 >> v2:172.21.15.172:3300/0 conn(0x7fb50c0cc590 msgr2=0x7fb50c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb51227a700 1 --2- 172.21.15.158:0/2875239500 >> v2:172.21.15.172:3300/0 conn(0x7fb50c0cc590 0x7fb50c06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb51227a700 1 -- 172.21.15.158:0/2875239500 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50c15de20 con 0x7fb50c15da40 2023-10-16T10:04:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb51227a700 1 --2- 172.21.15.158:0/2875239500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0caa20 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0d1380 tx=0x7fb4fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98dd00a44738c565 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb511278700 1 -- 172.21.15.158:0/2875239500 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc00b870 con 0x7fb50c15da40 2023-10-16T10:04:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb511278700 1 -- 172.21.15.158:0/2875239500 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc00b9d0 con 0x7fb50c15da40 2023-10-16T10:04:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb511278700 1 -- 172.21.15.158:0/2875239500 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc00bd60 con 0x7fb50c15da40 2023-10-16T10:04:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb5144de700 1 -- 172.21.15.158:0/2875239500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 msgr2=0x7fb50c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb5144de700 1 --2- 172.21.15.158:0/2875239500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0caa20 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0d1380 tx=0x7fb4fc009bd0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb5144de700 1 -- 172.21.15.158:0/2875239500 shutdown_connections 2023-10-16T10:04:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb5144de700 1 --2- 172.21.15.158:0/2875239500 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c06ae10 0x7fb50c0ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb5144de700 1 --2- 172.21.15.158:0/2875239500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0caa20 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb5144de700 1 --2- 172.21.15.158:0/2875239500 >> v2:172.21.15.172:3300/0 conn(0x7fb50c0cc590 0x7fb50c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.572+0000 7fb5144de700 1 -- 172.21.15.158:0/2875239500 >> 172.21.15.158:0/2875239500 conn(0x7fb50c099a10 msgr2=0x7fb50c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.573+0000 7fb5144de700 1 -- 172.21.15.158:0/2875239500 shutdown_connections 2023-10-16T10:04:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.573+0000 7fb5144de700 1 -- 172.21.15.158:0/2875239500 wait complete. 2023-10-16T10:04:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 Processor -- start 2023-10-16T10:04:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 -- start start 2023-10-16T10:04:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb50c06ae10 0x7fb50c159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c0cc590 0x7fb50c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c1f6730 con 0x7fb50c15da40 2023-10-16T10:04:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c1f69a0 con 0x7fb50c06ae10 2023-10-16T10:04:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.574+0000 7fb5144de700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c1f6c80 con 0x7fb50c0cc590 2023-10-16T10:04:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59642/0 (socket says 172.21.15.158:59642) 2023-10-16T10:04:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 -- 172.21.15.158:0/1590305409 learned_addr learned my addr 172.21.15.158:0/1590305409 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb51227a700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb50c06ae10 0x7fb50c159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb511a79700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c0cc590 0x7fb50c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 -- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c0cc590 msgr2=0x7fb50c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c0cc590 0x7fb50c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 -- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb50c06ae10 msgr2=0x7fb50c159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb50c06ae10 0x7fb50c159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 -- 172.21.15.158:0/1590305409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc009550 con 0x7fb50c15da40 2023-10-16T10:04:01.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb512a7b700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0d09e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb50400ccf0 tx=0x7fb50400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb502ffd700 1 -- 172.21.15.158:0/1590305409 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504003a20 con 0x7fb50c15da40 2023-10-16T10:04:01.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb502ffd700 1 -- 172.21.15.158:0/1590305409 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb504003b80 con 0x7fb50c15da40 2023-10-16T10:04:01.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50c1f6f60 con 0x7fb50c15da40 2023-10-16T10:04:01.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb502ffd700 1 -- 172.21.15.158:0/1590305409 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504004020 con 0x7fb50c15da40 2023-10-16T10:04:01.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.575+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50c1f7480 con 0x7fb50c15da40 2023-10-16T10:04:01.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.576+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50c1f71a0 con 0x7fb50c15da40 2023-10-16T10:04:01.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.577+0000 7fb502ffd700 1 -- 172.21.15.158:0/1590305409 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb504004180 con 0x7fb50c15da40 2023-10-16T10:04:01.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.577+0000 7fb502ffd700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb4f006b020 0x7fb4f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.577+0000 7fb502ffd700 1 -- 172.21.15.158:0/1590305409 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5629+0+0 (secure 0 0 0) 0x7fb504012070 con 0x7fb50c15da40 2023-10-16T10:04:01.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.577+0000 7fb51227a700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb4f006b020 0x7fb4f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.577+0000 7fb51227a700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb4f006b020 0x7fb4f006d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc014040 tx=0x7fb4fc008030 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.581+0000 7fb502ffd700 1 -- 172.21.15.158:0/1590305409 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb50c1f71a0 con 0x7fb50c15da40 2023-10-16T10:04:01.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.775+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb50c1f71a0 con 0x7fb50c15da40 2023-10-16T10:04:01.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.775+0000 7fb502ffd700 1 -- 172.21.15.158:0/1590305409 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb50c1f71a0 con 0x7fb50c15da40 2023-10-16T10:04:01.777 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T10:04:01.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb4f006b020 msgr2=0x7fb4f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb4f006b020 0x7fb4f006d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc014040 tx=0x7fb4fc008030 comp rx=0 tx=0).stop 2023-10-16T10:04:01.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 msgr2=0x7fb50c0d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0d09e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb50400ccf0 tx=0x7fb50400a570 comp rx=0 tx=0).stop 2023-10-16T10:04:01.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 shutdown_connections 2023-10-16T10:04:01.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb50c15da40 0x7fb50c0d09e0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb50c06ae10 0x7fb50c159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb4f006b020 0x7fb4f006d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 --2- 172.21.15.158:0/1590305409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb50c0cc590 0x7fb50c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.777+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 >> 172.21.15.158:0/1590305409 conn(0x7fb50c099a10 msgr2=0x7fb50c0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:01.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.778+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 shutdown_connections 2023-10-16T10:04:01.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.778+0000 7fb5144de700 1 -- 172.21.15.158:0/1590305409 wait complete. 2023-10-16T10:04:01.791 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T10:04:01.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.911+0000 7f5e230b2700 1 Processor -- start 2023-10-16T10:04:01.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.911+0000 7f5e230b2700 1 -- start start 2023-10-16T10:04:01.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e230b2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5e1c06ab40 0x7f5e1c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e230b2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c161270 con 0x7f5e1c0729b0 2023-10-16T10:04:01.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e230b2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c161510 con 0x7f5e1c06e5d0 2023-10-16T10:04:01.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e230b2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c1617f0 con 0x7f5e1c06ab40 2023-10-16T10:04:01.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e2164f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5e1c0729b0 0x7f5e1c15df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59828/0 (socket says 172.21.15.158:59828) 2023-10-16T10:04:01.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e2164f700 1 -- 172.21.15.158:0/1948723580 learned_addr learned my addr 172.21.15.158:0/1948723580 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:01.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238385997 0 0) 0x7f5e1c161270 con 0x7f5e1c0729b0 2023-10-16T10:04:01.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e20e4e700 1 --2- 172.21.15.158:0/1948723580 >> v2:172.21.15.172:3300/0 conn(0x7f5e1c06ab40 0x7f5e1c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c0035b0 con 0x7f5e1c0729b0 2023-10-16T10:04:01.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1765366849 0 0) 0x7f5e0c0035b0 con 0x7f5e1c0729b0 2023-10-16T10:04:01.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 >> v2:172.21.15.172:3300/0 conn(0x7f5e1c06ab40 msgr2=0x7f5e1c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:01.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 --2- 172.21.15.158:0/1948723580 >> v2:172.21.15.172:3300/0 conn(0x7f5e1c06ab40 0x7f5e1c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 >> v1:172.21.15.172:6789/0 conn(0x7f5e1c06e5d0 legacy=0x7f5e1c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c161ad0 con 0x7f5e1c0729b0 2023-10-16T10:04:01.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f5e140019f0 con 0x7f5e1c0729b0 2023-10-16T10:04:01.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 >> v1:172.21.15.158:6789/0 conn(0x7f5e1c0729b0 legacy=0x7f5e1c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c003a00 0x7f5e0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.912+0000 7f5e1b7fe700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e0c004850 0x7f5e0c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1b7fe700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c005210 0x7f5e0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c1617f0 con 0x7f5e0c004850 2023-10-16T10:04:01.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c00c0b0 con 0x7f5e0c003a00 2023-10-16T10:04:01.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c00c270 con 0x7f5e0c005210 2023-10-16T10:04:01.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1bfff700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e0c004850 0x7f5e0c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e20e4e700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c003a00 0x7f5e0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e2164f700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c005210 0x7f5e0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1bfff700 1 -- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c005210 msgr2=0x7f5e0c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:01.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1bfff700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c005210 0x7f5e0c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1bfff700 1 -- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c003a00 msgr2=0x7f5e0c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:01.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1bfff700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c003a00 0x7f5e0c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1bfff700 1 -- 172.21.15.158:0/1948723580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c0035b0 con 0x7f5e0c004850 2023-10-16T10:04:01.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e2164f700 1 -- 172.21.15.158:0/1948723580 reap_dead start 2023-10-16T10:04:01.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1bfff700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e0c004850 0x7f5e0c004cd0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c06b560 tx=0x7f5e0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614a4d75251f309 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e080012f0 con 0x7f5e0c004850 2023-10-16T10:04:01.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e08005310 con 0x7f5e0c004850 2023-10-16T10:04:01.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.913+0000 7f5e1b7fe700 1 -- 172.21.15.158:0/1948723580 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08006860 con 0x7f5e0c004850 2023-10-16T10:04:01.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.914+0000 7f5e230b2700 1 -- 172.21.15.158:0/1948723580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e0c004850 msgr2=0x7f5e0c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.914+0000 7f5e230b2700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e0c004850 0x7f5e0c004cd0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c06b560 tx=0x7f5e0c005880 comp rx=0 tx=0).stop 2023-10-16T10:04:01.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.914+0000 7f5e230b2700 1 -- 172.21.15.158:0/1948723580 shutdown_connections 2023-10-16T10:04:01.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.914+0000 7f5e230b2700 1 --2- 172.21.15.158:0/1948723580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e0c004850 0x7f5e0c004cd0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.914+0000 7f5e230b2700 1 -- 172.21.15.158:0/1948723580 >> 172.21.15.158:0/1948723580 conn(0x7f5e1c0d02b0 msgr2=0x7f5e1c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:01.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.914+0000 7f5e230b2700 1 -- 172.21.15.158:0/1948723580 shutdown_connections 2023-10-16T10:04:01.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.914+0000 7f5e230b2700 1 -- 172.21.15.158:0/1948723580 wait complete. 2023-10-16T10:04:01.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.915+0000 7f5e230b2700 1 Processor -- start 2023-10-16T10:04:01.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.915+0000 7f5e230b2700 1 -- start start 2023-10-16T10:04:01.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e230b2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c004850 0x7f5e1c15e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e230b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e1c0c3c80 0x7f5e1c0b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e230b2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e1c156e90 0x7f5e1c0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e230b2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c0b50d0 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e230b2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c157340 con 0x7f5e1c156e90 2023-10-16T10:04:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e230b2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c15e810 con 0x7f5e0c004850 2023-10-16T10:04:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e1c0c3c80 0x7f5e1c0b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e1c0c3c80 0x7f5e1c0b4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59660/0 (socket says 172.21.15.158:59660) 2023-10-16T10:04:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 -- 172.21.15.158:0/2578867548 learned_addr learned my addr 172.21.15.158:0/2578867548 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e2164f700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e1c156e90 0x7f5e1c0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e20e4e700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c004850 0x7f5e1c15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 -- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c004850 msgr2=0x7f5e1c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c004850 0x7f5e1c15e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 -- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e1c156e90 msgr2=0x7f5e1c0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e1c156e90 0x7f5e1c0b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 -- 172.21.15.158:0/2578867548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c15eaf0 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e20e4e700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c004850 0x7f5e1c15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e1bfff700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e1c0c3c80 0x7f5e1c0b4b90 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f5e08014040 tx=0x7f5e08001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e197fa700 1 -- 172.21.15.158:0/2578867548 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08001290 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.916+0000 7f5e197fa700 1 -- 172.21.15.158:0/2578867548 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e080050d0 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.917+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c0c83d0 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.917+0000 7f5e197fa700 1 -- 172.21.15.158:0/2578867548 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08007280 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.917+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c0c8990 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.918+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c0c8b50 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.918+0000 7f5e197fa700 1 -- 172.21.15.158:0/2578867548 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5e080073e0 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.918+0000 7f5e197fa700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5e0006b020 0x7f5e0006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:01.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.919+0000 7f5e197fa700 1 -- 172.21.15.158:0/2578867548 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f5e0808a6e0 con 0x7f5e1c0c3c80 2023-10-16T10:04:01.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.919+0000 7f5e20e4e700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5e0006b020 0x7f5e0006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:01.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.919+0000 7f5e20e4e700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5e0006b020 0x7f5e0006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009ce0 tx=0x7f5e10008040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:01.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:01.923+0000 7f5e197fa700 1 -- 172.21.15.158:0/2578867548 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5e08055880 con 0x7f5e1c0c3c80 2023-10-16T10:04:02.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.116+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5e1c0c8d00 con 0x7f5e1c0c3c80 2023-10-16T10:04:02.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.116+0000 7f5e197fa700 1 -- 172.21.15.158:0/2578867548 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v149) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5e0802b060 con 0x7f5e1c0c3c80 2023-10-16T10:04:02.118 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:02.118 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:04:02.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:02.119 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T10:04:02.119 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T10:04:02.119 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:02.119 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:04:02.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5e0006b020 msgr2=0x7f5e0006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5e0006b020 0x7f5e0006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009ce0 tx=0x7f5e10008040 comp rx=0 tx=0).stop 2023-10-16T10:04:02.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e1c0c3c80 msgr2=0x7f5e1c0b4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e1c0c3c80 0x7f5e1c0b4b90 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f5e08014040 tx=0x7f5e08001120 comp rx=0 tx=0).stop 2023-10-16T10:04:02.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 shutdown_connections 2023-10-16T10:04:02.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e1c156e90 0x7f5e1c0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5e0006b020 0x7f5e0006d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c004850 0x7f5e1c15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 --2- 172.21.15.158:0/2578867548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5e1c0c3c80 0x7f5e1c0b4b90 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 >> 172.21.15.158:0/2578867548 conn(0x7f5e1c0d02b0 msgr2=0x7f5e1c0b1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:02.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 shutdown_connections 2023-10-16T10:04:02.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.118+0000 7f5e230b2700 1 -- 172.21.15.158:0/2578867548 wait complete. 2023-10-16T10:04:02.132 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T10:04:02.132 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T10:04:02.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.257+0000 7f4a067ce700 1 Processor -- start 2023-10-16T10:04:02.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.257+0000 7f4a067ce700 1 -- start start 2023-10-16T10:04:02.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.257+0000 7f4a067ce700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4a000c4e60 0x7f4a0006b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.257+0000 7f4a067ce700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a000c52b0 con 0x7f4a00160d40 2023-10-16T10:04:02.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.257+0000 7f4a067ce700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a000ce110 con 0x7f4a000c49b0 2023-10-16T10:04:02.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.257+0000 7f4a067ce700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a000c61e0 con 0x7f4a000c4e60 2023-10-16T10:04:02.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49fffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4a00160d40 0x7f4a000ceee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59842/0 (socket says 172.21.15.158:59842) 2023-10-16T10:04:02.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49fffff700 1 -- 172.21.15.158:0/2273387003 learned_addr learned my addr 172.21.15.158:0/2273387003 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:02.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f4a04d6b700 1 --2- 172.21.15.158:0/2273387003 >> v2:172.21.15.172:3300/0 conn(0x7f4a000c4e60 0x7f4a0006b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855991737 0 0) 0x7f4a000c52b0 con 0x7f4a00160d40 2023-10-16T10:04:02.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec0035b0 con 0x7f4a00160d40 2023-10-16T10:04:02.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204689256 0 0) 0x7f49ec0035b0 con 0x7f4a00160d40 2023-10-16T10:04:02.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 >> v2:172.21.15.172:3300/0 conn(0x7f4a000c4e60 msgr2=0x7f4a0006b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:02.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 --2- 172.21.15.158:0/2273387003 >> v2:172.21.15.172:3300/0 conn(0x7f4a000c4e60 0x7f4a0006b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 >> v1:172.21.15.172:6789/0 conn(0x7f4a000c49b0 legacy=0x7f4a0006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000ce3f0 con 0x7f4a00160d40 2023-10-16T10:04:02.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f49f0001670 con 0x7f4a00160d40 2023-10-16T10:04:02.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f4a04d6b700 1 --2- 172.21.15.158:0/2273387003 >> v2:172.21.15.172:3300/0 conn(0x7f4a000c4e60 0x7f4a0006b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:04:02.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 >> v1:172.21.15.158:6789/0 conn(0x7f4a00160d40 legacy=0x7f4a000ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49ec003a00 0x7f49ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49ec004850 0x7f49ec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f49ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec00c080 con 0x7f49ec005210 2023-10-16T10:04:02.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec00c1f0 con 0x7f49ec004850 2023-10-16T10:04:02.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec00c3b0 con 0x7f49ec003a00 2023-10-16T10:04:02.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f4a04d6b700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f49ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49f7fff700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49ec004850 0x7f49ec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.258+0000 7f49fffff700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49ec003a00 0x7f49ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a04d6b700 1 -- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49ec003a00 msgr2=0x7f49ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:02.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a04d6b700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49ec003a00 0x7f49ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a04d6b700 1 -- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49ec004850 msgr2=0x7f49ec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:02.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a04d6b700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49ec004850 0x7f49ec004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a04d6b700 1 -- 172.21.15.158:0/2273387003 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec0035b0 con 0x7f49ec005210 2023-10-16T10:04:02.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a04d6b700 1 -- 172.21.15.158:0/2273387003 reap_dead start 2023-10-16T10:04:02.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a04d6b700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f49ec00ba10 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f49f800e350 tx=0x7f49ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4361d8e08c44de1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:02.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f800edf0 con 0x7f49ec005210 2023-10-16T10:04:02.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f800f040 con 0x7f49ec005210 2023-10-16T10:04:02.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f49ff7fe700 1 -- 172.21.15.158:0/2273387003 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f80146c0 con 0x7f49ec005210 2023-10-16T10:04:02.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a067ce700 1 -- 172.21.15.158:0/2273387003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 msgr2=0x7f49ec00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.259+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f49ec00ba10 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f49f800e350 tx=0x7f49ec005880 comp rx=0 tx=0).stop 2023-10-16T10:04:02.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.260+0000 7f4a067ce700 1 -- 172.21.15.158:0/2273387003 shutdown_connections 2023-10-16T10:04:02.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.260+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2273387003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f49ec00ba10 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.260+0000 7f4a067ce700 1 -- 172.21.15.158:0/2273387003 >> 172.21.15.158:0/2273387003 conn(0x7f4a000cc5b0 msgr2=0x7f4a000cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:02.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.260+0000 7f4a067ce700 1 -- 172.21.15.158:0/2273387003 shutdown_connections 2023-10-16T10:04:02.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.260+0000 7f4a067ce700 1 -- 172.21.15.158:0/2273387003 wait complete. 2023-10-16T10:04:02.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 Processor -- start 2023-10-16T10:04:02.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 -- start start 2023-10-16T10:04:02.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f4a00155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a00155980 0x7f4a000b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a00157690 0x7f4a00157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a000b3f50 con 0x7f49ec005210 2023-10-16T10:04:02.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a000b47f0 con 0x7f4a00155980 2023-10-16T10:04:02.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.261+0000 7f4a067ce700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a00155780 con 0x7f4a00157690 2023-10-16T10:04:02.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f4a00155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f4a00155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59676/0 (socket says 172.21.15.158:59676) 2023-10-16T10:04:02.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 -- 172.21.15.158:0/2784429990 learned_addr learned my addr 172.21.15.158:0/2784429990 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:02.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49f7fff700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a00155980 0x7f4a000b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f4a04d6b700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a00157690 0x7f4a00157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 -- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a00157690 msgr2=0x7f4a00157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a00157690 0x7f4a00157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 -- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a00155980 msgr2=0x7f4a000b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a00155980 0x7f4a000b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 -- 172.21.15.158:0/2784429990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000b36e0 con 0x7f49ec005210 2023-10-16T10:04:02.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fffff700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f4a00155240 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f49f00072c0 tx=0x7f49f00075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:02.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fd7fa700 1 -- 172.21.15.158:0/2784429990 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0011580 con 0x7f49ec005210 2023-10-16T10:04:02.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fd7fa700 1 -- 172.21.15.158:0/2784429990 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f00116e0 con 0x7f49ec005210 2023-10-16T10:04:02.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a000b3980 con 0x7f49ec005210 2023-10-16T10:04:02.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f49fd7fa700 1 -- 172.21.15.158:0/2784429990 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0011a90 con 0x7f49ec005210 2023-10-16T10:04:02.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.262+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a000b4480 con 0x7f49ec005210 2023-10-16T10:04:02.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.263+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a00066db0 con 0x7f49ec005210 2023-10-16T10:04:02.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.264+0000 7f49fd7fa700 1 -- 172.21.15.158:0/2784429990 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f49f0011bf0 con 0x7f49ec005210 2023-10-16T10:04:02.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.264+0000 7f49fd7fa700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f49e006b020 0x7f49e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.264+0000 7f49fd7fa700 1 -- 172.21.15.158:0/2784429990 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f49f0088c60 con 0x7f49ec005210 2023-10-16T10:04:02.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.264+0000 7f49f7fff700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f49e006b020 0x7f49e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.265+0000 7f49f7fff700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f49e006b020 0x7f49e006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a000cf6b0 tx=0x7f49e8009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:02.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.268+0000 7f49fd7fa700 1 -- 172.21.15.158:0/2784429990 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49f0053ce0 con 0x7f49ec005210 2023-10-16T10:04:02.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.462+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4a000681b0 con 0x7f49ec005210 2023-10-16T10:04:02.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.645+0000 7f49fd7fa700 1 -- 172.21.15.158:0/2784429990 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v150) v1 ==== 138+0+0 (secure 0 0 0) 0x7f49f001a0b0 con 0x7f49ec005210 2023-10-16T10:04:02.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f49e006b020 msgr2=0x7f49e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f49e006b020 0x7f49e006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a000cf6b0 tx=0x7f49e8009040 comp rx=0 tx=0).stop 2023-10-16T10:04:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 msgr2=0x7f4a00155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f4a00155240 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f49f00072c0 tx=0x7f49f00075d0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 shutdown_connections 2023-10-16T10:04:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f49e006b020 0x7f49e006d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a00157690 0x7f4a00157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49ec005210 0x7f4a00155240 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 --2- 172.21.15.158:0/2784429990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a00155980 0x7f4a000b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 >> 172.21.15.158:0/2784429990 conn(0x7f4a000cc5b0 msgr2=0x7f4a000b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 shutdown_connections 2023-10-16T10:04:02.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.649+0000 7f4a067ce700 1 -- 172.21.15.158:0/2784429990 wait complete. 2023-10-16T10:04:02.653 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T10:04:02.666 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T10:04:02.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.794+0000 7f97b6bf2700 1 Processor -- start 2023-10-16T10:04:02.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.794+0000 7f97b6bf2700 1 -- start start 2023-10-16T10:04:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97b6bf2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f97b006a8d0 0x7f97b006ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97b6bf2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b0071d70 con 0x7f97b00b3b60 2023-10-16T10:04:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97b6bf2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b00af9d0 con 0x7f97b00af5f0 2023-10-16T10:04:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97b6bf2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b006b2c0 con 0x7f97b006a8d0 2023-10-16T10:04:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97affff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f97b00b3b60 0x7f97b00724e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59862/0 (socket says 172.21.15.158:59862) 2023-10-16T10:04:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97affff700 1 -- 172.21.15.158:0/1338734054 learned_addr learned my addr 172.21.15.158:0/1338734054 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97b518f700 1 --2- 172.21.15.158:0/1338734054 >> v2:172.21.15.172:3300/0 conn(0x7f97b006a8d0 0x7f97b006ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277559562 0 0) 0x7f97b0071d70 con 0x7f97b00b3b60 2023-10-16T10:04:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c0035b0 con 0x7f97b00b3b60 2023-10-16T10:04:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.795+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695455657 0 0) 0x7f97b00af9d0 con 0x7f97b00af5f0 2023-10-16T10:04:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b0071d70 con 0x7f97b00af5f0 2023-10-16T10:04:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97b518f700 1 -- 172.21.15.158:0/1338734054 >> v1:172.21.15.172:6789/0 conn(0x7f97b00af5f0 legacy=0x7f97b0161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97b518f700 1 -- 172.21.15.158:0/1338734054 >> v1:172.21.15.158:6789/0 conn(0x7f97b00b3b60 legacy=0x7f97b00724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 800808360 0 0) 0x7f979c0035b0 con 0x7f97b00b3b60 2023-10-16T10:04:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97b518f700 1 -- 172.21.15.158:0/1338734054 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b00b3970 con 0x7f97b006a8d0 2023-10-16T10:04:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97b518f700 1 --2- 172.21.15.158:0/1338734054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b006a8d0 0x7f97b006ad80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009840 tx=0x7f97a0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b10054975564bd0b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798001670 con 0x7f97b006a8d0 2023-10-16T10:04:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a0004030 con 0x7f97b006a8d0 2023-10-16T10:04:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.796+0000 7f97af7fe700 1 -- 172.21.15.158:0/1338734054 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a00043a0 con 0x7f97b006a8d0 2023-10-16T10:04:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.797+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1338734054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b006a8d0 msgr2=0x7f97b006ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.797+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1338734054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b006a8d0 0x7f97b006ad80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009840 tx=0x7f97a0009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.797+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1338734054 shutdown_connections 2023-10-16T10:04:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.797+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1338734054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b006a8d0 0x7f97b006ad80 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.797+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1338734054 >> 172.21.15.158:0/1338734054 conn(0x7f97b00ccd30 msgr2=0x7f97b00cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.797+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1338734054 shutdown_connections 2023-10-16T10:04:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.797+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1338734054 wait complete. 2023-10-16T10:04:02.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.798+0000 7f97b6bf2700 1 Processor -- start 2023-10-16T10:04:02.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.798+0000 7f97b6bf2700 1 -- start start 2023-10-16T10:04:02.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b6bf2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b006a8d0 0x7f97b01f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b6bf2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b00af5f0 0x7f97b00b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b6bf2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b00b3b60 0x7f97b00b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b6bf2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b00b1f50 con 0x7f97b00b3b60 2023-10-16T10:04:02.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b6bf2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b01f4230 con 0x7f97b00af5f0 2023-10-16T10:04:02.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b6bf2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b00b1340 con 0x7f97b006a8d0 2023-10-16T10:04:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b00b3b60 0x7f97b00b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b00b3b60 0x7f97b00b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59694/0 (socket says 172.21.15.158:59694) 2023-10-16T10:04:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 -- 172.21.15.158:0/1872719942 learned_addr learned my addr 172.21.15.158:0/1872719942 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97affff700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b00af5f0 0x7f97b00b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b498e700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b006a8d0 0x7f97b01f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 -- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b006a8d0 msgr2=0x7f97b01f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b006a8d0 0x7f97b01f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 -- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b00af5f0 msgr2=0x7f97b00b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b00af5f0 0x7f97b00b7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 -- 172.21.15.158:0/1872719942 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a00094a0 con 0x7f97b00b3b60 2023-10-16T10:04:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b518f700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b00b3b60 0x7f97b00b23e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009ea0 tx=0x7f97a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97ad7fa700 1 -- 172.21.15.158:0/1872719942 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0003e60 con 0x7f97b00b3b60 2023-10-16T10:04:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97ad7fa700 1 -- 172.21.15.158:0/1872719942 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a001f070 con 0x7f97b00b3b60 2023-10-16T10:04:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b00b1620 con 0x7f97b00b3b60 2023-10-16T10:04:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.799+0000 7f97ad7fa700 1 -- 172.21.15.158:0/1872719942 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0011870 con 0x7f97b00b3b60 2023-10-16T10:04:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.800+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b00b1be0 con 0x7f97b00b3b60 2023-10-16T10:04:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.800+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b00681b0 con 0x7f97b00b3b60 2023-10-16T10:04:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.801+0000 7f97ad7fa700 1 -- 172.21.15.158:0/1872719942 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97a0021030 con 0x7f97b00b3b60 2023-10-16T10:04:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.801+0000 7f97ad7fa700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f979006b020 0x7f979006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.801+0000 7f97ad7fa700 1 -- 172.21.15.158:0/1872719942 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5999+0+0 (secure 0 0 0) 0x7f97a008aba0 con 0x7f97b00b3b60 2023-10-16T10:04:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.801+0000 7f97b498e700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f979006b020 0x7f979006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.802+0000 7f97b498e700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f979006b020 0x7f979006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97b00c83e0 tx=0x7f97a8009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:02.806+0000 7f97ad7fa700 1 -- 172.21.15.158:0/1872719942 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97a0055ab0 con 0x7f97b00b3b60 2023-10-16T10:04:03.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.000+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f97b006acb0 con 0x7f97b00b3b60 2023-10-16T10:04:03.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.647+0000 7f97ad7fa700 1 -- 172.21.15.158:0/1872719942 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v151) v1 ==== 170+0+0 (secure 0 0 0) 0x7f97a00590d0 con 0x7f97b00b3b60 2023-10-16T10:04:03.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f979006b020 msgr2=0x7f979006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:03.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f979006b020 0x7f979006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97b00c83e0 tx=0x7f97a8009040 comp rx=0 tx=0).stop 2023-10-16T10:04:03.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b00b3b60 msgr2=0x7f97b00b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:03.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b00b3b60 0x7f97b00b23e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009ea0 tx=0x7f97a0004750 comp rx=0 tx=0).stop 2023-10-16T10:04:03.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 shutdown_connections 2023-10-16T10:04:03.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b00b3b60 0x7f97b00b23e0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f979006b020 0x7f979006d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b006a8d0 0x7f97b01f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 --2- 172.21.15.158:0/1872719942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b00af5f0 0x7f97b00b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.650+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 >> 172.21.15.158:0/1872719942 conn(0x7f97b00ccd30 msgr2=0x7f97b006de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:03.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.651+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 shutdown_connections 2023-10-16T10:04:03.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.651+0000 7f97b6bf2700 1 -- 172.21.15.158:0/1872719942 wait complete. 2023-10-16T10:04:03.667 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T10:04:03.676 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T10:04:03.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.800+0000 7f964db49700 1 Processor -- start 2023-10-16T10:04:03.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.800+0000 7f964db49700 1 -- start start 2023-10-16T10:04:03.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.800+0000 7f964db49700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 0x7f96480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.800+0000 7f964db49700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480cb750 con 0x7f964806b8e0 2023-10-16T10:04:03.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.800+0000 7f964db49700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480cba30 con 0x7f964806f4a0 2023-10-16T10:04:03.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.800+0000 7f964db49700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480cbd10 con 0x7f96480ceed0 2023-10-16T10:04:03.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f9647fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f964806b8e0 0x7f964806bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59878/0 (socket says 172.21.15.158:59878) 2023-10-16T10:04:03.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f9647fff700 1 -- 172.21.15.158:0/1400273949 learned_addr learned my addr 172.21.15.158:0/1400273949 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:03.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f9646ffd700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 0x7f96480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109509065 0 0) 0x7f96480cb750 con 0x7f964806b8e0 2023-10-16T10:04:03.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96300035b0 con 0x7f964806b8e0 2023-10-16T10:04:03.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046665020 0 0) 0x7f96480cba30 con 0x7f964806f4a0 2023-10-16T10:04:03.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96480cb750 con 0x7f964806f4a0 2023-10-16T10:04:03.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128891511 0 0) 0x7f96300035b0 con 0x7f964806b8e0 2023-10-16T10:04:03.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 msgr2=0x7f96480cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:03.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 0x7f96480cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 >> v1:172.21.15.172:6789/0 conn(0x7f964806f4a0 legacy=0x7f964815a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:03.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480cbff0 con 0x7f964806b8e0 2023-10-16T10:04:03.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f96400019f0 con 0x7f964806b8e0 2023-10-16T10:04:03.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 >> v1:172.21.15.158:6789/0 conn(0x7f964806b8e0 legacy=0x7f964806bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:03.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9630003bc0 0x7f9630003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630004850 0x7f9630004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 0x7f9630005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480cbd10 con 0x7f9630003bc0 2023-10-16T10:04:03.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963000bac0 con 0x7f9630004850 2023-10-16T10:04:03.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963000bc80 con 0x7f96480ceed0 2023-10-16T10:04:03.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96477fe700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9630003bc0 0x7f9630003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f9646ffd700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630004850 0x7f9630004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f9647fff700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 0x7f9630005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96477fe700 1 -- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 msgr2=0x7f9630005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:03.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96477fe700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480ceed0 0x7f9630005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96477fe700 1 -- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630004850 msgr2=0x7f9630004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:03.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96477fe700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630004850 0x7f9630004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.801+0000 7f96477fe700 1 -- 172.21.15.158:0/1400273949 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300035b0 con 0x7f9630003bc0 2023-10-16T10:04:03.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f96477fe700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9630003bc0 0x7f9630003fc0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f9638007fd0 tx=0x7f9630005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02dd14fdf5e39a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:03.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638001430 con 0x7f9630003bc0 2023-10-16T10:04:03.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96380052d0 con 0x7f9630003bc0 2023-10-16T10:04:03.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f96467fc700 1 -- 172.21.15.158:0/1400273949 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638001d40 con 0x7f9630003bc0 2023-10-16T10:04:03.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f964db49700 1 -- 172.21.15.158:0/1400273949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9630003bc0 msgr2=0x7f9630003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:03.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f964db49700 1 --2- 172.21.15.158:0/1400273949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9630003bc0 0x7f9630003fc0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f9638007fd0 tx=0x7f9630005880 comp rx=0 tx=0).stop 2023-10-16T10:04:03.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f9647fff700 1 -- 172.21.15.158:0/1400273949 reap_dead start 2023-10-16T10:04:03.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f964db49700 1 -- 172.21.15.158:0/1400273949 shutdown_connections 2023-10-16T10:04:03.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f964db49700 1 -- 172.21.15.158:0/1400273949 >> 172.21.15.158:0/1400273949 conn(0x7f9648070fe0 msgr2=0x7f96480713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:03.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f964db49700 1 -- 172.21.15.158:0/1400273949 shutdown_connections 2023-10-16T10:04:03.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.802+0000 7f964db49700 1 -- 172.21.15.158:0/1400273949 wait complete. 2023-10-16T10:04:03.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.804+0000 7f964db49700 1 Processor -- start 2023-10-16T10:04:03.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.804+0000 7f964db49700 1 -- start start 2023-10-16T10:04:03.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f964db49700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630003bc0 0x7f96480b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f964db49700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f964814a770 0x7f9648148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f964db49700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480b0cd0 0x7f9648148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f964db49700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964815d5c0 con 0x7f964814a770 2023-10-16T10:04:03.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f964db49700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964814a520 con 0x7f9630003bc0 2023-10-16T10:04:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f964db49700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964814a240 con 0x7f96480b0cd0 2023-10-16T10:04:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f9646ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f964814a770 0x7f9648148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f9646ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f964814a770 0x7f9648148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59712/0 (socket says 172.21.15.158:59712) 2023-10-16T10:04:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f9646ffd700 1 -- 172.21.15.158:0/1802604873 learned_addr learned my addr 172.21.15.158:0/1802604873 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:03.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f96477fe700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630003bc0 0x7f96480b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.805+0000 7f9647fff700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480b0cd0 0x7f9648148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f9646ffd700 1 -- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480b0cd0 msgr2=0x7f9648148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f9646ffd700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480b0cd0 0x7f9648148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f9646ffd700 1 -- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630003bc0 msgr2=0x7f96480b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f9646ffd700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630003bc0 0x7f96480b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:03.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f9646ffd700 1 -- 172.21.15.158:0/1802604873 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964815d000 con 0x7f964814a770 2023-10-16T10:04:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f9647fff700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480b0cd0 0x7f9648148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f9646ffd700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f964814a770 0x7f9648148700 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f963c011aa0 tx=0x7f963c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f962ffff700 1 -- 172.21.15.158:0/1802604873 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c00f980 con 0x7f964814a770 2023-10-16T10:04:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f962ffff700 1 -- 172.21.15.158:0/1802604873 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963c00fae0 con 0x7f964814a770 2023-10-16T10:04:03.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964815d2e0 con 0x7f964814a770 2023-10-16T10:04:03.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f962ffff700 1 -- 172.21.15.158:0/1802604873 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c00fe90 con 0x7f964814a770 2023-10-16T10:04:03.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.806+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9648141300 con 0x7f964814a770 2023-10-16T10:04:03.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.807+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96480c6f60 con 0x7f964814a770 2023-10-16T10:04:03.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.807+0000 7f962ffff700 1 -- 172.21.15.158:0/1802604873 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f963c014440 con 0x7f964814a770 2023-10-16T10:04:03.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.808+0000 7f962ffff700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f963406b020 0x7f963406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:03.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.808+0000 7f962ffff700 1 -- 172.21.15.158:0/1802604873 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f963c08a1c0 con 0x7f964814a770 2023-10-16T10:04:03.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.808+0000 7f96477fe700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f963406b020 0x7f963406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:03.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.808+0000 7f96477fe700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f963406b020 0x7f963406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9638000c00 tx=0x7f963800a000 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:03.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:03.813+0000 7f962ffff700 1 -- 172.21.15.158:0/1802604873 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f963c054f80 con 0x7f964814a770 2023-10-16T10:04:04.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.047+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f96480c58d0 con 0x7f964814a770 2023-10-16T10:04:04.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.679+0000 7f962ffff700 1 -- 172.21.15.158:0/1802604873 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v73) v1 ==== 174+0+0 (secure 0 0 0) 0x7f963c0585a0 con 0x7f964814a770 2023-10-16T10:04:04.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f963406b020 msgr2=0x7f963406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:04.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f963406b020 0x7f963406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9638000c00 tx=0x7f963800a000 comp rx=0 tx=0).stop 2023-10-16T10:04:04.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f964814a770 msgr2=0x7f9648148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:04.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f964814a770 0x7f9648148700 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f963c011aa0 tx=0x7f963c011e60 comp rx=0 tx=0).stop 2023-10-16T10:04:04.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 shutdown_connections 2023-10-16T10:04:04.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f963406b020 0x7f963406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96480b0cd0 0x7f9648148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9630003bc0 0x7f96480b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 --2- 172.21.15.158:0/1802604873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f964814a770 0x7f9648148700 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 >> 172.21.15.158:0/1802604873 conn(0x7f9648070fe0 msgr2=0x7f96480c4860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:04.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 shutdown_connections 2023-10-16T10:04:04.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.684+0000 7f964db49700 1 -- 172.21.15.158:0/1802604873 wait complete. 2023-10-16T10:04:04.688 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 19 and data pool 20 2023-10-16T10:04:04.702 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:04:04.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.823+0000 7f5aa91de700 1 Processor -- start 2023-10-16T10:04:04.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.823+0000 7f5aa91de700 1 -- start start 2023-10-16T10:04:04.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa91de700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5aa415ed60 0x7f5aa40cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:04.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa91de700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa406a8d0 0x7f5aa406ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:04.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa91de700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa40b3f30 con 0x7f5aa40b3480 2023-10-16T10:04:04.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa91de700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa40b3910 con 0x7f5aa415ed60 2023-10-16T10:04:04.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa91de700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa406b2c0 con 0x7f5aa406a8d0 2023-10-16T10:04:04.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa2d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5aa415ed60 0x7f5aa40cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:04.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5aa40b3480 0x7f5aa40b43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59896/0 (socket says 172.21.15.158:59896) 2023-10-16T10:04:04.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa259c700 1 -- 172.21.15.158:0/1790640397 learned_addr learned my addr 172.21.15.158:0/1790640397 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:04.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa2d9d700 1 -- 172.21.15.158:0/1790640397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa406a8d0 msgr2=0x7f5aa406ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:04.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa2d9d700 1 --2- 172.21.15.158:0/1790640397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa406a8d0 0x7f5aa406ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa2d9d700 1 -- 172.21.15.158:0/1790640397 >> v1:172.21.15.158:6789/0 conn(0x7f5aa40b3480 legacy=0x7f5aa40b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:04.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa2d9d700 1 -- 172.21.15.158:0/1790640397 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa40b4ad0 con 0x7f5aa415ed60 2023-10-16T10:04:04.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.824+0000 7f5aa1d9b700 1 -- 172.21.15.158:0/1790640397 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998896597 0 0) 0x7f5aa40b3f30 con 0x7f5aa40b3480 2023-10-16T10:04:04.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa2d9d700 1 --2- 172.21.15.158:0/1790640397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa40cd890 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009820 tx=0x7f5a98009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e63004c51c6929e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:04.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa1d9b700 1 -- 172.21.15.158:0/1790640397 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98005e30 con 0x7f5aa415ed60 2023-10-16T10:04:04.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa1d9b700 1 -- 172.21.15.158:0/1790640397 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a98006030 con 0x7f5aa415ed60 2023-10-16T10:04:04.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa1d9b700 1 -- 172.21.15.158:0/1790640397 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980063e0 con 0x7f5aa415ed60 2023-10-16T10:04:04.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 -- 172.21.15.158:0/1790640397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 msgr2=0x7f5aa40cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:04.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1790640397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa40cd890 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009820 tx=0x7f5a98009b30 comp rx=0 tx=0).stop 2023-10-16T10:04:04.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 -- 172.21.15.158:0/1790640397 shutdown_connections 2023-10-16T10:04:04.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1790640397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa40cd890 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1790640397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa406a8d0 0x7f5aa406ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 -- 172.21.15.158:0/1790640397 >> 172.21.15.158:0/1790640397 conn(0x7f5aa40ccda0 msgr2=0x7f5aa40cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:04.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 -- 172.21.15.158:0/1790640397 shutdown_connections 2023-10-16T10:04:04.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.825+0000 7f5aa91de700 1 -- 172.21.15.158:0/1790640397 wait complete. 2023-10-16T10:04:04.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.827+0000 7f5aa91de700 1 Processor -- start 2023-10-16T10:04:04.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.827+0000 7f5aa91de700 1 -- start start 2023-10-16T10:04:04.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aa406a8d0 0x7f5aa4153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:04.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa4157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:04.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa4157f90 0x7f5aa415cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:04.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa4153970 con 0x7f5aa415ed60 2023-10-16T10:04:04.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa415c220 con 0x7f5aa406a8d0 2023-10-16T10:04:04.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa415c4c0 con 0x7f5aa4157f90 2023-10-16T10:04:04.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa4157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:04.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa4157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59726/0 (socket says 172.21.15.158:59726) 2023-10-16T10:04:04.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 -- 172.21.15.158:0/1239087930 learned_addr learned my addr 172.21.15.158:0/1239087930 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:04.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa359e700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa4157f90 0x7f5aa415cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:04.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa2d9d700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aa406a8d0 0x7f5aa4153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:04.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 -- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa4157f90 msgr2=0x7f5aa415cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:04.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa4157f90 0x7f5aa415cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 -- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aa406a8d0 msgr2=0x7f5aa4153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:04.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aa406a8d0 0x7f5aa4153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:04.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 -- 172.21.15.158:0/1239087930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98009430 con 0x7f5aa415ed60 2023-10-16T10:04:04.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa2d9d700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aa406a8d0 0x7f5aa4153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:04.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa359e700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa4157f90 0x7f5aa415cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:04.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa259c700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa4157a50 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f5a940011e0 tx=0x7f5a9400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:04.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5a8b7fe700 1 -- 172.21.15.158:0/1239087930 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a94004c80 con 0x7f5aa415ed60 2023-10-16T10:04:04.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5a8b7fe700 1 -- 172.21.15.158:0/1239087930 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a94004e00 con 0x7f5aa415ed60 2023-10-16T10:04:04.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa415c7a0 con 0x7f5aa415ed60 2023-10-16T10:04:04.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5a8b7fe700 1 -- 172.21.15.158:0/1239087930 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a94007310 con 0x7f5aa415ed60 2023-10-16T10:04:04.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.828+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa40b0a50 con 0x7f5aa415ed60 2023-10-16T10:04:04.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.829+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa415ccc0 con 0x7f5aa415ed60 2023-10-16T10:04:04.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.830+0000 7f5a8b7fe700 1 -- 172.21.15.158:0/1239087930 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a94007470 con 0x7f5aa415ed60 2023-10-16T10:04:04.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.830+0000 7f5a8b7fe700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a9006b020 0x7f5a9006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:04.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.830+0000 7f5a8b7fe700 1 -- 172.21.15.158:0/1239087930 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f5a94089880 con 0x7f5aa415ed60 2023-10-16T10:04:04.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.830+0000 7f5aa2d9d700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a9006b020 0x7f5a9006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:04.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.831+0000 7f5aa2d9d700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a9006b020 0x7f5a9006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a98013040 tx=0x7f5a98008070 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:04.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:04.835+0000 7f5a8b7fe700 1 -- 172.21.15.158:0/1239087930 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5aa415ccc0 con 0x7f5aa415ed60 2023-10-16T10:04:05.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.028+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa415ccc0 con 0x7f5aa415ed60 2023-10-16T10:04:05.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.029+0000 7f5a8b7fe700 1 -- 172.21.15.158:0/1239087930 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+17447 (secure 0 0 0) 0x7f5aa415ccc0 con 0x7f5aa415ed60 2023-10-16T10:04:05.030 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:05.031 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":152,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:04:04.049239+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:04:02.482303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T10:04:03.019780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:04:05.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a9006b020 msgr2=0x7f5a9006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a9006b020 0x7f5a9006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a98013040 tx=0x7f5a98008070 comp rx=0 tx=0).stop 2023-10-16T10:04:05.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 msgr2=0x7f5aa4157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa4157a50 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f5a940011e0 tx=0x7f5a9400cae0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 shutdown_connections 2023-10-16T10:04:05.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f5a9006b020 0x7f5a9006d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aa4157f90 0x7f5aa415cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aa406a8d0 0x7f5aa4153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 --2- 172.21.15.158:0/1239087930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aa415ed60 0x7f5aa4157a50 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 >> 172.21.15.158:0/1239087930 conn(0x7f5aa40ccda0 msgr2=0x7f5aa40acf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:05.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 shutdown_connections 2023-10-16T10:04:05.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.031+0000 7f5aa91de700 1 -- 172.21.15.158:0/1239087930 wait complete. 2023-10-16T10:04:05.045 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:04:05.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.164+0000 7f335ff7a700 1 Processor -- start 2023-10-16T10:04:05.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.164+0000 7f335ff7a700 1 -- start start 2023-10-16T10:04:05.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335ff7a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f33580ce640 0x7f33580cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335ff7a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33580cef60 con 0x7f33580cae30 2023-10-16T10:04:05.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335ff7a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335815d800 con 0x7f33580c65b0 2023-10-16T10:04:05.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335ff7a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f335815e480 con 0x7f33580ce640 2023-10-16T10:04:05.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335e517700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f33580cae30 0x7f33580cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59912/0 (socket says 172.21.15.158:59912) 2023-10-16T10:04:05.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335e517700 1 -- 172.21.15.158:0/2663665402 learned_addr learned my addr 172.21.15.158:0/2663665402 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:05.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335dd16700 1 --2- 172.21.15.158:0/2663665402 >> v2:172.21.15.172:3300/0 conn(0x7f33580ce640 0x7f33580cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534362833 0 0) 0x7f33580cef60 con 0x7f33580cae30 2023-10-16T10:04:05.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33440035b0 con 0x7f33580cae30 2023-10-16T10:04:05.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239931722 0 0) 0x7f335815d800 con 0x7f33580c65b0 2023-10-16T10:04:05.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33580cef60 con 0x7f33580c65b0 2023-10-16T10:04:05.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335dd16700 1 -- 172.21.15.158:0/2663665402 >> v1:172.21.15.172:6789/0 conn(0x7f33580c65b0 legacy=0x7f335806f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335dd16700 1 -- 172.21.15.158:0/2663665402 >> v1:172.21.15.158:6789/0 conn(0x7f33580cae30 legacy=0x7f33580cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2446488581 0 0) 0x7f33440035b0 con 0x7f33580cae30 2023-10-16T10:04:05.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.165+0000 7f335dd16700 1 -- 172.21.15.158:0/2663665402 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335815dae0 con 0x7f33580ce640 2023-10-16T10:04:05.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335dd16700 1 --2- 172.21.15.158:0/2663665402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580ce640 0x7f33580cea20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3354009840 tx=0x7f3354009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1815a9f48c3b9888 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:05.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33500019f0 con 0x7f33580ce640 2023-10-16T10:04:05.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3354004030 con 0x7f33580ce640 2023-10-16T10:04:05.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335cd14700 1 -- 172.21.15.158:0/2663665402 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540043a0 con 0x7f33580ce640 2023-10-16T10:04:05.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335ff7a700 1 -- 172.21.15.158:0/2663665402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580ce640 msgr2=0x7f33580cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2663665402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580ce640 0x7f33580cea20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3354009840 tx=0x7f3354009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:05.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335ff7a700 1 -- 172.21.15.158:0/2663665402 shutdown_connections 2023-10-16T10:04:05.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2663665402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580ce640 0x7f33580cea20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335ff7a700 1 -- 172.21.15.158:0/2663665402 >> 172.21.15.158:0/2663665402 conn(0x7f33580cc4b0 msgr2=0x7f33580cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:05.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335ff7a700 1 -- 172.21.15.158:0/2663665402 shutdown_connections 2023-10-16T10:04:05.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.166+0000 7f335ff7a700 1 -- 172.21.15.158:0/2663665402 wait complete. 2023-10-16T10:04:05.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 Processor -- start 2023-10-16T10:04:05.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 -- start start 2023-10-16T10:04:05.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580c65b0 0x7f3358139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33580cae30 0x7f33580b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33580ce640 0x7f3358136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33580b5700 con 0x7f33580ce640 2023-10-16T10:04:05.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3358137830 con 0x7f33580c65b0 2023-10-16T10:04:05.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335ff7a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3358137df0 con 0x7f33580cae30 2023-10-16T10:04:05.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335e517700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33580ce640 0x7f3358136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335e517700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33580ce640 0x7f3358136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59742/0 (socket says 172.21.15.158:59742) 2023-10-16T10:04:05.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.168+0000 7f335e517700 1 -- 172.21.15.158:0/2166242882 learned_addr learned my addr 172.21.15.158:0/2166242882 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:05.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335dd16700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580c65b0 0x7f3358139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335d515700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33580cae30 0x7f33580b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335e517700 1 -- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33580cae30 msgr2=0x7f33580b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335e517700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33580cae30 0x7f33580b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335e517700 1 -- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580c65b0 msgr2=0x7f3358139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335e517700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580c65b0 0x7f3358139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335e517700 1 -- 172.21.15.158:0/2166242882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33540094a0 con 0x7f33580ce640 2023-10-16T10:04:05.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335d515700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33580cae30 0x7f33580b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:05.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335e517700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33580ce640 0x7f3358136f80 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f335000cc70 tx=0x7f335000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:05.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f334e7fc700 1 -- 172.21.15.158:0/2166242882 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335000ade0 con 0x7f33580ce640 2023-10-16T10:04:05.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f334e7fc700 1 -- 172.21.15.158:0/2166242882 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335000ce70 con 0x7f33580ce640 2023-10-16T10:04:05.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33580b2fa0 con 0x7f33580ce640 2023-10-16T10:04:05.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f334e7fc700 1 -- 172.21.15.158:0/2166242882 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33500072f0 con 0x7f33580ce640 2023-10-16T10:04:05.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.169+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3358137a80 con 0x7f33580ce640 2023-10-16T10:04:05.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.170+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335815d800 con 0x7f33580ce640 2023-10-16T10:04:05.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.170+0000 7f334e7fc700 1 -- 172.21.15.158:0/2166242882 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3350007450 con 0x7f33580ce640 2023-10-16T10:04:05.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.171+0000 7f334e7fc700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f333c06b020 0x7f333c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.171+0000 7f334e7fc700 1 -- 172.21.15.158:0/2166242882 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f33500899c0 con 0x7f33580ce640 2023-10-16T10:04:05.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.171+0000 7f335dd16700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f333c06b020 0x7f333c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.171+0000 7f335dd16700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f333c06b020 0x7f333c06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f3354008030 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:05.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.175+0000 7f334e7fc700 1 -- 172.21.15.158:0/2166242882 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33500546e0 con 0x7f33580ce640 2023-10-16T10:04:05.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.368+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33580681b0 con 0x7f33580ce640 2023-10-16T10:04:05.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.369+0000 7f334e7fc700 1 -- 172.21.15.158:0/2166242882 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+17447 (secure 0 0 0) 0x7f3350018090 con 0x7f33580ce640 2023-10-16T10:04:05.371 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:05.371 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":152,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:04:04.049239+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:04:02.482303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T10:04:03.019780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:04:05.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f333c06b020 msgr2=0x7f333c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f333c06b020 0x7f333c06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f3354008030 comp rx=0 tx=0).stop 2023-10-16T10:04:05.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33580ce640 msgr2=0x7f3358136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33580ce640 0x7f3358136f80 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f335000cc70 tx=0x7f335000a340 comp rx=0 tx=0).stop 2023-10-16T10:04:05.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 shutdown_connections 2023-10-16T10:04:05.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33580ce640 0x7f3358136f80 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33580c65b0 0x7f3358139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f333c06b020 0x7f333c06d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 --2- 172.21.15.158:0/2166242882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33580cae30 0x7f33580b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 >> 172.21.15.158:0/2166242882 conn(0x7f33580cc4b0 msgr2=0x7f33580c59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:05.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 shutdown_connections 2023-10-16T10:04:05.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.371+0000 7f335ff7a700 1 -- 172.21.15.158:0/2166242882 wait complete. 2023-10-16T10:04:05.385 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T10:04:05.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.508+0000 7f4f1cb92700 1 Processor -- start 2023-10-16T10:04:05.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.508+0000 7f4f1cb92700 1 -- start start 2023-10-16T10:04:05.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1cb92700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4f18141920 0x7f4f1806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1cb92700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f180cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1cb92700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4f180cdc40 0x7f4f1815f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1cb92700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f180ce0d0 con 0x7f4f18141920 2023-10-16T10:04:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1cb92700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1806c210 con 0x7f4f180cdc40 2023-10-16T10:04:05.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1cb92700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18141d00 con 0x7f4f18144840 2023-10-16T10:04:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4f18141920 0x7f4f1806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4f18141920 0x7f4f1806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59746/0 (socket says 172.21.15.158:59746) 2023-10-16T10:04:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 -- 172.21.15.158:0/2101564460 learned_addr learned my addr 172.21.15.158:0/2101564460 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:05.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f15d9b700 1 --2- 172.21.15.158:0/2101564460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f180cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f16d9d700 1 --2- 172.21.15.158:0/2101564460 >> v2:172.21.15.172:3300/0 conn(0x7f4f180cdc40 0x7f4f1815f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 -- 172.21.15.158:0/2101564460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 msgr2=0x7f4f180cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 --2- 172.21.15.158:0/2101564460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f180cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 -- 172.21.15.158:0/2101564460 >> v2:172.21.15.172:3300/0 conn(0x7f4f180cdc40 msgr2=0x7f4f1815f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 --2- 172.21.15.158:0/2101564460 >> v2:172.21.15.172:3300/0 conn(0x7f4f180cdc40 0x7f4f1815f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 -- 172.21.15.158:0/2101564460 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1806b970 con 0x7f4f18141920 2023-10-16T10:04:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f15d9b700 1 --2- 172.21.15.158:0/2101564460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f180cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f16d9d700 1 --2- 172.21.15.158:0/2101564460 >> v2:172.21.15.172:3300/0 conn(0x7f4f180cdc40 0x7f4f1815f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1659c700 1 --2- 172.21.15.158:0/2101564460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f18141920 0x7f4f1806b2e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f4f180ca110 tx=0x7f4f0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca67a4fd8b9ad89 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:05.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1559a700 1 -- 172.21.15.158:0/2101564460 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c00b870 con 0x7f4f18141920 2023-10-16T10:04:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1559a700 1 -- 172.21.15.158:0/2101564460 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c00b9d0 con 0x7f4f18141920 2023-10-16T10:04:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.509+0000 7f4f1559a700 1 -- 172.21.15.158:0/2101564460 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c00bd60 con 0x7f4f18141920 2023-10-16T10:04:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 -- 172.21.15.158:0/2101564460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f18141920 msgr2=0x7f4f1806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/2101564460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f18141920 0x7f4f1806b2e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f4f180ca110 tx=0x7f4f0c009bd0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 -- 172.21.15.158:0/2101564460 shutdown_connections 2023-10-16T10:04:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/2101564460 >> v2:172.21.15.172:3300/0 conn(0x7f4f180cdc40 0x7f4f1815f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/2101564460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f18141920 0x7f4f1806b2e0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/2101564460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f180cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 -- 172.21.15.158:0/2101564460 >> 172.21.15.158:0/2101564460 conn(0x7f4f180cc5b0 msgr2=0x7f4f180cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:05.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 -- 172.21.15.158:0/2101564460 shutdown_connections 2023-10-16T10:04:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.510+0000 7f4f1cb92700 1 -- 172.21.15.158:0/2101564460 wait complete. 2023-10-16T10:04:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.511+0000 7f4f1cb92700 1 Processor -- start 2023-10-16T10:04:05.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.511+0000 7f4f1cb92700 1 -- start start 2023-10-16T10:04:05.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1cb92700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f180cdc40 0x7f4f181409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1cb92700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f18141920 0x7f4f18140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1cb92700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f18143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1cb92700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f181446f0 con 0x7f4f180cdc40 2023-10-16T10:04:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1cb92700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18142e30 con 0x7f4f18141920 2023-10-16T10:04:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1cb92700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18142070 con 0x7f4f18144840 2023-10-16T10:04:05.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f180cdc40 0x7f4f181409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f180cdc40 0x7f4f181409a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59752/0 (socket says 172.21.15.158:59752) 2023-10-16T10:04:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 -- 172.21.15.158:0/3532857688 learned_addr learned my addr 172.21.15.158:0/3532857688 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 -- 172.21.15.158:0/3532857688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 msgr2=0x7f4f18143ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f18143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 -- 172.21.15.158:0/3532857688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f18141920 msgr2=0x7f4f18140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f18141920 0x7f4f18140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.512+0000 7f4f1659c700 1 -- 172.21.15.158:0/3532857688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c009550 con 0x7f4f180cdc40 2023-10-16T10:04:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.513+0000 7f4f1659c700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f180cdc40 0x7f4f181409a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c000c00 tx=0x7f4f0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.513+0000 7f4f06ffd700 1 -- 172.21.15.158:0/3532857688 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c004540 con 0x7f4f180cdc40 2023-10-16T10:04:05.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.513+0000 7f4f06ffd700 1 -- 172.21.15.158:0/3532857688 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c0036a0 con 0x7f4f180cdc40 2023-10-16T10:04:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.513+0000 7f4f06ffd700 1 -- 172.21.15.158:0/3532857688 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c003a50 con 0x7f4f180cdc40 2023-10-16T10:04:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.513+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f18142350 con 0x7f4f180cdc40 2023-10-16T10:04:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.513+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f18142910 con 0x7f4f180cdc40 2023-10-16T10:04:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.514+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f180681b0 con 0x7f4f180cdc40 2023-10-16T10:04:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.514+0000 7f4f06ffd700 1 -- 172.21.15.158:0/3532857688 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f0c021070 con 0x7f4f180cdc40 2023-10-16T10:04:05.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.514+0000 7f4f06ffd700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f4ef806b020 0x7f4ef806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.515+0000 7f4f06ffd700 1 -- 172.21.15.158:0/3532857688 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f4f0c08b130 con 0x7f4f180cdc40 2023-10-16T10:04:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.515+0000 7f4f15d9b700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f4ef806b020 0x7f4ef806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.515+0000 7f4f15d9b700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f4ef806b020 0x7f4ef806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f08006fd0 tx=0x7f4f08008040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:05.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.519+0000 7f4f06ffd700 1 -- 172.21.15.158:0/3532857688 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f0c055e50 con 0x7f4f180cdc40 2023-10-16T10:04:05.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:05.759+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4f1806ec90 con 0x7f4f180cdc40 2023-10-16T10:04:06.648 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T10:04:06.647+0000 7f1041dae700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T10:04:06.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.684+0000 7f4f06ffd700 1 -- 172.21.15.158:0/3532857688 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4f0c059470 con 0x7f4f180cdc40 2023-10-16T10:04:06.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f4ef806b020 msgr2=0x7f4ef806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:06.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f4ef806b020 0x7f4ef806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f08006fd0 tx=0x7f4f08008040 comp rx=0 tx=0).stop 2023-10-16T10:04:06.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f180cdc40 msgr2=0x7f4f181409a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:06.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f180cdc40 0x7f4f181409a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c000c00 tx=0x7f4f0c004750 comp rx=0 tx=0).stop 2023-10-16T10:04:06.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 shutdown_connections 2023-10-16T10:04:06.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f4ef806b020 0x7f4ef806d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f18144840 0x7f4f18143ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f180cdc40 0x7f4f181409a0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 --2- 172.21.15.158:0/3532857688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f18141920 0x7f4f18140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 >> 172.21.15.158:0/3532857688 conn(0x7f4f180cc5b0 msgr2=0x7f4f1806e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:06.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 shutdown_connections 2023-10-16T10:04:06.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.687+0000 7f4f1cb92700 1 -- 172.21.15.158:0/3532857688 wait complete. 2023-10-16T10:04:06.703 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T10:04:06.703 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T10:04:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.834+0000 7f232f874700 1 Processor -- start 2023-10-16T10:04:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.834+0000 7f232f874700 1 -- start start 2023-10-16T10:04:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232f874700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2328141920 0x7f232806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232f874700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23280cde10 0x7f232806da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232f874700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232806c210 con 0x7f2328144840 2023-10-16T10:04:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232f874700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2328141d00 con 0x7f2328141920 2023-10-16T10:04:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232f874700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232806b970 con 0x7f23280cde10 2023-10-16T10:04:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232d610700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2328141920 0x7f232806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232ce0f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2328144840 0x7f23280cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59938/0 (socket says 172.21.15.158:59938) 2023-10-16T10:04:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232ce0f700 1 -- 172.21.15.158:0/2025084940 learned_addr learned my addr 172.21.15.158:0/2025084940 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232de11700 1 --2- 172.21.15.158:0/2025084940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23280cde10 0x7f232806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232d610700 1 -- 172.21.15.158:0/2025084940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23280cde10 msgr2=0x7f232806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232d610700 1 --2- 172.21.15.158:0/2025084940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23280cde10 0x7f232806da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232d610700 1 -- 172.21.15.158:0/2025084940 >> v1:172.21.15.158:6789/0 conn(0x7f2328144840 legacy=0x7f23280cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:06.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232d610700 1 -- 172.21.15.158:0/2025084940 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232806bc50 con 0x7f2328141920 2023-10-16T10:04:06.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232de11700 1 --2- 172.21.15.158:0/2025084940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23280cde10 0x7f232806da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:06.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.835+0000 7f232d610700 1 --2- 172.21.15.158:0/2025084940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328141920 0x7f232806b2e0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f2324009840 tx=0x7f2324009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b58aa3346baf1af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:06.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f231ffff700 1 -- 172.21.15.158:0/2025084940 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324011830 con 0x7f2328141920 2023-10-16T10:04:06.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f231ffff700 1 -- 172.21.15.158:0/2025084940 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2324011990 con 0x7f2328141920 2023-10-16T10:04:06.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f231ffff700 1 -- 172.21.15.158:0/2025084940 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324011d20 con 0x7f2328141920 2023-10-16T10:04:06.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 -- 172.21.15.158:0/2025084940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328141920 msgr2=0x7f232806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:06.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 --2- 172.21.15.158:0/2025084940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328141920 0x7f232806b2e0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f2324009840 tx=0x7f2324009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:06.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 -- 172.21.15.158:0/2025084940 shutdown_connections 2023-10-16T10:04:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 --2- 172.21.15.158:0/2025084940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328141920 0x7f232806b2e0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 --2- 172.21.15.158:0/2025084940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23280cde10 0x7f232806da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 -- 172.21.15.158:0/2025084940 >> 172.21.15.158:0/2025084940 conn(0x7f23280cc5b0 msgr2=0x7f23280cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 -- 172.21.15.158:0/2025084940 shutdown_connections 2023-10-16T10:04:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.836+0000 7f232f874700 1 -- 172.21.15.158:0/2025084940 wait complete. 2023-10-16T10:04:06.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.837+0000 7f232f874700 1 Processor -- start 2023-10-16T10:04:06.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.837+0000 7f232f874700 1 -- start start 2023-10-16T10:04:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232f874700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23280cde10 0x7f2328142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232f874700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2328141920 0x7f23281e5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232f874700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328144840 0x7f23280b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232f874700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23281e6100 con 0x7f2328144840 2023-10-16T10:04:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232f874700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328140eb0 con 0x7f23280cde10 2023-10-16T10:04:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232f874700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23280c51a0 con 0x7f2328141920 2023-10-16T10:04:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232de11700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328144840 0x7f23280b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232de11700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328144840 0x7f23280b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59770/0 (socket says 172.21.15.158:59770) 2023-10-16T10:04:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.838+0000 7f232de11700 1 -- 172.21.15.158:0/386703785 learned_addr learned my addr 172.21.15.158:0/386703785 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232ce0f700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2328141920 0x7f23281e5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:06.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232d610700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23280cde10 0x7f2328142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232de11700 1 -- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2328141920 msgr2=0x7f23281e5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232de11700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2328141920 0x7f23281e5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232de11700 1 -- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23280cde10 msgr2=0x7f2328142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232de11700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23280cde10 0x7f2328142780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:06.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232de11700 1 -- 172.21.15.158:0/386703785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240094a0 con 0x7f2328144840 2023-10-16T10:04:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232de11700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328144840 0x7f23280b47f0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f232000cea0 tx=0x7f232000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f231dffb700 1 -- 172.21.15.158:0/386703785 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232000ae10 con 0x7f2328144840 2023-10-16T10:04:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f231dffb700 1 -- 172.21.15.158:0/386703785 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320003a20 con 0x7f2328144840 2023-10-16T10:04:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23281e5e20 con 0x7f2328144840 2023-10-16T10:04:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f231dffb700 1 -- 172.21.15.158:0/386703785 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320003dd0 con 0x7f2328144840 2023-10-16T10:04:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.839+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23280c4e30 con 0x7f2328144840 2023-10-16T10:04:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.840+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23280c49b0 con 0x7f2328144840 2023-10-16T10:04:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.840+0000 7f231dffb700 1 -- 172.21.15.158:0/386703785 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2320003f30 con 0x7f2328144840 2023-10-16T10:04:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.841+0000 7f231dffb700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f230c06b020 0x7f230c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.841+0000 7f231dffb700 1 -- 172.21.15.158:0/386703785 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f2320089e60 con 0x7f2328144840 2023-10-16T10:04:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.841+0000 7f232d610700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f230c06b020 0x7f230c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.841+0000 7f232d610700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f230c06b020 0x7f230c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23240090b0 tx=0x7f2324008030 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:06.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:06.845+0000 7f231dffb700 1 -- 172.21.15.158:0/386703785 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2320054af0 con 0x7f2328144840 2023-10-16T10:04:07.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.084+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f23280681b0 con 0x7f2328144840 2023-10-16T10:04:07.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.685+0000 7f231dffb700 1 -- 172.21.15.158:0/386703785 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2320058110 con 0x7f2328144840 2023-10-16T10:04:07.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f230c06b020 msgr2=0x7f230c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:07.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f230c06b020 0x7f230c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23240090b0 tx=0x7f2324008030 comp rx=0 tx=0).stop 2023-10-16T10:04:07.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328144840 msgr2=0x7f23280b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:07.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328144840 0x7f23280b47f0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f232000cea0 tx=0x7f232000cf80 comp rx=0 tx=0).stop 2023-10-16T10:04:07.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 shutdown_connections 2023-10-16T10:04:07.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2328144840 0x7f23280b47f0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:07.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23280cde10 0x7f2328142780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:07.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f230c06b020 0x7f230c06d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:07.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 --2- 172.21.15.158:0/386703785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2328141920 0x7f23281e5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:07.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 >> 172.21.15.158:0/386703785 conn(0x7f23280cc5b0 msgr2=0x7f23280ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:07.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.688+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 shutdown_connections 2023-10-16T10:04:07.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.689+0000 7f232f874700 1 -- 172.21.15.158:0/386703785 wait complete. 2023-10-16T10:04:07.704 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T10:04:07.705 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:04:07.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.830+0000 7f90c951f700 1 Processor -- start 2023-10-16T10:04:07.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.830+0000 7f90c951f700 1 -- start start 2023-10-16T10:04:07.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c951f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90c40b7810 0x7f90c41683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:07.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c951f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c40d3bd0 con 0x7f90c40d1210 2023-10-16T10:04:07.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c951f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c40d2fc0 con 0x7f90c415d7c0 2023-10-16T10:04:07.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c951f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c40d4060 con 0x7f90c40b7810 2023-10-16T10:04:07.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c37fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90c40b7810 0x7f90c41683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:07.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c27fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f90c40d1210 0x7f90c40ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59950/0 (socket says 172.21.15.158:59950) 2023-10-16T10:04:07.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c27fc700 1 -- 172.21.15.158:0/3382121900 learned_addr learned my addr 172.21.15.158:0/3382121900 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:07.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c37fe700 1 -- 172.21.15.158:0/3382121900 >> v1:172.21.15.172:6789/0 conn(0x7f90c415d7c0 legacy=0x7f90c40d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:07.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c37fe700 1 -- 172.21.15.158:0/3382121900 >> v1:172.21.15.158:6789/0 conn(0x7f90c40d1210 legacy=0x7f90c40ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:07.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c37fe700 1 -- 172.21.15.158:0/3382121900 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c40d32a0 con 0x7f90c40b7810 2023-10-16T10:04:07.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c37fe700 1 --2- 172.21.15.158:0/3382121900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40b7810 0x7f90c41683f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d985b6b0cd26d1c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:07.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.831+0000 7f90c1ffb700 1 -- 172.21.15.158:0/3382121900 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc00e070 con 0x7f90c40b7810 2023-10-16T10:04:07.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c1ffb700 1 -- 172.21.15.158:0/3382121900 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc00bad0 con 0x7f90c40b7810 2023-10-16T10:04:07.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c1ffb700 1 -- 172.21.15.158:0/3382121900 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc00be60 con 0x7f90c40b7810 2023-10-16T10:04:07.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c951f700 1 -- 172.21.15.158:0/3382121900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40b7810 msgr2=0x7f90c41683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:07.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c951f700 1 --2- 172.21.15.158:0/3382121900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40b7810 0x7f90c41683f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc00b330 comp rx=0 tx=0).stop 2023-10-16T10:04:07.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c951f700 1 -- 172.21.15.158:0/3382121900 shutdown_connections 2023-10-16T10:04:07.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c951f700 1 --2- 172.21.15.158:0/3382121900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40b7810 0x7f90c41683f0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:07.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c951f700 1 -- 172.21.15.158:0/3382121900 >> 172.21.15.158:0/3382121900 conn(0x7f90c40669f0 msgr2=0x7f90c4066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:07.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c951f700 1 -- 172.21.15.158:0/3382121900 shutdown_connections 2023-10-16T10:04:07.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.832+0000 7f90c951f700 1 -- 172.21.15.158:0/3382121900 wait complete. 2023-10-16T10:04:07.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.833+0000 7f90c951f700 1 Processor -- start 2023-10-16T10:04:07.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.833+0000 7f90c951f700 1 -- start start 2023-10-16T10:04:07.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c951f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90c40b7810 0x7f90c40c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:07.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c951f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40d1210 0x7f90c40b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:07.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c951f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90c415d7c0 0x7f90c4149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:07.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c951f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c40c8de0 con 0x7f90c40d1210 2023-10-16T10:04:07.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c951f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c40c9640 con 0x7f90c40b7810 2023-10-16T10:04:07.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c951f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c41f71c0 con 0x7f90c415d7c0 2023-10-16T10:04:07.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40d1210 0x7f90c40b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:07.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40d1210 0x7f90c40b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59782/0 (socket says 172.21.15.158:59782) 2023-10-16T10:04:07.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c27fc700 1 -- 172.21.15.158:0/3996445597 learned_addr learned my addr 172.21.15.158:0/3996445597 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:07.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.834+0000 7f90c37fe700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90c415d7c0 0x7f90c4149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:07.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c2ffd700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90c40b7810 0x7f90c40c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:07.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c27fc700 1 -- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90c415d7c0 msgr2=0x7f90c4149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:07.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c27fc700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90c415d7c0 0x7f90c4149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:07.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c27fc700 1 -- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90c40b7810 msgr2=0x7f90c40c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:07.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c27fc700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90c40b7810 0x7f90c40c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:07.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c27fc700 1 -- 172.21.15.158:0/3996445597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc009a70 con 0x7f90c40d1210 2023-10-16T10:04:07.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c27fc700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40d1210 0x7f90c40b4360 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c7b0 tx=0x7f90b400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:07.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90ab7fe700 1 -- 172.21.15.158:0/3996445597 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4004c80 con 0x7f90c40d1210 2023-10-16T10:04:07.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90ab7fe700 1 -- 172.21.15.158:0/3996445597 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b4004de0 con 0x7f90c40d1210 2023-10-16T10:04:07.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c41f6ee0 con 0x7f90c40d1210 2023-10-16T10:04:07.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90ab7fe700 1 -- 172.21.15.158:0/3996445597 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4007310 con 0x7f90c40d1210 2023-10-16T10:04:07.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.835+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c40c92d0 con 0x7f90c40d1210 2023-10-16T10:04:07.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.836+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c40c97f0 con 0x7f90c40d1210 2023-10-16T10:04:07.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.836+0000 7f90ab7fe700 1 -- 172.21.15.158:0/3996445597 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f90b4007470 con 0x7f90c40d1210 2023-10-16T10:04:07.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.837+0000 7f90ab7fe700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f90b006aff0 0x7f90b006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:07.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.837+0000 7f90ab7fe700 1 -- 172.21.15.158:0/3996445597 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f90b4088fc0 con 0x7f90c40d1210 2023-10-16T10:04:07.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.837+0000 7f90c2ffd700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f90b006aff0 0x7f90b006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:07.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.837+0000 7f90c2ffd700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f90b006aff0 0x7f90b006d4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90b8004f30 tx=0x7f90b8009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:07.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:07.841+0000 7f90ab7fe700 1 -- 172.21.15.158:0/3996445597 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f90b40bbfe0 con 0x7f90c40d1210 2023-10-16T10:04:08.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.076+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90c40ca070 con 0x7f90c40d1210 2023-10-16T10:04:08.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.076+0000 7f90ab7fe700 1 -- 172.21.15.158:0/3996445597 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f90b4018020 con 0x7f90c40d1210 2023-10-16T10:04:08.078 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:08.078 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8422,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1604199546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1604199546},{"type":"v1","addr":"172.21.15.172:6834","nonce":1604199546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/95492242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":95492242},{"type":"v1","addr":"172.21.15.158:6836","nonce":95492242}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9393,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112424210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112424210},{"type":"v1","addr":"172.21.15.158:6837","nonce":112424210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:04:04.049571+0000","modified":"2023-10-16T10:04:07.086241+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37352":{"gid":37352,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/253283745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":253283745},{"type":"v1","addr":"172.21.15.172:6835","nonce":253283745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T10:04:08.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f90b006aff0 msgr2=0x7f90b006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f90b006aff0 0x7f90b006d4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90b8004f30 tx=0x7f90b8009040 comp rx=0 tx=0).stop 2023-10-16T10:04:08.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40d1210 msgr2=0x7f90c40b4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40d1210 0x7f90c40b4360 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c7b0 tx=0x7f90b400cb70 comp rx=0 tx=0).stop 2023-10-16T10:04:08.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 shutdown_connections 2023-10-16T10:04:08.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f90b006aff0 0x7f90b006d4b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90c415d7c0 0x7f90c4149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90c40b7810 0x7f90c40c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 --2- 172.21.15.158:0/3996445597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c40d1210 0x7f90c40b4360 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 >> 172.21.15.158:0/3996445597 conn(0x7f90c40669f0 msgr2=0x7f90c40d2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:08.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 shutdown_connections 2023-10-16T10:04:08.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.079+0000 7f90c951f700 1 -- 172.21.15.158:0/3996445597 wait complete. 2023-10-16T10:04:08.083 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 77 2023-10-16T10:04:08.092 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T10:04:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.212+0000 7f220017b700 1 Processor -- start 2023-10-16T10:04:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.212+0000 7f220017b700 1 -- start start 2023-10-16T10:04:08.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f220017b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21f815dfb0 0x7f21f806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f220017b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f806e770 con 0x7f21f80cc5d0 2023-10-16T10:04:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f220017b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f80cbe00 con 0x7f21f806f340 2023-10-16T10:04:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f220017b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f80cf270 con 0x7f21f815dfb0 2023-10-16T10:04:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f21fdf17700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f21f80cc5d0 0x7f21f80cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:59964/0 (socket says 172.21.15.158:59964) 2023-10-16T10:04:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f21fdf17700 1 -- 172.21.15.158:0/2663392847 learned_addr learned my addr 172.21.15.158:0/2663392847 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f21fd716700 1 --2- 172.21.15.158:0/2663392847 >> v2:172.21.15.172:3300/0 conn(0x7f21f815dfb0 0x7f21f806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f21fcf15700 1 -- 172.21.15.158:0/2663392847 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933838543 0 0) 0x7f21f0001110 con 0x7f21f80cc5d0 2023-10-16T10:04:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.213+0000 7f21fcf15700 1 -- 172.21.15.158:0/2663392847 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e00035b0 con 0x7f21f80cc5d0 2023-10-16T10:04:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fcf15700 1 -- 172.21.15.158:0/2663392847 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2451370354 0 0) 0x7f21e00035b0 con 0x7f21f80cc5d0 2023-10-16T10:04:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fd716700 1 -- 172.21.15.158:0/2663392847 >> v1:172.21.15.172:6789/0 conn(0x7f21f806f340 legacy=0x7f21f8071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fd716700 1 -- 172.21.15.158:0/2663392847 >> v1:172.21.15.158:6789/0 conn(0x7f21f80cc5d0 legacy=0x7f21f80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fd716700 1 -- 172.21.15.158:0/2663392847 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f815e4e0 con 0x7f21f815dfb0 2023-10-16T10:04:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fd716700 1 --2- 172.21.15.158:0/2663392847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f815dfb0 0x7f21f806e230 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009840 tx=0x7f21ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d074853161da3b1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fcf15700 1 -- 172.21.15.158:0/2663392847 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0001800 con 0x7f21f815dfb0 2023-10-16T10:04:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fcf15700 1 -- 172.21.15.158:0/2663392847 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec004030 con 0x7f21f815dfb0 2023-10-16T10:04:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.214+0000 7f21fcf15700 1 -- 172.21.15.158:0/2663392847 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec0043c0 con 0x7f21f815dfb0 2023-10-16T10:04:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.215+0000 7f220017b700 1 -- 172.21.15.158:0/2663392847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f815dfb0 msgr2=0x7f21f806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.215+0000 7f220017b700 1 --2- 172.21.15.158:0/2663392847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f815dfb0 0x7f21f806e230 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009840 tx=0x7f21ec009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.215+0000 7f220017b700 1 -- 172.21.15.158:0/2663392847 shutdown_connections 2023-10-16T10:04:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.215+0000 7f220017b700 1 --2- 172.21.15.158:0/2663392847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f815dfb0 0x7f21f806e230 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.215+0000 7f220017b700 1 -- 172.21.15.158:0/2663392847 >> 172.21.15.158:0/2663392847 conn(0x7f21f80d02b0 msgr2=0x7f21f80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.215+0000 7f220017b700 1 -- 172.21.15.158:0/2663392847 shutdown_connections 2023-10-16T10:04:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.215+0000 7f220017b700 1 -- 172.21.15.158:0/2663392847 wait complete. 2023-10-16T10:04:08.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.216+0000 7f220017b700 1 Processor -- start 2023-10-16T10:04:08.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.216+0000 7f220017b700 1 -- start start 2023-10-16T10:04:08.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21f806f340 0x7f21f814cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f80cc5d0 0x7f21f80a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21f815dfb0 0x7f21f80b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f80aa410 con 0x7f21f806f340 2023-10-16T10:04:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f80a9790 con 0x7f21f80cc5d0 2023-10-16T10:04:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f80a9a70 con 0x7f21f815dfb0 2023-10-16T10:04:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21f806f340 0x7f21f814cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21f806f340 0x7f21f814cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59794/0 (socket says 172.21.15.158:59794) 2023-10-16T10:04:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 -- 172.21.15.158:0/3722294856 learned_addr learned my addr 172.21.15.158:0/3722294856 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fd716700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f80cc5d0 0x7f21f80a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fe718700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21f815dfb0 0x7f21f80b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 -- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21f815dfb0 msgr2=0x7f21f80b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21f815dfb0 0x7f21f80b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 -- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f80cc5d0 msgr2=0x7f21f80a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f80cc5d0 0x7f21f80a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 -- 172.21.15.158:0/3722294856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec0094a0 con 0x7f21f806f340 2023-10-16T10:04:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21fdf17700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21f806f340 0x7f21f814cc20 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f21f000c750 tx=0x7f21f000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21f67fc700 1 -- 172.21.15.158:0/3722294856 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0004c80 con 0x7f21f806f340 2023-10-16T10:04:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21f67fc700 1 -- 172.21.15.158:0/3722294856 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f0004de0 con 0x7f21f806f340 2023-10-16T10:04:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f80a9d50 con 0x7f21f806f340 2023-10-16T10:04:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f21f67fc700 1 -- 172.21.15.158:0/3722294856 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0007310 con 0x7f21f806f340 2023-10-16T10:04:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.217+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f80a9380 con 0x7f21f806f340 2023-10-16T10:04:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.218+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f80aa550 con 0x7f21f806f340 2023-10-16T10:04:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.219+0000 7f21f67fc700 1 -- 172.21.15.158:0/3722294856 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f21f0007470 con 0x7f21f806f340 2023-10-16T10:04:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.219+0000 7f21f67fc700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f21e406b020 0x7f21e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.219+0000 7f21f67fc700 1 -- 172.21.15.158:0/3722294856 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f21f0089590 con 0x7f21f806f340 2023-10-16T10:04:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.219+0000 7f21fd716700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f21e406b020 0x7f21e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.220+0000 7f21fd716700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f21e406b020 0x7f21e406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009840 tx=0x7f21ec008000 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.224+0000 7f21f67fc700 1 -- 172.21.15.158:0/3722294856 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21f0054210 con 0x7f21f806f340 2023-10-16T10:04:08.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.415+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21f806e770 con 0x7f21f806f340 2023-10-16T10:04:08.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.416+0000 7f21f67fc700 1 -- 172.21.15.158:0/3722294856 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+17434 (secure 0 0 0) 0x7f21f806e770 con 0x7f21f806f340 2023-10-16T10:04:08.418 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:08.418 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":153,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T10:04:05.659603+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:04:02.482303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T10:04:03.019780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T10:04:08.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f21e406b020 msgr2=0x7f21e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f21e406b020 0x7f21e406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009840 tx=0x7f21ec008000 comp rx=0 tx=0).stop 2023-10-16T10:04:08.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21f806f340 msgr2=0x7f21f814cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21f806f340 0x7f21f814cc20 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f21f000c750 tx=0x7f21f000ca60 comp rx=0 tx=0).stop 2023-10-16T10:04:08.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 shutdown_connections 2023-10-16T10:04:08.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f21e406b020 0x7f21e406d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21f815dfb0 0x7f21f80b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21f806f340 0x7f21f814cc20 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 --2- 172.21.15.158:0/3722294856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21f80cc5d0 0x7f21f80a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 >> 172.21.15.158:0/3722294856 conn(0x7f21f80d02b0 msgr2=0x7f21f80c9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:08.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 shutdown_connections 2023-10-16T10:04:08.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.418+0000 7f220017b700 1 -- 172.21.15.158:0/3722294856 wait complete. 2023-10-16T10:04:08.432 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T10:04:08.433 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:04:08.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.553+0000 7f3ba2416700 1 Processor -- start 2023-10-16T10:04:08.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.553+0000 7f3ba2416700 1 -- start start 2023-10-16T10:04:08.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3ba2416700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3b9c0d1d50 0x7f3b9c161540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3ba2416700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3b9c06ba90 0x7f3b9c161a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3ba2416700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c0d21e0 con 0x7f3b9c15dbc0 2023-10-16T10:04:08.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3ba2416700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c15f320 con 0x7f3b9c0d1d50 2023-10-16T10:04:08.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3ba2416700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c15ef10 con 0x7f3b9c06ba90 2023-10-16T10:04:08.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3b9c0d1d50 0x7f3b9c161540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3b9c0d1d50 0x7f3b9c161540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59804/0 (socket says 172.21.15.158:59804) 2023-10-16T10:04:08.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/4225144752 learned_addr learned my addr 172.21.15.158:0/4225144752 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:08.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3ba09b3700 1 --2- 172.21.15.158:0/4225144752 >> v2:172.21.15.172:3300/0 conn(0x7f3b9c06ba90 0x7f3b9c161a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/4225144752 >> v2:172.21.15.172:3300/0 conn(0x7f3b9c06ba90 msgr2=0x7f3b9c161a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:08.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 --2- 172.21.15.158:0/4225144752 >> v2:172.21.15.172:3300/0 conn(0x7f3b9c06ba90 0x7f3b9c161a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/4225144752 >> v1:172.21.15.172:6789/0 conn(0x7f3b9c15dbc0 legacy=0x7f3b9c0cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/4225144752 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c15e150 con 0x7f3b9c0d1d50 2023-10-16T10:04:08.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9b7fe700 1 --2- 172.21.15.158:0/4225144752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0d1d50 0x7f3b9c161540 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0b3220 tx=0x7f3b8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6399e5528b153eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.554+0000 7f3b9affd700 1 -- 172.21.15.158:0/4225144752 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c011830 con 0x7f3b9c0d1d50 2023-10-16T10:04:08.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3b9affd700 1 -- 172.21.15.158:0/4225144752 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c011990 con 0x7f3b9c0d1d50 2023-10-16T10:04:08.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3b9affd700 1 -- 172.21.15.158:0/4225144752 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c011d20 con 0x7f3b9c0d1d50 2023-10-16T10:04:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 -- 172.21.15.158:0/4225144752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0d1d50 msgr2=0x7f3b9c161540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 --2- 172.21.15.158:0/4225144752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0d1d50 0x7f3b9c161540 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0b3220 tx=0x7f3b8c009bd0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 -- 172.21.15.158:0/4225144752 shutdown_connections 2023-10-16T10:04:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 --2- 172.21.15.158:0/4225144752 >> v2:172.21.15.172:3300/0 conn(0x7f3b9c06ba90 0x7f3b9c161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 --2- 172.21.15.158:0/4225144752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0d1d50 0x7f3b9c161540 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 -- 172.21.15.158:0/4225144752 >> 172.21.15.158:0/4225144752 conn(0x7f3b9c06a8d0 msgr2=0x7f3b9c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 -- 172.21.15.158:0/4225144752 shutdown_connections 2023-10-16T10:04:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.555+0000 7f3ba2416700 1 -- 172.21.15.158:0/4225144752 wait complete. 2023-10-16T10:04:08.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.556+0000 7f3ba2416700 1 Processor -- start 2023-10-16T10:04:08.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.556+0000 7f3ba2416700 1 -- start start 2023-10-16T10:04:08.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3ba2416700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b9c06ba90 0x7f3b9c0b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3ba2416700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0b5700 0x7f3b9c151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3ba2416700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b9c0c7af0 0x7f3b9c0c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3ba2416700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0b5c30 con 0x7f3b9c0b5700 2023-10-16T10:04:08.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3ba2416700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0b5f10 con 0x7f3b9c0c7af0 2023-10-16T10:04:08.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3ba2416700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0b61f0 con 0x7f3b9c06ba90 2023-10-16T10:04:08.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3b9b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0b5700 0x7f3b9c151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3b9b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0b5700 0x7f3b9c151990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59810/0 (socket says 172.21.15.158:59810) 2023-10-16T10:04:08.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/3362806075 learned_addr learned my addr 172.21.15.158:0/3362806075 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:08.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3ba09b3700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b9c0c7af0 0x7f3b9c0c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.557+0000 7f3b9bfff700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b9c06ba90 0x7f3b9c0b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b9c06ba90 msgr2=0x7f3b9c0b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b9b7fe700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b9c06ba90 0x7f3b9c0b51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b9c0c7af0 msgr2=0x7f3b9c0c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b9b7fe700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b9c0c7af0 0x7f3b9c0c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b9b7fe700 1 -- 172.21.15.158:0/3362806075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c009550 con 0x7f3b9c0b5700 2023-10-16T10:04:08.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b9b7fe700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0b5700 0x7f3b9c151990 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c000c00 tx=0x7f3b8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b98ff9700 1 -- 172.21.15.158:0/3362806075 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c004570 con 0x7f3b9c0b5700 2023-10-16T10:04:08.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b98ff9700 1 -- 172.21.15.158:0/3362806075 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c01f070 con 0x7f3b9c0b5700 2023-10-16T10:04:08.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c0b64d0 con 0x7f3b9c0b5700 2023-10-16T10:04:08.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3b98ff9700 1 -- 172.21.15.158:0/3362806075 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c0038e0 con 0x7f3b9c0b5700 2023-10-16T10:04:08.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.558+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c151ed0 con 0x7f3b9c0b5700 2023-10-16T10:04:08.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.559+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c0b69f0 con 0x7f3b9c0b5700 2023-10-16T10:04:08.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.559+0000 7f3b98ff9700 1 -- 172.21.15.158:0/3362806075 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3b8c021030 con 0x7f3b9c0b5700 2023-10-16T10:04:08.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.560+0000 7f3b98ff9700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3b8806b020 0x7f3b8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.560+0000 7f3b98ff9700 1 -- 172.21.15.158:0/3362806075 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f3b8c08aed0 con 0x7f3b9c0b5700 2023-10-16T10:04:08.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.560+0000 7f3b9bfff700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3b8806b020 0x7f3b8806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.560+0000 7f3b9bfff700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3b8806b020 0x7f3b8806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b90000f60 tx=0x7f3b90009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.564+0000 7f3b98ff9700 1 -- 172.21.15.158:0/3362806075 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b9c0b69f0 con 0x7f3b9c0b5700 2023-10-16T10:04:08.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.757+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 --> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b9c15e710 con 0x7f3b8806b020 2023-10-16T10:04:08.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.766+0000 7f3b98ff9700 1 -- 172.21.15.158:0/3362806075 <== mgr.9141 v2:172.21.15.158:6832/3458664017 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289781 (secure 0 0 0) 0x7f3b90001490 con 0x7f3b8806b020 2023-10-16T10:04:08.773 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:08.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3b8806b020 msgr2=0x7f3b8806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3b8806b020 0x7f3b8806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b90000f60 tx=0x7f3b90009040 comp rx=0 tx=0).stop 2023-10-16T10:04:08.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0b5700 msgr2=0x7f3b9c151990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0b5700 0x7f3b9c151990 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c000c00 tx=0x7f3b8c004750 comp rx=0 tx=0).stop 2023-10-16T10:04:08.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 shutdown_connections 2023-10-16T10:04:08.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b9c0c7af0 0x7f3b9c0c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3b8806b020 0x7f3b8806d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b9c06ba90 0x7f3b9c0b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 --2- 172.21.15.158:0/3362806075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b9c0b5700 0x7f3b9c151990 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 >> 172.21.15.158:0/3362806075 conn(0x7f3b9c06a8d0 msgr2=0x7f3b9c0cbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:08.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 shutdown_connections 2023-10-16T10:04:08.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.769+0000 7f3ba2416700 1 -- 172.21.15.158:0/3362806075 wait complete. 2023-10-16T10:04:08.778 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:04:08.783 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T10:04:08.373581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":352964,"kb_used_data":16076,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754621756,"statfs":{"total":773094113280,"available":772732678144,"internally_reserved":0,"allocated":16461824,"data_stored":9181996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":344837687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.723944"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.681669+0000","last_change":"2023-10-16T10:04:04.676987+0000","last_active":"2023-10-16T10:04:04.681669+0000","last_peered":"2023-10-16T10:04:04.681669+0000","last_clean":"2023-10-16T10:04:04.681669+0000","last_became_active":"2023-10-16T10:04:04.674754+0000","last_became_peered":"2023-10-16T10:04:04.674754+0000","last_unstale":"2023-10-16T10:04:04.681669+0000","last_undegraded":"2023-10-16T10:04:04.681669+0000","last_fullsized":"2023-10-16T10:04:04.681669+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:23:59.897682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795693+0000","last_change":"2023-10-16T10:04:04.680606+0000","last_active":"2023-10-16T10:04:05.795693+0000","last_peered":"2023-10-16T10:04:05.795693+0000","last_clean":"2023-10-16T10:04:05.795693+0000","last_became_active":"2023-10-16T10:04:04.679028+0000","last_became_peered":"2023-10-16T10:04:04.679028+0000","last_unstale":"2023-10-16T10:04:05.795693+0000","last_undegraded":"2023-10-16T10:04:05.795693+0000","last_fullsized":"2023-10-16T10:04:05.795693+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:22.835248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.687338+0000","last_change":"2023-10-16T10:04:04.680960+0000","last_active":"2023-10-16T10:04:04.687338+0000","last_peered":"2023-10-16T10:04:04.687338+0000","last_clean":"2023-10-16T10:04:04.687338+0000","last_became_active":"2023-10-16T10:04:04.680705+0000","last_became_peered":"2023-10-16T10:04:04.680705+0000","last_unstale":"2023-10-16T10:04:04.687338+0000","last_undegraded":"2023-10-16T10:04:04.687338+0000","last_fullsized":"2023-10-16T10:04:04.687338+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:02.354619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795690+0000","last_change":"2023-10-16T10:04:04.680244+0000","last_active":"2023-10-16T10:04:05.795690+0000","last_peered":"2023-10-16T10:04:05.795690+0000","last_clean":"2023-10-16T10:04:05.795690+0000","last_became_active":"2023-10-16T10:04:04.679919+0000","last_became_peered":"2023-10-16T10:04:04.679919+0000","last_unstale":"2023-10-16T10:04:05.795690+0000","last_undegraded":"2023-10-16T10:04:05.795690+0000","last_fullsized":"2023-10-16T10:04:05.795690+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:56.774704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668597+0000","last_change":"2023-10-16T10:04:04.671623+0000","last_active":"2023-10-16T10:04:05.668597+0000","last_peered":"2023-10-16T10:04:05.668597+0000","last_clean":"2023-10-16T10:04:05.668597+0000","last_became_active":"2023-10-16T10:04:04.671283+0000","last_became_peered":"2023-10-16T10:04:04.671283+0000","last_unstale":"2023-10-16T10:04:05.668597+0000","last_undegraded":"2023-10-16T10:04:05.668597+0000","last_fullsized":"2023-10-16T10:04:05.668597+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:58.128654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.686623+0000","last_change":"2023-10-16T10:04:04.686263+0000","last_active":"2023-10-16T10:04:04.686623+0000","last_peered":"2023-10-16T10:04:04.686623+0000","last_clean":"2023-10-16T10:04:04.686623+0000","last_became_active":"2023-10-16T10:04:04.685981+0000","last_became_peered":"2023-10-16T10:04:04.685981+0000","last_unstale":"2023-10-16T10:04:04.686623+0000","last_undegraded":"2023-10-16T10:04:04.686623+0000","last_fullsized":"2023-10-16T10:04:04.686623+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:16.110780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667769+0000","last_change":"2023-10-16T10:04:04.684746+0000","last_active":"2023-10-16T10:04:05.667769+0000","last_peered":"2023-10-16T10:04:05.667769+0000","last_clean":"2023-10-16T10:04:05.667769+0000","last_became_active":"2023-10-16T10:04:04.684456+0000","last_became_peered":"2023-10-16T10:04:04.684456+0000","last_unstale":"2023-10-16T10:04:05.667769+0000","last_undegraded":"2023-10-16T10:04:05.667769+0000","last_fullsized":"2023-10-16T10:04:05.667769+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:53.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667896+0000","last_change":"2023-10-16T10:04:04.671456+0000","last_active":"2023-10-16T10:04:05.667896+0000","last_peered":"2023-10-16T10:04:05.667896+0000","last_clean":"2023-10-16T10:04:05.667896+0000","last_became_active":"2023-10-16T10:04:04.670194+0000","last_became_peered":"2023-10-16T10:04:04.670194+0000","last_unstale":"2023-10-16T10:04:05.667896+0000","last_undegraded":"2023-10-16T10:04:05.667896+0000","last_fullsized":"2023-10-16T10:04:05.667896+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:04.243581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.686596+0000","last_change":"2023-10-16T10:04:04.686214+0000","last_active":"2023-10-16T10:04:04.686596+0000","last_peered":"2023-10-16T10:04:04.686596+0000","last_clean":"2023-10-16T10:04:04.686596+0000","last_became_active":"2023-10-16T10:04:04.685824+0000","last_became_peered":"2023-10-16T10:04:04.685824+0000","last_unstale":"2023-10-16T10:04:04.686596+0000","last_undegraded":"2023-10-16T10:04:04.686596+0000","last_fullsized":"2023-10-16T10:04:04.686596+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:54.877564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.668643+0000","last_change":"2023-10-16T10:04:04.667804+0000","last_active":"2023-10-16T10:04:04.668643+0000","last_peered":"2023-10-16T10:04:04.668643+0000","last_clean":"2023-10-16T10:04:04.668643+0000","last_became_active":"2023-10-16T10:04:04.667526+0000","last_became_peered":"2023-10-16T10:04:04.667526+0000","last_unstale":"2023-10-16T10:04:04.668643+0000","last_undegraded":"2023-10-16T10:04:04.668643+0000","last_fullsized":"2023-10-16T10:04:04.668643+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:56.774704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795583+0000","last_change":"2023-10-16T10:04:04.682075+0000","last_active":"2023-10-16T10:04:05.795583+0000","last_peered":"2023-10-16T10:04:05.795583+0000","last_clean":"2023-10-16T10:04:05.795583+0000","last_became_active":"2023-10-16T10:04:04.681804+0000","last_became_peered":"2023-10-16T10:04:04.681804+0000","last_unstale":"2023-10-16T10:04:05.795583+0000","last_undegraded":"2023-10-16T10:04:05.795583+0000","last_fullsized":"2023-10-16T10:04:05.795583+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:49.159622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.677869+0000","last_change":"2023-10-16T10:04:04.674949+0000","last_active":"2023-10-16T10:04:04.677869+0000","last_peered":"2023-10-16T10:04:04.677869+0000","last_clean":"2023-10-16T10:04:04.677869+0000","last_became_active":"2023-10-16T10:04:04.674654+0000","last_became_peered":"2023-10-16T10:04:04.674654+0000","last_unstale":"2023-10-16T10:04:04.677869+0000","last_undegraded":"2023-10-16T10:04:04.677869+0000","last_fullsized":"2023-10-16T10:04:04.677869+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:49:28.649692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682793+0000","last_change":"2023-10-16T10:04:04.681903+0000","last_active":"2023-10-16T10:04:04.682793+0000","last_peered":"2023-10-16T10:04:04.682793+0000","last_clean":"2023-10-16T10:04:04.682793+0000","last_became_active":"2023-10-16T10:04:04.681624+0000","last_became_peered":"2023-10-16T10:04:04.681624+0000","last_unstale":"2023-10-16T10:04:04.682793+0000","last_undegraded":"2023-10-16T10:04:04.682793+0000","last_fullsized":"2023-10-16T10:04:04.682793+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:37.689661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682002+0000","last_change":"2023-10-16T10:04:04.677083+0000","last_active":"2023-10-16T10:04:04.682002+0000","last_peered":"2023-10-16T10:04:04.682002+0000","last_clean":"2023-10-16T10:04:04.682002+0000","last_became_active":"2023-10-16T10:04:04.676806+0000","last_became_peered":"2023-10-16T10:04:04.676806+0000","last_unstale":"2023-10-16T10:04:04.682002+0000","last_undegraded":"2023-10-16T10:04:04.682002+0000","last_fullsized":"2023-10-16T10:04:04.682002+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:49.083338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.677672+0000","last_change":"2023-10-16T10:04:04.674409+0000","last_active":"2023-10-16T10:04:04.677672+0000","last_peered":"2023-10-16T10:04:04.677672+0000","last_clean":"2023-10-16T10:04:04.677672+0000","last_became_active":"2023-10-16T10:04:04.674201+0000","last_became_peered":"2023-10-16T10:04:04.674201+0000","last_unstale":"2023-10-16T10:04:04.677672+0000","last_undegraded":"2023-10-16T10:04:04.677672+0000","last_fullsized":"2023-10-16T10:04:04.677672+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:44.211979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.681770+0000","last_change":"2023-10-16T10:04:04.679952+0000","last_active":"2023-10-16T10:04:04.681770+0000","last_peered":"2023-10-16T10:04:04.681770+0000","last_clean":"2023-10-16T10:04:04.681770+0000","last_became_active":"2023-10-16T10:04:04.679698+0000","last_became_peered":"2023-10-16T10:04:04.679698+0000","last_unstale":"2023-10-16T10:04:04.681770+0000","last_undegraded":"2023-10-16T10:04:04.681770+0000","last_fullsized":"2023-10-16T10:04:04.681770+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:17.004744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.674253+0000","last_change":"2023-10-16T10:04:04.671866+0000","last_active":"2023-10-16T10:04:04.674253+0000","last_peered":"2023-10-16T10:04:04.674253+0000","last_clean":"2023-10-16T10:04:04.674253+0000","last_became_active":"2023-10-16T10:04:04.671614+0000","last_became_peered":"2023-10-16T10:04:04.671614+0000","last_unstale":"2023-10-16T10:04:04.674253+0000","last_undegraded":"2023-10-16T10:04:04.674253+0000","last_fullsized":"2023-10-16T10:04:04.674253+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:23.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668589+0000","last_change":"2023-10-16T10:04:04.671623+0000","last_active":"2023-10-16T10:04:05.668589+0000","last_peered":"2023-10-16T10:04:05.668589+0000","last_clean":"2023-10-16T10:04:05.668589+0000","last_became_active":"2023-10-16T10:04:04.671338+0000","last_became_peered":"2023-10-16T10:04:04.671338+0000","last_unstale":"2023-10-16T10:04:05.668589+0000","last_undegraded":"2023-10-16T10:04:05.668589+0000","last_fullsized":"2023-10-16T10:04:05.668589+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:50.556411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669520+0000","last_change":"2023-10-16T10:04:04.670291+0000","last_active":"2023-10-16T10:04:05.669520+0000","last_peered":"2023-10-16T10:04:05.669520+0000","last_clean":"2023-10-16T10:04:05.669520+0000","last_became_active":"2023-10-16T10:04:04.670012+0000","last_became_peered":"2023-10-16T10:04:04.670012+0000","last_unstale":"2023-10-16T10:04:05.669520+0000","last_undegraded":"2023-10-16T10:04:05.669520+0000","last_fullsized":"2023-10-16T10:04:05.669520+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:21.671698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668734+0000","last_change":"2023-10-16T10:04:04.675490+0000","last_active":"2023-10-16T10:04:05.668734+0000","last_peered":"2023-10-16T10:04:05.668734+0000","last_clean":"2023-10-16T10:04:05.668734+0000","last_became_active":"2023-10-16T10:04:04.675146+0000","last_became_peered":"2023-10-16T10:04:04.675146+0000","last_unstale":"2023-10-16T10:04:05.668734+0000","last_undegraded":"2023-10-16T10:04:05.668734+0000","last_fullsized":"2023-10-16T10:04:05.668734+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:10.171478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795473+0000","last_change":"2023-10-16T10:04:04.675392+0000","last_active":"2023-10-16T10:04:05.795473+0000","last_peered":"2023-10-16T10:04:05.795473+0000","last_clean":"2023-10-16T10:04:05.795473+0000","last_became_active":"2023-10-16T10:04:04.675118+0000","last_became_peered":"2023-10-16T10:04:04.675118+0000","last_unstale":"2023-10-16T10:04:05.795473+0000","last_undegraded":"2023-10-16T10:04:05.795473+0000","last_fullsized":"2023-10-16T10:04:05.795473+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:13.137832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.683023+0000","last_change":"2023-10-16T10:04:04.682461+0000","last_active":"2023-10-16T10:04:04.683023+0000","last_peered":"2023-10-16T10:04:04.683023+0000","last_clean":"2023-10-16T10:04:04.683023+0000","last_became_active":"2023-10-16T10:04:04.682162+0000","last_became_peered":"2023-10-16T10:04:04.682162+0000","last_unstale":"2023-10-16T10:04:04.683023+0000","last_undegraded":"2023-10-16T10:04:04.683023+0000","last_fullsized":"2023-10-16T10:04:04.683023+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:02.495908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795574+0000","last_change":"2023-10-16T10:04:04.679961+0000","last_active":"2023-10-16T10:04:05.795574+0000","last_peered":"2023-10-16T10:04:05.795574+0000","last_clean":"2023-10-16T10:04:05.795574+0000","last_became_active":"2023-10-16T10:04:04.678205+0000","last_became_peered":"2023-10-16T10:04:04.678205+0000","last_unstale":"2023-10-16T10:04:05.795574+0000","last_undegraded":"2023-10-16T10:04:05.795574+0000","last_fullsized":"2023-10-16T10:04:05.795574+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:31.215405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682560+0000","last_change":"2023-10-16T10:04:04.681519+0000","last_active":"2023-10-16T10:04:04.682560+0000","last_peered":"2023-10-16T10:04:04.682560+0000","last_clean":"2023-10-16T10:04:04.682560+0000","last_became_active":"2023-10-16T10:04:04.681228+0000","last_became_peered":"2023-10-16T10:04:04.681228+0000","last_unstale":"2023-10-16T10:04:04.682560+0000","last_undegraded":"2023-10-16T10:04:04.682560+0000","last_fullsized":"2023-10-16T10:04:04.682560+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:20.563663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.690739+0000","last_change":"2023-10-16T10:04:04.688549+0000","last_active":"2023-10-16T10:04:04.690739+0000","last_peered":"2023-10-16T10:04:04.690739+0000","last_clean":"2023-10-16T10:04:04.690739+0000","last_became_active":"2023-10-16T10:04:04.688283+0000","last_became_peered":"2023-10-16T10:04:04.688283+0000","last_unstale":"2023-10-16T10:04:04.690739+0000","last_undegraded":"2023-10-16T10:04:04.690739+0000","last_fullsized":"2023-10-16T10:04:04.690739+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:04.243581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.689160+0000","last_change":"2023-10-16T10:04:04.686692+0000","last_active":"2023-10-16T10:04:04.689160+0000","last_peered":"2023-10-16T10:04:04.689160+0000","last_clean":"2023-10-16T10:04:04.689160+0000","last_became_active":"2023-10-16T10:04:04.686350+0000","last_became_peered":"2023-10-16T10:04:04.686350+0000","last_unstale":"2023-10-16T10:04:04.689160+0000","last_undegraded":"2023-10-16T10:04:04.689160+0000","last_fullsized":"2023-10-16T10:04:04.689160+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:59.022801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669390+0000","last_change":"2023-10-16T10:04:04.672514+0000","last_active":"2023-10-16T10:04:05.669390+0000","last_peered":"2023-10-16T10:04:05.669390+0000","last_clean":"2023-10-16T10:04:05.669390+0000","last_became_active":"2023-10-16T10:04:04.671994+0000","last_became_peered":"2023-10-16T10:04:04.671994+0000","last_unstale":"2023-10-16T10:04:05.669390+0000","last_undegraded":"2023-10-16T10:04:05.669390+0000","last_fullsized":"2023-10-16T10:04:05.669390+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.659825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.668598+0000","last_change":"2023-10-16T10:04:04.667799+0000","last_active":"2023-10-16T10:04:04.668598+0000","last_peered":"2023-10-16T10:04:04.668598+0000","last_clean":"2023-10-16T10:04:04.668598+0000","last_became_active":"2023-10-16T10:04:04.667500+0000","last_became_peered":"2023-10-16T10:04:04.667500+0000","last_unstale":"2023-10-16T10:04:04.668598+0000","last_undegraded":"2023-10-16T10:04:04.668598+0000","last_fullsized":"2023-10-16T10:04:04.668598+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:10.931058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.688270+0000","last_change":"2023-10-16T10:04:04.683690+0000","last_active":"2023-10-16T10:04:04.688270+0000","last_peered":"2023-10-16T10:04:04.688270+0000","last_clean":"2023-10-16T10:04:04.688270+0000","last_became_active":"2023-10-16T10:04:04.683406+0000","last_became_peered":"2023-10-16T10:04:04.683406+0000","last_unstale":"2023-10-16T10:04:04.688270+0000","last_undegraded":"2023-10-16T10:04:04.688270+0000","last_fullsized":"2023-10-16T10:04:04.688270+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:12.851592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.685072+0000","last_change":"2023-10-16T10:04:04.682211+0000","last_active":"2023-10-16T10:04:04.685072+0000","last_peered":"2023-10-16T10:04:04.685072+0000","last_clean":"2023-10-16T10:04:04.685072+0000","last_became_active":"2023-10-16T10:04:04.681919+0000","last_became_peered":"2023-10-16T10:04:04.681919+0000","last_unstale":"2023-10-16T10:04:04.685072+0000","last_undegraded":"2023-10-16T10:04:04.685072+0000","last_fullsized":"2023-10-16T10:04:04.685072+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:38.754410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668029+0000","last_change":"2023-10-16T10:04:04.674400+0000","last_active":"2023-10-16T10:04:05.668029+0000","last_peered":"2023-10-16T10:04:05.668029+0000","last_clean":"2023-10-16T10:04:05.668029+0000","last_became_active":"2023-10-16T10:04:04.674135+0000","last_became_peered":"2023-10-16T10:04:04.674135+0000","last_unstale":"2023-10-16T10:04:05.668029+0000","last_undegraded":"2023-10-16T10:04:05.668029+0000","last_fullsized":"2023-10-16T10:04:05.668029+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:43.191440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.671785+0000","last_change":"2023-10-16T10:04:04.670635+0000","last_active":"2023-10-16T10:04:04.671785+0000","last_peered":"2023-10-16T10:04:04.671785+0000","last_clean":"2023-10-16T10:04:04.671785+0000","last_became_active":"2023-10-16T10:04:04.670313+0000","last_became_peered":"2023-10-16T10:04:04.670313+0000","last_unstale":"2023-10-16T10:04:04.671785+0000","last_undegraded":"2023-10-16T10:04:04.671785+0000","last_fullsized":"2023-10-16T10:04:04.671785+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:27.536546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669192+0000","last_change":"2023-10-16T10:04:04.671537+0000","last_active":"2023-10-16T10:04:05.669192+0000","last_peered":"2023-10-16T10:04:05.669192+0000","last_clean":"2023-10-16T10:04:05.669192+0000","last_became_active":"2023-10-16T10:04:04.671185+0000","last_became_peered":"2023-10-16T10:04:04.671185+0000","last_unstale":"2023-10-16T10:04:05.669192+0000","last_undegraded":"2023-10-16T10:04:05.669192+0000","last_fullsized":"2023-10-16T10:04:05.669192+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:54.508051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"152'4","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695664+0000","last_change":"2023-10-16T10:04:03.670215+0000","last_active":"2023-10-16T10:04:04.695664+0000","last_peered":"2023-10-16T10:04:04.695664+0000","last_clean":"2023-10-16T10:04:04.695664+0000","last_became_active":"2023-10-16T10:04:03.669921+0000","last_became_peered":"2023-10-16T10:04:03.669921+0000","last_unstale":"2023-10-16T10:04:04.695664+0000","last_undegraded":"2023-10-16T10:04:04.695664+0000","last_fullsized":"2023-10-16T10:04:04.695664+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:42.184281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682845+0000","last_change":"2023-10-16T10:04:04.681943+0000","last_active":"2023-10-16T10:04:04.682845+0000","last_peered":"2023-10-16T10:04:04.682845+0000","last_clean":"2023-10-16T10:04:04.682845+0000","last_became_active":"2023-10-16T10:04:04.681654+0000","last_became_peered":"2023-10-16T10:04:04.681654+0000","last_unstale":"2023-10-16T10:04:04.682845+0000","last_undegraded":"2023-10-16T10:04:04.682845+0000","last_fullsized":"2023-10-16T10:04:04.682845+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:38.449242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668741+0000","last_change":"2023-10-16T10:04:03.667947+0000","last_active":"2023-10-16T10:04:05.668741+0000","last_peered":"2023-10-16T10:04:05.668741+0000","last_clean":"2023-10-16T10:04:05.668741+0000","last_became_active":"2023-10-16T10:04:03.667637+0000","last_became_peered":"2023-10-16T10:04:03.667637+0000","last_unstale":"2023-10-16T10:04:05.668741+0000","last_undegraded":"2023-10-16T10:04:05.668741+0000","last_fullsized":"2023-10-16T10:04:05.668741+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:55.767546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795633+0000","last_change":"2023-10-16T10:04:04.678888+0000","last_active":"2023-10-16T10:04:05.795633+0000","last_peered":"2023-10-16T10:04:05.795633+0000","last_clean":"2023-10-16T10:04:05.795633+0000","last_became_active":"2023-10-16T10:04:04.676457+0000","last_became_peered":"2023-10-16T10:04:04.676457+0000","last_unstale":"2023-10-16T10:04:05.795633+0000","last_undegraded":"2023-10-16T10:04:05.795633+0000","last_fullsized":"2023-10-16T10:04:05.795633+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:27.536546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"152'2","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669409+0000","last_change":"2023-10-16T10:04:03.665596+0000","last_active":"2023-10-16T10:04:05.669409+0000","last_peered":"2023-10-16T10:04:05.669409+0000","last_clean":"2023-10-16T10:04:05.669409+0000","last_became_active":"2023-10-16T10:04:03.665287+0000","last_became_peered":"2023-10-16T10:04:03.665287+0000","last_unstale":"2023-10-16T10:04:05.669409+0000","last_undegraded":"2023-10-16T10:04:05.669409+0000","last_fullsized":"2023-10-16T10:04:05.669409+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:58:36.866060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682613+0000","last_change":"2023-10-16T10:04:04.680200+0000","last_active":"2023-10-16T10:04:04.682613+0000","last_peered":"2023-10-16T10:04:04.682613+0000","last_clean":"2023-10-16T10:04:04.682613+0000","last_became_active":"2023-10-16T10:04:04.679939+0000","last_became_peered":"2023-10-16T10:04:04.679939+0000","last_unstale":"2023-10-16T10:04:04.682613+0000","last_undegraded":"2023-10-16T10:04:04.682613+0000","last_fullsized":"2023-10-16T10:04:04.682613+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:41.949411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"152'2","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667847+0000","last_change":"2023-10-16T10:04:03.666061+0000","last_active":"2023-10-16T10:04:05.667847+0000","last_peered":"2023-10-16T10:04:05.667847+0000","last_clean":"2023-10-16T10:04:05.667847+0000","last_became_active":"2023-10-16T10:04:03.665741+0000","last_became_peered":"2023-10-16T10:04:03.665741+0000","last_unstale":"2023-10-16T10:04:05.667847+0000","last_undegraded":"2023-10-16T10:04:05.667847+0000","last_fullsized":"2023-10-16T10:04:05.667847+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:05:48.559362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668420+0000","last_change":"2023-10-16T10:04:04.675007+0000","last_active":"2023-10-16T10:04:05.668420+0000","last_peered":"2023-10-16T10:04:05.668420+0000","last_clean":"2023-10-16T10:04:05.668420+0000","last_became_active":"2023-10-16T10:04:04.674725+0000","last_became_peered":"2023-10-16T10:04:04.674725+0000","last_unstale":"2023-10-16T10:04:05.668420+0000","last_undegraded":"2023-10-16T10:04:05.668420+0000","last_fullsized":"2023-10-16T10:04:05.668420+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:37.689661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669129+0000","last_change":"2023-10-16T10:04:03.668304+0000","last_active":"2023-10-16T10:04:05.669129+0000","last_peered":"2023-10-16T10:04:05.669129+0000","last_clean":"2023-10-16T10:04:05.669129+0000","last_became_active":"2023-10-16T10:04:03.668036+0000","last_became_peered":"2023-10-16T10:04:03.668036+0000","last_unstale":"2023-10-16T10:04:05.669129+0000","last_undegraded":"2023-10-16T10:04:05.669129+0000","last_fullsized":"2023-10-16T10:04:05.669129+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:58.015643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668365+0000","last_change":"2023-10-16T10:04:04.670372+0000","last_active":"2023-10-16T10:04:05.668365+0000","last_peered":"2023-10-16T10:04:05.668365+0000","last_clean":"2023-10-16T10:04:05.668365+0000","last_became_active":"2023-10-16T10:04:04.670097+0000","last_became_peered":"2023-10-16T10:04:04.670097+0000","last_unstale":"2023-10-16T10:04:05.668365+0000","last_undegraded":"2023-10-16T10:04:05.668365+0000","last_fullsized":"2023-10-16T10:04:05.668365+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:40.048526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668400+0000","last_change":"2023-10-16T10:04:03.663804+0000","last_active":"2023-10-16T10:04:05.668400+0000","last_peered":"2023-10-16T10:04:05.668400+0000","last_clean":"2023-10-16T10:04:05.668400+0000","last_became_active":"2023-10-16T10:04:03.663643+0000","last_became_peered":"2023-10-16T10:04:03.663643+0000","last_unstale":"2023-10-16T10:04:05.668400+0000","last_undegraded":"2023-10-16T10:04:05.668400+0000","last_fullsized":"2023-10-16T10:04:05.668400+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:57.305053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668689+0000","last_change":"2023-10-16T10:04:04.675744+0000","last_active":"2023-10-16T10:04:05.668689+0000","last_peered":"2023-10-16T10:04:05.668689+0000","last_clean":"2023-10-16T10:04:05.668689+0000","last_became_active":"2023-10-16T10:04:04.675483+0000","last_became_peered":"2023-10-16T10:04:04.675483+0000","last_unstale":"2023-10-16T10:04:05.668689+0000","last_undegraded":"2023-10-16T10:04:05.668689+0000","last_fullsized":"2023-10-16T10:04:05.668689+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:21.671698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.667076+0000","last_change":"2023-10-16T10:04:03.666452+0000","last_active":"2023-10-16T10:04:04.667076+0000","last_peered":"2023-10-16T10:04:04.667076+0000","last_clean":"2023-10-16T10:04:04.667076+0000","last_became_active":"2023-10-16T10:04:03.666179+0000","last_became_peered":"2023-10-16T10:04:03.666179+0000","last_unstale":"2023-10-16T10:04:04.667076+0000","last_undegraded":"2023-10-16T10:04:04.667076+0000","last_fullsized":"2023-10-16T10:04:04.667076+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:49.610888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668639+0000","last_change":"2023-10-16T10:04:04.676179+0000","last_active":"2023-10-16T10:04:05.668639+0000","last_peered":"2023-10-16T10:04:05.668639+0000","last_clean":"2023-10-16T10:04:05.668639+0000","last_became_active":"2023-10-16T10:04:04.675871+0000","last_became_peered":"2023-10-16T10:04:04.675871+0000","last_unstale":"2023-10-16T10:04:05.668639+0000","last_undegraded":"2023-10-16T10:04:05.668639+0000","last_fullsized":"2023-10-16T10:04:05.668639+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:50.556411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.675499+0000","last_change":"2023-10-16T10:04:03.664965+0000","last_active":"2023-10-16T10:04:04.675499+0000","last_peered":"2023-10-16T10:04:04.675499+0000","last_clean":"2023-10-16T10:04:04.675499+0000","last_became_active":"2023-10-16T10:04:03.664686+0000","last_became_peered":"2023-10-16T10:04:03.664686+0000","last_unstale":"2023-10-16T10:04:04.675499+0000","last_undegraded":"2023-10-16T10:04:04.675499+0000","last_fullsized":"2023-10-16T10:04:04.675499+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:22.950332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.685168+0000","last_change":"2023-10-16T10:04:04.683631+0000","last_active":"2023-10-16T10:04:04.685168+0000","last_peered":"2023-10-16T10:04:04.685168+0000","last_clean":"2023-10-16T10:04:04.685168+0000","last_became_active":"2023-10-16T10:04:04.683468+0000","last_became_peered":"2023-10-16T10:04:04.683468+0000","last_unstale":"2023-10-16T10:04:04.685168+0000","last_undegraded":"2023-10-16T10:04:04.685168+0000","last_fullsized":"2023-10-16T10:04:04.685168+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:33.742220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795605+0000","last_change":"2023-10-16T10:04:03.666216+0000","last_active":"2023-10-16T10:04:05.795605+0000","last_peered":"2023-10-16T10:04:05.795605+0000","last_clean":"2023-10-16T10:04:05.795605+0000","last_became_active":"2023-10-16T10:04:03.665934+0000","last_became_peered":"2023-10-16T10:04:03.665934+0000","last_unstale":"2023-10-16T10:04:05.795605+0000","last_undegraded":"2023-10-16T10:04:05.795605+0000","last_fullsized":"2023-10-16T10:04:05.795605+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:57.305053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.674778+0000","last_change":"2023-10-16T10:04:04.672949+0000","last_active":"2023-10-16T10:04:04.674778+0000","last_peered":"2023-10-16T10:04:04.674778+0000","last_clean":"2023-10-16T10:04:04.674778+0000","last_became_active":"2023-10-16T10:04:04.672676+0000","last_became_peered":"2023-10-16T10:04:04.672676+0000","last_unstale":"2023-10-16T10:04:04.674778+0000","last_undegraded":"2023-10-16T10:04:04.674778+0000","last_fullsized":"2023-10-16T10:04:04.674778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:54.508051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.664863+0000","last_change":"2023-10-16T10:04:03.669827+0000","last_active":"2023-10-16T10:04:04.664863+0000","last_peered":"2023-10-16T10:04:04.664863+0000","last_clean":"2023-10-16T10:04:04.664863+0000","last_became_active":"2023-10-16T10:04:03.669574+0000","last_became_peered":"2023-10-16T10:04:03.669574+0000","last_unstale":"2023-10-16T10:04:04.664863+0000","last_undegraded":"2023-10-16T10:04:04.664863+0000","last_fullsized":"2023-10-16T10:04:04.664863+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:05.685692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.689010+0000","last_change":"2023-10-16T10:04:04.684881+0000","last_active":"2023-10-16T10:04:04.689010+0000","last_peered":"2023-10-16T10:04:04.689010+0000","last_clean":"2023-10-16T10:04:04.689010+0000","last_became_active":"2023-10-16T10:04:04.684601+0000","last_became_peered":"2023-10-16T10:04:04.684601+0000","last_unstale":"2023-10-16T10:04:04.689010+0000","last_undegraded":"2023-10-16T10:04:04.689010+0000","last_fullsized":"2023-10-16T10:04:04.689010+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:38.109699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"152'3","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669512+0000","last_change":"2023-10-16T10:04:03.668456+0000","last_active":"2023-10-16T10:04:05.669512+0000","last_peered":"2023-10-16T10:04:05.669512+0000","last_clean":"2023-10-16T10:04:05.669512+0000","last_became_active":"2023-10-16T10:04:03.668160+0000","last_became_peered":"2023-10-16T10:04:03.668160+0000","last_unstale":"2023-10-16T10:04:05.669512+0000","last_undegraded":"2023-10-16T10:04:05.669512+0000","last_fullsized":"2023-10-16T10:04:05.669512+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:44.090615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.683789+0000","last_change":"2023-10-16T10:04:04.681525+0000","last_active":"2023-10-16T10:04:04.683789+0000","last_peered":"2023-10-16T10:04:04.683789+0000","last_clean":"2023-10-16T10:04:04.683789+0000","last_became_active":"2023-10-16T10:04:04.681020+0000","last_became_peered":"2023-10-16T10:04:04.681020+0000","last_unstale":"2023-10-16T10:04:04.683789+0000","last_undegraded":"2023-10-16T10:04:04.683789+0000","last_fullsized":"2023-10-16T10:04:04.683789+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:23:59.419140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"152'6","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668058+0000","last_change":"2023-10-16T10:04:03.668664+0000","last_active":"2023-10-16T10:04:05.668058+0000","last_peered":"2023-10-16T10:04:05.668058+0000","last_clean":"2023-10-16T10:04:05.668058+0000","last_became_active":"2023-10-16T10:04:03.666911+0000","last_became_peered":"2023-10-16T10:04:03.666911+0000","last_unstale":"2023-10-16T10:04:05.668058+0000","last_undegraded":"2023-10-16T10:04:05.668058+0000","last_fullsized":"2023-10-16T10:04:05.668058+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:07.288560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.678782+0000","last_change":"2023-10-16T10:04:04.677240+0000","last_active":"2023-10-16T10:04:04.678782+0000","last_peered":"2023-10-16T10:04:04.678782+0000","last_clean":"2023-10-16T10:04:04.678782+0000","last_became_active":"2023-10-16T10:04:04.676959+0000","last_became_peered":"2023-10-16T10:04:04.676959+0000","last_unstale":"2023-10-16T10:04:04.678782+0000","last_undegraded":"2023-10-16T10:04:04.678782+0000","last_fullsized":"2023-10-16T10:04:04.678782+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:22.835248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"152'2","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.694325+0000","last_change":"2023-10-16T10:04:03.670480+0000","last_active":"2023-10-16T10:04:04.694325+0000","last_peered":"2023-10-16T10:04:04.694325+0000","last_clean":"2023-10-16T10:04:04.694325+0000","last_became_active":"2023-10-16T10:04:03.670317+0000","last_became_peered":"2023-10-16T10:04:03.670317+0000","last_unstale":"2023-10-16T10:04:04.694325+0000","last_undegraded":"2023-10-16T10:04:04.694325+0000","last_fullsized":"2023-10-16T10:04:04.694325+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:26.719464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668262+0000","last_change":"2023-10-16T10:04:04.686196+0000","last_active":"2023-10-16T10:04:05.668262+0000","last_peered":"2023-10-16T10:04:05.668262+0000","last_clean":"2023-10-16T10:04:05.668262+0000","last_became_active":"2023-10-16T10:04:04.685908+0000","last_became_peered":"2023-10-16T10:04:04.685908+0000","last_unstale":"2023-10-16T10:04:05.668262+0000","last_undegraded":"2023-10-16T10:04:05.668262+0000","last_fullsized":"2023-10-16T10:04:05.668262+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:04.733721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.675897+0000","last_change":"2023-10-16T10:04:03.667625+0000","last_active":"2023-10-16T10:04:04.675897+0000","last_peered":"2023-10-16T10:04:04.675897+0000","last_clean":"2023-10-16T10:04:04.675897+0000","last_became_active":"2023-10-16T10:04:03.667335+0000","last_became_peered":"2023-10-16T10:04:03.667335+0000","last_unstale":"2023-10-16T10:04:04.675897+0000","last_undegraded":"2023-10-16T10:04:04.675897+0000","last_fullsized":"2023-10-16T10:04:04.675897+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:57.114434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668900+0000","last_change":"2023-10-16T10:04:04.676237+0000","last_active":"2023-10-16T10:04:05.668900+0000","last_peered":"2023-10-16T10:04:05.668900+0000","last_clean":"2023-10-16T10:04:05.668900+0000","last_became_active":"2023-10-16T10:04:04.675981+0000","last_became_peered":"2023-10-16T10:04:04.675981+0000","last_unstale":"2023-10-16T10:04:05.668900+0000","last_undegraded":"2023-10-16T10:04:05.668900+0000","last_fullsized":"2023-10-16T10:04:05.668900+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:51.953692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"152'8","reported_seq":19,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.696626+0000","last_change":"2023-10-16T10:04:03.660640+0000","last_active":"2023-10-16T10:04:04.696626+0000","last_peered":"2023-10-16T10:04:04.696626+0000","last_clean":"2023-10-16T10:04:04.696626+0000","last_became_active":"2023-10-16T10:04:03.660385+0000","last_became_peered":"2023-10-16T10:04:03.660385+0000","last_unstale":"2023-10-16T10:04:04.696626+0000","last_undegraded":"2023-10-16T10:04:04.696626+0000","last_fullsized":"2023-10-16T10:04:04.696626+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:26.057348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.684269+0000","last_change":"2023-10-16T10:04:04.678632+0000","last_active":"2023-10-16T10:04:04.684269+0000","last_peered":"2023-10-16T10:04:04.684269+0000","last_clean":"2023-10-16T10:04:04.684269+0000","last_became_active":"2023-10-16T10:04:04.678350+0000","last_became_peered":"2023-10-16T10:04:04.678350+0000","last_unstale":"2023-10-16T10:04:04.684269+0000","last_undegraded":"2023-10-16T10:04:04.684269+0000","last_fullsized":"2023-10-16T10:04:04.684269+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:34.618183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795732+0000","last_change":"2023-10-16T10:04:03.664840+0000","last_active":"2023-10-16T10:04:05.795732+0000","last_peered":"2023-10-16T10:04:05.795732+0000","last_clean":"2023-10-16T10:04:05.795732+0000","last_became_active":"2023-10-16T10:04:03.664560+0000","last_became_peered":"2023-10-16T10:04:03.664560+0000","last_unstale":"2023-10-16T10:04:05.795732+0000","last_undegraded":"2023-10-16T10:04:05.795732+0000","last_fullsized":"2023-10-16T10:04:05.795732+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:19.652102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667934+0000","last_change":"2023-10-16T10:04:04.672154+0000","last_active":"2023-10-16T10:04:05.667934+0000","last_peered":"2023-10-16T10:04:05.667934+0000","last_clean":"2023-10-16T10:04:05.667934+0000","last_became_active":"2023-10-16T10:04:04.671855+0000","last_became_peered":"2023-10-16T10:04:04.671855+0000","last_unstale":"2023-10-16T10:04:05.667934+0000","last_undegraded":"2023-10-16T10:04:05.667934+0000","last_fullsized":"2023-10-16T10:04:05.667934+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:56.774704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"152'2","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695803+0000","last_change":"2023-10-16T10:04:03.670010+0000","last_active":"2023-10-16T10:04:04.695803+0000","last_peered":"2023-10-16T10:04:04.695803+0000","last_clean":"2023-10-16T10:04:04.695803+0000","last_became_active":"2023-10-16T10:04:03.669737+0000","last_became_peered":"2023-10-16T10:04:03.669737+0000","last_unstale":"2023-10-16T10:04:04.695803+0000","last_undegraded":"2023-10-16T10:04:04.695803+0000","last_fullsized":"2023-10-16T10:04:04.695803+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:54.712464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668031+0000","last_change":"2023-10-16T10:04:04.671117+0000","last_active":"2023-10-16T10:04:05.668031+0000","last_peered":"2023-10-16T10:04:05.668031+0000","last_clean":"2023-10-16T10:04:05.668031+0000","last_became_active":"2023-10-16T10:04:04.670865+0000","last_became_peered":"2023-10-16T10:04:04.670865+0000","last_unstale":"2023-10-16T10:04:05.668031+0000","last_undegraded":"2023-10-16T10:04:05.668031+0000","last_fullsized":"2023-10-16T10:04:05.668031+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:22.835248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"152'1","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695308+0000","last_change":"2023-10-16T10:04:03.666149+0000","last_active":"2023-10-16T10:04:04.695308+0000","last_peered":"2023-10-16T10:04:04.695308+0000","last_clean":"2023-10-16T10:04:04.695308+0000","last_became_active":"2023-10-16T10:04:03.665863+0000","last_became_peered":"2023-10-16T10:04:03.665863+0000","last_unstale":"2023-10-16T10:04:04.695308+0000","last_undegraded":"2023-10-16T10:04:04.695308+0000","last_fullsized":"2023-10-16T10:04:04.695308+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:03.236423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.683221+0000","last_change":"2023-10-16T10:04:04.681427+0000","last_active":"2023-10-16T10:04:04.683221+0000","last_peered":"2023-10-16T10:04:04.683221+0000","last_clean":"2023-10-16T10:04:04.683221+0000","last_became_active":"2023-10-16T10:04:04.681124+0000","last_became_peered":"2023-10-16T10:04:04.681124+0000","last_unstale":"2023-10-16T10:04:04.683221+0000","last_undegraded":"2023-10-16T10:04:04.683221+0000","last_fullsized":"2023-10-16T10:04:04.683221+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:53.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668732+0000","last_change":"2023-10-16T10:04:03.668680+0000","last_active":"2023-10-16T10:04:05.668732+0000","last_peered":"2023-10-16T10:04:05.668732+0000","last_clean":"2023-10-16T10:04:05.668732+0000","last_became_active":"2023-10-16T10:04:03.668339+0000","last_became_peered":"2023-10-16T10:04:03.668339+0000","last_unstale":"2023-10-16T10:04:05.668732+0000","last_undegraded":"2023-10-16T10:04:05.668732+0000","last_fullsized":"2023-10-16T10:04:05.668732+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:21.828090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668646+0000","last_change":"2023-10-16T10:04:04.677663+0000","last_active":"2023-10-16T10:04:05.668646+0000","last_peered":"2023-10-16T10:04:05.668646+0000","last_clean":"2023-10-16T10:04:05.668646+0000","last_became_active":"2023-10-16T10:04:04.677417+0000","last_became_peered":"2023-10-16T10:04:04.677417+0000","last_unstale":"2023-10-16T10:04:05.668646+0000","last_undegraded":"2023-10-16T10:04:05.668646+0000","last_fullsized":"2023-10-16T10:04:05.668646+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.659825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"152'2","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669269+0000","last_change":"2023-10-16T10:04:03.664473+0000","last_active":"2023-10-16T10:04:05.669269+0000","last_peered":"2023-10-16T10:04:05.669269+0000","last_clean":"2023-10-16T10:04:05.669269+0000","last_became_active":"2023-10-16T10:04:03.664239+0000","last_became_peered":"2023-10-16T10:04:03.664239+0000","last_unstale":"2023-10-16T10:04:05.669269+0000","last_undegraded":"2023-10-16T10:04:05.669269+0000","last_fullsized":"2023-10-16T10:04:05.669269+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:54.712464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682519+0000","last_change":"2023-10-16T10:04:04.677507+0000","last_active":"2023-10-16T10:04:04.682519+0000","last_peered":"2023-10-16T10:04:04.682519+0000","last_clean":"2023-10-16T10:04:04.682519+0000","last_became_active":"2023-10-16T10:04:04.677239+0000","last_became_peered":"2023-10-16T10:04:04.677239+0000","last_unstale":"2023-10-16T10:04:04.682519+0000","last_undegraded":"2023-10-16T10:04:04.682519+0000","last_fullsized":"2023-10-16T10:04:04.682519+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:17.157813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.673607+0000","last_change":"2023-10-16T10:04:03.668707+0000","last_active":"2023-10-16T10:04:04.673607+0000","last_peered":"2023-10-16T10:04:04.673607+0000","last_clean":"2023-10-16T10:04:04.673607+0000","last_became_active":"2023-10-16T10:04:03.667936+0000","last_became_peered":"2023-10-16T10:04:03.667936+0000","last_unstale":"2023-10-16T10:04:04.673607+0000","last_undegraded":"2023-10-16T10:04:04.673607+0000","last_fullsized":"2023-10-16T10:04:04.673607+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:28.886145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668740+0000","last_change":"2023-10-16T10:04:04.687193+0000","last_active":"2023-10-16T10:04:05.668740+0000","last_peered":"2023-10-16T10:04:05.668740+0000","last_clean":"2023-10-16T10:04:05.668740+0000","last_became_active":"2023-10-16T10:04:04.686926+0000","last_became_peered":"2023-10-16T10:04:04.686926+0000","last_unstale":"2023-10-16T10:04:05.668740+0000","last_undegraded":"2023-10-16T10:04:05.668740+0000","last_fullsized":"2023-10-16T10:04:05.668740+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:13.703024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668174+0000","last_change":"2023-10-16T10:04:03.672658+0000","last_active":"2023-10-16T10:04:05.668174+0000","last_peered":"2023-10-16T10:04:05.668174+0000","last_clean":"2023-10-16T10:04:05.668174+0000","last_became_active":"2023-10-16T10:04:03.672432+0000","last_became_peered":"2023-10-16T10:04:03.672432+0000","last_unstale":"2023-10-16T10:04:05.668174+0000","last_undegraded":"2023-10-16T10:04:05.668174+0000","last_fullsized":"2023-10-16T10:04:05.668174+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:26.529388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795636+0000","last_change":"2023-10-16T10:04:04.688730+0000","last_active":"2023-10-16T10:04:05.795636+0000","last_peered":"2023-10-16T10:04:05.795636+0000","last_clean":"2023-10-16T10:04:05.795636+0000","last_became_active":"2023-10-16T10:04:04.688438+0000","last_became_peered":"2023-10-16T10:04:04.688438+0000","last_unstale":"2023-10-16T10:04:05.795636+0000","last_undegraded":"2023-10-16T10:04:05.795636+0000","last_fullsized":"2023-10-16T10:04:05.795636+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:53.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.668878+0000","last_change":"2023-10-16T10:04:03.668943+0000","last_active":"2023-10-16T10:04:04.668878+0000","last_peered":"2023-10-16T10:04:04.668878+0000","last_clean":"2023-10-16T10:04:04.668878+0000","last_became_active":"2023-10-16T10:04:03.668665+0000","last_became_peered":"2023-10-16T10:04:03.668665+0000","last_unstale":"2023-10-16T10:04:04.668878+0000","last_undegraded":"2023-10-16T10:04:04.668878+0000","last_fullsized":"2023-10-16T10:04:04.668878+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:43:50.425071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668434+0000","last_change":"2023-10-16T10:04:04.676104+0000","last_active":"2023-10-16T10:04:05.668434+0000","last_peered":"2023-10-16T10:04:05.668434+0000","last_clean":"2023-10-16T10:04:05.668434+0000","last_became_active":"2023-10-16T10:04:04.675498+0000","last_became_peered":"2023-10-16T10:04:04.675498+0000","last_unstale":"2023-10-16T10:04:05.668434+0000","last_undegraded":"2023-10-16T10:04:05.668434+0000","last_fullsized":"2023-10-16T10:04:05.668434+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:04.733721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"152'2","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695643+0000","last_change":"2023-10-16T10:04:03.669278+0000","last_active":"2023-10-16T10:04:04.695643+0000","last_peered":"2023-10-16T10:04:04.695643+0000","last_clean":"2023-10-16T10:04:04.695643+0000","last_became_active":"2023-10-16T10:04:03.669015+0000","last_became_peered":"2023-10-16T10:04:03.669015+0000","last_unstale":"2023-10-16T10:04:04.695643+0000","last_undegraded":"2023-10-16T10:04:04.695643+0000","last_fullsized":"2023-10-16T10:04:04.695643+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.307298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.688151+0000","last_change":"2023-10-16T10:04:04.683596+0000","last_active":"2023-10-16T10:04:04.688151+0000","last_peered":"2023-10-16T10:04:04.688151+0000","last_clean":"2023-10-16T10:04:04.688151+0000","last_became_active":"2023-10-16T10:04:04.683329+0000","last_became_peered":"2023-10-16T10:04:04.683329+0000","last_unstale":"2023-10-16T10:04:04.688151+0000","last_undegraded":"2023-10-16T10:04:04.688151+0000","last_fullsized":"2023-10-16T10:04:04.688151+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:54.877564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668443+0000","last_change":"2023-10-16T10:04:03.667617+0000","last_active":"2023-10-16T10:04:05.668443+0000","last_peered":"2023-10-16T10:04:05.668443+0000","last_clean":"2023-10-16T10:04:05.668443+0000","last_became_active":"2023-10-16T10:04:03.667346+0000","last_became_peered":"2023-10-16T10:04:03.667346+0000","last_unstale":"2023-10-16T10:04:05.668443+0000","last_undegraded":"2023-10-16T10:04:05.668443+0000","last_fullsized":"2023-10-16T10:04:05.668443+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:37.747252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668219+0000","last_change":"2023-10-16T10:04:04.686263+0000","last_active":"2023-10-16T10:04:05.668219+0000","last_peered":"2023-10-16T10:04:05.668219+0000","last_clean":"2023-10-16T10:04:05.668219+0000","last_became_active":"2023-10-16T10:04:04.685980+0000","last_became_peered":"2023-10-16T10:04:04.685980+0000","last_unstale":"2023-10-16T10:04:05.668219+0000","last_undegraded":"2023-10-16T10:04:05.668219+0000","last_fullsized":"2023-10-16T10:04:05.668219+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:10.931058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668462+0000","last_change":"2023-10-16T10:04:03.669954+0000","last_active":"2023-10-16T10:04:05.668462+0000","last_peered":"2023-10-16T10:04:05.668462+0000","last_clean":"2023-10-16T10:04:05.668462+0000","last_became_active":"2023-10-16T10:04:03.669762+0000","last_became_peered":"2023-10-16T10:04:03.669762+0000","last_unstale":"2023-10-16T10:04:05.668462+0000","last_undegraded":"2023-10-16T10:04:05.668462+0000","last_fullsized":"2023-10-16T10:04:05.668462+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:57.114434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.672375+0000","last_change":"2023-10-16T10:04:04.671104+0000","last_active":"2023-10-16T10:04:04.672375+0000","last_peered":"2023-10-16T10:04:04.672375+0000","last_clean":"2023-10-16T10:04:04.672375+0000","last_became_active":"2023-10-16T10:04:04.670822+0000","last_became_peered":"2023-10-16T10:04:04.670822+0000","last_unstale":"2023-10-16T10:04:04.672375+0000","last_undegraded":"2023-10-16T10:04:04.672375+0000","last_fullsized":"2023-10-16T10:04:04.672375+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:04.733721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.666141+0000","last_change":"2023-10-16T10:04:03.666173+0000","last_active":"2023-10-16T10:04:04.666141+0000","last_peered":"2023-10-16T10:04:04.666141+0000","last_clean":"2023-10-16T10:04:04.666141+0000","last_became_active":"2023-10-16T10:04:03.665917+0000","last_became_peered":"2023-10-16T10:04:03.665917+0000","last_unstale":"2023-10-16T10:04:04.666141+0000","last_undegraded":"2023-10-16T10:04:04.666141+0000","last_fullsized":"2023-10-16T10:04:04.666141+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:39.041368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.680945+0000","last_change":"2023-10-16T10:04:04.678587+0000","last_active":"2023-10-16T10:04:04.680945+0000","last_peered":"2023-10-16T10:04:04.680945+0000","last_clean":"2023-10-16T10:04:04.680945+0000","last_became_active":"2023-10-16T10:04:04.678319+0000","last_became_peered":"2023-10-16T10:04:04.678319+0000","last_unstale":"2023-10-16T10:04:04.680945+0000","last_undegraded":"2023-10-16T10:04:04.680945+0000","last_fullsized":"2023-10-16T10:04:04.680945+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:43.191440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795444+0000","last_change":"2023-10-16T10:04:03.668013+0000","last_active":"2023-10-16T10:04:05.795444+0000","last_peered":"2023-10-16T10:04:05.795444+0000","last_clean":"2023-10-16T10:04:05.795444+0000","last_became_active":"2023-10-16T10:04:03.667714+0000","last_became_peered":"2023-10-16T10:04:03.667714+0000","last_unstale":"2023-10-16T10:04:05.795444+0000","last_undegraded":"2023-10-16T10:04:05.795444+0000","last_fullsized":"2023-10-16T10:04:05.795444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:44.090615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682032+0000","last_change":"2023-10-16T10:04:04.679521+0000","last_active":"2023-10-16T10:04:04.682032+0000","last_peered":"2023-10-16T10:04:04.682032+0000","last_clean":"2023-10-16T10:04:04.682032+0000","last_became_active":"2023-10-16T10:04:04.679254+0000","last_became_peered":"2023-10-16T10:04:04.679254+0000","last_unstale":"2023-10-16T10:04:04.682032+0000","last_undegraded":"2023-10-16T10:04:04.682032+0000","last_fullsized":"2023-10-16T10:04:04.682032+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:12.851592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668377+0000","last_change":"2023-10-16T10:04:03.666014+0000","last_active":"2023-10-16T10:04:05.668377+0000","last_peered":"2023-10-16T10:04:05.668377+0000","last_clean":"2023-10-16T10:04:05.668377+0000","last_became_active":"2023-10-16T10:04:03.665748+0000","last_became_peered":"2023-10-16T10:04:03.665748+0000","last_unstale":"2023-10-16T10:04:05.668377+0000","last_undegraded":"2023-10-16T10:04:05.668377+0000","last_fullsized":"2023-10-16T10:04:05.668377+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:39:27.408754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.673596+0000","last_change":"2023-10-16T10:04:04.670413+0000","last_active":"2023-10-16T10:04:04.673596+0000","last_peered":"2023-10-16T10:04:04.673596+0000","last_clean":"2023-10-16T10:04:04.673596+0000","last_became_active":"2023-10-16T10:04:04.670162+0000","last_became_peered":"2023-10-16T10:04:04.670162+0000","last_unstale":"2023-10-16T10:04:04.673596+0000","last_undegraded":"2023-10-16T10:04:04.673596+0000","last_fullsized":"2023-10-16T10:04:04.673596+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:58.121592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.667069+0000","last_change":"2023-10-16T10:04:03.663448+0000","last_active":"2023-10-16T10:04:04.667069+0000","last_peered":"2023-10-16T10:04:04.667069+0000","last_clean":"2023-10-16T10:04:04.667069+0000","last_became_active":"2023-10-16T10:04:03.663166+0000","last_became_peered":"2023-10-16T10:04:03.663166+0000","last_unstale":"2023-10-16T10:04:04.667069+0000","last_undegraded":"2023-10-16T10:04:04.667069+0000","last_fullsized":"2023-10-16T10:04:04.667069+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:48:52.234268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668777+0000","last_change":"2023-10-16T10:04:04.690645+0000","last_active":"2023-10-16T10:04:05.668777+0000","last_peered":"2023-10-16T10:04:05.668777+0000","last_clean":"2023-10-16T10:04:05.668777+0000","last_became_active":"2023-10-16T10:04:04.689661+0000","last_became_peered":"2023-10-16T10:04:04.689661+0000","last_unstale":"2023-10-16T10:04:05.668777+0000","last_undegraded":"2023-10-16T10:04:05.668777+0000","last_fullsized":"2023-10-16T10:04:05.668777+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:58.128654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668652+0000","last_change":"2023-10-16T10:04:03.668573+0000","last_active":"2023-10-16T10:04:05.668652+0000","last_peered":"2023-10-16T10:04:05.668652+0000","last_clean":"2023-10-16T10:04:05.668652+0000","last_became_active":"2023-10-16T10:04:03.668300+0000","last_became_peered":"2023-10-16T10:04:03.668300+0000","last_unstale":"2023-10-16T10:04:05.668652+0000","last_undegraded":"2023-10-16T10:04:05.668652+0000","last_fullsized":"2023-10-16T10:04:05.668652+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:12.130674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.687359+0000","last_change":"2023-10-16T10:04:04.681776+0000","last_active":"2023-10-16T10:04:04.687359+0000","last_peered":"2023-10-16T10:04:04.687359+0000","last_clean":"2023-10-16T10:04:04.687359+0000","last_became_active":"2023-10-16T10:04:04.681180+0000","last_became_peered":"2023-10-16T10:04:04.681180+0000","last_unstale":"2023-10-16T10:04:04.687359+0000","last_undegraded":"2023-10-16T10:04:04.687359+0000","last_fullsized":"2023-10-16T10:04:04.687359+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:17.265041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668275+0000","last_change":"2023-10-16T10:04:03.672672+0000","last_active":"2023-10-16T10:04:05.668275+0000","last_peered":"2023-10-16T10:04:05.668275+0000","last_clean":"2023-10-16T10:04:05.668275+0000","last_became_active":"2023-10-16T10:04:03.672413+0000","last_became_peered":"2023-10-16T10:04:03.672413+0000","last_unstale":"2023-10-16T10:04:05.668275+0000","last_undegraded":"2023-10-16T10:04:05.668275+0000","last_fullsized":"2023-10-16T10:04:05.668275+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:57.305053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"143'249","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.666591+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:04:04.666591+0000","last_peered":"2023-10-16T10:04:04.666591+0000","last_clean":"2023-10-16T10:04:04.666591+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:04:04.666591+0000","last_undegraded":"2023-10-16T10:04:04.666591+0000","last_fullsized":"2023-10-16T10:04:04.666591+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575092,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60008,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311832,"statfs":{"total":96636764160,"available":96575315968,"internally_reserved":0,"allocated":2519040,"data_stored":1596045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.76500000000000001,"15min":1.2150000000000001},"last":0.497},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55400000000000005,"15min":0.57099999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.254},"max":{"1min":0.83999999999999997,"5min":1.911,"15min":1.911},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57599999999999996,"15min":0.59799999999999998},"min":{"1min":0.432,"5min":0.34599999999999997,"15min":0.29199999999999998},"max":{"1min":0.85199999999999998,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.1279999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.625},"min":{"1min":0.33800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78800000000000003,"5min":3.3820000000000001,"15min":3.3820000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.63600000000000001},"min":{"1min":0.41899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.84399999999999997,"5min":3.1640000000000001,"15min":3.1640000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.59799999999999998},"min":{"1min":0.39800000000000002,"5min":0.33800000000000002,"15min":0.32300000000000001},"max":{"1min":0.81000000000000005,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64900000000000002,"15min":0.65400000000000003},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.874,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.57899999999999996}]},{"osd":4,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.501,"15min":0.495},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.72499999999999998,"5min":1.8220000000000001,"15min":1.9550000000000001},"last":0.373},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56100000000000005,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33000000000000002,"15min":0.27200000000000002},"max":{"1min":0.83899999999999997,"5min":3.8149999999999999,"15min":3.8149999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.80200000000000005,"5min":0.81999999999999995,"15min":0.84099999999999997},"last":0.439},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.40600000000000003,"5min":0.37,"15min":0.308},"max":{"1min":0.84599999999999997,"5min":1.6619999999999999,"15min":2.0150000000000001},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.7110000000000001,"15min":1.9870000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57999999999999996,"15min":0.57399999999999995},"min":{"1min":0.377,"5min":0.31,"15min":0.31},"max":{"1min":0.88100000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.46100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575092,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":997575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53800000000000003,"15min":0.56999999999999995},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":0.72099999999999997,"5min":0.89100000000000001,"15min":1.0920000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.249},"max":{"1min":0.72999999999999998,"5min":0.80900000000000005,"15min":0.82099999999999995},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.745,"5min":0.77100000000000002,"15min":0.90000000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.55400000000000005,"15min":0.59199999999999997},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.77400000000000002,"5min":0.78900000000000003,"15min":0.83199999999999996},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54200000000000004,"15min":0.57399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.27100000000000002},"max":{"1min":0.75700000000000001,"5min":0.78200000000000003,"15min":1.448},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56599999999999995,"15min":0.59499999999999997},"min":{"1min":0.40899999999999997,"5min":0.35899999999999999,"15min":0.35799999999999998},"max":{"1min":0.78700000000000003,"5min":0.83499999999999996,"15min":1.708},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.42999999999999999,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.315},"max":{"1min":0.71799999999999997,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.499,"15min":0.47999999999999998},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.249},"max":{"1min":0.74399999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.434},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.27500000000000002},"max":{"1min":0.78000000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54300000000000004}]},{"osd":5,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.253},"max":{"1min":0.69699999999999995,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57099999999999995,"15min":0.54900000000000004},"min":{"1min":0.433,"5min":0.35799999999999998,"15min":0.311},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.505}]},{"osd":7,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.80200000000000005,"5min":0.86499999999999999,"15min":0.91900000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.35799999999999998,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.80500000000000005,"15min":0.94599999999999995},"last":0.46200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1900544,"data_stored":996973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.35199999999999998,"15min":0.38600000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.188},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":1.6000000000000001},"last":0.308},{"interface":"front","average":{"1min":0.379,"5min":0.39800000000000002,"15min":0.43099999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.70199999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36099999999999999,"15min":0.40200000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":0.63300000000000001,"5min":0.98699999999999999,"15min":4.923},"last":0.252},{"interface":"front","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.42699999999999999},"min":{"1min":0.28000000000000003,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":1.712},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42899999999999999,"15min":0.45300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.984},"last":0.37},{"interface":"front","average":{"1min":0.375,"5min":0.40400000000000003,"15min":0.42499999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.27000000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.87,"15min":0.93100000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.63700000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"max":{"1min":0.89000000000000001,"5min":1.0840000000000001,"15min":5.0069999999999997},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61599999999999999,"15min":0.65400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92300000000000004,"5min":1.04,"15min":4.9749999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.66200000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.085,"5min":1.085,"15min":1.7769999999999999},"last":0.61599999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60199999999999998,"15min":0.63100000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.63},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61499999999999999,"15min":0.64900000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.052,"5min":1.3029999999999999,"15min":5.0419999999999998},"last":0.56299999999999994}]},{"osd":7,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.744},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61299999999999999,"15min":0.64200000000000002},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575093,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37780,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334060,"statfs":{"total":96636764160,"available":96598077440,"internally_reserved":0,"allocated":2498560,"data_stored":1587327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35799999999999998,"15min":0.38100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.192},"max":{"1min":0.52000000000000002,"5min":0.65700000000000003,"15min":1.2110000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39600000000000002,"15min":0.42699999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26500000000000001},"max":{"1min":0.54000000000000004,"5min":0.63,"15min":1.3169999999999999},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.20599999999999999},"max":{"1min":0.56299999999999994,"5min":0.61899999999999999,"15min":0.70099999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.45500000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.249},"max":{"1min":0.48499999999999999,"5min":0.63600000000000001,"15min":0.79900000000000004},"last":0.40100000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.38100000000000001,"15min":0.41299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.83399999999999996},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42799999999999999,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.624,"5min":0.67300000000000004,"15min":1.1859999999999999},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.33700000000000002},"max":{"1min":0.81799999999999995,"5min":0.83999999999999997,"15min":0.93700000000000006},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59199999999999997,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":1.1659999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59299999999999997,"15min":0.63},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.33300000000000002},"max":{"1min":0.80500000000000005,"5min":0.86499999999999999,"15min":0.99299999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.625},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.311},"max":{"1min":0.82899999999999996,"5min":0.89100000000000001,"15min":0.91000000000000003},"last":0.47399999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.40899999999999997},"max":{"1min":0.89200000000000002,"5min":0.90900000000000003,"15min":1.2410000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.624,"15min":0.66400000000000003},"min":{"1min":0.44800000000000001,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.877,"15min":0.97199999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60799999999999998,"15min":0.63900000000000001},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.316},"max":{"1min":0.88500000000000001,"5min":0.89200000000000002,"15min":1.02},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.60399999999999998,"15min":0.64000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.34899999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.44700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59416,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312424,"statfs":{"total":96636764160,"available":96575922176,"internally_reserved":0,"allocated":1904640,"data_stored":997519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.372},"min":{"1min":0.22900000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.496,"5min":0.57099999999999995,"15min":0.61599999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39500000000000002,"15min":0.40899999999999997},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39900000000000002,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.67900000000000005,"15min":0.68000000000000005},"last":0.374},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.623,"5min":0.624,"15min":0.624},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.246,"15min":0.23499999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.63500000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.47099999999999997},"min":{"1min":0.315,"5min":0.29899999999999999,"15min":0.25600000000000001},"max":{"1min":0.57399999999999995,"5min":0.72499999999999998,"15min":0.79800000000000004},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.629},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84999999999999998,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.33400000000000002},"max":{"1min":0.84899999999999998,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59599999999999997,"15min":0.63800000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76600000000000001,"5min":1.0249999999999999,"15min":1.103},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"max":{"1min":0.77700000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":1.046,"15min":1.1699999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60699999999999998,"15min":0.66100000000000003},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.84399999999999997,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81499999999999995,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.501},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.53000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1925120,"data_stored":1005657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34399999999999997,"15min":0.36599999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.17999999999999999},"max":{"1min":0.51100000000000001,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.38600000000000001},"min":{"1min":0.249,"5min":0.216,"15min":0.216},"max":{"1min":0.56299999999999994,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.245}]},{"osd":1,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35599999999999998,"15min":0.39000000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.51700000000000002,"5min":0.58899999999999997,"15min":0.65400000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39200000000000002,"15min":0.42299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.67500000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.45900000000000002},"min":{"1min":0.32700000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.65000000000000002,"5min":0.96099999999999997,"15min":1.0760000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45600000000000002,"15min":0.48299999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68600000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.57799999999999996,"15min":0.61599999999999999},"min":{"1min":0.35599999999999998,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.96499999999999997},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.33700000000000002},"max":{"1min":0.81599999999999995,"5min":0.83399999999999996,"15min":1.6910000000000001},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":1.6259999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.38700000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89100000000000001,"5min":1.673,"15min":1.673},"last":0.81899999999999995}]},{"osd":6,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58599999999999997,"15min":0.63900000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.374},"max":{"1min":0.83099999999999996,"5min":0.83299999999999996,"15min":0.98799999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59499999999999997,"15min":0.64400000000000002},"min":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86199999999999999,"5min":0.92600000000000005,"15min":1.0589999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.62,"15min":0.66200000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85699999999999998,"5min":0.878,"15min":1.0029999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.83099999999999996,"15min":0.72299999999999998},"min":{"1min":0.42799999999999999,"5min":0.42699999999999999,"15min":0.36899999999999999},"max":{"1min":0.96999999999999997,"5min":24.861000000000001,"15min":24.861000000000001},"last":0.69099999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575092,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1908736,"data_stored":1003975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.72199999999999998,"5min":0.82099999999999995,"15min":1.0269999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57699999999999996,"15min":0.58799999999999997},"min":{"1min":0.29799999999999999,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":0.76000000000000001,"5min":0.83099999999999996,"15min":1.0860000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.58499999999999996},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.312},"max":{"1min":0.70699999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.41899999999999998,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":1.6359999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.85199999999999998,"15min":0.88300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.38600000000000001,"15min":0.32500000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.2270000000000001},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55700000000000005,"15min":0.58299999999999996},"min":{"1min":0.40000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.69099999999999995,"5min":0.75800000000000001,"15min":0.86099999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60399999999999998,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.38300000000000001,"15min":0.34200000000000003},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.495},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36199999999999999,"5min":0.26400000000000001,"15min":0.26300000000000001},"max":{"1min":0.72899999999999998,"5min":0.77500000000000002,"15min":0.96599999999999997},"last":0.76400000000000001}]},{"osd":6,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.24199999999999999},"max":{"1min":0.71899999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.57199999999999995},"min":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":0.73199999999999998,"5min":0.80000000000000004,"15min":0.92400000000000004},"last":0.85899999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58899999999999997,"15min":0.57499999999999996},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.77900000000000003,"5min":0.82399999999999995,"15min":0.91100000000000003},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.626,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.89400000000000002,"15min":1.0429999999999999},"last":0.93300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575092,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51088,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320752,"statfs":{"total":96636764160,"available":96584450048,"internally_reserved":0,"allocated":1900544,"data_stored":996925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.61099999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.792},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.435,"5min":0.34599999999999997,"15min":0.27300000000000002},"max":{"1min":0.80800000000000005,"5min":0.85799999999999998,"15min":1.7090000000000001},"last":0.56599999999999995}]},{"osd":1,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55500000000000005,"15min":0.57899999999999996},"min":{"1min":0.34999999999999998,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.79400000000000004,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54200000000000004,"15min":0.56999999999999995},"min":{"1min":0.34399999999999997,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.82899999999999996},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.28599999999999998,"15min":0.25800000000000001},"max":{"1min":0.754,"5min":0.80800000000000005,"15min":1.496},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.59499999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34200000000000003},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":2.069},"last":0.71199999999999997}]},{"osd":3,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.317,"15min":0.317},"max":{"1min":0.70599999999999996,"5min":1,"15min":1},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.39600000000000002,"5min":0.34499999999999997,"15min":0.33400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":1.7689999999999999},"last":0.73599999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.5,"15min":0.48599999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27600000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.84199999999999997},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52500000000000002,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.86899999999999999},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54500000000000004},"min":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.72099999999999997,"5min":0.78400000000000003,"15min":0.88200000000000001},"last":0.76500000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.25900000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82899999999999996,"5min":0.88,"15min":0.88},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:04:08.789 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T10:04:08.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.942+0000 7f6a2e61a700 1 Processor -- start 2023-10-16T10:04:08.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.942+0000 7f6a2e61a700 1 -- start start 2023-10-16T10:04:08.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.942+0000 7f6a2e61a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.942+0000 7f6a2e61a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6a2806e090 0x7f6a280ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.942+0000 7f6a2e61a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a28072910 con 0x7f6a280af590 2023-10-16T10:04:08.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.942+0000 7f6a2e61a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a280ca7e0 con 0x7f6a2806e090 2023-10-16T10:04:08.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.942+0000 7f6a2e61a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280b1e40 con 0x7f6a2815a6c0 2023-10-16T10:04:08.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a2cbb7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6a2806e090 0x7f6a280ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:42418/0 (socket says 172.21.15.158:42418) 2023-10-16T10:04:08.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 -- 172.21.15.158:0/120551994 learned_addr learned my addr 172.21.15.158:0/120551994 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:08.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 -- 172.21.15.158:0/120551994 >> v2:172.21.15.172:3300/0 conn(0x7f6a2806e090 msgr2=0x7f6a280ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:08.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 --2- 172.21.15.158:0/120551994 >> v2:172.21.15.172:3300/0 conn(0x7f6a2806e090 0x7f6a280ad430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 -- 172.21.15.158:0/120551994 >> v1:172.21.15.172:6789/0 conn(0x7f6a280af590 legacy=0x7f6a2806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 -- 172.21.15.158:0/120551994 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a280b2120 con 0x7f6a2815a6c0 2023-10-16T10:04:08.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a2cbb7700 1 --2- 172.21.15.158:0/120551994 >> v2:172.21.15.172:3300/0 conn(0x7f6a2806e090 0x7f6a280ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:04:08.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a27fff700 1 --2- 172.21.15.158:0/120551994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280723d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006fd0 tx=0x7f6a1800be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a32fc7039ea90617 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.943+0000 7f6a26ffd700 1 -- 172.21.15.158:0/120551994 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18005e30 con 0x7f6a2815a6c0 2023-10-16T10:04:08.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a26ffd700 1 -- 172.21.15.158:0/120551994 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a18004030 con 0x7f6a2815a6c0 2023-10-16T10:04:08.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a26ffd700 1 -- 172.21.15.158:0/120551994 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a180043e0 con 0x7f6a2815a6c0 2023-10-16T10:04:08.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 -- 172.21.15.158:0/120551994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 msgr2=0x7f6a280723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/120551994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280723d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006fd0 tx=0x7f6a1800be70 comp rx=0 tx=0).stop 2023-10-16T10:04:08.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 -- 172.21.15.158:0/120551994 shutdown_connections 2023-10-16T10:04:08.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/120551994 >> v2:172.21.15.172:3300/0 conn(0x7f6a2806e090 0x7f6a280ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/120551994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280723d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 -- 172.21.15.158:0/120551994 >> 172.21.15.158:0/120551994 conn(0x7f6a280ccda0 msgr2=0x7f6a280cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:08.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 -- 172.21.15.158:0/120551994 shutdown_connections 2023-10-16T10:04:08.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.944+0000 7f6a2e61a700 1 -- 172.21.15.158:0/120551994 wait complete. 2023-10-16T10:04:08.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.945+0000 7f6a2e61a700 1 Processor -- start 2023-10-16T10:04:08.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a2e61a700 1 -- start start 2023-10-16T10:04:08.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a2e61a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2806e090 0x7f6a280c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a2e61a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a280af590 0x7f6a280c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a2e61a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a2e61a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280c9010 con 0x7f6a2806e090 2023-10-16T10:04:08.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a2e61a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280c6a60 con 0x7f6a280af590 2023-10-16T10:04:08.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a2e61a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280c5760 con 0x7f6a2815a6c0 2023-10-16T10:04:08.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a27fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2806e090 0x7f6a280c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a27fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2806e090 0x7f6a280c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59822/0 (socket says 172.21.15.158:59822) 2023-10-16T10:04:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.946+0000 7f6a27fff700 1 -- 172.21.15.158:0/3607860636 learned_addr learned my addr 172.21.15.158:0/3607860636 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a2cbb7700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a277fe700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a280af590 0x7f6a280c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a27fff700 1 -- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 msgr2=0x7f6a280c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a27fff700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a27fff700 1 -- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a280af590 msgr2=0x7f6a280c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a27fff700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a280af590 0x7f6a280c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a27fff700 1 -- 172.21.15.158:0/3607860636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20009a70 con 0x7f6a2806e090 2023-10-16T10:04:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a2cbb7700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a27fff700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2806e090 0x7f6a280c6520 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800beb0 tx=0x7f6a18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a24ff9700 1 -- 172.21.15.158:0/3607860636 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18003b30 con 0x7f6a2806e090 2023-10-16T10:04:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1800b7a0 con 0x7f6a2806e090 2023-10-16T10:04:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a24ff9700 1 -- 172.21.15.158:0/3607860636 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a18003cd0 con 0x7f6a2806e090 2023-10-16T10:04:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a24ff9700 1 -- 172.21.15.158:0/3607860636 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a180106b0 con 0x7f6a2806e090 2023-10-16T10:04:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.947+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a280c5d20 con 0x7f6a2806e090 2023-10-16T10:04:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.948+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a280681b0 con 0x7f6a2806e090 2023-10-16T10:04:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.949+0000 7f6a24ff9700 1 -- 172.21.15.158:0/3607860636 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a18024400 con 0x7f6a2806e090 2023-10-16T10:04:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.949+0000 7f6a24ff9700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f6a1406b020 0x7f6a1406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.949+0000 7f6a24ff9700 1 -- 172.21.15.158:0/3607860636 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f6a1808d2d0 con 0x7f6a2806e090 2023-10-16T10:04:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.949+0000 7f6a277fe700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f6a1406b020 0x7f6a1406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:08.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.949+0000 7f6a277fe700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f6a1406b020 0x7f6a1406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6a280c3a70 tx=0x7f6a1c009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:08.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:08.954+0000 7f6a24ff9700 1 -- 172.21.15.158:0/3607860636 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a18057f70 con 0x7f6a2806e090 2023-10-16T10:04:09.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.141+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 --> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a280b26e0 con 0x7f6a1406b020 2023-10-16T10:04:09.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.149+0000 7f6a24ff9700 1 -- 172.21.15.158:0/3607860636 <== mgr.9141 v2:172.21.15.158:6832/3458664017 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289781 (secure 0 0 0) 0x7f6a1c001490 con 0x7f6a1406b020 2023-10-16T10:04:09.153 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:09.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f6a1406b020 msgr2=0x7f6a1406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f6a1406b020 0x7f6a1406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6a280c3a70 tx=0x7f6a1c009040 comp rx=0 tx=0).stop 2023-10-16T10:04:09.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2806e090 msgr2=0x7f6a280c6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2806e090 0x7f6a280c6520 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800beb0 tx=0x7f6a18004750 comp rx=0 tx=0).stop 2023-10-16T10:04:09.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 shutdown_connections 2023-10-16T10:04:09.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f6a1406b020 0x7f6a1406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a2815a6c0 0x7f6a280c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2806e090 0x7f6a280c6520 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 --2- 172.21.15.158:0/3607860636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a280af590 0x7f6a280c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 >> 172.21.15.158:0/3607860636 conn(0x7f6a280ccda0 msgr2=0x7f6a280afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:09.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 shutdown_connections 2023-10-16T10:04:09.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.152+0000 7f6a2e61a700 1 -- 172.21.15.158:0/3607860636 wait complete. 2023-10-16T10:04:09.158 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T10:04:09.165 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T10:04:08.373581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":352964,"kb_used_data":16076,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754621756,"statfs":{"total":773094113280,"available":772732678144,"internally_reserved":0,"allocated":16461824,"data_stored":9181996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":344837687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.723944"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.681669+0000","last_change":"2023-10-16T10:04:04.676987+0000","last_active":"2023-10-16T10:04:04.681669+0000","last_peered":"2023-10-16T10:04:04.681669+0000","last_clean":"2023-10-16T10:04:04.681669+0000","last_became_active":"2023-10-16T10:04:04.674754+0000","last_became_peered":"2023-10-16T10:04:04.674754+0000","last_unstale":"2023-10-16T10:04:04.681669+0000","last_undegraded":"2023-10-16T10:04:04.681669+0000","last_fullsized":"2023-10-16T10:04:04.681669+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:23:59.897682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795693+0000","last_change":"2023-10-16T10:04:04.680606+0000","last_active":"2023-10-16T10:04:05.795693+0000","last_peered":"2023-10-16T10:04:05.795693+0000","last_clean":"2023-10-16T10:04:05.795693+0000","last_became_active":"2023-10-16T10:04:04.679028+0000","last_became_peered":"2023-10-16T10:04:04.679028+0000","last_unstale":"2023-10-16T10:04:05.795693+0000","last_undegraded":"2023-10-16T10:04:05.795693+0000","last_fullsized":"2023-10-16T10:04:05.795693+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:22.835248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.687338+0000","last_change":"2023-10-16T10:04:04.680960+0000","last_active":"2023-10-16T10:04:04.687338+0000","last_peered":"2023-10-16T10:04:04.687338+0000","last_clean":"2023-10-16T10:04:04.687338+0000","last_became_active":"2023-10-16T10:04:04.680705+0000","last_became_peered":"2023-10-16T10:04:04.680705+0000","last_unstale":"2023-10-16T10:04:04.687338+0000","last_undegraded":"2023-10-16T10:04:04.687338+0000","last_fullsized":"2023-10-16T10:04:04.687338+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:02.354619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795690+0000","last_change":"2023-10-16T10:04:04.680244+0000","last_active":"2023-10-16T10:04:05.795690+0000","last_peered":"2023-10-16T10:04:05.795690+0000","last_clean":"2023-10-16T10:04:05.795690+0000","last_became_active":"2023-10-16T10:04:04.679919+0000","last_became_peered":"2023-10-16T10:04:04.679919+0000","last_unstale":"2023-10-16T10:04:05.795690+0000","last_undegraded":"2023-10-16T10:04:05.795690+0000","last_fullsized":"2023-10-16T10:04:05.795690+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:56.774704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668597+0000","last_change":"2023-10-16T10:04:04.671623+0000","last_active":"2023-10-16T10:04:05.668597+0000","last_peered":"2023-10-16T10:04:05.668597+0000","last_clean":"2023-10-16T10:04:05.668597+0000","last_became_active":"2023-10-16T10:04:04.671283+0000","last_became_peered":"2023-10-16T10:04:04.671283+0000","last_unstale":"2023-10-16T10:04:05.668597+0000","last_undegraded":"2023-10-16T10:04:05.668597+0000","last_fullsized":"2023-10-16T10:04:05.668597+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:58.128654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.686623+0000","last_change":"2023-10-16T10:04:04.686263+0000","last_active":"2023-10-16T10:04:04.686623+0000","last_peered":"2023-10-16T10:04:04.686623+0000","last_clean":"2023-10-16T10:04:04.686623+0000","last_became_active":"2023-10-16T10:04:04.685981+0000","last_became_peered":"2023-10-16T10:04:04.685981+0000","last_unstale":"2023-10-16T10:04:04.686623+0000","last_undegraded":"2023-10-16T10:04:04.686623+0000","last_fullsized":"2023-10-16T10:04:04.686623+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:16.110780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667769+0000","last_change":"2023-10-16T10:04:04.684746+0000","last_active":"2023-10-16T10:04:05.667769+0000","last_peered":"2023-10-16T10:04:05.667769+0000","last_clean":"2023-10-16T10:04:05.667769+0000","last_became_active":"2023-10-16T10:04:04.684456+0000","last_became_peered":"2023-10-16T10:04:04.684456+0000","last_unstale":"2023-10-16T10:04:05.667769+0000","last_undegraded":"2023-10-16T10:04:05.667769+0000","last_fullsized":"2023-10-16T10:04:05.667769+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:53.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667896+0000","last_change":"2023-10-16T10:04:04.671456+0000","last_active":"2023-10-16T10:04:05.667896+0000","last_peered":"2023-10-16T10:04:05.667896+0000","last_clean":"2023-10-16T10:04:05.667896+0000","last_became_active":"2023-10-16T10:04:04.670194+0000","last_became_peered":"2023-10-16T10:04:04.670194+0000","last_unstale":"2023-10-16T10:04:05.667896+0000","last_undegraded":"2023-10-16T10:04:05.667896+0000","last_fullsized":"2023-10-16T10:04:05.667896+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:04.243581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.686596+0000","last_change":"2023-10-16T10:04:04.686214+0000","last_active":"2023-10-16T10:04:04.686596+0000","last_peered":"2023-10-16T10:04:04.686596+0000","last_clean":"2023-10-16T10:04:04.686596+0000","last_became_active":"2023-10-16T10:04:04.685824+0000","last_became_peered":"2023-10-16T10:04:04.685824+0000","last_unstale":"2023-10-16T10:04:04.686596+0000","last_undegraded":"2023-10-16T10:04:04.686596+0000","last_fullsized":"2023-10-16T10:04:04.686596+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:54.877564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.668643+0000","last_change":"2023-10-16T10:04:04.667804+0000","last_active":"2023-10-16T10:04:04.668643+0000","last_peered":"2023-10-16T10:04:04.668643+0000","last_clean":"2023-10-16T10:04:04.668643+0000","last_became_active":"2023-10-16T10:04:04.667526+0000","last_became_peered":"2023-10-16T10:04:04.667526+0000","last_unstale":"2023-10-16T10:04:04.668643+0000","last_undegraded":"2023-10-16T10:04:04.668643+0000","last_fullsized":"2023-10-16T10:04:04.668643+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:56.774704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795583+0000","last_change":"2023-10-16T10:04:04.682075+0000","last_active":"2023-10-16T10:04:05.795583+0000","last_peered":"2023-10-16T10:04:05.795583+0000","last_clean":"2023-10-16T10:04:05.795583+0000","last_became_active":"2023-10-16T10:04:04.681804+0000","last_became_peered":"2023-10-16T10:04:04.681804+0000","last_unstale":"2023-10-16T10:04:05.795583+0000","last_undegraded":"2023-10-16T10:04:05.795583+0000","last_fullsized":"2023-10-16T10:04:05.795583+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:49.159622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.677869+0000","last_change":"2023-10-16T10:04:04.674949+0000","last_active":"2023-10-16T10:04:04.677869+0000","last_peered":"2023-10-16T10:04:04.677869+0000","last_clean":"2023-10-16T10:04:04.677869+0000","last_became_active":"2023-10-16T10:04:04.674654+0000","last_became_peered":"2023-10-16T10:04:04.674654+0000","last_unstale":"2023-10-16T10:04:04.677869+0000","last_undegraded":"2023-10-16T10:04:04.677869+0000","last_fullsized":"2023-10-16T10:04:04.677869+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:49:28.649692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682793+0000","last_change":"2023-10-16T10:04:04.681903+0000","last_active":"2023-10-16T10:04:04.682793+0000","last_peered":"2023-10-16T10:04:04.682793+0000","last_clean":"2023-10-16T10:04:04.682793+0000","last_became_active":"2023-10-16T10:04:04.681624+0000","last_became_peered":"2023-10-16T10:04:04.681624+0000","last_unstale":"2023-10-16T10:04:04.682793+0000","last_undegraded":"2023-10-16T10:04:04.682793+0000","last_fullsized":"2023-10-16T10:04:04.682793+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:37.689661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682002+0000","last_change":"2023-10-16T10:04:04.677083+0000","last_active":"2023-10-16T10:04:04.682002+0000","last_peered":"2023-10-16T10:04:04.682002+0000","last_clean":"2023-10-16T10:04:04.682002+0000","last_became_active":"2023-10-16T10:04:04.676806+0000","last_became_peered":"2023-10-16T10:04:04.676806+0000","last_unstale":"2023-10-16T10:04:04.682002+0000","last_undegraded":"2023-10-16T10:04:04.682002+0000","last_fullsized":"2023-10-16T10:04:04.682002+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:49.083338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.677672+0000","last_change":"2023-10-16T10:04:04.674409+0000","last_active":"2023-10-16T10:04:04.677672+0000","last_peered":"2023-10-16T10:04:04.677672+0000","last_clean":"2023-10-16T10:04:04.677672+0000","last_became_active":"2023-10-16T10:04:04.674201+0000","last_became_peered":"2023-10-16T10:04:04.674201+0000","last_unstale":"2023-10-16T10:04:04.677672+0000","last_undegraded":"2023-10-16T10:04:04.677672+0000","last_fullsized":"2023-10-16T10:04:04.677672+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:44.211979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.681770+0000","last_change":"2023-10-16T10:04:04.679952+0000","last_active":"2023-10-16T10:04:04.681770+0000","last_peered":"2023-10-16T10:04:04.681770+0000","last_clean":"2023-10-16T10:04:04.681770+0000","last_became_active":"2023-10-16T10:04:04.679698+0000","last_became_peered":"2023-10-16T10:04:04.679698+0000","last_unstale":"2023-10-16T10:04:04.681770+0000","last_undegraded":"2023-10-16T10:04:04.681770+0000","last_fullsized":"2023-10-16T10:04:04.681770+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:17.004744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.674253+0000","last_change":"2023-10-16T10:04:04.671866+0000","last_active":"2023-10-16T10:04:04.674253+0000","last_peered":"2023-10-16T10:04:04.674253+0000","last_clean":"2023-10-16T10:04:04.674253+0000","last_became_active":"2023-10-16T10:04:04.671614+0000","last_became_peered":"2023-10-16T10:04:04.671614+0000","last_unstale":"2023-10-16T10:04:04.674253+0000","last_undegraded":"2023-10-16T10:04:04.674253+0000","last_fullsized":"2023-10-16T10:04:04.674253+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:23.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668589+0000","last_change":"2023-10-16T10:04:04.671623+0000","last_active":"2023-10-16T10:04:05.668589+0000","last_peered":"2023-10-16T10:04:05.668589+0000","last_clean":"2023-10-16T10:04:05.668589+0000","last_became_active":"2023-10-16T10:04:04.671338+0000","last_became_peered":"2023-10-16T10:04:04.671338+0000","last_unstale":"2023-10-16T10:04:05.668589+0000","last_undegraded":"2023-10-16T10:04:05.668589+0000","last_fullsized":"2023-10-16T10:04:05.668589+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:50.556411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669520+0000","last_change":"2023-10-16T10:04:04.670291+0000","last_active":"2023-10-16T10:04:05.669520+0000","last_peered":"2023-10-16T10:04:05.669520+0000","last_clean":"2023-10-16T10:04:05.669520+0000","last_became_active":"2023-10-16T10:04:04.670012+0000","last_became_peered":"2023-10-16T10:04:04.670012+0000","last_unstale":"2023-10-16T10:04:05.669520+0000","last_undegraded":"2023-10-16T10:04:05.669520+0000","last_fullsized":"2023-10-16T10:04:05.669520+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:21.671698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668734+0000","last_change":"2023-10-16T10:04:04.675490+0000","last_active":"2023-10-16T10:04:05.668734+0000","last_peered":"2023-10-16T10:04:05.668734+0000","last_clean":"2023-10-16T10:04:05.668734+0000","last_became_active":"2023-10-16T10:04:04.675146+0000","last_became_peered":"2023-10-16T10:04:04.675146+0000","last_unstale":"2023-10-16T10:04:05.668734+0000","last_undegraded":"2023-10-16T10:04:05.668734+0000","last_fullsized":"2023-10-16T10:04:05.668734+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:10.171478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795473+0000","last_change":"2023-10-16T10:04:04.675392+0000","last_active":"2023-10-16T10:04:05.795473+0000","last_peered":"2023-10-16T10:04:05.795473+0000","last_clean":"2023-10-16T10:04:05.795473+0000","last_became_active":"2023-10-16T10:04:04.675118+0000","last_became_peered":"2023-10-16T10:04:04.675118+0000","last_unstale":"2023-10-16T10:04:05.795473+0000","last_undegraded":"2023-10-16T10:04:05.795473+0000","last_fullsized":"2023-10-16T10:04:05.795473+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:13.137832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.683023+0000","last_change":"2023-10-16T10:04:04.682461+0000","last_active":"2023-10-16T10:04:04.683023+0000","last_peered":"2023-10-16T10:04:04.683023+0000","last_clean":"2023-10-16T10:04:04.683023+0000","last_became_active":"2023-10-16T10:04:04.682162+0000","last_became_peered":"2023-10-16T10:04:04.682162+0000","last_unstale":"2023-10-16T10:04:04.683023+0000","last_undegraded":"2023-10-16T10:04:04.683023+0000","last_fullsized":"2023-10-16T10:04:04.683023+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:02.495908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795574+0000","last_change":"2023-10-16T10:04:04.679961+0000","last_active":"2023-10-16T10:04:05.795574+0000","last_peered":"2023-10-16T10:04:05.795574+0000","last_clean":"2023-10-16T10:04:05.795574+0000","last_became_active":"2023-10-16T10:04:04.678205+0000","last_became_peered":"2023-10-16T10:04:04.678205+0000","last_unstale":"2023-10-16T10:04:05.795574+0000","last_undegraded":"2023-10-16T10:04:05.795574+0000","last_fullsized":"2023-10-16T10:04:05.795574+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:31.215405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682560+0000","last_change":"2023-10-16T10:04:04.681519+0000","last_active":"2023-10-16T10:04:04.682560+0000","last_peered":"2023-10-16T10:04:04.682560+0000","last_clean":"2023-10-16T10:04:04.682560+0000","last_became_active":"2023-10-16T10:04:04.681228+0000","last_became_peered":"2023-10-16T10:04:04.681228+0000","last_unstale":"2023-10-16T10:04:04.682560+0000","last_undegraded":"2023-10-16T10:04:04.682560+0000","last_fullsized":"2023-10-16T10:04:04.682560+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:20.563663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.690739+0000","last_change":"2023-10-16T10:04:04.688549+0000","last_active":"2023-10-16T10:04:04.690739+0000","last_peered":"2023-10-16T10:04:04.690739+0000","last_clean":"2023-10-16T10:04:04.690739+0000","last_became_active":"2023-10-16T10:04:04.688283+0000","last_became_peered":"2023-10-16T10:04:04.688283+0000","last_unstale":"2023-10-16T10:04:04.690739+0000","last_undegraded":"2023-10-16T10:04:04.690739+0000","last_fullsized":"2023-10-16T10:04:04.690739+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:04.243581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.689160+0000","last_change":"2023-10-16T10:04:04.686692+0000","last_active":"2023-10-16T10:04:04.689160+0000","last_peered":"2023-10-16T10:04:04.689160+0000","last_clean":"2023-10-16T10:04:04.689160+0000","last_became_active":"2023-10-16T10:04:04.686350+0000","last_became_peered":"2023-10-16T10:04:04.686350+0000","last_unstale":"2023-10-16T10:04:04.689160+0000","last_undegraded":"2023-10-16T10:04:04.689160+0000","last_fullsized":"2023-10-16T10:04:04.689160+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:59.022801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669390+0000","last_change":"2023-10-16T10:04:04.672514+0000","last_active":"2023-10-16T10:04:05.669390+0000","last_peered":"2023-10-16T10:04:05.669390+0000","last_clean":"2023-10-16T10:04:05.669390+0000","last_became_active":"2023-10-16T10:04:04.671994+0000","last_became_peered":"2023-10-16T10:04:04.671994+0000","last_unstale":"2023-10-16T10:04:05.669390+0000","last_undegraded":"2023-10-16T10:04:05.669390+0000","last_fullsized":"2023-10-16T10:04:05.669390+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.659825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.668598+0000","last_change":"2023-10-16T10:04:04.667799+0000","last_active":"2023-10-16T10:04:04.668598+0000","last_peered":"2023-10-16T10:04:04.668598+0000","last_clean":"2023-10-16T10:04:04.668598+0000","last_became_active":"2023-10-16T10:04:04.667500+0000","last_became_peered":"2023-10-16T10:04:04.667500+0000","last_unstale":"2023-10-16T10:04:04.668598+0000","last_undegraded":"2023-10-16T10:04:04.668598+0000","last_fullsized":"2023-10-16T10:04:04.668598+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:10.931058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.688270+0000","last_change":"2023-10-16T10:04:04.683690+0000","last_active":"2023-10-16T10:04:04.688270+0000","last_peered":"2023-10-16T10:04:04.688270+0000","last_clean":"2023-10-16T10:04:04.688270+0000","last_became_active":"2023-10-16T10:04:04.683406+0000","last_became_peered":"2023-10-16T10:04:04.683406+0000","last_unstale":"2023-10-16T10:04:04.688270+0000","last_undegraded":"2023-10-16T10:04:04.688270+0000","last_fullsized":"2023-10-16T10:04:04.688270+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:12.851592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.685072+0000","last_change":"2023-10-16T10:04:04.682211+0000","last_active":"2023-10-16T10:04:04.685072+0000","last_peered":"2023-10-16T10:04:04.685072+0000","last_clean":"2023-10-16T10:04:04.685072+0000","last_became_active":"2023-10-16T10:04:04.681919+0000","last_became_peered":"2023-10-16T10:04:04.681919+0000","last_unstale":"2023-10-16T10:04:04.685072+0000","last_undegraded":"2023-10-16T10:04:04.685072+0000","last_fullsized":"2023-10-16T10:04:04.685072+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:38.754410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668029+0000","last_change":"2023-10-16T10:04:04.674400+0000","last_active":"2023-10-16T10:04:05.668029+0000","last_peered":"2023-10-16T10:04:05.668029+0000","last_clean":"2023-10-16T10:04:05.668029+0000","last_became_active":"2023-10-16T10:04:04.674135+0000","last_became_peered":"2023-10-16T10:04:04.674135+0000","last_unstale":"2023-10-16T10:04:05.668029+0000","last_undegraded":"2023-10-16T10:04:05.668029+0000","last_fullsized":"2023-10-16T10:04:05.668029+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:43.191440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.671785+0000","last_change":"2023-10-16T10:04:04.670635+0000","last_active":"2023-10-16T10:04:04.671785+0000","last_peered":"2023-10-16T10:04:04.671785+0000","last_clean":"2023-10-16T10:04:04.671785+0000","last_became_active":"2023-10-16T10:04:04.670313+0000","last_became_peered":"2023-10-16T10:04:04.670313+0000","last_unstale":"2023-10-16T10:04:04.671785+0000","last_undegraded":"2023-10-16T10:04:04.671785+0000","last_fullsized":"2023-10-16T10:04:04.671785+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:27.536546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669192+0000","last_change":"2023-10-16T10:04:04.671537+0000","last_active":"2023-10-16T10:04:05.669192+0000","last_peered":"2023-10-16T10:04:05.669192+0000","last_clean":"2023-10-16T10:04:05.669192+0000","last_became_active":"2023-10-16T10:04:04.671185+0000","last_became_peered":"2023-10-16T10:04:04.671185+0000","last_unstale":"2023-10-16T10:04:05.669192+0000","last_undegraded":"2023-10-16T10:04:05.669192+0000","last_fullsized":"2023-10-16T10:04:05.669192+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:54.508051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"152'4","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695664+0000","last_change":"2023-10-16T10:04:03.670215+0000","last_active":"2023-10-16T10:04:04.695664+0000","last_peered":"2023-10-16T10:04:04.695664+0000","last_clean":"2023-10-16T10:04:04.695664+0000","last_became_active":"2023-10-16T10:04:03.669921+0000","last_became_peered":"2023-10-16T10:04:03.669921+0000","last_unstale":"2023-10-16T10:04:04.695664+0000","last_undegraded":"2023-10-16T10:04:04.695664+0000","last_fullsized":"2023-10-16T10:04:04.695664+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:42.184281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682845+0000","last_change":"2023-10-16T10:04:04.681943+0000","last_active":"2023-10-16T10:04:04.682845+0000","last_peered":"2023-10-16T10:04:04.682845+0000","last_clean":"2023-10-16T10:04:04.682845+0000","last_became_active":"2023-10-16T10:04:04.681654+0000","last_became_peered":"2023-10-16T10:04:04.681654+0000","last_unstale":"2023-10-16T10:04:04.682845+0000","last_undegraded":"2023-10-16T10:04:04.682845+0000","last_fullsized":"2023-10-16T10:04:04.682845+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:38.449242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668741+0000","last_change":"2023-10-16T10:04:03.667947+0000","last_active":"2023-10-16T10:04:05.668741+0000","last_peered":"2023-10-16T10:04:05.668741+0000","last_clean":"2023-10-16T10:04:05.668741+0000","last_became_active":"2023-10-16T10:04:03.667637+0000","last_became_peered":"2023-10-16T10:04:03.667637+0000","last_unstale":"2023-10-16T10:04:05.668741+0000","last_undegraded":"2023-10-16T10:04:05.668741+0000","last_fullsized":"2023-10-16T10:04:05.668741+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:55.767546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795633+0000","last_change":"2023-10-16T10:04:04.678888+0000","last_active":"2023-10-16T10:04:05.795633+0000","last_peered":"2023-10-16T10:04:05.795633+0000","last_clean":"2023-10-16T10:04:05.795633+0000","last_became_active":"2023-10-16T10:04:04.676457+0000","last_became_peered":"2023-10-16T10:04:04.676457+0000","last_unstale":"2023-10-16T10:04:05.795633+0000","last_undegraded":"2023-10-16T10:04:05.795633+0000","last_fullsized":"2023-10-16T10:04:05.795633+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:27.536546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"152'2","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669409+0000","last_change":"2023-10-16T10:04:03.665596+0000","last_active":"2023-10-16T10:04:05.669409+0000","last_peered":"2023-10-16T10:04:05.669409+0000","last_clean":"2023-10-16T10:04:05.669409+0000","last_became_active":"2023-10-16T10:04:03.665287+0000","last_became_peered":"2023-10-16T10:04:03.665287+0000","last_unstale":"2023-10-16T10:04:05.669409+0000","last_undegraded":"2023-10-16T10:04:05.669409+0000","last_fullsized":"2023-10-16T10:04:05.669409+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:58:36.866060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682613+0000","last_change":"2023-10-16T10:04:04.680200+0000","last_active":"2023-10-16T10:04:04.682613+0000","last_peered":"2023-10-16T10:04:04.682613+0000","last_clean":"2023-10-16T10:04:04.682613+0000","last_became_active":"2023-10-16T10:04:04.679939+0000","last_became_peered":"2023-10-16T10:04:04.679939+0000","last_unstale":"2023-10-16T10:04:04.682613+0000","last_undegraded":"2023-10-16T10:04:04.682613+0000","last_fullsized":"2023-10-16T10:04:04.682613+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:41.949411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"152'2","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667847+0000","last_change":"2023-10-16T10:04:03.666061+0000","last_active":"2023-10-16T10:04:05.667847+0000","last_peered":"2023-10-16T10:04:05.667847+0000","last_clean":"2023-10-16T10:04:05.667847+0000","last_became_active":"2023-10-16T10:04:03.665741+0000","last_became_peered":"2023-10-16T10:04:03.665741+0000","last_unstale":"2023-10-16T10:04:05.667847+0000","last_undegraded":"2023-10-16T10:04:05.667847+0000","last_fullsized":"2023-10-16T10:04:05.667847+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:05:48.559362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668420+0000","last_change":"2023-10-16T10:04:04.675007+0000","last_active":"2023-10-16T10:04:05.668420+0000","last_peered":"2023-10-16T10:04:05.668420+0000","last_clean":"2023-10-16T10:04:05.668420+0000","last_became_active":"2023-10-16T10:04:04.674725+0000","last_became_peered":"2023-10-16T10:04:04.674725+0000","last_unstale":"2023-10-16T10:04:05.668420+0000","last_undegraded":"2023-10-16T10:04:05.668420+0000","last_fullsized":"2023-10-16T10:04:05.668420+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:37.689661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669129+0000","last_change":"2023-10-16T10:04:03.668304+0000","last_active":"2023-10-16T10:04:05.669129+0000","last_peered":"2023-10-16T10:04:05.669129+0000","last_clean":"2023-10-16T10:04:05.669129+0000","last_became_active":"2023-10-16T10:04:03.668036+0000","last_became_peered":"2023-10-16T10:04:03.668036+0000","last_unstale":"2023-10-16T10:04:05.669129+0000","last_undegraded":"2023-10-16T10:04:05.669129+0000","last_fullsized":"2023-10-16T10:04:05.669129+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:58.015643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668365+0000","last_change":"2023-10-16T10:04:04.670372+0000","last_active":"2023-10-16T10:04:05.668365+0000","last_peered":"2023-10-16T10:04:05.668365+0000","last_clean":"2023-10-16T10:04:05.668365+0000","last_became_active":"2023-10-16T10:04:04.670097+0000","last_became_peered":"2023-10-16T10:04:04.670097+0000","last_unstale":"2023-10-16T10:04:05.668365+0000","last_undegraded":"2023-10-16T10:04:05.668365+0000","last_fullsized":"2023-10-16T10:04:05.668365+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:40.048526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668400+0000","last_change":"2023-10-16T10:04:03.663804+0000","last_active":"2023-10-16T10:04:05.668400+0000","last_peered":"2023-10-16T10:04:05.668400+0000","last_clean":"2023-10-16T10:04:05.668400+0000","last_became_active":"2023-10-16T10:04:03.663643+0000","last_became_peered":"2023-10-16T10:04:03.663643+0000","last_unstale":"2023-10-16T10:04:05.668400+0000","last_undegraded":"2023-10-16T10:04:05.668400+0000","last_fullsized":"2023-10-16T10:04:05.668400+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:57.305053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668689+0000","last_change":"2023-10-16T10:04:04.675744+0000","last_active":"2023-10-16T10:04:05.668689+0000","last_peered":"2023-10-16T10:04:05.668689+0000","last_clean":"2023-10-16T10:04:05.668689+0000","last_became_active":"2023-10-16T10:04:04.675483+0000","last_became_peered":"2023-10-16T10:04:04.675483+0000","last_unstale":"2023-10-16T10:04:05.668689+0000","last_undegraded":"2023-10-16T10:04:05.668689+0000","last_fullsized":"2023-10-16T10:04:05.668689+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:26:21.671698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.667076+0000","last_change":"2023-10-16T10:04:03.666452+0000","last_active":"2023-10-16T10:04:04.667076+0000","last_peered":"2023-10-16T10:04:04.667076+0000","last_clean":"2023-10-16T10:04:04.667076+0000","last_became_active":"2023-10-16T10:04:03.666179+0000","last_became_peered":"2023-10-16T10:04:03.666179+0000","last_unstale":"2023-10-16T10:04:04.667076+0000","last_undegraded":"2023-10-16T10:04:04.667076+0000","last_fullsized":"2023-10-16T10:04:04.667076+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:49.610888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668639+0000","last_change":"2023-10-16T10:04:04.676179+0000","last_active":"2023-10-16T10:04:05.668639+0000","last_peered":"2023-10-16T10:04:05.668639+0000","last_clean":"2023-10-16T10:04:05.668639+0000","last_became_active":"2023-10-16T10:04:04.675871+0000","last_became_peered":"2023-10-16T10:04:04.675871+0000","last_unstale":"2023-10-16T10:04:05.668639+0000","last_undegraded":"2023-10-16T10:04:05.668639+0000","last_fullsized":"2023-10-16T10:04:05.668639+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:50.556411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.675499+0000","last_change":"2023-10-16T10:04:03.664965+0000","last_active":"2023-10-16T10:04:04.675499+0000","last_peered":"2023-10-16T10:04:04.675499+0000","last_clean":"2023-10-16T10:04:04.675499+0000","last_became_active":"2023-10-16T10:04:03.664686+0000","last_became_peered":"2023-10-16T10:04:03.664686+0000","last_unstale":"2023-10-16T10:04:04.675499+0000","last_undegraded":"2023-10-16T10:04:04.675499+0000","last_fullsized":"2023-10-16T10:04:04.675499+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:22.950332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.685168+0000","last_change":"2023-10-16T10:04:04.683631+0000","last_active":"2023-10-16T10:04:04.685168+0000","last_peered":"2023-10-16T10:04:04.685168+0000","last_clean":"2023-10-16T10:04:04.685168+0000","last_became_active":"2023-10-16T10:04:04.683468+0000","last_became_peered":"2023-10-16T10:04:04.683468+0000","last_unstale":"2023-10-16T10:04:04.685168+0000","last_undegraded":"2023-10-16T10:04:04.685168+0000","last_fullsized":"2023-10-16T10:04:04.685168+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:33.742220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795605+0000","last_change":"2023-10-16T10:04:03.666216+0000","last_active":"2023-10-16T10:04:05.795605+0000","last_peered":"2023-10-16T10:04:05.795605+0000","last_clean":"2023-10-16T10:04:05.795605+0000","last_became_active":"2023-10-16T10:04:03.665934+0000","last_became_peered":"2023-10-16T10:04:03.665934+0000","last_unstale":"2023-10-16T10:04:05.795605+0000","last_undegraded":"2023-10-16T10:04:05.795605+0000","last_fullsized":"2023-10-16T10:04:05.795605+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:57.305053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.674778+0000","last_change":"2023-10-16T10:04:04.672949+0000","last_active":"2023-10-16T10:04:04.674778+0000","last_peered":"2023-10-16T10:04:04.674778+0000","last_clean":"2023-10-16T10:04:04.674778+0000","last_became_active":"2023-10-16T10:04:04.672676+0000","last_became_peered":"2023-10-16T10:04:04.672676+0000","last_unstale":"2023-10-16T10:04:04.674778+0000","last_undegraded":"2023-10-16T10:04:04.674778+0000","last_fullsized":"2023-10-16T10:04:04.674778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:54.508051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.664863+0000","last_change":"2023-10-16T10:04:03.669827+0000","last_active":"2023-10-16T10:04:04.664863+0000","last_peered":"2023-10-16T10:04:04.664863+0000","last_clean":"2023-10-16T10:04:04.664863+0000","last_became_active":"2023-10-16T10:04:03.669574+0000","last_became_peered":"2023-10-16T10:04:03.669574+0000","last_unstale":"2023-10-16T10:04:04.664863+0000","last_undegraded":"2023-10-16T10:04:04.664863+0000","last_fullsized":"2023-10-16T10:04:04.664863+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:18:05.685692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.689010+0000","last_change":"2023-10-16T10:04:04.684881+0000","last_active":"2023-10-16T10:04:04.689010+0000","last_peered":"2023-10-16T10:04:04.689010+0000","last_clean":"2023-10-16T10:04:04.689010+0000","last_became_active":"2023-10-16T10:04:04.684601+0000","last_became_peered":"2023-10-16T10:04:04.684601+0000","last_unstale":"2023-10-16T10:04:04.689010+0000","last_undegraded":"2023-10-16T10:04:04.689010+0000","last_fullsized":"2023-10-16T10:04:04.689010+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:38.109699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"152'3","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669512+0000","last_change":"2023-10-16T10:04:03.668456+0000","last_active":"2023-10-16T10:04:05.669512+0000","last_peered":"2023-10-16T10:04:05.669512+0000","last_clean":"2023-10-16T10:04:05.669512+0000","last_became_active":"2023-10-16T10:04:03.668160+0000","last_became_peered":"2023-10-16T10:04:03.668160+0000","last_unstale":"2023-10-16T10:04:05.669512+0000","last_undegraded":"2023-10-16T10:04:05.669512+0000","last_fullsized":"2023-10-16T10:04:05.669512+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:44.090615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.683789+0000","last_change":"2023-10-16T10:04:04.681525+0000","last_active":"2023-10-16T10:04:04.683789+0000","last_peered":"2023-10-16T10:04:04.683789+0000","last_clean":"2023-10-16T10:04:04.683789+0000","last_became_active":"2023-10-16T10:04:04.681020+0000","last_became_peered":"2023-10-16T10:04:04.681020+0000","last_unstale":"2023-10-16T10:04:04.683789+0000","last_undegraded":"2023-10-16T10:04:04.683789+0000","last_fullsized":"2023-10-16T10:04:04.683789+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:23:59.419140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"152'6","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668058+0000","last_change":"2023-10-16T10:04:03.668664+0000","last_active":"2023-10-16T10:04:05.668058+0000","last_peered":"2023-10-16T10:04:05.668058+0000","last_clean":"2023-10-16T10:04:05.668058+0000","last_became_active":"2023-10-16T10:04:03.666911+0000","last_became_peered":"2023-10-16T10:04:03.666911+0000","last_unstale":"2023-10-16T10:04:05.668058+0000","last_undegraded":"2023-10-16T10:04:05.668058+0000","last_fullsized":"2023-10-16T10:04:05.668058+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:07.288560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.678782+0000","last_change":"2023-10-16T10:04:04.677240+0000","last_active":"2023-10-16T10:04:04.678782+0000","last_peered":"2023-10-16T10:04:04.678782+0000","last_clean":"2023-10-16T10:04:04.678782+0000","last_became_active":"2023-10-16T10:04:04.676959+0000","last_became_peered":"2023-10-16T10:04:04.676959+0000","last_unstale":"2023-10-16T10:04:04.678782+0000","last_undegraded":"2023-10-16T10:04:04.678782+0000","last_fullsized":"2023-10-16T10:04:04.678782+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:22.835248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"152'2","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.694325+0000","last_change":"2023-10-16T10:04:03.670480+0000","last_active":"2023-10-16T10:04:04.694325+0000","last_peered":"2023-10-16T10:04:04.694325+0000","last_clean":"2023-10-16T10:04:04.694325+0000","last_became_active":"2023-10-16T10:04:03.670317+0000","last_became_peered":"2023-10-16T10:04:03.670317+0000","last_unstale":"2023-10-16T10:04:04.694325+0000","last_undegraded":"2023-10-16T10:04:04.694325+0000","last_fullsized":"2023-10-16T10:04:04.694325+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:26.719464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668262+0000","last_change":"2023-10-16T10:04:04.686196+0000","last_active":"2023-10-16T10:04:05.668262+0000","last_peered":"2023-10-16T10:04:05.668262+0000","last_clean":"2023-10-16T10:04:05.668262+0000","last_became_active":"2023-10-16T10:04:04.685908+0000","last_became_peered":"2023-10-16T10:04:04.685908+0000","last_unstale":"2023-10-16T10:04:05.668262+0000","last_undegraded":"2023-10-16T10:04:05.668262+0000","last_fullsized":"2023-10-16T10:04:05.668262+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:04.733721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.675897+0000","last_change":"2023-10-16T10:04:03.667625+0000","last_active":"2023-10-16T10:04:04.675897+0000","last_peered":"2023-10-16T10:04:04.675897+0000","last_clean":"2023-10-16T10:04:04.675897+0000","last_became_active":"2023-10-16T10:04:03.667335+0000","last_became_peered":"2023-10-16T10:04:03.667335+0000","last_unstale":"2023-10-16T10:04:04.675897+0000","last_undegraded":"2023-10-16T10:04:04.675897+0000","last_fullsized":"2023-10-16T10:04:04.675897+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:57.114434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668900+0000","last_change":"2023-10-16T10:04:04.676237+0000","last_active":"2023-10-16T10:04:05.668900+0000","last_peered":"2023-10-16T10:04:05.668900+0000","last_clean":"2023-10-16T10:04:05.668900+0000","last_became_active":"2023-10-16T10:04:04.675981+0000","last_became_peered":"2023-10-16T10:04:04.675981+0000","last_unstale":"2023-10-16T10:04:05.668900+0000","last_undegraded":"2023-10-16T10:04:05.668900+0000","last_fullsized":"2023-10-16T10:04:05.668900+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:51.953692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"152'8","reported_seq":19,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.696626+0000","last_change":"2023-10-16T10:04:03.660640+0000","last_active":"2023-10-16T10:04:04.696626+0000","last_peered":"2023-10-16T10:04:04.696626+0000","last_clean":"2023-10-16T10:04:04.696626+0000","last_became_active":"2023-10-16T10:04:03.660385+0000","last_became_peered":"2023-10-16T10:04:03.660385+0000","last_unstale":"2023-10-16T10:04:04.696626+0000","last_undegraded":"2023-10-16T10:04:04.696626+0000","last_fullsized":"2023-10-16T10:04:04.696626+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:26.057348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.684269+0000","last_change":"2023-10-16T10:04:04.678632+0000","last_active":"2023-10-16T10:04:04.684269+0000","last_peered":"2023-10-16T10:04:04.684269+0000","last_clean":"2023-10-16T10:04:04.684269+0000","last_became_active":"2023-10-16T10:04:04.678350+0000","last_became_peered":"2023-10-16T10:04:04.678350+0000","last_unstale":"2023-10-16T10:04:04.684269+0000","last_undegraded":"2023-10-16T10:04:04.684269+0000","last_fullsized":"2023-10-16T10:04:04.684269+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:34.618183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"152'1","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795732+0000","last_change":"2023-10-16T10:04:03.664840+0000","last_active":"2023-10-16T10:04:05.795732+0000","last_peered":"2023-10-16T10:04:05.795732+0000","last_clean":"2023-10-16T10:04:05.795732+0000","last_became_active":"2023-10-16T10:04:03.664560+0000","last_became_peered":"2023-10-16T10:04:03.664560+0000","last_unstale":"2023-10-16T10:04:05.795732+0000","last_undegraded":"2023-10-16T10:04:05.795732+0000","last_fullsized":"2023-10-16T10:04:05.795732+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:19.652102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.667934+0000","last_change":"2023-10-16T10:04:04.672154+0000","last_active":"2023-10-16T10:04:05.667934+0000","last_peered":"2023-10-16T10:04:05.667934+0000","last_clean":"2023-10-16T10:04:05.667934+0000","last_became_active":"2023-10-16T10:04:04.671855+0000","last_became_peered":"2023-10-16T10:04:04.671855+0000","last_unstale":"2023-10-16T10:04:05.667934+0000","last_undegraded":"2023-10-16T10:04:05.667934+0000","last_fullsized":"2023-10-16T10:04:05.667934+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:56.774704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"152'2","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695803+0000","last_change":"2023-10-16T10:04:03.670010+0000","last_active":"2023-10-16T10:04:04.695803+0000","last_peered":"2023-10-16T10:04:04.695803+0000","last_clean":"2023-10-16T10:04:04.695803+0000","last_became_active":"2023-10-16T10:04:03.669737+0000","last_became_peered":"2023-10-16T10:04:03.669737+0000","last_unstale":"2023-10-16T10:04:04.695803+0000","last_undegraded":"2023-10-16T10:04:04.695803+0000","last_fullsized":"2023-10-16T10:04:04.695803+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:54.712464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668031+0000","last_change":"2023-10-16T10:04:04.671117+0000","last_active":"2023-10-16T10:04:05.668031+0000","last_peered":"2023-10-16T10:04:05.668031+0000","last_clean":"2023-10-16T10:04:05.668031+0000","last_became_active":"2023-10-16T10:04:04.670865+0000","last_became_peered":"2023-10-16T10:04:04.670865+0000","last_unstale":"2023-10-16T10:04:05.668031+0000","last_undegraded":"2023-10-16T10:04:05.668031+0000","last_fullsized":"2023-10-16T10:04:05.668031+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:22.835248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"152'1","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695308+0000","last_change":"2023-10-16T10:04:03.666149+0000","last_active":"2023-10-16T10:04:04.695308+0000","last_peered":"2023-10-16T10:04:04.695308+0000","last_clean":"2023-10-16T10:04:04.695308+0000","last_became_active":"2023-10-16T10:04:03.665863+0000","last_became_peered":"2023-10-16T10:04:03.665863+0000","last_unstale":"2023-10-16T10:04:04.695308+0000","last_undegraded":"2023-10-16T10:04:04.695308+0000","last_fullsized":"2023-10-16T10:04:04.695308+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:03.236423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.683221+0000","last_change":"2023-10-16T10:04:04.681427+0000","last_active":"2023-10-16T10:04:04.683221+0000","last_peered":"2023-10-16T10:04:04.683221+0000","last_clean":"2023-10-16T10:04:04.683221+0000","last_became_active":"2023-10-16T10:04:04.681124+0000","last_became_peered":"2023-10-16T10:04:04.681124+0000","last_unstale":"2023-10-16T10:04:04.683221+0000","last_undegraded":"2023-10-16T10:04:04.683221+0000","last_fullsized":"2023-10-16T10:04:04.683221+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:53.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668732+0000","last_change":"2023-10-16T10:04:03.668680+0000","last_active":"2023-10-16T10:04:05.668732+0000","last_peered":"2023-10-16T10:04:05.668732+0000","last_clean":"2023-10-16T10:04:05.668732+0000","last_became_active":"2023-10-16T10:04:03.668339+0000","last_became_peered":"2023-10-16T10:04:03.668339+0000","last_unstale":"2023-10-16T10:04:05.668732+0000","last_undegraded":"2023-10-16T10:04:05.668732+0000","last_fullsized":"2023-10-16T10:04:05.668732+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:21.828090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668646+0000","last_change":"2023-10-16T10:04:04.677663+0000","last_active":"2023-10-16T10:04:05.668646+0000","last_peered":"2023-10-16T10:04:05.668646+0000","last_clean":"2023-10-16T10:04:05.668646+0000","last_became_active":"2023-10-16T10:04:04.677417+0000","last_became_peered":"2023-10-16T10:04:04.677417+0000","last_unstale":"2023-10-16T10:04:05.668646+0000","last_undegraded":"2023-10-16T10:04:05.668646+0000","last_fullsized":"2023-10-16T10:04:05.668646+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.659825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"152'2","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.669269+0000","last_change":"2023-10-16T10:04:03.664473+0000","last_active":"2023-10-16T10:04:05.669269+0000","last_peered":"2023-10-16T10:04:05.669269+0000","last_clean":"2023-10-16T10:04:05.669269+0000","last_became_active":"2023-10-16T10:04:03.664239+0000","last_became_peered":"2023-10-16T10:04:03.664239+0000","last_unstale":"2023-10-16T10:04:05.669269+0000","last_undegraded":"2023-10-16T10:04:05.669269+0000","last_fullsized":"2023-10-16T10:04:05.669269+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:54.712464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682519+0000","last_change":"2023-10-16T10:04:04.677507+0000","last_active":"2023-10-16T10:04:04.682519+0000","last_peered":"2023-10-16T10:04:04.682519+0000","last_clean":"2023-10-16T10:04:04.682519+0000","last_became_active":"2023-10-16T10:04:04.677239+0000","last_became_peered":"2023-10-16T10:04:04.677239+0000","last_unstale":"2023-10-16T10:04:04.682519+0000","last_undegraded":"2023-10-16T10:04:04.682519+0000","last_fullsized":"2023-10-16T10:04:04.682519+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:40:17.157813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.673607+0000","last_change":"2023-10-16T10:04:03.668707+0000","last_active":"2023-10-16T10:04:04.673607+0000","last_peered":"2023-10-16T10:04:04.673607+0000","last_clean":"2023-10-16T10:04:04.673607+0000","last_became_active":"2023-10-16T10:04:03.667936+0000","last_became_peered":"2023-10-16T10:04:03.667936+0000","last_unstale":"2023-10-16T10:04:04.673607+0000","last_undegraded":"2023-10-16T10:04:04.673607+0000","last_fullsized":"2023-10-16T10:04:04.673607+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:28.886145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668740+0000","last_change":"2023-10-16T10:04:04.687193+0000","last_active":"2023-10-16T10:04:05.668740+0000","last_peered":"2023-10-16T10:04:05.668740+0000","last_clean":"2023-10-16T10:04:05.668740+0000","last_became_active":"2023-10-16T10:04:04.686926+0000","last_became_peered":"2023-10-16T10:04:04.686926+0000","last_unstale":"2023-10-16T10:04:05.668740+0000","last_undegraded":"2023-10-16T10:04:05.668740+0000","last_fullsized":"2023-10-16T10:04:05.668740+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:13.703024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668174+0000","last_change":"2023-10-16T10:04:03.672658+0000","last_active":"2023-10-16T10:04:05.668174+0000","last_peered":"2023-10-16T10:04:05.668174+0000","last_clean":"2023-10-16T10:04:05.668174+0000","last_became_active":"2023-10-16T10:04:03.672432+0000","last_became_peered":"2023-10-16T10:04:03.672432+0000","last_unstale":"2023-10-16T10:04:05.668174+0000","last_undegraded":"2023-10-16T10:04:05.668174+0000","last_fullsized":"2023-10-16T10:04:05.668174+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:26.529388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795636+0000","last_change":"2023-10-16T10:04:04.688730+0000","last_active":"2023-10-16T10:04:05.795636+0000","last_peered":"2023-10-16T10:04:05.795636+0000","last_clean":"2023-10-16T10:04:05.795636+0000","last_became_active":"2023-10-16T10:04:04.688438+0000","last_became_peered":"2023-10-16T10:04:04.688438+0000","last_unstale":"2023-10-16T10:04:05.795636+0000","last_undegraded":"2023-10-16T10:04:05.795636+0000","last_fullsized":"2023-10-16T10:04:05.795636+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:53.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.668878+0000","last_change":"2023-10-16T10:04:03.668943+0000","last_active":"2023-10-16T10:04:04.668878+0000","last_peered":"2023-10-16T10:04:04.668878+0000","last_clean":"2023-10-16T10:04:04.668878+0000","last_became_active":"2023-10-16T10:04:03.668665+0000","last_became_peered":"2023-10-16T10:04:03.668665+0000","last_unstale":"2023-10-16T10:04:04.668878+0000","last_undegraded":"2023-10-16T10:04:04.668878+0000","last_fullsized":"2023-10-16T10:04:04.668878+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:43:50.425071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668434+0000","last_change":"2023-10-16T10:04:04.676104+0000","last_active":"2023-10-16T10:04:05.668434+0000","last_peered":"2023-10-16T10:04:05.668434+0000","last_clean":"2023-10-16T10:04:05.668434+0000","last_became_active":"2023-10-16T10:04:04.675498+0000","last_became_peered":"2023-10-16T10:04:04.675498+0000","last_unstale":"2023-10-16T10:04:05.668434+0000","last_undegraded":"2023-10-16T10:04:05.668434+0000","last_fullsized":"2023-10-16T10:04:05.668434+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:04.733721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"152'2","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.695643+0000","last_change":"2023-10-16T10:04:03.669278+0000","last_active":"2023-10-16T10:04:04.695643+0000","last_peered":"2023-10-16T10:04:04.695643+0000","last_clean":"2023-10-16T10:04:04.695643+0000","last_became_active":"2023-10-16T10:04:03.669015+0000","last_became_peered":"2023-10-16T10:04:03.669015+0000","last_unstale":"2023-10-16T10:04:04.695643+0000","last_undegraded":"2023-10-16T10:04:04.695643+0000","last_fullsized":"2023-10-16T10:04:04.695643+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.307298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.688151+0000","last_change":"2023-10-16T10:04:04.683596+0000","last_active":"2023-10-16T10:04:04.688151+0000","last_peered":"2023-10-16T10:04:04.688151+0000","last_clean":"2023-10-16T10:04:04.688151+0000","last_became_active":"2023-10-16T10:04:04.683329+0000","last_became_peered":"2023-10-16T10:04:04.683329+0000","last_unstale":"2023-10-16T10:04:04.688151+0000","last_undegraded":"2023-10-16T10:04:04.688151+0000","last_fullsized":"2023-10-16T10:04:04.688151+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:54.877564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668443+0000","last_change":"2023-10-16T10:04:03.667617+0000","last_active":"2023-10-16T10:04:05.668443+0000","last_peered":"2023-10-16T10:04:05.668443+0000","last_clean":"2023-10-16T10:04:05.668443+0000","last_became_active":"2023-10-16T10:04:03.667346+0000","last_became_peered":"2023-10-16T10:04:03.667346+0000","last_unstale":"2023-10-16T10:04:05.668443+0000","last_undegraded":"2023-10-16T10:04:05.668443+0000","last_fullsized":"2023-10-16T10:04:05.668443+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:37.747252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668219+0000","last_change":"2023-10-16T10:04:04.686263+0000","last_active":"2023-10-16T10:04:05.668219+0000","last_peered":"2023-10-16T10:04:05.668219+0000","last_clean":"2023-10-16T10:04:05.668219+0000","last_became_active":"2023-10-16T10:04:04.685980+0000","last_became_peered":"2023-10-16T10:04:04.685980+0000","last_unstale":"2023-10-16T10:04:05.668219+0000","last_undegraded":"2023-10-16T10:04:05.668219+0000","last_fullsized":"2023-10-16T10:04:05.668219+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:10.931058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668462+0000","last_change":"2023-10-16T10:04:03.669954+0000","last_active":"2023-10-16T10:04:05.668462+0000","last_peered":"2023-10-16T10:04:05.668462+0000","last_clean":"2023-10-16T10:04:05.668462+0000","last_became_active":"2023-10-16T10:04:03.669762+0000","last_became_peered":"2023-10-16T10:04:03.669762+0000","last_unstale":"2023-10-16T10:04:05.668462+0000","last_undegraded":"2023-10-16T10:04:05.668462+0000","last_fullsized":"2023-10-16T10:04:05.668462+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:57.114434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.672375+0000","last_change":"2023-10-16T10:04:04.671104+0000","last_active":"2023-10-16T10:04:04.672375+0000","last_peered":"2023-10-16T10:04:04.672375+0000","last_clean":"2023-10-16T10:04:04.672375+0000","last_became_active":"2023-10-16T10:04:04.670822+0000","last_became_peered":"2023-10-16T10:04:04.670822+0000","last_unstale":"2023-10-16T10:04:04.672375+0000","last_undegraded":"2023-10-16T10:04:04.672375+0000","last_fullsized":"2023-10-16T10:04:04.672375+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:04.733721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.666141+0000","last_change":"2023-10-16T10:04:03.666173+0000","last_active":"2023-10-16T10:04:04.666141+0000","last_peered":"2023-10-16T10:04:04.666141+0000","last_clean":"2023-10-16T10:04:04.666141+0000","last_became_active":"2023-10-16T10:04:03.665917+0000","last_became_peered":"2023-10-16T10:04:03.665917+0000","last_unstale":"2023-10-16T10:04:04.666141+0000","last_undegraded":"2023-10-16T10:04:04.666141+0000","last_fullsized":"2023-10-16T10:04:04.666141+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:39.041368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.680945+0000","last_change":"2023-10-16T10:04:04.678587+0000","last_active":"2023-10-16T10:04:04.680945+0000","last_peered":"2023-10-16T10:04:04.680945+0000","last_clean":"2023-10-16T10:04:04.680945+0000","last_became_active":"2023-10-16T10:04:04.678319+0000","last_became_peered":"2023-10-16T10:04:04.678319+0000","last_unstale":"2023-10-16T10:04:04.680945+0000","last_undegraded":"2023-10-16T10:04:04.680945+0000","last_fullsized":"2023-10-16T10:04:04.680945+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:43.191440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.795444+0000","last_change":"2023-10-16T10:04:03.668013+0000","last_active":"2023-10-16T10:04:05.795444+0000","last_peered":"2023-10-16T10:04:05.795444+0000","last_clean":"2023-10-16T10:04:05.795444+0000","last_became_active":"2023-10-16T10:04:03.667714+0000","last_became_peered":"2023-10-16T10:04:03.667714+0000","last_unstale":"2023-10-16T10:04:05.795444+0000","last_undegraded":"2023-10-16T10:04:05.795444+0000","last_fullsized":"2023-10-16T10:04:05.795444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:44.090615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.682032+0000","last_change":"2023-10-16T10:04:04.679521+0000","last_active":"2023-10-16T10:04:04.682032+0000","last_peered":"2023-10-16T10:04:04.682032+0000","last_clean":"2023-10-16T10:04:04.682032+0000","last_became_active":"2023-10-16T10:04:04.679254+0000","last_became_peered":"2023-10-16T10:04:04.679254+0000","last_unstale":"2023-10-16T10:04:04.682032+0000","last_undegraded":"2023-10-16T10:04:04.682032+0000","last_fullsized":"2023-10-16T10:04:04.682032+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:12.851592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668377+0000","last_change":"2023-10-16T10:04:03.666014+0000","last_active":"2023-10-16T10:04:05.668377+0000","last_peered":"2023-10-16T10:04:05.668377+0000","last_clean":"2023-10-16T10:04:05.668377+0000","last_became_active":"2023-10-16T10:04:03.665748+0000","last_became_peered":"2023-10-16T10:04:03.665748+0000","last_unstale":"2023-10-16T10:04:05.668377+0000","last_undegraded":"2023-10-16T10:04:05.668377+0000","last_fullsized":"2023-10-16T10:04:05.668377+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:39:27.408754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.673596+0000","last_change":"2023-10-16T10:04:04.670413+0000","last_active":"2023-10-16T10:04:04.673596+0000","last_peered":"2023-10-16T10:04:04.673596+0000","last_clean":"2023-10-16T10:04:04.673596+0000","last_became_active":"2023-10-16T10:04:04.670162+0000","last_became_peered":"2023-10-16T10:04:04.670162+0000","last_unstale":"2023-10-16T10:04:04.673596+0000","last_undegraded":"2023-10-16T10:04:04.673596+0000","last_fullsized":"2023-10-16T10:04:04.673596+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:58.121592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.667069+0000","last_change":"2023-10-16T10:04:03.663448+0000","last_active":"2023-10-16T10:04:04.667069+0000","last_peered":"2023-10-16T10:04:04.667069+0000","last_clean":"2023-10-16T10:04:04.667069+0000","last_became_active":"2023-10-16T10:04:03.663166+0000","last_became_peered":"2023-10-16T10:04:03.663166+0000","last_unstale":"2023-10-16T10:04:04.667069+0000","last_undegraded":"2023-10-16T10:04:04.667069+0000","last_fullsized":"2023-10-16T10:04:04.667069+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T10:48:52.234268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668777+0000","last_change":"2023-10-16T10:04:04.690645+0000","last_active":"2023-10-16T10:04:05.668777+0000","last_peered":"2023-10-16T10:04:05.668777+0000","last_clean":"2023-10-16T10:04:05.668777+0000","last_became_active":"2023-10-16T10:04:04.689661+0000","last_became_peered":"2023-10-16T10:04:04.689661+0000","last_unstale":"2023-10-16T10:04:05.668777+0000","last_undegraded":"2023-10-16T10:04:05.668777+0000","last_fullsized":"2023-10-16T10:04:05.668777+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:58.128654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668652+0000","last_change":"2023-10-16T10:04:03.668573+0000","last_active":"2023-10-16T10:04:05.668652+0000","last_peered":"2023-10-16T10:04:05.668652+0000","last_clean":"2023-10-16T10:04:05.668652+0000","last_became_active":"2023-10-16T10:04:03.668300+0000","last_became_peered":"2023-10-16T10:04:03.668300+0000","last_unstale":"2023-10-16T10:04:05.668652+0000","last_undegraded":"2023-10-16T10:04:05.668652+0000","last_fullsized":"2023-10-16T10:04:05.668652+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:12.130674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.687359+0000","last_change":"2023-10-16T10:04:04.681776+0000","last_active":"2023-10-16T10:04:04.687359+0000","last_peered":"2023-10-16T10:04:04.687359+0000","last_clean":"2023-10-16T10:04:04.687359+0000","last_became_active":"2023-10-16T10:04:04.681180+0000","last_became_peered":"2023-10-16T10:04:04.681180+0000","last_unstale":"2023-10-16T10:04:04.687359+0000","last_undegraded":"2023-10-16T10:04:04.687359+0000","last_fullsized":"2023-10-16T10:04:04.687359+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:03.646559+0000","last_clean_scrub_stamp":"2023-10-16T10:04:03.646559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:17.265041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-16T10:04:05.668275+0000","last_change":"2023-10-16T10:04:03.672672+0000","last_active":"2023-10-16T10:04:05.668275+0000","last_peered":"2023-10-16T10:04:05.668275+0000","last_clean":"2023-10-16T10:04:05.668275+0000","last_became_active":"2023-10-16T10:04:03.672413+0000","last_became_peered":"2023-10-16T10:04:03.672413+0000","last_unstale":"2023-10-16T10:04:05.668275+0000","last_undegraded":"2023-10-16T10:04:05.668275+0000","last_fullsized":"2023-10-16T10:04:05.668275+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T10:04:02.639401+0000","last_clean_scrub_stamp":"2023-10-16T10:04:02.639401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:57.305053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"143'249","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-16T10:04:04.666591+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T10:04:04.666591+0000","last_peered":"2023-10-16T10:04:04.666591+0000","last_clean":"2023-10-16T10:04:04.666591+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T10:04:04.666591+0000","last_undegraded":"2023-10-16T10:04:04.666591+0000","last_fullsized":"2023-10-16T10:04:04.666591+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575092,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60008,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311832,"statfs":{"total":96636764160,"available":96575315968,"internally_reserved":0,"allocated":2519040,"data_stored":1596045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.76500000000000001,"15min":1.2150000000000001},"last":0.497},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55400000000000005,"15min":0.57099999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.254},"max":{"1min":0.83999999999999997,"5min":1.911,"15min":1.911},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57599999999999996,"15min":0.59799999999999998},"min":{"1min":0.432,"5min":0.34599999999999997,"15min":0.29199999999999998},"max":{"1min":0.85199999999999998,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.1279999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.625},"min":{"1min":0.33800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78800000000000003,"5min":3.3820000000000001,"15min":3.3820000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.63600000000000001},"min":{"1min":0.41899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.84399999999999997,"5min":3.1640000000000001,"15min":3.1640000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.59799999999999998},"min":{"1min":0.39800000000000002,"5min":0.33800000000000002,"15min":0.32300000000000001},"max":{"1min":0.81000000000000005,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64900000000000002,"15min":0.65400000000000003},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.874,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.57899999999999996}]},{"osd":4,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.501,"15min":0.495},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.72499999999999998,"5min":1.8220000000000001,"15min":1.9550000000000001},"last":0.373},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56100000000000005,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33000000000000002,"15min":0.27200000000000002},"max":{"1min":0.83899999999999997,"5min":3.8149999999999999,"15min":3.8149999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.80200000000000005,"5min":0.81999999999999995,"15min":0.84099999999999997},"last":0.439},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.40600000000000003,"5min":0.37,"15min":0.308},"max":{"1min":0.84599999999999997,"5min":1.6619999999999999,"15min":2.0150000000000001},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.7110000000000001,"15min":1.9870000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57999999999999996,"15min":0.57399999999999995},"min":{"1min":0.377,"5min":0.31,"15min":0.31},"max":{"1min":0.88100000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.46100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575092,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":997575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53800000000000003,"15min":0.56999999999999995},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":0.72099999999999997,"5min":0.89100000000000001,"15min":1.0920000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.249},"max":{"1min":0.72999999999999998,"5min":0.80900000000000005,"15min":0.82099999999999995},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.745,"5min":0.77100000000000002,"15min":0.90000000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.55400000000000005,"15min":0.59199999999999997},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.77400000000000002,"5min":0.78900000000000003,"15min":0.83199999999999996},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54200000000000004,"15min":0.57399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.27100000000000002},"max":{"1min":0.75700000000000001,"5min":0.78200000000000003,"15min":1.448},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56599999999999995,"15min":0.59499999999999997},"min":{"1min":0.40899999999999997,"5min":0.35899999999999999,"15min":0.35799999999999998},"max":{"1min":0.78700000000000003,"5min":0.83499999999999996,"15min":1.708},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.42999999999999999,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.315},"max":{"1min":0.71799999999999997,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.499,"15min":0.47999999999999998},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.249},"max":{"1min":0.74399999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.434},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.27500000000000002},"max":{"1min":0.78000000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54300000000000004}]},{"osd":5,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.253},"max":{"1min":0.69699999999999995,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57099999999999995,"15min":0.54900000000000004},"min":{"1min":0.433,"5min":0.35799999999999998,"15min":0.311},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.505}]},{"osd":7,"last update":"Mon Oct 16 10:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":0.80200000000000005,"5min":0.86499999999999999,"15min":0.91900000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.35799999999999998,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.80500000000000005,"15min":0.94599999999999995},"last":0.46200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1900544,"data_stored":996973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.35199999999999998,"15min":0.38600000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.188},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":1.6000000000000001},"last":0.308},{"interface":"front","average":{"1min":0.379,"5min":0.39800000000000002,"15min":0.43099999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.70199999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36099999999999999,"15min":0.40200000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":0.63300000000000001,"5min":0.98699999999999999,"15min":4.923},"last":0.252},{"interface":"front","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.42699999999999999},"min":{"1min":0.28000000000000003,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":1.712},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42899999999999999,"15min":0.45300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.984},"last":0.37},{"interface":"front","average":{"1min":0.375,"5min":0.40400000000000003,"15min":0.42499999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.27000000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.87,"15min":0.93100000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.63700000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"max":{"1min":0.89000000000000001,"5min":1.0840000000000001,"15min":5.0069999999999997},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61599999999999999,"15min":0.65400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92300000000000004,"5min":1.04,"15min":4.9749999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.66200000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.085,"5min":1.085,"15min":1.7769999999999999},"last":0.61599999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60199999999999998,"15min":0.63100000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.63},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61499999999999999,"15min":0.64900000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.052,"5min":1.3029999999999999,"15min":5.0419999999999998},"last":0.56299999999999994}]},{"osd":7,"last update":"Mon Oct 16 10:04:04 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.744},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61299999999999999,"15min":0.64200000000000002},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575093,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37780,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334060,"statfs":{"total":96636764160,"available":96598077440,"internally_reserved":0,"allocated":2498560,"data_stored":1587327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35799999999999998,"15min":0.38100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.192},"max":{"1min":0.52000000000000002,"5min":0.65700000000000003,"15min":1.2110000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39600000000000002,"15min":0.42699999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26500000000000001},"max":{"1min":0.54000000000000004,"5min":0.63,"15min":1.3169999999999999},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.20599999999999999},"max":{"1min":0.56299999999999994,"5min":0.61899999999999999,"15min":0.70099999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.45500000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.249},"max":{"1min":0.48499999999999999,"5min":0.63600000000000001,"15min":0.79900000000000004},"last":0.40100000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.38100000000000001,"15min":0.41299999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.83399999999999996},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42799999999999999,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.624,"5min":0.67300000000000004,"15min":1.1859999999999999},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.33700000000000002},"max":{"1min":0.81799999999999995,"5min":0.83999999999999997,"15min":0.93700000000000006},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59199999999999997,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":1.1659999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59299999999999997,"15min":0.63},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.33300000000000002},"max":{"1min":0.80500000000000005,"5min":0.86499999999999999,"15min":0.99299999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.625},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.311},"max":{"1min":0.82899999999999996,"5min":0.89100000000000001,"15min":0.91000000000000003},"last":0.47399999999999998}]},{"osd":6,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.40899999999999997},"max":{"1min":0.89200000000000002,"5min":0.90900000000000003,"15min":1.2410000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.624,"15min":0.66400000000000003},"min":{"1min":0.44800000000000001,"5min":0.39900000000000002,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.877,"15min":0.97199999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Oct 16 10:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60799999999999998,"15min":0.63900000000000001},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.316},"max":{"1min":0.88500000000000001,"5min":0.89200000000000002,"15min":1.02},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.60399999999999998,"15min":0.64000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.34899999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.44700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59416,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312424,"statfs":{"total":96636764160,"available":96575922176,"internally_reserved":0,"allocated":1904640,"data_stored":997519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.372},"min":{"1min":0.22900000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.496,"5min":0.57099999999999995,"15min":0.61599999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39500000000000002,"15min":0.40899999999999997},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39900000000000002,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.67900000000000005,"15min":0.68000000000000005},"last":0.374},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.623,"5min":0.624,"15min":0.624},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.246,"15min":0.23499999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.63500000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.47099999999999997},"min":{"1min":0.315,"5min":0.29899999999999999,"15min":0.25600000000000001},"max":{"1min":0.57399999999999995,"5min":0.72499999999999998,"15min":0.79800000000000004},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.629},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84999999999999998,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.33400000000000002},"max":{"1min":0.84899999999999998,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59599999999999997,"15min":0.63800000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76600000000000001,"5min":1.0249999999999999,"15min":1.103},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"max":{"1min":0.77700000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":1.046,"15min":1.1699999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60699999999999998,"15min":0.66100000000000003},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.84399999999999997,"5min":0.96599999999999997,"15min":0.97299999999999998},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Oct 16 10:03:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81499999999999995,"5min":0.94699999999999995,"15min":1.0069999999999999},"last":0.501},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.53000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1925120,"data_stored":1005657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34399999999999997,"15min":0.36599999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.17999999999999999},"max":{"1min":0.51100000000000001,"5min":0.61799999999999999,"15min":0.63700000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.38600000000000001},"min":{"1min":0.249,"5min":0.216,"15min":0.216},"max":{"1min":0.56299999999999994,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.245}]},{"osd":1,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35599999999999998,"15min":0.39000000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.51700000000000002,"5min":0.58899999999999997,"15min":0.65400000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39200000000000002,"15min":0.42299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.67500000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.45900000000000002},"min":{"1min":0.32700000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.65000000000000002,"5min":0.96099999999999997,"15min":1.0760000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45600000000000002,"15min":0.48299999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68600000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.57799999999999996,"15min":0.61599999999999999},"min":{"1min":0.35599999999999998,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.96499999999999997},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.33700000000000002},"max":{"1min":0.81599999999999995,"5min":0.83399999999999996,"15min":1.6910000000000001},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":1.6259999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.38700000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89100000000000001,"5min":1.673,"15min":1.673},"last":0.81899999999999995}]},{"osd":6,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58599999999999997,"15min":0.63900000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.374},"max":{"1min":0.83099999999999996,"5min":0.83299999999999996,"15min":0.98799999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59499999999999997,"15min":0.64400000000000002},"min":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86199999999999999,"5min":0.92600000000000005,"15min":1.0589999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Oct 16 10:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.62,"15min":0.66200000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85699999999999998,"5min":0.878,"15min":1.0029999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.83099999999999996,"15min":0.72299999999999998},"min":{"1min":0.42799999999999999,"5min":0.42699999999999999,"15min":0.36899999999999999},"max":{"1min":0.96999999999999997,"5min":24.861000000000001,"15min":24.861000000000001},"last":0.69099999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575092,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1908736,"data_stored":1003975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.72199999999999998,"5min":0.82099999999999995,"15min":1.0269999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57699999999999996,"15min":0.58799999999999997},"min":{"1min":0.29799999999999999,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":0.76000000000000001,"5min":0.83099999999999996,"15min":1.0860000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.58499999999999996},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.312},"max":{"1min":0.70699999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.41899999999999998,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":1.6359999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.85199999999999998,"15min":0.88300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.38600000000000001,"15min":0.32500000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.2270000000000001},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55700000000000005,"15min":0.58299999999999996},"min":{"1min":0.40000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.69099999999999995,"5min":0.75800000000000001,"15min":0.86099999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60399999999999998,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.38300000000000001,"15min":0.34200000000000003},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.495},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36199999999999999,"5min":0.26400000000000001,"15min":0.26300000000000001},"max":{"1min":0.72899999999999998,"5min":0.77500000000000002,"15min":0.96599999999999997},"last":0.76400000000000001}]},{"osd":6,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.24199999999999999},"max":{"1min":0.71899999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.57199999999999995},"min":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":0.73199999999999998,"5min":0.80000000000000004,"15min":0.92400000000000004},"last":0.85899999999999999}]},{"osd":7,"last update":"Mon Oct 16 10:03:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58899999999999997,"15min":0.57499999999999996},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.77900000000000003,"5min":0.82399999999999995,"15min":0.91100000000000003},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.626,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.89400000000000002,"15min":1.0429999999999999},"last":0.93300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575092,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51088,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320752,"statfs":{"total":96636764160,"available":96584450048,"internally_reserved":0,"allocated":1900544,"data_stored":996925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.61099999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.792},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.435,"5min":0.34599999999999997,"15min":0.27300000000000002},"max":{"1min":0.80800000000000005,"5min":0.85799999999999998,"15min":1.7090000000000001},"last":0.56599999999999995}]},{"osd":1,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55500000000000005,"15min":0.57899999999999996},"min":{"1min":0.34999999999999998,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.79400000000000004,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54200000000000004,"15min":0.56999999999999995},"min":{"1min":0.34399999999999997,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.82899999999999996},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.28599999999999998,"15min":0.25800000000000001},"max":{"1min":0.754,"5min":0.80800000000000005,"15min":1.496},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.59499999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34200000000000003},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":2.069},"last":0.71199999999999997}]},{"osd":3,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.317,"15min":0.317},"max":{"1min":0.70599999999999996,"5min":1,"15min":1},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.39600000000000002,"5min":0.34499999999999997,"15min":0.33400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":1.7689999999999999},"last":0.73599999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.5,"15min":0.48599999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27600000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.84199999999999997},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52500000000000002,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.86899999999999999},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54500000000000004},"min":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.72099999999999997,"5min":0.78400000000000003,"15min":0.88200000000000001},"last":0.76500000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:03:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.25900000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82899999999999996,"5min":0.88,"15min":0.88},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T10:04:09.171 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T10:04:09.171 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T10:04:09.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.331+0000 7fa094c1e700 1 Processor -- start 2023-10-16T10:04:09.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.331+0000 7fa094c1e700 1 -- start start 2023-10-16T10:04:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa094c1e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa0900cbf50 0x7fa090144b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa094c1e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa090070b20 0x7fa0900cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa094c1e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09015f6a0 con 0x7fa090145040 2023-10-16T10:04:09.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa094c1e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa090160220 con 0x7fa0900cbf50 2023-10-16T10:04:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa094c1e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09015f980 con 0x7fa090070b20 2023-10-16T10:04:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08e59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa0900cbf50 0x7fa090144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08dd9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa090145040 0x7fa090070410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60006/0 (socket says 172.21.15.158:60006) 2023-10-16T10:04:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08dd9b700 1 -- 172.21.15.158:0/2391574369 learned_addr learned my addr 172.21.15.158:0/2391574369 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:09.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08ed9d700 1 --2- 172.21.15.158:0/2391574369 >> v2:172.21.15.172:3300/0 conn(0x7fa090070b20 0x7fa0900cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08e59c700 1 -- 172.21.15.158:0/2391574369 >> v2:172.21.15.172:3300/0 conn(0x7fa090070b20 msgr2=0x7fa0900cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08e59c700 1 --2- 172.21.15.158:0/2391574369 >> v2:172.21.15.172:3300/0 conn(0x7fa090070b20 0x7fa0900cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08e59c700 1 -- 172.21.15.158:0/2391574369 >> v1:172.21.15.158:6789/0 conn(0x7fa090145040 legacy=0x7fa090070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08e59c700 1 -- 172.21.15.158:0/2391574369 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09015fc60 con 0x7fa0900cbf50 2023-10-16T10:04:09.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08d59a700 1 -- 172.21.15.158:0/2391574369 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213600680 0 0) 0x7fa09015f6a0 con 0x7fa090145040 2023-10-16T10:04:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08ed9d700 1 --2- 172.21.15.158:0/2391574369 >> v2:172.21.15.172:3300/0 conn(0x7fa090070b20 0x7fa0900cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08e59c700 1 --2- 172.21.15.158:0/2391574369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0900cbf50 0x7fa090144b00 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa0900cde30 tx=0x7fa080009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885cdc3eef8e3ea1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.332+0000 7fa08d59a700 1 -- 172.21.15.158:0/2391574369 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa080011830 con 0x7fa0900cbf50 2023-10-16T10:04:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa08d59a700 1 -- 172.21.15.158:0/2391574369 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa080011990 con 0x7fa0900cbf50 2023-10-16T10:04:09.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa08d59a700 1 -- 172.21.15.158:0/2391574369 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa080011d20 con 0x7fa0900cbf50 2023-10-16T10:04:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 -- 172.21.15.158:0/2391574369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0900cbf50 msgr2=0x7fa090144b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2391574369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0900cbf50 0x7fa090144b00 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa0900cde30 tx=0x7fa080009b20 comp rx=0 tx=0).stop 2023-10-16T10:04:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 -- 172.21.15.158:0/2391574369 shutdown_connections 2023-10-16T10:04:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2391574369 >> v2:172.21.15.172:3300/0 conn(0x7fa090070b20 0x7fa0900cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2391574369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0900cbf50 0x7fa090144b00 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 -- 172.21.15.158:0/2391574369 >> 172.21.15.158:0/2391574369 conn(0x7fa0900d1410 msgr2=0x7fa0900d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 -- 172.21.15.158:0/2391574369 shutdown_connections 2023-10-16T10:04:09.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.333+0000 7fa094c1e700 1 -- 172.21.15.158:0/2391574369 wait complete. 2023-10-16T10:04:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.334+0000 7fa094c1e700 1 Processor -- start 2023-10-16T10:04:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa094c1e700 1 -- start start 2023-10-16T10:04:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa094c1e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa090070b20 0x7fa0900ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa094c1e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0900cbf50 0x7fa0900caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa094c1e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa090145040 0x7fa0901f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa094c1e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900c84e0 con 0x7fa090145040 2023-10-16T10:04:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa094c1e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09015cb30 con 0x7fa0900cbf50 2023-10-16T10:04:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa094c1e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09015ce10 con 0x7fa090070b20 2023-10-16T10:04:09.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa08ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa090145040 0x7fa0901f08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa08ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa090145040 0x7fa0901f08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59840/0 (socket says 172.21.15.158:59840) 2023-10-16T10:04:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.335+0000 7fa08ed9d700 1 -- 172.21.15.158:0/2427553401 learned_addr learned my addr 172.21.15.158:0/2427553401 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08dd9b700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0900cbf50 0x7fa0900caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08e59c700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa090070b20 0x7fa0900ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08ed9d700 1 -- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa090070b20 msgr2=0x7fa0900ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08ed9d700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa090070b20 0x7fa0900ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08ed9d700 1 -- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0900cbf50 msgr2=0x7fa0900caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08ed9d700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0900cbf50 0x7fa0900caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08ed9d700 1 -- 172.21.15.158:0/2427553401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0800094a0 con 0x7fa090145040 2023-10-16T10:04:09.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa08ed9d700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa090145040 0x7fa0901f08e0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa08800cd50 tx=0x7fa08800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa07effd700 1 -- 172.21.15.158:0/2427553401 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08800f480 con 0x7fa090145040 2023-10-16T10:04:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa07effd700 1 -- 172.21.15.158:0/2427553401 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08800f600 con 0x7fa090145040 2023-10-16T10:04:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0900c8730 con 0x7fa090145040 2023-10-16T10:04:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa07effd700 1 -- 172.21.15.158:0/2427553401 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08800f9b0 con 0x7fa090145040 2023-10-16T10:04:09.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.336+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0901f0e20 con 0x7fa090145040 2023-10-16T10:04:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.337+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0900c8c50 con 0x7fa090145040 2023-10-16T10:04:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.338+0000 7fa07effd700 1 -- 172.21.15.158:0/2427553401 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa08800b040 con 0x7fa090145040 2023-10-16T10:04:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.338+0000 7fa07effd700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa07006b020 0x7fa07006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.338+0000 7fa07effd700 1 -- 172.21.15.158:0/2427553401 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fa08808a420 con 0x7fa090145040 2023-10-16T10:04:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.338+0000 7fa08e59c700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa07006b020 0x7fa07006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.338+0000 7fa08e59c700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa07006b020 0x7fa07006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0800090b0 tx=0x7fa080008030 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.342+0000 7fa07effd700 1 -- 172.21.15.158:0/2427553401 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0900c8c50 con 0x7fa090145040 2023-10-16T10:04:09.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.612+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa0900c8c50 con 0x7fa090145040 2023-10-16T10:04:09.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.614+0000 7fa07effd700 1 -- 172.21.15.158:0/2427553401 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa0900c8c50 con 0x7fa090145040 2023-10-16T10:04:09.616 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T10:04:09.616 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T10:04:09.616 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T10:04:09.616 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T10:04:09.616 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T10:04:09.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.616+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa07006b020 msgr2=0x7fa07006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.616+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa07006b020 0x7fa07006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0800090b0 tx=0x7fa080008030 comp rx=0 tx=0).stop 2023-10-16T10:04:09.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.616+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa090145040 msgr2=0x7fa0901f08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.616+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa090145040 0x7fa0901f08e0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa08800cd50 tx=0x7fa08800a560 comp rx=0 tx=0).stop 2023-10-16T10:04:09.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.616+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 shutdown_connections 2023-10-16T10:04:09.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.617+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa090145040 0x7fa0901f08e0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.617+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa07006b020 0x7fa07006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.617+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa090070b20 0x7fa0900ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.617+0000 7fa094c1e700 1 --2- 172.21.15.158:0/2427553401 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0900cbf50 0x7fa0900caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.617+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 >> 172.21.15.158:0/2427553401 conn(0x7fa0900d1410 msgr2=0x7fa0900b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:09.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.617+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 shutdown_connections 2023-10-16T10:04:09.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.617+0000 7fa094c1e700 1 -- 172.21.15.158:0/2427553401 wait complete. 2023-10-16T10:04:09.620 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T10:04:09.630 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T10:04:09.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.750+0000 7fe835fe1700 1 Processor -- start 2023-10-16T10:04:09.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.750+0000 7fe835fe1700 1 -- start start 2023-10-16T10:04:09.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe835fe1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe83006e5d0 0x7fe83006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe835fe1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8300729b0 0x7fe830071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe835fe1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe830071e50 con 0x7fe83006ab40 2023-10-16T10:04:09.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe835fe1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe830161270 con 0x7fe83006e5d0 2023-10-16T10:04:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe835fe1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe830161510 con 0x7fe8300729b0 2023-10-16T10:04:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe83006e5d0 0x7fe83006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe83006ab40 0x7fe8300722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60020/0 (socket says 172.21.15.158:60020) 2023-10-16T10:04:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82f7fe700 1 -- 172.21.15.158:0/386350694 learned_addr learned my addr 172.21.15.158:0/386350694 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82effd700 1 -- 172.21.15.158:0/386350694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8300729b0 msgr2=0x7fe830071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82effd700 1 --2- 172.21.15.158:0/386350694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8300729b0 0x7fe830071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82effd700 1 -- 172.21.15.158:0/386350694 >> v1:172.21.15.158:6789/0 conn(0x7fe83006ab40 legacy=0x7fe8300722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82e7fc700 1 -- 172.21.15.158:0/386350694 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649387719 0 0) 0x7fe830071e50 con 0x7fe83006ab40 2023-10-16T10:04:09.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82effd700 1 -- 172.21.15.158:0/386350694 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8301617f0 con 0x7fe83006e5d0 2023-10-16T10:04:09.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.751+0000 7fe82effd700 1 --2- 172.21.15.158:0/386350694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe83006d820 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fe820009840 tx=0x7fe820009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5550db410995927 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:09.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe82e7fc700 1 -- 172.21.15.158:0/386350694 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820011830 con 0x7fe83006e5d0 2023-10-16T10:04:09.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe82e7fc700 1 -- 172.21.15.158:0/386350694 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe820011990 con 0x7fe83006e5d0 2023-10-16T10:04:09.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe82e7fc700 1 -- 172.21.15.158:0/386350694 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820011d20 con 0x7fe83006e5d0 2023-10-16T10:04:09.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 -- 172.21.15.158:0/386350694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 msgr2=0x7fe83006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 --2- 172.21.15.158:0/386350694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe83006d820 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fe820009840 tx=0x7fe820009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:09.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 -- 172.21.15.158:0/386350694 shutdown_connections 2023-10-16T10:04:09.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 --2- 172.21.15.158:0/386350694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8300729b0 0x7fe830071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 --2- 172.21.15.158:0/386350694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe83006d820 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 -- 172.21.15.158:0/386350694 >> 172.21.15.158:0/386350694 conn(0x7fe8300d02b0 msgr2=0x7fe8300d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:09.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 -- 172.21.15.158:0/386350694 shutdown_connections 2023-10-16T10:04:09.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.752+0000 7fe835fe1700 1 -- 172.21.15.158:0/386350694 wait complete. 2023-10-16T10:04:09.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.753+0000 7fe835fe1700 1 Processor -- start 2023-10-16T10:04:09.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe835fe1700 1 -- start start 2023-10-16T10:04:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe835fe1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83006ab40 0x7fe830155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe835fe1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe8300b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe835fe1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8300729b0 0x7fe8300c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe835fe1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300b5db0 con 0x7fe83006e5d0 2023-10-16T10:04:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe835fe1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c9450 con 0x7fe8300729b0 2023-10-16T10:04:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe835fe1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c9730 con 0x7fe83006ab40 2023-10-16T10:04:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe82effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe8300b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe82effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe8300b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59850/0 (socket says 172.21.15.158:59850) 2023-10-16T10:04:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.754+0000 7fe82effd700 1 -- 172.21.15.158:0/2404643119 learned_addr learned my addr 172.21.15.158:0/2404643119 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82f7fe700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83006ab40 0x7fe830155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82ffff700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8300729b0 0x7fe8300c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82effd700 1 -- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83006ab40 msgr2=0x7fe830155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82effd700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83006ab40 0x7fe830155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82effd700 1 -- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8300729b0 msgr2=0x7fe8300c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:09.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82effd700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8300729b0 0x7fe8300c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:09.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82effd700 1 -- 172.21.15.158:0/2404643119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8200094a0 con 0x7fe83006e5d0 2023-10-16T10:04:09.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82ffff700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8300729b0 0x7fe8300c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:09.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe82effd700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe8300b0aa0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fe8200090b0 tx=0x7fe820004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:09.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe817fff700 1 -- 172.21.15.158:0/2404643119 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200038a0 con 0x7fe83006e5d0 2023-10-16T10:04:09.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe817fff700 1 -- 172.21.15.158:0/2404643119 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82001f070 con 0x7fe83006e5d0 2023-10-16T10:04:09.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8300c9a10 con 0x7fe83006e5d0 2023-10-16T10:04:09.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe817fff700 1 -- 172.21.15.158:0/2404643119 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820003c40 con 0x7fe83006e5d0 2023-10-16T10:04:09.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.755+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8300c9fd0 con 0x7fe83006e5d0 2023-10-16T10:04:09.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.756+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8300681b0 con 0x7fe83006e5d0 2023-10-16T10:04:09.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.757+0000 7fe817fff700 1 -- 172.21.15.158:0/2404643119 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe82001a430 con 0x7fe83006e5d0 2023-10-16T10:04:09.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.757+0000 7fe817fff700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fe81c06b020 0x7fe81c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:09.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.757+0000 7fe817fff700 1 -- 172.21.15.158:0/2404643119 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fe820021070 con 0x7fe83006e5d0 2023-10-16T10:04:09.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.757+0000 7fe82f7fe700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fe81c06b020 0x7fe81c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:09.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.757+0000 7fe82f7fe700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fe81c06b020 0x7fe81c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe824000f60 tx=0x7fe824007be0 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:09.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.762+0000 7fe817fff700 1 -- 172.21.15.158:0/2404643119 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe820056290 con 0x7fe83006e5d0 2023-10-16T10:04:09.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.996+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe830161aa0 con 0x7fe83006e5d0 2023-10-16T10:04:09.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.997+0000 7fe817fff700 1 -- 172.21.15.158:0/2404643119 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fe8200598b0 con 0x7fe83006e5d0 2023-10-16T10:04:09.999 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:09.999 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8422,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1604199546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1604199546},{"type":"v1","addr":"172.21.15.172:6834","nonce":1604199546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/95492242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":95492242},{"type":"v1","addr":"172.21.15.158:6836","nonce":95492242}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9393,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112424210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112424210},{"type":"v1","addr":"172.21.15.158:6837","nonce":112424210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:04:04.049571+0000","modified":"2023-10-16T10:04:07.086241+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37352":{"gid":37352,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/253283745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":253283745},{"type":"v1","addr":"172.21.15.172:6835","nonce":253283745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T10:04:10.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fe81c06b020 msgr2=0x7fe81c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fe81c06b020 0x7fe81c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe824000f60 tx=0x7fe824007be0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 msgr2=0x7fe8300b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe8300b0aa0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fe8200090b0 tx=0x7fe820004750 comp rx=0 tx=0).stop 2023-10-16T10:04:10.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 shutdown_connections 2023-10-16T10:04:10.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8300729b0 0x7fe8300c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fe81c06b020 0x7fe81c06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe83006ab40 0x7fe830155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 --2- 172.21.15.158:0/2404643119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83006e5d0 0x7fe8300b0aa0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 >> 172.21.15.158:0/2404643119 conn(0x7fe8300d02b0 msgr2=0x7fe8301576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 shutdown_connections 2023-10-16T10:04:10.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:09.999+0000 7fe835fe1700 1 -- 172.21.15.158:0/2404643119 wait complete. 2023-10-16T10:04:10.003 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 77 2023-10-16T10:04:10.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T10:04:04.049571+0000', 'modified': '2023-10-16T10:04:07.086241+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 37352}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37352': {'gid': 37352, 'name': 'b', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/253283745', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 253283745}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 253283745}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T10:04:10.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T10:04:10.415 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T10:04:10.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.539+0000 7fa22813a700 1 Processor -- start 2023-10-16T10:04:10.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.539+0000 7fa22813a700 1 -- start start 2023-10-16T10:04:10.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.540+0000 7fa22813a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa220160cf0 0x7fa22006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.540+0000 7fa22813a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2200b3290 0x7fa22006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.540+0000 7fa22813a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22006f100 con 0x7fa2200cf7c0 2023-10-16T10:04:10.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.540+0000 7fa22813a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2200701a0 con 0x7fa2200b3290 2023-10-16T10:04:10.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.540+0000 7fa22813a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22006f3e0 con 0x7fa220160cf0 2023-10-16T10:04:10.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.540+0000 7fa2266d7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2200b3290 0x7fa22006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa2266d7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2200b3290 0x7fa22006ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59862/0 (socket says 172.21.15.158:59862) 2023-10-16T10:04:10.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa2266d7700 1 -- 172.21.15.158:0/1199962149 learned_addr learned my addr 172.21.15.158:0/1199962149 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:10.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa225ed6700 1 --2- 172.21.15.158:0/1199962149 >> v2:172.21.15.172:3300/0 conn(0x7fa220160cf0 0x7fa22006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa2266d7700 1 -- 172.21.15.158:0/1199962149 >> v2:172.21.15.172:3300/0 conn(0x7fa220160cf0 msgr2=0x7fa22006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa2266d7700 1 --2- 172.21.15.158:0/1199962149 >> v2:172.21.15.172:3300/0 conn(0x7fa220160cf0 0x7fa22006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa2266d7700 1 -- 172.21.15.158:0/1199962149 >> v1:172.21.15.172:6789/0 conn(0x7fa2200cf7c0 legacy=0x7fa220161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa2266d7700 1 -- 172.21.15.158:0/1199962149 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22006f6c0 con 0x7fa2200b3290 2023-10-16T10:04:10.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa2266d7700 1 --2- 172.21.15.158:0/1199962149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2200b3290 0x7fa22006ae10 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fa22006d130 tx=0x7fa21800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8e714c5b2a5750 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa224ed4700 1 -- 172.21.15.158:0/1199962149 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800e070 con 0x7fa2200b3290 2023-10-16T10:04:10.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa224ed4700 1 -- 172.21.15.158:0/1199962149 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21800b910 con 0x7fa2200b3290 2023-10-16T10:04:10.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.541+0000 7fa224ed4700 1 -- 172.21.15.158:0/1199962149 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800bcc0 con 0x7fa2200b3290 2023-10-16T10:04:10.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 -- 172.21.15.158:0/1199962149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2200b3290 msgr2=0x7fa22006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 --2- 172.21.15.158:0/1199962149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2200b3290 0x7fa22006ae10 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fa22006d130 tx=0x7fa21800b330 comp rx=0 tx=0).stop 2023-10-16T10:04:10.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 -- 172.21.15.158:0/1199962149 shutdown_connections 2023-10-16T10:04:10.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 --2- 172.21.15.158:0/1199962149 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2200b3290 0x7fa22006ae10 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 --2- 172.21.15.158:0/1199962149 >> v2:172.21.15.172:3300/0 conn(0x7fa220160cf0 0x7fa22006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 -- 172.21.15.158:0/1199962149 >> 172.21.15.158:0/1199962149 conn(0x7fa22006da00 msgr2=0x7fa22006dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 -- 172.21.15.158:0/1199962149 shutdown_connections 2023-10-16T10:04:10.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.542+0000 7fa22813a700 1 -- 172.21.15.158:0/1199962149 wait complete. 2023-10-16T10:04:10.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.543+0000 7fa22813a700 1 Processor -- start 2023-10-16T10:04:10.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.543+0000 7fa22813a700 1 -- start start 2023-10-16T10:04:10.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2200b3290 0x7fa2200ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2200cf7c0 0x7fa2201470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa220160cf0 0x7fa2200d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200ad0b0 con 0x7fa220160cf0 2023-10-16T10:04:10.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2201f4130 con 0x7fa2200cf7c0 2023-10-16T10:04:10.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2201f4410 con 0x7fa2200b3290 2023-10-16T10:04:10.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa220160cf0 0x7fa2200d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa220160cf0 0x7fa2200d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59868/0 (socket says 172.21.15.158:59868) 2023-10-16T10:04:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 -- 172.21.15.158:0/3271901935 learned_addr learned my addr 172.21.15.158:0/3271901935 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2256d5700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2200cf7c0 0x7fa2201470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa225ed6700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2200b3290 0x7fa2200ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 -- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2200b3290 msgr2=0x7fa2200ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2200b3290 0x7fa2200ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 -- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2200cf7c0 msgr2=0x7fa2201470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2200cf7c0 0x7fa2201470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 -- 172.21.15.158:0/3271901935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa218009b30 con 0x7fa220160cf0 2023-10-16T10:04:10.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa225ed6700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2200b3290 0x7fa2200ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:10.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2266d7700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa220160cf0 0x7fa2200d0260 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fa2200d1480 tx=0x7fa218004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2167fc700 1 -- 172.21.15.158:0/3271901935 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218004930 con 0x7fa220160cf0 2023-10-16T10:04:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2167fc700 1 -- 172.21.15.158:0/3271901935 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2180099b0 con 0x7fa220160cf0 2023-10-16T10:04:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa2167fc700 1 -- 172.21.15.158:0/3271901935 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21801c680 con 0x7fa220160cf0 2023-10-16T10:04:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2201f46f0 con 0x7fa220160cf0 2023-10-16T10:04:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.544+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2200acdc0 con 0x7fa220160cf0 2023-10-16T10:04:10.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.546+0000 7fa2167fc700 1 -- 172.21.15.158:0/3271901935 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa21801c7e0 con 0x7fa220160cf0 2023-10-16T10:04:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.546+0000 7fa2167fc700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa20406b020 0x7fa20406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.546+0000 7fa2167fc700 1 -- 172.21.15.158:0/3271901935 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fa21808b450 con 0x7fa220160cf0 2023-10-16T10:04:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.546+0000 7fa225ed6700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa20406b020 0x7fa20406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.547+0000 7fa225ed6700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa20406b020 0x7fa20406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa22006b120 tx=0x7fa210008040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.548+0000 7fa234324b80 1 Processor -- start 2023-10-16T10:04:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.548+0000 7fa234324b80 1 -- start start 2023-10-16T10:04:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d24017220 0x563d2401ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa234324b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d2401efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d23ff2010 0x563d23ff26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa234324b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d23f68c20 con 0x563d2401c630 2023-10-16T10:04:10.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d23f4dc10 con 0x563d23ff2010 2023-10-16T10:04:10.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d23f4b530 con 0x563d24017220 2023-10-16T10:04:10.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa2256d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d2401efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa2256d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d2401efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59872/0 (socket says 172.21.15.158:59872) 2023-10-16T10:04:10.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa2256d5700 1 -- 172.21.15.158:0/2336449084 learned_addr learned my addr 172.21.15.158:0/2336449084 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:10.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa2266d7700 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d23ff2010 0x563d23ff26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.549+0000 7fa225ed6700 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d24017220 0x563d2401ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2256d5700 1 -- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d24017220 msgr2=0x563d2401ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2256d5700 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d24017220 0x563d2401ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2256d5700 1 -- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d23ff2010 msgr2=0x563d23ff26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2256d5700 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d23ff2010 0x563d23ff26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2256d5700 1 -- 172.21.15.158:0/2336449084 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d23f043a0 con 0x563d2401c630 2023-10-16T10:04:10.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2266d7700 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d23ff2010 0x563d23ff26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:10.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2256d5700 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d2401efb0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fa21c001740 tx=0x7fa21c007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6469081bc58419b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2037fe700 1 -- 172.21.15.158:0/2336449084 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0052e0 con 0x563d2401c630 2023-10-16T10:04:10.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2037fe700 1 -- 172.21.15.158:0/2336449084 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c005440 con 0x563d2401c630 2023-10-16T10:04:10.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa2037fe700 1 -- 172.21.15.158:0/2336449084 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c003880 con 0x563d2401c630 2023-10-16T10:04:10.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa234324b80 1 -- 172.21.15.158:0/2336449084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 msgr2=0x563d2401efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.550+0000 7fa234324b80 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d2401efb0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fa21c001740 tx=0x7fa21c007ec0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 -- 172.21.15.158:0/2336449084 shutdown_connections 2023-10-16T10:04:10.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d23ff2010 0x563d23ff26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d24017220 0x563d2401ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 --2- 172.21.15.158:0/2336449084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d2401efb0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 -- 172.21.15.158:0/2336449084 >> 172.21.15.158:0/2336449084 conn(0x563d23f21d80 msgr2=0x563d2403cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 -- 172.21.15.158:0/2336449084 shutdown_connections 2023-10-16T10:04:10.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 -- 172.21.15.158:0/2336449084 wait complete. 2023-10-16T10:04:10.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 -- start start 2023-10-16T10:04:10.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.551+0000 7fa234324b80 1 Processor -- start 2023-10-16T10:04:10.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d2401b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d24020f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d23fe8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d23f46e20 con 0x563d24017220 2023-10-16T10:04:10.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d23ee36c0 con 0x563d2401c630 2023-10-16T10:04:10.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d23ee3360 con 0x563d23ff2010 2023-10-16T10:04:10.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 10 client.0 _ll_register_callbacks cb 0x563d23d407b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:04:10.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d24020f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d24020f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59880/0 (socket says 172.21.15.158:59880) 2023-10-16T10:04:10.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 -- 172.21.15.158:0/2999819074 learned_addr learned my addr 172.21.15.158:0/2999819074 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:10.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa225ed6700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d2401b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2266d7700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d23fe8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 -- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 msgr2=0x563d2401b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d2401b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 -- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 msgr2=0x563d23fe8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d23fe8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 -- 172.21.15.158:0/2999819074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22006f100 con 0x563d24017220 2023-10-16T10:04:10.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa225ed6700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d2401b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:10.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2266d7700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d23fe8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:10.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa2256d5700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d24020f30 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fa21c001eb0 tx=0x7fa21c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a83dbf723c94af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa200ff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:04:10.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa200ff9700 1 -- 172.21.15.158:0/2999819074 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0052e0 con 0x563d24017220 2023-10-16T10:04:10.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa200ff9700 1 -- 172.21.15.158:0/2999819074 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fa21c007b80 con 0x563d24017220 2023-10-16T10:04:10.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.553+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d23fe75b0 con 0x563d24017220 2023-10-16T10:04:10.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa200ff9700 1 -- 172.21.15.158:0/2999819074 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa21c006a60 con 0x563d24017220 2023-10-16T10:04:10.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 10 client.9534 fetch_fsmap learned FSMap version 77 2023-10-16T10:04:10.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x563d23d71d90 con 0x563d24017220 2023-10-16T10:04:10.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa200ff9700 1 -- 172.21.15.158:0/2999819074 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563d23fe75b0 con 0x563d24017220 2023-10-16T10:04:10.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa200ff9700 1 -- 172.21.15.158:0/2999819074 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 77) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa21c012070 con 0x563d24017220 2023-10-16T10:04:10.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 10 client.9534 fetch_fsmap finished waiting for FSMap version 77 2023-10-16T10:04:10.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 10 client.9534 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T10:04:10.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 20 client.9534 populate_metadata read hostname 'smithi158' 2023-10-16T10:04:10.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d23fe9480 0x563d2401fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 4 client.9534 mds_command: new command op to 37352 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T10:04:10.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 --> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d23f3fe00 con 0x563d23fe9480 2023-10-16T10:04:10.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa225ed6700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d23fe9480 0x563d2401fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa200ff9700 10 client.9534 ms_handle_connect on v2:172.21.15.172:6833/253283745 2023-10-16T10:04:10.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.554+0000 7fa225ed6700 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d23fe9480 0x563d2401fd50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2b569a41d82cd9e server_cookie=855327001702e93 in_seq=0 out_seq=0 2023-10-16T10:04:10.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa200ff9700 1 -- 172.21.15.158:0/2999819074 <== mds.0 v2:172.21.15.172:6833/253283745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563d23f3fe00 con 0x563d23fe9480 2023-10-16T10:04:10.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa200ff9700 10 client.9534 handle_command_reply: tid=0 2023-10-16T10:04:10.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 client.9534 shutdown 2023-10-16T10:04:10.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 msgr2=0x563d24020f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d24020f30 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fa21c001eb0 tx=0x7fa21c000ed0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 shutdown_connections 2023-10-16T10:04:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d23fe9480 0x563d2401fd50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d23fe8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa200ff9700 0 client.9534 ms_handle_reset on v2:172.21.15.172:6833/253283745 2023-10-16T10:04:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d2401b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 --2- 172.21.15.158:0/2999819074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d24020f30 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.555+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 >> 172.21.15.158:0/2999819074 conn(0x563d23f21d80 msgr2=0x563d2403a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.556+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 shutdown_connections 2023-10-16T10:04:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.556+0000 7fa234324b80 1 -- 172.21.15.158:0/2999819074 wait complete. 2023-10-16T10:04:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.556+0000 7fa234324b80 20 client.9534 trim_cache size 0 max 16384 2023-10-16T10:04:10.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 Processor -- start 2023-10-16T10:04:10.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 -- start start 2023-10-16T10:04:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d240161d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d2401b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d2409a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d23dc78d0 con 0x563d24017220 2023-10-16T10:04:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d23ff3110 con 0x563d2401c630 2023-10-16T10:04:10.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.574+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d24016710 con 0x563d23ff2010 2023-10-16T10:04:10.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d2401b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d2401b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59888/0 (socket says 172.21.15.158:59888) 2023-10-16T10:04:10.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 -- 172.21.15.158:0/4062467075 learned_addr learned my addr 172.21.15.158:0/4062467075 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:10.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2266d7700 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d2409a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa225ed6700 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d240161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 -- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 msgr2=0x563d240161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d240161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 -- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 msgr2=0x563d2409a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d2409a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 -- 172.21.15.158:0/4062467075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d23d5ac10 con 0x563d24017220 2023-10-16T10:04:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2266d7700 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d2409a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa225ed6700 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d240161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa2256d5700 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d2401b8c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fa21c006fd0 tx=0x7fa21c020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537657372ed7b71b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa203fff700 1 -- 172.21.15.158:0/4062467075 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c007b80 con 0x563d24017220 2023-10-16T10:04:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa203fff700 1 -- 172.21.15.158:0/4062467075 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c006860 con 0x563d24017220 2023-10-16T10:04:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.575+0000 7fa203fff700 1 -- 172.21.15.158:0/4062467075 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c007070 con 0x563d24017220 2023-10-16T10:04:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 -- 172.21.15.158:0/4062467075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 msgr2=0x563d2401b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d2401b8c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fa21c006fd0 tx=0x7fa21c020070 comp rx=0 tx=0).stop 2023-10-16T10:04:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 -- 172.21.15.158:0/4062467075 shutdown_connections 2023-10-16T10:04:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d2401c630 0x563d2409a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d240161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 --2- 172.21.15.158:0/4062467075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d24017220 0x563d2401b8c0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 -- 172.21.15.158:0/4062467075 >> 172.21.15.158:0/4062467075 conn(0x563d23f21d80 msgr2=0x563d2403a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 -- 172.21.15.158:0/4062467075 shutdown_connections 2023-10-16T10:04:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 -- 172.21.15.158:0/4062467075 wait complete. 2023-10-16T10:04:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 -- start start 2023-10-16T10:04:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.576+0000 7fa234324b80 1 Processor -- start 2023-10-16T10:04:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d24099500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa234324b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d24017220 0x563d24022300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa234324b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d24022840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa234324b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d24099a40 con 0x563d2401c630 2023-10-16T10:04:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d24037ff0 con 0x563d24017220 2023-10-16T10:04:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa234324b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d240203d0 con 0x563d23ff2010 2023-10-16T10:04:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa234324b80 10 client.0 _ll_register_callbacks cb 0x7fa21c0015e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T10:04:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa2266d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d24022840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa2266d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d24022840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59896/0 (socket says 172.21.15.158:59896) 2023-10-16T10:04:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa2266d7700 1 -- 172.21.15.158:0/3914544776 learned_addr learned my addr 172.21.15.158:0/3914544776 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa225ed6700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d24099500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.577+0000 7fa2256d5700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d24017220 0x563d24022300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa2266d7700 1 -- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 msgr2=0x563d24099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa2266d7700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d24099500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa2266d7700 1 -- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d24017220 msgr2=0x563d24022300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa2266d7700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d24017220 0x563d24022300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa2266d7700 1 -- 172.21.15.158:0/3914544776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c00f040 con 0x563d2401c630 2023-10-16T10:04:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa2256d5700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d24017220 0x563d24022300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa2266d7700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d24022840 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fa2180894d0 tx=0x7fa218059680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e396f46d88ba6bbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa200ff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T10:04:10.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa200ff9700 1 -- 172.21.15.158:0/3914544776 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21808b2b0 con 0x563d2401c630 2023-10-16T10:04:10.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa200ff9700 1 -- 172.21.15.158:0/3914544776 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fa218059d20 con 0x563d2401c630 2023-10-16T10:04:10.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa21c005440 con 0x563d2401c630 2023-10-16T10:04:10.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa200ff9700 1 -- 172.21.15.158:0/3914544776 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa21805de70 con 0x563d2401c630 2023-10-16T10:04:10.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 10 client.9540 fetch_fsmap learned FSMap version 77 2023-10-16T10:04:10.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x563d23d98200 con 0x563d2401c630 2023-10-16T10:04:10.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa200ff9700 1 -- 172.21.15.158:0/3914544776 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c005440 con 0x563d2401c630 2023-10-16T10:04:10.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa200ff9700 1 -- 172.21.15.158:0/3914544776 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 77) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa218059f80 con 0x563d2401c630 2023-10-16T10:04:10.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 10 client.9540 fetch_fsmap finished waiting for FSMap version 77 2023-10-16T10:04:10.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 10 client.9540 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T10:04:10.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 20 client.9540 populate_metadata read hostname 'smithi158' 2023-10-16T10:04:10.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d240370e0 0x563d240374c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 4 client.9540 mds_command: new command op to 37352 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T10:04:10.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 --> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d23f3fb20 con 0x563d240370e0 2023-10-16T10:04:10.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.578+0000 7fa225ed6700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d240370e0 0x563d240374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa200ff9700 10 client.9540 ms_handle_connect on v2:172.21.15.172:6833/253283745 2023-10-16T10:04:10.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa225ed6700 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d240370e0 0x563d240374c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a4c927009136a11 server_cookie=52d2b23cc21e1830 in_seq=0 out_seq=0 2023-10-16T10:04:10.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa200ff9700 1 -- 172.21.15.158:0/3914544776 <== mds.0 v2:172.21.15.172:6833/253283745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x563d23f3fb20 con 0x563d240370e0 2023-10-16T10:04:10.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa200ff9700 10 client.9540 handle_command_reply: tid=0 2023-10-16T10:04:10.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 client.9540 shutdown 2023-10-16T10:04:10.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 msgr2=0x563d24022840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d24022840 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fa2180894d0 tx=0x7fa218059680 comp rx=0 tx=0).stop 2023-10-16T10:04:10.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 shutdown_connections 2023-10-16T10:04:10.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:6833/253283745,v1:172.21.15.172:6835/253283745] conn(0x563d240370e0 0x563d240374c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x563d2401c630 0x563d24022840 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa200ff9700 0 client.9540 ms_handle_reset on v2:172.21.15.172:6833/253283745 2023-10-16T10:04:10.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x563d23ff2010 0x563d24099500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 --2- 172.21.15.158:0/3914544776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563d24017220 0x563d24022300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 >> 172.21.15.158:0/3914544776 conn(0x563d23f21d80 msgr2=0x563d24019820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 shutdown_connections 2023-10-16T10:04:10.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 1 -- 172.21.15.158:0/3914544776 wait complete. 2023-10-16T10:04:10.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.579+0000 7fa234324b80 20 client.9540 trim_cache size 0 max 16384 2023-10-16T10:04:10.603 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T10:04:10.603 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T10:04:10.604 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T10:04:10.604 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 37352, 2023-10-16T10:04:10.604 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T10:04:10.604 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T10:04:10.604 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T10:04:10.604 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.8976896830000003, 2023-10-16T10:04:10.605 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 77, 2023-10-16T10:04:10.605 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 153, 2023-10-16T10:04:10.605 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 153, 2023-10-16T10:04:10.605 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.842418644 2023-10-16T10:04:10.605 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T10:04:10.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa20406b020 msgr2=0x7fa20406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa20406b020 0x7fa20406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa22006b120 tx=0x7fa210008040 comp rx=0 tx=0).stop 2023-10-16T10:04:10.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa220160cf0 msgr2=0x7fa2200d0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa220160cf0 0x7fa2200d0260 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fa2200d1480 tx=0x7fa218004ad0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 shutdown_connections 2023-10-16T10:04:10.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa220160cf0 0x7fa2200d0260 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fa20406b020 0x7fa20406d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2200b3290 0x7fa2200ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 --2- 172.21.15.158:0/3271901935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2200cf7c0 0x7fa2201470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 >> 172.21.15.158:0/3271901935 conn(0x7fa22006da00 msgr2=0x7fa22015ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 shutdown_connections 2023-10-16T10:04:10.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.580+0000 7fa22813a700 1 -- 172.21.15.158:0/3271901935 wait complete. 2023-10-16T10:04:10.610 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T10:04:10.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.730+0000 7fb98bed7700 1 Processor -- start 2023-10-16T10:04:10.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.730+0000 7fb98bed7700 1 -- start start 2023-10-16T10:04:10.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98bed7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb98406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98bed7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb9840729b0 0x7fb984071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98bed7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb984071e50 con 0x7fb98406ab40 2023-10-16T10:04:10.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98bed7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb984161270 con 0x7fb9840729b0 2023-10-16T10:04:10.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98bed7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb984161510 con 0x7fb98406e5d0 2023-10-16T10:04:10.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb989c73700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb98406ab40 0x7fb9840722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60076/0 (socket says 172.21.15.158:60076) 2023-10-16T10:04:10.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb989c73700 1 -- 172.21.15.158:0/3903427906 learned_addr learned my addr 172.21.15.158:0/3903427906 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:10.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98a474700 1 --2- 172.21.15.158:0/3903427906 >> v2:172.21.15.158:3300/0 conn(0x7fb9840729b0 0x7fb984071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb989472700 1 --2- 172.21.15.158:0/3903427906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb98406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb988c71700 1 -- 172.21.15.158:0/3903427906 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896104548 0 0) 0x7fb984071e50 con 0x7fb98406ab40 2023-10-16T10:04:10.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb988c71700 1 -- 172.21.15.158:0/3903427906 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9700035b0 con 0x7fb98406ab40 2023-10-16T10:04:10.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98a474700 1 -- 172.21.15.158:0/3903427906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 msgr2=0x7fb98406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:10.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98a474700 1 --2- 172.21.15.158:0/3903427906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb98406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98a474700 1 -- 172.21.15.158:0/3903427906 >> v1:172.21.15.158:6789/0 conn(0x7fb98406ab40 legacy=0x7fb9840722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98a474700 1 -- 172.21.15.158:0/3903427906 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9841617f0 con 0x7fb9840729b0 2023-10-16T10:04:10.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.731+0000 7fb98a474700 1 --2- 172.21.15.158:0/3903427906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb984071910 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009d90 tx=0x7fb97c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42e21f8b0bb02fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb988c71700 1 -- 172.21.15.158:0/3903427906 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00e070 con 0x7fb9840729b0 2023-10-16T10:04:10.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb988c71700 1 -- 172.21.15.158:0/3903427906 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c00bb20 con 0x7fb9840729b0 2023-10-16T10:04:10.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb988c71700 1 -- 172.21.15.158:0/3903427906 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c012430 con 0x7fb9840729b0 2023-10-16T10:04:10.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 -- 172.21.15.158:0/3903427906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 msgr2=0x7fb984071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3903427906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb984071910 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009d90 tx=0x7fb97c00b330 comp rx=0 tx=0).stop 2023-10-16T10:04:10.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 -- 172.21.15.158:0/3903427906 shutdown_connections 2023-10-16T10:04:10.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3903427906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb984071910 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3903427906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb98406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 -- 172.21.15.158:0/3903427906 >> 172.21.15.158:0/3903427906 conn(0x7fb9840d02b0 msgr2=0x7fb9840d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 -- 172.21.15.158:0/3903427906 shutdown_connections 2023-10-16T10:04:10.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.732+0000 7fb98bed7700 1 -- 172.21.15.158:0/3903427906 wait complete. 2023-10-16T10:04:10.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.733+0000 7fb98bed7700 1 Processor -- start 2023-10-16T10:04:10.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98bed7700 1 -- start start 2023-10-16T10:04:10.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98bed7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb98406ab40 0x7fb984148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98bed7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb9840b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98bed7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb9840b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98bed7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb984148da0 con 0x7fb9840729b0 2023-10-16T10:04:10.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98bed7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9841492e0 con 0x7fb98406ab40 2023-10-16T10:04:10.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98bed7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb984160d10 con 0x7fb98406e5d0 2023-10-16T10:04:10.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98a474700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb9840b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98a474700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb9840b55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59910/0 (socket says 172.21.15.158:59910) 2023-10-16T10:04:10.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb98a474700 1 -- 172.21.15.158:0/3341039138 learned_addr learned my addr 172.21.15.158:0/3341039138 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:10.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.734+0000 7fb989c73700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb98406ab40 0x7fb984148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb989472700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb9840b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98a474700 1 -- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 msgr2=0x7fb9840b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98a474700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb9840b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98a474700 1 -- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb98406ab40 msgr2=0x7fb984148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98a474700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb98406ab40 0x7fb984148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98a474700 1 -- 172.21.15.158:0/3341039138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c009a70 con 0x7fb9840729b0 2023-10-16T10:04:10.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb989c73700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb98406ab40 0x7fb984148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T10:04:10.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98a474700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb9840b55b0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fb97c014040 tx=0x7fb97c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb97a7fc700 1 -- 172.21.15.158:0/3341039138 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c012430 con 0x7fb9840729b0 2023-10-16T10:04:10.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb97a7fc700 1 -- 172.21.15.158:0/3341039138 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c020070 con 0x7fb9840729b0 2023-10-16T10:04:10.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb97a7fc700 1 -- 172.21.15.158:0/3341039138 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c003ea0 con 0x7fb9840729b0 2023-10-16T10:04:10.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb984149580 con 0x7fb9840729b0 2023-10-16T10:04:10.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.735+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb984160ff0 con 0x7fb9840729b0 2023-10-16T10:04:10.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.736+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9840681b0 con 0x7fb9840729b0 2023-10-16T10:04:10.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.736+0000 7fb97a7fc700 1 -- 172.21.15.158:0/3341039138 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb97c004280 con 0x7fb9840729b0 2023-10-16T10:04:10.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.737+0000 7fb97a7fc700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb96806b020 0x7fb96806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:10.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.737+0000 7fb97a7fc700 1 -- 172.21.15.158:0/3341039138 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fb97c022070 con 0x7fb9840729b0 2023-10-16T10:04:10.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.737+0000 7fb989c73700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb96806b020 0x7fb96806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:10.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.737+0000 7fb989c73700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb96806b020 0x7fb96806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9800011e0 tx=0x7fb980009040 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:10.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.741+0000 7fb97a7fc700 1 -- 172.21.15.158:0/3341039138 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb97c056480 con 0x7fb9840729b0 2023-10-16T10:04:10.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.979+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb9840b6230 con 0x7fb9840729b0 2023-10-16T10:04:10.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.980+0000 7fb97a7fc700 1 -- 172.21.15.158:0/3341039138 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb97c059aa0 con 0x7fb9840729b0 2023-10-16T10:04:10.982 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T10:04:10.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb96806b020 msgr2=0x7fb96806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb96806b020 0x7fb96806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9800011e0 tx=0x7fb980009040 comp rx=0 tx=0).stop 2023-10-16T10:04:10.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 msgr2=0x7fb9840b55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:10.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb9840b55b0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fb97c014040 tx=0x7fb97c004ad0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 shutdown_connections 2023-10-16T10:04:10.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9840729b0 0x7fb9840b55b0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb98406ab40 0x7fb984148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fb96806b020 0x7fb96806d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 --2- 172.21.15.158:0/3341039138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb98406e5d0 0x7fb9840b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:10.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 >> 172.21.15.158:0/3341039138 conn(0x7fb9840d02b0 msgr2=0x7fb9840ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:10.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 shutdown_connections 2023-10-16T10:04:10.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:10.982+0000 7fb98bed7700 1 -- 172.21.15.158:0/3341039138 wait complete. 2023-10-16T10:04:10.995 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T10:04:10.996 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T10:04:10.996 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T10:04:10.996 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T10:04:10.997 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T10:04:10.997 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T10:04:10.997 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T10:04:10.998 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T10:04:10.998 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:04:10.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:10.999 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T10:04:11.016 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T10:04:11.016 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T10:04:11.017 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T10:04:11.017 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:04:11.017 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T10:04:11.017 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:04:11.017 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:04:11.018 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:04:11.018 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:04:11.018 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T10:04:11.018 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T10:04:11.018 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T10:04:11.019 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T10:04:11.019 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T10:04:11.019 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T10:04:11.019 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 41182sec preferred_lft 41182sec 2023-10-16T10:04:11.019 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T10:04:11.020 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:04:11.020 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T10:04:11.020 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T10:04:11.020 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T10:04:11.020 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:04:11.021 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T10:04:11.021 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:04:11.021 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:04:11.021 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T10:04:11.021 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T10:04:11.021 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:04:11.022 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T10:04:11.022 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T10:04:11.022 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T10:04:11.022 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T10:04:11.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.023 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T10:04:11.082 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T10:04:11.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.084 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T10:04:11.142 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T10:04:11.142 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T10:04:11.142 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T10:04:11.142 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T10:04:11.143 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T10:04:11.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.144 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T10:04:11.144 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T10:04:11.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T10:04:11.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T10:04:11.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T10:04:11.146 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T10:04:11.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.249 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T10:04:11.266 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T10:04:11.266 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T10:04:11.266 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:04:11.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.267 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T10:04:11.333 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:04:11.333 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T10:04:11.389 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T10:04:11.390 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T10:04:11.390 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:04:11.446 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T10:04:11.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.447 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:04:11.531 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:04:11.532 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:04:11.558 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:04:11.558 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:04:11.558 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:04:11.559 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:04:11.559 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:04:11.559 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:04:11.559 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:04:11.559 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:04:11.636 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T10:04:11.637 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T10:04:11.637 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T10:04:11.637 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T10:04:11.637 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T10:04:11.637 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T10:04:11.638 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T10:04:11.638 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T10:04:11.638 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T10:04:11.638 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T10:04:11.638 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T10:04:11.639 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T10:04:11.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.640 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T10:04:11.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:11.667 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T10:04:11.758 INFO:teuthology.orchestra.run.smithi158.stdout:client8545 2023-10-16T10:04:11.759 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T10:04:11.759 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T10:04:11.776 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T10:04:11.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.937+0000 7f4512cd2700 1 Processor -- start 2023-10-16T10:04:11.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.937+0000 7f4512cd2700 1 -- start start 2023-10-16T10:04:11.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.938+0000 7f4512cd2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f450c15eb10 0x7f450c15f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:11.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.938+0000 7f4512cd2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c15f6f0 con 0x7f450c0d0ae0 2023-10-16T10:04:11.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.938+0000 7f4512cd2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c15f990 con 0x7f450c0cfad0 2023-10-16T10:04:11.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.938+0000 7f4512cd2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c0cdef0 con 0x7f450c15eb10 2023-10-16T10:04:11.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.938+0000 7f450bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f450c0d0ae0 0x7f450c0b4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60096/0 (socket says 172.21.15.158:60096) 2023-10-16T10:04:11.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.938+0000 7f450bfff700 1 -- 172.21.15.158:0/2011567203 learned_addr learned my addr 172.21.15.158:0/2011567203 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:11.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.938+0000 7f4510a6e700 1 --2- 172.21.15.158:0/2011567203 >> v2:172.21.15.172:3300/0 conn(0x7f450c15eb10 0x7f450c15f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:11.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605369043 0 0) 0x7f450c15f6f0 con 0x7f450c0d0ae0 2023-10-16T10:04:11.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc0035b0 con 0x7f450c0d0ae0 2023-10-16T10:04:11.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287729096 0 0) 0x7f450c15f990 con 0x7f450c0cfad0 2023-10-16T10:04:11.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c15f6f0 con 0x7f450c0cfad0 2023-10-16T10:04:11.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184153082 0 0) 0x7f44fc0035b0 con 0x7f450c0d0ae0 2023-10-16T10:04:11.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f4510a6e700 1 -- 172.21.15.158:0/2011567203 >> v1:172.21.15.172:6789/0 conn(0x7f450c0cfad0 legacy=0x7f450c0cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f4510a6e700 1 -- 172.21.15.158:0/2011567203 >> v1:172.21.15.158:6789/0 conn(0x7f450c0d0ae0 legacy=0x7f450c0b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f4510a6e700 1 -- 172.21.15.158:0/2011567203 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c15fc70 con 0x7f450c15eb10 2023-10-16T10:04:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f4510a6e700 1 --2- 172.21.15.158:0/2011567203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 0x7f450c15f1b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4500009840 tx=0x7f4500009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8e142d6b65421fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8001670 con 0x7f450c15eb10 2023-10-16T10:04:11.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4500004030 con 0x7f450c15eb10 2023-10-16T10:04:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.939+0000 7f450b7fe700 1 -- 172.21.15.158:0/2011567203 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45000043c0 con 0x7f450c15eb10 2023-10-16T10:04:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.940+0000 7f4512cd2700 1 -- 172.21.15.158:0/2011567203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 msgr2=0x7f450c15f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.940+0000 7f4512cd2700 1 --2- 172.21.15.158:0/2011567203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 0x7f450c15f1b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4500009840 tx=0x7f4500009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.940+0000 7f4512cd2700 1 -- 172.21.15.158:0/2011567203 shutdown_connections 2023-10-16T10:04:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.940+0000 7f4512cd2700 1 --2- 172.21.15.158:0/2011567203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 0x7f450c15f1b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:11.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.940+0000 7f4512cd2700 1 -- 172.21.15.158:0/2011567203 >> 172.21.15.158:0/2011567203 conn(0x7f450c0ce040 msgr2=0x7f450c0ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:11.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.940+0000 7f4512cd2700 1 -- 172.21.15.158:0/2011567203 shutdown_connections 2023-10-16T10:04:11.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.940+0000 7f4512cd2700 1 -- 172.21.15.158:0/2011567203 wait complete. 2023-10-16T10:04:11.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.941+0000 7f4512cd2700 1 Processor -- start 2023-10-16T10:04:11.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.941+0000 7f4512cd2700 1 -- start start 2023-10-16T10:04:11.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4512cd2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f450c0cfad0 0x7f450c148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:11.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4512cd2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f450c0d0ae0 0x7f450c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:11.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4512cd2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 0x7f450c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:11.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4512cd2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c149150 con 0x7f450c0cfad0 2023-10-16T10:04:11.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4512cd2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c06b9e0 con 0x7f450c15eb10 2023-10-16T10:04:11.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4512cd2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c0b0200 con 0x7f450c0d0ae0 2023-10-16T10:04:11.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f450c0cfad0 0x7f450c148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:11.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f450c0cfad0 0x7f450c148c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59924/0 (socket says 172.21.15.158:59924) 2023-10-16T10:04:11.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 -- 172.21.15.158:0/1587696750 learned_addr learned my addr 172.21.15.158:0/1587696750 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:11.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f451126f700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 0x7f450c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:11.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f450bfff700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f450c0d0ae0 0x7f450c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:11.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 -- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f450c0d0ae0 msgr2=0x7f450c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:11.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f450c0d0ae0 0x7f450c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:11.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 -- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 msgr2=0x7f450c0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:11.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 0x7f450c0acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:11.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 -- 172.21.15.158:0/1587696750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45000094a0 con 0x7f450c0cfad0 2023-10-16T10:04:11.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f4510a6e700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f450c0cfad0 0x7f450c148c10 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f450000a010 tx=0x7f4500004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:11.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.942+0000 7f45097fa700 1 -- 172.21.15.158:0/1587696750 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500003d00 con 0x7f450c0cfad0 2023-10-16T10:04:11.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.943+0000 7f45097fa700 1 -- 172.21.15.158:0/1587696750 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450001f070 con 0x7f450c0cfad0 2023-10-16T10:04:11.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.943+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c06bca0 con 0x7f450c0cfad0 2023-10-16T10:04:11.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.943+0000 7f45097fa700 1 -- 172.21.15.158:0/1587696750 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500011870 con 0x7f450c0cfad0 2023-10-16T10:04:11.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.943+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c06c260 con 0x7f450c0cfad0 2023-10-16T10:04:11.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.943+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c06c450 con 0x7f450c0cfad0 2023-10-16T10:04:11.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.944+0000 7f45097fa700 1 -- 172.21.15.158:0/1587696750 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4500021030 con 0x7f450c0cfad0 2023-10-16T10:04:11.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.944+0000 7f45097fa700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f44f006b020 0x7f44f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:11.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.944+0000 7f45097fa700 1 -- 172.21.15.158:0/1587696750 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f450008acc0 con 0x7f450c0cfad0 2023-10-16T10:04:11.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.944+0000 7f450bfff700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f44f006b020 0x7f44f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:11.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.945+0000 7f450bfff700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f44f006b020 0x7f44f006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f450c0ca1c0 tx=0x7f44f8001c00 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:11.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:11.949+0000 7f45097fa700 1 -- 172.21.15.158:0/1587696750 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4500055b00 con 0x7f450c0cfad0 2023-10-16T10:04:12.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.171+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 --> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f450c0cf5b0 con 0x7f44f006b020 2023-10-16T10:04:12.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.172+0000 7f45097fa700 1 -- 172.21.15.158:0/1587696750 <== mgr.9141 v2:172.21.15.158:6832/3458664017 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f44f8001510 con 0x7f44f006b020 2023-10-16T10:04:12.173 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T10:04:12.174 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T10:04:12.174 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T10:04:12.174 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T10:04:12.174 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T10:04:12.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f44f006b020 msgr2=0x7f44f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f44f006b020 0x7f44f006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f450c0ca1c0 tx=0x7f44f8001c00 comp rx=0 tx=0).stop 2023-10-16T10:04:12.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f450c0cfad0 msgr2=0x7f450c148c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f450c0cfad0 0x7f450c148c10 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f450000a010 tx=0x7f4500004750 comp rx=0 tx=0).stop 2023-10-16T10:04:12.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 shutdown_connections 2023-10-16T10:04:12.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c15eb10 0x7f450c0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f450c0cfad0 0x7f450c148c10 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f44f006b020 0x7f44f006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 --2- 172.21.15.158:0/1587696750 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f450c0d0ae0 0x7f450c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 >> 172.21.15.158:0/1587696750 conn(0x7f450c0ce040 msgr2=0x7f450c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:12.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 shutdown_connections 2023-10-16T10:04:12.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.174+0000 7f4512cd2700 1 -- 172.21.15.158:0/1587696750 wait complete. 2023-10-16T10:04:12.188 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T10:04:12.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.309+0000 7fae82df4700 1 Processor -- start 2023-10-16T10:04:12.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.309+0000 7fae82df4700 1 -- start start 2023-10-16T10:04:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae82df4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae7c06afe0 0x7fae7c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae82df4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 0x7fae7c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae82df4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c0cbd30 con 0x7fae7c15da40 2023-10-16T10:04:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae82df4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c0cc1c0 con 0x7fae7c06afe0 2023-10-16T10:04:12.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae82df4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c15de20 con 0x7fae7c0cc590 2023-10-16T10:04:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae7c06afe0 0x7fae7c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae7c06afe0 0x7fae7c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59934/0 (socket says 172.21.15.158:59934) 2023-10-16T10:04:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 -- 172.21.15.158:0/272319640 learned_addr learned my addr 172.21.15.158:0/272319640 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae81391700 1 --2- 172.21.15.158:0/272319640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 0x7fae7c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 -- 172.21.15.158:0/272319640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 msgr2=0x7fae7c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 --2- 172.21.15.158:0/272319640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 0x7fae7c0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 -- 172.21.15.158:0/272319640 >> v1:172.21.15.172:6789/0 conn(0x7fae7c15da40 legacy=0x7fae7c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 -- 172.21.15.158:0/272319640 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c0ce260 con 0x7fae7c06afe0 2023-10-16T10:04:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7bfff700 1 --2- 172.21.15.158:0/272319640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c0caa20 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fae680098f0 tx=0x7fae68009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10563f1fdb3bf1a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:12.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7b7fe700 1 -- 172.21.15.158:0/272319640 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6800b8a0 con 0x7fae7c06afe0 2023-10-16T10:04:12.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.310+0000 7fae7b7fe700 1 -- 172.21.15.158:0/272319640 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6800ba00 con 0x7fae7c06afe0 2023-10-16T10:04:12.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae7b7fe700 1 -- 172.21.15.158:0/272319640 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6800bd90 con 0x7fae7c06afe0 2023-10-16T10:04:12.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 -- 172.21.15.158:0/272319640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 msgr2=0x7fae7c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 --2- 172.21.15.158:0/272319640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c0caa20 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fae680098f0 tx=0x7fae68009c00 comp rx=0 tx=0).stop 2023-10-16T10:04:12.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 -- 172.21.15.158:0/272319640 shutdown_connections 2023-10-16T10:04:12.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 --2- 172.21.15.158:0/272319640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 0x7fae7c0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 --2- 172.21.15.158:0/272319640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c0caa20 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 -- 172.21.15.158:0/272319640 >> 172.21.15.158:0/272319640 conn(0x7fae7c099a10 msgr2=0x7fae7c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:12.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 -- 172.21.15.158:0/272319640 shutdown_connections 2023-10-16T10:04:12.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.311+0000 7fae82df4700 1 -- 172.21.15.158:0/272319640 wait complete. 2023-10-16T10:04:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.312+0000 7fae82df4700 1 Processor -- start 2023-10-16T10:04:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.312+0000 7fae82df4700 1 -- start start 2023-10-16T10:04:12.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae82df4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae82df4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 0x7fae7c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae82df4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae7c15da40 0x7fae7c14f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae82df4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c15c9d0 con 0x7fae7c06afe0 2023-10-16T10:04:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae82df4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0c44c0 con 0x7fae7c15da40 2023-10-16T10:04:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae82df4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0c47a0 con 0x7fae7c0cc590 2023-10-16T10:04:12.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae80b90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae80b90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c15cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59940/0 (socket says 172.21.15.158:59940) 2023-10-16T10:04:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.313+0000 7fae80b90700 1 -- 172.21.15.158:0/2079202430 learned_addr learned my addr 172.21.15.158:0/2079202430 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae80b90700 1 -- 172.21.15.158:0/2079202430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 msgr2=0x7fae7c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae80b90700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 0x7fae7c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae80b90700 1 -- 172.21.15.158:0/2079202430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae7c15da40 msgr2=0x7fae7c14f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae80b90700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae7c15da40 0x7fae7c14f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae80b90700 1 -- 172.21.15.158:0/2079202430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae68009550 con 0x7fae7c06afe0 2023-10-16T10:04:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae80b90700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c15cbf0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fae7000c490 tx=0x7fae7000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae797fa700 1 -- 172.21.15.158:0/2079202430 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae700052d0 con 0x7fae7c06afe0 2023-10-16T10:04:12.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae797fa700 1 -- 172.21.15.158:0/2079202430 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae70005450 con 0x7fae7c06afe0 2023-10-16T10:04:12.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae797fa700 1 -- 172.21.15.158:0/2079202430 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70006aa0 con 0x7fae7c06afe0 2023-10-16T10:04:12.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c0c4a80 con 0x7fae7c06afe0 2023-10-16T10:04:12.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.314+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c0c5040 con 0x7fae7c06afe0 2023-10-16T10:04:12.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.315+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c0681b0 con 0x7fae7c06afe0 2023-10-16T10:04:12.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.315+0000 7fae797fa700 1 -- 172.21.15.158:0/2079202430 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fae70003680 con 0x7fae7c06afe0 2023-10-16T10:04:12.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.316+0000 7fae797fa700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fae6006b020 0x7fae6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.316+0000 7fae797fa700 1 -- 172.21.15.158:0/2079202430 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fae70088c90 con 0x7fae7c06afe0 2023-10-16T10:04:12.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.316+0000 7fae7bfff700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fae6006b020 0x7fae6006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.316+0000 7fae7bfff700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fae6006b020 0x7fae6006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0d1000 tx=0x7fae68010000 comp rx=0 tx=0).ready entity=mgr.9141 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:12.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.320+0000 7fae797fa700 1 -- 172.21.15.158:0/2079202430 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae700539b0 con 0x7fae7c06afe0 2023-10-16T10:04:12.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.570+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fae7c0cd820 con 0x7fae7c06afe0 2023-10-16T10:04:12.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.708+0000 7fae797fa700 1 -- 172.21.15.158:0/2079202430 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v86) v1 ==== 92+0+0 (secure 0 0 0) 0x7fae70056fd0 con 0x7fae7c06afe0 2023-10-16T10:04:12.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.711+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fae6006b020 msgr2=0x7fae6006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.711+0000 7fae82df4700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fae6006b020 0x7fae6006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0d1000 tx=0x7fae68010000 comp rx=0 tx=0).stop 2023-10-16T10:04:12.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.711+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 msgr2=0x7fae7c15cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.711+0000 7fae82df4700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c15cbf0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fae7000c490 tx=0x7fae7000c850 comp rx=0 tx=0).stop 2023-10-16T10:04:12.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 shutdown_connections 2023-10-16T10:04:12.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae7c15da40 0x7fae7c14f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae7c06afe0 0x7fae7c15cbf0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7fae6006b020 0x7fae6006d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 --2- 172.21.15.158:0/2079202430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae7c0cc590 0x7fae7c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 >> 172.21.15.158:0/2079202430 conn(0x7fae7c099a10 msgr2=0x7fae7c0b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:12.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 shutdown_connections 2023-10-16T10:04:12.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.712+0000 7fae82df4700 1 -- 172.21.15.158:0/2079202430 wait complete. 2023-10-16T10:04:12.727 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T10:04:12.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.854+0000 7f3e0e218700 1 Processor -- start 2023-10-16T10:04:12.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.855+0000 7f3e0e218700 1 -- start start 2023-10-16T10:04:12.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.855+0000 7f3e0e218700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3e0815dad0 0x7f3e080cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.855+0000 7f3e0e218700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3e0815deb0 0x7f3e080ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.855+0000 7f3e0e218700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e080c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.855+0000 7f3e0e218700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e080cee50 con 0x7f3e0815dad0 2023-10-16T10:04:12.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.855+0000 7f3e0e218700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e080d0ae0 con 0x7f3e0815deb0 2023-10-16T10:04:12.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.855+0000 7f3e0e218700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080d12d0 con 0x7f3e080cea20 2023-10-16T10:04:12.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3e0815dad0 0x7f3e080cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3e0815dad0 0x7f3e080cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59972/0 (socket says 172.21.15.158:59972) 2023-10-16T10:04:12.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 -- 172.21.15.158:0/3992084326 learned_addr learned my addr 172.21.15.158:0/3992084326 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:12.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3dfffff700 1 --2- 172.21.15.158:0/3992084326 >> v2:172.21.15.172:3300/0 conn(0x7f3e0815deb0 0x7f3e080ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e07fff700 1 --2- 172.21.15.158:0/3992084326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e080c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 -- 172.21.15.158:0/3992084326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 msgr2=0x7f3e080c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:12.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 --2- 172.21.15.158:0/3992084326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e080c3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 -- 172.21.15.158:0/3992084326 >> v2:172.21.15.172:3300/0 conn(0x7f3e0815deb0 msgr2=0x7f3e080ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:12.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 --2- 172.21.15.158:0/3992084326 >> v2:172.21.15.172:3300/0 conn(0x7f3e0815deb0 0x7f3e080ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 -- 172.21.15.158:0/3992084326 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080cbdf0 con 0x7f3e0815dad0 2023-10-16T10:04:12.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e077fe700 1 --2- 172.21.15.158:0/3992084326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815dad0 0x7f3e080cdfa0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3e08072c30 tx=0x7f3df800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7167f6d2fb94b45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:12.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e06ffd700 1 -- 172.21.15.158:0/3992084326 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df800dcf0 con 0x7f3e0815dad0 2023-10-16T10:04:12.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e06ffd700 1 -- 172.21.15.158:0/3992084326 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df800de50 con 0x7f3e0815dad0 2023-10-16T10:04:12.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.856+0000 7f3e06ffd700 1 -- 172.21.15.158:0/3992084326 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8010690 con 0x7f3e0815dad0 2023-10-16T10:04:12.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 -- 172.21.15.158:0/3992084326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815dad0 msgr2=0x7f3e080cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 --2- 172.21.15.158:0/3992084326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815dad0 0x7f3e080cdfa0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3e08072c30 tx=0x7f3df800bea0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 -- 172.21.15.158:0/3992084326 shutdown_connections 2023-10-16T10:04:12.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 --2- 172.21.15.158:0/3992084326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e080c3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 --2- 172.21.15.158:0/3992084326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815dad0 0x7f3e080cdfa0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 --2- 172.21.15.158:0/3992084326 >> v2:172.21.15.172:3300/0 conn(0x7f3e0815deb0 0x7f3e080ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 -- 172.21.15.158:0/3992084326 >> 172.21.15.158:0/3992084326 conn(0x7f3e08099a10 msgr2=0x7f3e080d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:12.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 -- 172.21.15.158:0/3992084326 shutdown_connections 2023-10-16T10:04:12.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.857+0000 7f3e0e218700 1 -- 172.21.15.158:0/3992084326 wait complete. 2023-10-16T10:04:12.864 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:12.856+0000 7fbb3082d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:04:12.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.858+0000 7f3e0e218700 1 Processor -- start 2023-10-16T10:04:12.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.858+0000 7f3e0e218700 1 -- start start 2023-10-16T10:04:12.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e0e218700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e0814aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e0e218700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0815dad0 0x7f3e080c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e0e218700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815deb0 0x7f3e08149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e0e218700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080c83f0 con 0x7f3e0815deb0 2023-10-16T10:04:12.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e0e218700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080c8620 con 0x7f3e0815dad0 2023-10-16T10:04:12.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e0e218700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080c8b40 con 0x7f3e080cea20 2023-10-16T10:04:12.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e07fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815deb0 0x7f3e08149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3dfffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0815dad0 0x7f3e080c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e07fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815deb0 0x7f3e08149c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:59982/0 (socket says 172.21.15.158:59982) 2023-10-16T10:04:12.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e07fff700 1 -- 172.21.15.158:0/322274786 learned_addr learned my addr 172.21.15.158:0/322274786 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:12.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e077fe700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e0814aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:12.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3dfffff700 1 -- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 msgr2=0x7f3e0814aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3dfffff700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e0814aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3dfffff700 1 -- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815deb0 msgr2=0x7f3e08149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:12.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3dfffff700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815deb0 0x7f3e08149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:12.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3dfffff700 1 -- 172.21.15.158:0/322274786 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df800b7f0 con 0x7f3e0815dad0 2023-10-16T10:04:12.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3e07fff700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815deb0 0x7f3e08149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:04:12.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.859+0000 7f3dfffff700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0815dad0 0x7f3e080c7eb0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3df000e870 tx=0x7f3df000eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:12.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.860+0000 7f3e04ff9700 1 -- 172.21.15.158:0/322274786 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0013680 con 0x7f3e0815dad0 2023-10-16T10:04:12.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.860+0000 7f3e04ff9700 1 -- 172.21.15.158:0/322274786 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df00137e0 con 0x7f3e0815dad0 2023-10-16T10:04:12.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.860+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e080c8e20 con 0x7f3e0815dad0 2023-10-16T10:04:12.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.860+0000 7f3e04ff9700 1 -- 172.21.15.158:0/322274786 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0013b90 con 0x7f3e0815dad0 2023-10-16T10:04:12.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.860+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e080c93e0 con 0x7f3e0815dad0 2023-10-16T10:04:12.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.861+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e080cc260 con 0x7f3e0815dad0 2023-10-16T10:04:12.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.861+0000 7f3e04ff9700 1 -- 172.21.15.158:0/322274786 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 86) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3df0013cf0 con 0x7f3e0815dad0 2023-10-16T10:04:12.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.861+0000 7f3e04ff9700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 0x7f3de806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:12.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.862+0000 7f3e077fe700 1 -- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 msgr2=0x7f3de806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:12.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.862+0000 7f3e04ff9700 1 -- 172.21.15.158:0/322274786 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f3df0089880 con 0x7f3e0815dad0 2023-10-16T10:04:12.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.862+0000 7f3e077fe700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 0x7f3de806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:04:12.873 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:12.864+0000 7efce7872200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T10:04:12.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:12.866+0000 7f3e04ff9700 1 -- 172.21.15.158:0/322274786 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e080cc260 con 0x7f3e0815dad0 2023-10-16T10:04:12.924 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:12.923+0000 7fbb3082d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:04:12.934 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:12.932+0000 7efce7872200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T10:04:12.997 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:12.996+0000 7fbb3082d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:04:13.007 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:13.006+0000 7efce7872200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T10:04:13.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.063+0000 7f3e077fe700 1 -- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 msgr2=0x7f3de806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:13.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.063+0000 7f3e077fe700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 0x7f3de806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:04:13.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.066+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3e080cc260 con 0x7f3e0815dad0 2023-10-16T10:04:13.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.071+0000 7f3e04ff9700 1 -- 172.21.15.158:0/322274786 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3e080cc260 con 0x7f3e0815dad0 2023-10-16T10:04:13.075 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:13.074+0000 7fbb3082d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:04:13.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 msgr2=0x7f3de806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:04:13.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 0x7f3de806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0815dad0 msgr2=0x7f3e080c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0815dad0 0x7f3e080c7eb0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3df000e870 tx=0x7f3df000eb80 comp rx=0 tx=0).stop 2023-10-16T10:04:13.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 shutdown_connections 2023-10-16T10:04:13.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0815deb0 0x7f3e08149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f3de806b070 0x7f3de806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.074+0000 7f3e0e218700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e080cea20 0x7f3e0814aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.075+0000 7f3e0e218700 1 --2- 172.21.15.158:0/322274786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0815dad0 0x7f3e080c7eb0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.075+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 >> 172.21.15.158:0/322274786 conn(0x7f3e08099a10 msgr2=0x7f3e080cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:13.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.075+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 shutdown_connections 2023-10-16T10:04:13.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.075+0000 7f3e0e218700 1 -- 172.21.15.158:0/322274786 wait complete. 2023-10-16T10:04:13.094 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T10:04:13.096 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:13.095+0000 7efce7872200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T10:04:13.217 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:13.216+0000 7fbb3082d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:04:13.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.219+0000 7ffb52c1c700 1 Processor -- start 2023-10-16T10:04:13.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.219+0000 7ffb52c1c700 1 -- start start 2023-10-16T10:04:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb52c1c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffb4c160d30 0x7ffb4c0b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb52c1c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb52c1c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c0b7c90 con 0x7ffb4c0d19a0 2023-10-16T10:04:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb52c1c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c0b8910 con 0x7ffb4c160d30 2023-10-16T10:04:13.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb52c1c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0d0ef0 con 0x7ffb4c0d0530 2023-10-16T10:04:13.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffb4c160d30 0x7ffb4c0b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffb4c160d30 0x7ffb4c0b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60032/0 (socket says 172.21.15.158:60032) 2023-10-16T10:04:13.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 -- 172.21.15.158:0/1010354884 learned_addr learned my addr 172.21.15.158:0/1010354884 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:13.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb4bfff700 1 --2- 172.21.15.158:0/1010354884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 -- 172.21.15.158:0/1010354884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 msgr2=0x7ffb4c0d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:13.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 --2- 172.21.15.158:0/1010354884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c0d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 -- 172.21.15.158:0/1010354884 >> v1:172.21.15.172:6789/0 conn(0x7ffb4c0d19a0 legacy=0x7ffb4c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 -- 172.21.15.158:0/1010354884 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0b7f70 con 0x7ffb4c160d30 2023-10-16T10:04:13.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb509b8700 1 --2- 172.21.15.158:0/1010354884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c160d30 0x7ffb4c0b8f50 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7ffb400098f0 tx=0x7ffb40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ba0193ec93ea2a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:13.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.220+0000 7ffb4b7fe700 1 -- 172.21.15.158:0/1010354884 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40011830 con 0x7ffb4c160d30 2023-10-16T10:04:13.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb4b7fe700 1 -- 172.21.15.158:0/1010354884 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb40011990 con 0x7ffb4c160d30 2023-10-16T10:04:13.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb4b7fe700 1 -- 172.21.15.158:0/1010354884 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40011d20 con 0x7ffb4c160d30 2023-10-16T10:04:13.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 -- 172.21.15.158:0/1010354884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c160d30 msgr2=0x7ffb4c0b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/1010354884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c160d30 0x7ffb4c0b8f50 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7ffb400098f0 tx=0x7ffb40009c00 comp rx=0 tx=0).stop 2023-10-16T10:04:13.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 -- 172.21.15.158:0/1010354884 shutdown_connections 2023-10-16T10:04:13.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/1010354884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c160d30 0x7ffb4c0b8f50 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/1010354884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c0d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 -- 172.21.15.158:0/1010354884 >> 172.21.15.158:0/1010354884 conn(0x7ffb4c0669f0 msgr2=0x7ffb4c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:13.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 -- 172.21.15.158:0/1010354884 shutdown_connections 2023-10-16T10:04:13.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.221+0000 7ffb52c1c700 1 -- 172.21.15.158:0/1010354884 wait complete. 2023-10-16T10:04:13.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.222+0000 7ffb52c1c700 1 Processor -- start 2023-10-16T10:04:13.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.222+0000 7ffb52c1c700 1 -- start start 2023-10-16T10:04:13.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb52c1c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c1f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb52c1c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c0d19a0 0x7ffb4c148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb52c1c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4c160d30 0x7ffb4c156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb52c1c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c9f20 con 0x7ffb4c0d19a0 2023-10-16T10:04:13.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb52c1c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1588d0 con 0x7ffb4c160d30 2023-10-16T10:04:13.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb52c1c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c158bb0 con 0x7ffb4c0d0530 2023-10-16T10:04:13.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb4bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c0d19a0 0x7ffb4c148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb4bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c0d19a0 0x7ffb4c148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60040/0 (socket says 172.21.15.158:60040) 2023-10-16T10:04:13.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb4bfff700 1 -- 172.21.15.158:0/263095726 learned_addr learned my addr 172.21.15.158:0/263095726 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:13.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb509b8700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c1f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.223+0000 7ffb511b9700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4c160d30 0x7ffb4c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb509b8700 1 -- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4c160d30 msgr2=0x7ffb4c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb509b8700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4c160d30 0x7ffb4c156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb509b8700 1 -- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c0d19a0 msgr2=0x7ffb4c148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb509b8700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c0d19a0 0x7ffb4c148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb509b8700 1 -- 172.21.15.158:0/263095726 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb40009550 con 0x7ffb4c0d0530 2023-10-16T10:04:13.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb4bfff700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c0d19a0 0x7ffb4c148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:04:13.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb509b8700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c1f2870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffb40016000 tx=0x7ffb40004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:13.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb497fa700 1 -- 172.21.15.158:0/263095726 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40004510 con 0x7ffb4c0d0530 2023-10-16T10:04:13.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb497fa700 1 -- 172.21.15.158:0/263095726 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4001e070 con 0x7ffb4c0d0530 2023-10-16T10:04:13.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb497fa700 1 -- 172.21.15.158:0/263095726 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb400038c0 con 0x7ffb4c0d0530 2023-10-16T10:04:13.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c158e90 con 0x7ffb4c0d0530 2023-10-16T10:04:13.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.224+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c148b50 con 0x7ffb4c0d0530 2023-10-16T10:04:13.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.225+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c0d09b0 con 0x7ffb4c0d0530 2023-10-16T10:04:13.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.226+0000 7ffb497fa700 1 -- 172.21.15.158:0/263095726 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 86) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ffb40003a20 con 0x7ffb4c0d0530 2023-10-16T10:04:13.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.226+0000 7ffb497fa700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 0x7ffb3006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.226+0000 7ffb4bfff700 1 -- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 msgr2=0x7ffb3006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:13.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.226+0000 7ffb497fa700 1 -- 172.21.15.158:0/263095726 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7ffb4008a880 con 0x7ffb4c0d0530 2023-10-16T10:04:13.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.226+0000 7ffb4bfff700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 0x7ffb3006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:04:13.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.231+0000 7ffb497fa700 1 -- 172.21.15.158:0/263095726 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb40055520 con 0x7ffb4c0d0530 2023-10-16T10:04:13.248 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:13.247+0000 7efce7872200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T10:04:13.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.427+0000 7ffb4bfff700 1 -- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 msgr2=0x7ffb3006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:13.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.427+0000 7ffb4bfff700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 0x7ffb3006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:04:13.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.437+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ffb4c0681b0 con 0x7ffb4c0d0530 2023-10-16T10:04:13.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.441+0000 7ffb497fa700 1 -- 172.21.15.158:0/263095726 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7ffb40058b40 con 0x7ffb4c0d0530 2023-10-16T10:04:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 msgr2=0x7ffb3006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:04:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 0x7ffb3006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 msgr2=0x7ffb4c1f2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c1f2870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffb40016000 tx=0x7ffb40004750 comp rx=0 tx=0).stop 2023-10-16T10:04:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 shutdown_connections 2023-10-16T10:04:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4c160d30 0x7ffb4c156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7ffb3006b070 0x7ffb3006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4c0d0530 0x7ffb4c1f2870 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 --2- 172.21.15.158:0/263095726 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb4c0d19a0 0x7ffb4c148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 >> 172.21.15.158:0/263095726 conn(0x7ffb4c0669f0 msgr2=0x7ffb4c066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 shutdown_connections 2023-10-16T10:04:13.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.444+0000 7ffb52c1c700 1 -- 172.21.15.158:0/263095726 wait complete. 2023-10-16T10:04:13.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T10:04:13.457 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T10:04:13.478 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2023-10-16T10:04:13.545 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:13.544+0000 7fbb3082d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:04:13.589 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:13.588+0000 7efce7872200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T10:04:13.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.644+0000 7f1efb774700 1 Processor -- start 2023-10-16T10:04:13.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.644+0000 7f1efb774700 1 -- start start 2023-10-16T10:04:13.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.645+0000 7f1efb774700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415b590 0x7f1ef40ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.645+0000 7f1efb774700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ef415f1d0 0x7f1ef40b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.645+0000 7f1efb774700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef40cf4c0 con 0x7f1ef40cbaf0 2023-10-16T10:04:13.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.645+0000 7f1efb774700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef415f970 con 0x7f1ef415f1d0 2023-10-16T10:04:13.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.645+0000 7f1efb774700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef415fc50 con 0x7f1ef415b590 2023-10-16T10:04:13.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef8d0f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ef415f1d0 0x7f1ef40b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef9d11700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1ef40cbaf0 0x7f1ef40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60270/0 (socket says 172.21.15.158:60270) 2023-10-16T10:04:13.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef8d0f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ef415f1d0 0x7f1ef40b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60094/0 (socket says 172.21.15.158:60094) 2023-10-16T10:04:13.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef9d11700 1 -- 172.21.15.158:0/1649284816 learned_addr learned my addr 172.21.15.158:0/1649284816 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:13.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef9510700 1 --2- 172.21.15.158:0/1649284816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415b590 0x7f1ef40ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef8d0f700 1 -- 172.21.15.158:0/1649284816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415b590 msgr2=0x7f1ef40ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:13.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef8d0f700 1 --2- 172.21.15.158:0/1649284816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415b590 0x7f1ef40ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef8d0f700 1 -- 172.21.15.158:0/1649284816 >> v1:172.21.15.158:6789/0 conn(0x7f1ef40cbaf0 legacy=0x7f1ef40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1eebfff700 1 -- 172.21.15.158:0/1649284816 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511468489 0 0) 0x7f1ef40cf4c0 con 0x7f1ef40cbaf0 2023-10-16T10:04:13.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef8d0f700 1 -- 172.21.15.158:0/1649284816 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef415ff30 con 0x7f1ef415f1d0 2023-10-16T10:04:13.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1ef8d0f700 1 --2- 172.21.15.158:0/1649284816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415f1d0 0x7f1ef40b42d0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009840 tx=0x7f1ee4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe99d1e44ef6f4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:13.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1eebfff700 1 -- 172.21.15.158:0/1649284816 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4011830 con 0x7f1ef415f1d0 2023-10-16T10:04:13.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1eebfff700 1 -- 172.21.15.158:0/1649284816 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee4011990 con 0x7f1ef415f1d0 2023-10-16T10:04:13.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.646+0000 7f1eebfff700 1 -- 172.21.15.158:0/1649284816 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4011d20 con 0x7f1ef415f1d0 2023-10-16T10:04:13.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 -- 172.21.15.158:0/1649284816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415f1d0 msgr2=0x7f1ef40b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 --2- 172.21.15.158:0/1649284816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415f1d0 0x7f1ef40b42d0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009840 tx=0x7f1ee4009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:13.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 -- 172.21.15.158:0/1649284816 shutdown_connections 2023-10-16T10:04:13.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 --2- 172.21.15.158:0/1649284816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415b590 0x7f1ef40ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 --2- 172.21.15.158:0/1649284816 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415f1d0 0x7f1ef40b42d0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 -- 172.21.15.158:0/1649284816 >> 172.21.15.158:0/1649284816 conn(0x7f1ef406a8d0 msgr2=0x7f1ef406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:13.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 -- 172.21.15.158:0/1649284816 shutdown_connections 2023-10-16T10:04:13.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.647+0000 7f1efb774700 1 -- 172.21.15.158:0/1649284816 wait complete. 2023-10-16T10:04:13.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.649+0000 7f1efb774700 1 Processor -- start 2023-10-16T10:04:13.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.649+0000 7f1efb774700 1 -- start start 2023-10-16T10:04:13.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1efb774700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ef40cbaf0 0x7f1ef415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1efb774700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415b590 0x7f1ef40ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1efb774700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415f1d0 0x7f1ef40ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1efb774700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef415d4a0 con 0x7f1ef415b590 2023-10-16T10:04:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1efb774700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41563a0 con 0x7f1ef40cbaf0 2023-10-16T10:04:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1efb774700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef415cc00 con 0x7f1ef415f1d0 2023-10-16T10:04:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415b590 0x7f1ef40ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415b590 0x7f1ef40ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60100/0 (socket says 172.21.15.158:60100) 2023-10-16T10:04:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 -- 172.21.15.158:0/588078623 learned_addr learned my addr 172.21.15.158:0/588078623 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:13.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef9d11700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415f1d0 0x7f1ef40ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ef40cbaf0 0x7f1ef415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:13.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415f1d0 msgr2=0x7f1ef40ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ef415f1d0 0x7f1ef40ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ef40cbaf0 msgr2=0x7f1ef415c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ef40cbaf0 0x7f1ef415c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.650+0000 7f1ef8d0f700 1 -- 172.21.15.158:0/588078623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee40094a0 con 0x7f1ef415b590 2023-10-16T10:04:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.651+0000 7f1ef8d0f700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415b590 0x7f1ef40ca380 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009b90 tx=0x7f1ee4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.651+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee40036a0 con 0x7f1ef415b590 2023-10-16T10:04:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.651+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee401e070 con 0x7f1ef415b590 2023-10-16T10:04:13.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.651+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4003a60 con 0x7f1ef415b590 2023-10-16T10:04:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.651+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef415cee0 con 0x7f1ef415b590 2023-10-16T10:04:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.651+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef4156640 con 0x7f1ef415b590 2023-10-16T10:04:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.652+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef40681b0 con 0x7f1ef415b590 2023-10-16T10:04:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.652+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 86) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1ee4003bc0 con 0x7f1ef415b590 2023-10-16T10:04:13.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.653+0000 7f1ee9ffb700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 0x7f1ed806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.653+0000 7f1ef9510700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 msgr2=0x7f1ed806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.653+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f1ee408a8d0 con 0x7f1ef415b590 2023-10-16T10:04:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.653+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 0x7f1ed806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T10:04:13.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.657+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ee40555f0 con 0x7f1ef415b590 2023-10-16T10:04:13.664 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:13.662+0000 7fbb3082d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:04:13.719 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:13.718+0000 7efce7872200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T10:04:13.738 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:13.737+0000 7fbb3082d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:04:13.795 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:13.793+0000 7efce7872200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T10:04:13.818 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:13.817+0000 7fbb3082d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:04:13.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.854+0000 7f1ef9510700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 msgr2=0x7f1ed806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:13.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.854+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 0x7f1ed806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T10:04:13.876 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:13.875+0000 7efce7872200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T10:04:13.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:13.893+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 --> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ef415b970 con 0x7f1ed806b070 2023-10-16T10:04:14.187 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:14.186+0000 7fbb3082d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:04:14.246 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:14.245+0000 7efce7872200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T10:04:14.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:14.255+0000 7f1ef9510700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 msgr2=0x7f1ed806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:14.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:14.255+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 0x7f1ed806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T10:04:14.759 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:14.757+0000 7fbb3082d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:04:14.818 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:14.817+0000 7efce7872200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T10:04:14.828 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:14.828+0000 7fbb3082d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:04:14.888 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:14.886+0000 7efce7872200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T10:04:15.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:15.056+0000 7f1ef9510700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 msgr2=0x7f1ed806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:15.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:15.056+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 0x7f1ed806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T10:04:15.458 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:15.457+0000 7fbb3082d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:04:15.522 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:15.521+0000 7efce7872200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:04:15.535 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:15.534+0000 7fbb3082d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:04:15.600 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:15.599+0000 7efce7872200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T10:04:15.678 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:15.678+0000 7fbb3082d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:04:15.746 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:15.745+0000 7efce7872200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T10:04:15.905 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:15.904+0000 7fbb3082d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:04:15.976 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:15.975+0000 7efce7872200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T10:04:15.978 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:15.978+0000 7fbb3082d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:04:16.050 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:16.049+0000 7efce7872200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T10:04:16.244 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:16.243+0000 7fbb3082d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:04:16.338 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:16.337+0000 7efce7872200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T10:04:16.464 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:16.464+0000 7fbb3082d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:04:16.564 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:16.563+0000 7efce7872200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T10:04:16.613 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:16.613+0000 7fbb3082d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:04:16.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:16.657+0000 7f1ef9510700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 msgr2=0x7f1ed806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3458664017 2023-10-16T10:04:16.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:16.657+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 0x7f1ed806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T10:04:16.716 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:16.714+0000 7efce7872200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T10:04:16.728 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:16.728+0000 7fbb3082d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:04:16.830 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:16.829+0000 7efce7872200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T10:04:16.884 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:16.884+0000 7fbb3082d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:04:16.988 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:16.986+0000 7efce7872200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T10:04:17.147 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:17.147+0000 7fbb3082d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:04:17.252 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:17.250+0000 7efce7872200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T10:04:17.309 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:17.309+0000 7fbb3082d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:04:17.413 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:17.412+0000 7efce7872200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T10:04:17.834 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T10:04:17.834+0000 7fbb3082d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:04:17.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:17.903+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 87) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1ee40559c0 con 0x7f1ef415b590 2023-10-16T10:04:17.945 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:17.944+0000 7efce7872200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T10:04:17.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:17.963+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 88) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f1ee4088ab0 con 0x7f1ef415b590 2023-10-16T10:04:17.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:17.963+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 msgr2=0x7f1ed806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T10:04:17.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:17.963+0000 7f1ee9ffb700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/3458664017,v1:172.21.15.158:6833/3458664017] conn(0x7f1ed806b070 0x7f1ed806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:18.108 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5d853700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.109 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5d853700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.109 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5d853700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.109 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5d853700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.109 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5d853700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.110 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5a84d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.110 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5a84d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.110 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5a84d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.110 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5a84d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.110 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:04:18.107+0000 7efc5a84d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:04:18.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:18.967+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 89) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1ee4056290 con 0x7f1ef415b590 2023-10-16T10:04:18.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:18.968+0000 7f1ee9ffb700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f1ed80279c0 0x7f1ed8027da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:18.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:18.968+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 --> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ed8071580 con 0x7f1ed80279c0 2023-10-16T10:04:18.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:18.970+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f1ed80279c0 0x7f1ed8027da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:18.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:18.971+0000 7f1ef9510700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f1ed80279c0 0x7f1ed8027da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0010820 tx=0x7f1ef0012040 comp rx=0 tx=0).ready entity=mgr.9573 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:19.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.035+0000 7f1ee9ffb700 1 -- 172.21.15.158:0/588078623 <== mgr.9573 v2:172.21.15.158:6832/2326422563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1ed8071580 con 0x7f1ed80279c0 2023-10-16T10:04:19.037 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1 2023-10-16T10:04:19.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f1ed80279c0 msgr2=0x7f1ed8027da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f1ed80279c0 0x7f1ed8027da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0010820 tx=0x7f1ef0012040 comp rx=0 tx=0).stop 2023-10-16T10:04:19.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415b590 msgr2=0x7f1ef40ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 --2- 172.21.15.158:0/588078623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ef415b590 0x7f1ef40ca380 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009b90 tx=0x7f1ee4004750 comp rx=0 tx=0).stop 2023-10-16T10:04:19.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1ef9d11700 1 -- 172.21.15.158:0/588078623 reap_dead start 2023-10-16T10:04:19.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 shutdown_connections 2023-10-16T10:04:19.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 >> 172.21.15.158:0/588078623 conn(0x7f1ef406a8d0 msgr2=0x7f1ef40cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:19.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 shutdown_connections 2023-10-16T10:04:19.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.038+0000 7f1efb774700 1 -- 172.21.15.158:0/588078623 wait complete. 2023-10-16T10:04:19.052 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-10-16T10:04:19.053 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2023-10-16T10:04:19.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.177+0000 7f36363a2700 1 Processor -- start 2023-10-16T10:04:19.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.177+0000 7f36363a2700 1 -- start start 2023-10-16T10:04:19.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f36363a2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f363015da40 0x7f36300caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f36363a2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f36300cc590 0x7f363006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f36363a2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300cbd30 con 0x7f363006ae10 2023-10-16T10:04:19.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f36363a2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36300cc1c0 con 0x7f363015da40 2023-10-16T10:04:19.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f36363a2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363006b240 con 0x7f36300cc590 2023-10-16T10:04:19.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f362ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f363015da40 0x7f36300caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f362ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f363015da40 0x7f36300caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60278/0 (socket says 172.21.15.158:60278) 2023-10-16T10:04:19.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f362ffff700 1 -- 172.21.15.158:0/2914848420 learned_addr learned my addr 172.21.15.158:0/2914848420 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:19.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f362f7fe700 1 --2- 172.21.15.158:0/2914848420 >> v2:172.21.15.172:3300/0 conn(0x7f36300cc590 0x7f363006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f362ffff700 1 -- 172.21.15.158:0/2914848420 >> v2:172.21.15.172:3300/0 conn(0x7f36300cc590 msgr2=0x7f363006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:19.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f362ffff700 1 --2- 172.21.15.158:0/2914848420 >> v2:172.21.15.172:3300/0 conn(0x7f36300cc590 0x7f363006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.178+0000 7f362ffff700 1 -- 172.21.15.158:0/2914848420 >> v1:172.21.15.172:6789/0 conn(0x7f363006ae10 legacy=0x7f36300ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:19.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f362ffff700 1 -- 172.21.15.158:0/2914848420 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363015de20 con 0x7f363015da40 2023-10-16T10:04:19.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f362f7fe700 1 --2- 172.21.15.158:0/2914848420 >> v2:172.21.15.172:3300/0 conn(0x7f36300cc590 0x7f363006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T10:04:19.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f362ffff700 1 --2- 172.21.15.158:0/2914848420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f363015da40 0x7f36300caa20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f36300d1380 tx=0x7f3624009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645af21f8c34dea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:19.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f362effd700 1 -- 172.21.15.158:0/2914848420 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624004230 con 0x7f363015da40 2023-10-16T10:04:19.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f362effd700 1 -- 172.21.15.158:0/2914848420 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624004390 con 0x7f363015da40 2023-10-16T10:04:19.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f362effd700 1 -- 172.21.15.158:0/2914848420 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36240036a0 con 0x7f363015da40 2023-10-16T10:04:19.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 -- 172.21.15.158:0/2914848420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f363015da40 msgr2=0x7f36300caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 --2- 172.21.15.158:0/2914848420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f363015da40 0x7f36300caa20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f36300d1380 tx=0x7f3624009bd0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 -- 172.21.15.158:0/2914848420 shutdown_connections 2023-10-16T10:04:19.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 --2- 172.21.15.158:0/2914848420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f363015da40 0x7f36300caa20 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 --2- 172.21.15.158:0/2914848420 >> v2:172.21.15.172:3300/0 conn(0x7f36300cc590 0x7f363006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 -- 172.21.15.158:0/2914848420 >> 172.21.15.158:0/2914848420 conn(0x7f3630099a10 msgr2=0x7f36300d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:19.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 -- 172.21.15.158:0/2914848420 shutdown_connections 2023-10-16T10:04:19.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.179+0000 7f36363a2700 1 -- 172.21.15.158:0/2914848420 wait complete. 2023-10-16T10:04:19.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 Processor -- start 2023-10-16T10:04:19.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 -- start start 2023-10-16T10:04:19.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f363006ae10 0x7f363015b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36300cc590 0x7f36300c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f363015da40 0x7f36301e5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36300c6850 con 0x7f36300cc590 2023-10-16T10:04:19.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363015bef0 con 0x7f363015da40 2023-10-16T10:04:19.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.181+0000 7f36363a2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36300c7c10 con 0x7f363006ae10 2023-10-16T10:04:19.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36300cc590 0x7f36300c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36300cc590 0x7f36300c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60286/0 (socket says 172.21.15.158:60286) 2023-10-16T10:04:19.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 -- 172.21.15.158:0/2694117620 learned_addr learned my addr 172.21.15.158:0/2694117620 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:19.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362ffff700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f363006ae10 0x7f363015b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f363493f700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f363015da40 0x7f36301e5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 -- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f363006ae10 msgr2=0x7f363015b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f363006ae10 0x7f363015b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 -- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f363015da40 msgr2=0x7f36301e5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f363015da40 0x7f36301e5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 -- 172.21.15.158:0/2694117620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624009550 con 0x7f36300cc590 2023-10-16T10:04:19.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362f7fe700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36300cc590 0x7f36300c76d0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f362000c740 tx=0x7f362000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:19.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362cff9700 1 -- 172.21.15.158:0/2694117620 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620004220 con 0x7f36300cc590 2023-10-16T10:04:19.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362cff9700 1 -- 172.21.15.158:0/2694117620 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3620004380 con 0x7f36300cc590 2023-10-16T10:04:19.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36301e6f40 con 0x7f36300cc590 2023-10-16T10:04:19.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f362cff9700 1 -- 172.21.15.158:0/2694117620 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36200036c0 con 0x7f36300cc590 2023-10-16T10:04:19.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.182+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36301e6460 con 0x7f36300cc590 2023-10-16T10:04:19.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.183+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36300681b0 con 0x7f36300cc590 2023-10-16T10:04:19.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.184+0000 7f362cff9700 1 -- 172.21.15.158:0/2694117620 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 89) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3620003820 con 0x7f36300cc590 2023-10-16T10:04:19.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.184+0000 7f362cff9700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f361c06ada0 0x7f361c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.184+0000 7f362cff9700 1 -- 172.21.15.158:0/2694117620 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f3620089810 con 0x7f36300cc590 2023-10-16T10:04:19.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.184+0000 7f362ffff700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f361c06ada0 0x7f361c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.184+0000 7f362ffff700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f361c06ada0 0x7f361c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3624009130 tx=0x7f3624008030 comp rx=0 tx=0).ready entity=mgr.9573 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:19.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.188+0000 7f362cff9700 1 -- 172.21.15.158:0/2694117620 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f36200547a0 con 0x7f36300cc590 2023-10-16T10:04:19.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.409+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 --> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f36300cb120 con 0x7f361c06ada0 2023-10-16T10:04:19.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.469+0000 7f362cff9700 1 -- 172.21.15.158:0/2694117620 <== mgr.9573 v2:172.21.15.158:6832/2326422563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f3624006c60 con 0x7f361c06ada0 2023-10-16T10:04:19.471 INFO:teuthology.orchestra.run.smithi158.stdout:Retention added to path /snap_test_dir1 2023-10-16T10:04:19.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f361c06ada0 msgr2=0x7f361c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f361c06ada0 0x7f361c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3624009130 tx=0x7f3624008030 comp rx=0 tx=0).stop 2023-10-16T10:04:19.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36300cc590 msgr2=0x7f36300c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36300cc590 0x7f36300c76d0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f362000c740 tx=0x7f362000ca50 comp rx=0 tx=0).stop 2023-10-16T10:04:19.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 shutdown_connections 2023-10-16T10:04:19.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f361c06ada0 0x7f361c06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f363015da40 0x7f36301e5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f363006ae10 0x7f363015b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 --2- 172.21.15.158:0/2694117620 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36300cc590 0x7f36300c76d0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.471+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 >> 172.21.15.158:0/2694117620 conn(0x7f3630099a10 msgr2=0x7f363015cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:19.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.472+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 shutdown_connections 2023-10-16T10:04:19.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.472+0000 7f36363a2700 1 -- 172.21.15.158:0/2694117620 wait complete. 2023-10-16T10:04:19.485 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-10-16T10:04:19.485 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T10:04:19.486 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2023-10-16T10:04:19.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.608+0000 7fd1d9553700 1 Processor -- start 2023-10-16T10:04:19.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.608+0000 7fd1d9553700 1 -- start start 2023-10-16T10:04:19.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.608+0000 7fd1d9553700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd1d4159af0 0x7fd1d40ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.608+0000 7fd1d9553700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40ce3e0 0x7fd1d415de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.608+0000 7fd1d9553700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d40cd3d0 con 0x7fd1d40b0c00 2023-10-16T10:04:19.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.608+0000 7fd1d9553700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d40cdf50 con 0x7fd1d4159af0 2023-10-16T10:04:19.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.608+0000 7fd1d9553700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40cd620 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40ce3e0 0x7fd1d415de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd1d4159af0 0x7fd1d40ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d37fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd1d40b0c00 0x7fd1d4070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:36142/0 (socket says 172.21.15.158:36142) 2023-10-16T10:04:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d37fe700 1 -- 172.21.15.158:0/2075159576 learned_addr learned my addr 172.21.15.158:0/2075159576 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T10:04:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40ce3e0 0x7fd1d415de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:42894/0 (socket says 172.21.15.158:42894) 2023-10-16T10:04:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d2ffd700 1 -- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40ce3e0 msgr2=0x7fd1d415de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T10:04:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d2ffd700 1 --2- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40ce3e0 0x7fd1d415de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d2ffd700 1 -- 172.21.15.158:0/2075159576 >> v1:172.21.15.172:6789/0 conn(0x7fd1d40b0c00 legacy=0x7fd1d4070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d2ffd700 1 -- 172.21.15.158:0/2075159576 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40cd900 con 0x7fd1d4159af0 2023-10-16T10:04:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d27fc700 1 --2- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40ce3e0 0x7fd1d415de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T10:04:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.609+0000 7fd1d2ffd700 1 --2- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 0x7fd1d40ca8b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009840 tx=0x7fd1c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1887f814ab42183 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d1ffb700 1 -- 172.21.15.158:0/2075159576 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4011830 con 0x7fd1d4159af0 2023-10-16T10:04:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d1ffb700 1 -- 172.21.15.158:0/2075159576 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c4011990 con 0x7fd1d4159af0 2023-10-16T10:04:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d1ffb700 1 -- 172.21.15.158:0/2075159576 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4011d20 con 0x7fd1d4159af0 2023-10-16T10:04:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 -- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 msgr2=0x7fd1d40ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 --2- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 0x7fd1d40ca8b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009840 tx=0x7fd1c4009b50 comp rx=0 tx=0).stop 2023-10-16T10:04:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 -- 172.21.15.158:0/2075159576 shutdown_connections 2023-10-16T10:04:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 --2- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 0x7fd1d40ca8b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 --2- 172.21.15.158:0/2075159576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40ce3e0 0x7fd1d415de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 -- 172.21.15.158:0/2075159576 >> 172.21.15.158:0/2075159576 conn(0x7fd1d406af00 msgr2=0x7fd1d40cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 -- 172.21.15.158:0/2075159576 shutdown_connections 2023-10-16T10:04:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.610+0000 7fd1d9553700 1 -- 172.21.15.158:0/2075159576 wait complete. 2023-10-16T10:04:19.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 Processor -- start 2023-10-16T10:04:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 -- start start 2023-10-16T10:04:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40b0c00 0x7fd1d41f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1d40ce3e0 0x7fd1d41f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 0x7fd1d41f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40b03b0 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d415aa80 con 0x7fd1d4159af0 2023-10-16T10:04:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d9553700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d415ad60 con 0x7fd1d40b0c00 2023-10-16T10:04:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1d40ce3e0 0x7fd1d41f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1d40ce3e0 0x7fd1d41f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60298/0 (socket says 172.21.15.158:60298) 2023-10-16T10:04:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.612+0000 7fd1d27fc700 1 -- 172.21.15.158:0/968760004 learned_addr learned my addr 172.21.15.158:0/968760004 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T10:04:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d2ffd700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40b0c00 0x7fd1d41f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d37fe700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 0x7fd1d41f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d27fc700 1 -- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40b0c00 msgr2=0x7fd1d41f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d27fc700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40b0c00 0x7fd1d41f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d27fc700 1 -- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 msgr2=0x7fd1d41f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d27fc700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 0x7fd1d41f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d27fc700 1 -- 172.21.15.158:0/968760004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc00a640 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d27fc700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1d40ce3e0 0x7fd1d41f69f0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00e9a0 tx=0x7fd1bc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1cb7fe700 1 -- 172.21.15.158:0/968760004 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00cb50 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1cb7fe700 1 -- 172.21.15.158:0/968760004 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc00ccb0 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1cb7fe700 1 -- 172.21.15.158:0/968760004 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc011610 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c40094a0 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.613+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d415b320 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.614+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d40c2cb0 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.615+0000 7fd1cb7fe700 1 -- 172.21.15.158:0/968760004 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 89) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd1bc011770 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.615+0000 7fd1cb7fe700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fd1b406ada0 0x7fd1b406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T10:04:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.615+0000 7fd1cb7fe700 1 -- 172.21.15.158:0/968760004 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fd1bc089480 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.615+0000 7fd1d2ffd700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fd1b406ada0 0x7fd1b406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T10:04:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.616+0000 7fd1d2ffd700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fd1b406ada0 0x7fd1b406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009fd0 tx=0x7fd1c401c040 comp rx=0 tx=0).ready entity=mgr.9573 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T10:04:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.620+0000 7fd1cb7fe700 1 -- 172.21.15.158:0/968760004 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1bc054410 con 0x7fd1d40ce3e0 2023-10-16T10:04:19.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.842+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 --> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1d40cdbe0 con 0x7fd1b406ada0 2023-10-16T10:04:19.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.853+0000 7fd1cb7fe700 1 -- 172.21.15.158:0/968760004 <== mgr.9573 v2:172.21.15.158:6832/2326422563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7fd1c4011b80 con 0x7fd1b406ada0 2023-10-16T10:04:19.855 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T10:04:19.855 INFO:teuthology.orchestra.run.smithi158.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T10:04:19.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fd1b406ada0 msgr2=0x7fd1b406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fd1b406ada0 0x7fd1b406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009fd0 tx=0x7fd1c401c040 comp rx=0 tx=0).stop 2023-10-16T10:04:19.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1d40ce3e0 msgr2=0x7fd1d41f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T10:04:19.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1d40ce3e0 0x7fd1d41f69f0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00e9a0 tx=0x7fd1bc00ed60 comp rx=0 tx=0).stop 2023-10-16T10:04:19.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 shutdown_connections 2023-10-16T10:04:19.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fd1b406ada0 0x7fd1b406d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1d4159af0 0x7fd1d41f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1d40b0c00 0x7fd1d41f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 --2- 172.21.15.158:0/968760004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1d40ce3e0 0x7fd1d41f69f0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T10:04:19.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 >> 172.21.15.158:0/968760004 conn(0x7fd1d406af00 msgr2=0x7fd1d415d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T10:04:19.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 shutdown_connections 2023-10-16T10:04:19.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T10:04:19.856+0000 7fd1d9553700 1 -- 172.21.15.158:0/968760004 wait complete. 2023-10-16T10:04:19.870 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T10:04:19.870 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2023-10-16T10:04:29.237 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:04:29.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:04:59.303 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:04:59.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:05:00.481 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:05:00.480+0000 7efc4c831700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:05:00.482 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:05:00.480+0000 7efc4c831700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:05:00.482 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:05:00.480+0000 7efc4c831700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:05:00.482 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:05:00.480+0000 7efc4c831700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:05:00.482 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T10:05:00.480+0000 7efc4c831700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T10:05:29.366 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:05:29.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:05:59.472 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:05:59.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:06:29.547 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:06:29.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:06:59.649 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:06:59.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:07:29.724 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:07:29.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:07:59.803 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:07:59.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:08:29.866 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:08:29.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:09:00.000 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:09:00.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:09:30.070 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:09:30.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:10:00.444 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:10:00.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:10:30.510 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:10:30.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:11:00.574 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:11:00.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:11:30.652 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:11:30.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:12:00.732 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:12:00.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:12:30.804 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:12:30.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:13:00.892 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:13:00.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:13:30.993 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:13:30.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:14:01.076 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:14:01.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:14:31.144 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:14:31.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:15:01.424 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:15:01.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:15:31.498 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:15:31.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:16:01.567 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:16:01.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:16:31.631 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:16:31.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:17:01.720 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:17:01.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:17:31.793 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:17:31.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:18:01.880 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:18:01.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:18:31.992 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:18:31.996 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:19:02.055 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:19:02.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:19:32.119 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:19:32.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:20:02.440 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:20:02.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:20:32.520 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:20:32.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:21:02.608 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:21:02.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:21:32.696 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:21:32.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:22:02.760 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:22:02.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:22:32.829 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:22:32.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:23:02.934 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:23:02.938 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:23:33.006 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:23:33.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:24:03.078 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:24:03.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:24:33.165 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:24:33.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:25:03.228 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:25:03.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:25:33.300 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:25:33.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:26:03.530 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:26:03.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:26:33.634 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:26:33.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:27:03.723 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:27:03.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:27:33.805 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:27:33.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:28:03.887 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:28:03.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:28:34.003 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:28:34.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:29:04.074 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:29:04.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:29:34.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:29:34.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:30:04.206 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:30:04.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:30:34.280 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:30:34.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:31:04.368 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:31:04.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:31:34.523 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:31:34.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:32:04.592 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:32:04.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:32:34.679 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:32:34.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:33:04.769 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:33:04.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:33:34.848 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:33:34.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:34:04.919 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:34:04.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:34:34.993 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:34:34.996 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:35:05.271 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:35:05.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:35:35.378 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:35:35.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:36:05.466 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:36:05.470 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:36:35.537 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:36:35.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:37:05.609 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:37:05.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:37:35.697 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:37:35.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:38:05.793 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:38:05.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:38:35.865 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:38:35.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:39:05.936 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:39:05.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:39:36.007 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:39:36.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:40:06.119 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:40:06.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:40:36.191 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:40:36.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:41:06.271 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:41:06.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:41:36.333 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:41:36.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:42:06.472 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:42:06.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:42:36.584 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:42:36.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:43:06.822 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:43:06.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:43:36.885 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:43:36.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:44:07.036 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:44:07.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:44:37.208 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:44:37.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:45:07.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:45:07.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:45:37.616 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:45:37.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:46:07.729 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:46:07.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:46:37.858 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:46:37.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:47:08.110 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:47:08.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:47:38.257 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:47:38.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:48:08.372 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:48:08.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:48:38.518 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:48:38.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:49:08.589 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:49:08.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:49:38.694 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:49:38.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:50:08.805 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:50:08.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:50:38.877 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:50:38.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:51:09.016 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:51:09.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:51:39.170 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:51:39.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:52:09.300 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:52:09.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:52:39.363 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:52:39.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:53:09.509 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:53:09.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:53:39.665 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:53:39.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:54:09.727 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:54:09.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:54:39.873 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:54:39.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:55:09.936 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:55:09.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:55:40.065 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:55:40.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:56:10.195 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:56:10.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:56:40.318 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:56:40.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:57:10.448 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:57:10.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:57:40.576 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:57:40.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:58:10.663 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:58:10.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:58:40.978 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:58:40.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:59:11.041 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:59:11.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:59:41.194 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T10:59:41.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:00:11.341 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:00:11.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:00:15.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:15.871 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:00:15.898 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_06_00_UTC 2023-10-16T11:00:15.898 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_07_00_UTC 2023-10-16T11:00:15.898 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_08_00_UTC 2023-10-16T11:00:15.898 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_09_00_UTC 2023-10-16T11:00:15.899 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_10_00_UTC 2023-10-16T11:00:15.899 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_11_00_UTC 2023-10-16T11:00:15.899 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_12_00_UTC 2023-10-16T11:00:15.899 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_13_00_UTC 2023-10-16T11:00:15.899 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_14_00_UTC 2023-10-16T11:00:15.900 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_15_00_UTC 2023-10-16T11:00:15.900 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_16_00_UTC 2023-10-16T11:00:15.900 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_17_00_UTC 2023-10-16T11:00:15.900 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_18_00_UTC 2023-10-16T11:00:15.900 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_19_00_UTC 2023-10-16T11:00:15.900 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_20_00_UTC 2023-10-16T11:00:15.901 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_21_00_UTC 2023-10-16T11:00:15.901 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_22_00_UTC 2023-10-16T11:00:15.901 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_23_00_UTC 2023-10-16T11:00:15.901 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_24_00_UTC 2023-10-16T11:00:15.901 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_25_00_UTC 2023-10-16T11:00:15.901 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_26_00_UTC 2023-10-16T11:00:15.902 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_27_00_UTC 2023-10-16T11:00:15.902 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_28_00_UTC 2023-10-16T11:00:15.902 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_29_00_UTC 2023-10-16T11:00:15.902 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_30_00_UTC 2023-10-16T11:00:15.902 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_31_00_UTC 2023-10-16T11:00:15.902 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_32_00_UTC 2023-10-16T11:00:15.903 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_33_00_UTC 2023-10-16T11:00:15.903 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_34_00_UTC 2023-10-16T11:00:15.903 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_35_00_UTC 2023-10-16T11:00:15.903 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_36_00_UTC 2023-10-16T11:00:15.903 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_37_00_UTC 2023-10-16T11:00:15.904 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_38_00_UTC 2023-10-16T11:00:15.904 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_39_00_UTC 2023-10-16T11:00:15.904 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_40_00_UTC 2023-10-16T11:00:15.904 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_41_00_UTC 2023-10-16T11:00:15.904 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_42_00_UTC 2023-10-16T11:00:15.905 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_43_00_UTC 2023-10-16T11:00:15.905 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_44_00_UTC 2023-10-16T11:00:15.905 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_45_00_UTC 2023-10-16T11:00:15.905 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_46_00_UTC 2023-10-16T11:00:15.905 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_47_00_UTC 2023-10-16T11:00:15.906 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_48_00_UTC 2023-10-16T11:00:15.906 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_49_00_UTC 2023-10-16T11:00:15.906 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_50_00_UTC 2023-10-16T11:00:15.906 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_51_00_UTC 2023-10-16T11:00:15.906 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_52_00_UTC 2023-10-16T11:00:15.906 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_53_00_UTC 2023-10-16T11:00:15.907 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_54_00_UTC 2023-10-16T11:00:15.907 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_55_00_UTC 2023-10-16T11:00:15.907 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_56_00_UTC 2023-10-16T11:00:15.907 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_57_00_UTC 2023-10-16T11:00:15.907 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_58_00_UTC 2023-10-16T11:00:15.908 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_59_00_UTC 2023-10-16T11:00:15.908 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_00_00_UTC 2023-10-16T11:00:15.908 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-10_06_00_UTC', 'scheduled-2023-10-16-10_07_00_UTC', 'scheduled-2023-10-16-10_08_00_UTC', 'scheduled-2023-10-16-10_09_00_UTC', 'scheduled-2023-10-16-10_10_00_UTC', 'scheduled-2023-10-16-10_11_00_UTC', 'scheduled-2023-10-16-10_12_00_UTC', 'scheduled-2023-10-16-10_13_00_UTC', 'scheduled-2023-10-16-10_14_00_UTC', 'scheduled-2023-10-16-10_15_00_UTC', 'scheduled-2023-10-16-10_16_00_UTC', 'scheduled-2023-10-16-10_17_00_UTC', 'scheduled-2023-10-16-10_18_00_UTC', 'scheduled-2023-10-16-10_19_00_UTC', 'scheduled-2023-10-16-10_20_00_UTC', 'scheduled-2023-10-16-10_21_00_UTC', 'scheduled-2023-10-16-10_22_00_UTC', 'scheduled-2023-10-16-10_23_00_UTC', 'scheduled-2023-10-16-10_24_00_UTC', 'scheduled-2023-10-16-10_25_00_UTC', 'scheduled-2023-10-16-10_26_00_UTC', 'scheduled-2023-10-16-10_27_00_UTC', 'scheduled-2023-10-16-10_28_00_UTC', 'scheduled-2023-10-16-10_29_00_UTC', 'scheduled-2023-10-16-10_30_00_UTC', 'scheduled-2023-10-16-10_31_00_UTC', 'scheduled-2023-10-16-10_32_00_UTC', 'scheduled-2023-10-16-10_33_00_UTC', 'scheduled-2023-10-16-10_34_00_UTC', 'scheduled-2023-10-16-10_35_00_UTC', 'scheduled-2023-10-16-10_36_00_UTC', 'scheduled-2023-10-16-10_37_00_UTC', 'scheduled-2023-10-16-10_38_00_UTC', 'scheduled-2023-10-16-10_39_00_UTC', 'scheduled-2023-10-16-10_40_00_UTC', 'scheduled-2023-10-16-10_41_00_UTC', 'scheduled-2023-10-16-10_42_00_UTC', 'scheduled-2023-10-16-10_43_00_UTC', 'scheduled-2023-10-16-10_44_00_UTC', 'scheduled-2023-10-16-10_45_00_UTC', 'scheduled-2023-10-16-10_46_00_UTC', 'scheduled-2023-10-16-10_47_00_UTC', 'scheduled-2023-10-16-10_48_00_UTC', 'scheduled-2023-10-16-10_49_00_UTC', 'scheduled-2023-10-16-10_50_00_UTC', 'scheduled-2023-10-16-10_51_00_UTC', 'scheduled-2023-10-16-10_52_00_UTC', 'scheduled-2023-10-16-10_53_00_UTC', 'scheduled-2023-10-16-10_54_00_UTC', 'scheduled-2023-10-16-10_55_00_UTC', 'scheduled-2023-10-16-10_56_00_UTC', 'scheduled-2023-10-16-10_57_00_UTC', 'scheduled-2023-10-16-10_58_00_UTC', 'scheduled-2023-10-16-10_59_00_UTC', 'scheduled-2023-10-16-11_00_00_UTC'] 2023-10-16T11:00:15.908 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2023-10-16T11:00:16.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.068+0000 7fcbab370700 1 Processor -- start 2023-10-16T11:00:16.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.069+0000 7fcbab370700 1 -- start start 2023-10-16T11:00:16.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.069+0000 7fcbab370700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcba406f180 0x7fcba406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.069+0000 7fcbab370700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 0x7fcba406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.069+0000 7fcbab370700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba415ee60 con 0x7fcba415b540 2023-10-16T11:00:16.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.069+0000 7fcbab370700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba415b920 con 0x7fcba406f180 2023-10-16T11:00:16.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.069+0000 7fcbab370700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba415e2e0 con 0x7fcba406fea0 2023-10-16T11:00:16.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcba910c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcba415b540 0x7fcba406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60598/0 (socket says 172.21.15.158:60598) 2023-10-16T11:00:16.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcba910c700 1 -- 172.21.15.158:0/1185564727 learned_addr learned my addr 172.21.15.158:0/1185564727 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:16.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcba990d700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 0x7fcba406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcba890b700 1 --2- 172.21.15.158:0/1185564727 >> v2:172.21.15.172:3300/0 conn(0x7fcba406f180 0x7fcba406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967298236 0 0) 0x7fcba415ee60 con 0x7fcba415b540 2023-10-16T11:00:16.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb900035b0 con 0x7fcba415b540 2023-10-16T11:00:16.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1848982668 0 0) 0x7fcb900035b0 con 0x7fcba415b540 2023-10-16T11:00:16.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 msgr2=0x7fcba406ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:16.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 0x7fcba406ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 >> v2:172.21.15.172:3300/0 conn(0x7fcba406f180 msgr2=0x7fcba406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:16.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 --2- 172.21.15.158:0/1185564727 >> v2:172.21.15.172:3300/0 conn(0x7fcba406f180 0x7fcba406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba415e5c0 con 0x7fcba415b540 2023-10-16T11:00:16.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcb94001670 con 0x7fcba415b540 2023-10-16T11:00:16.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 >> v1:172.21.15.158:6789/0 conn(0x7fcba415b540 legacy=0x7fcba406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 0x7fcb90003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90003f80 0x7fcb90004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb90004d90 0x7fcb90005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba415b920 con 0x7fcb90004d90 2023-10-16T11:00:16.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba415e2e0 con 0x7fcb90003f80 2023-10-16T11:00:16.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9000ba60 con 0x7fcba406fea0 2023-10-16T11:00:16.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcba990d700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb90004d90 0x7fcb90005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcba890b700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90003f80 0x7fcb90004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.070+0000 7fcba910c700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 0x7fcb90003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcba990d700 1 -- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 msgr2=0x7fcb90003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:16.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcba990d700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba406fea0 0x7fcb90003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcba990d700 1 -- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90003f80 msgr2=0x7fcb90004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:16.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcba990d700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90003f80 0x7fcb90004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcba990d700 1 -- 172.21.15.158:0/1185564727 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb900035b0 con 0x7fcb90004d90 2023-10-16T11:00:16.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcba990d700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb90004d90 0x7fcb90005210 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00e170 tx=0x7fcb90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b54ee9adca8b4ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:16.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c00ec90 con 0x7fcb90004d90 2023-10-16T11:00:16.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c00f040 con 0x7fcb90004d90 2023-10-16T11:00:16.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcb9bfff700 1 -- 172.21.15.158:0/1185564727 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c014480 con 0x7fcb90004d90 2023-10-16T11:00:16.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcbab370700 1 -- 172.21.15.158:0/1185564727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb90004d90 msgr2=0x7fcb90005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcbab370700 1 --2- 172.21.15.158:0/1185564727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb90004d90 0x7fcb90005210 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00e170 tx=0x7fcb90005880 comp rx=0 tx=0).stop 2023-10-16T11:00:16.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcba990d700 1 -- 172.21.15.158:0/1185564727 reap_dead start 2023-10-16T11:00:16.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcbab370700 1 -- 172.21.15.158:0/1185564727 shutdown_connections 2023-10-16T11:00:16.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.071+0000 7fcbab370700 1 -- 172.21.15.158:0/1185564727 >> 172.21.15.158:0/1185564727 conn(0x7fcba40d03b0 msgr2=0x7fcba40d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:16.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.072+0000 7fcbab370700 1 -- 172.21.15.158:0/1185564727 shutdown_connections 2023-10-16T11:00:16.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.072+0000 7fcbab370700 1 -- 172.21.15.158:0/1185564727 wait complete. 2023-10-16T11:00:16.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.073+0000 7fcbab370700 1 Processor -- start 2023-10-16T11:00:16.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.073+0000 7fcbab370700 1 -- start start 2023-10-16T11:00:16.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcbab370700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90004d90 0x7fcba40c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcbab370700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba415fa20 0x7fcba40ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcbab370700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcba40ae650 0x7fcba40aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcbab370700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40c95f0 con 0x7fcba40ae650 2023-10-16T11:00:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcbab370700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba4160b30 con 0x7fcb90004d90 2023-10-16T11:00:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcbab370700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40af040 con 0x7fcba415fa20 2023-10-16T11:00:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcba40ae650 0x7fcba40aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcba40ae650 0x7fcba40aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60436/0 (socket says 172.21.15.158:60436) 2023-10-16T11:00:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 -- 172.21.15.158:0/3709678595 learned_addr learned my addr 172.21.15.158:0/3709678595 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba890b700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba415fa20 0x7fcba40ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba910c700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90004d90 0x7fcba40c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 -- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba415fa20 msgr2=0x7fcba40ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba415fa20 0x7fcba40ae110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 -- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90004d90 msgr2=0x7fcba40c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90004d90 0x7fcba40c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 -- 172.21.15.158:0/3709678595 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba415feb0 con 0x7fcba40ae650 2023-10-16T11:00:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcba990d700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcba40ae650 0x7fcba40aeb00 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fcb90004790 tx=0x7fcb9c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcb99ffb700 1 -- 172.21.15.158:0/3709678595 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c007000 con 0x7fcba40ae650 2023-10-16T11:00:16.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcb99ffb700 1 -- 172.21.15.158:0/3709678595 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c00ee80 con 0x7fcba40ae650 2023-10-16T11:00:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba4160190 con 0x7fcba40ae650 2023-10-16T11:00:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.074+0000 7fcb99ffb700 1 -- 172.21.15.158:0/3709678595 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c003c60 con 0x7fcba40ae650 2023-10-16T11:00:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.075+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba4160750 con 0x7fcba40ae650 2023-10-16T11:00:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.075+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba40681b0 con 0x7fcba40ae650 2023-10-16T11:00:16.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.076+0000 7fcb99ffb700 1 -- 172.21.15.158:0/3709678595 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 93) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fcb9c024030 con 0x7fcba40ae650 2023-10-16T11:00:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.076+0000 7fcb99ffb700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fcb8806b070 0x7fcb8806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.076+0000 7fcb99ffb700 1 -- 172.21.15.158:0/3709678595 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fcb9c08b570 con 0x7fcba40ae650 2023-10-16T11:00:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.076+0000 7fcba910c700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fcb8806b070 0x7fcb8806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.077+0000 7fcba910c700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fcb8806b070 0x7fcb8806d530 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb940010e0 tx=0x7fcb94009040 comp rx=0 tx=0).ready entity=mgr.9573 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:16.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.081+0000 7fcb99ffb700 1 -- 172.21.15.158:0/3709678595 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb9c056040 con 0x7fcba40ae650 2023-10-16T11:00:16.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.298+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 --> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcba415eb80 con 0x7fcb8806b070 2023-10-16T11:00:16.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.306+0000 7fcb99ffb700 1 -- 172.21.15.158:0/3709678595 <== mgr.9573 v2:172.21.15.158:6832/2326422563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7fcb94007e20 con 0x7fcb8806b070 2023-10-16T11:00:16.308 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:16.308 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T10:04:18", "first": "2023-10-16T10:05:00", "last": "2023-10-16T11:00:00", "last_pruned": "2023-10-16T11:00:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T11:00:16.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fcb8806b070 msgr2=0x7fcb8806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fcb8806b070 0x7fcb8806d530 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb940010e0 tx=0x7fcb94009040 comp rx=0 tx=0).stop 2023-10-16T11:00:16.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcba40ae650 msgr2=0x7fcba40aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcba40ae650 0x7fcba40aeb00 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fcb90004790 tx=0x7fcb9c004b40 comp rx=0 tx=0).stop 2023-10-16T11:00:16.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 shutdown_connections 2023-10-16T11:00:16.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fcb8806b070 0x7fcb8806d530 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcba40ae650 0x7fcba40aeb00 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb90004d90 0x7fcba40c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 --2- 172.21.15.158:0/3709678595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcba415fa20 0x7fcba40ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.308+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 >> 172.21.15.158:0/3709678595 conn(0x7fcba40d03b0 msgr2=0x7fcba406dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:16.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.309+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 shutdown_connections 2023-10-16T11:00:16.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.309+0000 7fcbab370700 1 -- 172.21.15.158:0/3709678595 wait complete. 2023-10-16T11:00:16.321 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T10:04:18", "first": "2023-10-16T10:05:00", "last": "2023-10-16T11:00:00", "last_pruned": "2023-10-16T11:00:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T11:00:16.322 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T10:04:18', 'first': '2023-10-16T10:05:00', 'last': '2023-10-16T11:00:00', 'last_pruned': '2023-10-16T11:00:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-10-16T11:00:16.322 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-10-16T11:00:16.322 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-10-16T11:00:16.323 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2023-10-16T11:00:16.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.446+0000 7fb1829c5700 1 Processor -- start 2023-10-16T11:00:16.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.446+0000 7fb1829c5700 1 -- start start 2023-10-16T11:00:16.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb1829c5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb17c070dd0 0x7fb17c0cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb1829c5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0cfc90 con 0x7fb17c15b4a0 2023-10-16T11:00:16.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb1829c5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c15f5c0 con 0x7fb17c15f050 2023-10-16T11:00:16.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb1829c5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb17c15f8a0 con 0x7fb17c070dd0 2023-10-16T11:00:16.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb180f62700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb17c070dd0 0x7fb17c0cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb17bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb17c15b4a0 0x7fb17c0b1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60616/0 (socket says 172.21.15.158:60616) 2023-10-16T11:00:16.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb17bfff700 1 -- 172.21.15.158:0/2219201372 learned_addr learned my addr 172.21.15.158:0/2219201372 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:16.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb180f62700 1 -- 172.21.15.158:0/2219201372 >> v1:172.21.15.172:6789/0 conn(0x7fb17c15f050 legacy=0x7fb17c0706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:16.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb180f62700 1 -- 172.21.15.158:0/2219201372 >> v1:172.21.15.158:6789/0 conn(0x7fb17c15b4a0 legacy=0x7fb17c0b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb180f62700 1 -- 172.21.15.158:0/2219201372 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c15fb80 con 0x7fb17c070dd0 2023-10-16T11:00:16.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb180f62700 1 --2- 172.21.15.158:0/2219201372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c0cb2d0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb174009d90 tx=0x7fb17400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd933bdbe434555d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:16.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb17affd700 1 -- 172.21.15.158:0/2219201372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17400e070 con 0x7fb17c070dd0 2023-10-16T11:00:16.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb17affd700 1 -- 172.21.15.158:0/2219201372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17400bb90 con 0x7fb17c070dd0 2023-10-16T11:00:16.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.447+0000 7fb17affd700 1 -- 172.21.15.158:0/2219201372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb174003a20 con 0x7fb17c070dd0 2023-10-16T11:00:16.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.448+0000 7fb1829c5700 1 -- 172.21.15.158:0/2219201372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 msgr2=0x7fb17c0cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.448+0000 7fb1829c5700 1 --2- 172.21.15.158:0/2219201372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c0cb2d0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb174009d90 tx=0x7fb17400b330 comp rx=0 tx=0).stop 2023-10-16T11:00:16.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.448+0000 7fb1829c5700 1 -- 172.21.15.158:0/2219201372 shutdown_connections 2023-10-16T11:00:16.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.448+0000 7fb1829c5700 1 --2- 172.21.15.158:0/2219201372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c0cb2d0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.448+0000 7fb1829c5700 1 -- 172.21.15.158:0/2219201372 >> 172.21.15.158:0/2219201372 conn(0x7fb17c06af00 msgr2=0x7fb17c0cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:16.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.448+0000 7fb1829c5700 1 -- 172.21.15.158:0/2219201372 shutdown_connections 2023-10-16T11:00:16.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.448+0000 7fb1829c5700 1 -- 172.21.15.158:0/2219201372 wait complete. 2023-10-16T11:00:16.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.449+0000 7fb1829c5700 1 Processor -- start 2023-10-16T11:00:16.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.449+0000 7fb1829c5700 1 -- start start 2023-10-16T11:00:16.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb1829c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c15d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb1829c5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb17c15b4a0 0x7fb17c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb1829c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb17c15f050 0x7fb17c0b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb1829c5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c159170 con 0x7fb17c070dd0 2023-10-16T11:00:16.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb1829c5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c15e260 con 0x7fb17c15f050 2023-10-16T11:00:16.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb1829c5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c15e540 con 0x7fb17c15b4a0 2023-10-16T11:00:16.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb17bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c15d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb17bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c15d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60446/0 (socket says 172.21.15.158:60446) 2023-10-16T11:00:16.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.450+0000 7fb17bfff700 1 -- 172.21.15.158:0/321406508 learned_addr learned my addr 172.21.15.158:0/321406508 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:16.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17b7fe700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb17c15b4a0 0x7fb17c0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb180f62700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb17c15f050 0x7fb17c0b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17bfff700 1 -- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb17c15b4a0 msgr2=0x7fb17c0b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17bfff700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb17c15b4a0 0x7fb17c0b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17bfff700 1 -- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb17c15f050 msgr2=0x7fb17c0b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17bfff700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb17c15f050 0x7fb17c0b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17bfff700 1 -- 172.21.15.158:0/321406508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb174009a70 con 0x7fb17c070dd0 2023-10-16T11:00:16.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17b7fe700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb17c15b4a0 0x7fb17c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:16.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb180f62700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb17c15f050 0x7fb17c0b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:16.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb17bfff700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c15d960 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb17000c490 tx=0x7fb17000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:16.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb178ff9700 1 -- 172.21.15.158:0/321406508 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170005d70 con 0x7fb17c070dd0 2023-10-16T11:00:16.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb178ff9700 1 -- 172.21.15.158:0/321406508 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1700070d0 con 0x7fb17c070dd0 2023-10-16T11:00:16.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17c15e820 con 0x7fb17c070dd0 2023-10-16T11:00:16.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb178ff9700 1 -- 172.21.15.158:0/321406508 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170007460 con 0x7fb17c070dd0 2023-10-16T11:00:16.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.451+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c15dea0 con 0x7fb17c070dd0 2023-10-16T11:00:16.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.452+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17c15be50 con 0x7fb17c070dd0 2023-10-16T11:00:16.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.452+0000 7fb178ff9700 1 -- 172.21.15.158:0/321406508 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 93) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fb1700075c0 con 0x7fb17c070dd0 2023-10-16T11:00:16.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.453+0000 7fb178ff9700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb16806b070 0x7fb16806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:16.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.453+0000 7fb178ff9700 1 -- 172.21.15.158:0/321406508 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fb1700890e0 con 0x7fb17c070dd0 2023-10-16T11:00:16.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.453+0000 7fb17b7fe700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb16806b070 0x7fb16806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:16.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.453+0000 7fb17b7fe700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb16806b070 0x7fb16806d530 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb16c003fc0 tx=0x7fb16c009040 comp rx=0 tx=0).ready entity=mgr.9573 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:16.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.457+0000 7fb178ff9700 1 -- 172.21.15.158:0/321406508 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb17c15be50 con 0x7fb17c070dd0 2023-10-16T11:00:16.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.673+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 --> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb17c15ba90 con 0x7fb16806b070 2023-10-16T11:00:16.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.754+0000 7fb178ff9700 1 -- 172.21.15.158:0/321406508 <== mgr.9573 v2:172.21.15.158:6832/2326422563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7fb16c003bb0 con 0x7fb16806b070 2023-10-16T11:00:16.756 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1 2023-10-16T11:00:16.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb16806b070 msgr2=0x7fb16806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb16806b070 0x7fb16806d530 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb16c003fc0 tx=0x7fb16c009040 comp rx=0 tx=0).stop 2023-10-16T11:00:16.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 msgr2=0x7fb17c15d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:16.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c15d960 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb17000c490 tx=0x7fb17000c850 comp rx=0 tx=0).stop 2023-10-16T11:00:16.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 shutdown_connections 2023-10-16T11:00:16.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb16806b070 0x7fb16806d530 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb17c15f050 0x7fb17c0b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb17c070dd0 0x7fb17c15d960 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 --2- 172.21.15.158:0/321406508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb17c15b4a0 0x7fb17c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:16.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 >> 172.21.15.158:0/321406508 conn(0x7fb17c06af00 msgr2=0x7fb17c0c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:16.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 shutdown_connections 2023-10-16T11:00:16.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:16.757+0000 7fb1829c5700 1 -- 172.21.15.158:0/321406508 wait complete. 2023-10-16T11:00:16.770 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-10-16T11:00:16.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:16.771 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:00:16.790 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_06_00_UTC 2023-10-16T11:00:16.791 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_07_00_UTC 2023-10-16T11:00:16.791 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_08_00_UTC 2023-10-16T11:00:16.791 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_09_00_UTC 2023-10-16T11:00:16.791 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_10_00_UTC 2023-10-16T11:00:16.792 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_11_00_UTC 2023-10-16T11:00:16.792 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_12_00_UTC 2023-10-16T11:00:16.792 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_13_00_UTC 2023-10-16T11:00:16.792 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_14_00_UTC 2023-10-16T11:00:16.792 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_15_00_UTC 2023-10-16T11:00:16.792 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_16_00_UTC 2023-10-16T11:00:16.793 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_17_00_UTC 2023-10-16T11:00:16.793 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_18_00_UTC 2023-10-16T11:00:16.793 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_19_00_UTC 2023-10-16T11:00:16.793 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_20_00_UTC 2023-10-16T11:00:16.793 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_21_00_UTC 2023-10-16T11:00:16.794 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_22_00_UTC 2023-10-16T11:00:16.794 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_23_00_UTC 2023-10-16T11:00:16.794 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_24_00_UTC 2023-10-16T11:00:16.794 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_25_00_UTC 2023-10-16T11:00:16.794 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_26_00_UTC 2023-10-16T11:00:16.794 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_27_00_UTC 2023-10-16T11:00:16.795 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_28_00_UTC 2023-10-16T11:00:16.795 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_29_00_UTC 2023-10-16T11:00:16.795 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_30_00_UTC 2023-10-16T11:00:16.795 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_31_00_UTC 2023-10-16T11:00:16.796 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_32_00_UTC 2023-10-16T11:00:16.796 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_33_00_UTC 2023-10-16T11:00:16.796 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_34_00_UTC 2023-10-16T11:00:16.796 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_35_00_UTC 2023-10-16T11:00:16.796 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_36_00_UTC 2023-10-16T11:00:16.796 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_37_00_UTC 2023-10-16T11:00:16.797 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_38_00_UTC 2023-10-16T11:00:16.797 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_39_00_UTC 2023-10-16T11:00:16.797 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_40_00_UTC 2023-10-16T11:00:16.797 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_41_00_UTC 2023-10-16T11:00:16.797 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_42_00_UTC 2023-10-16T11:00:16.797 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_43_00_UTC 2023-10-16T11:00:16.798 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_44_00_UTC 2023-10-16T11:00:16.798 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_45_00_UTC 2023-10-16T11:00:16.798 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_46_00_UTC 2023-10-16T11:00:16.798 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_47_00_UTC 2023-10-16T11:00:16.798 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_48_00_UTC 2023-10-16T11:00:16.799 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_49_00_UTC 2023-10-16T11:00:16.799 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_50_00_UTC 2023-10-16T11:00:16.799 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_51_00_UTC 2023-10-16T11:00:16.799 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_52_00_UTC 2023-10-16T11:00:16.799 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_53_00_UTC 2023-10-16T11:00:16.799 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_54_00_UTC 2023-10-16T11:00:16.800 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_55_00_UTC 2023-10-16T11:00:16.800 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_56_00_UTC 2023-10-16T11:00:16.800 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_57_00_UTC 2023-10-16T11:00:16.800 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_58_00_UTC 2023-10-16T11:00:16.800 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-10_59_00_UTC 2023-10-16T11:00:16.801 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_00_00_UTC 2023-10-16T11:00:16.801 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_06_00_UTC 2023-10-16T11:00:16.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:16.802 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_06_00_UTC) 2023-10-16T11:00:16.853 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_07_00_UTC 2023-10-16T11:00:16.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:16.854 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_07_00_UTC) 2023-10-16T11:00:16.915 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_08_00_UTC 2023-10-16T11:00:16.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:16.916 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_08_00_UTC) 2023-10-16T11:00:16.978 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_09_00_UTC 2023-10-16T11:00:16.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:16.979 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_09_00_UTC) 2023-10-16T11:00:17.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_10_00_UTC 2023-10-16T11:00:17.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.041 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_10_00_UTC) 2023-10-16T11:00:17.103 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_11_00_UTC 2023-10-16T11:00:17.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.103 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_11_00_UTC) 2023-10-16T11:00:17.165 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_12_00_UTC 2023-10-16T11:00:17.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.166 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_12_00_UTC) 2023-10-16T11:00:17.227 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_13_00_UTC 2023-10-16T11:00:17.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.228 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_13_00_UTC) 2023-10-16T11:00:17.290 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_14_00_UTC 2023-10-16T11:00:17.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.290 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_14_00_UTC) 2023-10-16T11:00:17.352 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_15_00_UTC 2023-10-16T11:00:17.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.353 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_15_00_UTC) 2023-10-16T11:00:17.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_16_00_UTC 2023-10-16T11:00:17.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.415 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_16_00_UTC) 2023-10-16T11:00:17.476 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_17_00_UTC 2023-10-16T11:00:17.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.477 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_17_00_UTC) 2023-10-16T11:00:17.539 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_18_00_UTC 2023-10-16T11:00:17.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.540 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_18_00_UTC) 2023-10-16T11:00:17.602 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_19_00_UTC 2023-10-16T11:00:17.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.603 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_19_00_UTC) 2023-10-16T11:00:17.664 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_20_00_UTC 2023-10-16T11:00:17.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.665 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_20_00_UTC) 2023-10-16T11:00:17.726 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_21_00_UTC 2023-10-16T11:00:17.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.727 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_21_00_UTC) 2023-10-16T11:00:17.789 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_22_00_UTC 2023-10-16T11:00:17.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.790 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_22_00_UTC) 2023-10-16T11:00:17.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_23_00_UTC 2023-10-16T11:00:17.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.853 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_23_00_UTC) 2023-10-16T11:00:17.914 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_24_00_UTC 2023-10-16T11:00:17.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.915 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_24_00_UTC) 2023-10-16T11:00:17.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_25_00_UTC 2023-10-16T11:00:17.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:17.976 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_25_00_UTC) 2023-10-16T11:00:18.038 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_26_00_UTC 2023-10-16T11:00:18.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.039 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_26_00_UTC) 2023-10-16T11:00:18.101 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_27_00_UTC 2023-10-16T11:00:18.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.102 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_27_00_UTC) 2023-10-16T11:00:18.164 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_28_00_UTC 2023-10-16T11:00:18.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.165 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_28_00_UTC) 2023-10-16T11:00:18.226 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_29_00_UTC 2023-10-16T11:00:18.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.227 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_29_00_UTC) 2023-10-16T11:00:18.289 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_30_00_UTC 2023-10-16T11:00:18.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.290 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_30_00_UTC) 2023-10-16T11:00:18.352 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_31_00_UTC 2023-10-16T11:00:18.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.353 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_31_00_UTC) 2023-10-16T11:00:18.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_32_00_UTC 2023-10-16T11:00:18.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.415 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_32_00_UTC) 2023-10-16T11:00:18.477 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_33_00_UTC 2023-10-16T11:00:18.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.478 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_33_00_UTC) 2023-10-16T11:00:18.540 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_34_00_UTC 2023-10-16T11:00:18.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.541 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_34_00_UTC) 2023-10-16T11:00:18.602 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_35_00_UTC 2023-10-16T11:00:18.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.603 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_35_00_UTC) 2023-10-16T11:00:18.664 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_36_00_UTC 2023-10-16T11:00:18.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.665 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_36_00_UTC) 2023-10-16T11:00:18.726 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_37_00_UTC 2023-10-16T11:00:18.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.727 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_37_00_UTC) 2023-10-16T11:00:18.789 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_38_00_UTC 2023-10-16T11:00:18.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.790 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_38_00_UTC) 2023-10-16T11:00:18.851 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_39_00_UTC 2023-10-16T11:00:18.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.852 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_39_00_UTC) 2023-10-16T11:00:18.913 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_40_00_UTC 2023-10-16T11:00:18.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.914 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_40_00_UTC) 2023-10-16T11:00:18.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_41_00_UTC 2023-10-16T11:00:18.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:18.978 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_41_00_UTC) 2023-10-16T11:00:19.039 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_42_00_UTC 2023-10-16T11:00:19.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.040 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_42_00_UTC) 2023-10-16T11:00:19.101 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_43_00_UTC 2023-10-16T11:00:19.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.102 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_43_00_UTC) 2023-10-16T11:00:19.163 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_44_00_UTC 2023-10-16T11:00:19.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.164 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_44_00_UTC) 2023-10-16T11:00:19.226 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_45_00_UTC 2023-10-16T11:00:19.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.227 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_45_00_UTC) 2023-10-16T11:00:19.289 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_46_00_UTC 2023-10-16T11:00:19.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.290 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_46_00_UTC) 2023-10-16T11:00:19.351 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_47_00_UTC 2023-10-16T11:00:19.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.352 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_47_00_UTC) 2023-10-16T11:00:19.415 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_48_00_UTC 2023-10-16T11:00:19.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.416 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_48_00_UTC) 2023-10-16T11:00:19.478 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_49_00_UTC 2023-10-16T11:00:19.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.479 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_49_00_UTC) 2023-10-16T11:00:19.541 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_50_00_UTC 2023-10-16T11:00:19.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.541 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_50_00_UTC) 2023-10-16T11:00:19.602 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_51_00_UTC 2023-10-16T11:00:19.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.603 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_51_00_UTC) 2023-10-16T11:00:19.666 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_52_00_UTC 2023-10-16T11:00:19.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.667 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_52_00_UTC) 2023-10-16T11:00:19.728 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_53_00_UTC 2023-10-16T11:00:19.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.730 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_53_00_UTC) 2023-10-16T11:00:19.791 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_54_00_UTC 2023-10-16T11:00:19.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.792 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_54_00_UTC) 2023-10-16T11:00:19.853 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_55_00_UTC 2023-10-16T11:00:19.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.854 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_55_00_UTC) 2023-10-16T11:00:19.916 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_56_00_UTC 2023-10-16T11:00:19.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.917 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_56_00_UTC) 2023-10-16T11:00:19.978 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_57_00_UTC 2023-10-16T11:00:19.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:19.979 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_57_00_UTC) 2023-10-16T11:00:20.041 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_58_00_UTC 2023-10-16T11:00:20.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:20.041 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_58_00_UTC) 2023-10-16T11:00:20.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-10_59_00_UTC 2023-10-16T11:00:20.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:20.103 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-10_59_00_UTC) 2023-10-16T11:00:20.164 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-11_00_00_UTC 2023-10-16T11:00:20.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:20.165 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-11_00_00_UTC) 2023-10-16T11:00:20.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:20.228 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T11:00:20.287 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:00:20.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.482+0000 7f31e02c7700 1 Processor -- start 2023-10-16T11:00:20.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.482+0000 7f31e02c7700 1 -- start start 2023-10-16T11:00:20.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.484+0000 7f31e02c7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f31d80c27c0 0x7f31d80b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:20.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.484+0000 7f31e02c7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f31d8161c70 0x7f31d80cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:20.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.484+0000 7f31e02c7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d80ca280 con 0x7f31d80b0c00 2023-10-16T11:00:20.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.484+0000 7f31e02c7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d80c9740 con 0x7f31d8161c70 2023-10-16T11:00:20.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.484+0000 7f31e02c7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d80b1520 con 0x7f31d80c27c0 2023-10-16T11:00:20.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.484+0000 7f31de063700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f31d80b0c00 0x7f31d8165ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60630/0 (socket says 172.21.15.158:60630) 2023-10-16T11:00:20.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.484+0000 7f31de063700 1 -- 172.21.15.158:0/2712853034 learned_addr learned my addr 172.21.15.158:0/2712853034 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:20.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.485+0000 7f31dd061700 1 -- 172.21.15.158:0/2712853034 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346279530 0 0) 0x7f31d80ca280 con 0x7f31d80b0c00 2023-10-16T11:00:20.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.485+0000 7f31dd061700 1 -- 172.21.15.158:0/2712853034 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c40035b0 con 0x7f31d80b0c00 2023-10-16T11:00:20.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.485+0000 7f31de864700 1 --2- 172.21.15.158:0/2712853034 >> v2:172.21.15.158:3300/0 conn(0x7f31d8161c70 0x7f31d80cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:20.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.485+0000 7f31dd862700 1 --2- 172.21.15.158:0/2712853034 >> v2:172.21.15.172:3300/0 conn(0x7f31d80c27c0 0x7f31d80b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:20.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.486+0000 7f31de864700 1 -- 172.21.15.158:0/2712853034 >> v2:172.21.15.172:3300/0 conn(0x7f31d80c27c0 msgr2=0x7f31d80b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:20.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.486+0000 7f31de864700 1 --2- 172.21.15.158:0/2712853034 >> v2:172.21.15.172:3300/0 conn(0x7f31d80c27c0 0x7f31d80b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:20.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.486+0000 7f31de864700 1 -- 172.21.15.158:0/2712853034 >> v1:172.21.15.158:6789/0 conn(0x7f31d80b0c00 legacy=0x7f31d8165ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:20.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.486+0000 7f31de864700 1 -- 172.21.15.158:0/2712853034 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d80c99e0 con 0x7f31d8161c70 2023-10-16T11:00:20.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.486+0000 7f31de864700 1 --2- 172.21.15.158:0/2712853034 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d8161c70 0x7f31d80cd0c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f31d80cede0 tx=0x7f31d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f028a9c63e8ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:20.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.487+0000 7f31dd061700 1 -- 172.21.15.158:0/2712853034 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d000e070 con 0x7f31d8161c70 2023-10-16T11:00:20.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.487+0000 7f31dd061700 1 -- 172.21.15.158:0/2712853034 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d000bae0 con 0x7f31d8161c70 2023-10-16T11:00:20.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.487+0000 7f31dd061700 1 -- 172.21.15.158:0/2712853034 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d000be90 con 0x7f31d8161c70 2023-10-16T11:00:20.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.488+0000 7f31e02c7700 1 -- 172.21.15.158:0/2712853034 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d8161c70 msgr2=0x7f31d80cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:20.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.488+0000 7f31e02c7700 1 --2- 172.21.15.158:0/2712853034 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d8161c70 0x7f31d80cd0c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f31d80cede0 tx=0x7f31d000b330 comp rx=0 tx=0).stop 2023-10-16T11:00:20.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.488+0000 7f31e02c7700 1 -- 172.21.15.158:0/2712853034 shutdown_connections 2023-10-16T11:00:20.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.488+0000 7f31e02c7700 1 --2- 172.21.15.158:0/2712853034 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d8161c70 0x7f31d80cd0c0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:20.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.488+0000 7f31e02c7700 1 --2- 172.21.15.158:0/2712853034 >> v2:172.21.15.172:3300/0 conn(0x7f31d80c27c0 0x7f31d80b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:20.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.488+0000 7f31e02c7700 1 -- 172.21.15.158:0/2712853034 >> 172.21.15.158:0/2712853034 conn(0x7f31d806af00 msgr2=0x7f31d80cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:20.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.490+0000 7f31e02c7700 1 -- 172.21.15.158:0/2712853034 shutdown_connections 2023-10-16T11:00:20.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.490+0000 7f31e02c7700 1 -- 172.21.15.158:0/2712853034 wait complete. 2023-10-16T11:00:20.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.492+0000 7f31e02c7700 1 Processor -- start 2023-10-16T11:00:20.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.493+0000 7f31e02c7700 1 -- start start 2023-10-16T11:00:20.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31e02c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d80b0c00 0x7f31d814c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:20.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31e02c7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d80c27c0 0x7f31d81519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:20.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31e02c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80b2060 0x7f31d80b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:20.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31e02c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d81fb540 con 0x7f31d80b0c00 2023-10-16T11:00:20.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31e02c7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d814c5e0 con 0x7f31d80c27c0 2023-10-16T11:00:20.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31e02c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80b3ce0 con 0x7f31d80b2060 2023-10-16T11:00:20.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31dd862700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d80c27c0 0x7f31d81519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:20.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31dd862700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d80c27c0 0x7f31d81519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:33410/0 (socket says 172.21.15.158:33410) 2023-10-16T11:00:20.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.494+0000 7f31dd862700 1 -- 172.21.15.158:0/525852010 learned_addr learned my addr 172.21.15.158:0/525852010 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:20.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.495+0000 7f31de063700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d80b0c00 0x7f31d814c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:20.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.495+0000 7f31dd862700 1 -- 172.21.15.158:0/525852010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80b2060 msgr2=0x7f31d80b24f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:20.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.495+0000 7f31dd862700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80b2060 0x7f31d80b24f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:20.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.495+0000 7f31dd862700 1 -- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d80b0c00 msgr2=0x7f31d814c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:20.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.495+0000 7f31dd862700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d80b0c00 0x7f31d814c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:20.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.495+0000 7f31dd862700 1 -- 172.21.15.158:0/525852010 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0009a70 con 0x7f31d80c27c0 2023-10-16T11:00:20.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.496+0000 7f31dd862700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d80c27c0 0x7f31d81519c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f31d400e810 tx=0x7f31d400ebd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:20.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.496+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4013610 con 0x7f31d80c27c0 2023-10-16T11:00:20.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.496+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d4013770 con 0x7f31d80c27c0 2023-10-16T11:00:20.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.496+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d8151f00 con 0x7f31d80c27c0 2023-10-16T11:00:20.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.496+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4013b20 con 0x7f31d80c27c0 2023-10-16T11:00:20.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.496+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d81faca0 con 0x7f31d80c27c0 2023-10-16T11:00:20.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.498+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d80b3ea0 con 0x7f31d80c27c0 2023-10-16T11:00:20.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.500+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 93) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f31d4013c80 con 0x7f31d80c27c0 2023-10-16T11:00:20.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.500+0000 7f31ceffd700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f31bc06b070 0x7f31bc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:20.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.500+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f31d4089ec0 con 0x7f31d80c27c0 2023-10-16T11:00:20.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.500+0000 7f31de063700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f31bc06b070 0x7f31bc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:20.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.502+0000 7f31de063700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f31bc06b070 0x7f31bc06d530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f31c8007fd0 tx=0x7f31c8007c20 comp rx=0 tx=0).ready entity=mgr.9573 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:20.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.507+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31d4054880 con 0x7f31d80c27c0 2023-10-16T11:00:20.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:20.930+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f31d80681b0 con 0x7f31d80c27c0 2023-10-16T11:00:21.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.382+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f31d4054c50 con 0x7f31d80c27c0 2023-10-16T11:00:21.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.384+0000 7f31ceffd700 1 -- 172.21.15.158:0/525852010 <== mon.1 v2:172.21.15.172:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v94) v1 ==== 93+0+0 (secure 0 0 0) 0x7f31d40723f0 con 0x7f31d80c27c0 2023-10-16T11:00:21.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f31bc06b070 msgr2=0x7f31bc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f31bc06b070 0x7f31bc06d530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f31c8007fd0 tx=0x7f31c8007c20 comp rx=0 tx=0).stop 2023-10-16T11:00:21.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d80c27c0 msgr2=0x7f31d81519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d80c27c0 0x7f31d81519c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f31d400e810 tx=0x7f31d400ebd0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 shutdown_connections 2023-10-16T11:00:21.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f31bc06b070 0x7f31bc06d530 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31d80b2060 0x7f31d80b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31d80b0c00 0x7f31d814c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 --2- 172.21.15.158:0/525852010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31d80c27c0 0x7f31d81519c0 secure :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f31d400e810 tx=0x7f31d400ebd0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 >> 172.21.15.158:0/525852010 conn(0x7f31d806af00 msgr2=0x7f31d815d720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:21.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 shutdown_connections 2023-10-16T11:00:21.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.389+0000 7f31e02c7700 1 -- 172.21.15.158:0/525852010 wait complete. 2023-10-16T11:00:21.411 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:00:21.411 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:00:21.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:00:21.499 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:00:21.500 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:21.500 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:21.519 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:21.520 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:21.541 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:21.541+0000 7fe0ee3c7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:00:21.543 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:21.542+0000 7f2ab7708200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:00:21.561 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:21.607 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:21.606+0000 7fe0ee3c7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:21.615 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:21.614+0000 7f2ab7708200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:21.679 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:21.679+0000 7fe0ee3c7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:00:21.692 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:21.691+0000 7f2ab7708200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:00:21.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.758+0000 7f5a37bc0700 1 Processor -- start 2023-10-16T11:00:21.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.758+0000 7f5a37bc0700 1 -- start start 2023-10-16T11:00:21.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.759+0000 7f5a37bc0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a3015f600 0x7f5a3006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:21.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.759+0000 7f5a37bc0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a30162fe0 con 0x7f5a3006ea70 2023-10-16T11:00:21.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.759+0000 7f5a37bc0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3006dcb0 con 0x7f5a30162980 2023-10-16T11:00:21.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.759+0000 7f5a37bc0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a30162df0 con 0x7f5a3015f600 2023-10-16T11:00:21.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.760+0000 7f5a3615d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5a30162980 0x7f5a300ccff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:36380/0 (socket says 172.21.15.158:36380) 2023-10-16T11:00:21.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.760+0000 7f5a3615d700 1 -- 172.21.15.158:0/3838688785 learned_addr learned my addr 172.21.15.158:0/3838688785 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:21.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.760+0000 7f5a3595c700 1 --2- 172.21.15.158:0/3838688785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a3015f600 0x7f5a3006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:21.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.760+0000 7f5a3495a700 1 -- 172.21.15.158:0/3838688785 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329459491 0 0) 0x7f5a30162fe0 con 0x7f5a3006ea70 2023-10-16T11:00:21.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.760+0000 7f5a3495a700 1 -- 172.21.15.158:0/3838688785 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c0035b0 con 0x7f5a3006ea70 2023-10-16T11:00:21.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.760+0000 7f5a3495a700 1 -- 172.21.15.158:0/3838688785 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84049137 0 0) 0x7f5a3006dcb0 con 0x7f5a30162980 2023-10-16T11:00:21.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.760+0000 7f5a3495a700 1 -- 172.21.15.158:0/3838688785 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a30162fe0 con 0x7f5a30162980 2023-10-16T11:00:21.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.761+0000 7f5a3595c700 1 -- 172.21.15.158:0/3838688785 >> v1:172.21.15.172:6789/0 conn(0x7f5a30162980 legacy=0x7f5a300ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.761+0000 7f5a3595c700 1 -- 172.21.15.158:0/3838688785 >> v1:172.21.15.158:6789/0 conn(0x7f5a3006ea70 legacy=0x7f5a30162270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.761+0000 7f5a3595c700 1 -- 172.21.15.158:0/3838688785 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3006df90 con 0x7f5a3015f600 2023-10-16T11:00:21.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.761+0000 7f5a3595c700 1 --2- 172.21.15.158:0/3838688785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a3015f600 0x7f5a3006c550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009870 tx=0x7f5a2c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e88333a4d95ddbaf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:21.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.761+0000 7f5a3495a700 1 -- 172.21.15.158:0/3838688785 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c011830 con 0x7f5a3015f600 2023-10-16T11:00:21.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.761+0000 7f5a3495a700 1 -- 172.21.15.158:0/3838688785 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c011990 con 0x7f5a3015f600 2023-10-16T11:00:21.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.761+0000 7f5a3495a700 1 -- 172.21.15.158:0/3838688785 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c011d20 con 0x7f5a3015f600 2023-10-16T11:00:21.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.762+0000 7f5a37bc0700 1 -- 172.21.15.158:0/3838688785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a3015f600 msgr2=0x7f5a3006c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.762+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/3838688785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a3015f600 0x7f5a3006c550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009870 tx=0x7f5a2c009b80 comp rx=0 tx=0).stop 2023-10-16T11:00:21.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.763+0000 7f5a37bc0700 1 -- 172.21.15.158:0/3838688785 shutdown_connections 2023-10-16T11:00:21.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.763+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/3838688785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a3015f600 0x7f5a3006c550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.763+0000 7f5a37bc0700 1 -- 172.21.15.158:0/3838688785 >> 172.21.15.158:0/3838688785 conn(0x7f5a300cbc00 msgr2=0x7f5a300cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:21.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.763+0000 7f5a37bc0700 1 -- 172.21.15.158:0/3838688785 shutdown_connections 2023-10-16T11:00:21.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.763+0000 7f5a37bc0700 1 -- 172.21.15.158:0/3838688785 wait complete. 2023-10-16T11:00:21.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.765+0000 7f5a37bc0700 1 Processor -- start 2023-10-16T11:00:21.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.765+0000 7f5a37bc0700 1 -- start start 2023-10-16T11:00:21.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a37bc0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a3006ea70 0x7f5a30149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:21.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a37bc0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a3015f600 0x7f5a300c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:21.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a37bc0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30162980 0x7f5a30159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:21.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a37bc0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300c83e0 con 0x7f5a3006ea70 2023-10-16T11:00:21.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a37bc0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300c86c0 con 0x7f5a3015f600 2023-10-16T11:00:21.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a37bc0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300c89a0 con 0x7f5a30162980 2023-10-16T11:00:21.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a3006ea70 0x7f5a30149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:21.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a3006ea70 0x7f5a30149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60536/0 (socket says 172.21.15.158:60536) 2023-10-16T11:00:21.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 -- 172.21.15.158:0/373585597 learned_addr learned my addr 172.21.15.158:0/373585597 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:21.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3615d700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30162980 0x7f5a30159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:21.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3515b700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a3015f600 0x7f5a300c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:21.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 -- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30162980 msgr2=0x7f5a30159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30162980 0x7f5a30159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 -- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a3015f600 msgr2=0x7f5a300c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a3015f600 0x7f5a300c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.766+0000 7f5a3595c700 1 -- 172.21.15.158:0/373585597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c0094a0 con 0x7f5a3006ea70 2023-10-16T11:00:21.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.767+0000 7f5a3595c700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a3006ea70 0x7f5a30149380 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0090b0 tx=0x7f5a2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:21.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.767+0000 7f5a267fc700 1 -- 172.21.15.158:0/373585597 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c01a6e0 con 0x7f5a3006ea70 2023-10-16T11:00:21.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.767+0000 7f5a267fc700 1 -- 172.21.15.158:0/373585597 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c01a840 con 0x7f5a3006ea70 2023-10-16T11:00:21.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.767+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a300c8c80 con 0x7f5a3006ea70 2023-10-16T11:00:21.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.767+0000 7f5a267fc700 1 -- 172.21.15.158:0/373585597 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c01abb0 con 0x7f5a3006ea70 2023-10-16T11:00:21.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.767+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a301498c0 con 0x7f5a3006ea70 2023-10-16T11:00:21.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.768+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a300c91a0 con 0x7f5a3006ea70 2023-10-16T11:00:21.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.768+0000 7f5a267fc700 1 -- 172.21.15.158:0/373585597 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5a2c022070 con 0x7f5a3006ea70 2023-10-16T11:00:21.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.769+0000 7f5a267fc700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 0x7f5a1406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.769+0000 7f5a3515b700 1 -- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 msgr2=0x7f5a1406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.769+0000 7f5a267fc700 1 -- 172.21.15.158:0/373585597 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f5a2c08abb0 con 0x7f5a3006ea70 2023-10-16T11:00:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.769+0000 7f5a3515b700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 0x7f5a1406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.774+0000 7f5a267fc700 1 -- 172.21.15.158:0/373585597 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a300c91a0 con 0x7f5a3006ea70 2023-10-16T11:00:21.783 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:21.782+0000 7fe0ee3c7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:00:21.801 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:21.800+0000 7f2ab7708200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:00:21.924 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:21.924+0000 7fe0ee3c7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:00:21.946 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:21.945+0000 7f2ab7708200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:00:21.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.970+0000 7f5a3515b700 1 -- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 msgr2=0x7f5a1406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:21.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.970+0000 7f5a3515b700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 0x7f5a1406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:21.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.972+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a300c91a0 con 0x7f5a3006ea70 2023-10-16T11:00:21.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.974+0000 7f5a267fc700 1 -- 172.21.15.158:0/373585597 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+18109 (secure 0 0 0) 0x7f5a300c91a0 con 0x7f5a3006ea70 2023-10-16T11:00:21.976 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:21.976 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":158,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:21.373390+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:04:02.482303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T10:04:03.019780+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2640446942":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3643659736":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1187395764":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/2828352389":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6833/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6832/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3333502274":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1458403109":"2023-10-17T10:04:17.957825+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":50,"length":7},{"begin":106,"length":7}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:21.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 msgr2=0x7f5a1406d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:21.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 0x7f5a1406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a3006ea70 msgr2=0x7f5a30149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:21.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a3006ea70 0x7f5a30149380 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0090b0 tx=0x7f5a2c004750 comp rx=0 tx=0).stop 2023-10-16T11:00:21.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 shutdown_connections 2023-10-16T11:00:21.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f5a1406aff0 0x7f5a1406d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a30162980 0x7f5a30159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a3006ea70 0x7f5a30149380 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 --2- 172.21.15.158:0/373585597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a3015f600 0x7f5a300c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:21.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 >> 172.21.15.158:0/373585597 conn(0x7f5a300cbc00 msgr2=0x7f5a3015e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:21.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 shutdown_connections 2023-10-16T11:00:21.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:21.976+0000 7f5a37bc0700 1 -- 172.21.15.158:0/373585597 wait complete. 2023-10-16T11:00:21.991 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:04:02.482303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:04:03.019780+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 158, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:00:21.992 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:00:22.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.155+0000 7fb95469d700 1 Processor -- start 2023-10-16T11:00:22.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.155+0000 7fb95469d700 1 -- start start 2023-10-16T11:00:22.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.156+0000 7fb95469d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c15dc40 0x7fb94c15ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.156+0000 7fb95469d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb94c06d9c0 0x7fb94c06de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.156+0000 7fb95469d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb94c06f180 0x7fb94c06f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.156+0000 7fb95469d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c06e380 con 0x7fb94c06f180 2023-10-16T11:00:22.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.156+0000 7fb95469d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c15f380 con 0x7fb94c06d9c0 2023-10-16T11:00:22.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.156+0000 7fb95469d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c071e70 con 0x7fb94c15dc40 2023-10-16T11:00:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb94c06f180 0x7fb94c06f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb94c06f180 0x7fb94c06f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60594/0 (socket says 172.21.15.158:60594) 2023-10-16T11:00:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 -- 172.21.15.158:0/2768666840 learned_addr learned my addr 172.21.15.158:0/2768666840 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:22.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952439700 1 --2- 172.21.15.158:0/2768666840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c15dc40 0x7fb94c15ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb951c38700 1 --2- 172.21.15.158:0/2768666840 >> v2:172.21.15.172:3300/0 conn(0x7fb94c06d9c0 0x7fb94c06de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 -- 172.21.15.158:0/2768666840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c15dc40 msgr2=0x7fb94c15ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 --2- 172.21.15.158:0/2768666840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c15dc40 0x7fb94c15ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 -- 172.21.15.158:0/2768666840 >> v2:172.21.15.172:3300/0 conn(0x7fb94c06d9c0 msgr2=0x7fb94c06de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 --2- 172.21.15.158:0/2768666840 >> v2:172.21.15.172:3300/0 conn(0x7fb94c06d9c0 0x7fb94c06de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 -- 172.21.15.158:0/2768666840 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c072150 con 0x7fb94c06f180 2023-10-16T11:00:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb952c3a700 1 --2- 172.21.15.158:0/2768666840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c06f180 0x7fb94c06f610 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fb94c04fd10 tx=0x7fb94400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4446521bbaf72f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb951437700 1 -- 172.21.15.158:0/2768666840 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94400e070 con 0x7fb94c06f180 2023-10-16T11:00:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb951437700 1 -- 172.21.15.158:0/2768666840 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94400b910 con 0x7fb94c06f180 2023-10-16T11:00:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.157+0000 7fb951437700 1 -- 172.21.15.158:0/2768666840 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94400bcc0 con 0x7fb94c06f180 2023-10-16T11:00:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 -- 172.21.15.158:0/2768666840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c06f180 msgr2=0x7fb94c06f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 --2- 172.21.15.158:0/2768666840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c06f180 0x7fb94c06f610 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fb94c04fd10 tx=0x7fb94400b330 comp rx=0 tx=0).stop 2023-10-16T11:00:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 -- 172.21.15.158:0/2768666840 shutdown_connections 2023-10-16T11:00:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 --2- 172.21.15.158:0/2768666840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c06f180 0x7fb94c06f610 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 --2- 172.21.15.158:0/2768666840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c15dc40 0x7fb94c15ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 --2- 172.21.15.158:0/2768666840 >> v2:172.21.15.172:3300/0 conn(0x7fb94c06d9c0 0x7fb94c06de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 -- 172.21.15.158:0/2768666840 >> 172.21.15.158:0/2768666840 conn(0x7fb94c0d03b0 msgr2=0x7fb94c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 -- 172.21.15.158:0/2768666840 shutdown_connections 2023-10-16T11:00:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.158+0000 7fb95469d700 1 -- 172.21.15.158:0/2768666840 wait complete. 2023-10-16T11:00:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.160+0000 7fb95469d700 1 Processor -- start 2023-10-16T11:00:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.160+0000 7fb95469d700 1 -- start start 2023-10-16T11:00:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.160+0000 7fb95469d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c06d9c0 0x7fb94c0aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.160+0000 7fb95469d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c06f180 0x7fb94c0ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.160+0000 7fb95469d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c15dc40 0x7fb94c0c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.160+0000 7fb95469d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0af3b0 con 0x7fb94c15dc40 2023-10-16T11:00:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb95469d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1f33a0 con 0x7fb94c06d9c0 2023-10-16T11:00:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb95469d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1f3680 con 0x7fb94c06f180 2023-10-16T11:00:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c15dc40 0x7fb94c0c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c15dc40 0x7fb94c0c3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60600/0 (socket says 172.21.15.158:60600) 2023-10-16T11:00:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb951c38700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c06f180 0x7fb94c0ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952439700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c06d9c0 0x7fb94c0aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 -- 172.21.15.158:0/2432779624 learned_addr learned my addr 172.21.15.158:0/2432779624 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 -- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c06f180 msgr2=0x7fb94c0ad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c06f180 0x7fb94c0ad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 -- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c06d9c0 msgr2=0x7fb94c0aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c06d9c0 0x7fb94c0aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 -- 172.21.15.158:0/2432779624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944009b30 con 0x7fb94c15dc40 2023-10-16T11:00:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952439700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c06d9c0 0x7fb94c0aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:22.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb951c38700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c06f180 0x7fb94c0ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:22.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb952c3a700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c15dc40 0x7fb94c0c3ee0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0cdf30 tx=0x7fb944004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:22.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb942ffd700 1 -- 172.21.15.158:0/2432779624 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440099b0 con 0x7fb94c15dc40 2023-10-16T11:00:22.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb942ffd700 1 -- 172.21.15.158:0/2432779624 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94400ce80 con 0x7fb94c15dc40 2023-10-16T11:00:22.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb942ffd700 1 -- 172.21.15.158:0/2432779624 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944004490 con 0x7fb94c15dc40 2023-10-16T11:00:22.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c1f3960 con 0x7fb94c15dc40 2023-10-16T11:00:22.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.161+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c1f2fb0 con 0x7fb94c15dc40 2023-10-16T11:00:22.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.162+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c1f3170 con 0x7fb94c15dc40 2023-10-16T11:00:22.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.163+0000 7fb942ffd700 1 -- 172.21.15.158:0/2432779624 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb9440045f0 con 0x7fb94c15dc40 2023-10-16T11:00:22.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.163+0000 7fb942ffd700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.163+0000 7fb952439700 1 -- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 msgr2=0x7fb93006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:22.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.163+0000 7fb942ffd700 1 -- 172.21.15.158:0/2432779624 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fb94408b2c0 con 0x7fb94c15dc40 2023-10-16T11:00:22.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.163+0000 7fb952439700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:22.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.168+0000 7fb942ffd700 1 -- 172.21.15.158:0/2432779624 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb944055d20 con 0x7fb94c15dc40 2023-10-16T11:00:22.248 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:22.248+0000 7fe0ee3c7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:00:22.290 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:22.289+0000 7f2ab7708200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:00:22.362 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:22.362+0000 7fe0ee3c7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:00:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.364+0000 7fb952439700 1 -- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 msgr2=0x7fb93006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.364+0000 7fb952439700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.366+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb94c0681b0 con 0x7fb94c15dc40 2023-10-16T11:00:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.366+0000 7fb942ffd700 1 -- 172.21.15.158:0/2432779624 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb944020020 con 0x7fb94c15dc40 2023-10-16T11:00:22.368 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:00:22.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 msgr2=0x7fb93006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c15dc40 msgr2=0x7fb94c0c3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c15dc40 0x7fb94c0c3ee0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0cdf30 tx=0x7fb944004ad0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 shutdown_connections 2023-10-16T11:00:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb94c15dc40 0x7fb94c0c3ee0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c06d9c0 0x7fb94c0aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 --2- 172.21.15.158:0/2432779624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb94c06f180 0x7fb94c0ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 >> 172.21.15.158:0/2432779624 conn(0x7fb94c0d03b0 msgr2=0x7fb94c0cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 shutdown_connections 2023-10-16T11:00:22.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.369+0000 7fb95469d700 1 -- 172.21.15.158:0/2432779624 wait complete. 2023-10-16T11:00:22.384 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:00:22.412 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:22.411+0000 7f2ab7708200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:00:22.435 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:22.435+0000 7fe0ee3c7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:00:22.490 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:22.489+0000 7f2ab7708200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:00:22.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.509+0000 7fe18fc93700 1 Processor -- start 2023-10-16T11:00:22.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.509+0000 7fe18fc93700 1 -- start start 2023-10-16T11:00:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18fc93700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe18815d6c0 0x7fe1880cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18fc93700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe18815daa0 0x7fe1880d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18fc93700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1880d1ee0 con 0x7fe1880d2370 2023-10-16T11:00:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18fc93700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1880d40a0 con 0x7fe18815d6c0 2023-10-16T11:00:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18fc93700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1880d12d0 con 0x7fe18815daa0 2023-10-16T11:00:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18da2f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe18815d6c0 0x7fe1880cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18da2f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe18815d6c0 0x7fe1880cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60606/0 (socket says 172.21.15.158:60606) 2023-10-16T11:00:22.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18da2f700 1 -- 172.21.15.158:0/560291507 learned_addr learned my addr 172.21.15.158:0/560291507 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:22.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.510+0000 7fe18d22e700 1 --2- 172.21.15.158:0/560291507 >> v2:172.21.15.172:3300/0 conn(0x7fe18815daa0 0x7fe1880d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18da2f700 1 -- 172.21.15.158:0/560291507 >> v2:172.21.15.172:3300/0 conn(0x7fe18815daa0 msgr2=0x7fe1880d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18da2f700 1 --2- 172.21.15.158:0/560291507 >> v2:172.21.15.172:3300/0 conn(0x7fe18815daa0 0x7fe1880d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18da2f700 1 -- 172.21.15.158:0/560291507 >> v1:172.21.15.172:6789/0 conn(0x7fe1880d2370 legacy=0x7fe1880cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18da2f700 1 -- 172.21.15.158:0/560291507 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880d15b0 con 0x7fe18815d6c0 2023-10-16T11:00:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18da2f700 1 --2- 172.21.15.158:0/560291507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe18815d6c0 0x7fe1880cff30 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fe188162470 tx=0x7fe184009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee2cb581b9e896c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18ca2d700 1 -- 172.21.15.158:0/560291507 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184011830 con 0x7fe18815d6c0 2023-10-16T11:00:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18ca2d700 1 -- 172.21.15.158:0/560291507 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184011990 con 0x7fe18815d6c0 2023-10-16T11:00:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18ca2d700 1 -- 172.21.15.158:0/560291507 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184011d20 con 0x7fe18815d6c0 2023-10-16T11:00:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18fc93700 1 -- 172.21.15.158:0/560291507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe18815d6c0 msgr2=0x7fe1880cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18fc93700 1 --2- 172.21.15.158:0/560291507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe18815d6c0 0x7fe1880cff30 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fe188162470 tx=0x7fe184009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18fc93700 1 -- 172.21.15.158:0/560291507 shutdown_connections 2023-10-16T11:00:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18fc93700 1 --2- 172.21.15.158:0/560291507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe18815d6c0 0x7fe1880cff30 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18fc93700 1 --2- 172.21.15.158:0/560291507 >> v2:172.21.15.172:3300/0 conn(0x7fe18815daa0 0x7fe1880d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.511+0000 7fe18fc93700 1 -- 172.21.15.158:0/560291507 >> 172.21.15.158:0/560291507 conn(0x7fe18809c940 msgr2=0x7fe1880d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.512+0000 7fe18fc93700 1 -- 172.21.15.158:0/560291507 shutdown_connections 2023-10-16T11:00:22.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.512+0000 7fe18fc93700 1 -- 172.21.15.158:0/560291507 wait complete. 2023-10-16T11:00:22.520 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:22.516+0000 7fe0ee3c7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.513+0000 7fe18fc93700 1 Processor -- start 2023-10-16T11:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.513+0000 7fe18fc93700 1 -- start start 2023-10-16T11:00:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.513+0000 7fe18fc93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1880d2370 0x7fe1881f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.513+0000 7fe18fc93700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe18815d6c0 0x7fe18814e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18fc93700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe18815daa0 0x7fe1881601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18fc93700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1880ca6a0 con 0x7fe1880d2370 2023-10-16T11:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18fc93700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1880ca980 con 0x7fe18815d6c0 2023-10-16T11:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18fc93700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1881f6cc0 con 0x7fe18815daa0 2023-10-16T11:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1880d2370 0x7fe1881f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1880d2370 0x7fe1881f6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60612/0 (socket says 172.21.15.158:60612) 2023-10-16T11:00:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 -- 172.21.15.158:0/4014565673 learned_addr learned my addr 172.21.15.158:0/4014565673 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18d22e700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe18815d6c0 0x7fe18814e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18e230700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe18815daa0 0x7fe1881601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 -- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe18815daa0 msgr2=0x7fe1881601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe18815daa0 0x7fe1881601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 -- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe18815d6c0 msgr2=0x7fe18814e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe18815d6c0 0x7fe18814e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 -- 172.21.15.158:0/4014565673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe184009550 con 0x7fe1880d2370 2023-10-16T11:00:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18da2f700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1880d2370 0x7fe1881f6780 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fe184000c00 tx=0x7fe184004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe17e7fc700 1 -- 172.21.15.158:0/4014565673 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184004400 con 0x7fe1880d2370 2023-10-16T11:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe17e7fc700 1 -- 172.21.15.158:0/4014565673 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18401f070 con 0x7fe1880d2370 2023-10-16T11:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe17e7fc700 1 -- 172.21.15.158:0/4014565673 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1840038e0 con 0x7fe1880d2370 2023-10-16T11:00:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1881f6f60 con 0x7fe1880d2370 2023-10-16T11:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.514+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1881f7520 con 0x7fe1880d2370 2023-10-16T11:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.515+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1880681b0 con 0x7fe1880d2370 2023-10-16T11:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.516+0000 7fe17e7fc700 1 -- 172.21.15.158:0/4014565673 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe184021030 con 0x7fe1880d2370 2023-10-16T11:00:22.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.516+0000 7fe17e7fc700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 0x7fe16c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.516+0000 7fe18d22e700 1 -- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 msgr2=0x7fe16c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.516+0000 7fe17e7fc700 1 -- 172.21.15.158:0/4014565673 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fe18408af60 con 0x7fe1880d2370 2023-10-16T11:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.516+0000 7fe18d22e700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 0x7fe16c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:22.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.521+0000 7fe17e7fc700 1 -- 172.21.15.158:0/4014565673 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe184055a30 con 0x7fe1880d2370 2023-10-16T11:00:22.576 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:22.575+0000 7f2ab7708200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:00:22.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.717+0000 7fe18d22e700 1 -- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 msgr2=0x7fe16c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:22.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.717+0000 7fe18d22e700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 0x7fe16c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:22.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.720+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe1880d0d30 con 0x7fe1880d2370 2023-10-16T11:00:22.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.720+0000 7fe17e7fc700 1 -- 172.21.15.158:0/4014565673 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v158) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe184059050 con 0x7fe1880d2370 2023-10-16T11:00:22.722 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:00:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 msgr2=0x7fe16c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 0x7fe16c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1880d2370 msgr2=0x7fe1881f6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1880d2370 0x7fe1881f6780 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fe184000c00 tx=0x7fe184004750 comp rx=0 tx=0).stop 2023-10-16T11:00:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 shutdown_connections 2023-10-16T11:00:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fe16c06b020 0x7fe16c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe18815daa0 0x7fe1881601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1880d2370 0x7fe1881f6780 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 --2- 172.21.15.158:0/4014565673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe18815d6c0 0x7fe18814e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 >> 172.21.15.158:0/4014565673 conn(0x7fe18809c940 msgr2=0x7fe18806d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:22.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.722+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 shutdown_connections 2023-10-16T11:00:22.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.723+0000 7fe18fc93700 1 -- 172.21.15.158:0/4014565673 wait complete. 2023-10-16T11:00:22.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:00:22.885 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:22.884+0000 7fe0ee3c7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:00:22.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.901+0000 7f500acfd700 1 Processor -- start 2023-10-16T11:00:22.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.901+0000 7f500acfd700 1 -- start start 2023-10-16T11:00:22.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f500acfd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f500406e0d0 0x7f500415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f500acfd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040cbcf0 0x7f50040cc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f500acfd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500415f380 con 0x7f500415dc40 2023-10-16T11:00:22.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f500acfd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5004071e70 con 0x7f500406e0d0 2023-10-16T11:00:22.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f500acfd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5004072150 con 0x7f50040cbcf0 2023-10-16T11:00:22.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f5008a99700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f500415dc40 0x7f500406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60842/0 (socket says 172.21.15.158:60842) 2023-10-16T11:00:22.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f5008a99700 1 -- 172.21.15.158:0/4015468836 learned_addr learned my addr 172.21.15.158:0/4015468836 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:22.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f5003fff700 1 --2- 172.21.15.158:0/4015468836 >> v2:172.21.15.172:3300/0 conn(0x7f500406e0d0 0x7f500415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f500929a700 1 --2- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040cbcf0 0x7f50040cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f50037fe700 1 -- 172.21.15.158:0/4015468836 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181661733 0 0) 0x7f4ff8001110 con 0x7f500415dc40 2023-10-16T11:00:22.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f50037fe700 1 -- 172.21.15.158:0/4015468836 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff40035b0 con 0x7f500415dc40 2023-10-16T11:00:22.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f5003fff700 1 -- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040cbcf0 msgr2=0x7f50040cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:22.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f50037fe700 1 -- 172.21.15.158:0/4015468836 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788122989 0 0) 0x7f4ff40035b0 con 0x7f500415dc40 2023-10-16T11:00:22.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f5003fff700 1 --2- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040cbcf0 0x7f50040cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f5003fff700 1 -- 172.21.15.158:0/4015468836 >> v1:172.21.15.158:6789/0 conn(0x7f500415dc40 legacy=0x7f500406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f5003fff700 1 -- 172.21.15.158:0/4015468836 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5004072430 con 0x7f500406e0d0 2023-10-16T11:00:22.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.902+0000 7f500929a700 1 --2- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040cbcf0 0x7f50040cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:22.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f5003fff700 1 --2- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 0x7f500415ee40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009840 tx=0x7f4ff0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abb1df94a6d65bbb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:22.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f50037fe700 1 -- 172.21.15.158:0/4015468836 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8001800 con 0x7f500406e0d0 2023-10-16T11:00:22.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f50037fe700 1 -- 172.21.15.158:0/4015468836 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff0004030 con 0x7f500406e0d0 2023-10-16T11:00:22.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f50037fe700 1 -- 172.21.15.158:0/4015468836 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff00043a0 con 0x7f500406e0d0 2023-10-16T11:00:22.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f500acfd700 1 -- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 msgr2=0x7f500415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f500acfd700 1 --2- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 0x7f500415ee40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009840 tx=0x7f4ff0009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:22.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f500acfd700 1 -- 172.21.15.158:0/4015468836 shutdown_connections 2023-10-16T11:00:22.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f500acfd700 1 --2- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040cbcf0 0x7f50040cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f500acfd700 1 --2- 172.21.15.158:0/4015468836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 0x7f500415ee40 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.903+0000 7f500acfd700 1 -- 172.21.15.158:0/4015468836 >> 172.21.15.158:0/4015468836 conn(0x7f50040d03b0 msgr2=0x7f50040d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:22.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.904+0000 7f500acfd700 1 -- 172.21.15.158:0/4015468836 shutdown_connections 2023-10-16T11:00:22.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.904+0000 7f500acfd700 1 -- 172.21.15.158:0/4015468836 wait complete. 2023-10-16T11:00:22.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.905+0000 7f500acfd700 1 Processor -- start 2023-10-16T11:00:22.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.905+0000 7f500acfd700 1 -- start start 2023-10-16T11:00:22.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 0x7f5004155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50040cbcf0 0x7f50040b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f500415dc40 0x7f50041e50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500414f0e0 con 0x7f50040cbcf0 2023-10-16T11:00:22.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50041f3040 con 0x7f500406e0d0 2023-10-16T11:00:22.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040b52b0 con 0x7f500415dc40 2023-10-16T11:00:22.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50040cbcf0 0x7f50040b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50040cbcf0 0x7f50040b4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60674/0 (socket says 172.21.15.158:60674) 2023-10-16T11:00:22.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 -- 172.21.15.158:0/3317974907 learned_addr learned my addr 172.21.15.158:0/3317974907 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:22.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5008a99700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 0x7f5004155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500929a700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f500415dc40 0x7f50041e50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:22.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 -- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f500415dc40 msgr2=0x7f50041e50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f500415dc40 0x7f50041e50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 -- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 msgr2=0x7f5004155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:22.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 0x7f5004155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:22.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 -- 172.21.15.158:0/3317974907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc009a70 con 0x7f50040cbcf0 2023-10-16T11:00:22.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f5003fff700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50040cbcf0 0x7f50040b4940 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009b90 tx=0x7f4ff0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:22.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f50017fa700 1 -- 172.21.15.158:0/3317974907 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0004590 con 0x7f50040cbcf0 2023-10-16T11:00:22.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f50017fa700 1 -- 172.21.15.158:0/3317974907 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff001f070 con 0x7f50040cbcf0 2023-10-16T11:00:22.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f50017fa700 1 -- 172.21.15.158:0/3317974907 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0011870 con 0x7f50040cbcf0 2023-10-16T11:00:22.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff00094a0 con 0x7f50040cbcf0 2023-10-16T11:00:22.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.906+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500415b6b0 con 0x7f50040cbcf0 2023-10-16T11:00:22.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.907+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500415b230 con 0x7f50040cbcf0 2023-10-16T11:00:22.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.908+0000 7f50017fa700 1 -- 172.21.15.158:0/3317974907 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f4ff00119d0 con 0x7f50040cbcf0 2023-10-16T11:00:22.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.908+0000 7f50017fa700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 0x7f4fe806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:22.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.908+0000 7f5008a99700 1 -- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 msgr2=0x7f4fe806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:22.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.908+0000 7f50017fa700 1 -- 172.21.15.158:0/3317974907 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f4ff0021070 con 0x7f50040cbcf0 2023-10-16T11:00:22.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.908+0000 7f5008a99700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 0x7f4fe806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:22.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:22.913+0000 7f50017fa700 1 -- 172.21.15.158:0/3317974907 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f500415b230 con 0x7f50040cbcf0 2023-10-16T11:00:22.967 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:22.966+0000 7f2ab7708200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:00:23.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.109+0000 7f5008a99700 1 -- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 msgr2=0x7f4fe806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:23.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.109+0000 7f5008a99700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 0x7f4fe806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:23.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.112+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f500415b230 con 0x7f50040cbcf0 2023-10-16T11:00:23.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.113+0000 7f50017fa700 1 -- 172.21.15.158:0/3317974907 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v158) v1 ==== 100+0+11 (secure 0 0 0) 0x7f500415b230 con 0x7f50040cbcf0 2023-10-16T11:00:23.114 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:00:23.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 msgr2=0x7f4fe806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:23.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 0x7f4fe806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50040cbcf0 msgr2=0x7f50040b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50040cbcf0 0x7f50040b4940 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009b90 tx=0x7f4ff0004750 comp rx=0 tx=0).stop 2023-10-16T11:00:23.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 shutdown_connections 2023-10-16T11:00:23.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f4fe806b020 0x7f4fe806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f500415dc40 0x7f50041e50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500406e0d0 0x7f5004155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 --2- 172.21.15.158:0/3317974907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50040cbcf0 0x7f50040b4940 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 >> 172.21.15.158:0/3317974907 conn(0x7f50040d03b0 msgr2=0x7f50040d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:23.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 shutdown_connections 2023-10-16T11:00:23.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.115+0000 7f500acfd700 1 -- 172.21.15.158:0/3317974907 wait complete. 2023-10-16T11:00:23.129 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:00:23.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.251+0000 7f53bb0d1700 1 Processor -- start 2023-10-16T11:00:23.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.251+0000 7f53bb0d1700 1 -- start start 2023-10-16T11:00:23.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.252+0000 7f53bb0d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b4168750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.252+0000 7f53bb0d1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40d1db0 con 0x7f53b415da60 2023-10-16T11:00:23.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.252+0000 7f53bb0d1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40cfd60 con 0x7f53b40ce5b0 2023-10-16T11:00:23.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.252+0000 7f53bb0d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40cfb40 con 0x7f53b40b5870 2023-10-16T11:00:23.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.252+0000 7f53b8e6d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f53b415da60 0x7f53b40cf430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60858/0 (socket says 172.21.15.158:60858) 2023-10-16T11:00:23.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.252+0000 7f53b8e6d700 1 -- 172.21.15.158:0/1590667066 learned_addr learned my addr 172.21.15.158:0/1590667066 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:23.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.252+0000 7f53b966e700 1 --2- 172.21.15.158:0/1590667066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b4168750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180452690 0 0) 0x7f53b40d1db0 con 0x7f53b415da60 2023-10-16T11:00:23.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a40035b0 con 0x7f53b415da60 2023-10-16T11:00:23.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390245596 0 0) 0x7f53b40cfd60 con 0x7f53b40ce5b0 2023-10-16T11:00:23.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b40d1db0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 489152910 0 0) 0x7f53a40035b0 con 0x7f53b415da60 2023-10-16T11:00:23.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b966e700 1 -- 172.21.15.158:0/1590667066 >> v1:172.21.15.172:6789/0 conn(0x7f53b40ce5b0 legacy=0x7f53b40b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b966e700 1 -- 172.21.15.158:0/1590667066 >> v1:172.21.15.158:6789/0 conn(0x7f53b415da60 legacy=0x7f53b40cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b966e700 1 -- 172.21.15.158:0/1590667066 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40d1760 con 0x7f53b40b5870 2023-10-16T11:00:23.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b966e700 1 --2- 172.21.15.158:0/1590667066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b4168750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009dc0 tx=0x7f53ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5679931535a189c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:23.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac00e070 con 0x7f53b40b5870 2023-10-16T11:00:23.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac003a20 con 0x7f53b40b5870 2023-10-16T11:00:23.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.253+0000 7f53b37fe700 1 -- 172.21.15.158:0/1590667066 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a80017f0 con 0x7f53b40b5870 2023-10-16T11:00:23.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.254+0000 7f53bb0d1700 1 -- 172.21.15.158:0/1590667066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 msgr2=0x7f53b4168750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.254+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/1590667066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b4168750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009dc0 tx=0x7f53ac00b330 comp rx=0 tx=0).stop 2023-10-16T11:00:23.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.254+0000 7f53bb0d1700 1 -- 172.21.15.158:0/1590667066 shutdown_connections 2023-10-16T11:00:23.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.254+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/1590667066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b4168750 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.254+0000 7f53bb0d1700 1 -- 172.21.15.158:0/1590667066 >> 172.21.15.158:0/1590667066 conn(0x7f53b409c9b0 msgr2=0x7f53b40d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:23.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.254+0000 7f53bb0d1700 1 -- 172.21.15.158:0/1590667066 shutdown_connections 2023-10-16T11:00:23.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.254+0000 7f53bb0d1700 1 -- 172.21.15.158:0/1590667066 wait complete. 2023-10-16T11:00:23.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.255+0000 7f53bb0d1700 1 Processor -- start 2023-10-16T11:00:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.255+0000 7f53bb0d1700 1 -- start start 2023-10-16T11:00:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53bb0d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b40cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53bb0d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b40ce5b0 0x7f53b414fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53bb0d1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53b415da60 0x7f53b4150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53bb0d1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b414dda0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53bb0d1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4160fa0 con 0x7f53b415da60 2023-10-16T11:00:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53bb0d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b414d6e0 con 0x7f53b40b5870 2023-10-16T11:00:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53b3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b40ce5b0 0x7f53b414fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53b8e6d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b40cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53b3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b40ce5b0 0x7f53b414fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60690/0 (socket says 172.21.15.158:60690) 2023-10-16T11:00:23.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.256+0000 7f53b3fff700 1 -- 172.21.15.158:0/711855779 learned_addr learned my addr 172.21.15.158:0/711855779 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:23.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b966e700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53b415da60 0x7f53b4150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b3fff700 1 -- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 msgr2=0x7f53b40cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b3fff700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b40cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b3fff700 1 -- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53b415da60 msgr2=0x7f53b4150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b3fff700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53b415da60 0x7f53b4150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b3fff700 1 -- 172.21.15.158:0/711855779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac009a70 con 0x7f53b40ce5b0 2023-10-16T11:00:23.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b8e6d700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b40cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:23.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b3fff700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b40ce5b0 0x7f53b414fae0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f53a000c500 tx=0x7f53a000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b17fa700 1 -- 172.21.15.158:0/711855779 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00052d0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b17fa700 1 -- 172.21.15.158:0/711855779 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0005430 con 0x7f53b40ce5b0 2023-10-16T11:00:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b414d9c0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53b17fa700 1 -- 172.21.15.158:0/711855779 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00038c0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.257+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b40cd0c0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.258+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b414dee0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.259+0000 7f53b17fa700 1 -- 172.21.15.158:0/711855779 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f53a0003a20 con 0x7f53b40ce5b0 2023-10-16T11:00:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.260+0000 7f53b17fa700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 0x7f539806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.260+0000 7f53b8e6d700 1 -- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 msgr2=0x7f539806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.260+0000 7f53b17fa700 1 -- 172.21.15.158:0/711855779 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f53a0088f40 con 0x7f53b40ce5b0 2023-10-16T11:00:23.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.260+0000 7f53b8e6d700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 0x7f539806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:23.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.264+0000 7f53b17fa700 1 -- 172.21.15.158:0/711855779 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53b414dee0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.454 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:23.454+0000 7fe0ee3c7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:00:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.461+0000 7f53b8e6d700 1 -- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 msgr2=0x7f539806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.461+0000 7f53b8e6d700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 0x7f539806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:23.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.513+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f53b414dee0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.514+0000 7f53b17fa700 1 -- 172.21.15.158:0/711855779 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v158) v1 ==== 80+0+478 (secure 0 0 0) 0x7f53b414dee0 con 0x7f53b40ce5b0 2023-10-16T11:00:23.515 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2640446942 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.516 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3643659736 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.516 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1187395764 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.516 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2828352389 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.516 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/3458664017 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.516 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/3458664017 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.517 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3333502274 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.517 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1458403109 2023-10-17T10:04:17.957825+0000 2023-10-16T11:00:23.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 msgr2=0x7f539806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:23.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 0x7f539806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b40ce5b0 msgr2=0x7f53b414fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b40ce5b0 0x7f53b414fae0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f53a000c500 tx=0x7f53a000c810 comp rx=0 tx=0).stop 2023-10-16T11:00:23.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 shutdown_connections 2023-10-16T11:00:23.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f539806b020 0x7f539806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53b415da60 0x7f53b4150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53b40b5870 0x7f53b40cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 --2- 172.21.15.158:0/711855779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b40ce5b0 0x7f53b414fae0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 >> 172.21.15.158:0/711855779 conn(0x7f53b409c9b0 msgr2=0x7f53b406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:23.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 shutdown_connections 2023-10-16T11:00:23.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.516+0000 7f53bb0d1700 1 -- 172.21.15.158:0/711855779 wait complete. 2023-10-16T11:00:23.520 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T11:00:23.523 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:23.523+0000 7fe0ee3c7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:00:23.532 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:00:23.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:23.532 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:00:23.570 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:23.569+0000 7f2ab7708200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:00:23.575 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:00:23.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:23.576 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:00:23.633 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:00:23.634 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:23.646 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:23.645+0000 7f2ab7708200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:00:23.652 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:00:23.652 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:23.652 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:23.669 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:00:23.670 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:00:23.670 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:23.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:00:23.709 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:00:23.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.831+0000 7f72f448b700 1 Processor -- start 2023-10-16T11:00:23.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.831+0000 7f72f448b700 1 -- start start 2023-10-16T11:00:23.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.832+0000 7f72f448b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72ec072360 0x7f72ec072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.832+0000 7f72f448b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec06f850 con 0x7f72ec06f470 2023-10-16T11:00:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.832+0000 7f72f448b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec0b4260 con 0x7f72ec06a8d0 2023-10-16T11:00:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.832+0000 7f72f448b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec072d50 con 0x7f72ec072360 2023-10-16T11:00:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.832+0000 7f72f1a26700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72ec072360 0x7f72ec072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f2227700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f72ec06f470 0x7f72ec06e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60874/0 (socket says 172.21.15.158:60874) 2023-10-16T11:00:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f2227700 1 -- 172.21.15.158:0/3687149942 learned_addr learned my addr 172.21.15.158:0/3687149942 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1a26700 1 -- 172.21.15.158:0/3687149942 >> v1:172.21.15.172:6789/0 conn(0x7f72ec06a8d0 legacy=0x7f72ec06ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1a26700 1 -- 172.21.15.158:0/3687149942 >> v1:172.21.15.158:6789/0 conn(0x7f72ec06f470 legacy=0x7f72ec06e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1225700 1 -- 172.21.15.158:0/3687149942 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701414930 0 0) 0x7f72ec06f850 con 0x7f72ec06f470 2023-10-16T11:00:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1a26700 1 -- 172.21.15.158:0/3687149942 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0b4540 con 0x7f72ec072360 2023-10-16T11:00:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1a26700 1 --2- 172.21.15.158:0/3687149942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec072360 0x7f72ec072810 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009840 tx=0x7f72dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6890912834132110 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1225700 1 -- 172.21.15.158:0/3687149942 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc011830 con 0x7f72ec072360 2023-10-16T11:00:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1225700 1 -- 172.21.15.158:0/3687149942 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc011990 con 0x7f72ec072360 2023-10-16T11:00:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.833+0000 7f72f1225700 1 -- 172.21.15.158:0/3687149942 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc011d20 con 0x7f72ec072360 2023-10-16T11:00:23.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.834+0000 7f72f448b700 1 -- 172.21.15.158:0/3687149942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec072360 msgr2=0x7f72ec072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.834+0000 7f72f448b700 1 --2- 172.21.15.158:0/3687149942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec072360 0x7f72ec072810 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009840 tx=0x7f72dc009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:23.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.834+0000 7f72f448b700 1 -- 172.21.15.158:0/3687149942 shutdown_connections 2023-10-16T11:00:23.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.834+0000 7f72f448b700 1 --2- 172.21.15.158:0/3687149942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec072360 0x7f72ec072810 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.834+0000 7f72f448b700 1 -- 172.21.15.158:0/3687149942 >> 172.21.15.158:0/3687149942 conn(0x7f72ec0ccda0 msgr2=0x7f72ec0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:23.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.834+0000 7f72f448b700 1 -- 172.21.15.158:0/3687149942 shutdown_connections 2023-10-16T11:00:23.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.834+0000 7f72f448b700 1 -- 172.21.15.158:0/3687149942 wait complete. 2023-10-16T11:00:23.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.835+0000 7f72f448b700 1 Processor -- start 2023-10-16T11:00:23.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.835+0000 7f72f448b700 1 -- start start 2023-10-16T11:00:23.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f448b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72ec06a8d0 0x7f72ec0cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f448b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec06f470 0x7f72ec0cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f448b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72ec072360 0x7f72ec150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f448b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0cfb10 con 0x7f72ec06f470 2023-10-16T11:00:23.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f448b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0cede0 con 0x7f72ec072360 2023-10-16T11:00:23.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f448b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0ce540 con 0x7f72ec06a8d0 2023-10-16T11:00:23.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec06f470 0x7f72ec0cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec06f470 0x7f72ec0cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60706/0 (socket says 172.21.15.158:60706) 2023-10-16T11:00:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 -- 172.21.15.158:0/1753684375 learned_addr learned my addr 172.21.15.158:0/1753684375 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f2227700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72ec06a8d0 0x7f72ec0cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f2a28700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72ec072360 0x7f72ec150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 -- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72ec06a8d0 msgr2=0x7f72ec0cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72ec06a8d0 0x7f72ec0cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 -- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72ec072360 msgr2=0x7f72ec150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:23.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72ec072360 0x7f72ec150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:23.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f1a26700 1 -- 172.21.15.158:0/1753684375 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc0094a0 con 0x7f72ec06f470 2023-10-16T11:00:23.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.836+0000 7f72f2a28700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72ec072360 0x7f72ec150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:23.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.837+0000 7f72f1a26700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec06f470 0x7f72ec0cff90 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f72dc014040 tx=0x7f72dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:23.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.837+0000 7f72e2ffd700 1 -- 172.21.15.158:0/1753684375 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc01a850 con 0x7f72ec06f470 2023-10-16T11:00:23.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.837+0000 7f72e2ffd700 1 -- 172.21.15.158:0/1753684375 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc01a9b0 con 0x7f72ec06f470 2023-10-16T11:00:23.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.837+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec0ce820 con 0x7f72ec06f470 2023-10-16T11:00:23.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.837+0000 7f72e2ffd700 1 -- 172.21.15.158:0/1753684375 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc01ad20 con 0x7f72ec06f470 2023-10-16T11:00:23.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.837+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec0c7ee0 con 0x7f72ec06f470 2023-10-16T11:00:23.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.838+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec0c7d40 con 0x7f72ec06f470 2023-10-16T11:00:23.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.838+0000 7f72e2ffd700 1 -- 172.21.15.158:0/1753684375 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f72dc022070 con 0x7f72ec06f470 2023-10-16T11:00:23.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.839+0000 7f72e2ffd700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 0x7f72d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:23.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.839+0000 7f72f2227700 1 -- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 msgr2=0x7f72d006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:23.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.839+0000 7f72e2ffd700 1 -- 172.21.15.158:0/1753684375 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f72dc08b000 con 0x7f72ec06f470 2023-10-16T11:00:23.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.839+0000 7f72f2227700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 0x7f72d006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:23.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:23.843+0000 7f72e2ffd700 1 -- 172.21.15.158:0/1753684375 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72dc055c50 con 0x7f72ec06f470 2023-10-16T11:00:24.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.040+0000 7f72f2227700 1 -- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 msgr2=0x7f72d006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:24.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.040+0000 7f72f2227700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 0x7f72d006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:24.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.084+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72ec071e20 con 0x7f72ec06f470 2023-10-16T11:00:24.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.085+0000 7f72e2ffd700 1 -- 172.21.15.158:0/1753684375 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f72ec071e20 con 0x7f72ec06f470 2023-10-16T11:00:24.086 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:24.086 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8422,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1604199546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1604199546},{"type":"v1","addr":"172.21.15.172:6834","nonce":1604199546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/95492242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":95492242},{"type":"v1","addr":"172.21.15.158:6836","nonce":95492242}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9393,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112424210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112424210},{"type":"v1","addr":"172.21.15.158:6837","nonce":112424210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:04:04.049571+0000","modified":"2023-10-16T10:04:07.086241+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37352":{"gid":37352,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/253283745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":253283745},{"type":"v1","addr":"172.21.15.172:6835","nonce":253283745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T11:00:24.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 msgr2=0x7f72d006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:24.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 0x7f72d006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec06f470 msgr2=0x7f72ec0cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec06f470 0x7f72ec0cff90 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f72dc014040 tx=0x7f72dc004750 comp rx=0 tx=0).stop 2023-10-16T11:00:24.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 shutdown_connections 2023-10-16T11:00:24.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f72d006b020 0x7f72d006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72ec072360 0x7f72ec150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72ec06a8d0 0x7f72ec0cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 --2- 172.21.15.158:0/1753684375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72ec06f470 0x7f72ec0cff90 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 >> 172.21.15.158:0/1753684375 conn(0x7f72ec0ccda0 msgr2=0x7f72ec06adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:24.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 shutdown_connections 2023-10-16T11:00:24.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.087+0000 7f72f448b700 1 -- 172.21.15.158:0/1753684375 wait complete. 2023-10-16T11:00:24.092 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 77 2023-10-16T11:00:24.101 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:24.161 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:24.161+0000 7fe0ee3c7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:24.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.226+0000 7f9c15cb5700 1 Processor -- start 2023-10-16T11:00:24.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.226+0000 7f9c15cb5700 1 -- start start 2023-10-16T11:00:24.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c15cb5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c100d0bd0 0x7f9c100d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c15cb5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c100ce4d0 con 0x7f9c1015daa0 2023-10-16T11:00:24.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c15cb5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1006c1b0 con 0x7f9c100d3570 2023-10-16T11:00:24.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c15cb5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1006c490 con 0x7f9c100d0bd0 2023-10-16T11:00:24.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c100d0bd0 0x7f9c100d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9c1015daa0 0x7f9c1006b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60890/0 (socket says 172.21.15.158:60890) 2023-10-16T11:00:24.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0f7fe700 1 -- 172.21.15.158:0/1354145701 learned_addr learned my addr 172.21.15.158:0/1354145701 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:24.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0effd700 1 -- 172.21.15.158:0/1354145701 >> v1:172.21.15.172:6789/0 conn(0x7f9c100d3570 legacy=0x7f9c100d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0effd700 1 -- 172.21.15.158:0/1354145701 >> v1:172.21.15.158:6789/0 conn(0x7f9c1015daa0 legacy=0x7f9c1006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0effd700 1 -- 172.21.15.158:0/1354145701 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1006c770 con 0x7f9c100d0bd0 2023-10-16T11:00:24.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0effd700 1 --2- 172.21.15.158:0/1354145701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c100d1510 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009840 tx=0x7f9bf8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f570dda9b542c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:24.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.227+0000 7f9c0e7fc700 1 -- 172.21.15.158:0/1354145701 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8004230 con 0x7f9c100d0bd0 2023-10-16T11:00:24.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c0e7fc700 1 -- 172.21.15.158:0/1354145701 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf8004390 con 0x7f9c100d0bd0 2023-10-16T11:00:24.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c0e7fc700 1 -- 172.21.15.158:0/1354145701 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf80036a0 con 0x7f9c100d0bd0 2023-10-16T11:00:24.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c15cb5700 1 -- 172.21.15.158:0/1354145701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 msgr2=0x7f9c100d1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/1354145701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c100d1510 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009840 tx=0x7f9bf8009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:24.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c15cb5700 1 -- 172.21.15.158:0/1354145701 shutdown_connections 2023-10-16T11:00:24.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/1354145701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c100d1510 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c15cb5700 1 -- 172.21.15.158:0/1354145701 >> 172.21.15.158:0/1354145701 conn(0x7f9c1008d040 msgr2=0x7f9c1009c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:24.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c15cb5700 1 -- 172.21.15.158:0/1354145701 shutdown_connections 2023-10-16T11:00:24.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.228+0000 7f9c15cb5700 1 -- 172.21.15.158:0/1354145701 wait complete. 2023-10-16T11:00:24.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.229+0000 7f9c15cb5700 1 Processor -- start 2023-10-16T11:00:24.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.229+0000 7f9c15cb5700 1 -- start start 2023-10-16T11:00:24.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c15cb5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c1015f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c15cb5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c100d3570 0x7f9c101f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c15cb5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c1015daa0 0x7f9c10152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c15cb5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c101f7970 con 0x7f9c100d0bd0 2023-10-16T11:00:24.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c15cb5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c10160430 con 0x7f9c1015daa0 2023-10-16T11:00:24.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c15cb5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c10153310 con 0x7f9c100d3570 2023-10-16T11:00:24.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c0f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c1015f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c0f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c1015f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60720/0 (socket says 172.21.15.158:60720) 2023-10-16T11:00:24.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c0f7fe700 1 -- 172.21.15.158:0/936964644 learned_addr learned my addr 172.21.15.158:0/936964644 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:24.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c0effd700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c100d3570 0x7f9c101f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.230+0000 7f9c0ffff700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c1015daa0 0x7f9c10152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c0f7fe700 1 -- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c100d3570 msgr2=0x7f9c101f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c0f7fe700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c100d3570 0x7f9c101f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c0f7fe700 1 -- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c1015daa0 msgr2=0x7f9c10152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c0f7fe700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c1015daa0 0x7f9c10152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c0f7fe700 1 -- 172.21.15.158:0/936964644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80094a0 con 0x7f9c100d0bd0 2023-10-16T11:00:24.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c0f7fe700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c1015f1f0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c520 tx=0x7f9c0800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:24.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c07fff700 1 -- 172.21.15.158:0/936964644 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080074e0 con 0x7f9c100d0bd0 2023-10-16T11:00:24.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c07fff700 1 -- 172.21.15.158:0/936964644 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08007660 con 0x7f9c100d0bd0 2023-10-16T11:00:24.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c07fff700 1 -- 172.21.15.158:0/936964644 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08005a70 con 0x7f9c100d0bd0 2023-10-16T11:00:24.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c100cd3a0 con 0x7f9c100d0bd0 2023-10-16T11:00:24.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.231+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1015fb00 con 0x7f9c100d0bd0 2023-10-16T11:00:24.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.232+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c100681b0 con 0x7f9c100d0bd0 2023-10-16T11:00:24.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.233+0000 7f9c07fff700 1 -- 172.21.15.158:0/936964644 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f9c08005bd0 con 0x7f9c100d0bd0 2023-10-16T11:00:24.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.234+0000 7f9c07fff700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 0x7f9bf006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.234+0000 7f9c0effd700 1 -- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 msgr2=0x7f9bf006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:24.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.234+0000 7f9c07fff700 1 -- 172.21.15.158:0/936964644 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f9c080891a0 con 0x7f9c100d0bd0 2023-10-16T11:00:24.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.234+0000 7f9c0effd700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 0x7f9bf006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:24.243 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:24.238+0000 7fe0ee3c7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:00:24.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.238+0000 7f9c07fff700 1 -- 172.21.15.158:0/936964644 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c08053b60 con 0x7f9c100d0bd0 2023-10-16T11:00:24.317 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:24.316+0000 7f2ab7708200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:24.385 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:24.385+0000 7fe0ee3c7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:24.399 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:24.398+0000 7f2ab7708200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:00:24.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.435+0000 7f9c0effd700 1 -- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 msgr2=0x7f9bf006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:24.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.435+0000 7f9c0effd700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 0x7f9bf006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:24.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.441+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c100d4450 con 0x7f9c100d0bd0 2023-10-16T11:00:24.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.442+0000 7f9c07fff700 1 -- 172.21.15.158:0/936964644 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+18109 (secure 0 0 0) 0x7f9c08057180 con 0x7f9c100d0bd0 2023-10-16T11:00:24.443 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:24.444 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":158,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:21.373390+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:04:02.482303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T10:04:03.019780+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2640446942":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3643659736":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1187395764":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/2828352389":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6833/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6832/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3333502274":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1458403109":"2023-10-17T10:04:17.957825+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":50,"length":7},{"begin":106,"length":7}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:24.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 msgr2=0x7f9bf006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:24.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 0x7f9bf006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 msgr2=0x7f9c1015f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c1015f1f0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c520 tx=0x7f9c0800c830 comp rx=0 tx=0).stop 2023-10-16T11:00:24.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 shutdown_connections 2023-10-16T11:00:24.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f9bf006b020 0x7f9bf006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c1015daa0 0x7f9c10152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c100d0bd0 0x7f9c1015f1f0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 --2- 172.21.15.158:0/936964644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c100d3570 0x7f9c101f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 >> 172.21.15.158:0/936964644 conn(0x7f9c1008d040 msgr2=0x7f9c100b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:24.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 shutdown_connections 2023-10-16T11:00:24.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.444+0000 7f9c15cb5700 1 -- 172.21.15.158:0/936964644 wait complete. 2023-10-16T11:00:24.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T10:04:02.482303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T10:04:03.019780+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 158, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:00:24.460 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:00:24.555 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:24.554+0000 7f2ab7708200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:24.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.586+0000 7f7545c90700 1 Processor -- start 2023-10-16T11:00:24.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.586+0000 7f7545c90700 1 -- start start 2023-10-16T11:00:24.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.586+0000 7f7545c90700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f75400ce9c0 0x7f75401686b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.586+0000 7f7545c90700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754015ec50 con 0x7f754006af20 2023-10-16T11:00:24.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.586+0000 7f7545c90700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7540161450 con 0x7f754006ab40 2023-10-16T11:00:24.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.586+0000 7f7545c90700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7540161730 con 0x7f75400ce9c0 2023-10-16T11:00:24.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.586+0000 7f753ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f75400ce9c0 0x7f75401686b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f754006af20 0x7f754015d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60936/0 (socket says 172.21.15.158:60936) 2023-10-16T11:00:24.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753effd700 1 -- 172.21.15.158:0/1793699970 learned_addr learned my addr 172.21.15.158:0/1793699970 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:24.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753ffff700 1 -- 172.21.15.158:0/1793699970 >> v1:172.21.15.172:6789/0 conn(0x7f754006ab40 legacy=0x7f75400ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753ffff700 1 -- 172.21.15.158:0/1793699970 >> v1:172.21.15.158:6789/0 conn(0x7f754006af20 legacy=0x7f754015d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753e7fc700 1 -- 172.21.15.158:0/1793699970 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058275439 0 0) 0x7f7528001110 con 0x7f754006af20 2023-10-16T11:00:24.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753ffff700 1 -- 172.21.15.158:0/1793699970 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7540161a10 con 0x7f75400ce9c0 2023-10-16T11:00:24.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753ffff700 1 --2- 172.21.15.158:0/1793699970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75400ce9c0 0x7f75401686b0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f75400cfec0 tx=0x7f753000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eccf50d48a6263e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:24.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753e7fc700 1 -- 172.21.15.158:0/1793699970 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7530016070 con 0x7f75400ce9c0 2023-10-16T11:00:24.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753e7fc700 1 -- 172.21.15.158:0/1793699970 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753000bbd0 con 0x7f75400ce9c0 2023-10-16T11:00:24.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.587+0000 7f753e7fc700 1 -- 172.21.15.158:0/1793699970 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753000e070 con 0x7f75400ce9c0 2023-10-16T11:00:24.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.588+0000 7f7545c90700 1 -- 172.21.15.158:0/1793699970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75400ce9c0 msgr2=0x7f75401686b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.588+0000 7f7545c90700 1 --2- 172.21.15.158:0/1793699970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75400ce9c0 0x7f75401686b0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f75400cfec0 tx=0x7f753000b330 comp rx=0 tx=0).stop 2023-10-16T11:00:24.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.588+0000 7f7545c90700 1 -- 172.21.15.158:0/1793699970 shutdown_connections 2023-10-16T11:00:24.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.588+0000 7f7545c90700 1 --2- 172.21.15.158:0/1793699970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75400ce9c0 0x7f75401686b0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.588+0000 7f7545c90700 1 -- 172.21.15.158:0/1793699970 >> 172.21.15.158:0/1793699970 conn(0x7f75400b1f00 msgr2=0x7f75400b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:24.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.588+0000 7f7545c90700 1 -- 172.21.15.158:0/1793699970 shutdown_connections 2023-10-16T11:00:24.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.588+0000 7f7545c90700 1 -- 172.21.15.158:0/1793699970 wait complete. 2023-10-16T11:00:24.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.589+0000 7f7545c90700 1 Processor -- start 2023-10-16T11:00:24.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.589+0000 7f7545c90700 1 -- start start 2023-10-16T11:00:24.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f7545c90700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f754006ab40 0x7f75400b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f7545c90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f754006af20 0x7f75401f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f7545c90700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75400ce9c0 0x7f75400ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f7545c90700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75401f3a10 con 0x7f754006af20 2023-10-16T11:00:24.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f7545c90700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75401f3cf0 con 0x7f754006ab40 2023-10-16T11:00:24.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f7545c90700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75401f3fd0 con 0x7f75400ce9c0 2023-10-16T11:00:24.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f753effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f754006af20 0x7f75401f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f753effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f754006af20 0x7f75401f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60766/0 (socket says 172.21.15.158:60766) 2023-10-16T11:00:24.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f753effd700 1 -- 172.21.15.158:0/423093402 learned_addr learned my addr 172.21.15.158:0/423093402 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:24.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f753f7fe700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f754006ab40 0x7f75400b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.590+0000 7f753ffff700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75400ce9c0 0x7f75400ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f753effd700 1 -- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75400ce9c0 msgr2=0x7f75400ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f753effd700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75400ce9c0 0x7f75400ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f753effd700 1 -- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f754006ab40 msgr2=0x7f75400b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f753effd700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f754006ab40 0x7f75400b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f753effd700 1 -- 172.21.15.158:0/423093402 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7530009a70 con 0x7f754006af20 2023-10-16T11:00:24.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f753f7fe700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f754006ab40 0x7f75400b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:24.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f753effd700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f754006af20 0x7f75401f30c0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f752800c450 tx=0x7f752800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:24.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f7537fff700 1 -- 172.21.15.158:0/423093402 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528004c80 con 0x7f754006af20 2023-10-16T11:00:24.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f7537fff700 1 -- 172.21.15.158:0/423093402 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7528004e00 con 0x7f754006af20 2023-10-16T11:00:24.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75400ad980 con 0x7f754006af20 2023-10-16T11:00:24.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f7537fff700 1 -- 172.21.15.158:0/423093402 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528007310 con 0x7f754006af20 2023-10-16T11:00:24.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.591+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75400b57d0 con 0x7f754006af20 2023-10-16T11:00:24.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.592+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75400adea0 con 0x7f754006af20 2023-10-16T11:00:24.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.592+0000 7f7537fff700 1 -- 172.21.15.158:0/423093402 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7528007470 con 0x7f754006af20 2023-10-16T11:00:24.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.593+0000 7f7537fff700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 0x7f752006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.593+0000 7f753f7fe700 1 -- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 msgr2=0x7f752006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:24.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.593+0000 7f7537fff700 1 -- 172.21.15.158:0/423093402 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f75280895f0 con 0x7f754006af20 2023-10-16T11:00:24.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.593+0000 7f753f7fe700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 0x7f752006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:24.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.597+0000 7f7537fff700 1 -- 172.21.15.158:0/423093402 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75400adea0 con 0x7f754006af20 2023-10-16T11:00:24.613 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:24.613+0000 7fe0ee3c7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:00:24.687 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:24.687+0000 7fe0ee3c7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:00:24.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.793+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f754015ec50 con 0x7f754006af20 2023-10-16T11:00:24.795 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:24.793+0000 7f2ab7708200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:00:24.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.794+0000 7f7537fff700 1 -- 172.21.15.158:0/423093402 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7f754015ec50 con 0x7f754006af20 2023-10-16T11:00:24.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.794+0000 7f753f7fe700 1 -- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 msgr2=0x7f752006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:24.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.794+0000 7f753f7fe700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 0x7f752006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:24.796 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:00:24.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 msgr2=0x7f752006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:24.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 0x7f752006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f754006af20 msgr2=0x7f75401f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f754006af20 0x7f75401f30c0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f752800c450 tx=0x7f752800c760 comp rx=0 tx=0).stop 2023-10-16T11:00:24.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 shutdown_connections 2023-10-16T11:00:24.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f752006b020 0x7f752006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75400ce9c0 0x7f75400ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f754006ab40 0x7f75400b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 --2- 172.21.15.158:0/423093402 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f754006af20 0x7f75401f30c0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 >> 172.21.15.158:0/423093402 conn(0x7f75400b1f00 msgr2=0x7f754015edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:24.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 shutdown_connections 2023-10-16T11:00:24.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.796+0000 7f7545c90700 1 -- 172.21.15.158:0/423093402 wait complete. 2023-10-16T11:00:24.810 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:00:24.876 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:24.875+0000 7f2ab7708200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:00:24.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.936+0000 7fa55cf80700 1 Processor -- start 2023-10-16T11:00:24.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.936+0000 7fa55cf80700 1 -- start start 2023-10-16T11:00:24.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.937+0000 7fa55cf80700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa558160d30 0x7fa5580b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.937+0000 7fa55cf80700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa5580b7590 0x7fa5580d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.937+0000 7fa55cf80700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5580b7a20 con 0x7fa5580d0530 2023-10-16T11:00:24.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.937+0000 7fa55cf80700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5580b8910 con 0x7fa5580b7590 2023-10-16T11:00:24.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.937+0000 7fa55cf80700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580d0ef0 con 0x7fa558160d30 2023-10-16T11:00:24.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa5580b7590 0x7fa5580d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa5580b7590 0x7fa5580d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60778/0 (socket says 172.21.15.158:60778) 2023-10-16T11:00:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 -- 172.21.15.158:0/1202864343 learned_addr learned my addr 172.21.15.158:0/1202864343 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa55659c700 1 --2- 172.21.15.158:0/1202864343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa558160d30 0x7fa5580b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 -- 172.21.15.158:0/1202864343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa558160d30 msgr2=0x7fa5580b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 --2- 172.21.15.158:0/1202864343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa558160d30 0x7fa5580b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 -- 172.21.15.158:0/1202864343 >> v1:172.21.15.172:6789/0 conn(0x7fa5580d0530 legacy=0x7fa5580d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 -- 172.21.15.158:0/1202864343 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580b7c90 con 0x7fa5580b7590 2023-10-16T11:00:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa556d9d700 1 --2- 172.21.15.158:0/1202864343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580b7590 0x7fa5580d09b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fa550009e50 tx=0x7fa55000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0225e7500635f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa55559a700 1 -- 172.21.15.158:0/1202864343 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55000e070 con 0x7fa5580b7590 2023-10-16T11:00:24.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa55559a700 1 -- 172.21.15.158:0/1202864343 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55000b910 con 0x7fa5580b7590 2023-10-16T11:00:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.938+0000 7fa55559a700 1 -- 172.21.15.158:0/1202864343 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55000bcc0 con 0x7fa5580b7590 2023-10-16T11:00:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 -- 172.21.15.158:0/1202864343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580b7590 msgr2=0x7fa5580d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 --2- 172.21.15.158:0/1202864343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580b7590 0x7fa5580d09b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fa550009e50 tx=0x7fa55000b330 comp rx=0 tx=0).stop 2023-10-16T11:00:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 -- 172.21.15.158:0/1202864343 shutdown_connections 2023-10-16T11:00:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 --2- 172.21.15.158:0/1202864343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580b7590 0x7fa5580d09b0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 --2- 172.21.15.158:0/1202864343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa558160d30 0x7fa5580b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 -- 172.21.15.158:0/1202864343 >> 172.21.15.158:0/1202864343 conn(0x7fa5580669f0 msgr2=0x7fa558066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 -- 172.21.15.158:0/1202864343 shutdown_connections 2023-10-16T11:00:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.939+0000 7fa55cf80700 1 -- 172.21.15.158:0/1202864343 wait complete. 2023-10-16T11:00:24.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 Processor -- start 2023-10-16T11:00:24.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 -- start start 2023-10-16T11:00:24.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5580b7590 0x7fa558151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580d0530 0x7fa5580cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558160d30 0x7fa55815db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580c9a40 con 0x7fa5580d0530 2023-10-16T11:00:24.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55815cee0 con 0x7fa558160d30 2023-10-16T11:00:24.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa55cf80700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580c8b50 con 0x7fa5580b7590 2023-10-16T11:00:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa555d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580d0530 0x7fa5580cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa555d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580d0530 0x7fa5580cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60782/0 (socket says 172.21.15.158:60782) 2023-10-16T11:00:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.941+0000 7fa555d9b700 1 -- 172.21.15.158:0/3589315667 learned_addr learned my addr 172.21.15.158:0/3589315667 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa55659c700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5580b7590 0x7fa558151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa556d9d700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558160d30 0x7fa55815db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa555d9b700 1 -- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5580b7590 msgr2=0x7fa558151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa555d9b700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5580b7590 0x7fa558151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa555d9b700 1 -- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558160d30 msgr2=0x7fa55815db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa555d9b700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558160d30 0x7fa55815db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa555d9b700 1 -- 172.21.15.158:0/3589315667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa550009b30 con 0x7fa5580d0530 2023-10-16T11:00:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa555d9b700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580d0530 0x7fa5580cae70 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00c4e0 tx=0x7fa54c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa546ffd700 1 -- 172.21.15.158:0/3589315667 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0052d0 con 0x7fa5580d0530 2023-10-16T11:00:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa546ffd700 1 -- 172.21.15.158:0/3589315667 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54c005430 con 0x7fa5580d0530 2023-10-16T11:00:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa546ffd700 1 -- 172.21.15.158:0/3589315667 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0038c0 con 0x7fa5580d0530 2023-10-16T11:00:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5580c8e30 con 0x7fa5580d0530 2023-10-16T11:00:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.942+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5580c93f0 con 0x7fa5580d0530 2023-10-16T11:00:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.943+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5580681b0 con 0x7fa5580d0530 2023-10-16T11:00:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.944+0000 7fa546ffd700 1 -- 172.21.15.158:0/3589315667 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa54c003a20 con 0x7fa5580d0530 2023-10-16T11:00:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.944+0000 7fa546ffd700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 0x7fa53806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.944+0000 7fa55659c700 1 -- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 msgr2=0x7fa53806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.944+0000 7fa546ffd700 1 -- 172.21.15.158:0/3589315667 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fa54c088f50 con 0x7fa5580d0530 2023-10-16T11:00:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.944+0000 7fa55659c700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 0x7fa53806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:24.954 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:24.952+0000 7fe0ee3c7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:00:24.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:24.949+0000 7fa546ffd700 1 -- 172.21.15.158:0/3589315667 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa54c053a30 con 0x7fa5580d0530 2023-10-16T11:00:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.145+0000 7fa55659c700 1 -- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 msgr2=0x7fa53806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.145+0000 7fa55659c700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 0x7fa53806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:25.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.147+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa558066dd0 con 0x7fa5580d0530 2023-10-16T11:00:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.148+0000 7fa546ffd700 1 -- 172.21.15.158:0/3589315667 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v158) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa54c017020 con 0x7fa5580d0530 2023-10-16T11:00:25.149 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:00:25.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 msgr2=0x7fa53806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:25.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 0x7fa53806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580d0530 msgr2=0x7fa5580cae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580d0530 0x7fa5580cae70 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00c4e0 tx=0x7fa54c00c7f0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 shutdown_connections 2023-10-16T11:00:25.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fa53806b020 0x7fa53806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558160d30 0x7fa55815db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5580b7590 0x7fa558151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 --2- 172.21.15.158:0/3589315667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5580d0530 0x7fa5580cae70 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 >> 172.21.15.158:0/3589315667 conn(0x7fa5580669f0 msgr2=0x7fa55815b520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:25.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 shutdown_connections 2023-10-16T11:00:25.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.150+0000 7fa55cf80700 1 -- 172.21.15.158:0/3589315667 wait complete. 2023-10-16T11:00:25.156 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:25.155+0000 7f2ab7708200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:00:25.164 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:00:25.174 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:25.174+0000 7fe0ee3c7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:00:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.290+0000 7ff4fa0fe700 1 Processor -- start 2023-10-16T11:00:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.290+0000 7ff4fa0fe700 1 -- start start 2023-10-16T11:00:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.290+0000 7ff4fa0fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff4f415deb0 0x7ff4f40ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.290+0000 7ff4fa0fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f40ccc00 0x7ff4f40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.290+0000 7ff4fa0fe700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f40d0ae0 con 0x7ff4f415dad0 2023-10-16T11:00:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.290+0000 7ff4fa0fe700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f40d12d0 con 0x7ff4f415deb0 2023-10-16T11:00:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.290+0000 7ff4fa0fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f40cbdf0 con 0x7ff4f40ccc00 2023-10-16T11:00:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f40ccc00 0x7ff4f40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff4f415deb0 0x7ff4f40ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f40ccc00 0x7ff4f40cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:43396/0 (socket says 172.21.15.158:43396) 2023-10-16T11:00:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff4f415deb0 0x7ff4f40ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:33740/0 (socket says 172.21.15.158:33740) 2023-10-16T11:00:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f37fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff4f415dad0 0x7ff4f40cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:36638/0 (socket says 172.21.15.158:36638) 2023-10-16T11:00:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f3fff700 1 -- 172.21.15.158:0/4002828072 learned_addr learned my addr 172.21.15.158:0/4002828072 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f2ffd700 1 -- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f40ccc00 msgr2=0x7ff4f40cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f2ffd700 1 --2- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f40ccc00 0x7ff4f40cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f2ffd700 1 -- 172.21.15.158:0/4002828072 >> v1:172.21.15.172:6789/0 conn(0x7ff4f415dad0 legacy=0x7ff4f40cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f2ffd700 1 -- 172.21.15.158:0/4002828072 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f40cc280 con 0x7ff4f415deb0 2023-10-16T11:00:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f27fc700 1 -- 172.21.15.158:0/4002828072 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12606834 0 0) 0x7ff4f40d0ae0 con 0x7ff4f415dad0 2023-10-16T11:00:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f3fff700 1 --2- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f40ccc00 0x7ff4f40cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f2ffd700 1 --2- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f415deb0 0x7ff4f40ce6b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009840 tx=0x7ff4e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0019cb393aade4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f27fc700 1 -- 172.21.15.158:0/4002828072 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8011830 con 0x7ff4f415deb0 2023-10-16T11:00:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f27fc700 1 -- 172.21.15.158:0/4002828072 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e8011990 con 0x7ff4f415deb0 2023-10-16T11:00:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.291+0000 7ff4f27fc700 1 -- 172.21.15.158:0/4002828072 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8011d20 con 0x7ff4f415deb0 2023-10-16T11:00:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f415deb0 msgr2=0x7ff4f40ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f415deb0 0x7ff4f40ce6b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009840 tx=0x7ff4e8009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/4002828072 shutdown_connections 2023-10-16T11:00:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f40ccc00 0x7ff4f40cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/4002828072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f415deb0 0x7ff4f40ce6b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/4002828072 >> 172.21.15.158:0/4002828072 conn(0x7ff4f4099a10 msgr2=0x7ff4f40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:25.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/4002828072 shutdown_connections 2023-10-16T11:00:25.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.292+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/4002828072 wait complete. 2023-10-16T11:00:25.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.293+0000 7ff4fa0fe700 1 Processor -- start 2023-10-16T11:00:25.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.293+0000 7ff4fa0fe700 1 -- start start 2023-10-16T11:00:25.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4fa0fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f40ccc00 0x7ff4f4153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4fa0fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4f415dad0 0x7ff4f4152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4fa0fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f415deb0 0x7ff4f4155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4fa0fe700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f40c74a0 con 0x7ff4f415dad0 2023-10-16T11:00:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4fa0fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f40c7740 con 0x7ff4f40ccc00 2023-10-16T11:00:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4fa0fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f40c7a20 con 0x7ff4f415deb0 2023-10-16T11:00:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4f2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4f415dad0 0x7ff4f4152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4f2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4f415dad0 0x7ff4f4152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60798/0 (socket says 172.21.15.158:60798) 2023-10-16T11:00:25.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4f2ffd700 1 -- 172.21.15.158:0/2029798348 learned_addr learned my addr 172.21.15.158:0/2029798348 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:25.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4f3fff700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f415deb0 0x7ff4f4155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.294+0000 7ff4f37fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f40ccc00 0x7ff4f4153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4f2ffd700 1 -- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f415deb0 msgr2=0x7ff4f4155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4f2ffd700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f415deb0 0x7ff4f4155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4f2ffd700 1 -- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f40ccc00 msgr2=0x7ff4f4153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4f2ffd700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f40ccc00 0x7ff4f4153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4f2ffd700 1 -- 172.21.15.158:0/2029798348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec009b30 con 0x7ff4f415dad0 2023-10-16T11:00:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4f2ffd700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4f415dad0 0x7ff4f4152160 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8014040 tx=0x7ff4e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4dbfff700 1 -- 172.21.15.158:0/2029798348 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8004550 con 0x7ff4f415dad0 2023-10-16T11:00:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4dbfff700 1 -- 172.21.15.158:0/2029798348 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e801f070 con 0x7ff4f415dad0 2023-10-16T11:00:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4dbfff700 1 -- 172.21.15.158:0/2029798348 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e80038c0 con 0x7ff4f415dad0 2023-10-16T11:00:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e80094a0 con 0x7ff4f415dad0 2023-10-16T11:00:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.295+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f40c7fe0 con 0x7ff4f415dad0 2023-10-16T11:00:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.296+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f40681b0 con 0x7ff4f415dad0 2023-10-16T11:00:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.296+0000 7ff4dbfff700 1 -- 172.21.15.158:0/2029798348 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff4e8003a20 con 0x7ff4f415dad0 2023-10-16T11:00:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.297+0000 7ff4dbfff700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 0x7ff4e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.297+0000 7ff4f37fe700 1 -- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 msgr2=0x7ff4e006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.297+0000 7ff4dbfff700 1 -- 172.21.15.158:0/2029798348 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6913+0+0 (secure 0 0 0) 0x7ff4e8021070 con 0x7ff4f415dad0 2023-10-16T11:00:25.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.297+0000 7ff4f37fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 0x7ff4e006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:25.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.301+0000 7ff4dbfff700 1 -- 172.21.15.158:0/2029798348 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4e8055e70 con 0x7ff4f415dad0 2023-10-16T11:00:25.322 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:25.322+0000 7fe0ee3c7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:00:25.398 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:25.397+0000 7f2ab7708200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:00:25.440 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:25.438+0000 7fe0ee3c7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:00:25.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.498+0000 7ff4f37fe700 1 -- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 msgr2=0x7ff4e006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:25.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.498+0000 7ff4f37fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 0x7ff4e006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:25.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.507+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff4f406e6c0 con 0x7ff4f415dad0 2023-10-16T11:00:25.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.507+0000 7ff4dbfff700 1 -- 172.21.15.158:0/2029798348 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v159) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4e8059490 con 0x7ff4f415dad0 2023-10-16T11:00:25.509 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:00:25.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 msgr2=0x7ff4e006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:25.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 0x7ff4e006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4f415dad0 msgr2=0x7ff4f4152160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4f415dad0 0x7ff4f4152160 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8014040 tx=0x7ff4e8004750 comp rx=0 tx=0).stop 2023-10-16T11:00:25.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 shutdown_connections 2023-10-16T11:00:25.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7ff4e006b020 0x7ff4e006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4f415deb0 0x7ff4f4155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4f40ccc00 0x7ff4f4153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 --2- 172.21.15.158:0/2029798348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4f415dad0 0x7ff4f4152160 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 >> 172.21.15.158:0/2029798348 conn(0x7ff4f4099a10 msgr2=0x7ff4f40ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:25.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 shutdown_connections 2023-10-16T11:00:25.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.510+0000 7ff4fa0fe700 1 -- 172.21.15.158:0/2029798348 wait complete. 2023-10-16T11:00:25.524 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:00:25.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:25.559+0000 7f2ab7708200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:00:25.594 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:25.593+0000 7fe0ee3c7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:00:25.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.647+0000 7f34ce036700 1 Processor -- start 2023-10-16T11:00:25.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.647+0000 7f34ce036700 1 -- start start 2023-10-16T11:00:25.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.647+0000 7f34ce036700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.647+0000 7f34ce036700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f34c80b6e10 0x7f34c81521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.647+0000 7f34ce036700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c815c5d0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.647+0000 7f34ce036700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c815d2c0 con 0x7f34c80b6e10 2023-10-16T11:00:25.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.647+0000 7f34ce036700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c815c8b0 con 0x7f34c80cdee0 2023-10-16T11:00:25.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f34c80ce2c0 0x7f34c80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:60984/0 (socket says 172.21.15.158:60984) 2023-10-16T11:00:25.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c7fff700 1 -- 172.21.15.158:0/3179279988 learned_addr learned my addr 172.21.15.158:0/3179279988 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:25.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c77fe700 1 --2- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c6ffd700 1 --2- 172.21.15.158:0/3179279988 >> v2:172.21.15.172:3300/0 conn(0x7f34c80b6e10 0x7f34c81521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c67fc700 1 -- 172.21.15.158:0/3179279988 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718017852 0 0) 0x7f34c815c5d0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c67fc700 1 -- 172.21.15.158:0/3179279988 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b00035b0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c6ffd700 1 -- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 msgr2=0x7f34c815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:25.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c6ffd700 1 --2- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c6ffd700 1 -- 172.21.15.158:0/3179279988 >> v1:172.21.15.158:6789/0 conn(0x7f34c80ce2c0 legacy=0x7f34c80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c6ffd700 1 -- 172.21.15.158:0/3179279988 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c815cb90 con 0x7f34c80b6e10 2023-10-16T11:00:25.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c77fe700 1 --2- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:25.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c6ffd700 1 --2- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 0x7f34c81521a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009840 tx=0x7f34bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62bcfa98bede271b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:25.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c67fc700 1 -- 172.21.15.158:0/3179279988 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc011830 con 0x7f34c80b6e10 2023-10-16T11:00:25.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c67fc700 1 -- 172.21.15.158:0/3179279988 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc011990 con 0x7f34c80b6e10 2023-10-16T11:00:25.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.648+0000 7f34c67fc700 1 -- 172.21.15.158:0/3179279988 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc011d20 con 0x7f34c80b6e10 2023-10-16T11:00:25.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 -- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 msgr2=0x7f34c81521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 --2- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 0x7f34c81521a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009840 tx=0x7f34bc009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:25.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 -- 172.21.15.158:0/3179279988 shutdown_connections 2023-10-16T11:00:25.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 --2- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 --2- 172.21.15.158:0/3179279988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 0x7f34c81521a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 -- 172.21.15.158:0/3179279988 >> 172.21.15.158:0/3179279988 conn(0x7f34c806a8d0 msgr2=0x7f34c806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:25.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 -- 172.21.15.158:0/3179279988 shutdown_connections 2023-10-16T11:00:25.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.649+0000 7f34ce036700 1 -- 172.21.15.158:0/3179279988 wait complete. 2023-10-16T11:00:25.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.650+0000 7f34ce036700 1 Processor -- start 2023-10-16T11:00:25.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.650+0000 7f34ce036700 1 -- start start 2023-10-16T11:00:25.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34ce036700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 0x7f34c81f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34ce036700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34ce036700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34c80ce2c0 0x7f34c815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34ce036700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81f2f90 con 0x7f34c80ce2c0 2023-10-16T11:00:25.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34ce036700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c8140450 con 0x7f34c80b6e10 2023-10-16T11:00:25.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34ce036700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c80c6eb0 con 0x7f34c80cdee0 2023-10-16T11:00:25.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34c7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34c80ce2c0 0x7f34c815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34c7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34c80ce2c0 0x7f34c815ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60814/0 (socket says 172.21.15.158:60814) 2023-10-16T11:00:25.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34c7fff700 1 -- 172.21.15.158:0/1068371241 learned_addr learned my addr 172.21.15.158:0/1068371241 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34c77fe700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 0x7f34c81f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.651+0000 7f34c6ffd700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34c7fff700 1 -- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 msgr2=0x7f34c815a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34c7fff700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34c7fff700 1 -- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 msgr2=0x7f34c81f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34c7fff700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 0x7f34c81f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34c7fff700 1 -- 172.21.15.158:0/1068371241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b80094a0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34c7fff700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34c80ce2c0 0x7f34c815ac90 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f34c000caa0 tx=0x7f34c000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34affff700 1 -- 172.21.15.158:0/1068371241 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c000a9e0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34affff700 1 -- 172.21.15.158:0/1068371241 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c000ab40 con 0x7f34c80ce2c0 2023-10-16T11:00:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34affff700 1 -- 172.21.15.158:0/1068371241 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0012400 con 0x7f34c80ce2c0 2023-10-16T11:00:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc0094a0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.652+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c80c63d0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.653+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c80681b0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.654+0000 7f34affff700 1 -- 172.21.15.158:0/1068371241 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f34c001b3b0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.654+0000 7f34affff700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 0x7f34b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:25.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.654+0000 7f34c77fe700 1 -- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 msgr2=0x7f34b406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:25.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.654+0000 7f34affff700 1 -- 172.21.15.158:0/1068371241 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f34c0089860 con 0x7f34c80ce2c0 2023-10-16T11:00:25.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.654+0000 7f34c77fe700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 0x7f34b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:25.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.662+0000 7f34affff700 1 -- 172.21.15.158:0/1068371241 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34c0054400 con 0x7f34c80ce2c0 2023-10-16T11:00:25.687 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:25.686+0000 7f2ab7708200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:00:25.855 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:25.854+0000 7f2ab7708200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:00:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.855+0000 7f34c77fe700 1 -- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 msgr2=0x7f34b406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.855+0000 7f34c77fe700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 0x7f34b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:25.861 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:25.860+0000 7fe0ee3c7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:25.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.909+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34c80cc5e0 con 0x7f34c80ce2c0 2023-10-16T11:00:25.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.910+0000 7f34affff700 1 -- 172.21.15.158:0/1068371241 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f34c0057a20 con 0x7f34c80ce2c0 2023-10-16T11:00:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:25.912 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8422,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1604199546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1604199546},{"type":"v1","addr":"172.21.15.172:6834","nonce":1604199546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/95492242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":95492242},{"type":"v1","addr":"172.21.15.158:6836","nonce":95492242}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9393,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112424210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112424210},{"type":"v1","addr":"172.21.15.158:6837","nonce":112424210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:04:04.049571+0000","modified":"2023-10-16T10:04:07.086241+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37352":{"gid":37352,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/253283745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":253283745},{"type":"v1","addr":"172.21.15.172:6835","nonce":253283745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T11:00:25.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 msgr2=0x7f34b406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:25.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 0x7f34b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34c80ce2c0 msgr2=0x7f34c815ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34c80ce2c0 0x7f34c815ac90 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f34c000caa0 tx=0x7f34c000cdb0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 shutdown_connections 2023-10-16T11:00:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f34b406b020 0x7f34b406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34c80ce2c0 0x7f34c815ac90 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34c80b6e10 0x7f34c81f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 --2- 172.21.15.158:0/1068371241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34c80cdee0 0x7f34c815a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:25.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 >> 172.21.15.158:0/1068371241 conn(0x7f34c806a8d0 msgr2=0x7f34c80b5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:25.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 shutdown_connections 2023-10-16T11:00:25.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:25.913+0000 7f34ce036700 1 -- 172.21.15.158:0/1068371241 wait complete. 2023-10-16T11:00:25.917 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 77 2023-10-16T11:00:25.928 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:26.023 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:26.022+0000 7fe0ee3c7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:00:26.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.091+0000 7f72b7c43700 1 Processor -- start 2023-10-16T11:00:26.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.091+0000 7f72b7c43700 1 -- start start 2023-10-16T11:00:26.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b7c43700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72b006ab40 0x7f72b006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b7c43700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f72b006e5d0 0x7f72b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b7c43700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b0071e50 con 0x7f72b006af20 2023-10-16T11:00:26.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b7c43700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b0161270 con 0x7f72b006ab40 2023-10-16T11:00:26.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b7c43700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b0161510 con 0x7f72b006e5d0 2023-10-16T11:00:26.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b59df700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72b006ab40 0x7f72b006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b61e0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f72b006af20 0x7f72b00722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:32768/0 (socket says 172.21.15.158:32768) 2023-10-16T11:00:26.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b59df700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72b006ab40 0x7f72b006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60820/0 (socket says 172.21.15.158:60820) 2023-10-16T11:00:26.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b61e0700 1 -- 172.21.15.158:0/3837107558 learned_addr learned my addr 172.21.15.158:0/3837107558 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:26.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b51de700 1 --2- 172.21.15.158:0/3837107558 >> v2:172.21.15.172:3300/0 conn(0x7f72b006e5d0 0x7f72b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b59df700 1 -- 172.21.15.158:0/3837107558 >> v2:172.21.15.172:3300/0 conn(0x7f72b006e5d0 msgr2=0x7f72b0071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:26.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b59df700 1 --2- 172.21.15.158:0/3837107558 >> v2:172.21.15.172:3300/0 conn(0x7f72b006e5d0 0x7f72b0071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b59df700 1 -- 172.21.15.158:0/3837107558 >> v1:172.21.15.158:6789/0 conn(0x7f72b006af20 legacy=0x7f72b00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b59df700 1 -- 172.21.15.158:0/3837107558 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b01617f0 con 0x7f72b006ab40 2023-10-16T11:00:26.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.092+0000 7f72b59df700 1 --2- 172.21.15.158:0/3837107558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b006d820 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f72b00c3bf0 tx=0x7f72ac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26b4ac9da4c7521 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:26.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b49dd700 1 -- 172.21.15.158:0/3837107558 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac011830 con 0x7f72b006ab40 2023-10-16T11:00:26.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b49dd700 1 -- 172.21.15.158:0/3837107558 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac011990 con 0x7f72b006ab40 2023-10-16T11:00:26.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b49dd700 1 -- 172.21.15.158:0/3837107558 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac011d20 con 0x7f72b006ab40 2023-10-16T11:00:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 -- 172.21.15.158:0/3837107558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 msgr2=0x7f72b006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 --2- 172.21.15.158:0/3837107558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b006d820 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f72b00c3bf0 tx=0x7f72ac009b20 comp rx=0 tx=0).stop 2023-10-16T11:00:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 -- 172.21.15.158:0/3837107558 shutdown_connections 2023-10-16T11:00:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 --2- 172.21.15.158:0/3837107558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b006d820 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 --2- 172.21.15.158:0/3837107558 >> v2:172.21.15.172:3300/0 conn(0x7f72b006e5d0 0x7f72b0071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 -- 172.21.15.158:0/3837107558 >> 172.21.15.158:0/3837107558 conn(0x7f72b00d02b0 msgr2=0x7f72b00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 -- 172.21.15.158:0/3837107558 shutdown_connections 2023-10-16T11:00:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.093+0000 7f72b7c43700 1 -- 172.21.15.158:0/3837107558 wait complete. 2023-10-16T11:00:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.095+0000 7f72b7c43700 1 Processor -- start 2023-10-16T11:00:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.095+0000 7f72b7c43700 1 -- start start 2023-10-16T11:00:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b7c43700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b00b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b7c43700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b006af20 0x7f72b0148010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b7c43700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b006e5d0 0x7f72b00c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b7c43700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b00b0ae0 con 0x7f72b006ab40 2023-10-16T11:00:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b7c43700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0148550 con 0x7f72b006e5d0 2023-10-16T11:00:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b7c43700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b00c9160 con 0x7f72b006af20 2023-10-16T11:00:26.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b59df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b00b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b59df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b00b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60826/0 (socket says 172.21.15.158:60826) 2023-10-16T11:00:26.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b59df700 1 -- 172.21.15.158:0/724789300 learned_addr learned my addr 172.21.15.158:0/724789300 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:26.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b61e0700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b006e5d0 0x7f72b00c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.096+0000 7f72b51de700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b006af20 0x7f72b0148010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b61e0700 1 -- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b006af20 msgr2=0x7f72b0148010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b61e0700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b006af20 0x7f72b0148010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b61e0700 1 -- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 msgr2=0x7f72b00b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b61e0700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b00b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b61e0700 1 -- 172.21.15.158:0/724789300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac0094a0 con 0x7f72b006e5d0 2023-10-16T11:00:26.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b59df700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b00b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:26.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b61e0700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b006e5d0 0x7f72b00c8c20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f72a800caa0 tx=0x7f72a800cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:26.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72a67fc700 1 -- 172.21.15.158:0/724789300 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a800a9a0 con 0x7f72b006e5d0 2023-10-16T11:00:26.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72a67fc700 1 -- 172.21.15.158:0/724789300 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a800ab00 con 0x7f72b006e5d0 2023-10-16T11:00:26.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b00c78b0 con 0x7f72b006e5d0 2023-10-16T11:00:26.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72a67fc700 1 -- 172.21.15.158:0/724789300 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a800ae70 con 0x7f72b006e5d0 2023-10-16T11:00:26.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.097+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b00c7e70 con 0x7f72b006e5d0 2023-10-16T11:00:26.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.098+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b00681b0 con 0x7f72b006e5d0 2023-10-16T11:00:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.098+0000 7f72a67fc700 1 -- 172.21.15.158:0/724789300 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f72a801b3b0 con 0x7f72b006e5d0 2023-10-16T11:00:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.099+0000 7f72a67fc700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 0x7f729406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.099+0000 7f72b59df700 1 -- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 msgr2=0x7f729406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.099+0000 7f72a67fc700 1 -- 172.21.15.158:0/724789300 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f72a8089530 con 0x7f72b006e5d0 2023-10-16T11:00:26.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.099+0000 7f72b59df700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 0x7f729406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:26.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.103+0000 7f72a67fc700 1 -- 172.21.15.158:0/724789300 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72a8054210 con 0x7f72b006e5d0 2023-10-16T11:00:26.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.136+0000 7f2ab7708200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:26.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.300+0000 7f72b59df700 1 -- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 msgr2=0x7f729406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:26.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.300+0000 7f72b59df700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 0x7f729406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:26.305 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.304+0000 7f2ab7708200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:00:26.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.304+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72b00b6230 con 0x7f72b006e5d0 2023-10-16T11:00:26.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.306+0000 7f72a67fc700 1 -- 172.21.15.158:0/724789300 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+18109 (secure 0 0 0) 0x7f72a8057830 con 0x7f72b006e5d0 2023-10-16T11:00:26.307 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:26.308 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":159,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:25.381476+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:04:02.482303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T10:04:03.019780+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2640446942":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3643659736":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1187395764":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/2828352389":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6833/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6832/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3333502274":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1458403109":"2023-10-17T10:04:17.957825+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:26.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 msgr2=0x7f729406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:26.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 0x7f729406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b006e5d0 msgr2=0x7f72b00c8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b006e5d0 0x7f72b00c8c20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f72a800caa0 tx=0x7f72a800cdb0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 shutdown_connections 2023-10-16T11:00:26.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f729406b020 0x7f729406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b006e5d0 0x7f72b00c8c20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72b006ab40 0x7f72b00b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 --2- 172.21.15.158:0/724789300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b006af20 0x7f72b0148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 >> 172.21.15.158:0/724789300 conn(0x7f72b00d02b0 msgr2=0x7f72b00ad790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:26.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 shutdown_connections 2023-10-16T11:00:26.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.308+0000 7f72b7c43700 1 -- 172.21.15.158:0/724789300 wait complete. 2023-10-16T11:00:26.322 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:00:26.323 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:00:26.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.446+0000 7f405cf8b700 1 Processor -- start 2023-10-16T11:00:26.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.446+0000 7f405cf8b700 1 -- start start 2023-10-16T11:00:26.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f405cf8b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f40580b7810 0x7f40581683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f405cf8b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40580d3bd0 con 0x7f405815d7c0 2023-10-16T11:00:26.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f405cf8b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40580d2fc0 con 0x7f40580d1210 2023-10-16T11:00:26.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f405cf8b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40580d4060 con 0x7f40580b7810 2023-10-16T11:00:26.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f4056d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f40580b7810 0x7f40581683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f405659c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f405815d7c0 0x7f40580d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:32780/0 (socket says 172.21.15.158:32780) 2023-10-16T11:00:26.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f405659c700 1 -- 172.21.15.158:0/3643882991 learned_addr learned my addr 172.21.15.158:0/3643882991 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:26.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.447+0000 7f4056d9d700 1 -- 172.21.15.158:0/3643882991 >> v1:172.21.15.172:6789/0 conn(0x7f40580d1210 legacy=0x7f40580ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f4056d9d700 1 -- 172.21.15.158:0/3643882991 >> v1:172.21.15.158:6789/0 conn(0x7f405815d7c0 legacy=0x7f40580d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f4056d9d700 1 -- 172.21.15.158:0/3643882991 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580d32a0 con 0x7f40580b7810 2023-10-16T11:00:26.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f4056d9d700 1 --2- 172.21.15.158:0/3643882991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580b7810 0x7f40581683f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f4050009d90 tx=0x7f405000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732860f27349dbf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:26.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f405559a700 1 -- 172.21.15.158:0/3643882991 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405000e070 con 0x7f40580b7810 2023-10-16T11:00:26.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f405559a700 1 -- 172.21.15.158:0/3643882991 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405000bbd0 con 0x7f40580b7810 2023-10-16T11:00:26.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f405559a700 1 -- 172.21.15.158:0/3643882991 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4050012430 con 0x7f40580b7810 2023-10-16T11:00:26.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f405cf8b700 1 -- 172.21.15.158:0/3643882991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580b7810 msgr2=0x7f40581683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.448+0000 7f405cf8b700 1 --2- 172.21.15.158:0/3643882991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580b7810 0x7f40581683f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f4050009d90 tx=0x7f405000b330 comp rx=0 tx=0).stop 2023-10-16T11:00:26.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.449+0000 7f405cf8b700 1 -- 172.21.15.158:0/3643882991 shutdown_connections 2023-10-16T11:00:26.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.449+0000 7f405cf8b700 1 --2- 172.21.15.158:0/3643882991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580b7810 0x7f40581683f0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.449+0000 7f405cf8b700 1 -- 172.21.15.158:0/3643882991 >> 172.21.15.158:0/3643882991 conn(0x7f40580669f0 msgr2=0x7f4058066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:26.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.449+0000 7f405cf8b700 1 -- 172.21.15.158:0/3643882991 shutdown_connections 2023-10-16T11:00:26.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.449+0000 7f405cf8b700 1 -- 172.21.15.158:0/3643882991 wait complete. 2023-10-16T11:00:26.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.450+0000 7f405cf8b700 1 Processor -- start 2023-10-16T11:00:26.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.450+0000 7f405cf8b700 1 -- start start 2023-10-16T11:00:26.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405cf8b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40580b7810 0x7f40580a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405cf8b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580d1210 0x7f40581f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405cf8b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f405815d7c0 0x7f40581f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405cf8b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580aa650 con 0x7f40580d1210 2023-10-16T11:00:26.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405cf8b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581f3c80 con 0x7f40580b7810 2023-10-16T11:00:26.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405cf8b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405815f600 con 0x7f405815d7c0 2023-10-16T11:00:26.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580d1210 0x7f40581f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580d1210 0x7f40581f3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60844/0 (socket says 172.21.15.158:60844) 2023-10-16T11:00:26.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 -- 172.21.15.158:0/2933186081 learned_addr learned my addr 172.21.15.158:0/2933186081 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:26.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405659c700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40580b7810 0x7f40580a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4056d9d700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f405815d7c0 0x7f40581f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 -- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f405815d7c0 msgr2=0x7f40581f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f405815d7c0 0x7f40581f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 -- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40580b7810 msgr2=0x7f40580a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40580b7810 0x7f40580a9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 -- 172.21.15.158:0/2933186081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4050009a70 con 0x7f40580d1210 2023-10-16T11:00:26.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f405659c700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40580b7810 0x7f40580a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:26.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4055d9b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580d1210 0x7f40581f3740 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f404c00c4e0 tx=0x7f404c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:26.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4046ffd700 1 -- 172.21.15.158:0/2933186081 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c0052d0 con 0x7f40580d1210 2023-10-16T11:00:26.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.451+0000 7f4046ffd700 1 -- 172.21.15.158:0/2933186081 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c005430 con 0x7f40580d1210 2023-10-16T11:00:26.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.452+0000 7f4046ffd700 1 -- 172.21.15.158:0/2933186081 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c0038c0 con 0x7f40580d1210 2023-10-16T11:00:26.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.452+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405815f8e0 con 0x7f40580d1210 2023-10-16T11:00:26.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.452+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405815fea0 con 0x7f40580d1210 2023-10-16T11:00:26.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.452+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40581600c0 con 0x7f40580d1210 2023-10-16T11:00:26.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.453+0000 7f4046ffd700 1 -- 172.21.15.158:0/2933186081 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f404c003a20 con 0x7f40580d1210 2023-10-16T11:00:26.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.453+0000 7f4046ffd700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 0x7f403806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.454+0000 7f405659c700 1 -- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 msgr2=0x7f403806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:26.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.454+0000 7f4046ffd700 1 -- 172.21.15.158:0/2933186081 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f404c089340 con 0x7f40580d1210 2023-10-16T11:00:26.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.454+0000 7f405659c700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 0x7f403806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:26.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.457+0000 7f4046ffd700 1 -- 172.21.15.158:0/2933186081 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f404c053f00 con 0x7f40580d1210 2023-10-16T11:00:26.555 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:26.553+0000 7fe0ee3c7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:00:26.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.655+0000 7f405659c700 1 -- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 msgr2=0x7f403806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:26.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.655+0000 7f405659c700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 0x7f403806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:26.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.701+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40580681b0 con 0x7f40580d1210 2023-10-16T11:00:26.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.702+0000 7f4046ffd700 1 -- 172.21.15.158:0/2933186081 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f404c057520 con 0x7f40580d1210 2023-10-16T11:00:26.703 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:26.703 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8422,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1604199546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1604199546},{"type":"v1","addr":"172.21.15.172:6834","nonce":1604199546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/95492242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":95492242},{"type":"v1","addr":"172.21.15.158:6836","nonce":95492242}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9393,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112424210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112424210},{"type":"v1","addr":"172.21.15.158:6837","nonce":112424210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:04:04.049571+0000","modified":"2023-10-16T10:04:07.086241+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37352":{"gid":37352,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/253283745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":253283745},{"type":"v1","addr":"172.21.15.172:6835","nonce":253283745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T11:00:26.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.704+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 msgr2=0x7f403806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:26.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.704+0000 7f405cf8b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 0x7f403806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.704+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580d1210 msgr2=0x7f40581f3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.704+0000 7f405cf8b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580d1210 0x7f40581f3740 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f404c00c4e0 tx=0x7f404c00c7f0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.704+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 shutdown_connections 2023-10-16T11:00:26.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.704+0000 7f405cf8b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7f403806b020 0x7f403806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.705+0000 7f405cf8b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f405815d7c0 0x7f40581f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.705+0000 7f405cf8b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40580b7810 0x7f40580a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.705+0000 7f405cf8b700 1 --2- 172.21.15.158:0/2933186081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40580d1210 0x7f40581f3740 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.705+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 >> 172.21.15.158:0/2933186081 conn(0x7f40580669f0 msgr2=0x7f40580b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:26.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.705+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 shutdown_connections 2023-10-16T11:00:26.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.705+0000 7f405cf8b700 1 -- 172.21.15.158:0/2933186081 wait complete. 2023-10-16T11:00:26.708 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 77 2023-10-16T11:00:26.718 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:00:26.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.840+0000 7fdf14a14700 1 Processor -- start 2023-10-16T11:00:26.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.841+0000 7fdf14a14700 1 -- start start 2023-10-16T11:00:26.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.841+0000 7fdf14a14700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdf1006bfb0 0x7fdf100cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.841+0000 7fdf14a14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf100cf2f0 0x7fdf100cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf14a14700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1015b810 con 0x7fdf100cc0c0 2023-10-16T11:00:26.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf14a14700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf100cf7a0 con 0x7fdf1006bfb0 2023-10-16T11:00:26.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf14a14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1015a990 con 0x7fdf100cf2f0 2023-10-16T11:00:26.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0e59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdf100cc0c0 0x7fdf100aeda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:32796/0 (socket says 172.21.15.158:32796) 2023-10-16T11:00:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0e59c700 1 -- 172.21.15.158:0/1105929107 learned_addr learned my addr 172.21.15.158:0/1105929107 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0dd9b700 1 --2- 172.21.15.158:0/1105929107 >> v2:172.21.15.172:3300/0 conn(0x7fdf1006bfb0 0x7fdf100cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0ed9d700 1 --2- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf100cf2f0 0x7fdf100cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0d59a700 1 -- 172.21.15.158:0/1105929107 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275388455 0 0) 0x7fdf1015b810 con 0x7fdf100cc0c0 2023-10-16T11:00:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0d59a700 1 -- 172.21.15.158:0/1105929107 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef80035b0 con 0x7fdf100cc0c0 2023-10-16T11:00:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0dd9b700 1 -- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf100cf2f0 msgr2=0x7fdf100cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0dd9b700 1 --2- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf100cf2f0 0x7fdf100cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0dd9b700 1 -- 172.21.15.158:0/1105929107 >> v1:172.21.15.158:6789/0 conn(0x7fdf100cc0c0 legacy=0x7fdf100aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0dd9b700 1 -- 172.21.15.158:0/1105929107 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1015ac00 con 0x7fdf1006bfb0 2023-10-16T11:00:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0ed9d700 1 --2- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf100cf2f0 0x7fdf100cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:26.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0dd9b700 1 --2- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf1006bfb0 0x7fdf100cedb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009840 tx=0x7fdf04009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1096237a1f8e01ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:26.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0d59a700 1 -- 172.21.15.158:0/1105929107 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04011830 con 0x7fdf1006bfb0 2023-10-16T11:00:26.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.842+0000 7fdf0d59a700 1 -- 172.21.15.158:0/1105929107 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf04011990 con 0x7fdf1006bfb0 2023-10-16T11:00:26.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf0d59a700 1 -- 172.21.15.158:0/1105929107 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04011d20 con 0x7fdf1006bfb0 2023-10-16T11:00:26.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 -- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf1006bfb0 msgr2=0x7fdf100cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf1006bfb0 0x7fdf100cedb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009840 tx=0x7fdf04009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:26.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 -- 172.21.15.158:0/1105929107 shutdown_connections 2023-10-16T11:00:26.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf100cf2f0 0x7fdf100cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1105929107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf1006bfb0 0x7fdf100cedb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 -- 172.21.15.158:0/1105929107 >> 172.21.15.158:0/1105929107 conn(0x7fdf100cc4b0 msgr2=0x7fdf100cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:26.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 -- 172.21.15.158:0/1105929107 shutdown_connections 2023-10-16T11:00:26.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.843+0000 7fdf14a14700 1 -- 172.21.15.158:0/1105929107 wait complete. 2023-10-16T11:00:26.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 Processor -- start 2023-10-16T11:00:26.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 -- start start 2023-10-16T11:00:26.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf1006bfb0 0x7fdf100c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf100cf2f0 0x7fdf101f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf10144370 0x7fdf101594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf101f72d0 con 0x7fdf100cf2f0 2023-10-16T11:00:26.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf100b2750 con 0x7fdf10144370 2023-10-16T11:00:26.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf14a14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf100c5760 con 0x7fdf1006bfb0 2023-10-16T11:00:26.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.845+0000 7fdf0dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf100cf2f0 0x7fdf101f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf100cf2f0 0x7fdf101f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60860/0 (socket says 172.21.15.158:60860) 2023-10-16T11:00:26.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 -- 172.21.15.158:0/1376807316 learned_addr learned my addr 172.21.15.158:0/1376807316 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:26.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0e59c700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf1006bfb0 0x7fdf100c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0ed9d700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf10144370 0x7fdf101594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:26.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 -- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf1006bfb0 msgr2=0x7fdf100c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf1006bfb0 0x7fdf100c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 -- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf10144370 msgr2=0x7fdf101594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:26.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf10144370 0x7fdf101594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 -- 172.21.15.158:0/1376807316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf08009a70 con 0x7fdf100cf2f0 2023-10-16T11:00:26.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf0dd9b700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf100cf2f0 0x7fdf101f6d90 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fdf040090b0 tx=0x7fdf04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:26.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04004550 con 0x7fdf100cf2f0 2023-10-16T11:00:26.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0401f070 con 0x7fdf100cf2f0 2023-10-16T11:00:26.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040094a0 con 0x7fdf100cf2f0 2023-10-16T11:00:26.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf040038e0 con 0x7fdf100cf2f0 2023-10-16T11:00:26.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.846+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf100b2c80 con 0x7fdf100cf2f0 2023-10-16T11:00:26.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.847+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf100681b0 con 0x7fdf100cf2f0 2023-10-16T11:00:26.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.848+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fdf04003a40 con 0x7fdf100cf2f0 2023-10-16T11:00:26.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.848+0000 7fdefeffd700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fdef006b020 0x7fdef006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:26.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.848+0000 7fdf0e59c700 1 -- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fdef006b020 msgr2=0x7fdef006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2326422563 2023-10-16T11:00:26.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.848+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fdf04021070 con 0x7fdf100cf2f0 2023-10-16T11:00:26.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.848+0000 7fdf0e59c700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fdef006b020 0x7fdef006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:26.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.852+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf04055dd0 con 0x7fdf100cf2f0 2023-10-16T11:00:26.873 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.872+0000 7f2ab7708200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:00:26.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.892+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fdf04003cf0 con 0x7fdf100cf2f0 2023-10-16T11:00:26.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.892+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fdef006b020 msgr2=0x7fdef006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:26.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:26.892+0000 7fdefeffd700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fdef006b020 0x7fdef006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:26.954 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a2feee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.955 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a2feee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.955 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a2feee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.955 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a2feee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.955 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a2feee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.955 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a32ef4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.956 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a32ef4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.956 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a32ef4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.956 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a32ef4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:26.956 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:26.953+0000 7f2a32ef4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:27.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.110+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf1015c3a0 con 0x7fdf100cf2f0 2023-10-16T11:00:27.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.111+0000 7fdefeffd700 1 -- 172.21.15.158:0/1376807316 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fdf0405c700 con 0x7fdf100cf2f0 2023-10-16T11:00:27.113 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:27.113 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8422,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1604199546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1604199546},{"type":"v1","addr":"172.21.15.172:6834","nonce":1604199546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/95492242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":95492242},{"type":"v1","addr":"172.21.15.158:6836","nonce":95492242}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9393,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112424210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112424210},{"type":"v1","addr":"172.21.15.158:6837","nonce":112424210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T10:04:04.049571+0000","modified":"2023-10-16T10:04:07.086241+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37352":{"gid":37352,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/253283745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":253283745},{"type":"v1","addr":"172.21.15.172:6835","nonce":253283745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T11:00:27.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf100cf2f0 msgr2=0x7fdf101f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf100cf2f0 0x7fdf101f6d90 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fdf040090b0 tx=0x7fdf04004750 comp rx=0 tx=0).stop 2023-10-16T11:00:27.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 shutdown_connections 2023-10-16T11:00:27.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:6832/2326422563,v1:172.21.15.158:6833/2326422563] conn(0x7fdef006b020 0x7fdef006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf10144370 0x7fdf101594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf1006bfb0 0x7fdf100c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 --2- 172.21.15.158:0/1376807316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf100cf2f0 0x7fdf101f6d90 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 >> 172.21.15.158:0/1376807316 conn(0x7fdf100cc4b0 msgr2=0x7fdf100cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:27.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 shutdown_connections 2023-10-16T11:00:27.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.113+0000 7fdf14a14700 1 -- 172.21.15.158:0/1376807316 wait complete. 2023-10-16T11:00:27.117 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 77 2023-10-16T11:00:27.127 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:27.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.246+0000 7fb202741700 1 Processor -- start 2023-10-16T11:00:27.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.246+0000 7fb202741700 1 -- start start 2023-10-16T11:00:27.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb202741700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb202741700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb1fc06e5d0 0x7fb1fc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb202741700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1fc06af20 0x7fb1fc0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb202741700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc071e50 con 0x7fb1fc06af20 2023-10-16T11:00:27.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb202741700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc161270 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb202741700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc161510 con 0x7fb1fc06ab40 2023-10-16T11:00:27.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1fc06af20 0x7fb1fc0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1fc06af20 0x7fb1fc0722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60940/0 (socket says 172.21.15.158:60940) 2023-10-16T11:00:27.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 -- 172.21.15.158:0/166359545 learned_addr learned my addr 172.21.15.158:0/166359545 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:27.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb1fbfff700 1 --2- 172.21.15.158:0/166359545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb1fb7fe700 1 --2- 172.21.15.158:0/166359545 >> v2:172.21.15.172:3300/0 conn(0x7fb1fc06e5d0 0x7fb1fc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 -- 172.21.15.158:0/166359545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 msgr2=0x7fb1fc06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:27.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 --2- 172.21.15.158:0/166359545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 -- 172.21.15.158:0/166359545 >> v2:172.21.15.172:3300/0 conn(0x7fb1fc06e5d0 msgr2=0x7fb1fc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:27.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 --2- 172.21.15.158:0/166359545 >> v2:172.21.15.172:3300/0 conn(0x7fb1fc06e5d0 0x7fb1fc071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.247+0000 7fb200cde700 1 -- 172.21.15.158:0/166359545 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc1617f0 con 0x7fb1fc06af20 2023-10-16T11:00:27.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb200cde700 1 --2- 172.21.15.158:0/166359545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0722a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc074da0 tx=0x7fb1f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0bad8a9f2f0d19 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:27.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb1faffd700 1 -- 172.21.15.158:0/166359545 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400e070 con 0x7fb1fc06af20 2023-10-16T11:00:27.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb1faffd700 1 -- 172.21.15.158:0/166359545 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f400b9c0 con 0x7fb1fc06af20 2023-10-16T11:00:27.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb1faffd700 1 -- 172.21.15.158:0/166359545 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400bd70 con 0x7fb1fc06af20 2023-10-16T11:00:27.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 -- 172.21.15.158:0/166359545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06af20 msgr2=0x7fb1fc0722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 --2- 172.21.15.158:0/166359545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0722a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc074da0 tx=0x7fb1f400b330 comp rx=0 tx=0).stop 2023-10-16T11:00:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 -- 172.21.15.158:0/166359545 shutdown_connections 2023-10-16T11:00:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 --2- 172.21.15.158:0/166359545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0722a0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 --2- 172.21.15.158:0/166359545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 --2- 172.21.15.158:0/166359545 >> v2:172.21.15.172:3300/0 conn(0x7fb1fc06e5d0 0x7fb1fc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 -- 172.21.15.158:0/166359545 >> 172.21.15.158:0/166359545 conn(0x7fb1fc0d02b0 msgr2=0x7fb1fc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:27.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 -- 172.21.15.158:0/166359545 shutdown_connections 2023-10-16T11:00:27.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.248+0000 7fb202741700 1 -- 172.21.15.158:0/166359545 wait complete. 2023-10-16T11:00:27.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.250+0000 7fb202741700 1 Processor -- start 2023-10-16T11:00:27.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.250+0000 7fb202741700 1 -- start start 2023-10-16T11:00:27.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.250+0000 7fb202741700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.250+0000 7fb202741700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb202741700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06e5d0 0x7fb1fc0c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb202741700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc1f65f0 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb202741700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc1f6890 con 0x7fb1fc06af20 2023-10-16T11:00:27.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb202741700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc1f6b70 con 0x7fb1fc06ab40 2023-10-16T11:00:27.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06e5d0 0x7fb1fc0c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06e5d0 0x7fb1fc0c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60946/0 (socket says 172.21.15.158:60946) 2023-10-16T11:00:27.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 -- 172.21.15.158:0/3960687434 learned_addr learned my addr 172.21.15.158:0/3960687434 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:27.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb1fbfff700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb1fb7fe700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 -- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 msgr2=0x7fb1fc152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 -- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1fc06af20 msgr2=0x7fb1fc0b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 -- 172.21.15.158:0/3960687434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f4009b30 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb1fb7fe700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:27.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb1fbfff700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:27.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb200cde700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06e5d0 0x7fb1fc0c6050 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc074ec0 tx=0x7fb1f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:27.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb1f8ff9700 1 -- 172.21.15.158:0/3960687434 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4004250 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb1f8ff9700 1 -- 172.21.15.158:0/3960687434 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f40043d0 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc1f6e50 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb1f8ff9700 1 -- 172.21.15.158:0/3960687434 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4004780 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.251+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc152f60 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.252+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc1f7370 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.253+0000 7fb1f8ff9700 1 -- 172.21.15.158:0/3960687434 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb1f401c440 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.253+0000 7fb1f8ff9700 1 -- 172.21.15.158:0/3960687434 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fb1f408ac40 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.257+0000 7fb1f8ff9700 1 -- 172.21.15.158:0/3960687434 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1f4055c10 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.454+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1fc161aa0 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.455+0000 7fb1f8ff9700 1 -- 172.21.15.158:0/3960687434 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18545 (secure 0 0 0) 0x7fb1f4020090 con 0x7fb1fc06e5d0 2023-10-16T11:00:27.457 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:27.457 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":160,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:26.886736+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T10:04:02.482303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T10:04:03.019780+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/963025365":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/89982933":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/4116889804":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/4154474342":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/3915457102":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/559163437":"2023-10-17T11:00:26.886700+0000","172.21.15.158:6833/2326422563":"2023-10-17T11:00:26.886700+0000","172.21.15.158:6832/2326422563":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/2640446942":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3643659736":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1187395764":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/2828352389":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6833/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6832/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3333502274":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1458403109":"2023-10-17T10:04:17.957825+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06e5d0 msgr2=0x7fb1fc0c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06e5d0 0x7fb1fc0c6050 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc074ec0 tx=0x7fb1f4004ad0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 shutdown_connections 2023-10-16T11:00:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1fc06e5d0 0x7fb1fc0c6050 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1fc06ab40 0x7fb1fc152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 --2- 172.21.15.158:0/3960687434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1fc06af20 0x7fb1fc0b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 >> 172.21.15.158:0/3960687434 conn(0x7fb1fc0d02b0 msgr2=0x7fb1fc0ad870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 shutdown_connections 2023-10-16T11:00:27.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.457+0000 7fb202741700 1 -- 172.21.15.158:0/3960687434 wait complete. 2023-10-16T11:00:27.471 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:00:27.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.592+0000 7f7f185c5700 1 Processor -- start 2023-10-16T11:00:27.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.592+0000 7f7f185c5700 1 -- start start 2023-10-16T11:00:27.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f185c5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 0x7f7f100b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f185c5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f100af600 con 0x7f7f1006e480 2023-10-16T11:00:27.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f185c5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1006e9b0 con 0x7f7f1015d990 2023-10-16T11:00:27.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f185c5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100b2c90 con 0x7f7f100cfd60 2023-10-16T11:00:27.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f15b60700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7f1006e480 0x7f7f100cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:32894/0 (socket says 172.21.15.158:32894) 2023-10-16T11:00:27.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f15b60700 1 -- 172.21.15.158:0/1757065958 learned_addr learned my addr 172.21.15.158:0/1757065958 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:27.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f16b62700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 0x7f7f100b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202274652 0 0) 0x7f7f100af600 con 0x7f7f1006e480 2023-10-16T11:00:27.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7efc0035b0 con 0x7f7f1006e480 2023-10-16T11:00:27.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769310895 0 0) 0x7f7f1006e9b0 con 0x7f7f1015d990 2023-10-16T11:00:27.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f100af600 con 0x7f7f1015d990 2023-10-16T11:00:27.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782213048 0 0) 0x7f7efc0035b0 con 0x7f7f1006e480 2023-10-16T11:00:27.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 msgr2=0x7f7f100b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:27.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 0x7f7f100b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 >> v1:172.21.15.172:6789/0 conn(0x7f7f1015d990 legacy=0x7f7f100b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f100ce890 con 0x7f7f1006e480 2023-10-16T11:00:27.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7f000015f0 con 0x7f7f1006e480 2023-10-16T11:00:27.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 >> v1:172.21.15.158:6789/0 conn(0x7f7f1006e480 legacy=0x7f7f100cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7efc003bc0 0x7f7efc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7efc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 0x7f7efc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100b2c90 con 0x7f7efc004850 2023-10-16T11:00:27.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc00bac0 con 0x7f7efc003bc0 2023-10-16T11:00:27.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.593+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc00bc80 con 0x7f7f100cfd60 2023-10-16T11:00:27.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f15b60700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7efc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f16b62700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 0x7f7efc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f16361700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7efc003bc0 0x7f7efc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f15b60700 1 -- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 msgr2=0x7f7efc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:27.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f15b60700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f100cfd60 0x7f7efc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f15b60700 1 -- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7efc003bc0 msgr2=0x7f7efc003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:27.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f15b60700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7efc003bc0 0x7f7efc003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f15b60700 1 -- 172.21.15.158:0/1757065958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc0035b0 con 0x7f7efc004850 2023-10-16T11:00:27.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f15b60700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7efc004cd0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f7f00007fa0 tx=0x7f7efc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff5f6787a3c4e23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:27.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0000cc80 con 0x7f7efc004850 2023-10-16T11:00:27.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0000ce00 con 0x7f7efc004850 2023-10-16T11:00:27.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.594+0000 7f7f1535f700 1 -- 172.21.15.158:0/1757065958 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00005ba0 con 0x7f7efc004850 2023-10-16T11:00:27.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.595+0000 7f7f185c5700 1 -- 172.21.15.158:0/1757065958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 msgr2=0x7f7efc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.595+0000 7f7f185c5700 1 --2- 172.21.15.158:0/1757065958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7efc004cd0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f7f00007fa0 tx=0x7f7efc005880 comp rx=0 tx=0).stop 2023-10-16T11:00:27.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.595+0000 7f7f16b62700 1 -- 172.21.15.158:0/1757065958 reap_dead start 2023-10-16T11:00:27.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.595+0000 7f7f185c5700 1 -- 172.21.15.158:0/1757065958 shutdown_connections 2023-10-16T11:00:27.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.595+0000 7f7f185c5700 1 -- 172.21.15.158:0/1757065958 >> 172.21.15.158:0/1757065958 conn(0x7f7f1006b9e0 msgr2=0x7f7f1006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:27.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.595+0000 7f7f185c5700 1 -- 172.21.15.158:0/1757065958 shutdown_connections 2023-10-16T11:00:27.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.595+0000 7f7f185c5700 1 -- 172.21.15.158:0/1757065958 wait complete. 2023-10-16T11:00:27.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.596+0000 7f7f185c5700 1 Processor -- start 2023-10-16T11:00:27.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.596+0000 7f7f185c5700 1 -- start start 2023-10-16T11:00:27.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7f100cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f1015dd70 0x7f7f101f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f101f2ce0 0x7f7f100ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100c4b30 con 0x7f7efc004850 2023-10-16T11:00:27.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1015e5c0 con 0x7f7f1015dd70 2023-10-16T11:00:27.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1015e8a0 con 0x7f7f101f2ce0 2023-10-16T11:00:27.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7f100cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7f100cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60960/0 (socket says 172.21.15.158:60960) 2023-10-16T11:00:27.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 -- 172.21.15.158:0/1576339355 learned_addr learned my addr 172.21.15.158:0/1576339355 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:27.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16b62700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f101f2ce0 0x7f7f100ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f15b60700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f1015dd70 0x7f7f101f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 -- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f101f2ce0 msgr2=0x7f7f100ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f101f2ce0 0x7f7f100ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 -- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f1015dd70 msgr2=0x7f7f101f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f1015dd70 0x7f7f101f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 -- 172.21.15.158:0/1576339355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1015eb80 con 0x7f7efc004850 2023-10-16T11:00:27.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16b62700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f101f2ce0 0x7f7f100ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:27.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f16361700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7f100cd580 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00f800 tx=0x7f7f0c00fbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:27.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f06ffd700 1 -- 172.21.15.158:0/1576339355 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c005320 con 0x7f7efc004850 2023-10-16T11:00:27.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f06ffd700 1 -- 172.21.15.158:0/1576339355 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c0054a0 con 0x7f7efc004850 2023-10-16T11:00:27.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f06ffd700 1 -- 172.21.15.158:0/1576339355 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c006aa0 con 0x7f7efc004850 2023-10-16T11:00:27.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1015ee60 con 0x7f7efc004850 2023-10-16T11:00:27.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.597+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f100ae640 con 0x7f7efc004850 2023-10-16T11:00:27.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.598+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f100681b0 con 0x7f7efc004850 2023-10-16T11:00:27.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.599+0000 7f7f06ffd700 1 -- 172.21.15.158:0/1576339355 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7f0c006c00 con 0x7f7efc004850 2023-10-16T11:00:27.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.599+0000 7f7f06ffd700 1 -- 172.21.15.158:0/1576339355 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f7f0c088d00 con 0x7f7efc004850 2023-10-16T11:00:27.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.604+0000 7f7f06ffd700 1 -- 172.21.15.158:0/1576339355 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7f0c08c030 con 0x7f7efc004850 2023-10-16T11:00:27.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.842+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7f1015c130 con 0x7f7efc004850 2023-10-16T11:00:27.844 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:00:27.843+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:00:27.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.844+0000 7f7f06ffd700 1 -- 172.21.15.158:0/1576339355 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7f0c01a090 con 0x7f7efc004850 2023-10-16T11:00:27.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.847+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 msgr2=0x7f7f100cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.847+0000 7f7f185c5700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7f100cd580 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00f800 tx=0x7f7f0c00fbc0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.847+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 shutdown_connections 2023-10-16T11:00:27.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.847+0000 7f7f185c5700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f101f2ce0 0x7f7f100ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.847+0000 7f7f185c5700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7efc004850 0x7f7f100cd580 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.847+0000 7f7f185c5700 1 --2- 172.21.15.158:0/1576339355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f1015dd70 0x7f7f101f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.847+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 >> 172.21.15.158:0/1576339355 conn(0x7f7f1006b9e0 msgr2=0x7f7f100acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:27.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.848+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 shutdown_connections 2023-10-16T11:00:27.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.848+0000 7f7f185c5700 1 -- 172.21.15.158:0/1576339355 wait complete. 2023-10-16T11:00:27.851 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:00:27.861 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:00:27.887 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:00:27.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.988+0000 7fa6a48d0700 1 Processor -- start 2023-10-16T11:00:27.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.988+0000 7fa6a48d0700 1 -- start start 2023-10-16T11:00:27.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa6a48d0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa6a00d0e10 0x7fa6a015c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa6a48d0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a0159660 con 0x7fa6a0159280 2023-10-16T11:00:27.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa6a48d0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a006b120 con 0x7fa6a00cbef0 2023-10-16T11:00:27.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa6a48d0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a00cb710 con 0x7fa6a00d0e10 2023-10-16T11:00:27.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa69dd9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa6a00d0e10 0x7fa6a015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:27.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa69e59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa6a0159280 0x7fa6a006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:32932/0 (socket says 172.21.15.158:32932) 2023-10-16T11:00:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa69e59c700 1 -- 172.21.15.158:0/3408114053 learned_addr learned my addr 172.21.15.158:0/3408114053 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa69dd9b700 1 -- 172.21.15.158:0/3408114053 >> v1:172.21.15.172:6789/0 conn(0x7fa6a00cbef0 legacy=0x7fa6a00cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa69dd9b700 1 -- 172.21.15.158:0/3408114053 >> v1:172.21.15.158:6789/0 conn(0x7fa6a0159280 legacy=0x7fa6a006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa69dd9b700 1 -- 172.21.15.158:0/3408114053 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a006b400 con 0x7fa6a00d0e10 2023-10-16T11:00:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.989+0000 7fa69d59a700 1 -- 172.21.15.158:0/3408114053 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716797106 0 0) 0x7fa6a0159660 con 0x7fa6a0159280 2023-10-16T11:00:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa69dd9b700 1 --2- 172.21.15.158:0/3408114053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a015c590 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fa694009840 tx=0x7fa694009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e94e0ddc61b5daa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa69d59a700 1 -- 172.21.15.158:0/3408114053 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694004230 con 0x7fa6a00d0e10 2023-10-16T11:00:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa69d59a700 1 -- 172.21.15.158:0/3408114053 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa694004390 con 0x7fa6a00d0e10 2023-10-16T11:00:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa69d59a700 1 -- 172.21.15.158:0/3408114053 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6940036a0 con 0x7fa6a00d0e10 2023-10-16T11:00:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa6a48d0700 1 -- 172.21.15.158:0/3408114053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 msgr2=0x7fa6a015c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/3408114053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a015c590 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fa694009840 tx=0x7fa694009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa6a48d0700 1 -- 172.21.15.158:0/3408114053 shutdown_connections 2023-10-16T11:00:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/3408114053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a015c590 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa6a48d0700 1 -- 172.21.15.158:0/3408114053 >> 172.21.15.158:0/3408114053 conn(0x7fa6a0099a10 msgr2=0x7fa6a00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa6a48d0700 1 -- 172.21.15.158:0/3408114053 shutdown_connections 2023-10-16T11:00:27.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.990+0000 7fa6a48d0700 1 -- 172.21.15.158:0/3408114053 wait complete. 2023-10-16T11:00:27.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.992+0000 7fa6a48d0700 1 Processor -- start 2023-10-16T11:00:27.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.992+0000 7fa6a48d0700 1 -- start start 2023-10-16T11:00:27.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa6a48d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6a00cbef0 0x7fa6a0158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa6a48d0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a01f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa6a48d0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6a0159280 0x7fa6a01f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa6a48d0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00c5960 con 0x7fa6a00d0e10 2023-10-16T11:00:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa6a48d0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00cfe40 con 0x7fa6a0159280 2023-10-16T11:00:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa6a48d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00c5bb0 con 0x7fa6a00cbef0 2023-10-16T11:00:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa69dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a01f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa69dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a01f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:60996/0 (socket says 172.21.15.158:60996) 2023-10-16T11:00:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.993+0000 7fa69dd9b700 1 -- 172.21.15.158:0/4170330916 learned_addr learned my addr 172.21.15.158:0/4170330916 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69ed9d700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6a0159280 0x7fa6a01f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69e59c700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6a00cbef0 0x7fa6a0158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69dd9b700 1 -- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6a00cbef0 msgr2=0x7fa6a0158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69dd9b700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6a00cbef0 0x7fa6a0158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69dd9b700 1 -- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6a0159280 msgr2=0x7fa6a01f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69dd9b700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6a0159280 0x7fa6a01f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69dd9b700 1 -- 172.21.15.158:0/4170330916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940094a0 con 0x7fa6a00d0e10 2023-10-16T11:00:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69e59c700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6a00cbef0 0x7fa6a0158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:28.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa69dd9b700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a01f2c00 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fa6940090b0 tx=0x7fa6940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:28.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa68effd700 1 -- 172.21.15.158:0/4170330916 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694011830 con 0x7fa6a00d0e10 2023-10-16T11:00:28.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa68effd700 1 -- 172.21.15.158:0/4170330916 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa694011990 con 0x7fa6a00d0e10 2023-10-16T11:00:28.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a00c5e90 con 0x7fa6a00d0e10 2023-10-16T11:00:28.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa68effd700 1 -- 172.21.15.158:0/4170330916 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694011d40 con 0x7fa6a00d0e10 2023-10-16T11:00:28.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.994+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a00ce670 con 0x7fa6a00d0e10 2023-10-16T11:00:28.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.995+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a00b13a0 con 0x7fa6a00d0e10 2023-10-16T11:00:28.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.995+0000 7fa68effd700 1 -- 172.21.15.158:0/4170330916 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa69401b440 con 0x7fa6a00d0e10 2023-10-16T11:00:28.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.996+0000 7fa68effd700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa68006ad50 0x7fa68006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:28.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.996+0000 7fa68effd700 1 -- 172.21.15.158:0/4170330916 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 7299+0+0 (secure 0 0 0) 0x7fa69408b1d0 con 0x7fa6a00d0e10 2023-10-16T11:00:28.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.996+0000 7fa69e59c700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa68006ad50 0x7fa68006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:28.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:27.996+0000 7fa69e59c700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa68006ad50 0x7fa68006d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa698007fd0 tx=0x7fa698009040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:28.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.000+0000 7fa68effd700 1 -- 172.21.15.158:0/4170330916 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa694055f80 con 0x7fa6a00d0e10 2023-10-16T11:00:28.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.240+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa6a00c63b0 con 0x7fa6a00d0e10 2023-10-16T11:00:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.897+0000 7fa68effd700 1 -- 172.21.15.158:0/4170330916 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa6a00c63b0 con 0x7fa6a00d0e10 2023-10-16T11:00:28.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa68006ad50 msgr2=0x7fa68006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:28.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa68006ad50 0x7fa68006d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa698007fd0 tx=0x7fa698009040 comp rx=0 tx=0).stop 2023-10-16T11:00:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 msgr2=0x7fa6a01f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a01f2c00 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fa6940090b0 tx=0x7fa6940047c0 comp rx=0 tx=0).stop 2023-10-16T11:00:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 shutdown_connections 2023-10-16T11:00:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa68006ad50 0x7fa68006d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6a0159280 0x7fa6a01f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6a00cbef0 0x7fa6a0158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:28.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 --2- 172.21.15.158:0/4170330916 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6a00d0e10 0x7fa6a01f2c00 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:28.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 >> 172.21.15.158:0/4170330916 conn(0x7fa6a0099a10 msgr2=0x7fa6a006da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:28.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 shutdown_connections 2023-10-16T11:00:28.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:28.901+0000 7fa6a48d0700 1 -- 172.21.15.158:0/4170330916 wait complete. 2023-10-16T11:00:28.917 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:00:29.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.050+0000 7fc06f3a6700 1 Processor -- start 2023-10-16T11:00:29.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.050+0000 7fc06f3a6700 1 -- start start 2023-10-16T11:00:29.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06f3a6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc06815da40 0x7fc0680caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06f3a6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0680cc590 0x7fc06806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06f3a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06806ae10 0x7fc0680ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06f3a6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06806b280 con 0x7fc0680cc590 2023-10-16T11:00:29.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06f3a6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680cbd30 con 0x7fc06815da40 2023-10-16T11:00:29.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06f3a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680cc1c0 con 0x7fc06806ae10 2023-10-16T11:00:29.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06c941700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0680cc590 0x7fc06806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06c941700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc0680cc590 0x7fc06806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32790/0 (socket says 172.21.15.158:32790) 2023-10-16T11:00:29.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06c941700 1 -- 172.21.15.158:0/3902034266 learned_addr learned my addr 172.21.15.158:0/3902034266 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:29.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.051+0000 7fc06d943700 1 --2- 172.21.15.158:0/3902034266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06806ae10 0x7fc0680ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc06d142700 1 --2- 172.21.15.158:0/3902034266 >> v2:172.21.15.172:3300/0 conn(0x7fc06815da40 0x7fc0680caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc06c941700 1 -- 172.21.15.158:0/3902034266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06806ae10 msgr2=0x7fc0680ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:29.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc06c941700 1 --2- 172.21.15.158:0/3902034266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06806ae10 0x7fc0680ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc06c941700 1 -- 172.21.15.158:0/3902034266 >> v2:172.21.15.172:3300/0 conn(0x7fc06815da40 msgr2=0x7fc0680caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:29.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc06c941700 1 --2- 172.21.15.158:0/3902034266 >> v2:172.21.15.172:3300/0 conn(0x7fc06815da40 0x7fc0680caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc06c941700 1 -- 172.21.15.158:0/3902034266 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06815de20 con 0x7fc0680cc590 2023-10-16T11:00:29.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc06c941700 1 --2- 172.21.15.158:0/3902034266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0680cc590 0x7fc06806a8d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fc06806c9c0 tx=0x7fc058009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1973daa3245f3986 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:29.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc05ffff700 1 -- 172.21.15.158:0/3902034266 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05800b870 con 0x7fc0680cc590 2023-10-16T11:00:29.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc05ffff700 1 -- 172.21.15.158:0/3902034266 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05800b9d0 con 0x7fc0680cc590 2023-10-16T11:00:29.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.052+0000 7fc05ffff700 1 -- 172.21.15.158:0/3902034266 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05800bd60 con 0x7fc0680cc590 2023-10-16T11:00:29.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 -- 172.21.15.158:0/3902034266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0680cc590 msgr2=0x7fc06806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/3902034266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0680cc590 0x7fc06806a8d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fc06806c9c0 tx=0x7fc058009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 -- 172.21.15.158:0/3902034266 shutdown_connections 2023-10-16T11:00:29.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/3902034266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06806ae10 0x7fc0680ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/3902034266 >> v2:172.21.15.172:3300/0 conn(0x7fc06815da40 0x7fc0680caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/3902034266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0680cc590 0x7fc06806a8d0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 -- 172.21.15.158:0/3902034266 >> 172.21.15.158:0/3902034266 conn(0x7fc068099a10 msgr2=0x7fc0680d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:29.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 -- 172.21.15.158:0/3902034266 shutdown_connections 2023-10-16T11:00:29.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.053+0000 7fc06f3a6700 1 -- 172.21.15.158:0/3902034266 wait complete. 2023-10-16T11:00:29.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.054+0000 7fc06f3a6700 1 Processor -- start 2023-10-16T11:00:29.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.054+0000 7fc06f3a6700 1 -- start start 2023-10-16T11:00:29.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06f3a6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc06806ae10 0x7fc06815bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06f3a6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0680cc590 0x7fc0681f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06f3a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06815da40 0x7fc0681f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06f3a6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06815c260 con 0x7fc06806ae10 2023-10-16T11:00:29.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06f3a6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0681f3ac0 con 0x7fc0680cc590 2023-10-16T11:00:29.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06f3a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0681f3da0 con 0x7fc06815da40 2023-10-16T11:00:29.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc06806ae10 0x7fc06815bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc06806ae10 0x7fc06815bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32794/0 (socket says 172.21.15.158:32794) 2023-10-16T11:00:29.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 -- 172.21.15.158:0/1699163404 learned_addr learned my addr 172.21.15.158:0/1699163404 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:29.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06c941700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0680cc590 0x7fc0681f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d943700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06815da40 0x7fc0681f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 -- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06815da40 msgr2=0x7fc0681f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06815da40 0x7fc0681f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 -- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0680cc590 msgr2=0x7fc0681f41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0680cc590 0x7fc0681f41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 -- 172.21.15.158:0/1699163404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc058009550 con 0x7fc06806ae10 2023-10-16T11:00:29.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06d142700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc06806ae10 0x7fc06815bd20 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fc06400d800 tx=0x7fc06400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:29.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc05dffb700 1 -- 172.21.15.158:0/1699163404 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0640125f0 con 0x7fc06806ae10 2023-10-16T11:00:29.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc05dffb700 1 -- 172.21.15.158:0/1699163404 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc064012750 con 0x7fc06806ae10 2023-10-16T11:00:29.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0681e5070 con 0x7fc06806ae10 2023-10-16T11:00:29.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.055+0000 7fc05dffb700 1 -- 172.21.15.158:0/1699163404 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064012b00 con 0x7fc06806ae10 2023-10-16T11:00:29.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.056+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0681f4700 con 0x7fc06806ae10 2023-10-16T11:00:29.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.057+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0681f3fc0 con 0x7fc06806ae10 2023-10-16T11:00:29.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.057+0000 7fc05dffb700 1 -- 172.21.15.158:0/1699163404 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc064012c60 con 0x7fc06806ae10 2023-10-16T11:00:29.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.057+0000 7fc05dffb700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc04c06ad50 0x7fc04c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.058+0000 7fc06c941700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc04c06ad50 0x7fc04c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.058+0000 7fc05dffb700 1 -- 172.21.15.158:0/1699163404 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 7299+0+0 (secure 0 0 0) 0x7fc064058cb0 con 0x7fc06806ae10 2023-10-16T11:00:29.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.058+0000 7fc06c941700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc04c06ad50 0x7fc04c06d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc06806c9c0 tx=0x7fc058008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:29.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.062+0000 7fc05dffb700 1 -- 172.21.15.158:0/1699163404 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0640548f0 con 0x7fc06806ae10 2023-10-16T11:00:29.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.254+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc0680681b0 con 0x7fc06806ae10 2023-10-16T11:00:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.365+0000 7fc05dffb700 1 -- 172.21.15.158:0/1699163404 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v162) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc064057f10 con 0x7fc06806ae10 2023-10-16T11:00:29.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc04c06ad50 msgr2=0x7fc04c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc04c06ad50 0x7fc04c06d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc06806c9c0 tx=0x7fc058008030 comp rx=0 tx=0).stop 2023-10-16T11:00:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc06806ae10 msgr2=0x7fc06815bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc06806ae10 0x7fc06815bd20 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fc06400d800 tx=0x7fc06400db10 comp rx=0 tx=0).stop 2023-10-16T11:00:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 shutdown_connections 2023-10-16T11:00:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc06815da40 0x7fc0681f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc06806ae10 0x7fc06815bd20 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc04c06ad50 0x7fc04c06d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 --2- 172.21.15.158:0/1699163404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0680cc590 0x7fc0681f41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 >> 172.21.15.158:0/1699163404 conn(0x7fc068099a10 msgr2=0x7fc06806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 shutdown_connections 2023-10-16T11:00:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.367+0000 7fc06f3a6700 1 -- 172.21.15.158:0/1699163404 wait complete. 2023-10-16T11:00:29.371 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:00:29.383 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:00:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.507+0000 7f10ff716700 1 Processor -- start 2023-10-16T11:00:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.508+0000 7f10ff716700 1 -- start start 2023-10-16T11:00:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.508+0000 7f10ff716700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 0x7f10f80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.508+0000 7f10ff716700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10f80cc590 0x7f10f806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.508+0000 7f10ff716700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80cbd30 con 0x7f10f806ae10 2023-10-16T11:00:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.508+0000 7f10ff716700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f806b280 con 0x7f10f80cc590 2023-10-16T11:00:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.508+0000 7f10ff716700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80cc1c0 con 0x7f10f815da40 2023-10-16T11:00:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.508+0000 7f10fdcb3700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f10f806ae10 0x7f10f80ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:32980/0 (socket says 172.21.15.158:32980) 2023-10-16T11:00:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10fdcb3700 1 -- 172.21.15.158:0/1664398167 learned_addr learned my addr 172.21.15.158:0/1664398167 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10fccb1700 1 --2- 172.21.15.158:0/1664398167 >> v2:172.21.15.172:3300/0 conn(0x7f10f80cc590 0x7f10f806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10fd4b2700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 0x7f10f80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785557946 0 0) 0x7f10f80cbd30 con 0x7f10f806ae10 2023-10-16T11:00:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e40035b0 con 0x7f10f806ae10 2023-10-16T11:00:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2467933202 0 0) 0x7f10e40035b0 con 0x7f10f806ae10 2023-10-16T11:00:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 msgr2=0x7f10f80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 0x7f10f80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 >> v2:172.21.15.172:3300/0 conn(0x7f10f80cc590 msgr2=0x7f10f806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 --2- 172.21.15.158:0/1664398167 >> v2:172.21.15.172:3300/0 conn(0x7f10f80cc590 0x7f10f806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f815de20 con 0x7f10f806ae10 2023-10-16T11:00:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f10f00019f0 con 0x7f10f806ae10 2023-10-16T11:00:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 >> v1:172.21.15.158:6789/0 conn(0x7f10f806ae10 legacy=0x7f10f80ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4003a00 0x7f10e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e4004850 0x7f10e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 0x7f10e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f806b280 con 0x7f10e4004850 2023-10-16T11:00:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80cc1c0 con 0x7f10e4003a00 2023-10-16T11:00:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e400ba10 con 0x7f10f815da40 2023-10-16T11:00:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10fccb1700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e4004850 0x7f10e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10fd4b2700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4003a00 0x7f10e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.509+0000 7f10fdcb3700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 0x7f10e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fccb1700 1 -- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 msgr2=0x7f10e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fccb1700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f815da40 0x7f10e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fccb1700 1 -- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4003a00 msgr2=0x7f10e4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fccb1700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4003a00 0x7f10e4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fccb1700 1 -- 172.21.15.158:0/1664398167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e40035b0 con 0x7f10e4004850 2023-10-16T11:00:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fd4b2700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4003a00 0x7f10e4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fccb1700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e4004850 0x7f10e4004cd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009b60 tx=0x7f10e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674656ded8ce412 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f400e6c0 con 0x7f10e4004850 2023-10-16T11:00:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f400e820 con 0x7f10e4004850 2023-10-16T11:00:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10effff700 1 -- 172.21.15.158:0/1664398167 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f400eb90 con 0x7f10e4004850 2023-10-16T11:00:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10ff716700 1 -- 172.21.15.158:0/1664398167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e4004850 msgr2=0x7f10e4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10ff716700 1 --2- 172.21.15.158:0/1664398167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10e4004850 0x7f10e4004cd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009b60 tx=0x7f10e4005880 comp rx=0 tx=0).stop 2023-10-16T11:00:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10fdcb3700 1 -- 172.21.15.158:0/1664398167 reap_dead start 2023-10-16T11:00:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10ff716700 1 -- 172.21.15.158:0/1664398167 shutdown_connections 2023-10-16T11:00:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.510+0000 7f10ff716700 1 -- 172.21.15.158:0/1664398167 >> 172.21.15.158:0/1664398167 conn(0x7f10f8099a10 msgr2=0x7f10f80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.511+0000 7f10ff716700 1 -- 172.21.15.158:0/1664398167 shutdown_connections 2023-10-16T11:00:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.511+0000 7f10ff716700 1 -- 172.21.15.158:0/1664398167 wait complete. 2023-10-16T11:00:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.512+0000 7f10ff716700 1 Processor -- start 2023-10-16T11:00:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.512+0000 7f10ff716700 1 -- start start 2023-10-16T11:00:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10ff716700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4004850 0x7f10f815ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10ff716700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80b4cc0 0x7f10f80b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10ff716700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f80b2630 0x7f10f80c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10ff716700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80b5d70 con 0x7f10f80b2630 2023-10-16T11:00:29.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10ff716700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80b5160 con 0x7f10e4004850 2023-10-16T11:00:29.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10ff716700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80b5440 con 0x7f10f80b4cc0 2023-10-16T11:00:29.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fdcb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f80b2630 0x7f10f80c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fdcb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f80b2630 0x7f10f80c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32818/0 (socket says 172.21.15.158:32818) 2023-10-16T11:00:29.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fdcb3700 1 -- 172.21.15.158:0/1558315243 learned_addr learned my addr 172.21.15.158:0/1558315243 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:29.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fd4b2700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4004850 0x7f10f815ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fccb1700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80b4cc0 0x7f10f80b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fdcb3700 1 -- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80b4cc0 msgr2=0x7f10f80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fdcb3700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80b4cc0 0x7f10f80b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.513+0000 7f10fdcb3700 1 -- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4004850 msgr2=0x7f10f815ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:29.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10fdcb3700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4004850 0x7f10f815ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:29.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10fdcb3700 1 -- 172.21.15.158:0/1558315243 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80b5720 con 0x7f10f80b2630 2023-10-16T11:00:29.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10fccb1700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80b4cc0 0x7f10f80b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:29.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10fdcb3700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f80b2630 0x7f10f80c5680 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f10f806c9c0 tx=0x7f10f000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:29.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f000ad50 con 0x7f10f80b2630 2023-10-16T11:00:29.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f00133f0 con 0x7f10f80b2630 2023-10-16T11:00:29.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00137a0 con 0x7f10f80b2630 2023-10-16T11:00:29.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f80b5a00 con 0x7f10f80b2630 2023-10-16T11:00:29.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.514+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f80c5da0 con 0x7f10f80b2630 2023-10-16T11:00:29.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.515+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f80681b0 con 0x7f10f80b2630 2023-10-16T11:00:29.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.515+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f10f0013900 con 0x7f10f80b2630 2023-10-16T11:00:29.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.516+0000 7f10edffb700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f10dc06ad50 0x7f10dc06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:29.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.516+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f10f008a0c0 con 0x7f10f80b2630 2023-10-16T11:00:29.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.516+0000 7f10fd4b2700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f10dc06ad50 0x7f10dc06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:29.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.516+0000 7f10fd4b2700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f10dc06ad50 0x7f10dc06d210 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009b00 tx=0x7f10e8009210 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:29.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.520+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10f0055030 con 0x7f10f80b2630 2023-10-16T11:00:29.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.717+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10f80b1c40 con 0x7f10f80b2630 2023-10-16T11:00:29.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:29.910+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f10f0055400 con 0x7f10f80b2630 2023-10-16T11:00:30.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.394+0000 7f10edffb700 1 -- 172.21.15.158:0/1558315243 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v163) v1 ==== 169+0+0 (secure 0 0 0) 0x7f10f0070d60 con 0x7f10f80b2630 2023-10-16T11:00:30.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f10dc06ad50 msgr2=0x7f10dc06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f10dc06ad50 0x7f10dc06d210 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009b00 tx=0x7f10e8009210 comp rx=0 tx=0).stop 2023-10-16T11:00:30.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f80b2630 msgr2=0x7f10f80c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f80b2630 0x7f10f80c5680 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f10f806c9c0 tx=0x7f10f000a640 comp rx=0 tx=0).stop 2023-10-16T11:00:30.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 shutdown_connections 2023-10-16T11:00:30.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10f80b2630 0x7f10f80c5680 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f10dc06ad50 0x7f10dc06d210 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10e4004850 0x7f10f815ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 --2- 172.21.15.158:0/1558315243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10f80b4cc0 0x7f10f80b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 >> 172.21.15.158:0/1558315243 conn(0x7f10f8099a10 msgr2=0x7f10f80cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:30.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 shutdown_connections 2023-10-16T11:00:30.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.397+0000 7f10ff716700 1 -- 172.21.15.158:0/1558315243 wait complete. 2023-10-16T11:00:30.401 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:00:30.416 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:00:30.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.543+0000 7f1c6cb2b700 1 Processor -- start 2023-10-16T11:00:30.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.543+0000 7f1c6cb2b700 1 -- start start 2023-10-16T11:00:30.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c6cb2b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c68160d30 0x7f1c680b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c6cb2b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c680b7590 0x7f1c680d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c6cb2b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c680b7a20 con 0x7f1c680d0530 2023-10-16T11:00:30.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c6cb2b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c680b8910 con 0x7f1c680b7590 2023-10-16T11:00:30.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c6cb2b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c680d0ef0 con 0x7f1c68160d30 2023-10-16T11:00:30.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c680b7590 0x7f1c680d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c680b7590 0x7f1c680d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32826/0 (socket says 172.21.15.158:32826) 2023-10-16T11:00:30.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 -- 172.21.15.158:0/3175994066 learned_addr learned my addr 172.21.15.158:0/3175994066 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:30.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c6659c700 1 --2- 172.21.15.158:0/3175994066 >> v2:172.21.15.172:3300/0 conn(0x7f1c68160d30 0x7f1c680b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 -- 172.21.15.158:0/3175994066 >> v2:172.21.15.172:3300/0 conn(0x7f1c68160d30 msgr2=0x7f1c680b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:30.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 --2- 172.21.15.158:0/3175994066 >> v2:172.21.15.172:3300/0 conn(0x7f1c68160d30 0x7f1c680b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 -- 172.21.15.158:0/3175994066 >> v1:172.21.15.172:6789/0 conn(0x7f1c680d0530 legacy=0x7f1c680d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:30.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 -- 172.21.15.158:0/3175994066 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c680b7c90 con 0x7f1c680b7590 2023-10-16T11:00:30.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c66d9d700 1 --2- 172.21.15.158:0/3175994066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c680b7590 0x7f1c680d09b0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009e50 tx=0x7f1c6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfe2caa32577b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:30.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.544+0000 7f1c6559a700 1 -- 172.21.15.158:0/3175994066 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6000e070 con 0x7f1c680b7590 2023-10-16T11:00:30.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6559a700 1 -- 172.21.15.158:0/3175994066 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c60003a20 con 0x7f1c680b7590 2023-10-16T11:00:30.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6559a700 1 -- 172.21.15.158:0/3175994066 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60003dd0 con 0x7f1c680b7590 2023-10-16T11:00:30.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/3175994066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c680b7590 msgr2=0x7f1c680d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/3175994066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c680b7590 0x7f1c680d09b0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009e50 tx=0x7f1c6000b330 comp rx=0 tx=0).stop 2023-10-16T11:00:30.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/3175994066 shutdown_connections 2023-10-16T11:00:30.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/3175994066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c680b7590 0x7f1c680d09b0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/3175994066 >> v2:172.21.15.172:3300/0 conn(0x7f1c68160d30 0x7f1c680b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/3175994066 >> 172.21.15.158:0/3175994066 conn(0x7f1c680669f0 msgr2=0x7f1c68066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:30.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/3175994066 shutdown_connections 2023-10-16T11:00:30.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.545+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/3175994066 wait complete. 2023-10-16T11:00:30.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.546+0000 7f1c6cb2b700 1 Processor -- start 2023-10-16T11:00:30.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.546+0000 7f1c6cb2b700 1 -- start start 2023-10-16T11:00:30.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c6cb2b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c680b7590 0x7f1c681555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c6cb2b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c680d0530 0x7f1c6814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c6cb2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c68160d30 0x7f1c6815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c6cb2b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6815d760 con 0x7f1c68160d30 2023-10-16T11:00:30.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c6cb2b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6815d950 con 0x7f1c680d0530 2023-10-16T11:00:30.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c6cb2b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6815dc30 con 0x7f1c680b7590 2023-10-16T11:00:30.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c66d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c68160d30 0x7f1c6815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c66d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c68160d30 0x7f1c6815d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32832/0 (socket says 172.21.15.158:32832) 2023-10-16T11:00:30.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c66d9d700 1 -- 172.21.15.158:0/2663322907 learned_addr learned my addr 172.21.15.158:0/2663322907 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:30.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c65d9b700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c680d0530 0x7f1c6814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.547+0000 7f1c6659c700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c680b7590 0x7f1c681555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c66d9d700 1 -- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c680b7590 msgr2=0x7f1c681555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c66d9d700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c680b7590 0x7f1c681555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c66d9d700 1 -- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c680d0530 msgr2=0x7f1c6814aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c66d9d700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c680d0530 0x7f1c6814aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c66d9d700 1 -- 172.21.15.158:0/2663322907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c60009b30 con 0x7f1c68160d30 2023-10-16T11:00:30.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c6659c700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c680b7590 0x7f1c681555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:30.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c66d9d700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c68160d30 0x7f1c6815d220 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1c60015040 tx=0x7f1c60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:30.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c56ffd700 1 -- 172.21.15.158:0/2663322907 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60004440 con 0x7f1c68160d30 2023-10-16T11:00:30.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c56ffd700 1 -- 172.21.15.158:0/2663322907 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c600045c0 con 0x7f1c68160d30 2023-10-16T11:00:30.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6815df10 con 0x7f1c68160d30 2023-10-16T11:00:30.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c56ffd700 1 -- 172.21.15.158:0/2663322907 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60004970 con 0x7f1c68160d30 2023-10-16T11:00:30.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.548+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6815e4d0 con 0x7f1c68160d30 2023-10-16T11:00:30.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.549+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c680681b0 con 0x7f1c68160d30 2023-10-16T11:00:30.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.549+0000 7f1c56ffd700 1 -- 172.21.15.158:0/2663322907 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1c60012430 con 0x7f1c68160d30 2023-10-16T11:00:30.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.550+0000 7f1c56ffd700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1c4806af90 0x7f1c4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.550+0000 7f1c56ffd700 1 -- 172.21.15.158:0/2663322907 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f1c6008ad60 con 0x7f1c68160d30 2023-10-16T11:00:30.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.550+0000 7f1c6659c700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1c4806af90 0x7f1c4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.550+0000 7f1c6659c700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1c4806af90 0x7f1c4806d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1c58007a50 tx=0x7f1c58008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:30.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.554+0000 7f1c56ffd700 1 -- 172.21.15.158:0/2663322907 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c60059570 con 0x7f1c68160d30 2023-10-16T11:00:30.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.748+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1c680b8220 con 0x7f1c68160d30 2023-10-16T11:00:30.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.752+0000 7f1c56ffd700 1 -- 172.21.15.158:0/2663322907 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1c600170b0 con 0x7f1c68160d30 2023-10-16T11:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1c4806af90 msgr2=0x7f1c4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1c4806af90 0x7f1c4806d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1c58007a50 tx=0x7f1c58008040 comp rx=0 tx=0).stop 2023-10-16T11:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c68160d30 msgr2=0x7f1c6815d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c68160d30 0x7f1c6815d220 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1c60015040 tx=0x7f1c60004ad0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 shutdown_connections 2023-10-16T11:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c68160d30 0x7f1c6815d220 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c680b7590 0x7f1c681555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1c4806af90 0x7f1c4806d450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 --2- 172.21.15.158:0/2663322907 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c680d0530 0x7f1c6814aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 >> 172.21.15.158:0/2663322907 conn(0x7f1c680669f0 msgr2=0x7f1c680d3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:30.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 shutdown_connections 2023-10-16T11:00:30.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.754+0000 7f1c6cb2b700 1 -- 172.21.15.158:0/2663322907 wait complete. 2023-10-16T11:00:30.769 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:00:30.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.889+0000 7fa17d795700 1 Processor -- start 2023-10-16T11:00:30.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.890+0000 7fa17d795700 1 -- start start 2023-10-16T11:00:30.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.890+0000 7fa17d795700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa17815e870 0x7fa1780cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.890+0000 7fa17d795700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa1780cdbe0 0x7fa17806e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.890+0000 7fa17d795700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1780ce160 con 0x7fa17815e490 2023-10-16T11:00:30.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.890+0000 7fa17d795700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17806ccd0 con 0x7fa17815e870 2023-10-16T11:00:30.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.890+0000 7fa17d795700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17806cfb0 con 0x7fa1780cdbe0 2023-10-16T11:00:30.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.890+0000 7fa1767fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa17815e870 0x7fa1780cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1767fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa17815e870 0x7fa1780cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32840/0 (socket says 172.21.15.158:32840) 2023-10-16T11:00:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1767fc700 1 -- 172.21.15.158:0/2247497922 learned_addr learned my addr 172.21.15.158:0/2247497922 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1777fe700 1 --2- 172.21.15.158:0/2247497922 >> v2:172.21.15.172:3300/0 conn(0x7fa1780cdbe0 0x7fa17806e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1767fc700 1 -- 172.21.15.158:0/2247497922 >> v2:172.21.15.172:3300/0 conn(0x7fa1780cdbe0 msgr2=0x7fa17806e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1767fc700 1 --2- 172.21.15.158:0/2247497922 >> v2:172.21.15.172:3300/0 conn(0x7fa1780cdbe0 0x7fa17806e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1767fc700 1 -- 172.21.15.158:0/2247497922 >> v1:172.21.15.172:6789/0 conn(0x7fa17815e490 legacy=0x7fa17806da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1767fc700 1 -- 172.21.15.158:0/2247497922 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17806d290 con 0x7fa17815e870 2023-10-16T11:00:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1777fe700 1 --2- 172.21.15.158:0/2247497922 >> v2:172.21.15.172:3300/0 conn(0x7fa1780cdbe0 0x7fa17806e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa1767fc700 1 --2- 172.21.15.158:0/2247497922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e870 0x7fa1780cd6a0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fa1780cf4c0 tx=0x7fa168009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182165559082659e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa175ffb700 1 -- 172.21.15.158:0/2247497922 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16800b870 con 0x7fa17815e870 2023-10-16T11:00:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa175ffb700 1 -- 172.21.15.158:0/2247497922 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16800b9d0 con 0x7fa17815e870 2023-10-16T11:00:30.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.891+0000 7fa175ffb700 1 -- 172.21.15.158:0/2247497922 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16800bd60 con 0x7fa17815e870 2023-10-16T11:00:30.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 -- 172.21.15.158:0/2247497922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e870 msgr2=0x7fa1780cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 --2- 172.21.15.158:0/2247497922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e870 0x7fa1780cd6a0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fa1780cf4c0 tx=0x7fa168009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 -- 172.21.15.158:0/2247497922 shutdown_connections 2023-10-16T11:00:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 --2- 172.21.15.158:0/2247497922 >> v2:172.21.15.172:3300/0 conn(0x7fa1780cdbe0 0x7fa17806e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 --2- 172.21.15.158:0/2247497922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e870 0x7fa1780cd6a0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 -- 172.21.15.158:0/2247497922 >> 172.21.15.158:0/2247497922 conn(0x7fa1780b15c0 msgr2=0x7fa1780b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 -- 172.21.15.158:0/2247497922 shutdown_connections 2023-10-16T11:00:30.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.892+0000 7fa17d795700 1 -- 172.21.15.158:0/2247497922 wait complete. 2023-10-16T11:00:30.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.893+0000 7fa17d795700 1 Processor -- start 2023-10-16T11:00:30.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.893+0000 7fa17d795700 1 -- start start 2023-10-16T11:00:30.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa17d795700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1780cdbe0 0x7fa178151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa17d795700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e490 0x7fa1781e7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa17d795700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa17815e870 0x7fa1781e8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa17d795700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17815b4d0 con 0x7fa17815e490 2023-10-16T11:00:30.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa17d795700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780ae250 con 0x7fa1780cdbe0 2023-10-16T11:00:30.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa17d795700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780ad6d0 con 0x7fa17815e870 2023-10-16T11:00:30.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e490 0x7fa1781e7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e490 0x7fa1781e7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32844/0 (socket says 172.21.15.158:32844) 2023-10-16T11:00:30.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 -- 172.21.15.158:0/421423066 learned_addr learned my addr 172.21.15.158:0/421423066 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:30.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa176ffd700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1780cdbe0 0x7fa178151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1777fe700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa17815e870 0x7fa1781e8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 -- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa17815e870 msgr2=0x7fa1781e8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa17815e870 0x7fa1781e8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 -- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1780cdbe0 msgr2=0x7fa178151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:30.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1780cdbe0 0x7fa178151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:30.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 -- 172.21.15.158:0/421423066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa168009550 con 0x7fa17815e490 2023-10-16T11:00:30.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.894+0000 7fa1767fc700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e490 0x7fa1781e7bd0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fa168009c10 tx=0x7fa168004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:30.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.895+0000 7fa15f7fe700 1 -- 172.21.15.158:0/421423066 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168004250 con 0x7fa17815e490 2023-10-16T11:00:30.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.895+0000 7fa15f7fe700 1 -- 172.21.15.158:0/421423066 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16801f070 con 0x7fa17815e490 2023-10-16T11:00:30.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.895+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1780ad9b0 con 0x7fa17815e490 2023-10-16T11:00:30.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.895+0000 7fa15f7fe700 1 -- 172.21.15.158:0/421423066 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1680045b0 con 0x7fa17815e490 2023-10-16T11:00:30.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.895+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1780adf70 con 0x7fa17815e490 2023-10-16T11:00:30.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.896+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa178066db0 con 0x7fa17815e490 2023-10-16T11:00:30.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.896+0000 7fa15f7fe700 1 -- 172.21.15.158:0/421423066 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa168021030 con 0x7fa17815e490 2023-10-16T11:00:30.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.896+0000 7fa15f7fe700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa16406af90 0x7fa16406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:30.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.896+0000 7fa15f7fe700 1 -- 172.21.15.158:0/421423066 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fa16808aaf0 con 0x7fa17815e490 2023-10-16T11:00:30.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.896+0000 7fa176ffd700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa16406af90 0x7fa16406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:30.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.897+0000 7fa176ffd700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa16406af90 0x7fa16406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0018a0 tx=0x7fa16c009040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:30.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:30.901+0000 7fa15f7fe700 1 -- 172.21.15.158:0/421423066 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa168055900 con 0x7fa17815e490 2023-10-16T11:00:31.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.097+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa1780b22f0 con 0x7fa17815e490 2023-10-16T11:00:31.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.100+0000 7fa15f7fe700 1 -- 172.21.15.158:0/421423066 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa168058f20 con 0x7fa17815e490 2023-10-16T11:00:31.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa16406af90 msgr2=0x7fa16406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa16406af90 0x7fa16406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0018a0 tx=0x7fa16c009040 comp rx=0 tx=0).stop 2023-10-16T11:00:31.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e490 msgr2=0x7fa1781e7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e490 0x7fa1781e7bd0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fa168009c10 tx=0x7fa168004750 comp rx=0 tx=0).stop 2023-10-16T11:00:31.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 shutdown_connections 2023-10-16T11:00:31.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa17815e870 0x7fa1781e8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa16406af90 0x7fa16406d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1780cdbe0 0x7fa178151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 --2- 172.21.15.158:0/421423066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa17815e490 0x7fa1781e7bd0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 >> 172.21.15.158:0/421423066 conn(0x7fa1780b15c0 msgr2=0x7fa17815fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:31.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 shutdown_connections 2023-10-16T11:00:31.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.102+0000 7fa17d795700 1 -- 172.21.15.158:0/421423066 wait complete. 2023-10-16T11:00:31.115 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:00:31.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.237+0000 7fb0a02f8700 1 Processor -- start 2023-10-16T11:00:31.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.237+0000 7fb0a02f8700 1 -- start start 2023-10-16T11:00:31.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb0a02f8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb098143790 0x7fb09806bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb0a02f8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0980c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb0a02f8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb09806e540 0x7fb09806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb0a02f8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb098143b70 con 0x7fb09806e540 2023-10-16T11:00:31.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb0a02f8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0980c5a60 con 0x7fb098143790 2023-10-16T11:00:31.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb0a02f8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09806ef10 con 0x7fb0980c50a0 2023-10-16T11:00:31.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09e895700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb09806e540 0x7fb09806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09e895700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb09806e540 0x7fb09806e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32854/0 (socket says 172.21.15.158:32854) 2023-10-16T11:00:31.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09e895700 1 -- 172.21.15.158:0/507448908 learned_addr learned my addr 172.21.15.158:0/507448908 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:31.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09e094700 1 --2- 172.21.15.158:0/507448908 >> v2:172.21.15.172:3300/0 conn(0x7fb098143790 0x7fb09806bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09d893700 1 --2- 172.21.15.158:0/507448908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0980c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09e895700 1 -- 172.21.15.158:0/507448908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 msgr2=0x7fb0980c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:31.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09e895700 1 --2- 172.21.15.158:0/507448908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0980c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.238+0000 7fb09e895700 1 -- 172.21.15.158:0/507448908 >> v2:172.21.15.172:3300/0 conn(0x7fb098143790 msgr2=0x7fb09806bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:31.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb09e895700 1 --2- 172.21.15.158:0/507448908 >> v2:172.21.15.172:3300/0 conn(0x7fb098143790 0x7fb09806bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb09e895700 1 -- 172.21.15.158:0/507448908 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09806aa70 con 0x7fb09806e540 2023-10-16T11:00:31.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb09e094700 1 --2- 172.21.15.158:0/507448908 >> v2:172.21.15.172:3300/0 conn(0x7fb098143790 0x7fb09806bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:31.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb09e895700 1 --2- 172.21.15.158:0/507448908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09806e9d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fb0980cd2a0 tx=0x7fb09000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46ca957e21413c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:31.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb09d092700 1 -- 172.21.15.158:0/507448908 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09000e070 con 0x7fb09806e540 2023-10-16T11:00:31.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb09d092700 1 -- 172.21.15.158:0/507448908 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09000b9c0 con 0x7fb09806e540 2023-10-16T11:00:31.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb09d092700 1 -- 172.21.15.158:0/507448908 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09000bd70 con 0x7fb09806e540 2023-10-16T11:00:31.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb0a02f8700 1 -- 172.21.15.158:0/507448908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 msgr2=0x7fb09806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/507448908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09806e9d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fb0980cd2a0 tx=0x7fb09000b330 comp rx=0 tx=0).stop 2023-10-16T11:00:31.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb0a02f8700 1 -- 172.21.15.158:0/507448908 shutdown_connections 2023-10-16T11:00:31.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/507448908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09806e9d0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/507448908 >> v2:172.21.15.172:3300/0 conn(0x7fb098143790 0x7fb09806bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/507448908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0980c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.239+0000 7fb0a02f8700 1 -- 172.21.15.158:0/507448908 >> 172.21.15.158:0/507448908 conn(0x7fb0980cc5b0 msgr2=0x7fb0980cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:31.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.240+0000 7fb0a02f8700 1 -- 172.21.15.158:0/507448908 shutdown_connections 2023-10-16T11:00:31.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.240+0000 7fb0a02f8700 1 -- 172.21.15.158:0/507448908 wait complete. 2023-10-16T11:00:31.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 Processor -- start 2023-10-16T11:00:31.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 -- start start 2023-10-16T11:00:31.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09813e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0981648c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb098143790 0x7fb098164e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09813fba0 con 0x7fb09806e540 2023-10-16T11:00:31.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb098143030 con 0x7fb098143790 2023-10-16T11:00:31.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.241+0000 7fb0a02f8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09813ef20 con 0x7fb0980c50a0 2023-10-16T11:00:31.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09813e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09813e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32856/0 (socket says 172.21.15.158:32856) 2023-10-16T11:00:31.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 -- 172.21.15.158:0/3874035995 learned_addr learned my addr 172.21.15.158:0/3874035995 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:31.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09d893700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0981648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e895700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb098143790 0x7fb098164e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 -- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 msgr2=0x7fb0981648c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0981648c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 -- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb098143790 msgr2=0x7fb098164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb098143790 0x7fb098164e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 -- 172.21.15.158:0/3874035995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090009b30 con 0x7fb09806e540 2023-10-16T11:00:31.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb09e094700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09813e8d0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fb08800c820 tx=0x7fb08800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:31.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb08effd700 1 -- 172.21.15.158:0/3874035995 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088011800 con 0x7fb09806e540 2023-10-16T11:00:31.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb08effd700 1 -- 172.21.15.158:0/3874035995 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb088011960 con 0x7fb09806e540 2023-10-16T11:00:31.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb08effd700 1 -- 172.21.15.158:0/3874035995 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088011d10 con 0x7fb09806e540 2023-10-16T11:00:31.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09813f200 con 0x7fb09806e540 2023-10-16T11:00:31.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.242+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09813f7c0 con 0x7fb09806e540 2023-10-16T11:00:31.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.243+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0980681b0 con 0x7fb09806e540 2023-10-16T11:00:31.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.244+0000 7fb08effd700 1 -- 172.21.15.158:0/3874035995 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb088018360 con 0x7fb09806e540 2023-10-16T11:00:31.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.244+0000 7fb08effd700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fb07c06afc0 0x7fb07c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.244+0000 7fb08effd700 1 -- 172.21.15.158:0/3874035995 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fb088089930 con 0x7fb09806e540 2023-10-16T11:00:31.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.244+0000 7fb09d893700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fb07c06afc0 0x7fb07c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.244+0000 7fb09d893700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fb07c06afc0 0x7fb07c06d480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb09813a230 tx=0x7fb094008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:31.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.248+0000 7fb08effd700 1 -- 172.21.15.158:0/3874035995 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0880545a0 con 0x7fb09806e540 2023-10-16T11:00:31.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.448+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fb0980c5d90 con 0x7fb09806e540 2023-10-16T11:00:31.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.452+0000 7fb08effd700 1 -- 172.21.15.158:0/3874035995 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb088057bc0 con 0x7fb09806e540 2023-10-16T11:00:31.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.454+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fb07c06afc0 msgr2=0x7fb07c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.454+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fb07c06afc0 0x7fb07c06d480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb09813a230 tx=0x7fb094008040 comp rx=0 tx=0).stop 2023-10-16T11:00:31.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.454+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 msgr2=0x7fb09813e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.454+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09813e8d0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fb08800c820 tx=0x7fb08800cbe0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 shutdown_connections 2023-10-16T11:00:31.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fb07c06afc0 0x7fb07c06d480 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb098143790 0x7fb098164e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb09806e540 0x7fb09813e8d0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 --2- 172.21.15.158:0/3874035995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0980c50a0 0x7fb0981648c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 >> 172.21.15.158:0/3874035995 conn(0x7fb0980cc5b0 msgr2=0x7fb098073d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:31.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 shutdown_connections 2023-10-16T11:00:31.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.455+0000 7fb0a02f8700 1 -- 172.21.15.158:0/3874035995 wait complete. 2023-10-16T11:00:31.468 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T11:00:31.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.587+0000 7feb3cf3f700 1 Processor -- start 2023-10-16T11:00:31.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.587+0000 7feb3cf3f700 1 -- start start 2023-10-16T11:00:31.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb3cf3f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 0x7feb3806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb3cf3f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3806e770 con 0x7feb3806f340 2023-10-16T11:00:31.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb3cf3f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb380cbe00 con 0x7feb380cc5d0 2023-10-16T11:00:31.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb3cf3f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb380cf270 con 0x7feb3815dfb0 2023-10-16T11:00:31.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb36d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7feb3806f340 0x7feb38071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33044/0 (socket says 172.21.15.158:33044) 2023-10-16T11:00:31.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb36d9d700 1 -- 172.21.15.158:0/266048208 learned_addr learned my addr 172.21.15.158:0/266048208 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:31.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb35d9b700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 0x7feb3806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842101635 0 0) 0x7feb30001490 con 0x7feb3806f340 2023-10-16T11:00:31.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.588+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb200035b0 con 0x7feb3806f340 2023-10-16T11:00:31.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2087475249 0 0) 0x7feb200035b0 con 0x7feb3806f340 2023-10-16T11:00:31.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 msgr2=0x7feb3806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:31.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 0x7feb3806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 >> v1:172.21.15.172:6789/0 conn(0x7feb380cc5d0 legacy=0x7feb380cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3815e4e0 con 0x7feb3806f340 2023-10-16T11:00:31.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7feb30001b80 con 0x7feb3806f340 2023-10-16T11:00:31.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 >> v1:172.21.15.158:6789/0 conn(0x7feb3806f340 legacy=0x7feb38071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb20003a30 0x7feb20003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb20004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 0x7feb20005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb380cf270 con 0x7feb20004850 2023-10-16T11:00:31.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2000bac0 con 0x7feb20003a30 2023-10-16T11:00:31.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2000bc80 con 0x7feb3815dfb0 2023-10-16T11:00:31.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb35d9b700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb20004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3659c700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb20003a30 0x7feb20003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb36d9d700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 0x7feb20005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb35d9b700 1 -- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 msgr2=0x7feb20005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:31.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb35d9b700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 0x7feb20005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb35d9b700 1 -- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb20003a30 msgr2=0x7feb20003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:31.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb35d9b700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb20003a30 0x7feb20003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb35d9b700 1 -- 172.21.15.158:0/266048208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200035b0 con 0x7feb20004850 2023-10-16T11:00:31.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb36d9d700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815dfb0 0x7feb20005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:31.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb35d9b700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb20004cd0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7feb380cee90 tx=0x7feb20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521222f6012357e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:31.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800e690 con 0x7feb20004850 2023-10-16T11:00:31.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2800e7f0 con 0x7feb20004850 2023-10-16T11:00:31.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.589+0000 7feb3559a700 1 -- 172.21.15.158:0/266048208 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800eb60 con 0x7feb20004850 2023-10-16T11:00:31.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.590+0000 7feb3cf3f700 1 -- 172.21.15.158:0/266048208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 msgr2=0x7feb20004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.590+0000 7feb3cf3f700 1 --2- 172.21.15.158:0/266048208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb20004cd0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7feb380cee90 tx=0x7feb20005880 comp rx=0 tx=0).stop 2023-10-16T11:00:31.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.590+0000 7feb36d9d700 1 -- 172.21.15.158:0/266048208 reap_dead start 2023-10-16T11:00:31.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.590+0000 7feb3cf3f700 1 -- 172.21.15.158:0/266048208 shutdown_connections 2023-10-16T11:00:31.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.590+0000 7feb3cf3f700 1 -- 172.21.15.158:0/266048208 >> 172.21.15.158:0/266048208 conn(0x7feb380d02b0 msgr2=0x7feb380d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:31.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.590+0000 7feb3cf3f700 1 -- 172.21.15.158:0/266048208 shutdown_connections 2023-10-16T11:00:31.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.590+0000 7feb3cf3f700 1 -- 172.21.15.158:0/266048208 wait complete. 2023-10-16T11:00:31.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.591+0000 7feb3cf3f700 1 Processor -- start 2023-10-16T11:00:31.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.591+0000 7feb3cf3f700 1 -- start start 2023-10-16T11:00:31.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3cf3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb3814f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3cf3f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb38155eb0 0x7feb38156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3cf3f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815bb70 0x7feb3815bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3cf3f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb38156870 con 0x7feb20004850 2023-10-16T11:00:31.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3cf3f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3815c530 con 0x7feb38155eb0 2023-10-16T11:00:31.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3cf3f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb380c8600 con 0x7feb3815bb70 2023-10-16T11:00:31.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb3814f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb3814f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32876/0 (socket says 172.21.15.158:32876) 2023-10-16T11:00:31.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3659c700 1 -- 172.21.15.158:0/2892180962 learned_addr learned my addr 172.21.15.158:0/2892180962 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:31.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb35d9b700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb38155eb0 0x7feb38156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb36d9d700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815bb70 0x7feb3815bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.592+0000 7feb3659c700 1 -- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815bb70 msgr2=0x7feb3815bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb3659c700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815bb70 0x7feb3815bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb3659c700 1 -- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb38155eb0 msgr2=0x7feb38156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:31.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb3659c700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb38155eb0 0x7feb38156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:31.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb3659c700 1 -- 172.21.15.158:0/2892180962 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380c8870 con 0x7feb20004850 2023-10-16T11:00:31.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb3659c700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb3814f740 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7feb2c011960 tx=0x7feb2c011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:31.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c001650 con 0x7feb20004850 2023-10-16T11:00:31.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c0052e0 con 0x7feb20004850 2023-10-16T11:00:31.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c003680 con 0x7feb20004850 2023-10-16T11:00:31.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb380c8b50 con 0x7feb20004850 2023-10-16T11:00:31.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.593+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb380c9110 con 0x7feb20004850 2023-10-16T11:00:31.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.594+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb380c9480 con 0x7feb20004850 2023-10-16T11:00:31.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.594+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7feb2c0037e0 con 0x7feb20004850 2023-10-16T11:00:31.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.595+0000 7feb26ffd700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feb1806af90 0x7feb1806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:31.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.595+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7feb2c08b980 con 0x7feb20004850 2023-10-16T11:00:31.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.595+0000 7feb35d9b700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feb1806af90 0x7feb1806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:31.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.595+0000 7feb35d9b700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feb1806af90 0x7feb1806d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feb28000c00 tx=0x7feb28008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:31.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.599+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb2c056780 con 0x7feb20004850 2023-10-16T11:00:31.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.762+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb2c056b50 con 0x7feb20004850 2023-10-16T11:00:31.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:31.878+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7feb380c92a0 con 0x7feb20004850 2023-10-16T11:00:32.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.397+0000 7feb26ffd700 1 -- 172.21.15.158:0/2892180962 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2503) v1 ==== 178+0+0 (secure 0 0 0) 0x7feb2c072690 con 0x7feb20004850 2023-10-16T11:00:32.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feb1806af90 msgr2=0x7feb1806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feb1806af90 0x7feb1806d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feb28000c00 tx=0x7feb28008030 comp rx=0 tx=0).stop 2023-10-16T11:00:32.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 msgr2=0x7feb3814f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb3814f740 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7feb2c011960 tx=0x7feb2c011d20 comp rx=0 tx=0).stop 2023-10-16T11:00:32.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 shutdown_connections 2023-10-16T11:00:32.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3815bb70 0x7feb3815bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb20004850 0x7feb3814f740 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feb1806af90 0x7feb1806d450 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 --2- 172.21.15.158:0/2892180962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb38155eb0 0x7feb38156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 >> 172.21.15.158:0/2892180962 conn(0x7feb380d02b0 msgr2=0x7feb3806fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:32.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 shutdown_connections 2023-10-16T11:00:32.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.400+0000 7feb3cf3f700 1 -- 172.21.15.158:0/2892180962 wait complete. 2023-10-16T11:00:32.415 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-10-16T11:00:32.415 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-10-16T11:00:32.416 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T11:00:32.416 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:32.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.540+0000 7fcf24491700 1 Processor -- start 2023-10-16T11:00:32.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.540+0000 7fcf24491700 1 -- start start 2023-10-16T11:00:32.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf24491700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcf1c0cdee0 0x7fcf1c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf24491700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf1c0b6e10 0x7fcf1c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf24491700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c15c5d0 con 0x7fcf1c0ce2c0 2023-10-16T11:00:32.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf24491700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c15d2c0 con 0x7fcf1c0b6e10 2023-10-16T11:00:32.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf24491700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c15c8b0 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf1c0b6e10 0x7fcf1c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf1c0b6e10 0x7fcf1c1521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32888/0 (socket says 172.21.15.158:32888) 2023-10-16T11:00:32.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 -- 172.21.15.158:0/3283551022 learned_addr learned my addr 172.21.15.158:0/3283551022 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:32.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf2222d700 1 --2- 172.21.15.158:0/3283551022 >> v2:172.21.15.172:3300/0 conn(0x7fcf1c0cdee0 0x7fcf1c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 -- 172.21.15.158:0/3283551022 >> v2:172.21.15.172:3300/0 conn(0x7fcf1c0cdee0 msgr2=0x7fcf1c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:32.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 --2- 172.21.15.158:0/3283551022 >> v2:172.21.15.172:3300/0 conn(0x7fcf1c0cdee0 0x7fcf1c15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 -- 172.21.15.158:0/3283551022 >> v1:172.21.15.172:6789/0 conn(0x7fcf1c0ce2c0 legacy=0x7fcf1c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 -- 172.21.15.158:0/3283551022 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c15cb90 con 0x7fcf1c0b6e10 2023-10-16T11:00:32.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf21a2c700 1 --2- 172.21.15.158:0/3283551022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0b6e10 0x7fcf1c1521a0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0d2470 tx=0x7fcf0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfc4455d7e7259d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:32.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf2122b700 1 -- 172.21.15.158:0/3283551022 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c011830 con 0x7fcf1c0b6e10 2023-10-16T11:00:32.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.541+0000 7fcf2122b700 1 -- 172.21.15.158:0/3283551022 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c011990 con 0x7fcf1c0b6e10 2023-10-16T11:00:32.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf2122b700 1 -- 172.21.15.158:0/3283551022 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c011d20 con 0x7fcf1c0b6e10 2023-10-16T11:00:32.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 -- 172.21.15.158:0/3283551022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0b6e10 msgr2=0x7fcf1c1521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 --2- 172.21.15.158:0/3283551022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0b6e10 0x7fcf1c1521a0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0d2470 tx=0x7fcf0c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 -- 172.21.15.158:0/3283551022 shutdown_connections 2023-10-16T11:00:32.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 --2- 172.21.15.158:0/3283551022 >> v2:172.21.15.172:3300/0 conn(0x7fcf1c0cdee0 0x7fcf1c15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 --2- 172.21.15.158:0/3283551022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0b6e10 0x7fcf1c1521a0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 -- 172.21.15.158:0/3283551022 >> 172.21.15.158:0/3283551022 conn(0x7fcf1c06a8d0 msgr2=0x7fcf1c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:32.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 -- 172.21.15.158:0/3283551022 shutdown_connections 2023-10-16T11:00:32.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.542+0000 7fcf24491700 1 -- 172.21.15.158:0/3283551022 wait complete. 2023-10-16T11:00:32.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.543+0000 7fcf24491700 1 Processor -- start 2023-10-16T11:00:32.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.543+0000 7fcf24491700 1 -- start start 2023-10-16T11:00:32.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf24491700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf1c0b6e10 0x7fcf1c14dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf24491700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0cdee0 0x7fcf1c1538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf24491700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf1c0ce2c0 0x7fcf1c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf24491700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c14e120 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf24491700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0c4f00 con 0x7fcf1c0ce2c0 2023-10-16T11:00:32.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf24491700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0c51e0 con 0x7fcf1c0b6e10 2023-10-16T11:00:32.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0cdee0 0x7fcf1c1538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0cdee0 0x7fcf1c1538d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32892/0 (socket says 172.21.15.158:32892) 2023-10-16T11:00:32.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 -- 172.21.15.158:0/1072964506 learned_addr learned my addr 172.21.15.158:0/1072964506 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:32.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf2222d700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf1c0b6e10 0x7fcf1c14dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf22a2e700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf1c0ce2c0 0x7fcf1c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 -- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf1c0b6e10 msgr2=0x7fcf1c14dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf1c0b6e10 0x7fcf1c14dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 -- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf1c0ce2c0 msgr2=0x7fcf1c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf1c0ce2c0 0x7fcf1c0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf21a2c700 1 -- 172.21.15.158:0/1072964506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c009550 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf22a2e700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf1c0ce2c0 0x7fcf1c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:32.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.544+0000 7fcf2222d700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf1c0b6e10 0x7fcf1c14dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:32.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.545+0000 7fcf21a2c700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0cdee0 0x7fcf1c1538d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c000c00 tx=0x7fcf0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:32.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.545+0000 7fcf12ffd700 1 -- 172.21.15.158:0/1072964506 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c004570 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.545+0000 7fcf12ffd700 1 -- 172.21.15.158:0/1072964506 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c01f070 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.545+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c0c54c0 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.545+0000 7fcf12ffd700 1 -- 172.21.15.158:0/1072964506 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0038e0 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.545+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c0c5a80 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.546+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c0681b0 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.546+0000 7fcf12ffd700 1 -- 172.21.15.158:0/1072964506 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf0c021030 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.546+0000 7fcf12ffd700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcf0006b020 0x7fcf0006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.547+0000 7fcf12ffd700 1 -- 172.21.15.158:0/1072964506 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fcf0c08aed0 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.547+0000 7fcf2222d700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcf0006b020 0x7fcf0006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.547+0000 7fcf2222d700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcf0006b020 0x7fcf0006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009c30 tx=0x7fcf18008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:32.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.551+0000 7fcf12ffd700 1 -- 172.21.15.158:0/1072964506 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf0c055c40 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.744+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf1c0c4960 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.745+0000 7fcf12ffd700 1 -- 172.21.15.158:0/1072964506 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+14919 (secure 0 0 0) 0x7fcf0c015020 con 0x7fcf1c0cdee0 2023-10-16T11:00:32.747 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:32.747 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":163,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:30.390396+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6835/253283745":"2023-10-17T11:00:27.844038+0000","172.21.15.172:6833/253283745":"2023-10-17T11:00:27.844038+0000","172.21.15.158:0/963025365":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/89982933":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/4116889804":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/4154474342":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/3915457102":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/559163437":"2023-10-17T11:00:26.886700+0000","172.21.15.158:6833/2326422563":"2023-10-17T11:00:26.886700+0000","172.21.15.158:6832/2326422563":"2023-10-17T11:00:26.886700+0000","172.21.15.158:0/2640446942":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3643659736":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1187395764":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/2828352389":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6833/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:6832/3458664017":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/3333502274":"2023-10-17T10:04:17.957825+0000","172.21.15.158:0/1458403109":"2023-10-17T10:04:17.957825+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:32.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcf0006b020 msgr2=0x7fcf0006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcf0006b020 0x7fcf0006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009c30 tx=0x7fcf18008040 comp rx=0 tx=0).stop 2023-10-16T11:00:32.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0cdee0 msgr2=0x7fcf1c1538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0cdee0 0x7fcf1c1538d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c000c00 tx=0x7fcf0c004750 comp rx=0 tx=0).stop 2023-10-16T11:00:32.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 shutdown_connections 2023-10-16T11:00:32.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcf0006b020 0x7fcf0006d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf1c0ce2c0 0x7fcf1c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf1c0b6e10 0x7fcf1c14dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 --2- 172.21.15.158:0/1072964506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf1c0cdee0 0x7fcf1c1538d0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 >> 172.21.15.158:0/1072964506 conn(0x7fcf1c06a8d0 msgr2=0x7fcf1c151f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:32.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 shutdown_connections 2023-10-16T11:00:32.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.748+0000 7fcf24491700 1 -- 172.21.15.158:0/1072964506 wait complete. 2023-10-16T11:00:32.761 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:00:32.762 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:00:32.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.883+0000 7f2284452700 1 Processor -- start 2023-10-16T11:00:32.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.883+0000 7f2284452700 1 -- start start 2023-10-16T11:00:32.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f2284452700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f227c15dad0 0x7f227c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f2284452700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227c0d0ae0 con 0x7f227c15deb0 2023-10-16T11:00:32.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f2284452700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227c0d12d0 con 0x7f227c0ccc00 2023-10-16T11:00:32.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f2284452700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227c0cbdf0 con 0x7f227c15dad0 2023-10-16T11:00:32.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f22819ed700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f227c15deb0 0x7f227c0ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33076/0 (socket says 172.21.15.158:33076) 2023-10-16T11:00:32.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f22819ed700 1 -- 172.21.15.158:0/2985597188 learned_addr learned my addr 172.21.15.158:0/2985597188 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:32.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f22821ee700 1 --2- 172.21.15.158:0/2985597188 >> v2:172.21.15.172:3300/0 conn(0x7f227c15dad0 0x7f227c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027304374 0 0) 0x7f227c0d0ae0 con 0x7f227c15deb0 2023-10-16T11:00:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22680035b0 con 0x7f227c15deb0 2023-10-16T11:00:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.884+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867406003 0 0) 0x7f22680035b0 con 0x7f227c15deb0 2023-10-16T11:00:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 >> v2:172.21.15.172:3300/0 conn(0x7f227c15dad0 msgr2=0x7f227c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 --2- 172.21.15.158:0/2985597188 >> v2:172.21.15.172:3300/0 conn(0x7f227c15dad0 0x7f227c0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 >> v1:172.21.15.172:6789/0 conn(0x7f227c0ccc00 legacy=0x7f227c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c0cc280 con 0x7f227c15deb0 2023-10-16T11:00:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f2278001670 con 0x7f227c15deb0 2023-10-16T11:00:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 >> v1:172.21.15.158:6789/0 conn(0x7f227c15deb0 legacy=0x7f227c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268003a00 0x7f2268003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2268004850 0x7f2268004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2268005210 0x7f226800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0cbdf0 con 0x7f2268005210 2023-10-16T11:00:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226800c1b0 con 0x7f2268004850 2023-10-16T11:00:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226800c370 con 0x7f2268003a00 2023-10-16T11:00:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2268005210 0x7f226800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22819ed700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2268004850 0x7f2268004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22821ee700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268003a00 0x7f2268003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 -- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268003a00 msgr2=0x7f2268003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268003a00 0x7f2268003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 -- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2268004850 msgr2=0x7f2268004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2268004850 0x7f2268004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 -- 172.21.15.158:0/2985597188 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22680035b0 con 0x7f2268005210 2023-10-16T11:00:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 -- 172.21.15.158:0/2985597188 reap_dead start 2023-10-16T11:00:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22829ef700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2268005210 0x7f226800ba10 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f227400c190 tx=0x7f2268005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03dabe500d079f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274001db0 con 0x7f2268005210 2023-10-16T11:00:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227400ccf0 con 0x7f2268005210 2023-10-16T11:00:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.885+0000 7f22811ec700 1 -- 172.21.15.158:0/2985597188 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274003c40 con 0x7f2268005210 2023-10-16T11:00:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.886+0000 7f2284452700 1 -- 172.21.15.158:0/2985597188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2268005210 msgr2=0x7f226800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.886+0000 7f2284452700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2268005210 0x7f226800ba10 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f227400c190 tx=0x7f2268005880 comp rx=0 tx=0).stop 2023-10-16T11:00:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.886+0000 7f2284452700 1 -- 172.21.15.158:0/2985597188 shutdown_connections 2023-10-16T11:00:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.886+0000 7f2284452700 1 --2- 172.21.15.158:0/2985597188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2268005210 0x7f226800ba10 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.886+0000 7f2284452700 1 -- 172.21.15.158:0/2985597188 >> 172.21.15.158:0/2985597188 conn(0x7f227c099a10 msgr2=0x7f227c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:32.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.886+0000 7f2284452700 1 -- 172.21.15.158:0/2985597188 shutdown_connections 2023-10-16T11:00:32.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.886+0000 7f2284452700 1 -- 172.21.15.158:0/2985597188 wait complete. 2023-10-16T11:00:32.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.888+0000 7f2284452700 1 Processor -- start 2023-10-16T11:00:32.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.888+0000 7f2284452700 1 -- start start 2023-10-16T11:00:32.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268005210 0x7f227c152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f227c0b51d0 0x7f227c0b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f227c15b4f0 0x7f227c15b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0c9690 con 0x7f227c0b51d0 2023-10-16T11:00:32.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0b5b90 con 0x7f227c15b4f0 2023-10-16T11:00:32.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0c9ab0 con 0x7f2268005210 2023-10-16T11:00:32.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f227c0b51d0 0x7f227c0b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f227c0b51d0 0x7f227c0b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32912/0 (socket says 172.21.15.158:32912) 2023-10-16T11:00:32.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 -- 172.21.15.158:0/3590838652 learned_addr learned my addr 172.21.15.158:0/3590838652 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:32.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22821ee700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268005210 0x7f227c152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22829ef700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f227c15b4f0 0x7f227c15b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 -- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268005210 msgr2=0x7f227c152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268005210 0x7f227c152870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 -- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f227c15b4f0 msgr2=0x7f227c15b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:32.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f227c15b4f0 0x7f227c15b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:32.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 -- 172.21.15.158:0/3590838652 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c0c8a10 con 0x7f227c0b51d0 2023-10-16T11:00:32.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22821ee700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268005210 0x7f227c152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:32.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f22819ed700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f227c0b51d0 0x7f227c0b5650 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f22780072c0 tx=0x7f2278007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:32.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2272ffd700 1 -- 172.21.15.158:0/3590838652 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22780077a0 con 0x7f227c0b51d0 2023-10-16T11:00:32.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2272ffd700 1 -- 172.21.15.158:0/3590838652 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227800f8d0 con 0x7f227c0b51d0 2023-10-16T11:00:32.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227c0c8cf0 con 0x7f227c0b51d0 2023-10-16T11:00:32.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2272ffd700 1 -- 172.21.15.158:0/3590838652 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227800fc80 con 0x7f227c0b51d0 2023-10-16T11:00:32.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.889+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227c0c92b0 con 0x7f227c0b51d0 2023-10-16T11:00:32.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.890+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227c0c97d0 con 0x7f227c0b51d0 2023-10-16T11:00:32.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.891+0000 7f2272ffd700 1 -- 172.21.15.158:0/3590838652 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f227801d460 con 0x7f227c0b51d0 2023-10-16T11:00:32.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.891+0000 7f2272ffd700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f226006b020 0x7f226006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:32.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.891+0000 7f2272ffd700 1 -- 172.21.15.158:0/3590838652 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f22780899d0 con 0x7f227c0b51d0 2023-10-16T11:00:32.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.891+0000 7f22821ee700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f226006b020 0x7f226006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:32.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.892+0000 7f22821ee700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f226006b020 0x7f226006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f226c009be0 tx=0x7f226c008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:32.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:32.896+0000 7f2272ffd700 1 -- 172.21.15.158:0/3590838652 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f227c0c97d0 con 0x7f227c0b51d0 2023-10-16T11:00:33.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.085+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f227c0c97d0 con 0x7f227c0b51d0 2023-10-16T11:00:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.085+0000 7f2272ffd700 1 -- 172.21.15.158:0/3590838652 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) v1 ==== 93+0+10 (secure 0 0 0) 0x7f227c0c97d0 con 0x7f227c0b51d0 2023-10-16T11:00:33.087 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:00:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.087+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f226006b020 msgr2=0x7f226006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.087+0000 7f2284452700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f226006b020 0x7f226006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f226c009be0 tx=0x7f226c008040 comp rx=0 tx=0).stop 2023-10-16T11:00:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f227c0b51d0 msgr2=0x7f227c0b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f227c0b51d0 0x7f227c0b5650 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f22780072c0 tx=0x7f2278007680 comp rx=0 tx=0).stop 2023-10-16T11:00:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 shutdown_connections 2023-10-16T11:00:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f226006b020 0x7f226006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f227c15b4f0 0x7f227c15b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2268005210 0x7f227c152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 --2- 172.21.15.158:0/3590838652 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f227c0b51d0 0x7f227c0b5650 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 >> 172.21.15.158:0/3590838652 conn(0x7f227c099a10 msgr2=0x7f227c0c6460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 shutdown_connections 2023-10-16T11:00:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.088+0000 7f2284452700 1 -- 172.21.15.158:0/3590838652 wait complete. 2023-10-16T11:00:33.101 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T11:00:33.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.220+0000 7f6ee60b7700 1 Processor -- start 2023-10-16T11:00:33.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.220+0000 7f6ee60b7700 1 -- start start 2023-10-16T11:00:33.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.221+0000 7f6ee60b7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6ee015b4a0 0x7f6ee00cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:33.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.221+0000 7f6ee60b7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00cb2c0 0x7f6ee00b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.221+0000 7f6ee60b7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6ee00b1140 0x7f6ee006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.221+0000 7f6ee60b7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee015efa0 con 0x7f6ee015b4a0 2023-10-16T11:00:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.221+0000 7f6ee60b7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00b15d0 con 0x7f6ee00b1140 2023-10-16T11:00:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.221+0000 7f6ee60b7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee015fab0 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.221+0000 7f6edf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6ee015b4a0 0x7f6ee00cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6ee015b4a0 0x7f6ee00cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32918/0 (socket says 172.21.15.158:32918) 2023-10-16T11:00:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 -- 172.21.15.158:0/3607812209 learned_addr learned my addr 172.21.15.158:0/3607812209 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edffff700 1 --2- 172.21.15.158:0/3607812209 >> v2:172.21.15.172:3300/0 conn(0x7f6ee00b1140 0x7f6ee006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edeffd700 1 --2- 172.21.15.158:0/3607812209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00cb2c0 0x7f6ee00b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 -- 172.21.15.158:0/3607812209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00cb2c0 msgr2=0x7f6ee00b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 --2- 172.21.15.158:0/3607812209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00cb2c0 0x7f6ee00b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 -- 172.21.15.158:0/3607812209 >> v2:172.21.15.172:3300/0 conn(0x7f6ee00b1140 msgr2=0x7f6ee006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 --2- 172.21.15.158:0/3607812209 >> v2:172.21.15.172:3300/0 conn(0x7f6ee00b1140 0x7f6ee006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 -- 172.21.15.158:0/3607812209 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee015b880 con 0x7f6ee015b4a0 2023-10-16T11:00:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edffff700 1 --2- 172.21.15.158:0/3607812209 >> v2:172.21.15.172:3300/0 conn(0x7f6ee00b1140 0x7f6ee006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6edf7fe700 1 --2- 172.21.15.158:0/3607812209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee015b4a0 0x7f6ee00cfcc0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00098f0 tx=0x7f6ed0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5347e5fddd7ad18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6ede7fc700 1 -- 172.21.15.158:0/3607812209 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000b860 con 0x7f6ee015b4a0 2023-10-16T11:00:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6ede7fc700 1 -- 172.21.15.158:0/3607812209 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed000b9c0 con 0x7f6ee015b4a0 2023-10-16T11:00:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.222+0000 7f6ede7fc700 1 -- 172.21.15.158:0/3607812209 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000bd50 con 0x7f6ee015b4a0 2023-10-16T11:00:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 -- 172.21.15.158:0/3607812209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee015b4a0 msgr2=0x7f6ee00cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/3607812209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee015b4a0 0x7f6ee00cfcc0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00098f0 tx=0x7f6ed0009c00 comp rx=0 tx=0).stop 2023-10-16T11:00:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 -- 172.21.15.158:0/3607812209 shutdown_connections 2023-10-16T11:00:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/3607812209 >> v2:172.21.15.172:3300/0 conn(0x7f6ee00b1140 0x7f6ee006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/3607812209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee015b4a0 0x7f6ee00cfcc0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/3607812209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00cb2c0 0x7f6ee00b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 -- 172.21.15.158:0/3607812209 >> 172.21.15.158:0/3607812209 conn(0x7f6ee006af00 msgr2=0x7f6ee00cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 -- 172.21.15.158:0/3607812209 shutdown_connections 2023-10-16T11:00:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.223+0000 7f6ee60b7700 1 -- 172.21.15.158:0/3607812209 wait complete. 2023-10-16T11:00:33.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.224+0000 7f6ee60b7700 1 Processor -- start 2023-10-16T11:00:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6ee60b7700 1 -- start start 2023-10-16T11:00:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6ee60b7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00b1140 0x7f6ee014f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6ee60b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee00cb2c0 0x7f6ee00c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6ee60b7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ee015b4a0 0x7f6ee00b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6ee60b7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee015bd40 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6ee60b7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee015e8e0 con 0x7f6ee015b4a0 2023-10-16T11:00:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6ee60b7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee015d030 con 0x7f6ee00b1140 2023-10-16T11:00:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.225+0000 7f6edeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee00cb2c0 0x7f6ee00c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee00cb2c0 0x7f6ee00c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32926/0 (socket says 172.21.15.158:32926) 2023-10-16T11:00:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 -- 172.21.15.158:0/1533698914 learned_addr learned my addr 172.21.15.158:0/1533698914 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:33.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edf7fe700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00b1140 0x7f6ee014f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:33.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edffff700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ee015b4a0 0x7f6ee00b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:33.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 -- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00b1140 msgr2=0x7f6ee014f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:33.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00b1140 0x7f6ee014f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 -- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ee015b4a0 msgr2=0x7f6ee00b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:33.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ee015b4a0 0x7f6ee00b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:33.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 -- 172.21.15.158:0/1533698914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0009550 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edf7fe700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00b1140 0x7f6ee014f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:33.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6edeffd700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee00cb2c0 0x7f6ee00c8150 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800e810 tx=0x7f6ec800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:33.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6ed7fff700 1 -- 172.21.15.158:0/1533698914 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8013610 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6ed7fff700 1 -- 172.21.15.158:0/1533698914 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec8013770 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee015bf90 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6ed7fff700 1 -- 172.21.15.158:0/1533698914 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8013b20 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.226+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee015c550 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.227+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee00681b0 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.228+0000 7f6ed7fff700 1 -- 172.21.15.158:0/1533698914 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ec8013c80 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.228+0000 7f6ed7fff700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f6ec006b020 0x7f6ec006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:33.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.229+0000 7f6ed7fff700 1 -- 172.21.15.158:0/1533698914 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f6ec8089c00 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.229+0000 7f6edf7fe700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f6ec006b020 0x7f6ec006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:33.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.229+0000 7f6edf7fe700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f6ec006b020 0x7f6ec006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0014040 tx=0x7f6ed0008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:33.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.232+0000 7f6ed7fff700 1 -- 172.21.15.158:0/1533698914 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ec8054860 con 0x7f6ee00cb2c0 2023-10-16T11:00:33.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:33.506+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f6ee015f710 con 0x7f6ee00cb2c0 2023-10-16T11:00:34.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.415+0000 7f6ed7fff700 1 -- 172.21.15.158:0/1533698914 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2505) v1 ==== 177+0+0 (secure 0 0 0) 0x7f6ec80210f0 con 0x7f6ee00cb2c0 2023-10-16T11:00:34.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f6ec006b020 msgr2=0x7f6ec006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f6ec006b020 0x7f6ec006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0014040 tx=0x7f6ed0008030 comp rx=0 tx=0).stop 2023-10-16T11:00:34.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee00cb2c0 msgr2=0x7f6ee00c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee00cb2c0 0x7f6ee00c8150 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800e810 tx=0x7f6ec800ebd0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 shutdown_connections 2023-10-16T11:00:34.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f6ec006b020 0x7f6ec006d4e0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0014040 tx=0x7f6ed0008030 comp rx=0 tx=0).stop 2023-10-16T11:00:34.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ee015b4a0 0x7f6ee00b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ee00b1140 0x7f6ee014f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 --2- 172.21.15.158:0/1533698914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ee00cb2c0 0x7f6ee00c8150 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 >> 172.21.15.158:0/1533698914 conn(0x7f6ee006af00 msgr2=0x7f6ee00aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:34.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 shutdown_connections 2023-10-16T11:00:34.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.418+0000 7f6ee60b7700 1 -- 172.21.15.158:0/1533698914 wait complete. 2023-10-16T11:00:34.433 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T11:00:34.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.557+0000 7fd601106700 1 Processor -- start 2023-10-16T11:00:34.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.557+0000 7fd601106700 1 -- start start 2023-10-16T11:00:34.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.557+0000 7fd601106700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:34.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.557+0000 7fd601106700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5fc0cc590 0x7fd5fc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:34.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.557+0000 7fd601106700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5fc0cbd30 con 0x7fd5fc06ae10 2023-10-16T11:00:34.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.557+0000 7fd601106700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc06b280 con 0x7fd5fc0cc590 2023-10-16T11:00:34.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.557+0000 7fd601106700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0cc1c0 con 0x7fd5fc15da40 2023-10-16T11:00:34.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fa59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5fc0cc590 0x7fd5fc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:34.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fb59e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd5fc06ae10 0x7fd5fc0ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33112/0 (socket says 172.21.15.158:33112) 2023-10-16T11:00:34.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fb59e700 1 -- 172.21.15.158:0/198438160 learned_addr learned my addr 172.21.15.158:0/198438160 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:34.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fa59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5fc0cc590 0x7fd5fc06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32932/0 (socket says 172.21.15.158:32932) 2023-10-16T11:00:34.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fad9d700 1 --2- 172.21.15.158:0/198438160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:34.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fa59c700 1 -- 172.21.15.158:0/198438160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 msgr2=0x7fd5fc0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:34.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fa59c700 1 --2- 172.21.15.158:0/198438160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fa59c700 1 -- 172.21.15.158:0/198438160 >> v1:172.21.15.158:6789/0 conn(0x7fd5fc06ae10 legacy=0x7fd5fc0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fa59c700 1 -- 172.21.15.158:0/198438160 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc15de20 con 0x7fd5fc0cc590 2023-10-16T11:00:34.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5f9d9b700 1 -- 172.21.15.158:0/198438160 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001916949 0 0) 0x7fd5fc0cbd30 con 0x7fd5fc06ae10 2023-10-16T11:00:34.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5fa59c700 1 --2- 172.21.15.158:0/198438160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc06a8d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009840 tx=0x7fd5f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec19a05c5c7b748e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:34.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5f9d9b700 1 -- 172.21.15.158:0/198438160 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0011830 con 0x7fd5fc0cc590 2023-10-16T11:00:34.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5f9d9b700 1 -- 172.21.15.158:0/198438160 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f0011990 con 0x7fd5fc0cc590 2023-10-16T11:00:34.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.558+0000 7fd5f9d9b700 1 -- 172.21.15.158:0/198438160 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0011d20 con 0x7fd5fc0cc590 2023-10-16T11:00:34.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 -- 172.21.15.158:0/198438160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 msgr2=0x7fd5fc06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 --2- 172.21.15.158:0/198438160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc06a8d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009840 tx=0x7fd5f0009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:34.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 -- 172.21.15.158:0/198438160 shutdown_connections 2023-10-16T11:00:34.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 --2- 172.21.15.158:0/198438160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 --2- 172.21.15.158:0/198438160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc06a8d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 -- 172.21.15.158:0/198438160 >> 172.21.15.158:0/198438160 conn(0x7fd5fc099a10 msgr2=0x7fd5fc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:34.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 -- 172.21.15.158:0/198438160 shutdown_connections 2023-10-16T11:00:34.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.559+0000 7fd601106700 1 -- 172.21.15.158:0/198438160 wait complete. 2023-10-16T11:00:34.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.560+0000 7fd601106700 1 Processor -- start 2023-10-16T11:00:34.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.560+0000 7fd601106700 1 -- start start 2023-10-16T11:00:34.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd601106700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5fc06ae10 0x7fd5fc15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:34.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd601106700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc0c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:34.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd601106700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:34.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd601106700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc15b150 con 0x7fd5fc0cc590 2023-10-16T11:00:34.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd601106700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1f6570 con 0x7fd5fc06ae10 2023-10-16T11:00:34.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd601106700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1f6850 con 0x7fd5fc15da40 2023-10-16T11:00:34.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc0c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:34.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc0c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32938/0 (socket says 172.21.15.158:32938) 2023-10-16T11:00:34.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 -- 172.21.15.158:0/1648380070 learned_addr learned my addr 172.21.15.158:0/1648380070 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fad9d700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5fc06ae10 0x7fd5fc15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 -- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 msgr2=0x7fd5fc0c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fb59e700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 -- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5fc06ae10 msgr2=0x7fd5fc15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5fc06ae10 0x7fd5fc15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fa59c700 1 -- 172.21.15.158:0/1648380070 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f00094a0 con 0x7fd5fc0cc590 2023-10-16T11:00:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.561+0000 7fd5fad9d700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5fc06ae10 0x7fd5fc15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.562+0000 7fd5fa59c700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc0c6320 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0702f0 tx=0x7fd5f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.562+0000 7fd5e37fe700 1 -- 172.21.15.158:0/1648380070 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f00038a0 con 0x7fd5fc0cc590 2023-10-16T11:00:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.562+0000 7fd5e37fe700 1 -- 172.21.15.158:0/1648380070 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f001f070 con 0x7fd5fc0cc590 2023-10-16T11:00:34.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.562+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc1f6b30 con 0x7fd5fc0cc590 2023-10-16T11:00:34.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.562+0000 7fd5e37fe700 1 -- 172.21.15.158:0/1648380070 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0003c40 con 0x7fd5fc0cc590 2023-10-16T11:00:34.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.562+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc1e6860 con 0x7fd5fc0cc590 2023-10-16T11:00:34.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.563+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc0681b0 con 0x7fd5fc0cc590 2023-10-16T11:00:34.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.563+0000 7fd5e37fe700 1 -- 172.21.15.158:0/1648380070 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd5f0026480 con 0x7fd5fc0cc590 2023-10-16T11:00:34.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.564+0000 7fd5e37fe700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd5e806b020 0x7fd5e806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:34.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.564+0000 7fd5e37fe700 1 -- 172.21.15.158:0/1648380070 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fd5f0021070 con 0x7fd5fc0cc590 2023-10-16T11:00:34.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.564+0000 7fd5fad9d700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd5e806b020 0x7fd5e806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:34.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.564+0000 7fd5fad9d700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd5e806b020 0x7fd5e806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009820 tx=0x7fd5ec008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:34.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.568+0000 7fd5e37fe700 1 -- 172.21.15.158:0/1648380070 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5f00561e0 con 0x7fd5fc0cc590 2023-10-16T11:00:34.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.762+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd5fc0cdc40 con 0x7fd5fc0cc590 2023-10-16T11:00:34.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.765+0000 7fd5e37fe700 1 -- 172.21.15.158:0/1648380070 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd5f0059800 con 0x7fd5fc0cc590 2023-10-16T11:00:34.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd5e806b020 msgr2=0x7fd5e806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd5e806b020 0x7fd5e806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009820 tx=0x7fd5ec008040 comp rx=0 tx=0).stop 2023-10-16T11:00:34.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 msgr2=0x7fd5fc0c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:34.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc0c6320 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0702f0 tx=0x7fd5f0004750 comp rx=0 tx=0).stop 2023-10-16T11:00:34.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 shutdown_connections 2023-10-16T11:00:34.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5fc15da40 0x7fd5fc0c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd5e806b020 0x7fd5e806d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5fc06ae10 0x7fd5fc15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 --2- 172.21.15.158:0/1648380070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5fc0cc590 0x7fd5fc0c6320 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:34.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 >> 172.21.15.158:0/1648380070 conn(0x7fd5fc099a10 msgr2=0x7fd5fc15cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:34.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 shutdown_connections 2023-10-16T11:00:34.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:34.768+0000 7fd601106700 1 -- 172.21.15.158:0/1648380070 wait complete. 2023-10-16T11:00:34.781 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:00:34.782 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:00:34.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:00:34.866 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:00:34.866 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:34.867 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:34.884 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:00:34.885 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:00:34.885 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:34.904 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:00:34.905 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:34.905 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:34.941 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:00:34.942 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:00:34.942 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:34.964 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:00:34.964 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T11:00:34.965 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:00:35.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.101+0000 7fc97cbb0700 1 Processor -- start 2023-10-16T11:00:35.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.101+0000 7fc97cbb0700 1 -- start start 2023-10-16T11:00:35.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc97cbb0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 0x7fc97806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc97cbb0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc978161270 con 0x7fc9780729b0 2023-10-16T11:00:35.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc97cbb0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc978161510 con 0x7fc97806ab40 2023-10-16T11:00:35.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc97cbb0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9781617f0 con 0x7fc97806e5d0 2023-10-16T11:00:35.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc976d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9780729b0 0x7fc97815df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33126/0 (socket says 172.21.15.158:33126) 2023-10-16T11:00:35.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc976d9d700 1 -- 172.21.15.158:0/1159384312 learned_addr learned my addr 172.21.15.158:0/1159384312 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:35.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc975d9b700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 0x7fc97806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.102+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801403395 0 0) 0x7fc978161270 con 0x7fc9780729b0 2023-10-16T11:00:35.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9600035b0 con 0x7fc9780729b0 2023-10-16T11:00:35.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686427550 0 0) 0x7fc978161510 con 0x7fc97806ab40 2023-10-16T11:00:35.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc978161270 con 0x7fc97806ab40 2023-10-16T11:00:35.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347637032 0 0) 0x7fc9600035b0 con 0x7fc9780729b0 2023-10-16T11:00:35.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 msgr2=0x7fc97806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:35.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 0x7fc97806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 >> v1:172.21.15.172:6789/0 conn(0x7fc97806ab40 legacy=0x7fc9780722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978161ad0 con 0x7fc9780729b0 2023-10-16T11:00:35.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc9700019f0 con 0x7fc9780729b0 2023-10-16T11:00:35.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 >> v1:172.21.15.158:6789/0 conn(0x7fc9780729b0 legacy=0x7fc97815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc960003bc0 0x7fc960003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc960004850 0x7fc960004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 0x7fc960005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9781617f0 con 0x7fc960003bc0 2023-10-16T11:00:35.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96000bac0 con 0x7fc960004850 2023-10-16T11:00:35.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96000bc50 con 0x7fc97806e5d0 2023-10-16T11:00:35.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97659c700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc960003bc0 0x7fc960003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc975d9b700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc960004850 0x7fc960004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc976d9d700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 0x7fc960005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97659c700 1 -- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 msgr2=0x7fc960005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:35.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97659c700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc97806e5d0 0x7fc960005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97659c700 1 -- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc960004850 msgr2=0x7fc960004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:35.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97659c700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc960004850 0x7fc960004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97659c700 1 -- 172.21.15.158:0/1159384312 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600035b0 con 0x7fc960003bc0 2023-10-16T11:00:35.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97659c700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc960003bc0 0x7fc960003fc0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fc96c001980 tx=0x7fc960005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6985294ba4a146ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:35.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c001470 con 0x7fc960003bc0 2023-10-16T11:00:35.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.103+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c0052d0 con 0x7fc960003bc0 2023-10-16T11:00:35.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc97559a700 1 -- 172.21.15.158:0/1159384312 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c003680 con 0x7fc960003bc0 2023-10-16T11:00:35.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc97cbb0700 1 -- 172.21.15.158:0/1159384312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc960003bc0 msgr2=0x7fc960003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc97cbb0700 1 --2- 172.21.15.158:0/1159384312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc960003bc0 0x7fc960003fc0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fc96c001980 tx=0x7fc960005880 comp rx=0 tx=0).stop 2023-10-16T11:00:35.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc976d9d700 1 -- 172.21.15.158:0/1159384312 reap_dead start 2023-10-16T11:00:35.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc97cbb0700 1 -- 172.21.15.158:0/1159384312 shutdown_connections 2023-10-16T11:00:35.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc97cbb0700 1 -- 172.21.15.158:0/1159384312 >> 172.21.15.158:0/1159384312 conn(0x7fc9780d02b0 msgr2=0x7fc9780d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:35.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc97cbb0700 1 -- 172.21.15.158:0/1159384312 shutdown_connections 2023-10-16T11:00:35.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.104+0000 7fc97cbb0700 1 -- 172.21.15.158:0/1159384312 wait complete. 2023-10-16T11:00:35.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.106+0000 7fc97cbb0700 1 Processor -- start 2023-10-16T11:00:35.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.106+0000 7fc97cbb0700 1 -- start start 2023-10-16T11:00:35.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc97cbb0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc960003bc0 0x7fc9780aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc97cbb0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9780af380 0x7fc9781f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc97cbb0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9781f2680 0x7fc9780cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc97cbb0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9780c5c20 con 0x7fc9781f2680 2023-10-16T11:00:35.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc97cbb0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9780c5e90 con 0x7fc9780af380 2023-10-16T11:00:35.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc97cbb0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9780c6170 con 0x7fc960003bc0 2023-10-16T11:00:35.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9781f2680 0x7fc9780cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9781f2680 0x7fc9780cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32962/0 (socket says 172.21.15.158:32962) 2023-10-16T11:00:35.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 -- 172.21.15.158:0/226379758 learned_addr learned my addr 172.21.15.158:0/226379758 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:35.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc97659c700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc960003bc0 0x7fc9780aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc975d9b700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9780af380 0x7fc9781f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 -- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc960003bc0 msgr2=0x7fc9780aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc960003bc0 0x7fc9780aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 -- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9780af380 msgr2=0x7fc9781f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9780af380 0x7fc9781f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 -- 172.21.15.158:0/226379758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9780c6450 con 0x7fc9781f2680 2023-10-16T11:00:35.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.107+0000 7fc976d9d700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9781f2680 0x7fc9780cbb10 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fc970000f80 tx=0x7fc970007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:35.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.108+0000 7fc966ffd700 1 -- 172.21.15.158:0/226379758 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97000ec20 con 0x7fc9781f2680 2023-10-16T11:00:35.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.108+0000 7fc966ffd700 1 -- 172.21.15.158:0/226379758 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97000ed80 con 0x7fc9781f2680 2023-10-16T11:00:35.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.108+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9780c6730 con 0x7fc9781f2680 2023-10-16T11:00:35.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.108+0000 7fc966ffd700 1 -- 172.21.15.158:0/226379758 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970013640 con 0x7fc9781f2680 2023-10-16T11:00:35.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.108+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9781f2b10 con 0x7fc9781f2680 2023-10-16T11:00:35.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.109+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9780c6920 con 0x7fc9781f2680 2023-10-16T11:00:35.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.109+0000 7fc966ffd700 1 -- 172.21.15.158:0/226379758 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc9700137a0 con 0x7fc9781f2680 2023-10-16T11:00:35.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.109+0000 7fc966ffd700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc95806b020 0x7fc95806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.110+0000 7fc966ffd700 1 -- 172.21.15.158:0/226379758 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fc970089f40 con 0x7fc9781f2680 2023-10-16T11:00:35.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.110+0000 7fc97659c700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc95806b020 0x7fc95806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.110+0000 7fc97659c700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc95806b020 0x7fc95806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc96c015040 tx=0x7fc96c00a000 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:35.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.114+0000 7fc966ffd700 1 -- 172.21.15.158:0/226379758 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc970054c20 con 0x7fc9781f2680 2023-10-16T11:00:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.350+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9780681b0 con 0x7fc9781f2680 2023-10-16T11:00:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.350+0000 7fc966ffd700 1 -- 172.21.15.158:0/226379758 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fc970058240 con 0x7fc9781f2680 2023-10-16T11:00:35.352 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:35.352 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8422,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/1604199546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1604199546},{"type":"v1","addr":"172.21.15.172:6834","nonce":1604199546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":8800,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2166115210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2166115210},{"type":"v1","addr":"172.21.15.172:6835","nonce":2166115210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/95492242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":95492242},{"type":"v1","addr":"172.21.15.158:6836","nonce":95492242}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9393,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/112424210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":112424210},{"type":"v1","addr":"172.21.15.158:6837","nonce":112424210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[]} 2023-10-16T11:00:35.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc95806b020 msgr2=0x7fc95806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc95806b020 0x7fc95806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc96c015040 tx=0x7fc96c00a000 comp rx=0 tx=0).stop 2023-10-16T11:00:35.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9781f2680 msgr2=0x7fc9780cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9781f2680 0x7fc9780cbb10 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fc970000f80 tx=0x7fc970007c90 comp rx=0 tx=0).stop 2023-10-16T11:00:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 shutdown_connections 2023-10-16T11:00:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9781f2680 0x7fc9780cbb10 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc960003bc0 0x7fc9780aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc95806b020 0x7fc95806d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 --2- 172.21.15.158:0/226379758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9780af380 0x7fc9781f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 >> 172.21.15.158:0/226379758 conn(0x7fc9780d02b0 msgr2=0x7fc9780721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.352+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 shutdown_connections 2023-10-16T11:00:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.353+0000 7fc97cbb0700 1 -- 172.21.15.158:0/226379758 wait complete. 2023-10-16T11:00:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 79 2023-10-16T11:00:35.366 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T11:00:35.366 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T11:00:35.367 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:00:35.367 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:00:35.367 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T11:00:35.367 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T11:00:35.368 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:00:35.368 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:00:35.368 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T11:00:35.368 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T11:00:35.369 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:00:35.369 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:00:35.369 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T11:00:35.369 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T11:00:35.370 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:00:35.370 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:00:35.376 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:00:35.376 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T11:00:35.377 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:00:35.377 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T11:00:35.380 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:00:35.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T11:00:35.382 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:00:35.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T11:00:35.418 INFO:tasks.ceph.mds.a:Started 2023-10-16T11:00:35.418 DEBUG:teuthology.parallel:result is None 2023-10-16T11:00:35.419 INFO:tasks.ceph.mds.c:Started 2023-10-16T11:00:35.419 DEBUG:teuthology.parallel:result is None 2023-10-16T11:00:35.424 INFO:tasks.ceph.mds.b:Started 2023-10-16T11:00:35.424 DEBUG:teuthology.parallel:result is None 2023-10-16T11:00:35.425 INFO:tasks.ceph.mds.d:Started 2023-10-16T11:00:35.426 DEBUG:teuthology.parallel:result is None 2023-10-16T11:00:35.426 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T11:00:35.497 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:00:35.498 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T11:00:35.512 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:00:35.512 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:00:35.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.553+0000 7ffa50d8a700 1 Processor -- start 2023-10-16T11:00:35.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.553+0000 7ffa50d8a700 1 -- start start 2023-10-16T11:00:35.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa50d8a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffa4c06afe0 0x7ffa4c0719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa50d8a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c15dc50 con 0x7ffa4c15d870 2023-10-16T11:00:35.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa50d8a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c06f100 con 0x7ffa4c0cf7c0 2023-10-16T11:00:35.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa50d8a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c0701a0 con 0x7ffa4c06afe0 2023-10-16T11:00:35.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa49d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffa4c06afe0 0x7ffa4c0719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa4a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffa4c15d870 0x7ffa4c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33166/0 (socket says 172.21.15.158:33166) 2023-10-16T11:00:35.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa4a59c700 1 -- 172.21.15.158:0/1588030440 learned_addr learned my addr 172.21.15.158:0/1588030440 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:35.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa4959a700 1 -- 172.21.15.158:0/1588030440 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086289075 0 0) 0x7ffa4c15dc50 con 0x7ffa4c15d870 2023-10-16T11:00:35.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa4959a700 1 -- 172.21.15.158:0/1588030440 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa340035b0 con 0x7ffa4c15d870 2023-10-16T11:00:35.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa49d9b700 1 -- 172.21.15.158:0/1588030440 >> v1:172.21.15.172:6789/0 conn(0x7ffa4c0cf7c0 legacy=0x7ffa4c071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa49d9b700 1 -- 172.21.15.158:0/1588030440 >> v1:172.21.15.158:6789/0 conn(0x7ffa4c15d870 legacy=0x7ffa4c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa49d9b700 1 -- 172.21.15.158:0/1588030440 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c06f3e0 con 0x7ffa4c06afe0 2023-10-16T11:00:35.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa49d9b700 1 --2- 172.21.15.158:0/1588030440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c0719f0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009840 tx=0x7ffa3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355de1217757f239 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:35.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa4959a700 1 -- 172.21.15.158:0/1588030440 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c011830 con 0x7ffa4c06afe0 2023-10-16T11:00:35.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa4959a700 1 -- 172.21.15.158:0/1588030440 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c011990 con 0x7ffa4c06afe0 2023-10-16T11:00:35.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.554+0000 7ffa4959a700 1 -- 172.21.15.158:0/1588030440 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c011d20 con 0x7ffa4c06afe0 2023-10-16T11:00:35.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.555+0000 7ffa50d8a700 1 -- 172.21.15.158:0/1588030440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 msgr2=0x7ffa4c0719f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.555+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/1588030440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c0719f0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009840 tx=0x7ffa3c009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:35.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.555+0000 7ffa50d8a700 1 -- 172.21.15.158:0/1588030440 shutdown_connections 2023-10-16T11:00:35.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.555+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/1588030440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c0719f0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.555+0000 7ffa50d8a700 1 -- 172.21.15.158:0/1588030440 >> 172.21.15.158:0/1588030440 conn(0x7ffa4c06da00 msgr2=0x7ffa4c06dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:35.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.555+0000 7ffa50d8a700 1 -- 172.21.15.158:0/1588030440 shutdown_connections 2023-10-16T11:00:35.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.555+0000 7ffa50d8a700 1 -- 172.21.15.158:0/1588030440 wait complete. 2023-10-16T11:00:35.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.557+0000 7ffa50d8a700 1 Processor -- start 2023-10-16T11:00:35.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.557+0000 7ffa50d8a700 1 -- start start 2023-10-16T11:00:35.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c14bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa4c0cf7c0 0x7ffa4c1605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa4c15d870 0x7ffa4c0d0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c157a70 con 0x7ffa4c06afe0 2023-10-16T11:00:35.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c160f60 con 0x7ffa4c15d870 2023-10-16T11:00:35.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c157c80 con 0x7ffa4c0cf7c0 2023-10-16T11:00:35.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c14bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c14bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:32996/0 (socket says 172.21.15.158:32996) 2023-10-16T11:00:35.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 -- 172.21.15.158:0/2681408480 learned_addr learned my addr 172.21.15.158:0/2681408480 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:35.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa49d9b700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa4c0cf7c0 0x7ffa4c1605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4ad9d700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa4c15d870 0x7ffa4c0d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 -- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa4c0cf7c0 msgr2=0x7ffa4c1605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa4c0cf7c0 0x7ffa4c1605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 -- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa4c15d870 msgr2=0x7ffa4c0d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:35.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa4c15d870 0x7ffa4c0d0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:35.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 -- 172.21.15.158:0/2681408480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3c0094a0 con 0x7ffa4c06afe0 2023-10-16T11:00:35.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4ad9d700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa4c15d870 0x7ffa4c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:35.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa49d9b700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa4c0cf7c0 0x7ffa4c1605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:35.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa4a59c700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c14bca0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7ffa40001eb0 tx=0x7ffa4000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:35.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa3affd700 1 -- 172.21.15.158:0/2681408480 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40005d70 con 0x7ffa4c06afe0 2023-10-16T11:00:35.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa3affd700 1 -- 172.21.15.158:0/2681408480 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa400070d0 con 0x7ffa4c06afe0 2023-10-16T11:00:35.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c157f60 con 0x7ffa4c06afe0 2023-10-16T11:00:35.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa3affd700 1 -- 172.21.15.158:0/2681408480 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40007460 con 0x7ffa4c06afe0 2023-10-16T11:00:35.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.558+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c160ae0 con 0x7ffa4c06afe0 2023-10-16T11:00:35.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.559+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c0681b0 con 0x7ffa4c06afe0 2023-10-16T11:00:35.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.560+0000 7ffa3affd700 1 -- 172.21.15.158:0/2681408480 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffa400075c0 con 0x7ffa4c06afe0 2023-10-16T11:00:35.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.560+0000 7ffa3affd700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ffa2c06b020 0x7ffa2c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:35.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.560+0000 7ffa3affd700 1 -- 172.21.15.158:0/2681408480 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6403+0+0 (secure 0 0 0) 0x7ffa40088f60 con 0x7ffa4c06afe0 2023-10-16T11:00:35.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.560+0000 7ffa49d9b700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ffa2c06b020 0x7ffa2c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:35.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.561+0000 7ffa49d9b700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ffa2c06b020 0x7ffa2c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0090b0 tx=0x7ffa3c008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:35.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.565+0000 7ffa3affd700 1 -- 172.21.15.158:0/2681408480 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa40053c40 con 0x7ffa4c06afe0 2023-10-16T11:00:35.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:35.773+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffa4c1565e0 con 0x7ffa4c06afe0 2023-10-16T11:00:36.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.788+0000 7ffa3affd700 1 -- 172.21.15.158:0/2681408480 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v168) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffa40017080 con 0x7ffa4c06afe0 2023-10-16T11:00:36.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ffa2c06b020 msgr2=0x7ffa2c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ffa2c06b020 0x7ffa2c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0090b0 tx=0x7ffa3c008030 comp rx=0 tx=0).stop 2023-10-16T11:00:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 msgr2=0x7ffa4c14bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c14bca0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7ffa40001eb0 tx=0x7ffa4000c860 comp rx=0 tx=0).stop 2023-10-16T11:00:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 shutdown_connections 2023-10-16T11:00:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ffa2c06b020 0x7ffa2c06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa4c15d870 0x7ffa4c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa4c06afe0 0x7ffa4c14bca0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 --2- 172.21.15.158:0/2681408480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa4c0cf7c0 0x7ffa4c1605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 >> 172.21.15.158:0/2681408480 conn(0x7ffa4c06da00 msgr2=0x7ffa4c0724c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 shutdown_connections 2023-10-16T11:00:36.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.791+0000 7ffa50d8a700 1 -- 172.21.15.158:0/2681408480 wait complete. 2023-10-16T11:00:36.795 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T11:00:36.807 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T11:00:36.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.932+0000 7fcdf04d8700 1 Processor -- start 2023-10-16T11:00:36.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.932+0000 7fcdf04d8700 1 -- start start 2023-10-16T11:00:36.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.933+0000 7fcdf04d8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 0x7fcde8072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.933+0000 7fcdf04d8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde806f850 con 0x7fcde8072360 2023-10-16T11:00:36.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.933+0000 7fcdf04d8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80b4260 con 0x7fcde806f470 2023-10-16T11:00:36.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.933+0000 7fcdf04d8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde8072d50 con 0x7fcde806afe0 2023-10-16T11:00:36.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcdeda73700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcde8072360 0x7fcde806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33186/0 (socket says 172.21.15.158:33186) 2023-10-16T11:00:36.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcdeda73700 1 -- 172.21.15.158:0/4048808124 learned_addr learned my addr 172.21.15.158:0/4048808124 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:36.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcdeea75700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 0x7fcde8072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200188117 0 0) 0x7fcde806f850 con 0x7fcde8072360 2023-10-16T11:00:36.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd40035b0 con 0x7fcde8072360 2023-10-16T11:00:36.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220022357 0 0) 0x7fcdd40035b0 con 0x7fcde8072360 2023-10-16T11:00:36.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 msgr2=0x7fcde8072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:36.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 0x7fcde8072810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 >> v1:172.21.15.172:6789/0 conn(0x7fcde806f470 legacy=0x7fcde806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:36.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80b4540 con 0x7fcde8072360 2023-10-16T11:00:36.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcde4001670 con 0x7fcde8072360 2023-10-16T11:00:36.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 >> v1:172.21.15.158:6789/0 conn(0x7fcde8072360 legacy=0x7fcde806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:36.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdd4003a30 0x7fcdd4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 0x7fcdd4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd4004d90 0x7fcdd4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde8072d50 con 0x7fcdd4003a30 2023-10-16T11:00:36.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd400ba40 con 0x7fcdd4004d90 2023-10-16T11:00:36.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd400bc00 con 0x7fcde806afe0 2023-10-16T11:00:36.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcdee274700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdd4003a30 0x7fcdd4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcdeda73700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 0x7fcdd4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.934+0000 7fcdeea75700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd4004d90 0x7fcdd4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdee274700 1 -- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 msgr2=0x7fcdd4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:36.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdee274700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcde806afe0 0x7fcdd4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdee274700 1 -- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd4004d90 msgr2=0x7fcdd4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:36.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdee274700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd4004d90 0x7fcdd4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdee274700 1 -- 172.21.15.158:0/4048808124 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd40035b0 con 0x7fcdd4003a30 2023-10-16T11:00:36.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdee274700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdd4003a30 0x7fcdd4003e10 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80019e0 tx=0x7fcdd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd56c6a1c2bd724c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:36.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8016070 con 0x7fcdd4003a30 2023-10-16T11:00:36.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd8001d50 con 0x7fcdd4003a30 2023-10-16T11:00:36.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcded272700 1 -- 172.21.15.158:0/4048808124 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8006af0 con 0x7fcdd4003a30 2023-10-16T11:00:36.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdf04d8700 1 -- 172.21.15.158:0/4048808124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdd4003a30 msgr2=0x7fcdd4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:36.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdf04d8700 1 --2- 172.21.15.158:0/4048808124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdd4003a30 0x7fcdd4003e10 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80019e0 tx=0x7fcdd4005880 comp rx=0 tx=0).stop 2023-10-16T11:00:36.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.935+0000 7fcdeea75700 1 -- 172.21.15.158:0/4048808124 reap_dead start 2023-10-16T11:00:36.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.936+0000 7fcdf04d8700 1 -- 172.21.15.158:0/4048808124 shutdown_connections 2023-10-16T11:00:36.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.936+0000 7fcdf04d8700 1 -- 172.21.15.158:0/4048808124 >> 172.21.15.158:0/4048808124 conn(0x7fcde80ccda0 msgr2=0x7fcde80cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:36.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.936+0000 7fcdf04d8700 1 -- 172.21.15.158:0/4048808124 shutdown_connections 2023-10-16T11:00:36.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.936+0000 7fcdf04d8700 1 -- 172.21.15.158:0/4048808124 wait complete. 2023-10-16T11:00:36.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.937+0000 7fcdf04d8700 1 Processor -- start 2023-10-16T11:00:36.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.937+0000 7fcdf04d8700 1 -- start start 2023-10-16T11:00:36.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd4003a30 0x7fcde80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcde815c530 0x7fcde81f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcde81f6ee0 0x7fcde81f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80c73f0 con 0x7fcde81f6ee0 2023-10-16T11:00:36.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80c4790 con 0x7fcde815c530 2023-10-16T11:00:36.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80c4a70 con 0x7fcdd4003a30 2023-10-16T11:00:36.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcde81f6ee0 0x7fcde81f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcde81f6ee0 0x7fcde81f7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33024/0 (socket says 172.21.15.158:33024) 2023-10-16T11:00:36.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 -- 172.21.15.158:0/1041174859 learned_addr learned my addr 172.21.15.158:0/1041174859 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:36.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdee274700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd4003a30 0x7fcde80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeda73700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcde815c530 0x7fcde81f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 -- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd4003a30 msgr2=0x7fcde80cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:36.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd4003a30 0x7fcde80cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 -- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcde815c530 msgr2=0x7fcde81f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:36.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcde815c530 0x7fcde81f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:36.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 -- 172.21.15.158:0/1041174859 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80b0e30 con 0x7fcde81f6ee0 2023-10-16T11:00:36.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdee274700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd4003a30 0x7fcde80cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:36.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdeea75700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcde81f6ee0 0x7fcde81f7360 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fcde806dd60 tx=0x7fcde0011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:36.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcddeffd700 1 -- 172.21.15.158:0/1041174859 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde000e040 con 0x7fcde81f6ee0 2023-10-16T11:00:36.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcddeffd700 1 -- 172.21.15.158:0/1041174859 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde000f040 con 0x7fcde81f6ee0 2023-10-16T11:00:36.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde80b1110 con 0x7fcde81f6ee0 2023-10-16T11:00:36.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcddeffd700 1 -- 172.21.15.158:0/1041174859 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde000fdb0 con 0x7fcde81f6ee0 2023-10-16T11:00:36.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.938+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde80cfc90 con 0x7fcde81f6ee0 2023-10-16T11:00:36.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.939+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde80b1630 con 0x7fcde81f6ee0 2023-10-16T11:00:36.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.940+0000 7fcddeffd700 1 -- 172.21.15.158:0/1041174859 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcde0022460 con 0x7fcde81f6ee0 2023-10-16T11:00:36.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.940+0000 7fcddeffd700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcdcc06b020 0x7fcdcc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:36.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.940+0000 7fcddeffd700 1 -- 172.21.15.158:0/1041174859 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5629+0+0 (secure 0 0 0) 0x7fcde0089d40 con 0x7fcde81f6ee0 2023-10-16T11:00:36.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.940+0000 7fcdee274700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcdcc06b020 0x7fcdcc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:36.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.940+0000 7fcdee274700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcdcc06b020 0x7fcdcc06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8014040 tx=0x7fcdd8009030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:36.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:36.944+0000 7fcddeffd700 1 -- 172.21.15.158:0/1041174859 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcde80b1630 con 0x7fcde81f6ee0 2023-10-16T11:00:37.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.213+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcde80b1630 con 0x7fcde81f6ee0 2023-10-16T11:00:37.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.213+0000 7fcddeffd700 1 -- 172.21.15.158:0/1041174859 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v43) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcde80b1630 con 0x7fcde81f6ee0 2023-10-16T11:00:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:37.215 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:00:37.215 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T11:00:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T11:00:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T11:00:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.216 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:00:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:00:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.217 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T11:00:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T11:00:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.218 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:00:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:00:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.219 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T11:00:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T11:00:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:00:37.220 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:00:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T11:00:37.221 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T11:00:37.222 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.222 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.222 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:00:37.222 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:00:37.222 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.223 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.223 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.223 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.223 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T11:00:37.223 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T11:00:37.223 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.224 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.224 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.224 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.224 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.224 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.225 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.225 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T11:00:37.225 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T11:00:37.225 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.225 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.225 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.226 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.226 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.226 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.226 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.226 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T11:00:37.227 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T11:00:37.227 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.227 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.227 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.227 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.228 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.228 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.228 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.228 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T11:00:37.228 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T11:00:37.228 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.229 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.229 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.229 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.229 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.229 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.230 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.230 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T11:00:37.230 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T11:00:37.230 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.230 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.230 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.231 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.231 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.231 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.231 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.231 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T11:00:37.232 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T11:00:37.232 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.232 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.232 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.232 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.232 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.233 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.233 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.233 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T11:00:37.233 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T11:00:37.233 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.234 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.234 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.234 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.234 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.234 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.235 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.235 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T11:00:37.235 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T11:00:37.235 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.235 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:00:37.235 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:00:37.236 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.236 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.236 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.236 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.236 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T11:00:37.237 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T11:00:37.237 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.237 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.237 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T11:00:37.237 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T11:00:37.237 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.238 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.238 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.238 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T11:00:37.238 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T11:00:37.238 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.239 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.239 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:00:37.239 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:00:37.239 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:00:37.239 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.240 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.240 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.240 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T11:00:37.240 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T11:00:37.240 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.240 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.241 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:00:37.241 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:00:37.241 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:00:37.241 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.241 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.241 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.242 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T11:00:37.242 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T11:00:37.242 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.242 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:00:37.242 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:00:37.243 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:00:37.243 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:00:37.243 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.243 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.243 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.243 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T11:00:37.244 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T11:00:37.244 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.244 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:00:37.244 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:00:37.244 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:00:37.245 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.245 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.245 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.245 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.245 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T11:00:37.246 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T11:00:37.246 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.246 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:00:37.246 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:00:37.246 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:00:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T11:00:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T11:00:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:00:37.248 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:00:37.248 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:00:37.248 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:00:37.248 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.248 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:37.249 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T11:00:37.249 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:00:37.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcdcc06b020 msgr2=0x7fcdcc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcdcc06b020 0x7fcdcc06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8014040 tx=0x7fcdd8009030 comp rx=0 tx=0).stop 2023-10-16T11:00:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcde81f6ee0 msgr2=0x7fcde81f7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcde81f6ee0 0x7fcde81f7360 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fcde806dd60 tx=0x7fcde0011f80 comp rx=0 tx=0).stop 2023-10-16T11:00:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 shutdown_connections 2023-10-16T11:00:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcde81f6ee0 0x7fcde81f7360 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd4003a30 0x7fcde80cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fcdcc06b020 0x7fcdcc06d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 --2- 172.21.15.158:0/1041174859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcde815c530 0x7fcde81f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 >> 172.21.15.158:0/1041174859 conn(0x7fcde80ccda0 msgr2=0x7fcde80cbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 shutdown_connections 2023-10-16T11:00:37.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.215+0000 7fcdf04d8700 1 -- 172.21.15.158:0/1041174859 wait complete. 2023-10-16T11:00:37.253 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:37.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.373+0000 7f1f36bb6700 1 Processor -- start 2023-10-16T11:00:37.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.373+0000 7f1f36bb6700 1 -- start start 2023-10-16T11:00:37.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.373+0000 7f1f36bb6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f3015d7b0 0x7f1f30161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.373+0000 7f1f36bb6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3015db90 con 0x7f1f300cf760 2023-10-16T11:00:37.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.373+0000 7f1f36bb6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3006bea0 con 0x7f1f300b3770 2023-10-16T11:00:37.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.373+0000 7f1f36bb6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f301621e0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f35153700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1f300cf760 0x7f1f300cbad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33208/0 (socket says 172.21.15.158:33208) 2023-10-16T11:00:37.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f35153700 1 -- 172.21.15.158:0/518849267 learned_addr learned my addr 172.21.15.158:0/518849267 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:37.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f34952700 1 --2- 172.21.15.158:0/518849267 >> v2:172.21.15.172:3300/0 conn(0x7f1f3015d7b0 0x7f1f30161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f2f7fe700 1 -- 172.21.15.158:0/518849267 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676873093 0 0) 0x7f1f3015db90 con 0x7f1f300cf760 2023-10-16T11:00:37.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f2f7fe700 1 -- 172.21.15.158:0/518849267 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f200035b0 con 0x7f1f300cf760 2023-10-16T11:00:37.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f2f7fe700 1 -- 172.21.15.158:0/518849267 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604771400 0 0) 0x7f1f200035b0 con 0x7f1f300cf760 2023-10-16T11:00:37.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f34952700 1 -- 172.21.15.158:0/518849267 >> v1:172.21.15.172:6789/0 conn(0x7f1f300b3770 legacy=0x7f1f300cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f34952700 1 -- 172.21.15.158:0/518849267 >> v1:172.21.15.158:6789/0 conn(0x7f1f300cf760 legacy=0x7f1f300cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f34952700 1 -- 172.21.15.158:0/518849267 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3006c140 con 0x7f1f3015d7b0 2023-10-16T11:00:37.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f34952700 1 --2- 172.21.15.158:0/518849267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f3015d7b0 0x7f1f30161ca0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009840 tx=0x7f1f24009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfaf23ea08043319 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:37.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f2f7fe700 1 -- 172.21.15.158:0/518849267 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f280019f0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.374+0000 7f1f2f7fe700 1 -- 172.21.15.158:0/518849267 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f24004030 con 0x7f1f3015d7b0 2023-10-16T11:00:37.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f2f7fe700 1 -- 172.21.15.158:0/518849267 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f240043c0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f36bb6700 1 -- 172.21.15.158:0/518849267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f3015d7b0 msgr2=0x7f1f30161ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/518849267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f3015d7b0 0x7f1f30161ca0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009840 tx=0x7f1f24009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:37.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f36bb6700 1 -- 172.21.15.158:0/518849267 shutdown_connections 2023-10-16T11:00:37.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/518849267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f3015d7b0 0x7f1f30161ca0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f36bb6700 1 -- 172.21.15.158:0/518849267 >> 172.21.15.158:0/518849267 conn(0x7f1f3006a8d0 msgr2=0x7f1f3006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:37.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f36bb6700 1 -- 172.21.15.158:0/518849267 shutdown_connections 2023-10-16T11:00:37.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.375+0000 7f1f36bb6700 1 -- 172.21.15.158:0/518849267 wait complete. 2023-10-16T11:00:37.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.376+0000 7f1f36bb6700 1 Processor -- start 2023-10-16T11:00:37.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.376+0000 7f1f36bb6700 1 -- start start 2023-10-16T11:00:37.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f36bb6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f300b3770 0x7f1f301f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f36bb6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f300cf760 0x7f1f300b6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f36bb6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f3015d7b0 0x7f1f300b6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f36bb6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3015b260 con 0x7f1f3015d7b0 2023-10-16T11:00:37.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f36bb6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f301f41f0 con 0x7f1f300cf760 2023-10-16T11:00:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f36bb6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300c89a0 con 0x7f1f300b3770 2023-10-16T11:00:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f35153700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f3015d7b0 0x7f1f300b6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f35153700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f3015d7b0 0x7f1f300b6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33038/0 (socket says 172.21.15.158:33038) 2023-10-16T11:00:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f35153700 1 -- 172.21.15.158:0/3879614970 learned_addr learned my addr 172.21.15.158:0/3879614970 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f34952700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f300b3770 0x7f1f301f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.377+0000 7f1f2ffff700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f300cf760 0x7f1f300b6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f35153700 1 -- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f300b3770 msgr2=0x7f1f301f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f35153700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f300b3770 0x7f1f301f3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f35153700 1 -- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f300cf760 msgr2=0x7f1f300b6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f35153700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f300cf760 0x7f1f300b6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f35153700 1 -- 172.21.15.158:0/3879614970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240094a0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f2ffff700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f300cf760 0x7f1f300b6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f35153700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f3015d7b0 0x7f1f300b6880 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f1f28001560 tx=0x7f1f2800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f2d7fa700 1 -- 172.21.15.158:0/3879614970 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2800ab50 con 0x7f1f3015d7b0 2023-10-16T11:00:37.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f2d7fa700 1 -- 172.21.15.158:0/3879614970 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2800acb0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f300c8c80 con 0x7f1f3015d7b0 2023-10-16T11:00:37.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f2d7fa700 1 -- 172.21.15.158:0/3879614970 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28007310 con 0x7f1f3015d7b0 2023-10-16T11:00:37.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.378+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f300c9240 con 0x7f1f3015d7b0 2023-10-16T11:00:37.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.379+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f300681b0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.379+0000 7f1f2d7fa700 1 -- 172.21.15.158:0/3879614970 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f28007470 con 0x7f1f3015d7b0 2023-10-16T11:00:37.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.380+0000 7f1f2d7fa700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1f1406b020 0x7f1f1406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.380+0000 7f1f2d7fa700 1 -- 172.21.15.158:0/3879614970 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f1f28089530 con 0x7f1f3015d7b0 2023-10-16T11:00:37.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.380+0000 7f1f34952700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1f1406b020 0x7f1f1406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.380+0000 7f1f34952700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1f1406b020 0x7f1f1406d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f240090b0 tx=0x7f1f24013040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:37.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.384+0000 7f1f2d7fa700 1 -- 172.21.15.158:0/3879614970 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f280545b0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.581+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f3015cfe0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.582+0000 7f1f2d7fa700 1 -- 172.21.15.158:0/3879614970 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+13774 (secure 0 0 0) 0x7f1f28057bd0 con 0x7f1f3015d7b0 2023-10-16T11:00:37.584 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:37.584 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":168,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:36.787470+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.584+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1f1406b020 msgr2=0x7f1f1406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.584+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1f1406b020 0x7f1f1406d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f240090b0 tx=0x7f1f24013040 comp rx=0 tx=0).stop 2023-10-16T11:00:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.584+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f3015d7b0 msgr2=0x7f1f300b6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.584+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f3015d7b0 0x7f1f300b6880 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f1f28001560 tx=0x7f1f2800cf90 comp rx=0 tx=0).stop 2023-10-16T11:00:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 shutdown_connections 2023-10-16T11:00:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f3015d7b0 0x7f1f300b6880 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f300b3770 0x7f1f301f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f1f1406b020 0x7f1f1406d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 --2- 172.21.15.158:0/3879614970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f300cf760 0x7f1f300b6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 >> 172.21.15.158:0/3879614970 conn(0x7f1f3006a8d0 msgr2=0x7f1f300cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 shutdown_connections 2023-10-16T11:00:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.585+0000 7f1f36bb6700 1 -- 172.21.15.158:0/3879614970 wait complete. 2023-10-16T11:00:37.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:00:37.599 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:00:37.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.722+0000 7ff6f5ede700 1 Processor -- start 2023-10-16T11:00:37.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.722+0000 7ff6f5ede700 1 -- start start 2023-10-16T11:00:37.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.722+0000 7ff6f5ede700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6f015dad0 0x7ff6f00cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.722+0000 7ff6f5ede700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff6f015deb0 0x7ff6f00ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.722+0000 7ff6f5ede700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00d0ae0 con 0x7ff6f00cea20 2023-10-16T11:00:37.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.722+0000 7ff6f5ede700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00cee50 con 0x7ff6f015dad0 2023-10-16T11:00:37.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.722+0000 7ff6f5ede700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00d12d0 con 0x7ff6f015deb0 2023-10-16T11:00:37.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6f015dad0 0x7ff6f00cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6f015dad0 0x7ff6f00cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33042/0 (socket says 172.21.15.158:33042) 2023-10-16T11:00:37.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 -- 172.21.15.158:0/1387023474 learned_addr learned my addr 172.21.15.158:0/1387023474 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 -- 172.21.15.158:0/1387023474 >> v2:172.21.15.172:3300/0 conn(0x7ff6f015deb0 msgr2=0x7ff6f00ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 --2- 172.21.15.158:0/1387023474 >> v2:172.21.15.172:3300/0 conn(0x7ff6f015deb0 0x7ff6f00ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 -- 172.21.15.158:0/1387023474 >> v1:172.21.15.172:6789/0 conn(0x7ff6f00cea20 legacy=0x7ff6f00c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 -- 172.21.15.158:0/1387023474 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00cbdf0 con 0x7ff6f015dad0 2023-10-16T11:00:37.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ef7fe700 1 --2- 172.21.15.158:0/1387023474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015dad0 0x7ff6f00cdfa0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009e50 tx=0x7ff6e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf3f373b41b2812 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ee7fc700 1 -- 172.21.15.158:0/1387023474 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e800e070 con 0x7ff6f015dad0 2023-10-16T11:00:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ee7fc700 1 -- 172.21.15.158:0/1387023474 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e8003a20 con 0x7ff6f015dad0 2023-10-16T11:00:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.723+0000 7ff6ee7fc700 1 -- 172.21.15.158:0/1387023474 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8003dd0 con 0x7ff6f015dad0 2023-10-16T11:00:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 -- 172.21.15.158:0/1387023474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015dad0 msgr2=0x7ff6f00cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/1387023474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015dad0 0x7ff6f00cdfa0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009e50 tx=0x7ff6e800b330 comp rx=0 tx=0).stop 2023-10-16T11:00:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 -- 172.21.15.158:0/1387023474 shutdown_connections 2023-10-16T11:00:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/1387023474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015dad0 0x7ff6f00cdfa0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/1387023474 >> v2:172.21.15.172:3300/0 conn(0x7ff6f015deb0 0x7ff6f00ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 -- 172.21.15.158:0/1387023474 >> 172.21.15.158:0/1387023474 conn(0x7ff6f0099a10 msgr2=0x7ff6f00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 -- 172.21.15.158:0/1387023474 shutdown_connections 2023-10-16T11:00:37.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.724+0000 7ff6f5ede700 1 -- 172.21.15.158:0/1387023474 wait complete. 2023-10-16T11:00:37.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.725+0000 7ff6f5ede700 1 Processor -- start 2023-10-16T11:00:37.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.725+0000 7ff6f5ede700 1 -- start start 2023-10-16T11:00:37.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6f5ede700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6f00cea20 0x7ff6f0151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6f5ede700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6f015dad0 0x7ff6f014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6f5ede700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015deb0 0x7ff6f01553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6f5ede700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00ca6a0 con 0x7ff6f015deb0 2023-10-16T11:00:37.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6f5ede700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f0152e60 con 0x7ff6f00cea20 2023-10-16T11:00:37.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6f5ede700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f0152410 con 0x7ff6f015dad0 2023-10-16T11:00:37.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015deb0 0x7ff6f01553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015deb0 0x7ff6f01553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33052/0 (socket says 172.21.15.158:33052) 2023-10-16T11:00:37.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 -- 172.21.15.158:0/21621722 learned_addr learned my addr 172.21.15.158:0/21621722 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:37.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6ef7fe700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6f00cea20 0x7ff6f0151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6eeffd700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6f015dad0 0x7ff6f014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 -- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6f015dad0 msgr2=0x7ff6f014d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6f015dad0 0x7ff6f014d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 -- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6f00cea20 msgr2=0x7ff6f0151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6f00cea20 0x7ff6f0151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 -- 172.21.15.158:0/21621722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8009b30 con 0x7ff6f015deb0 2023-10-16T11:00:37.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6eeffd700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6f015dad0 0x7ff6f014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:37.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.726+0000 7ff6effff700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015deb0 0x7ff6f01553d0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000c490 tx=0x7ff6e0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:37.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.727+0000 7ff6d7fff700 1 -- 172.21.15.158:0/21621722 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00054e0 con 0x7ff6f015deb0 2023-10-16T11:00:37.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.727+0000 7ff6d7fff700 1 -- 172.21.15.158:0/21621722 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e0003680 con 0x7ff6f015deb0 2023-10-16T11:00:37.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.727+0000 7ff6d7fff700 1 -- 172.21.15.158:0/21621722 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0003a30 con 0x7ff6f015deb0 2023-10-16T11:00:37.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.727+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f00c9fe0 con 0x7ff6f015deb0 2023-10-16T11:00:37.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.727+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f0152100 con 0x7ff6f015deb0 2023-10-16T11:00:37.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.728+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f00681b0 con 0x7ff6f015deb0 2023-10-16T11:00:37.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.728+0000 7ff6d7fff700 1 -- 172.21.15.158:0/21621722 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff6e0003b90 con 0x7ff6f015deb0 2023-10-16T11:00:37.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.728+0000 7ff6d7fff700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff6dc06b020 0x7ff6dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:37.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.728+0000 7ff6d7fff700 1 -- 172.21.15.158:0/21621722 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5629+0+0 (secure 0 0 0) 0x7ff6e0088680 con 0x7ff6f015deb0 2023-10-16T11:00:37.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.729+0000 7ff6ef7fe700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff6dc06b020 0x7ff6dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:37.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.729+0000 7ff6ef7fe700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff6dc06b020 0x7ff6dc06d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8015040 tx=0x7ff6e8009000 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:37.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.733+0000 7ff6d7fff700 1 -- 172.21.15.158:0/21621722 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6e0053670 con 0x7ff6f015deb0 2023-10-16T11:00:37.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.925+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6f00cca90 con 0x7ff6f015deb0 2023-10-16T11:00:37.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.925+0000 7ff6d7fff700 1 -- 172.21.15.158:0/21621722 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6e0056c90 con 0x7ff6f015deb0 2023-10-16T11:00:37.927 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:00:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff6dc06b020 msgr2=0x7ff6dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff6dc06b020 0x7ff6dc06d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8015040 tx=0x7ff6e8009000 comp rx=0 tx=0).stop 2023-10-16T11:00:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015deb0 msgr2=0x7ff6f01553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015deb0 0x7ff6f01553d0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000c490 tx=0x7ff6e0001e70 comp rx=0 tx=0).stop 2023-10-16T11:00:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 shutdown_connections 2023-10-16T11:00:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6f015deb0 0x7ff6f01553d0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff6dc06b020 0x7ff6dc06d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6f00cea20 0x7ff6f0151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 --2- 172.21.15.158:0/21621722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6f015dad0 0x7ff6f014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 >> 172.21.15.158:0/21621722 conn(0x7ff6f0099a10 msgr2=0x7ff6f006efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 shutdown_connections 2023-10-16T11:00:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:37.928+0000 7ff6f5ede700 1 -- 172.21.15.158:0/21621722 wait complete. 2023-10-16T11:00:37.941 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:00:38.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.061+0000 7f940dbaf700 1 Processor -- start 2023-10-16T11:00:38.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.061+0000 7f940dbaf700 1 -- start start 2023-10-16T11:00:38.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.061+0000 7f940dbaf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f940815dfb0 0x7f940806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.061+0000 7f940dbaf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f940806f340 0x7f940806f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.061+0000 7f940dbaf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940806e770 con 0x7f94080cc5d0 2023-10-16T11:00:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.061+0000 7f940dbaf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940806fd30 con 0x7f940815dfb0 2023-10-16T11:00:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.061+0000 7f940dbaf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94080cf270 con 0x7f940806f340 2023-10-16T11:00:38.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94077fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f94080cc5d0 0x7f94080cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33232/0 (socket says 172.21.15.158:33232) 2023-10-16T11:00:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94077fe700 1 -- 172.21.15.158:0/2089040385 learned_addr learned my addr 172.21.15.158:0/2089040385 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583160110 0 0) 0x7f93f8001110 con 0x7f94080cc5d0 2023-10-16T11:00:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f00035b0 con 0x7f94080cc5d0 2023-10-16T11:00:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f9406ffd700 1 --2- 172.21.15.158:0/2089040385 >> v2:172.21.15.172:3300/0 conn(0x7f940815dfb0 0x7f940806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f9407fff700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f940806f340 0x7f940806f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2163806657 0 0) 0x7f93f00035b0 con 0x7f94080cc5d0 2023-10-16T11:00:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f940806f340 msgr2=0x7f940806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f940806f340 0x7f940806f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 >> v2:172.21.15.172:3300/0 conn(0x7f940815dfb0 msgr2=0x7f940806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:38.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 --2- 172.21.15.158:0/2089040385 >> v2:172.21.15.172:3300/0 conn(0x7f940815dfb0 0x7f940806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080cbe00 con 0x7f94080cc5d0 2023-10-16T11:00:38.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f93f8001800 con 0x7f94080cc5d0 2023-10-16T11:00:38.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 >> v1:172.21.15.158:6789/0 conn(0x7f94080cc5d0 legacy=0x7f94080cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93f0003a00 0x7f93f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f93f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93f0005210 0x7f93f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940806fd30 con 0x7f93f0004850 2023-10-16T11:00:38.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94080cf270 con 0x7f93f0003a00 2023-10-16T11:00:38.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f000bf80 con 0x7f93f0005210 2023-10-16T11:00:38.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f9407fff700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f940806f340 0x7f940806f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:38.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f9406ffd700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f93f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f94077fe700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93f0003a00 0x7f93f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.062+0000 7f9407fff700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93f0005210 0x7f93f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9406ffd700 1 -- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93f0005210 msgr2=0x7f93f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:38.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9406ffd700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93f0005210 0x7f93f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9406ffd700 1 -- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93f0003a00 msgr2=0x7f93f0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:38.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9406ffd700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93f0003a00 0x7f93f0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9406ffd700 1 -- 172.21.15.158:0/2089040385 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f00035b0 con 0x7f93f0004850 2023-10-16T11:00:38.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f94077fe700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93f0003a00 0x7f93f0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:38.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9407fff700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93f0005210 0x7f93f000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:38.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9406ffd700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f93f0004cd0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f940815e410 tx=0x7f93f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c22b6b445c60e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:38.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00e660 con 0x7f93f0004850 2023-10-16T11:00:38.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc00e7c0 con 0x7f93f0004850 2023-10-16T11:00:38.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f94067fc700 1 -- 172.21.15.158:0/2089040385 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00eb30 con 0x7f93f0004850 2023-10-16T11:00:38.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f940dbaf700 1 -- 172.21.15.158:0/2089040385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 msgr2=0x7f93f0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f940dbaf700 1 --2- 172.21.15.158:0/2089040385 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f93f0004cd0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f940815e410 tx=0x7f93f0005880 comp rx=0 tx=0).stop 2023-10-16T11:00:38.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f9407fff700 1 -- 172.21.15.158:0/2089040385 reap_dead start 2023-10-16T11:00:38.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f940dbaf700 1 -- 172.21.15.158:0/2089040385 shutdown_connections 2023-10-16T11:00:38.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.063+0000 7f940dbaf700 1 -- 172.21.15.158:0/2089040385 >> 172.21.15.158:0/2089040385 conn(0x7f94080d02b0 msgr2=0x7f94080d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:38.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.064+0000 7f940dbaf700 1 -- 172.21.15.158:0/2089040385 shutdown_connections 2023-10-16T11:00:38.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.064+0000 7f940dbaf700 1 -- 172.21.15.158:0/2089040385 wait complete. 2023-10-16T11:00:38.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.065+0000 7f940dbaf700 1 Processor -- start 2023-10-16T11:00:38.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.065+0000 7f940dbaf700 1 -- start start 2023-10-16T11:00:38.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f94081f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94080a89b0 0x7f94080c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94080a8e30 0x7f9408158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94080b47c0 con 0x7f93f0004850 2023-10-16T11:00:38.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94081f4270 con 0x7f94080a8e30 2023-10-16T11:00:38.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94081f4550 con 0x7f94080a89b0 2023-10-16T11:00:38.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f94081f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f94081f0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33068/0 (socket says 172.21.15.158:33068) 2023-10-16T11:00:38.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 -- 172.21.15.158:0/3452910503 learned_addr learned my addr 172.21.15.158:0/3452910503 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f9406ffd700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94080a89b0 0x7f94080c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 -- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94080a89b0 msgr2=0x7f94080c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f9407fff700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94080a8e30 0x7f9408158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94080a89b0 0x7f94080c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 -- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94080a8e30 msgr2=0x7f9408158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94080a8e30 0x7f9408158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 -- 172.21.15.158:0/3452910503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94081f4830 con 0x7f93f0004850 2023-10-16T11:00:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f94077fe700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f94081f0b60 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f93f800eaf0 tx=0x7f93f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f93effff700 1 -- 172.21.15.158:0/3452910503 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f800c7b0 con 0x7f93f0004850 2023-10-16T11:00:38.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f93effff700 1 -- 172.21.15.158:0/3452910503 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f800c910 con 0x7f93f0004850 2023-10-16T11:00:38.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f93effff700 1 -- 172.21.15.158:0/3452910503 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f800cc80 con 0x7f93f0004850 2023-10-16T11:00:38.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94081f4b10 con 0x7f93f0004850 2023-10-16T11:00:38.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.066+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94081f10a0 con 0x7f93f0004850 2023-10-16T11:00:38.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.067+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94080681b0 con 0x7f93f0004850 2023-10-16T11:00:38.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.068+0000 7f93effff700 1 -- 172.21.15.158:0/3452910503 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f93f801d460 con 0x7f93f0004850 2023-10-16T11:00:38.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.068+0000 7f93effff700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f93f406b020 0x7f93f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.068+0000 7f9406ffd700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f93f406b020 0x7f93f406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.068+0000 7f93effff700 1 -- 172.21.15.158:0/3452910503 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f93f80894c0 con 0x7f93f0004850 2023-10-16T11:00:38.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.069+0000 7f9406ffd700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f93f406b020 0x7f93f406d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f94080b1d00 tx=0x7f93fc008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:38.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.073+0000 7f93effff700 1 -- 172.21.15.158:0/3452910503 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93f80544b0 con 0x7f93f0004850 2023-10-16T11:00:38.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.267+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f94080b6230 con 0x7f93f0004850 2023-10-16T11:00:38.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.268+0000 7f93effff700 1 -- 172.21.15.158:0/3452910503 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v168) v1 ==== 84+0+65 (secure 0 0 0) 0x7f93f8087e60 con 0x7f93f0004850 2023-10-16T11:00:38.269 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:38.269 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:00:38.270 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:38.270 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:00:38.270 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:00:38.270 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:38.270 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:00:38.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f93f406b020 msgr2=0x7f93f406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f93f406b020 0x7f93f406d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f94080b1d00 tx=0x7f93fc008030 comp rx=0 tx=0).stop 2023-10-16T11:00:38.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 msgr2=0x7f94081f0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f94081f0b60 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f93f800eaf0 tx=0x7f93f800eeb0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 shutdown_connections 2023-10-16T11:00:38.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f93f406b020 0x7f93f406d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94080a8e30 0x7f9408158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93f0004850 0x7f94081f0b60 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 --2- 172.21.15.158:0/3452910503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94080a89b0 0x7f94080c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 >> 172.21.15.158:0/3452910503 conn(0x7f94080d02b0 msgr2=0x7f940806d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:38.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 shutdown_connections 2023-10-16T11:00:38.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.270+0000 7f940dbaf700 1 -- 172.21.15.158:0/3452910503 wait complete. 2023-10-16T11:00:38.283 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T11:00:38.284 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T11:00:38.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.407+0000 7f9b62bb6700 1 Processor -- start 2023-10-16T11:00:38.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.407+0000 7f9b62bb6700 1 -- start start 2023-10-16T11:00:38.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b62bb6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b5c15da40 0x7f9b5c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b62bb6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c06afe0 0x7f9b5c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b62bb6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5c0cbd30 con 0x7f9b5c0cc590 2023-10-16T11:00:38.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b62bb6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5c0cc1c0 con 0x7f9b5c15da40 2023-10-16T11:00:38.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b62bb6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c15de20 con 0x7f9b5c06afe0 2023-10-16T11:00:38.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b60952700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b5c15da40 0x7f9b5c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b5bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9b5c0cc590 0x7f9b5c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33252/0 (socket says 172.21.15.158:33252) 2023-10-16T11:00:38.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b5bfff700 1 -- 172.21.15.158:0/2770625339 learned_addr learned my addr 172.21.15.158:0/2770625339 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:38.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b61153700 1 --2- 172.21.15.158:0/2770625339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c06afe0 0x7f9b5c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b60952700 1 -- 172.21.15.158:0/2770625339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c06afe0 msgr2=0x7f9b5c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:38.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b60952700 1 --2- 172.21.15.158:0/2770625339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c06afe0 0x7f9b5c0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b60952700 1 -- 172.21.15.158:0/2770625339 >> v1:172.21.15.158:6789/0 conn(0x7f9b5c0cc590 legacy=0x7f9b5c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b5b7fe700 1 -- 172.21.15.158:0/2770625339 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935546162 0 0) 0x7f9b5c0cbd30 con 0x7f9b5c0cc590 2023-10-16T11:00:38.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b60952700 1 -- 172.21.15.158:0/2770625339 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0ce260 con 0x7f9b5c15da40 2023-10-16T11:00:38.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b60952700 1 --2- 172.21.15.158:0/2770625339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c15da40 0x7f9b5c0caa20 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009840 tx=0x7f9b50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6a3ecf7ede1c25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:38.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b5b7fe700 1 -- 172.21.15.158:0/2770625339 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50011830 con 0x7f9b5c15da40 2023-10-16T11:00:38.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b5b7fe700 1 -- 172.21.15.158:0/2770625339 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b50011990 con 0x7f9b5c15da40 2023-10-16T11:00:38.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.408+0000 7f9b5b7fe700 1 -- 172.21.15.158:0/2770625339 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50011d20 con 0x7f9b5c15da40 2023-10-16T11:00:38.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 -- 172.21.15.158:0/2770625339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c15da40 msgr2=0x7f9b5c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/2770625339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c15da40 0x7f9b5c0caa20 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009840 tx=0x7f9b50009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 -- 172.21.15.158:0/2770625339 shutdown_connections 2023-10-16T11:00:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/2770625339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c15da40 0x7f9b5c0caa20 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/2770625339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c06afe0 0x7f9b5c0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 -- 172.21.15.158:0/2770625339 >> 172.21.15.158:0/2770625339 conn(0x7f9b5c099a10 msgr2=0x7f9b5c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:38.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 -- 172.21.15.158:0/2770625339 shutdown_connections 2023-10-16T11:00:38.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.409+0000 7f9b62bb6700 1 -- 172.21.15.158:0/2770625339 wait complete. 2023-10-16T11:00:38.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.410+0000 7f9b62bb6700 1 Processor -- start 2023-10-16T11:00:38.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.410+0000 7f9b62bb6700 1 -- start start 2023-10-16T11:00:38.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b62bb6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b5c06afe0 0x7f9b5c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b62bb6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c0cc590 0x7f9b5c1e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b62bb6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c15da40 0x7f9b5c0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b62bb6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0c5b70 con 0x7f9b5c0cc590 2023-10-16T11:00:38.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b62bb6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0c52a0 con 0x7f9b5c06afe0 2023-10-16T11:00:38.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b62bb6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1e5390 con 0x7f9b5c15da40 2023-10-16T11:00:38.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c0cc590 0x7f9b5c1e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c0cc590 0x7f9b5c1e7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33082/0 (socket says 172.21.15.158:33082) 2023-10-16T11:00:38.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 -- 172.21.15.158:0/1433619080 learned_addr learned my addr 172.21.15.158:0/1433619080 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:38.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b60952700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b5c06afe0 0x7f9b5c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b61153700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c15da40 0x7f9b5c0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 -- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c15da40 msgr2=0x7f9b5c0b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c15da40 0x7f9b5c0b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 -- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b5c06afe0 msgr2=0x7f9b5c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:38.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b5c06afe0 0x7f9b5c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:38.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.411+0000 7f9b5bfff700 1 -- 172.21.15.158:0/1433619080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b500094a0 con 0x7f9b5c0cc590 2023-10-16T11:00:38.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.412+0000 7f9b61153700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c15da40 0x7f9b5c0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:38.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.412+0000 7f9b5bfff700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c0cc590 0x7f9b5c1e7a00 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c540 tx=0x7f9b4800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:38.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.412+0000 7f9b597fa700 1 -- 172.21.15.158:0/1433619080 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48004c80 con 0x7f9b5c0cc590 2023-10-16T11:00:38.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.412+0000 7f9b597fa700 1 -- 172.21.15.158:0/1433619080 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b48004de0 con 0x7f9b5c0cc590 2023-10-16T11:00:38.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.412+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c0c54b0 con 0x7f9b5c0cc590 2023-10-16T11:00:38.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.412+0000 7f9b597fa700 1 -- 172.21.15.158:0/1433619080 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48007310 con 0x7f9b5c0cc590 2023-10-16T11:00:38.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.412+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c0cab50 con 0x7f9b5c0cc590 2023-10-16T11:00:38.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.413+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c0c5cb0 con 0x7f9b5c0cc590 2023-10-16T11:00:38.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.413+0000 7f9b597fa700 1 -- 172.21.15.158:0/1433619080 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9b48007470 con 0x7f9b5c0cc590 2023-10-16T11:00:38.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.414+0000 7f9b597fa700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f9b4006b020 0x7f9b4006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:38.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.414+0000 7f9b597fa700 1 -- 172.21.15.158:0/1433619080 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f9b48089460 con 0x7f9b5c0cc590 2023-10-16T11:00:38.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.414+0000 7f9b60952700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f9b4006b020 0x7f9b4006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:38.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.414+0000 7f9b60952700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f9b4006b020 0x7f9b4006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9b500090b0 tx=0x7f9b50008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:38.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.418+0000 7f9b597fa700 1 -- 172.21.15.158:0/1433619080 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b5c0c5cb0 con 0x7f9b5c0cc590 2023-10-16T11:00:38.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:38.612+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9b5c0c5cb0 con 0x7f9b5c0cc590 2023-10-16T11:00:39.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.432+0000 7f9b597fa700 1 -- 172.21.15.158:0/1433619080 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v169) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9b5c0c5cb0 con 0x7f9b5c0cc590 2023-10-16T11:00:39.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f9b4006b020 msgr2=0x7f9b4006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:39.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f9b4006b020 0x7f9b4006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9b500090b0 tx=0x7f9b50008030 comp rx=0 tx=0).stop 2023-10-16T11:00:39.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c0cc590 msgr2=0x7f9b5c1e7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:39.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c0cc590 0x7f9b5c1e7a00 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c540 tx=0x7f9b4800c850 comp rx=0 tx=0).stop 2023-10-16T11:00:39.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 shutdown_connections 2023-10-16T11:00:39.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b5c15da40 0x7f9b5c0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:39.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f9b4006b020 0x7f9b4006d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:39.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b5c06afe0 0x7f9b5c0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:39.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 --2- 172.21.15.158:0/1433619080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b5c0cc590 0x7f9b5c1e7a00 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:39.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.435+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 >> 172.21.15.158:0/1433619080 conn(0x7f9b5c099a10 msgr2=0x7f9b5c15ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:39.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.436+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 shutdown_connections 2023-10-16T11:00:39.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.436+0000 7f9b62bb6700 1 -- 172.21.15.158:0/1433619080 wait complete. 2023-10-16T11:00:39.442 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T11:00:39.454 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:00:39.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.588+0000 7fa4cf75b700 1 Processor -- start 2023-10-16T11:00:39.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.588+0000 7fa4cf75b700 1 -- start start 2023-10-16T11:00:39.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.589+0000 7fa4cf75b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa4c806f610 0x7fa4c80aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:39.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.589+0000 7fa4cf75b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c815e090 con 0x7fa4c806bff0 2023-10-16T11:00:39.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.589+0000 7fa4cf75b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c815e370 con 0x7fa4c8072170 2023-10-16T11:00:39.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.589+0000 7fa4cf75b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c815ed10 con 0x7fa4c806f610 2023-10-16T11:00:39.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.589+0000 7fa4cdcf8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa4c806bff0 0x7fa4c8150c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33266/0 (socket says 172.21.15.158:33266) 2023-10-16T11:00:39.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.589+0000 7fa4cdcf8700 1 -- 172.21.15.158:0/2905399664 learned_addr learned my addr 172.21.15.158:0/2905399664 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:39.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4cd4f7700 1 --2- 172.21.15.158:0/2905399664 >> v2:172.21.15.172:3300/0 conn(0x7fa4c806f610 0x7fa4c80aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:39.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010769844 0 0) 0x7fa4c815e090 con 0x7fa4c806bff0 2023-10-16T11:00:39.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b40035b0 con 0x7fa4c806bff0 2023-10-16T11:00:39.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517207899 0 0) 0x7fa4c815e370 con 0x7fa4c8072170 2023-10-16T11:00:39.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c815e090 con 0x7fa4c8072170 2023-10-16T11:00:39.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3689081086 0 0) 0x7fa4b40035b0 con 0x7fa4c806bff0 2023-10-16T11:00:39.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4cd4f7700 1 -- 172.21.15.158:0/2905399664 >> v1:172.21.15.172:6789/0 conn(0x7fa4c8072170 legacy=0x7fa4c806b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:39.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4cd4f7700 1 -- 172.21.15.158:0/2905399664 >> v1:172.21.15.158:6789/0 conn(0x7fa4c806bff0 legacy=0x7fa4c8150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:39.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4cd4f7700 1 -- 172.21.15.158:0/2905399664 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c815e650 con 0x7fa4c806f610 2023-10-16T11:00:39.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4cd4f7700 1 --2- 172.21.15.158:0/2905399664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 0x7fa4c80aedc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009840 tx=0x7fa4b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=689b88d3d3fb7b2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:39.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c00019f0 con 0x7fa4c806f610 2023-10-16T11:00:39.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b8004030 con 0x7fa4c806f610 2023-10-16T11:00:39.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.590+0000 7fa4bffff700 1 -- 172.21.15.158:0/2905399664 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80043c0 con 0x7fa4c806f610 2023-10-16T11:00:39.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.591+0000 7fa4cf75b700 1 -- 172.21.15.158:0/2905399664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 msgr2=0x7fa4c80aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:39.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.591+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/2905399664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 0x7fa4c80aedc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009840 tx=0x7fa4b8009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:39.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.591+0000 7fa4cf75b700 1 -- 172.21.15.158:0/2905399664 shutdown_connections 2023-10-16T11:00:39.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.591+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/2905399664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 0x7fa4c80aedc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:39.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.591+0000 7fa4cf75b700 1 -- 172.21.15.158:0/2905399664 >> 172.21.15.158:0/2905399664 conn(0x7fa4c80710f0 msgr2=0x7fa4c8071500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:39.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.591+0000 7fa4cf75b700 1 -- 172.21.15.158:0/2905399664 shutdown_connections 2023-10-16T11:00:39.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.591+0000 7fa4cf75b700 1 -- 172.21.15.158:0/2905399664 wait complete. 2023-10-16T11:00:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.592+0000 7fa4cf75b700 1 Processor -- start 2023-10-16T11:00:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.592+0000 7fa4cf75b700 1 -- start start 2023-10-16T11:00:39.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cf75b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4c806bff0 0x7fa4c815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cf75b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 0x7fa4c814e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cf75b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c8072170 0x7fa4c80cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cf75b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c814eb00 con 0x7fa4c8072170 2023-10-16T11:00:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cf75b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c815d840 con 0x7fa4c806f610 2023-10-16T11:00:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cf75b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80ce010 con 0x7fa4c806bff0 2023-10-16T11:00:39.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c8072170 0x7fa4c80cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c8072170 0x7fa4c80cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33100/0 (socket says 172.21.15.158:33100) 2023-10-16T11:00:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 -- 172.21.15.158:0/3577725077 learned_addr learned my addr 172.21.15.158:0/3577725077 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cccf6700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 0x7fa4c814e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cd4f7700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4c806bff0 0x7fa4c815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 -- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4c806bff0 msgr2=0x7fa4c815c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4c806bff0 0x7fa4c815c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 -- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 msgr2=0x7fa4c814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 0x7fa4c814e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cdcf8700 1 -- 172.21.15.158:0/3577725077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b80094a0 con 0x7fa4c8072170 2023-10-16T11:00:39.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.593+0000 7fa4cd4f7700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4c806bff0 0x7fa4c815c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.594+0000 7fa4cdcf8700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c8072170 0x7fa4c80cdad0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000cbc0 tx=0x7fa4c000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.594+0000 7fa4bdffb700 1 -- 172.21.15.158:0/3577725077 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000abb0 con 0x7fa4c8072170 2023-10-16T11:00:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.594+0000 7fa4bdffb700 1 -- 172.21.15.158:0/3577725077 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c000ad30 con 0x7fa4c8072170 2023-10-16T11:00:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.594+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c815ccc0 con 0x7fa4c8072170 2023-10-16T11:00:39.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.594+0000 7fa4bdffb700 1 -- 172.21.15.158:0/3577725077 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000aea0 con 0x7fa4c8072170 2023-10-16T11:00:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.594+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c815d280 con 0x7fa4c8072170 2023-10-16T11:00:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.595+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c80681b0 con 0x7fa4c8072170 2023-10-16T11:00:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.595+0000 7fa4bdffb700 1 -- 172.21.15.158:0/3577725077 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa4c0007310 con 0x7fa4c8072170 2023-10-16T11:00:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.595+0000 7fa4bdffb700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa4ac06b020 0x7fa4ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.596+0000 7fa4bdffb700 1 -- 172.21.15.158:0/3577725077 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5999+0+0 (secure 0 0 0) 0x7fa4c0089650 con 0x7fa4c8072170 2023-10-16T11:00:39.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.596+0000 7fa4cd4f7700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa4ac06b020 0x7fa4ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:39.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.596+0000 7fa4cd4f7700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa4ac06b020 0x7fa4ac06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8000c00 tx=0x7fa4b8013040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:39.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.600+0000 7fa4bdffb700 1 -- 172.21.15.158:0/3577725077 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa4c0054560 con 0x7fa4c8072170 2023-10-16T11:00:39.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:39.790+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa4c80b24b0 con 0x7fa4c8072170 2023-10-16T11:00:40.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.434+0000 7fa4bdffb700 1 -- 172.21.15.158:0/3577725077 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v170) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa4c0057b80 con 0x7fa4c8072170 2023-10-16T11:00:40.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa4ac06b020 msgr2=0x7fa4ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:40.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa4ac06b020 0x7fa4ac06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8000c00 tx=0x7fa4b8013040 comp rx=0 tx=0).stop 2023-10-16T11:00:40.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c8072170 msgr2=0x7fa4c80cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:40.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c8072170 0x7fa4c80cdad0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000cbc0 tx=0x7fa4c000cf80 comp rx=0 tx=0).stop 2023-10-16T11:00:40.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 shutdown_connections 2023-10-16T11:00:40.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c8072170 0x7fa4c80cdad0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4c806bff0 0x7fa4c815c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa4ac06b020 0x7fa4ac06d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 --2- 172.21.15.158:0/3577725077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c806f610 0x7fa4c814e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.437+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 >> 172.21.15.158:0/3577725077 conn(0x7fa4c80710f0 msgr2=0x7fa4c80b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:40.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.439+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 shutdown_connections 2023-10-16T11:00:40.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.444+0000 7fa4cf75b700 1 -- 172.21.15.158:0/3577725077 wait complete. 2023-10-16T11:00:40.453 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T11:00:40.465 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T11:00:40.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.588+0000 7fc9b2872700 1 Processor -- start 2023-10-16T11:00:40.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.588+0000 7fc9b2872700 1 -- start start 2023-10-16T11:00:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9b2872700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 0x7fc9ac0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9b2872700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9ac06afe0 0x7fc9ac0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9b2872700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ac0cbd30 con 0x7fc9ac0cc590 2023-10-16T11:00:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9b2872700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ac0cc1c0 con 0x7fc9ac06afe0 2023-10-16T11:00:40.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9b2872700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac15de20 con 0x7fc9ac15da40 2023-10-16T11:00:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9ab7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9ac0cc590 0x7fc9ac06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33280/0 (socket says 172.21.15.158:33280) 2023-10-16T11:00:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9ab7fe700 1 -- 172.21.15.158:0/3251013091 learned_addr learned my addr 172.21.15.158:0/3251013091 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9abfff700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 0x7fc9ac0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/3251013091 >> v2:172.21.15.172:3300/0 conn(0x7fc9ac06afe0 0x7fc9ac0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872635183 0 0) 0x7fc9ac0cbd30 con 0x7fc9ac0cc590 2023-10-16T11:00:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.589+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9940035b0 con 0x7fc9ac0cc590 2023-10-16T11:00:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 802044163 0 0) 0x7fc9940035b0 con 0x7fc9ac0cc590 2023-10-16T11:00:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 msgr2=0x7fc9ac0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 0x7fc9ac0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 >> v2:172.21.15.172:3300/0 conn(0x7fc9ac06afe0 msgr2=0x7fc9ac0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 --2- 172.21.15.158:0/3251013091 >> v2:172.21.15.172:3300/0 conn(0x7fc9ac06afe0 0x7fc9ac0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac0ce260 con 0x7fc9ac0cc590 2023-10-16T11:00:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc99c001670 con 0x7fc9ac0cc590 2023-10-16T11:00:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 >> v1:172.21.15.158:6789/0 conn(0x7fc9ac0cc590 legacy=0x7fc9ac06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:40.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 0x7fc994003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc994003f80 0x7fc994004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc994004d90 0x7fc994005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac0cc1c0 con 0x7fc994004d90 2023-10-16T11:00:40.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac15de20 con 0x7fc994003f80 2023-10-16T11:00:40.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99400ba10 con 0x7fc9ac15da40 2023-10-16T11:00:40.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/3251013091 >> v2:172.21.15.172:3300/0 conn(0x7fc9ac06afe0 0x7fc9ac0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:40.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc994004d90 0x7fc994005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9ab7fe700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc994003f80 0x7fc994004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9abfff700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 0x7fc994003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 msgr2=0x7fc994003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:40.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac15da40 0x7fc994003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc994003f80 msgr2=0x7fc994004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:40.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc994003f80 0x7fc994004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/3251013091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940035b0 con 0x7fc994004d90 2023-10-16T11:00:40.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.590+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc994004d90 0x7fc994005210 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac072cc0 tx=0x7fc994005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c7d710e6d231ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:40.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a400ed30 con 0x7fc994004d90 2023-10-16T11:00:40.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a400f040 con 0x7fc994004d90 2023-10-16T11:00:40.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9aaffd700 1 -- 172.21.15.158:0/3251013091 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4003c60 con 0x7fc994004d90 2023-10-16T11:00:40.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9b2872700 1 -- 172.21.15.158:0/3251013091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc994004d90 msgr2=0x7fc994005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:40.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9b2872700 1 --2- 172.21.15.158:0/3251013091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc994004d90 0x7fc994005210 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac072cc0 tx=0x7fc994005880 comp rx=0 tx=0).stop 2023-10-16T11:00:40.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/3251013091 reap_dead start 2023-10-16T11:00:40.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9b2872700 1 -- 172.21.15.158:0/3251013091 shutdown_connections 2023-10-16T11:00:40.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9b2872700 1 -- 172.21.15.158:0/3251013091 >> 172.21.15.158:0/3251013091 conn(0x7fc9ac099a10 msgr2=0x7fc9ac0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:40.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9b2872700 1 -- 172.21.15.158:0/3251013091 shutdown_connections 2023-10-16T11:00:40.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.591+0000 7fc9b2872700 1 -- 172.21.15.158:0/3251013091 wait complete. 2023-10-16T11:00:40.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.593+0000 7fc9b2872700 1 Processor -- start 2023-10-16T11:00:40.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.593+0000 7fc9b2872700 1 -- start start 2023-10-16T11:00:40.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b2872700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ac15b890 0x7fc9ac0c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b2872700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac0c8530 0x7fc9ac15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b2872700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ac15bc70 0x7fc9ac1600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b2872700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac0c95e0 con 0x7fc9ac15bc70 2023-10-16T11:00:40.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b2872700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac0c89d0 con 0x7fc9ac15b890 2023-10-16T11:00:40.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b2872700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac0c8cb0 con 0x7fc9ac0c8530 2023-10-16T11:00:40.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ac15bc70 0x7fc9ac1600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ac15bc70 0x7fc9ac1600e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33118/0 (socket says 172.21.15.158:33118) 2023-10-16T11:00:40.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/1136604372 learned_addr learned my addr 172.21.15.158:0/1136604372 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:40.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9ab7fe700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac0c8530 0x7fc9ac15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9abfff700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ac15b890 0x7fc9ac0c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac0c8530 msgr2=0x7fc9ac15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:40.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac0c8530 0x7fc9ac15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ac15b890 msgr2=0x7fc9ac0c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:40.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ac15b890 0x7fc9ac0c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:40.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 -- 172.21.15.158:0/1136604372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac0c8f90 con 0x7fc9ac15bc70 2023-10-16T11:00:40.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.594+0000 7fc9b0e0f700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ac15bc70 0x7fc9ac1600e0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4004000 tx=0x7fc9a4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:40.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.595+0000 7fc9a8ff9700 1 -- 172.21.15.158:0/1136604372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4004440 con 0x7fc9ac15bc70 2023-10-16T11:00:40.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.595+0000 7fc9a8ff9700 1 -- 172.21.15.158:0/1136604372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a40045a0 con 0x7fc9ac15bc70 2023-10-16T11:00:40.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.595+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac0c9270 con 0x7fc9ac15bc70 2023-10-16T11:00:40.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.595+0000 7fc9a8ff9700 1 -- 172.21.15.158:0/1136604372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4004950 con 0x7fc9ac15bc70 2023-10-16T11:00:40.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.595+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac0b2230 con 0x7fc9ac15bc70 2023-10-16T11:00:40.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.596+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac0681b0 con 0x7fc9ac15bc70 2023-10-16T11:00:40.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.596+0000 7fc9a8ff9700 1 -- 172.21.15.158:0/1136604372 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc9a4025020 con 0x7fc9ac15bc70 2023-10-16T11:00:40.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.597+0000 7fc9a8ff9700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc99806b020 0x7fc99806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:40.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.597+0000 7fc9abfff700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc99806b020 0x7fc99806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:40.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.597+0000 7fc9a8ff9700 1 -- 172.21.15.158:0/1136604372 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6381+0+0 (secure 0 0 0) 0x7fc9a408b5b0 con 0x7fc9ac15bc70 2023-10-16T11:00:40.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.597+0000 7fc9abfff700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc99806b020 0x7fc99806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac1f62a0 tx=0x7fc9a0009210 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:40.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.602+0000 7fc9a8ff9700 1 -- 172.21.15.158:0/1136604372 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9a4056400 con 0x7fc9ac15bc70 2023-10-16T11:00:40.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:40.838+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc9ac0b2230 con 0x7fc9ac15bc70 2023-10-16T11:00:41.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.458+0000 7fc9a8ff9700 1 -- 172.21.15.158:0/1136604372 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v81) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc9ac0b2230 con 0x7fc9ac15bc70 2023-10-16T11:00:41.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc99806b020 msgr2=0x7fc99806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc99806b020 0x7fc99806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac1f62a0 tx=0x7fc9a0009210 comp rx=0 tx=0).stop 2023-10-16T11:00:41.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ac15bc70 msgr2=0x7fc9ac1600e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ac15bc70 0x7fc9ac1600e0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4004000 tx=0x7fc9a4009750 comp rx=0 tx=0).stop 2023-10-16T11:00:41.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 shutdown_connections 2023-10-16T11:00:41.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ac15bc70 0x7fc9ac1600e0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fc99806b020 0x7fc99806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ac15b890 0x7fc9ac0c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 --2- 172.21.15.158:0/1136604372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ac0c8530 0x7fc9ac15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.463+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 >> 172.21.15.158:0/1136604372 conn(0x7fc9ac099a10 msgr2=0x7fc9ac15f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:41.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.464+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 shutdown_connections 2023-10-16T11:00:41.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.464+0000 7fc9b2872700 1 -- 172.21.15.158:0/1136604372 wait complete. 2023-10-16T11:00:41.468 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 21 and data pool 22 2023-10-16T11:00:41.478 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:00:41.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:00:41.485 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.644+0000 7f413fc65700 1 Processor -- start 2023-10-16T11:00:41.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.645+0000 7f413fc65700 1 -- start start 2023-10-16T11:00:41.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413fc65700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4138161cd0 0x7f41380ae920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:41.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413fc65700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413806b2e0 con 0x7f41380cd810 2023-10-16T11:00:41.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413fc65700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413806b530 con 0x7f41380ce380 2023-10-16T11:00:41.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413fc65700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413806c1b0 con 0x7f4138161cd0 2023-10-16T11:00:41.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413d200700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f41380cd810 0x7f41380cdc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33298/0 (socket says 172.21.15.158:33298) 2023-10-16T11:00:41.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413d200700 1 -- 172.21.15.158:0/3655924158 learned_addr learned my addr 172.21.15.158:0/3655924158 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:41.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413da01700 1 --2- 172.21.15.158:0/3655924158 >> v2:172.21.15.158:3300/0 conn(0x7f4138161cd0 0x7f41380ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:41.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413c9ff700 1 -- 172.21.15.158:0/3655924158 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462046137 0 0) 0x7f413806b2e0 con 0x7f41380cd810 2023-10-16T11:00:41.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413c9ff700 1 -- 172.21.15.158:0/3655924158 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41240035b0 con 0x7f41380cd810 2023-10-16T11:00:41.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413c9ff700 1 -- 172.21.15.158:0/3655924158 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719363517 0 0) 0x7f413806b530 con 0x7f41380ce380 2023-10-16T11:00:41.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413da01700 1 -- 172.21.15.158:0/3655924158 >> v1:172.21.15.172:6789/0 conn(0x7f41380ce380 legacy=0x7f41380c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413da01700 1 -- 172.21.15.158:0/3655924158 >> v1:172.21.15.158:6789/0 conn(0x7f41380cd810 legacy=0x7f41380cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.646+0000 7f413da01700 1 -- 172.21.15.158:0/3655924158 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413806b810 con 0x7f4138161cd0 2023-10-16T11:00:41.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413da01700 1 --2- 172.21.15.158:0/3655924158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380ae920 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f4134009840 tx=0x7f4134009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d61d3258d7b9a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:41.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413c9ff700 1 -- 172.21.15.158:0/3655924158 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4134011830 con 0x7f4138161cd0 2023-10-16T11:00:41.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413c9ff700 1 -- 172.21.15.158:0/3655924158 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4134011990 con 0x7f4138161cd0 2023-10-16T11:00:41.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413c9ff700 1 -- 172.21.15.158:0/3655924158 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4134011d20 con 0x7f4138161cd0 2023-10-16T11:00:41.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413fc65700 1 -- 172.21.15.158:0/3655924158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 msgr2=0x7f41380ae920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413fc65700 1 --2- 172.21.15.158:0/3655924158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380ae920 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f4134009840 tx=0x7f4134009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:41.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413fc65700 1 -- 172.21.15.158:0/3655924158 shutdown_connections 2023-10-16T11:00:41.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413fc65700 1 --2- 172.21.15.158:0/3655924158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380ae920 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.647+0000 7f413fc65700 1 -- 172.21.15.158:0/3655924158 >> 172.21.15.158:0/3655924158 conn(0x7f41380cc4b0 msgr2=0x7f41380cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:41.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.648+0000 7f413fc65700 1 -- 172.21.15.158:0/3655924158 shutdown_connections 2023-10-16T11:00:41.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.648+0000 7f413fc65700 1 -- 172.21.15.158:0/3655924158 wait complete. 2023-10-16T11:00:41.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.649+0000 7f413fc65700 1 Processor -- start 2023-10-16T11:00:41.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.649+0000 7f413fc65700 1 -- start start 2023-10-16T11:00:41.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41380cd810 0x7f4138163160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:41.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41380ce380 0x7f41381576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:41.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:41.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4138157c20 con 0x7f4138161cd0 2023-10-16T11:00:41.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41381639e0 con 0x7f41380ce380 2023-10-16T11:00:41.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4138162540 con 0x7f41380cd810 2023-10-16T11:00:41.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:41.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380b06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33130/0 (socket says 172.21.15.158:33130) 2023-10-16T11:00:41.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 -- 172.21.15.158:0/127516718 learned_addr learned my addr 172.21.15.158:0/127516718 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:41.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413da01700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41380cd810 0x7f4138163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:41.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413d200700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41380ce380 0x7f41381576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:41.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 -- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41380cd810 msgr2=0x7f4138163160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41380cd810 0x7f4138163160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 -- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41380ce380 msgr2=0x7f41381576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41380ce380 0x7f41381576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 -- 172.21.15.158:0/127516718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340094a0 con 0x7f4138161cd0 2023-10-16T11:00:41.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413d200700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41380ce380 0x7f41381576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:41.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413e202700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380b06a0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f4130001560 tx=0x7f413000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:41.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f412e7fc700 1 -- 172.21.15.158:0/127516718 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130003e30 con 0x7f4138161cd0 2023-10-16T11:00:41.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f412e7fc700 1 -- 172.21.15.158:0/127516718 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130005070 con 0x7f4138161cd0 2023-10-16T11:00:41.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41381f7320 con 0x7f4138161cd0 2023-10-16T11:00:41.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f412e7fc700 1 -- 172.21.15.158:0/127516718 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130005420 con 0x7f4138161cd0 2023-10-16T11:00:41.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.650+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41381f78e0 con 0x7f4138161cd0 2023-10-16T11:00:41.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.651+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413806bad0 con 0x7f4138161cd0 2023-10-16T11:00:41.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.652+0000 7f412e7fc700 1 -- 172.21.15.158:0/127516718 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4130005580 con 0x7f4138161cd0 2023-10-16T11:00:41.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.652+0000 7f412e7fc700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f411c06b020 0x7f411c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:41.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.652+0000 7f412e7fc700 1 -- 172.21.15.158:0/127516718 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f4130089210 con 0x7f4138161cd0 2023-10-16T11:00:41.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.652+0000 7f413da01700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f411c06b020 0x7f411c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:41.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.653+0000 7f413da01700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f411c06b020 0x7f411c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4134009fd0 tx=0x7f413401c040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:41.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.657+0000 7f412e7fc700 1 -- 172.21.15.158:0/127516718 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f413806bad0 con 0x7f4138161cd0 2023-10-16T11:00:41.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.850+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f413806bad0 con 0x7f4138161cd0 2023-10-16T11:00:41.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.852+0000 7f412e7fc700 1 -- 172.21.15.158:0/127516718 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+17439 (secure 0 0 0) 0x7f413806bad0 con 0x7f4138161cd0 2023-10-16T11:00:41.853 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:41.854 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":171,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:40.839985+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:00:38.634284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T11:00:39.812150+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:41.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f411c06b020 msgr2=0x7f411c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f411c06b020 0x7f411c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4134009fd0 tx=0x7f413401c040 comp rx=0 tx=0).stop 2023-10-16T11:00:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 msgr2=0x7f41380b06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380b06a0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f4130001560 tx=0x7f413000cc00 comp rx=0 tx=0).stop 2023-10-16T11:00:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 shutdown_connections 2023-10-16T11:00:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4138161cd0 0x7f41380b06a0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41380cd810 0x7f4138163160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f411c06b020 0x7f411c06d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 --2- 172.21.15.158:0/127516718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41380ce380 0x7f41381576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 >> 172.21.15.158:0/127516718 conn(0x7f41380cc4b0 msgr2=0x7f41380c4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 shutdown_connections 2023-10-16T11:00:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.854+0000 7f413fc65700 1 -- 172.21.15.158:0/127516718 wait complete. 2023-10-16T11:00:41.868 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:41.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.988+0000 7fdc0cfe8700 1 Processor -- start 2023-10-16T11:00:41.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.988+0000 7fdc0cfe8700 1 -- start start 2023-10-16T11:00:41.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0cfe8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc080cc5d0 0x7fdc0806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:41.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0cfe8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0806e770 con 0x7fdc0806f340 2023-10-16T11:00:41.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0cfe8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc080cbe00 con 0x7fdc0815dfb0 2023-10-16T11:00:41.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0cfe8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc080cf270 con 0x7fdc080cc5d0 2023-10-16T11:00:41.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc06d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdc0806f340 0x7fdc08071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33314/0 (socket says 172.21.15.158:33314) 2023-10-16T11:00:41.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc06d9d700 1 -- 172.21.15.158:0/4206785698 learned_addr learned my addr 172.21.15.158:0/4206785698 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:41.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0659c700 1 --2- 172.21.15.158:0/4206785698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc080cc5d0 0x7fdc0806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:41.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995741991 0 0) 0x7fdc00001490 con 0x7fdc0806f340 2023-10-16T11:00:41.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf00035b0 con 0x7fdc0806f340 2023-10-16T11:00:41.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962153196 0 0) 0x7fdc080cbe00 con 0x7fdc0815dfb0 2023-10-16T11:00:41.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc00001490 con 0x7fdc0815dfb0 2023-10-16T11:00:41.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948753461 0 0) 0x7fdbf00035b0 con 0x7fdc0806f340 2023-10-16T11:00:41.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0659c700 1 -- 172.21.15.158:0/4206785698 >> v1:172.21.15.172:6789/0 conn(0x7fdc0815dfb0 legacy=0x7fdc080cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0659c700 1 -- 172.21.15.158:0/4206785698 >> v1:172.21.15.158:6789/0 conn(0x7fdc0806f340 legacy=0x7fdc08071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.989+0000 7fdc0659c700 1 -- 172.21.15.158:0/4206785698 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0815e4e0 con 0x7fdc080cc5d0 2023-10-16T11:00:41.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0659c700 1 --2- 172.21.15.158:0/4206785698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc080cc5d0 0x7fdc0806e230 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009870 tx=0x7fdbf8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cc658d7b0cd4a24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:41.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00001b80 con 0x7fdc080cc5d0 2023-10-16T11:00:41.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8004030 con 0x7fdc080cc5d0 2023-10-16T11:00:41.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0559a700 1 -- 172.21.15.158:0/4206785698 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf80043a0 con 0x7fdc080cc5d0 2023-10-16T11:00:41.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/4206785698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc080cc5d0 msgr2=0x7fdc0806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:41.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/4206785698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc080cc5d0 0x7fdc0806e230 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009870 tx=0x7fdbf8009b80 comp rx=0 tx=0).stop 2023-10-16T11:00:41.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/4206785698 shutdown_connections 2023-10-16T11:00:41.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/4206785698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc080cc5d0 0x7fdc0806e230 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:41.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/4206785698 >> 172.21.15.158:0/4206785698 conn(0x7fdc080d02b0 msgr2=0x7fdc080d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:41.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/4206785698 shutdown_connections 2023-10-16T11:00:41.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.990+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/4206785698 wait complete. 2023-10-16T11:00:41.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.992+0000 7fdc0cfe8700 1 Processor -- start 2023-10-16T11:00:41.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.992+0000 7fdc0cfe8700 1 -- start start 2023-10-16T11:00:41.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc0806f340 0x7fdc0806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc080cc5d0 0x7fdc0815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc0815dfb0 0x7fdc08158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081f4d20 con 0x7fdc080cc5d0 2023-10-16T11:00:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081f4150 con 0x7fdc0806f340 2023-10-16T11:00:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081f43f0 con 0x7fdc0815dfb0 2023-10-16T11:00:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc080cc5d0 0x7fdc0815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc080cc5d0 0x7fdc0815cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33142/0 (socket says 172.21.15.158:33142) 2023-10-16T11:00:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 -- 172.21.15.158:0/3106004623 learned_addr learned my addr 172.21.15.158:0/3106004623 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0659c700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc0806f340 0x7fdc0806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc06d9d700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc0815dfb0 0x7fdc08158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 -- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc0815dfb0 msgr2=0x7fdc08158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc0815dfb0 0x7fdc08158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 -- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc0806f340 msgr2=0x7fdc0806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc0806f340 0x7fdc0806b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 -- 172.21.15.158:0/3106004623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80094a0 con 0x7fdc080cc5d0 2023-10-16T11:00:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0659c700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc0806f340 0x7fdc0806b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc05d9b700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc080cc5d0 0x7fdc0815cbc0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00c490 tx=0x7fdbfc00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdbf6ffd700 1 -- 172.21.15.158:0/3106004623 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0052d0 con 0x7fdc080cc5d0 2023-10-16T11:00:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdbf6ffd700 1 -- 172.21.15.158:0/3106004623 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc005450 con 0x7fdc080cc5d0 2023-10-16T11:00:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc081f46d0 con 0x7fdc080cc5d0 2023-10-16T11:00:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdbf6ffd700 1 -- 172.21.15.158:0/3106004623 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0038c0 con 0x7fdc080cc5d0 2023-10-16T11:00:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.993+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0806b730 con 0x7fdc080cc5d0 2023-10-16T11:00:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.994+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc081f4e90 con 0x7fdc080cc5d0 2023-10-16T11:00:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.995+0000 7fdbf6ffd700 1 -- 172.21.15.158:0/3106004623 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdbfc003a20 con 0x7fdc080cc5d0 2023-10-16T11:00:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.995+0000 7fdbf6ffd700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fdbe806b020 0x7fdbe806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.995+0000 7fdbf6ffd700 1 -- 172.21.15.158:0/3106004623 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fdbfc088e90 con 0x7fdc080cc5d0 2023-10-16T11:00:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.995+0000 7fdc0659c700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fdbe806b020 0x7fdbe806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.995+0000 7fdc0659c700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fdbe806b020 0x7fdbe806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80097c0 tx=0x7fdbf8008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:42.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:41.999+0000 7fdbf6ffd700 1 -- 172.21.15.158:0/3106004623 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbfc053b20 con 0x7fdc080cc5d0 2023-10-16T11:00:42.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.190+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc080681b0 con 0x7fdc080cc5d0 2023-10-16T11:00:42.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.191+0000 7fdbf6ffd700 1 -- 172.21.15.158:0/3106004623 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+17439 (secure 0 0 0) 0x7fdbfc017090 con 0x7fdc080cc5d0 2023-10-16T11:00:42.192 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:42.193 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":171,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:40.839985+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:00:38.634284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T11:00:39.812150+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:42.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fdbe806b020 msgr2=0x7fdbe806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fdbe806b020 0x7fdbe806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80097c0 tx=0x7fdbf8008030 comp rx=0 tx=0).stop 2023-10-16T11:00:42.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc080cc5d0 msgr2=0x7fdc0815cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc080cc5d0 0x7fdc0815cbc0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00c490 tx=0x7fdbfc00c7a0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 shutdown_connections 2023-10-16T11:00:42.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc0815dfb0 0x7fdc08158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fdbe806b020 0x7fdbe806d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc0806f340 0x7fdc0806b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 --2- 172.21.15.158:0/3106004623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc080cc5d0 0x7fdc0815cbc0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 >> 172.21.15.158:0/3106004623 conn(0x7fdc080d02b0 msgr2=0x7fdc0806f760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:42.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 shutdown_connections 2023-10-16T11:00:42.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.193+0000 7fdc0cfe8700 1 -- 172.21.15.158:0/3106004623 wait complete. 2023-10-16T11:00:42.207 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T11:00:42.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.329+0000 7feceff96700 1 Processor -- start 2023-10-16T11:00:42.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.329+0000 7feceff96700 1 -- start start 2023-10-16T11:00:42.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7feceff96700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fece80ce510 0x7fece815f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7feceff96700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 0x7fece80cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7feceff96700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80cf4c0 con 0x7fece815b590 2023-10-16T11:00:42.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7feceff96700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece80cc4c0 con 0x7fece80ce510 2023-10-16T11:00:42.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7feceff96700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece80b49e0 con 0x7fece80cbaf0 2023-10-16T11:00:42.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7fecedd32700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fece815b590 0x7fece80b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33324/0 (socket says 172.21.15.158:33324) 2023-10-16T11:00:42.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7fecedd32700 1 -- 172.21.15.158:0/1636012424 learned_addr learned my addr 172.21.15.158:0/1636012424 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:42.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7feced531700 1 --2- 172.21.15.158:0/1636012424 >> v2:172.21.15.172:3300/0 conn(0x7fece80ce510 0x7fece815f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.330+0000 7fecee533700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 0x7fece80cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352429120 0 0) 0x7fece80cf4c0 con 0x7fece815b590 2023-10-16T11:00:42.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd40035b0 con 0x7fece815b590 2023-10-16T11:00:42.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892462328 0 0) 0x7fecd40035b0 con 0x7fece815b590 2023-10-16T11:00:42.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 msgr2=0x7fece80cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:42.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 0x7fece80cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 >> v2:172.21.15.172:3300/0 conn(0x7fece80ce510 msgr2=0x7fece815f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:42.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 --2- 172.21.15.158:0/1636012424 >> v2:172.21.15.172:3300/0 conn(0x7fece80ce510 0x7fece815f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece815f710 con 0x7fece815b590 2023-10-16T11:00:42.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fecd8001670 con 0x7fece815b590 2023-10-16T11:00:42.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 --2- 172.21.15.158:0/1636012424 >> v2:172.21.15.172:3300/0 conn(0x7fece80ce510 0x7fece815f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:42.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fecee533700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 0x7fece80cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:42.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 >> v1:172.21.15.158:6789/0 conn(0x7fece815b590 legacy=0x7fece80b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 0x7fecd4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecd4003f70 0x7fecd4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecd4004d90 0x7fecd4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80cc4c0 con 0x7fecd4003f70 2023-10-16T11:00:42.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80b49e0 con 0x7fecd4004d90 2023-10-16T11:00:42.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd400fe50 con 0x7fece80cbaf0 2023-10-16T11:00:42.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecd4003f70 0x7fecd4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fecedd32700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 0x7fecd4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fecee533700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecd4004d90 0x7fecd4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 -- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 msgr2=0x7fecd4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:42.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cbaf0 0x7fecd4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 -- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecd4004d90 msgr2=0x7fecd4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:42.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecd4004d90 0x7fecd4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 -- 172.21.15.158:0/1636012424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0009a70 con 0x7fecd4003f70 2023-10-16T11:00:42.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fecee533700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecd4004d90 0x7fecd4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:42.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7feced531700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecd4003f70 0x7fecd4004850 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fece4009d30 tx=0x7fecd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b9dc3c816d5c79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:42.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.331+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4009e80 con 0x7fecd4003f70 2023-10-16T11:00:42.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece400e9a0 con 0x7fecd4003f70 2023-10-16T11:00:42.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7fececd30700 1 -- 172.21.15.158:0/1636012424 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400ed30 con 0x7fecd4003f70 2023-10-16T11:00:42.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7feceff96700 1 -- 172.21.15.158:0/1636012424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecd4003f70 msgr2=0x7fecd4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7feceff96700 1 --2- 172.21.15.158:0/1636012424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecd4003f70 0x7fecd4004850 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fece4009d30 tx=0x7fecd4005880 comp rx=0 tx=0).stop 2023-10-16T11:00:42.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7fecee533700 1 -- 172.21.15.158:0/1636012424 reap_dead start 2023-10-16T11:00:42.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7feceff96700 1 -- 172.21.15.158:0/1636012424 shutdown_connections 2023-10-16T11:00:42.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7feceff96700 1 -- 172.21.15.158:0/1636012424 >> 172.21.15.158:0/1636012424 conn(0x7fece806a8d0 msgr2=0x7fece806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:42.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7feceff96700 1 -- 172.21.15.158:0/1636012424 shutdown_connections 2023-10-16T11:00:42.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.332+0000 7feceff96700 1 -- 172.21.15.158:0/1636012424 wait complete. 2023-10-16T11:00:42.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.333+0000 7feceff96700 1 Processor -- start 2023-10-16T11:00:42.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.333+0000 7feceff96700 1 -- start start 2023-10-16T11:00:42.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7feceff96700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecd4003f70 0x7fece8155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7feceff96700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece8155670 0x7fece8154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7feceff96700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece80c8570 0x7fece80c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7feceff96700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece8155500 con 0x7fece80c8570 2023-10-16T11:00:42.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7feceff96700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80b6ea0 con 0x7fece8155670 2023-10-16T11:00:42.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7feceff96700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece815cbf0 con 0x7fecd4003f70 2023-10-16T11:00:42.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece80c8570 0x7fece80c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece80c8570 0x7fece80c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33164/0 (socket says 172.21.15.158:33164) 2023-10-16T11:00:42.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 -- 172.21.15.158:0/3188543671 learned_addr learned my addr 172.21.15.158:0/3188543671 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:42.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecedd32700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecd4003f70 0x7fece8155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7feced531700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece8155670 0x7fece8154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 -- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecd4003f70 msgr2=0x7fece8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecd4003f70 0x7fece8155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 -- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece8155670 msgr2=0x7fece8154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:42.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece8155670 0x7fece8154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:42.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.334+0000 7fecee533700 1 -- 172.21.15.158:0/3188543671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece81f30b0 con 0x7fece80c8570 2023-10-16T11:00:42.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.335+0000 7fecee533700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece80c8570 0x7fece80c8a00 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fece80cee80 tx=0x7fece00114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:42.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.335+0000 7fecde7fc700 1 -- 172.21.15.158:0/3188543671 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece000e070 con 0x7fece80c8570 2023-10-16T11:00:42.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.335+0000 7fecde7fc700 1 -- 172.21.15.158:0/3188543671 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece0011ce0 con 0x7fece80c8570 2023-10-16T11:00:42.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.335+0000 7feceff96700 1 -- 172.21.15.158:0/3188543671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece81f35e0 con 0x7fece80c8570 2023-10-16T11:00:42.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.335+0000 7fecde7fc700 1 -- 172.21.15.158:0/3188543671 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0011e70 con 0x7fece80c8570 2023-10-16T11:00:42.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.335+0000 7feceff96700 1 -- 172.21.15.158:0/3188543671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece815d470 con 0x7fece80c8570 2023-10-16T11:00:42.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.336+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece815d600 con 0x7fece80c8570 2023-10-16T11:00:42.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.336+0000 7fecde7fc700 1 -- 172.21.15.158:0/3188543671 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fece0003a20 con 0x7fece80c8570 2023-10-16T11:00:42.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.337+0000 7fecde7fc700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feccc06b020 0x7feccc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:42.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.337+0000 7fecde7fc700 1 -- 172.21.15.158:0/3188543671 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fece008a090 con 0x7fece80c8570 2023-10-16T11:00:42.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.337+0000 7fecedd32700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feccc06b020 0x7feccc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:42.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.337+0000 7fecedd32700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feccc06b020 0x7feccc06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fece81f2600 tx=0x7fecd8009040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:42.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.341+0000 7fecde7fc700 1 -- 172.21.15.158:0/3188543671 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fece0054b70 con 0x7fece80c8570 2023-10-16T11:00:42.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:42.581+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fece80681b0 con 0x7fece80c8570 2023-10-16T11:00:43.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.470+0000 7fecde7fc700 1 -- 172.21.15.158:0/3188543671 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v84) v1 ==== 118+0+0 (secure 0 0 0) 0x7fece0058190 con 0x7fece80c8570 2023-10-16T11:00:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feccc06b020 msgr2=0x7feccc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feccc06b020 0x7feccc06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fece81f2600 tx=0x7fecd8009040 comp rx=0 tx=0).stop 2023-10-16T11:00:43.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece80c8570 msgr2=0x7fece80c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:43.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece80c8570 0x7fece80c8a00 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fece80cee80 tx=0x7fece00114b0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 shutdown_connections 2023-10-16T11:00:43.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece80c8570 0x7fece80c8a00 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecd4003f70 0x7fece8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7feccc06b020 0x7feccc06d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 --2- 172.21.15.158:0/3188543671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece8155670 0x7fece8154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 >> 172.21.15.158:0/3188543671 conn(0x7fece806a8d0 msgr2=0x7fece806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:43.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 shutdown_connections 2023-10-16T11:00:43.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.474+0000 7feccbfff700 1 -- 172.21.15.158:0/3188543671 wait complete. 2023-10-16T11:00:43.490 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:00:43.490 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T11:00:43.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.618+0000 7f937d8c5700 1 Processor -- start 2023-10-16T11:00:43.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.618+0000 7f937d8c5700 1 -- start start 2023-10-16T11:00:43.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f937d8c5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f93780ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:43.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f937d8c5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f937815f1d0 0x7f93780b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:43.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f937d8c5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93780cbaf0 0x7f93780cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:43.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f937d8c5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93780cc4c0 con 0x7f937815f1d0 2023-10-16T11:00:43.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f937d8c5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93780cf4c0 con 0x7f93780cbaf0 2023-10-16T11:00:43.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f937d8c5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937815f970 con 0x7f937815b590 2023-10-16T11:00:43.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f93767fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f937815f1d0 0x7f93780b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:43.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f93767fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f937815f1d0 0x7f93780b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33168/0 (socket says 172.21.15.158:33168) 2023-10-16T11:00:43.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f93767fc700 1 -- 172.21.15.158:0/1577045115 learned_addr learned my addr 172.21.15.158:0/1577045115 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:43.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f9376ffd700 1 --2- 172.21.15.158:0/1577045115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f93780ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:43.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.619+0000 7f93777fe700 1 --2- 172.21.15.158:0/1577045115 >> v2:172.21.15.172:3300/0 conn(0x7f93780cbaf0 0x7f93780cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:43.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f93767fc700 1 -- 172.21.15.158:0/1577045115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 msgr2=0x7f93780ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:43.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f93767fc700 1 --2- 172.21.15.158:0/1577045115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f93780ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f93767fc700 1 -- 172.21.15.158:0/1577045115 >> v2:172.21.15.172:3300/0 conn(0x7f93780cbaf0 msgr2=0x7f93780cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:43.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f93767fc700 1 --2- 172.21.15.158:0/1577045115 >> v2:172.21.15.172:3300/0 conn(0x7f93780cbaf0 0x7f93780cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f93767fc700 1 -- 172.21.15.158:0/1577045115 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937815fc50 con 0x7f937815f1d0 2023-10-16T11:00:43.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f93767fc700 1 --2- 172.21.15.158:0/1577045115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f93780b42d0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f93780b3640 tx=0x7f9368009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8668f0328d7a063 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:43.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f9375ffb700 1 -- 172.21.15.158:0/1577045115 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936800b870 con 0x7f937815f1d0 2023-10-16T11:00:43.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f9375ffb700 1 -- 172.21.15.158:0/1577045115 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936800b9d0 con 0x7f937815f1d0 2023-10-16T11:00:43.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f9375ffb700 1 -- 172.21.15.158:0/1577045115 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936800bd60 con 0x7f937815f1d0 2023-10-16T11:00:43.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f937d8c5700 1 -- 172.21.15.158:0/1577045115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 msgr2=0x7f93780b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:43.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f937d8c5700 1 --2- 172.21.15.158:0/1577045115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f93780b42d0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f93780b3640 tx=0x7f9368009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f937d8c5700 1 -- 172.21.15.158:0/1577045115 shutdown_connections 2023-10-16T11:00:43.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f937d8c5700 1 --2- 172.21.15.158:0/1577045115 >> v2:172.21.15.172:3300/0 conn(0x7f93780cbaf0 0x7f93780cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f937d8c5700 1 --2- 172.21.15.158:0/1577045115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f93780ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f937d8c5700 1 --2- 172.21.15.158:0/1577045115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f93780b42d0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.620+0000 7f937d8c5700 1 -- 172.21.15.158:0/1577045115 >> 172.21.15.158:0/1577045115 conn(0x7f937806a8d0 msgr2=0x7f937806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:43.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.621+0000 7f937d8c5700 1 -- 172.21.15.158:0/1577045115 shutdown_connections 2023-10-16T11:00:43.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.621+0000 7f937d8c5700 1 -- 172.21.15.158:0/1577045115 wait complete. 2023-10-16T11:00:43.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.622+0000 7f937d8c5700 1 Processor -- start 2023-10-16T11:00:43.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.622+0000 7f937d8c5700 1 -- start start 2023-10-16T11:00:43.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93780cbaf0 0x7f937815d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f937815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f937815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780cb1d0 con 0x7f937815f1d0 2023-10-16T11:00:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780ca5c0 con 0x7f93780cbaf0 2023-10-16T11:00:43.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93780ca8a0 con 0x7f937815b590 2023-10-16T11:00:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f937815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f937815c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33176/0 (socket says 172.21.15.158:33176) 2023-10-16T11:00:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 -- 172.21.15.158:0/3606967249 learned_addr learned my addr 172.21.15.158:0/3606967249 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93767fc700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f937815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f9376ffd700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93780cbaf0 0x7f937815d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:43.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 -- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 msgr2=0x7f937815bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f937815bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 -- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93780cbaf0 msgr2=0x7f937815d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93780cbaf0 0x7f937815d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 -- 172.21.15.158:0/3606967249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9368009550 con 0x7f937815f1d0 2023-10-16T11:00:43.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f9376ffd700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93780cbaf0 0x7f937815d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f93777fe700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f937815c120 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f9370007e60 tx=0x7f937000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f935ffff700 1 -- 172.21.15.158:0/3606967249 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937000ede0 con 0x7f937815f1d0 2023-10-16T11:00:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f935ffff700 1 -- 172.21.15.158:0/3606967249 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9370014420 con 0x7f937815f1d0 2023-10-16T11:00:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93780cab80 con 0x7f937815f1d0 2023-10-16T11:00:43.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f935ffff700 1 -- 172.21.15.158:0/3606967249 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93700147d0 con 0x7f937815f1d0 2023-10-16T11:00:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.623+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93780c79b0 con 0x7f937815f1d0 2023-10-16T11:00:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.624+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93780cb340 con 0x7f937815f1d0 2023-10-16T11:00:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.625+0000 7f935ffff700 1 -- 172.21.15.158:0/3606967249 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9370014930 con 0x7f937815f1d0 2023-10-16T11:00:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.625+0000 7f935ffff700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f936406b020 0x7f936406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:43.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.625+0000 7f935ffff700 1 -- 172.21.15.158:0/3606967249 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f937008a180 con 0x7f937815f1d0 2023-10-16T11:00:43.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.625+0000 7f9376ffd700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f936406b020 0x7f936406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:43.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.626+0000 7f9376ffd700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f936406b020 0x7f936406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f936c0099f0 tx=0x7f936c008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:43.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.630+0000 7f935ffff700 1 -- 172.21.15.158:0/3606967249 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9370054e10 con 0x7f937815f1d0 2023-10-16T11:00:43.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:43.868+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f93780681b0 con 0x7f937815f1d0 2023-10-16T11:00:44.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.480+0000 7f935ffff700 1 -- 172.21.15.158:0/3606967249 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v85) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9370058430 con 0x7f937815f1d0 2023-10-16T11:00:44.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f936406b020 msgr2=0x7f936406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f936406b020 0x7f936406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f936c0099f0 tx=0x7f936c008040 comp rx=0 tx=0).stop 2023-10-16T11:00:44.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 msgr2=0x7f937815c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f937815c120 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f9370007e60 tx=0x7f937000e3c0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 shutdown_connections 2023-10-16T11:00:44.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937815f1d0 0x7f937815c120 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f936406b020 0x7f936406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93780cbaf0 0x7f937815d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.482+0000 7f937d8c5700 1 --2- 172.21.15.158:0/3606967249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f937815b590 0x7f937815bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.483+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 >> 172.21.15.158:0/3606967249 conn(0x7f937806a8d0 msgr2=0x7f93780d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:44.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.483+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 shutdown_connections 2023-10-16T11:00:44.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.483+0000 7f937d8c5700 1 -- 172.21.15.158:0/3606967249 wait complete. 2023-10-16T11:00:44.499 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T11:00:44.499 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:00:44.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.624+0000 7f3d710f3700 1 Processor -- start 2023-10-16T11:00:44.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.624+0000 7f3d710f3700 1 -- start start 2023-10-16T11:00:44.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d710f3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3d6c150b80 0x7f3d6c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d710f3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d6c0cf610 0x7f3d6c15dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d710f3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c06e7a0 con 0x7f3d6c15d7e0 2023-10-16T11:00:44.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d710f3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6c06f420 con 0x7f3d6c150b80 2023-10-16T11:00:44.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d710f3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c15e1d0 con 0x7f3d6c0cf610 2023-10-16T11:00:44.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d6a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3d6c15d7e0 0x7f3d6c15a1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33356/0 (socket says 172.21.15.158:33356) 2023-10-16T11:00:44.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d6a59c700 1 -- 172.21.15.158:0/1840649914 learned_addr learned my addr 172.21.15.158:0/1840649914 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:44.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d6b59e700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d6c0cf610 0x7f3d6c15dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/1840649914 >> v2:172.21.15.172:3300/0 conn(0x7f3d6c150b80 0x7f3d6c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575489318 0 0) 0x7f3d6c06e7a0 con 0x7f3d6c15d7e0 2023-10-16T11:00:44.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d540035b0 con 0x7f3d6c15d7e0 2023-10-16T11:00:44.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3558403868 0 0) 0x7f3d540035b0 con 0x7f3d6c15d7e0 2023-10-16T11:00:44.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d6c0cf610 msgr2=0x7f3d6c15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:44.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d6c0cf610 0x7f3d6c15dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 >> v2:172.21.15.172:3300/0 conn(0x7f3d6c150b80 msgr2=0x7f3d6c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:44.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 --2- 172.21.15.158:0/1840649914 >> v2:172.21.15.172:3300/0 conn(0x7f3d6c150b80 0x7f3d6c06df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c06ea80 con 0x7f3d6c15d7e0 2023-10-16T11:00:44.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3d5c001670 con 0x7f3d6c15d7e0 2023-10-16T11:00:44.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d6b59e700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d6c0cf610 0x7f3d6c15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:44.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.625+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 >> v1:172.21.15.158:6789/0 conn(0x7f3d6c15d7e0 legacy=0x7f3d6c15a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d54003a00 0x7f3d54003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d54004850 0x7f3d54004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54005210 0x7f3d5400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c06f420 con 0x7f3d54003a00 2023-10-16T11:00:44.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c15e1d0 con 0x7f3d54004850 2023-10-16T11:00:44.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540056c0 con 0x7f3d54005210 2023-10-16T11:00:44.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d54003a00 0x7f3d54003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6a59c700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d54004850 0x7f3d54004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6b59e700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54005210 0x7f3d5400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6ad9d700 1 -- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54005210 msgr2=0x7f3d5400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:44.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54005210 0x7f3d5400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6ad9d700 1 -- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d54004850 msgr2=0x7f3d54004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:44.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d54004850 0x7f3d54004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6ad9d700 1 -- 172.21.15.158:0/1840649914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d64009a70 con 0x7f3d54003a00 2023-10-16T11:00:44.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6a59c700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d54004850 0x7f3d54004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:44.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d54003a00 0x7f3d54003e20 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009b60 tx=0x7f3d54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9b8906da14e528 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:44.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000e660 con 0x7f3d54003a00 2023-10-16T11:00:44.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6000e7c0 con 0x7f3d54003a00 2023-10-16T11:00:44.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.626+0000 7f3d69d9b700 1 -- 172.21.15.158:0/1840649914 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000eb30 con 0x7f3d54003a00 2023-10-16T11:00:44.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.627+0000 7f3d710f3700 1 -- 172.21.15.158:0/1840649914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d54003a00 msgr2=0x7f3d54003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.627+0000 7f3d710f3700 1 --2- 172.21.15.158:0/1840649914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d54003a00 0x7f3d54003e20 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009b60 tx=0x7f3d54005880 comp rx=0 tx=0).stop 2023-10-16T11:00:44.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.627+0000 7f3d6b59e700 1 -- 172.21.15.158:0/1840649914 reap_dead start 2023-10-16T11:00:44.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.627+0000 7f3d710f3700 1 -- 172.21.15.158:0/1840649914 shutdown_connections 2023-10-16T11:00:44.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.627+0000 7f3d710f3700 1 -- 172.21.15.158:0/1840649914 >> 172.21.15.158:0/1840649914 conn(0x7f3d6c0cc5a0 msgr2=0x7f3d6c0cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:44.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.627+0000 7f3d710f3700 1 -- 172.21.15.158:0/1840649914 shutdown_connections 2023-10-16T11:00:44.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.627+0000 7f3d710f3700 1 -- 172.21.15.158:0/1840649914 wait complete. 2023-10-16T11:00:44.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.628+0000 7f3d710f3700 1 Processor -- start 2023-10-16T11:00:44.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.628+0000 7f3d710f3700 1 -- start start 2023-10-16T11:00:44.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d710f3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54003a00 0x7f3d6c0c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d710f3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d6c0b31d0 0x7f3d6c14e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d710f3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d6c151800 0x7f3d6c151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d710f3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c14e5e0 con 0x7f3d6c151800 2023-10-16T11:00:44.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d710f3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c0af920 con 0x7f3d6c0b31d0 2023-10-16T11:00:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d710f3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c0b04a0 con 0x7f3d54003a00 2023-10-16T11:00:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d6c151800 0x7f3d6c151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d6c151800 0x7f3d6c151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33194/0 (socket says 172.21.15.158:33194) 2023-10-16T11:00:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 -- 172.21.15.158:0/9817564 learned_addr learned my addr 172.21.15.158:0/9817564 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:44.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54003a00 0x7f3d6c0c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6a59c700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d6c0b31d0 0x7f3d6c14e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 -- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54003a00 msgr2=0x7f3d6c0c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54003a00 0x7f3d6c0c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 -- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d6c0b31d0 msgr2=0x7f3d6c14e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d6c0b31d0 0x7f3d6c14e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.629+0000 7f3d6b59e700 1 -- 172.21.15.158:0/9817564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c0b3650 con 0x7f3d6c151800 2023-10-16T11:00:44.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.630+0000 7f3d6b59e700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d6c151800 0x7f3d6c151c80 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0cd670 tx=0x7f3d64011640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:44.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.630+0000 7f3d537fe700 1 -- 172.21.15.158:0/9817564 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64003c20 con 0x7f3d6c151800 2023-10-16T11:00:44.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.630+0000 7f3d537fe700 1 -- 172.21.15.158:0/9817564 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d64003d80 con 0x7f3d6c151800 2023-10-16T11:00:44.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.630+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c0afb70 con 0x7f3d6c151800 2023-10-16T11:00:44.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.630+0000 7f3d537fe700 1 -- 172.21.15.158:0/9817564 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64004250 con 0x7f3d6c151800 2023-10-16T11:00:44.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.630+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c0b0130 con 0x7f3d6c151800 2023-10-16T11:00:44.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.631+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6c0681b0 con 0x7f3d6c151800 2023-10-16T11:00:44.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.631+0000 7f3d537fe700 1 -- 172.21.15.158:0/9817564 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3d640043b0 con 0x7f3d6c151800 2023-10-16T11:00:44.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.632+0000 7f3d537fe700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f3d5806b020 0x7f3d5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:44.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.632+0000 7f3d537fe700 1 -- 172.21.15.158:0/9817564 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f3d6408c830 con 0x7f3d6c151800 2023-10-16T11:00:44.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.632+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f3d5806b020 0x7f3d5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:44.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.632+0000 7f3d6ad9d700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f3d5806b020 0x7f3d5806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3d600090b0 tx=0x7f3d60008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:44.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.636+0000 7f3d537fe700 1 -- 172.21.15.158:0/9817564 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3d64057550 con 0x7f3d6c151800 2023-10-16T11:00:44.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.875+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d6c0b05e0 con 0x7f3d6c151800 2023-10-16T11:00:44.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.875+0000 7f3d537fe700 1 -- 172.21.15.158:0/9817564 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f3d6405ab70 con 0x7f3d6c151800 2023-10-16T11:00:44.877 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:44.877 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3392606972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3392606972},{"type":"v1","addr":"172.21.15.172:6834","nonce":3392606972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9930,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3321991635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3321991635},{"type":"v1","addr":"172.21.15.158:6836","nonce":3321991635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4013459160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4013459160},{"type":"v1","addr":"172.21.15.158:6837","nonce":4013459160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:00:40.840318+0000","modified":"2023-10-16T11:00:43.870189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37712":{"gid":37712,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/551179403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":551179403},{"type":"v1","addr":"172.21.15.172:6835","nonce":551179403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T11:00:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f3d5806b020 msgr2=0x7f3d5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f3d5806b020 0x7f3d5806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3d600090b0 tx=0x7f3d60008030 comp rx=0 tx=0).stop 2023-10-16T11:00:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d6c151800 msgr2=0x7f3d6c151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d6c151800 0x7f3d6c151c80 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0cd670 tx=0x7f3d64011640 comp rx=0 tx=0).stop 2023-10-16T11:00:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 shutdown_connections 2023-10-16T11:00:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d6c151800 0x7f3d6c151c80 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d54003a00 0x7f3d6c0c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f3d5806b020 0x7f3d5806d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 --2- 172.21.15.158:0/9817564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d6c0b31d0 0x7f3d6c14e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 >> 172.21.15.158:0/9817564 conn(0x7f3d6c0cc5a0 msgr2=0x7f3d6c0cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 shutdown_connections 2023-10-16T11:00:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:44.878+0000 7f3d710f3700 1 -- 172.21.15.158:0/9817564 wait complete. 2023-10-16T11:00:44.882 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 85 2023-10-16T11:00:44.892 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:45.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.011+0000 7f608b6a4700 1 Processor -- start 2023-10-16T11:00:45.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.012+0000 7f608b6a4700 1 -- start start 2023-10-16T11:00:45.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.012+0000 7f608b6a4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840ce640 0x7f60840cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.012+0000 7f608b6a4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f60840c65b0 0x7f608406f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.012+0000 7f608b6a4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60840cd600 0x7f60840cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.012+0000 7f608b6a4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608415e480 con 0x7f60840cd600 2023-10-16T11:00:45.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.012+0000 7f608b6a4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60840cef60 con 0x7f60840c65b0 2023-10-16T11:00:45.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.012+0000 7f608b6a4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608415d800 con 0x7f60840ce640 2023-10-16T11:00:45.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60840cd600 0x7f60840cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f60840cd600 0x7f60840cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33200/0 (socket says 172.21.15.158:33200) 2023-10-16T11:00:45.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 -- 172.21.15.158:0/188532163 learned_addr learned my addr 172.21.15.158:0/188532163 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:45.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089440700 1 --2- 172.21.15.158:0/188532163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840ce640 0x7f60840cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6088c3f700 1 --2- 172.21.15.158:0/188532163 >> v2:172.21.15.172:3300/0 conn(0x7f60840c65b0 0x7f608406f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 -- 172.21.15.158:0/188532163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840ce640 msgr2=0x7f60840cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:45.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 --2- 172.21.15.158:0/188532163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840ce640 0x7f60840cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 -- 172.21.15.158:0/188532163 >> v2:172.21.15.172:3300/0 conn(0x7f60840c65b0 msgr2=0x7f608406f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:45.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 --2- 172.21.15.158:0/188532163 >> v2:172.21.15.172:3300/0 conn(0x7f60840c65b0 0x7f608406f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 -- 172.21.15.158:0/188532163 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608415dae0 con 0x7f60840cd600 2023-10-16T11:00:45.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089440700 1 --2- 172.21.15.158:0/188532163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840ce640 0x7f60840cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:45.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6088c3f700 1 --2- 172.21.15.158:0/188532163 >> v2:172.21.15.172:3300/0 conn(0x7f60840c65b0 0x7f608406f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:45.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f6089c41700 1 --2- 172.21.15.158:0/188532163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840cd600 0x7f60840cda90 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f60840b26f0 tx=0x7f607c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644fe56e61814aef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f607bfff700 1 -- 172.21.15.158:0/188532163 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c00e070 con 0x7f60840cd600 2023-10-16T11:00:45.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f607bfff700 1 -- 172.21.15.158:0/188532163 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c00b9c0 con 0x7f60840cd600 2023-10-16T11:00:45.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.013+0000 7f607bfff700 1 -- 172.21.15.158:0/188532163 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c00bd70 con 0x7f60840cd600 2023-10-16T11:00:45.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 -- 172.21.15.158:0/188532163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840cd600 msgr2=0x7f60840cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 --2- 172.21.15.158:0/188532163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840cd600 0x7f60840cda90 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f60840b26f0 tx=0x7f607c00b330 comp rx=0 tx=0).stop 2023-10-16T11:00:45.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 -- 172.21.15.158:0/188532163 shutdown_connections 2023-10-16T11:00:45.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 --2- 172.21.15.158:0/188532163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840cd600 0x7f60840cda90 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 --2- 172.21.15.158:0/188532163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840ce640 0x7f60840cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 --2- 172.21.15.158:0/188532163 >> v2:172.21.15.172:3300/0 conn(0x7f60840c65b0 0x7f608406f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 -- 172.21.15.158:0/188532163 >> 172.21.15.158:0/188532163 conn(0x7f60840cc4b0 msgr2=0x7f60840cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:45.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 -- 172.21.15.158:0/188532163 shutdown_connections 2023-10-16T11:00:45.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.014+0000 7f608b6a4700 1 -- 172.21.15.158:0/188532163 wait complete. 2023-10-16T11:00:45.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.015+0000 7f608b6a4700 1 Processor -- start 2023-10-16T11:00:45.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.015+0000 7f608b6a4700 1 -- start start 2023-10-16T11:00:45.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f608b6a4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840c65b0 0x7f6084158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f608b6a4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60840cd600 0x7f6084140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f608b6a4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840ce640 0x7f60841413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f608b6a4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840c7130 con 0x7f60840ce640 2023-10-16T11:00:45.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f608b6a4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840c6b00 con 0x7f60840cd600 2023-10-16T11:00:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f608b6a4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60840ae7b0 con 0x7f60840c65b0 2023-10-16T11:00:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840ce640 0x7f60841413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840ce640 0x7f60841413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33206/0 (socket says 172.21.15.158:33206) 2023-10-16T11:00:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 -- 172.21.15.158:0/2077247250 learned_addr learned my addr 172.21.15.158:0/2077247250 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089440700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840c65b0 0x7f6084158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6088c3f700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60840cd600 0x7f6084140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 -- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840c65b0 msgr2=0x7f6084158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840c65b0 0x7f6084158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 -- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60840cd600 msgr2=0x7f6084140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60840cd600 0x7f6084140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 -- 172.21.15.158:0/2077247250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c009b30 con 0x7f60840ce640 2023-10-16T11:00:45.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6089c41700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840ce640 0x7f60841413e0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f607c00b370 tx=0x7f607c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6079ffb700 1 -- 172.21.15.158:0/2077247250 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c004310 con 0x7f60840ce640 2023-10-16T11:00:45.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6079ffb700 1 -- 172.21.15.158:0/2077247250 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c004470 con 0x7f60840ce640 2023-10-16T11:00:45.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60840aea90 con 0x7f60840ce640 2023-10-16T11:00:45.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.016+0000 7f6079ffb700 1 -- 172.21.15.158:0/2077247250 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c004820 con 0x7f60840ce640 2023-10-16T11:00:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.017+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60840b2750 con 0x7f60840ce640 2023-10-16T11:00:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.018+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60840b2940 con 0x7f60840ce640 2023-10-16T11:00:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.018+0000 7f6079ffb700 1 -- 172.21.15.158:0/2077247250 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f607c01b430 con 0x7f60840ce640 2023-10-16T11:00:45.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.018+0000 7f6079ffb700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f606806b020 0x7f606806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.019+0000 7f6089440700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f606806b020 0x7f606806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.019+0000 7f6079ffb700 1 -- 172.21.15.158:0/2077247250 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f607c08adf0 con 0x7f60840ce640 2023-10-16T11:00:45.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.019+0000 7f6089440700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f606806b020 0x7f606806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6074006fd0 tx=0x7f6074008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.023+0000 7f6079ffb700 1 -- 172.21.15.158:0/2077247250 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f607c055a00 con 0x7f60840ce640 2023-10-16T11:00:45.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.216+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60840681b0 con 0x7f60840ce640 2023-10-16T11:00:45.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.217+0000 7f6079ffb700 1 -- 172.21.15.158:0/2077247250 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+17426 (secure 0 0 0) 0x7f607c01f020 con 0x7f60840ce640 2023-10-16T11:00:45.219 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:45.219 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":172,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:42.445964+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:00:38.634284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T11:00:39.812150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:45.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f606806b020 msgr2=0x7f606806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f606806b020 0x7f606806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6074006fd0 tx=0x7f6074008040 comp rx=0 tx=0).stop 2023-10-16T11:00:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840ce640 msgr2=0x7f60841413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840ce640 0x7f60841413e0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f607c00b370 tx=0x7f607c004ad0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 shutdown_connections 2023-10-16T11:00:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60840ce640 0x7f60841413e0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f60840c65b0 0x7f6084158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f606806b020 0x7f606806d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 --2- 172.21.15.158:0/2077247250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60840cd600 0x7f6084140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 >> 172.21.15.158:0/2077247250 conn(0x7f60840cc4b0 msgr2=0x7f60840af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:45.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 shutdown_connections 2023-10-16T11:00:45.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.219+0000 7f608b6a4700 1 -- 172.21.15.158:0/2077247250 wait complete. 2023-10-16T11:00:45.233 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:00:45.234 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:00:45.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.353+0000 7fee56530700 1 Processor -- start 2023-10-16T11:00:45.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.353+0000 7fee56530700 1 -- start start 2023-10-16T11:00:45.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.354+0000 7fee56530700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee500cdee0 0x7fee5015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.354+0000 7fee56530700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5015c5d0 con 0x7fee500b6e10 2023-10-16T11:00:45.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.354+0000 7fee56530700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5015c8b0 con 0x7fee500ce2c0 2023-10-16T11:00:45.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.354+0000 7fee56530700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5015d2c0 con 0x7fee500cdee0 2023-10-16T11:00:45.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.354+0000 7fee4f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fee500b6e10 0x7fee501521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33386/0 (socket says 172.21.15.158:33386) 2023-10-16T11:00:45.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.354+0000 7fee4f7fe700 1 -- 172.21.15.158:0/1908459399 learned_addr learned my addr 172.21.15.158:0/1908459399 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:45.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.354+0000 7fee4ffff700 1 --2- 172.21.15.158:0/1908459399 >> v2:172.21.15.172:3300/0 conn(0x7fee500cdee0 0x7fee5015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781085376 0 0) 0x7fee5015c5d0 con 0x7fee500b6e10 2023-10-16T11:00:45.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c0035b0 con 0x7fee500b6e10 2023-10-16T11:00:45.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1006100074 0 0) 0x7fee3c0035b0 con 0x7fee500b6e10 2023-10-16T11:00:45.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 >> v2:172.21.15.172:3300/0 conn(0x7fee500cdee0 msgr2=0x7fee5015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:45.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 --2- 172.21.15.158:0/1908459399 >> v2:172.21.15.172:3300/0 conn(0x7fee500cdee0 0x7fee5015bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 >> v1:172.21.15.172:6789/0 conn(0x7fee500ce2c0 legacy=0x7fee500cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5015cb90 con 0x7fee500b6e10 2023-10-16T11:00:45.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fee40001670 con 0x7fee500b6e10 2023-10-16T11:00:45.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 >> v1:172.21.15.158:6789/0 conn(0x7fee500b6e10 legacy=0x7fee501521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee3c003a00 0x7fee3c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee3c004850 0x7fee3c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c005210 0x7fee3c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5015d2c0 con 0x7fee3c004850 2023-10-16T11:00:45.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c00c1b0 con 0x7fee3c005210 2023-10-16T11:00:45.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c00c370 con 0x7fee3c003a00 2023-10-16T11:00:45.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4f7fe700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee3c004850 0x7fee3c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4ffff700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee3c003a00 0x7fee3c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee54acd700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c005210 0x7fee3c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4f7fe700 1 -- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee3c003a00 msgr2=0x7fee3c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:45.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4f7fe700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee3c003a00 0x7fee3c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4f7fe700 1 -- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c005210 msgr2=0x7fee3c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:45.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4f7fe700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c005210 0x7fee3c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee54acd700 1 -- 172.21.15.158:0/1908459399 reap_dead start 2023-10-16T11:00:45.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.355+0000 7fee4f7fe700 1 -- 172.21.15.158:0/1908459399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c0035b0 con 0x7fee3c004850 2023-10-16T11:00:45.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee4f7fe700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee3c004850 0x7fee3c004cd0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fee4000c240 tx=0x7fee3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0be62b7e3902835 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40007c10 con 0x7fee3c004850 2023-10-16T11:00:45.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4000cde0 con 0x7fee3c004850 2023-10-16T11:00:45.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee4effd700 1 -- 172.21.15.158:0/1908459399 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40005ba0 con 0x7fee3c004850 2023-10-16T11:00:45.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee56530700 1 -- 172.21.15.158:0/1908459399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee3c004850 msgr2=0x7fee3c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee56530700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee3c004850 0x7fee3c004cd0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fee4000c240 tx=0x7fee3c005880 comp rx=0 tx=0).stop 2023-10-16T11:00:45.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee56530700 1 -- 172.21.15.158:0/1908459399 shutdown_connections 2023-10-16T11:00:45.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee56530700 1 --2- 172.21.15.158:0/1908459399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee3c004850 0x7fee3c004cd0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee56530700 1 -- 172.21.15.158:0/1908459399 >> 172.21.15.158:0/1908459399 conn(0x7fee5006a8d0 msgr2=0x7fee5006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:45.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee56530700 1 -- 172.21.15.158:0/1908459399 shutdown_connections 2023-10-16T11:00:45.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.356+0000 7fee56530700 1 -- 172.21.15.158:0/1908459399 wait complete. 2023-10-16T11:00:45.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.357+0000 7fee56530700 1 Processor -- start 2023-10-16T11:00:45.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee56530700 1 -- start start 2023-10-16T11:00:45.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee56530700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c004850 0x7fee5014ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee56530700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee500c5c80 0x7fee5014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee56530700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee500c6c70 0x7fee50154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee56530700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee500c57f0 con 0x7fee500c6c70 2023-10-16T11:00:45.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee56530700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5014aa50 con 0x7fee3c004850 2023-10-16T11:00:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee56530700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5014e240 con 0x7fee500c5c80 2023-10-16T11:00:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.358+0000 7fee54acd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee500c6c70 0x7fee50154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee500c6c70 0x7fee50154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33224/0 (socket says 172.21.15.158:33224) 2023-10-16T11:00:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 -- 172.21.15.158:0/1644706045 learned_addr learned my addr 172.21.15.158:0/1644706045 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee4f7fe700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee500c5c80 0x7fee5014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee4ffff700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c004850 0x7fee5014ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 -- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee500c5c80 msgr2=0x7fee5014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee500c5c80 0x7fee5014dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 -- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c004850 msgr2=0x7fee5014ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c004850 0x7fee5014ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 -- 172.21.15.158:0/1644706045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500c4ec0 con 0x7fee500c6c70 2023-10-16T11:00:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee4f7fe700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee500c5c80 0x7fee5014dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee54acd700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee500c6c70 0x7fee50154450 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fee500b72a0 tx=0x7fee4800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee4cff9700 1 -- 172.21.15.158:0/1644706045 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48003cd0 con 0x7fee500c6c70 2023-10-16T11:00:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee4cff9700 1 -- 172.21.15.158:0/1644706045 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee48003e30 con 0x7fee500c6c70 2023-10-16T11:00:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee500c51a0 con 0x7fee500c6c70 2023-10-16T11:00:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee4cff9700 1 -- 172.21.15.158:0/1644706045 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4800d9b0 con 0x7fee500c6c70 2023-10-16T11:00:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.359+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee500c7100 con 0x7fee500c6c70 2023-10-16T11:00:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.360+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee500c59a0 con 0x7fee500c6c70 2023-10-16T11:00:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.361+0000 7fee4cff9700 1 -- 172.21.15.158:0/1644706045 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee4800db10 con 0x7fee500c6c70 2023-10-16T11:00:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.361+0000 7fee4cff9700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fee3006b020 0x7fee3006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.361+0000 7fee4cff9700 1 -- 172.21.15.158:0/1644706045 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fee48089b70 con 0x7fee500c6c70 2023-10-16T11:00:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.361+0000 7fee4ffff700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fee3006b020 0x7fee3006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.362+0000 7fee4ffff700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fee3006b020 0x7fee3006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee38009b20 tx=0x7fee38008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.366+0000 7fee4cff9700 1 -- 172.21.15.158:0/1644706045 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee500c59a0 con 0x7fee500c6c70 2023-10-16T11:00:45.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.556+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 --> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee5015cee0 con 0x7fee3006b020 2023-10-16T11:00:45.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.565+0000 7fee4cff9700 1 -- 172.21.15.158:0/1644706045 <== mgr.9690 v2:172.21.15.158:6832/1293733893 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290643 (secure 0 0 0) 0x7fee380137d0 con 0x7fee3006b020 2023-10-16T11:00:45.569 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:45.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fee3006b020 msgr2=0x7fee3006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fee3006b020 0x7fee3006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee38009b20 tx=0x7fee38008040 comp rx=0 tx=0).stop 2023-10-16T11:00:45.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee500c6c70 msgr2=0x7fee50154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee500c6c70 0x7fee50154450 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fee500b72a0 tx=0x7fee4800d150 comp rx=0 tx=0).stop 2023-10-16T11:00:45.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 shutdown_connections 2023-10-16T11:00:45.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee500c6c70 0x7fee50154450 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fee3006b020 0x7fee3006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee3c004850 0x7fee5014ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 --2- 172.21.15.158:0/1644706045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee500c5c80 0x7fee5014dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 >> 172.21.15.158:0/1644706045 conn(0x7fee5006a8d0 msgr2=0x7fee500b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:45.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 shutdown_connections 2023-10-16T11:00:45.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.568+0000 7fee56530700 1 -- 172.21.15.158:0/1644706045 wait complete. 2023-10-16T11:00:45.574 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:00:45.581 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T11:00:44.903932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":372868,"kb_used_data":17940,"kb_used_omap":12,"kb_used_meta":354803,"kb_avail":754601852,"statfs":{"total":773094113280,"available":772712296448,"internally_reserved":0,"allocated":18370560,"data_stored":10253596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":363318839},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.470052"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453490+0000","last_change":"2023-10-16T11:00:41.477008+0000","last_active":"2023-10-16T11:00:42.453490+0000","last_peered":"2023-10-16T11:00:42.453490+0000","last_clean":"2023-10-16T11:00:42.453490+0000","last_became_active":"2023-10-16T11:00:41.476709+0000","last_became_peered":"2023-10-16T11:00:41.476709+0000","last_unstale":"2023-10-16T11:00:42.453490+0000","last_undegraded":"2023-10-16T11:00:42.453490+0000","last_fullsized":"2023-10-16T11:00:42.453490+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:53.069037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.472343+0000","last_change":"2023-10-16T11:00:41.471591+0000","last_active":"2023-10-16T11:00:41.472343+0000","last_peered":"2023-10-16T11:00:41.472343+0000","last_clean":"2023-10-16T11:00:41.472343+0000","last_became_active":"2023-10-16T11:00:41.471350+0000","last_became_peered":"2023-10-16T11:00:41.471350+0000","last_unstale":"2023-10-16T11:00:41.472343+0000","last_undegraded":"2023-10-16T11:00:41.472343+0000","last_fullsized":"2023-10-16T11:00:41.472343+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:17.745318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463282+0000","last_change":"2023-10-16T11:00:41.460551+0000","last_active":"2023-10-16T11:00:41.463282+0000","last_peered":"2023-10-16T11:00:41.463282+0000","last_clean":"2023-10-16T11:00:41.463282+0000","last_became_active":"2023-10-16T11:00:41.460282+0000","last_became_peered":"2023-10-16T11:00:41.460282+0000","last_unstale":"2023-10-16T11:00:41.463282+0000","last_undegraded":"2023-10-16T11:00:41.463282+0000","last_fullsized":"2023-10-16T11:00:41.463282+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:11:36.916150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471633+0000","last_change":"2023-10-16T11:00:41.465831+0000","last_active":"2023-10-16T11:00:41.471633+0000","last_peered":"2023-10-16T11:00:41.471633+0000","last_clean":"2023-10-16T11:00:41.471633+0000","last_became_active":"2023-10-16T11:00:41.465575+0000","last_became_peered":"2023-10-16T11:00:41.465575+0000","last_unstale":"2023-10-16T11:00:41.471633+0000","last_undegraded":"2023-10-16T11:00:41.471633+0000","last_fullsized":"2023-10-16T11:00:41.471633+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:49.958609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462787+0000","last_change":"2023-10-16T11:00:41.458769+0000","last_active":"2023-10-16T11:00:41.462787+0000","last_peered":"2023-10-16T11:00:41.462787+0000","last_clean":"2023-10-16T11:00:41.462787+0000","last_became_active":"2023-10-16T11:00:41.458521+0000","last_became_peered":"2023-10-16T11:00:41.458521+0000","last_unstale":"2023-10-16T11:00:41.462787+0000","last_undegraded":"2023-10-16T11:00:41.462787+0000","last_fullsized":"2023-10-16T11:00:41.462787+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:10.376601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.466788+0000","last_change":"2023-10-16T11:00:41.466213+0000","last_active":"2023-10-16T11:00:41.466788+0000","last_peered":"2023-10-16T11:00:41.466788+0000","last_clean":"2023-10-16T11:00:41.466788+0000","last_became_active":"2023-10-16T11:00:41.465883+0000","last_became_peered":"2023-10-16T11:00:41.465883+0000","last_unstale":"2023-10-16T11:00:41.466788+0000","last_undegraded":"2023-10-16T11:00:41.466788+0000","last_fullsized":"2023-10-16T11:00:41.466788+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:10.193252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453294+0000","last_change":"2023-10-16T11:00:41.461538+0000","last_active":"2023-10-16T11:00:42.453294+0000","last_peered":"2023-10-16T11:00:42.453294+0000","last_clean":"2023-10-16T11:00:42.453294+0000","last_became_active":"2023-10-16T11:00:41.458949+0000","last_became_peered":"2023-10-16T11:00:41.458949+0000","last_unstale":"2023-10-16T11:00:42.453294+0000","last_undegraded":"2023-10-16T11:00:42.453294+0000","last_fullsized":"2023-10-16T11:00:42.453294+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:10.376601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451783+0000","last_change":"2023-10-16T11:00:41.459242+0000","last_active":"2023-10-16T11:00:42.451783+0000","last_peered":"2023-10-16T11:00:42.451783+0000","last_clean":"2023-10-16T11:00:42.451783+0000","last_became_active":"2023-10-16T11:00:41.458953+0000","last_became_peered":"2023-10-16T11:00:41.458953+0000","last_unstale":"2023-10-16T11:00:42.451783+0000","last_undegraded":"2023-10-16T11:00:42.451783+0000","last_fullsized":"2023-10-16T11:00:42.451783+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:24:55.187152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463998+0000","last_change":"2023-10-16T11:00:41.462286+0000","last_active":"2023-10-16T11:00:41.463998+0000","last_peered":"2023-10-16T11:00:41.463998+0000","last_clean":"2023-10-16T11:00:41.463998+0000","last_became_active":"2023-10-16T11:00:41.462015+0000","last_became_peered":"2023-10-16T11:00:41.462015+0000","last_unstale":"2023-10-16T11:00:41.463998+0000","last_undegraded":"2023-10-16T11:00:41.463998+0000","last_fullsized":"2023-10-16T11:00:41.463998+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:52.568190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464030+0000","last_change":"2023-10-16T11:00:41.459684+0000","last_active":"2023-10-16T11:00:41.464030+0000","last_peered":"2023-10-16T11:00:41.464030+0000","last_clean":"2023-10-16T11:00:41.464030+0000","last_became_active":"2023-10-16T11:00:41.459422+0000","last_became_peered":"2023-10-16T11:00:41.459422+0000","last_unstale":"2023-10-16T11:00:41.464030+0000","last_undegraded":"2023-10-16T11:00:41.464030+0000","last_fullsized":"2023-10-16T11:00:41.464030+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:53.069037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462834+0000","last_change":"2023-10-16T11:00:41.457993+0000","last_active":"2023-10-16T11:00:41.462834+0000","last_peered":"2023-10-16T11:00:41.462834+0000","last_clean":"2023-10-16T11:00:41.462834+0000","last_became_active":"2023-10-16T11:00:41.457232+0000","last_became_peered":"2023-10-16T11:00:41.457232+0000","last_unstale":"2023-10-16T11:00:41.462834+0000","last_undegraded":"2023-10-16T11:00:41.462834+0000","last_fullsized":"2023-10-16T11:00:41.462834+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:34.345528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471657+0000","last_change":"2023-10-16T11:00:41.466799+0000","last_active":"2023-10-16T11:00:41.471657+0000","last_peered":"2023-10-16T11:00:41.471657+0000","last_clean":"2023-10-16T11:00:41.471657+0000","last_became_active":"2023-10-16T11:00:41.466512+0000","last_became_peered":"2023-10-16T11:00:41.466512+0000","last_unstale":"2023-10-16T11:00:41.471657+0000","last_undegraded":"2023-10-16T11:00:41.471657+0000","last_fullsized":"2023-10-16T11:00:41.471657+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:53.923623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463155+0000","last_change":"2023-10-16T11:00:41.459488+0000","last_active":"2023-10-16T11:00:41.463155+0000","last_peered":"2023-10-16T11:00:41.463155+0000","last_clean":"2023-10-16T11:00:41.463155+0000","last_became_active":"2023-10-16T11:00:41.459214+0000","last_became_peered":"2023-10-16T11:00:41.459214+0000","last_unstale":"2023-10-16T11:00:41.463155+0000","last_undegraded":"2023-10-16T11:00:41.463155+0000","last_fullsized":"2023-10-16T11:00:41.463155+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:41.526891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473394+0000","last_change":"2023-10-16T11:00:41.470475+0000","last_active":"2023-10-16T11:00:41.473394+0000","last_peered":"2023-10-16T11:00:41.473394+0000","last_clean":"2023-10-16T11:00:41.473394+0000","last_became_active":"2023-10-16T11:00:41.469828+0000","last_became_peered":"2023-10-16T11:00:41.469828+0000","last_unstale":"2023-10-16T11:00:41.473394+0000","last_undegraded":"2023-10-16T11:00:41.473394+0000","last_fullsized":"2023-10-16T11:00:41.473394+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:48.349715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.472146+0000","last_change":"2023-10-16T11:00:41.471040+0000","last_active":"2023-10-16T11:00:41.472146+0000","last_peered":"2023-10-16T11:00:41.472146+0000","last_clean":"2023-10-16T11:00:41.472146+0000","last_became_active":"2023-10-16T11:00:41.470747+0000","last_became_peered":"2023-10-16T11:00:41.470747+0000","last_unstale":"2023-10-16T11:00:41.472146+0000","last_undegraded":"2023-10-16T11:00:41.472146+0000","last_fullsized":"2023-10-16T11:00:41.472146+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:57.876948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468210+0000","last_change":"2023-10-16T11:00:41.467035+0000","last_active":"2023-10-16T11:00:41.468210+0000","last_peered":"2023-10-16T11:00:41.468210+0000","last_clean":"2023-10-16T11:00:41.468210+0000","last_became_active":"2023-10-16T11:00:41.466784+0000","last_became_peered":"2023-10-16T11:00:41.466784+0000","last_unstale":"2023-10-16T11:00:41.468210+0000","last_undegraded":"2023-10-16T11:00:41.468210+0000","last_fullsized":"2023-10-16T11:00:41.468210+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:54.842603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.478263+0000","last_change":"2023-10-16T11:00:41.472267+0000","last_active":"2023-10-16T11:00:41.478263+0000","last_peered":"2023-10-16T11:00:41.478263+0000","last_clean":"2023-10-16T11:00:41.478263+0000","last_became_active":"2023-10-16T11:00:41.471848+0000","last_became_peered":"2023-10-16T11:00:41.471848+0000","last_unstale":"2023-10-16T11:00:41.478263+0000","last_undegraded":"2023-10-16T11:00:41.478263+0000","last_fullsized":"2023-10-16T11:00:41.478263+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:54.257655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.479047+0000","last_change":"2023-10-16T11:00:41.478064+0000","last_active":"2023-10-16T11:00:41.479047+0000","last_peered":"2023-10-16T11:00:41.479047+0000","last_clean":"2023-10-16T11:00:41.479047+0000","last_became_active":"2023-10-16T11:00:41.477789+0000","last_became_peered":"2023-10-16T11:00:41.477789+0000","last_unstale":"2023-10-16T11:00:41.479047+0000","last_undegraded":"2023-10-16T11:00:41.479047+0000","last_fullsized":"2023-10-16T11:00:41.479047+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:37.346803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449699+0000","last_change":"2023-10-16T11:00:41.469762+0000","last_active":"2023-10-16T11:00:42.449699+0000","last_peered":"2023-10-16T11:00:42.449699+0000","last_clean":"2023-10-16T11:00:42.449699+0000","last_became_active":"2023-10-16T11:00:41.469455+0000","last_became_peered":"2023-10-16T11:00:41.469455+0000","last_unstale":"2023-10-16T11:00:42.449699+0000","last_undegraded":"2023-10-16T11:00:42.449699+0000","last_fullsized":"2023-10-16T11:00:42.449699+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:53.876230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.465046+0000","last_change":"2023-10-16T11:00:41.461267+0000","last_active":"2023-10-16T11:00:41.465046+0000","last_peered":"2023-10-16T11:00:41.465046+0000","last_clean":"2023-10-16T11:00:41.465046+0000","last_became_active":"2023-10-16T11:00:41.460984+0000","last_became_peered":"2023-10-16T11:00:41.460984+0000","last_unstale":"2023-10-16T11:00:41.465046+0000","last_undegraded":"2023-10-16T11:00:41.465046+0000","last_fullsized":"2023-10-16T11:00:41.465046+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:16.445945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464024+0000","last_change":"2023-10-16T11:00:41.461959+0000","last_active":"2023-10-16T11:00:41.464024+0000","last_peered":"2023-10-16T11:00:41.464024+0000","last_clean":"2023-10-16T11:00:41.464024+0000","last_became_active":"2023-10-16T11:00:41.460926+0000","last_became_peered":"2023-10-16T11:00:41.460926+0000","last_unstale":"2023-10-16T11:00:41.464024+0000","last_undegraded":"2023-10-16T11:00:41.464024+0000","last_fullsized":"2023-10-16T11:00:41.464024+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:10.075072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453383+0000","last_change":"2023-10-16T11:00:41.464610+0000","last_active":"2023-10-16T11:00:42.453383+0000","last_peered":"2023-10-16T11:00:42.453383+0000","last_clean":"2023-10-16T11:00:42.453383+0000","last_became_active":"2023-10-16T11:00:41.464287+0000","last_became_peered":"2023-10-16T11:00:41.464287+0000","last_unstale":"2023-10-16T11:00:42.453383+0000","last_undegraded":"2023-10-16T11:00:42.453383+0000","last_fullsized":"2023-10-16T11:00:42.453383+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:57:28.884434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451572+0000","last_change":"2023-10-16T11:00:41.466639+0000","last_active":"2023-10-16T11:00:42.451572+0000","last_peered":"2023-10-16T11:00:42.451572+0000","last_clean":"2023-10-16T11:00:42.451572+0000","last_became_active":"2023-10-16T11:00:41.466383+0000","last_became_peered":"2023-10-16T11:00:41.466383+0000","last_unstale":"2023-10-16T11:00:42.451572+0000","last_undegraded":"2023-10-16T11:00:42.451572+0000","last_fullsized":"2023-10-16T11:00:42.451572+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:34.831144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.456643+0000","last_change":"2023-10-16T11:00:41.455674+0000","last_active":"2023-10-16T11:00:41.456643+0000","last_peered":"2023-10-16T11:00:41.456643+0000","last_clean":"2023-10-16T11:00:41.456643+0000","last_became_active":"2023-10-16T11:00:41.455364+0000","last_became_peered":"2023-10-16T11:00:41.455364+0000","last_unstale":"2023-10-16T11:00:41.456643+0000","last_undegraded":"2023-10-16T11:00:41.456643+0000","last_fullsized":"2023-10-16T11:00:41.456643+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:11:28.754077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453551+0000","last_change":"2023-10-16T11:00:41.481083+0000","last_active":"2023-10-16T11:00:42.453551+0000","last_peered":"2023-10-16T11:00:42.453551+0000","last_clean":"2023-10-16T11:00:42.453551+0000","last_became_active":"2023-10-16T11:00:41.480816+0000","last_became_peered":"2023-10-16T11:00:41.480816+0000","last_unstale":"2023-10-16T11:00:42.453551+0000","last_undegraded":"2023-10-16T11:00:42.453551+0000","last_fullsized":"2023-10-16T11:00:42.453551+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:52.568190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453717+0000","last_change":"2023-10-16T11:00:41.466489+0000","last_active":"2023-10-16T11:00:42.453717+0000","last_peered":"2023-10-16T11:00:42.453717+0000","last_clean":"2023-10-16T11:00:42.453717+0000","last_became_active":"2023-10-16T11:00:41.466202+0000","last_became_peered":"2023-10-16T11:00:41.466202+0000","last_unstale":"2023-10-16T11:00:42.453717+0000","last_undegraded":"2023-10-16T11:00:42.453717+0000","last_fullsized":"2023-10-16T11:00:42.453717+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:10.075072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.460060+0000","last_change":"2023-10-16T11:00:41.456615+0000","last_active":"2023-10-16T11:00:41.460060+0000","last_peered":"2023-10-16T11:00:41.460060+0000","last_clean":"2023-10-16T11:00:41.460060+0000","last_became_active":"2023-10-16T11:00:41.456353+0000","last_became_peered":"2023-10-16T11:00:41.456353+0000","last_unstale":"2023-10-16T11:00:41.460060+0000","last_undegraded":"2023-10-16T11:00:41.460060+0000","last_fullsized":"2023-10-16T11:00:41.460060+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.516140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462949+0000","last_change":"2023-10-16T11:00:41.460594+0000","last_active":"2023-10-16T11:00:41.462949+0000","last_peered":"2023-10-16T11:00:41.462949+0000","last_clean":"2023-10-16T11:00:41.462949+0000","last_became_active":"2023-10-16T11:00:41.460223+0000","last_became_peered":"2023-10-16T11:00:41.460223+0000","last_unstale":"2023-10-16T11:00:41.462949+0000","last_undegraded":"2023-10-16T11:00:41.462949+0000","last_fullsized":"2023-10-16T11:00:41.462949+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:44.082575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451841+0000","last_change":"2023-10-16T11:00:41.463809+0000","last_active":"2023-10-16T11:00:42.451841+0000","last_peered":"2023-10-16T11:00:42.451841+0000","last_clean":"2023-10-16T11:00:42.451841+0000","last_became_active":"2023-10-16T11:00:41.463182+0000","last_became_peered":"2023-10-16T11:00:41.463182+0000","last_unstale":"2023-10-16T11:00:42.451841+0000","last_undegraded":"2023-10-16T11:00:42.451841+0000","last_fullsized":"2023-10-16T11:00:42.451841+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:14.514749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468812+0000","last_change":"2023-10-16T11:00:41.468511+0000","last_active":"2023-10-16T11:00:41.468812+0000","last_peered":"2023-10-16T11:00:41.468812+0000","last_clean":"2023-10-16T11:00:41.468812+0000","last_became_active":"2023-10-16T11:00:41.468241+0000","last_became_peered":"2023-10-16T11:00:41.468241+0000","last_unstale":"2023-10-16T11:00:41.468812+0000","last_undegraded":"2023-10-16T11:00:41.468812+0000","last_fullsized":"2023-10-16T11:00:41.468812+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:02.241701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468304+0000","last_change":"2023-10-16T11:00:41.467197+0000","last_active":"2023-10-16T11:00:41.468304+0000","last_peered":"2023-10-16T11:00:41.468304+0000","last_clean":"2023-10-16T11:00:41.468304+0000","last_became_active":"2023-10-16T11:00:41.466709+0000","last_became_peered":"2023-10-16T11:00:41.466709+0000","last_unstale":"2023-10-16T11:00:41.468304+0000","last_undegraded":"2023-10-16T11:00:41.468304+0000","last_fullsized":"2023-10-16T11:00:41.468304+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:57.181305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451596+0000","last_change":"2023-10-16T11:00:41.462986+0000","last_active":"2023-10-16T11:00:42.451596+0000","last_peered":"2023-10-16T11:00:42.451596+0000","last_clean":"2023-10-16T11:00:42.451596+0000","last_became_active":"2023-10-16T11:00:41.462665+0000","last_became_peered":"2023-10-16T11:00:41.462665+0000","last_unstale":"2023-10-16T11:00:42.451596+0000","last_undegraded":"2023-10-16T11:00:42.451596+0000","last_fullsized":"2023-10-16T11:00:42.451596+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:17.941549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463413+0000","last_change":"2023-10-16T11:00:41.460338+0000","last_active":"2023-10-16T11:00:41.463413+0000","last_peered":"2023-10-16T11:00:41.463413+0000","last_clean":"2023-10-16T11:00:41.463413+0000","last_became_active":"2023-10-16T11:00:41.460070+0000","last_became_peered":"2023-10-16T11:00:41.460070+0000","last_unstale":"2023-10-16T11:00:41.463413+0000","last_undegraded":"2023-10-16T11:00:41.463413+0000","last_fullsized":"2023-10-16T11:00:41.463413+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:08.498765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"171'4","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480397+0000","last_change":"2023-10-16T11:00:40.456582+0000","last_active":"2023-10-16T11:00:41.480397+0000","last_peered":"2023-10-16T11:00:41.480397+0000","last_clean":"2023-10-16T11:00:41.480397+0000","last_became_active":"2023-10-16T11:00:40.456323+0000","last_became_peered":"2023-10-16T11:00:40.456323+0000","last_unstale":"2023-10-16T11:00:41.480397+0000","last_undegraded":"2023-10-16T11:00:41.480397+0000","last_fullsized":"2023-10-16T11:00:41.480397+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:52.872001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471874+0000","last_change":"2023-10-16T11:00:41.470993+0000","last_active":"2023-10-16T11:00:41.471874+0000","last_peered":"2023-10-16T11:00:41.471874+0000","last_clean":"2023-10-16T11:00:41.471874+0000","last_became_active":"2023-10-16T11:00:41.470691+0000","last_became_peered":"2023-10-16T11:00:41.470691+0000","last_unstale":"2023-10-16T11:00:41.471874+0000","last_undegraded":"2023-10-16T11:00:41.471874+0000","last_fullsized":"2023-10-16T11:00:41.471874+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:50.860884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"171'1","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449094+0000","last_change":"2023-10-16T11:00:40.452203+0000","last_active":"2023-10-16T11:00:42.449094+0000","last_peered":"2023-10-16T11:00:42.449094+0000","last_clean":"2023-10-16T11:00:42.449094+0000","last_became_active":"2023-10-16T11:00:40.451986+0000","last_became_peered":"2023-10-16T11:00:40.451986+0000","last_unstale":"2023-10-16T11:00:42.449094+0000","last_undegraded":"2023-10-16T11:00:42.449094+0000","last_fullsized":"2023-10-16T11:00:42.449094+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:09.372371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.452763+0000","last_change":"2023-10-16T11:00:41.471082+0000","last_active":"2023-10-16T11:00:42.452763+0000","last_peered":"2023-10-16T11:00:42.452763+0000","last_clean":"2023-10-16T11:00:42.452763+0000","last_became_active":"2023-10-16T11:00:41.470664+0000","last_became_peered":"2023-10-16T11:00:41.470664+0000","last_unstale":"2023-10-16T11:00:42.452763+0000","last_undegraded":"2023-10-16T11:00:42.452763+0000","last_fullsized":"2023-10-16T11:00:42.452763+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:10.863889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"171'2","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449793+0000","last_change":"2023-10-16T11:00:40.457440+0000","last_active":"2023-10-16T11:00:42.449793+0000","last_peered":"2023-10-16T11:00:42.449793+0000","last_clean":"2023-10-16T11:00:42.449793+0000","last_became_active":"2023-10-16T11:00:40.457213+0000","last_became_peered":"2023-10-16T11:00:40.457213+0000","last_unstale":"2023-10-16T11:00:42.449793+0000","last_undegraded":"2023-10-16T11:00:42.449793+0000","last_fullsized":"2023-10-16T11:00:42.449793+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:09.070842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.450039+0000","last_change":"2023-10-16T11:00:41.466643+0000","last_active":"2023-10-16T11:00:42.450039+0000","last_peered":"2023-10-16T11:00:42.450039+0000","last_clean":"2023-10-16T11:00:42.450039+0000","last_became_active":"2023-10-16T11:00:41.465951+0000","last_became_peered":"2023-10-16T11:00:41.465951+0000","last_unstale":"2023-10-16T11:00:42.450039+0000","last_undegraded":"2023-10-16T11:00:42.450039+0000","last_fullsized":"2023-10-16T11:00:42.450039+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:48.788715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480311+0000","last_change":"2023-10-16T11:00:40.451643+0000","last_active":"2023-10-16T11:00:41.480311+0000","last_peered":"2023-10-16T11:00:41.480311+0000","last_clean":"2023-10-16T11:00:41.480311+0000","last_became_active":"2023-10-16T11:00:40.451380+0000","last_became_peered":"2023-10-16T11:00:40.451380+0000","last_unstale":"2023-10-16T11:00:41.480311+0000","last_undegraded":"2023-10-16T11:00:41.480311+0000","last_fullsized":"2023-10-16T11:00:41.480311+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:22.248351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.465075+0000","last_change":"2023-10-16T11:00:41.463587+0000","last_active":"2023-10-16T11:00:41.465075+0000","last_peered":"2023-10-16T11:00:41.465075+0000","last_clean":"2023-10-16T11:00:41.465075+0000","last_became_active":"2023-10-16T11:00:41.463407+0000","last_became_peered":"2023-10-16T11:00:41.463407+0000","last_unstale":"2023-10-16T11:00:41.465075+0000","last_undegraded":"2023-10-16T11:00:41.465075+0000","last_fullsized":"2023-10-16T11:00:41.465075+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:47.907213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"171'1","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451952+0000","last_change":"2023-10-16T11:00:40.458502+0000","last_active":"2023-10-16T11:00:42.451952+0000","last_peered":"2023-10-16T11:00:42.451952+0000","last_clean":"2023-10-16T11:00:42.451952+0000","last_became_active":"2023-10-16T11:00:40.458211+0000","last_became_peered":"2023-10-16T11:00:40.458211+0000","last_unstale":"2023-10-16T11:00:42.451952+0000","last_undegraded":"2023-10-16T11:00:42.451952+0000","last_fullsized":"2023-10-16T11:00:42.451952+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:34.299145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473260+0000","last_change":"2023-10-16T11:00:41.471193+0000","last_active":"2023-10-16T11:00:41.473260+0000","last_peered":"2023-10-16T11:00:41.473260+0000","last_clean":"2023-10-16T11:00:41.473260+0000","last_became_active":"2023-10-16T11:00:41.470752+0000","last_became_peered":"2023-10-16T11:00:41.470752+0000","last_unstale":"2023-10-16T11:00:41.473260+0000","last_undegraded":"2023-10-16T11:00:41.473260+0000","last_fullsized":"2023-10-16T11:00:41.473260+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:36.882752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"171'1","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.482519+0000","last_change":"2023-10-16T11:00:40.458570+0000","last_active":"2023-10-16T11:00:41.482519+0000","last_peered":"2023-10-16T11:00:41.482519+0000","last_clean":"2023-10-16T11:00:41.482519+0000","last_became_active":"2023-10-16T11:00:40.458318+0000","last_became_peered":"2023-10-16T11:00:40.458318+0000","last_unstale":"2023-10-16T11:00:41.482519+0000","last_undegraded":"2023-10-16T11:00:41.482519+0000","last_fullsized":"2023-10-16T11:00:41.482519+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:48.455944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463254+0000","last_change":"2023-10-16T11:00:41.459535+0000","last_active":"2023-10-16T11:00:41.463254+0000","last_peered":"2023-10-16T11:00:41.463254+0000","last_clean":"2023-10-16T11:00:41.463254+0000","last_became_active":"2023-10-16T11:00:41.459286+0000","last_became_peered":"2023-10-16T11:00:41.459286+0000","last_unstale":"2023-10-16T11:00:41.463254+0000","last_undegraded":"2023-10-16T11:00:41.463254+0000","last_fullsized":"2023-10-16T11:00:41.463254+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:21.242313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453195+0000","last_change":"2023-10-16T11:00:40.453494+0000","last_active":"2023-10-16T11:00:42.453195+0000","last_peered":"2023-10-16T11:00:42.453195+0000","last_clean":"2023-10-16T11:00:42.453195+0000","last_became_active":"2023-10-16T11:00:40.453198+0000","last_became_peered":"2023-10-16T11:00:40.453198+0000","last_unstale":"2023-10-16T11:00:42.453195+0000","last_undegraded":"2023-10-16T11:00:42.453195+0000","last_fullsized":"2023-10-16T11:00:42.453195+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:15.441715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453472+0000","last_change":"2023-10-16T11:00:41.474672+0000","last_active":"2023-10-16T11:00:42.453472+0000","last_peered":"2023-10-16T11:00:42.453472+0000","last_clean":"2023-10-16T11:00:42.453472+0000","last_became_active":"2023-10-16T11:00:41.474402+0000","last_became_peered":"2023-10-16T11:00:41.474402+0000","last_unstale":"2023-10-16T11:00:42.453472+0000","last_undegraded":"2023-10-16T11:00:42.453472+0000","last_fullsized":"2023-10-16T11:00:42.453472+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:51.120388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.451833+0000","last_change":"2023-10-16T11:00:40.450232+0000","last_active":"2023-10-16T11:00:41.451833+0000","last_peered":"2023-10-16T11:00:41.451833+0000","last_clean":"2023-10-16T11:00:41.451833+0000","last_became_active":"2023-10-16T11:00:40.449973+0000","last_became_peered":"2023-10-16T11:00:40.449973+0000","last_unstale":"2023-10-16T11:00:41.451833+0000","last_undegraded":"2023-10-16T11:00:41.451833+0000","last_fullsized":"2023-10-16T11:00:41.451833+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:36.847056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.452165+0000","last_change":"2023-10-16T11:00:41.463974+0000","last_active":"2023-10-16T11:00:42.452165+0000","last_peered":"2023-10-16T11:00:42.452165+0000","last_clean":"2023-10-16T11:00:42.452165+0000","last_became_active":"2023-10-16T11:00:41.463675+0000","last_became_peered":"2023-10-16T11:00:41.463675+0000","last_unstale":"2023-10-16T11:00:42.452165+0000","last_undegraded":"2023-10-16T11:00:42.452165+0000","last_fullsized":"2023-10-16T11:00:42.452165+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:26.220332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.450232+0000","last_change":"2023-10-16T11:00:40.460012+0000","last_active":"2023-10-16T11:00:42.450232+0000","last_peered":"2023-10-16T11:00:42.450232+0000","last_clean":"2023-10-16T11:00:42.450232+0000","last_became_active":"2023-10-16T11:00:40.459824+0000","last_became_peered":"2023-10-16T11:00:40.459824+0000","last_unstale":"2023-10-16T11:00:42.450232+0000","last_undegraded":"2023-10-16T11:00:42.450232+0000","last_fullsized":"2023-10-16T11:00:42.450232+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:51.563960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462857+0000","last_change":"2023-10-16T11:00:41.458088+0000","last_active":"2023-10-16T11:00:41.462857+0000","last_peered":"2023-10-16T11:00:41.462857+0000","last_clean":"2023-10-16T11:00:41.462857+0000","last_became_active":"2023-10-16T11:00:41.457608+0000","last_became_peered":"2023-10-16T11:00:41.457608+0000","last_unstale":"2023-10-16T11:00:41.462857+0000","last_undegraded":"2023-10-16T11:00:41.462857+0000","last_fullsized":"2023-10-16T11:00:41.462857+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:42.910704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453464+0000","last_change":"2023-10-16T11:00:40.447087+0000","last_active":"2023-10-16T11:00:42.453464+0000","last_peered":"2023-10-16T11:00:42.453464+0000","last_clean":"2023-10-16T11:00:42.453464+0000","last_became_active":"2023-10-16T11:00:40.446802+0000","last_became_peered":"2023-10-16T11:00:40.446802+0000","last_unstale":"2023-10-16T11:00:42.453464+0000","last_undegraded":"2023-10-16T11:00:42.453464+0000","last_fullsized":"2023-10-16T11:00:42.453464+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:44.511910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.478496+0000","last_change":"2023-10-16T11:00:41.477892+0000","last_active":"2023-10-16T11:00:41.478496+0000","last_peered":"2023-10-16T11:00:41.478496+0000","last_clean":"2023-10-16T11:00:41.478496+0000","last_became_active":"2023-10-16T11:00:41.477574+0000","last_became_peered":"2023-10-16T11:00:41.477574+0000","last_unstale":"2023-10-16T11:00:41.478496+0000","last_undegraded":"2023-10-16T11:00:41.478496+0000","last_fullsized":"2023-10-16T11:00:41.478496+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:13:44.623651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"171'3","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480231+0000","last_change":"2023-10-16T11:00:40.447101+0000","last_active":"2023-10-16T11:00:41.480231+0000","last_peered":"2023-10-16T11:00:41.480231+0000","last_clean":"2023-10-16T11:00:41.480231+0000","last_became_active":"2023-10-16T11:00:40.446836+0000","last_became_peered":"2023-10-16T11:00:40.446836+0000","last_unstale":"2023-10-16T11:00:41.480231+0000","last_undegraded":"2023-10-16T11:00:41.480231+0000","last_fullsized":"2023-10-16T11:00:41.480231+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:45.697533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451895+0000","last_change":"2023-10-16T11:00:41.464724+0000","last_active":"2023-10-16T11:00:42.451895+0000","last_peered":"2023-10-16T11:00:42.451895+0000","last_clean":"2023-10-16T11:00:42.451895+0000","last_became_active":"2023-10-16T11:00:41.463275+0000","last_became_peered":"2023-10-16T11:00:41.463275+0000","last_unstale":"2023-10-16T11:00:42.451895+0000","last_undegraded":"2023-10-16T11:00:42.451895+0000","last_fullsized":"2023-10-16T11:00:42.451895+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:54:54.378090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480042+0000","last_change":"2023-10-16T11:00:40.454974+0000","last_active":"2023-10-16T11:00:41.480042+0000","last_peered":"2023-10-16T11:00:41.480042+0000","last_clean":"2023-10-16T11:00:41.480042+0000","last_became_active":"2023-10-16T11:00:40.454679+0000","last_became_peered":"2023-10-16T11:00:40.454679+0000","last_unstale":"2023-10-16T11:00:41.480042+0000","last_undegraded":"2023-10-16T11:00:41.480042+0000","last_fullsized":"2023-10-16T11:00:41.480042+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:37.018071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.472784+0000","last_change":"2023-10-16T11:00:41.472240+0000","last_active":"2023-10-16T11:00:41.472784+0000","last_peered":"2023-10-16T11:00:41.472784+0000","last_clean":"2023-10-16T11:00:41.472784+0000","last_became_active":"2023-10-16T11:00:41.471959+0000","last_became_peered":"2023-10-16T11:00:41.471959+0000","last_unstale":"2023-10-16T11:00:41.472784+0000","last_undegraded":"2023-10-16T11:00:41.472784+0000","last_fullsized":"2023-10-16T11:00:41.472784+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:03.479161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464071+0000","last_change":"2023-10-16T11:00:40.458656+0000","last_active":"2023-10-16T11:00:41.464071+0000","last_peered":"2023-10-16T11:00:41.464071+0000","last_clean":"2023-10-16T11:00:41.464071+0000","last_became_active":"2023-10-16T11:00:40.458376+0000","last_became_peered":"2023-10-16T11:00:40.458376+0000","last_unstale":"2023-10-16T11:00:41.464071+0000","last_undegraded":"2023-10-16T11:00:41.464071+0000","last_fullsized":"2023-10-16T11:00:41.464071+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:08.926226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.458028+0000","last_change":"2023-10-16T11:00:41.455832+0000","last_active":"2023-10-16T11:00:41.458028+0000","last_peered":"2023-10-16T11:00:41.458028+0000","last_clean":"2023-10-16T11:00:41.458028+0000","last_became_active":"2023-10-16T11:00:41.455588+0000","last_became_peered":"2023-10-16T11:00:41.455588+0000","last_unstale":"2023-10-16T11:00:41.458028+0000","last_undegraded":"2023-10-16T11:00:41.458028+0000","last_fullsized":"2023-10-16T11:00:41.458028+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:03.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464963+0000","last_change":"2023-10-16T11:00:40.450240+0000","last_active":"2023-10-16T11:00:41.464963+0000","last_peered":"2023-10-16T11:00:41.464963+0000","last_clean":"2023-10-16T11:00:41.464963+0000","last_became_active":"2023-10-16T11:00:40.449697+0000","last_became_peered":"2023-10-16T11:00:40.449697+0000","last_unstale":"2023-10-16T11:00:41.464963+0000","last_undegraded":"2023-10-16T11:00:41.464963+0000","last_fullsized":"2023-10-16T11:00:41.464963+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:52.897742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463321+0000","last_change":"2023-10-16T11:00:41.462245+0000","last_active":"2023-10-16T11:00:41.463321+0000","last_peered":"2023-10-16T11:00:41.463321+0000","last_clean":"2023-10-16T11:00:41.463321+0000","last_became_active":"2023-10-16T11:00:41.461952+0000","last_became_peered":"2023-10-16T11:00:41.461952+0000","last_unstale":"2023-10-16T11:00:41.463321+0000","last_undegraded":"2023-10-16T11:00:41.463321+0000","last_fullsized":"2023-10-16T11:00:41.463321+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:14:59.407937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"171'8","reported_seq":20,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453044+0000","last_change":"2023-10-16T11:00:40.457630+0000","last_active":"2023-10-16T11:00:42.453044+0000","last_peered":"2023-10-16T11:00:42.453044+0000","last_clean":"2023-10-16T11:00:42.453044+0000","last_became_active":"2023-10-16T11:00:40.457356+0000","last_became_peered":"2023-10-16T11:00:40.457356+0000","last_unstale":"2023-10-16T11:00:42.453044+0000","last_undegraded":"2023-10-16T11:00:42.453044+0000","last_fullsized":"2023-10-16T11:00:42.453044+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:11:33.334005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449387+0000","last_change":"2023-10-16T11:00:41.457860+0000","last_active":"2023-10-16T11:00:42.449387+0000","last_peered":"2023-10-16T11:00:42.449387+0000","last_clean":"2023-10-16T11:00:42.449387+0000","last_became_active":"2023-10-16T11:00:41.457589+0000","last_became_peered":"2023-10-16T11:00:41.457589+0000","last_unstale":"2023-10-16T11:00:42.449387+0000","last_undegraded":"2023-10-16T11:00:42.449387+0000","last_fullsized":"2023-10-16T11:00:42.449387+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:49.460174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"171'1","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.482746+0000","last_change":"2023-10-16T11:00:40.451064+0000","last_active":"2023-10-16T11:00:41.482746+0000","last_peered":"2023-10-16T11:00:41.482746+0000","last_clean":"2023-10-16T11:00:41.482746+0000","last_became_active":"2023-10-16T11:00:40.450691+0000","last_became_peered":"2023-10-16T11:00:40.450691+0000","last_unstale":"2023-10-16T11:00:41.482746+0000","last_undegraded":"2023-10-16T11:00:41.482746+0000","last_fullsized":"2023-10-16T11:00:41.482746+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:54.767876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449565+0000","last_change":"2023-10-16T11:00:41.469796+0000","last_active":"2023-10-16T11:00:42.449565+0000","last_peered":"2023-10-16T11:00:42.449565+0000","last_clean":"2023-10-16T11:00:42.449565+0000","last_became_active":"2023-10-16T11:00:41.469499+0000","last_became_peered":"2023-10-16T11:00:41.469499+0000","last_unstale":"2023-10-16T11:00:42.449565+0000","last_undegraded":"2023-10-16T11:00:42.449565+0000","last_fullsized":"2023-10-16T11:00:42.449565+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:29.208905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"171'2","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449429+0000","last_change":"2023-10-16T11:00:40.453105+0000","last_active":"2023-10-16T11:00:42.449429+0000","last_peered":"2023-10-16T11:00:42.449429+0000","last_clean":"2023-10-16T11:00:42.449429+0000","last_became_active":"2023-10-16T11:00:40.452856+0000","last_became_peered":"2023-10-16T11:00:40.452856+0000","last_unstale":"2023-10-16T11:00:42.449429+0000","last_undegraded":"2023-10-16T11:00:42.449429+0000","last_fullsized":"2023-10-16T11:00:42.449429+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:57:27.880204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453570+0000","last_change":"2023-10-16T11:00:41.466433+0000","last_active":"2023-10-16T11:00:42.453570+0000","last_peered":"2023-10-16T11:00:42.453570+0000","last_clean":"2023-10-16T11:00:42.453570+0000","last_became_active":"2023-10-16T11:00:41.465838+0000","last_became_peered":"2023-10-16T11:00:41.465838+0000","last_unstale":"2023-10-16T11:00:42.453570+0000","last_undegraded":"2023-10-16T11:00:42.453570+0000","last_fullsized":"2023-10-16T11:00:42.453570+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:10.193252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"171'1","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.479863+0000","last_change":"2023-10-16T11:00:40.456532+0000","last_active":"2023-10-16T11:00:41.479863+0000","last_peered":"2023-10-16T11:00:41.479863+0000","last_clean":"2023-10-16T11:00:41.479863+0000","last_became_active":"2023-10-16T11:00:40.456311+0000","last_became_peered":"2023-10-16T11:00:40.456311+0000","last_unstale":"2023-10-16T11:00:41.479863+0000","last_undegraded":"2023-10-16T11:00:41.479863+0000","last_fullsized":"2023-10-16T11:00:41.479863+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:30.954112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453618+0000","last_change":"2023-10-16T11:00:41.477820+0000","last_active":"2023-10-16T11:00:42.453618+0000","last_peered":"2023-10-16T11:00:42.453618+0000","last_clean":"2023-10-16T11:00:42.453618+0000","last_became_active":"2023-10-16T11:00:41.477529+0000","last_became_peered":"2023-10-16T11:00:41.477529+0000","last_unstale":"2023-10-16T11:00:42.453618+0000","last_undegraded":"2023-10-16T11:00:42.453618+0000","last_fullsized":"2023-10-16T11:00:42.453618+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:03.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464057+0000","last_change":"2023-10-16T11:00:40.455764+0000","last_active":"2023-10-16T11:00:41.464057+0000","last_peered":"2023-10-16T11:00:41.464057+0000","last_clean":"2023-10-16T11:00:41.464057+0000","last_became_active":"2023-10-16T11:00:40.455478+0000","last_became_peered":"2023-10-16T11:00:40.455478+0000","last_unstale":"2023-10-16T11:00:41.464057+0000","last_undegraded":"2023-10-16T11:00:41.464057+0000","last_fullsized":"2023-10-16T11:00:41.464057+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:48.025931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.448994+0000","last_change":"2023-10-16T11:00:41.470677+0000","last_active":"2023-10-16T11:00:42.448994+0000","last_peered":"2023-10-16T11:00:42.448994+0000","last_clean":"2023-10-16T11:00:42.448994+0000","last_became_active":"2023-10-16T11:00:41.470326+0000","last_became_peered":"2023-10-16T11:00:41.470326+0000","last_unstale":"2023-10-16T11:00:42.448994+0000","last_undegraded":"2023-10-16T11:00:42.448994+0000","last_fullsized":"2023-10-16T11:00:42.448994+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:37.851286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.477561+0000","last_change":"2023-10-16T11:00:40.452057+0000","last_active":"2023-10-16T11:00:41.477561+0000","last_peered":"2023-10-16T11:00:41.477561+0000","last_clean":"2023-10-16T11:00:41.477561+0000","last_became_active":"2023-10-16T11:00:40.451786+0000","last_became_peered":"2023-10-16T11:00:40.451786+0000","last_unstale":"2023-10-16T11:00:41.477561+0000","last_undegraded":"2023-10-16T11:00:41.477561+0000","last_fullsized":"2023-10-16T11:00:41.477561+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:16:48.178739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.467877+0000","last_change":"2023-10-16T11:00:41.466266+0000","last_active":"2023-10-16T11:00:41.467877+0000","last_peered":"2023-10-16T11:00:41.467877+0000","last_clean":"2023-10-16T11:00:41.467877+0000","last_became_active":"2023-10-16T11:00:41.465995+0000","last_became_peered":"2023-10-16T11:00:41.465995+0000","last_unstale":"2023-10-16T11:00:41.467877+0000","last_undegraded":"2023-10-16T11:00:41.467877+0000","last_fullsized":"2023-10-16T11:00:41.467877+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:05.708282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451900+0000","last_change":"2023-10-16T11:00:40.452653+0000","last_active":"2023-10-16T11:00:42.451900+0000","last_peered":"2023-10-16T11:00:42.451900+0000","last_clean":"2023-10-16T11:00:42.451900+0000","last_became_active":"2023-10-16T11:00:40.452394+0000","last_became_peered":"2023-10-16T11:00:40.452394+0000","last_unstale":"2023-10-16T11:00:42.451900+0000","last_undegraded":"2023-10-16T11:00:42.451900+0000","last_fullsized":"2023-10-16T11:00:42.451900+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:39.763178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449517+0000","last_change":"2023-10-16T11:00:41.458899+0000","last_active":"2023-10-16T11:00:42.449517+0000","last_peered":"2023-10-16T11:00:42.449517+0000","last_clean":"2023-10-16T11:00:42.449517+0000","last_became_active":"2023-10-16T11:00:41.458598+0000","last_became_peered":"2023-10-16T11:00:41.458598+0000","last_unstale":"2023-10-16T11:00:42.449517+0000","last_undegraded":"2023-10-16T11:00:42.449517+0000","last_fullsized":"2023-10-16T11:00:42.449517+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:38.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.456356+0000","last_change":"2023-10-16T11:00:40.451703+0000","last_active":"2023-10-16T11:00:41.456356+0000","last_peered":"2023-10-16T11:00:41.456356+0000","last_clean":"2023-10-16T11:00:41.456356+0000","last_became_active":"2023-10-16T11:00:40.451430+0000","last_became_peered":"2023-10-16T11:00:40.451430+0000","last_unstale":"2023-10-16T11:00:41.456356+0000","last_undegraded":"2023-10-16T11:00:41.456356+0000","last_fullsized":"2023-10-16T11:00:41.456356+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:10.925267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468034+0000","last_change":"2023-10-16T11:00:41.466814+0000","last_active":"2023-10-16T11:00:41.468034+0000","last_peered":"2023-10-16T11:00:41.468034+0000","last_clean":"2023-10-16T11:00:41.468034+0000","last_became_active":"2023-10-16T11:00:41.466551+0000","last_became_peered":"2023-10-16T11:00:41.466551+0000","last_unstale":"2023-10-16T11:00:41.468034+0000","last_undegraded":"2023-10-16T11:00:41.468034+0000","last_fullsized":"2023-10-16T11:00:41.468034+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:53.901971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451656+0000","last_change":"2023-10-16T11:00:40.451714+0000","last_active":"2023-10-16T11:00:42.451656+0000","last_peered":"2023-10-16T11:00:42.451656+0000","last_clean":"2023-10-16T11:00:42.451656+0000","last_became_active":"2023-10-16T11:00:40.451453+0000","last_became_peered":"2023-10-16T11:00:40.451453+0000","last_unstale":"2023-10-16T11:00:42.451656+0000","last_undegraded":"2023-10-16T11:00:42.451656+0000","last_fullsized":"2023-10-16T11:00:42.451656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:33.128802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462868+0000","last_change":"2023-10-16T11:00:41.458666+0000","last_active":"2023-10-16T11:00:41.462868+0000","last_peered":"2023-10-16T11:00:41.462868+0000","last_clean":"2023-10-16T11:00:41.462868+0000","last_became_active":"2023-10-16T11:00:41.458375+0000","last_became_peered":"2023-10-16T11:00:41.458375+0000","last_unstale":"2023-10-16T11:00:41.462868+0000","last_undegraded":"2023-10-16T11:00:41.462868+0000","last_fullsized":"2023-10-16T11:00:41.462868+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:51.120388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.483618+0000","last_change":"2023-10-16T11:00:40.454868+0000","last_active":"2023-10-16T11:00:41.483618+0000","last_peered":"2023-10-16T11:00:41.483618+0000","last_clean":"2023-10-16T11:00:41.483618+0000","last_became_active":"2023-10-16T11:00:40.454544+0000","last_became_peered":"2023-10-16T11:00:40.454544+0000","last_unstale":"2023-10-16T11:00:41.483618+0000","last_undegraded":"2023-10-16T11:00:41.483618+0000","last_fullsized":"2023-10-16T11:00:41.483618+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:07.604913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471683+0000","last_change":"2023-10-16T11:00:41.466832+0000","last_active":"2023-10-16T11:00:41.471683+0000","last_peered":"2023-10-16T11:00:41.471683+0000","last_clean":"2023-10-16T11:00:41.471683+0000","last_became_active":"2023-10-16T11:00:41.466558+0000","last_became_peered":"2023-10-16T11:00:41.466558+0000","last_unstale":"2023-10-16T11:00:41.471683+0000","last_undegraded":"2023-10-16T11:00:41.471683+0000","last_fullsized":"2023-10-16T11:00:41.471683+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:47.451264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453347+0000","last_change":"2023-10-16T11:00:40.450422+0000","last_active":"2023-10-16T11:00:42.453347+0000","last_peered":"2023-10-16T11:00:42.453347+0000","last_clean":"2023-10-16T11:00:42.453347+0000","last_became_active":"2023-10-16T11:00:40.450128+0000","last_became_peered":"2023-10-16T11:00:40.450128+0000","last_unstale":"2023-10-16T11:00:42.453347+0000","last_undegraded":"2023-10-16T11:00:42.453347+0000","last_fullsized":"2023-10-16T11:00:42.453347+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:53.838374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453626+0000","last_change":"2023-10-16T11:00:41.474760+0000","last_active":"2023-10-16T11:00:42.453626+0000","last_peered":"2023-10-16T11:00:42.453626+0000","last_clean":"2023-10-16T11:00:42.453626+0000","last_became_active":"2023-10-16T11:00:41.474499+0000","last_became_peered":"2023-10-16T11:00:41.474499+0000","last_unstale":"2023-10-16T11:00:42.453626+0000","last_undegraded":"2023-10-16T11:00:42.453626+0000","last_fullsized":"2023-10-16T11:00:42.453626+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:41.526891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.453130+0000","last_change":"2023-10-16T11:00:40.444269+0000","last_active":"2023-10-16T11:00:41.453130+0000","last_peered":"2023-10-16T11:00:41.453130+0000","last_clean":"2023-10-16T11:00:41.453130+0000","last_became_active":"2023-10-16T11:00:40.444004+0000","last_became_peered":"2023-10-16T11:00:40.444004+0000","last_unstale":"2023-10-16T11:00:41.453130+0000","last_undegraded":"2023-10-16T11:00:41.453130+0000","last_fullsized":"2023-10-16T11:00:41.453130+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:22.567572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473767+0000","last_change":"2023-10-16T11:00:41.473441+0000","last_active":"2023-10-16T11:00:41.473767+0000","last_peered":"2023-10-16T11:00:41.473767+0000","last_clean":"2023-10-16T11:00:41.473767+0000","last_became_active":"2023-10-16T11:00:41.473215+0000","last_became_peered":"2023-10-16T11:00:41.473215+0000","last_unstale":"2023-10-16T11:00:41.473767+0000","last_undegraded":"2023-10-16T11:00:41.473767+0000","last_fullsized":"2023-10-16T11:00:41.473767+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:57:28.884434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.452151+0000","last_change":"2023-10-16T11:00:40.459689+0000","last_active":"2023-10-16T11:00:41.452151+0000","last_peered":"2023-10-16T11:00:41.452151+0000","last_clean":"2023-10-16T11:00:41.452151+0000","last_became_active":"2023-10-16T11:00:40.459479+0000","last_became_peered":"2023-10-16T11:00:40.459479+0000","last_unstale":"2023-10-16T11:00:41.452151+0000","last_undegraded":"2023-10-16T11:00:41.452151+0000","last_fullsized":"2023-10-16T11:00:41.452151+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:28.204675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451459+0000","last_change":"2023-10-16T11:00:41.464765+0000","last_active":"2023-10-16T11:00:42.451459+0000","last_peered":"2023-10-16T11:00:42.451459+0000","last_clean":"2023-10-16T11:00:42.451459+0000","last_became_active":"2023-10-16T11:00:41.464494+0000","last_became_peered":"2023-10-16T11:00:41.464494+0000","last_unstale":"2023-10-16T11:00:42.451459+0000","last_undegraded":"2023-10-16T11:00:42.451459+0000","last_fullsized":"2023-10-16T11:00:42.451459+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:27.689829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453308+0000","last_change":"2023-10-16T11:00:40.456501+0000","last_active":"2023-10-16T11:00:42.453308+0000","last_peered":"2023-10-16T11:00:42.453308+0000","last_clean":"2023-10-16T11:00:42.453308+0000","last_became_active":"2023-10-16T11:00:40.456223+0000","last_became_peered":"2023-10-16T11:00:40.456223+0000","last_unstale":"2023-10-16T11:00:42.453308+0000","last_undegraded":"2023-10-16T11:00:42.453308+0000","last_fullsized":"2023-10-16T11:00:42.453308+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:49:59.623653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473239+0000","last_change":"2023-10-16T11:00:41.471002+0000","last_active":"2023-10-16T11:00:41.473239+0000","last_peered":"2023-10-16T11:00:41.473239+0000","last_clean":"2023-10-16T11:00:41.473239+0000","last_became_active":"2023-10-16T11:00:41.470406+0000","last_became_peered":"2023-10-16T11:00:41.470406+0000","last_unstale":"2023-10-16T11:00:41.473239+0000","last_undegraded":"2023-10-16T11:00:41.473239+0000","last_fullsized":"2023-10-16T11:00:41.473239+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:30.075302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.450979+0000","last_change":"2023-10-16T11:00:40.450797+0000","last_active":"2023-10-16T11:00:41.450979+0000","last_peered":"2023-10-16T11:00:41.450979+0000","last_clean":"2023-10-16T11:00:41.450979+0000","last_became_active":"2023-10-16T11:00:40.450301+0000","last_became_peered":"2023-10-16T11:00:40.450301+0000","last_unstale":"2023-10-16T11:00:41.450979+0000","last_undegraded":"2023-10-16T11:00:41.450979+0000","last_fullsized":"2023-10-16T11:00:41.450979+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:56.869490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.466177+0000","last_change":"2023-10-16T11:00:41.465374+0000","last_active":"2023-10-16T11:00:41.466177+0000","last_peered":"2023-10-16T11:00:41.466177+0000","last_clean":"2023-10-16T11:00:41.466177+0000","last_became_active":"2023-10-16T11:00:41.465021+0000","last_became_peered":"2023-10-16T11:00:41.465021+0000","last_unstale":"2023-10-16T11:00:41.466177+0000","last_undegraded":"2023-10-16T11:00:41.466177+0000","last_fullsized":"2023-10-16T11:00:41.466177+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:08.609143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453320+0000","last_change":"2023-10-16T11:00:40.447040+0000","last_active":"2023-10-16T11:00:42.453320+0000","last_peered":"2023-10-16T11:00:42.453320+0000","last_clean":"2023-10-16T11:00:42.453320+0000","last_became_active":"2023-10-16T11:00:40.446750+0000","last_became_peered":"2023-10-16T11:00:40.446750+0000","last_unstale":"2023-10-16T11:00:42.453320+0000","last_undegraded":"2023-10-16T11:00:42.453320+0000","last_fullsized":"2023-10-16T11:00:42.453320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:16:48.178739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451916+0000","last_change":"2023-10-16T11:00:41.463978+0000","last_active":"2023-10-16T11:00:42.451916+0000","last_peered":"2023-10-16T11:00:42.451916+0000","last_clean":"2023-10-16T11:00:42.451916+0000","last_became_active":"2023-10-16T11:00:41.463506+0000","last_became_peered":"2023-10-16T11:00:41.463506+0000","last_unstale":"2023-10-16T11:00:42.451916+0000","last_undegraded":"2023-10-16T11:00:42.451916+0000","last_fullsized":"2023-10-16T11:00:42.451916+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:14:40.752577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449375+0000","last_change":"2023-10-16T11:00:40.453238+0000","last_active":"2023-10-16T11:00:42.449375+0000","last_peered":"2023-10-16T11:00:42.449375+0000","last_clean":"2023-10-16T11:00:42.449375+0000","last_became_active":"2023-10-16T11:00:40.452980+0000","last_became_peered":"2023-10-16T11:00:40.452980+0000","last_unstale":"2023-10-16T11:00:42.449375+0000","last_undegraded":"2023-10-16T11:00:42.449375+0000","last_fullsized":"2023-10-16T11:00:42.449375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:53.838374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.452420+0000","last_change":"2023-10-16T11:00:41.463407+0000","last_active":"2023-10-16T11:00:42.452420+0000","last_peered":"2023-10-16T11:00:42.452420+0000","last_clean":"2023-10-16T11:00:42.452420+0000","last_became_active":"2023-10-16T11:00:41.463061+0000","last_became_peered":"2023-10-16T11:00:41.463061+0000","last_unstale":"2023-10-16T11:00:42.452420+0000","last_undegraded":"2023-10-16T11:00:42.452420+0000","last_fullsized":"2023-10-16T11:00:42.452420+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:10.847238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"171'6","reported_seq":17,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.485550+0000","last_change":"2023-10-16T11:00:40.451786+0000","last_active":"2023-10-16T11:00:41.485550+0000","last_peered":"2023-10-16T11:00:41.485550+0000","last_clean":"2023-10-16T11:00:41.485550+0000","last_became_active":"2023-10-16T11:00:40.451529+0000","last_became_peered":"2023-10-16T11:00:40.451529+0000","last_unstale":"2023-10-16T11:00:41.485550+0000","last_undegraded":"2023-10-16T11:00:41.485550+0000","last_fullsized":"2023-10-16T11:00:41.485550+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:18:57.514911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"161'271","reported_seq":886,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.448356+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:00:41.448356+0000","last_peered":"2023-10-16T11:00:41.448356+0000","last_clean":"2023-10-16T11:00:41.448356+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:00:41.448356+0000","last_undegraded":"2023-10-16T11:00:41.448356+0000","last_fullsized":"2023-10-16T11:00:41.448356+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575770,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47740,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324100,"statfs":{"total":96636764160,"available":96587878400,"internally_reserved":0,"allocated":2736128,"data_stored":1723011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.38600000000000001,"15min":0.32500000000000001},"max":{"1min":0.83199999999999996,"5min":0.83399999999999996,"15min":0.88900000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.46600000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":0.86199999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.81999999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.60599999999999998,"5min":0.46300000000000002,"15min":0.313},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.85499999999999998},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.38400000000000001},"max":{"1min":0.79000000000000004,"5min":0.82199999999999995,"15min":0.86399999999999999},"last":0.44},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67300000000000004,"15min":0.66900000000000004},"min":{"1min":0.51900000000000002,"5min":0.498,"15min":0.32300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.57399999999999995}]},{"osd":3,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.35499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.50800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.90200000000000002},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.38700000000000001,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":0.67700000000000005,"5min":0.78500000000000003,"15min":0.81699999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.44900000000000001,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.84799999999999998},"last":0.40100000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53300000000000003,"15min":0.52200000000000002},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.878},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.41699999999999998,"5min":0.40999999999999998,"15min":0.36199999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.92000000000000004},"last":0.46700000000000003}]},{"osd":6,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.42999999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.96099999999999997},"last":0.504},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.60399999999999998,"15min":0.58999999999999997},"min":{"1min":0.46899999999999997,"5min":0.38700000000000001,"15min":0.36699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.47999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575770,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2142208,"data_stored":1132691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.495,"5min":0.36399999999999999,"15min":0.29799999999999999},"max":{"1min":0.79400000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.61299999999999999,"15min":0.625},"min":{"1min":0.432,"5min":0.42099999999999999,"15min":0.32400000000000001},"max":{"1min":0.70799999999999996,"5min":0.80100000000000005,"15min":0.89200000000000002},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63200000000000001,"15min":0.61799999999999999},"min":{"1min":0.45300000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83299999999999996,"5min":0.872,"15min":0.872},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.621,"15min":0.624},"min":{"1min":0.5,"5min":0.40899999999999997,"15min":0.38600000000000001},"max":{"1min":0.77200000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.625,"15min":0.61199999999999999},"min":{"1min":0.47899999999999998,"5min":0.39600000000000002,"15min":0.38100000000000001},"max":{"1min":0.80800000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.49199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76400000000000001,"5min":0.90700000000000003,"15min":0.91000000000000003},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63600000000000001,"15min":0.629},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41599999999999998},"max":{"1min":0.75600000000000001,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.48999999999999999,"5min":0.39200000000000002,"15min":0.36399999999999999},"max":{"1min":0.80200000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.505,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.32100000000000001,"15min":0.29999999999999999},"max":{"1min":0.72499999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.75700000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60099999999999998}]},{"osd":5,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.53300000000000003},"min":{"1min":0.40100000000000002,"5min":0.35599999999999998,"15min":0.35399999999999998},"max":{"1min":0.69199999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.39000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.629}]},{"osd":7,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78800000000000003,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.89900000000000002,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575771,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":2142208,"data_stored":1132691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.432},"min":{"1min":0.245,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.64100000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46000000000000002,"15min":0.47199999999999998},"min":{"1min":0.35899999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.73599999999999999},"last":0.436}]},{"osd":2,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.54100000000000004,"5min":0.59499999999999997,"15min":0.66000000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45800000000000002,"15min":0.46500000000000002},"min":{"1min":0.36499999999999999,"5min":0.29099999999999998,"15min":0.27800000000000002},"max":{"1min":0.60599999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48099999999999998,"15min":0.49099999999999999},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32200000000000001},"max":{"1min":0.628,"5min":0.73099999999999998,"15min":0.748},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45800000000000002},"min":{"1min":0.33700000000000002,"5min":0.313,"15min":0.28999999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.78300000000000003},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.51700000000000002,"5min":0.48399999999999999,"15min":0.46000000000000002},"max":{"1min":0.83799999999999997,"5min":0.89000000000000001,"15min":0.93899999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88300000000000001,"5min":0.88500000000000001,"15min":0.91300000000000003},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46300000000000002,"15min":0.44},"max":{"1min":0.78600000000000003,"5min":0.92100000000000004,"15min":0.96399999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72399999999999998,"15min":0.72299999999999998},"min":{"1min":0.54300000000000004,"5min":0.48099999999999998,"15min":0.47199999999999998},"max":{"1min":0.92200000000000004,"5min":0.93400000000000005,"15min":0.97999999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.51000000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.94499999999999995,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.52300000000000002,"5min":0.48699999999999999,"15min":0.44},"max":{"1min":0.84199999999999997,"5min":0.93799999999999994,"15min":0.99399999999999999},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.54200000000000004,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.85799999999999998,"5min":0.96899999999999997,"15min":0.98799999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70099999999999996,"15min":0.70699999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.98999999999999999},"last":0.55900000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38032,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333808,"statfs":{"total":96636764160,"available":96597819392,"internally_reserved":0,"allocated":2756608,"data_stored":1725307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.39400000000000002},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55000000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.67400000000000004,"15min":0.67600000000000005},"last":0.313}]},{"osd":2,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.434},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.94899999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49399999999999999,"15min":0.49299999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.044},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56999999999999995,"5min":0.64500000000000002,"15min":0.67000000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":0.67800000000000005,"15min":0.68100000000000005},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.36699999999999999},"max":{"1min":0.77100000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39400000000000002},"max":{"1min":0.79300000000000004,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.435,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81299999999999994,"5min":0.96999999999999997,"15min":0.98199999999999998},"last":0.59699999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.69199999999999995,"15min":0.69499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.98699999999999999,"15min":1.0209999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.69799999999999995,"15min":0.70199999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.96299999999999997},"last":0.621}]},{"osd":7,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.997,"15min":0.997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68100000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85099999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.65000000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575770,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63812,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308028,"statfs":{"total":96636764160,"available":96571420672,"internally_reserved":0,"allocated":2146304,"data_stored":1132733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40600000000000003,"15min":0.39600000000000002},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.97199999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.60899999999999999,"5min":0.67500000000000004,"15min":1.3180000000000001},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.30599999999999999,"5min":0.27800000000000002,"15min":0.27100000000000002},"max":{"1min":0.625,"5min":0.749,"15min":1.161},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.29099999999999998,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45100000000000001,"15min":0.45000000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.23799999999999999},"max":{"1min":0.58299999999999996,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70099999999999996,"5min":0.73699999999999999,"15min":1.431},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.66700000000000004,"15min":0.66200000000000003},"min":{"1min":0.44900000000000001,"5min":0.41599999999999998,"15min":0.40699999999999997},"max":{"1min":0.86699999999999999,"5min":0.98899999999999999,"15min":1.0129999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69299999999999995,"15min":0.68500000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.435},"max":{"1min":0.93999999999999995,"5min":0.96299999999999997,"15min":1.234},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":0.97399999999999998,"15min":1.1299999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96799999999999997,"5min":0.98799999999999999,"15min":1.278},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68799999999999994,"15min":0.67100000000000004},"min":{"1min":0.46999999999999997,"5min":0.432,"15min":0.432},"max":{"1min":0.84799999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.70899999999999996},"min":{"1min":0.54700000000000004,"5min":0.51100000000000001,"15min":0.432},"max":{"1min":0.89500000000000002,"5min":1.004,"15min":1.1299999999999999},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.48199999999999998},"max":{"1min":0.99099999999999999,"5min":1.02,"15min":1.3879999999999999},"last":0.496},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71099999999999997,"15min":0.70299999999999996},"min":{"1min":0.502,"5min":0.47699999999999998,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":0.94499999999999995,"15min":1.2190000000000001},"last":0.54400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575770,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47168,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324672,"statfs":{"total":96636764160,"available":96588464128,"internally_reserved":0,"allocated":2146304,"data_stored":1133237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.438},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.41699999999999998,"15min":0.41999999999999998},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.47699999999999998,"5min":0.61099999999999999,"15min":0.68300000000000005},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56699999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":0.54500000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57499999999999996,"5min":0.69199999999999995,"15min":0.75600000000000001},"last":0.499},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.58699999999999997,"5min":0.70899999999999996,"15min":0.748},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81299999999999994,"5min":0.97199999999999998,"15min":0.97699999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67600000000000005,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":0.98899999999999999,"15min":1.0629999999999999},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.72399999999999998,"5min":0.99299999999999999,"15min":1.0629999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.68899999999999995},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81399999999999995,"5min":1.054,"15min":1.054},"last":0.81399999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.68899999999999995},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.82199999999999995,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.755},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.45800000000000002},"max":{"1min":0.79700000000000004,"5min":0.99399999999999999,"15min":1.02},"last":0.79700000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.53000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.84299999999999997,"5min":1.012,"15min":1.012},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.70699999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79400000000000004,"5min":0.97899999999999998,"15min":1.002},"last":0.77400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575770,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2162688,"data_stored":1141305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.81200000000000006,"15min":0.83799999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.629},"min":{"1min":0.42399999999999999,"5min":0.41099999999999998,"15min":0.315},"max":{"1min":0.77000000000000002,"5min":0.80500000000000005,"15min":0.84899999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61899999999999999},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78500000000000003,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.504},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.66600000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72599999999999998,"5min":0.85599999999999998,"15min":0.879},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.71299999999999997,"5min":0.83699999999999997,"15min":0.875},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.74099999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63700000000000001,"15min":0.63800000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77100000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61199999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.501},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73099999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.69399999999999995,"5min":0.83199999999999996,"15min":0.84499999999999997},"last":0.39100000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.375},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56599999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.41699999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38900000000000001},"max":{"1min":0.68899999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33400000000000002},"max":{"1min":0.70899999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59640,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312200,"statfs":{"total":96636764160,"available":96575692800,"internally_reserved":0,"allocated":2138112,"data_stored":1132621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64800000000000002,"15min":0.65600000000000003},"min":{"1min":0.504,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.77100000000000002,"5min":0.85899999999999999,"15min":0.86499999999999999},"last":0.503},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.46899999999999997,"15min":0.41599999999999998},"max":{"1min":0.78700000000000003,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59399999999999997,"15min":0.59799999999999998},"min":{"1min":0.36799999999999999,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":0.79600000000000004,"15min":0.82299999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70799999999999996,"5min":0.82099999999999995,"15min":0.83299999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":0.88200000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.623,"15min":0.623},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.82799999999999996},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.47999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71199999999999997,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.37,"5min":0.33500000000000002,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.78600000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.318,"15min":0.28000000000000003},"max":{"1min":0.69999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.63700000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.40000000000000002,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":0.66800000000000004,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56100000000000005,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68300000000000005,"5min":0.82999999999999996,"15min":0.84099999999999997},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.57699999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72699999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63600000000000001,"15min":0.63200000000000001},"min":{"1min":0.46000000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":0.77300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.71799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:00:45.588 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:00:45.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.742+0000 7f13d326c700 1 Processor -- start 2023-10-16T11:00:45.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.742+0000 7f13d326c700 1 -- start start 2023-10-16T11:00:45.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d326c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f13cc06e5d0 0x7f13cc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d326c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13cc0729b0 0x7f13cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d326c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13cc071e50 con 0x7f13cc06ab40 2023-10-16T11:00:45.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d326c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13cc161270 con 0x7f13cc06e5d0 2023-10-16T11:00:45.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d326c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13cc161510 con 0x7f13cc0729b0 2023-10-16T11:00:45.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f13cc06e5d0 0x7f13cc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f13cc06e5d0 0x7f13cc06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33230/0 (socket says 172.21.15.158:33230) 2023-10-16T11:00:45.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 -- 172.21.15.158:0/4021895566 learned_addr learned my addr 172.21.15.158:0/4021895566 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:45.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d1809700 1 --2- 172.21.15.158:0/4021895566 >> v2:172.21.15.172:3300/0 conn(0x7f13cc0729b0 0x7f13cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 -- 172.21.15.158:0/4021895566 >> v2:172.21.15.172:3300/0 conn(0x7f13cc0729b0 msgr2=0x7f13cc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:45.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 --2- 172.21.15.158:0/4021895566 >> v2:172.21.15.172:3300/0 conn(0x7f13cc0729b0 0x7f13cc071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 -- 172.21.15.158:0/4021895566 >> v1:172.21.15.172:6789/0 conn(0x7f13cc06ab40 legacy=0x7f13cc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:45.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 -- 172.21.15.158:0/4021895566 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc1617f0 con 0x7f13cc06e5d0 2023-10-16T11:00:45.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d1809700 1 --2- 172.21.15.158:0/4021895566 >> v2:172.21.15.172:3300/0 conn(0x7f13cc0729b0 0x7f13cc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:45.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13d0807700 1 --2- 172.21.15.158:0/4021895566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc06e5d0 0x7f13cc06d820 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f13cc158240 tx=0x7f13c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d886b52985b4169 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13c3fff700 1 -- 172.21.15.158:0/4021895566 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8004230 con 0x7f13cc06e5d0 2023-10-16T11:00:45.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13c3fff700 1 -- 172.21.15.158:0/4021895566 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c8004390 con 0x7f13cc06e5d0 2023-10-16T11:00:45.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.743+0000 7f13c3fff700 1 -- 172.21.15.158:0/4021895566 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c80036a0 con 0x7f13cc06e5d0 2023-10-16T11:00:45.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 -- 172.21.15.158:0/4021895566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc06e5d0 msgr2=0x7f13cc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 --2- 172.21.15.158:0/4021895566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc06e5d0 0x7f13cc06d820 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f13cc158240 tx=0x7f13c8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 -- 172.21.15.158:0/4021895566 shutdown_connections 2023-10-16T11:00:45.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 --2- 172.21.15.158:0/4021895566 >> v2:172.21.15.172:3300/0 conn(0x7f13cc0729b0 0x7f13cc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 --2- 172.21.15.158:0/4021895566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc06e5d0 0x7f13cc06d820 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 -- 172.21.15.158:0/4021895566 >> 172.21.15.158:0/4021895566 conn(0x7f13cc0d02b0 msgr2=0x7f13cc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:45.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 -- 172.21.15.158:0/4021895566 shutdown_connections 2023-10-16T11:00:45.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.744+0000 7f13d326c700 1 -- 172.21.15.158:0/4021895566 wait complete. 2023-10-16T11:00:45.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.745+0000 7f13d326c700 1 Processor -- start 2023-10-16T11:00:45.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.745+0000 7f13d326c700 1 -- start start 2023-10-16T11:00:45.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d326c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13cc06ab40 0x7f13cc151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d326c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13cc06e5d0 0x7f13cc151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d326c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc0729b0 0x7f13cc1f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d326c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc15ff70 con 0x7f13cc0729b0 2023-10-16T11:00:45.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d326c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc0c3ce0 con 0x7f13cc06ab40 2023-10-16T11:00:45.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d326c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc160460 con 0x7f13cc06e5d0 2023-10-16T11:00:45.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc0729b0 0x7f13cc1f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc0729b0 0x7f13cc1f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33238/0 (socket says 172.21.15.158:33238) 2023-10-16T11:00:45.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 -- 172.21.15.158:0/4112305910 learned_addr learned my addr 172.21.15.158:0/4112305910 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:45.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d0807700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13cc06e5d0 0x7f13cc151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1008700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13cc06ab40 0x7f13cc151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 -- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13cc06e5d0 msgr2=0x7f13cc151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13cc06e5d0 0x7f13cc151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 -- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13cc06ab40 msgr2=0x7f13cc151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13cc06ab40 0x7f13cc151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 -- 172.21.15.158:0/4112305910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c8009550 con 0x7f13cc0729b0 2023-10-16T11:00:45.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.746+0000 7f13d1809700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc0729b0 0x7f13cc1f66b0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f13c400ceb0 tx=0x7f13c400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.747+0000 7f13c1ffb700 1 -- 172.21.15.158:0/4112305910 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c400ad00 con 0x7f13cc0729b0 2023-10-16T11:00:45.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.747+0000 7f13c1ffb700 1 -- 172.21.15.158:0/4112305910 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c400ae60 con 0x7f13cc0729b0 2023-10-16T11:00:45.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.747+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13cc0c3f50 con 0x7f13cc0729b0 2023-10-16T11:00:45.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.747+0000 7f13c1ffb700 1 -- 172.21.15.158:0/4112305910 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4015680 con 0x7f13cc0729b0 2023-10-16T11:00:45.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.747+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13cc160740 con 0x7f13cc0729b0 2023-10-16T11:00:45.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.748+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13cc0c4190 con 0x7f13cc0729b0 2023-10-16T11:00:45.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.748+0000 7f13c1ffb700 1 -- 172.21.15.158:0/4112305910 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f13c400d430 con 0x7f13cc0729b0 2023-10-16T11:00:45.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.748+0000 7f13c1ffb700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f13b006b020 0x7f13b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:45.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.749+0000 7f13d1008700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f13b006b020 0x7f13b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:45.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.749+0000 7f13c1ffb700 1 -- 172.21.15.158:0/4112305910 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f13c4089e30 con 0x7f13cc0729b0 2023-10-16T11:00:45.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.749+0000 7f13d1008700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f13b006b020 0x7f13b006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009960 tx=0x7f13bc0090e0 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:45.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.753+0000 7f13c1ffb700 1 -- 172.21.15.158:0/4112305910 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13c4054b50 con 0x7f13cc0729b0 2023-10-16T11:00:45.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.942+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 --> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13cc161ad0 con 0x7f13b006b020 2023-10-16T11:00:45.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.950+0000 7f13c1ffb700 1 -- 172.21.15.158:0/4112305910 <== mgr.9690 v2:172.21.15.158:6832/1293733893 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290643 (secure 0 0 0) 0x7f13bc009e30 con 0x7f13b006b020 2023-10-16T11:00:45.954 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:45.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f13b006b020 msgr2=0x7f13b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f13b006b020 0x7f13b006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009960 tx=0x7f13bc0090e0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc0729b0 msgr2=0x7f13cc1f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:45.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc0729b0 0x7f13cc1f66b0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f13c400ceb0 tx=0x7f13c400cee0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 shutdown_connections 2023-10-16T11:00:45.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13cc0729b0 0x7f13cc1f66b0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f13b006b020 0x7f13b006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13cc06ab40 0x7f13cc151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 --2- 172.21.15.158:0/4112305910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13cc06e5d0 0x7f13cc151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:45.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 >> 172.21.15.158:0/4112305910 conn(0x7f13cc0d02b0 msgr2=0x7f13cc06d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:45.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.953+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 shutdown_connections 2023-10-16T11:00:45.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:45.954+0000 7f13d326c700 1 -- 172.21.15.158:0/4112305910 wait complete. 2023-10-16T11:00:45.959 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:00:45.967 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T11:00:44.903932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":372868,"kb_used_data":17940,"kb_used_omap":12,"kb_used_meta":354803,"kb_avail":754601852,"statfs":{"total":773094113280,"available":772712296448,"internally_reserved":0,"allocated":18370560,"data_stored":10253596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":363318839},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.470052"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453490+0000","last_change":"2023-10-16T11:00:41.477008+0000","last_active":"2023-10-16T11:00:42.453490+0000","last_peered":"2023-10-16T11:00:42.453490+0000","last_clean":"2023-10-16T11:00:42.453490+0000","last_became_active":"2023-10-16T11:00:41.476709+0000","last_became_peered":"2023-10-16T11:00:41.476709+0000","last_unstale":"2023-10-16T11:00:42.453490+0000","last_undegraded":"2023-10-16T11:00:42.453490+0000","last_fullsized":"2023-10-16T11:00:42.453490+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:53.069037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.472343+0000","last_change":"2023-10-16T11:00:41.471591+0000","last_active":"2023-10-16T11:00:41.472343+0000","last_peered":"2023-10-16T11:00:41.472343+0000","last_clean":"2023-10-16T11:00:41.472343+0000","last_became_active":"2023-10-16T11:00:41.471350+0000","last_became_peered":"2023-10-16T11:00:41.471350+0000","last_unstale":"2023-10-16T11:00:41.472343+0000","last_undegraded":"2023-10-16T11:00:41.472343+0000","last_fullsized":"2023-10-16T11:00:41.472343+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:17.745318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463282+0000","last_change":"2023-10-16T11:00:41.460551+0000","last_active":"2023-10-16T11:00:41.463282+0000","last_peered":"2023-10-16T11:00:41.463282+0000","last_clean":"2023-10-16T11:00:41.463282+0000","last_became_active":"2023-10-16T11:00:41.460282+0000","last_became_peered":"2023-10-16T11:00:41.460282+0000","last_unstale":"2023-10-16T11:00:41.463282+0000","last_undegraded":"2023-10-16T11:00:41.463282+0000","last_fullsized":"2023-10-16T11:00:41.463282+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:11:36.916150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471633+0000","last_change":"2023-10-16T11:00:41.465831+0000","last_active":"2023-10-16T11:00:41.471633+0000","last_peered":"2023-10-16T11:00:41.471633+0000","last_clean":"2023-10-16T11:00:41.471633+0000","last_became_active":"2023-10-16T11:00:41.465575+0000","last_became_peered":"2023-10-16T11:00:41.465575+0000","last_unstale":"2023-10-16T11:00:41.471633+0000","last_undegraded":"2023-10-16T11:00:41.471633+0000","last_fullsized":"2023-10-16T11:00:41.471633+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:49.958609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462787+0000","last_change":"2023-10-16T11:00:41.458769+0000","last_active":"2023-10-16T11:00:41.462787+0000","last_peered":"2023-10-16T11:00:41.462787+0000","last_clean":"2023-10-16T11:00:41.462787+0000","last_became_active":"2023-10-16T11:00:41.458521+0000","last_became_peered":"2023-10-16T11:00:41.458521+0000","last_unstale":"2023-10-16T11:00:41.462787+0000","last_undegraded":"2023-10-16T11:00:41.462787+0000","last_fullsized":"2023-10-16T11:00:41.462787+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:10.376601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.466788+0000","last_change":"2023-10-16T11:00:41.466213+0000","last_active":"2023-10-16T11:00:41.466788+0000","last_peered":"2023-10-16T11:00:41.466788+0000","last_clean":"2023-10-16T11:00:41.466788+0000","last_became_active":"2023-10-16T11:00:41.465883+0000","last_became_peered":"2023-10-16T11:00:41.465883+0000","last_unstale":"2023-10-16T11:00:41.466788+0000","last_undegraded":"2023-10-16T11:00:41.466788+0000","last_fullsized":"2023-10-16T11:00:41.466788+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:10.193252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453294+0000","last_change":"2023-10-16T11:00:41.461538+0000","last_active":"2023-10-16T11:00:42.453294+0000","last_peered":"2023-10-16T11:00:42.453294+0000","last_clean":"2023-10-16T11:00:42.453294+0000","last_became_active":"2023-10-16T11:00:41.458949+0000","last_became_peered":"2023-10-16T11:00:41.458949+0000","last_unstale":"2023-10-16T11:00:42.453294+0000","last_undegraded":"2023-10-16T11:00:42.453294+0000","last_fullsized":"2023-10-16T11:00:42.453294+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:10.376601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451783+0000","last_change":"2023-10-16T11:00:41.459242+0000","last_active":"2023-10-16T11:00:42.451783+0000","last_peered":"2023-10-16T11:00:42.451783+0000","last_clean":"2023-10-16T11:00:42.451783+0000","last_became_active":"2023-10-16T11:00:41.458953+0000","last_became_peered":"2023-10-16T11:00:41.458953+0000","last_unstale":"2023-10-16T11:00:42.451783+0000","last_undegraded":"2023-10-16T11:00:42.451783+0000","last_fullsized":"2023-10-16T11:00:42.451783+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:24:55.187152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463998+0000","last_change":"2023-10-16T11:00:41.462286+0000","last_active":"2023-10-16T11:00:41.463998+0000","last_peered":"2023-10-16T11:00:41.463998+0000","last_clean":"2023-10-16T11:00:41.463998+0000","last_became_active":"2023-10-16T11:00:41.462015+0000","last_became_peered":"2023-10-16T11:00:41.462015+0000","last_unstale":"2023-10-16T11:00:41.463998+0000","last_undegraded":"2023-10-16T11:00:41.463998+0000","last_fullsized":"2023-10-16T11:00:41.463998+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:52.568190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464030+0000","last_change":"2023-10-16T11:00:41.459684+0000","last_active":"2023-10-16T11:00:41.464030+0000","last_peered":"2023-10-16T11:00:41.464030+0000","last_clean":"2023-10-16T11:00:41.464030+0000","last_became_active":"2023-10-16T11:00:41.459422+0000","last_became_peered":"2023-10-16T11:00:41.459422+0000","last_unstale":"2023-10-16T11:00:41.464030+0000","last_undegraded":"2023-10-16T11:00:41.464030+0000","last_fullsized":"2023-10-16T11:00:41.464030+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:53.069037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462834+0000","last_change":"2023-10-16T11:00:41.457993+0000","last_active":"2023-10-16T11:00:41.462834+0000","last_peered":"2023-10-16T11:00:41.462834+0000","last_clean":"2023-10-16T11:00:41.462834+0000","last_became_active":"2023-10-16T11:00:41.457232+0000","last_became_peered":"2023-10-16T11:00:41.457232+0000","last_unstale":"2023-10-16T11:00:41.462834+0000","last_undegraded":"2023-10-16T11:00:41.462834+0000","last_fullsized":"2023-10-16T11:00:41.462834+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:34.345528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471657+0000","last_change":"2023-10-16T11:00:41.466799+0000","last_active":"2023-10-16T11:00:41.471657+0000","last_peered":"2023-10-16T11:00:41.471657+0000","last_clean":"2023-10-16T11:00:41.471657+0000","last_became_active":"2023-10-16T11:00:41.466512+0000","last_became_peered":"2023-10-16T11:00:41.466512+0000","last_unstale":"2023-10-16T11:00:41.471657+0000","last_undegraded":"2023-10-16T11:00:41.471657+0000","last_fullsized":"2023-10-16T11:00:41.471657+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:53.923623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463155+0000","last_change":"2023-10-16T11:00:41.459488+0000","last_active":"2023-10-16T11:00:41.463155+0000","last_peered":"2023-10-16T11:00:41.463155+0000","last_clean":"2023-10-16T11:00:41.463155+0000","last_became_active":"2023-10-16T11:00:41.459214+0000","last_became_peered":"2023-10-16T11:00:41.459214+0000","last_unstale":"2023-10-16T11:00:41.463155+0000","last_undegraded":"2023-10-16T11:00:41.463155+0000","last_fullsized":"2023-10-16T11:00:41.463155+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:41.526891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473394+0000","last_change":"2023-10-16T11:00:41.470475+0000","last_active":"2023-10-16T11:00:41.473394+0000","last_peered":"2023-10-16T11:00:41.473394+0000","last_clean":"2023-10-16T11:00:41.473394+0000","last_became_active":"2023-10-16T11:00:41.469828+0000","last_became_peered":"2023-10-16T11:00:41.469828+0000","last_unstale":"2023-10-16T11:00:41.473394+0000","last_undegraded":"2023-10-16T11:00:41.473394+0000","last_fullsized":"2023-10-16T11:00:41.473394+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:48.349715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.472146+0000","last_change":"2023-10-16T11:00:41.471040+0000","last_active":"2023-10-16T11:00:41.472146+0000","last_peered":"2023-10-16T11:00:41.472146+0000","last_clean":"2023-10-16T11:00:41.472146+0000","last_became_active":"2023-10-16T11:00:41.470747+0000","last_became_peered":"2023-10-16T11:00:41.470747+0000","last_unstale":"2023-10-16T11:00:41.472146+0000","last_undegraded":"2023-10-16T11:00:41.472146+0000","last_fullsized":"2023-10-16T11:00:41.472146+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:57.876948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468210+0000","last_change":"2023-10-16T11:00:41.467035+0000","last_active":"2023-10-16T11:00:41.468210+0000","last_peered":"2023-10-16T11:00:41.468210+0000","last_clean":"2023-10-16T11:00:41.468210+0000","last_became_active":"2023-10-16T11:00:41.466784+0000","last_became_peered":"2023-10-16T11:00:41.466784+0000","last_unstale":"2023-10-16T11:00:41.468210+0000","last_undegraded":"2023-10-16T11:00:41.468210+0000","last_fullsized":"2023-10-16T11:00:41.468210+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:54.842603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.478263+0000","last_change":"2023-10-16T11:00:41.472267+0000","last_active":"2023-10-16T11:00:41.478263+0000","last_peered":"2023-10-16T11:00:41.478263+0000","last_clean":"2023-10-16T11:00:41.478263+0000","last_became_active":"2023-10-16T11:00:41.471848+0000","last_became_peered":"2023-10-16T11:00:41.471848+0000","last_unstale":"2023-10-16T11:00:41.478263+0000","last_undegraded":"2023-10-16T11:00:41.478263+0000","last_fullsized":"2023-10-16T11:00:41.478263+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:54.257655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.479047+0000","last_change":"2023-10-16T11:00:41.478064+0000","last_active":"2023-10-16T11:00:41.479047+0000","last_peered":"2023-10-16T11:00:41.479047+0000","last_clean":"2023-10-16T11:00:41.479047+0000","last_became_active":"2023-10-16T11:00:41.477789+0000","last_became_peered":"2023-10-16T11:00:41.477789+0000","last_unstale":"2023-10-16T11:00:41.479047+0000","last_undegraded":"2023-10-16T11:00:41.479047+0000","last_fullsized":"2023-10-16T11:00:41.479047+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:37.346803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449699+0000","last_change":"2023-10-16T11:00:41.469762+0000","last_active":"2023-10-16T11:00:42.449699+0000","last_peered":"2023-10-16T11:00:42.449699+0000","last_clean":"2023-10-16T11:00:42.449699+0000","last_became_active":"2023-10-16T11:00:41.469455+0000","last_became_peered":"2023-10-16T11:00:41.469455+0000","last_unstale":"2023-10-16T11:00:42.449699+0000","last_undegraded":"2023-10-16T11:00:42.449699+0000","last_fullsized":"2023-10-16T11:00:42.449699+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:53.876230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.465046+0000","last_change":"2023-10-16T11:00:41.461267+0000","last_active":"2023-10-16T11:00:41.465046+0000","last_peered":"2023-10-16T11:00:41.465046+0000","last_clean":"2023-10-16T11:00:41.465046+0000","last_became_active":"2023-10-16T11:00:41.460984+0000","last_became_peered":"2023-10-16T11:00:41.460984+0000","last_unstale":"2023-10-16T11:00:41.465046+0000","last_undegraded":"2023-10-16T11:00:41.465046+0000","last_fullsized":"2023-10-16T11:00:41.465046+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:16.445945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464024+0000","last_change":"2023-10-16T11:00:41.461959+0000","last_active":"2023-10-16T11:00:41.464024+0000","last_peered":"2023-10-16T11:00:41.464024+0000","last_clean":"2023-10-16T11:00:41.464024+0000","last_became_active":"2023-10-16T11:00:41.460926+0000","last_became_peered":"2023-10-16T11:00:41.460926+0000","last_unstale":"2023-10-16T11:00:41.464024+0000","last_undegraded":"2023-10-16T11:00:41.464024+0000","last_fullsized":"2023-10-16T11:00:41.464024+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:10.075072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453383+0000","last_change":"2023-10-16T11:00:41.464610+0000","last_active":"2023-10-16T11:00:42.453383+0000","last_peered":"2023-10-16T11:00:42.453383+0000","last_clean":"2023-10-16T11:00:42.453383+0000","last_became_active":"2023-10-16T11:00:41.464287+0000","last_became_peered":"2023-10-16T11:00:41.464287+0000","last_unstale":"2023-10-16T11:00:42.453383+0000","last_undegraded":"2023-10-16T11:00:42.453383+0000","last_fullsized":"2023-10-16T11:00:42.453383+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:57:28.884434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451572+0000","last_change":"2023-10-16T11:00:41.466639+0000","last_active":"2023-10-16T11:00:42.451572+0000","last_peered":"2023-10-16T11:00:42.451572+0000","last_clean":"2023-10-16T11:00:42.451572+0000","last_became_active":"2023-10-16T11:00:41.466383+0000","last_became_peered":"2023-10-16T11:00:41.466383+0000","last_unstale":"2023-10-16T11:00:42.451572+0000","last_undegraded":"2023-10-16T11:00:42.451572+0000","last_fullsized":"2023-10-16T11:00:42.451572+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:34.831144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.456643+0000","last_change":"2023-10-16T11:00:41.455674+0000","last_active":"2023-10-16T11:00:41.456643+0000","last_peered":"2023-10-16T11:00:41.456643+0000","last_clean":"2023-10-16T11:00:41.456643+0000","last_became_active":"2023-10-16T11:00:41.455364+0000","last_became_peered":"2023-10-16T11:00:41.455364+0000","last_unstale":"2023-10-16T11:00:41.456643+0000","last_undegraded":"2023-10-16T11:00:41.456643+0000","last_fullsized":"2023-10-16T11:00:41.456643+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:11:28.754077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453551+0000","last_change":"2023-10-16T11:00:41.481083+0000","last_active":"2023-10-16T11:00:42.453551+0000","last_peered":"2023-10-16T11:00:42.453551+0000","last_clean":"2023-10-16T11:00:42.453551+0000","last_became_active":"2023-10-16T11:00:41.480816+0000","last_became_peered":"2023-10-16T11:00:41.480816+0000","last_unstale":"2023-10-16T11:00:42.453551+0000","last_undegraded":"2023-10-16T11:00:42.453551+0000","last_fullsized":"2023-10-16T11:00:42.453551+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:52.568190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453717+0000","last_change":"2023-10-16T11:00:41.466489+0000","last_active":"2023-10-16T11:00:42.453717+0000","last_peered":"2023-10-16T11:00:42.453717+0000","last_clean":"2023-10-16T11:00:42.453717+0000","last_became_active":"2023-10-16T11:00:41.466202+0000","last_became_peered":"2023-10-16T11:00:41.466202+0000","last_unstale":"2023-10-16T11:00:42.453717+0000","last_undegraded":"2023-10-16T11:00:42.453717+0000","last_fullsized":"2023-10-16T11:00:42.453717+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:10.075072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.460060+0000","last_change":"2023-10-16T11:00:41.456615+0000","last_active":"2023-10-16T11:00:41.460060+0000","last_peered":"2023-10-16T11:00:41.460060+0000","last_clean":"2023-10-16T11:00:41.460060+0000","last_became_active":"2023-10-16T11:00:41.456353+0000","last_became_peered":"2023-10-16T11:00:41.456353+0000","last_unstale":"2023-10-16T11:00:41.460060+0000","last_undegraded":"2023-10-16T11:00:41.460060+0000","last_fullsized":"2023-10-16T11:00:41.460060+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.516140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462949+0000","last_change":"2023-10-16T11:00:41.460594+0000","last_active":"2023-10-16T11:00:41.462949+0000","last_peered":"2023-10-16T11:00:41.462949+0000","last_clean":"2023-10-16T11:00:41.462949+0000","last_became_active":"2023-10-16T11:00:41.460223+0000","last_became_peered":"2023-10-16T11:00:41.460223+0000","last_unstale":"2023-10-16T11:00:41.462949+0000","last_undegraded":"2023-10-16T11:00:41.462949+0000","last_fullsized":"2023-10-16T11:00:41.462949+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:44.082575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451841+0000","last_change":"2023-10-16T11:00:41.463809+0000","last_active":"2023-10-16T11:00:42.451841+0000","last_peered":"2023-10-16T11:00:42.451841+0000","last_clean":"2023-10-16T11:00:42.451841+0000","last_became_active":"2023-10-16T11:00:41.463182+0000","last_became_peered":"2023-10-16T11:00:41.463182+0000","last_unstale":"2023-10-16T11:00:42.451841+0000","last_undegraded":"2023-10-16T11:00:42.451841+0000","last_fullsized":"2023-10-16T11:00:42.451841+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:14.514749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468812+0000","last_change":"2023-10-16T11:00:41.468511+0000","last_active":"2023-10-16T11:00:41.468812+0000","last_peered":"2023-10-16T11:00:41.468812+0000","last_clean":"2023-10-16T11:00:41.468812+0000","last_became_active":"2023-10-16T11:00:41.468241+0000","last_became_peered":"2023-10-16T11:00:41.468241+0000","last_unstale":"2023-10-16T11:00:41.468812+0000","last_undegraded":"2023-10-16T11:00:41.468812+0000","last_fullsized":"2023-10-16T11:00:41.468812+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:02.241701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468304+0000","last_change":"2023-10-16T11:00:41.467197+0000","last_active":"2023-10-16T11:00:41.468304+0000","last_peered":"2023-10-16T11:00:41.468304+0000","last_clean":"2023-10-16T11:00:41.468304+0000","last_became_active":"2023-10-16T11:00:41.466709+0000","last_became_peered":"2023-10-16T11:00:41.466709+0000","last_unstale":"2023-10-16T11:00:41.468304+0000","last_undegraded":"2023-10-16T11:00:41.468304+0000","last_fullsized":"2023-10-16T11:00:41.468304+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:57.181305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451596+0000","last_change":"2023-10-16T11:00:41.462986+0000","last_active":"2023-10-16T11:00:42.451596+0000","last_peered":"2023-10-16T11:00:42.451596+0000","last_clean":"2023-10-16T11:00:42.451596+0000","last_became_active":"2023-10-16T11:00:41.462665+0000","last_became_peered":"2023-10-16T11:00:41.462665+0000","last_unstale":"2023-10-16T11:00:42.451596+0000","last_undegraded":"2023-10-16T11:00:42.451596+0000","last_fullsized":"2023-10-16T11:00:42.451596+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:17.941549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463413+0000","last_change":"2023-10-16T11:00:41.460338+0000","last_active":"2023-10-16T11:00:41.463413+0000","last_peered":"2023-10-16T11:00:41.463413+0000","last_clean":"2023-10-16T11:00:41.463413+0000","last_became_active":"2023-10-16T11:00:41.460070+0000","last_became_peered":"2023-10-16T11:00:41.460070+0000","last_unstale":"2023-10-16T11:00:41.463413+0000","last_undegraded":"2023-10-16T11:00:41.463413+0000","last_fullsized":"2023-10-16T11:00:41.463413+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:08.498765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"171'4","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480397+0000","last_change":"2023-10-16T11:00:40.456582+0000","last_active":"2023-10-16T11:00:41.480397+0000","last_peered":"2023-10-16T11:00:41.480397+0000","last_clean":"2023-10-16T11:00:41.480397+0000","last_became_active":"2023-10-16T11:00:40.456323+0000","last_became_peered":"2023-10-16T11:00:40.456323+0000","last_unstale":"2023-10-16T11:00:41.480397+0000","last_undegraded":"2023-10-16T11:00:41.480397+0000","last_fullsized":"2023-10-16T11:00:41.480397+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:52.872001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471874+0000","last_change":"2023-10-16T11:00:41.470993+0000","last_active":"2023-10-16T11:00:41.471874+0000","last_peered":"2023-10-16T11:00:41.471874+0000","last_clean":"2023-10-16T11:00:41.471874+0000","last_became_active":"2023-10-16T11:00:41.470691+0000","last_became_peered":"2023-10-16T11:00:41.470691+0000","last_unstale":"2023-10-16T11:00:41.471874+0000","last_undegraded":"2023-10-16T11:00:41.471874+0000","last_fullsized":"2023-10-16T11:00:41.471874+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:50.860884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"171'1","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449094+0000","last_change":"2023-10-16T11:00:40.452203+0000","last_active":"2023-10-16T11:00:42.449094+0000","last_peered":"2023-10-16T11:00:42.449094+0000","last_clean":"2023-10-16T11:00:42.449094+0000","last_became_active":"2023-10-16T11:00:40.451986+0000","last_became_peered":"2023-10-16T11:00:40.451986+0000","last_unstale":"2023-10-16T11:00:42.449094+0000","last_undegraded":"2023-10-16T11:00:42.449094+0000","last_fullsized":"2023-10-16T11:00:42.449094+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:09.372371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.452763+0000","last_change":"2023-10-16T11:00:41.471082+0000","last_active":"2023-10-16T11:00:42.452763+0000","last_peered":"2023-10-16T11:00:42.452763+0000","last_clean":"2023-10-16T11:00:42.452763+0000","last_became_active":"2023-10-16T11:00:41.470664+0000","last_became_peered":"2023-10-16T11:00:41.470664+0000","last_unstale":"2023-10-16T11:00:42.452763+0000","last_undegraded":"2023-10-16T11:00:42.452763+0000","last_fullsized":"2023-10-16T11:00:42.452763+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:10.863889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"171'2","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449793+0000","last_change":"2023-10-16T11:00:40.457440+0000","last_active":"2023-10-16T11:00:42.449793+0000","last_peered":"2023-10-16T11:00:42.449793+0000","last_clean":"2023-10-16T11:00:42.449793+0000","last_became_active":"2023-10-16T11:00:40.457213+0000","last_became_peered":"2023-10-16T11:00:40.457213+0000","last_unstale":"2023-10-16T11:00:42.449793+0000","last_undegraded":"2023-10-16T11:00:42.449793+0000","last_fullsized":"2023-10-16T11:00:42.449793+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:09.070842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.450039+0000","last_change":"2023-10-16T11:00:41.466643+0000","last_active":"2023-10-16T11:00:42.450039+0000","last_peered":"2023-10-16T11:00:42.450039+0000","last_clean":"2023-10-16T11:00:42.450039+0000","last_became_active":"2023-10-16T11:00:41.465951+0000","last_became_peered":"2023-10-16T11:00:41.465951+0000","last_unstale":"2023-10-16T11:00:42.450039+0000","last_undegraded":"2023-10-16T11:00:42.450039+0000","last_fullsized":"2023-10-16T11:00:42.450039+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:48.788715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480311+0000","last_change":"2023-10-16T11:00:40.451643+0000","last_active":"2023-10-16T11:00:41.480311+0000","last_peered":"2023-10-16T11:00:41.480311+0000","last_clean":"2023-10-16T11:00:41.480311+0000","last_became_active":"2023-10-16T11:00:40.451380+0000","last_became_peered":"2023-10-16T11:00:40.451380+0000","last_unstale":"2023-10-16T11:00:41.480311+0000","last_undegraded":"2023-10-16T11:00:41.480311+0000","last_fullsized":"2023-10-16T11:00:41.480311+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:22.248351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.465075+0000","last_change":"2023-10-16T11:00:41.463587+0000","last_active":"2023-10-16T11:00:41.465075+0000","last_peered":"2023-10-16T11:00:41.465075+0000","last_clean":"2023-10-16T11:00:41.465075+0000","last_became_active":"2023-10-16T11:00:41.463407+0000","last_became_peered":"2023-10-16T11:00:41.463407+0000","last_unstale":"2023-10-16T11:00:41.465075+0000","last_undegraded":"2023-10-16T11:00:41.465075+0000","last_fullsized":"2023-10-16T11:00:41.465075+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:47.907213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"171'1","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451952+0000","last_change":"2023-10-16T11:00:40.458502+0000","last_active":"2023-10-16T11:00:42.451952+0000","last_peered":"2023-10-16T11:00:42.451952+0000","last_clean":"2023-10-16T11:00:42.451952+0000","last_became_active":"2023-10-16T11:00:40.458211+0000","last_became_peered":"2023-10-16T11:00:40.458211+0000","last_unstale":"2023-10-16T11:00:42.451952+0000","last_undegraded":"2023-10-16T11:00:42.451952+0000","last_fullsized":"2023-10-16T11:00:42.451952+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:34.299145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473260+0000","last_change":"2023-10-16T11:00:41.471193+0000","last_active":"2023-10-16T11:00:41.473260+0000","last_peered":"2023-10-16T11:00:41.473260+0000","last_clean":"2023-10-16T11:00:41.473260+0000","last_became_active":"2023-10-16T11:00:41.470752+0000","last_became_peered":"2023-10-16T11:00:41.470752+0000","last_unstale":"2023-10-16T11:00:41.473260+0000","last_undegraded":"2023-10-16T11:00:41.473260+0000","last_fullsized":"2023-10-16T11:00:41.473260+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:36.882752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"171'1","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.482519+0000","last_change":"2023-10-16T11:00:40.458570+0000","last_active":"2023-10-16T11:00:41.482519+0000","last_peered":"2023-10-16T11:00:41.482519+0000","last_clean":"2023-10-16T11:00:41.482519+0000","last_became_active":"2023-10-16T11:00:40.458318+0000","last_became_peered":"2023-10-16T11:00:40.458318+0000","last_unstale":"2023-10-16T11:00:41.482519+0000","last_undegraded":"2023-10-16T11:00:41.482519+0000","last_fullsized":"2023-10-16T11:00:41.482519+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:48.455944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463254+0000","last_change":"2023-10-16T11:00:41.459535+0000","last_active":"2023-10-16T11:00:41.463254+0000","last_peered":"2023-10-16T11:00:41.463254+0000","last_clean":"2023-10-16T11:00:41.463254+0000","last_became_active":"2023-10-16T11:00:41.459286+0000","last_became_peered":"2023-10-16T11:00:41.459286+0000","last_unstale":"2023-10-16T11:00:41.463254+0000","last_undegraded":"2023-10-16T11:00:41.463254+0000","last_fullsized":"2023-10-16T11:00:41.463254+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:21.242313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453195+0000","last_change":"2023-10-16T11:00:40.453494+0000","last_active":"2023-10-16T11:00:42.453195+0000","last_peered":"2023-10-16T11:00:42.453195+0000","last_clean":"2023-10-16T11:00:42.453195+0000","last_became_active":"2023-10-16T11:00:40.453198+0000","last_became_peered":"2023-10-16T11:00:40.453198+0000","last_unstale":"2023-10-16T11:00:42.453195+0000","last_undegraded":"2023-10-16T11:00:42.453195+0000","last_fullsized":"2023-10-16T11:00:42.453195+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:15.441715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453472+0000","last_change":"2023-10-16T11:00:41.474672+0000","last_active":"2023-10-16T11:00:42.453472+0000","last_peered":"2023-10-16T11:00:42.453472+0000","last_clean":"2023-10-16T11:00:42.453472+0000","last_became_active":"2023-10-16T11:00:41.474402+0000","last_became_peered":"2023-10-16T11:00:41.474402+0000","last_unstale":"2023-10-16T11:00:42.453472+0000","last_undegraded":"2023-10-16T11:00:42.453472+0000","last_fullsized":"2023-10-16T11:00:42.453472+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:51.120388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.451833+0000","last_change":"2023-10-16T11:00:40.450232+0000","last_active":"2023-10-16T11:00:41.451833+0000","last_peered":"2023-10-16T11:00:41.451833+0000","last_clean":"2023-10-16T11:00:41.451833+0000","last_became_active":"2023-10-16T11:00:40.449973+0000","last_became_peered":"2023-10-16T11:00:40.449973+0000","last_unstale":"2023-10-16T11:00:41.451833+0000","last_undegraded":"2023-10-16T11:00:41.451833+0000","last_fullsized":"2023-10-16T11:00:41.451833+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:36.847056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.452165+0000","last_change":"2023-10-16T11:00:41.463974+0000","last_active":"2023-10-16T11:00:42.452165+0000","last_peered":"2023-10-16T11:00:42.452165+0000","last_clean":"2023-10-16T11:00:42.452165+0000","last_became_active":"2023-10-16T11:00:41.463675+0000","last_became_peered":"2023-10-16T11:00:41.463675+0000","last_unstale":"2023-10-16T11:00:42.452165+0000","last_undegraded":"2023-10-16T11:00:42.452165+0000","last_fullsized":"2023-10-16T11:00:42.452165+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:26.220332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.450232+0000","last_change":"2023-10-16T11:00:40.460012+0000","last_active":"2023-10-16T11:00:42.450232+0000","last_peered":"2023-10-16T11:00:42.450232+0000","last_clean":"2023-10-16T11:00:42.450232+0000","last_became_active":"2023-10-16T11:00:40.459824+0000","last_became_peered":"2023-10-16T11:00:40.459824+0000","last_unstale":"2023-10-16T11:00:42.450232+0000","last_undegraded":"2023-10-16T11:00:42.450232+0000","last_fullsized":"2023-10-16T11:00:42.450232+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:51.563960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462857+0000","last_change":"2023-10-16T11:00:41.458088+0000","last_active":"2023-10-16T11:00:41.462857+0000","last_peered":"2023-10-16T11:00:41.462857+0000","last_clean":"2023-10-16T11:00:41.462857+0000","last_became_active":"2023-10-16T11:00:41.457608+0000","last_became_peered":"2023-10-16T11:00:41.457608+0000","last_unstale":"2023-10-16T11:00:41.462857+0000","last_undegraded":"2023-10-16T11:00:41.462857+0000","last_fullsized":"2023-10-16T11:00:41.462857+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:42.910704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453464+0000","last_change":"2023-10-16T11:00:40.447087+0000","last_active":"2023-10-16T11:00:42.453464+0000","last_peered":"2023-10-16T11:00:42.453464+0000","last_clean":"2023-10-16T11:00:42.453464+0000","last_became_active":"2023-10-16T11:00:40.446802+0000","last_became_peered":"2023-10-16T11:00:40.446802+0000","last_unstale":"2023-10-16T11:00:42.453464+0000","last_undegraded":"2023-10-16T11:00:42.453464+0000","last_fullsized":"2023-10-16T11:00:42.453464+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:44.511910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.478496+0000","last_change":"2023-10-16T11:00:41.477892+0000","last_active":"2023-10-16T11:00:41.478496+0000","last_peered":"2023-10-16T11:00:41.478496+0000","last_clean":"2023-10-16T11:00:41.478496+0000","last_became_active":"2023-10-16T11:00:41.477574+0000","last_became_peered":"2023-10-16T11:00:41.477574+0000","last_unstale":"2023-10-16T11:00:41.478496+0000","last_undegraded":"2023-10-16T11:00:41.478496+0000","last_fullsized":"2023-10-16T11:00:41.478496+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:13:44.623651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"171'3","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480231+0000","last_change":"2023-10-16T11:00:40.447101+0000","last_active":"2023-10-16T11:00:41.480231+0000","last_peered":"2023-10-16T11:00:41.480231+0000","last_clean":"2023-10-16T11:00:41.480231+0000","last_became_active":"2023-10-16T11:00:40.446836+0000","last_became_peered":"2023-10-16T11:00:40.446836+0000","last_unstale":"2023-10-16T11:00:41.480231+0000","last_undegraded":"2023-10-16T11:00:41.480231+0000","last_fullsized":"2023-10-16T11:00:41.480231+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:45.697533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451895+0000","last_change":"2023-10-16T11:00:41.464724+0000","last_active":"2023-10-16T11:00:42.451895+0000","last_peered":"2023-10-16T11:00:42.451895+0000","last_clean":"2023-10-16T11:00:42.451895+0000","last_became_active":"2023-10-16T11:00:41.463275+0000","last_became_peered":"2023-10-16T11:00:41.463275+0000","last_unstale":"2023-10-16T11:00:42.451895+0000","last_undegraded":"2023-10-16T11:00:42.451895+0000","last_fullsized":"2023-10-16T11:00:42.451895+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:54:54.378090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.480042+0000","last_change":"2023-10-16T11:00:40.454974+0000","last_active":"2023-10-16T11:00:41.480042+0000","last_peered":"2023-10-16T11:00:41.480042+0000","last_clean":"2023-10-16T11:00:41.480042+0000","last_became_active":"2023-10-16T11:00:40.454679+0000","last_became_peered":"2023-10-16T11:00:40.454679+0000","last_unstale":"2023-10-16T11:00:41.480042+0000","last_undegraded":"2023-10-16T11:00:41.480042+0000","last_fullsized":"2023-10-16T11:00:41.480042+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:37.018071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.472784+0000","last_change":"2023-10-16T11:00:41.472240+0000","last_active":"2023-10-16T11:00:41.472784+0000","last_peered":"2023-10-16T11:00:41.472784+0000","last_clean":"2023-10-16T11:00:41.472784+0000","last_became_active":"2023-10-16T11:00:41.471959+0000","last_became_peered":"2023-10-16T11:00:41.471959+0000","last_unstale":"2023-10-16T11:00:41.472784+0000","last_undegraded":"2023-10-16T11:00:41.472784+0000","last_fullsized":"2023-10-16T11:00:41.472784+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:03.479161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464071+0000","last_change":"2023-10-16T11:00:40.458656+0000","last_active":"2023-10-16T11:00:41.464071+0000","last_peered":"2023-10-16T11:00:41.464071+0000","last_clean":"2023-10-16T11:00:41.464071+0000","last_became_active":"2023-10-16T11:00:40.458376+0000","last_became_peered":"2023-10-16T11:00:40.458376+0000","last_unstale":"2023-10-16T11:00:41.464071+0000","last_undegraded":"2023-10-16T11:00:41.464071+0000","last_fullsized":"2023-10-16T11:00:41.464071+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:08.926226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.458028+0000","last_change":"2023-10-16T11:00:41.455832+0000","last_active":"2023-10-16T11:00:41.458028+0000","last_peered":"2023-10-16T11:00:41.458028+0000","last_clean":"2023-10-16T11:00:41.458028+0000","last_became_active":"2023-10-16T11:00:41.455588+0000","last_became_peered":"2023-10-16T11:00:41.455588+0000","last_unstale":"2023-10-16T11:00:41.458028+0000","last_undegraded":"2023-10-16T11:00:41.458028+0000","last_fullsized":"2023-10-16T11:00:41.458028+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:03.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464963+0000","last_change":"2023-10-16T11:00:40.450240+0000","last_active":"2023-10-16T11:00:41.464963+0000","last_peered":"2023-10-16T11:00:41.464963+0000","last_clean":"2023-10-16T11:00:41.464963+0000","last_became_active":"2023-10-16T11:00:40.449697+0000","last_became_peered":"2023-10-16T11:00:40.449697+0000","last_unstale":"2023-10-16T11:00:41.464963+0000","last_undegraded":"2023-10-16T11:00:41.464963+0000","last_fullsized":"2023-10-16T11:00:41.464963+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:52.897742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.463321+0000","last_change":"2023-10-16T11:00:41.462245+0000","last_active":"2023-10-16T11:00:41.463321+0000","last_peered":"2023-10-16T11:00:41.463321+0000","last_clean":"2023-10-16T11:00:41.463321+0000","last_became_active":"2023-10-16T11:00:41.461952+0000","last_became_peered":"2023-10-16T11:00:41.461952+0000","last_unstale":"2023-10-16T11:00:41.463321+0000","last_undegraded":"2023-10-16T11:00:41.463321+0000","last_fullsized":"2023-10-16T11:00:41.463321+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:14:59.407937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"171'8","reported_seq":20,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453044+0000","last_change":"2023-10-16T11:00:40.457630+0000","last_active":"2023-10-16T11:00:42.453044+0000","last_peered":"2023-10-16T11:00:42.453044+0000","last_clean":"2023-10-16T11:00:42.453044+0000","last_became_active":"2023-10-16T11:00:40.457356+0000","last_became_peered":"2023-10-16T11:00:40.457356+0000","last_unstale":"2023-10-16T11:00:42.453044+0000","last_undegraded":"2023-10-16T11:00:42.453044+0000","last_fullsized":"2023-10-16T11:00:42.453044+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:11:33.334005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449387+0000","last_change":"2023-10-16T11:00:41.457860+0000","last_active":"2023-10-16T11:00:42.449387+0000","last_peered":"2023-10-16T11:00:42.449387+0000","last_clean":"2023-10-16T11:00:42.449387+0000","last_became_active":"2023-10-16T11:00:41.457589+0000","last_became_peered":"2023-10-16T11:00:41.457589+0000","last_unstale":"2023-10-16T11:00:42.449387+0000","last_undegraded":"2023-10-16T11:00:42.449387+0000","last_fullsized":"2023-10-16T11:00:42.449387+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:49.460174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"171'1","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.482746+0000","last_change":"2023-10-16T11:00:40.451064+0000","last_active":"2023-10-16T11:00:41.482746+0000","last_peered":"2023-10-16T11:00:41.482746+0000","last_clean":"2023-10-16T11:00:41.482746+0000","last_became_active":"2023-10-16T11:00:40.450691+0000","last_became_peered":"2023-10-16T11:00:40.450691+0000","last_unstale":"2023-10-16T11:00:41.482746+0000","last_undegraded":"2023-10-16T11:00:41.482746+0000","last_fullsized":"2023-10-16T11:00:41.482746+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:54.767876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449565+0000","last_change":"2023-10-16T11:00:41.469796+0000","last_active":"2023-10-16T11:00:42.449565+0000","last_peered":"2023-10-16T11:00:42.449565+0000","last_clean":"2023-10-16T11:00:42.449565+0000","last_became_active":"2023-10-16T11:00:41.469499+0000","last_became_peered":"2023-10-16T11:00:41.469499+0000","last_unstale":"2023-10-16T11:00:42.449565+0000","last_undegraded":"2023-10-16T11:00:42.449565+0000","last_fullsized":"2023-10-16T11:00:42.449565+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:29.208905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"171'2","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449429+0000","last_change":"2023-10-16T11:00:40.453105+0000","last_active":"2023-10-16T11:00:42.449429+0000","last_peered":"2023-10-16T11:00:42.449429+0000","last_clean":"2023-10-16T11:00:42.449429+0000","last_became_active":"2023-10-16T11:00:40.452856+0000","last_became_peered":"2023-10-16T11:00:40.452856+0000","last_unstale":"2023-10-16T11:00:42.449429+0000","last_undegraded":"2023-10-16T11:00:42.449429+0000","last_fullsized":"2023-10-16T11:00:42.449429+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:57:27.880204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453570+0000","last_change":"2023-10-16T11:00:41.466433+0000","last_active":"2023-10-16T11:00:42.453570+0000","last_peered":"2023-10-16T11:00:42.453570+0000","last_clean":"2023-10-16T11:00:42.453570+0000","last_became_active":"2023-10-16T11:00:41.465838+0000","last_became_peered":"2023-10-16T11:00:41.465838+0000","last_unstale":"2023-10-16T11:00:42.453570+0000","last_undegraded":"2023-10-16T11:00:42.453570+0000","last_fullsized":"2023-10-16T11:00:42.453570+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:10.193252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"171'1","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.479863+0000","last_change":"2023-10-16T11:00:40.456532+0000","last_active":"2023-10-16T11:00:41.479863+0000","last_peered":"2023-10-16T11:00:41.479863+0000","last_clean":"2023-10-16T11:00:41.479863+0000","last_became_active":"2023-10-16T11:00:40.456311+0000","last_became_peered":"2023-10-16T11:00:40.456311+0000","last_unstale":"2023-10-16T11:00:41.479863+0000","last_undegraded":"2023-10-16T11:00:41.479863+0000","last_fullsized":"2023-10-16T11:00:41.479863+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:30.954112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453618+0000","last_change":"2023-10-16T11:00:41.477820+0000","last_active":"2023-10-16T11:00:42.453618+0000","last_peered":"2023-10-16T11:00:42.453618+0000","last_clean":"2023-10-16T11:00:42.453618+0000","last_became_active":"2023-10-16T11:00:41.477529+0000","last_became_peered":"2023-10-16T11:00:41.477529+0000","last_unstale":"2023-10-16T11:00:42.453618+0000","last_undegraded":"2023-10-16T11:00:42.453618+0000","last_fullsized":"2023-10-16T11:00:42.453618+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:03.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.464057+0000","last_change":"2023-10-16T11:00:40.455764+0000","last_active":"2023-10-16T11:00:41.464057+0000","last_peered":"2023-10-16T11:00:41.464057+0000","last_clean":"2023-10-16T11:00:41.464057+0000","last_became_active":"2023-10-16T11:00:40.455478+0000","last_became_peered":"2023-10-16T11:00:40.455478+0000","last_unstale":"2023-10-16T11:00:41.464057+0000","last_undegraded":"2023-10-16T11:00:41.464057+0000","last_fullsized":"2023-10-16T11:00:41.464057+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:48.025931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.448994+0000","last_change":"2023-10-16T11:00:41.470677+0000","last_active":"2023-10-16T11:00:42.448994+0000","last_peered":"2023-10-16T11:00:42.448994+0000","last_clean":"2023-10-16T11:00:42.448994+0000","last_became_active":"2023-10-16T11:00:41.470326+0000","last_became_peered":"2023-10-16T11:00:41.470326+0000","last_unstale":"2023-10-16T11:00:42.448994+0000","last_undegraded":"2023-10-16T11:00:42.448994+0000","last_fullsized":"2023-10-16T11:00:42.448994+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:37.851286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.477561+0000","last_change":"2023-10-16T11:00:40.452057+0000","last_active":"2023-10-16T11:00:41.477561+0000","last_peered":"2023-10-16T11:00:41.477561+0000","last_clean":"2023-10-16T11:00:41.477561+0000","last_became_active":"2023-10-16T11:00:40.451786+0000","last_became_peered":"2023-10-16T11:00:40.451786+0000","last_unstale":"2023-10-16T11:00:41.477561+0000","last_undegraded":"2023-10-16T11:00:41.477561+0000","last_fullsized":"2023-10-16T11:00:41.477561+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:16:48.178739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.467877+0000","last_change":"2023-10-16T11:00:41.466266+0000","last_active":"2023-10-16T11:00:41.467877+0000","last_peered":"2023-10-16T11:00:41.467877+0000","last_clean":"2023-10-16T11:00:41.467877+0000","last_became_active":"2023-10-16T11:00:41.465995+0000","last_became_peered":"2023-10-16T11:00:41.465995+0000","last_unstale":"2023-10-16T11:00:41.467877+0000","last_undegraded":"2023-10-16T11:00:41.467877+0000","last_fullsized":"2023-10-16T11:00:41.467877+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:05.708282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451900+0000","last_change":"2023-10-16T11:00:40.452653+0000","last_active":"2023-10-16T11:00:42.451900+0000","last_peered":"2023-10-16T11:00:42.451900+0000","last_clean":"2023-10-16T11:00:42.451900+0000","last_became_active":"2023-10-16T11:00:40.452394+0000","last_became_peered":"2023-10-16T11:00:40.452394+0000","last_unstale":"2023-10-16T11:00:42.451900+0000","last_undegraded":"2023-10-16T11:00:42.451900+0000","last_fullsized":"2023-10-16T11:00:42.451900+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:39.763178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449517+0000","last_change":"2023-10-16T11:00:41.458899+0000","last_active":"2023-10-16T11:00:42.449517+0000","last_peered":"2023-10-16T11:00:42.449517+0000","last_clean":"2023-10-16T11:00:42.449517+0000","last_became_active":"2023-10-16T11:00:41.458598+0000","last_became_peered":"2023-10-16T11:00:41.458598+0000","last_unstale":"2023-10-16T11:00:42.449517+0000","last_undegraded":"2023-10-16T11:00:42.449517+0000","last_fullsized":"2023-10-16T11:00:42.449517+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:38.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.456356+0000","last_change":"2023-10-16T11:00:40.451703+0000","last_active":"2023-10-16T11:00:41.456356+0000","last_peered":"2023-10-16T11:00:41.456356+0000","last_clean":"2023-10-16T11:00:41.456356+0000","last_became_active":"2023-10-16T11:00:40.451430+0000","last_became_peered":"2023-10-16T11:00:40.451430+0000","last_unstale":"2023-10-16T11:00:41.456356+0000","last_undegraded":"2023-10-16T11:00:41.456356+0000","last_fullsized":"2023-10-16T11:00:41.456356+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:10.925267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.468034+0000","last_change":"2023-10-16T11:00:41.466814+0000","last_active":"2023-10-16T11:00:41.468034+0000","last_peered":"2023-10-16T11:00:41.468034+0000","last_clean":"2023-10-16T11:00:41.468034+0000","last_became_active":"2023-10-16T11:00:41.466551+0000","last_became_peered":"2023-10-16T11:00:41.466551+0000","last_unstale":"2023-10-16T11:00:41.468034+0000","last_undegraded":"2023-10-16T11:00:41.468034+0000","last_fullsized":"2023-10-16T11:00:41.468034+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:53.901971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451656+0000","last_change":"2023-10-16T11:00:40.451714+0000","last_active":"2023-10-16T11:00:42.451656+0000","last_peered":"2023-10-16T11:00:42.451656+0000","last_clean":"2023-10-16T11:00:42.451656+0000","last_became_active":"2023-10-16T11:00:40.451453+0000","last_became_peered":"2023-10-16T11:00:40.451453+0000","last_unstale":"2023-10-16T11:00:42.451656+0000","last_undegraded":"2023-10-16T11:00:42.451656+0000","last_fullsized":"2023-10-16T11:00:42.451656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:33.128802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.462868+0000","last_change":"2023-10-16T11:00:41.458666+0000","last_active":"2023-10-16T11:00:41.462868+0000","last_peered":"2023-10-16T11:00:41.462868+0000","last_clean":"2023-10-16T11:00:41.462868+0000","last_became_active":"2023-10-16T11:00:41.458375+0000","last_became_peered":"2023-10-16T11:00:41.458375+0000","last_unstale":"2023-10-16T11:00:41.462868+0000","last_undegraded":"2023-10-16T11:00:41.462868+0000","last_fullsized":"2023-10-16T11:00:41.462868+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:51.120388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"171'2","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.483618+0000","last_change":"2023-10-16T11:00:40.454868+0000","last_active":"2023-10-16T11:00:41.483618+0000","last_peered":"2023-10-16T11:00:41.483618+0000","last_clean":"2023-10-16T11:00:41.483618+0000","last_became_active":"2023-10-16T11:00:40.454544+0000","last_became_peered":"2023-10-16T11:00:40.454544+0000","last_unstale":"2023-10-16T11:00:41.483618+0000","last_undegraded":"2023-10-16T11:00:41.483618+0000","last_fullsized":"2023-10-16T11:00:41.483618+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:07.604913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.471683+0000","last_change":"2023-10-16T11:00:41.466832+0000","last_active":"2023-10-16T11:00:41.471683+0000","last_peered":"2023-10-16T11:00:41.471683+0000","last_clean":"2023-10-16T11:00:41.471683+0000","last_became_active":"2023-10-16T11:00:41.466558+0000","last_became_peered":"2023-10-16T11:00:41.466558+0000","last_unstale":"2023-10-16T11:00:41.471683+0000","last_undegraded":"2023-10-16T11:00:41.471683+0000","last_fullsized":"2023-10-16T11:00:41.471683+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:47.451264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453347+0000","last_change":"2023-10-16T11:00:40.450422+0000","last_active":"2023-10-16T11:00:42.453347+0000","last_peered":"2023-10-16T11:00:42.453347+0000","last_clean":"2023-10-16T11:00:42.453347+0000","last_became_active":"2023-10-16T11:00:40.450128+0000","last_became_peered":"2023-10-16T11:00:40.450128+0000","last_unstale":"2023-10-16T11:00:42.453347+0000","last_undegraded":"2023-10-16T11:00:42.453347+0000","last_fullsized":"2023-10-16T11:00:42.453347+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:53.838374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453626+0000","last_change":"2023-10-16T11:00:41.474760+0000","last_active":"2023-10-16T11:00:42.453626+0000","last_peered":"2023-10-16T11:00:42.453626+0000","last_clean":"2023-10-16T11:00:42.453626+0000","last_became_active":"2023-10-16T11:00:41.474499+0000","last_became_peered":"2023-10-16T11:00:41.474499+0000","last_unstale":"2023-10-16T11:00:42.453626+0000","last_undegraded":"2023-10-16T11:00:42.453626+0000","last_fullsized":"2023-10-16T11:00:42.453626+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:41.526891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.453130+0000","last_change":"2023-10-16T11:00:40.444269+0000","last_active":"2023-10-16T11:00:41.453130+0000","last_peered":"2023-10-16T11:00:41.453130+0000","last_clean":"2023-10-16T11:00:41.453130+0000","last_became_active":"2023-10-16T11:00:40.444004+0000","last_became_peered":"2023-10-16T11:00:40.444004+0000","last_unstale":"2023-10-16T11:00:41.453130+0000","last_undegraded":"2023-10-16T11:00:41.453130+0000","last_fullsized":"2023-10-16T11:00:41.453130+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:22.567572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473767+0000","last_change":"2023-10-16T11:00:41.473441+0000","last_active":"2023-10-16T11:00:41.473767+0000","last_peered":"2023-10-16T11:00:41.473767+0000","last_clean":"2023-10-16T11:00:41.473767+0000","last_became_active":"2023-10-16T11:00:41.473215+0000","last_became_peered":"2023-10-16T11:00:41.473215+0000","last_unstale":"2023-10-16T11:00:41.473767+0000","last_undegraded":"2023-10-16T11:00:41.473767+0000","last_fullsized":"2023-10-16T11:00:41.473767+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:57:28.884434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.452151+0000","last_change":"2023-10-16T11:00:40.459689+0000","last_active":"2023-10-16T11:00:41.452151+0000","last_peered":"2023-10-16T11:00:41.452151+0000","last_clean":"2023-10-16T11:00:41.452151+0000","last_became_active":"2023-10-16T11:00:40.459479+0000","last_became_peered":"2023-10-16T11:00:40.459479+0000","last_unstale":"2023-10-16T11:00:41.452151+0000","last_undegraded":"2023-10-16T11:00:41.452151+0000","last_fullsized":"2023-10-16T11:00:41.452151+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:28.204675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451459+0000","last_change":"2023-10-16T11:00:41.464765+0000","last_active":"2023-10-16T11:00:42.451459+0000","last_peered":"2023-10-16T11:00:42.451459+0000","last_clean":"2023-10-16T11:00:42.451459+0000","last_became_active":"2023-10-16T11:00:41.464494+0000","last_became_peered":"2023-10-16T11:00:41.464494+0000","last_unstale":"2023-10-16T11:00:42.451459+0000","last_undegraded":"2023-10-16T11:00:42.451459+0000","last_fullsized":"2023-10-16T11:00:42.451459+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:27.689829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453308+0000","last_change":"2023-10-16T11:00:40.456501+0000","last_active":"2023-10-16T11:00:42.453308+0000","last_peered":"2023-10-16T11:00:42.453308+0000","last_clean":"2023-10-16T11:00:42.453308+0000","last_became_active":"2023-10-16T11:00:40.456223+0000","last_became_peered":"2023-10-16T11:00:40.456223+0000","last_unstale":"2023-10-16T11:00:42.453308+0000","last_undegraded":"2023-10-16T11:00:42.453308+0000","last_fullsized":"2023-10-16T11:00:42.453308+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:49:59.623653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.473239+0000","last_change":"2023-10-16T11:00:41.471002+0000","last_active":"2023-10-16T11:00:41.473239+0000","last_peered":"2023-10-16T11:00:41.473239+0000","last_clean":"2023-10-16T11:00:41.473239+0000","last_became_active":"2023-10-16T11:00:41.470406+0000","last_became_peered":"2023-10-16T11:00:41.470406+0000","last_unstale":"2023-10-16T11:00:41.473239+0000","last_undegraded":"2023-10-16T11:00:41.473239+0000","last_fullsized":"2023-10-16T11:00:41.473239+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:30.075302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.450979+0000","last_change":"2023-10-16T11:00:40.450797+0000","last_active":"2023-10-16T11:00:41.450979+0000","last_peered":"2023-10-16T11:00:41.450979+0000","last_clean":"2023-10-16T11:00:41.450979+0000","last_became_active":"2023-10-16T11:00:40.450301+0000","last_became_peered":"2023-10-16T11:00:40.450301+0000","last_unstale":"2023-10-16T11:00:41.450979+0000","last_undegraded":"2023-10-16T11:00:41.450979+0000","last_fullsized":"2023-10-16T11:00:41.450979+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:56.869490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.466177+0000","last_change":"2023-10-16T11:00:41.465374+0000","last_active":"2023-10-16T11:00:41.466177+0000","last_peered":"2023-10-16T11:00:41.466177+0000","last_clean":"2023-10-16T11:00:41.466177+0000","last_became_active":"2023-10-16T11:00:41.465021+0000","last_became_peered":"2023-10-16T11:00:41.465021+0000","last_unstale":"2023-10-16T11:00:41.466177+0000","last_undegraded":"2023-10-16T11:00:41.466177+0000","last_fullsized":"2023-10-16T11:00:41.466177+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:24:08.609143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.453320+0000","last_change":"2023-10-16T11:00:40.447040+0000","last_active":"2023-10-16T11:00:42.453320+0000","last_peered":"2023-10-16T11:00:42.453320+0000","last_clean":"2023-10-16T11:00:42.453320+0000","last_became_active":"2023-10-16T11:00:40.446750+0000","last_became_peered":"2023-10-16T11:00:40.446750+0000","last_unstale":"2023-10-16T11:00:42.453320+0000","last_undegraded":"2023-10-16T11:00:42.453320+0000","last_fullsized":"2023-10-16T11:00:42.453320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:16:48.178739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.451916+0000","last_change":"2023-10-16T11:00:41.463978+0000","last_active":"2023-10-16T11:00:42.451916+0000","last_peered":"2023-10-16T11:00:42.451916+0000","last_clean":"2023-10-16T11:00:42.451916+0000","last_became_active":"2023-10-16T11:00:41.463506+0000","last_became_peered":"2023-10-16T11:00:41.463506+0000","last_unstale":"2023-10-16T11:00:42.451916+0000","last_undegraded":"2023-10-16T11:00:42.451916+0000","last_fullsized":"2023-10-16T11:00:42.451916+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:14:40.752577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.449375+0000","last_change":"2023-10-16T11:00:40.453238+0000","last_active":"2023-10-16T11:00:42.449375+0000","last_peered":"2023-10-16T11:00:42.449375+0000","last_clean":"2023-10-16T11:00:42.449375+0000","last_became_active":"2023-10-16T11:00:40.452980+0000","last_became_peered":"2023-10-16T11:00:40.452980+0000","last_unstale":"2023-10-16T11:00:42.449375+0000","last_undegraded":"2023-10-16T11:00:42.449375+0000","last_fullsized":"2023-10-16T11:00:42.449375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:53.838374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-10-16T11:00:42.452420+0000","last_change":"2023-10-16T11:00:41.463407+0000","last_active":"2023-10-16T11:00:42.452420+0000","last_peered":"2023-10-16T11:00:42.452420+0000","last_clean":"2023-10-16T11:00:42.452420+0000","last_became_active":"2023-10-16T11:00:41.463061+0000","last_became_peered":"2023-10-16T11:00:41.463061+0000","last_unstale":"2023-10-16T11:00:42.452420+0000","last_undegraded":"2023-10-16T11:00:42.452420+0000","last_fullsized":"2023-10-16T11:00:42.452420+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:40.433356+0000","last_clean_scrub_stamp":"2023-10-16T11:00:40.433356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:10.847238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"171'6","reported_seq":17,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.485550+0000","last_change":"2023-10-16T11:00:40.451786+0000","last_active":"2023-10-16T11:00:41.485550+0000","last_peered":"2023-10-16T11:00:41.485550+0000","last_clean":"2023-10-16T11:00:41.485550+0000","last_became_active":"2023-10-16T11:00:40.451529+0000","last_became_peered":"2023-10-16T11:00:40.451529+0000","last_unstale":"2023-10-16T11:00:41.485550+0000","last_undegraded":"2023-10-16T11:00:41.485550+0000","last_fullsized":"2023-10-16T11:00:41.485550+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:00:39.429126+0000","last_clean_scrub_stamp":"2023-10-16T11:00:39.429126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:18:57.514911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"161'271","reported_seq":886,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-16T11:00:41.448356+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:00:41.448356+0000","last_peered":"2023-10-16T11:00:41.448356+0000","last_clean":"2023-10-16T11:00:41.448356+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:00:41.448356+0000","last_undegraded":"2023-10-16T11:00:41.448356+0000","last_fullsized":"2023-10-16T11:00:41.448356+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575770,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47740,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324100,"statfs":{"total":96636764160,"available":96587878400,"internally_reserved":0,"allocated":2736128,"data_stored":1723011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.38600000000000001,"15min":0.32500000000000001},"max":{"1min":0.83199999999999996,"5min":0.83399999999999996,"15min":0.88900000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.46600000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":0.86199999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.81999999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.60599999999999998,"5min":0.46300000000000002,"15min":0.313},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.85499999999999998},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.38400000000000001},"max":{"1min":0.79000000000000004,"5min":0.82199999999999995,"15min":0.86399999999999999},"last":0.44},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67300000000000004,"15min":0.66900000000000004},"min":{"1min":0.51900000000000002,"5min":0.498,"15min":0.32300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.57399999999999995}]},{"osd":3,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.35499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.50800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.90200000000000002},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.38700000000000001,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":0.67700000000000005,"5min":0.78500000000000003,"15min":0.81699999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.44900000000000001,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.84799999999999998},"last":0.40100000000000002}]},{"osd":5,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53300000000000003,"15min":0.52200000000000002},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.878},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.41699999999999998,"5min":0.40999999999999998,"15min":0.36199999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.92000000000000004},"last":0.46700000000000003}]},{"osd":6,"last update":"Mon Oct 16 10:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.42999999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.96099999999999997},"last":0.504},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.60399999999999998,"15min":0.58999999999999997},"min":{"1min":0.46899999999999997,"5min":0.38700000000000001,"15min":0.36699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.47999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575770,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2142208,"data_stored":1132691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.495,"5min":0.36399999999999999,"15min":0.29799999999999999},"max":{"1min":0.79400000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.61299999999999999,"15min":0.625},"min":{"1min":0.432,"5min":0.42099999999999999,"15min":0.32400000000000001},"max":{"1min":0.70799999999999996,"5min":0.80100000000000005,"15min":0.89200000000000002},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63200000000000001,"15min":0.61799999999999999},"min":{"1min":0.45300000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83299999999999996,"5min":0.872,"15min":0.872},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.621,"15min":0.624},"min":{"1min":0.5,"5min":0.40899999999999997,"15min":0.38600000000000001},"max":{"1min":0.77200000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.625,"15min":0.61199999999999999},"min":{"1min":0.47899999999999998,"5min":0.39600000000000002,"15min":0.38100000000000001},"max":{"1min":0.80800000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.49199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76400000000000001,"5min":0.90700000000000003,"15min":0.91000000000000003},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63600000000000001,"15min":0.629},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41599999999999998},"max":{"1min":0.75600000000000001,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.48999999999999999,"5min":0.39200000000000002,"15min":0.36399999999999999},"max":{"1min":0.80200000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.505,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.32100000000000001,"15min":0.29999999999999999},"max":{"1min":0.72499999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.75700000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60099999999999998}]},{"osd":5,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.53300000000000003},"min":{"1min":0.40100000000000002,"5min":0.35599999999999998,"15min":0.35399999999999998},"max":{"1min":0.69199999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.39000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.629}]},{"osd":7,"last update":"Mon Oct 16 10:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78800000000000003,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.89900000000000002,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575771,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":2142208,"data_stored":1132691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.432},"min":{"1min":0.245,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.64100000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46000000000000002,"15min":0.47199999999999998},"min":{"1min":0.35899999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.73599999999999999},"last":0.436}]},{"osd":2,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.54100000000000004,"5min":0.59499999999999997,"15min":0.66000000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45800000000000002,"15min":0.46500000000000002},"min":{"1min":0.36499999999999999,"5min":0.29099999999999998,"15min":0.27800000000000002},"max":{"1min":0.60599999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48099999999999998,"15min":0.49099999999999999},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32200000000000001},"max":{"1min":0.628,"5min":0.73099999999999998,"15min":0.748},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45800000000000002},"min":{"1min":0.33700000000000002,"5min":0.313,"15min":0.28999999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.78300000000000003},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.51700000000000002,"5min":0.48399999999999999,"15min":0.46000000000000002},"max":{"1min":0.83799999999999997,"5min":0.89000000000000001,"15min":0.93899999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88300000000000001,"5min":0.88500000000000001,"15min":0.91300000000000003},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46300000000000002,"15min":0.44},"max":{"1min":0.78600000000000003,"5min":0.92100000000000004,"15min":0.96399999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72399999999999998,"15min":0.72299999999999998},"min":{"1min":0.54300000000000004,"5min":0.48099999999999998,"15min":0.47199999999999998},"max":{"1min":0.92200000000000004,"5min":0.93400000000000005,"15min":0.97999999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.51000000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.94499999999999995,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.52300000000000002,"5min":0.48699999999999999,"15min":0.44},"max":{"1min":0.84199999999999997,"5min":0.93799999999999994,"15min":0.99399999999999999},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Oct 16 10:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.54200000000000004,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.85799999999999998,"5min":0.96899999999999997,"15min":0.98799999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70099999999999996,"15min":0.70699999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.98999999999999999},"last":0.55900000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38032,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333808,"statfs":{"total":96636764160,"available":96597819392,"internally_reserved":0,"allocated":2756608,"data_stored":1725307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.39400000000000002},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55000000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.67400000000000004,"15min":0.67600000000000005},"last":0.313}]},{"osd":2,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.434},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.94899999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49399999999999999,"15min":0.49299999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.044},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56999999999999995,"5min":0.64500000000000002,"15min":0.67000000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":0.67800000000000005,"15min":0.68100000000000005},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.36699999999999999},"max":{"1min":0.77100000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39400000000000002},"max":{"1min":0.79300000000000004,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.435,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81299999999999994,"5min":0.96999999999999997,"15min":0.98199999999999998},"last":0.59699999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.69199999999999995,"15min":0.69499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.98699999999999999,"15min":1.0209999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.69799999999999995,"15min":0.70199999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.96299999999999997},"last":0.621}]},{"osd":7,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.997,"15min":0.997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68100000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85099999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.65000000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575770,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63812,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308028,"statfs":{"total":96636764160,"available":96571420672,"internally_reserved":0,"allocated":2146304,"data_stored":1132733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40600000000000003,"15min":0.39600000000000002},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.97199999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.60899999999999999,"5min":0.67500000000000004,"15min":1.3180000000000001},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.30599999999999999,"5min":0.27800000000000002,"15min":0.27100000000000002},"max":{"1min":0.625,"5min":0.749,"15min":1.161},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.29099999999999998,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45100000000000001,"15min":0.45000000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.23799999999999999},"max":{"1min":0.58299999999999996,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70099999999999996,"5min":0.73699999999999999,"15min":1.431},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.66700000000000004,"15min":0.66200000000000003},"min":{"1min":0.44900000000000001,"5min":0.41599999999999998,"15min":0.40699999999999997},"max":{"1min":0.86699999999999999,"5min":0.98899999999999999,"15min":1.0129999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69299999999999995,"15min":0.68500000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.435},"max":{"1min":0.93999999999999995,"5min":0.96299999999999997,"15min":1.234},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":0.97399999999999998,"15min":1.1299999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96799999999999997,"5min":0.98799999999999999,"15min":1.278},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68799999999999994,"15min":0.67100000000000004},"min":{"1min":0.46999999999999997,"5min":0.432,"15min":0.432},"max":{"1min":0.84799999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.70899999999999996},"min":{"1min":0.54700000000000004,"5min":0.51100000000000001,"15min":0.432},"max":{"1min":0.89500000000000002,"5min":1.004,"15min":1.1299999999999999},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Oct 16 10:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.48199999999999998},"max":{"1min":0.99099999999999999,"5min":1.02,"15min":1.3879999999999999},"last":0.496},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71099999999999997,"15min":0.70299999999999996},"min":{"1min":0.502,"5min":0.47699999999999998,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":0.94499999999999995,"15min":1.2190000000000001},"last":0.54400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575770,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47168,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324672,"statfs":{"total":96636764160,"available":96588464128,"internally_reserved":0,"allocated":2146304,"data_stored":1133237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.438},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.41699999999999998,"15min":0.41999999999999998},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.47699999999999998,"5min":0.61099999999999999,"15min":0.68300000000000005},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56699999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":0.54500000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57499999999999996,"5min":0.69199999999999995,"15min":0.75600000000000001},"last":0.499},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.58699999999999997,"5min":0.70899999999999996,"15min":0.748},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81299999999999994,"5min":0.97199999999999998,"15min":0.97699999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67600000000000005,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":0.98899999999999999,"15min":1.0629999999999999},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.72399999999999998,"5min":0.99299999999999999,"15min":1.0629999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.68899999999999995},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81399999999999995,"5min":1.054,"15min":1.054},"last":0.81399999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.68899999999999995},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.82199999999999995,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.755},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.45800000000000002},"max":{"1min":0.79700000000000004,"5min":0.99399999999999999,"15min":1.02},"last":0.79700000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.53000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.84299999999999997,"5min":1.012,"15min":1.012},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.70699999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79400000000000004,"5min":0.97899999999999998,"15min":1.002},"last":0.77400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575770,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2162688,"data_stored":1141305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.81200000000000006,"15min":0.83799999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.629},"min":{"1min":0.42399999999999999,"5min":0.41099999999999998,"15min":0.315},"max":{"1min":0.77000000000000002,"5min":0.80500000000000005,"15min":0.84899999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61899999999999999},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78500000000000003,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.504},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.66600000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72599999999999998,"5min":0.85599999999999998,"15min":0.879},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.71299999999999997,"5min":0.83699999999999997,"15min":0.875},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.74099999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63700000000000001,"15min":0.63800000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77100000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61199999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.501},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73099999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.69399999999999995,"5min":0.83199999999999996,"15min":0.84499999999999997},"last":0.39100000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.375},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56599999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.41699999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38900000000000001},"max":{"1min":0.68899999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33400000000000002},"max":{"1min":0.70899999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59640,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312200,"statfs":{"total":96636764160,"available":96575692800,"internally_reserved":0,"allocated":2138112,"data_stored":1132621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64800000000000002,"15min":0.65600000000000003},"min":{"1min":0.504,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.77100000000000002,"5min":0.85899999999999999,"15min":0.86499999999999999},"last":0.503},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.46899999999999997,"15min":0.41599999999999998},"max":{"1min":0.78700000000000003,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59399999999999997,"15min":0.59799999999999998},"min":{"1min":0.36799999999999999,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":0.79600000000000004,"15min":0.82299999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70799999999999996,"5min":0.82099999999999995,"15min":0.83299999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":0.88200000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.623,"15min":0.623},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.82799999999999996},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.47999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71199999999999997,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.37,"5min":0.33500000000000002,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.78600000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.318,"15min":0.28000000000000003},"max":{"1min":0.69999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.63700000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.40000000000000002,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":0.66800000000000004,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56100000000000005,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68300000000000005,"5min":0.82999999999999996,"15min":0.84099999999999997},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.57699999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72699999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63600000000000001,"15min":0.63200000000000001},"min":{"1min":0.46000000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":0.77300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.71799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:00:45.973 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:00:45.973 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T11:00:46.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.133+0000 7fd314ad3700 1 Processor -- start 2023-10-16T11:00:46.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.133+0000 7fd314ad3700 1 -- start start 2023-10-16T11:00:46.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd314ad3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd310161e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd314ad3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd31006a8d0 0x7fd31006ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd314ad3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd310071d70 con 0x7fd3100b3b60 2023-10-16T11:00:46.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd314ad3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31006b2c0 con 0x7fd31006a8d0 2023-10-16T11:00:46.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd314ad3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100af9d0 con 0x7fd3100af5f0 2023-10-16T11:00:46.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd31006a8d0 0x7fd31006ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd31006a8d0 0x7fd31006ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33246/0 (socket says 172.21.15.158:33246) 2023-10-16T11:00:46.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 -- 172.21.15.158:0/1249578143 learned_addr learned my addr 172.21.15.158:0/1249578143 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:46.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30e59c700 1 --2- 172.21.15.158:0/1249578143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd310161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 -- 172.21.15.158:0/1249578143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 msgr2=0x7fd310161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:46.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 --2- 172.21.15.158:0/1249578143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd310161e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 -- 172.21.15.158:0/1249578143 >> v1:172.21.15.172:6789/0 conn(0x7fd3100b3b60 legacy=0x7fd3100724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:46.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 -- 172.21.15.158:0/1249578143 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3100b3970 con 0x7fd31006a8d0 2023-10-16T11:00:46.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30e59c700 1 --2- 172.21.15.158:0/1249578143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd310161e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:46.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.134+0000 7fd30ed9d700 1 --2- 172.21.15.158:0/1249578143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31006a8d0 0x7fd31006ad80 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd308009e50 tx=0x7fd30800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db23416b62ae5cd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:46.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd30d59a700 1 -- 172.21.15.158:0/1249578143 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800e070 con 0x7fd31006a8d0 2023-10-16T11:00:46.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd30d59a700 1 -- 172.21.15.158:0/1249578143 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd308003a20 con 0x7fd31006a8d0 2023-10-16T11:00:46.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd30d59a700 1 -- 172.21.15.158:0/1249578143 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308003dd0 con 0x7fd31006a8d0 2023-10-16T11:00:46.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 -- 172.21.15.158:0/1249578143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31006a8d0 msgr2=0x7fd31006ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 --2- 172.21.15.158:0/1249578143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31006a8d0 0x7fd31006ad80 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd308009e50 tx=0x7fd30800b330 comp rx=0 tx=0).stop 2023-10-16T11:00:46.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 -- 172.21.15.158:0/1249578143 shutdown_connections 2023-10-16T11:00:46.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 --2- 172.21.15.158:0/1249578143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31006a8d0 0x7fd31006ad80 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 --2- 172.21.15.158:0/1249578143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd310161e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 -- 172.21.15.158:0/1249578143 >> 172.21.15.158:0/1249578143 conn(0x7fd3100ccd30 msgr2=0x7fd3100cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:46.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 -- 172.21.15.158:0/1249578143 shutdown_connections 2023-10-16T11:00:46.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.135+0000 7fd314ad3700 1 -- 172.21.15.158:0/1249578143 wait complete. 2023-10-16T11:00:46.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 Processor -- start 2023-10-16T11:00:46.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 -- start start 2023-10-16T11:00:46.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31006a8d0 0x7fd3100ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd3100c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3100b3b60 0x7fd3101f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100b0cd0 con 0x7fd3100b3b60 2023-10-16T11:00:46.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3101f3300 con 0x7fd31006a8d0 2023-10-16T11:00:46.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd314ad3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3101f35a0 con 0x7fd3100af5f0 2023-10-16T11:00:46.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd30ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3100b3b60 0x7fd3101f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd30ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3100b3b60 0x7fd3101f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33252/0 (socket says 172.21.15.158:33252) 2023-10-16T11:00:46.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.137+0000 7fd30ed9d700 1 -- 172.21.15.158:0/3851153471 learned_addr learned my addr 172.21.15.158:0/3851153471 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:46.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30ed9d700 1 -- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 msgr2=0x7fd3100c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30e59c700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31006a8d0 0x7fd3100ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30dd9b700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd3100c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30ed9d700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd3100c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30ed9d700 1 -- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31006a8d0 msgr2=0x7fd3100ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30ed9d700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31006a8d0 0x7fd3100ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30ed9d700 1 -- 172.21.15.158:0/3851153471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd308009b30 con 0x7fd3100b3b60 2023-10-16T11:00:46.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30e59c700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31006a8d0 0x7fd3100ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:46.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd30ed9d700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3100b3b60 0x7fd3101f2dc0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd31006d2f0 tx=0x7fd308004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:46.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd2feffd700 1 -- 172.21.15.158:0/3851153471 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308004240 con 0x7fd3100b3b60 2023-10-16T11:00:46.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd2feffd700 1 -- 172.21.15.158:0/3851153471 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3080043a0 con 0x7fd3100b3b60 2023-10-16T11:00:46.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3101f3880 con 0x7fd3100b3b60 2023-10-16T11:00:46.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd2feffd700 1 -- 172.21.15.158:0/3851153471 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308004750 con 0x7fd3100b3b60 2023-10-16T11:00:46.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.138+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3100b0ee0 con 0x7fd3100b3b60 2023-10-16T11:00:46.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.139+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3100681b0 con 0x7fd3100b3b60 2023-10-16T11:00:46.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.140+0000 7fd2feffd700 1 -- 172.21.15.158:0/3851153471 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd308019070 con 0x7fd3100b3b60 2023-10-16T11:00:46.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.140+0000 7fd2feffd700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd2f006b020 0x7fd2f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.140+0000 7fd2feffd700 1 -- 172.21.15.158:0/3851153471 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fd30808ab70 con 0x7fd3100b3b60 2023-10-16T11:00:46.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.140+0000 7fd30e59c700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd2f006b020 0x7fd2f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.140+0000 7fd30e59c700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd2f006b020 0x7fd2f006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3100c7930 tx=0x7fd300008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:46.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.144+0000 7fd2feffd700 1 -- 172.21.15.158:0/3851153471 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd308055700 con 0x7fd3100b3b60 2023-10-16T11:00:46.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.418+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd3100cd960 con 0x7fd3100b3b60 2023-10-16T11:00:46.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.420+0000 7fd2feffd700 1 -- 172.21.15.158:0/3851153471 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v44) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd3080200b0 con 0x7fd3100b3b60 2023-10-16T11:00:46.422 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T11:00:46.422 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T11:00:46.422 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T11:00:46.422 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T11:00:46.423 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T11:00:46.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd2f006b020 msgr2=0x7fd2f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd2f006b020 0x7fd2f006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3100c7930 tx=0x7fd300008040 comp rx=0 tx=0).stop 2023-10-16T11:00:46.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3100b3b60 msgr2=0x7fd3101f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3100b3b60 0x7fd3101f2dc0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd31006d2f0 tx=0x7fd308004ad0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 shutdown_connections 2023-10-16T11:00:46.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3100b3b60 0x7fd3101f2dc0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fd2f006b020 0x7fd2f006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd31006a8d0 0x7fd3100ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 --2- 172.21.15.158:0/3851153471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3100af5f0 0x7fd3100c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 >> 172.21.15.158:0/3851153471 conn(0x7fd3100ccd30 msgr2=0x7fd3100cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:46.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 shutdown_connections 2023-10-16T11:00:46.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.423+0000 7fd314ad3700 1 -- 172.21.15.158:0/3851153471 wait complete. 2023-10-16T11:00:46.427 INFO:tasks.ceph.mds.d.smithi172.stderr:2023-10-16T11:00:46.424+0000 7fc0a2da7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:00:46.427 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T11:00:46.437 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:00:46.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.558+0000 7fba19524700 1 Processor -- start 2023-10-16T11:00:46.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.558+0000 7fba19524700 1 -- start start 2023-10-16T11:00:46.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.558+0000 7fba19524700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fba140cba60 0x7fba140cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.558+0000 7fba19524700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba140b4240 0x7fba1415f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.558+0000 7fba19524700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140cf4f0 con 0x7fba1415f5a0 2023-10-16T11:00:46.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.558+0000 7fba19524700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba14160070 con 0x7fba140cba60 2023-10-16T11:00:46.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.558+0000 7fba19524700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba140cc380 con 0x7fba140b4240 2023-10-16T11:00:46.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba12ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fba140cba60 0x7fba140cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba137fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fba1415f5a0 0x7fba140cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33436/0 (socket says 172.21.15.158:33436) 2023-10-16T11:00:46.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba137fe700 1 -- 172.21.15.158:0/509913668 learned_addr learned my addr 172.21.15.158:0/509913668 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:46.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba127fc700 1 --2- 172.21.15.158:0/509913668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba140b4240 0x7fba1415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba12ffd700 1 -- 172.21.15.158:0/509913668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba140b4240 msgr2=0x7fba1415f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:46.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba12ffd700 1 --2- 172.21.15.158:0/509913668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba140b4240 0x7fba1415f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba12ffd700 1 -- 172.21.15.158:0/509913668 >> v1:172.21.15.158:6789/0 conn(0x7fba1415f5a0 legacy=0x7fba140cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba12ffd700 1 -- 172.21.15.158:0/509913668 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1415fab0 con 0x7fba140cba60 2023-10-16T11:00:46.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba11ffb700 1 -- 172.21.15.158:0/509913668 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176327709 0 0) 0x7fba140cf4f0 con 0x7fba1415f5a0 2023-10-16T11:00:46.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba127fc700 1 --2- 172.21.15.158:0/509913668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba140b4240 0x7fba1415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:46.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba12ffd700 1 --2- 172.21.15.158:0/509913668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140cba60 0x7fba140cbe40 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fba04009840 tx=0x7fba04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c869d9db9917aa03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:46.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba11ffb700 1 -- 172.21.15.158:0/509913668 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04011830 con 0x7fba140cba60 2023-10-16T11:00:46.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba11ffb700 1 -- 172.21.15.158:0/509913668 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba04011990 con 0x7fba140cba60 2023-10-16T11:00:46.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.559+0000 7fba11ffb700 1 -- 172.21.15.158:0/509913668 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04011d20 con 0x7fba140cba60 2023-10-16T11:00:46.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 -- 172.21.15.158:0/509913668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140cba60 msgr2=0x7fba140cbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 --2- 172.21.15.158:0/509913668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140cba60 0x7fba140cbe40 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fba04009840 tx=0x7fba04009b50 comp rx=0 tx=0).stop 2023-10-16T11:00:46.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 -- 172.21.15.158:0/509913668 shutdown_connections 2023-10-16T11:00:46.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 --2- 172.21.15.158:0/509913668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140cba60 0x7fba140cbe40 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 --2- 172.21.15.158:0/509913668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba140b4240 0x7fba1415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 -- 172.21.15.158:0/509913668 >> 172.21.15.158:0/509913668 conn(0x7fba14071220 msgr2=0x7fba1406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:46.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 -- 172.21.15.158:0/509913668 shutdown_connections 2023-10-16T11:00:46.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.560+0000 7fba19524700 1 -- 172.21.15.158:0/509913668 wait complete. 2023-10-16T11:00:46.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.561+0000 7fba19524700 1 Processor -- start 2023-10-16T11:00:46.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.561+0000 7fba19524700 1 -- start start 2023-10-16T11:00:46.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba19524700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140b4240 0x7fba141f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba19524700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba140cba60 0x7fba1415da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba19524700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1415f5a0 0x7fba140c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba19524700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1415c770 con 0x7fba140b4240 2023-10-16T11:00:46.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba19524700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba141f3f50 con 0x7fba140cba60 2023-10-16T11:00:46.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba19524700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1415c9c0 con 0x7fba1415f5a0 2023-10-16T11:00:46.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140b4240 0x7fba141f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140b4240 0x7fba141f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33262/0 (socket says 172.21.15.158:33262) 2023-10-16T11:00:46.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 -- 172.21.15.158:0/1289227432 learned_addr learned my addr 172.21.15.158:0/1289227432 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:46.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba137fe700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1415f5a0 0x7fba140c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba127fc700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba140cba60 0x7fba1415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 -- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1415f5a0 msgr2=0x7fba140c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1415f5a0 0x7fba140c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 -- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba140cba60 msgr2=0x7fba1415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba140cba60 0x7fba1415da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 -- 172.21.15.158:0/1289227432 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba040094a0 con 0x7fba140b4240 2023-10-16T11:00:46.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.562+0000 7fba12ffd700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140b4240 0x7fba141f3a10 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fba04000c00 tx=0x7fba04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:46.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.563+0000 7fb9fb7fe700 1 -- 172.21.15.158:0/1289227432 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba040045f0 con 0x7fba140b4240 2023-10-16T11:00:46.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.563+0000 7fb9fb7fe700 1 -- 172.21.15.158:0/1289227432 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0401f070 con 0x7fba140b4240 2023-10-16T11:00:46.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.563+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1415cca0 con 0x7fba140b4240 2023-10-16T11:00:46.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.563+0000 7fb9fb7fe700 1 -- 172.21.15.158:0/1289227432 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04003ae0 con 0x7fba140b4240 2023-10-16T11:00:46.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.563+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1415d260 con 0x7fba140b4240 2023-10-16T11:00:46.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.564+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba140681b0 con 0x7fba140b4240 2023-10-16T11:00:46.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.564+0000 7fb9fb7fe700 1 -- 172.21.15.158:0/1289227432 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fba04003c40 con 0x7fba140b4240 2023-10-16T11:00:46.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.565+0000 7fb9fb7fe700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fba0006b020 0x7fba0006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:46.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.565+0000 7fb9fb7fe700 1 -- 172.21.15.158:0/1289227432 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fba04021070 con 0x7fba140b4240 2023-10-16T11:00:46.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.565+0000 7fba127fc700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fba0006b020 0x7fba0006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:46.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.565+0000 7fba127fc700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fba0006b020 0x7fba0006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba1406dbc0 tx=0x7fba08008040 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:46.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.569+0000 7fb9fb7fe700 1 -- 172.21.15.158:0/1289227432 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba040566a0 con 0x7fba140b4240 2023-10-16T11:00:46.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.805+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba140d1180 con 0x7fba140b4240 2023-10-16T11:00:46.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.805+0000 7fb9fb7fe700 1 -- 172.21.15.158:0/1289227432 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4929 (secure 0 0 0) 0x7fba04015090 con 0x7fba140b4240 2023-10-16T11:00:46.807 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:46.807 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3392606972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3392606972},{"type":"v1","addr":"172.21.15.172:6834","nonce":3392606972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9930,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3321991635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3321991635},{"type":"v1","addr":"172.21.15.158:6836","nonce":3321991635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4013459160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4013459160},{"type":"v1","addr":"172.21.15.158:6837","nonce":4013459160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:00:40.840318+0000","modified":"2023-10-16T11:00:43.870189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37712":{"gid":37712,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/551179403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":551179403},{"type":"v1","addr":"172.21.15.172:6835","nonce":551179403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T11:00:46.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fba0006b020 msgr2=0x7fba0006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fba0006b020 0x7fba0006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba1406dbc0 tx=0x7fba08008040 comp rx=0 tx=0).stop 2023-10-16T11:00:46.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140b4240 msgr2=0x7fba141f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:46.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140b4240 0x7fba141f3a10 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fba04000c00 tx=0x7fba04004750 comp rx=0 tx=0).stop 2023-10-16T11:00:46.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 shutdown_connections 2023-10-16T11:00:46.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1415f5a0 0x7fba140c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba140b4240 0x7fba141f3a10 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fba0006b020 0x7fba0006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 --2- 172.21.15.158:0/1289227432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba140cba60 0x7fba1415da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:46.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 >> 172.21.15.158:0/1289227432 conn(0x7fba14071220 msgr2=0x7fba1406ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:46.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 shutdown_connections 2023-10-16T11:00:46.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:46.807+0000 7fba19524700 1 -- 172.21.15.158:0/1289227432 wait complete. 2023-10-16T11:00:46.812 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 85 2023-10-16T11:00:47.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:00:40.840318+0000', 'modified': '2023-10-16T11:00:43.870189+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 37712}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37712': {'gid': 37712, 'name': 'd', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/551179403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 551179403}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 551179403}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:00:47.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:00:47.225 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T11:00:47.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.346+0000 7f1751758700 1 Processor -- start 2023-10-16T11:00:47.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.346+0000 7f1751758700 1 -- start start 2023-10-16T11:00:47.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f1751758700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f174c15da60 0x7f174c0cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f1751758700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c0d1db0 con 0x7f174c0ce5b0 2023-10-16T11:00:47.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f1751758700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c0cfd60 con 0x7f174c0b5870 2023-10-16T11:00:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f1751758700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174c0d1760 con 0x7f174c15da60 2023-10-16T11:00:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f174a7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f174c0ce5b0 0x7f174c0b5160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33446/0 (socket says 172.21.15.158:33446) 2023-10-16T11:00:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f174a7fc700 1 -- 172.21.15.158:0/2162393858 learned_addr learned my addr 172.21.15.158:0/2162393858 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701785697 0 0) 0x7f174c0d1db0 con 0x7f174c0ce5b0 2023-10-16T11:00:47.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f174affd700 1 --2- 172.21.15.158:0/2162393858 >> v2:172.21.15.172:3300/0 conn(0x7f174c15da60 0x7f174c0cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.347+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17340035b0 con 0x7f174c0ce5b0 2023-10-16T11:00:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648298809 0 0) 0x7f17340035b0 con 0x7f174c0ce5b0 2023-10-16T11:00:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 >> v2:172.21.15.172:3300/0 conn(0x7f174c15da60 msgr2=0x7f174c0cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 --2- 172.21.15.158:0/2162393858 >> v2:172.21.15.172:3300/0 conn(0x7f174c15da60 0x7f174c0cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 >> v1:172.21.15.172:6789/0 conn(0x7f174c0b5870 legacy=0x7f174c0d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c0d1a40 con 0x7f174c0ce5b0 2023-10-16T11:00:47.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f173c001670 con 0x7f174c0ce5b0 2023-10-16T11:00:47.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 >> v1:172.21.15.158:6789/0 conn(0x7f174c0ce5b0 legacy=0x7f174c0b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1734003a00 0x7f1734003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1734004850 0x7f1734004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734005210 0x7f173400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0d1760 con 0x7f1734003a00 2023-10-16T11:00:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173400c0b0 con 0x7f1734005210 2023-10-16T11:00:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173400c270 con 0x7f1734004850 2023-10-16T11:00:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174affd700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1734003a00 0x7f1734003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174a7fc700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1734004850 0x7f1734004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174b7fe700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734005210 0x7f173400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174affd700 1 -- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1734004850 msgr2=0x7f1734004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174affd700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1734004850 0x7f1734004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174affd700 1 -- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734005210 msgr2=0x7f173400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174affd700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734005210 0x7f173400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174affd700 1 -- 172.21.15.158:0/2162393858 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340035b0 con 0x7f1734003a00 2023-10-16T11:00:47.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174a7fc700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1734004850 0x7f1734004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174b7fe700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734005210 0x7f173400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174b7fe700 1 -- 172.21.15.158:0/2162393858 reap_dead start 2023-10-16T11:00:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f174affd700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1734003a00 0x7f1734003e20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f1740009b40 tx=0x7f1734005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2504aa80190271 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740005a20 con 0x7f1734003a00 2023-10-16T11:00:47.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1740005b80 con 0x7f1734003a00 2023-10-16T11:00:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.348+0000 7f1749ffb700 1 -- 172.21.15.158:0/2162393858 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174000e6c0 con 0x7f1734003a00 2023-10-16T11:00:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.349+0000 7f1751758700 1 -- 172.21.15.158:0/2162393858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1734003a00 msgr2=0x7f1734003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.349+0000 7f1751758700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1734003a00 0x7f1734003e20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f1740009b40 tx=0x7f1734005880 comp rx=0 tx=0).stop 2023-10-16T11:00:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.349+0000 7f1751758700 1 -- 172.21.15.158:0/2162393858 shutdown_connections 2023-10-16T11:00:47.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.349+0000 7f1751758700 1 --2- 172.21.15.158:0/2162393858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1734003a00 0x7f1734003e20 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.349+0000 7f1751758700 1 -- 172.21.15.158:0/2162393858 >> 172.21.15.158:0/2162393858 conn(0x7f174c09c9b0 msgr2=0x7f174c0d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.349+0000 7f1751758700 1 -- 172.21.15.158:0/2162393858 shutdown_connections 2023-10-16T11:00:47.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.349+0000 7f1751758700 1 -- 172.21.15.158:0/2162393858 wait complete. 2023-10-16T11:00:47.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.350+0000 7f1751758700 1 Processor -- start 2023-10-16T11:00:47.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.350+0000 7f1751758700 1 -- start start 2023-10-16T11:00:47.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f1751758700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734003a00 0x7f174c0b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f1751758700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f174c0d2710 0x7f174c0d2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f1751758700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c1f69b0 0x7f174c1f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f1751758700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0d30d0 con 0x7f174c1f69b0 2023-10-16T11:00:47.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f1751758700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1f7730 con 0x7f1734003a00 2023-10-16T11:00:47.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f1751758700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0cb180 con 0x7f174c0d2710 2023-10-16T11:00:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c1f69b0 0x7f174c1f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c1f69b0 0x7f174c1f6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33282/0 (socket says 172.21.15.158:33282) 2023-10-16T11:00:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f174b7fe700 1 -- 172.21.15.158:0/1068187736 learned_addr learned my addr 172.21.15.158:0/1068187736 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f174a7fc700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f174c0d2710 0x7f174c0d2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.351+0000 7f174affd700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734003a00 0x7f174c0b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174b7fe700 1 -- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f174c0d2710 msgr2=0x7f174c0d2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174b7fe700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f174c0d2710 0x7f174c0d2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174b7fe700 1 -- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734003a00 msgr2=0x7f174c0b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174b7fe700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734003a00 0x7f174c0b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174b7fe700 1 -- 172.21.15.158:0/1068187736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c0ca850 con 0x7f174c1f69b0 2023-10-16T11:00:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174a7fc700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f174c0d2710 0x7f174c0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:47.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174affd700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734003a00 0x7f174c0b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f174b7fe700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c1f69b0 0x7f174c1f6e60 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f174c06b100 tx=0x7f174400ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f17337fe700 1 -- 172.21.15.158:0/1068187736 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744003e10 con 0x7f174c1f69b0 2023-10-16T11:00:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f17337fe700 1 -- 172.21.15.158:0/1068187736 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1744005070 con 0x7f174c1f69b0 2023-10-16T11:00:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c0cab30 con 0x7f174c1f69b0 2023-10-16T11:00:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f17337fe700 1 -- 172.21.15.158:0/1068187736 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744005420 con 0x7f174c1f69b0 2023-10-16T11:00:47.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.352+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c0cb610 con 0x7f174c1f69b0 2023-10-16T11:00:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.353+0000 7f17337fe700 1 -- 172.21.15.158:0/1068187736 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1744005580 con 0x7f174c1f69b0 2023-10-16T11:00:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.354+0000 7f17337fe700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f173806b020 0x7f173806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.354+0000 7f17337fe700 1 -- 172.21.15.158:0/1068187736 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f17440895e0 con 0x7f174c1f69b0 2023-10-16T11:00:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.354+0000 7f174affd700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f173806b020 0x7f173806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.354+0000 7f174affd700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f173806b020 0x7f173806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1740000c00 tx=0x7f1740008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.356+0000 7f175d942b80 1 Processor -- start 2023-10-16T11:00:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.356+0000 7f175d942b80 1 -- start start 2023-10-16T11:00:47.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.356+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe057a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe055630 0x562bfe057fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f175d942b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe02b010 0x562bfe02b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f175d942b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfdfa1c20 con 0x562bfe02b010 2023-10-16T11:00:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfdf86c10 con 0x562bfe050220 2023-10-16T11:00:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bfdf84530 con 0x562bfe055630 2023-10-16T11:00:47.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe02b010 0x562bfe02b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe02b010 0x562bfe02b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33290/0 (socket says 172.21.15.158:33290) 2023-10-16T11:00:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 -- 172.21.15.158:0/674779359 learned_addr learned my addr 172.21.15.158:0/674779359 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 -- 172.21.15.158:0/674779359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe055630 msgr2=0x562bfe057fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 --2- 172.21.15.158:0/674779359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe055630 0x562bfe057fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 -- 172.21.15.158:0/674779359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 msgr2=0x562bfe057a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 --2- 172.21.15.158:0/674779359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe057a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 -- 172.21.15.158:0/674779359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bfdf3d3a0 con 0x562bfe02b010 2023-10-16T11:00:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f174b7fe700 1 --2- 172.21.15.158:0/674779359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe02b010 0x562bfe02b6c0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f1744087660 tx=0x7f1744057890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6c3612e0f4c0eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f1730ff9700 1 -- 172.21.15.158:0/674779359 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174405c000 con 0x562bfe02b010 2023-10-16T11:00:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f1730ff9700 1 -- 172.21.15.158:0/674779359 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17440601b0 con 0x562bfe02b010 2023-10-16T11:00:47.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.357+0000 7f1730ff9700 1 -- 172.21.15.158:0/674779359 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174405e0c0 con 0x562bfe02b010 2023-10-16T11:00:47.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 -- 172.21.15.158:0/674779359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe02b010 msgr2=0x562bfe02b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 --2- 172.21.15.158:0/674779359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe02b010 0x562bfe02b6c0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f1744087660 tx=0x7f1744057890 comp rx=0 tx=0).stop 2023-10-16T11:00:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 -- 172.21.15.158:0/674779359 shutdown_connections 2023-10-16T11:00:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 --2- 172.21.15.158:0/674779359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe02b010 0x562bfe02b6c0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 --2- 172.21.15.158:0/674779359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe057a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 --2- 172.21.15.158:0/674779359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe055630 0x562bfe057fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 -- 172.21.15.158:0/674779359 >> 172.21.15.158:0/674779359 conn(0x562bfdf5ad80 msgr2=0x562bfe075ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 -- 172.21.15.158:0/674779359 shutdown_connections 2023-10-16T11:00:47.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.358+0000 7f175d942b80 1 -- 172.21.15.158:0/674779359 wait complete. 2023-10-16T11:00:47.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 -- start start 2023-10-16T11:00:47.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 Processor -- start 2023-10-16T11:00:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe05adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe085650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe055630 0x562bfe085b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfdf7fe20 con 0x562bfe055630 2023-10-16T11:00:47.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfdf1c6c0 con 0x562bfe050220 2023-10-16T11:00:47.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bfdf1c360 con 0x562bfe02b010 2023-10-16T11:00:47.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f175d942b80 10 client.0 _ll_register_callbacks cb 0x562bfdd797b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:00:47.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.359+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe055630 0x562bfe085b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe055630 0x562bfe085b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33296/0 (socket says 172.21.15.158:33296) 2023-10-16T11:00:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 -- 172.21.15.158:0/980417237 learned_addr learned my addr 172.21.15.158:0/980417237 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174affd700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe05adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 -- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 msgr2=0x562bfe05adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174a7fc700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe085650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe05adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 -- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 msgr2=0x562bfe085650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe085650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 -- 172.21.15.158:0/980417237 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c0cb610 con 0x562bfe055630 2023-10-16T11:00:47.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174affd700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe05adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:47.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174a7fc700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe085650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:47.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f174b7fe700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe055630 0x562bfe085b90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f1744015040 tx=0x7f174400bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78acef7360b9591 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f1729ffb700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:00:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f1729ffb700 1 -- 172.21.15.158:0/980417237 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174405e0c0 con 0x562bfe055630 2023-10-16T11:00:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562bfe0205b0 con 0x562bfe055630 2023-10-16T11:00:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f1729ffb700 1 -- 172.21.15.158:0/980417237 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f17440871f0 con 0x562bfe055630 2023-10-16T11:00:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f1729ffb700 1 -- 172.21.15.158:0/980417237 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17440573a0 con 0x562bfe055630 2023-10-16T11:00:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 10 client.10080 fetch_fsmap learned FSMap version 85 2023-10-16T11:00:47.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) v3 -- 0x562bfddaad90 con 0x562bfe055630 2023-10-16T11:00:47.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f1729ffb700 1 -- 172.21.15.158:0/980417237 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562bfe0205b0 con 0x562bfe055630 2023-10-16T11:00:47.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f1729ffb700 1 -- 172.21.15.158:0/980417237 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 85) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f174405fd20 con 0x562bfe055630 2023-10-16T11:00:47.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 10 client.10080 fetch_fsmap finished waiting for FSMap version 85 2023-10-16T11:00:47.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 10 client.10080 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:00:47.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 20 client.10080 populate_metadata read hostname 'smithi158' 2023-10-16T11:00:47.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe04f260 0x562bfe058730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 4 client.10080 mds_command: new command op to 37712 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:00:47.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.360+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 --> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562bfdf78e00 con 0x562bfe04f260 2023-10-16T11:00:47.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.361+0000 7f174affd700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe04f260 0x562bfe058730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.361+0000 7f1729ffb700 10 client.10080 ms_handle_connect on v2:172.21.15.172:6833/551179403 2023-10-16T11:00:47.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.361+0000 7f174affd700 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe04f260 0x562bfe058730 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9560d4103a84230a server_cookie=cd3a09b385c10b63 in_seq=0 out_seq=0 2023-10-16T11:00:47.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f1729ffb700 1 -- 172.21.15.158:0/980417237 <== mds.0 v2:172.21.15.172:6833/551179403 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562bfdf78e00 con 0x562bfe04f260 2023-10-16T11:00:47.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f1729ffb700 10 client.10080 handle_command_reply: tid=0 2023-10-16T11:00:47.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 client.10080 shutdown 2023-10-16T11:00:47.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe055630 msgr2=0x562bfe085b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe055630 0x562bfe085b90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f1744015040 tx=0x7f174400bfb0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 shutdown_connections 2023-10-16T11:00:47.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe04f260 0x562bfe058730 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe055630 0x562bfe085b90 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe05adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f1729ffb700 0 client.10080 ms_handle_reset on v2:172.21.15.172:6833/551179403 2023-10-16T11:00:47.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 --2- 172.21.15.158:0/980417237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe085650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 >> 172.21.15.158:0/980417237 conn(0x562bfdf5ad80 msgr2=0x562bfe057a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 shutdown_connections 2023-10-16T11:00:47.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 1 -- 172.21.15.158:0/980417237 wait complete. 2023-10-16T11:00:47.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.362+0000 7f175d942b80 20 client.10080 trim_cache size 0 max 16384 2023-10-16T11:00:47.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 Processor -- start 2023-10-16T11:00:47.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 -- start start 2023-10-16T11:00:47.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe028200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe04f260 0x562bfe08c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe04d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfde008d0 con 0x562bfe04f260 2023-10-16T11:00:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfe04e780 con 0x562bfe050220 2023-10-16T11:00:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.380+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bfe02b7f0 con 0x562bfe02b010 2023-10-16T11:00:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe04f260 0x562bfe08c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe04f260 0x562bfe08c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33302/0 (socket says 172.21.15.158:33302) 2023-10-16T11:00:47.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 -- 172.21.15.158:0/4208449473 learned_addr learned my addr 172.21.15.158:0/4208449473 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174affd700 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe028200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174b7fe700 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe04d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 -- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 msgr2=0x562bfe028200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe028200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 -- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 msgr2=0x562bfe04d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe04d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 -- 172.21.15.158:0/4208449473 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bfdd93c10 con 0x562bfe04f260 2023-10-16T11:00:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f174a7fc700 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe04f260 0x562bfe08c710 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f173c000c00 tx=0x7f173c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b14fa18900d3f33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f17317fa700 1 -- 172.21.15.158:0/4208449473 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c011ac0 con 0x562bfe04f260 2023-10-16T11:00:47.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f17317fa700 1 -- 172.21.15.158:0/4208449473 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c011c20 con 0x562bfe04f260 2023-10-16T11:00:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.381+0000 7f17317fa700 1 -- 172.21.15.158:0/4208449473 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c005e60 con 0x562bfe04f260 2023-10-16T11:00:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 -- 172.21.15.158:0/4208449473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe04f260 msgr2=0x562bfe08c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe04f260 0x562bfe08c710 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f173c000c00 tx=0x7f173c009040 comp rx=0 tx=0).stop 2023-10-16T11:00:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 -- 172.21.15.158:0/4208449473 shutdown_connections 2023-10-16T11:00:47.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe050220 0x562bfe04d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe02b010 0x562bfe028200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 --2- 172.21.15.158:0/4208449473 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe04f260 0x562bfe08c710 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 -- 172.21.15.158:0/4208449473 >> 172.21.15.158:0/4208449473 conn(0x562bfdf5ad80 msgr2=0x562bfe021f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 -- 172.21.15.158:0/4208449473 shutdown_connections 2023-10-16T11:00:47.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 -- 172.21.15.158:0/4208449473 wait complete. 2023-10-16T11:00:47.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 -- start start 2023-10-16T11:00:47.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.382+0000 7f175d942b80 1 Processor -- start 2023-10-16T11:00:47.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe02b010 0x562bfe0d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f175d942b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe04f260 0x562bfe0589c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f175d942b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe050220 0x562bfe08fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f175d942b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfe0224e0 con 0x562bfe050220 2023-10-16T11:00:47.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bfe090680 con 0x562bfe02b010 2023-10-16T11:00:47.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f175d942b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bfe022650 con 0x562bfe04f260 2023-10-16T11:00:47.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f175d942b80 10 client.0 _ll_register_callbacks cb 0x7f1744057a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:00:47.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe050220 0x562bfe08fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe050220 0x562bfe08fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33308/0 (socket says 172.21.15.158:33308) 2023-10-16T11:00:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 -- 172.21.15.158:0/2024733944 learned_addr learned my addr 172.21.15.158:0/2024733944 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174a7fc700 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe04f260 0x562bfe0589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174affd700 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe02b010 0x562bfe0d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 -- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe04f260 msgr2=0x562bfe0589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe04f260 0x562bfe0589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 -- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe02b010 msgr2=0x562bfe0d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe02b010 0x562bfe0d0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.383+0000 7f174b7fe700 1 -- 172.21.15.158:0/2024733944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c0d1a40 con 0x562bfe050220 2023-10-16T11:00:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f174b7fe700 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe050220 0x562bfe08fbb0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f1744001330 tx=0x7f17440541f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4d59b7f745c9fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f1729ffb700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:00:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f1729ffb700 1 -- 172.21.15.158:0/2024733944 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744057c80 con 0x562bfe050220 2023-10-16T11:00:47.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f1729ffb700 1 -- 172.21.15.158:0/2024733944 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f1744076e40 con 0x562bfe050220 2023-10-16T11:00:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17440601b0 con 0x562bfe050220 2023-10-16T11:00:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f1729ffb700 1 -- 172.21.15.158:0/2024733944 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f174405e0c0 con 0x562bfe050220 2023-10-16T11:00:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 10 client.10086 fetch_fsmap learned FSMap version 85 2023-10-16T11:00:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) v3 -- 0x562bfddd1200 con 0x562bfe050220 2023-10-16T11:00:47.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f1729ffb700 1 -- 172.21.15.158:0/2024733944 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17440601b0 con 0x562bfe050220 2023-10-16T11:00:47.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f1729ffb700 1 -- 172.21.15.158:0/2024733944 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 85) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f174405fd20 con 0x562bfe050220 2023-10-16T11:00:47.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 10 client.10086 fetch_fsmap finished waiting for FSMap version 85 2023-10-16T11:00:47.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 10 client.10086 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:00:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 20 client.10086 populate_metadata read hostname 'smithi158' 2023-10-16T11:00:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe055630 0x562bfe02c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 4 client.10086 mds_command: new command op to 37712 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:00:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 --> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562bfdf78b20 con 0x562bfe055630 2023-10-16T11:00:47.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.384+0000 7f174affd700 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe055630 0x562bfe02c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f1729ffb700 10 client.10086 ms_handle_connect on v2:172.21.15.172:6833/551179403 2023-10-16T11:00:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f174affd700 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe055630 0x562bfe02c7c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=316be5df7fd28719 server_cookie=b2afb8c6925bc15b in_seq=0 out_seq=0 2023-10-16T11:00:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f1729ffb700 1 -- 172.21.15.158:0/2024733944 <== mds.0 v2:172.21.15.172:6833/551179403 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562bfdf78b20 con 0x562bfe055630 2023-10-16T11:00:47.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f1729ffb700 10 client.10086 handle_command_reply: tid=0 2023-10-16T11:00:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 client.10086 shutdown 2023-10-16T11:00:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe050220 msgr2=0x562bfe08fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe050220 0x562bfe08fbb0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f1744001330 tx=0x7f17440541f0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 shutdown_connections 2023-10-16T11:00:47.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:6833/551179403,v1:172.21.15.172:6835/551179403] conn(0x562bfe055630 0x562bfe02c7c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562bfe050220 0x562bfe08fbb0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f1729ffb700 0 client.10086 ms_handle_reset on v2:172.21.15.172:6833/551179403 2023-10-16T11:00:47.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bfe02b010 0x562bfe0d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 --2- 172.21.15.158:0/2024733944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x562bfe04f260 0x562bfe0589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 >> 172.21.15.158:0/2024733944 conn(0x562bfdf5ad80 msgr2=0x562bfe08c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 shutdown_connections 2023-10-16T11:00:47.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 1 -- 172.21.15.158:0/2024733944 wait complete. 2023-10-16T11:00:47.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.385+0000 7f175d942b80 20 client.10086 trim_cache size 0 max 16384 2023-10-16T11:00:47.419 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:00:47.419 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:00:47.420 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:00:47.420 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 37712, 2023-10-16T11:00:47.420 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:00:47.420 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:00:47.421 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T11:00:47.421 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9222421750000001, 2023-10-16T11:00:47.421 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 85, 2023-10-16T11:00:47.421 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 172, 2023-10-16T11:00:47.421 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 172, 2023-10-16T11:00:47.422 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 11.87219749 2023-10-16T11:00:47.422 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:00:47.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f173806b020 msgr2=0x7f173806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f173806b020 0x7f173806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1740000c00 tx=0x7f1740008030 comp rx=0 tx=0).stop 2023-10-16T11:00:47.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c1f69b0 msgr2=0x7f174c1f6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c1f69b0 0x7f174c1f6e60 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f174c06b100 tx=0x7f174400ddf0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 shutdown_connections 2023-10-16T11:00:47.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c1f69b0 0x7f174c1f6e60 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f173806b020 0x7f173806d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1734003a00 0x7f174c0b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 --2- 172.21.15.158:0/1068187736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f174c0d2710 0x7f174c0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 >> 172.21.15.158:0/1068187736 conn(0x7f174c09c9b0 msgr2=0x7f174c1e63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 shutdown_connections 2023-10-16T11:00:47.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.386+0000 7f1751758700 1 -- 172.21.15.158:0/1068187736 wait complete. 2023-10-16T11:00:47.427 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T11:00:47.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.548+0000 7ff860d5c700 1 Processor -- start 2023-10-16T11:00:47.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.548+0000 7ff860d5c700 1 -- start start 2023-10-16T11:00:47.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff860d5c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff85c15b330 0x7ff85c0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff860d5c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff85c15f760 0x7ff85c0716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff860d5c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c15ff10 con 0x7ff85c15b710 2023-10-16T11:00:47.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff860d5c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85c072440 con 0x7ff85c15b330 2023-10-16T11:00:47.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff860d5c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85c15c250 con 0x7ff85c15f760 2023-10-16T11:00:47.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff85a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff85c15b330 0x7ff85c0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff85a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff85c15b330 0x7ff85c0711a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33312/0 (socket says 172.21.15.158:33312) 2023-10-16T11:00:47.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.549+0000 7ff85a59c700 1 -- 172.21.15.158:0/565855378 learned_addr learned my addr 172.21.15.158:0/565855378 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:47.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85ad9d700 1 --2- 172.21.15.158:0/565855378 >> v2:172.21.15.172:3300/0 conn(0x7ff85c15f760 0x7ff85c0716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85a59c700 1 -- 172.21.15.158:0/565855378 >> v2:172.21.15.172:3300/0 conn(0x7ff85c15f760 msgr2=0x7ff85c0716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85a59c700 1 --2- 172.21.15.158:0/565855378 >> v2:172.21.15.172:3300/0 conn(0x7ff85c15f760 0x7ff85c0716e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85a59c700 1 -- 172.21.15.158:0/565855378 >> v1:172.21.15.172:6789/0 conn(0x7ff85c15b710 legacy=0x7ff85c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:47.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85a59c700 1 -- 172.21.15.158:0/565855378 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c15fc30 con 0x7ff85c15b330 2023-10-16T11:00:47.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85ad9d700 1 --2- 172.21.15.158:0/565855378 >> v2:172.21.15.172:3300/0 conn(0x7ff85c15f760 0x7ff85c0716e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:47.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85a59c700 1 --2- 172.21.15.158:0/565855378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15b330 0x7ff85c0711a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0cecc0 tx=0x7ff850009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d614c24b81703868 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85959a700 1 -- 172.21.15.158:0/565855378 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850004230 con 0x7ff85c15b330 2023-10-16T11:00:47.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85959a700 1 -- 172.21.15.158:0/565855378 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff850004390 con 0x7ff85c15b330 2023-10-16T11:00:47.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.550+0000 7ff85959a700 1 -- 172.21.15.158:0/565855378 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8500036a0 con 0x7ff85c15b330 2023-10-16T11:00:47.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 -- 172.21.15.158:0/565855378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15b330 msgr2=0x7ff85c0711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 --2- 172.21.15.158:0/565855378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15b330 0x7ff85c0711a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0cecc0 tx=0x7ff850009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 -- 172.21.15.158:0/565855378 shutdown_connections 2023-10-16T11:00:47.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 --2- 172.21.15.158:0/565855378 >> v2:172.21.15.172:3300/0 conn(0x7ff85c15f760 0x7ff85c0716e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 --2- 172.21.15.158:0/565855378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15b330 0x7ff85c0711a0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 -- 172.21.15.158:0/565855378 >> 172.21.15.158:0/565855378 conn(0x7ff85c0cfaa0 msgr2=0x7ff85c0cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 -- 172.21.15.158:0/565855378 shutdown_connections 2023-10-16T11:00:47.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.551+0000 7ff860d5c700 1 -- 172.21.15.158:0/565855378 wait complete. 2023-10-16T11:00:47.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.552+0000 7ff860d5c700 1 Processor -- start 2023-10-16T11:00:47.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.552+0000 7ff860d5c700 1 -- start start 2023-10-16T11:00:47.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff860d5c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff85c15b330 0x7ff85c0c6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff860d5c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff85c15b710 0x7ff85c06b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff860d5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15f760 0x7ff85c0c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff860d5c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0c5fa0 con 0x7ff85c15f760 2023-10-16T11:00:47.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff860d5c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0c6970 con 0x7ff85c15b710 2023-10-16T11:00:47.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff860d5c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0c5390 con 0x7ff85c15b330 2023-10-16T11:00:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15f760 0x7ff85c0c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15f760 0x7ff85c0c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33322/0 (socket says 172.21.15.158:33322) 2023-10-16T11:00:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 -- 172.21.15.158:0/2140662699 learned_addr learned my addr 172.21.15.158:0/2140662699 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff859d9b700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff85c15b710 0x7ff85c06b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85a59c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff85c15b330 0x7ff85c0c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 -- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff85c15b330 msgr2=0x7ff85c0c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff85c15b330 0x7ff85c0c6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 -- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff85c15b710 msgr2=0x7ff85c06b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff85c15b710 0x7ff85c06b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 -- 172.21.15.158:0/2140662699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850009550 con 0x7ff85c15f760 2023-10-16T11:00:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff85ad9d700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15f760 0x7ff85c0c4940 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7ff85400cd50 tx=0x7ff85400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.553+0000 7ff84affd700 1 -- 172.21.15.158:0/2140662699 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85400cea0 con 0x7ff85c15f760 2023-10-16T11:00:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.554+0000 7ff84affd700 1 -- 172.21.15.158:0/2140662699 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85400f480 con 0x7ff85c15f760 2023-10-16T11:00:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.554+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c0c5670 con 0x7ff85c15f760 2023-10-16T11:00:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.554+0000 7ff84affd700 1 -- 172.21.15.158:0/2140662699 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85400f830 con 0x7ff85c15f760 2023-10-16T11:00:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.554+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c0c5c30 con 0x7ff85c15f760 2023-10-16T11:00:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.554+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c15c6c0 con 0x7ff85c15f760 2023-10-16T11:00:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.555+0000 7ff84affd700 1 -- 172.21.15.158:0/2140662699 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff85400b040 con 0x7ff85c15f760 2023-10-16T11:00:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.555+0000 7ff84affd700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff83c06b050 0x7ff83c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.556+0000 7ff84affd700 1 -- 172.21.15.158:0/2140662699 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7ff85408a290 con 0x7ff85c15f760 2023-10-16T11:00:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.556+0000 7ff85a59c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff83c06b050 0x7ff83c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:47.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.556+0000 7ff85a59c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff83c06b050 0x7ff83c06d510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff850009130 tx=0x7ff850008030 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:47.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.560+0000 7ff84affd700 1 -- 172.21.15.158:0/2140662699 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff85c15c6c0 con 0x7ff85c15f760 2023-10-16T11:00:47.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.795+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff85c15c6c0 con 0x7ff85c15f760 2023-10-16T11:00:47.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.796+0000 7ff84affd700 1 -- 172.21.15.158:0/2140662699 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v85) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff85401d060 con 0x7ff85c15f760 2023-10-16T11:00:47.797 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T11:00:47.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff83c06b050 msgr2=0x7ff83c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff83c06b050 0x7ff83c06d510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff850009130 tx=0x7ff850008030 comp rx=0 tx=0).stop 2023-10-16T11:00:47.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15f760 msgr2=0x7ff85c0c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:47.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15f760 0x7ff85c0c4940 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7ff85400cd50 tx=0x7ff85400a560 comp rx=0 tx=0).stop 2023-10-16T11:00:47.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 shutdown_connections 2023-10-16T11:00:47.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff85c15f760 0x7ff85c0c4940 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff85c15b330 0x7ff85c0c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7ff83c06b050 0x7ff83c06d510 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 --2- 172.21.15.158:0/2140662699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff85c15b710 0x7ff85c06b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:47.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 >> 172.21.15.158:0/2140662699 conn(0x7ff85c0cfaa0 msgr2=0x7ff85c0b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:47.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 shutdown_connections 2023-10-16T11:00:47.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:47.798+0000 7ff860d5c700 1 -- 172.21.15.158:0/2140662699 wait complete. 2023-10-16T11:00:47.811 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T11:00:47.812 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T11:00:47.812 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T11:00:47.813 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T11:00:47.813 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T11:00:47.813 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T11:00:47.814 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T11:00:47.814 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T11:00:47.815 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:00:47.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:47.816 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T11:00:47.833 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T11:00:47.833 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T11:00:47.834 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T11:00:47.834 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:00:47.834 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T11:00:47.834 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:00:47.835 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:00:47.835 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:00:47.835 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:00:47.835 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:00:47.836 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:00:47.836 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T11:00:47.836 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T11:00:47.836 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T11:00:47.836 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T11:00:47.837 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 37784sec preferred_lft 37784sec 2023-10-16T11:00:47.837 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T11:00:47.837 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:00:47.837 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T11:00:47.838 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T11:00:47.838 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T11:00:47.838 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:00:47.838 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T11:00:47.839 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:00:47.839 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:00:47.839 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:00:47.839 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T11:00:47.839 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:00:47.840 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:00:47.840 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:00:47.840 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T11:00:47.840 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:00:47.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:47.841 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T11:00:47.899 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T11:00:47.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:47.901 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T11:00:47.960 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T11:00:47.960 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T11:00:47.961 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T11:00:47.961 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T11:00:47.961 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T11:00:47.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:47.962 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T11:00:47.962 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T11:00:47.963 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T11:00:47.963 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T11:00:47.963 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T11:00:47.964 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T11:00:48.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:48.066 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:00:48.082 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:00:48.082 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:00:48.083 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:00:48.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:48.083 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T11:00:48.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:00:48.149 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T11:00:48.205 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:00:48.206 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:00:48.206 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:00:48.262 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T11:00:48.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:48.263 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:00:48.348 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:00:48.348 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:00:48.372 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:00:48.373 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:00:48.373 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:00:48.373 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:00:48.373 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:00:48.373 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:00:48.374 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:00:48.374 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:00:48.448 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:00:48.449 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:00:48.449 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:00:48.449 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:00:48.449 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:00:48.449 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:00:48.450 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:00:48.450 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:00:48.450 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T11:00:48.450 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T11:00:48.450 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T11:00:48.451 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T11:00:48.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:48.452 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:00:48.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:48.479 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T11:00:48.567 INFO:teuthology.orchestra.run.smithi158.stdout:client37793 2023-10-16T11:00:48.568 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:48.568 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:48.585 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T11:00:48.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.749+0000 7f243281b700 1 Processor -- start 2023-10-16T11:00:48.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.749+0000 7f243281b700 1 -- start start 2023-10-16T11:00:48.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f243281b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f242c159a10 0x7f242c071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:48.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f243281b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f242c06ce90 0x7f242c159df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:48.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f243281b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242c0716b0 con 0x7f242c15e220 2023-10-16T11:00:48.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f243281b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f242c071990 con 0x7f242c06ce90 2023-10-16T11:00:48.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f243281b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f242c071c70 con 0x7f242c159a10 2023-10-16T11:00:48.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f2430db8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f242c06ce90 0x7f242c159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:48.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f2430db8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f242c06ce90 0x7f242c159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33334/0 (socket says 172.21.15.158:33334) 2023-10-16T11:00:48.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f2430db8700 1 -- 172.21.15.158:0/3541415872 learned_addr learned my addr 172.21.15.158:0/3541415872 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:48.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.750+0000 7f242bfff700 1 --2- 172.21.15.158:0/3541415872 >> v2:172.21.15.172:3300/0 conn(0x7f242c159a10 0x7f242c071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:48.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f2430db8700 1 -- 172.21.15.158:0/3541415872 >> v2:172.21.15.172:3300/0 conn(0x7f242c159a10 msgr2=0x7f242c071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:48.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f2430db8700 1 --2- 172.21.15.158:0/3541415872 >> v2:172.21.15.172:3300/0 conn(0x7f242c159a10 0x7f242c071010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f2430db8700 1 -- 172.21.15.158:0/3541415872 >> v1:172.21.15.172:6789/0 conn(0x7f242c15e220 legacy=0x7f242c06c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:48.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f2430db8700 1 -- 172.21.15.158:0/3541415872 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c071f50 con 0x7f242c06ce90 2023-10-16T11:00:48.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f2430db8700 1 --2- 172.21.15.158:0/3541415872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c06ce90 0x7f242c159df0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f242c06eb10 tx=0x7f242400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480f35451ca90575 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:48.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f242affd700 1 -- 172.21.15.158:0/3541415872 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242400f070 con 0x7f242c06ce90 2023-10-16T11:00:48.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f242affd700 1 -- 172.21.15.158:0/3541415872 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242400a2e0 con 0x7f242c06ce90 2023-10-16T11:00:48.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.751+0000 7f242affd700 1 -- 172.21.15.158:0/3541415872 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242400cc10 con 0x7f242c06ce90 2023-10-16T11:00:48.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 -- 172.21.15.158:0/3541415872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c06ce90 msgr2=0x7f242c159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:48.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 --2- 172.21.15.158:0/3541415872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c06ce90 0x7f242c159df0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f242c06eb10 tx=0x7f242400c330 comp rx=0 tx=0).stop 2023-10-16T11:00:48.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 -- 172.21.15.158:0/3541415872 shutdown_connections 2023-10-16T11:00:48.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 --2- 172.21.15.158:0/3541415872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c06ce90 0x7f242c159df0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 --2- 172.21.15.158:0/3541415872 >> v2:172.21.15.172:3300/0 conn(0x7f242c159a10 0x7f242c071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 -- 172.21.15.158:0/3541415872 >> 172.21.15.158:0/3541415872 conn(0x7f242c0cbd30 msgr2=0x7f242c0cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:48.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 -- 172.21.15.158:0/3541415872 shutdown_connections 2023-10-16T11:00:48.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.752+0000 7f243281b700 1 -- 172.21.15.158:0/3541415872 wait complete. 2023-10-16T11:00:48.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.753+0000 7f243281b700 1 Processor -- start 2023-10-16T11:00:48.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.753+0000 7f243281b700 1 -- start start 2023-10-16T11:00:48.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f243281b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c06ce90 0x7f242c1548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:48.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f243281b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159a10 0x7f242c0c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:48.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f243281b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c15e220 0x7f242c0b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:48.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f243281b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0b1c30 con 0x7f242c159a10 2023-10-16T11:00:48.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f243281b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0c70d0 con 0x7f242c06ce90 2023-10-16T11:00:48.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f243281b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0afe50 con 0x7f242c15e220 2023-10-16T11:00:48.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159a10 0x7f242c0c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:48.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159a10 0x7f242c0c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33338/0 (socket says 172.21.15.158:33338) 2023-10-16T11:00:48.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 -- 172.21.15.158:0/2076065360 learned_addr learned my addr 172.21.15.158:0/2076065360 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:48.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242bfff700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c06ce90 0x7f242c1548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:48.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f2430db8700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c15e220 0x7f242c0b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:48.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 -- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c15e220 msgr2=0x7f242c0b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:48.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c15e220 0x7f242c0b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 -- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c06ce90 msgr2=0x7f242c1548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:48.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c06ce90 0x7f242c1548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.754+0000 7f242b7fe700 1 -- 172.21.15.158:0/2076065360 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242400ac40 con 0x7f242c159a10 2023-10-16T11:00:48.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.755+0000 7f242b7fe700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159a10 0x7f242c0c6380 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f242000c450 tx=0x7f242000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:48.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.755+0000 7f2428ff9700 1 -- 172.21.15.158:0/2076065360 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24200052d0 con 0x7f242c159a10 2023-10-16T11:00:48.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.755+0000 7f2428ff9700 1 -- 172.21.15.158:0/2076065360 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420005430 con 0x7f242c159a10 2023-10-16T11:00:48.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.755+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c0b0130 con 0x7f242c159a10 2023-10-16T11:00:48.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.755+0000 7f2428ff9700 1 -- 172.21.15.158:0/2076065360 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24200038c0 con 0x7f242c159a10 2023-10-16T11:00:48.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.755+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c0c73b0 con 0x7f242c159a10 2023-10-16T11:00:48.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.756+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c0681b0 con 0x7f242c159a10 2023-10-16T11:00:48.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.757+0000 7f2428ff9700 1 -- 172.21.15.158:0/2076065360 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2420003a20 con 0x7f242c159a10 2023-10-16T11:00:48.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.757+0000 7f2428ff9700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f241806b020 0x7f241806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:48.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.757+0000 7f2428ff9700 1 -- 172.21.15.158:0/2076065360 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f2420088cf0 con 0x7f242c159a10 2023-10-16T11:00:48.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.757+0000 7f242bfff700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f241806b020 0x7f241806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:48.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.757+0000 7f242bfff700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f241806b020 0x7f241806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f242c0c5880 tx=0x7f241c009210 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:48.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.761+0000 7f2428ff9700 1 -- 172.21.15.158:0/2076065360 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f242008d070 con 0x7f242c159a10 2023-10-16T11:00:48.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.989+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 --> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f242c15e6b0 con 0x7f241806b020 2023-10-16T11:00:48.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.991+0000 7f2428ff9700 1 -- 172.21.15.158:0/2076065360 <== mgr.9690 v2:172.21.15.158:6832/1293733893 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f241c011860 con 0x7f241806b020 2023-10-16T11:00:48.992 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:00:48.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:00:48.993 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T11:00:48.993 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:00:48.993 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:00:48.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f241806b020 msgr2=0x7f241806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f241806b020 0x7f241806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f242c0c5880 tx=0x7f241c009210 comp rx=0 tx=0).stop 2023-10-16T11:00:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159a10 msgr2=0x7f242c0c6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159a10 0x7f242c0c6380 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f242000c450 tx=0x7f242000c760 comp rx=0 tx=0).stop 2023-10-16T11:00:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 shutdown_connections 2023-10-16T11:00:48.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f242c15e220 0x7f242c0b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f241806b020 0x7f241806d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f242c06ce90 0x7f242c1548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 --2- 172.21.15.158:0/2076065360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f242c159a10 0x7f242c0c6380 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 >> 172.21.15.158:0/2076065360 conn(0x7f242c0cbd30 msgr2=0x7f242c158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 shutdown_connections 2023-10-16T11:00:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:48.993+0000 7f243281b700 1 -- 172.21.15.158:0/2076065360 wait complete. 2023-10-16T11:00:49.008 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T11:00:49.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.136+0000 7fa8d79bf700 1 Processor -- start 2023-10-16T11:00:49.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.137+0000 7fa8d79bf700 1 -- start start 2023-10-16T11:00:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.137+0000 7fa8d79bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d79bf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa8d00b0c00 0x7fa8d00b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d79bf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d00cd3d0 con 0x7fa8d00ce3e0 2023-10-16T11:00:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d79bf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d015e560 con 0x7fa8d00b0c00 2023-10-16T11:00:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d79bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00b15d0 con 0x7fa8d0159af0 2023-10-16T11:00:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d5f5c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa8d00b0c00 0x7fa8d00b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d4f5a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa8d00ce3e0 0x7fa8d015de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:37424/0 (socket says 172.21.15.158:37424) 2023-10-16T11:00:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d5f5c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa8d00b0c00 0x7fa8d00b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:34528/0 (socket says 172.21.15.158:34528) 2023-10-16T11:00:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d4f5a700 1 -- 172.21.15.158:0/2834197231 learned_addr learned my addr 172.21.15.158:0/2834197231 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d575b700 1 --2- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d5f5c700 1 -- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 msgr2=0x7fa8d00ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d5f5c700 1 --2- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d5f5c700 1 -- 172.21.15.158:0/2834197231 >> v1:172.21.15.172:6789/0 conn(0x7fa8d00ce3e0 legacy=0x7fa8d015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:49.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8d5f5c700 1 -- 172.21.15.158:0/2834197231 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d00cdf50 con 0x7fa8d00b0c00 2023-10-16T11:00:49.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.138+0000 7fa8c7fff700 1 -- 172.21.15.158:0/2834197231 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076086096 0 0) 0x7fa8d00cd3d0 con 0x7fa8d00ce3e0 2023-10-16T11:00:49.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8d5f5c700 1 --2- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00b0c00 0x7fa8d00b1090 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800aea0 tx=0x7fa8c800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6934e16af8a49e17 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:49.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8c7fff700 1 -- 172.21.15.158:0/2834197231 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800f070 con 0x7fa8d00b0c00 2023-10-16T11:00:49.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8c7fff700 1 -- 172.21.15.158:0/2834197231 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c800a2e0 con 0x7fa8d00b0c00 2023-10-16T11:00:49.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8c7fff700 1 -- 172.21.15.158:0/2834197231 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800cb50 con 0x7fa8d00b0c00 2023-10-16T11:00:49.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8d79bf700 1 -- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00b0c00 msgr2=0x7fa8d00b1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:49.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00b0c00 0x7fa8d00b1090 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800aea0 tx=0x7fa8c800c330 comp rx=0 tx=0).stop 2023-10-16T11:00:49.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8d79bf700 1 -- 172.21.15.158:0/2834197231 shutdown_connections 2023-10-16T11:00:49.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00b0c00 0x7fa8d00b1090 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:49.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/2834197231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:49.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.139+0000 7fa8d79bf700 1 -- 172.21.15.158:0/2834197231 >> 172.21.15.158:0/2834197231 conn(0x7fa8d006af00 msgr2=0x7fa8d00cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:49.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.140+0000 7fa8d79bf700 1 -- 172.21.15.158:0/2834197231 shutdown_connections 2023-10-16T11:00:49.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.140+0000 7fa8d79bf700 1 -- 172.21.15.158:0/2834197231 wait complete. 2023-10-16T11:00:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.141+0000 7fa8d79bf700 1 Processor -- start 2023-10-16T11:00:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.141+0000 7fa8d79bf700 1 -- start start 2023-10-16T11:00:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.141+0000 7fa8d79bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00b0c00 0x7fa8d01f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.141+0000 7fa8d79bf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00ce3e0 0x7fa8d015c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d79bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d79bf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d015a310 con 0x7fa8d00b0c00 2023-10-16T11:00:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d79bf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00c5b00 con 0x7fa8d00ce3e0 2023-10-16T11:00:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d79bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d01f74d0 con 0x7fa8d0159af0 2023-10-16T11:00:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00b0c00 0x7fa8d01f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00b0c00 0x7fa8d01f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33350/0 (socket says 172.21.15.158:33350) 2023-10-16T11:00:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 -- 172.21.15.158:0/677641696 learned_addr learned my addr 172.21.15.158:0/677641696 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d4f5a700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00ce3e0 0x7fa8d015c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d5f5c700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 -- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 msgr2=0x7fa8d00c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 -- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00ce3e0 msgr2=0x7fa8d015c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00ce3e0 0x7fa8d015c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 -- 172.21.15.158:0/677641696 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c800ab80 con 0x7fa8d00b0c00 2023-10-16T11:00:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d575b700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00b0c00 0x7fa8d01f6f90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000e870 tx=0x7fa8c000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8c5ffb700 1 -- 172.21.15.158:0/677641696 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0009590 con 0x7fa8d00b0c00 2023-10-16T11:00:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8c5ffb700 1 -- 172.21.15.158:0/677641696 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0013680 con 0x7fa8d00b0c00 2023-10-16T11:00:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d015b700 con 0x7fa8d00b0c00 2023-10-16T11:00:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.142+0000 7fa8c5ffb700 1 -- 172.21.15.158:0/677641696 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00139f0 con 0x7fa8d00b0c00 2023-10-16T11:00:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.143+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d015bc80 con 0x7fa8d00b0c00 2023-10-16T11:00:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.144+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d00681b0 con 0x7fa8d00b0c00 2023-10-16T11:00:49.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.144+0000 7fa8c5ffb700 1 -- 172.21.15.158:0/677641696 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa8c0013b50 con 0x7fa8d00b0c00 2023-10-16T11:00:49.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.144+0000 7fa8c5ffb700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa8b406b020 0x7fa8b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:49.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.144+0000 7fa8c5ffb700 1 -- 172.21.15.158:0/677641696 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fa8c0089c20 con 0x7fa8d00b0c00 2023-10-16T11:00:49.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.144+0000 7fa8d4f5a700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa8b406b020 0x7fa8b406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:49.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.145+0000 7fa8d4f5a700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa8b406b020 0x7fa8b406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc007fd0 tx=0x7fa8cc007de0 comp rx=0 tx=0).ready entity=mgr.9690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:49.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.149+0000 7fa8c5ffb700 1 -- 172.21.15.158:0/677641696 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8c00548b0 con 0x7fa8d00b0c00 2023-10-16T11:00:49.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:49.417+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa8d0157480 con 0x7fa8d00b0c00 2023-10-16T11:00:50.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.378+0000 7fa8c5ffb700 1 -- 172.21.15.158:0/677641696 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v99) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa8c00170a0 con 0x7fa8d00b0c00 2023-10-16T11:00:50.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.381+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa8b406b020 msgr2=0x7fa8b406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.381+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa8b406b020 0x7fa8b406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc007fd0 tx=0x7fa8cc007de0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.381+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00b0c00 msgr2=0x7fa8d01f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.381+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00b0c00 0x7fa8d01f6f90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000e870 tx=0x7fa8c000eb80 comp rx=0 tx=0).stop 2023-10-16T11:00:50.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.381+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 shutdown_connections 2023-10-16T11:00:50.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.381+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8d0159af0 0x7fa8d00c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.382+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00b0c00 0x7fa8d01f6f90 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.382+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fa8b406b020 0x7fa8b406d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.382+0000 7fa8d79bf700 1 --2- 172.21.15.158:0/677641696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d00ce3e0 0x7fa8d015c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.382+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 >> 172.21.15.158:0/677641696 conn(0x7fa8d006af00 msgr2=0x7fa8d0157f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:50.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.383+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 shutdown_connections 2023-10-16T11:00:50.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.384+0000 7fa8d79bf700 1 -- 172.21.15.158:0/677641696 wait complete. 2023-10-16T11:00:50.399 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T11:00:50.527 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:50.526+0000 7f24a6d6b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:00:50.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.528+0000 7f103c359700 1 Processor -- start 2023-10-16T11:00:50.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.528+0000 7f103c359700 1 -- start start 2023-10-16T11:00:50.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103c359700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f103415da40 0x7f10340caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103c359700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f103406afe0 0x7f10340afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103c359700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10340cbd30 con 0x7f10340cc590 2023-10-16T11:00:50.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103c359700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10340cc1c0 con 0x7f103415da40 2023-10-16T11:00:50.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103c359700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f103415de20 con 0x7f103406afe0 2023-10-16T11:00:50.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a0f5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f103415da40 0x7f10340caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a0f5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f103415da40 0x7f10340caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33384/0 (socket says 172.21.15.158:33384) 2023-10-16T11:00:50.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a0f5700 1 -- 172.21.15.158:0/2464219081 learned_addr learned my addr 172.21.15.158:0/2464219081 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:50.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a8f6700 1 --2- 172.21.15.158:0/2464219081 >> v2:172.21.15.172:3300/0 conn(0x7f103406afe0 0x7f10340afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a0f5700 1 -- 172.21.15.158:0/2464219081 >> v2:172.21.15.172:3300/0 conn(0x7f103406afe0 msgr2=0x7f10340afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:50.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a0f5700 1 --2- 172.21.15.158:0/2464219081 >> v2:172.21.15.172:3300/0 conn(0x7f103406afe0 0x7f10340afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a0f5700 1 -- 172.21.15.158:0/2464219081 >> v1:172.21.15.172:6789/0 conn(0x7f10340cc590 legacy=0x7f103406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.529+0000 7f103a0f5700 1 -- 172.21.15.158:0/2464219081 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10340ce260 con 0x7f103415da40 2023-10-16T11:00:50.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.530+0000 7f103a0f5700 1 --2- 172.21.15.158:0/2464219081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f103415da40 0x7f10340caa20 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f10340729e0 tx=0x7f1030009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e04709a52eb835 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:50.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.530+0000 7f10390f3700 1 -- 172.21.15.158:0/2464219081 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030011830 con 0x7f103415da40 2023-10-16T11:00:50.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.530+0000 7f10390f3700 1 -- 172.21.15.158:0/2464219081 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1030011990 con 0x7f103415da40 2023-10-16T11:00:50.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.530+0000 7f10390f3700 1 -- 172.21.15.158:0/2464219081 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030011d20 con 0x7f103415da40 2023-10-16T11:00:50.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.530+0000 7f103c359700 1 -- 172.21.15.158:0/2464219081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f103415da40 msgr2=0x7f10340caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.530+0000 7f103c359700 1 --2- 172.21.15.158:0/2464219081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f103415da40 0x7f10340caa20 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f10340729e0 tx=0x7f1030009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.531+0000 7f103c359700 1 -- 172.21.15.158:0/2464219081 shutdown_connections 2023-10-16T11:00:50.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.531+0000 7f103c359700 1 --2- 172.21.15.158:0/2464219081 >> v2:172.21.15.172:3300/0 conn(0x7f103406afe0 0x7f10340afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.531+0000 7f103c359700 1 --2- 172.21.15.158:0/2464219081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f103415da40 0x7f10340caa20 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.531+0000 7f103c359700 1 -- 172.21.15.158:0/2464219081 >> 172.21.15.158:0/2464219081 conn(0x7f1034099a10 msgr2=0x7f10340d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:50.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.531+0000 7f103c359700 1 -- 172.21.15.158:0/2464219081 shutdown_connections 2023-10-16T11:00:50.536 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:50.531+0000 7fa81df3b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:00:50.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.531+0000 7f103c359700 1 -- 172.21.15.158:0/2464219081 wait complete. 2023-10-16T11:00:50.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.532+0000 7f103c359700 1 Processor -- start 2023-10-16T11:00:50.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.532+0000 7f103c359700 1 -- start start 2023-10-16T11:00:50.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103c359700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f103406afe0 0x7f103415cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103c359700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10340cc590 0x7f103409cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103c359700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f103415da40 0x7f103414e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103c359700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103409a660 con 0x7f10340cc590 2023-10-16T11:00:50.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103c359700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103409d260 con 0x7f103415da40 2023-10-16T11:00:50.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103c359700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103414e860 con 0x7f103406afe0 2023-10-16T11:00:50.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10340cc590 0x7f103409cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10340cc590 0x7f103409cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33392/0 (socket says 172.21.15.158:33392) 2023-10-16T11:00:50.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 -- 172.21.15.158:0/2366742722 learned_addr learned my addr 172.21.15.158:0/2366742722 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:50.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103a0f5700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f103406afe0 0x7f103415cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f103a8f6700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f103415da40 0x7f103414e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 -- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f103406afe0 msgr2=0x7f103415cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f103406afe0 0x7f103415cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 -- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f103415da40 msgr2=0x7f103414e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f103415da40 0x7f103414e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 -- 172.21.15.158:0/2366742722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1030009550 con 0x7f10340cc590 2023-10-16T11:00:50.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.533+0000 7f10398f4700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10340cc590 0x7f103409cd20 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f102400c4b0 tx=0x7f102400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:50.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.534+0000 7f102affd700 1 -- 172.21.15.158:0/2366742722 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10240052d0 con 0x7f10340cc590 2023-10-16T11:00:50.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.534+0000 7f102affd700 1 -- 172.21.15.158:0/2366742722 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1024005430 con 0x7f10340cc590 2023-10-16T11:00:50.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.534+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103409a8b0 con 0x7f10340cc590 2023-10-16T11:00:50.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.534+0000 7f102affd700 1 -- 172.21.15.158:0/2366742722 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10240038c0 con 0x7f10340cc590 2023-10-16T11:00:50.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.534+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103409ae70 con 0x7f10340cc590 2023-10-16T11:00:50.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.535+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10340681b0 con 0x7f10340cc590 2023-10-16T11:00:50.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.535+0000 7f102affd700 1 -- 172.21.15.158:0/2366742722 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1024003a20 con 0x7f10340cc590 2023-10-16T11:00:50.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.536+0000 7f102affd700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 0x7f101806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.536+0000 7f103a0f5700 1 -- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 msgr2=0x7f101806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:50.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.536+0000 7f102affd700 1 -- 172.21.15.158:0/2366742722 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f1024088d70 con 0x7f10340cc590 2023-10-16T11:00:50.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.536+0000 7f103a0f5700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 0x7f101806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:50.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.540+0000 7f102affd700 1 -- 172.21.15.158:0/2366742722 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1024053a90 con 0x7f10340cc590 2023-10-16T11:00:50.594 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:50.592+0000 7f24a6d6b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:50.601 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:50.601+0000 7fa81df3b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:50.666 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:50.665+0000 7f24a6d6b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:00:50.676 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:50.675+0000 7fa81df3b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:00:50.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.737+0000 7f103a0f5700 1 -- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 msgr2=0x7f101806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:50.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.737+0000 7f103a0f5700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 0x7f101806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:50.744 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:50.742+0000 7f24a6d6b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:00:50.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.757+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f103409a440 con 0x7f10340cc590 2023-10-16T11:00:50.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.760+0000 7f102affd700 1 -- 172.21.15.158:0/2366742722 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f10240170d0 con 0x7f10340cc590 2023-10-16T11:00:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 msgr2=0x7f101806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 0x7f101806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10340cc590 msgr2=0x7f103409cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10340cc590 0x7f103409cd20 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f102400c4b0 tx=0x7f102400c7c0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 shutdown_connections 2023-10-16T11:00:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f101806b070 0x7f101806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f103415da40 0x7f103414e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f103406afe0 0x7f103415cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 --2- 172.21.15.158:0/2366742722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10340cc590 0x7f103409cd20 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 >> 172.21.15.158:0/2366742722 conn(0x7f1034099a10 msgr2=0x7f103406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 shutdown_connections 2023-10-16T11:00:50.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.763+0000 7f103c359700 1 -- 172.21.15.158:0/2366742722 wait complete. 2023-10-16T11:00:50.768 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:50.767+0000 7fa81df3b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:00:50.777 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T11:00:50.884 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:50.883+0000 7f24a6d6b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:00:50.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.902+0000 7fe382c30700 1 Processor -- start 2023-10-16T11:00:50.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe382c30700 1 -- start start 2023-10-16T11:00:50.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe382c30700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c160d40 0x7fe37c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe382c30700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe37c0cf420 0x7fe37c0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe382c30700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe37c0c4ef0 0x7fe37c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe382c30700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe37c0c53a0 con 0x7fe37c0cf420 2023-10-16T11:00:50.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe382c30700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe37c0c61e0 con 0x7fe37c0c4ef0 2023-10-16T11:00:50.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe382c30700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0ce110 con 0x7fe37c160d40 2023-10-16T11:00:50.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.903+0000 7fe37bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe37c0cf420 0x7fe37c0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe37c0cf420 0x7fe37c0c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33418/0 (socket says 172.21.15.158:33418) 2023-10-16T11:00:50.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 -- 172.21.15.158:0/1880034943 learned_addr learned my addr 172.21.15.158:0/1880034943 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:50.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe3809cc700 1 --2- 172.21.15.158:0/1880034943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c160d40 0x7fe37c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe3811cd700 1 --2- 172.21.15.158:0/1880034943 >> v2:172.21.15.172:3300/0 conn(0x7fe37c0c4ef0 0x7fe37c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 -- 172.21.15.158:0/1880034943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c160d40 msgr2=0x7fe37c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:50.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 --2- 172.21.15.158:0/1880034943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c160d40 0x7fe37c0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 -- 172.21.15.158:0/1880034943 >> v2:172.21.15.172:3300/0 conn(0x7fe37c0c4ef0 msgr2=0x7fe37c06d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:50.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 --2- 172.21.15.158:0/1880034943 >> v2:172.21.15.172:3300/0 conn(0x7fe37c0c4ef0 0x7fe37c06d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 -- 172.21.15.158:0/1880034943 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37c0ce3f0 con 0x7fe37c0cf420 2023-10-16T11:00:50.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37bfff700 1 --2- 172.21.15.158:0/1880034943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c49b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0cec90 tx=0x7fe368009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2608fca59b90cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:50.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37b7fe700 1 -- 172.21.15.158:0/1880034943 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368011830 con 0x7fe37c0cf420 2023-10-16T11:00:50.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37b7fe700 1 -- 172.21.15.158:0/1880034943 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe368011990 con 0x7fe37c0cf420 2023-10-16T11:00:50.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.904+0000 7fe37b7fe700 1 -- 172.21.15.158:0/1880034943 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368011d20 con 0x7fe37c0cf420 2023-10-16T11:00:50.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 -- 172.21.15.158:0/1880034943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 msgr2=0x7fe37c0c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 --2- 172.21.15.158:0/1880034943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c49b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0cec90 tx=0x7fe368009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 -- 172.21.15.158:0/1880034943 shutdown_connections 2023-10-16T11:00:50.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 --2- 172.21.15.158:0/1880034943 >> v2:172.21.15.172:3300/0 conn(0x7fe37c0c4ef0 0x7fe37c06d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 --2- 172.21.15.158:0/1880034943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c160d40 0x7fe37c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 --2- 172.21.15.158:0/1880034943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c49b0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 -- 172.21.15.158:0/1880034943 >> 172.21.15.158:0/1880034943 conn(0x7fe37c0cc5b0 msgr2=0x7fe37c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:50.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 -- 172.21.15.158:0/1880034943 shutdown_connections 2023-10-16T11:00:50.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.905+0000 7fe382c30700 1 -- 172.21.15.158:0/1880034943 wait complete. 2023-10-16T11:00:50.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.906+0000 7fe382c30700 1 Processor -- start 2023-10-16T11:00:50.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.906+0000 7fe382c30700 1 -- start start 2023-10-16T11:00:50.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe382c30700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c0c4ef0 0x7fe37c0ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe382c30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe382c30700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe37c160d40 0x7fe37c15e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe382c30700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0c6f30 con 0x7fe37c0cf420 2023-10-16T11:00:50.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe382c30700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0c8870 con 0x7fe37c160d40 2023-10-16T11:00:50.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe382c30700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c1461d0 con 0x7fe37c0c4ef0 2023-10-16T11:00:50.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe37bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe37bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33426/0 (socket says 172.21.15.158:33426) 2023-10-16T11:00:50.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe37bfff700 1 -- 172.21.15.158:0/477070627 learned_addr learned my addr 172.21.15.158:0/477070627 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:50.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe3809cc700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c0c4ef0 0x7fe37c0ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.907+0000 7fe3811cd700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe37c160d40 0x7fe37c15e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:50.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe37bfff700 1 -- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c0c4ef0 msgr2=0x7fe37c0ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe37bfff700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c0c4ef0 0x7fe37c0ae560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe37bfff700 1 -- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe37c160d40 msgr2=0x7fe37c15e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:50.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe37bfff700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe37c160d40 0x7fe37c15e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:50.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe37bfff700 1 -- 172.21.15.158:0/477070627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe368009550 con 0x7fe37c0cf420 2023-10-16T11:00:50.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe37bfff700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c9580 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fe368009c10 tx=0x7fe368004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:50.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe3797fa700 1 -- 172.21.15.158:0/477070627 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368004240 con 0x7fe37c0cf420 2023-10-16T11:00:50.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe3797fa700 1 -- 172.21.15.158:0/477070627 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36801f070 con 0x7fe37c0cf420 2023-10-16T11:00:50.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37c0aeaa0 con 0x7fe37c0cf420 2023-10-16T11:00:50.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe3797fa700 1 -- 172.21.15.158:0/477070627 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3680036a0 con 0x7fe37c0cf420 2023-10-16T11:00:50.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.908+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37c0c8aa0 con 0x7fe37c0cf420 2023-10-16T11:00:50.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.909+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe37c066db0 con 0x7fe37c0cf420 2023-10-16T11:00:50.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.910+0000 7fe3797fa700 1 -- 172.21.15.158:0/477070627 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe368021030 con 0x7fe37c0cf420 2023-10-16T11:00:50.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.911+0000 7fe3797fa700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 0x7fe36006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:50.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.911+0000 7fe3809cc700 1 -- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 msgr2=0x7fe36006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:50.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.911+0000 7fe3797fa700 1 -- 172.21.15.158:0/477070627 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fe36808ad80 con 0x7fe37c0cf420 2023-10-16T11:00:50.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.911+0000 7fe3809cc700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 0x7fe36006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:50.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:50.915+0000 7fe3797fa700 1 -- 172.21.15.158:0/477070627 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe368055b30 con 0x7fe37c0cf420 2023-10-16T11:00:50.929 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:50.928+0000 7fa81df3b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:00:51.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.112+0000 7fe3809cc700 1 -- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 msgr2=0x7fe36006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:51.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.112+0000 7fe3809cc700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 0x7fe36006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:51.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.122+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe37c074a90 con 0x7fe37c0cf420 2023-10-16T11:00:51.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.125+0000 7fe3797fa700 1 -- 172.21.15.158:0/477070627 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe368015020 con 0x7fe37c0cf420 2023-10-16T11:00:51.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 msgr2=0x7fe36006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:51.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 0x7fe36006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 msgr2=0x7fe37c0c9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:51.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c9580 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fe368009c10 tx=0x7fe368004750 comp rx=0 tx=0).stop 2023-10-16T11:00:51.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 shutdown_connections 2023-10-16T11:00:51.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7fe36006b070 0x7fe36006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe37c160d40 0x7fe37c15e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe37c0c4ef0 0x7fe37c0ae560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 --2- 172.21.15.158:0/477070627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe37c0cf420 0x7fe37c0c9580 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 >> 172.21.15.158:0/477070627 conn(0x7fe37c0cc5b0 msgr2=0x7fe37c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:51.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 shutdown_connections 2023-10-16T11:00:51.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.127+0000 7fe382c30700 1 -- 172.21.15.158:0/477070627 wait complete. 2023-10-16T11:00:51.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:51.142 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T11:00:51.163 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2023-10-16T11:00:51.213 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:51.211+0000 7f24a6d6b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:00:51.274 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:51.272+0000 7fa81df3b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:00:51.329 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:51.327+0000 7f24a6d6b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:00:51.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.329+0000 7f56f1d42700 1 Processor -- start 2023-10-16T11:00:51.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.329+0000 7f56f1d42700 1 -- start start 2023-10-16T11:00:51.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56f1d42700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec15da40 0x7f56ec0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:51.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56f1d42700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56ec0cc590 0x7f56ec06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:51.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56f1d42700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0cbd30 con 0x7f56ec06ae10 2023-10-16T11:00:51.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56f1d42700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec06b280 con 0x7f56ec0cc590 2023-10-16T11:00:51.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56f1d42700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0cc1c0 con 0x7f56ec15da40 2023-10-16T11:00:51.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56ec0cc590 0x7f56ec06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:51.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f56ec0cc590 0x7f56ec06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33454/0 (socket says 172.21.15.158:33454) 2023-10-16T11:00:51.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 -- 172.21.15.158:0/1980357860 learned_addr learned my addr 172.21.15.158:0/1980357860 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:51.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1980357860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec15da40 0x7f56ec0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:51.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 -- 172.21.15.158:0/1980357860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec15da40 msgr2=0x7f56ec0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:51.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 --2- 172.21.15.158:0/1980357860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec15da40 0x7f56ec0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 -- 172.21.15.158:0/1980357860 >> v1:172.21.15.172:6789/0 conn(0x7f56ec06ae10 legacy=0x7f56ec0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:51.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 -- 172.21.15.158:0/1980357860 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec15de20 con 0x7f56ec0cc590 2023-10-16T11:00:51.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56eaffd700 1 --2- 172.21.15.158:0/1980357860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec0cc590 0x7f56ec06a8d0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f56e00098f0 tx=0x7f56e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7386ef6b7186f652 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:51.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.330+0000 7f56ea7fc700 1 -- 172.21.15.158:0/1980357860 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e0011830 con 0x7f56ec0cc590 2023-10-16T11:00:51.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56ea7fc700 1 -- 172.21.15.158:0/1980357860 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e0011990 con 0x7f56ec0cc590 2023-10-16T11:00:51.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56ea7fc700 1 -- 172.21.15.158:0/1980357860 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e0011d20 con 0x7f56ec0cc590 2023-10-16T11:00:51.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 -- 172.21.15.158:0/1980357860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec0cc590 msgr2=0x7f56ec06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:51.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 --2- 172.21.15.158:0/1980357860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec0cc590 0x7f56ec06a8d0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f56e00098f0 tx=0x7f56e0009c00 comp rx=0 tx=0).stop 2023-10-16T11:00:51.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 -- 172.21.15.158:0/1980357860 shutdown_connections 2023-10-16T11:00:51.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 --2- 172.21.15.158:0/1980357860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec15da40 0x7f56ec0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 --2- 172.21.15.158:0/1980357860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec0cc590 0x7f56ec06a8d0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 -- 172.21.15.158:0/1980357860 >> 172.21.15.158:0/1980357860 conn(0x7f56ec099a10 msgr2=0x7f56ec0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:51.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 -- 172.21.15.158:0/1980357860 shutdown_connections 2023-10-16T11:00:51.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.331+0000 7f56f1d42700 1 -- 172.21.15.158:0/1980357860 wait complete. 2023-10-16T11:00:51.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.332+0000 7f56f1d42700 1 Processor -- start 2023-10-16T11:00:51.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56f1d42700 1 -- start start 2023-10-16T11:00:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56f1d42700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec06ae10 0x7f56ec0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56f1d42700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56ec0cc590 0x7f56ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56f1d42700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec15da40 0x7f56ec0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56f1d42700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec15af30 con 0x7f56ec15da40 2023-10-16T11:00:51.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56f1d42700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0c8010 con 0x7f56ec0cc590 2023-10-16T11:00:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56f1d42700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0c82f0 con 0x7f56ec06ae10 2023-10-16T11:00:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.333+0000 7f56ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec15da40 0x7f56ec0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec15da40 0x7f56ec0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33462/0 (socket says 172.21.15.158:33462) 2023-10-16T11:00:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 -- 172.21.15.158:0/1143935919 learned_addr learned my addr 172.21.15.158:0/1143935919 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:51.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56eaffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56ec0cc590 0x7f56ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:51.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec06ae10 0x7f56ec0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:51.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec06ae10 msgr2=0x7f56ec0c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:51.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56ec06ae10 0x7f56ec0c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56ec0cc590 msgr2=0x7f56ec06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:51.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56ec0cc590 0x7f56ec06dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:51.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 -- 172.21.15.158:0/1143935919 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e0009550 con 0x7f56ec15da40 2023-10-16T11:00:51.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56ebfff700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec15da40 0x7f56ec0c74a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f56e400cc50 tx=0x7f56e400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:51.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4003c00 con 0x7f56ec15da40 2023-10-16T11:00:51.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e4003d60 con 0x7f56ec15da40 2023-10-16T11:00:51.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec0c85d0 con 0x7f56ec15da40 2023-10-16T11:00:51.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400aab0 con 0x7f56ec15da40 2023-10-16T11:00:51.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.334+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec0c8b90 con 0x7f56ec15da40 2023-10-16T11:00:51.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.335+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec0681b0 con 0x7f56ec15da40 2023-10-16T11:00:51.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.336+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f56e400ac10 con 0x7f56ec15da40 2023-10-16T11:00:51.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.336+0000 7f56d3fff700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 0x7f56d806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:51.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.336+0000 7f56eb7fe700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 msgr2=0x7f56d806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:51.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.336+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f56e4089aa0 con 0x7f56ec15da40 2023-10-16T11:00:51.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.336+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 0x7f56d806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:51.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.340+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56e4054730 con 0x7f56ec15da40 2023-10-16T11:00:51.398 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:51.397+0000 7fa81df3b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:00:51.403 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:51.402+0000 7f24a6d6b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:00:51.474 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:51.473+0000 7fa81df3b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:00:51.483 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:51.482+0000 7f24a6d6b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:00:51.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.537+0000 7f56eb7fe700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 msgr2=0x7f56d806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:51.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.537+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 0x7f56d806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:51.557 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:51.557+0000 7fa81df3b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:00:51.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.567+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 --> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f56ec0cb120 con 0x7f56d806b070 2023-10-16T11:00:51.852 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:51.851+0000 7f24a6d6b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:00:51.927 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:51.926+0000 7fa81df3b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:00:51.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.938+0000 7f56eb7fe700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 msgr2=0x7f56d806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:51.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:51.938+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 0x7f56d806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T11:00:52.421 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:52.420+0000 7f24a6d6b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:00:52.492 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:52.490+0000 7f24a6d6b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:00:52.497 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:52.496+0000 7fa81df3b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:00:52.568 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:52.566+0000 7fa81df3b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:00:52.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:52.739+0000 7f56eb7fe700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 msgr2=0x7f56d806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:52.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:52.739+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 0x7f56d806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T11:00:53.133 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:53.131+0000 7f24a6d6b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:53.198 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:53.197+0000 7fa81df3b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:53.210 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:53.209+0000 7f24a6d6b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:00:53.276 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:53.275+0000 7fa81df3b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:00:53.354 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:53.353+0000 7f24a6d6b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:53.422 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:53.421+0000 7fa81df3b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:00:53.583 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:53.582+0000 7f24a6d6b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:00:53.653 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:53.652+0000 7fa81df3b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:00:53.657 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:53.655+0000 7f24a6d6b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:00:53.729 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:53.727+0000 7fa81df3b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:00:53.926 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:53.925+0000 7f24a6d6b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:00:53.995 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:53.994+0000 7fa81df3b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:00:54.149 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:54.147+0000 7f24a6d6b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:00:54.219 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:54.218+0000 7fa81df3b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:00:54.298 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:54.296+0000 7f24a6d6b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:00:54.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:54.340+0000 7f56eb7fe700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 msgr2=0x7f56d806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1293733893 2023-10-16T11:00:54.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:54.340+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 0x7f56d806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T11:00:54.371 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:54.370+0000 7fa81df3b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:00:54.412 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:54.411+0000 7f24a6d6b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:00:54.486 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:54.485+0000 7fa81df3b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:00:54.569 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:54.568+0000 7f24a6d6b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:00:54.643 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:54.641+0000 7fa81df3b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:00:54.833 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:54.831+0000 7f24a6d6b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:54.906 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:54.905+0000 7fa81df3b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:54.994 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:54.993+0000 7f24a6d6b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:00:55.068 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.067+0000 7fa81df3b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:00:55.522 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:55.520+0000 7f24a6d6b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:00:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:55.591+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 100) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f56e405e050 con 0x7f56ec15da40 2023-10-16T11:00:55.602 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.601+0000 7fa81df3b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:00:55.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:55.622+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 101) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f56e4087cb0 con 0x7f56ec15da40 2023-10-16T11:00:55.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:55.622+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 msgr2=0x7f56d806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:55.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:55.622+0000 7f56d3fff700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/1293733893,v1:172.21.15.158:6833/1293733893] conn(0x7f56d806b070 0x7f56d806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:55.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa793f1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa793f1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa793f1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa793f1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa793f1c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa790f16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa790f16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa790f16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.783 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa790f16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:55.783 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:55.779+0000 7fa790f16700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:00:56.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.625+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f56e40553d0 con 0x7f56ec15da40 2023-10-16T11:00:56.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.625+0000 7f56d3fff700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f56d801d250 0x7f56d8075aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:56.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.625+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 --> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f56d8071580 con 0x7f56d801d250 2023-10-16T11:00:56.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.628+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f56d801d250 0x7f56d8075aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:56.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.630+0000 7f56eb7fe700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f56d801d250 0x7f56d8075aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009680 tx=0x7f56dc009210 comp rx=0 tx=0).ready entity=mgr.37814 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:56.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.695+0000 7f56d3fff700 1 -- 172.21.15.158:0/1143935919 <== mgr.37814 v2:172.21.15.158:6832/4231823616 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f56d8071580 con 0x7f56d801d250 2023-10-16T11:00:56.697 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1 2023-10-16T11:00:56.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f56d801d250 msgr2=0x7f56d8075aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f56d801d250 0x7f56d8075aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009680 tx=0x7f56dc009210 comp rx=0 tx=0).stop 2023-10-16T11:00:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec15da40 msgr2=0x7f56ec0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 --2- 172.21.15.158:0/1143935919 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56ec15da40 0x7f56ec0c74a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f56e400cc50 tx=0x7f56e400cf60 comp rx=0 tx=0).stop 2023-10-16T11:00:56.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56ebfff700 1 -- 172.21.15.158:0/1143935919 reap_dead start 2023-10-16T11:00:56.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 shutdown_connections 2023-10-16T11:00:56.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 >> 172.21.15.158:0/1143935919 conn(0x7f56ec099a10 msgr2=0x7f56ec0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:56.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 shutdown_connections 2023-10-16T11:00:56.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.698+0000 7f56f1d42700 1 -- 172.21.15.158:0/1143935919 wait complete. 2023-10-16T11:00:56.712 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T11:00:56.713 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T11:00:56.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.836+0000 7f398713a700 1 Processor -- start 2023-10-16T11:00:56.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.836+0000 7f398713a700 1 -- start start 2023-10-16T11:00:56.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f398713a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f39800d0c60 0x7f39800b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:56.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f398713a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f39800b3460 0x7f39800b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:56.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f398713a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800b4420 con 0x7f398015b6b0 2023-10-16T11:00:56.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f398713a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398015ba90 con 0x7f39800d0c60 2023-10-16T11:00:56.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f398713a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800b38d0 con 0x7f39800b3460 2023-10-16T11:00:56.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f397ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f39800d0c60 0x7f39800b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:56.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f397ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f39800d0c60 0x7f39800b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33624/0 (socket says 172.21.15.158:33624) 2023-10-16T11:00:56.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f397ffff700 1 -- 172.21.15.158:0/2741267000 learned_addr learned my addr 172.21.15.158:0/2741267000 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:56.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f39856d7700 1 --2- 172.21.15.158:0/2741267000 >> v2:172.21.15.172:3300/0 conn(0x7f39800b3460 0x7f39800b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:56.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f397ffff700 1 -- 172.21.15.158:0/2741267000 >> v2:172.21.15.172:3300/0 conn(0x7f39800b3460 msgr2=0x7f39800b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:56.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f397ffff700 1 --2- 172.21.15.158:0/2741267000 >> v2:172.21.15.172:3300/0 conn(0x7f39800b3460 0x7f39800b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:56.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f397ffff700 1 -- 172.21.15.158:0/2741267000 >> v1:172.21.15.172:6789/0 conn(0x7f398015b6b0 legacy=0x7f39800cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:56.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.837+0000 7f397ffff700 1 -- 172.21.15.158:0/2741267000 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800cbc20 con 0x7f39800d0c60 2023-10-16T11:00:56.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f397ffff700 1 --2- 172.21.15.158:0/2741267000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800d0c60 0x7f39800b2f20 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f39800b0cb0 tx=0x7f396c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61ecba9e721a524 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:56.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f397f7fe700 1 -- 172.21.15.158:0/2741267000 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c00b860 con 0x7f39800d0c60 2023-10-16T11:00:56.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f397f7fe700 1 -- 172.21.15.158:0/2741267000 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c00b9c0 con 0x7f39800d0c60 2023-10-16T11:00:56.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f397f7fe700 1 -- 172.21.15.158:0/2741267000 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c00bd50 con 0x7f39800d0c60 2023-10-16T11:00:56.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 -- 172.21.15.158:0/2741267000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800d0c60 msgr2=0x7f39800b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:56.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 --2- 172.21.15.158:0/2741267000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800d0c60 0x7f39800b2f20 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f39800b0cb0 tx=0x7f396c009c00 comp rx=0 tx=0).stop 2023-10-16T11:00:56.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 -- 172.21.15.158:0/2741267000 shutdown_connections 2023-10-16T11:00:56.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 --2- 172.21.15.158:0/2741267000 >> v2:172.21.15.172:3300/0 conn(0x7f39800b3460 0x7f39800b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:56.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 --2- 172.21.15.158:0/2741267000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800d0c60 0x7f39800b2f20 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:56.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 -- 172.21.15.158:0/2741267000 >> 172.21.15.158:0/2741267000 conn(0x7f39800cdf20 msgr2=0x7f39800ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:56.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 -- 172.21.15.158:0/2741267000 shutdown_connections 2023-10-16T11:00:56.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.838+0000 7f398713a700 1 -- 172.21.15.158:0/2741267000 wait complete. 2023-10-16T11:00:56.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 Processor -- start 2023-10-16T11:00:56.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 -- start start 2023-10-16T11:00:56.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39800b3460 0x7f398006c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:56.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f398015b6b0 0x7f39801583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:56.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800ca5e0 0x7f39800caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:56.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39801521f0 con 0x7f39800ca5e0 2023-10-16T11:00:56.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800cafd0 con 0x7f398015b6b0 2023-10-16T11:00:56.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.840+0000 7f398713a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39801f3d40 con 0x7f39800b3460 2023-10-16T11:00:56.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800ca5e0 0x7f39800caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:56.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800ca5e0 0x7f39800caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33632/0 (socket says 172.21.15.158:33632) 2023-10-16T11:00:56.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 -- 172.21.15.158:0/2448998005 learned_addr learned my addr 172.21.15.158:0/2448998005 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:56.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f397ffff700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f398015b6b0 0x7f39801583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:56.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f3984ed6700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39800b3460 0x7f398006c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:56.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 -- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39800b3460 msgr2=0x7f398006c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:56.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39800b3460 0x7f398006c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:56.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 -- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f398015b6b0 msgr2=0x7f39801583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:56.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f398015b6b0 0x7f39801583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:56.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 -- 172.21.15.158:0/2448998005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c009550 con 0x7f39800ca5e0 2023-10-16T11:00:56.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f397ffff700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f398015b6b0 0x7f39801583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:56.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f39856d7700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800ca5e0 0x7f39800caa90 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f3978007e90 tx=0x7f397800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:56.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f397d7fa700 1 -- 172.21.15.158:0/2448998005 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397800ede0 con 0x7f39800ca5e0 2023-10-16T11:00:56.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f397d7fa700 1 -- 172.21.15.158:0/2448998005 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3978014420 con 0x7f39800ca5e0 2023-10-16T11:00:56.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398006bb10 con 0x7f39800ca5e0 2023-10-16T11:00:56.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f397d7fa700 1 -- 172.21.15.158:0/2448998005 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39780147d0 con 0x7f39800ca5e0 2023-10-16T11:00:56.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.841+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398006c0d0 con 0x7f39800ca5e0 2023-10-16T11:00:56.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.842+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39800681b0 con 0x7f39800ca5e0 2023-10-16T11:00:56.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.842+0000 7f397d7fa700 1 -- 172.21.15.158:0/2448998005 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3978014930 con 0x7f39800ca5e0 2023-10-16T11:00:56.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.843+0000 7f397d7fa700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f396406ada0 0x7f396406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:56.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.843+0000 7f3984ed6700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f396406ada0 0x7f396406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:56.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.843+0000 7f397d7fa700 1 -- 172.21.15.158:0/2448998005 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f3978089ea0 con 0x7f39800ca5e0 2023-10-16T11:00:56.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.843+0000 7f3984ed6700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f396406ada0 0x7f396406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39800cb2c0 tx=0x7f3974009040 comp rx=0 tx=0).ready entity=mgr.37814 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:56.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:56.848+0000 7f397d7fa700 1 -- 172.21.15.158:0/2448998005 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3978054db0 con 0x7f39800ca5e0 2023-10-16T11:00:57.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.064+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 --> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f39800cc1e0 con 0x7f396406ada0 2023-10-16T11:00:57.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.129+0000 7f397d7fa700 1 -- 172.21.15.158:0/2448998005 <== mgr.37814 v2:172.21.15.158:6832/4231823616 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f39740050d0 con 0x7f396406ada0 2023-10-16T11:00:57.131 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path snap_test_dir1 2023-10-16T11:00:57.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f396406ada0 msgr2=0x7f396406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f396406ada0 0x7f396406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39800cb2c0 tx=0x7f3974009040 comp rx=0 tx=0).stop 2023-10-16T11:00:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800ca5e0 msgr2=0x7f39800caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800ca5e0 0x7f39800caa90 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f3978007e90 tx=0x7f397800e3c0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 shutdown_connections 2023-10-16T11:00:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f396406ada0 0x7f396406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39800ca5e0 0x7f39800caa90 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39800b3460 0x7f398006c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 --2- 172.21.15.158:0/2448998005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f398015b6b0 0x7f39801583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 >> 172.21.15.158:0/2448998005 conn(0x7f39800cdf20 msgr2=0x7f398015e370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 shutdown_connections 2023-10-16T11:00:57.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.132+0000 7f398713a700 1 -- 172.21.15.158:0/2448998005 wait complete. 2023-10-16T11:00:57.145 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T11:00:57.146 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T11:00:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.269+0000 7f49bb16e700 1 Processor -- start 2023-10-16T11:00:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.269+0000 7f49bb16e700 1 -- start start 2023-10-16T11:00:57.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.269+0000 7f49bb16e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49b415b590 0x7f49b40ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.269+0000 7f49bb16e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b40cf4c0 con 0x7f49b415f1d0 2023-10-16T11:00:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.269+0000 7f49bb16e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b415f970 con 0x7f49b40cbaf0 2023-10-16T11:00:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.269+0000 7f49bb16e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b40b49e0 con 0x7f49b415b590 2023-10-16T11:00:57.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b8f0a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49b415b590 0x7f49b40ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b8f0a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49b415b590 0x7f49b40ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33636/0 (socket says 172.21.15.158:33636) 2023-10-16T11:00:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b8f0a700 1 -- 172.21.15.158:0/147979528 learned_addr learned my addr 172.21.15.158:0/147979528 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:57.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b3fff700 1 --1- 172.21.15.158:0/147979528 >> v1:172.21.15.158:6789/0 conn(0x7f49b415f1d0 0x7f49b40b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33816/0 (socket says 172.21.15.158:33816) 2023-10-16T11:00:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b8f0a700 1 -- 172.21.15.158:0/147979528 >> v1:172.21.15.172:6789/0 conn(0x7f49b40cbaf0 legacy=0x7f49b40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b8f0a700 1 -- 172.21.15.158:0/147979528 >> v1:172.21.15.158:6789/0 conn(0x7f49b415f1d0 legacy=0x7f49b40b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b8f0a700 1 -- 172.21.15.158:0/147979528 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b415fc50 con 0x7f49b415b590 2023-10-16T11:00:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b8f0a700 1 --2- 172.21.15.158:0/147979528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b415b590 0x7f49b40ce510 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f49a80098f0 tx=0x7f49a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4a6182e96a0bbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:57.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b37fe700 1 -- 172.21.15.158:0/147979528 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8004230 con 0x7f49b415b590 2023-10-16T11:00:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b37fe700 1 -- 172.21.15.158:0/147979528 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a8004390 con 0x7f49b415b590 2023-10-16T11:00:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.270+0000 7f49b37fe700 1 -- 172.21.15.158:0/147979528 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a80036a0 con 0x7f49b415b590 2023-10-16T11:00:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.271+0000 7f49bb16e700 1 -- 172.21.15.158:0/147979528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b415b590 msgr2=0x7f49b40ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.271+0000 7f49bb16e700 1 --2- 172.21.15.158:0/147979528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b415b590 0x7f49b40ce510 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f49a80098f0 tx=0x7f49a8009c00 comp rx=0 tx=0).stop 2023-10-16T11:00:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.271+0000 7f49bb16e700 1 -- 172.21.15.158:0/147979528 shutdown_connections 2023-10-16T11:00:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.271+0000 7f49bb16e700 1 --2- 172.21.15.158:0/147979528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b415b590 0x7f49b40ce510 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.271+0000 7f49bb16e700 1 -- 172.21.15.158:0/147979528 >> 172.21.15.158:0/147979528 conn(0x7f49b406a8d0 msgr2=0x7f49b406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.271+0000 7f49bb16e700 1 -- 172.21.15.158:0/147979528 shutdown_connections 2023-10-16T11:00:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.271+0000 7f49bb16e700 1 -- 172.21.15.158:0/147979528 wait complete. 2023-10-16T11:00:57.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.272+0000 7f49bb16e700 1 Processor -- start 2023-10-16T11:00:57.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.272+0000 7f49bb16e700 1 -- start start 2023-10-16T11:00:57.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49bb16e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b40cbaf0 0x7f49b40cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49bb16e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b415b590 0x7f49b40c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49bb16e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b415f1d0 0x7f49b40c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49bb16e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b406ba80 con 0x7f49b40cbaf0 2023-10-16T11:00:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49bb16e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b40c7dd0 con 0x7f49b415f1d0 2023-10-16T11:00:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49bb16e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b40c7540 con 0x7f49b415b590 2023-10-16T11:00:57.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b40cbaf0 0x7f49b40cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b40cbaf0 0x7f49b40cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33642/0 (socket says 172.21.15.158:33642) 2023-10-16T11:00:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 -- 172.21.15.158:0/17763073 learned_addr learned my addr 172.21.15.158:0/17763073 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b3fff700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b415b590 0x7f49b40c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b970b700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b415f1d0 0x7f49b40c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 -- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b415b590 msgr2=0x7f49b40c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b415b590 0x7f49b40c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 -- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b415f1d0 msgr2=0x7f49b40c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b415f1d0 0x7f49b40c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 -- 172.21.15.158:0/17763073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a8009550 con 0x7f49b40cbaf0 2023-10-16T11:00:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b3fff700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49b415b590 0x7f49b40c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:57.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b970b700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49b415f1d0 0x7f49b40c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:00:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.273+0000 7f49b8f0a700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49b40cbaf0 0x7f49b40cafa0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f49a8000c00 tx=0x7f49a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.274+0000 7f49b17fa700 1 -- 172.21.15.158:0/17763073 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8003db0 con 0x7f49b40cbaf0 2023-10-16T11:00:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.274+0000 7f49b17fa700 1 -- 172.21.15.158:0/17763073 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a801f070 con 0x7f49b40cbaf0 2023-10-16T11:00:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.274+0000 7f49b17fa700 1 -- 172.21.15.158:0/17763073 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a800b8a0 con 0x7f49b40cbaf0 2023-10-16T11:00:57.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.274+0000 7f49bb16e700 1 -- 172.21.15.158:0/17763073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b40c7810 con 0x7f49b40cbaf0 2023-10-16T11:00:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.274+0000 7f49bb16e700 1 -- 172.21.15.158:0/17763073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b406bd20 con 0x7f49b40cbaf0 2023-10-16T11:00:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.274+0000 7f49bb16e700 1 -- 172.21.15.158:0/17763073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b40681b0 con 0x7f49b40cbaf0 2023-10-16T11:00:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.275+0000 7f49b17fa700 1 -- 172.21.15.158:0/17763073 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f49a8021030 con 0x7f49b40cbaf0 2023-10-16T11:00:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.275+0000 7f49b17fa700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f499806ada0 0x7f499806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.276+0000 7f49b17fa700 1 -- 172.21.15.158:0/17763073 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f49a808aea0 con 0x7f49b40cbaf0 2023-10-16T11:00:57.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.276+0000 7f49b3fff700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f499806ada0 0x7f499806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.276+0000 7f49b3fff700 1 --2- 172.21.15.158:0/17763073 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f499806ada0 0x7f499806d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49b40ca810 tx=0x7f49a0009040 comp rx=0 tx=0).ready entity=mgr.37814 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:57.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.280+0000 7f49b17fa700 1 -- 172.21.15.158:0/17763073 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49a8055e30 con 0x7f49b40cbaf0 2023-10-16T11:00:57.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.502+0000 7f49bb16e700 1 -- 172.21.15.158:0/17763073 --> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f49b415ff30 con 0x7f499806ada0 2023-10-16T11:00:57.510 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:57.509+0000 7fa79af6a700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T11:00:57.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.509+0000 7f49b17fa700 1 -- 172.21.15.158:0/17763073 <== mgr.37814 v2:172.21.15.158:6832/4231823616 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f49a00013d0 con 0x7f499806ada0 2023-10-16T11:00:57.512 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T11:00:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T11:00:57.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:00:57.517 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T11:00:57.537 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:00:57.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.706+0000 7fbec75bd700 1 Processor -- start 2023-10-16T11:00:57.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.706+0000 7fbec75bd700 1 -- start start 2023-10-16T11:00:57.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec75bd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbec00c49b0 0x7fbec00c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec75bd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbec006abb0 0x7fbec006b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec75bd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00ce110 con 0x7fbec0160d40 2023-10-16T11:00:57.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec75bd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00c53a0 con 0x7fbec00c49b0 2023-10-16T11:00:57.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec75bd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00c61e0 con 0x7fbec006abb0 2023-10-16T11:00:57.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec4b58700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbec00c49b0 0x7fbec00c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec5359700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbec0160d40 0x7fbec00ceee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33844/0 (socket says 172.21.15.158:33844) 2023-10-16T11:00:57.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec5359700 1 -- 172.21.15.158:0/2204120262 learned_addr learned my addr 172.21.15.158:0/2204120262 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:57.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec5b5a700 1 --2- 172.21.15.158:0/2204120262 >> v2:172.21.15.172:3300/0 conn(0x7fbec006abb0 0x7fbec006b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec4b58700 1 -- 172.21.15.158:0/2204120262 >> v2:172.21.15.172:3300/0 conn(0x7fbec006abb0 msgr2=0x7fbec006b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:57.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec4b58700 1 --2- 172.21.15.158:0/2204120262 >> v2:172.21.15.172:3300/0 conn(0x7fbec006abb0 0x7fbec006b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec4b58700 1 -- 172.21.15.158:0/2204120262 >> v1:172.21.15.158:6789/0 conn(0x7fbec0160d40 legacy=0x7fbec00ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec4b58700 1 -- 172.21.15.158:0/2204120262 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00ce3f0 con 0x7fbec00c49b0 2023-10-16T11:00:57.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbeb7fff700 1 -- 172.21.15.158:0/2204120262 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480135601 0 0) 0x7fbec00ce110 con 0x7fbec0160d40 2023-10-16T11:00:57.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.707+0000 7fbec4b58700 1 --2- 172.21.15.158:0/2204120262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec00c49b0 0x7fbec00c4e60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fbec0071130 tx=0x7fbeb0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197452a1e1e92497 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:57.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbeb7fff700 1 -- 172.21.15.158:0/2204120262 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0011830 con 0x7fbec00c49b0 2023-10-16T11:00:57.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbeb7fff700 1 -- 172.21.15.158:0/2204120262 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb0011990 con 0x7fbec00c49b0 2023-10-16T11:00:57.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbeb7fff700 1 -- 172.21.15.158:0/2204120262 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0011d20 con 0x7fbec00c49b0 2023-10-16T11:00:57.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 -- 172.21.15.158:0/2204120262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec00c49b0 msgr2=0x7fbec00c4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 --2- 172.21.15.158:0/2204120262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec00c49b0 0x7fbec00c4e60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fbec0071130 tx=0x7fbeb0009b20 comp rx=0 tx=0).stop 2023-10-16T11:00:57.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 -- 172.21.15.158:0/2204120262 shutdown_connections 2023-10-16T11:00:57.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 --2- 172.21.15.158:0/2204120262 >> v2:172.21.15.172:3300/0 conn(0x7fbec006abb0 0x7fbec006b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 --2- 172.21.15.158:0/2204120262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec00c49b0 0x7fbec00c4e60 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 -- 172.21.15.158:0/2204120262 >> 172.21.15.158:0/2204120262 conn(0x7fbec00cc5b0 msgr2=0x7fbec00cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:57.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 -- 172.21.15.158:0/2204120262 shutdown_connections 2023-10-16T11:00:57.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.708+0000 7fbec75bd700 1 -- 172.21.15.158:0/2204120262 wait complete. 2023-10-16T11:00:57.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.710+0000 7fbec75bd700 1 Processor -- start 2023-10-16T11:00:57.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.710+0000 7fbec75bd700 1 -- start start 2023-10-16T11:00:57.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec75bd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec006abb0 0x7fbec0156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec75bd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbec00c49b0 0x7fbec01f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec75bd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbec0160d40 0x7fbec0146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec75bd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec0156e50 con 0x7fbec006abb0 2023-10-16T11:00:57.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec75bd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec0159ac0 con 0x7fbec00c49b0 2023-10-16T11:00:57.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec75bd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec0144100 con 0x7fbec0160d40 2023-10-16T11:00:57.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec006abb0 0x7fbec0156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec006abb0 0x7fbec0156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33674/0 (socket says 172.21.15.158:33674) 2023-10-16T11:00:57.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 -- 172.21.15.158:0/839733162 learned_addr learned my addr 172.21.15.158:0/839733162 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:57.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 -- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbec0160d40 msgr2=0x7fbec0146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5b5a700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbec0160d40 0x7fbec0146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec4b58700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbec00c49b0 0x7fbec01f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbec0160d40 0x7fbec0146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 -- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbec00c49b0 msgr2=0x7fbec01f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:57.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbec00c49b0 0x7fbec01f2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:57.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 -- 172.21.15.158:0/839733162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb00094a0 con 0x7fbec006abb0 2023-10-16T11:00:57.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec5359700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec006abb0 0x7fbec0156910 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00c580 tx=0x7fbebc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:57.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbeb5ffb700 1 -- 172.21.15.158:0/839733162 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc004c80 con 0x7fbec006abb0 2023-10-16T11:00:57.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbeb5ffb700 1 -- 172.21.15.158:0/839733162 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc004e00 con 0x7fbec006abb0 2023-10-16T11:00:57.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec01f2f40 con 0x7fbec006abb0 2023-10-16T11:00:57.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.711+0000 7fbeb5ffb700 1 -- 172.21.15.158:0/839733162 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc007310 con 0x7fbec006abb0 2023-10-16T11:00:57.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.712+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec00ad6d0 con 0x7fbec006abb0 2023-10-16T11:00:57.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.713+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec0066db0 con 0x7fbec006abb0 2023-10-16T11:00:57.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.713+0000 7fbeb5ffb700 1 -- 172.21.15.158:0/839733162 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fbebc007470 con 0x7fbec006abb0 2023-10-16T11:00:57.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.713+0000 7fbeb5ffb700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fbea406ada0 0x7fbea406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:57.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.713+0000 7fbeb5ffb700 1 -- 172.21.15.158:0/839733162 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fbebc089270 con 0x7fbec006abb0 2023-10-16T11:00:57.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.713+0000 7fbec4b58700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fbea406ada0 0x7fbea406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:57.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.714+0000 7fbec4b58700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fbea406ada0 0x7fbea406d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00090b0 tx=0x7fbeb0008030 comp rx=0 tx=0).ready entity=mgr.37814 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:57.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.718+0000 7fbeb5ffb700 1 -- 172.21.15.158:0/839733162 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbebc054200 con 0x7fbec006abb0 2023-10-16T11:00:57.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:57.969+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fbec01f2f40 con 0x7fbec006abb0 2023-10-16T11:00:58.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.632+0000 7fbeb5ffb700 1 -- 172.21.15.158:0/839733162 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v103) v1 ==== 93+0+0 (secure 0 0 0) 0x7fbec01f2f40 con 0x7fbec006abb0 2023-10-16T11:00:58.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.635+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fbea406ada0 msgr2=0x7fbea406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:58.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.635+0000 7fbec75bd700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fbea406ada0 0x7fbea406d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00090b0 tx=0x7fbeb0008030 comp rx=0 tx=0).stop 2023-10-16T11:00:58.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.635+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec006abb0 msgr2=0x7fbec0156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:58.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.635+0000 7fbec75bd700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec006abb0 0x7fbec0156910 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00c580 tx=0x7fbebc00c940 comp rx=0 tx=0).stop 2023-10-16T11:00:58.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.636+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 shutdown_connections 2023-10-16T11:00:58.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.636+0000 7fbec75bd700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fbea406ada0 0x7fbea406d260 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00090b0 tx=0x7fbeb0008030 comp rx=0 tx=0).stop 2023-10-16T11:00:58.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.636+0000 7fbec75bd700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbec0160d40 0x7fbec0146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.636+0000 7fbec75bd700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbec006abb0 0x7fbec0156910 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.636+0000 7fbec75bd700 1 --2- 172.21.15.158:0/839733162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbec00c49b0 0x7fbec01f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.636+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 >> 172.21.15.158:0/839733162 conn(0x7fbec00cc5b0 msgr2=0x7fbec006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:58.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.637+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 shutdown_connections 2023-10-16T11:00:58.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.637+0000 7fbec75bd700 1 -- 172.21.15.158:0/839733162 wait complete. 2023-10-16T11:00:58.653 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:00:58.654 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:00:58.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:00:58.757 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:00:58.758 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:58.759 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:58.778 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:00:58.779 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:00:58.786 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:58.785+0000 7f231ff64200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:00:58.788 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:58.785+0000 7f8877926200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:00:58.806 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:00:58.853 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:58.852+0000 7f8877926200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:58.856 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:58.855+0000 7f231ff64200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:00:58.927 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:58.925+0000 7f8877926200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:00:58.933 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:58.932+0000 7f231ff64200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:00:58.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.974+0000 7f6ca14d4700 1 Processor -- start 2023-10-16T11:00:58.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.974+0000 7f6ca14d4700 1 -- start start 2023-10-16T11:00:58.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6ca14d4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c0ce530 0x7f6c9c0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:58.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6ca14d4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c9c15f0f0 0x7f6c9c0cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:58.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6ca14d4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c9c0cbfe0 0x7f6c9c0d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:58.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6ca14d4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9c0cc490 con 0x7f6c9c15f0f0 2023-10-16T11:00:58.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6ca14d4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9c0cf020 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6ca14d4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c15b930 con 0x7f6c9c0ce530 2023-10-16T11:00:58.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c9c15f0f0 0x7f6c9c0cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:58.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c9c15f0f0 0x7f6c9c0cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33740/0 (socket says 172.21.15.158:33740) 2023-10-16T11:00:58.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 -- 172.21.15.158:0/3616269137 learned_addr learned my addr 172.21.15.158:0/3616269137 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:58.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9b7fe700 1 --2- 172.21.15.158:0/3616269137 >> v2:172.21.15.172:3300/0 conn(0x7f6c9c0cbfe0 0x7f6c9c0d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:58.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9affd700 1 --2- 172.21.15.158:0/3616269137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c0ce530 0x7f6c9c0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:58.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 -- 172.21.15.158:0/3616269137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c0ce530 msgr2=0x7f6c9c0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:58.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 --2- 172.21.15.158:0/3616269137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c0ce530 0x7f6c9c0b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 -- 172.21.15.158:0/3616269137 >> v2:172.21.15.172:3300/0 conn(0x7f6c9c0cbfe0 msgr2=0x7f6c9c0d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:58.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 --2- 172.21.15.158:0/3616269137 >> v2:172.21.15.172:3300/0 conn(0x7f6c9c0cbfe0 0x7f6c9c0d13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.975+0000 7f6c9a7fc700 1 -- 172.21.15.158:0/3616269137 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c15f5f0 con 0x7f6c9c15f0f0 2023-10-16T11:00:58.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6c9a7fc700 1 --2- 172.21.15.158:0/3616269137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c15f0f0 0x7f6c9c0cbaa0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c1453b0 tx=0x7f6c90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3fc088c9f73503 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:58.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6c99ffb700 1 -- 172.21.15.158:0/3616269137 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90011830 con 0x7f6c9c15f0f0 2023-10-16T11:00:58.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6c99ffb700 1 -- 172.21.15.158:0/3616269137 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90011990 con 0x7f6c9c15f0f0 2023-10-16T11:00:58.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6c99ffb700 1 -- 172.21.15.158:0/3616269137 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90011d20 con 0x7f6c9c15f0f0 2023-10-16T11:00:58.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 -- 172.21.15.158:0/3616269137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c15f0f0 msgr2=0x7f6c9c0cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:58.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/3616269137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c15f0f0 0x7f6c9c0cbaa0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c1453b0 tx=0x7f6c90009bd0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 -- 172.21.15.158:0/3616269137 shutdown_connections 2023-10-16T11:00:58.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/3616269137 >> v2:172.21.15.172:3300/0 conn(0x7f6c9c0cbfe0 0x7f6c9c0d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/3616269137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c0ce530 0x7f6c9c0b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/3616269137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c15f0f0 0x7f6c9c0cbaa0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 -- 172.21.15.158:0/3616269137 >> 172.21.15.158:0/3616269137 conn(0x7f6c9c06a8d0 msgr2=0x7f6c9c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:58.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.976+0000 7f6ca14d4700 1 -- 172.21.15.158:0/3616269137 shutdown_connections 2023-10-16T11:00:58.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.977+0000 7f6ca14d4700 1 -- 172.21.15.158:0/3616269137 wait complete. 2023-10-16T11:00:58.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.978+0000 7f6ca14d4700 1 Processor -- start 2023-10-16T11:00:58.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.978+0000 7f6ca14d4700 1 -- start start 2023-10-16T11:00:58.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6ca14d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c0cbfe0 0x7f6c9c158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6ca14d4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c9c0ce530 0x7f6c9c15def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6ca14d4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c15f0f0 0x7f6c9c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6ca14d4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0c7820 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6ca14d4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c158650 con 0x7f6c9c0ce530 2023-10-16T11:00:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6ca14d4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0b5b90 con 0x7f6c9c15f0f0 2023-10-16T11:00:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c0cbfe0 0x7f6c9c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c0cbfe0 0x7f6c9c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33744/0 (socket says 172.21.15.158:33744) 2023-10-16T11:00:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 -- 172.21.15.158:0/156250751 learned_addr learned my addr 172.21.15.158:0/156250751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9a7fc700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c9c0ce530 0x7f6c9c15def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:58.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9b7fe700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c15f0f0 0x7f6c9c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 -- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c15f0f0 msgr2=0x7f6c9c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c15f0f0 0x7f6c9c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 -- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c9c0ce530 msgr2=0x7f6c9c15def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c9c0ce530 0x7f6c9c15def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 -- 172.21.15.158:0/156250751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009550 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9a7fc700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c9c0ce530 0x7f6c9c15def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c9affd700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c0cbfe0 0x7f6c9c158110 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e870 tx=0x7f6c8c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c837fe700 1 -- 172.21.15.158:0/156250751 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c009590 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c837fe700 1 -- 172.21.15.158:0/156250751 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c013680 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c0b5e30 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.979+0000 7f6c837fe700 1 -- 172.21.15.158:0/156250751 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0139f0 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.980+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c0b63f0 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.980+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c0b6a40 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.981+0000 7f6c837fe700 1 -- 172.21.15.158:0/156250751 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f6c8c013b50 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.982+0000 7f6c837fe700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 0x7f6c8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.982+0000 7f6c9a7fc700 1 -- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 msgr2=0x7f6c8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:00:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.982+0000 7f6c837fe700 1 -- 172.21.15.158:0/156250751 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f6c8c08a050 con 0x7f6c9c0cbfe0 2023-10-16T11:00:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.982+0000 7f6c9a7fc700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 0x7f6c8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:58.986+0000 7f6c837fe700 1 -- 172.21.15.158:0/156250751 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c8c054d00 con 0x7f6c9c0cbfe0 2023-10-16T11:00:59.005 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:59.004+0000 7f8877926200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:00:59.018 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:59.017+0000 7f231ff64200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:00:59.148 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:59.147+0000 7f8877926200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:00:59.164 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:59.163+0000 7f231ff64200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:00:59.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.183+0000 7f6c9a7fc700 1 -- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 msgr2=0x7f6c8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:00:59.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.183+0000 7f6c9a7fc700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 0x7f6c8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:59.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.185+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c9c0681b0 con 0x7f6c9c0cbfe0 2023-10-16T11:00:59.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.187+0000 7f6c837fe700 1 -- 172.21.15.158:0/156250751 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+17933 (secure 0 0 0) 0x7f6c8c017090 con 0x7f6c9c0cbfe0 2023-10-16T11:00:59.188 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:00:59.188 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":173,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:55.616760+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:00:38.634284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T11:00:39.812150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/982917811":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3695355538":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/2070324944":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/1259583827":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/915803440":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3743813404":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6833/1293733893":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6832/1293733893":"2023-10-17T11:00:55.616726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:00:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 msgr2=0x7f6c8806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 0x7f6c8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c0cbfe0 msgr2=0x7f6c9c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c0cbfe0 0x7f6c9c158110 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e870 tx=0x7f6c8c00eb80 comp rx=0 tx=0).stop 2023-10-16T11:00:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 shutdown_connections 2023-10-16T11:00:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6c8806b020 0x7f6c8806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c9c15f0f0 0x7f6c9c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c9c0cbfe0 0x7f6c9c158110 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 --2- 172.21.15.158:0/156250751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c9c0ce530 0x7f6c9c15def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 >> 172.21.15.158:0/156250751 conn(0x7f6c9c06a8d0 msgr2=0x7f6c9c0cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 shutdown_connections 2023-10-16T11:00:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.189+0000 7f6ca14d4700 1 -- 172.21.15.158:0/156250751 wait complete. 2023-10-16T11:00:59.203 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:00:38.634284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:00:39.812150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:00:59.204 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:00:59.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.372+0000 7fa0356a8700 1 Processor -- start 2023-10-16T11:00:59.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.372+0000 7fa0356a8700 1 -- start start 2023-10-16T11:00:59.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa0356a8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa030143630 0x7fa0300b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa0356a8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0300b3a60 0x7fa0300716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa0356a8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03015b420 con 0x7fa030071bf0 2023-10-16T11:00:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa0356a8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa030072080 con 0x7fa030143630 2023-10-16T11:00:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa0356a8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa030066cb0 con 0x7fa0300b3a60 2023-10-16T11:00:59.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa030071bf0 0x7fa0300665a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33960/0 (socket says 172.21.15.158:33960) 2023-10-16T11:00:59.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02f7fe700 1 -- 172.21.15.158:0/110988300 learned_addr learned my addr 172.21.15.158:0/110988300 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:59.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02effd700 1 --2- 172.21.15.158:0/110988300 >> v2:172.21.15.172:3300/0 conn(0x7fa030143630 0x7fa0300b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02e7fc700 1 --2- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0300b3a60 0x7fa0300716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02dffb700 1 -- 172.21.15.158:0/110988300 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523837475 0 0) 0x7fa03015b420 con 0x7fa030071bf0 2023-10-16T11:00:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02dffb700 1 -- 172.21.15.158:0/110988300 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0180035b0 con 0x7fa030071bf0 2023-10-16T11:00:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02effd700 1 -- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0300b3a60 msgr2=0x7fa0300716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02effd700 1 --2- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0300b3a60 0x7fa0300716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02effd700 1 -- 172.21.15.158:0/110988300 >> v1:172.21.15.158:6789/0 conn(0x7fa030071bf0 legacy=0x7fa0300665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02effd700 1 -- 172.21.15.158:0/110988300 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03006c150 con 0x7fa030143630 2023-10-16T11:00:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02e7fc700 1 --2- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0300b3a60 0x7fa0300716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:00:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.373+0000 7fa02effd700 1 --2- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa030143630 0x7fa0300b3fb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b920 tx=0x7fa02000bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0bf39e071e75ee2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:59.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.375+0000 7fa02dffb700 1 -- 172.21.15.158:0/110988300 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02000c9f0 con 0x7fa030143630 2023-10-16T11:00:59.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.375+0000 7fa02dffb700 1 -- 172.21.15.158:0/110988300 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02000cb50 con 0x7fa030143630 2023-10-16T11:00:59.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.375+0000 7fa02dffb700 1 -- 172.21.15.158:0/110988300 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02000f460 con 0x7fa030143630 2023-10-16T11:00:59.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 -- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa030143630 msgr2=0x7fa0300b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 --2- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa030143630 0x7fa0300b3fb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b920 tx=0x7fa02000bc30 comp rx=0 tx=0).stop 2023-10-16T11:00:59.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 -- 172.21.15.158:0/110988300 shutdown_connections 2023-10-16T11:00:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 --2- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa030143630 0x7fa0300b3fb0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 --2- 172.21.15.158:0/110988300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0300b3a60 0x7fa0300716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 -- 172.21.15.158:0/110988300 >> 172.21.15.158:0/110988300 conn(0x7fa0300b1bd0 msgr2=0x7fa0300b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 -- 172.21.15.158:0/110988300 shutdown_connections 2023-10-16T11:00:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.376+0000 7fa0356a8700 1 -- 172.21.15.158:0/110988300 wait complete. 2023-10-16T11:00:59.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.377+0000 7fa0356a8700 1 Processor -- start 2023-10-16T11:00:59.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.377+0000 7fa0356a8700 1 -- start start 2023-10-16T11:00:59.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa0356a8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa030071bf0 0x7fa030158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa0356a8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0300b3a60 0x7fa03015c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa0356a8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa030143630 0x7fa03015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa0356a8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa030158a60 con 0x7fa030071bf0 2023-10-16T11:00:59.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa0356a8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03013a540 con 0x7fa0300b3a60 2023-10-16T11:00:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa0356a8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03015a5c0 con 0x7fa030143630 2023-10-16T11:00:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa02effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa030071bf0 0x7fa030158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa02effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa030071bf0 0x7fa030158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33786/0 (socket says 172.21.15.158:33786) 2023-10-16T11:00:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa02effd700 1 -- 172.21.15.158:0/1294727558 learned_addr learned my addr 172.21.15.158:0/1294727558 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:59.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa02f7fe700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa030143630 0x7fa03015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.378+0000 7fa02e7fc700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0300b3a60 0x7fa03015c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa02effd700 1 -- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa030143630 msgr2=0x7fa03015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa02effd700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa030143630 0x7fa03015cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa02effd700 1 -- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0300b3a60 msgr2=0x7fa03015c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa02effd700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0300b3a60 0x7fa03015c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa02effd700 1 -- 172.21.15.158:0/1294727558 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02400d940 con 0x7fa030071bf0 2023-10-16T11:00:59.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa02effd700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa030071bf0 0x7fa030158520 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b870 tx=0x7fa020004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa0177fe700 1 -- 172.21.15.158:0/1294727558 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02001c070 con 0x7fa030071bf0 2023-10-16T11:00:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa0177fe700 1 -- 172.21.15.158:0/1294727558 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02000ce00 con 0x7fa030071bf0 2023-10-16T11:00:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa0177fe700 1 -- 172.21.15.158:0/1294727558 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0200038e0 con 0x7fa030071bf0 2023-10-16T11:00:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02000b550 con 0x7fa030071bf0 2023-10-16T11:00:59.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.379+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03015ab80 con 0x7fa030071bf0 2023-10-16T11:00:59.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.380+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0300681b0 con 0x7fa030071bf0 2023-10-16T11:00:59.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.380+0000 7fa0177fe700 1 -- 172.21.15.158:0/1294727558 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa020020070 con 0x7fa030071bf0 2023-10-16T11:00:59.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.381+0000 7fa0177fe700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 0x7fa01c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.381+0000 7fa02e7fc700 1 -- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 msgr2=0x7fa01c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:00:59.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.381+0000 7fa0177fe700 1 -- 172.21.15.158:0/1294727558 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fa02008a940 con 0x7fa030071bf0 2023-10-16T11:00:59.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.381+0000 7fa02e7fc700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 0x7fa01c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:59.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.386+0000 7fa0177fe700 1 -- 172.21.15.158:0/1294727558 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa020055400 con 0x7fa030071bf0 2023-10-16T11:00:59.476 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:59.475+0000 7f8877926200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:00:59.509 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:59.507+0000 7f231ff64200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:00:59.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.582+0000 7fa02e7fc700 1 -- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 msgr2=0x7fa01c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:00:59.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.582+0000 7fa02e7fc700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 0x7fa01c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:59.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.584+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa030066920 con 0x7fa030071bf0 2023-10-16T11:00:59.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.585+0000 7fa0177fe700 1 -- 172.21.15.158:0/1294727558 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa020058a20 con 0x7fa030071bf0 2023-10-16T11:00:59.587 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:00:59.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 msgr2=0x7fa01c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:59.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 0x7fa01c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa030071bf0 msgr2=0x7fa030158520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa030071bf0 0x7fa030158520 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b870 tx=0x7fa020004750 comp rx=0 tx=0).stop 2023-10-16T11:00:59.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 shutdown_connections 2023-10-16T11:00:59.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fa01c06b020 0x7fa01c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa030143630 0x7fa03015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa030071bf0 0x7fa030158520 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 --2- 172.21.15.158:0/1294727558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0300b3a60 0x7fa03015c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.587+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 >> 172.21.15.158:0/1294727558 conn(0x7fa0300b1bd0 msgr2=0x7fa03006afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:59.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.588+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 shutdown_connections 2023-10-16T11:00:59.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.588+0000 7fa0356a8700 1 -- 172.21.15.158:0/1294727558 wait complete. 2023-10-16T11:00:59.593 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:59.591+0000 7f8877926200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:00:59.603 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:00:59.630 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:59.629+0000 7f231ff64200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:00:59.666 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:59.665+0000 7f8877926200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:00:59.709 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:59.708+0000 7f231ff64200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:00:59.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.728+0000 7f85aaf85700 1 Processor -- start 2023-10-16T11:00:59.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.728+0000 7f85aaf85700 1 -- start start 2023-10-16T11:00:59.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85aaf85700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85a415daa0 0x7f85a40cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85aaf85700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f85a40d0640 0x7f85a40cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85aaf85700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a40d1ee0 con 0x7f85a415d6c0 2023-10-16T11:00:59.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85aaf85700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a40d40a0 con 0x7f85a415daa0 2023-10-16T11:00:59.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85aaf85700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a40d12d0 con 0x7f85a40d0640 2023-10-16T11:00:59.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85a415daa0 0x7f85a40cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a8d21700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f85a415d6c0 0x7f85a40cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:33970/0 (socket says 172.21.15.158:33970) 2023-10-16T11:00:59.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85a415daa0 0x7f85a40cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33796/0 (socket says 172.21.15.158:33796) 2023-10-16T11:00:59.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a8d21700 1 -- 172.21.15.158:0/50043576 learned_addr learned my addr 172.21.15.158:0/50043576 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:00:59.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a9522700 1 --2- 172.21.15.158:0/50043576 >> v2:172.21.15.172:3300/0 conn(0x7f85a40d0640 0x7f85a40cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a3fff700 1 -- 172.21.15.158:0/50043576 >> v2:172.21.15.172:3300/0 conn(0x7f85a40d0640 msgr2=0x7f85a40cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:00:59.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a3fff700 1 --2- 172.21.15.158:0/50043576 >> v2:172.21.15.172:3300/0 conn(0x7f85a40d0640 0x7f85a40cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a3fff700 1 -- 172.21.15.158:0/50043576 >> v1:172.21.15.158:6789/0 conn(0x7f85a415d6c0 legacy=0x7f85a40cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a3fff700 1 -- 172.21.15.158:0/50043576 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a40d15b0 con 0x7f85a415daa0 2023-10-16T11:00:59.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.729+0000 7f85a37fe700 1 -- 172.21.15.158:0/50043576 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006860937 0 0) 0x7f85a40d1ee0 con 0x7f85a415d6c0 2023-10-16T11:00:59.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85a9522700 1 --2- 172.21.15.158:0/50043576 >> v2:172.21.15.172:3300/0 conn(0x7f85a40d0640 0x7f85a40cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:00:59.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85a3fff700 1 --2- 172.21.15.158:0/50043576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a415daa0 0x7f85a40cec50 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f85a40b5050 tx=0x7f8590009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcec1fd0b0c5033 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:59.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85a37fe700 1 -- 172.21.15.158:0/50043576 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590011830 con 0x7f85a415daa0 2023-10-16T11:00:59.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85a37fe700 1 -- 172.21.15.158:0/50043576 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8590011990 con 0x7f85a415daa0 2023-10-16T11:00:59.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85a37fe700 1 -- 172.21.15.158:0/50043576 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590011d20 con 0x7f85a415daa0 2023-10-16T11:00:59.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85aaf85700 1 -- 172.21.15.158:0/50043576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a415daa0 msgr2=0x7f85a40cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85aaf85700 1 --2- 172.21.15.158:0/50043576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a415daa0 0x7f85a40cec50 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f85a40b5050 tx=0x7f8590009b20 comp rx=0 tx=0).stop 2023-10-16T11:00:59.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85aaf85700 1 -- 172.21.15.158:0/50043576 shutdown_connections 2023-10-16T11:00:59.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85aaf85700 1 --2- 172.21.15.158:0/50043576 >> v2:172.21.15.172:3300/0 conn(0x7f85a40d0640 0x7f85a40cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85aaf85700 1 --2- 172.21.15.158:0/50043576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a415daa0 0x7f85a40cec50 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85aaf85700 1 -- 172.21.15.158:0/50043576 >> 172.21.15.158:0/50043576 conn(0x7f85a409c940 msgr2=0x7f85a40d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:59.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.730+0000 7f85aaf85700 1 -- 172.21.15.158:0/50043576 shutdown_connections 2023-10-16T11:00:59.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.731+0000 7f85aaf85700 1 -- 172.21.15.158:0/50043576 wait complete. 2023-10-16T11:00:59.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.732+0000 7f85aaf85700 1 Processor -- start 2023-10-16T11:00:59.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.732+0000 7f85aaf85700 1 -- start start 2023-10-16T11:00:59.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40d0640 0x7f85a415bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415d6c0 0x7f85a40b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a415daa0 0x7f85a40b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a40b67b0 con 0x7f85a40d0640 2023-10-16T11:00:59.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a415c070 con 0x7f85a415d6c0 2023-10-16T11:00:59.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a415eda0 con 0x7f85a415daa0 2023-10-16T11:00:59.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40d0640 0x7f85a415bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40d0640 0x7f85a415bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33800/0 (socket says 172.21.15.158:33800) 2023-10-16T11:00:59.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 -- 172.21.15.158:0/3841412572 learned_addr learned my addr 172.21.15.158:0/3841412572 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:00:59.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a3fff700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415d6c0 0x7f85a40b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a9522700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a415daa0 0x7f85a40b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:00:59.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 -- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a415daa0 msgr2=0x7f85a40b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a415daa0 0x7f85a40b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 -- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415d6c0 msgr2=0x7f85a40b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415d6c0 0x7f85a40b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 -- 172.21.15.158:0/3841412572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85900094a0 con 0x7f85a40d0640 2023-10-16T11:00:59.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a8d21700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40d0640 0x7f85a415bb30 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f85980011e0 tx=0x7f859800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:00:59.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a17fa700 1 -- 172.21.15.158:0/3841412572 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598004c80 con 0x7f85a40d0640 2023-10-16T11:00:59.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a17fa700 1 -- 172.21.15.158:0/3841412572 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8598004e00 con 0x7f85a40d0640 2023-10-16T11:00:59.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a415e220 con 0x7f85a40d0640 2023-10-16T11:00:59.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85a17fa700 1 -- 172.21.15.158:0/3841412572 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598007310 con 0x7f85a40d0640 2023-10-16T11:00:59.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.733+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a415e7e0 con 0x7f85a40d0640 2023-10-16T11:00:59.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.735+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a40d2350 con 0x7f85a40d0640 2023-10-16T11:00:59.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.735+0000 7f85a17fa700 1 -- 172.21.15.158:0/3841412572 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8598007470 con 0x7f85a40d0640 2023-10-16T11:00:59.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.735+0000 7f85a17fa700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:00:59.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.735+0000 7f85a3fff700 1 -- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 msgr2=0x7f858806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:00:59.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.735+0000 7f85a17fa700 1 -- 172.21.15.158:0/3841412572 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f8598089820 con 0x7f85a40d0640 2023-10-16T11:00:59.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.735+0000 7f85a3fff700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:00:59.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.740+0000 7f85a17fa700 1 -- 172.21.15.158:0/3841412572 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85a40d2350 con 0x7f85a40d0640 2023-10-16T11:00:59.748 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:00:59.747+0000 7f8877926200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:00:59.795 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:00:59.794+0000 7f231ff64200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:00:59.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.936+0000 7f85a3fff700 1 -- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 msgr2=0x7f858806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:00:59.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.936+0000 7f85a3fff700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:00:59.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.937+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85a40d2350 con 0x7f85a40d0640 2023-10-16T11:00:59.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.938+0000 7f85a17fa700 1 -- 172.21.15.158:0/3841412572 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v173) v1 ==== 104+0+11 (secure 0 0 0) 0x7f85a40d2350 con 0x7f85a40d0640 2023-10-16T11:00:59.939 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:00:59.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 msgr2=0x7f858806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:00:59.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40d0640 msgr2=0x7f85a415bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:00:59.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40d0640 0x7f85a415bb30 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f85980011e0 tx=0x7f859800ca30 comp rx=0 tx=0).stop 2023-10-16T11:00:59.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 shutdown_connections 2023-10-16T11:00:59.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f858806b020 0x7f858806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85a415daa0 0x7f85a40b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a40d0640 0x7f85a415bb30 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 --2- 172.21.15.158:0/3841412572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85a415d6c0 0x7f85a40b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:00:59.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 >> 172.21.15.158:0/3841412572 conn(0x7f85a409c940 msgr2=0x7f85a406c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:00:59.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 shutdown_connections 2023-10-16T11:00:59.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:00:59.940+0000 7f85aaf85700 1 -- 172.21.15.158:0/3841412572 wait complete. 2023-10-16T11:00:59.959 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:01:00.121 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:00.120+0000 7f8877926200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:01:00.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.125+0000 7efe96f4b700 1 Processor -- start 2023-10-16T11:01:00.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.125+0000 7efe96f4b700 1 -- start start 2023-10-16T11:01:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe96f4b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe900cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe96f4b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efe900b3630 0x7efe900b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe96f4b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe900b4420 con 0x7efe900d0c60 2023-10-16T11:01:00.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe96f4b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9015ba90 con 0x7efe900b3630 2023-10-16T11:01:00.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe96f4b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe900cbc20 con 0x7efe9015b6b0 2023-10-16T11:01:00.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe94ce7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe900cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe954e8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efe900b3630 0x7efe900b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe8ffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7efe900d0c60 0x7efe900b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:37920/0 (socket says 172.21.15.158:37920) 2023-10-16T11:01:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe8ffff700 1 -- 172.21.15.158:0/51683300 learned_addr learned my addr 172.21.15.158:0/51683300 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe954e8700 1 -- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 msgr2=0x7efe900cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:00.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe954e8700 1 --2- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe900cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe954e8700 1 -- 172.21.15.158:0/51683300 >> v1:172.21.15.172:6789/0 conn(0x7efe900d0c60 legacy=0x7efe900b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe954e8700 1 -- 172.21.15.158:0/51683300 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe900cbf00 con 0x7efe900b3630 2023-10-16T11:01:00.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe94ce7700 1 --2- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe900cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:00.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.127+0000 7efe8f7fe700 1 -- 172.21.15.158:0/51683300 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662383355 0 0) 0x7efe900b4420 con 0x7efe900d0c60 2023-10-16T11:01:00.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe954e8700 1 --2- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900b3630 0x7efe900b48b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efe88009d90 tx=0x7efe8800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51996ab817f65b7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:00.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe8f7fe700 1 -- 172.21.15.158:0/51683300 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8800e070 con 0x7efe900b3630 2023-10-16T11:01:00.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe8f7fe700 1 -- 172.21.15.158:0/51683300 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8800bb20 con 0x7efe900b3630 2023-10-16T11:01:00.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe8f7fe700 1 -- 172.21.15.158:0/51683300 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88012430 con 0x7efe900b3630 2023-10-16T11:01:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 -- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900b3630 msgr2=0x7efe900b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 --2- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900b3630 0x7efe900b48b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efe88009d90 tx=0x7efe8800b330 comp rx=0 tx=0).stop 2023-10-16T11:01:00.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 -- 172.21.15.158:0/51683300 shutdown_connections 2023-10-16T11:01:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 --2- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900b3630 0x7efe900b48b0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 --2- 172.21.15.158:0/51683300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe900cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 -- 172.21.15.158:0/51683300 >> 172.21.15.158:0/51683300 conn(0x7efe900cdf20 msgr2=0x7efe900ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 -- 172.21.15.158:0/51683300 shutdown_connections 2023-10-16T11:01:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.128+0000 7efe96f4b700 1 -- 172.21.15.158:0/51683300 wait complete. 2023-10-16T11:01:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.130+0000 7efe96f4b700 1 Processor -- start 2023-10-16T11:01:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.130+0000 7efe96f4b700 1 -- start start 2023-10-16T11:01:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe96f4b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe900b3630 0x7efe9006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe96f4b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900d0c60 0x7efe90151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe96f4b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe90152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe96f4b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe901f3ef0 con 0x7efe900b3630 2023-10-16T11:01:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe96f4b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90152a00 con 0x7efe900d0c60 2023-10-16T11:01:00.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe96f4b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe90151860 con 0x7efe9015b6b0 2023-10-16T11:01:00.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe900b3630 0x7efe9006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe900b3630 0x7efe9006d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33844/0 (socket says 172.21.15.158:33844) 2023-10-16T11:01:00.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 -- 172.21.15.158:0/2083334137 learned_addr learned my addr 172.21.15.158:0/2083334137 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:00.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe8ffff700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900d0c60 0x7efe90151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe954e8700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe90152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 -- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 msgr2=0x7efe90152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe90152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 -- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900d0c60 msgr2=0x7efe90151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900d0c60 0x7efe90151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 -- 172.21.15.158:0/2083334137 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe840094a0 con 0x7efe900b3630 2023-10-16T11:01:00.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe94ce7700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe900b3630 0x7efe9006d510 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7efe8400ebe0 tx=0x7efe8400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.132+0000 7efe8d7fa700 1 -- 172.21.15.158:0/2083334137 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8400c850 con 0x7efe900b3630 2023-10-16T11:01:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.133+0000 7efe8d7fa700 1 -- 172.21.15.158:0/2083334137 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8400c9b0 con 0x7efe900b3630 2023-10-16T11:01:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.133+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe88009a70 con 0x7efe900b3630 2023-10-16T11:01:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.133+0000 7efe8d7fa700 1 -- 172.21.15.158:0/2083334137 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8400cd40 con 0x7efe900b3630 2023-10-16T11:01:00.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.133+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe901f3130 con 0x7efe900b3630 2023-10-16T11:01:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.133+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe901f3d40 con 0x7efe900b3630 2023-10-16T11:01:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.134+0000 7efe8d7fa700 1 -- 172.21.15.158:0/2083334137 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7efe84013470 con 0x7efe900b3630 2023-10-16T11:01:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.135+0000 7efe8d7fa700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 0x7efe7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.135+0000 7efe8ffff700 1 -- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 msgr2=0x7efe7406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.135+0000 7efe8d7fa700 1 -- 172.21.15.158:0/2083334137 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7efe84089d20 con 0x7efe900b3630 2023-10-16T11:01:00.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.135+0000 7efe8ffff700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 0x7efe7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:00.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.139+0000 7efe8d7fa700 1 -- 172.21.15.158:0/2083334137 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe840547c0 con 0x7efe900b3630 2023-10-16T11:01:00.196 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:00.194+0000 7f231ff64200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:01:00.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.336+0000 7efe8ffff700 1 -- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 msgr2=0x7efe7406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:00.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.336+0000 7efe8ffff700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 0x7efe7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:00.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.337+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efe900681b0 con 0x7efe900b3630 2023-10-16T11:01:00.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.338+0000 7efe8d7fa700 1 -- 172.21.15.158:0/2083334137 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v173) v1 ==== 100+0+11 (secure 0 0 0) 0x7efe840210a0 con 0x7efe900b3630 2023-10-16T11:01:00.339 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:01:00.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.340+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 msgr2=0x7efe7406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:00.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.340+0000 7efe96f4b700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 0x7efe7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.340+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe900b3630 msgr2=0x7efe9006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.340+0000 7efe96f4b700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe900b3630 0x7efe9006d510 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7efe8400ebe0 tx=0x7efe8400eef0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 shutdown_connections 2023-10-16T11:01:00.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7efe7406b020 0x7efe7406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe9015b6b0 0x7efe90152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe900b3630 0x7efe9006d510 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 --2- 172.21.15.158:0/2083334137 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe900d0c60 0x7efe90151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 >> 172.21.15.158:0/2083334137 conn(0x7efe900cdf20 msgr2=0x7efe900d1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:00.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 shutdown_connections 2023-10-16T11:01:00.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.341+0000 7efe96f4b700 1 -- 172.21.15.158:0/2083334137 wait complete. 2023-10-16T11:01:00.355 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:01:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.478+0000 7fcd2d46e700 1 Processor -- start 2023-10-16T11:01:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.478+0000 7fcd2d46e700 1 -- start start 2023-10-16T11:01:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd2d46e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd280cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd2d46e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcd280b5870 0x7fcd280d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd2d46e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd280d1db0 con 0x7fcd280ce5b0 2023-10-16T11:01:00.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd2d46e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd280cfd60 con 0x7fcd280b5870 2023-10-16T11:01:00.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd2d46e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd280d1760 con 0x7fcd2815da60 2023-10-16T11:01:00.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd277fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcd280b5870 0x7fcd280d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd277fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcd280b5870 0x7fcd280d2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33858/0 (socket says 172.21.15.158:33858) 2023-10-16T11:01:00.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd277fe700 1 -- 172.21.15.158:0/3923817326 learned_addr learned my addr 172.21.15.158:0/3923817326 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:00.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.479+0000 7fcd26ffd700 1 --2- 172.21.15.158:0/3923817326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd280cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd25ffb700 1 -- 172.21.15.158:0/3923817326 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964349894 0 0) 0x7fcd280d1db0 con 0x7fcd280ce5b0 2023-10-16T11:01:00.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd277fe700 1 -- 172.21.15.158:0/3923817326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 msgr2=0x7fcd280cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:00.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd277fe700 1 --2- 172.21.15.158:0/3923817326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd280cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd277fe700 1 -- 172.21.15.158:0/3923817326 >> v1:172.21.15.172:6789/0 conn(0x7fcd280ce5b0 legacy=0x7fcd280b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd277fe700 1 -- 172.21.15.158:0/3923817326 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd280d1a40 con 0x7fcd280b5870 2023-10-16T11:01:00.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd26ffd700 1 --2- 172.21.15.158:0/3923817326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd280cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:00.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd277fe700 1 --2- 172.21.15.158:0/3923817326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd280d2240 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009e50 tx=0x7fcd2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80aa97a77cec9acb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:00.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd25ffb700 1 -- 172.21.15.158:0/3923817326 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2000e070 con 0x7fcd280b5870 2023-10-16T11:01:00.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd25ffb700 1 -- 172.21.15.158:0/3923817326 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2000b910 con 0x7fcd280b5870 2023-10-16T11:01:00.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.480+0000 7fcd25ffb700 1 -- 172.21.15.158:0/3923817326 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2000bcc0 con 0x7fcd280b5870 2023-10-16T11:01:00.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 -- 172.21.15.158:0/3923817326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 msgr2=0x7fcd280d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/3923817326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd280d2240 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009e50 tx=0x7fcd2000b330 comp rx=0 tx=0).stop 2023-10-16T11:01:00.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 -- 172.21.15.158:0/3923817326 shutdown_connections 2023-10-16T11:01:00.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/3923817326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd280d2240 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/3923817326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd280cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 -- 172.21.15.158:0/3923817326 >> 172.21.15.158:0/3923817326 conn(0x7fcd2809c9b0 msgr2=0x7fcd280d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:00.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 -- 172.21.15.158:0/3923817326 shutdown_connections 2023-10-16T11:01:00.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.481+0000 7fcd2d46e700 1 -- 172.21.15.158:0/3923817326 wait complete. 2023-10-16T11:01:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.482+0000 7fcd2d46e700 1 Processor -- start 2023-10-16T11:01:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.482+0000 7fcd2d46e700 1 -- start start 2023-10-16T11:01:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd2d46e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd2815c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd2d46e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd280ce5b0 0x7fcd280b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd2d46e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd281e63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd2d46e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2815d1a0 con 0x7fcd280b5870 2023-10-16T11:01:00.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd2d46e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd280b7500 con 0x7fcd280ce5b0 2023-10-16T11:01:00.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd2d46e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2815c790 con 0x7fcd2815da60 2023-10-16T11:01:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd2815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd2815c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33860/0 (socket says 172.21.15.158:33860) 2023-10-16T11:01:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 -- 172.21.15.158:0/1625642058 learned_addr learned my addr 172.21.15.158:0/1625642058 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd267fc700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd280ce5b0 0x7fcd280b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd277fe700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd281e63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 -- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 msgr2=0x7fcd281e63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd281e63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 -- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd280ce5b0 msgr2=0x7fcd280b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd280ce5b0 0x7fcd280b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.483+0000 7fcd26ffd700 1 -- 172.21.15.158:0/1625642058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18009430 con 0x7fcd280b5870 2023-10-16T11:01:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.484+0000 7fcd26ffd700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd2815c250 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800ebe0 tx=0x7fcd1800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.484+0000 7fcd0f7fe700 1 -- 172.21.15.158:0/1625642058 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18005a20 con 0x7fcd280b5870 2023-10-16T11:01:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.484+0000 7fcd0f7fe700 1 -- 172.21.15.158:0/1625642058 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd18005b80 con 0x7fcd280b5870 2023-10-16T11:01:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.484+0000 7fcd0f7fe700 1 -- 172.21.15.158:0/1625642058 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1800c850 con 0x7fcd280b5870 2023-10-16T11:01:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.484+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd20009b30 con 0x7fcd280b5870 2023-10-16T11:01:00.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.484+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd280b6eb0 con 0x7fcd280b5870 2023-10-16T11:01:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.485+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd280b6d10 con 0x7fcd280b5870 2023-10-16T11:01:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.486+0000 7fcd0f7fe700 1 -- 172.21.15.158:0/1625642058 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fcd1800c9b0 con 0x7fcd280b5870 2023-10-16T11:01:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.486+0000 7fcd0f7fe700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 0x7fcd1406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.486+0000 7fcd267fc700 1 -- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 msgr2=0x7fcd1406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:00.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.486+0000 7fcd0f7fe700 1 -- 172.21.15.158:0/1625642058 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fcd18089e10 con 0x7fcd280b5870 2023-10-16T11:01:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.486+0000 7fcd267fc700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 0x7fcd1406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.491+0000 7fcd0f7fe700 1 -- 172.21.15.158:0/1625642058 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd280b6d10 con 0x7fcd280b5870 2023-10-16T11:01:00.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.687+0000 7fcd267fc700 1 -- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 msgr2=0x7fcd1406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:00.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.687+0000 7fcd267fc700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 0x7fcd1406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:00.698 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:00.698+0000 7f8877926200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:01:00.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.756+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcd280b6d10 con 0x7fcd280b5870 2023-10-16T11:01:00.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.756+0000 7fcd0f7fe700 1 -- 172.21.15.158:0/1625642058 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v173) v1 ==== 80+0+476 (secure 0 0 0) 0x7fcd280b6d10 con 0x7fcd280b5870 2023-10-16T11:01:00.758 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/982917811 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.758 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3695355538 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.759 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2070324944 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.759 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1259583827 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.759 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/915803440 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.759 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3743813404 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.760 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/1293733893 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.760 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/1293733893 2023-10-17T11:00:55.616726+0000 2023-10-16T11:01:00.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 msgr2=0x7fcd1406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:00.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 0x7fcd1406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 msgr2=0x7fcd2815c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:00.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd2815c250 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800ebe0 tx=0x7fcd1800eef0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 shutdown_connections 2023-10-16T11:01:00.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7fcd1406b020 0x7fcd1406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd2815da60 0x7fcd281e63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd280b5870 0x7fcd2815c250 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 --2- 172.21.15.158:0/1625642058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd280ce5b0 0x7fcd280b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:00.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 >> 172.21.15.158:0/1625642058 conn(0x7fcd2809c9b0 msgr2=0x7fcd280b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:00.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 shutdown_connections 2023-10-16T11:01:00.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:00.759+0000 7fcd2d46e700 1 -- 172.21.15.158:0/1625642058 wait complete. 2023-10-16T11:01:00.764 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T11:01:00.768 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:00.768+0000 7f8877926200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:01:00.774 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:01:00.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:00.775 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:01:00.814 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:00.813+0000 7f231ff64200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:01:00.816 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:01:00.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:00.817 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:01:00.873 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:01:00.874 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:00.889 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:00.888+0000 7f231ff64200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:01:00.892 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:01:00.892 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:01:00.893 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:00.909 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:01:00.910 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:01:00.910 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:00.949 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:01:00.950 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:01:01.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.072+0000 7f8b73a98700 1 Processor -- start 2023-10-16T11:01:01.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.072+0000 7f8b73a98700 1 -- start start 2023-10-16T11:01:01.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b73a98700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b6c15b4a0 0x7f8b6c0cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b73a98700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b6c0b1310 0x7f8b6c0c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b73a98700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c15fab0 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b73a98700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6c15efa0 con 0x7f8b6c0b1310 2023-10-16T11:01:01.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b73a98700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6c15b880 con 0x7f8b6c15b4a0 2023-10-16T11:01:01.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b72035700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b6c0b1310 0x7f8b6c0c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b71033700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8b6c0cb2c0 0x7f8b6c0b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34046/0 (socket says 172.21.15.158:34046) 2023-10-16T11:01:01.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b72035700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b6c0b1310 0x7f8b6c0c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33872/0 (socket says 172.21.15.158:33872) 2023-10-16T11:01:01.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b71033700 1 -- 172.21.15.158:0/2547130851 learned_addr learned my addr 172.21.15.158:0/2547130851 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:01.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b71834700 1 --2- 172.21.15.158:0/2547130851 >> v2:172.21.15.172:3300/0 conn(0x7f8b6c15b4a0 0x7f8b6c0cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b72035700 1 -- 172.21.15.158:0/2547130851 >> v2:172.21.15.172:3300/0 conn(0x7f8b6c15b4a0 msgr2=0x7f8b6c0cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:01.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b72035700 1 --2- 172.21.15.158:0/2547130851 >> v2:172.21.15.172:3300/0 conn(0x7f8b6c15b4a0 0x7f8b6c0cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b72035700 1 -- 172.21.15.158:0/2547130851 >> v1:172.21.15.158:6789/0 conn(0x7f8b6c0cb2c0 legacy=0x7f8b6c0b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b72035700 1 -- 172.21.15.158:0/2547130851 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c15f180 con 0x7f8b6c0b1310 2023-10-16T11:01:01.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.073+0000 7f8b70832700 1 -- 172.21.15.158:0/2547130851 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856903996 0 0) 0x7f8b6c15fab0 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b71834700 1 --2- 172.21.15.158:0/2547130851 >> v2:172.21.15.172:3300/0 conn(0x7f8b6c15b4a0 0x7f8b6c0cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:01.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b72035700 1 --2- 172.21.15.158:0/2547130851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0b1310 0x7f8b6c0c9950 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c06ecd0 tx=0x7f8b6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee73982e147ecd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:01.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b70832700 1 -- 172.21.15.158:0/2547130851 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6400e070 con 0x7f8b6c0b1310 2023-10-16T11:01:01.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b70832700 1 -- 172.21.15.158:0/2547130851 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6400bbd0 con 0x7f8b6c0b1310 2023-10-16T11:01:01.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b70832700 1 -- 172.21.15.158:0/2547130851 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64012430 con 0x7f8b6c0b1310 2023-10-16T11:01:01.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b73a98700 1 -- 172.21.15.158:0/2547130851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0b1310 msgr2=0x7f8b6c0c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2547130851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0b1310 0x7f8b6c0c9950 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c06ecd0 tx=0x7f8b6400b330 comp rx=0 tx=0).stop 2023-10-16T11:01:01.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b73a98700 1 -- 172.21.15.158:0/2547130851 shutdown_connections 2023-10-16T11:01:01.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2547130851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0b1310 0x7f8b6c0c9950 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2547130851 >> v2:172.21.15.172:3300/0 conn(0x7f8b6c15b4a0 0x7f8b6c0cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.074+0000 7f8b73a98700 1 -- 172.21.15.158:0/2547130851 >> 172.21.15.158:0/2547130851 conn(0x7f8b6c06af00 msgr2=0x7f8b6c0cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:01.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.075+0000 7f8b73a98700 1 -- 172.21.15.158:0/2547130851 shutdown_connections 2023-10-16T11:01:01.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.075+0000 7f8b73a98700 1 -- 172.21.15.158:0/2547130851 wait complete. 2023-10-16T11:01:01.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 Processor -- start 2023-10-16T11:01:01.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 -- start start 2023-10-16T11:01:01.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b6c0b1310 0x7f8b6c14fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0cb2c0 0x7f8b6c0ad270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b6c15b4a0 0x7f8b6c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0ac890 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0c6820 con 0x7f8b6c15b4a0 2023-10-16T11:01:01.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.076+0000 7f8b73a98700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0ade20 con 0x7f8b6c0b1310 2023-10-16T11:01:01.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0cb2c0 0x7f8b6c0ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0cb2c0 0x7f8b6c0ad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33876/0 (socket says 172.21.15.158:33876) 2023-10-16T11:01:01.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 -- 172.21.15.158:0/2262431828 learned_addr learned my addr 172.21.15.158:0/2262431828 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:01.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71834700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b6c0b1310 0x7f8b6c14fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b72035700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b6c15b4a0 0x7f8b6c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 -- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b6c0b1310 msgr2=0x7f8b6c14fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b6c0b1310 0x7f8b6c14fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 -- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b6c15b4a0 msgr2=0x7f8b6c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b6c15b4a0 0x7f8b6c0ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 -- 172.21.15.158:0/2262431828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64009a70 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b71033700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0cb2c0 0x7f8b6c0ad270 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c0011e0 tx=0x7f8b5c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:01.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b627fc700 1 -- 172.21.15.158:0/2262431828 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c004c80 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b627fc700 1 -- 172.21.15.158:0/2262431828 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c004e00 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b627fc700 1 -- 172.21.15.158:0/2262431828 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c007310 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c0c6260 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.077+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c0aca70 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.078+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c0681b0 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.079+0000 7f8b627fc700 1 -- 172.21.15.158:0/2262431828 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8b5c007470 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.079+0000 7f8b627fc700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 0x7f8b5006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.079+0000 7f8b71834700 1 -- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 msgr2=0x7f8b5006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:01.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.079+0000 7f8b627fc700 1 -- 172.21.15.158:0/2262431828 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f8b5c089780 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.079+0000 7f8b71834700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 0x7f8b5006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:01.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.084+0000 7f8b627fc700 1 -- 172.21.15.158:0/2262431828 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b5c054350 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.280+0000 7f8b71834700 1 -- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 msgr2=0x7f8b5006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:01.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.280+0000 7f8b71834700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 0x7f8b5006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:01.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.333+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b6c0ae030 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.334+0000 7f8b627fc700 1 -- 172.21.15.158:0/2262431828 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f8b5c057970 con 0x7f8b6c0cb2c0 2023-10-16T11:01:01.335 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:01.336 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3392606972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3392606972},{"type":"v1","addr":"172.21.15.172:6834","nonce":3392606972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9930,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3321991635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3321991635},{"type":"v1","addr":"172.21.15.158:6836","nonce":3321991635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4013459160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4013459160},{"type":"v1","addr":"172.21.15.158:6837","nonce":4013459160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:00:40.840318+0000","modified":"2023-10-16T11:00:43.870189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37712":{"gid":37712,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/551179403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":551179403},{"type":"v1","addr":"172.21.15.172:6835","nonce":551179403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T11:01:01.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 msgr2=0x7f8b5006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 0x7f8b5006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0cb2c0 msgr2=0x7f8b6c0ad270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0cb2c0 0x7f8b6c0ad270 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c0011e0 tx=0x7f8b5c00ca30 comp rx=0 tx=0).stop 2023-10-16T11:01:01.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 shutdown_connections 2023-10-16T11:01:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f8b5006b020 0x7f8b5006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b6c15b4a0 0x7f8b6c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b6c0b1310 0x7f8b6c14fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 --2- 172.21.15.158:0/2262431828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b6c0cb2c0 0x7f8b6c0ad270 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 >> 172.21.15.158:0/2262431828 conn(0x7f8b6c06af00 msgr2=0x7f8b6c0af310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:01.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 shutdown_connections 2023-10-16T11:01:01.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.336+0000 7f8b73a98700 1 -- 172.21.15.158:0/2262431828 wait complete. 2023-10-16T11:01:01.340 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 85 2023-10-16T11:01:01.350 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:01.421 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:01.421+0000 7f8877926200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:01.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.478+0000 7f0882a3d700 1 Processor -- start 2023-10-16T11:01:01.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.478+0000 7f0882a3d700 1 -- start start 2023-10-16T11:01:01.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f0882a3d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f087c06f470 0x7f087c06e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f0882a3d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f087c06e810 0x7f087c072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f0882a3d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c0728a0 0x7f087c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f0882a3d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c072cd0 con 0x7f087c06e810 2023-10-16T11:01:01.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f0882a3d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c06f850 con 0x7f087c06f470 2023-10-16T11:01:01.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f0882a3d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0b4260 con 0x7f087c0728a0 2023-10-16T11:01:01.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f087b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f087c06e810 0x7f087c072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f087b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f087c06e810 0x7f087c072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33886/0 (socket says 172.21.15.158:33886) 2023-10-16T11:01:01.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f087b7fe700 1 -- 172.21.15.158:0/2192892536 learned_addr learned my addr 172.21.15.158:0/2192892536 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:01.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f0880fda700 1 --2- 172.21.15.158:0/2192892536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c0728a0 0x7f087c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.479+0000 7f087bfff700 1 --2- 172.21.15.158:0/2192892536 >> v2:172.21.15.172:3300/0 conn(0x7f087c06f470 0x7f087c06e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087b7fe700 1 -- 172.21.15.158:0/2192892536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c0728a0 msgr2=0x7f087c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:01.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087b7fe700 1 --2- 172.21.15.158:0/2192892536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c0728a0 0x7f087c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087b7fe700 1 -- 172.21.15.158:0/2192892536 >> v2:172.21.15.172:3300/0 conn(0x7f087c06f470 msgr2=0x7f087c06e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:01.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087b7fe700 1 --2- 172.21.15.158:0/2192892536 >> v2:172.21.15.172:3300/0 conn(0x7f087c06f470 0x7f087c06e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087b7fe700 1 -- 172.21.15.158:0/2192892536 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0b4540 con 0x7f087c06e810 2023-10-16T11:01:01.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087b7fe700 1 --2- 172.21.15.158:0/2192892536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c072360 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f087c0737c0 tx=0x7f0870009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b724fa48dcce2ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:01.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087affd700 1 -- 172.21.15.158:0/2192892536 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870011830 con 0x7f087c06e810 2023-10-16T11:01:01.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087affd700 1 -- 172.21.15.158:0/2192892536 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0870011990 con 0x7f087c06e810 2023-10-16T11:01:01.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.480+0000 7f087affd700 1 -- 172.21.15.158:0/2192892536 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870011d20 con 0x7f087c06e810 2023-10-16T11:01:01.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 -- 172.21.15.158:0/2192892536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 msgr2=0x7f087c072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 --2- 172.21.15.158:0/2192892536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c072360 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f087c0737c0 tx=0x7f0870009bd0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 -- 172.21.15.158:0/2192892536 shutdown_connections 2023-10-16T11:01:01.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 --2- 172.21.15.158:0/2192892536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c0728a0 0x7f087c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 --2- 172.21.15.158:0/2192892536 >> v2:172.21.15.172:3300/0 conn(0x7f087c06f470 0x7f087c06e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 --2- 172.21.15.158:0/2192892536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c072360 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 -- 172.21.15.158:0/2192892536 >> 172.21.15.158:0/2192892536 conn(0x7f087c0ccda0 msgr2=0x7f087c0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:01.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 -- 172.21.15.158:0/2192892536 shutdown_connections 2023-10-16T11:01:01.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.481+0000 7f0882a3d700 1 -- 172.21.15.158:0/2192892536 wait complete. 2023-10-16T11:01:01.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.482+0000 7f0882a3d700 1 Processor -- start 2023-10-16T11:01:01.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.483+0000 7f0882a3d700 1 -- start start 2023-10-16T11:01:01.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.483+0000 7f0882a3d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.483+0000 7f0882a3d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c06f470 0x7f087c15d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.483+0000 7f0882a3d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f087c0728a0 0x7f087c15dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.483+0000 7f0882a3d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c15e110 con 0x7f087c06e810 2023-10-16T11:01:01.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.483+0000 7f0882a3d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c15adf0 con 0x7f087c0728a0 2023-10-16T11:01:01.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.483+0000 7f0882a3d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cf4e0 con 0x7f087c06f470 2023-10-16T11:01:01.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33890/0 (socket says 172.21.15.158:33890) 2023-10-16T11:01:01.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087bfff700 1 -- 172.21.15.158:0/1914921821 learned_addr learned my addr 172.21.15.158:0/1914921821 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:01.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087b7fe700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c06f470 0x7f087c15d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f0880fda700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f087c0728a0 0x7f087c15dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087b7fe700 1 -- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f087c0728a0 msgr2=0x7f087c15dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087b7fe700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f087c0728a0 0x7f087c15dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087b7fe700 1 -- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 msgr2=0x7f087c15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087b7fe700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c15a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087b7fe700 1 -- 172.21.15.158:0/1914921821 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0870009550 con 0x7f087c06f470 2023-10-16T11:01:01.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087bfff700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:01.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f087b7fe700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c06f470 0x7f087c15d690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0870016000 tx=0x7f0870004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:01.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f0878ff9700 1 -- 172.21.15.158:0/1914921821 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870004240 con 0x7f087c06f470 2023-10-16T11:01:01.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f0878ff9700 1 -- 172.21.15.158:0/1914921821 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087001f070 con 0x7f087c06f470 2023-10-16T11:01:01.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c0cf7c0 con 0x7f087c06f470 2023-10-16T11:01:01.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f0878ff9700 1 -- 172.21.15.158:0/1914921821 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08700045e0 con 0x7f087c06f470 2023-10-16T11:01:01.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.484+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c0cfd80 con 0x7f087c06f470 2023-10-16T11:01:01.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.485+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c0681b0 con 0x7f087c06f470 2023-10-16T11:01:01.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.486+0000 7f0878ff9700 1 -- 172.21.15.158:0/1914921821 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0870021030 con 0x7f087c06f470 2023-10-16T11:01:01.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.486+0000 7f0878ff9700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 0x7f086806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.486+0000 7f087bfff700 1 -- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 msgr2=0x7f086806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:01.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.486+0000 7f0878ff9700 1 -- 172.21.15.158:0/1914921821 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f087008ac30 con 0x7f087c06f470 2023-10-16T11:01:01.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.486+0000 7f087bfff700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 0x7f086806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:01.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.491+0000 7f0878ff9700 1 -- 172.21.15.158:0/1914921821 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0870055800 con 0x7f087c06f470 2023-10-16T11:01:01.502 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:01.501+0000 7f8877926200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:01:01.569 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:01.568+0000 7f231ff64200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:01.648 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:01.648+0000 7f8877926200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:01.650 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:01.649+0000 7f231ff64200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:01:01.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.687+0000 7f087bfff700 1 -- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 msgr2=0x7f086806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:01.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.687+0000 7f087bfff700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 0x7f086806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:01.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.693+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f087c161660 con 0x7f087c06f470 2023-10-16T11:01:01.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.694+0000 7f0878ff9700 1 -- 172.21.15.158:0/1914921821 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+17933 (secure 0 0 0) 0x7f0870015020 con 0x7f087c06f470 2023-10-16T11:01:01.695 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:01.696 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":173,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:55.616760+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:00:38.634284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T11:00:39.812150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/982917811":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3695355538":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/2070324944":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/1259583827":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/915803440":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3743813404":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6833/1293733893":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6832/1293733893":"2023-10-17T11:00:55.616726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:01.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 msgr2=0x7f086806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:01.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 0x7f086806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c06f470 msgr2=0x7f087c15d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c06f470 0x7f087c15d690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0870016000 tx=0x7f0870004750 comp rx=0 tx=0).stop 2023-10-16T11:01:01.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 shutdown_connections 2023-10-16T11:01:01.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f086806b020 0x7f086806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f087c0728a0 0x7f087c15dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c06e810 0x7f087c15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.696+0000 7f0882a3d700 1 --2- 172.21.15.158:0/1914921821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f087c06f470 0x7f087c15d690 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.697+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 >> 172.21.15.158:0/1914921821 conn(0x7f087c0ccda0 msgr2=0x7f087c0b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:01.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.697+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 shutdown_connections 2023-10-16T11:01:01.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.697+0000 7f0882a3d700 1 -- 172.21.15.158:0/1914921821 wait complete. 2023-10-16T11:01:01.711 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:00:38.634284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:00:39.812150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:01:01.712 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:01:01.818 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:01.817+0000 7f231ff64200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:01.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.850+0000 7f1ae25a9700 1 Processor -- start 2023-10-16T11:01:01.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.850+0000 7f1ae25a9700 1 -- start start 2023-10-16T11:01:01.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1ae25a9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1adc15daa0 0x7f1adc0cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1ae25a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1ae25a9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1adc0d1ee0 con 0x7f1adc15d6c0 2023-10-16T11:01:01.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1ae25a9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1adc0d40a0 con 0x7f1adc15daa0 2023-10-16T11:01:01.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1ae25a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc0d12d0 con 0x7f1adc0d0640 2023-10-16T11:01:01.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1adbfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1adc15d6c0 0x7f1adc0cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34100/0 (socket says 172.21.15.158:34100) 2023-10-16T11:01:01.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1adbfff700 1 -- 172.21.15.158:0/1104368715 learned_addr learned my addr 172.21.15.158:0/1104368715 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:01.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1ae0b46700 1 --2- 172.21.15.158:0/1104368715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1adb7fe700 1 --2- 172.21.15.158:0/1104368715 >> v2:172.21.15.172:3300/0 conn(0x7f1adc15daa0 0x7f1adc0cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1adaffd700 1 -- 172.21.15.158:0/1104368715 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407258624 0 0) 0x7f1adc0d1ee0 con 0x7f1adc15d6c0 2023-10-16T11:01:01.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.851+0000 7f1adaffd700 1 -- 172.21.15.158:0/1104368715 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac40035b0 con 0x7f1adc15d6c0 2023-10-16T11:01:01.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1ae0b46700 1 -- 172.21.15.158:0/1104368715 >> v2:172.21.15.172:3300/0 conn(0x7f1adc15daa0 msgr2=0x7f1adc0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:01.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1ae0b46700 1 --2- 172.21.15.158:0/1104368715 >> v2:172.21.15.172:3300/0 conn(0x7f1adc15daa0 0x7f1adc0cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1ae0b46700 1 -- 172.21.15.158:0/1104368715 >> v1:172.21.15.158:6789/0 conn(0x7f1adc15d6c0 legacy=0x7f1adc0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1ae0b46700 1 -- 172.21.15.158:0/1104368715 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc0d15b0 con 0x7f1adc0d0640 2023-10-16T11:01:01.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1adb7fe700 1 --2- 172.21.15.158:0/1104368715 >> v2:172.21.15.172:3300/0 conn(0x7f1adc15daa0 0x7f1adc0cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:01.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1ae0b46700 1 --2- 172.21.15.158:0/1104368715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cf190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009d90 tx=0x7f1ad400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b412cb592b8018a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:01.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1adaffd700 1 -- 172.21.15.158:0/1104368715 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad400e070 con 0x7f1adc0d0640 2023-10-16T11:01:01.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1adaffd700 1 -- 172.21.15.158:0/1104368715 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad400bb20 con 0x7f1adc0d0640 2023-10-16T11:01:01.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.852+0000 7f1adaffd700 1 -- 172.21.15.158:0/1104368715 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4012430 con 0x7f1adc0d0640 2023-10-16T11:01:01.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 -- 172.21.15.158:0/1104368715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 msgr2=0x7f1adc0cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/1104368715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cf190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009d90 tx=0x7f1ad400b330 comp rx=0 tx=0).stop 2023-10-16T11:01:01.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 -- 172.21.15.158:0/1104368715 shutdown_connections 2023-10-16T11:01:01.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/1104368715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cf190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/1104368715 >> v2:172.21.15.172:3300/0 conn(0x7f1adc15daa0 0x7f1adc0cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 -- 172.21.15.158:0/1104368715 >> 172.21.15.158:0/1104368715 conn(0x7f1adc09c940 msgr2=0x7f1adc0d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:01.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 -- 172.21.15.158:0/1104368715 shutdown_connections 2023-10-16T11:01:01.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.853+0000 7f1ae25a9700 1 -- 172.21.15.158:0/1104368715 wait complete. 2023-10-16T11:01:01.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.854+0000 7f1ae25a9700 1 Processor -- start 2023-10-16T11:01:01.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.854+0000 7f1ae25a9700 1 -- start start 2023-10-16T11:01:01.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ae25a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ae25a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1adc15d6c0 0x7f1adc1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ae25a9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1adc15daa0 0x7f1adc152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ae25a9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc15be80 con 0x7f1adc15d6c0 2023-10-16T11:01:01.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ae25a9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc15b300 con 0x7f1adc15daa0 2023-10-16T11:01:01.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ae25a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc15b5e0 con 0x7f1adc0d0640 2023-10-16T11:01:01.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1adc15d6c0 0x7f1adc1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1adc15d6c0 0x7f1adc1526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33932/0 (socket says 172.21.15.158:33932) 2023-10-16T11:01:01.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 -- 172.21.15.158:0/4115085318 learned_addr learned my addr 172.21.15.158:0/4115085318 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:01.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adbfff700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ae0b46700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1adc15daa0 0x7f1adc152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:01.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 -- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 msgr2=0x7f1adc0cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 -- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1adc15daa0 msgr2=0x7f1adc152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:01.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1adc15daa0 0x7f1adc152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:01.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 -- 172.21.15.158:0/4115085318 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad00094a0 con 0x7f1adc15d6c0 2023-10-16T11:01:01.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1adb7fe700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1adc15d6c0 0x7f1adc1526c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000e9b0 tx=0x7f1ad000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:01.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ad8ff9700 1 -- 172.21.15.158:0/4115085318 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000ee50 con 0x7f1adc15d6c0 2023-10-16T11:01:01.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.855+0000 7f1ad8ff9700 1 -- 172.21.15.158:0/4115085318 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad000c850 con 0x7f1adc15d6c0 2023-10-16T11:01:01.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.856+0000 7f1ad8ff9700 1 -- 172.21.15.158:0/4115085318 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000cc00 con 0x7f1adc15d6c0 2023-10-16T11:01:01.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.856+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad4009a70 con 0x7f1adc15d6c0 2023-10-16T11:01:01.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.856+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc15bba0 con 0x7f1adc15d6c0 2023-10-16T11:01:01.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.857+0000 7f1ad8ff9700 1 -- 172.21.15.158:0/4115085318 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1ad0013470 con 0x7f1adc15d6c0 2023-10-16T11:01:01.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.857+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1adc0681b0 con 0x7f1adc15d6c0 2023-10-16T11:01:01.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.857+0000 7f1ad8ff9700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 0x7f1ac806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:01.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.857+0000 7f1adbfff700 1 -- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 msgr2=0x7f1ac806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:01.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.857+0000 7f1ad8ff9700 1 -- 172.21.15.158:0/4115085318 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f1ad0089770 con 0x7f1adc15d6c0 2023-10-16T11:01:01.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.857+0000 7f1adbfff700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 0x7f1ac806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:01.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:01.862+0000 7f1ad8ff9700 1 -- 172.21.15.158:0/4115085318 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ad0054340 con 0x7f1adc15d6c0 2023-10-16T11:01:01.878 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:01.878+0000 7f8877926200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:01:01.953 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:01.952+0000 7f8877926200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:01:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.058+0000 7f1adbfff700 1 -- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 msgr2=0x7f1ac806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.058+0000 7f1adbfff700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 0x7f1ac806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:02.062 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:02.061+0000 7f231ff64200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:01:02.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.066+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1adc0d4500 con 0x7f1adc15d6c0 2023-10-16T11:01:02.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.066+0000 7f1ad8ff9700 1 -- 172.21.15.158:0/4115085318 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ad0057960 con 0x7f1adc15d6c0 2023-10-16T11:01:02.068 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:01:02.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 msgr2=0x7f1ac806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:02.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 0x7f1ac806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1adc15d6c0 msgr2=0x7f1adc1526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1adc15d6c0 0x7f1adc1526c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000e9b0 tx=0x7f1ad000ecc0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 shutdown_connections 2023-10-16T11:01:02.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1ac806b020 0x7f1ac806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1adc15daa0 0x7f1adc152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1adc0d0640 0x7f1adc0cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 --2- 172.21.15.158:0/4115085318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1adc15d6c0 0x7f1adc1526c0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 >> 172.21.15.158:0/4115085318 conn(0x7f1adc09c940 msgr2=0x7f1adc0cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:02.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.068+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 shutdown_connections 2023-10-16T11:01:02.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.069+0000 7f1ae25a9700 1 -- 172.21.15.158:0/4115085318 wait complete. 2023-10-16T11:01:02.082 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:01:02.145 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:02.143+0000 7f231ff64200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:01:02.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.209+0000 7f42bcf66700 1 Processor -- start 2023-10-16T11:01:02.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.209+0000 7f42bcf66700 1 -- start start 2023-10-16T11:01:02.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42bcf66700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42b806e0d0 0x7f42b815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42bcf66700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f42b80cbcf0 0x7f42b80cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42bcf66700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b815f380 con 0x7f42b815dc40 2023-10-16T11:01:02.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42bcf66700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b8071e70 con 0x7f42b806e0d0 2023-10-16T11:01:02.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42bcf66700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b8072150 con 0x7f42b80cbcf0 2023-10-16T11:01:02.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42b806e0d0 0x7f42b815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42b806e0d0 0x7f42b815ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33942/0 (socket says 172.21.15.158:33942) 2023-10-16T11:01:02.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 -- 172.21.15.158:0/2229832357 learned_addr learned my addr 172.21.15.158:0/2229832357 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:02.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b6d9d700 1 --2- 172.21.15.158:0/2229832357 >> v2:172.21.15.172:3300/0 conn(0x7f42b80cbcf0 0x7f42b80cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 -- 172.21.15.158:0/2229832357 >> v2:172.21.15.172:3300/0 conn(0x7f42b80cbcf0 msgr2=0x7f42b80cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:02.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 --2- 172.21.15.158:0/2229832357 >> v2:172.21.15.172:3300/0 conn(0x7f42b80cbcf0 0x7f42b80cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 -- 172.21.15.158:0/2229832357 >> v1:172.21.15.172:6789/0 conn(0x7f42b815dc40 legacy=0x7f42b806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 -- 172.21.15.158:0/2229832357 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b8072430 con 0x7f42b806e0d0 2023-10-16T11:01:02.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b6d9d700 1 --2- 172.21.15.158:0/2229832357 >> v2:172.21.15.172:3300/0 conn(0x7f42b80cbcf0 0x7f42b80cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:02.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b5d9b700 1 --2- 172.21.15.158:0/2229832357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b815ee40 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f42b806e610 tx=0x7f42ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6677bf6be59308ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:02.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b559a700 1 -- 172.21.15.158:0/2229832357 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac00b870 con 0x7f42b806e0d0 2023-10-16T11:01:02.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b559a700 1 -- 172.21.15.158:0/2229832357 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ac00b9d0 con 0x7f42b806e0d0 2023-10-16T11:01:02.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.211+0000 7f42b559a700 1 -- 172.21.15.158:0/2229832357 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac00bd60 con 0x7f42b806e0d0 2023-10-16T11:01:02.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 -- 172.21.15.158:0/2229832357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 msgr2=0x7f42b815ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 --2- 172.21.15.158:0/2229832357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b815ee40 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f42b806e610 tx=0x7f42ac009bd0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 -- 172.21.15.158:0/2229832357 shutdown_connections 2023-10-16T11:01:02.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 --2- 172.21.15.158:0/2229832357 >> v2:172.21.15.172:3300/0 conn(0x7f42b80cbcf0 0x7f42b80cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 --2- 172.21.15.158:0/2229832357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b815ee40 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 -- 172.21.15.158:0/2229832357 >> 172.21.15.158:0/2229832357 conn(0x7f42b80d03b0 msgr2=0x7f42b80d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:02.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 -- 172.21.15.158:0/2229832357 shutdown_connections 2023-10-16T11:01:02.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.212+0000 7f42bcf66700 1 -- 172.21.15.158:0/2229832357 wait complete. 2023-10-16T11:01:02.219 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:02.218+0000 7f8877926200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:01:02.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.214+0000 7f42bcf66700 1 Processor -- start 2023-10-16T11:01:02.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.214+0000 7f42bcf66700 1 -- start start 2023-10-16T11:01:02.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42bcf66700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b81530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42bcf66700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b80cbcf0 0x7f42b80b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42bcf66700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b815dc40 0x7f42b80cf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42bcf66700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80b16c0 con 0x7f42b806e0d0 2023-10-16T11:01:02.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42bcf66700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80af410 con 0x7f42b815dc40 2023-10-16T11:01:02.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42bcf66700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b8159670 con 0x7f42b80cbcf0 2023-10-16T11:01:02.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42b659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b81530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42b6d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b815dc40 0x7f42b80cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42b5d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b80cbcf0 0x7f42b80b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42b659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b81530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33946/0 (socket says 172.21.15.158:33946) 2023-10-16T11:01:02.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.215+0000 7f42b659c700 1 -- 172.21.15.158:0/612203334 learned_addr learned my addr 172.21.15.158:0/612203334 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:02.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b659c700 1 -- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b80cbcf0 msgr2=0x7f42b80b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b659c700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b80cbcf0 0x7f42b80b0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b659c700 1 -- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b815dc40 msgr2=0x7f42b80cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b659c700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b815dc40 0x7f42b80cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b659c700 1 -- 172.21.15.158:0/612203334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac009550 con 0x7f42b806e0d0 2023-10-16T11:01:02.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b5d9b700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b80cbcf0 0x7f42b80b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:02.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b6d9d700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b815dc40 0x7f42b80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:02.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42b659c700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b81530e0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c450 tx=0x7f42a800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:02.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42a6ffd700 1 -- 172.21.15.158:0/612203334 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a80052d0 con 0x7f42b806e0d0 2023-10-16T11:01:02.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42a6ffd700 1 -- 172.21.15.158:0/612203334 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a8005430 con 0x7f42b806e0d0 2023-10-16T11:01:02.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b8159950 con 0x7f42b806e0d0 2023-10-16T11:01:02.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42a6ffd700 1 -- 172.21.15.158:0/612203334 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a80038c0 con 0x7f42b806e0d0 2023-10-16T11:01:02.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.216+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b8159f10 con 0x7f42b806e0d0 2023-10-16T11:01:02.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.217+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b80681b0 con 0x7f42b806e0d0 2023-10-16T11:01:02.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.218+0000 7f42a6ffd700 1 -- 172.21.15.158:0/612203334 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f42a8003a20 con 0x7f42b806e0d0 2023-10-16T11:01:02.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.218+0000 7f42a6ffd700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 0x7f429806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.218+0000 7f42b5d9b700 1 -- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 msgr2=0x7f429806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:02.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.218+0000 7f42a6ffd700 1 -- 172.21.15.158:0/612203334 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f42a8088c90 con 0x7f42b806e0d0 2023-10-16T11:01:02.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.218+0000 7f42b5d9b700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 0x7f429806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:02.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.223+0000 7f42a6ffd700 1 -- 172.21.15.158:0/612203334 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f42a8053860 con 0x7f42b806e0d0 2023-10-16T11:01:02.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.419+0000 7f42b5d9b700 1 -- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 msgr2=0x7f429806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:02.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.419+0000 7f42b5d9b700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 0x7f429806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:02.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.424+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42b815f380 con 0x7f42b806e0d0 2023-10-16T11:01:02.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.424+0000 7f42a6ffd700 1 -- 172.21.15.158:0/612203334 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v173) v1 ==== 104+0+11 (secure 0 0 0) 0x7f42a8017020 con 0x7f42b806e0d0 2023-10-16T11:01:02.426 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:01:02.428 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f231ff64200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:01:02.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 msgr2=0x7f429806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:02.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 0x7f429806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 msgr2=0x7f42b81530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b81530e0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c450 tx=0x7f42a800c760 comp rx=0 tx=0).stop 2023-10-16T11:01:02.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 shutdown_connections 2023-10-16T11:01:02.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f429806b020 0x7f429806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b815dc40 0x7f42b80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42b806e0d0 0x7f42b81530e0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 --2- 172.21.15.158:0/612203334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b80cbcf0 0x7f42b80b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 >> 172.21.15.158:0/612203334 conn(0x7f42b80d03b0 msgr2=0x7f42b806d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:02.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 shutdown_connections 2023-10-16T11:01:02.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.427+0000 7f42bcf66700 1 -- 172.21.15.158:0/612203334 wait complete. 2023-10-16T11:01:02.441 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:02.440+0000 7f8877926200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:01:02.441 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:01:02.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.567+0000 7f6bb5fc0700 1 Processor -- start 2023-10-16T11:01:02.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.567+0000 7f6bb5fc0700 1 -- start start 2023-10-16T11:01:02.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6bb5fc0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6bb0159280 0x7f6bb006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6bb5fc0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6bb00cbef0 0x7f6bb00cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6bb5fc0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb0159660 con 0x7f6bb00d0e10 2023-10-16T11:01:02.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6bb5fc0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb00cb710 con 0x7f6bb00cbef0 2023-10-16T11:01:02.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6bb5fc0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb006b120 con 0x7f6bb0159280 2023-10-16T11:01:02.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6baffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6bb00cbef0 0x7f6bb00cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6baffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6bb00cbef0 0x7f6bb00cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33960/0 (socket says 172.21.15.158:33960) 2023-10-16T11:01:02.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.568+0000 7f6baffff700 1 -- 172.21.15.158:0/252506209 learned_addr learned my addr 172.21.15.158:0/252506209 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:02.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6baf7fe700 1 --2- 172.21.15.158:0/252506209 >> v2:172.21.15.172:3300/0 conn(0x7f6bb0159280 0x7f6bb006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6baffff700 1 -- 172.21.15.158:0/252506209 >> v2:172.21.15.172:3300/0 conn(0x7f6bb0159280 msgr2=0x7f6bb006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:02.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6baffff700 1 --2- 172.21.15.158:0/252506209 >> v2:172.21.15.172:3300/0 conn(0x7f6bb0159280 0x7f6bb006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6baffff700 1 -- 172.21.15.158:0/252506209 >> v1:172.21.15.172:6789/0 conn(0x7f6bb00d0e10 legacy=0x7f6bb015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6baffff700 1 -- 172.21.15.158:0/252506209 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb006b400 con 0x7f6bb00cbef0 2023-10-16T11:01:02.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6baffff700 1 --2- 172.21.15.158:0/252506209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00cbef0 0x7f6bb00cc340 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00cdf50 tx=0x7f6ba8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31dc88304370e5f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:02.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6bae7fc700 1 -- 172.21.15.158:0/252506209 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8011830 con 0x7f6bb00cbef0 2023-10-16T11:01:02.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6bae7fc700 1 -- 172.21.15.158:0/252506209 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba8011990 con 0x7f6bb00cbef0 2023-10-16T11:01:02.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.569+0000 7f6bae7fc700 1 -- 172.21.15.158:0/252506209 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8011d20 con 0x7f6bb00cbef0 2023-10-16T11:01:02.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/252506209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00cbef0 msgr2=0x7f6bb00cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/252506209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00cbef0 0x7f6bb00cc340 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00cdf50 tx=0x7f6ba8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/252506209 shutdown_connections 2023-10-16T11:01:02.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/252506209 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00cbef0 0x7f6bb00cc340 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/252506209 >> v2:172.21.15.172:3300/0 conn(0x7f6bb0159280 0x7f6bb006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/252506209 >> 172.21.15.158:0/252506209 conn(0x7f6bb0099a10 msgr2=0x7f6bb00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:02.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/252506209 shutdown_connections 2023-10-16T11:01:02.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.570+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/252506209 wait complete. 2023-10-16T11:01:02.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.572+0000 7f6bb5fc0700 1 Processor -- start 2023-10-16T11:01:02.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.572+0000 7f6bb5fc0700 1 -- start start 2023-10-16T11:01:02.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6bb5fc0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bb00cbef0 0x7f6bb0156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6bb5fc0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00d0e10 0x7f6bb00ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6bb5fc0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bb0159280 0x7f6bb00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6bb5fc0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00ae290 con 0x7f6bb00d0e10 2023-10-16T11:01:02.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6bb5fc0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00b5030 con 0x7f6bb0159280 2023-10-16T11:01:02.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6bb5fc0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00c4250 con 0x7f6bb00cbef0 2023-10-16T11:01:02.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6baeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00d0e10 0x7f6bb00ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6baeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00d0e10 0x7f6bb00ad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33962/0 (socket says 172.21.15.158:33962) 2023-10-16T11:01:02.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6baeffd700 1 -- 172.21.15.158:0/2017793887 learned_addr learned my addr 172.21.15.158:0/2017793887 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:02.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6baffff700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bb0159280 0x7f6bb00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.573+0000 7f6baf7fe700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bb00cbef0 0x7f6bb0156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6baeffd700 1 -- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bb00cbef0 msgr2=0x7f6bb0156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6baeffd700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bb00cbef0 0x7f6bb0156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6baeffd700 1 -- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bb0159280 msgr2=0x7f6bb00b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6baeffd700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bb0159280 0x7f6bb00b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6baeffd700 1 -- 172.21.15.158:0/2017793887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba8009550 con 0x7f6bb00d0e10 2023-10-16T11:01:02.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6baeffd700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00d0e10 0x7f6bb00ad080 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400c4e0 tx=0x7f6ba400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:02.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6b97fff700 1 -- 172.21.15.158:0/2017793887 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40052d0 con 0x7f6bb00d0e10 2023-10-16T11:01:02.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6b97fff700 1 -- 172.21.15.158:0/2017793887 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba4005430 con 0x7f6bb00d0e10 2023-10-16T11:01:02.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb00ad710 con 0x7f6bb00d0e10 2023-10-16T11:01:02.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6b97fff700 1 -- 172.21.15.158:0/2017793887 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40038c0 con 0x7f6bb00d0e10 2023-10-16T11:01:02.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.574+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb00adcd0 con 0x7f6bb00d0e10 2023-10-16T11:01:02.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.575+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb00c4410 con 0x7f6bb00d0e10 2023-10-16T11:01:02.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.576+0000 7f6b97fff700 1 -- 172.21.15.158:0/2017793887 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f6ba4003a20 con 0x7f6bb00d0e10 2023-10-16T11:01:02.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.576+0000 7f6b97fff700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 0x7f6b9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.576+0000 7f6baf7fe700 1 -- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 msgr2=0x7f6b9c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:02.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.576+0000 7f6b97fff700 1 -- 172.21.15.158:0/2017793887 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f6ba4088f40 con 0x7f6bb00d0e10 2023-10-16T11:01:02.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.576+0000 7f6baf7fe700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 0x7f6b9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:02.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.581+0000 7f6b97fff700 1 -- 172.21.15.158:0/2017793887 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ba4053a80 con 0x7f6bb00d0e10 2023-10-16T11:01:02.592 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:02.592+0000 7f8877926200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:01:02.683 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:02.682+0000 7f231ff64200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:01:02.708 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:02.707+0000 7f8877926200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:01:02.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.777+0000 7f6baf7fe700 1 -- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 msgr2=0x7f6b9c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:02.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.777+0000 7f6baf7fe700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 0x7f6b9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:02.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.792+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6bb00681b0 con 0x7f6bb00d0e10 2023-10-16T11:01:02.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.792+0000 7f6b97fff700 1 -- 172.21.15.158:0/2017793887 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v173) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6ba40170a0 con 0x7f6bb00d0e10 2023-10-16T11:01:02.793 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:01:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 msgr2=0x7f6b9c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 0x7f6b9c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00d0e10 msgr2=0x7f6bb00ad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00d0e10 0x7f6bb00ad080 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400c4e0 tx=0x7f6ba400c7f0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 shutdown_connections 2023-10-16T11:01:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f6b9c06b020 0x7f6b9c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bb0159280 0x7f6bb00b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bb00cbef0 0x7f6bb0156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 --2- 172.21.15.158:0/2017793887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bb00d0e10 0x7f6bb00ad080 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.794+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 >> 172.21.15.158:0/2017793887 conn(0x7f6bb0099a10 msgr2=0x7f6bb0159e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.795+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 shutdown_connections 2023-10-16T11:01:02.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.795+0000 7f6bb5fc0700 1 -- 172.21.15.158:0/2017793887 wait complete. 2023-10-16T11:01:02.809 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:01:02.845 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:02.844+0000 7f231ff64200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:01:02.863 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:02.863+0000 7f8877926200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:01:02.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.930+0000 7f2df8272700 1 Processor -- start 2023-10-16T11:01:02.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.930+0000 7f2df8272700 1 -- start start 2023-10-16T11:01:02.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.931+0000 7f2df8272700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2df00cc1b0 0x7f2df00b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.931+0000 7f2df8272700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df015b930 con 0x7f2df015f0f0 2023-10-16T11:01:02.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.931+0000 7f2df8272700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df015f5f0 con 0x7f2df00ce530 2023-10-16T11:01:02.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.931+0000 7f2df8272700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df00cf020 con 0x7f2df00cc1b0 2023-10-16T11:01:02.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.931+0000 7f2df580d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2df00cc1b0 0x7f2df00b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.931+0000 7f2df680f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2df015f0f0 0x7f2df00cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34150/0 (socket says 172.21.15.158:34150) 2023-10-16T11:01:02.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.931+0000 7f2df680f700 1 -- 172.21.15.158:0/2801334155 learned_addr learned my addr 172.21.15.158:0/2801334155 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:02.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.932+0000 7f2df580d700 1 -- 172.21.15.158:0/2801334155 >> v1:172.21.15.172:6789/0 conn(0x7f2df00ce530 legacy=0x7f2df00cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.932+0000 7f2df580d700 1 -- 172.21.15.158:0/2801334155 >> v1:172.21.15.158:6789/0 conn(0x7f2df015f0f0 legacy=0x7f2df00cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.932+0000 7f2df580d700 1 -- 172.21.15.158:0/2801334155 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df015f8d0 con 0x7f2df00cc1b0 2023-10-16T11:01:02.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.932+0000 7f2df580d700 1 --2- 172.21.15.158:0/2801334155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df00cc1b0 0x7f2df00b3b20 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009840 tx=0x7f2de0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad2e3382b3bf500 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:02.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.932+0000 7f2df500c700 1 -- 172.21.15.158:0/2801334155 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0011830 con 0x7f2df00cc1b0 2023-10-16T11:01:02.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.932+0000 7f2df500c700 1 -- 172.21.15.158:0/2801334155 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de0011990 con 0x7f2df00cc1b0 2023-10-16T11:01:02.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.932+0000 7f2df500c700 1 -- 172.21.15.158:0/2801334155 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0011d20 con 0x7f2df00cc1b0 2023-10-16T11:01:02.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.933+0000 7f2df8272700 1 -- 172.21.15.158:0/2801334155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df00cc1b0 msgr2=0x7f2df00b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.933+0000 7f2df8272700 1 --2- 172.21.15.158:0/2801334155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df00cc1b0 0x7f2df00b3b20 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009840 tx=0x7f2de0009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:02.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.933+0000 7f2df8272700 1 -- 172.21.15.158:0/2801334155 shutdown_connections 2023-10-16T11:01:02.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.933+0000 7f2df8272700 1 --2- 172.21.15.158:0/2801334155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df00cc1b0 0x7f2df00b3b20 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.933+0000 7f2df8272700 1 -- 172.21.15.158:0/2801334155 >> 172.21.15.158:0/2801334155 conn(0x7f2df006a8d0 msgr2=0x7f2df006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:02.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.933+0000 7f2df8272700 1 -- 172.21.15.158:0/2801334155 shutdown_connections 2023-10-16T11:01:02.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.933+0000 7f2df8272700 1 -- 172.21.15.158:0/2801334155 wait complete. 2023-10-16T11:01:02.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.934+0000 7f2df8272700 1 Processor -- start 2023-10-16T11:01:02.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.934+0000 7f2df8272700 1 -- start start 2023-10-16T11:01:02.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2df00cc1b0 0x7f2df00c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2df00ce530 0x7f2df015acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df015f0f0 0x7f2df01f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df00c9f40 con 0x7f2df015f0f0 2023-10-16T11:01:02.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df00c9470 con 0x7f2df00ce530 2023-10-16T11:01:02.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df015b210 con 0x7f2df00cc1b0 2023-10-16T11:01:02.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df015f0f0 0x7f2df01f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df015f0f0 0x7f2df01f45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33980/0 (socket says 172.21.15.158:33980) 2023-10-16T11:01:02.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 -- 172.21.15.158:0/496271067 learned_addr learned my addr 172.21.15.158:0/496271067 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:02.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df580d700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2df00ce530 0x7f2df015acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df600e700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2df00cc1b0 0x7f2df00c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:02.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 -- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2df00cc1b0 msgr2=0x7f2df00c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2df00cc1b0 0x7f2df00c8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 -- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2df00ce530 msgr2=0x7f2df015acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:02.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2df00ce530 0x7f2df015acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:02.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 -- 172.21.15.158:0/496271067 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de00094a0 con 0x7f2df015f0f0 2023-10-16T11:01:02.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df680f700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df015f0f0 0x7f2df01f45e0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f2de800cb20 tx=0x7f2de800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:02.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2de6ffd700 1 -- 172.21.15.158:0/496271067 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de800a9a0 con 0x7f2df015f0f0 2023-10-16T11:01:02.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2de6ffd700 1 -- 172.21.15.158:0/496271067 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de800ab00 con 0x7f2df015f0f0 2023-10-16T11:01:02.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df00c96a0 con 0x7f2df015f0f0 2023-10-16T11:01:02.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2de6ffd700 1 -- 172.21.15.158:0/496271067 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de800ae70 con 0x7f2df015f0f0 2023-10-16T11:01:02.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.935+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df00c9c60 con 0x7f2df015f0f0 2023-10-16T11:01:02.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.936+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df00681b0 con 0x7f2df015f0f0 2023-10-16T11:01:02.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.937+0000 7f2de6ffd700 1 -- 172.21.15.158:0/496271067 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2de801b3b0 con 0x7f2df015f0f0 2023-10-16T11:01:02.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.937+0000 7f2de6ffd700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 0x7f2dd406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:02.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.937+0000 7f2df600e700 1 -- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 msgr2=0x7f2dd406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:02.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.938+0000 7f2de6ffd700 1 -- 172.21.15.158:0/496271067 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f2de8089690 con 0x7f2df015f0f0 2023-10-16T11:01:02.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.938+0000 7f2df600e700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 0x7f2dd406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:02.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:02.946+0000 7f2de6ffd700 1 -- 172.21.15.158:0/496271067 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2de8054280 con 0x7f2df015f0f0 2023-10-16T11:01:02.971 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:02.970+0000 7f231ff64200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:01:03.126 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:03.126+0000 7f8877926200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:01:03.136 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:03.135+0000 7f231ff64200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:01:03.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.139+0000 7f2df600e700 1 -- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 msgr2=0x7f2dd406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:03.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.139+0000 7f2df600e700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 0x7f2dd406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:03.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.190+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2df00b3900 con 0x7f2df015f0f0 2023-10-16T11:01:03.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.190+0000 7f2de6ffd700 1 -- 172.21.15.158:0/496271067 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f2de80578a0 con 0x7f2df015f0f0 2023-10-16T11:01:03.192 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:03.192 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3392606972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3392606972},{"type":"v1","addr":"172.21.15.172:6834","nonce":3392606972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9930,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3321991635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3321991635},{"type":"v1","addr":"172.21.15.158:6836","nonce":3321991635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4013459160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4013459160},{"type":"v1","addr":"172.21.15.158:6837","nonce":4013459160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:00:40.840318+0000","modified":"2023-10-16T11:00:43.870189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37712":{"gid":37712,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/551179403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":551179403},{"type":"v1","addr":"172.21.15.172:6835","nonce":551179403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T11:01:03.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.192+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 msgr2=0x7f2dd406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:03.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.192+0000 7f2df8272700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 0x7f2dd406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.192+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df015f0f0 msgr2=0x7f2df01f45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.192+0000 7f2df8272700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df015f0f0 0x7f2df01f45e0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f2de800cb20 tx=0x7f2de800ce30 comp rx=0 tx=0).stop 2023-10-16T11:01:03.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 shutdown_connections 2023-10-16T11:01:03.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f2dd406b020 0x7f2dd406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2df015f0f0 0x7f2df01f45e0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2df00cc1b0 0x7f2df00c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 --2- 172.21.15.158:0/496271067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2df00ce530 0x7f2df015acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 >> 172.21.15.158:0/496271067 conn(0x7f2df006a8d0 msgr2=0x7f2df00b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:03.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 shutdown_connections 2023-10-16T11:01:03.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.193+0000 7f2df8272700 1 -- 172.21.15.158:0/496271067 wait complete. 2023-10-16T11:01:03.197 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 85 2023-10-16T11:01:03.208 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:03.287 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:03.287+0000 7f8877926200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:01:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.372+0000 7ff172cae700 1 Processor -- start 2023-10-16T11:01:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.372+0000 7ff172cae700 1 -- start start 2023-10-16T11:01:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.372+0000 7ff172cae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.372+0000 7ff172cae700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff16c0729b0 0x7ff16c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.372+0000 7ff172cae700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16c071e50 con 0x7ff16c06ab40 2023-10-16T11:01:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.372+0000 7ff172cae700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff16c161270 con 0x7ff16c0729b0 2023-10-16T11:01:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.372+0000 7ff172cae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c161510 con 0x7ff16c06e5d0 2023-10-16T11:01:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff16c0729b0 0x7ff16c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff16c0729b0 0x7ff16c071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33990/0 (socket says 172.21.15.158:33990) 2023-10-16T11:01:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 -- 172.21.15.158:0/1848423193 learned_addr learned my addr 172.21.15.158:0/1848423193 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff16bfff700 1 --2- 172.21.15.158:0/1848423193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 -- 172.21.15.158:0/1848423193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 msgr2=0x7ff16c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 --2- 172.21.15.158:0/1848423193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 -- 172.21.15.158:0/1848423193 >> v1:172.21.15.172:6789/0 conn(0x7ff16c06ab40 legacy=0x7ff16c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff16b7fe700 1 -- 172.21.15.158:0/1848423193 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401804597 0 0) 0x7ff16c071e50 con 0x7ff16c06ab40 2023-10-16T11:01:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 -- 172.21.15.158:0/1848423193 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c1617f0 con 0x7ff16c0729b0 2023-10-16T11:01:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff17124b700 1 --2- 172.21.15.158:0/1848423193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c071910 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7ff164009e50 tx=0x7ff16400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7ac8d37a460d5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff16b7fe700 1 -- 172.21.15.158:0/1848423193 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16400e070 con 0x7ff16c0729b0 2023-10-16T11:01:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff16b7fe700 1 -- 172.21.15.158:0/1848423193 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16400b910 con 0x7ff16c0729b0 2023-10-16T11:01:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.373+0000 7ff16b7fe700 1 -- 172.21.15.158:0/1848423193 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16400bcc0 con 0x7ff16c0729b0 2023-10-16T11:01:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 -- 172.21.15.158:0/1848423193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 msgr2=0x7ff16c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 --2- 172.21.15.158:0/1848423193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c071910 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7ff164009e50 tx=0x7ff16400b330 comp rx=0 tx=0).stop 2023-10-16T11:01:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 -- 172.21.15.158:0/1848423193 shutdown_connections 2023-10-16T11:01:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 --2- 172.21.15.158:0/1848423193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c071910 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 --2- 172.21.15.158:0/1848423193 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 -- 172.21.15.158:0/1848423193 >> 172.21.15.158:0/1848423193 conn(0x7ff16c0d02b0 msgr2=0x7ff16c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 -- 172.21.15.158:0/1848423193 shutdown_connections 2023-10-16T11:01:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.374+0000 7ff172cae700 1 -- 172.21.15.158:0/1848423193 wait complete. 2023-10-16T11:01:03.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.375+0000 7ff172cae700 1 Processor -- start 2023-10-16T11:01:03.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.375+0000 7ff172cae700 1 -- start start 2023-10-16T11:01:03.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff172cae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff16c06ab40 0x7ff16c0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff172cae700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c0b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff172cae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c0c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff172cae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c154030 con 0x7ff16c0729b0 2023-10-16T11:01:03.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff172cae700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c0b1630 con 0x7ff16c06ab40 2023-10-16T11:01:03.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff172cae700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c0b1870 con 0x7ff16c06e5d0 2023-10-16T11:01:03.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff17124b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c0c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff17124b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c0c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:33994/0 (socket says 172.21.15.158:33994) 2023-10-16T11:01:03.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff17124b700 1 -- 172.21.15.158:0/802772824 learned_addr learned my addr 172.21.15.158:0/802772824 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:03.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff16bfff700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c0b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.376+0000 7ff170a4a700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff16c06ab40 0x7ff16c0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff17124b700 1 -- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 msgr2=0x7ff16c0b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff17124b700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c0b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff17124b700 1 -- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff16c06ab40 msgr2=0x7ff16c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff17124b700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff16c06ab40 0x7ff16c0b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff17124b700 1 -- 172.21.15.158:0/802772824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff164009b30 con 0x7ff16c0729b0 2023-10-16T11:01:03.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff170a4a700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff16c06ab40 0x7ff16c0b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:03.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff17124b700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c0c9780 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7ff16c074ec0 tx=0x7ff164004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:03.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff1697fa700 1 -- 172.21.15.158:0/802772824 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff164004880 con 0x7ff16c0729b0 2023-10-16T11:01:03.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff1697fa700 1 -- 172.21.15.158:0/802772824 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1640099b0 con 0x7ff16c0729b0 2023-10-16T11:01:03.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16c0b1b50 con 0x7ff16c0729b0 2023-10-16T11:01:03.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff1697fa700 1 -- 172.21.15.158:0/802772824 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16401b670 con 0x7ff16c0729b0 2023-10-16T11:01:03.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.377+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16c0b2110 con 0x7ff16c0729b0 2023-10-16T11:01:03.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.378+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16c0681b0 con 0x7ff16c0729b0 2023-10-16T11:01:03.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.379+0000 7ff1697fa700 1 -- 172.21.15.158:0/802772824 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff16401b7d0 con 0x7ff16c0729b0 2023-10-16T11:01:03.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.379+0000 7ff1697fa700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 0x7ff15006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.379+0000 7ff170a4a700 1 -- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 msgr2=0x7ff15006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:03.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.379+0000 7ff1697fa700 1 -- 172.21.15.158:0/802772824 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7ff16408b120 con 0x7ff16c0729b0 2023-10-16T11:01:03.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.379+0000 7ff170a4a700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 0x7ff15006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:03.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.387+0000 7ff1697fa700 1 -- 172.21.15.158:0/802772824 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff164055c90 con 0x7ff16c0729b0 2023-10-16T11:01:03.421 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:03.420+0000 7f231ff64200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:01:03.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.580+0000 7ff170a4a700 1 -- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 msgr2=0x7ff15006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:03.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.580+0000 7ff170a4a700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 0x7ff15006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:03.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.588+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff16c0b6230 con 0x7ff16c0729b0 2023-10-16T11:01:03.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.589+0000 7ff1697fa700 1 -- 172.21.15.158:0/802772824 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+17933 (secure 0 0 0) 0x7ff1640592b0 con 0x7ff16c0729b0 2023-10-16T11:01:03.591 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:03.591 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":173,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:00:55.616760+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:00:38.634284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T11:00:39.812150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/982917811":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3695355538":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/2070324944":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/1259583827":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/915803440":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3743813404":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6833/1293733893":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6832/1293733893":"2023-10-17T11:00:55.616726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:03.592 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:03.591+0000 7f231ff64200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:01:03.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 msgr2=0x7ff15006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:03.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 0x7ff15006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 msgr2=0x7ff16c0c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c0c9780 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7ff16c074ec0 tx=0x7ff164004ad0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 shutdown_connections 2023-10-16T11:01:03.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7ff15006b020 0x7ff15006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff16c0729b0 0x7ff16c0c9780 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff16c06ab40 0x7ff16c0b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 --2- 172.21.15.158:0/802772824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff16c06e5d0 0x7ff16c0b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 >> 172.21.15.158:0/802772824 conn(0x7ff16c0d02b0 msgr2=0x7ff16c0ae350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:03.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 shutdown_connections 2023-10-16T11:01:03.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.592+0000 7ff172cae700 1 -- 172.21.15.158:0/802772824 wait complete. 2023-10-16T11:01:03.606 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:01:03.606 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:01:03.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.728+0000 7f219b058700 1 Processor -- start 2023-10-16T11:01:03.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.729+0000 7f219b058700 1 -- start start 2023-10-16T11:01:03.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.729+0000 7f219b058700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f219415daa0 0x7f219406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.729+0000 7f219b058700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21940d3570 0x7f21940d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.729+0000 7f219b058700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21940ce4d0 con 0x7f21940d0bd0 2023-10-16T11:01:03.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.729+0000 7f219b058700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21940d1c20 con 0x7f21940d3570 2023-10-16T11:01:03.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.729+0000 7f219b058700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21940d3f00 con 0x7f219415daa0 2023-10-16T11:01:03.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2198df4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f219415daa0 0x7f219406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f21995f5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21940d3570 0x7f21940d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2193fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f21940d0bd0 0x7f21940d1510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:38082/0 (socket says 172.21.15.158:38082) 2023-10-16T11:01:03.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2193fff700 1 -- 172.21.15.158:0/1439600654 learned_addr learned my addr 172.21.15.158:0/1439600654 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:03.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f21937fe700 1 -- 172.21.15.158:0/1439600654 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309920956 0 0) 0x7f21940ce4d0 con 0x7f21940d0bd0 2023-10-16T11:01:03.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f21937fe700 1 -- 172.21.15.158:0/1439600654 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21840035b0 con 0x7f21940d0bd0 2023-10-16T11:01:03.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2198df4700 1 -- 172.21.15.158:0/1439600654 >> v2:172.21.15.172:3300/0 conn(0x7f21940d3570 msgr2=0x7f21940d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:03.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2198df4700 1 --2- 172.21.15.158:0/1439600654 >> v2:172.21.15.172:3300/0 conn(0x7f21940d3570 0x7f21940d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2198df4700 1 -- 172.21.15.158:0/1439600654 >> v1:172.21.15.172:6789/0 conn(0x7f21940d0bd0 legacy=0x7f21940d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2198df4700 1 -- 172.21.15.158:0/1439600654 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219406c1b0 con 0x7f219415daa0 2023-10-16T11:01:03.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f21995f5700 1 --2- 172.21.15.158:0/1439600654 >> v2:172.21.15.172:3300/0 conn(0x7f21940d3570 0x7f21940d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f2198df4700 1 --2- 172.21.15.158:0/1439600654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f219415daa0 0x7f219406b8d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2188009820 tx=0x7f2188009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=940865469642bd44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f21937fe700 1 -- 172.21.15.158:0/1439600654 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188005e30 con 0x7f219415daa0 2023-10-16T11:01:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f21937fe700 1 -- 172.21.15.158:0/1439600654 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2188006030 con 0x7f219415daa0 2023-10-16T11:01:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.730+0000 7f21937fe700 1 -- 172.21.15.158:0/1439600654 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21880063c0 con 0x7f219415daa0 2023-10-16T11:01:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 -- 172.21.15.158:0/1439600654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f219415daa0 msgr2=0x7f219406b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 --2- 172.21.15.158:0/1439600654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f219415daa0 0x7f219406b8d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2188009820 tx=0x7f2188009b30 comp rx=0 tx=0).stop 2023-10-16T11:01:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 -- 172.21.15.158:0/1439600654 shutdown_connections 2023-10-16T11:01:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 --2- 172.21.15.158:0/1439600654 >> v2:172.21.15.172:3300/0 conn(0x7f21940d3570 0x7f21940d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 --2- 172.21.15.158:0/1439600654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f219415daa0 0x7f219406b8d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 -- 172.21.15.158:0/1439600654 >> 172.21.15.158:0/1439600654 conn(0x7f219408d040 msgr2=0x7f219409c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 -- 172.21.15.158:0/1439600654 shutdown_connections 2023-10-16T11:01:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.731+0000 7f219b058700 1 -- 172.21.15.158:0/1439600654 wait complete. 2023-10-16T11:01:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.732+0000 7f219b058700 1 Processor -- start 2023-10-16T11:01:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.732+0000 7f219b058700 1 -- start start 2023-10-16T11:01:03.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f219b058700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21940d0bd0 0x7f21941f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f219b058700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21940d3570 0x7f21940cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f219b058700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f219415daa0 0x7f21940b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f219b058700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21941f3e90 con 0x7f219415daa0 2023-10-16T11:01:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f219b058700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21940b3550 con 0x7f21940d0bd0 2023-10-16T11:01:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f219b058700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21940b3830 con 0x7f21940d3570 2023-10-16T11:01:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21940d0bd0 0x7f21941f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21940d0bd0 0x7f21941f3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:35186/0 (socket says 172.21.15.158:35186) 2023-10-16T11:01:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 -- 172.21.15.158:0/2883635900 learned_addr learned my addr 172.21.15.158:0/2883635900 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f21995f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f219415daa0 0x7f21940b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2193fff700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21940d3570 0x7f21940cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 -- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21940d3570 msgr2=0x7f21940cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21940d3570 0x7f21940cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 -- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f219415daa0 msgr2=0x7f21940b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f219415daa0 0x7f21940b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.733+0000 7f2198df4700 1 -- 172.21.15.158:0/2883635900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c009a70 con 0x7f21940d0bd0 2023-10-16T11:01:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.734+0000 7f2198df4700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21940d0bd0 0x7f21941f3950 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2188000c00 tx=0x7f2188004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.734+0000 7f21917fa700 1 -- 172.21.15.158:0/2883635900 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21880063a0 con 0x7f21940d0bd0 2023-10-16T11:01:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.734+0000 7f21917fa700 1 -- 172.21.15.158:0/2883635900 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2188020070 con 0x7f21940d0bd0 2023-10-16T11:01:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.734+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2188009430 con 0x7f21940d0bd0 2023-10-16T11:01:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.734+0000 7f21917fa700 1 -- 172.21.15.158:0/2883635900 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188006760 con 0x7f21940d0bd0 2023-10-16T11:01:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.734+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21940b3df0 con 0x7f21940d0bd0 2023-10-16T11:01:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.735+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21940681b0 con 0x7f21940d0bd0 2023-10-16T11:01:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.735+0000 7f21917fa700 1 -- 172.21.15.158:0/2883635900 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2188022030 con 0x7f21940d0bd0 2023-10-16T11:01:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.736+0000 7f21917fa700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 0x7f217806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.736+0000 7f2193fff700 1 -- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 msgr2=0x7f217806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.736+0000 7f21917fa700 1 -- 172.21.15.158:0/2883635900 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f218808ae70 con 0x7f21940d0bd0 2023-10-16T11:01:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.736+0000 7f2193fff700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 0x7f217806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:03.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.740+0000 7f21917fa700 1 -- 172.21.15.158:0/2883635900 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2188055a40 con 0x7f21940d0bd0 2023-10-16T11:01:03.822 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:03.819+0000 7f8877926200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:01:03.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.937+0000 7f2193fff700 1 -- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 msgr2=0x7f217806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:03.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.937+0000 7f2193fff700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 0x7f217806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:03.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.983+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21940d4450 con 0x7f21940d0bd0 2023-10-16T11:01:03.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.984+0000 7f21917fa700 1 -- 172.21.15.158:0/2883635900 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f2188059060 con 0x7f21940d0bd0 2023-10-16T11:01:03.985 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:03.986 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3392606972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3392606972},{"type":"v1","addr":"172.21.15.172:6834","nonce":3392606972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9930,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3321991635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3321991635},{"type":"v1","addr":"172.21.15.158:6836","nonce":3321991635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4013459160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4013459160},{"type":"v1","addr":"172.21.15.158:6837","nonce":4013459160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:00:40.840318+0000","modified":"2023-10-16T11:00:43.870189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37712":{"gid":37712,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/551179403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":551179403},{"type":"v1","addr":"172.21.15.172:6835","nonce":551179403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T11:01:03.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 msgr2=0x7f217806d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:03.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 0x7f217806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21940d0bd0 msgr2=0x7f21941f3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:03.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21940d0bd0 0x7f21941f3950 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2188000c00 tx=0x7f2188004750 comp rx=0 tx=0).stop 2023-10-16T11:01:03.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 shutdown_connections 2023-10-16T11:01:03.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f217806b050 0x7f217806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f219415daa0 0x7f21940b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21940d0bd0 0x7f21941f3950 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 --2- 172.21.15.158:0/2883635900 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21940d3570 0x7f21940cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:03.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 >> 172.21.15.158:0/2883635900 conn(0x7f219408d040 msgr2=0x7f21940b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:03.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 shutdown_connections 2023-10-16T11:01:03.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:03.986+0000 7f219b058700 1 -- 172.21.15.158:0/2883635900 wait complete. 2023-10-16T11:01:03.991 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 85 2023-10-16T11:01:04.000 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:01:04.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.126+0000 7f1e3a3e8700 1 Processor -- start 2023-10-16T11:01:04.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.126+0000 7f1e3a3e8700 1 -- start start 2023-10-16T11:01:04.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e3a3e8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1e3415d7c0 0x7f1e340d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e3a3e8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1e340d1210 0x7f1e340ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e3a3e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340ce950 0x7f1e340b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e3a3e8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e340d4060 con 0x7f1e340d1210 2023-10-16T11:01:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e3a3e8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e340d3bd0 con 0x7f1e3415d7c0 2023-10-16T11:01:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e3a3e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340cedc0 con 0x7f1e340ce950 2023-10-16T11:01:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1e340d1210 0x7f1e340ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1e340d1210 0x7f1e340ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34018/0 (socket says 172.21.15.158:34018) 2023-10-16T11:01:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 -- 172.21.15.158:0/177224914 learned_addr learned my addr 172.21.15.158:0/177224914 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e33fff700 1 --2- 172.21.15.158:0/177224914 >> v2:172.21.15.172:3300/0 conn(0x7f1e3415d7c0 0x7f1e340d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e38985700 1 --2- 172.21.15.158:0/177224914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340ce950 0x7f1e340b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 -- 172.21.15.158:0/177224914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340ce950 msgr2=0x7f1e340b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 --2- 172.21.15.158:0/177224914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340ce950 0x7f1e340b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 -- 172.21.15.158:0/177224914 >> v2:172.21.15.172:3300/0 conn(0x7f1e3415d7c0 msgr2=0x7f1e340d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 --2- 172.21.15.158:0/177224914 >> v2:172.21.15.172:3300/0 conn(0x7f1e3415d7c0 0x7f1e340d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 -- 172.21.15.158:0/177224914 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e340d2cf0 con 0x7f1e340d1210 2023-10-16T11:01:04.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e33fff700 1 --2- 172.21.15.158:0/177224914 >> v2:172.21.15.172:3300/0 conn(0x7f1e3415d7c0 0x7f1e340d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:04.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e337fe700 1 --2- 172.21.15.158:0/177224914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340d1210 0x7f1e340ce410 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f1e340d0bc0 tx=0x7f1e28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69859733dd2be60 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:04.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e32ffd700 1 -- 172.21.15.158:0/177224914 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2800b870 con 0x7f1e340d1210 2023-10-16T11:01:04.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e32ffd700 1 -- 172.21.15.158:0/177224914 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2800b9d0 con 0x7f1e340d1210 2023-10-16T11:01:04.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.127+0000 7f1e32ffd700 1 -- 172.21.15.158:0/177224914 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2800bd60 con 0x7f1e340d1210 2023-10-16T11:01:04.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/177224914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340d1210 msgr2=0x7f1e340ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/177224914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340d1210 0x7f1e340ce410 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f1e340d0bc0 tx=0x7f1e28009bd0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/177224914 shutdown_connections 2023-10-16T11:01:04.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/177224914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340ce950 0x7f1e340b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/177224914 >> v2:172.21.15.172:3300/0 conn(0x7f1e3415d7c0 0x7f1e340d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/177224914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340d1210 0x7f1e340ce410 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/177224914 >> 172.21.15.158:0/177224914 conn(0x7f1e340669f0 msgr2=0x7f1e34066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/177224914 shutdown_connections 2023-10-16T11:01:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.128+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/177224914 wait complete. 2023-10-16T11:01:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.130+0000 7f1e3a3e8700 1 Processor -- start 2023-10-16T11:01:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.130+0000 7f1e3a3e8700 1 -- start start 2023-10-16T11:01:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e3a3e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340ce950 0x7f1e3414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e3a3e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340d1210 0x7f1e341f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e3a3e8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e3415d7c0 0x7f1e34160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e3a3e8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e341f6290 con 0x7f1e340ce950 2023-10-16T11:01:04.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e3a3e8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3414db00 con 0x7f1e3415d7c0 2023-10-16T11:01:04.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e3a3e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3414dde0 con 0x7f1e340d1210 2023-10-16T11:01:04.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340ce950 0x7f1e3414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340ce950 0x7f1e3414d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34022/0 (socket says 172.21.15.158:34022) 2023-10-16T11:01:04.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 -- 172.21.15.158:0/4093995452 learned_addr learned my addr 172.21.15.158:0/4093995452 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:04.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e337fe700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340d1210 0x7f1e341f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e38985700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e3415d7c0 0x7f1e34160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 -- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340d1210 msgr2=0x7f1e341f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340d1210 0x7f1e341f6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 -- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e3415d7c0 msgr2=0x7f1e34160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e3415d7c0 0x7f1e34160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.131+0000 7f1e33fff700 1 -- 172.21.15.158:0/4093995452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28009550 con 0x7f1e340ce950 2023-10-16T11:01:04.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.132+0000 7f1e33fff700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340ce950 0x7f1e3414d200 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c750 tx=0x7f1e2400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:04.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.132+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24011800 con 0x7f1e340ce950 2023-10-16T11:01:04.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.132+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e24011960 con 0x7f1e340ce950 2023-10-16T11:01:04.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.132+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e341f5f20 con 0x7f1e340ce950 2023-10-16T11:01:04.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.132+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24011cd0 con 0x7f1e340ce950 2023-10-16T11:01:04.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.132+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3414d740 con 0x7f1e340ce950 2023-10-16T11:01:04.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.133+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e340681b0 con 0x7f1e340ce950 2023-10-16T11:01:04.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.133+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1e24021350 con 0x7f1e340ce950 2023-10-16T11:01:04.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.134+0000 7f1e30ff9700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1e2006b020 0x7f1e2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.134+0000 7f1e337fe700 1 -- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1e2006b020 msgr2=0x7f1e2006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/4231823616 2023-10-16T11:01:04.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.134+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f1e24089a80 con 0x7f1e340ce950 2023-10-16T11:01:04.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.134+0000 7f1e337fe700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1e2006b020 0x7f1e2006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:04.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.138+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e240544b0 con 0x7f1e340ce950 2023-10-16T11:01:04.163 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.162+0000 7f231ff64200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:01:04.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.182+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 104) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1e24087b40 con 0x7f1e340ce950 2023-10-16T11:01:04.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.182+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1e2006b020 msgr2=0x7f1e2006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:04.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.182+0000 7f1e30ff9700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1e2006b020 0x7f1e2006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.247 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.245+0000 7f229874a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.247 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.245+0000 7f229874a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.248 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.245+0000 7f229874a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.248 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.245+0000 7f229874a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.248 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.245+0000 7f229874a700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.248 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.246+0000 7f229b750700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.248 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.246+0000 7f229b750700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.248 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.246+0000 7f229b750700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.249 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.246+0000 7f229b750700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.249 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:04.246+0000 7f229b750700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:04.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.396+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e340ce280 con 0x7f1e340ce950 2023-10-16T11:01:04.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.397+0000 7f1e30ff9700 1 -- 172.21.15.158:0/4093995452 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f1e24075b20 con 0x7f1e340ce950 2023-10-16T11:01:04.400 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:04.400 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3392606972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3392606972},{"type":"v1","addr":"172.21.15.172:6834","nonce":3392606972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9930,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3321991635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3321991635},{"type":"v1","addr":"172.21.15.158:6836","nonce":3321991635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4013459160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4013459160},{"type":"v1","addr":"172.21.15.158:6837","nonce":4013459160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:00:40.840318+0000","modified":"2023-10-16T11:00:43.870189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37712":{"gid":37712,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/551179403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":551179403},{"type":"v1","addr":"172.21.15.172:6835","nonce":551179403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T11:01:04.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340ce950 msgr2=0x7f1e3414d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340ce950 0x7f1e3414d200 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c750 tx=0x7f1e2400ca60 comp rx=0 tx=0).stop 2023-10-16T11:01:04.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 shutdown_connections 2023-10-16T11:01:04.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:6832/4231823616,v1:172.21.15.158:6833/4231823616] conn(0x7f1e2006b020 0x7f1e2006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e3415d7c0 0x7f1e34160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e340ce950 0x7f1e3414d200 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 --2- 172.21.15.158:0/4093995452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e340d1210 0x7f1e341f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 >> 172.21.15.158:0/4093995452 conn(0x7f1e340669f0 msgr2=0x7f1e340ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:04.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 shutdown_connections 2023-10-16T11:01:04.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.399+0000 7f1e3a3e8700 1 -- 172.21.15.158:0/4093995452 wait complete. 2023-10-16T11:01:04.404 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 85 2023-10-16T11:01:04.414 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:04.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.534+0000 7f6ae1b80700 1 Processor -- start 2023-10-16T11:01:04.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.534+0000 7f6ae1b80700 1 -- start start 2023-10-16T11:01:04.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ae1b80700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6adc06fa50 0x7f6adc06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ae1b80700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc06e770 con 0x7f6adc0cc5d0 2023-10-16T11:01:04.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ae1b80700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc0cbe00 con 0x7f6adc15dfb0 2023-10-16T11:01:04.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ae1b80700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6adc0cf270 con 0x7f6adc06fa50 2023-10-16T11:01:04.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6adb7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6adc0cc5d0 0x7f6adc0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34278/0 (socket says 172.21.15.158:34278) 2023-10-16T11:01:04.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6adb7fe700 1 -- 172.21.15.158:0/2971241298 learned_addr learned my addr 172.21.15.158:0/2971241298 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:04.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6adbfff700 1 --2- 172.21.15.158:0/2971241298 >> v2:172.21.15.172:3300/0 conn(0x7f6adc06fa50 0x7f6adc06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399817098 0 0) 0x7f6acc001110 con 0x7f6adc0cc5d0 2023-10-16T11:01:04.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac40035b0 con 0x7f6adc0cc5d0 2023-10-16T11:01:04.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174616938 0 0) 0x7f6ac40035b0 con 0x7f6adc0cc5d0 2023-10-16T11:01:04.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 >> v2:172.21.15.172:3300/0 conn(0x7f6adc06fa50 msgr2=0x7f6adc06e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 --2- 172.21.15.158:0/2971241298 >> v2:172.21.15.172:3300/0 conn(0x7f6adc06fa50 0x7f6adc06e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 >> v1:172.21.15.172:6789/0 conn(0x7f6adc15dfb0 legacy=0x7f6adc06f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc15e4e0 con 0x7f6adc0cc5d0 2023-10-16T11:01:04.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f6acc001800 con 0x7f6adc0cc5d0 2023-10-16T11:01:04.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 >> v1:172.21.15.158:6789/0 conn(0x7f6adc0cc5d0 legacy=0x7f6adc0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ac4003a00 0x7f6ac4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6ac40048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ac40050d0 0x7f6ac400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0cf270 con 0x7f6ac4004400 2023-10-16T11:01:04.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac400c030 con 0x7f6ac40050d0 2023-10-16T11:01:04.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.535+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac400c1f0 con 0x7f6ac4003a00 2023-10-16T11:01:04.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adaffd700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6ac40048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adb7fe700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ac4003a00 0x7f6ac4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adbfff700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ac40050d0 0x7f6ac400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adaffd700 1 -- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ac4003a00 msgr2=0x7f6ac4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adaffd700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ac4003a00 0x7f6ac4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adaffd700 1 -- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ac40050d0 msgr2=0x7f6ac400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adaffd700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ac40050d0 0x7f6ac400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adbfff700 1 -- 172.21.15.158:0/2971241298 reap_dead start 2023-10-16T11:01:04.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adaffd700 1 -- 172.21.15.158:0/2971241298 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac40035b0 con 0x7f6ac4004400 2023-10-16T11:01:04.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6adaffd700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6ac40048d0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0cee90 tx=0x7f6ac4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffac9826b9c3c8ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:04.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad00052d0 con 0x7f6ac4004400 2023-10-16T11:01:04.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad0005450 con 0x7f6ac4004400 2023-10-16T11:01:04.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.536+0000 7f6ada7fc700 1 -- 172.21.15.158:0/2971241298 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad00038c0 con 0x7f6ac4004400 2023-10-16T11:01:04.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.537+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2971241298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 msgr2=0x7f6ac40048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.537+0000 7f6ae1b80700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6ac40048d0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0cee90 tx=0x7f6ac4005880 comp rx=0 tx=0).stop 2023-10-16T11:01:04.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.537+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2971241298 shutdown_connections 2023-10-16T11:01:04.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.537+0000 7f6ae1b80700 1 --2- 172.21.15.158:0/2971241298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6ac40048d0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.537+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2971241298 >> 172.21.15.158:0/2971241298 conn(0x7f6adc0d02b0 msgr2=0x7f6adc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:04.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.537+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2971241298 shutdown_connections 2023-10-16T11:01:04.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.537+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2971241298 wait complete. 2023-10-16T11:01:04.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.538+0000 7f6ae1b80700 1 Processor -- start 2023-10-16T11:01:04.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6ae1b80700 1 -- start start 2023-10-16T11:01:04.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6ae1b80700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6adc15c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6ae1b80700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6adc14f740 0x7f6adc0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6ae1b80700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6adc0c78c0 0x7f6adc1545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6ae1b80700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc15b7f0 con 0x7f6ac4004400 2023-10-16T11:01:04.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6ae1b80700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc15bad0 con 0x7f6adc14f740 2023-10-16T11:01:04.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6ae1b80700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc15bdb0 con 0x7f6adc0c78c0 2023-10-16T11:01:04.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6adb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6adc15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.539+0000 7f6adb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6adc15c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34114/0 (socket says 172.21.15.158:34114) 2023-10-16T11:01:04.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adb7fe700 1 -- 172.21.15.158:0/2209544382 learned_addr learned my addr 172.21.15.158:0/2209544382 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:04.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adaffd700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6adc14f740 0x7f6adc0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adbfff700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6adc0c78c0 0x7f6adc1545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adb7fe700 1 -- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6adc0c78c0 msgr2=0x7f6adc1545f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adb7fe700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6adc0c78c0 0x7f6adc1545f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adb7fe700 1 -- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6adc14f740 msgr2=0x7f6adc0c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adb7fe700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6adc14f740 0x7f6adc0c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adb7fe700 1 -- 172.21.15.158:0/2209544382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc15c090 con 0x7f6ac4004400 2023-10-16T11:01:04.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6adb7fe700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6adc15c890 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f6acc007fd0 tx=0x7f6acc007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:04.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6ac3fff700 1 -- 172.21.15.158:0/2209544382 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc007e70 con 0x7f6ac4004400 2023-10-16T11:01:04.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6ac3fff700 1 -- 172.21.15.158:0/2209544382 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc012780 con 0x7f6ac4004400 2023-10-16T11:01:04.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc15c370 con 0x7f6ac4004400 2023-10-16T11:01:04.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6ac3fff700 1 -- 172.21.15.158:0/2209544382 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc012b30 con 0x7f6ac4004400 2023-10-16T11:01:04.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.540+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc0b5b20 con 0x7f6ac4004400 2023-10-16T11:01:04.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.541+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc0681b0 con 0x7f6ac4004400 2023-10-16T11:01:04.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.542+0000 7f6ac3fff700 1 -- 172.21.15.158:0/2209544382 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 104) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f6acc018020 con 0x7f6ac4004400 2023-10-16T11:01:04.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.542+0000 7f6ac3fff700 1 -- 172.21.15.158:0/2209544382 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f6acc08ac50 con 0x7f6ac4004400 2023-10-16T11:01:04.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.546+0000 7f6ac3fff700 1 -- 172.21.15.158:0/2209544382 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6acc0559c0 con 0x7f6ac4004400 2023-10-16T11:01:04.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.741+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6adc06e770 con 0x7f6ac4004400 2023-10-16T11:01:04.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.742+0000 7f6ac3fff700 1 -- 172.21.15.158:0/2209544382 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+18443 (secure 0 0 0) 0x7f6adc06e770 con 0x7f6ac4004400 2023-10-16T11:01:04.744 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:04.745 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":174,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:01:04.176052+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:00:38.634284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T11:00:39.812150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3738809292":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/2918364282":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/1160958831":"2023-10-17T11:01:04.175999+0000","172.21.15.158:6833/4231823616":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/2347685989":"2023-10-17T11:01:04.175999+0000","172.21.15.158:6832/4231823616":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/982917811":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3695355538":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/2070324944":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/1259583827":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3480878312":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/915803440":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3743813404":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6833/1293733893":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/2282704909":"2023-10-17T11:01:04.175999+0000","172.21.15.158:6832/1293733893":"2023-10-17T11:00:55.616726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:04.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 msgr2=0x7f6adc15c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6adc15c890 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f6acc007fd0 tx=0x7f6acc007570 comp rx=0 tx=0).stop 2023-10-16T11:01:04.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 shutdown_connections 2023-10-16T11:01:04.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6adc0c78c0 0x7f6adc1545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ac4004400 0x7f6adc15c890 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 --2- 172.21.15.158:0/2209544382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6adc14f740 0x7f6adc0c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 >> 172.21.15.158:0/2209544382 conn(0x7f6adc0d02b0 msgr2=0x7f6adc0d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:04.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 shutdown_connections 2023-10-16T11:01:04.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.744+0000 7f6ae1b80700 1 -- 172.21.15.158:0/2209544382 wait complete. 2023-10-16T11:01:04.759 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:01:04.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c5d389700 1 Processor -- start 2023-10-16T11:01:04.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c5d389700 1 -- start start 2023-10-16T11:01:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c5d389700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c5815d670 0x7f9c5806d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c5d389700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c580cf0a0 0x7f9c580cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c5d389700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c58161320 con 0x7f9c58070340 2023-10-16T11:01:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c5d389700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c580cfa70 con 0x7f9c580cf0a0 2023-10-16T11:01:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c5d389700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c58161600 con 0x7f9c5815d670 2023-10-16T11:01:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.880+0000 7f9c567fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9c58070340 0x7f9c58161a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34296/0 (socket says 172.21.15.158:34296) 2023-10-16T11:01:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c567fc700 1 -- 172.21.15.158:0/2455695954 learned_addr learned my addr 172.21.15.158:0/2455695954 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c5815d670 0x7f9c5806d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c577fe700 1 --2- 172.21.15.158:0/2455695954 >> v2:172.21.15.172:3300/0 conn(0x7f9c580cf0a0 0x7f9c580cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866331778 0 0) 0x7f9c58161320 con 0x7f9c58070340 2023-10-16T11:01:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c440035b0 con 0x7f9c58070340 2023-10-16T11:01:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464855113 0 0) 0x7f9c440035b0 con 0x7f9c58070340 2023-10-16T11:01:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c5815d670 msgr2=0x7f9c5806d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c5815d670 0x7f9c5806d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 >> v2:172.21.15.172:3300/0 conn(0x7f9c580cf0a0 msgr2=0x7f9c580cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 --2- 172.21.15.158:0/2455695954 >> v2:172.21.15.172:3300/0 conn(0x7f9c580cf0a0 0x7f9c580cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c581621a0 con 0x7f9c58070340 2023-10-16T11:01:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f9c40001670 con 0x7f9c58070340 2023-10-16T11:01:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 >> v1:172.21.15.158:6789/0 conn(0x7f9c58070340 legacy=0x7f9c58161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c44003a00 0x7f9c44003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c44004850 0x7f9c44004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44005210 0x7f9c4400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580cfa70 con 0x7f9c44004850 2023-10-16T11:01:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c58161600 con 0x7f9c44003a00 2023-10-16T11:01:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4400bf80 con 0x7f9c44005210 2023-10-16T11:01:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c5815d670 0x7f9c5806d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c577fe700 1 --2- 172.21.15.158:0/2455695954 >> v2:172.21.15.172:3300/0 conn(0x7f9c580cf0a0 0x7f9c580cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c567fc700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c44004850 0x7f9c44004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c44003a00 0x7f9c44003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c577fe700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44005210 0x7f9c4400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c567fc700 1 -- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44005210 msgr2=0x7f9c4400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c567fc700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44005210 0x7f9c4400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c567fc700 1 -- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c44003a00 msgr2=0x7f9c44003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.881+0000 7f9c567fc700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c44003a00 0x7f9c44003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c567fc700 1 -- 172.21.15.158:0/2455695954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c440035b0 con 0x7f9c44004850 2023-10-16T11:01:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c577fe700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44005210 0x7f9c4400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c567fc700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c44004850 0x7f9c44004cd0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000c320 tx=0x7f9c44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e712900f4b220a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4000ce00 con 0x7f9c44004850 2023-10-16T11:01:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40005960 con 0x7f9c44004850 2023-10-16T11:01:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c55ffb700 1 -- 172.21.15.158:0/2455695954 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40005d10 con 0x7f9c44004850 2023-10-16T11:01:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c5d389700 1 -- 172.21.15.158:0/2455695954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c44004850 msgr2=0x7f9c44004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c5d389700 1 --2- 172.21.15.158:0/2455695954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c44004850 0x7f9c44004cd0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000c320 tx=0x7f9c44005880 comp rx=0 tx=0).stop 2023-10-16T11:01:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c577fe700 1 -- 172.21.15.158:0/2455695954 reap_dead start 2023-10-16T11:01:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c5d389700 1 -- 172.21.15.158:0/2455695954 shutdown_connections 2023-10-16T11:01:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c5d389700 1 -- 172.21.15.158:0/2455695954 >> 172.21.15.158:0/2455695954 conn(0x7f9c580cdf50 msgr2=0x7f9c580ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c5d389700 1 -- 172.21.15.158:0/2455695954 shutdown_connections 2023-10-16T11:01:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.882+0000 7f9c5d389700 1 -- 172.21.15.158:0/2455695954 wait complete. 2023-10-16T11:01:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 Processor -- start 2023-10-16T11:01:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 -- start start 2023-10-16T11:01:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44004850 0x7f9c580c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5815ee50 0x7f9c581512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c580c62b0 0x7f9c580b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c581f2e40 con 0x7f9c5815ee50 2023-10-16T11:01:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5815f2d0 con 0x7f9c580c62b0 2023-10-16T11:01:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.884+0000 7f9c5d389700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580c6ae0 con 0x7f9c44004850 2023-10-16T11:01:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5815ee50 0x7f9c581512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5815ee50 0x7f9c581512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34134/0 (socket says 172.21.15.158:34134) 2023-10-16T11:01:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 -- 172.21.15.158:0/4033146227 learned_addr learned my addr 172.21.15.158:0/4033146227 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c56ffd700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44004850 0x7f9c580c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c577fe700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c580c62b0 0x7f9c580b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 -- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44004850 msgr2=0x7f9c580c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44004850 0x7f9c580c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 -- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c580c62b0 msgr2=0x7f9c580b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c580c62b0 0x7f9c580b5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 -- 172.21.15.158:0/4033146227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580c6dc0 con 0x7f9c5815ee50 2023-10-16T11:01:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c567fc700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5815ee50 0x7f9c581512f0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f9c40008fd0 tx=0x7f9c40001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c4f7fe700 1 -- 172.21.15.158:0/4033146227 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4000ce70 con 0x7f9c5815ee50 2023-10-16T11:01:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c4f7fe700 1 -- 172.21.15.158:0/4033146227 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40004a80 con 0x7f9c5815ee50 2023-10-16T11:01:04.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c580c70a0 con 0x7f9c5815ee50 2023-10-16T11:01:04.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c4f7fe700 1 -- 172.21.15.158:0/4033146227 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40004e30 con 0x7f9c5815ee50 2023-10-16T11:01:04.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.885+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c580c7660 con 0x7f9c5815ee50 2023-10-16T11:01:04.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.886+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c580c7820 con 0x7f9c5815ee50 2023-10-16T11:01:04.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.887+0000 7f9c4f7fe700 1 -- 172.21.15.158:0/4033146227 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 104) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9c400070d0 con 0x7f9c5815ee50 2023-10-16T11:01:04.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.887+0000 7f9c4f7fe700 1 -- 172.21.15.158:0/4033146227 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f9c4008a9a0 con 0x7f9c5815ee50 2023-10-16T11:01:04.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:04.891+0000 7f9c4f7fe700 1 -- 172.21.15.158:0/4033146227 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c400558e0 con 0x7f9c5815ee50 2023-10-16T11:01:05.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.126+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9c580681b0 con 0x7f9c5815ee50 2023-10-16T11:01:05.128 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:01:05.126+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:01:05.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.127+0000 7f9c4f7fe700 1 -- 172.21.15.158:0/4033146227 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v86) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9c40007da0 con 0x7f9c5815ee50 2023-10-16T11:01:05.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5815ee50 msgr2=0x7f9c581512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:05.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5815ee50 0x7f9c581512f0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f9c40008fd0 tx=0x7f9c40001a00 comp rx=0 tx=0).stop 2023-10-16T11:01:05.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 shutdown_connections 2023-10-16T11:01:05.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c580c62b0 0x7f9c580b5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c44004850 0x7f9c580c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 --2- 172.21.15.158:0/4033146227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5815ee50 0x7f9c581512f0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 >> 172.21.15.158:0/4033146227 conn(0x7f9c580cdf50 msgr2=0x7f9c580ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:05.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 shutdown_connections 2023-10-16T11:01:05.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.130+0000 7f9c5d389700 1 -- 172.21.15.158:0/4033146227 wait complete. 2023-10-16T11:01:05.135 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:01:05.136 INFO:tasks.ceph.mds.d.smithi172.stderr: -4732> 2023-10-16T11:00:46.424+0000 7fc0a2da7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:01:05.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:01:05.165 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:01:05.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.270+0000 7ff8c2534700 1 Processor -- start 2023-10-16T11:01:05.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8c2534700 1 -- start start 2023-10-16T11:01:05.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8c2534700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff8bc160d40 0x7ff8bc0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:05.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8c2534700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff8bc0c50c0 0x7ff8bc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:05.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8c2534700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc0ce110 con 0x7ff8bc0cf420 2023-10-16T11:01:05.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8c2534700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8bc0c61e0 con 0x7ff8bc0c50c0 2023-10-16T11:01:05.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8c2534700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8bc0ce3f0 con 0x7ff8bc160d40 2023-10-16T11:01:05.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8c0ad1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff8bc0c50c0 0x7ff8bc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:05.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8bb7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff8bc0cf420 0x7ff8bc0c49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34336/0 (socket says 172.21.15.158:34336) 2023-10-16T11:01:05.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.271+0000 7ff8bb7fe700 1 -- 172.21.15.158:0/653206189 learned_addr learned my addr 172.21.15.158:0/653206189 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:05.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8bbfff700 1 --2- 172.21.15.158:0/653206189 >> v2:172.21.15.172:3300/0 conn(0x7ff8bc160d40 0x7ff8bc0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:05.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8c0ad1700 1 -- 172.21.15.158:0/653206189 >> v2:172.21.15.172:3300/0 conn(0x7ff8bc160d40 msgr2=0x7ff8bc0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8c0ad1700 1 --2- 172.21.15.158:0/653206189 >> v2:172.21.15.172:3300/0 conn(0x7ff8bc160d40 0x7ff8bc0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8c0ad1700 1 -- 172.21.15.158:0/653206189 >> v1:172.21.15.158:6789/0 conn(0x7ff8bc0cf420 legacy=0x7ff8bc0c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8c0ad1700 1 -- 172.21.15.158:0/653206189 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc0ce6d0 con 0x7ff8bc0c50c0 2023-10-16T11:01:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8baffd700 1 -- 172.21.15.158:0/653206189 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390377768 0 0) 0x7ff8bc0ce110 con 0x7ff8bc0cf420 2023-10-16T11:01:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8c0ad1700 1 --2- 172.21.15.158:0/653206189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc06abb0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc0d0b80 tx=0x7ff8b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45bc6afb9369722 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8baffd700 1 -- 172.21.15.158:0/653206189 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400e070 con 0x7ff8bc0c50c0 2023-10-16T11:01:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8baffd700 1 -- 172.21.15.158:0/653206189 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b400bb20 con 0x7ff8bc0c50c0 2023-10-16T11:01:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.272+0000 7ff8baffd700 1 -- 172.21.15.158:0/653206189 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4012430 con 0x7ff8bc0c50c0 2023-10-16T11:01:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 -- 172.21.15.158:0/653206189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0c50c0 msgr2=0x7ff8bc06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 --2- 172.21.15.158:0/653206189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc06abb0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc0d0b80 tx=0x7ff8b400b330 comp rx=0 tx=0).stop 2023-10-16T11:01:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 -- 172.21.15.158:0/653206189 shutdown_connections 2023-10-16T11:01:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 --2- 172.21.15.158:0/653206189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc06abb0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 --2- 172.21.15.158:0/653206189 >> v2:172.21.15.172:3300/0 conn(0x7ff8bc160d40 0x7ff8bc0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 -- 172.21.15.158:0/653206189 >> 172.21.15.158:0/653206189 conn(0x7ff8bc0cc5b0 msgr2=0x7ff8bc0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 -- 172.21.15.158:0/653206189 shutdown_connections 2023-10-16T11:01:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.273+0000 7ff8c2534700 1 -- 172.21.15.158:0/653206189 wait complete. 2023-10-16T11:01:05.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.274+0000 7ff8c2534700 1 Processor -- start 2023-10-16T11:01:05.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.274+0000 7ff8c2534700 1 -- start start 2023-10-16T11:01:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c2534700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c2534700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8bc0cf420 0x7ff8bc1f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c2534700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc160d40 0x7ff8bc1f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c2534700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc157660 con 0x7ff8bc160d40 2023-10-16T11:01:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c2534700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc154470 con 0x7ff8bc0c50c0 2023-10-16T11:01:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c2534700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc156a50 con 0x7ff8bc0cf420 2023-10-16T11:01:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc160d40 0x7ff8bc1f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:35346/0 (socket says 172.21.15.158:35346) 2023-10-16T11:01:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc160d40 0x7ff8bc1f6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34170/0 (socket says 172.21.15.158:34170) 2023-10-16T11:01:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8bbfff700 1 -- 172.21.15.158:0/2932395026 learned_addr learned my addr 172.21.15.158:0/2932395026 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8bb7fe700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8bc0cf420 0x7ff8bc1f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 -- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8bc0cf420 msgr2=0x7ff8bc1f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8bc0cf420 0x7ff8bc1f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 -- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0c50c0 msgr2=0x7ff8bc155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 -- 172.21.15.158:0/2932395026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4009a70 con 0x7ff8bc160d40 2023-10-16T11:01:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8bbfff700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.275+0000 7ff8c0ad1700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc160d40 0x7ff8bc1f6ae0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4016000 tx=0x7ff8b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.276+0000 7ff8b8ff9700 1 -- 172.21.15.158:0/2932395026 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4012430 con 0x7ff8bc160d40 2023-10-16T11:01:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.276+0000 7ff8b8ff9700 1 -- 172.21.15.158:0/2932395026 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b4020070 con 0x7ff8bc160d40 2023-10-16T11:01:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.276+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc1612a0 con 0x7ff8bc160d40 2023-10-16T11:01:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.276+0000 7ff8b8ff9700 1 -- 172.21.15.158:0/2932395026 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4004280 con 0x7ff8bc160d40 2023-10-16T11:01:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.276+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc157010 con 0x7ff8bc160d40 2023-10-16T11:01:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.277+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8bc066db0 con 0x7ff8bc160d40 2023-10-16T11:01:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.277+0000 7ff8b8ff9700 1 -- 172.21.15.158:0/2932395026 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 105) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff8b40043e0 con 0x7ff8bc160d40 2023-10-16T11:01:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.277+0000 7ff8b8ff9700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff8a806ad50 0x7ff8a806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.278+0000 7ff8b8ff9700 1 -- 172.21.15.158:0/2932395026 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 7255+0+0 (secure 0 0 0) 0x7ff8b4022070 con 0x7ff8bc160d40 2023-10-16T11:01:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.278+0000 7ff8bbfff700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff8a806ad50 0x7ff8a806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.278+0000 7ff8bbfff700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff8a806ad50 0x7ff8a806d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00112c0 tx=0x7ff8b0009210 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:05.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.282+0000 7ff8b8ff9700 1 -- 172.21.15.158:0/2932395026 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8b40563b0 con 0x7ff8bc160d40 2023-10-16T11:01:05.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:05.527+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff8bc074a90 con 0x7ff8bc160d40 2023-10-16T11:01:06.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.184+0000 7ff8b8ff9700 1 -- 172.21.15.158:0/2932395026 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v87) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff8b40599d0 con 0x7ff8bc160d40 2023-10-16T11:01:06.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.186+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff8a806ad50 msgr2=0x7ff8a806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:06.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.186+0000 7ff8c2534700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff8a806ad50 0x7ff8a806d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00112c0 tx=0x7ff8b0009210 comp rx=0 tx=0).stop 2023-10-16T11:01:06.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.186+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc160d40 msgr2=0x7ff8bc1f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:06.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc160d40 0x7ff8bc1f6ae0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4016000 tx=0x7ff8b4004ad0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 shutdown_connections 2023-10-16T11:01:06.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff8a806ad50 0x7ff8a806d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc160d40 0x7ff8bc1f6ae0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0c50c0 0x7ff8bc155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 --2- 172.21.15.158:0/2932395026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8bc0cf420 0x7ff8bc1f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 >> 172.21.15.158:0/2932395026 conn(0x7ff8bc0cc5b0 msgr2=0x7ff8bc06b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:06.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 shutdown_connections 2023-10-16T11:01:06.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.187+0000 7ff8c2534700 1 -- 172.21.15.158:0/2932395026 wait complete. 2023-10-16T11:01:06.203 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:01:06.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.337+0000 7fe5e7773700 1 Processor -- start 2023-10-16T11:01:06.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.338+0000 7fe5e7773700 1 -- start start 2023-10-16T11:01:06.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.338+0000 7fe5e7773700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe5e006d5d0 0x7fe5e0070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:06.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.338+0000 7fe5e7773700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e015dd70 con 0x7fe5e015d990 2023-10-16T11:01:06.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.338+0000 7fe5e7773700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e0070a00 con 0x7fe5e00cbaf0 2023-10-16T11:01:06.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.338+0000 7fe5e7773700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e01616b0 con 0x7fe5e006d5d0 2023-10-16T11:01:06.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5e550f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe5e015d990 0x7fe5e0161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34368/0 (socket says 172.21.15.158:34368) 2023-10-16T11:01:06.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5e550f700 1 -- 172.21.15.158:0/931103409 learned_addr learned my addr 172.21.15.158:0/931103409 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5e4d0e700 1 --2- 172.21.15.158:0/931103409 >> v2:172.21.15.172:3300/0 conn(0x7fe5e006d5d0 0x7fe5e0070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5d7fff700 1 -- 172.21.15.158:0/931103409 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477633971 0 0) 0x7fe5dc001110 con 0x7fe5e015d990 2023-10-16T11:01:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5d7fff700 1 -- 172.21.15.158:0/931103409 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5cc0035b0 con 0x7fe5e015d990 2023-10-16T11:01:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5e4d0e700 1 -- 172.21.15.158:0/931103409 >> v1:172.21.15.172:6789/0 conn(0x7fe5e00cbaf0 legacy=0x7fe5e00b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5e4d0e700 1 -- 172.21.15.158:0/931103409 >> v1:172.21.15.158:6789/0 conn(0x7fe5e015d990 legacy=0x7fe5e0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5e4d0e700 1 -- 172.21.15.158:0/931103409 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e0070ce0 con 0x7fe5e006d5d0 2023-10-16T11:01:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5e4d0e700 1 --2- 172.21.15.158:0/931103409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e006d5d0 0x7fe5e0070300 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009840 tx=0x7fe5d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d4c1c5faa89ad1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5d7fff700 1 -- 172.21.15.158:0/931103409 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0011830 con 0x7fe5e006d5d0 2023-10-16T11:01:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5d7fff700 1 -- 172.21.15.158:0/931103409 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d0011990 con 0x7fe5e006d5d0 2023-10-16T11:01:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.339+0000 7fe5d7fff700 1 -- 172.21.15.158:0/931103409 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0011d20 con 0x7fe5e006d5d0 2023-10-16T11:01:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.340+0000 7fe5e7773700 1 -- 172.21.15.158:0/931103409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e006d5d0 msgr2=0x7fe5e0070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.340+0000 7fe5e7773700 1 --2- 172.21.15.158:0/931103409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e006d5d0 0x7fe5e0070300 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009840 tx=0x7fe5d0009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.340+0000 7fe5e7773700 1 -- 172.21.15.158:0/931103409 shutdown_connections 2023-10-16T11:01:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.340+0000 7fe5e7773700 1 --2- 172.21.15.158:0/931103409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e006d5d0 0x7fe5e0070300 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.340+0000 7fe5e7773700 1 -- 172.21.15.158:0/931103409 >> 172.21.15.158:0/931103409 conn(0x7fe5e00cdf20 msgr2=0x7fe5e00ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.340+0000 7fe5e7773700 1 -- 172.21.15.158:0/931103409 shutdown_connections 2023-10-16T11:01:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.340+0000 7fe5e7773700 1 -- 172.21.15.158:0/931103409 wait complete. 2023-10-16T11:01:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.341+0000 7fe5e7773700 1 Processor -- start 2023-10-16T11:01:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.341+0000 7fe5e7773700 1 -- start start 2023-10-16T11:01:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e7773700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e006d5d0 0x7fe5e00c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e7773700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e00cbaf0 0x7fe5e01536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e7773700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e015d990 0x7fe5e015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e7773700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e0153c00 con 0x7fe5e015d990 2023-10-16T11:01:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e7773700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00c6640 con 0x7fe5e00cbaf0 2023-10-16T11:01:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e7773700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00c6920 con 0x7fe5e006d5d0 2023-10-16T11:01:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e015d990 0x7fe5e015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e550f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e006d5d0 0x7fe5e00c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e015d990 0x7fe5e015ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34202/0 (socket says 172.21.15.158:34202) 2023-10-16T11:01:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 -- 172.21.15.158:0/2335437516 learned_addr learned my addr 172.21.15.158:0/2335437516 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e4d0e700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e00cbaf0 0x7fe5e01536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 -- 172.21.15.158:0/2335437516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e006d5d0 msgr2=0x7fe5e00c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e006d5d0 0x7fe5e00c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 -- 172.21.15.158:0/2335437516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e00cbaf0 msgr2=0x7fe5e01536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e00cbaf0 0x7fe5e01536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.342+0000 7fe5e5d10700 1 -- 172.21.15.158:0/2335437516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d00094a0 con 0x7fe5e015d990 2023-10-16T11:01:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.343+0000 7fe5e5d10700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e015d990 0x7fe5e015ecc0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800cc50 tx=0x7fe5d800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.343+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8003c00 con 0x7fe5e015d990 2023-10-16T11:01:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.343+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8003d60 con 0x7fe5e015d990 2023-10-16T11:01:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.343+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e00c6c00 con 0x7fe5e015d990 2023-10-16T11:01:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.343+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d800aab0 con 0x7fe5e015d990 2023-10-16T11:01:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.343+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e00c71c0 con 0x7fe5e015d990 2023-10-16T11:01:06.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.344+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e00681b0 con 0x7fe5e015d990 2023-10-16T11:01:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.344+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 105) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe5d800ac10 con 0x7fe5e015d990 2023-10-16T11:01:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.345+0000 7fe5d5ffb700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe5c406ad50 0x7fe5c406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.345+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 7255+0+0 (secure 0 0 0) 0x7fe5d80897f0 con 0x7fe5e015d990 2023-10-16T11:01:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.345+0000 7fe5e550f700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe5c406ad50 0x7fe5c406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:06.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.345+0000 7fe5e550f700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe5c406ad50 0x7fe5c406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc007fd0 tx=0x7fe5dc007c40 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:06.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.349+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5d80545d0 con 0x7fe5e015d990 2023-10-16T11:01:06.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:06.542+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe5e006d100 con 0x7fe5e015d990 2023-10-16T11:01:07.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.190+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v176) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe5d8057bf0 con 0x7fe5e015d990 2023-10-16T11:01:07.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.198+0000 7fe5d5ffb700 1 -- 172.21.15.158:0/2335437516 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 106) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe5d80549a0 con 0x7fe5e015d990 2023-10-16T11:01:07.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.198+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe5c406ad50 msgr2=0x7fe5c406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:07.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.198+0000 7fe5e7773700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe5c406ad50 0x7fe5c406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc007fd0 tx=0x7fe5dc007c40 comp rx=0 tx=0).stop 2023-10-16T11:01:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.198+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e015d990 msgr2=0x7fe5e015ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.198+0000 7fe5e7773700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e015d990 0x7fe5e015ecc0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800cc50 tx=0x7fe5d800cf60 comp rx=0 tx=0).stop 2023-10-16T11:01:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 shutdown_connections 2023-10-16T11:01:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe5c406ad50 0x7fe5c406d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e015d990 0x7fe5e015ecc0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e006d5d0 0x7fe5e00c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 --2- 172.21.15.158:0/2335437516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e00cbaf0 0x7fe5e01536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 >> 172.21.15.158:0/2335437516 conn(0x7fe5e00cdf20 msgr2=0x7fe5e00b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 shutdown_connections 2023-10-16T11:01:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.199+0000 7fe5e7773700 1 -- 172.21.15.158:0/2335437516 wait complete. 2023-10-16T11:01:07.204 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:01:07.215 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:01:07.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.340+0000 7f9d6efdb700 1 Processor -- start 2023-10-16T11:01:07.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.340+0000 7f9d6efdb700 1 -- start start 2023-10-16T11:01:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6efdb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6815da40 0x7f9d680caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6efdb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d680cc590 0x7f9d6806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6efdb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d680cbd30 con 0x7f9d6806ae10 2023-10-16T11:01:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6efdb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6806b280 con 0x7f9d680cc590 2023-10-16T11:01:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6efdb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680cc1c0 con 0x7f9d6815da40 2023-10-16T11:01:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d67fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d680cc590 0x7f9d6806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6d578700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9d6806ae10 0x7f9d680ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34386/0 (socket says 172.21.15.158:34386) 2023-10-16T11:01:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6d578700 1 -- 172.21.15.158:0/2240779091 learned_addr learned my addr 172.21.15.158:0/2240779091 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6cd77700 1 --2- 172.21.15.158:0/2240779091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6815da40 0x7f9d680caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d67fff700 1 -- 172.21.15.158:0/2240779091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6815da40 msgr2=0x7f9d680caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d67fff700 1 --2- 172.21.15.158:0/2240779091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6815da40 0x7f9d680caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d67fff700 1 -- 172.21.15.158:0/2240779091 >> v1:172.21.15.158:6789/0 conn(0x7f9d6806ae10 legacy=0x7f9d680ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d67fff700 1 -- 172.21.15.158:0/2240779091 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6815de20 con 0x7f9d680cc590 2023-10-16T11:01:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d6cd77700 1 --2- 172.21.15.158:0/2240779091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6815da40 0x7f9d680caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d677fe700 1 -- 172.21.15.158:0/2240779091 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014944310 0 0) 0x7f9d680cbd30 con 0x7f9d6806ae10 2023-10-16T11:01:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d67fff700 1 --2- 172.21.15.158:0/2240779091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d680cc590 0x7f9d6806a8d0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009840 tx=0x7f9d54009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0ce0db66cd1844 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.341+0000 7f9d677fe700 1 -- 172.21.15.158:0/2240779091 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d54011830 con 0x7f9d680cc590 2023-10-16T11:01:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d677fe700 1 -- 172.21.15.158:0/2240779091 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d54011990 con 0x7f9d680cc590 2023-10-16T11:01:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d677fe700 1 -- 172.21.15.158:0/2240779091 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d54011d20 con 0x7f9d680cc590 2023-10-16T11:01:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 -- 172.21.15.158:0/2240779091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d680cc590 msgr2=0x7f9d6806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/2240779091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d680cc590 0x7f9d6806a8d0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009840 tx=0x7f9d54009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 -- 172.21.15.158:0/2240779091 shutdown_connections 2023-10-16T11:01:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/2240779091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6815da40 0x7f9d680caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/2240779091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d680cc590 0x7f9d6806a8d0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 -- 172.21.15.158:0/2240779091 >> 172.21.15.158:0/2240779091 conn(0x7f9d68099a10 msgr2=0x7f9d680d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 -- 172.21.15.158:0/2240779091 shutdown_connections 2023-10-16T11:01:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.342+0000 7f9d6efdb700 1 -- 172.21.15.158:0/2240779091 wait complete. 2023-10-16T11:01:07.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.343+0000 7f9d6efdb700 1 Processor -- start 2023-10-16T11:01:07.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.343+0000 7f9d6efdb700 1 -- start start 2023-10-16T11:01:07.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6efdb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6806ae10 0x7f9d68149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:07.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6efdb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d680cc590 0x7f9d680b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6efdb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d6815da40 0x7f9d680c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6efdb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680b4a60 con 0x7f9d6815da40 2023-10-16T11:01:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6efdb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680b4d40 con 0x7f9d680cc590 2023-10-16T11:01:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6efdb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680b5020 con 0x7f9d6806ae10 2023-10-16T11:01:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6d578700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d6815da40 0x7f9d680c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6d578700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d6815da40 0x7f9d680c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34216/0 (socket says 172.21.15.158:34216) 2023-10-16T11:01:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.344+0000 7f9d6d578700 1 -- 172.21.15.158:0/1098949064 learned_addr learned my addr 172.21.15.158:0/1098949064 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6cd77700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6806ae10 0x7f9d68149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6d578700 1 -- 172.21.15.158:0/1098949064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6806ae10 msgr2=0x7f9d68149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6d578700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6806ae10 0x7f9d68149990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6d578700 1 -- 172.21.15.158:0/1098949064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d680cc590 msgr2=0x7f9d680b5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6d578700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d680cc590 0x7f9d680b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6d578700 1 -- 172.21.15.158:0/1098949064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d540094a0 con 0x7f9d6815da40 2023-10-16T11:01:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6d578700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d6815da40 0x7f9d680c3ec0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000cca0 tx=0x7f9d6000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d657fa700 1 -- 172.21.15.158:0/1098949064 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000cea0 con 0x7f9d6815da40 2023-10-16T11:01:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d657fa700 1 -- 172.21.15.158:0/1098949064 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6000ade0 con 0x7f9d6815da40 2023-10-16T11:01:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d680b5300 con 0x7f9d6815da40 2023-10-16T11:01:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d657fa700 1 -- 172.21.15.158:0/1098949064 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60007310 con 0x7f9d6815da40 2023-10-16T11:01:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.345+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d680b46f0 con 0x7f9d6815da40 2023-10-16T11:01:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.346+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d680b5820 con 0x7f9d6815da40 2023-10-16T11:01:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.346+0000 7f9d657fa700 1 -- 172.21.15.158:0/1098949064 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 106) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9d60007470 con 0x7f9d6815da40 2023-10-16T11:01:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.347+0000 7f9d657fa700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9d4c06af90 0x7f9d4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.347+0000 7f9d657fa700 1 -- 172.21.15.158:0/1098949064 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f9d60089a10 con 0x7f9d6815da40 2023-10-16T11:01:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.347+0000 7f9d6cd77700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9d4c06af90 0x7f9d4c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.347+0000 7f9d6cd77700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9d4c06af90 0x7f9d4c06d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.351+0000 7f9d657fa700 1 -- 172.21.15.158:0/1098949064 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d680b5820 con 0x7f9d6815da40 2023-10-16T11:01:07.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:07.550+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9d680b5820 con 0x7f9d6815da40 2023-10-16T11:01:08.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.192+0000 7f9d657fa700 1 -- 172.21.15.158:0/1098949064 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v177) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9d680b5820 con 0x7f9d6815da40 2023-10-16T11:01:08.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9d4c06af90 msgr2=0x7f9d4c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9d4c06af90 0x7f9d4c06d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c008040 comp rx=0 tx=0).stop 2023-10-16T11:01:08.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d6815da40 msgr2=0x7f9d680c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d6815da40 0x7f9d680c3ec0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000cca0 tx=0x7f9d6000a340 comp rx=0 tx=0).stop 2023-10-16T11:01:08.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 shutdown_connections 2023-10-16T11:01:08.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9d4c06af90 0x7f9d4c06d450 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d6815da40 0x7f9d680c3ec0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d6806ae10 0x7f9d68149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 --2- 172.21.15.158:0/1098949064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d680cc590 0x7f9d680b5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 >> 172.21.15.158:0/1098949064 conn(0x7f9d68099a10 msgr2=0x7f9d680b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:08.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 shutdown_connections 2023-10-16T11:01:08.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.195+0000 7f9d6efdb700 1 -- 172.21.15.158:0/1098949064 wait complete. 2023-10-16T11:01:08.200 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:01:08.216 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:01:08.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.342+0000 7ffbe8c58700 1 Processor -- start 2023-10-16T11:01:08.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.342+0000 7ffbe8c58700 1 -- start start 2023-10-16T11:01:08.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe8c58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe8c58700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffbe40cc590 0x7ffbe406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe8c58700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40cbd30 con 0x7ffbe406ae10 2023-10-16T11:01:08.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe8c58700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe406b280 con 0x7ffbe40cc590 2023-10-16T11:01:08.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe8c58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40cc1c0 con 0x7ffbe415da40 2023-10-16T11:01:08.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffbe40cc590 0x7ffbe406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe2d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffbe406ae10 0x7ffbe40ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34400/0 (socket says 172.21.15.158:34400) 2023-10-16T11:01:08.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffbe40cc590 0x7ffbe406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34222/0 (socket says 172.21.15.158:34222) 2023-10-16T11:01:08.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.343+0000 7ffbe2d9d700 1 -- 172.21.15.158:0/159757707 learned_addr learned my addr 172.21.15.158:0/159757707 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:08.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe1d9b700 1 -- 172.21.15.158:0/159757707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 msgr2=0x7ffbe40caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:08.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe259c700 1 --2- 172.21.15.158:0/159757707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe1d9b700 1 --2- 172.21.15.158:0/159757707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe40caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe1d9b700 1 -- 172.21.15.158:0/159757707 >> v1:172.21.15.158:6789/0 conn(0x7ffbe406ae10 legacy=0x7ffbe40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe1d9b700 1 -- 172.21.15.158:0/159757707 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe415de20 con 0x7ffbe40cc590 2023-10-16T11:01:08.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe159a700 1 -- 172.21.15.158:0/159757707 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420515849 0 0) 0x7ffbe40cbd30 con 0x7ffbe406ae10 2023-10-16T11:01:08.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe1d9b700 1 --2- 172.21.15.158:0/159757707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe406a8d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4009840 tx=0x7ffbd4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e4e0cf4936a5c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:08.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe159a700 1 -- 172.21.15.158:0/159757707 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4011830 con 0x7ffbe40cc590 2023-10-16T11:01:08.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe159a700 1 -- 172.21.15.158:0/159757707 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd4011990 con 0x7ffbe40cc590 2023-10-16T11:01:08.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.344+0000 7ffbe159a700 1 -- 172.21.15.158:0/159757707 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4011d20 con 0x7ffbe40cc590 2023-10-16T11:01:08.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 -- 172.21.15.158:0/159757707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 msgr2=0x7ffbe406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/159757707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe406a8d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4009840 tx=0x7ffbd4009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:08.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 -- 172.21.15.158:0/159757707 shutdown_connections 2023-10-16T11:01:08.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/159757707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe40caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/159757707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe406a8d0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 -- 172.21.15.158:0/159757707 >> 172.21.15.158:0/159757707 conn(0x7ffbe4099a10 msgr2=0x7ffbe40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:08.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 -- 172.21.15.158:0/159757707 shutdown_connections 2023-10-16T11:01:08.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.345+0000 7ffbe8c58700 1 -- 172.21.15.158:0/159757707 wait complete. 2023-10-16T11:01:08.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.346+0000 7ffbe8c58700 1 Processor -- start 2023-10-16T11:01:08.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.346+0000 7ffbe8c58700 1 -- start start 2023-10-16T11:01:08.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe8c58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe406ae10 0x7ffbe41e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe8c58700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe41f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe8c58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe41e5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe8c58700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe41f7480 con 0x7ffbe40cc590 2023-10-16T11:01:08.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe8c58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40c5f20 con 0x7ffbe406ae10 2023-10-16T11:01:08.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe8c58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40c6200 con 0x7ffbe415da40 2023-10-16T11:01:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe41f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe41f6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34228/0 (socket says 172.21.15.158:34228) 2023-10-16T11:01:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 -- 172.21.15.158:0/246577293 learned_addr learned my addr 172.21.15.158:0/246577293 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe2d9d700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe41e5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe259c700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe406ae10 0x7ffbe41e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 -- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 msgr2=0x7ffbe41e5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe41e5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 -- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe406ae10 msgr2=0x7ffbe41e4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe406ae10 0x7ffbe41e4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 -- 172.21.15.158:0/246577293 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40094a0 con 0x7ffbe40cc590 2023-10-16T11:01:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe2d9d700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe41e5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbe1d9b700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe41f6f40 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40090b0 tx=0x7ffbd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbd2ffd700 1 -- 172.21.15.158:0/246577293 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40038a0 con 0x7ffbe40cc590 2023-10-16T11:01:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.347+0000 7ffbd2ffd700 1 -- 172.21.15.158:0/246577293 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd401f070 con 0x7ffbe40cc590 2023-10-16T11:01:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.348+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe40c64e0 con 0x7ffbe40cc590 2023-10-16T11:01:08.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.348+0000 7ffbd2ffd700 1 -- 172.21.15.158:0/246577293 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4003c40 con 0x7ffbe40cc590 2023-10-16T11:01:08.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.348+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe41e66c0 con 0x7ffbe40cc590 2023-10-16T11:01:08.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.348+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe40c6a00 con 0x7ffbe40cc590 2023-10-16T11:01:08.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.349+0000 7ffbd2ffd700 1 -- 172.21.15.158:0/246577293 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 106) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ffbd4026480 con 0x7ffbe40cc590 2023-10-16T11:01:08.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.349+0000 7ffbd2ffd700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ffbc406af90 0x7ffbc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.349+0000 7ffbd2ffd700 1 -- 172.21.15.158:0/246577293 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7ffbd4021070 con 0x7ffbe40cc590 2023-10-16T11:01:08.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.349+0000 7ffbe259c700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ffbc406af90 0x7ffbc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.350+0000 7ffbe259c700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ffbc406af90 0x7ffbc406d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800a950 tx=0x7ffbd8008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:08.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.354+0000 7ffbd2ffd700 1 -- 172.21.15.158:0/246577293 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffbe40c6a00 con 0x7ffbe40cc590 2023-10-16T11:01:08.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.552+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7ffbe40c6a00 con 0x7ffbe40cc590 2023-10-16T11:01:08.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.555+0000 7ffbd2ffd700 1 -- 172.21.15.158:0/246577293 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7ffbe40c6a00 con 0x7ffbe40cc590 2023-10-16T11:01:08.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ffbc406af90 msgr2=0x7ffbc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ffbc406af90 0x7ffbc406d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800a950 tx=0x7ffbd8008040 comp rx=0 tx=0).stop 2023-10-16T11:01:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 msgr2=0x7ffbe41f6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe41f6f40 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40090b0 tx=0x7ffbd4004750 comp rx=0 tx=0).stop 2023-10-16T11:01:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 shutdown_connections 2023-10-16T11:01:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ffbc406af90 0x7ffbc406d450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe415da40 0x7ffbe41e5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe406ae10 0x7ffbe41e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 --2- 172.21.15.158:0/246577293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffbe40cc590 0x7ffbe41f6f40 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 >> 172.21.15.158:0/246577293 conn(0x7ffbe4099a10 msgr2=0x7ffbe40b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 shutdown_connections 2023-10-16T11:01:08.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.558+0000 7ffbe8c58700 1 -- 172.21.15.158:0/246577293 wait complete. 2023-10-16T11:01:08.573 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:01:08.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.694+0000 7f783095a700 1 Processor -- start 2023-10-16T11:01:08.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.694+0000 7f783095a700 1 -- start start 2023-10-16T11:01:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f783095a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f782c160d40 0x7f782c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f783095a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0ce110 con 0x7f782c0c50c0 2023-10-16T11:01:08.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f783095a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0ce3f0 con 0x7f782c0cf420 2023-10-16T11:01:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f783095a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782c0c61e0 con 0x7f782c160d40 2023-10-16T11:01:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f782a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f782c160d40 0x7f782c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f782ad9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f782c0c50c0 0x7f782c06abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34412/0 (socket says 172.21.15.158:34412) 2023-10-16T11:01:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f782ad9d700 1 -- 172.21.15.158:0/1591587766 learned_addr learned my addr 172.21.15.158:0/1591587766 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f782a59c700 1 -- 172.21.15.158:0/1591587766 >> v1:172.21.15.172:6789/0 conn(0x7f782c0cf420 legacy=0x7f782c0c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f782a59c700 1 -- 172.21.15.158:0/1591587766 >> v1:172.21.15.158:6789/0 conn(0x7f782c0c50c0 legacy=0x7f782c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.695+0000 7f782a59c700 1 -- 172.21.15.158:0/1591587766 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c0ce6d0 con 0x7f782c160d40 2023-10-16T11:01:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f782a59c700 1 --2- 172.21.15.158:0/1591587766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c160d40 0x7f782c0ceee0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f7820009840 tx=0x7f7820009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505e61e50761d43b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f782959a700 1 -- 172.21.15.158:0/1591587766 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820011830 con 0x7f782c160d40 2023-10-16T11:01:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f782959a700 1 -- 172.21.15.158:0/1591587766 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7820011990 con 0x7f782c160d40 2023-10-16T11:01:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f782959a700 1 -- 172.21.15.158:0/1591587766 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820011d20 con 0x7f782c160d40 2023-10-16T11:01:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f783095a700 1 -- 172.21.15.158:0/1591587766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c160d40 msgr2=0x7f782c0ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f783095a700 1 --2- 172.21.15.158:0/1591587766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c160d40 0x7f782c0ceee0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f7820009840 tx=0x7f7820009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:08.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f783095a700 1 -- 172.21.15.158:0/1591587766 shutdown_connections 2023-10-16T11:01:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f783095a700 1 --2- 172.21.15.158:0/1591587766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c160d40 0x7f782c0ceee0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f783095a700 1 -- 172.21.15.158:0/1591587766 >> 172.21.15.158:0/1591587766 conn(0x7f782c0cc5b0 msgr2=0x7f782c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f783095a700 1 -- 172.21.15.158:0/1591587766 shutdown_connections 2023-10-16T11:01:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.696+0000 7f783095a700 1 -- 172.21.15.158:0/1591587766 wait complete. 2023-10-16T11:01:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 Processor -- start 2023-10-16T11:01:08.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 -- start start 2023-10-16T11:01:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f782c0c50c0 0x7f782c0c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c0cf420 0x7f782c0c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f782c160d40 0x7f782c1f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c1523c0 con 0x7f782c0cf420 2023-10-16T11:01:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c1526a0 con 0x7f782c0c50c0 2023-10-16T11:01:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.698+0000 7f783095a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c152980 con 0x7f782c160d40 2023-10-16T11:01:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c0cf420 0x7f782c0c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c0cf420 0x7f782c0c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34240/0 (socket says 172.21.15.158:34240) 2023-10-16T11:01:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 -- 172.21.15.158:0/2569402376 learned_addr learned my addr 172.21.15.158:0/2569402376 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f782ad9d700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f782c160d40 0x7f782c1f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f782a59c700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f782c0c50c0 0x7f782c0c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 -- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f782c160d40 msgr2=0x7f782c1f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f782c160d40 0x7f782c1f63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 -- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f782c0c50c0 msgr2=0x7f782c0c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f782c0c50c0 0x7f782c0c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 -- 172.21.15.158:0/2569402376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78200094a0 con 0x7f782c0cf420 2023-10-16T11:01:08.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f7829d9b700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c0cf420 0x7f782c0c8cd0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f781c00c4e0 tx=0x7f781c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:08.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f781affd700 1 -- 172.21.15.158:0/2569402376 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c0052d0 con 0x7f782c0cf420 2023-10-16T11:01:08.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f781affd700 1 -- 172.21.15.158:0/2569402376 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c005430 con 0x7f782c0cf420 2023-10-16T11:01:08.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c1612a0 con 0x7f782c0cf420 2023-10-16T11:01:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f781affd700 1 -- 172.21.15.158:0/2569402376 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c0038c0 con 0x7f782c0cf420 2023-10-16T11:01:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.699+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c152f40 con 0x7f782c0cf420 2023-10-16T11:01:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.700+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c14ffc0 con 0x7f782c0cf420 2023-10-16T11:01:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.701+0000 7f781affd700 1 -- 172.21.15.158:0/2569402376 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 106) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f781c003a20 con 0x7f782c0cf420 2023-10-16T11:01:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.701+0000 7f781affd700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f780c06af90 0x7f780c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.701+0000 7f781affd700 1 -- 172.21.15.158:0/2569402376 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f781c088d00 con 0x7f782c0cf420 2023-10-16T11:01:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.701+0000 7f782a59c700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f780c06af90 0x7f780c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.701+0000 7f782a59c700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f780c06af90 0x7f780c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78200090b0 tx=0x7f7820008030 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.705+0000 7f781affd700 1 -- 172.21.15.158:0/2569402376 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f782c14ffc0 con 0x7f782c0cf420 2023-10-16T11:01:08.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.899+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f782c14ffc0 con 0x7f782c0cf420 2023-10-16T11:01:08.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.902+0000 7f781affd700 1 -- 172.21.15.158:0/2569402376 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f782c14ffc0 con 0x7f782c0cf420 2023-10-16T11:01:08.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f780c06af90 msgr2=0x7f780c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f780c06af90 0x7f780c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78200090b0 tx=0x7f7820008030 comp rx=0 tx=0).stop 2023-10-16T11:01:08.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c0cf420 msgr2=0x7f782c0c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:08.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c0cf420 0x7f782c0c8cd0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f781c00c4e0 tx=0x7f781c00c8a0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 shutdown_connections 2023-10-16T11:01:08.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f780c06af90 0x7f780c06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f782c160d40 0x7f782c1f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f782c0c50c0 0x7f782c0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 --2- 172.21.15.158:0/2569402376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f782c0cf420 0x7f782c0c8cd0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:08.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 >> 172.21.15.158:0/2569402376 conn(0x7f782c0cc5b0 msgr2=0x7f782c071180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:08.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 shutdown_connections 2023-10-16T11:01:08.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:08.904+0000 7f783095a700 1 -- 172.21.15.158:0/2569402376 wait complete. 2023-10-16T11:01:08.918 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:01:09.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.038+0000 7efecac85700 1 Processor -- start 2023-10-16T11:01:09.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.038+0000 7efecac85700 1 -- start start 2023-10-16T11:01:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efecac85700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec415dad0 0x7efec40cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efecac85700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efec415deb0 0x7efec40ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efecac85700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec40d0ae0 con 0x7efec40cea20 2023-10-16T11:01:09.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efecac85700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec40d12d0 con 0x7efec415deb0 2023-10-16T11:01:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efecac85700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec40cbdf0 con 0x7efec415dad0 2023-10-16T11:01:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efec3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efec415deb0 0x7efec40ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efec3fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efec415deb0 0x7efec40ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34248/0 (socket says 172.21.15.158:34248) 2023-10-16T11:01:09.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.039+0000 7efec3fff700 1 -- 172.21.15.158:0/3148950006 learned_addr learned my addr 172.21.15.158:0/3148950006 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec8a21700 1 --2- 172.21.15.158:0/3148950006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec415dad0 0x7efec40cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec3fff700 1 -- 172.21.15.158:0/3148950006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec415dad0 msgr2=0x7efec40cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec3fff700 1 --2- 172.21.15.158:0/3148950006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec415dad0 0x7efec40cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec3fff700 1 -- 172.21.15.158:0/3148950006 >> v1:172.21.15.172:6789/0 conn(0x7efec40cea20 legacy=0x7efec40c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec3fff700 1 -- 172.21.15.158:0/3148950006 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec40cc280 con 0x7efec415deb0 2023-10-16T11:01:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec3fff700 1 --2- 172.21.15.158:0/3148950006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec40ce4e0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7efeb00098f0 tx=0x7efeb0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde96cdf98338cde server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec37fe700 1 -- 172.21.15.158:0/3148950006 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0004230 con 0x7efec415deb0 2023-10-16T11:01:09.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec37fe700 1 -- 172.21.15.158:0/3148950006 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb0004390 con 0x7efec415deb0 2023-10-16T11:01:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efec37fe700 1 -- 172.21.15.158:0/3148950006 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00036a0 con 0x7efec415deb0 2023-10-16T11:01:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efecac85700 1 -- 172.21.15.158:0/3148950006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 msgr2=0x7efec40ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.040+0000 7efecac85700 1 --2- 172.21.15.158:0/3148950006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec40ce4e0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7efeb00098f0 tx=0x7efeb0009c00 comp rx=0 tx=0).stop 2023-10-16T11:01:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.041+0000 7efecac85700 1 -- 172.21.15.158:0/3148950006 shutdown_connections 2023-10-16T11:01:09.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.041+0000 7efecac85700 1 --2- 172.21.15.158:0/3148950006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec415dad0 0x7efec40cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.041+0000 7efecac85700 1 --2- 172.21.15.158:0/3148950006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec40ce4e0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.041+0000 7efecac85700 1 -- 172.21.15.158:0/3148950006 >> 172.21.15.158:0/3148950006 conn(0x7efec4099a10 msgr2=0x7efec40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:09.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.041+0000 7efecac85700 1 -- 172.21.15.158:0/3148950006 shutdown_connections 2023-10-16T11:01:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.041+0000 7efecac85700 1 -- 172.21.15.158:0/3148950006 wait complete. 2023-10-16T11:01:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.042+0000 7efecac85700 1 Processor -- start 2023-10-16T11:01:09.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.042+0000 7efecac85700 1 -- start start 2023-10-16T11:01:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efecac85700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec40cea20 0x7efec40b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efecac85700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efec415dad0 0x7efec40c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efecac85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec406ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efecac85700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec4149ac0 con 0x7efec415deb0 2023-10-16T11:01:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efecac85700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec4148340 con 0x7efec415dad0 2023-10-16T11:01:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efecac85700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec40b3530 con 0x7efec40cea20 2023-10-16T11:01:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efec9222700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec406ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efec9222700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec406ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34256/0 (socket says 172.21.15.158:34256) 2023-10-16T11:01:09.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.043+0000 7efec9222700 1 -- 172.21.15.158:0/3897337594 learned_addr learned my addr 172.21.15.158:0/3897337594 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec3fff700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efec415dad0 0x7efec40c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec8a21700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec40cea20 0x7efec40b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec9222700 1 -- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec40cea20 msgr2=0x7efec40b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec9222700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec40cea20 0x7efec40b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec9222700 1 -- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efec415dad0 msgr2=0x7efec40c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec9222700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efec415dad0 0x7efec40c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec9222700 1 -- 172.21.15.158:0/3897337594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0009550 con 0x7efec415deb0 2023-10-16T11:01:09.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec3fff700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efec415dad0 0x7efec40c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:09.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec9222700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec406ee10 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7efebc00cc40 tx=0x7efebc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:09.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc00aba0 con 0x7efec415deb0 2023-10-16T11:01:09.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc00ad00 con 0x7efec415deb0 2023-10-16T11:01:09.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec40b37d0 con 0x7efec415deb0 2023-10-16T11:01:09.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc012660 con 0x7efec415deb0 2023-10-16T11:01:09.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.044+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec41485b0 con 0x7efec415deb0 2023-10-16T11:01:09.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.045+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec41487a0 con 0x7efec415deb0 2023-10-16T11:01:09.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.045+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 106) v1 ==== 89275+0+0 (secure 0 0 0) 0x7efebc0127c0 con 0x7efec415deb0 2023-10-16T11:01:09.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.046+0000 7efec17fa700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efea806af90 0x7efea806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.046+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7efebc089fa0 con 0x7efec415deb0 2023-10-16T11:01:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.046+0000 7efec8a21700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efea806af90 0x7efea806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.046+0000 7efec8a21700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efea806af90 0x7efea806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009930 tx=0x7efeb8008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.050+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efebc054e30 con 0x7efec415deb0 2023-10-16T11:01:09.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.200+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efebc055200 con 0x7efec415deb0 2023-10-16T11:01:09.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.248+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7efec40681b0 con 0x7efec415deb0 2023-10-16T11:01:09.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.251+0000 7efec17fa700 1 -- 172.21.15.158:0/3897337594 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7efebc073e40 con 0x7efec415deb0 2023-10-16T11:01:09.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.253+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efea806af90 msgr2=0x7efea806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.253+0000 7efecac85700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efea806af90 0x7efea806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009930 tx=0x7efeb8008040 comp rx=0 tx=0).stop 2023-10-16T11:01:09.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.253+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 msgr2=0x7efec406ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.253+0000 7efecac85700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec406ee10 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7efebc00cc40 tx=0x7efebc00a060 comp rx=0 tx=0).stop 2023-10-16T11:01:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.253+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 shutdown_connections 2023-10-16T11:01:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.254+0000 7efecac85700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efea806af90 0x7efea806d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.254+0000 7efecac85700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efec415deb0 0x7efec406ee10 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.254+0000 7efecac85700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efec40cea20 0x7efec40b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.254+0000 7efecac85700 1 --2- 172.21.15.158:0/3897337594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efec415dad0 0x7efec40c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.254+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 >> 172.21.15.158:0/3897337594 conn(0x7efec4099a10 msgr2=0x7efec4156580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.254+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 shutdown_connections 2023-10-16T11:01:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.254+0000 7efecac85700 1 -- 172.21.15.158:0/3897337594 wait complete. 2023-10-16T11:01:09.269 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T11:01:09.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.388+0000 7fd402887700 1 Processor -- start 2023-10-16T11:01:09.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.388+0000 7fd402887700 1 -- start start 2023-10-16T11:01:09.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd402887700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 0x7fd3fc161270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd402887700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc15de20 con 0x7fd3fc06bd10 2023-10-16T11:01:09.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd402887700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc162180 con 0x7fd3fc0cf4c0 2023-10-16T11:01:09.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd402887700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc1618e0 con 0x7fd3fc0cf8a0 2023-10-16T11:01:09.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd400e24700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd3fc06bd10 0x7fd3fc15e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34440/0 (socket says 172.21.15.158:34440) 2023-10-16T11:01:09.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd400e24700 1 -- 172.21.15.158:0/990849293 learned_addr learned my addr 172.21.15.158:0/990849293 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:09.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd3fb7fe700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 0x7fd3fc161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037632142 0 0) 0x7fd3fc15de20 con 0x7fd3fc06bd10 2023-10-16T11:01:09.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.389+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e40035b0 con 0x7fd3fc06bd10 2023-10-16T11:01:09.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446969606 0 0) 0x7fd3e40035b0 con 0x7fd3fc06bd10 2023-10-16T11:01:09.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 msgr2=0x7fd3fc161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:09.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 0x7fd3fc161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 >> v1:172.21.15.172:6789/0 conn(0x7fd3fc0cf4c0 legacy=0x7fd3fc06d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc161bc0 con 0x7fd3fc06bd10 2023-10-16T11:01:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd3f40019f0 con 0x7fd3fc06bd10 2023-10-16T11:01:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 >> v1:172.21.15.158:6789/0 conn(0x7fd3fc06bd10 legacy=0x7fd3fc15e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 0x7fd3e4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e4003fb0 0x7fd3e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4004d90 0x7fd3e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc1618e0 con 0x7fd3e4003fb0 2023-10-16T11:01:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e400bc50 con 0x7fd3e4004d90 2023-10-16T11:01:09.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e400be10 con 0x7fd3fc0cf8a0 2023-10-16T11:01:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fb7fe700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e4003fb0 0x7fd3e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fbfff700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 0x7fd3e4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd400e24700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4004d90 0x7fd3e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fb7fe700 1 -- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 msgr2=0x7fd3e4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:09.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fb7fe700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc0cf8a0 0x7fd3e4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fb7fe700 1 -- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4004d90 msgr2=0x7fd3e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:09.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fb7fe700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4004d90 0x7fd3e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fb7fe700 1 -- 172.21.15.158:0/990849293 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e40035b0 con 0x7fd3e4003fb0 2023-10-16T11:01:09.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd400e24700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4004d90 0x7fd3e4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:09.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3fb7fe700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e4003fb0 0x7fd3e4004850 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009b60 tx=0x7fd3e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a1e4c600df383e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00e6c0 con 0x7fd3e4003fb0 2023-10-16T11:01:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec00e820 con 0x7fd3e4003fb0 2023-10-16T11:01:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.390+0000 7fd3faffd700 1 -- 172.21.15.158:0/990849293 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00eb90 con 0x7fd3e4003fb0 2023-10-16T11:01:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.391+0000 7fd402887700 1 -- 172.21.15.158:0/990849293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e4003fb0 msgr2=0x7fd3e4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.391+0000 7fd402887700 1 --2- 172.21.15.158:0/990849293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e4003fb0 0x7fd3e4004850 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009b60 tx=0x7fd3e4005880 comp rx=0 tx=0).stop 2023-10-16T11:01:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.391+0000 7fd400e24700 1 -- 172.21.15.158:0/990849293 reap_dead start 2023-10-16T11:01:09.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.391+0000 7fd402887700 1 -- 172.21.15.158:0/990849293 shutdown_connections 2023-10-16T11:01:09.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.391+0000 7fd402887700 1 -- 172.21.15.158:0/990849293 >> 172.21.15.158:0/990849293 conn(0x7fd3fc06a8d0 msgr2=0x7fd3fc06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:09.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.391+0000 7fd402887700 1 -- 172.21.15.158:0/990849293 shutdown_connections 2023-10-16T11:01:09.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.391+0000 7fd402887700 1 -- 172.21.15.158:0/990849293 wait complete. 2023-10-16T11:01:09.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.392+0000 7fd402887700 1 Processor -- start 2023-10-16T11:01:09.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.392+0000 7fd402887700 1 -- start start 2023-10-16T11:01:09.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd402887700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4003fb0 0x7fd3fc152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd402887700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc1603f0 0x7fd3fc0c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd402887700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3fc0c4f80 0x7fd3fc0c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd402887700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0b60f0 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd402887700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc159760 con 0x7fd3e4003fb0 2023-10-16T11:01:09.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd402887700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0c7f70 con 0x7fd3fc1603f0 2023-10-16T11:01:09.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd400e24700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3fc0c4f80 0x7fd3fc0c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd400e24700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3fc0c4f80 0x7fd3fc0c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34276/0 (socket says 172.21.15.158:34276) 2023-10-16T11:01:09.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.393+0000 7fd400e24700 1 -- 172.21.15.158:0/2127391342 learned_addr learned my addr 172.21.15.158:0/2127391342 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:09.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd400e24700 1 -- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc1603f0 msgr2=0x7fd3fc0c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd3fb7fe700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc1603f0 0x7fd3fc0c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd400e24700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc1603f0 0x7fd3fc0c74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd3fbfff700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4003fb0 0x7fd3fc152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd400e24700 1 -- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4003fb0 msgr2=0x7fd3fc152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:09.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd400e24700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4003fb0 0x7fd3fc152010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:09.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd400e24700 1 -- 172.21.15.158:0/2127391342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc0cd450 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd400e24700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3fc0c4f80 0x7fd3fc0c7a30 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc15b850 tx=0x7fd3f400e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:09.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd3f8ff9700 1 -- 172.21.15.158:0/2127391342 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4017070 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd3f8ff9700 1 -- 172.21.15.158:0/2127391342 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f400ee00 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc0b6340 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd3f8ff9700 1 -- 172.21.15.158:0/2127391342 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4013420 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.394+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc0b6900 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.395+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc0681b0 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.395+0000 7fd3f8ff9700 1 -- 172.21.15.158:0/2127391342 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd3f4013580 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.396+0000 7fd3f8ff9700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd3e806b020 0x7fd3e806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.396+0000 7fd3fbfff700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd3e806b020 0x7fd3e806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.396+0000 7fd3f8ff9700 1 -- 172.21.15.158:0/2127391342 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fd3f4089880 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.396+0000 7fd3fbfff700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd3e806b020 0x7fd3e806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0007fd0 tx=0x7fd3f0009040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:09.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.400+0000 7fd3f8ff9700 1 -- 172.21.15.158:0/2127391342 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3f40544d0 con 0x7fd3fc0c4f80 2023-10-16T11:01:09.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:09.675+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fd3fc1598a0 con 0x7fd3fc0c4f80 2023-10-16T11:01:10.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.204+0000 7fd3f8ff9700 1 -- 172.21.15.158:0/2127391342 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2536) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd3f4057af0 con 0x7fd3fc0c4f80 2023-10-16T11:01:10.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.206+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd3e806b020 msgr2=0x7fd3e806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.206+0000 7fd402887700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd3e806b020 0x7fd3e806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0007fd0 tx=0x7fd3f0009040 comp rx=0 tx=0).stop 2023-10-16T11:01:10.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.206+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3fc0c4f80 msgr2=0x7fd3fc0c7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.206+0000 7fd402887700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3fc0c4f80 0x7fd3fc0c7a30 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc15b850 tx=0x7fd3f400e4f0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 shutdown_connections 2023-10-16T11:01:10.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd3e806b020 0x7fd3e806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3fc0c4f80 0x7fd3fc0c7a30 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3e4003fb0 0x7fd3fc152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 --2- 172.21.15.158:0/2127391342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3fc1603f0 0x7fd3fc0c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 >> 172.21.15.158:0/2127391342 conn(0x7fd3fc06a8d0 msgr2=0x7fd3fc15b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:10.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 shutdown_connections 2023-10-16T11:01:10.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.207+0000 7fd402887700 1 -- 172.21.15.158:0/2127391342 wait complete. 2023-10-16T11:01:10.222 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-10-16T11:01:10.222 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-10-16T11:01:10.223 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T11:01:10.223 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:10.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.341+0000 7f1f680e2700 1 Processor -- start 2023-10-16T11:01:10.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.342+0000 7f1f680e2700 1 -- start start 2023-10-16T11:01:10.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.342+0000 7f1f680e2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f600b6e10 0x7f1f601521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.342+0000 7f1f680e2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600ce2c0 0x7f1f601526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.342+0000 7f1f680e2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6015c5d0 con 0x7f1f600cdee0 2023-10-16T11:01:10.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.342+0000 7f1f680e2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f6015d2c0 con 0x7f1f600b6e10 2023-10-16T11:01:10.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.342+0000 7f1f680e2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f60152c20 con 0x7f1f600ce2c0 2023-10-16T11:01:10.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600ce2c0 0x7f1f601526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6567d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f600b6e10 0x7f1f601521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600ce2c0 0x7f1f601526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:45118/0 (socket says 172.21.15.158:45118) 2023-10-16T11:01:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6567d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f600b6e10 0x7f1f601521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:35462/0 (socket says 172.21.15.158:35462) 2023-10-16T11:01:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 -- 172.21.15.158:0/1540703816 learned_addr learned my addr 172.21.15.158:0/1540703816 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f65e7e700 1 --1- 172.21.15.158:0/1540703816 >> v1:172.21.15.172:6789/0 conn(0x7f1f600cdee0 0x7f1f6015bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:38360/0 (socket says 172.21.15.158:38360) 2023-10-16T11:01:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 -- 172.21.15.158:0/1540703816 >> v2:172.21.15.172:3300/0 conn(0x7f1f600b6e10 msgr2=0x7f1f601521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:10.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 --2- 172.21.15.158:0/1540703816 >> v2:172.21.15.172:3300/0 conn(0x7f1f600b6e10 0x7f1f601521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 -- 172.21.15.158:0/1540703816 >> v1:172.21.15.172:6789/0 conn(0x7f1f600cdee0 legacy=0x7f1f6015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 -- 172.21.15.158:0/1540703816 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6015c8b0 con 0x7f1f600ce2c0 2023-10-16T11:01:10.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6567d700 1 --2- 172.21.15.158:0/1540703816 >> v2:172.21.15.172:3300/0 conn(0x7f1f600b6e10 0x7f1f601521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:10.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f6667f700 1 --2- 172.21.15.158:0/1540703816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600ce2c0 0x7f1f601526e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009e50 tx=0x7f1f5800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa9dfa0aa8a062bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:10.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.343+0000 7f1f64e7c700 1 -- 172.21.15.158:0/1540703816 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5800e070 con 0x7f1f600ce2c0 2023-10-16T11:01:10.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f64e7c700 1 -- 172.21.15.158:0/1540703816 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5800b910 con 0x7f1f600ce2c0 2023-10-16T11:01:10.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f64e7c700 1 -- 172.21.15.158:0/1540703816 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5800bcc0 con 0x7f1f600ce2c0 2023-10-16T11:01:10.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 -- 172.21.15.158:0/1540703816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600ce2c0 msgr2=0x7f1f601526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1540703816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600ce2c0 0x7f1f601526e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009e50 tx=0x7f1f5800b330 comp rx=0 tx=0).stop 2023-10-16T11:01:10.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 -- 172.21.15.158:0/1540703816 shutdown_connections 2023-10-16T11:01:10.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1540703816 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600ce2c0 0x7f1f601526e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1540703816 >> v2:172.21.15.172:3300/0 conn(0x7f1f600b6e10 0x7f1f601521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 -- 172.21.15.158:0/1540703816 >> 172.21.15.158:0/1540703816 conn(0x7f1f6006a8d0 msgr2=0x7f1f6006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:10.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 -- 172.21.15.158:0/1540703816 shutdown_connections 2023-10-16T11:01:10.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.344+0000 7f1f680e2700 1 -- 172.21.15.158:0/1540703816 wait complete. 2023-10-16T11:01:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.345+0000 7f1f680e2700 1 Processor -- start 2023-10-16T11:01:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.345+0000 7f1f680e2700 1 -- start start 2023-10-16T11:01:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f680e2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f600b6e10 0x7f1f6013cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f680e2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cdee0 0x7f1f600ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f680e2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600ce2c0 0x7f1f6015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f680e2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f60140ff0 con 0x7f1f600ce2c0 2023-10-16T11:01:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f680e2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6013d280 con 0x7f1f600b6e10 2023-10-16T11:01:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f680e2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6013e0d0 con 0x7f1f600cdee0 2023-10-16T11:01:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6667f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600ce2c0 0x7f1f6015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6667f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600ce2c0 0x7f1f6015a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34290/0 (socket says 172.21.15.158:34290) 2023-10-16T11:01:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6667f700 1 -- 172.21.15.158:0/1160053215 learned_addr learned my addr 172.21.15.158:0/1160053215 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f65e7e700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f600b6e10 0x7f1f6013cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6567d700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cdee0 0x7f1f600ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6667f700 1 -- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cdee0 msgr2=0x7f1f600ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6667f700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cdee0 0x7f1f600ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6667f700 1 -- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f600b6e10 msgr2=0x7f1f6013cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.346+0000 7f1f6667f700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f600b6e10 0x7f1f6013cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.347+0000 7f1f6667f700 1 -- 172.21.15.158:0/1160053215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c0094a0 con 0x7f1f600ce2c0 2023-10-16T11:01:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.347+0000 7f1f6667f700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600ce2c0 0x7f1f6015a730 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f1f58008e50 tx=0x7f1f58004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:10.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.347+0000 7f1f567fc700 1 -- 172.21.15.158:0/1160053215 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5801b630 con 0x7f1f600ce2c0 2023-10-16T11:01:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.347+0000 7f1f567fc700 1 -- 172.21.15.158:0/1160053215 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58007080 con 0x7f1f600ce2c0 2023-10-16T11:01:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.347+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f58009b30 con 0x7f1f600ce2c0 2023-10-16T11:01:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.347+0000 7f1f567fc700 1 -- 172.21.15.158:0/1160053215 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5801b9d0 con 0x7f1f600ce2c0 2023-10-16T11:01:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.347+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6013d7a0 con 0x7f1f600ce2c0 2023-10-16T11:01:10.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.348+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f600d0b50 con 0x7f1f600ce2c0 2023-10-16T11:01:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.348+0000 7f1f567fc700 1 -- 172.21.15.158:0/1160053215 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f5801bb30 con 0x7f1f600ce2c0 2023-10-16T11:01:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.348+0000 7f1f567fc700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.349+0000 7f1f567fc700 1 -- 172.21.15.158:0/1160053215 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f1f5808b390 con 0x7f1f600ce2c0 2023-10-16T11:01:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.349+0000 7f1f65e7e700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.349+0000 7f1f65e7e700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f1f4406b020 0x7f1f4406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f50000f60 tx=0x7f1f50007f10 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:10.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.353+0000 7f1f567fc700 1 -- 172.21.15.158:0/1160053215 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f58055f70 con 0x7f1f600ce2c0 2023-10-16T11:01:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.550+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f600681b0 con 0x7f1f600ce2c0 2023-10-16T11:01:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.551+0000 7f1f567fc700 1 -- 172.21.15.158:0/1160053215 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+14921 (secure 0 0 0) 0x7f1f58059590 con 0x7f1f600ce2c0 2023-10-16T11:01:10.553 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:10.553 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":177,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:01:08.192680+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6833/551179403":"2023-10-17T11:01:05.128296+0000","172.21.15.158:0/3738809292":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/2918364282":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/1160958831":"2023-10-17T11:01:04.175999+0000","172.21.15.158:6833/4231823616":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/2347685989":"2023-10-17T11:01:04.175999+0000","172.21.15.158:6832/4231823616":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/982917811":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3695355538":"2023-10-17T11:00:55.616726+0000","172.21.15.172:6835/551179403":"2023-10-17T11:01:05.128296+0000","172.21.15.158:0/2070324944":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/1259583827":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3480878312":"2023-10-17T11:01:04.175999+0000","172.21.15.158:0/915803440":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/3743813404":"2023-10-17T11:00:55.616726+0000","172.21.15.158:6833/1293733893":"2023-10-17T11:00:55.616726+0000","172.21.15.158:0/2282704909":"2023-10-17T11:01:04.175999+0000","172.21.15.158:6832/1293733893":"2023-10-17T11:00:55.616726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:10.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f1f4406b020 msgr2=0x7f1f4406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f1f4406b020 0x7f1f4406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f50000f60 tx=0x7f1f50007f10 comp rx=0 tx=0).stop 2023-10-16T11:01:10.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600ce2c0 msgr2=0x7f1f6015a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600ce2c0 0x7f1f6015a730 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f1f58008e50 tx=0x7f1f58004ad0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 shutdown_connections 2023-10-16T11:01:10.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600ce2c0 0x7f1f6015a730 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f600b6e10 0x7f1f6013cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 --2- 172.21.15.158:0/1160053215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cdee0 0x7f1f600ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 >> 172.21.15.158:0/1160053215 conn(0x7f1f6006a8d0 msgr2=0x7f1f600b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:10.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 shutdown_connections 2023-10-16T11:01:10.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.553+0000 7f1f680e2700 1 -- 172.21.15.158:0/1160053215 wait complete. 2023-10-16T11:01:10.567 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:01:10.568 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:01:10.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.689+0000 7f4ded4d6700 1 Processor -- start 2023-10-16T11:01:10.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.689+0000 7f4ded4d6700 1 -- start start 2023-10-16T11:01:10.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4ded4d6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4de80b0c00 0x7f4de80b1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4ded4d6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de80b1590 con 0x7f4de80ca8b0 2023-10-16T11:01:10.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4ded4d6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de80cd3d0 con 0x7f4de8159af0 2023-10-16T11:01:10.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4ded4d6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de815e560 con 0x7f4de80b0c00 2023-10-16T11:01:10.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de67fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4de80ca8b0 0x7f4de815de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34472/0 (socket says 172.21.15.158:34472) 2023-10-16T11:01:10.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de67fc700 1 -- 172.21.15.158:0/3345763018 learned_addr learned my addr 172.21.15.158:0/3345763018 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:10.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de77fe700 1 --2- 172.21.15.158:0/3345763018 >> v2:172.21.15.172:3300/0 conn(0x7f4de80b0c00 0x7f4de80b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439010124 0 0) 0x7f4dd8001110 con 0x7f4de80ca8b0 2023-10-16T11:01:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd00035b0 con 0x7f4de80ca8b0 2023-10-16T11:01:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1489381900 0 0) 0x7f4dd00035b0 con 0x7f4de80ca8b0 2023-10-16T11:01:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 >> v2:172.21.15.172:3300/0 conn(0x7f4de80b0c00 msgr2=0x7f4de80b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:10.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 --2- 172.21.15.158:0/3345763018 >> v2:172.21.15.172:3300/0 conn(0x7f4de80b0c00 0x7f4de80b1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 >> v1:172.21.15.172:6789/0 conn(0x7f4de8159af0 legacy=0x7f4de80ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de80cdf50 con 0x7f4de80ca8b0 2023-10-16T11:01:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4dd8001800 con 0x7f4de80ca8b0 2023-10-16T11:01:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 >> v1:172.21.15.158:6789/0 conn(0x7f4de80ca8b0 legacy=0x7f4de815de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dd0003a00 0x7f4dd0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd0004850 0x7f4dd0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0005210 0x7f4dd000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd000c080 con 0x7f4dd0003a00 2023-10-16T11:01:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd000c1f0 con 0x7f4dd0004850 2023-10-16T11:01:10.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.690+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd000e5c0 con 0x7f4dd0005210 2023-10-16T11:01:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de6ffd700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dd0003a00 0x7f4dd0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de67fc700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd0004850 0x7f4dd0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de77fe700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0005210 0x7f4dd000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de6ffd700 1 -- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0005210 msgr2=0x7f4dd000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:10.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de6ffd700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0005210 0x7f4dd000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de6ffd700 1 -- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd0004850 msgr2=0x7f4dd0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:10.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de6ffd700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd0004850 0x7f4dd0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de77fe700 1 -- 172.21.15.158:0/3345763018 reap_dead start 2023-10-16T11:01:10.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de6ffd700 1 -- 172.21.15.158:0/3345763018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00035b0 con 0x7f4dd0003a00 2023-10-16T11:01:10.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de67fc700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd0004850 0x7f4dd0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:10.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de6ffd700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dd0003a00 0x7f4dd0003e20 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0017c0 tx=0x7f4dd0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e781064bf1ada4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:10.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc0052d0 con 0x7f4dd0003a00 2023-10-16T11:01:10.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc005430 con 0x7f4dd0003a00 2023-10-16T11:01:10.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.691+0000 7f4de5ffb700 1 -- 172.21.15.158:0/3345763018 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc0038c0 con 0x7f4dd0003a00 2023-10-16T11:01:10.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.692+0000 7f4ded4d6700 1 -- 172.21.15.158:0/3345763018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dd0003a00 msgr2=0x7f4dd0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.692+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dd0003a00 0x7f4dd0003e20 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0017c0 tx=0x7f4dd0005880 comp rx=0 tx=0).stop 2023-10-16T11:01:10.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.692+0000 7f4ded4d6700 1 -- 172.21.15.158:0/3345763018 shutdown_connections 2023-10-16T11:01:10.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.692+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/3345763018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dd0003a00 0x7f4dd0003e20 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.692+0000 7f4ded4d6700 1 -- 172.21.15.158:0/3345763018 >> 172.21.15.158:0/3345763018 conn(0x7f4de806af00 msgr2=0x7f4de80cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:10.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.692+0000 7f4ded4d6700 1 -- 172.21.15.158:0/3345763018 shutdown_connections 2023-10-16T11:01:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.692+0000 7f4ded4d6700 1 -- 172.21.15.158:0/3345763018 wait complete. 2023-10-16T11:01:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.693+0000 7f4ded4d6700 1 Processor -- start 2023-10-16T11:01:10.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.693+0000 7f4ded4d6700 1 -- start start 2023-10-16T11:01:10.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4ded4d6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0003a00 0x7f4de814e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4ded4d6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4de80c4680 0x7f4de80c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4ded4d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4de814f400 0x7f4de814f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4ded4d6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de814ecd0 con 0x7f4de814f400 2023-10-16T11:01:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4ded4d6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de8159f70 con 0x7f4de80c4680 2023-10-16T11:01:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4ded4d6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80c5140 con 0x7f4dd0003a00 2023-10-16T11:01:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4de814f400 0x7f4de814f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4de814f400 0x7f4de814f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34310/0 (socket says 172.21.15.158:34310) 2023-10-16T11:01:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 -- 172.21.15.158:0/4112172414 learned_addr learned my addr 172.21.15.158:0/4112172414 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:10.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de6ffd700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0003a00 0x7f4de814e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de67fc700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4de80c4680 0x7f4de80c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 -- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0003a00 msgr2=0x7f4de814e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0003a00 0x7f4de814e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 -- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4de80c4680 msgr2=0x7f4de80c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4de80c4680 0x7f4de80c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 -- 172.21.15.158:0/4112172414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de80c5420 con 0x7f4de814f400 2023-10-16T11:01:10.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4de77fe700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4de814f400 0x7f4de814f8b0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f4de815d6a0 tx=0x7f4de0009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:10.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4dcf7fe700 1 -- 172.21.15.158:0/4112172414 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de000e070 con 0x7f4de814f400 2023-10-16T11:01:10.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4dcf7fe700 1 -- 172.21.15.158:0/4112172414 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de0003cf0 con 0x7f4de814f400 2023-10-16T11:01:10.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4dcf7fe700 1 -- 172.21.15.158:0/4112172414 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de0011720 con 0x7f4de814f400 2023-10-16T11:01:10.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.694+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de815a210 con 0x7f4de814f400 2023-10-16T11:01:10.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.695+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de8152370 con 0x7f4de814f400 2023-10-16T11:01:10.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.696+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de80681b0 con 0x7f4de814f400 2023-10-16T11:01:10.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.696+0000 7f4dcf7fe700 1 -- 172.21.15.158:0/4112172414 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4de0011880 con 0x7f4de814f400 2023-10-16T11:01:10.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.696+0000 7f4dcf7fe700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4dd406b020 0x7f4dd406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:10.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.697+0000 7f4de6ffd700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4dd406b020 0x7f4dd406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:10.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.697+0000 7f4dcf7fe700 1 -- 172.21.15.158:0/4112172414 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f4de0089c40 con 0x7f4de814f400 2023-10-16T11:01:10.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.697+0000 7f4de6ffd700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4dd406b020 0x7f4dd406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc005040 tx=0x7f4ddc009030 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:10.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.701+0000 7f4dcf7fe700 1 -- 172.21.15.158:0/4112172414 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4de00548a0 con 0x7f4de814f400 2023-10-16T11:01:10.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.891+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4de80cd1b0 con 0x7f4de814f400 2023-10-16T11:01:10.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.891+0000 7f4dcf7fe700 1 -- 172.21.15.158:0/4112172414 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4de0057ec0 con 0x7f4de814f400 2023-10-16T11:01:10.894 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:01:10.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4dd406b020 msgr2=0x7f4dd406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4dd406b020 0x7f4dd406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc005040 tx=0x7f4ddc009030 comp rx=0 tx=0).stop 2023-10-16T11:01:10.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4de814f400 msgr2=0x7f4de814f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:10.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4de814f400 0x7f4de814f8b0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f4de815d6a0 tx=0x7f4de0009f70 comp rx=0 tx=0).stop 2023-10-16T11:01:10.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 shutdown_connections 2023-10-16T11:01:10.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4dd406b020 0x7f4dd406d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4de814f400 0x7f4de814f8b0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dd0003a00 0x7f4de814e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 --2- 172.21.15.158:0/4112172414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4de80c4680 0x7f4de80c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:10.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 >> 172.21.15.158:0/4112172414 conn(0x7f4de806af00 msgr2=0x7f4de80ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:10.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 shutdown_connections 2023-10-16T11:01:10.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:10.893+0000 7f4ded4d6700 1 -- 172.21.15.158:0/4112172414 wait complete. 2023-10-16T11:01:10.908 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T11:01:11.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.026+0000 7efda5ceb700 1 Processor -- start 2023-10-16T11:01:11.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.026+0000 7efda5ceb700 1 -- start start 2023-10-16T11:01:11.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.027+0000 7efda5ceb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efda00cf4c0 0x7efda0161270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.027+0000 7efda5ceb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda015de20 con 0x7efda006bd10 2023-10-16T11:01:11.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.027+0000 7efda5ceb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda0162180 con 0x7efda00cf8a0 2023-10-16T11:01:11.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.027+0000 7efda5ceb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda01618e0 con 0x7efda00cf4c0 2023-10-16T11:01:11.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.027+0000 7efd9ffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efda006bd10 0x7efda015e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34494/0 (socket says 172.21.15.158:34494) 2023-10-16T11:01:11.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.027+0000 7efd9ffff700 1 -- 172.21.15.158:0/3520332139 learned_addr learned my addr 172.21.15.158:0/3520332139 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:11.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731783883 0 0) 0x7efda015de20 con 0x7efda006bd10 2023-10-16T11:01:11.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9f7fe700 1 --2- 172.21.15.158:0/3520332139 >> v2:172.21.15.172:3300/0 conn(0x7efda00cf4c0 0x7efda0161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c0035b0 con 0x7efda006bd10 2023-10-16T11:01:11.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749724620 0 0) 0x7efd8c0035b0 con 0x7efda006bd10 2023-10-16T11:01:11.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 >> v2:172.21.15.172:3300/0 conn(0x7efda00cf4c0 msgr2=0x7efda0161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:11.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 --2- 172.21.15.158:0/3520332139 >> v2:172.21.15.172:3300/0 conn(0x7efda00cf4c0 0x7efda0161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:11.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 >> v1:172.21.15.172:6789/0 conn(0x7efda00cf8a0 legacy=0x7efda006d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:11.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0161bc0 con 0x7efda006bd10 2023-10-16T11:01:11.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7efd980019f0 con 0x7efda006bd10 2023-10-16T11:01:11.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 >> v1:172.21.15.158:6789/0 conn(0x7efda006bd10 legacy=0x7efda015e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:11.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c003a00 0x7efd8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd8c004850 0x7efd8c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd8c005210 0x7efd8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda01618e0 con 0x7efd8c005210 2023-10-16T11:01:11.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c00c1b0 con 0x7efd8c003a00 2023-10-16T11:01:11.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c00c370 con 0x7efd8c004850 2023-10-16T11:01:11.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9ffff700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd8c005210 0x7efd8c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9f7fe700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c003a00 0x7efd8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9effd700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd8c004850 0x7efd8c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9ffff700 1 -- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd8c004850 msgr2=0x7efd8c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:11.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9ffff700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd8c004850 0x7efd8c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:11.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9ffff700 1 -- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c003a00 msgr2=0x7efd8c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:11.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9ffff700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c003a00 0x7efd8c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:11.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9ffff700 1 -- 172.21.15.158:0/3520332139 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c0035b0 con 0x7efd8c005210 2023-10-16T11:01:11.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.028+0000 7efd9ffff700 1 -- 172.21.15.158:0/3520332139 reap_dead start 2023-10-16T11:01:11.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efd9ffff700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd8c005210 0x7efd8c00ba10 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7efd9800a820 tx=0x7efd8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5c6f85b5cb330c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:11.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9800a2f0 con 0x7efd8c005210 2023-10-16T11:01:11.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd98012600 con 0x7efd8c005210 2023-10-16T11:01:11.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efd9e7fc700 1 -- 172.21.15.158:0/3520332139 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd980129b0 con 0x7efd8c005210 2023-10-16T11:01:11.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efda5ceb700 1 -- 172.21.15.158:0/3520332139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd8c005210 msgr2=0x7efd8c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:11.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efda5ceb700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd8c005210 0x7efd8c00ba10 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7efd9800a820 tx=0x7efd8c005880 comp rx=0 tx=0).stop 2023-10-16T11:01:11.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efda5ceb700 1 -- 172.21.15.158:0/3520332139 shutdown_connections 2023-10-16T11:01:11.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efda5ceb700 1 --2- 172.21.15.158:0/3520332139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd8c005210 0x7efd8c00ba10 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:11.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efda5ceb700 1 -- 172.21.15.158:0/3520332139 >> 172.21.15.158:0/3520332139 conn(0x7efda006a8d0 msgr2=0x7efda006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:11.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efda5ceb700 1 -- 172.21.15.158:0/3520332139 shutdown_connections 2023-10-16T11:01:11.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.029+0000 7efda5ceb700 1 -- 172.21.15.158:0/3520332139 wait complete. 2023-10-16T11:01:11.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.030+0000 7efda5ceb700 1 Processor -- start 2023-10-16T11:01:11.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efda5ceb700 1 -- start start 2023-10-16T11:01:11.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efda5ceb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c005210 0x7efda00c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efda5ceb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efda00c4f80 0x7efda015ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efda5ceb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0158e70 0x7efda0159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efda5ceb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda00c7570 con 0x7efda00c4f80 2023-10-16T11:01:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efda5ceb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda00c5480 con 0x7efd8c005210 2023-10-16T11:01:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efda5ceb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda00c5760 con 0x7efda0158e70 2023-10-16T11:01:11.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.031+0000 7efd9effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efda00c4f80 0x7efda015ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efda00c4f80 0x7efda015ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34328/0 (socket says 172.21.15.158:34328) 2023-10-16T11:01:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 -- 172.21.15.158:0/1741538273 learned_addr learned my addr 172.21.15.158:0/1741538273 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9ffff700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0158e70 0x7efda0159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9f7fe700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c005210 0x7efda00c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 -- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0158e70 msgr2=0x7efda0159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0158e70 0x7efda0159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 -- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c005210 msgr2=0x7efda00c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c005210 0x7efda00c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 -- 172.21.15.158:0/1741538273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00c5a40 con 0x7efda00c4f80 2023-10-16T11:01:11.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd9effd700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efda00c4f80 0x7efda015ffb0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7efd9000f820 tx=0x7efd9000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd97fff700 1 -- 172.21.15.158:0/1741538273 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900052e0 con 0x7efda00c4f80 2023-10-16T11:01:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd97fff700 1 -- 172.21.15.158:0/1741538273 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90005460 con 0x7efda00c4f80 2023-10-16T11:01:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda00c5d20 con 0x7efda00c4f80 2023-10-16T11:01:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efd97fff700 1 -- 172.21.15.158:0/1741538273 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900038c0 con 0x7efda00c4f80 2023-10-16T11:01:11.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.032+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda01604f0 con 0x7efda00c4f80 2023-10-16T11:01:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.033+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda01606b0 con 0x7efda00c4f80 2023-10-16T11:01:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.034+0000 7efd97fff700 1 -- 172.21.15.158:0/1741538273 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd90003a20 con 0x7efda00c4f80 2023-10-16T11:01:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.034+0000 7efd97fff700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efd8006b020 0x7efd8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.034+0000 7efd97fff700 1 -- 172.21.15.158:0/1741538273 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7efd900897d0 con 0x7efda00c4f80 2023-10-16T11:01:11.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.034+0000 7efd9f7fe700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efd8006b020 0x7efd8006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:11.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.034+0000 7efd9f7fe700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efd8006b020 0x7efd8006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efd88009b20 tx=0x7efd88008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:11.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.038+0000 7efd97fff700 1 -- 172.21.15.158:0/1741538273 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd900544b0 con 0x7efda00c4f80 2023-10-16T11:01:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:11.313+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7efda00681b0 con 0x7efda00c4f80 2023-10-16T11:01:11.566 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:01:11.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:01:12.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.210+0000 7efd97fff700 1 -- 172.21.15.158:0/1741538273 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2538) v1 ==== 168+0+0 (secure 0 0 0) 0x7efd90087e60 con 0x7efda00c4f80 2023-10-16T11:01:12.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efd8006b020 msgr2=0x7efd8006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efd8006b020 0x7efd8006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efd88009b20 tx=0x7efd88008040 comp rx=0 tx=0).stop 2023-10-16T11:01:12.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efda00c4f80 msgr2=0x7efda015ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efda00c4f80 0x7efda015ffb0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7efd9000f820 tx=0x7efd9000fbe0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 shutdown_connections 2023-10-16T11:01:12.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7efd8006b020 0x7efd8006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0158e70 0x7efda0159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd8c005210 0x7efda00c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 --2- 172.21.15.158:0/1741538273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efda00c4f80 0x7efda015ffb0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 >> 172.21.15.158:0/1741538273 conn(0x7efda006a8d0 msgr2=0x7efda0156b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:12.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 shutdown_connections 2023-10-16T11:01:12.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.213+0000 7efda5ceb700 1 -- 172.21.15.158:0/1741538273 wait complete. 2023-10-16T11:01:12.231 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T11:01:12.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.360+0000 7f4add840700 1 Processor -- start 2023-10-16T11:01:12.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.360+0000 7f4add840700 1 -- start start 2023-10-16T11:01:12.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4add840700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ad8160d30 0x7f4ad80b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4add840700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 0x7f4ad80d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4add840700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad80b7c90 con 0x7f4ad80d19a0 2023-10-16T11:01:12.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4add840700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad80b8910 con 0x7f4ad8160d30 2023-10-16T11:01:12.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4add840700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80d0ef0 con 0x7f4ad80d0530 2023-10-16T11:01:12.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4ad80d19a0 0x7f4ad80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34514/0 (socket says 172.21.15.158:34514) 2023-10-16T11:01:12.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 -- 172.21.15.158:0/842881933 learned_addr learned my addr 172.21.15.158:0/842881933 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:12.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/842881933 >> v2:172.21.15.172:3300/0 conn(0x7f4ad8160d30 0x7f4ad80b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad67fc700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 0x7f4ad80d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174012034 0 0) 0x7f4ad80b7c90 con 0x7f4ad80d19a0 2023-10-16T11:01:12.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac00035b0 con 0x7f4ad80d19a0 2023-10-16T11:01:12.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 882344421 0 0) 0x7f4ac00035b0 con 0x7f4ad80d19a0 2023-10-16T11:01:12.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 msgr2=0x7f4ad80d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:12.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 0x7f4ad80d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 >> v2:172.21.15.172:3300/0 conn(0x7f4ad8160d30 msgr2=0x7f4ad80b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:12.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 --2- 172.21.15.158:0/842881933 >> v2:172.21.15.172:3300/0 conn(0x7f4ad8160d30 0x7f4ad80b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80b7f70 con 0x7f4ad80d19a0 2023-10-16T11:01:12.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4ad00019f0 con 0x7f4ad80d19a0 2023-10-16T11:01:12.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 >> v1:172.21.15.158:6789/0 conn(0x7f4ad80d19a0 legacy=0x7f4ad80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 0x7f4ac0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ac0003f80 0x7f4ac0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ac0004d90 0x7f4ac0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80b8910 con 0x7f4ac0004d90 2023-10-16T11:01:12.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80d0ef0 con 0x7f4ac0003f80 2023-10-16T11:01:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0014270 con 0x7f4ad80d0530 2023-10-16T11:01:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ac0004d90 0x7f4ac0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 0x7f4ac0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad67fc700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ac0003f80 0x7f4ac0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 -- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 msgr2=0x7f4ac0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80d0530 0x7f4ac0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 -- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ac0003f80 msgr2=0x7f4ac0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ac0003f80 0x7f4ac0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 -- 172.21.15.158:0/842881933 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac00035b0 con 0x7f4ac0004d90 2023-10-16T11:01:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ac0004d90 0x7f4ac0005210 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000a860 tx=0x7f4ac0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb3033e846c05a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:12.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000a320 con 0x7f4ac0004d90 2023-10-16T11:01:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad0012600 con 0x7f4ac0004d90 2023-10-16T11:01:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.362+0000 7f4ad5ffb700 1 -- 172.21.15.158:0/842881933 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad00129b0 con 0x7f4ac0004d90 2023-10-16T11:01:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.363+0000 7f4add840700 1 -- 172.21.15.158:0/842881933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ac0004d90 msgr2=0x7f4ac0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.363+0000 7f4add840700 1 --2- 172.21.15.158:0/842881933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ac0004d90 0x7f4ac0005210 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000a860 tx=0x7f4ac0005880 comp rx=0 tx=0).stop 2023-10-16T11:01:12.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.363+0000 7f4ad77fe700 1 -- 172.21.15.158:0/842881933 reap_dead start 2023-10-16T11:01:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.363+0000 7f4add840700 1 -- 172.21.15.158:0/842881933 shutdown_connections 2023-10-16T11:01:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.363+0000 7f4add840700 1 -- 172.21.15.158:0/842881933 >> 172.21.15.158:0/842881933 conn(0x7f4ad80669f0 msgr2=0x7f4ad8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.363+0000 7f4add840700 1 -- 172.21.15.158:0/842881933 shutdown_connections 2023-10-16T11:01:12.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.363+0000 7f4add840700 1 -- 172.21.15.158:0/842881933 wait complete. 2023-10-16T11:01:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.364+0000 7f4add840700 1 Processor -- start 2023-10-16T11:01:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.364+0000 7f4add840700 1 -- start start 2023-10-16T11:01:12.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4add840700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad8154da0 0x7f4ad80ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4add840700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad806db30 0x7f4ad80cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4add840700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80cc460 0x7f4ad806f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4add840700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80cc8e0 con 0x7f4ad8154da0 2023-10-16T11:01:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4add840700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8155180 con 0x7f4ad806db30 2023-10-16T11:01:12.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4add840700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80711e0 con 0x7f4ad80cc460 2023-10-16T11:01:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad8154da0 0x7f4ad80ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad8154da0 0x7f4ad80ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34346/0 (socket says 172.21.15.158:34346) 2023-10-16T11:01:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/97982745 learned_addr learned my addr 172.21.15.158:0/97982745 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad67fc700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad806db30 0x7f4ad80cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad77fe700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80cc460 0x7f4ad806f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80cc460 msgr2=0x7f4ad806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80cc460 0x7f4ad806f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad806db30 msgr2=0x7f4ad80cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad806db30 0x7f4ad80cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 -- 172.21.15.158:0/97982745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80714c0 con 0x7f4ad8154da0 2023-10-16T11:01:12.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.365+0000 7f4ad6ffd700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad8154da0 0x7f4ad80ccae0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f4acc011ad0 tx=0x7f4acc011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:12.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.366+0000 7f4abffff700 1 -- 172.21.15.158:0/97982745 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc00f940 con 0x7f4ad8154da0 2023-10-16T11:01:12.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.366+0000 7f4abffff700 1 -- 172.21.15.158:0/97982745 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc00faa0 con 0x7f4ad8154da0 2023-10-16T11:01:12.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.366+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad80717a0 con 0x7f4ad8154da0 2023-10-16T11:01:12.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.366+0000 7f4abffff700 1 -- 172.21.15.158:0/97982745 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc00fe50 con 0x7f4ad8154da0 2023-10-16T11:01:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.366+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad806dfb0 con 0x7f4ad8154da0 2023-10-16T11:01:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.367+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad80c7300 con 0x7f4ad8154da0 2023-10-16T11:01:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.367+0000 7f4abffff700 1 -- 172.21.15.158:0/97982745 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4acc014440 con 0x7f4ad8154da0 2023-10-16T11:01:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.367+0000 7f4abffff700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4ac406b020 0x7f4ac406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.367+0000 7f4abffff700 1 -- 172.21.15.158:0/97982745 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f4acc08a1c0 con 0x7f4ad8154da0 2023-10-16T11:01:12.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.367+0000 7f4ad67fc700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4ac406b020 0x7f4ac406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.368+0000 7f4ad67fc700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4ac406b020 0x7f4ac406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ad815bc20 tx=0x7f4ac8009210 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.373+0000 7f4abffff700 1 -- 172.21.15.158:0/97982745 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4acc054e00 con 0x7f4ad8154da0 2023-10-16T11:01:12.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.567+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4ad80c70f0 con 0x7f4ad8154da0 2023-10-16T11:01:12.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.571+0000 7f4abffff700 1 -- 172.21.15.158:0/97982745 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4acc058420 con 0x7f4ad8154da0 2023-10-16T11:01:12.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4ac406b020 msgr2=0x7f4ac406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4ac406b020 0x7f4ac406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ad815bc20 tx=0x7f4ac8009210 comp rx=0 tx=0).stop 2023-10-16T11:01:12.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad8154da0 msgr2=0x7f4ad80ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad8154da0 0x7f4ad80ccae0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f4acc011ad0 tx=0x7f4acc011de0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 shutdown_connections 2023-10-16T11:01:12.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f4ac406b020 0x7f4ac406d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ad80cc460 0x7f4ad806f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ad8154da0 0x7f4ad80ccae0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 --2- 172.21.15.158:0/97982745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ad806db30 0x7f4ad80cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 >> 172.21.15.158:0/97982745 conn(0x7f4ad80669f0 msgr2=0x7f4ad80d3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:12.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 shutdown_connections 2023-10-16T11:01:12.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.573+0000 7f4add840700 1 -- 172.21.15.158:0/97982745 wait complete. 2023-10-16T11:01:12.587 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:01:12.588 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:01:12.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:01:12.679 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:01:12.679 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:01:12.680 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:12.697 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:01:12.698 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:01:12.698 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:12.718 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:01:12.719 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:01:12.719 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:12.755 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:01:12.755 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:01:12.756 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:12.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:01:12.778 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T11:01:12.778 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:01:12.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9ba53e4700 1 Processor -- start 2023-10-16T11:01:12.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9ba53e4700 1 -- start start 2023-10-16T11:01:12.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9ba53e4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9ba00d1210 0x7f9ba00ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9ba53e4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba00ce950 0x7f9ba00b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9ba53e4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00d3bd0 con 0x7f9ba015d7c0 2023-10-16T11:01:12.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9ba53e4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba00d4060 con 0x7f9ba00d1210 2023-10-16T11:01:12.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9ba53e4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00cedc0 con 0x7f9ba00ce950 2023-10-16T11:01:12.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9ba00d1210 0x7f9ba00ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9ba00d1210 0x7f9ba00ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34356/0 (socket says 172.21.15.158:34356) 2023-10-16T11:01:12.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2224439298 learned_addr learned my addr 172.21.15.158:0/2224439298 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:12.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9f7fe700 1 --2- 172.21.15.158:0/2224439298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba00ce950 0x7f9ba00b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2224439298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba00ce950 msgr2=0x7f9ba00b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:12.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 --2- 172.21.15.158:0/2224439298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba00ce950 0x7f9ba00b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2224439298 >> v1:172.21.15.172:6789/0 conn(0x7f9ba015d7c0 legacy=0x7f9ba00d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:12.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2224439298 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba00d2fc0 con 0x7f9ba00d1210 2023-10-16T11:01:12.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9f7fe700 1 --2- 172.21.15.158:0/2224439298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba00ce950 0x7f9ba00b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:12.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9e7fc700 1 --2- 172.21.15.158:0/2224439298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ce410 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f9b940098f0 tx=0x7f9b94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ca0b17c8184717 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:12.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9dffb700 1 -- 172.21.15.158:0/2224439298 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94004230 con 0x7f9ba00d1210 2023-10-16T11:01:12.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9dffb700 1 -- 172.21.15.158:0/2224439298 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94004390 con 0x7f9ba00d1210 2023-10-16T11:01:12.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.913+0000 7f9b9dffb700 1 -- 172.21.15.158:0/2224439298 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940036a0 con 0x7f9ba00d1210 2023-10-16T11:01:12.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2224439298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 msgr2=0x7f9ba00ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2224439298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ce410 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f9b940098f0 tx=0x7f9b94009c00 comp rx=0 tx=0).stop 2023-10-16T11:01:12.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2224439298 shutdown_connections 2023-10-16T11:01:12.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2224439298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba00ce950 0x7f9ba00b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2224439298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ce410 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2224439298 >> 172.21.15.158:0/2224439298 conn(0x7f9ba00669f0 msgr2=0x7f9ba0066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:12.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2224439298 shutdown_connections 2023-10-16T11:01:12.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.914+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2224439298 wait complete. 2023-10-16T11:01:12.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.915+0000 7f9ba53e4700 1 Processor -- start 2023-10-16T11:01:12.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.915+0000 7f9ba53e4700 1 -- start start 2023-10-16T11:01:12.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9ba53e4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba00ce950 0x7f9ba0151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9ba53e4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9ba53e4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba015d7c0 0x7f9ba01604b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9ba53e4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0151bb0 con 0x7f9ba00d1210 2023-10-16T11:01:12.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9ba53e4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0152090 con 0x7f9ba00ce950 2023-10-16T11:01:12.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9ba53e4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00aa330 con 0x7f9ba015d7c0 2023-10-16T11:01:12.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ab3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ab3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34362/0 (socket says 172.21.15.158:34362) 2023-10-16T11:01:12.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2484477501 learned_addr learned my addr 172.21.15.158:0/2484477501 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:12.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9f7fe700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba015d7c0 0x7f9ba01604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9effd700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba00ce950 0x7f9ba0151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba015d7c0 msgr2=0x7f9ba01604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba015d7c0 0x7f9ba01604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba00ce950 msgr2=0x7f9ba0151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:12.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba00ce950 0x7f9ba0151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:12.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 -- 172.21.15.158:0/2484477501 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b94009550 con 0x7f9ba00d1210 2023-10-16T11:01:12.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.916+0000 7f9b9e7fc700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ab3d0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009130 tx=0x7f9b94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:12.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.917+0000 7f9b877fe700 1 -- 172.21.15.158:0/2484477501 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94003cd0 con 0x7f9ba00d1210 2023-10-16T11:01:12.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.917+0000 7f9b877fe700 1 -- 172.21.15.158:0/2484477501 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9401f070 con 0x7f9ba00d1210 2023-10-16T11:01:12.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.917+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba00aa610 con 0x7f9ba00d1210 2023-10-16T11:01:12.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.917+0000 7f9b877fe700 1 -- 172.21.15.158:0/2484477501 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9400b8a0 con 0x7f9ba00d1210 2023-10-16T11:01:12.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.917+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba00aabd0 con 0x7f9ba00d1210 2023-10-16T11:01:12.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.918+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba00681b0 con 0x7f9ba00d1210 2023-10-16T11:01:12.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.918+0000 7f9b877fe700 1 -- 172.21.15.158:0/2484477501 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9b94021030 con 0x7f9ba00d1210 2023-10-16T11:01:12.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.918+0000 7f9b877fe700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9b8c06b050 0x7f9b8c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:12.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.919+0000 7f9b877fe700 1 -- 172.21.15.158:0/2484477501 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f9b9408b020 con 0x7f9ba00d1210 2023-10-16T11:01:12.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.919+0000 7f9b9effd700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9b8c06b050 0x7f9b8c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:12.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.919+0000 7f9b9effd700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9b8c06b050 0x7f9b8c06d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9b90007fd0 tx=0x7f9b90007ec0 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:12.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:12.924+0000 7f9b877fe700 1 -- 172.21.15.158:0/2484477501 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b94055d00 con 0x7f9ba00d1210 2023-10-16T11:01:13.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.158+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ba00d3550 con 0x7f9ba00d1210 2023-10-16T11:01:13.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.159+0000 7f9b877fe700 1 -- 172.21.15.158:0/2484477501 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f9b94059320 con 0x7f9ba00d1210 2023-10-16T11:01:13.161 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:13.162 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":87,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3392606972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3392606972},{"type":"v1","addr":"172.21.15.172:6834","nonce":3392606972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/187629256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":187629256},{"type":"v1","addr":"172.21.15.172:6835","nonce":187629256}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":9930,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/3321991635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3321991635},{"type":"v1","addr":"172.21.15.158:6836","nonce":3321991635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":9933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4013459160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4013459160},{"type":"v1","addr":"172.21.15.158:6837","nonce":4013459160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[]} 2023-10-16T11:01:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9b8c06b050 msgr2=0x7f9b8c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9b8c06b050 0x7f9b8c06d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9b90007fd0 tx=0x7f9b90007ec0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 msgr2=0x7f9ba00ab3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ab3d0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009130 tx=0x7f9b94004750 comp rx=0 tx=0).stop 2023-10-16T11:01:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 shutdown_connections 2023-10-16T11:01:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9b8c06b050 0x7f9b8c06d510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ba015d7c0 0x7f9ba01604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba00ce950 0x7f9ba0151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 --2- 172.21.15.158:0/2484477501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba00d1210 0x7f9ba00ab3d0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 >> 172.21.15.158:0/2484477501 conn(0x7f9ba00669f0 msgr2=0x7f9ba0066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:13.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 shutdown_connections 2023-10-16T11:01:13.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.161+0000 7f9ba53e4700 1 -- 172.21.15.158:0/2484477501 wait complete. 2023-10-16T11:01:13.166 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 87 2023-10-16T11:01:13.175 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T11:01:13.176 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T11:01:13.176 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:01:13.176 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:01:13.177 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T11:01:13.177 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T11:01:13.177 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:01:13.177 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:01:13.178 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T11:01:13.178 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T11:01:13.178 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:01:13.179 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:01:13.179 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T11:01:13.179 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T11:01:13.179 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:01:13.180 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:01:13.186 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:01:13.187 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T11:01:13.188 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:01:13.188 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T11:01:13.190 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:01:13.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T11:01:13.191 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:01:13.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T11:01:13.230 INFO:tasks.ceph.mds.a:Started 2023-10-16T11:01:13.230 DEBUG:teuthology.parallel:result is None 2023-10-16T11:01:13.232 INFO:tasks.ceph.mds.c:Started 2023-10-16T11:01:13.232 DEBUG:teuthology.parallel:result is None 2023-10-16T11:01:13.233 INFO:tasks.ceph.mds.b:Started 2023-10-16T11:01:13.234 DEBUG:teuthology.parallel:result is None 2023-10-16T11:01:13.234 INFO:tasks.ceph.mds.d:Started 2023-10-16T11:01:13.235 DEBUG:teuthology.parallel:result is None 2023-10-16T11:01:13.235 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T11:01:13.309 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T11:01:13.310 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:01:13.322 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:01:13.322 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:01:13.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.361+0000 7f0f6d393700 1 Processor -- start 2023-10-16T11:01:13.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.361+0000 7f0f6d393700 1 -- start start 2023-10-16T11:01:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f6d393700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0f6815a500 0x7f0f680b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f6d393700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f680b4240 con 0x7f0f680cfa70 2023-10-16T11:01:13.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f6d393700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6815ee90 con 0x7f0f680b3c70 2023-10-16T11:01:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f6d393700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f680cd7a0 con 0x7f0f6815a500 2023-10-16T11:01:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f66ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0f6815a500 0x7f0f680b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f667fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0f680cfa70 0x7f0f6815e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34570/0 (socket says 172.21.15.158:34570) 2023-10-16T11:01:13.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f667fc700 1 -- 172.21.15.158:0/1523657546 learned_addr learned my addr 172.21.15.158:0/1523657546 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f66ffd700 1 -- 172.21.15.158:0/1523657546 >> v1:172.21.15.172:6789/0 conn(0x7f0f680b3c70 legacy=0x7f0f6815e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f65ffb700 1 -- 172.21.15.158:0/1523657546 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638103259 0 0) 0x7f0f680b4240 con 0x7f0f680cfa70 2023-10-16T11:01:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f66ffd700 1 -- 172.21.15.158:0/1523657546 >> v1:172.21.15.158:6789/0 conn(0x7f0f680cfa70 legacy=0x7f0f6815e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.362+0000 7f0f66ffd700 1 -- 172.21.15.158:0/1523657546 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f68071b20 con 0x7f0f6815a500 2023-10-16T11:01:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f66ffd700 1 --2- 172.21.15.158:0/1523657546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f6815a500 0x7f0f680b4a00 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009840 tx=0x7f0f60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03742a317155da6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:13.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f65ffb700 1 -- 172.21.15.158:0/1523657546 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60011830 con 0x7f0f6815a500 2023-10-16T11:01:13.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f65ffb700 1 -- 172.21.15.158:0/1523657546 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f60011990 con 0x7f0f6815a500 2023-10-16T11:01:13.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f65ffb700 1 -- 172.21.15.158:0/1523657546 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60011d20 con 0x7f0f6815a500 2023-10-16T11:01:13.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f6d393700 1 -- 172.21.15.158:0/1523657546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f6815a500 msgr2=0x7f0f680b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:13.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f6d393700 1 --2- 172.21.15.158:0/1523657546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f6815a500 0x7f0f680b4a00 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009840 tx=0x7f0f60009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:13.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f6d393700 1 -- 172.21.15.158:0/1523657546 shutdown_connections 2023-10-16T11:01:13.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f6d393700 1 --2- 172.21.15.158:0/1523657546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f6815a500 0x7f0f680b4a00 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f6d393700 1 -- 172.21.15.158:0/1523657546 >> 172.21.15.158:0/1523657546 conn(0x7f0f6806da00 msgr2=0x7f0f6806dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:13.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f6d393700 1 -- 172.21.15.158:0/1523657546 shutdown_connections 2023-10-16T11:01:13.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.363+0000 7f0f6d393700 1 -- 172.21.15.158:0/1523657546 wait complete. 2023-10-16T11:01:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 Processor -- start 2023-10-16T11:01:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 -- start start 2023-10-16T11:01:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f680b3c70 0x7f0f681f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:13.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f680cfa70 0x7f0f6815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f6815a500 0x7f0f6815de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68153fc0 con 0x7f0f680cfa70 2023-10-16T11:01:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68153720 con 0x7f0f6815a500 2023-10-16T11:01:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f6d393700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68153a00 con 0x7f0f680b3c70 2023-10-16T11:01:13.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f667fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f680cfa70 0x7f0f6815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.365+0000 7f0f667fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f680cfa70 0x7f0f6815d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34400/0 (socket says 172.21.15.158:34400) 2023-10-16T11:01:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f667fc700 1 -- 172.21.15.158:0/3588415995 learned_addr learned my addr 172.21.15.158:0/3588415995 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f66ffd700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f680b3c70 0x7f0f681f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f677fe700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f6815a500 0x7f0f6815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:13.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f667fc700 1 -- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f680b3c70 msgr2=0x7f0f681f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f667fc700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f680b3c70 0x7f0f681f6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f667fc700 1 -- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f6815a500 msgr2=0x7f0f6815de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f667fc700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f6815a500 0x7f0f6815de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f667fc700 1 -- 172.21.15.158:0/3588415995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f600094a0 con 0x7f0f680cfa70 2023-10-16T11:01:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f667fc700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f680cfa70 0x7f0f6815d8d0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f0f50001eb0 tx=0x7f0f5000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:13.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f5f7fe700 1 -- 172.21.15.158:0/3588415995 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50004c80 con 0x7f0f680cfa70 2023-10-16T11:01:13.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f5f7fe700 1 -- 172.21.15.158:0/3588415995 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f50004de0 con 0x7f0f680cfa70 2023-10-16T11:01:13.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f68153ce0 con 0x7f0f680cfa70 2023-10-16T11:01:13.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f5f7fe700 1 -- 172.21.15.158:0/3588415995 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50007310 con 0x7f0f680cfa70 2023-10-16T11:01:13.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.366+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6814af20 con 0x7f0f680cfa70 2023-10-16T11:01:13.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.367+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f680b4870 con 0x7f0f680cfa70 2023-10-16T11:01:13.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.368+0000 7f0f5f7fe700 1 -- 172.21.15.158:0/3588415995 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0f50007470 con 0x7f0f680cfa70 2023-10-16T11:01:13.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.368+0000 7f0f5f7fe700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0f4806b020 0x7f0f4806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:13.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.368+0000 7f0f5f7fe700 1 -- 172.21.15.158:0/3588415995 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f0f50089570 con 0x7f0f680cfa70 2023-10-16T11:01:13.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.368+0000 7f0f66ffd700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0f4806b020 0x7f0f4806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:13.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.368+0000 7f0f66ffd700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0f4806b020 0x7f0f4806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0f600090b0 tx=0x7f0f6001c040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:13.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.373+0000 7f0f5f7fe700 1 -- 172.21.15.158:0/3588415995 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f500542e0 con 0x7f0f680cfa70 2023-10-16T11:01:13.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:13.583+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0f68071da0 con 0x7f0f680cfa70 2023-10-16T11:01:14.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.228+0000 7f0f5f7fe700 1 -- 172.21.15.158:0/3588415995 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v179) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0f68071da0 con 0x7f0f680cfa70 2023-10-16T11:01:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0f4806b020 msgr2=0x7f0f4806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0f4806b020 0x7f0f4806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0f600090b0 tx=0x7f0f6001c040 comp rx=0 tx=0).stop 2023-10-16T11:01:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f680cfa70 msgr2=0x7f0f6815d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f680cfa70 0x7f0f6815d8d0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f0f50001eb0 tx=0x7f0f5000c800 comp rx=0 tx=0).stop 2023-10-16T11:01:14.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 shutdown_connections 2023-10-16T11:01:14.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0f4806b020 0x7f0f4806d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f6815a500 0x7f0f6815de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f680b3c70 0x7f0f681f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 --2- 172.21.15.158:0/3588415995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f680cfa70 0x7f0f6815d8d0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 >> 172.21.15.158:0/3588415995 conn(0x7f0f6806da00 msgr2=0x7f0f680b3180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 shutdown_connections 2023-10-16T11:01:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.232+0000 7f0f6d393700 1 -- 172.21.15.158:0/3588415995 wait complete. 2023-10-16T11:01:14.236 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T11:01:14.248 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T11:01:14.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.368+0000 7f9abdde2700 1 Processor -- start 2023-10-16T11:01:14.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.368+0000 7f9abdde2700 1 -- start start 2023-10-16T11:01:14.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9abdde2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9ab80729b0 0x7f9ab806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9abdde2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab8161270 con 0x7f9ab806e5d0 2023-10-16T11:01:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9abdde2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab8161510 con 0x7f9ab806ab40 2023-10-16T11:01:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9abdde2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab81617f0 con 0x7f9ab80729b0 2023-10-16T11:01:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9ab80729b0 0x7f9ab806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9ab806e5d0 0x7f9ab815df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34586/0 (socket says 172.21.15.158:34586) 2023-10-16T11:01:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab6ffd700 1 -- 172.21.15.158:0/3600424262 learned_addr learned my addr 172.21.15.158:0/3600424262 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab7fff700 1 -- 172.21.15.158:0/3600424262 >> v1:172.21.15.172:6789/0 conn(0x7f9ab806ab40 legacy=0x7f9ab80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab7fff700 1 -- 172.21.15.158:0/3600424262 >> v1:172.21.15.158:6789/0 conn(0x7f9ab806e5d0 legacy=0x7f9ab815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab67fc700 1 -- 172.21.15.158:0/3600424262 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290338190 0 0) 0x7f9ab8161270 con 0x7f9ab806e5d0 2023-10-16T11:01:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab7fff700 1 -- 172.21.15.158:0/3600424262 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8161ad0 con 0x7f9ab80729b0 2023-10-16T11:01:14.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab7fff700 1 --2- 172.21.15.158:0/3600424262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab80729b0 0x7f9ab806d820 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009d90 tx=0x7f9ab000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f521fe2b4498c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:14.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab67fc700 1 -- 172.21.15.158:0/3600424262 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab000e070 con 0x7f9ab80729b0 2023-10-16T11:01:14.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab67fc700 1 -- 172.21.15.158:0/3600424262 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab000bbd0 con 0x7f9ab80729b0 2023-10-16T11:01:14.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.369+0000 7f9ab67fc700 1 -- 172.21.15.158:0/3600424262 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0012430 con 0x7f9ab80729b0 2023-10-16T11:01:14.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.370+0000 7f9abdde2700 1 -- 172.21.15.158:0/3600424262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab80729b0 msgr2=0x7f9ab806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.370+0000 7f9abdde2700 1 --2- 172.21.15.158:0/3600424262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab80729b0 0x7f9ab806d820 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009d90 tx=0x7f9ab000b330 comp rx=0 tx=0).stop 2023-10-16T11:01:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.370+0000 7f9abdde2700 1 -- 172.21.15.158:0/3600424262 shutdown_connections 2023-10-16T11:01:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.370+0000 7f9abdde2700 1 --2- 172.21.15.158:0/3600424262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab80729b0 0x7f9ab806d820 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.370+0000 7f9abdde2700 1 -- 172.21.15.158:0/3600424262 >> 172.21.15.158:0/3600424262 conn(0x7f9ab80d02b0 msgr2=0x7f9ab80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:14.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.370+0000 7f9abdde2700 1 -- 172.21.15.158:0/3600424262 shutdown_connections 2023-10-16T11:01:14.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.370+0000 7f9abdde2700 1 -- 172.21.15.158:0/3600424262 wait complete. 2023-10-16T11:01:14.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.372+0000 7f9abdde2700 1 Processor -- start 2023-10-16T11:01:14.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.372+0000 7f9abdde2700 1 -- start start 2023-10-16T11:01:14.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ab806ab40 0x7f9ab8153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab806e5d0 0x7f9ab80b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ab80729b0 0x7f9ab80ad620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80b2820 con 0x7f9ab806e5d0 2023-10-16T11:01:14.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab81f2a70 con 0x7f9ab806ab40 2023-10-16T11:01:14.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8153640 con 0x7f9ab80729b0 2023-10-16T11:01:14.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab806e5d0 0x7f9ab80b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab806e5d0 0x7f9ab80b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34416/0 (socket says 172.21.15.158:34416) 2023-10-16T11:01:14.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 -- 172.21.15.158:0/463139615 learned_addr learned my addr 172.21.15.158:0/463139615 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:14.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab7fff700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ab80729b0 0x7f9ab80ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab77fe700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ab806ab40 0x7f9ab8153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 -- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ab80729b0 msgr2=0x7f9ab80ad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ab80729b0 0x7f9ab80ad620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 -- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ab806ab40 msgr2=0x7f9ab8153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ab806ab40 0x7f9ab8153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 -- 172.21.15.158:0/463139615 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab0009a70 con 0x7f9ab806e5d0 2023-10-16T11:01:14.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab7fff700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ab80729b0 0x7f9ab80ad620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:14.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9ab6ffd700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab806e5d0 0x7f9ab80b15c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c520 tx=0x7f9aac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:14.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9a9ffff700 1 -- 172.21.15.158:0/463139615 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac004c80 con 0x7f9ab806e5d0 2023-10-16T11:01:14.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9a9ffff700 1 -- 172.21.15.158:0/463139615 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac004de0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab80b37f0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9a9ffff700 1 -- 172.21.15.158:0/463139615 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac007310 con 0x7f9ab806e5d0 2023-10-16T11:01:14.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.373+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab80b3db0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.374+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab80681b0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.375+0000 7f9a9ffff700 1 -- 172.21.15.158:0/463139615 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9aac007470 con 0x7f9ab806e5d0 2023-10-16T11:01:14.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.375+0000 7f9a9ffff700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9aa406b020 0x7f9aa406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.375+0000 7f9a9ffff700 1 -- 172.21.15.158:0/463139615 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f9aac0893e0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.375+0000 7f9ab77fe700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9aa406b020 0x7f9aa406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.376+0000 7f9ab77fe700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9aa406b020 0x7f9aa406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8000f60 tx=0x7f9aa8009040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:14.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.380+0000 7f9a9ffff700 1 -- 172.21.15.158:0/463139615 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9aac0543d0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.649+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9ab80c39d0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.649+0000 7f9a9ffff700 1 -- 172.21.15.158:0/463139615 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9aac0579f0 con 0x7f9ab806e5d0 2023-10-16T11:01:14.651 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:14.651 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:01:14.652 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T11:01:14.652 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.652 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T11:01:14.652 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T11:01:14.653 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.653 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.653 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:01:14.653 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:01:14.653 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.654 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.654 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.654 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.654 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T11:01:14.655 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T11:01:14.655 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.655 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.655 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:01:14.656 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:01:14.656 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.656 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.656 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.656 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.657 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T11:01:14.657 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T11:01:14.657 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.657 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.658 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:01:14.658 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:01:14.658 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.658 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.659 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.659 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T11:01:14.659 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T11:01:14.660 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.660 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.660 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:01:14.661 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:01:14.661 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.661 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.661 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.661 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.662 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T11:01:14.662 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T11:01:14.662 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.662 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.663 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.663 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.663 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.664 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.664 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T11:01:14.664 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T11:01:14.664 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.664 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.664 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.665 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.665 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.665 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.665 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.665 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T11:01:14.666 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T11:01:14.666 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.666 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.666 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.667 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.667 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.667 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.667 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.667 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T11:01:14.668 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T11:01:14.668 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.668 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.668 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.668 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.668 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.669 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.669 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.669 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T11:01:14.669 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T11:01:14.669 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.670 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.670 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.670 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.670 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.671 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.671 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.671 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T11:01:14.671 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T11:01:14.671 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.672 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.672 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.672 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.672 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.672 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.673 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.673 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T11:01:14.673 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T11:01:14.673 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.673 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.674 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.674 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.674 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.674 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.675 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T11:01:14.675 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T11:01:14.675 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.675 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:01:14.675 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:01:14.676 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.676 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.676 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.676 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.676 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T11:01:14.677 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T11:01:14.677 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.677 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.677 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T11:01:14.677 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T11:01:14.678 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.678 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.678 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.678 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T11:01:14.678 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T11:01:14.679 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.679 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.679 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:01:14.679 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:01:14.679 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:01:14.680 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.680 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.680 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.680 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T11:01:14.680 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T11:01:14.680 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.681 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.681 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:01:14.681 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:01:14.681 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:01:14.681 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.682 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.682 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.682 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T11:01:14.682 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T11:01:14.682 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.683 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:01:14.683 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:01:14.683 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:01:14.683 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:01:14.683 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.684 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.684 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.684 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T11:01:14.684 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T11:01:14.684 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.684 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:01:14.685 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:01:14.685 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:01:14.685 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.685 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.685 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.686 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.686 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T11:01:14.686 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T11:01:14.686 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.686 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:01:14.687 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:01:14.687 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.687 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.687 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:01:14.688 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:14.688 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T11:01:14.688 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T11:01:14.688 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:01:14.688 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:01:14.689 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:01:14.689 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:01:14.689 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.689 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:14.689 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T11:01:14.690 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:01:14.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.651+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9aa406b020 msgr2=0x7f9aa406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.651+0000 7f9abdde2700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9aa406b020 0x7f9aa406d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8000f60 tx=0x7f9aa8009040 comp rx=0 tx=0).stop 2023-10-16T11:01:14.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.651+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab806e5d0 msgr2=0x7f9ab80b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.651+0000 7f9abdde2700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab806e5d0 0x7f9ab80b15c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c520 tx=0x7f9aac00c8e0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 shutdown_connections 2023-10-16T11:01:14.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f9aa406b020 0x7f9aa406d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ab80729b0 0x7f9ab80ad620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ab806ab40 0x7f9ab8153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 --2- 172.21.15.158:0/463139615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ab806e5d0 0x7f9ab80b15c0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 >> 172.21.15.158:0/463139615 conn(0x7f9ab80d02b0 msgr2=0x7f9ab815d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:14.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 shutdown_connections 2023-10-16T11:01:14.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.652+0000 7f9abdde2700 1 -- 172.21.15.158:0/463139615 wait complete. 2023-10-16T11:01:14.694 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:14.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.815+0000 7ff644d8a700 1 Processor -- start 2023-10-16T11:01:14.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.815+0000 7ff644d8a700 1 -- start start 2023-10-16T11:01:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.816+0000 7ff644d8a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff64015d7c0 0x7ff6400d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.816+0000 7ff644d8a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6400d1210 0x7ff6400ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.816+0000 7ff644d8a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400d3bd0 con 0x7ff6400ce950 2023-10-16T11:01:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.816+0000 7ff644d8a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6400d4060 con 0x7ff6400d1210 2023-10-16T11:01:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.816+0000 7ff644d8a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400cede0 con 0x7ff64015d7c0 2023-10-16T11:01:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.816+0000 7ff63dd9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6400d1210 0x7ff6400ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63dd9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6400d1210 0x7ff6400ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34422/0 (socket says 172.21.15.158:34422) 2023-10-16T11:01:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63dd9b700 1 -- 172.21.15.158:0/3661727490 learned_addr learned my addr 172.21.15.158:0/3661727490 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63e59c700 1 --2- 172.21.15.158:0/3661727490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff64015d7c0 0x7ff6400d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63dd9b700 1 -- 172.21.15.158:0/3661727490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff64015d7c0 msgr2=0x7ff6400d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63dd9b700 1 --2- 172.21.15.158:0/3661727490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff64015d7c0 0x7ff6400d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63dd9b700 1 -- 172.21.15.158:0/3661727490 >> v1:172.21.15.172:6789/0 conn(0x7ff6400ce950 legacy=0x7ff6400b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63dd9b700 1 -- 172.21.15.158:0/3661727490 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400d2cf0 con 0x7ff6400d1210 2023-10-16T11:01:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63e59c700 1 --2- 172.21.15.158:0/3661727490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff64015d7c0 0x7ff6400d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63dd9b700 1 --2- 172.21.15.158:0/3661727490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6400d1210 0x7ff6400ce410 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7ff6300098f0 tx=0x7ff630009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd57677af5678a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63d59a700 1 -- 172.21.15.158:0/3661727490 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630004230 con 0x7ff6400d1210 2023-10-16T11:01:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63d59a700 1 -- 172.21.15.158:0/3661727490 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff630004390 con 0x7ff6400d1210 2023-10-16T11:01:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.817+0000 7ff63d59a700 1 -- 172.21.15.158:0/3661727490 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300036a0 con 0x7ff6400d1210 2023-10-16T11:01:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 -- 172.21.15.158:0/3661727490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6400d1210 msgr2=0x7ff6400ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 --2- 172.21.15.158:0/3661727490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6400d1210 0x7ff6400ce410 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7ff6300098f0 tx=0x7ff630009c00 comp rx=0 tx=0).stop 2023-10-16T11:01:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 -- 172.21.15.158:0/3661727490 shutdown_connections 2023-10-16T11:01:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 --2- 172.21.15.158:0/3661727490 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff64015d7c0 0x7ff6400d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 --2- 172.21.15.158:0/3661727490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6400d1210 0x7ff6400ce410 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 -- 172.21.15.158:0/3661727490 >> 172.21.15.158:0/3661727490 conn(0x7ff6400669f0 msgr2=0x7ff640066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:14.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 -- 172.21.15.158:0/3661727490 shutdown_connections 2023-10-16T11:01:14.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.818+0000 7ff644d8a700 1 -- 172.21.15.158:0/3661727490 wait complete. 2023-10-16T11:01:14.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.819+0000 7ff644d8a700 1 Processor -- start 2023-10-16T11:01:14.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.819+0000 7ff644d8a700 1 -- start start 2023-10-16T11:01:14.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff644d8a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6400ce950 0x7ff6400c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff644d8a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6400d1210 0x7ff6401587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff644d8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64015d7c0 0x7ff640158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff644d8a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400c8a50 con 0x7ff64015d7c0 2023-10-16T11:01:14.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff644d8a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400c7e40 con 0x7ff6400ce950 2023-10-16T11:01:14.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff644d8a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400c8120 con 0x7ff6400d1210 2023-10-16T11:01:14.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64015d7c0 0x7ff640158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64015d7c0 0x7ff640158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34432/0 (socket says 172.21.15.158:34432) 2023-10-16T11:01:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 -- 172.21.15.158:0/1203024132 learned_addr learned my addr 172.21.15.158:0/1203024132 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63dd9b700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6400d1210 0x7ff6401587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63e59c700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6400ce950 0x7ff6400c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 -- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6400d1210 msgr2=0x7ff6401587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6400d1210 0x7ff6401587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 -- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6400ce950 msgr2=0x7ff6400c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6400ce950 0x7ff6400c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 -- 172.21.15.158:0/1203024132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630009550 con 0x7ff64015d7c0 2023-10-16T11:01:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.820+0000 7ff63ed9d700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64015d7c0 0x7ff640158d10 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b970 tx=0x7ff63800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.821+0000 7ff636ffd700 1 -- 172.21.15.158:0/1203024132 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63800d860 con 0x7ff64015d7c0 2023-10-16T11:01:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.821+0000 7ff636ffd700 1 -- 172.21.15.158:0/1203024132 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63800d9c0 con 0x7ff64015d7c0 2023-10-16T11:01:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.821+0000 7ff636ffd700 1 -- 172.21.15.158:0/1203024132 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63800dd70 con 0x7ff64015d7c0 2023-10-16T11:01:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.821+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6400c8400 con 0x7ff64015d7c0 2023-10-16T11:01:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.821+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6401581f0 con 0x7ff64015d7c0 2023-10-16T11:01:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.822+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6400c8b90 con 0x7ff64015d7c0 2023-10-16T11:01:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.822+0000 7ff636ffd700 1 -- 172.21.15.158:0/1203024132 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff63800e430 con 0x7ff64015d7c0 2023-10-16T11:01:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.822+0000 7ff636ffd700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff62006b020 0x7ff62006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.822+0000 7ff636ffd700 1 -- 172.21.15.158:0/1203024132 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5629+0+0 (secure 0 0 0) 0x7ff6380898b0 con 0x7ff64015d7c0 2023-10-16T11:01:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.822+0000 7ff63e59c700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff62006b020 0x7ff62006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.823+0000 7ff63e59c700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff62006b020 0x7ff62006d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff628006fd0 tx=0x7ff628008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:14.827+0000 7ff636ffd700 1 -- 172.21.15.158:0/1203024132 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6400c8b90 con 0x7ff64015d7c0 2023-10-16T11:01:15.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.020+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6400c8b90 con 0x7ff64015d7c0 2023-10-16T11:01:15.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.021+0000 7ff636ffd700 1 -- 172.21.15.158:0/1203024132 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+13774 (secure 0 0 0) 0x7ff6400c8b90 con 0x7ff64015d7c0 2023-10-16T11:01:15.023 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:15.023 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":181,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:01:14.240811+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:15.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff62006b020 msgr2=0x7ff62006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff62006b020 0x7ff62006d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff628006fd0 tx=0x7ff628008040 comp rx=0 tx=0).stop 2023-10-16T11:01:15.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64015d7c0 msgr2=0x7ff640158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64015d7c0 0x7ff640158d10 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b970 tx=0x7ff63800bc80 comp rx=0 tx=0).stop 2023-10-16T11:01:15.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 shutdown_connections 2023-10-16T11:01:15.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff62006b020 0x7ff62006d4e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64015d7c0 0x7ff640158d10 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6400ce950 0x7ff6400c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 --2- 172.21.15.158:0/1203024132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6400d1210 0x7ff6401587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 >> 172.21.15.158:0/1203024132 conn(0x7ff6400669f0 msgr2=0x7ff64015be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:15.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 shutdown_connections 2023-10-16T11:01:15.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.023+0000 7ff644d8a700 1 -- 172.21.15.158:0/1203024132 wait complete. 2023-10-16T11:01:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:01:15.038 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:01:15.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.161+0000 7f2177b20700 1 Processor -- start 2023-10-16T11:01:15.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.161+0000 7f2177b20700 1 -- start start 2023-10-16T11:01:15.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f2177b20700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f217015da40 0x7f21700caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f2177b20700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21700cc590 0x7f217006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f2177b20700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21700cbd30 con 0x7f217006ae10 2023-10-16T11:01:15.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f2177b20700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21700cc1c0 con 0x7f217015da40 2023-10-16T11:01:15.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f2177b20700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217006b240 con 0x7f21700cc590 2023-10-16T11:01:15.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f217015da40 0x7f21700caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f217015da40 0x7f21700caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34436/0 (socket says 172.21.15.158:34436) 2023-10-16T11:01:15.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 -- 172.21.15.158:0/3443880508 learned_addr learned my addr 172.21.15.158:0/3443880508 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:15.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 -- 172.21.15.158:0/3443880508 >> v2:172.21.15.172:3300/0 conn(0x7f21700cc590 msgr2=0x7f217006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:15.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 --2- 172.21.15.158:0/3443880508 >> v2:172.21.15.172:3300/0 conn(0x7f21700cc590 0x7f217006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 -- 172.21.15.158:0/3443880508 >> v1:172.21.15.172:6789/0 conn(0x7f217006ae10 legacy=0x7f21700ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 -- 172.21.15.158:0/3443880508 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217015de20 con 0x7f217015da40 2023-10-16T11:01:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21758bc700 1 --2- 172.21.15.158:0/3443880508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f217015da40 0x7f21700caa20 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f21700d1380 tx=0x7f216c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44b7a94bd8f7acb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.162+0000 7f21748ba700 1 -- 172.21.15.158:0/3443880508 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c004230 con 0x7f217015da40 2023-10-16T11:01:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f21748ba700 1 -- 172.21.15.158:0/3443880508 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c004390 con 0x7f217015da40 2023-10-16T11:01:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f21748ba700 1 -- 172.21.15.158:0/3443880508 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c0036a0 con 0x7f217015da40 2023-10-16T11:01:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 -- 172.21.15.158:0/3443880508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f217015da40 msgr2=0x7f21700caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 --2- 172.21.15.158:0/3443880508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f217015da40 0x7f21700caa20 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f21700d1380 tx=0x7f216c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 -- 172.21.15.158:0/3443880508 shutdown_connections 2023-10-16T11:01:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 --2- 172.21.15.158:0/3443880508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f217015da40 0x7f21700caa20 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 --2- 172.21.15.158:0/3443880508 >> v2:172.21.15.172:3300/0 conn(0x7f21700cc590 0x7f217006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 -- 172.21.15.158:0/3443880508 >> 172.21.15.158:0/3443880508 conn(0x7f2170099a10 msgr2=0x7f21700d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 -- 172.21.15.158:0/3443880508 shutdown_connections 2023-10-16T11:01:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.163+0000 7f2177b20700 1 -- 172.21.15.158:0/3443880508 wait complete. 2023-10-16T11:01:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.164+0000 7f2177b20700 1 Processor -- start 2023-10-16T11:01:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.164+0000 7f2177b20700 1 -- start start 2023-10-16T11:01:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f2177b20700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f217006ae10 0x7f21700c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f2177b20700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21700cc590 0x7f21701f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f2177b20700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f217015da40 0x7f21700af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f2177b20700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21700ad080 con 0x7f21700cc590 2023-10-16T11:01:15.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f2177b20700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21701f31b0 con 0x7f217006ae10 2023-10-16T11:01:15.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f2177b20700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21700af840 con 0x7f217015da40 2023-10-16T11:01:15.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f21750bb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21700cc590 0x7f21701f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f21750bb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21700cc590 0x7f21701f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34444/0 (socket says 172.21.15.158:34444) 2023-10-16T11:01:15.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f21750bb700 1 -- 172.21.15.158:0/1725915705 learned_addr learned my addr 172.21.15.158:0/1725915705 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:15.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f21758bc700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f217006ae10 0x7f21700c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.165+0000 7f21760bd700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f217015da40 0x7f21700af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21750bb700 1 -- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f217015da40 msgr2=0x7f21700af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21750bb700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f217015da40 0x7f21700af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21750bb700 1 -- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f217006ae10 msgr2=0x7f21700c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21750bb700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f217006ae10 0x7f21700c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21750bb700 1 -- 172.21.15.158:0/1725915705 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c009550 con 0x7f21700cc590 2023-10-16T11:01:15.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21750bb700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21700cc590 0x7f21701f2c70 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f216000b500 tx=0x7f216000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21667fc700 1 -- 172.21.15.158:0/1725915705 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160011830 con 0x7f21700cc590 2023-10-16T11:01:15.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21667fc700 1 -- 172.21.15.158:0/1725915705 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21600119b0 con 0x7f21700cc590 2023-10-16T11:01:15.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f21667fc700 1 -- 172.21.15.158:0/1725915705 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160011d60 con 0x7f21700cc590 2023-10-16T11:01:15.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21701f6750 con 0x7f21700cc590 2023-10-16T11:01:15.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.166+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21701f6d10 con 0x7f21700cc590 2023-10-16T11:01:15.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.167+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21700ad1f0 con 0x7f21700cc590 2023-10-16T11:01:15.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.167+0000 7f21667fc700 1 -- 172.21.15.158:0/1725915705 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f216001a400 con 0x7f21700cc590 2023-10-16T11:01:15.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.168+0000 7f21667fc700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f215406b020 0x7f215406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.168+0000 7f21667fc700 1 -- 172.21.15.158:0/1725915705 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f21600897f0 con 0x7f21700cc590 2023-10-16T11:01:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.168+0000 7f21758bc700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f215406b020 0x7f215406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.168+0000 7f21758bc700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f215406b020 0x7f215406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f216c009130 tx=0x7f216c008030 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.172+0000 7f21667fc700 1 -- 172.21.15.158:0/1725915705 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21700ad1f0 con 0x7f21700cc590 2023-10-16T11:01:15.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.369+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21700ad1f0 con 0x7f21700cc590 2023-10-16T11:01:15.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.370+0000 7f21667fc700 1 -- 172.21.15.158:0/1725915705 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v181) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21700ad1f0 con 0x7f21700cc590 2023-10-16T11:01:15.371 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:01:15.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.371+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f215406b020 msgr2=0x7f215406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.371+0000 7f2177b20700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f215406b020 0x7f215406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f216c009130 tx=0x7f216c008030 comp rx=0 tx=0).stop 2023-10-16T11:01:15.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.371+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21700cc590 msgr2=0x7f21701f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.371+0000 7f2177b20700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21700cc590 0x7f21701f2c70 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f216000b500 tx=0x7f216000b8c0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 shutdown_connections 2023-10-16T11:01:15.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f215406b020 0x7f215406d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f217015da40 0x7f21700af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f217006ae10 0x7f21700c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 --2- 172.21.15.158:0/1725915705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21700cc590 0x7f21701f2c70 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 >> 172.21.15.158:0/1725915705 conn(0x7f2170099a10 msgr2=0x7f21700ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 shutdown_connections 2023-10-16T11:01:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.372+0000 7f2177b20700 1 -- 172.21.15.158:0/1725915705 wait complete. 2023-10-16T11:01:15.385 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:01:15.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f2301fb3700 1 Processor -- start 2023-10-16T11:01:15.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f2301fb3700 1 -- start start 2023-10-16T11:01:15.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f2301fb3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22fc072360 0x7f22fc072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f2301fb3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f22fc06a8d0 0x7f22fc06ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f2301fb3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc06f850 con 0x7f22fc06f470 2023-10-16T11:01:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f2301fb3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc072d50 con 0x7f22fc072360 2023-10-16T11:01:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f2301fb3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc06b220 con 0x7f22fc06a8d0 2023-10-16T11:01:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.505+0000 7f22faffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22fc072360 0x7f22fc072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22faffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22fc072360 0x7f22fc072810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34454/0 (socket says 172.21.15.158:34454) 2023-10-16T11:01:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22faffd700 1 -- 172.21.15.158:0/1668181142 learned_addr learned my addr 172.21.15.158:0/1668181142 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22faffd700 1 -- 172.21.15.158:0/1668181142 >> v2:172.21.15.172:3300/0 conn(0x7f22fc06a8d0 msgr2=0x7f22fc06ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22faffd700 1 --2- 172.21.15.158:0/1668181142 >> v2:172.21.15.172:3300/0 conn(0x7f22fc06a8d0 0x7f22fc06ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22faffd700 1 -- 172.21.15.158:0/1668181142 >> v1:172.21.15.172:6789/0 conn(0x7f22fc06f470 legacy=0x7f22fc06e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22faffd700 1 -- 172.21.15.158:0/1668181142 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0b4260 con 0x7f22fc072360 2023-10-16T11:01:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22faffd700 1 --2- 172.21.15.158:0/1668181142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc072360 0x7f22fc072810 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0ca900 tx=0x7f22e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07f6ceb39947f29 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22fa7fc700 1 -- 172.21.15.158:0/1668181142 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e4004230 con 0x7f22fc072360 2023-10-16T11:01:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22fa7fc700 1 -- 172.21.15.158:0/1668181142 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e4004390 con 0x7f22fc072360 2023-10-16T11:01:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.506+0000 7f22fa7fc700 1 -- 172.21.15.158:0/1668181142 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e40036a0 con 0x7f22fc072360 2023-10-16T11:01:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 -- 172.21.15.158:0/1668181142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc072360 msgr2=0x7f22fc072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 --2- 172.21.15.158:0/1668181142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc072360 0x7f22fc072810 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0ca900 tx=0x7f22e4009bd0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 -- 172.21.15.158:0/1668181142 shutdown_connections 2023-10-16T11:01:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 --2- 172.21.15.158:0/1668181142 >> v2:172.21.15.172:3300/0 conn(0x7f22fc06a8d0 0x7f22fc06ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 --2- 172.21.15.158:0/1668181142 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc072360 0x7f22fc072810 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 -- 172.21.15.158:0/1668181142 >> 172.21.15.158:0/1668181142 conn(0x7f22fc0ccda0 msgr2=0x7f22fc0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 -- 172.21.15.158:0/1668181142 shutdown_connections 2023-10-16T11:01:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.507+0000 7f2301fb3700 1 -- 172.21.15.158:0/1668181142 wait complete. 2023-10-16T11:01:15.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.508+0000 7f2301fb3700 1 Processor -- start 2023-10-16T11:01:15.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.508+0000 7f2301fb3700 1 -- start start 2023-10-16T11:01:15.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f2301fb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc06a8d0 0x7f22fc0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f2301fb3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc06f470 0x7f22fc1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f2301fb3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc072360 0x7f22fc0c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f2301fb3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0c4ad0 con 0x7f22fc06a8d0 2023-10-16T11:01:15.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f2301fb3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0c99d0 con 0x7f22fc072360 2023-10-16T11:01:15.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f2301fb3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc1f2ed0 con 0x7f22fc06f470 2023-10-16T11:01:15.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f22fb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc06a8d0 0x7f22fc0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f22fb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc06a8d0 0x7f22fc0ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34458/0 (socket says 172.21.15.158:34458) 2023-10-16T11:01:15.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.509+0000 7f22fb7fe700 1 -- 172.21.15.158:0/4180095486 learned_addr learned my addr 172.21.15.158:0/4180095486 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:15.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22faffd700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc06f470 0x7f22fc1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22fbfff700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc072360 0x7f22fc0c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22fb7fe700 1 -- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc06f470 msgr2=0x7f22fc1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22fb7fe700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc06f470 0x7f22fc1540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22fb7fe700 1 -- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc072360 msgr2=0x7f22fc0c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22fb7fe700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc072360 0x7f22fc0c5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22fb7fe700 1 -- 172.21.15.158:0/4180095486 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e4009550 con 0x7f22fc06a8d0 2023-10-16T11:01:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22fb7fe700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc06a8d0 0x7f22fc0ce820 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00c520 tx=0x7f22ec00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22f3fff700 1 -- 172.21.15.158:0/4180095486 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0054e0 con 0x7f22fc06a8d0 2023-10-16T11:01:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22f3fff700 1 -- 172.21.15.158:0/4180095486 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec0036d0 con 0x7f22fc06a8d0 2023-10-16T11:01:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc154630 con 0x7f22fc06a8d0 2023-10-16T11:01:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f22f3fff700 1 -- 172.21.15.158:0/4180095486 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec003a80 con 0x7f22fc06a8d0 2023-10-16T11:01:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.510+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc0c4410 con 0x7f22fc06a8d0 2023-10-16T11:01:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.511+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc0681b0 con 0x7f22fc06a8d0 2023-10-16T11:01:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.512+0000 7f22f3fff700 1 -- 172.21.15.158:0/4180095486 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f22ec003be0 con 0x7f22fc06a8d0 2023-10-16T11:01:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.512+0000 7f22f3fff700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f22dc06b020 0x7f22dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.512+0000 7f22f3fff700 1 -- 172.21.15.158:0/4180095486 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f22ec088b60 con 0x7f22fc06a8d0 2023-10-16T11:01:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.512+0000 7f22faffd700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f22dc06b020 0x7f22dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.512+0000 7f22faffd700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f22dc06b020 0x7f22dc06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0c5da0 tx=0x7f22e4008030 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.516+0000 7f22f3fff700 1 -- 172.21.15.158:0/4180095486 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22ec053be0 con 0x7f22fc06a8d0 2023-10-16T11:01:15.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.711+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f22fc161660 con 0x7f22fc06a8d0 2023-10-16T11:01:15.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.712+0000 7f22f3fff700 1 -- 172.21.15.158:0/4180095486 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v181) v1 ==== 84+0+65 (secure 0 0 0) 0x7f22ec017020 con 0x7f22fc06a8d0 2023-10-16T11:01:15.713 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:15.714 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:01:15.714 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:15.714 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:01:15.714 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:01:15.715 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:15.715 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:01:15.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f22dc06b020 msgr2=0x7f22dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f22dc06b020 0x7f22dc06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0c5da0 tx=0x7f22e4008030 comp rx=0 tx=0).stop 2023-10-16T11:01:15.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc06a8d0 msgr2=0x7f22fc0ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc06a8d0 0x7f22fc0ce820 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00c520 tx=0x7f22ec00c8e0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 shutdown_connections 2023-10-16T11:01:15.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f22dc06b020 0x7f22dc06d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc072360 0x7f22fc0c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc06a8d0 0x7f22fc0ce820 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 --2- 172.21.15.158:0/4180095486 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc06f470 0x7f22fc1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 >> 172.21.15.158:0/4180095486 conn(0x7f22fc0ccda0 msgr2=0x7f22fc0b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:15.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 shutdown_connections 2023-10-16T11:01:15.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.714+0000 7f2301fb3700 1 -- 172.21.15.158:0/4180095486 wait complete. 2023-10-16T11:01:15.727 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T11:01:15.728 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T11:01:15.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.851+0000 7f52e5513700 1 Processor -- start 2023-10-16T11:01:15.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.851+0000 7f52e5513700 1 -- start start 2023-10-16T11:01:15.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.852+0000 7f52e5513700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f52e015b6b0 0x7f52e00cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.852+0000 7f52e5513700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00b4420 con 0x7f52e00d0c60 2023-10-16T11:01:15.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.852+0000 7f52e5513700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00cbc20 con 0x7f52e00b3630 2023-10-16T11:01:15.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.852+0000 7f52e5513700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e015ba90 con 0x7f52e015b6b0 2023-10-16T11:01:15.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.852+0000 7f52de7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f52e00d0c60 0x7f52e00b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34646/0 (socket says 172.21.15.158:34646) 2023-10-16T11:01:15.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.852+0000 7f52de7fc700 1 -- 172.21.15.158:0/1946763471 learned_addr learned my addr 172.21.15.158:0/1946763471 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:15.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52deffd700 1 --2- 172.21.15.158:0/1946763471 >> v2:172.21.15.172:3300/0 conn(0x7f52e015b6b0 0x7f52e00cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320467753 0 0) 0x7f52e00b4420 con 0x7f52e00d0c60 2023-10-16T11:01:15.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c80035b0 con 0x7f52e00d0c60 2023-10-16T11:01:15.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046263945 0 0) 0x7f52c80035b0 con 0x7f52e00d0c60 2023-10-16T11:01:15.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 >> v2:172.21.15.172:3300/0 conn(0x7f52e015b6b0 msgr2=0x7f52e00cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:15.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 --2- 172.21.15.158:0/1946763471 >> v2:172.21.15.172:3300/0 conn(0x7f52e015b6b0 0x7f52e00cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 >> v1:172.21.15.172:6789/0 conn(0x7f52e00b3630 legacy=0x7f52e00b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00cbf00 con 0x7f52e00d0c60 2023-10-16T11:01:15.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f52d0001670 con 0x7f52e00d0c60 2023-10-16T11:01:15.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 >> v1:172.21.15.158:6789/0 conn(0x7f52e00d0c60 legacy=0x7f52e00b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8003a00 0x7f52c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52c8004850 0x7f52c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c8005210 0x7f52c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c800c180 con 0x7f52c8004850 2023-10-16T11:01:15.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c800c2f0 con 0x7f52c8005210 2023-10-16T11:01:15.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c800c4b0 con 0x7f52c8003a00 2023-10-16T11:01:15.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52de7fc700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52c8004850 0x7f52c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52deffd700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8003a00 0x7f52c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52df7fe700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c8005210 0x7f52c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52de7fc700 1 -- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8003a00 msgr2=0x7f52c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:15.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52de7fc700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8003a00 0x7f52c8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52de7fc700 1 -- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c8005210 msgr2=0x7f52c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:15.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52de7fc700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c8005210 0x7f52c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52de7fc700 1 -- 172.21.15.158:0/1946763471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80035b0 con 0x7f52c8004850 2023-10-16T11:01:15.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52df7fe700 1 -- 172.21.15.158:0/1946763471 reap_dead start 2023-10-16T11:01:15.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52deffd700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8003a00 0x7f52c8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:15.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.853+0000 7f52de7fc700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52c8004850 0x7f52c8004cd0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f52d000c210 tx=0x7f52c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e4e9bb8dc9dd27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0007c10 con 0x7f52c8004850 2023-10-16T11:01:15.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d000cdb0 con 0x7f52c8004850 2023-10-16T11:01:15.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52ddffb700 1 -- 172.21.15.158:0/1946763471 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0005ba0 con 0x7f52c8004850 2023-10-16T11:01:15.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52e5513700 1 -- 172.21.15.158:0/1946763471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52c8004850 msgr2=0x7f52c8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52e5513700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52c8004850 0x7f52c8004cd0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f52d000c210 tx=0x7f52c8005880 comp rx=0 tx=0).stop 2023-10-16T11:01:15.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52e5513700 1 -- 172.21.15.158:0/1946763471 shutdown_connections 2023-10-16T11:01:15.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52e5513700 1 --2- 172.21.15.158:0/1946763471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52c8004850 0x7f52c8004cd0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52e5513700 1 -- 172.21.15.158:0/1946763471 >> 172.21.15.158:0/1946763471 conn(0x7f52e00cdf20 msgr2=0x7f52e00ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:15.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52e5513700 1 -- 172.21.15.158:0/1946763471 shutdown_connections 2023-10-16T11:01:15.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.854+0000 7f52e5513700 1 -- 172.21.15.158:0/1946763471 wait complete. 2023-10-16T11:01:15.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.855+0000 7f52e5513700 1 Processor -- start 2023-10-16T11:01:15.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52e5513700 1 -- start start 2023-10-16T11:01:15.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52e5513700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8004850 0x7f52e0155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52e5513700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52e00ca5a0 0x7f52e01f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52e5513700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52e00caad0 0x7f52e00a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52e5513700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e01561e0 con 0x7f52e00ca5a0 2023-10-16T11:01:15.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52e5513700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00c8610 con 0x7f52e00caad0 2023-10-16T11:01:15.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52e5513700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00c88b0 con 0x7f52c8004850 2023-10-16T11:01:15.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52de7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52e00ca5a0 0x7f52e01f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52de7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52e00ca5a0 0x7f52e01f1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34480/0 (socket says 172.21.15.158:34480) 2023-10-16T11:01:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.856+0000 7f52de7fc700 1 -- 172.21.15.158:0/2595973350 learned_addr learned my addr 172.21.15.158:0/2595973350 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52deffd700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8004850 0x7f52e0155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52df7fe700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52e00caad0 0x7f52e00a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52de7fc700 1 -- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8004850 msgr2=0x7f52e0155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52de7fc700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8004850 0x7f52e0155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52de7fc700 1 -- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52e00caad0 msgr2=0x7f52e00a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52de7fc700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52e00caad0 0x7f52e00a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52de7fc700 1 -- 172.21.15.158:0/2595973350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00c8b90 con 0x7f52e00ca5a0 2023-10-16T11:01:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52de7fc700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52e00ca5a0 0x7f52e01f1e00 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c2c0 tx=0x7f52d0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52c77fe700 1 -- 172.21.15.158:0/2595973350 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0000ed0 con 0x7f52e00ca5a0 2023-10-16T11:01:15.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52c77fe700 1 -- 172.21.15.158:0/2595973350 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d00070d0 con 0x7f52e00ca5a0 2023-10-16T11:01:15.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e00c8e70 con 0x7f52e00ca5a0 2023-10-16T11:01:15.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52c77fe700 1 -- 172.21.15.158:0/2595973350 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0007460 con 0x7f52e00ca5a0 2023-10-16T11:01:15.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.857+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e01f2340 con 0x7f52e00ca5a0 2023-10-16T11:01:15.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.858+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e01f24d0 con 0x7f52e00ca5a0 2023-10-16T11:01:15.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.858+0000 7f52c77fe700 1 -- 172.21.15.158:0/2595973350 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f52d00075c0 con 0x7f52e00ca5a0 2023-10-16T11:01:15.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.859+0000 7f52c77fe700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f52cc06b020 0x7f52cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.859+0000 7f52c77fe700 1 -- 172.21.15.158:0/2595973350 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f52d008a680 con 0x7f52e00ca5a0 2023-10-16T11:01:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.859+0000 7f52deffd700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f52cc06b020 0x7f52cc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.859+0000 7f52deffd700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f52cc06b020 0x7f52cc06d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52e00b0a00 tx=0x7f52d4008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:15.863+0000 7f52c77fe700 1 -- 172.21.15.158:0/2595973350 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52d0055700 con 0x7f52e00ca5a0 2023-10-16T11:01:16.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.056+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f52e00681b0 con 0x7f52e00ca5a0 2023-10-16T11:01:16.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.229+0000 7f52c77fe700 1 -- 172.21.15.158:0/2595973350 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v182) v1 ==== 138+0+0 (secure 0 0 0) 0x7f52d0058d20 con 0x7f52e00ca5a0 2023-10-16T11:01:16.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f52cc06b020 msgr2=0x7f52cc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:16.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f52cc06b020 0x7f52cc06d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52e00b0a00 tx=0x7f52d4008040 comp rx=0 tx=0).stop 2023-10-16T11:01:16.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52e00ca5a0 msgr2=0x7f52e01f1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:16.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52e00ca5a0 0x7f52e01f1e00 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c2c0 tx=0x7f52d0001a00 comp rx=0 tx=0).stop 2023-10-16T11:01:16.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 shutdown_connections 2023-10-16T11:01:16.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f52cc06b020 0x7f52cc06d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52e00caad0 0x7f52e00a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c8004850 0x7f52e0155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 --2- 172.21.15.158:0/2595973350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52e00ca5a0 0x7f52e01f1e00 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 >> 172.21.15.158:0/2595973350 conn(0x7f52e00cdf20 msgr2=0x7f52e00ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:16.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 shutdown_connections 2023-10-16T11:01:16.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.231+0000 7f52e5513700 1 -- 172.21.15.158:0/2595973350 wait complete. 2023-10-16T11:01:16.236 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T11:01:16.249 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:01:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.375+0000 7fd92dbfb700 1 Processor -- start 2023-10-16T11:01:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.375+0000 7fd92dbfb700 1 -- start start 2023-10-16T11:01:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd92dbfb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd928070410 0x7fd928070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd92dbfb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd9280cf6b0 0x7fd9280cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd92dbfb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd928070db0 con 0x7fd9280cbf50 2023-10-16T11:01:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd92dbfb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd928160220 con 0x7fd9280cf6b0 2023-10-16T11:01:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd92dbfb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92815f6a0 con 0x7fd928070410 2023-10-16T11:01:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd927fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd9280cf6b0 0x7fd9280cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd927fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd9280cf6b0 0x7fd9280cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34490/0 (socket says 172.21.15.158:34490) 2023-10-16T11:01:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd9277fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd9280cbf50 0x7fd928144b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34662/0 (socket says 172.21.15.158:34662) 2023-10-16T11:01:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd927fff700 1 -- 172.21.15.158:0/865537559 learned_addr learned my addr 172.21.15.158:0/865537559 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd926ffd700 1 --2- 172.21.15.158:0/865537559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd928070410 0x7fd928070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd927fff700 1 -- 172.21.15.158:0/865537559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd928070410 msgr2=0x7fd928070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd927fff700 1 --2- 172.21.15.158:0/865537559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd928070410 0x7fd928070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd927fff700 1 -- 172.21.15.158:0/865537559 >> v1:172.21.15.158:6789/0 conn(0x7fd9280cbf50 legacy=0x7fd928144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd927fff700 1 -- 172.21.15.158:0/865537559 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92815f980 con 0x7fd9280cf6b0 2023-10-16T11:01:16.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.376+0000 7fd9267fc700 1 -- 172.21.15.158:0/865537559 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190535614 0 0) 0x7fd928070db0 con 0x7fd9280cbf50 2023-10-16T11:01:16.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd926ffd700 1 --2- 172.21.15.158:0/865537559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd928070410 0x7fd928070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:16.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd927fff700 1 --2- 172.21.15.158:0/865537559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9280cf6b0 0x7fd9280cfac0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fd920009e50 tx=0x7fd92000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c124f1f3c05439 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:16.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd9267fc700 1 -- 172.21.15.158:0/865537559 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000e070 con 0x7fd9280cf6b0 2023-10-16T11:01:16.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd9267fc700 1 -- 172.21.15.158:0/865537559 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92000b9c0 con 0x7fd9280cf6b0 2023-10-16T11:01:16.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd9267fc700 1 -- 172.21.15.158:0/865537559 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000bd70 con 0x7fd9280cf6b0 2023-10-16T11:01:16.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 -- 172.21.15.158:0/865537559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9280cf6b0 msgr2=0x7fd9280cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:16.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/865537559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9280cf6b0 0x7fd9280cfac0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fd920009e50 tx=0x7fd92000b330 comp rx=0 tx=0).stop 2023-10-16T11:01:16.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 -- 172.21.15.158:0/865537559 shutdown_connections 2023-10-16T11:01:16.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/865537559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9280cf6b0 0x7fd9280cfac0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/865537559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd928070410 0x7fd928070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 -- 172.21.15.158:0/865537559 >> 172.21.15.158:0/865537559 conn(0x7fd9280d1410 msgr2=0x7fd9280d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:16.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 -- 172.21.15.158:0/865537559 shutdown_connections 2023-10-16T11:01:16.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.377+0000 7fd92dbfb700 1 -- 172.21.15.158:0/865537559 wait complete. 2023-10-16T11:01:16.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 Processor -- start 2023-10-16T11:01:16.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 -- start start 2023-10-16T11:01:16.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd928070410 0x7fd92815d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:16.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9280cbf50 0x7fd92815bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:16.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9280cf6b0 0x7fd9280b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:16.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92815dd40 con 0x7fd928070410 2023-10-16T11:01:16.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92815df90 con 0x7fd9280cbf50 2023-10-16T11:01:16.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.379+0000 7fd92dbfb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92815e270 con 0x7fd9280cf6b0 2023-10-16T11:01:16.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd928070410 0x7fd92815d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:16.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd928070410 0x7fd92815d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34492/0 (socket says 172.21.15.158:34492) 2023-10-16T11:01:16.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 -- 172.21.15.158:0/729392148 learned_addr learned my addr 172.21.15.158:0/729392148 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:16.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd927fff700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9280cf6b0 0x7fd9280b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:16.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd926ffd700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9280cbf50 0x7fd92815bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:16.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 -- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9280cf6b0 msgr2=0x7fd9280b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:16.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9280cf6b0 0x7fd9280b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 -- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9280cbf50 msgr2=0x7fd92815bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:16.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9280cbf50 0x7fd92815bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:16.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 -- 172.21.15.158:0/729392148 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920009b30 con 0x7fd928070410 2023-10-16T11:01:16.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd927fff700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9280cf6b0 0x7fd9280b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:16.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd9277fe700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd928070410 0x7fd92815d800 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0011e0 tx=0x7fd91c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:16.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd90ffff700 1 -- 172.21.15.158:0/729392148 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c004c80 con 0x7fd928070410 2023-10-16T11:01:16.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd90ffff700 1 -- 172.21.15.158:0/729392148 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91c004e00 con 0x7fd928070410 2023-10-16T11:01:16.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92815e550 con 0x7fd928070410 2023-10-16T11:01:16.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd90ffff700 1 -- 172.21.15.158:0/729392148 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c007310 con 0x7fd928070410 2023-10-16T11:01:16.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.380+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92815eb10 con 0x7fd928070410 2023-10-16T11:01:16.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.381+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92815c390 con 0x7fd928070410 2023-10-16T11:01:16.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.382+0000 7fd90ffff700 1 -- 172.21.15.158:0/729392148 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd91c007470 con 0x7fd928070410 2023-10-16T11:01:16.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.382+0000 7fd90ffff700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd91406b020 0x7fd91406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:16.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.382+0000 7fd90ffff700 1 -- 172.21.15.158:0/729392148 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5999+0+0 (secure 0 0 0) 0x7fd91c089660 con 0x7fd928070410 2023-10-16T11:01:16.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.382+0000 7fd926ffd700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd91406b020 0x7fd91406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:16.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.383+0000 7fd926ffd700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd91406b020 0x7fd91406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c900 tx=0x7fd918008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:16.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.387+0000 7fd90ffff700 1 -- 172.21.15.158:0/729392148 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd91c0544d0 con 0x7fd928070410 2023-10-16T11:01:16.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:16.577+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd9280681b0 con 0x7fd928070410 2023-10-16T11:01:17.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.234+0000 7fd90ffff700 1 -- 172.21.15.158:0/729392148 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v183) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd91c057af0 con 0x7fd928070410 2023-10-16T11:01:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.240+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd91406b020 msgr2=0x7fd91406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.240+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd91406b020 0x7fd91406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c900 tx=0x7fd918008040 comp rx=0 tx=0).stop 2023-10-16T11:01:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.240+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd928070410 msgr2=0x7fd92815d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.240+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd928070410 0x7fd92815d800 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0011e0 tx=0x7fd91c00cae0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.243+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 shutdown_connections 2023-10-16T11:01:17.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.243+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fd91406b020 0x7fd91406d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.243+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9280cf6b0 0x7fd9280b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.243+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd928070410 0x7fd92815d800 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.243+0000 7fd92dbfb700 1 --2- 172.21.15.158:0/729392148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9280cbf50 0x7fd92815bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.243+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 >> 172.21.15.158:0/729392148 conn(0x7fd9280d1410 msgr2=0x7fd9280c57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:17.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.244+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 shutdown_connections 2023-10-16T11:01:17.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.244+0000 7fd92dbfb700 1 -- 172.21.15.158:0/729392148 wait complete. 2023-10-16T11:01:17.256 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T11:01:17.267 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T11:01:17.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.392+0000 7fcd03ac1700 1 Processor -- start 2023-10-16T11:01:17.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.392+0000 7fcd03ac1700 1 -- start start 2023-10-16T11:01:17.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd03ac1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fccfc15da40 0x7fccfc0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:17.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd03ac1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccfc0cbd30 con 0x7fccfc0cc590 2023-10-16T11:01:17.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd03ac1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccfc0cc1c0 con 0x7fccfc06afe0 2023-10-16T11:01:17.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd03ac1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccfc15de20 con 0x7fccfc15da40 2023-10-16T11:01:17.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0185d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fccfc15da40 0x7fccfc0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:17.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0105c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fccfc0cc590 0x7fccfc06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34676/0 (socket says 172.21.15.158:34676) 2023-10-16T11:01:17.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0105c700 1 -- 172.21.15.158:0/899505536 learned_addr learned my addr 172.21.15.158:0/899505536 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:17.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0185d700 1 -- 172.21.15.158:0/899505536 >> v1:172.21.15.172:6789/0 conn(0x7fccfc06afe0 legacy=0x7fccfc0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:17.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0185d700 1 -- 172.21.15.158:0/899505536 >> v1:172.21.15.158:6789/0 conn(0x7fccfc0cc590 legacy=0x7fccfc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:17.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0185d700 1 -- 172.21.15.158:0/899505536 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc0ce260 con 0x7fccfc15da40 2023-10-16T11:01:17.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0085b700 1 -- 172.21.15.158:0/899505536 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228795322 0 0) 0x7fccfc0cbd30 con 0x7fccfc0cc590 2023-10-16T11:01:17.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0185d700 1 --2- 172.21.15.158:0/899505536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0caa20 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009840 tx=0x7fccf8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87f5c39e10510cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:17.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.393+0000 7fcd0085b700 1 -- 172.21.15.158:0/899505536 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8011830 con 0x7fccfc15da40 2023-10-16T11:01:17.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd0085b700 1 -- 172.21.15.158:0/899505536 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf8011990 con 0x7fccfc15da40 2023-10-16T11:01:17.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd0085b700 1 -- 172.21.15.158:0/899505536 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8011d20 con 0x7fccfc15da40 2023-10-16T11:01:17.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd03ac1700 1 -- 172.21.15.158:0/899505536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 msgr2=0x7fccfc0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:17.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/899505536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0caa20 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009840 tx=0x7fccf8009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:17.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd03ac1700 1 -- 172.21.15.158:0/899505536 shutdown_connections 2023-10-16T11:01:17.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/899505536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0caa20 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd03ac1700 1 -- 172.21.15.158:0/899505536 >> 172.21.15.158:0/899505536 conn(0x7fccfc099a10 msgr2=0x7fccfc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:17.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd03ac1700 1 -- 172.21.15.158:0/899505536 shutdown_connections 2023-10-16T11:01:17.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.394+0000 7fcd03ac1700 1 -- 172.21.15.158:0/899505536 wait complete. 2023-10-16T11:01:17.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.395+0000 7fcd03ac1700 1 Processor -- start 2023-10-16T11:01:17.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.395+0000 7fcd03ac1700 1 -- start start 2023-10-16T11:01:17.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd03ac1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccfc06afe0 0x7fccfc15a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:17.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd03ac1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccfc0cc590 0x7fccfc1f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:17.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd03ac1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:17.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd03ac1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc0c4e00 con 0x7fccfc15da40 2023-10-16T11:01:17.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd03ac1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc1f7170 con 0x7fccfc06afe0 2023-10-16T11:01:17.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd03ac1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc0c41f0 con 0x7fccfc0cc590 2023-10-16T11:01:17.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:17.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34510/0 (socket says 172.21.15.158:34510) 2023-10-16T11:01:17.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 -- 172.21.15.158:0/1122332665 learned_addr learned my addr 172.21.15.158:0/1122332665 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:17.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0105c700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccfc0cc590 0x7fccfc1f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:17.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0185d700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccfc06afe0 0x7fccfc15a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:17.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 -- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccfc0cc590 msgr2=0x7fccfc1f6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:17.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccfc0cc590 0x7fccfc1f6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 -- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccfc06afe0 msgr2=0x7fccfc15a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:17.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccfc06afe0 0x7fccfc15a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:17.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.396+0000 7fcd0205e700 1 -- 172.21.15.158:0/1122332665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf80094a0 con 0x7fccfc15da40 2023-10-16T11:01:17.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.397+0000 7fcd0205e700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0b38b0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fccf400cbc0 tx=0x7fccf400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:17.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.397+0000 7fccf27fc700 1 -- 172.21.15.158:0/1122332665 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4003c00 con 0x7fccfc15da40 2023-10-16T11:01:17.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.397+0000 7fccf27fc700 1 -- 172.21.15.158:0/1122332665 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf4003d60 con 0x7fccfc15da40 2023-10-16T11:01:17.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.397+0000 7fccf27fc700 1 -- 172.21.15.158:0/1122332665 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf400aae0 con 0x7fccfc15da40 2023-10-16T11:01:17.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.397+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccfc0c44d0 con 0x7fccfc15da40 2023-10-16T11:01:17.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.397+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc0c4a90 con 0x7fccfc15da40 2023-10-16T11:01:17.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.398+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc0c5190 con 0x7fccfc15da40 2023-10-16T11:01:17.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.399+0000 7fccf27fc700 1 -- 172.21.15.158:0/1122332665 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fccf400ac40 con 0x7fccfc15da40 2023-10-16T11:01:17.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.399+0000 7fccf27fc700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcce006b020 0x7fcce006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:17.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.399+0000 7fccf27fc700 1 -- 172.21.15.158:0/1122332665 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6381+0+0 (secure 0 0 0) 0x7fccf4089a30 con 0x7fccfc15da40 2023-10-16T11:01:17.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.399+0000 7fcd0185d700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcce006b020 0x7fcce006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:17.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.399+0000 7fcd0185d700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcce006b020 0x7fcce006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fccf80090b0 tx=0x7fccf801c040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:17.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.404+0000 7fccf27fc700 1 -- 172.21.15.158:0/1122332665 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccf4054730 con 0x7fccfc15da40 2023-10-16T11:01:17.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:17.639+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fccfc0681b0 con 0x7fccfc15da40 2023-10-16T11:01:18.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.252+0000 7fccf27fc700 1 -- 172.21.15.158:0/1122332665 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v90) v1 ==== 174+0+0 (secure 0 0 0) 0x7fccf4057d50 con 0x7fccfc15da40 2023-10-16T11:01:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.259+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcce006b020 msgr2=0x7fcce006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.259+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcce006b020 0x7fcce006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fccf80090b0 tx=0x7fccf801c040 comp rx=0 tx=0).stop 2023-10-16T11:01:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.259+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 msgr2=0x7fccfc0b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.259+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0b38b0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fccf400cbc0 tx=0x7fccf400ced0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.260+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 shutdown_connections 2023-10-16T11:01:18.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.260+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcce006b020 0x7fcce006d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.260+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccfc15da40 0x7fccfc0b38b0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.260+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccfc06afe0 0x7fccfc15a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.260+0000 7fcd03ac1700 1 --2- 172.21.15.158:0/1122332665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccfc0cc590 0x7fccfc1f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.260+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 >> 172.21.15.158:0/1122332665 conn(0x7fccfc099a10 msgr2=0x7fccfc0b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.262+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 shutdown_connections 2023-10-16T11:01:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.262+0000 7fcd03ac1700 1 -- 172.21.15.158:0/1122332665 wait complete. 2023-10-16T11:01:18.264 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 23 and data pool 24 2023-10-16T11:01:18.282 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:18.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.403+0000 7f7d31e8a700 1 Processor -- start 2023-10-16T11:01:18.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.403+0000 7f7d31e8a700 1 -- start start 2023-10-16T11:01:18.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d31e8a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d2c0cc5d0 0x7f7d2c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d31e8a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d31e8a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c06e770 con 0x7f7d2c06f340 2023-10-16T11:01:18.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d31e8a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c0cf270 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d31e8a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c15e4e0 con 0x7f7d2c15dfb0 2023-10-16T11:01:18.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d2b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d2c0cc5d0 0x7f7d2c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d2b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d2c0cc5d0 0x7f7d2c06e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34526/0 (socket says 172.21.15.158:34526) 2023-10-16T11:01:18.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d2b7fe700 1 -- 172.21.15.158:0/3603919267 learned_addr learned my addr 172.21.15.158:0/3603919267 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:18.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.404+0000 7f7d2affd700 1 --2- 172.21.15.158:0/3603919267 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2b7fe700 1 -- 172.21.15.158:0/3603919267 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 msgr2=0x7f7d2c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:18.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2b7fe700 1 --2- 172.21.15.158:0/3603919267 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2b7fe700 1 -- 172.21.15.158:0/3603919267 >> v1:172.21.15.172:6789/0 conn(0x7f7d2c06f340 legacy=0x7f7d2c071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:18.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2b7fe700 1 -- 172.21.15.158:0/3603919267 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c15e7c0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2b7fe700 1 --2- 172.21.15.158:0/3603919267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c06e230 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f7d140098f0 tx=0x7f7d14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5b6441eabb1ea5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:18.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2a7fc700 1 -- 172.21.15.158:0/3603919267 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14004230 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2a7fc700 1 -- 172.21.15.158:0/3603919267 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d14004390 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d2a7fc700 1 -- 172.21.15.158:0/3603919267 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d140036a0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d31e8a700 1 -- 172.21.15.158:0/3603919267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 msgr2=0x7f7d2c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/3603919267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c06e230 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f7d140098f0 tx=0x7f7d14009c00 comp rx=0 tx=0).stop 2023-10-16T11:01:18.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d31e8a700 1 -- 172.21.15.158:0/3603919267 shutdown_connections 2023-10-16T11:01:18.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/3603919267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c06e230 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/3603919267 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d31e8a700 1 -- 172.21.15.158:0/3603919267 >> 172.21.15.158:0/3603919267 conn(0x7f7d2c0d02b0 msgr2=0x7f7d2c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:18.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.405+0000 7f7d31e8a700 1 -- 172.21.15.158:0/3603919267 shutdown_connections 2023-10-16T11:01:18.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.406+0000 7f7d31e8a700 1 -- 172.21.15.158:0/3603919267 wait complete. 2023-10-16T11:01:18.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.407+0000 7f7d31e8a700 1 Processor -- start 2023-10-16T11:01:18.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.407+0000 7f7d31e8a700 1 -- start start 2023-10-16T11:01:18.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d31e8a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d2c06f340 0x7f7d2c0c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d31e8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d31e8a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c1ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d31e8a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0c9150 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d31e8a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0c9430 con 0x7f7d2c06f340 2023-10-16T11:01:18.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d31e8a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0c9710 con 0x7f7d2c15dfb0 2023-10-16T11:01:18.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d2affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d2affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c15c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34534/0 (socket says 172.21.15.158:34534) 2023-10-16T11:01:18.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d2affd700 1 -- 172.21.15.158:0/4027906800 learned_addr learned my addr 172.21.15.158:0/4027906800 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:18.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d2b7fe700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d2c06f340 0x7f7d2c0c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.408+0000 7f7d2bfff700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c1ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d2affd700 1 -- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 msgr2=0x7f7d2c1ef3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d2affd700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c1ef3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d2affd700 1 -- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d2c06f340 msgr2=0x7f7d2c0c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d2affd700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d2c06f340 0x7f7d2c0c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d2affd700 1 -- 172.21.15.158:0/4027906800 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d14009550 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d2affd700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c15c4a0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009f40 tx=0x7f7d1c00eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d23fff700 1 -- 172.21.15.158:0/4027906800 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c00c070 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d23fff700 1 -- 172.21.15.158:0/4027906800 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c011940 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c0c99f0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d23fff700 1 -- 172.21.15.158:0/4027906800 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c011cf0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.409+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c15c9e0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.410+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c0681b0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.411+0000 7f7d23fff700 1 -- 172.21.15.158:0/4027906800 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7d1c0193f0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.411+0000 7f7d23fff700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f7d0c06b020 0x7f7d0c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.411+0000 7f7d23fff700 1 -- 172.21.15.158:0/4027906800 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f7d1c08b010 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.411+0000 7f7d2b7fe700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f7d0c06b020 0x7f7d0c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.411+0000 7f7d2b7fe700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f7d0c06b020 0x7f7d0c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009130 tx=0x7f7d14008030 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:18.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.415+0000 7f7d23fff700 1 -- 172.21.15.158:0/4027906800 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d1c052fe0 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.608+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d2c0c9290 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.609+0000 7f7d23fff700 1 -- 172.21.15.158:0/4027906800 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+17431 (secure 0 0 0) 0x7f7d2c0c9290 con 0x7f7d2c0cc5d0 2023-10-16T11:01:18.610 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:18.611 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":184,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:01:17.640341+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:01:16.077848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T11:01:16.598638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:18.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f7d0c06b020 msgr2=0x7f7d0c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f7d0c06b020 0x7f7d0c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009130 tx=0x7f7d14008030 comp rx=0 tx=0).stop 2023-10-16T11:01:18.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 msgr2=0x7f7d2c15c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c15c4a0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009f40 tx=0x7f7d1c00eac0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 shutdown_connections 2023-10-16T11:01:18.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f7d0c06b020 0x7f7d0c06d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d2c15dfb0 0x7f7d2c1ef3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d2c06f340 0x7f7d2c0c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 --2- 172.21.15.158:0/4027906800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d2c0cc5d0 0x7f7d2c15c4a0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 >> 172.21.15.158:0/4027906800 conn(0x7f7d2c0d02b0 msgr2=0x7f7d2c159c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:18.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 shutdown_connections 2023-10-16T11:01:18.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.611+0000 7f7d31e8a700 1 -- 172.21.15.158:0/4027906800 wait complete. 2023-10-16T11:01:18.625 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:18.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.744+0000 7f093f207700 1 Processor -- start 2023-10-16T11:01:18.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.744+0000 7f093f207700 1 -- start start 2023-10-16T11:01:18.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093f207700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f093806ab40 0x7f093806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093f207700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 0x7f0938071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093f207700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0938071e50 con 0x7f093806e5d0 2023-10-16T11:01:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093f207700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0938161270 con 0x7f093806ab40 2023-10-16T11:01:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093f207700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0938161510 con 0x7f09380729b0 2023-10-16T11:01:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093cfa3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f093806ab40 0x7f093806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f0937fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f093806e5d0 0x7f09380722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34718/0 (socket says 172.21.15.158:34718) 2023-10-16T11:01:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f0937fff700 1 -- 172.21.15.158:0/764826262 learned_addr learned my addr 172.21.15.158:0/764826262 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093d7a4700 1 --2- 172.21.15.158:0/764826262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 0x7f0938071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093cfa3700 1 -- 172.21.15.158:0/764826262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 msgr2=0x7f0938071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:18.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093cfa3700 1 --2- 172.21.15.158:0/764826262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 0x7f0938071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093cfa3700 1 -- 172.21.15.158:0/764826262 >> v1:172.21.15.158:6789/0 conn(0x7f093806e5d0 legacy=0x7f09380722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093cfa3700 1 -- 172.21.15.158:0/764826262 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09381617f0 con 0x7f093806ab40 2023-10-16T11:01:18.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f09377fe700 1 -- 172.21.15.158:0/764826262 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267855312 0 0) 0x7f0938071e50 con 0x7f093806e5d0 2023-10-16T11:01:18.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.745+0000 7f093cfa3700 1 --2- 172.21.15.158:0/764826262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806ab40 0x7f093806d820 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f092c009840 tx=0x7f092c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319de5cd243d0a73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:18.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f09377fe700 1 -- 172.21.15.158:0/764826262 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011830 con 0x7f093806ab40 2023-10-16T11:01:18.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f09377fe700 1 -- 172.21.15.158:0/764826262 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c011990 con 0x7f093806ab40 2023-10-16T11:01:18.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f09377fe700 1 -- 172.21.15.158:0/764826262 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011d20 con 0x7f093806ab40 2023-10-16T11:01:18.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 -- 172.21.15.158:0/764826262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806ab40 msgr2=0x7f093806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 --2- 172.21.15.158:0/764826262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806ab40 0x7f093806d820 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f092c009840 tx=0x7f092c009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:18.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 -- 172.21.15.158:0/764826262 shutdown_connections 2023-10-16T11:01:18.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 --2- 172.21.15.158:0/764826262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806ab40 0x7f093806d820 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 --2- 172.21.15.158:0/764826262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 0x7f0938071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 -- 172.21.15.158:0/764826262 >> 172.21.15.158:0/764826262 conn(0x7f09380d02b0 msgr2=0x7f09380d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 -- 172.21.15.158:0/764826262 shutdown_connections 2023-10-16T11:01:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.746+0000 7f093f207700 1 -- 172.21.15.158:0/764826262 wait complete. 2023-10-16T11:01:18.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.747+0000 7f093f207700 1 Processor -- start 2023-10-16T11:01:18.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.747+0000 7f093f207700 1 -- start start 2023-10-16T11:01:18.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f093f207700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f093806ab40 0x7f09380b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f093f207700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806e5d0 0x7f09380b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f093f207700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 0x7f09381f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f093f207700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938148fa0 con 0x7f093806e5d0 2023-10-16T11:01:18.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f093f207700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09380b3710 con 0x7f093806ab40 2023-10-16T11:01:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f093f207700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09380b6040 con 0x7f09380729b0 2023-10-16T11:01:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f0937fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806e5d0 0x7f09380b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f0937fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806e5d0 0x7f09380b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34548/0 (socket says 172.21.15.158:34548) 2023-10-16T11:01:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.748+0000 7f0937fff700 1 -- 172.21.15.158:0/2789288591 learned_addr learned my addr 172.21.15.158:0/2789288591 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:18.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f0937fff700 1 -- 172.21.15.158:0/2789288591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 msgr2=0x7f09381f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f0937fff700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 0x7f09381f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f0937fff700 1 -- 172.21.15.158:0/2789288591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f093806ab40 msgr2=0x7f09380b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f093cfa3700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f093806ab40 0x7f09380b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f0937fff700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f093806ab40 0x7f09380b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f0937fff700 1 -- 172.21.15.158:0/2789288591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c0094a0 con 0x7f093806e5d0 2023-10-16T11:01:18.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f0937fff700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806e5d0 0x7f09380b31d0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f092400c540 tx=0x7f092400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:18.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f09357fa700 1 -- 172.21.15.158:0/2789288591 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924004c80 con 0x7f093806e5d0 2023-10-16T11:01:18.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f09357fa700 1 -- 172.21.15.158:0/2789288591 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924004de0 con 0x7f093806e5d0 2023-10-16T11:01:18.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093815f110 con 0x7f093806e5d0 2023-10-16T11:01:18.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f09357fa700 1 -- 172.21.15.158:0/2789288591 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924007310 con 0x7f093806e5d0 2023-10-16T11:01:18.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.749+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093815f3b0 con 0x7f093806e5d0 2023-10-16T11:01:18.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.750+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09380681b0 con 0x7f093806e5d0 2023-10-16T11:01:18.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.750+0000 7f09357fa700 1 -- 172.21.15.158:0/2789288591 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0924007470 con 0x7f093806e5d0 2023-10-16T11:01:18.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.751+0000 7f09357fa700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f091c06b020 0x7f091c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:18.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.751+0000 7f09357fa700 1 -- 172.21.15.158:0/2789288591 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f0924089770 con 0x7f093806e5d0 2023-10-16T11:01:18.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.751+0000 7f093cfa3700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f091c06b020 0x7f091c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:18.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.751+0000 7f093cfa3700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f091c06b020 0x7f091c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f092c000c00 tx=0x7f092c008030 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:18.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.755+0000 7f09357fa700 1 -- 172.21.15.158:0/2789288591 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0924054490 con 0x7f093806e5d0 2023-10-16T11:01:18.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.948+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09380af3c0 con 0x7f093806e5d0 2023-10-16T11:01:18.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.949+0000 7f09357fa700 1 -- 172.21.15.158:0/2789288591 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+17431 (secure 0 0 0) 0x7f0924057ab0 con 0x7f093806e5d0 2023-10-16T11:01:18.951 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:18.951 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":184,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:01:17.640341+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:01:16.077848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T11:01:16.598638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:18.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f091c06b020 msgr2=0x7f091c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f091c06b020 0x7f091c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f092c000c00 tx=0x7f092c008030 comp rx=0 tx=0).stop 2023-10-16T11:01:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806e5d0 msgr2=0x7f09380b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806e5d0 0x7f09380b31d0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f092400c540 tx=0x7f092400c900 comp rx=0 tx=0).stop 2023-10-16T11:01:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 shutdown_connections 2023-10-16T11:01:18.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f091c06b020 0x7f091c06d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09380729b0 0x7f09381f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f093806ab40 0x7f09380b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 --2- 172.21.15.158:0/2789288591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f093806e5d0 0x7f09380b31d0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 >> 172.21.15.158:0/2789288591 conn(0x7f09380d02b0 msgr2=0x7f093806d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:18.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 shutdown_connections 2023-10-16T11:01:18.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:18.951+0000 7f093f207700 1 -- 172.21.15.158:0/2789288591 wait complete. 2023-10-16T11:01:18.965 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T11:01:19.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.089+0000 7fe773554700 1 Processor -- start 2023-10-16T11:01:19.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.089+0000 7fe773554700 1 -- start start 2023-10-16T11:01:19.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.090+0000 7fe773554700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe76c0ce510 0x7fe76c15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe773554700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 0x7fe76c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe773554700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c0cf4c0 con 0x7fe76c15b590 2023-10-16T11:01:19.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe773554700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c0cc4c0 con 0x7fe76c0ce510 2023-10-16T11:01:19.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe773554700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0b49e0 con 0x7fe76c0cbaf0 2023-10-16T11:01:19.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe7712f0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe76c15b590 0x7fe76c0b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34730/0 (socket says 172.21.15.158:34730) 2023-10-16T11:01:19.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe7712f0700 1 -- 172.21.15.158:0/71905738 learned_addr learned my addr 172.21.15.158:0/71905738 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:19.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe771af1700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 0x7fe76c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe770aef700 1 --2- 172.21.15.158:0/71905738 >> v2:172.21.15.172:3300/0 conn(0x7fe76c0ce510 0x7fe76c15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185644343 0 0) 0x7fe76c0cf4c0 con 0x7fe76c15b590 2023-10-16T11:01:19.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7580035b0 con 0x7fe76c15b590 2023-10-16T11:01:19.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754321184 0 0) 0x7fe7580035b0 con 0x7fe76c15b590 2023-10-16T11:01:19.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 msgr2=0x7fe76c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:19.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 0x7fe76c0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:19.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 >> v2:172.21.15.172:3300/0 conn(0x7fe76c0ce510 msgr2=0x7fe76c15f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:19.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 --2- 172.21.15.158:0/71905738 >> v2:172.21.15.172:3300/0 conn(0x7fe76c0ce510 0x7fe76c15f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:19.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c15f710 con 0x7fe76c15b590 2023-10-16T11:01:19.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe75c001670 con 0x7fe76c15b590 2023-10-16T11:01:19.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 >> v1:172.21.15.158:6789/0 conn(0x7fe76c15b590 legacy=0x7fe76c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:19.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe758003a30 0x7fe758003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758004850 0x7fe758004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 0x7fe758005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0cc4c0 con 0x7fe758003a30 2023-10-16T11:01:19.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0b49e0 con 0x7fe758004850 2023-10-16T11:01:19.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75800ba60 con 0x7fe76c0cbaf0 2023-10-16T11:01:19.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe7712f0700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe758003a30 0x7fe758003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.091+0000 7fe770aef700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758004850 0x7fe758004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe771af1700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 0x7fe758005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe7712f0700 1 -- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 msgr2=0x7fe758005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:19.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe7712f0700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c0cbaf0 0x7fe758005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:19.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe7712f0700 1 -- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758004850 msgr2=0x7fe758004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:19.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe7712f0700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758004850 0x7fe758004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:19.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe7712f0700 1 -- 172.21.15.158:0/71905738 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580035b0 con 0x7fe758003a30 2023-10-16T11:01:19.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe7712f0700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe758003a30 0x7fe758003e50 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00c2a0 tx=0x7fe758005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a515f56ff5a56e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:19.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c007c10 con 0x7fe758003a30 2023-10-16T11:01:19.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c00cde0 con 0x7fe758003a30 2023-10-16T11:01:19.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.092+0000 7fe763fff700 1 -- 172.21.15.158:0/71905738 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c005ba0 con 0x7fe758003a30 2023-10-16T11:01:19.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.093+0000 7fe773554700 1 -- 172.21.15.158:0/71905738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe758003a30 msgr2=0x7fe758003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:19.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.093+0000 7fe773554700 1 --2- 172.21.15.158:0/71905738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe758003a30 0x7fe758003e50 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00c2a0 tx=0x7fe758005880 comp rx=0 tx=0).stop 2023-10-16T11:01:19.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.093+0000 7fe771af1700 1 -- 172.21.15.158:0/71905738 reap_dead start 2023-10-16T11:01:19.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.093+0000 7fe773554700 1 -- 172.21.15.158:0/71905738 shutdown_connections 2023-10-16T11:01:19.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.093+0000 7fe773554700 1 -- 172.21.15.158:0/71905738 >> 172.21.15.158:0/71905738 conn(0x7fe76c06a8d0 msgr2=0x7fe76c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:19.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.093+0000 7fe773554700 1 -- 172.21.15.158:0/71905738 shutdown_connections 2023-10-16T11:01:19.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.093+0000 7fe773554700 1 -- 172.21.15.158:0/71905738 wait complete. 2023-10-16T11:01:19.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.094+0000 7fe773554700 1 Processor -- start 2023-10-16T11:01:19.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.094+0000 7fe773554700 1 -- start start 2023-10-16T11:01:19.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe773554700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758003a30 0x7fe76c0d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe773554700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe76c15b970 0x7fe76c154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe773554700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c15bdf0 0x7fe76c155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe773554700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c15c980 con 0x7fe76c15b970 2023-10-16T11:01:19.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe773554700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c15c3c0 con 0x7fe758003a30 2023-10-16T11:01:19.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe773554700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c15c6a0 con 0x7fe76c15bdf0 2023-10-16T11:01:19.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe770aef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe76c15b970 0x7fe76c154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe770aef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe76c15b970 0x7fe76c154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34568/0 (socket says 172.21.15.158:34568) 2023-10-16T11:01:19.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe770aef700 1 -- 172.21.15.158:0/1660237709 learned_addr learned my addr 172.21.15.158:0/1660237709 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:19.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe7712f0700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758003a30 0x7fe76c0d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.095+0000 7fe771af1700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c15bdf0 0x7fe76c155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe770aef700 1 -- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c15bdf0 msgr2=0x7fe76c155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:19.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe770aef700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c15bdf0 0x7fe76c155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe770aef700 1 -- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758003a30 msgr2=0x7fe76c0d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe770aef700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758003a30 0x7fe76c0d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe770aef700 1 -- 172.21.15.158:0/1660237709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c154710 con 0x7fe76c15b970 2023-10-16T11:01:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe770aef700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe76c15b970 0x7fe76c154f10 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fe768011aa0 tx=0x7fe768011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe761ffb700 1 -- 172.21.15.158:0/1660237709 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768011ee0 con 0x7fe76c15b970 2023-10-16T11:01:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe761ffb700 1 -- 172.21.15.158:0/1660237709 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76800f940 con 0x7fe76c15b970 2023-10-16T11:01:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c1549f0 con 0x7fe76c15b970 2023-10-16T11:01:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe761ffb700 1 -- 172.21.15.158:0/1660237709 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76800fd10 con 0x7fe76c15b970 2023-10-16T11:01:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.096+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c0d0780 con 0x7fe76c15b970 2023-10-16T11:01:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.097+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c154570 con 0x7fe76c15b970 2023-10-16T11:01:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.098+0000 7fe761ffb700 1 -- 172.21.15.158:0/1660237709 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe768014440 con 0x7fe76c15b970 2023-10-16T11:01:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.098+0000 7fe761ffb700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe75006b020 0x7fe75006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.098+0000 7fe761ffb700 1 -- 172.21.15.158:0/1660237709 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fe768089fd0 con 0x7fe76c15b970 2023-10-16T11:01:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.098+0000 7fe7712f0700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe75006b020 0x7fe75006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.098+0000 7fe7712f0700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe75006b020 0x7fe75006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe75c000c00 tx=0x7fe75c007000 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:19.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.102+0000 7fe761ffb700 1 -- 172.21.15.158:0/1660237709 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe76c154570 con 0x7fe76c15b970 2023-10-16T11:01:19.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:19.344+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe76c154570 con 0x7fe76c15b970 2023-10-16T11:01:20.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.263+0000 7fe761ffb700 1 -- 172.21.15.158:0/1660237709 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v93) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe76c154570 con 0x7fe76c15b970 2023-10-16T11:01:20.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe75006b020 msgr2=0x7fe75006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe75006b020 0x7fe75006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe75c000c00 tx=0x7fe75c007000 comp rx=0 tx=0).stop 2023-10-16T11:01:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe76c15b970 msgr2=0x7fe76c154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe76c15b970 0x7fe76c154f10 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fe768011aa0 tx=0x7fe768011db0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 shutdown_connections 2023-10-16T11:01:20.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fe75006b020 0x7fe75006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe76c15bdf0 0x7fe76c155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe758003a30 0x7fe76c0d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 --2- 172.21.15.158:0/1660237709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe76c15b970 0x7fe76c154f10 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 >> 172.21.15.158:0/1660237709 conn(0x7fe76c06a8d0 msgr2=0x7fe76c159800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:20.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 shutdown_connections 2023-10-16T11:01:20.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.266+0000 7fe773554700 1 -- 172.21.15.158:0/1660237709 wait complete. 2023-10-16T11:01:20.282 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:01:20.283 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T11:01:20.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.412+0000 7f8f991d5700 1 Processor -- start 2023-10-16T11:01:20.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.412+0000 7f8f991d5700 1 -- start start 2023-10-16T11:01:20.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.412+0000 7f8f991d5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f940cc590 0x7f8f940caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:20.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f991d5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f940cbd30 con 0x7f8f9406afe0 2023-10-16T11:01:20.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f991d5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f940cc1c0 con 0x7f8f9415da40 2023-10-16T11:01:20.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f991d5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9415de20 con 0x7f8f940cc590 2023-10-16T11:01:20.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f9359e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f940cc590 0x7f8f940caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:20.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f9259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8f9406afe0 0x7f8f940afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34750/0 (socket says 172.21.15.158:34750) 2023-10-16T11:01:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f9259c700 1 -- 172.21.15.158:0/1646716459 learned_addr learned my addr 172.21.15.158:0/1646716459 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f9359e700 1 -- 172.21.15.158:0/1646716459 >> v1:172.21.15.172:6789/0 conn(0x7f8f9415da40 legacy=0x7f8f9406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f9359e700 1 -- 172.21.15.158:0/1646716459 >> v1:172.21.15.158:6789/0 conn(0x7f8f9406afe0 legacy=0x7f8f940afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f9359e700 1 -- 172.21.15.158:0/1646716459 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f940ce260 con 0x7f8f940cc590 2023-10-16T11:01:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f9359e700 1 --2- 172.21.15.158:0/1646716459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f940cc590 0x7f8f940caa20 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009d90 tx=0x7f8f8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192a5ae772c318e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f91d9b700 1 -- 172.21.15.158:0/1646716459 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c00e070 con 0x7f8f940cc590 2023-10-16T11:01:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f91d9b700 1 -- 172.21.15.158:0/1646716459 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c00bb90 con 0x7f8f940cc590 2023-10-16T11:01:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.413+0000 7f8f91d9b700 1 -- 172.21.15.158:0/1646716459 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c003a20 con 0x7f8f940cc590 2023-10-16T11:01:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.414+0000 7f8f991d5700 1 -- 172.21.15.158:0/1646716459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f940cc590 msgr2=0x7f8f940caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.414+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1646716459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f940cc590 0x7f8f940caa20 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009d90 tx=0x7f8f8c00b330 comp rx=0 tx=0).stop 2023-10-16T11:01:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.414+0000 7f8f991d5700 1 -- 172.21.15.158:0/1646716459 shutdown_connections 2023-10-16T11:01:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.414+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1646716459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f940cc590 0x7f8f940caa20 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.414+0000 7f8f991d5700 1 -- 172.21.15.158:0/1646716459 >> 172.21.15.158:0/1646716459 conn(0x7f8f94099a10 msgr2=0x7f8f940d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.414+0000 7f8f991d5700 1 -- 172.21.15.158:0/1646716459 shutdown_connections 2023-10-16T11:01:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.414+0000 7f8f991d5700 1 -- 172.21.15.158:0/1646716459 wait complete. 2023-10-16T11:01:20.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.415+0000 7f8f991d5700 1 Processor -- start 2023-10-16T11:01:20.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.415+0000 7f8f991d5700 1 -- start start 2023-10-16T11:01:20.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f991d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f9406afe0 0x7f8f941584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:20.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f991d5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f940cc590 0x7f8f941e7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:20.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f991d5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9415da40 0x7f8f940c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:20.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f991d5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f941e6140 con 0x7f8f9406afe0 2023-10-16T11:01:20.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f991d5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9415c550 con 0x7f8f940cc590 2023-10-16T11:01:20.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f991d5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f941e6390 con 0x7f8f9415da40 2023-10-16T11:01:20.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f92d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f9406afe0 0x7f8f941584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:20.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f92d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f9406afe0 0x7f8f941584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34578/0 (socket says 172.21.15.158:34578) 2023-10-16T11:01:20.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f92d9d700 1 -- 172.21.15.158:0/1000676270 learned_addr learned my addr 172.21.15.158:0/1000676270 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:20.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f9259c700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f940cc590 0x7f8f941e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:20.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.416+0000 7f8f9359e700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9415da40 0x7f8f940c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:20.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f92d9d700 1 -- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9415da40 msgr2=0x7f8f940c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:20.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f92d9d700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9415da40 0x7f8f940c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f92d9d700 1 -- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f940cc590 msgr2=0x7f8f941e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:20.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f92d9d700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f940cc590 0x7f8f941e7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:20.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f92d9d700 1 -- 172.21.15.158:0/1000676270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c009a70 con 0x7f8f9406afe0 2023-10-16T11:01:20.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f9259c700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f940cc590 0x7f8f941e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:20.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f92d9d700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f9406afe0 0x7f8f941584a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400b4d0 tx=0x7f8f8400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:20.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f7b7fe700 1 -- 172.21.15.158:0/1000676270 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84011830 con 0x7f8f9406afe0 2023-10-16T11:01:20.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f7b7fe700 1 -- 172.21.15.158:0/1000676270 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f84011990 con 0x7f8f9406afe0 2023-10-16T11:01:20.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f941e6670 con 0x7f8f9406afe0 2023-10-16T11:01:20.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f7b7fe700 1 -- 172.21.15.158:0/1000676270 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84011d40 con 0x7f8f9406afe0 2023-10-16T11:01:20.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.417+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f941e6c30 con 0x7f8f9406afe0 2023-10-16T11:01:20.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.418+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f940681b0 con 0x7f8f9406afe0 2023-10-16T11:01:20.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.418+0000 7f8f7b7fe700 1 -- 172.21.15.158:0/1000676270 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8f8401a400 con 0x7f8f9406afe0 2023-10-16T11:01:20.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.419+0000 7f8f7b7fe700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8f8006b020 0x7f8f8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:20.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.419+0000 7f8f7b7fe700 1 -- 172.21.15.158:0/1000676270 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f8f840897b0 con 0x7f8f9406afe0 2023-10-16T11:01:20.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.419+0000 7f8f9259c700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8f8006b020 0x7f8f8006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:20.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.419+0000 7f8f9259c700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8f8006b020 0x7f8f8006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f9406ed20 tx=0x7f8f88009040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:20.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.423+0000 7f8f7b7fe700 1 -- 172.21.15.158:0/1000676270 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f84054440 con 0x7f8f9406afe0 2023-10-16T11:01:20.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:20.660+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8f9406ea70 con 0x7f8f9406afe0 2023-10-16T11:01:21.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.379+0000 7f8f7b7fe700 1 -- 172.21.15.158:0/1000676270 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8f84057a60 con 0x7f8f9406afe0 2023-10-16T11:01:21.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.382+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8f8006b020 msgr2=0x7f8f8006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.382+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8f8006b020 0x7f8f8006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f9406ed20 tx=0x7f8f88009040 comp rx=0 tx=0).stop 2023-10-16T11:01:21.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.382+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f9406afe0 msgr2=0x7f8f941584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.382+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f9406afe0 0x7f8f941584a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400b4d0 tx=0x7f8f8400b890 comp rx=0 tx=0).stop 2023-10-16T11:01:21.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 shutdown_connections 2023-10-16T11:01:21.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8f8006b020 0x7f8f8006d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f9415da40 0x7f8f940c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f9406afe0 0x7f8f941584a0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 --2- 172.21.15.158:0/1000676270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f940cc590 0x7f8f941e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 >> 172.21.15.158:0/1000676270 conn(0x7f8f94099a10 msgr2=0x7f8f9415cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:21.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 shutdown_connections 2023-10-16T11:01:21.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.383+0000 7f8f991d5700 1 -- 172.21.15.158:0/1000676270 wait complete. 2023-10-16T11:01:21.399 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T11:01:21.399 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:01:21.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.525+0000 7f0cf1eaa700 1 Processor -- start 2023-10-16T11:01:21.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.525+0000 7f0cf1eaa700 1 -- start start 2023-10-16T11:01:21.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cf1eaa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0cec15ed60 0x7f0cec0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cf1eaa700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0cec06a8d0 0x7f0cec06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cf1eaa700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cec0b3f30 con 0x7f0cec0b3480 2023-10-16T11:01:21.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cf1eaa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cec0b3910 con 0x7f0cec15ed60 2023-10-16T11:01:21.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cf1eaa700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cec06b280 con 0x7f0cec06a8d0 2023-10-16T11:01:21.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0cec15ed60 0x7f0cec0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0cec15ed60 0x7f0cec0cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34586/0 (socket says 172.21.15.158:34586) 2023-10-16T11:01:21.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 -- 172.21.15.158:0/1958553476 learned_addr learned my addr 172.21.15.158:0/1958553476 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:21.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 -- 172.21.15.158:0/1958553476 >> v2:172.21.15.172:3300/0 conn(0x7f0cec06a8d0 msgr2=0x7f0cec06ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:21.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cebfff700 1 --2- 172.21.15.158:0/1958553476 >> v2:172.21.15.172:3300/0 conn(0x7f0cec06a8d0 0x7f0cec06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 --2- 172.21.15.158:0/1958553476 >> v2:172.21.15.172:3300/0 conn(0x7f0cec06a8d0 0x7f0cec06ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 -- 172.21.15.158:0/1958553476 >> v1:172.21.15.172:6789/0 conn(0x7f0cec0b3480 legacy=0x7f0cec0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:21.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 -- 172.21.15.158:0/1958553476 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec0b4ad0 con 0x7f0cec15ed60 2023-10-16T11:01:21.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0ceb7fe700 1 --2- 172.21.15.158:0/1958553476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0cd890 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f0cec06b5d0 tx=0x7f0ce0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3098de36c5e7160c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:21.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cea7fc700 1 -- 172.21.15.158:0/1958553476 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0005e30 con 0x7f0cec15ed60 2023-10-16T11:01:21.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cea7fc700 1 -- 172.21.15.158:0/1958553476 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0006030 con 0x7f0cec15ed60 2023-10-16T11:01:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.526+0000 7f0cea7fc700 1 -- 172.21.15.158:0/1958553476 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00063c0 con 0x7f0cec15ed60 2023-10-16T11:01:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1958553476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 msgr2=0x7f0cec0cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1958553476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0cd890 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f0cec06b5d0 tx=0x7f0ce0009bd0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1958553476 shutdown_connections 2023-10-16T11:01:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1958553476 >> v2:172.21.15.172:3300/0 conn(0x7f0cec06a8d0 0x7f0cec06ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1958553476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0cd890 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1958553476 >> 172.21.15.158:0/1958553476 conn(0x7f0cec0ccda0 msgr2=0x7f0cec0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:21.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1958553476 shutdown_connections 2023-10-16T11:01:21.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.527+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1958553476 wait complete. 2023-10-16T11:01:21.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.528+0000 7f0cf1eaa700 1 Processor -- start 2023-10-16T11:01:21.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.528+0000 7f0cf1eaa700 1 -- start start 2023-10-16T11:01:21.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0cf1eaa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cec06a8d0 0x7f0cec0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0cf1eaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0cf1eaa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cec0b1ca0 0x7f0cec15d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0cf1eaa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0b0a50 con 0x7f0cec15ed60 2023-10-16T11:01:21.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0cf1eaa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0afe40 con 0x7f0cec0b1ca0 2023-10-16T11:01:21.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0cf1eaa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0b0120 con 0x7f0cec06a8d0 2023-10-16T11:01:21.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34594/0 (socket says 172.21.15.158:34594) 2023-10-16T11:01:21.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 -- 172.21.15.158:0/1739086020 learned_addr learned my addr 172.21.15.158:0/1739086020 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:21.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceb7fe700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cec06a8d0 0x7f0cec0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0cebfff700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cec0b1ca0 0x7f0cec15d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 -- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cec06a8d0 msgr2=0x7f0cec0adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cec06a8d0 0x7f0cec0adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 -- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cec0b1ca0 msgr2=0x7f0cec15d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cec0b1ca0 0x7f0cec15d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.529+0000 7f0ceaffd700 1 -- 172.21.15.158:0/1739086020 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce0009500 con 0x7f0cec15ed60 2023-10-16T11:01:21.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.530+0000 7f0ceaffd700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0b1760 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00c4a0 tx=0x7f0cdc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:21.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.530+0000 7f0cd3fff700 1 -- 172.21.15.158:0/1739086020 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0054e0 con 0x7f0cec15ed60 2023-10-16T11:01:21.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.530+0000 7f0cd3fff700 1 -- 172.21.15.158:0/1739086020 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc003680 con 0x7f0cec15ed60 2023-10-16T11:01:21.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.530+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec0b0400 con 0x7f0cec15ed60 2023-10-16T11:01:21.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.530+0000 7f0cd3fff700 1 -- 172.21.15.158:0/1739086020 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc003a30 con 0x7f0cec15ed60 2023-10-16T11:01:21.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.530+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec0b2100 con 0x7f0cec15ed60 2023-10-16T11:01:21.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.531+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec0b0c00 con 0x7f0cec15ed60 2023-10-16T11:01:21.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.531+0000 7f0cd3fff700 1 -- 172.21.15.158:0/1739086020 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0cdc003b90 con 0x7f0cec15ed60 2023-10-16T11:01:21.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.532+0000 7f0cd3fff700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0cd806b020 0x7f0cd806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.532+0000 7f0cd3fff700 1 -- 172.21.15.158:0/1739086020 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f0cdc088f00 con 0x7f0cec15ed60 2023-10-16T11:01:21.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.532+0000 7f0ceb7fe700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0cd806b020 0x7f0cd806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.532+0000 7f0ceb7fe700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0cd806b020 0x7f0cd806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0000c00 tx=0x7f0ce0008070 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:21.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.536+0000 7f0cd3fff700 1 -- 172.21.15.158:0/1739086020 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0cec0b0c00 con 0x7f0cec15ed60 2023-10-16T11:01:21.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.772+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cec0681b0 con 0x7f0cec15ed60 2023-10-16T11:01:21.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.773+0000 7f0cd3fff700 1 -- 172.21.15.158:0/1739086020 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4930 (secure 0 0 0) 0x7f0cdc017020 con 0x7f0cec15ed60 2023-10-16T11:01:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:21.775 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3580267390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3580267390},{"type":"v1","addr":"172.21.15.172:6834","nonce":3580267390}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":9328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3433152480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3433152480},{"type":"v1","addr":"172.21.15.172:6835","nonce":3433152480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1170239465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1170239465},{"type":"v1","addr":"172.21.15.158:6836","nonce":1170239465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:01:17.640665+0000","modified":"2023-10-16T11:01:20.662098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10449":{"gid":10449,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/773046671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":773046671},{"type":"v1","addr":"172.21.15.158:6837","nonce":773046671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T11:01:21.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0cd806b020 msgr2=0x7f0cd806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0cd806b020 0x7f0cd806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0000c00 tx=0x7f0ce0008070 comp rx=0 tx=0).stop 2023-10-16T11:01:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 msgr2=0x7f0cec0b1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0b1760 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00c4a0 tx=0x7f0cdc00c860 comp rx=0 tx=0).stop 2023-10-16T11:01:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 shutdown_connections 2023-10-16T11:01:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0cd806b020 0x7f0cd806d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cec0b1ca0 0x7f0cec15d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cec06a8d0 0x7f0cec0adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 --2- 172.21.15.158:0/1739086020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cec15ed60 0x7f0cec0b1760 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 >> 172.21.15.158:0/1739086020 conn(0x7f0cec0ccda0 msgr2=0x7f0cec15a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 shutdown_connections 2023-10-16T11:01:21.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.775+0000 7f0cf1eaa700 1 -- 172.21.15.158:0/1739086020 wait complete. 2023-10-16T11:01:21.780 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 94 2023-10-16T11:01:21.789 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:01:21.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.908+0000 7f0bdbc33700 1 Processor -- start 2023-10-16T11:01:21.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.908+0000 7f0bdbc33700 1 -- start start 2023-10-16T11:01:21.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bdbc33700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0bd406fa50 0x7f0bd406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bdbc33700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd406e770 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bdbc33700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd40cbe00 con 0x7f0bd415dfb0 2023-10-16T11:01:21.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bdbc33700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd40cf270 con 0x7f0bd406fa50 2023-10-16T11:01:21.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bd99cf700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0bd40cc5d0 0x7f0bd40cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34776/0 (socket says 172.21.15.158:34776) 2023-10-16T11:01:21.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bd99cf700 1 -- 172.21.15.158:0/19166718 learned_addr learned my addr 172.21.15.158:0/19166718 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:21.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bda1d0700 1 --2- 172.21.15.158:0/19166718 >> v2:172.21.15.172:3300/0 conn(0x7f0bd406fa50 0x7f0bd406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bd89cd700 1 -- 172.21.15.158:0/19166718 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799329626 0 0) 0x7f0bd0001110 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bd89cd700 1 -- 172.21.15.158:0/19166718 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc00035b0 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bda1d0700 1 -- 172.21.15.158:0/19166718 >> v1:172.21.15.172:6789/0 conn(0x7f0bd415dfb0 legacy=0x7f0bd406f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bd89cd700 1 -- 172.21.15.158:0/19166718 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 195808045 0 0) 0x7f0bc00035b0 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bda1d0700 1 -- 172.21.15.158:0/19166718 >> v1:172.21.15.158:6789/0 conn(0x7f0bd40cc5d0 legacy=0x7f0bd40cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.909+0000 7f0bda1d0700 1 -- 172.21.15.158:0/19166718 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd415e4e0 con 0x7f0bd406fa50 2023-10-16T11:01:21.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bda1d0700 1 --2- 172.21.15.158:0/19166718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd406e230 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009d90 tx=0x7f0bcc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=942bc03738045a13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:21.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bd89cd700 1 -- 172.21.15.158:0/19166718 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc00e070 con 0x7f0bd406fa50 2023-10-16T11:01:21.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bd89cd700 1 -- 172.21.15.158:0/19166718 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc003a20 con 0x7f0bd406fa50 2023-10-16T11:01:21.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bd89cd700 1 -- 172.21.15.158:0/19166718 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0001800 con 0x7f0bd406fa50 2023-10-16T11:01:21.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bdbc33700 1 -- 172.21.15.158:0/19166718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 msgr2=0x7f0bd406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/19166718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd406e230 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009d90 tx=0x7f0bcc00b330 comp rx=0 tx=0).stop 2023-10-16T11:01:21.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bdbc33700 1 -- 172.21.15.158:0/19166718 shutdown_connections 2023-10-16T11:01:21.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/19166718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd406e230 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bdbc33700 1 -- 172.21.15.158:0/19166718 >> 172.21.15.158:0/19166718 conn(0x7f0bd40d02b0 msgr2=0x7f0bd40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:21.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bdbc33700 1 -- 172.21.15.158:0/19166718 shutdown_connections 2023-10-16T11:01:21.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.910+0000 7f0bdbc33700 1 -- 172.21.15.158:0/19166718 wait complete. 2023-10-16T11:01:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.911+0000 7f0bdbc33700 1 Processor -- start 2023-10-16T11:01:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bdbc33700 1 -- start start 2023-10-16T11:01:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bdbc33700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd4143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bdbc33700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bd40cc5d0 0x7f0bd40c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bdbc33700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bd415dfb0 0x7f0bd415cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bdbc33700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd41573e0 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bdbc33700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4156760 con 0x7f0bd406fa50 2023-10-16T11:01:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bdbc33700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4156a40 con 0x7f0bd415dfb0 2023-10-16T11:01:21.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bd91ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bd40cc5d0 0x7f0bd40c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bd91ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bd40cc5d0 0x7f0bd40c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34608/0 (socket says 172.21.15.158:34608) 2023-10-16T11:01:21.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.912+0000 7f0bd91ce700 1 -- 172.21.15.158:0/3331470493 learned_addr learned my addr 172.21.15.158:0/3331470493 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:21.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bda1d0700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bd415dfb0 0x7f0bd415cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd99cf700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd4143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd91ce700 1 -- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bd415dfb0 msgr2=0x7f0bd415cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd91ce700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bd415dfb0 0x7f0bd415cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd91ce700 1 -- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 msgr2=0x7f0bd4143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:21.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd91ce700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd4143fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:21.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd91ce700 1 -- 172.21.15.158:0/3331470493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc009a70 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bda1d0700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bd415dfb0 0x7f0bd415cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:21.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd99cf700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd4143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:21.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bd91ce700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bd40cc5d0 0x7f0bd40c9d50 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c5d0 tx=0x7f0bc400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:21.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bca7fc700 1 -- 172.21.15.158:0/3331470493 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc40054e0 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bca7fc700 1 -- 172.21.15.158:0/3331470493 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4003680 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd4156d20 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bca7fc700 1 -- 172.21.15.158:0/3331470493 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4003a30 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.913+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd4144500 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.914+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd4157520 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.914+0000 7f0bca7fc700 1 -- 172.21.15.158:0/3331470493 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0bc4003b90 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.915+0000 7f0bca7fc700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0bb806b020 0x7f0bb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:21.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.915+0000 7f0bd99cf700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0bb806b020 0x7f0bb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:21.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.915+0000 7f0bca7fc700 1 -- 172.21.15.158:0/3331470493 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f0bc4089060 con 0x7f0bd40cc5d0 2023-10-16T11:01:21.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.915+0000 7f0bd99cf700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0bb806b020 0x7f0bb806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0001eb0 tx=0x7f0bd0009040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:21.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:21.919+0000 7f0bca7fc700 1 -- 172.21.15.158:0/3331470493 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bd4157520 con 0x7f0bd40cc5d0 2023-10-16T11:01:22.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.115+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bd4157520 con 0x7f0bd40cc5d0 2023-10-16T11:01:22.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.116+0000 7f0bca7fc700 1 -- 172.21.15.158:0/3331470493 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+17418 (secure 0 0 0) 0x7f0bd4157520 con 0x7f0bd40cc5d0 2023-10-16T11:01:22.118 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:22.118 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":185,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:01:19.237420+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:01:16.077848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T11:01:16.598638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:01:22.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.118+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0bb806b020 msgr2=0x7f0bb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.118+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0bb806b020 0x7f0bb806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0001eb0 tx=0x7f0bd0009040 comp rx=0 tx=0).stop 2023-10-16T11:01:22.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.118+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bd40cc5d0 msgr2=0x7f0bd40c9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bd40cc5d0 0x7f0bd40c9d50 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c5d0 tx=0x7f0bc400c990 comp rx=0 tx=0).stop 2023-10-16T11:01:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 shutdown_connections 2023-10-16T11:01:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f0bb806b020 0x7f0bb806d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bd415dfb0 0x7f0bd415cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bd406fa50 0x7f0bd4143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 --2- 172.21.15.158:0/3331470493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bd40cc5d0 0x7f0bd40c9d50 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 >> 172.21.15.158:0/3331470493 conn(0x7f0bd40d02b0 msgr2=0x7f0bd40afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:22.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 shutdown_connections 2023-10-16T11:01:22.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.119+0000 7f0bdbc33700 1 -- 172.21.15.158:0/3331470493 wait complete. 2023-10-16T11:01:22.132 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:01:22.133 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:01:22.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.252+0000 7f5789ce5700 1 Processor -- start 2023-10-16T11:01:22.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.252+0000 7f5789ce5700 1 -- start start 2023-10-16T11:01:22.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.253+0000 7f5789ce5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 0x7f578406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.253+0000 7f5789ce5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578406e770 con 0x7f578406f340 2023-10-16T11:01:22.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.253+0000 7f5789ce5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57840cbe00 con 0x7f57840cc5d0 2023-10-16T11:01:22.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.253+0000 7f5789ce5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57840cf270 con 0x7f578415dfb0 2023-10-16T11:01:22.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.253+0000 7f5783fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f578406f340 0x7f5784071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34794/0 (socket says 172.21.15.158:34794) 2023-10-16T11:01:22.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.253+0000 7f5783fff700 1 -- 172.21.15.158:0/1102765784 learned_addr learned my addr 172.21.15.158:0/1102765784 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:22.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5782ffd700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 0x7f578406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397675816 0 0) 0x7f577c001490 con 0x7f578406f340 2023-10-16T11:01:22.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57700035b0 con 0x7f578406f340 2023-10-16T11:01:22.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1253542345 0 0) 0x7f57700035b0 con 0x7f578406f340 2023-10-16T11:01:22.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 msgr2=0x7f578406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:22.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 0x7f578406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 >> v1:172.21.15.172:6789/0 conn(0x7f57840cc5d0 legacy=0x7f57840cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578415e4e0 con 0x7f578406f340 2023-10-16T11:01:22.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f577c001b80 con 0x7f578406f340 2023-10-16T11:01:22.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 >> v1:172.21.15.158:6789/0 conn(0x7f578406f340 legacy=0x7f5784071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 0x7f5770003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770003fb0 0x7f5770004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5770004d90 0x7f5770005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840cf270 con 0x7f5770004d90 2023-10-16T11:01:22.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577000baf0 con 0x7f5770003fb0 2023-10-16T11:01:22.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577000bcb0 con 0x7f578415dfb0 2023-10-16T11:01:22.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5783fff700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5770004d90 0x7f5770005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5782ffd700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770003fb0 0x7f5770004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f57837fe700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 0x7f5770003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5783fff700 1 -- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 msgr2=0x7f5770003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:22.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5783fff700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415dfb0 0x7f5770003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5783fff700 1 -- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770003fb0 msgr2=0x7f5770004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:22.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5783fff700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770003fb0 0x7f5770004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.254+0000 7f5783fff700 1 -- 172.21.15.158:0/1102765784 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700035b0 con 0x7f5770004d90 2023-10-16T11:01:22.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5783fff700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5770004d90 0x7f5770005210 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f577c00a830 tx=0x7f5770005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58adeac8a1a715d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:22.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c00a490 con 0x7f5770004d90 2023-10-16T11:01:22.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c012620 con 0x7f5770004d90 2023-10-16T11:01:22.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f57827fc700 1 -- 172.21.15.158:0/1102765784 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c0129d0 con 0x7f5770004d90 2023-10-16T11:01:22.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5789ce5700 1 -- 172.21.15.158:0/1102765784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5770004d90 msgr2=0x7f5770005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5789ce5700 1 --2- 172.21.15.158:0/1102765784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5770004d90 0x7f5770005210 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f577c00a830 tx=0x7f5770005880 comp rx=0 tx=0).stop 2023-10-16T11:01:22.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5783fff700 1 -- 172.21.15.158:0/1102765784 reap_dead start 2023-10-16T11:01:22.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5789ce5700 1 -- 172.21.15.158:0/1102765784 shutdown_connections 2023-10-16T11:01:22.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5789ce5700 1 -- 172.21.15.158:0/1102765784 >> 172.21.15.158:0/1102765784 conn(0x7f57840d02b0 msgr2=0x7f57840d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:22.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5789ce5700 1 -- 172.21.15.158:0/1102765784 shutdown_connections 2023-10-16T11:01:22.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.255+0000 7f5789ce5700 1 -- 172.21.15.158:0/1102765784 wait complete. 2023-10-16T11:01:22.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.257+0000 7f5789ce5700 1 Processor -- start 2023-10-16T11:01:22.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.257+0000 7f5789ce5700 1 -- start start 2023-10-16T11:01:22.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5789ce5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770004d90 0x7f57840b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5789ce5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415c570 0x7f57840ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5789ce5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5784149540 0x7f578406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5789ce5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840c9180 con 0x7f5784149540 2023-10-16T11:01:22.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5789ce5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840acdd0 con 0x7f5770004d90 2023-10-16T11:01:22.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5789ce5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578415c9f0 con 0x7f578415c570 2023-10-16T11:01:22.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5784149540 0x7f578406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5784149540 0x7f578406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34630/0 (socket says 172.21.15.158:34630) 2023-10-16T11:01:22.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 -- 172.21.15.158:0/123998931 learned_addr learned my addr 172.21.15.158:0/123998931 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:22.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5782ffd700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415c570 0x7f57840ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f57837fe700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770004d90 0x7f57840b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 -- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415c570 msgr2=0x7f57840ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415c570 0x7f57840ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 -- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770004d90 msgr2=0x7f57840b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770004d90 0x7f57840b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.258+0000 7f5783fff700 1 -- 172.21.15.158:0/123998931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57840c8570 con 0x7f5784149540 2023-10-16T11:01:22.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.259+0000 7f5783fff700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5784149540 0x7f578406a8d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f577c012000 tx=0x7f577c001f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:22.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.259+0000 7f577bfff700 1 -- 172.21.15.158:0/123998931 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c00a490 con 0x7f5784149540 2023-10-16T11:01:22.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.259+0000 7f577bfff700 1 -- 172.21.15.158:0/123998931 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c001490 con 0x7f5784149540 2023-10-16T11:01:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.259+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57840c8850 con 0x7f5784149540 2023-10-16T11:01:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.259+0000 7f577bfff700 1 -- 172.21.15.158:0/123998931 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c001b80 con 0x7f5784149540 2023-10-16T11:01:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.259+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57840c8e10 con 0x7f5784149540 2023-10-16T11:01:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.260+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57840681b0 con 0x7f5784149540 2023-10-16T11:01:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.260+0000 7f577bfff700 1 -- 172.21.15.158:0/123998931 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f577c02b670 con 0x7f5784149540 2023-10-16T11:01:22.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.261+0000 7f577bfff700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f576406b020 0x7f576406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.261+0000 7f577bfff700 1 -- 172.21.15.158:0/123998931 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f577c08b9b0 con 0x7f5784149540 2023-10-16T11:01:22.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.261+0000 7f57837fe700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f576406b020 0x7f576406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.261+0000 7f57837fe700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f576406b020 0x7f576406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f576c000c00 tx=0x7f576c009040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:22.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.265+0000 7f577bfff700 1 -- 172.21.15.158:0/123998931 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f577c0566d0 con 0x7f5784149540 2023-10-16T11:01:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.455+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 --> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f578415eaa0 con 0x7f576406b020 2023-10-16T11:01:22.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.464+0000 7f577bfff700 1 -- 172.21.15.158:0/123998931 <== mgr.10212 v2:172.21.15.158:6832/1491011698 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290511 (secure 0 0 0) 0x7f576c001620 con 0x7f576406b020 2023-10-16T11:01:22.468 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:22.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f576406b020 msgr2=0x7f576406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f576406b020 0x7f576406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f576c000c00 tx=0x7f576c009040 comp rx=0 tx=0).stop 2023-10-16T11:01:22.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5784149540 msgr2=0x7f578406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5784149540 0x7f578406a8d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f577c012000 tx=0x7f577c001f10 comp rx=0 tx=0).stop 2023-10-16T11:01:22.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 shutdown_connections 2023-10-16T11:01:22.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f576406b020 0x7f576406d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5784149540 0x7f578406a8d0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5770004d90 0x7f57840b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 --2- 172.21.15.158:0/123998931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f578415c570 0x7f57840ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 >> 172.21.15.158:0/123998931 conn(0x7f57840d02b0 msgr2=0x7f57840d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:22.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 shutdown_connections 2023-10-16T11:01:22.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.467+0000 7f5789ce5700 1 -- 172.21.15.158:0/123998931 wait complete. 2023-10-16T11:01:22.473 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:01:22.481 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T11:01:22.194948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":404772,"kb_used_data":19288,"kb_used_omap":12,"kb_used_meta":385331,"kb_avail":754569948,"statfs":{"total":773094113280,"available":772679626752,"internally_reserved":0,"allocated":19750912,"data_stored":10995436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":394579511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.963985"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244373+0000","last_change":"2023-10-16T11:01:18.255169+0000","last_active":"2023-10-16T11:01:19.244373+0000","last_peered":"2023-10-16T11:01:19.244373+0000","last_clean":"2023-10-16T11:01:19.244373+0000","last_became_active":"2023-10-16T11:01:18.254846+0000","last_became_peered":"2023-10-16T11:01:18.254846+0000","last_unstale":"2023-10-16T11:01:19.244373+0000","last_undegraded":"2023-10-16T11:01:19.244373+0000","last_fullsized":"2023-10-16T11:01:19.244373+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:54.542505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.256868+0000","last_change":"2023-10-16T11:01:18.252580+0000","last_active":"2023-10-16T11:01:18.256868+0000","last_peered":"2023-10-16T11:01:18.256868+0000","last_clean":"2023-10-16T11:01:18.256868+0000","last_became_active":"2023-10-16T11:01:18.252289+0000","last_became_peered":"2023-10-16T11:01:18.252289+0000","last_unstale":"2023-10-16T11:01:18.256868+0000","last_undegraded":"2023-10-16T11:01:18.256868+0000","last_fullsized":"2023-10-16T11:01:18.256868+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:37.646535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244077+0000","last_change":"2023-10-16T11:01:18.255742+0000","last_active":"2023-10-16T11:01:19.244077+0000","last_peered":"2023-10-16T11:01:19.244077+0000","last_clean":"2023-10-16T11:01:19.244077+0000","last_became_active":"2023-10-16T11:01:18.255490+0000","last_became_peered":"2023-10-16T11:01:18.255490+0000","last_unstale":"2023-10-16T11:01:19.244077+0000","last_undegraded":"2023-10-16T11:01:19.244077+0000","last_fullsized":"2023-10-16T11:01:19.244077+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:08.755529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244254+0000","last_change":"2023-10-16T11:01:18.254231+0000","last_active":"2023-10-16T11:01:19.244254+0000","last_peered":"2023-10-16T11:01:19.244254+0000","last_clean":"2023-10-16T11:01:19.244254+0000","last_became_active":"2023-10-16T11:01:18.253964+0000","last_became_peered":"2023-10-16T11:01:18.253964+0000","last_unstale":"2023-10-16T11:01:19.244254+0000","last_undegraded":"2023-10-16T11:01:19.244254+0000","last_fullsized":"2023-10-16T11:01:19.244254+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:00.368989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240703+0000","last_change":"2023-10-16T11:01:18.253401+0000","last_active":"2023-10-16T11:01:19.240703+0000","last_peered":"2023-10-16T11:01:19.240703+0000","last_clean":"2023-10-16T11:01:19.240703+0000","last_became_active":"2023-10-16T11:01:18.253103+0000","last_became_peered":"2023-10-16T11:01:18.253103+0000","last_unstale":"2023-10-16T11:01:19.240703+0000","last_undegraded":"2023-10-16T11:01:19.240703+0000","last_fullsized":"2023-10-16T11:01:19.240703+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:27.026714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.246921+0000","last_change":"2023-10-16T11:01:18.244774+0000","last_active":"2023-10-16T11:01:18.246921+0000","last_peered":"2023-10-16T11:01:18.246921+0000","last_clean":"2023-10-16T11:01:18.246921+0000","last_became_active":"2023-10-16T11:01:18.244464+0000","last_became_peered":"2023-10-16T11:01:18.244464+0000","last_unstale":"2023-10-16T11:01:18.246921+0000","last_undegraded":"2023-10-16T11:01:18.246921+0000","last_fullsized":"2023-10-16T11:01:18.246921+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:52.905843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801078+0000","last_change":"2023-10-16T11:01:18.266009+0000","last_active":"2023-10-16T11:01:19.801078+0000","last_peered":"2023-10-16T11:01:19.801078+0000","last_clean":"2023-10-16T11:01:19.801078+0000","last_became_active":"2023-10-16T11:01:18.265719+0000","last_became_peered":"2023-10-16T11:01:18.265719+0000","last_unstale":"2023-10-16T11:01:19.801078+0000","last_undegraded":"2023-10-16T11:01:19.801078+0000","last_fullsized":"2023-10-16T11:01:19.801078+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:47.173788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.800999+0000","last_change":"2023-10-16T11:01:18.255229+0000","last_active":"2023-10-16T11:01:19.800999+0000","last_peered":"2023-10-16T11:01:19.800999+0000","last_clean":"2023-10-16T11:01:19.800999+0000","last_became_active":"2023-10-16T11:01:18.254951+0000","last_became_peered":"2023-10-16T11:01:18.254951+0000","last_unstale":"2023-10-16T11:01:19.800999+0000","last_undegraded":"2023-10-16T11:01:19.800999+0000","last_fullsized":"2023-10-16T11:01:19.800999+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:22.313327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801436+0000","last_change":"2023-10-16T11:01:18.266965+0000","last_active":"2023-10-16T11:01:19.801436+0000","last_peered":"2023-10-16T11:01:19.801436+0000","last_clean":"2023-10-16T11:01:19.801436+0000","last_became_active":"2023-10-16T11:01:18.265343+0000","last_became_peered":"2023-10-16T11:01:18.265343+0000","last_unstale":"2023-10-16T11:01:19.801436+0000","last_undegraded":"2023-10-16T11:01:19.801436+0000","last_fullsized":"2023-10-16T11:01:19.801436+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:40.734466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801474+0000","last_change":"2023-10-16T11:01:18.270104+0000","last_active":"2023-10-16T11:01:19.801474+0000","last_peered":"2023-10-16T11:01:19.801474+0000","last_clean":"2023-10-16T11:01:19.801474+0000","last_became_active":"2023-10-16T11:01:18.269462+0000","last_became_peered":"2023-10-16T11:01:18.269462+0000","last_unstale":"2023-10-16T11:01:19.801474+0000","last_undegraded":"2023-10-16T11:01:19.801474+0000","last_fullsized":"2023-10-16T11:01:19.801474+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:46.872260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258609+0000","last_change":"2023-10-16T11:01:18.253298+0000","last_active":"2023-10-16T11:01:18.258609+0000","last_peered":"2023-10-16T11:01:18.258609+0000","last_clean":"2023-10-16T11:01:18.258609+0000","last_became_active":"2023-10-16T11:01:18.253048+0000","last_became_peered":"2023-10-16T11:01:18.253048+0000","last_unstale":"2023-10-16T11:01:18.258609+0000","last_undegraded":"2023-10-16T11:01:18.258609+0000","last_fullsized":"2023-10-16T11:01:18.258609+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:12.531044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260294+0000","last_change":"2023-10-16T11:01:18.256459+0000","last_active":"2023-10-16T11:01:18.260294+0000","last_peered":"2023-10-16T11:01:18.260294+0000","last_clean":"2023-10-16T11:01:18.260294+0000","last_became_active":"2023-10-16T11:01:18.256165+0000","last_became_peered":"2023-10-16T11:01:18.256165+0000","last_unstale":"2023-10-16T11:01:18.260294+0000","last_undegraded":"2023-10-16T11:01:18.260294+0000","last_fullsized":"2023-10-16T11:01:18.260294+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:44:26.428306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243503+0000","last_change":"2023-10-16T11:01:18.246189+0000","last_active":"2023-10-16T11:01:19.243503+0000","last_peered":"2023-10-16T11:01:19.243503+0000","last_clean":"2023-10-16T11:01:19.243503+0000","last_became_active":"2023-10-16T11:01:18.245955+0000","last_became_peered":"2023-10-16T11:01:18.245955+0000","last_unstale":"2023-10-16T11:01:19.243503+0000","last_undegraded":"2023-10-16T11:01:19.243503+0000","last_fullsized":"2023-10-16T11:01:19.243503+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:19.358042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240569+0000","last_change":"2023-10-16T11:01:18.256243+0000","last_active":"2023-10-16T11:01:19.240569+0000","last_peered":"2023-10-16T11:01:19.240569+0000","last_clean":"2023-10-16T11:01:19.240569+0000","last_became_active":"2023-10-16T11:01:18.255971+0000","last_became_peered":"2023-10-16T11:01:18.255971+0000","last_unstale":"2023-10-16T11:01:19.240569+0000","last_undegraded":"2023-10-16T11:01:19.240569+0000","last_fullsized":"2023-10-16T11:01:19.240569+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:16.766240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260398+0000","last_change":"2023-10-16T11:01:18.256579+0000","last_active":"2023-10-16T11:01:18.260398+0000","last_peered":"2023-10-16T11:01:18.260398+0000","last_clean":"2023-10-16T11:01:18.260398+0000","last_became_active":"2023-10-16T11:01:18.255894+0000","last_became_peered":"2023-10-16T11:01:18.255894+0000","last_unstale":"2023-10-16T11:01:18.260398+0000","last_undegraded":"2023-10-16T11:01:18.260398+0000","last_fullsized":"2023-10-16T11:01:18.260398+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:27.026714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243749+0000","last_change":"2023-10-16T11:01:18.250473+0000","last_active":"2023-10-16T11:01:19.243749+0000","last_peered":"2023-10-16T11:01:19.243749+0000","last_clean":"2023-10-16T11:01:19.243749+0000","last_became_active":"2023-10-16T11:01:18.250014+0000","last_became_peered":"2023-10-16T11:01:18.250014+0000","last_unstale":"2023-10-16T11:01:19.243749+0000","last_undegraded":"2023-10-16T11:01:19.243749+0000","last_fullsized":"2023-10-16T11:01:19.243749+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:13.713337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801467+0000","last_change":"2023-10-16T11:01:18.265316+0000","last_active":"2023-10-16T11:01:19.801467+0000","last_peered":"2023-10-16T11:01:19.801467+0000","last_clean":"2023-10-16T11:01:19.801467+0000","last_became_active":"2023-10-16T11:01:18.264960+0000","last_became_peered":"2023-10-16T11:01:18.264960+0000","last_unstale":"2023-10-16T11:01:19.801467+0000","last_undegraded":"2023-10-16T11:01:19.801467+0000","last_fullsized":"2023-10-16T11:01:19.801467+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:18.324079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.272773+0000","last_change":"2023-10-16T11:01:18.269626+0000","last_active":"2023-10-16T11:01:18.272773+0000","last_peered":"2023-10-16T11:01:18.272773+0000","last_clean":"2023-10-16T11:01:18.272773+0000","last_became_active":"2023-10-16T11:01:18.269307+0000","last_became_peered":"2023-10-16T11:01:18.269307+0000","last_unstale":"2023-10-16T11:01:18.272773+0000","last_undegraded":"2023-10-16T11:01:18.272773+0000","last_fullsized":"2023-10-16T11:01:18.272773+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:14.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.268595+0000","last_change":"2023-10-16T11:01:18.265253+0000","last_active":"2023-10-16T11:01:18.268595+0000","last_peered":"2023-10-16T11:01:18.268595+0000","last_clean":"2023-10-16T11:01:18.268595+0000","last_became_active":"2023-10-16T11:01:18.264988+0000","last_became_peered":"2023-10-16T11:01:18.264988+0000","last_unstale":"2023-10-16T11:01:18.268595+0000","last_undegraded":"2023-10-16T11:01:18.268595+0000","last_fullsized":"2023-10-16T11:01:18.268595+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:46.822996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260931+0000","last_change":"2023-10-16T11:01:18.257678+0000","last_active":"2023-10-16T11:01:18.260931+0000","last_peered":"2023-10-16T11:01:18.260931+0000","last_clean":"2023-10-16T11:01:18.260931+0000","last_became_active":"2023-10-16T11:01:18.257478+0000","last_became_peered":"2023-10-16T11:01:18.257478+0000","last_unstale":"2023-10-16T11:01:18.260931+0000","last_undegraded":"2023-10-16T11:01:18.260931+0000","last_fullsized":"2023-10-16T11:01:18.260931+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:32.903365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243683+0000","last_change":"2023-10-16T11:01:18.246002+0000","last_active":"2023-10-16T11:01:19.243683+0000","last_peered":"2023-10-16T11:01:19.243683+0000","last_clean":"2023-10-16T11:01:19.243683+0000","last_became_active":"2023-10-16T11:01:18.245715+0000","last_became_peered":"2023-10-16T11:01:18.245715+0000","last_unstale":"2023-10-16T11:01:19.243683+0000","last_undegraded":"2023-10-16T11:01:19.243683+0000","last_fullsized":"2023-10-16T11:01:19.243683+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:58.914088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.256694+0000","last_change":"2023-10-16T11:01:18.250100+0000","last_active":"2023-10-16T11:01:18.256694+0000","last_peered":"2023-10-16T11:01:18.256694+0000","last_clean":"2023-10-16T11:01:18.256694+0000","last_became_active":"2023-10-16T11:01:18.249784+0000","last_became_peered":"2023-10-16T11:01:18.249784+0000","last_unstale":"2023-10-16T11:01:18.256694+0000","last_undegraded":"2023-10-16T11:01:18.256694+0000","last_fullsized":"2023-10-16T11:01:18.256694+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:29.401941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240450+0000","last_change":"2023-10-16T11:01:18.255427+0000","last_active":"2023-10-16T11:01:19.240450+0000","last_peered":"2023-10-16T11:01:19.240450+0000","last_clean":"2023-10-16T11:01:19.240450+0000","last_became_active":"2023-10-16T11:01:18.255159+0000","last_became_peered":"2023-10-16T11:01:18.255159+0000","last_unstale":"2023-10-16T11:01:19.240450+0000","last_undegraded":"2023-10-16T11:01:19.240450+0000","last_fullsized":"2023-10-16T11:01:19.240450+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:16.339682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243848+0000","last_change":"2023-10-16T11:01:18.250294+0000","last_active":"2023-10-16T11:01:19.243848+0000","last_peered":"2023-10-16T11:01:19.243848+0000","last_clean":"2023-10-16T11:01:19.243848+0000","last_became_active":"2023-10-16T11:01:18.249664+0000","last_became_peered":"2023-10-16T11:01:18.249664+0000","last_unstale":"2023-10-16T11:01:19.243848+0000","last_undegraded":"2023-10-16T11:01:19.243848+0000","last_fullsized":"2023-10-16T11:01:19.243848+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:00.049769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243747+0000","last_change":"2023-10-16T11:01:18.255135+0000","last_active":"2023-10-16T11:01:19.243747+0000","last_peered":"2023-10-16T11:01:19.243747+0000","last_clean":"2023-10-16T11:01:19.243747+0000","last_became_active":"2023-10-16T11:01:18.254875+0000","last_became_peered":"2023-10-16T11:01:18.254875+0000","last_unstale":"2023-10-16T11:01:19.243747+0000","last_undegraded":"2023-10-16T11:01:19.243747+0000","last_fullsized":"2023-10-16T11:01:19.243747+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:08.634340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257514+0000","last_change":"2023-10-16T11:01:18.254259+0000","last_active":"2023-10-16T11:01:18.257514+0000","last_peered":"2023-10-16T11:01:18.257514+0000","last_clean":"2023-10-16T11:01:18.257514+0000","last_became_active":"2023-10-16T11:01:18.254015+0000","last_became_peered":"2023-10-16T11:01:18.254015+0000","last_unstale":"2023-10-16T11:01:18.257514+0000","last_undegraded":"2023-10-16T11:01:18.257514+0000","last_fullsized":"2023-10-16T11:01:18.257514+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:20.651168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260140+0000","last_change":"2023-10-16T11:01:18.250740+0000","last_active":"2023-10-16T11:01:18.260140+0000","last_peered":"2023-10-16T11:01:18.260140+0000","last_clean":"2023-10-16T11:01:18.260140+0000","last_became_active":"2023-10-16T11:01:18.250280+0000","last_became_peered":"2023-10-16T11:01:18.250280+0000","last_unstale":"2023-10-16T11:01:18.260140+0000","last_undegraded":"2023-10-16T11:01:18.260140+0000","last_fullsized":"2023-10-16T11:01:18.260140+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:45:28.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243557+0000","last_change":"2023-10-16T11:01:18.259015+0000","last_active":"2023-10-16T11:01:19.243557+0000","last_peered":"2023-10-16T11:01:19.243557+0000","last_clean":"2023-10-16T11:01:19.243557+0000","last_became_active":"2023-10-16T11:01:18.258706+0000","last_became_peered":"2023-10-16T11:01:18.258706+0000","last_unstale":"2023-10-16T11:01:19.243557+0000","last_undegraded":"2023-10-16T11:01:19.243557+0000","last_fullsized":"2023-10-16T11:01:19.243557+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:32.959040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260367+0000","last_change":"2023-10-16T11:01:18.256478+0000","last_active":"2023-10-16T11:01:18.260367+0000","last_peered":"2023-10-16T11:01:18.260367+0000","last_clean":"2023-10-16T11:01:18.260367+0000","last_became_active":"2023-10-16T11:01:18.256189+0000","last_became_peered":"2023-10-16T11:01:18.256189+0000","last_unstale":"2023-10-16T11:01:18.260367+0000","last_undegraded":"2023-10-16T11:01:18.260367+0000","last_fullsized":"2023-10-16T11:01:18.260367+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:15.391349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257970+0000","last_change":"2023-10-16T11:01:18.254231+0000","last_active":"2023-10-16T11:01:18.257970+0000","last_peered":"2023-10-16T11:01:18.257970+0000","last_clean":"2023-10-16T11:01:18.257970+0000","last_became_active":"2023-10-16T11:01:18.253953+0000","last_became_peered":"2023-10-16T11:01:18.253953+0000","last_unstale":"2023-10-16T11:01:18.257970+0000","last_undegraded":"2023-10-16T11:01:18.257970+0000","last_fullsized":"2023-10-16T11:01:18.257970+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:54.822998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260377+0000","last_change":"2023-10-16T11:01:18.255668+0000","last_active":"2023-10-16T11:01:18.260377+0000","last_peered":"2023-10-16T11:01:18.260377+0000","last_clean":"2023-10-16T11:01:18.260377+0000","last_became_active":"2023-10-16T11:01:18.255360+0000","last_became_peered":"2023-10-16T11:01:18.255360+0000","last_unstale":"2023-10-16T11:01:18.260377+0000","last_undegraded":"2023-10-16T11:01:18.260377+0000","last_fullsized":"2023-10-16T11:01:18.260377+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:16.339682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260355+0000","last_change":"2023-10-16T11:01:18.258816+0000","last_active":"2023-10-16T11:01:18.260355+0000","last_peered":"2023-10-16T11:01:18.260355+0000","last_clean":"2023-10-16T11:01:18.260355+0000","last_became_active":"2023-10-16T11:01:18.258547+0000","last_became_peered":"2023-10-16T11:01:18.258547+0000","last_unstale":"2023-10-16T11:01:18.260355+0000","last_undegraded":"2023-10-16T11:01:18.260355+0000","last_fullsized":"2023-10-16T11:01:18.260355+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:30.425859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260057+0000","last_change":"2023-10-16T11:01:18.250795+0000","last_active":"2023-10-16T11:01:18.260057+0000","last_peered":"2023-10-16T11:01:18.260057+0000","last_clean":"2023-10-16T11:01:18.260057+0000","last_became_active":"2023-10-16T11:01:18.250514+0000","last_became_peered":"2023-10-16T11:01:18.250514+0000","last_unstale":"2023-10-16T11:01:18.260057+0000","last_undegraded":"2023-10-16T11:01:18.260057+0000","last_fullsized":"2023-10-16T11:01:18.260057+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:16.766240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"184'4","reported_seq":16,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244435+0000","last_change":"2023-10-16T11:01:17.249870+0000","last_active":"2023-10-16T11:01:19.244435+0000","last_peered":"2023-10-16T11:01:19.244435+0000","last_clean":"2023-10-16T11:01:19.244435+0000","last_became_active":"2023-10-16T11:01:17.249572+0000","last_became_peered":"2023-10-16T11:01:17.249572+0000","last_unstale":"2023-10-16T11:01:19.244435+0000","last_undegraded":"2023-10-16T11:01:19.244435+0000","last_fullsized":"2023-10-16T11:01:19.244435+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:15:35.200519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243336+0000","last_change":"2023-10-16T11:01:18.252603+0000","last_active":"2023-10-16T11:01:19.243336+0000","last_peered":"2023-10-16T11:01:19.243336+0000","last_clean":"2023-10-16T11:01:19.243336+0000","last_became_active":"2023-10-16T11:01:18.252190+0000","last_became_peered":"2023-10-16T11:01:18.252190+0000","last_unstale":"2023-10-16T11:01:19.243336+0000","last_undegraded":"2023-10-16T11:01:19.243336+0000","last_fullsized":"2023-10-16T11:01:19.243336+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:20.787643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.241057+0000","last_change":"2023-10-16T11:01:17.264091+0000","last_active":"2023-10-16T11:01:19.241057+0000","last_peered":"2023-10-16T11:01:19.241057+0000","last_clean":"2023-10-16T11:01:19.241057+0000","last_became_active":"2023-10-16T11:01:17.263800+0000","last_became_peered":"2023-10-16T11:01:17.263800+0000","last_unstale":"2023-10-16T11:01:19.241057+0000","last_undegraded":"2023-10-16T11:01:19.241057+0000","last_fullsized":"2023-10-16T11:01:19.241057+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:11.526438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240573+0000","last_change":"2023-10-16T11:01:18.261131+0000","last_active":"2023-10-16T11:01:19.240573+0000","last_peered":"2023-10-16T11:01:19.240573+0000","last_clean":"2023-10-16T11:01:19.240573+0000","last_became_active":"2023-10-16T11:01:18.260822+0000","last_became_peered":"2023-10-16T11:01:18.260822+0000","last_unstale":"2023-10-16T11:01:19.240573+0000","last_undegraded":"2023-10-16T11:01:19.240573+0000","last_fullsized":"2023-10-16T11:01:19.240573+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:15.391349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"184'2","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.273535+0000","last_change":"2023-10-16T11:01:17.248279+0000","last_active":"2023-10-16T11:01:18.273535+0000","last_peered":"2023-10-16T11:01:18.273535+0000","last_clean":"2023-10-16T11:01:18.273535+0000","last_became_active":"2023-10-16T11:01:17.247994+0000","last_became_peered":"2023-10-16T11:01:17.247994+0000","last_unstale":"2023-10-16T11:01:18.273535+0000","last_undegraded":"2023-10-16T11:01:18.273535+0000","last_fullsized":"2023-10-16T11:01:18.273535+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:55.782004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.256777+0000","last_change":"2023-10-16T11:01:18.251617+0000","last_active":"2023-10-16T11:01:18.256777+0000","last_peered":"2023-10-16T11:01:18.256777+0000","last_clean":"2023-10-16T11:01:18.256777+0000","last_became_active":"2023-10-16T11:01:18.251166+0000","last_became_peered":"2023-10-16T11:01:18.251166+0000","last_unstale":"2023-10-16T11:01:18.256777+0000","last_undegraded":"2023-10-16T11:01:18.256777+0000","last_fullsized":"2023-10-16T11:01:18.256777+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:56.786610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"184'2","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.268022+0000","last_change":"2023-10-16T11:01:17.251826+0000","last_active":"2023-10-16T11:01:18.268022+0000","last_peered":"2023-10-16T11:01:18.268022+0000","last_clean":"2023-10-16T11:01:18.268022+0000","last_became_active":"2023-10-16T11:01:17.251537+0000","last_became_peered":"2023-10-16T11:01:17.251537+0000","last_unstale":"2023-10-16T11:01:18.268022+0000","last_undegraded":"2023-10-16T11:01:18.268022+0000","last_fullsized":"2023-10-16T11:01:18.268022+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:09.418488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801203+0000","last_change":"2023-10-16T11:01:18.255277+0000","last_active":"2023-10-16T11:01:19.801203+0000","last_peered":"2023-10-16T11:01:19.801203+0000","last_clean":"2023-10-16T11:01:19.801203+0000","last_became_active":"2023-10-16T11:01:18.255011+0000","last_became_peered":"2023-10-16T11:01:18.255011+0000","last_unstale":"2023-10-16T11:01:19.801203+0000","last_undegraded":"2023-10-16T11:01:19.801203+0000","last_fullsized":"2023-10-16T11:01:19.801203+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:00:29.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801149+0000","last_change":"2023-10-16T11:01:17.256175+0000","last_active":"2023-10-16T11:01:19.801149+0000","last_peered":"2023-10-16T11:01:19.801149+0000","last_clean":"2023-10-16T11:01:19.801149+0000","last_became_active":"2023-10-16T11:01:17.255810+0000","last_became_peered":"2023-10-16T11:01:17.255810+0000","last_unstale":"2023-10-16T11:01:19.801149+0000","last_undegraded":"2023-10-16T11:01:19.801149+0000","last_fullsized":"2023-10-16T11:01:19.801149+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:53.042731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258638+0000","last_change":"2023-10-16T11:01:18.256094+0000","last_active":"2023-10-16T11:01:18.258638+0000","last_peered":"2023-10-16T11:01:18.258638+0000","last_clean":"2023-10-16T11:01:18.258638+0000","last_became_active":"2023-10-16T11:01:18.255820+0000","last_became_peered":"2023-10-16T11:01:18.255820+0000","last_unstale":"2023-10-16T11:01:18.258638+0000","last_undegraded":"2023-10-16T11:01:18.258638+0000","last_fullsized":"2023-10-16T11:01:18.258638+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:04.154686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240666+0000","last_change":"2023-10-16T11:01:17.249761+0000","last_active":"2023-10-16T11:01:19.240666+0000","last_peered":"2023-10-16T11:01:19.240666+0000","last_clean":"2023-10-16T11:01:19.240666+0000","last_became_active":"2023-10-16T11:01:17.249502+0000","last_became_peered":"2023-10-16T11:01:17.249502+0000","last_unstale":"2023-10-16T11:01:19.240666+0000","last_undegraded":"2023-10-16T11:01:19.240666+0000","last_fullsized":"2023-10-16T11:01:19.240666+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:11.693570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258584+0000","last_change":"2023-10-16T11:01:18.256079+0000","last_active":"2023-10-16T11:01:18.258584+0000","last_peered":"2023-10-16T11:01:18.258584+0000","last_clean":"2023-10-16T11:01:18.258584+0000","last_became_active":"2023-10-16T11:01:18.255800+0000","last_became_peered":"2023-10-16T11:01:18.255800+0000","last_unstale":"2023-10-16T11:01:18.258584+0000","last_undegraded":"2023-10-16T11:01:18.258584+0000","last_fullsized":"2023-10-16T11:01:18.258584+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:58.891749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.245445+0000","last_change":"2023-10-16T11:01:17.257413+0000","last_active":"2023-10-16T11:01:18.245445+0000","last_peered":"2023-10-16T11:01:18.245445+0000","last_clean":"2023-10-16T11:01:18.245445+0000","last_became_active":"2023-10-16T11:01:17.256961+0000","last_became_peered":"2023-10-16T11:01:17.256961+0000","last_unstale":"2023-10-16T11:01:18.245445+0000","last_undegraded":"2023-10-16T11:01:18.245445+0000","last_fullsized":"2023-10-16T11:01:18.245445+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.959786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244468+0000","last_change":"2023-10-16T11:01:18.259493+0000","last_active":"2023-10-16T11:01:19.244468+0000","last_peered":"2023-10-16T11:01:19.244468+0000","last_clean":"2023-10-16T11:01:19.244468+0000","last_became_active":"2023-10-16T11:01:18.259196+0000","last_became_peered":"2023-10-16T11:01:18.259196+0000","last_unstale":"2023-10-16T11:01:19.244468+0000","last_undegraded":"2023-10-16T11:01:19.244468+0000","last_fullsized":"2023-10-16T11:01:19.244468+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:58.071953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.800952+0000","last_change":"2023-10-16T11:01:17.255013+0000","last_active":"2023-10-16T11:01:19.800952+0000","last_peered":"2023-10-16T11:01:19.800952+0000","last_clean":"2023-10-16T11:01:19.800952+0000","last_became_active":"2023-10-16T11:01:17.254681+0000","last_became_peered":"2023-10-16T11:01:17.254681+0000","last_unstale":"2023-10-16T11:01:19.800952+0000","last_undegraded":"2023-10-16T11:01:19.800952+0000","last_fullsized":"2023-10-16T11:01:19.800952+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:52.238527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260034+0000","last_change":"2023-10-16T11:01:18.250323+0000","last_active":"2023-10-16T11:01:18.260034+0000","last_peered":"2023-10-16T11:01:18.260034+0000","last_clean":"2023-10-16T11:01:18.260034+0000","last_became_active":"2023-10-16T11:01:18.249985+0000","last_became_peered":"2023-10-16T11:01:18.249985+0000","last_unstale":"2023-10-16T11:01:18.260034+0000","last_undegraded":"2023-10-16T11:01:18.260034+0000","last_fullsized":"2023-10-16T11:01:18.260034+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:27.526202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243589+0000","last_change":"2023-10-16T11:01:17.256567+0000","last_active":"2023-10-16T11:01:19.243589+0000","last_peered":"2023-10-16T11:01:19.243589+0000","last_clean":"2023-10-16T11:01:19.243589+0000","last_became_active":"2023-10-16T11:01:17.256294+0000","last_became_peered":"2023-10-16T11:01:17.256294+0000","last_unstale":"2023-10-16T11:01:19.243589+0000","last_undegraded":"2023-10-16T11:01:19.243589+0000","last_fullsized":"2023-10-16T11:01:19.243589+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:01.846023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243751+0000","last_change":"2023-10-16T11:01:18.248345+0000","last_active":"2023-10-16T11:01:19.243751+0000","last_peered":"2023-10-16T11:01:19.243751+0000","last_clean":"2023-10-16T11:01:19.243751+0000","last_became_active":"2023-10-16T11:01:18.248078+0000","last_became_peered":"2023-10-16T11:01:18.248078+0000","last_unstale":"2023-10-16T11:01:19.243751+0000","last_undegraded":"2023-10-16T11:01:19.243751+0000","last_fullsized":"2023-10-16T11:01:19.243751+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:50:27.706495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"184'3","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244088+0000","last_change":"2023-10-16T11:01:17.249021+0000","last_active":"2023-10-16T11:01:19.244088+0000","last_peered":"2023-10-16T11:01:19.244088+0000","last_clean":"2023-10-16T11:01:19.244088+0000","last_became_active":"2023-10-16T11:01:17.248730+0000","last_became_peered":"2023-10-16T11:01:17.248730+0000","last_unstale":"2023-10-16T11:01:19.244088+0000","last_undegraded":"2023-10-16T11:01:19.244088+0000","last_fullsized":"2023-10-16T11:01:19.244088+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:39.271743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257801+0000","last_change":"2023-10-16T11:01:18.254241+0000","last_active":"2023-10-16T11:01:18.257801+0000","last_peered":"2023-10-16T11:01:18.257801+0000","last_clean":"2023-10-16T11:01:18.257801+0000","last_became_active":"2023-10-16T11:01:18.253963+0000","last_became_peered":"2023-10-16T11:01:18.253963+0000","last_unstale":"2023-10-16T11:01:18.257801+0000","last_undegraded":"2023-10-16T11:01:18.257801+0000","last_fullsized":"2023-10-16T11:01:18.257801+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:56.315152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"184'6","reported_seq":18,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243622+0000","last_change":"2023-10-16T11:01:17.263874+0000","last_active":"2023-10-16T11:01:19.243622+0000","last_peered":"2023-10-16T11:01:19.243622+0000","last_clean":"2023-10-16T11:01:19.243622+0000","last_became_active":"2023-10-16T11:01:17.263556+0000","last_became_peered":"2023-10-16T11:01:17.263556+0000","last_unstale":"2023-10-16T11:01:19.243622+0000","last_undegraded":"2023-10-16T11:01:19.243622+0000","last_fullsized":"2023-10-16T11:01:19.243622+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:09.448108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260316+0000","last_change":"2023-10-16T11:01:18.258109+0000","last_active":"2023-10-16T11:01:18.260316+0000","last_peered":"2023-10-16T11:01:18.260316+0000","last_clean":"2023-10-16T11:01:18.260316+0000","last_became_active":"2023-10-16T11:01:18.257792+0000","last_became_peered":"2023-10-16T11:01:18.257792+0000","last_unstale":"2023-10-16T11:01:18.260316+0000","last_undegraded":"2023-10-16T11:01:18.260316+0000","last_fullsized":"2023-10-16T11:01:18.260316+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:16.601581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240667+0000","last_change":"2023-10-16T11:01:17.254494+0000","last_active":"2023-10-16T11:01:19.240667+0000","last_peered":"2023-10-16T11:01:19.240667+0000","last_clean":"2023-10-16T11:01:19.240667+0000","last_became_active":"2023-10-16T11:01:17.254186+0000","last_became_peered":"2023-10-16T11:01:17.254186+0000","last_unstale":"2023-10-16T11:01:19.240667+0000","last_undegraded":"2023-10-16T11:01:19.240667+0000","last_fullsized":"2023-10-16T11:01:19.240667+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:10.252312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801301+0000","last_change":"2023-10-16T11:01:18.256547+0000","last_active":"2023-10-16T11:01:19.801301+0000","last_peered":"2023-10-16T11:01:19.801301+0000","last_clean":"2023-10-16T11:01:19.801301+0000","last_became_active":"2023-10-16T11:01:18.256177+0000","last_became_peered":"2023-10-16T11:01:18.256177+0000","last_unstale":"2023-10-16T11:01:19.801301+0000","last_undegraded":"2023-10-16T11:01:19.801301+0000","last_fullsized":"2023-10-16T11:01:19.801301+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:46.990440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243423+0000","last_change":"2023-10-16T11:01:17.249754+0000","last_active":"2023-10-16T11:01:19.243423+0000","last_peered":"2023-10-16T11:01:19.243423+0000","last_clean":"2023-10-16T11:01:19.243423+0000","last_became_active":"2023-10-16T11:01:17.249475+0000","last_became_peered":"2023-10-16T11:01:17.249475+0000","last_unstale":"2023-10-16T11:01:19.243423+0000","last_undegraded":"2023-10-16T11:01:19.243423+0000","last_fullsized":"2023-10-16T11:01:19.243423+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:20.247320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.261762+0000","last_change":"2023-10-16T11:01:18.258206+0000","last_active":"2023-10-16T11:01:18.261762+0000","last_peered":"2023-10-16T11:01:18.261762+0000","last_clean":"2023-10-16T11:01:18.261762+0000","last_became_active":"2023-10-16T11:01:18.257932+0000","last_became_peered":"2023-10-16T11:01:18.257932+0000","last_unstale":"2023-10-16T11:01:18.261762+0000","last_undegraded":"2023-10-16T11:01:18.261762+0000","last_fullsized":"2023-10-16T11:01:18.261762+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:16.766240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.247699+0000","last_change":"2023-10-16T11:01:17.254588+0000","last_active":"2023-10-16T11:01:18.247699+0000","last_peered":"2023-10-16T11:01:18.247699+0000","last_clean":"2023-10-16T11:01:18.247699+0000","last_became_active":"2023-10-16T11:01:17.254280+0000","last_became_peered":"2023-10-16T11:01:17.254280+0000","last_unstale":"2023-10-16T11:01:18.247699+0000","last_undegraded":"2023-10-16T11:01:18.247699+0000","last_fullsized":"2023-10-16T11:01:18.247699+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:28.397335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.262168+0000","last_change":"2023-10-16T11:01:18.257804+0000","last_active":"2023-10-16T11:01:18.262168+0000","last_peered":"2023-10-16T11:01:18.262168+0000","last_clean":"2023-10-16T11:01:18.262168+0000","last_became_active":"2023-10-16T11:01:18.257545+0000","last_became_peered":"2023-10-16T11:01:18.257545+0000","last_unstale":"2023-10-16T11:01:18.262168+0000","last_undegraded":"2023-10-16T11:01:18.262168+0000","last_fullsized":"2023-10-16T11:01:18.262168+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:16.332113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"184'8","reported_seq":20,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801110+0000","last_change":"2023-10-16T11:01:17.251687+0000","last_active":"2023-10-16T11:01:19.801110+0000","last_peered":"2023-10-16T11:01:19.801110+0000","last_clean":"2023-10-16T11:01:19.801110+0000","last_became_active":"2023-10-16T11:01:17.251419+0000","last_became_peered":"2023-10-16T11:01:17.251419+0000","last_unstale":"2023-10-16T11:01:19.801110+0000","last_undegraded":"2023-10-16T11:01:19.801110+0000","last_fullsized":"2023-10-16T11:01:19.801110+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:24.975550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801190+0000","last_change":"2023-10-16T11:01:18.272702+0000","last_active":"2023-10-16T11:01:19.801190+0000","last_peered":"2023-10-16T11:01:19.801190+0000","last_clean":"2023-10-16T11:01:19.801190+0000","last_became_active":"2023-10-16T11:01:18.272391+0000","last_became_peered":"2023-10-16T11:01:18.272391+0000","last_unstale":"2023-10-16T11:01:19.801190+0000","last_undegraded":"2023-10-16T11:01:19.801190+0000","last_fullsized":"2023-10-16T11:01:19.801190+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:29.866224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243679+0000","last_change":"2023-10-16T11:01:17.247703+0000","last_active":"2023-10-16T11:01:19.243679+0000","last_peered":"2023-10-16T11:01:19.243679+0000","last_clean":"2023-10-16T11:01:19.243679+0000","last_became_active":"2023-10-16T11:01:17.247411+0000","last_became_peered":"2023-10-16T11:01:17.247411+0000","last_unstale":"2023-10-16T11:01:19.243679+0000","last_undegraded":"2023-10-16T11:01:19.243679+0000","last_fullsized":"2023-10-16T11:01:19.243679+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:15.421754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257287+0000","last_change":"2023-10-16T11:01:18.252422+0000","last_active":"2023-10-16T11:01:18.257287+0000","last_peered":"2023-10-16T11:01:18.257287+0000","last_clean":"2023-10-16T11:01:18.257287+0000","last_became_active":"2023-10-16T11:01:18.252131+0000","last_became_peered":"2023-10-16T11:01:18.252131+0000","last_unstale":"2023-10-16T11:01:18.257287+0000","last_undegraded":"2023-10-16T11:01:18.257287+0000","last_fullsized":"2023-10-16T11:01:18.257287+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:07.220099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801380+0000","last_change":"2023-10-16T11:01:17.256863+0000","last_active":"2023-10-16T11:01:19.801380+0000","last_peered":"2023-10-16T11:01:19.801380+0000","last_clean":"2023-10-16T11:01:19.801380+0000","last_became_active":"2023-10-16T11:01:17.256609+0000","last_became_peered":"2023-10-16T11:01:17.256609+0000","last_unstale":"2023-10-16T11:01:19.801380+0000","last_undegraded":"2023-10-16T11:01:19.801380+0000","last_fullsized":"2023-10-16T11:01:19.801380+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:32.973887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.261812+0000","last_change":"2023-10-16T11:01:18.259143+0000","last_active":"2023-10-16T11:01:18.261812+0000","last_peered":"2023-10-16T11:01:18.261812+0000","last_clean":"2023-10-16T11:01:18.261812+0000","last_became_active":"2023-10-16T11:01:18.258882+0000","last_became_peered":"2023-10-16T11:01:18.258882+0000","last_unstale":"2023-10-16T11:01:18.261812+0000","last_undegraded":"2023-10-16T11:01:18.261812+0000","last_fullsized":"2023-10-16T11:01:18.261812+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:11.256918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.241082+0000","last_change":"2023-10-16T11:01:17.250819+0000","last_active":"2023-10-16T11:01:19.241082+0000","last_peered":"2023-10-16T11:01:19.241082+0000","last_clean":"2023-10-16T11:01:19.241082+0000","last_became_active":"2023-10-16T11:01:17.250532+0000","last_became_peered":"2023-10-16T11:01:17.250532+0000","last_unstale":"2023-10-16T11:01:19.241082+0000","last_undegraded":"2023-10-16T11:01:19.241082+0000","last_fullsized":"2023-10-16T11:01:19.241082+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:29.070461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260305+0000","last_change":"2023-10-16T11:01:18.255052+0000","last_active":"2023-10-16T11:01:18.260305+0000","last_peered":"2023-10-16T11:01:18.260305+0000","last_clean":"2023-10-16T11:01:18.260305+0000","last_became_active":"2023-10-16T11:01:18.254772+0000","last_became_peered":"2023-10-16T11:01:18.254772+0000","last_unstale":"2023-10-16T11:01:18.260305+0000","last_undegraded":"2023-10-16T11:01:18.260305+0000","last_fullsized":"2023-10-16T11:01:18.260305+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:15.391349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801383+0000","last_change":"2023-10-16T11:01:17.255507+0000","last_active":"2023-10-16T11:01:19.801383+0000","last_peered":"2023-10-16T11:01:19.801383+0000","last_clean":"2023-10-16T11:01:19.801383+0000","last_became_active":"2023-10-16T11:01:17.255280+0000","last_became_peered":"2023-10-16T11:01:17.255280+0000","last_unstale":"2023-10-16T11:01:19.801383+0000","last_undegraded":"2023-10-16T11:01:19.801383+0000","last_fullsized":"2023-10-16T11:01:19.801383+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:33.602849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243595+0000","last_change":"2023-10-16T11:01:18.250487+0000","last_active":"2023-10-16T11:01:19.243595+0000","last_peered":"2023-10-16T11:01:19.243595+0000","last_clean":"2023-10-16T11:01:19.243595+0000","last_became_active":"2023-10-16T11:01:18.250238+0000","last_became_peered":"2023-10-16T11:01:18.250238+0000","last_unstale":"2023-10-16T11:01:19.243595+0000","last_undegraded":"2023-10-16T11:01:19.243595+0000","last_fullsized":"2023-10-16T11:01:19.243595+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:16.317321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240727+0000","last_change":"2023-10-16T11:01:17.253306+0000","last_active":"2023-10-16T11:01:19.240727+0000","last_peered":"2023-10-16T11:01:19.240727+0000","last_clean":"2023-10-16T11:01:19.240727+0000","last_became_active":"2023-10-16T11:01:17.253049+0000","last_became_peered":"2023-10-16T11:01:17.253049+0000","last_unstale":"2023-10-16T11:01:19.240727+0000","last_undegraded":"2023-10-16T11:01:19.240727+0000","last_fullsized":"2023-10-16T11:01:19.240727+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:45.818390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240989+0000","last_change":"2023-10-16T11:01:18.266452+0000","last_active":"2023-10-16T11:01:19.240989+0000","last_peered":"2023-10-16T11:01:19.240989+0000","last_clean":"2023-10-16T11:01:19.240989+0000","last_became_active":"2023-10-16T11:01:18.266057+0000","last_became_peered":"2023-10-16T11:01:18.266057+0000","last_unstale":"2023-10-16T11:01:19.240989+0000","last_undegraded":"2023-10-16T11:01:19.240989+0000","last_fullsized":"2023-10-16T11:01:19.240989+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:27.526202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.246601+0000","last_change":"2023-10-16T11:01:17.250980+0000","last_active":"2023-10-16T11:01:18.246601+0000","last_peered":"2023-10-16T11:01:18.246601+0000","last_clean":"2023-10-16T11:01:18.246601+0000","last_became_active":"2023-10-16T11:01:17.250422+0000","last_became_peered":"2023-10-16T11:01:17.250422+0000","last_unstale":"2023-10-16T11:01:18.246601+0000","last_undegraded":"2023-10-16T11:01:18.246601+0000","last_fullsized":"2023-10-16T11:01:18.246601+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:08:25.935666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240975+0000","last_change":"2023-10-16T11:01:18.269869+0000","last_active":"2023-10-16T11:01:19.240975+0000","last_peered":"2023-10-16T11:01:19.240975+0000","last_clean":"2023-10-16T11:01:19.240975+0000","last_became_active":"2023-10-16T11:01:18.269590+0000","last_became_peered":"2023-10-16T11:01:18.269590+0000","last_unstale":"2023-10-16T11:01:19.240975+0000","last_undegraded":"2023-10-16T11:01:19.240975+0000","last_fullsized":"2023-10-16T11:01:19.240975+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:29.401941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.246840+0000","last_change":"2023-10-16T11:01:17.251070+0000","last_active":"2023-10-16T11:01:18.246840+0000","last_peered":"2023-10-16T11:01:18.246840+0000","last_clean":"2023-10-16T11:01:18.246840+0000","last_became_active":"2023-10-16T11:01:17.250797+0000","last_became_peered":"2023-10-16T11:01:17.250797+0000","last_unstale":"2023-10-16T11:01:18.246840+0000","last_undegraded":"2023-10-16T11:01:18.246840+0000","last_fullsized":"2023-10-16T11:01:18.246840+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:16.559989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260424+0000","last_change":"2023-10-16T11:01:18.259030+0000","last_active":"2023-10-16T11:01:18.260424+0000","last_peered":"2023-10-16T11:01:18.260424+0000","last_clean":"2023-10-16T11:01:18.260424+0000","last_became_active":"2023-10-16T11:01:18.258738+0000","last_became_peered":"2023-10-16T11:01:18.258738+0000","last_unstale":"2023-10-16T11:01:18.260424+0000","last_undegraded":"2023-10-16T11:01:18.260424+0000","last_fullsized":"2023-10-16T11:01:18.260424+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:50.323773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.240227+0000","last_change":"2023-10-16T11:01:17.253326+0000","last_active":"2023-10-16T11:01:18.240227+0000","last_peered":"2023-10-16T11:01:18.240227+0000","last_clean":"2023-10-16T11:01:18.240227+0000","last_became_active":"2023-10-16T11:01:17.253064+0000","last_became_peered":"2023-10-16T11:01:17.253064+0000","last_unstale":"2023-10-16T11:01:18.240227+0000","last_undegraded":"2023-10-16T11:01:18.240227+0000","last_fullsized":"2023-10-16T11:01:18.240227+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:26.022108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260338+0000","last_change":"2023-10-16T11:01:18.255264+0000","last_active":"2023-10-16T11:01:18.260338+0000","last_peered":"2023-10-16T11:01:18.260338+0000","last_clean":"2023-10-16T11:01:18.260338+0000","last_became_active":"2023-10-16T11:01:18.254989+0000","last_became_peered":"2023-10-16T11:01:18.254989+0000","last_unstale":"2023-10-16T11:01:18.260338+0000","last_undegraded":"2023-10-16T11:01:18.260338+0000","last_fullsized":"2023-10-16T11:01:18.260338+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:30.075067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.241108+0000","last_change":"2023-10-16T11:01:17.249899+0000","last_active":"2023-10-16T11:01:19.241108+0000","last_peered":"2023-10-16T11:01:19.241108+0000","last_clean":"2023-10-16T11:01:19.241108+0000","last_became_active":"2023-10-16T11:01:17.249628+0000","last_became_peered":"2023-10-16T11:01:17.249628+0000","last_unstale":"2023-10-16T11:01:19.241108+0000","last_undegraded":"2023-10-16T11:01:19.241108+0000","last_fullsized":"2023-10-16T11:01:19.241108+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:02.176992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801189+0000","last_change":"2023-10-16T11:01:18.266040+0000","last_active":"2023-10-16T11:01:19.801189+0000","last_peered":"2023-10-16T11:01:19.801189+0000","last_clean":"2023-10-16T11:01:19.801189+0000","last_became_active":"2023-10-16T11:01:18.265007+0000","last_became_peered":"2023-10-16T11:01:18.265007+0000","last_unstale":"2023-10-16T11:01:19.801189+0000","last_undegraded":"2023-10-16T11:01:19.801189+0000","last_fullsized":"2023-10-16T11:01:19.801189+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:27.917576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243618+0000","last_change":"2023-10-16T11:01:17.263877+0000","last_active":"2023-10-16T11:01:19.243618+0000","last_peered":"2023-10-16T11:01:19.243618+0000","last_clean":"2023-10-16T11:01:19.243618+0000","last_became_active":"2023-10-16T11:01:17.263621+0000","last_became_peered":"2023-10-16T11:01:17.263621+0000","last_unstale":"2023-10-16T11:01:19.243618+0000","last_undegraded":"2023-10-16T11:01:19.243618+0000","last_fullsized":"2023-10-16T11:01:19.243618+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:08.557266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258791+0000","last_change":"2023-10-16T11:01:18.257227+0000","last_active":"2023-10-16T11:01:18.258791+0000","last_peered":"2023-10-16T11:01:18.258791+0000","last_clean":"2023-10-16T11:01:18.258791+0000","last_became_active":"2023-10-16T11:01:18.256934+0000","last_became_peered":"2023-10-16T11:01:18.256934+0000","last_unstale":"2023-10-16T11:01:18.258791+0000","last_undegraded":"2023-10-16T11:01:18.258791+0000","last_fullsized":"2023-10-16T11:01:18.258791+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:18.809460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243712+0000","last_change":"2023-10-16T11:01:17.253342+0000","last_active":"2023-10-16T11:01:19.243712+0000","last_peered":"2023-10-16T11:01:19.243712+0000","last_clean":"2023-10-16T11:01:19.243712+0000","last_became_active":"2023-10-16T11:01:17.252830+0000","last_became_peered":"2023-10-16T11:01:17.252830+0000","last_unstale":"2023-10-16T11:01:19.243712+0000","last_undegraded":"2023-10-16T11:01:19.243712+0000","last_fullsized":"2023-10-16T11:01:19.243712+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:33.628241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801163+0000","last_change":"2023-10-16T11:01:18.267828+0000","last_active":"2023-10-16T11:01:19.801163+0000","last_peered":"2023-10-16T11:01:19.801163+0000","last_clean":"2023-10-16T11:01:19.801163+0000","last_became_active":"2023-10-16T11:01:18.266155+0000","last_became_peered":"2023-10-16T11:01:18.266155+0000","last_unstale":"2023-10-16T11:01:19.801163+0000","last_undegraded":"2023-10-16T11:01:19.801163+0000","last_fullsized":"2023-10-16T11:01:19.801163+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:58:05.681621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.247797+0000","last_change":"2023-10-16T11:01:17.263961+0000","last_active":"2023-10-16T11:01:18.247797+0000","last_peered":"2023-10-16T11:01:18.247797+0000","last_clean":"2023-10-16T11:01:18.247797+0000","last_became_active":"2023-10-16T11:01:17.263678+0000","last_became_peered":"2023-10-16T11:01:17.263678+0000","last_unstale":"2023-10-16T11:01:18.247797+0000","last_undegraded":"2023-10-16T11:01:18.247797+0000","last_fullsized":"2023-10-16T11:01:18.247797+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:09:21.097748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244141+0000","last_change":"2023-10-16T11:01:18.251283+0000","last_active":"2023-10-16T11:01:19.244141+0000","last_peered":"2023-10-16T11:01:19.244141+0000","last_clean":"2023-10-16T11:01:19.244141+0000","last_became_active":"2023-10-16T11:01:18.249191+0000","last_became_peered":"2023-10-16T11:01:18.249191+0000","last_unstale":"2023-10-16T11:01:19.244141+0000","last_undegraded":"2023-10-16T11:01:19.244141+0000","last_fullsized":"2023-10-16T11:01:19.244141+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:27.658071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243942+0000","last_change":"2023-10-16T11:01:17.251718+0000","last_active":"2023-10-16T11:01:19.243942+0000","last_peered":"2023-10-16T11:01:19.243942+0000","last_clean":"2023-10-16T11:01:19.243942+0000","last_became_active":"2023-10-16T11:01:17.251439+0000","last_became_peered":"2023-10-16T11:01:17.251439+0000","last_unstale":"2023-10-16T11:01:19.243942+0000","last_undegraded":"2023-10-16T11:01:19.243942+0000","last_fullsized":"2023-10-16T11:01:19.243942+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:44.291347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260101+0000","last_change":"2023-10-16T11:01:18.250562+0000","last_active":"2023-10-16T11:01:18.260101+0000","last_peered":"2023-10-16T11:01:18.260101+0000","last_clean":"2023-10-16T11:01:18.260101+0000","last_became_active":"2023-10-16T11:01:18.249661+0000","last_became_peered":"2023-10-16T11:01:18.249661+0000","last_unstale":"2023-10-16T11:01:18.260101+0000","last_undegraded":"2023-10-16T11:01:18.260101+0000","last_fullsized":"2023-10-16T11:01:18.260101+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:24.017251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243555+0000","last_change":"2023-10-16T11:01:17.251560+0000","last_active":"2023-10-16T11:01:19.243555+0000","last_peered":"2023-10-16T11:01:19.243555+0000","last_clean":"2023-10-16T11:01:19.243555+0000","last_became_active":"2023-10-16T11:01:17.251279+0000","last_became_peered":"2023-10-16T11:01:17.251279+0000","last_unstale":"2023-10-16T11:01:19.243555+0000","last_undegraded":"2023-10-16T11:01:19.243555+0000","last_fullsized":"2023-10-16T11:01:19.243555+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:57.146524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243464+0000","last_change":"2023-10-16T11:01:18.244960+0000","last_active":"2023-10-16T11:01:19.243464+0000","last_peered":"2023-10-16T11:01:19.243464+0000","last_clean":"2023-10-16T11:01:19.243464+0000","last_became_active":"2023-10-16T11:01:18.244715+0000","last_became_peered":"2023-10-16T11:01:18.244715+0000","last_unstale":"2023-10-16T11:01:19.243464+0000","last_undegraded":"2023-10-16T11:01:19.243464+0000","last_fullsized":"2023-10-16T11:01:19.243464+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:06.021662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.248055+0000","last_change":"2023-10-16T11:01:17.253322+0000","last_active":"2023-10-16T11:01:18.248055+0000","last_peered":"2023-10-16T11:01:18.248055+0000","last_clean":"2023-10-16T11:01:18.248055+0000","last_became_active":"2023-10-16T11:01:17.252991+0000","last_became_peered":"2023-10-16T11:01:17.252991+0000","last_unstale":"2023-10-16T11:01:18.248055+0000","last_undegraded":"2023-10-16T11:01:18.248055+0000","last_fullsized":"2023-10-16T11:01:18.248055+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:26.521597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257342+0000","last_change":"2023-10-16T11:01:18.248602+0000","last_active":"2023-10-16T11:01:18.257342+0000","last_peered":"2023-10-16T11:01:18.257342+0000","last_clean":"2023-10-16T11:01:18.257342+0000","last_became_active":"2023-10-16T11:01:18.248340+0000","last_became_peered":"2023-10-16T11:01:18.248340+0000","last_unstale":"2023-10-16T11:01:18.257342+0000","last_undegraded":"2023-10-16T11:01:18.257342+0000","last_fullsized":"2023-10-16T11:01:18.257342+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:32.330447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240342+0000","last_change":"2023-10-16T11:01:17.254006+0000","last_active":"2023-10-16T11:01:19.240342+0000","last_peered":"2023-10-16T11:01:19.240342+0000","last_clean":"2023-10-16T11:01:19.240342+0000","last_became_active":"2023-10-16T11:01:17.253685+0000","last_became_peered":"2023-10-16T11:01:17.253685+0000","last_unstale":"2023-10-16T11:01:19.240342+0000","last_undegraded":"2023-10-16T11:01:19.240342+0000","last_fullsized":"2023-10-16T11:01:19.240342+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:34.709625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260449+0000","last_change":"2023-10-16T11:01:18.257890+0000","last_active":"2023-10-16T11:01:18.260449+0000","last_peered":"2023-10-16T11:01:18.260449+0000","last_clean":"2023-10-16T11:01:18.260449+0000","last_became_active":"2023-10-16T11:01:18.257624+0000","last_became_peered":"2023-10-16T11:01:18.257624+0000","last_unstale":"2023-10-16T11:01:18.260449+0000","last_undegraded":"2023-10-16T11:01:18.260449+0000","last_fullsized":"2023-10-16T11:01:18.260449+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:16.339682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243552+0000","last_change":"2023-10-16T11:01:17.263866+0000","last_active":"2023-10-16T11:01:19.243552+0000","last_peered":"2023-10-16T11:01:19.243552+0000","last_clean":"2023-10-16T11:01:19.243552+0000","last_became_active":"2023-10-16T11:01:17.263576+0000","last_became_peered":"2023-10-16T11:01:17.263576+0000","last_unstale":"2023-10-16T11:01:19.243552+0000","last_undegraded":"2023-10-16T11:01:19.243552+0000","last_fullsized":"2023-10-16T11:01:19.243552+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:06.856271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"175'293","reported_seq":974,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T11:01:17.249905+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:01:17.249905+0000","last_peered":"2023-10-16T11:01:17.249905+0000","last_clean":"2023-10-16T11:01:17.249905+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:01:17.249905+0000","last_undegraded":"2023-10-16T11:01:17.249905+0000","last_fullsized":"2023-10-16T11:01:17.249905+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575776,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47904,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323936,"statfs":{"total":96636764160,"available":96587710464,"internally_reserved":0,"allocated":2899968,"data_stored":1813067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.83399999999999996,"15min":0.88900000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.45200000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":0.86199999999999999},"last":0.495}]},{"osd":1,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.61899999999999999,"15min":0.63600000000000001},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.80700000000000005,"15min":0.81999999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.313},"max":{"1min":0.71699999999999997,"5min":0.83299999999999996,"15min":0.85499999999999998},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.63400000000000001,"15min":0.65300000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38400000000000001},"max":{"1min":0.72799999999999998,"5min":0.80400000000000005,"15min":0.86399999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.32300000000000001},"max":{"1min":0.77800000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35499999999999998},"max":{"1min":0.71399999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.67100000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85899999999999999,"5min":0.879,"15min":0.90200000000000002},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.503,"15min":0.499},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53200000000000003,"15min":0.52300000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.79100000000000004,"15min":0.84799999999999998},"last":0.63}]},{"osd":5,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.54200000000000004,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38400000000000001,"15min":0.33200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.749},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83699999999999997,"5min":0.86399999999999999,"15min":0.92000000000000004},"last":0.73999999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.60299999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.96099999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60999999999999999,"15min":0.59599999999999997},"min":{"1min":0.44600000000000001,"5min":0.41199999999999998,"15min":0.36799999999999999},"max":{"1min":0.83899999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.67400000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575776,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37600,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334240,"statfs":{"total":96636764160,"available":96598261760,"internally_reserved":0,"allocated":2306048,"data_stored":1222699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60599999999999998,"15min":0.60099999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77000000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.621},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32400000000000001},"max":{"1min":0.78800000000000003,"5min":0.80100000000000005,"15min":0.82699999999999996},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.752,"5min":0.872,"15min":0.872},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76900000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.751,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.622,"15min":0.62},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81399999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63300000000000001,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.41899999999999998,"15min":0.41599999999999998},"max":{"1min":0.80000000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.36399999999999999},"max":{"1min":0.83199999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57799999999999996}]},{"osd":4,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.497},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.29999999999999999},"max":{"1min":0.79200000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.625},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.45100000000000001,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77200000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83499999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.69999999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84399999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.46600000000000003,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.81399999999999995,"5min":0.89900000000000002,"15min":0.96399999999999997},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575777,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59824,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312016,"statfs":{"total":96636764160,"available":96575504384,"internally_reserved":0,"allocated":2330624,"data_stored":1231383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41699999999999998,"15min":0.42899999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":0.46700000000000003,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.45300000000000001,"15min":0.46800000000000003},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.55200000000000005,"5min":0.64300000000000002,"15min":0.73599999999999999},"last":0.442}]},{"osd":2,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.51100000000000001,"5min":0.59499999999999997,"15min":0.66000000000000003},"last":0.317},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.51600000000000001,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56499999999999995,"5min":0.73099999999999998,"15min":0.748},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55300000000000005,"5min":0.69299999999999995,"15min":0.78300000000000003},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.68400000000000005,"5min":0.88200000000000001,"15min":0.89000000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84299999999999997,"5min":0.88300000000000001,"15min":0.91300000000000003},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.70699999999999996,"15min":0.71799999999999997},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.47199999999999998},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.66300000000000003,"15min":0.67200000000000004},"min":{"1min":0.48599999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.86799999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.90900000000000003,"5min":0.93799999999999994,"15min":0.96499999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67900000000000005,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.96899999999999997,"15min":0.98799999999999999},"last":0.502},{"interface":"front","average":{"1min":0.62,"5min":0.68300000000000005,"15min":0.69999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92400000000000004,"5min":0.97899999999999998,"15min":0.98999999999999999},"last":0.56799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575777,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42356,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329484,"statfs":{"total":96636764160,"available":96593391616,"internally_reserved":0,"allocated":2920448,"data_stored":1820243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.39400000000000002},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55000000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.67400000000000004,"15min":0.67600000000000005},"last":0.377}]},{"osd":2,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.434},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.94899999999999995},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49399999999999999,"15min":0.49299999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.044},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56999999999999995,"5min":0.64500000000000002,"15min":0.67000000000000004},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":0.67800000000000005,"15min":0.68100000000000005},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.36699999999999999},"max":{"1min":0.77100000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42399999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39400000000000002},"max":{"1min":0.79300000000000004,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.435,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81299999999999994,"5min":0.96999999999999997,"15min":0.98199999999999998},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.69199999999999995,"15min":0.69499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.98699999999999999,"15min":1.0209999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.69799999999999995,"15min":0.70199999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.96299999999999997},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.997,"15min":0.997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68100000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85099999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.46600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575776,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63968,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307872,"statfs":{"total":96636764160,"available":96571260928,"internally_reserved":0,"allocated":2306048,"data_stored":1222699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39500000000000002,"15min":0.39400000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58699999999999997,"5min":0.66500000000000004,"15min":0.97199999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46000000000000002,"15min":0.45400000000000001},"min":{"1min":0.30399999999999999,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":1.3180000000000001},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":1.161},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.435,"15min":0.42699999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.28000000000000003},"max":{"1min":0.64000000000000001,"5min":0.67600000000000005,"15min":0.69499999999999995},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.33800000000000002,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.431},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.503,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68999999999999995,"15min":0.68400000000000005},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.435},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67100000000000004,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.44900000000000001,"15min":0.40300000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.1299999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69699999999999995},"min":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85299999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70999999999999996,"15min":0.70899999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.432},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.70199999999999996,"15min":0.71099999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.48199999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.3879999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.70199999999999996,"15min":0.70099999999999996},"min":{"1min":0.50900000000000001,"5min":0.47699999999999998,"15min":0.46100000000000002},"max":{"1min":0.90700000000000003,"5min":0.93400000000000005,"15min":1.2190000000000001},"last":0.61499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47360,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324480,"statfs":{"total":96636764160,"available":96588267520,"internally_reserved":0,"allocated":2338816,"data_stored":1231987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.41699999999999998,"15min":0.41999999999999998},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.47699999999999998,"5min":0.61099999999999999,"15min":0.68300000000000005},"last":0.44}]},{"osd":1,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56699999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":0.54500000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52800000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57499999999999996,"5min":0.69199999999999995,"15min":0.75600000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.58699999999999997,"5min":0.70899999999999996,"15min":0.748},"last":0.57499999999999996}]},{"osd":4,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81299999999999994,"5min":0.97199999999999998,"15min":0.97699999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67600000000000005,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":0.98899999999999999,"15min":1.0629999999999999},"last":0.86499999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.72399999999999998,"5min":0.99299999999999999,"15min":1.0629999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.68899999999999995},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81399999999999995,"5min":1.054,"15min":1.054},"last":0.876}]},{"osd":6,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.68899999999999995},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.82199999999999995,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.45800000000000002},"max":{"1min":0.79700000000000004,"5min":0.99399999999999999,"15min":1.02},"last":0.78100000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.53000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.84299999999999997,"5min":1.012,"15min":1.012},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.70699999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79400000000000004,"5min":0.97899999999999998,"15min":1.002},"last":0.92100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575776,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45944,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325896,"statfs":{"total":96636764160,"available":96589717504,"internally_reserved":0,"allocated":2334720,"data_stored":1230535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.81200000000000006,"15min":0.83799999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.629},"min":{"1min":0.42399999999999999,"5min":0.41099999999999998,"15min":0.315},"max":{"1min":0.77000000000000002,"5min":0.80500000000000005,"15min":0.84899999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61899999999999999},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78500000000000003,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.66600000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.63100000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72599999999999998,"5min":0.85599999999999998,"15min":0.879},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.71299999999999997,"5min":0.83699999999999997,"15min":0.875},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.74099999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63700000000000001,"15min":0.63800000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77100000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.501},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73099999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.69399999999999995,"5min":0.83199999999999996,"15min":0.84499999999999997},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56599999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38900000000000001},"max":{"1min":0.68899999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33400000000000002},"max":{"1min":0.70899999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.66700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575776,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59816,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312024,"statfs":{"total":96636764160,"available":96575512576,"internally_reserved":0,"allocated":2314240,"data_stored":1222823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64800000000000002,"15min":0.65600000000000003},"min":{"1min":0.504,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.77100000000000002,"5min":0.85899999999999999,"15min":0.86499999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.46899999999999997,"15min":0.41599999999999998},"max":{"1min":0.78700000000000003,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59399999999999997,"15min":0.59799999999999998},"min":{"1min":0.36799999999999999,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":0.79600000000000004,"15min":0.82299999999999995},"last":0.432},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70799999999999996,"5min":0.82099999999999995,"15min":0.83299999999999996},"last":0.42299999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":0.88200000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.623,"15min":0.623},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.82799999999999996},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.47999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71199999999999997,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.45100000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.37,"5min":0.33500000000000002,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.78600000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.318,"15min":0.28000000000000003},"max":{"1min":0.69999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51200000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.40000000000000002,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":0.66800000000000004,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.375},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56100000000000005,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68300000000000005,"5min":0.82999999999999996,"15min":0.84099999999999997},"last":0.46800000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.57699999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72699999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63600000000000001,"15min":0.63200000000000001},"min":{"1min":0.46000000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":0.77300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:01:22.487 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:01:22.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.641+0000 7f0a07084700 1 Processor -- start 2023-10-16T11:01:22.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.641+0000 7f0a07084700 1 -- start start 2023-10-16T11:01:22.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f0a07084700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a000b6e10 0x7f0a001521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f0a07084700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0015c5d0 con 0x7f0a000cdee0 2023-10-16T11:01:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f0a07084700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0015c8b0 con 0x7f0a000ce2c0 2023-10-16T11:01:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f0a07084700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0015d2c0 con 0x7f0a000b6e10 2023-10-16T11:01:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f0a04e20700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0a000cdee0 0x7f0a0015bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34810/0 (socket says 172.21.15.158:34810) 2023-10-16T11:01:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f0a04e20700 1 -- 172.21.15.158:0/4282955177 learned_addr learned my addr 172.21.15.158:0/4282955177 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09fffff700 1 --2- 172.21.15.158:0/4282955177 >> v2:172.21.15.172:3300/0 conn(0x7f0a000b6e10 0x7f0a001521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919454090 0 0) 0x7f0a0015c5d0 con 0x7f0a000cdee0 2023-10-16T11:01:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f00035b0 con 0x7f0a000cdee0 2023-10-16T11:01:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1317938674 0 0) 0x7f09f00035b0 con 0x7f0a000cdee0 2023-10-16T11:01:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 >> v2:172.21.15.172:3300/0 conn(0x7f0a000b6e10 msgr2=0x7f0a001521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 --2- 172.21.15.158:0/4282955177 >> v2:172.21.15.172:3300/0 conn(0x7f0a000b6e10 0x7f0a001521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 >> v1:172.21.15.172:6789/0 conn(0x7f0a000ce2c0 legacy=0x7f0a000cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0015cb90 con 0x7f0a000cdee0 2023-10-16T11:01:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f09f4001670 con 0x7f0a000cdee0 2023-10-16T11:01:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 >> v1:172.21.15.158:6789/0 conn(0x7f0a000cdee0 legacy=0x7f0a0015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.642+0000 7f09ff7fe700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0003a00 0x7f09f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09f0004850 0x7f09f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f0005210 0x7f09f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0015d2c0 con 0x7f09f0004850 2023-10-16T11:01:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f000c1b0 con 0x7f09f0005210 2023-10-16T11:01:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f000c370 con 0x7f09f0003a00 2023-10-16T11:01:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09fffff700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09f0004850 0x7f09f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f0a04e20700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0003a00 0x7f09f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f0a05621700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f0005210 0x7f09f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09fffff700 1 -- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0003a00 msgr2=0x7f09f0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09fffff700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0003a00 0x7f09f0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09fffff700 1 -- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f0005210 msgr2=0x7f09f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09fffff700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f0005210 0x7f09f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09fffff700 1 -- 172.21.15.158:0/4282955177 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f00035b0 con 0x7f09f0004850 2023-10-16T11:01:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f0a05621700 1 -- 172.21.15.158:0/4282955177 reap_dead start 2023-10-16T11:01:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f0a04e20700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0003a00 0x7f09f0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09fffff700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09f0004850 0x7f09f0004cd0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009b60 tx=0x7f09f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c592d899414c4260 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:22.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00e6c0 con 0x7f09f0004850 2023-10-16T11:01:22.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec00e820 con 0x7f09f0004850 2023-10-16T11:01:22.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.643+0000 7f09ff7fe700 1 -- 172.21.15.158:0/4282955177 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00eb90 con 0x7f09f0004850 2023-10-16T11:01:22.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.644+0000 7f0a07084700 1 -- 172.21.15.158:0/4282955177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09f0004850 msgr2=0x7f09f0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.644+0000 7f0a07084700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09f0004850 0x7f09f0004cd0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009b60 tx=0x7f09f0005880 comp rx=0 tx=0).stop 2023-10-16T11:01:22.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.644+0000 7f0a07084700 1 -- 172.21.15.158:0/4282955177 shutdown_connections 2023-10-16T11:01:22.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.644+0000 7f0a07084700 1 --2- 172.21.15.158:0/4282955177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09f0004850 0x7f09f0004cd0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.644+0000 7f0a07084700 1 -- 172.21.15.158:0/4282955177 >> 172.21.15.158:0/4282955177 conn(0x7f0a0006a8d0 msgr2=0x7f0a0006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:22.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.644+0000 7f0a07084700 1 -- 172.21.15.158:0/4282955177 shutdown_connections 2023-10-16T11:01:22.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.644+0000 7f0a07084700 1 -- 172.21.15.158:0/4282955177 wait complete. 2023-10-16T11:01:22.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.645+0000 7f0a07084700 1 Processor -- start 2023-10-16T11:01:22.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.645+0000 7f0a07084700 1 -- start start 2023-10-16T11:01:22.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a07084700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0004850 0x7f0a0014f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a07084700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c7730 0x7f0a000c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a07084700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a001f1270 0x7f0a0015f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a07084700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0014ff00 con 0x7f0a001f1270 2023-10-16T11:01:22.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a07084700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a001f2210 con 0x7f0a000c7730 2023-10-16T11:01:22.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a07084700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a001f1c50 con 0x7f09f0004850 2023-10-16T11:01:22.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a05621700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a001f1270 0x7f0a0015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.646+0000 7f0a05621700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a001f1270 0x7f0a0015f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34650/0 (socket says 172.21.15.158:34650) 2023-10-16T11:01:22.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a05621700 1 -- 172.21.15.158:0/423386283 learned_addr learned my addr 172.21.15.158:0/423386283 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:22.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a04e20700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0004850 0x7f0a0014f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f09fffff700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c7730 0x7f0a000c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a05621700 1 -- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0004850 msgr2=0x7f0a0014f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a05621700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0004850 0x7f0a0014f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a05621700 1 -- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c7730 msgr2=0x7f0a000c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a05621700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c7730 0x7f0a000c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a05621700 1 -- 172.21.15.158:0/423386283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a001f1f30 con 0x7f0a001f1270 2023-10-16T11:01:22.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a05621700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a001f1270 0x7f0a0015f550 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f0a000715e0 tx=0x7f09f800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:22.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f09fd7fa700 1 -- 172.21.15.158:0/423386283 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8003c10 con 0x7f0a001f1270 2023-10-16T11:01:22.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f09fd7fa700 1 -- 172.21.15.158:0/423386283 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f8003d90 con 0x7f0a001f1270 2023-10-16T11:01:22.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f09fd7fa700 1 -- 172.21.15.158:0/423386283 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8005310 con 0x7f0a001f1270 2023-10-16T11:01:22.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a000c6080 con 0x7f0a001f1270 2023-10-16T11:01:22.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.647+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a000c6da0 con 0x7f0a001f1270 2023-10-16T11:01:22.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.648+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a000c66d0 con 0x7f0a001f1270 2023-10-16T11:01:22.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.649+0000 7f09fd7fa700 1 -- 172.21.15.158:0/423386283 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f09f8005470 con 0x7f0a001f1270 2023-10-16T11:01:22.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.649+0000 7f09fd7fa700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f09e406b020 0x7f09e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:22.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.649+0000 7f09fd7fa700 1 -- 172.21.15.158:0/423386283 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f09f8089910 con 0x7f0a001f1270 2023-10-16T11:01:22.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.649+0000 7f0a04e20700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f09e406b020 0x7f09e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:22.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.650+0000 7f0a04e20700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f09e406b020 0x7f09e406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09f4001c90 tx=0x7f09f4007040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:22.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.654+0000 7f09fd7fa700 1 -- 172.21.15.158:0/423386283 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09f8054630 con 0x7f0a001f1270 2023-10-16T11:01:22.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.843+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 --> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a0015cee0 con 0x7f09e406b020 2023-10-16T11:01:22.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.851+0000 7f09fd7fa700 1 -- 172.21.15.158:0/423386283 <== mgr.10212 v2:172.21.15.158:6832/1491011698 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290511 (secure 0 0 0) 0x7f09f40117c0 con 0x7f09e406b020 2023-10-16T11:01:22.855 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:22.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f09e406b020 msgr2=0x7f09e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f09e406b020 0x7f09e406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09f4001c90 tx=0x7f09f4007040 comp rx=0 tx=0).stop 2023-10-16T11:01:22.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a001f1270 msgr2=0x7f0a0015f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:22.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a001f1270 0x7f0a0015f550 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f0a000715e0 tx=0x7f09f800edd0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 shutdown_connections 2023-10-16T11:01:22.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f09e406b020 0x7f09e406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a001f1270 0x7f0a0015f550 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09f0004850 0x7f0a0014f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 --2- 172.21.15.158:0/423386283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c7730 0x7f0a000c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:22.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 >> 172.21.15.158:0/423386283 conn(0x7f0a0006a8d0 msgr2=0x7f0a0015b070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:22.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 shutdown_connections 2023-10-16T11:01:22.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:22.854+0000 7f0a07084700 1 -- 172.21.15.158:0/423386283 wait complete. 2023-10-16T11:01:22.860 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:01:22.868 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T11:01:22.194948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":404772,"kb_used_data":19288,"kb_used_omap":12,"kb_used_meta":385331,"kb_avail":754569948,"statfs":{"total":773094113280,"available":772679626752,"internally_reserved":0,"allocated":19750912,"data_stored":10995436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":394579511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.963985"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244373+0000","last_change":"2023-10-16T11:01:18.255169+0000","last_active":"2023-10-16T11:01:19.244373+0000","last_peered":"2023-10-16T11:01:19.244373+0000","last_clean":"2023-10-16T11:01:19.244373+0000","last_became_active":"2023-10-16T11:01:18.254846+0000","last_became_peered":"2023-10-16T11:01:18.254846+0000","last_unstale":"2023-10-16T11:01:19.244373+0000","last_undegraded":"2023-10-16T11:01:19.244373+0000","last_fullsized":"2023-10-16T11:01:19.244373+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:54.542505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.256868+0000","last_change":"2023-10-16T11:01:18.252580+0000","last_active":"2023-10-16T11:01:18.256868+0000","last_peered":"2023-10-16T11:01:18.256868+0000","last_clean":"2023-10-16T11:01:18.256868+0000","last_became_active":"2023-10-16T11:01:18.252289+0000","last_became_peered":"2023-10-16T11:01:18.252289+0000","last_unstale":"2023-10-16T11:01:18.256868+0000","last_undegraded":"2023-10-16T11:01:18.256868+0000","last_fullsized":"2023-10-16T11:01:18.256868+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:37.646535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244077+0000","last_change":"2023-10-16T11:01:18.255742+0000","last_active":"2023-10-16T11:01:19.244077+0000","last_peered":"2023-10-16T11:01:19.244077+0000","last_clean":"2023-10-16T11:01:19.244077+0000","last_became_active":"2023-10-16T11:01:18.255490+0000","last_became_peered":"2023-10-16T11:01:18.255490+0000","last_unstale":"2023-10-16T11:01:19.244077+0000","last_undegraded":"2023-10-16T11:01:19.244077+0000","last_fullsized":"2023-10-16T11:01:19.244077+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:08.755529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244254+0000","last_change":"2023-10-16T11:01:18.254231+0000","last_active":"2023-10-16T11:01:19.244254+0000","last_peered":"2023-10-16T11:01:19.244254+0000","last_clean":"2023-10-16T11:01:19.244254+0000","last_became_active":"2023-10-16T11:01:18.253964+0000","last_became_peered":"2023-10-16T11:01:18.253964+0000","last_unstale":"2023-10-16T11:01:19.244254+0000","last_undegraded":"2023-10-16T11:01:19.244254+0000","last_fullsized":"2023-10-16T11:01:19.244254+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:00.368989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240703+0000","last_change":"2023-10-16T11:01:18.253401+0000","last_active":"2023-10-16T11:01:19.240703+0000","last_peered":"2023-10-16T11:01:19.240703+0000","last_clean":"2023-10-16T11:01:19.240703+0000","last_became_active":"2023-10-16T11:01:18.253103+0000","last_became_peered":"2023-10-16T11:01:18.253103+0000","last_unstale":"2023-10-16T11:01:19.240703+0000","last_undegraded":"2023-10-16T11:01:19.240703+0000","last_fullsized":"2023-10-16T11:01:19.240703+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:27.026714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.246921+0000","last_change":"2023-10-16T11:01:18.244774+0000","last_active":"2023-10-16T11:01:18.246921+0000","last_peered":"2023-10-16T11:01:18.246921+0000","last_clean":"2023-10-16T11:01:18.246921+0000","last_became_active":"2023-10-16T11:01:18.244464+0000","last_became_peered":"2023-10-16T11:01:18.244464+0000","last_unstale":"2023-10-16T11:01:18.246921+0000","last_undegraded":"2023-10-16T11:01:18.246921+0000","last_fullsized":"2023-10-16T11:01:18.246921+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:52.905843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801078+0000","last_change":"2023-10-16T11:01:18.266009+0000","last_active":"2023-10-16T11:01:19.801078+0000","last_peered":"2023-10-16T11:01:19.801078+0000","last_clean":"2023-10-16T11:01:19.801078+0000","last_became_active":"2023-10-16T11:01:18.265719+0000","last_became_peered":"2023-10-16T11:01:18.265719+0000","last_unstale":"2023-10-16T11:01:19.801078+0000","last_undegraded":"2023-10-16T11:01:19.801078+0000","last_fullsized":"2023-10-16T11:01:19.801078+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:08:47.173788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.800999+0000","last_change":"2023-10-16T11:01:18.255229+0000","last_active":"2023-10-16T11:01:19.800999+0000","last_peered":"2023-10-16T11:01:19.800999+0000","last_clean":"2023-10-16T11:01:19.800999+0000","last_became_active":"2023-10-16T11:01:18.254951+0000","last_became_peered":"2023-10-16T11:01:18.254951+0000","last_unstale":"2023-10-16T11:01:19.800999+0000","last_undegraded":"2023-10-16T11:01:19.800999+0000","last_fullsized":"2023-10-16T11:01:19.800999+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:22.313327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801436+0000","last_change":"2023-10-16T11:01:18.266965+0000","last_active":"2023-10-16T11:01:19.801436+0000","last_peered":"2023-10-16T11:01:19.801436+0000","last_clean":"2023-10-16T11:01:19.801436+0000","last_became_active":"2023-10-16T11:01:18.265343+0000","last_became_peered":"2023-10-16T11:01:18.265343+0000","last_unstale":"2023-10-16T11:01:19.801436+0000","last_undegraded":"2023-10-16T11:01:19.801436+0000","last_fullsized":"2023-10-16T11:01:19.801436+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:40.734466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801474+0000","last_change":"2023-10-16T11:01:18.270104+0000","last_active":"2023-10-16T11:01:19.801474+0000","last_peered":"2023-10-16T11:01:19.801474+0000","last_clean":"2023-10-16T11:01:19.801474+0000","last_became_active":"2023-10-16T11:01:18.269462+0000","last_became_peered":"2023-10-16T11:01:18.269462+0000","last_unstale":"2023-10-16T11:01:19.801474+0000","last_undegraded":"2023-10-16T11:01:19.801474+0000","last_fullsized":"2023-10-16T11:01:19.801474+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:46.872260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258609+0000","last_change":"2023-10-16T11:01:18.253298+0000","last_active":"2023-10-16T11:01:18.258609+0000","last_peered":"2023-10-16T11:01:18.258609+0000","last_clean":"2023-10-16T11:01:18.258609+0000","last_became_active":"2023-10-16T11:01:18.253048+0000","last_became_peered":"2023-10-16T11:01:18.253048+0000","last_unstale":"2023-10-16T11:01:18.258609+0000","last_undegraded":"2023-10-16T11:01:18.258609+0000","last_fullsized":"2023-10-16T11:01:18.258609+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:12.531044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260294+0000","last_change":"2023-10-16T11:01:18.256459+0000","last_active":"2023-10-16T11:01:18.260294+0000","last_peered":"2023-10-16T11:01:18.260294+0000","last_clean":"2023-10-16T11:01:18.260294+0000","last_became_active":"2023-10-16T11:01:18.256165+0000","last_became_peered":"2023-10-16T11:01:18.256165+0000","last_unstale":"2023-10-16T11:01:18.260294+0000","last_undegraded":"2023-10-16T11:01:18.260294+0000","last_fullsized":"2023-10-16T11:01:18.260294+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:44:26.428306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243503+0000","last_change":"2023-10-16T11:01:18.246189+0000","last_active":"2023-10-16T11:01:19.243503+0000","last_peered":"2023-10-16T11:01:19.243503+0000","last_clean":"2023-10-16T11:01:19.243503+0000","last_became_active":"2023-10-16T11:01:18.245955+0000","last_became_peered":"2023-10-16T11:01:18.245955+0000","last_unstale":"2023-10-16T11:01:19.243503+0000","last_undegraded":"2023-10-16T11:01:19.243503+0000","last_fullsized":"2023-10-16T11:01:19.243503+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:19.358042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240569+0000","last_change":"2023-10-16T11:01:18.256243+0000","last_active":"2023-10-16T11:01:19.240569+0000","last_peered":"2023-10-16T11:01:19.240569+0000","last_clean":"2023-10-16T11:01:19.240569+0000","last_became_active":"2023-10-16T11:01:18.255971+0000","last_became_peered":"2023-10-16T11:01:18.255971+0000","last_unstale":"2023-10-16T11:01:19.240569+0000","last_undegraded":"2023-10-16T11:01:19.240569+0000","last_fullsized":"2023-10-16T11:01:19.240569+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:16.766240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260398+0000","last_change":"2023-10-16T11:01:18.256579+0000","last_active":"2023-10-16T11:01:18.260398+0000","last_peered":"2023-10-16T11:01:18.260398+0000","last_clean":"2023-10-16T11:01:18.260398+0000","last_became_active":"2023-10-16T11:01:18.255894+0000","last_became_peered":"2023-10-16T11:01:18.255894+0000","last_unstale":"2023-10-16T11:01:18.260398+0000","last_undegraded":"2023-10-16T11:01:18.260398+0000","last_fullsized":"2023-10-16T11:01:18.260398+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:27.026714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243749+0000","last_change":"2023-10-16T11:01:18.250473+0000","last_active":"2023-10-16T11:01:19.243749+0000","last_peered":"2023-10-16T11:01:19.243749+0000","last_clean":"2023-10-16T11:01:19.243749+0000","last_became_active":"2023-10-16T11:01:18.250014+0000","last_became_peered":"2023-10-16T11:01:18.250014+0000","last_unstale":"2023-10-16T11:01:19.243749+0000","last_undegraded":"2023-10-16T11:01:19.243749+0000","last_fullsized":"2023-10-16T11:01:19.243749+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:13.713337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801467+0000","last_change":"2023-10-16T11:01:18.265316+0000","last_active":"2023-10-16T11:01:19.801467+0000","last_peered":"2023-10-16T11:01:19.801467+0000","last_clean":"2023-10-16T11:01:19.801467+0000","last_became_active":"2023-10-16T11:01:18.264960+0000","last_became_peered":"2023-10-16T11:01:18.264960+0000","last_unstale":"2023-10-16T11:01:19.801467+0000","last_undegraded":"2023-10-16T11:01:19.801467+0000","last_fullsized":"2023-10-16T11:01:19.801467+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:18.324079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.272773+0000","last_change":"2023-10-16T11:01:18.269626+0000","last_active":"2023-10-16T11:01:18.272773+0000","last_peered":"2023-10-16T11:01:18.272773+0000","last_clean":"2023-10-16T11:01:18.272773+0000","last_became_active":"2023-10-16T11:01:18.269307+0000","last_became_peered":"2023-10-16T11:01:18.269307+0000","last_unstale":"2023-10-16T11:01:18.272773+0000","last_undegraded":"2023-10-16T11:01:18.272773+0000","last_fullsized":"2023-10-16T11:01:18.272773+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:14.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.268595+0000","last_change":"2023-10-16T11:01:18.265253+0000","last_active":"2023-10-16T11:01:18.268595+0000","last_peered":"2023-10-16T11:01:18.268595+0000","last_clean":"2023-10-16T11:01:18.268595+0000","last_became_active":"2023-10-16T11:01:18.264988+0000","last_became_peered":"2023-10-16T11:01:18.264988+0000","last_unstale":"2023-10-16T11:01:18.268595+0000","last_undegraded":"2023-10-16T11:01:18.268595+0000","last_fullsized":"2023-10-16T11:01:18.268595+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:46.822996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260931+0000","last_change":"2023-10-16T11:01:18.257678+0000","last_active":"2023-10-16T11:01:18.260931+0000","last_peered":"2023-10-16T11:01:18.260931+0000","last_clean":"2023-10-16T11:01:18.260931+0000","last_became_active":"2023-10-16T11:01:18.257478+0000","last_became_peered":"2023-10-16T11:01:18.257478+0000","last_unstale":"2023-10-16T11:01:18.260931+0000","last_undegraded":"2023-10-16T11:01:18.260931+0000","last_fullsized":"2023-10-16T11:01:18.260931+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:32.903365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243683+0000","last_change":"2023-10-16T11:01:18.246002+0000","last_active":"2023-10-16T11:01:19.243683+0000","last_peered":"2023-10-16T11:01:19.243683+0000","last_clean":"2023-10-16T11:01:19.243683+0000","last_became_active":"2023-10-16T11:01:18.245715+0000","last_became_peered":"2023-10-16T11:01:18.245715+0000","last_unstale":"2023-10-16T11:01:19.243683+0000","last_undegraded":"2023-10-16T11:01:19.243683+0000","last_fullsized":"2023-10-16T11:01:19.243683+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:58.914088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.256694+0000","last_change":"2023-10-16T11:01:18.250100+0000","last_active":"2023-10-16T11:01:18.256694+0000","last_peered":"2023-10-16T11:01:18.256694+0000","last_clean":"2023-10-16T11:01:18.256694+0000","last_became_active":"2023-10-16T11:01:18.249784+0000","last_became_peered":"2023-10-16T11:01:18.249784+0000","last_unstale":"2023-10-16T11:01:18.256694+0000","last_undegraded":"2023-10-16T11:01:18.256694+0000","last_fullsized":"2023-10-16T11:01:18.256694+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:29.401941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240450+0000","last_change":"2023-10-16T11:01:18.255427+0000","last_active":"2023-10-16T11:01:19.240450+0000","last_peered":"2023-10-16T11:01:19.240450+0000","last_clean":"2023-10-16T11:01:19.240450+0000","last_became_active":"2023-10-16T11:01:18.255159+0000","last_became_peered":"2023-10-16T11:01:18.255159+0000","last_unstale":"2023-10-16T11:01:19.240450+0000","last_undegraded":"2023-10-16T11:01:19.240450+0000","last_fullsized":"2023-10-16T11:01:19.240450+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:16.339682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243848+0000","last_change":"2023-10-16T11:01:18.250294+0000","last_active":"2023-10-16T11:01:19.243848+0000","last_peered":"2023-10-16T11:01:19.243848+0000","last_clean":"2023-10-16T11:01:19.243848+0000","last_became_active":"2023-10-16T11:01:18.249664+0000","last_became_peered":"2023-10-16T11:01:18.249664+0000","last_unstale":"2023-10-16T11:01:19.243848+0000","last_undegraded":"2023-10-16T11:01:19.243848+0000","last_fullsized":"2023-10-16T11:01:19.243848+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:00.049769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243747+0000","last_change":"2023-10-16T11:01:18.255135+0000","last_active":"2023-10-16T11:01:19.243747+0000","last_peered":"2023-10-16T11:01:19.243747+0000","last_clean":"2023-10-16T11:01:19.243747+0000","last_became_active":"2023-10-16T11:01:18.254875+0000","last_became_peered":"2023-10-16T11:01:18.254875+0000","last_unstale":"2023-10-16T11:01:19.243747+0000","last_undegraded":"2023-10-16T11:01:19.243747+0000","last_fullsized":"2023-10-16T11:01:19.243747+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:08.634340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257514+0000","last_change":"2023-10-16T11:01:18.254259+0000","last_active":"2023-10-16T11:01:18.257514+0000","last_peered":"2023-10-16T11:01:18.257514+0000","last_clean":"2023-10-16T11:01:18.257514+0000","last_became_active":"2023-10-16T11:01:18.254015+0000","last_became_peered":"2023-10-16T11:01:18.254015+0000","last_unstale":"2023-10-16T11:01:18.257514+0000","last_undegraded":"2023-10-16T11:01:18.257514+0000","last_fullsized":"2023-10-16T11:01:18.257514+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:20.651168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260140+0000","last_change":"2023-10-16T11:01:18.250740+0000","last_active":"2023-10-16T11:01:18.260140+0000","last_peered":"2023-10-16T11:01:18.260140+0000","last_clean":"2023-10-16T11:01:18.260140+0000","last_became_active":"2023-10-16T11:01:18.250280+0000","last_became_peered":"2023-10-16T11:01:18.250280+0000","last_unstale":"2023-10-16T11:01:18.260140+0000","last_undegraded":"2023-10-16T11:01:18.260140+0000","last_fullsized":"2023-10-16T11:01:18.260140+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:45:28.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243557+0000","last_change":"2023-10-16T11:01:18.259015+0000","last_active":"2023-10-16T11:01:19.243557+0000","last_peered":"2023-10-16T11:01:19.243557+0000","last_clean":"2023-10-16T11:01:19.243557+0000","last_became_active":"2023-10-16T11:01:18.258706+0000","last_became_peered":"2023-10-16T11:01:18.258706+0000","last_unstale":"2023-10-16T11:01:19.243557+0000","last_undegraded":"2023-10-16T11:01:19.243557+0000","last_fullsized":"2023-10-16T11:01:19.243557+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:32.959040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260367+0000","last_change":"2023-10-16T11:01:18.256478+0000","last_active":"2023-10-16T11:01:18.260367+0000","last_peered":"2023-10-16T11:01:18.260367+0000","last_clean":"2023-10-16T11:01:18.260367+0000","last_became_active":"2023-10-16T11:01:18.256189+0000","last_became_peered":"2023-10-16T11:01:18.256189+0000","last_unstale":"2023-10-16T11:01:18.260367+0000","last_undegraded":"2023-10-16T11:01:18.260367+0000","last_fullsized":"2023-10-16T11:01:18.260367+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:15.391349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257970+0000","last_change":"2023-10-16T11:01:18.254231+0000","last_active":"2023-10-16T11:01:18.257970+0000","last_peered":"2023-10-16T11:01:18.257970+0000","last_clean":"2023-10-16T11:01:18.257970+0000","last_became_active":"2023-10-16T11:01:18.253953+0000","last_became_peered":"2023-10-16T11:01:18.253953+0000","last_unstale":"2023-10-16T11:01:18.257970+0000","last_undegraded":"2023-10-16T11:01:18.257970+0000","last_fullsized":"2023-10-16T11:01:18.257970+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:54.822998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260377+0000","last_change":"2023-10-16T11:01:18.255668+0000","last_active":"2023-10-16T11:01:18.260377+0000","last_peered":"2023-10-16T11:01:18.260377+0000","last_clean":"2023-10-16T11:01:18.260377+0000","last_became_active":"2023-10-16T11:01:18.255360+0000","last_became_peered":"2023-10-16T11:01:18.255360+0000","last_unstale":"2023-10-16T11:01:18.260377+0000","last_undegraded":"2023-10-16T11:01:18.260377+0000","last_fullsized":"2023-10-16T11:01:18.260377+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:16.339682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260355+0000","last_change":"2023-10-16T11:01:18.258816+0000","last_active":"2023-10-16T11:01:18.260355+0000","last_peered":"2023-10-16T11:01:18.260355+0000","last_clean":"2023-10-16T11:01:18.260355+0000","last_became_active":"2023-10-16T11:01:18.258547+0000","last_became_peered":"2023-10-16T11:01:18.258547+0000","last_unstale":"2023-10-16T11:01:18.260355+0000","last_undegraded":"2023-10-16T11:01:18.260355+0000","last_fullsized":"2023-10-16T11:01:18.260355+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:30.425859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260057+0000","last_change":"2023-10-16T11:01:18.250795+0000","last_active":"2023-10-16T11:01:18.260057+0000","last_peered":"2023-10-16T11:01:18.260057+0000","last_clean":"2023-10-16T11:01:18.260057+0000","last_became_active":"2023-10-16T11:01:18.250514+0000","last_became_peered":"2023-10-16T11:01:18.250514+0000","last_unstale":"2023-10-16T11:01:18.260057+0000","last_undegraded":"2023-10-16T11:01:18.260057+0000","last_fullsized":"2023-10-16T11:01:18.260057+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:16.766240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"184'4","reported_seq":16,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244435+0000","last_change":"2023-10-16T11:01:17.249870+0000","last_active":"2023-10-16T11:01:19.244435+0000","last_peered":"2023-10-16T11:01:19.244435+0000","last_clean":"2023-10-16T11:01:19.244435+0000","last_became_active":"2023-10-16T11:01:17.249572+0000","last_became_peered":"2023-10-16T11:01:17.249572+0000","last_unstale":"2023-10-16T11:01:19.244435+0000","last_undegraded":"2023-10-16T11:01:19.244435+0000","last_fullsized":"2023-10-16T11:01:19.244435+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:15:35.200519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243336+0000","last_change":"2023-10-16T11:01:18.252603+0000","last_active":"2023-10-16T11:01:19.243336+0000","last_peered":"2023-10-16T11:01:19.243336+0000","last_clean":"2023-10-16T11:01:19.243336+0000","last_became_active":"2023-10-16T11:01:18.252190+0000","last_became_peered":"2023-10-16T11:01:18.252190+0000","last_unstale":"2023-10-16T11:01:19.243336+0000","last_undegraded":"2023-10-16T11:01:19.243336+0000","last_fullsized":"2023-10-16T11:01:19.243336+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:20.787643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.241057+0000","last_change":"2023-10-16T11:01:17.264091+0000","last_active":"2023-10-16T11:01:19.241057+0000","last_peered":"2023-10-16T11:01:19.241057+0000","last_clean":"2023-10-16T11:01:19.241057+0000","last_became_active":"2023-10-16T11:01:17.263800+0000","last_became_peered":"2023-10-16T11:01:17.263800+0000","last_unstale":"2023-10-16T11:01:19.241057+0000","last_undegraded":"2023-10-16T11:01:19.241057+0000","last_fullsized":"2023-10-16T11:01:19.241057+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:11.526438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240573+0000","last_change":"2023-10-16T11:01:18.261131+0000","last_active":"2023-10-16T11:01:19.240573+0000","last_peered":"2023-10-16T11:01:19.240573+0000","last_clean":"2023-10-16T11:01:19.240573+0000","last_became_active":"2023-10-16T11:01:18.260822+0000","last_became_peered":"2023-10-16T11:01:18.260822+0000","last_unstale":"2023-10-16T11:01:19.240573+0000","last_undegraded":"2023-10-16T11:01:19.240573+0000","last_fullsized":"2023-10-16T11:01:19.240573+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:15.391349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"184'2","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.273535+0000","last_change":"2023-10-16T11:01:17.248279+0000","last_active":"2023-10-16T11:01:18.273535+0000","last_peered":"2023-10-16T11:01:18.273535+0000","last_clean":"2023-10-16T11:01:18.273535+0000","last_became_active":"2023-10-16T11:01:17.247994+0000","last_became_peered":"2023-10-16T11:01:17.247994+0000","last_unstale":"2023-10-16T11:01:18.273535+0000","last_undegraded":"2023-10-16T11:01:18.273535+0000","last_fullsized":"2023-10-16T11:01:18.273535+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:55.782004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.256777+0000","last_change":"2023-10-16T11:01:18.251617+0000","last_active":"2023-10-16T11:01:18.256777+0000","last_peered":"2023-10-16T11:01:18.256777+0000","last_clean":"2023-10-16T11:01:18.256777+0000","last_became_active":"2023-10-16T11:01:18.251166+0000","last_became_peered":"2023-10-16T11:01:18.251166+0000","last_unstale":"2023-10-16T11:01:18.256777+0000","last_undegraded":"2023-10-16T11:01:18.256777+0000","last_fullsized":"2023-10-16T11:01:18.256777+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:56.786610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"184'2","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.268022+0000","last_change":"2023-10-16T11:01:17.251826+0000","last_active":"2023-10-16T11:01:18.268022+0000","last_peered":"2023-10-16T11:01:18.268022+0000","last_clean":"2023-10-16T11:01:18.268022+0000","last_became_active":"2023-10-16T11:01:17.251537+0000","last_became_peered":"2023-10-16T11:01:17.251537+0000","last_unstale":"2023-10-16T11:01:18.268022+0000","last_undegraded":"2023-10-16T11:01:18.268022+0000","last_fullsized":"2023-10-16T11:01:18.268022+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:09.418488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801203+0000","last_change":"2023-10-16T11:01:18.255277+0000","last_active":"2023-10-16T11:01:19.801203+0000","last_peered":"2023-10-16T11:01:19.801203+0000","last_clean":"2023-10-16T11:01:19.801203+0000","last_became_active":"2023-10-16T11:01:18.255011+0000","last_became_peered":"2023-10-16T11:01:18.255011+0000","last_unstale":"2023-10-16T11:01:19.801203+0000","last_undegraded":"2023-10-16T11:01:19.801203+0000","last_fullsized":"2023-10-16T11:01:19.801203+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:00:29.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801149+0000","last_change":"2023-10-16T11:01:17.256175+0000","last_active":"2023-10-16T11:01:19.801149+0000","last_peered":"2023-10-16T11:01:19.801149+0000","last_clean":"2023-10-16T11:01:19.801149+0000","last_became_active":"2023-10-16T11:01:17.255810+0000","last_became_peered":"2023-10-16T11:01:17.255810+0000","last_unstale":"2023-10-16T11:01:19.801149+0000","last_undegraded":"2023-10-16T11:01:19.801149+0000","last_fullsized":"2023-10-16T11:01:19.801149+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:53.042731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258638+0000","last_change":"2023-10-16T11:01:18.256094+0000","last_active":"2023-10-16T11:01:18.258638+0000","last_peered":"2023-10-16T11:01:18.258638+0000","last_clean":"2023-10-16T11:01:18.258638+0000","last_became_active":"2023-10-16T11:01:18.255820+0000","last_became_peered":"2023-10-16T11:01:18.255820+0000","last_unstale":"2023-10-16T11:01:18.258638+0000","last_undegraded":"2023-10-16T11:01:18.258638+0000","last_fullsized":"2023-10-16T11:01:18.258638+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:04.154686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240666+0000","last_change":"2023-10-16T11:01:17.249761+0000","last_active":"2023-10-16T11:01:19.240666+0000","last_peered":"2023-10-16T11:01:19.240666+0000","last_clean":"2023-10-16T11:01:19.240666+0000","last_became_active":"2023-10-16T11:01:17.249502+0000","last_became_peered":"2023-10-16T11:01:17.249502+0000","last_unstale":"2023-10-16T11:01:19.240666+0000","last_undegraded":"2023-10-16T11:01:19.240666+0000","last_fullsized":"2023-10-16T11:01:19.240666+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:11.693570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258584+0000","last_change":"2023-10-16T11:01:18.256079+0000","last_active":"2023-10-16T11:01:18.258584+0000","last_peered":"2023-10-16T11:01:18.258584+0000","last_clean":"2023-10-16T11:01:18.258584+0000","last_became_active":"2023-10-16T11:01:18.255800+0000","last_became_peered":"2023-10-16T11:01:18.255800+0000","last_unstale":"2023-10-16T11:01:18.258584+0000","last_undegraded":"2023-10-16T11:01:18.258584+0000","last_fullsized":"2023-10-16T11:01:18.258584+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:58.891749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.245445+0000","last_change":"2023-10-16T11:01:17.257413+0000","last_active":"2023-10-16T11:01:18.245445+0000","last_peered":"2023-10-16T11:01:18.245445+0000","last_clean":"2023-10-16T11:01:18.245445+0000","last_became_active":"2023-10-16T11:01:17.256961+0000","last_became_peered":"2023-10-16T11:01:17.256961+0000","last_unstale":"2023-10-16T11:01:18.245445+0000","last_undegraded":"2023-10-16T11:01:18.245445+0000","last_fullsized":"2023-10-16T11:01:18.245445+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.959786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244468+0000","last_change":"2023-10-16T11:01:18.259493+0000","last_active":"2023-10-16T11:01:19.244468+0000","last_peered":"2023-10-16T11:01:19.244468+0000","last_clean":"2023-10-16T11:01:19.244468+0000","last_became_active":"2023-10-16T11:01:18.259196+0000","last_became_peered":"2023-10-16T11:01:18.259196+0000","last_unstale":"2023-10-16T11:01:19.244468+0000","last_undegraded":"2023-10-16T11:01:19.244468+0000","last_fullsized":"2023-10-16T11:01:19.244468+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:58.071953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.800952+0000","last_change":"2023-10-16T11:01:17.255013+0000","last_active":"2023-10-16T11:01:19.800952+0000","last_peered":"2023-10-16T11:01:19.800952+0000","last_clean":"2023-10-16T11:01:19.800952+0000","last_became_active":"2023-10-16T11:01:17.254681+0000","last_became_peered":"2023-10-16T11:01:17.254681+0000","last_unstale":"2023-10-16T11:01:19.800952+0000","last_undegraded":"2023-10-16T11:01:19.800952+0000","last_fullsized":"2023-10-16T11:01:19.800952+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:52.238527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260034+0000","last_change":"2023-10-16T11:01:18.250323+0000","last_active":"2023-10-16T11:01:18.260034+0000","last_peered":"2023-10-16T11:01:18.260034+0000","last_clean":"2023-10-16T11:01:18.260034+0000","last_became_active":"2023-10-16T11:01:18.249985+0000","last_became_peered":"2023-10-16T11:01:18.249985+0000","last_unstale":"2023-10-16T11:01:18.260034+0000","last_undegraded":"2023-10-16T11:01:18.260034+0000","last_fullsized":"2023-10-16T11:01:18.260034+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:27.526202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243589+0000","last_change":"2023-10-16T11:01:17.256567+0000","last_active":"2023-10-16T11:01:19.243589+0000","last_peered":"2023-10-16T11:01:19.243589+0000","last_clean":"2023-10-16T11:01:19.243589+0000","last_became_active":"2023-10-16T11:01:17.256294+0000","last_became_peered":"2023-10-16T11:01:17.256294+0000","last_unstale":"2023-10-16T11:01:19.243589+0000","last_undegraded":"2023-10-16T11:01:19.243589+0000","last_fullsized":"2023-10-16T11:01:19.243589+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:01.846023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243751+0000","last_change":"2023-10-16T11:01:18.248345+0000","last_active":"2023-10-16T11:01:19.243751+0000","last_peered":"2023-10-16T11:01:19.243751+0000","last_clean":"2023-10-16T11:01:19.243751+0000","last_became_active":"2023-10-16T11:01:18.248078+0000","last_became_peered":"2023-10-16T11:01:18.248078+0000","last_unstale":"2023-10-16T11:01:19.243751+0000","last_undegraded":"2023-10-16T11:01:19.243751+0000","last_fullsized":"2023-10-16T11:01:19.243751+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:50:27.706495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"184'3","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244088+0000","last_change":"2023-10-16T11:01:17.249021+0000","last_active":"2023-10-16T11:01:19.244088+0000","last_peered":"2023-10-16T11:01:19.244088+0000","last_clean":"2023-10-16T11:01:19.244088+0000","last_became_active":"2023-10-16T11:01:17.248730+0000","last_became_peered":"2023-10-16T11:01:17.248730+0000","last_unstale":"2023-10-16T11:01:19.244088+0000","last_undegraded":"2023-10-16T11:01:19.244088+0000","last_fullsized":"2023-10-16T11:01:19.244088+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:39.271743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257801+0000","last_change":"2023-10-16T11:01:18.254241+0000","last_active":"2023-10-16T11:01:18.257801+0000","last_peered":"2023-10-16T11:01:18.257801+0000","last_clean":"2023-10-16T11:01:18.257801+0000","last_became_active":"2023-10-16T11:01:18.253963+0000","last_became_peered":"2023-10-16T11:01:18.253963+0000","last_unstale":"2023-10-16T11:01:18.257801+0000","last_undegraded":"2023-10-16T11:01:18.257801+0000","last_fullsized":"2023-10-16T11:01:18.257801+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:56.315152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"184'6","reported_seq":18,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243622+0000","last_change":"2023-10-16T11:01:17.263874+0000","last_active":"2023-10-16T11:01:19.243622+0000","last_peered":"2023-10-16T11:01:19.243622+0000","last_clean":"2023-10-16T11:01:19.243622+0000","last_became_active":"2023-10-16T11:01:17.263556+0000","last_became_peered":"2023-10-16T11:01:17.263556+0000","last_unstale":"2023-10-16T11:01:19.243622+0000","last_undegraded":"2023-10-16T11:01:19.243622+0000","last_fullsized":"2023-10-16T11:01:19.243622+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:09.448108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260316+0000","last_change":"2023-10-16T11:01:18.258109+0000","last_active":"2023-10-16T11:01:18.260316+0000","last_peered":"2023-10-16T11:01:18.260316+0000","last_clean":"2023-10-16T11:01:18.260316+0000","last_became_active":"2023-10-16T11:01:18.257792+0000","last_became_peered":"2023-10-16T11:01:18.257792+0000","last_unstale":"2023-10-16T11:01:18.260316+0000","last_undegraded":"2023-10-16T11:01:18.260316+0000","last_fullsized":"2023-10-16T11:01:18.260316+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:16.601581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240667+0000","last_change":"2023-10-16T11:01:17.254494+0000","last_active":"2023-10-16T11:01:19.240667+0000","last_peered":"2023-10-16T11:01:19.240667+0000","last_clean":"2023-10-16T11:01:19.240667+0000","last_became_active":"2023-10-16T11:01:17.254186+0000","last_became_peered":"2023-10-16T11:01:17.254186+0000","last_unstale":"2023-10-16T11:01:19.240667+0000","last_undegraded":"2023-10-16T11:01:19.240667+0000","last_fullsized":"2023-10-16T11:01:19.240667+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:10.252312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801301+0000","last_change":"2023-10-16T11:01:18.256547+0000","last_active":"2023-10-16T11:01:19.801301+0000","last_peered":"2023-10-16T11:01:19.801301+0000","last_clean":"2023-10-16T11:01:19.801301+0000","last_became_active":"2023-10-16T11:01:18.256177+0000","last_became_peered":"2023-10-16T11:01:18.256177+0000","last_unstale":"2023-10-16T11:01:19.801301+0000","last_undegraded":"2023-10-16T11:01:19.801301+0000","last_fullsized":"2023-10-16T11:01:19.801301+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:46.990440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243423+0000","last_change":"2023-10-16T11:01:17.249754+0000","last_active":"2023-10-16T11:01:19.243423+0000","last_peered":"2023-10-16T11:01:19.243423+0000","last_clean":"2023-10-16T11:01:19.243423+0000","last_became_active":"2023-10-16T11:01:17.249475+0000","last_became_peered":"2023-10-16T11:01:17.249475+0000","last_unstale":"2023-10-16T11:01:19.243423+0000","last_undegraded":"2023-10-16T11:01:19.243423+0000","last_fullsized":"2023-10-16T11:01:19.243423+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:01:20.247320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.261762+0000","last_change":"2023-10-16T11:01:18.258206+0000","last_active":"2023-10-16T11:01:18.261762+0000","last_peered":"2023-10-16T11:01:18.261762+0000","last_clean":"2023-10-16T11:01:18.261762+0000","last_became_active":"2023-10-16T11:01:18.257932+0000","last_became_peered":"2023-10-16T11:01:18.257932+0000","last_unstale":"2023-10-16T11:01:18.261762+0000","last_undegraded":"2023-10-16T11:01:18.261762+0000","last_fullsized":"2023-10-16T11:01:18.261762+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:07:16.766240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.247699+0000","last_change":"2023-10-16T11:01:17.254588+0000","last_active":"2023-10-16T11:01:18.247699+0000","last_peered":"2023-10-16T11:01:18.247699+0000","last_clean":"2023-10-16T11:01:18.247699+0000","last_became_active":"2023-10-16T11:01:17.254280+0000","last_became_peered":"2023-10-16T11:01:17.254280+0000","last_unstale":"2023-10-16T11:01:18.247699+0000","last_undegraded":"2023-10-16T11:01:18.247699+0000","last_fullsized":"2023-10-16T11:01:18.247699+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:28.397335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.262168+0000","last_change":"2023-10-16T11:01:18.257804+0000","last_active":"2023-10-16T11:01:18.262168+0000","last_peered":"2023-10-16T11:01:18.262168+0000","last_clean":"2023-10-16T11:01:18.262168+0000","last_became_active":"2023-10-16T11:01:18.257545+0000","last_became_peered":"2023-10-16T11:01:18.257545+0000","last_unstale":"2023-10-16T11:01:18.262168+0000","last_undegraded":"2023-10-16T11:01:18.262168+0000","last_fullsized":"2023-10-16T11:01:18.262168+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:16.332113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"184'8","reported_seq":20,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801110+0000","last_change":"2023-10-16T11:01:17.251687+0000","last_active":"2023-10-16T11:01:19.801110+0000","last_peered":"2023-10-16T11:01:19.801110+0000","last_clean":"2023-10-16T11:01:19.801110+0000","last_became_active":"2023-10-16T11:01:17.251419+0000","last_became_peered":"2023-10-16T11:01:17.251419+0000","last_unstale":"2023-10-16T11:01:19.801110+0000","last_undegraded":"2023-10-16T11:01:19.801110+0000","last_fullsized":"2023-10-16T11:01:19.801110+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:24.975550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801190+0000","last_change":"2023-10-16T11:01:18.272702+0000","last_active":"2023-10-16T11:01:19.801190+0000","last_peered":"2023-10-16T11:01:19.801190+0000","last_clean":"2023-10-16T11:01:19.801190+0000","last_became_active":"2023-10-16T11:01:18.272391+0000","last_became_peered":"2023-10-16T11:01:18.272391+0000","last_unstale":"2023-10-16T11:01:19.801190+0000","last_undegraded":"2023-10-16T11:01:19.801190+0000","last_fullsized":"2023-10-16T11:01:19.801190+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:29.866224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243679+0000","last_change":"2023-10-16T11:01:17.247703+0000","last_active":"2023-10-16T11:01:19.243679+0000","last_peered":"2023-10-16T11:01:19.243679+0000","last_clean":"2023-10-16T11:01:19.243679+0000","last_became_active":"2023-10-16T11:01:17.247411+0000","last_became_peered":"2023-10-16T11:01:17.247411+0000","last_unstale":"2023-10-16T11:01:19.243679+0000","last_undegraded":"2023-10-16T11:01:19.243679+0000","last_fullsized":"2023-10-16T11:01:19.243679+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:15.421754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257287+0000","last_change":"2023-10-16T11:01:18.252422+0000","last_active":"2023-10-16T11:01:18.257287+0000","last_peered":"2023-10-16T11:01:18.257287+0000","last_clean":"2023-10-16T11:01:18.257287+0000","last_became_active":"2023-10-16T11:01:18.252131+0000","last_became_peered":"2023-10-16T11:01:18.252131+0000","last_unstale":"2023-10-16T11:01:18.257287+0000","last_undegraded":"2023-10-16T11:01:18.257287+0000","last_fullsized":"2023-10-16T11:01:18.257287+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:07.220099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801380+0000","last_change":"2023-10-16T11:01:17.256863+0000","last_active":"2023-10-16T11:01:19.801380+0000","last_peered":"2023-10-16T11:01:19.801380+0000","last_clean":"2023-10-16T11:01:19.801380+0000","last_became_active":"2023-10-16T11:01:17.256609+0000","last_became_peered":"2023-10-16T11:01:17.256609+0000","last_unstale":"2023-10-16T11:01:19.801380+0000","last_undegraded":"2023-10-16T11:01:19.801380+0000","last_fullsized":"2023-10-16T11:01:19.801380+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:32.973887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.261812+0000","last_change":"2023-10-16T11:01:18.259143+0000","last_active":"2023-10-16T11:01:18.261812+0000","last_peered":"2023-10-16T11:01:18.261812+0000","last_clean":"2023-10-16T11:01:18.261812+0000","last_became_active":"2023-10-16T11:01:18.258882+0000","last_became_peered":"2023-10-16T11:01:18.258882+0000","last_unstale":"2023-10-16T11:01:18.261812+0000","last_undegraded":"2023-10-16T11:01:18.261812+0000","last_fullsized":"2023-10-16T11:01:18.261812+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:11.256918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.241082+0000","last_change":"2023-10-16T11:01:17.250819+0000","last_active":"2023-10-16T11:01:19.241082+0000","last_peered":"2023-10-16T11:01:19.241082+0000","last_clean":"2023-10-16T11:01:19.241082+0000","last_became_active":"2023-10-16T11:01:17.250532+0000","last_became_peered":"2023-10-16T11:01:17.250532+0000","last_unstale":"2023-10-16T11:01:19.241082+0000","last_undegraded":"2023-10-16T11:01:19.241082+0000","last_fullsized":"2023-10-16T11:01:19.241082+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:29.070461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260305+0000","last_change":"2023-10-16T11:01:18.255052+0000","last_active":"2023-10-16T11:01:18.260305+0000","last_peered":"2023-10-16T11:01:18.260305+0000","last_clean":"2023-10-16T11:01:18.260305+0000","last_became_active":"2023-10-16T11:01:18.254772+0000","last_became_peered":"2023-10-16T11:01:18.254772+0000","last_unstale":"2023-10-16T11:01:18.260305+0000","last_undegraded":"2023-10-16T11:01:18.260305+0000","last_fullsized":"2023-10-16T11:01:18.260305+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:05:15.391349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801383+0000","last_change":"2023-10-16T11:01:17.255507+0000","last_active":"2023-10-16T11:01:19.801383+0000","last_peered":"2023-10-16T11:01:19.801383+0000","last_clean":"2023-10-16T11:01:19.801383+0000","last_became_active":"2023-10-16T11:01:17.255280+0000","last_became_peered":"2023-10-16T11:01:17.255280+0000","last_unstale":"2023-10-16T11:01:19.801383+0000","last_undegraded":"2023-10-16T11:01:19.801383+0000","last_fullsized":"2023-10-16T11:01:19.801383+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:06:33.602849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243595+0000","last_change":"2023-10-16T11:01:18.250487+0000","last_active":"2023-10-16T11:01:19.243595+0000","last_peered":"2023-10-16T11:01:19.243595+0000","last_clean":"2023-10-16T11:01:19.243595+0000","last_became_active":"2023-10-16T11:01:18.250238+0000","last_became_peered":"2023-10-16T11:01:18.250238+0000","last_unstale":"2023-10-16T11:01:19.243595+0000","last_undegraded":"2023-10-16T11:01:19.243595+0000","last_fullsized":"2023-10-16T11:01:19.243595+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:16.317321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240727+0000","last_change":"2023-10-16T11:01:17.253306+0000","last_active":"2023-10-16T11:01:19.240727+0000","last_peered":"2023-10-16T11:01:19.240727+0000","last_clean":"2023-10-16T11:01:19.240727+0000","last_became_active":"2023-10-16T11:01:17.253049+0000","last_became_peered":"2023-10-16T11:01:17.253049+0000","last_unstale":"2023-10-16T11:01:19.240727+0000","last_undegraded":"2023-10-16T11:01:19.240727+0000","last_fullsized":"2023-10-16T11:01:19.240727+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:45.818390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240989+0000","last_change":"2023-10-16T11:01:18.266452+0000","last_active":"2023-10-16T11:01:19.240989+0000","last_peered":"2023-10-16T11:01:19.240989+0000","last_clean":"2023-10-16T11:01:19.240989+0000","last_became_active":"2023-10-16T11:01:18.266057+0000","last_became_peered":"2023-10-16T11:01:18.266057+0000","last_unstale":"2023-10-16T11:01:19.240989+0000","last_undegraded":"2023-10-16T11:01:19.240989+0000","last_fullsized":"2023-10-16T11:01:19.240989+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:27.526202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.246601+0000","last_change":"2023-10-16T11:01:17.250980+0000","last_active":"2023-10-16T11:01:18.246601+0000","last_peered":"2023-10-16T11:01:18.246601+0000","last_clean":"2023-10-16T11:01:18.246601+0000","last_became_active":"2023-10-16T11:01:17.250422+0000","last_became_peered":"2023-10-16T11:01:17.250422+0000","last_unstale":"2023-10-16T11:01:18.246601+0000","last_undegraded":"2023-10-16T11:01:18.246601+0000","last_fullsized":"2023-10-16T11:01:18.246601+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:08:25.935666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240975+0000","last_change":"2023-10-16T11:01:18.269869+0000","last_active":"2023-10-16T11:01:19.240975+0000","last_peered":"2023-10-16T11:01:19.240975+0000","last_clean":"2023-10-16T11:01:19.240975+0000","last_became_active":"2023-10-16T11:01:18.269590+0000","last_became_peered":"2023-10-16T11:01:18.269590+0000","last_unstale":"2023-10-16T11:01:19.240975+0000","last_undegraded":"2023-10-16T11:01:19.240975+0000","last_fullsized":"2023-10-16T11:01:19.240975+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:29.401941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.246840+0000","last_change":"2023-10-16T11:01:17.251070+0000","last_active":"2023-10-16T11:01:18.246840+0000","last_peered":"2023-10-16T11:01:18.246840+0000","last_clean":"2023-10-16T11:01:18.246840+0000","last_became_active":"2023-10-16T11:01:17.250797+0000","last_became_peered":"2023-10-16T11:01:17.250797+0000","last_unstale":"2023-10-16T11:01:18.246840+0000","last_undegraded":"2023-10-16T11:01:18.246840+0000","last_fullsized":"2023-10-16T11:01:18.246840+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:16.559989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260424+0000","last_change":"2023-10-16T11:01:18.259030+0000","last_active":"2023-10-16T11:01:18.260424+0000","last_peered":"2023-10-16T11:01:18.260424+0000","last_clean":"2023-10-16T11:01:18.260424+0000","last_became_active":"2023-10-16T11:01:18.258738+0000","last_became_peered":"2023-10-16T11:01:18.258738+0000","last_unstale":"2023-10-16T11:01:18.260424+0000","last_undegraded":"2023-10-16T11:01:18.260424+0000","last_fullsized":"2023-10-16T11:01:18.260424+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:50.323773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.240227+0000","last_change":"2023-10-16T11:01:17.253326+0000","last_active":"2023-10-16T11:01:18.240227+0000","last_peered":"2023-10-16T11:01:18.240227+0000","last_clean":"2023-10-16T11:01:18.240227+0000","last_became_active":"2023-10-16T11:01:17.253064+0000","last_became_peered":"2023-10-16T11:01:17.253064+0000","last_unstale":"2023-10-16T11:01:18.240227+0000","last_undegraded":"2023-10-16T11:01:18.240227+0000","last_fullsized":"2023-10-16T11:01:18.240227+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:26.022108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260338+0000","last_change":"2023-10-16T11:01:18.255264+0000","last_active":"2023-10-16T11:01:18.260338+0000","last_peered":"2023-10-16T11:01:18.260338+0000","last_clean":"2023-10-16T11:01:18.260338+0000","last_became_active":"2023-10-16T11:01:18.254989+0000","last_became_peered":"2023-10-16T11:01:18.254989+0000","last_unstale":"2023-10-16T11:01:18.260338+0000","last_undegraded":"2023-10-16T11:01:18.260338+0000","last_fullsized":"2023-10-16T11:01:18.260338+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:30.075067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.241108+0000","last_change":"2023-10-16T11:01:17.249899+0000","last_active":"2023-10-16T11:01:19.241108+0000","last_peered":"2023-10-16T11:01:19.241108+0000","last_clean":"2023-10-16T11:01:19.241108+0000","last_became_active":"2023-10-16T11:01:17.249628+0000","last_became_peered":"2023-10-16T11:01:17.249628+0000","last_unstale":"2023-10-16T11:01:19.241108+0000","last_undegraded":"2023-10-16T11:01:19.241108+0000","last_fullsized":"2023-10-16T11:01:19.241108+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:02.176992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801189+0000","last_change":"2023-10-16T11:01:18.266040+0000","last_active":"2023-10-16T11:01:19.801189+0000","last_peered":"2023-10-16T11:01:19.801189+0000","last_clean":"2023-10-16T11:01:19.801189+0000","last_became_active":"2023-10-16T11:01:18.265007+0000","last_became_peered":"2023-10-16T11:01:18.265007+0000","last_unstale":"2023-10-16T11:01:19.801189+0000","last_undegraded":"2023-10-16T11:01:19.801189+0000","last_fullsized":"2023-10-16T11:01:19.801189+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:27.917576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243618+0000","last_change":"2023-10-16T11:01:17.263877+0000","last_active":"2023-10-16T11:01:19.243618+0000","last_peered":"2023-10-16T11:01:19.243618+0000","last_clean":"2023-10-16T11:01:19.243618+0000","last_became_active":"2023-10-16T11:01:17.263621+0000","last_became_peered":"2023-10-16T11:01:17.263621+0000","last_unstale":"2023-10-16T11:01:19.243618+0000","last_undegraded":"2023-10-16T11:01:19.243618+0000","last_fullsized":"2023-10-16T11:01:19.243618+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:08.557266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.258791+0000","last_change":"2023-10-16T11:01:18.257227+0000","last_active":"2023-10-16T11:01:18.258791+0000","last_peered":"2023-10-16T11:01:18.258791+0000","last_clean":"2023-10-16T11:01:18.258791+0000","last_became_active":"2023-10-16T11:01:18.256934+0000","last_became_peered":"2023-10-16T11:01:18.256934+0000","last_unstale":"2023-10-16T11:01:18.258791+0000","last_undegraded":"2023-10-16T11:01:18.258791+0000","last_fullsized":"2023-10-16T11:01:18.258791+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:18.809460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243712+0000","last_change":"2023-10-16T11:01:17.253342+0000","last_active":"2023-10-16T11:01:19.243712+0000","last_peered":"2023-10-16T11:01:19.243712+0000","last_clean":"2023-10-16T11:01:19.243712+0000","last_became_active":"2023-10-16T11:01:17.252830+0000","last_became_peered":"2023-10-16T11:01:17.252830+0000","last_unstale":"2023-10-16T11:01:19.243712+0000","last_undegraded":"2023-10-16T11:01:19.243712+0000","last_fullsized":"2023-10-16T11:01:19.243712+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:33.628241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.801163+0000","last_change":"2023-10-16T11:01:18.267828+0000","last_active":"2023-10-16T11:01:19.801163+0000","last_peered":"2023-10-16T11:01:19.801163+0000","last_clean":"2023-10-16T11:01:19.801163+0000","last_became_active":"2023-10-16T11:01:18.266155+0000","last_became_peered":"2023-10-16T11:01:18.266155+0000","last_unstale":"2023-10-16T11:01:19.801163+0000","last_undegraded":"2023-10-16T11:01:19.801163+0000","last_fullsized":"2023-10-16T11:01:19.801163+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:58:05.681621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.247797+0000","last_change":"2023-10-16T11:01:17.263961+0000","last_active":"2023-10-16T11:01:18.247797+0000","last_peered":"2023-10-16T11:01:18.247797+0000","last_clean":"2023-10-16T11:01:18.247797+0000","last_became_active":"2023-10-16T11:01:17.263678+0000","last_became_peered":"2023-10-16T11:01:17.263678+0000","last_unstale":"2023-10-16T11:01:18.247797+0000","last_undegraded":"2023-10-16T11:01:18.247797+0000","last_fullsized":"2023-10-16T11:01:18.247797+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:09:21.097748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.244141+0000","last_change":"2023-10-16T11:01:18.251283+0000","last_active":"2023-10-16T11:01:19.244141+0000","last_peered":"2023-10-16T11:01:19.244141+0000","last_clean":"2023-10-16T11:01:19.244141+0000","last_became_active":"2023-10-16T11:01:18.249191+0000","last_became_peered":"2023-10-16T11:01:18.249191+0000","last_unstale":"2023-10-16T11:01:19.244141+0000","last_undegraded":"2023-10-16T11:01:19.244141+0000","last_fullsized":"2023-10-16T11:01:19.244141+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:27.658071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243942+0000","last_change":"2023-10-16T11:01:17.251718+0000","last_active":"2023-10-16T11:01:19.243942+0000","last_peered":"2023-10-16T11:01:19.243942+0000","last_clean":"2023-10-16T11:01:19.243942+0000","last_became_active":"2023-10-16T11:01:17.251439+0000","last_became_peered":"2023-10-16T11:01:17.251439+0000","last_unstale":"2023-10-16T11:01:19.243942+0000","last_undegraded":"2023-10-16T11:01:19.243942+0000","last_fullsized":"2023-10-16T11:01:19.243942+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:44.291347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260101+0000","last_change":"2023-10-16T11:01:18.250562+0000","last_active":"2023-10-16T11:01:18.260101+0000","last_peered":"2023-10-16T11:01:18.260101+0000","last_clean":"2023-10-16T11:01:18.260101+0000","last_became_active":"2023-10-16T11:01:18.249661+0000","last_became_peered":"2023-10-16T11:01:18.249661+0000","last_unstale":"2023-10-16T11:01:18.260101+0000","last_undegraded":"2023-10-16T11:01:18.260101+0000","last_fullsized":"2023-10-16T11:01:18.260101+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:24.017251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243555+0000","last_change":"2023-10-16T11:01:17.251560+0000","last_active":"2023-10-16T11:01:19.243555+0000","last_peered":"2023-10-16T11:01:19.243555+0000","last_clean":"2023-10-16T11:01:19.243555+0000","last_became_active":"2023-10-16T11:01:17.251279+0000","last_became_peered":"2023-10-16T11:01:17.251279+0000","last_unstale":"2023-10-16T11:01:19.243555+0000","last_undegraded":"2023-10-16T11:01:19.243555+0000","last_fullsized":"2023-10-16T11:01:19.243555+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:57.146524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243464+0000","last_change":"2023-10-16T11:01:18.244960+0000","last_active":"2023-10-16T11:01:19.243464+0000","last_peered":"2023-10-16T11:01:19.243464+0000","last_clean":"2023-10-16T11:01:19.243464+0000","last_became_active":"2023-10-16T11:01:18.244715+0000","last_became_peered":"2023-10-16T11:01:18.244715+0000","last_unstale":"2023-10-16T11:01:19.243464+0000","last_undegraded":"2023-10-16T11:01:19.243464+0000","last_fullsized":"2023-10-16T11:01:19.243464+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:06.021662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.248055+0000","last_change":"2023-10-16T11:01:17.253322+0000","last_active":"2023-10-16T11:01:18.248055+0000","last_peered":"2023-10-16T11:01:18.248055+0000","last_clean":"2023-10-16T11:01:18.248055+0000","last_became_active":"2023-10-16T11:01:17.252991+0000","last_became_peered":"2023-10-16T11:01:17.252991+0000","last_unstale":"2023-10-16T11:01:18.248055+0000","last_undegraded":"2023-10-16T11:01:18.248055+0000","last_fullsized":"2023-10-16T11:01:18.248055+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:26.521597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.257342+0000","last_change":"2023-10-16T11:01:18.248602+0000","last_active":"2023-10-16T11:01:18.257342+0000","last_peered":"2023-10-16T11:01:18.257342+0000","last_clean":"2023-10-16T11:01:18.257342+0000","last_became_active":"2023-10-16T11:01:18.248340+0000","last_became_peered":"2023-10-16T11:01:18.248340+0000","last_unstale":"2023-10-16T11:01:18.257342+0000","last_undegraded":"2023-10-16T11:01:18.257342+0000","last_fullsized":"2023-10-16T11:01:18.257342+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:32.330447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.240342+0000","last_change":"2023-10-16T11:01:17.254006+0000","last_active":"2023-10-16T11:01:19.240342+0000","last_peered":"2023-10-16T11:01:19.240342+0000","last_clean":"2023-10-16T11:01:19.240342+0000","last_became_active":"2023-10-16T11:01:17.253685+0000","last_became_peered":"2023-10-16T11:01:17.253685+0000","last_unstale":"2023-10-16T11:01:19.240342+0000","last_undegraded":"2023-10-16T11:01:19.240342+0000","last_fullsized":"2023-10-16T11:01:19.240342+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:34.709625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-10-16T11:01:18.260449+0000","last_change":"2023-10-16T11:01:18.257890+0000","last_active":"2023-10-16T11:01:18.260449+0000","last_peered":"2023-10-16T11:01:18.260449+0000","last_clean":"2023-10-16T11:01:18.260449+0000","last_became_active":"2023-10-16T11:01:18.257624+0000","last_became_peered":"2023-10-16T11:01:18.257624+0000","last_unstale":"2023-10-16T11:01:18.260449+0000","last_undegraded":"2023-10-16T11:01:18.260449+0000","last_fullsized":"2023-10-16T11:01:18.260449+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:17.230544+0000","last_clean_scrub_stamp":"2023-10-16T11:01:17.230544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:16.339682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-10-16T11:01:19.243552+0000","last_change":"2023-10-16T11:01:17.263866+0000","last_active":"2023-10-16T11:01:19.243552+0000","last_peered":"2023-10-16T11:01:19.243552+0000","last_clean":"2023-10-16T11:01:19.243552+0000","last_became_active":"2023-10-16T11:01:17.263576+0000","last_became_peered":"2023-10-16T11:01:17.263576+0000","last_unstale":"2023-10-16T11:01:19.243552+0000","last_undegraded":"2023-10-16T11:01:19.243552+0000","last_fullsized":"2023-10-16T11:01:19.243552+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:01:16.225938+0000","last_clean_scrub_stamp":"2023-10-16T11:01:16.225938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:06.856271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"175'293","reported_seq":974,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T11:01:17.249905+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:01:17.249905+0000","last_peered":"2023-10-16T11:01:17.249905+0000","last_clean":"2023-10-16T11:01:17.249905+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:01:17.249905+0000","last_undegraded":"2023-10-16T11:01:17.249905+0000","last_fullsized":"2023-10-16T11:01:17.249905+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575776,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47904,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323936,"statfs":{"total":96636764160,"available":96587710464,"internally_reserved":0,"allocated":2899968,"data_stored":1813067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.83399999999999996,"15min":0.88900000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.45200000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":0.86199999999999999},"last":0.495}]},{"osd":1,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.61899999999999999,"15min":0.63600000000000001},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.80700000000000005,"15min":0.81999999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.313},"max":{"1min":0.71699999999999997,"5min":0.83299999999999996,"15min":0.85499999999999998},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.63400000000000001,"15min":0.65300000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38400000000000001},"max":{"1min":0.72799999999999998,"5min":0.80400000000000005,"15min":0.86399999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.32300000000000001},"max":{"1min":0.77800000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35499999999999998},"max":{"1min":0.71399999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.67100000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85899999999999999,"5min":0.879,"15min":0.90200000000000002},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.503,"15min":0.499},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53200000000000003,"15min":0.52300000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.79100000000000004,"15min":0.84799999999999998},"last":0.63}]},{"osd":5,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.54200000000000004,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38400000000000001,"15min":0.33200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.749},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83699999999999997,"5min":0.86399999999999999,"15min":0.92000000000000004},"last":0.73999999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.60299999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.96099999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60999999999999999,"15min":0.59599999999999997},"min":{"1min":0.44600000000000001,"5min":0.41199999999999998,"15min":0.36799999999999999},"max":{"1min":0.83899999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.67400000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575776,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37600,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334240,"statfs":{"total":96636764160,"available":96598261760,"internally_reserved":0,"allocated":2306048,"data_stored":1222699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60599999999999998,"15min":0.60099999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77000000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.621},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32400000000000001},"max":{"1min":0.78800000000000003,"5min":0.80100000000000005,"15min":0.82699999999999996},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.752,"5min":0.872,"15min":0.872},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76900000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.751,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.622,"15min":0.62},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81399999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63300000000000001,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.41899999999999998,"15min":0.41599999999999998},"max":{"1min":0.80000000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.36399999999999999},"max":{"1min":0.83199999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57799999999999996}]},{"osd":4,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.497},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.29999999999999999},"max":{"1min":0.79200000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.625},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.45100000000000001,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77200000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83499999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.69999999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84399999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.46600000000000003,"5min":0.39000000000000001,"15min":0.34899999999999998},"max":{"1min":0.81399999999999995,"5min":0.89900000000000002,"15min":0.96399999999999997},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575777,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59824,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312016,"statfs":{"total":96636764160,"available":96575504384,"internally_reserved":0,"allocated":2330624,"data_stored":1231383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41699999999999998,"15min":0.42899999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":0.46700000000000003,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.45300000000000001,"15min":0.46800000000000003},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.55200000000000005,"5min":0.64300000000000002,"15min":0.73599999999999999},"last":0.442}]},{"osd":2,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.51100000000000001,"5min":0.59499999999999997,"15min":0.66000000000000003},"last":0.317},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.51600000000000001,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56499999999999995,"5min":0.73099999999999998,"15min":0.748},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55300000000000005,"5min":0.69299999999999995,"15min":0.78300000000000003},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.68400000000000005,"5min":0.88200000000000001,"15min":0.89000000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84299999999999997,"5min":0.88300000000000001,"15min":0.91300000000000003},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.70699999999999996,"15min":0.71799999999999997},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.47199999999999998},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.66300000000000003,"15min":0.67200000000000004},"min":{"1min":0.48599999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.86799999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.90900000000000003,"5min":0.93799999999999994,"15min":0.96499999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:00:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67900000000000005,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.96899999999999997,"15min":0.98799999999999999},"last":0.502},{"interface":"front","average":{"1min":0.62,"5min":0.68300000000000005,"15min":0.69999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92400000000000004,"5min":0.97899999999999998,"15min":0.98999999999999999},"last":0.56799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575777,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42356,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329484,"statfs":{"total":96636764160,"available":96593391616,"internally_reserved":0,"allocated":2920448,"data_stored":1820243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.39400000000000002},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55000000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.67400000000000004,"15min":0.67600000000000005},"last":0.377}]},{"osd":2,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.434},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.94899999999999995},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49399999999999999,"15min":0.49299999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.044},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56999999999999995,"5min":0.64500000000000002,"15min":0.67000000000000004},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":0.67800000000000005,"15min":0.68100000000000005},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.36699999999999999},"max":{"1min":0.77100000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42399999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39400000000000002},"max":{"1min":0.79300000000000004,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.435,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81299999999999994,"5min":0.96999999999999997,"15min":0.98199999999999998},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.69199999999999995,"15min":0.69499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.98699999999999999,"15min":1.0209999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.69799999999999995,"15min":0.70199999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.96299999999999997},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.997,"15min":0.997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68100000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85099999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.46600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575776,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63968,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307872,"statfs":{"total":96636764160,"available":96571260928,"internally_reserved":0,"allocated":2306048,"data_stored":1222699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39500000000000002,"15min":0.39400000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58699999999999997,"5min":0.66500000000000004,"15min":0.97199999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46000000000000002,"15min":0.45400000000000001},"min":{"1min":0.30399999999999999,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":1.3180000000000001},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":1.161},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.435,"15min":0.42699999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.28000000000000003},"max":{"1min":0.64000000000000001,"5min":0.67600000000000005,"15min":0.69499999999999995},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.33800000000000002,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.431},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.503,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68999999999999995,"15min":0.68400000000000005},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.435},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67100000000000004,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.44900000000000001,"15min":0.40300000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.1299999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69699999999999995},"min":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85299999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70999999999999996,"15min":0.70899999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.432},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.70199999999999996,"15min":0.71099999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.48199999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.3879999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.70199999999999996,"15min":0.70099999999999996},"min":{"1min":0.50900000000000001,"5min":0.47699999999999998,"15min":0.46100000000000002},"max":{"1min":0.90700000000000003,"5min":0.93400000000000005,"15min":1.2190000000000001},"last":0.61499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47360,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324480,"statfs":{"total":96636764160,"available":96588267520,"internally_reserved":0,"allocated":2338816,"data_stored":1231987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.41699999999999998,"15min":0.41999999999999998},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.47699999999999998,"5min":0.61099999999999999,"15min":0.68300000000000005},"last":0.44}]},{"osd":1,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56699999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":0.54500000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52800000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57499999999999996,"5min":0.69199999999999995,"15min":0.75600000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.58699999999999997,"5min":0.70899999999999996,"15min":0.748},"last":0.57499999999999996}]},{"osd":4,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81299999999999994,"5min":0.97199999999999998,"15min":0.97699999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67600000000000005,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":0.98899999999999999,"15min":1.0629999999999999},"last":0.86499999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.72399999999999998,"5min":0.99299999999999999,"15min":1.0629999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.68899999999999995},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81399999999999995,"5min":1.054,"15min":1.054},"last":0.876}]},{"osd":6,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.68899999999999995},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.82199999999999995,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.45800000000000002},"max":{"1min":0.79700000000000004,"5min":0.99399999999999999,"15min":1.02},"last":0.78100000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.53000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.84299999999999997,"5min":1.012,"15min":1.012},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.70699999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79400000000000004,"5min":0.97899999999999998,"15min":1.002},"last":0.92100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575776,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45944,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325896,"statfs":{"total":96636764160,"available":96589717504,"internally_reserved":0,"allocated":2334720,"data_stored":1230535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":0.81200000000000006,"15min":0.83799999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.629},"min":{"1min":0.42399999999999999,"5min":0.41099999999999998,"15min":0.315},"max":{"1min":0.77000000000000002,"5min":0.80500000000000005,"15min":0.84899999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61899999999999999},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78500000000000003,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.66600000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.63100000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72599999999999998,"5min":0.85599999999999998,"15min":0.879},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.71299999999999997,"5min":0.83699999999999997,"15min":0.875},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.74099999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63700000000000001,"15min":0.63800000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77100000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.501},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73099999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.69399999999999995,"5min":0.83199999999999996,"15min":0.84499999999999997},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56599999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38900000000000001},"max":{"1min":0.68899999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33400000000000002},"max":{"1min":0.70899999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.66700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575776,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59816,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312024,"statfs":{"total":96636764160,"available":96575512576,"internally_reserved":0,"allocated":2314240,"data_stored":1222823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64800000000000002,"15min":0.65600000000000003},"min":{"1min":0.504,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.77100000000000002,"5min":0.85899999999999999,"15min":0.86499999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.46899999999999997,"15min":0.41599999999999998},"max":{"1min":0.78700000000000003,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59399999999999997,"15min":0.59799999999999998},"min":{"1min":0.36799999999999999,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":0.79600000000000004,"15min":0.82299999999999995},"last":0.432},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70799999999999996,"5min":0.82099999999999995,"15min":0.83299999999999996},"last":0.42299999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":0.88200000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.623,"15min":0.623},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.82799999999999996},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.35599999999999998},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.47999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71199999999999997,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.45100000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.37,"5min":0.33500000000000002,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.78600000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.318,"15min":0.28000000000000003},"max":{"1min":0.69999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51200000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.40000000000000002,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":0.66800000000000004,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.375},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56100000000000005,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68300000000000005,"5min":0.82999999999999996,"15min":0.84099999999999997},"last":0.46800000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.57699999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72699999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63600000000000001,"15min":0.63200000000000001},"min":{"1min":0.46000000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":0.77300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:01:22.874 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:01:22.874 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T11:01:23.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.034+0000 7f30672e1700 1 Processor -- start 2023-10-16T11:01:23.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.034+0000 7f30672e1700 1 -- start start 2023-10-16T11:01:23.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f30672e1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f306015b5b0 0x7f306015f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f30672e1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30600cddd0 0x7f30600ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f30672e1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306015f450 con 0x7f30600d0b60 2023-10-16T11:01:23.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f30672e1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30600ce7a0 con 0x7f306015b5b0 2023-10-16T11:01:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f30672e1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306015c9a0 con 0x7f30600cddd0 2023-10-16T11:01:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306507d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f306015b5b0 0x7f306015f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306487c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f30600d0b60 0x7f306015b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34830/0 (socket says 172.21.15.158:34830) 2023-10-16T11:01:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306487c700 1 -- 172.21.15.158:0/3447410014 learned_addr learned my addr 172.21.15.158:0/3447410014 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306587e700 1 --2- 172.21.15.158:0/3447410014 >> v2:172.21.15.172:3300/0 conn(0x7f30600cddd0 0x7f30600ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306507d700 1 -- 172.21.15.158:0/3447410014 >> v2:172.21.15.172:3300/0 conn(0x7f30600cddd0 msgr2=0x7f30600ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306507d700 1 --2- 172.21.15.158:0/3447410014 >> v2:172.21.15.172:3300/0 conn(0x7f30600cddd0 0x7f30600ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f3057fff700 1 -- 172.21.15.158:0/3447410014 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977701718 0 0) 0x7f306015f450 con 0x7f30600d0b60 2023-10-16T11:01:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306507d700 1 -- 172.21.15.158:0/3447410014 >> v1:172.21.15.158:6789/0 conn(0x7f30600d0b60 legacy=0x7f306015b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306507d700 1 -- 172.21.15.158:0/3447410014 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306015c3e0 con 0x7f306015b5b0 2023-10-16T11:01:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306587e700 1 --2- 172.21.15.158:0/3447410014 >> v2:172.21.15.172:3300/0 conn(0x7f30600cddd0 0x7f30600ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.035+0000 7f306507d700 1 --2- 172.21.15.158:0/3447410014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f306015f8e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f30600b74e0 tx=0x7f305c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31b8a663dcdfbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f3057fff700 1 -- 172.21.15.158:0/3447410014 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c011830 con 0x7f306015b5b0 2023-10-16T11:01:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f3057fff700 1 -- 172.21.15.158:0/3447410014 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305c011990 con 0x7f306015b5b0 2023-10-16T11:01:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f3057fff700 1 -- 172.21.15.158:0/3447410014 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c011d20 con 0x7f306015b5b0 2023-10-16T11:01:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 -- 172.21.15.158:0/3447410014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 msgr2=0x7f306015f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 --2- 172.21.15.158:0/3447410014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f306015f8e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f30600b74e0 tx=0x7f305c009b20 comp rx=0 tx=0).stop 2023-10-16T11:01:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 -- 172.21.15.158:0/3447410014 shutdown_connections 2023-10-16T11:01:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 --2- 172.21.15.158:0/3447410014 >> v2:172.21.15.172:3300/0 conn(0x7f30600cddd0 0x7f30600ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 --2- 172.21.15.158:0/3447410014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f306015f8e0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 -- 172.21.15.158:0/3447410014 >> 172.21.15.158:0/3447410014 conn(0x7f306006ec70 msgr2=0x7f306006f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 -- 172.21.15.158:0/3447410014 shutdown_connections 2023-10-16T11:01:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.036+0000 7f30672e1700 1 -- 172.21.15.158:0/3447410014 wait complete. 2023-10-16T11:01:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.037+0000 7f30672e1700 1 Processor -- start 2023-10-16T11:01:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f30672e1700 1 -- start start 2023-10-16T11:01:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f30672e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30600cddd0 0x7f306015e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f30672e1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30600d0b60 0x7f30600c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f30672e1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f30600c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f30672e1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306006c3a0 con 0x7f306015b5b0 2023-10-16T11:01:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f30672e1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306006c640 con 0x7f30600d0b60 2023-10-16T11:01:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f30672e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306006c920 con 0x7f30600cddd0 2023-10-16T11:01:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f306587e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f30600c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f306587e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f30600c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34662/0 (socket says 172.21.15.158:34662) 2023-10-16T11:01:23.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.038+0000 7f306587e700 1 -- 172.21.15.158:0/547090074 learned_addr learned my addr 172.21.15.158:0/547090074 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:23.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306507d700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30600cddd0 0x7f306015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306487c700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30600d0b60 0x7f30600c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306587e700 1 -- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30600cddd0 msgr2=0x7f306015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306587e700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30600cddd0 0x7f306015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306587e700 1 -- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30600d0b60 msgr2=0x7f30600c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306587e700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30600d0b60 0x7f30600c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306587e700 1 -- 172.21.15.158:0/547090074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c0094a0 con 0x7f306015b5b0 2023-10-16T11:01:23.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f306587e700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f30600c5430 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f305800cef0 tx=0x7f305800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:23.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f3055ffb700 1 -- 172.21.15.158:0/547090074 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305800ae50 con 0x7f306015b5b0 2023-10-16T11:01:23.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f3055ffb700 1 -- 172.21.15.158:0/547090074 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3058015440 con 0x7f306015b5b0 2023-10-16T11:01:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f3055ffb700 1 -- 172.21.15.158:0/547090074 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30580157f0 con 0x7f306015b5b0 2023-10-16T11:01:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306006cc00 con 0x7f306015b5b0 2023-10-16T11:01:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.039+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30600b50a0 con 0x7f306015b5b0 2023-10-16T11:01:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.040+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30600b52c0 con 0x7f306015b5b0 2023-10-16T11:01:23.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.040+0000 7f3055ffb700 1 -- 172.21.15.158:0/547090074 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f305800d430 con 0x7f306015b5b0 2023-10-16T11:01:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.041+0000 7f3055ffb700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f304406b020 0x7f304406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.041+0000 7f3055ffb700 1 -- 172.21.15.158:0/547090074 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f305808a0d0 con 0x7f306015b5b0 2023-10-16T11:01:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.041+0000 7f306507d700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f304406b020 0x7f304406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.041+0000 7f306507d700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f304406b020 0x7f304406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f305c0090b0 tx=0x7f305c01c040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:23.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.045+0000 7f3055ffb700 1 -- 172.21.15.158:0/547090074 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3058054d60 con 0x7f306015b5b0 2023-10-16T11:01:23.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.318+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f30600681b0 con 0x7f306015b5b0 2023-10-16T11:01:23.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.320+0000 7f3055ffb700 1 -- 172.21.15.158:0/547090074 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v45) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3058058380 con 0x7f306015b5b0 2023-10-16T11:01:23.321 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T11:01:23.321 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T11:01:23.321 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T11:01:23.322 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T11:01:23.322 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T11:01:23.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f304406b020 msgr2=0x7f304406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f304406b020 0x7f304406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f305c0090b0 tx=0x7f305c01c040 comp rx=0 tx=0).stop 2023-10-16T11:01:23.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 msgr2=0x7f30600c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f30600c5430 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f305800cef0 tx=0x7f305800a340 comp rx=0 tx=0).stop 2023-10-16T11:01:23.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 shutdown_connections 2023-10-16T11:01:23.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f304406b020 0x7f304406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306015b5b0 0x7f30600c5430 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30600cddd0 0x7f306015e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 --2- 172.21.15.158:0/547090074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30600d0b60 0x7f30600c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 >> 172.21.15.158:0/547090074 conn(0x7f306006ec70 msgr2=0x7f30600cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:23.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 shutdown_connections 2023-10-16T11:01:23.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.322+0000 7f30672e1700 1 -- 172.21.15.158:0/547090074 wait complete. 2023-10-16T11:01:23.326 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T11:01:23.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:01:23.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.456+0000 7f3f966d4700 1 Processor -- start 2023-10-16T11:01:23.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.456+0000 7f3f966d4700 1 -- start start 2023-10-16T11:01:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f966d4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 0x7f3f90072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f966d4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3f9006afe0 0x7f3f900caae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f966d4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f900ceac0 con 0x7f3f900727d0 2023-10-16T11:01:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f966d4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f900cdeb0 con 0x7f3f9006afe0 2023-10-16T11:01:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f966d4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f900ce190 con 0x7f3f9015d7c0 2023-10-16T11:01:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3f900727d0 0x7f3f9006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34842/0 (socket says 172.21.15.158:34842) 2023-10-16T11:01:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8f7fe700 1 -- 172.21.15.158:0/943742648 learned_addr learned my addr 172.21.15.158:0/943742648 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747233430 0 0) 0x7f3f900ceac0 con 0x7f3f900727d0 2023-10-16T11:01:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f780035b0 con 0x7f3f900727d0 2023-10-16T11:01:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8ffff700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 0x7f3f90072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f94c71700 1 --2- 172.21.15.158:0/943742648 >> v2:172.21.15.172:3300/0 conn(0x7f3f9006afe0 0x7f3f900caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2992198471 0 0) 0x7f3f780035b0 con 0x7f3f900727d0 2023-10-16T11:01:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 msgr2=0x7f3f90072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 0x7f3f90072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 >> v2:172.21.15.172:3300/0 conn(0x7f3f9006afe0 msgr2=0x7f3f900caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:23.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 --2- 172.21.15.158:0/943742648 >> v2:172.21.15.172:3300/0 conn(0x7f3f9006afe0 0x7f3f900caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f900ce470 con 0x7f3f900727d0 2023-10-16T11:01:23.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.457+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3f80001670 con 0x7f3f900727d0 2023-10-16T11:01:23.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 >> v1:172.21.15.158:6789/0 conn(0x7f3f900727d0 legacy=0x7f3f9006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 0x7f3f78003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f78003fd0 0x7f3f78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78004d90 0x7f3f78005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f900cdeb0 con 0x7f3f78003fd0 2023-10-16T11:01:23.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f900ce190 con 0x7f3f78004d90 2023-10-16T11:01:23.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7800bac0 con 0x7f3f9015d7c0 2023-10-16T11:01:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8f7fe700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f78003fd0 0x7f3f78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8ffff700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 0x7f3f78003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f94c71700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78004d90 0x7f3f78005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8f7fe700 1 -- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 msgr2=0x7f3f78003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8f7fe700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f9015d7c0 0x7f3f78003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8f7fe700 1 -- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78004d90 msgr2=0x7f3f78005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8f7fe700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78004d90 0x7f3f78005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8f7fe700 1 -- 172.21.15.158:0/943742648 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f780035b0 con 0x7f3f78003fd0 2023-10-16T11:01:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8f7fe700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f78003fd0 0x7f3f78004850 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000c210 tx=0x7f3f78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96425d1b650e498 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8000cd50 con 0x7f3f78003fd0 2023-10-16T11:01:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f80005960 con 0x7f3f78003fd0 2023-10-16T11:01:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.458+0000 7f3f8effd700 1 -- 172.21.15.158:0/943742648 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80005d10 con 0x7f3f78003fd0 2023-10-16T11:01:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.459+0000 7f3f966d4700 1 -- 172.21.15.158:0/943742648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f78003fd0 msgr2=0x7f3f78004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.459+0000 7f3f966d4700 1 --2- 172.21.15.158:0/943742648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f78003fd0 0x7f3f78004850 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000c210 tx=0x7f3f78005880 comp rx=0 tx=0).stop 2023-10-16T11:01:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.459+0000 7f3f94c71700 1 -- 172.21.15.158:0/943742648 reap_dead start 2023-10-16T11:01:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.459+0000 7f3f966d4700 1 -- 172.21.15.158:0/943742648 shutdown_connections 2023-10-16T11:01:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.459+0000 7f3f966d4700 1 -- 172.21.15.158:0/943742648 >> 172.21.15.158:0/943742648 conn(0x7f3f900ccd60 msgr2=0x7f3f900cd170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.459+0000 7f3f966d4700 1 -- 172.21.15.158:0/943742648 shutdown_connections 2023-10-16T11:01:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.459+0000 7f3f966d4700 1 -- 172.21.15.158:0/943742648 wait complete. 2023-10-16T11:01:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.460+0000 7f3f966d4700 1 Processor -- start 2023-10-16T11:01:23.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.460+0000 7f3f966d4700 1 -- start start 2023-10-16T11:01:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f966d4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78003fd0 0x7f3f900b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f966d4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f901516b0 0x7f3f901f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f966d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f901f22a0 0x7f3f900c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f966d4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f901f2700 con 0x7f3f901f22a0 2023-10-16T11:01:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f966d4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f90151e20 con 0x7f3f78003fd0 2023-10-16T11:01:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f966d4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f90152100 con 0x7f3f901516b0 2023-10-16T11:01:23.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f901f22a0 0x7f3f900c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f901f22a0 0x7f3f900c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34682/0 (socket says 172.21.15.158:34682) 2023-10-16T11:01:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 -- 172.21.15.158:0/214798158 learned_addr learned my addr 172.21.15.158:0/214798158 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f8f7fe700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f901516b0 0x7f3f901f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f8ffff700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78003fd0 0x7f3f900b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 -- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f901516b0 msgr2=0x7f3f901f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f901516b0 0x7f3f901f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 -- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78003fd0 msgr2=0x7f3f900b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78003fd0 0x7f3f900b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 -- 172.21.15.158:0/214798158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f901523e0 con 0x7f3f901f22a0 2023-10-16T11:01:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.461+0000 7f3f94c71700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f901f22a0 0x7f3f900c9580 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f3f900670a0 tx=0x7f3f88004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.462+0000 7f3f8cff9700 1 -- 172.21.15.158:0/214798158 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88007c00 con 0x7f3f901f22a0 2023-10-16T11:01:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.462+0000 7f3f8cff9700 1 -- 172.21.15.158:0/214798158 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8800f050 con 0x7f3f901f22a0 2023-10-16T11:01:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.462+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9014f7a0 con 0x7f3f901f22a0 2023-10-16T11:01:23.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.462+0000 7f3f8cff9700 1 -- 172.21.15.158:0/214798158 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88003e30 con 0x7f3f901f22a0 2023-10-16T11:01:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.462+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f900c9ac0 con 0x7f3f901f22a0 2023-10-16T11:01:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.462+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9014f990 con 0x7f3f901f22a0 2023-10-16T11:01:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.463+0000 7f3f8cff9700 1 -- 172.21.15.158:0/214798158 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3f8801e460 con 0x7f3f901f22a0 2023-10-16T11:01:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.463+0000 7f3f8cff9700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f3f7c06b020 0x7f3f7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.463+0000 7f3f8cff9700 1 -- 172.21.15.158:0/214798158 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f3f88089f30 con 0x7f3f901f22a0 2023-10-16T11:01:23.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.463+0000 7f3f8ffff700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f3f7c06b020 0x7f3f7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:23.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.464+0000 7f3f8ffff700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f3f7c06b020 0x7f3f7c06d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f840099c0 tx=0x7f3f84008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:23.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.468+0000 7f3f8cff9700 1 -- 172.21.15.158:0/214798158 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f88054ab0 con 0x7f3f901f22a0 2023-10-16T11:01:23.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.704+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f900681b0 con 0x7f3f901f22a0 2023-10-16T11:01:23.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.704+0000 7f3f8cff9700 1 -- 172.21.15.158:0/214798158 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4930 (secure 0 0 0) 0x7f3f880580d0 con 0x7f3f901f22a0 2023-10-16T11:01:23.706 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:01:23.706 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3580267390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3580267390},{"type":"v1","addr":"172.21.15.172:6834","nonce":3580267390}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":9328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3433152480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3433152480},{"type":"v1","addr":"172.21.15.172:6835","nonce":3433152480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1170239465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1170239465},{"type":"v1","addr":"172.21.15.158:6836","nonce":1170239465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:01:17.640665+0000","modified":"2023-10-16T11:01:20.662098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10449":{"gid":10449,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/773046671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":773046671},{"type":"v1","addr":"172.21.15.158:6837","nonce":773046671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T11:01:23.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f3f7c06b020 msgr2=0x7f3f7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f3f7c06b020 0x7f3f7c06d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f840099c0 tx=0x7f3f84008040 comp rx=0 tx=0).stop 2023-10-16T11:01:23.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f901f22a0 msgr2=0x7f3f900c9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:23.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f901f22a0 0x7f3f900c9580 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f3f900670a0 tx=0x7f3f88004970 comp rx=0 tx=0).stop 2023-10-16T11:01:23.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 shutdown_connections 2023-10-16T11:01:23.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f3f7c06b020 0x7f3f7c06d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f901f22a0 0x7f3f900c9580 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f78003fd0 0x7f3f900b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 --2- 172.21.15.158:0/214798158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f901516b0 0x7f3f901f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:23.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.706+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 >> 172.21.15.158:0/214798158 conn(0x7f3f900ccd60 msgr2=0x7f3f900ca9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:23.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.707+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 shutdown_connections 2023-10-16T11:01:23.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:23.707+0000 7f3f966d4700 1 -- 172.21.15.158:0/214798158 wait complete. 2023-10-16T11:01:23.711 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 94 2023-10-16T11:01:24.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:01:17.640665+0000', 'modified': '2023-10-16T11:01:20.662098+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10449}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10449': {'gid': 10449, 'name': 'a', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6837/773046671', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6835', 'nonce': 773046671}, {'type': 'v1', 'addr': '172.21.15.158:6837', 'nonce': 773046671}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:01:24.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:01:24.124 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T11:01:24.230 INFO:tasks.ceph.mds.a.smithi158.stderr:2023-10-16T11:01:24.228+0000 7f51895da700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:01:24.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.247+0000 7f8fe023f700 1 Processor -- start 2023-10-16T11:01:24.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.248+0000 7f8fe023f700 1 -- start start 2023-10-16T11:01:24.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.248+0000 7f8fe023f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8fd80caa80 0x7f8fd8071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.248+0000 7f8fe023f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd806cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.248+0000 7f8fe023f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80716b0 con 0x7f8fd8159a10 2023-10-16T11:01:24.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.248+0000 7f8fe023f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd806d150 con 0x7f8fd80caa80 2023-10-16T11:01:24.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.248+0000 7f8fe023f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd8071990 con 0x7f8fd806c780 2023-10-16T11:01:24.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8fd80caa80 0x7f8fd8071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8fd80caa80 0x7f8fd8071010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:35870/0 (socket says 172.21.15.158:35870) 2023-10-16T11:01:24.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2068626850 learned_addr learned my addr 172.21.15.158:0/2068626850 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:24.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd806cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 msgr2=0x7f8fd806cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd806cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2068626850 >> v1:172.21.15.172:6789/0 conn(0x7f8fd8159a10 legacy=0x7f8fd815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2068626850 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd8071c70 con 0x7f8fd80caa80 2023-10-16T11:01:24.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd806cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:01:24.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd80caa80 0x7f8fd8071010 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0098f0 tx=0x7f8fcc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2515343f91de4231 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.249+0000 7f8fdcfd9700 1 -- 172.21.15.158:0/2068626850 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc011830 con 0x7f8fd80caa80 2023-10-16T11:01:24.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fdcfd9700 1 -- 172.21.15.158:0/2068626850 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc011990 con 0x7f8fd80caa80 2023-10-16T11:01:24.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fdcfd9700 1 -- 172.21.15.158:0/2068626850 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc011d20 con 0x7f8fd80caa80 2023-10-16T11:01:24.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 -- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd80caa80 msgr2=0x7f8fd8071010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd80caa80 0x7f8fd8071010 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0098f0 tx=0x7f8fcc009c00 comp rx=0 tx=0).stop 2023-10-16T11:01:24.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 -- 172.21.15.158:0/2068626850 shutdown_connections 2023-10-16T11:01:24.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd806cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2068626850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd80caa80 0x7f8fd8071010 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 -- 172.21.15.158:0/2068626850 >> 172.21.15.158:0/2068626850 conn(0x7f8fd80cbd30 msgr2=0x7f8fd80cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 -- 172.21.15.158:0/2068626850 shutdown_connections 2023-10-16T11:01:24.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.250+0000 7f8fe023f700 1 -- 172.21.15.158:0/2068626850 wait complete. 2023-10-16T11:01:24.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 Processor -- start 2023-10-16T11:01:24.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 -- start start 2023-10-16T11:01:24.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd8156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd80caa80 0x7f8fd8154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd8159a10 0x7f8fd8154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd8158c50 con 0x7f8fd80caa80 2023-10-16T11:01:24.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd814e1a0 con 0x7f8fd8159a10 2023-10-16T11:01:24.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fe023f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd814e480 con 0x7f8fd806c780 2023-10-16T11:01:24.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fdd7da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd80caa80 0x7f8fd8154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fdd7da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd80caa80 0x7f8fd8154700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34694/0 (socket says 172.21.15.158:34694) 2023-10-16T11:01:24.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.252+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2017546831 learned_addr learned my addr 172.21.15.158:0/2017546831 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:24.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd8156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd8159a10 0x7f8fd8154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 msgr2=0x7f8fd8156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd8156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd8159a10 msgr2=0x7f8fd8154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd8159a10 0x7f8fd8154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2017546831 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd0009a70 con 0x7f8fd80caa80 2023-10-16T11:01:24.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd80caa80 0x7f8fd8154700 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc000c00 tx=0x7f8fcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fca7fc700 1 -- 172.21.15.158:0/2017546831 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0043c0 con 0x7f8fd80caa80 2023-10-16T11:01:24.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fca7fc700 1 -- 172.21.15.158:0/2017546831 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc01f070 con 0x7f8fd80caa80 2023-10-16T11:01:24.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fca7fc700 1 -- 172.21.15.158:0/2017546831 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0038c0 con 0x7f8fd80caa80 2023-10-16T11:01:24.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc009550 con 0x7f8fd80caa80 2023-10-16T11:01:24.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.253+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd814ea40 con 0x7f8fd80caa80 2023-10-16T11:01:24.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.254+0000 7f8fca7fc700 1 -- 172.21.15.158:0/2017546831 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8fcc021030 con 0x7f8fd80caa80 2023-10-16T11:01:24.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.255+0000 7f8fca7fc700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8fbc06b020 0x7f8fbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.255+0000 7f8fca7fc700 1 -- 172.21.15.158:0/2017546831 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f8fcc08ad40 con 0x7f8fd80caa80 2023-10-16T11:01:24.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.255+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8fbc06b020 0x7f8fbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.255+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8fbc06b020 0x7f8fbc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007fd0 tx=0x7f8fd4009040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.257+0000 7f8fec429b80 1 Processor -- start 2023-10-16T11:01:24.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.257+0000 7f8fec429b80 1 -- start start 2023-10-16T11:01:24.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211fbca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211f906c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fec429b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211f06c20 con 0x564211fb5220 2023-10-16T11:01:24.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211eebc10 con 0x564211f90010 2023-10-16T11:01:24.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564211ee9530 con 0x564211fba630 2023-10-16T11:01:24.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fddfdb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211fbca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fddfdb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211fbca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34704/0 (socket says 172.21.15.158:34704) 2023-10-16T11:01:24.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.258+0000 7f8fddfdb700 1 -- 172.21.15.158:0/4132404330 learned_addr learned my addr 172.21.15.158:0/4132404330 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:24.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211f906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fddfdb700 1 -- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 msgr2=0x564211fbcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fddfdb700 1 -- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 msgr2=0x564211f906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211f906c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fddfdb700 1 -- 172.21.15.158:0/4132404330 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564211ea23a0 con 0x564211fb5220 2023-10-16T11:01:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211f906c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211fbca70 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400f6c0 tx=0x7f8fd4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29504a2e075847d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fbb7fe700 1 -- 172.21.15.158:0/4132404330 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd400fe40 con 0x564211fb5220 2023-10-16T11:01:24.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fbb7fe700 1 -- 172.21.15.158:0/4132404330 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd4014450 con 0x564211fb5220 2023-10-16T11:01:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.259+0000 7f8fbb7fe700 1 -- 172.21.15.158:0/4132404330 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4014820 con 0x564211fb5220 2023-10-16T11:01:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 -- 172.21.15.158:0/4132404330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 msgr2=0x564211fbca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211fbca70 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400f6c0 tx=0x7f8fd4012000 comp rx=0 tx=0).stop 2023-10-16T11:01:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 -- 172.21.15.158:0/4132404330 shutdown_connections 2023-10-16T11:01:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211f906c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211fbca70 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 --2- 172.21.15.158:0/4132404330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 -- 172.21.15.158:0/4132404330 >> 172.21.15.158:0/4132404330 conn(0x564211ebfd80 msgr2=0x564211fdaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 -- 172.21.15.158:0/4132404330 shutdown_connections 2023-10-16T11:01:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 -- 172.21.15.158:0/4132404330 wait complete. 2023-10-16T11:01:24.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 -- start start 2023-10-16T11:01:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.260+0000 7f8fec429b80 1 Processor -- start 2023-10-16T11:01:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211f90010 0x564211fd7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211fb5220 0x564211fbee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211f90480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fec429b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211ee4e20 con 0x564211fba630 2023-10-16T11:01:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211e816c0 con 0x564211fb5220 2023-10-16T11:01:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564211e81360 con 0x564211f90010 2023-10-16T11:01:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fec429b80 10 client.0 _ll_register_callbacks cb 0x564211cde7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:01:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fde7dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211f90480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fde7dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211f90480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34710/0 (socket says 172.21.15.158:34710) 2023-10-16T11:01:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.261+0000 7f8fde7dc700 1 -- 172.21.15.158:0/1928744986 learned_addr learned my addr 172.21.15.158:0/1928744986 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211f90010 0x564211fd7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fde7dc700 1 -- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211f90010 msgr2=0x564211fd7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211fb5220 0x564211fbee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211f90010 0x564211fd7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fde7dc700 1 -- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211fb5220 msgr2=0x564211fbee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211fb5220 0x564211fbee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fde7dc700 1 -- 172.21.15.158:0/1928744986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd80716b0 con 0x564211fba630 2023-10-16T11:01:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211f90010 0x564211fd7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211fb5220 0x564211fbee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:24.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211f90480 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009980 tx=0x7f8fd0012520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724c6fbf2748c7b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fb8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:01:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/1928744986 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd000e040 con 0x564211fba630 2023-10-16T11:01:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/1928744986 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f8fd0018470 con 0x564211fba630 2023-10-16T11:01:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564211f855b0 con 0x564211fba630 2023-10-16T11:01:24.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/1928744986 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fd0018630 con 0x564211fba630 2023-10-16T11:01:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fec429b80 10 client.10605 fetch_fsmap learned FSMap version 94 2023-10-16T11:01:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x564211d0fd90 con 0x564211fba630 2023-10-16T11:01:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.262+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/1928744986 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564211f855b0 con 0x564211fba630 2023-10-16T11:01:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/1928744986 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 94) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8fd0014040 con 0x564211fba630 2023-10-16T11:01:24.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fec429b80 10 client.10605 fetch_fsmap finished waiting for FSMap version 94 2023-10-16T11:01:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fec429b80 10 client.10605 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T11:01:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fec429b80 20 client.10605 populate_metadata read hostname 'smithi158' 2023-10-16T11:01:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fec429b80 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fe6e50 0x564211fbdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fec429b80 4 client.10605 mds_command: new command op to 10449 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:01:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 --> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564211edde00 con 0x564211fe6e50 2023-10-16T11:01:24.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fe6e50 0x564211fbdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fb8ff9700 10 client.10605 ms_handle_connect on v2:172.21.15.158:6835/773046671 2023-10-16T11:01:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.263+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fe6e50 0x564211fbdf70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39c22b3f8e42bd66 server_cookie=c5a335fba5d0a86a in_seq=0 out_seq=0 2023-10-16T11:01:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/1928744986 <== mds.0 v2:172.21.15.158:6835/773046671 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564211edde00 con 0x564211fe6e50 2023-10-16T11:01:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fb8ff9700 10 client.10605 handle_command_reply: tid=0 2023-10-16T11:01:24.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 client.10605 shutdown 2023-10-16T11:01:24.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 msgr2=0x564211f90480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211f90480 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009980 tx=0x7f8fd0012520 comp rx=0 tx=0).stop 2023-10-16T11:01:24.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 shutdown_connections 2023-10-16T11:01:24.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fe6e50 0x564211fbdf70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211f90480 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fb8ff9700 0 client.10605 ms_handle_reset on v2:172.21.15.158:6835/773046671 2023-10-16T11:01:24.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211f90010 0x564211fd7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 --2- 172.21.15.158:0/1928744986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211fb5220 0x564211fbee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 >> 172.21.15.158:0/1928744986 conn(0x564211ebfd80 msgr2=0x564211fbca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 shutdown_connections 2023-10-16T11:01:24.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 1 -- 172.21.15.158:0/1928744986 wait complete. 2023-10-16T11:01:24.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.264+0000 7f8fec429b80 20 client.10605 trim_cache size 0 max 16384 2023-10-16T11:01:24.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.282+0000 7f8fec429b80 1 Processor -- start 2023-10-16T11:01:24.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.282+0000 7f8fec429b80 1 -- start start 2023-10-16T11:01:24.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211ff0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211f92540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fec429b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211d658d0 con 0x564211fb5220 2023-10-16T11:01:24.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211fb4950 con 0x564211f90010 2023-10-16T11:01:24.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564211fdb2b0 con 0x564211fba630 2023-10-16T11:01:24.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211f92540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211f92540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34714/0 (socket says 172.21.15.158:34714) 2023-10-16T11:01:24.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2720703533 learned_addr learned my addr 172.21.15.158:0/2720703533 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:24.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211ff0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 msgr2=0x564211fbdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 msgr2=0x564211ff0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211ff0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 -- 172.21.15.158:0/2720703533 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564211cf8c10 con 0x564211fb5220 2023-10-16T11:01:24.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211ff0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:24.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.283+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211f92540 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc006b20 tx=0x7f8fcc059330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe0fc231810d82e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fbbfff700 1 -- 172.21.15.158:0/2720703533 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc059720 con 0x564211fb5220 2023-10-16T11:01:24.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fbbfff700 1 -- 172.21.15.158:0/2720703533 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc05d870 con 0x564211fb5220 2023-10-16T11:01:24.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fbbfff700 1 -- 172.21.15.158:0/2720703533 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc061a20 con 0x564211fb5220 2023-10-16T11:01:24.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 -- 172.21.15.158:0/2720703533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 msgr2=0x564211f92540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211f92540 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc006b20 tx=0x7f8fcc059330 comp rx=0 tx=0).stop 2023-10-16T11:01:24.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 -- 172.21.15.158:0/2720703533 shutdown_connections 2023-10-16T11:01:24.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fba630 0x564211fbdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x564211ff0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 --2- 172.21.15.158:0/2720703533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fb5220 0x564211f92540 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 -- 172.21.15.158:0/2720703533 >> 172.21.15.158:0/2720703533 conn(0x564211ebfd80 msgr2=0x564211ff9770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 -- 172.21.15.158:0/2720703533 shutdown_connections 2023-10-16T11:01:24.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.284+0000 7f8fec429b80 1 -- 172.21.15.158:0/2720703533 wait complete. 2023-10-16T11:01:24.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.285+0000 7f8fec429b80 1 -- start start 2023-10-16T11:01:24.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.285+0000 7f8fec429b80 1 Processor -- start 2023-10-16T11:01:24.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x56421203e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fb5220 0x564211fbdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211fbe430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211ff0c70 con 0x564211fba630 2023-10-16T11:01:24.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x564211fd2710 con 0x564211f90010 2023-10-16T11:01:24.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x564211ff0e70 con 0x564211fb5220 2023-10-16T11:01:24.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 10 client.0 _ll_register_callbacks cb 0x7f8fd0007b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:01:24.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211fbe430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211fbe430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34724/0 (socket says 172.21.15.158:34724) 2023-10-16T11:01:24.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 -- 172.21.15.158:0/782903647 learned_addr learned my addr 172.21.15.158:0/782903647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:24.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fdd7da700 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fb5220 0x564211fbdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x56421203e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 -- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fb5220 msgr2=0x564211fbdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fb5220 0x564211fbdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 -- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 msgr2=0x56421203e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x56421203e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 -- 172.21.15.158:0/782903647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd814ea40 con 0x564211fba630 2023-10-16T11:01:24.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fde7dc700 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211fbe430 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0007e10 tx=0x7f8fd0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3af117c86e151 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fb8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:01:24.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/782903647 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd000e050 con 0x564211fba630 2023-10-16T11:01:24.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/782903647 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f8fd0003a20 con 0x564211fba630 2023-10-16T11:01:24.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.286+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fd4014450 con 0x564211fba630 2023-10-16T11:01:24.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/782903647 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fd0003be0 con 0x564211fba630 2023-10-16T11:01:24.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 10 client.10611 fetch_fsmap learned FSMap version 94 2023-10-16T11:01:24.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x564211d36200 con 0x564211fba630 2023-10-16T11:01:24.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/782903647 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd4014450 con 0x564211fba630 2023-10-16T11:01:24.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/782903647 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 94) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8fd0012240 con 0x564211fba630 2023-10-16T11:01:24.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 10 client.10611 fetch_fsmap finished waiting for FSMap version 94 2023-10-16T11:01:24.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 10 client.10611 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T11:01:24.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 20 client.10611 populate_metadata read hostname 'smithi158' 2023-10-16T11:01:24.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fd6430 0x564211fd6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 4 client.10611 mds_command: new command op to 10449 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:01:24.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 --> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564211eddb20 con 0x564211fd6430 2023-10-16T11:01:24.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fd6430 0x564211fd6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fb8ff9700 10 client.10611 ms_handle_connect on v2:172.21.15.158:6835/773046671 2023-10-16T11:01:24.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.287+0000 7f8fddfdb700 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fd6430 0x564211fd6810 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6af48a640dba055 server_cookie=4f529f25355347a8 in_seq=0 out_seq=0 2023-10-16T11:01:24.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fb8ff9700 1 -- 172.21.15.158:0/782903647 <== mds.0 v2:172.21.15.158:6835/773046671 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x564211eddb20 con 0x564211fd6430 2023-10-16T11:01:24.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fb8ff9700 10 client.10611 handle_command_reply: tid=0 2023-10-16T11:01:24.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 client.10611 shutdown 2023-10-16T11:01:24.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 msgr2=0x564211fbe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211fbe430 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0007e10 tx=0x7f8fd0015000 comp rx=0 tx=0).stop 2023-10-16T11:01:24.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 shutdown_connections 2023-10-16T11:01:24.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:6835/773046671,v1:172.21.15.158:6837/773046671] conn(0x564211fd6430 0x564211fd6810 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x564211fba630 0x564211fbe430 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fb8ff9700 0 client.10611 ms_handle_reset on v2:172.21.15.158:6835/773046671 2023-10-16T11:01:24.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564211f90010 0x56421203e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 --2- 172.21.15.158:0/782903647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x564211fb5220 0x564211fbdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 >> 172.21.15.158:0/782903647 conn(0x564211ebfd80 msgr2=0x564211fb7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 shutdown_connections 2023-10-16T11:01:24.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 1 -- 172.21.15.158:0/782903647 wait complete. 2023-10-16T11:01:24.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fec429b80 20 client.10611 trim_cache size 0 max 16384 2023-10-16T11:01:24.320 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:01:24.320 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:01:24.320 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:01:24.321 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 10449, 2023-10-16T11:01:24.321 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:01:24.321 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:01:24.321 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T11:01:24.321 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 6.0403113089999998, 2023-10-16T11:01:24.322 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 94, 2023-10-16T11:01:24.322 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 185, 2023-10-16T11:01:24.322 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 185, 2023-10-16T11:01:24.322 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.977742037000001 2023-10-16T11:01:24.323 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:01:24.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8fbc06b020 msgr2=0x7f8fbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8fbc06b020 0x7f8fbc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007fd0 tx=0x7f8fd4009040 comp rx=0 tx=0).stop 2023-10-16T11:01:24.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.288+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd80caa80 msgr2=0x7f8fd8154700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd80caa80 0x7f8fd8154700 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc000c00 tx=0x7f8fcc004750 comp rx=0 tx=0).stop 2023-10-16T11:01:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 shutdown_connections 2023-10-16T11:01:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f8fbc06b020 0x7f8fbc06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fd8159a10 0x7f8fd8154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fd806c780 0x7f8fd8156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 --2- 172.21.15.158:0/2017546831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd80caa80 0x7f8fd8154700 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 >> 172.21.15.158:0/2017546831 conn(0x7f8fd80cbd30 msgr2=0x7f8fd8158240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 shutdown_connections 2023-10-16T11:01:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.289+0000 7f8fe023f700 1 -- 172.21.15.158:0/2017546831 wait complete. 2023-10-16T11:01:24.327 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T11:01:24.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.452+0000 7fcc7af8c700 1 Processor -- start 2023-10-16T11:01:24.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.452+0000 7fcc7af8c700 1 -- start start 2023-10-16T11:01:24.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.453+0000 7fcc7af8c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc7415d640 0x7fcc740cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.453+0000 7fcc7af8c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc74161340 con 0x7fcc740c39b0 2023-10-16T11:01:24.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.453+0000 7fcc7af8c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc74161ec0 con 0x7fcc7406a8d0 2023-10-16T11:01:24.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.453+0000 7fcc7af8c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc74072970 con 0x7fcc7415d640 2023-10-16T11:01:24.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.453+0000 7fcc79529700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcc740c39b0 0x7fcc7406f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34904/0 (socket says 172.21.15.158:34904) 2023-10-16T11:01:24.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.453+0000 7fcc79529700 1 -- 172.21.15.158:0/1331625320 learned_addr learned my addr 172.21.15.158:0/1331625320 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:24.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553580633 0 0) 0x7fcc74161340 con 0x7fcc740c39b0 2023-10-16T11:01:24.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc78d28700 1 --2- 172.21.15.158:0/1331625320 >> v2:172.21.15.172:3300/0 conn(0x7fcc7415d640 0x7fcc740cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc640035b0 con 0x7fcc740c39b0 2023-10-16T11:01:24.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095253404 0 0) 0x7fcc640035b0 con 0x7fcc740c39b0 2023-10-16T11:01:24.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 >> v2:172.21.15.172:3300/0 conn(0x7fcc7415d640 msgr2=0x7fcc740cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 --2- 172.21.15.158:0/1331625320 >> v2:172.21.15.172:3300/0 conn(0x7fcc7415d640 0x7fcc740cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 >> v1:172.21.15.172:6789/0 conn(0x7fcc7406a8d0 legacy=0x7fcc7406eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7406fcd0 con 0x7fcc740c39b0 2023-10-16T11:01:24.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcc6c0019f0 con 0x7fcc740c39b0 2023-10-16T11:01:24.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 >> v1:172.21.15.158:6789/0 conn(0x7fcc740c39b0 legacy=0x7fcc7406f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc64003a00 0x7fcc64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc64004850 0x7fcc64004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc6400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74072970 con 0x7fcc64005210 2023-10-16T11:01:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6400c0c0 con 0x7fcc64004850 2023-10-16T11:01:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6400c280 con 0x7fcc64003a00 2023-10-16T11:01:24.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc79529700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc6400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc73fff700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc64004850 0x7fcc64004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc78d28700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc64003a00 0x7fcc64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc79529700 1 -- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc64003a00 msgr2=0x7fcc64003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc79529700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc64003a00 0x7fcc64003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc79529700 1 -- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc64004850 msgr2=0x7fcc64004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:24.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc79529700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc64004850 0x7fcc64004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.454+0000 7fcc79529700 1 -- 172.21.15.158:0/1331625320 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640035b0 con 0x7fcc64005210 2023-10-16T11:01:24.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc79529700 1 -- 172.21.15.158:0/1331625320 reap_dead start 2023-10-16T11:01:24.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc79529700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc6400ba10 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00a820 tx=0x7fcc64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88f5e7196c921fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c00a2f0 con 0x7fcc64005210 2023-10-16T11:01:24.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c012600 con 0x7fcc64005210 2023-10-16T11:01:24.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc737fe700 1 -- 172.21.15.158:0/1331625320 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0129b0 con 0x7fcc64005210 2023-10-16T11:01:24.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc7af8c700 1 -- 172.21.15.158:0/1331625320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 msgr2=0x7fcc6400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc6400ba10 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00a820 tx=0x7fcc64005880 comp rx=0 tx=0).stop 2023-10-16T11:01:24.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc7af8c700 1 -- 172.21.15.158:0/1331625320 shutdown_connections 2023-10-16T11:01:24.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/1331625320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc6400ba10 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc7af8c700 1 -- 172.21.15.158:0/1331625320 >> 172.21.15.158:0/1331625320 conn(0x7fcc740b1e10 msgr2=0x7fcc740b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc7af8c700 1 -- 172.21.15.158:0/1331625320 shutdown_connections 2023-10-16T11:01:24.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.455+0000 7fcc7af8c700 1 -- 172.21.15.158:0/1331625320 wait complete. 2023-10-16T11:01:24.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.457+0000 7fcc7af8c700 1 Processor -- start 2023-10-16T11:01:24.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.457+0000 7fcc7af8c700 1 -- start start 2023-10-16T11:01:24.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.457+0000 7fcc7af8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc74148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.457+0000 7fcc7af8c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc74159ae0 0x7fcc7415e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.457+0000 7fcc7af8c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc740c95c0 0x7fcc740c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc7af8c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740c9fb0 con 0x7fcc64005210 2023-10-16T11:01:24.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc7af8c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74158840 con 0x7fcc740c95c0 2023-10-16T11:01:24.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc7af8c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74159f60 con 0x7fcc74159ae0 2023-10-16T11:01:24.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc74148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc74148430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34738/0 (socket says 172.21.15.158:34738) 2023-10-16T11:01:24.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 -- 172.21.15.158:0/2970377295 learned_addr learned my addr 172.21.15.158:0/2970377295 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:24.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 -- 172.21.15.158:0/2970377295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc74159ae0 msgr2=0x7fcc7415e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc74159ae0 0x7fcc7415e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 -- 172.21.15.158:0/2970377295 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc740c95c0 msgr2=0x7fcc740c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc740c95c0 0x7fcc740c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 -- 172.21.15.158:0/2970377295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7414a830 con 0x7fcc64005210 2023-10-16T11:01:24.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc78d28700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc74148430 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fcc680119d0 tx=0x7fcc68011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc717fa700 1 -- 172.21.15.158:0/2970377295 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6800f8e0 con 0x7fcc64005210 2023-10-16T11:01:24.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc717fa700 1 -- 172.21.15.158:0/2970377295 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6800fa40 con 0x7fcc64005210 2023-10-16T11:01:24.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7414ab10 con 0x7fcc64005210 2023-10-16T11:01:24.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc717fa700 1 -- 172.21.15.158:0/2970377295 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6800fdd0 con 0x7fcc64005210 2023-10-16T11:01:24.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.458+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7414b0d0 con 0x7fcc64005210 2023-10-16T11:01:24.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.459+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc740cea50 con 0x7fcc64005210 2023-10-16T11:01:24.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.460+0000 7fcc717fa700 1 -- 172.21.15.158:0/2970377295 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc68014440 con 0x7fcc64005210 2023-10-16T11:01:24.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.460+0000 7fcc717fa700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcc5806b020 0x7fcc5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:24.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.460+0000 7fcc73fff700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcc5806b020 0x7fcc5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:24.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.460+0000 7fcc717fa700 1 -- 172.21.15.158:0/2970377295 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fcc6808a060 con 0x7fcc64005210 2023-10-16T11:01:24.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.461+0000 7fcc73fff700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcc5806b020 0x7fcc5806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc60000c00 tx=0x7fcc60009040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:24.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.465+0000 7fcc717fa700 1 -- 172.21.15.158:0/2970377295 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc68054d80 con 0x7fcc64005210 2023-10-16T11:01:24.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.701+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcc740681b0 con 0x7fcc64005210 2023-10-16T11:01:24.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.701+0000 7fcc717fa700 1 -- 172.21.15.158:0/2970377295 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v94) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcc680583a0 con 0x7fcc64005210 2023-10-16T11:01:24.705 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T11:01:24.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcc5806b020 msgr2=0x7fcc5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcc5806b020 0x7fcc5806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc60000c00 tx=0x7fcc60009040 comp rx=0 tx=0).stop 2023-10-16T11:01:24.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 msgr2=0x7fcc74148430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:24.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc74148430 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fcc680119d0 tx=0x7fcc68011d90 comp rx=0 tx=0).stop 2023-10-16T11:01:24.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 shutdown_connections 2023-10-16T11:01:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fcc5806b020 0x7fcc5806d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc740c95c0 0x7fcc740c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc64005210 0x7fcc74148430 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 --2- 172.21.15.158:0/2970377295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc74159ae0 0x7fcc7415e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 >> 172.21.15.158:0/2970377295 conn(0x7fcc740b1e10 msgr2=0x7fcc740afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:24.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 shutdown_connections 2023-10-16T11:01:24.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:24.704+0000 7fcc7af8c700 1 -- 172.21.15.158:0/2970377295 wait complete. 2023-10-16T11:01:24.718 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T11:01:24.718 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T11:01:24.719 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T11:01:24.719 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T11:01:24.719 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T11:01:24.720 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T11:01:24.720 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T11:01:24.720 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T11:01:24.721 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:01:24.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:24.722 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T11:01:24.739 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T11:01:24.739 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T11:01:24.739 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T11:01:24.740 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:01:24.740 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T11:01:24.740 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:01:24.740 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:01:24.741 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:01:24.741 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:01:24.741 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:01:24.741 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:01:24.741 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T11:01:24.741 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T11:01:24.742 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T11:01:24.742 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T11:01:24.742 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 37748sec preferred_lft 37748sec 2023-10-16T11:01:24.742 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T11:01:24.743 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:01:24.743 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T11:01:24.743 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T11:01:24.743 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T11:01:24.743 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:01:24.744 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T11:01:24.744 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:01:24.744 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:01:24.744 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:01:24.744 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T11:01:24.745 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:01:24.745 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:01:24.745 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:01:24.745 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T11:01:24.745 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:01:24.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:24.746 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T11:01:24.804 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T11:01:24.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:24.806 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T11:01:24.865 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T11:01:24.865 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T11:01:24.865 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T11:01:24.866 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T11:01:24.866 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T11:01:24.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:24.867 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T11:01:24.867 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T11:01:24.868 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T11:01:24.868 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T11:01:24.868 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T11:01:24.869 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T11:01:24.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:24.970 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:01:24.987 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:01:24.988 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:01:24.988 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:01:24.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:24.989 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T11:01:25.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:01:25.055 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T11:01:25.114 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:01:25.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:01:25.115 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:01:25.170 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T11:01:25.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:25.171 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:01:25.257 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:01:25.257 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:01:25.289 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:01:25.289 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:01:25.289 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:01:25.289 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:01:25.289 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:01:25.290 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:01:25.290 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:01:25.290 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:01:25.318 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:01:25.319 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:01:25.319 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:01:25.319 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:01:25.319 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:01:25.320 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:01:25.320 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:01:25.320 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:01:25.321 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T11:01:25.321 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T11:01:25.321 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T11:01:25.345 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T11:01:25.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:25.346 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:01:25.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:25.374 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T11:01:25.456 INFO:teuthology.orchestra.run.smithi158.stdout:client38132 2023-10-16T11:01:25.456 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:01:25.457 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:01:25.513 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T11:01:25.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.674+0000 7fc8fa28d700 1 Processor -- start 2023-10-16T11:01:25.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.674+0000 7fc8fa28d700 1 -- start start 2023-10-16T11:01:25.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8fa28d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc8f4159a10 0x7fc8f4071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:25.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8fa28d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f415e220 0x7fc8f406c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:25.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8fa28d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8f406ccc0 0x7fc8f40c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:25.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8fa28d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f406d100 con 0x7fc8f406ccc0 2023-10-16T11:01:25.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8fa28d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f40716b0 con 0x7fc8f4159a10 2023-10-16T11:01:25.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8fa28d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4071990 con 0x7fc8f415e220 2023-10-16T11:01:25.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8f406ccc0 0x7fc8f40c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:25.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8f406ccc0 0x7fc8f40c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34754/0 (socket says 172.21.15.158:34754) 2023-10-16T11:01:25.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 -- 172.21.15.158:0/1639598886 learned_addr learned my addr 172.21.15.158:0/1639598886 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:25.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f37fe700 1 --2- 172.21.15.158:0/1639598886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f415e220 0x7fc8f406c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:25.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 -- 172.21.15.158:0/1639598886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f415e220 msgr2=0x7fc8f406c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:25.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 --2- 172.21.15.158:0/1639598886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f415e220 0x7fc8f406c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f3fff700 1 --2- 172.21.15.158:0/1639598886 >> v2:172.21.15.172:3300/0 conn(0x7fc8f4159a10 0x7fc8f4071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:25.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 -- 172.21.15.158:0/1639598886 >> v2:172.21.15.172:3300/0 conn(0x7fc8f4159a10 msgr2=0x7fc8f4071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:25.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 --2- 172.21.15.158:0/1639598886 >> v2:172.21.15.172:3300/0 conn(0x7fc8f4159a10 0x7fc8f4071010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 -- 172.21.15.158:0/1639598886 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f4071c70 con 0x7fc8f406ccc0 2023-10-16T11:01:25.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f37fe700 1 --2- 172.21.15.158:0/1639598886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f415e220 0x7fc8f406c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:25.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f882a700 1 --2- 172.21.15.158:0/1639598886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f406ccc0 0x7fc8f40c9a70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fc8f406eb10 tx=0x7fc8ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d424800001688f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:25.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f2ffd700 1 -- 172.21.15.158:0/1639598886 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec00e070 con 0x7fc8f406ccc0 2023-10-16T11:01:25.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f2ffd700 1 -- 172.21.15.158:0/1639598886 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec00b9c0 con 0x7fc8f406ccc0 2023-10-16T11:01:25.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.676+0000 7fc8f2ffd700 1 -- 172.21.15.158:0/1639598886 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec00bd70 con 0x7fc8f406ccc0 2023-10-16T11:01:25.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 -- 172.21.15.158:0/1639598886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f406ccc0 msgr2=0x7fc8f40c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:25.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/1639598886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f406ccc0 0x7fc8f40c9a70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fc8f406eb10 tx=0x7fc8ec00b330 comp rx=0 tx=0).stop 2023-10-16T11:01:25.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 -- 172.21.15.158:0/1639598886 shutdown_connections 2023-10-16T11:01:25.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/1639598886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f406ccc0 0x7fc8f40c9a70 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/1639598886 >> v2:172.21.15.172:3300/0 conn(0x7fc8f4159a10 0x7fc8f4071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/1639598886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f415e220 0x7fc8f406c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 -- 172.21.15.158:0/1639598886 >> 172.21.15.158:0/1639598886 conn(0x7fc8f40cbd30 msgr2=0x7fc8f40cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:25.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 -- 172.21.15.158:0/1639598886 shutdown_connections 2023-10-16T11:01:25.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.677+0000 7fc8fa28d700 1 -- 172.21.15.158:0/1639598886 wait complete. 2023-10-16T11:01:25.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.678+0000 7fc8fa28d700 1 Processor -- start 2023-10-16T11:01:25.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.678+0000 7fc8fa28d700 1 -- start start 2023-10-16T11:01:25.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8fa28d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f406ccc0 0x7fc8f40b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:25.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8fa28d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4159a10 0x7fc8f415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:25.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8fa28d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f415e220 0x7fc8f41462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:25.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8fa28d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4159df0 con 0x7fc8f415e220 2023-10-16T11:01:25.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8fa28d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f415a040 con 0x7fc8f4159a10 2023-10-16T11:01:25.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8fa28d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f415a320 con 0x7fc8f406ccc0 2023-10-16T11:01:25.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8f882a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f415e220 0x7fc8f41462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:25.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8f882a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f415e220 0x7fc8f41462b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34760/0 (socket says 172.21.15.158:34760) 2023-10-16T11:01:25.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.679+0000 7fc8f882a700 1 -- 172.21.15.158:0/33150516 learned_addr learned my addr 172.21.15.158:0/33150516 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:25.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f3fff700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f406ccc0 0x7fc8f40b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:25.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f37fe700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4159a10 0x7fc8f415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:25.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f882a700 1 -- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f406ccc0 msgr2=0x7fc8f40b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:25.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f882a700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f406ccc0 0x7fc8f40b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f882a700 1 -- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4159a10 msgr2=0x7fc8f415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:25.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f882a700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4159a10 0x7fc8f415b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f882a700 1 -- 172.21.15.158:0/33150516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009b30 con 0x7fc8f415e220 2023-10-16T11:01:25.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f37fe700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4159a10 0x7fc8f415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:25.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f882a700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f415e220 0x7fc8f41462b0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00b370 tx=0x7fc8ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:25.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f0ff9700 1 -- 172.21.15.158:0/33150516 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec004310 con 0x7fc8f415e220 2023-10-16T11:01:25.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f415a600 con 0x7fc8f415e220 2023-10-16T11:01:25.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f0ff9700 1 -- 172.21.15.158:0/33150516 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec004490 con 0x7fc8f415e220 2023-10-16T11:01:25.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8f0ff9700 1 -- 172.21.15.158:0/33150516 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec004840 con 0x7fc8f415e220 2023-10-16T11:01:25.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.680+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f415abc0 con 0x7fc8f415e220 2023-10-16T11:01:25.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.681+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f40681b0 con 0x7fc8f415e220 2023-10-16T11:01:25.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.682+0000 7fc8f0ff9700 1 -- 172.21.15.158:0/33150516 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc8ec01c440 con 0x7fc8f415e220 2023-10-16T11:01:25.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.682+0000 7fc8f0ff9700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc8e006b020 0x7fc8e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:25.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.682+0000 7fc8f0ff9700 1 -- 172.21.15.158:0/33150516 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fc8ec08b4e0 con 0x7fc8f415e220 2023-10-16T11:01:25.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.682+0000 7fc8f3fff700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc8e006b020 0x7fc8e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:25.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.682+0000 7fc8f3fff700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc8e006b020 0x7fc8e006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800ab60 tx=0x7fc8e8008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:25.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.686+0000 7fc8f0ff9700 1 -- 172.21.15.158:0/33150516 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc8ec056070 con 0x7fc8f415e220 2023-10-16T11:01:25.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.908+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 --> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8f4072230 con 0x7fc8e006b020 2023-10-16T11:01:25.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.910+0000 7fc8f0ff9700 1 -- 172.21.15.158:0/33150516 <== mgr.10212 v2:172.21.15.158:6832/1491011698 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc8e800c040 con 0x7fc8e006b020 2023-10-16T11:01:25.911 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:01:25.911 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:01:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T11:01:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:01:25.912 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:01:25.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc8e006b020 msgr2=0x7fc8e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:25.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc8e006b020 0x7fc8e006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800ab60 tx=0x7fc8e8008040 comp rx=0 tx=0).stop 2023-10-16T11:01:25.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f415e220 msgr2=0x7fc8f41462b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:25.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f415e220 0x7fc8f41462b0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00b370 tx=0x7fc8ec004ad0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 shutdown_connections 2023-10-16T11:01:25.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc8e006b020 0x7fc8e006d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f415e220 0x7fc8f41462b0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f406ccc0 0x7fc8f40b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 --2- 172.21.15.158:0/33150516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4159a10 0x7fc8f415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 >> 172.21.15.158:0/33150516 conn(0x7fc8f40cbd30 msgr2=0x7fc8f40c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:25.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 shutdown_connections 2023-10-16T11:01:25.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:25.912+0000 7fc8fa28d700 1 -- 172.21.15.158:0/33150516 wait complete. 2023-10-16T11:01:25.925 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T11:01:26.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.047+0000 7ff6e2aed700 1 Processor -- start 2023-10-16T11:01:26.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.047+0000 7ff6e2aed700 1 -- start start 2023-10-16T11:01:26.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.047+0000 7ff6e2aed700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc0cde10 0x7ff6dc06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.047+0000 7ff6e2aed700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6dc144840 0x7ff6dc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.047+0000 7ff6e2aed700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc06c210 con 0x7ff6dc141920 2023-10-16T11:01:26.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.047+0000 7ff6e2aed700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc141d00 con 0x7ff6dc144840 2023-10-16T11:01:26.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.047+0000 7ff6e2aed700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc06b970 con 0x7ff6dc0cde10 2023-10-16T11:01:26.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6dc144840 0x7ff6dc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6dc144840 0x7ff6dc06da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34768/0 (socket says 172.21.15.158:34768) 2023-10-16T11:01:26.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 -- 172.21.15.158:0/1379159439 learned_addr learned my addr 172.21.15.158:0/1379159439 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:26.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6dbfff700 1 --2- 172.21.15.158:0/1379159439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc0cde10 0x7ff6dc06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 -- 172.21.15.158:0/1379159439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc0cde10 msgr2=0x7ff6dc06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:26.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 --2- 172.21.15.158:0/1379159439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc0cde10 0x7ff6dc06b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 -- 172.21.15.158:0/1379159439 >> v1:172.21.15.172:6789/0 conn(0x7ff6dc141920 legacy=0x7ff6dc0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:26.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 -- 172.21.15.158:0/1379159439 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc06bc50 con 0x7ff6dc144840 2023-10-16T11:01:26.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6e108a700 1 --2- 172.21.15.158:0/1379159439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc144840 0x7ff6dc06da60 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009e50 tx=0x7ff6d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590dfaa0654bf5d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:26.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6db7fe700 1 -- 172.21.15.158:0/1379159439 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d400e070 con 0x7ff6dc144840 2023-10-16T11:01:26.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6db7fe700 1 -- 172.21.15.158:0/1379159439 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d4003a20 con 0x7ff6dc144840 2023-10-16T11:01:26.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.048+0000 7ff6db7fe700 1 -- 172.21.15.158:0/1379159439 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4003dd0 con 0x7ff6dc144840 2023-10-16T11:01:26.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1379159439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc144840 msgr2=0x7ff6dc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1379159439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc144840 0x7ff6dc06da60 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009e50 tx=0x7ff6d400b330 comp rx=0 tx=0).stop 2023-10-16T11:01:26.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1379159439 shutdown_connections 2023-10-16T11:01:26.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1379159439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc144840 0x7ff6dc06da60 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1379159439 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc0cde10 0x7ff6dc06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1379159439 >> 172.21.15.158:0/1379159439 conn(0x7ff6dc0cc5b0 msgr2=0x7ff6dc0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:26.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1379159439 shutdown_connections 2023-10-16T11:01:26.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.049+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1379159439 wait complete. 2023-10-16T11:01:26.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.050+0000 7ff6e2aed700 1 Processor -- start 2023-10-16T11:01:26.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.050+0000 7ff6e2aed700 1 -- start start 2023-10-16T11:01:26.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e2aed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc0cde10 0x7ff6dc1e39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e2aed700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6dc141920 0x7ff6dc0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e2aed700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc144840 0x7ff6dc13d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e2aed700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0c8c10 con 0x7ff6dc0cde10 2023-10-16T11:01:26.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e2aed700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0b4860 con 0x7ff6dc141920 2023-10-16T11:01:26.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e2aed700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0b4e60 con 0x7ff6dc144840 2023-10-16T11:01:26.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc0cde10 0x7ff6dc1e39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc0cde10 0x7ff6dc1e39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34770/0 (socket says 172.21.15.158:34770) 2023-10-16T11:01:26.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 -- 172.21.15.158:0/1154436647 learned_addr learned my addr 172.21.15.158:0/1154436647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:26.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e108a700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc144840 0x7ff6dc13d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6dbfff700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6dc141920 0x7ff6dc0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 -- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc144840 msgr2=0x7ff6dc13d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc144840 0x7ff6dc13d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 -- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6dc141920 msgr2=0x7ff6dc0b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6dc141920 0x7ff6dc0b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 -- 172.21.15.158:0/1154436647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009b30 con 0x7ff6dc0cde10 2023-10-16T11:01:26.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6dbfff700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6dc141920 0x7ff6dc0b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:01:26.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.051+0000 7ff6e0889700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc0cde10 0x7ff6dc1e39b0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c4d0 tx=0x7ff6d000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:26.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.052+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d00054e0 con 0x7ff6dc0cde10 2023-10-16T11:01:26.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.052+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d0003680 con 0x7ff6dc0cde10 2023-10-16T11:01:26.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.052+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc0b3280 con 0x7ff6dc0cde10 2023-10-16T11:01:26.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.052+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0003a30 con 0x7ff6dc0cde10 2023-10-16T11:01:26.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.052+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc0b3b20 con 0x7ff6dc0cde10 2023-10-16T11:01:26.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.053+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc0681b0 con 0x7ff6dc0cde10 2023-10-16T11:01:26.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.053+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff6d0003b90 con 0x7ff6dc0cde10 2023-10-16T11:01:26.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.053+0000 7ff6d97fa700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff6c006b020 0x7ff6c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.054+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7ff6d0088f20 con 0x7ff6dc0cde10 2023-10-16T11:01:26.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.054+0000 7ff6dbfff700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff6c006b020 0x7ff6c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.054+0000 7ff6dbfff700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff6c006b020 0x7ff6c006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8006fd0 tx=0x7ff6c8008040 comp rx=0 tx=0).ready entity=mgr.10212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:26.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.058+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6d0053bb0 con 0x7ff6dc0cde10 2023-10-16T11:01:26.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.312+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff6dc0ca650 con 0x7ff6dc0cde10 2023-10-16T11:01:26.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.396+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v108) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff6d0017020 con 0x7ff6dc0cde10 2023-10-16T11:01:26.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.401+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff6c006b020 msgr2=0x7ff6c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.401+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff6c006b020 0x7ff6c006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8006fd0 tx=0x7ff6c8008040 comp rx=0 tx=0).stop 2023-10-16T11:01:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.401+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc0cde10 msgr2=0x7ff6dc1e39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6d97fa700 1 -- 172.21.15.158:0/1154436647 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 108) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff6d0087130 con 0x7ff6dc0cde10 2023-10-16T11:01:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc0cde10 0x7ff6dc1e39b0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c4d0 tx=0x7ff6d000c890 comp rx=0 tx=0).stop 2023-10-16T11:01:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 shutdown_connections 2023-10-16T11:01:26.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7ff6c006b020 0x7ff6c006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6dc144840 0x7ff6dc13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6dc0cde10 0x7ff6dc1e39b0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 --2- 172.21.15.158:0/1154436647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6dc141920 0x7ff6dc0b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 >> 172.21.15.158:0/1154436647 conn(0x7ff6dc0cc5b0 msgr2=0x7ff6dc0cb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:26.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 shutdown_connections 2023-10-16T11:01:26.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.402+0000 7ff6e2aed700 1 -- 172.21.15.158:0/1154436647 wait complete. 2023-10-16T11:01:26.416 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T11:01:26.539 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:26.538+0000 7f8b0c548200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:01:26.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.543+0000 7f63aa2e0700 1 Processor -- start 2023-10-16T11:01:26.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.544+0000 7f63aa2e0700 1 -- start start 2023-10-16T11:01:26.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.544+0000 7f63aa2e0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63a406d5d0 0x7f63a4070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.544+0000 7f63aa2e0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a415dd70 con 0x7f63a40cbaf0 2023-10-16T11:01:26.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.544+0000 7f63aa2e0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a4070a00 con 0x7f63a415d990 2023-10-16T11:01:26.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.544+0000 7f63aa2e0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a41616b0 con 0x7f63a406d5d0 2023-10-16T11:01:26.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.544+0000 7f63a37fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63a406d5d0 0x7f63a4070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.544+0000 7f63a887d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f63a40cbaf0 0x7f63a40b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:34984/0 (socket says 172.21.15.158:34984) 2023-10-16T11:01:26.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a887d700 1 -- 172.21.15.158:0/3672186754 learned_addr learned my addr 172.21.15.158:0/3672186754 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:01:26.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a37fe700 1 -- 172.21.15.158:0/3672186754 >> v1:172.21.15.172:6789/0 conn(0x7f63a415d990 legacy=0x7f63a4161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a37fe700 1 -- 172.21.15.158:0/3672186754 >> v1:172.21.15.158:6789/0 conn(0x7f63a40cbaf0 legacy=0x7f63a40b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a37fe700 1 -- 172.21.15.158:0/3672186754 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4070ce0 con 0x7f63a406d5d0 2023-10-16T11:01:26.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a2ffd700 1 -- 172.21.15.158:0/3672186754 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189818932 0 0) 0x7f639c001490 con 0x7f63a40cbaf0 2023-10-16T11:01:26.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a37fe700 1 --2- 172.21.15.158:0/3672186754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a406d5d0 0x7f63a4070300 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f6398009840 tx=0x7f6398009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d976d7f8ad09432 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:26.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a2ffd700 1 -- 172.21.15.158:0/3672186754 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398011830 con 0x7f63a406d5d0 2023-10-16T11:01:26.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a2ffd700 1 -- 172.21.15.158:0/3672186754 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6398011990 con 0x7f63a406d5d0 2023-10-16T11:01:26.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.545+0000 7f63a2ffd700 1 -- 172.21.15.158:0/3672186754 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398011d20 con 0x7f63a406d5d0 2023-10-16T11:01:26.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.546+0000 7f63aa2e0700 1 -- 172.21.15.158:0/3672186754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a406d5d0 msgr2=0x7f63a4070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.546+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/3672186754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a406d5d0 0x7f63a4070300 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f6398009840 tx=0x7f6398009b50 comp rx=0 tx=0).stop 2023-10-16T11:01:26.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.546+0000 7f63aa2e0700 1 -- 172.21.15.158:0/3672186754 shutdown_connections 2023-10-16T11:01:26.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.546+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/3672186754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a406d5d0 0x7f63a4070300 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.546+0000 7f63aa2e0700 1 -- 172.21.15.158:0/3672186754 >> 172.21.15.158:0/3672186754 conn(0x7f63a40cdf20 msgr2=0x7f63a40ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.546+0000 7f63aa2e0700 1 -- 172.21.15.158:0/3672186754 shutdown_connections 2023-10-16T11:01:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.546+0000 7f63aa2e0700 1 -- 172.21.15.158:0/3672186754 wait complete. 2023-10-16T11:01:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.547+0000 7f63aa2e0700 1 Processor -- start 2023-10-16T11:01:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.547+0000 7f63aa2e0700 1 -- start start 2023-10-16T11:01:26.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63aa2e0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a406d5d0 0x7f63a4156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63aa2e0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a40cbaf0 0x7f63a40c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63aa2e0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a415d990 0x7f63a415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63aa2e0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a41f41f0 con 0x7f63a40cbaf0 2023-10-16T11:01:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63aa2e0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40c7c80 con 0x7f63a415d990 2023-10-16T11:01:26.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63aa2e0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a41f3670 con 0x7f63a406d5d0 2023-10-16T11:01:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a40cbaf0 0x7f63a40c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a40cbaf0 0x7f63a40c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34812/0 (socket says 172.21.15.158:34812) 2023-10-16T11:01:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 -- 172.21.15.158:0/701836454 learned_addr learned my addr 172.21.15.158:0/701836454 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a887d700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a415d990 0x7f63a415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a3fff700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a406d5d0 0x7f63a4156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 -- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a406d5d0 msgr2=0x7f63a4156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a406d5d0 0x7f63a4156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 -- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a415d990 msgr2=0x7f63a415ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a415d990 0x7f63a415ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.548+0000 7f63a37fe700 1 -- 172.21.15.158:0/701836454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63980094a0 con 0x7f63a40cbaf0 2023-10-16T11:01:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.549+0000 7f63a37fe700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a40cbaf0 0x7f63a40c96e0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f63980090b0 tx=0x7f6398004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.549+0000 7f63a0ff9700 1 -- 172.21.15.158:0/701836454 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63980045d0 con 0x7f63a40cbaf0 2023-10-16T11:01:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.549+0000 7f63a0ff9700 1 -- 172.21.15.158:0/701836454 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639801e070 con 0x7f63a40cbaf0 2023-10-16T11:01:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.549+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a41f3950 con 0x7f63a40cbaf0 2023-10-16T11:01:26.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.549+0000 7f63a0ff9700 1 -- 172.21.15.158:0/701836454 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63980038e0 con 0x7f63a40cbaf0 2023-10-16T11:01:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.549+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a41f3f10 con 0x7f63a40cbaf0 2023-10-16T11:01:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.550+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a40c9f60 con 0x7f63a40cbaf0 2023-10-16T11:01:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.550+0000 7f63a0ff9700 1 -- 172.21.15.158:0/701836454 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 108) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f6398003a40 con 0x7f63a40cbaf0 2023-10-16T11:01:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.551+0000 7f63a0ff9700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 0x7f639006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.551+0000 7f63a3fff700 1 -- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 msgr2=0x7f639006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:26.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.551+0000 7f63a0ff9700 1 -- 172.21.15.158:0/701836454 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f639808ab60 con 0x7f63a40cbaf0 2023-10-16T11:01:26.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.551+0000 7f63a3fff700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 0x7f639006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:26.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:26.553+0000 7f24167e5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:01:26.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.555+0000 7f63a0ff9700 1 -- 172.21.15.158:0/701836454 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63a40c9f60 con 0x7f63a40cbaf0 2023-10-16T11:01:26.606 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:26.605+0000 7f8b0c548200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:01:26.622 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:26.621+0000 7f24167e5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:01:26.678 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:26.676+0000 7f8b0c548200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:01:26.696 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:26.695+0000 7f24167e5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:01:26.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.752+0000 7f63a3fff700 1 -- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 msgr2=0x7f639006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:26.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.752+0000 7f63a3fff700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 0x7f639006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:26.754 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:26.753+0000 7f8b0c548200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:01:26.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.756+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f63a40c9f60 con 0x7f63a40cbaf0 2023-10-16T11:01:26.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.760+0000 7f63a0ff9700 1 -- 172.21.15.158:0/701836454 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f63a40c9f60 con 0x7f63a40cbaf0 2023-10-16T11:01:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.762+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 msgr2=0x7f639006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.762+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 0x7f639006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.762+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a40cbaf0 msgr2=0x7f63a40c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.762+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a40cbaf0 0x7f63a40c96e0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f63980090b0 tx=0x7f6398004750 comp rx=0 tx=0).stop 2023-10-16T11:01:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 shutdown_connections 2023-10-16T11:01:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f639006b070 0x7f639006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a415d990 0x7f63a415ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a406d5d0 0x7f63a4156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 --2- 172.21.15.158:0/701836454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a40cbaf0 0x7f63a40c96e0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 >> 172.21.15.158:0/701836454 conn(0x7f63a40cdf20 msgr2=0x7f63a40ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 shutdown_connections 2023-10-16T11:01:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.763+0000 7f63aa2e0700 1 -- 172.21.15.158:0/701836454 wait complete. 2023-10-16T11:01:26.776 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T11:01:26.777 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:26.776+0000 7f24167e5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:01:26.895 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:26.894+0000 7f8b0c548200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:01:26.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.903+0000 7f79cd6fb700 1 Processor -- start 2023-10-16T11:01:26.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.903+0000 7f79cd6fb700 1 -- start start 2023-10-16T11:01:26.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79cd6fb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f79c80cf190 0x7f79c806d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79cd6fb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f79c806d950 0x7f79c80c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79cd6fb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c80cf930 con 0x7f79c8151220 2023-10-16T11:01:26.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79cd6fb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80cf650 con 0x7f79c80cf190 2023-10-16T11:01:26.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79cd6fb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80c51b0 con 0x7f79c806d950 2023-10-16T11:01:26.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f79c80cf190 0x7f79c806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f79c80cf190 0x7f79c806d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34838/0 (socket says 172.21.15.158:34838) 2023-10-16T11:01:26.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 -- 172.21.15.158:0/3941324941 learned_addr learned my addr 172.21.15.158:0/3941324941 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:26.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c77fe700 1 --2- 172.21.15.158:0/3941324941 >> v2:172.21.15.172:3300/0 conn(0x7f79c806d950 0x7f79c80c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 -- 172.21.15.158:0/3941324941 >> v2:172.21.15.172:3300/0 conn(0x7f79c806d950 msgr2=0x7f79c80c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:26.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 --2- 172.21.15.158:0/3941324941 >> v2:172.21.15.172:3300/0 conn(0x7f79c806d950 0x7f79c80c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 -- 172.21.15.158:0/3941324941 >> v1:172.21.15.172:6789/0 conn(0x7f79c8151220 legacy=0x7f79c80665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 -- 172.21.15.158:0/3941324941 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80c5490 con 0x7f79c80cf190 2023-10-16T11:01:26.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.904+0000 7f79c67fc700 1 --2- 172.21.15.158:0/3941324941 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c80cf190 0x7f79c806d410 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00bb90 tx=0x7f79bc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a49b19dcf6c07 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:26.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79c5ffb700 1 -- 172.21.15.158:0/3941324941 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00dcf0 con 0x7f79c80cf190 2023-10-16T11:01:26.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79c5ffb700 1 -- 172.21.15.158:0/3941324941 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79bc00de50 con 0x7f79c80cf190 2023-10-16T11:01:26.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79c5ffb700 1 -- 172.21.15.158:0/3941324941 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79bc010690 con 0x7f79c80cf190 2023-10-16T11:01:26.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3941324941 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c80cf190 msgr2=0x7f79c806d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3941324941 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c80cf190 0x7f79c806d410 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00bb90 tx=0x7f79bc00bea0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3941324941 shutdown_connections 2023-10-16T11:01:26.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3941324941 >> v2:172.21.15.172:3300/0 conn(0x7f79c806d950 0x7f79c80c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3941324941 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c80cf190 0x7f79c806d410 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3941324941 >> 172.21.15.158:0/3941324941 conn(0x7f79c80cc5b0 msgr2=0x7f79c80cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:26.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3941324941 shutdown_connections 2023-10-16T11:01:26.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.905+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3941324941 wait complete. 2023-10-16T11:01:26.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 Processor -- start 2023-10-16T11:01:26.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 -- start start 2023-10-16T11:01:26.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79c806d950 0x7f79c8145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c80cf190 0x7f79c8152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c8151220 0x7f79c8159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c8145b80 con 0x7f79c8151220 2023-10-16T11:01:26.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c8154880 con 0x7f79c80cf190 2023-10-16T11:01:26.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.907+0000 7f79cd6fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c81537e0 con 0x7f79c806d950 2023-10-16T11:01:26.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c8151220 0x7f79c8159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c8151220 0x7f79c8159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34846/0 (socket says 172.21.15.158:34846) 2023-10-16T11:01:26.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 -- 172.21.15.158:0/3977915944 learned_addr learned my addr 172.21.15.158:0/3977915944 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:26.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c6ffd700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79c806d950 0x7f79c8145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c67fc700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c80cf190 0x7f79c8152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:26.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 -- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79c806d950 msgr2=0x7f79c8145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79c806d950 0x7f79c8145640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 -- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c80cf190 msgr2=0x7f79c8152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:26.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c80cf190 0x7f79c8152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:26.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 -- 172.21.15.158:0/3977915944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc00b7f0 con 0x7f79c8151220 2023-10-16T11:01:26.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79c77fe700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c8151220 0x7f79c8159420 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f79c000efc0 tx=0x7f79c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:26.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79af7fe700 1 -- 172.21.15.158:0/3977915944 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c0009ea0 con 0x7f79c8151220 2023-10-16T11:01:26.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79af7fe700 1 -- 172.21.15.158:0/3977915944 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c000f040 con 0x7f79c8151220 2023-10-16T11:01:26.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c8153ac0 con 0x7f79c8151220 2023-10-16T11:01:26.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79af7fe700 1 -- 172.21.15.158:0/3977915944 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c000ced0 con 0x7f79c8151220 2023-10-16T11:01:26.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.908+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c8154080 con 0x7f79c8151220 2023-10-16T11:01:26.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.909+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c815e120 con 0x7f79c8151220 2023-10-16T11:01:26.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.910+0000 7f79af7fe700 1 -- 172.21.15.158:0/3977915944 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 108) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f79c0014480 con 0x7f79c8151220 2023-10-16T11:01:26.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.910+0000 7f79af7fe700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 0x7f79b406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:26.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.910+0000 7f79c6ffd700 1 -- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 msgr2=0x7f79b406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:26.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.910+0000 7f79af7fe700 1 -- 172.21.15.158:0/3977915944 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f79c008aa90 con 0x7f79c8151220 2023-10-16T11:01:26.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.910+0000 7f79c6ffd700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 0x7f79b406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:26.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:26.914+0000 7f79af7fe700 1 -- 172.21.15.158:0/3977915944 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79c00556f0 con 0x7f79c8151220 2023-10-16T11:01:26.928 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:26.927+0000 7f24167e5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:01:27.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.111+0000 7f79c6ffd700 1 -- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 msgr2=0x7f79b406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:27.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.111+0000 7f79c6ffd700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 0x7f79b406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:27.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.114+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f79c806d410 con 0x7f79c8151220 2023-10-16T11:01:27.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.117+0000 7f79af7fe700 1 -- 172.21.15.158:0/3977915944 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f79c00180c0 con 0x7f79c8151220 2023-10-16T11:01:27.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 msgr2=0x7f79b406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:27.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 0x7f79b406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c8151220 msgr2=0x7f79c8159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:27.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c8151220 0x7f79c8159420 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f79c000efc0 tx=0x7f79c000c3a0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 shutdown_connections 2023-10-16T11:01:27.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7f79b406b070 0x7f79b406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c8151220 0x7f79c8159420 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79c806d950 0x7f79c8145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 --2- 172.21.15.158:0/3977915944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79c80cf190 0x7f79c8152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 >> 172.21.15.158:0/3977915944 conn(0x7f79c80cc5b0 msgr2=0x7f79c806eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:27.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 shutdown_connections 2023-10-16T11:01:27.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.119+0000 7f79cd6fb700 1 -- 172.21.15.158:0/3977915944 wait complete. 2023-10-16T11:01:27.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:27.134 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T11:01:27.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:01:27.156 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-10-16T11:01:27.216 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2023-10-16T11:01:27.219 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:27.218+0000 7f8b0c548200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:01:27.276 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:27.275+0000 7f24167e5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:01:27.334 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:27.333+0000 7f8b0c548200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:01:27.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.385+0000 7fc7178d0700 1 Processor -- start 2023-10-16T11:01:27.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.385+0000 7fc7178d0700 1 -- start start 2023-10-16T11:01:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc7178d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc71015ddc0 0x7fc7100cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc7178d0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7100cf120 0x7fc7100adc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc7178d0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7100ad800 con 0x7fc7100d0dc0 2023-10-16T11:01:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc7178d0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71015e1a0 con 0x7fc7100cf120 2023-10-16T11:01:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc7178d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100cc3a0 con 0x7fc71015ddc0 2023-10-16T11:01:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc715e6d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7100cf120 0x7fc7100adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc715e6d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7100cf120 0x7fc7100adc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34882/0 (socket says 172.21.15.158:34882) 2023-10-16T11:01:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc714e6b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc7100d0dc0 0x7fc7100cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:35056/0 (socket says 172.21.15.158:35056) 2023-10-16T11:01:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc715e6d700 1 -- 172.21.15.158:0/2859527269 learned_addr learned my addr 172.21.15.158:0/2859527269 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc71566c700 1 --2- 172.21.15.158:0/2859527269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc71015ddc0 0x7fc7100cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc715e6d700 1 -- 172.21.15.158:0/2859527269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc71015ddc0 msgr2=0x7fc7100cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.386+0000 7fc715e6d700 1 --2- 172.21.15.158:0/2859527269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc71015ddc0 0x7fc7100cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc715e6d700 1 -- 172.21.15.158:0/2859527269 >> v1:172.21.15.158:6789/0 conn(0x7fc7100d0dc0 legacy=0x7fc7100cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc707fff700 1 -- 172.21.15.158:0/2859527269 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754899313 0 0) 0x7fc7100ad800 con 0x7fc7100d0dc0 2023-10-16T11:01:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc715e6d700 1 -- 172.21.15.158:0/2859527269 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7100cc0c0 con 0x7fc7100cf120 2023-10-16T11:01:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc715e6d700 1 --2- 172.21.15.158:0/2859527269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100cf120 0x7fc7100adc90 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fc708009e50 tx=0x7fc70800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13470e336ececad0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc707fff700 1 -- 172.21.15.158:0/2859527269 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800e070 con 0x7fc7100cf120 2023-10-16T11:01:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc707fff700 1 -- 172.21.15.158:0/2859527269 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70800b910 con 0x7fc7100cf120 2023-10-16T11:01:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc707fff700 1 -- 172.21.15.158:0/2859527269 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800bcc0 con 0x7fc7100cf120 2023-10-16T11:01:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc7178d0700 1 -- 172.21.15.158:0/2859527269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100cf120 msgr2=0x7fc7100adc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc7178d0700 1 --2- 172.21.15.158:0/2859527269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100cf120 0x7fc7100adc90 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fc708009e50 tx=0x7fc70800b330 comp rx=0 tx=0).stop 2023-10-16T11:01:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc7178d0700 1 -- 172.21.15.158:0/2859527269 shutdown_connections 2023-10-16T11:01:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc7178d0700 1 --2- 172.21.15.158:0/2859527269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100cf120 0x7fc7100adc90 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc7178d0700 1 --2- 172.21.15.158:0/2859527269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc71015ddc0 0x7fc7100cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.387+0000 7fc7178d0700 1 -- 172.21.15.158:0/2859527269 >> 172.21.15.158:0/2859527269 conn(0x7fc710099a10 msgr2=0x7fc7100d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.388+0000 7fc7178d0700 1 -- 172.21.15.158:0/2859527269 shutdown_connections 2023-10-16T11:01:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.388+0000 7fc7178d0700 1 -- 172.21.15.158:0/2859527269 wait complete. 2023-10-16T11:01:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.389+0000 7fc7178d0700 1 Processor -- start 2023-10-16T11:01:27.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.389+0000 7fc7178d0700 1 -- start start 2023-10-16T11:01:27.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.389+0000 7fc7178d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7100cf120 0x7fc7101f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc7178d0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100d0dc0 0x7fc710149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc7178d0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc71015ddc0 0x7fc7100b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc7178d0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7101f4290 con 0x7fc7100d0dc0 2023-10-16T11:01:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc7178d0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71014c920 con 0x7fc71015ddc0 2023-10-16T11:01:27.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc7178d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100c6960 con 0x7fc7100cf120 2023-10-16T11:01:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100d0dc0 0x7fc710149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100d0dc0 0x7fc710149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34886/0 (socket says 172.21.15.158:34886) 2023-10-16T11:01:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 -- 172.21.15.158:0/2376340790 learned_addr learned my addr 172.21.15.158:0/2376340790 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7100cf120 0x7fc7101f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:27.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc715e6d700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc71015ddc0 0x7fc7100b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7100cf120 msgr2=0x7fc7101f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7100cf120 0x7fc7101f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc71015ddc0 msgr2=0x7fc7100b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc71015ddc0 0x7fc7100b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 -- 172.21.15.158:0/2376340790 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708009b30 con 0x7fc7100d0dc0 2023-10-16T11:01:27.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc715e6d700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc71015ddc0 0x7fc7100b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc714e6b700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100d0dc0 0x7fc710149f80 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fc70000c580 tx=0x7fc70000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700004c80 con 0x7fc7100d0dc0 2023-10-16T11:01:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc700004e00 con 0x7fc7100d0dc0 2023-10-16T11:01:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700007310 con 0x7fc7100d0dc0 2023-10-16T11:01:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7100c7f20 con 0x7fc7100d0dc0 2023-10-16T11:01:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.390+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7100c84e0 con 0x7fc7100d0dc0 2023-10-16T11:01:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.391+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71014cad0 con 0x7fc7100d0dc0 2023-10-16T11:01:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.392+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 108) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc700007470 con 0x7fc7100d0dc0 2023-10-16T11:01:27.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.393+0000 7fc705ffb700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 0x7fc6f406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.393+0000 7fc71566c700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 msgr2=0x7fc6f406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.393+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fc700089cf0 con 0x7fc7100d0dc0 2023-10-16T11:01:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.393+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 0x7fc6f406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:01:27.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.401+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7000548e0 con 0x7fc7100d0dc0 2023-10-16T11:01:27.405 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:27.402+0000 7f24167e5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:01:27.407 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:27.406+0000 7f8b0c548200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:01:27.480 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:27.479+0000 7f24167e5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:01:27.489 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:27.487+0000 7f8b0c548200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:01:27.561 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:27.560+0000 7f24167e5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:01:27.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.594+0000 7fc71566c700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 msgr2=0x7fc6f406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:27.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.594+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 0x7fc6f406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:01:27.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.627+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 --> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7100ad490 con 0x7fc6f406b070 2023-10-16T11:01:27.856 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:27.855+0000 7f8b0c548200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:01:27.932 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:27.931+0000 7f24167e5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:01:27.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.995+0000 7fc71566c700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 msgr2=0x7fc6f406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:27.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:27.995+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 0x7fc6f406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T11:01:28.422 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:28.421+0000 7f8b0c548200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:01:28.492 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:28.490+0000 7f8b0c548200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:01:28.503 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:28.502+0000 7f24167e5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:01:28.573 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:28.572+0000 7f24167e5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:01:28.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:28.796+0000 7fc71566c700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 msgr2=0x7fc6f406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:28.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:28.796+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 0x7fc6f406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T11:01:29.120 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:29.119+0000 7f8b0c548200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:29.197 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:29.196+0000 7f8b0c548200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:01:29.205 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:29.204+0000 7f24167e5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:29.283 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:29.282+0000 7f24167e5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:01:29.341 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:29.340+0000 7f8b0c548200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:29.431 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:29.428+0000 7f24167e5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:01:29.571 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:29.570+0000 7f8b0c548200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:01:29.645 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:29.643+0000 7f8b0c548200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:01:29.659 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:29.657+0000 7f24167e5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:01:29.733 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:29.732+0000 7f24167e5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:01:29.914 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:29.913+0000 7f8b0c548200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:01:29.999 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:29.998+0000 7f24167e5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:01:30.136 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:30.134+0000 7f8b0c548200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:01:30.223 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:30.222+0000 7f24167e5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:01:30.285 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:30.283+0000 7f8b0c548200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:01:30.374 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:30.373+0000 7f24167e5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:01:30.397 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:30.396+0000 7f8b0c548200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:01:30.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:30.397+0000 7fc71566c700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 msgr2=0x7fc6f406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1491011698 2023-10-16T11:01:30.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:30.397+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 0x7fc6f406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T11:01:30.488 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:30.487+0000 7f24167e5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:01:30.553 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:30.551+0000 7f8b0c548200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:01:30.646 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:30.644+0000 7f24167e5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:01:30.814 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:30.812+0000 7f8b0c548200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:01:30.911 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:30.910+0000 7f24167e5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:01:30.974 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:30.972+0000 7f8b0c548200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:01:31.074 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.073+0000 7f24167e5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:01:31.497 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:01:31.496+0000 7f8b0c548200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:01:31.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:31.567+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 109) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc70001ad90 con 0x7fc7100d0dc0 2023-10-16T11:01:31.611 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.610+0000 7f24167e5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:01:31.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:31.632+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 110) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fc700054cb0 con 0x7fc7100d0dc0 2023-10-16T11:01:31.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:31.632+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 msgr2=0x7fc6f406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:01:31.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:31.632+0000 7fc705ffb700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/1491011698,v1:172.21.15.158:6833/1491011698] conn(0x7fc6f406b070 0x7fc6f406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:31.780 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f23897c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.780 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f23897c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.780 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f23897c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f23897c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f23897c0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f238c7c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.781 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f238c7c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f238c7c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f238c7c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:31.782 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:01:31.778+0000 7f238c7c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:01:32.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.639+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 111) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc700057ec0 con 0x7fc7100d0dc0 2023-10-16T11:01:32.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.639+0000 7fc705ffb700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7fc6f401d250 0x7fc6f4075aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:32.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.639+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 --> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6f4071580 con 0x7fc6f401d250 2023-10-16T11:01:32.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.640+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7fc6f401d250 0x7fc6f4075aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:32.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.643+0000 7fc71566c700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7fc6f401d250 0x7fc6f4075aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc7101574e0 tx=0x7fc70c009210 comp rx=0 tx=0).ready entity=mgr.10644 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:32.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.700+0000 7fc705ffb700 1 -- 172.21.15.158:0/2376340790 <== mgr.10644 v2:172.21.15.158:6832/129037312 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc6f4071580 con 0x7fc6f401d250 2023-10-16T11:01:32.702 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-10-16T11:01:32.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7fc6f401d250 msgr2=0x7fc6f4075aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:32.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7fc6f401d250 0x7fc6f4075aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc7101574e0 tx=0x7fc70c009210 comp rx=0 tx=0).stop 2023-10-16T11:01:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100d0dc0 msgr2=0x7fc710149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 --2- 172.21.15.158:0/2376340790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7100d0dc0 0x7fc710149f80 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fc70000c580 tx=0x7fc70000c890 comp rx=0 tx=0).stop 2023-10-16T11:01:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc715e6d700 1 -- 172.21.15.158:0/2376340790 reap_dead start 2023-10-16T11:01:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 shutdown_connections 2023-10-16T11:01:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 >> 172.21.15.158:0/2376340790 conn(0x7fc710099a10 msgr2=0x7fc71015c360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:32.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 shutdown_connections 2023-10-16T11:01:32.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.703+0000 7fc7178d0700 1 -- 172.21.15.158:0/2376340790 wait complete. 2023-10-16T11:01:32.716 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-10-16T11:01:32.717 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2023-10-16T11:01:32.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.880+0000 7f17df3c6700 1 Processor -- start 2023-10-16T11:01:32.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.880+0000 7f17df3c6700 1 -- start start 2023-10-16T11:01:32.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17df3c6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17d8144840 0x7f17d806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:32.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17df3c6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d806c210 con 0x7f17d8141920 2023-10-16T11:01:32.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17df3c6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d8141d00 con 0x7f17d80cde10 2023-10-16T11:01:32.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17df3c6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d806b970 con 0x7f17d8144840 2023-10-16T11:01:32.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17dd963700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17d8144840 0x7f17d806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:32.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17dd963700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17d8144840 0x7f17d806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35070/0 (socket says 172.21.15.158:35070) 2023-10-16T11:01:32.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17dd963700 1 -- 172.21.15.158:0/6222684 learned_addr learned my addr 172.21.15.158:0/6222684 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:32.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17dd963700 1 -- 172.21.15.158:0/6222684 >> v1:172.21.15.172:6789/0 conn(0x7f17d80cde10 legacy=0x7f17d806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:01:32.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17dd963700 1 -- 172.21.15.158:0/6222684 >> v1:172.21.15.158:6789/0 conn(0x7f17d8141920 legacy=0x7f17d80cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:32.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17dd963700 1 -- 172.21.15.158:0/6222684 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d806bc50 con 0x7f17d8144840 2023-10-16T11:01:32.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17dd963700 1 --2- 172.21.15.158:0/6222684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8144840 0x7f17d806b2e0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009e50 tx=0x7f17d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d53104b3b5b211f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:32.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17cffff700 1 -- 172.21.15.158:0/6222684 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d000e070 con 0x7f17d8144840 2023-10-16T11:01:32.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17cffff700 1 -- 172.21.15.158:0/6222684 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0003a20 con 0x7f17d8144840 2023-10-16T11:01:32.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.881+0000 7f17cffff700 1 -- 172.21.15.158:0/6222684 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0003dd0 con 0x7f17d8144840 2023-10-16T11:01:32.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.882+0000 7f17df3c6700 1 -- 172.21.15.158:0/6222684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8144840 msgr2=0x7f17d806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:32.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.882+0000 7f17df3c6700 1 --2- 172.21.15.158:0/6222684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8144840 0x7f17d806b2e0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009e50 tx=0x7f17d000b330 comp rx=0 tx=0).stop 2023-10-16T11:01:32.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.882+0000 7f17df3c6700 1 -- 172.21.15.158:0/6222684 shutdown_connections 2023-10-16T11:01:32.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.882+0000 7f17df3c6700 1 --2- 172.21.15.158:0/6222684 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8144840 0x7f17d806b2e0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:32.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.882+0000 7f17df3c6700 1 -- 172.21.15.158:0/6222684 >> 172.21.15.158:0/6222684 conn(0x7f17d80cc5b0 msgr2=0x7f17d80cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:32.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.882+0000 7f17df3c6700 1 -- 172.21.15.158:0/6222684 shutdown_connections 2023-10-16T11:01:32.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.882+0000 7f17df3c6700 1 -- 172.21.15.158:0/6222684 wait complete. 2023-10-16T11:01:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.883+0000 7f17df3c6700 1 Processor -- start 2023-10-16T11:01:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.883+0000 7f17df3c6700 1 -- start start 2023-10-16T11:01:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17df3c6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d80cde10 0x7f17d81f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:32.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17df3c6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8141920 0x7f17d81f5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17df3c6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17d8144840 0x7f17d80b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17df3c6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d81f2ce0 con 0x7f17d8141920 2023-10-16T11:01:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17df3c6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d81f5aa0 con 0x7f17d80cde10 2023-10-16T11:01:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17df3c6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80b41f0 con 0x7f17d8144840 2023-10-16T11:01:32.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8141920 0x7f17d81f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8141920 0x7f17d81f5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35074/0 (socket says 172.21.15.158:35074) 2023-10-16T11:01:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 -- 172.21.15.158:0/2991729389 learned_addr learned my addr 172.21.15.158:0/2991729389 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:01:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dd162700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d80cde10 0x7f17d81f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dd963700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17d8144840 0x7f17d80b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:32.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 -- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17d8144840 msgr2=0x7f17d80b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17d8144840 0x7f17d80b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 -- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d80cde10 msgr2=0x7f17d81f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d80cde10 0x7f17d81f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dc961700 1 -- 172.21.15.158:0/2991729389 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d0009b30 con 0x7f17d8141920 2023-10-16T11:01:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dd162700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d80cde10 0x7f17d81f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.884+0000 7f17dd963700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17d8144840 0x7f17d80b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:01:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.885+0000 7f17dc961700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8141920 0x7f17d81f5560 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f17c800c4d0 tx=0x7f17c800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.885+0000 7f17cdffb700 1 -- 172.21.15.158:0/2991729389 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8011790 con 0x7f17d8141920 2023-10-16T11:01:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.885+0000 7f17cdffb700 1 -- 172.21.15.158:0/2991729389 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c80118f0 con 0x7f17d8141920 2023-10-16T11:01:32.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.885+0000 7f17cdffb700 1 -- 172.21.15.158:0/2991729389 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8011ca0 con 0x7f17d8141920 2023-10-16T11:01:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.885+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d80b44d0 con 0x7f17d8141920 2023-10-16T11:01:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.885+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d80c4d30 con 0x7f17d8141920 2023-10-16T11:01:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.886+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d80681b0 con 0x7f17d8141920 2023-10-16T11:01:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.886+0000 7f17cdffb700 1 -- 172.21.15.158:0/2991729389 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 111) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f17c801c460 con 0x7f17d8141920 2023-10-16T11:01:32.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.887+0000 7f17cdffb700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f17bc06ada0 0x7f17bc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:01:32.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.887+0000 7f17cdffb700 1 -- 172.21.15.158:0/2991729389 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f17c8089590 con 0x7f17d8141920 2023-10-16T11:01:32.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.887+0000 7f17dd162700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f17bc06ada0 0x7f17bc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:01:32.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.887+0000 7f17dd162700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f17bc06ada0 0x7f17bc06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17d4001780 tx=0x7f17d4009040 comp rx=0 tx=0).ready entity=mgr.10644 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:01:32.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:32.891+0000 7f17cdffb700 1 -- 172.21.15.158:0/2991729389 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17c8054520 con 0x7f17d8141920 2023-10-16T11:01:33.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.110+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 --> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f17d806d540 con 0x7f17bc06ada0 2023-10-16T11:01:33.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.150+0000 7f17cdffb700 1 -- 172.21.15.158:0/2991729389 <== mgr.10644 v2:172.21.15.158:6832/129037312 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f17d40050d0 con 0x7f17bc06ada0 2023-10-16T11:01:33.152 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T11:01:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f17bc06ada0 msgr2=0x7f17bc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f17bc06ada0 0x7f17bc06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17d4001780 tx=0x7f17d4009040 comp rx=0 tx=0).stop 2023-10-16T11:01:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8141920 msgr2=0x7f17d81f5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:01:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8141920 0x7f17d81f5560 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f17c800c4d0 tx=0x7f17c800c890 comp rx=0 tx=0).stop 2023-10-16T11:01:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 shutdown_connections 2023-10-16T11:01:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17d8144840 0x7f17d80b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d80cde10 0x7f17d81f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f17bc06ada0 0x7f17bc06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 --2- 172.21.15.158:0/2991729389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d8141920 0x7f17d81f5560 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:01:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 >> 172.21.15.158:0/2991729389 conn(0x7f17d80cc5b0 msgr2=0x7f17d813ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:01:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 shutdown_connections 2023-10-16T11:01:33.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:01:33.153+0000 7f17df3c6700 1 -- 172.21.15.158:0/2991729389 wait complete. 2023-10-16T11:01:33.167 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T11:01:41.705 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:01:41.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:02:00.162 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:02:00.160+0000 7f237b764700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:02:00.162 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:02:00.160+0000 7f237b764700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:02:00.163 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:02:00.160+0000 7f237b764700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:02:00.163 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:02:00.160+0000 7f237b764700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:02:00.163 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:02:00.160+0000 7f237b764700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:02:11.843 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:02:11.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:02:41.981 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:02:41.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:03:12.143 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:03:12.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:03:42.215 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:03:42.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:04:12.279 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:04:12.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:04:42.416 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:04:42.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:05:12.756 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:05:12.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:05:42.892 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:05:42.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:06:13.004 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:06:13.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:06:43.125 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:06:43.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:07:13.246 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:07:13.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:07:43.325 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:07:43.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:08:13.505 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:08:13.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:08:43.652 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:08:43.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:09:13.789 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:09:13.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:09:43.883 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:09:43.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:10:14.031 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:10:14.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:10:44.202 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:10:44.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:11:14.282 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:11:14.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:11:44.420 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:11:44.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:12:14.549 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:12:14.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:12:16.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:12:16.168 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T11:12:16.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_02_00_UTC 2023-10-16T11:12:16.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_03_00_UTC 2023-10-16T11:12:16.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_04_00_UTC 2023-10-16T11:12:16.191 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_05_00_UTC 2023-10-16T11:12:16.192 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_06_00_UTC 2023-10-16T11:12:16.192 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_07_00_UTC 2023-10-16T11:12:16.192 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_08_00_UTC 2023-10-16T11:12:16.192 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_09_00_UTC 2023-10-16T11:12:16.192 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_10_00_UTC 2023-10-16T11:12:16.193 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_11_00_UTC 2023-10-16T11:12:16.193 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_12_00_UTC 2023-10-16T11:12:16.193 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-11_02_00_UTC', 'scheduled-2023-10-16-11_03_00_UTC', 'scheduled-2023-10-16-11_04_00_UTC', 'scheduled-2023-10-16-11_05_00_UTC', 'scheduled-2023-10-16-11_06_00_UTC', 'scheduled-2023-10-16-11_07_00_UTC', 'scheduled-2023-10-16-11_08_00_UTC', 'scheduled-2023-10-16-11_09_00_UTC', 'scheduled-2023-10-16-11_10_00_UTC', 'scheduled-2023-10-16-11_11_00_UTC', 'scheduled-2023-10-16-11_12_00_UTC'] 2023-10-16T11:12:16.193 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T11:12:16.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.364+0000 7f054fcd0700 1 Processor -- start 2023-10-16T11:12:16.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.364+0000 7f054fcd0700 1 -- start start 2023-10-16T11:12:16.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.365+0000 7f054fcd0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0548160d30 0x7f05480b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.365+0000 7f054fcd0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480b7590 0x7f05480d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.365+0000 7f054fcd0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480b7a20 con 0x7f05480d0530 2023-10-16T11:12:16.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.365+0000 7f054fcd0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05480b8910 con 0x7f0548160d30 2023-10-16T11:12:16.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.365+0000 7f054fcd0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05480d0ef0 con 0x7f05480b7590 2023-10-16T11:12:16.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.365+0000 7f054da6c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0548160d30 0x7f05480b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.365+0000 7f054da6c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0548160d30 0x7f05480b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35112/0 (socket says 172.21.15.158:35112) 2023-10-16T11:12:16.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054da6c700 1 -- 172.21.15.158:0/2199357606 learned_addr learned my addr 172.21.15.158:0/2199357606 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:12:16.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054e26d700 1 --2- 172.21.15.158:0/2199357606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480b7590 0x7f05480d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054da6c700 1 -- 172.21.15.158:0/2199357606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480b7590 msgr2=0x7f05480d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:12:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054da6c700 1 --2- 172.21.15.158:0/2199357606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480b7590 0x7f05480d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054da6c700 1 -- 172.21.15.158:0/2199357606 >> v1:172.21.15.172:6789/0 conn(0x7f05480d0530 legacy=0x7f05480d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:12:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054da6c700 1 -- 172.21.15.158:0/2199357606 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480b7c90 con 0x7f0548160d30 2023-10-16T11:12:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054e26d700 1 --2- 172.21.15.158:0/2199357606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480b7590 0x7f05480d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:12:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054da6c700 1 --2- 172.21.15.158:0/2199357606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0548160d30 0x7f05480b8f50 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f05380098f0 tx=0x7f0538009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68c891f8ecd79a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:12:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054ca6a700 1 -- 172.21.15.158:0/2199357606 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538004230 con 0x7f0548160d30 2023-10-16T11:12:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054ca6a700 1 -- 172.21.15.158:0/2199357606 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0538004390 con 0x7f0548160d30 2023-10-16T11:12:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.366+0000 7f054ca6a700 1 -- 172.21.15.158:0/2199357606 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05380036a0 con 0x7f0548160d30 2023-10-16T11:12:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 -- 172.21.15.158:0/2199357606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0548160d30 msgr2=0x7f05480b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 --2- 172.21.15.158:0/2199357606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0548160d30 0x7f05480b8f50 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f05380098f0 tx=0x7f0538009c00 comp rx=0 tx=0).stop 2023-10-16T11:12:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 -- 172.21.15.158:0/2199357606 shutdown_connections 2023-10-16T11:12:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 --2- 172.21.15.158:0/2199357606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480b7590 0x7f05480d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 --2- 172.21.15.158:0/2199357606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0548160d30 0x7f05480b8f50 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 -- 172.21.15.158:0/2199357606 >> 172.21.15.158:0/2199357606 conn(0x7f05480669f0 msgr2=0x7f0548066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:12:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 -- 172.21.15.158:0/2199357606 shutdown_connections 2023-10-16T11:12:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.367+0000 7f054fcd0700 1 -- 172.21.15.158:0/2199357606 wait complete. 2023-10-16T11:12:16.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.368+0000 7f054fcd0700 1 Processor -- start 2023-10-16T11:12:16.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.368+0000 7f054fcd0700 1 -- start start 2023-10-16T11:12:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054fcd0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05480b7590 0x7f05481f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054fcd0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480d0530 0x7f0548157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054fcd0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0548160d30 0x7f05480d27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054fcd0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481f5750 con 0x7f05480b7590 2023-10-16T11:12:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054fcd0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05480b4cd0 con 0x7f0548160d30 2023-10-16T11:12:16.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054fcd0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05481f4ad0 con 0x7f05480d0530 2023-10-16T11:12:16.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05480b7590 0x7f05481f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05480b7590 0x7f05481f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35116/0 (socket says 172.21.15.158:35116) 2023-10-16T11:12:16.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 -- 172.21.15.158:0/4230746830 learned_addr learned my addr 172.21.15.158:0/4230746830 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:12:16.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054d26b700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480d0530 0x7f0548157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054e26d700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0548160d30 0x7f05480d27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 -- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480d0530 msgr2=0x7f0548157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480d0530 0x7f0548157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 -- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0548160d30 msgr2=0x7f05480d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0548160d30 0x7f05480d27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 -- 172.21.15.158:0/4230746830 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0538009550 con 0x7f05480b7590 2023-10-16T11:12:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f054da6c700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05480b7590 0x7f05481f5b70 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f0538009130 tx=0x7f0538004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:12:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.369+0000 7f053e7fc700 1 -- 172.21.15.158:0/4230746830 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053800b660 con 0x7f05480b7590 2023-10-16T11:12:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.370+0000 7f053e7fc700 1 -- 172.21.15.158:0/4230746830 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053800b7c0 con 0x7f05480b7590 2023-10-16T11:12:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.370+0000 7f053e7fc700 1 -- 172.21.15.158:0/4230746830 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053800bb70 con 0x7f05480b7590 2023-10-16T11:12:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.370+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05481f4db0 con 0x7f05480b7590 2023-10-16T11:12:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.370+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05481f5370 con 0x7f05480b7590 2023-10-16T11:12:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.371+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05480681b0 con 0x7f05480b7590 2023-10-16T11:12:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.371+0000 7f053e7fc700 1 -- 172.21.15.158:0/4230746830 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 115) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f053800bcd0 con 0x7f05480b7590 2023-10-16T11:12:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.371+0000 7f053e7fc700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f052c06b070 0x7f052c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.371+0000 7f053e7fc700 1 -- 172.21.15.158:0/4230746830 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f053808b3e0 con 0x7f05480b7590 2023-10-16T11:12:16.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.371+0000 7f054d26b700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f052c06b070 0x7f052c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.372+0000 7f054d26b700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f052c06b070 0x7f052c06d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f054815d470 tx=0x7f0544007ec0 comp rx=0 tx=0).ready entity=mgr.10644 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:12:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.376+0000 7f053e7fc700 1 -- 172.21.15.158:0/4230746830 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f05380560d0 con 0x7f05480b7590 2023-10-16T11:12:16.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.595+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 --> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05480b8250 con 0x7f052c06b070 2023-10-16T11:12:16.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.600+0000 7f053e7fc700 1 -- 172.21.15.158:0/4230746830 <== mgr.10644 v2:172.21.15.158:6832/129037312 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f0544003a90 con 0x7f052c06b070 2023-10-16T11:12:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:12:16.602 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T11:01:32", "first": "2023-10-16T11:02:00", "last": "2023-10-16T11:12:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T11:12:16.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f052c06b070 msgr2=0x7f052c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f052c06b070 0x7f052c06d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f054815d470 tx=0x7f0544007ec0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05480b7590 msgr2=0x7f05481f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05480b7590 0x7f05481f5b70 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f0538009130 tx=0x7f0538004750 comp rx=0 tx=0).stop 2023-10-16T11:12:16.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 shutdown_connections 2023-10-16T11:12:16.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0548160d30 0x7f05480d27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f052c06b070 0x7f052c06d530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05480b7590 0x7f05481f5b70 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 --2- 172.21.15.158:0/4230746830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05480d0530 0x7f0548157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 >> 172.21.15.158:0/4230746830 conn(0x7f05480669f0 msgr2=0x7f05480ca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:12:16.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 shutdown_connections 2023-10-16T11:12:16.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.602+0000 7f054fcd0700 1 -- 172.21.15.158:0/4230746830 wait complete. 2023-10-16T11:12:16.615 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T11:01:32", "first": "2023-10-16T11:02:00", "last": "2023-10-16T11:12:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T11:12:16.616 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T11:01:32', 'first': '2023-10-16T11:02:00', 'last': '2023-10-16T11:12:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-10-16T11:12:16.616 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T11:12:16.617 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T11:12:16.617 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T11:12:16.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.736+0000 7f0b6b1aa700 1 Processor -- start 2023-10-16T11:12:16.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.736+0000 7f0b6b1aa700 1 -- start start 2023-10-16T11:12:16.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b6b1aa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b640cc5d0 0x7f0b6406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b6b1aa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0b6406f340 0x7f0b6406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b6b1aa700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6406fce0 con 0x7f0b6415dfb0 2023-10-16T11:12:16.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b6b1aa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b640cf270 con 0x7f0b6406f340 2023-10-16T11:12:16.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b6b1aa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6406e770 con 0x7f0b640cc5d0 2023-10-16T11:12:16.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0b6406f340 0x7f0b6406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0b6406f340 0x7f0b6406f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35128/0 (socket says 172.21.15.158:35128) 2023-10-16T11:12:16.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 -- 172.21.15.158:0/1085488045 learned_addr learned my addr 172.21.15.158:0/1085488045 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:12:16.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b68f46700 1 --2- 172.21.15.158:0/1085488045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b640cc5d0 0x7f0b6406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 -- 172.21.15.158:0/1085488045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b640cc5d0 msgr2=0x7f0b6406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:12:16.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 --2- 172.21.15.158:0/1085488045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b640cc5d0 0x7f0b6406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 -- 172.21.15.158:0/1085488045 >> v1:172.21.15.172:6789/0 conn(0x7f0b6415dfb0 legacy=0x7f0b640cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 -- 172.21.15.158:0/1085488045 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b640cbe00 con 0x7f0b6406f340 2023-10-16T11:12:16.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b68f46700 1 --2- 172.21.15.158:0/1085488045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b640cc5d0 0x7f0b6406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:12:16.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b69747700 1 --2- 172.21.15.158:0/1085488045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b6406f340 0x7f0b6406f7a0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c009e50 tx=0x7f0b5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a84d052a139441 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:12:16.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.737+0000 7f0b637fe700 1 -- 172.21.15.158:0/1085488045 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c00e070 con 0x7f0b6406f340 2023-10-16T11:12:16.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b637fe700 1 -- 172.21.15.158:0/1085488045 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5c00b9c0 con 0x7f0b6406f340 2023-10-16T11:12:16.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b637fe700 1 -- 172.21.15.158:0/1085488045 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c00bd70 con 0x7f0b6406f340 2023-10-16T11:12:16.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1085488045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b6406f340 msgr2=0x7f0b6406f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1085488045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b6406f340 0x7f0b6406f7a0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c009e50 tx=0x7f0b5c00b330 comp rx=0 tx=0).stop 2023-10-16T11:12:16.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1085488045 shutdown_connections 2023-10-16T11:12:16.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1085488045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b6406f340 0x7f0b6406f7a0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1085488045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b640cc5d0 0x7f0b6406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1085488045 >> 172.21.15.158:0/1085488045 conn(0x7f0b640d02b0 msgr2=0x7f0b640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:12:16.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1085488045 shutdown_connections 2023-10-16T11:12:16.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.738+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1085488045 wait complete. 2023-10-16T11:12:16.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.739+0000 7f0b6b1aa700 1 Processor -- start 2023-10-16T11:12:16.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.739+0000 7f0b6b1aa700 1 -- start start 2023-10-16T11:12:16.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b6b1aa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b6406f340 0x7f0b6415d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b6b1aa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b640cc5d0 0x7f0b64154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b6b1aa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b6415dfb0 0x7f0b640c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b6b1aa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b640b4a30 con 0x7f0b640cc5d0 2023-10-16T11:12:16.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b6b1aa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b640b4c40 con 0x7f0b6406f340 2023-10-16T11:12:16.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b6b1aa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b640b4f20 con 0x7f0b6415dfb0 2023-10-16T11:12:16.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b640cc5d0 0x7f0b64154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b640cc5d0 0x7f0b64154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35132/0 (socket says 172.21.15.158:35132) 2023-10-16T11:12:16.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b63fff700 1 -- 172.21.15.158:0/1870152961 learned_addr learned my addr 172.21.15.158:0/1870152961 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:12:16.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b68f46700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b6406f340 0x7f0b6415d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.740+0000 7f0b69747700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b6415dfb0 0x7f0b640c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b63fff700 1 -- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b6415dfb0 msgr2=0x7f0b640c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b63fff700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b6415dfb0 0x7f0b640c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b63fff700 1 -- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b6406f340 msgr2=0x7f0b6415d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:16.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b63fff700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b6406f340 0x7f0b6415d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:16.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b63fff700 1 -- 172.21.15.158:0/1870152961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c009b30 con 0x7f0b640cc5d0 2023-10-16T11:12:16.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b69747700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b6415dfb0 0x7f0b640c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:12:16.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b63fff700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b640cc5d0 0x7f0b64154440 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000c4b0 tx=0x7f0b5000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:12:16.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b617fa700 1 -- 172.21.15.158:0/1870152961 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b500052d0 con 0x7f0b640cc5d0 2023-10-16T11:12:16.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b617fa700 1 -- 172.21.15.158:0/1870152961 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b50005430 con 0x7f0b640cc5d0 2023-10-16T11:12:16.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b640b5200 con 0x7f0b640cc5d0 2023-10-16T11:12:16.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b617fa700 1 -- 172.21.15.158:0/1870152961 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b500038c0 con 0x7f0b640cc5d0 2023-10-16T11:12:16.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.741+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6415d720 con 0x7f0b640cc5d0 2023-10-16T11:12:16.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.742+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b640681b0 con 0x7f0b640cc5d0 2023-10-16T11:12:16.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.743+0000 7f0b617fa700 1 -- 172.21.15.158:0/1870152961 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 115) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f0b50003a20 con 0x7f0b640cc5d0 2023-10-16T11:12:16.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.744+0000 7f0b617fa700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f0b4806b070 0x7f0b4806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:12:16.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.744+0000 7f0b617fa700 1 -- 172.21.15.158:0/1870152961 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6825+0+0 (secure 0 0 0) 0x7f0b50089480 con 0x7f0b640cc5d0 2023-10-16T11:12:16.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.744+0000 7f0b68f46700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f0b4806b070 0x7f0b4806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:12:16.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.744+0000 7f0b68f46700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f0b4806b070 0x7f0b4806d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b58006f10 tx=0x7f0b58008040 comp rx=0 tx=0).ready entity=mgr.10644 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:12:16.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.747+0000 7f0b617fa700 1 -- 172.21.15.158:0/1870152961 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b50053fb0 con 0x7f0b640cc5d0 2023-10-16T11:12:16.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.994+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0b640b6230 con 0x7f0b640cc5d0 2023-10-16T11:12:16.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:16.997+0000 7f0b617fa700 1 -- 172.21.15.158:0/1870152961 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v115) v1 ==== 74+0+171246 (secure 0 0 0) 0x7f0b500575d0 con 0x7f0b640cc5d0 2023-10-16T11:12:17.001 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:12:17.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f0b4806b070 msgr2=0x7f0b4806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:17.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f0b4806b070 0x7f0b4806d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b58006f10 tx=0x7f0b58008040 comp rx=0 tx=0).stop 2023-10-16T11:12:17.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b640cc5d0 msgr2=0x7f0b64154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:12:17.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b640cc5d0 0x7f0b64154440 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000c4b0 tx=0x7f0b5000c7c0 comp rx=0 tx=0).stop 2023-10-16T11:12:17.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 shutdown_connections 2023-10-16T11:12:17.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b6415dfb0 0x7f0b640c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:17.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b6406f340 0x7f0b6415d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:17.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:6832/129037312,v1:172.21.15.158:6833/129037312] conn(0x7f0b4806b070 0x7f0b4806d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:17.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 --2- 172.21.15.158:0/1870152961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b640cc5d0 0x7f0b64154440 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:12:17.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 >> 172.21.15.158:0/1870152961 conn(0x7f0b640d02b0 msgr2=0x7f0b6415ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:12:17.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 shutdown_connections 2023-10-16T11:12:17.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:12:17.000+0000 7f0b6b1aa700 1 -- 172.21.15.158:0/1870152961 wait complete. 2023-10-16T11:12:17.013 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":115,"active_gid":10644,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":129037312},{"type":"v1","addr":"172.21.15.158:6833","nonce":129037312}]},"active_addr":"172.21.15.158:6833/129037312","active_change":"2023-10-16T11:01:31.624340+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":38144,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":186,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":3186172973}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":1399927564}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":2719309194}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":2531840430}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":2545298209}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":2065912370}]}]} 2023-10-16T11:12:17.018 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: y 2023-10-16T11:12:17.018 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-16T11:12:17.019 INFO:tasks.ceph.mgr.y:Stopping old one... 2023-10-16T11:12:17.019 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-16T11:12:17.019 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:12:17.048 INFO:tasks.ceph.mgr.y:Stopped 2023-10-16T11:12:17.048 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-16T11:12:17.090 INFO:tasks.ceph.mgr.y:Started 2023-10-16T11:12:17.267 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:17.265+0000 7ff72e7b6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:12:17.333 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:17.332+0000 7ff72e7b6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:12:17.406 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:17.404+0000 7ff72e7b6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:12:17.484 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:17.483+0000 7ff72e7b6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:12:17.627 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:17.625+0000 7ff72e7b6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:12:17.953 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:17.951+0000 7ff72e7b6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:12:18.071 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:18.070+0000 7ff72e7b6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:12:18.144 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:18.143+0000 7ff72e7b6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:12:18.225 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:18.223+0000 7ff72e7b6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:12:18.598 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:18.596+0000 7ff72e7b6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:12:19.170 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:19.169+0000 7ff72e7b6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:12:19.241 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:19.239+0000 7ff72e7b6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:12:19.879 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:19.877+0000 7ff72e7b6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:12:19.958 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:19.956+0000 7ff72e7b6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:12:20.105 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:20.103+0000 7ff72e7b6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:12:20.336 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:20.334+0000 7ff72e7b6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:12:20.411 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:20.410+0000 7ff72e7b6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:12:20.679 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:20.678+0000 7ff72e7b6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:12:20.905 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:20.904+0000 7ff72e7b6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:12:21.058 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:21.057+0000 7ff72e7b6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:12:21.174 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:21.172+0000 7ff72e7b6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:12:21.333 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:21.332+0000 7ff72e7b6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:12:21.600 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:21.598+0000 7ff72e7b6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:12:21.763 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:21.762+0000 7ff72e7b6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:12:22.297 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.295+0000 7ff72e7b6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:12:22.457 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a3f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.457 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a3f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.457 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a3f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.457 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a3f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.458 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a3f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.458 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a0f90700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.458 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a0f90700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.458 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a0f90700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.458 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a0f90700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:22.459 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:12:22.455+0000 7ff6a0f90700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:12:44.711 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:12:44.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:13:00.454 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:13:00.452+0000 7ff6abfa6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:13:00.455 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:13:00.452+0000 7ff6abfa6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:13:00.455 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:13:00.452+0000 7ff6abfa6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:13:00.455 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:13:00.452+0000 7ff6abfa6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:13:00.455 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:13:00.452+0000 7ff6abfa6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:13:14.875 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:13:14.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:13:44.937 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:13:44.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:14:15.125 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:14:15.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:14:45.221 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:14:45.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:15:15.317 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:15:15.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:15:45.398 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:15:45.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:16:15.535 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:16:15.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:16:45.840 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:16:45.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:17:15.903 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:17:15.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:17:17.092 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2023-10-16T11:17:17.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.226+0000 7f8666cef700 1 Processor -- start 2023-10-16T11:17:17.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.226+0000 7f8666cef700 1 -- start start 2023-10-16T11:17:17.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.227+0000 7f8666cef700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f866015deb0 0x7f86600ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.227+0000 7f8666cef700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86600ccc00 0x7f86600cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.227+0000 7f8666cef700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600d0ae0 con 0x7f866015dad0 2023-10-16T11:17:17.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.227+0000 7f8666cef700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86600d12d0 con 0x7f866015deb0 2023-10-16T11:17:17.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.227+0000 7f8666cef700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86600cbdf0 con 0x7f86600ccc00 2023-10-16T11:17:17.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.227+0000 7f865ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f866015deb0 0x7f86600ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f866015deb0 0x7f86600ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35502/0 (socket says 172.21.15.158:35502) 2023-10-16T11:17:17.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865ffff700 1 -- 172.21.15.158:0/3258670456 learned_addr learned my addr 172.21.15.158:0/3258670456 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:17.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f866528c700 1 --2- 172.21.15.158:0/3258670456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86600ccc00 0x7f86600cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865ffff700 1 -- 172.21.15.158:0/3258670456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86600ccc00 msgr2=0x7f86600cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:17.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865ffff700 1 --2- 172.21.15.158:0/3258670456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86600ccc00 0x7f86600cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865ffff700 1 -- 172.21.15.158:0/3258670456 >> v1:172.21.15.172:6789/0 conn(0x7f866015dad0 legacy=0x7f86600cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865ffff700 1 -- 172.21.15.158:0/3258670456 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600cc280 con 0x7f866015deb0 2023-10-16T11:17:17.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865ffff700 1 --2- 172.21.15.158:0/3258670456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015deb0 0x7f86600ce6b0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f864c0098f0 tx=0x7f864c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4284d6f3b46607b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:17.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865f7fe700 1 -- 172.21.15.158:0/3258670456 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c00b8a0 con 0x7f866015deb0 2023-10-16T11:17:17.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865f7fe700 1 -- 172.21.15.158:0/3258670456 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c00ba00 con 0x7f866015deb0 2023-10-16T11:17:17.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.228+0000 7f865f7fe700 1 -- 172.21.15.158:0/3258670456 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c00bd90 con 0x7f866015deb0 2023-10-16T11:17:17.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 -- 172.21.15.158:0/3258670456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015deb0 msgr2=0x7f86600ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 --2- 172.21.15.158:0/3258670456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015deb0 0x7f86600ce6b0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f864c0098f0 tx=0x7f864c009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:17.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 -- 172.21.15.158:0/3258670456 shutdown_connections 2023-10-16T11:17:17.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 --2- 172.21.15.158:0/3258670456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86600ccc00 0x7f86600cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 --2- 172.21.15.158:0/3258670456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015deb0 0x7f86600ce6b0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 -- 172.21.15.158:0/3258670456 >> 172.21.15.158:0/3258670456 conn(0x7f8660099a10 msgr2=0x7f86600d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:17.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 -- 172.21.15.158:0/3258670456 shutdown_connections 2023-10-16T11:17:17.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.229+0000 7f8666cef700 1 -- 172.21.15.158:0/3258670456 wait complete. 2023-10-16T11:17:17.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.230+0000 7f8666cef700 1 Processor -- start 2023-10-16T11:17:17.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.230+0000 7f8666cef700 1 -- start start 2023-10-16T11:17:17.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f8666cef700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86600ccc00 0x7f86601e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f8666cef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015dad0 0x7f86601f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f8666cef700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f866015deb0 0x7f86600c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f8666cef700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86601e3220 con 0x7f866015dad0 2023-10-16T11:17:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f8666cef700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86600c9e10 con 0x7f86600ccc00 2023-10-16T11:17:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f8666cef700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86600ca0f0 con 0x7f866015deb0 2023-10-16T11:17:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015dad0 0x7f86601f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015dad0 0x7f86601f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35508/0 (socket says 172.21.15.158:35508) 2023-10-16T11:17:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 -- 172.21.15.158:0/2742900608 learned_addr learned my addr 172.21.15.158:0/2742900608 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f8664a8b700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86600ccc00 0x7f86601e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f866528c700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f866015deb0 0x7f86600c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 -- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f866015deb0 msgr2=0x7f86600c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f866015deb0 0x7f86600c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 -- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86600ccc00 msgr2=0x7f86601e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86600ccc00 0x7f86601e59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f865ffff700 1 -- 172.21.15.158:0/2742900608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c009550 con 0x7f866015dad0 2023-10-16T11:17:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.231+0000 7f866528c700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f866015deb0 0x7f86600c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.232+0000 7f865ffff700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015dad0 0x7f86601f2690 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f864c000c00 tx=0x7f864c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:17.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.232+0000 7f865d7fa700 1 -- 172.21.15.158:0/2742900608 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c004380 con 0x7f866015dad0 2023-10-16T11:17:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.232+0000 7f865d7fa700 1 -- 172.21.15.158:0/2742900608 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c01f070 con 0x7f866015dad0 2023-10-16T11:17:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.232+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86600ca3d0 con 0x7f866015dad0 2023-10-16T11:17:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.232+0000 7f865d7fa700 1 -- 172.21.15.158:0/2742900608 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0036a0 con 0x7f866015dad0 2023-10-16T11:17:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.232+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8660149ff0 con 0x7f866015dad0 2023-10-16T11:17:17.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.233+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866014a180 con 0x7f866015dad0 2023-10-16T11:17:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.233+0000 7f865d7fa700 1 -- 172.21.15.158:0/2742900608 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f864c021030 con 0x7f866015dad0 2023-10-16T11:17:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.234+0000 7f865d7fa700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f864406b070 0x7f864406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.234+0000 7f865d7fa700 1 -- 172.21.15.158:0/2742900608 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f864c08b150 con 0x7f866015dad0 2023-10-16T11:17:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.234+0000 7f8664a8b700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f864406b070 0x7f864406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.234+0000 7f8664a8b700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f864406b070 0x7f864406d530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8654007fd0 tx=0x7f8654009040 comp rx=0 tx=0).ready entity=mgr.10740 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:17.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.238+0000 7f865d7fa700 1 -- 172.21.15.158:0/2742900608 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f864c055ce0 con 0x7f866015dad0 2023-10-16T11:17:17.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.466+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 --> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f86600d0ff0 con 0x7f864406b070 2023-10-16T11:17:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.511+0000 7f865d7fa700 1 -- 172.21.15.158:0/2742900608 <== mgr.10740 v2:172.21.15.158:6832/3817845958 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f8654001560 con 0x7f864406b070 2023-10-16T11:17:17.514 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T11:17:17.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.513+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f864406b070 msgr2=0x7f864406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.513+0000 7f8666cef700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f864406b070 0x7f864406d530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8654007fd0 tx=0x7f8654009040 comp rx=0 tx=0).stop 2023-10-16T11:17:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.513+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015dad0 msgr2=0x7f86601f2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.513+0000 7f8666cef700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015dad0 0x7f86601f2690 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f864c000c00 tx=0x7f864c004750 comp rx=0 tx=0).stop 2023-10-16T11:17:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 shutdown_connections 2023-10-16T11:17:17.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f864406b070 0x7f864406d530 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f866015deb0 0x7f86600c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86600ccc00 0x7f86601e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 --2- 172.21.15.158:0/2742900608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f866015dad0 0x7f86601f2690 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 >> 172.21.15.158:0/2742900608 conn(0x7f8660099a10 msgr2=0x7f866015cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:17.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 shutdown_connections 2023-10-16T11:17:17.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.514+0000 7f8666cef700 1 -- 172.21.15.158:0/2742900608 wait complete. 2023-10-16T11:17:17.528 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T11:17:17.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:17.529 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T11:17:17.546 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_02_00_UTC 2023-10-16T11:17:17.546 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_03_00_UTC 2023-10-16T11:17:17.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_04_00_UTC 2023-10-16T11:17:17.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_05_00_UTC 2023-10-16T11:17:17.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_06_00_UTC 2023-10-16T11:17:17.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_07_00_UTC 2023-10-16T11:17:17.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_08_00_UTC 2023-10-16T11:17:17.547 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_09_00_UTC 2023-10-16T11:17:17.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_10_00_UTC 2023-10-16T11:17:17.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_11_00_UTC 2023-10-16T11:17:17.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_12_00_UTC 2023-10-16T11:17:17.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_13_00_UTC 2023-10-16T11:17:17.548 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_14_00_UTC 2023-10-16T11:17:17.549 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_15_00_UTC 2023-10-16T11:17:17.549 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_16_00_UTC 2023-10-16T11:17:17.549 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_17_00_UTC 2023-10-16T11:17:17.549 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-11_02_00_UTC', 'scheduled-2023-10-16-11_03_00_UTC', 'scheduled-2023-10-16-11_04_00_UTC', 'scheduled-2023-10-16-11_05_00_UTC', 'scheduled-2023-10-16-11_06_00_UTC', 'scheduled-2023-10-16-11_07_00_UTC', 'scheduled-2023-10-16-11_08_00_UTC', 'scheduled-2023-10-16-11_09_00_UTC', 'scheduled-2023-10-16-11_10_00_UTC', 'scheduled-2023-10-16-11_11_00_UTC', 'scheduled-2023-10-16-11_12_00_UTC', 'scheduled-2023-10-16-11_13_00_UTC', 'scheduled-2023-10-16-11_14_00_UTC', 'scheduled-2023-10-16-11_15_00_UTC', 'scheduled-2023-10-16-11_16_00_UTC', 'scheduled-2023-10-16-11_17_00_UTC'] 2023-10-16T11:17:17.550 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T11:17:17.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.705+0000 7f765f815700 1 Processor -- start 2023-10-16T11:17:17.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.706+0000 7f765f815700 1 -- start start 2023-10-16T11:17:17.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.706+0000 7f765f815700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f765815b540 0x7f765806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.706+0000 7f765f815700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f765806afe0 0x7f7658161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.706+0000 7f765f815700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765815ee60 con 0x7f765806f180 2023-10-16T11:17:17.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.706+0000 7f765f815700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765815b920 con 0x7f765806afe0 2023-10-16T11:17:17.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.706+0000 7f765f815700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765815e2e0 con 0x7f765815b540 2023-10-16T11:17:17.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.706+0000 7f765ddb2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f765806afe0 0x7f7658161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765ddb2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f765806afe0 0x7f7658161860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35520/0 (socket says 172.21.15.158:35520) 2023-10-16T11:17:17.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765ddb2700 1 -- 172.21.15.158:0/3748988412 learned_addr learned my addr 172.21.15.158:0/3748988412 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:17.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765d5b1700 1 --2- 172.21.15.158:0/3748988412 >> v2:172.21.15.172:3300/0 conn(0x7f765815b540 0x7f765806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765ddb2700 1 -- 172.21.15.158:0/3748988412 >> v2:172.21.15.172:3300/0 conn(0x7f765815b540 msgr2=0x7f765806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:17.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765ddb2700 1 --2- 172.21.15.158:0/3748988412 >> v2:172.21.15.172:3300/0 conn(0x7f765815b540 0x7f765806d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765ddb2700 1 -- 172.21.15.158:0/3748988412 >> v1:172.21.15.172:6789/0 conn(0x7f765806f180 legacy=0x7f765806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:17.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765ddb2700 1 -- 172.21.15.158:0/3748988412 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765815e5c0 con 0x7f765806afe0 2023-10-16T11:17:17.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f765ddb2700 1 --2- 172.21.15.158:0/3748988412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f7658161860 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f765815ba60 tx=0x7f765000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5532dce527be672f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:17.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f764ffff700 1 -- 172.21.15.158:0/3748988412 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765000e070 con 0x7f765806afe0 2023-10-16T11:17:17.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f764ffff700 1 -- 172.21.15.158:0/3748988412 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765000b950 con 0x7f765806afe0 2023-10-16T11:17:17.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.707+0000 7f764ffff700 1 -- 172.21.15.158:0/3748988412 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765000bd00 con 0x7f765806afe0 2023-10-16T11:17:17.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 -- 172.21.15.158:0/3748988412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 msgr2=0x7f7658161860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 --2- 172.21.15.158:0/3748988412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f7658161860 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f765815ba60 tx=0x7f765000b330 comp rx=0 tx=0).stop 2023-10-16T11:17:17.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 -- 172.21.15.158:0/3748988412 shutdown_connections 2023-10-16T11:17:17.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 --2- 172.21.15.158:0/3748988412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f7658161860 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 --2- 172.21.15.158:0/3748988412 >> v2:172.21.15.172:3300/0 conn(0x7f765815b540 0x7f765806d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 -- 172.21.15.158:0/3748988412 >> 172.21.15.158:0/3748988412 conn(0x7f76580d03b0 msgr2=0x7f76580d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:17.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 -- 172.21.15.158:0/3748988412 shutdown_connections 2023-10-16T11:17:17.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.708+0000 7f765f815700 1 -- 172.21.15.158:0/3748988412 wait complete. 2023-10-16T11:17:17.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.709+0000 7f765f815700 1 Processor -- start 2023-10-16T11:17:17.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.709+0000 7f765f815700 1 -- start start 2023-10-16T11:17:17.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765f815700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f76581f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765f815700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f765806f180 0x7f765815d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765f815700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f765815b540 0x7f7658158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765f815700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76581f3d00 con 0x7f765806afe0 2023-10-16T11:17:17.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765f815700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765815dcc0 con 0x7f765815b540 2023-10-16T11:17:17.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765f815700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76581f3460 con 0x7f765806f180 2023-10-16T11:17:17.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f76581f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f76581f2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35522/0 (socket says 172.21.15.158:35522) 2023-10-16T11:17:17.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 -- 172.21.15.158:0/1429083191 learned_addr learned my addr 172.21.15.158:0/1429083191 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:17.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765ddb2700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f765815b540 0x7f7658158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765cdb0700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f765806f180 0x7f765815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 -- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f765806f180 msgr2=0x7f765815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f765806f180 0x7f765815d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 -- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f765815b540 msgr2=0x7f7658158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f765815b540 0x7f7658158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 -- 172.21.15.158:0/1429083191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7650009b30 con 0x7f765806afe0 2023-10-16T11:17:17.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765ddb2700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f765815b540 0x7f7658158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:17.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765cdb0700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f765806f180 0x7f765815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:17.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765d5b1700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f76581f2d60 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f765400c7f0 tx=0x7f765400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:17.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f764dffb700 1 -- 172.21.15.158:0/1429083191 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654011800 con 0x7f765806afe0 2023-10-16T11:17:17.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f764dffb700 1 -- 172.21.15.158:0/1429083191 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7654011960 con 0x7f765806afe0 2023-10-16T11:17:17.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f764dffb700 1 -- 172.21.15.158:0/1429083191 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654011cf0 con 0x7f765806afe0 2023-10-16T11:17:17.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.710+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76581f3740 con 0x7f765806afe0 2023-10-16T11:17:17.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.711+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765815bbc0 con 0x7f765806afe0 2023-10-16T11:17:17.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.711+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76580681b0 con 0x7f765806afe0 2023-10-16T11:17:17.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.712+0000 7f764dffb700 1 -- 172.21.15.158:0/1429083191 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f7654018360 con 0x7f765806afe0 2023-10-16T11:17:17.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.712+0000 7f764dffb700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f763c06b070 0x7f763c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:17.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.712+0000 7f764dffb700 1 -- 172.21.15.158:0/1429083191 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f7654089d10 con 0x7f765806afe0 2023-10-16T11:17:17.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.712+0000 7f765cdb0700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f763c06b070 0x7f763c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:17.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.713+0000 7f765cdb0700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f763c06b070 0x7f763c06d530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f76580cf690 tx=0x7f7648009040 comp rx=0 tx=0).ready entity=mgr.10740 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:17.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.717+0000 7f764dffb700 1 -- 172.21.15.158:0/1429083191 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7654054670 con 0x7f765806afe0 2023-10-16T11:17:17.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.935+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 --> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f765815eb80 con 0x7f763c06b070 2023-10-16T11:17:17.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.941+0000 7f764dffb700 1 -- 172.21.15.158:0/1429083191 <== mgr.10740 v2:172.21.15.158:6832/3817845958 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7f7648001c70 con 0x7f763c06b070 2023-10-16T11:17:17.944 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:17.944 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T11:01:32", "first": "2023-10-16T11:02:00", "last": "2023-10-16T11:17:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T11:17:17.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f763c06b070 msgr2=0x7f763c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f763c06b070 0x7f763c06d530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f76580cf690 tx=0x7f7648009040 comp rx=0 tx=0).stop 2023-10-16T11:17:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 msgr2=0x7f76581f2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f76581f2d60 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f765400c7f0 tx=0x7f765400cbb0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 shutdown_connections 2023-10-16T11:17:17.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f765815b540 0x7f7658158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f765806afe0 0x7f76581f2d60 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f763c06b070 0x7f763c06d530 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 --2- 172.21.15.158:0/1429083191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f765806f180 0x7f765815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 >> 172.21.15.158:0/1429083191 conn(0x7f76580d03b0 msgr2=0x7f765806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:17.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.943+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 shutdown_connections 2023-10-16T11:17:17.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:17.944+0000 7f765f815700 1 -- 172.21.15.158:0/1429083191 wait complete. 2023-10-16T11:17:17.958 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T11:01:32", "first": "2023-10-16T11:02:00", "last": "2023-10-16T11:17:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T11:17:17.958 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T11:01:32', 'first': '2023-10-16T11:02:00', 'last': '2023-10-16T11:17:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-10-16T11:17:17.958 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-10-16T11:17:17.959 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-10-16T11:17:17.959 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2023-10-16T11:17:18.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.080+0000 7f6f46275700 1 Processor -- start 2023-10-16T11:17:18.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.080+0000 7f6f46275700 1 -- start start 2023-10-16T11:17:18.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f46275700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6f4015da40 0x7f6f400caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f46275700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f400cbd30 con 0x7f6f400cc590 2023-10-16T11:17:18.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f46275700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f400cc1c0 con 0x7f6f4006afe0 2023-10-16T11:17:18.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f46275700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4015de20 con 0x7f6f4015da40 2023-10-16T11:17:18.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6f400cc590 0x7f6f4006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:35706/0 (socket says 172.21.15.158:35706) 2023-10-16T11:17:18.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3f7fe700 1 -- 172.21.15.158:0/3553459088 learned_addr learned my addr 172.21.15.158:0/3553459088 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:18.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125460209 0 0) 0x7f6f400cbd30 con 0x7f6f400cc590 2023-10-16T11:17:18.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/3553459088 >> v2:172.21.15.172:3300/0 conn(0x7f6f4015da40 0x7f6f400caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f280035b0 con 0x7f6f400cc590 2023-10-16T11:17:18.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327495396 0 0) 0x7f6f280035b0 con 0x7f6f400cc590 2023-10-16T11:17:18.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 >> v2:172.21.15.172:3300/0 conn(0x7f6f4015da40 msgr2=0x7f6f400caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:18.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 --2- 172.21.15.158:0/3553459088 >> v2:172.21.15.172:3300/0 conn(0x7f6f4015da40 0x7f6f400caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 >> v1:172.21.15.172:6789/0 conn(0x7f6f4006afe0 legacy=0x7f6f400afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:18.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400ce260 con 0x7f6f400cc590 2023-10-16T11:17:18.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f6f30001670 con 0x7f6f400cc590 2023-10-16T11:17:18.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 >> v1:172.21.15.158:6789/0 conn(0x7f6f400cc590 legacy=0x7f6f4006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:18.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f28004850 0x7f6f28004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f28005210 0x7f6f2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4015de20 con 0x7f6f28003a00 2023-10-16T11:17:18.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2800c0b0 con 0x7f6f28005210 2023-10-16T11:17:18.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.081+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2800c270 con 0x7f6f28004850 2023-10-16T11:17:18.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f44812700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f28005210 0x7f6f2800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3f7fe700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f28004850 0x7f6f28004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3ffff700 1 -- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f28004850 msgr2=0x7f6f28004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:18.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f28004850 0x7f6f28004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3ffff700 1 -- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f28005210 msgr2=0x7f6f2800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:18.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f28005210 0x7f6f2800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3ffff700 1 -- 172.21.15.158:0/3553459088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f280035b0 con 0x7f6f28003a00 2023-10-16T11:17:18.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f44812700 1 -- 172.21.15.158:0/3553459088 reap_dead start 2023-10-16T11:17:18.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f28003e20 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009b60 tx=0x7f6f28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b48a50bef9806f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:18.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400e6c0 con 0x7f6f28003a00 2023-10-16T11:17:18.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3400e820 con 0x7f6f28003a00 2023-10-16T11:17:18.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.082+0000 7f6f3effd700 1 -- 172.21.15.158:0/3553459088 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400eb90 con 0x7f6f28003a00 2023-10-16T11:17:18.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.083+0000 7f6f46275700 1 -- 172.21.15.158:0/3553459088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 msgr2=0x7f6f28003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:18.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.083+0000 7f6f46275700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f28003e20 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009b60 tx=0x7f6f28005880 comp rx=0 tx=0).stop 2023-10-16T11:17:18.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.083+0000 7f6f46275700 1 -- 172.21.15.158:0/3553459088 shutdown_connections 2023-10-16T11:17:18.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.083+0000 7f6f46275700 1 --2- 172.21.15.158:0/3553459088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f28003e20 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.083+0000 7f6f46275700 1 -- 172.21.15.158:0/3553459088 >> 172.21.15.158:0/3553459088 conn(0x7f6f40099a10 msgr2=0x7f6f400d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:18.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.083+0000 7f6f46275700 1 -- 172.21.15.158:0/3553459088 shutdown_connections 2023-10-16T11:17:18.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.083+0000 7f6f46275700 1 -- 172.21.15.158:0/3553459088 wait complete. 2023-10-16T11:17:18.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.084+0000 7f6f46275700 1 Processor -- start 2023-10-16T11:17:18.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.084+0000 7f6f46275700 1 -- start start 2023-10-16T11:17:18.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f4014d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f400c7770 0x7f6f400ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f400b1e90 0x7f6f400b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400b2880 con 0x7f6f28003a00 2023-10-16T11:17:18.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4014de10 con 0x7f6f400b1e90 2023-10-16T11:17:18.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4015efe0 con 0x7f6f400c7770 2023-10-16T11:17:18.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f4014d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f4014d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35542/0 (socket says 172.21.15.158:35542) 2023-10-16T11:17:18.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 -- 172.21.15.158:0/2207280475 learned_addr learned my addr 172.21.15.158:0/2207280475 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:18.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3f7fe700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f400c7770 0x7f6f400ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f44812700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f400b1e90 0x7f6f400b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 -- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f400c7770 msgr2=0x7f6f400ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:18.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f400c7770 0x7f6f400ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 -- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f400b1e90 msgr2=0x7f6f400b2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:18.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f400b1e90 0x7f6f400b2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 -- 172.21.15.158:0/2207280475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400c8230 con 0x7f6f28003a00 2023-10-16T11:17:18.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3ffff700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f4014d8d0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e4b0 tx=0x7f6f340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:18.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3cff9700 1 -- 172.21.15.158:0/2207280475 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400e6c0 con 0x7f6f28003a00 2023-10-16T11:17:18.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3cff9700 1 -- 172.21.15.158:0/2207280475 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f340038c0 con 0x7f6f28003a00 2023-10-16T11:17:18.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f3cff9700 1 -- 172.21.15.158:0/2207280475 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34003c70 con 0x7f6f28003a00 2023-10-16T11:17:18.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f400c7c70 con 0x7f6f28003a00 2023-10-16T11:17:18.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.085+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4015ec70 con 0x7f6f28003a00 2023-10-16T11:17:18.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.086+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4015ee00 con 0x7f6f28003a00 2023-10-16T11:17:18.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.087+0000 7f6f3cff9700 1 -- 172.21.15.158:0/2207280475 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f6f34013470 con 0x7f6f28003a00 2023-10-16T11:17:18.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.087+0000 7f6f3cff9700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f6f2c06b070 0x7f6f2c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:18.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.087+0000 7f6f3cff9700 1 -- 172.21.15.158:0/2207280475 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f6f3408b440 con 0x7f6f28003a00 2023-10-16T11:17:18.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.087+0000 7f6f3f7fe700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f6f2c06b070 0x7f6f2c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:18.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.088+0000 7f6f3f7fe700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f6f2c06b070 0x7f6f2c06d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f4015d090 tx=0x7f6f30007040 comp rx=0 tx=0).ready entity=mgr.10740 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:18.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.092+0000 7f6f3cff9700 1 -- 172.21.15.158:0/2207280475 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f34055fd0 con 0x7f6f28003a00 2023-10-16T11:17:18.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.313+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 --> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f400cb400 con 0x7f6f2c06b070 2023-10-16T11:17:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.367+0000 7f6f3cff9700 1 -- 172.21.15.158:0/2207280475 <== mgr.10740 v2:172.21.15.158:6832/3817845958 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f6f30011710 con 0x7f6f2c06b070 2023-10-16T11:17:18.374 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T11:17:18.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f6f2c06b070 msgr2=0x7f6f2c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f6f2c06b070 0x7f6f2c06d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f4015d090 tx=0x7f6f30007040 comp rx=0 tx=0).stop 2023-10-16T11:17:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 msgr2=0x7f6f4014d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f4014d8d0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e4b0 tx=0x7f6f340047c0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 shutdown_connections 2023-10-16T11:17:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f400b1e90 0x7f6f400b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f28003a00 0x7f6f4014d8d0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f6f2c06b070 0x7f6f2c06d530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 --2- 172.21.15.158:0/2207280475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f400c7770 0x7f6f400ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:18.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 >> 172.21.15.158:0/2207280475 conn(0x7f6f40099a10 msgr2=0x7f6f401e7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:18.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 shutdown_connections 2023-10-16T11:17:18.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:18.370+0000 7f6f46275700 1 -- 172.21.15.158:0/2207280475 wait complete. 2023-10-16T11:17:18.384 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T11:17:18.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.385 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T11:17:18.403 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_02_00_UTC 2023-10-16T11:17:18.403 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_03_00_UTC 2023-10-16T11:17:18.403 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_04_00_UTC 2023-10-16T11:17:18.404 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_05_00_UTC 2023-10-16T11:17:18.404 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_06_00_UTC 2023-10-16T11:17:18.404 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_07_00_UTC 2023-10-16T11:17:18.404 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_08_00_UTC 2023-10-16T11:17:18.405 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_09_00_UTC 2023-10-16T11:17:18.405 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_10_00_UTC 2023-10-16T11:17:18.405 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_11_00_UTC 2023-10-16T11:17:18.405 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_12_00_UTC 2023-10-16T11:17:18.405 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_13_00_UTC 2023-10-16T11:17:18.406 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_14_00_UTC 2023-10-16T11:17:18.406 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_15_00_UTC 2023-10-16T11:17:18.406 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_16_00_UTC 2023-10-16T11:17:18.406 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_17_00_UTC 2023-10-16T11:17:18.407 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_02_00_UTC 2023-10-16T11:17:18.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.407 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_02_00_UTC) 2023-10-16T11:17:18.464 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_03_00_UTC 2023-10-16T11:17:18.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.465 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_03_00_UTC) 2023-10-16T11:17:18.525 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_04_00_UTC 2023-10-16T11:17:18.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.526 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_04_00_UTC) 2023-10-16T11:17:18.586 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_05_00_UTC 2023-10-16T11:17:18.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.587 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_05_00_UTC) 2023-10-16T11:17:18.647 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_06_00_UTC 2023-10-16T11:17:18.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.648 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_06_00_UTC) 2023-10-16T11:17:18.708 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_07_00_UTC 2023-10-16T11:17:18.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.709 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_07_00_UTC) 2023-10-16T11:17:18.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_08_00_UTC 2023-10-16T11:17:18.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.770 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_08_00_UTC) 2023-10-16T11:17:18.830 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_09_00_UTC 2023-10-16T11:17:18.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.831 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_09_00_UTC) 2023-10-16T11:17:18.892 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_10_00_UTC 2023-10-16T11:17:18.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.893 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_10_00_UTC) 2023-10-16T11:17:18.954 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_11_00_UTC 2023-10-16T11:17:18.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:18.955 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_11_00_UTC) 2023-10-16T11:17:19.016 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_12_00_UTC 2023-10-16T11:17:19.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:19.017 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_12_00_UTC) 2023-10-16T11:17:19.077 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_13_00_UTC 2023-10-16T11:17:19.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:19.078 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_13_00_UTC) 2023-10-16T11:17:19.139 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_14_00_UTC 2023-10-16T11:17:19.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:19.140 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_14_00_UTC) 2023-10-16T11:17:19.200 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_15_00_UTC 2023-10-16T11:17:19.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:19.201 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_15_00_UTC) 2023-10-16T11:17:19.262 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_16_00_UTC 2023-10-16T11:17:19.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:19.263 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_16_00_UTC) 2023-10-16T11:17:19.324 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_17_00_UTC 2023-10-16T11:17:19.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:19.324 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-11_17_00_UTC) 2023-10-16T11:17:19.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:19.386 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-10-16T11:17:19.446 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:17:19.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.608+0000 7fb65d61d700 1 Processor -- start 2023-10-16T11:17:19.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.608+0000 7fb65d61d700 1 -- start start 2023-10-16T11:17:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.608+0000 7fb65d61d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb6580b38f0 0x7fb6580d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.608+0000 7fb65d61d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6580d2010 0x7fb65806bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:19.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.608+0000 7fb65d61d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65815e3e0 con 0x7fb65815d840 2023-10-16T11:17:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb65d61d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65815ec80 con 0x7fb6580d2010 2023-10-16T11:17:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb65d61d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580d13b0 con 0x7fb6580b38f0 2023-10-16T11:17:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6580d2010 0x7fb65806bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6580d2010 0x7fb65806bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35554/0 (socket says 172.21.15.158:35554) 2023-10-16T11:17:19.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 -- 172.21.15.158:0/1687146395 learned_addr learned my addr 172.21.15.158:0/1687146395 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6567fc700 1 --2- 172.21.15.158:0/1687146395 >> v2:172.21.15.172:3300/0 conn(0x7fb6580b38f0 0x7fb6580d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 -- 172.21.15.158:0/1687146395 >> v2:172.21.15.172:3300/0 conn(0x7fb6580b38f0 msgr2=0x7fb6580d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 --2- 172.21.15.158:0/1687146395 >> v2:172.21.15.172:3300/0 conn(0x7fb6580b38f0 0x7fb6580d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 -- 172.21.15.158:0/1687146395 >> v1:172.21.15.172:6789/0 conn(0x7fb65815d840 legacy=0x7fb65815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:19.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 -- 172.21.15.158:0/1687146395 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65815e6c0 con 0x7fb6580d2010 2023-10-16T11:17:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6567fc700 1 --2- 172.21.15.158:0/1687146395 >> v2:172.21.15.172:3300/0 conn(0x7fb6580b38f0 0x7fb6580d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb6577fe700 1 --2- 172.21.15.158:0/1687146395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6580d2010 0x7fb65806bbf0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fb6580d0e20 tx=0x7fb65000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387a45bb6507d7ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb655ffb700 1 -- 172.21.15.158:0/1687146395 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65000e070 con 0x7fb6580d2010 2023-10-16T11:17:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb655ffb700 1 -- 172.21.15.158:0/1687146395 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65000b9c0 con 0x7fb6580d2010 2023-10-16T11:17:19.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.609+0000 7fb655ffb700 1 -- 172.21.15.158:0/1687146395 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65000bd70 con 0x7fb6580d2010 2023-10-16T11:17:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 -- 172.21.15.158:0/1687146395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6580d2010 msgr2=0x7fb65806bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 --2- 172.21.15.158:0/1687146395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6580d2010 0x7fb65806bbf0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fb6580d0e20 tx=0x7fb65000b330 comp rx=0 tx=0).stop 2023-10-16T11:17:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 -- 172.21.15.158:0/1687146395 shutdown_connections 2023-10-16T11:17:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 --2- 172.21.15.158:0/1687146395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6580d2010 0x7fb65806bbf0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:19.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 --2- 172.21.15.158:0/1687146395 >> v2:172.21.15.172:3300/0 conn(0x7fb6580b38f0 0x7fb6580d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:19.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 -- 172.21.15.158:0/1687146395 >> 172.21.15.158:0/1687146395 conn(0x7fb65806ec00 msgr2=0x7fb65806efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:19.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 -- 172.21.15.158:0/1687146395 shutdown_connections 2023-10-16T11:17:19.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.610+0000 7fb65d61d700 1 -- 172.21.15.158:0/1687146395 wait complete. 2023-10-16T11:17:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.611+0000 7fb65d61d700 1 Processor -- start 2023-10-16T11:17:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.611+0000 7fb65d61d700 1 -- start start 2023-10-16T11:17:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb65d61d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6580b38f0 0x7fb6581e6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:19.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb65d61d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6580d2010 0x7fb6580b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb65d61d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65815d840 0x7fb6580b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb65d61d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581e8760 con 0x7fb65815d840 2023-10-16T11:17:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb65d61d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580b4fa0 con 0x7fb6580d2010 2023-10-16T11:17:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb65d61d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580b5220 con 0x7fb6580b38f0 2023-10-16T11:17:19.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65815d840 0x7fb6580b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65815d840 0x7fb6580b4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35560/0 (socket says 172.21.15.158:35560) 2023-10-16T11:17:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 -- 172.21.15.158:0/3278133865 learned_addr learned my addr 172.21.15.158:0/3278133865 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb656ffd700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6580b38f0 0x7fb6581e6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 -- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6580b38f0 msgr2=0x7fb6581e6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:19.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6567fc700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6580d2010 0x7fb6580b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6580b38f0 0x7fb6581e6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 -- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6580d2010 msgr2=0x7fb6580b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6580d2010 0x7fb6580b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 -- 172.21.15.158:0/3278133865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650009b30 con 0x7fb65815d840 2023-10-16T11:17:19.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6567fc700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6580d2010 0x7fb6580b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.612+0000 7fb6577fe700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65815d840 0x7fb6580b4a60 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fb6580cebc0 tx=0x7fb650004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.613+0000 7fb63f7fe700 1 -- 172.21.15.158:0/3278133865 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500099b0 con 0x7fb65815d840 2023-10-16T11:17:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.613+0000 7fb63f7fe700 1 -- 172.21.15.158:0/3278133865 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6500092e0 con 0x7fb65815d840 2023-10-16T11:17:19.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.613+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6581e8990 con 0x7fb65815d840 2023-10-16T11:17:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.613+0000 7fb63f7fe700 1 -- 172.21.15.158:0/3278133865 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65001c640 con 0x7fb65815d840 2023-10-16T11:17:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.613+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6581e8f10 con 0x7fb65815d840 2023-10-16T11:17:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.614+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6580681b0 con 0x7fb65815d840 2023-10-16T11:17:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.614+0000 7fb63f7fe700 1 -- 172.21.15.158:0/3278133865 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fb65001c7a0 con 0x7fb65815d840 2023-10-16T11:17:19.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.615+0000 7fb63f7fe700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fb64406b070 0x7fb64406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.615+0000 7fb656ffd700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fb64406b070 0x7fb64406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.615+0000 7fb63f7fe700 1 -- 172.21.15.158:0/3278133865 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7fb65008b7e0 con 0x7fb65815d840 2023-10-16T11:17:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.615+0000 7fb656ffd700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fb64406b070 0x7fb64406d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb648007fd0 tx=0x7fb648009040 comp rx=0 tx=0).ready entity=mgr.10740 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:19.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.619+0000 7fb63f7fe700 1 -- 172.21.15.158:0/3278133865 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb650056140 con 0x7fb65815d840 2023-10-16T11:17:19.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:19.866+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb6580b7400 con 0x7fb65815d840 2023-10-16T11:17:20.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.590+0000 7fb63f7fe700 1 -- 172.21.15.158:0/3278133865 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v122) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb650059760 con 0x7fb65815d840 2023-10-16T11:17:20.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.594+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fb64406b070 msgr2=0x7fb64406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:20.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.594+0000 7fb65d61d700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fb64406b070 0x7fb64406d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb648007fd0 tx=0x7fb648009040 comp rx=0 tx=0).stop 2023-10-16T11:17:20.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.594+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65815d840 msgr2=0x7fb6580b4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:20.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.594+0000 7fb65d61d700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65815d840 0x7fb6580b4a60 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fb6580cebc0 tx=0x7fb650004ad0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.595+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 shutdown_connections 2023-10-16T11:17:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.595+0000 7fb65d61d700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65815d840 0x7fb6580b4a60 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.595+0000 7fb65d61d700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fb64406b070 0x7fb64406d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.595+0000 7fb65d61d700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6580b38f0 0x7fb6581e6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.595+0000 7fb65d61d700 1 --2- 172.21.15.158:0/3278133865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6580d2010 0x7fb6580b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.595+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 >> 172.21.15.158:0/3278133865 conn(0x7fb65806ec00 msgr2=0x7fb65815ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.596+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 shutdown_connections 2023-10-16T11:17:20.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.597+0000 7fb65d61d700 1 -- 172.21.15.158:0/3278133865 wait complete. 2023-10-16T11:17:20.615 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:17:20.615 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:17:20.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:17:20.695 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:17:20.696 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:17:20.697 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:20.716 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:17:20.716 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:20.745 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:20.743+0000 7f8e793c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:17:20.751 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:20.749+0000 7fd9a47fa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:17:20.768 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:20.811 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:20.809+0000 7f8e793c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:20.819 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:20.817+0000 7fd9a47fa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:20.882 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:20.880+0000 7f8e793c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:17:20.895 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:20.893+0000 7fd9a47fa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:17:20.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.930+0000 7fa7a1f8d700 1 Processor -- start 2023-10-16T11:17:20.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.930+0000 7fa7a1f8d700 1 -- start start 2023-10-16T11:17:20.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.930+0000 7fa7a1f8d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa79c15d7c0 0x7fa79c0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:20.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa7a1f8d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c0d1210 0x7fa79c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:20.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa7a1f8d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa79c0ce950 0x7fa79c0b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:20.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa7a1f8d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79c0d4060 con 0x7fa79c15d7c0 2023-10-16T11:17:20.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa7a1f8d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79c0d3bd0 con 0x7fa79c0ce950 2023-10-16T11:17:20.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa7a1f8d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c0cede0 con 0x7fa79c0d1210 2023-10-16T11:17:20.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa79c15d7c0 0x7fa79c0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:20.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa79c15d7c0 0x7fa79c0d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35620/0 (socket says 172.21.15.158:35620) 2023-10-16T11:17:20.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2701142516 learned_addr learned my addr 172.21.15.158:0/2701142516 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:20.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79affd700 1 --2- 172.21.15.158:0/2701142516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c0d1210 0x7fa79c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:20.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79bfff700 1 --2- 172.21.15.158:0/2701142516 >> v2:172.21.15.172:3300/0 conn(0x7fa79c0ce950 0x7fa79c0b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:20.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2701142516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c0d1210 msgr2=0x7fa79c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:20.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 --2- 172.21.15.158:0/2701142516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c0d1210 0x7fa79c0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2701142516 >> v2:172.21.15.172:3300/0 conn(0x7fa79c0ce950 msgr2=0x7fa79c0b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:20.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 --2- 172.21.15.158:0/2701142516 >> v2:172.21.15.172:3300/0 conn(0x7fa79c0ce950 0x7fa79c0b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2701142516 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c0d2cf0 con 0x7fa79c15d7c0 2023-10-16T11:17:20.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79bfff700 1 --2- 172.21.15.158:0/2701142516 >> v2:172.21.15.172:3300/0 conn(0x7fa79c0ce950 0x7fa79c0b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79b7fe700 1 --2- 172.21.15.158:0/2701142516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c15d7c0 0x7fa79c0d27b0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0b2d60 tx=0x7fa78c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695e955f5bc8c06f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79a7fc700 1 -- 172.21.15.158:0/2701142516 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c011830 con 0x7fa79c15d7c0 2023-10-16T11:17:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79a7fc700 1 -- 172.21.15.158:0/2701142516 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c011990 con 0x7fa79c15d7c0 2023-10-16T11:17:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.931+0000 7fa79a7fc700 1 -- 172.21.15.158:0/2701142516 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c011d20 con 0x7fa79c15d7c0 2023-10-16T11:17:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2701142516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c15d7c0 msgr2=0x7fa79c0d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2701142516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c15d7c0 0x7fa79c0d27b0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0b2d60 tx=0x7fa78c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2701142516 shutdown_connections 2023-10-16T11:17:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2701142516 >> v2:172.21.15.172:3300/0 conn(0x7fa79c0ce950 0x7fa79c0b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2701142516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c15d7c0 0x7fa79c0d27b0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2701142516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c0d1210 0x7fa79c0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2701142516 >> 172.21.15.158:0/2701142516 conn(0x7fa79c0669f0 msgr2=0x7fa79c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2701142516 shutdown_connections 2023-10-16T11:17:20.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.932+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2701142516 wait complete. 2023-10-16T11:17:20.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.933+0000 7fa7a1f8d700 1 Processor -- start 2023-10-16T11:17:20.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.933+0000 7fa7a1f8d700 1 -- start start 2023-10-16T11:17:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa7a1f8d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c0ce950 0x7fa79c1574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa7a1f8d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa79c0d1210 0x7fa79c0cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa7a1f8d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c15d7c0 0x7fa79c0a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:20.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa7a1f8d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c157bd0 con 0x7fa79c0ce950 2023-10-16T11:17:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa7a1f8d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c1f2b20 con 0x7fa79c0d1210 2023-10-16T11:17:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa7a1f8d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c0cc0a0 con 0x7fa79c15d7c0 2023-10-16T11:17:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa79b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c0ce950 0x7fa79c1574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa79b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c0ce950 0x7fa79c1574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35626/0 (socket says 172.21.15.158:35626) 2023-10-16T11:17:20.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2307388304 learned_addr learned my addr 172.21.15.158:0/2307388304 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:20.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa79bfff700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c15d7c0 0x7fa79c0a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:20.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.934+0000 7fa79affd700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa79c0d1210 0x7fa79c0cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:20.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c15d7c0 msgr2=0x7fa79c0a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:20.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa79b7fe700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c15d7c0 0x7fa79c0a9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa79c0d1210 msgr2=0x7fa79c0cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:20.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa79b7fe700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa79c0d1210 0x7fa79c0cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:20.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa79b7fe700 1 -- 172.21.15.158:0/2307388304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c009550 con 0x7fa79c0ce950 2023-10-16T11:17:20.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa79b7fe700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c0ce950 0x7fa79c1574a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fa78c000c00 tx=0x7fa78c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:20.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa783fff700 1 -- 172.21.15.158:0/2307388304 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c004240 con 0x7fa79c0ce950 2023-10-16T11:17:20.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa783fff700 1 -- 172.21.15.158:0/2307388304 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c01f070 con 0x7fa79c0ce950 2023-10-16T11:17:20.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa783fff700 1 -- 172.21.15.158:0/2307388304 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c0036a0 con 0x7fa79c0ce950 2023-10-16T11:17:20.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79c0aae50 con 0x7fa79c0ce950 2023-10-16T11:17:20.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.935+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79c0cbae0 con 0x7fa79c0ce950 2023-10-16T11:17:20.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.936+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79c157d10 con 0x7fa79c0ce950 2023-10-16T11:17:20.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.937+0000 7fa783fff700 1 -- 172.21.15.158:0/2307388304 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa78c021030 con 0x7fa79c0ce950 2023-10-16T11:17:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.937+0000 7fa783fff700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 0x7fa78806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.937+0000 7fa79affd700 1 -- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 msgr2=0x7fa78806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.937+0000 7fa783fff700 1 -- 172.21.15.158:0/2307388304 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7fa78c08ae60 con 0x7fa79c0ce950 2023-10-16T11:17:20.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.937+0000 7fa79affd700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 0x7fa78806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:20.942+0000 7fa783fff700 1 -- 172.21.15.158:0/2307388304 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa79c157d10 con 0x7fa79c0ce950 2023-10-16T11:17:20.960 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:20.958+0000 7f8e793c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:17:20.980 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:20.978+0000 7fd9a47fa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:17:21.102 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:21.100+0000 7f8e793c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:17:21.126 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:21.124+0000 7fd9a47fa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:17:21.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.138+0000 7fa79affd700 1 -- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 msgr2=0x7fa78806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:21.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.138+0000 7fa79affd700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 0x7fa78806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:21.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.143+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa79c157d10 con 0x7fa79c0ce950 2023-10-16T11:17:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.144+0000 7fa783fff700 1 -- 172.21.15.158:0/2307388304 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+18432 (secure 0 0 0) 0x7fa79c157d10 con 0x7fa79c0ce950 2023-10-16T11:17:21.147 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:21.148 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":187,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:12:22.309951+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:01:16.077848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T11:01:16.598638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2531840430":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2719309194":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2065912370":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/1399927564":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6833/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2545298209":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/549775613":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/827805412":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/2974007506":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/4132457601":"2023-10-17T11:01:31.624108+0000","172.21.15.158:6833/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3186172973":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3804388680":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/411715851":"2023-10-17T11:01:31.624108+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:21.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.146+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 msgr2=0x7fa78806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.146+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 0x7fa78806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c0ce950 msgr2=0x7fa79c1574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c0ce950 0x7fa79c1574a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fa78c000c00 tx=0x7fa78c004750 comp rx=0 tx=0).stop 2023-10-16T11:17:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 shutdown_connections 2023-10-16T11:17:21.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fa78806b020 0x7fa78806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa79c15d7c0 0x7fa79c0a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa79c0ce950 0x7fa79c1574a0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 --2- 172.21.15.158:0/2307388304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa79c0d1210 0x7fa79c0cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 >> 172.21.15.158:0/2307388304 conn(0x7fa79c0669f0 msgr2=0x7fa79c0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:21.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 shutdown_connections 2023-10-16T11:17:21.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.147+0000 7fa7a1f8d700 1 -- 172.21.15.158:0/2307388304 wait complete. 2023-10-16T11:17:21.163 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:01:16.077848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:01:16.598638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:17:21.163 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:17:21.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.326+0000 7f2d7b8a6700 1 Processor -- start 2023-10-16T11:17:21.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.326+0000 7f2d7b8a6700 1 -- start start 2023-10-16T11:17:21.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.327+0000 7f2d7b8a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d7415d6c0 0x7f2d740cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.327+0000 7f2d7b8a6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2d7415daa0 0x7f2d740d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.327+0000 7f2d7b8a6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d740d1ee0 con 0x7f2d740d2370 2023-10-16T11:17:21.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.327+0000 7f2d7b8a6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d740d40a0 con 0x7f2d7415daa0 2023-10-16T11:17:21.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.327+0000 7f2d7b8a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740d12d0 con 0x7f2d7415d6c0 2023-10-16T11:17:21.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d78e41700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2d7415daa0 0x7f2d740d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d79e43700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2d740d2370 0x7f2d740cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:35850/0 (socket says 172.21.15.158:35850) 2023-10-16T11:17:21.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d78e41700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2d7415daa0 0x7f2d740d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35672/0 (socket says 172.21.15.158:35672) 2023-10-16T11:17:21.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d79e43700 1 -- 172.21.15.158:0/4151658951 learned_addr learned my addr 172.21.15.158:0/4151658951 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:21.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d79642700 1 --2- 172.21.15.158:0/4151658951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d7415d6c0 0x7f2d740cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d78e41700 1 -- 172.21.15.158:0/4151658951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d7415d6c0 msgr2=0x7f2d740cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:21.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d78e41700 1 --2- 172.21.15.158:0/4151658951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d7415d6c0 0x7f2d740cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d78e41700 1 -- 172.21.15.158:0/4151658951 >> v1:172.21.15.158:6789/0 conn(0x7f2d740d2370 legacy=0x7f2d740cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d78e41700 1 -- 172.21.15.158:0/4151658951 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d740d15b0 con 0x7f2d7415daa0 2023-10-16T11:17:21.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d78e41700 1 --2- 172.21.15.158:0/4151658951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740d0470 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009840 tx=0x7f2d64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225de3adfaa680d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:21.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d6bfff700 1 -- 172.21.15.158:0/4151658951 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64011830 con 0x7f2d7415daa0 2023-10-16T11:17:21.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d6bfff700 1 -- 172.21.15.158:0/4151658951 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d64011990 con 0x7f2d7415daa0 2023-10-16T11:17:21.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.328+0000 7f2d6bfff700 1 -- 172.21.15.158:0/4151658951 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64011d20 con 0x7f2d7415daa0 2023-10-16T11:17:21.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/4151658951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 msgr2=0x7f2d740d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/4151658951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740d0470 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009840 tx=0x7f2d64009b50 comp rx=0 tx=0).stop 2023-10-16T11:17:21.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/4151658951 shutdown_connections 2023-10-16T11:17:21.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/4151658951 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d7415d6c0 0x7f2d740cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/4151658951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740d0470 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/4151658951 >> 172.21.15.158:0/4151658951 conn(0x7f2d7409c940 msgr2=0x7f2d740d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:21.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/4151658951 shutdown_connections 2023-10-16T11:17:21.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.329+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/4151658951 wait complete. 2023-10-16T11:17:21.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.330+0000 7f2d7b8a6700 1 Processor -- start 2023-10-16T11:17:21.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.330+0000 7f2d7b8a6700 1 -- start start 2023-10-16T11:17:21.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d7b8a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d740d2370 0x7f2d74151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d7b8a6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d7415d6c0 0x7f2d740c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d7b8a6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d7b8a6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74152000 con 0x7f2d7415daa0 2023-10-16T11:17:21.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d7b8a6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740c9d00 con 0x7f2d7415d6c0 2023-10-16T11:17:21.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d7b8a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740cb790 con 0x7f2d740d2370 2023-10-16T11:17:21.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d79e43700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d79e43700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35680/0 (socket says 172.21.15.158:35680) 2023-10-16T11:17:21.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d79e43700 1 -- 172.21.15.158:0/866451046 learned_addr learned my addr 172.21.15.158:0/866451046 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:21.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d79642700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d740d2370 0x7f2d74151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.331+0000 7f2d78e41700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d7415d6c0 0x7f2d740c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d79e43700 1 -- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d740d2370 msgr2=0x7f2d74151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d79e43700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d740d2370 0x7f2d74151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d79e43700 1 -- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d7415d6c0 msgr2=0x7f2d740c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d79e43700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d7415d6c0 0x7f2d740c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d79e43700 1 -- 172.21.15.158:0/866451046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d640094a0 con 0x7f2d7415daa0 2023-10-16T11:17:21.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d79642700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d740d2370 0x7f2d74151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:21.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d79e43700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740c9680 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00cbc0 tx=0x7f2d6c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:21.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d69ffb700 1 -- 172.21.15.158:0/866451046 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00a9e0 con 0x7f2d7415daa0 2023-10-16T11:17:21.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d69ffb700 1 -- 172.21.15.158:0/866451046 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c00ab40 con 0x7f2d7415daa0 2023-10-16T11:17:21.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d69ffb700 1 -- 172.21.15.158:0/866451046 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c012400 con 0x7f2d7415daa0 2023-10-16T11:17:21.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d740cba70 con 0x7f2d7415daa0 2023-10-16T11:17:21.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.332+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d740cafd0 con 0x7f2d7415daa0 2023-10-16T11:17:21.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.333+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d740cb5f0 con 0x7f2d7415daa0 2023-10-16T11:17:21.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.333+0000 7f2d69ffb700 1 -- 172.21.15.158:0/866451046 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f2d6c01b3b0 con 0x7f2d7415daa0 2023-10-16T11:17:21.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.334+0000 7f2d69ffb700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.334+0000 7f2d79642700 1 -- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 msgr2=0x7f2d5806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:21.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.334+0000 7f2d69ffb700 1 -- 172.21.15.158:0/866451046 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f2d6c089820 con 0x7f2d7415daa0 2023-10-16T11:17:21.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.334+0000 7f2d79642700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:21.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.338+0000 7f2d69ffb700 1 -- 172.21.15.158:0/866451046 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d740cb5f0 con 0x7f2d7415daa0 2023-10-16T11:17:21.424 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:21.423+0000 7f8e793c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:17:21.479 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:21.477+0000 7fd9a47fa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:17:21.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.535+0000 7f2d79642700 1 -- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 msgr2=0x7f2d5806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:21.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.535+0000 7f2d79642700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:21.539 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:21.538+0000 7f8e793c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:17:21.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.551+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d740cb5f0 con 0x7f2d7415daa0 2023-10-16T11:17:21.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.551+0000 7f2d69ffb700 1 -- 172.21.15.158:0/866451046 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v187) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d740cb5f0 con 0x7f2d7415daa0 2023-10-16T11:17:21.554 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:17:21.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 msgr2=0x7f2d5806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:21.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 msgr2=0x7f2d740c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740c9680 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00cbc0 tx=0x7f2d6c00ced0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 shutdown_connections 2023-10-16T11:17:21.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d7415daa0 0x7f2d740c9680 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d740d2370 0x7f2d74151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 --2- 172.21.15.158:0/866451046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d7415d6c0 0x7f2d740c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.553+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 >> 172.21.15.158:0/866451046 conn(0x7f2d7409c940 msgr2=0x7f2d7415c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:21.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.554+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 shutdown_connections 2023-10-16T11:17:21.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.554+0000 7f2d7b8a6700 1 -- 172.21.15.158:0/866451046 wait complete. 2023-10-16T11:17:21.570 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:17:21.602 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:21.600+0000 7fd9a47fa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:17:21.612 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:21.611+0000 7f8e793c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:17:21.679 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:21.677+0000 7fd9a47fa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:17:21.692 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:21.690+0000 7f8e793c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:17:21.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.696+0000 7f4b15d23700 1 Processor -- start 2023-10-16T11:17:21.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.696+0000 7f4b15d23700 1 -- start start 2023-10-16T11:17:21.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.697+0000 7f4b15d23700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4b1015daa0 0x7f4b100cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.697+0000 7f4b15d23700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b100d0640 0x7f4b100cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.697+0000 7f4b15d23700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b100d1ee0 con 0x7f4b1015d6c0 2023-10-16T11:17:21.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.697+0000 7f4b15d23700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b100d40a0 con 0x7f4b1015daa0 2023-10-16T11:17:21.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.697+0000 7f4b15d23700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100d12d0 con 0x7f4b100d0640 2023-10-16T11:17:21.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4b1015d6c0 0x7f4b100cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:39766/0 (socket says 172.21.15.158:39766) 2023-10-16T11:17:21.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0effd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4b1015daa0 0x7f4b100cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0f7fe700 1 -- 172.21.15.158:0/408415250 learned_addr learned my addr 172.21.15.158:0/408415250 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:21.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b100d0640 0x7f4b100cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0e7fc700 1 -- 172.21.15.158:0/408415250 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826170062 0 0) 0x7f4b100d1ee0 con 0x7f4b1015d6c0 2023-10-16T11:17:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0e7fc700 1 -- 172.21.15.158:0/408415250 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4afc0035b0 con 0x7f4b1015d6c0 2023-10-16T11:17:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0effd700 1 -- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b100d0640 msgr2=0x7f4b100cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0effd700 1 --2- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b100d0640 0x7f4b100cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0effd700 1 -- 172.21.15.158:0/408415250 >> v1:172.21.15.172:6789/0 conn(0x7f4b1015d6c0 legacy=0x7f4b100cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0effd700 1 -- 172.21.15.158:0/408415250 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b100d15b0 con 0x7f4b1015daa0 2023-10-16T11:17:21.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0ffff700 1 --2- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b100d0640 0x7f4b100cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:21.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0effd700 1 --2- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 0x7f4b100cec50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009840 tx=0x7f4af8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f73bcfab1292582a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:21.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0e7fc700 1 -- 172.21.15.158:0/408415250 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8011830 con 0x7f4b1015daa0 2023-10-16T11:17:21.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0e7fc700 1 -- 172.21.15.158:0/408415250 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af8011990 con 0x7f4b1015daa0 2023-10-16T11:17:21.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.698+0000 7f4b0e7fc700 1 -- 172.21.15.158:0/408415250 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8011d20 con 0x7f4b1015daa0 2023-10-16T11:17:21.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 -- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 msgr2=0x7f4b100cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 --2- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 0x7f4b100cec50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009840 tx=0x7f4af8009b50 comp rx=0 tx=0).stop 2023-10-16T11:17:21.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 -- 172.21.15.158:0/408415250 shutdown_connections 2023-10-16T11:17:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 --2- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b100d0640 0x7f4b100cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 --2- 172.21.15.158:0/408415250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 0x7f4b100cec50 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 -- 172.21.15.158:0/408415250 >> 172.21.15.158:0/408415250 conn(0x7f4b1009c940 msgr2=0x7f4b100d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:21.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 -- 172.21.15.158:0/408415250 shutdown_connections 2023-10-16T11:17:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.699+0000 7f4b15d23700 1 -- 172.21.15.158:0/408415250 wait complete. 2023-10-16T11:17:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.700+0000 7f4b15d23700 1 Processor -- start 2023-10-16T11:17:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.700+0000 7f4b15d23700 1 -- start start 2023-10-16T11:17:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b15d23700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b100d0640 0x7f4b100ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b15d23700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b1015d6c0 0x7f4b100cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b15d23700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 0x7f4b100c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b15d23700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100cc950 con 0x7f4b100d0640 2023-10-16T11:17:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b15d23700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b101f5ed0 con 0x7f4b1015daa0 2023-10-16T11:17:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b15d23700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100ccb40 con 0x7f4b1015d6c0 2023-10-16T11:17:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b100d0640 0x7f4b100ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b100d0640 0x7f4b100ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35692/0 (socket says 172.21.15.158:35692) 2023-10-16T11:17:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 -- 172.21.15.158:0/3856159632 learned_addr learned my addr 172.21.15.158:0/3856159632 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:21.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0ffff700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 0x7f4b100c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0effd700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b1015d6c0 0x7f4b100cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:21.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 -- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b1015d6c0 msgr2=0x7f4b100cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b1015d6c0 0x7f4b100cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 -- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 msgr2=0x7f4b100c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 0x7f4b100c7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.701+0000 7f4b0f7fe700 1 -- 172.21.15.158:0/3856159632 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080094a0 con 0x7f4b100d0640 2023-10-16T11:17:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.702+0000 7f4b0f7fe700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b100d0640 0x7f4b100ccff0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000cbc0 tx=0x7f4b0000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.702+0000 7f4b07fff700 1 -- 172.21.15.158:0/3856159632 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0000aa40 con 0x7f4b100d0640 2023-10-16T11:17:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.702+0000 7f4b07fff700 1 -- 172.21.15.158:0/3856159632 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0000aba0 con 0x7f4b100d0640 2023-10-16T11:17:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.702+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af80094a0 con 0x7f4b100d0640 2023-10-16T11:17:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.702+0000 7f4b07fff700 1 -- 172.21.15.158:0/3856159632 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00012400 con 0x7f4b100d0640 2023-10-16T11:17:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.702+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b100cb430 con 0x7f4b100d0640 2023-10-16T11:17:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.703+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b100681b0 con 0x7f4b100d0640 2023-10-16T11:17:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.703+0000 7f4b07fff700 1 -- 172.21.15.158:0/3856159632 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f4b0001b3b0 con 0x7f4b100d0640 2023-10-16T11:17:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.704+0000 7f4b07fff700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 0x7f4af006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.704+0000 7f4b0effd700 1 -- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 msgr2=0x7f4af006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.704+0000 7f4b07fff700 1 -- 172.21.15.158:0/3856159632 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f4b00089a10 con 0x7f4b100d0640 2023-10-16T11:17:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.704+0000 7f4b0effd700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 0x7f4af006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.712+0000 7f4b07fff700 1 -- 172.21.15.158:0/3856159632 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4b00054400 con 0x7f4b100d0640 2023-10-16T11:17:21.769 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:21.766+0000 7fd9a47fa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:17:21.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.905+0000 7f4b0effd700 1 -- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 msgr2=0x7f4af006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:21.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.905+0000 7f4b0effd700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 0x7f4af006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:21.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.913+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4b100cd540 con 0x7f4b100d0640 2023-10-16T11:17:21.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.914+0000 7f4b07fff700 1 -- 172.21.15.158:0/3856159632 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v187) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4b00057a20 con 0x7f4b100d0640 2023-10-16T11:17:21.916 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:17:21.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 msgr2=0x7f4af006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:21.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 0x7f4af006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b100d0640 msgr2=0x7f4b100ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b100d0640 0x7f4b100ccff0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000cbc0 tx=0x7f4b0000ced0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 shutdown_connections 2023-10-16T11:17:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b1015daa0 0x7f4b100c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b100d0640 0x7f4b100ccff0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f4af006b020 0x7f4af006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 --2- 172.21.15.158:0/3856159632 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b1015d6c0 0x7f4b100cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 >> 172.21.15.158:0/3856159632 conn(0x7f4b1009c940 msgr2=0x7f4b1006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 shutdown_connections 2023-10-16T11:17:21.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:21.916+0000 7f4b15d23700 1 -- 172.21.15.158:0/3856159632 wait complete. 2023-10-16T11:17:21.931 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:17:22.063 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:22.061+0000 7f8e793c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:17:22.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.095+0000 7f2e85093700 1 Processor -- start 2023-10-16T11:17:22.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.095+0000 7f2e85093700 1 -- start start 2023-10-16T11:17:22.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.096+0000 7f2e85093700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e8015b540 0x7f2e8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.096+0000 7f2e85093700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8015ee60 con 0x7f2e8006f180 2023-10-16T11:17:22.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.096+0000 7f2e85093700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8015e2e0 con 0x7f2e8006afe0 2023-10-16T11:17:22.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.096+0000 7f2e85093700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e8015b920 con 0x7f2e8015b540 2023-10-16T11:17:22.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.096+0000 7f2e7e59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2e8006f180 0x7f2e8006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:35912/0 (socket says 172.21.15.158:35912) 2023-10-16T11:17:22.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.096+0000 7f2e7e59c700 1 -- 172.21.15.158:0/3819623847 learned_addr learned my addr 172.21.15.158:0/3819623847 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:22.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.096+0000 7f2e7ed9d700 1 --2- 172.21.15.158:0/3819623847 >> v2:172.21.15.172:3300/0 conn(0x7f2e8015b540 0x7f2e8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7dd9b700 1 -- 172.21.15.158:0/3819623847 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380374402 0 0) 0x7f2e8015ee60 con 0x7f2e8006f180 2023-10-16T11:17:22.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7dd9b700 1 -- 172.21.15.158:0/3819623847 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e680035b0 con 0x7f2e8006f180 2023-10-16T11:17:22.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7ed9d700 1 -- 172.21.15.158:0/3819623847 >> v1:172.21.15.172:6789/0 conn(0x7f2e8006afe0 legacy=0x7f2e80161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7ed9d700 1 -- 172.21.15.158:0/3819623847 >> v1:172.21.15.158:6789/0 conn(0x7f2e8006f180 legacy=0x7f2e8006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7ed9d700 1 -- 172.21.15.158:0/3819623847 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8015e5c0 con 0x7f2e8015b540 2023-10-16T11:17:22.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7ed9d700 1 --2- 172.21.15.158:0/3819623847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 0x7f2e8006d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009840 tx=0x7f2e70009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6659aae2f85f3fd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:22.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7dd9b700 1 -- 172.21.15.158:0/3819623847 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70011830 con 0x7f2e8015b540 2023-10-16T11:17:22.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7dd9b700 1 -- 172.21.15.158:0/3819623847 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70011990 con 0x7f2e8015b540 2023-10-16T11:17:22.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.097+0000 7f2e7dd9b700 1 -- 172.21.15.158:0/3819623847 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70011d20 con 0x7f2e8015b540 2023-10-16T11:17:22.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.098+0000 7f2e85093700 1 -- 172.21.15.158:0/3819623847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 msgr2=0x7f2e8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.098+0000 7f2e85093700 1 --2- 172.21.15.158:0/3819623847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 0x7f2e8006d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009840 tx=0x7f2e70009b50 comp rx=0 tx=0).stop 2023-10-16T11:17:22.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.098+0000 7f2e85093700 1 -- 172.21.15.158:0/3819623847 shutdown_connections 2023-10-16T11:17:22.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.098+0000 7f2e85093700 1 --2- 172.21.15.158:0/3819623847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 0x7f2e8006d9c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.098+0000 7f2e85093700 1 -- 172.21.15.158:0/3819623847 >> 172.21.15.158:0/3819623847 conn(0x7f2e800d03b0 msgr2=0x7f2e800d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:22.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.098+0000 7f2e85093700 1 -- 172.21.15.158:0/3819623847 shutdown_connections 2023-10-16T11:17:22.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.098+0000 7f2e85093700 1 -- 172.21.15.158:0/3819623847 wait complete. 2023-10-16T11:17:22.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.099+0000 7f2e85093700 1 Processor -- start 2023-10-16T11:17:22.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.099+0000 7f2e85093700 1 -- start start 2023-10-16T11:17:22.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e85093700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e8006afe0 0x7f2e801f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e85093700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8006f180 0x7f2e80160d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e85093700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 0x7f2e801f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e85093700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e801f3460 con 0x7f2e8006afe0 2023-10-16T11:17:22.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e85093700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e801f6c10 con 0x7f2e8015b540 2023-10-16T11:17:22.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e85093700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e801f6eb0 con 0x7f2e8006f180 2023-10-16T11:17:22.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e8006afe0 0x7f2e801f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e8006afe0 0x7f2e801f66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35742/0 (socket says 172.21.15.158:35742) 2023-10-16T11:17:22.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 -- 172.21.15.158:0/3336858902 learned_addr learned my addr 172.21.15.158:0/3336858902 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:22.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7f59e700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 0x7f2e801f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7e59c700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8006f180 0x7f2e80160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 -- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8006f180 msgr2=0x7f2e80160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8006f180 0x7f2e80160d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 -- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 msgr2=0x7f2e801f3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 0x7f2e801f3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7ed9d700 1 -- 172.21.15.158:0/3336858902 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e700094a0 con 0x7f2e8006afe0 2023-10-16T11:17:22.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.100+0000 7f2e7e59c700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8006f180 0x7f2e80160d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:22.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.101+0000 7f2e7ed9d700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e8006afe0 0x7f2e801f66d0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009b90 tx=0x7f2e70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:22.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.101+0000 7f2e677fe700 1 -- 172.21.15.158:0/3336858902 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e700036a0 con 0x7f2e8006afe0 2023-10-16T11:17:22.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.101+0000 7f2e677fe700 1 -- 172.21.15.158:0/3336858902 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7001e070 con 0x7f2e8006afe0 2023-10-16T11:17:22.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.101+0000 7f2e677fe700 1 -- 172.21.15.158:0/3336858902 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70003a20 con 0x7f2e8006afe0 2023-10-16T11:17:22.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.101+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e801f7190 con 0x7f2e8006afe0 2023-10-16T11:17:22.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.101+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e801f3080 con 0x7f2e8006afe0 2023-10-16T11:17:22.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.102+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8015eb60 con 0x7f2e8006afe0 2023-10-16T11:17:22.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.102+0000 7f2e677fe700 1 -- 172.21.15.158:0/3336858902 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f2e70003b80 con 0x7f2e8006afe0 2023-10-16T11:17:22.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.103+0000 7f2e677fe700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 0x7f2e6c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.103+0000 7f2e7e59c700 1 -- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 msgr2=0x7f2e6c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:22.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.103+0000 7f2e677fe700 1 -- 172.21.15.158:0/3336858902 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f2e7008acb0 con 0x7f2e8006afe0 2023-10-16T11:17:22.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.103+0000 7f2e7e59c700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 0x7f2e6c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:22.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.107+0000 7f2e677fe700 1 -- 172.21.15.158:0/3336858902 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e8015eb60 con 0x7f2e8006afe0 2023-10-16T11:17:22.158 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:22.156+0000 7fd9a47fa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:17:22.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.304+0000 7f2e7e59c700 1 -- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 msgr2=0x7f2e6c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:22.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.304+0000 7f2e7e59c700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 0x7f2e6c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:22.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.306+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2e8015eb60 con 0x7f2e8006afe0 2023-10-16T11:17:22.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.306+0000 7f2e677fe700 1 -- 172.21.15.158:0/3336858902 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v187) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2e8015eb60 con 0x7f2e8006afe0 2023-10-16T11:17:22.309 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:17:22.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 msgr2=0x7f2e6c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:22.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 0x7f2e6c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e8006afe0 msgr2=0x7f2e801f66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e8006afe0 0x7f2e801f66d0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009b90 tx=0x7f2e70004750 comp rx=0 tx=0).stop 2023-10-16T11:17:22.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 shutdown_connections 2023-10-16T11:17:22.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e8015b540 0x7f2e801f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e8006afe0 0x7f2e801f66d0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f2e6c06b020 0x7f2e6c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 --2- 172.21.15.158:0/3336858902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8006f180 0x7f2e80160d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 >> 172.21.15.158:0/3336858902 conn(0x7f2e800d03b0 msgr2=0x7f2e8006a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:22.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 shutdown_connections 2023-10-16T11:17:22.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.309+0000 7f2e85093700 1 -- 172.21.15.158:0/3336858902 wait complete. 2023-10-16T11:17:22.324 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:17:22.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.444+0000 7f9d0fe52700 1 Processor -- start 2023-10-16T11:17:22.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.445+0000 7f9d0fe52700 1 -- start start 2023-10-16T11:17:22.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.445+0000 7f9d0fe52700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d0815b650 0x7f9d0815fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.445+0000 7f9d0fe52700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b4320 0x7f9d080b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.445+0000 7f9d0fe52700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d080cf550 con 0x7f9d080b47a0 2023-10-16T11:17:22.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.445+0000 7f9d0fe52700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d080cc6d0 con 0x7f9d0815b650 2023-10-16T11:17:22.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.445+0000 7f9d0fe52700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080cf2b0 con 0x7f9d080b4320 2023-10-16T11:17:22.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d0815b650 0x7f9d0815fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d0815b650 0x7f9d0815fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35750/0 (socket says 172.21.15.158:35750) 2023-10-16T11:17:22.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 -- 172.21.15.158:0/3957373631 learned_addr learned my addr 172.21.15.158:0/3957373631 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:22.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0d3ed700 1 --2- 172.21.15.158:0/3957373631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b4320 0x7f9d080b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 -- 172.21.15.158:0/3957373631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b4320 msgr2=0x7f9d080b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:22.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 --2- 172.21.15.158:0/3957373631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b4320 0x7f9d080b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 -- 172.21.15.158:0/3957373631 >> v1:172.21.15.172:6789/0 conn(0x7f9d080b47a0 legacy=0x7f9d080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 -- 172.21.15.158:0/3957373631 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d081600f0 con 0x7f9d0815b650 2023-10-16T11:17:22.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0dbee700 1 --2- 172.21.15.158:0/3957373631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0815fbb0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f9d040098f0 tx=0x7f9d04009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8eb5f9705f7453b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:22.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0cbec700 1 -- 172.21.15.158:0/3957373631 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04011830 con 0x7f9d0815b650 2023-10-16T11:17:22.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0cbec700 1 -- 172.21.15.158:0/3957373631 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04011990 con 0x7f9d0815b650 2023-10-16T11:17:22.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.446+0000 7f9d0cbec700 1 -- 172.21.15.158:0/3957373631 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04011d20 con 0x7f9d0815b650 2023-10-16T11:17:22.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3957373631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 msgr2=0x7f9d0815fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3957373631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0815fbb0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f9d040098f0 tx=0x7f9d04009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3957373631 shutdown_connections 2023-10-16T11:17:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3957373631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0815fbb0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3957373631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b4320 0x7f9d080b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3957373631 >> 172.21.15.158:0/3957373631 conn(0x7f9d080cdf50 msgr2=0x7f9d080ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3957373631 shutdown_connections 2023-10-16T11:17:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.447+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3957373631 wait complete. 2023-10-16T11:17:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.448+0000 7f9d0fe52700 1 Processor -- start 2023-10-16T11:17:22.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.448+0000 7f9d0fe52700 1 -- start start 2023-10-16T11:17:22.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0fe52700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d080b4320 0x7f9d0806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0fe52700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b47a0 0x7f9d081f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0fe52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0fe52700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0806da30 con 0x7f9d0815b650 2023-10-16T11:17:22.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0fe52700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d081f36e0 con 0x7f9d080b4320 2023-10-16T11:17:22.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0fe52700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d081f39c0 con 0x7f9d080b47a0 2023-10-16T11:17:22.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0e3ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0e3ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35760/0 (socket says 172.21.15.158:35760) 2023-10-16T11:17:22.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0d3ed700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b47a0 0x7f9d081f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0e3ef700 1 -- 172.21.15.158:0/3574628710 learned_addr learned my addr 172.21.15.158:0/3574628710 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:22.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.449+0000 7f9d0dbee700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d080b4320 0x7f9d0806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:22.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0e3ef700 1 -- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b47a0 msgr2=0x7f9d081f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0e3ef700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b47a0 0x7f9d081f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0e3ef700 1 -- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d080b4320 msgr2=0x7f9d0806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0e3ef700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d080b4320 0x7f9d0806d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0e3ef700 1 -- 172.21.15.158:0/3574628710 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04009550 con 0x7f9d0815b650 2023-10-16T11:17:22.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0d3ed700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b47a0 0x7f9d081f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:22.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0e3ef700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0806b9e0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000cc50 tx=0x7f9d0000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:22.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9cfe7fc700 1 -- 172.21.15.158:0/3574628710 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00003c00 con 0x7f9d0815b650 2023-10-16T11:17:22.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9cfe7fc700 1 -- 172.21.15.158:0/3574628710 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d00003d60 con 0x7f9d0815b650 2023-10-16T11:17:22.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d081f3ca0 con 0x7f9d0815b650 2023-10-16T11:17:22.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9cfe7fc700 1 -- 172.21.15.158:0/3574628710 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000aab0 con 0x7f9d0815b650 2023-10-16T11:17:22.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.450+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d081f7160 con 0x7f9d0815b650 2023-10-16T11:17:22.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.451+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d081f6560 con 0x7f9d0815b650 2023-10-16T11:17:22.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.452+0000 7f9cfe7fc700 1 -- 172.21.15.158:0/3574628710 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f9d0000ac10 con 0x7f9d0815b650 2023-10-16T11:17:22.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.452+0000 7f9cfe7fc700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 0x7f9cec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:22.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.452+0000 7f9d0dbee700 1 -- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 msgr2=0x7f9cec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:22.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.452+0000 7f9cfe7fc700 1 -- 172.21.15.158:0/3574628710 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f9d00089d90 con 0x7f9d0815b650 2023-10-16T11:17:22.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.452+0000 7f9d0dbee700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 0x7f9cec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:22.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.457+0000 7f9cfe7fc700 1 -- 172.21.15.158:0/3574628710 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d081f6560 con 0x7f9d0815b650 2023-10-16T11:17:22.632 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:22.630+0000 7f8e793c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:17:22.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.653+0000 7f9d0dbee700 1 -- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 msgr2=0x7f9cec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:22.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.653+0000 7f9d0dbee700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 0x7f9cec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:22.701 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:22.700+0000 7f8e793c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:17:22.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.709+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9d081f6940 con 0x7f9d0815b650 2023-10-16T11:17:22.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.709+0000 7f9cfe7fc700 1 -- 172.21.15.158:0/3574628710 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v187) v1 ==== 81+0+951 (secure 0 0 0) 0x7f9d081f6940 con 0x7f9d0815b650 2023-10-16T11:17:22.711 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2531840430 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.711 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2719309194 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.711 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2065912370 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.712 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1399927564 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.712 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/129037312 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.712 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/129037312 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.712 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2545298209 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.712 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/549775613 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.713 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/827805412 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.713 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2974007506 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.713 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/4132457601 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.713 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/1491011698 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.713 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3186172973 2023-10-17T11:12:22.309907+0000 2023-10-16T11:17:22.713 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/1491011698 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.714 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3804388680 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.714 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/411715851 2023-10-17T11:01:31.624108+0000 2023-10-16T11:17:22.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 msgr2=0x7f9cec06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:22.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 0x7f9cec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 msgr2=0x7f9d0806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:22.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0806b9e0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000cc50 tx=0x7f9d0000cf60 comp rx=0 tx=0).stop 2023-10-16T11:17:22.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 shutdown_connections 2023-10-16T11:17:22.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0815b650 0x7f9d0806b9e0 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d080b4320 0x7f9d0806d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f9cec06b020 0x7f9cec06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 --2- 172.21.15.158:0/3574628710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d080b47a0 0x7f9d081f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:22.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 >> 172.21.15.158:0/3574628710 conn(0x7f9d080cdf50 msgr2=0x7f9d080d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:22.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 shutdown_connections 2023-10-16T11:17:22.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:22.712+0000 7f9d0fe52700 1 -- 172.21.15.158:0/3574628710 wait complete. 2023-10-16T11:17:22.717 INFO:teuthology.orchestra.run.smithi158.stderr:listed 16 entries 2023-10-16T11:17:22.727 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:17:22.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:22.728 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:17:22.760 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:22.759+0000 7fd9a47fa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:17:22.781 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:17:22.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:22.781 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:17:22.836 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:22.835+0000 7fd9a47fa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:17:22.837 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:17:22.838 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:22.855 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:17:22.855 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:17:22.855 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:22.895 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:17:22.895 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:17:22.895 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:22.913 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:17:22.913 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:17:23.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.058+0000 7f0288c3f700 1 Processor -- start 2023-10-16T11:17:23.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.058+0000 7f0288c3f700 1 -- start start 2023-10-16T11:17:23.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0288c3f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f028406ab40 0x7f028406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0288c3f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f028406e5d0 0x7f0284071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0288c3f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 0x7f02840722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0288c3f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0284071e50 con 0x7f028406e5d0 2023-10-16T11:17:23.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0288c3f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0284161270 con 0x7f028406ab40 2023-10-16T11:17:23.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0288c3f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0284161510 con 0x7f028406af20 2023-10-16T11:17:23.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f028406e5d0 0x7f0284071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f028406e5d0 0x7f0284071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35766/0 (socket says 172.21.15.158:35766) 2023-10-16T11:17:23.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 -- 172.21.15.158:0/1193568945 learned_addr learned my addr 172.21.15.158:0/1193568945 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:23.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 -- 172.21.15.158:0/1193568945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 msgr2=0x7f02840722a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T11:17:23.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 --2- 172.21.15.158:0/1193568945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 0x7f02840722a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 -- 172.21.15.158:0/1193568945 >> v2:172.21.15.172:3300/0 conn(0x7f028406ab40 msgr2=0x7f028406d820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T11:17:23.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 --2- 172.21.15.158:0/1193568945 >> v2:172.21.15.172:3300/0 conn(0x7f028406ab40 0x7f028406d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 -- 172.21.15.158:0/1193568945 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02841617f0 con 0x7f028406e5d0 2023-10-16T11:17:23.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f0281d9b700 1 --2- 172.21.15.158:0/1193568945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406e5d0 0x7f0284071910 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f0284074d70 tx=0x7f0278009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f8dc07948a3570 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:23.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f028159a700 1 -- 172.21.15.158:0/1193568945 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027800b870 con 0x7f028406e5d0 2023-10-16T11:17:23.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f028159a700 1 -- 172.21.15.158:0/1193568945 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027800b9d0 con 0x7f028406e5d0 2023-10-16T11:17:23.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.059+0000 7f028159a700 1 -- 172.21.15.158:0/1193568945 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027800bd60 con 0x7f028406e5d0 2023-10-16T11:17:23.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 -- 172.21.15.158:0/1193568945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406e5d0 msgr2=0x7f0284071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1193568945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406e5d0 0x7f0284071910 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f0284074d70 tx=0x7f0278009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 -- 172.21.15.158:0/1193568945 shutdown_connections 2023-10-16T11:17:23.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1193568945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 0x7f02840722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1193568945 >> v2:172.21.15.172:3300/0 conn(0x7f028406ab40 0x7f028406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1193568945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406e5d0 0x7f0284071910 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 -- 172.21.15.158:0/1193568945 >> 172.21.15.158:0/1193568945 conn(0x7f02840d02b0 msgr2=0x7f02840d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:23.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 -- 172.21.15.158:0/1193568945 shutdown_connections 2023-10-16T11:17:23.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.060+0000 7f0288c3f700 1 -- 172.21.15.158:0/1193568945 wait complete. 2023-10-16T11:17:23.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.061+0000 7f0288c3f700 1 Processor -- start 2023-10-16T11:17:23.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.061+0000 7f0288c3f700 1 -- start start 2023-10-16T11:17:23.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0288c3f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406ab40 0x7f0284160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0288c3f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 0x7f028414b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0288c3f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f028406e5d0 0x7f0284155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0288c3f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840b22d0 con 0x7f028406ab40 2023-10-16T11:17:23.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0288c3f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840c3e20 con 0x7f028406e5d0 2023-10-16T11:17:23.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0288c3f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0284160eb0 con 0x7f028406af20 2023-10-16T11:17:23.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f028259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406ab40 0x7f0284160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f028259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406ab40 0x7f0284160970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35772/0 (socket says 172.21.15.158:35772) 2023-10-16T11:17:23.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f028259c700 1 -- 172.21.15.158:0/1964155259 learned_addr learned my addr 172.21.15.158:0/1964155259 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:23.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0281d9b700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 0x7f028414b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.062+0000 7f0282d9d700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f028406e5d0 0x7f0284155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f028259c700 1 -- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 msgr2=0x7f028414b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f028259c700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 0x7f028414b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f028259c700 1 -- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f028406e5d0 msgr2=0x7f0284155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f028259c700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f028406e5d0 0x7f0284155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f028259c700 1 -- 172.21.15.158:0/1964155259 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0278009550 con 0x7f028406ab40 2023-10-16T11:17:23.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f028259c700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406ab40 0x7f0284160970 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f027400b550 tx=0x7f027400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f0272ffd700 1 -- 172.21.15.158:0/1964155259 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274011830 con 0x7f028406ab40 2023-10-16T11:17:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f0272ffd700 1 -- 172.21.15.158:0/1964155259 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02740119b0 con 0x7f028406ab40 2023-10-16T11:17:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f0272ffd700 1 -- 172.21.15.158:0/1964155259 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274011d60 con 0x7f028406ab40 2023-10-16T11:17:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02840b1a30 con 0x7f028406ab40 2023-10-16T11:17:23.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.063+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02840b1ff0 con 0x7f028406ab40 2023-10-16T11:17:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.064+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02840c3c80 con 0x7f028406ab40 2023-10-16T11:17:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.065+0000 7f0272ffd700 1 -- 172.21.15.158:0/1964155259 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f027401a400 con 0x7f028406ab40 2023-10-16T11:17:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.065+0000 7f0272ffd700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 0x7f026406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.065+0000 7f0281d9b700 1 -- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 msgr2=0x7f026406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:23.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.065+0000 7f0272ffd700 1 -- 172.21.15.158:0/1964155259 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f0274089cf0 con 0x7f028406ab40 2023-10-16T11:17:23.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.065+0000 7f0281d9b700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 0x7f026406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:23.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.069+0000 7f0272ffd700 1 -- 172.21.15.158:0/1964155259 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02840c3c80 con 0x7f028406ab40 2023-10-16T11:17:23.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.266+0000 7f0281d9b700 1 -- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 msgr2=0x7f026406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:23.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.266+0000 7f0281d9b700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 0x7f026406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:23.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.314+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02840c3c80 con 0x7f028406ab40 2023-10-16T11:17:23.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.315+0000 7f0272ffd700 1 -- 172.21.15.158:0/1964155259 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4930 (secure 0 0 0) 0x7f02840c3c80 con 0x7f028406ab40 2023-10-16T11:17:23.317 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:23.317 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3580267390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3580267390},{"type":"v1","addr":"172.21.15.172:6834","nonce":3580267390}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":9328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3433152480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3433152480},{"type":"v1","addr":"172.21.15.172:6835","nonce":3433152480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1170239465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1170239465},{"type":"v1","addr":"172.21.15.158:6836","nonce":1170239465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:01:17.640665+0000","modified":"2023-10-16T11:01:20.662098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10449":{"gid":10449,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/773046671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":773046671},{"type":"v1","addr":"172.21.15.158:6837","nonce":773046671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T11:17:23.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 msgr2=0x7f026406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:23.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 0x7f026406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406ab40 msgr2=0x7f0284160970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406ab40 0x7f0284160970 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f027400b550 tx=0x7f027400b860 comp rx=0 tx=0).stop 2023-10-16T11:17:23.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 shutdown_connections 2023-10-16T11:17:23.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f028406e5d0 0x7f0284155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f028406ab40 0x7f0284160970 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f026406b020 0x7f026406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 --2- 172.21.15.158:0/1964155259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f028406af20 0x7f028414b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 >> 172.21.15.158:0/1964155259 conn(0x7f02840d02b0 msgr2=0x7f0284072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:23.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 shutdown_connections 2023-10-16T11:17:23.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.317+0000 7f0288c3f700 1 -- 172.21.15.158:0/1964155259 wait complete. 2023-10-16T11:17:23.322 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 94 2023-10-16T11:17:23.332 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:23.340 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:23.338+0000 7f8e793c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:23.417 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:23.415+0000 7f8e793c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:17:23.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.457+0000 7f65733f1700 1 Processor -- start 2023-10-16T11:17:23.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.457+0000 7f65733f1700 1 -- start start 2023-10-16T11:17:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f65733f1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f656c0cc1b0 0x7f656c0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f65733f1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f656c15f0f0 0x7f656c0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f65733f1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c15b930 con 0x7f656c0ce530 2023-10-16T11:17:23.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f65733f1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c0cf020 con 0x7f656c0cc1b0 2023-10-16T11:17:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f65733f1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c15f5f0 con 0x7f656c15f0f0 2023-10-16T11:17:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f656c0cc1b0 0x7f656c0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f656c0cc1b0 0x7f656c0b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35784/0 (socket says 172.21.15.158:35784) 2023-10-16T11:17:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 -- 172.21.15.158:0/405930923 learned_addr learned my addr 172.21.15.158:0/405930923 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:23.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657198e700 1 --2- 172.21.15.158:0/405930923 >> v2:172.21.15.172:3300/0 conn(0x7f656c15f0f0 0x7f656c0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 -- 172.21.15.158:0/405930923 >> v2:172.21.15.172:3300/0 conn(0x7f656c15f0f0 msgr2=0x7f656c0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:23.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 --2- 172.21.15.158:0/405930923 >> v2:172.21.15.172:3300/0 conn(0x7f656c15f0f0 0x7f656c0cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 -- 172.21.15.158:0/405930923 >> v1:172.21.15.172:6789/0 conn(0x7f656c0ce530 legacy=0x7f656c0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 -- 172.21.15.158:0/405930923 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c15f8d0 con 0x7f656c0cc1b0 2023-10-16T11:17:23.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.458+0000 7f657098c700 1 --2- 172.21.15.158:0/405930923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c0cc1b0 0x7f656c0b3b20 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f656c144d40 tx=0x7f655c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc784932083c8cf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f6563fff700 1 -- 172.21.15.158:0/405930923 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c011830 con 0x7f656c0cc1b0 2023-10-16T11:17:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f6563fff700 1 -- 172.21.15.158:0/405930923 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c011990 con 0x7f656c0cc1b0 2023-10-16T11:17:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f6563fff700 1 -- 172.21.15.158:0/405930923 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c011d20 con 0x7f656c0cc1b0 2023-10-16T11:17:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 -- 172.21.15.158:0/405930923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c0cc1b0 msgr2=0x7f656c0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 --2- 172.21.15.158:0/405930923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c0cc1b0 0x7f656c0b3b20 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f656c144d40 tx=0x7f655c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 -- 172.21.15.158:0/405930923 shutdown_connections 2023-10-16T11:17:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 --2- 172.21.15.158:0/405930923 >> v2:172.21.15.172:3300/0 conn(0x7f656c15f0f0 0x7f656c0cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 --2- 172.21.15.158:0/405930923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c0cc1b0 0x7f656c0b3b20 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 -- 172.21.15.158:0/405930923 >> 172.21.15.158:0/405930923 conn(0x7f656c06a8d0 msgr2=0x7f656c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:23.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 -- 172.21.15.158:0/405930923 shutdown_connections 2023-10-16T11:17:23.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.459+0000 7f65733f1700 1 -- 172.21.15.158:0/405930923 wait complete. 2023-10-16T11:17:23.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.461+0000 7f65733f1700 1 Processor -- start 2023-10-16T11:17:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.461+0000 7f65733f1700 1 -- start start 2023-10-16T11:17:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f65733f1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f656c0cc1b0 0x7f656c15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f65733f1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f656c0ce530 0x7f656c0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f65733f1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c15f0f0 0x7f656c0cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f65733f1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0cb520 con 0x7f656c15f0f0 2023-10-16T11:17:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f65733f1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0cb940 con 0x7f656c0ce530 2023-10-16T11:17:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f65733f1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c1f3620 con 0x7f656c0cc1b0 2023-10-16T11:17:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c15f0f0 0x7f656c0cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657098c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f656c0ce530 0x7f656c0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c15f0f0 0x7f656c0cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35792/0 (socket says 172.21.15.158:35792) 2023-10-16T11:17:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657118d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f656c0cc1b0 0x7f656c15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657098c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f656c0ce530 0x7f656c0cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:36970/0 (socket says 172.21.15.158:36970) 2023-10-16T11:17:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 -- 172.21.15.158:0/754049327 learned_addr learned my addr 172.21.15.158:0/754049327 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 -- 172.21.15.158:0/754049327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f656c0cc1b0 msgr2=0x7f656c15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f656c0cc1b0 0x7f656c15aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 -- 172.21.15.158:0/754049327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f656c0ce530 msgr2=0x7f656c0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f656c0ce530 0x7f656c0cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 -- 172.21.15.158:0/754049327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c009550 con 0x7f656c15f0f0 2023-10-16T11:17:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657098c700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f656c0ce530 0x7f656c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:23.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.462+0000 7f657198e700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c15f0f0 0x7f656c0cfa10 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f6564009e30 tx=0x7f656400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:23.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.463+0000 7f6561ffb700 1 -- 172.21.15.158:0/754049327 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564009ca0 con 0x7f656c15f0f0 2023-10-16T11:17:23.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.463+0000 7f6561ffb700 1 -- 172.21.15.158:0/754049327 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656400f040 con 0x7f656c15f0f0 2023-10-16T11:17:23.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.463+0000 7f6561ffb700 1 -- 172.21.15.158:0/754049327 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564014480 con 0x7f656c15f0f0 2023-10-16T11:17:23.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.463+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c0ca8a0 con 0x7f656c15f0f0 2023-10-16T11:17:23.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.463+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c0cae60 con 0x7f656c15f0f0 2023-10-16T11:17:23.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.464+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c0cb660 con 0x7f656c15f0f0 2023-10-16T11:17:23.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.464+0000 7f6561ffb700 1 -- 172.21.15.158:0/754049327 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f65640145e0 con 0x7f656c15f0f0 2023-10-16T11:17:23.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.465+0000 7f6561ffb700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 0x7f655006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.465+0000 7f657118d700 1 -- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 msgr2=0x7f655006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:23.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.465+0000 7f6561ffb700 1 -- 172.21.15.158:0/754049327 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f656408adb0 con 0x7f656c15f0f0 2023-10-16T11:17:23.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.465+0000 7f657118d700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 0x7f655006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:23.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.469+0000 7f6561ffb700 1 -- 172.21.15.158:0/754049327 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6564055760 con 0x7f656c15f0f0 2023-10-16T11:17:23.509 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:23.507+0000 7fd9a47fa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:23.562 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:23.560+0000 7f8e793c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:23.590 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:23.588+0000 7fd9a47fa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:17:23.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.666+0000 7f657118d700 1 -- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 msgr2=0x7f655006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:23.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.666+0000 7f657118d700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 0x7f655006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:23.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.719+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f656c0cb660 con 0x7f656c15f0f0 2023-10-16T11:17:23.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.720+0000 7f6561ffb700 1 -- 172.21.15.158:0/754049327 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+18547 (secure 0 0 0) 0x7f656c0cb660 con 0x7f656c15f0f0 2023-10-16T11:17:23.723 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:23.723 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":188,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:23.595580+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:01:16.077848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T11:01:16.598638+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":188,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2531840430":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2719309194":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2065912370":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/1399927564":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6833/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2545298209":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/549775613":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/827805412":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/2974007506":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/4132457601":"2023-10-17T11:01:31.624108+0000","172.21.15.158:6833/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3186172973":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3804388680":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/411715851":"2023-10-17T11:01:31.624108+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:23.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.724+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 msgr2=0x7f655006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:23.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.724+0000 7f65733f1700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 0x7f655006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.724+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c15f0f0 msgr2=0x7f656c0cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.724+0000 7f65733f1700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c15f0f0 0x7f656c0cfa10 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f6564009e30 tx=0x7f656400c3a0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.725+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 shutdown_connections 2023-10-16T11:17:23.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.725+0000 7f65733f1700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f656c15f0f0 0x7f656c0cfa10 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.725+0000 7f65733f1700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f655006b020 0x7f655006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.725+0000 7f65733f1700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f656c0cc1b0 0x7f656c15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.725+0000 7f65733f1700 1 --2- 172.21.15.158:0/754049327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f656c0ce530 0x7f656c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.725+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 >> 172.21.15.158:0/754049327 conn(0x7f656c06a8d0 msgr2=0x7f656c0b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:23.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.730+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 shutdown_connections 2023-10-16T11:17:23.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.732+0000 7f65733f1700 1 -- 172.21.15.158:0/754049327 wait complete. 2023-10-16T11:17:23.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:01:16.077848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:01:16.598638+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 188, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:17:23.760 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:17:23.797 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:23.796+0000 7fd9a47fa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:23.833 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:23.831+0000 7f8e793c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:17:23.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.913+0000 7f8517b6b700 1 Processor -- start 2023-10-16T11:17:23.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.914+0000 7f8517b6b700 1 -- start start 2023-10-16T11:17:23.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.914+0000 7f8517b6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f851015d990 0x7f851006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.914+0000 7f8517b6b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85100cbaf0 0x7f85100cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.914+0000 7f8517b6b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851015dd70 con 0x7f8510070300 2023-10-16T11:17:23.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8517b6b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85101616b0 con 0x7f851015d990 2023-10-16T11:17:23.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8517b6b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85100ced70 con 0x7f85100cbaf0 2023-10-16T11:17:23.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515907700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f851015d990 0x7f851006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515106700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8510070300 0x7f8510161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36008/0 (socket says 172.21.15.158:36008) 2023-10-16T11:17:23.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515106700 1 -- 172.21.15.158:0/3104177989 learned_addr learned my addr 172.21.15.158:0/3104177989 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:23.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8516108700 1 --2- 172.21.15.158:0/3104177989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85100cbaf0 0x7f85100cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515907700 1 -- 172.21.15.158:0/3104177989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85100cbaf0 msgr2=0x7f85100cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:23.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515907700 1 --2- 172.21.15.158:0/3104177989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85100cbaf0 0x7f85100cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515907700 1 -- 172.21.15.158:0/3104177989 >> v1:172.21.15.158:6789/0 conn(0x7f8510070300 legacy=0x7f8510161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515907700 1 -- 172.21.15.158:0/3104177989 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8510070780 con 0x7f851015d990 2023-10-16T11:17:23.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8514905700 1 -- 172.21.15.158:0/3104177989 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975917944 0 0) 0x7f8500001110 con 0x7f8510070300 2023-10-16T11:17:23.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8515907700 1 --2- 172.21.15.158:0/3104177989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f851015d990 0x7f851006d5d0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f850c009870 tx=0x7f850c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59505b51f0302f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:23.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8514905700 1 -- 172.21.15.158:0/3104177989 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c011830 con 0x7f851015d990 2023-10-16T11:17:23.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8514905700 1 -- 172.21.15.158:0/3104177989 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c011990 con 0x7f851015d990 2023-10-16T11:17:23.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.915+0000 7f8514905700 1 -- 172.21.15.158:0/3104177989 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c011d20 con 0x7f851015d990 2023-10-16T11:17:23.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 -- 172.21.15.158:0/3104177989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f851015d990 msgr2=0x7f851006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 --2- 172.21.15.158:0/3104177989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f851015d990 0x7f851006d5d0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f850c009870 tx=0x7f850c009b80 comp rx=0 tx=0).stop 2023-10-16T11:17:23.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 -- 172.21.15.158:0/3104177989 shutdown_connections 2023-10-16T11:17:23.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 --2- 172.21.15.158:0/3104177989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f851015d990 0x7f851006d5d0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 --2- 172.21.15.158:0/3104177989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85100cbaf0 0x7f85100cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 -- 172.21.15.158:0/3104177989 >> 172.21.15.158:0/3104177989 conn(0x7f85100cdf20 msgr2=0x7f85100ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:23.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 -- 172.21.15.158:0/3104177989 shutdown_connections 2023-10-16T11:17:23.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.916+0000 7f8517b6b700 1 -- 172.21.15.158:0/3104177989 wait complete. 2023-10-16T11:17:23.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.917+0000 7f8517b6b700 1 Processor -- start 2023-10-16T11:17:23.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.917+0000 7f8517b6b700 1 -- start start 2023-10-16T11:17:23.925 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:23.922+0000 7f8e793c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:17:23.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8517b6b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8510070300 0x7f851015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8517b6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85100cbaf0 0x7f851014ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8517b6b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f851015d990 0x7f85100cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8517b6b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101e69d0 con 0x7f85100cbaf0 2023-10-16T11:17:23.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8517b6b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101e6cb0 con 0x7f8510070300 2023-10-16T11:17:23.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8517b6b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85101e6f90 con 0x7f851015d990 2023-10-16T11:17:23.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85100cbaf0 0x7f851014ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85100cbaf0 0x7f851014ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35838/0 (socket says 172.21.15.158:35838) 2023-10-16T11:17:23.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 -- 172.21.15.158:0/2614382604 learned_addr learned my addr 172.21.15.158:0/2614382604 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:23.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8516108700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f851015d990 0x7f85100cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515907700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8510070300 0x7f851015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:23.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 -- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f851015d990 msgr2=0x7f85100cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f851015d990 0x7f85100cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 -- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8510070300 msgr2=0x7f851015a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:23.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8510070300 0x7f851015a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:23.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.918+0000 7f8515106700 1 -- 172.21.15.158:0/2614382604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0094a0 con 0x7f85100cbaf0 2023-10-16T11:17:23.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.919+0000 7f8515106700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85100cbaf0 0x7f851014ea10 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f850000c550 tx=0x7f850000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:23.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.919+0000 7f85067fc700 1 -- 172.21.15.158:0/2614382604 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500004c80 con 0x7f85100cbaf0 2023-10-16T11:17:23.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.919+0000 7f85067fc700 1 -- 172.21.15.158:0/2614382604 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500004e00 con 0x7f85100cbaf0 2023-10-16T11:17:23.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.919+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85101e7270 con 0x7f85100cbaf0 2023-10-16T11:17:23.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.919+0000 7f85067fc700 1 -- 172.21.15.158:0/2614382604 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500007310 con 0x7f85100cbaf0 2023-10-16T11:17:23.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.919+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85100b4b20 con 0x7f85100cbaf0 2023-10-16T11:17:23.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.920+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85101e74b0 con 0x7f85100cbaf0 2023-10-16T11:17:23.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.920+0000 7f85067fc700 1 -- 172.21.15.158:0/2614382604 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f8500007470 con 0x7f85100cbaf0 2023-10-16T11:17:23.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.921+0000 7f85067fc700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 0x7f84f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:23.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.921+0000 7f8515907700 1 -- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 msgr2=0x7f84f406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:23.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.921+0000 7f85067fc700 1 -- 172.21.15.158:0/2614382604 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f8500089b70 con 0x7f85100cbaf0 2023-10-16T11:17:23.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.921+0000 7f8515907700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 0x7f84f406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:23.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:23.925+0000 7f85067fc700 1 -- 172.21.15.158:0/2614382604 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8500054690 con 0x7f85100cbaf0 2023-10-16T11:17:24.066 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:24.064+0000 7fd9a47fa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:17:24.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.122+0000 7f8515907700 1 -- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 msgr2=0x7f84f406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:24.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.122+0000 7f8515907700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 0x7f84f406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:24.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.129+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f851015dd70 con 0x7f85100cbaf0 2023-10-16T11:17:24.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.129+0000 7f85067fc700 1 -- 172.21.15.158:0/2614382604 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7f851015dd70 con 0x7f85100cbaf0 2023-10-16T11:17:24.131 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:17:24.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 msgr2=0x7f84f406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:24.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 0x7f84f406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85100cbaf0 msgr2=0x7f851014ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85100cbaf0 0x7f851014ea10 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f850000c550 tx=0x7f850000c860 comp rx=0 tx=0).stop 2023-10-16T11:17:24.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 shutdown_connections 2023-10-16T11:17:24.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f84f406b020 0x7f84f406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f851015d990 0x7f85100cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8510070300 0x7f851015a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 --2- 172.21.15.158:0/2614382604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85100cbaf0 0x7f851014ea10 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 >> 172.21.15.158:0/2614382604 conn(0x7f85100cdf20 msgr2=0x7f85100b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:24.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 shutdown_connections 2023-10-16T11:17:24.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.132+0000 7f8517b6b700 1 -- 172.21.15.158:0/2614382604 wait complete. 2023-10-16T11:17:24.144 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:24.143+0000 7fd9a47fa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:17:24.146 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:17:24.198 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:24.198+0000 7f8e793c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:17:24.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.318+0000 7f10c6390700 1 Processor -- start 2023-10-16T11:17:24.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.318+0000 7f10c6390700 1 -- start start 2023-10-16T11:17:24.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.318+0000 7f10c6390700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10c006d5d0 0x7f10c0070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.318+0000 7f10c6390700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c00cbaf0 0x7f10c00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.318+0000 7f10c6390700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c015dd70 con 0x7f10c015d990 2023-10-16T11:17:24.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.318+0000 7f10c6390700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c01616b0 con 0x7f10c006d5d0 2023-10-16T11:17:24.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.318+0000 7f10c6390700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c006da50 con 0x7f10c00cbaf0 2023-10-16T11:17:24.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10bffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f10c015d990 0x7f10c0161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36022/0 (socket says 172.21.15.158:36022) 2023-10-16T11:17:24.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10bffff700 1 -- 172.21.15.158:0/2141051266 learned_addr learned my addr 172.21.15.158:0/2141051266 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:24.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10c492d700 1 --2- 172.21.15.158:0/2141051266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c00cbaf0 0x7f10c00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10bf7fe700 1 --2- 172.21.15.158:0/2141051266 >> v2:172.21.15.172:3300/0 conn(0x7f10c006d5d0 0x7f10c0070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10beffd700 1 -- 172.21.15.158:0/2141051266 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401586282 0 0) 0x7f10b0001110 con 0x7f10c015d990 2023-10-16T11:17:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10beffd700 1 -- 172.21.15.158:0/2141051266 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a80035b0 con 0x7f10c015d990 2023-10-16T11:17:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10c492d700 1 -- 172.21.15.158:0/2141051266 >> v2:172.21.15.172:3300/0 conn(0x7f10c006d5d0 msgr2=0x7f10c0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10c492d700 1 --2- 172.21.15.158:0/2141051266 >> v2:172.21.15.172:3300/0 conn(0x7f10c006d5d0 0x7f10c0070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10c492d700 1 -- 172.21.15.158:0/2141051266 >> v1:172.21.15.158:6789/0 conn(0x7f10c015d990 legacy=0x7f10c0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10c492d700 1 -- 172.21.15.158:0/2141051266 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c0070a00 con 0x7f10c00cbaf0 2023-10-16T11:17:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10bf7fe700 1 --2- 172.21.15.158:0/2141051266 >> v2:172.21.15.172:3300/0 conn(0x7f10c006d5d0 0x7f10c0070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10c492d700 1 --2- 172.21.15.158:0/2141051266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c00cbaf0 0x7f10c00cbf80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009d90 tx=0x7f10b800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97aca6a53157bac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:24.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10beffd700 1 -- 172.21.15.158:0/2141051266 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b800e070 con 0x7f10c00cbaf0 2023-10-16T11:17:24.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10beffd700 1 -- 172.21.15.158:0/2141051266 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b800bb20 con 0x7f10c00cbaf0 2023-10-16T11:17:24.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.319+0000 7f10beffd700 1 -- 172.21.15.158:0/2141051266 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8012430 con 0x7f10c00cbaf0 2023-10-16T11:17:24.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 -- 172.21.15.158:0/2141051266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c00cbaf0 msgr2=0x7f10c00cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 --2- 172.21.15.158:0/2141051266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c00cbaf0 0x7f10c00cbf80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009d90 tx=0x7f10b800b330 comp rx=0 tx=0).stop 2023-10-16T11:17:24.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 -- 172.21.15.158:0/2141051266 shutdown_connections 2023-10-16T11:17:24.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 --2- 172.21.15.158:0/2141051266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c00cbaf0 0x7f10c00cbf80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 --2- 172.21.15.158:0/2141051266 >> v2:172.21.15.172:3300/0 conn(0x7f10c006d5d0 0x7f10c0070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 -- 172.21.15.158:0/2141051266 >> 172.21.15.158:0/2141051266 conn(0x7f10c00cdf20 msgr2=0x7f10c00ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:24.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 -- 172.21.15.158:0/2141051266 shutdown_connections 2023-10-16T11:17:24.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.320+0000 7f10c6390700 1 -- 172.21.15.158:0/2141051266 wait complete. 2023-10-16T11:17:24.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.321+0000 7f10c6390700 1 Processor -- start 2023-10-16T11:17:24.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.321+0000 7f10c6390700 1 -- start start 2023-10-16T11:17:24.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10c6390700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c006d5d0 0x7f10c0159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10c6390700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c00cbaf0 0x7f10c00c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10c6390700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c015d990 0x7f10c00c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10c6390700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00c7be0 con 0x7f10c006d5d0 2023-10-16T11:17:24.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10c6390700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00c4e70 con 0x7f10c00cbaf0 2023-10-16T11:17:24.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10c6390700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00c5e80 con 0x7f10c015d990 2023-10-16T11:17:24.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c006d5d0 0x7f10c0159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c006d5d0 0x7f10c0159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35852/0 (socket says 172.21.15.158:35852) 2023-10-16T11:17:24.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10bffff700 1 -- 172.21.15.158:0/2417981918 learned_addr learned my addr 172.21.15.158:0/2417981918 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:24.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10c492d700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c015d990 0x7f10c00c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.322+0000 7f10bf7fe700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c00cbaf0 0x7f10c00c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bffff700 1 -- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c015d990 msgr2=0x7f10c00c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bffff700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c015d990 0x7f10c00c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bffff700 1 -- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c00cbaf0 msgr2=0x7f10c00c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bffff700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c00cbaf0 0x7f10c00c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bffff700 1 -- 172.21.15.158:0/2417981918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b40094a0 con 0x7f10c006d5d0 2023-10-16T11:17:24.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bf7fe700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c00cbaf0 0x7f10c00c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:24.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bffff700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c006d5d0 0x7f10c0159de0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f10b0007fd0 tx=0x7f10b000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:24.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bcff9700 1 -- 172.21.15.158:0/2417981918 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0005d70 con 0x7f10c006d5d0 2023-10-16T11:17:24.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bcff9700 1 -- 172.21.15.158:0/2417981918 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b00070d0 con 0x7f10c006d5d0 2023-10-16T11:17:24.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b8009a70 con 0x7f10c006d5d0 2023-10-16T11:17:24.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10bcff9700 1 -- 172.21.15.158:0/2417981918 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0007460 con 0x7f10c006d5d0 2023-10-16T11:17:24.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.323+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c00c53a0 con 0x7f10c006d5d0 2023-10-16T11:17:24.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.324+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c00681b0 con 0x7f10c006d5d0 2023-10-16T11:17:24.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.324+0000 7f10bcff9700 1 -- 172.21.15.158:0/2417981918 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f10b00075c0 con 0x7f10c006d5d0 2023-10-16T11:17:24.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.325+0000 7f10bcff9700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 0x7f10ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.325+0000 7f10bf7fe700 1 -- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 msgr2=0x7f10ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:24.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.325+0000 7f10bcff9700 1 -- 172.21.15.158:0/2417981918 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f10b0089020 con 0x7f10c006d5d0 2023-10-16T11:17:24.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.325+0000 7f10bf7fe700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 0x7f10ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:24.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.333+0000 7f10bcff9700 1 -- 172.21.15.158:0/2417981918 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10b00539e0 con 0x7f10c006d5d0 2023-10-16T11:17:24.420 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:24.419+0000 7f8e793c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:17:24.430 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:24.429+0000 7fd9a47fa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:17:24.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.526+0000 7f10bf7fe700 1 -- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 msgr2=0x7f10ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:24.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.526+0000 7f10bf7fe700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 0x7f10ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:24.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.529+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10c015dd70 con 0x7f10c006d5d0 2023-10-16T11:17:24.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.529+0000 7f10bcff9700 1 -- 172.21.15.158:0/2417981918 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v188) v1 ==== 104+0+11 (secure 0 0 0) 0x7f10c015dd70 con 0x7f10c006d5d0 2023-10-16T11:17:24.531 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:17:24.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 msgr2=0x7f10ac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:24.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 0x7f10ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c006d5d0 msgr2=0x7f10c0159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c006d5d0 0x7f10c0159de0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f10b0007fd0 tx=0x7f10b000c9e0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 shutdown_connections 2023-10-16T11:17:24.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f10ac06b020 0x7f10ac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c015d990 0x7f10c00c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c006d5d0 0x7f10c0159de0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 --2- 172.21.15.158:0/2417981918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c00cbaf0 0x7f10c00c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 >> 172.21.15.158:0/2417981918 conn(0x7f10c00cdf20 msgr2=0x7f10c00b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:24.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.531+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 shutdown_connections 2023-10-16T11:17:24.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.532+0000 7f10c6390700 1 -- 172.21.15.158:0/2417981918 wait complete. 2023-10-16T11:17:24.545 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:17:24.567 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:24.567+0000 7f8e793c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:17:24.666 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:24.665+0000 7fd9a47fa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:17:24.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.673+0000 7f46d3fa8700 1 Processor -- start 2023-10-16T11:17:24.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.673+0000 7f46d3fa8700 1 -- start start 2023-10-16T11:17:24.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d3fa8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46cc15d990 0x7f46cc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d3fa8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f46cc0cbaf0 0x7f46cc0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d3fa8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc15dd70 con 0x7f46cc070300 2023-10-16T11:17:24.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d3fa8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc1616b0 con 0x7f46cc15d990 2023-10-16T11:17:24.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d3fa8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc0ced70 con 0x7f46cc0cbaf0 2023-10-16T11:17:24.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d1d44700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46cc15d990 0x7f46cc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d1d44700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46cc15d990 0x7f46cc06d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35862/0 (socket says 172.21.15.158:35862) 2023-10-16T11:17:24.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d1d44700 1 -- 172.21.15.158:0/3278234334 learned_addr learned my addr 172.21.15.158:0/3278234334 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:24.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.674+0000 7f46d2545700 1 --2- 172.21.15.158:0/3278234334 >> v2:172.21.15.172:3300/0 conn(0x7f46cc0cbaf0 0x7f46cc0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d1d44700 1 -- 172.21.15.158:0/3278234334 >> v2:172.21.15.172:3300/0 conn(0x7f46cc0cbaf0 msgr2=0x7f46cc0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:24.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d1d44700 1 --2- 172.21.15.158:0/3278234334 >> v2:172.21.15.172:3300/0 conn(0x7f46cc0cbaf0 0x7f46cc0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d1d44700 1 -- 172.21.15.158:0/3278234334 >> v1:172.21.15.172:6789/0 conn(0x7f46cc070300 legacy=0x7f46cc161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d1d44700 1 -- 172.21.15.158:0/3278234334 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc070780 con 0x7f46cc15d990 2023-10-16T11:17:24.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d1d44700 1 --2- 172.21.15.158:0/3278234334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc06d5d0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0b73d0 tx=0x7f46c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e92793d2a69031 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:24.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d0d42700 1 -- 172.21.15.158:0/3278234334 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8011830 con 0x7f46cc15d990 2023-10-16T11:17:24.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d0d42700 1 -- 172.21.15.158:0/3278234334 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c8011990 con 0x7f46cc15d990 2023-10-16T11:17:24.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d0d42700 1 -- 172.21.15.158:0/3278234334 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8011d20 con 0x7f46cc15d990 2023-10-16T11:17:24.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d3fa8700 1 -- 172.21.15.158:0/3278234334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 msgr2=0x7f46cc06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.675+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/3278234334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc06d5d0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0b73d0 tx=0x7f46c8009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:24.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.676+0000 7f46d3fa8700 1 -- 172.21.15.158:0/3278234334 shutdown_connections 2023-10-16T11:17:24.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.676+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/3278234334 >> v2:172.21.15.172:3300/0 conn(0x7f46cc0cbaf0 0x7f46cc0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.676+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/3278234334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc06d5d0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.676+0000 7f46d3fa8700 1 -- 172.21.15.158:0/3278234334 >> 172.21.15.158:0/3278234334 conn(0x7f46cc0cdf20 msgr2=0x7f46cc0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:24.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.676+0000 7f46d3fa8700 1 -- 172.21.15.158:0/3278234334 shutdown_connections 2023-10-16T11:17:24.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.676+0000 7f46d3fa8700 1 -- 172.21.15.158:0/3278234334 wait complete. 2023-10-16T11:17:24.683 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:24.681+0000 7f8e793c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:17:24.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.677+0000 7f46d3fa8700 1 Processor -- start 2023-10-16T11:17:24.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.677+0000 7f46d3fa8700 1 -- start start 2023-10-16T11:17:24.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d3fa8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46cc070300 0x7f46cc0cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d3fa8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46cc0cbaf0 0x7f46cc1e8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d3fa8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc1e91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d3fa8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c96b0 con 0x7f46cc15d990 2023-10-16T11:17:24.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d3fa8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c9950 con 0x7f46cc070300 2023-10-16T11:17:24.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d3fa8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c9c30 con 0x7f46cc0cbaf0 2023-10-16T11:17:24.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc1e91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc1e91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35872/0 (socket says 172.21.15.158:35872) 2023-10-16T11:17:24.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 -- 172.21.15.158:0/1253772897 learned_addr learned my addr 172.21.15.158:0/1253772897 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:24.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d1d44700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46cc070300 0x7f46cc0cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d1543700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46cc0cbaf0 0x7f46cc1e8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:24.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 -- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46cc0cbaf0 msgr2=0x7f46cc1e8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46cc0cbaf0 0x7f46cc1e8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 -- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46cc070300 msgr2=0x7f46cc0cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46cc070300 0x7f46cc0cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 -- 172.21.15.158:0/1253772897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c8009550 con 0x7f46cc15d990 2023-10-16T11:17:24.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d1d44700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46cc070300 0x7f46cc0cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:24.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d2545700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc1e91a0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f46c400efc0 tx=0x7f46c400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:24.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46c27fc700 1 -- 172.21.15.158:0/1253772897 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400cad0 con 0x7f46cc15d990 2023-10-16T11:17:24.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46c27fc700 1 -- 172.21.15.158:0/1253772897 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c400f040 con 0x7f46cc15d990 2023-10-16T11:17:24.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc0c9f10 con 0x7f46cc15d990 2023-10-16T11:17:24.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.678+0000 7f46c27fc700 1 -- 172.21.15.158:0/1253772897 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400ce70 con 0x7f46cc15d990 2023-10-16T11:17:24.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.679+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc0ca4d0 con 0x7f46cc15d990 2023-10-16T11:17:24.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.680+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc0681b0 con 0x7f46cc15d990 2023-10-16T11:17:24.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.680+0000 7f46c27fc700 1 -- 172.21.15.158:0/1253772897 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f46c4014480 con 0x7f46cc15d990 2023-10-16T11:17:24.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.680+0000 7f46c27fc700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:24.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.681+0000 7f46d1d44700 1 -- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 msgr2=0x7f46b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:24.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.681+0000 7f46c27fc700 1 -- 172.21.15.158:0/1253772897 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f46c408a5a0 con 0x7f46cc15d990 2023-10-16T11:17:24.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.681+0000 7f46d1d44700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:24.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.685+0000 7f46c27fc700 1 -- 172.21.15.158:0/1253772897 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46c4054e40 con 0x7f46cc15d990 2023-10-16T11:17:24.827 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:24.826+0000 7fd9a47fa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:17:24.838 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:24.838+0000 7f8e793c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:17:24.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.882+0000 7f46d1d44700 1 -- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 msgr2=0x7f46b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:24.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.882+0000 7f46d1d44700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:24.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.883+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46cc0b0c90 con 0x7f46cc15d990 2023-10-16T11:17:24.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.883+0000 7f46c27fc700 1 -- 172.21.15.158:0/1253772897 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v188) v1 ==== 100+0+11 (secure 0 0 0) 0x7f46c4058460 con 0x7f46cc15d990 2023-10-16T11:17:24.885 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:17:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 msgr2=0x7f46b006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 msgr2=0x7f46cc1e91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc1e91a0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f46c400efc0 tx=0x7f46c400c0c0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 shutdown_connections 2023-10-16T11:17:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc15d990 0x7f46cc1e91a0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46cc070300 0x7f46cc0cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 --2- 172.21.15.158:0/1253772897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46cc0cbaf0 0x7f46cc1e8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 >> 172.21.15.158:0/1253772897 conn(0x7f46cc0cdf20 msgr2=0x7f46cc0b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:24.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 shutdown_connections 2023-10-16T11:17:24.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:24.886+0000 7f46d3fa8700 1 -- 172.21.15.158:0/1253772897 wait complete. 2023-10-16T11:17:24.900 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:17:24.950 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:24.949+0000 7fd9a47fa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:17:25.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.023+0000 7f4042f09700 1 Processor -- start 2023-10-16T11:17:25.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.023+0000 7f4042f09700 1 -- start start 2023-10-16T11:17:25.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f4042f09700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f4042f09700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c15f0f0 0x7f403c0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f4042f09700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c15b930 con 0x7f403c0ce530 2023-10-16T11:17:25.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f4042f09700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403c0cf020 con 0x7f403c0cc1b0 2023-10-16T11:17:25.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f4042f09700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c15f5f0 con 0x7f403c15f0f0 2023-10-16T11:17:25.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f4040ca5700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f403c0ce530 0x7f403c0cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36054/0 (socket says 172.21.15.158:36054) 2023-10-16T11:17:25.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35880/0 (socket says 172.21.15.158:35880) 2023-10-16T11:17:25.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f4040ca5700 1 -- 172.21.15.158:0/1640009602 learned_addr learned my addr 172.21.15.158:0/1640009602 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:25.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f40414a6700 1 --2- 172.21.15.158:0/1640009602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c15f0f0 0x7f403c0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403bfff700 1 -- 172.21.15.158:0/1640009602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c15f0f0 msgr2=0x7f403c0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:25.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403bfff700 1 --2- 172.21.15.158:0/1640009602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c15f0f0 0x7f403c0cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403bfff700 1 -- 172.21.15.158:0/1640009602 >> v1:172.21.15.158:6789/0 conn(0x7f403c0ce530 legacy=0x7f403c0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403bfff700 1 -- 172.21.15.158:0/1640009602 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c15f8d0 con 0x7f403c0cc1b0 2023-10-16T11:17:25.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403bfff700 1 --2- 172.21.15.158:0/1640009602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f4028009840 tx=0x7f4028009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad340ae5b2a155b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:25.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403b7fe700 1 -- 172.21.15.158:0/1640009602 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028011830 con 0x7f403c0cc1b0 2023-10-16T11:17:25.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403b7fe700 1 -- 172.21.15.158:0/1640009602 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028011990 con 0x7f403c0cc1b0 2023-10-16T11:17:25.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.025+0000 7f403b7fe700 1 -- 172.21.15.158:0/1640009602 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028011d20 con 0x7f403c0cc1b0 2023-10-16T11:17:25.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 -- 172.21.15.158:0/1640009602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 msgr2=0x7f403c0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 --2- 172.21.15.158:0/1640009602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f4028009840 tx=0x7f4028009b50 comp rx=0 tx=0).stop 2023-10-16T11:17:25.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 -- 172.21.15.158:0/1640009602 shutdown_connections 2023-10-16T11:17:25.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 --2- 172.21.15.158:0/1640009602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c15f0f0 0x7f403c0cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 --2- 172.21.15.158:0/1640009602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 -- 172.21.15.158:0/1640009602 >> 172.21.15.158:0/1640009602 conn(0x7f403c06a8d0 msgr2=0x7f403c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:25.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 -- 172.21.15.158:0/1640009602 shutdown_connections 2023-10-16T11:17:25.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.026+0000 7f4042f09700 1 -- 172.21.15.158:0/1640009602 wait complete. 2023-10-16T11:17:25.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.027+0000 7f4042f09700 1 Processor -- start 2023-10-16T11:17:25.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4042f09700 1 -- start start 2023-10-16T11:17:25.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4042f09700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4042f09700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c0ce530 0x7f403c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4042f09700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f403c15f0f0 0x7f403c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4042f09700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0cb7a0 con 0x7f403c0cc1b0 2023-10-16T11:17:25.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4042f09700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0c8520 con 0x7f403c15f0f0 2023-10-16T11:17:25.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4042f09700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c15b060 con 0x7f403c0ce530 2023-10-16T11:17:25.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.028+0000 7f4040ca5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35884/0 (socket says 172.21.15.158:35884) 2023-10-16T11:17:25.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 -- 172.21.15.158:0/2456042042 learned_addr learned my addr 172.21.15.158:0/2456042042 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:25.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f403bfff700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c0ce530 0x7f403c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f40414a6700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f403c15f0f0 0x7f403c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 -- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c0ce530 msgr2=0x7f403c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c0ce530 0x7f403c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 -- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f403c15f0f0 msgr2=0x7f403c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f403c15f0f0 0x7f403c15ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 -- 172.21.15.158:0/2456042042 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40280094a0 con 0x7f403c0cc1b0 2023-10-16T11:17:25.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f403bfff700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c0ce530 0x7f403c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:25.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4040ca5700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f403000c490 tx=0x7f403000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:25.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f40397fa700 1 -- 172.21.15.158:0/2456042042 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403000cea0 con 0x7f403c0cc1b0 2023-10-16T11:17:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f40397fa700 1 -- 172.21.15.158:0/2456042042 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030005d70 con 0x7f403c0cc1b0 2023-10-16T11:17:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c0b4100 con 0x7f403c0cc1b0 2023-10-16T11:17:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f40397fa700 1 -- 172.21.15.158:0/2456042042 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40300072f0 con 0x7f403c0cc1b0 2023-10-16T11:17:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.029+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c0b46c0 con 0x7f403c0cc1b0 2023-10-16T11:17:25.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.030+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c0681b0 con 0x7f403c0cc1b0 2023-10-16T11:17:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.031+0000 7f40397fa700 1 -- 172.21.15.158:0/2456042042 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f4030007450 con 0x7f403c0cc1b0 2023-10-16T11:17:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.031+0000 7f40397fa700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 0x7f402006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.031+0000 7f403bfff700 1 -- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 msgr2=0x7f402006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.031+0000 7f40397fa700 1 -- 172.21.15.158:0/2456042042 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f4030089040 con 0x7f403c0cc1b0 2023-10-16T11:17:25.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.031+0000 7f403bfff700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 0x7f402006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:25.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.036+0000 7f40397fa700 1 -- 172.21.15.158:0/2456042042 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4030053a80 con 0x7f403c0cc1b0 2023-10-16T11:17:25.103 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:25.102+0000 7f8e793c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:25.125 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:25.124+0000 7fd9a47fa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:17:25.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.232+0000 7f403bfff700 1 -- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 msgr2=0x7f402006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:25.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.232+0000 7f403bfff700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 0x7f402006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:25.264 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:25.264+0000 7f8e793c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:17:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.289+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f403c0b3900 con 0x7f403c0cc1b0 2023-10-16T11:17:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.289+0000 7f40397fa700 1 -- 172.21.15.158:0/2456042042 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4930 (secure 0 0 0) 0x7f4030017090 con 0x7f403c0cc1b0 2023-10-16T11:17:25.291 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:25.292 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3580267390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3580267390},{"type":"v1","addr":"172.21.15.172:6834","nonce":3580267390}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":9328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3433152480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3433152480},{"type":"v1","addr":"172.21.15.172:6835","nonce":3433152480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1170239465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1170239465},{"type":"v1","addr":"172.21.15.158:6836","nonce":1170239465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:01:17.640665+0000","modified":"2023-10-16T11:01:20.662098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10449":{"gid":10449,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/773046671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":773046671},{"type":"v1","addr":"172.21.15.158:6837","nonce":773046671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T11:17:25.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 msgr2=0x7f402006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 0x7f402006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 msgr2=0x7f403c0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f403000c490 tx=0x7f403000c7a0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 shutdown_connections 2023-10-16T11:17:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f403c15f0f0 0x7f403c15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f403c0cc1b0 0x7f403c0b3b20 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f402006b020 0x7f402006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 --2- 172.21.15.158:0/2456042042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f403c0ce530 0x7f403c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 >> 172.21.15.158:0/2456042042 conn(0x7f403c06a8d0 msgr2=0x7f403c0b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 shutdown_connections 2023-10-16T11:17:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.292+0000 7f4042f09700 1 -- 172.21.15.158:0/2456042042 wait complete. 2023-10-16T11:17:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 94 2023-10-16T11:17:25.306 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:25.406 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:25.405+0000 7fd9a47fa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:25.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.428+0000 7f3a8a151700 1 Processor -- start 2023-10-16T11:17:25.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.429+0000 7f3a8a151700 1 -- start start 2023-10-16T11:17:25.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.429+0000 7f3a8a151700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.429+0000 7f3a8a151700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a8415f0f0 0x7f3a840cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.429+0000 7f3a8a151700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8415b930 con 0x7f3a840ce530 2023-10-16T11:17:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.429+0000 7f3a8a151700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a840cf020 con 0x7f3a8415f0f0 2023-10-16T11:17:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.429+0000 7f3a8a151700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8415f5f0 con 0x7f3a840cc1b0 2023-10-16T11:17:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.429+0000 7f3a83fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a8415f0f0 0x7f3a840cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a83fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a8415f0f0 0x7f3a840cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35898/0 (socket says 172.21.15.158:35898) 2023-10-16T11:17:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a83fff700 1 -- 172.21.15.158:0/2243126751 learned_addr learned my addr 172.21.15.158:0/2243126751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a82ffd700 1 --2- 172.21.15.158:0/2243126751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a83fff700 1 -- 172.21.15.158:0/2243126751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 msgr2=0x7f3a840b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a83fff700 1 --2- 172.21.15.158:0/2243126751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a83fff700 1 -- 172.21.15.158:0/2243126751 >> v1:172.21.15.172:6789/0 conn(0x7f3a840ce530 legacy=0x7f3a840cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a83fff700 1 -- 172.21.15.158:0/2243126751 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8415f8d0 con 0x7f3a8415f0f0 2023-10-16T11:17:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a83fff700 1 --2- 172.21.15.158:0/2243126751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf470 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0098f0 tx=0x7f3a7c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23db39898bfb79a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:25.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a827fc700 1 -- 172.21.15.158:0/2243126751 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c011830 con 0x7f3a8415f0f0 2023-10-16T11:17:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a827fc700 1 -- 172.21.15.158:0/2243126751 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7c011990 con 0x7f3a8415f0f0 2023-10-16T11:17:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.430+0000 7f3a827fc700 1 -- 172.21.15.158:0/2243126751 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c011d20 con 0x7f3a8415f0f0 2023-10-16T11:17:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 -- 172.21.15.158:0/2243126751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 msgr2=0x7f3a840cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2243126751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf470 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0098f0 tx=0x7f3a7c009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:25.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 -- 172.21.15.158:0/2243126751 shutdown_connections 2023-10-16T11:17:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2243126751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf470 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2243126751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 -- 172.21.15.158:0/2243126751 >> 172.21.15.158:0/2243126751 conn(0x7f3a8406a8d0 msgr2=0x7f3a8406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 -- 172.21.15.158:0/2243126751 shutdown_connections 2023-10-16T11:17:25.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.431+0000 7f3a8a151700 1 -- 172.21.15.158:0/2243126751 wait complete. 2023-10-16T11:17:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.432+0000 7f3a8a151700 1 Processor -- start 2023-10-16T11:17:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.432+0000 7f3a8a151700 1 -- start start 2023-10-16T11:17:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a8a151700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a8a151700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a840ce530 0x7f3a84154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a8a151700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a8a151700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840c54c0 con 0x7f3a8415f0f0 2023-10-16T11:17:25.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a8a151700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840b7110 con 0x7f3a840ce530 2023-10-16T11:17:25.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a8a151700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840cb350 con 0x7f3a840cc1b0 2023-10-16T11:17:25.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a837fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35904/0 (socket says 172.21.15.158:35904) 2023-10-16T11:17:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a82ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a840ce530 0x7f3a84154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 -- 172.21.15.158:0/2366515232 learned_addr learned my addr 172.21.15.158:0/2366515232 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 -- 172.21.15.158:0/2366515232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 msgr2=0x7f3a840c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 -- 172.21.15.158:0/2366515232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a840ce530 msgr2=0x7f3a84154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a840ce530 0x7f3a84154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 -- 172.21.15.158:0/2366515232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c009550 con 0x7f3a8415f0f0 2023-10-16T11:17:25.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a837fe700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:25.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.433+0000 7f3a83fff700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf370 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c013000 tx=0x7f3a7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:25.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.434+0000 7f3a7bfff700 1 -- 172.21.15.158:0/2366515232 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c004380 con 0x7f3a8415f0f0 2023-10-16T11:17:25.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.434+0000 7f3a7bfff700 1 -- 172.21.15.158:0/2366515232 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7c01f070 con 0x7f3a8415f0f0 2023-10-16T11:17:25.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.434+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a840c5e20 con 0x7f3a8415f0f0 2023-10-16T11:17:25.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.434+0000 7f3a7bfff700 1 -- 172.21.15.158:0/2366515232 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c0036a0 con 0x7f3a8415f0f0 2023-10-16T11:17:25.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.434+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a840c4e00 con 0x7f3a8415f0f0 2023-10-16T11:17:25.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.435+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a840681b0 con 0x7f3a8415f0f0 2023-10-16T11:17:25.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.435+0000 7f3a7bfff700 1 -- 172.21.15.158:0/2366515232 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f3a7c021030 con 0x7f3a8415f0f0 2023-10-16T11:17:25.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.435+0000 7f3a7bfff700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 0x7f3a6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.436+0000 7f3a837fe700 1 -- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 msgr2=0x7f3a6406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:25.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.436+0000 7f3a7bfff700 1 -- 172.21.15.158:0/2366515232 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f3a7c08afd0 con 0x7f3a8415f0f0 2023-10-16T11:17:25.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.436+0000 7f3a837fe700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 0x7f3a6406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:25.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.440+0000 7f3a7bfff700 1 -- 172.21.15.158:0/2366515232 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a7c055990 con 0x7f3a8415f0f0 2023-10-16T11:17:25.622 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:25.621+0000 7fd9a47fa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:17:25.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.637+0000 7f3a837fe700 1 -- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 msgr2=0x7f3a6406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:25.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.637+0000 7f3a837fe700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 0x7f3a6406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:25.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.699+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a8415fb80 con 0x7f3a8415f0f0 2023-10-16T11:17:25.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.700+0000 7f3a7bfff700 1 -- 172.21.15.158:0/2366515232 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+18547 (secure 0 0 0) 0x7f3a7c076890 con 0x7f3a8415f0f0 2023-10-16T11:17:25.702 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:25.703 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":188,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:23.595580+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:01:16.077848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T11:01:16.598638+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":188,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2531840430":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2719309194":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2065912370":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/1399927564":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6833/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2545298209":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/549775613":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/827805412":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/2974007506":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/4132457601":"2023-10-17T11:01:31.624108+0000","172.21.15.158:6833/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3186172973":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3804388680":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/411715851":"2023-10-17T11:01:31.624108+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:25.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 msgr2=0x7f3a6406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:25.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 0x7f3a6406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 msgr2=0x7f3a840cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf370 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c013000 tx=0x7f3a7c004750 comp rx=0 tx=0).stop 2023-10-16T11:17:25.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 shutdown_connections 2023-10-16T11:17:25.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a8415f0f0 0x7f3a840cf370 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7f3a6406b020 0x7f3a6406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a840cc1b0 0x7f3a840c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 --2- 172.21.15.158:0/2366515232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a840ce530 0x7f3a84154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 >> 172.21.15.158:0/2366515232 conn(0x7f3a8406a8d0 msgr2=0x7f3a840cba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:25.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 shutdown_connections 2023-10-16T11:17:25.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.703+0000 7f3a8a151700 1 -- 172.21.15.158:0/2366515232 wait complete. 2023-10-16T11:17:25.717 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:17:25.718 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:17:25.790 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:25.789+0000 7f8e793c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:17:25.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.839+0000 7fbffd52d700 1 Processor -- start 2023-10-16T11:17:25.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.839+0000 7fbffd52d700 1 -- start start 2023-10-16T11:17:25.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbffd52d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbff815b330 0x7fbff80711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbffd52d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbff815b710 0x7fbff80716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbffd52d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff815ff10 con 0x7fbff815c6e0 2023-10-16T11:17:25.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbffd52d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff8072440 con 0x7fbff815b710 2023-10-16T11:17:25.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbffd52d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff815c250 con 0x7fbff815b330 2023-10-16T11:17:25.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbff67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbff815b710 0x7fbff80716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbff67fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbff815b710 0x7fbff80716e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35912/0 (socket says 172.21.15.158:35912) 2023-10-16T11:17:25.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbff77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbff815c6e0 0x7fbff815f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36090/0 (socket says 172.21.15.158:36090) 2023-10-16T11:17:25.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbff67fc700 1 -- 172.21.15.158:0/1574702701 learned_addr learned my addr 172.21.15.158:0/1574702701 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:25.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbff6ffd700 1 --2- 172.21.15.158:0/1574702701 >> v2:172.21.15.172:3300/0 conn(0x7fbff815b330 0x7fbff80711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbff67fc700 1 -- 172.21.15.158:0/1574702701 >> v2:172.21.15.172:3300/0 conn(0x7fbff815b330 msgr2=0x7fbff80711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:25.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.840+0000 7fbff67fc700 1 --2- 172.21.15.158:0/1574702701 >> v2:172.21.15.172:3300/0 conn(0x7fbff815b330 0x7fbff80711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff67fc700 1 -- 172.21.15.158:0/1574702701 >> v1:172.21.15.158:6789/0 conn(0x7fbff815c6e0 legacy=0x7fbff815f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff67fc700 1 -- 172.21.15.158:0/1574702701 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff815fc30 con 0x7fbff815b710 2023-10-16T11:17:25.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff5ffb700 1 -- 172.21.15.158:0/1574702701 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138564345 0 0) 0x7fbff815ff10 con 0x7fbff815c6e0 2023-10-16T11:17:25.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff6ffd700 1 --2- 172.21.15.158:0/1574702701 >> v2:172.21.15.172:3300/0 conn(0x7fbff815b330 0x7fbff80711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:25.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff67fc700 1 --2- 172.21.15.158:0/1574702701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b710 0x7fbff80716e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fbff80d08c0 tx=0x7fbfec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638194ac928aedad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:25.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff5ffb700 1 -- 172.21.15.158:0/1574702701 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec00b870 con 0x7fbff815b710 2023-10-16T11:17:25.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff5ffb700 1 -- 172.21.15.158:0/1574702701 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec00b9d0 con 0x7fbff815b710 2023-10-16T11:17:25.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbff5ffb700 1 -- 172.21.15.158:0/1574702701 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec00bd60 con 0x7fbff815b710 2023-10-16T11:17:25.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbffd52d700 1 -- 172.21.15.158:0/1574702701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b710 msgr2=0x7fbff80716e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.841+0000 7fbffd52d700 1 --2- 172.21.15.158:0/1574702701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b710 0x7fbff80716e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fbff80d08c0 tx=0x7fbfec009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.842+0000 7fbffd52d700 1 -- 172.21.15.158:0/1574702701 shutdown_connections 2023-10-16T11:17:25.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.842+0000 7fbffd52d700 1 --2- 172.21.15.158:0/1574702701 >> v2:172.21.15.172:3300/0 conn(0x7fbff815b330 0x7fbff80711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.842+0000 7fbffd52d700 1 --2- 172.21.15.158:0/1574702701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b710 0x7fbff80716e0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.842+0000 7fbffd52d700 1 -- 172.21.15.158:0/1574702701 >> 172.21.15.158:0/1574702701 conn(0x7fbff80cfaa0 msgr2=0x7fbff80cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:25.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.842+0000 7fbffd52d700 1 -- 172.21.15.158:0/1574702701 shutdown_connections 2023-10-16T11:17:25.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.842+0000 7fbffd52d700 1 -- 172.21.15.158:0/1574702701 wait complete. 2023-10-16T11:17:25.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.843+0000 7fbffd52d700 1 Processor -- start 2023-10-16T11:17:25.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.843+0000 7fbffd52d700 1 -- start start 2023-10-16T11:17:25.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbffd52d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b330 0x7fbff80b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbffd52d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff815b710 0x7fbff8159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbffd52d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbff815c6e0 0x7fbff81440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbffd52d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff815a650 con 0x7fbff815b330 2023-10-16T11:17:25.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbffd52d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff8159ba0 con 0x7fbff815b710 2023-10-16T11:17:25.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbffd52d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff8144600 con 0x7fbff815c6e0 2023-10-16T11:17:25.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b330 0x7fbff80b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b330 0x7fbff80b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35916/0 (socket says 172.21.15.158:35916) 2023-10-16T11:17:25.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 -- 172.21.15.158:0/2066622276 learned_addr learned my addr 172.21.15.158:0/2066622276 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:25.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff77fe700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbff815c6e0 0x7fbff81440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff67fc700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff815b710 0x7fbff8159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:25.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 -- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbff815c6e0 msgr2=0x7fbff81440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbff815c6e0 0x7fbff81440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 -- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff815b710 msgr2=0x7fbff8159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:25.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff815b710 0x7fbff8159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:25.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 -- 172.21.15.158:0/2066622276 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec009550 con 0x7fbff815b330 2023-10-16T11:17:25.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.844+0000 7fbff6ffd700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b330 0x7fbff80b2750 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800e810 tx=0x7fbfe800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:25.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.845+0000 7fbfdf7fe700 1 -- 172.21.15.158:0/2066622276 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8013610 con 0x7fbff815b330 2023-10-16T11:17:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.845+0000 7fbfdf7fe700 1 -- 172.21.15.158:0/2066622276 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe8013770 con 0x7fbff815b330 2023-10-16T11:17:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.845+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff8153c30 con 0x7fbff815b330 2023-10-16T11:17:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.845+0000 7fbfdf7fe700 1 -- 172.21.15.158:0/2066622276 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8013b20 con 0x7fbff815b330 2023-10-16T11:17:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.845+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff815a090 con 0x7fbff815b330 2023-10-16T11:17:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.846+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff8153a00 con 0x7fbff815b330 2023-10-16T11:17:25.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.846+0000 7fbfdf7fe700 1 -- 172.21.15.158:0/2066622276 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fbfe8013c80 con 0x7fbff815b330 2023-10-16T11:17:25.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.846+0000 7fbfdf7fe700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 0x7fbfe406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:25.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.847+0000 7fbff67fc700 1 -- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 msgr2=0x7fbfe406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:25.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.847+0000 7fbfdf7fe700 1 -- 172.21.15.158:0/2066622276 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7225+0+0 (secure 0 0 0) 0x7fbfe80899a0 con 0x7fbff815b330 2023-10-16T11:17:25.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.847+0000 7fbff67fc700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 0x7fbfe406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:25.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:25.851+0000 7fbfdf7fe700 1 -- 172.21.15.158:0/2066622276 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbfe8054240 con 0x7fbff815b330 2023-10-16T11:17:26.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.048+0000 7fbff67fc700 1 -- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 msgr2=0x7fbfe406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3817845958 2023-10-16T11:17:26.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.048+0000 7fbff67fc700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 0x7fbfe406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:26.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.093+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbff80681b0 con 0x7fbff815b330 2023-10-16T11:17:26.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.094+0000 7fbfdf7fe700 1 -- 172.21.15.158:0/2066622276 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4930 (secure 0 0 0) 0x7fbfe8057860 con 0x7fbff815b330 2023-10-16T11:17:26.096 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:26.096 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3580267390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3580267390},{"type":"v1","addr":"172.21.15.172:6834","nonce":3580267390}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":9328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3433152480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3433152480},{"type":"v1","addr":"172.21.15.172:6835","nonce":3433152480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1170239465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1170239465},{"type":"v1","addr":"172.21.15.158:6836","nonce":1170239465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:01:17.640665+0000","modified":"2023-10-16T11:01:20.662098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10449":{"gid":10449,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/773046671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":773046671},{"type":"v1","addr":"172.21.15.158:6837","nonce":773046671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T11:17:26.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 msgr2=0x7fbfe406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:26.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 0x7fbfe406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b330 msgr2=0x7fbff80b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b330 0x7fbff80b2750 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800e810 tx=0x7fbfe800eb20 comp rx=0 tx=0).stop 2023-10-16T11:17:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 shutdown_connections 2023-10-16T11:17:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:6832/3817845958,v1:172.21.15.158:6833/3817845958] conn(0x7fbfe406b020 0x7fbfe406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbff815c6e0 0x7fbff81440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbff815b330 0x7fbff80b2750 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 --2- 172.21.15.158:0/2066622276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff815b710 0x7fbff8159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 >> 172.21.15.158:0/2066622276 conn(0x7fbff80cfaa0 msgr2=0x7fbff80cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 shutdown_connections 2023-10-16T11:17:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.096+0000 7fbffd52d700 1 -- 172.21.15.158:0/2066622276 wait complete. 2023-10-16T11:17:26.101 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 94 2023-10-16T11:17:26.110 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:17:26.180 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.179+0000 7fd9a47fa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:17:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.238+0000 7f4a0159a700 1 Processor -- start 2023-10-16T11:17:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.238+0000 7f4a0159a700 1 -- start start 2023-10-16T11:17:26.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f4a0159a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f49fc15a6c0 0x7f49fc0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f4a0159a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49fc06e090 0x7f49fc0ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f4a0159a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49fc072910 con 0x7f49fc0af590 2023-10-16T11:17:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f4a0159a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49fc0ca7e0 con 0x7f49fc06e090 2023-10-16T11:17:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f4a0159a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49fc0b1e40 con 0x7f49fc15a6c0 2023-10-16T11:17:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f49fb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49fc06e090 0x7f49fc0ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f49fb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f49fc06e090 0x7f49fc0ad430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35956/0 (socket says 172.21.15.158:35956) 2023-10-16T11:17:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f49fb7fe700 1 -- 172.21.15.158:0/1545243323 learned_addr learned my addr 172.21.15.158:0/1545243323 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.239+0000 7f49faffd700 1 --2- 172.21.15.158:0/1545243323 >> v2:172.21.15.172:3300/0 conn(0x7f49fc15a6c0 0x7f49fc0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49fb7fe700 1 -- 172.21.15.158:0/1545243323 >> v2:172.21.15.172:3300/0 conn(0x7f49fc15a6c0 msgr2=0x7f49fc0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49fb7fe700 1 --2- 172.21.15.158:0/1545243323 >> v2:172.21.15.172:3300/0 conn(0x7f49fc15a6c0 0x7f49fc0723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49fb7fe700 1 -- 172.21.15.158:0/1545243323 >> v1:172.21.15.172:6789/0 conn(0x7f49fc0af590 legacy=0x7f49fc06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49fb7fe700 1 -- 172.21.15.158:0/1545243323 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc0b2120 con 0x7f49fc06e090 2023-10-16T11:17:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49fb7fe700 1 --2- 172.21.15.158:0/1545243323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc06e090 0x7f49fc0ad430 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0737c0 tx=0x7f49f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ccb9a2ee2f80fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49f9ffb700 1 -- 172.21.15.158:0/1545243323 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f400e070 con 0x7f49fc06e090 2023-10-16T11:17:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49f9ffb700 1 -- 172.21.15.158:0/1545243323 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f400b910 con 0x7f49fc06e090 2023-10-16T11:17:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.240+0000 7f49f9ffb700 1 -- 172.21.15.158:0/1545243323 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f400bcc0 con 0x7f49fc06e090 2023-10-16T11:17:26.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 -- 172.21.15.158:0/1545243323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc06e090 msgr2=0x7f49fc0ad430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 --2- 172.21.15.158:0/1545243323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc06e090 0x7f49fc0ad430 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0737c0 tx=0x7f49f400b330 comp rx=0 tx=0).stop 2023-10-16T11:17:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 -- 172.21.15.158:0/1545243323 shutdown_connections 2023-10-16T11:17:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 --2- 172.21.15.158:0/1545243323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc06e090 0x7f49fc0ad430 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 --2- 172.21.15.158:0/1545243323 >> v2:172.21.15.172:3300/0 conn(0x7f49fc15a6c0 0x7f49fc0723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 -- 172.21.15.158:0/1545243323 >> 172.21.15.158:0/1545243323 conn(0x7f49fc0ccda0 msgr2=0x7f49fc0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 -- 172.21.15.158:0/1545243323 shutdown_connections 2023-10-16T11:17:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.241+0000 7f4a0159a700 1 -- 172.21.15.158:0/1545243323 wait complete. 2023-10-16T11:17:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 Processor -- start 2023-10-16T11:17:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 -- start start 2023-10-16T11:17:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49fc06e090 0x7f49fc1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49fc0af590 0x7f49fc0c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc15a6c0 0x7f49fc0d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc0c4e90 con 0x7f49fc15a6c0 2023-10-16T11:17:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc156910 con 0x7f49fc0af590 2023-10-16T11:17:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f4a0159a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc0c7960 con 0x7f49fc06e090 2023-10-16T11:17:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.243+0000 7f49fb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc15a6c0 0x7f49fc0d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc15a6c0 0x7f49fc0d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:35968/0 (socket says 172.21.15.158:35968) 2023-10-16T11:17:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 -- 172.21.15.158:0/3079337264 learned_addr learned my addr 172.21.15.158:0/3079337264 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49faffd700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49fc06e090 0x7f49fc1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fa7fc700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49fc0af590 0x7f49fc0c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 -- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49fc06e090 msgr2=0x7f49fc1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49fc06e090 0x7f49fc1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 -- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49fc0af590 msgr2=0x7f49fc0c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49fc0af590 0x7f49fc0c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 -- 172.21.15.158:0/3079337264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f4009b30 con 0x7f49fc15a6c0 2023-10-16T11:17:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49fb7fe700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc15a6c0 0x7f49fc0d01b0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f49f4012000 tx=0x7f49f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49e37fe700 1 -- 172.21.15.158:0/3079337264 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f4004840 con 0x7f49fc15a6c0 2023-10-16T11:17:26.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49e37fe700 1 -- 172.21.15.158:0/3079337264 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f40099b0 con 0x7f49fc15a6c0 2023-10-16T11:17:26.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f49e37fe700 1 -- 172.21.15.158:0/3079337264 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f401c680 con 0x7f49fc15a6c0 2023-10-16T11:17:26.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49fc0c7c40 con 0x7f49fc15a6c0 2023-10-16T11:17:26.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.244+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49fc0c8200 con 0x7f49fc15a6c0 2023-10-16T11:17:26.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.245+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49fc0681b0 con 0x7f49fc15a6c0 2023-10-16T11:17:26.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.246+0000 7f49e37fe700 1 -- 172.21.15.158:0/3079337264 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 123) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f49f401c7e0 con 0x7f49fc15a6c0 2023-10-16T11:17:26.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.246+0000 7f49e37fe700 1 -- 172.21.15.158:0/3079337264 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f49f408afd0 con 0x7f49fc15a6c0 2023-10-16T11:17:26.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.251+0000 7f49e37fe700 1 -- 172.21.15.158:0/3079337264 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49f4055da0 con 0x7f49fc15a6c0 2023-10-16T11:17:26.263 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.261+0000 7fd91c7df700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.263 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.261+0000 7fd91c7df700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.263 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91c7df700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.263 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91c7df700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.264 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91c7df700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.264 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91ffe6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.264 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91ffe6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.264 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91ffe6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.265 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91ffe6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.265 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:26.262+0000 7fd91ffe6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:26.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.496+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49fc09e030 con 0x7f49fc15a6c0 2023-10-16T11:17:26.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.496+0000 7f49e37fe700 1 -- 172.21.15.158:0/3079337264 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4930 (secure 0 0 0) 0x7f49f40593c0 con 0x7f49fc15a6c0 2023-10-16T11:17:26.498 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:26.498 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3580267390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3580267390},{"type":"v1","addr":"172.21.15.172:6834","nonce":3580267390}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":9328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3433152480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3433152480},{"type":"v1","addr":"172.21.15.172:6835","nonce":3433152480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1170239465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1170239465},{"type":"v1","addr":"172.21.15.158:6836","nonce":1170239465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:01:17.640665+0000","modified":"2023-10-16T11:01:20.662098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10449":{"gid":10449,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6837/773046671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":773046671},{"type":"v1","addr":"172.21.15.158:6837","nonce":773046671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T11:17:26.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc15a6c0 msgr2=0x7f49fc0d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc15a6c0 0x7f49fc0d01b0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f49f4012000 tx=0x7f49f4004ad0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 shutdown_connections 2023-10-16T11:17:26.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49fc15a6c0 0x7f49fc0d01b0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49fc06e090 0x7f49fc1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 --2- 172.21.15.158:0/3079337264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49fc0af590 0x7f49fc0c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 >> 172.21.15.158:0/3079337264 conn(0x7f49fc0ccda0 msgr2=0x7f49fc0cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:26.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 shutdown_connections 2023-10-16T11:17:26.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.498+0000 7f4a0159a700 1 -- 172.21.15.158:0/3079337264 wait complete. 2023-10-16T11:17:26.502 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 94 2023-10-16T11:17:26.512 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:26.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.633+0000 7fc969f36700 1 Processor -- start 2023-10-16T11:17:26.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.633+0000 7fc969f36700 1 -- start start 2023-10-16T11:17:26.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc969f36700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96415dfb0 0x7fc96406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc969f36700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc96406f340 0x7fc96406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc969f36700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96406e770 con 0x7fc9640cc5d0 2023-10-16T11:17:26.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc969f36700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96406fd30 con 0x7fc96406f340 2023-10-16T11:17:26.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc969f36700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9640cf270 con 0x7fc96415dfb0 2023-10-16T11:17:26.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9637fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9640cc5d0 0x7fc9640cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36186/0 (socket says 172.21.15.158:36186) 2023-10-16T11:17:26.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9637fe700 1 -- 172.21.15.158:0/1228604551 learned_addr learned my addr 172.21.15.158:0/1228604551 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:26.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc963fff700 1 --2- 172.21.15.158:0/1228604551 >> v2:172.21.15.172:3300/0 conn(0x7fc96406f340 0x7fc96406f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc962ffd700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96415dfb0 0x7fc96406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738963704 0 0) 0x7fc94c001110 con 0x7fc9640cc5d0 2023-10-16T11:17:26.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9500035b0 con 0x7fc9640cc5d0 2023-10-16T11:17:26.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386591986 0 0) 0x7fc9500035b0 con 0x7fc9640cc5d0 2023-10-16T11:17:26.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96415dfb0 msgr2=0x7fc96406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96415dfb0 0x7fc96406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 >> v2:172.21.15.172:3300/0 conn(0x7fc96406f340 msgr2=0x7fc96406f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 --2- 172.21.15.158:0/1228604551 >> v2:172.21.15.172:3300/0 conn(0x7fc96406f340 0x7fc96406f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9640cbe00 con 0x7fc9640cc5d0 2023-10-16T11:17:26.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc94c001800 con 0x7fc9640cc5d0 2023-10-16T11:17:26.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 >> v1:172.21.15.158:6789/0 conn(0x7fc9640cc5d0 legacy=0x7fc9640cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc950003a00 0x7fc950003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.634+0000 7fc9627fc700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9500043f0 0x7fc9500048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9627fc700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc95000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96406fd30 con 0x7fc950004f40 2023-10-16T11:17:26.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9640cf270 con 0x7fc9500043f0 2023-10-16T11:17:26.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95000bf50 con 0x7fc950003a00 2023-10-16T11:17:26.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc962ffd700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96415dfb0 0x7fc96406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:26.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 --2- 172.21.15.158:0/1228604551 >> v2:172.21.15.172:3300/0 conn(0x7fc96406f340 0x7fc96406f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:26.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9637fe700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc950003a00 0x7fc950003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc95000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc962ffd700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9500043f0 0x7fc9500048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 -- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc950003a00 msgr2=0x7fc950003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc950003a00 0x7fc950003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 -- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9500043f0 msgr2=0x7fc9500048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9500043f0 0x7fc9500048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 -- 172.21.15.158:0/1228604551 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500035b0 con 0x7fc950004f40 2023-10-16T11:17:26.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9637fe700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc950003a00 0x7fc950003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:26.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc963fff700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc95000ba10 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00e350 tx=0x7fc950005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e2c2eb854e4a8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:26.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c00ee10 con 0x7fc950004f40 2023-10-16T11:17:26.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c00f040 con 0x7fc950004f40 2023-10-16T11:17:26.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.635+0000 7fc9627fc700 1 -- 172.21.15.158:0/1228604551 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c003c40 con 0x7fc950004f40 2023-10-16T11:17:26.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.636+0000 7fc969f36700 1 -- 172.21.15.158:0/1228604551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 msgr2=0x7fc95000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.636+0000 7fc969f36700 1 --2- 172.21.15.158:0/1228604551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc95000ba10 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00e350 tx=0x7fc950005880 comp rx=0 tx=0).stop 2023-10-16T11:17:26.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.636+0000 7fc963fff700 1 -- 172.21.15.158:0/1228604551 reap_dead start 2023-10-16T11:17:26.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.636+0000 7fc969f36700 1 -- 172.21.15.158:0/1228604551 shutdown_connections 2023-10-16T11:17:26.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.636+0000 7fc969f36700 1 -- 172.21.15.158:0/1228604551 >> 172.21.15.158:0/1228604551 conn(0x7fc9640d02b0 msgr2=0x7fc9640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:26.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.636+0000 7fc969f36700 1 -- 172.21.15.158:0/1228604551 shutdown_connections 2023-10-16T11:17:26.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.636+0000 7fc969f36700 1 -- 172.21.15.158:0/1228604551 wait complete. 2023-10-16T11:17:26.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.638+0000 7fc969f36700 1 Processor -- start 2023-10-16T11:17:26.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.638+0000 7fc969f36700 1 -- start start 2023-10-16T11:17:26.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc9640b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9640b5010 0x7fc964152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9640b5490 0x7fc964150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9640b59d0 con 0x7fc950004f40 2023-10-16T11:17:26.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9640ca330 con 0x7fc9640b5010 2023-10-16T11:17:26.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96414fc00 con 0x7fc9640b5490 2023-10-16T11:17:26.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc9640b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc9640b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36022/0 (socket says 172.21.15.158:36022) 2023-10-16T11:17:26.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 -- 172.21.15.158:0/1813447691 learned_addr learned my addr 172.21.15.158:0/1813447691 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:26.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc962ffd700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9640b5010 0x7fc964152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc963fff700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9640b5490 0x7fc964150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 -- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9640b5490 msgr2=0x7fc964150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9640b5490 0x7fc964150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 -- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9640b5010 msgr2=0x7fc964152320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9640b5010 0x7fc964152320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 -- 172.21.15.158:0/1813447691 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96414f8d0 con 0x7fc950004f40 2023-10-16T11:17:26.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc9637fe700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc9640b38b0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fc94c007310 tx=0x7fc94c007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:26.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc95bfff700 1 -- 172.21.15.158:0/1813447691 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c00f6c0 con 0x7fc950004f40 2023-10-16T11:17:26.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc95bfff700 1 -- 172.21.15.158:0/1813447691 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c00f820 con 0x7fc950004f40 2023-10-16T11:17:26.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9640c9d70 con 0x7fc950004f40 2023-10-16T11:17:26.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc95bfff700 1 -- 172.21.15.158:0/1813447691 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c00fb90 con 0x7fc950004f40 2023-10-16T11:17:26.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.639+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9640b3df0 con 0x7fc950004f40 2023-10-16T11:17:26.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.640+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96414f730 con 0x7fc950004f40 2023-10-16T11:17:26.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.641+0000 7fc95bfff700 1 -- 172.21.15.158:0/1813447691 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 123) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc94c00fcf0 con 0x7fc950004f40 2023-10-16T11:17:26.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.641+0000 7fc95bfff700 1 -- 172.21.15.158:0/1813447691 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 7541+0+0 (secure 0 0 0) 0x7fc94c089770 con 0x7fc950004f40 2023-10-16T11:17:26.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.646+0000 7fc95bfff700 1 -- 172.21.15.158:0/1813447691 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc96414f730 con 0x7fc950004f40 2023-10-16T11:17:26.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.840+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc96414f730 con 0x7fc950004f40 2023-10-16T11:17:26.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.841+0000 7fc95bfff700 1 -- 172.21.15.158:0/1813447691 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+19010 (secure 0 0 0) 0x7fc96414f730 con 0x7fc950004f40 2023-10-16T11:17:26.843 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:26.844 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":189,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:26.192933+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:01:16.077848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T11:01:16.598638+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":188,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/995001036":"2023-10-17T11:17:26.192865+0000","172.21.15.158:6833/3817845958":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/3732316088":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/1124225642":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/1870965787":"2023-10-17T11:17:26.192865+0000","172.21.15.158:6832/3817845958":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/2531840430":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2719309194":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2065912370":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/1399927564":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6833/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2251797820":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/2545298209":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/549775613":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/827805412":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/2974007506":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/4132457601":"2023-10-17T11:01:31.624108+0000","172.21.15.158:6833/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/276622814":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/3186172973":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3804388680":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/411715851":"2023-10-17T11:01:31.624108+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 msgr2=0x7fc9640b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc9640b38b0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fc94c007310 tx=0x7fc94c007620 comp rx=0 tx=0).stop 2023-10-16T11:17:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 shutdown_connections 2023-10-16T11:17:26.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9640b5490 0x7fc964150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc950004f40 0x7fc9640b38b0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 --2- 172.21.15.158:0/1813447691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9640b5010 0x7fc964152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 >> 172.21.15.158:0/1813447691 conn(0x7fc9640d02b0 msgr2=0x7fc9640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:26.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 shutdown_connections 2023-10-16T11:17:26.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.844+0000 7fc969f36700 1 -- 172.21.15.158:0/1813447691 wait complete. 2023-10-16T11:17:26.858 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:17:26.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.978+0000 7fe408c4b700 1 Processor -- start 2023-10-16T11:17:26.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.978+0000 7fe408c4b700 1 -- start start 2023-10-16T11:17:26.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe408c4b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 0x7fe404071e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe408c4b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4040b3290 con 0x7fe4040b3ce0 2023-10-16T11:17:26.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe408c4b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4040724f0 con 0x7fe40415a570 2023-10-16T11:17:26.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe408c4b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040727d0 con 0x7fe404071a30 2023-10-16T11:17:26.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe402d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe4040b3ce0 0x7fe4040b40f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36208/0 (socket says 172.21.15.158:36208) 2023-10-16T11:17:26.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe402d9d700 1 -- 172.21.15.158:0/2120670838 learned_addr learned my addr 172.21.15.158:0/2120670838 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:26.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe401d9b700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 0x7fe404071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762384831 0 0) 0x7fe4040b3290 con 0x7fe4040b3ce0 2023-10-16T11:17:26.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ec0035b0 con 0x7fe4040b3ce0 2023-10-16T11:17:26.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2037873399 0 0) 0x7fe3ec0035b0 con 0x7fe4040b3ce0 2023-10-16T11:17:26.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 msgr2=0x7fe404071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 0x7fe404071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 >> v1:172.21.15.172:6789/0 conn(0x7fe40415a570 legacy=0x7fe4040ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4040b34e0 con 0x7fe4040b3ce0 2023-10-16T11:17:26.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe3fc0019f0 con 0x7fe4040b3ce0 2023-10-16T11:17:26.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.979+0000 7fe401d9b700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 0x7fe404071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:26.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 >> v1:172.21.15.158:6789/0 conn(0x7fe4040b3ce0 legacy=0x7fe4040b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe3ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3ec0043e0 0x7fe3ec0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 0x7fe3ec0050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040727d0 con 0x7fe3ec003a30 2023-10-16T11:17:26.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec00ba10 con 0x7fe3ec0043e0 2023-10-16T11:17:26.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec00bb80 con 0x7fe404071a30 2023-10-16T11:17:26.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40259c700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe3ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe401d9b700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3ec0043e0 0x7fe3ec0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe402d9d700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 0x7fe3ec0050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40259c700 1 -- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 msgr2=0x7fe3ec0050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40259c700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe404071a30 0x7fe3ec0050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40259c700 1 -- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3ec0043e0 msgr2=0x7fe3ec0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:26.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40259c700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3ec0043e0 0x7fe3ec0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40259c700 1 -- 172.21.15.158:0/2120670838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f80094a0 con 0x7fe3ec003a30 2023-10-16T11:17:26.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40259c700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe3ec003e10 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40018d0 tx=0x7fe3ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227a0f2d6d0d10ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:26.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400c690 con 0x7fe3ec003a30 2023-10-16T11:17:26.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f400c810 con 0x7fe3ec003a30 2023-10-16T11:17:26.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.980+0000 7fe40159a700 1 -- 172.21.15.158:0/2120670838 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400cbc0 con 0x7fe3ec003a30 2023-10-16T11:17:26.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.981+0000 7fe408c4b700 1 -- 172.21.15.158:0/2120670838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 msgr2=0x7fe3ec003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.981+0000 7fe408c4b700 1 --2- 172.21.15.158:0/2120670838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe3ec003e10 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40018d0 tx=0x7fe3ec005880 comp rx=0 tx=0).stop 2023-10-16T11:17:26.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.981+0000 7fe402d9d700 1 -- 172.21.15.158:0/2120670838 reap_dead start 2023-10-16T11:17:26.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.981+0000 7fe408c4b700 1 -- 172.21.15.158:0/2120670838 shutdown_connections 2023-10-16T11:17:26.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.981+0000 7fe408c4b700 1 -- 172.21.15.158:0/2120670838 >> 172.21.15.158:0/2120670838 conn(0x7fe40406da40 msgr2=0x7fe40406de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:26.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.981+0000 7fe408c4b700 1 -- 172.21.15.158:0/2120670838 shutdown_connections 2023-10-16T11:17:26.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.981+0000 7fe408c4b700 1 -- 172.21.15.158:0/2120670838 wait complete. 2023-10-16T11:17:26.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.982+0000 7fe408c4b700 1 Processor -- start 2023-10-16T11:17:26.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.982+0000 7fe408c4b700 1 -- start start 2023-10-16T11:17:26.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe408c4b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe4041f6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe408c4b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40414cc60 0x7fe4041f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe408c4b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4041f28f0 0x7fe40406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:26.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe408c4b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4041f3b50 con 0x7fe3ec003a30 2023-10-16T11:17:26.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe408c4b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40414d420 con 0x7fe40414cc60 2023-10-16T11:17:26.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe408c4b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4041f2db0 con 0x7fe4041f28f0 2023-10-16T11:17:26.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe4041f6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe4041f6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36040/0 (socket says 172.21.15.158:36040) 2023-10-16T11:17:26.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 -- 172.21.15.158:0/1132505264 learned_addr learned my addr 172.21.15.158:0/1132505264 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:26.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe402d9d700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4041f28f0 0x7fe40406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe401d9b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40414cc60 0x7fe4041f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:26.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 -- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4041f28f0 msgr2=0x7fe40406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4041f28f0 0x7fe40406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 -- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40414cc60 msgr2=0x7fe4041f23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:26.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40414cc60 0x7fe4041f23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:26.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 -- 172.21.15.158:0/1132505264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4041f3070 con 0x7fe3ec003a30 2023-10-16T11:17:26.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe401d9b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40414cc60 0x7fe4041f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:26.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe40259c700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe4041f6c50 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c0e0 tx=0x7fe3f4001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:26.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.983+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400cb80 con 0x7fe3ec003a30 2023-10-16T11:17:26.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.984+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f4021070 con 0x7fe3ec003a30 2023-10-16T11:17:26.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.984+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4041f3350 con 0x7fe3ec003a30 2023-10-16T11:17:26.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.984+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4001330 con 0x7fe3ec003a30 2023-10-16T11:17:26.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.984+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40414d0e0 con 0x7fe3ec003a30 2023-10-16T11:17:26.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.984+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4041f39a0 con 0x7fe3ec003a30 2023-10-16T11:17:26.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.985+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 123) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe3f4023030 con 0x7fe3ec003a30 2023-10-16T11:17:26.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.985+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 7541+0+0 (secure 0 0 0) 0x7fe3f408a9f0 con 0x7fe3ec003a30 2023-10-16T11:17:26.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:26.990+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3f408ce60 con 0x7fe3ec003a30 2023-10-16T11:17:27.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.202+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 124) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe3f4001490 con 0x7fe3ec003a30 2023-10-16T11:17:27.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.203+0000 7fe3f2ffd700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe3e4071570 0x7fe3e4073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.204+0000 7fe401d9b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe3e4071570 0x7fe3e4073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:27.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.206+0000 7fe401d9b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe3e4071570 0x7fe3e4073960 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009b60 tx=0x7fe3f8008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:27.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.230+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe4040681b0 con 0x7fe3ec003a30 2023-10-16T11:17:27.232 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:17:27.230+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:17:27.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.232+0000 7fe3f2ffd700 1 -- 172.21.15.158:0/1132505264 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe3f406d1d0 con 0x7fe3ec003a30 2023-10-16T11:17:27.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe3e4071570 msgr2=0x7fe3e4073960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:27.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe3e4071570 0x7fe3e4073960 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009b60 tx=0x7fe3f8008040 comp rx=0 tx=0).stop 2023-10-16T11:17:27.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 msgr2=0x7fe4041f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:27.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe4041f6c50 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c0e0 tx=0x7fe3f4001230 comp rx=0 tx=0).stop 2023-10-16T11:17:27.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 shutdown_connections 2023-10-16T11:17:27.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4041f28f0 0x7fe40406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3ec003a30 0x7fe4041f6c50 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe3e4071570 0x7fe3e4073960 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 --2- 172.21.15.158:0/1132505264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40414cc60 0x7fe4041f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 >> 172.21.15.158:0/1132505264 conn(0x7fe40406da40 msgr2=0x7fe4040afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:27.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.235+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 shutdown_connections 2023-10-16T11:17:27.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.236+0000 7fe408c4b700 1 -- 172.21.15.158:0/1132505264 wait complete. 2023-10-16T11:17:27.240 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:17:27.250 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:17:27.272 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:17:27.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.374+0000 7fd74f99b700 1 Processor -- start 2023-10-16T11:17:27.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.374+0000 7fd74f99b700 1 -- start start 2023-10-16T11:17:27.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74f99b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd74815d990 0x7fd74806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74f99b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 0x7fd748161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74f99b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd748162010 0x7fd7480cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74f99b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7481616b0 con 0x7fd74815d990 2023-10-16T11:17:27.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74f99b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7480ced70 con 0x7fd748162010 2023-10-16T11:17:27.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74f99b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74815dd70 con 0x7fd748070300 2023-10-16T11:17:27.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd74815d990 0x7fd74806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:27.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd74815d990 0x7fd74806d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36082/0 (socket says 172.21.15.158:36082) 2023-10-16T11:17:27.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 -- 172.21.15.158:0/3339462692 learned_addr learned my addr 172.21.15.158:0/3339462692 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:27.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 -- 172.21.15.158:0/3339462692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 msgr2=0x7fd748161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:27.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 --2- 172.21.15.158:0/3339462692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 0x7fd748161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 -- 172.21.15.158:0/3339462692 >> v2:172.21.15.172:3300/0 conn(0x7fd748162010 msgr2=0x7fd7480cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:27.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 --2- 172.21.15.158:0/3339462692 >> v2:172.21.15.172:3300/0 conn(0x7fd748162010 0x7fd7480cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 -- 172.21.15.158:0/3339462692 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd748070780 con 0x7fd74815d990 2023-10-16T11:17:27.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd74d737700 1 --2- 172.21.15.158:0/3339462692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd74806d5d0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fd7480d0160 tx=0x7fd744009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369ee6c79a07877e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:27.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd73ffff700 1 -- 172.21.15.158:0/3339462692 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74400b860 con 0x7fd74815d990 2023-10-16T11:17:27.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd73ffff700 1 -- 172.21.15.158:0/3339462692 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74400b9c0 con 0x7fd74815d990 2023-10-16T11:17:27.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.375+0000 7fd73ffff700 1 -- 172.21.15.158:0/3339462692 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74400bd50 con 0x7fd74815d990 2023-10-16T11:17:27.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 -- 172.21.15.158:0/3339462692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 msgr2=0x7fd74806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:27.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 --2- 172.21.15.158:0/3339462692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd74806d5d0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fd7480d0160 tx=0x7fd744009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:27.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 -- 172.21.15.158:0/3339462692 shutdown_connections 2023-10-16T11:17:27.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 --2- 172.21.15.158:0/3339462692 >> v2:172.21.15.172:3300/0 conn(0x7fd748162010 0x7fd7480cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 --2- 172.21.15.158:0/3339462692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd74806d5d0 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 --2- 172.21.15.158:0/3339462692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 0x7fd748161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 -- 172.21.15.158:0/3339462692 >> 172.21.15.158:0/3339462692 conn(0x7fd7480cdf20 msgr2=0x7fd7480ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:27.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 -- 172.21.15.158:0/3339462692 shutdown_connections 2023-10-16T11:17:27.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.376+0000 7fd74f99b700 1 -- 172.21.15.158:0/3339462692 wait complete. 2023-10-16T11:17:27.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.377+0000 7fd74f99b700 1 Processor -- start 2023-10-16T11:17:27.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.377+0000 7fd74f99b700 1 -- start start 2023-10-16T11:17:27.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74f99b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 0x7fd7481529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74f99b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd7481f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74f99b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd748162010 0x7fd7481f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74f99b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7480c5280 con 0x7fd74815d990 2023-10-16T11:17:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74f99b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7480c6d40 con 0x7fd748162010 2023-10-16T11:17:27.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74f99b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd748152f20 con 0x7fd748070300 2023-10-16T11:17:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd7481f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd7481f68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36090/0 (socket says 172.21.15.158:36090) 2023-10-16T11:17:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 -- 172.21.15.158:0/75794302 learned_addr learned my addr 172.21.15.158:0/75794302 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 -- 172.21.15.158:0/75794302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 msgr2=0x7fd7481529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:27.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 0x7fd7481529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 -- 172.21.15.158:0/75794302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd748162010 msgr2=0x7fd7481f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74df38700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd748162010 0x7fd7481f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd748162010 0x7fd7481f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:27.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.378+0000 7fd74cf36700 1 -- 172.21.15.158:0/75794302 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd744009550 con 0x7fd74815d990 2023-10-16T11:17:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.379+0000 7fd74cf36700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd7481f68d0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fd73800b4a0 tx=0x7fd73800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.379+0000 7fd73dffb700 1 -- 172.21.15.158:0/75794302 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738011830 con 0x7fd74815d990 2023-10-16T11:17:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.379+0000 7fd73dffb700 1 -- 172.21.15.158:0/75794302 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd738011990 con 0x7fd74815d990 2023-10-16T11:17:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.379+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7480b4070 con 0x7fd74815d990 2023-10-16T11:17:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.379+0000 7fd73dffb700 1 -- 172.21.15.158:0/75794302 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738011d40 con 0x7fd74815d990 2023-10-16T11:17:27.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.379+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd748159070 con 0x7fd74815d990 2023-10-16T11:17:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.380+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7480681b0 con 0x7fd74815d990 2023-10-16T11:17:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.380+0000 7fd73dffb700 1 -- 172.21.15.158:0/75794302 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 124) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd73801a400 con 0x7fd74815d990 2023-10-16T11:17:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.381+0000 7fd73dffb700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fd72c06ad50 0x7fd72c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:27.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.381+0000 7fd73dffb700 1 -- 172.21.15.158:0/75794302 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 7627+0+0 (secure 0 0 0) 0x7fd738089a00 con 0x7fd74815d990 2023-10-16T11:17:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.381+0000 7fd74d737700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fd72c06ad50 0x7fd72c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:27.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.381+0000 7fd74d737700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fd72c06ad50 0x7fd72c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd744009c40 tx=0x7fd744010000 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:27.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.385+0000 7fd73dffb700 1 -- 172.21.15.158:0/75794302 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7380545f0 con 0x7fd74815d990 2023-10-16T11:17:27.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:27.627+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd7480c53c0 con 0x7fd74815d990 2023-10-16T11:17:28.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.233+0000 7fd73dffb700 1 -- 172.21.15.158:0/75794302 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd7480c53c0 con 0x7fd74815d990 2023-10-16T11:17:28.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fd72c06ad50 msgr2=0x7fd72c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:28.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fd72c06ad50 0x7fd72c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd744009c40 tx=0x7fd744010000 comp rx=0 tx=0).stop 2023-10-16T11:17:28.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 msgr2=0x7fd7481f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:28.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd7481f68d0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fd73800b4a0 tx=0x7fd73800b860 comp rx=0 tx=0).stop 2023-10-16T11:17:28.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 shutdown_connections 2023-10-16T11:17:28.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fd72c06ad50 0x7fd72c06d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd748162010 0x7fd7481f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd748070300 0x7fd7481529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 --2- 172.21.15.158:0/75794302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd74815d990 0x7fd7481f68d0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 >> 172.21.15.158:0/75794302 conn(0x7fd7480cdf20 msgr2=0x7fd74815df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:28.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 shutdown_connections 2023-10-16T11:17:28.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.236+0000 7fd74f99b700 1 -- 172.21.15.158:0/75794302 wait complete. 2023-10-16T11:17:28.253 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:17:28.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.420+0000 7f9ae90e3700 1 Processor -- start 2023-10-16T11:17:28.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.420+0000 7f9ae90e3700 1 -- start start 2023-10-16T11:17:28.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae90e3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ae40cf070 0x7f9ae40cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:28.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae90e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cbdd0 0x7f9ae40cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:28.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae90e3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae415dc20 con 0x7f9ae415d840 2023-10-16T11:17:28.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae90e3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae40cfa60 con 0x7f9ae40cf070 2023-10-16T11:17:28.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae90e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae406c0e0 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ae40cf070 0x7f9ae40cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:28.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cbdd0 0x7f9ae40cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:28.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae2d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9ae415d840 0x7f9ae4161d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:40196/0 (socket says 172.21.15.158:40196) 2023-10-16T11:17:28.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.421+0000 7f9ae2d9d700 1 -- 172.21.15.158:0/3110284634 learned_addr learned my addr 172.21.15.158:0/3110284634 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:28.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae359e700 1 -- 172.21.15.158:0/3110284634 >> v2:172.21.15.172:3300/0 conn(0x7f9ae40cf070 msgr2=0x7f9ae40cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:28.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae359e700 1 --2- 172.21.15.158:0/3110284634 >> v2:172.21.15.172:3300/0 conn(0x7f9ae40cf070 0x7f9ae40cf520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae359e700 1 -- 172.21.15.158:0/3110284634 >> v1:172.21.15.172:6789/0 conn(0x7f9ae415d840 legacy=0x7f9ae4161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:28.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae359e700 1 -- 172.21.15.158:0/3110284634 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae406c3c0 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae1d9b700 1 -- 172.21.15.158:0/3110284634 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217579053 0 0) 0x7f9ad8001110 con 0x7f9ae415d840 2023-10-16T11:17:28.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae259c700 1 --2- 172.21.15.158:0/3110284634 >> v2:172.21.15.172:3300/0 conn(0x7f9ae40cf070 0x7f9ae40cf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:28.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae359e700 1 --2- 172.21.15.158:0/3110284634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cbdd0 0x7f9ae40cc260 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a3a8fb80c790fbe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:28.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae1d9b700 1 -- 172.21.15.158:0/3110284634 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00e070 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae1d9b700 1 -- 172.21.15.158:0/3110284634 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc00bb20 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.422+0000 7f9ae1d9b700 1 -- 172.21.15.158:0/3110284634 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc012430 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 -- 172.21.15.158:0/3110284634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cbdd0 msgr2=0x7f9ae40cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:28.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/3110284634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cbdd0 0x7f9ae40cc260 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc00b330 comp rx=0 tx=0).stop 2023-10-16T11:17:28.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 -- 172.21.15.158:0/3110284634 shutdown_connections 2023-10-16T11:17:28.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/3110284634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cbdd0 0x7f9ae40cc260 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/3110284634 >> v2:172.21.15.172:3300/0 conn(0x7f9ae40cf070 0x7f9ae40cf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 -- 172.21.15.158:0/3110284634 >> 172.21.15.158:0/3110284634 conn(0x7f9ae406a8d0 msgr2=0x7f9ae406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:28.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 -- 172.21.15.158:0/3110284634 shutdown_connections 2023-10-16T11:17:28.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.423+0000 7f9ae90e3700 1 -- 172.21.15.158:0/3110284634 wait complete. 2023-10-16T11:17:28.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.425+0000 7f9ae90e3700 1 Processor -- start 2023-10-16T11:17:28.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.425+0000 7f9ae90e3700 1 -- start start 2023-10-16T11:17:28.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ae40cbdd0 0x7f9ae40caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:28.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cf070 0x7f9ae4150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:28.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ae415d840 0x7f9ae4150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:28.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c91c0 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c94a0 con 0x7f9ae415d840 2023-10-16T11:17:28.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c9780 con 0x7f9ae40cf070 2023-10-16T11:17:28.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ae40cbdd0 0x7f9ae40caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:28.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ae40cbdd0 0x7f9ae40caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36122/0 (socket says 172.21.15.158:36122) 2023-10-16T11:17:28.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 -- 172.21.15.158:0/1412509875 learned_addr learned my addr 172.21.15.158:0/1412509875 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:28.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae259c700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cf070 0x7f9ae4150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:28.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae359e700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ae415d840 0x7f9ae4150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:28.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 -- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cf070 msgr2=0x7f9ae4150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:28.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cf070 0x7f9ae4150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 -- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ae415d840 msgr2=0x7f9ae4150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:28.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ae415d840 0x7f9ae4150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:28.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 -- 172.21.15.158:0/1412509875 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40094a0 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae259c700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cf070 0x7f9ae4150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:28.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae359e700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ae415d840 0x7f9ae4150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:28.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae2d9d700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ae40cbdd0 0x7f9ae40caa60 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c550 tx=0x7f9ad800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:28.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8004c80 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad8004e00 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8007310 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc009a70 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.426+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae40c9d40 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.427+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae40c9f60 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.428+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 124) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9ad8007470 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.428+0000 7f9acb7fe700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9ad006ad50 0x7f9ad006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:28.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.428+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f9ad8089520 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.428+0000 7f9ae259c700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9ad006ad50 0x7f9ad006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:28.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.429+0000 7f9ae259c700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9ad006ad50 0x7f9ad006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9ae41f68e0 tx=0x7f9ad4008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:28.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.433+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ad8054190 con 0x7f9ae40cbdd0 2023-10-16T11:17:28.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:28.626+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9ae415dc20 con 0x7f9ae40cbdd0 2023-10-16T11:17:29.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.210+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 125) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9ad801ad90 con 0x7f9ae40cbdd0 2023-10-16T11:17:29.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.591+0000 7f9acb7fe700 1 -- 172.21.15.158:0/1412509875 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v191) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9ad8073830 con 0x7f9ae40cbdd0 2023-10-16T11:17:29.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.598+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9ad006ad50 msgr2=0x7f9ad006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:29.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.598+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9ad006ad50 0x7f9ad006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9ae41f68e0 tx=0x7f9ad4008040 comp rx=0 tx=0).stop 2023-10-16T11:17:29.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.598+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ae40cbdd0 msgr2=0x7f9ae40caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:29.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.598+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ae40cbdd0 0x7f9ae40caa60 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c550 tx=0x7f9ad800c910 comp rx=0 tx=0).stop 2023-10-16T11:17:29.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.599+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 shutdown_connections 2023-10-16T11:17:29.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.599+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9ad006ad50 0x7f9ad006d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.599+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ae415d840 0x7f9ae4150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.599+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ae40cbdd0 0x7f9ae40caa60 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.599+0000 7f9ae90e3700 1 --2- 172.21.15.158:0/1412509875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ae40cf070 0x7f9ae4150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.599+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 >> 172.21.15.158:0/1412509875 conn(0x7f9ae406a8d0 msgr2=0x7f9ae40cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:29.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.600+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 shutdown_connections 2023-10-16T11:17:29.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.602+0000 7f9ae90e3700 1 -- 172.21.15.158:0/1412509875 wait complete. 2023-10-16T11:17:29.606 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:17:29.620 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:17:29.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.748+0000 7f67cea36700 1 Processor -- start 2023-10-16T11:17:29.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.748+0000 7f67cea36700 1 -- start start 2023-10-16T11:17:29.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67cea36700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f67c8160d40 0x7f67c80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:29.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67cea36700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c80c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:29.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67cea36700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c80ce110 con 0x7f67c80c4ef0 2023-10-16T11:17:29.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67cea36700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c80c53a0 con 0x7f67c8160d40 2023-10-16T11:17:29.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67cea36700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80c61e0 con 0x7f67c80cf420 2023-10-16T11:17:29.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f67c8160d40 0x7f67c80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:29.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f67c8160d40 0x7f67c80ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36130/0 (socket says 172.21.15.158:36130) 2023-10-16T11:17:29.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 -- 172.21.15.158:0/2097234601 learned_addr learned my addr 172.21.15.158:0/2097234601 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:29.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67ccfd3700 1 --1- 172.21.15.158:0/2097234601 >> v1:172.21.15.158:6789/0 conn(0x7f67c80c4ef0 0x7f67c806d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36310/0 (socket says 172.21.15.158:36310) 2023-10-16T11:17:29.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 -- 172.21.15.158:0/2097234601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 msgr2=0x7f67c80c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:29.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c77fe700 1 --2- 172.21.15.158:0/2097234601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c80c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:29.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 --2- 172.21.15.158:0/2097234601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c80c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c6ffd700 1 -- 172.21.15.158:0/2097234601 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784895412 0 0) 0x7f67c80ce110 con 0x7f67c80c4ef0 2023-10-16T11:17:29.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 -- 172.21.15.158:0/2097234601 >> v1:172.21.15.158:6789/0 conn(0x7f67c80c4ef0 legacy=0x7f67c806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:29.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 -- 172.21.15.158:0/2097234601 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c80ce3f0 con 0x7f67c8160d40 2023-10-16T11:17:29.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c77fe700 1 --2- 172.21.15.158:0/2097234601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c80c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:29.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.749+0000 7f67c7fff700 1 --2- 172.21.15.158:0/2097234601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8160d40 0x7f67c80ceee0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0098f0 tx=0x7f67bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b9cd57efccd135 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:29.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67c6ffd700 1 -- 172.21.15.158:0/2097234601 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc00b8a0 con 0x7f67c8160d40 2023-10-16T11:17:29.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67c6ffd700 1 -- 172.21.15.158:0/2097234601 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc00ba00 con 0x7f67c8160d40 2023-10-16T11:17:29.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67c6ffd700 1 -- 172.21.15.158:0/2097234601 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc00bd90 con 0x7f67c8160d40 2023-10-16T11:17:29.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 -- 172.21.15.158:0/2097234601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8160d40 msgr2=0x7f67c80ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:29.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 --2- 172.21.15.158:0/2097234601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8160d40 0x7f67c80ceee0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0098f0 tx=0x7f67bc009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:29.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 -- 172.21.15.158:0/2097234601 shutdown_connections 2023-10-16T11:17:29.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 --2- 172.21.15.158:0/2097234601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c8160d40 0x7f67c80ceee0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 --2- 172.21.15.158:0/2097234601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c80c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 -- 172.21.15.158:0/2097234601 >> 172.21.15.158:0/2097234601 conn(0x7f67c80cc5b0 msgr2=0x7f67c80cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:29.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 -- 172.21.15.158:0/2097234601 shutdown_connections 2023-10-16T11:17:29.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.750+0000 7f67cea36700 1 -- 172.21.15.158:0/2097234601 wait complete. 2023-10-16T11:17:29.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.751+0000 7f67cea36700 1 Processor -- start 2023-10-16T11:17:29.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67cea36700 1 -- start start 2023-10-16T11:17:29.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67cea36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c80c4ef0 0x7f67c806ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:29.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67cea36700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c8159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:29.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67cea36700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8160d40 0x7f67c815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:29.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67cea36700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80ae670 con 0x7f67c80c4ef0 2023-10-16T11:17:29.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67cea36700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80ada60 con 0x7f67c8160d40 2023-10-16T11:17:29.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67cea36700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80add40 con 0x7f67c80cf420 2023-10-16T11:17:29.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67c7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c80c4ef0 0x7f67c806ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:29.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67c7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c80c4ef0 0x7f67c806ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36140/0 (socket says 172.21.15.158:36140) 2023-10-16T11:17:29.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.752+0000 7f67c7fff700 1 -- 172.21.15.158:0/1779624948 learned_addr learned my addr 172.21.15.158:0/1779624948 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:29.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c7fff700 1 -- 172.21.15.158:0/1779624948 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 msgr2=0x7f67c8159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:29.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c7fff700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c8159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c7fff700 1 -- 172.21.15.158:0/1779624948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8160d40 msgr2=0x7f67c815a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:29.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c7fff700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8160d40 0x7f67c815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:29.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c7fff700 1 -- 172.21.15.158:0/1779624948 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc009550 con 0x7f67c80c4ef0 2023-10-16T11:17:29.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c7fff700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c80c4ef0 0x7f67c806ae20 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f67bc000c00 tx=0x7f67bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:29.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc0038a0 con 0x7f67c80c4ef0 2023-10-16T11:17:29.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc01e070 con 0x7f67c80c4ef0 2023-10-16T11:17:29.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c80ae020 con 0x7f67c80c4ef0 2023-10-16T11:17:29.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc003c20 con 0x7f67c80c4ef0 2023-10-16T11:17:29.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.753+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c806b360 con 0x7f67c80c4ef0 2023-10-16T11:17:29.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.754+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c8066db0 con 0x7f67c80c4ef0 2023-10-16T11:17:29.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.754+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 125) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f67bc025480 con 0x7f67c80c4ef0 2023-10-16T11:17:29.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.755+0000 7f67c4ff9700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f67b406af90 0x7f67b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:29.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.755+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 7189+0+0 (secure 0 0 0) 0x7f67bc020070 con 0x7f67c80c4ef0 2023-10-16T11:17:29.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.755+0000 7f67c77fe700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f67b406af90 0x7f67b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:29.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.755+0000 7f67c77fe700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f67b406af90 0x7f67b406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f67c81f5c90 tx=0x7f67b8008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:29.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.759+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f67bc0bdfe0 con 0x7f67c80c4ef0 2023-10-16T11:17:29.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:29.953+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f67c80681b0 con 0x7f67c80c4ef0 2023-10-16T11:17:30.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.624+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v192) v1 ==== 169+0+0 (secure 0 0 0) 0x7f67bc015020 con 0x7f67c80c4ef0 2023-10-16T11:17:30.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.630+0000 7f67c4ff9700 1 -- 172.21.15.158:0/1779624948 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f67bc056070 con 0x7f67c80c4ef0 2023-10-16T11:17:30.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f67b406af90 msgr2=0x7f67b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f67b406af90 0x7f67b406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f67c81f5c90 tx=0x7f67b8008040 comp rx=0 tx=0).stop 2023-10-16T11:17:30.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c80c4ef0 msgr2=0x7f67c806ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c80c4ef0 0x7f67c806ae20 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f67bc000c00 tx=0x7f67bc004750 comp rx=0 tx=0).stop 2023-10-16T11:17:30.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 shutdown_connections 2023-10-16T11:17:30.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f67b406af90 0x7f67b406d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c8160d40 0x7f67c815a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67c80c4ef0 0x7f67c806ae20 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 --2- 172.21.15.158:0/1779624948 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c80cf420 0x7f67c8159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.635+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 >> 172.21.15.158:0/1779624948 conn(0x7f67c80cc5b0 msgr2=0x7f67c815eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:30.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.636+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 shutdown_connections 2023-10-16T11:17:30.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.636+0000 7f67cea36700 1 -- 172.21.15.158:0/1779624948 wait complete. 2023-10-16T11:17:30.639 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:17:30.652 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:17:30.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.774+0000 7f43c89cd700 1 Processor -- start 2023-10-16T11:17:30.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.774+0000 7f43c89cd700 1 -- start start 2023-10-16T11:17:30.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c89cd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f43c4160d40 0x7f43c40ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:30.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c89cd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f43c40c50c0 0x7f43c406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:30.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c89cd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c40ce110 con 0x7f43c40cf420 2023-10-16T11:17:30.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c89cd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c40c61e0 con 0x7f43c4160d40 2023-10-16T11:17:30.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c89cd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c40ce3f0 con 0x7f43c40c50c0 2023-10-16T11:17:30.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f43c4160d40 0x7f43c40ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:30.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f43c4160d40 0x7f43c40ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36144/0 (socket says 172.21.15.158:36144) 2023-10-16T11:17:30.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c259c700 1 -- 172.21.15.158:0/975211491 learned_addr learned my addr 172.21.15.158:0/975211491 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:30.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c2d9d700 1 --2- 172.21.15.158:0/975211491 >> v2:172.21.15.172:3300/0 conn(0x7f43c40c50c0 0x7f43c406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:30.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c259c700 1 -- 172.21.15.158:0/975211491 >> v2:172.21.15.172:3300/0 conn(0x7f43c40c50c0 msgr2=0x7f43c406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:30.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c259c700 1 --2- 172.21.15.158:0/975211491 >> v2:172.21.15.172:3300/0 conn(0x7f43c40c50c0 0x7f43c406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c259c700 1 -- 172.21.15.158:0/975211491 >> v1:172.21.15.158:6789/0 conn(0x7f43c40cf420 legacy=0x7f43c40c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c159a700 1 -- 172.21.15.158:0/975211491 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387104034 0 0) 0x7f43c40ce110 con 0x7f43c40cf420 2023-10-16T11:17:30.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.775+0000 7f43c259c700 1 -- 172.21.15.158:0/975211491 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c40ce6d0 con 0x7f43c4160d40 2023-10-16T11:17:30.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c259c700 1 --2- 172.21.15.158:0/975211491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40ceee0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f43c40ca110 tx=0x7f43b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5123e99a54eceffd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:30.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c159a700 1 -- 172.21.15.158:0/975211491 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b4011830 con 0x7f43c4160d40 2023-10-16T11:17:30.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c159a700 1 -- 172.21.15.158:0/975211491 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b4011990 con 0x7f43c4160d40 2023-10-16T11:17:30.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c159a700 1 -- 172.21.15.158:0/975211491 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b4011d20 con 0x7f43c4160d40 2023-10-16T11:17:30.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 -- 172.21.15.158:0/975211491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 msgr2=0x7f43c40ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 --2- 172.21.15.158:0/975211491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40ceee0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f43c40ca110 tx=0x7f43b4009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 -- 172.21.15.158:0/975211491 shutdown_connections 2023-10-16T11:17:30.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 --2- 172.21.15.158:0/975211491 >> v2:172.21.15.172:3300/0 conn(0x7f43c40c50c0 0x7f43c406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 --2- 172.21.15.158:0/975211491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40ceee0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 -- 172.21.15.158:0/975211491 >> 172.21.15.158:0/975211491 conn(0x7f43c40cc5b0 msgr2=0x7f43c40cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:30.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 -- 172.21.15.158:0/975211491 shutdown_connections 2023-10-16T11:17:30.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.776+0000 7f43c89cd700 1 -- 172.21.15.158:0/975211491 wait complete. 2023-10-16T11:17:30.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.778+0000 7f43c89cd700 1 Processor -- start 2023-10-16T11:17:30.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.778+0000 7f43c89cd700 1 -- start start 2023-10-16T11:17:30.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.778+0000 7f43c89cd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43c40c50c0 0x7f43c415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:30.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.778+0000 7f43c89cd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43c40cf420 0x7f43c40c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:30.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c89cd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:30.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c89cd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40ae9b0 con 0x7f43c4160d40 2023-10-16T11:17:30.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c89cd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40aec90 con 0x7f43c40cf420 2023-10-16T11:17:30.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c89cd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40aef70 con 0x7f43c40c50c0 2023-10-16T11:17:30.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:30.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36154/0 (socket says 172.21.15.158:36154) 2023-10-16T11:17:30.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 -- 172.21.15.158:0/2908000177 learned_addr learned my addr 172.21.15.158:0/2908000177 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:30.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c259c700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43c40c50c0 0x7f43c415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:30.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c1d9b700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43c40cf420 0x7f43c40c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:30.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 -- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43c40c50c0 msgr2=0x7f43c415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43c40c50c0 0x7f43c415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 -- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43c40cf420 msgr2=0x7f43c40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43c40cf420 0x7f43c40c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 -- 172.21.15.158:0/2908000177 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b4009550 con 0x7f43c4160d40 2023-10-16T11:17:30.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c2d9d700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40adf60 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00efc0 tx=0x7f43bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:30.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43b2ffd700 1 -- 172.21.15.158:0/2908000177 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc009ea0 con 0x7f43c4160d40 2023-10-16T11:17:30.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43b2ffd700 1 -- 172.21.15.158:0/2908000177 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc00f040 con 0x7f43c4160d40 2023-10-16T11:17:30.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c41612a0 con 0x7f43c4160d40 2023-10-16T11:17:30.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43b2ffd700 1 -- 172.21.15.158:0/2908000177 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc00ced0 con 0x7f43c4160d40 2023-10-16T11:17:30.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.779+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c415be60 con 0x7f43c4160d40 2023-10-16T11:17:30.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.780+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c4066db0 con 0x7f43c4160d40 2023-10-16T11:17:30.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.781+0000 7f43b2ffd700 1 -- 172.21.15.158:0/2908000177 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f43bc014480 con 0x7f43c4160d40 2023-10-16T11:17:30.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.781+0000 7f43b2ffd700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f43a406b020 0x7f43a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:30.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.781+0000 7f43b2ffd700 1 -- 172.21.15.158:0/2908000177 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f43bc08a7e0 con 0x7f43c4160d40 2023-10-16T11:17:30.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.781+0000 7f43c259c700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f43a406b020 0x7f43a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:30.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.782+0000 7f43c259c700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f43a406b020 0x7f43a406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f43b4014040 tx=0x7f43b4008030 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:30.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.786+0000 7f43b2ffd700 1 -- 172.21.15.158:0/2908000177 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43bc0551e0 con 0x7f43c4160d40 2023-10-16T11:17:30.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.977+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f43c40681b0 con 0x7f43c4160d40 2023-10-16T11:17:30.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.981+0000 7f43b2ffd700 1 -- 172.21.15.158:0/2908000177 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7f43bc058800 con 0x7f43c4160d40 2023-10-16T11:17:30.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f43a406b020 msgr2=0x7f43a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f43a406b020 0x7f43a406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f43b4014040 tx=0x7f43b4008030 comp rx=0 tx=0).stop 2023-10-16T11:17:30.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 msgr2=0x7f43c40adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:30.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40adf60 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00efc0 tx=0x7f43bc00c3a0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 shutdown_connections 2023-10-16T11:17:30.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43c4160d40 0x7f43c40adf60 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43c40c50c0 0x7f43c415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f43a406b020 0x7f43a406d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 --2- 172.21.15.158:0/2908000177 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43c40cf420 0x7f43c40c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:30.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 >> 172.21.15.158:0/2908000177 conn(0x7f43c40cc5b0 msgr2=0x7f43c415eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:30.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 shutdown_connections 2023-10-16T11:17:30.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:30.984+0000 7f43c89cd700 1 -- 172.21.15.158:0/2908000177 wait complete. 2023-10-16T11:17:30.998 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:17:31.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.119+0000 7f4352a5a700 1 Processor -- start 2023-10-16T11:17:31.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.119+0000 7f4352a5a700 1 -- start start 2023-10-16T11:17:31.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4352a5a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f434c15d6c0 0x7f434c0cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4352a5a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4352a5a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f434c0d2370 0x7f434c0cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4352a5a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c0d40a0 con 0x7f434c0d2370 2023-10-16T11:17:31.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4352a5a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c0d1ee0 con 0x7f434c15d6c0 2023-10-16T11:17:31.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4352a5a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0d12d0 con 0x7f434c15daa0 2023-10-16T11:17:31.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4350ff7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f434c0d2370 0x7f434c0cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4350ff7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f434c0d2370 0x7f434c0cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36162/0 (socket says 172.21.15.158:36162) 2023-10-16T11:17:31.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.120+0000 7f4350ff7700 1 -- 172.21.15.158:0/2483882158 learned_addr learned my addr 172.21.15.158:0/2483882158 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:31.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4350ff7700 1 -- 172.21.15.158:0/2483882158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 msgr2=0x7f434c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:31.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f434b7fe700 1 --2- 172.21.15.158:0/2483882158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4350ff7700 1 --2- 172.21.15.158:0/2483882158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4350ff7700 1 -- 172.21.15.158:0/2483882158 >> v2:172.21.15.172:3300/0 conn(0x7f434c15d6c0 msgr2=0x7f434c0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:31.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4350ff7700 1 --2- 172.21.15.158:0/2483882158 >> v2:172.21.15.172:3300/0 conn(0x7f434c15d6c0 0x7f434c0cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4350ff7700 1 -- 172.21.15.158:0/2483882158 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c0d15b0 con 0x7f434c0d2370 2023-10-16T11:17:31.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4350ff7700 1 --2- 172.21.15.158:0/2483882158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c0d2370 0x7f434c0cec50 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f434c0674f0 tx=0x7f434400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b04dd712d0eca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f434affd700 1 -- 172.21.15.158:0/2483882158 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434400e070 con 0x7f434c0d2370 2023-10-16T11:17:31.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f434affd700 1 -- 172.21.15.158:0/2483882158 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434400b9c0 con 0x7f434c0d2370 2023-10-16T11:17:31.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f434affd700 1 -- 172.21.15.158:0/2483882158 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434400bd70 con 0x7f434c0d2370 2023-10-16T11:17:31.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4352a5a700 1 -- 172.21.15.158:0/2483882158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c0d2370 msgr2=0x7f434c0cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.121+0000 7f4352a5a700 1 --2- 172.21.15.158:0/2483882158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c0d2370 0x7f434c0cec50 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f434c0674f0 tx=0x7f434400b330 comp rx=0 tx=0).stop 2023-10-16T11:17:31.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.122+0000 7f4352a5a700 1 -- 172.21.15.158:0/2483882158 shutdown_connections 2023-10-16T11:17:31.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.122+0000 7f4352a5a700 1 --2- 172.21.15.158:0/2483882158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c0d2370 0x7f434c0cec50 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.122+0000 7f4352a5a700 1 --2- 172.21.15.158:0/2483882158 >> v2:172.21.15.172:3300/0 conn(0x7f434c15d6c0 0x7f434c0cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.122+0000 7f4352a5a700 1 --2- 172.21.15.158:0/2483882158 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.122+0000 7f4352a5a700 1 -- 172.21.15.158:0/2483882158 >> 172.21.15.158:0/2483882158 conn(0x7f434c09c940 msgr2=0x7f434c0d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:31.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.122+0000 7f4352a5a700 1 -- 172.21.15.158:0/2483882158 shutdown_connections 2023-10-16T11:17:31.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.122+0000 7f4352a5a700 1 -- 172.21.15.158:0/2483882158 wait complete. 2023-10-16T11:17:31.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.123+0000 7f4352a5a700 1 Processor -- start 2023-10-16T11:17:31.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.123+0000 7f4352a5a700 1 -- start start 2023-10-16T11:17:31.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f434c0d2370 0x7f434c1503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c15d6c0 0x7f434c0b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0b43e0 con 0x7f434c15d6c0 2023-10-16T11:17:31.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0b49a0 con 0x7f434c0d2370 2023-10-16T11:17:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c150e60 con 0x7f434c15daa0 2023-10-16T11:17:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c15d6c0 0x7f434c0b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c15d6c0 0x7f434c0b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36166/0 (socket says 172.21.15.158:36166) 2023-10-16T11:17:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 -- 172.21.15.158:0/4248637239 learned_addr learned my addr 172.21.15.158:0/4248637239 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4350ff7700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434bfff700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f434c0d2370 0x7f434c1503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 -- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 msgr2=0x7f434c0b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 -- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f434c0d2370 msgr2=0x7f434c1503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f434c0d2370 0x7f434c1503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 -- 172.21.15.158:0/4248637239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4344009b30 con 0x7f434c15d6c0 2023-10-16T11:17:31.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f434b7fe700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c15d6c0 0x7f434c0b4e30 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f433c00c7f0 tx=0x7f433c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4348ff9700 1 -- 172.21.15.158:0/4248637239 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c004220 con 0x7f434c15d6c0 2023-10-16T11:17:31.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4348ff9700 1 -- 172.21.15.158:0/4248637239 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c004380 con 0x7f434c15d6c0 2023-10-16T11:17:31.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c0b3ba0 con 0x7f434c15d6c0 2023-10-16T11:17:31.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4348ff9700 1 -- 172.21.15.158:0/4248637239 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c0036a0 con 0x7f434c15d6c0 2023-10-16T11:17:31.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.124+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c0b4630 con 0x7f434c15d6c0 2023-10-16T11:17:31.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.125+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c0b4b10 con 0x7f434c15d6c0 2023-10-16T11:17:31.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.126+0000 7f4348ff9700 1 -- 172.21.15.158:0/4248637239 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f433c003800 con 0x7f434c15d6c0 2023-10-16T11:17:31.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.126+0000 7f4348ff9700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f432c06b020 0x7f432c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.126+0000 7f4348ff9700 1 -- 172.21.15.158:0/4248637239 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f433c0895e0 con 0x7f434c15d6c0 2023-10-16T11:17:31.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.126+0000 7f434bfff700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f432c06b020 0x7f432c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.127+0000 7f434bfff700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f432c06b020 0x7f432c06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f434c0b60a0 tx=0x7f4334009190 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.131+0000 7f4348ff9700 1 -- 172.21.15.158:0/4248637239 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f433c0540f0 con 0x7f434c15d6c0 2023-10-16T11:17:31.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.328+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f434c0cf110 con 0x7f434c15d6c0 2023-10-16T11:17:31.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.332+0000 7f4348ff9700 1 -- 172.21.15.158:0/4248637239 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f433c057710 con 0x7f434c15d6c0 2023-10-16T11:17:31.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f432c06b020 msgr2=0x7f432c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f432c06b020 0x7f432c06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f434c0b60a0 tx=0x7f4334009190 comp rx=0 tx=0).stop 2023-10-16T11:17:31.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c15d6c0 msgr2=0x7f434c0b4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c15d6c0 0x7f434c0b4e30 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f433c00c7f0 tx=0x7f433c00cb00 comp rx=0 tx=0).stop 2023-10-16T11:17:31.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 shutdown_connections 2023-10-16T11:17:31.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f434c15daa0 0x7f434c0b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f432c06b020 0x7f432c06d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f434c0d2370 0x7f434c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 --2- 172.21.15.158:0/4248637239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f434c15d6c0 0x7f434c0b4e30 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 >> 172.21.15.158:0/4248637239 conn(0x7f434c09c940 msgr2=0x7f434c15c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:31.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 shutdown_connections 2023-10-16T11:17:31.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.334+0000 7f4352a5a700 1 -- 172.21.15.158:0/4248637239 wait complete. 2023-10-16T11:17:31.348 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:17:31.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.469+0000 7f12c7c0f700 1 Processor -- start 2023-10-16T11:17:31.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.469+0000 7f12c7c0f700 1 -- start start 2023-10-16T11:17:31.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c7c0f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c7c0f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12c00cc590 0x7f12c006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c7c0f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12c006ae10 0x7f12c00ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c7c0f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c006b280 con 0x7f12c006ae10 2023-10-16T11:17:31.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c7c0f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c00cbd30 con 0x7f12c00cc590 2023-10-16T11:17:31.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c7c0f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00cc1c0 con 0x7f12c015da40 2023-10-16T11:17:31.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12c006ae10 0x7f12c00ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12c006ae10 0x7f12c00ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36176/0 (socket says 172.21.15.158:36176) 2023-10-16T11:17:31.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 -- 172.21.15.158:0/3037358596 learned_addr learned my addr 172.21.15.158:0/3037358596 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:31.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c59ab700 1 --2- 172.21.15.158:0/3037358596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c51aa700 1 --2- 172.21.15.158:0/3037358596 >> v2:172.21.15.172:3300/0 conn(0x7f12c00cc590 0x7f12c006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 -- 172.21.15.158:0/3037358596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 msgr2=0x7f12c00caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:31.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 --2- 172.21.15.158:0/3037358596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c00caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 -- 172.21.15.158:0/3037358596 >> v2:172.21.15.172:3300/0 conn(0x7f12c00cc590 msgr2=0x7f12c006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:31.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 --2- 172.21.15.158:0/3037358596 >> v2:172.21.15.172:3300/0 conn(0x7f12c00cc590 0x7f12c006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 -- 172.21.15.158:0/3037358596 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c015de20 con 0x7f12c006ae10 2023-10-16T11:17:31.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.470+0000 7f12c61ac700 1 --2- 172.21.15.158:0/3037358596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c00ccd60 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f12c0070430 tx=0x7f12b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa93fd500969e549 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c49a9700 1 -- 172.21.15.158:0/3037358596 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b800e070 con 0x7f12c006ae10 2023-10-16T11:17:31.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c49a9700 1 -- 172.21.15.158:0/3037358596 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b800b9c0 con 0x7f12c006ae10 2023-10-16T11:17:31.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c49a9700 1 -- 172.21.15.158:0/3037358596 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b800bd70 con 0x7f12c006ae10 2023-10-16T11:17:31.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 -- 172.21.15.158:0/3037358596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 msgr2=0x7f12c00ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/3037358596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c00ccd60 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f12c0070430 tx=0x7f12b800b330 comp rx=0 tx=0).stop 2023-10-16T11:17:31.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 -- 172.21.15.158:0/3037358596 shutdown_connections 2023-10-16T11:17:31.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/3037358596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c00ccd60 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/3037358596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c00caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/3037358596 >> v2:172.21.15.172:3300/0 conn(0x7f12c00cc590 0x7f12c006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 -- 172.21.15.158:0/3037358596 >> 172.21.15.158:0/3037358596 conn(0x7f12c0099a10 msgr2=0x7f12c00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:31.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 -- 172.21.15.158:0/3037358596 shutdown_connections 2023-10-16T11:17:31.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.471+0000 7f12c7c0f700 1 -- 172.21.15.158:0/3037358596 wait complete. 2023-10-16T11:17:31.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.472+0000 7f12c7c0f700 1 Processor -- start 2023-10-16T11:17:31.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.472+0000 7f12c7c0f700 1 -- start start 2023-10-16T11:17:31.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c7c0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c01e5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c7c0f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12c00cc590 0x7f12c00b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c7c0f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c7c0f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01f2870 con 0x7f12c006ae10 2023-10-16T11:17:31.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c7c0f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01e6df0 con 0x7f12c00cc590 2023-10-16T11:17:31.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c7c0f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01e6170 con 0x7f12c015da40 2023-10-16T11:17:31.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c59ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c01e5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c59ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c01e5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36178/0 (socket says 172.21.15.158:36178) 2023-10-16T11:17:31.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c59ab700 1 -- 172.21.15.158:0/2921834829 learned_addr learned my addr 172.21.15.158:0/2921834829 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:31.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.473+0000 7f12c61ac700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c51aa700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12c00cc590 0x7f12c00b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c59ab700 1 -- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 msgr2=0x7f12c014e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c59ab700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c014e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c59ab700 1 -- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12c00cc590 msgr2=0x7f12c00b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c59ab700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12c00cc590 0x7f12c00b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c59ab700 1 -- 172.21.15.158:0/2921834829 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b8009b30 con 0x7f12c006ae10 2023-10-16T11:17:31.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c59ab700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c01e5860 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00d800 tx=0x7f12bc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12b67fc700 1 -- 172.21.15.158:0/2921834829 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc0125f0 con 0x7f12c006ae10 2023-10-16T11:17:31.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12b67fc700 1 -- 172.21.15.158:0/2921834829 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc012750 con 0x7f12c006ae10 2023-10-16T11:17:31.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c01e6450 con 0x7f12c006ae10 2023-10-16T11:17:31.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12b67fc700 1 -- 172.21.15.158:0/2921834829 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc012b00 con 0x7f12c006ae10 2023-10-16T11:17:31.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.474+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c01e6a10 con 0x7f12c006ae10 2023-10-16T11:17:31.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.475+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c00681b0 con 0x7f12c006ae10 2023-10-16T11:17:31.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.475+0000 7f12b67fc700 1 -- 172.21.15.158:0/2921834829 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f12bc012c60 con 0x7f12c006ae10 2023-10-16T11:17:31.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.476+0000 7f12b67fc700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f12a406b020 0x7f12a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.476+0000 7f12b67fc700 1 -- 172.21.15.158:0/2921834829 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f12bc089f40 con 0x7f12c006ae10 2023-10-16T11:17:31.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.476+0000 7f12c51aa700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f12a406b020 0x7f12a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.476+0000 7f12c51aa700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f12a406b020 0x7f12a406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f12c00c57a0 tx=0x7f12b0008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.480+0000 7f12b67fc700 1 -- 172.21.15.158:0/2921834829 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12bc054ad0 con 0x7f12c006ae10 2023-10-16T11:17:31.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.673+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f12c006f6b0 con 0x7f12c006ae10 2023-10-16T11:17:31.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.677+0000 7f12b67fc700 1 -- 172.21.15.158:0/2921834829 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7f12bc0580f0 con 0x7f12c006ae10 2023-10-16T11:17:31.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f12a406b020 msgr2=0x7f12a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f12a406b020 0x7f12a406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f12c00c57a0 tx=0x7f12b0008040 comp rx=0 tx=0).stop 2023-10-16T11:17:31.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 msgr2=0x7f12c01e5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c01e5860 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00d800 tx=0x7f12bc00dbc0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 shutdown_connections 2023-10-16T11:17:31.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12c015da40 0x7f12c014e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12c006ae10 0x7f12c01e5860 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f12a406b020 0x7f12a406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 --2- 172.21.15.158:0/2921834829 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12c00cc590 0x7f12c00b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 >> 172.21.15.158:0/2921834829 conn(0x7f12c0099a10 msgr2=0x7f12c00b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:31.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 shutdown_connections 2023-10-16T11:17:31.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.679+0000 7f12c7c0f700 1 -- 172.21.15.158:0/2921834829 wait complete. 2023-10-16T11:17:31.693 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T11:17:31.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.813+0000 7f82c8ca3700 1 Processor -- start 2023-10-16T11:17:31.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.813+0000 7f82c8ca3700 1 -- start start 2023-10-16T11:17:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c8ca3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f82c406ab40 0x7f82c406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c8ca3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82c40729b0 0x7f82c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c8ca3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c4071e50 con 0x7f82c406e5d0 2023-10-16T11:17:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c8ca3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c4161270 con 0x7f82c40729b0 2023-10-16T11:17:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c8ca3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c4161510 con 0x7f82c406ab40 2023-10-16T11:17:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82c40729b0 0x7f82c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82c40729b0 0x7f82c4071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36188/0 (socket says 172.21.15.158:36188) 2023-10-16T11:17:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 -- 172.21.15.158:0/3580924176 learned_addr learned my addr 172.21.15.158:0/3580924176 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c259c700 1 --2- 172.21.15.158:0/3580924176 >> v2:172.21.15.172:3300/0 conn(0x7f82c406ab40 0x7f82c406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 -- 172.21.15.158:0/3580924176 >> v2:172.21.15.172:3300/0 conn(0x7f82c406ab40 msgr2=0x7f82c406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 --2- 172.21.15.158:0/3580924176 >> v2:172.21.15.172:3300/0 conn(0x7f82c406ab40 0x7f82c406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 -- 172.21.15.158:0/3580924176 >> v1:172.21.15.172:6789/0 conn(0x7f82c406e5d0 legacy=0x7f82c40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 -- 172.21.15.158:0/3580924176 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c41617f0 con 0x7f82c40729b0 2023-10-16T11:17:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c259c700 1 --2- 172.21.15.158:0/3580924176 >> v2:172.21.15.172:3300/0 conn(0x7f82c406ab40 0x7f82c406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c2d9d700 1 --2- 172.21.15.158:0/3580924176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c40729b0 0x7f82c4071910 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f82c4074d70 tx=0x7f82bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff9d15f745cc355 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c159a700 1 -- 172.21.15.158:0/3580924176 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc00e070 con 0x7f82c40729b0 2023-10-16T11:17:31.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c159a700 1 -- 172.21.15.158:0/3580924176 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc00b9c0 con 0x7f82c40729b0 2023-10-16T11:17:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.814+0000 7f82c159a700 1 -- 172.21.15.158:0/3580924176 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc00bd70 con 0x7f82c40729b0 2023-10-16T11:17:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3580924176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c40729b0 msgr2=0x7f82c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3580924176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c40729b0 0x7f82c4071910 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f82c4074d70 tx=0x7f82bc00b330 comp rx=0 tx=0).stop 2023-10-16T11:17:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3580924176 shutdown_connections 2023-10-16T11:17:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3580924176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c40729b0 0x7f82c4071910 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3580924176 >> v2:172.21.15.172:3300/0 conn(0x7f82c406ab40 0x7f82c406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3580924176 >> 172.21.15.158:0/3580924176 conn(0x7f82c40d02b0 msgr2=0x7f82c40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3580924176 shutdown_connections 2023-10-16T11:17:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.815+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3580924176 wait complete. 2023-10-16T11:17:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.816+0000 7f82c8ca3700 1 Processor -- start 2023-10-16T11:17:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.816+0000 7f82c8ca3700 1 -- start start 2023-10-16T11:17:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c8ca3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c406ab40 0x7f82c4154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c8ca3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c406e5d0 0x7f82c40c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c8ca3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82c40729b0 0x7f82c40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c8ca3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40b1050 con 0x7f82c406ab40 2023-10-16T11:17:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c8ca3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40b1330 con 0x7f82c406e5d0 2023-10-16T11:17:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c8ca3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40b1610 con 0x7f82c40729b0 2023-10-16T11:17:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c406ab40 0x7f82c4154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c406ab40 0x7f82c4154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36190/0 (socket says 172.21.15.158:36190) 2023-10-16T11:17:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c259c700 1 -- 172.21.15.158:0/3218862143 learned_addr learned my addr 172.21.15.158:0/3218862143 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c2d9d700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82c40729b0 0x7f82c40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.817+0000 7f82c1d9b700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c406e5d0 0x7f82c40c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c259c700 1 -- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82c40729b0 msgr2=0x7f82c40c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c259c700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82c40729b0 0x7f82c40c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c259c700 1 -- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c406e5d0 msgr2=0x7f82c40c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c259c700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c406e5d0 0x7f82c40c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:31.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c259c700 1 -- 172.21.15.158:0/3218862143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc009b30 con 0x7f82c406ab40 2023-10-16T11:17:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c259c700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c406ab40 0x7f82c4154b00 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f82b400e760 tx=0x7f82b400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82b2ffd700 1 -- 172.21.15.158:0/3218862143 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b400c070 con 0x7f82c406ab40 2023-10-16T11:17:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82b2ffd700 1 -- 172.21.15.158:0/3218862143 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b4013610 con 0x7f82c406ab40 2023-10-16T11:17:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c40b18f0 con 0x7f82c406ab40 2023-10-16T11:17:31.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82b2ffd700 1 -- 172.21.15.158:0/3218862143 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40139c0 con 0x7f82c406ab40 2023-10-16T11:17:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.818+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c40b1eb0 con 0x7f82c406ab40 2023-10-16T11:17:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.819+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c40681b0 con 0x7f82c406ab40 2023-10-16T11:17:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.819+0000 7f82b2ffd700 1 -- 172.21.15.158:0/3218862143 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f82b4013b20 con 0x7f82c406ab40 2023-10-16T11:17:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.820+0000 7f82b2ffd700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f82a406b020 0x7f82a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:31.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.820+0000 7f82b2ffd700 1 -- 172.21.15.158:0/3218862143 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f82b4089920 con 0x7f82c406ab40 2023-10-16T11:17:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.820+0000 7f82c1d9b700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f82a406b020 0x7f82a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.820+0000 7f82c1d9b700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f82a406b020 0x7f82a406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f82b8007fd0 tx=0x7f82b8009040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:31.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:31.824+0000 7f82b2ffd700 1 -- 172.21.15.158:0/3218862143 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82b40543a0 con 0x7f82c406ab40 2023-10-16T11:17:32.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.099+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f82c40c70e0 con 0x7f82c406ab40 2023-10-16T11:17:32.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.627+0000 7f82b2ffd700 1 -- 172.21.15.158:0/3218862143 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3044) v1 ==== 165+0+0 (secure 0 0 0) 0x7f82b40579c0 con 0x7f82c406ab40 2023-10-16T11:17:32.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f82a406b020 msgr2=0x7f82a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f82a406b020 0x7f82a406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f82b8007fd0 tx=0x7f82b8009040 comp rx=0 tx=0).stop 2023-10-16T11:17:32.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c406ab40 msgr2=0x7f82c4154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c406ab40 0x7f82c4154b00 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f82b400e760 tx=0x7f82b400ea70 comp rx=0 tx=0).stop 2023-10-16T11:17:32.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 shutdown_connections 2023-10-16T11:17:32.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82c40729b0 0x7f82c40c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82c406ab40 0x7f82c4154b00 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f82a406b020 0x7f82a406d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 --2- 172.21.15.158:0/3218862143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c406e5d0 0x7f82c40c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 >> 172.21.15.158:0/3218862143 conn(0x7f82c40d02b0 msgr2=0x7f82c406d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:32.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 shutdown_connections 2023-10-16T11:17:32.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.630+0000 7f82c8ca3700 1 -- 172.21.15.158:0/3218862143 wait complete. 2023-10-16T11:17:32.646 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-10-16T11:17:32.647 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-10-16T11:17:32.647 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T11:17:32.648 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:32.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.773+0000 7f32f35bd700 1 Processor -- start 2023-10-16T11:17:32.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.773+0000 7f32f35bd700 1 -- start start 2023-10-16T11:17:32.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.773+0000 7f32f35bd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec15dfb0 0x7f32ec06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:32.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f35bd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32ec06f340 0x7f32ec06f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:32.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f35bd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec06e770 con 0x7f32ec0cc5d0 2023-10-16T11:17:32.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f35bd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32ec06fd30 con 0x7f32ec06f340 2023-10-16T11:17:32.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f35bd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec0cf270 con 0x7f32ec15dfb0 2023-10-16T11:17:32.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f0b58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec15dfb0 0x7f32ec06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:32.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f1b5a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32ec06f340 0x7f32ec06f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f0b58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec15dfb0 0x7f32ec06e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:47034/0 (socket says 172.21.15.158:47034) 2023-10-16T11:17:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f1359700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f32ec0cc5d0 0x7f32ec0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:40278/0 (socket says 172.21.15.158:40278) 2023-10-16T11:17:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f0b58700 1 -- 172.21.15.158:0/2970264536 learned_addr learned my addr 172.21.15.158:0/2970264536 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f0b58700 1 -- 172.21.15.158:0/2970264536 >> v2:172.21.15.172:3300/0 conn(0x7f32ec06f340 msgr2=0x7f32ec06f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f0b58700 1 --2- 172.21.15.158:0/2970264536 >> v2:172.21.15.172:3300/0 conn(0x7f32ec06f340 0x7f32ec06f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f0b58700 1 -- 172.21.15.158:0/2970264536 >> v1:172.21.15.172:6789/0 conn(0x7f32ec0cc5d0 legacy=0x7f32ec0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f0b58700 1 -- 172.21.15.158:0/2970264536 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec0cbe00 con 0x7f32ec15dfb0 2023-10-16T11:17:32.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.774+0000 7f32f1b5a700 1 --2- 172.21.15.158:0/2970264536 >> v2:172.21.15.172:3300/0 conn(0x7f32ec06f340 0x7f32ec06f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:32.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f0b58700 1 --2- 172.21.15.158:0/2970264536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec15dfb0 0x7f32ec06e230 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098f0 tx=0x7f32e8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=369fc3f08b737059 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:32.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32e3fff700 1 -- 172.21.15.158:0/2970264536 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8011830 con 0x7f32ec15dfb0 2023-10-16T11:17:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32e3fff700 1 -- 172.21.15.158:0/2970264536 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e8011990 con 0x7f32ec15dfb0 2023-10-16T11:17:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32e3fff700 1 -- 172.21.15.158:0/2970264536 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8011d20 con 0x7f32ec15dfb0 2023-10-16T11:17:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 -- 172.21.15.158:0/2970264536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec15dfb0 msgr2=0x7f32ec06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 --2- 172.21.15.158:0/2970264536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec15dfb0 0x7f32ec06e230 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098f0 tx=0x7f32e8009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:32.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 -- 172.21.15.158:0/2970264536 shutdown_connections 2023-10-16T11:17:32.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 --2- 172.21.15.158:0/2970264536 >> v2:172.21.15.172:3300/0 conn(0x7f32ec06f340 0x7f32ec06f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 --2- 172.21.15.158:0/2970264536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec15dfb0 0x7f32ec06e230 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 -- 172.21.15.158:0/2970264536 >> 172.21.15.158:0/2970264536 conn(0x7f32ec0d02b0 msgr2=0x7f32ec0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:32.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 -- 172.21.15.158:0/2970264536 shutdown_connections 2023-10-16T11:17:32.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.775+0000 7f32f35bd700 1 -- 172.21.15.158:0/2970264536 wait complete. 2023-10-16T11:17:32.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.776+0000 7f32f35bd700 1 Processor -- start 2023-10-16T11:17:32.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f35bd700 1 -- start start 2023-10-16T11:17:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f35bd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32ec06f340 0x7f32ec1f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f35bd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec0cc5d0 0x7f32ec1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f35bd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32ec15dfb0 0x7f32ec0c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f35bd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1f4f00 con 0x7f32ec06f340 2023-10-16T11:17:32.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f35bd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec0c8a50 con 0x7f32ec15dfb0 2023-10-16T11:17:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f35bd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1f0960 con 0x7f32ec0cc5d0 2023-10-16T11:17:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f1359700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32ec06f340 0x7f32ec1f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f1359700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32ec06f340 0x7f32ec1f49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36204/0 (socket says 172.21.15.158:36204) 2023-10-16T11:17:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f1359700 1 -- 172.21.15.158:0/1070924303 learned_addr learned my addr 172.21.15.158:0/1070924303 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:32.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f0b58700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec0cc5d0 0x7f32ec1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.777+0000 7f32f1b5a700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32ec15dfb0 0x7f32ec0c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f1359700 1 -- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec0cc5d0 msgr2=0x7f32ec1570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f1359700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec0cc5d0 0x7f32ec1570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f1359700 1 -- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32ec15dfb0 msgr2=0x7f32ec0c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f1359700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32ec15dfb0 0x7f32ec0c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f1359700 1 -- 172.21.15.158:0/1070924303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4009a70 con 0x7f32ec06f340 2023-10-16T11:17:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f1359700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32ec06f340 0x7f32ec1f49c0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00c460 tx=0x7f32dc00c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32e1ffb700 1 -- 172.21.15.158:0/1070924303 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc004a70 con 0x7f32ec06f340 2023-10-16T11:17:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32e1ffb700 1 -- 172.21.15.158:0/1070924303 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32dc004bd0 con 0x7f32ec06f340 2023-10-16T11:17:32.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32e1ffb700 1 -- 172.21.15.158:0/1070924303 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc0070d0 con 0x7f32ec06f340 2023-10-16T11:17:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e8009550 con 0x7f32ec06f340 2023-10-16T11:17:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.778+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec0c7ed0 con 0x7f32ec06f340 2023-10-16T11:17:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.779+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ec0681b0 con 0x7f32ec06f340 2023-10-16T11:17:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.779+0000 7f32e1ffb700 1 -- 172.21.15.158:0/1070924303 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f32dc007230 con 0x7f32ec06f340 2023-10-16T11:17:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.780+0000 7f32e1ffb700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f32d006b020 0x7f32d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:32.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.780+0000 7f32e1ffb700 1 -- 172.21.15.158:0/1070924303 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f32dc089130 con 0x7f32ec06f340 2023-10-16T11:17:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.780+0000 7f32f0b58700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f32d006b020 0x7f32d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.780+0000 7f32f0b58700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f32d006b020 0x7f32d006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098f0 tx=0x7f32e801d040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:32.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.784+0000 7f32e1ffb700 1 -- 172.21.15.158:0/1070924303 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32dc053c20 con 0x7f32ec06f340 2023-10-16T11:17:32.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.980+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32ec150d50 con 0x7f32ec06f340 2023-10-16T11:17:32.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.981+0000 7f32e1ffb700 1 -- 172.21.15.158:0/1070924303 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+15426 (secure 0 0 0) 0x7f32dc057240 con 0x7f32ec06f340 2023-10-16T11:17:32.982 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:32.983 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":192,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:30.614366+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6837/773046671":"2023-10-17T11:17:27.231371+0000","172.21.15.158:6835/773046671":"2023-10-17T11:17:27.231371+0000","172.21.15.158:0/995001036":"2023-10-17T11:17:26.192865+0000","172.21.15.158:6833/3817845958":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/3732316088":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/1124225642":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/1870965787":"2023-10-17T11:17:26.192865+0000","172.21.15.158:6832/3817845958":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/2531840430":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2719309194":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2065912370":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/1399927564":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6833/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/129037312":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/2251797820":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/2545298209":"2023-10-17T11:12:22.309907+0000","172.21.15.158:0/549775613":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/827805412":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/2974007506":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/4132457601":"2023-10-17T11:01:31.624108+0000","172.21.15.158:6833/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/276622814":"2023-10-17T11:17:26.192865+0000","172.21.15.158:0/3186172973":"2023-10-17T11:12:22.309907+0000","172.21.15.158:6832/1491011698":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/3804388680":"2023-10-17T11:01:31.624108+0000","172.21.15.158:0/411715851":"2023-10-17T11:01:31.624108+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:32.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f32d006b020 msgr2=0x7f32d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f32d006b020 0x7f32d006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098f0 tx=0x7f32e801d040 comp rx=0 tx=0).stop 2023-10-16T11:17:32.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32ec06f340 msgr2=0x7f32ec1f49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:32.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32ec06f340 0x7f32ec1f49c0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00c460 tx=0x7f32dc00c770 comp rx=0 tx=0).stop 2023-10-16T11:17:32.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 shutdown_connections 2023-10-16T11:17:32.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f32d006b020 0x7f32d006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32ec15dfb0 0x7f32ec0c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32ec06f340 0x7f32ec1f49c0 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 --2- 172.21.15.158:0/1070924303 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32ec0cc5d0 0x7f32ec1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:32.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 >> 172.21.15.158:0/1070924303 conn(0x7f32ec0d02b0 msgr2=0x7f32ec0d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:32.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 shutdown_connections 2023-10-16T11:17:32.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:32.983+0000 7f32f35bd700 1 -- 172.21.15.158:0/1070924303 wait complete. 2023-10-16T11:17:32.997 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:17:32.998 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:17:33.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.120+0000 7fa8a13a4700 1 Processor -- start 2023-10-16T11:17:33.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.120+0000 7fa8a13a4700 1 -- start start 2023-10-16T11:17:33.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa8a13a4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa89c0d29b0 0x7fa89c0d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa8a13a4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d0a70 0x7fa89c0b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa8a13a4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c0b7c90 con 0x7fa89c160d30 2023-10-16T11:17:33.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa8a13a4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa89c0b8910 con 0x7fa89c0d29b0 2023-10-16T11:17:33.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa8a13a4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c0d0f00 con 0x7fa89c0d0a70 2023-10-16T11:17:33.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa89c160d30 0x7fa89c0b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36388/0 (socket says 172.21.15.158:36388) 2023-10-16T11:17:33.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89affd700 1 -- 172.21.15.158:0/2623481358 learned_addr learned my addr 172.21.15.158:0/2623481358 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:33.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89b7fe700 1 --2- 172.21.15.158:0/2623481358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d0a70 0x7fa89c0b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89a7fc700 1 --2- 172.21.15.158:0/2623481358 >> v2:172.21.15.172:3300/0 conn(0x7fa89c0d29b0 0x7fa89c0d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa899ffb700 1 -- 172.21.15.158:0/2623481358 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406702423 0 0) 0x7fa89c0b7c90 con 0x7fa89c160d30 2023-10-16T11:17:33.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa899ffb700 1 -- 172.21.15.158:0/2623481358 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8840035b0 con 0x7fa89c160d30 2023-10-16T11:17:33.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89b7fe700 1 -- 172.21.15.158:0/2623481358 >> v2:172.21.15.172:3300/0 conn(0x7fa89c0d29b0 msgr2=0x7fa89c0d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:33.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89b7fe700 1 --2- 172.21.15.158:0/2623481358 >> v2:172.21.15.172:3300/0 conn(0x7fa89c0d29b0 0x7fa89c0d0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89b7fe700 1 -- 172.21.15.158:0/2623481358 >> v1:172.21.15.158:6789/0 conn(0x7fa89c160d30 legacy=0x7fa89c0b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa899ffb700 1 -- 172.21.15.158:0/2623481358 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1800655155 0 0) 0x7fa8840035b0 con 0x7fa89c160d30 2023-10-16T11:17:33.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89b7fe700 1 -- 172.21.15.158:0/2623481358 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c0b7f70 con 0x7fa89c0d0a70 2023-10-16T11:17:33.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.121+0000 7fa89b7fe700 1 --2- 172.21.15.158:0/2623481358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d0a70 0x7fa89c0b6580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa894009d90 tx=0x7fa89400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9c4af3a98c0ac0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:33.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa899ffb700 1 -- 172.21.15.158:0/2623481358 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89400e070 con 0x7fa89c0d0a70 2023-10-16T11:17:33.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa899ffb700 1 -- 172.21.15.158:0/2623481358 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894003a20 con 0x7fa89c0d0a70 2023-10-16T11:17:33.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa899ffb700 1 -- 172.21.15.158:0/2623481358 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c001670 con 0x7fa89c0d0a70 2023-10-16T11:17:33.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 -- 172.21.15.158:0/2623481358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d0a70 msgr2=0x7fa89c0b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/2623481358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d0a70 0x7fa89c0b6580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa894009d90 tx=0x7fa89400b330 comp rx=0 tx=0).stop 2023-10-16T11:17:33.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 -- 172.21.15.158:0/2623481358 shutdown_connections 2023-10-16T11:17:33.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/2623481358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d0a70 0x7fa89c0b6580 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/2623481358 >> v2:172.21.15.172:3300/0 conn(0x7fa89c0d29b0 0x7fa89c0d0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 -- 172.21.15.158:0/2623481358 >> 172.21.15.158:0/2623481358 conn(0x7fa89c0669f0 msgr2=0x7fa89c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:33.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 -- 172.21.15.158:0/2623481358 shutdown_connections 2023-10-16T11:17:33.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.122+0000 7fa8a13a4700 1 -- 172.21.15.158:0/2623481358 wait complete. 2023-10-16T11:17:33.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 Processor -- start 2023-10-16T11:17:33.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 -- start start 2023-10-16T11:17:33.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa89c0d0a70 0x7fa89c157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d29b0 0x7fa89c0c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa89c160d30 0x7fa89c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c0c8040 con 0x7fa89c160d30 2023-10-16T11:17:33.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1552b0 con 0x7fa89c0d0a70 2023-10-16T11:17:33.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa8a13a4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c155b50 con 0x7fa89c0d29b0 2023-10-16T11:17:33.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa89b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa89c160d30 0x7fa89c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa89b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa89c160d30 0x7fa89c152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36222/0 (socket says 172.21.15.158:36222) 2023-10-16T11:17:33.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.124+0000 7fa89b7fe700 1 -- 172.21.15.158:0/941567728 learned_addr learned my addr 172.21.15.158:0/941567728 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:33.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89a7fc700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d29b0 0x7fa89c0c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89affd700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa89c0d0a70 0x7fa89c157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89b7fe700 1 -- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d29b0 msgr2=0x7fa89c0c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89b7fe700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d29b0 0x7fa89c0c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89b7fe700 1 -- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa89c0d0a70 msgr2=0x7fa89c157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89b7fe700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa89c0d0a70 0x7fa89c157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89b7fe700 1 -- 172.21.15.158:0/941567728 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa894009a70 con 0x7fa89c160d30 2023-10-16T11:17:33.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa89b7fe700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa89c160d30 0x7fa89c152be0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fa89400b680 tx=0x7fa894004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:33.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa8837fe700 1 -- 172.21.15.158:0/941567728 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894004440 con 0x7fa89c160d30 2023-10-16T11:17:33.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa8837fe700 1 -- 172.21.15.158:0/941567728 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8940045e0 con 0x7fa89c160d30 2023-10-16T11:17:33.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c0c8250 con 0x7fa89c160d30 2023-10-16T11:17:33.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa8837fe700 1 -- 172.21.15.158:0/941567728 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894012430 con 0x7fa89c160d30 2023-10-16T11:17:33.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.125+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c1557e0 con 0x7fa89c160d30 2023-10-16T11:17:33.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.126+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c155ce0 con 0x7fa89c160d30 2023-10-16T11:17:33.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.127+0000 7fa8837fe700 1 -- 172.21.15.158:0/941567728 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa894012590 con 0x7fa89c160d30 2023-10-16T11:17:33.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.127+0000 7fa8837fe700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fa88806b020 0x7fa88806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.127+0000 7fa8837fe700 1 -- 172.21.15.158:0/941567728 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7fa89408bc60 con 0x7fa89c160d30 2023-10-16T11:17:33.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.127+0000 7fa89affd700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fa88806b020 0x7fa88806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.127+0000 7fa89affd700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fa88806b020 0x7fa88806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa89c158750 tx=0x7fa88c001da0 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:33.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.131+0000 7fa8837fe700 1 -- 172.21.15.158:0/941567728 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa894056770 con 0x7fa89c160d30 2023-10-16T11:17:33.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.324+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa89c0681b0 con 0x7fa89c160d30 2023-10-16T11:17:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.324+0000 7fa8837fe700 1 -- 172.21.15.158:0/941567728 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa894059d90 con 0x7fa89c160d30 2023-10-16T11:17:33.326 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:17:33.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fa88806b020 msgr2=0x7fa88806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fa88806b020 0x7fa88806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa89c158750 tx=0x7fa88c001da0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa89c160d30 msgr2=0x7fa89c152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa89c160d30 0x7fa89c152be0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fa89400b680 tx=0x7fa894004ad0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 shutdown_connections 2023-10-16T11:17:33.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa89c160d30 0x7fa89c152be0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fa88806b020 0x7fa88806d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa89c0d0a70 0x7fa89c157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 --2- 172.21.15.158:0/941567728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa89c0d29b0 0x7fa89c0c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 >> 172.21.15.158:0/941567728 conn(0x7fa89c0669f0 msgr2=0x7fa89c0b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:33.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 shutdown_connections 2023-10-16T11:17:33.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.326+0000 7fa8a13a4700 1 -- 172.21.15.158:0/941567728 wait complete. 2023-10-16T11:17:33.340 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T11:17:33.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.460+0000 7f93a97ec700 1 Processor -- start 2023-10-16T11:17:33.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.460+0000 7f93a97ec700 1 -- start start 2023-10-16T11:17:33.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.460+0000 7f93a97ec700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a415b4a0 0x7f93a40cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.460+0000 7f93a97ec700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93a40b1310 0x7f93a40c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.460+0000 7f93a97ec700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a415fab0 con 0x7f93a40cb2c0 2023-10-16T11:17:33.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.460+0000 7f93a97ec700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a415efa0 con 0x7f93a40b1310 2023-10-16T11:17:33.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.460+0000 7f93a97ec700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a415b880 con 0x7f93a415b4a0 2023-10-16T11:17:33.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a415b4a0 0x7f93a40cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a37fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93a40b1310 0x7f93a40c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a415b4a0 0x7f93a40cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:47060/0 (socket says 172.21.15.158:47060) 2023-10-16T11:17:33.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 -- 172.21.15.158:0/868584164 learned_addr learned my addr 172.21.15.158:0/868584164 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:33.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a27fc700 1 --1- 172.21.15.158:0/868584164 >> v1:172.21.15.172:6789/0 conn(0x7f93a40cb2c0 0x7f93a40b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:40306/0 (socket says 172.21.15.158:40306) 2023-10-16T11:17:33.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 -- 172.21.15.158:0/868584164 >> v2:172.21.15.172:3300/0 conn(0x7f93a40b1310 msgr2=0x7f93a40c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:33.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 --2- 172.21.15.158:0/868584164 >> v2:172.21.15.172:3300/0 conn(0x7f93a40b1310 0x7f93a40c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 -- 172.21.15.158:0/868584164 >> v1:172.21.15.172:6789/0 conn(0x7f93a40cb2c0 legacy=0x7f93a40b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a1ffb700 1 -- 172.21.15.158:0/868584164 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590920691 0 0) 0x7f93a415fab0 con 0x7f93a40cb2c0 2023-10-16T11:17:33.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 -- 172.21.15.158:0/868584164 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a415f180 con 0x7f93a415b4a0 2023-10-16T11:17:33.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a2ffd700 1 --2- 172.21.15.158:0/868584164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a415b4a0 0x7f93a40cfcc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f93980098f0 tx=0x7f9398009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9c617fbf18bdfe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:33.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.461+0000 7f93a1ffb700 1 -- 172.21.15.158:0/868584164 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398011830 con 0x7f93a415b4a0 2023-10-16T11:17:33.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a1ffb700 1 -- 172.21.15.158:0/868584164 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9398011990 con 0x7f93a415b4a0 2023-10-16T11:17:33.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a1ffb700 1 -- 172.21.15.158:0/868584164 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398011d20 con 0x7f93a415b4a0 2023-10-16T11:17:33.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 -- 172.21.15.158:0/868584164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a415b4a0 msgr2=0x7f93a40cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 --2- 172.21.15.158:0/868584164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a415b4a0 0x7f93a40cfcc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f93980098f0 tx=0x7f9398009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:33.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 -- 172.21.15.158:0/868584164 shutdown_connections 2023-10-16T11:17:33.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 --2- 172.21.15.158:0/868584164 >> v2:172.21.15.172:3300/0 conn(0x7f93a40b1310 0x7f93a40c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 --2- 172.21.15.158:0/868584164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a415b4a0 0x7f93a40cfcc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 -- 172.21.15.158:0/868584164 >> 172.21.15.158:0/868584164 conn(0x7f93a406af00 msgr2=0x7f93a40cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:33.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 -- 172.21.15.158:0/868584164 shutdown_connections 2023-10-16T11:17:33.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.462+0000 7f93a97ec700 1 -- 172.21.15.158:0/868584164 wait complete. 2023-10-16T11:17:33.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.463+0000 7f93a97ec700 1 Processor -- start 2023-10-16T11:17:33.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.463+0000 7f93a97ec700 1 -- start start 2023-10-16T11:17:33.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a97ec700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a40b1310 0x7f93a40addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a97ec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93a40cb2c0 0x7f93a41f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a97ec700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93a415b4a0 0x7f93a40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a97ec700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a41f2ae0 con 0x7f93a40cb2c0 2023-10-16T11:17:33.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a97ec700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40acdd0 con 0x7f93a415b4a0 2023-10-16T11:17:33.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a97ec700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40ad0a0 con 0x7f93a40b1310 2023-10-16T11:17:33.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93a40cb2c0 0x7f93a41f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93a40cb2c0 0x7f93a41f25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36232/0 (socket says 172.21.15.158:36232) 2023-10-16T11:17:33.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a27fc700 1 -- 172.21.15.158:0/675806602 learned_addr learned my addr 172.21.15.158:0/675806602 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:33.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a2ffd700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a40b1310 0x7f93a40addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a37fe700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93a415b4a0 0x7f93a40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a27fc700 1 -- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a40b1310 msgr2=0x7f93a40addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.464+0000 7f93a27fc700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a40b1310 0x7f93a40addb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f93a27fc700 1 -- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93a415b4a0 msgr2=0x7f93a40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:33.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f93a27fc700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93a415b4a0 0x7f93a40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:33.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f93a27fc700 1 -- 172.21.15.158:0/675806602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398009550 con 0x7f93a40cb2c0 2023-10-16T11:17:33.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f93a37fe700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93a415b4a0 0x7f93a40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:33.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f93a27fc700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93a40cb2c0 0x7f93a41f25a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f93940011e0 tx=0x7f939400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:33.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f938b7fe700 1 -- 172.21.15.158:0/675806602 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394004c80 con 0x7f93a40cb2c0 2023-10-16T11:17:33.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f938b7fe700 1 -- 172.21.15.158:0/675806602 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9394004de0 con 0x7f93a40cb2c0 2023-10-16T11:17:33.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a40ad380 con 0x7f93a40cb2c0 2023-10-16T11:17:33.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f938b7fe700 1 -- 172.21.15.158:0/675806602 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394007310 con 0x7f93a40cb2c0 2023-10-16T11:17:33.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.465+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a40c7cc0 con 0x7f93a40cb2c0 2023-10-16T11:17:33.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.466+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a40c7b20 con 0x7f93a40cb2c0 2023-10-16T11:17:33.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.466+0000 7f938b7fe700 1 -- 172.21.15.158:0/675806602 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9394007470 con 0x7f93a40cb2c0 2023-10-16T11:17:33.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.467+0000 7f938b7fe700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f939006b020 0x7f939006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:33.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.467+0000 7f938b7fe700 1 -- 172.21.15.158:0/675806602 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f9394089860 con 0x7f93a40cb2c0 2023-10-16T11:17:33.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.467+0000 7f93a2ffd700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f939006b020 0x7f939006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:33.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.467+0000 7f93a2ffd700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f939006b020 0x7f939006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9398009c40 tx=0x7f939801d040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:33.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.471+0000 7f938b7fe700 1 -- 172.21.15.158:0/675806602 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93a40c7b20 con 0x7f93a40cb2c0 2023-10-16T11:17:33.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:33.741+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f93a40c7b20 con 0x7f93a40cb2c0 2023-10-16T11:17:34.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.632+0000 7f938b7fe700 1 -- 172.21.15.158:0/675806602 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3046) v1 ==== 166+0+0 (secure 0 0 0) 0x7f93a40c7b20 con 0x7f93a40cb2c0 2023-10-16T11:17:34.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.635+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f939006b020 msgr2=0x7f939006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:34.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.635+0000 7f93a97ec700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f939006b020 0x7f939006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9398009c40 tx=0x7f939801d040 comp rx=0 tx=0).stop 2023-10-16T11:17:34.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.635+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93a40cb2c0 msgr2=0x7f93a41f25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:34.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93a40cb2c0 0x7f93a41f25a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f93940011e0 tx=0x7f939400ca30 comp rx=0 tx=0).stop 2023-10-16T11:17:34.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 shutdown_connections 2023-10-16T11:17:34.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f939006b020 0x7f939006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93a415b4a0 0x7f93a40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93a40b1310 0x7f93a40addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 --2- 172.21.15.158:0/675806602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f93a40cb2c0 0x7f93a41f25a0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 >> 172.21.15.158:0/675806602 conn(0x7f93a406af00 msgr2=0x7f93a415a840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:34.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 shutdown_connections 2023-10-16T11:17:34.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.636+0000 7f93a97ec700 1 -- 172.21.15.158:0/675806602 wait complete. 2023-10-16T11:17:34.652 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T11:17:34.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.780+0000 7fc7fe59d700 1 Processor -- start 2023-10-16T11:17:34.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.780+0000 7fc7fe59d700 1 -- start start 2023-10-16T11:17:34.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fe59d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc7f815da40 0x7fc7f80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:34.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fe59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f80cc590 0x7fc7f806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:34.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fe59d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7f806ae10 0x7fc7f80ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:34.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fe59d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f806b2c0 con 0x7fc7f806ae10 2023-10-16T11:17:34.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fe59d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80cbd30 con 0x7fc7f815da40 2023-10-16T11:17:34.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fe59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80cc1c0 con 0x7fc7f80cc590 2023-10-16T11:17:34.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7f806ae10 0x7fc7f80ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:34.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7f806ae10 0x7fc7f80ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36244/0 (socket says 172.21.15.158:36244) 2023-10-16T11:17:34.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 -- 172.21.15.158:0/1292272701 learned_addr learned my addr 172.21.15.158:0/1292272701 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:34.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7f77fe700 1 --2- 172.21.15.158:0/1292272701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f80cc590 0x7fc7f806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:34.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7f7fff700 1 --2- 172.21.15.158:0/1292272701 >> v2:172.21.15.172:3300/0 conn(0x7fc7f815da40 0x7fc7f80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:34.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 -- 172.21.15.158:0/1292272701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f80cc590 msgr2=0x7fc7f806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:34.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 --2- 172.21.15.158:0/1292272701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f80cc590 0x7fc7f806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 -- 172.21.15.158:0/1292272701 >> v2:172.21.15.172:3300/0 conn(0x7fc7f815da40 msgr2=0x7fc7f80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:34.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 --2- 172.21.15.158:0/1292272701 >> v2:172.21.15.172:3300/0 conn(0x7fc7f815da40 0x7fc7f80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.781+0000 7fc7fcb3a700 1 -- 172.21.15.158:0/1292272701 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f815de20 con 0x7fc7f806ae10 2023-10-16T11:17:34.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fcb3a700 1 --2- 172.21.15.158:0/1292272701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ccd60 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806c9c0 tx=0x7fc7f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f327ee3c38779e8b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:34.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7f6ffd700 1 -- 172.21.15.158:0/1292272701 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f000e070 con 0x7fc7f806ae10 2023-10-16T11:17:34.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7f6ffd700 1 -- 172.21.15.158:0/1292272701 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f000b9c0 con 0x7fc7f806ae10 2023-10-16T11:17:34.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7f6ffd700 1 -- 172.21.15.158:0/1292272701 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f000bd70 con 0x7fc7f806ae10 2023-10-16T11:17:34.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 -- 172.21.15.158:0/1292272701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 msgr2=0x7fc7f80ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:34.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/1292272701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ccd60 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806c9c0 tx=0x7fc7f000b330 comp rx=0 tx=0).stop 2023-10-16T11:17:34.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 -- 172.21.15.158:0/1292272701 shutdown_connections 2023-10-16T11:17:34.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/1292272701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ccd60 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/1292272701 >> v2:172.21.15.172:3300/0 conn(0x7fc7f815da40 0x7fc7f80caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/1292272701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f80cc590 0x7fc7f806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 -- 172.21.15.158:0/1292272701 >> 172.21.15.158:0/1292272701 conn(0x7fc7f8099a10 msgr2=0x7fc7f80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:34.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 -- 172.21.15.158:0/1292272701 shutdown_connections 2023-10-16T11:17:34.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.782+0000 7fc7fe59d700 1 -- 172.21.15.158:0/1292272701 wait complete. 2023-10-16T11:17:34.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.783+0000 7fc7fe59d700 1 Processor -- start 2023-10-16T11:17:34.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7fe59d700 1 -- start start 2023-10-16T11:17:34.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7fe59d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:34.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7fe59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f80cc590 0x7fc7f80c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:34.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7fe59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f815da40 0x7fc7f80c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:34.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7fe59d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80b4780 con 0x7fc7f806ae10 2023-10-16T11:17:34.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7fe59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80c6cd0 con 0x7fc7f80cc590 2023-10-16T11:17:34.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7fe59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80c48e0 con 0x7fc7f815da40 2023-10-16T11:17:34.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.784+0000 7fc7f7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:34.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36246/0 (socket says 172.21.15.158:36246) 2023-10-16T11:17:34.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 -- 172.21.15.158:0/3918994945 learned_addr learned my addr 172.21.15.158:0/3918994945 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:34.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7fcb3a700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f815da40 0x7fc7f80c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:34.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f77fe700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f80cc590 0x7fc7f80c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:34.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 -- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f815da40 msgr2=0x7fc7f80c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:34.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f815da40 0x7fc7f80c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 -- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f80cc590 msgr2=0x7fc7f80c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f80cc590 0x7fc7f80c6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 -- 172.21.15.158:0/3918994945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0009b30 con 0x7fc7f806ae10 2023-10-16T11:17:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f7fff700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ca2d0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00d800 tx=0x7fc7ec00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:34.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f4ff9700 1 -- 172.21.15.158:0/3918994945 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec0125f0 con 0x7fc7f806ae10 2023-10-16T11:17:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f4ff9700 1 -- 172.21.15.158:0/3918994945 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ec012750 con 0x7fc7f806ae10 2023-10-16T11:17:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f81f3ae0 con 0x7fc7f806ae10 2023-10-16T11:17:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7f4ff9700 1 -- 172.21.15.158:0/3918994945 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec012b00 con 0x7fc7f806ae10 2023-10-16T11:17:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.785+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f81f3dc0 con 0x7fc7f806ae10 2023-10-16T11:17:34.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.786+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f80b4a20 con 0x7fc7f806ae10 2023-10-16T11:17:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.787+0000 7fc7f4ff9700 1 -- 172.21.15.158:0/3918994945 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc7ec012c60 con 0x7fc7f806ae10 2023-10-16T11:17:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.787+0000 7fc7f4ff9700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fc7e406b020 0x7fc7e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.787+0000 7fc7f4ff9700 1 -- 172.21.15.158:0/3918994945 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7fc7ec089de0 con 0x7fc7f806ae10 2023-10-16T11:17:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.787+0000 7fc7f77fe700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fc7e406b020 0x7fc7e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.787+0000 7fc7f77fe700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fc7e406b020 0x7fc7e406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b4470 tx=0x7fc7e8008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:34.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.792+0000 7fc7f4ff9700 1 -- 172.21.15.158:0/3918994945 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7ec054a00 con 0x7fc7f806ae10 2023-10-16T11:17:34.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.984+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc7f80cbac0 con 0x7fc7f806ae10 2023-10-16T11:17:34.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.987+0000 7fc7f4ff9700 1 -- 172.21.15.158:0/3918994945 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc7ec058020 con 0x7fc7f806ae10 2023-10-16T11:17:34.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fc7e406b020 msgr2=0x7fc7e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:34.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fc7e406b020 0x7fc7e406d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b4470 tx=0x7fc7e8008040 comp rx=0 tx=0).stop 2023-10-16T11:17:34.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 msgr2=0x7fc7f80ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:34.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ca2d0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00d800 tx=0x7fc7ec00db10 comp rx=0 tx=0).stop 2023-10-16T11:17:34.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 shutdown_connections 2023-10-16T11:17:34.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7f815da40 0x7fc7f80c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7f806ae10 0x7fc7f80ca2d0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fc7e406b020 0x7fc7e406d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 --2- 172.21.15.158:0/3918994945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f80cc590 0x7fc7f80c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 >> 172.21.15.158:0/3918994945 conn(0x7fc7f8099a10 msgr2=0x7fc7f80b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 shutdown_connections 2023-10-16T11:17:34.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:34.989+0000 7fc7fe59d700 1 -- 172.21.15.158:0/3918994945 wait complete. 2023-10-16T11:17:35.003 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:17:35.003 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:17:35.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:17:35.087 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:17:35.088 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:17:35.088 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:35.105 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:17:35.106 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:17:35.106 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:35.125 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:17:35.126 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:17:35.126 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:35.162 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:17:35.162 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:17:35.163 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:35.184 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:17:35.185 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T11:17:35.185 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:17:35.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.321+0000 7ffb8b9f8700 1 Processor -- start 2023-10-16T11:17:35.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.321+0000 7ffb8b9f8700 1 -- start start 2023-10-16T11:17:35.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb8b9f8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 0x7ffb8415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb8b9f8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8415c5d0 con 0x7ffb840ce2c0 2023-10-16T11:17:35.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb8b9f8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8415c8b0 con 0x7ffb840b6e10 2023-10-16T11:17:35.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb8b9f8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8415d2c0 con 0x7ffb840cdee0 2023-10-16T11:17:35.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb89f95700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffb840ce2c0 0x7ffb840cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36434/0 (socket says 172.21.15.158:36434) 2023-10-16T11:17:35.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb89f95700 1 -- 172.21.15.158:0/3258520231 learned_addr learned my addr 172.21.15.158:0/3258520231 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:35.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb89794700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 0x7ffb8415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982200011 0 0) 0x7ffb8415c5d0 con 0x7ffb840ce2c0 2023-10-16T11:17:35.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb700035b0 con 0x7ffb840ce2c0 2023-10-16T11:17:35.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201739972 0 0) 0x7ffb700035b0 con 0x7ffb840ce2c0 2023-10-16T11:17:35.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 msgr2=0x7ffb8415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:35.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 0x7ffb8415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 >> v1:172.21.15.172:6789/0 conn(0x7ffb840b6e10 legacy=0x7ffb841521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8415cb90 con 0x7ffb840ce2c0 2023-10-16T11:17:35.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ffb7c0019f0 con 0x7ffb840ce2c0 2023-10-16T11:17:35.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 >> v1:172.21.15.158:6789/0 conn(0x7ffb840ce2c0 legacy=0x7ffb840cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 0x7ffb70003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.322+0000 7ffb7bfff700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb70003f70 0x7ffb70004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb7bfff700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb70004d90 0x7ffb70005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8415d2c0 con 0x7ffb70004d90 2023-10-16T11:17:35.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7000bb20 con 0x7ffb70003f70 2023-10-16T11:17:35.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7000bce0 con 0x7ffb840cdee0 2023-10-16T11:17:35.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89f95700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb70004d90 0x7ffb70005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89794700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 0x7ffb70003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb88f93700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb70003f70 0x7ffb70004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89f95700 1 -- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 msgr2=0x7ffb70003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:35.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89f95700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb840cdee0 0x7ffb70003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89f95700 1 -- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb70003f70 msgr2=0x7ffb70004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:35.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89f95700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb70003f70 0x7ffb70004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89f95700 1 -- 172.21.15.158:0/3258520231 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700035b0 con 0x7ffb70004d90 2023-10-16T11:17:35.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb89f95700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb70004d90 0x7ffb70005210 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00a830 tx=0x7ffb70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289c24b1a3ea1cc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:35.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c012600 con 0x7ffb70004d90 2023-10-16T11:17:35.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c012780 con 0x7ffb70004d90 2023-10-16T11:17:35.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.323+0000 7ffb7bfff700 1 -- 172.21.15.158:0/3258520231 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c012b30 con 0x7ffb70004d90 2023-10-16T11:17:35.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.324+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3258520231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb70004d90 msgr2=0x7ffb70005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.324+0000 7ffb8b9f8700 1 --2- 172.21.15.158:0/3258520231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb70004d90 0x7ffb70005210 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00a830 tx=0x7ffb70005880 comp rx=0 tx=0).stop 2023-10-16T11:17:35.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.324+0000 7ffb89f95700 1 -- 172.21.15.158:0/3258520231 reap_dead start 2023-10-16T11:17:35.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.324+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3258520231 shutdown_connections 2023-10-16T11:17:35.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.324+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3258520231 >> 172.21.15.158:0/3258520231 conn(0x7ffb8406a8d0 msgr2=0x7ffb8406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:35.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.324+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3258520231 shutdown_connections 2023-10-16T11:17:35.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.324+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3258520231 wait complete. 2023-10-16T11:17:35.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.325+0000 7ffb8b9f8700 1 Processor -- start 2023-10-16T11:17:35.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.325+0000 7ffb8b9f8700 1 -- start start 2023-10-16T11:17:35.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb8b9f8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb70004d90 0x7ffb8414a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb8b9f8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb8406c660 0x7ffb840c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb8b9f8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb840c73d0 0x7ffb841547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb8b9f8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb840c7bf0 con 0x7ffb8406c660 2023-10-16T11:17:35.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb8b9f8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb840c7910 con 0x7ffb840c73d0 2023-10-16T11:17:35.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb8b9f8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8406bb80 con 0x7ffb70004d90 2023-10-16T11:17:35.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb8406c660 0x7ffb840c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb8406c660 0x7ffb840c6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36268/0 (socket says 172.21.15.158:36268) 2023-10-16T11:17:35.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 -- 172.21.15.158:0/3773071085 learned_addr learned my addr 172.21.15.158:0/3773071085 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:35.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb89794700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb70004d90 0x7ffb8414a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb89f95700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb840c73d0 0x7ffb841547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 -- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb70004d90 msgr2=0x7ffb8414a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb70004d90 0x7ffb8414a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 -- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb840c73d0 msgr2=0x7ffb841547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb840c73d0 0x7ffb841547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 -- 172.21.15.158:0/3773071085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8406be60 con 0x7ffb8406c660 2023-10-16T11:17:35.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb88f93700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb8406c660 0x7ffb840c6e90 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400f860 tx=0x7ffb7400fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:35.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb79ffb700 1 -- 172.21.15.158:0/3773071085 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb740052e0 con 0x7ffb8406c660 2023-10-16T11:17:35.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb79ffb700 1 -- 172.21.15.158:0/3773071085 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb74005460 con 0x7ffb8406c660 2023-10-16T11:17:35.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb79ffb700 1 -- 172.21.15.158:0/3773071085 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb740038c0 con 0x7ffb8406c660 2023-10-16T11:17:35.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.326+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8406c140 con 0x7ffb8406c660 2023-10-16T11:17:35.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.327+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8414a860 con 0x7ffb8406c660 2023-10-16T11:17:35.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.328+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8406b9e0 con 0x7ffb8406c660 2023-10-16T11:17:35.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.328+0000 7ffb79ffb700 1 -- 172.21.15.158:0/3773071085 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffb74003a20 con 0x7ffb8406c660 2023-10-16T11:17:35.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.328+0000 7ffb79ffb700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7ffb6806b020 0x7ffb6806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.328+0000 7ffb89794700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7ffb6806b020 0x7ffb6806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.328+0000 7ffb79ffb700 1 -- 172.21.15.158:0/3773071085 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7ffb740899b0 con 0x7ffb8406c660 2023-10-16T11:17:35.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.329+0000 7ffb89794700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7ffb6806b020 0x7ffb6806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009c60 tx=0x7ffb80009210 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:35.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.333+0000 7ffb79ffb700 1 -- 172.21.15.158:0/3773071085 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb8406b9e0 con 0x7ffb8406c660 2023-10-16T11:17:35.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.569+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb8406b9e0 con 0x7ffb8406c660 2023-10-16T11:17:35.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.569+0000 7ffb79ffb700 1 -- 172.21.15.158:0/3773071085 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3713 (secure 0 0 0) 0x7ffb8406b9e0 con 0x7ffb8406c660 2023-10-16T11:17:35.571 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:35.572 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3580267390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3580267390},{"type":"v1","addr":"172.21.15.172:6834","nonce":3580267390}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":9328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3433152480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3433152480},{"type":"v1","addr":"172.21.15.172:6835","nonce":3433152480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1170239465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1170239465},{"type":"v1","addr":"172.21.15.158:6836","nonce":1170239465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89},{"gid":10965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/4174815672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":4174815672},{"type":"v1","addr":"172.21.15.158:6837","nonce":4174815672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[]} 2023-10-16T11:17:35.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.571+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7ffb6806b020 msgr2=0x7ffb6806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.571+0000 7ffb8b9f8700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7ffb6806b020 0x7ffb6806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009c60 tx=0x7ffb80009210 comp rx=0 tx=0).stop 2023-10-16T11:17:35.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb8406c660 msgr2=0x7ffb840c6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb8406c660 0x7ffb840c6e90 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400f860 tx=0x7ffb7400fb70 comp rx=0 tx=0).stop 2023-10-16T11:17:35.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 shutdown_connections 2023-10-16T11:17:35.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7ffb6806b020 0x7ffb6806d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb840c73d0 0x7ffb841547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb70004d90 0x7ffb8414a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 --2- 172.21.15.158:0/3773071085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb8406c660 0x7ffb840c6e90 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 >> 172.21.15.158:0/3773071085 conn(0x7ffb8406a8d0 msgr2=0x7ffb840b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:35.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 shutdown_connections 2023-10-16T11:17:35.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.572+0000 7ffb8b9f8700 1 -- 172.21.15.158:0/3773071085 wait complete. 2023-10-16T11:17:35.577 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 96 2023-10-16T11:17:35.586 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T11:17:35.586 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T11:17:35.586 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:17:35.587 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:17:35.587 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T11:17:35.587 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T11:17:35.587 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:17:35.588 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:17:35.588 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T11:17:35.588 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T11:17:35.589 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:17:35.589 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:17:35.589 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T11:17:35.589 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T11:17:35.590 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:17:35.590 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:17:35.595 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:17:35.595 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T11:17:35.597 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:17:35.597 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T11:17:35.600 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:17:35.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T11:17:35.602 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:17:35.602 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T11:17:35.639 INFO:tasks.ceph.mds.a:Started 2023-10-16T11:17:35.639 DEBUG:teuthology.parallel:result is None 2023-10-16T11:17:35.640 INFO:tasks.ceph.mds.c:Started 2023-10-16T11:17:35.641 DEBUG:teuthology.parallel:result is None 2023-10-16T11:17:35.643 INFO:tasks.ceph.mds.b:Started 2023-10-16T11:17:35.643 DEBUG:teuthology.parallel:result is None 2023-10-16T11:17:35.644 INFO:tasks.ceph.mds.d:Started 2023-10-16T11:17:35.644 DEBUG:teuthology.parallel:result is None 2023-10-16T11:17:35.645 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T11:17:35.719 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T11:17:35.719 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:17:35.730 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:17:35.731 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:17:35.731 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T11:17:35.729+0000 7fc58b3d3700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-10-16T11:17:35.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.771+0000 7f2f0be39700 1 Processor -- start 2023-10-16T11:17:35.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.771+0000 7f2f0be39700 1 -- start start 2023-10-16T11:17:35.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0be39700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f04161a00 0x7f2f040cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0be39700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2f0406a8d0 0x7f2f0406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0be39700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0406b260 con 0x7f2f04161de0 2023-10-16T11:17:35.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0be39700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0415e460 con 0x7f2f0406a8d0 2023-10-16T11:17:35.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0be39700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f040ce570 con 0x7f2f04161a00 2023-10-16T11:17:35.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0a3d6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2f0406a8d0 0x7f2f0406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f093d4700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2f04161de0 0x7f2f0415e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36476/0 (socket says 172.21.15.158:36476) 2023-10-16T11:17:35.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f093d4700 1 -- 172.21.15.158:0/3046644685 learned_addr learned my addr 172.21.15.158:0/3046644685 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:35.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f09bd5700 1 --2- 172.21.15.158:0/3046644685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f04161a00 0x7f2f040cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0a3d6700 1 -- 172.21.15.158:0/3046644685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f04161a00 msgr2=0x7f2f040cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:35.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0a3d6700 1 --2- 172.21.15.158:0/3046644685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f04161a00 0x7f2f040cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0a3d6700 1 -- 172.21.15.158:0/3046644685 >> v1:172.21.15.158:6789/0 conn(0x7f2f04161de0 legacy=0x7f2f0415e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0a3d6700 1 -- 172.21.15.158:0/3046644685 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f040ce850 con 0x7f2f0406a8d0 2023-10-16T11:17:35.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f08bd3700 1 -- 172.21.15.158:0/3046644685 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795377990 0 0) 0x7f2f00001110 con 0x7f2f04161de0 2023-10-16T11:17:35.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.772+0000 7f2f0a3d6700 1 --2- 172.21.15.158:0/3046644685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f0406a8d0 0x7f2f0406ad20 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009d90 tx=0x7f2efc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ed1e849929fc7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:35.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f08bd3700 1 -- 172.21.15.158:0/3046644685 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00e070 con 0x7f2f0406a8d0 2023-10-16T11:17:35.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f08bd3700 1 -- 172.21.15.158:0/3046644685 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc00bb20 con 0x7f2f0406a8d0 2023-10-16T11:17:35.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f08bd3700 1 -- 172.21.15.158:0/3046644685 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc012430 con 0x7f2f0406a8d0 2023-10-16T11:17:35.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 -- 172.21.15.158:0/3046644685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f0406a8d0 msgr2=0x7f2f0406ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3046644685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f0406a8d0 0x7f2f0406ad20 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009d90 tx=0x7f2efc00b330 comp rx=0 tx=0).stop 2023-10-16T11:17:35.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 -- 172.21.15.158:0/3046644685 shutdown_connections 2023-10-16T11:17:35.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3046644685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f0406a8d0 0x7f2f0406ad20 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3046644685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f04161a00 0x7f2f040cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 -- 172.21.15.158:0/3046644685 >> 172.21.15.158:0/3046644685 conn(0x7f2f040cce40 msgr2=0x7f2f040cd250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:35.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 -- 172.21.15.158:0/3046644685 shutdown_connections 2023-10-16T11:17:35.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.773+0000 7f2f0be39700 1 -- 172.21.15.158:0/3046644685 wait complete. 2023-10-16T11:17:35.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.774+0000 7f2f0be39700 1 Processor -- start 2023-10-16T11:17:35.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.774+0000 7f2f0be39700 1 -- start start 2023-10-16T11:17:35.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0be39700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f0406a8d0 0x7f2f041e9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0be39700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f04161a00 0x7f2f040b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0be39700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f04161de0 0x7f2f040b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0be39700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f040acec0 con 0x7f2f04161de0 2023-10-16T11:17:35.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0be39700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04155eb0 con 0x7f2f04161a00 2023-10-16T11:17:35.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0be39700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04154fe0 con 0x7f2f0406a8d0 2023-10-16T11:17:35.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0a3d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f04161de0 0x7f2f040b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0a3d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f04161de0 0x7f2f040b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36304/0 (socket says 172.21.15.158:36304) 2023-10-16T11:17:35.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f0a3d6700 1 -- 172.21.15.158:0/3342331845 learned_addr learned my addr 172.21.15.158:0/3342331845 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:35.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f093d4700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f04161a00 0x7f2f040b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.775+0000 7f2f09bd5700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f0406a8d0 0x7f2f041e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0a3d6700 1 -- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f0406a8d0 msgr2=0x7f2f041e9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0a3d6700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f0406a8d0 0x7f2f041e9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0a3d6700 1 -- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f04161a00 msgr2=0x7f2f040b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:35.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0a3d6700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f04161a00 0x7f2f040b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:35.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0a3d6700 1 -- 172.21.15.158:0/3342331845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc009a70 con 0x7f2f04161de0 2023-10-16T11:17:35.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0a3d6700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f04161de0 0x7f2f040b5270 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f2efc014040 tx=0x7f2efc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:35.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2efa7fc700 1 -- 172.21.15.158:0/3342331845 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc012430 con 0x7f2f04161de0 2023-10-16T11:17:35.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2efa7fc700 1 -- 172.21.15.158:0/3342331845 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc020070 con 0x7f2f04161de0 2023-10-16T11:17:35.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f04155230 con 0x7f2f04161de0 2023-10-16T11:17:35.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2efa7fc700 1 -- 172.21.15.158:0/3342331845 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc004280 con 0x7f2f04161de0 2023-10-16T11:17:35.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.776+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f041557f0 con 0x7f2f04161de0 2023-10-16T11:17:35.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.777+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f040681b0 con 0x7f2f04161de0 2023-10-16T11:17:35.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.777+0000 7f2efa7fc700 1 -- 172.21.15.158:0/3342331845 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2efc0043e0 con 0x7f2f04161de0 2023-10-16T11:17:35.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.778+0000 7f2efa7fc700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2ee806b020 0x7f2ee806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:35.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.778+0000 7f2f09bd5700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2ee806b020 0x7f2ee806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:35.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.778+0000 7f2efa7fc700 1 -- 172.21.15.158:0/3342331845 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f2efc022070 con 0x7f2f04161de0 2023-10-16T11:17:35.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.778+0000 7f2f09bd5700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2ee806b020 0x7f2ee806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009b40 tx=0x7f2ef4009210 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:35.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.782+0000 7f2efa7fc700 1 -- 172.21.15.158:0/3342331845 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2efc0566c0 con 0x7f2f04161de0 2023-10-16T11:17:35.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:35.994+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2f0415b660 con 0x7f2f04161de0 2023-10-16T11:17:36.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.648+0000 7f2efa7fc700 1 -- 172.21.15.158:0/3342331845 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v194) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2efc059ce0 con 0x7f2f04161de0 2023-10-16T11:17:36.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.651+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2ee806b020 msgr2=0x7f2ee806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:36.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.651+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2ee806b020 0x7f2ee806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009b40 tx=0x7f2ef4009210 comp rx=0 tx=0).stop 2023-10-16T11:17:36.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.651+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f04161de0 msgr2=0x7f2f040b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:36.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.651+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f04161de0 0x7f2f040b5270 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f2efc014040 tx=0x7f2efc004ad0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 shutdown_connections 2023-10-16T11:17:36.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f04161de0 0x7f2f040b5270 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f0406a8d0 0x7f2f041e9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2ee806b020 0x7f2ee806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 --2- 172.21.15.158:0/3342331845 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f04161a00 0x7f2f040b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 >> 172.21.15.158:0/3342331845 conn(0x7f2f040cce40 msgr2=0x7f2f0406e220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 shutdown_connections 2023-10-16T11:17:36.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.652+0000 7f2f0be39700 1 -- 172.21.15.158:0/3342331845 wait complete. 2023-10-16T11:17:36.656 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T11:17:36.668 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T11:17:36.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.788+0000 7fee101ac700 1 Processor -- start 2023-10-16T11:17:36.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.788+0000 7fee101ac700 1 -- start start 2023-10-16T11:17:36.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee101ac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee080cc5d0 0x7fee0806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee101ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee080cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee101ac700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0806e770 con 0x7fee0806f340 2023-10-16T11:17:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee101ac700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080cf270 con 0x7fee080cc5d0 2023-10-16T11:17:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee101ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0815e4e0 con 0x7fee0815dfb0 2023-10-16T11:17:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee0d747700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee080cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee0df48700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee080cc5d0 0x7fee0806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee0d747700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee080cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:47146/0 (socket says 172.21.15.158:47146) 2023-10-16T11:17:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.789+0000 7fee0d747700 1 -- 172.21.15.158:0/3291128714 learned_addr learned my addr 172.21.15.158:0/3291128714 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0d747700 1 -- 172.21.15.158:0/3291128714 >> v2:172.21.15.172:3300/0 conn(0x7fee080cc5d0 msgr2=0x7fee0806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:36.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0d747700 1 --2- 172.21.15.158:0/3291128714 >> v2:172.21.15.172:3300/0 conn(0x7fee080cc5d0 0x7fee0806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0d747700 1 -- 172.21.15.158:0/3291128714 >> v1:172.21.15.172:6789/0 conn(0x7fee0806f340 legacy=0x7fee08071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:36.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0d747700 1 -- 172.21.15.158:0/3291128714 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0815e7c0 con 0x7fee0815dfb0 2023-10-16T11:17:36.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0df48700 1 --2- 172.21.15.158:0/3291128714 >> v2:172.21.15.172:3300/0 conn(0x7fee080cc5d0 0x7fee0806e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:36.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0d747700 1 --2- 172.21.15.158:0/3291128714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee080cb6f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fee040098f0 tx=0x7fee04009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77eef1bbf0c6d79c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:36.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0cf46700 1 -- 172.21.15.158:0/3291128714 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04011830 con 0x7fee0815dfb0 2023-10-16T11:17:36.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0cf46700 1 -- 172.21.15.158:0/3291128714 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04011990 con 0x7fee0815dfb0 2023-10-16T11:17:36.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.790+0000 7fee0cf46700 1 -- 172.21.15.158:0/3291128714 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04011d20 con 0x7fee0815dfb0 2023-10-16T11:17:36.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 -- 172.21.15.158:0/3291128714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 msgr2=0x7fee080cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:36.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 --2- 172.21.15.158:0/3291128714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee080cb6f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fee040098f0 tx=0x7fee04009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:36.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 -- 172.21.15.158:0/3291128714 shutdown_connections 2023-10-16T11:17:36.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 --2- 172.21.15.158:0/3291128714 >> v2:172.21.15.172:3300/0 conn(0x7fee080cc5d0 0x7fee0806e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 --2- 172.21.15.158:0/3291128714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee080cb6f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 -- 172.21.15.158:0/3291128714 >> 172.21.15.158:0/3291128714 conn(0x7fee080d02b0 msgr2=0x7fee080d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:36.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 -- 172.21.15.158:0/3291128714 shutdown_connections 2023-10-16T11:17:36.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.791+0000 7fee101ac700 1 -- 172.21.15.158:0/3291128714 wait complete. 2023-10-16T11:17:36.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.792+0000 7fee101ac700 1 Processor -- start 2023-10-16T11:17:36.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.792+0000 7fee101ac700 1 -- start start 2023-10-16T11:17:36.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee101ac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0806f340 0x7fee08150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:36.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee101ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee080cc5d0 0x7fee08157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:36.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee101ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee081579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:36.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee101ac700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee08151f60 con 0x7fee080cc5d0 2023-10-16T11:17:36.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee101ac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081ef340 con 0x7fee0806f340 2023-10-16T11:17:36.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee101ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee081ef620 con 0x7fee0815dfb0 2023-10-16T11:17:36.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee080cc5d0 0x7fee08157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:36.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee080cc5d0 0x7fee08157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36320/0 (socket says 172.21.15.158:36320) 2023-10-16T11:17:36.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 -- 172.21.15.158:0/4280397758 learned_addr learned my addr 172.21.15.158:0/4280397758 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:36.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0e749700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee081579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:36.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0df48700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0806f340 0x7fee08150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:36.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 -- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 msgr2=0x7fee081579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:36.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee081579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 -- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0806f340 msgr2=0x7fee08150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:36.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0806f340 0x7fee08150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 -- 172.21.15.158:0/4280397758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80094a0 con 0x7fee080cc5d0 2023-10-16T11:17:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fee0d747700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee080cc5d0 0x7fee08157490 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fee04009c40 tx=0x7fee04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.793+0000 7fedfe7fc700 1 -- 172.21.15.158:0/4280397758 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee040036a0 con 0x7fee080cc5d0 2023-10-16T11:17:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.794+0000 7fedfe7fc700 1 -- 172.21.15.158:0/4280397758 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04003800 con 0x7fee080cc5d0 2023-10-16T11:17:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.794+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee04009550 con 0x7fee080cc5d0 2023-10-16T11:17:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.794+0000 7fedfe7fc700 1 -- 172.21.15.158:0/4280397758 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04003bb0 con 0x7fee080cc5d0 2023-10-16T11:17:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.794+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee081efbe0 con 0x7fee080cc5d0 2023-10-16T11:17:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.794+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee080681b0 con 0x7fee080cc5d0 2023-10-16T11:17:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.795+0000 7fedfe7fc700 1 -- 172.21.15.158:0/4280397758 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee04003d10 con 0x7fee080cc5d0 2023-10-16T11:17:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.795+0000 7fedfe7fc700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fedec06b020 0x7fedec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.795+0000 7fedfe7fc700 1 -- 172.21.15.158:0/4280397758 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5629+0+0 (secure 0 0 0) 0x7fee0408a8d0 con 0x7fee080cc5d0 2023-10-16T11:17:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.795+0000 7fee0df48700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fedec06b020 0x7fedec06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.796+0000 7fee0df48700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fedec06b020 0x7fedec06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee080c6c30 tx=0x7fedf8008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:36.800+0000 7fedfe7fc700 1 -- 172.21.15.158:0/4280397758 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee040558b0 con 0x7fee080cc5d0 2023-10-16T11:17:37.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.071+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fee080b6230 con 0x7fee080cc5d0 2023-10-16T11:17:37.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.071+0000 7fedfe7fc700 1 -- 172.21.15.158:0/4280397758 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fee04058ed0 con 0x7fee080cc5d0 2023-10-16T11:17:37.073 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:37.074 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:17:37.074 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T11:17:37.074 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.074 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T11:17:37.075 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T11:17:37.075 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.075 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.075 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:17:37.075 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:17:37.076 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.076 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.076 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.076 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.076 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T11:17:37.076 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T11:17:37.077 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.077 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.077 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:17:37.077 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:17:37.077 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.078 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.078 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.078 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.078 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T11:17:37.078 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T11:17:37.079 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.079 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.079 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:17:37.079 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:17:37.079 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.080 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.080 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.080 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.080 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T11:17:37.080 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T11:17:37.081 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.081 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.081 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:17:37.081 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:17:37.081 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.081 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.082 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.082 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.082 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T11:17:37.082 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T11:17:37.082 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.083 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.083 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.083 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.083 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.083 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.083 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.084 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T11:17:37.084 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T11:17:37.084 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.084 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.084 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.084 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.085 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.085 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.085 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.085 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T11:17:37.086 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T11:17:37.086 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.086 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.086 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.087 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.087 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.087 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.087 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.087 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T11:17:37.088 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T11:17:37.088 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.088 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.088 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.088 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.089 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.089 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.089 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.089 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T11:17:37.089 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T11:17:37.090 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.090 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.090 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.090 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.090 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.091 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.091 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.091 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T11:17:37.091 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T11:17:37.091 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.091 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.092 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.092 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.092 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.092 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.092 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.093 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T11:17:37.093 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T11:17:37.093 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.093 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.093 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.093 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.094 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.094 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.094 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.094 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T11:17:37.095 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T11:17:37.095 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.095 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:17:37.095 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:17:37.095 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.096 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.096 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.096 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.096 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T11:17:37.096 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T11:17:37.097 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.097 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.097 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T11:17:37.097 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T11:17:37.097 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.097 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.098 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.098 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T11:17:37.098 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T11:17:37.098 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.098 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.099 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:17:37.099 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:17:37.099 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:17:37.099 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.100 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.100 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.100 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T11:17:37.100 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T11:17:37.100 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.101 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.101 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:17:37.101 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:17:37.101 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:17:37.101 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.102 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.102 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.102 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T11:17:37.102 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T11:17:37.102 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.103 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:17:37.103 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:17:37.103 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:17:37.103 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:17:37.103 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.104 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.104 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.104 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T11:17:37.104 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T11:17:37.104 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.104 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:17:37.105 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:17:37.105 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:17:37.105 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.105 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.105 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.106 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.106 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T11:17:37.106 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T11:17:37.106 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.106 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:17:37.107 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:17:37.107 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.107 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.107 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:17:37.107 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:37.108 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T11:17:37.108 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T11:17:37.108 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:17:37.108 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:17:37.108 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:17:37.109 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:17:37.109 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.109 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:37.109 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T11:17:37.109 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:17:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fedec06b020 msgr2=0x7fedec06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fedec06b020 0x7fedec06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee080c6c30 tx=0x7fedf8008040 comp rx=0 tx=0).stop 2023-10-16T11:17:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee080cc5d0 msgr2=0x7fee08157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee080cc5d0 0x7fee08157490 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fee04009c40 tx=0x7fee04004750 comp rx=0 tx=0).stop 2023-10-16T11:17:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 shutdown_connections 2023-10-16T11:17:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee0815dfb0 0x7fee081579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fedec06b020 0x7fedec06d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0806f340 0x7fee08150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 --2- 172.21.15.158:0/4280397758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee080cc5d0 0x7fee08157490 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 >> 172.21.15.158:0/4280397758 conn(0x7fee080d02b0 msgr2=0x7fee080ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 shutdown_connections 2023-10-16T11:17:37.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.074+0000 7fee101ac700 1 -- 172.21.15.158:0/4280397758 wait complete. 2023-10-16T11:17:37.114 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:37.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.234+0000 7fecfbc35700 1 Processor -- start 2023-10-16T11:17:37.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.235+0000 7fecfbc35700 1 -- start start 2023-10-16T11:17:37.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.235+0000 7fecfbc35700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fecf415b730 0x7fecf40ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.235+0000 7fecfbc35700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf40ca780 con 0x7fecf40ce630 2023-10-16T11:17:37.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.235+0000 7fecfbc35700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf406aa90 con 0x7fecf415b350 2023-10-16T11:17:37.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.235+0000 7fecfbc35700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf40cf820 con 0x7fecf415b730 2023-10-16T11:17:37.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecfa1d2700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fecf40ce630 0x7fecf406af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36506/0 (socket says 172.21.15.158:36506) 2023-10-16T11:17:37.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecfa1d2700 1 -- 172.21.15.158:0/2224969165 learned_addr learned my addr 172.21.15.158:0/2224969165 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:37.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124263986 0 0) 0x7fecf40ca780 con 0x7fecf40ce630 2023-10-16T11:17:37.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece00035b0 con 0x7fecf40ce630 2023-10-16T11:17:37.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf91d0700 1 --2- 172.21.15.158:0/2224969165 >> v2:172.21.15.172:3300/0 conn(0x7fecf415b730 0x7fecf40ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411228686 0 0) 0x7fece00035b0 con 0x7fecf40ce630 2023-10-16T11:17:37.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 >> v2:172.21.15.172:3300/0 conn(0x7fecf415b730 msgr2=0x7fecf40ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:37.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 --2- 172.21.15.158:0/2224969165 >> v2:172.21.15.172:3300/0 conn(0x7fecf415b730 0x7fecf40ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 >> v1:172.21.15.172:6789/0 conn(0x7fecf415b350 legacy=0x7fecf40cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40cf260 con 0x7fecf40ce630 2023-10-16T11:17:37.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fecec0019f0 con 0x7fecf40ce630 2023-10-16T11:17:37.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 >> v1:172.21.15.158:6789/0 conn(0x7fecf40ce630 legacy=0x7fecf406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fece0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece0004850 0x7fece0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece0005330 0x7fece000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40cf820 con 0x7fece0003a00 2023-10-16T11:17:37.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece000c200 con 0x7fece0004850 2023-10-16T11:17:37.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece000c3c0 con 0x7fece0005330 2023-10-16T11:17:37.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf99d1700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fece0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecf91d0700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece0004850 0x7fece0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.236+0000 7fecfa1d2700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece0005330 0x7fece000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf99d1700 1 -- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece0005330 msgr2=0x7fece000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:37.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf99d1700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece0005330 0x7fece000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf99d1700 1 -- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece0004850 msgr2=0x7fece0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:37.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf99d1700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece0004850 0x7fece0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecfa1d2700 1 -- 172.21.15.158:0/2224969165 reap_dead start 2023-10-16T11:17:37.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf99d1700 1 -- 172.21.15.158:0/2224969165 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece00035b0 con 0x7fece0003a00 2023-10-16T11:17:37.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf99d1700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fece0003e20 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fece4007fd0 tx=0x7fece0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4227af9d6e57e83 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40052d0 con 0x7fece0003a00 2023-10-16T11:17:37.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4005430 con 0x7fece0003a00 2023-10-16T11:17:37.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecf89cf700 1 -- 172.21.15.158:0/2224969165 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40038c0 con 0x7fece0003a00 2023-10-16T11:17:37.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecfbc35700 1 -- 172.21.15.158:0/2224969165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 msgr2=0x7fece0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fece0003e20 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fece4007fd0 tx=0x7fece0005880 comp rx=0 tx=0).stop 2023-10-16T11:17:37.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecfbc35700 1 -- 172.21.15.158:0/2224969165 shutdown_connections 2023-10-16T11:17:37.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2224969165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fece0003e20 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.237+0000 7fecfbc35700 1 -- 172.21.15.158:0/2224969165 >> 172.21.15.158:0/2224969165 conn(0x7fecf40cc4b0 msgr2=0x7fecf40cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:37.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.238+0000 7fecfbc35700 1 -- 172.21.15.158:0/2224969165 shutdown_connections 2023-10-16T11:17:37.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.238+0000 7fecfbc35700 1 -- 172.21.15.158:0/2224969165 wait complete. 2023-10-16T11:17:37.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.239+0000 7fecfbc35700 1 Processor -- start 2023-10-16T11:17:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.239+0000 7fecfbc35700 1 -- start start 2023-10-16T11:17:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.239+0000 7fecfbc35700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fecf40b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.239+0000 7fecfbc35700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecf40b2750 0x7fecf40c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecfbc35700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecf41517c0 0x7fecf4150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecfbc35700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b37f0 con 0x7fece0003a00 2023-10-16T11:17:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecfbc35700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b2be0 con 0x7fecf40b2750 2023-10-16T11:17:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecfbc35700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b2ec0 con 0x7fecf41517c0 2023-10-16T11:17:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fecf40b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fecf40b3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36338/0 (socket says 172.21.15.158:36338) 2023-10-16T11:17:37.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 -- 172.21.15.158:0/2609975468 learned_addr learned my addr 172.21.15.158:0/2609975468 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:37.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf91d0700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecf40b2750 0x7fecf40c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecfa1d2700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecf41517c0 0x7fecf4150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 -- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecf41517c0 msgr2=0x7fecf4150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecf41517c0 0x7fecf4150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 -- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecf40b2750 msgr2=0x7fecf40c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecf40b2750 0x7fecf40c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 -- 172.21.15.158:0/2609975468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40b31a0 con 0x7fece0003a00 2023-10-16T11:17:37.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecf99d1700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fecf40b3c80 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fece4005040 tx=0x7fece4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecea7fc700 1 -- 172.21.15.158:0/2609975468 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece401f070 con 0x7fece0003a00 2023-10-16T11:17:37.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecea7fc700 1 -- 172.21.15.158:0/2609975468 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4011620 con 0x7fece0003a00 2023-10-16T11:17:37.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf40b3480 con 0x7fece0003a00 2023-10-16T11:17:37.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecea7fc700 1 -- 172.21.15.158:0/2609975468 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40119d0 con 0x7fece0003a00 2023-10-16T11:17:37.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.240+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf40c9970 con 0x7fece0003a00 2023-10-16T11:17:37.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.241+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf40b3ad0 con 0x7fece0003a00 2023-10-16T11:17:37.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.242+0000 7fecea7fc700 1 -- 172.21.15.158:0/2609975468 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fece4011b30 con 0x7fece0003a00 2023-10-16T11:17:37.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.242+0000 7fecea7fc700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fecd806b020 0x7fecd806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.242+0000 7fecea7fc700 1 -- 172.21.15.158:0/2609975468 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5629+0+0 (secure 0 0 0) 0x7fece408d640 con 0x7fece0003a00 2023-10-16T11:17:37.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.242+0000 7fecf91d0700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fecd806b020 0x7fecd806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.242+0000 7fecf91d0700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fecd806b020 0x7fecd806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fecf4151f60 tx=0x7fecf0009210 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.246+0000 7fecea7fc700 1 -- 172.21.15.158:0/2609975468 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fece40586c0 con 0x7fece0003a00 2023-10-16T11:17:37.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.440+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecf40c9970 con 0x7fece0003a00 2023-10-16T11:17:37.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.441+0000 7fecea7fc700 1 -- 172.21.15.158:0/2609975468 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+13774 (secure 0 0 0) 0x7fecf40c9970 con 0x7fece0003a00 2023-10-16T11:17:37.443 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:37.443 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":196,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:36.660742+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:37.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fecd806b020 msgr2=0x7fecd806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fecd806b020 0x7fecd806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fecf4151f60 tx=0x7fecf0009210 comp rx=0 tx=0).stop 2023-10-16T11:17:37.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 msgr2=0x7fecf40b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fecf40b3c80 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fece4005040 tx=0x7fece4001120 comp rx=0 tx=0).stop 2023-10-16T11:17:37.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 shutdown_connections 2023-10-16T11:17:37.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecf41517c0 0x7fecf4150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fece0003a00 0x7fecf40b3c80 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fecd806b020 0x7fecd806d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 --2- 172.21.15.158:0/2609975468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecf40b2750 0x7fecf40c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.443+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 >> 172.21.15.158:0/2609975468 conn(0x7fecf40cc4b0 msgr2=0x7fecf40af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:37.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.444+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 shutdown_connections 2023-10-16T11:17:37.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.444+0000 7fecfbc35700 1 -- 172.21.15.158:0/2609975468 wait complete. 2023-10-16T11:17:37.457 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:17:37.458 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:17:37.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.580+0000 7f66016ab700 1 Processor -- start 2023-10-16T11:17:37.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.580+0000 7f66016ab700 1 -- start start 2023-10-16T11:17:37.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f66016ab700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f65fc0d0c40 0x7f65fc0b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f66016ab700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65fc0b7c90 con 0x7f65fc160d30 2023-10-16T11:17:37.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f66016ab700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65fc0b7f70 con 0x7f65fc0d29b0 2023-10-16T11:17:37.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f66016ab700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65fc0b8910 con 0x7f65fc0d0c40 2023-10-16T11:17:37.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f65fb7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f65fc0d0c40 0x7f65fc0b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f65faffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f65fc160d30 0x7f65fc0b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36522/0 (socket says 172.21.15.158:36522) 2023-10-16T11:17:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f65faffd700 1 -- 172.21.15.158:0/2190047441 learned_addr learned my addr 172.21.15.158:0/2190047441 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f65fb7fe700 1 -- 172.21.15.158:0/2190047441 >> v1:172.21.15.172:6789/0 conn(0x7f65fc0d29b0 legacy=0x7f65fc0d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f65fb7fe700 1 -- 172.21.15.158:0/2190047441 >> v1:172.21.15.158:6789/0 conn(0x7f65fc160d30 legacy=0x7f65fc0b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f65fb7fe700 1 -- 172.21.15.158:0/2190047441 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc0b8250 con 0x7f65fc0d0c40 2023-10-16T11:17:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.581+0000 7f65f9ffb700 1 -- 172.21.15.158:0/2190047441 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058442775 0 0) 0x7f65fc0b7c90 con 0x7f65fc160d30 2023-10-16T11:17:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f65fb7fe700 1 --2- 172.21.15.158:0/2190047441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d0c40 0x7f65fc0b7590 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009d90 tx=0x7f65f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c5193488dfde0d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f65f9ffb700 1 -- 172.21.15.158:0/2190047441 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f400e070 con 0x7f65fc0d0c40 2023-10-16T11:17:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f65f9ffb700 1 -- 172.21.15.158:0/2190047441 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f400bb90 con 0x7f65fc0d0c40 2023-10-16T11:17:37.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f65f9ffb700 1 -- 172.21.15.158:0/2190047441 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4003a20 con 0x7f65fc0d0c40 2023-10-16T11:17:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f66016ab700 1 -- 172.21.15.158:0/2190047441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d0c40 msgr2=0x7f65fc0b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f66016ab700 1 --2- 172.21.15.158:0/2190047441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d0c40 0x7f65fc0b7590 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009d90 tx=0x7f65f400b330 comp rx=0 tx=0).stop 2023-10-16T11:17:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f66016ab700 1 -- 172.21.15.158:0/2190047441 shutdown_connections 2023-10-16T11:17:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f66016ab700 1 --2- 172.21.15.158:0/2190047441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d0c40 0x7f65fc0b7590 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f66016ab700 1 -- 172.21.15.158:0/2190047441 >> 172.21.15.158:0/2190047441 conn(0x7f65fc0669f0 msgr2=0x7f65fc066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:37.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f66016ab700 1 -- 172.21.15.158:0/2190047441 shutdown_connections 2023-10-16T11:17:37.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.582+0000 7f66016ab700 1 -- 172.21.15.158:0/2190047441 wait complete. 2023-10-16T11:17:37.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 Processor -- start 2023-10-16T11:17:37.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 -- start start 2023-10-16T11:17:37.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65fc0d0c40 0x7f65fc15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d29b0 0x7f65fc14b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65fc160d30 0x7f65fc0ce030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc14c4a0 con 0x7f65fc0d29b0 2023-10-16T11:17:37.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc15c430 con 0x7f65fc160d30 2023-10-16T11:17:37.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f66016ab700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc15be50 con 0x7f65fc0d0c40 2023-10-16T11:17:37.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.584+0000 7f65fa7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d29b0 0x7f65fc14b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d29b0 0x7f65fc14b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36352/0 (socket says 172.21.15.158:36352) 2023-10-16T11:17:37.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 -- 172.21.15.158:0/4191140979 learned_addr learned my addr 172.21.15.158:0/4191140979 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:37.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 -- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65fc0d0c40 msgr2=0x7f65fc15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fb7fe700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65fc160d30 0x7f65fc0ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65faffd700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65fc0d0c40 0x7f65fc15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65fc0d0c40 0x7f65fc15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 -- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65fc160d30 msgr2=0x7f65fc0ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65fc160d30 0x7f65fc0ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 -- 172.21.15.158:0/4191140979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f4009a70 con 0x7f65fc0d29b0 2023-10-16T11:17:37.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65fa7fc700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d29b0 0x7f65fc14b8b0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c4d0 tx=0x7f65ec00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65f37fe700 1 -- 172.21.15.158:0/4191140979 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec011790 con 0x7f65fc0d29b0 2023-10-16T11:17:37.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65f37fe700 1 -- 172.21.15.158:0/4191140979 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0118f0 con 0x7f65fc0d29b0 2023-10-16T11:17:37.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc0c9790 con 0x7f65fc0d29b0 2023-10-16T11:17:37.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f65f37fe700 1 -- 172.21.15.158:0/4191140979 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec011ca0 con 0x7f65fc0d29b0 2023-10-16T11:17:37.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.585+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc0c9a70 con 0x7f65fc0d29b0 2023-10-16T11:17:37.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.586+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc15c5a0 con 0x7f65fc0d29b0 2023-10-16T11:17:37.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.587+0000 7f65f37fe700 1 -- 172.21.15.158:0/4191140979 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65ec01c460 con 0x7f65fc0d29b0 2023-10-16T11:17:37.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.587+0000 7f65f37fe700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f65dc06b020 0x7f65dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.587+0000 7f65f37fe700 1 -- 172.21.15.158:0/4191140979 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f65ec089540 con 0x7f65fc0d29b0 2023-10-16T11:17:37.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.587+0000 7f65faffd700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f65dc06b020 0x7f65dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.588+0000 7f65faffd700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f65dc06b020 0x7f65dc06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f65e4000f60 tx=0x7f65e4009040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.592+0000 7f65f37fe700 1 -- 172.21.15.158:0/4191140979 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65ec054530 con 0x7f65fc0d29b0 2023-10-16T11:17:37.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.788+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65fc0681b0 con 0x7f65fc0d29b0 2023-10-16T11:17:37.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.788+0000 7f65f37fe700 1 -- 172.21.15.158:0/4191140979 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v196) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65ec057b50 con 0x7f65fc0d29b0 2023-10-16T11:17:37.790 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:17:37.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f65dc06b020 msgr2=0x7f65dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f65dc06b020 0x7f65dc06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f65e4000f60 tx=0x7f65e4009040 comp rx=0 tx=0).stop 2023-10-16T11:17:37.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d29b0 msgr2=0x7f65fc14b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d29b0 0x7f65fc14b8b0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c4d0 tx=0x7f65ec00c890 comp rx=0 tx=0).stop 2023-10-16T11:17:37.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 shutdown_connections 2023-10-16T11:17:37.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f65dc06b020 0x7f65dc06d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65fc160d30 0x7f65fc0ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65fc0d0c40 0x7f65fc15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 --2- 172.21.15.158:0/4191140979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65fc0d29b0 0x7f65fc14b8b0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 >> 172.21.15.158:0/4191140979 conn(0x7f65fc0669f0 msgr2=0x7f65fc150340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:37.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 shutdown_connections 2023-10-16T11:17:37.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.790+0000 7f66016ab700 1 -- 172.21.15.158:0/4191140979 wait complete. 2023-10-16T11:17:37.804 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:17:37.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.923+0000 7f15a8cf2700 1 Processor -- start 2023-10-16T11:17:37.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.923+0000 7f15a8cf2700 1 -- start start 2023-10-16T11:17:37.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a8cf2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f15a40cc5d0 0x7f15a406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a8cf2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f15a406f340 0x7f15a406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a8cf2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a406e770 con 0x7f15a415dfb0 2023-10-16T11:17:37.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a8cf2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40cf270 con 0x7f15a40cc5d0 2023-10-16T11:17:37.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a8cf2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a406fd10 con 0x7f15a406f340 2023-10-16T11:17:37.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f15a40cc5d0 0x7f15a406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f15a40cc5d0 0x7f15a406e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36362/0 (socket says 172.21.15.158:36362) 2023-10-16T11:17:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 -- 172.21.15.158:0/3500071815 learned_addr learned my addr 172.21.15.158:0/3500071815 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a2d9d700 1 --2- 172.21.15.158:0/3500071815 >> v2:172.21.15.172:3300/0 conn(0x7f15a406f340 0x7f15a406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 -- 172.21.15.158:0/3500071815 >> v2:172.21.15.172:3300/0 conn(0x7f15a406f340 msgr2=0x7f15a406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 --2- 172.21.15.158:0/3500071815 >> v2:172.21.15.172:3300/0 conn(0x7f15a406f340 0x7f15a406f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 -- 172.21.15.158:0/3500071815 >> v1:172.21.15.172:6789/0 conn(0x7f15a415dfb0 legacy=0x7f15a40cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 -- 172.21.15.158:0/3500071815 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40cbe00 con 0x7f15a40cc5d0 2023-10-16T11:17:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.924+0000 7f15a259c700 1 --2- 172.21.15.158:0/3500071815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a40cc5d0 0x7f15a406e230 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f15a40b61e0 tx=0x7f159400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268d081d4febda39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a159a700 1 -- 172.21.15.158:0/3500071815 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159400dcf0 con 0x7f15a40cc5d0 2023-10-16T11:17:37.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a159a700 1 -- 172.21.15.158:0/3500071815 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159400de50 con 0x7f15a40cc5d0 2023-10-16T11:17:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a159a700 1 -- 172.21.15.158:0/3500071815 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594004250 con 0x7f15a40cc5d0 2023-10-16T11:17:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 -- 172.21.15.158:0/3500071815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a40cc5d0 msgr2=0x7f15a406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/3500071815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a40cc5d0 0x7f15a406e230 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f15a40b61e0 tx=0x7f159400be70 comp rx=0 tx=0).stop 2023-10-16T11:17:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 -- 172.21.15.158:0/3500071815 shutdown_connections 2023-10-16T11:17:37.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/3500071815 >> v2:172.21.15.172:3300/0 conn(0x7f15a406f340 0x7f15a406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/3500071815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a40cc5d0 0x7f15a406e230 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 -- 172.21.15.158:0/3500071815 >> 172.21.15.158:0/3500071815 conn(0x7f15a40d02b0 msgr2=0x7f15a40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:37.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 -- 172.21.15.158:0/3500071815 shutdown_connections 2023-10-16T11:17:37.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.925+0000 7f15a8cf2700 1 -- 172.21.15.158:0/3500071815 wait complete. 2023-10-16T11:17:37.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.926+0000 7f15a8cf2700 1 Processor -- start 2023-10-16T11:17:37.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a8cf2700 1 -- start start 2023-10-16T11:17:37.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a8cf2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a406f340 0x7f15a4158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a8cf2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a40cc5d0 0x7f15a4159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a8cf2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a415dfb0 0x7f15a4159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a8cf2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4158d80 con 0x7f15a406f340 2023-10-16T11:17:37.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a8cf2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40a9490 con 0x7f15a40cc5d0 2023-10-16T11:17:37.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a8cf2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40c9450 con 0x7f15a415dfb0 2023-10-16T11:17:37.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a406f340 0x7f15a4158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a406f340 0x7f15a4158840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36368/0 (socket says 172.21.15.158:36368) 2023-10-16T11:17:37.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.927+0000 7f15a259c700 1 -- 172.21.15.158:0/2029796825 learned_addr learned my addr 172.21.15.158:0/2029796825 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:37.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a1d9b700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a40cc5d0 0x7f15a4159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a2d9d700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a415dfb0 0x7f15a4159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a259c700 1 -- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a415dfb0 msgr2=0x7f15a4159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a259c700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a415dfb0 0x7f15a4159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a259c700 1 -- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a40cc5d0 msgr2=0x7f15a4159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:37.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a259c700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a40cc5d0 0x7f15a4159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:37.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a259c700 1 -- 172.21.15.158:0/2029796825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159400b7f0 con 0x7f15a406f340 2023-10-16T11:17:37.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a1d9b700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a40cc5d0 0x7f15a4159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:37.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a259c700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a406f340 0x7f15a4158840 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1594000c00 tx=0x7f1594004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f1592ffd700 1 -- 172.21.15.158:0/2029796825 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594020070 con 0x7f15a406f340 2023-10-16T11:17:37.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40a8480 con 0x7f15a406f340 2023-10-16T11:17:37.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f1592ffd700 1 -- 172.21.15.158:0/2029796825 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1594004030 con 0x7f15a406f340 2023-10-16T11:17:37.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f1592ffd700 1 -- 172.21.15.158:0/2029796825 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594010470 con 0x7f15a406f340 2023-10-16T11:17:37.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.928+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40a9000 con 0x7f15a406f340 2023-10-16T11:17:37.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.929+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a40681b0 con 0x7f15a406f340 2023-10-16T11:17:37.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.929+0000 7f1592ffd700 1 -- 172.21.15.158:0/2029796825 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f15940193f0 con 0x7f15a406f340 2023-10-16T11:17:37.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.930+0000 7f1592ffd700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f158406b020 0x7f158406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:37.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.930+0000 7f1592ffd700 1 -- 172.21.15.158:0/2029796825 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f159408cd40 con 0x7f15a406f340 2023-10-16T11:17:37.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.930+0000 7f15a1d9b700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f158406b020 0x7f158406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:37.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.930+0000 7f15a1d9b700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f158406b020 0x7f158406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f15a40a9940 tx=0x7f1598009040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:37.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:37.934+0000 7f1592ffd700 1 -- 172.21.15.158:0/2029796825 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1594057c30 con 0x7f15a406f340 2023-10-16T11:17:38.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.131+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f15a40a92e0 con 0x7f15a406f340 2023-10-16T11:17:38.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.131+0000 7f1592ffd700 1 -- 172.21.15.158:0/2029796825 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v196) v1 ==== 84+0+65 (secure 0 0 0) 0x7f159405b250 con 0x7f15a406f340 2023-10-16T11:17:38.133 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:38.133 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:17:38.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:38.134 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:17:38.134 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:17:38.134 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:38.135 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:17:38.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f158406b020 msgr2=0x7f158406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f158406b020 0x7f158406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f15a40a9940 tx=0x7f1598009040 comp rx=0 tx=0).stop 2023-10-16T11:17:38.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a406f340 msgr2=0x7f15a4158840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a406f340 0x7f15a4158840 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1594000c00 tx=0x7f1594004750 comp rx=0 tx=0).stop 2023-10-16T11:17:38.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 shutdown_connections 2023-10-16T11:17:38.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15a415dfb0 0x7f15a4159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15a406f340 0x7f15a4158840 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f158406b020 0x7f158406d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 --2- 172.21.15.158:0/2029796825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15a40cc5d0 0x7f15a4159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 >> 172.21.15.158:0/2029796825 conn(0x7f15a40d02b0 msgr2=0x7f15a415aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:38.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 shutdown_connections 2023-10-16T11:17:38.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.133+0000 7f15a8cf2700 1 -- 172.21.15.158:0/2029796825 wait complete. 2023-10-16T11:17:38.147 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T11:17:38.148 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T11:17:38.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.273+0000 7f420602c700 1 Processor -- start 2023-10-16T11:17:38.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.273+0000 7f420602c700 1 -- start start 2023-10-16T11:17:38.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.273+0000 7f420602c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 0x7f42000caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.273+0000 7f420602c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000cbd30 con 0x7f420015da40 2023-10-16T11:17:38.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.273+0000 7f420602c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000cc1c0 con 0x7f42000cc590 2023-10-16T11:17:38.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.273+0000 7f420602c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420015de20 con 0x7f420006afe0 2023-10-16T11:17:38.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41ff7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f420015da40 0x7f420006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36552/0 (socket says 172.21.15.158:36552) 2023-10-16T11:17:38.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41ff7fe700 1 -- 172.21.15.158:0/536682579 learned_addr learned my addr 172.21.15.158:0/536682579 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:38.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41feffd700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 0x7f42000caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887447479 0 0) 0x7f42000cbd30 con 0x7f420015da40 2023-10-16T11:17:38.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e80035b0 con 0x7f420015da40 2023-10-16T11:17:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 442072748 0 0) 0x7f41e80035b0 con 0x7f420015da40 2023-10-16T11:17:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 msgr2=0x7f42000caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 0x7f42000caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 >> v1:172.21.15.172:6789/0 conn(0x7f42000cc590 legacy=0x7f42000afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000ce260 con 0x7f420015da40 2023-10-16T11:17:38.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f41f0001670 con 0x7f420015da40 2023-10-16T11:17:38.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 >> v1:172.21.15.158:6789/0 conn(0x7f420015da40 legacy=0x7f420006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41e8003a30 0x7f41e8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 0x7f41e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41e8004d90 0x7f41e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420015de20 con 0x7f41e8004d90 2023-10-16T11:17:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e800ba70 con 0x7f41e8003a30 2023-10-16T11:17:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e800bc30 con 0x7f420006afe0 2023-10-16T11:17:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41ff7fe700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41e8003a30 0x7f41e8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41fffff700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41e8004d90 0x7f41e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.274+0000 7f41feffd700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 0x7f41e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fffff700 1 -- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 msgr2=0x7f41e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fffff700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420006afe0 0x7f41e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fffff700 1 -- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41e8003a30 msgr2=0x7f41e8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fffff700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41e8003a30 0x7f41e8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fffff700 1 -- 172.21.15.158:0/536682579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80035b0 con 0x7f41e8004d90 2023-10-16T11:17:38.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fffff700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41e8004d90 0x7f41e8005210 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f41f800c320 tx=0x7f41e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e24d4246f365d66 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:38.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8001ef0 con 0x7f41e8004d90 2023-10-16T11:17:38.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f800ce80 con 0x7f41e8004d90 2023-10-16T11:17:38.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fe7fc700 1 -- 172.21.15.158:0/536682579 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8012660 con 0x7f41e8004d90 2023-10-16T11:17:38.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f420602c700 1 -- 172.21.15.158:0/536682579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41e8004d90 msgr2=0x7f41e8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f420602c700 1 --2- 172.21.15.158:0/536682579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41e8004d90 0x7f41e8005210 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f41f800c320 tx=0x7f41e8005880 comp rx=0 tx=0).stop 2023-10-16T11:17:38.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.275+0000 7f41fffff700 1 -- 172.21.15.158:0/536682579 reap_dead start 2023-10-16T11:17:38.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.276+0000 7f420602c700 1 -- 172.21.15.158:0/536682579 shutdown_connections 2023-10-16T11:17:38.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.276+0000 7f420602c700 1 -- 172.21.15.158:0/536682579 >> 172.21.15.158:0/536682579 conn(0x7f4200099a10 msgr2=0x7f42000d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:38.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.276+0000 7f420602c700 1 -- 172.21.15.158:0/536682579 shutdown_connections 2023-10-16T11:17:38.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.276+0000 7f420602c700 1 -- 172.21.15.158:0/536682579 wait complete. 2023-10-16T11:17:38.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.277+0000 7f420602c700 1 Processor -- start 2023-10-16T11:17:38.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.277+0000 7f420602c700 1 -- start start 2023-10-16T11:17:38.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f420602c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4200148760 0x7f420014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f420602c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42000c74a0 0x7f42000b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f420602c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42000b4fe0 0x7f42000c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f420602c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000b5470 con 0x7f4200148760 2023-10-16T11:17:38.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f420602c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420015b110 con 0x7f42000c74a0 2023-10-16T11:17:38.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f420602c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420014bcd0 con 0x7f42000b4fe0 2023-10-16T11:17:38.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4200148760 0x7f420014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4200148760 0x7f420014b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36388/0 (socket says 172.21.15.158:36388) 2023-10-16T11:17:38.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 -- 172.21.15.158:0/1621222189 learned_addr learned my addr 172.21.15.158:0/1621222189 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:38.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41feffd700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42000c74a0 0x7f42000b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41fffff700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42000b4fe0 0x7f42000c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 -- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42000b4fe0 msgr2=0x7f42000c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42000b4fe0 0x7f42000c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 -- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42000c74a0 msgr2=0x7f42000b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42000c74a0 0x7f42000b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 -- 172.21.15.158:0/1621222189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000c7960 con 0x7f4200148760 2023-10-16T11:17:38.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41ff7fe700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4200148760 0x7f420014b790 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f41f00011e0 tx=0x7f41f0007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:38.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41e7fff700 1 -- 172.21.15.158:0/1621222189 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f00077e0 con 0x7f4200148760 2023-10-16T11:17:38.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.278+0000 7f41e7fff700 1 -- 172.21.15.158:0/1621222189 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f000f870 con 0x7f4200148760 2023-10-16T11:17:38.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.279+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42000c7c40 con 0x7f4200148760 2023-10-16T11:17:38.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.279+0000 7f41e7fff700 1 -- 172.21.15.158:0/1621222189 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f000fc00 con 0x7f4200148760 2023-10-16T11:17:38.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.279+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42000c8200 con 0x7f4200148760 2023-10-16T11:17:38.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.280+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42000681b0 con 0x7f4200148760 2023-10-16T11:17:38.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.280+0000 7f41e7fff700 1 -- 172.21.15.158:0/1621222189 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f41f000fd60 con 0x7f4200148760 2023-10-16T11:17:38.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.280+0000 7f41e7fff700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f41ec06b020 0x7f41ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.280+0000 7f41e7fff700 1 -- 172.21.15.158:0/1621222189 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5629+0+0 (secure 0 0 0) 0x7f41f00896f0 con 0x7f4200148760 2023-10-16T11:17:38.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.280+0000 7f41feffd700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f41ec06b020 0x7f41ec06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.281+0000 7f41feffd700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f41ec06b020 0x7f41ec06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f420006e090 tx=0x7f41f4008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:38.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.285+0000 7f41e7fff700 1 -- 172.21.15.158:0/1621222189 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f41f0054820 con 0x7f4200148760 2023-10-16T11:17:38.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.477+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f420006dde0 con 0x7f4200148760 2023-10-16T11:17:38.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.657+0000 7f41e7fff700 1 -- 172.21.15.158:0/1621222189 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v197) v1 ==== 138+0+0 (secure 0 0 0) 0x7f41f0057e40 con 0x7f4200148760 2023-10-16T11:17:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f41ec06b020 msgr2=0x7f41ec06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f41ec06b020 0x7f41ec06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f420006e090 tx=0x7f41f4008040 comp rx=0 tx=0).stop 2023-10-16T11:17:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4200148760 msgr2=0x7f420014b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4200148760 0x7f420014b790 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f41f00011e0 tx=0x7f41f0007600 comp rx=0 tx=0).stop 2023-10-16T11:17:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 shutdown_connections 2023-10-16T11:17:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42000b4fe0 0x7f42000c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4200148760 0x7f420014b790 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f41ec06b020 0x7f41ec06d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 --2- 172.21.15.158:0/1621222189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42000c74a0 0x7f42000b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 >> 172.21.15.158:0/1621222189 conn(0x7f4200099a10 msgr2=0x7f420015ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.660+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 shutdown_connections 2023-10-16T11:17:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.661+0000 7f420602c700 1 -- 172.21.15.158:0/1621222189 wait complete. 2023-10-16T11:17:38.665 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T11:17:38.678 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:17:38.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.804+0000 7f61d13e7700 1 Processor -- start 2023-10-16T11:17:38.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.805+0000 7f61d13e7700 1 -- start start 2023-10-16T11:17:38.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.805+0000 7f61d13e7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f61cc0c97e0 0x7f61cc06cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.805+0000 7f61d13e7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc071790 con 0x7f61cc0c9e50 2023-10-16T11:17:38.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.805+0000 7f61d13e7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc070ed0 con 0x7f61cc159c20 2023-10-16T11:17:38.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.805+0000 7f61d13e7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61cc072330 con 0x7f61cc0c97e0 2023-10-16T11:17:38.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61cb7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f61cc0c9e50 0x7f61cc0ca2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36576/0 (socket says 172.21.15.158:36576) 2023-10-16T11:17:38.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61cb7fe700 1 -- 172.21.15.158:0/3455933203 learned_addr learned my addr 172.21.15.158:0/3455933203 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:38.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61ca7fc700 1 --2- 172.21.15.158:0/3455933203 >> v2:172.21.15.172:3300/0 conn(0x7f61cc0c97e0 0x7f61cc06cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578607160 0 0) 0x7f61cc071790 con 0x7f61cc0c9e50 2023-10-16T11:17:38.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b40057c0 con 0x7f61cc0c9e50 2023-10-16T11:17:38.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113387647 0 0) 0x7f61b40057c0 con 0x7f61cc0c9e50 2023-10-16T11:17:38.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 >> v2:172.21.15.172:3300/0 conn(0x7f61cc0c97e0 msgr2=0x7f61cc06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:38.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 --2- 172.21.15.158:0/3455933203 >> v2:172.21.15.172:3300/0 conn(0x7f61cc0c97e0 0x7f61cc06cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 >> v1:172.21.15.172:6789/0 conn(0x7f61cc159c20 legacy=0x7f61cc071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc15a000 con 0x7f61cc0c9e50 2023-10-16T11:17:38.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f61c40019f0 con 0x7f61cc0c9e50 2023-10-16T11:17:38.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 >> v1:172.21.15.158:6789/0 conn(0x7f61cc0c9e50 legacy=0x7f61cc0ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4005c10 0x7f61b4006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61b4006a60 0x7f61b4006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61b4007420 0x7f61b400dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc072330 con 0x7f61b4007420 2023-10-16T11:17:38.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b400e2c0 con 0x7f61b4005c10 2023-10-16T11:17:38.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b400e480 con 0x7f61b4006a60 2023-10-16T11:17:38.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61cb7fe700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61b4007420 0x7f61b400dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61caffd700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4005c10 0x7f61b4006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.806+0000 7f61ca7fc700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61b4006a60 0x7f61b4006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61cb7fe700 1 -- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61b4006a60 msgr2=0x7f61b4006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:38.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61cb7fe700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61b4006a60 0x7f61b4006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61cb7fe700 1 -- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4005c10 msgr2=0x7f61b4006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:38.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61cb7fe700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4005c10 0x7f61b4006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61cb7fe700 1 -- 172.21.15.158:0/3455933203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40057c0 con 0x7f61b4007420 2023-10-16T11:17:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61cb7fe700 1 -- 172.21.15.158:0/3455933203 reap_dead start 2023-10-16T11:17:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61cb7fe700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61b4007420 0x7f61b400dc20 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f61c400a830 tx=0x7f61b4007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5957cdd5c32ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c400a2f0 con 0x7f61b4007420 2023-10-16T11:17:38.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c4012600 con 0x7f61b4007420 2023-10-16T11:17:38.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61c9ffb700 1 -- 172.21.15.158:0/3455933203 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c40129b0 con 0x7f61b4007420 2023-10-16T11:17:38.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61d13e7700 1 -- 172.21.15.158:0/3455933203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61b4007420 msgr2=0x7f61b400dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61d13e7700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61b4007420 0x7f61b400dc20 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f61c400a830 tx=0x7f61b4007a90 comp rx=0 tx=0).stop 2023-10-16T11:17:38.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61d13e7700 1 -- 172.21.15.158:0/3455933203 shutdown_connections 2023-10-16T11:17:38.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61d13e7700 1 --2- 172.21.15.158:0/3455933203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61b4007420 0x7f61b400dc20 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61d13e7700 1 -- 172.21.15.158:0/3455933203 >> 172.21.15.158:0/3455933203 conn(0x7f61cc0cbc10 msgr2=0x7f61cc0cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:38.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61d13e7700 1 -- 172.21.15.158:0/3455933203 shutdown_connections 2023-10-16T11:17:38.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.807+0000 7f61d13e7700 1 -- 172.21.15.158:0/3455933203 wait complete. 2023-10-16T11:17:38.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.809+0000 7f61d13e7700 1 Processor -- start 2023-10-16T11:17:38.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.809+0000 7f61d13e7700 1 -- start start 2023-10-16T11:17:38.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4007420 0x7f61cc158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61cc156520 0x7f61cc0c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61cc0c8950 0x7f61cc15c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc1551f0 con 0x7f61cc0c8950 2023-10-16T11:17:38.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc158990 con 0x7f61b4007420 2023-10-16T11:17:38.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0c9140 con 0x7f61cc156520 2023-10-16T11:17:38.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61cc0c8950 0x7f61cc15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61caffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4007420 0x7f61cc158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61caffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4007420 0x7f61cc158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:37588/0 (socket says 172.21.15.158:37588) 2023-10-16T11:17:38.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61caffd700 1 -- 172.21.15.158:0/2250758118 learned_addr learned my addr 172.21.15.158:0/2250758118 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:38.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61cc0c8950 0x7f61cc15c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36412/0 (socket says 172.21.15.158:36412) 2023-10-16T11:17:38.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61ca7fc700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61cc156520 0x7f61cc0c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 -- 172.21.15.158:0/2250758118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61cc156520 msgr2=0x7f61cc0c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61cc156520 0x7f61cc0c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 -- 172.21.15.158:0/2250758118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4007420 msgr2=0x7f61cc158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:38.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4007420 0x7f61cc158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:38.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 -- 172.21.15.158:0/2250758118 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc0c8e60 con 0x7f61cc0c8950 2023-10-16T11:17:38.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61cb7fe700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61cc0c8950 0x7f61cc15c440 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0cc820 tx=0x7f61c4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:38.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61b37fe700 1 -- 172.21.15.158:0/2250758118 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4024070 con 0x7f61cc0c8950 2023-10-16T11:17:38.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61b37fe700 1 -- 172.21.15.158:0/2250758118 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c4012400 con 0x7f61cc0c8950 2023-10-16T11:17:38.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61cc1e2440 con 0x7f61cc0c8950 2023-10-16T11:17:38.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61b37fe700 1 -- 172.21.15.158:0/2250758118 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c401e430 con 0x7f61cc0c8950 2023-10-16T11:17:38.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.810+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61cc1e2a00 con 0x7f61cc0c8950 2023-10-16T11:17:38.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.811+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61cc0681b0 con 0x7f61cc0c8950 2023-10-16T11:17:38.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.812+0000 7f61b37fe700 1 -- 172.21.15.158:0/2250758118 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f61c401e590 con 0x7f61cc0c8950 2023-10-16T11:17:38.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.812+0000 7f61b37fe700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f61b806b020 0x7f61b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:38.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.813+0000 7f61b37fe700 1 -- 172.21.15.158:0/2250758118 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5999+0+0 (secure 0 0 0) 0x7f61c408aac0 con 0x7f61cc0c8950 2023-10-16T11:17:38.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.813+0000 7f61caffd700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f61b806b020 0x7f61b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:38.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.813+0000 7f61caffd700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f61b806b020 0x7f61b806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61c00018f0 tx=0x7f61c0009040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:38.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:38.817+0000 7f61b37fe700 1 -- 172.21.15.158:0/2250758118 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61c4055a70 con 0x7f61cc0c8950 2023-10-16T11:17:39.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.012+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f61cc0b1af0 con 0x7f61cc0c8950 2023-10-16T11:17:39.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.660+0000 7f61b37fe700 1 -- 172.21.15.158:0/2250758118 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v198) v1 ==== 170+0+0 (secure 0 0 0) 0x7f61c408ae60 con 0x7f61cc0c8950 2023-10-16T11:17:39.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f61b806b020 msgr2=0x7f61b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:39.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f61b806b020 0x7f61b806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61c00018f0 tx=0x7f61c0009040 comp rx=0 tx=0).stop 2023-10-16T11:17:39.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61cc0c8950 msgr2=0x7f61cc15c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:39.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61cc0c8950 0x7f61cc15c440 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0cc820 tx=0x7f61c4001d80 comp rx=0 tx=0).stop 2023-10-16T11:17:39.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 shutdown_connections 2023-10-16T11:17:39.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61cc0c8950 0x7f61cc15c440 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f61b806b020 0x7f61b806d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61b4007420 0x7f61cc158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 --2- 172.21.15.158:0/2250758118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f61cc156520 0x7f61cc0c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.664+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 >> 172.21.15.158:0/2250758118 conn(0x7f61cc0cbc10 msgr2=0x7f61cc0ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.670+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 shutdown_connections 2023-10-16T11:17:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.671+0000 7f61d13e7700 1 -- 172.21.15.158:0/2250758118 wait complete. 2023-10-16T11:17:39.681 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T11:17:39.696 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T11:17:39.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.821+0000 7fecb8a5f700 1 Processor -- start 2023-10-16T11:17:39.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.821+0000 7fecb8a5f700 1 -- start start 2023-10-16T11:17:39.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb8a5f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fecb406afe0 0x7fecb40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:39.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb8a5f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fecb40cc590 0x7fecb40afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:39.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb8a5f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40cbd30 con 0x7fecb415da40 2023-10-16T11:17:39.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb8a5f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb40cc1c0 con 0x7fecb406afe0 2023-10-16T11:17:39.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb8a5f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb415de20 con 0x7fecb40cc590 2023-10-16T11:17:39.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fecb406afe0 0x7fecb40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:39.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fecb406afe0 0x7fecb40caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36418/0 (socket says 172.21.15.158:36418) 2023-10-16T11:17:39.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb1d9b700 1 -- 172.21.15.158:0/1617992394 learned_addr learned my addr 172.21.15.158:0/1617992394 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:39.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb2d9d700 1 --2- 172.21.15.158:0/1617992394 >> v2:172.21.15.172:3300/0 conn(0x7fecb40cc590 0x7fecb40afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:39.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb1d9b700 1 -- 172.21.15.158:0/1617992394 >> v2:172.21.15.172:3300/0 conn(0x7fecb40cc590 msgr2=0x7fecb40afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:39.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.822+0000 7fecb1d9b700 1 --2- 172.21.15.158:0/1617992394 >> v2:172.21.15.172:3300/0 conn(0x7fecb40cc590 0x7fecb40afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb1d9b700 1 -- 172.21.15.158:0/1617992394 >> v1:172.21.15.172:6789/0 conn(0x7fecb415da40 legacy=0x7fecb406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:39.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb1d9b700 1 -- 172.21.15.158:0/1617992394 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb40ce260 con 0x7fecb406afe0 2023-10-16T11:17:39.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb1d9b700 1 --2- 172.21.15.158:0/1617992394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40caa20 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fecb40cdd70 tx=0x7feca4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2b00439b93acd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:39.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb159a700 1 -- 172.21.15.158:0/1617992394 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca400b870 con 0x7fecb406afe0 2023-10-16T11:17:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb159a700 1 -- 172.21.15.158:0/1617992394 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca400b9d0 con 0x7fecb406afe0 2023-10-16T11:17:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb159a700 1 -- 172.21.15.158:0/1617992394 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca400bd60 con 0x7fecb406afe0 2023-10-16T11:17:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1617992394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 msgr2=0x7fecb40caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.823+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1617992394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40caa20 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fecb40cdd70 tx=0x7feca4009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.824+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1617992394 shutdown_connections 2023-10-16T11:17:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.824+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1617992394 >> v2:172.21.15.172:3300/0 conn(0x7fecb40cc590 0x7fecb40afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.824+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1617992394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40caa20 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.824+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1617992394 >> 172.21.15.158:0/1617992394 conn(0x7fecb4099a10 msgr2=0x7fecb40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:39.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.824+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1617992394 shutdown_connections 2023-10-16T11:17:39.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.824+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1617992394 wait complete. 2023-10-16T11:17:39.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.825+0000 7fecb8a5f700 1 Processor -- start 2023-10-16T11:17:39.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.825+0000 7fecb8a5f700 1 -- start start 2023-10-16T11:17:39.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:39.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecb40cc590 0x7fecb40c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:39.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecb415da40 0x7fecb40c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:39.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb40c9bf0 con 0x7fecb406afe0 2023-10-16T11:17:39.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb40c83c0 con 0x7fecb40cc590 2023-10-16T11:17:39.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb40c9670 con 0x7fecb415da40 2023-10-16T11:17:39.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:39.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36422/0 (socket says 172.21.15.158:36422) 2023-10-16T11:17:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 -- 172.21.15.158:0/1955642883 learned_addr learned my addr 172.21.15.158:0/1955642883 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb2d9d700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecb415da40 0x7fecb40c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb1d9b700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecb40cc590 0x7fecb40c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 -- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecb415da40 msgr2=0x7fecb40c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecb415da40 0x7fecb40c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 -- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecb40cc590 msgr2=0x7fecb40c8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecb40cc590 0x7fecb40c8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 -- 172.21.15.158:0/1955642883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca4009550 con 0x7fecb406afe0 2023-10-16T11:17:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb259c700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40b1880 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7feca800c450 tx=0x7feca800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7feca2ffd700 1 -- 172.21.15.158:0/1955642883 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80052d0 con 0x7fecb406afe0 2023-10-16T11:17:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7feca2ffd700 1 -- 172.21.15.158:0/1955642883 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca8005430 con 0x7fecb406afe0 2023-10-16T11:17:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb40c9910 con 0x7fecb406afe0 2023-10-16T11:17:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7feca2ffd700 1 -- 172.21.15.158:0/1955642883 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80038c0 con 0x7fecb406afe0 2023-10-16T11:17:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.826+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb40b1dc0 con 0x7fecb406afe0 2023-10-16T11:17:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.827+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb40c7eb0 con 0x7fecb406afe0 2023-10-16T11:17:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.828+0000 7feca2ffd700 1 -- 172.21.15.158:0/1955642883 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feca8003a20 con 0x7fecb406afe0 2023-10-16T11:17:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.828+0000 7feca2ffd700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fec9406b020 0x7fec9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.828+0000 7feca2ffd700 1 -- 172.21.15.158:0/1955642883 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6381+0+0 (secure 0 0 0) 0x7feca8088e40 con 0x7fecb406afe0 2023-10-16T11:17:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.828+0000 7fecb1d9b700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fec9406b020 0x7fec9406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.829+0000 7fecb1d9b700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fec9406b020 0x7fec9406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca4008030 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:39.833+0000 7feca2ffd700 1 -- 172.21.15.158:0/1955642883 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fecb40c7eb0 con 0x7fecb406afe0 2023-10-16T11:17:40.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.070+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fecb40c7eb0 con 0x7fecb406afe0 2023-10-16T11:17:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.688+0000 7feca2ffd700 1 -- 172.21.15.158:0/1955642883 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v99) v1 ==== 174+0+0 (secure 0 0 0) 0x7fecb40c7eb0 con 0x7fecb406afe0 2023-10-16T11:17:40.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fec9406b020 msgr2=0x7fec9406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:40.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fec9406b020 0x7fec9406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca4008030 comp rx=0 tx=0).stop 2023-10-16T11:17:40.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 msgr2=0x7fecb40b1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:40.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40b1880 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7feca800c450 tx=0x7feca800c760 comp rx=0 tx=0).stop 2023-10-16T11:17:40.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 shutdown_connections 2023-10-16T11:17:40.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fecb415da40 0x7fecb40c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:40.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecb406afe0 0x7fecb40b1880 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:40.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fec9406b020 0x7fec9406d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:40.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 --2- 172.21.15.158:0/1955642883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecb40cc590 0x7fecb40c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:40.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.701+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 >> 172.21.15.158:0/1955642883 conn(0x7fecb4099a10 msgr2=0x7fecb40b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:40.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.702+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 shutdown_connections 2023-10-16T11:17:40.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.702+0000 7fecb8a5f700 1 -- 172.21.15.158:0/1955642883 wait complete. 2023-10-16T11:17:40.707 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 25 and data pool 26 2023-10-16T11:17:40.723 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:40.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.846+0000 7f86a2c32700 1 Processor -- start 2023-10-16T11:17:40.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.846+0000 7f86a2c32700 1 -- start start 2023-10-16T11:17:40.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a2c32700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f869c0729b0 0x7f869c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a2c32700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c161270 con 0x7f869c06e5d0 2023-10-16T11:17:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a2c32700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c161510 con 0x7f869c06ab40 2023-10-16T11:17:40.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a2c32700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c1617f0 con 0x7f869c0729b0 2023-10-16T11:17:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f869bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f869c06e5d0 0x7f869c15df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36608/0 (socket says 172.21.15.158:36608) 2023-10-16T11:17:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f869bfff700 1 -- 172.21.15.158:0/3514571305 learned_addr learned my addr 172.21.15.158:0/3514571305 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a11cf700 1 --2- 172.21.15.158:0/3514571305 >> v2:172.21.15.172:3300/0 conn(0x7f869c0729b0 0x7f869c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f869b7fe700 1 -- 172.21.15.158:0/3514571305 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246936986 0 0) 0x7f869c161270 con 0x7f869c06e5d0 2023-10-16T11:17:40.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f869b7fe700 1 -- 172.21.15.158:0/3514571305 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c0035b0 con 0x7f869c06e5d0 2023-10-16T11:17:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a11cf700 1 -- 172.21.15.158:0/3514571305 >> v1:172.21.15.172:6789/0 conn(0x7f869c06ab40 legacy=0x7f869c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a11cf700 1 -- 172.21.15.158:0/3514571305 >> v1:172.21.15.158:6789/0 conn(0x7f869c06e5d0 legacy=0x7f869c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f869b7fe700 1 -- 172.21.15.158:0/3514571305 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 458462365 0 0) 0x7f868c0035b0 con 0x7f869c06e5d0 2023-10-16T11:17:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.847+0000 7f86a11cf700 1 -- 172.21.15.158:0/3514571305 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c161ad0 con 0x7f869c0729b0 2023-10-16T11:17:40.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a11cf700 1 --2- 172.21.15.158:0/3514571305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 0x7f869c06d820 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8694009d90 tx=0x7f869400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7af11b54eab02bec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f869b7fe700 1 -- 172.21.15.158:0/3514571305 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869400e070 con 0x7f869c0729b0 2023-10-16T11:17:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f869b7fe700 1 -- 172.21.15.158:0/3514571305 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869400bb20 con 0x7f869c0729b0 2023-10-16T11:17:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f869b7fe700 1 -- 172.21.15.158:0/3514571305 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869400bed0 con 0x7f869c0729b0 2023-10-16T11:17:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a2c32700 1 -- 172.21.15.158:0/3514571305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 msgr2=0x7f869c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:40.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a2c32700 1 --2- 172.21.15.158:0/3514571305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 0x7f869c06d820 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8694009d90 tx=0x7f869400b330 comp rx=0 tx=0).stop 2023-10-16T11:17:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a2c32700 1 -- 172.21.15.158:0/3514571305 shutdown_connections 2023-10-16T11:17:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a2c32700 1 --2- 172.21.15.158:0/3514571305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 0x7f869c06d820 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a2c32700 1 -- 172.21.15.158:0/3514571305 >> 172.21.15.158:0/3514571305 conn(0x7f869c0d02b0 msgr2=0x7f869c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a2c32700 1 -- 172.21.15.158:0/3514571305 shutdown_connections 2023-10-16T11:17:40.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.848+0000 7f86a2c32700 1 -- 172.21.15.158:0/3514571305 wait complete. 2023-10-16T11:17:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 Processor -- start 2023-10-16T11:17:40.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 -- start start 2023-10-16T11:17:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f869c06ab40 0x7f869c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c06e5d0 0x7f869c0c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 0x7f869c15de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0b3f80 con 0x7f869c06e5d0 2023-10-16T11:17:40.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0b4260 con 0x7f869c0729b0 2023-10-16T11:17:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.850+0000 7f86a2c32700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0b4540 con 0x7f869c06ab40 2023-10-16T11:17:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c06e5d0 0x7f869c0c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c06e5d0 0x7f869c0c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36438/0 (socket says 172.21.15.158:36438) 2023-10-16T11:17:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 -- 172.21.15.158:0/4252629339 learned_addr learned my addr 172.21.15.158:0/4252629339 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:40.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f86a11cf700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 0x7f869c15de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f86a09ce700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f869c06ab40 0x7f869c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 -- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f869c06ab40 msgr2=0x7f869c0b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f869c06ab40 0x7f869c0b5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 -- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 msgr2=0x7f869c15de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:40.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 0x7f869c15de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 -- 172.21.15.158:0/4252629339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694009a70 con 0x7f869c06e5d0 2023-10-16T11:17:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f869bfff700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c06e5d0 0x7f869c0c9180 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f868800c750 tx=0x7f868800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f86997fa700 1 -- 172.21.15.158:0/4252629339 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688005d70 con 0x7f869c06e5d0 2023-10-16T11:17:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f86997fa700 1 -- 172.21.15.158:0/4252629339 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86880070d0 con 0x7f869c06e5d0 2023-10-16T11:17:40.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c0b4820 con 0x7f869c06e5d0 2023-10-16T11:17:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f86997fa700 1 -- 172.21.15.158:0/4252629339 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688007480 con 0x7f869c06e5d0 2023-10-16T11:17:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.851+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c0b5540 con 0x7f869c06e5d0 2023-10-16T11:17:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.852+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c0681b0 con 0x7f869c06e5d0 2023-10-16T11:17:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.853+0000 7f86997fa700 1 -- 172.21.15.158:0/4252629339 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f86880075e0 con 0x7f869c06e5d0 2023-10-16T11:17:40.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.853+0000 7f86997fa700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f868006b020 0x7f868006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.853+0000 7f86997fa700 1 -- 172.21.15.158:0/4252629339 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f8688089050 con 0x7f869c06e5d0 2023-10-16T11:17:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.853+0000 7f86a09ce700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f868006b020 0x7f868006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:40.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.853+0000 7f86a09ce700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f868006b020 0x7f868006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8690000f60 tx=0x7f8690009040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:40.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:40.857+0000 7f86997fa700 1 -- 172.21.15.158:0/4252629339 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8688053ce0 con 0x7f869c06e5d0 2023-10-16T11:17:41.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.054+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f869c162060 con 0x7f869c06e5d0 2023-10-16T11:17:41.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.055+0000 7f86997fa700 1 -- 172.21.15.158:0/4252629339 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+17435 (secure 0 0 0) 0x7f8688057300 con 0x7f869c06e5d0 2023-10-16T11:17:41.057 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:41.057 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":199,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:40.072131+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:17:38.499055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T11:17:39.035057+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:41.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.057+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f868006b020 msgr2=0x7f868006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.057+0000 7f86a2c32700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f868006b020 0x7f868006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8690000f60 tx=0x7f8690009040 comp rx=0 tx=0).stop 2023-10-16T11:17:41.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.057+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c06e5d0 msgr2=0x7f869c0c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.057+0000 7f86a2c32700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c06e5d0 0x7f869c0c9180 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f868800c750 tx=0x7f868800cb10 comp rx=0 tx=0).stop 2023-10-16T11:17:41.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 shutdown_connections 2023-10-16T11:17:41.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f868006b020 0x7f868006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f869c0729b0 0x7f869c15de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f869c06ab40 0x7f869c0b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 --2- 172.21.15.158:0/4252629339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c06e5d0 0x7f869c0c9180 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 >> 172.21.15.158:0/4252629339 conn(0x7f869c0d02b0 msgr2=0x7f869c0add20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:41.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 shutdown_connections 2023-10-16T11:17:41.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.058+0000 7f86a2c32700 1 -- 172.21.15.158:0/4252629339 wait complete. 2023-10-16T11:17:41.072 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:41.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.191+0000 7f98d131c700 1 Processor -- start 2023-10-16T11:17:41.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.191+0000 7f98d131c700 1 -- start start 2023-10-16T11:17:41.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98d131c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98cc0d1210 0x7f98cc0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98d131c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0d3bd0 con 0x7f98cc15d7c0 2023-10-16T11:17:41.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98d131c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0d2fc0 con 0x7f98cc0ce950 2023-10-16T11:17:41.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98d131c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc0d4060 con 0x7f98cc0d1210 2023-10-16T11:17:41.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98caffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f98cc15d7c0 0x7f98cc0d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36620/0 (socket says 172.21.15.158:36620) 2023-10-16T11:17:41.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98caffd700 1 -- 172.21.15.158:0/1932987508 learned_addr learned my addr 172.21.15.158:0/1932987508 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:41.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98ca7fc700 1 --2- 172.21.15.158:0/1932987508 >> v2:172.21.15.172:3300/0 conn(0x7f98cc0d1210 0x7f98cc0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747206095 0 0) 0x7f98cc0d3bd0 con 0x7f98cc15d7c0 2023-10-16T11:17:41.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.192+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b40035b0 con 0x7f98cc15d7c0 2023-10-16T11:17:41.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843682546 0 0) 0x7f98b40035b0 con 0x7f98cc15d7c0 2023-10-16T11:17:41.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 >> v2:172.21.15.172:3300/0 conn(0x7f98cc0d1210 msgr2=0x7f98cc0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:41.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 --2- 172.21.15.158:0/1932987508 >> v2:172.21.15.172:3300/0 conn(0x7f98cc0d1210 0x7f98cc0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 >> v1:172.21.15.172:6789/0 conn(0x7f98cc0ce950 legacy=0x7f98cc0b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc0d32a0 con 0x7f98cc15d7c0 2023-10-16T11:17:41.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f98bc0019f0 con 0x7f98cc15d7c0 2023-10-16T11:17:41.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 >> v1:172.21.15.158:6789/0 conn(0x7f98cc15d7c0 legacy=0x7f98cc0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b4003a00 0x7f98b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b4004850 0x7f98b4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4005210 0x7f98b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0d4060 con 0x7f98b4003a00 2023-10-16T11:17:41.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b400c0b0 con 0x7f98b4004850 2023-10-16T11:17:41.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b400c270 con 0x7f98b4005210 2023-10-16T11:17:41.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98caffd700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b4003a00 0x7f98b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98ca7fc700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b4004850 0x7f98b4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98cb7fe700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4005210 0x7f98b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98caffd700 1 -- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4005210 msgr2=0x7f98b400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:41.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98caffd700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4005210 0x7f98b400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98caffd700 1 -- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b4004850 msgr2=0x7f98b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:41.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98caffd700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b4004850 0x7f98b4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98caffd700 1 -- 172.21.15.158:0/1932987508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40035b0 con 0x7f98b4003a00 2023-10-16T11:17:41.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98cb7fe700 1 -- 172.21.15.158:0/1932987508 reap_dead start 2023-10-16T11:17:41.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98caffd700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b4003a00 0x7f98b4003e20 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00a830 tx=0x7f98b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8413b1de41b85731 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:41.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00a2f0 con 0x7f98b4003a00 2023-10-16T11:17:41.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc012620 con 0x7f98b4003a00 2023-10-16T11:17:41.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.193+0000 7f98c9ffb700 1 -- 172.21.15.158:0/1932987508 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0129d0 con 0x7f98b4003a00 2023-10-16T11:17:41.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.194+0000 7f98d131c700 1 -- 172.21.15.158:0/1932987508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b4003a00 msgr2=0x7f98b4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.194+0000 7f98d131c700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b4003a00 0x7f98b4003e20 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00a830 tx=0x7f98b4005880 comp rx=0 tx=0).stop 2023-10-16T11:17:41.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.194+0000 7f98d131c700 1 -- 172.21.15.158:0/1932987508 shutdown_connections 2023-10-16T11:17:41.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.194+0000 7f98d131c700 1 --2- 172.21.15.158:0/1932987508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b4003a00 0x7f98b4003e20 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.194+0000 7f98d131c700 1 -- 172.21.15.158:0/1932987508 >> 172.21.15.158:0/1932987508 conn(0x7f98cc0669f0 msgr2=0x7f98cc066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:41.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.194+0000 7f98d131c700 1 -- 172.21.15.158:0/1932987508 shutdown_connections 2023-10-16T11:17:41.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.194+0000 7f98d131c700 1 -- 172.21.15.158:0/1932987508 wait complete. 2023-10-16T11:17:41.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.195+0000 7f98d131c700 1 Processor -- start 2023-10-16T11:17:41.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.195+0000 7f98d131c700 1 -- start start 2023-10-16T11:17:41.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98d131c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4003a00 0x7f98cc1f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98d131c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98cc0c7d20 0x7f98cc151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98d131c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cae10 0x7f98cc0cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98d131c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0b3180 con 0x7f98cc0cae10 2023-10-16T11:17:41.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98d131c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0cb7d0 con 0x7f98cc0c7d20 2023-10-16T11:17:41.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98d131c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0c82e0 con 0x7f98b4003a00 2023-10-16T11:17:41.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cae10 0x7f98cc0cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cae10 0x7f98cc0cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36460/0 (socket says 172.21.15.158:36460) 2023-10-16T11:17:41.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 -- 172.21.15.158:0/1121037017 learned_addr learned my addr 172.21.15.158:0/1121037017 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:41.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98caffd700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4003a00 0x7f98cc1f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98ca7fc700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98cc0c7d20 0x7f98cc151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 -- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4003a00 msgr2=0x7f98cc1f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4003a00 0x7f98cc1f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 -- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98cc0c7d20 msgr2=0x7f98cc151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98cc0c7d20 0x7f98cc151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98cb7fe700 1 -- 172.21.15.158:0/1121037017 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc0c85c0 con 0x7f98cc0cae10 2023-10-16T11:17:41.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.196+0000 7f98ca7fc700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98cc0c7d20 0x7f98cc151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:41.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.197+0000 7f98cb7fe700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cae10 0x7f98cc0cb290 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000e60 tx=0x7f98c4001ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:41.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.197+0000 7f98b37fe700 1 -- 172.21.15.158:0/1121037017 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c400ed90 con 0x7f98cc0cae10 2023-10-16T11:17:41.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.197+0000 7f98b37fe700 1 -- 172.21.15.158:0/1121037017 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c40050d0 con 0x7f98cc0cae10 2023-10-16T11:17:41.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.197+0000 7f98b37fe700 1 -- 172.21.15.158:0/1121037017 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4005460 con 0x7f98cc0cae10 2023-10-16T11:17:41.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.197+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc0c88a0 con 0x7f98cc0cae10 2023-10-16T11:17:41.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.197+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc1f2c70 con 0x7f98cc0cae10 2023-10-16T11:17:41.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.198+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc0b32c0 con 0x7f98cc0cae10 2023-10-16T11:17:41.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.198+0000 7f98b37fe700 1 -- 172.21.15.158:0/1121037017 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f98c40036d0 con 0x7f98cc0cae10 2023-10-16T11:17:41.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.199+0000 7f98b37fe700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f98b806b050 0x7f98b806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.199+0000 7f98b37fe700 1 -- 172.21.15.158:0/1121037017 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f98c40888f0 con 0x7f98cc0cae10 2023-10-16T11:17:41.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.199+0000 7f98caffd700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f98b806b050 0x7f98b806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.199+0000 7f98caffd700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f98b806b050 0x7f98b806d510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f98bc000f80 tx=0x7f98bc007000 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:41.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.203+0000 7f98b37fe700 1 -- 172.21.15.158:0/1121037017 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98cc0b32c0 con 0x7f98cc0cae10 2023-10-16T11:17:41.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.395+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98cc0b32c0 con 0x7f98cc0cae10 2023-10-16T11:17:41.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.397+0000 7f98b37fe700 1 -- 172.21.15.158:0/1121037017 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+17435 (secure 0 0 0) 0x7f98cc0b32c0 con 0x7f98cc0cae10 2023-10-16T11:17:41.399 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:41.399 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":199,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:40.072131+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:17:38.499055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T11:17:39.035057+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:41.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f98b806b050 msgr2=0x7f98b806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f98b806b050 0x7f98b806d510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f98bc000f80 tx=0x7f98bc007000 comp rx=0 tx=0).stop 2023-10-16T11:17:41.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cae10 msgr2=0x7f98cc0cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cae10 0x7f98cc0cb290 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000e60 tx=0x7f98c4001ec0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 shutdown_connections 2023-10-16T11:17:41.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cae10 0x7f98cc0cb290 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b4003a00 0x7f98cc1f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f98b806b050 0x7f98b806d510 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 --2- 172.21.15.158:0/1121037017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98cc0c7d20 0x7f98cc151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 >> 172.21.15.158:0/1121037017 conn(0x7f98cc0669f0 msgr2=0x7f98cc0b8810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:41.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 shutdown_connections 2023-10-16T11:17:41.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.399+0000 7f98d131c700 1 -- 172.21.15.158:0/1121037017 wait complete. 2023-10-16T11:17:41.413 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T11:17:41.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.535+0000 7fdf6a4fc700 1 Processor -- start 2023-10-16T11:17:41.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.535+0000 7fdf6a4fc700 1 -- start start 2023-10-16T11:17:41.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf6a4fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf64071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf6a4fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdf640cc9e0 0x7fdf64071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf6a4fc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6406a8d0 con 0x7fdf6415d910 2023-10-16T11:17:41.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf6a4fc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf64161e50 con 0x7fdf640cc9e0 2023-10-16T11:17:41.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf6a4fc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6415dcf0 con 0x7fdf640ce680 2023-10-16T11:17:41.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf63fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdf6415d910 0x7fdf640cdf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36640/0 (socket says 172.21.15.158:36640) 2023-10-16T11:17:41.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf63fff700 1 -- 172.21.15.158:0/2257277509 learned_addr learned my addr 172.21.15.158:0/2257277509 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:41.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf637fe700 1 --2- 172.21.15.158:0/2257277509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf64071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf68a99700 1 --2- 172.21.15.158:0/2257277509 >> v2:172.21.15.172:3300/0 conn(0x7fdf640cc9e0 0x7fdf64071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf62ffd700 1 -- 172.21.15.158:0/2257277509 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818949815 0 0) 0x7fdf6406a8d0 con 0x7fdf6415d910 2023-10-16T11:17:41.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf62ffd700 1 -- 172.21.15.158:0/2257277509 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c0035b0 con 0x7fdf6415d910 2023-10-16T11:17:41.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf637fe700 1 -- 172.21.15.158:0/2257277509 >> v2:172.21.15.172:3300/0 conn(0x7fdf640cc9e0 msgr2=0x7fdf64071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:41.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf637fe700 1 --2- 172.21.15.158:0/2257277509 >> v2:172.21.15.172:3300/0 conn(0x7fdf640cc9e0 0x7fdf64071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf637fe700 1 -- 172.21.15.158:0/2257277509 >> v1:172.21.15.158:6789/0 conn(0x7fdf6415d910 legacy=0x7fdf640cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf62ffd700 1 -- 172.21.15.158:0/2257277509 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 217074113 0 0) 0x7fdf4c0035b0 con 0x7fdf6415d910 2023-10-16T11:17:41.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.536+0000 7fdf637fe700 1 -- 172.21.15.158:0/2257277509 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf64161520 con 0x7fdf640ce680 2023-10-16T11:17:41.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf637fe700 1 --2- 172.21.15.158:0/2257277509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf64071960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009840 tx=0x7fdf54009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c132fd9a7086732d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:41.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf62ffd700 1 -- 172.21.15.158:0/2257277509 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54011830 con 0x7fdf640ce680 2023-10-16T11:17:41.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf62ffd700 1 -- 172.21.15.158:0/2257277509 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf54011990 con 0x7fdf640ce680 2023-10-16T11:17:41.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf62ffd700 1 -- 172.21.15.158:0/2257277509 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54011d20 con 0x7fdf640ce680 2023-10-16T11:17:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/2257277509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 msgr2=0x7fdf64071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/2257277509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf64071960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009840 tx=0x7fdf54009b50 comp rx=0 tx=0).stop 2023-10-16T11:17:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/2257277509 shutdown_connections 2023-10-16T11:17:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/2257277509 >> v2:172.21.15.172:3300/0 conn(0x7fdf640cc9e0 0x7fdf64071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/2257277509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf64071960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/2257277509 >> 172.21.15.158:0/2257277509 conn(0x7fdf640b1e90 msgr2=0x7fdf640b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:41.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/2257277509 shutdown_connections 2023-10-16T11:17:41.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.537+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/2257277509 wait complete. 2023-10-16T11:17:41.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 Processor -- start 2023-10-16T11:17:41.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 -- start start 2023-10-16T11:17:41.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf640cc9e0 0x7fdf640c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf640b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf6415d910 0x7fdf640b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640ca430 con 0x7fdf640cc9e0 2023-10-16T11:17:41.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640b3af0 con 0x7fdf6415d910 2023-10-16T11:17:41.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.539+0000 7fdf6a4fc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640b3dd0 con 0x7fdf640ce680 2023-10-16T11:17:41.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf640cc9e0 0x7fdf640c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf640cc9e0 0x7fdf640c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36470/0 (socket says 172.21.15.158:36470) 2023-10-16T11:17:41.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 -- 172.21.15.158:0/4113491608 learned_addr learned my addr 172.21.15.158:0/4113491608 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:41.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf637fe700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf640b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf68a99700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf6415d910 0x7fdf640b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 -- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 msgr2=0x7fdf640b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf640b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 -- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf6415d910 msgr2=0x7fdf640b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:41.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf6415d910 0x7fdf640b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:41.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 -- 172.21.15.158:0/4113491608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540094a0 con 0x7fdf640cc9e0 2023-10-16T11:17:41.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf63fff700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf640cc9e0 0x7fdf640c9ef0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c4d0 tx=0x7fdf5800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:41.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf60ff9700 1 -- 172.21.15.158:0/4113491608 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58005d70 con 0x7fdf640cc9e0 2023-10-16T11:17:41.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf60ff9700 1 -- 172.21.15.158:0/4113491608 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf580070d0 con 0x7fdf640cc9e0 2023-10-16T11:17:41.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf640b40b0 con 0x7fdf640cc9e0 2023-10-16T11:17:41.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf60ff9700 1 -- 172.21.15.158:0/4113491608 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58007460 con 0x7fdf640cc9e0 2023-10-16T11:17:41.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.540+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf640b4670 con 0x7fdf640cc9e0 2023-10-16T11:17:41.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.541+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf640c7ec0 con 0x7fdf640cc9e0 2023-10-16T11:17:41.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.542+0000 7fdf60ff9700 1 -- 172.21.15.158:0/4113491608 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf580075c0 con 0x7fdf640cc9e0 2023-10-16T11:17:41.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.542+0000 7fdf60ff9700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fdf5006b020 0x7fdf5006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:41.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.542+0000 7fdf60ff9700 1 -- 172.21.15.158:0/4113491608 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fdf58088de0 con 0x7fdf640cc9e0 2023-10-16T11:17:41.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.542+0000 7fdf637fe700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fdf5006b020 0x7fdf5006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:41.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.543+0000 7fdf637fe700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fdf5006b020 0x7fdf5006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009fd0 tx=0x7fdf5401c040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:41.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.546+0000 7fdf60ff9700 1 -- 172.21.15.158:0/4113491608 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf640c7ec0 con 0x7fdf640cc9e0 2023-10-16T11:17:41.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:41.789+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdf640c7ec0 con 0x7fdf640cc9e0 2023-10-16T11:17:42.666 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T11:17:42.664+0000 7fc5853c7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:17:42.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.705+0000 7fdf60ff9700 1 -- 172.21.15.158:0/4113491608 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v102) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdf640c7ec0 con 0x7fdf640cc9e0 2023-10-16T11:17:42.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fdf5006b020 msgr2=0x7fdf5006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:42.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fdf5006b020 0x7fdf5006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009fd0 tx=0x7fdf5401c040 comp rx=0 tx=0).stop 2023-10-16T11:17:42.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf640cc9e0 msgr2=0x7fdf640c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:42.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf640cc9e0 0x7fdf640c9ef0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c4d0 tx=0x7fdf5800c7e0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 shutdown_connections 2023-10-16T11:17:42.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fdf5006b020 0x7fdf5006d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf6415d910 0x7fdf640b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf640cc9e0 0x7fdf640c9ef0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 --2- 172.21.15.158:0/4113491608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf640ce680 0x7fdf640b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 >> 172.21.15.158:0/4113491608 conn(0x7fdf640b1e90 msgr2=0x7fdf6406adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:42.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 shutdown_connections 2023-10-16T11:17:42.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.708+0000 7fdf6a4fc700 1 -- 172.21.15.158:0/4113491608 wait complete. 2023-10-16T11:17:42.725 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:17:42.725 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T11:17:42.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.853+0000 7f98a81fb700 1 Processor -- start 2023-10-16T11:17:42.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.853+0000 7f98a81fb700 1 -- start start 2023-10-16T11:17:42.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.854+0000 7f98a81fb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98a0163380 0x7f98a0165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.854+0000 7f98a81fb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00d0ae0 con 0x7f98a015dad0 2023-10-16T11:17:42.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.854+0000 7f98a81fb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00d12d0 con 0x7f98a015deb0 2023-10-16T11:17:42.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.854+0000 7f98a81fb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a00cbdf0 con 0x7f98a0163380 2023-10-16T11:17:42.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a5f97700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f98a015dad0 0x7f98a00cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36654/0 (socket says 172.21.15.158:36654) 2023-10-16T11:17:42.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a5f97700 1 -- 172.21.15.158:0/2898734929 learned_addr learned my addr 172.21.15.158:0/2898734929 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:42.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a6798700 1 --2- 172.21.15.158:0/2898734929 >> v2:172.21.15.172:3300/0 conn(0x7f98a0163380 0x7f98a0165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632437529 0 0) 0x7f98a00d0ae0 con 0x7f98a015dad0 2023-10-16T11:17:42.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c0035b0 con 0x7f98a015dad0 2023-10-16T11:17:42.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922759979 0 0) 0x7f988c0035b0 con 0x7f98a015dad0 2023-10-16T11:17:42.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 >> v2:172.21.15.172:3300/0 conn(0x7f98a0163380 msgr2=0x7f98a0165770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:42.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 --2- 172.21.15.158:0/2898734929 >> v2:172.21.15.172:3300/0 conn(0x7f98a0163380 0x7f98a0165770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 >> v1:172.21.15.172:6789/0 conn(0x7f98a015deb0 legacy=0x7f98a00ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:42.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00cc280 con 0x7f98a015dad0 2023-10-16T11:17:42.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f9890001670 con 0x7f98a015dad0 2023-10-16T11:17:42.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 >> v1:172.21.15.158:6789/0 conn(0x7f98a015dad0 legacy=0x7f98a00cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:42.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f988c003a00 0x7f988c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f988c004850 0x7f988c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c005330 0x7f988c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00cbdf0 con 0x7f988c003a00 2023-10-16T11:17:42.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c00c200 con 0x7f988c005330 2023-10-16T11:17:42.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c00e5d0 con 0x7f988c004850 2023-10-16T11:17:42.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a5f97700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f988c003a00 0x7f988c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a6798700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c005330 0x7f988c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.855+0000 7f98a5796700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f988c004850 0x7f988c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a5f97700 1 -- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f988c004850 msgr2=0x7f988c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:42.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a5f97700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f988c004850 0x7f988c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a5f97700 1 -- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c005330 msgr2=0x7f988c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:42.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a5f97700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c005330 0x7f988c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a5f97700 1 -- 172.21.15.158:0/2898734929 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0035b0 con 0x7f988c003a00 2023-10-16T11:17:42.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a6798700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c005330 0x7f988c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:42.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a6798700 1 -- 172.21.15.158:0/2898734929 reap_dead start 2023-10-16T11:17:42.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a5f97700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f988c003a00 0x7f988c003e20 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f989000c210 tx=0x7f988c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499ca2ca6e960024 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:42.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890007c10 con 0x7f988c003a00 2023-10-16T11:17:42.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989000cdb0 con 0x7f988c003a00 2023-10-16T11:17:42.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a4f95700 1 -- 172.21.15.158:0/2898734929 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890005ba0 con 0x7f988c003a00 2023-10-16T11:17:42.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a81fb700 1 -- 172.21.15.158:0/2898734929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f988c003a00 msgr2=0x7f988c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:42.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a81fb700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f988c003a00 0x7f988c003e20 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f989000c210 tx=0x7f988c005880 comp rx=0 tx=0).stop 2023-10-16T11:17:42.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a81fb700 1 -- 172.21.15.158:0/2898734929 shutdown_connections 2023-10-16T11:17:42.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a81fb700 1 --2- 172.21.15.158:0/2898734929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f988c003a00 0x7f988c003e20 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a81fb700 1 -- 172.21.15.158:0/2898734929 >> 172.21.15.158:0/2898734929 conn(0x7f98a0099a10 msgr2=0x7f98a00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:42.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a81fb700 1 -- 172.21.15.158:0/2898734929 shutdown_connections 2023-10-16T11:17:42.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.856+0000 7f98a81fb700 1 -- 172.21.15.158:0/2898734929 wait complete. 2023-10-16T11:17:42.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 Processor -- start 2023-10-16T11:17:42.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 -- start start 2023-10-16T11:17:42.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c003a00 0x7f98a015bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98a00b4270 0x7f98a014d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98a015a320 0x7f98a00b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c74a0 con 0x7f98a015a320 2023-10-16T11:17:42.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a015ab50 con 0x7f988c003a00 2023-10-16T11:17:42.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a81fb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a015ae30 con 0x7f98a00b4270 2023-10-16T11:17:42.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a6798700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98a015a320 0x7f98a00b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a6798700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98a015a320 0x7f98a00b4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36494/0 (socket says 172.21.15.158:36494) 2023-10-16T11:17:42.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a6798700 1 -- 172.21.15.158:0/3641712470 learned_addr learned my addr 172.21.15.158:0/3641712470 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:42.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a5f97700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c003a00 0x7f98a015bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.858+0000 7f98a5796700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98a00b4270 0x7f98a014d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a6798700 1 -- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98a00b4270 msgr2=0x7f98a014d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:42.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a6798700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98a00b4270 0x7f98a014d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a6798700 1 -- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c003a00 msgr2=0x7f98a015bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:42.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a6798700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c003a00 0x7f98a015bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:42.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a6798700 1 -- 172.21.15.158:0/3641712470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a015b110 con 0x7f98a015a320 2023-10-16T11:17:42.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a6798700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98a015a320 0x7f98a00b4c40 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f98a00cbd60 tx=0x7f98980101c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:42.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98967fc700 1 -- 172.21.15.158:0/3641712470 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989801b070 con 0x7f98a015a320 2023-10-16T11:17:42.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98967fc700 1 -- 172.21.15.158:0/3641712470 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989800f040 con 0x7f98a015a320 2023-10-16T11:17:42.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a015b3f0 con 0x7f98a015a320 2023-10-16T11:17:42.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98967fc700 1 -- 172.21.15.158:0/3641712470 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800e050 con 0x7f98a015a320 2023-10-16T11:17:42.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.859+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a014d700 con 0x7f98a015a320 2023-10-16T11:17:42.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.860+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a015b910 con 0x7f98a015a320 2023-10-16T11:17:42.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.861+0000 7f98967fc700 1 -- 172.21.15.158:0/3641712470 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9898010a60 con 0x7f98a015a320 2023-10-16T11:17:42.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.861+0000 7f98967fc700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f988406b020 0x7f988406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:42.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.861+0000 7f98967fc700 1 -- 172.21.15.158:0/3641712470 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f9898089c40 con 0x7f98a015a320 2023-10-16T11:17:42.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.861+0000 7f98a5f97700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f988406b020 0x7f988406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:42.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.861+0000 7f98a5f97700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f988406b020 0x7f988406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9890000c00 tx=0x7f9890007000 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:42.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:42.865+0000 7f98967fc700 1 -- 172.21.15.158:0/3641712470 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98a015b910 con 0x7f98a015a320 2023-10-16T11:17:43.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.104+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f98a015b910 con 0x7f98a015a320 2023-10-16T11:17:43.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.713+0000 7f98967fc700 1 -- 172.21.15.158:0/3641712470 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v103) v1 ==== 122+0+0 (secure 0 0 0) 0x7f98a015b910 con 0x7f98a015a320 2023-10-16T11:17:43.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f988406b020 msgr2=0x7f988406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:43.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f988406b020 0x7f988406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9890000c00 tx=0x7f9890007000 comp rx=0 tx=0).stop 2023-10-16T11:17:43.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98a015a320 msgr2=0x7f98a00b4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:43.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98a015a320 0x7f98a00b4c40 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f98a00cbd60 tx=0x7f98980101c0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 shutdown_connections 2023-10-16T11:17:43.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98a015a320 0x7f98a00b4c40 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f988406b020 0x7f988406d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f988c003a00 0x7f98a015bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 --2- 172.21.15.158:0/3641712470 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98a00b4270 0x7f98a014d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 >> 172.21.15.158:0/3641712470 conn(0x7f98a0099a10 msgr2=0x7f98a00cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:43.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 shutdown_connections 2023-10-16T11:17:43.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.716+0000 7f98a81fb700 1 -- 172.21.15.158:0/3641712470 wait complete. 2023-10-16T11:17:43.733 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T11:17:43.733 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:17:43.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.858+0000 7f3403cd0700 1 Processor -- start 2023-10-16T11:17:43.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.858+0000 7f3403cd0700 1 -- start start 2023-10-16T11:17:43.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f3403cd0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33fc06e5d0 0x7f33fc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:43.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f3403cd0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f33fc0729b0 0x7f33fc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:43.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f3403cd0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc071e50 con 0x7f33fc06ab40 2023-10-16T11:17:43.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f3403cd0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33fc161270 con 0x7f33fc06e5d0 2023-10-16T11:17:43.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f3403cd0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33fc161510 con 0x7f33fc0729b0 2023-10-16T11:17:43.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33fc06e5d0 0x7f33fc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:43.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33fc06e5d0 0x7f33fc06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36500/0 (socket says 172.21.15.158:36500) 2023-10-16T11:17:43.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 -- 172.21.15.158:0/4226905567 learned_addr learned my addr 172.21.15.158:0/4226905567 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:43.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340226d700 1 --2- 172.21.15.158:0/4226905567 >> v2:172.21.15.172:3300/0 conn(0x7f33fc0729b0 0x7f33fc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:43.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 -- 172.21.15.158:0/4226905567 >> v2:172.21.15.172:3300/0 conn(0x7f33fc0729b0 msgr2=0x7f33fc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:43.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 --2- 172.21.15.158:0/4226905567 >> v2:172.21.15.172:3300/0 conn(0x7f33fc0729b0 0x7f33fc071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 -- 172.21.15.158:0/4226905567 >> v1:172.21.15.172:6789/0 conn(0x7f33fc06ab40 legacy=0x7f33fc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:43.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 -- 172.21.15.158:0/4226905567 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc1617f0 con 0x7f33fc06e5d0 2023-10-16T11:17:43.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.859+0000 7f340126b700 1 --2- 172.21.15.158:0/4226905567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc06e5d0 0x7f33fc06d820 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f33fc158240 tx=0x7f33ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf04c42547c720f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:43.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3400a6a700 1 -- 172.21.15.158:0/4226905567 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec004230 con 0x7f33fc06e5d0 2023-10-16T11:17:43.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3400a6a700 1 -- 172.21.15.158:0/4226905567 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec004390 con 0x7f33fc06e5d0 2023-10-16T11:17:43.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3400a6a700 1 -- 172.21.15.158:0/4226905567 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0036a0 con 0x7f33fc06e5d0 2023-10-16T11:17:43.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 -- 172.21.15.158:0/4226905567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc06e5d0 msgr2=0x7f33fc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:43.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 --2- 172.21.15.158:0/4226905567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc06e5d0 0x7f33fc06d820 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f33fc158240 tx=0x7f33ec009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 -- 172.21.15.158:0/4226905567 shutdown_connections 2023-10-16T11:17:43.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 --2- 172.21.15.158:0/4226905567 >> v2:172.21.15.172:3300/0 conn(0x7f33fc0729b0 0x7f33fc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 --2- 172.21.15.158:0/4226905567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc06e5d0 0x7f33fc06d820 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 -- 172.21.15.158:0/4226905567 >> 172.21.15.158:0/4226905567 conn(0x7f33fc0d02b0 msgr2=0x7f33fc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:43.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 -- 172.21.15.158:0/4226905567 shutdown_connections 2023-10-16T11:17:43.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.860+0000 7f3403cd0700 1 -- 172.21.15.158:0/4226905567 wait complete. 2023-10-16T11:17:43.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.862+0000 7f3403cd0700 1 Processor -- start 2023-10-16T11:17:43.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.862+0000 7f3403cd0700 1 -- start start 2023-10-16T11:17:43.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33fc06ab40 0x7f33fc1f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:43.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33fc06e5d0 0x7f33fc160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:43.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc0729b0 0x7f33fc0c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:43.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc160d60 con 0x7f33fc0729b0 2023-10-16T11:17:43.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc15ef10 con 0x7f33fc06e5d0 2023-10-16T11:17:43.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc1f3680 con 0x7f33fc06ab40 2023-10-16T11:17:43.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc0729b0 0x7f33fc0c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:43.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc0729b0 0x7f33fc0c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36508/0 (socket says 172.21.15.158:36508) 2023-10-16T11:17:43.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 -- 172.21.15.158:0/1088634150 learned_addr learned my addr 172.21.15.158:0/1088634150 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:43.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340126b700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33fc06e5d0 0x7f33fc160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:43.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3401a6c700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33fc06ab40 0x7f33fc1f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:43.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 -- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33fc06ab40 msgr2=0x7f33fc1f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:43.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33fc06ab40 0x7f33fc1f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 -- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33fc06e5d0 msgr2=0x7f33fc160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:43.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33fc06e5d0 0x7f33fc160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:43.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 -- 172.21.15.158:0/1088634150 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec009550 con 0x7f33fc0729b0 2023-10-16T11:17:43.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340126b700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33fc06e5d0 0x7f33fc160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:43.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3401a6c700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33fc06ab40 0x7f33fc1f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:43.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f340226d700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc0729b0 0x7f33fc0c6250 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f33f4007e30 tx=0x7f33f400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:43.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f33f27fc700 1 -- 172.21.15.158:0/1088634150 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f400eea0 con 0x7f33fc0729b0 2023-10-16T11:17:43.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f33f27fc700 1 -- 172.21.15.158:0/1088634150 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f4014420 con 0x7f33fc0729b0 2023-10-16T11:17:43.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc1f3960 con 0x7f33fc0729b0 2023-10-16T11:17:43.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f33f27fc700 1 -- 172.21.15.158:0/1088634150 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f40147d0 con 0x7f33fc0729b0 2023-10-16T11:17:43.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.863+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc0c6790 con 0x7f33fc0729b0 2023-10-16T11:17:43.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.864+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc15f0e0 con 0x7f33fc0729b0 2023-10-16T11:17:43.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.865+0000 7f33f27fc700 1 -- 172.21.15.158:0/1088634150 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f33f4014930 con 0x7f33fc0729b0 2023-10-16T11:17:43.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.865+0000 7f33f27fc700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f33e006b020 0x7f33e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:43.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.865+0000 7f33f27fc700 1 -- 172.21.15.158:0/1088634150 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f33f408a280 con 0x7f33fc0729b0 2023-10-16T11:17:43.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.865+0000 7f3401a6c700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f33e006b020 0x7f33e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:43.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.866+0000 7f3401a6c700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f33e006b020 0x7f33e006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f33f8006fd0 tx=0x7f33f8008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:43.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:43.870+0000 7f33f27fc700 1 -- 172.21.15.158:0/1088634150 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33f4054f10 con 0x7f33fc0729b0 2023-10-16T11:17:44.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.104+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33fc161d80 con 0x7f33fc0729b0 2023-10-16T11:17:44.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.104+0000 7f33f27fc700 1 -- 172.21.15.158:0/1088634150 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f33f4058530 con 0x7f33fc0729b0 2023-10-16T11:17:44.106 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:44.107 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":103,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2912297853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2912297853},{"type":"v1","addr":"172.21.15.158:6836","nonce":2912297853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":38441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2297507228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2297507228},{"type":"v1","addr":"172.21.15.158:6837","nonce":2297507228}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38447,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3795197457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3795197457},{"type":"v1","addr":"172.21.15.172:6834","nonce":3795197457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:17:40.072523+0000","modified":"2023-10-16T11:17:43.106178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38450":{"gid":38450,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1279760034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1279760034},{"type":"v1","addr":"172.21.15.172:6835","nonce":1279760034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T11:17:44.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.106+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f33e006b020 msgr2=0x7f33e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.106+0000 7f3403cd0700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f33e006b020 0x7f33e006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f33f8006fd0 tx=0x7f33f8008040 comp rx=0 tx=0).stop 2023-10-16T11:17:44.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.106+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc0729b0 msgr2=0x7f33fc0c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.106+0000 7f3403cd0700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc0729b0 0x7f33fc0c6250 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f33f4007e30 tx=0x7f33f400e3c0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 shutdown_connections 2023-10-16T11:17:44.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33fc0729b0 0x7f33fc0c6250 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33fc06ab40 0x7f33fc1f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f33e006b020 0x7f33e006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 --2- 172.21.15.158:0/1088634150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33fc06e5d0 0x7f33fc160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 >> 172.21.15.158:0/1088634150 conn(0x7f33fc0d02b0 msgr2=0x7f33fc06d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:44.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 shutdown_connections 2023-10-16T11:17:44.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.107+0000 7f3403cd0700 1 -- 172.21.15.158:0/1088634150 wait complete. 2023-10-16T11:17:44.111 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 103 2023-10-16T11:17:44.120 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:17:44.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.239+0000 7f50ae709700 1 Processor -- start 2023-10-16T11:17:44.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.240+0000 7f50ae709700 1 -- start start 2023-10-16T11:17:44.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50ae709700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 0x7f50a806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50ae709700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a806e770 con 0x7f50a806f340 2023-10-16T11:17:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50ae709700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a80cbe00 con 0x7f50a815dfb0 2023-10-16T11:17:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50ae709700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80cf270 con 0x7f50a80cc5d0 2023-10-16T11:17:44.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50acca6700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f50a806f340 0x7f50a8071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36692/0 (socket says 172.21.15.158:36692) 2023-10-16T11:17:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50acca6700 1 -- 172.21.15.158:0/1252673792 learned_addr learned my addr 172.21.15.158:0/1252673792 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a7fff700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 0x7f50a806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504787221 0 0) 0x7f50a0001490 con 0x7f50a806f340 2023-10-16T11:17:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50900035b0 con 0x7f50a806f340 2023-10-16T11:17:44.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473265301 0 0) 0x7f50a80cbe00 con 0x7f50a815dfb0 2023-10-16T11:17:44.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a0001490 con 0x7f50a815dfb0 2023-10-16T11:17:44.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866129630 0 0) 0x7f50900035b0 con 0x7f50a806f340 2023-10-16T11:17:44.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 msgr2=0x7f50a806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 0x7f50a806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 >> v1:172.21.15.172:6789/0 conn(0x7f50a815dfb0 legacy=0x7f50a80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a815e4e0 con 0x7f50a806f340 2023-10-16T11:17:44.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f50a0001b80 con 0x7f50a806f340 2023-10-16T11:17:44.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a7fff700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 0x7f50a806e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:44.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 >> v1:172.21.15.158:6789/0 conn(0x7f50a806f340 legacy=0x7f50a8071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f5090003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5090004850 0x7f5090004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.241+0000 7f50a6ffd700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 0x7f5090005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80cf270 con 0x7f5090003bc0 2023-10-16T11:17:44.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509000ba40 con 0x7f5090004850 2023-10-16T11:17:44.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509000bc00 con 0x7f50a80cc5d0 2023-10-16T11:17:44.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a7fff700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f5090003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a77fe700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5090004850 0x7f5090004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50acca6700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 0x7f5090005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a7fff700 1 -- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 msgr2=0x7f5090005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a7fff700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80cc5d0 0x7f5090005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a7fff700 1 -- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5090004850 msgr2=0x7f5090004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a7fff700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5090004850 0x7f5090004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a7fff700 1 -- 172.21.15.158:0/1252673792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50980094a0 con 0x7f5090003bc0 2023-10-16T11:17:44.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a77fe700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5090004850 0x7f5090004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:44.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a7fff700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f5090003fc0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f5098009d00 tx=0x7f5090005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde690e400cdf199 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098009e50 con 0x7f5090003bc0 2023-10-16T11:17:44.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509800e9a0 con 0x7f5090003bc0 2023-10-16T11:17:44.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.242+0000 7f50a6ffd700 1 -- 172.21.15.158:0/1252673792 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509800ed30 con 0x7f5090003bc0 2023-10-16T11:17:44.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.243+0000 7f50ae709700 1 -- 172.21.15.158:0/1252673792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 msgr2=0x7f5090003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.243+0000 7f50ae709700 1 --2- 172.21.15.158:0/1252673792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f5090003fc0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f5098009d00 tx=0x7f5090005880 comp rx=0 tx=0).stop 2023-10-16T11:17:44.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.243+0000 7f50acca6700 1 -- 172.21.15.158:0/1252673792 reap_dead start 2023-10-16T11:17:44.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.243+0000 7f50ae709700 1 -- 172.21.15.158:0/1252673792 shutdown_connections 2023-10-16T11:17:44.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.243+0000 7f50ae709700 1 -- 172.21.15.158:0/1252673792 >> 172.21.15.158:0/1252673792 conn(0x7f50a80d02b0 msgr2=0x7f50a80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:44.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.243+0000 7f50ae709700 1 -- 172.21.15.158:0/1252673792 shutdown_connections 2023-10-16T11:17:44.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.243+0000 7f50ae709700 1 -- 172.21.15.158:0/1252673792 wait complete. 2023-10-16T11:17:44.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.244+0000 7f50ae709700 1 Processor -- start 2023-10-16T11:17:44.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.244+0000 7f50ae709700 1 -- start start 2023-10-16T11:17:44.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50ae709700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f50a80c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50ae709700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80ad2a0 0x7f50a81562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50ae709700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50a806a8d0 0x7f50a806ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50ae709700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a806b2c0 con 0x7f5090003bc0 2023-10-16T11:17:44.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50ae709700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80aeae0 con 0x7f50a806a8d0 2023-10-16T11:17:44.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50ae709700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80ada40 con 0x7f50a80ad2a0 2023-10-16T11:17:44.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f50a80c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f50a80c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36524/0 (socket says 172.21.15.158:36524) 2023-10-16T11:17:44.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 -- 172.21.15.158:0/2024256901 learned_addr learned my addr 172.21.15.158:0/2024256901 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:44.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a77fe700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80ad2a0 0x7f50a81562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50acca6700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50a806a8d0 0x7f50a806ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 -- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80ad2a0 msgr2=0x7f50a81562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80ad2a0 0x7f50a81562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 -- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50a806a8d0 msgr2=0x7f50a806ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50a806a8d0 0x7f50a806ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 -- 172.21.15.158:0/2024256901 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a80add20 con 0x7f5090003bc0 2023-10-16T11:17:44.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50acca6700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50a806a8d0 0x7f50a806ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:44.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.245+0000 7f50a7fff700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f50a80c9470 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f509000ba10 tx=0x7f50980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.246+0000 7f50a4ff9700 1 -- 172.21.15.158:0/2024256901 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098021070 con 0x7f5090003bc0 2023-10-16T11:17:44.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.246+0000 7f50a4ff9700 1 -- 172.21.15.158:0/2024256901 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5098009e50 con 0x7f5090003bc0 2023-10-16T11:17:44.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.246+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a80ae000 con 0x7f5090003bc0 2023-10-16T11:17:44.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.246+0000 7f50a4ff9700 1 -- 172.21.15.158:0/2024256901 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50980038e0 con 0x7f5090003bc0 2023-10-16T11:17:44.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.246+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a80ae5c0 con 0x7f5090003bc0 2023-10-16T11:17:44.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.247+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a80ae780 con 0x7f5090003bc0 2023-10-16T11:17:44.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.247+0000 7f50a4ff9700 1 -- 172.21.15.158:0/2024256901 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5098025070 con 0x7f5090003bc0 2023-10-16T11:17:44.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.247+0000 7f50a4ff9700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f509406b020 0x7f509406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.247+0000 7f50a4ff9700 1 -- 172.21.15.158:0/2024256901 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f509808ad10 con 0x7f5090003bc0 2023-10-16T11:17:44.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.247+0000 7f50a77fe700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f509406b020 0x7f509406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.248+0000 7f50a77fe700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f509406b020 0x7f509406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f509c000f60 tx=0x7f509c009040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.252+0000 7f50a4ff9700 1 -- 172.21.15.158:0/2024256901 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5098055a30 con 0x7f5090003bc0 2023-10-16T11:17:44.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.444+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50a80681b0 con 0x7f5090003bc0 2023-10-16T11:17:44.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.445+0000 7f50a4ff9700 1 -- 172.21.15.158:0/2024256901 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) v1 ==== 74+0+17422 (secure 0 0 0) 0x7f5098017090 con 0x7f5090003bc0 2023-10-16T11:17:44.447 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:44.447 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":200,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:17:41.688989+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:17:38.499055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T11:17:39.035057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"200","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:17:44.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f509406b020 msgr2=0x7f509406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f509406b020 0x7f509406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f509c000f60 tx=0x7f509c009040 comp rx=0 tx=0).stop 2023-10-16T11:17:44.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 msgr2=0x7f50a80c9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f50a80c9470 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f509000ba10 tx=0x7f50980047c0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 shutdown_connections 2023-10-16T11:17:44.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f509406b020 0x7f509406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50a806a8d0 0x7f50a806ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5090003bc0 0x7f50a80c9470 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 --2- 172.21.15.158:0/2024256901 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50a80ad2a0 0x7f50a81562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 >> 172.21.15.158:0/2024256901 conn(0x7f50a80d02b0 msgr2=0x7f50a8072140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:44.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 shutdown_connections 2023-10-16T11:17:44.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.447+0000 7f50ae709700 1 -- 172.21.15.158:0/2024256901 wait complete. 2023-10-16T11:17:44.461 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:17:44.462 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:17:44.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.580+0000 7fb51d90c700 1 Processor -- start 2023-10-16T11:17:44.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.580+0000 7fb51d90c700 1 -- start start 2023-10-16T11:17:44.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb51d90c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb51815d7c0 0x7fb5180d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb51d90c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180ceb20 0x7fb5180b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb51d90c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5180d3bd0 con 0x7fb5180d1210 2023-10-16T11:17:44.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb51d90c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5180d4060 con 0x7fb51815d7c0 2023-10-16T11:17:44.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb51d90c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5180d2cf0 con 0x7fb5180ceb20 2023-10-16T11:17:44.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb51815d7c0 0x7fb5180d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb51815d7c0 0x7fb5180d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36532/0 (socket says 172.21.15.158:36532) 2023-10-16T11:17:44.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 -- 172.21.15.158:0/1913571341 learned_addr learned my addr 172.21.15.158:0/1913571341 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:44.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb5177fe700 1 --2- 172.21.15.158:0/1913571341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180ceb20 0x7fb5180b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 -- 172.21.15.158:0/1913571341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180ceb20 msgr2=0x7fb5180b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 --2- 172.21.15.158:0/1913571341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180ceb20 0x7fb5180b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 -- 172.21.15.158:0/1913571341 >> v1:172.21.15.172:6789/0 conn(0x7fb5180d1210 legacy=0x7fb5180ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 -- 172.21.15.158:0/1913571341 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5180d2fc0 con 0x7fb51815d7c0 2023-10-16T11:17:44.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb516ffd700 1 --2- 172.21.15.158:0/1913571341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180d27b0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fb5080098f0 tx=0x7fb508009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234082323345dfe9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb515ffb700 1 -- 172.21.15.158:0/1913571341 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb508011830 con 0x7fb51815d7c0 2023-10-16T11:17:44.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.581+0000 7fb515ffb700 1 -- 172.21.15.158:0/1913571341 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb508011990 con 0x7fb51815d7c0 2023-10-16T11:17:44.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb515ffb700 1 -- 172.21.15.158:0/1913571341 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb508011d20 con 0x7fb51815d7c0 2023-10-16T11:17:44.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 -- 172.21.15.158:0/1913571341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 msgr2=0x7fb5180d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 --2- 172.21.15.158:0/1913571341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180d27b0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fb5080098f0 tx=0x7fb508009c00 comp rx=0 tx=0).stop 2023-10-16T11:17:44.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 -- 172.21.15.158:0/1913571341 shutdown_connections 2023-10-16T11:17:44.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 --2- 172.21.15.158:0/1913571341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180ceb20 0x7fb5180b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 --2- 172.21.15.158:0/1913571341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180d27b0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 -- 172.21.15.158:0/1913571341 >> 172.21.15.158:0/1913571341 conn(0x7fb5180669f0 msgr2=0x7fb518066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:44.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 -- 172.21.15.158:0/1913571341 shutdown_connections 2023-10-16T11:17:44.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.582+0000 7fb51d90c700 1 -- 172.21.15.158:0/1913571341 wait complete. 2023-10-16T11:17:44.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.583+0000 7fb51d90c700 1 Processor -- start 2023-10-16T11:17:44.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.583+0000 7fb51d90c700 1 -- start start 2023-10-16T11:17:44.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb51d90c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5180ceb20 0x7fb5180b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb51d90c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180d1210 0x7fb5180b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb51d90c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb51d90c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5181f6c10 con 0x7fb51815d7c0 2023-10-16T11:17:44.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb51d90c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5181f6270 con 0x7fb5180ceb20 2023-10-16T11:17:44.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb51d90c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5181f6550 con 0x7fb5180d1210 2023-10-16T11:17:44.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb5177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb5177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180b8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36542/0 (socket says 172.21.15.158:36542) 2023-10-16T11:17:44.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb5177fe700 1 -- 172.21.15.158:0/2990737835 learned_addr learned my addr 172.21.15.158:0/2990737835 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:44.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb516ffd700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5180ceb20 0x7fb5180b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.584+0000 7fb5167fc700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180d1210 0x7fb5180b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb5177fe700 1 -- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180d1210 msgr2=0x7fb5180b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb5177fe700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180d1210 0x7fb5180b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb5177fe700 1 -- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5180ceb20 msgr2=0x7fb5180b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb5177fe700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5180ceb20 0x7fb5180b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb5177fe700 1 -- 172.21.15.158:0/2990737835 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508009550 con 0x7fb51815d7c0 2023-10-16T11:17:44.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb5177fe700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180b8d50 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fb51000efc0 tx=0x7fb51000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb4fffff700 1 -- 172.21.15.158:0/2990737835 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510009ea0 con 0x7fb51815d7c0 2023-10-16T11:17:44.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb4fffff700 1 -- 172.21.15.158:0/2990737835 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51000f040 con 0x7fb51815d7c0 2023-10-16T11:17:44.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5181f6830 con 0x7fb51815d7c0 2023-10-16T11:17:44.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb4fffff700 1 -- 172.21.15.158:0/2990737835 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510014480 con 0x7fb51815d7c0 2023-10-16T11:17:44.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.585+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5181f7030 con 0x7fb51815d7c0 2023-10-16T11:17:44.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.586+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5180d3840 con 0x7fb51815d7c0 2023-10-16T11:17:44.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.586+0000 7fb4fffff700 1 -- 172.21.15.158:0/2990737835 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb5100145e0 con 0x7fb51815d7c0 2023-10-16T11:17:44.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.587+0000 7fb4fffff700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fb50406b020 0x7fb50406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.587+0000 7fb4fffff700 1 -- 172.21.15.158:0/2990737835 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fb51008a760 con 0x7fb51815d7c0 2023-10-16T11:17:44.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.587+0000 7fb516ffd700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fb50406b020 0x7fb50406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.587+0000 7fb516ffd700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fb50406b020 0x7fb50406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb508014040 tx=0x7fb508008030 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.591+0000 7fb4fffff700 1 -- 172.21.15.158:0/2990737835 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5180d3840 con 0x7fb51815d7c0 2023-10-16T11:17:44.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.781+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 --> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5180d3580 con 0x7fb50406b020 2023-10-16T11:17:44.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.789+0000 7fb4fffff700 1 -- 172.21.15.158:0/2990737835 <== mgr.10824 v2:172.21.15.158:6832/3585499115 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290163 (secure 0 0 0) 0x7fb508011b80 con 0x7fb50406b020 2023-10-16T11:17:44.795 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:44.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fb50406b020 msgr2=0x7fb50406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fb50406b020 0x7fb50406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb508014040 tx=0x7fb508008030 comp rx=0 tx=0).stop 2023-10-16T11:17:44.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 msgr2=0x7fb5180b8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180b8d50 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fb51000efc0 tx=0x7fb51000c3a0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 shutdown_connections 2023-10-16T11:17:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb51815d7c0 0x7fb5180b8d50 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fb50406b020 0x7fb50406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5180ceb20 0x7fb5180b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 --2- 172.21.15.158:0/2990737835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5180d1210 0x7fb5180b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 >> 172.21.15.158:0/2990737835 conn(0x7fb5180669f0 msgr2=0x7fb51815caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 shutdown_connections 2023-10-16T11:17:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.793+0000 7fb51d90c700 1 -- 172.21.15.158:0/2990737835 wait complete. 2023-10-16T11:17:44.800 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:17:44.807 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T11:17:44.211191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":386824,"kb_used_data":20756,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754587896,"statfs":{"total":773094113280,"available":772698005504,"internally_reserved":0,"allocated":21254144,"data_stored":11836260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":374722103},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552006"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700776+0000","last_change":"2023-10-16T11:17:40.694497+0000","last_active":"2023-10-16T11:17:40.700776+0000","last_peered":"2023-10-16T11:17:40.700776+0000","last_clean":"2023-10-16T11:17:40.700776+0000","last_became_active":"2023-10-16T11:17:40.694220+0000","last_became_peered":"2023-10-16T11:17:40.694220+0000","last_unstale":"2023-10-16T11:17:40.700776+0000","last_undegraded":"2023-10-16T11:17:40.700776+0000","last_fullsized":"2023-10-16T11:17:40.700776+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:21.672778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700917+0000","last_change":"2023-10-16T11:17:40.695794+0000","last_active":"2023-10-16T11:17:40.700917+0000","last_peered":"2023-10-16T11:17:40.700917+0000","last_clean":"2023-10-16T11:17:40.700917+0000","last_became_active":"2023-10-16T11:17:40.695539+0000","last_became_peered":"2023-10-16T11:17:40.695539+0000","last_unstale":"2023-10-16T11:17:40.700917+0000","last_undegraded":"2023-10-16T11:17:40.700917+0000","last_fullsized":"2023-10-16T11:17:40.700917+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:22.032657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709672+0000","last_change":"2023-10-16T11:17:40.702396+0000","last_active":"2023-10-16T11:17:40.709672+0000","last_peered":"2023-10-16T11:17:40.709672+0000","last_clean":"2023-10-16T11:17:40.709672+0000","last_became_active":"2023-10-16T11:17:40.702104+0000","last_became_peered":"2023-10-16T11:17:40.702104+0000","last_unstale":"2023-10-16T11:17:40.709672+0000","last_undegraded":"2023-10-16T11:17:40.709672+0000","last_fullsized":"2023-10-16T11:17:40.709672+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:44.925107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917705+0000","last_change":"2023-10-16T11:17:40.696695+0000","last_active":"2023-10-16T11:17:41.917705+0000","last_peered":"2023-10-16T11:17:41.917705+0000","last_clean":"2023-10-16T11:17:41.917705+0000","last_became_active":"2023-10-16T11:17:40.693646+0000","last_became_peered":"2023-10-16T11:17:40.693646+0000","last_unstale":"2023-10-16T11:17:41.917705+0000","last_undegraded":"2023-10-16T11:17:41.917705+0000","last_fullsized":"2023-10-16T11:17:41.917705+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:51.324037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702933+0000","last_change":"2023-10-16T11:17:40.695772+0000","last_active":"2023-10-16T11:17:41.702933+0000","last_peered":"2023-10-16T11:17:41.702933+0000","last_clean":"2023-10-16T11:17:41.702933+0000","last_became_active":"2023-10-16T11:17:40.694840+0000","last_became_peered":"2023-10-16T11:17:40.694840+0000","last_unstale":"2023-10-16T11:17:41.702933+0000","last_undegraded":"2023-10-16T11:17:41.702933+0000","last_fullsized":"2023-10-16T11:17:41.702933+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:45:49.801058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702756+0000","last_change":"2023-10-16T11:17:40.690289+0000","last_active":"2023-10-16T11:17:41.702756+0000","last_peered":"2023-10-16T11:17:41.702756+0000","last_clean":"2023-10-16T11:17:41.702756+0000","last_became_active":"2023-10-16T11:17:40.689954+0000","last_became_peered":"2023-10-16T11:17:40.689954+0000","last_unstale":"2023-10-16T11:17:41.702756+0000","last_undegraded":"2023-10-16T11:17:41.702756+0000","last_fullsized":"2023-10-16T11:17:41.702756+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:36.680230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697609+0000","last_change":"2023-10-16T11:17:40.700103+0000","last_active":"2023-10-16T11:17:41.697609+0000","last_peered":"2023-10-16T11:17:41.697609+0000","last_clean":"2023-10-16T11:17:41.697609+0000","last_became_active":"2023-10-16T11:17:40.699829+0000","last_became_peered":"2023-10-16T11:17:40.699829+0000","last_unstale":"2023-10-16T11:17:41.697609+0000","last_undegraded":"2023-10-16T11:17:41.697609+0000","last_fullsized":"2023-10-16T11:17:41.697609+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:11.131193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708532+0000","last_change":"2023-10-16T11:17:40.703979+0000","last_active":"2023-10-16T11:17:40.708532+0000","last_peered":"2023-10-16T11:17:40.708532+0000","last_clean":"2023-10-16T11:17:40.708532+0000","last_became_active":"2023-10-16T11:17:40.703393+0000","last_became_peered":"2023-10-16T11:17:40.703393+0000","last_unstale":"2023-10-16T11:17:40.708532+0000","last_undegraded":"2023-10-16T11:17:40.708532+0000","last_fullsized":"2023-10-16T11:17:40.708532+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:31.159612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701140+0000","last_change":"2023-10-16T11:17:40.698076+0000","last_active":"2023-10-16T11:17:40.701140+0000","last_peered":"2023-10-16T11:17:40.701140+0000","last_clean":"2023-10-16T11:17:40.701140+0000","last_became_active":"2023-10-16T11:17:40.697767+0000","last_became_peered":"2023-10-16T11:17:40.697767+0000","last_unstale":"2023-10-16T11:17:40.701140+0000","last_undegraded":"2023-10-16T11:17:40.701140+0000","last_fullsized":"2023-10-16T11:17:40.701140+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:02.108714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703029+0000","last_change":"2023-10-16T11:17:40.692946+0000","last_active":"2023-10-16T11:17:41.703029+0000","last_peered":"2023-10-16T11:17:41.703029+0000","last_clean":"2023-10-16T11:17:41.703029+0000","last_became_active":"2023-10-16T11:17:40.692079+0000","last_became_peered":"2023-10-16T11:17:40.692079+0000","last_unstale":"2023-10-16T11:17:41.703029+0000","last_undegraded":"2023-10-16T11:17:41.703029+0000","last_fullsized":"2023-10-16T11:17:41.703029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:32.845329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.703985+0000","last_change":"2023-10-16T11:17:40.697631+0000","last_active":"2023-10-16T11:17:40.703985+0000","last_peered":"2023-10-16T11:17:40.703985+0000","last_clean":"2023-10-16T11:17:40.703985+0000","last_became_active":"2023-10-16T11:17:40.697311+0000","last_became_peered":"2023-10-16T11:17:40.697311+0000","last_unstale":"2023-10-16T11:17:40.703985+0000","last_undegraded":"2023-10-16T11:17:40.703985+0000","last_fullsized":"2023-10-16T11:17:40.703985+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:37.784073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917779+0000","last_change":"2023-10-16T11:17:40.695856+0000","last_active":"2023-10-16T11:17:41.917779+0000","last_peered":"2023-10-16T11:17:41.917779+0000","last_clean":"2023-10-16T11:17:41.917779+0000","last_became_active":"2023-10-16T11:17:40.695592+0000","last_became_peered":"2023-10-16T11:17:40.695592+0000","last_unstale":"2023-10-16T11:17:41.917779+0000","last_undegraded":"2023-10-16T11:17:41.917779+0000","last_fullsized":"2023-10-16T11:17:41.917779+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:10.797324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703482+0000","last_change":"2023-10-16T11:17:40.700186+0000","last_active":"2023-10-16T11:17:41.703482+0000","last_peered":"2023-10-16T11:17:41.703482+0000","last_clean":"2023-10-16T11:17:41.703482+0000","last_became_active":"2023-10-16T11:17:40.699545+0000","last_became_peered":"2023-10-16T11:17:40.699545+0000","last_unstale":"2023-10-16T11:17:41.703482+0000","last_undegraded":"2023-10-16T11:17:41.703482+0000","last_fullsized":"2023-10-16T11:17:41.703482+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:50.278406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697639+0000","last_change":"2023-10-16T11:17:40.692724+0000","last_active":"2023-10-16T11:17:41.697639+0000","last_peered":"2023-10-16T11:17:41.697639+0000","last_clean":"2023-10-16T11:17:41.697639+0000","last_became_active":"2023-10-16T11:17:40.692481+0000","last_became_peered":"2023-10-16T11:17:40.692481+0000","last_unstale":"2023-10-16T11:17:41.697639+0000","last_undegraded":"2023-10-16T11:17:41.697639+0000","last_fullsized":"2023-10-16T11:17:41.697639+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:08.747508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.696494+0000","last_change":"2023-10-16T11:17:40.693320+0000","last_active":"2023-10-16T11:17:40.696494+0000","last_peered":"2023-10-16T11:17:40.696494+0000","last_clean":"2023-10-16T11:17:40.696494+0000","last_became_active":"2023-10-16T11:17:40.692889+0000","last_became_peered":"2023-10-16T11:17:40.692889+0000","last_unstale":"2023-10-16T11:17:40.696494+0000","last_undegraded":"2023-10-16T11:17:40.696494+0000","last_fullsized":"2023-10-16T11:17:40.696494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:27.293703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702641+0000","last_change":"2023-10-16T11:17:40.695710+0000","last_active":"2023-10-16T11:17:41.702641+0000","last_peered":"2023-10-16T11:17:41.702641+0000","last_clean":"2023-10-16T11:17:41.702641+0000","last_became_active":"2023-10-16T11:17:40.695423+0000","last_became_peered":"2023-10-16T11:17:40.695423+0000","last_unstale":"2023-10-16T11:17:41.702641+0000","last_undegraded":"2023-10-16T11:17:41.702641+0000","last_fullsized":"2023-10-16T11:17:41.702641+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:10.682957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702918+0000","last_change":"2023-10-16T11:17:40.696331+0000","last_active":"2023-10-16T11:17:41.702918+0000","last_peered":"2023-10-16T11:17:41.702918+0000","last_clean":"2023-10-16T11:17:41.702918+0000","last_became_active":"2023-10-16T11:17:40.696061+0000","last_became_peered":"2023-10-16T11:17:40.696061+0000","last_unstale":"2023-10-16T11:17:41.702918+0000","last_undegraded":"2023-10-16T11:17:41.702918+0000","last_fullsized":"2023-10-16T11:17:41.702918+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:32.691142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703799+0000","last_change":"2023-10-16T11:17:40.697835+0000","last_active":"2023-10-16T11:17:41.703799+0000","last_peered":"2023-10-16T11:17:41.703799+0000","last_clean":"2023-10-16T11:17:41.703799+0000","last_became_active":"2023-10-16T11:17:40.695770+0000","last_became_peered":"2023-10-16T11:17:40.695770+0000","last_unstale":"2023-10-16T11:17:41.703799+0000","last_undegraded":"2023-10-16T11:17:41.703799+0000","last_fullsized":"2023-10-16T11:17:41.703799+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:26.640546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703604+0000","last_change":"2023-10-16T11:17:40.691107+0000","last_active":"2023-10-16T11:17:41.703604+0000","last_peered":"2023-10-16T11:17:41.703604+0000","last_clean":"2023-10-16T11:17:41.703604+0000","last_became_active":"2023-10-16T11:17:40.690833+0000","last_became_peered":"2023-10-16T11:17:40.690833+0000","last_unstale":"2023-10-16T11:17:41.703604+0000","last_undegraded":"2023-10-16T11:17:41.703604+0000","last_fullsized":"2023-10-16T11:17:41.703604+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:44.925107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703085+0000","last_change":"2023-10-16T11:17:40.696392+0000","last_active":"2023-10-16T11:17:41.703085+0000","last_peered":"2023-10-16T11:17:41.703085+0000","last_clean":"2023-10-16T11:17:41.703085+0000","last_became_active":"2023-10-16T11:17:40.696121+0000","last_became_peered":"2023-10-16T11:17:40.696121+0000","last_unstale":"2023-10-16T11:17:41.703085+0000","last_undegraded":"2023-10-16T11:17:41.703085+0000","last_fullsized":"2023-10-16T11:17:41.703085+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:12.243702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701178+0000","last_change":"2023-10-16T11:17:40.698440+0000","last_active":"2023-10-16T11:17:40.701178+0000","last_peered":"2023-10-16T11:17:40.701178+0000","last_clean":"2023-10-16T11:17:40.701178+0000","last_became_active":"2023-10-16T11:17:40.698164+0000","last_became_peered":"2023-10-16T11:17:40.698164+0000","last_unstale":"2023-10-16T11:17:40.701178+0000","last_undegraded":"2023-10-16T11:17:40.701178+0000","last_fullsized":"2023-10-16T11:17:40.701178+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:11.032489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700971+0000","last_change":"2023-10-16T11:17:40.697560+0000","last_active":"2023-10-16T11:17:40.700971+0000","last_peered":"2023-10-16T11:17:40.700971+0000","last_clean":"2023-10-16T11:17:40.700971+0000","last_became_active":"2023-10-16T11:17:40.697244+0000","last_became_peered":"2023-10-16T11:17:40.697244+0000","last_unstale":"2023-10-16T11:17:40.700971+0000","last_undegraded":"2023-10-16T11:17:40.700971+0000","last_fullsized":"2023-10-16T11:17:40.700971+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:38.240047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705902+0000","last_change":"2023-10-16T11:17:40.703106+0000","last_active":"2023-10-16T11:17:40.705902+0000","last_peered":"2023-10-16T11:17:40.705902+0000","last_clean":"2023-10-16T11:17:40.705902+0000","last_became_active":"2023-10-16T11:17:40.702829+0000","last_became_peered":"2023-10-16T11:17:40.702829+0000","last_unstale":"2023-10-16T11:17:40.705902+0000","last_undegraded":"2023-10-16T11:17:40.705902+0000","last_fullsized":"2023-10-16T11:17:40.705902+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:35.114359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701100+0000","last_change":"2023-10-16T11:17:40.698969+0000","last_active":"2023-10-16T11:17:40.701100+0000","last_peered":"2023-10-16T11:17:40.701100+0000","last_clean":"2023-10-16T11:17:40.701100+0000","last_became_active":"2023-10-16T11:17:40.698694+0000","last_became_peered":"2023-10-16T11:17:40.698694+0000","last_unstale":"2023-10-16T11:17:40.701100+0000","last_undegraded":"2023-10-16T11:17:40.701100+0000","last_fullsized":"2023-10-16T11:17:40.701100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:18.701459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697477+0000","last_change":"2023-10-16T11:17:40.699654+0000","last_active":"2023-10-16T11:17:41.697477+0000","last_peered":"2023-10-16T11:17:41.697477+0000","last_clean":"2023-10-16T11:17:41.697477+0000","last_became_active":"2023-10-16T11:17:40.699360+0000","last_became_peered":"2023-10-16T11:17:40.699360+0000","last_unstale":"2023-10-16T11:17:41.697477+0000","last_undegraded":"2023-10-16T11:17:41.697477+0000","last_fullsized":"2023-10-16T11:17:41.697477+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:01.988974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705211+0000","last_change":"2023-10-16T11:17:40.702431+0000","last_active":"2023-10-16T11:17:40.705211+0000","last_peered":"2023-10-16T11:17:40.705211+0000","last_clean":"2023-10-16T11:17:40.705211+0000","last_became_active":"2023-10-16T11:17:40.701745+0000","last_became_peered":"2023-10-16T11:17:40.701745+0000","last_unstale":"2023-10-16T11:17:40.705211+0000","last_undegraded":"2023-10-16T11:17:40.705211+0000","last_fullsized":"2023-10-16T11:17:40.705211+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:54:45.421589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708501+0000","last_change":"2023-10-16T11:17:40.703451+0000","last_active":"2023-10-16T11:17:40.708501+0000","last_peered":"2023-10-16T11:17:40.708501+0000","last_clean":"2023-10-16T11:17:40.708501+0000","last_became_active":"2023-10-16T11:17:40.703020+0000","last_became_peered":"2023-10-16T11:17:40.703020+0000","last_unstale":"2023-10-16T11:17:40.708501+0000","last_undegraded":"2023-10-16T11:17:40.708501+0000","last_fullsized":"2023-10-16T11:17:40.708501+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:45:55.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702852+0000","last_change":"2023-10-16T11:17:40.694622+0000","last_active":"2023-10-16T11:17:41.702852+0000","last_peered":"2023-10-16T11:17:41.702852+0000","last_clean":"2023-10-16T11:17:41.702852+0000","last_became_active":"2023-10-16T11:17:40.694348+0000","last_became_peered":"2023-10-16T11:17:40.694348+0000","last_unstale":"2023-10-16T11:17:41.702852+0000","last_undegraded":"2023-10-16T11:17:41.702852+0000","last_fullsized":"2023-10-16T11:17:41.702852+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.852772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.696826+0000","last_change":"2023-10-16T11:17:40.693732+0000","last_active":"2023-10-16T11:17:40.696826+0000","last_peered":"2023-10-16T11:17:40.696826+0000","last_clean":"2023-10-16T11:17:40.696826+0000","last_became_active":"2023-10-16T11:17:40.693400+0000","last_became_peered":"2023-10-16T11:17:40.693400+0000","last_unstale":"2023-10-16T11:17:40.696826+0000","last_undegraded":"2023-10-16T11:17:40.696826+0000","last_fullsized":"2023-10-16T11:17:40.696826+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:30.857760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.696625+0000","last_change":"2023-10-16T11:17:40.692417+0000","last_active":"2023-10-16T11:17:40.696625+0000","last_peered":"2023-10-16T11:17:40.696625+0000","last_clean":"2023-10-16T11:17:40.696625+0000","last_became_active":"2023-10-16T11:17:40.692043+0000","last_became_peered":"2023-10-16T11:17:40.692043+0000","last_unstale":"2023-10-16T11:17:40.696625+0000","last_undegraded":"2023-10-16T11:17:40.696625+0000","last_fullsized":"2023-10-16T11:17:40.696625+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:08.300873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697446+0000","last_change":"2023-10-16T11:17:40.692158+0000","last_active":"2023-10-16T11:17:41.697446+0000","last_peered":"2023-10-16T11:17:41.697446+0000","last_clean":"2023-10-16T11:17:41.697446+0000","last_became_active":"2023-10-16T11:17:40.691311+0000","last_became_peered":"2023-10-16T11:17:40.691311+0000","last_unstale":"2023-10-16T11:17:41.697446+0000","last_undegraded":"2023-10-16T11:17:41.697446+0000","last_fullsized":"2023-10-16T11:17:41.697446+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:49.546837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702785+0000","last_change":"2023-10-16T11:17:40.691547+0000","last_active":"2023-10-16T11:17:41.702785+0000","last_peered":"2023-10-16T11:17:41.702785+0000","last_clean":"2023-10-16T11:17:41.702785+0000","last_became_active":"2023-10-16T11:17:40.691291+0000","last_became_peered":"2023-10-16T11:17:40.691291+0000","last_unstale":"2023-10-16T11:17:41.702785+0000","last_undegraded":"2023-10-16T11:17:41.702785+0000","last_fullsized":"2023-10-16T11:17:41.702785+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:40.386764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709322+0000","last_change":"2023-10-16T11:17:40.705533+0000","last_active":"2023-10-16T11:17:40.709322+0000","last_peered":"2023-10-16T11:17:40.709322+0000","last_clean":"2023-10-16T11:17:40.709322+0000","last_became_active":"2023-10-16T11:17:40.705271+0000","last_became_peered":"2023-10-16T11:17:40.705271+0000","last_unstale":"2023-10-16T11:17:40.709322+0000","last_undegraded":"2023-10-16T11:17:40.709322+0000","last_fullsized":"2023-10-16T11:17:40.709322+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:12.327351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"199'4","reported_seq":16,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917810+0000","last_change":"2023-10-16T11:17:39.675277+0000","last_active":"2023-10-16T11:17:41.917810+0000","last_peered":"2023-10-16T11:17:41.917810+0000","last_clean":"2023-10-16T11:17:41.917810+0000","last_became_active":"2023-10-16T11:17:39.675018+0000","last_became_peered":"2023-10-16T11:17:39.675018+0000","last_unstale":"2023-10-16T11:17:41.917810+0000","last_undegraded":"2023-10-16T11:17:41.917810+0000","last_fullsized":"2023-10-16T11:17:41.917810+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:58.903873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697711+0000","last_change":"2023-10-16T11:17:40.699332+0000","last_active":"2023-10-16T11:17:41.697711+0000","last_peered":"2023-10-16T11:17:41.697711+0000","last_clean":"2023-10-16T11:17:41.697711+0000","last_became_active":"2023-10-16T11:17:40.699049+0000","last_became_peered":"2023-10-16T11:17:40.699049+0000","last_unstale":"2023-10-16T11:17:41.697711+0000","last_undegraded":"2023-10-16T11:17:41.697711+0000","last_fullsized":"2023-10-16T11:17:41.697711+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:38.866372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"199'1","reported_seq":13,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703103+0000","last_change":"2023-10-16T11:17:39.683955+0000","last_active":"2023-10-16T11:17:41.703103+0000","last_peered":"2023-10-16T11:17:41.703103+0000","last_clean":"2023-10-16T11:17:41.703103+0000","last_became_active":"2023-10-16T11:17:39.683703+0000","last_became_peered":"2023-10-16T11:17:39.683703+0000","last_unstale":"2023-10-16T11:17:41.703103+0000","last_undegraded":"2023-10-16T11:17:41.703103+0000","last_fullsized":"2023-10-16T11:17:41.703103+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:33.590796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705176+0000","last_change":"2023-10-16T11:17:40.698772+0000","last_active":"2023-10-16T11:17:40.705176+0000","last_peered":"2023-10-16T11:17:40.705176+0000","last_clean":"2023-10-16T11:17:40.705176+0000","last_became_active":"2023-10-16T11:17:40.698491+0000","last_became_peered":"2023-10-16T11:17:40.698491+0000","last_unstale":"2023-10-16T11:17:40.705176+0000","last_undegraded":"2023-10-16T11:17:40.705176+0000","last_fullsized":"2023-10-16T11:17:40.705176+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:29.439041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"199'2","reported_seq":13,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.711796+0000","last_change":"2023-10-16T11:17:39.673701+0000","last_active":"2023-10-16T11:17:40.711796+0000","last_peered":"2023-10-16T11:17:40.711796+0000","last_clean":"2023-10-16T11:17:40.711796+0000","last_became_active":"2023-10-16T11:17:39.673444+0000","last_became_peered":"2023-10-16T11:17:39.673444+0000","last_unstale":"2023-10-16T11:17:40.711796+0000","last_undegraded":"2023-10-16T11:17:40.711796+0000","last_fullsized":"2023-10-16T11:17:40.711796+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:35.427673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708784+0000","last_change":"2023-10-16T11:17:40.698034+0000","last_active":"2023-10-16T11:17:40.708784+0000","last_peered":"2023-10-16T11:17:40.708784+0000","last_clean":"2023-10-16T11:17:40.708784+0000","last_became_active":"2023-10-16T11:17:40.697399+0000","last_became_peered":"2023-10-16T11:17:40.697399+0000","last_unstale":"2023-10-16T11:17:40.708784+0000","last_undegraded":"2023-10-16T11:17:40.708784+0000","last_fullsized":"2023-10-16T11:17:40.708784+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:38.510740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703568+0000","last_change":"2023-10-16T11:17:39.674379+0000","last_active":"2023-10-16T11:17:41.703568+0000","last_peered":"2023-10-16T11:17:41.703568+0000","last_clean":"2023-10-16T11:17:41.703568+0000","last_became_active":"2023-10-16T11:17:39.674096+0000","last_became_peered":"2023-10-16T11:17:39.674096+0000","last_unstale":"2023-10-16T11:17:41.703568+0000","last_undegraded":"2023-10-16T11:17:41.703568+0000","last_fullsized":"2023-10-16T11:17:41.703568+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:14.587208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703161+0000","last_change":"2023-10-16T11:17:40.700985+0000","last_active":"2023-10-16T11:17:41.703161+0000","last_peered":"2023-10-16T11:17:41.703161+0000","last_clean":"2023-10-16T11:17:41.703161+0000","last_became_active":"2023-10-16T11:17:40.700696+0000","last_became_peered":"2023-10-16T11:17:40.700696+0000","last_unstale":"2023-10-16T11:17:41.703161+0000","last_undegraded":"2023-10-16T11:17:41.703161+0000","last_fullsized":"2023-10-16T11:17:41.703161+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:59.721389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"199'1","reported_seq":13,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703058+0000","last_change":"2023-10-16T11:17:39.683852+0000","last_active":"2023-10-16T11:17:41.703058+0000","last_peered":"2023-10-16T11:17:41.703058+0000","last_clean":"2023-10-16T11:17:41.703058+0000","last_became_active":"2023-10-16T11:17:39.681486+0000","last_became_peered":"2023-10-16T11:17:39.681486+0000","last_unstale":"2023-10-16T11:17:41.703058+0000","last_undegraded":"2023-10-16T11:17:41.703058+0000","last_fullsized":"2023-10-16T11:17:41.703058+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:08.123480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.693073+0000","last_change":"2023-10-16T11:17:40.691203+0000","last_active":"2023-10-16T11:17:40.693073+0000","last_peered":"2023-10-16T11:17:40.693073+0000","last_clean":"2023-10-16T11:17:40.693073+0000","last_became_active":"2023-10-16T11:17:40.690915+0000","last_became_peered":"2023-10-16T11:17:40.690915+0000","last_unstale":"2023-10-16T11:17:40.693073+0000","last_undegraded":"2023-10-16T11:17:40.693073+0000","last_fullsized":"2023-10-16T11:17:40.693073+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:47.296282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"199'1","reported_seq":13,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917572+0000","last_change":"2023-10-16T11:17:39.683877+0000","last_active":"2023-10-16T11:17:41.917572+0000","last_peered":"2023-10-16T11:17:41.917572+0000","last_clean":"2023-10-16T11:17:41.917572+0000","last_became_active":"2023-10-16T11:17:39.683583+0000","last_became_peered":"2023-10-16T11:17:39.683583+0000","last_unstale":"2023-10-16T11:17:41.917572+0000","last_undegraded":"2023-10-16T11:17:41.917572+0000","last_fullsized":"2023-10-16T11:17:41.917572+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:38.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.694412+0000","last_change":"2023-10-16T11:17:40.693377+0000","last_active":"2023-10-16T11:17:40.694412+0000","last_peered":"2023-10-16T11:17:40.694412+0000","last_clean":"2023-10-16T11:17:40.694412+0000","last_became_active":"2023-10-16T11:17:40.693046+0000","last_became_peered":"2023-10-16T11:17:40.693046+0000","last_unstale":"2023-10-16T11:17:40.694412+0000","last_undegraded":"2023-10-16T11:17:40.694412+0000","last_fullsized":"2023-10-16T11:17:40.694412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:38.661148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703280+0000","last_change":"2023-10-16T11:17:39.678628+0000","last_active":"2023-10-16T11:17:41.703280+0000","last_peered":"2023-10-16T11:17:41.703280+0000","last_clean":"2023-10-16T11:17:41.703280+0000","last_became_active":"2023-10-16T11:17:39.678352+0000","last_became_peered":"2023-10-16T11:17:39.678352+0000","last_unstale":"2023-10-16T11:17:41.703280+0000","last_undegraded":"2023-10-16T11:17:41.703280+0000","last_fullsized":"2023-10-16T11:17:41.703280+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:19.163238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.710130+0000","last_change":"2023-10-16T11:17:40.709430+0000","last_active":"2023-10-16T11:17:40.710130+0000","last_peered":"2023-10-16T11:17:40.710130+0000","last_clean":"2023-10-16T11:17:40.710130+0000","last_became_active":"2023-10-16T11:17:40.708515+0000","last_became_peered":"2023-10-16T11:17:40.708515+0000","last_unstale":"2023-10-16T11:17:40.710130+0000","last_undegraded":"2023-10-16T11:17:40.710130+0000","last_fullsized":"2023-10-16T11:17:40.710130+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:31.963145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703038+0000","last_change":"2023-10-16T11:17:39.683136+0000","last_active":"2023-10-16T11:17:41.703038+0000","last_peered":"2023-10-16T11:17:41.703038+0000","last_clean":"2023-10-16T11:17:41.703038+0000","last_became_active":"2023-10-16T11:17:39.682216+0000","last_became_peered":"2023-10-16T11:17:39.682216+0000","last_unstale":"2023-10-16T11:17:41.703038+0000","last_undegraded":"2023-10-16T11:17:41.703038+0000","last_fullsized":"2023-10-16T11:17:41.703038+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:59.797957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702752+0000","last_change":"2023-10-16T11:17:40.688250+0000","last_active":"2023-10-16T11:17:41.702752+0000","last_peered":"2023-10-16T11:17:41.702752+0000","last_clean":"2023-10-16T11:17:41.702752+0000","last_became_active":"2023-10-16T11:17:40.688000+0000","last_became_peered":"2023-10-16T11:17:40.688000+0000","last_unstale":"2023-10-16T11:17:41.702752+0000","last_undegraded":"2023-10-16T11:17:41.702752+0000","last_fullsized":"2023-10-16T11:17:41.702752+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:26.210836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.680588+0000","last_change":"2023-10-16T11:17:39.687390+0000","last_active":"2023-10-16T11:17:40.680588+0000","last_peered":"2023-10-16T11:17:40.680588+0000","last_clean":"2023-10-16T11:17:40.680588+0000","last_became_active":"2023-10-16T11:17:39.687111+0000","last_became_peered":"2023-10-16T11:17:39.687111+0000","last_unstale":"2023-10-16T11:17:40.680588+0000","last_undegraded":"2023-10-16T11:17:40.680588+0000","last_fullsized":"2023-10-16T11:17:40.680588+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:56.248764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703934+0000","last_change":"2023-10-16T11:17:40.696043+0000","last_active":"2023-10-16T11:17:41.703934+0000","last_peered":"2023-10-16T11:17:41.703934+0000","last_clean":"2023-10-16T11:17:41.703934+0000","last_became_active":"2023-10-16T11:17:40.695381+0000","last_became_peered":"2023-10-16T11:17:40.695381+0000","last_unstale":"2023-10-16T11:17:41.703934+0000","last_undegraded":"2023-10-16T11:17:41.703934+0000","last_fullsized":"2023-10-16T11:17:41.703934+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:48.707835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"199'3","reported_seq":15,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917687+0000","last_change":"2023-10-16T11:17:39.684566+0000","last_active":"2023-10-16T11:17:41.917687+0000","last_peered":"2023-10-16T11:17:41.917687+0000","last_clean":"2023-10-16T11:17:41.917687+0000","last_became_active":"2023-10-16T11:17:39.684287+0000","last_became_peered":"2023-10-16T11:17:39.684287+0000","last_unstale":"2023-10-16T11:17:41.917687+0000","last_undegraded":"2023-10-16T11:17:41.917687+0000","last_fullsized":"2023-10-16T11:17:41.917687+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:51:55.403425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700798+0000","last_change":"2023-10-16T11:17:40.697024+0000","last_active":"2023-10-16T11:17:40.700798+0000","last_peered":"2023-10-16T11:17:40.700798+0000","last_clean":"2023-10-16T11:17:40.700798+0000","last_became_active":"2023-10-16T11:17:40.696758+0000","last_became_peered":"2023-10-16T11:17:40.696758+0000","last_unstale":"2023-10-16T11:17:40.700798+0000","last_undegraded":"2023-10-16T11:17:40.700798+0000","last_fullsized":"2023-10-16T11:17:40.700798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:33.929739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"199'6","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703654+0000","last_change":"2023-10-16T11:17:39.670423+0000","last_active":"2023-10-16T11:17:41.703654+0000","last_peered":"2023-10-16T11:17:41.703654+0000","last_clean":"2023-10-16T11:17:41.703654+0000","last_became_active":"2023-10-16T11:17:39.670158+0000","last_became_peered":"2023-10-16T11:17:39.670158+0000","last_unstale":"2023-10-16T11:17:41.703654+0000","last_undegraded":"2023-10-16T11:17:41.703654+0000","last_fullsized":"2023-10-16T11:17:41.703654+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:13.682735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697280+0000","last_change":"2023-10-16T11:17:40.699423+0000","last_active":"2023-10-16T11:17:41.697280+0000","last_peered":"2023-10-16T11:17:41.697280+0000","last_clean":"2023-10-16T11:17:41.697280+0000","last_became_active":"2023-10-16T11:17:40.698999+0000","last_became_peered":"2023-10-16T11:17:40.698999+0000","last_unstale":"2023-10-16T11:17:41.697280+0000","last_undegraded":"2023-10-16T11:17:41.697280+0000","last_fullsized":"2023-10-16T11:17:41.697280+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:01.675519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703684+0000","last_change":"2023-10-16T11:17:39.679123+0000","last_active":"2023-10-16T11:17:41.703684+0000","last_peered":"2023-10-16T11:17:41.703684+0000","last_clean":"2023-10-16T11:17:41.703684+0000","last_became_active":"2023-10-16T11:17:39.678883+0000","last_became_peered":"2023-10-16T11:17:39.678883+0000","last_unstale":"2023-10-16T11:17:41.703684+0000","last_undegraded":"2023-10-16T11:17:41.703684+0000","last_fullsized":"2023-10-16T11:17:41.703684+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:25.492564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703066+0000","last_change":"2023-10-16T11:17:40.697723+0000","last_active":"2023-10-16T11:17:41.703066+0000","last_peered":"2023-10-16T11:17:41.703066+0000","last_clean":"2023-10-16T11:17:41.703066+0000","last_became_active":"2023-10-16T11:17:40.697466+0000","last_became_peered":"2023-10-16T11:17:40.697466+0000","last_unstale":"2023-10-16T11:17:41.703066+0000","last_undegraded":"2023-10-16T11:17:41.703066+0000","last_fullsized":"2023-10-16T11:17:41.703066+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:41.224618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.681174+0000","last_change":"2023-10-16T11:17:39.682261+0000","last_active":"2023-10-16T11:17:40.681174+0000","last_peered":"2023-10-16T11:17:40.681174+0000","last_clean":"2023-10-16T11:17:40.681174+0000","last_became_active":"2023-10-16T11:17:39.681991+0000","last_became_peered":"2023-10-16T11:17:39.681991+0000","last_unstale":"2023-10-16T11:17:40.681174+0000","last_undegraded":"2023-10-16T11:17:40.681174+0000","last_fullsized":"2023-10-16T11:17:40.681174+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:33.523191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703521+0000","last_change":"2023-10-16T11:17:40.691839+0000","last_active":"2023-10-16T11:17:41.703521+0000","last_peered":"2023-10-16T11:17:41.703521+0000","last_clean":"2023-10-16T11:17:41.703521+0000","last_became_active":"2023-10-16T11:17:40.691181+0000","last_became_peered":"2023-10-16T11:17:40.691181+0000","last_unstale":"2023-10-16T11:17:41.703521+0000","last_undegraded":"2023-10-16T11:17:41.703521+0000","last_fullsized":"2023-10-16T11:17:41.703521+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:37.678572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917495+0000","last_change":"2023-10-16T11:17:39.673769+0000","last_active":"2023-10-16T11:17:41.917495+0000","last_peered":"2023-10-16T11:17:41.917495+0000","last_clean":"2023-10-16T11:17:41.917495+0000","last_became_active":"2023-10-16T11:17:39.673483+0000","last_became_peered":"2023-10-16T11:17:39.673483+0000","last_unstale":"2023-10-16T11:17:41.917495+0000","last_undegraded":"2023-10-16T11:17:41.917495+0000","last_fullsized":"2023-10-16T11:17:41.917495+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:36.779758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703912+0000","last_change":"2023-10-16T11:17:40.694569+0000","last_active":"2023-10-16T11:17:41.703912+0000","last_peered":"2023-10-16T11:17:41.703912+0000","last_clean":"2023-10-16T11:17:41.703912+0000","last_became_active":"2023-10-16T11:17:40.694235+0000","last_became_peered":"2023-10-16T11:17:40.694235+0000","last_unstale":"2023-10-16T11:17:41.703912+0000","last_undegraded":"2023-10-16T11:17:41.703912+0000","last_fullsized":"2023-10-16T11:17:41.703912+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:38.510740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"199'8","reported_seq":19,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.714318+0000","last_change":"2023-10-16T11:17:39.675603+0000","last_active":"2023-10-16T11:17:40.714318+0000","last_peered":"2023-10-16T11:17:40.714318+0000","last_clean":"2023-10-16T11:17:40.714318+0000","last_became_active":"2023-10-16T11:17:39.675356+0000","last_became_peered":"2023-10-16T11:17:39.675356+0000","last_unstale":"2023-10-16T11:17:40.714318+0000","last_undegraded":"2023-10-16T11:17:40.714318+0000","last_fullsized":"2023-10-16T11:17:40.714318+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:38.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697433+0000","last_change":"2023-10-16T11:17:40.693857+0000","last_active":"2023-10-16T11:17:41.697433+0000","last_peered":"2023-10-16T11:17:41.697433+0000","last_clean":"2023-10-16T11:17:41.697433+0000","last_became_active":"2023-10-16T11:17:40.693603+0000","last_became_peered":"2023-10-16T11:17:40.693603+0000","last_unstale":"2023-10-16T11:17:41.697433+0000","last_undegraded":"2023-10-16T11:17:41.697433+0000","last_fullsized":"2023-10-16T11:17:41.697433+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:00.469870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"199'1","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.713527+0000","last_change":"2023-10-16T11:17:39.675704+0000","last_active":"2023-10-16T11:17:40.713527+0000","last_peered":"2023-10-16T11:17:40.713527+0000","last_clean":"2023-10-16T11:17:40.713527+0000","last_became_active":"2023-10-16T11:17:39.675418+0000","last_became_peered":"2023-10-16T11:17:39.675418+0000","last_unstale":"2023-10-16T11:17:40.713527+0000","last_undegraded":"2023-10-16T11:17:40.713527+0000","last_fullsized":"2023-10-16T11:17:40.713527+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:33.039808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702804+0000","last_change":"2023-10-16T11:17:40.688585+0000","last_active":"2023-10-16T11:17:41.702804+0000","last_peered":"2023-10-16T11:17:41.702804+0000","last_clean":"2023-10-16T11:17:41.702804+0000","last_became_active":"2023-10-16T11:17:40.687493+0000","last_became_peered":"2023-10-16T11:17:40.687493+0000","last_unstale":"2023-10-16T11:17:41.702804+0000","last_undegraded":"2023-10-16T11:17:41.702804+0000","last_fullsized":"2023-10-16T11:17:41.702804+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:42.635777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917701+0000","last_change":"2023-10-16T11:17:39.674738+0000","last_active":"2023-10-16T11:17:41.917701+0000","last_peered":"2023-10-16T11:17:41.917701+0000","last_clean":"2023-10-16T11:17:41.917701+0000","last_became_active":"2023-10-16T11:17:39.674479+0000","last_became_peered":"2023-10-16T11:17:39.674479+0000","last_unstale":"2023-10-16T11:17:41.917701+0000","last_undegraded":"2023-10-16T11:17:41.917701+0000","last_fullsized":"2023-10-16T11:17:41.917701+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:54.405847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705195+0000","last_change":"2023-10-16T11:17:40.702264+0000","last_active":"2023-10-16T11:17:40.705195+0000","last_peered":"2023-10-16T11:17:40.705195+0000","last_clean":"2023-10-16T11:17:40.705195+0000","last_became_active":"2023-10-16T11:17:40.701407+0000","last_became_peered":"2023-10-16T11:17:40.701407+0000","last_unstale":"2023-10-16T11:17:40.705195+0000","last_undegraded":"2023-10-16T11:17:40.705195+0000","last_fullsized":"2023-10-16T11:17:40.705195+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:03.169653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"199'1","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709719+0000","last_change":"2023-10-16T11:17:39.684443+0000","last_active":"2023-10-16T11:17:40.709719+0000","last_peered":"2023-10-16T11:17:40.709719+0000","last_clean":"2023-10-16T11:17:40.709719+0000","last_became_active":"2023-10-16T11:17:39.684164+0000","last_became_peered":"2023-10-16T11:17:39.684164+0000","last_unstale":"2023-10-16T11:17:40.709719+0000","last_undegraded":"2023-10-16T11:17:40.709719+0000","last_fullsized":"2023-10-16T11:17:40.709719+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:25.636230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701159+0000","last_change":"2023-10-16T11:17:40.699740+0000","last_active":"2023-10-16T11:17:40.701159+0000","last_peered":"2023-10-16T11:17:40.701159+0000","last_clean":"2023-10-16T11:17:40.701159+0000","last_became_active":"2023-10-16T11:17:40.699468+0000","last_became_peered":"2023-10-16T11:17:40.699468+0000","last_unstale":"2023-10-16T11:17:40.701159+0000","last_undegraded":"2023-10-16T11:17:40.701159+0000","last_fullsized":"2023-10-16T11:17:40.701159+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:20.114136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.691776+0000","last_change":"2023-10-16T11:17:39.676587+0000","last_active":"2023-10-16T11:17:40.691776+0000","last_peered":"2023-10-16T11:17:40.691776+0000","last_clean":"2023-10-16T11:17:40.691776+0000","last_became_active":"2023-10-16T11:17:39.676297+0000","last_became_peered":"2023-10-16T11:17:39.676297+0000","last_unstale":"2023-10-16T11:17:40.691776+0000","last_undegraded":"2023-10-16T11:17:40.691776+0000","last_fullsized":"2023-10-16T11:17:40.691776+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:10.448777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.691976+0000","last_change":"2023-10-16T11:17:40.690713+0000","last_active":"2023-10-16T11:17:40.691976+0000","last_peered":"2023-10-16T11:17:40.691976+0000","last_clean":"2023-10-16T11:17:40.691976+0000","last_became_active":"2023-10-16T11:17:40.690472+0000","last_became_peered":"2023-10-16T11:17:40.690472+0000","last_unstale":"2023-10-16T11:17:40.691976+0000","last_undegraded":"2023-10-16T11:17:40.691976+0000","last_fullsized":"2023-10-16T11:17:40.691976+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:30.669145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703774+0000","last_change":"2023-10-16T11:17:39.681349+0000","last_active":"2023-10-16T11:17:41.703774+0000","last_peered":"2023-10-16T11:17:41.703774+0000","last_clean":"2023-10-16T11:17:41.703774+0000","last_became_active":"2023-10-16T11:17:39.681055+0000","last_became_peered":"2023-10-16T11:17:39.681055+0000","last_unstale":"2023-10-16T11:17:41.703774+0000","last_undegraded":"2023-10-16T11:17:41.703774+0000","last_fullsized":"2023-10-16T11:17:41.703774+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:39.368829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917552+0000","last_change":"2023-10-16T11:17:40.693825+0000","last_active":"2023-10-16T11:17:41.917552+0000","last_peered":"2023-10-16T11:17:41.917552+0000","last_clean":"2023-10-16T11:17:41.917552+0000","last_became_active":"2023-10-16T11:17:40.693406+0000","last_became_peered":"2023-10-16T11:17:40.693406+0000","last_unstale":"2023-10-16T11:17:41.917552+0000","last_undegraded":"2023-10-16T11:17:41.917552+0000","last_fullsized":"2023-10-16T11:17:41.917552+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:34.044124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917727+0000","last_change":"2023-10-16T11:17:39.674089+0000","last_active":"2023-10-16T11:17:41.917727+0000","last_peered":"2023-10-16T11:17:41.917727+0000","last_clean":"2023-10-16T11:17:41.917727+0000","last_became_active":"2023-10-16T11:17:39.673817+0000","last_became_peered":"2023-10-16T11:17:39.673817+0000","last_unstale":"2023-10-16T11:17:41.917727+0000","last_undegraded":"2023-10-16T11:17:41.917727+0000","last_fullsized":"2023-10-16T11:17:41.917727+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:10.448777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917731+0000","last_change":"2023-10-16T11:17:40.694458+0000","last_active":"2023-10-16T11:17:41.917731+0000","last_peered":"2023-10-16T11:17:41.917731+0000","last_clean":"2023-10-16T11:17:41.917731+0000","last_became_active":"2023-10-16T11:17:40.693271+0000","last_became_peered":"2023-10-16T11:17:40.693271+0000","last_unstale":"2023-10-16T11:17:41.917731+0000","last_undegraded":"2023-10-16T11:17:41.917731+0000","last_fullsized":"2023-10-16T11:17:41.917731+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:51.908463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697223+0000","last_change":"2023-10-16T11:17:39.669434+0000","last_active":"2023-10-16T11:17:41.697223+0000","last_peered":"2023-10-16T11:17:41.697223+0000","last_clean":"2023-10-16T11:17:41.697223+0000","last_became_active":"2023-10-16T11:17:39.669186+0000","last_became_peered":"2023-10-16T11:17:39.669186+0000","last_unstale":"2023-10-16T11:17:41.697223+0000","last_undegraded":"2023-10-16T11:17:41.697223+0000","last_fullsized":"2023-10-16T11:17:41.697223+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:59.072882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702683+0000","last_change":"2023-10-16T11:17:40.690074+0000","last_active":"2023-10-16T11:17:41.702683+0000","last_peered":"2023-10-16T11:17:41.702683+0000","last_clean":"2023-10-16T11:17:41.702683+0000","last_became_active":"2023-10-16T11:17:40.689766+0000","last_became_peered":"2023-10-16T11:17:40.689766+0000","last_unstale":"2023-10-16T11:17:41.702683+0000","last_undegraded":"2023-10-16T11:17:41.702683+0000","last_fullsized":"2023-10-16T11:17:41.702683+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:00:41.301924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.688656+0000","last_change":"2023-10-16T11:17:39.676742+0000","last_active":"2023-10-16T11:17:40.688656+0000","last_peered":"2023-10-16T11:17:40.688656+0000","last_clean":"2023-10-16T11:17:40.688656+0000","last_became_active":"2023-10-16T11:17:39.676477+0000","last_became_peered":"2023-10-16T11:17:39.676477+0000","last_unstale":"2023-10-16T11:17:40.688656+0000","last_undegraded":"2023-10-16T11:17:40.688656+0000","last_fullsized":"2023-10-16T11:17:40.688656+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:49.095636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697546+0000","last_change":"2023-10-16T11:17:40.695234+0000","last_active":"2023-10-16T11:17:41.697546+0000","last_peered":"2023-10-16T11:17:41.697546+0000","last_clean":"2023-10-16T11:17:41.697546+0000","last_became_active":"2023-10-16T11:17:40.694963+0000","last_became_peered":"2023-10-16T11:17:40.694963+0000","last_unstale":"2023-10-16T11:17:41.697546+0000","last_undegraded":"2023-10-16T11:17:41.697546+0000","last_fullsized":"2023-10-16T11:17:41.697546+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:27.403827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917811+0000","last_change":"2023-10-16T11:17:39.676664+0000","last_active":"2023-10-16T11:17:41.917811+0000","last_peered":"2023-10-16T11:17:41.917811+0000","last_clean":"2023-10-16T11:17:41.917811+0000","last_became_active":"2023-10-16T11:17:39.676211+0000","last_became_peered":"2023-10-16T11:17:39.676211+0000","last_unstale":"2023-10-16T11:17:41.917811+0000","last_undegraded":"2023-10-16T11:17:41.917811+0000","last_fullsized":"2023-10-16T11:17:41.917811+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:07.296557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.690582+0000","last_change":"2023-10-16T11:17:40.689614+0000","last_active":"2023-10-16T11:17:40.690582+0000","last_peered":"2023-10-16T11:17:40.690582+0000","last_clean":"2023-10-16T11:17:40.690582+0000","last_became_active":"2023-10-16T11:17:40.689341+0000","last_became_peered":"2023-10-16T11:17:40.689341+0000","last_unstale":"2023-10-16T11:17:40.690582+0000","last_undegraded":"2023-10-16T11:17:40.690582+0000","last_fullsized":"2023-10-16T11:17:40.690582+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:16.262244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697762+0000","last_change":"2023-10-16T11:17:39.670518+0000","last_active":"2023-10-16T11:17:41.697762+0000","last_peered":"2023-10-16T11:17:41.697762+0000","last_clean":"2023-10-16T11:17:41.697762+0000","last_became_active":"2023-10-16T11:17:39.670260+0000","last_became_peered":"2023-10-16T11:17:39.670260+0000","last_unstale":"2023-10-16T11:17:41.697762+0000","last_undegraded":"2023-10-16T11:17:41.697762+0000","last_fullsized":"2023-10-16T11:17:41.697762+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:01.529436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708572+0000","last_change":"2023-10-16T11:17:40.703935+0000","last_active":"2023-10-16T11:17:40.708572+0000","last_peered":"2023-10-16T11:17:40.708572+0000","last_clean":"2023-10-16T11:17:40.708572+0000","last_became_active":"2023-10-16T11:17:40.703641+0000","last_became_peered":"2023-10-16T11:17:40.703641+0000","last_unstale":"2023-10-16T11:17:40.708572+0000","last_undegraded":"2023-10-16T11:17:40.708572+0000","last_fullsized":"2023-10-16T11:17:40.708572+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:24.300682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702932+0000","last_change":"2023-10-16T11:17:39.682853+0000","last_active":"2023-10-16T11:17:41.702932+0000","last_peered":"2023-10-16T11:17:41.702932+0000","last_clean":"2023-10-16T11:17:41.702932+0000","last_became_active":"2023-10-16T11:17:39.682595+0000","last_became_peered":"2023-10-16T11:17:39.682595+0000","last_unstale":"2023-10-16T11:17:41.702932+0000","last_undegraded":"2023-10-16T11:17:41.702932+0000","last_fullsized":"2023-10-16T11:17:41.702932+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:37.010036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697553+0000","last_change":"2023-10-16T11:17:40.695544+0000","last_active":"2023-10-16T11:17:41.697553+0000","last_peered":"2023-10-16T11:17:41.697553+0000","last_clean":"2023-10-16T11:17:41.697553+0000","last_became_active":"2023-10-16T11:17:40.695271+0000","last_became_peered":"2023-10-16T11:17:40.695271+0000","last_unstale":"2023-10-16T11:17:41.697553+0000","last_undegraded":"2023-10-16T11:17:41.697553+0000","last_fullsized":"2023-10-16T11:17:41.697553+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:15.875869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702939+0000","last_change":"2023-10-16T11:17:39.685403+0000","last_active":"2023-10-16T11:17:41.702939+0000","last_peered":"2023-10-16T11:17:41.702939+0000","last_clean":"2023-10-16T11:17:41.702939+0000","last_became_active":"2023-10-16T11:17:39.685168+0000","last_became_peered":"2023-10-16T11:17:39.685168+0000","last_unstale":"2023-10-16T11:17:41.702939+0000","last_undegraded":"2023-10-16T11:17:41.702939+0000","last_fullsized":"2023-10-16T11:17:41.702939+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:21.717775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708995+0000","last_change":"2023-10-16T11:17:40.700629+0000","last_active":"2023-10-16T11:17:40.708995+0000","last_peered":"2023-10-16T11:17:40.708995+0000","last_clean":"2023-10-16T11:17:40.708995+0000","last_became_active":"2023-10-16T11:17:40.700334+0000","last_became_peered":"2023-10-16T11:17:40.700334+0000","last_unstale":"2023-10-16T11:17:40.708995+0000","last_undegraded":"2023-10-16T11:17:40.708995+0000","last_fullsized":"2023-10-16T11:17:40.708995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:59.721389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.693658+0000","last_change":"2023-10-16T11:17:39.680002+0000","last_active":"2023-10-16T11:17:40.693658+0000","last_peered":"2023-10-16T11:17:40.693658+0000","last_clean":"2023-10-16T11:17:40.693658+0000","last_became_active":"2023-10-16T11:17:39.679711+0000","last_became_peered":"2023-10-16T11:17:39.679711+0000","last_unstale":"2023-10-16T11:17:40.693658+0000","last_undegraded":"2023-10-16T11:17:40.693658+0000","last_fullsized":"2023-10-16T11:17:40.693658+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:39.988284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703255+0000","last_change":"2023-10-16T11:17:40.699067+0000","last_active":"2023-10-16T11:17:41.703255+0000","last_peered":"2023-10-16T11:17:41.703255+0000","last_clean":"2023-10-16T11:17:41.703255+0000","last_became_active":"2023-10-16T11:17:40.698789+0000","last_became_peered":"2023-10-16T11:17:40.698789+0000","last_unstale":"2023-10-16T11:17:41.703255+0000","last_undegraded":"2023-10-16T11:17:41.703255+0000","last_fullsized":"2023-10-16T11:17:41.703255+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:50.099951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.692684+0000","last_change":"2023-10-16T11:17:39.685541+0000","last_active":"2023-10-16T11:17:40.692684+0000","last_peered":"2023-10-16T11:17:40.692684+0000","last_clean":"2023-10-16T11:17:40.692684+0000","last_became_active":"2023-10-16T11:17:39.685324+0000","last_became_peered":"2023-10-16T11:17:39.685324+0000","last_unstale":"2023-10-16T11:17:40.692684+0000","last_undegraded":"2023-10-16T11:17:40.692684+0000","last_fullsized":"2023-10-16T11:17:40.692684+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:53.860398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.694390+0000","last_change":"2023-10-16T11:17:40.693330+0000","last_active":"2023-10-16T11:17:40.694390+0000","last_peered":"2023-10-16T11:17:40.694390+0000","last_clean":"2023-10-16T11:17:40.694390+0000","last_became_active":"2023-10-16T11:17:40.692609+0000","last_became_peered":"2023-10-16T11:17:40.692609+0000","last_unstale":"2023-10-16T11:17:40.694390+0000","last_undegraded":"2023-10-16T11:17:40.694390+0000","last_fullsized":"2023-10-16T11:17:40.694390+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:10.608318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697532+0000","last_change":"2023-10-16T11:17:39.677643+0000","last_active":"2023-10-16T11:17:41.697532+0000","last_peered":"2023-10-16T11:17:41.697532+0000","last_clean":"2023-10-16T11:17:41.697532+0000","last_became_active":"2023-10-16T11:17:39.677170+0000","last_became_peered":"2023-10-16T11:17:39.677170+0000","last_unstale":"2023-10-16T11:17:41.697532+0000","last_undegraded":"2023-10-16T11:17:41.697532+0000","last_fullsized":"2023-10-16T11:17:41.697532+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:18.641776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705872+0000","last_change":"2023-10-16T11:17:40.702439+0000","last_active":"2023-10-16T11:17:40.705872+0000","last_peered":"2023-10-16T11:17:40.705872+0000","last_clean":"2023-10-16T11:17:40.705872+0000","last_became_active":"2023-10-16T11:17:40.702140+0000","last_became_peered":"2023-10-16T11:17:40.702140+0000","last_unstale":"2023-10-16T11:17:40.705872+0000","last_undegraded":"2023-10-16T11:17:40.705872+0000","last_fullsized":"2023-10-16T11:17:40.705872+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:15:43.365279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.688800+0000","last_change":"2023-10-16T11:17:39.687613+0000","last_active":"2023-10-16T11:17:40.688800+0000","last_peered":"2023-10-16T11:17:40.688800+0000","last_clean":"2023-10-16T11:17:40.688800+0000","last_became_active":"2023-10-16T11:17:39.687403+0000","last_became_peered":"2023-10-16T11:17:39.687403+0000","last_unstale":"2023-10-16T11:17:40.688800+0000","last_undegraded":"2023-10-16T11:17:40.688800+0000","last_fullsized":"2023-10-16T11:17:40.688800+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:49.274091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709202+0000","last_change":"2023-10-16T11:17:40.707377+0000","last_active":"2023-10-16T11:17:40.709202+0000","last_peered":"2023-10-16T11:17:40.709202+0000","last_clean":"2023-10-16T11:17:40.709202+0000","last_became_active":"2023-10-16T11:17:40.707085+0000","last_became_peered":"2023-10-16T11:17:40.707085+0000","last_unstale":"2023-10-16T11:17:40.709202+0000","last_undegraded":"2023-10-16T11:17:40.709202+0000","last_fullsized":"2023-10-16T11:17:40.709202+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:37.678572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917515+0000","last_change":"2023-10-16T11:17:39.674731+0000","last_active":"2023-10-16T11:17:41.917515+0000","last_peered":"2023-10-16T11:17:41.917515+0000","last_clean":"2023-10-16T11:17:41.917515+0000","last_became_active":"2023-10-16T11:17:39.674462+0000","last_became_peered":"2023-10-16T11:17:39.674462+0000","last_unstale":"2023-10-16T11:17:41.917515+0000","last_undegraded":"2023-10-16T11:17:41.917515+0000","last_fullsized":"2023-10-16T11:17:41.917515+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:28.434726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"190'326","reported_seq":1114,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-10-16T11:17:39.671226+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:17:39.671226+0000","last_peered":"2023-10-16T11:17:39.671226+0000","last_clean":"2023-10-16T11:17:39.671226+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:17:39.671226+0000","last_undegraded":"2023-10-16T11:17:39.671226+0000","last_fullsized":"2023-10-16T11:17:39.671226+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575970,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52248,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319592,"statfs":{"total":96636764160,"available":96583262208,"internally_reserved":0,"allocated":3092480,"data_stored":1920822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.374,"5min":0.372,"15min":0.372},"max":{"1min":0.76200000000000001,"5min":0.84299999999999997,"15min":0.98499999999999999},"last":0.629},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72899999999999998,"5min":0.79500000000000004,"15min":0.999},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.64600000000000002},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.39500000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":1.0640000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.81799999999999995,"5min":0.83499999999999996,"15min":1.0349999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78800000000000003,"5min":0.84199999999999997,"15min":1.117},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66100000000000003,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.41899999999999998},"max":{"1min":0.75700000000000001,"5min":0.90400000000000003,"15min":1.05},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.39500000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78400000000000003,"5min":0.83999999999999997,"15min":1.022},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67800000000000005,"15min":0.68799999999999994},"min":{"1min":0.51600000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.85499999999999998,"15min":1.083},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.50900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.313},"max":{"1min":0.71199999999999997,"5min":0.81599999999999995,"15min":0.94799999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.52300000000000002,"15min":0.53800000000000003},"min":{"1min":0.372,"5min":0.35299999999999998,"15min":0.34399999999999997},"max":{"1min":0.71699999999999997,"5min":0.83299999999999996,"15min":0.96299999999999997},"last":0.64500000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53100000000000003,"15min":0.54300000000000004},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77100000000000002,"5min":0.83899999999999997,"15min":0.91200000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.42599999999999999,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.80700000000000005,"5min":0.85299999999999998,"15min":0.92000000000000004},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61599999999999999},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.373},"max":{"1min":0.86099999999999999,"5min":0.88500000000000001,"15min":0.92600000000000005},"last":0.747},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.45100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88600000000000001,"5min":0.88700000000000001,"15min":0.91300000000000003},"last":0.58899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575970,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37784,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334056,"statfs":{"total":96636764160,"available":96598073344,"internally_reserved":0,"allocated":2502656,"data_stored":1330524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57899999999999996,"15min":0.59899999999999998},"min":{"1min":0.29499999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70999999999999996,"5min":0.81000000000000005,"15min":0.91000000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.78200000000000003,"5min":0.82599999999999996,"15min":0.88500000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":0.85099999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60199999999999998,"15min":0.628},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.95499999999999996},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.371,"5min":0.314,"15min":0.314},"max":{"1min":0.78600000000000003,"5min":0.83099999999999996,"15min":0.95599999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.622,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72999999999999998,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.61499999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.624},"min":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.32300000000000001},"max":{"1min":0.754,"5min":0.84099999999999997,"15min":0.92700000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61899999999999999,"15min":0.63300000000000001},"min":{"1min":0.433,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82399999999999995,"5min":0.84099999999999997,"15min":0.97799999999999998},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.79400000000000004,"5min":0.87,"15min":0.87},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.78500000000000003,"5min":0.83199999999999996,"15min":0.83899999999999997},"last":0.67600000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.85499999999999998,"15min":0.89900000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.42599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.754,"5min":0.92500000000000004,"15min":1.1599999999999999},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.79200000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.48999999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76700000000000002,"5min":0.90700000000000003,"15min":0.93300000000000005},"last":0.73699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575971,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47516,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324324,"statfs":{"total":96636764160,"available":96588107776,"internally_reserved":0,"allocated":2498560,"data_stored":1330454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59499999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.64100000000000001,"15min":0.71699999999999997},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.50900000000000001,"5min":0.66600000000000004,"15min":0.70099999999999996},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47599999999999998,"15min":0.47699999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.56699999999999995,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.45000000000000001,"15min":0.44500000000000001},"min":{"1min":0.32600000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":0.70499999999999996},"last":0.375}]},{"osd":4,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.66200000000000003,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":0.97699999999999998,"15min":1.0309999999999999},"last":0.503}]},{"osd":5,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68400000000000005,"15min":0.69599999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94099999999999995,"5min":1.01,"15min":1.0760000000000001},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.442,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86799999999999999,"5min":0.93100000000000005,"15min":1.0149999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.98699999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.88700000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.83799999999999997,"5min":0.99399999999999999,"15min":1.016},"last":0.47799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42540,"kb_used_data":3036,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329300,"statfs":{"total":96636764160,"available":96593203200,"internally_reserved":0,"allocated":3108864,"data_stored":1921550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.26900000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.55500000000000005,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45800000000000002},"min":{"1min":0.376,"5min":0.27800000000000002,"15min":0.255},"max":{"1min":0.58099999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.443,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58799999999999997,"5min":0.64300000000000002,"15min":0.89000000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":0.626,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.40000000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.44700000000000001},"min":{"1min":0.30599999999999999,"5min":0.29599999999999999,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.64700000000000002,"15min":0.86699999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.63600000000000001,"5min":0.67400000000000004,"15min":0.80100000000000005},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.66100000000000003},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77800000000000002,"5min":0.999,"15min":1.1850000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64500000000000002,"15min":0.65900000000000003},"min":{"1min":0.499,"5min":0.47199999999999998,"15min":0.45200000000000001},"max":{"1min":0.872,"5min":0.94799999999999995,"15min":0.96899999999999997},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.45800000000000002},"max":{"1min":0.81699999999999995,"5min":0.86899999999999999,"15min":1.089},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.44700000000000001,"15min":0.438},"max":{"1min":0.95599999999999996,"5min":0.96499999999999997,"15min":1.216},"last":0.495}]},{"osd":6,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69799999999999995,"15min":0.70499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.1020000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69099999999999995,"15min":0.70199999999999996},"min":{"1min":0.57699999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.038,"5min":1.038,"15min":1.1180000000000001},"last":0.57699999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67500000000000004,"15min":0.68200000000000005},"min":{"1min":0.55400000000000005,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.503},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.69099999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.45800000000000002},"max":{"1min":0.88500000000000001,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.46899999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47516,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324324,"statfs":{"total":96636764160,"available":96588107776,"internally_reserved":0,"allocated":2502656,"data_stored":1331036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.307,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55100000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.45400000000000001,"15min":0.45000000000000001},"min":{"1min":0.39200000000000002,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":0.59499999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.34599999999999997,"5min":0.311,"15min":0.29499999999999998},"max":{"1min":0.623,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.27200000000000002},"max":{"1min":0.55900000000000005,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.33100000000000002,"5min":0.29099999999999998,"15min":0.26100000000000001},"max":{"1min":0.59599999999999997,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50700000000000001,"15min":0.51000000000000001},"min":{"1min":0.39500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.77800000000000002},"last":0.41599999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.67000000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93400000000000005,"5min":0.995,"15min":1.097},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.68700000000000006},"min":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92600000000000005,"5min":0.95799999999999996,"15min":1.016},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67400000000000004,"15min":0.68300000000000005},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.435},"max":{"1min":0.87,"5min":0.95099999999999996,"15min":1.0669999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.70199999999999996},"min":{"1min":0.56999999999999995,"5min":0.51200000000000001,"15min":0.48599999999999999},"max":{"1min":1.0169999999999999,"5min":1.022,"15min":1.0389999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":1.022,"15min":1.022},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71199999999999997,"15min":0.71899999999999997},"min":{"1min":0.55600000000000005,"5min":0.504,"15min":0.46899999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.0980000000000001},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.72999999999999998,"15min":0.72199999999999998},"min":{"1min":0.59999999999999998,"5min":0.49099999999999999,"15min":0.45200000000000001},"max":{"1min":0.99099999999999999,"5min":1.0569999999999999,"15min":1.071},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.55300000000000005,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.57299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575970,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51688,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320152,"statfs":{"total":96636764160,"available":96583835648,"internally_reserved":0,"allocated":2519040,"data_stored":1337686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.503,"5min":0.63800000000000001,"15min":0.68700000000000006},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42299999999999999,"15min":0.41999999999999998},"min":{"1min":0.316,"5min":0.27100000000000002,"15min":0.27000000000000002},"max":{"1min":0.503,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41899999999999998,"15min":0.42799999999999999},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.22600000000000001},"max":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.67300000000000004},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.45300000000000001},"min":{"1min":0.375,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":0.53100000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.371,"5min":0.314,"15min":0.314},"max":{"1min":0.66800000000000004,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.35199999999999998,"15min":0.34699999999999998},"max":{"1min":0.58799999999999997,"5min":1.024,"15min":1.024},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48999999999999999,"5min":0.432,"15min":0.42099999999999999},"max":{"1min":0.94699999999999995,"5min":0.97199999999999998,"15min":1.107},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90300000000000002,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.504}]},{"osd":5,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.48799999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"max":{"1min":0.79600000000000004,"5min":1,"15min":1.026},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70199999999999996,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.438},"max":{"1min":0.92200000000000004,"5min":0.99199999999999999,"15min":1.0640000000000001},"last":0.436}]},{"osd":6,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.53300000000000003,"5min":0.46400000000000002,"15min":0.41799999999999998},"max":{"1min":0.92000000000000004,"5min":0.98099999999999998,"15min":1.0629999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70799999999999996,"15min":0.70299999999999996},"min":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.45100000000000001},"max":{"1min":0.878,"5min":1.05,"15min":1.0620000000000001},"last":0.45900000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52900000000000003,"5min":0.47999999999999998,"15min":0.434},"max":{"1min":0.96299999999999997,"5min":0.97699999999999998,"15min":1.0509999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71499999999999997},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93700000000000006,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.51400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575970,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47528,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324312,"statfs":{"total":96636764160,"available":96588095488,"internally_reserved":0,"allocated":2514944,"data_stored":1332100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.45900000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.83499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.624,"15min":0.63800000000000001},"min":{"1min":0.45000000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83699999999999997,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.629,"15min":0.63200000000000001},"min":{"1min":0.45000000000000001,"5min":0.41999999999999998,"15min":0.41399999999999998},"max":{"1min":0.79400000000000004,"5min":0.83899999999999997,"15min":0.90700000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.436,"5min":0.315,"15min":0.315},"max":{"1min":0.80200000000000005,"5min":0.83999999999999997,"15min":0.95499999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63400000000000001,"15min":0.64600000000000002},"min":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.375},"max":{"1min":0.81599999999999995,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.46999999999999997,"5min":0.41999999999999998,"15min":0.41599999999999998},"max":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.63300000000000001,"15min":0.64600000000000002},"min":{"1min":0.44,"5min":0.375,"15min":0.375},"max":{"1min":0.78200000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.495,"15min":0.501},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.32500000000000001},"max":{"1min":0.68500000000000005,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.33400000000000002},"max":{"1min":0.64700000000000002,"5min":0.78000000000000003,"15min":0.82399999999999995},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.30299999999999999},"max":{"1min":0.878,"5min":0.878,"15min":1.0609999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.90500000000000003,"15min":1.115},"last":0.60099999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72699999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60999999999999999},"min":{"1min":0.41699999999999998,"5min":0.38500000000000001,"15min":0.34899999999999998},"max":{"1min":0.80800000000000005,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.58599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575970,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60004,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311836,"statfs":{"total":96636764160,"available":96575320064,"internally_reserved":0,"allocated":2514944,"data_stored":1332088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.502,"5min":0.42199999999999999,"15min":0.28999999999999998},"max":{"1min":0.80900000000000005,"5min":0.85799999999999998,"15min":0.89300000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.66700000000000004},"min":{"1min":0.499,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":0.81399999999999995,"5min":0.88200000000000001,"15min":0.88600000000000001},"last":0.495}]},{"osd":1,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.46000000000000002,"5min":0.42899999999999999,"15min":0.34399999999999997},"max":{"1min":0.78300000000000003,"5min":0.83199999999999996,"15min":0.84399999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60999999999999999},"min":{"1min":0.46899999999999997,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.80300000000000005,"5min":0.86399999999999999,"15min":0.91300000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.70299999999999996,"5min":0.84899999999999998,"15min":0.86499999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":0.89400000000000002,"15min":0.91400000000000003},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.47399999999999998,"5min":0.42399999999999999,"15min":0.373},"max":{"1min":0.748,"5min":0.81000000000000005,"15min":0.88600000000000001},"last":0.434},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.45000000000000001,"5min":0.40999999999999998,"15min":0.39700000000000002},"max":{"1min":0.76200000000000001,"5min":0.83399999999999996,"15min":0.90500000000000003},"last":0.56299999999999994}]},{"osd":4,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.374,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.51400000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.61799999999999999,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.379,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.748,"5min":0.83599999999999997,"15min":0.874},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.45300000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83299999999999996,"5min":0.878,"15min":0.89700000000000002},"last":0.60399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:17:44.813 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:17:44.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.966+0000 7fca83181700 1 Processor -- start 2023-10-16T11:17:44.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.966+0000 7fca83181700 1 -- start start 2023-10-16T11:17:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca83181700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fca7c0b6e10 0x7fca7c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca83181700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 0x7fca7c1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca83181700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c15c5d0 con 0x7fca7c0cdee0 2023-10-16T11:17:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca83181700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c15d2c0 con 0x7fca7c0b6e10 2023-10-16T11:17:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca83181700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c152c20 con 0x7fca7c0ce2c0 2023-10-16T11:17:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca80f1d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fca7c0cdee0 0x7fca7c15bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36722/0 (socket says 172.21.15.158:36722) 2023-10-16T11:17:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca80f1d700 1 -- 172.21.15.158:0/3680823147 learned_addr learned my addr 172.21.15.158:0/3680823147 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3680823147 >> v2:172.21.15.172:3300/0 conn(0x7fca7c0b6e10 0x7fca7c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca8171e700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 0x7fca7c1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446150283 0 0) 0x7fca7c15c5d0 con 0x7fca7c0cdee0 2023-10-16T11:17:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca6c0035b0 con 0x7fca7c0cdee0 2023-10-16T11:17:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3198692817 0 0) 0x7fca6c0035b0 con 0x7fca7c0cdee0 2023-10-16T11:17:44.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 msgr2=0x7fca7c1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 0x7fca7c1526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 >> v2:172.21.15.172:3300/0 conn(0x7fca7c0b6e10 msgr2=0x7fca7c1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 --2- 172.21.15.158:0/3680823147 >> v2:172.21.15.172:3300/0 conn(0x7fca7c0b6e10 0x7fca7c1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c15c8b0 con 0x7fca7c0cdee0 2023-10-16T11:17:44.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca8171e700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 0x7fca7c1526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:44.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fca70001670 con 0x7fca7c0cdee0 2023-10-16T11:17:44.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 >> v1:172.21.15.158:6789/0 conn(0x7fca7c0cdee0 legacy=0x7fca7c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca6c003a30 0x7fca6c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 0x7fca6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c15d2c0 con 0x7fca6c004850 2023-10-16T11:17:44.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c152c20 con 0x7fca6c003a30 2023-10-16T11:17:44.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.967+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c00ba90 con 0x7fca7c0ce2c0 2023-10-16T11:17:44.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca80f1d700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca6c003a30 0x7fca6c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca8171e700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 0x7fca6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7bfff700 1 -- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 msgr2=0x7fca6c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0ce2c0 0x7fca6c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7bfff700 1 -- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca6c003a30 msgr2=0x7fca6c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:44.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca6c003a30 0x7fca6c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7bfff700 1 -- 172.21.15.158:0/3680823147 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c0035b0 con 0x7fca6c004850 2023-10-16T11:17:44.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca6c004cd0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7fca68009b60 tx=0x7fca6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1237fb93e9ebc91f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6800e6c0 con 0x7fca6c004850 2023-10-16T11:17:44.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6800e820 con 0x7fca6c004850 2023-10-16T11:17:44.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.968+0000 7fca7b7fe700 1 -- 172.21.15.158:0/3680823147 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6800eb90 con 0x7fca6c004850 2023-10-16T11:17:44.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.969+0000 7fca83181700 1 -- 172.21.15.158:0/3680823147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 msgr2=0x7fca6c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.969+0000 7fca83181700 1 --2- 172.21.15.158:0/3680823147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca6c004cd0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7fca68009b60 tx=0x7fca6c005880 comp rx=0 tx=0).stop 2023-10-16T11:17:44.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.969+0000 7fca8171e700 1 -- 172.21.15.158:0/3680823147 reap_dead start 2023-10-16T11:17:44.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.969+0000 7fca83181700 1 -- 172.21.15.158:0/3680823147 shutdown_connections 2023-10-16T11:17:44.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.969+0000 7fca83181700 1 -- 172.21.15.158:0/3680823147 >> 172.21.15.158:0/3680823147 conn(0x7fca7c06a8d0 msgr2=0x7fca7c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:44.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.969+0000 7fca83181700 1 -- 172.21.15.158:0/3680823147 shutdown_connections 2023-10-16T11:17:44.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.969+0000 7fca83181700 1 -- 172.21.15.158:0/3680823147 wait complete. 2023-10-16T11:17:44.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.970+0000 7fca83181700 1 Processor -- start 2023-10-16T11:17:44.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.970+0000 7fca83181700 1 -- start start 2023-10-16T11:17:44.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca83181700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca7c14d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca83181700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0b0010 0x7fca7c1f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca83181700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca7c1f5170 0x7fca7c0d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca83181700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c1f4990 con 0x7fca6c004850 2023-10-16T11:17:44.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca83181700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c1f1c80 con 0x7fca7c1f5170 2023-10-16T11:17:44.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca83181700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c1f1f60 con 0x7fca7c0b0010 2023-10-16T11:17:44.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca7c14d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca7c14d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36558/0 (socket says 172.21.15.158:36558) 2023-10-16T11:17:44.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 -- 172.21.15.158:0/3002695861 learned_addr learned my addr 172.21.15.158:0/3002695861 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:44.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0b0010 0x7fca7c1f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca8171e700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca7c1f5170 0x7fca7c0d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 -- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0b0010 msgr2=0x7fca7c1f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0b0010 0x7fca7c1f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 -- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca7c1f5170 msgr2=0x7fca7c0d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:44.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca7c1f5170 0x7fca7c0d11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:44.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 -- 172.21.15.158:0/3002695861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c1f2240 con 0x7fca6c004850 2023-10-16T11:17:44.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0b0010 0x7fca7c1f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:44.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca80f1d700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca7c14d9c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fca700011e0 tx=0x7fca70007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca797fa700 1 -- 172.21.15.158:0/3002695861 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70011580 con 0x7fca6c004850 2023-10-16T11:17:44.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.971+0000 7fca797fa700 1 -- 172.21.15.158:0/3002695861 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca700116e0 con 0x7fca6c004850 2023-10-16T11:17:44.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.972+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c1f4c50 con 0x7fca6c004850 2023-10-16T11:17:44.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.972+0000 7fca797fa700 1 -- 172.21.15.158:0/3002695861 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70011a70 con 0x7fca6c004850 2023-10-16T11:17:44.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.972+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c1f5620 con 0x7fca6c004850 2023-10-16T11:17:44.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.972+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c0681b0 con 0x7fca6c004850 2023-10-16T11:17:44.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.973+0000 7fca797fa700 1 -- 172.21.15.158:0/3002695861 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fca70011bd0 con 0x7fca6c004850 2023-10-16T11:17:44.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.973+0000 7fca797fa700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fca6006b020 0x7fca6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:44.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.973+0000 7fca797fa700 1 -- 172.21.15.158:0/3002695861 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fca70089a90 con 0x7fca6c004850 2023-10-16T11:17:44.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.973+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fca6006b020 0x7fca6006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:44.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.974+0000 7fca7bfff700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fca6006b020 0x7fca6006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fca680090b0 tx=0x7fca68008030 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:44.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:44.978+0000 7fca797fa700 1 -- 172.21.15.158:0/3002695861 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca70054730 con 0x7fca6c004850 2023-10-16T11:17:45.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.163+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 --> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca7c15cee0 con 0x7fca6006b020 2023-10-16T11:17:45.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.171+0000 7fca797fa700 1 -- 172.21.15.158:0/3002695861 <== mgr.10824 v2:172.21.15.158:6832/3585499115 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290163 (secure 0 0 0) 0x7fca6800eb10 con 0x7fca6006b020 2023-10-16T11:17:45.177 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fca6006b020 msgr2=0x7fca6006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fca6006b020 0x7fca6006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fca680090b0 tx=0x7fca68008030 comp rx=0 tx=0).stop 2023-10-16T11:17:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 msgr2=0x7fca7c14d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca7c14d9c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fca700011e0 tx=0x7fca70007570 comp rx=0 tx=0).stop 2023-10-16T11:17:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 shutdown_connections 2023-10-16T11:17:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fca6006b020 0x7fca6006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fca7c1f5170 0x7fca7c0d11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca6c004850 0x7fca7c14d9c0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 --2- 172.21.15.158:0/3002695861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fca7c0b0010 0x7fca7c1f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 >> 172.21.15.158:0/3002695861 conn(0x7fca7c06a8d0 msgr2=0x7fca7c15ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:45.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 shutdown_connections 2023-10-16T11:17:45.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.175+0000 7fca83181700 1 -- 172.21.15.158:0/3002695861 wait complete. 2023-10-16T11:17:45.181 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:17:45.188 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T11:17:44.211191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":386824,"kb_used_data":20756,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754587896,"statfs":{"total":773094113280,"available":772698005504,"internally_reserved":0,"allocated":21254144,"data_stored":11836260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":374722103},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552006"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700776+0000","last_change":"2023-10-16T11:17:40.694497+0000","last_active":"2023-10-16T11:17:40.700776+0000","last_peered":"2023-10-16T11:17:40.700776+0000","last_clean":"2023-10-16T11:17:40.700776+0000","last_became_active":"2023-10-16T11:17:40.694220+0000","last_became_peered":"2023-10-16T11:17:40.694220+0000","last_unstale":"2023-10-16T11:17:40.700776+0000","last_undegraded":"2023-10-16T11:17:40.700776+0000","last_fullsized":"2023-10-16T11:17:40.700776+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:21.672778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700917+0000","last_change":"2023-10-16T11:17:40.695794+0000","last_active":"2023-10-16T11:17:40.700917+0000","last_peered":"2023-10-16T11:17:40.700917+0000","last_clean":"2023-10-16T11:17:40.700917+0000","last_became_active":"2023-10-16T11:17:40.695539+0000","last_became_peered":"2023-10-16T11:17:40.695539+0000","last_unstale":"2023-10-16T11:17:40.700917+0000","last_undegraded":"2023-10-16T11:17:40.700917+0000","last_fullsized":"2023-10-16T11:17:40.700917+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:22.032657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709672+0000","last_change":"2023-10-16T11:17:40.702396+0000","last_active":"2023-10-16T11:17:40.709672+0000","last_peered":"2023-10-16T11:17:40.709672+0000","last_clean":"2023-10-16T11:17:40.709672+0000","last_became_active":"2023-10-16T11:17:40.702104+0000","last_became_peered":"2023-10-16T11:17:40.702104+0000","last_unstale":"2023-10-16T11:17:40.709672+0000","last_undegraded":"2023-10-16T11:17:40.709672+0000","last_fullsized":"2023-10-16T11:17:40.709672+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:44.925107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917705+0000","last_change":"2023-10-16T11:17:40.696695+0000","last_active":"2023-10-16T11:17:41.917705+0000","last_peered":"2023-10-16T11:17:41.917705+0000","last_clean":"2023-10-16T11:17:41.917705+0000","last_became_active":"2023-10-16T11:17:40.693646+0000","last_became_peered":"2023-10-16T11:17:40.693646+0000","last_unstale":"2023-10-16T11:17:41.917705+0000","last_undegraded":"2023-10-16T11:17:41.917705+0000","last_fullsized":"2023-10-16T11:17:41.917705+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:51.324037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702933+0000","last_change":"2023-10-16T11:17:40.695772+0000","last_active":"2023-10-16T11:17:41.702933+0000","last_peered":"2023-10-16T11:17:41.702933+0000","last_clean":"2023-10-16T11:17:41.702933+0000","last_became_active":"2023-10-16T11:17:40.694840+0000","last_became_peered":"2023-10-16T11:17:40.694840+0000","last_unstale":"2023-10-16T11:17:41.702933+0000","last_undegraded":"2023-10-16T11:17:41.702933+0000","last_fullsized":"2023-10-16T11:17:41.702933+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:45:49.801058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702756+0000","last_change":"2023-10-16T11:17:40.690289+0000","last_active":"2023-10-16T11:17:41.702756+0000","last_peered":"2023-10-16T11:17:41.702756+0000","last_clean":"2023-10-16T11:17:41.702756+0000","last_became_active":"2023-10-16T11:17:40.689954+0000","last_became_peered":"2023-10-16T11:17:40.689954+0000","last_unstale":"2023-10-16T11:17:41.702756+0000","last_undegraded":"2023-10-16T11:17:41.702756+0000","last_fullsized":"2023-10-16T11:17:41.702756+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:36.680230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697609+0000","last_change":"2023-10-16T11:17:40.700103+0000","last_active":"2023-10-16T11:17:41.697609+0000","last_peered":"2023-10-16T11:17:41.697609+0000","last_clean":"2023-10-16T11:17:41.697609+0000","last_became_active":"2023-10-16T11:17:40.699829+0000","last_became_peered":"2023-10-16T11:17:40.699829+0000","last_unstale":"2023-10-16T11:17:41.697609+0000","last_undegraded":"2023-10-16T11:17:41.697609+0000","last_fullsized":"2023-10-16T11:17:41.697609+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:11.131193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708532+0000","last_change":"2023-10-16T11:17:40.703979+0000","last_active":"2023-10-16T11:17:40.708532+0000","last_peered":"2023-10-16T11:17:40.708532+0000","last_clean":"2023-10-16T11:17:40.708532+0000","last_became_active":"2023-10-16T11:17:40.703393+0000","last_became_peered":"2023-10-16T11:17:40.703393+0000","last_unstale":"2023-10-16T11:17:40.708532+0000","last_undegraded":"2023-10-16T11:17:40.708532+0000","last_fullsized":"2023-10-16T11:17:40.708532+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:31.159612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701140+0000","last_change":"2023-10-16T11:17:40.698076+0000","last_active":"2023-10-16T11:17:40.701140+0000","last_peered":"2023-10-16T11:17:40.701140+0000","last_clean":"2023-10-16T11:17:40.701140+0000","last_became_active":"2023-10-16T11:17:40.697767+0000","last_became_peered":"2023-10-16T11:17:40.697767+0000","last_unstale":"2023-10-16T11:17:40.701140+0000","last_undegraded":"2023-10-16T11:17:40.701140+0000","last_fullsized":"2023-10-16T11:17:40.701140+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:02.108714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703029+0000","last_change":"2023-10-16T11:17:40.692946+0000","last_active":"2023-10-16T11:17:41.703029+0000","last_peered":"2023-10-16T11:17:41.703029+0000","last_clean":"2023-10-16T11:17:41.703029+0000","last_became_active":"2023-10-16T11:17:40.692079+0000","last_became_peered":"2023-10-16T11:17:40.692079+0000","last_unstale":"2023-10-16T11:17:41.703029+0000","last_undegraded":"2023-10-16T11:17:41.703029+0000","last_fullsized":"2023-10-16T11:17:41.703029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:32.845329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.703985+0000","last_change":"2023-10-16T11:17:40.697631+0000","last_active":"2023-10-16T11:17:40.703985+0000","last_peered":"2023-10-16T11:17:40.703985+0000","last_clean":"2023-10-16T11:17:40.703985+0000","last_became_active":"2023-10-16T11:17:40.697311+0000","last_became_peered":"2023-10-16T11:17:40.697311+0000","last_unstale":"2023-10-16T11:17:40.703985+0000","last_undegraded":"2023-10-16T11:17:40.703985+0000","last_fullsized":"2023-10-16T11:17:40.703985+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:37.784073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917779+0000","last_change":"2023-10-16T11:17:40.695856+0000","last_active":"2023-10-16T11:17:41.917779+0000","last_peered":"2023-10-16T11:17:41.917779+0000","last_clean":"2023-10-16T11:17:41.917779+0000","last_became_active":"2023-10-16T11:17:40.695592+0000","last_became_peered":"2023-10-16T11:17:40.695592+0000","last_unstale":"2023-10-16T11:17:41.917779+0000","last_undegraded":"2023-10-16T11:17:41.917779+0000","last_fullsized":"2023-10-16T11:17:41.917779+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:10.797324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703482+0000","last_change":"2023-10-16T11:17:40.700186+0000","last_active":"2023-10-16T11:17:41.703482+0000","last_peered":"2023-10-16T11:17:41.703482+0000","last_clean":"2023-10-16T11:17:41.703482+0000","last_became_active":"2023-10-16T11:17:40.699545+0000","last_became_peered":"2023-10-16T11:17:40.699545+0000","last_unstale":"2023-10-16T11:17:41.703482+0000","last_undegraded":"2023-10-16T11:17:41.703482+0000","last_fullsized":"2023-10-16T11:17:41.703482+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:50.278406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697639+0000","last_change":"2023-10-16T11:17:40.692724+0000","last_active":"2023-10-16T11:17:41.697639+0000","last_peered":"2023-10-16T11:17:41.697639+0000","last_clean":"2023-10-16T11:17:41.697639+0000","last_became_active":"2023-10-16T11:17:40.692481+0000","last_became_peered":"2023-10-16T11:17:40.692481+0000","last_unstale":"2023-10-16T11:17:41.697639+0000","last_undegraded":"2023-10-16T11:17:41.697639+0000","last_fullsized":"2023-10-16T11:17:41.697639+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:08.747508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.696494+0000","last_change":"2023-10-16T11:17:40.693320+0000","last_active":"2023-10-16T11:17:40.696494+0000","last_peered":"2023-10-16T11:17:40.696494+0000","last_clean":"2023-10-16T11:17:40.696494+0000","last_became_active":"2023-10-16T11:17:40.692889+0000","last_became_peered":"2023-10-16T11:17:40.692889+0000","last_unstale":"2023-10-16T11:17:40.696494+0000","last_undegraded":"2023-10-16T11:17:40.696494+0000","last_fullsized":"2023-10-16T11:17:40.696494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:27.293703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702641+0000","last_change":"2023-10-16T11:17:40.695710+0000","last_active":"2023-10-16T11:17:41.702641+0000","last_peered":"2023-10-16T11:17:41.702641+0000","last_clean":"2023-10-16T11:17:41.702641+0000","last_became_active":"2023-10-16T11:17:40.695423+0000","last_became_peered":"2023-10-16T11:17:40.695423+0000","last_unstale":"2023-10-16T11:17:41.702641+0000","last_undegraded":"2023-10-16T11:17:41.702641+0000","last_fullsized":"2023-10-16T11:17:41.702641+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:10.682957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702918+0000","last_change":"2023-10-16T11:17:40.696331+0000","last_active":"2023-10-16T11:17:41.702918+0000","last_peered":"2023-10-16T11:17:41.702918+0000","last_clean":"2023-10-16T11:17:41.702918+0000","last_became_active":"2023-10-16T11:17:40.696061+0000","last_became_peered":"2023-10-16T11:17:40.696061+0000","last_unstale":"2023-10-16T11:17:41.702918+0000","last_undegraded":"2023-10-16T11:17:41.702918+0000","last_fullsized":"2023-10-16T11:17:41.702918+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:32.691142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703799+0000","last_change":"2023-10-16T11:17:40.697835+0000","last_active":"2023-10-16T11:17:41.703799+0000","last_peered":"2023-10-16T11:17:41.703799+0000","last_clean":"2023-10-16T11:17:41.703799+0000","last_became_active":"2023-10-16T11:17:40.695770+0000","last_became_peered":"2023-10-16T11:17:40.695770+0000","last_unstale":"2023-10-16T11:17:41.703799+0000","last_undegraded":"2023-10-16T11:17:41.703799+0000","last_fullsized":"2023-10-16T11:17:41.703799+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:26.640546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703604+0000","last_change":"2023-10-16T11:17:40.691107+0000","last_active":"2023-10-16T11:17:41.703604+0000","last_peered":"2023-10-16T11:17:41.703604+0000","last_clean":"2023-10-16T11:17:41.703604+0000","last_became_active":"2023-10-16T11:17:40.690833+0000","last_became_peered":"2023-10-16T11:17:40.690833+0000","last_unstale":"2023-10-16T11:17:41.703604+0000","last_undegraded":"2023-10-16T11:17:41.703604+0000","last_fullsized":"2023-10-16T11:17:41.703604+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:44.925107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703085+0000","last_change":"2023-10-16T11:17:40.696392+0000","last_active":"2023-10-16T11:17:41.703085+0000","last_peered":"2023-10-16T11:17:41.703085+0000","last_clean":"2023-10-16T11:17:41.703085+0000","last_became_active":"2023-10-16T11:17:40.696121+0000","last_became_peered":"2023-10-16T11:17:40.696121+0000","last_unstale":"2023-10-16T11:17:41.703085+0000","last_undegraded":"2023-10-16T11:17:41.703085+0000","last_fullsized":"2023-10-16T11:17:41.703085+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:12.243702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701178+0000","last_change":"2023-10-16T11:17:40.698440+0000","last_active":"2023-10-16T11:17:40.701178+0000","last_peered":"2023-10-16T11:17:40.701178+0000","last_clean":"2023-10-16T11:17:40.701178+0000","last_became_active":"2023-10-16T11:17:40.698164+0000","last_became_peered":"2023-10-16T11:17:40.698164+0000","last_unstale":"2023-10-16T11:17:40.701178+0000","last_undegraded":"2023-10-16T11:17:40.701178+0000","last_fullsized":"2023-10-16T11:17:40.701178+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:11.032489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700971+0000","last_change":"2023-10-16T11:17:40.697560+0000","last_active":"2023-10-16T11:17:40.700971+0000","last_peered":"2023-10-16T11:17:40.700971+0000","last_clean":"2023-10-16T11:17:40.700971+0000","last_became_active":"2023-10-16T11:17:40.697244+0000","last_became_peered":"2023-10-16T11:17:40.697244+0000","last_unstale":"2023-10-16T11:17:40.700971+0000","last_undegraded":"2023-10-16T11:17:40.700971+0000","last_fullsized":"2023-10-16T11:17:40.700971+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:38.240047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705902+0000","last_change":"2023-10-16T11:17:40.703106+0000","last_active":"2023-10-16T11:17:40.705902+0000","last_peered":"2023-10-16T11:17:40.705902+0000","last_clean":"2023-10-16T11:17:40.705902+0000","last_became_active":"2023-10-16T11:17:40.702829+0000","last_became_peered":"2023-10-16T11:17:40.702829+0000","last_unstale":"2023-10-16T11:17:40.705902+0000","last_undegraded":"2023-10-16T11:17:40.705902+0000","last_fullsized":"2023-10-16T11:17:40.705902+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:35.114359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701100+0000","last_change":"2023-10-16T11:17:40.698969+0000","last_active":"2023-10-16T11:17:40.701100+0000","last_peered":"2023-10-16T11:17:40.701100+0000","last_clean":"2023-10-16T11:17:40.701100+0000","last_became_active":"2023-10-16T11:17:40.698694+0000","last_became_peered":"2023-10-16T11:17:40.698694+0000","last_unstale":"2023-10-16T11:17:40.701100+0000","last_undegraded":"2023-10-16T11:17:40.701100+0000","last_fullsized":"2023-10-16T11:17:40.701100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:25:18.701459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697477+0000","last_change":"2023-10-16T11:17:40.699654+0000","last_active":"2023-10-16T11:17:41.697477+0000","last_peered":"2023-10-16T11:17:41.697477+0000","last_clean":"2023-10-16T11:17:41.697477+0000","last_became_active":"2023-10-16T11:17:40.699360+0000","last_became_peered":"2023-10-16T11:17:40.699360+0000","last_unstale":"2023-10-16T11:17:41.697477+0000","last_undegraded":"2023-10-16T11:17:41.697477+0000","last_fullsized":"2023-10-16T11:17:41.697477+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:01.988974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705211+0000","last_change":"2023-10-16T11:17:40.702431+0000","last_active":"2023-10-16T11:17:40.705211+0000","last_peered":"2023-10-16T11:17:40.705211+0000","last_clean":"2023-10-16T11:17:40.705211+0000","last_became_active":"2023-10-16T11:17:40.701745+0000","last_became_peered":"2023-10-16T11:17:40.701745+0000","last_unstale":"2023-10-16T11:17:40.705211+0000","last_undegraded":"2023-10-16T11:17:40.705211+0000","last_fullsized":"2023-10-16T11:17:40.705211+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:54:45.421589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708501+0000","last_change":"2023-10-16T11:17:40.703451+0000","last_active":"2023-10-16T11:17:40.708501+0000","last_peered":"2023-10-16T11:17:40.708501+0000","last_clean":"2023-10-16T11:17:40.708501+0000","last_became_active":"2023-10-16T11:17:40.703020+0000","last_became_peered":"2023-10-16T11:17:40.703020+0000","last_unstale":"2023-10-16T11:17:40.708501+0000","last_undegraded":"2023-10-16T11:17:40.708501+0000","last_fullsized":"2023-10-16T11:17:40.708501+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:45:55.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702852+0000","last_change":"2023-10-16T11:17:40.694622+0000","last_active":"2023-10-16T11:17:41.702852+0000","last_peered":"2023-10-16T11:17:41.702852+0000","last_clean":"2023-10-16T11:17:41.702852+0000","last_became_active":"2023-10-16T11:17:40.694348+0000","last_became_peered":"2023-10-16T11:17:40.694348+0000","last_unstale":"2023-10-16T11:17:41.702852+0000","last_undegraded":"2023-10-16T11:17:41.702852+0000","last_fullsized":"2023-10-16T11:17:41.702852+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.852772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.696826+0000","last_change":"2023-10-16T11:17:40.693732+0000","last_active":"2023-10-16T11:17:40.696826+0000","last_peered":"2023-10-16T11:17:40.696826+0000","last_clean":"2023-10-16T11:17:40.696826+0000","last_became_active":"2023-10-16T11:17:40.693400+0000","last_became_peered":"2023-10-16T11:17:40.693400+0000","last_unstale":"2023-10-16T11:17:40.696826+0000","last_undegraded":"2023-10-16T11:17:40.696826+0000","last_fullsized":"2023-10-16T11:17:40.696826+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:30.857760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.696625+0000","last_change":"2023-10-16T11:17:40.692417+0000","last_active":"2023-10-16T11:17:40.696625+0000","last_peered":"2023-10-16T11:17:40.696625+0000","last_clean":"2023-10-16T11:17:40.696625+0000","last_became_active":"2023-10-16T11:17:40.692043+0000","last_became_peered":"2023-10-16T11:17:40.692043+0000","last_unstale":"2023-10-16T11:17:40.696625+0000","last_undegraded":"2023-10-16T11:17:40.696625+0000","last_fullsized":"2023-10-16T11:17:40.696625+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:08.300873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697446+0000","last_change":"2023-10-16T11:17:40.692158+0000","last_active":"2023-10-16T11:17:41.697446+0000","last_peered":"2023-10-16T11:17:41.697446+0000","last_clean":"2023-10-16T11:17:41.697446+0000","last_became_active":"2023-10-16T11:17:40.691311+0000","last_became_peered":"2023-10-16T11:17:40.691311+0000","last_unstale":"2023-10-16T11:17:41.697446+0000","last_undegraded":"2023-10-16T11:17:41.697446+0000","last_fullsized":"2023-10-16T11:17:41.697446+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:49.546837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702785+0000","last_change":"2023-10-16T11:17:40.691547+0000","last_active":"2023-10-16T11:17:41.702785+0000","last_peered":"2023-10-16T11:17:41.702785+0000","last_clean":"2023-10-16T11:17:41.702785+0000","last_became_active":"2023-10-16T11:17:40.691291+0000","last_became_peered":"2023-10-16T11:17:40.691291+0000","last_unstale":"2023-10-16T11:17:41.702785+0000","last_undegraded":"2023-10-16T11:17:41.702785+0000","last_fullsized":"2023-10-16T11:17:41.702785+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:40.386764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709322+0000","last_change":"2023-10-16T11:17:40.705533+0000","last_active":"2023-10-16T11:17:40.709322+0000","last_peered":"2023-10-16T11:17:40.709322+0000","last_clean":"2023-10-16T11:17:40.709322+0000","last_became_active":"2023-10-16T11:17:40.705271+0000","last_became_peered":"2023-10-16T11:17:40.705271+0000","last_unstale":"2023-10-16T11:17:40.709322+0000","last_undegraded":"2023-10-16T11:17:40.709322+0000","last_fullsized":"2023-10-16T11:17:40.709322+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:12.327351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"199'4","reported_seq":16,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917810+0000","last_change":"2023-10-16T11:17:39.675277+0000","last_active":"2023-10-16T11:17:41.917810+0000","last_peered":"2023-10-16T11:17:41.917810+0000","last_clean":"2023-10-16T11:17:41.917810+0000","last_became_active":"2023-10-16T11:17:39.675018+0000","last_became_peered":"2023-10-16T11:17:39.675018+0000","last_unstale":"2023-10-16T11:17:41.917810+0000","last_undegraded":"2023-10-16T11:17:41.917810+0000","last_fullsized":"2023-10-16T11:17:41.917810+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:58.903873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697711+0000","last_change":"2023-10-16T11:17:40.699332+0000","last_active":"2023-10-16T11:17:41.697711+0000","last_peered":"2023-10-16T11:17:41.697711+0000","last_clean":"2023-10-16T11:17:41.697711+0000","last_became_active":"2023-10-16T11:17:40.699049+0000","last_became_peered":"2023-10-16T11:17:40.699049+0000","last_unstale":"2023-10-16T11:17:41.697711+0000","last_undegraded":"2023-10-16T11:17:41.697711+0000","last_fullsized":"2023-10-16T11:17:41.697711+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:38.866372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"199'1","reported_seq":13,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703103+0000","last_change":"2023-10-16T11:17:39.683955+0000","last_active":"2023-10-16T11:17:41.703103+0000","last_peered":"2023-10-16T11:17:41.703103+0000","last_clean":"2023-10-16T11:17:41.703103+0000","last_became_active":"2023-10-16T11:17:39.683703+0000","last_became_peered":"2023-10-16T11:17:39.683703+0000","last_unstale":"2023-10-16T11:17:41.703103+0000","last_undegraded":"2023-10-16T11:17:41.703103+0000","last_fullsized":"2023-10-16T11:17:41.703103+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:33.590796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705176+0000","last_change":"2023-10-16T11:17:40.698772+0000","last_active":"2023-10-16T11:17:40.705176+0000","last_peered":"2023-10-16T11:17:40.705176+0000","last_clean":"2023-10-16T11:17:40.705176+0000","last_became_active":"2023-10-16T11:17:40.698491+0000","last_became_peered":"2023-10-16T11:17:40.698491+0000","last_unstale":"2023-10-16T11:17:40.705176+0000","last_undegraded":"2023-10-16T11:17:40.705176+0000","last_fullsized":"2023-10-16T11:17:40.705176+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:29.439041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"199'2","reported_seq":13,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.711796+0000","last_change":"2023-10-16T11:17:39.673701+0000","last_active":"2023-10-16T11:17:40.711796+0000","last_peered":"2023-10-16T11:17:40.711796+0000","last_clean":"2023-10-16T11:17:40.711796+0000","last_became_active":"2023-10-16T11:17:39.673444+0000","last_became_peered":"2023-10-16T11:17:39.673444+0000","last_unstale":"2023-10-16T11:17:40.711796+0000","last_undegraded":"2023-10-16T11:17:40.711796+0000","last_fullsized":"2023-10-16T11:17:40.711796+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:35.427673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708784+0000","last_change":"2023-10-16T11:17:40.698034+0000","last_active":"2023-10-16T11:17:40.708784+0000","last_peered":"2023-10-16T11:17:40.708784+0000","last_clean":"2023-10-16T11:17:40.708784+0000","last_became_active":"2023-10-16T11:17:40.697399+0000","last_became_peered":"2023-10-16T11:17:40.697399+0000","last_unstale":"2023-10-16T11:17:40.708784+0000","last_undegraded":"2023-10-16T11:17:40.708784+0000","last_fullsized":"2023-10-16T11:17:40.708784+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:38.510740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703568+0000","last_change":"2023-10-16T11:17:39.674379+0000","last_active":"2023-10-16T11:17:41.703568+0000","last_peered":"2023-10-16T11:17:41.703568+0000","last_clean":"2023-10-16T11:17:41.703568+0000","last_became_active":"2023-10-16T11:17:39.674096+0000","last_became_peered":"2023-10-16T11:17:39.674096+0000","last_unstale":"2023-10-16T11:17:41.703568+0000","last_undegraded":"2023-10-16T11:17:41.703568+0000","last_fullsized":"2023-10-16T11:17:41.703568+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:14.587208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703161+0000","last_change":"2023-10-16T11:17:40.700985+0000","last_active":"2023-10-16T11:17:41.703161+0000","last_peered":"2023-10-16T11:17:41.703161+0000","last_clean":"2023-10-16T11:17:41.703161+0000","last_became_active":"2023-10-16T11:17:40.700696+0000","last_became_peered":"2023-10-16T11:17:40.700696+0000","last_unstale":"2023-10-16T11:17:41.703161+0000","last_undegraded":"2023-10-16T11:17:41.703161+0000","last_fullsized":"2023-10-16T11:17:41.703161+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:59.721389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"199'1","reported_seq":13,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703058+0000","last_change":"2023-10-16T11:17:39.683852+0000","last_active":"2023-10-16T11:17:41.703058+0000","last_peered":"2023-10-16T11:17:41.703058+0000","last_clean":"2023-10-16T11:17:41.703058+0000","last_became_active":"2023-10-16T11:17:39.681486+0000","last_became_peered":"2023-10-16T11:17:39.681486+0000","last_unstale":"2023-10-16T11:17:41.703058+0000","last_undegraded":"2023-10-16T11:17:41.703058+0000","last_fullsized":"2023-10-16T11:17:41.703058+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:08.123480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.693073+0000","last_change":"2023-10-16T11:17:40.691203+0000","last_active":"2023-10-16T11:17:40.693073+0000","last_peered":"2023-10-16T11:17:40.693073+0000","last_clean":"2023-10-16T11:17:40.693073+0000","last_became_active":"2023-10-16T11:17:40.690915+0000","last_became_peered":"2023-10-16T11:17:40.690915+0000","last_unstale":"2023-10-16T11:17:40.693073+0000","last_undegraded":"2023-10-16T11:17:40.693073+0000","last_fullsized":"2023-10-16T11:17:40.693073+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:47.296282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"199'1","reported_seq":13,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917572+0000","last_change":"2023-10-16T11:17:39.683877+0000","last_active":"2023-10-16T11:17:41.917572+0000","last_peered":"2023-10-16T11:17:41.917572+0000","last_clean":"2023-10-16T11:17:41.917572+0000","last_became_active":"2023-10-16T11:17:39.683583+0000","last_became_peered":"2023-10-16T11:17:39.683583+0000","last_unstale":"2023-10-16T11:17:41.917572+0000","last_undegraded":"2023-10-16T11:17:41.917572+0000","last_fullsized":"2023-10-16T11:17:41.917572+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:38.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.694412+0000","last_change":"2023-10-16T11:17:40.693377+0000","last_active":"2023-10-16T11:17:40.694412+0000","last_peered":"2023-10-16T11:17:40.694412+0000","last_clean":"2023-10-16T11:17:40.694412+0000","last_became_active":"2023-10-16T11:17:40.693046+0000","last_became_peered":"2023-10-16T11:17:40.693046+0000","last_unstale":"2023-10-16T11:17:40.694412+0000","last_undegraded":"2023-10-16T11:17:40.694412+0000","last_fullsized":"2023-10-16T11:17:40.694412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:38.661148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703280+0000","last_change":"2023-10-16T11:17:39.678628+0000","last_active":"2023-10-16T11:17:41.703280+0000","last_peered":"2023-10-16T11:17:41.703280+0000","last_clean":"2023-10-16T11:17:41.703280+0000","last_became_active":"2023-10-16T11:17:39.678352+0000","last_became_peered":"2023-10-16T11:17:39.678352+0000","last_unstale":"2023-10-16T11:17:41.703280+0000","last_undegraded":"2023-10-16T11:17:41.703280+0000","last_fullsized":"2023-10-16T11:17:41.703280+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:56:19.163238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.710130+0000","last_change":"2023-10-16T11:17:40.709430+0000","last_active":"2023-10-16T11:17:40.710130+0000","last_peered":"2023-10-16T11:17:40.710130+0000","last_clean":"2023-10-16T11:17:40.710130+0000","last_became_active":"2023-10-16T11:17:40.708515+0000","last_became_peered":"2023-10-16T11:17:40.708515+0000","last_unstale":"2023-10-16T11:17:40.710130+0000","last_undegraded":"2023-10-16T11:17:40.710130+0000","last_fullsized":"2023-10-16T11:17:40.710130+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:31.963145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703038+0000","last_change":"2023-10-16T11:17:39.683136+0000","last_active":"2023-10-16T11:17:41.703038+0000","last_peered":"2023-10-16T11:17:41.703038+0000","last_clean":"2023-10-16T11:17:41.703038+0000","last_became_active":"2023-10-16T11:17:39.682216+0000","last_became_peered":"2023-10-16T11:17:39.682216+0000","last_unstale":"2023-10-16T11:17:41.703038+0000","last_undegraded":"2023-10-16T11:17:41.703038+0000","last_fullsized":"2023-10-16T11:17:41.703038+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:59.797957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702752+0000","last_change":"2023-10-16T11:17:40.688250+0000","last_active":"2023-10-16T11:17:41.702752+0000","last_peered":"2023-10-16T11:17:41.702752+0000","last_clean":"2023-10-16T11:17:41.702752+0000","last_became_active":"2023-10-16T11:17:40.688000+0000","last_became_peered":"2023-10-16T11:17:40.688000+0000","last_unstale":"2023-10-16T11:17:41.702752+0000","last_undegraded":"2023-10-16T11:17:41.702752+0000","last_fullsized":"2023-10-16T11:17:41.702752+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:26.210836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.680588+0000","last_change":"2023-10-16T11:17:39.687390+0000","last_active":"2023-10-16T11:17:40.680588+0000","last_peered":"2023-10-16T11:17:40.680588+0000","last_clean":"2023-10-16T11:17:40.680588+0000","last_became_active":"2023-10-16T11:17:39.687111+0000","last_became_peered":"2023-10-16T11:17:39.687111+0000","last_unstale":"2023-10-16T11:17:40.680588+0000","last_undegraded":"2023-10-16T11:17:40.680588+0000","last_fullsized":"2023-10-16T11:17:40.680588+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:56.248764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703934+0000","last_change":"2023-10-16T11:17:40.696043+0000","last_active":"2023-10-16T11:17:41.703934+0000","last_peered":"2023-10-16T11:17:41.703934+0000","last_clean":"2023-10-16T11:17:41.703934+0000","last_became_active":"2023-10-16T11:17:40.695381+0000","last_became_peered":"2023-10-16T11:17:40.695381+0000","last_unstale":"2023-10-16T11:17:41.703934+0000","last_undegraded":"2023-10-16T11:17:41.703934+0000","last_fullsized":"2023-10-16T11:17:41.703934+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:48.707835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"199'3","reported_seq":15,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917687+0000","last_change":"2023-10-16T11:17:39.684566+0000","last_active":"2023-10-16T11:17:41.917687+0000","last_peered":"2023-10-16T11:17:41.917687+0000","last_clean":"2023-10-16T11:17:41.917687+0000","last_became_active":"2023-10-16T11:17:39.684287+0000","last_became_peered":"2023-10-16T11:17:39.684287+0000","last_unstale":"2023-10-16T11:17:41.917687+0000","last_undegraded":"2023-10-16T11:17:41.917687+0000","last_fullsized":"2023-10-16T11:17:41.917687+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:51:55.403425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.700798+0000","last_change":"2023-10-16T11:17:40.697024+0000","last_active":"2023-10-16T11:17:40.700798+0000","last_peered":"2023-10-16T11:17:40.700798+0000","last_clean":"2023-10-16T11:17:40.700798+0000","last_became_active":"2023-10-16T11:17:40.696758+0000","last_became_peered":"2023-10-16T11:17:40.696758+0000","last_unstale":"2023-10-16T11:17:40.700798+0000","last_undegraded":"2023-10-16T11:17:40.700798+0000","last_fullsized":"2023-10-16T11:17:40.700798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:33.929739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"199'6","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703654+0000","last_change":"2023-10-16T11:17:39.670423+0000","last_active":"2023-10-16T11:17:41.703654+0000","last_peered":"2023-10-16T11:17:41.703654+0000","last_clean":"2023-10-16T11:17:41.703654+0000","last_became_active":"2023-10-16T11:17:39.670158+0000","last_became_peered":"2023-10-16T11:17:39.670158+0000","last_unstale":"2023-10-16T11:17:41.703654+0000","last_undegraded":"2023-10-16T11:17:41.703654+0000","last_fullsized":"2023-10-16T11:17:41.703654+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:13.682735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697280+0000","last_change":"2023-10-16T11:17:40.699423+0000","last_active":"2023-10-16T11:17:41.697280+0000","last_peered":"2023-10-16T11:17:41.697280+0000","last_clean":"2023-10-16T11:17:41.697280+0000","last_became_active":"2023-10-16T11:17:40.698999+0000","last_became_peered":"2023-10-16T11:17:40.698999+0000","last_unstale":"2023-10-16T11:17:41.697280+0000","last_undegraded":"2023-10-16T11:17:41.697280+0000","last_fullsized":"2023-10-16T11:17:41.697280+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:01.675519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703684+0000","last_change":"2023-10-16T11:17:39.679123+0000","last_active":"2023-10-16T11:17:41.703684+0000","last_peered":"2023-10-16T11:17:41.703684+0000","last_clean":"2023-10-16T11:17:41.703684+0000","last_became_active":"2023-10-16T11:17:39.678883+0000","last_became_peered":"2023-10-16T11:17:39.678883+0000","last_unstale":"2023-10-16T11:17:41.703684+0000","last_undegraded":"2023-10-16T11:17:41.703684+0000","last_fullsized":"2023-10-16T11:17:41.703684+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:25.492564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703066+0000","last_change":"2023-10-16T11:17:40.697723+0000","last_active":"2023-10-16T11:17:41.703066+0000","last_peered":"2023-10-16T11:17:41.703066+0000","last_clean":"2023-10-16T11:17:41.703066+0000","last_became_active":"2023-10-16T11:17:40.697466+0000","last_became_peered":"2023-10-16T11:17:40.697466+0000","last_unstale":"2023-10-16T11:17:41.703066+0000","last_undegraded":"2023-10-16T11:17:41.703066+0000","last_fullsized":"2023-10-16T11:17:41.703066+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:41.224618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.681174+0000","last_change":"2023-10-16T11:17:39.682261+0000","last_active":"2023-10-16T11:17:40.681174+0000","last_peered":"2023-10-16T11:17:40.681174+0000","last_clean":"2023-10-16T11:17:40.681174+0000","last_became_active":"2023-10-16T11:17:39.681991+0000","last_became_peered":"2023-10-16T11:17:39.681991+0000","last_unstale":"2023-10-16T11:17:40.681174+0000","last_undegraded":"2023-10-16T11:17:40.681174+0000","last_fullsized":"2023-10-16T11:17:40.681174+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:33.523191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703521+0000","last_change":"2023-10-16T11:17:40.691839+0000","last_active":"2023-10-16T11:17:41.703521+0000","last_peered":"2023-10-16T11:17:41.703521+0000","last_clean":"2023-10-16T11:17:41.703521+0000","last_became_active":"2023-10-16T11:17:40.691181+0000","last_became_peered":"2023-10-16T11:17:40.691181+0000","last_unstale":"2023-10-16T11:17:41.703521+0000","last_undegraded":"2023-10-16T11:17:41.703521+0000","last_fullsized":"2023-10-16T11:17:41.703521+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:37.678572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917495+0000","last_change":"2023-10-16T11:17:39.673769+0000","last_active":"2023-10-16T11:17:41.917495+0000","last_peered":"2023-10-16T11:17:41.917495+0000","last_clean":"2023-10-16T11:17:41.917495+0000","last_became_active":"2023-10-16T11:17:39.673483+0000","last_became_peered":"2023-10-16T11:17:39.673483+0000","last_unstale":"2023-10-16T11:17:41.917495+0000","last_undegraded":"2023-10-16T11:17:41.917495+0000","last_fullsized":"2023-10-16T11:17:41.917495+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:36.779758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703912+0000","last_change":"2023-10-16T11:17:40.694569+0000","last_active":"2023-10-16T11:17:41.703912+0000","last_peered":"2023-10-16T11:17:41.703912+0000","last_clean":"2023-10-16T11:17:41.703912+0000","last_became_active":"2023-10-16T11:17:40.694235+0000","last_became_peered":"2023-10-16T11:17:40.694235+0000","last_unstale":"2023-10-16T11:17:41.703912+0000","last_undegraded":"2023-10-16T11:17:41.703912+0000","last_fullsized":"2023-10-16T11:17:41.703912+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:38.510740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"199'8","reported_seq":19,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.714318+0000","last_change":"2023-10-16T11:17:39.675603+0000","last_active":"2023-10-16T11:17:40.714318+0000","last_peered":"2023-10-16T11:17:40.714318+0000","last_clean":"2023-10-16T11:17:40.714318+0000","last_became_active":"2023-10-16T11:17:39.675356+0000","last_became_peered":"2023-10-16T11:17:39.675356+0000","last_unstale":"2023-10-16T11:17:40.714318+0000","last_undegraded":"2023-10-16T11:17:40.714318+0000","last_fullsized":"2023-10-16T11:17:40.714318+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:38.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697433+0000","last_change":"2023-10-16T11:17:40.693857+0000","last_active":"2023-10-16T11:17:41.697433+0000","last_peered":"2023-10-16T11:17:41.697433+0000","last_clean":"2023-10-16T11:17:41.697433+0000","last_became_active":"2023-10-16T11:17:40.693603+0000","last_became_peered":"2023-10-16T11:17:40.693603+0000","last_unstale":"2023-10-16T11:17:41.697433+0000","last_undegraded":"2023-10-16T11:17:41.697433+0000","last_fullsized":"2023-10-16T11:17:41.697433+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:00.469870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"199'1","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.713527+0000","last_change":"2023-10-16T11:17:39.675704+0000","last_active":"2023-10-16T11:17:40.713527+0000","last_peered":"2023-10-16T11:17:40.713527+0000","last_clean":"2023-10-16T11:17:40.713527+0000","last_became_active":"2023-10-16T11:17:39.675418+0000","last_became_peered":"2023-10-16T11:17:39.675418+0000","last_unstale":"2023-10-16T11:17:40.713527+0000","last_undegraded":"2023-10-16T11:17:40.713527+0000","last_fullsized":"2023-10-16T11:17:40.713527+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:33.039808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702804+0000","last_change":"2023-10-16T11:17:40.688585+0000","last_active":"2023-10-16T11:17:41.702804+0000","last_peered":"2023-10-16T11:17:41.702804+0000","last_clean":"2023-10-16T11:17:41.702804+0000","last_became_active":"2023-10-16T11:17:40.687493+0000","last_became_peered":"2023-10-16T11:17:40.687493+0000","last_unstale":"2023-10-16T11:17:41.702804+0000","last_undegraded":"2023-10-16T11:17:41.702804+0000","last_fullsized":"2023-10-16T11:17:41.702804+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:42.635777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917701+0000","last_change":"2023-10-16T11:17:39.674738+0000","last_active":"2023-10-16T11:17:41.917701+0000","last_peered":"2023-10-16T11:17:41.917701+0000","last_clean":"2023-10-16T11:17:41.917701+0000","last_became_active":"2023-10-16T11:17:39.674479+0000","last_became_peered":"2023-10-16T11:17:39.674479+0000","last_unstale":"2023-10-16T11:17:41.917701+0000","last_undegraded":"2023-10-16T11:17:41.917701+0000","last_fullsized":"2023-10-16T11:17:41.917701+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:54.405847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705195+0000","last_change":"2023-10-16T11:17:40.702264+0000","last_active":"2023-10-16T11:17:40.705195+0000","last_peered":"2023-10-16T11:17:40.705195+0000","last_clean":"2023-10-16T11:17:40.705195+0000","last_became_active":"2023-10-16T11:17:40.701407+0000","last_became_peered":"2023-10-16T11:17:40.701407+0000","last_unstale":"2023-10-16T11:17:40.705195+0000","last_undegraded":"2023-10-16T11:17:40.705195+0000","last_fullsized":"2023-10-16T11:17:40.705195+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:03.169653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"199'1","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709719+0000","last_change":"2023-10-16T11:17:39.684443+0000","last_active":"2023-10-16T11:17:40.709719+0000","last_peered":"2023-10-16T11:17:40.709719+0000","last_clean":"2023-10-16T11:17:40.709719+0000","last_became_active":"2023-10-16T11:17:39.684164+0000","last_became_peered":"2023-10-16T11:17:39.684164+0000","last_unstale":"2023-10-16T11:17:40.709719+0000","last_undegraded":"2023-10-16T11:17:40.709719+0000","last_fullsized":"2023-10-16T11:17:40.709719+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:25.636230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.701159+0000","last_change":"2023-10-16T11:17:40.699740+0000","last_active":"2023-10-16T11:17:40.701159+0000","last_peered":"2023-10-16T11:17:40.701159+0000","last_clean":"2023-10-16T11:17:40.701159+0000","last_became_active":"2023-10-16T11:17:40.699468+0000","last_became_peered":"2023-10-16T11:17:40.699468+0000","last_unstale":"2023-10-16T11:17:40.701159+0000","last_undegraded":"2023-10-16T11:17:40.701159+0000","last_fullsized":"2023-10-16T11:17:40.701159+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:20.114136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.691776+0000","last_change":"2023-10-16T11:17:39.676587+0000","last_active":"2023-10-16T11:17:40.691776+0000","last_peered":"2023-10-16T11:17:40.691776+0000","last_clean":"2023-10-16T11:17:40.691776+0000","last_became_active":"2023-10-16T11:17:39.676297+0000","last_became_peered":"2023-10-16T11:17:39.676297+0000","last_unstale":"2023-10-16T11:17:40.691776+0000","last_undegraded":"2023-10-16T11:17:40.691776+0000","last_fullsized":"2023-10-16T11:17:40.691776+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:10.448777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.691976+0000","last_change":"2023-10-16T11:17:40.690713+0000","last_active":"2023-10-16T11:17:40.691976+0000","last_peered":"2023-10-16T11:17:40.691976+0000","last_clean":"2023-10-16T11:17:40.691976+0000","last_became_active":"2023-10-16T11:17:40.690472+0000","last_became_peered":"2023-10-16T11:17:40.690472+0000","last_unstale":"2023-10-16T11:17:40.691976+0000","last_undegraded":"2023-10-16T11:17:40.691976+0000","last_fullsized":"2023-10-16T11:17:40.691976+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:30.669145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703774+0000","last_change":"2023-10-16T11:17:39.681349+0000","last_active":"2023-10-16T11:17:41.703774+0000","last_peered":"2023-10-16T11:17:41.703774+0000","last_clean":"2023-10-16T11:17:41.703774+0000","last_became_active":"2023-10-16T11:17:39.681055+0000","last_became_peered":"2023-10-16T11:17:39.681055+0000","last_unstale":"2023-10-16T11:17:41.703774+0000","last_undegraded":"2023-10-16T11:17:41.703774+0000","last_fullsized":"2023-10-16T11:17:41.703774+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:39.368829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917552+0000","last_change":"2023-10-16T11:17:40.693825+0000","last_active":"2023-10-16T11:17:41.917552+0000","last_peered":"2023-10-16T11:17:41.917552+0000","last_clean":"2023-10-16T11:17:41.917552+0000","last_became_active":"2023-10-16T11:17:40.693406+0000","last_became_peered":"2023-10-16T11:17:40.693406+0000","last_unstale":"2023-10-16T11:17:41.917552+0000","last_undegraded":"2023-10-16T11:17:41.917552+0000","last_fullsized":"2023-10-16T11:17:41.917552+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:34.044124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917727+0000","last_change":"2023-10-16T11:17:39.674089+0000","last_active":"2023-10-16T11:17:41.917727+0000","last_peered":"2023-10-16T11:17:41.917727+0000","last_clean":"2023-10-16T11:17:41.917727+0000","last_became_active":"2023-10-16T11:17:39.673817+0000","last_became_peered":"2023-10-16T11:17:39.673817+0000","last_unstale":"2023-10-16T11:17:41.917727+0000","last_undegraded":"2023-10-16T11:17:41.917727+0000","last_fullsized":"2023-10-16T11:17:41.917727+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:10.448777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917731+0000","last_change":"2023-10-16T11:17:40.694458+0000","last_active":"2023-10-16T11:17:41.917731+0000","last_peered":"2023-10-16T11:17:41.917731+0000","last_clean":"2023-10-16T11:17:41.917731+0000","last_became_active":"2023-10-16T11:17:40.693271+0000","last_became_peered":"2023-10-16T11:17:40.693271+0000","last_unstale":"2023-10-16T11:17:41.917731+0000","last_undegraded":"2023-10-16T11:17:41.917731+0000","last_fullsized":"2023-10-16T11:17:41.917731+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:51.908463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697223+0000","last_change":"2023-10-16T11:17:39.669434+0000","last_active":"2023-10-16T11:17:41.697223+0000","last_peered":"2023-10-16T11:17:41.697223+0000","last_clean":"2023-10-16T11:17:41.697223+0000","last_became_active":"2023-10-16T11:17:39.669186+0000","last_became_peered":"2023-10-16T11:17:39.669186+0000","last_unstale":"2023-10-16T11:17:41.697223+0000","last_undegraded":"2023-10-16T11:17:41.697223+0000","last_fullsized":"2023-10-16T11:17:41.697223+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:59.072882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702683+0000","last_change":"2023-10-16T11:17:40.690074+0000","last_active":"2023-10-16T11:17:41.702683+0000","last_peered":"2023-10-16T11:17:41.702683+0000","last_clean":"2023-10-16T11:17:41.702683+0000","last_became_active":"2023-10-16T11:17:40.689766+0000","last_became_peered":"2023-10-16T11:17:40.689766+0000","last_unstale":"2023-10-16T11:17:41.702683+0000","last_undegraded":"2023-10-16T11:17:41.702683+0000","last_fullsized":"2023-10-16T11:17:41.702683+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:00:41.301924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.688656+0000","last_change":"2023-10-16T11:17:39.676742+0000","last_active":"2023-10-16T11:17:40.688656+0000","last_peered":"2023-10-16T11:17:40.688656+0000","last_clean":"2023-10-16T11:17:40.688656+0000","last_became_active":"2023-10-16T11:17:39.676477+0000","last_became_peered":"2023-10-16T11:17:39.676477+0000","last_unstale":"2023-10-16T11:17:40.688656+0000","last_undegraded":"2023-10-16T11:17:40.688656+0000","last_fullsized":"2023-10-16T11:17:40.688656+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:49.095636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697546+0000","last_change":"2023-10-16T11:17:40.695234+0000","last_active":"2023-10-16T11:17:41.697546+0000","last_peered":"2023-10-16T11:17:41.697546+0000","last_clean":"2023-10-16T11:17:41.697546+0000","last_became_active":"2023-10-16T11:17:40.694963+0000","last_became_peered":"2023-10-16T11:17:40.694963+0000","last_unstale":"2023-10-16T11:17:41.697546+0000","last_undegraded":"2023-10-16T11:17:41.697546+0000","last_fullsized":"2023-10-16T11:17:41.697546+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:27.403827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"199'2","reported_seq":14,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917811+0000","last_change":"2023-10-16T11:17:39.676664+0000","last_active":"2023-10-16T11:17:41.917811+0000","last_peered":"2023-10-16T11:17:41.917811+0000","last_clean":"2023-10-16T11:17:41.917811+0000","last_became_active":"2023-10-16T11:17:39.676211+0000","last_became_peered":"2023-10-16T11:17:39.676211+0000","last_unstale":"2023-10-16T11:17:41.917811+0000","last_undegraded":"2023-10-16T11:17:41.917811+0000","last_fullsized":"2023-10-16T11:17:41.917811+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:07.296557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.690582+0000","last_change":"2023-10-16T11:17:40.689614+0000","last_active":"2023-10-16T11:17:40.690582+0000","last_peered":"2023-10-16T11:17:40.690582+0000","last_clean":"2023-10-16T11:17:40.690582+0000","last_became_active":"2023-10-16T11:17:40.689341+0000","last_became_peered":"2023-10-16T11:17:40.689341+0000","last_unstale":"2023-10-16T11:17:40.690582+0000","last_undegraded":"2023-10-16T11:17:40.690582+0000","last_fullsized":"2023-10-16T11:17:40.690582+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:16.262244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697762+0000","last_change":"2023-10-16T11:17:39.670518+0000","last_active":"2023-10-16T11:17:41.697762+0000","last_peered":"2023-10-16T11:17:41.697762+0000","last_clean":"2023-10-16T11:17:41.697762+0000","last_became_active":"2023-10-16T11:17:39.670260+0000","last_became_peered":"2023-10-16T11:17:39.670260+0000","last_unstale":"2023-10-16T11:17:41.697762+0000","last_undegraded":"2023-10-16T11:17:41.697762+0000","last_fullsized":"2023-10-16T11:17:41.697762+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:01.529436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708572+0000","last_change":"2023-10-16T11:17:40.703935+0000","last_active":"2023-10-16T11:17:40.708572+0000","last_peered":"2023-10-16T11:17:40.708572+0000","last_clean":"2023-10-16T11:17:40.708572+0000","last_became_active":"2023-10-16T11:17:40.703641+0000","last_became_peered":"2023-10-16T11:17:40.703641+0000","last_unstale":"2023-10-16T11:17:40.708572+0000","last_undegraded":"2023-10-16T11:17:40.708572+0000","last_fullsized":"2023-10-16T11:17:40.708572+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:24.300682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702932+0000","last_change":"2023-10-16T11:17:39.682853+0000","last_active":"2023-10-16T11:17:41.702932+0000","last_peered":"2023-10-16T11:17:41.702932+0000","last_clean":"2023-10-16T11:17:41.702932+0000","last_became_active":"2023-10-16T11:17:39.682595+0000","last_became_peered":"2023-10-16T11:17:39.682595+0000","last_unstale":"2023-10-16T11:17:41.702932+0000","last_undegraded":"2023-10-16T11:17:41.702932+0000","last_fullsized":"2023-10-16T11:17:41.702932+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:37.010036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697553+0000","last_change":"2023-10-16T11:17:40.695544+0000","last_active":"2023-10-16T11:17:41.697553+0000","last_peered":"2023-10-16T11:17:41.697553+0000","last_clean":"2023-10-16T11:17:41.697553+0000","last_became_active":"2023-10-16T11:17:40.695271+0000","last_became_peered":"2023-10-16T11:17:40.695271+0000","last_unstale":"2023-10-16T11:17:41.697553+0000","last_undegraded":"2023-10-16T11:17:41.697553+0000","last_fullsized":"2023-10-16T11:17:41.697553+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:15.875869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.702939+0000","last_change":"2023-10-16T11:17:39.685403+0000","last_active":"2023-10-16T11:17:41.702939+0000","last_peered":"2023-10-16T11:17:41.702939+0000","last_clean":"2023-10-16T11:17:41.702939+0000","last_became_active":"2023-10-16T11:17:39.685168+0000","last_became_peered":"2023-10-16T11:17:39.685168+0000","last_unstale":"2023-10-16T11:17:41.702939+0000","last_undegraded":"2023-10-16T11:17:41.702939+0000","last_fullsized":"2023-10-16T11:17:41.702939+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:21.717775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.708995+0000","last_change":"2023-10-16T11:17:40.700629+0000","last_active":"2023-10-16T11:17:40.708995+0000","last_peered":"2023-10-16T11:17:40.708995+0000","last_clean":"2023-10-16T11:17:40.708995+0000","last_became_active":"2023-10-16T11:17:40.700334+0000","last_became_peered":"2023-10-16T11:17:40.700334+0000","last_unstale":"2023-10-16T11:17:40.708995+0000","last_undegraded":"2023-10-16T11:17:40.708995+0000","last_fullsized":"2023-10-16T11:17:40.708995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:59.721389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.693658+0000","last_change":"2023-10-16T11:17:39.680002+0000","last_active":"2023-10-16T11:17:40.693658+0000","last_peered":"2023-10-16T11:17:40.693658+0000","last_clean":"2023-10-16T11:17:40.693658+0000","last_became_active":"2023-10-16T11:17:39.679711+0000","last_became_peered":"2023-10-16T11:17:39.679711+0000","last_unstale":"2023-10-16T11:17:40.693658+0000","last_undegraded":"2023-10-16T11:17:40.693658+0000","last_fullsized":"2023-10-16T11:17:40.693658+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:39.988284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.703255+0000","last_change":"2023-10-16T11:17:40.699067+0000","last_active":"2023-10-16T11:17:41.703255+0000","last_peered":"2023-10-16T11:17:41.703255+0000","last_clean":"2023-10-16T11:17:41.703255+0000","last_became_active":"2023-10-16T11:17:40.698789+0000","last_became_peered":"2023-10-16T11:17:40.698789+0000","last_unstale":"2023-10-16T11:17:41.703255+0000","last_undegraded":"2023-10-16T11:17:41.703255+0000","last_fullsized":"2023-10-16T11:17:41.703255+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:50.099951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.692684+0000","last_change":"2023-10-16T11:17:39.685541+0000","last_active":"2023-10-16T11:17:40.692684+0000","last_peered":"2023-10-16T11:17:40.692684+0000","last_clean":"2023-10-16T11:17:40.692684+0000","last_became_active":"2023-10-16T11:17:39.685324+0000","last_became_peered":"2023-10-16T11:17:39.685324+0000","last_unstale":"2023-10-16T11:17:40.692684+0000","last_undegraded":"2023-10-16T11:17:40.692684+0000","last_fullsized":"2023-10-16T11:17:40.692684+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:53.860398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.694390+0000","last_change":"2023-10-16T11:17:40.693330+0000","last_active":"2023-10-16T11:17:40.694390+0000","last_peered":"2023-10-16T11:17:40.694390+0000","last_clean":"2023-10-16T11:17:40.694390+0000","last_became_active":"2023-10-16T11:17:40.692609+0000","last_became_peered":"2023-10-16T11:17:40.692609+0000","last_unstale":"2023-10-16T11:17:40.694390+0000","last_undegraded":"2023-10-16T11:17:40.694390+0000","last_fullsized":"2023-10-16T11:17:40.694390+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:10.608318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.697532+0000","last_change":"2023-10-16T11:17:39.677643+0000","last_active":"2023-10-16T11:17:41.697532+0000","last_peered":"2023-10-16T11:17:41.697532+0000","last_clean":"2023-10-16T11:17:41.697532+0000","last_became_active":"2023-10-16T11:17:39.677170+0000","last_became_peered":"2023-10-16T11:17:39.677170+0000","last_unstale":"2023-10-16T11:17:41.697532+0000","last_undegraded":"2023-10-16T11:17:41.697532+0000","last_fullsized":"2023-10-16T11:17:41.697532+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:18.641776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.705872+0000","last_change":"2023-10-16T11:17:40.702439+0000","last_active":"2023-10-16T11:17:40.705872+0000","last_peered":"2023-10-16T11:17:40.705872+0000","last_clean":"2023-10-16T11:17:40.705872+0000","last_became_active":"2023-10-16T11:17:40.702140+0000","last_became_peered":"2023-10-16T11:17:40.702140+0000","last_unstale":"2023-10-16T11:17:40.705872+0000","last_undegraded":"2023-10-16T11:17:40.705872+0000","last_fullsized":"2023-10-16T11:17:40.705872+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:15:43.365279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.688800+0000","last_change":"2023-10-16T11:17:39.687613+0000","last_active":"2023-10-16T11:17:40.688800+0000","last_peered":"2023-10-16T11:17:40.688800+0000","last_clean":"2023-10-16T11:17:40.688800+0000","last_became_active":"2023-10-16T11:17:39.687403+0000","last_became_peered":"2023-10-16T11:17:39.687403+0000","last_unstale":"2023-10-16T11:17:40.688800+0000","last_undegraded":"2023-10-16T11:17:40.688800+0000","last_fullsized":"2023-10-16T11:17:40.688800+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:49.274091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-10-16T11:17:40.709202+0000","last_change":"2023-10-16T11:17:40.707377+0000","last_active":"2023-10-16T11:17:40.709202+0000","last_peered":"2023-10-16T11:17:40.709202+0000","last_clean":"2023-10-16T11:17:40.709202+0000","last_became_active":"2023-10-16T11:17:40.707085+0000","last_became_peered":"2023-10-16T11:17:40.707085+0000","last_unstale":"2023-10-16T11:17:40.709202+0000","last_undegraded":"2023-10-16T11:17:40.709202+0000","last_fullsized":"2023-10-16T11:17:40.709202+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:39.658746+0000","last_clean_scrub_stamp":"2023-10-16T11:17:39.658746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:37.678572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-16T11:17:41.917515+0000","last_change":"2023-10-16T11:17:39.674731+0000","last_active":"2023-10-16T11:17:41.917515+0000","last_peered":"2023-10-16T11:17:41.917515+0000","last_clean":"2023-10-16T11:17:41.917515+0000","last_became_active":"2023-10-16T11:17:39.674462+0000","last_became_peered":"2023-10-16T11:17:39.674462+0000","last_unstale":"2023-10-16T11:17:41.917515+0000","last_undegraded":"2023-10-16T11:17:41.917515+0000","last_fullsized":"2023-10-16T11:17:41.917515+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:17:38.654431+0000","last_clean_scrub_stamp":"2023-10-16T11:17:38.654431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:30:28.434726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"190'326","reported_seq":1114,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-10-16T11:17:39.671226+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:17:39.671226+0000","last_peered":"2023-10-16T11:17:39.671226+0000","last_clean":"2023-10-16T11:17:39.671226+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:17:39.671226+0000","last_undegraded":"2023-10-16T11:17:39.671226+0000","last_fullsized":"2023-10-16T11:17:39.671226+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575970,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52248,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319592,"statfs":{"total":96636764160,"available":96583262208,"internally_reserved":0,"allocated":3092480,"data_stored":1920822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.374,"5min":0.372,"15min":0.372},"max":{"1min":0.76200000000000001,"5min":0.84299999999999997,"15min":0.98499999999999999},"last":0.629},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72899999999999998,"5min":0.79500000000000004,"15min":0.999},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.64600000000000002},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.39500000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":1.0640000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.81799999999999995,"5min":0.83499999999999996,"15min":1.0349999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78800000000000003,"5min":0.84199999999999997,"15min":1.117},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66100000000000003,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.41899999999999998},"max":{"1min":0.75700000000000001,"5min":0.90400000000000003,"15min":1.05},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.39500000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78400000000000003,"5min":0.83999999999999997,"15min":1.022},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67800000000000005,"15min":0.68799999999999994},"min":{"1min":0.51600000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.85499999999999998,"15min":1.083},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.50900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.313},"max":{"1min":0.71199999999999997,"5min":0.81599999999999995,"15min":0.94799999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.52300000000000002,"15min":0.53800000000000003},"min":{"1min":0.372,"5min":0.35299999999999998,"15min":0.34399999999999997},"max":{"1min":0.71699999999999997,"5min":0.83299999999999996,"15min":0.96299999999999997},"last":0.64500000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53100000000000003,"15min":0.54300000000000004},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77100000000000002,"5min":0.83899999999999997,"15min":0.91200000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.42599999999999999,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.80700000000000005,"5min":0.85299999999999998,"15min":0.92000000000000004},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:17:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61599999999999999},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.373},"max":{"1min":0.86099999999999999,"5min":0.88500000000000001,"15min":0.92600000000000005},"last":0.747},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.45100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88600000000000001,"5min":0.88700000000000001,"15min":0.91300000000000003},"last":0.58899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575970,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37784,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334056,"statfs":{"total":96636764160,"available":96598073344,"internally_reserved":0,"allocated":2502656,"data_stored":1330524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57899999999999996,"15min":0.59899999999999998},"min":{"1min":0.29499999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70999999999999996,"5min":0.81000000000000005,"15min":0.91000000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.78200000000000003,"5min":0.82599999999999996,"15min":0.88500000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":0.85099999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60199999999999998,"15min":0.628},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.95499999999999996},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.371,"5min":0.314,"15min":0.314},"max":{"1min":0.78600000000000003,"5min":0.83099999999999996,"15min":0.95599999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.622,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72999999999999998,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.61499999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.624},"min":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.32300000000000001},"max":{"1min":0.754,"5min":0.84099999999999997,"15min":0.92700000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61899999999999999,"15min":0.63300000000000001},"min":{"1min":0.433,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82399999999999995,"5min":0.84099999999999997,"15min":0.97799999999999998},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.79400000000000004,"5min":0.87,"15min":0.87},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.78500000000000003,"5min":0.83199999999999996,"15min":0.83899999999999997},"last":0.67600000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.85499999999999998,"15min":0.89900000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.42599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.754,"5min":0.92500000000000004,"15min":1.1599999999999999},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.79200000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.48999999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76700000000000002,"5min":0.90700000000000003,"15min":0.93300000000000005},"last":0.73699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575971,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47516,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324324,"statfs":{"total":96636764160,"available":96588107776,"internally_reserved":0,"allocated":2498560,"data_stored":1330454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59499999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.64100000000000001,"15min":0.71699999999999997},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.50900000000000001,"5min":0.66600000000000004,"15min":0.70099999999999996},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47599999999999998,"15min":0.47699999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.56699999999999995,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.45000000000000001,"15min":0.44500000000000001},"min":{"1min":0.32600000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":0.70499999999999996},"last":0.375}]},{"osd":4,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.66200000000000003,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":0.97699999999999998,"15min":1.0309999999999999},"last":0.503}]},{"osd":5,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68400000000000005,"15min":0.69599999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94099999999999995,"5min":1.01,"15min":1.0760000000000001},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.442,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86799999999999999,"5min":0.93100000000000005,"15min":1.0149999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.98699999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:17:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.88700000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.83799999999999997,"5min":0.99399999999999999,"15min":1.016},"last":0.47799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42540,"kb_used_data":3036,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329300,"statfs":{"total":96636764160,"available":96593203200,"internally_reserved":0,"allocated":3108864,"data_stored":1921550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.26900000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.55500000000000005,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45800000000000002},"min":{"1min":0.376,"5min":0.27800000000000002,"15min":0.255},"max":{"1min":0.58099999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.443,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58799999999999997,"5min":0.64300000000000002,"15min":0.89000000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":0.626,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.40000000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.44700000000000001},"min":{"1min":0.30599999999999999,"5min":0.29599999999999999,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.64700000000000002,"15min":0.86699999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.63600000000000001,"5min":0.67400000000000004,"15min":0.80100000000000005},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.66100000000000003},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77800000000000002,"5min":0.999,"15min":1.1850000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64500000000000002,"15min":0.65900000000000003},"min":{"1min":0.499,"5min":0.47199999999999998,"15min":0.45200000000000001},"max":{"1min":0.872,"5min":0.94799999999999995,"15min":0.96899999999999997},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.45800000000000002},"max":{"1min":0.81699999999999995,"5min":0.86899999999999999,"15min":1.089},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.44700000000000001,"15min":0.438},"max":{"1min":0.95599999999999996,"5min":0.96499999999999997,"15min":1.216},"last":0.495}]},{"osd":6,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69799999999999995,"15min":0.70499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.1020000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69099999999999995,"15min":0.70199999999999996},"min":{"1min":0.57699999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.038,"5min":1.038,"15min":1.1180000000000001},"last":0.57699999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:17:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67500000000000004,"15min":0.68200000000000005},"min":{"1min":0.55400000000000005,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.503},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.69099999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.45800000000000002},"max":{"1min":0.88500000000000001,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.46899999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47516,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324324,"statfs":{"total":96636764160,"available":96588107776,"internally_reserved":0,"allocated":2502656,"data_stored":1331036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.307,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55100000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.45400000000000001,"15min":0.45000000000000001},"min":{"1min":0.39200000000000002,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":0.59499999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.34599999999999997,"5min":0.311,"15min":0.29499999999999998},"max":{"1min":0.623,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.27200000000000002},"max":{"1min":0.55900000000000005,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.33100000000000002,"5min":0.29099999999999998,"15min":0.26100000000000001},"max":{"1min":0.59599999999999997,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50700000000000001,"15min":0.51000000000000001},"min":{"1min":0.39500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.77800000000000002},"last":0.41599999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.67000000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93400000000000005,"5min":0.995,"15min":1.097},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.68700000000000006},"min":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92600000000000005,"5min":0.95799999999999996,"15min":1.016},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67400000000000004,"15min":0.68300000000000005},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.435},"max":{"1min":0.87,"5min":0.95099999999999996,"15min":1.0669999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.70199999999999996},"min":{"1min":0.56999999999999995,"5min":0.51200000000000001,"15min":0.48599999999999999},"max":{"1min":1.0169999999999999,"5min":1.022,"15min":1.0389999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":1.022,"15min":1.022},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71199999999999997,"15min":0.71899999999999997},"min":{"1min":0.55600000000000005,"5min":0.504,"15min":0.46899999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.0980000000000001},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.72999999999999998,"15min":0.72199999999999998},"min":{"1min":0.59999999999999998,"5min":0.49099999999999999,"15min":0.45200000000000001},"max":{"1min":0.99099999999999999,"5min":1.0569999999999999,"15min":1.071},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.55300000000000005,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.57299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575970,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51688,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320152,"statfs":{"total":96636764160,"available":96583835648,"internally_reserved":0,"allocated":2519040,"data_stored":1337686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.503,"5min":0.63800000000000001,"15min":0.68700000000000006},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42299999999999999,"15min":0.41999999999999998},"min":{"1min":0.316,"5min":0.27100000000000002,"15min":0.27000000000000002},"max":{"1min":0.503,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41899999999999998,"15min":0.42799999999999999},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.22600000000000001},"max":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.67300000000000004},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.45300000000000001},"min":{"1min":0.375,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":0.53100000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.371,"5min":0.314,"15min":0.314},"max":{"1min":0.66800000000000004,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.35199999999999998,"15min":0.34699999999999998},"max":{"1min":0.58799999999999997,"5min":1.024,"15min":1.024},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48999999999999999,"5min":0.432,"15min":0.42099999999999999},"max":{"1min":0.94699999999999995,"5min":0.97199999999999998,"15min":1.107},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90300000000000002,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.504}]},{"osd":5,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.48799999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"max":{"1min":0.79600000000000004,"5min":1,"15min":1.026},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70199999999999996,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.438},"max":{"1min":0.92200000000000004,"5min":0.99199999999999999,"15min":1.0640000000000001},"last":0.436}]},{"osd":6,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.53300000000000003,"5min":0.46400000000000002,"15min":0.41799999999999998},"max":{"1min":0.92000000000000004,"5min":0.98099999999999998,"15min":1.0629999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70799999999999996,"15min":0.70299999999999996},"min":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.45100000000000001},"max":{"1min":0.878,"5min":1.05,"15min":1.0620000000000001},"last":0.45900000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:17:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52900000000000003,"5min":0.47999999999999998,"15min":0.434},"max":{"1min":0.96299999999999997,"5min":0.97699999999999998,"15min":1.0509999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71499999999999997},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93700000000000006,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.51400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575970,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47528,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324312,"statfs":{"total":96636764160,"available":96588095488,"internally_reserved":0,"allocated":2514944,"data_stored":1332100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.45900000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.83499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.624,"15min":0.63800000000000001},"min":{"1min":0.45000000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83699999999999997,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.629,"15min":0.63200000000000001},"min":{"1min":0.45000000000000001,"5min":0.41999999999999998,"15min":0.41399999999999998},"max":{"1min":0.79400000000000004,"5min":0.83899999999999997,"15min":0.90700000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.436,"5min":0.315,"15min":0.315},"max":{"1min":0.80200000000000005,"5min":0.83999999999999997,"15min":0.95499999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63400000000000001,"15min":0.64600000000000002},"min":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.375},"max":{"1min":0.81599999999999995,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.46999999999999997,"5min":0.41999999999999998,"15min":0.41599999999999998},"max":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.63300000000000001,"15min":0.64600000000000002},"min":{"1min":0.44,"5min":0.375,"15min":0.375},"max":{"1min":0.78200000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.495,"15min":0.501},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.32500000000000001},"max":{"1min":0.68500000000000005,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.33400000000000002},"max":{"1min":0.64700000000000002,"5min":0.78000000000000003,"15min":0.82399999999999995},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.30299999999999999},"max":{"1min":0.878,"5min":0.878,"15min":1.0609999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.90500000000000003,"15min":1.115},"last":0.60099999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72699999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60999999999999999},"min":{"1min":0.41699999999999998,"5min":0.38500000000000001,"15min":0.34899999999999998},"max":{"1min":0.80800000000000005,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.58599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575970,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60004,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311836,"statfs":{"total":96636764160,"available":96575320064,"internally_reserved":0,"allocated":2514944,"data_stored":1332088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.502,"5min":0.42199999999999999,"15min":0.28999999999999998},"max":{"1min":0.80900000000000005,"5min":0.85799999999999998,"15min":0.89300000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.66700000000000004},"min":{"1min":0.499,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":0.81399999999999995,"5min":0.88200000000000001,"15min":0.88600000000000001},"last":0.495}]},{"osd":1,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.46000000000000002,"5min":0.42899999999999999,"15min":0.34399999999999997},"max":{"1min":0.78300000000000003,"5min":0.83199999999999996,"15min":0.84399999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60999999999999999},"min":{"1min":0.46899999999999997,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.80300000000000005,"5min":0.86399999999999999,"15min":0.91300000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.70299999999999996,"5min":0.84899999999999998,"15min":0.86499999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":0.89400000000000002,"15min":0.91400000000000003},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.47399999999999998,"5min":0.42399999999999999,"15min":0.373},"max":{"1min":0.748,"5min":0.81000000000000005,"15min":0.88600000000000001},"last":0.434},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.45000000000000001,"5min":0.40999999999999998,"15min":0.39700000000000002},"max":{"1min":0.76200000000000001,"5min":0.83399999999999996,"15min":0.90500000000000003},"last":0.56299999999999994}]},{"osd":4,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.374,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.51400000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.61799999999999999,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.379,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.748,"5min":0.83599999999999997,"15min":0.874},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.45300000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83299999999999996,"5min":0.878,"15min":0.89700000000000002},"last":0.60399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:17:45.195 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:17:45.195 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T11:17:45.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.364+0000 7f19c3ec1700 1 Processor -- start 2023-10-16T11:17:45.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.365+0000 7f19c3ec1700 1 -- start start 2023-10-16T11:17:45.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.365+0000 7f19c3ec1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 0x7f19bc0724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.365+0000 7f19c3ec1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc071d70 con 0x7f19bc0af5f0 2023-10-16T11:17:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.365+0000 7f19c3ec1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc0af9d0 con 0x7f19bc072a20 2023-10-16T11:17:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.365+0000 7f19c3ec1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0b3970 con 0x7f19bc0b3b60 2023-10-16T11:17:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c1c5d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f19bc0af5f0 0x7f19bc161e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36742/0 (socket says 172.21.15.158:36742) 2023-10-16T11:17:45.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c1c5d700 1 -- 172.21.15.158:0/1096131554 learned_addr learned my addr 172.21.15.158:0/1096131554 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c145c700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 0x7f19bc0724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997894667 0 0) 0x7f19bc071d70 con 0x7f19bc0af5f0 2023-10-16T11:17:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a80035b0 con 0x7f19bc0af5f0 2023-10-16T11:17:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684039902 0 0) 0x7f19bc0af9d0 con 0x7f19bc072a20 2023-10-16T11:17:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc071d70 con 0x7f19bc072a20 2023-10-16T11:17:45.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598402492 0 0) 0x7f19a80035b0 con 0x7f19bc0af5f0 2023-10-16T11:17:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 msgr2=0x7f19bc0724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 0x7f19bc0724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 >> v1:172.21.15.172:6789/0 conn(0x7f19bc072a20 legacy=0x7f19bc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc0b4060 con 0x7f19bc0af5f0 2023-10-16T11:17:45.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f19ac001670 con 0x7f19bc0af5f0 2023-10-16T11:17:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 >> v1:172.21.15.158:6789/0 conn(0x7f19bc0af5f0 legacy=0x7f19bc161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19a8003bc0 0x7f19a8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 0x7f19a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8004d90 0x7f19a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0b3970 con 0x7f19a8003bc0 2023-10-16T11:17:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a800bac0 con 0x7f19a8004d90 2023-10-16T11:17:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a800bc30 con 0x7f19bc0b3b60 2023-10-16T11:17:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c1c5d700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19a8003bc0 0x7f19a8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c145c700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 0x7f19a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.366+0000 7f19c245e700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8004d90 0x7f19a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c1c5d700 1 -- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 msgr2=0x7f19a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:45.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c1c5d700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b3b60 0x7f19a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c1c5d700 1 -- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8004d90 msgr2=0x7f19a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:45.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c1c5d700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8004d90 0x7f19a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c1c5d700 1 -- 172.21.15.158:0/1096131554 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a80035b0 con 0x7f19a8003bc0 2023-10-16T11:17:45.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c1c5d700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19a8003bc0 0x7f19a8003fc0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00c210 tx=0x7f19a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106b0d7986d7641d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:45.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00cdb0 con 0x7f19a8003bc0 2023-10-16T11:17:45.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac001af0 con 0x7f19a8003bc0 2023-10-16T11:17:45.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c0c5b700 1 -- 172.21.15.158:0/1096131554 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac005ba0 con 0x7f19a8003bc0 2023-10-16T11:17:45.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c3ec1700 1 -- 172.21.15.158:0/1096131554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19a8003bc0 msgr2=0x7f19a8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c3ec1700 1 --2- 172.21.15.158:0/1096131554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19a8003bc0 0x7f19a8003fc0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00c210 tx=0x7f19a8005880 comp rx=0 tx=0).stop 2023-10-16T11:17:45.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.367+0000 7f19c245e700 1 -- 172.21.15.158:0/1096131554 reap_dead start 2023-10-16T11:17:45.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.368+0000 7f19c3ec1700 1 -- 172.21.15.158:0/1096131554 shutdown_connections 2023-10-16T11:17:45.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.368+0000 7f19c3ec1700 1 -- 172.21.15.158:0/1096131554 >> 172.21.15.158:0/1096131554 conn(0x7f19bc0ccd30 msgr2=0x7f19bc0cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:45.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.368+0000 7f19c3ec1700 1 -- 172.21.15.158:0/1096131554 shutdown_connections 2023-10-16T11:17:45.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.368+0000 7f19c3ec1700 1 -- 172.21.15.158:0/1096131554 wait complete. 2023-10-16T11:17:45.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.369+0000 7f19c3ec1700 1 Processor -- start 2023-10-16T11:17:45.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.369+0000 7f19c3ec1700 1 -- start start 2023-10-16T11:17:45.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8003bc0 0x7f19bc0c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc0b2720 0x7f19bc155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b04e0 0x7f19bc1f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc155660 con 0x7f19bc0b2720 2023-10-16T11:17:45.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0b2ba0 con 0x7f19a8003bc0 2023-10-16T11:17:45.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c9ac0 con 0x7f19bc0b04e0 2023-10-16T11:17:45.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc0b2720 0x7f19bc155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc0b2720 0x7f19bc155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36580/0 (socket says 172.21.15.158:36580) 2023-10-16T11:17:45.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 -- 172.21.15.158:0/2733117781 learned_addr learned my addr 172.21.15.158:0/2733117781 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:45.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c1c5d700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8003bc0 0x7f19bc0c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c245e700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b04e0 0x7f19bc1f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 -- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b04e0 msgr2=0x7f19bc1f6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b04e0 0x7f19bc1f6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 -- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8003bc0 msgr2=0x7f19bc0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8003bc0 0x7f19bc0c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 -- 172.21.15.158:0/2733117781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc0c9da0 con 0x7f19bc0b2720 2023-10-16T11:17:45.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c245e700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b04e0 0x7f19bc1f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:45.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c145c700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc0b2720 0x7f19bc155120 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f19b8011aa0 tx=0x7f19b8011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:45.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19b27fc700 1 -- 172.21.15.158:0/2733117781 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b800f940 con 0x7f19bc0b2720 2023-10-16T11:17:45.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19b27fc700 1 -- 172.21.15.158:0/2733117781 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b800faa0 con 0x7f19bc0b2720 2023-10-16T11:17:45.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc0ca080 con 0x7f19bc0b2720 2023-10-16T11:17:45.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19b27fc700 1 -- 172.21.15.158:0/2733117781 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b800fe50 con 0x7f19bc0b2720 2023-10-16T11:17:45.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.370+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc155af0 con 0x7f19bc0b2720 2023-10-16T11:17:45.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.371+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc0681b0 con 0x7f19bc0b2720 2023-10-16T11:17:45.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.372+0000 7f19b27fc700 1 -- 172.21.15.158:0/2733117781 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f19b8014440 con 0x7f19bc0b2720 2023-10-16T11:17:45.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.372+0000 7f19b27fc700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f19a006b050 0x7f19a006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.372+0000 7f19b27fc700 1 -- 172.21.15.158:0/2733117781 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f19b808a150 con 0x7f19bc0b2720 2023-10-16T11:17:45.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.372+0000 7f19c1c5d700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f19a006b050 0x7f19a006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.372+0000 7f19c1c5d700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f19a006b050 0x7f19a006d510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19ac000c00 tx=0x7f19ac00a000 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:45.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.376+0000 7f19b27fc700 1 -- 172.21.15.158:0/2733117781 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19b8054f50 con 0x7f19bc0b2720 2023-10-16T11:17:45.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.647+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f19bc0cd960 con 0x7f19bc0b2720 2023-10-16T11:17:45.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.648+0000 7f19b27fc700 1 -- 172.21.15.158:0/2733117781 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) v1 ==== 190+0+147 (secure 0 0 0) 0x7f19b80220b0 con 0x7f19bc0b2720 2023-10-16T11:17:45.650 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T11:17:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T11:17:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T11:17:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T11:17:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T11:17:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f19a006b050 msgr2=0x7f19a006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f19a006b050 0x7f19a006d510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19ac000c00 tx=0x7f19ac00a000 comp rx=0 tx=0).stop 2023-10-16T11:17:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc0b2720 msgr2=0x7f19bc155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc0b2720 0x7f19bc155120 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f19b8011aa0 tx=0x7f19b8011db0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 shutdown_connections 2023-10-16T11:17:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc0b04e0 0x7f19bc1f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f19a006b050 0x7f19a006d510 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a8003bc0 0x7f19bc0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 --2- 172.21.15.158:0/2733117781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc0b2720 0x7f19bc155120 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 >> 172.21.15.158:0/2733117781 conn(0x7f19bc0ccd30 msgr2=0x7f19bc06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 shutdown_connections 2023-10-16T11:17:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.652+0000 7f19c3ec1700 1 -- 172.21.15.158:0/2733117781 wait complete. 2023-10-16T11:17:45.656 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T11:17:45.665 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:17:45.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.786+0000 7f7b68de2700 1 Processor -- start 2023-10-16T11:17:45.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.786+0000 7f7b68de2700 1 -- start start 2023-10-16T11:17:45.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b68de2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b64152580 0x7f7b640aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b68de2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7b640ca080 0x7f7b640cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b68de2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6406aa70 con 0x7f7b6415f0b0 2023-10-16T11:17:45.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b68de2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6406ad50 con 0x7f7b640ca080 2023-10-16T11:17:45.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b68de2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6406b030 con 0x7f7b64152580 2023-10-16T11:17:45.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b61d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7b640ca080 0x7f7b640cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b62d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7b6415f0b0 0x7f7b6415f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36766/0 (socket says 172.21.15.158:36766) 2023-10-16T11:17:45.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b62d9d700 1 -- 172.21.15.158:0/2095040427 learned_addr learned my addr 172.21.15.158:0/2095040427 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:45.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b61d9b700 1 -- 172.21.15.158:0/2095040427 >> v2:172.21.15.172:3300/0 conn(0x7f7b64152580 msgr2=0x7f7b640aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:45.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b61d9b700 1 --2- 172.21.15.158:0/2095040427 >> v2:172.21.15.172:3300/0 conn(0x7f7b64152580 0x7f7b640aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b61d9b700 1 -- 172.21.15.158:0/2095040427 >> v1:172.21.15.158:6789/0 conn(0x7f7b6415f0b0 legacy=0x7f7b6415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b61d9b700 1 -- 172.21.15.158:0/2095040427 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6406b310 con 0x7f7b640ca080 2023-10-16T11:17:45.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.787+0000 7f7b6159a700 1 -- 172.21.15.158:0/2095040427 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436194538 0 0) 0x7f7b6406aa70 con 0x7f7b6415f0b0 2023-10-16T11:17:45.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b61d9b700 1 --2- 172.21.15.158:0/2095040427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b640cdaf0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f7b640725b0 tx=0x7f7b54009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eac446f24d14842 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:45.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b6159a700 1 -- 172.21.15.158:0/2095040427 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54011830 con 0x7f7b640ca080 2023-10-16T11:17:45.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b6159a700 1 -- 172.21.15.158:0/2095040427 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b54011990 con 0x7f7b640ca080 2023-10-16T11:17:45.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b6159a700 1 -- 172.21.15.158:0/2095040427 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54011d20 con 0x7f7b640ca080 2023-10-16T11:17:45.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 -- 172.21.15.158:0/2095040427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 msgr2=0x7f7b640cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2095040427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b640cdaf0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f7b640725b0 tx=0x7f7b54009b20 comp rx=0 tx=0).stop 2023-10-16T11:17:45.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 -- 172.21.15.158:0/2095040427 shutdown_connections 2023-10-16T11:17:45.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2095040427 >> v2:172.21.15.172:3300/0 conn(0x7f7b64152580 0x7f7b640aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2095040427 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b640cdaf0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 -- 172.21.15.158:0/2095040427 >> 172.21.15.158:0/2095040427 conn(0x7f7b640cc4b0 msgr2=0x7f7b640cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:45.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 -- 172.21.15.158:0/2095040427 shutdown_connections 2023-10-16T11:17:45.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.788+0000 7f7b68de2700 1 -- 172.21.15.158:0/2095040427 wait complete. 2023-10-16T11:17:45.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 Processor -- start 2023-10-16T11:17:45.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 -- start start 2023-10-16T11:17:45.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b64146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b64152580 0x7f7b640c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b6415f0b0 0x7f7b640c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640b4a10 con 0x7f7b640ca080 2023-10-16T11:17:45.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640b4cf0 con 0x7f7b6415f0b0 2023-10-16T11:17:45.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.790+0000 7f7b68de2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640b4fd0 con 0x7f7b64152580 2023-10-16T11:17:45.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b64146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b64146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36592/0 (socket says 172.21.15.158:36592) 2023-10-16T11:17:45.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 -- 172.21.15.158:0/2885669018 learned_addr learned my addr 172.21.15.158:0/2885669018 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:45.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b61d9b700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b64152580 0x7f7b640c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b62d9d700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b6415f0b0 0x7f7b640c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 -- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b64152580 msgr2=0x7f7b640c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b64152580 0x7f7b640c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 -- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b6415f0b0 msgr2=0x7f7b640c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:45.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b6415f0b0 0x7f7b640c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:45.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 -- 172.21.15.158:0/2885669018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b540094a0 con 0x7f7b640ca080 2023-10-16T11:17:45.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b6259c700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b64146aa0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800b530 tx=0x7f7b5800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b52ffd700 1 -- 172.21.15.158:0/2885669018 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58011830 con 0x7f7b640ca080 2023-10-16T11:17:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b52ffd700 1 -- 172.21.15.158:0/2885669018 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b580119b0 con 0x7f7b640ca080 2023-10-16T11:17:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b640c6fd0 con 0x7f7b640ca080 2023-10-16T11:17:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b52ffd700 1 -- 172.21.15.158:0/2885669018 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58011d60 con 0x7f7b640ca080 2023-10-16T11:17:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.791+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b640b4690 con 0x7f7b640ca080 2023-10-16T11:17:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.792+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b64066df0 con 0x7f7b640ca080 2023-10-16T11:17:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.793+0000 7f7b52ffd700 1 -- 172.21.15.158:0/2885669018 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b5801a400 con 0x7f7b640ca080 2023-10-16T11:17:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.793+0000 7f7b52ffd700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7b4406b020 0x7f7b4406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.793+0000 7f7b52ffd700 1 -- 172.21.15.158:0/2885669018 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f7b58089b30 con 0x7f7b640ca080 2023-10-16T11:17:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.793+0000 7f7b61d9b700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7b4406b020 0x7f7b4406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.794+0000 7f7b61d9b700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7b4406b020 0x7f7b4406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7b540090b0 tx=0x7f7b54008030 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:45.798+0000 7f7b52ffd700 1 -- 172.21.15.158:0/2885669018 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b580547c0 con 0x7f7b640ca080 2023-10-16T11:17:45.991 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:17:45.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:17:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.039+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b640681b0 con 0x7f7b640ca080 2023-10-16T11:17:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.040+0000 7f7b52ffd700 1 -- 172.21.15.158:0/2885669018 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f7b58015020 con 0x7f7b640ca080 2023-10-16T11:17:46.041 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:46.042 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":103,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2912297853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2912297853},{"type":"v1","addr":"172.21.15.158:6836","nonce":2912297853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":38441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2297507228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2297507228},{"type":"v1","addr":"172.21.15.158:6837","nonce":2297507228}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38447,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3795197457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3795197457},{"type":"v1","addr":"172.21.15.172:6834","nonce":3795197457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:17:40.072523+0000","modified":"2023-10-16T11:17:43.106178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38450":{"gid":38450,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1279760034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1279760034},{"type":"v1","addr":"172.21.15.172:6835","nonce":1279760034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T11:17:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7b4406b020 msgr2=0x7f7b4406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7b4406b020 0x7f7b4406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7b540090b0 tx=0x7f7b54008030 comp rx=0 tx=0).stop 2023-10-16T11:17:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 msgr2=0x7f7b64146aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b64146aa0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800b530 tx=0x7f7b5800b8f0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 shutdown_connections 2023-10-16T11:17:46.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7b4406b020 0x7f7b4406d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b6415f0b0 0x7f7b640c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b640ca080 0x7f7b64146aa0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 --2- 172.21.15.158:0/2885669018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b64152580 0x7f7b640c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 >> 172.21.15.158:0/2885669018 conn(0x7f7b640cc4b0 msgr2=0x7f7b640cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 shutdown_connections 2023-10-16T11:17:46.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.042+0000 7f7b68de2700 1 -- 172.21.15.158:0/2885669018 wait complete. 2023-10-16T11:17:46.046 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 103 2023-10-16T11:17:46.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:17:40.072523+0000', 'modified': '2023-10-16T11:17:43.106178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 38450}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_38450': {'gid': 38450, 'name': 'b', 'rank': 0, 'incarnation': 99, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/1279760034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 1279760034}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 1279760034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:17:46.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:17:46.459 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T11:17:46.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.580+0000 7f05a8801700 1 Processor -- start 2023-10-16T11:17:46.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.580+0000 7f05a8801700 1 -- start start 2023-10-16T11:17:46.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a8801700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05a415a6d0 0x7f05a40ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a8801700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05a40cfe30 0x7f05a40ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a8801700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a40cd190 0x7f05a406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a8801700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a40afcd0 con 0x7f05a40cfe30 2023-10-16T11:17:46.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a8801700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a40cd620 con 0x7f05a415a6d0 2023-10-16T11:17:46.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a8801700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a415ec70 con 0x7f05a40cd190 2023-10-16T11:17:46.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05a40cfe30 0x7f05a40ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05a40cfe30 0x7f05a40ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36600/0 (socket says 172.21.15.158:36600) 2023-10-16T11:17:46.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.581+0000 7f05a1d9b700 1 -- 172.21.15.158:0/3810896336 learned_addr learned my addr 172.21.15.158:0/3810896336 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:46.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/3810896336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a40cd190 0x7f05a406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a1d9b700 1 -- 172.21.15.158:0/3810896336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a40cd190 msgr2=0x7f05a406ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a259c700 1 --2- 172.21.15.158:0/3810896336 >> v2:172.21.15.172:3300/0 conn(0x7f05a415a6d0 0x7f05a40ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/3810896336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a40cd190 0x7f05a406ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a1d9b700 1 -- 172.21.15.158:0/3810896336 >> v2:172.21.15.172:3300/0 conn(0x7f05a415a6d0 msgr2=0x7f05a40ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/3810896336 >> v2:172.21.15.172:3300/0 conn(0x7f05a415a6d0 0x7f05a40ce2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a1d9b700 1 -- 172.21.15.158:0/3810896336 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a415f1b0 con 0x7f05a40cfe30 2023-10-16T11:17:46.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/3810896336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cfe30 0x7f05a40ccc50 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f05a406c770 tx=0x7f0594009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5406de45d0f739 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a159a700 1 -- 172.21.15.158:0/3810896336 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059400b870 con 0x7f05a40cfe30 2023-10-16T11:17:46.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a159a700 1 -- 172.21.15.158:0/3810896336 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059400b9d0 con 0x7f05a40cfe30 2023-10-16T11:17:46.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a159a700 1 -- 172.21.15.158:0/3810896336 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059400bd60 con 0x7f05a40cfe30 2023-10-16T11:17:46.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a8801700 1 -- 172.21.15.158:0/3810896336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cfe30 msgr2=0x7f05a40ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.582+0000 7f05a8801700 1 --2- 172.21.15.158:0/3810896336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cfe30 0x7f05a40ccc50 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f05a406c770 tx=0x7f0594009bd0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.583+0000 7f05a8801700 1 -- 172.21.15.158:0/3810896336 shutdown_connections 2023-10-16T11:17:46.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.583+0000 7f05a8801700 1 --2- 172.21.15.158:0/3810896336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a40cd190 0x7f05a406ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.583+0000 7f05a8801700 1 --2- 172.21.15.158:0/3810896336 >> v2:172.21.15.172:3300/0 conn(0x7f05a415a6d0 0x7f05a40ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.583+0000 7f05a8801700 1 --2- 172.21.15.158:0/3810896336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cfe30 0x7f05a40ccc50 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.583+0000 7f05a8801700 1 -- 172.21.15.158:0/3810896336 >> 172.21.15.158:0/3810896336 conn(0x7f05a406b9e0 msgr2=0x7f05a406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.583+0000 7f05a8801700 1 -- 172.21.15.158:0/3810896336 shutdown_connections 2023-10-16T11:17:46.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.583+0000 7f05a8801700 1 -- 172.21.15.158:0/3810896336 wait complete. 2023-10-16T11:17:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.584+0000 7f05a8801700 1 Processor -- start 2023-10-16T11:17:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.584+0000 7f05a8801700 1 -- start start 2023-10-16T11:17:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a8801700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cd190 0x7f05a41f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a8801700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05a40cfe30 0x7f05a4157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a8801700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415a6d0 0x7f05a41ef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a8801700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a41efb20 con 0x7f05a40cd190 2023-10-16T11:17:46.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a8801700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40c9870 con 0x7f05a40cfe30 2023-10-16T11:17:46.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a8801700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40c9b50 con 0x7f05a415a6d0 2023-10-16T11:17:46.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cd190 0x7f05a41f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cd190 0x7f05a41f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36606/0 (socket says 172.21.15.158:36606) 2023-10-16T11:17:46.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 -- 172.21.15.158:0/1081243112 learned_addr learned my addr 172.21.15.158:0/1081243112 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:46.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05a40cfe30 0x7f05a4157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415a6d0 0x7f05a41ef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 -- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415a6d0 msgr2=0x7f05a41ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415a6d0 0x7f05a41ef5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 -- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05a40cfe30 msgr2=0x7f05a4157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05a40cfe30 0x7f05a4157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 -- 172.21.15.158:0/1081243112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594009550 con 0x7f05a40cd190 2023-10-16T11:17:46.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a259c700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cd190 0x7f05a41f2c90 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f059800c7f0 tx=0x7f059800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f0592ffd700 1 -- 172.21.15.158:0/1081243112 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598011800 con 0x7f05a40cd190 2023-10-16T11:17:46.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f0592ffd700 1 -- 172.21.15.158:0/1081243112 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0598011960 con 0x7f05a40cd190 2023-10-16T11:17:46.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a41f21b0 con 0x7f05a40cd190 2023-10-16T11:17:46.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.585+0000 7f0592ffd700 1 -- 172.21.15.158:0/1081243112 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598011cd0 con 0x7f05a40cd190 2023-10-16T11:17:46.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.586+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a41f2770 con 0x7f05a40cd190 2023-10-16T11:17:46.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.587+0000 7f0592ffd700 1 -- 172.21.15.158:0/1081243112 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0598004020 con 0x7f05a40cd190 2023-10-16T11:17:46.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.587+0000 7f0592ffd700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f058406b020 0x7f058406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.587+0000 7f0592ffd700 1 -- 172.21.15.158:0/1081243112 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f05980895c0 con 0x7f05a40cd190 2023-10-16T11:17:46.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.587+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f058406b020 0x7f058406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.588+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f058406b020 0x7f058406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f05a40d0270 tx=0x7f0594008030 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.589+0000 7f05b49ebb80 1 Processor -- start 2023-10-16T11:17:46.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.589+0000 7f05b49ebb80 1 -- start start 2023-10-16T11:17:46.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 0x55deb703cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb703c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7063300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb6fb0c20 con 0x55deb7063f30 2023-10-16T11:17:46.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb6f95c10 con 0x55deb7037f00 2023-10-16T11:17:46.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55deb6f93530 con 0x55deb703cb40 2023-10-16T11:17:46.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb703c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb703c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36616/0 (socket says 172.21.15.158:36616) 2023-10-16T11:17:46.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 -- 172.21.15.158:0/48521332 learned_addr learned my addr 172.21.15.158:0/48521332 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:46.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a259c700 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 0x55deb703cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7063300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 -- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 msgr2=0x55deb703cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 0x55deb703cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 -- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 msgr2=0x55deb7063300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7063300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.590+0000 7f05a1d9b700 1 -- 172.21.15.158:0/48521332 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55deb6f4c3a0 con 0x55deb7063f30 2023-10-16T11:17:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb703c280 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f0594009fd0 tx=0x7f0594010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d93b9b624f9fde server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f0583fff700 1 -- 172.21.15.158:0/48521332 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594020070 con 0x55deb7063f30 2023-10-16T11:17:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f0583fff700 1 -- 172.21.15.158:0/48521332 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05940036a0 con 0x55deb7063f30 2023-10-16T11:17:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f0583fff700 1 -- 172.21.15.158:0/48521332 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594003a10 con 0x55deb7063f30 2023-10-16T11:17:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f05b49ebb80 1 -- 172.21.15.158:0/48521332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 msgr2=0x55deb703c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb703c280 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f0594009fd0 tx=0x7f0594010000 comp rx=0 tx=0).stop 2023-10-16T11:17:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f05b49ebb80 1 -- 172.21.15.158:0/48521332 shutdown_connections 2023-10-16T11:17:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7063300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 0x55deb703cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.591+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/48521332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb703c280 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.592+0000 7f05b49ebb80 1 -- 172.21.15.158:0/48521332 >> 172.21.15.158:0/48521332 conn(0x55deb6f69d80 msgr2=0x55deb7084ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.592+0000 7f05b49ebb80 1 -- 172.21.15.158:0/48521332 shutdown_connections 2023-10-16T11:17:46.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.592+0000 7f05b49ebb80 1 -- 172.21.15.158:0/48521332 wait complete. 2023-10-16T11:17:46.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.592+0000 7f05b49ebb80 1 -- start start 2023-10-16T11:17:46.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.592+0000 7f05b49ebb80 1 Processor -- start 2023-10-16T11:17:46.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7068a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 0x55deb7060760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb7060ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb6f8ee20 con 0x55deb7063f30 2023-10-16T11:17:46.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb6f2b6c0 con 0x55deb7037f00 2023-10-16T11:17:46.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55deb6f2b360 con 0x55deb703cb40 2023-10-16T11:17:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05b49ebb80 10 client.0 _ll_register_callbacks cb 0x55deb6d887b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:17:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb7060ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb7060ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36624/0 (socket says 172.21.15.158:36624) 2023-10-16T11:17:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.593+0000 7f05a2d9d700 1 -- 172.21.15.158:0/2454872213 learned_addr learned my addr 172.21.15.158:0/2454872213 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:46.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05a2d9d700 1 -- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 msgr2=0x55deb7060760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 0x55deb7060760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05a2d9d700 1 -- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 msgr2=0x55deb7068a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05a259c700 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7068a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7068a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05a2d9d700 1 -- 172.21.15.158:0/2454872213 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a415f1b0 con 0x55deb7063f30 2023-10-16T11:17:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb7060ca0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f059c00fbd0 tx=0x7f059c00ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae26602bfa67a1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05817fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:17:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05817fa700 1 -- 172.21.15.158:0/2454872213 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c014070 con 0x55deb7063f30 2023-10-16T11:17:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05817fa700 1 -- 172.21.15.158:0/2454872213 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f059c00dbd0 con 0x55deb7063f30 2023-10-16T11:17:46.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55deb702f5b0 con 0x55deb7063f30 2023-10-16T11:17:46.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05817fa700 1 -- 172.21.15.158:0/2454872213 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) v2 ==== 24+0+0 (secure 0 0 0) 0x7f059c00dd90 con 0x55deb7063f30 2023-10-16T11:17:46.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 10 client.11193 fetch_fsmap learned FSMap version 103 2023-10-16T11:17:46.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) v3 -- 0x55deb6db9d90 con 0x55deb7063f30 2023-10-16T11:17:46.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05817fa700 1 -- 172.21.15.158:0/2454872213 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55deb702f5b0 con 0x55deb7063f30 2023-10-16T11:17:46.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05817fa700 1 -- 172.21.15.158:0/2454872213 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 103) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f059c018410 con 0x55deb7063f30 2023-10-16T11:17:46.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 10 client.11193 fetch_fsmap finished waiting for FSMap version 103 2023-10-16T11:17:46.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 10 client.11193 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:17:46.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 20 client.11193 populate_metadata read hostname 'smithi158' 2023-10-16T11:17:46.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb705d270 0x55deb70a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 4 client.11193 mds_command: new command op to 38450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:17:46.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.594+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 --> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55deb6f87e00 con 0x55deb705d270 2023-10-16T11:17:46.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.595+0000 7f05a259c700 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb705d270 0x55deb70a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.595+0000 7f05817fa700 10 client.11193 ms_handle_connect on v2:172.21.15.172:6833/1279760034 2023-10-16T11:17:46.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.595+0000 7f05a259c700 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb705d270 0x55deb70a2d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a03cb02bdfb3e0a0 server_cookie=e7031598daf506ad in_seq=0 out_seq=0 2023-10-16T11:17:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05817fa700 1 -- 172.21.15.158:0/2454872213 <== mds.0 v2:172.21.15.172:6833/1279760034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55deb6f87e00 con 0x55deb705d270 2023-10-16T11:17:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05817fa700 10 client.11193 handle_command_reply: tid=0 2023-10-16T11:17:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 client.11193 shutdown 2023-10-16T11:17:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 msgr2=0x55deb7060ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb7060ca0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f059c00fbd0 tx=0x7f059c00ff90 comp rx=0 tx=0).stop 2023-10-16T11:17:46.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 shutdown_connections 2023-10-16T11:17:46.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7063f30 0x55deb7060ca0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb7068a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb705d270 0x55deb70a2d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/2454872213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb703cb40 0x55deb7060760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05817fa700 0 client.11193 ms_handle_reset on v2:172.21.15.172:6833/1279760034 2023-10-16T11:17:46.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 >> 172.21.15.158:0/2454872213 conn(0x55deb6f69d80 msgr2=0x55deb7036ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 shutdown_connections 2023-10-16T11:17:46.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 1 -- 172.21.15.158:0/2454872213 wait complete. 2023-10-16T11:17:46.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.596+0000 7f05b49ebb80 20 client.11193 trim_cache size 0 max 16384 2023-10-16T11:17:46.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.613+0000 7f05b49ebb80 1 Processor -- start 2023-10-16T11:17:46.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.613+0000 7f05b49ebb80 1 -- start start 2023-10-16T11:17:46.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb703da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb703cb40 0x55deb7038850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb6e0f8d0 con 0x55deb703cb40 2023-10-16T11:17:46.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb7069790 con 0x55deb7037f00 2023-10-16T11:17:46.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55deb70a75f0 con 0x55deb7063f30 2023-10-16T11:17:46.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb703cb40 0x55deb7038850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb703cb40 0x55deb7038850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36630/0 (socket says 172.21.15.158:36630) 2023-10-16T11:17:46.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 -- 172.21.15.158:0/1120141516 learned_addr learned my addr 172.21.15.158:0/1120141516 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:46.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 -- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 msgr2=0x55deb7060930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a259c700 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb703da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 -- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 msgr2=0x55deb703da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb703da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a1d9b700 1 -- 172.21.15.158:0/1120141516 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55deb6da2c10 con 0x55deb703cb40 2023-10-16T11:17:46.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a259c700 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb703da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:46.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.614+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:46.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb703cb40 0x55deb7038850 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x55deb703df70 tx=0x7f059400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f43d8f441a235e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f0590ff9700 1 -- 172.21.15.158:0/1120141516 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594003b50 con 0x55deb703cb40 2023-10-16T11:17:46.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f0590ff9700 1 -- 172.21.15.158:0/1120141516 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0594003cd0 con 0x55deb703cb40 2023-10-16T11:17:46.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f0590ff9700 1 -- 172.21.15.158:0/1120141516 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594032470 con 0x55deb703cb40 2023-10-16T11:17:46.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 -- 172.21.15.158:0/1120141516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb703cb40 msgr2=0x55deb7038850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb703cb40 0x55deb7038850 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x55deb703df70 tx=0x7f059400e000 comp rx=0 tx=0).stop 2023-10-16T11:17:46.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 -- 172.21.15.158:0/1120141516 shutdown_connections 2023-10-16T11:17:46.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb7037f00 0x55deb703da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/1120141516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb703cb40 0x55deb7038850 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 -- 172.21.15.158:0/1120141516 >> 172.21.15.158:0/1120141516 conn(0x55deb6f69d80 msgr2=0x55deb7067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 -- 172.21.15.158:0/1120141516 shutdown_connections 2023-10-16T11:17:46.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.615+0000 7f05b49ebb80 1 -- 172.21.15.158:0/1120141516 wait complete. 2023-10-16T11:17:46.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.616+0000 7f05b49ebb80 1 -- start start 2023-10-16T11:17:46.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.616+0000 7f05b49ebb80 1 Processor -- start 2023-10-16T11:17:46.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7037f00 0x55deb70ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb703cb40 0x55deb70976a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb70980b0 con 0x55deb7037f00 2023-10-16T11:17:46.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb7037d40 con 0x55deb703cb40 2023-10-16T11:17:46.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55deb7095410 con 0x55deb7063f30 2023-10-16T11:17:46.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 10 client.0 _ll_register_callbacks cb 0x7f059c00f490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:17:46.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7037f00 0x55deb70ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7037f00 0x55deb70ec6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36634/0 (socket says 172.21.15.158:36634) 2023-10-16T11:17:46.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 -- 172.21.15.158:0/3843038861 learned_addr learned my addr 172.21.15.158:0/3843038861 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:46.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a2d9d700 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb703cb40 0x55deb70976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 -- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 msgr2=0x55deb7060930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 -- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb703cb40 msgr2=0x55deb70976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb703cb40 0x55deb70976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 -- 172.21.15.158:0/3843038861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55deb6f4c3a0 con 0x55deb7037f00 2023-10-16T11:17:46.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05a259c700 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7037f00 0x55deb70ec6d0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f059800ce50 tx=0x7f0598057a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a4f35a1638cf99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05817fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:17:46.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05817fa700 1 -- 172.21.15.158:0/3843038861 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059805a040 con 0x55deb7037f00 2023-10-16T11:17:46.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05817fa700 1 -- 172.21.15.158:0/3843038861 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f0598082ea0 con 0x55deb7037f00 2023-10-16T11:17:46.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.617+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05940036a0 con 0x55deb7037f00 2023-10-16T11:17:46.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05817fa700 1 -- 172.21.15.158:0/3843038861 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) v2 ==== 24+0+0 (secure 0 0 0) 0x7f059805bf50 con 0x55deb7037f00 2023-10-16T11:17:46.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 10 client.11199 fetch_fsmap learned FSMap version 103 2023-10-16T11:17:46.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) v3 -- 0x55deb6de0200 con 0x55deb7037f00 2023-10-16T11:17:46.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05817fa700 1 -- 172.21.15.158:0/3843038861 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0598073070 con 0x55deb7037f00 2023-10-16T11:17:46.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05817fa700 1 -- 172.21.15.158:0/3843038861 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 103) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0598058060 con 0x55deb7037f00 2023-10-16T11:17:46.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 10 client.11199 fetch_fsmap finished waiting for FSMap version 103 2023-10-16T11:17:46.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 10 client.11199 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:17:46.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 20 client.11199 populate_metadata read hostname 'smithi158' 2023-10-16T11:17:46.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb7030f40 0x55deb7031320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 4 client.11199 mds_command: new command op to 38450 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:17:46.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 --> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55deb6f87b20 con 0x55deb7030f40 2023-10-16T11:17:46.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb7030f40 0x55deb7031320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05817fa700 10 client.11199 ms_handle_connect on v2:172.21.15.172:6833/1279760034 2023-10-16T11:17:46.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.618+0000 7f05a1d9b700 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb7030f40 0x55deb7031320 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d949c9a79188704a server_cookie=1a564a1744dbcdf6 in_seq=0 out_seq=0 2023-10-16T11:17:46.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05817fa700 1 -- 172.21.15.158:0/3843038861 <== mds.0 v2:172.21.15.172:6833/1279760034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55deb6f87b20 con 0x55deb7030f40 2023-10-16T11:17:46.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05817fa700 10 client.11199 handle_command_reply: tid=0 2023-10-16T11:17:46.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 client.11199 shutdown 2023-10-16T11:17:46.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7037f00 msgr2=0x55deb70ec6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7037f00 0x55deb70ec6d0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f059800ce50 tx=0x7f0598057a70 comp rx=0 tx=0).stop 2023-10-16T11:17:46.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 shutdown_connections 2023-10-16T11:17:46.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:6833/1279760034,v1:172.21.15.172:6835/1279760034] conn(0x55deb7030f40 0x55deb7031320 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55deb7063f30 0x55deb7060930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05817fa700 0 client.11199 ms_handle_reset on v2:172.21.15.172:6833/1279760034 2023-10-16T11:17:46.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55deb7037f00 0x55deb70ec6d0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 --2- 172.21.15.158:0/3843038861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55deb703cb40 0x55deb70976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 >> 172.21.15.158:0/3843038861 conn(0x55deb6f69d80 msgr2=0x55deb70388e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 shutdown_connections 2023-10-16T11:17:46.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 1 -- 172.21.15.158:0/3843038861 wait complete. 2023-10-16T11:17:46.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05b49ebb80 20 client.11199 trim_cache size 0 max 16384 2023-10-16T11:17:46.644 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:17:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:17:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:17:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 38450, 2023-10-16T11:17:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:17:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:17:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T11:17:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9233653579999999, 2023-10-16T11:17:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 103, 2023-10-16T11:17:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 200, 2023-10-16T11:17:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 200, 2023-10-16T11:17:46.647 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.888434932999999 2023-10-16T11:17:46.647 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:17:46.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f058406b020 msgr2=0x7f058406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f058406b020 0x7f058406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f05a40d0270 tx=0x7f0594008030 comp rx=0 tx=0).stop 2023-10-16T11:17:46.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cd190 msgr2=0x7f05a41f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cd190 0x7f05a41f2c90 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f059800c7f0 tx=0x7f059800cb00 comp rx=0 tx=0).stop 2023-10-16T11:17:46.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 shutdown_connections 2023-10-16T11:17:46.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415a6d0 0x7f05a41ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a40cd190 0x7f05a41f2c90 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f058406b020 0x7f058406d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 --2- 172.21.15.158:0/1081243112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05a40cfe30 0x7f05a4157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.619+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 >> 172.21.15.158:0/1081243112 conn(0x7f05a406b9e0 msgr2=0x7f05a406e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.620+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 shutdown_connections 2023-10-16T11:17:46.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.620+0000 7f05a8801700 1 -- 172.21.15.158:0/1081243112 wait complete. 2023-10-16T11:17:46.651 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T11:17:46.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.772+0000 7f8574b9e700 1 Processor -- start 2023-10-16T11:17:46.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.772+0000 7f8574b9e700 1 -- start start 2023-10-16T11:17:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f8574b9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857015dbc0 0x7f85700cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f8574b9e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f857006ba90 0x7f85700cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f8574b9e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85700d21e0 con 0x7f85700d1d50 2023-10-16T11:17:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f8574b9e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8570162550 con 0x7f857006ba90 2023-10-16T11:17:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f8574b9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857015f320 con 0x7f857015dbc0 2023-10-16T11:17:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856ed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f857006ba90 0x7f85700cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857015dbc0 0x7f85700cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856dd9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f85700d1d50 0x7f857006beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:40726/0 (socket says 172.21.15.158:40726) 2023-10-16T11:17:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857015dbc0 0x7f85700cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:47476/0 (socket says 172.21.15.158:47476) 2023-10-16T11:17:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856ed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f857006ba90 0x7f85700cf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:37824/0 (socket says 172.21.15.158:37824) 2023-10-16T11:17:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856dd9b700 1 -- 172.21.15.158:0/2193834012 learned_addr learned my addr 172.21.15.158:0/2193834012 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856e59c700 1 -- 172.21.15.158:0/2193834012 >> v2:172.21.15.172:3300/0 conn(0x7f857006ba90 msgr2=0x7f85700cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856e59c700 1 --2- 172.21.15.158:0/2193834012 >> v2:172.21.15.172:3300/0 conn(0x7f857006ba90 0x7f85700cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856e59c700 1 -- 172.21.15.158:0/2193834012 >> v1:172.21.15.172:6789/0 conn(0x7f85700d1d50 legacy=0x7f857006beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856e59c700 1 -- 172.21.15.158:0/2193834012 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857015ef10 con 0x7f857015dbc0 2023-10-16T11:17:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.773+0000 7f856ed9d700 1 --2- 172.21.15.158:0/2193834012 >> v2:172.21.15.172:3300/0 conn(0x7f857006ba90 0x7f85700cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f856e59c700 1 --2- 172.21.15.158:0/2193834012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857015dbc0 0x7f85700cefa0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8560009840 tx=0x7f8560009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fae4ebd2aadcf77c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f856d59a700 1 -- 172.21.15.158:0/2193834012 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8560011830 con 0x7f857015dbc0 2023-10-16T11:17:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f856d59a700 1 -- 172.21.15.158:0/2193834012 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8560011990 con 0x7f857015dbc0 2023-10-16T11:17:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f856d59a700 1 -- 172.21.15.158:0/2193834012 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8560011d20 con 0x7f857015dbc0 2023-10-16T11:17:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 -- 172.21.15.158:0/2193834012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857015dbc0 msgr2=0x7f85700cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 --2- 172.21.15.158:0/2193834012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857015dbc0 0x7f85700cefa0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8560009840 tx=0x7f8560009b50 comp rx=0 tx=0).stop 2023-10-16T11:17:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 -- 172.21.15.158:0/2193834012 shutdown_connections 2023-10-16T11:17:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 --2- 172.21.15.158:0/2193834012 >> v2:172.21.15.172:3300/0 conn(0x7f857006ba90 0x7f85700cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 --2- 172.21.15.158:0/2193834012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857015dbc0 0x7f85700cefa0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 -- 172.21.15.158:0/2193834012 >> 172.21.15.158:0/2193834012 conn(0x7f857006a8d0 msgr2=0x7f857006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 -- 172.21.15.158:0/2193834012 shutdown_connections 2023-10-16T11:17:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.774+0000 7f8574b9e700 1 -- 172.21.15.158:0/2193834012 wait complete. 2023-10-16T11:17:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 Processor -- start 2023-10-16T11:17:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 -- start start 2023-10-16T11:17:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f857006ba90 0x7f8570160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857014e9e0 0x7f8570161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85700b5760 0x7f85700b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85700c9ef0 con 0x7f857006ba90 2023-10-16T11:17:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85700b6870 con 0x7f85700b5760 2023-10-16T11:17:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.776+0000 7f8574b9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85700c9650 con 0x7f857014e9e0 2023-10-16T11:17:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f857006ba90 0x7f8570160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f857006ba90 0x7f8570160f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36648/0 (socket says 172.21.15.158:36648) 2023-10-16T11:17:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 -- 172.21.15.158:0/4082812848 learned_addr learned my addr 172.21.15.158:0/4082812848 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856dd9b700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857014e9e0 0x7f8570161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856ed9d700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85700b5760 0x7f85700b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 -- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857014e9e0 msgr2=0x7f8570161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857014e9e0 0x7f8570161490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 -- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85700b5760 msgr2=0x7f85700b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85700b5760 0x7f85700b5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 -- 172.21.15.158:0/4082812848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8568009a70 con 0x7f857006ba90 2023-10-16T11:17:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f856e59c700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f857006ba90 0x7f8570160f50 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f8560009fd0 tx=0x7f8560004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f855effd700 1 -- 172.21.15.158:0/4082812848 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85600045d0 con 0x7f857006ba90 2023-10-16T11:17:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f855effd700 1 -- 172.21.15.158:0/4082812848 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856001f070 con 0x7f857006ba90 2023-10-16T11:17:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85600094a0 con 0x7f857006ba90 2023-10-16T11:17:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f855effd700 1 -- 172.21.15.158:0/4082812848 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85600038e0 con 0x7f857006ba90 2023-10-16T11:17:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.777+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85700c9c10 con 0x7f857006ba90 2023-10-16T11:17:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.778+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85700681b0 con 0x7f857006ba90 2023-10-16T11:17:46.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.779+0000 7f855effd700 1 -- 172.21.15.158:0/4082812848 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8560003a40 con 0x7f857006ba90 2023-10-16T11:17:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.779+0000 7f855effd700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f855006b020 0x7f855006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.779+0000 7f855effd700 1 -- 172.21.15.158:0/4082812848 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f8560021070 con 0x7f857006ba90 2023-10-16T11:17:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.779+0000 7f856dd9b700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f855006b020 0x7f855006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.779+0000 7f856dd9b700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f855006b020 0x7f855006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8564000f60 tx=0x7f8564007f10 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:46.783+0000 7f855effd700 1 -- 172.21.15.158:0/4082812848 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85600bdfe0 con 0x7f857006ba90 2023-10-16T11:17:47.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.023+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f857006b0b0 con 0x7f857006ba90 2023-10-16T11:17:47.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.023+0000 7f855effd700 1 -- 172.21.15.158:0/4082812848 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v103) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8560015070 con 0x7f857006ba90 2023-10-16T11:17:47.025 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T11:17:47.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f855006b020 msgr2=0x7f855006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:47.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f855006b020 0x7f855006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8564000f60 tx=0x7f8564007f10 comp rx=0 tx=0).stop 2023-10-16T11:17:47.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f857006ba90 msgr2=0x7f8570160f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:47.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f857006ba90 0x7f8570160f50 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f8560009fd0 tx=0x7f8560004750 comp rx=0 tx=0).stop 2023-10-16T11:17:47.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 shutdown_connections 2023-10-16T11:17:47.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f855006b020 0x7f855006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:47.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85700b5760 0x7f85700b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:47.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f857006ba90 0x7f8570160f50 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:47.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 --2- 172.21.15.158:0/4082812848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f857014e9e0 0x7f8570161490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:47.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 >> 172.21.15.158:0/4082812848 conn(0x7f857006a8d0 msgr2=0x7f857006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:47.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 shutdown_connections 2023-10-16T11:17:47.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.026+0000 7f8574b9e700 1 -- 172.21.15.158:0/4082812848 wait complete. 2023-10-16T11:17:47.039 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T11:17:47.040 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T11:17:47.040 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T11:17:47.040 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T11:17:47.041 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T11:17:47.041 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T11:17:47.041 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T11:17:47.042 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T11:17:47.045 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:17:47.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.046 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T11:17:47.063 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T11:17:47.063 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T11:17:47.063 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T11:17:47.063 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:17:47.064 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T11:17:47.064 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:17:47.064 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:17:47.064 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:17:47.064 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:17:47.065 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:17:47.065 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:17:47.065 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T11:17:47.066 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T11:17:47.066 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T11:17:47.066 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T11:17:47.066 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 36765sec preferred_lft 36765sec 2023-10-16T11:17:47.066 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T11:17:47.067 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:17:47.067 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T11:17:47.067 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T11:17:47.067 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T11:17:47.067 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:17:47.068 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T11:17:47.068 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:17:47.068 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:17:47.068 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:17:47.068 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T11:17:47.069 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:17:47.069 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:17:47.069 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:17:47.069 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T11:17:47.069 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:17:47.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.070 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T11:17:47.128 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T11:17:47.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.130 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T11:17:47.188 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T11:17:47.189 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T11:17:47.189 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T11:17:47.189 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T11:17:47.189 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T11:17:47.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.190 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T11:17:47.190 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T11:17:47.191 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T11:17:47.191 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T11:17:47.192 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T11:17:47.192 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T11:17:47.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.294 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:17:47.311 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:17:47.311 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:17:47.311 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:17:47.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.312 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T11:17:47.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:17:47.379 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T11:17:47.435 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:17:47.435 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:17:47.436 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:17:47.491 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T11:17:47.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.492 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:17:47.576 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:17:47.576 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:17:47.600 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:17:47.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:17:47.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:17:47.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:17:47.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:17:47.601 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:17:47.602 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:17:47.602 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:17:47.676 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:17:47.677 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:17:47.677 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:17:47.677 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:17:47.677 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:17:47.677 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:17:47.678 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:17:47.678 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:17:47.678 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T11:17:47.678 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T11:17:47.678 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T11:17:47.680 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T11:17:47.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.680 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:17:47.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:47.707 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T11:17:47.796 INFO:teuthology.orchestra.run.smithi158.stdout:client9946 2023-10-16T11:17:47.797 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:17:47.797 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:17:47.815 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T11:17:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.976+0000 7f2e77156700 1 Processor -- start 2023-10-16T11:17:47.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.976+0000 7f2e77156700 1 -- start start 2023-10-16T11:17:47.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.976+0000 7f2e77156700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e7006afe0 0x7f2e700cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:47.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.976+0000 7f2e77156700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e70071af0 con 0x7f2e7015f4d0 2023-10-16T11:17:47.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.976+0000 7f2e77156700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7015fa40 con 0x7f2e7015b6e0 2023-10-16T11:17:47.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.976+0000 7f2e77156700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e700afb60 con 0x7f2e7006afe0 2023-10-16T11:17:47.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e756f3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e7006afe0 0x7f2e700cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:47.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e756f3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e7006afe0 0x7f2e700cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36664/0 (socket says 172.21.15.158:36664) 2023-10-16T11:17:47.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e756f3700 1 -- 172.21.15.158:0/2192240598 learned_addr learned my addr 172.21.15.158:0/2192240598 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:47.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e6ffff700 1 --1- 172.21.15.158:0/2192240598 >> v1:172.21.15.158:6789/0 conn(0x7f2e7015f4d0 0x7f2e7006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36836/0 (socket says 172.21.15.158:36836) 2023-10-16T11:17:47.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e756f3700 1 -- 172.21.15.158:0/2192240598 >> v1:172.21.15.172:6789/0 conn(0x7f2e7015b6e0 legacy=0x7f2e7006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:47.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e756f3700 1 -- 172.21.15.158:0/2192240598 >> v1:172.21.15.158:6789/0 conn(0x7f2e7015f4d0 legacy=0x7f2e7006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:47.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e756f3700 1 -- 172.21.15.158:0/2192240598 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7015fd20 con 0x7f2e7006afe0 2023-10-16T11:17:47.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e6f7fe700 1 -- 172.21.15.158:0/2192240598 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119844548 0 0) 0x7f2e70071af0 con 0x7f2e7015f4d0 2023-10-16T11:17:47.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e756f3700 1 --2- 172.21.15.158:0/2192240598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7006afe0 0x7f2e700cfa80 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009e50 tx=0x7f2e6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6e2792221ae8ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:47.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e6f7fe700 1 -- 172.21.15.158:0/2192240598 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800e070 con 0x7f2e7006afe0 2023-10-16T11:17:47.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e6f7fe700 1 -- 172.21.15.158:0/2192240598 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6800b9c0 con 0x7f2e7006afe0 2023-10-16T11:17:47.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.977+0000 7f2e6f7fe700 1 -- 172.21.15.158:0/2192240598 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800bd70 con 0x7f2e7006afe0 2023-10-16T11:17:47.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.978+0000 7f2e77156700 1 -- 172.21.15.158:0/2192240598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7006afe0 msgr2=0x7f2e700cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:47.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.978+0000 7f2e77156700 1 --2- 172.21.15.158:0/2192240598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7006afe0 0x7f2e700cfa80 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009e50 tx=0x7f2e6800b330 comp rx=0 tx=0).stop 2023-10-16T11:17:47.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.978+0000 7f2e77156700 1 -- 172.21.15.158:0/2192240598 shutdown_connections 2023-10-16T11:17:47.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.978+0000 7f2e77156700 1 --2- 172.21.15.158:0/2192240598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7006afe0 0x7f2e700cfa80 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:47.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.978+0000 7f2e77156700 1 -- 172.21.15.158:0/2192240598 >> 172.21.15.158:0/2192240598 conn(0x7f2e7006b9e0 msgr2=0x7f2e7006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:47.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.978+0000 7f2e77156700 1 -- 172.21.15.158:0/2192240598 shutdown_connections 2023-10-16T11:17:47.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.978+0000 7f2e77156700 1 -- 172.21.15.158:0/2192240598 wait complete. 2023-10-16T11:17:47.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.979+0000 7f2e77156700 1 Processor -- start 2023-10-16T11:17:47.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.979+0000 7f2e77156700 1 -- start start 2023-10-16T11:17:47.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e77156700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7006afe0 0x7f2e700c41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:47.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e77156700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7015b6e0 0x7f2e700c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:47.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e77156700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7015f4d0 0x7f2e700b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:47.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e77156700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700c74a0 con 0x7f2e7015f4d0 2023-10-16T11:17:47.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e77156700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e70157e40 con 0x7f2e7015b6e0 2023-10-16T11:17:47.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e77156700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e70158120 con 0x7f2e7006afe0 2023-10-16T11:17:47.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7015f4d0 0x7f2e700b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:47.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7015f4d0 0x7f2e700b4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36670/0 (socket says 172.21.15.158:36670) 2023-10-16T11:17:47.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 -- 172.21.15.158:0/3870512451 learned_addr learned my addr 172.21.15.158:0/3870512451 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:47.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e6ffff700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7015b6e0 0x7f2e700c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:47.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 -- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7006afe0 msgr2=0x7f2e700c41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:47.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e74ef2700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7006afe0 0x7f2e700c41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:47.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7006afe0 0x7f2e700c41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:47.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 -- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7015b6e0 msgr2=0x7f2e700c2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:47.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7015b6e0 0x7f2e700c2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:47.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e756f3700 1 -- 172.21.15.158:0/3870512451 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009b30 con 0x7f2e7015f4d0 2023-10-16T11:17:47.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.980+0000 7f2e6ffff700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7015b6e0 0x7f2e700c2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:47.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.981+0000 7f2e756f3700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7015f4d0 0x7f2e700b4770 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f2e700cdf50 tx=0x7f2e68004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:47.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.981+0000 7f2e6d7fa700 1 -- 172.21.15.158:0/3870512451 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6801c640 con 0x7f2e7015f4d0 2023-10-16T11:17:47.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.981+0000 7f2e6d7fa700 1 -- 172.21.15.158:0/3870512451 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6801c7c0 con 0x7f2e7015f4d0 2023-10-16T11:17:47.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.981+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e70158400 con 0x7f2e7015f4d0 2023-10-16T11:17:47.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.981+0000 7f2e6d7fa700 1 -- 172.21.15.158:0/3870512451 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6801cb70 con 0x7f2e7015f4d0 2023-10-16T11:17:47.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.981+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e700c46e0 con 0x7f2e7015f4d0 2023-10-16T11:17:47.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.982+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e700681b0 con 0x7f2e7015f4d0 2023-10-16T11:17:47.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.982+0000 7f2e6d7fa700 1 -- 172.21.15.158:0/3870512451 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2e6801ccd0 con 0x7f2e7015f4d0 2023-10-16T11:17:47.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.983+0000 7f2e6d7fa700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2e5406b020 0x7f2e5406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:47.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.983+0000 7f2e6d7fa700 1 -- 172.21.15.158:0/3870512451 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f2e6808b660 con 0x7f2e7015f4d0 2023-10-16T11:17:47.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.983+0000 7f2e74ef2700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2e5406b020 0x7f2e5406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:47.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.983+0000 7f2e74ef2700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2e5406b020 0x7f2e5406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e64000f60 tx=0x7f2e64009040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:47.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:47.987+0000 7f2e6d7fa700 1 -- 172.21.15.158:0/3870512451 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e680561f0 con 0x7f2e7015f4d0 2023-10-16T11:17:48.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.207+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 --> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e701602e0 con 0x7f2e5406b020 2023-10-16T11:17:48.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.208+0000 7f2e6d7fa700 1 -- 172.21.15.158:0/3870512451 <== mgr.10824 v2:172.21.15.158:6832/3585499115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2e640050d0 con 0x7f2e5406b020 2023-10-16T11:17:48.210 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:17:48.210 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:17:48.211 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T11:17:48.211 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:17:48.211 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:17:48.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2e5406b020 msgr2=0x7f2e5406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2e5406b020 0x7f2e5406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e64000f60 tx=0x7f2e64009040 comp rx=0 tx=0).stop 2023-10-16T11:17:48.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7015f4d0 msgr2=0x7f2e700b4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7015f4d0 0x7f2e700b4770 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f2e700cdf50 tx=0x7f2e68004ad0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 shutdown_connections 2023-10-16T11:17:48.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7015f4d0 0x7f2e700b4770 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7006afe0 0x7f2e700c41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f2e5406b020 0x7f2e5406d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 --2- 172.21.15.158:0/3870512451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7015b6e0 0x7f2e700c2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 >> 172.21.15.158:0/3870512451 conn(0x7f2e7006b9e0 msgr2=0x7f2e70159ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:48.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 shutdown_connections 2023-10-16T11:17:48.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.210+0000 7f2e77156700 1 -- 172.21.15.158:0/3870512451 wait complete. 2023-10-16T11:17:48.224 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T11:17:48.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.345+0000 7f05011ea700 1 Processor -- start 2023-10-16T11:17:48.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.345+0000 7f05011ea700 1 -- start start 2023-10-16T11:17:48.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f05011ea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f04fc15da40 0x7f04fc0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f05011ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04fc06afe0 0x7f04fc0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f05011ea700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc0cbd30 con 0x7f04fc0cc590 2023-10-16T11:17:48.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f05011ea700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04fc0cc1c0 con 0x7f04fc15da40 2023-10-16T11:17:48.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f05011ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc15de20 con 0x7f04fc06afe0 2023-10-16T11:17:48.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f04fa59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f04fc0cc590 0x7f04fc06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36852/0 (socket says 172.21.15.158:36852) 2023-10-16T11:17:48.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f04fa59c700 1 -- 172.21.15.158:0/3677069009 learned_addr learned my addr 172.21.15.158:0/3677069009 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:48.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f04fb59e700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04fc06afe0 0x7f04fc0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.346+0000 7f04fad9d700 1 --2- 172.21.15.158:0/3677069009 >> v2:172.21.15.172:3300/0 conn(0x7f04fc15da40 0x7f04fc0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020135302 0 0) 0x7f04fc0cbd30 con 0x7f04fc0cc590 2023-10-16T11:17:48.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e40035b0 con 0x7f04fc0cc590 2023-10-16T11:17:48.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170655166 0 0) 0x7f04e40035b0 con 0x7f04fc0cc590 2023-10-16T11:17:48.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04fc06afe0 msgr2=0x7f04fc0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04fc06afe0 0x7f04fc0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 >> v2:172.21.15.172:3300/0 conn(0x7f04fc15da40 msgr2=0x7f04fc0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 --2- 172.21.15.158:0/3677069009 >> v2:172.21.15.172:3300/0 conn(0x7f04fc15da40 0x7f04fc0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0ce260 con 0x7f04fc0cc590 2023-10-16T11:17:48.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f04ec001670 con 0x7f04fc0cc590 2023-10-16T11:17:48.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 >> v1:172.21.15.158:6789/0 conn(0x7f04fc0cc590 legacy=0x7f04fc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e4003a00 0x7f04e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04e4004850 0x7f04e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4005210 0x7f04e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0cc1c0 con 0x7f04e4004850 2023-10-16T11:17:48.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc15de20 con 0x7f04e4003a00 2023-10-16T11:17:48.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e400bf50 con 0x7f04e4005210 2023-10-16T11:17:48.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fad9d700 1 --2- 172.21.15.158:0/3677069009 >> v2:172.21.15.172:3300/0 conn(0x7f04fc15da40 0x7f04fc0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:48.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fb59e700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04fc06afe0 0x7f04fc0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:48.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fa59c700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04e4004850 0x7f04e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fad9d700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e4003a00 0x7f04e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fb59e700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4005210 0x7f04e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fa59c700 1 -- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4005210 msgr2=0x7f04e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fa59c700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4005210 0x7f04e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fa59c700 1 -- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e4003a00 msgr2=0x7f04e4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fa59c700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e4003a00 0x7f04e4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fa59c700 1 -- 172.21.15.158:0/3677069009 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e40035b0 con 0x7f04e4004850 2023-10-16T11:17:48.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fb59e700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4005210 0x7f04e400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:48.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.347+0000 7f04fa59c700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04e4004850 0x7f04e4004cd0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00c320 tx=0x7f04e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55207ae3ba478fd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:48.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec00ce20 con 0x7f04e4004850 2023-10-16T11:17:48.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec005960 con 0x7f04e4004850 2023-10-16T11:17:48.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f04f9d9b700 1 -- 172.21.15.158:0/3677069009 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec005d10 con 0x7f04e4004850 2023-10-16T11:17:48.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f05011ea700 1 -- 172.21.15.158:0/3677069009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04e4004850 msgr2=0x7f04e4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f05011ea700 1 --2- 172.21.15.158:0/3677069009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04e4004850 0x7f04e4004cd0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00c320 tx=0x7f04e4005880 comp rx=0 tx=0).stop 2023-10-16T11:17:48.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f04fb59e700 1 -- 172.21.15.158:0/3677069009 reap_dead start 2023-10-16T11:17:48.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f05011ea700 1 -- 172.21.15.158:0/3677069009 shutdown_connections 2023-10-16T11:17:48.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f05011ea700 1 -- 172.21.15.158:0/3677069009 >> 172.21.15.158:0/3677069009 conn(0x7f04fc099a10 msgr2=0x7f04fc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:48.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f05011ea700 1 -- 172.21.15.158:0/3677069009 shutdown_connections 2023-10-16T11:17:48.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.348+0000 7f05011ea700 1 -- 172.21.15.158:0/3677069009 wait complete. 2023-10-16T11:17:48.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 Processor -- start 2023-10-16T11:17:48.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 -- start start 2023-10-16T11:17:48.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4004850 0x7f04fc155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04fc155840 0x7f04fc159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04fc0c8f10 0x7f04fc0c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0c98d0 con 0x7f04fc155840 2023-10-16T11:17:48.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc1e5f50 con 0x7f04fc0c8f10 2023-10-16T11:17:48.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.350+0000 7f05011ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc1e4f40 con 0x7f04e4004850 2023-10-16T11:17:48.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04fc155840 0x7f04fc159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04fc155840 0x7f04fc159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36688/0 (socket says 172.21.15.158:36688) 2023-10-16T11:17:48.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 -- 172.21.15.158:0/2698088249 learned_addr learned my addr 172.21.15.158:0/2698088249 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:48.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fad9d700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4004850 0x7f04fc155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fb59e700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04fc0c8f10 0x7f04fc0c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 -- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4004850 msgr2=0x7f04fc155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4004850 0x7f04fc155200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 -- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04fc0c8f10 msgr2=0x7f04fc0c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04fc0c8f10 0x7f04fc0c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 -- 172.21.15.158:0/2698088249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc1e5190 con 0x7f04fc155840 2023-10-16T11:17:48.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fb59e700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04fc0c8f10 0x7f04fc0c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:17:48.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04fa59c700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04fc155840 0x7f04fc159250 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f04ec008fd0 tx=0x7f04ec001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:48.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04e37fe700 1 -- 172.21.15.158:0/2698088249 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec00ce90 con 0x7f04fc155840 2023-10-16T11:17:48.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04e37fe700 1 -- 172.21.15.158:0/2698088249 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec004a80 con 0x7f04fc155840 2023-10-16T11:17:48.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f04e37fe700 1 -- 172.21.15.158:0/2698088249 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec004e70 con 0x7f04fc155840 2023-10-16T11:17:48.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc1e5470 con 0x7f04fc155840 2023-10-16T11:17:48.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.351+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc1e5a30 con 0x7f04fc155840 2023-10-16T11:17:48.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.352+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc1e5da0 con 0x7f04fc155840 2023-10-16T11:17:48.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.353+0000 7f04e37fe700 1 -- 172.21.15.158:0/2698088249 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f04ec0070d0 con 0x7f04fc155840 2023-10-16T11:17:48.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.353+0000 7f04e37fe700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f04e806b020 0x7f04e806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.353+0000 7f04e37fe700 1 -- 172.21.15.158:0/2698088249 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f04ec08b0c0 con 0x7f04fc155840 2023-10-16T11:17:48.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.353+0000 7f04fad9d700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f04e806b020 0x7f04e806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.353+0000 7f04fad9d700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f04e806b020 0x7f04e806d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04f0004510 tx=0x7f04f0008040 comp rx=0 tx=0).ready entity=mgr.10824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:48.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.357+0000 7f04e37fe700 1 -- 172.21.15.158:0/2698088249 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f04ec055ea0 con 0x7f04fc155840 2023-10-16T11:17:48.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.605+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f04fc1e5bc0 con 0x7f04fc155840 2023-10-16T11:17:48.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.734+0000 7f04e37fe700 1 -- 172.21.15.158:0/2698088249 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v127) v1 ==== 92+0+0 (secure 0 0 0) 0x7f04ec007da0 con 0x7f04fc155840 2023-10-16T11:17:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.736+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f04e806b020 msgr2=0x7f04e806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.736+0000 7f05011ea700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f04e806b020 0x7f04e806d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04f0004510 tx=0x7f04f0008040 comp rx=0 tx=0).stop 2023-10-16T11:17:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.736+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04fc155840 msgr2=0x7f04fc159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.736+0000 7f05011ea700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04fc155840 0x7f04fc159250 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f04ec008fd0 tx=0x7f04ec001a00 comp rx=0 tx=0).stop 2023-10-16T11:17:48.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.737+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 shutdown_connections 2023-10-16T11:17:48.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.737+0000 7f05011ea700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f04e806b020 0x7f04e806d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.737+0000 7f05011ea700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04fc0c8f10 0x7f04fc0c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.737+0000 7f05011ea700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e4004850 0x7f04fc155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.737+0000 7f05011ea700 1 --2- 172.21.15.158:0/2698088249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04fc155840 0x7f04fc159250 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.737+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 >> 172.21.15.158:0/2698088249 conn(0x7f04fc099a10 msgr2=0x7f04fc0cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.738+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 shutdown_connections 2023-10-16T11:17:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.738+0000 7f05011ea700 1 -- 172.21.15.158:0/2698088249 wait complete. 2023-10-16T11:17:48.753 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T11:17:48.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.878+0000 7f7b0c9b0700 1 Processor -- start 2023-10-16T11:17:48.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.879+0000 7f7b0c9b0700 1 -- start start 2023-10-16T11:17:48.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.879+0000 7f7b0c9b0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0815b6b0 0x7f7b080cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.879+0000 7f7b0c9b0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b080d0c60 0x7f7b080b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.879+0000 7f7b0c9b0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080b4420 con 0x7f7b080b3460 2023-10-16T11:17:48.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.879+0000 7f7b0c9b0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b080b3910 con 0x7f7b080d0c60 2023-10-16T11:17:48.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.879+0000 7f7b0c9b0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0815ba90 con 0x7f7b0815b6b0 2023-10-16T11:17:48.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b06d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7b080b3460 0x7f7b08070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:36902/0 (socket says 172.21.15.158:36902) 2023-10-16T11:17:48.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b06d9d700 1 -- 172.21.15.158:0/2821647416 learned_addr learned my addr 172.21.15.158:0/2821647416 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:48.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0659c700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0815b6b0 0x7f7b080cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b05d9b700 1 --2- 172.21.15.158:0/2821647416 >> v2:172.21.15.172:3300/0 conn(0x7f7b080d0c60 0x7f7b080b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881876094 0 0) 0x7f7b080b4420 con 0x7f7b080b3460 2023-10-16T11:17:48.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af00035b0 con 0x7f7b080b3460 2023-10-16T11:17:48.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377660182 0 0) 0x7f7af00035b0 con 0x7f7b080b3460 2023-10-16T11:17:48.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0815b6b0 msgr2=0x7f7b080cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0815b6b0 0x7f7b080cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 >> v2:172.21.15.172:3300/0 conn(0x7f7b080d0c60 msgr2=0x7f7b080b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 --2- 172.21.15.158:0/2821647416 >> v2:172.21.15.172:3300/0 conn(0x7f7b080d0c60 0x7f7b080b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080cbc20 con 0x7f7b080b3460 2023-10-16T11:17:48.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7b000019f0 con 0x7f7b080b3460 2023-10-16T11:17:48.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 >> v1:172.21.15.158:6789/0 conn(0x7f7b080b3460 legacy=0x7f7b08070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7af0003a00 0x7f7af0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af00043f0 0x7f7af00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af0004fe0 0x7f7af000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080b3910 con 0x7f7af00043f0 2023-10-16T11:17:48.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af000bf50 con 0x7f7af0004fe0 2023-10-16T11:17:48.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af000c0c0 con 0x7f7af0003a00 2023-10-16T11:17:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0659c700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0815b6b0 0x7f7b080cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b05d9b700 1 --2- 172.21.15.158:0/2821647416 >> v2:172.21.15.172:3300/0 conn(0x7f7b080d0c60 0x7f7b080b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b06d9d700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af0004fe0 0x7f7af000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b05d9b700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af00043f0 0x7f7af00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.880+0000 7f7b0659c700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7af0003a00 0x7f7af0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b06d9d700 1 -- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7af0003a00 msgr2=0x7f7af0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b06d9d700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7af0003a00 0x7f7af0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b06d9d700 1 -- 172.21.15.158:0/2821647416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af00043f0 msgr2=0x7f7af00048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b06d9d700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af00043f0 0x7f7af00048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b06d9d700 1 -- 172.21.15.158:0/2821647416 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00035b0 con 0x7f7af0004fe0 2023-10-16T11:17:48.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b05d9b700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af00043f0 0x7f7af00048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:48.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b06d9d700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af0004fe0 0x7f7af000ba10 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000a860 tx=0x7f7af0005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e465dc12272c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:48.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00012600 con 0x7f7af0004fe0 2023-10-16T11:17:48.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b00012780 con 0x7f7af0004fe0 2023-10-16T11:17:48.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f7b0559a700 1 -- 172.21.15.158:0/2821647416 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00012b30 con 0x7f7af0004fe0 2023-10-16T11:17:48.892 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:48.881+0000 7f275e54f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:17:48.894 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:48.881+0000 7f67261ca200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:17:48.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.882+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af0004fe0 msgr2=0x7f7af000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.882+0000 7f7b0c9b0700 1 --2- 172.21.15.158:0/2821647416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af0004fe0 0x7f7af000ba10 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000a860 tx=0x7f7af0005880 comp rx=0 tx=0).stop 2023-10-16T11:17:48.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.882+0000 7f7b06d9d700 1 -- 172.21.15.158:0/2821647416 reap_dead start 2023-10-16T11:17:48.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.882+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/2821647416 shutdown_connections 2023-10-16T11:17:48.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.882+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/2821647416 >> 172.21.15.158:0/2821647416 conn(0x7f7b080cdf20 msgr2=0x7f7b080ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:48.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.882+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/2821647416 shutdown_connections 2023-10-16T11:17:48.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.882+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/2821647416 wait complete. 2023-10-16T11:17:48.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.883+0000 7f7b0c9b0700 1 Processor -- start 2023-10-16T11:17:48.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.883+0000 7f7b0c9b0700 1 -- start start 2023-10-16T11:17:48.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0c9b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af0004fe0 0x7f7b08155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0c9b0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0806b9e0 0x7f7b080adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0c9b0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b080af7e0 0x7f7b081592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0c9b0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0806bea0 con 0x7f7af0004fe0 2023-10-16T11:17:48.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0c9b0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0806c180 con 0x7f7b080af7e0 2023-10-16T11:17:48.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0c9b0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0806c460 con 0x7f7b0806b9e0 2023-10-16T11:17:48.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af0004fe0 0x7f7b08155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af0004fe0 0x7f7b08155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36734/0 (socket says 172.21.15.158:36734) 2023-10-16T11:17:48.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 -- 172.21.15.158:0/3384014203 learned_addr learned my addr 172.21.15.158:0/3384014203 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:48.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b06d9d700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b080af7e0 0x7f7b081592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b05d9b700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0806b9e0 0x7f7b080adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:48.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 -- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0806b9e0 msgr2=0x7f7b080adc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0806b9e0 0x7f7b080adc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 -- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b080af7e0 msgr2=0x7f7b081592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:48.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b080af7e0 0x7f7b081592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:48.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.884+0000 7f7b0659c700 1 -- 172.21.15.158:0/3384014203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0806c740 con 0x7f7af0004fe0 2023-10-16T11:17:48.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.885+0000 7f7b0659c700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af0004fe0 0x7f7b08155c30 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f7af8011be0 tx=0x7f7af8011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:48.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.885+0000 7f7af6ffd700 1 -- 172.21.15.158:0/3384014203 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af800f940 con 0x7f7af0004fe0 2023-10-16T11:17:48.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.885+0000 7f7af6ffd700 1 -- 172.21.15.158:0/3384014203 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af800faa0 con 0x7f7af0004fe0 2023-10-16T11:17:48.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.885+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0806ca20 con 0x7f7af0004fe0 2023-10-16T11:17:48.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.885+0000 7f7af6ffd700 1 -- 172.21.15.158:0/3384014203 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af800fe10 con 0x7f7af0004fe0 2023-10-16T11:17:48.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.885+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b080ae1d0 con 0x7f7af0004fe0 2023-10-16T11:17:48.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.886+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b080681b0 con 0x7f7af0004fe0 2023-10-16T11:17:48.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.886+0000 7f7af6ffd700 1 -- 172.21.15.158:0/3384014203 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 127) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f7af8014440 con 0x7f7af0004fe0 2023-10-16T11:17:48.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.887+0000 7f7af6ffd700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 0x7f7ae806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:48.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.887+0000 7f7b05d9b700 1 -- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 msgr2=0x7f7ae806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:48.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.887+0000 7f7af6ffd700 1 -- 172.21.15.158:0/3384014203 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f7af808a200 con 0x7f7af0004fe0 2023-10-16T11:17:48.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.887+0000 7f7b05d9b700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 0x7f7ae806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:48.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:48.891+0000 7f7af6ffd700 1 -- 172.21.15.158:0/3384014203 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7af8054f20 con 0x7f7af0004fe0 2023-10-16T11:17:48.949 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:48.947+0000 7f67261ca200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:48.952 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:48.951+0000 7f275e54f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:49.021 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:49.020+0000 7f67261ca200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:17:49.026 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:49.024+0000 7f275e54f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:17:49.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.088+0000 7f7b05d9b700 1 -- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 msgr2=0x7f7ae806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:49.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.088+0000 7f7b05d9b700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 0x7f7ae806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:49.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.094+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f7b0806dbf0 con 0x7f7af0004fe0 2023-10-16T11:17:49.099 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:49.097+0000 7f67261ca200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:17:49.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.098+0000 7f7af6ffd700 1 -- 172.21.15.158:0/3384014203 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7af8058540 con 0x7f7af0004fe0 2023-10-16T11:17:49.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 msgr2=0x7f7ae806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:49.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 0x7f7ae806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af0004fe0 msgr2=0x7f7b08155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af0004fe0 0x7f7b08155c30 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f7af8011be0 tx=0x7f7af8011ef0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 shutdown_connections 2023-10-16T11:17:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f7ae806b070 0x7f7ae806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b080af7e0 0x7f7b081592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7af0004fe0 0x7f7b08155c30 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 --2- 172.21.15.158:0/3384014203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b0806b9e0 0x7f7b080adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 >> 172.21.15.158:0/3384014203 conn(0x7f7b080cdf20 msgr2=0x7f7b0815e470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 shutdown_connections 2023-10-16T11:17:49.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.100+0000 7f7b0c9b0700 1 -- 172.21.15.158:0/3384014203 wait complete. 2023-10-16T11:17:49.107 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:49.106+0000 7f275e54f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:17:49.115 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T11:17:49.240 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:49.239+0000 7f67261ca200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:17:49.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.239+0000 7f9d4044b700 1 Processor -- start 2023-10-16T11:17:49.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.239+0000 7f9d4044b700 1 -- start start 2023-10-16T11:17:49.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d4044b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d380cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d4044b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9d380d0c60 0x7f9d380b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d4044b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d380b3460 0x7f9d38070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d4044b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d380b3910 con 0x7f9d380b3460 2023-10-16T11:17:49.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d4044b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3815ba90 con 0x7f9d380d0c60 2023-10-16T11:17:49.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d4044b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380b4420 con 0x7f9d3815b6b0 2023-10-16T11:17:49.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d3e9e8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d380b3460 0x7f9d38070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d3e9e8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d380b3460 0x7f9d38070300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36792/0 (socket says 172.21.15.158:36792) 2023-10-16T11:17:49.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d3e9e8700 1 -- 172.21.15.158:0/1584120739 learned_addr learned my addr 172.21.15.158:0/1584120739 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:49.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d3e1e7700 1 --2- 172.21.15.158:0/1584120739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d380cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.240+0000 7f9d3d9e6700 1 --2- 172.21.15.158:0/1584120739 >> v2:172.21.15.172:3300/0 conn(0x7f9d380d0c60 0x7f9d380b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3e9e8700 1 -- 172.21.15.158:0/1584120739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 msgr2=0x7f9d380cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:49.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3e9e8700 1 --2- 172.21.15.158:0/1584120739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d380cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3e9e8700 1 -- 172.21.15.158:0/1584120739 >> v2:172.21.15.172:3300/0 conn(0x7f9d380d0c60 msgr2=0x7f9d380b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:49.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3e9e8700 1 --2- 172.21.15.158:0/1584120739 >> v2:172.21.15.172:3300/0 conn(0x7f9d380d0c60 0x7f9d380b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3e9e8700 1 -- 172.21.15.158:0/1584120739 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d380cbc20 con 0x7f9d380b3460 2023-10-16T11:17:49.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3d9e6700 1 --2- 172.21.15.158:0/1584120739 >> v2:172.21.15.172:3300/0 conn(0x7f9d380d0c60 0x7f9d380b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:49.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3e9e8700 1 --2- 172.21.15.158:0/1584120739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380b3460 0x7f9d38070300 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f9d38067260 tx=0x7f9d3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a754037c5c1a564 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:49.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3d1e5700 1 -- 172.21.15.158:0/1584120739 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000e070 con 0x7f9d380b3460 2023-10-16T11:17:49.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3d1e5700 1 -- 172.21.15.158:0/1584120739 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3000b910 con 0x7f9d380b3460 2023-10-16T11:17:49.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.241+0000 7f9d3d1e5700 1 -- 172.21.15.158:0/1584120739 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000bcc0 con 0x7f9d380b3460 2023-10-16T11:17:49.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 -- 172.21.15.158:0/1584120739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380b3460 msgr2=0x7f9d38070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 --2- 172.21.15.158:0/1584120739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380b3460 0x7f9d38070300 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f9d38067260 tx=0x7f9d3000b330 comp rx=0 tx=0).stop 2023-10-16T11:17:49.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 -- 172.21.15.158:0/1584120739 shutdown_connections 2023-10-16T11:17:49.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 --2- 172.21.15.158:0/1584120739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380b3460 0x7f9d38070300 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 --2- 172.21.15.158:0/1584120739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d380cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 --2- 172.21.15.158:0/1584120739 >> v2:172.21.15.172:3300/0 conn(0x7f9d380d0c60 0x7f9d380b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 -- 172.21.15.158:0/1584120739 >> 172.21.15.158:0/1584120739 conn(0x7f9d380cdf20 msgr2=0x7f9d380ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 -- 172.21.15.158:0/1584120739 shutdown_connections 2023-10-16T11:17:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.242+0000 7f9d4044b700 1 -- 172.21.15.158:0/1584120739 wait complete. 2023-10-16T11:17:49.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.244+0000 7f9d4044b700 1 Processor -- start 2023-10-16T11:17:49.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.244+0000 7f9d4044b700 1 -- start start 2023-10-16T11:17:49.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.244+0000 7f9d4044b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d380b3460 0x7f9d381f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.244+0000 7f9d4044b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380d0c60 0x7f9d380a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.244+0000 7f9d4044b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d381edd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.244+0000 7f9d4044b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380ab5b0 con 0x7f9d380d0c60 2023-10-16T11:17:49.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.244+0000 7f9d4044b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380ab2a0 con 0x7f9d380b3460 2023-10-16T11:17:49.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d4044b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380a8df0 con 0x7f9d3815b6b0 2023-10-16T11:17:49.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380d0c60 0x7f9d380a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380d0c60 0x7f9d380a8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36796/0 (socket says 172.21.15.158:36796) 2023-10-16T11:17:49.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 -- 172.21.15.158:0/2263124151 learned_addr learned my addr 172.21.15.158:0/2263124151 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:49.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3e1e7700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d380b3460 0x7f9d381f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3e9e8700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d381edd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 -- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 msgr2=0x7f9d381edd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d381edd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 -- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d380b3460 msgr2=0x7f9d381f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d380b3460 0x7f9d381f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 -- 172.21.15.158:0/2263124151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30009b30 con 0x7f9d380d0c60 2023-10-16T11:17:49.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d3d9e6700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380d0c60 0x7f9d380a8480 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800d7d0 tx=0x7f9d2800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:49.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d2effd700 1 -- 172.21.15.158:0/2263124151 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28004030 con 0x7f9d380d0c60 2023-10-16T11:17:49.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d2effd700 1 -- 172.21.15.158:0/2263124151 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28004190 con 0x7f9d380d0c60 2023-10-16T11:17:49.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d2effd700 1 -- 172.21.15.158:0/2263124151 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28004500 con 0x7f9d380d0c60 2023-10-16T11:17:49.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d380a90d0 con 0x7f9d380d0c60 2023-10-16T11:17:49.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.245+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d381f22c0 con 0x7f9d380d0c60 2023-10-16T11:17:49.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.246+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d380ab820 con 0x7f9d380d0c60 2023-10-16T11:17:49.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.247+0000 7f9d2effd700 1 -- 172.21.15.158:0/2263124151 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 127) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9d280036a0 con 0x7f9d380d0c60 2023-10-16T11:17:49.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.247+0000 7f9d2effd700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 0x7f9d1c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.247+0000 7f9d3e1e7700 1 -- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 msgr2=0x7f9d1c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:49.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.247+0000 7f9d2effd700 1 -- 172.21.15.158:0/2263124151 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f9d28089770 con 0x7f9d380d0c60 2023-10-16T11:17:49.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.247+0000 7f9d3e1e7700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 0x7f9d1c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:49.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.252+0000 7f9d2effd700 1 -- 172.21.15.158:0/2263124151 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d28054490 con 0x7f9d380d0c60 2023-10-16T11:17:49.258 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:49.256+0000 7f275e54f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:17:49.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.448+0000 7f9d3e1e7700 1 -- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 msgr2=0x7f9d1c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:49.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.448+0000 7f9d3e1e7700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 0x7f9d1c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:49.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.453+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9d380681b0 con 0x7f9d380d0c60 2023-10-16T11:17:49.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.457+0000 7f9d2effd700 1 -- 172.21.15.158:0/2263124151 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9d28057ab0 con 0x7f9d380d0c60 2023-10-16T11:17:49.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 msgr2=0x7f9d1c06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:49.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 0x7f9d1c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380d0c60 msgr2=0x7f9d380a8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380d0c60 0x7f9d380a8480 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800d7d0 tx=0x7f9d2800dae0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 shutdown_connections 2023-10-16T11:17:49.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d3815b6b0 0x7f9d381edd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7f9d1c06b070 0x7f9d1c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d380b3460 0x7f9d381f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 --2- 172.21.15.158:0/2263124151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d380d0c60 0x7f9d380a8480 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 >> 172.21.15.158:0/2263124151 conn(0x7f9d380cdf20 msgr2=0x7f9d3815c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:49.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 shutdown_connections 2023-10-16T11:17:49.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.459+0000 7f9d4044b700 1 -- 172.21.15.158:0/2263124151 wait complete. 2023-10-16T11:17:49.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:49.473 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T11:17:49.496 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T11:17:49.568 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:49.566+0000 7f67261ca200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:17:49.597 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:49.596+0000 7f275e54f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:17:49.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.662+0000 7fe7367e1700 1 Processor -- start 2023-10-16T11:17:49.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.662+0000 7fe7367e1700 1 -- start start 2023-10-16T11:17:49.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.663+0000 7fe7367e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73015a840 0x7fe7300ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.663+0000 7fe7367e1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7300c9a70 con 0x7fe730157680 2023-10-16T11:17:49.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.663+0000 7fe7367e1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73015aed0 con 0x7fe7300b1310 2023-10-16T11:17:49.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.663+0000 7fe7367e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7300cace0 con 0x7fe73015a840 2023-10-16T11:17:49.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72ffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe730157680 0x7fe7300b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37024/0 (socket says 172.21.15.158:37024) 2023-10-16T11:17:49.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72ffff700 1 -- 172.21.15.158:0/1094104108 learned_addr learned my addr 172.21.15.158:0/1094104108 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:49.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe734d7e700 1 --2- 172.21.15.158:0/1094104108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73015a840 0x7fe7300ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72effd700 1 -- 172.21.15.158:0/1094104108 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848227920 0 0) 0x7fe7300c9a70 con 0x7fe730157680 2023-10-16T11:17:49.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72effd700 1 -- 172.21.15.158:0/1094104108 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7180035b0 con 0x7fe730157680 2023-10-16T11:17:49.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72effd700 1 -- 172.21.15.158:0/1094104108 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862922878 0 0) 0x7fe73015aed0 con 0x7fe7300b1310 2023-10-16T11:17:49.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72effd700 1 -- 172.21.15.158:0/1094104108 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7300c9a70 con 0x7fe7300b1310 2023-10-16T11:17:49.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe734d7e700 1 -- 172.21.15.158:0/1094104108 >> v1:172.21.15.172:6789/0 conn(0x7fe7300b1310 legacy=0x7fe73015a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe734d7e700 1 -- 172.21.15.158:0/1094104108 >> v1:172.21.15.158:6789/0 conn(0x7fe730157680 legacy=0x7fe7300b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe734d7e700 1 -- 172.21.15.158:0/1094104108 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7300c9cc0 con 0x7fe73015a840 2023-10-16T11:17:49.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe734d7e700 1 --2- 172.21.15.158:0/1094104108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73015a840 0x7fe7300ca7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe728009dc0 tx=0x7fe72800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102964ed5a9d0762 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:49.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72effd700 1 -- 172.21.15.158:0/1094104108 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72800e070 con 0x7fe73015a840 2023-10-16T11:17:49.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.664+0000 7fe72effd700 1 -- 172.21.15.158:0/1094104108 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72800bb20 con 0x7fe73015a840 2023-10-16T11:17:49.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe72effd700 1 -- 172.21.15.158:0/1094104108 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728012430 con 0x7fe73015a840 2023-10-16T11:17:49.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe7367e1700 1 -- 172.21.15.158:0/1094104108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73015a840 msgr2=0x7fe7300ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe7367e1700 1 --2- 172.21.15.158:0/1094104108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73015a840 0x7fe7300ca7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe728009dc0 tx=0x7fe72800b330 comp rx=0 tx=0).stop 2023-10-16T11:17:49.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe7367e1700 1 -- 172.21.15.158:0/1094104108 shutdown_connections 2023-10-16T11:17:49.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe7367e1700 1 --2- 172.21.15.158:0/1094104108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe73015a840 0x7fe7300ca7a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe7367e1700 1 -- 172.21.15.158:0/1094104108 >> 172.21.15.158:0/1094104108 conn(0x7fe73006af00 msgr2=0x7fe7300cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:49.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe7367e1700 1 -- 172.21.15.158:0/1094104108 shutdown_connections 2023-10-16T11:17:49.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.665+0000 7fe7367e1700 1 -- 172.21.15.158:0/1094104108 wait complete. 2023-10-16T11:17:49.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.667+0000 7fe7367e1700 1 Processor -- start 2023-10-16T11:17:49.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.667+0000 7fe7367e1700 1 -- start start 2023-10-16T11:17:49.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe7367e1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7300b1310 0x7fe7300adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe7367e1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe730157680 0x7fe7300c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe7367e1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73015a840 0x7fe730153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe7367e1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730153e90 con 0x7fe7300b1310 2023-10-16T11:17:49.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe7367e1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730157b90 con 0x7fe73015a840 2023-10-16T11:17:49.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe7367e1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe730157e70 con 0x7fe730157680 2023-10-16T11:17:49.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7300b1310 0x7fe7300adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7300b1310 0x7fe7300adfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:36854/0 (socket says 172.21.15.158:36854) 2023-10-16T11:17:49.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe730157680 0x7fe7300c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72ffff700 1 -- 172.21.15.158:0/3808419371 learned_addr learned my addr 172.21.15.158:0/3808419371 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:49.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe734d7e700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73015a840 0x7fe730153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:49.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72f7fe700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73015a840 msgr2=0x7fe730153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72f7fe700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe73015a840 0x7fe730153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72f7fe700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7300b1310 msgr2=0x7fe7300adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:49.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72f7fe700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7300b1310 0x7fe7300adfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:49.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72f7fe700 1 -- 172.21.15.158:0/3808419371 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728009a70 con 0x7fe730157680 2023-10-16T11:17:49.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.668+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7300b1310 0x7fe7300adfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:17:49.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.669+0000 7fe72f7fe700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe730157680 0x7fe7300c7be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe72400c450 tx=0x7fe72400c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:49.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.669+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240052d0 con 0x7fe730157680 2023-10-16T11:17:49.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.669+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe724005430 con 0x7fe730157680 2023-10-16T11:17:49.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.669+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe730158150 con 0x7fe730157680 2023-10-16T11:17:49.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.669+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240038c0 con 0x7fe730157680 2023-10-16T11:17:49.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.669+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe730158710 con 0x7fe730157680 2023-10-16T11:17:49.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.670+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7300b16f0 con 0x7fe730157680 2023-10-16T11:17:49.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.670+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 127) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe724003a20 con 0x7fe730157680 2023-10-16T11:17:49.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.670+0000 7fe72cff9700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 0x7fe71c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:49.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.671+0000 7fe72ffff700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 msgr2=0x7fe71c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:49.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.671+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fe724088d50 con 0x7fe730157680 2023-10-16T11:17:49.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.671+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 0x7fe71c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:17:49.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.675+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe72408c030 con 0x7fe730157680 2023-10-16T11:17:49.686 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:49.685+0000 7f67261ca200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:17:49.720 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:49.719+0000 7f275e54f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:17:49.761 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:49.760+0000 7f67261ca200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:17:49.796 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:49.795+0000 7f275e54f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:17:49.841 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:49.839+0000 7f67261ca200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:17:49.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.872+0000 7fe72ffff700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 msgr2=0x7fe71c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:49.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.872+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 0x7fe71c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:17:49.879 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:49.878+0000 7f275e54f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:17:49.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:49.903+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 --> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7300d0080 con 0x7fe71c06b070 2023-10-16T11:17:50.210 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:50.208+0000 7f67261ca200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:17:50.251 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:50.249+0000 7f275e54f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:17:50.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:50.273+0000 7fe72ffff700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 msgr2=0x7fe71c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:50.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:50.273+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 0x7fe71c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T11:17:50.778 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:50.777+0000 7f67261ca200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:17:50.821 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:50.819+0000 7f275e54f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:17:50.847 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:50.846+0000 7f67261ca200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:17:50.891 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:50.889+0000 7f275e54f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:17:51.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:51.074+0000 7fe72ffff700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 msgr2=0x7fe71c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:51.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:51.074+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 0x7fe71c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T11:17:51.490 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:51.488+0000 7f67261ca200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:51.521 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:51.520+0000 7f275e54f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:51.567 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:51.565+0000 7f67261ca200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:17:51.599 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:51.597+0000 7f275e54f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:17:51.712 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:51.711+0000 7f67261ca200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:51.744 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:51.743+0000 7f275e54f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:17:51.941 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:51.939+0000 7f67261ca200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:17:51.974 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:51.972+0000 7f275e54f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:17:52.015 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:52.013+0000 7f67261ca200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:17:52.047 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:52.046+0000 7f275e54f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:17:52.279 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:52.277+0000 7f67261ca200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:17:52.314 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:52.312+0000 7f275e54f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:17:52.501 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:52.499+0000 7f67261ca200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:17:52.539 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:52.538+0000 7f275e54f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:17:52.648 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:52.647+0000 7f67261ca200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:17:52.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:52.675+0000 7fe72ffff700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 msgr2=0x7fe71c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3585499115 2023-10-16T11:17:52.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:52.675+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 0x7fe71c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T11:17:52.690 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:52.688+0000 7f275e54f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:17:52.762 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:52.761+0000 7f67261ca200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:17:52.804 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:52.803+0000 7f275e54f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:17:52.918 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:52.916+0000 7f67261ca200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:17:52.961 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:52.960+0000 7f275e54f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:17:53.180 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:53.178+0000 7f67261ca200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:53.225 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:53.224+0000 7f275e54f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:17:53.340 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:53.338+0000 7f67261ca200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:17:53.386 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:53.384+0000 7f275e54f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:17:53.864 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:17:53.863+0000 7f67261ca200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:17:53.913 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:53.911+0000 7f275e54f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:17:53.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:53.935+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mgrmap(e 128) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fe724053f00 con 0x7fe730157680 2023-10-16T11:17:53.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:53.935+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 msgr2=0x7fe71c06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:17:53.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:53.935+0000 7fe72cff9700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/3585499115,v1:172.21.15.158:6833/3585499115] conn(0x7fe71c06b070 0x7fe71c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:54.093 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d3d2f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.093 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d3d2f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.093 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d3d2f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.093 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d3d2f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.094 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d3d2f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.094 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d0d29700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.094 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d0d29700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.094 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d0d29700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.094 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d0d29700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.095 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:17:54.091+0000 7f26d0d29700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:17:54.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:54.953+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mon.2 v2:172.21.15.172:3301/0 8 ==== mgrmap(e 129) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe724080a70 con 0x7fe730157680 2023-10-16T11:17:54.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:54.953+0000 7fe72cff9700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fe71c035910 0x7fe71c075aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:54.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:54.953+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 --> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7300d0080 con 0x7fe71c035910 2023-10-16T11:17:54.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:54.954+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fe71c035910 0x7fe71c075aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:54.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:54.955+0000 7fe72ffff700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fe71c035910 0x7fe71c075aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe720000f60 tx=0x7fe720007ea0 comp rx=0 tx=0).ready entity=mgr.11229 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:55.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.014+0000 7fe72cff9700 1 -- 172.21.15.158:0/3808419371 <== mgr.11229 v2:172.21.15.158:6832/2839558305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe7300d0080 con 0x7fe71c035910 2023-10-16T11:17:55.016 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1 2023-10-16T11:17:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fe71c035910 msgr2=0x7fe71c075aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fe71c035910 0x7fe71c075aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe720000f60 tx=0x7fe720007ea0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe730157680 msgr2=0x7fe7300c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 --2- 172.21.15.158:0/3808419371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe730157680 0x7fe7300c7be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe72400c450 tx=0x7fe72400c760 comp rx=0 tx=0).stop 2023-10-16T11:17:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe734d7e700 1 -- 172.21.15.158:0/3808419371 reap_dead start 2023-10-16T11:17:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 shutdown_connections 2023-10-16T11:17:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 >> 172.21.15.158:0/3808419371 conn(0x7fe73006af00 msgr2=0x7fe730071460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 shutdown_connections 2023-10-16T11:17:55.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.016+0000 7fe7367e1700 1 -- 172.21.15.158:0/3808419371 wait complete. 2023-10-16T11:17:55.032 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T11:17:55.032 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T11:17:55.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.157+0000 7faa8f62c700 1 Processor -- start 2023-10-16T11:17:55.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.157+0000 7faa8f62c700 1 -- start start 2023-10-16T11:17:55.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa8f62c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 0x7faa880c4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa8f62c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa880ce110 con 0x7faa8806abb0 2023-10-16T11:17:55.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa8f62c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa880ce3f0 con 0x7faa88160d40 2023-10-16T11:17:55.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa8f62c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa880c61e0 con 0x7faa880c49b0 2023-10-16T11:17:55.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa8dbc9700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7faa8806abb0 0x7faa8806b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37222/0 (socket says 172.21.15.158:37222) 2023-10-16T11:17:55.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa8dbc9700 1 -- 172.21.15.158:0/856240278 learned_addr learned my addr 172.21.15.158:0/856240278 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:55.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa8cbc7700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 0x7faa880c4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442253114 0 0) 0x7faa880ce110 con 0x7faa8806abb0 2023-10-16T11:17:55.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.158+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa740035b0 con 0x7faa8806abb0 2023-10-16T11:17:55.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1969775697 0 0) 0x7faa740035b0 con 0x7faa8806abb0 2023-10-16T11:17:55.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 msgr2=0x7faa880c4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:55.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 0x7faa880c4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 >> v1:172.21.15.172:6789/0 conn(0x7faa88160d40 legacy=0x7faa880ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa880ce6d0 con 0x7faa8806abb0 2023-10-16T11:17:55.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7faa800019f0 con 0x7faa8806abb0 2023-10-16T11:17:55.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 >> v1:172.21.15.158:6789/0 conn(0x7faa8806abb0 legacy=0x7faa8806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 0x7faa74003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa74003f70 0x7faa74004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa74004d90 0x7faa74005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7400ba40 con 0x7faa74004d90 2023-10-16T11:17:55.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7400bbb0 con 0x7faa74003f70 2023-10-16T11:17:55.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7400bd70 con 0x7faa880c49b0 2023-10-16T11:17:55.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8dbc9700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa74004d90 0x7faa74005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8d3c8700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 0x7faa74003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8cbc7700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa74003f70 0x7faa74004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8dbc9700 1 -- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 msgr2=0x7faa74003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:55.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8dbc9700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880c49b0 0x7faa74003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8dbc9700 1 -- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa74003f70 msgr2=0x7faa74004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:55.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8dbc9700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa74003f70 0x7faa74004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8dbc9700 1 -- 172.21.15.158:0/856240278 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740035b0 con 0x7faa74004d90 2023-10-16T11:17:55.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa8dbc9700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa74004d90 0x7faa74005210 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7faa8000a830 tx=0x7faa74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6926cfa0c178d9ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:55.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80012600 con 0x7faa74004d90 2023-10-16T11:17:55.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa80012780 con 0x7faa74004d90 2023-10-16T11:17:55.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.159+0000 7faa7ffff700 1 -- 172.21.15.158:0/856240278 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80012b30 con 0x7faa74004d90 2023-10-16T11:17:55.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.160+0000 7faa8f62c700 1 -- 172.21.15.158:0/856240278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa74004d90 msgr2=0x7faa74005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.160+0000 7faa8f62c700 1 --2- 172.21.15.158:0/856240278 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa74004d90 0x7faa74005210 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7faa8000a830 tx=0x7faa74005880 comp rx=0 tx=0).stop 2023-10-16T11:17:55.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.160+0000 7faa8dbc9700 1 -- 172.21.15.158:0/856240278 reap_dead start 2023-10-16T11:17:55.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.160+0000 7faa8f62c700 1 -- 172.21.15.158:0/856240278 shutdown_connections 2023-10-16T11:17:55.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.160+0000 7faa8f62c700 1 -- 172.21.15.158:0/856240278 >> 172.21.15.158:0/856240278 conn(0x7faa880cc5b0 msgr2=0x7faa880cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:55.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.160+0000 7faa8f62c700 1 -- 172.21.15.158:0/856240278 shutdown_connections 2023-10-16T11:17:55.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.160+0000 7faa8f62c700 1 -- 172.21.15.158:0/856240278 wait complete. 2023-10-16T11:17:55.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.162+0000 7faa8f62c700 1 Processor -- start 2023-10-16T11:17:55.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.162+0000 7faa8f62c700 1 -- start start 2023-10-16T11:17:55.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8f62c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa8814e8d0 0x7faa881e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8f62c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa8815db10 0x7faa880b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8f62c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880b4060 0x7faa880b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8f62c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880b0c50 con 0x7faa8815db10 2023-10-16T11:17:55.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8f62c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880b4a30 con 0x7faa8814e8d0 2023-10-16T11:17:55.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8f62c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa880b1510 con 0x7faa880b4060 2023-10-16T11:17:55.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa8815db10 0x7faa880b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa8815db10 0x7faa880b0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37056/0 (socket says 172.21.15.158:37056) 2023-10-16T11:17:55.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 -- 172.21.15.158:0/3341657033 learned_addr learned my addr 172.21.15.158:0/3341657033 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:55.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8dbc9700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880b4060 0x7faa880b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8d3c8700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa8814e8d0 0x7faa881e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 -- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880b4060 msgr2=0x7faa880b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880b4060 0x7faa880b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 -- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa8814e8d0 msgr2=0x7faa881e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa8814e8d0 0x7faa881e7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 -- 172.21.15.158:0/3341657033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa880b0970 con 0x7faa8815db10 2023-10-16T11:17:55.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8d3c8700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa8814e8d0 0x7faa881e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:55.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8cbc7700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa8815db10 0x7faa880b0fd0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7faa84011aa0 tx=0x7faa84011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:55.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa7dffb700 1 -- 172.21.15.158:0/3341657033 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8400f8e0 con 0x7faa8815db10 2023-10-16T11:17:55.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa7dffb700 1 -- 172.21.15.158:0/3341657033 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8400fa40 con 0x7faa8815db10 2023-10-16T11:17:55.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa881e7220 con 0x7faa8815db10 2023-10-16T11:17:55.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.163+0000 7faa7dffb700 1 -- 172.21.15.158:0/3341657033 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8400fdf0 con 0x7faa8815db10 2023-10-16T11:17:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.164+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa881e77e0 con 0x7faa8815db10 2023-10-16T11:17:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.165+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa881e7080 con 0x7faa8815db10 2023-10-16T11:17:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.165+0000 7faa7dffb700 1 -- 172.21.15.158:0/3341657033 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 129) v1 ==== 89084+0+0 (secure 0 0 0) 0x7faa84014440 con 0x7faa8815db10 2023-10-16T11:17:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.165+0000 7faa7dffb700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faa6c06ada0 0x7faa6c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.166+0000 7faa8d3c8700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faa6c06ada0 0x7faa6c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.166+0000 7faa7dffb700 1 -- 172.21.15.158:0/3341657033 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6825+0+0 (secure 0 0 0) 0x7faa84089d80 con 0x7faa8815db10 2023-10-16T11:17:55.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.166+0000 7faa8d3c8700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faa6c06ada0 0x7faa6c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa881f5d20 tx=0x7faa78009040 comp rx=0 tx=0).ready entity=mgr.11229 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:55.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.170+0000 7faa7dffb700 1 -- 172.21.15.158:0/3341657033 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa881e7080 con 0x7faa8815db10 2023-10-16T11:17:55.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.390+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 --> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faa880ce9b0 con 0x7faa6c06ada0 2023-10-16T11:17:55.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.439+0000 7faa7dffb700 1 -- 172.21.15.158:0/3341657033 <== mgr.11229 v2:172.21.15.158:6832/2839558305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7faa780053b0 con 0x7faa6c06ada0 2023-10-16T11:17:55.441 INFO:teuthology.orchestra.run.smithi158.stdout:Retention added to path snap_test_dir1 2023-10-16T11:17:55.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.441+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faa6c06ada0 msgr2=0x7faa6c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.441+0000 7faa8f62c700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faa6c06ada0 0x7faa6c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa881f5d20 tx=0x7faa78009040 comp rx=0 tx=0).stop 2023-10-16T11:17:55.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.441+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa8815db10 msgr2=0x7faa880b0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.441+0000 7faa8f62c700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa8815db10 0x7faa880b0fd0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7faa84011aa0 tx=0x7faa84011db0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 shutdown_connections 2023-10-16T11:17:55.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faa6c06ada0 0x7faa6c06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faa880b4060 0x7faa880b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa8814e8d0 0x7faa881e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 --2- 172.21.15.158:0/3341657033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa8815db10 0x7faa880b0fd0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 >> 172.21.15.158:0/3341657033 conn(0x7faa880cc5b0 msgr2=0x7faa880ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:55.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 shutdown_connections 2023-10-16T11:17:55.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.442+0000 7faa8f62c700 1 -- 172.21.15.158:0/3341657033 wait complete. 2023-10-16T11:17:55.455 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T11:17:55.456 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-11_18_00 in ~5s... 2023-10-16T11:17:55.456 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T11:17:55.456 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T11:17:55.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.577+0000 7fea254e8700 1 Processor -- start 2023-10-16T11:17:55.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.577+0000 7fea254e8700 1 -- start start 2023-10-16T11:17:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.578+0000 7fea254e8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fea200b3630 0x7fea20166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.578+0000 7fea254e8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea200b4420 con 0x7fea200d0c60 2023-10-16T11:17:55.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.578+0000 7fea254e8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea200cbc20 con 0x7fea2015b6b0 2023-10-16T11:17:55.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.578+0000 7fea254e8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2015ba90 con 0x7fea200b3630 2023-10-16T11:17:55.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.578+0000 7fea1f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fea200b3630 0x7fea20166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.578+0000 7fea1e7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fea200d0c60 0x7fea200b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37240/0 (socket says 172.21.15.158:37240) 2023-10-16T11:17:55.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.578+0000 7fea1e7fc700 1 -- 172.21.15.158:0/3718939693 learned_addr learned my addr 172.21.15.158:0/3718939693 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:17:55.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1f7fe700 1 -- 172.21.15.158:0/3718939693 >> v1:172.21.15.172:6789/0 conn(0x7fea2015b6b0 legacy=0x7fea200cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:17:55.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1f7fe700 1 -- 172.21.15.158:0/3718939693 >> v1:172.21.15.158:6789/0 conn(0x7fea200d0c60 legacy=0x7fea200b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1dffb700 1 -- 172.21.15.158:0/3718939693 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997425880 0 0) 0x7fea200b4420 con 0x7fea200d0c60 2023-10-16T11:17:55.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1f7fe700 1 -- 172.21.15.158:0/3718939693 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea200cbf00 con 0x7fea200b3630 2023-10-16T11:17:55.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1f7fe700 1 --2- 172.21.15.158:0/3718939693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea20166120 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7fea18009d90 tx=0x7fea1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8844ea80a82859d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:55.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1dffb700 1 -- 172.21.15.158:0/3718939693 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1800e070 con 0x7fea200b3630 2023-10-16T11:17:55.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1dffb700 1 -- 172.21.15.158:0/3718939693 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1800bb90 con 0x7fea200b3630 2023-10-16T11:17:55.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea1dffb700 1 -- 172.21.15.158:0/3718939693 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea18003a20 con 0x7fea200b3630 2023-10-16T11:17:55.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea254e8700 1 -- 172.21.15.158:0/3718939693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 msgr2=0x7fea20166120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea254e8700 1 --2- 172.21.15.158:0/3718939693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea20166120 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7fea18009d90 tx=0x7fea1800b330 comp rx=0 tx=0).stop 2023-10-16T11:17:55.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea254e8700 1 -- 172.21.15.158:0/3718939693 shutdown_connections 2023-10-16T11:17:55.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.579+0000 7fea254e8700 1 --2- 172.21.15.158:0/3718939693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea20166120 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.580+0000 7fea254e8700 1 -- 172.21.15.158:0/3718939693 >> 172.21.15.158:0/3718939693 conn(0x7fea200cdf20 msgr2=0x7fea200ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:55.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.580+0000 7fea254e8700 1 -- 172.21.15.158:0/3718939693 shutdown_connections 2023-10-16T11:17:55.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.580+0000 7fea254e8700 1 -- 172.21.15.158:0/3718939693 wait complete. 2023-10-16T11:17:55.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.581+0000 7fea254e8700 1 Processor -- start 2023-10-16T11:17:55.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.581+0000 7fea254e8700 1 -- start start 2023-10-16T11:17:55.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea200c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fea200d0c60 0x7fea200ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea2015b6b0 0x7fea200aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea201f0800 con 0x7fea200b3630 2023-10-16T11:17:55.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea201f0a80 con 0x7fea2015b6b0 2023-10-16T11:17:55.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea201f0d60 con 0x7fea200d0c60 2023-10-16T11:17:55.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea200c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea200c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37068/0 (socket says 172.21.15.158:37068) 2023-10-16T11:17:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 -- 172.21.15.158:0/3525103896 learned_addr learned my addr 172.21.15.158:0/3525103896 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:17:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1f7fe700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea2015b6b0 0x7fea200aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1e7fc700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fea200d0c60 0x7fea200ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 -- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fea200d0c60 msgr2=0x7fea200ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fea200d0c60 0x7fea200ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 -- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea2015b6b0 msgr2=0x7fea200aa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea2015b6b0 0x7fea200aa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 -- 172.21.15.158:0/3525103896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18009a70 con 0x7fea200b3630 2023-10-16T11:17:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1f7fe700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea2015b6b0 0x7fea200aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:17:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea1effd700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea200c95c0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fea1400b4a0 tx=0x7fea1400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:55.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea077fe700 1 -- 172.21.15.158:0/3525103896 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14011830 con 0x7fea200b3630 2023-10-16T11:17:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea077fe700 1 -- 172.21.15.158:0/3525103896 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea14011990 con 0x7fea200b3630 2023-10-16T11:17:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea201f1040 con 0x7fea200b3630 2023-10-16T11:17:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea077fe700 1 -- 172.21.15.158:0/3525103896 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14011d40 con 0x7fea200b3630 2023-10-16T11:17:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.582+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea200ca6d0 con 0x7fea200b3630 2023-10-16T11:17:55.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.583+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea200681b0 con 0x7fea200b3630 2023-10-16T11:17:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.584+0000 7fea077fe700 1 -- 172.21.15.158:0/3525103896 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 129) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fea1401a400 con 0x7fea200b3630 2023-10-16T11:17:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.584+0000 7fea077fe700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fea0c06ada0 0x7fea0c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:17:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.584+0000 7fea077fe700 1 -- 172.21.15.158:0/3525103896 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fea14089360 con 0x7fea200b3630 2023-10-16T11:17:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.584+0000 7fea1e7fc700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fea0c06ada0 0x7fea0c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:17:55.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.585+0000 7fea1e7fc700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fea0c06ada0 0x7fea0c06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fea200cb670 tx=0x7fea10007be0 comp rx=0 tx=0).ready entity=mgr.11229 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:17:55.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.589+0000 7fea077fe700 1 -- 172.21.15.158:0/3525103896 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fea140542f0 con 0x7fea200b3630 2023-10-16T11:17:55.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.809+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 --> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea200cc4c0 con 0x7fea0c06ada0 2023-10-16T11:17:55.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.818+0000 7fea077fe700 1 -- 172.21.15.158:0/3525103896 <== mgr.11229 v2:172.21.15.158:6832/2839558305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fea10005aa0 con 0x7fea0c06ada0 2023-10-16T11:17:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:17:55.820 INFO:teuthology.orchestra.run.smithi158.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T11:17:55.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fea0c06ada0 msgr2=0x7fea0c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fea0c06ada0 0x7fea0c06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fea200cb670 tx=0x7fea10007be0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 msgr2=0x7fea200c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:17:55.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea200c95c0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fea1400b4a0 tx=0x7fea1400b860 comp rx=0 tx=0).stop 2023-10-16T11:17:55.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 shutdown_connections 2023-10-16T11:17:55.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fea0c06ada0 0x7fea0c06d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea2015b6b0 0x7fea200aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea200b3630 0x7fea200c95c0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 --2- 172.21.15.158:0/3525103896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fea200d0c60 0x7fea200ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:17:55.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 >> 172.21.15.158:0/3525103896 conn(0x7fea200cdf20 msgr2=0x7fea200b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:17:55.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 shutdown_connections 2023-10-16T11:17:55.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:17:55.820+0000 7fea254e8700 1 -- 172.21.15.158:0/3525103896 wait complete. 2023-10-16T11:17:55.834 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T11:17:55.834 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T11:17:55.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:55.835 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:17:55.853 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T11:17:55.854 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:17:55.854 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:17:56.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:56.856 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:17:56.878 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T11:17:56.879 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:17:56.879 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:17:57.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:57.881 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:17:57.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T11:17:57.903 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:17:57.903 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:17:58.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:58.905 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:17:58.928 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T11:17:58.928 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:17:58.928 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:17:59.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:17:59.930 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:17:59.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T11:17:59.954 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:17:59.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:00.451 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:18:00.449+0000 7f26c350e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:18:00.451 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:18:00.449+0000 7f26c350e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:18:00.452 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:18:00.449+0000 7f26c350e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:18:00.452 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:18:00.449+0000 7f26c350e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:18:00.452 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:18:00.449+0000 7f26c350e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:18:00.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:00.956 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:00.979 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:00.979 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:00.979 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:00.980 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:00.980 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-11_18_00_UTC'] 2023-10-16T11:18:00.980 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697455075.4561868, now = 1697455080.9808967, timo = 5 2023-10-16T11:18:01.982 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-10-16-11_18_00 in ~65s... 2023-10-16T11:18:01.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:01.983 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:02.004 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:02.005 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:02.006 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:02.006 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:03.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:03.008 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:03.030 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:03.031 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:03.031 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:03.031 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:04.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:04.033 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:04.054 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:04.055 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:04.055 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:04.056 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:05.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:05.057 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:05.078 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:05.079 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:05.080 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:05.080 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:06.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:06.081 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:06.103 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:06.104 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:06.104 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:06.104 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:07.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:07.107 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:07.128 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:07.129 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:07.130 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:07.130 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:08.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:08.131 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:08.153 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:08.154 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:08.154 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:08.154 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:09.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:09.156 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:09.177 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:09.178 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:09.178 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:09.179 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:10.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:10.180 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:10.201 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:10.202 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:10.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:10.203 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:11.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:11.205 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:11.227 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:11.227 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:11.228 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:11.228 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:12.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:12.230 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:12.251 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:12.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:12.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:12.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:13.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:13.254 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:13.276 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:13.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:13.277 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:13.277 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:14.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:14.279 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:14.300 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:14.302 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:14.302 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:14.302 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:15.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:15.304 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:15.327 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:15.329 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:15.329 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:15.330 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:16.078 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:18:16.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:18:16.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:16.332 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:16.353 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:16.354 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:16.355 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:16.355 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:17.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:17.356 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:17.377 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:17.379 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:17.379 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:17.379 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:18.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:18.381 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:18.403 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:18.404 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:18.404 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:18.405 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:19.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:19.407 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:19.428 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:19.429 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:19.429 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:19.430 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:20.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:20.431 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:20.453 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:20.454 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:20.455 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:20.455 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:21.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:21.457 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:21.478 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:21.480 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:21.480 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:21.480 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:22.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:22.482 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:22.504 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:22.505 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:22.505 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:22.505 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:23.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:23.507 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:23.528 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:23.529 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:23.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:23.530 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:24.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:24.532 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:24.554 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:24.554 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:24.555 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:24.555 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:25.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:25.557 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:25.579 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:25.580 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:25.580 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:25.580 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:26.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:26.582 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:26.603 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:26.604 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:26.604 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:26.605 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:27.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:27.607 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:27.628 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:27.630 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:27.630 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:27.630 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:28.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:28.632 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:28.653 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:28.654 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:28.654 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:28.655 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:29.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:29.657 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:29.679 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:29.679 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:29.680 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:29.680 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:30.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:30.682 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:30.703 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:30.704 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:30.704 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:30.705 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:31.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:31.706 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:31.727 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:31.728 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:31.728 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:31.729 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:32.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:32.730 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:32.751 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:32.752 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:32.752 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:32.753 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:33.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:33.754 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:33.776 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:33.777 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:33.777 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:33.777 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:34.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:34.780 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:34.801 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:34.802 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:34.802 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:34.803 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:35.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:35.805 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:35.826 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:35.827 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:35.828 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:35.828 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:36.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:36.830 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:36.851 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:36.852 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:36.852 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:36.853 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:37.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:37.854 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:37.875 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:37.876 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:37.877 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:37.877 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:38.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:38.879 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:38.902 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:38.902 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:38.903 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:38.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:39.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:39.906 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:39.927 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:39.928 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:39.929 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:39.929 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:40.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:40.931 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:40.952 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:40.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:40.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:40.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:41.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:41.955 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:41.977 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:41.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:41.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:41.978 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:42.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:42.980 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:43.002 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:43.002 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:43.002 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:43.003 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:44.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:44.004 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:44.026 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:44.027 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:44.027 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:44.028 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:45.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:45.029 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:45.051 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:45.052 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:45.052 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:45.052 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:46.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:46.054 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:46.076 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:46.077 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:46.077 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:46.077 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:46.216 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:18:46.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:18:47.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:47.079 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:47.101 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:47.101 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:47.102 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:47.102 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:48.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:48.104 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:48.125 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:48.126 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:48.127 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:48.127 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:49.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:49.129 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:49.150 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:49.151 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:49.152 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:49.152 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:50.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:50.154 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:50.177 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:50.177 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:50.177 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:50.178 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:51.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:51.180 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:51.201 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:51.202 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:51.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:51.203 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:52.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:52.204 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:52.225 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:52.226 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:52.227 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:52.227 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:53.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:53.229 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:53.250 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:53.251 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:53.252 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:53.252 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:54.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:54.254 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:54.275 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:54.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:54.277 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:54.277 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:55.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:55.279 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:55.300 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:55.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:55.302 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:55.302 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:56.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:56.304 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:56.325 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:56.326 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:56.327 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:56.327 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:57.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:57.328 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:57.350 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:57.350 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:57.351 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:57.351 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:58.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:58.353 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:58.375 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:58.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:58.375 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:58.376 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:18:59.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:18:59.377 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:18:59.399 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:18:59.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:18:59.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:18:59.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:19:00.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:00.402 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:19:00.424 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_18_00_UTC 2023-10-16T11:19:00.425 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:19:00.425 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T11:19:00.425 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T11:19:01.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:01.427 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:19:01.448 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_19_00_UTC 2023-10-16T11:19:01.449 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-11_19_00_UTC'} 2023-10-16T11:19:01.450 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-11_19_00_UTC'} 2023-10-16T11:19:01.450 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-10-16-11_18_00_UTC'} 2023-10-16T11:19:01.450 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-10-16-11_18_00_UTC'] 2023-10-16T11:19:01.451 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697455075.4561868, now = 1697455141.450984, timo = 65 2023-10-16T11:19:02.452 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T11:19:02.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.586+0000 7fba5c524700 1 Processor -- start 2023-10-16T11:19:02.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.586+0000 7fba5c524700 1 -- start start 2023-10-16T11:19:02.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.587+0000 7fba5c524700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fba540ce510 0x7fba5415f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.587+0000 7fba5c524700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 0x7fba540cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.587+0000 7fba5c524700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540cf4c0 con 0x7fba5415b590 2023-10-16T11:19:02.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.587+0000 7fba5c524700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba540cc4c0 con 0x7fba540ce510 2023-10-16T11:19:02.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.587+0000 7fba5c524700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba540b49e0 con 0x7fba540cbaf0 2023-10-16T11:19:02.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.587+0000 7fba5a2c0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fba5415b590 0x7fba540b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37284/0 (socket says 172.21.15.158:37284) 2023-10-16T11:19:02.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.587+0000 7fba5a2c0700 1 -- 172.21.15.158:0/588364567 learned_addr learned my addr 172.21.15.158:0/588364567 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:02.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307800050 0 0) 0x7fba540cf4c0 con 0x7fba5415b590 2023-10-16T11:19:02.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5aac1700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 0x7fba540cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba59abf700 1 --2- 172.21.15.158:0/588364567 >> v2:172.21.15.172:3300/0 conn(0x7fba540ce510 0x7fba5415f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba400035b0 con 0x7fba5415b590 2023-10-16T11:19:02.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381054063 0 0) 0x7fba400035b0 con 0x7fba5415b590 2023-10-16T11:19:02.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 msgr2=0x7fba540cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:02.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 0x7fba540cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 >> v2:172.21.15.172:3300/0 conn(0x7fba540ce510 msgr2=0x7fba5415f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:02.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 --2- 172.21.15.158:0/588364567 >> v2:172.21.15.172:3300/0 conn(0x7fba540ce510 0x7fba5415f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5415f710 con 0x7fba5415b590 2023-10-16T11:19:02.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fba44001670 con 0x7fba5415b590 2023-10-16T11:19:02.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 >> v1:172.21.15.158:6789/0 conn(0x7fba5415b590 legacy=0x7fba540b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:02.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 0x7fba40003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba40003fb0 0x7fba40004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba40004d90 0x7fba40005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba540cc4c0 con 0x7fba40004d90 2023-10-16T11:19:02.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba540b49e0 con 0x7fba40003fb0 2023-10-16T11:19:02.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4000ba90 con 0x7fba540cbaf0 2023-10-16T11:19:02.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5aac1700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba40004d90 0x7fba40005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5a2c0700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 0x7fba40003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba59abf700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba40003fb0 0x7fba40004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5aac1700 1 -- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 msgr2=0x7fba40003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:02.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5aac1700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 0x7fba40003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5aac1700 1 -- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba40003fb0 msgr2=0x7fba40004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:02.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5aac1700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba40003fb0 0x7fba40004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5aac1700 1 -- 172.21.15.158:0/588364567 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba400035b0 con 0x7fba40004d90 2023-10-16T11:19:02.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.588+0000 7fba5a2c0700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba540cbaf0 0x7fba40003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:02.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5aac1700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba40004d90 0x7fba40005210 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009fc0 tx=0x7fba40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24470930d63d2207 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:02.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00ec30 con 0x7fba40004d90 2023-10-16T11:19:02.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c00f040 con 0x7fba40004d90 2023-10-16T11:19:02.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba592be700 1 -- 172.21.15.158:0/588364567 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c014480 con 0x7fba40004d90 2023-10-16T11:19:02.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5c524700 1 -- 172.21.15.158:0/588364567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba40004d90 msgr2=0x7fba40005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:02.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5c524700 1 --2- 172.21.15.158:0/588364567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba40004d90 0x7fba40005210 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009fc0 tx=0x7fba40005880 comp rx=0 tx=0).stop 2023-10-16T11:19:02.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5aac1700 1 -- 172.21.15.158:0/588364567 reap_dead start 2023-10-16T11:19:02.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5c524700 1 -- 172.21.15.158:0/588364567 shutdown_connections 2023-10-16T11:19:02.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5c524700 1 -- 172.21.15.158:0/588364567 >> 172.21.15.158:0/588364567 conn(0x7fba5406a8d0 msgr2=0x7fba5406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:02.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5c524700 1 -- 172.21.15.158:0/588364567 shutdown_connections 2023-10-16T11:19:02.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.589+0000 7fba5c524700 1 -- 172.21.15.158:0/588364567 wait complete. 2023-10-16T11:19:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.591+0000 7fba5c524700 1 Processor -- start 2023-10-16T11:19:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.591+0000 7fba5c524700 1 -- start start 2023-10-16T11:19:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5c524700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba40004d90 0x7fba54152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5c524700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5415c8d0 0x7fba5415d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5c524700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5415cd50 0x7fba5415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5c524700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5415dd60 con 0x7fba5415cd50 2023-10-16T11:19:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5c524700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba54152f90 con 0x7fba5415c8d0 2023-10-16T11:19:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5c524700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5415c440 con 0x7fba40004d90 2023-10-16T11:19:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5415cd50 0x7fba5415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5415cd50 0x7fba5415d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37122/0 (socket says 172.21.15.158:37122) 2023-10-16T11:19:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 -- 172.21.15.158:0/3040253408 learned_addr learned my addr 172.21.15.158:0/3040253408 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5a2c0700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba40004d90 0x7fba54152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba59abf700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5415c8d0 0x7fba5415d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 -- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba40004d90 msgr2=0x7fba54152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba40004d90 0x7fba54152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 -- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5415c8d0 msgr2=0x7fba5415d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:02.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5415c8d0 0x7fba5415d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba5aac1700 1 -- 172.21.15.158:0/3040253408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba540cb3b0 con 0x7fba5415cd50 2023-10-16T11:19:02.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.592+0000 7fba59abf700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5415c8d0 0x7fba5415d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:02.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.593+0000 7fba5aac1700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5415cd50 0x7fba5415d730 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fba4c017040 tx=0x7fba4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.593+0000 7fba4affd700 1 -- 172.21.15.158:0/3040253408 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c007000 con 0x7fba5415cd50 2023-10-16T11:19:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.593+0000 7fba4affd700 1 -- 172.21.15.158:0/3040253408 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c003e50 con 0x7fba5415cd50 2023-10-16T11:19:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.593+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5415bb10 con 0x7fba5415cd50 2023-10-16T11:19:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.593+0000 7fba4affd700 1 -- 172.21.15.158:0/3040253408 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c014b50 con 0x7fba5415cd50 2023-10-16T11:19:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.593+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5415c0d0 con 0x7fba5415cd50 2023-10-16T11:19:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.594+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5415c290 con 0x7fba5415cd50 2023-10-16T11:19:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.594+0000 7fba4affd700 1 -- 172.21.15.158:0/3040253408 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 132) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fba4c023020 con 0x7fba5415cd50 2023-10-16T11:19:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.594+0000 7fba4affd700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fba3806b190 0x7fba3806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.595+0000 7fba4affd700 1 -- 172.21.15.158:0/3040253408 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6941+0+0 (secure 0 0 0) 0x7fba4c08b4b0 con 0x7fba5415cd50 2023-10-16T11:19:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.595+0000 7fba5a2c0700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fba3806b190 0x7fba3806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.595+0000 7fba5a2c0700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fba3806b190 0x7fba3806d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba540ca470 tx=0x7fba44009040 comp rx=0 tx=0).ready entity=mgr.11229 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:02.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.599+0000 7fba4affd700 1 -- 172.21.15.158:0/3040253408 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba4c055ee0 con 0x7fba5415cd50 2023-10-16T11:19:02.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.821+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 --> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fba5415fc50 con 0x7fba3806b190 2023-10-16T11:19:02.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.879+0000 7fba4affd700 1 -- 172.21.15.158:0/3040253408 <== mgr.11229 v2:172.21.15.158:6832/2839558305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fba44004940 con 0x7fba3806b190 2023-10-16T11:19:02.881 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path snap_test_dir1 2023-10-16T11:19:02.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fba3806b190 msgr2=0x7fba3806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:02.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fba3806b190 0x7fba3806d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba540ca470 tx=0x7fba44009040 comp rx=0 tx=0).stop 2023-10-16T11:19:02.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5415cd50 msgr2=0x7fba5415d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:02.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5415cd50 0x7fba5415d730 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fba4c017040 tx=0x7fba4c009750 comp rx=0 tx=0).stop 2023-10-16T11:19:02.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 shutdown_connections 2023-10-16T11:19:02.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fba3806b190 0x7fba3806d650 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5415cd50 0x7fba5415d730 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba40004d90 0x7fba54152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 --2- 172.21.15.158:0/3040253408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5415c8d0 0x7fba5415d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:02.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 >> 172.21.15.158:0/3040253408 conn(0x7fba5406a8d0 msgr2=0x7fba5415ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:02.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 shutdown_connections 2023-10-16T11:19:02.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:02.881+0000 7fba5c524700 1 -- 172.21.15.158:0/3040253408 wait complete. 2023-10-16T11:19:02.895 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T11:19:02.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:02.896 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:19:02.913 INFO:teuthology.orchestra.run.smithi158.stdout:scheduled-2023-10-16-11_19_00_UTC 2023-10-16T11:19:02.913 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-11_19_00_UTC 2023-10-16T11:19:02.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:02.914 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-11_19_00_UTC) 2023-10-16T11:19:02.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:02.976 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T11:19:03.035 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:19:03.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.196+0000 7f91e56d2700 1 Processor -- start 2023-10-16T11:19:03.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.196+0000 7f91e56d2700 1 -- start start 2023-10-16T11:19:03.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.196+0000 7f91e56d2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91e00cddb0 0x7f91e006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:03.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.196+0000 7f91e56d2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f91e006e400 0x7f91e00ac970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:03.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.196+0000 7f91e56d2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00af360 con 0x7f91e015d9d0 2023-10-16T11:19:03.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.196+0000 7f91e56d2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e006b480 con 0x7f91e00cddb0 2023-10-16T11:19:03.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.196+0000 7f91e56d2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e006b760 con 0x7f91e006e400 2023-10-16T11:19:03.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91e00cddb0 0x7f91e006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:03.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91e00cddb0 0x7f91e006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37130/0 (socket says 172.21.15.158:37130) 2023-10-16T11:19:03.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 -- 172.21.15.158:0/3419279813 learned_addr learned my addr 172.21.15.158:0/3419279813 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:03.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 -- 172.21.15.158:0/3419279813 >> v2:172.21.15.172:3300/0 conn(0x7f91e006e400 msgr2=0x7f91e00ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:03.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91df7fe700 1 --2- 172.21.15.158:0/3419279813 >> v2:172.21.15.172:3300/0 conn(0x7f91e006e400 0x7f91e00ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:03.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 --2- 172.21.15.158:0/3419279813 >> v2:172.21.15.172:3300/0 conn(0x7f91e006e400 0x7f91e00ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 -- 172.21.15.158:0/3419279813 >> v1:172.21.15.172:6789/0 conn(0x7f91e015d9d0 legacy=0x7f91e00cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:03.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 -- 172.21.15.158:0/3419279813 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e006ba40 con 0x7f91e00cddb0 2023-10-16T11:19:03.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91de7fc700 1 --2- 172.21.15.158:0/3419279813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e006e930 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f91e00cf9b0 tx=0x7f91d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5227978881fe01d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:03.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91ddffb700 1 -- 172.21.15.158:0/3419279813 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4004230 con 0x7f91e00cddb0 2023-10-16T11:19:03.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91ddffb700 1 -- 172.21.15.158:0/3419279813 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4004390 con 0x7f91e00cddb0 2023-10-16T11:19:03.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.197+0000 7f91ddffb700 1 -- 172.21.15.158:0/3419279813 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d40036a0 con 0x7f91e00cddb0 2023-10-16T11:19:03.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 -- 172.21.15.158:0/3419279813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 msgr2=0x7f91e006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:03.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 --2- 172.21.15.158:0/3419279813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e006e930 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f91e00cf9b0 tx=0x7f91d4009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 -- 172.21.15.158:0/3419279813 shutdown_connections 2023-10-16T11:19:03.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 --2- 172.21.15.158:0/3419279813 >> v2:172.21.15.172:3300/0 conn(0x7f91e006e400 0x7f91e00ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 --2- 172.21.15.158:0/3419279813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e006e930 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 -- 172.21.15.158:0/3419279813 >> 172.21.15.158:0/3419279813 conn(0x7f91e00b15c0 msgr2=0x7f91e00b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:03.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 -- 172.21.15.158:0/3419279813 shutdown_connections 2023-10-16T11:19:03.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.198+0000 7f91e56d2700 1 -- 172.21.15.158:0/3419279813 wait complete. 2023-10-16T11:19:03.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.199+0000 7f91e56d2700 1 Processor -- start 2023-10-16T11:19:03.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.199+0000 7f91e56d2700 1 -- start start 2023-10-16T11:19:03.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91e56d2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91e006e400 0x7f91e0154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:03.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91e56d2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:03.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91e56d2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91e015d9d0 0x7f91e0160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:03.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91e56d2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00c8e40 con 0x7f91e00cddb0 2023-10-16T11:19:03.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91e56d2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e015b6b0 con 0x7f91e015d9d0 2023-10-16T11:19:03.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91e56d2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00c6960 con 0x7f91e006e400 2023-10-16T11:19:03.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:03.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e015fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37136/0 (socket says 172.21.15.158:37136) 2023-10-16T11:19:03.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 -- 172.21.15.158:0/671962380 learned_addr learned my addr 172.21.15.158:0/671962380 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:03.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91df7fe700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91e015d9d0 0x7f91e0160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:03.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91deffd700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91e006e400 0x7f91e0154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:03.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 -- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91e006e400 msgr2=0x7f91e0154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:03.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91e006e400 0x7f91e0154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 -- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91e015d9d0 msgr2=0x7f91e0160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:03.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91e015d9d0 0x7f91e0160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 -- 172.21.15.158:0/671962380 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4009550 con 0x7f91e00cddb0 2023-10-16T11:19:03.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.200+0000 7f91de7fc700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e015fdc0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009130 tx=0x7f91d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:03.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.201+0000 7f91c77fe700 1 -- 172.21.15.158:0/671962380 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4003dc0 con 0x7f91e00cddb0 2023-10-16T11:19:03.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.201+0000 7f91c77fe700 1 -- 172.21.15.158:0/671962380 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d401f070 con 0x7f91e00cddb0 2023-10-16T11:19:03.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.201+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e01f2ea0 con 0x7f91e00cddb0 2023-10-16T11:19:03.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.201+0000 7f91c77fe700 1 -- 172.21.15.158:0/671962380 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4011870 con 0x7f91e00cddb0 2023-10-16T11:19:03.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.201+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e01f3410 con 0x7f91e00cddb0 2023-10-16T11:19:03.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.202+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e00681b0 con 0x7f91e00cddb0 2023-10-16T11:19:03.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.202+0000 7f91c77fe700 1 -- 172.21.15.158:0/671962380 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 132) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f91d4021030 con 0x7f91e00cddb0 2023-10-16T11:19:03.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.202+0000 7f91c77fe700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f91cc06b190 0x7f91cc06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:03.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.202+0000 7f91c77fe700 1 -- 172.21.15.158:0/671962380 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6941+0+0 (secure 0 0 0) 0x7f91d408b550 con 0x7f91e00cddb0 2023-10-16T11:19:03.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.202+0000 7f91deffd700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f91cc06b190 0x7f91cc06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:03.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.203+0000 7f91deffd700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f91cc06b190 0x7f91cc06d650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f91e015b120 tx=0x7f91d0008040 comp rx=0 tx=0).ready entity=mgr.11229 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:03.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.207+0000 7f91c77fe700 1 -- 172.21.15.158:0/671962380 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91d4055e90 con 0x7f91e00cddb0 2023-10-16T11:19:03.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.454+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f91e006f4b0 con 0x7f91e00cddb0 2023-10-16T11:19:03.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.774+0000 7f91c77fe700 1 -- 172.21.15.158:0/671962380 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v133) v1 ==== 93+0+0 (secure 0 0 0) 0x7f91d40594b0 con 0x7f91e00cddb0 2023-10-16T11:19:03.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.777+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f91cc06b190 msgr2=0x7f91cc06d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:03.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.777+0000 7f91e56d2700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f91cc06b190 0x7f91cc06d650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f91e015b120 tx=0x7f91d0008040 comp rx=0 tx=0).stop 2023-10-16T11:19:03.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.777+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 msgr2=0x7f91e015fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:03.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.777+0000 7f91e56d2700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e015fdc0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009130 tx=0x7f91d4004750 comp rx=0 tx=0).stop 2023-10-16T11:19:03.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.778+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 shutdown_connections 2023-10-16T11:19:03.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.778+0000 7f91e56d2700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f91cc06b190 0x7f91cc06d650 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.778+0000 7f91e56d2700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91e015d9d0 0x7f91e0160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.778+0000 7f91e56d2700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91e006e400 0x7f91e0154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.778+0000 7f91e56d2700 1 --2- 172.21.15.158:0/671962380 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91e00cddb0 0x7f91e015fdc0 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:03.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.778+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 >> 172.21.15.158:0/671962380 conn(0x7f91e00b15c0 msgr2=0x7f91e0070fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:03.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.779+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 shutdown_connections 2023-10-16T11:19:03.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:03.779+0000 7f91e56d2700 1 -- 172.21.15.158:0/671962380 wait complete. 2023-10-16T11:19:03.796 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:19:03.796 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:19:03.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:19:03.879 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:19:03.879 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:03.880 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:03.896 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:03.897 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:03.926 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:03.925+0000 7faf26f8d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:19:03.927 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:03.925+0000 7f34fb1c9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:19:03.943 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:03.993 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:03.991+0000 7f34fb1c9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:03.995 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:03.993+0000 7faf26f8d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:04.066 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:04.064+0000 7f34fb1c9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:19:04.070 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:04.069+0000 7faf26f8d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:19:04.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.104+0000 7ff92439f700 1 Processor -- start 2023-10-16T11:19:04.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.105+0000 7ff92439f700 1 -- start start 2023-10-16T11:19:04.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.105+0000 7ff92439f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c0729b0 0x7ff91c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.105+0000 7ff92439f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c161270 con 0x7ff91c06ab40 2023-10-16T11:19:04.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.105+0000 7ff92439f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c161510 con 0x7ff91c06e5d0 2023-10-16T11:19:04.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.105+0000 7ff92439f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c1617f0 con 0x7ff91c0729b0 2023-10-16T11:19:04.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff92213b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff91c06ab40 0x7ff91c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37366/0 (socket says 172.21.15.158:37366) 2023-10-16T11:19:04.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff92213b700 1 -- 172.21.15.158:0/2030693114 learned_addr learned my addr 172.21.15.158:0/2030693114 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:04.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff92293c700 1 --2- 172.21.15.158:0/2030693114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c0729b0 0x7ff91c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff921139700 1 -- 172.21.15.158:0/2030693114 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780986498 0 0) 0x7ff91c161270 con 0x7ff91c06ab40 2023-10-16T11:19:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff921139700 1 -- 172.21.15.158:0/2030693114 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9080035b0 con 0x7ff91c06ab40 2023-10-16T11:19:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff921139700 1 -- 172.21.15.158:0/2030693114 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937195590 0 0) 0x7ff91c161510 con 0x7ff91c06e5d0 2023-10-16T11:19:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff92293c700 1 -- 172.21.15.158:0/2030693114 >> v1:172.21.15.172:6789/0 conn(0x7ff91c06e5d0 legacy=0x7ff91c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff92293c700 1 -- 172.21.15.158:0/2030693114 >> v1:172.21.15.158:6789/0 conn(0x7ff91c06ab40 legacy=0x7ff91c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff92293c700 1 -- 172.21.15.158:0/2030693114 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c161ad0 con 0x7ff91c0729b0 2023-10-16T11:19:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff92293c700 1 --2- 172.21.15.158:0/2030693114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c0729b0 0x7ff91c06d820 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff914009dc0 tx=0x7ff91400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4557e8e407cfb65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff921139700 1 -- 172.21.15.158:0/2030693114 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91400e070 con 0x7ff91c0729b0 2023-10-16T11:19:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff921139700 1 -- 172.21.15.158:0/2030693114 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91400bb20 con 0x7ff91c0729b0 2023-10-16T11:19:04.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.106+0000 7ff921139700 1 -- 172.21.15.158:0/2030693114 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91400bed0 con 0x7ff91c0729b0 2023-10-16T11:19:04.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.107+0000 7ff92439f700 1 -- 172.21.15.158:0/2030693114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c0729b0 msgr2=0x7ff91c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.107+0000 7ff92439f700 1 --2- 172.21.15.158:0/2030693114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c0729b0 0x7ff91c06d820 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff914009dc0 tx=0x7ff91400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:04.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.107+0000 7ff92439f700 1 -- 172.21.15.158:0/2030693114 shutdown_connections 2023-10-16T11:19:04.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.107+0000 7ff92439f700 1 --2- 172.21.15.158:0/2030693114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c0729b0 0x7ff91c06d820 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.107+0000 7ff92439f700 1 -- 172.21.15.158:0/2030693114 >> 172.21.15.158:0/2030693114 conn(0x7ff91c0d02b0 msgr2=0x7ff91c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:04.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.107+0000 7ff92439f700 1 -- 172.21.15.158:0/2030693114 shutdown_connections 2023-10-16T11:19:04.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.107+0000 7ff92439f700 1 -- 172.21.15.158:0/2030693114 wait complete. 2023-10-16T11:19:04.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.108+0000 7ff92439f700 1 Processor -- start 2023-10-16T11:19:04.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.108+0000 7ff92439f700 1 -- start start 2023-10-16T11:19:04.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92439f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c06ab40 0x7ff91c1f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92439f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff91c06e5d0 0x7ff91c0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92439f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff91c0729b0 0x7ff91c0b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92439f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c14c170 con 0x7ff91c06e5d0 2023-10-16T11:19:04.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92439f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c15f470 con 0x7ff91c0729b0 2023-10-16T11:19:04.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92439f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c14cac0 con 0x7ff91c06ab40 2023-10-16T11:19:04.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92193a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff91c06e5d0 0x7ff91c0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92193a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff91c06e5d0 0x7ff91c0b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37196/0 (socket says 172.21.15.158:37196) 2023-10-16T11:19:04.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92193a700 1 -- 172.21.15.158:0/3041673026 learned_addr learned my addr 172.21.15.158:0/3041673026 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:04.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92213b700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c06ab40 0x7ff91c1f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92293c700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff91c0729b0 0x7ff91c0b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.109+0000 7ff92193a700 1 -- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c06ab40 msgr2=0x7ff91c1f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff92193a700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c06ab40 0x7ff91c1f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff92193a700 1 -- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff91c0729b0 msgr2=0x7ff91c0b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff92193a700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff91c0729b0 0x7ff91c0b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff92193a700 1 -- 172.21.15.158:0/3041673026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914009a70 con 0x7ff91c06e5d0 2023-10-16T11:19:04.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff92193a700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff91c06e5d0 0x7ff91c0b0bb0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c520 tx=0x7ff91800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:04.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff912ffd700 1 -- 172.21.15.158:0/3041673026 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9180054e0 con 0x7ff91c06e5d0 2023-10-16T11:19:04.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff912ffd700 1 -- 172.21.15.158:0/3041673026 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff918003680 con 0x7ff91c06e5d0 2023-10-16T11:19:04.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c0b1700 con 0x7ff91c06e5d0 2023-10-16T11:19:04.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff912ffd700 1 -- 172.21.15.158:0/3041673026 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918003a30 con 0x7ff91c06e5d0 2023-10-16T11:19:04.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.110+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c0b1cc0 con 0x7ff91c06e5d0 2023-10-16T11:19:04.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.111+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c0b1e50 con 0x7ff91c06e5d0 2023-10-16T11:19:04.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.112+0000 7ff912ffd700 1 -- 172.21.15.158:0/3041673026 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff918003b90 con 0x7ff91c06e5d0 2023-10-16T11:19:04.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.112+0000 7ff912ffd700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 0x7ff90006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.112+0000 7ff92213b700 1 -- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 msgr2=0x7ff90006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:04.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.112+0000 7ff912ffd700 1 -- 172.21.15.158:0/3041673026 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6941+0+0 (secure 0 0 0) 0x7ff918089270 con 0x7ff91c06e5d0 2023-10-16T11:19:04.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.112+0000 7ff92213b700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 0x7ff90006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:04.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.116+0000 7ff912ffd700 1 -- 172.21.15.158:0/3041673026 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff918053b80 con 0x7ff91c06e5d0 2023-10-16T11:19:04.144 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:04.143+0000 7f34fb1c9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:19:04.156 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:04.154+0000 7faf26f8d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:19:04.285 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:04.284+0000 7f34fb1c9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:19:04.300 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:04.299+0000 7faf26f8d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:19:04.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.313+0000 7ff92213b700 1 -- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 msgr2=0x7ff90006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:04.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.313+0000 7ff92213b700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 0x7ff90006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.315+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff91c15db20 con 0x7ff91c06e5d0 2023-10-16T11:19:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.317+0000 7ff912ffd700 1 -- 172.21.15.158:0/3041673026 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) v1 ==== 74+0+18118 (secure 0 0 0) 0x7ff9180571a0 con 0x7ff91c06e5d0 2023-10-16T11:19:04.319 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":203,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:18:55.697135+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:17:38.499055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T11:17:39.035057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"200","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.1b","mappings":[{"from":3,"to":0}]},{"pgid":"26.29","mappings":[{"from":1,"to":6}]},{"pgid":"26.3e","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2415272176":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6833/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6832/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/778612447":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/1354527416":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/3047051699":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/4256533896":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/716693565":"2023-10-17T11:17:53.925811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 msgr2=0x7ff90006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 0x7ff90006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff91c06e5d0 msgr2=0x7ff91c0b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff91c06e5d0 0x7ff91c0b0bb0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c520 tx=0x7ff91800c830 comp rx=0 tx=0).stop 2023-10-16T11:19:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 shutdown_connections 2023-10-16T11:19:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7ff90006b140 0x7ff90006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff91c0729b0 0x7ff91c0b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff91c06ab40 0x7ff91c1f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 --2- 172.21.15.158:0/3041673026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff91c06e5d0 0x7ff91c0b0bb0 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 >> 172.21.15.158:0/3041673026 conn(0x7ff91c0d02b0 msgr2=0x7ff91c072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 shutdown_connections 2023-10-16T11:19:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.319+0000 7ff92439f700 1 -- 172.21.15.158:0/3041673026 wait complete. 2023-10-16T11:19:04.334 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:17:38.499055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:17:39.035057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '200', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:19:04.336 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:19:04.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.500+0000 7f0cfd705700 1 Processor -- start 2023-10-16T11:19:04.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.500+0000 7f0cfd705700 1 -- start start 2023-10-16T11:19:04.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.500+0000 7f0cfd705700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf815b540 0x7f0cf806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.500+0000 7f0cfd705700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf815ee60 con 0x7f0cf806afe0 2023-10-16T11:19:04.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.500+0000 7f0cfd705700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf815e2e0 con 0x7f0cf806f180 2023-10-16T11:19:04.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.500+0000 7f0cfd705700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf815b920 con 0x7f0cf815b540 2023-10-16T11:19:04.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0cf806afe0 0x7f0cf8161860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37410/0 (socket says 172.21.15.158:37410) 2023-10-16T11:19:04.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf77fe700 1 -- 172.21.15.158:0/3576350435 learned_addr learned my addr 172.21.15.158:0/3576350435 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:04.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf6ffd700 1 --2- 172.21.15.158:0/3576350435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf815b540 0x7f0cf806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf5ffb700 1 -- 172.21.15.158:0/3576350435 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577663183 0 0) 0x7f0cf815ee60 con 0x7f0cf806afe0 2023-10-16T11:19:04.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf5ffb700 1 -- 172.21.15.158:0/3576350435 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00035b0 con 0x7f0cf806afe0 2023-10-16T11:19:04.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf5ffb700 1 -- 172.21.15.158:0/3576350435 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157101898 0 0) 0x7f0cf815e2e0 con 0x7f0cf806f180 2023-10-16T11:19:04.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf5ffb700 1 -- 172.21.15.158:0/3576350435 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf815ee60 con 0x7f0cf806f180 2023-10-16T11:19:04.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf6ffd700 1 -- 172.21.15.158:0/3576350435 >> v1:172.21.15.172:6789/0 conn(0x7f0cf806f180 legacy=0x7f0cf806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf6ffd700 1 -- 172.21.15.158:0/3576350435 >> v1:172.21.15.158:6789/0 conn(0x7f0cf806afe0 legacy=0x7f0cf8161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf6ffd700 1 -- 172.21.15.158:0/3576350435 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf815e5c0 con 0x7f0cf815b540 2023-10-16T11:19:04.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf6ffd700 1 --2- 172.21.15.158:0/3576350435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf815b540 0x7f0cf806d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009870 tx=0x7f0ce8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d52ce8cb9a1e96e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:04.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf5ffb700 1 -- 172.21.15.158:0/3576350435 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8011830 con 0x7f0cf815b540 2023-10-16T11:19:04.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf5ffb700 1 -- 172.21.15.158:0/3576350435 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce8011990 con 0x7f0cf815b540 2023-10-16T11:19:04.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.501+0000 7f0cf5ffb700 1 -- 172.21.15.158:0/3576350435 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8011d00 con 0x7f0cf815b540 2023-10-16T11:19:04.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.502+0000 7f0cfd705700 1 -- 172.21.15.158:0/3576350435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf815b540 msgr2=0x7f0cf806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.502+0000 7f0cfd705700 1 --2- 172.21.15.158:0/3576350435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf815b540 0x7f0cf806d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009870 tx=0x7f0ce8009b80 comp rx=0 tx=0).stop 2023-10-16T11:19:04.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.502+0000 7f0cfd705700 1 -- 172.21.15.158:0/3576350435 shutdown_connections 2023-10-16T11:19:04.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.502+0000 7f0cfd705700 1 --2- 172.21.15.158:0/3576350435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf815b540 0x7f0cf806d9c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.502+0000 7f0cfd705700 1 -- 172.21.15.158:0/3576350435 >> 172.21.15.158:0/3576350435 conn(0x7f0cf80d03b0 msgr2=0x7f0cf80d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:04.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.502+0000 7f0cfd705700 1 -- 172.21.15.158:0/3576350435 shutdown_connections 2023-10-16T11:19:04.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.502+0000 7f0cfd705700 1 -- 172.21.15.158:0/3576350435 wait complete. 2023-10-16T11:19:04.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.504+0000 7f0cfd705700 1 Processor -- start 2023-10-16T11:19:04.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.504+0000 7f0cfd705700 1 -- start start 2023-10-16T11:19:04.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cfd705700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf806afe0 0x7f0cf81f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cfd705700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf806f180 0x7f0cf815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cfd705700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf815b540 0x7f0cf80b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cfd705700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf815caa0 con 0x7f0cf815b540 2023-10-16T11:19:04.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cfd705700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81f30e0 con 0x7f0cf806f180 2023-10-16T11:19:04.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cfd705700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf814d2f0 con 0x7f0cf806afe0 2023-10-16T11:19:04.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf815b540 0x7f0cf80b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf67fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf806f180 0x7f0cf815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf6ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf806afe0 0x7f0cf81f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf67fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf806f180 0x7f0cf815ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:38418/0 (socket says 172.21.15.158:38418) 2023-10-16T11:19:04.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf67fc700 1 -- 172.21.15.158:0/22810085 learned_addr learned my addr 172.21.15.158:0/22810085 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:04.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf815b540 0x7f0cf80b06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37240/0 (socket says 172.21.15.158:37240) 2023-10-16T11:19:04.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 -- 172.21.15.158:0/22810085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf806afe0 msgr2=0x7f0cf81f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf806afe0 0x7f0cf81f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 -- 172.21.15.158:0/22810085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf806f180 msgr2=0x7f0cf815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf806f180 0x7f0cf815ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 -- 172.21.15.158:0/22810085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80094a0 con 0x7f0cf815b540 2023-10-16T11:19:04.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf6ffd700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf806afe0 0x7f0cf81f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:04.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.505+0000 7f0cf77fe700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf815b540 0x7f0cf80b06c0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80ca9b0 tx=0x7f0cf000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.506+0000 7f0cdf7fe700 1 -- 172.21.15.158:0/22810085 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000a9a0 con 0x7f0cf815b540 2023-10-16T11:19:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.506+0000 7f0cdf7fe700 1 -- 172.21.15.158:0/22810085 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf000ab00 con 0x7f0cf815b540 2023-10-16T11:19:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.506+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf815c4e0 con 0x7f0cf815b540 2023-10-16T11:19:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.506+0000 7f0cdf7fe700 1 -- 172.21.15.158:0/22810085 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000ae70 con 0x7f0cf815b540 2023-10-16T11:19:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.506+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf80aca30 con 0x7f0cf815b540 2023-10-16T11:19:04.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.507+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf80ac890 con 0x7f0cf815b540 2023-10-16T11:19:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.507+0000 7f0cdf7fe700 1 -- 172.21.15.158:0/22810085 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f0cf001b3b0 con 0x7f0cf815b540 2023-10-16T11:19:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.508+0000 7f0cdf7fe700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 0x7f0ce406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.508+0000 7f0cf6ffd700 1 -- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 msgr2=0x7f0ce406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.508+0000 7f0cdf7fe700 1 -- 172.21.15.158:0/22810085 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6941+0+0 (secure 0 0 0) 0x7f0cf0089930 con 0x7f0cf815b540 2023-10-16T11:19:04.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.508+0000 7f0cf6ffd700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 0x7f0ce406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:04.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.512+0000 7f0cdf7fe700 1 -- 172.21.15.158:0/22810085 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0cf80ac890 con 0x7f0cf815b540 2023-10-16T11:19:04.610 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:04.608+0000 7f34fb1c9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:19:04.656 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:04.655+0000 7faf26f8d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:19:04.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.709+0000 7f0cf6ffd700 1 -- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 msgr2=0x7f0ce406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:04.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.709+0000 7f0cf6ffd700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 0x7f0ce406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:04.725 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:04.724+0000 7f34fb1c9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:19:04.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.728+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0cf80ac890 con 0x7f0cf815b540 2023-10-16T11:19:04.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.729+0000 7f0cdf7fe700 1 -- 172.21.15.158:0/22810085 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v203) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0cf80ac890 con 0x7f0cf815b540 2023-10-16T11:19:04.731 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:19:04.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 msgr2=0x7f0ce406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:04.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 0x7f0ce406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf815b540 msgr2=0x7f0cf80b06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf815b540 0x7f0cf80b06c0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80ca9b0 tx=0x7f0cf000cd50 comp rx=0 tx=0).stop 2023-10-16T11:19:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 shutdown_connections 2023-10-16T11:19:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f0ce406b140 0x7f0ce406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf815b540 0x7f0cf80b06c0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf806afe0 0x7f0cf81f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 --2- 172.21.15.158:0/22810085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf806f180 0x7f0cf815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 >> 172.21.15.158:0/22810085 conn(0x7f0cf80d03b0 msgr2=0x7f0cf80d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:04.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 shutdown_connections 2023-10-16T11:19:04.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.731+0000 7f0cfd705700 1 -- 172.21.15.158:0/22810085 wait complete. 2023-10-16T11:19:04.748 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:19:04.779 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:04.778+0000 7faf26f8d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:19:04.802 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:04.798+0000 7f34fb1c9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:19:04.856 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:04.854+0000 7faf26f8d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:19:04.881 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:04.879+0000 7f34fb1c9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:19:04.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.873+0000 7faf9529c700 1 Processor -- start 2023-10-16T11:19:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.874+0000 7faf9529c700 1 -- start start 2023-10-16T11:19:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.874+0000 7faf9529c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf90159280 0x7faf9006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.874+0000 7faf9529c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf9015c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.874+0000 7faf9529c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf90159660 con 0x7faf900cbef0 2023-10-16T11:19:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.874+0000 7faf9529c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf900cb710 con 0x7faf90159280 2023-10-16T11:19:04.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.874+0000 7faf9529c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9006b120 con 0x7faf900d0e10 2023-10-16T11:19:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.874+0000 7faf8effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf90159280 0x7faf9006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf90159280 0x7faf9006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37246/0 (socket says 172.21.15.158:37246) 2023-10-16T11:19:04.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8effd700 1 -- 172.21.15.158:0/18747239 learned_addr learned my addr 172.21.15.158:0/18747239 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8e7fc700 1 --2- 172.21.15.158:0/18747239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf9015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8effd700 1 -- 172.21.15.158:0/18747239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 msgr2=0x7faf9015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8effd700 1 --2- 172.21.15.158:0/18747239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf9015c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8effd700 1 -- 172.21.15.158:0/18747239 >> v1:172.21.15.172:6789/0 conn(0x7faf900cbef0 legacy=0x7faf900cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8effd700 1 -- 172.21.15.158:0/18747239 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9006b400 con 0x7faf90159280 2023-10-16T11:19:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8dffb700 1 -- 172.21.15.158:0/18747239 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270299277 0 0) 0x7faf90159660 con 0x7faf900cbef0 2023-10-16T11:19:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8effd700 1 --2- 172.21.15.158:0/18747239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf9006a8d0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7faf840098f0 tx=0x7faf84009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e48b42dd559c94 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8dffb700 1 -- 172.21.15.158:0/18747239 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84011830 con 0x7faf90159280 2023-10-16T11:19:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8dffb700 1 -- 172.21.15.158:0/18747239 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf84011990 con 0x7faf90159280 2023-10-16T11:19:04.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.875+0000 7faf8dffb700 1 -- 172.21.15.158:0/18747239 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84011d20 con 0x7faf90159280 2023-10-16T11:19:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 -- 172.21.15.158:0/18747239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 msgr2=0x7faf9006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 --2- 172.21.15.158:0/18747239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf9006a8d0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7faf840098f0 tx=0x7faf84009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 -- 172.21.15.158:0/18747239 shutdown_connections 2023-10-16T11:19:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 --2- 172.21.15.158:0/18747239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf9006a8d0 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 --2- 172.21.15.158:0/18747239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf9015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 -- 172.21.15.158:0/18747239 >> 172.21.15.158:0/18747239 conn(0x7faf90099a10 msgr2=0x7faf900d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 -- 172.21.15.158:0/18747239 shutdown_connections 2023-10-16T11:19:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.876+0000 7faf9529c700 1 -- 172.21.15.158:0/18747239 wait complete. 2023-10-16T11:19:04.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.878+0000 7faf9529c700 1 Processor -- start 2023-10-16T11:19:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.878+0000 7faf9529c700 1 -- start start 2023-10-16T11:19:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.878+0000 7faf9529c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf900cbef0 0x7faf901577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.878+0000 7faf9529c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf900af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf9529c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf900af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf9529c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9015d3b0 con 0x7faf90159280 2023-10-16T11:19:04.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf9529c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf90157cf0 con 0x7faf900cbef0 2023-10-16T11:19:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf9529c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9015c730 con 0x7faf900d0e10 2023-10-16T11:19:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf900af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf900af790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37256/0 (socket says 172.21.15.158:37256) 2023-10-16T11:19:04.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 -- 172.21.15.158:0/866430861 learned_addr learned my addr 172.21.15.158:0/866430861 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8effd700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf900cbef0 0x7faf901577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8e7fc700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf900af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 -- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 msgr2=0x7faf900af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf900af250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 -- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf900cbef0 msgr2=0x7faf901577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf900cbef0 0x7faf901577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 -- 172.21.15.158:0/866430861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf84009550 con 0x7faf90159280 2023-10-16T11:19:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf8f7fe700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf900af790 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c7d0 tx=0x7faf8800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf777fe700 1 -- 172.21.15.158:0/866430861 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88003e30 con 0x7faf90159280 2023-10-16T11:19:04.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf777fe700 1 -- 172.21.15.158:0/866430861 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88005070 con 0x7faf90159280 2023-10-16T11:19:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf777fe700 1 -- 172.21.15.158:0/866430861 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88005420 con 0x7faf90159280 2023-10-16T11:19:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9015ca10 con 0x7faf90159280 2023-10-16T11:19:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.879+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9015cfd0 con 0x7faf90159280 2023-10-16T11:19:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.880+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9015d4f0 con 0x7faf90159280 2023-10-16T11:19:04.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.881+0000 7faf777fe700 1 -- 172.21.15.158:0/866430861 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7faf88005580 con 0x7faf90159280 2023-10-16T11:19:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.881+0000 7faf777fe700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 0x7faf7c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.881+0000 7faf8effd700 1 -- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 msgr2=0x7faf7c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.881+0000 7faf777fe700 1 -- 172.21.15.158:0/866430861 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6941+0+0 (secure 0 0 0) 0x7faf880893f0 con 0x7faf90159280 2023-10-16T11:19:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.881+0000 7faf8effd700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 0x7faf7c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:04.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:04.886+0000 7faf777fe700 1 -- 172.21.15.158:0/866430861 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf9015d4f0 con 0x7faf90159280 2023-10-16T11:19:04.942 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:04.940+0000 7faf26f8d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:19:05.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.082+0000 7faf8effd700 1 -- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 msgr2=0x7faf7c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:05.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.082+0000 7faf8effd700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 0x7faf7c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:05.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.082+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf9015d4f0 con 0x7faf90159280 2023-10-16T11:19:05.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.082+0000 7faf777fe700 1 -- 172.21.15.158:0/866430861 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v203) v1 ==== 104+0+11 (secure 0 0 0) 0x7faf88017020 con 0x7faf90159280 2023-10-16T11:19:05.085 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:19:05.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 msgr2=0x7faf7c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:05.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 0x7faf7c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 msgr2=0x7faf900af790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf900af790 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c7d0 tx=0x7faf8800cae0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 shutdown_connections 2023-10-16T11:19:05.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7faf7c06b140 0x7faf7c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf90159280 0x7faf900af790 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf900cbef0 0x7faf901577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 --2- 172.21.15.158:0/866430861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf900d0e10 0x7faf900af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 >> 172.21.15.158:0/866430861 conn(0x7faf90099a10 msgr2=0x7faf90158870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:05.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 shutdown_connections 2023-10-16T11:19:05.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.084+0000 7faf9529c700 1 -- 172.21.15.158:0/866430861 wait complete. 2023-10-16T11:19:05.098 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:19:05.252 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:05.250+0000 7f34fb1c9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:19:05.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.264+0000 7f54393d9700 1 Processor -- start 2023-10-16T11:19:05.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.264+0000 7f54393d9700 1 -- start start 2023-10-16T11:19:05.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.264+0000 7f54393d9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f543415b5d0 0x7f543406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.264+0000 7f54393d9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543415b9b0 0x7f543406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f54393d9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5434072be0 con 0x7f543406f180 2023-10-16T11:19:05.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f54393d9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5434160370 con 0x7f543415b5d0 2023-10-16T11:19:05.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f54393d9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543415f6f0 con 0x7f543415b9b0 2023-10-16T11:19:05.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5432ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f543415b5d0 0x7f543406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f54327fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f543406f180 0x7f543406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37472/0 (socket says 172.21.15.158:37472) 2023-10-16T11:19:05.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f54327fc700 1 -- 172.21.15.158:0/1067029815 learned_addr learned my addr 172.21.15.158:0/1067029815 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:05.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f54337fe700 1 --2- 172.21.15.158:0/1067029815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543415b9b0 0x7f543406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5432ffd700 1 -- 172.21.15.158:0/1067029815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543415b9b0 msgr2=0x7f543406ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:05.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5432ffd700 1 --2- 172.21.15.158:0/1067029815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543415b9b0 0x7f543406ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5432ffd700 1 -- 172.21.15.158:0/1067029815 >> v1:172.21.15.158:6789/0 conn(0x7f543406f180 legacy=0x7f543406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5432ffd700 1 -- 172.21.15.158:0/1067029815 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543415f9d0 con 0x7f543415b5d0 2023-10-16T11:19:05.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5431ffb700 1 -- 172.21.15.158:0/1067029815 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965727802 0 0) 0x7f5434072be0 con 0x7f543406f180 2023-10-16T11:19:05.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f54337fe700 1 --2- 172.21.15.158:0/1067029815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543415b9b0 0x7f543406ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:05.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5432ffd700 1 --2- 172.21.15.158:0/1067029815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b5d0 0x7f543406a8d0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f5424009840 tx=0x7f5424009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7965aca79ebfa555 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:05.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5431ffb700 1 -- 172.21.15.158:0/1067029815 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424011830 con 0x7f543415b5d0 2023-10-16T11:19:05.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5431ffb700 1 -- 172.21.15.158:0/1067029815 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5424011990 con 0x7f543415b5d0 2023-10-16T11:19:05.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.265+0000 7f5431ffb700 1 -- 172.21.15.158:0/1067029815 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424011d20 con 0x7f543415b5d0 2023-10-16T11:19:05.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 -- 172.21.15.158:0/1067029815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b5d0 msgr2=0x7f543406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 --2- 172.21.15.158:0/1067029815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b5d0 0x7f543406a8d0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f5424009840 tx=0x7f5424009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:05.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 -- 172.21.15.158:0/1067029815 shutdown_connections 2023-10-16T11:19:05.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 --2- 172.21.15.158:0/1067029815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b5d0 0x7f543406a8d0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 --2- 172.21.15.158:0/1067029815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543415b9b0 0x7f543406ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 -- 172.21.15.158:0/1067029815 >> 172.21.15.158:0/1067029815 conn(0x7f54340d03b0 msgr2=0x7f54340d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:05.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 -- 172.21.15.158:0/1067029815 shutdown_connections 2023-10-16T11:19:05.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.266+0000 7f54393d9700 1 -- 172.21.15.158:0/1067029815 wait complete. 2023-10-16T11:19:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.267+0000 7f54393d9700 1 Processor -- start 2023-10-16T11:19:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.267+0000 7f54393d9700 1 -- start start 2023-10-16T11:19:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54393d9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543406f180 0x7f543415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54393d9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f543415b5d0 0x7f543415e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54393d9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b9b0 0x7f5434157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54393d9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543415ed20 con 0x7f543415b9b0 2023-10-16T11:19:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54393d9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543415d590 con 0x7f543415b5d0 2023-10-16T11:19:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54393d9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543415d870 con 0x7f543406f180 2023-10-16T11:19:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b9b0 0x7f5434157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b9b0 0x7f5434157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37306/0 (socket says 172.21.15.158:37306) 2023-10-16T11:19:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54337fe700 1 -- 172.21.15.158:0/3387364647 learned_addr learned my addr 172.21.15.158:0/3387364647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f5432ffd700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543406f180 0x7f543415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.268+0000 7f54327fc700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f543415b5d0 0x7f543415e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54337fe700 1 -- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543406f180 msgr2=0x7f543415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54337fe700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543406f180 0x7f543415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54337fe700 1 -- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f543415b5d0 msgr2=0x7f543415e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54337fe700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f543415b5d0 0x7f543415e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54337fe700 1 -- 172.21.15.158:0/3387364647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240094a0 con 0x7f543415b9b0 2023-10-16T11:19:05.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54337fe700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b9b0 0x7f5434157c50 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f542c007df0 tx=0x7f542c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f541b7fe700 1 -- 172.21.15.158:0/3387364647 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c00dd20 con 0x7f543415b9b0 2023-10-16T11:19:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f541b7fe700 1 -- 172.21.15.158:0/3387364647 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542c00f040 con 0x7f543415b9b0 2023-10-16T11:19:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543415db50 con 0x7f543415b9b0 2023-10-16T11:19:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f541b7fe700 1 -- 172.21.15.158:0/3387364647 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c003c60 con 0x7f543415b9b0 2023-10-16T11:19:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.269+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54340c5080 con 0x7f543415b9b0 2023-10-16T11:19:05.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.270+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54340c4ee0 con 0x7f543415b9b0 2023-10-16T11:19:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.271+0000 7f541b7fe700 1 -- 172.21.15.158:0/3387364647 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f542c004030 con 0x7f543415b9b0 2023-10-16T11:19:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.271+0000 7f541b7fe700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 0x7f542006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.271+0000 7f5432ffd700 1 -- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 msgr2=0x7f542006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.271+0000 7f541b7fe700 1 -- 172.21.15.158:0/3387364647 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6941+0+0 (secure 0 0 0) 0x7f542c089f90 con 0x7f543415b9b0 2023-10-16T11:19:05.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.271+0000 7f5432ffd700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 0x7f542006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.276+0000 7f541b7fe700 1 -- 172.21.15.158:0/3387364647 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f542c0bcfe0 con 0x7f543415b9b0 2023-10-16T11:19:05.334 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:05.332+0000 7faf26f8d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:19:05.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.470+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f54340c52a0 con 0x7f543415b9b0 2023-10-16T11:19:05.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.471+0000 7f541b7fe700 1 -- 172.21.15.158:0/3387364647 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v203) v1 ==== 100+0+11 (secure 0 0 0) 0x7f542c01a0a0 con 0x7f543415b9b0 2023-10-16T11:19:05.473 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:19:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.472+0000 7f5432ffd700 1 -- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 msgr2=0x7f542006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.472+0000 7f5432ffd700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 0x7f542006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 msgr2=0x7f542006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 0x7f542006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b9b0 msgr2=0x7f5434157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b9b0 0x7f5434157c50 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f542c007df0 tx=0x7f542c00d3b0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 shutdown_connections 2023-10-16T11:19:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f542006b140 0x7f542006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f543415b9b0 0x7f5434157c50 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f543406f180 0x7f543415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 --2- 172.21.15.158:0/3387364647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f543415b5d0 0x7f543415e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 >> 172.21.15.158:0/3387364647 conn(0x7f54340d03b0 msgr2=0x7f54340b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 shutdown_connections 2023-10-16T11:19:05.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.473+0000 7f54393d9700 1 -- 172.21.15.158:0/3387364647 wait complete. 2023-10-16T11:19:05.488 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:19:05.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.609+0000 7fd5464a9700 1 Processor -- start 2023-10-16T11:19:05.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.609+0000 7fd5464a9700 1 -- start start 2023-10-16T11:19:05.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.609+0000 7fd5464a9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5400b4320 0x7fd5400b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.609+0000 7fd5464a9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd5400b47d0 0x7fd5400b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.609+0000 7fd5464a9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5400cf550 con 0x7fd54015b650 2023-10-16T11:19:05.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.609+0000 7fd5464a9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5400cc6d0 con 0x7fd5400b4320 2023-10-16T11:19:05.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.609+0000 7fd5464a9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5400cf2b0 con 0x7fd5400b47d0 2023-10-16T11:19:05.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5400b4320 0x7fd5400b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5400b4320 0x7fd5400b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37314/0 (socket says 172.21.15.158:37314) 2023-10-16T11:19:05.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 -- 172.21.15.158:0/1563584455 learned_addr learned my addr 172.21.15.158:0/1563584455 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:05.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd544a46700 1 --2- 172.21.15.158:0/1563584455 >> v2:172.21.15.172:3300/0 conn(0x7fd5400b47d0 0x7fd5400b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 -- 172.21.15.158:0/1563584455 >> v2:172.21.15.172:3300/0 conn(0x7fd5400b47d0 msgr2=0x7fd5400b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:05.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 --2- 172.21.15.158:0/1563584455 >> v2:172.21.15.172:3300/0 conn(0x7fd5400b47d0 0x7fd5400b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 -- 172.21.15.158:0/1563584455 >> v1:172.21.15.172:6789/0 conn(0x7fd54015b650 legacy=0x7fd54015fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 -- 172.21.15.158:0/1563584455 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54015ba30 con 0x7fd5400b4320 2023-10-16T11:19:05.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53f7fe700 1 --2- 172.21.15.158:0/1563584455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5400b2f50 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fd5400722f0 tx=0x7fd534009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb3dfc79783cff9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:05.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53effd700 1 -- 172.21.15.158:0/1563584455 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534011830 con 0x7fd5400b4320 2023-10-16T11:19:05.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53effd700 1 -- 172.21.15.158:0/1563584455 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534011990 con 0x7fd5400b4320 2023-10-16T11:19:05.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.610+0000 7fd53effd700 1 -- 172.21.15.158:0/1563584455 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534011d20 con 0x7fd5400b4320 2023-10-16T11:19:05.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 -- 172.21.15.158:0/1563584455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 msgr2=0x7fd5400b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 --2- 172.21.15.158:0/1563584455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5400b2f50 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fd5400722f0 tx=0x7fd534009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 -- 172.21.15.158:0/1563584455 shutdown_connections 2023-10-16T11:19:05.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 --2- 172.21.15.158:0/1563584455 >> v2:172.21.15.172:3300/0 conn(0x7fd5400b47d0 0x7fd5400b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 --2- 172.21.15.158:0/1563584455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5400b2f50 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 -- 172.21.15.158:0/1563584455 >> 172.21.15.158:0/1563584455 conn(0x7fd5400cdf50 msgr2=0x7fd5400ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:05.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 -- 172.21.15.158:0/1563584455 shutdown_connections 2023-10-16T11:19:05.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.611+0000 7fd5464a9700 1 -- 172.21.15.158:0/1563584455 wait complete. 2023-10-16T11:19:05.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.612+0000 7fd5464a9700 1 Processor -- start 2023-10-16T11:19:05.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.612+0000 7fd5464a9700 1 -- start start 2023-10-16T11:19:05.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd5464a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5401e6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd5464a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400b47d0 0x7fd5400c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd5464a9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54015b650 0x7fd5401e4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd5464a9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400c5000 con 0x7fd5400b4320 2023-10-16T11:19:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd5464a9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400c52c0 con 0x7fd54015b650 2023-10-16T11:19:05.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd5464a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400c55a0 con 0x7fd5400b47d0 2023-10-16T11:19:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd53ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5401e6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd53ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5401e6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37318/0 (socket says 172.21.15.158:37318) 2023-10-16T11:19:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd53ffff700 1 -- 172.21.15.158:0/689226131 learned_addr learned my addr 172.21.15.158:0/689226131 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd53f7fe700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400b47d0 0x7fd5400c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.613+0000 7fd544a46700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54015b650 0x7fd5401e4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53f7fe700 1 -- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54015b650 msgr2=0x7fd5401e4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53f7fe700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54015b650 0x7fd5401e4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53f7fe700 1 -- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 msgr2=0x7fd5401e6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53f7fe700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5401e6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53f7fe700 1 -- 172.21.15.158:0/689226131 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534009550 con 0x7fd5400b47d0 2023-10-16T11:19:05.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53ffff700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5401e6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:05.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53f7fe700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400b47d0 0x7fd5400c4730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd534016000 tx=0x7fd534004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:05.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53cff9700 1 -- 172.21.15.158:0/689226131 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534004380 con 0x7fd5400b47d0 2023-10-16T11:19:05.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53cff9700 1 -- 172.21.15.158:0/689226131 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53401f070 con 0x7fd5400b47d0 2023-10-16T11:19:05.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd53cff9700 1 -- 172.21.15.158:0/689226131 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5340036a0 con 0x7fd5400b47d0 2023-10-16T11:19:05.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5400c5880 con 0x7fd5400b47d0 2023-10-16T11:19:05.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.614+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd540156ab0 con 0x7fd5400b47d0 2023-10-16T11:19:05.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.615+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5400681b0 con 0x7fd5400b47d0 2023-10-16T11:19:05.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.616+0000 7fd53cff9700 1 -- 172.21.15.158:0/689226131 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd534021030 con 0x7fd5400b47d0 2023-10-16T11:19:05.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.616+0000 7fd53cff9700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 0x7fd52c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:05.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.616+0000 7fd53ffff700 1 -- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 msgr2=0x7fd52c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:05.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.616+0000 7fd53cff9700 1 -- 172.21.15.158:0/689226131 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6941+0+0 (secure 0 0 0) 0x7fd53408b150 con 0x7fd5400b47d0 2023-10-16T11:19:05.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.616+0000 7fd53ffff700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 0x7fd52c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:05.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.624+0000 7fd53cff9700 1 -- 172.21.15.158:0/689226131 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd534055ab0 con 0x7fd5400b47d0 2023-10-16T11:19:05.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.817+0000 7fd53ffff700 1 -- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 msgr2=0x7fd52c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:05.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.817+0000 7fd53ffff700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 0x7fd52c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:05.826 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:05.824+0000 7f34fb1c9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:19:05.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.888+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd54015f250 con 0x7fd5400b47d0 2023-10-16T11:19:05.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.889+0000 7fd53cff9700 1 -- 172.21.15.158:0/689226131 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v204) v1 ==== 80+0+476 (secure 0 0 0) 0x7fd5340590d0 con 0x7fd5400b47d0 2023-10-16T11:19:05.891 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2415272176 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.891 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/3585499115 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.891 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/3585499115 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.891 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/778612447 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.891 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1354527416 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.892 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3047051699 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.892 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/4256533896 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.892 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/716693565 2023-10-17T11:17:53.925811+0000 2023-10-16T11:19:05.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 msgr2=0x7fd52c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:05.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 0x7fd52c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400b47d0 msgr2=0x7fd5400c4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:05.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400b47d0 0x7fd5400c4730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd534016000 tx=0x7fd534004750 comp rx=0 tx=0).stop 2023-10-16T11:19:05.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 shutdown_connections 2023-10-16T11:19:05.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fd52c06b140 0x7fd52c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54015b650 0x7fd5401e4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5400b4320 0x7fd5401e6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 --2- 172.21.15.158:0/689226131 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400b47d0 0x7fd5400c4730 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:05.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 >> 172.21.15.158:0/689226131 conn(0x7fd5400cdf50 msgr2=0x7fd54015e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:05.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 shutdown_connections 2023-10-16T11:19:05.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:05.891+0000 7fd5464a9700 1 -- 172.21.15.158:0/689226131 wait complete. 2023-10-16T11:19:05.897 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:05.894+0000 7f34fb1c9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:19:05.897 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T11:19:05.906 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:19:05.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:05.907 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:19:05.940 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:05.938+0000 7faf26f8d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:19:05.950 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:19:05.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:05.951 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:19:06.010 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:19:06.010 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:06.017 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:06.016+0000 7faf26f8d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:19:06.028 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:19:06.028 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:06.028 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:06.046 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:19:06.046 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:19:06.046 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:06.084 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:19:06.085 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:06.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.205+0000 7f440fa95700 1 Processor -- start 2023-10-16T11:19:06.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.206+0000 7f440fa95700 1 -- start start 2023-10-16T11:19:06.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440fa95700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44080b7810 0x7f44081683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440fa95700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44080d3bd0 con 0x7f44080d1210 2023-10-16T11:19:06.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440fa95700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44080d2fc0 con 0x7f440815d7c0 2023-10-16T11:19:06.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440fa95700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44080d4060 con 0x7f44080b7810 2023-10-16T11:19:06.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440d030700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f44080d1210 0x7f44080ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37506/0 (socket says 172.21.15.158:37506) 2023-10-16T11:19:06.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440d030700 1 -- 172.21.15.158:0/336662025 learned_addr learned my addr 172.21.15.158:0/336662025 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:06.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440e032700 1 --2- 172.21.15.158:0/336662025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44080b7810 0x7f44081683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585570957 0 0) 0x7f44080d3bd0 con 0x7f44080d1210 2023-10-16T11:19:06.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f40035b0 con 0x7f44080d1210 2023-10-16T11:19:06.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969508916 0 0) 0x7f44080d2fc0 con 0x7f440815d7c0 2023-10-16T11:19:06.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.207+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44080d3bd0 con 0x7f440815d7c0 2023-10-16T11:19:06.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440e032700 1 -- 172.21.15.158:0/336662025 >> v1:172.21.15.172:6789/0 conn(0x7f440815d7c0 legacy=0x7f44080d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173727692 0 0) 0x7f44080d3bd0 con 0x7f440815d7c0 2023-10-16T11:19:06.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440e032700 1 -- 172.21.15.158:0/336662025 >> v1:172.21.15.158:6789/0 conn(0x7f44080d1210 legacy=0x7f44080ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440e032700 1 -- 172.21.15.158:0/336662025 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44080d32a0 con 0x7f44080b7810 2023-10-16T11:19:06.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440e032700 1 --2- 172.21.15.158:0/336662025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44080b7810 0x7f44081683f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4400009dc0 tx=0x7f440000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e094a49bfb36dff2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:06.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440000e070 con 0x7f44080b7810 2023-10-16T11:19:06.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4400003a70 con 0x7f44080b7810 2023-10-16T11:19:06.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.208+0000 7f440c82f700 1 -- 172.21.15.158:0/336662025 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f80015f0 con 0x7f44080b7810 2023-10-16T11:19:06.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.209+0000 7f440fa95700 1 -- 172.21.15.158:0/336662025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44080b7810 msgr2=0x7f44081683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.209+0000 7f440fa95700 1 --2- 172.21.15.158:0/336662025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44080b7810 0x7f44081683f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4400009dc0 tx=0x7f440000b330 comp rx=0 tx=0).stop 2023-10-16T11:19:06.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.209+0000 7f440fa95700 1 -- 172.21.15.158:0/336662025 shutdown_connections 2023-10-16T11:19:06.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.209+0000 7f440fa95700 1 --2- 172.21.15.158:0/336662025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44080b7810 0x7f44081683f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.209+0000 7f440fa95700 1 -- 172.21.15.158:0/336662025 >> 172.21.15.158:0/336662025 conn(0x7f44080669f0 msgr2=0x7f4408066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:06.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.209+0000 7f440fa95700 1 -- 172.21.15.158:0/336662025 shutdown_connections 2023-10-16T11:19:06.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.209+0000 7f440fa95700 1 -- 172.21.15.158:0/336662025 wait complete. 2023-10-16T11:19:06.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.211+0000 7f440fa95700 1 Processor -- start 2023-10-16T11:19:06.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.211+0000 7f440fa95700 1 -- start start 2023-10-16T11:19:06.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440fa95700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44080b7810 0x7f44080b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440fa95700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44080d1210 0x7f44081f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440fa95700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f440815d7c0 0x7f4408152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440fa95700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44080b5330 con 0x7f44080b7810 2023-10-16T11:19:06.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440fa95700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44080b5610 con 0x7f44080d1210 2023-10-16T11:19:06.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440fa95700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440815a3a0 con 0x7f440815d7c0 2023-10-16T11:19:06.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440d831700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44080b7810 0x7f44080b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440d831700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44080b7810 0x7f44080b4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37334/0 (socket says 172.21.15.158:37334) 2023-10-16T11:19:06.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440d831700 1 -- 172.21.15.158:0/1727077709 learned_addr learned my addr 172.21.15.158:0/1727077709 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:06.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440e032700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f440815d7c0 0x7f4408152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.212+0000 7f440d030700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44080d1210 0x7f44081f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440d831700 1 -- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f440815d7c0 msgr2=0x7f4408152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440d831700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f440815d7c0 0x7f4408152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440d831700 1 -- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44080d1210 msgr2=0x7f44081f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440d831700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44080d1210 0x7f44081f67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440d831700 1 -- 172.21.15.158:0/1727077709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400009a70 con 0x7f44080b7810 2023-10-16T11:19:06.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440d831700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44080b7810 0x7f44080b4a30 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f440400c810 tx=0x7f440400cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:06.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f43fe7fc700 1 -- 172.21.15.158:0/1727077709 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404005480 con 0x7f44080b7810 2023-10-16T11:19:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f43fe7fc700 1 -- 172.21.15.158:0/1727077709 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404003680 con 0x7f44080b7810 2023-10-16T11:19:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f43fe7fc700 1 -- 172.21.15.158:0/1727077709 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404003a30 con 0x7f44080b7810 2023-10-16T11:19:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440815a680 con 0x7f44080b7810 2023-10-16T11:19:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.213+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44080b4f70 con 0x7f44080b7810 2023-10-16T11:19:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.214+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440815a200 con 0x7f44080b7810 2023-10-16T11:19:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.214+0000 7f43fe7fc700 1 -- 172.21.15.158:0/1727077709 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f4404003b90 con 0x7f44080b7810 2023-10-16T11:19:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.215+0000 7f43fe7fc700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 0x7f43ec06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.215+0000 7f440d030700 1 -- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 msgr2=0x7f43ec06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.215+0000 7f43fe7fc700 1 -- 172.21.15.158:0/1727077709 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f4404089490 con 0x7f44080b7810 2023-10-16T11:19:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.215+0000 7f440d030700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 0x7f43ec06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:06.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.222+0000 7f43fe7fc700 1 -- 172.21.15.158:0/1727077709 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f440815a200 con 0x7f44080b7810 2023-10-16T11:19:06.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.416+0000 7f440d030700 1 -- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 msgr2=0x7f43ec06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:06.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.416+0000 7f440d030700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 0x7f43ec06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:06.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.466+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f440815a200 con 0x7f44080b7810 2023-10-16T11:19:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.466+0000 7f43fe7fc700 1 -- 172.21.15.158:0/1727077709 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f440815a200 con 0x7f44080b7810 2023-10-16T11:19:06.468 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:06.468 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":103,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2912297853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2912297853},{"type":"v1","addr":"172.21.15.158:6836","nonce":2912297853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":38441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2297507228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2297507228},{"type":"v1","addr":"172.21.15.158:6837","nonce":2297507228}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38447,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3795197457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3795197457},{"type":"v1","addr":"172.21.15.172:6834","nonce":3795197457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:17:40.072523+0000","modified":"2023-10-16T11:17:43.106178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38450":{"gid":38450,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1279760034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1279760034},{"type":"v1","addr":"172.21.15.172:6835","nonce":1279760034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T11:19:06.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 msgr2=0x7f43ec06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:06.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 0x7f43ec06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44080b7810 msgr2=0x7f44080b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44080b7810 0x7f44080b4a30 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f440400c810 tx=0x7f440400cb20 comp rx=0 tx=0).stop 2023-10-16T11:19:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 shutdown_connections 2023-10-16T11:19:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f43ec06b140 0x7f43ec06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f440815d7c0 0x7f4408152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44080b7810 0x7f44080b4a30 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 --2- 172.21.15.158:0/1727077709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44080d1210 0x7f44081f67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 >> 172.21.15.158:0/1727077709 conn(0x7f44080669f0 msgr2=0x7f4408066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 shutdown_connections 2023-10-16T11:19:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.469+0000 7f440fa95700 1 -- 172.21.15.158:0/1727077709 wait complete. 2023-10-16T11:19:06.474 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 103 2023-10-16T11:19:06.483 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:06.543 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:06.542+0000 7f34fb1c9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:06.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.607+0000 7f41850a0700 1 Processor -- start 2023-10-16T11:19:06.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.607+0000 7f41850a0700 1 -- start start 2023-10-16T11:19:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f41850a0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f41800cc1b0 0x7f41800b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f41850a0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f418015f0f0 0x7f41800cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f41850a0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418015b930 con 0x7f41800ce530 2023-10-16T11:19:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f41850a0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41800cf020 con 0x7f418015f0f0 2023-10-16T11:19:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f41850a0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418015f5f0 con 0x7f41800cc1b0 2023-10-16T11:19:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f417f59e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f418015f0f0 0x7f41800cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f417ed9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f41800ce530 0x7f41800cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37520/0 (socket says 172.21.15.158:37520) 2023-10-16T11:19:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f417ed9d700 1 -- 172.21.15.158:0/1778656861 learned_addr learned my addr 172.21.15.158:0/1778656861 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f417f59e700 1 --2- 172.21.15.158:0/1778656861 >> v2:172.21.15.158:3300/0 conn(0x7f418015f0f0 0x7f41800cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37348/0 (socket says 172.21.15.158:37348) 2023-10-16T11:19:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.608+0000 7f417e59c700 1 --2- 172.21.15.158:0/1778656861 >> v2:172.21.15.172:3300/0 conn(0x7f41800cc1b0 0x7f41800b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417f59e700 1 -- 172.21.15.158:0/1778656861 >> v2:172.21.15.172:3300/0 conn(0x7f41800cc1b0 msgr2=0x7f41800b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417f59e700 1 --2- 172.21.15.158:0/1778656861 >> v2:172.21.15.172:3300/0 conn(0x7f41800cc1b0 0x7f41800b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417f59e700 1 -- 172.21.15.158:0/1778656861 >> v1:172.21.15.158:6789/0 conn(0x7f41800ce530 legacy=0x7f41800cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417f59e700 1 -- 172.21.15.158:0/1778656861 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418015f8d0 con 0x7f418015f0f0 2023-10-16T11:19:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417dd9b700 1 -- 172.21.15.158:0/1778656861 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054772418 0 0) 0x7f418015b930 con 0x7f41800ce530 2023-10-16T11:19:06.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417f59e700 1 --2- 172.21.15.158:0/1778656861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418015f0f0 0x7f41800cf470 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f41800cef50 tx=0x7f417800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e599347dc4434a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:06.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417dd9b700 1 -- 172.21.15.158:0/1778656861 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417800e070 con 0x7f418015f0f0 2023-10-16T11:19:06.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417dd9b700 1 -- 172.21.15.158:0/1778656861 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417800bb20 con 0x7f418015f0f0 2023-10-16T11:19:06.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f417dd9b700 1 -- 172.21.15.158:0/1778656861 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178012430 con 0x7f418015f0f0 2023-10-16T11:19:06.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f41850a0700 1 -- 172.21.15.158:0/1778656861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418015f0f0 msgr2=0x7f41800cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f41850a0700 1 --2- 172.21.15.158:0/1778656861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418015f0f0 0x7f41800cf470 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f41800cef50 tx=0x7f417800b330 comp rx=0 tx=0).stop 2023-10-16T11:19:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f41850a0700 1 -- 172.21.15.158:0/1778656861 shutdown_connections 2023-10-16T11:19:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f41850a0700 1 --2- 172.21.15.158:0/1778656861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418015f0f0 0x7f41800cf470 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f41850a0700 1 --2- 172.21.15.158:0/1778656861 >> v2:172.21.15.172:3300/0 conn(0x7f41800cc1b0 0x7f41800b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.609+0000 7f41850a0700 1 -- 172.21.15.158:0/1778656861 >> 172.21.15.158:0/1778656861 conn(0x7f418006a8d0 msgr2=0x7f418006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.610+0000 7f41850a0700 1 -- 172.21.15.158:0/1778656861 shutdown_connections 2023-10-16T11:19:06.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.610+0000 7f41850a0700 1 -- 172.21.15.158:0/1778656861 wait complete. 2023-10-16T11:19:06.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.611+0000 7f41850a0700 1 Processor -- start 2023-10-16T11:19:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.611+0000 7f41850a0700 1 -- start start 2023-10-16T11:19:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.612+0000 7f41850a0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41800cc1b0 0x7f4180157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.612+0000 7f41850a0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41800ce530 0x7f41800c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.612+0000 7f41850a0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f418015f0f0 0x7f41800b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.612+0000 7f41850a0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418006c520 con 0x7f41800cc1b0 2023-10-16T11:19:06.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.612+0000 7f41850a0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418006c800 con 0x7f41800ce530 2023-10-16T11:19:06.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.612+0000 7f41850a0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418006cae0 con 0x7f418015f0f0 2023-10-16T11:19:06.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41800cc1b0 0x7f4180157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41800cc1b0 0x7f4180157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37350/0 (socket says 172.21.15.158:37350) 2023-10-16T11:19:06.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 -- 172.21.15.158:0/41018637 learned_addr learned my addr 172.21.15.158:0/41018637 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:06.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417e59c700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41800ce530 0x7f41800c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417f59e700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f418015f0f0 0x7f41800b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 -- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f418015f0f0 msgr2=0x7f41800b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f418015f0f0 0x7f41800b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 -- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41800ce530 msgr2=0x7f41800c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41800ce530 0x7f41800c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 -- 172.21.15.158:0/41018637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178009a70 con 0x7f41800cc1b0 2023-10-16T11:19:06.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417e59c700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41800ce530 0x7f41800c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:06.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f417ed9d700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41800cc1b0 0x7f4180157a20 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f41700011e0 tx=0x7f417000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:06.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f41677fe700 1 -- 172.21.15.158:0/41018637 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170004c80 con 0x7f41800cc1b0 2023-10-16T11:19:06.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f41677fe700 1 -- 172.21.15.158:0/41018637 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4170004e00 con 0x7f41800cc1b0 2023-10-16T11:19:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418006cdc0 con 0x7f41800cc1b0 2023-10-16T11:19:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f41677fe700 1 -- 172.21.15.158:0/41018637 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170007310 con 0x7f41800cc1b0 2023-10-16T11:19:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.613+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418006d2e0 con 0x7f41800cc1b0 2023-10-16T11:19:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.614+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418006c660 con 0x7f41800cc1b0 2023-10-16T11:19:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.615+0000 7f41677fe700 1 -- 172.21.15.158:0/41018637 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f4170007470 con 0x7f41800cc1b0 2023-10-16T11:19:06.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.616+0000 7f41677fe700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 0x7f416c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.616+0000 7f417e59c700 1 -- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 msgr2=0x7f416c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:06.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.616+0000 7f41677fe700 1 -- 172.21.15.158:0/41018637 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f417008a0a0 con 0x7f41800cc1b0 2023-10-16T11:19:06.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.616+0000 7f417e59c700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 0x7f416c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:06.626 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:06.620+0000 7f34fb1c9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:19:06.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.624+0000 7f41677fe700 1 -- 172.21.15.158:0/41018637 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f41700548b0 con 0x7f41800cc1b0 2023-10-16T11:19:06.686 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:06.685+0000 7faf26f8d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:06.767 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:06.766+0000 7faf26f8d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:19:06.768 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:06.766+0000 7f34fb1c9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.817+0000 7f417e59c700 1 -- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 msgr2=0x7f416c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.817+0000 7f417e59c700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 0x7f416c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.837+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f418006c660 con 0x7f41800cc1b0 2023-10-16T11:19:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.838+0000 7f41677fe700 1 -- 172.21.15.158:0/41018637 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+18253 (secure 0 0 0) 0x7f418006c660 con 0x7f41800cc1b0 2023-10-16T11:19:06.840 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:06.841 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":205,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:06.802082+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:17:38.499055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T11:17:39.035057+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":205,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.1b","mappings":[{"from":3,"to":0}]},{"pgid":"26.29","mappings":[{"from":1,"to":6}]},{"pgid":"26.3e","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2415272176":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6833/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6832/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/778612447":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/1354527416":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/3047051699":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/4256533896":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/716693565":"2023-10-17T11:17:53.925811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.840+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 msgr2=0x7f416c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 0x7f416c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41800cc1b0 msgr2=0x7f4180157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41800cc1b0 0x7f4180157a20 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f41700011e0 tx=0x7f417000ca30 comp rx=0 tx=0).stop 2023-10-16T11:19:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 shutdown_connections 2023-10-16T11:19:06.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f416c06b140 0x7f416c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f418015f0f0 0x7f41800b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41800cc1b0 0x7f4180157a20 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 --2- 172.21.15.158:0/41018637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41800ce530 0x7f41800c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 >> 172.21.15.158:0/41018637 conn(0x7f418006a8d0 msgr2=0x7f418006f320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 shutdown_connections 2023-10-16T11:19:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.841+0000 7f41850a0700 1 -- 172.21.15.158:0/41018637 wait complete. 2023-10-16T11:19:06.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:17:38.499055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:17:39.035057+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 205, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:19:06.856 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:19:06.929 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:06.927+0000 7faf26f8d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:06.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.985+0000 7f98d02ed700 1 Processor -- start 2023-10-16T11:19:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.985+0000 7f98d02ed700 1 -- start start 2023-10-16T11:19:06.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98d02ed700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98c8160d30 0x7f98c80b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98d02ed700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98c80d0530 0x7f98c80d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98d02ed700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d19a0 0x7f98c80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98d02ed700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c80b8910 con 0x7f98c80d0530 2023-10-16T11:19:06.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98d02ed700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c80d0ef0 con 0x7f98c8160d30 2023-10-16T11:19:06.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98d02ed700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80b7c90 con 0x7f98c80d19a0 2023-10-16T11:19:06.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98c80d0530 0x7f98c80d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98c80d0530 0x7f98c80d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37388/0 (socket says 172.21.15.158:37388) 2023-10-16T11:19:06.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 -- 172.21.15.158:0/662146167 learned_addr learned my addr 172.21.15.158:0/662146167 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:06.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98ce88a700 1 --2- 172.21.15.158:0/662146167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d19a0 0x7f98c80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98ce089700 1 --2- 172.21.15.158:0/662146167 >> v2:172.21.15.172:3300/0 conn(0x7f98c8160d30 0x7f98c80b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:06.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 -- 172.21.15.158:0/662146167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d19a0 msgr2=0x7f98c80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:06.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 --2- 172.21.15.158:0/662146167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d19a0 0x7f98c80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 -- 172.21.15.158:0/662146167 >> v2:172.21.15.172:3300/0 conn(0x7f98c8160d30 msgr2=0x7f98c80b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:06.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 --2- 172.21.15.158:0/662146167 >> v2:172.21.15.172:3300/0 conn(0x7f98c8160d30 0x7f98c80b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98cd888700 1 -- 172.21.15.158:0/662146167 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c80b7f70 con 0x7f98c80d0530 2023-10-16T11:19:06.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98ce089700 1 --2- 172.21.15.158:0/662146167 >> v2:172.21.15.172:3300/0 conn(0x7f98c8160d30 0x7f98c80b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.986+0000 7f98ce88a700 1 --2- 172.21.15.158:0/662146167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d19a0 0x7f98c80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98cd888700 1 --2- 172.21.15.158:0/662146167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80d0530 0x7f98c80d09b0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f98c80d4690 tx=0x7f98b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168fbfd914b9d885 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98cd087700 1 -- 172.21.15.158:0/662146167 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8011830 con 0x7f98c80d0530 2023-10-16T11:19:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98cd087700 1 -- 172.21.15.158:0/662146167 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b8011990 con 0x7f98c80d0530 2023-10-16T11:19:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98cd087700 1 -- 172.21.15.158:0/662146167 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8011d20 con 0x7f98c80d0530 2023-10-16T11:19:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98d02ed700 1 -- 172.21.15.158:0/662146167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80d0530 msgr2=0x7f98c80d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98d02ed700 1 --2- 172.21.15.158:0/662146167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80d0530 0x7f98c80d09b0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f98c80d4690 tx=0x7f98b8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98d02ed700 1 -- 172.21.15.158:0/662146167 shutdown_connections 2023-10-16T11:19:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98d02ed700 1 --2- 172.21.15.158:0/662146167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d19a0 0x7f98c80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98d02ed700 1 --2- 172.21.15.158:0/662146167 >> v2:172.21.15.172:3300/0 conn(0x7f98c8160d30 0x7f98c80b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98d02ed700 1 --2- 172.21.15.158:0/662146167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c80d0530 0x7f98c80d09b0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.987+0000 7f98d02ed700 1 -- 172.21.15.158:0/662146167 >> 172.21.15.158:0/662146167 conn(0x7f98c80669f0 msgr2=0x7f98c8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.988+0000 7f98d02ed700 1 -- 172.21.15.158:0/662146167 shutdown_connections 2023-10-16T11:19:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.988+0000 7f98d02ed700 1 -- 172.21.15.158:0/662146167 wait complete. 2023-10-16T11:19:06.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 Processor -- start 2023-10-16T11:19:06.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 -- start start 2023-10-16T11:19:06.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d0530 0x7f98c8158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c80d19a0 0x7f98c80c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c8160d30 0x7f98c81f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:06.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f4ab0 con 0x7f98c8160d30 2023-10-16T11:19:06.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81522f0 con 0x7f98c80d19a0 2023-10-16T11:19:06.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.989+0000 7f98d02ed700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f3e30 con 0x7f98c80d0530 2023-10-16T11:19:07.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c8160d30 0x7f98c81f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c8160d30 0x7f98c81f4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37396/0 (socket says 172.21.15.158:37396) 2023-10-16T11:19:07.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 -- 172.21.15.158:0/2363597718 learned_addr learned my addr 172.21.15.158:0/2363597718 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:07.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce089700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d0530 0x7f98c8158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98cd888700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c80d19a0 0x7f98c80c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 -- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d0530 msgr2=0x7f98c8158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d0530 0x7f98c8158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 -- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c80d19a0 msgr2=0x7f98c80c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c80d19a0 0x7f98c80c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 -- 172.21.15.158:0/2363597718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0009a70 con 0x7f98c8160d30 2023-10-16T11:19:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98ce88a700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c8160d30 0x7f98c81f4ed0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f98c000ef00 tx=0x7f98c000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98beffd700 1 -- 172.21.15.158:0/2363597718 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0010040 con 0x7f98c8160d30 2023-10-16T11:19:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98beffd700 1 -- 172.21.15.158:0/2363597718 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000ba80 con 0x7f98c8160d30 2023-10-16T11:19:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c40094a0 con 0x7f98c8160d30 2023-10-16T11:19:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98beffd700 1 -- 172.21.15.158:0/2363597718 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c000be10 con 0x7f98c8160d30 2023-10-16T11:19:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.990+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b8009550 con 0x7f98c8160d30 2023-10-16T11:19:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.991+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c81f4bf0 con 0x7f98c8160d30 2023-10-16T11:19:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.992+0000 7f98beffd700 1 -- 172.21.15.158:0/2363597718 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f98c001d410 con 0x7f98c8160d30 2023-10-16T11:19:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.992+0000 7f98beffd700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 0x7f98ac06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.992+0000 7f98ce089700 1 -- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 msgr2=0x7f98ac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.992+0000 7f98beffd700 1 -- 172.21.15.158:0/2363597718 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7013+0+0 (secure 0 0 0) 0x7f98c008a1d0 con 0x7f98c8160d30 2023-10-16T11:19:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.992+0000 7f98ce089700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 0x7f98ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:06.996+0000 7f98beffd700 1 -- 172.21.15.158:0/2363597718 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98c0054a30 con 0x7f98c8160d30 2023-10-16T11:19:07.006 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:07.004+0000 7f34fb1c9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:19:07.081 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:07.079+0000 7f34fb1c9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:19:07.170 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:07.168+0000 7faf26f8d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:19:07.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.193+0000 7f98ce089700 1 -- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 msgr2=0x7f98ac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:07.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.193+0000 7f98ce089700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 0x7f98ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:07.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.195+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98c80681b0 con 0x7f98c8160d30 2023-10-16T11:19:07.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.196+0000 7f98beffd700 1 -- 172.21.15.158:0/2363597718 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v205) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98c0058050 con 0x7f98c8160d30 2023-10-16T11:19:07.198 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:19:07.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 msgr2=0x7f98ac06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:07.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 0x7f98ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c8160d30 msgr2=0x7f98c81f4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c8160d30 0x7f98c81f4ed0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f98c000ef00 tx=0x7f98c000b390 comp rx=0 tx=0).stop 2023-10-16T11:19:07.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 shutdown_connections 2023-10-16T11:19:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f98ac06b140 0x7f98ac06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98c8160d30 0x7f98c81f4ed0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c80d0530 0x7f98c8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 --2- 172.21.15.158:0/2363597718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c80d19a0 0x7f98c80c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 >> 172.21.15.158:0/2363597718 conn(0x7f98c80669f0 msgr2=0x7f98c815f330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:07.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 shutdown_connections 2023-10-16T11:19:07.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.198+0000 7f98d02ed700 1 -- 172.21.15.158:0/2363597718 wait complete. 2023-10-16T11:19:07.212 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:19:07.250 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:07.249+0000 7faf26f8d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:19:07.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.340+0000 7faebdbbc700 1 Processor -- start 2023-10-16T11:19:07.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faebdbbc700 1 -- start start 2023-10-16T11:19:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faebdbbc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb815d990 0x7faeb806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faebdbbc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faeb8070300 0x7faeb8161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faebdbbc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb815dd70 con 0x7faeb8162010 2023-10-16T11:19:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faebdbbc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb81616b0 con 0x7faeb8070300 2023-10-16T11:19:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faebdbbc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb80ced70 con 0x7faeb815d990 2023-10-16T11:19:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faeb7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7faeb8162010 0x7faeb80cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37582/0 (socket says 172.21.15.158:37582) 2023-10-16T11:19:07.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.341+0000 7faeb7fff700 1 -- 172.21.15.158:0/4255795220 learned_addr learned my addr 172.21.15.158:0/4255795220 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb6ffd700 1 --2- 172.21.15.158:0/4255795220 >> v2:172.21.15.172:3300/0 conn(0x7faeb8070300 0x7faeb8161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb77fe700 1 --2- 172.21.15.158:0/4255795220 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb815d990 0x7faeb806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb67fc700 1 -- 172.21.15.158:0/4255795220 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70643161 0 0) 0x7faeb0001110 con 0x7faeb8162010 2023-10-16T11:19:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb67fc700 1 -- 172.21.15.158:0/4255795220 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea40035b0 con 0x7faeb8162010 2023-10-16T11:19:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb77fe700 1 -- 172.21.15.158:0/4255795220 >> v2:172.21.15.172:3300/0 conn(0x7faeb8070300 msgr2=0x7faeb8161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb77fe700 1 --2- 172.21.15.158:0/4255795220 >> v2:172.21.15.172:3300/0 conn(0x7faeb8070300 0x7faeb8161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb77fe700 1 -- 172.21.15.158:0/4255795220 >> v1:172.21.15.158:6789/0 conn(0x7faeb8162010 legacy=0x7faeb80cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb77fe700 1 -- 172.21.15.158:0/4255795220 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8070780 con 0x7faeb815d990 2023-10-16T11:19:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb6ffd700 1 --2- 172.21.15.158:0/4255795220 >> v2:172.21.15.172:3300/0 conn(0x7faeb8070300 0x7faeb8161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb77fe700 1 --2- 172.21.15.158:0/4255795220 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb815d990 0x7faeb806d5d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faea8009d90 tx=0x7faea800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86bd7b0c68bf4cde server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb67fc700 1 -- 172.21.15.158:0/4255795220 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea800e070 con 0x7faeb815d990 2023-10-16T11:19:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb67fc700 1 -- 172.21.15.158:0/4255795220 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea800bb20 con 0x7faeb815d990 2023-10-16T11:19:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.342+0000 7faeb67fc700 1 -- 172.21.15.158:0/4255795220 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8012430 con 0x7faeb815d990 2023-10-16T11:19:07.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 -- 172.21.15.158:0/4255795220 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb815d990 msgr2=0x7faeb806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 --2- 172.21.15.158:0/4255795220 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb815d990 0x7faeb806d5d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faea8009d90 tx=0x7faea800b330 comp rx=0 tx=0).stop 2023-10-16T11:19:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 -- 172.21.15.158:0/4255795220 shutdown_connections 2023-10-16T11:19:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 --2- 172.21.15.158:0/4255795220 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb815d990 0x7faeb806d5d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 --2- 172.21.15.158:0/4255795220 >> v2:172.21.15.172:3300/0 conn(0x7faeb8070300 0x7faeb8161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 -- 172.21.15.158:0/4255795220 >> 172.21.15.158:0/4255795220 conn(0x7faeb80cdf20 msgr2=0x7faeb80ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:07.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 -- 172.21.15.158:0/4255795220 shutdown_connections 2023-10-16T11:19:07.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.343+0000 7faebdbbc700 1 -- 172.21.15.158:0/4255795220 wait complete. 2023-10-16T11:19:07.351 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:07.345+0000 7f34fb1c9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:19:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.344+0000 7faebdbbc700 1 Processor -- start 2023-10-16T11:19:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faebdbbc700 1 -- start start 2023-10-16T11:19:07.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faebdbbc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8070300 0x7faeb81558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faebdbbc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb815d990 0x7faeb80b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faebdbbc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faeb8162010 0x7faeb8157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faebdbbc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb80b4e70 con 0x7faeb8162010 2023-10-16T11:19:07.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faebdbbc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8155e30 con 0x7faeb815d990 2023-10-16T11:19:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faebdbbc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb80b1e20 con 0x7faeb8070300 2023-10-16T11:19:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faeb7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faeb8162010 0x7faeb8157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faeb7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faeb8162010 0x7faeb8157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37412/0 (socket says 172.21.15.158:37412) 2023-10-16T11:19:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faeb7fff700 1 -- 172.21.15.158:0/2442475647 learned_addr learned my addr 172.21.15.158:0/2442475647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:07.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faeb77fe700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8070300 0x7faeb81558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.345+0000 7faeb6ffd700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb815d990 0x7faeb80b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb7fff700 1 -- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8070300 msgr2=0x7faeb81558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb7fff700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8070300 0x7faeb81558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb7fff700 1 -- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb815d990 msgr2=0x7faeb80b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb7fff700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb815d990 0x7faeb80b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb7fff700 1 -- 172.21.15.158:0/2442475647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00094a0 con 0x7faeb8162010 2023-10-16T11:19:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb77fe700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8070300 0x7faeb81558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb6ffd700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb815d990 0x7faeb80b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeb7fff700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faeb8162010 0x7faeb8157890 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7faeb000c4d0 tx=0x7faeb000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:07.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeaffff700 1 -- 172.21.15.158:0/2442475647 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0005d70 con 0x7faeb8162010 2023-10-16T11:19:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeaffff700 1 -- 172.21.15.158:0/2442475647 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb00070d0 con 0x7faeb8162010 2023-10-16T11:19:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea8009a70 con 0x7faeb8162010 2023-10-16T11:19:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faeaffff700 1 -- 172.21.15.158:0/2442475647 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0007460 con 0x7faeb8162010 2023-10-16T11:19:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.346+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb80b44d0 con 0x7faeb8162010 2023-10-16T11:19:07.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.347+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb80b4fb0 con 0x7faeb8162010 2023-10-16T11:19:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.348+0000 7faeaffff700 1 -- 172.21.15.158:0/2442475647 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7faeb00075c0 con 0x7faeb8162010 2023-10-16T11:19:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.348+0000 7faeaffff700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 0x7fae9806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.348+0000 7faeaffff700 1 -- 172.21.15.158:0/2442475647 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7013+0+0 (secure 0 0 0) 0x7faeb0089140 con 0x7faeb8162010 2023-10-16T11:19:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.348+0000 7faeb77fe700 1 -- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 msgr2=0x7fae9806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:07.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.348+0000 7faeb77fe700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 0x7fae9806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:07.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.355+0000 7faeaffff700 1 -- 172.21.15.158:0/2442475647 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faeb80b4fb0 con 0x7faeb8162010 2023-10-16T11:19:07.533 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:07.531+0000 7faf26f8d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:19:07.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.549+0000 7faeb77fe700 1 -- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 msgr2=0x7fae9806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:07.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.549+0000 7faeb77fe700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 0x7fae9806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:07.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.557+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faeb815dd70 con 0x7faeb8162010 2023-10-16T11:19:07.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.557+0000 7faeaffff700 1 -- 172.21.15.158:0/2442475647 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v205) v1 ==== 104+0+11 (secure 0 0 0) 0x7faeb815dd70 con 0x7faeb8162010 2023-10-16T11:19:07.559 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:19:07.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 msgr2=0x7fae9806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:07.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 0x7fae9806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faeb8162010 msgr2=0x7faeb8157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faeb8162010 0x7faeb8157890 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7faeb000c4d0 tx=0x7faeb000c7e0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 shutdown_connections 2023-10-16T11:19:07.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7fae9806b140 0x7fae9806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faeb8162010 0x7faeb8157890 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8070300 0x7faeb81558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 --2- 172.21.15.158:0/2442475647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb815d990 0x7faeb80b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 >> 172.21.15.158:0/2442475647 conn(0x7faeb80cdf20 msgr2=0x7faeb815df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:07.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 shutdown_connections 2023-10-16T11:19:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.560+0000 7faebdbbc700 1 -- 172.21.15.158:0/2442475647 wait complete. 2023-10-16T11:19:07.570 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:07.569+0000 7f34fb1c9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:19:07.574 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:19:07.722 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:07.721+0000 7f34fb1c9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:19:07.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.740+0000 7f1cfea15700 1 Processor -- start 2023-10-16T11:19:07.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.740+0000 7f1cfea15700 1 -- start start 2023-10-16T11:19:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cfea15700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cf815b6b0 0x7f1cf80cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cfea15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80d0c60 0x7f1cf80b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cfea15700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80b4420 con 0x7f1cf80b3460 2023-10-16T11:19:07.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cfea15700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf815ba90 con 0x7f1cf815b6b0 2023-10-16T11:19:07.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cfea15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80b38f0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cf7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cf815b6b0 0x7f1cf80cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cfcfb2700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1cf80b3460 0x7f1cf8070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37598/0 (socket says 172.21.15.158:37598) 2023-10-16T11:19:07.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cfcfb2700 1 -- 172.21.15.158:0/1302283950 learned_addr learned my addr 172.21.15.158:0/1302283950 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:07.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cf77fe700 1 --2- 172.21.15.158:0/1302283950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80d0c60 0x7f1cf80b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cf7fff700 1 -- 172.21.15.158:0/1302283950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80d0c60 msgr2=0x7f1cf80b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:07.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cf7fff700 1 --2- 172.21.15.158:0/1302283950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80d0c60 0x7f1cf80b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cf7fff700 1 -- 172.21.15.158:0/1302283950 >> v1:172.21.15.158:6789/0 conn(0x7f1cf80b3460 legacy=0x7f1cf8070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cf7fff700 1 -- 172.21.15.158:0/1302283950 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80cbc20 con 0x7f1cf815b6b0 2023-10-16T11:19:07.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.741+0000 7f1cf6ffd700 1 -- 172.21.15.158:0/1302283950 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563809639 0 0) 0x7f1cf80b4420 con 0x7f1cf80b3460 2023-10-16T11:19:07.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cf7fff700 1 --2- 172.21.15.158:0/1302283950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf815b6b0 0x7f1cf80cfdb0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009840 tx=0x7f1ce8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc2368d100e8fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:07.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cf6ffd700 1 -- 172.21.15.158:0/1302283950 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8011830 con 0x7f1cf815b6b0 2023-10-16T11:19:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cf6ffd700 1 -- 172.21.15.158:0/1302283950 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce8011990 con 0x7f1cf815b6b0 2023-10-16T11:19:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cf6ffd700 1 -- 172.21.15.158:0/1302283950 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8011d20 con 0x7f1cf815b6b0 2023-10-16T11:19:07.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cfea15700 1 -- 172.21.15.158:0/1302283950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf815b6b0 msgr2=0x7f1cf80cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cfea15700 1 --2- 172.21.15.158:0/1302283950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf815b6b0 0x7f1cf80cfdb0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009840 tx=0x7f1ce8009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cfea15700 1 -- 172.21.15.158:0/1302283950 shutdown_connections 2023-10-16T11:19:07.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cfea15700 1 --2- 172.21.15.158:0/1302283950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf815b6b0 0x7f1cf80cfdb0 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cfea15700 1 --2- 172.21.15.158:0/1302283950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80d0c60 0x7f1cf80b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.742+0000 7f1cfea15700 1 -- 172.21.15.158:0/1302283950 >> 172.21.15.158:0/1302283950 conn(0x7f1cf80cdf20 msgr2=0x7f1cf80ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:07.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.743+0000 7f1cfea15700 1 -- 172.21.15.158:0/1302283950 shutdown_connections 2023-10-16T11:19:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.743+0000 7f1cfea15700 1 -- 172.21.15.158:0/1302283950 wait complete. 2023-10-16T11:19:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.744+0000 7f1cfea15700 1 Processor -- start 2023-10-16T11:19:07.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.744+0000 7f1cfea15700 1 -- start start 2023-10-16T11:19:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cfea15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80b3460 0x7f1cf8151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cfea15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf80d0c60 0x7f1cf8144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cfea15700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf815b6b0 0x7f1cf81f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cfea15700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8157bb0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cfea15700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8144800 con 0x7f1cf815b6b0 2023-10-16T11:19:07.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cfea15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8157de0 con 0x7f1cf80b3460 2023-10-16T11:19:07.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf80d0c60 0x7f1cf8144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf80d0c60 0x7f1cf8144290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37426/0 (socket says 172.21.15.158:37426) 2023-10-16T11:19:07.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.745+0000 7f1cf77fe700 1 -- 172.21.15.158:0/2256352559 learned_addr learned my addr 172.21.15.158:0/2256352559 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:07.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf7fff700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80b3460 0x7f1cf8151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cfcfb2700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf815b6b0 0x7f1cf81f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:07.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf77fe700 1 -- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80b3460 msgr2=0x7f1cf8151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf77fe700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80b3460 0x7f1cf8151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf77fe700 1 -- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf815b6b0 msgr2=0x7f1cf81f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf77fe700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf815b6b0 0x7f1cf81f0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf77fe700 1 -- 172.21.15.158:0/2256352559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80094a0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf77fe700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf80d0c60 0x7f1cf8144290 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00e980 tx=0x7f1cec00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:07.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf4ff9700 1 -- 172.21.15.158:0/2256352559 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec009710 con 0x7f1cf80d0c60 2023-10-16T11:19:07.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf4ff9700 1 -- 172.21.15.158:0/2256352559 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec00c690 con 0x7f1cf80d0c60 2023-10-16T11:19:07.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cf4ff9700 1 -- 172.21.15.158:0/2256352559 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00ca40 con 0x7f1cf80d0c60 2023-10-16T11:19:07.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf8153500 con 0x7f1cf80d0c60 2023-10-16T11:19:07.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.746+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf81f06d0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.747+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf80cc4a0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.748+0000 7f1cf4ff9700 1 -- 172.21.15.158:0/2256352559 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f1cec00cba0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.749+0000 7f1cf4ff9700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 0x7f1ce406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:07.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.749+0000 7f1cf7fff700 1 -- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 msgr2=0x7f1ce406d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:07.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.749+0000 7f1cf4ff9700 1 -- 172.21.15.158:0/2256352559 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7013+0+0 (secure 0 0 0) 0x7f1cec089fc0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.749+0000 7f1cf7fff700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 0x7f1ce406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:07.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.753+0000 7f1cf4ff9700 1 -- 172.21.15.158:0/2256352559 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cec0bcfe0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.774 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:07.772+0000 7faf26f8d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:19:07.841 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:07.839+0000 7f34fb1c9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:19:07.931 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:07.930+0000 7faf26f8d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:19:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.950+0000 7f1cf7fff700 1 -- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 msgr2=0x7f1ce406d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.950+0000 7f1cf7fff700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 0x7f1ce406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:07.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.955+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1cf80681b0 con 0x7f1cf80d0c60 2023-10-16T11:19:07.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.955+0000 7f1cf4ff9700 1 -- 172.21.15.158:0/2256352559 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v205) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1cec021020 con 0x7f1cf80d0c60 2023-10-16T11:19:07.957 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:19:07.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 msgr2=0x7f1ce406d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:07.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 0x7f1ce406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf80d0c60 msgr2=0x7f1cf8144290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf80d0c60 0x7f1cf8144290 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00e980 tx=0x7f1cec00ec90 comp rx=0 tx=0).stop 2023-10-16T11:19:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 shutdown_connections 2023-10-16T11:19:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1ce406b170 0x7f1ce406d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cf815b6b0 0x7f1cf81f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cf80b3460 0x7f1cf8151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 --2- 172.21.15.158:0/2256352559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cf80d0c60 0x7f1cf8144290 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 >> 172.21.15.158:0/2256352559 conn(0x7f1cf80cdf20 msgr2=0x7f1cf80b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 shutdown_connections 2023-10-16T11:19:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:07.957+0000 7f1cfea15700 1 -- 172.21.15.158:0/2256352559 wait complete. 2023-10-16T11:19:07.972 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:07.997 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:07.995+0000 7f34fb1c9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:19:08.056 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:08.054+0000 7faf26f8d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:19:08.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.095+0000 7f1fab782700 1 Processor -- start 2023-10-16T11:19:08.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.095+0000 7f1fab782700 1 -- start start 2023-10-16T11:19:08.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.096+0000 7f1fab782700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1fa40cf4c0 0x7f1fa4161270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.096+0000 7f1fab782700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa415de20 con 0x7f1fa406bd10 2023-10-16T11:19:08.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.096+0000 7f1fab782700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa4162180 con 0x7f1fa40cf8a0 2023-10-16T11:19:08.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.096+0000 7f1fab782700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa41618e0 con 0x7f1fa40cf4c0 2023-10-16T11:19:08.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.096+0000 7f1fa9d1f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1fa406bd10 0x7f1fa415e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37614/0 (socket says 172.21.15.158:37614) 2023-10-16T11:19:08.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.096+0000 7f1fa9d1f700 1 -- 172.21.15.158:0/2833638948 learned_addr learned my addr 172.21.15.158:0/2833638948 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:08.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.096+0000 7f1fa951e700 1 --2- 172.21.15.158:0/2833638948 >> v2:172.21.15.172:3300/0 conn(0x7f1fa40cf4c0 0x7f1fa4161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1f9bfff700 1 -- 172.21.15.158:0/2833638948 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361480366 0 0) 0x7f1fa415de20 con 0x7f1fa406bd10 2023-10-16T11:19:08.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1f9bfff700 1 -- 172.21.15.158:0/2833638948 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f900035b0 con 0x7f1fa406bd10 2023-10-16T11:19:08.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1fa951e700 1 -- 172.21.15.158:0/2833638948 >> v1:172.21.15.172:6789/0 conn(0x7f1fa40cf8a0 legacy=0x7f1fa406d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1f9bfff700 1 -- 172.21.15.158:0/2833638948 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906539798 0 0) 0x7f1fa4162180 con 0x7f1fa40cf8a0 2023-10-16T11:19:08.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1fa951e700 1 -- 172.21.15.158:0/2833638948 >> v1:172.21.15.158:6789/0 conn(0x7f1fa406bd10 legacy=0x7f1fa415e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1fa951e700 1 -- 172.21.15.158:0/2833638948 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa4161bc0 con 0x7f1fa40cf4c0 2023-10-16T11:19:08.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1fa951e700 1 --2- 172.21.15.158:0/2833638948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf4c0 0x7f1fa4161270 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009840 tx=0x7f1fa0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beecf6037265b88b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:08.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1f9bfff700 1 -- 172.21.15.158:0/2833638948 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c0019f0 con 0x7f1fa40cf4c0 2023-10-16T11:19:08.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1f9bfff700 1 -- 172.21.15.158:0/2833638948 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa0004030 con 0x7f1fa40cf4c0 2023-10-16T11:19:08.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.097+0000 7f1f9bfff700 1 -- 172.21.15.158:0/2833638948 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa00043c0 con 0x7f1fa40cf4c0 2023-10-16T11:19:08.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.098+0000 7f1fab782700 1 -- 172.21.15.158:0/2833638948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf4c0 msgr2=0x7f1fa4161270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.098+0000 7f1fab782700 1 --2- 172.21.15.158:0/2833638948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf4c0 0x7f1fa4161270 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009840 tx=0x7f1fa0009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:08.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.098+0000 7f1fab782700 1 -- 172.21.15.158:0/2833638948 shutdown_connections 2023-10-16T11:19:08.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.098+0000 7f1fab782700 1 --2- 172.21.15.158:0/2833638948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf4c0 0x7f1fa4161270 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.098+0000 7f1fab782700 1 -- 172.21.15.158:0/2833638948 >> 172.21.15.158:0/2833638948 conn(0x7f1fa406a8d0 msgr2=0x7f1fa406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:08.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.098+0000 7f1fab782700 1 -- 172.21.15.158:0/2833638948 shutdown_connections 2023-10-16T11:19:08.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.098+0000 7f1fab782700 1 -- 172.21.15.158:0/2833638948 wait complete. 2023-10-16T11:19:08.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.099+0000 7f1fab782700 1 Processor -- start 2023-10-16T11:19:08.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.099+0000 7f1fab782700 1 -- start start 2023-10-16T11:19:08.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fab782700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa406bd10 0x7f1fa40c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fab782700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf8a0 0x7f1fa40cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fab782700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fa40c87c0 0x7f1fa415ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fab782700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40c8d60 con 0x7f1fa40c87c0 2023-10-16T11:19:08.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fab782700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40c7dd0 con 0x7f1fa40cf8a0 2023-10-16T11:19:08.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fab782700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40c94c0 con 0x7f1fa406bd10 2023-10-16T11:19:08.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fa40c87c0 0x7f1fa415ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fa40c87c0 0x7f1fa415ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37444/0 (socket says 172.21.15.158:37444) 2023-10-16T11:19:08.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 -- 172.21.15.158:0/3815486512 learned_addr learned my addr 172.21.15.158:0/3815486512 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:08.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa951e700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa406bd10 0x7f1fa40c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa8d1d700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf8a0 0x7f1fa40cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 -- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa406bd10 msgr2=0x7f1fa40c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa406bd10 0x7f1fa40c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 -- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf8a0 msgr2=0x7f1fa40cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf8a0 0x7f1fa40cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.100+0000 7f1fa9d1f700 1 -- 172.21.15.158:0/3815486512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa00094a0 con 0x7f1fa40c87c0 2023-10-16T11:19:08.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.101+0000 7f1fa9d1f700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fa40c87c0 0x7f1fa415ffa0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c002040 tx=0x7f1f9c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:08.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.101+0000 7f1f99ffb700 1 -- 172.21.15.158:0/3815486512 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c00acd0 con 0x7f1fa40c87c0 2023-10-16T11:19:08.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.101+0000 7f1f99ffb700 1 -- 172.21.15.158:0/3815486512 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c00ae30 con 0x7f1fa40c87c0 2023-10-16T11:19:08.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.101+0000 7f1f99ffb700 1 -- 172.21.15.158:0/3815486512 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c007310 con 0x7f1fa40c87c0 2023-10-16T11:19:08.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.101+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa41f7050 con 0x7f1fa40c87c0 2023-10-16T11:19:08.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.101+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa41f74e0 con 0x7f1fa40c87c0 2023-10-16T11:19:08.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.102+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa41f71e0 con 0x7f1fa40c87c0 2023-10-16T11:19:08.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.103+0000 7f1f99ffb700 1 -- 172.21.15.158:0/3815486512 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f1f9c007470 con 0x7f1fa40c87c0 2023-10-16T11:19:08.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.103+0000 7f1f99ffb700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 0x7f1f8806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.103+0000 7f1fa951e700 1 -- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 msgr2=0x7f1f8806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:08.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.103+0000 7f1f99ffb700 1 -- 172.21.15.158:0/3815486512 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7013+0+0 (secure 0 0 0) 0x7f1f9c089ea0 con 0x7f1fa40c87c0 2023-10-16T11:19:08.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.103+0000 7f1fa951e700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 0x7f1f8806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:08.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.107+0000 7f1f99ffb700 1 -- 172.21.15.158:0/3815486512 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f9c0546d0 con 0x7f1fa40c87c0 2023-10-16T11:19:08.221 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:08.220+0000 7faf26f8d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:19:08.260 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:08.259+0000 7f34fb1c9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:08.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.304+0000 7f1fa951e700 1 -- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 msgr2=0x7f1f8806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:08.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.304+0000 7f1fa951e700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 0x7f1f8806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:08.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.352+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fa40681b0 con 0x7f1fa40c87c0 2023-10-16T11:19:08.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.352+0000 7f1f99ffb700 1 -- 172.21.15.158:0/3815486512 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1f9c018020 con 0x7f1fa40c87c0 2023-10-16T11:19:08.354 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:08.354 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":103,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2912297853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2912297853},{"type":"v1","addr":"172.21.15.158:6836","nonce":2912297853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":38441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2297507228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2297507228},{"type":"v1","addr":"172.21.15.158:6837","nonce":2297507228}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38447,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3795197457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3795197457},{"type":"v1","addr":"172.21.15.172:6834","nonce":3795197457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:17:40.072523+0000","modified":"2023-10-16T11:17:43.106178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38450":{"gid":38450,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1279760034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1279760034},{"type":"v1","addr":"172.21.15.172:6835","nonce":1279760034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T11:19:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 msgr2=0x7f1f8806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 0x7f1f8806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fa40c87c0 msgr2=0x7f1fa415ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fa40c87c0 0x7f1fa415ffa0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c002040 tx=0x7f1f9c00a340 comp rx=0 tx=0).stop 2023-10-16T11:19:08.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 shutdown_connections 2023-10-16T11:19:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1f8806b140 0x7f1f8806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fa40c87c0 0x7f1fa415ffa0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa406bd10 0x7f1fa40c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 --2- 172.21.15.158:0/3815486512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cf8a0 0x7f1fa40cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 >> 172.21.15.158:0/3815486512 conn(0x7f1fa406a8d0 msgr2=0x7f1fa406ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 shutdown_connections 2023-10-16T11:19:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.355+0000 7f1fab782700 1 -- 172.21.15.158:0/3815486512 wait complete. 2023-10-16T11:19:08.359 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 103 2023-10-16T11:19:08.370 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:08.422 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:08.420+0000 7f34fb1c9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:19:08.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.491+0000 7f1b2ee31700 1 Processor -- start 2023-10-16T11:19:08.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.491+0000 7f1b2ee31700 1 -- start start 2023-10-16T11:19:08.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.492+0000 7f1b2ee31700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1b2815a6c0 0x7f1b280723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.492+0000 7f1b2ee31700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b28072910 con 0x7f1b2806e090 2023-10-16T11:19:08.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.492+0000 7f1b2ee31700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b280b1e40 con 0x7f1b280af590 2023-10-16T11:19:08.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.492+0000 7f1b2ee31700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b280ca7e0 con 0x7f1b2815a6c0 2023-10-16T11:19:08.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.492+0000 7f1b2cbcd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1b2815a6c0 0x7f1b280723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.492+0000 7f1b2d3ce700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1b2806e090 0x7f1b280ad430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37630/0 (socket says 172.21.15.158:37630) 2023-10-16T11:19:08.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.492+0000 7f1b2d3ce700 1 -- 172.21.15.158:0/738523461 learned_addr learned my addr 172.21.15.158:0/738523461 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:08.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b2cbcd700 1 -- 172.21.15.158:0/738523461 >> v1:172.21.15.172:6789/0 conn(0x7f1b280af590 legacy=0x7f1b2806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b2cbcd700 1 -- 172.21.15.158:0/738523461 >> v1:172.21.15.158:6789/0 conn(0x7f1b2806e090 legacy=0x7f1b280ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b2cbcd700 1 -- 172.21.15.158:0/738523461 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b280b2120 con 0x7f1b2815a6c0 2023-10-16T11:19:08.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b277fe700 1 -- 172.21.15.158:0/738523461 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304885843 0 0) 0x7f1b28072910 con 0x7f1b2806e090 2023-10-16T11:19:08.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b2cbcd700 1 --2- 172.21.15.158:0/738523461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b280723d0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009820 tx=0x7f1b1c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdbb0b538f1c972 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:08.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b277fe700 1 -- 172.21.15.158:0/738523461 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c005e30 con 0x7f1b2815a6c0 2023-10-16T11:19:08.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b277fe700 1 -- 172.21.15.158:0/738523461 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c006030 con 0x7f1b2815a6c0 2023-10-16T11:19:08.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.493+0000 7f1b277fe700 1 -- 172.21.15.158:0/738523461 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c0063e0 con 0x7f1b2815a6c0 2023-10-16T11:19:08.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.494+0000 7f1b2ee31700 1 -- 172.21.15.158:0/738523461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 msgr2=0x7f1b280723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.494+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/738523461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b280723d0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009820 tx=0x7f1b1c009b30 comp rx=0 tx=0).stop 2023-10-16T11:19:08.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.494+0000 7f1b2ee31700 1 -- 172.21.15.158:0/738523461 shutdown_connections 2023-10-16T11:19:08.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.494+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/738523461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b280723d0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.494+0000 7f1b2ee31700 1 -- 172.21.15.158:0/738523461 >> 172.21.15.158:0/738523461 conn(0x7f1b280ccda0 msgr2=0x7f1b280cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:08.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.494+0000 7f1b2ee31700 1 -- 172.21.15.158:0/738523461 shutdown_connections 2023-10-16T11:19:08.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.494+0000 7f1b2ee31700 1 -- 172.21.15.158:0/738523461 wait complete. 2023-10-16T11:19:08.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.495+0000 7f1b2ee31700 1 Processor -- start 2023-10-16T11:19:08.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.495+0000 7f1b2ee31700 1 -- start start 2023-10-16T11:19:08.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2ee31700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b2806e090 0x7f1b280c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2ee31700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b280af590 0x7f1b281f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2ee31700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b281f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2ee31700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b280c6d10 con 0x7f1b2815a6c0 2023-10-16T11:19:08.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2ee31700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b281f29c0 con 0x7f1b2806e090 2023-10-16T11:19:08.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2ee31700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b281f2ca0 con 0x7f1b280af590 2023-10-16T11:19:08.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b281f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b281f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37460/0 (socket says 172.21.15.158:37460) 2023-10-16T11:19:08.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 -- 172.21.15.158:0/400044889 learned_addr learned my addr 172.21.15.158:0/400044889 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:08.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b27fff700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b280af590 0x7f1b281f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2cbcd700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b2806e090 0x7f1b280c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 -- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b280af590 msgr2=0x7f1b281f3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b280af590 0x7f1b281f3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 -- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b2806e090 msgr2=0x7f1b280c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b2806e090 0x7f1b280c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 -- 172.21.15.158:0/400044889 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c009430 con 0x7f1b2815a6c0 2023-10-16T11:19:08.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.496+0000 7f1b2d3ce700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b281f1e50 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000cc10 tx=0x7f1b2000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:08.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.497+0000 7f1b257fa700 1 -- 172.21.15.158:0/400044889 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2000aaa0 con 0x7f1b2815a6c0 2023-10-16T11:19:08.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.497+0000 7f1b257fa700 1 -- 172.21.15.158:0/400044889 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2000ac00 con 0x7f1b2815a6c0 2023-10-16T11:19:08.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.497+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b281f2f80 con 0x7f1b2815a6c0 2023-10-16T11:19:08.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.497+0000 7f1b257fa700 1 -- 172.21.15.158:0/400044889 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b200070d0 con 0x7f1b2815a6c0 2023-10-16T11:19:08.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.497+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b281f3540 con 0x7f1b2815a6c0 2023-10-16T11:19:08.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.498+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b281f3760 con 0x7f1b2815a6c0 2023-10-16T11:19:08.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.498+0000 7f1b257fa700 1 -- 172.21.15.158:0/400044889 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f1b20007230 con 0x7f1b2815a6c0 2023-10-16T11:19:08.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.498+0000 7f1b257fa700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 0x7f1b0c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.498+0000 7f1b2cbcd700 1 -- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 msgr2=0x7f1b0c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:08.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.498+0000 7f1b257fa700 1 -- 172.21.15.158:0/400044889 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7013+0+0 (secure 0 0 0) 0x7f1b20089c60 con 0x7f1b2815a6c0 2023-10-16T11:19:08.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.498+0000 7f1b2cbcd700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 0x7f1b0c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:08.509 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:08.501+0000 7faf26f8d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:08.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.503+0000 7f1b257fa700 1 -- 172.21.15.158:0/400044889 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b20054490 con 0x7f1b2815a6c0 2023-10-16T11:19:08.674 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:08.672+0000 7faf26f8d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:19:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.699+0000 7f1b2cbcd700 1 -- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 msgr2=0x7f1b0c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:08.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.699+0000 7f1b2cbcd700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 0x7f1b0c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:08.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.704+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b280681b0 con 0x7f1b2815a6c0 2023-10-16T11:19:08.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.706+0000 7f1b257fa700 1 -- 172.21.15.158:0/400044889 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+18253 (secure 0 0 0) 0x7f1b20018090 con 0x7f1b2815a6c0 2023-10-16T11:19:08.708 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:08.708 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":205,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:06.802082+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:17:38.499055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T11:17:39.035057+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":205,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.1b","mappings":[{"from":3,"to":0}]},{"pgid":"26.29","mappings":[{"from":1,"to":6}]},{"pgid":"26.3e","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2415272176":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6833/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6832/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/778612447":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/1354527416":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/3047051699":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/4256533896":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/716693565":"2023-10-17T11:17:53.925811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 msgr2=0x7f1b0c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 0x7f1b0c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 msgr2=0x7f1b281f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b281f1e50 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000cc10 tx=0x7f1b2000cf20 comp rx=0 tx=0).stop 2023-10-16T11:19:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 shutdown_connections 2023-10-16T11:19:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f1b0c06b140 0x7f1b0c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b2815a6c0 0x7f1b281f1e50 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b2806e090 0x7f1b280c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 --2- 172.21.15.158:0/400044889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b280af590 0x7f1b281f3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 >> 172.21.15.158:0/400044889 conn(0x7f1b280ccda0 msgr2=0x7f1b280cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 shutdown_connections 2023-10-16T11:19:08.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.708+0000 7f1b2ee31700 1 -- 172.21.15.158:0/400044889 wait complete. 2023-10-16T11:19:08.722 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:19:08.723 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:08.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.844+0000 7f45925fc700 1 Processor -- start 2023-10-16T11:19:08.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.845+0000 7f45925fc700 1 -- start start 2023-10-16T11:19:08.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f45925fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f458c0ca8f0 0x7f458c06bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f45925fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f45925fc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458c06b830 con 0x7f458c15d920 2023-10-16T11:19:08.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f45925fc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c06fb50 con 0x7f458c0ca8f0 2023-10-16T11:19:08.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f45925fc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c06c200 con 0x7f458c0cead0 2023-10-16T11:19:08.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f458c0ca8f0 0x7f458c06bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f458c15d920 0x7f458c0cacd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37642/0 (socket says 172.21.15.158:37642) 2023-10-16T11:19:08.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f458c0ca8f0 0x7f458c06bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37466/0 (socket says 172.21.15.158:37466) 2023-10-16T11:19:08.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458b7fe700 1 -- 172.21.15.158:0/4096736165 learned_addr learned my addr 172.21.15.158:0/4096736165 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:08.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f4590b99700 1 --2- 172.21.15.158:0/4096736165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458bfff700 1 -- 172.21.15.158:0/4096736165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 msgr2=0x7f458c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:08.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458bfff700 1 --2- 172.21.15.158:0/4096736165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458bfff700 1 -- 172.21.15.158:0/4096736165 >> v1:172.21.15.158:6789/0 conn(0x7f458c15d920 legacy=0x7f458c0cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458bfff700 1 -- 172.21.15.158:0/4096736165 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c06ac20 con 0x7f458c0ca8f0 2023-10-16T11:19:08.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.846+0000 7f458affd700 1 -- 172.21.15.158:0/4096736165 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60110916 0 0) 0x7f458c06b830 con 0x7f458c15d920 2023-10-16T11:19:08.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f4590b99700 1 --2- 172.21.15.158:0/4096736165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:08.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f458bfff700 1 --2- 172.21.15.158:0/4096736165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c06bcc0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f457c009840 tx=0x7f457c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73577e739d016671 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:08.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f458affd700 1 -- 172.21.15.158:0/4096736165 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c011830 con 0x7f458c0ca8f0 2023-10-16T11:19:08.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f458affd700 1 -- 172.21.15.158:0/4096736165 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c011990 con 0x7f458c0ca8f0 2023-10-16T11:19:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f458affd700 1 -- 172.21.15.158:0/4096736165 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c011d20 con 0x7f458c0ca8f0 2023-10-16T11:19:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 -- 172.21.15.158:0/4096736165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 msgr2=0x7f458c06bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 --2- 172.21.15.158:0/4096736165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c06bcc0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f457c009840 tx=0x7f457c009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 -- 172.21.15.158:0/4096736165 shutdown_connections 2023-10-16T11:19:08.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 --2- 172.21.15.158:0/4096736165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c06bcc0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 --2- 172.21.15.158:0/4096736165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 -- 172.21.15.158:0/4096736165 >> 172.21.15.158:0/4096736165 conn(0x7f458c0b1510 msgr2=0x7f458c0b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 -- 172.21.15.158:0/4096736165 shutdown_connections 2023-10-16T11:19:08.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.847+0000 7f45925fc700 1 -- 172.21.15.158:0/4096736165 wait complete. 2023-10-16T11:19:08.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.849+0000 7f45925fc700 1 Processor -- start 2023-10-16T11:19:08.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.849+0000 7f45925fc700 1 -- start start 2023-10-16T11:19:08.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c1f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f458c15d920 0x7f458c1515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c160140 con 0x7f458c0ca8f0 2023-10-16T11:19:08.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c1f2cb0 con 0x7f458c15d920 2023-10-16T11:19:08.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c15fb30 con 0x7f458c0cead0 2023-10-16T11:19:08.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c1f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c1f6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37472/0 (socket says 172.21.15.158:37472) 2023-10-16T11:19:08.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 -- 172.21.15.158:0/3465249524 learned_addr learned my addr 172.21.15.158:0/3465249524 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:08.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f4590b99700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f458c15d920 0x7f458c1515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458b7fe700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:08.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 -- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 msgr2=0x7f458c160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 -- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f458c15d920 msgr2=0x7f458c1515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:08.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f458c15d920 0x7f458c1515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:08.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 -- 172.21.15.158:0/3465249524 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0094a0 con 0x7f458c0ca8f0 2023-10-16T11:19:08.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f458bfff700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c1f6370 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f457c0090b0 tx=0x7f457c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:08.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f4588ff9700 1 -- 172.21.15.158:0/3465249524 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0036a0 con 0x7f458c0ca8f0 2023-10-16T11:19:08.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f4588ff9700 1 -- 172.21.15.158:0/3465249524 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c01e070 con 0x7f458c0ca8f0 2023-10-16T11:19:08.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c1f72a0 con 0x7f458c0ca8f0 2023-10-16T11:19:08.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f4588ff9700 1 -- 172.21.15.158:0/3465249524 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c003a60 con 0x7f458c0ca8f0 2023-10-16T11:19:08.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.850+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c1f6fc0 con 0x7f458c0ca8f0 2023-10-16T11:19:08.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.851+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c0681b0 con 0x7f458c0ca8f0 2023-10-16T11:19:08.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.852+0000 7f4588ff9700 1 -- 172.21.15.158:0/3465249524 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f457c003bc0 con 0x7f458c0ca8f0 2023-10-16T11:19:08.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.852+0000 7f4588ff9700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 0x7f457806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:08.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.852+0000 7f458b7fe700 1 -- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 msgr2=0x7f457806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:08.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.852+0000 7f4588ff9700 1 -- 172.21.15.158:0/3465249524 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7013+0+0 (secure 0 0 0) 0x7f457c08af10 con 0x7f458c0ca8f0 2023-10-16T11:19:08.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.852+0000 7f458b7fe700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 0x7f457806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:08.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:08.857+0000 7f4588ff9700 1 -- 172.21.15.158:0/3465249524 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f457c055750 con 0x7f458c0ca8f0 2023-10-16T11:19:08.955 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:08.953+0000 7f34fb1c9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:19:09.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.053+0000 7f458b7fe700 1 -- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 msgr2=0x7f457806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:09.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.053+0000 7f458b7fe700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 0x7f457806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:09.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.101+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458c06b490 con 0x7f458c0ca8f0 2023-10-16T11:19:09.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.101+0000 7f4588ff9700 1 -- 172.21.15.158:0/3465249524 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f457c020090 con 0x7f458c0ca8f0 2023-10-16T11:19:09.103 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:09.103 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":103,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2912297853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2912297853},{"type":"v1","addr":"172.21.15.158:6836","nonce":2912297853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":38441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2297507228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2297507228},{"type":"v1","addr":"172.21.15.158:6837","nonce":2297507228}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38447,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3795197457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3795197457},{"type":"v1","addr":"172.21.15.172:6834","nonce":3795197457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:17:40.072523+0000","modified":"2023-10-16T11:17:43.106178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38450":{"gid":38450,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1279760034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1279760034},{"type":"v1","addr":"172.21.15.172:6835","nonce":1279760034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T11:19:09.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 msgr2=0x7f457806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:09.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 0x7f457806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 msgr2=0x7f458c1f6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c1f6370 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f457c0090b0 tx=0x7f457c004750 comp rx=0 tx=0).stop 2023-10-16T11:19:09.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 shutdown_connections 2023-10-16T11:19:09.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f457806b140 0x7f457806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f458c15d920 0x7f458c1515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f458c0ca8f0 0x7f458c1f6370 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 --2- 172.21.15.158:0/3465249524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f458c0cead0 0x7f458c160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 >> 172.21.15.158:0/3465249524 conn(0x7f458c0b1510 msgr2=0x7f458c06ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:09.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 shutdown_connections 2023-10-16T11:19:09.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.104+0000 7f45925fc700 1 -- 172.21.15.158:0/3465249524 wait complete. 2023-10-16T11:19:09.108 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 103 2023-10-16T11:19:09.119 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:09.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.241+0000 7f4211fb7700 1 Processor -- start 2023-10-16T11:19:09.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.241+0000 7f4211fb7700 1 -- start start 2023-10-16T11:19:09.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.242+0000 7f4211fb7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f420c15a6c0 0x7f420c0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.242+0000 7f4211fb7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c072910 con 0x7f420c06e090 2023-10-16T11:19:09.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.242+0000 7f4211fb7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c0b1e40 con 0x7f420c0af590 2023-10-16T11:19:09.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.242+0000 7f4211fb7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c0ca7e0 con 0x7f420c15a6c0 2023-10-16T11:19:09.247 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.243+0000 7faf26f8d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:19:09.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f420c06e090 0x7f420c0ad430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37662/0 (socket says 172.21.15.158:37662) 2023-10-16T11:19:09.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420bfff700 1 -- 172.21.15.158:0/2822876037 learned_addr learned my addr 172.21.15.158:0/2822876037 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:09.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420b7fe700 1 --2- 172.21.15.158:0/2822876037 >> v2:172.21.15.158:3300/0 conn(0x7f420c15a6c0 0x7f420c0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420a7fc700 1 -- 172.21.15.158:0/2822876037 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818867721 0 0) 0x7f420c072910 con 0x7f420c06e090 2023-10-16T11:19:09.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420a7fc700 1 -- 172.21.15.158:0/2822876037 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f40035b0 con 0x7f420c06e090 2023-10-16T11:19:09.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420b7fe700 1 -- 172.21.15.158:0/2822876037 >> v1:172.21.15.172:6789/0 conn(0x7f420c0af590 legacy=0x7f420c06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420b7fe700 1 -- 172.21.15.158:0/2822876037 >> v1:172.21.15.158:6789/0 conn(0x7f420c06e090 legacy=0x7f420c0ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420b7fe700 1 -- 172.21.15.158:0/2822876037 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c0b2120 con 0x7f420c15a6c0 2023-10-16T11:19:09.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420b7fe700 1 --2- 172.21.15.158:0/2822876037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c15a6c0 0x7f420c0723d0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f41fc009820 tx=0x7f41fc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34d5559270a9ff5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:09.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420a7fc700 1 -- 172.21.15.158:0/2822876037 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc005e30 con 0x7f420c15a6c0 2023-10-16T11:19:09.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420a7fc700 1 -- 172.21.15.158:0/2822876037 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc006030 con 0x7f420c15a6c0 2023-10-16T11:19:09.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.243+0000 7f420a7fc700 1 -- 172.21.15.158:0/2822876037 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0063c0 con 0x7f420c15a6c0 2023-10-16T11:19:09.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.244+0000 7f4211fb7700 1 -- 172.21.15.158:0/2822876037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c15a6c0 msgr2=0x7f420c0723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.244+0000 7f4211fb7700 1 --2- 172.21.15.158:0/2822876037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c15a6c0 0x7f420c0723d0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f41fc009820 tx=0x7f41fc009b30 comp rx=0 tx=0).stop 2023-10-16T11:19:09.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.244+0000 7f4211fb7700 1 -- 172.21.15.158:0/2822876037 shutdown_connections 2023-10-16T11:19:09.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.244+0000 7f4211fb7700 1 --2- 172.21.15.158:0/2822876037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c15a6c0 0x7f420c0723d0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.244+0000 7f4211fb7700 1 -- 172.21.15.158:0/2822876037 >> 172.21.15.158:0/2822876037 conn(0x7f420c0ccda0 msgr2=0x7f420c0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:09.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.244+0000 7f4211fb7700 1 -- 172.21.15.158:0/2822876037 shutdown_connections 2023-10-16T11:19:09.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.244+0000 7f4211fb7700 1 -- 172.21.15.158:0/2822876037 wait complete. 2023-10-16T11:19:09.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.246+0000 7f4211fb7700 1 Processor -- start 2023-10-16T11:19:09.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.246+0000 7f4211fb7700 1 -- start start 2023-10-16T11:19:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c06e090 0x7f420c0c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0af590 0x7f420c1ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f420c15a6c0 0x7f420c148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0b43c0 con 0x7f420c0af590 2023-10-16T11:19:09.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0c6cc0 con 0x7f420c15a6c0 2023-10-16T11:19:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0b47e0 con 0x7f420c06e090 2023-10-16T11:19:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0af590 0x7f420c1ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0af590 0x7f420c1ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37490/0 (socket says 172.21.15.158:37490) 2023-10-16T11:19:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 -- 172.21.15.158:0/4075860227 learned_addr learned my addr 172.21.15.158:0/4075860227 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:09.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420b7fe700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c06e090 0x7f420c0c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420bfff700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f420c15a6c0 0x7f420c148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 -- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c06e090 msgr2=0x7f420c0c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c06e090 0x7f420c0c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 -- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f420c15a6c0 msgr2=0x7f420c148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f420c15a6c0 0x7f420c148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 -- 172.21.15.158:0/4075860227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc009430 con 0x7f420c0af590 2023-10-16T11:19:09.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f420affd700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0af590 0x7f420c1ee5b0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f420000c4b0 tx=0x7f420000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:09.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42000052d0 con 0x7f420c0af590 2023-10-16T11:19:09.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4200005430 con 0x7f420c0af590 2023-10-16T11:19:09.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420c0b3750 con 0x7f420c0af590 2023-10-16T11:19:09.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42000038c0 con 0x7f420c0af590 2023-10-16T11:19:09.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.247+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c0b3d00 con 0x7f420c0af590 2023-10-16T11:19:09.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.248+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420c0b26c0 con 0x7f420c0af590 2023-10-16T11:19:09.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.249+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 133) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f4200003a20 con 0x7f420c0af590 2023-10-16T11:19:09.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.249+0000 7f41f3fff700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f41f806b140 0x7f41f806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.249+0000 7f420b7fe700 1 -- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f41f806b140 msgr2=0x7f41f806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/2839558305 2023-10-16T11:19:09.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.250+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7013+0+0 (secure 0 0 0) 0x7f4200089120 con 0x7f420c0af590 2023-10-16T11:19:09.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.250+0000 7f420b7fe700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f41f806b140 0x7f41f806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:09.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.254+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f420c0b26c0 con 0x7f420c0af590 2023-10-16T11:19:09.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.262+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4200053e80 con 0x7f420c0af590 2023-10-16T11:19:09.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.262+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f41f806b140 msgr2=0x7f41f806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:09.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.262+0000 7f41f3fff700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f41f806b140 0x7f41f806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.328 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7faea27b9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.328 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7faea27b9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.328 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7faea27b9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.329 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7faea27b9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.329 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7faea27b9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.329 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7fae9f773700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.329 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7fae9f773700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.329 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7fae9f773700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.330 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7fae9f773700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.330 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:09.326+0000 7fae9f773700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.510+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f420c0b4500 con 0x7f420c0af590 2023-10-16T11:19:09.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.510+0000 7f41f3fff700 1 -- 172.21.15.158:0/4075860227 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f420c0b4500 con 0x7f420c0af590 2023-10-16T11:19:09.513 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:09.513 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":103,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2912297853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2912297853},{"type":"v1","addr":"172.21.15.158:6836","nonce":2912297853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":38441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2297507228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2297507228},{"type":"v1","addr":"172.21.15.158:6837","nonce":2297507228}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38447,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3795197457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3795197457},{"type":"v1","addr":"172.21.15.172:6834","nonce":3795197457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:17:40.072523+0000","modified":"2023-10-16T11:17:43.106178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38450":{"gid":38450,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/1279760034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1279760034},{"type":"v1","addr":"172.21.15.172:6835","nonce":1279760034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T11:19:09.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0af590 msgr2=0x7f420c1ee5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0af590 0x7f420c1ee5b0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f420000c4b0 tx=0x7f420000c7c0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 shutdown_connections 2023-10-16T11:19:09.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:6832/2839558305,v1:172.21.15.158:6833/2839558305] conn(0x7f41f806b140 0x7f41f806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f420c15a6c0 0x7f420c148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f420c06e090 0x7f420c0c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 --2- 172.21.15.158:0/4075860227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0af590 0x7f420c1ee5b0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.512+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 >> 172.21.15.158:0/4075860227 conn(0x7f420c0ccda0 msgr2=0x7f420c06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:09.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.513+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 shutdown_connections 2023-10-16T11:19:09.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.513+0000 7f4211fb7700 1 -- 172.21.15.158:0/4075860227 wait complete. 2023-10-16T11:19:09.517 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 103 2023-10-16T11:19:09.526 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:09.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.646+0000 7fc92be1d700 1 Processor -- start 2023-10-16T11:19:09.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.646+0000 7fc92be1d700 1 -- start start 2023-10-16T11:19:09.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc92be1d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc92415dfb0 0x7fc92406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc92be1d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 0x7fc92406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc92be1d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92406e770 con 0x7fc9240cc5d0 2023-10-16T11:19:09.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc92be1d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92406fd30 con 0x7fc92415dfb0 2023-10-16T11:19:09.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc92be1d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240cf270 con 0x7fc92406f340 2023-10-16T11:19:09.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc929bb9700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9240cc5d0 0x7fc9240cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37742/0 (socket says 172.21.15.158:37742) 2023-10-16T11:19:09.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc929bb9700 1 -- 172.21.15.158:0/2914172323 learned_addr learned my addr 172.21.15.158:0/2914172323 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:09.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc92a3ba700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 0x7fc92406f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc9293b8700 1 --2- 172.21.15.158:0/2914172323 >> v2:172.21.15.172:3300/0 conn(0x7fc92415dfb0 0x7fc92406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349594874 0 0) 0x7fc914001110 con 0x7fc9240cc5d0 2023-10-16T11:19:09.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.647+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9100035b0 con 0x7fc9240cc5d0 2023-10-16T11:19:09.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650418847 0 0) 0x7fc9100035b0 con 0x7fc9240cc5d0 2023-10-16T11:19:09.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 msgr2=0x7fc92406f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:09.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 0x7fc92406f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 >> v2:172.21.15.172:3300/0 conn(0x7fc92415dfb0 msgr2=0x7fc92406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:09.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 --2- 172.21.15.158:0/2914172323 >> v2:172.21.15.172:3300/0 conn(0x7fc92415dfb0 0x7fc92406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240cbe00 con 0x7fc9240cc5d0 2023-10-16T11:19:09.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc914001800 con 0x7fc9240cc5d0 2023-10-16T11:19:09.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc9293b8700 1 --2- 172.21.15.158:0/2914172323 >> v2:172.21.15.172:3300/0 conn(0x7fc92415dfb0 0x7fc92406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:09.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 >> v1:172.21.15.158:6789/0 conn(0x7fc9240cc5d0 legacy=0x7fc9240cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc910003a00 0x7fc910003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910004850 0x7fc910004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 0x7fc910005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92406fd30 con 0x7fc910003a00 2023-10-16T11:19:09.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240cf270 con 0x7fc910004850 2023-10-16T11:19:09.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91000ba10 con 0x7fc92406f340 2023-10-16T11:19:09.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc929bb9700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc910003a00 0x7fc910003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc9293b8700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910004850 0x7fc910004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc92a3ba700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 0x7fc910005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc929bb9700 1 -- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 msgr2=0x7fc910005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:09.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc929bb9700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc92406f340 0x7fc910005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc929bb9700 1 -- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910004850 msgr2=0x7fc910004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:09.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc929bb9700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910004850 0x7fc910004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc929bb9700 1 -- 172.21.15.158:0/2914172323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100035b0 con 0x7fc910003a00 2023-10-16T11:19:09.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc929bb9700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc910003a00 0x7fc910003e20 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c2f0 tx=0x7fc910005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1885cf6e0fd39a9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:09.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91400ce30 con 0x7fc910003a00 2023-10-16T11:19:09.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc914005960 con 0x7fc910003a00 2023-10-16T11:19:09.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.648+0000 7fc928bb7700 1 -- 172.21.15.158:0/2914172323 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914005d10 con 0x7fc910003a00 2023-10-16T11:19:09.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.649+0000 7fc92be1d700 1 -- 172.21.15.158:0/2914172323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc910003a00 msgr2=0x7fc910003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.649+0000 7fc92be1d700 1 --2- 172.21.15.158:0/2914172323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc910003a00 0x7fc910003e20 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c2f0 tx=0x7fc910005880 comp rx=0 tx=0).stop 2023-10-16T11:19:09.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.649+0000 7fc92a3ba700 1 -- 172.21.15.158:0/2914172323 reap_dead start 2023-10-16T11:19:09.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.649+0000 7fc92be1d700 1 -- 172.21.15.158:0/2914172323 shutdown_connections 2023-10-16T11:19:09.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.649+0000 7fc92be1d700 1 -- 172.21.15.158:0/2914172323 >> 172.21.15.158:0/2914172323 conn(0x7fc9240d02b0 msgr2=0x7fc9240d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:09.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.649+0000 7fc92be1d700 1 -- 172.21.15.158:0/2914172323 shutdown_connections 2023-10-16T11:19:09.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.649+0000 7fc92be1d700 1 -- 172.21.15.158:0/2914172323 wait complete. 2023-10-16T11:19:09.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.650+0000 7fc92be1d700 1 Processor -- start 2023-10-16T11:19:09.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.650+0000 7fc92be1d700 1 -- start start 2023-10-16T11:19:09.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92be1d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910003a00 0x7fc924153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92be1d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240c4c60 0x7fc9240ad270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92be1d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92406a8d0 0x7fc92406ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92be1d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92406b290 con 0x7fc92406a8d0 2023-10-16T11:19:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92be1d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240ae120 con 0x7fc910003a00 2023-10-16T11:19:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92be1d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240ae370 con 0x7fc9240c4c60 2023-10-16T11:19:09.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92a3ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92406a8d0 0x7fc92406ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92a3ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92406a8d0 0x7fc92406ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37580/0 (socket says 172.21.15.158:37580) 2023-10-16T11:19:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92a3ba700 1 -- 172.21.15.158:0/1638586262 learned_addr learned my addr 172.21.15.158:0/1638586262 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc9293b8700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240c4c60 0x7fc9240ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc929bb9700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910003a00 0x7fc924153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92a3ba700 1 -- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240c4c60 msgr2=0x7fc9240ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92a3ba700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240c4c60 0x7fc9240ad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92a3ba700 1 -- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910003a00 msgr2=0x7fc924153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.651+0000 7fc92a3ba700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910003a00 0x7fc924153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.652+0000 7fc92a3ba700 1 -- 172.21.15.158:0/1638586262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240ae650 con 0x7fc92406a8d0 2023-10-16T11:19:09.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.652+0000 7fc92a3ba700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92406a8d0 0x7fc92406ad50 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fc9240718c0 tx=0x7fc91c011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.652+0000 7fc91a7fc700 1 -- 172.21.15.158:0/1638586262 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00e040 con 0x7fc92406a8d0 2023-10-16T11:19:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.652+0000 7fc91a7fc700 1 -- 172.21.15.158:0/1638586262 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c00f040 con 0x7fc92406a8d0 2023-10-16T11:19:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.652+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9240c81e0 con 0x7fc92406a8d0 2023-10-16T11:19:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.652+0000 7fc91a7fc700 1 -- 172.21.15.158:0/1638586262 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00fbe0 con 0x7fc92406a8d0 2023-10-16T11:19:09.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.652+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9240c50e0 con 0x7fc92406a8d0 2023-10-16T11:19:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.653+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9240681b0 con 0x7fc92406a8d0 2023-10-16T11:19:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.653+0000 7fc91a7fc700 1 -- 172.21.15.158:0/1638586262 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc91c00fd40 con 0x7fc92406a8d0 2023-10-16T11:19:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.654+0000 7fc91a7fc700 1 -- 172.21.15.158:0/1638586262 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7399+0+0 (secure 0 0 0) 0x7fc91c089ed0 con 0x7fc92406a8d0 2023-10-16T11:19:09.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.658+0000 7fc91a7fc700 1 -- 172.21.15.158:0/1638586262 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc91c054cf0 con 0x7fc92406a8d0 2023-10-16T11:19:09.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.854+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9240b1780 con 0x7fc92406a8d0 2023-10-16T11:19:09.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.855+0000 7fc91a7fc700 1 -- 172.21.15.158:0/1638586262 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+18820 (secure 0 0 0) 0x7fc91c058310 con 0x7fc92406a8d0 2023-10-16T11:19:09.857 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:09.857 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":206,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:09.256745+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:17:38.499055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T11:17:39.035057+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":205,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.1b","mappings":[{"from":3,"to":0}]},{"pgid":"26.29","mappings":[{"from":1,"to":6}]},{"pgid":"26.3e","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/1772820648":"2023-10-17T11:19:09.256689+0000","172.21.15.158:6833/2839558305":"2023-10-17T11:19:09.256689+0000","172.21.15.158:6832/2839558305":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/1533815399":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/1067131743":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/1000153153":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/332114222":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/3542171936":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/786969372":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/2415272176":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6833/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6832/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/778612447":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/1354527416":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/3344401643":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/3047051699":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/4256533896":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/716693565":"2023-10-17T11:17:53.925811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:09.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92406a8d0 msgr2=0x7fc92406ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92406a8d0 0x7fc92406ad50 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fc9240718c0 tx=0x7fc91c011ed0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 shutdown_connections 2023-10-16T11:19:09.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc92406a8d0 0x7fc92406ad50 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc910003a00 0x7fc924153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 --2- 172.21.15.158:0/1638586262 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9240c4c60 0x7fc9240ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 >> 172.21.15.158:0/1638586262 conn(0x7fc9240d02b0 msgr2=0x7fc924073c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:09.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 shutdown_connections 2023-10-16T11:19:09.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.857+0000 7fc92be1d700 1 -- 172.21.15.158:0/1638586262 wait complete. 2023-10-16T11:19:09.872 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:19:09.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.991+0000 7fa3cdb90700 1 Processor -- start 2023-10-16T11:19:09.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.991+0000 7fa3cdb90700 1 -- start start 2023-10-16T11:19:09.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.992+0000 7fa3cdb90700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa3c80b3040 0x7fa3c80b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:09.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.992+0000 7fa3cdb90700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c815e150 con 0x7fa3c806bcc0 2023-10-16T11:19:09.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.992+0000 7fa3cdb90700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c815e430 con 0x7fa3c815d920 2023-10-16T11:19:09.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.992+0000 7fa3cdb90700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c80b3a00 con 0x7fa3c80b3040 2023-10-16T11:19:09.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.992+0000 7fa3c6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa3c80b3040 0x7fa3c80b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:09.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.992+0000 7fa3c6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa3c80b3040 0x7fa3c80b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37586/0 (socket says 172.21.15.158:37586) 2023-10-16T11:19:09.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.992+0000 7fa3c6ffd700 1 -- 172.21.15.158:0/3609891575 learned_addr learned my addr 172.21.15.158:0/3609891575 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:09.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3c6ffd700 1 -- 172.21.15.158:0/3609891575 >> v1:172.21.15.172:6789/0 conn(0x7fa3c815d920 legacy=0x7fa3c80cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:09.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3c6ffd700 1 -- 172.21.15.158:0/3609891575 >> v1:172.21.15.158:6789/0 conn(0x7fa3c806bcc0 legacy=0x7fa3c806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3c6ffd700 1 -- 172.21.15.158:0/3609891575 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c815e710 con 0x7fa3c80b3040 2023-10-16T11:19:09.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3c6ffd700 1 --2- 172.21.15.158:0/3609891575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c80b3040 0x7fa3c80b34c0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80098f0 tx=0x7fa3b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bc04e28ad9f8ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:09.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3c67fc700 1 -- 172.21.15.158:0/3609891575 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8004230 con 0x7fa3c80b3040 2023-10-16T11:19:09.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3c67fc700 1 -- 172.21.15.158:0/3609891575 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b8004390 con 0x7fa3c80b3040 2023-10-16T11:19:09.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3c67fc700 1 -- 172.21.15.158:0/3609891575 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b80036a0 con 0x7fa3c80b3040 2023-10-16T11:19:09.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3cdb90700 1 -- 172.21.15.158:0/3609891575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c80b3040 msgr2=0x7fa3c80b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:09.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3cdb90700 1 --2- 172.21.15.158:0/3609891575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c80b3040 0x7fa3c80b34c0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80098f0 tx=0x7fa3b8009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:09.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3cdb90700 1 -- 172.21.15.158:0/3609891575 shutdown_connections 2023-10-16T11:19:09.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3cdb90700 1 --2- 172.21.15.158:0/3609891575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c80b3040 0x7fa3c80b34c0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:09.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.993+0000 7fa3cdb90700 1 -- 172.21.15.158:0/3609891575 >> 172.21.15.158:0/3609891575 conn(0x7fa3c80ce040 msgr2=0x7fa3c80ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:09.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.994+0000 7fa3cdb90700 1 -- 172.21.15.158:0/3609891575 shutdown_connections 2023-10-16T11:19:10.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.994+0000 7fa3cdb90700 1 -- 172.21.15.158:0/3609891575 wait complete. 2023-10-16T11:19:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.995+0000 7fa3cdb90700 1 Processor -- start 2023-10-16T11:19:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.995+0000 7fa3cdb90700 1 -- start start 2023-10-16T11:19:10.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c806bcc0 0x7fa3c8158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3c80b3040 0x7fa3c80b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c815d920 0x7fa3c815bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80c8600 con 0x7fa3c806bcc0 2023-10-16T11:19:10.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80c8850 con 0x7fa3c815d920 2023-10-16T11:19:10.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80c8b30 con 0x7fa3c80b3040 2023-10-16T11:19:10.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c806bcc0 0x7fa3c8158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c806bcc0 0x7fa3c8158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37590/0 (socket says 172.21.15.158:37590) 2023-10-16T11:19:10.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 -- 172.21.15.158:0/1276871592 learned_addr learned my addr 172.21.15.158:0/1276871592 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:10.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c7fff700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c815d920 0x7fa3c815bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c6ffd700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3c80b3040 0x7fa3c80b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 -- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3c80b3040 msgr2=0x7fa3c80b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3c80b3040 0x7fa3c80b3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 -- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c815d920 msgr2=0x7fa3c815bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c815d920 0x7fa3c815bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 -- 172.21.15.158:0/1276871592 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8009550 con 0x7fa3c806bcc0 2023-10-16T11:19:10.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c7fff700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c815d920 0x7fa3c815bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:10.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3c77fe700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c806bcc0 0x7fa3c8158970 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00b470 tx=0x7fa3bc00b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:10.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3affff700 1 -- 172.21.15.158:0/1276871592 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc011830 con 0x7fa3c806bcc0 2023-10-16T11:19:10.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3affff700 1 -- 172.21.15.158:0/1276871592 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc011990 con 0x7fa3c806bcc0 2023-10-16T11:19:10.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c80c8e10 con 0x7fa3c806bcc0 2023-10-16T11:19:10.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3affff700 1 -- 172.21.15.158:0/1276871592 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc011d40 con 0x7fa3c806bcc0 2023-10-16T11:19:10.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.996+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c80c93d0 con 0x7fa3c806bcc0 2023-10-16T11:19:10.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.997+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c80681b0 con 0x7fa3c806bcc0 2023-10-16T11:19:10.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.998+0000 7fa3affff700 1 -- 172.21.15.158:0/1276871592 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa3bc01a400 con 0x7fa3c806bcc0 2023-10-16T11:19:10.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:09.998+0000 7fa3affff700 1 -- 172.21.15.158:0/1276871592 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7399+0+0 (secure 0 0 0) 0x7fa3bc089740 con 0x7fa3c806bcc0 2023-10-16T11:19:10.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.002+0000 7fa3affff700 1 -- 172.21.15.158:0/1276871592 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3bc0545a0 con 0x7fa3c806bcc0 2023-10-16T11:19:10.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.238+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa3c80c95c0 con 0x7fa3c806bcc0 2023-10-16T11:19:10.240 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:19:10.239+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:19:10.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.240+0000 7fa3affff700 1 -- 172.21.15.158:0/1276871592 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa3bc015090 con 0x7fa3c806bcc0 2023-10-16T11:19:10.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c806bcc0 msgr2=0x7fa3c8158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c806bcc0 0x7fa3c8158970 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00b470 tx=0x7fa3bc00b830 comp rx=0 tx=0).stop 2023-10-16T11:19:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 shutdown_connections 2023-10-16T11:19:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c815d920 0x7fa3c815bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3c806bcc0 0x7fa3c8158970 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 --2- 172.21.15.158:0/1276871592 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa3c80b3040 0x7fa3c80b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 >> 172.21.15.158:0/1276871592 conn(0x7fa3c80ce040 msgr2=0x7fa3c815a540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 shutdown_connections 2023-10-16T11:19:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.243+0000 7fa3cdb90700 1 -- 172.21.15.158:0/1276871592 wait complete. 2023-10-16T11:19:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:19:10.257 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:19:10.286 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:19:10.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.383+0000 7f0a6e00a700 1 Processor -- start 2023-10-16T11:19:10.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.383+0000 7f0a6e00a700 1 -- start start 2023-10-16T11:19:10.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a6e00a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a6e00a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680b4420 con 0x7f0a6815b6b0 2023-10-16T11:19:10.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a6e00a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680cbc20 con 0x7f0a680d0c60 2023-10-16T11:19:10.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a6e00a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6815ba90 con 0x7f0a680b3630 2023-10-16T11:19:10.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a677fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0a6815b6b0 0x7f0a680cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37794/0 (socket says 172.21.15.158:37794) 2023-10-16T11:19:10.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a677fe700 1 -- 172.21.15.158:0/4035352248 learned_addr learned my addr 172.21.15.158:0/4035352248 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:10.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a67fff700 1 --2- 172.21.15.158:0/4035352248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767182464 0 0) 0x7f0a680b4420 con 0x7f0a6815b6b0 2023-10-16T11:19:10.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a500035b0 con 0x7f0a6815b6b0 2023-10-16T11:19:10.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139475993 0 0) 0x7f0a680cbc20 con 0x7f0a680d0c60 2023-10-16T11:19:10.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a680b4420 con 0x7f0a680d0c60 2023-10-16T11:19:10.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a67fff700 1 -- 172.21.15.158:0/4035352248 >> v1:172.21.15.172:6789/0 conn(0x7f0a680d0c60 legacy=0x7f0a680b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782050801 0 0) 0x7f0a500035b0 con 0x7f0a6815b6b0 2023-10-16T11:19:10.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a67fff700 1 -- 172.21.15.158:0/4035352248 >> v1:172.21.15.158:6789/0 conn(0x7f0a6815b6b0 legacy=0x7f0a680cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.384+0000 7f0a67fff700 1 -- 172.21.15.158:0/4035352248 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a680cbf00 con 0x7f0a680b3630 2023-10-16T11:19:10.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a67fff700 1 --2- 172.21.15.158:0/4035352248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68166120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009dc0 tx=0x7f0a6000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5210600799e0f371 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:10.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000e070 con 0x7f0a680b3630 2023-10-16T11:19:10.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60003a20 con 0x7f0a680b3630 2023-10-16T11:19:10.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a667fc700 1 -- 172.21.15.158:0/4035352248 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58001670 con 0x7f0a680b3630 2023-10-16T11:19:10.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a6e00a700 1 -- 172.21.15.158:0/4035352248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 msgr2=0x7f0a68166120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/4035352248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68166120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009dc0 tx=0x7f0a6000b330 comp rx=0 tx=0).stop 2023-10-16T11:19:10.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a6e00a700 1 -- 172.21.15.158:0/4035352248 shutdown_connections 2023-10-16T11:19:10.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/4035352248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68166120 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a6e00a700 1 -- 172.21.15.158:0/4035352248 >> 172.21.15.158:0/4035352248 conn(0x7f0a680cdf20 msgr2=0x7f0a680ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:10.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a6e00a700 1 -- 172.21.15.158:0/4035352248 shutdown_connections 2023-10-16T11:19:10.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.385+0000 7f0a6e00a700 1 -- 172.21.15.158:0/4035352248 wait complete. 2023-10-16T11:19:10.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.386+0000 7f0a6e00a700 1 Processor -- start 2023-10-16T11:19:10.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a6e00a700 1 -- start start 2023-10-16T11:19:10.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a6e00a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a6e00a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a680d0c60 0x7f0a680ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a6e00a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6815b6b0 0x7f0a6806c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a6e00a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6815cf20 con 0x7f0a6815b6b0 2023-10-16T11:19:10.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a6e00a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6815cc40 con 0x7f0a680d0c60 2023-10-16T11:19:10.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a6e00a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6806bb80 con 0x7f0a680b3630 2023-10-16T11:19:10.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a67fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6815b6b0 0x7f0a6806c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a67fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6815b6b0 0x7f0a6806c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37628/0 (socket says 172.21.15.158:37628) 2023-10-16T11:19:10.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.387+0000 7f0a67fff700 1 -- 172.21.15.158:0/3072633512 learned_addr learned my addr 172.21.15.158:0/3072633512 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:10.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a677fe700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a66ffd700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a680d0c60 0x7f0a680ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a67fff700 1 -- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 msgr2=0x7f0a68158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a67fff700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a67fff700 1 -- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a680d0c60 msgr2=0x7f0a680ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:10.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a67fff700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a680d0c60 0x7f0a680ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:10.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a67fff700 1 -- 172.21.15.158:0/3072633512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60009a70 con 0x7f0a6815b6b0 2023-10-16T11:19:10.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a67fff700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6815b6b0 0x7f0a6806c660 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009d90 tx=0x7f0a60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:10.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a4ffff700 1 -- 172.21.15.158:0/3072633512 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600044c0 con 0x7f0a6815b6b0 2023-10-16T11:19:10.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a4ffff700 1 -- 172.21.15.158:0/3072633512 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60004620 con 0x7f0a6815b6b0 2023-10-16T11:19:10.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6806be60 con 0x7f0a6815b6b0 2023-10-16T11:19:10.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a4ffff700 1 -- 172.21.15.158:0/3072633512 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6001d470 con 0x7f0a6815b6b0 2023-10-16T11:19:10.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.388+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a68158cc0 con 0x7f0a6815b6b0 2023-10-16T11:19:10.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.389+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6806b9e0 con 0x7f0a6815b6b0 2023-10-16T11:19:10.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.390+0000 7f0a4ffff700 1 -- 172.21.15.158:0/3072633512 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0a6001d5d0 con 0x7f0a6815b6b0 2023-10-16T11:19:10.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.390+0000 7f0a4ffff700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0a5406ad50 0x7f0a5406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:10.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.390+0000 7f0a4ffff700 1 -- 172.21.15.158:0/3072633512 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 7485+0+0 (secure 0 0 0) 0x7f0a6008b580 con 0x7f0a6815b6b0 2023-10-16T11:19:10.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.390+0000 7f0a677fe700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0a5406ad50 0x7f0a5406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:10.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.390+0000 7f0a677fe700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0a5406ad50 0x7f0a5406d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a58000f60 tx=0x7f0a58001690 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:10.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.396+0000 7f0a4ffff700 1 -- 172.21.15.158:0/3072633512 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a6806b9e0 con 0x7f0a6815b6b0 2023-10-16T11:19:10.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:10.632+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0a6806c380 con 0x7f0a6815b6b0 2023-10-16T11:19:11.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.272+0000 7f0a4ffff700 1 -- 172.21.15.158:0/3072633512 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v105) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0a6806c380 con 0x7f0a6815b6b0 2023-10-16T11:19:11.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0a5406ad50 msgr2=0x7f0a5406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:11.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0a5406ad50 0x7f0a5406d210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a58000f60 tx=0x7f0a58001690 comp rx=0 tx=0).stop 2023-10-16T11:19:11.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6815b6b0 msgr2=0x7f0a6806c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:11.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6815b6b0 0x7f0a6806c660 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009d90 tx=0x7f0a60004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 shutdown_connections 2023-10-16T11:19:11.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0a5406ad50 0x7f0a5406d210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6815b6b0 0x7f0a6806c660 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a680b3630 0x7f0a68158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 --2- 172.21.15.158:0/3072633512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a680d0c60 0x7f0a680ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 >> 172.21.15.158:0/3072633512 conn(0x7f0a680cdf20 msgr2=0x7f0a680d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:11.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 shutdown_connections 2023-10-16T11:19:11.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.275+0000 7f0a6e00a700 1 -- 172.21.15.158:0/3072633512 wait complete. 2023-10-16T11:19:11.291 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:19:11.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.422+0000 7f5776a13700 1 Processor -- start 2023-10-16T11:19:11.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.422+0000 7f5776a13700 1 -- start start 2023-10-16T11:19:11.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f5776a13700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f577015da40 0x7f57700caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:11.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f5776a13700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57700cc590 0x7f577006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:11.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f5776a13700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700cbd30 con 0x7f577006ae10 2023-10-16T11:19:11.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f5776a13700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f577006b2c0 con 0x7f577015da40 2023-10-16T11:19:11.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f5776a13700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57700cc1c0 con 0x7f57700cc590 2023-10-16T11:19:11.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f5774fb0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f577006ae10 0x7f57700ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37830/0 (socket says 172.21.15.158:37830) 2023-10-16T11:19:11.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f5774fb0700 1 -- 172.21.15.158:0/4005640691 learned_addr learned my addr 172.21.15.158:0/4005640691 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:11.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576f7fe700 1 --2- 172.21.15.158:0/4005640691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57700cc590 0x7f577006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:11.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576ffff700 1 --2- 172.21.15.158:0/4005640691 >> v2:172.21.15.172:3300/0 conn(0x7f577015da40 0x7f57700caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:11.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576effd700 1 -- 172.21.15.158:0/4005640691 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424157825 0 0) 0x7f57700cbd30 con 0x7f577006ae10 2023-10-16T11:19:11.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576effd700 1 -- 172.21.15.158:0/4005640691 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57580035b0 con 0x7f577006ae10 2023-10-16T11:19:11.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576f7fe700 1 -- 172.21.15.158:0/4005640691 >> v2:172.21.15.172:3300/0 conn(0x7f577015da40 msgr2=0x7f57700caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:11.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576f7fe700 1 --2- 172.21.15.158:0/4005640691 >> v2:172.21.15.172:3300/0 conn(0x7f577015da40 0x7f57700caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576f7fe700 1 -- 172.21.15.158:0/4005640691 >> v1:172.21.15.158:6789/0 conn(0x7f577006ae10 legacy=0x7f57700ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:11.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.423+0000 7f576f7fe700 1 -- 172.21.15.158:0/4005640691 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577015de20 con 0x7f57700cc590 2023-10-16T11:19:11.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f576f7fe700 1 --2- 172.21.15.158:0/4005640691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57700cc590 0x7f577006a8d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5760009840 tx=0x7f5760009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9282745f1b44a249 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:11.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f576effd700 1 -- 172.21.15.158:0/4005640691 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760011830 con 0x7f57700cc590 2023-10-16T11:19:11.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f576effd700 1 -- 172.21.15.158:0/4005640691 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5760011990 con 0x7f57700cc590 2023-10-16T11:19:11.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f576effd700 1 -- 172.21.15.158:0/4005640691 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760011d00 con 0x7f57700cc590 2023-10-16T11:19:11.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 -- 172.21.15.158:0/4005640691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57700cc590 msgr2=0x7f577006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:11.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 --2- 172.21.15.158:0/4005640691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57700cc590 0x7f577006a8d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5760009840 tx=0x7f5760009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:11.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 -- 172.21.15.158:0/4005640691 shutdown_connections 2023-10-16T11:19:11.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 --2- 172.21.15.158:0/4005640691 >> v2:172.21.15.172:3300/0 conn(0x7f577015da40 0x7f57700caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 --2- 172.21.15.158:0/4005640691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57700cc590 0x7f577006a8d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 -- 172.21.15.158:0/4005640691 >> 172.21.15.158:0/4005640691 conn(0x7f5770099a10 msgr2=0x7f57700d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:11.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 -- 172.21.15.158:0/4005640691 shutdown_connections 2023-10-16T11:19:11.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.424+0000 7f5776a13700 1 -- 172.21.15.158:0/4005640691 wait complete. 2023-10-16T11:19:11.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.425+0000 7f5776a13700 1 Processor -- start 2023-10-16T11:19:11.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.426+0000 7f5776a13700 1 -- start start 2023-10-16T11:19:11.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.426+0000 7f5776a13700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f577006ae10 0x7f577015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:11.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.426+0000 7f5776a13700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57700cc590 0x7f57700c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:11.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.426+0000 7f5776a13700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f577015da40 0x7f577006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:11.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.426+0000 7f5776a13700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57700c93c0 con 0x7f577015da40 2023-10-16T11:19:11.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.426+0000 7f5776a13700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57700c7f50 con 0x7f57700cc590 2023-10-16T11:19:11.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.426+0000 7f5776a13700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57700c7620 con 0x7f577006ae10 2023-10-16T11:19:11.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f577015da40 0x7f577006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:11.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f577015da40 0x7f577006e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37658/0 (socket says 172.21.15.158:37658) 2023-10-16T11:19:11.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 -- 172.21.15.158:0/3931839374 learned_addr learned my addr 172.21.15.158:0/3931839374 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:11.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f576ffff700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f577006ae10 0x7f577015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:11.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f576f7fe700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57700cc590 0x7f57700c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:11.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 -- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f577006ae10 msgr2=0x7f577015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:11.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f577006ae10 0x7f577015ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 -- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57700cc590 msgr2=0x7f57700c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:11.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57700cc590 0x7f57700c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:11.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 -- 172.21.15.158:0/3931839374 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57600094a0 con 0x7f577015da40 2023-10-16T11:19:11.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5774fb0700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f577015da40 0x7f577006e000 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f576800cb40 tx=0x7f576800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:11.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576800ab50 con 0x7f577015da40 2023-10-16T11:19:11.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576800acb0 con 0x7f577015da40 2023-10-16T11:19:11.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57700c7900 con 0x7f577015da40 2023-10-16T11:19:11.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768012620 con 0x7f577015da40 2023-10-16T11:19:11.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.427+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57701f6180 con 0x7f577015da40 2023-10-16T11:19:11.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.428+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57700c80c0 con 0x7f577015da40 2023-10-16T11:19:11.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.429+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f576801b3b0 con 0x7f577015da40 2023-10-16T11:19:11.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.429+0000 7f576cff9700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f575c06ad50 0x7f575c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:11.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.429+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 7485+0+0 (secure 0 0 0) 0x7f5768089720 con 0x7f577015da40 2023-10-16T11:19:11.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.429+0000 7f576ffff700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f575c06ad50 0x7f575c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:11.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.429+0000 7f576ffff700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f575c06ad50 0x7f575c06d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f577009b210 tx=0x7f5764008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:11.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.434+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5768054420 con 0x7f577015da40 2023-10-16T11:19:11.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:11.626+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57700681b0 con 0x7f577015da40 2023-10-16T11:19:12.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.276+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 136) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f57680547f0 con 0x7f577015da40 2023-10-16T11:19:12.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.278+0000 7f576cff9700 1 -- 172.21.15.158:0/3931839374 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v208) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5768077390 con 0x7f577015da40 2023-10-16T11:19:12.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f575c06ad50 msgr2=0x7f575c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f575c06ad50 0x7f575c06d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f577009b210 tx=0x7f5764008040 comp rx=0 tx=0).stop 2023-10-16T11:19:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f577015da40 msgr2=0x7f577006e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f577015da40 0x7f577006e000 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f576800cb40 tx=0x7f576800cf00 comp rx=0 tx=0).stop 2023-10-16T11:19:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 shutdown_connections 2023-10-16T11:19:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f575c06ad50 0x7f575c06d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f577015da40 0x7f577006e000 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f577006ae10 0x7f577015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 --2- 172.21.15.158:0/3931839374 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57700cc590 0x7f57700c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.281+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 >> 172.21.15.158:0/3931839374 conn(0x7f5770099a10 msgr2=0x7f57700ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.282+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 shutdown_connections 2023-10-16T11:19:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.282+0000 7f5776a13700 1 -- 172.21.15.158:0/3931839374 wait complete. 2023-10-16T11:19:12.286 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:19:12.302 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:19:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.429+0000 7f6fd1f22700 1 Processor -- start 2023-10-16T11:19:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.430+0000 7f6fd1f22700 1 -- start start 2023-10-16T11:19:12.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.430+0000 7f6fd1f22700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6fcc15dad0 0x7f6fcc0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:12.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.430+0000 7f6fd1f22700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6fcc15deb0 0x7f6fcc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:12.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.430+0000 7f6fd1f22700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc0d0ae0 con 0x7f6fcc0cea20 2023-10-16T11:19:12.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.430+0000 7f6fd1f22700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fcc0cee50 con 0x7f6fcc15deb0 2023-10-16T11:19:12.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.430+0000 7f6fd1f22700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fcc0d12d0 con 0x7f6fcc15dad0 2023-10-16T11:19:12.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.430+0000 7f6fcaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6fcc15deb0 0x7f6fcc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:12.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6fcc15deb0 0x7f6fcc0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37666/0 (socket says 172.21.15.158:37666) 2023-10-16T11:19:12.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcaffd700 1 -- 172.21.15.158:0/3113880401 learned_addr learned my addr 172.21.15.158:0/3113880401 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:12.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcb7fe700 1 --2- 172.21.15.158:0/3113880401 >> v2:172.21.15.172:3300/0 conn(0x7f6fcc15dad0 0x7f6fcc0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:12.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcaffd700 1 -- 172.21.15.158:0/3113880401 >> v2:172.21.15.172:3300/0 conn(0x7f6fcc15dad0 msgr2=0x7f6fcc0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:12.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcaffd700 1 --2- 172.21.15.158:0/3113880401 >> v2:172.21.15.172:3300/0 conn(0x7f6fcc15dad0 0x7f6fcc0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcaffd700 1 -- 172.21.15.158:0/3113880401 >> v1:172.21.15.172:6789/0 conn(0x7f6fcc0cea20 legacy=0x7f6fcc0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:12.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcaffd700 1 -- 172.21.15.158:0/3113880401 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc0cbdf0 con 0x7f6fcc15deb0 2023-10-16T11:19:12.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcb7fe700 1 --2- 172.21.15.158:0/3113880401 >> v2:172.21.15.172:3300/0 conn(0x7f6fcc15dad0 0x7f6fcc0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:12.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fcaffd700 1 --2- 172.21.15.158:0/3113880401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15deb0 0x7f6fcc0ce4e0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000bb90 tx=0x7f6fc000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e595dfc40ed7b84a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:12.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fca7fc700 1 -- 172.21.15.158:0/3113880401 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc000dd10 con 0x7f6fcc15deb0 2023-10-16T11:19:12.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fca7fc700 1 -- 172.21.15.158:0/3113880401 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc000de70 con 0x7f6fcc15deb0 2023-10-16T11:19:12.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.431+0000 7f6fca7fc700 1 -- 172.21.15.158:0/3113880401 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0010670 con 0x7f6fcc15deb0 2023-10-16T11:19:12.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 -- 172.21.15.158:0/3113880401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15deb0 msgr2=0x7f6fcc0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:12.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/3113880401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15deb0 0x7f6fcc0ce4e0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000bb90 tx=0x7f6fc000bea0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 -- 172.21.15.158:0/3113880401 shutdown_connections 2023-10-16T11:19:12.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/3113880401 >> v2:172.21.15.172:3300/0 conn(0x7f6fcc15dad0 0x7f6fcc0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/3113880401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15deb0 0x7f6fcc0ce4e0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 -- 172.21.15.158:0/3113880401 >> 172.21.15.158:0/3113880401 conn(0x7f6fcc099a10 msgr2=0x7f6fcc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:12.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 -- 172.21.15.158:0/3113880401 shutdown_connections 2023-10-16T11:19:12.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.432+0000 7f6fd1f22700 1 -- 172.21.15.158:0/3113880401 wait complete. 2023-10-16T11:19:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.433+0000 7f6fd1f22700 1 Processor -- start 2023-10-16T11:19:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.433+0000 7f6fd1f22700 1 -- start start 2023-10-16T11:19:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fd1f22700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fcc0cea20 0x7f6fcc159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fd1f22700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15dad0 0x7f6fcc0b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:12.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fd1f22700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fcc15deb0 0x7f6fcc06e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:12.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fd1f22700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc0b4010 con 0x7f6fcc15dad0 2023-10-16T11:19:12.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fd1f22700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc159b80 con 0x7f6fcc15deb0 2023-10-16T11:19:12.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fd1f22700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc0b4260 con 0x7f6fcc0cea20 2023-10-16T11:19:12.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15dad0 0x7f6fcc0b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15dad0 0x7f6fcc0b58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37672/0 (socket says 172.21.15.158:37672) 2023-10-16T11:19:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 -- 172.21.15.158:0/307728048 learned_addr learned my addr 172.21.15.158:0/307728048 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcb7fe700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fcc0cea20 0x7f6fcc159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcbfff700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fcc15deb0 0x7f6fcc06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:12.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 -- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fcc0cea20 msgr2=0x7f6fcc159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fcc0cea20 0x7f6fcc159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 -- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fcc15deb0 msgr2=0x7f6fcc06e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fcc15deb0 0x7f6fcc06e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 -- 172.21.15.158:0/307728048 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc000b7f0 con 0x7f6fcc15dad0 2023-10-16T11:19:12.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.434+0000 7f6fcaffd700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15dad0 0x7f6fcc0b58d0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc070430 tx=0x7f6fc0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.435+0000 7f6fb3fff700 1 -- 172.21.15.158:0/307728048 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0003bf0 con 0x7f6fcc15dad0 2023-10-16T11:19:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.435+0000 7f6fb3fff700 1 -- 172.21.15.158:0/307728048 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc0003d50 con 0x7f6fcc15dad0 2023-10-16T11:19:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.435+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fcc0b4540 con 0x7f6fcc15dad0 2023-10-16T11:19:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.435+0000 7f6fb3fff700 1 -- 172.21.15.158:0/307728048 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00106b0 con 0x7f6fcc15dad0 2023-10-16T11:19:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.435+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fcc0b4b00 con 0x7f6fcc15dad0 2023-10-16T11:19:12.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.436+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fcc0681b0 con 0x7f6fcc15dad0 2023-10-16T11:19:12.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.436+0000 7f6fb3fff700 1 -- 172.21.15.158:0/307728048 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 136) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6fc0019400 con 0x7f6fcc15dad0 2023-10-16T11:19:12.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.436+0000 7f6fb3fff700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f6fb806af90 0x7f6fb806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:12.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.437+0000 7f6fb3fff700 1 -- 172.21.15.158:0/307728048 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f6fc008b1b0 con 0x7f6fcc15dad0 2023-10-16T11:19:12.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.437+0000 7f6fcb7fe700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f6fb806af90 0x7f6fb806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:12.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.437+0000 7f6fcb7fe700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f6fb806af90 0x7f6fb806d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0099d0 tx=0x7f6fbc008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:12.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.441+0000 7f6fb3fff700 1 -- 172.21.15.158:0/307728048 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fc0055d60 con 0x7f6fcc15dad0 2023-10-16T11:19:12.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:12.634+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6fcc0ce680 con 0x7f6fcc15dad0 2023-10-16T11:19:13.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.280+0000 7f6fb3fff700 1 -- 172.21.15.158:0/307728048 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v209) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6fc0059380 con 0x7f6fcc15dad0 2023-10-16T11:19:13.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f6fb806af90 msgr2=0x7f6fb806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f6fb806af90 0x7f6fb806d450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0099d0 tx=0x7f6fbc008040 comp rx=0 tx=0).stop 2023-10-16T11:19:13.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15dad0 msgr2=0x7f6fcc0b58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15dad0 0x7f6fcc0b58d0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc070430 tx=0x7f6fc0004750 comp rx=0 tx=0).stop 2023-10-16T11:19:13.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 shutdown_connections 2023-10-16T11:19:13.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fcc15deb0 0x7f6fcc06e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fcc0cea20 0x7f6fcc159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f6fb806af90 0x7f6fb806d450 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 --2- 172.21.15.158:0/307728048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fcc15dad0 0x7f6fcc0b58d0 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.284+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 >> 172.21.15.158:0/307728048 conn(0x7f6fcc099a10 msgr2=0x7f6fcc0c62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:13.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.285+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 shutdown_connections 2023-10-16T11:19:13.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.285+0000 7f6fd1f22700 1 -- 172.21.15.158:0/307728048 wait complete. 2023-10-16T11:19:13.289 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:19:13.303 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:19:13.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.428+0000 7fe1ea011700 1 Processor -- start 2023-10-16T11:19:13.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.428+0000 7fe1ea011700 1 -- start start 2023-10-16T11:19:13.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1ea011700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1e415b630 0x7fe1e40cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1ea011700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe1e40cdab0 0x7fe1e40b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1ea011700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40d0e70 0x7fe1e40b1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1ea011700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e40cc4d0 con 0x7fe1e40cdab0 2023-10-16T11:19:13.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1ea011700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e40cc7b0 con 0x7fe1e415b630 2023-10-16T11:19:13.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1ea011700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40cca90 con 0x7fe1e40d0e70 2023-10-16T11:19:13.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe1e40cdab0 0x7fe1e40b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe1e40cdab0 0x7fe1e40b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37678/0 (socket says 172.21.15.158:37678) 2023-10-16T11:19:13.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 -- 172.21.15.158:0/537502639 learned_addr learned my addr 172.21.15.158:0/537502639 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:13.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e37fe700 1 --2- 172.21.15.158:0/537502639 >> v2:172.21.15.172:3300/0 conn(0x7fe1e415b630 0x7fe1e40cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e3fff700 1 --2- 172.21.15.158:0/537502639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40d0e70 0x7fe1e40b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 -- 172.21.15.158:0/537502639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40d0e70 msgr2=0x7fe1e40b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:13.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 --2- 172.21.15.158:0/537502639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40d0e70 0x7fe1e40b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 -- 172.21.15.158:0/537502639 >> v2:172.21.15.172:3300/0 conn(0x7fe1e415b630 msgr2=0x7fe1e40cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:13.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 --2- 172.21.15.158:0/537502639 >> v2:172.21.15.172:3300/0 conn(0x7fe1e415b630 0x7fe1e40cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.429+0000 7fe1e2ffd700 1 -- 172.21.15.158:0/537502639 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e40ccd70 con 0x7fe1e40cdab0 2023-10-16T11:19:13.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1e2ffd700 1 --2- 172.21.15.158:0/537502639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e40cdab0 0x7fe1e40b12e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fe1e40d0320 tx=0x7fe1d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcf1a2964432773 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:13.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1e27fc700 1 -- 172.21.15.158:0/537502639 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400b870 con 0x7fe1e40cdab0 2023-10-16T11:19:13.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1e27fc700 1 -- 172.21.15.158:0/537502639 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400b9d0 con 0x7fe1e40cdab0 2023-10-16T11:19:13.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1e27fc700 1 -- 172.21.15.158:0/537502639 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400bd60 con 0x7fe1e40cdab0 2023-10-16T11:19:13.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 -- 172.21.15.158:0/537502639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e40cdab0 msgr2=0x7fe1e40b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 --2- 172.21.15.158:0/537502639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e40cdab0 0x7fe1e40b12e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fe1e40d0320 tx=0x7fe1d4009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 -- 172.21.15.158:0/537502639 shutdown_connections 2023-10-16T11:19:13.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 --2- 172.21.15.158:0/537502639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40d0e70 0x7fe1e40b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 --2- 172.21.15.158:0/537502639 >> v2:172.21.15.172:3300/0 conn(0x7fe1e415b630 0x7fe1e40cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 --2- 172.21.15.158:0/537502639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e40cdab0 0x7fe1e40b12e0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 -- 172.21.15.158:0/537502639 >> 172.21.15.158:0/537502639 conn(0x7fe1e4099a10 msgr2=0x7fe1e40d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:13.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 -- 172.21.15.158:0/537502639 shutdown_connections 2023-10-16T11:19:13.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.430+0000 7fe1ea011700 1 -- 172.21.15.158:0/537502639 wait complete. 2023-10-16T11:19:13.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.431+0000 7fe1ea011700 1 Processor -- start 2023-10-16T11:19:13.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.432+0000 7fe1ea011700 1 -- start start 2023-10-16T11:19:13.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.432+0000 7fe1ea011700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40cdab0 0x7fe1e41f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.432+0000 7fe1ea011700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1e40d0e70 0x7fe1e41517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.432+0000 7fe1ea011700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e415b630 0x7fe1e40ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.432+0000 7fe1ea011700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40ce0a0 con 0x7fe1e415b630 2023-10-16T11:19:13.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.432+0000 7fe1ea011700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40cfcd0 con 0x7fe1e40d0e70 2023-10-16T11:19:13.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.432+0000 7fe1ea011700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40ce340 con 0x7fe1e40cdab0 2023-10-16T11:19:13.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e415b630 0x7fe1e40ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e415b630 0x7fe1e40ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37684/0 (socket says 172.21.15.158:37684) 2023-10-16T11:19:13.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 -- 172.21.15.158:0/2552592921 learned_addr learned my addr 172.21.15.158:0/2552592921 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:13.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e37fe700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40cdab0 0x7fe1e41f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e2ffd700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1e40d0e70 0x7fe1e41517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 -- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40cdab0 msgr2=0x7fe1e41f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40cdab0 0x7fe1e41f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 -- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1e40d0e70 msgr2=0x7fe1e41517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1e40d0e70 0x7fe1e41517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 -- 172.21.15.158:0/2552592921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4009550 con 0x7fe1e415b630 2023-10-16T11:19:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1e3fff700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e415b630 0x7fe1e40ce860 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc007e30 tx=0x7fe1dc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1dbfff700 1 -- 172.21.15.158:0/2552592921 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc00ed30 con 0x7fe1e415b630 2023-10-16T11:19:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1dbfff700 1 -- 172.21.15.158:0/2552592921 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc00ee90 con 0x7fe1e415b630 2023-10-16T11:19:13.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e415bb90 con 0x7fe1e415b630 2023-10-16T11:19:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1dbfff700 1 -- 172.21.15.158:0/2552592921 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc014660 con 0x7fe1e415b630 2023-10-16T11:19:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.433+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e41524b0 con 0x7fe1e415b630 2023-10-16T11:19:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.434+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e40681b0 con 0x7fe1e415b630 2023-10-16T11:19:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.435+0000 7fe1dbfff700 1 -- 172.21.15.158:0/2552592921 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 136) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe1dc0147c0 con 0x7fe1e415b630 2023-10-16T11:19:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.435+0000 7fe1dbfff700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe1c406af90 0x7fe1c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.435+0000 7fe1dbfff700 1 -- 172.21.15.158:0/2552592921 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fe1dc089f60 con 0x7fe1e415b630 2023-10-16T11:19:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.435+0000 7fe1e37fe700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe1c406af90 0x7fe1c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.435+0000 7fe1e37fe700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe1c406af90 0x7fe1c406d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006b20 tx=0x7fe1cc008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:13.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.439+0000 7fe1dbfff700 1 -- 172.21.15.158:0/2552592921 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1dc0bcfe0 con 0x7fe1e415b630 2023-10-16T11:19:13.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.630+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe1e4159ee0 con 0x7fe1e415b630 2023-10-16T11:19:13.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.633+0000 7fe1dbfff700 1 -- 172.21.15.158:0/2552592921 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe1dc054cf0 con 0x7fe1e415b630 2023-10-16T11:19:13.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe1c406af90 msgr2=0x7fe1c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe1c406af90 0x7fe1c406d450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006b20 tx=0x7fe1cc008040 comp rx=0 tx=0).stop 2023-10-16T11:19:13.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e415b630 msgr2=0x7fe1e40ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e415b630 0x7fe1e40ce860 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc007e30 tx=0x7fe1dc00e3c0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 shutdown_connections 2023-10-16T11:19:13.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe1c406af90 0x7fe1c406d450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e415b630 0x7fe1e40ce860 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1e40cdab0 0x7fe1e41f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 --2- 172.21.15.158:0/2552592921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1e40d0e70 0x7fe1e41517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 >> 172.21.15.158:0/2552592921 conn(0x7fe1e4099a10 msgr2=0x7fe1e40c6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:13.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 shutdown_connections 2023-10-16T11:19:13.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.636+0000 7fe1ea011700 1 -- 172.21.15.158:0/2552592921 wait complete. 2023-10-16T11:19:13.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:19:13.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.775+0000 7fb06830a700 1 Processor -- start 2023-10-16T11:19:13.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.775+0000 7fb06830a700 1 -- start start 2023-10-16T11:19:13.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb06830a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb060151600 0x7fb0600c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb06830a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0600cf930 con 0x7fb060151220 2023-10-16T11:19:13.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb06830a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0600cf650 con 0x7fb0600cf190 2023-10-16T11:19:13.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb06830a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0600c51b0 con 0x7fb060151600 2023-10-16T11:19:13.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0668a7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb060151600 0x7fb0600c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0668a7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb060151600 0x7fb0600c5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37692/0 (socket says 172.21.15.158:37692) 2023-10-16T11:19:13.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0668a7700 1 -- 172.21.15.158:0/1342292395 learned_addr learned my addr 172.21.15.158:0/1342292395 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:13.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0650a4700 1 -- 172.21.15.158:0/1342292395 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130595497 0 0) 0x7fb0600cf930 con 0x7fb060151220 2023-10-16T11:19:13.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0650a4700 1 -- 172.21.15.158:0/1342292395 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c0035b0 con 0x7fb060151220 2023-10-16T11:19:13.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0650a4700 1 -- 172.21.15.158:0/1342292395 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537121141 0 0) 0x7fb0600cf650 con 0x7fb0600cf190 2023-10-16T11:19:13.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0650a4700 1 -- 172.21.15.158:0/1342292395 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0600cf930 con 0x7fb0600cf190 2023-10-16T11:19:13.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0668a7700 1 -- 172.21.15.158:0/1342292395 >> v1:172.21.15.172:6789/0 conn(0x7fb0600cf190 legacy=0x7fb06006d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0668a7700 1 -- 172.21.15.158:0/1342292395 >> v1:172.21.15.158:6789/0 conn(0x7fb060151220 legacy=0x7fb0600665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.777+0000 7fb0668a7700 1 -- 172.21.15.158:0/1342292395 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0600c5490 con 0x7fb060151600 2023-10-16T11:19:13.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.778+0000 7fb0668a7700 1 --2- 172.21.15.158:0/1342292395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb060151600 0x7fb0600c5c90 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fb058009e80 tx=0x7fb05800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ee819b774463a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:13.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.778+0000 7fb0650a4700 1 -- 172.21.15.158:0/1342292395 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800e070 con 0x7fb060151600 2023-10-16T11:19:13.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.778+0000 7fb0650a4700 1 -- 172.21.15.158:0/1342292395 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05800b8d0 con 0x7fb060151600 2023-10-16T11:19:13.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.778+0000 7fb0650a4700 1 -- 172.21.15.158:0/1342292395 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800bc80 con 0x7fb060151600 2023-10-16T11:19:13.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.779+0000 7fb06830a700 1 -- 172.21.15.158:0/1342292395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb060151600 msgr2=0x7fb0600c5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.779+0000 7fb06830a700 1 --2- 172.21.15.158:0/1342292395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb060151600 0x7fb0600c5c90 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fb058009e80 tx=0x7fb05800b330 comp rx=0 tx=0).stop 2023-10-16T11:19:13.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.779+0000 7fb06830a700 1 -- 172.21.15.158:0/1342292395 shutdown_connections 2023-10-16T11:19:13.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.779+0000 7fb06830a700 1 --2- 172.21.15.158:0/1342292395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb060151600 0x7fb0600c5c90 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.779+0000 7fb06830a700 1 -- 172.21.15.158:0/1342292395 >> 172.21.15.158:0/1342292395 conn(0x7fb0600cc5b0 msgr2=0x7fb0600cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:13.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.779+0000 7fb06830a700 1 -- 172.21.15.158:0/1342292395 shutdown_connections 2023-10-16T11:19:13.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.779+0000 7fb06830a700 1 -- 172.21.15.158:0/1342292395 wait complete. 2023-10-16T11:19:13.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.780+0000 7fb06830a700 1 Processor -- start 2023-10-16T11:19:13.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.780+0000 7fb06830a700 1 -- start start 2023-10-16T11:19:13.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb06830a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0600cf190 0x7fb060145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb06830a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb060151220 0x7fb060159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb06830a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb060151600 0x7fb060146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb06830a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06012e010 con 0x7fb0600cf190 2023-10-16T11:19:13.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb06830a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06012d400 con 0x7fb060151600 2023-10-16T11:19:13.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb06830a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06012d6e0 con 0x7fb060151220 2023-10-16T11:19:13.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb0660a6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0600cf190 0x7fb060145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb0660a6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0600cf190 0x7fb060145450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37694/0 (socket says 172.21.15.158:37694) 2023-10-16T11:19:13.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb0660a6700 1 -- 172.21.15.158:0/2871404353 learned_addr learned my addr 172.21.15.158:0/2871404353 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:13.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb0658a5700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb060151220 0x7fb060159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.781+0000 7fb0668a7700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb060151600 0x7fb060146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb0660a6700 1 -- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb060151220 msgr2=0x7fb060159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb0660a6700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb060151220 0x7fb060159690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb0660a6700 1 -- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb060151600 msgr2=0x7fb060146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb0660a6700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb060151600 0x7fb060146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb0660a6700 1 -- 172.21.15.158:0/2871404353 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058009b30 con 0x7fb0600cf190 2023-10-16T11:19:13.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb0660a6700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0600cf190 0x7fb060145450 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0010e0 tx=0x7fb05c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:13.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0052d0 con 0x7fb0600cf190 2023-10-16T11:19:13.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c005430 con 0x7fb0600cf190 2023-10-16T11:19:13.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0038a0 con 0x7fb0600cf190 2023-10-16T11:19:13.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06012d9c0 con 0x7fb0600cf190 2023-10-16T11:19:13.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.782+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb060144790 con 0x7fb0600cf190 2023-10-16T11:19:13.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.783+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0600681b0 con 0x7fb0600cf190 2023-10-16T11:19:13.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.784+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 136) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb05c003a00 con 0x7fb0600cf190 2023-10-16T11:19:13.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.784+0000 7fb056ffd700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb04406af90 0x7fb04406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:13.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.784+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb05c088ed0 con 0x7fb0600cf190 2023-10-16T11:19:13.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.784+0000 7fb0658a5700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb04406af90 0x7fb04406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:13.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.784+0000 7fb0658a5700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb04406af90 0x7fb04406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb060130050 tx=0x7fb050009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:13.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.788+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb05c053c70 con 0x7fb0600cf190 2023-10-16T11:19:13.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.823+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb05c054040 con 0x7fb0600cf190 2023-10-16T11:19:13.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.983+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb0600723f0 con 0x7fb0600cf190 2023-10-16T11:19:13.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.986+0000 7fb056ffd700 1 -- 172.21.15.158:0/2871404353 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb05c083400 con 0x7fb0600cf190 2023-10-16T11:19:13.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb04406af90 msgr2=0x7fb04406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb04406af90 0x7fb04406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb060130050 tx=0x7fb050009040 comp rx=0 tx=0).stop 2023-10-16T11:19:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0600cf190 msgr2=0x7fb060145450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0600cf190 0x7fb060145450 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0010e0 tx=0x7fb05c00ca90 comp rx=0 tx=0).stop 2023-10-16T11:19:13.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 shutdown_connections 2023-10-16T11:19:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb060151600 0x7fb060146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb04406af90 0x7fb04406d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0600cf190 0x7fb060145450 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 --2- 172.21.15.158:0/2871404353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb060151220 0x7fb060159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:13.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 >> 172.21.15.158:0/2871404353 conn(0x7fb0600cc5b0 msgr2=0x7fb0600cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:13.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 shutdown_connections 2023-10-16T11:19:13.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:13.989+0000 7fb06830a700 1 -- 172.21.15.158:0/2871404353 wait complete. 2023-10-16T11:19:14.002 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:19:14.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.123+0000 7f3b2c935700 1 Processor -- start 2023-10-16T11:19:14.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.123+0000 7f3b2c935700 1 -- start start 2023-10-16T11:19:14.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.124+0000 7f3b2c935700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3b2806f180 0x7f3b2806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.124+0000 7f3b2c935700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2806fea0 0x7f3b2806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.124+0000 7f3b2c935700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2815ee60 con 0x7f3b2815b540 2023-10-16T11:19:14.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.124+0000 7f3b2c935700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2815b920 con 0x7f3b2806f180 2023-10-16T11:19:14.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.124+0000 7f3b2c935700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2815e2e0 con 0x7f3b2806fea0 2023-10-16T11:19:14.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b26d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2806fea0 0x7f3b2806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b25d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3b2806f180 0x7f3b2806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b26d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2806fea0 0x7f3b2806ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:48540/0 (socket says 172.21.15.158:48540) 2023-10-16T11:19:14.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b26d9d700 1 -- 172.21.15.158:0/3261225129 learned_addr learned my addr 172.21.15.158:0/3261225129 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:14.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b2659c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3b2815b540 0x7f3b2806d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:41782/0 (socket says 172.21.15.158:41782) 2023-10-16T11:19:14.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b25d9b700 1 -- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2806fea0 msgr2=0x7f3b2806ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:14.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b25d9b700 1 --2- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2806fea0 0x7f3b2806ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b25d9b700 1 -- 172.21.15.158:0/3261225129 >> v1:172.21.15.172:6789/0 conn(0x7f3b2815b540 legacy=0x7f3b2806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b25d9b700 1 -- 172.21.15.158:0/3261225129 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2815e5c0 con 0x7f3b2806f180 2023-10-16T11:19:14.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b26d9d700 1 --2- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2806fea0 0x7f3b2806ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:14.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.125+0000 7f3b25d9b700 1 --2- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806f180 0x7f3b2806a8d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009840 tx=0x7f3b10009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7044762bfc44681f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:14.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2559a700 1 -- 172.21.15.158:0/3261225129 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10011830 con 0x7f3b2806f180 2023-10-16T11:19:14.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2559a700 1 -- 172.21.15.158:0/3261225129 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b10011990 con 0x7f3b2806f180 2023-10-16T11:19:14.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2559a700 1 -- 172.21.15.158:0/3261225129 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10011d20 con 0x7f3b2806f180 2023-10-16T11:19:14.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 -- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806f180 msgr2=0x7f3b2806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 --2- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806f180 0x7f3b2806a8d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009840 tx=0x7f3b10009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:14.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 -- 172.21.15.158:0/3261225129 shutdown_connections 2023-10-16T11:19:14.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 --2- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2806fea0 0x7f3b2806ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 --2- 172.21.15.158:0/3261225129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806f180 0x7f3b2806a8d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 -- 172.21.15.158:0/3261225129 >> 172.21.15.158:0/3261225129 conn(0x7f3b280d03b0 msgr2=0x7f3b280d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:14.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 -- 172.21.15.158:0/3261225129 shutdown_connections 2023-10-16T11:19:14.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.126+0000 7f3b2c935700 1 -- 172.21.15.158:0/3261225129 wait complete. 2023-10-16T11:19:14.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.128+0000 7f3b2c935700 1 Processor -- start 2023-10-16T11:19:14.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.128+0000 7f3b2c935700 1 -- start start 2023-10-16T11:19:14.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.128+0000 7f3b2c935700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b2806f180 0x7f3b2814f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.128+0000 7f3b2c935700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806fea0 0x7f3b280c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.128+0000 7f3b2c935700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2815b540 0x7f3b2815cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.128+0000 7f3b2c935700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281f7500 con 0x7f3b2806f180 2023-10-16T11:19:14.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2c935700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281f6980 con 0x7f3b2806fea0 2023-10-16T11:19:14.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2c935700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281f6c60 con 0x7f3b2815b540 2023-10-16T11:19:14.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b2806f180 0x7f3b2814f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b2806f180 0x7f3b2814f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37710/0 (socket says 172.21.15.158:37710) 2023-10-16T11:19:14.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 -- 172.21.15.158:0/1750095821 learned_addr learned my addr 172.21.15.158:0/1750095821 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:14.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b25d9b700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806fea0 0x7f3b280c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b26d9d700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2815b540 0x7f3b2815cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 -- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2815b540 msgr2=0x7f3b2815cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2815b540 0x7f3b2815cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 -- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806fea0 msgr2=0x7f3b280c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806fea0 0x7f3b280c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 -- 172.21.15.158:0/1750095821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20009b30 con 0x7f3b2806f180 2023-10-16T11:19:14.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b25d9b700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806fea0 0x7f3b280c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:14.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2659c700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b2806f180 0x7f3b2814f680 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c500 tx=0x7f3b1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:14.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b1effd700 1 -- 172.21.15.158:0/1750095821 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18005cb0 con 0x7f3b2806f180 2023-10-16T11:19:14.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b1effd700 1 -- 172.21.15.158:0/1750095821 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b18005e10 con 0x7f3b2806f180 2023-10-16T11:19:14.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b100094a0 con 0x7f3b2806f180 2023-10-16T11:19:14.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b1effd700 1 -- 172.21.15.158:0/1750095821 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b180072d0 con 0x7f3b2806f180 2023-10-16T11:19:14.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.129+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b281f7220 con 0x7f3b2806f180 2023-10-16T11:19:14.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.130+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b280681b0 con 0x7f3b2806f180 2023-10-16T11:19:14.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.131+0000 7f3b1effd700 1 -- 172.21.15.158:0/1750095821 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3b18007430 con 0x7f3b2806f180 2023-10-16T11:19:14.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.131+0000 7f3b1effd700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3b0806b020 0x7f3b0806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.131+0000 7f3b1effd700 1 -- 172.21.15.158:0/1750095821 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3b18088d30 con 0x7f3b2806f180 2023-10-16T11:19:14.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.131+0000 7f3b25d9b700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3b0806b020 0x7f3b0806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.131+0000 7f3b25d9b700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3b0806b020 0x7f3b0806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3b10000c00 tx=0x7f3b10008030 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:14.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.135+0000 7f3b1effd700 1 -- 172.21.15.158:0/1750095821 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b18053ae0 con 0x7f3b2806f180 2023-10-16T11:19:14.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.332+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f3b280b6330 con 0x7f3b2806f180 2023-10-16T11:19:14.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.335+0000 7f3b1effd700 1 -- 172.21.15.158:0/1750095821 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3b180170b0 con 0x7f3b2806f180 2023-10-16T11:19:14.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3b0806b020 msgr2=0x7f3b0806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3b0806b020 0x7f3b0806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3b10000c00 tx=0x7f3b10008030 comp rx=0 tx=0).stop 2023-10-16T11:19:14.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b2806f180 msgr2=0x7f3b2814f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b2806f180 0x7f3b2814f680 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c500 tx=0x7f3b1800c8c0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 shutdown_connections 2023-10-16T11:19:14.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b2815b540 0x7f3b2815cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3b0806b020 0x7f3b0806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b2806f180 0x7f3b2814f680 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 --2- 172.21.15.158:0/1750095821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b2806fea0 0x7f3b280c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 >> 172.21.15.158:0/1750095821 conn(0x7f3b280d03b0 msgr2=0x7f3b280cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:14.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 shutdown_connections 2023-10-16T11:19:14.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.338+0000 7f3b2c935700 1 -- 172.21.15.158:0/1750095821 wait complete. 2023-10-16T11:19:14.352 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T11:19:14.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.471+0000 7fbf05a40700 1 Processor -- start 2023-10-16T11:19:14.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.471+0000 7fbf05a40700 1 -- start start 2023-10-16T11:19:14.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbf05a40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000d1210 0x7fbf000ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbf05a40700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf000ce950 0x7fbf000b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbf05a40700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000d3bd0 con 0x7fbf0015d7c0 2023-10-16T11:19:14.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbf05a40700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf000d4060 con 0x7fbf000ce950 2023-10-16T11:19:14.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbf05a40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000cede0 con 0x7fbf000d1210 2023-10-16T11:19:14.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbeff7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf000ce950 0x7fbf000b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbefe7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000d1210 0x7fbf000ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbeff7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf000ce950 0x7fbf000b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:38900/0 (socket says 172.21.15.158:38900) 2023-10-16T11:19:14.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbeff7fe700 1 -- 172.21.15.158:0/1249626442 learned_addr learned my addr 172.21.15.158:0/1249626442 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:14.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbefeffd700 1 --1- 172.21.15.158:0/1249626442 >> v1:172.21.15.172:6789/0 conn(0x7fbf0015d7c0 0x7fbf000d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:41796/0 (socket says 172.21.15.158:41796) 2023-10-16T11:19:14.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbefe7fc700 1 -- 172.21.15.158:0/1249626442 >> v2:172.21.15.172:3300/0 conn(0x7fbf000ce950 msgr2=0x7fbf000b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:14.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbefe7fc700 1 --2- 172.21.15.158:0/1249626442 >> v2:172.21.15.172:3300/0 conn(0x7fbf000ce950 0x7fbf000b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbefe7fc700 1 -- 172.21.15.158:0/1249626442 >> v1:172.21.15.172:6789/0 conn(0x7fbf0015d7c0 legacy=0x7fbf000d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbefe7fc700 1 -- 172.21.15.158:0/1249626442 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000d2fc0 con 0x7fbf000d1210 2023-10-16T11:19:14.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.472+0000 7fbeff7fe700 1 --2- 172.21.15.158:0/1249626442 >> v2:172.21.15.172:3300/0 conn(0x7fbf000ce950 0x7fbf000b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:14.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbefe7fc700 1 --2- 172.21.15.158:0/1249626442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000d1210 0x7fbf000ce410 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009840 tx=0x7fbef0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5c597dd47d7583c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:14.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbefdffb700 1 -- 172.21.15.158:0/1249626442 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0011830 con 0x7fbf000d1210 2023-10-16T11:19:14.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbefdffb700 1 -- 172.21.15.158:0/1249626442 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef0011990 con 0x7fbf000d1210 2023-10-16T11:19:14.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbefdffb700 1 -- 172.21.15.158:0/1249626442 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0011d20 con 0x7fbf000d1210 2023-10-16T11:19:14.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 -- 172.21.15.158:0/1249626442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000d1210 msgr2=0x7fbf000ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1249626442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000d1210 0x7fbf000ce410 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009840 tx=0x7fbef0009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:14.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 -- 172.21.15.158:0/1249626442 shutdown_connections 2023-10-16T11:19:14.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1249626442 >> v2:172.21.15.172:3300/0 conn(0x7fbf000ce950 0x7fbf000b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1249626442 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000d1210 0x7fbf000ce410 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 -- 172.21.15.158:0/1249626442 >> 172.21.15.158:0/1249626442 conn(0x7fbf000669f0 msgr2=0x7fbf00066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:14.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 -- 172.21.15.158:0/1249626442 shutdown_connections 2023-10-16T11:19:14.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.473+0000 7fbf05a40700 1 -- 172.21.15.158:0/1249626442 wait complete. 2023-10-16T11:19:14.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 Processor -- start 2023-10-16T11:19:14.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 -- start start 2023-10-16T11:19:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000ce950 0x7fbf00159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf000d1210 0x7fbf00155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf0015d7c0 0x7fbf000cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00159960 con 0x7fbf0015d7c0 2023-10-16T11:19:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00155c90 con 0x7fbf000d1210 2023-10-16T11:19:14.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.475+0000 7fbf05a40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00158f50 con 0x7fbf000ce950 2023-10-16T11:19:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf0015d7c0 0x7fbf000cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf0015d7c0 0x7fbf000cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37726/0 (socket says 172.21.15.158:37726) 2023-10-16T11:19:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 -- 172.21.15.158:0/1711997743 learned_addr learned my addr 172.21.15.158:0/1711997743 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbefeffd700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000ce950 0x7fbf00159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbefe7fc700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf000d1210 0x7fbf00155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 -- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000ce950 msgr2=0x7fbf00159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000ce950 0x7fbf00159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 -- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf000d1210 msgr2=0x7fbf00155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf000d1210 0x7fbf00155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 -- 172.21.15.158:0/1711997743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8009b30 con 0x7fbf0015d7c0 2023-10-16T11:19:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbeff7fe700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf0015d7c0 0x7fbf000cc9c0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fbef800efc0 tx=0x7fbef800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbee7fff700 1 -- 172.21.15.158:0/1711997743 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8018070 con 0x7fbf0015d7c0 2023-10-16T11:19:14.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbee7fff700 1 -- 172.21.15.158:0/1711997743 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef800f040 con 0x7fbf0015d7c0 2023-10-16T11:19:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef00094a0 con 0x7fbf0015d7c0 2023-10-16T11:19:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbee7fff700 1 -- 172.21.15.158:0/1711997743 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8003c90 con 0x7fbf0015d7c0 2023-10-16T11:19:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.476+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf00157280 con 0x7fbf0015d7c0 2023-10-16T11:19:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.477+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf000681b0 con 0x7fbf0015d7c0 2023-10-16T11:19:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.478+0000 7fbee7fff700 1 -- 172.21.15.158:0/1711997743 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbef800c880 con 0x7fbf0015d7c0 2023-10-16T11:19:14.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.478+0000 7fbee7fff700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbeec06b050 0x7fbeec06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:14.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.478+0000 7fbee7fff700 1 -- 172.21.15.158:0/1711997743 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fbef8089bc0 con 0x7fbf0015d7c0 2023-10-16T11:19:14.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.478+0000 7fbefeffd700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbeec06b050 0x7fbeec06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:14.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.479+0000 7fbefeffd700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbeec06b050 0x7fbeec06d510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbef4000f60 tx=0x7fbef4007ed0 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:14.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.482+0000 7fbee7fff700 1 -- 172.21.15.158:0/1711997743 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbef80546a0 con 0x7fbf0015d7c0 2023-10-16T11:19:14.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:14.756+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fbf0015b380 con 0x7fbf0015d7c0 2023-10-16T11:19:15.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.283+0000 7fbee7fff700 1 -- 172.21.15.158:0/1711997743 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3112) v1 ==== 163+0+0 (secure 0 0 0) 0x7fbef8009ce0 con 0x7fbf0015d7c0 2023-10-16T11:19:15.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbeec06b050 msgr2=0x7fbeec06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbeec06b050 0x7fbeec06d510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbef4000f60 tx=0x7fbef4007ed0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf0015d7c0 msgr2=0x7fbf000cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf0015d7c0 0x7fbf000cc9c0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fbef800efc0 tx=0x7fbef800c3a0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 shutdown_connections 2023-10-16T11:19:15.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbeec06b050 0x7fbeec06d510 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf0015d7c0 0x7fbf000cc9c0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf000ce950 0x7fbf00159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 --2- 172.21.15.158:0/1711997743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf000d1210 0x7fbf00155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 >> 172.21.15.158:0/1711997743 conn(0x7fbf000669f0 msgr2=0x7fbf00066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:15.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 shutdown_connections 2023-10-16T11:19:15.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.285+0000 7fbf05a40700 1 -- 172.21.15.158:0/1711997743 wait complete. 2023-10-16T11:19:15.301 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-10-16T11:19:15.301 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2023-10-16T11:19:15.302 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T11:19:15.302 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:15.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.423+0000 7f2fd5fb8700 1 Processor -- start 2023-10-16T11:19:15.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.424+0000 7f2fd5fb8700 1 -- start start 2023-10-16T11:19:15.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.424+0000 7f2fd5fb8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2fd015d7c0 0x7f2fd006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.424+0000 7f2fd5fb8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2fd00ce3a0 0x7f2fd00c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.424+0000 7f2fd5fb8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00c4dd0 con 0x7f2fd00cd810 2023-10-16T11:19:15.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.424+0000 7f2fd5fb8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd00af0a0 con 0x7f2fd015d7c0 2023-10-16T11:19:15.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.424+0000 7f2fd5fb8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd015dba0 con 0x7f2fd00ce3a0 2023-10-16T11:19:15.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2fd015d7c0 0x7f2fd006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2fd015d7c0 0x7f2fd006b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37730/0 (socket says 172.21.15.158:37730) 2023-10-16T11:19:15.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3305527335 learned_addr learned my addr 172.21.15.158:0/3305527335 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:15.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3305527335 >> v2:172.21.15.172:3300/0 conn(0x7f2fd00ce3a0 msgr2=0x7f2fd00c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:15.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcffff700 1 --2- 172.21.15.158:0/3305527335 >> v2:172.21.15.172:3300/0 conn(0x7f2fd00ce3a0 0x7f2fd00c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 --2- 172.21.15.158:0/3305527335 >> v2:172.21.15.172:3300/0 conn(0x7f2fd00ce3a0 0x7f2fd00c5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3305527335 >> v1:172.21.15.172:6789/0 conn(0x7f2fd00cd810 legacy=0x7f2fd00cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:15.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3305527335 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd006b8c0 con 0x7f2fd015d7c0 2023-10-16T11:19:15.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fcf7fe700 1 --2- 172.21.15.158:0/3305527335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd015d7c0 0x7f2fd006b2b0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00c4aa0 tx=0x7f2fb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8039301fd42a56ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:15.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fce7fc700 1 -- 172.21.15.158:0/3305527335 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8004230 con 0x7f2fd015d7c0 2023-10-16T11:19:15.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fce7fc700 1 -- 172.21.15.158:0/3305527335 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8004390 con 0x7f2fd015d7c0 2023-10-16T11:19:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.425+0000 7f2fce7fc700 1 -- 172.21.15.158:0/3305527335 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80036a0 con 0x7f2fd015d7c0 2023-10-16T11:19:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3305527335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd015d7c0 msgr2=0x7f2fd006b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3305527335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd015d7c0 0x7f2fd006b2b0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00c4aa0 tx=0x7f2fb8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3305527335 shutdown_connections 2023-10-16T11:19:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3305527335 >> v2:172.21.15.172:3300/0 conn(0x7f2fd00ce3a0 0x7f2fd00c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3305527335 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd015d7c0 0x7f2fd006b2b0 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3305527335 >> 172.21.15.158:0/3305527335 conn(0x7f2fd00cc4b0 msgr2=0x7f2fd00cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3305527335 shutdown_connections 2023-10-16T11:19:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.426+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3305527335 wait complete. 2023-10-16T11:19:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.427+0000 7f2fd5fb8700 1 Processor -- start 2023-10-16T11:19:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.427+0000 7f2fd5fb8700 1 -- start start 2023-10-16T11:19:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fd5fb8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd00cd810 0x7f2fd00c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fd5fb8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fd00ce3a0 0x7f2fd00cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fd5fb8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fd015d7c0 0x7f2fd00cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fd5fb8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c7a50 con 0x7f2fd00cd810 2023-10-16T11:19:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fd5fb8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c8370 con 0x7f2fd00ce3a0 2023-10-16T11:19:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fd5fb8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c8650 con 0x7f2fd015d7c0 2023-10-16T11:19:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd00cd810 0x7f2fd00c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd00cd810 0x7f2fd00c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37736/0 (socket says 172.21.15.158:37736) 2023-10-16T11:19:15.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3025138791 learned_addr learned my addr 172.21.15.158:0/3025138791 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:15.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcffff700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fd015d7c0 0x7f2fd00cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fceffd700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fd00ce3a0 0x7f2fd00cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fd015d7c0 msgr2=0x7f2fd00cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fd015d7c0 0x7f2fd00cbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fd00ce3a0 msgr2=0x7f2fd00cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fd00ce3a0 0x7f2fd00cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 -- 172.21.15.158:0/3025138791 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8009550 con 0x7f2fd00cd810 2023-10-16T11:19:15.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.428+0000 7f2fcf7fe700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd00cd810 0x7f2fd00c7510 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009130 tx=0x7f2fb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:15.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.429+0000 7f2fc7fff700 1 -- 172.21.15.158:0/3025138791 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8003dc0 con 0x7f2fd00cd810 2023-10-16T11:19:15.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.429+0000 7f2fc7fff700 1 -- 172.21.15.158:0/3025138791 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb801f070 con 0x7f2fd00cd810 2023-10-16T11:19:15.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.429+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd00c8930 con 0x7f2fd00cd810 2023-10-16T11:19:15.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.429+0000 7f2fc7fff700 1 -- 172.21.15.158:0/3025138791 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8011870 con 0x7f2fd00cd810 2023-10-16T11:19:15.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.429+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd00c8ef0 con 0x7f2fd00cd810 2023-10-16T11:19:15.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.430+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd00681b0 con 0x7f2fd00cd810 2023-10-16T11:19:15.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.430+0000 7f2fc7fff700 1 -- 172.21.15.158:0/3025138791 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2fb8021030 con 0x7f2fd00cd810 2023-10-16T11:19:15.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.430+0000 7f2fc7fff700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f2fb006b020 0x7f2fb006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.430+0000 7f2fceffd700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f2fb006b020 0x7f2fb006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.430+0000 7f2fc7fff700 1 -- 172.21.15.158:0/3025138791 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f2fb808b110 con 0x7f2fd00cd810 2023-10-16T11:19:15.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.431+0000 7f2fceffd700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f2fb006b020 0x7f2fb006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00b0400 tx=0x7f2fc0008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:15.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.435+0000 7f2fc7fff700 1 -- 172.21.15.158:0/3025138791 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fb8055e10 con 0x7f2fd00cd810 2023-10-16T11:19:15.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.628+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fd006bb70 con 0x7f2fd00cd810 2023-10-16T11:19:15.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.629+0000 7f2fc7fff700 1 -- 172.21.15.158:0/3025138791 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+15092 (secure 0 0 0) 0x7f2fb8059430 con 0x7f2fd00cd810 2023-10-16T11:19:15.631 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:15.631 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":209,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:13.279748+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6833/1279760034":"2023-10-17T11:19:10.239973+0000","172.21.15.158:0/1772820648":"2023-10-17T11:19:09.256689+0000","172.21.15.158:6833/2839558305":"2023-10-17T11:19:09.256689+0000","172.21.15.158:6832/2839558305":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/1533815399":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/1067131743":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/1000153153":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/332114222":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/3542171936":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/786969372":"2023-10-17T11:19:09.256689+0000","172.21.15.172:6835/1279760034":"2023-10-17T11:19:10.239973+0000","172.21.15.158:0/2415272176":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6833/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:6832/3585499115":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/778612447":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/1354527416":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/3344401643":"2023-10-17T11:19:09.256689+0000","172.21.15.158:0/3047051699":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/4256533896":"2023-10-17T11:17:53.925811+0000","172.21.15.158:0/716693565":"2023-10-17T11:17:53.925811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:15.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f2fb006b020 msgr2=0x7f2fb006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f2fb006b020 0x7f2fb006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00b0400 tx=0x7f2fc0008040 comp rx=0 tx=0).stop 2023-10-16T11:19:15.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd00cd810 msgr2=0x7f2fd00c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd00cd810 0x7f2fd00c7510 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009130 tx=0x7f2fb8004750 comp rx=0 tx=0).stop 2023-10-16T11:19:15.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 shutdown_connections 2023-10-16T11:19:15.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fd015d7c0 0x7f2fd00cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f2fb006b020 0x7f2fb006d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fd00cd810 0x7f2fd00c7510 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 --2- 172.21.15.158:0/3025138791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fd00ce3a0 0x7f2fd00cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 >> 172.21.15.158:0/3025138791 conn(0x7f2fd00cc4b0 msgr2=0x7f2fd00b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:15.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 shutdown_connections 2023-10-16T11:19:15.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.631+0000 7f2fd5fb8700 1 -- 172.21.15.158:0/3025138791 wait complete. 2023-10-16T11:19:15.645 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:19:15.646 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:19:15.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.768+0000 7f30c67df700 1 Processor -- start 2023-10-16T11:19:15.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.769+0000 7f30c67df700 1 -- start start 2023-10-16T11:19:15.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.769+0000 7f30c67df700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 0x7f30c006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.769+0000 7f30c67df700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c015ee60 con 0x7f30c006fea0 2023-10-16T11:19:15.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.769+0000 7f30c67df700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c015e2e0 con 0x7f30c015b540 2023-10-16T11:19:15.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.769+0000 7f30c67df700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c015b920 con 0x7f30c006f180 2023-10-16T11:19:15.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.769+0000 7f30c4d7c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f30c006fea0 0x7f30c0161860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37924/0 (socket says 172.21.15.158:37924) 2023-10-16T11:19:15.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.769+0000 7f30c4d7c700 1 -- 172.21.15.158:0/2403392112 learned_addr learned my addr 172.21.15.158:0/2403392112 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:15.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30bf7fe700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 0x7f30c006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801770622 0 0) 0x7f30c015ee60 con 0x7f30c006fea0 2023-10-16T11:19:15.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac0035b0 con 0x7f30c006fea0 2023-10-16T11:19:15.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368731244 0 0) 0x7f30c015e2e0 con 0x7f30c015b540 2023-10-16T11:19:15.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c015ee60 con 0x7f30c015b540 2023-10-16T11:19:15.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3251617492 0 0) 0x7f30ac0035b0 con 0x7f30c006fea0 2023-10-16T11:19:15.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 msgr2=0x7f30c006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:15.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 0x7f30c006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 >> v1:172.21.15.172:6789/0 conn(0x7f30c015b540 legacy=0x7f30c006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c015e5c0 con 0x7f30c006fea0 2023-10-16T11:19:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f30b80019f0 con 0x7f30c006fea0 2023-10-16T11:19:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 >> v1:172.21.15.158:6789/0 conn(0x7f30c006fea0 legacy=0x7f30c0161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30ac003bc0 0x7f30ac003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 0x7f30ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30ac004d90 0x7f30ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c015b920 con 0x7f30ac004d90 2023-10-16T11:19:15.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac00bc00 con 0x7f30ac003bc0 2023-10-16T11:19:15.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac00bdc0 con 0x7f30c006f180 2023-10-16T11:19:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30c4d7c700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30ac004d90 0x7f30ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30bffff700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30ac003bc0 0x7f30ac003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.770+0000 7f30bf7fe700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 0x7f30ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c4d7c700 1 -- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 msgr2=0x7f30ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:15.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c4d7c700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c006f180 0x7f30ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c4d7c700 1 -- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30ac003bc0 msgr2=0x7f30ac003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c4d7c700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30ac003bc0 0x7f30ac003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c4d7c700 1 -- 172.21.15.158:0/2403392112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac0035b0 con 0x7f30ac004d90 2023-10-16T11:19:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30bffff700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30ac003bc0 0x7f30ac003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:15.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c4d7c700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30ac004d90 0x7f30ac005210 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f30b800a7c0 tx=0x7f30ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5027499147a1c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:15.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8012600 con 0x7f30ac004d90 2023-10-16T11:19:15.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b8012780 con 0x7f30ac004d90 2023-10-16T11:19:15.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30beffd700 1 -- 172.21.15.158:0/2403392112 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8012b30 con 0x7f30ac004d90 2023-10-16T11:19:15.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c67df700 1 -- 172.21.15.158:0/2403392112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30ac004d90 msgr2=0x7f30ac005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c67df700 1 --2- 172.21.15.158:0/2403392112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30ac004d90 0x7f30ac005210 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f30b800a7c0 tx=0x7f30ac005880 comp rx=0 tx=0).stop 2023-10-16T11:19:15.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c4d7c700 1 -- 172.21.15.158:0/2403392112 reap_dead start 2023-10-16T11:19:15.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c67df700 1 -- 172.21.15.158:0/2403392112 shutdown_connections 2023-10-16T11:19:15.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c67df700 1 -- 172.21.15.158:0/2403392112 >> 172.21.15.158:0/2403392112 conn(0x7f30c00d03b0 msgr2=0x7f30c00d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:15.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c67df700 1 -- 172.21.15.158:0/2403392112 shutdown_connections 2023-10-16T11:19:15.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.771+0000 7f30c67df700 1 -- 172.21.15.158:0/2403392112 wait complete. 2023-10-16T11:19:15.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.773+0000 7f30c67df700 1 Processor -- start 2023-10-16T11:19:15.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.773+0000 7f30c67df700 1 -- start start 2023-10-16T11:19:15.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.773+0000 7f30c67df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c0159950 0x7f30c00ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c67df700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c00ae3d0 0x7f30c0160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c67df700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c00ae850 0x7f30c01605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c67df700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c00af090 con 0x7f30c0159950 2023-10-16T11:19:15.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c67df700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c00ac890 con 0x7f30c00ae850 2023-10-16T11:19:15.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c67df700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c00ad130 con 0x7f30c00ae3d0 2023-10-16T11:19:15.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c0159950 0x7f30c00ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c0159950 0x7f30c00ad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37756/0 (socket says 172.21.15.158:37756) 2023-10-16T11:19:15.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 -- 172.21.15.158:0/1845814024 learned_addr learned my addr 172.21.15.158:0/1845814024 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:15.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bf7fe700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c00ae3d0 0x7f30c0160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c4d7c700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c00ae850 0x7f30c01605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 -- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c00ae3d0 msgr2=0x7f30c0160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c00ae3d0 0x7f30c0160080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 -- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c00ae850 msgr2=0x7f30c01605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c00ae850 0x7f30c01605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 -- 172.21.15.158:0/1845814024 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c00aedb0 con 0x7f30c0159950 2023-10-16T11:19:15.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c4d7c700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c00ae850 0x7f30c01605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:15.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bffff700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c0159950 0x7f30c00ad5c0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f30a800e7b0 tx=0x7f30a800eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:15.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bcff9700 1 -- 172.21.15.158:0/1845814024 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a80053a0 con 0x7f30c0159950 2023-10-16T11:19:15.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bcff9700 1 -- 172.21.15.158:0/1845814024 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a8005500 con 0x7f30c0159950 2023-10-16T11:19:15.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c00acae0 con 0x7f30c0159950 2023-10-16T11:19:15.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30bcff9700 1 -- 172.21.15.158:0/1845814024 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a80038c0 con 0x7f30c0159950 2023-10-16T11:19:15.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.774+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c00adb00 con 0x7f30c0159950 2023-10-16T11:19:15.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.775+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c00ad410 con 0x7f30c0159950 2023-10-16T11:19:15.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.776+0000 7f30bcff9700 1 -- 172.21.15.158:0/1845814024 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f30a8003a20 con 0x7f30c0159950 2023-10-16T11:19:15.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.776+0000 7f30bcff9700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f30a006b020 0x7f30a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:15.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.776+0000 7f30bcff9700 1 -- 172.21.15.158:0/1845814024 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f30a80891b0 con 0x7f30c0159950 2023-10-16T11:19:15.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.776+0000 7f30bf7fe700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f30a006b020 0x7f30a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:15.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.776+0000 7f30bf7fe700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f30a006b020 0x7f30a006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30c00cc120 tx=0x7f30b0009210 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:15.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.781+0000 7f30bcff9700 1 -- 172.21.15.158:0/1845814024 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30a8053ed0 con 0x7f30c0159950 2023-10-16T11:19:15.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.973+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30c00681b0 con 0x7f30c0159950 2023-10-16T11:19:15.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.973+0000 7f30bcff9700 1 -- 172.21.15.158:0/1845814024 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30a8019020 con 0x7f30c0159950 2023-10-16T11:19:15.975 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:19:15.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f30a006b020 msgr2=0x7f30a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f30a006b020 0x7f30a006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30c00cc120 tx=0x7f30b0009210 comp rx=0 tx=0).stop 2023-10-16T11:19:15.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c0159950 msgr2=0x7f30c00ad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:15.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c0159950 0x7f30c00ad5c0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f30a800e7b0 tx=0x7f30a800eb70 comp rx=0 tx=0).stop 2023-10-16T11:19:15.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 shutdown_connections 2023-10-16T11:19:15.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c00ae850 0x7f30c01605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f30a006b020 0x7f30a006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30c0159950 0x7f30c00ad5c0 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 --2- 172.21.15.158:0/1845814024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c00ae3d0 0x7f30c0160080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:15.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 >> 172.21.15.158:0/1845814024 conn(0x7f30c00d03b0 msgr2=0x7f30c00cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:15.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 shutdown_connections 2023-10-16T11:19:15.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:15.976+0000 7f30c67df700 1 -- 172.21.15.158:0/1845814024 wait complete. 2023-10-16T11:19:15.990 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T11:19:16.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.108+0000 7f971a646700 1 Processor -- start 2023-10-16T11:19:16.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.108+0000 7f971a646700 1 -- start start 2023-10-16T11:19:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f971a646700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f971406ab40 0x7f971406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f971a646700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f97140729b0 0x7f9714071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f971a646700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9714071e50 con 0x7f971406e5d0 2023-10-16T11:19:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f971a646700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9714161270 con 0x7f971406ab40 2023-10-16T11:19:16.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f971a646700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9714161510 con 0x7f97140729b0 2023-10-16T11:19:16.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f9713fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f971406ab40 0x7f971406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:16.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f9713fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f971406ab40 0x7f971406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37764/0 (socket says 172.21.15.158:37764) 2023-10-16T11:19:16.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.109+0000 7f9713fff700 1 -- 172.21.15.158:0/587074231 learned_addr learned my addr 172.21.15.158:0/587074231 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:16.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9718be3700 1 --2- 172.21.15.158:0/587074231 >> v2:172.21.15.172:3300/0 conn(0x7f97140729b0 0x7f9714071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9713fff700 1 -- 172.21.15.158:0/587074231 >> v2:172.21.15.172:3300/0 conn(0x7f97140729b0 msgr2=0x7f9714071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9713fff700 1 --2- 172.21.15.158:0/587074231 >> v2:172.21.15.172:3300/0 conn(0x7f97140729b0 0x7f9714071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9713fff700 1 -- 172.21.15.158:0/587074231 >> v1:172.21.15.172:6789/0 conn(0x7f971406e5d0 legacy=0x7f97140722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9713fff700 1 -- 172.21.15.158:0/587074231 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97141617f0 con 0x7f971406ab40 2023-10-16T11:19:16.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9713fff700 1 --2- 172.21.15.158:0/587074231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f971406d820 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f9714158240 tx=0x7f9708009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a816b66035eec62 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9712ffd700 1 -- 172.21.15.158:0/587074231 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9708011830 con 0x7f971406ab40 2023-10-16T11:19:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9712ffd700 1 -- 172.21.15.158:0/587074231 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9708011990 con 0x7f971406ab40 2023-10-16T11:19:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.110+0000 7f9712ffd700 1 -- 172.21.15.158:0/587074231 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9708011d20 con 0x7f971406ab40 2023-10-16T11:19:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 -- 172.21.15.158:0/587074231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 msgr2=0x7f971406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 --2- 172.21.15.158:0/587074231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f971406d820 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f9714158240 tx=0x7f9708009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:16.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 -- 172.21.15.158:0/587074231 shutdown_connections 2023-10-16T11:19:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 --2- 172.21.15.158:0/587074231 >> v2:172.21.15.172:3300/0 conn(0x7f97140729b0 0x7f9714071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 --2- 172.21.15.158:0/587074231 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f971406d820 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 -- 172.21.15.158:0/587074231 >> 172.21.15.158:0/587074231 conn(0x7f97140d02b0 msgr2=0x7f97140d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 -- 172.21.15.158:0/587074231 shutdown_connections 2023-10-16T11:19:16.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.111+0000 7f971a646700 1 -- 172.21.15.158:0/587074231 wait complete. 2023-10-16T11:19:16.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.112+0000 7f971a646700 1 Processor -- start 2023-10-16T11:19:16.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.112+0000 7f971a646700 1 -- start start 2023-10-16T11:19:16.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f971a646700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f9714154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f971a646700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f971406e5d0 0x7f97140b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f971a646700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97140729b0 0x7f971415f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f971a646700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9714155190 con 0x7f971406ab40 2023-10-16T11:19:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f971a646700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97140b3db0 con 0x7f971406e5d0 2023-10-16T11:19:16.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f971a646700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97140b4090 con 0x7f97140729b0 2023-10-16T11:19:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f9714154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f9714154c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37770/0 (socket says 172.21.15.158:37770) 2023-10-16T11:19:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 -- 172.21.15.158:0/882269053 learned_addr learned my addr 172.21.15.158:0/882269053 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 -- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97140729b0 msgr2=0x7f971415f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:16.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9718be3700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97140729b0 0x7f971415f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f97137fe700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f971406e5d0 0x7f97140b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97140729b0 0x7f971415f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 -- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f971406e5d0 msgr2=0x7f97140b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f971406e5d0 0x7f97140b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:16.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 -- 172.21.15.158:0/882269053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9708009550 con 0x7f971406ab40 2023-10-16T11:19:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9713fff700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f9714154c50 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f9708014040 tx=0x7f9708004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9710ff9700 1 -- 172.21.15.158:0/882269053 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97080036a0 con 0x7f971406ab40 2023-10-16T11:19:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9710ff9700 1 -- 172.21.15.158:0/882269053 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970801e070 con 0x7f971406ab40 2023-10-16T11:19:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.113+0000 7f9710ff9700 1 -- 172.21.15.158:0/882269053 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9708003a00 con 0x7f971406ab40 2023-10-16T11:19:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.114+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97140b4370 con 0x7f971406ab40 2023-10-16T11:19:16.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.114+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97140b4930 con 0x7f971406ab40 2023-10-16T11:19:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.115+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97140b4b20 con 0x7f971406ab40 2023-10-16T11:19:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.115+0000 7f9710ff9700 1 -- 172.21.15.158:0/882269053 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9708003b60 con 0x7f971406ab40 2023-10-16T11:19:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.115+0000 7f9710ff9700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f970006b020 0x7f970006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.116+0000 7f9710ff9700 1 -- 172.21.15.158:0/882269053 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f970808aa10 con 0x7f971406ab40 2023-10-16T11:19:16.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.116+0000 7f97137fe700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f970006b020 0x7f970006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.116+0000 7f97137fe700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f970006b020 0x7f970006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97140c6f90 tx=0x7f9704009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.120+0000 7f9710ff9700 1 -- 172.21.15.158:0/882269053 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9708055670 con 0x7f971406ab40 2023-10-16T11:19:16.286 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:19:16.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:19:16.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:16.402+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f97140681b0 con 0x7f971406ab40 2023-10-16T11:19:17.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.290+0000 7f9710ff9700 1 -- 172.21.15.158:0/882269053 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3114) v1 ==== 189+0+0 (secure 0 0 0) 0x7f9708020100 con 0x7f971406ab40 2023-10-16T11:19:17.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f970006b020 msgr2=0x7f970006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f970006b020 0x7f970006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97140c6f90 tx=0x7f9704009040 comp rx=0 tx=0).stop 2023-10-16T11:19:17.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 msgr2=0x7f9714154c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f9714154c50 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f9708014040 tx=0x7f9708004750 comp rx=0 tx=0).stop 2023-10-16T11:19:17.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 shutdown_connections 2023-10-16T11:19:17.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97140729b0 0x7f971415f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f970006b020 0x7f970006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f971406ab40 0x7f9714154c50 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 --2- 172.21.15.158:0/882269053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f971406e5d0 0x7f97140b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 >> 172.21.15.158:0/882269053 conn(0x7f97140d02b0 msgr2=0x7f971406d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:17.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 shutdown_connections 2023-10-16T11:19:17.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.293+0000 7f971a646700 1 -- 172.21.15.158:0/882269053 wait complete. 2023-10-16T11:19:17.310 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T11:19:17.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.440+0000 7f48748eb700 1 Processor -- start 2023-10-16T11:19:17.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.440+0000 7f48748eb700 1 -- start start 2023-10-16T11:19:17.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f48748eb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f487006afe0 0x7f48700caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:17.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f48748eb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700cbd30 con 0x7f487015da40 2023-10-16T11:19:17.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f48748eb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700cc1c0 con 0x7f48700cc590 2023-10-16T11:19:17.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f48748eb700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487015de20 con 0x7f487006afe0 2023-10-16T11:19:17.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486dd9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f487006afe0 0x7f48700caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:17.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486dd9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f487006afe0 0x7f48700caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37780/0 (socket says 172.21.15.158:37780) 2023-10-16T11:19:17.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486e59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f487015da40 0x7f487006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37954/0 (socket says 172.21.15.158:37954) 2023-10-16T11:19:17.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486dd9b700 1 -- 172.21.15.158:0/2039009926 learned_addr learned my addr 172.21.15.158:0/2039009926 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:17.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486dd9b700 1 -- 172.21.15.158:0/2039009926 >> v1:172.21.15.172:6789/0 conn(0x7f48700cc590 legacy=0x7f48700afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486dd9b700 1 -- 172.21.15.158:0/2039009926 >> v1:172.21.15.158:6789/0 conn(0x7f487015da40 legacy=0x7f487006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486dd9b700 1 -- 172.21.15.158:0/2039009926 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700ce260 con 0x7f487006afe0 2023-10-16T11:19:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.441+0000 7f486d59a700 1 -- 172.21.15.158:0/2039009926 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945423201 0 0) 0x7f48700cbd30 con 0x7f487015da40 2023-10-16T11:19:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f486dd9b700 1 --2- 172.21.15.158:0/2039009926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487006afe0 0x7f48700caa20 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f48600098f0 tx=0x7f4860009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f38469b52845e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:17.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f486d59a700 1 -- 172.21.15.158:0/2039009926 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860004230 con 0x7f487006afe0 2023-10-16T11:19:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f486d59a700 1 -- 172.21.15.158:0/2039009926 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860004390 con 0x7f487006afe0 2023-10-16T11:19:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f486d59a700 1 -- 172.21.15.158:0/2039009926 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48600036a0 con 0x7f487006afe0 2023-10-16T11:19:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f48748eb700 1 -- 172.21.15.158:0/2039009926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487006afe0 msgr2=0x7f48700caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f48748eb700 1 --2- 172.21.15.158:0/2039009926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487006afe0 0x7f48700caa20 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f48600098f0 tx=0x7f4860009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f48748eb700 1 -- 172.21.15.158:0/2039009926 shutdown_connections 2023-10-16T11:19:17.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f48748eb700 1 --2- 172.21.15.158:0/2039009926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487006afe0 0x7f48700caa20 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f48748eb700 1 -- 172.21.15.158:0/2039009926 >> 172.21.15.158:0/2039009926 conn(0x7f4870099a10 msgr2=0x7f48700d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:17.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f48748eb700 1 -- 172.21.15.158:0/2039009926 shutdown_connections 2023-10-16T11:19:17.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.442+0000 7f48748eb700 1 -- 172.21.15.158:0/2039009926 wait complete. 2023-10-16T11:19:17.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 Processor -- start 2023-10-16T11:19:17.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 -- start start 2023-10-16T11:19:17.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f487006afe0 0x7f4870145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:17.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48700cc590 0x7f4870146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:17.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487015da40 0x7f48700c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:17.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48700b52d0 con 0x7f487015da40 2023-10-16T11:19:17.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48700b4750 con 0x7f48700cc590 2023-10-16T11:19:17.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.444+0000 7f48748eb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48700b4a30 con 0x7f487006afe0 2023-10-16T11:19:17.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487015da40 0x7f48700c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:17.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487015da40 0x7f48700c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37786/0 (socket says 172.21.15.158:37786) 2023-10-16T11:19:17.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 -- 172.21.15.158:0/3351081505 learned_addr learned my addr 172.21.15.158:0/3351081505 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:17.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486dd9b700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48700cc590 0x7f4870146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:17.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486e59c700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f487006afe0 0x7f4870145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:17.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 -- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f487006afe0 msgr2=0x7f4870145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f487006afe0 0x7f4870145ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 -- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48700cc590 msgr2=0x7f4870146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48700cc590 0x7f4870146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 -- 172.21.15.158:0/3351081505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009550 con 0x7f487015da40 2023-10-16T11:19:17.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f486ed9d700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487015da40 0x7f48700c3e30 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f486800cc10 tx=0x7f486800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:17.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f485effd700 1 -- 172.21.15.158:0/3351081505 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800abb0 con 0x7f487015da40 2023-10-16T11:19:17.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f485effd700 1 -- 172.21.15.158:0/3351081505 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486800ad10 con 0x7f487015da40 2023-10-16T11:19:17.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f485effd700 1 -- 172.21.15.158:0/3351081505 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868012660 con 0x7f487015da40 2023-10-16T11:19:17.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48700b4d10 con 0x7f487015da40 2023-10-16T11:19:17.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.445+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48700c4370 con 0x7f487015da40 2023-10-16T11:19:17.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.446+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4870146aa0 con 0x7f487015da40 2023-10-16T11:19:17.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.447+0000 7f485effd700 1 -- 172.21.15.158:0/3351081505 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f48680127c0 con 0x7f487015da40 2023-10-16T11:19:17.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.447+0000 7f485effd700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f485006b020 0x7f485006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:17.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.447+0000 7f485effd700 1 -- 172.21.15.158:0/3351081505 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f486808a1f0 con 0x7f487015da40 2023-10-16T11:19:17.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.447+0000 7f486e59c700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f485006b020 0x7f485006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:17.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.447+0000 7f486e59c700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f485006b020 0x7f485006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4864007fd0 tx=0x7f4864009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:17.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.452+0000 7f485effd700 1 -- 172.21.15.158:0/3351081505 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4868054e60 con 0x7f487015da40 2023-10-16T11:19:17.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.654+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f48700681b0 con 0x7f487015da40 2023-10-16T11:19:17.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.657+0000 7f485effd700 1 -- 172.21.15.158:0/3351081505 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f48680160c0 con 0x7f487015da40 2023-10-16T11:19:17.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f485006b020 msgr2=0x7f485006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f485006b020 0x7f485006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4864007fd0 tx=0x7f4864009040 comp rx=0 tx=0).stop 2023-10-16T11:19:17.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487015da40 msgr2=0x7f48700c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:17.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487015da40 0x7f48700c3e30 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f486800cc10 tx=0x7f486800a060 comp rx=0 tx=0).stop 2023-10-16T11:19:17.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 shutdown_connections 2023-10-16T11:19:17.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f485006b020 0x7f485006d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f487015da40 0x7f48700c3e30 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f487006afe0 0x7f4870145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 --2- 172.21.15.158:0/3351081505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f48700cc590 0x7f4870146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:17.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 >> 172.21.15.158:0/3351081505 conn(0x7f4870099a10 msgr2=0x7f487015ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:17.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 shutdown_connections 2023-10-16T11:19:17.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.659+0000 7f48748eb700 1 -- 172.21.15.158:0/3351081505 wait complete. 2023-10-16T11:19:17.673 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:19:17.673 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:19:17.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:19:17.760 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:19:17.761 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:17.761 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:17.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:19:17.779 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:19:17.779 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:17.799 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:19:17.799 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:17.800 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:17.835 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:19:17.835 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:19:17.836 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:17.858 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:19:17.859 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T11:19:17.859 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:17.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.993+0000 7feb940b9700 1 Processor -- start 2023-10-16T11:19:17.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.993+0000 7feb940b9700 1 -- start start 2023-10-16T11:19:17.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb940b9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb8c06ab40 0x7feb8c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:17.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb940b9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c161270 con 0x7feb8c06e5d0 2023-10-16T11:19:17.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb940b9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c161510 con 0x7feb8c0729b0 2023-10-16T11:19:17.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb940b9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c1617f0 con 0x7feb8c06ab40 2023-10-16T11:19:18.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb91654700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7feb8c06e5d0 0x7feb8c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:37966/0 (socket says 172.21.15.158:37966) 2023-10-16T11:19:18.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb91654700 1 -- 172.21.15.158:0/1457982215 learned_addr learned my addr 172.21.15.158:0/1457982215 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:18.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb91e55700 1 --2- 172.21.15.158:0/1457982215 >> v2:172.21.15.172:3300/0 conn(0x7feb8c06ab40 0x7feb8c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513130208 0 0) 0x7feb8c161270 con 0x7feb8c06e5d0 2023-10-16T11:19:18.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.994+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb780035b0 con 0x7feb8c06e5d0 2023-10-16T11:19:18.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2899346392 0 0) 0x7feb780035b0 con 0x7feb8c06e5d0 2023-10-16T11:19:18.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 >> v2:172.21.15.172:3300/0 conn(0x7feb8c06ab40 msgr2=0x7feb8c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:18.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 --2- 172.21.15.158:0/1457982215 >> v2:172.21.15.172:3300/0 conn(0x7feb8c06ab40 0x7feb8c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 >> v1:172.21.15.172:6789/0 conn(0x7feb8c0729b0 legacy=0x7feb8c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c161ad0 con 0x7feb8c06e5d0 2023-10-16T11:19:18.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7feb7c001670 con 0x7feb8c06e5d0 2023-10-16T11:19:18.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 >> v1:172.21.15.158:6789/0 conn(0x7feb8c06e5d0 legacy=0x7feb8c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78003a00 0x7feb78003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb78004850 0x7feb78004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb78005210 0x7feb7800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c1617f0 con 0x7feb78004850 2023-10-16T11:19:18.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7800c0b0 con 0x7feb78005210 2023-10-16T11:19:18.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7800c270 con 0x7feb78003a00 2023-10-16T11:19:18.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91654700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb78004850 0x7feb78004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91e55700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78003a00 0x7feb78003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb92656700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb78005210 0x7feb7800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91654700 1 -- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78003a00 msgr2=0x7feb78003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:18.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91654700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78003a00 0x7feb78003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91654700 1 -- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb78005210 msgr2=0x7feb7800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:18.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91654700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb78005210 0x7feb7800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91654700 1 -- 172.21.15.158:0/1457982215 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb780035b0 con 0x7feb78004850 2023-10-16T11:19:18.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb92656700 1 -- 172.21.15.158:0/1457982215 reap_dead start 2023-10-16T11:19:18.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91e55700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78003a00 0x7feb78003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:18.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb91654700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb78004850 0x7feb78004cd0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c210 tx=0x7feb78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771bc3e6234714e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:18.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c007c10 con 0x7feb78004850 2023-10-16T11:19:18.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c00cdb0 con 0x7feb78004850 2023-10-16T11:19:18.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.995+0000 7feb90e53700 1 -- 172.21.15.158:0/1457982215 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c005ba0 con 0x7feb78004850 2023-10-16T11:19:18.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.996+0000 7feb940b9700 1 -- 172.21.15.158:0/1457982215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb78004850 msgr2=0x7feb78004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.996+0000 7feb940b9700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb78004850 0x7feb78004cd0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c210 tx=0x7feb78005880 comp rx=0 tx=0).stop 2023-10-16T11:19:18.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.996+0000 7feb940b9700 1 -- 172.21.15.158:0/1457982215 shutdown_connections 2023-10-16T11:19:18.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.996+0000 7feb940b9700 1 --2- 172.21.15.158:0/1457982215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb78004850 0x7feb78004cd0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.996+0000 7feb940b9700 1 -- 172.21.15.158:0/1457982215 >> 172.21.15.158:0/1457982215 conn(0x7feb8c0d02b0 msgr2=0x7feb8c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:18.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.996+0000 7feb940b9700 1 -- 172.21.15.158:0/1457982215 shutdown_connections 2023-10-16T11:19:18.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.996+0000 7feb940b9700 1 -- 172.21.15.158:0/1457982215 wait complete. 2023-10-16T11:19:18.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.997+0000 7feb940b9700 1 Processor -- start 2023-10-16T11:19:18.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.997+0000 7feb940b9700 1 -- start start 2023-10-16T11:19:18.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb940b9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78004850 0x7feb8c1f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb940b9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0ad210 0x7feb8c0ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb940b9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0c5fc0 0x7feb8c0c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb940b9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0adbd0 con 0x7feb8c0ad210 2023-10-16T11:19:18.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb940b9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0b3b80 con 0x7feb8c0c5fc0 2023-10-16T11:19:18.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb940b9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0b2f70 con 0x7feb78004850 2023-10-16T11:19:18.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0ad210 0x7feb8c0ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0ad210 0x7feb8c0ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37804/0 (socket says 172.21.15.158:37804) 2023-10-16T11:19:18.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 -- 172.21.15.158:0/1736224571 learned_addr learned my addr 172.21.15.158:0/1736224571 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:18.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91e55700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78004850 0x7feb8c1f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb92656700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0c5fc0 0x7feb8c0c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 -- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78004850 msgr2=0x7feb8c1f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78004850 0x7feb8c1f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 -- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0c5fc0 msgr2=0x7feb8c0c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0c5fc0 0x7feb8c0c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 -- 172.21.15.158:0/1736224571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0b3250 con 0x7feb8c0ad210 2023-10-16T11:19:18.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91e55700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78004850 0x7feb8c1f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:18.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.998+0000 7feb91654700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0ad210 0x7feb8c0ad690 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7feb7800c080 tx=0x7feb7c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:18.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.999+0000 7feb827fc700 1 -- 172.21.15.158:0/1736224571 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c0070d0 con 0x7feb8c0ad210 2023-10-16T11:19:18.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.999+0000 7feb827fc700 1 -- 172.21.15.158:0/1736224571 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c007230 con 0x7feb8c0ad210 2023-10-16T11:19:18.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.999+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c0b3530 con 0x7feb8c0ad210 2023-10-16T11:19:18.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.999+0000 7feb827fc700 1 -- 172.21.15.158:0/1736224571 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c0075c0 con 0x7feb8c0ad210 2023-10-16T11:19:18.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.999+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c1f2b40 con 0x7feb8c0ad210 2023-10-16T11:19:18.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:17.999+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8c1f2d00 con 0x7feb8c0ad210 2023-10-16T11:19:18.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.000+0000 7feb827fc700 1 -- 172.21.15.158:0/1736224571 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb7c01e430 con 0x7feb8c0ad210 2023-10-16T11:19:18.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.000+0000 7feb827fc700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7feb7006b020 0x7feb7006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.000+0000 7feb827fc700 1 -- 172.21.15.158:0/1736224571 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7feb7c08ae50 con 0x7feb8c0ad210 2023-10-16T11:19:18.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.000+0000 7feb91e55700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7feb7006b020 0x7feb7006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.001+0000 7feb91e55700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7feb7006b020 0x7feb7006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb8c04fec0 tx=0x7feb88008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:18.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.005+0000 7feb827fc700 1 -- 172.21.15.158:0/1736224571 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb7c055c70 con 0x7feb8c0ad210 2023-10-16T11:19:18.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.240+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb8c161d80 con 0x7feb8c0ad210 2023-10-16T11:19:18.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.240+0000 7feb827fc700 1 -- 172.21.15.158:0/1736224571 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3717 (secure 0 0 0) 0x7feb7c059290 con 0x7feb8c0ad210 2023-10-16T11:19:18.243 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:18.243 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":105,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2912297853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2912297853},{"type":"v1","addr":"172.21.15.158:6836","nonce":2912297853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":38441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/2297507228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":2297507228},{"type":"v1","addr":"172.21.15.158:6837","nonce":2297507228}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38447,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/3795197457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3795197457},{"type":"v1","addr":"172.21.15.172:6834","nonce":3795197457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":38750,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2618834466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2618834466},{"type":"v1","addr":"172.21.15.172:6835","nonce":2618834466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[]} 2023-10-16T11:19:18.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7feb7006b020 msgr2=0x7feb7006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7feb7006b020 0x7feb7006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb8c04fec0 tx=0x7feb88008040 comp rx=0 tx=0).stop 2023-10-16T11:19:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0ad210 msgr2=0x7feb8c0ad690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0ad210 0x7feb8c0ad690 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7feb7800c080 tx=0x7feb7c001a00 comp rx=0 tx=0).stop 2023-10-16T11:19:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 shutdown_connections 2023-10-16T11:19:18.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0c5fc0 0x7feb8c0c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb78004850 0x7feb8c1f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7feb7006b020 0x7feb7006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 --2- 172.21.15.158:0/1736224571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0ad210 0x7feb8c0ad690 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 >> 172.21.15.158:0/1736224571 conn(0x7feb8c0d02b0 msgr2=0x7feb8c15d510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:18.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 shutdown_connections 2023-10-16T11:19:18.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.243+0000 7feb940b9700 1 -- 172.21.15.158:0/1736224571 wait complete. 2023-10-16T11:19:18.247 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 105 2023-10-16T11:19:18.256 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T11:19:18.257 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T11:19:18.257 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:19:18.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:19:18.258 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T11:19:18.258 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T11:19:18.258 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:19:18.258 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:19:18.259 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T11:19:18.259 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T11:19:18.259 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:19:18.260 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:19:18.260 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T11:19:18.260 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T11:19:18.260 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:19:18.261 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:19:18.265 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:19:18.266 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T11:19:18.267 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:19:18.267 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T11:19:18.271 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:19:18.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T11:19:18.272 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:19:18.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T11:19:18.307 INFO:tasks.ceph.mds.a:Started 2023-10-16T11:19:18.308 DEBUG:teuthology.parallel:result is None 2023-10-16T11:19:18.309 INFO:tasks.ceph.mds.c:Started 2023-10-16T11:19:18.309 DEBUG:teuthology.parallel:result is None 2023-10-16T11:19:18.314 INFO:tasks.ceph.mds.b:Started 2023-10-16T11:19:18.314 DEBUG:teuthology.parallel:result is None 2023-10-16T11:19:18.315 INFO:tasks.ceph.mds.d:Started 2023-10-16T11:19:18.315 DEBUG:teuthology.parallel:result is None 2023-10-16T11:19:18.316 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T11:19:18.387 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:19:18.387 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T11:19:18.404 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:19:18.404 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:19:18.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.441+0000 7f00041e7700 1 Processor -- start 2023-10-16T11:19:18.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.442+0000 7f00041e7700 1 -- start start 2023-10-16T11:19:18.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.442+0000 7f00041e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0b5870 0x7efffc168750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.442+0000 7f00041e7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efffc0d1db0 con 0x7efffc15da60 2023-10-16T11:19:18.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.442+0000 7f00041e7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efffc0cfd60 con 0x7efffc0ce5b0 2023-10-16T11:19:18.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.442+0000 7f00041e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efffc0cfb40 con 0x7efffc0b5870 2023-10-16T11:19:18.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0001f83700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efffc15da60 0x7efffc0cf430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38010/0 (socket says 172.21.15.158:38010) 2023-10-16T11:19:18.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0001f83700 1 -- 172.21.15.158:0/2580441519 learned_addr learned my addr 172.21.15.158:0/2580441519 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:18.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0002784700 1 --2- 172.21.15.158:0/2580441519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0b5870 0x7efffc168750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0000f81700 1 -- 172.21.15.158:0/2580441519 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708273969 0 0) 0x7efffc0d1db0 con 0x7efffc15da60 2023-10-16T11:19:18.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0000f81700 1 -- 172.21.15.158:0/2580441519 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe80035b0 con 0x7efffc15da60 2023-10-16T11:19:18.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0002784700 1 -- 172.21.15.158:0/2580441519 >> v1:172.21.15.172:6789/0 conn(0x7efffc0ce5b0 legacy=0x7efffc0b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0000f81700 1 -- 172.21.15.158:0/2580441519 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979991656 0 0) 0x7efffc0cfd60 con 0x7efffc0ce5b0 2023-10-16T11:19:18.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0002784700 1 -- 172.21.15.158:0/2580441519 >> v1:172.21.15.158:6789/0 conn(0x7efffc15da60 legacy=0x7efffc0cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0002784700 1 -- 172.21.15.158:0/2580441519 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc0d1760 con 0x7efffc0b5870 2023-10-16T11:19:18.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0002784700 1 --2- 172.21.15.158:0/2580441519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0b5870 0x7efffc168750 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efff4009dc0 tx=0x7efff400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3bdf256a7ed1610 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:18.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0000f81700 1 -- 172.21.15.158:0/2580441519 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff400e070 con 0x7efffc0b5870 2023-10-16T11:19:18.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0000f81700 1 -- 172.21.15.158:0/2580441519 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff400bb20 con 0x7efffc0b5870 2023-10-16T11:19:18.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.443+0000 7f0000f81700 1 -- 172.21.15.158:0/2580441519 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff400be90 con 0x7efffc0b5870 2023-10-16T11:19:18.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.444+0000 7f00041e7700 1 -- 172.21.15.158:0/2580441519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0b5870 msgr2=0x7efffc168750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.444+0000 7f00041e7700 1 --2- 172.21.15.158:0/2580441519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0b5870 0x7efffc168750 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efff4009dc0 tx=0x7efff400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:18.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.444+0000 7f00041e7700 1 -- 172.21.15.158:0/2580441519 shutdown_connections 2023-10-16T11:19:18.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.444+0000 7f00041e7700 1 --2- 172.21.15.158:0/2580441519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0b5870 0x7efffc168750 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.444+0000 7f00041e7700 1 -- 172.21.15.158:0/2580441519 >> 172.21.15.158:0/2580441519 conn(0x7efffc09c9b0 msgr2=0x7efffc0d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:18.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.444+0000 7f00041e7700 1 -- 172.21.15.158:0/2580441519 shutdown_connections 2023-10-16T11:19:18.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.444+0000 7f00041e7700 1 -- 172.21.15.158:0/2580441519 wait complete. 2023-10-16T11:19:18.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.445+0000 7f00041e7700 1 Processor -- start 2023-10-16T11:19:18.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.445+0000 7f00041e7700 1 -- start start 2023-10-16T11:19:18.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f00041e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efffc0b5870 0x7efffc1f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f00041e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0ce5b0 0x7efffc1e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f00041e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efffc15da60 0x7efffc1603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f00041e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc1e8ad0 con 0x7efffc15da60 2023-10-16T11:19:18.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f00041e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc1e7750 con 0x7efffc0b5870 2023-10-16T11:19:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f00041e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efffc1e7a30 con 0x7efffc0ce5b0 2023-10-16T11:19:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efffc15da60 0x7efffc1603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efffc15da60 0x7efffc1603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37844/0 (socket says 172.21.15.158:37844) 2023-10-16T11:19:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 -- 172.21.15.158:0/3828941583 learned_addr learned my addr 172.21.15.158:0/3828941583 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0001f83700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efffc0b5870 0x7efffc1f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0001782700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0ce5b0 0x7efffc1e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 -- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0ce5b0 msgr2=0x7efffc1e6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0ce5b0 0x7efffc1e6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 -- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efffc0b5870 msgr2=0x7efffc1f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efffc0b5870 0x7efffc1f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:18.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 -- 172.21.15.158:0/3828941583 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff4009a70 con 0x7efffc15da60 2023-10-16T11:19:18.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.446+0000 7f0002784700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efffc15da60 0x7efffc1603c0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7efffc08cf60 tx=0x7efff4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:18.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.447+0000 7efff27fc700 1 -- 172.21.15.158:0/3828941583 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4004810 con 0x7efffc15da60 2023-10-16T11:19:18.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.447+0000 7efff27fc700 1 -- 172.21.15.158:0/3828941583 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff4021070 con 0x7efffc15da60 2023-10-16T11:19:18.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.447+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efffc1e7d10 con 0x7efffc15da60 2023-10-16T11:19:18.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.447+0000 7efff27fc700 1 -- 172.21.15.158:0/3828941583 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4026670 con 0x7efffc15da60 2023-10-16T11:19:18.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.447+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efffc1e82d0 con 0x7efffc15da60 2023-10-16T11:19:18.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.448+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efffc0681b0 con 0x7efffc15da60 2023-10-16T11:19:18.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.448+0000 7efff27fc700 1 -- 172.21.15.158:0/3828941583 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efff4023030 con 0x7efffc15da60 2023-10-16T11:19:18.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.448+0000 7efff27fc700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7effe006b020 0x7effe006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:18.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.449+0000 7efff27fc700 1 -- 172.21.15.158:0/3828941583 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6517+0+0 (secure 0 0 0) 0x7efff408bae0 con 0x7efffc15da60 2023-10-16T11:19:18.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.449+0000 7f0001f83700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7effe006b020 0x7effe006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:18.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.449+0000 7f0001f83700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7effe006b020 0x7effe006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effec007fd0 tx=0x7effec009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:18.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.453+0000 7efff27fc700 1 -- 172.21.15.158:0/3828941583 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efff4056860 con 0x7efffc15da60 2023-10-16T11:19:18.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:18.661+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7efffc1f3400 con 0x7efffc15da60 2023-10-16T11:19:19.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.299+0000 7efff27fc700 1 -- 172.21.15.158:0/3828941583 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v211) v1 ==== 97+0+0 (secure 0 0 0) 0x7efff4059e80 con 0x7efffc15da60 2023-10-16T11:19:19.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.302+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7effe006b020 msgr2=0x7effe006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.302+0000 7f00041e7700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7effe006b020 0x7effe006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effec007fd0 tx=0x7effec009040 comp rx=0 tx=0).stop 2023-10-16T11:19:19.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.302+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efffc15da60 msgr2=0x7efffc1603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.302+0000 7f00041e7700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efffc15da60 0x7efffc1603c0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7efffc08cf60 tx=0x7efff4004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 shutdown_connections 2023-10-16T11:19:19.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7effe006b020 0x7effe006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efffc15da60 0x7efffc1603c0 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efffc0b5870 0x7efffc1f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 --2- 172.21.15.158:0/3828941583 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efffc0ce5b0 0x7efffc1e6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 >> 172.21.15.158:0/3828941583 conn(0x7efffc09c9b0 msgr2=0x7efffc0d2240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:19.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 shutdown_connections 2023-10-16T11:19:19.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.303+0000 7f00041e7700 1 -- 172.21.15.158:0/3828941583 wait complete. 2023-10-16T11:19:19.307 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T11:19:19.318 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T11:19:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.438+0000 7f0b3674c700 1 Processor -- start 2023-10-16T11:19:19.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.438+0000 7f0b3674c700 1 -- start start 2023-10-16T11:19:19.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b3674c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0b300cbe70 0x7f0b30165380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b3674c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b300cecd0 con 0x7f0b300aefa0 2023-10-16T11:19:19.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b3674c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3015b1c0 con 0x7f0b300c6140 2023-10-16T11:19:19.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b3674c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b300af450 con 0x7f0b300cbe70 2023-10-16T11:19:19.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0b300aefa0 0x7f0b300cb760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38028/0 (socket says 172.21.15.158:38028) 2023-10-16T11:19:19.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2f7fe700 1 -- 172.21.15.158:0/2884033989 learned_addr learned my addr 172.21.15.158:0/2884033989 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:19.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b34ce9700 1 --2- 172.21.15.158:0/2884033989 >> v2:172.21.15.172:3300/0 conn(0x7f0b300cbe70 0x7f0b30165380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667521420 0 0) 0x7f0b300cecd0 con 0x7f0b300aefa0 2023-10-16T11:19:19.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b180035b0 con 0x7f0b300aefa0 2023-10-16T11:19:19.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384435105 0 0) 0x7f0b180035b0 con 0x7f0b300aefa0 2023-10-16T11:19:19.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 >> v2:172.21.15.172:3300/0 conn(0x7f0b300cbe70 msgr2=0x7f0b30165380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:19.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 --2- 172.21.15.158:0/2884033989 >> v2:172.21.15.172:3300/0 conn(0x7f0b300cbe70 0x7f0b30165380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 >> v1:172.21.15.172:6789/0 conn(0x7f0b300c6140 legacy=0x7f0b3015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3015b4a0 con 0x7f0b300aefa0 2023-10-16T11:19:19.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0b24001670 con 0x7f0b300aefa0 2023-10-16T11:19:19.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 >> v1:172.21.15.158:6789/0 conn(0x7f0b300aefa0 legacy=0x7f0b300cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b18003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b18004850 0x7f0b18004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b18005210 0x7f0b1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1800c080 con 0x7f0b18003a00 2023-10-16T11:19:19.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1800c1f0 con 0x7f0b18005210 2023-10-16T11:19:19.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.439+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1800c3b0 con 0x7f0b18004850 2023-10-16T11:19:19.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2ffff700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b18003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2f7fe700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b18004850 0x7f0b18004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b34ce9700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b18005210 0x7f0b1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2ffff700 1 -- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b18004850 msgr2=0x7f0b18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:19.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2ffff700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b18004850 0x7f0b18004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2ffff700 1 -- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b18005210 msgr2=0x7f0b1800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:19.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2ffff700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b18005210 0x7f0b1800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2ffff700 1 -- 172.21.15.158:0/2884033989 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180035b0 con 0x7f0b18003a00 2023-10-16T11:19:19.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b34ce9700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b18005210 0x7f0b1800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:19.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2f7fe700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b18004850 0x7f0b18004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:19.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b34ce9700 1 -- 172.21.15.158:0/2884033989 reap_dead start 2023-10-16T11:19:19.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2ffff700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b18003e20 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f0b200017c0 tx=0x7f0b18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a3665ed4a20189 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:19.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200052d0 con 0x7f0b18003a00 2023-10-16T11:19:19.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b20005430 con 0x7f0b18003a00 2023-10-16T11:19:19.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.440+0000 7f0b2effd700 1 -- 172.21.15.158:0/2884033989 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200038c0 con 0x7f0b18003a00 2023-10-16T11:19:19.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.441+0000 7f0b3674c700 1 -- 172.21.15.158:0/2884033989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 msgr2=0x7f0b18003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.441+0000 7f0b3674c700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b18003e20 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f0b200017c0 tx=0x7f0b18005880 comp rx=0 tx=0).stop 2023-10-16T11:19:19.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.441+0000 7f0b3674c700 1 -- 172.21.15.158:0/2884033989 shutdown_connections 2023-10-16T11:19:19.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.441+0000 7f0b3674c700 1 --2- 172.21.15.158:0/2884033989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b18003e20 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.441+0000 7f0b3674c700 1 -- 172.21.15.158:0/2884033989 >> 172.21.15.158:0/2884033989 conn(0x7f0b300cc4b0 msgr2=0x7f0b300cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:19.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.441+0000 7f0b3674c700 1 -- 172.21.15.158:0/2884033989 shutdown_connections 2023-10-16T11:19:19.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.441+0000 7f0b3674c700 1 -- 172.21.15.158:0/2884033989 wait complete. 2023-10-16T11:19:19.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.442+0000 7f0b3674c700 1 Processor -- start 2023-10-16T11:19:19.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b3674c700 1 -- start start 2023-10-16T11:19:19.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b3674c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b300c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b3674c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3013e280 0x7f0b30140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b3674c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b30140fa0 0x7f0b3015cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b3674c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3013e700 con 0x7f0b18003a00 2023-10-16T11:19:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b3674c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30144d90 con 0x7f0b3013e280 2023-10-16T11:19:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b3674c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30141620 con 0x7f0b30140fa0 2023-10-16T11:19:19.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b2ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b300c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b2ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b300c9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37864/0 (socket says 172.21.15.158:37864) 2023-10-16T11:19:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.443+0000 7f0b2ffff700 1 -- 172.21.15.158:0/4227453146 learned_addr learned my addr 172.21.15.158:0/4227453146 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b34ce9700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b30140fa0 0x7f0b3015cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2f7fe700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3013e280 0x7f0b30140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2ffff700 1 -- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b30140fa0 msgr2=0x7f0b3015cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2ffff700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b30140fa0 0x7f0b3015cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2ffff700 1 -- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3013e280 msgr2=0x7f0b30140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2ffff700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3013e280 0x7f0b30140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2ffff700 1 -- 172.21.15.158:0/4227453146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b30141900 con 0x7f0b18003a00 2023-10-16T11:19:19.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2ffff700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b300c9820 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f0b20015040 tx=0x7f0b20001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:19.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2cff9700 1 -- 172.21.15.158:0/4227453146 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20001290 con 0x7f0b18003a00 2023-10-16T11:19:19.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2cff9700 1 -- 172.21.15.158:0/4227453146 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2001d480 con 0x7f0b18003a00 2023-10-16T11:19:19.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b2cff9700 1 -- 172.21.15.158:0/4227453146 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2001d830 con 0x7f0b18003a00 2023-10-16T11:19:19.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b300c8d40 con 0x7f0b18003a00 2023-10-16T11:19:19.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.444+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b300c9300 con 0x7f0b18003a00 2023-10-16T11:19:19.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.445+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b300c94c0 con 0x7f0b18003a00 2023-10-16T11:19:19.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.446+0000 7f0b2cff9700 1 -- 172.21.15.158:0/4227453146 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0b20019040 con 0x7f0b18003a00 2023-10-16T11:19:19.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.446+0000 7f0b2cff9700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0b1c06b020 0x7f0b1c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.446+0000 7f0b2f7fe700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0b1c06b020 0x7f0b1c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.446+0000 7f0b2cff9700 1 -- 172.21.15.158:0/4227453146 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f0b2008a2a0 con 0x7f0b18003a00 2023-10-16T11:19:19.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.446+0000 7f0b2f7fe700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0b1c06b020 0x7f0b1c06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0b24000c00 tx=0x7f0b24009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:19.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.451+0000 7f0b2cff9700 1 -- 172.21.15.158:0/4227453146 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b20055300 con 0x7f0b18003a00 2023-10-16T11:19:19.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.725+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0b300c8ba0 con 0x7f0b18003a00 2023-10-16T11:19:19.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.725+0000 7f0b2cff9700 1 -- 172.21.15.158:0/4227453146 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0b300c8ba0 con 0x7f0b18003a00 2023-10-16T11:19:19.727 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:19.727 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:19:19.728 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T11:19:19.728 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.728 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T11:19:19.728 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T11:19:19.729 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.729 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.729 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:19:19.729 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:19:19.729 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.730 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.730 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.730 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.730 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T11:19:19.730 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T11:19:19.731 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.731 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.731 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:19:19.731 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:19:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T11:19:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T11:19:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:19:19.734 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:19:19.734 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.734 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.734 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.734 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.735 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T11:19:19.735 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T11:19:19.735 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.735 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.735 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:19:19.736 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:19:19.736 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.736 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.736 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.736 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.737 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T11:19:19.737 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T11:19:19.737 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.737 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.737 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.738 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.738 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.738 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.738 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T11:19:19.739 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T11:19:19.739 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.739 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.739 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.739 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.740 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.740 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.740 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.740 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T11:19:19.740 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T11:19:19.741 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.741 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.741 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.741 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.741 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.741 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.742 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.742 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T11:19:19.742 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T11:19:19.743 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.743 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.743 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.743 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.743 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.744 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.744 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.744 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T11:19:19.744 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T11:19:19.744 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.745 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.745 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.745 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.745 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.745 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.746 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.746 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T11:19:19.746 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T11:19:19.746 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.746 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.747 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.747 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.747 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.747 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.747 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.748 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T11:19:19.748 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T11:19:19.748 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.748 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.748 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.749 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.749 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.749 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.749 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.749 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T11:19:19.750 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T11:19:19.750 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.750 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:19:19.750 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:19:19.750 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.751 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.751 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.751 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.751 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T11:19:19.751 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T11:19:19.752 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.752 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.752 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T11:19:19.752 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T11:19:19.753 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.753 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.753 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.753 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T11:19:19.754 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T11:19:19.754 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.755 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.755 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:19:19.756 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:19:19.756 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:19:19.756 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.756 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T11:19:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T11:19:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:19:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:19:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:19:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T11:19:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T11:19:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:19:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:19:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:19:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:19:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T11:19:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T11:19:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:19:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:19:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:19:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T11:19:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T11:19:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:19:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:19:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:19.765 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:19.765 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T11:19:19.765 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T11:19:19.765 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:19:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:19:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:19:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:19:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.767 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:19.767 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T11:19:19.767 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:19:19.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0b1c06b020 msgr2=0x7f0b1c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0b1c06b020 0x7f0b1c06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0b24000c00 tx=0x7f0b24009040 comp rx=0 tx=0).stop 2023-10-16T11:19:19.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 msgr2=0x7f0b300c9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b300c9820 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f0b20015040 tx=0x7f0b20001120 comp rx=0 tx=0).stop 2023-10-16T11:19:19.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 shutdown_connections 2023-10-16T11:19:19.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b30140fa0 0x7f0b3015cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f0b1c06b020 0x7f0b1c06d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b18003a00 0x7f0b300c9820 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 --2- 172.21.15.158:0/4227453146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3013e280 0x7f0b30140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 >> 172.21.15.158:0/4227453146 conn(0x7f0b300cc4b0 msgr2=0x7f0b30154490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:19.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 shutdown_connections 2023-10-16T11:19:19.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.727+0000 7f0b3674c700 1 -- 172.21.15.158:0/4227453146 wait complete. 2023-10-16T11:19:19.772 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:19.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.892+0000 7f99dae6c700 1 Processor -- start 2023-10-16T11:19:19.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.892+0000 7f99dae6c700 1 -- start start 2023-10-16T11:19:19.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99dae6c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99dae6c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99d406e5d0 0x7f99d4071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99dae6c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4071e50 con 0x7f99d406af20 2023-10-16T11:19:19.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99dae6c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d4161270 con 0x7f99d406e5d0 2023-10-16T11:19:19.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99dae6c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4161510 con 0x7f99d406ab40 2023-10-16T11:19:19.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d8c08700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99d406e5d0 0x7f99d4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d8c08700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:48704/0 (socket says 172.21.15.158:48704) 2023-10-16T11:19:19.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d8c08700 1 -- 172.21.15.158:0/966119464 learned_addr learned my addr 172.21.15.158:0/966119464 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:19.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d3fff700 1 -- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 msgr2=0x7f99d406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:19.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d3fff700 1 --2- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d3fff700 1 -- 172.21.15.158:0/966119464 >> v1:172.21.15.172:6789/0 conn(0x7f99d406af20 legacy=0x7f99d40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.893+0000 7f99d37fe700 1 -- 172.21.15.158:0/966119464 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220003594 0 0) 0x7f99d4071e50 con 0x7f99d406af20 2023-10-16T11:19:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99d3fff700 1 -- 172.21.15.158:0/966119464 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d41617f0 con 0x7f99d406e5d0 2023-10-16T11:19:19.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99d8c08700 1 --2- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:19.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99d3fff700 1 --2- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406e5d0 0x7f99d4071910 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009840 tx=0x7f99c0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d38b2335d9403917 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:19.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99d37fe700 1 -- 172.21.15.158:0/966119464 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0011830 con 0x7f99d406e5d0 2023-10-16T11:19:19.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99d37fe700 1 -- 172.21.15.158:0/966119464 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c0011990 con 0x7f99d406e5d0 2023-10-16T11:19:19.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99d37fe700 1 -- 172.21.15.158:0/966119464 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0011d20 con 0x7f99d406e5d0 2023-10-16T11:19:19.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 -- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406e5d0 msgr2=0x7f99d4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 --2- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406e5d0 0x7f99d4071910 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009840 tx=0x7f99c0009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:19.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 -- 172.21.15.158:0/966119464 shutdown_connections 2023-10-16T11:19:19.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 --2- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 --2- 172.21.15.158:0/966119464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406e5d0 0x7f99d4071910 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 -- 172.21.15.158:0/966119464 >> 172.21.15.158:0/966119464 conn(0x7f99d40d02b0 msgr2=0x7f99d40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 -- 172.21.15.158:0/966119464 shutdown_connections 2023-10-16T11:19:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.894+0000 7f99dae6c700 1 -- 172.21.15.158:0/966119464 wait complete. 2023-10-16T11:19:19.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.896+0000 7f99dae6c700 1 Processor -- start 2023-10-16T11:19:19.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.896+0000 7f99dae6c700 1 -- start start 2023-10-16T11:19:19.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.896+0000 7f99dae6c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d40c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99dae6c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406af20 0x7f99d41f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99dae6c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d406e5d0 0x7f99d41f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99dae6c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40c9a50 con 0x7f99d406e5d0 2023-10-16T11:19:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99dae6c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d41f2a70 con 0x7f99d406af20 2023-10-16T11:19:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99dae6c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40c8dd0 con 0x7f99d406ab40 2023-10-16T11:19:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d406e5d0 0x7f99d41f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d406e5d0 0x7f99d41f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37880/0 (socket says 172.21.15.158:37880) 2023-10-16T11:19:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 -- 172.21.15.158:0/780400198 learned_addr learned my addr 172.21.15.158:0/780400198 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d3fff700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406af20 0x7f99d41f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d8c08700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d40c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 -- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 msgr2=0x7f99d40c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d40c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 -- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406af20 msgr2=0x7f99d41f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406af20 0x7f99d41f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:19.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 -- 172.21.15.158:0/780400198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8009550 con 0x7f99d406e5d0 2023-10-16T11:19:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d9409700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d406e5d0 0x7f99d41f34b0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f99cc001560 tx=0x7f99cc00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d17fa700 1 -- 172.21.15.158:0/780400198 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc003e30 con 0x7f99d406e5d0 2023-10-16T11:19:19.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d17fa700 1 -- 172.21.15.158:0/780400198 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc005070 con 0x7f99d406e5d0 2023-10-16T11:19:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c00094a0 con 0x7f99d406e5d0 2023-10-16T11:19:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99d17fa700 1 -- 172.21.15.158:0/780400198 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc005420 con 0x7f99d406e5d0 2023-10-16T11:19:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.897+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d40c9390 con 0x7f99d406e5d0 2023-10-16T11:19:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.898+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d40681b0 con 0x7f99d406e5d0 2023-10-16T11:19:19.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.899+0000 7f99d17fa700 1 -- 172.21.15.158:0/780400198 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f99cc005580 con 0x7f99d406e5d0 2023-10-16T11:19:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.899+0000 7f99d17fa700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99b806b020 0x7f99b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.899+0000 7f99d17fa700 1 -- 172.21.15.158:0/780400198 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f99cc088cc0 con 0x7f99d406e5d0 2023-10-16T11:19:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.899+0000 7f99d8c08700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99b806b020 0x7f99b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:19.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.900+0000 7f99d8c08700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99b806b020 0x7f99b806d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f99c8006d30 tx=0x7f99c80092c0 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:19.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:19.904+0000 7f99d17fa700 1 -- 172.21.15.158:0/780400198 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99cc053d20 con 0x7f99d406e5d0 2023-10-16T11:19:20.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.096+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99d40b6230 con 0x7f99d406e5d0 2023-10-16T11:19:20.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.097+0000 7f99d17fa700 1 -- 172.21.15.158:0/780400198 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+13819 (secure 0 0 0) 0x7f99cc017070 con 0x7f99d406e5d0 2023-10-16T11:19:20.099 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:20.099 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":213,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:19.309859+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:20.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99b806b020 msgr2=0x7f99b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99b806b020 0x7f99b806d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f99c8006d30 tx=0x7f99c80092c0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d406e5d0 msgr2=0x7f99d41f34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d406e5d0 0x7f99d41f34b0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f99cc001560 tx=0x7f99cc00cb50 comp rx=0 tx=0).stop 2023-10-16T11:19:20.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 shutdown_connections 2023-10-16T11:19:20.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99b806b020 0x7f99b806d4e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d406e5d0 0x7f99d41f34b0 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99d406ab40 0x7f99d40c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 --2- 172.21.15.158:0/780400198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d406af20 0x7f99d41f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 >> 172.21.15.158:0/780400198 conn(0x7f99d40d02b0 msgr2=0x7f99d40af040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:20.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.099+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 shutdown_connections 2023-10-16T11:19:20.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.100+0000 7f99dae6c700 1 -- 172.21.15.158:0/780400198 wait complete. 2023-10-16T11:19:20.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:19:20.114 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:19:20.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.235+0000 7efe72610700 1 Processor -- start 2023-10-16T11:19:20.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.236+0000 7efe72610700 1 -- start start 2023-10-16T11:19:20.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.236+0000 7efe72610700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efe6c15d7b0 0x7efe6c0721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.236+0000 7efe72610700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efe6c0b2a40 0x7efe6c0ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.236+0000 7efe72610700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c0b3dc0 con 0x7efe6c06b8e0 2023-10-16T11:19:20.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.236+0000 7efe72610700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6c15db90 con 0x7efe6c0b2a40 2023-10-16T11:19:20.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.236+0000 7efe72610700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6c0b31b0 con 0x7efe6c15d7b0 2023-10-16T11:19:20.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.236+0000 7efe6b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efe6c0b2a40 0x7efe6c0ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe70bad700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efe6c06b8e0 0x7efe6c0b4250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38062/0 (socket says 172.21.15.158:38062) 2023-10-16T11:19:20.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efe6c0b2a40 0x7efe6c0ac9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37888/0 (socket says 172.21.15.158:37888) 2023-10-16T11:19:20.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe70bad700 1 -- 172.21.15.158:0/3365856214 learned_addr learned my addr 172.21.15.158:0/3365856214 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:20.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6b7fe700 1 -- 172.21.15.158:0/3365856214 >> v2:172.21.15.172:3300/0 conn(0x7efe6c15d7b0 msgr2=0x7efe6c0721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:20.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6bfff700 1 --2- 172.21.15.158:0/3365856214 >> v2:172.21.15.172:3300/0 conn(0x7efe6c15d7b0 0x7efe6c0721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6b7fe700 1 --2- 172.21.15.158:0/3365856214 >> v2:172.21.15.172:3300/0 conn(0x7efe6c15d7b0 0x7efe6c0721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6affd700 1 -- 172.21.15.158:0/3365856214 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201926252 0 0) 0x7efe6c0b3dc0 con 0x7efe6c06b8e0 2023-10-16T11:19:20.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6b7fe700 1 -- 172.21.15.158:0/3365856214 >> v1:172.21.15.158:6789/0 conn(0x7efe6c06b8e0 legacy=0x7efe6c0b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6b7fe700 1 -- 172.21.15.158:0/3365856214 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c0b3490 con 0x7efe6c0b2a40 2023-10-16T11:19:20.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6b7fe700 1 --2- 172.21.15.158:0/3365856214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c0b2a40 0x7efe6c0ac9d0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7efe6c066650 tx=0x7efe60009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6db5a3e7ac296a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6affd700 1 -- 172.21.15.158:0/3365856214 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60011830 con 0x7efe6c0b2a40 2023-10-16T11:19:20.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6affd700 1 -- 172.21.15.158:0/3365856214 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60011990 con 0x7efe6c0b2a40 2023-10-16T11:19:20.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.237+0000 7efe6affd700 1 -- 172.21.15.158:0/3365856214 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60011d20 con 0x7efe6c0b2a40 2023-10-16T11:19:20.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 -- 172.21.15.158:0/3365856214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c0b2a40 msgr2=0x7efe6c0ac9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 --2- 172.21.15.158:0/3365856214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c0b2a40 0x7efe6c0ac9d0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7efe6c066650 tx=0x7efe60009b20 comp rx=0 tx=0).stop 2023-10-16T11:19:20.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 -- 172.21.15.158:0/3365856214 shutdown_connections 2023-10-16T11:19:20.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 --2- 172.21.15.158:0/3365856214 >> v2:172.21.15.172:3300/0 conn(0x7efe6c15d7b0 0x7efe6c0721d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 --2- 172.21.15.158:0/3365856214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c0b2a40 0x7efe6c0ac9d0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 -- 172.21.15.158:0/3365856214 >> 172.21.15.158:0/3365856214 conn(0x7efe6c071110 msgr2=0x7efe6c071520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:20.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 -- 172.21.15.158:0/3365856214 shutdown_connections 2023-10-16T11:19:20.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.238+0000 7efe72610700 1 -- 172.21.15.158:0/3365856214 wait complete. 2023-10-16T11:19:20.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.239+0000 7efe72610700 1 Processor -- start 2023-10-16T11:19:20.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.240+0000 7efe72610700 1 -- start start 2023-10-16T11:19:20.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.240+0000 7efe72610700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe6c06b8e0 0x7efe6c14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.240+0000 7efe72610700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe6c0b2a40 0x7efe6c159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.240+0000 7efe72610700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c15d7b0 0x7efe6c1614f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.240+0000 7efe72610700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c159d40 con 0x7efe6c15d7b0 2023-10-16T11:19:20.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.240+0000 7efe72610700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c15a020 con 0x7efe6c06b8e0 2023-10-16T11:19:20.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.240+0000 7efe72610700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c15a300 con 0x7efe6c0b2a40 2023-10-16T11:19:20.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c15d7b0 0x7efe6c1614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c15d7b0 0x7efe6c1614f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37894/0 (socket says 172.21.15.158:37894) 2023-10-16T11:19:20.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 -- 172.21.15.158:0/3763223412 learned_addr learned my addr 172.21.15.158:0/3763223412 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:20.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe6bfff700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe6c06b8e0 0x7efe6c14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe6b7fe700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe6c0b2a40 0x7efe6c159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 -- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe6c0b2a40 msgr2=0x7efe6c159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe6c0b2a40 0x7efe6c159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 -- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe6c06b8e0 msgr2=0x7efe6c14e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe6c06b8e0 0x7efe6c14e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 -- 172.21.15.158:0/3763223412 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe600094a0 con 0x7efe6c15d7b0 2023-10-16T11:19:20.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe6bfff700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe6c06b8e0 0x7efe6c14e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:20.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe70bad700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c15d7b0 0x7efe6c1614f0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7efe64002230 tx=0x7efe6400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe68ff9700 1 -- 172.21.15.158:0/3763223412 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6400ac20 con 0x7efe6c15d7b0 2023-10-16T11:19:20.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe68ff9700 1 -- 172.21.15.158:0/3763223412 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6400ada0 con 0x7efe6c15d7b0 2023-10-16T11:19:20.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe68ff9700 1 -- 172.21.15.158:0/3763223412 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64007310 con 0x7efe6c15d7b0 2023-10-16T11:19:20.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c15ee50 con 0x7efe6c15d7b0 2023-10-16T11:19:20.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.241+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c151f00 con 0x7efe6c15d7b0 2023-10-16T11:19:20.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.242+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c152120 con 0x7efe6c15d7b0 2023-10-16T11:19:20.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.243+0000 7efe68ff9700 1 -- 172.21.15.158:0/3763223412 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efe64007470 con 0x7efe6c15d7b0 2023-10-16T11:19:20.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.243+0000 7efe68ff9700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7efe5806b020 0x7efe5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.243+0000 7efe68ff9700 1 -- 172.21.15.158:0/3763223412 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5657+0+0 (secure 0 0 0) 0x7efe64089230 con 0x7efe6c15d7b0 2023-10-16T11:19:20.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.243+0000 7efe6bfff700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7efe5806b020 0x7efe5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.243+0000 7efe6bfff700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7efe5806b020 0x7efe5806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe5c006fd0 tx=0x7efe5c008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.247+0000 7efe68ff9700 1 -- 172.21.15.158:0/3763223412 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe6c152120 con 0x7efe6c15d7b0 2023-10-16T11:19:20.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.443+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe6c152120 con 0x7efe6c15d7b0 2023-10-16T11:19:20.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.443+0000 7efe68ff9700 1 -- 172.21.15.158:0/3763223412 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v213) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe6c152120 con 0x7efe6c15d7b0 2023-10-16T11:19:20.445 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:19:20.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7efe5806b020 msgr2=0x7efe5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7efe5806b020 0x7efe5806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe5c006fd0 tx=0x7efe5c008040 comp rx=0 tx=0).stop 2023-10-16T11:19:20.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c15d7b0 msgr2=0x7efe6c1614f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c15d7b0 0x7efe6c1614f0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7efe64002230 tx=0x7efe6400a060 comp rx=0 tx=0).stop 2023-10-16T11:19:20.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 shutdown_connections 2023-10-16T11:19:20.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7efe5806b020 0x7efe5806d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe6c15d7b0 0x7efe6c1614f0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe6c06b8e0 0x7efe6c14e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 --2- 172.21.15.158:0/3763223412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe6c0b2a40 0x7efe6c159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.445+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 >> 172.21.15.158:0/3763223412 conn(0x7efe6c071110 msgr2=0x7efe6c157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:20.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.446+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 shutdown_connections 2023-10-16T11:19:20.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.446+0000 7efe72610700 1 -- 172.21.15.158:0/3763223412 wait complete. 2023-10-16T11:19:20.459 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:19:20.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.578+0000 7f47fc322700 1 Processor -- start 2023-10-16T11:19:20.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.578+0000 7f47fc322700 1 -- start start 2023-10-16T11:19:20.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47fc322700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 0x7f47f41521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47fc322700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f415c5d0 con 0x7f47f40cdee0 2023-10-16T11:19:20.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47fc322700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f415c8b0 con 0x7f47f40ce2c0 2023-10-16T11:19:20.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47fc322700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415d2c0 con 0x7f47f40b6e10 2023-10-16T11:19:20.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47fa0be700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f47f40cdee0 0x7f47f415bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38076/0 (socket says 172.21.15.158:38076) 2023-10-16T11:19:20.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47fa0be700 1 -- 172.21.15.158:0/3606010072 learned_addr learned my addr 172.21.15.158:0/3606010072 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:20.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153873826 0 0) 0x7f47f415c5d0 con 0x7f47f40cdee0 2023-10-16T11:19:20.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e00035b0 con 0x7f47f40cdee0 2023-10-16T11:19:20.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f98bd700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 0x7f47f41521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3676771776 0 0) 0x7f47e00035b0 con 0x7f47f40cdee0 2023-10-16T11:19:20.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 msgr2=0x7f47f41521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:20.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 0x7f47f41521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 >> v1:172.21.15.172:6789/0 conn(0x7f47f40ce2c0 legacy=0x7f47f40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f415cb90 con 0x7f47f40cdee0 2023-10-16T11:19:20.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.579+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f47f0001670 con 0x7f47f40cdee0 2023-10-16T11:19:20.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 >> v1:172.21.15.158:6789/0 conn(0x7f47f40cdee0 legacy=0x7f47f415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47e0003a30 0x7f47e0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47e0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 0x7f47e0005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415d2c0 con 0x7f47e0004850 2023-10-16T11:19:20.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e000bb70 con 0x7f47e0003a30 2023-10-16T11:19:20.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e000bd30 con 0x7f47f40b6e10 2023-10-16T11:19:20.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f98bd700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47e0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47fa0be700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47e0003a30 0x7f47e0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47fa8bf700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 0x7f47e0005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f98bd700 1 -- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 msgr2=0x7f47e0005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:20.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f98bd700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b6e10 0x7f47e0005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f98bd700 1 -- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47e0003a30 msgr2=0x7f47e0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:20.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f98bd700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47e0003a30 0x7f47e0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f98bd700 1 -- 172.21.15.158:0/3606010072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e00035b0 con 0x7f47e0004850 2023-10-16T11:19:20.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f98bd700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47e0004cd0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009b60 tx=0x7f47e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7883f20a5b3a086 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e400e6c0 con 0x7f47e0004850 2023-10-16T11:19:20.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e400e820 con 0x7f47e0004850 2023-10-16T11:19:20.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.580+0000 7f47f90bc700 1 -- 172.21.15.158:0/3606010072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e400eb90 con 0x7f47e0004850 2023-10-16T11:19:20.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.581+0000 7f47fc322700 1 -- 172.21.15.158:0/3606010072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 msgr2=0x7f47e0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.581+0000 7f47fc322700 1 --2- 172.21.15.158:0/3606010072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47e0004cd0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009b60 tx=0x7f47e0005880 comp rx=0 tx=0).stop 2023-10-16T11:19:20.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.581+0000 7f47fa8bf700 1 -- 172.21.15.158:0/3606010072 reap_dead start 2023-10-16T11:19:20.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.581+0000 7f47fc322700 1 -- 172.21.15.158:0/3606010072 shutdown_connections 2023-10-16T11:19:20.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.581+0000 7f47fc322700 1 -- 172.21.15.158:0/3606010072 >> 172.21.15.158:0/3606010072 conn(0x7f47f406a8d0 msgr2=0x7f47f406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:20.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.581+0000 7f47fc322700 1 -- 172.21.15.158:0/3606010072 shutdown_connections 2023-10-16T11:19:20.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.581+0000 7f47fc322700 1 -- 172.21.15.158:0/3606010072 wait complete. 2023-10-16T11:19:20.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.582+0000 7f47fc322700 1 Processor -- start 2023-10-16T11:19:20.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.583+0000 7f47fc322700 1 -- start start 2023-10-16T11:19:20.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.583+0000 7f47fc322700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47f40c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.583+0000 7f47fc322700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40a8640 0x7f47f40a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.583+0000 7f47fc322700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f4147ff0 0x7f47f4148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.583+0000 7f47fc322700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40a8fe0 con 0x7f47e0004850 2023-10-16T11:19:20.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.583+0000 7f47fc322700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f4149d80 con 0x7f47f40a8640 2023-10-16T11:19:20.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.583+0000 7f47fc322700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f4148ce0 con 0x7f47f4147ff0 2023-10-16T11:19:20.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47f40c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47f40c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37912/0 (socket says 172.21.15.158:37912) 2023-10-16T11:19:20.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 -- 172.21.15.158:0/4204805493 learned_addr learned my addr 172.21.15.158:0/4204805493 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:20.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 -- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f4147ff0 msgr2=0x7f47f4148480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa8bf700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f4147ff0 0x7f47f4148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47f98bd700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40a8640 0x7f47f40a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f4147ff0 0x7f47f4148480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 -- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40a8640 msgr2=0x7f47f40a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40a8640 0x7f47f40a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 -- 172.21.15.158:0/4204805493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f4148fc0 con 0x7f47e0004850 2023-10-16T11:19:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fa0be700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47f40c78e0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f47f00072c0 tx=0x7f47f0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47eaffd700 1 -- 172.21.15.158:0/4204805493 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0001450 con 0x7f47e0004850 2023-10-16T11:19:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47eaffd700 1 -- 172.21.15.158:0/4204805493 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f000f850 con 0x7f47e0004850 2023-10-16T11:19:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47eaffd700 1 -- 172.21.15.158:0/4204805493 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f000fc00 con 0x7f47e0004850 2023-10-16T11:19:20.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f41492a0 con 0x7f47e0004850 2023-10-16T11:19:20.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.584+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f4149860 con 0x7f47e0004850 2023-10-16T11:19:20.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.585+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f40681b0 con 0x7f47e0004850 2023-10-16T11:19:20.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.586+0000 7f47eaffd700 1 -- 172.21.15.158:0/4204805493 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f47f000fd60 con 0x7f47e0004850 2023-10-16T11:19:20.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.586+0000 7f47eaffd700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f47d806b020 0x7f47d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.586+0000 7f47eaffd700 1 -- 172.21.15.158:0/4204805493 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f47f00897e0 con 0x7f47e0004850 2023-10-16T11:19:20.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.586+0000 7f47f98bd700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f47d806b020 0x7f47d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.586+0000 7f47f98bd700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f47d806b020 0x7f47d806d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f47e4000c00 tx=0x7f47e4008030 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.590+0000 7f47eaffd700 1 -- 172.21.15.158:0/4204805493 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47f00547b0 con 0x7f47e0004850 2023-10-16T11:19:20.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.787+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f47f40b3bf0 con 0x7f47e0004850 2023-10-16T11:19:20.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.787+0000 7f47eaffd700 1 -- 172.21.15.158:0/4204805493 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v213) v1 ==== 84+0+65 (secure 0 0 0) 0x7f47f0057dd0 con 0x7f47e0004850 2023-10-16T11:19:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:20.789 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:19:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:19:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:19:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:20.790 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:19:20.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f47d806b020 msgr2=0x7f47d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f47d806b020 0x7f47d806d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f47e4000c00 tx=0x7f47e4008030 comp rx=0 tx=0).stop 2023-10-16T11:19:20.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 msgr2=0x7f47f40c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47f40c78e0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f47f00072c0 tx=0x7f47f0007680 comp rx=0 tx=0).stop 2023-10-16T11:19:20.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 shutdown_connections 2023-10-16T11:19:20.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f4147ff0 0x7f47f4148480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f47d806b020 0x7f47d806d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47e0004850 0x7f47f40c78e0 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 --2- 172.21.15.158:0/4204805493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40a8640 0x7f47f40a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 >> 172.21.15.158:0/4204805493 conn(0x7f47f406a8d0 msgr2=0x7f47f415a310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:20.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 shutdown_connections 2023-10-16T11:19:20.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.789+0000 7f47fc322700 1 -- 172.21.15.158:0/4204805493 wait complete. 2023-10-16T11:19:20.803 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T11:19:20.803 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T11:19:20.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.926+0000 7f6961f82700 1 Processor -- start 2023-10-16T11:19:20.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.926+0000 7f6961f82700 1 -- start start 2023-10-16T11:19:20.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f6961f82700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c15da40 0x7f695c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f6961f82700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f695c06afe0 0x7f695c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f6961f82700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c0cbd30 con 0x7f695c0cc590 2023-10-16T11:19:20.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f6961f82700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c0cc1c0 con 0x7f695c06afe0 2023-10-16T11:19:20.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f6961f82700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c15de20 con 0x7f695c15da40 2023-10-16T11:19:20.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f695bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f695c06afe0 0x7f695c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f695bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f695c06afe0 0x7f695c0afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37924/0 (socket says 172.21.15.158:37924) 2023-10-16T11:19:20.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f695bfff700 1 -- 172.21.15.158:0/3007691359 learned_addr learned my addr 172.21.15.158:0/3007691359 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:20.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.927+0000 7f695b7fe700 1 --2- 172.21.15.158:0/3007691359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c15da40 0x7f695c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695bfff700 1 -- 172.21.15.158:0/3007691359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c15da40 msgr2=0x7f695c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:20.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695bfff700 1 --2- 172.21.15.158:0/3007691359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c15da40 0x7f695c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695bfff700 1 -- 172.21.15.158:0/3007691359 >> v1:172.21.15.172:6789/0 conn(0x7f695c0cc590 legacy=0x7f695c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:20.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695bfff700 1 -- 172.21.15.158:0/3007691359 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0ce260 con 0x7f695c06afe0 2023-10-16T11:19:20.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695bfff700 1 --2- 172.21.15.158:0/3007691359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c06afe0 0x7f695c0afa40 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f6954009e50 tx=0x7f695400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65abcd77cdcf3b93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695a7fc700 1 -- 172.21.15.158:0/3007691359 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695400e070 con 0x7f695c06afe0 2023-10-16T11:19:20.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695a7fc700 1 -- 172.21.15.158:0/3007691359 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6954003a20 con 0x7f695c06afe0 2023-10-16T11:19:20.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f695a7fc700 1 -- 172.21.15.158:0/3007691359 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954003dd0 con 0x7f695c06afe0 2023-10-16T11:19:20.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f6961f82700 1 -- 172.21.15.158:0/3007691359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c06afe0 msgr2=0x7f695c0afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f6961f82700 1 --2- 172.21.15.158:0/3007691359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c06afe0 0x7f695c0afa40 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f6954009e50 tx=0x7f695400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:20.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f6961f82700 1 -- 172.21.15.158:0/3007691359 shutdown_connections 2023-10-16T11:19:20.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f6961f82700 1 --2- 172.21.15.158:0/3007691359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c06afe0 0x7f695c0afa40 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.928+0000 7f6961f82700 1 --2- 172.21.15.158:0/3007691359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c15da40 0x7f695c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.929+0000 7f6961f82700 1 -- 172.21.15.158:0/3007691359 >> 172.21.15.158:0/3007691359 conn(0x7f695c099a10 msgr2=0x7f695c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:20.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.929+0000 7f6961f82700 1 -- 172.21.15.158:0/3007691359 shutdown_connections 2023-10-16T11:19:20.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.929+0000 7f6961f82700 1 -- 172.21.15.158:0/3007691359 wait complete. 2023-10-16T11:19:20.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.930+0000 7f6961f82700 1 Processor -- start 2023-10-16T11:19:20.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.930+0000 7f6961f82700 1 -- start start 2023-10-16T11:19:20.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f695c06afe0 0x7f695c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c0cc590 0x7f695c0b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c15da40 0x7f695c1f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c7a60 con 0x7f695c15da40 2023-10-16T11:19:20.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c7d40 con 0x7f695c06afe0 2023-10-16T11:19:20.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c8020 con 0x7f695c0cc590 2023-10-16T11:19:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c15da40 0x7f695c1f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c15da40 0x7f695c1f32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37930/0 (socket says 172.21.15.158:37930) 2023-10-16T11:19:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 -- 172.21.15.158:0/99498324 learned_addr learned my addr 172.21.15.158:0/99498324 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695b7fe700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f695c06afe0 0x7f695c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695affd700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c0cc590 0x7f695c0b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 -- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c0cc590 msgr2=0x7f695c0b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c0cc590 0x7f695c0b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 -- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f695c06afe0 msgr2=0x7f695c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f695c06afe0 0x7f695c0b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:20.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 -- 172.21.15.158:0/99498324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954009b30 con 0x7f695c15da40 2023-10-16T11:19:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f695bfff700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c15da40 0x7f695c1f32a0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f695c0702f0 tx=0x7f6954004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6943fff700 1 -- 172.21.15.158:0/99498324 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954004440 con 0x7f695c15da40 2023-10-16T11:19:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6943fff700 1 -- 172.21.15.158:0/99498324 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69540045c0 con 0x7f695c15da40 2023-10-16T11:19:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c0c8300 con 0x7f695c15da40 2023-10-16T11:19:20.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6943fff700 1 -- 172.21.15.158:0/99498324 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954004970 con 0x7f695c15da40 2023-10-16T11:19:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.931+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c0c74a0 con 0x7f695c15da40 2023-10-16T11:19:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.932+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c0c87c0 con 0x7f695c15da40 2023-10-16T11:19:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.933+0000 7f6943fff700 1 -- 172.21.15.158:0/99498324 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6954012430 con 0x7f695c15da40 2023-10-16T11:19:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.933+0000 7f6943fff700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f694806b020 0x7f694806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.933+0000 7f695b7fe700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f694806b020 0x7f694806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:20.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.933+0000 7f6943fff700 1 -- 172.21.15.158:0/99498324 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f695408af40 con 0x7f695c15da40 2023-10-16T11:19:20.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.934+0000 7f695b7fe700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f694806b020 0x7f694806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6950006fd0 tx=0x7f6950008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:20.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:20.938+0000 7f6943fff700 1 -- 172.21.15.158:0/99498324 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69540598a0 con 0x7f695c15da40 2023-10-16T11:19:21.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.129+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f695c0681b0 con 0x7f695c15da40 2023-10-16T11:19:21.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.306+0000 7f6943fff700 1 -- 172.21.15.158:0/99498324 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v214) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6954059430 con 0x7f695c15da40 2023-10-16T11:19:21.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f694806b020 msgr2=0x7f694806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:21.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f694806b020 0x7f694806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6950006fd0 tx=0x7f6950008040 comp rx=0 tx=0).stop 2023-10-16T11:19:21.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c15da40 msgr2=0x7f695c1f32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c15da40 0x7f695c1f32a0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f695c0702f0 tx=0x7f6954004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 shutdown_connections 2023-10-16T11:19:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f694806b020 0x7f694806d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f695c15da40 0x7f695c1f32a0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f695c06afe0 0x7f695c0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 --2- 172.21.15.158:0/99498324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f695c0cc590 0x7f695c0b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 >> 172.21.15.158:0/99498324 conn(0x7f695c099a10 msgr2=0x7f695c0b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 shutdown_connections 2023-10-16T11:19:21.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.308+0000 7f6961f82700 1 -- 172.21.15.158:0/99498324 wait complete. 2023-10-16T11:19:21.313 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T11:19:21.326 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:19:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.452+0000 7fa8f415b700 1 Processor -- start 2023-10-16T11:19:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.452+0000 7fa8f415b700 1 -- start start 2023-10-16T11:19:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f415b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec072b60 0x7fa8ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f415b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8ec06d960 0x7fa8ec0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:21.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f415b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec0b5040 con 0x7fa8ec15a780 2023-10-16T11:19:21.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f415b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec06dde0 con 0x7fa8ec06d960 2023-10-16T11:19:21.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f415b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec15e190 con 0x7fa8ec072b60 2023-10-16T11:19:21.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f26f8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8ec06d960 0x7fa8ec0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:21.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f1ef7700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa8ec15a780 0x7fa8ec0b4930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38110/0 (socket says 172.21.15.158:38110) 2023-10-16T11:19:21.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f1ef7700 1 -- 172.21.15.158:0/6717009 learned_addr learned my addr 172.21.15.158:0/6717009 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:21.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f16f6700 1 --2- 172.21.15.158:0/6717009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec072b60 0x7fa8ec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:21.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f26f8700 1 -- 172.21.15.158:0/6717009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec072b60 msgr2=0x7fa8ec06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:21.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f26f8700 1 --2- 172.21.15.158:0/6717009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec072b60 0x7fa8ec06d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f26f8700 1 -- 172.21.15.158:0/6717009 >> v1:172.21.15.158:6789/0 conn(0x7fa8ec15a780 legacy=0x7fa8ec0b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:21.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f0ef5700 1 -- 172.21.15.158:0/6717009 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730027252 0 0) 0x7fa8ec0b5040 con 0x7fa8ec15a780 2023-10-16T11:19:21.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.453+0000 7fa8f26f8700 1 -- 172.21.15.158:0/6717009 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec15e470 con 0x7fa8ec06d960 2023-10-16T11:19:21.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f26f8700 1 --2- 172.21.15.158:0/6717009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0b3590 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400bfa0 tx=0x7fa8e400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4773b7305e31d3c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:21.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f0ef5700 1 -- 172.21.15.158:0/6717009 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4010070 con 0x7fa8ec06d960 2023-10-16T11:19:21.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f0ef5700 1 -- 172.21.15.158:0/6717009 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e400b2e0 con 0x7fa8ec06d960 2023-10-16T11:19:21.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f0ef5700 1 -- 172.21.15.158:0/6717009 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e400dcf0 con 0x7fa8ec06d960 2023-10-16T11:19:21.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 -- 172.21.15.158:0/6717009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 msgr2=0x7fa8ec0b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:21.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 --2- 172.21.15.158:0/6717009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0b3590 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400bfa0 tx=0x7fa8e400d330 comp rx=0 tx=0).stop 2023-10-16T11:19:21.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 -- 172.21.15.158:0/6717009 shutdown_connections 2023-10-16T11:19:21.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 --2- 172.21.15.158:0/6717009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0b3590 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 --2- 172.21.15.158:0/6717009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec072b60 0x7fa8ec06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 -- 172.21.15.158:0/6717009 >> 172.21.15.158:0/6717009 conn(0x7fa8ec0ccd30 msgr2=0x7fa8ec0cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:21.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 -- 172.21.15.158:0/6717009 shutdown_connections 2023-10-16T11:19:21.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.454+0000 7fa8f415b700 1 -- 172.21.15.158:0/6717009 wait complete. 2023-10-16T11:19:21.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 Processor -- start 2023-10-16T11:19:21.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 -- start start 2023-10-16T11:19:21.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:21.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8ec072b60 0x7fa8ec1554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:21.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec15a780 0x7fa8ec14d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:21.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec155a20 con 0x7fa8ec06d960 2023-10-16T11:19:21.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0ae710 con 0x7fa8ec072b60 2023-10-16T11:19:21.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f415b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0ad950 con 0x7fa8ec15a780 2023-10-16T11:19:21.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f1ef7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:21.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f1ef7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0acfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37940/0 (socket says 172.21.15.158:37940) 2023-10-16T11:19:21.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.456+0000 7fa8f1ef7700 1 -- 172.21.15.158:0/2563190776 learned_addr learned my addr 172.21.15.158:0/2563190776 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:21.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f26f8700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec15a780 0x7fa8ec14d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:21.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f16f6700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8ec072b60 0x7fa8ec1554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:21.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f1ef7700 1 -- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec15a780 msgr2=0x7fa8ec14d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:21.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f1ef7700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec15a780 0x7fa8ec14d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f1ef7700 1 -- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8ec072b60 msgr2=0x7fa8ec1554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:21.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f1ef7700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8ec072b60 0x7fa8ec1554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:21.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f1ef7700 1 -- 172.21.15.158:0/2563190776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e400bc80 con 0x7fa8ec06d960 2023-10-16T11:19:21.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f1ef7700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0acfc0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c550 tx=0x7fa8e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:21.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8de7fc700 1 -- 172.21.15.158:0/2563190776 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0004c80 con 0x7fa8ec06d960 2023-10-16T11:19:21.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8de7fc700 1 -- 172.21.15.158:0/2563190776 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0004e00 con 0x7fa8ec06d960 2023-10-16T11:19:21.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec0adc30 con 0x7fa8ec06d960 2023-10-16T11:19:21.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8de7fc700 1 -- 172.21.15.158:0/2563190776 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0007310 con 0x7fa8ec06d960 2023-10-16T11:19:21.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.457+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec0ae1f0 con 0x7fa8ec06d960 2023-10-16T11:19:21.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.458+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec0681b0 con 0x7fa8ec06d960 2023-10-16T11:19:21.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.459+0000 7fa8de7fc700 1 -- 172.21.15.158:0/2563190776 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa8e0007470 con 0x7fa8ec06d960 2023-10-16T11:19:21.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.459+0000 7fa8de7fc700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa8d006b020 0x7fa8d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:21.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.459+0000 7fa8de7fc700 1 -- 172.21.15.158:0/2563190776 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6027+0+0 (secure 0 0 0) 0x7fa8e0089570 con 0x7fa8ec06d960 2023-10-16T11:19:21.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.459+0000 7fa8f16f6700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa8d006b020 0x7fa8d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:21.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.459+0000 7fa8f16f6700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa8d006b020 0x7fa8d006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0cff70 tx=0x7fa8e8008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:21.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.464+0000 7fa8de7fc700 1 -- 172.21.15.158:0/2563190776 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8e0054460 con 0x7fa8ec06d960 2023-10-16T11:19:21.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:21.656+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa8ec0728b0 con 0x7fa8ec06d960 2023-10-16T11:19:22.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.312+0000 7fa8de7fc700 1 -- 172.21.15.158:0/2563190776 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v215) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa8e0057a80 con 0x7fa8ec06d960 2023-10-16T11:19:22.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.316+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa8d006b020 msgr2=0x7fa8d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:22.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.316+0000 7fa8f415b700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa8d006b020 0x7fa8d006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0cff70 tx=0x7fa8e8008040 comp rx=0 tx=0).stop 2023-10-16T11:19:22.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.316+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 msgr2=0x7fa8ec0acfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:22.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.316+0000 7fa8f415b700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0acfc0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c550 tx=0x7fa8e000c910 comp rx=0 tx=0).stop 2023-10-16T11:19:22.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.318+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 shutdown_connections 2023-10-16T11:19:22.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.318+0000 7fa8f415b700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa8ec15a780 0x7fa8ec14d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.318+0000 7fa8f415b700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa8d006b020 0x7fa8d006d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.318+0000 7fa8f415b700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8ec06d960 0x7fa8ec0acfc0 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.318+0000 7fa8f415b700 1 --2- 172.21.15.158:0/2563190776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8ec072b60 0x7fa8ec1554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.318+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 >> 172.21.15.158:0/2563190776 conn(0x7fa8ec0ccd30 msgr2=0x7fa8ec1577c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:22.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.323+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 shutdown_connections 2023-10-16T11:19:22.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.323+0000 7fa8f415b700 1 -- 172.21.15.158:0/2563190776 wait complete. 2023-10-16T11:19:22.327 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T11:19:22.345 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T11:19:22.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.471+0000 7fae95967700 1 Processor -- start 2023-10-16T11:19:22.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.471+0000 7fae95967700 1 -- start start 2023-10-16T11:19:22.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.471+0000 7fae95967700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fae90160d30 0x7fae900b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:22.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.471+0000 7fae95967700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae900b7590 0x7fae900d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:22.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.471+0000 7fae95967700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae900b7a20 con 0x7fae900d0530 2023-10-16T11:19:22.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.471+0000 7fae95967700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae900b8910 con 0x7fae900b7590 2023-10-16T11:19:22.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.471+0000 7fae95967700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae900d0ef0 con 0x7fae90160d30 2023-10-16T11:19:22.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae900b7590 0x7fae900d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:22.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fae900b7590 0x7fae900d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37948/0 (socket says 172.21.15.158:37948) 2023-10-16T11:19:22.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 -- 172.21.15.158:0/927165803 learned_addr learned my addr 172.21.15.158:0/927165803 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:22.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8effd700 1 --2- 172.21.15.158:0/927165803 >> v2:172.21.15.172:3300/0 conn(0x7fae90160d30 0x7fae900b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:22.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 -- 172.21.15.158:0/927165803 >> v2:172.21.15.172:3300/0 conn(0x7fae90160d30 msgr2=0x7fae900b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:22.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 --2- 172.21.15.158:0/927165803 >> v2:172.21.15.172:3300/0 conn(0x7fae90160d30 0x7fae900b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 -- 172.21.15.158:0/927165803 >> v1:172.21.15.172:6789/0 conn(0x7fae900d0530 legacy=0x7fae900d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:22.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 -- 172.21.15.158:0/927165803 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae900b7c90 con 0x7fae900b7590 2023-10-16T11:19:22.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8effd700 1 --2- 172.21.15.158:0/927165803 >> v2:172.21.15.172:3300/0 conn(0x7fae90160d30 0x7fae900b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:22.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8f7fe700 1 --2- 172.21.15.158:0/927165803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900b7590 0x7fae900d09b0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fae88009e50 tx=0x7fae8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1509f9862fca330 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:22.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8dffb700 1 -- 172.21.15.158:0/927165803 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8800e070 con 0x7fae900b7590 2023-10-16T11:19:22.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8dffb700 1 -- 172.21.15.158:0/927165803 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae88003a20 con 0x7fae900b7590 2023-10-16T11:19:22.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.472+0000 7fae8dffb700 1 -- 172.21.15.158:0/927165803 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88003dd0 con 0x7fae900b7590 2023-10-16T11:19:22.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 -- 172.21.15.158:0/927165803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900b7590 msgr2=0x7fae900d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:22.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 --2- 172.21.15.158:0/927165803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900b7590 0x7fae900d09b0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fae88009e50 tx=0x7fae8800b330 comp rx=0 tx=0).stop 2023-10-16T11:19:22.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 -- 172.21.15.158:0/927165803 shutdown_connections 2023-10-16T11:19:22.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 --2- 172.21.15.158:0/927165803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900b7590 0x7fae900d09b0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 --2- 172.21.15.158:0/927165803 >> v2:172.21.15.172:3300/0 conn(0x7fae90160d30 0x7fae900b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 -- 172.21.15.158:0/927165803 >> 172.21.15.158:0/927165803 conn(0x7fae900669f0 msgr2=0x7fae90066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:22.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 -- 172.21.15.158:0/927165803 shutdown_connections 2023-10-16T11:19:22.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.473+0000 7fae95967700 1 -- 172.21.15.158:0/927165803 wait complete. 2023-10-16T11:19:22.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.474+0000 7fae95967700 1 Processor -- start 2023-10-16T11:19:22.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.474+0000 7fae95967700 1 -- start start 2023-10-16T11:19:22.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae95967700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae900b7590 0x7fae900cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:22.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae95967700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900d0530 0x7fae901536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:22.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae95967700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae90160d30 0x7fae900cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:22.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae95967700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c7770 con 0x7fae900d0530 2023-10-16T11:19:22.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae95967700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900cc780 con 0x7fae900b7590 2023-10-16T11:19:22.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae95967700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae90153be0 con 0x7fae90160d30 2023-10-16T11:19:22.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900d0530 0x7fae901536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:22.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900d0530 0x7fae901536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37952/0 (socket says 172.21.15.158:37952) 2023-10-16T11:19:22.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 -- 172.21.15.158:0/3705605868 learned_addr learned my addr 172.21.15.158:0/3705605868 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:22.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8effd700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae900b7590 0x7fae900cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:22.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8f7fe700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae90160d30 0x7fae900cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:22.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 -- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae90160d30 msgr2=0x7fae900cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:22.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae90160d30 0x7fae900cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 -- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae900b7590 msgr2=0x7fae900cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:22.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae900b7590 0x7fae900cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:22.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.475+0000 7fae8e7fc700 1 -- 172.21.15.158:0/3705605868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88009b30 con 0x7fae900d0530 2023-10-16T11:19:22.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.476+0000 7fae8e7fc700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900d0530 0x7fae901536a0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c500 tx=0x7fae8400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:22.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.476+0000 7fae77fff700 1 -- 172.21.15.158:0/3705605868 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84011790 con 0x7fae900d0530 2023-10-16T11:19:22.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.476+0000 7fae77fff700 1 -- 172.21.15.158:0/3705605868 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae840118f0 con 0x7fae900d0530 2023-10-16T11:19:22.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.476+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae900c7b90 con 0x7fae900d0530 2023-10-16T11:19:22.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.476+0000 7fae77fff700 1 -- 172.21.15.158:0/3705605868 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84011ca0 con 0x7fae900d0530 2023-10-16T11:19:22.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.476+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae900c70b0 con 0x7fae900d0530 2023-10-16T11:19:22.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.477+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae900c78b0 con 0x7fae900d0530 2023-10-16T11:19:22.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.477+0000 7fae77fff700 1 -- 172.21.15.158:0/3705605868 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fae8401c460 con 0x7fae900d0530 2023-10-16T11:19:22.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.478+0000 7fae77fff700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fae7c06b020 0x7fae7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:22.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.478+0000 7fae77fff700 1 -- 172.21.15.158:0/3705605868 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6409+0+0 (secure 0 0 0) 0x7fae84089ef0 con 0x7fae900d0530 2023-10-16T11:19:22.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.478+0000 7fae8effd700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fae7c06b020 0x7fae7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:22.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.478+0000 7fae8effd700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fae7c06b020 0x7fae7c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fae900c6980 tx=0x7fae80008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:22.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.482+0000 7fae77fff700 1 -- 172.21.15.158:0/3705605868 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae84054d40 con 0x7fae900d0530 2023-10-16T11:19:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:22.723+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fae900c78b0 con 0x7fae900d0530 2023-10-16T11:19:23.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.331+0000 7fae77fff700 1 -- 172.21.15.158:0/3705605868 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v108) v1 ==== 174+0+0 (secure 0 0 0) 0x7fae900c78b0 con 0x7fae900d0530 2023-10-16T11:19:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fae7c06b020 msgr2=0x7fae7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fae7c06b020 0x7fae7c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fae900c6980 tx=0x7fae80008040 comp rx=0 tx=0).stop 2023-10-16T11:19:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900d0530 msgr2=0x7fae901536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900d0530 0x7fae901536a0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c500 tx=0x7fae8400c810 comp rx=0 tx=0).stop 2023-10-16T11:19:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 shutdown_connections 2023-10-16T11:19:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae90160d30 0x7fae900cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae900b7590 0x7fae900cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fae7c06b020 0x7fae7c06d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 --2- 172.21.15.158:0/3705605868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae900d0530 0x7fae901536a0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 >> 172.21.15.158:0/3705605868 conn(0x7fae900669f0 msgr2=0x7fae9015f330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 shutdown_connections 2023-10-16T11:19:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.338+0000 7fae95967700 1 -- 172.21.15.158:0/3705605868 wait complete. 2023-10-16T11:19:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 27 and data pool 28 2023-10-16T11:19:23.358 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.478+0000 7fcfa1138700 1 Processor -- start 2023-10-16T11:19:23.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.478+0000 7fcfa1138700 1 -- start start 2023-10-16T11:19:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcfa1138700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcf9c15dfb0 0x7fcf9c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcfa1138700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c06e770 con 0x7fcf9c06f340 2023-10-16T11:19:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcfa1138700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0cbe00 con 0x7fcf9c0cc5d0 2023-10-16T11:19:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcfa1138700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9c0cf270 con 0x7fcf9c15dfb0 2023-10-16T11:19:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcf9b59e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcf9c06f340 0x7fcf9c071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38138/0 (socket says 172.21.15.158:38138) 2023-10-16T11:19:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcf9b59e700 1 -- 172.21.15.158:0/4027191100 learned_addr learned my addr 172.21.15.158:0/4027191100 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcf9a59c700 1 --2- 172.21.15.158:0/4027191100 >> v2:172.21.15.172:3300/0 conn(0x7fcf9c15dfb0 0x7fcf9c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.479+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976421263 0 0) 0x7fcf94001490 con 0x7fcf9c06f340 2023-10-16T11:19:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf840057c0 con 0x7fcf9c06f340 2023-10-16T11:19:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745702040 0 0) 0x7fcf9c0cbe00 con 0x7fcf9c0cc5d0 2023-10-16T11:19:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf94001490 con 0x7fcf9c0cc5d0 2023-10-16T11:19:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937779942 0 0) 0x7fcf840057c0 con 0x7fcf9c06f340 2023-10-16T11:19:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 >> v2:172.21.15.172:3300/0 conn(0x7fcf9c15dfb0 msgr2=0x7fcf9c06e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 --2- 172.21.15.158:0/4027191100 >> v2:172.21.15.172:3300/0 conn(0x7fcf9c15dfb0 0x7fcf9c06e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 >> v1:172.21.15.172:6789/0 conn(0x7fcf9c0cc5d0 legacy=0x7fcf9c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c15e4e0 con 0x7fcf9c06f340 2023-10-16T11:19:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcf94001b80 con 0x7fcf9c06f340 2023-10-16T11:19:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 >> v1:172.21.15.158:6789/0 conn(0x7fcf9c06f340 legacy=0x7fcf9c071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84005da0 0x7fcf840061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf84006a60 0x7fcf84006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf84007420 0x7fcf8400dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0cf270 con 0x7fcf84007420 2023-10-16T11:19:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8400e2a0 con 0x7fcf84006a60 2023-10-16T11:19:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8400e460 con 0x7fcf84005da0 2023-10-16T11:19:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf84007420 0x7fcf8400dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9a59c700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf84006a60 0x7fcf84006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9ad9d700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84005da0 0x7fcf840061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 -- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84005da0 msgr2=0x7fcf840061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84005da0 0x7fcf840061a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 -- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf84006a60 msgr2=0x7fcf84006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:23.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf84006a60 0x7fcf84006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 -- 172.21.15.158:0/4027191100 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf840057c0 con 0x7fcf84007420 2023-10-16T11:19:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 -- 172.21.15.158:0/4027191100 reap_dead start 2023-10-16T11:19:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.480+0000 7fcf9b59e700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf84007420 0x7fcf8400dc20 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400a790 tx=0x7fcf84007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3c099dd84761bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94012600 con 0x7fcf84007420 2023-10-16T11:19:23.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf940127a0 con 0x7fcf84007420 2023-10-16T11:19:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcf99d9b700 1 -- 172.21.15.158:0/4027191100 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94012b50 con 0x7fcf84007420 2023-10-16T11:19:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcfa1138700 1 -- 172.21.15.158:0/4027191100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf84007420 msgr2=0x7fcf8400dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcfa1138700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf84007420 0x7fcf8400dc20 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400a790 tx=0x7fcf84007a90 comp rx=0 tx=0).stop 2023-10-16T11:19:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcfa1138700 1 -- 172.21.15.158:0/4027191100 shutdown_connections 2023-10-16T11:19:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcfa1138700 1 --2- 172.21.15.158:0/4027191100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf84007420 0x7fcf8400dc20 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcfa1138700 1 -- 172.21.15.158:0/4027191100 >> 172.21.15.158:0/4027191100 conn(0x7fcf9c0d02b0 msgr2=0x7fcf9c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:23.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcfa1138700 1 -- 172.21.15.158:0/4027191100 shutdown_connections 2023-10-16T11:19:23.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.481+0000 7fcfa1138700 1 -- 172.21.15.158:0/4027191100 wait complete. 2023-10-16T11:19:23.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 Processor -- start 2023-10-16T11:19:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 -- start start 2023-10-16T11:19:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84007420 0x7fcf9c1f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf9c0ac890 0x7fcf9c1f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf9c0cf3f0 0x7fcf9c0cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0cfde0 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0b1e70 con 0x7fcf9c0ac890 2023-10-16T11:19:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.483+0000 7fcfa1138700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c1f4040 con 0x7fcf84007420 2023-10-16T11:19:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf9c0cf3f0 0x7fcf9c0cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf9c0cf3f0 0x7fcf9c0cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37974/0 (socket says 172.21.15.158:37974) 2023-10-16T11:19:23.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 -- 172.21.15.158:0/3188300399 learned_addr learned my addr 172.21.15.158:0/3188300399 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 -- 172.21.15.158:0/3188300399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84007420 msgr2=0x7fcf9c1f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84007420 0x7fcf9c1f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 -- 172.21.15.158:0/3188300399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf9c0ac890 msgr2=0x7fcf9c1f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9a59c700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf9c0ac890 0x7fcf9c1f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf9c0ac890 0x7fcf9c1f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 -- 172.21.15.158:0/3188300399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0ace50 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf9b59e700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf9c0cf3f0 0x7fcf9c0cf8a0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7fcf94002230 tx=0x7fcf94001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf837fe700 1 -- 172.21.15.158:0/3188300399 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9401c610 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf837fe700 1 -- 172.21.15.158:0/3188300399 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9401c790 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcf837fe700 1 -- 172.21.15.158:0/3188300399 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9401cb40 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c0ad130 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.484+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c0ad6f0 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.485+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9c0734b0 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.486+0000 7fcf837fe700 1 -- 172.21.15.158:0/3188300399 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf94019050 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.486+0000 7fcf837fe700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcf8806b020 0x7fcf8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.486+0000 7fcf837fe700 1 -- 172.21.15.158:0/3188300399 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fcf9408b140 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.486+0000 7fcf9ad9d700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcf8806b020 0x7fcf8806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.486+0000 7fcf9ad9d700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcf8806b020 0x7fcf8806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0aebd0 tx=0x7fcf90009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:23.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.490+0000 7fcf837fe700 1 -- 172.21.15.158:0/3188300399 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf9c0734b0 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.686+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf9c0c7380 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.687+0000 7fcf837fe700 1 -- 172.21.15.158:0/3188300399 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) v1 ==== 74+0+17544 (secure 0 0 0) 0x7fcf9c0c7380 con 0x7fcf9c0cf3f0 2023-10-16T11:19:23.689 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:23.689 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":216,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:22.724661+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:23.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcf8806b020 msgr2=0x7fcf8806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcf8806b020 0x7fcf8806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0aebd0 tx=0x7fcf90009040 comp rx=0 tx=0).stop 2023-10-16T11:19:23.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf9c0cf3f0 msgr2=0x7fcf9c0cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf9c0cf3f0 0x7fcf9c0cf8a0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7fcf94002230 tx=0x7fcf94001e90 comp rx=0 tx=0).stop 2023-10-16T11:19:23.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 shutdown_connections 2023-10-16T11:19:23.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcf8806b020 0x7fcf8806d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf9c0cf3f0 0x7fcf9c0cf8a0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf84007420 0x7fcf9c1f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 --2- 172.21.15.158:0/3188300399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf9c0ac890 0x7fcf9c1f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 >> 172.21.15.158:0/3188300399 conn(0x7fcf9c0d02b0 msgr2=0x7fcf9c06b160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:23.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 shutdown_connections 2023-10-16T11:19:23.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.689+0000 7fcfa1138700 1 -- 172.21.15.158:0/3188300399 wait complete. 2023-10-16T11:19:23.703 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:23.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.821+0000 7f9f7e4d2700 1 Processor -- start 2023-10-16T11:19:23.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.821+0000 7f9f7e4d2700 1 -- start start 2023-10-16T11:19:23.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.822+0000 7f9f7e4d2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f7815a6c0 0x7f9f780723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.822+0000 7f9f7e4d2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f7806e090 0x7f9f780ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.822+0000 7f9f7e4d2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f78072910 con 0x7f9f780af590 2023-10-16T11:19:23.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.822+0000 7f9f7e4d2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780ca7e0 con 0x7f9f7806e090 2023-10-16T11:19:23.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.822+0000 7f9f7e4d2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780b1e40 con 0x7f9f7815a6c0 2023-10-16T11:19:23.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.822+0000 7f9f7ca6f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f7806e090 0x7f9f780ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f7ca6f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f7806e090 0x7f9f780ad430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37982/0 (socket says 172.21.15.158:37982) 2023-10-16T11:19:23.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f7ca6f700 1 -- 172.21.15.158:0/1839077235 learned_addr learned my addr 172.21.15.158:0/1839077235 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:23.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f7ca6f700 1 -- 172.21.15.158:0/1839077235 >> v2:172.21.15.172:3300/0 conn(0x7f9f7815a6c0 msgr2=0x7f9f780723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:23.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f7ca6f700 1 --2- 172.21.15.158:0/1839077235 >> v2:172.21.15.172:3300/0 conn(0x7f9f7815a6c0 0x7f9f780723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f7ca6f700 1 -- 172.21.15.158:0/1839077235 >> v1:172.21.15.172:6789/0 conn(0x7f9f780af590 legacy=0x7f9f7806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:23.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f7ca6f700 1 -- 172.21.15.158:0/1839077235 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f780b2120 con 0x7f9f7806e090 2023-10-16T11:19:23.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f7ca6f700 1 --2- 172.21.15.158:0/1839077235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f780ad430 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f9f780737c0 tx=0x7f9f7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e294e915b2691be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:23.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f76ffd700 1 -- 172.21.15.158:0/1839077235 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000e070 con 0x7f9f7806e090 2023-10-16T11:19:23.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f76ffd700 1 -- 172.21.15.158:0/1839077235 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70003a20 con 0x7f9f7806e090 2023-10-16T11:19:23.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.823+0000 7f9f76ffd700 1 -- 172.21.15.158:0/1839077235 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70003dd0 con 0x7f9f7806e090 2023-10-16T11:19:23.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/1839077235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 msgr2=0x7f9f780ad430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/1839077235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f780ad430 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f9f780737c0 tx=0x7f9f7000b330 comp rx=0 tx=0).stop 2023-10-16T11:19:23.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/1839077235 shutdown_connections 2023-10-16T11:19:23.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/1839077235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f780ad430 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/1839077235 >> v2:172.21.15.172:3300/0 conn(0x7f9f7815a6c0 0x7f9f780723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/1839077235 >> 172.21.15.158:0/1839077235 conn(0x7f9f780ccda0 msgr2=0x7f9f780cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:23.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/1839077235 shutdown_connections 2023-10-16T11:19:23.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.824+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/1839077235 wait complete. 2023-10-16T11:19:23.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.825+0000 7f9f7e4d2700 1 Processor -- start 2023-10-16T11:19:23.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.825+0000 7f9f7e4d2700 1 -- start start 2023-10-16T11:19:23.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f78150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f780af590 0x7f9f780c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f7815a6c0 0x7f9f780c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78153bd0 con 0x7f9f7806e090 2023-10-16T11:19:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78150e70 con 0x7f9f780af590 2023-10-16T11:19:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781538f0 con 0x7f9f7815a6c0 2023-10-16T11:19:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f78150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f78150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37984/0 (socket says 172.21.15.158:37984) 2023-10-16T11:19:23.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 -- 172.21.15.158:0/868632631 learned_addr learned my addr 172.21.15.158:0/868632631 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f777fe700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f780af590 0x7f9f780c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7ca6f700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f7815a6c0 0x7f9f780c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 -- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f7815a6c0 msgr2=0x7f9f780c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f7815a6c0 0x7f9f780c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 -- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f780af590 msgr2=0x7f9f780c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f780af590 0x7f9f780c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 -- 172.21.15.158:0/868632631 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f70009b30 con 0x7f9f7806e090 2023-10-16T11:19:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7ca6f700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f7815a6c0 0x7f9f780c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f77fff700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f78150930 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800b480 tx=0x7f9f6800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:23.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f74ff9700 1 -- 172.21.15.158:0/868632631 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68005a00 con 0x7f9f7806e090 2023-10-16T11:19:23.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f74ff9700 1 -- 172.21.15.158:0/868632631 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f68005b60 con 0x7f9f7806e090 2023-10-16T11:19:23.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f780b2f80 con 0x7f9f7806e090 2023-10-16T11:19:23.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f74ff9700 1 -- 172.21.15.158:0/868632631 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68011630 con 0x7f9f7806e090 2023-10-16T11:19:23.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.826+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f780b3540 con 0x7f9f7806e090 2023-10-16T11:19:23.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.827+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f780b23e0 con 0x7f9f7806e090 2023-10-16T11:19:23.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.828+0000 7f9f74ff9700 1 -- 172.21.15.158:0/868632631 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9f68011790 con 0x7f9f7806e090 2023-10-16T11:19:23.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.828+0000 7f9f74ff9700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9f5806b020 0x7f9f5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:23.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.828+0000 7f9f74ff9700 1 -- 172.21.15.158:0/868632631 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f9f68089600 con 0x7f9f7806e090 2023-10-16T11:19:23.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.828+0000 7f9f777fe700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9f5806b020 0x7f9f5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:23.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.829+0000 7f9f777fe700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9f5806b020 0x7f9f5806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f78168790 tx=0x7f9f60009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:23.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:23.833+0000 7f9f74ff9700 1 -- 172.21.15.158:0/868632631 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f780b23e0 con 0x7f9f7806e090 2023-10-16T11:19:24.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.030+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f780b23e0 con 0x7f9f7806e090 2023-10-16T11:19:24.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.031+0000 7f9f74ff9700 1 -- 172.21.15.158:0/868632631 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) v1 ==== 74+0+17544 (secure 0 0 0) 0x7f9f780b23e0 con 0x7f9f7806e090 2023-10-16T11:19:24.033 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:24.034 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":216,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:22.724661+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:24.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.033+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9f5806b020 msgr2=0x7f9f5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.033+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9f5806b020 0x7f9f5806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f78168790 tx=0x7f9f60009040 comp rx=0 tx=0).stop 2023-10-16T11:19:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.033+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 msgr2=0x7f9f78150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.033+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f78150930 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800b480 tx=0x7f9f6800b790 comp rx=0 tx=0).stop 2023-10-16T11:19:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.033+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 shutdown_connections 2023-10-16T11:19:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.034+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f7815a6c0 0x7f9f780c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.034+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9f5806b020 0x7f9f5806d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.034+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f7806e090 0x7f9f78150930 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.034+0000 7f9f7e4d2700 1 --2- 172.21.15.158:0/868632631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f780af590 0x7f9f780c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.034+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 >> 172.21.15.158:0/868632631 conn(0x7f9f780ccda0 msgr2=0x7f9f780b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:24.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.034+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 shutdown_connections 2023-10-16T11:19:24.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.034+0000 7f9f7e4d2700 1 -- 172.21.15.158:0/868632631 wait complete. 2023-10-16T11:19:24.048 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T11:19:24.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.168+0000 7f4248f40700 1 Processor -- start 2023-10-16T11:19:24.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.169+0000 7f4248f40700 1 -- start start 2023-10-16T11:19:24.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.169+0000 7f4248f40700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f424415da40 0x7f42440caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:24.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.169+0000 7f4248f40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42440cc590 0x7f424406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:24.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.169+0000 7f4248f40700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f424406ae10 0x7f42440ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:24.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.169+0000 7f4248f40700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424406b2c0 con 0x7f424415da40 2023-10-16T11:19:24.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.169+0000 7f4248f40700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42440cbd30 con 0x7f424406ae10 2023-10-16T11:19:24.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.169+0000 7f4248f40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42440cc1c0 con 0x7f42440cc590 2023-10-16T11:19:24.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f424415da40 0x7f42440caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:24.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f424415da40 0x7f42440caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37992/0 (socket says 172.21.15.158:37992) 2023-10-16T11:19:24.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 -- 172.21.15.158:0/538288598 learned_addr learned my addr 172.21.15.158:0/538288598 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:24.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f4242d9d700 1 --2- 172.21.15.158:0/538288598 >> v2:172.21.15.172:3300/0 conn(0x7f424406ae10 0x7f42440ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:24.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f4241d9b700 1 --2- 172.21.15.158:0/538288598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42440cc590 0x7f424406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:24.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 -- 172.21.15.158:0/538288598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42440cc590 msgr2=0x7f424406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:24.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 --2- 172.21.15.158:0/538288598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42440cc590 0x7f424406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 -- 172.21.15.158:0/538288598 >> v2:172.21.15.172:3300/0 conn(0x7f424406ae10 msgr2=0x7f42440ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:24.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 --2- 172.21.15.158:0/538288598 >> v2:172.21.15.172:3300/0 conn(0x7f424406ae10 0x7f42440ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 -- 172.21.15.158:0/538288598 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424415de20 con 0x7f424415da40 2023-10-16T11:19:24.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424259c700 1 --2- 172.21.15.158:0/538288598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f424415da40 0x7f42440caa20 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f42440729e0 tx=0x7f4238009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c4d586e22a2915 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:24.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424159a700 1 -- 172.21.15.158:0/538288598 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238011830 con 0x7f424415da40 2023-10-16T11:19:24.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424159a700 1 -- 172.21.15.158:0/538288598 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238011990 con 0x7f424415da40 2023-10-16T11:19:24.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.170+0000 7f424159a700 1 -- 172.21.15.158:0/538288598 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238011d20 con 0x7f424415da40 2023-10-16T11:19:24.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 -- 172.21.15.158:0/538288598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f424415da40 msgr2=0x7f42440caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:24.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 --2- 172.21.15.158:0/538288598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f424415da40 0x7f42440caa20 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f42440729e0 tx=0x7f4238009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 -- 172.21.15.158:0/538288598 shutdown_connections 2023-10-16T11:19:24.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 --2- 172.21.15.158:0/538288598 >> v2:172.21.15.172:3300/0 conn(0x7f424406ae10 0x7f42440ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 --2- 172.21.15.158:0/538288598 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f424415da40 0x7f42440caa20 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 --2- 172.21.15.158:0/538288598 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42440cc590 0x7f424406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 -- 172.21.15.158:0/538288598 >> 172.21.15.158:0/538288598 conn(0x7f4244099a10 msgr2=0x7f42440d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:24.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 -- 172.21.15.158:0/538288598 shutdown_connections 2023-10-16T11:19:24.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.171+0000 7f4248f40700 1 -- 172.21.15.158:0/538288598 wait complete. 2023-10-16T11:19:24.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.173+0000 7f4248f40700 1 Processor -- start 2023-10-16T11:19:24.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.173+0000 7f4248f40700 1 -- start start 2023-10-16T11:19:24.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.173+0000 7f4248f40700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f424406ae10 0x7f42441594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:24.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4248f40700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42440cc590 0x7f42440c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:24.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4248f40700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f424415da40 0x7f424415f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:24.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4248f40700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42440cadc0 con 0x7f42440cc590 2023-10-16T11:19:24.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4248f40700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424415f460 con 0x7f424415da40 2023-10-16T11:19:24.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4248f40700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42440ca520 con 0x7f424406ae10 2023-10-16T11:19:24.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42440cc590 0x7f42440c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:24.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42440cc590 0x7f42440c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38000/0 (socket says 172.21.15.158:38000) 2023-10-16T11:19:24.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 -- 172.21.15.158:0/2082306452 learned_addr learned my addr 172.21.15.158:0/2082306452 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:24.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4242d9d700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f424415da40 0x7f424415f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:24.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f424259c700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f424406ae10 0x7f42441594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:24.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 -- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f424406ae10 msgr2=0x7f42441594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:24.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f424406ae10 0x7f42441594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 -- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f424415da40 msgr2=0x7f424415f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:24.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f424415da40 0x7f424415f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:24.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 -- 172.21.15.158:0/2082306452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238009550 con 0x7f42440cc590 2023-10-16T11:19:24.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4241d9b700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42440cc590 0x7f42440c64e0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f423400e980 tx=0x7f423400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:24.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4232ffd700 1 -- 172.21.15.158:0/2082306452 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4234009710 con 0x7f42440cc590 2023-10-16T11:19:24.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4232ffd700 1 -- 172.21.15.158:0/2082306452 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423400c690 con 0x7f42440cc590 2023-10-16T11:19:24.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42440ca800 con 0x7f42440cc590 2023-10-16T11:19:24.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4232ffd700 1 -- 172.21.15.158:0/2082306452 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423400ca20 con 0x7f42440cc590 2023-10-16T11:19:24.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.174+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42441599f0 con 0x7f42440cc590 2023-10-16T11:19:24.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.175+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424415f230 con 0x7f42440cc590 2023-10-16T11:19:24.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.176+0000 7f4232ffd700 1 -- 172.21.15.158:0/2082306452 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f423400cb80 con 0x7f42440cc590 2023-10-16T11:19:24.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.176+0000 7f4232ffd700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f422406b020 0x7f422406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:24.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.176+0000 7f4232ffd700 1 -- 172.21.15.158:0/2082306452 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f4234089aa0 con 0x7f42440cc590 2023-10-16T11:19:24.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.176+0000 7f424259c700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f422406b020 0x7f422406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:24.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.177+0000 7f424259c700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f422406b020 0x7f422406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4238014040 tx=0x7f4238008030 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:24.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.180+0000 7f4232ffd700 1 -- 172.21.15.158:0/2082306452 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f424415f230 con 0x7f42440cc590 2023-10-16T11:19:24.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:24.426+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f424415f610 con 0x7f42440cc590 2023-10-16T11:19:25.308 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T11:19:25.307+0000 7f8dbaac2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:19:25.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.345+0000 7f4232ffd700 1 -- 172.21.15.158:0/2082306452 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v111) v1 ==== 118+0+0 (secure 0 0 0) 0x7f424415f610 con 0x7f42440cc590 2023-10-16T11:19:25.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f422406b020 msgr2=0x7f422406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:25.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f422406b020 0x7f422406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4238014040 tx=0x7f4238008030 comp rx=0 tx=0).stop 2023-10-16T11:19:25.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42440cc590 msgr2=0x7f42440c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:25.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42440cc590 0x7f42440c64e0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f423400e980 tx=0x7f423400ec90 comp rx=0 tx=0).stop 2023-10-16T11:19:25.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 shutdown_connections 2023-10-16T11:19:25.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f424415da40 0x7f424415f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f424406ae10 0x7f42441594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f422406b020 0x7f422406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 --2- 172.21.15.158:0/2082306452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42440cc590 0x7f42440c64e0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 >> 172.21.15.158:0/2082306452 conn(0x7f4244099a10 msgr2=0x7f424415b960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:25.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 shutdown_connections 2023-10-16T11:19:25.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.348+0000 7f4248f40700 1 -- 172.21.15.158:0/2082306452 wait complete. 2023-10-16T11:19:25.364 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:19:25.365 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T11:19:25.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.491+0000 7f410769b700 1 Processor -- start 2023-10-16T11:19:25.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.491+0000 7f410769b700 1 -- start start 2023-10-16T11:19:25.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.492+0000 7f410769b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f410015dad0 0x7f41000cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:25.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.492+0000 7f410769b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f41000ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:25.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.492+0000 7f410769b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41000d0ae0 con 0x7f41000cea20 2023-10-16T11:19:25.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.492+0000 7f410769b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41000d12d0 con 0x7f410015dad0 2023-10-16T11:19:25.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.492+0000 7f410769b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41000cbdf0 con 0x7f410015deb0 2023-10-16T11:19:25.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.492+0000 7f4105437700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f410015dad0 0x7f41000cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:25.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4105437700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f410015dad0 0x7f41000cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38006/0 (socket says 172.21.15.158:38006) 2023-10-16T11:19:25.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4105437700 1 -- 172.21.15.158:0/1191124224 learned_addr learned my addr 172.21.15.158:0/1191124224 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:25.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4104c36700 1 --2- 172.21.15.158:0/1191124224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f41000ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:25.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4105437700 1 -- 172.21.15.158:0/1191124224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 msgr2=0x7f41000ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:25.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4105437700 1 --2- 172.21.15.158:0/1191124224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f41000ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4105437700 1 -- 172.21.15.158:0/1191124224 >> v1:172.21.15.172:6789/0 conn(0x7f41000cea20 legacy=0x7f41000c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:25.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4105437700 1 -- 172.21.15.158:0/1191124224 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41000cc280 con 0x7f410015dad0 2023-10-16T11:19:25.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4104c36700 1 --2- 172.21.15.158:0/1191124224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f41000ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:25.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f4105437700 1 --2- 172.21.15.158:0/1191124224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000cdfa0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f40f00098f0 tx=0x7f40f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea17c04fd26d6b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:25.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f40f7fff700 1 -- 172.21.15.158:0/1191124224 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0004230 con 0x7f410015dad0 2023-10-16T11:19:25.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f40f7fff700 1 -- 172.21.15.158:0/1191124224 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0004390 con 0x7f410015dad0 2023-10-16T11:19:25.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.493+0000 7f40f7fff700 1 -- 172.21.15.158:0/1191124224 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00036a0 con 0x7f410015dad0 2023-10-16T11:19:25.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 -- 172.21.15.158:0/1191124224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 msgr2=0x7f41000cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:25.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 --2- 172.21.15.158:0/1191124224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000cdfa0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f40f00098f0 tx=0x7f40f0009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:25.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 -- 172.21.15.158:0/1191124224 shutdown_connections 2023-10-16T11:19:25.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 --2- 172.21.15.158:0/1191124224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000cdfa0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 --2- 172.21.15.158:0/1191124224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f41000ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 -- 172.21.15.158:0/1191124224 >> 172.21.15.158:0/1191124224 conn(0x7f4100099a10 msgr2=0x7f41000d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:25.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 -- 172.21.15.158:0/1191124224 shutdown_connections 2023-10-16T11:19:25.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.494+0000 7f410769b700 1 -- 172.21.15.158:0/1191124224 wait complete. 2023-10-16T11:19:25.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.495+0000 7f410769b700 1 Processor -- start 2023-10-16T11:19:25.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.495+0000 7f410769b700 1 -- start start 2023-10-16T11:19:25.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f410769b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41000cea20 0x7f41000ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:25.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f410769b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:25.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f410769b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f410006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:25.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f410769b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41000c91a0 con 0x7f410015dad0 2023-10-16T11:19:25.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f410769b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41000c7c80 con 0x7f41000cea20 2023-10-16T11:19:25.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f410769b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41000c79e0 con 0x7f410015deb0 2023-10-16T11:19:25.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:25.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38014/0 (socket says 172.21.15.158:38014) 2023-10-16T11:19:25.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 -- 172.21.15.158:0/2511824346 learned_addr learned my addr 172.21.15.158:0/2511824346 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:25.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4105437700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41000cea20 0x7f41000ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:25.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4105c38700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f410006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:25.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 -- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 msgr2=0x7f410006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:25.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f410006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 -- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41000cea20 msgr2=0x7f41000ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:25.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41000cea20 0x7f41000ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:25.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 -- 172.21.15.158:0/2511824346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f0009550 con 0x7f410015dad0 2023-10-16T11:19:25.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f4104c36700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000c74a0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c7f0 tx=0x7f40fc00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:25.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f40f5ffb700 1 -- 172.21.15.158:0/2511824346 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc011800 con 0x7f410015dad0 2023-10-16T11:19:25.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f40f5ffb700 1 -- 172.21.15.158:0/2511824346 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc011960 con 0x7f410015dad0 2023-10-16T11:19:25.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f40f5ffb700 1 -- 172.21.15.158:0/2511824346 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc011cf0 con 0x7f410015dad0 2023-10-16T11:19:25.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.496+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41000c93d0 con 0x7f410015dad0 2023-10-16T11:19:25.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.497+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41000c9990 con 0x7f410015dad0 2023-10-16T11:19:25.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.498+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41000681b0 con 0x7f410015dad0 2023-10-16T11:19:25.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.498+0000 7f40f5ffb700 1 -- 172.21.15.158:0/2511824346 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f40fc018360 con 0x7f410015dad0 2023-10-16T11:19:25.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.498+0000 7f40f5ffb700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f40e406b020 0x7f40e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:25.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.498+0000 7f40f5ffb700 1 -- 172.21.15.158:0/2511824346 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f40fc089a30 con 0x7f410015dad0 2023-10-16T11:19:25.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.499+0000 7f4105437700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f40e406b020 0x7f40e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:25.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.499+0000 7f4105437700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f40e406b020 0x7f40e406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009130 tx=0x7f40f0008030 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:25.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.503+0000 7f40f5ffb700 1 -- 172.21.15.158:0/2511824346 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40fc0545a0 con 0x7f410015dad0 2023-10-16T11:19:25.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:25.740+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f41000caa00 con 0x7f410015dad0 2023-10-16T11:19:26.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.619+0000 7f40f5ffb700 1 -- 172.21.15.158:0/2511824346 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v112) v1 ==== 122+0+0 (secure 0 0 0) 0x7f40fc057bc0 con 0x7f410015dad0 2023-10-16T11:19:26.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f40e406b020 msgr2=0x7f40e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:26.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f40e406b020 0x7f40e406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009130 tx=0x7f40f0008030 comp rx=0 tx=0).stop 2023-10-16T11:19:26.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 msgr2=0x7f41000c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:26.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000c74a0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c7f0 tx=0x7f40fc00cb00 comp rx=0 tx=0).stop 2023-10-16T11:19:26.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 shutdown_connections 2023-10-16T11:19:26.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f410015deb0 0x7f410006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41000cea20 0x7f41000ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f40e406b020 0x7f40e406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 --2- 172.21.15.158:0/2511824346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f410015dad0 0x7f41000c74a0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 >> 172.21.15.158:0/2511824346 conn(0x7f4100099a10 msgr2=0x7f410006f320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:26.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 shutdown_connections 2023-10-16T11:19:26.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.622+0000 7f410769b700 1 -- 172.21.15.158:0/2511824346 wait complete. 2023-10-16T11:19:26.639 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T11:19:26.639 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:26.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.765+0000 7ff9d15f2700 1 Processor -- start 2023-10-16T11:19:26.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.765+0000 7ff9d15f2700 1 -- start start 2023-10-16T11:19:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.766+0000 7ff9d15f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 0x7ff9cc06e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.766+0000 7ff9d15f2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9cc15dcb0 con 0x7ff9cc0718d0 2023-10-16T11:19:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.766+0000 7ff9d15f2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9cc161740 con 0x7ff9cc1612a0 2023-10-16T11:19:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.766+0000 7ff9d15f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc161a20 con 0x7ff9cc06a8d0 2023-10-16T11:19:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.766+0000 7ff9cb7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff9cc0718d0 0x7ff9cc15e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38196/0 (socket says 172.21.15.158:38196) 2023-10-16T11:19:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.766+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/1924806263 learned_addr learned my addr 172.21.15.158:0/1924806263 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232888050 0 0) 0x7ff9cc15dcb0 con 0x7ff9cc0718d0 2023-10-16T11:19:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b40035b0 con 0x7ff9cc0718d0 2023-10-16T11:19:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9caffd700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 0x7ff9cc06e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4188085820 0 0) 0x7ff9b40035b0 con 0x7ff9cc0718d0 2023-10-16T11:19:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 msgr2=0x7ff9cc06e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 0x7ff9cc06e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 >> v1:172.21.15.172:6789/0 conn(0x7ff9cc1612a0 legacy=0x7ff9cc06acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc161d00 con 0x7ff9cc0718d0 2023-10-16T11:19:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff9c40019f0 con 0x7ff9cc0718d0 2023-10-16T11:19:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 >> v1:172.21.15.158:6789/0 conn(0x7ff9cc0718d0 legacy=0x7ff9cc15e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 0x7ff9b4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4003fb0 0x7ff9b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9b4004d90 0x7ff9b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc161a20 con 0x7ff9b4004d90 2023-10-16T11:19:26.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b400bba0 con 0x7ff9b4003fb0 2023-10-16T11:19:26.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b400bd60 con 0x7ff9cc06a8d0 2023-10-16T11:19:26.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9caffd700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 0x7ff9b4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9cb7fe700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9b4004d90 0x7ff9b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9ca7fc700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4003fb0 0x7ff9b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 msgr2=0x7ff9b4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:26.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9cb7fe700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 0x7ff9b4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4003fb0 msgr2=0x7ff9b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:26.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9cb7fe700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4003fb0 0x7ff9b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/1924806263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b40035b0 con 0x7ff9b4004d90 2023-10-16T11:19:26.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.767+0000 7ff9caffd700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc06a8d0 0x7ff9b4003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:26.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9cb7fe700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9b4004d90 0x7ff9b4005210 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400a830 tx=0x7ff9b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d274d41e9e56a39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:26.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400a2f0 con 0x7ff9b4004d90 2023-10-16T11:19:26.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c4012620 con 0x7ff9b4004d90 2023-10-16T11:19:26.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9c9ffb700 1 -- 172.21.15.158:0/1924806263 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40129d0 con 0x7ff9b4004d90 2023-10-16T11:19:26.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9d15f2700 1 -- 172.21.15.158:0/1924806263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9b4004d90 msgr2=0x7ff9b4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:26.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9d15f2700 1 --2- 172.21.15.158:0/1924806263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9b4004d90 0x7ff9b4005210 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400a830 tx=0x7ff9b4005880 comp rx=0 tx=0).stop 2023-10-16T11:19:26.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/1924806263 reap_dead start 2023-10-16T11:19:26.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9d15f2700 1 -- 172.21.15.158:0/1924806263 shutdown_connections 2023-10-16T11:19:26.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9d15f2700 1 -- 172.21.15.158:0/1924806263 >> 172.21.15.158:0/1924806263 conn(0x7ff9cc0d02a0 msgr2=0x7ff9cc0d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:26.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9d15f2700 1 -- 172.21.15.158:0/1924806263 shutdown_connections 2023-10-16T11:19:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.768+0000 7ff9d15f2700 1 -- 172.21.15.158:0/1924806263 wait complete. 2023-10-16T11:19:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 Processor -- start 2023-10-16T11:19:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 -- start start 2023-10-16T11:19:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4004d90 0x7ff9cc1f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc0c9d40 0x7ff9cc0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9cc0b05c0 0x7ff9cc152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc0b24f0 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc0b1b20 con 0x7ff9b4004d90 2023-10-16T11:19:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9d15f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc0b0f10 con 0x7ff9cc0c9d40 2023-10-16T11:19:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9cb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9cc0b05c0 0x7ff9cc152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.770+0000 7ff9cb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9cc0b05c0 0x7ff9cc152500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38034/0 (socket says 172.21.15.158:38034) 2023-10-16T11:19:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/393103394 learned_addr learned my addr 172.21.15.158:0/393103394 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9caffd700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4004d90 0x7ff9cc1f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9ca7fc700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc0c9d40 0x7ff9cc0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc0c9d40 msgr2=0x7ff9cc0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9cb7fe700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc0c9d40 0x7ff9cc0b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4004d90 msgr2=0x7ff9cc1f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9cb7fe700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4004d90 0x7ff9cc1f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9cb7fe700 1 -- 172.21.15.158:0/393103394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc0b11f0 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9caffd700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4004d90 0x7ff9cc1f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9ca7fc700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc0c9d40 0x7ff9cc0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9cb7fe700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9cc0b05c0 0x7ff9cc152500 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400a800 tx=0x7ff9c4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9b37fe700 1 -- 172.21.15.158:0/393103394 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4025070 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9b37fe700 1 -- 172.21.15.158:0/393103394 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c4001280 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc0b14d0 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9b37fe700 1 -- 172.21.15.158:0/393103394 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40017f0 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.771+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9cc0b0a70 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.772+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9cc0681b0 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.773+0000 7ff9b37fe700 1 -- 172.21.15.158:0/393103394 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9c401e430 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.773+0000 7ff9b37fe700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff9b806b020 0x7ff9b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.773+0000 7ff9b37fe700 1 -- 172.21.15.158:0/393103394 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff9c408b170 con 0x7ff9cc0b05c0 2023-10-16T11:19:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.773+0000 7ff9caffd700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff9b806b020 0x7ff9b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:26.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.773+0000 7ff9caffd700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff9b806b020 0x7ff9b806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009b50 tx=0x7ff9c0008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:26.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:26.777+0000 7ff9b37fe700 1 -- 172.21.15.158:0/393103394 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9c4055e80 con 0x7ff9cc0b05c0 2023-10-16T11:19:27.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.015+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9cc0b1c60 con 0x7ff9cc0b05c0 2023-10-16T11:19:27.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.015+0000 7ff9b37fe700 1 -- 172.21.15.158:0/393103394 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4938 (secure 0 0 0) 0x7ff9c40594a0 con 0x7ff9cc0b05c0 2023-10-16T11:19:27.018 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:27.018 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":11586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T11:19:27.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.017+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff9b806b020 msgr2=0x7ff9b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.017+0000 7ff9d15f2700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff9b806b020 0x7ff9b806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009b50 tx=0x7ff9c0008040 comp rx=0 tx=0).stop 2023-10-16T11:19:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9cc0b05c0 msgr2=0x7ff9cc152500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9cc0b05c0 0x7ff9cc152500 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400a800 tx=0x7ff9c4001d80 comp rx=0 tx=0).stop 2023-10-16T11:19:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 shutdown_connections 2023-10-16T11:19:27.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff9b806b020 0x7ff9b806d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9cc0b05c0 0x7ff9cc152500 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9b4004d90 0x7ff9cc1f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 --2- 172.21.15.158:0/393103394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9cc0c9d40 0x7ff9cc0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 >> 172.21.15.158:0/393103394 conn(0x7ff9cc0d02a0 msgr2=0x7ff9cc0ca8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:27.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 shutdown_connections 2023-10-16T11:19:27.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.018+0000 7ff9d15f2700 1 -- 172.21.15.158:0/393103394 wait complete. 2023-10-16T11:19:27.022 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 112 2023-10-16T11:19:27.032 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:27.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.152+0000 7fbf7dd49700 1 Processor -- start 2023-10-16T11:19:27.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.152+0000 7fbf7dd49700 1 -- start start 2023-10-16T11:19:27.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.152+0000 7fbf7dd49700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf7806e5d0 0x7fbf7806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.152+0000 7fbf7dd49700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf780729b0 0x7fbf78071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.152+0000 7fbf7dd49700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf78071e50 con 0x7fbf7806ab40 2023-10-16T11:19:27.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf7dd49700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf78161270 con 0x7fbf7806e5d0 2023-10-16T11:19:27.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf7dd49700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78161510 con 0x7fbf780729b0 2023-10-16T11:19:27.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf77fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf780729b0 0x7fbf78071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf76ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf7806e5d0 0x7fbf7806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf77fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf780729b0 0x7fbf78071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:48876/0 (socket says 172.21.15.158:48876) 2023-10-16T11:19:27.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf77fff700 1 -- 172.21.15.158:0/3654323489 learned_addr learned my addr 172.21.15.158:0/3654323489 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:27.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf76ffd700 1 -- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf780729b0 msgr2=0x7fbf78071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:27.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf76ffd700 1 --2- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf780729b0 0x7fbf78071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf76ffd700 1 -- 172.21.15.158:0/3654323489 >> v1:172.21.15.172:6789/0 conn(0x7fbf7806ab40 legacy=0x7fbf780722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf76ffd700 1 -- 172.21.15.158:0/3654323489 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf781617f0 con 0x7fbf7806e5d0 2023-10-16T11:19:27.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf77fff700 1 --2- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf780729b0 0x7fbf78071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:27.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.153+0000 7fbf76ffd700 1 --2- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806e5d0 0x7fbf7806d820 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009840 tx=0x7fbf6c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=840eb7513411c36a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf767fc700 1 -- 172.21.15.158:0/3654323489 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c011830 con 0x7fbf7806e5d0 2023-10-16T11:19:27.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf767fc700 1 -- 172.21.15.158:0/3654323489 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c011990 con 0x7fbf7806e5d0 2023-10-16T11:19:27.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf767fc700 1 -- 172.21.15.158:0/3654323489 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c011d20 con 0x7fbf7806e5d0 2023-10-16T11:19:27.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 -- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806e5d0 msgr2=0x7fbf7806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806e5d0 0x7fbf7806d820 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009840 tx=0x7fbf6c009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:27.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 -- 172.21.15.158:0/3654323489 shutdown_connections 2023-10-16T11:19:27.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf780729b0 0x7fbf78071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/3654323489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806e5d0 0x7fbf7806d820 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 -- 172.21.15.158:0/3654323489 >> 172.21.15.158:0/3654323489 conn(0x7fbf780d02b0 msgr2=0x7fbf780d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:27.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 -- 172.21.15.158:0/3654323489 shutdown_connections 2023-10-16T11:19:27.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.154+0000 7fbf7dd49700 1 -- 172.21.15.158:0/3654323489 wait complete. 2023-10-16T11:19:27.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.155+0000 7fbf7dd49700 1 Processor -- start 2023-10-16T11:19:27.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf7dd49700 1 -- start start 2023-10-16T11:19:27.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf7dd49700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806ab40 0x7fbf78154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf7dd49700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf7806e5d0 0x7fbf780afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf7dd49700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf780729b0 0x7fbf780c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf7dd49700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780b1210 con 0x7fbf780729b0 2023-10-16T11:19:27.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf7dd49700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780b0600 con 0x7fbf7806ab40 2023-10-16T11:19:27.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf7dd49700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780b08e0 con 0x7fbf7806e5d0 2023-10-16T11:19:27.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf780729b0 0x7fbf780c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf780729b0 0x7fbf780c4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38048/0 (socket says 172.21.15.158:38048) 2023-10-16T11:19:27.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.156+0000 7fbf77fff700 1 -- 172.21.15.158:0/2193623851 learned_addr learned my addr 172.21.15.158:0/2193623851 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:27.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf76ffd700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf7806e5d0 0x7fbf780afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf777fe700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806ab40 0x7fbf78154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf77fff700 1 -- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf7806e5d0 msgr2=0x7fbf780afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf77fff700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf7806e5d0 0x7fbf780afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf77fff700 1 -- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806ab40 msgr2=0x7fbf78154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf77fff700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806ab40 0x7fbf78154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf77fff700 1 -- 172.21.15.158:0/2193623851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf70009b30 con 0x7fbf780729b0 2023-10-16T11:19:27.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf76ffd700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf7806e5d0 0x7fbf780afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:27.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf777fe700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806ab40 0x7fbf78154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:27.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf77fff700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf780729b0 0x7fbf780c4690 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000ef90 tx=0x7fbf7000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2193623851 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70018070 con 0x7fbf780729b0 2023-10-16T11:19:27.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2193623851 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7000f040 con 0x7fbf780729b0 2023-10-16T11:19:27.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c0094a0 con 0x7fbf780729b0 2023-10-16T11:19:27.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2193623851 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70003c60 con 0x7fbf780729b0 2023-10-16T11:19:27.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.157+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf780b0ea0 con 0x7fbf780729b0 2023-10-16T11:19:27.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.158+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf780b1380 con 0x7fbf780729b0 2023-10-16T11:19:27.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.159+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2193623851 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf7000c9f0 con 0x7fbf780729b0 2023-10-16T11:19:27.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.159+0000 7fbf5ffff700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbf6406b020 0x7fbf6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.159+0000 7fbf777fe700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbf6406b020 0x7fbf6406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.159+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2193623851 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fbf70089cf0 con 0x7fbf780729b0 2023-10-16T11:19:27.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.159+0000 7fbf777fe700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbf6406b020 0x7fbf6406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000f60 tx=0x7fbf68009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.164+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2193623851 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf7008e070 con 0x7fbf780729b0 2023-10-16T11:19:27.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.358+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf780681b0 con 0x7fbf780729b0 2023-10-16T11:19:27.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.359+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2193623851 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+17531 (secure 0 0 0) 0x7fbf700548f0 con 0x7fbf780729b0 2023-10-16T11:19:27.361 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:27.362 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":217,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:24.314678+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:27.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbf6406b020 msgr2=0x7fbf6406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbf6406b020 0x7fbf6406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000f60 tx=0x7fbf68009040 comp rx=0 tx=0).stop 2023-10-16T11:19:27.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf780729b0 msgr2=0x7fbf780c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf780729b0 0x7fbf780c4690 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000ef90 tx=0x7fbf7000c3a0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 shutdown_connections 2023-10-16T11:19:27.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fbf6406b020 0x7fbf6406d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf780729b0 0x7fbf780c4690 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf7806ab40 0x7fbf78154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 --2- 172.21.15.158:0/2193623851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf7806e5d0 0x7fbf780afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 >> 172.21.15.158:0/2193623851 conn(0x7fbf780d02b0 msgr2=0x7fbf780ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:27.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 shutdown_connections 2023-10-16T11:19:27.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.362+0000 7fbf7dd49700 1 -- 172.21.15.158:0/2193623851 wait complete. 2023-10-16T11:19:27.376 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:19:27.376 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:19:27.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.495+0000 7fad3c6ce700 1 Processor -- start 2023-10-16T11:19:27.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.496+0000 7fad3c6ce700 1 -- start start 2023-10-16T11:19:27.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.496+0000 7fad3c6ce700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad3406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.496+0000 7fad3c6ce700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad340729b0 0x7fad34071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.496+0000 7fad3c6ce700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad34071e50 con 0x7fad3406e5d0 2023-10-16T11:19:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.496+0000 7fad3c6ce700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad34161270 con 0x7fad340729b0 2023-10-16T11:19:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.496+0000 7fad3c6ce700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad34161510 con 0x7fad3406ab40 2023-10-16T11:19:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad340729b0 0x7fad34071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad340729b0 0x7fad34071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38054/0 (socket says 172.21.15.158:38054) 2023-10-16T11:19:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 -- 172.21.15.158:0/4102416608 learned_addr learned my addr 172.21.15.158:0/4102416608 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3a46a700 1 --2- 172.21.15.158:0/4102416608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad3406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 -- 172.21.15.158:0/4102416608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 msgr2=0x7fad3406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:27.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 --2- 172.21.15.158:0/4102416608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad3406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 -- 172.21.15.158:0/4102416608 >> v1:172.21.15.172:6789/0 conn(0x7fad3406e5d0 legacy=0x7fad340722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 -- 172.21.15.158:0/4102416608 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad341617f0 con 0x7fad340729b0 2023-10-16T11:19:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad3ac6b700 1 --2- 172.21.15.158:0/4102416608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad34071910 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009e50 tx=0x7fad2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36de7fcb37b6cee8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad39468700 1 -- 172.21.15.158:0/4102416608 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c00e070 con 0x7fad340729b0 2023-10-16T11:19:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad39468700 1 -- 172.21.15.158:0/4102416608 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c00b9c0 con 0x7fad340729b0 2023-10-16T11:19:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.497+0000 7fad39468700 1 -- 172.21.15.158:0/4102416608 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c00bd70 con 0x7fad340729b0 2023-10-16T11:19:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 -- 172.21.15.158:0/4102416608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 msgr2=0x7fad34071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/4102416608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad34071910 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009e50 tx=0x7fad2c00b330 comp rx=0 tx=0).stop 2023-10-16T11:19:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 -- 172.21.15.158:0/4102416608 shutdown_connections 2023-10-16T11:19:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/4102416608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad34071910 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/4102416608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad3406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 -- 172.21.15.158:0/4102416608 >> 172.21.15.158:0/4102416608 conn(0x7fad340d02b0 msgr2=0x7fad340d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 -- 172.21.15.158:0/4102416608 shutdown_connections 2023-10-16T11:19:27.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.498+0000 7fad3c6ce700 1 -- 172.21.15.158:0/4102416608 wait complete. 2023-10-16T11:19:27.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.499+0000 7fad3c6ce700 1 Processor -- start 2023-10-16T11:19:27.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.499+0000 7fad3c6ce700 1 -- start start 2023-10-16T11:19:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad340c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad3406e5d0 0x7fad341f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad341f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340b12a0 con 0x7fad340729b0 2023-10-16T11:19:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340c5ba0 con 0x7fad3406e5d0 2023-10-16T11:19:27.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad340c5e80 con 0x7fad3406ab40 2023-10-16T11:19:27.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad341f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad341f67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38062/0 (socket says 172.21.15.158:38062) 2023-10-16T11:19:27.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 -- 172.21.15.158:0/3959259861 learned_addr learned my addr 172.21.15.158:0/3959259861 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:27.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad39c69700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad3406e5d0 0x7fad341f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3a46a700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad340c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 -- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 msgr2=0x7fad340c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad340c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 -- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad3406e5d0 msgr2=0x7fad341f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad3406e5d0 0x7fad341f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 -- 172.21.15.158:0/3959259861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c009b30 con 0x7fad340729b0 2023-10-16T11:19:27.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3a46a700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad340c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:27.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3ac6b700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad341f67e0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7fad2c012000 tx=0x7fad2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad2affd700 1 -- 172.21.15.158:0/3959259861 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c0048b0 con 0x7fad340729b0 2023-10-16T11:19:27.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad2affd700 1 -- 172.21.15.158:0/3959259861 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c0099b0 con 0x7fad340729b0 2023-10-16T11:19:27.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad340b0c50 con 0x7fad340729b0 2023-10-16T11:19:27.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad2affd700 1 -- 172.21.15.158:0/3959259861 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c01b670 con 0x7fad340729b0 2023-10-16T11:19:27.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.500+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad340adf20 con 0x7fad340729b0 2023-10-16T11:19:27.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.501+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad340b1410 con 0x7fad340729b0 2023-10-16T11:19:27.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.502+0000 7fad2affd700 1 -- 172.21.15.158:0/3959259861 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fad2c01b7d0 con 0x7fad340729b0 2023-10-16T11:19:27.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.502+0000 7fad2affd700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fad1806b020 0x7fad1806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.503+0000 7fad2affd700 1 -- 172.21.15.158:0/3959259861 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fad2c08b090 con 0x7fad340729b0 2023-10-16T11:19:27.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.503+0000 7fad3a46a700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fad1806b020 0x7fad1806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.503+0000 7fad3a46a700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fad1806b020 0x7fad1806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fad240099f0 tx=0x7fad24008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.507+0000 7fad2affd700 1 -- 172.21.15.158:0/3959259861 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fad2c055c90 con 0x7fad340729b0 2023-10-16T11:19:27.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.695+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad34161db0 con 0x7fad1806b020 2023-10-16T11:19:27.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.704+0000 7fad2affd700 1 -- 172.21.15.158:0/3959259861 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290149 (secure 0 0 0) 0x7fad24009680 con 0x7fad1806b020 2023-10-16T11:19:27.708 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:27.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fad1806b020 msgr2=0x7fad1806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fad1806b020 0x7fad1806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fad240099f0 tx=0x7fad24008040 comp rx=0 tx=0).stop 2023-10-16T11:19:27.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 msgr2=0x7fad341f67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad341f67e0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7fad2c012000 tx=0x7fad2c004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 shutdown_connections 2023-10-16T11:19:27.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fad1806b020 0x7fad1806d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad340729b0 0x7fad341f67e0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad3406ab40 0x7fad340c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 --2- 172.21.15.158:0/3959259861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad3406e5d0 0x7fad341f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 >> 172.21.15.158:0/3959259861 conn(0x7fad340d02b0 msgr2=0x7fad340ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:27.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 shutdown_connections 2023-10-16T11:19:27.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.707+0000 7fad3c6ce700 1 -- 172.21.15.158:0/3959259861 wait complete. 2023-10-16T11:19:27.713 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:19:27.722 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T11:19:27.274063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392720,"kb_used_data":22488,"kb_used_omap":12,"kb_used_meta":370099,"kb_avail":754582000,"statfs":{"total":773094113280,"available":772691968000,"internally_reserved":0,"allocated":23027712,"data_stored":12832196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":378981943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.956946"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835969+0000","last_change":"2023-10-16T11:19:23.334441+0000","last_active":"2023-10-16T11:19:24.835969+0000","last_peered":"2023-10-16T11:19:24.835969+0000","last_clean":"2023-10-16T11:19:24.835969+0000","last_became_active":"2023-10-16T11:19:23.334140+0000","last_became_peered":"2023-10-16T11:19:23.334140+0000","last_unstale":"2023-10-16T11:19:24.835969+0000","last_undegraded":"2023-10-16T11:19:24.835969+0000","last_fullsized":"2023-10-16T11:19:24.835969+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319516+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:24.319516+0000","last_peered":"2023-10-16T11:19:24.319516+0000","last_clean":"2023-10-16T11:19:24.319516+0000","last_became_active":"2023-10-16T11:19:23.331671+0000","last_became_peered":"2023-10-16T11:19:23.331671+0000","last_unstale":"2023-10-16T11:19:24.319516+0000","last_undegraded":"2023-10-16T11:19:24.319516+0000","last_fullsized":"2023-10-16T11:19:24.319516+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:13.976802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834982+0000","last_change":"2023-10-16T11:19:23.333266+0000","last_active":"2023-10-16T11:19:24.834982+0000","last_peered":"2023-10-16T11:19:24.834982+0000","last_clean":"2023-10-16T11:19:24.834982+0000","last_became_active":"2023-10-16T11:19:23.332412+0000","last_became_peered":"2023-10-16T11:19:23.332412+0000","last_unstale":"2023-10-16T11:19:24.834982+0000","last_undegraded":"2023-10-16T11:19:24.834982+0000","last_fullsized":"2023-10-16T11:19:24.834982+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:19.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836634+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:24.836634+0000","last_peered":"2023-10-16T11:19:24.836634+0000","last_clean":"2023-10-16T11:19:24.836634+0000","last_became_active":"2023-10-16T11:19:23.335513+0000","last_became_peered":"2023-10-16T11:19:23.335513+0000","last_unstale":"2023-10-16T11:19:24.836634+0000","last_undegraded":"2023-10-16T11:19:24.836634+0000","last_fullsized":"2023-10-16T11:19:24.836634+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:30.545707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318260+0000","last_change":"2023-10-16T11:19:23.330646+0000","last_active":"2023-10-16T11:19:24.318260+0000","last_peered":"2023-10-16T11:19:24.318260+0000","last_clean":"2023-10-16T11:19:24.318260+0000","last_became_active":"2023-10-16T11:19:23.330371+0000","last_became_peered":"2023-10-16T11:19:23.330371+0000","last_unstale":"2023-10-16T11:19:24.318260+0000","last_undegraded":"2023-10-16T11:19:24.318260+0000","last_fullsized":"2023-10-16T11:19:24.318260+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:55.845569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835127+0000","last_change":"2023-10-16T11:19:23.326663+0000","last_active":"2023-10-16T11:19:24.835127+0000","last_peered":"2023-10-16T11:19:24.835127+0000","last_clean":"2023-10-16T11:19:24.835127+0000","last_became_active":"2023-10-16T11:19:23.326378+0000","last_became_peered":"2023-10-16T11:19:23.326378+0000","last_unstale":"2023-10-16T11:19:24.835127+0000","last_undegraded":"2023-10-16T11:19:24.835127+0000","last_fullsized":"2023-10-16T11:19:24.835127+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:41.823015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318400+0000","last_change":"2023-10-16T11:19:23.329203+0000","last_active":"2023-10-16T11:19:24.318400+0000","last_peered":"2023-10-16T11:19:24.318400+0000","last_clean":"2023-10-16T11:19:24.318400+0000","last_became_active":"2023-10-16T11:19:23.328748+0000","last_became_peered":"2023-10-16T11:19:23.328748+0000","last_unstale":"2023-10-16T11:19:24.318400+0000","last_undegraded":"2023-10-16T11:19:24.318400+0000","last_fullsized":"2023-10-16T11:19:24.318400+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:09.942459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835119+0000","last_change":"2023-10-16T11:19:23.329954+0000","last_active":"2023-10-16T11:19:24.835119+0000","last_peered":"2023-10-16T11:19:24.835119+0000","last_clean":"2023-10-16T11:19:24.835119+0000","last_became_active":"2023-10-16T11:19:23.329688+0000","last_became_peered":"2023-10-16T11:19:23.329688+0000","last_unstale":"2023-10-16T11:19:24.835119+0000","last_undegraded":"2023-10-16T11:19:24.835119+0000","last_fullsized":"2023-10-16T11:19:24.835119+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:11.183201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834522+0000","last_change":"2023-10-16T11:19:23.336597+0000","last_active":"2023-10-16T11:19:24.834522+0000","last_peered":"2023-10-16T11:19:24.834522+0000","last_clean":"2023-10-16T11:19:24.834522+0000","last_became_active":"2023-10-16T11:19:23.336307+0000","last_became_peered":"2023-10-16T11:19:23.336307+0000","last_unstale":"2023-10-16T11:19:24.834522+0000","last_undegraded":"2023-10-16T11:19:24.834522+0000","last_fullsized":"2023-10-16T11:19:24.834522+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836687+0000","last_change":"2023-10-16T11:19:23.333382+0000","last_active":"2023-10-16T11:19:24.836687+0000","last_peered":"2023-10-16T11:19:24.836687+0000","last_clean":"2023-10-16T11:19:24.836687+0000","last_became_active":"2023-10-16T11:19:23.332643+0000","last_became_peered":"2023-10-16T11:19:23.332643+0000","last_unstale":"2023-10-16T11:19:24.836687+0000","last_undegraded":"2023-10-16T11:19:24.836687+0000","last_fullsized":"2023-10-16T11:19:24.836687+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:54.101849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835116+0000","last_change":"2023-10-16T11:19:23.333705+0000","last_active":"2023-10-16T11:19:24.835116+0000","last_peered":"2023-10-16T11:19:24.835116+0000","last_clean":"2023-10-16T11:19:24.835116+0000","last_became_active":"2023-10-16T11:19:23.333009+0000","last_became_peered":"2023-10-16T11:19:23.333009+0000","last_unstale":"2023-10-16T11:19:24.835116+0000","last_undegraded":"2023-10-16T11:19:24.835116+0000","last_fullsized":"2023-10-16T11:19:24.835116+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:41.811226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318204+0000","last_change":"2023-10-16T11:19:23.335413+0000","last_active":"2023-10-16T11:19:24.318204+0000","last_peered":"2023-10-16T11:19:24.318204+0000","last_clean":"2023-10-16T11:19:24.318204+0000","last_became_active":"2023-10-16T11:19:23.334994+0000","last_became_peered":"2023-10-16T11:19:23.334994+0000","last_unstale":"2023-10-16T11:19:24.318204+0000","last_undegraded":"2023-10-16T11:19:24.318204+0000","last_fullsized":"2023-10-16T11:19:24.318204+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:53.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836109+0000","last_change":"2023-10-16T11:19:23.330180+0000","last_active":"2023-10-16T11:19:24.836109+0000","last_peered":"2023-10-16T11:19:24.836109+0000","last_clean":"2023-10-16T11:19:24.836109+0000","last_became_active":"2023-10-16T11:19:23.329598+0000","last_became_peered":"2023-10-16T11:19:23.329598+0000","last_unstale":"2023-10-16T11:19:24.836109+0000","last_undegraded":"2023-10-16T11:19:24.836109+0000","last_fullsized":"2023-10-16T11:19:24.836109+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:59.613377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836143+0000","last_change":"2023-10-16T11:19:23.342633+0000","last_active":"2023-10-16T11:19:24.836143+0000","last_peered":"2023-10-16T11:19:24.836143+0000","last_clean":"2023-10-16T11:19:24.836143+0000","last_became_active":"2023-10-16T11:19:23.342346+0000","last_became_peered":"2023-10-16T11:19:23.342346+0000","last_unstale":"2023-10-16T11:19:24.836143+0000","last_undegraded":"2023-10-16T11:19:24.836143+0000","last_fullsized":"2023-10-16T11:19:24.836143+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:12.087797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319922+0000","last_change":"2023-10-16T11:19:23.325666+0000","last_active":"2023-10-16T11:19:24.319922+0000","last_peered":"2023-10-16T11:19:24.319922+0000","last_clean":"2023-10-16T11:19:24.319922+0000","last_became_active":"2023-10-16T11:19:23.325400+0000","last_became_peered":"2023-10-16T11:19:23.325400+0000","last_unstale":"2023-10-16T11:19:24.319922+0000","last_undegraded":"2023-10-16T11:19:24.319922+0000","last_fullsized":"2023-10-16T11:19:24.319922+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834781+0000","last_change":"2023-10-16T11:19:23.332953+0000","last_active":"2023-10-16T11:19:24.834781+0000","last_peered":"2023-10-16T11:19:24.834781+0000","last_clean":"2023-10-16T11:19:24.834781+0000","last_became_active":"2023-10-16T11:19:23.332662+0000","last_became_peered":"2023-10-16T11:19:23.332662+0000","last_unstale":"2023-10-16T11:19:24.834781+0000","last_undegraded":"2023-10-16T11:19:24.834781+0000","last_fullsized":"2023-10-16T11:19:24.834781+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:27.611049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834765+0000","last_change":"2023-10-16T11:19:23.334483+0000","last_active":"2023-10-16T11:19:24.834765+0000","last_peered":"2023-10-16T11:19:24.834765+0000","last_clean":"2023-10-16T11:19:24.834765+0000","last_became_active":"2023-10-16T11:19:23.333819+0000","last_became_peered":"2023-10-16T11:19:23.333819+0000","last_unstale":"2023-10-16T11:19:24.834765+0000","last_undegraded":"2023-10-16T11:19:24.834765+0000","last_fullsized":"2023-10-16T11:19:24.834765+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834629+0000","last_change":"2023-10-16T11:19:23.333633+0000","last_active":"2023-10-16T11:19:24.834629+0000","last_peered":"2023-10-16T11:19:24.834629+0000","last_clean":"2023-10-16T11:19:24.834629+0000","last_became_active":"2023-10-16T11:19:23.333280+0000","last_became_peered":"2023-10-16T11:19:23.333280+0000","last_unstale":"2023-10-16T11:19:24.834629+0000","last_undegraded":"2023-10-16T11:19:24.834629+0000","last_fullsized":"2023-10-16T11:19:24.834629+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835515+0000","last_change":"2023-10-16T11:19:23.337766+0000","last_active":"2023-10-16T11:19:24.835515+0000","last_peered":"2023-10-16T11:19:24.835515+0000","last_clean":"2023-10-16T11:19:24.835515+0000","last_became_active":"2023-10-16T11:19:23.337503+0000","last_became_peered":"2023-10-16T11:19:23.337503+0000","last_unstale":"2023-10-16T11:19:24.835515+0000","last_undegraded":"2023-10-16T11:19:24.835515+0000","last_fullsized":"2023-10-16T11:19:24.835515+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835489+0000","last_change":"2023-10-16T11:19:23.331248+0000","last_active":"2023-10-16T11:19:24.835489+0000","last_peered":"2023-10-16T11:19:24.835489+0000","last_clean":"2023-10-16T11:19:24.835489+0000","last_became_active":"2023-10-16T11:19:23.330931+0000","last_became_peered":"2023-10-16T11:19:23.330931+0000","last_unstale":"2023-10-16T11:19:24.835489+0000","last_undegraded":"2023-10-16T11:19:24.835489+0000","last_fullsized":"2023-10-16T11:19:24.835489+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835742+0000","last_change":"2023-10-16T11:19:23.334105+0000","last_active":"2023-10-16T11:19:24.835742+0000","last_peered":"2023-10-16T11:19:24.835742+0000","last_clean":"2023-10-16T11:19:24.835742+0000","last_became_active":"2023-10-16T11:19:23.333836+0000","last_became_peered":"2023-10-16T11:19:23.333836+0000","last_unstale":"2023-10-16T11:19:24.835742+0000","last_undegraded":"2023-10-16T11:19:24.835742+0000","last_fullsized":"2023-10-16T11:19:24.835742+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.432829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835415+0000","last_change":"2023-10-16T11:19:23.341632+0000","last_active":"2023-10-16T11:19:24.835415+0000","last_peered":"2023-10-16T11:19:24.835415+0000","last_clean":"2023-10-16T11:19:24.835415+0000","last_became_active":"2023-10-16T11:19:23.341312+0000","last_became_peered":"2023-10-16T11:19:23.341312+0000","last_unstale":"2023-10-16T11:19:24.835415+0000","last_undegraded":"2023-10-16T11:19:24.835415+0000","last_fullsized":"2023-10-16T11:19:24.835415+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319184+0000","last_change":"2023-10-16T11:19:23.341165+0000","last_active":"2023-10-16T11:19:24.319184+0000","last_peered":"2023-10-16T11:19:24.319184+0000","last_clean":"2023-10-16T11:19:24.319184+0000","last_became_active":"2023-10-16T11:19:23.340853+0000","last_became_peered":"2023-10-16T11:19:23.340853+0000","last_unstale":"2023-10-16T11:19:24.319184+0000","last_undegraded":"2023-10-16T11:19:24.319184+0000","last_fullsized":"2023-10-16T11:19:24.319184+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:41.539293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318052+0000","last_change":"2023-10-16T11:19:23.341264+0000","last_active":"2023-10-16T11:19:24.318052+0000","last_peered":"2023-10-16T11:19:24.318052+0000","last_clean":"2023-10-16T11:19:24.318052+0000","last_became_active":"2023-10-16T11:19:23.340921+0000","last_became_peered":"2023-10-16T11:19:23.340921+0000","last_unstale":"2023-10-16T11:19:24.318052+0000","last_undegraded":"2023-10-16T11:19:24.318052+0000","last_fullsized":"2023-10-16T11:19:24.318052+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.710291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834953+0000","last_change":"2023-10-16T11:19:23.334306+0000","last_active":"2023-10-16T11:19:24.834953+0000","last_peered":"2023-10-16T11:19:24.834953+0000","last_clean":"2023-10-16T11:19:24.834953+0000","last_became_active":"2023-10-16T11:19:23.333460+0000","last_became_peered":"2023-10-16T11:19:23.333460+0000","last_unstale":"2023-10-16T11:19:24.834953+0000","last_undegraded":"2023-10-16T11:19:24.834953+0000","last_fullsized":"2023-10-16T11:19:24.834953+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:53.175386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836320+0000","last_change":"2023-10-16T11:19:23.341307+0000","last_active":"2023-10-16T11:19:24.836320+0000","last_peered":"2023-10-16T11:19:24.836320+0000","last_clean":"2023-10-16T11:19:24.836320+0000","last_became_active":"2023-10-16T11:19:23.334457+0000","last_became_peered":"2023-10-16T11:19:23.334457+0000","last_unstale":"2023-10-16T11:19:24.836320+0000","last_undegraded":"2023-10-16T11:19:24.836320+0000","last_fullsized":"2023-10-16T11:19:24.836320+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835101+0000","last_change":"2023-10-16T11:19:23.332619+0000","last_active":"2023-10-16T11:19:24.835101+0000","last_peered":"2023-10-16T11:19:24.835101+0000","last_clean":"2023-10-16T11:19:24.835101+0000","last_became_active":"2023-10-16T11:19:23.332037+0000","last_became_peered":"2023-10-16T11:19:23.332037+0000","last_unstale":"2023-10-16T11:19:24.835101+0000","last_undegraded":"2023-10-16T11:19:24.835101+0000","last_fullsized":"2023-10-16T11:19:24.835101+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:48.332974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318285+0000","last_change":"2023-10-16T11:19:23.327468+0000","last_active":"2023-10-16T11:19:24.318285+0000","last_peered":"2023-10-16T11:19:24.318285+0000","last_clean":"2023-10-16T11:19:24.318285+0000","last_became_active":"2023-10-16T11:19:23.327176+0000","last_became_peered":"2023-10-16T11:19:23.327176+0000","last_unstale":"2023-10-16T11:19:24.318285+0000","last_undegraded":"2023-10-16T11:19:24.318285+0000","last_fullsized":"2023-10-16T11:19:24.318285+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:02.654427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834591+0000","last_change":"2023-10-16T11:19:23.330876+0000","last_active":"2023-10-16T11:19:24.834591+0000","last_peered":"2023-10-16T11:19:24.834591+0000","last_clean":"2023-10-16T11:19:24.834591+0000","last_became_active":"2023-10-16T11:19:23.330597+0000","last_became_peered":"2023-10-16T11:19:23.330597+0000","last_unstale":"2023-10-16T11:19:24.834591+0000","last_undegraded":"2023-10-16T11:19:24.834591+0000","last_fullsized":"2023-10-16T11:19:24.834591+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320234+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:24.320234+0000","last_peered":"2023-10-16T11:19:24.320234+0000","last_clean":"2023-10-16T11:19:24.320234+0000","last_became_active":"2023-10-16T11:19:23.335534+0000","last_became_peered":"2023-10-16T11:19:23.335534+0000","last_unstale":"2023-10-16T11:19:24.320234+0000","last_undegraded":"2023-10-16T11:19:24.320234+0000","last_fullsized":"2023-10-16T11:19:24.320234+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835252+0000","last_change":"2023-10-16T11:19:23.332826+0000","last_active":"2023-10-16T11:19:24.835252+0000","last_peered":"2023-10-16T11:19:24.835252+0000","last_clean":"2023-10-16T11:19:24.835252+0000","last_became_active":"2023-10-16T11:19:23.332541+0000","last_became_peered":"2023-10-16T11:19:23.332541+0000","last_unstale":"2023-10-16T11:19:24.835252+0000","last_undegraded":"2023-10-16T11:19:24.835252+0000","last_fullsized":"2023-10-16T11:19:24.835252+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:12:33.418278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835298+0000","last_change":"2023-10-16T11:19:23.323955+0000","last_active":"2023-10-16T11:19:24.835298+0000","last_peered":"2023-10-16T11:19:24.835298+0000","last_clean":"2023-10-16T11:19:24.835298+0000","last_became_active":"2023-10-16T11:19:23.323706+0000","last_became_peered":"2023-10-16T11:19:23.323706+0000","last_unstale":"2023-10-16T11:19:24.835298+0000","last_undegraded":"2023-10-16T11:19:24.835298+0000","last_fullsized":"2023-10-16T11:19:24.835298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.182127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319216+0000","last_change":"2023-10-16T11:19:23.341701+0000","last_active":"2023-10-16T11:19:24.319216+0000","last_peered":"2023-10-16T11:19:24.319216+0000","last_clean":"2023-10-16T11:19:24.319216+0000","last_became_active":"2023-10-16T11:19:23.341424+0000","last_became_peered":"2023-10-16T11:19:23.341424+0000","last_unstale":"2023-10-16T11:19:24.319216+0000","last_undegraded":"2023-10-16T11:19:24.319216+0000","last_fullsized":"2023-10-16T11:19:24.319216+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:25.448895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"216'4","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320219+0000","last_change":"2023-10-16T11:19:22.332308+0000","last_active":"2023-10-16T11:19:24.320219+0000","last_peered":"2023-10-16T11:19:24.320219+0000","last_clean":"2023-10-16T11:19:24.320219+0000","last_became_active":"2023-10-16T11:19:22.332005+0000","last_became_peered":"2023-10-16T11:19:22.332005+0000","last_unstale":"2023-10-16T11:19:24.320219+0000","last_undegraded":"2023-10-16T11:19:24.320219+0000","last_fullsized":"2023-10-16T11:19:24.320219+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836337+0000","last_change":"2023-10-16T11:19:23.342219+0000","last_active":"2023-10-16T11:19:24.836337+0000","last_peered":"2023-10-16T11:19:24.836337+0000","last_clean":"2023-10-16T11:19:24.836337+0000","last_became_active":"2023-10-16T11:19:23.334494+0000","last_became_peered":"2023-10-16T11:19:23.334494+0000","last_unstale":"2023-10-16T11:19:24.836337+0000","last_undegraded":"2023-10-16T11:19:24.836337+0000","last_fullsized":"2023-10-16T11:19:24.836337+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835938+0000","last_change":"2023-10-16T11:19:22.332064+0000","last_active":"2023-10-16T11:19:24.835938+0000","last_peered":"2023-10-16T11:19:24.835938+0000","last_clean":"2023-10-16T11:19:24.835938+0000","last_became_active":"2023-10-16T11:19:22.331746+0000","last_became_peered":"2023-10-16T11:19:22.331746+0000","last_unstale":"2023-10-16T11:19:24.835938+0000","last_undegraded":"2023-10-16T11:19:24.835938+0000","last_fullsized":"2023-10-16T11:19:24.835938+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836560+0000","last_change":"2023-10-16T11:19:23.326944+0000","last_active":"2023-10-16T11:19:24.836560+0000","last_peered":"2023-10-16T11:19:24.836560+0000","last_clean":"2023-10-16T11:19:24.836560+0000","last_became_active":"2023-10-16T11:19:23.326680+0000","last_became_peered":"2023-10-16T11:19:23.326680+0000","last_unstale":"2023-10-16T11:19:24.836560+0000","last_undegraded":"2023-10-16T11:19:24.836560+0000","last_fullsized":"2023-10-16T11:19:24.836560+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:28.070345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319282+0000","last_change":"2023-10-16T11:19:22.328385+0000","last_active":"2023-10-16T11:19:24.319282+0000","last_peered":"2023-10-16T11:19:24.319282+0000","last_clean":"2023-10-16T11:19:24.319282+0000","last_became_active":"2023-10-16T11:19:22.328096+0000","last_became_peered":"2023-10-16T11:19:22.328096+0000","last_unstale":"2023-10-16T11:19:24.319282+0000","last_undegraded":"2023-10-16T11:19:24.319282+0000","last_fullsized":"2023-10-16T11:19:24.319282+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:40.312176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318309+0000","last_change":"2023-10-16T11:19:23.335733+0000","last_active":"2023-10-16T11:19:24.318309+0000","last_peered":"2023-10-16T11:19:24.318309+0000","last_clean":"2023-10-16T11:19:24.318309+0000","last_became_active":"2023-10-16T11:19:23.334936+0000","last_became_peered":"2023-10-16T11:19:23.334936+0000","last_unstale":"2023-10-16T11:19:24.318309+0000","last_undegraded":"2023-10-16T11:19:24.318309+0000","last_fullsized":"2023-10-16T11:19:24.318309+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:08.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834582+0000","last_change":"2023-10-16T11:19:22.329968+0000","last_active":"2023-10-16T11:19:24.834582+0000","last_peered":"2023-10-16T11:19:24.834582+0000","last_clean":"2023-10-16T11:19:24.834582+0000","last_became_active":"2023-10-16T11:19:22.326360+0000","last_became_peered":"2023-10-16T11:19:22.326360+0000","last_unstale":"2023-10-16T11:19:24.834582+0000","last_undegraded":"2023-10-16T11:19:24.834582+0000","last_fullsized":"2023-10-16T11:19:24.834582+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320333+0000","last_change":"2023-10-16T11:19:23.336335+0000","last_active":"2023-10-16T11:19:24.320333+0000","last_peered":"2023-10-16T11:19:24.320333+0000","last_clean":"2023-10-16T11:19:24.320333+0000","last_became_active":"2023-10-16T11:19:23.335852+0000","last_became_peered":"2023-10-16T11:19:23.335852+0000","last_unstale":"2023-10-16T11:19:24.320333+0000","last_undegraded":"2023-10-16T11:19:24.320333+0000","last_fullsized":"2023-10-16T11:19:24.320333+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836093+0000","last_change":"2023-10-16T11:19:22.328217+0000","last_active":"2023-10-16T11:19:24.836093+0000","last_peered":"2023-10-16T11:19:24.836093+0000","last_clean":"2023-10-16T11:19:24.836093+0000","last_became_active":"2023-10-16T11:19:22.327901+0000","last_became_peered":"2023-10-16T11:19:22.327901+0000","last_unstale":"2023-10-16T11:19:24.836093+0000","last_undegraded":"2023-10-16T11:19:24.836093+0000","last_fullsized":"2023-10-16T11:19:24.836093+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:48.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320160+0000","last_change":"2023-10-16T11:19:23.330616+0000","last_active":"2023-10-16T11:19:24.320160+0000","last_peered":"2023-10-16T11:19:24.320160+0000","last_clean":"2023-10-16T11:19:24.320160+0000","last_became_active":"2023-10-16T11:19:23.330317+0000","last_became_peered":"2023-10-16T11:19:23.330317+0000","last_unstale":"2023-10-16T11:19:24.320160+0000","last_undegraded":"2023-10-16T11:19:24.320160+0000","last_fullsized":"2023-10-16T11:19:24.320160+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835202+0000","last_change":"2023-10-16T11:19:22.328061+0000","last_active":"2023-10-16T11:19:24.835202+0000","last_peered":"2023-10-16T11:19:24.835202+0000","last_clean":"2023-10-16T11:19:24.835202+0000","last_became_active":"2023-10-16T11:19:22.327773+0000","last_became_peered":"2023-10-16T11:19:22.327773+0000","last_unstale":"2023-10-16T11:19:24.835202+0000","last_undegraded":"2023-10-16T11:19:24.835202+0000","last_fullsized":"2023-10-16T11:19:24.835202+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:39.077769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834923+0000","last_change":"2023-10-16T11:19:23.334130+0000","last_active":"2023-10-16T11:19:24.834923+0000","last_peered":"2023-10-16T11:19:24.834923+0000","last_clean":"2023-10-16T11:19:24.834923+0000","last_became_active":"2023-10-16T11:19:23.333073+0000","last_became_peered":"2023-10-16T11:19:23.333073+0000","last_unstale":"2023-10-16T11:19:24.834923+0000","last_undegraded":"2023-10-16T11:19:24.834923+0000","last_fullsized":"2023-10-16T11:19:24.834923+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:38.491191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834767+0000","last_change":"2023-10-16T11:19:22.330754+0000","last_active":"2023-10-16T11:19:24.834767+0000","last_peered":"2023-10-16T11:19:24.834767+0000","last_clean":"2023-10-16T11:19:24.834767+0000","last_became_active":"2023-10-16T11:19:22.330490+0000","last_became_peered":"2023-10-16T11:19:22.330490+0000","last_unstale":"2023-10-16T11:19:24.834767+0000","last_undegraded":"2023-10-16T11:19:24.834767+0000","last_fullsized":"2023-10-16T11:19:24.834767+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:17.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319916+0000","last_change":"2023-10-16T11:19:23.330327+0000","last_active":"2023-10-16T11:19:24.319916+0000","last_peered":"2023-10-16T11:19:24.319916+0000","last_clean":"2023-10-16T11:19:24.319916+0000","last_became_active":"2023-10-16T11:19:23.330040+0000","last_became_peered":"2023-10-16T11:19:23.330040+0000","last_unstale":"2023-10-16T11:19:24.319916+0000","last_undegraded":"2023-10-16T11:19:24.319916+0000","last_fullsized":"2023-10-16T11:19:24.319916+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318527+0000","last_change":"2023-10-16T11:19:22.332253+0000","last_active":"2023-10-16T11:19:24.318527+0000","last_peered":"2023-10-16T11:19:24.318527+0000","last_clean":"2023-10-16T11:19:24.318527+0000","last_became_active":"2023-10-16T11:19:22.331970+0000","last_became_peered":"2023-10-16T11:19:22.331970+0000","last_unstale":"2023-10-16T11:19:24.318527+0000","last_undegraded":"2023-10-16T11:19:24.318527+0000","last_fullsized":"2023-10-16T11:19:24.318527+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:19.740978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834824+0000","last_change":"2023-10-16T11:19:23.331144+0000","last_active":"2023-10-16T11:19:24.834824+0000","last_peered":"2023-10-16T11:19:24.834824+0000","last_clean":"2023-10-16T11:19:24.834824+0000","last_became_active":"2023-10-16T11:19:23.330824+0000","last_became_peered":"2023-10-16T11:19:23.330824+0000","last_unstale":"2023-10-16T11:19:24.834824+0000","last_undegraded":"2023-10-16T11:19:24.834824+0000","last_fullsized":"2023-10-16T11:19:24.834824+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:44:27.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836013+0000","last_change":"2023-10-16T11:19:22.325211+0000","last_active":"2023-10-16T11:19:24.836013+0000","last_peered":"2023-10-16T11:19:24.836013+0000","last_clean":"2023-10-16T11:19:24.836013+0000","last_became_active":"2023-10-16T11:19:22.324901+0000","last_became_peered":"2023-10-16T11:19:22.324901+0000","last_unstale":"2023-10-16T11:19:24.836013+0000","last_undegraded":"2023-10-16T11:19:24.836013+0000","last_fullsized":"2023-10-16T11:19:24.836013+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:46.432434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834454+0000","last_change":"2023-10-16T11:19:23.338568+0000","last_active":"2023-10-16T11:19:24.834454+0000","last_peered":"2023-10-16T11:19:24.834454+0000","last_clean":"2023-10-16T11:19:24.834454+0000","last_became_active":"2023-10-16T11:19:23.338280+0000","last_became_peered":"2023-10-16T11:19:23.338280+0000","last_unstale":"2023-10-16T11:19:24.834454+0000","last_undegraded":"2023-10-16T11:19:24.834454+0000","last_fullsized":"2023-10-16T11:19:24.834454+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:40.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"216'3","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834606+0000","last_change":"2023-10-16T11:19:22.331150+0000","last_active":"2023-10-16T11:19:24.834606+0000","last_peered":"2023-10-16T11:19:24.834606+0000","last_clean":"2023-10-16T11:19:24.834606+0000","last_became_active":"2023-10-16T11:19:22.330820+0000","last_became_peered":"2023-10-16T11:19:22.330820+0000","last_unstale":"2023-10-16T11:19:24.834606+0000","last_undegraded":"2023-10-16T11:19:24.834606+0000","last_fullsized":"2023-10-16T11:19:24.834606+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318394+0000","last_change":"2023-10-16T11:19:23.336196+0000","last_active":"2023-10-16T11:19:24.318394+0000","last_peered":"2023-10-16T11:19:24.318394+0000","last_clean":"2023-10-16T11:19:24.318394+0000","last_became_active":"2023-10-16T11:19:23.335924+0000","last_became_peered":"2023-10-16T11:19:23.335924+0000","last_unstale":"2023-10-16T11:19:24.318394+0000","last_undegraded":"2023-10-16T11:19:24.318394+0000","last_fullsized":"2023-10-16T11:19:24.318394+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:27:01.350215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"216'6","reported_seq":18,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836624+0000","last_change":"2023-10-16T11:19:22.329082+0000","last_active":"2023-10-16T11:19:24.836624+0000","last_peered":"2023-10-16T11:19:24.836624+0000","last_clean":"2023-10-16T11:19:24.836624+0000","last_became_active":"2023-10-16T11:19:22.328756+0000","last_became_peered":"2023-10-16T11:19:22.328756+0000","last_unstale":"2023-10-16T11:19:24.836624+0000","last_undegraded":"2023-10-16T11:19:24.836624+0000","last_fullsized":"2023-10-16T11:19:24.836624+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.384963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319334+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:24.319334+0000","last_peered":"2023-10-16T11:19:24.319334+0000","last_clean":"2023-10-16T11:19:24.319334+0000","last_became_active":"2023-10-16T11:19:23.331725+0000","last_became_peered":"2023-10-16T11:19:23.331725+0000","last_unstale":"2023-10-16T11:19:24.319334+0000","last_undegraded":"2023-10-16T11:19:24.319334+0000","last_fullsized":"2023-10-16T11:19:24.319334+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:45.043730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319327+0000","last_change":"2023-10-16T11:19:22.325030+0000","last_active":"2023-10-16T11:19:24.319327+0000","last_peered":"2023-10-16T11:19:24.319327+0000","last_clean":"2023-10-16T11:19:24.319327+0000","last_became_active":"2023-10-16T11:19:22.324733+0000","last_became_peered":"2023-10-16T11:19:22.324733+0000","last_unstale":"2023-10-16T11:19:24.319327+0000","last_undegraded":"2023-10-16T11:19:24.319327+0000","last_fullsized":"2023-10-16T11:19:24.319327+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:29.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835180+0000","last_change":"2023-10-16T11:19:23.337758+0000","last_active":"2023-10-16T11:19:24.835180+0000","last_peered":"2023-10-16T11:19:24.835180+0000","last_clean":"2023-10-16T11:19:24.835180+0000","last_became_active":"2023-10-16T11:19:23.337147+0000","last_became_peered":"2023-10-16T11:19:23.337147+0000","last_unstale":"2023-10-16T11:19:24.835180+0000","last_undegraded":"2023-10-16T11:19:24.835180+0000","last_fullsized":"2023-10-16T11:19:24.835180+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:06.136692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835090+0000","last_change":"2023-10-16T11:19:22.327307+0000","last_active":"2023-10-16T11:19:24.835090+0000","last_peered":"2023-10-16T11:19:24.835090+0000","last_clean":"2023-10-16T11:19:24.835090+0000","last_became_active":"2023-10-16T11:19:22.327036+0000","last_became_peered":"2023-10-16T11:19:22.327036+0000","last_unstale":"2023-10-16T11:19:24.835090+0000","last_undegraded":"2023-10-16T11:19:24.835090+0000","last_fullsized":"2023-10-16T11:19:24.835090+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:24.276671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834767+0000","last_change":"2023-10-16T11:19:23.333500+0000","last_active":"2023-10-16T11:19:24.834767+0000","last_peered":"2023-10-16T11:19:24.834767+0000","last_clean":"2023-10-16T11:19:24.834767+0000","last_became_active":"2023-10-16T11:19:23.333215+0000","last_became_peered":"2023-10-16T11:19:23.333215+0000","last_unstale":"2023-10-16T11:19:24.834767+0000","last_undegraded":"2023-10-16T11:19:24.834767+0000","last_fullsized":"2023-10-16T11:19:24.834767+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834354+0000","last_change":"2023-10-16T11:19:22.331044+0000","last_active":"2023-10-16T11:19:24.834354+0000","last_peered":"2023-10-16T11:19:24.834354+0000","last_clean":"2023-10-16T11:19:24.834354+0000","last_became_active":"2023-10-16T11:19:22.330757+0000","last_became_peered":"2023-10-16T11:19:22.330757+0000","last_unstale":"2023-10-16T11:19:24.834354+0000","last_undegraded":"2023-10-16T11:19:24.834354+0000","last_fullsized":"2023-10-16T11:19:24.834354+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835584+0000","last_change":"2023-10-16T11:19:23.336992+0000","last_active":"2023-10-16T11:19:24.835584+0000","last_peered":"2023-10-16T11:19:24.835584+0000","last_clean":"2023-10-16T11:19:24.835584+0000","last_became_active":"2023-10-16T11:19:23.336724+0000","last_became_peered":"2023-10-16T11:19:23.336724+0000","last_unstale":"2023-10-16T11:19:24.835584+0000","last_undegraded":"2023-10-16T11:19:24.835584+0000","last_fullsized":"2023-10-16T11:19:24.835584+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:01.326738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"216'8","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.317995+0000","last_change":"2023-10-16T11:19:22.332156+0000","last_active":"2023-10-16T11:19:24.317995+0000","last_peered":"2023-10-16T11:19:24.317995+0000","last_clean":"2023-10-16T11:19:24.317995+0000","last_became_active":"2023-10-16T11:19:22.331863+0000","last_became_peered":"2023-10-16T11:19:22.331863+0000","last_unstale":"2023-10-16T11:19:24.317995+0000","last_undegraded":"2023-10-16T11:19:24.317995+0000","last_fullsized":"2023-10-16T11:19:24.317995+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:36.175530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835297+0000","last_change":"2023-10-16T11:19:23.333018+0000","last_active":"2023-10-16T11:19:24.835297+0000","last_peered":"2023-10-16T11:19:24.835297+0000","last_clean":"2023-10-16T11:19:24.835297+0000","last_became_active":"2023-10-16T11:19:23.332725+0000","last_became_peered":"2023-10-16T11:19:23.332725+0000","last_unstale":"2023-10-16T11:19:24.835297+0000","last_undegraded":"2023-10-16T11:19:24.835297+0000","last_fullsized":"2023-10-16T11:19:24.835297+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:39.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836118+0000","last_change":"2023-10-16T11:19:22.331432+0000","last_active":"2023-10-16T11:19:24.836118+0000","last_peered":"2023-10-16T11:19:24.836118+0000","last_clean":"2023-10-16T11:19:24.836118+0000","last_became_active":"2023-10-16T11:19:22.331124+0000","last_became_peered":"2023-10-16T11:19:22.331124+0000","last_unstale":"2023-10-16T11:19:24.836118+0000","last_undegraded":"2023-10-16T11:19:24.836118+0000","last_fullsized":"2023-10-16T11:19:24.836118+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836283+0000","last_change":"2023-10-16T11:19:23.338701+0000","last_active":"2023-10-16T11:19:24.836283+0000","last_peered":"2023-10-16T11:19:24.836283+0000","last_clean":"2023-10-16T11:19:24.836283+0000","last_became_active":"2023-10-16T11:19:23.334033+0000","last_became_peered":"2023-10-16T11:19:23.334033+0000","last_unstale":"2023-10-16T11:19:24.836283+0000","last_undegraded":"2023-10-16T11:19:24.836283+0000","last_fullsized":"2023-10-16T11:19:24.836283+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836041+0000","last_change":"2023-10-16T11:19:22.330569+0000","last_active":"2023-10-16T11:19:24.836041+0000","last_peered":"2023-10-16T11:19:24.836041+0000","last_clean":"2023-10-16T11:19:24.836041+0000","last_became_active":"2023-10-16T11:19:22.329740+0000","last_became_peered":"2023-10-16T11:19:22.329740+0000","last_unstale":"2023-10-16T11:19:24.836041+0000","last_undegraded":"2023-10-16T11:19:24.836041+0000","last_fullsized":"2023-10-16T11:19:24.836041+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835745+0000","last_change":"2023-10-16T11:19:23.337274+0000","last_active":"2023-10-16T11:19:24.835745+0000","last_peered":"2023-10-16T11:19:24.835745+0000","last_clean":"2023-10-16T11:19:24.835745+0000","last_became_active":"2023-10-16T11:19:23.336889+0000","last_became_peered":"2023-10-16T11:19:23.336889+0000","last_unstale":"2023-10-16T11:19:24.835745+0000","last_undegraded":"2023-10-16T11:19:24.835745+0000","last_fullsized":"2023-10-16T11:19:24.835745+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319614+0000","last_change":"2023-10-16T11:19:22.318314+0000","last_active":"2023-10-16T11:19:24.319614+0000","last_peered":"2023-10-16T11:19:24.319614+0000","last_clean":"2023-10-16T11:19:24.319614+0000","last_became_active":"2023-10-16T11:19:22.318015+0000","last_became_peered":"2023-10-16T11:19:22.318015+0000","last_unstale":"2023-10-16T11:19:24.319614+0000","last_undegraded":"2023-10-16T11:19:24.319614+0000","last_fullsized":"2023-10-16T11:19:24.319614+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.421816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835880+0000","last_change":"2023-10-16T11:19:23.331171+0000","last_active":"2023-10-16T11:19:24.835880+0000","last_peered":"2023-10-16T11:19:24.835880+0000","last_clean":"2023-10-16T11:19:24.835880+0000","last_became_active":"2023-10-16T11:19:23.330836+0000","last_became_peered":"2023-10-16T11:19:23.330836+0000","last_unstale":"2023-10-16T11:19:24.835880+0000","last_undegraded":"2023-10-16T11:19:24.835880+0000","last_fullsized":"2023-10-16T11:19:24.835880+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:45.818409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318309+0000","last_change":"2023-10-16T11:19:22.332355+0000","last_active":"2023-10-16T11:19:24.318309+0000","last_peered":"2023-10-16T11:19:24.318309+0000","last_clean":"2023-10-16T11:19:24.318309+0000","last_became_active":"2023-10-16T11:19:22.331520+0000","last_became_peered":"2023-10-16T11:19:22.331520+0000","last_unstale":"2023-10-16T11:19:24.318309+0000","last_undegraded":"2023-10-16T11:19:24.318309+0000","last_fullsized":"2023-10-16T11:19:24.318309+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:10.682623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319436+0000","last_change":"2023-10-16T11:19:23.331460+0000","last_active":"2023-10-16T11:19:24.319436+0000","last_peered":"2023-10-16T11:19:24.319436+0000","last_clean":"2023-10-16T11:19:24.319436+0000","last_became_active":"2023-10-16T11:19:23.331178+0000","last_became_peered":"2023-10-16T11:19:23.331178+0000","last_unstale":"2023-10-16T11:19:24.319436+0000","last_undegraded":"2023-10-16T11:19:24.319436+0000","last_fullsized":"2023-10-16T11:19:24.319436+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:36.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318426+0000","last_change":"2023-10-16T11:19:22.329655+0000","last_active":"2023-10-16T11:19:24.318426+0000","last_peered":"2023-10-16T11:19:24.318426+0000","last_clean":"2023-10-16T11:19:24.318426+0000","last_became_active":"2023-10-16T11:19:22.329366+0000","last_became_peered":"2023-10-16T11:19:22.329366+0000","last_unstale":"2023-10-16T11:19:24.318426+0000","last_undegraded":"2023-10-16T11:19:24.318426+0000","last_fullsized":"2023-10-16T11:19:24.318426+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:47.331154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835742+0000","last_change":"2023-10-16T11:19:23.333742+0000","last_active":"2023-10-16T11:19:24.835742+0000","last_peered":"2023-10-16T11:19:24.835742+0000","last_clean":"2023-10-16T11:19:24.835742+0000","last_became_active":"2023-10-16T11:19:23.333424+0000","last_became_peered":"2023-10-16T11:19:23.333424+0000","last_unstale":"2023-10-16T11:19:24.835742+0000","last_undegraded":"2023-10-16T11:19:24.835742+0000","last_fullsized":"2023-10-16T11:19:24.835742+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.156569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319530+0000","last_change":"2023-10-16T11:19:22.329858+0000","last_active":"2023-10-16T11:19:24.319530+0000","last_peered":"2023-10-16T11:19:24.319530+0000","last_clean":"2023-10-16T11:19:24.319530+0000","last_became_active":"2023-10-16T11:19:22.329586+0000","last_became_peered":"2023-10-16T11:19:22.329586+0000","last_unstale":"2023-10-16T11:19:24.319530+0000","last_undegraded":"2023-10-16T11:19:24.319530+0000","last_fullsized":"2023-10-16T11:19:24.319530+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:25.906324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834796+0000","last_change":"2023-10-16T11:19:23.333035+0000","last_active":"2023-10-16T11:19:24.834796+0000","last_peered":"2023-10-16T11:19:24.834796+0000","last_clean":"2023-10-16T11:19:24.834796+0000","last_became_active":"2023-10-16T11:19:23.332692+0000","last_became_peered":"2023-10-16T11:19:23.332692+0000","last_unstale":"2023-10-16T11:19:24.834796+0000","last_undegraded":"2023-10-16T11:19:24.834796+0000","last_fullsized":"2023-10-16T11:19:24.834796+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320007+0000","last_change":"2023-10-16T11:19:22.331457+0000","last_active":"2023-10-16T11:19:24.320007+0000","last_peered":"2023-10-16T11:19:24.320007+0000","last_clean":"2023-10-16T11:19:24.320007+0000","last_became_active":"2023-10-16T11:19:22.331010+0000","last_became_peered":"2023-10-16T11:19:22.331010+0000","last_unstale":"2023-10-16T11:19:24.320007+0000","last_undegraded":"2023-10-16T11:19:24.320007+0000","last_fullsized":"2023-10-16T11:19:24.320007+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:38.048633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836566+0000","last_change":"2023-10-16T11:19:23.336920+0000","last_active":"2023-10-16T11:19:24.836566+0000","last_peered":"2023-10-16T11:19:24.836566+0000","last_clean":"2023-10-16T11:19:24.836566+0000","last_became_active":"2023-10-16T11:19:23.336473+0000","last_became_peered":"2023-10-16T11:19:23.336473+0000","last_unstale":"2023-10-16T11:19:24.836566+0000","last_undegraded":"2023-10-16T11:19:24.836566+0000","last_fullsized":"2023-10-16T11:19:24.836566+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:22.421887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835914+0000","last_change":"2023-10-16T11:19:22.326294+0000","last_active":"2023-10-16T11:19:24.835914+0000","last_peered":"2023-10-16T11:19:24.835914+0000","last_clean":"2023-10-16T11:19:24.835914+0000","last_became_active":"2023-10-16T11:19:22.326012+0000","last_became_peered":"2023-10-16T11:19:22.326012+0000","last_unstale":"2023-10-16T11:19:24.835914+0000","last_undegraded":"2023-10-16T11:19:24.835914+0000","last_fullsized":"2023-10-16T11:19:24.835914+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835497+0000","last_change":"2023-10-16T11:19:23.333192+0000","last_active":"2023-10-16T11:19:24.835497+0000","last_peered":"2023-10-16T11:19:24.835497+0000","last_clean":"2023-10-16T11:19:24.835497+0000","last_became_active":"2023-10-16T11:19:23.332739+0000","last_became_peered":"2023-10-16T11:19:23.332739+0000","last_unstale":"2023-10-16T11:19:24.835497+0000","last_undegraded":"2023-10-16T11:19:24.835497+0000","last_fullsized":"2023-10-16T11:19:24.835497+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835009+0000","last_change":"2023-10-16T11:19:22.332950+0000","last_active":"2023-10-16T11:19:24.835009+0000","last_peered":"2023-10-16T11:19:24.835009+0000","last_clean":"2023-10-16T11:19:24.835009+0000","last_became_active":"2023-10-16T11:19:22.332650+0000","last_became_peered":"2023-10-16T11:19:22.332650+0000","last_unstale":"2023-10-16T11:19:24.835009+0000","last_undegraded":"2023-10-16T11:19:24.835009+0000","last_fullsized":"2023-10-16T11:19:24.835009+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834946+0000","last_change":"2023-10-16T11:19:23.325482+0000","last_active":"2023-10-16T11:19:24.834946+0000","last_peered":"2023-10-16T11:19:24.834946+0000","last_clean":"2023-10-16T11:19:24.834946+0000","last_became_active":"2023-10-16T11:19:23.325193+0000","last_became_peered":"2023-10-16T11:19:23.325193+0000","last_unstale":"2023-10-16T11:19:24.834946+0000","last_undegraded":"2023-10-16T11:19:24.834946+0000","last_fullsized":"2023-10-16T11:19:24.834946+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.191635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836563+0000","last_change":"2023-10-16T11:19:22.331182+0000","last_active":"2023-10-16T11:19:24.836563+0000","last_peered":"2023-10-16T11:19:24.836563+0000","last_clean":"2023-10-16T11:19:24.836563+0000","last_became_active":"2023-10-16T11:19:22.330860+0000","last_became_peered":"2023-10-16T11:19:22.330860+0000","last_unstale":"2023-10-16T11:19:24.836563+0000","last_undegraded":"2023-10-16T11:19:24.836563+0000","last_fullsized":"2023-10-16T11:19:24.836563+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834593+0000","last_change":"2023-10-16T11:19:23.331571+0000","last_active":"2023-10-16T11:19:24.834593+0000","last_peered":"2023-10-16T11:19:24.834593+0000","last_clean":"2023-10-16T11:19:24.834593+0000","last_became_active":"2023-10-16T11:19:23.331242+0000","last_became_peered":"2023-10-16T11:19:23.331242+0000","last_unstale":"2023-10-16T11:19:24.834593+0000","last_undegraded":"2023-10-16T11:19:24.834593+0000","last_fullsized":"2023-10-16T11:19:24.834593+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320127+0000","last_change":"2023-10-16T11:19:22.326754+0000","last_active":"2023-10-16T11:19:24.320127+0000","last_peered":"2023-10-16T11:19:24.320127+0000","last_clean":"2023-10-16T11:19:24.320127+0000","last_became_active":"2023-10-16T11:19:22.326480+0000","last_became_peered":"2023-10-16T11:19:22.326480+0000","last_unstale":"2023-10-16T11:19:24.320127+0000","last_undegraded":"2023-10-16T11:19:24.320127+0000","last_fullsized":"2023-10-16T11:19:24.320127+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835594+0000","last_change":"2023-10-16T11:19:23.336160+0000","last_active":"2023-10-16T11:19:24.835594+0000","last_peered":"2023-10-16T11:19:24.835594+0000","last_clean":"2023-10-16T11:19:24.835594+0000","last_became_active":"2023-10-16T11:19:23.333920+0000","last_became_peered":"2023-10-16T11:19:23.333920+0000","last_unstale":"2023-10-16T11:19:24.835594+0000","last_undegraded":"2023-10-16T11:19:24.835594+0000","last_fullsized":"2023-10-16T11:19:24.835594+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836151+0000","last_change":"2023-10-16T11:19:22.335923+0000","last_active":"2023-10-16T11:19:24.836151+0000","last_peered":"2023-10-16T11:19:24.836151+0000","last_clean":"2023-10-16T11:19:24.836151+0000","last_became_active":"2023-10-16T11:19:22.335705+0000","last_became_peered":"2023-10-16T11:19:22.335705+0000","last_unstale":"2023-10-16T11:19:24.836151+0000","last_undegraded":"2023-10-16T11:19:24.836151+0000","last_fullsized":"2023-10-16T11:19:24.836151+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:40.803363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835646+0000","last_change":"2023-10-16T11:19:23.329220+0000","last_active":"2023-10-16T11:19:24.835646+0000","last_peered":"2023-10-16T11:19:24.835646+0000","last_clean":"2023-10-16T11:19:24.835646+0000","last_became_active":"2023-10-16T11:19:23.328923+0000","last_became_peered":"2023-10-16T11:19:23.328923+0000","last_unstale":"2023-10-16T11:19:24.835646+0000","last_undegraded":"2023-10-16T11:19:24.835646+0000","last_fullsized":"2023-10-16T11:19:24.835646+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835972+0000","last_change":"2023-10-16T11:19:22.333752+0000","last_active":"2023-10-16T11:19:24.835972+0000","last_peered":"2023-10-16T11:19:24.835972+0000","last_clean":"2023-10-16T11:19:24.835972+0000","last_became_active":"2023-10-16T11:19:22.333497+0000","last_became_peered":"2023-10-16T11:19:22.333497+0000","last_unstale":"2023-10-16T11:19:24.835972+0000","last_undegraded":"2023-10-16T11:19:24.835972+0000","last_fullsized":"2023-10-16T11:19:24.835972+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835880+0000","last_change":"2023-10-16T11:19:23.330624+0000","last_active":"2023-10-16T11:19:24.835880+0000","last_peered":"2023-10-16T11:19:24.835880+0000","last_clean":"2023-10-16T11:19:24.835880+0000","last_became_active":"2023-10-16T11:19:23.329588+0000","last_became_peered":"2023-10-16T11:19:23.329588+0000","last_unstale":"2023-10-16T11:19:24.835880+0000","last_undegraded":"2023-10-16T11:19:24.835880+0000","last_fullsized":"2023-10-16T11:19:24.835880+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:10.835410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834650+0000","last_change":"2023-10-16T11:19:22.329623+0000","last_active":"2023-10-16T11:19:24.834650+0000","last_peered":"2023-10-16T11:19:24.834650+0000","last_clean":"2023-10-16T11:19:24.834650+0000","last_became_active":"2023-10-16T11:19:22.329344+0000","last_became_peered":"2023-10-16T11:19:22.329344+0000","last_unstale":"2023-10-16T11:19:24.834650+0000","last_undegraded":"2023-10-16T11:19:24.834650+0000","last_fullsized":"2023-10-16T11:19:24.834650+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835841+0000","last_change":"2023-10-16T11:19:23.329115+0000","last_active":"2023-10-16T11:19:24.835841+0000","last_peered":"2023-10-16T11:19:24.835841+0000","last_clean":"2023-10-16T11:19:24.835841+0000","last_became_active":"2023-10-16T11:19:23.328830+0000","last_became_peered":"2023-10-16T11:19:23.328830+0000","last_unstale":"2023-10-16T11:19:24.835841+0000","last_undegraded":"2023-10-16T11:19:24.835841+0000","last_fullsized":"2023-10-16T11:19:24.835841+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:17.247488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836376+0000","last_change":"2023-10-16T11:19:22.332640+0000","last_active":"2023-10-16T11:19:24.836376+0000","last_peered":"2023-10-16T11:19:24.836376+0000","last_clean":"2023-10-16T11:19:24.836376+0000","last_became_active":"2023-10-16T11:19:22.331608+0000","last_became_peered":"2023-10-16T11:19:22.331608+0000","last_unstale":"2023-10-16T11:19:24.836376+0000","last_undegraded":"2023-10-16T11:19:24.836376+0000","last_fullsized":"2023-10-16T11:19:24.836376+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835155+0000","last_change":"2023-10-16T11:19:23.324999+0000","last_active":"2023-10-16T11:19:24.835155+0000","last_peered":"2023-10-16T11:19:24.835155+0000","last_clean":"2023-10-16T11:19:24.835155+0000","last_became_active":"2023-10-16T11:19:23.324686+0000","last_became_peered":"2023-10-16T11:19:23.324686+0000","last_unstale":"2023-10-16T11:19:24.835155+0000","last_undegraded":"2023-10-16T11:19:24.835155+0000","last_fullsized":"2023-10-16T11:19:24.835155+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:42.281964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318104+0000","last_change":"2023-10-16T11:19:22.334152+0000","last_active":"2023-10-16T11:19:24.318104+0000","last_peered":"2023-10-16T11:19:24.318104+0000","last_clean":"2023-10-16T11:19:24.318104+0000","last_became_active":"2023-10-16T11:19:22.333859+0000","last_became_peered":"2023-10-16T11:19:22.333859+0000","last_unstale":"2023-10-16T11:19:24.318104+0000","last_undegraded":"2023-10-16T11:19:24.318104+0000","last_fullsized":"2023-10-16T11:19:24.318104+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:50.619981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836232+0000","last_change":"2023-10-16T11:19:23.334041+0000","last_active":"2023-10-16T11:19:24.836232+0000","last_peered":"2023-10-16T11:19:24.836232+0000","last_clean":"2023-10-16T11:19:24.836232+0000","last_became_active":"2023-10-16T11:19:23.333752+0000","last_became_peered":"2023-10-16T11:19:23.333752+0000","last_unstale":"2023-10-16T11:19:24.836232+0000","last_undegraded":"2023-10-16T11:19:24.836232+0000","last_fullsized":"2023-10-16T11:19:24.836232+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834559+0000","last_change":"2023-10-16T11:19:22.332936+0000","last_active":"2023-10-16T11:19:24.834559+0000","last_peered":"2023-10-16T11:19:24.834559+0000","last_clean":"2023-10-16T11:19:24.834559+0000","last_became_active":"2023-10-16T11:19:22.332598+0000","last_became_peered":"2023-10-16T11:19:22.332598+0000","last_unstale":"2023-10-16T11:19:24.834559+0000","last_undegraded":"2023-10-16T11:19:24.834559+0000","last_fullsized":"2023-10-16T11:19:24.834559+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:34.597859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"207'348","reported_seq":1208,"reported_epoch":216,"state":"active+clean","last_fresh":"2023-10-16T11:19:23.318091+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:19:23.318091+0000","last_peered":"2023-10-16T11:19:23.318091+0000","last_clean":"2023-10-16T11:19:23.318091+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:19:23.318091+0000","last_undegraded":"2023-10-16T11:19:23.318091+0000","last_fullsized":"2023-10-16T11:19:23.318091+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575989,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52476,"kb_used_data":3248,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319364,"statfs":{"total":96636764160,"available":96583028736,"internally_reserved":0,"allocated":3325952,"data_stored":2044330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.75700000000000001,"5min":0.84299999999999997,"15min":0.98499999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71499999999999997,"5min":0.79500000000000004,"15min":0.999},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.749,"5min":0.83999999999999997,"15min":1.0640000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63200000000000001,"15min":0.65100000000000002},"min":{"1min":0.40100000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.83099999999999996,"15min":1.0349999999999999},"last":0.621}]},{"osd":2,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":1.117},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.90400000000000003,"15min":1.05},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.442,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69799999999999995,"5min":0.83999999999999997,"15min":1.022},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.67200000000000004,"15min":0.68600000000000005},"min":{"1min":0.56399999999999995,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.083},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":0.77700000000000002,"5min":0.81599999999999995,"15min":0.94799999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"max":{"1min":0.78000000000000003,"5min":0.83299999999999996,"15min":0.96299999999999997},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.91200000000000003},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.80500000000000005,"5min":0.85299999999999998,"15min":0.92000000000000004},"last":0.73499999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.88500000000000001,"15min":0.92600000000000005},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59599999999999997,"15min":0.61099999999999999},"min":{"1min":0.43099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77600000000000002,"5min":0.88700000000000001,"15min":0.91300000000000003},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38020,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333820,"statfs":{"total":96636764160,"available":96597831680,"internally_reserved":0,"allocated":2744320,"data_stored":1461574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.40400000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.80600000000000005,"5min":0.81000000000000005,"15min":0.91000000000000003},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.73799999999999999,"5min":0.82599999999999996,"15min":0.88500000000000001},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61399999999999999},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59499999999999997,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.95499999999999996},"last":0.67900000000000005}]},{"osd":2,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.77400000000000002,"5min":0.83099999999999996,"15min":0.95599999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.621,"15min":0.63},"min":{"1min":0.435,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82899999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.69599999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61399999999999999,"15min":0.621},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":0.81699999999999995,"5min":0.84099999999999997,"15min":0.92700000000000005},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61599999999999999,"15min":0.629},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":0.97799999999999998},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34499999999999997,"5min":0.33900000000000002,"15min":0.30299999999999999},"max":{"1min":0.72899999999999998,"5min":0.87,"15min":0.87},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.75700000000000001,"5min":0.85499999999999998,"15min":0.89900000000000002},"last":0.443},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83299999999999996,"5min":0.92500000000000004,"15min":1.1599999999999999},"last":0.5}]},{"osd":7,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.42399999999999999,"15min":0.38300000000000001},"max":{"1min":0.80000000000000004,"5min":0.90700000000000003,"15min":0.93300000000000005},"last":0.378}]}]},{"osd":1,"up_from":13,"seq":55834575990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47720,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324120,"statfs":{"total":96636764160,"available":96587898880,"internally_reserved":0,"allocated":2715648,"data_stored":1452280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40899999999999997,"15min":0.41399999999999998},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.622,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.32300000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63900000000000001,"5min":0.64100000000000001,"15min":0.71699999999999997},"last":0.35799999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.46400000000000002,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.62,"5min":0.66600000000000004,"15min":0.70099999999999996},"last":0.308}]},{"osd":3,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54100000000000004,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.441,"15min":0.443},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53200000000000003,"5min":0.623,"15min":0.70499999999999996},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82099999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":1.0309999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88900000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90100000000000002,"5min":1.01,"15min":1.0760000000000001},"last":0.499}]},{"osd":6,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81999999999999995,"5min":0.93100000000000005,"15min":1.0149999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.68799999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.70299999999999996,"15min":0.70399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83399999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.81599999999999995,"5min":0.99399999999999999,"15min":1.016},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575990,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42740,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329100,"statfs":{"total":96636764160,"available":96592998400,"internally_reserved":0,"allocated":3313664,"data_stored":2042738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44400000000000001,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60699999999999998,"5min":1.272,"15min":1.272},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67700000000000005,"5min":1.351,"15min":1.351},"last":0.439}]},{"osd":4,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.46800000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.875,"5min":1.391,"15min":1.391},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80000000000000004,"5min":0.874,"15min":1.089},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.95599999999999996,"15min":1.216},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.69799999999999995},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90900000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.47199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89100000000000001,"5min":1.038,"15min":1.1180000000000001},"last":0.71499999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.621},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.72799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47744,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324096,"statfs":{"total":96636764160,"available":96587874304,"internally_reserved":0,"allocated":2732032,"data_stored":1459470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.63300000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.432}]},{"osd":3,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69799999999999995,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77800000000000002},"last":0.61099999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.95399999999999996,"15min":1.097},"last":0.626},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67200000000000004,"15min":0.68200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.91800000000000004,"5min":0.96199999999999997,"15min":1.016},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90500000000000003,"5min":1.022,"15min":1.0389999999999999},"last":0.74399999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91800000000000004,"5min":1.022,"15min":1.022},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91900000000000004,"5min":0.98999999999999999,"15min":1.004},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90400000000000003,"5min":1.0569999999999999,"15min":1.071},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.75600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575989,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51912,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319928,"statfs":{"total":96636764160,"available":96583606272,"internally_reserved":0,"allocated":2740224,"data_stored":1459506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38100000000000001,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.68700000000000006},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":1.0029999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.60899999999999999,"15min":0.67300000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.74299999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.58999999999999997,"5min":0.66800000000000004,"15min":0.96299999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64200000000000002,"5min":0.69899999999999995,"15min":1.024},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90600000000000003,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"last":0.749}]},{"osd":5,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.99099999999999999,"15min":1.026},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0640000000000001},"last":0.76300000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41799999999999998},"max":{"1min":0.85999999999999999,"5min":0.92900000000000005,"15min":1.0369999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.0620000000000001},"last":0.79000000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":1.002},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88400000000000001,"5min":1.006,"15min":1.1899999999999999},"last":0.77800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575989,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47736,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324104,"statfs":{"total":96636764160,"available":96587882496,"internally_reserved":0,"allocated":2732032,"data_stored":1459962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.625},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.623},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71599999999999997,"5min":0.82499999999999996,"15min":0.90700000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58799999999999997,"15min":0.623},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60599999999999998,"15min":0.63900000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.84399999999999997,"15min":0.88400000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.624},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.83799999999999997,"15min":0.995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.51500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.90500000000000003,"15min":1.115},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.35499999999999998,"15min":0.34200000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.83899999999999997},"last":0.622},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.64900000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575989,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64372,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307468,"statfs":{"total":96636764160,"available":96570847232,"internally_reserved":0,"allocated":2723840,"data_stored":1452336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.88600000000000001},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60299999999999998},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.86499999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.91400000000000003},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40899999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85899999999999999,"5min":0.878,"15min":0.878},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55300000000000005,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.68500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:19:27.728 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:19:27.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.880+0000 7fcfc6030700 1 Processor -- start 2023-10-16T11:19:27.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.881+0000 7fcfc6030700 1 -- start start 2023-10-16T11:19:27.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.881+0000 7fcfc6030700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcfc00b6e10 0x7fcfc01521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.881+0000 7fcfc6030700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc015c5d0 con 0x7fcfc00cdee0 2023-10-16T11:19:27.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.881+0000 7fcfc6030700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc015c8b0 con 0x7fcfc00ce2c0 2023-10-16T11:19:27.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.881+0000 7fcfc6030700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc015d2c0 con 0x7fcfc00b6e10 2023-10-16T11:19:27.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.881+0000 7fcfbf7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcfc00cdee0 0x7fcfc015bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38242/0 (socket says 172.21.15.158:38242) 2023-10-16T11:19:27.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.881+0000 7fcfbf7fe700 1 -- 172.21.15.158:0/670796505 learned_addr learned my addr 172.21.15.158:0/670796505 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:27.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbeffd700 1 --2- 172.21.15.158:0/670796505 >> v2:172.21.15.172:3300/0 conn(0x7fcfc00b6e10 0x7fcfc01521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589438498 0 0) 0x7fcfc015c5d0 con 0x7fcfc00cdee0 2023-10-16T11:19:27.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfac0035b0 con 0x7fcfc00cdee0 2023-10-16T11:19:27.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 551139890 0 0) 0x7fcfac0035b0 con 0x7fcfc00cdee0 2023-10-16T11:19:27.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 >> v2:172.21.15.172:3300/0 conn(0x7fcfc00b6e10 msgr2=0x7fcfc01521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:27.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 --2- 172.21.15.158:0/670796505 >> v2:172.21.15.172:3300/0 conn(0x7fcfc00b6e10 0x7fcfc01521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 >> v1:172.21.15.172:6789/0 conn(0x7fcfc00ce2c0 legacy=0x7fcfc00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc015cb90 con 0x7fcfc00cdee0 2023-10-16T11:19:27.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcfa8001670 con 0x7fcfc00cdee0 2023-10-16T11:19:27.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 >> v1:172.21.15.158:6789/0 conn(0x7fcfc00cdee0 legacy=0x7fcfc015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac003a00 0x7fcfac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac004400 0x7fcfac0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac005170 0x7fcfac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc015d2c0 con 0x7fcfac003a00 2023-10-16T11:19:27.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac00c0b0 con 0x7fcfac004400 2023-10-16T11:19:27.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac00c270 con 0x7fcfac005170 2023-10-16T11:19:27.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbf7fe700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac003a00 0x7fcfac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbeffd700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac004400 0x7fcfac0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbffff700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac005170 0x7fcfac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbf7fe700 1 -- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac005170 msgr2=0x7fcfac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:27.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbf7fe700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac005170 0x7fcfac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbf7fe700 1 -- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac004400 msgr2=0x7fcfac0048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:27.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbf7fe700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac004400 0x7fcfac0048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbffff700 1 -- 172.21.15.158:0/670796505 reap_dead start 2023-10-16T11:19:27.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.882+0000 7fcfbf7fe700 1 -- 172.21.15.158:0/670796505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac0035b0 con 0x7fcfac003a00 2023-10-16T11:19:27.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfbf7fe700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac003a00 0x7fcfac003de0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800c2f0 tx=0x7fcfac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7610e762c8b186 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800cdf0 con 0x7fcfac003a00 2023-10-16T11:19:27.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa8005960 con 0x7fcfac003a00 2023-10-16T11:19:27.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfbe7fc700 1 -- 172.21.15.158:0/670796505 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8005cf0 con 0x7fcfac003a00 2023-10-16T11:19:27.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfc6030700 1 -- 172.21.15.158:0/670796505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac003a00 msgr2=0x7fcfac003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfc6030700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac003a00 0x7fcfac003de0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800c2f0 tx=0x7fcfac005880 comp rx=0 tx=0).stop 2023-10-16T11:19:27.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfc6030700 1 -- 172.21.15.158:0/670796505 shutdown_connections 2023-10-16T11:19:27.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfc6030700 1 --2- 172.21.15.158:0/670796505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac003a00 0x7fcfac003de0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfc6030700 1 -- 172.21.15.158:0/670796505 >> 172.21.15.158:0/670796505 conn(0x7fcfc006a8d0 msgr2=0x7fcfc006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:27.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfc6030700 1 -- 172.21.15.158:0/670796505 shutdown_connections 2023-10-16T11:19:27.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.883+0000 7fcfc6030700 1 -- 172.21.15.158:0/670796505 wait complete. 2023-10-16T11:19:27.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 Processor -- start 2023-10-16T11:19:27.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 -- start start 2023-10-16T11:19:27.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac003a00 0x7fcfc014f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc00cf8f0 0x7fcfc00d1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfc00c6cf0 0x7fcfc00c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc014fb60 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c76b0 con 0x7fcfac003a00 2023-10-16T11:19:27.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.885+0000 7fcfc6030700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01515d0 con 0x7fcfc00cf8f0 2023-10-16T11:19:27.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfc00c6cf0 0x7fcfc00c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfc00c6cf0 0x7fcfc00c7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38082/0 (socket says 172.21.15.158:38082) 2023-10-16T11:19:27.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 -- 172.21.15.158:0/2144455616 learned_addr learned my addr 172.21.15.158:0/2144455616 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:27.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 -- 172.21.15.158:0/2144455616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc00cf8f0 msgr2=0x7fcfc00d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc00cf8f0 0x7fcfc00d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 -- 172.21.15.158:0/2144455616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac003a00 msgr2=0x7fcfc014f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:27.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbf7fe700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac003a00 0x7fcfc014f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac003a00 0x7fcfc014f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:27.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 -- 172.21.15.158:0/2144455616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00d09c0 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfbffff700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfc00c6cf0 0x7fcfc00c7170 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00715e0 tx=0x7fcfb800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfb7fff700 1 -- 172.21.15.158:0/2144455616 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8003c10 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfb7fff700 1 -- 172.21.15.158:0/2144455616 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8003db0 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc00cfe40 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfb7fff700 1 -- 172.21.15.158:0/2144455616 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb800dbd0 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.886+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc00d0400 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.887+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc0151760 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.888+0000 7fcfb7fff700 1 -- 172.21.15.158:0/2144455616 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcfb800dd30 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.888+0000 7fcfb7fff700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcfa006b020 0x7fcfa006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:27.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.888+0000 7fcfb7fff700 1 -- 172.21.15.158:0/2144455616 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fcfb8089da0 con 0x7fcfc00c6cf0 2023-10-16T11:19:27.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.888+0000 7fcfbf7fe700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcfa006b020 0x7fcfa006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:27.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.889+0000 7fcfbf7fe700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcfa006b020 0x7fcfa006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00c7940 tx=0x7fcfa8007000 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:27.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:27.893+0000 7fcfb7fff700 1 -- 172.21.15.158:0/2144455616 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcfc0151760 con 0x7fcfc00c6cf0 2023-10-16T11:19:28.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.080+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcfc015cee0 con 0x7fcfa006b020 2023-10-16T11:19:28.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.088+0000 7fcfb7fff700 1 -- 172.21.15.158:0/2144455616 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290149 (secure 0 0 0) 0x7fcfa8001430 con 0x7fcfa006b020 2023-10-16T11:19:28.093 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:28.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcfa006b020 msgr2=0x7fcfa006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcfa006b020 0x7fcfa006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00c7940 tx=0x7fcfa8007000 comp rx=0 tx=0).stop 2023-10-16T11:19:28.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfc00c6cf0 msgr2=0x7fcfc00c7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfc00c6cf0 0x7fcfc00c7170 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00715e0 tx=0x7fcfb800d430 comp rx=0 tx=0).stop 2023-10-16T11:19:28.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 shutdown_connections 2023-10-16T11:19:28.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fcfa006b020 0x7fcfa006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfc00c6cf0 0x7fcfc00c7170 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac003a00 0x7fcfc014f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 --2- 172.21.15.158:0/2144455616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc00cf8f0 0x7fcfc00d1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 >> 172.21.15.158:0/2144455616 conn(0x7fcfc006a8d0 msgr2=0x7fcfc0159730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:28.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 shutdown_connections 2023-10-16T11:19:28.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.091+0000 7fcfc6030700 1 -- 172.21.15.158:0/2144455616 wait complete. 2023-10-16T11:19:28.098 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:19:28.105 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T11:19:27.274063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392720,"kb_used_data":22488,"kb_used_omap":12,"kb_used_meta":370099,"kb_avail":754582000,"statfs":{"total":773094113280,"available":772691968000,"internally_reserved":0,"allocated":23027712,"data_stored":12832196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":378981943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.956946"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835969+0000","last_change":"2023-10-16T11:19:23.334441+0000","last_active":"2023-10-16T11:19:24.835969+0000","last_peered":"2023-10-16T11:19:24.835969+0000","last_clean":"2023-10-16T11:19:24.835969+0000","last_became_active":"2023-10-16T11:19:23.334140+0000","last_became_peered":"2023-10-16T11:19:23.334140+0000","last_unstale":"2023-10-16T11:19:24.835969+0000","last_undegraded":"2023-10-16T11:19:24.835969+0000","last_fullsized":"2023-10-16T11:19:24.835969+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319516+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:24.319516+0000","last_peered":"2023-10-16T11:19:24.319516+0000","last_clean":"2023-10-16T11:19:24.319516+0000","last_became_active":"2023-10-16T11:19:23.331671+0000","last_became_peered":"2023-10-16T11:19:23.331671+0000","last_unstale":"2023-10-16T11:19:24.319516+0000","last_undegraded":"2023-10-16T11:19:24.319516+0000","last_fullsized":"2023-10-16T11:19:24.319516+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:13.976802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834982+0000","last_change":"2023-10-16T11:19:23.333266+0000","last_active":"2023-10-16T11:19:24.834982+0000","last_peered":"2023-10-16T11:19:24.834982+0000","last_clean":"2023-10-16T11:19:24.834982+0000","last_became_active":"2023-10-16T11:19:23.332412+0000","last_became_peered":"2023-10-16T11:19:23.332412+0000","last_unstale":"2023-10-16T11:19:24.834982+0000","last_undegraded":"2023-10-16T11:19:24.834982+0000","last_fullsized":"2023-10-16T11:19:24.834982+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:19.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836634+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:24.836634+0000","last_peered":"2023-10-16T11:19:24.836634+0000","last_clean":"2023-10-16T11:19:24.836634+0000","last_became_active":"2023-10-16T11:19:23.335513+0000","last_became_peered":"2023-10-16T11:19:23.335513+0000","last_unstale":"2023-10-16T11:19:24.836634+0000","last_undegraded":"2023-10-16T11:19:24.836634+0000","last_fullsized":"2023-10-16T11:19:24.836634+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:30.545707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318260+0000","last_change":"2023-10-16T11:19:23.330646+0000","last_active":"2023-10-16T11:19:24.318260+0000","last_peered":"2023-10-16T11:19:24.318260+0000","last_clean":"2023-10-16T11:19:24.318260+0000","last_became_active":"2023-10-16T11:19:23.330371+0000","last_became_peered":"2023-10-16T11:19:23.330371+0000","last_unstale":"2023-10-16T11:19:24.318260+0000","last_undegraded":"2023-10-16T11:19:24.318260+0000","last_fullsized":"2023-10-16T11:19:24.318260+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:55.845569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835127+0000","last_change":"2023-10-16T11:19:23.326663+0000","last_active":"2023-10-16T11:19:24.835127+0000","last_peered":"2023-10-16T11:19:24.835127+0000","last_clean":"2023-10-16T11:19:24.835127+0000","last_became_active":"2023-10-16T11:19:23.326378+0000","last_became_peered":"2023-10-16T11:19:23.326378+0000","last_unstale":"2023-10-16T11:19:24.835127+0000","last_undegraded":"2023-10-16T11:19:24.835127+0000","last_fullsized":"2023-10-16T11:19:24.835127+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:41.823015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318400+0000","last_change":"2023-10-16T11:19:23.329203+0000","last_active":"2023-10-16T11:19:24.318400+0000","last_peered":"2023-10-16T11:19:24.318400+0000","last_clean":"2023-10-16T11:19:24.318400+0000","last_became_active":"2023-10-16T11:19:23.328748+0000","last_became_peered":"2023-10-16T11:19:23.328748+0000","last_unstale":"2023-10-16T11:19:24.318400+0000","last_undegraded":"2023-10-16T11:19:24.318400+0000","last_fullsized":"2023-10-16T11:19:24.318400+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:09.942459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835119+0000","last_change":"2023-10-16T11:19:23.329954+0000","last_active":"2023-10-16T11:19:24.835119+0000","last_peered":"2023-10-16T11:19:24.835119+0000","last_clean":"2023-10-16T11:19:24.835119+0000","last_became_active":"2023-10-16T11:19:23.329688+0000","last_became_peered":"2023-10-16T11:19:23.329688+0000","last_unstale":"2023-10-16T11:19:24.835119+0000","last_undegraded":"2023-10-16T11:19:24.835119+0000","last_fullsized":"2023-10-16T11:19:24.835119+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:11.183201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834522+0000","last_change":"2023-10-16T11:19:23.336597+0000","last_active":"2023-10-16T11:19:24.834522+0000","last_peered":"2023-10-16T11:19:24.834522+0000","last_clean":"2023-10-16T11:19:24.834522+0000","last_became_active":"2023-10-16T11:19:23.336307+0000","last_became_peered":"2023-10-16T11:19:23.336307+0000","last_unstale":"2023-10-16T11:19:24.834522+0000","last_undegraded":"2023-10-16T11:19:24.834522+0000","last_fullsized":"2023-10-16T11:19:24.834522+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836687+0000","last_change":"2023-10-16T11:19:23.333382+0000","last_active":"2023-10-16T11:19:24.836687+0000","last_peered":"2023-10-16T11:19:24.836687+0000","last_clean":"2023-10-16T11:19:24.836687+0000","last_became_active":"2023-10-16T11:19:23.332643+0000","last_became_peered":"2023-10-16T11:19:23.332643+0000","last_unstale":"2023-10-16T11:19:24.836687+0000","last_undegraded":"2023-10-16T11:19:24.836687+0000","last_fullsized":"2023-10-16T11:19:24.836687+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:54.101849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835116+0000","last_change":"2023-10-16T11:19:23.333705+0000","last_active":"2023-10-16T11:19:24.835116+0000","last_peered":"2023-10-16T11:19:24.835116+0000","last_clean":"2023-10-16T11:19:24.835116+0000","last_became_active":"2023-10-16T11:19:23.333009+0000","last_became_peered":"2023-10-16T11:19:23.333009+0000","last_unstale":"2023-10-16T11:19:24.835116+0000","last_undegraded":"2023-10-16T11:19:24.835116+0000","last_fullsized":"2023-10-16T11:19:24.835116+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:41.811226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318204+0000","last_change":"2023-10-16T11:19:23.335413+0000","last_active":"2023-10-16T11:19:24.318204+0000","last_peered":"2023-10-16T11:19:24.318204+0000","last_clean":"2023-10-16T11:19:24.318204+0000","last_became_active":"2023-10-16T11:19:23.334994+0000","last_became_peered":"2023-10-16T11:19:23.334994+0000","last_unstale":"2023-10-16T11:19:24.318204+0000","last_undegraded":"2023-10-16T11:19:24.318204+0000","last_fullsized":"2023-10-16T11:19:24.318204+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:53.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836109+0000","last_change":"2023-10-16T11:19:23.330180+0000","last_active":"2023-10-16T11:19:24.836109+0000","last_peered":"2023-10-16T11:19:24.836109+0000","last_clean":"2023-10-16T11:19:24.836109+0000","last_became_active":"2023-10-16T11:19:23.329598+0000","last_became_peered":"2023-10-16T11:19:23.329598+0000","last_unstale":"2023-10-16T11:19:24.836109+0000","last_undegraded":"2023-10-16T11:19:24.836109+0000","last_fullsized":"2023-10-16T11:19:24.836109+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:59.613377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836143+0000","last_change":"2023-10-16T11:19:23.342633+0000","last_active":"2023-10-16T11:19:24.836143+0000","last_peered":"2023-10-16T11:19:24.836143+0000","last_clean":"2023-10-16T11:19:24.836143+0000","last_became_active":"2023-10-16T11:19:23.342346+0000","last_became_peered":"2023-10-16T11:19:23.342346+0000","last_unstale":"2023-10-16T11:19:24.836143+0000","last_undegraded":"2023-10-16T11:19:24.836143+0000","last_fullsized":"2023-10-16T11:19:24.836143+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:12.087797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319922+0000","last_change":"2023-10-16T11:19:23.325666+0000","last_active":"2023-10-16T11:19:24.319922+0000","last_peered":"2023-10-16T11:19:24.319922+0000","last_clean":"2023-10-16T11:19:24.319922+0000","last_became_active":"2023-10-16T11:19:23.325400+0000","last_became_peered":"2023-10-16T11:19:23.325400+0000","last_unstale":"2023-10-16T11:19:24.319922+0000","last_undegraded":"2023-10-16T11:19:24.319922+0000","last_fullsized":"2023-10-16T11:19:24.319922+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834781+0000","last_change":"2023-10-16T11:19:23.332953+0000","last_active":"2023-10-16T11:19:24.834781+0000","last_peered":"2023-10-16T11:19:24.834781+0000","last_clean":"2023-10-16T11:19:24.834781+0000","last_became_active":"2023-10-16T11:19:23.332662+0000","last_became_peered":"2023-10-16T11:19:23.332662+0000","last_unstale":"2023-10-16T11:19:24.834781+0000","last_undegraded":"2023-10-16T11:19:24.834781+0000","last_fullsized":"2023-10-16T11:19:24.834781+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:27.611049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834765+0000","last_change":"2023-10-16T11:19:23.334483+0000","last_active":"2023-10-16T11:19:24.834765+0000","last_peered":"2023-10-16T11:19:24.834765+0000","last_clean":"2023-10-16T11:19:24.834765+0000","last_became_active":"2023-10-16T11:19:23.333819+0000","last_became_peered":"2023-10-16T11:19:23.333819+0000","last_unstale":"2023-10-16T11:19:24.834765+0000","last_undegraded":"2023-10-16T11:19:24.834765+0000","last_fullsized":"2023-10-16T11:19:24.834765+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834629+0000","last_change":"2023-10-16T11:19:23.333633+0000","last_active":"2023-10-16T11:19:24.834629+0000","last_peered":"2023-10-16T11:19:24.834629+0000","last_clean":"2023-10-16T11:19:24.834629+0000","last_became_active":"2023-10-16T11:19:23.333280+0000","last_became_peered":"2023-10-16T11:19:23.333280+0000","last_unstale":"2023-10-16T11:19:24.834629+0000","last_undegraded":"2023-10-16T11:19:24.834629+0000","last_fullsized":"2023-10-16T11:19:24.834629+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835515+0000","last_change":"2023-10-16T11:19:23.337766+0000","last_active":"2023-10-16T11:19:24.835515+0000","last_peered":"2023-10-16T11:19:24.835515+0000","last_clean":"2023-10-16T11:19:24.835515+0000","last_became_active":"2023-10-16T11:19:23.337503+0000","last_became_peered":"2023-10-16T11:19:23.337503+0000","last_unstale":"2023-10-16T11:19:24.835515+0000","last_undegraded":"2023-10-16T11:19:24.835515+0000","last_fullsized":"2023-10-16T11:19:24.835515+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835489+0000","last_change":"2023-10-16T11:19:23.331248+0000","last_active":"2023-10-16T11:19:24.835489+0000","last_peered":"2023-10-16T11:19:24.835489+0000","last_clean":"2023-10-16T11:19:24.835489+0000","last_became_active":"2023-10-16T11:19:23.330931+0000","last_became_peered":"2023-10-16T11:19:23.330931+0000","last_unstale":"2023-10-16T11:19:24.835489+0000","last_undegraded":"2023-10-16T11:19:24.835489+0000","last_fullsized":"2023-10-16T11:19:24.835489+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835742+0000","last_change":"2023-10-16T11:19:23.334105+0000","last_active":"2023-10-16T11:19:24.835742+0000","last_peered":"2023-10-16T11:19:24.835742+0000","last_clean":"2023-10-16T11:19:24.835742+0000","last_became_active":"2023-10-16T11:19:23.333836+0000","last_became_peered":"2023-10-16T11:19:23.333836+0000","last_unstale":"2023-10-16T11:19:24.835742+0000","last_undegraded":"2023-10-16T11:19:24.835742+0000","last_fullsized":"2023-10-16T11:19:24.835742+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.432829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835415+0000","last_change":"2023-10-16T11:19:23.341632+0000","last_active":"2023-10-16T11:19:24.835415+0000","last_peered":"2023-10-16T11:19:24.835415+0000","last_clean":"2023-10-16T11:19:24.835415+0000","last_became_active":"2023-10-16T11:19:23.341312+0000","last_became_peered":"2023-10-16T11:19:23.341312+0000","last_unstale":"2023-10-16T11:19:24.835415+0000","last_undegraded":"2023-10-16T11:19:24.835415+0000","last_fullsized":"2023-10-16T11:19:24.835415+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319184+0000","last_change":"2023-10-16T11:19:23.341165+0000","last_active":"2023-10-16T11:19:24.319184+0000","last_peered":"2023-10-16T11:19:24.319184+0000","last_clean":"2023-10-16T11:19:24.319184+0000","last_became_active":"2023-10-16T11:19:23.340853+0000","last_became_peered":"2023-10-16T11:19:23.340853+0000","last_unstale":"2023-10-16T11:19:24.319184+0000","last_undegraded":"2023-10-16T11:19:24.319184+0000","last_fullsized":"2023-10-16T11:19:24.319184+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:41.539293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318052+0000","last_change":"2023-10-16T11:19:23.341264+0000","last_active":"2023-10-16T11:19:24.318052+0000","last_peered":"2023-10-16T11:19:24.318052+0000","last_clean":"2023-10-16T11:19:24.318052+0000","last_became_active":"2023-10-16T11:19:23.340921+0000","last_became_peered":"2023-10-16T11:19:23.340921+0000","last_unstale":"2023-10-16T11:19:24.318052+0000","last_undegraded":"2023-10-16T11:19:24.318052+0000","last_fullsized":"2023-10-16T11:19:24.318052+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.710291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834953+0000","last_change":"2023-10-16T11:19:23.334306+0000","last_active":"2023-10-16T11:19:24.834953+0000","last_peered":"2023-10-16T11:19:24.834953+0000","last_clean":"2023-10-16T11:19:24.834953+0000","last_became_active":"2023-10-16T11:19:23.333460+0000","last_became_peered":"2023-10-16T11:19:23.333460+0000","last_unstale":"2023-10-16T11:19:24.834953+0000","last_undegraded":"2023-10-16T11:19:24.834953+0000","last_fullsized":"2023-10-16T11:19:24.834953+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:53.175386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836320+0000","last_change":"2023-10-16T11:19:23.341307+0000","last_active":"2023-10-16T11:19:24.836320+0000","last_peered":"2023-10-16T11:19:24.836320+0000","last_clean":"2023-10-16T11:19:24.836320+0000","last_became_active":"2023-10-16T11:19:23.334457+0000","last_became_peered":"2023-10-16T11:19:23.334457+0000","last_unstale":"2023-10-16T11:19:24.836320+0000","last_undegraded":"2023-10-16T11:19:24.836320+0000","last_fullsized":"2023-10-16T11:19:24.836320+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835101+0000","last_change":"2023-10-16T11:19:23.332619+0000","last_active":"2023-10-16T11:19:24.835101+0000","last_peered":"2023-10-16T11:19:24.835101+0000","last_clean":"2023-10-16T11:19:24.835101+0000","last_became_active":"2023-10-16T11:19:23.332037+0000","last_became_peered":"2023-10-16T11:19:23.332037+0000","last_unstale":"2023-10-16T11:19:24.835101+0000","last_undegraded":"2023-10-16T11:19:24.835101+0000","last_fullsized":"2023-10-16T11:19:24.835101+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:48.332974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318285+0000","last_change":"2023-10-16T11:19:23.327468+0000","last_active":"2023-10-16T11:19:24.318285+0000","last_peered":"2023-10-16T11:19:24.318285+0000","last_clean":"2023-10-16T11:19:24.318285+0000","last_became_active":"2023-10-16T11:19:23.327176+0000","last_became_peered":"2023-10-16T11:19:23.327176+0000","last_unstale":"2023-10-16T11:19:24.318285+0000","last_undegraded":"2023-10-16T11:19:24.318285+0000","last_fullsized":"2023-10-16T11:19:24.318285+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:02.654427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834591+0000","last_change":"2023-10-16T11:19:23.330876+0000","last_active":"2023-10-16T11:19:24.834591+0000","last_peered":"2023-10-16T11:19:24.834591+0000","last_clean":"2023-10-16T11:19:24.834591+0000","last_became_active":"2023-10-16T11:19:23.330597+0000","last_became_peered":"2023-10-16T11:19:23.330597+0000","last_unstale":"2023-10-16T11:19:24.834591+0000","last_undegraded":"2023-10-16T11:19:24.834591+0000","last_fullsized":"2023-10-16T11:19:24.834591+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320234+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:24.320234+0000","last_peered":"2023-10-16T11:19:24.320234+0000","last_clean":"2023-10-16T11:19:24.320234+0000","last_became_active":"2023-10-16T11:19:23.335534+0000","last_became_peered":"2023-10-16T11:19:23.335534+0000","last_unstale":"2023-10-16T11:19:24.320234+0000","last_undegraded":"2023-10-16T11:19:24.320234+0000","last_fullsized":"2023-10-16T11:19:24.320234+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835252+0000","last_change":"2023-10-16T11:19:23.332826+0000","last_active":"2023-10-16T11:19:24.835252+0000","last_peered":"2023-10-16T11:19:24.835252+0000","last_clean":"2023-10-16T11:19:24.835252+0000","last_became_active":"2023-10-16T11:19:23.332541+0000","last_became_peered":"2023-10-16T11:19:23.332541+0000","last_unstale":"2023-10-16T11:19:24.835252+0000","last_undegraded":"2023-10-16T11:19:24.835252+0000","last_fullsized":"2023-10-16T11:19:24.835252+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:12:33.418278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835298+0000","last_change":"2023-10-16T11:19:23.323955+0000","last_active":"2023-10-16T11:19:24.835298+0000","last_peered":"2023-10-16T11:19:24.835298+0000","last_clean":"2023-10-16T11:19:24.835298+0000","last_became_active":"2023-10-16T11:19:23.323706+0000","last_became_peered":"2023-10-16T11:19:23.323706+0000","last_unstale":"2023-10-16T11:19:24.835298+0000","last_undegraded":"2023-10-16T11:19:24.835298+0000","last_fullsized":"2023-10-16T11:19:24.835298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.182127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319216+0000","last_change":"2023-10-16T11:19:23.341701+0000","last_active":"2023-10-16T11:19:24.319216+0000","last_peered":"2023-10-16T11:19:24.319216+0000","last_clean":"2023-10-16T11:19:24.319216+0000","last_became_active":"2023-10-16T11:19:23.341424+0000","last_became_peered":"2023-10-16T11:19:23.341424+0000","last_unstale":"2023-10-16T11:19:24.319216+0000","last_undegraded":"2023-10-16T11:19:24.319216+0000","last_fullsized":"2023-10-16T11:19:24.319216+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:25.448895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"216'4","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320219+0000","last_change":"2023-10-16T11:19:22.332308+0000","last_active":"2023-10-16T11:19:24.320219+0000","last_peered":"2023-10-16T11:19:24.320219+0000","last_clean":"2023-10-16T11:19:24.320219+0000","last_became_active":"2023-10-16T11:19:22.332005+0000","last_became_peered":"2023-10-16T11:19:22.332005+0000","last_unstale":"2023-10-16T11:19:24.320219+0000","last_undegraded":"2023-10-16T11:19:24.320219+0000","last_fullsized":"2023-10-16T11:19:24.320219+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836337+0000","last_change":"2023-10-16T11:19:23.342219+0000","last_active":"2023-10-16T11:19:24.836337+0000","last_peered":"2023-10-16T11:19:24.836337+0000","last_clean":"2023-10-16T11:19:24.836337+0000","last_became_active":"2023-10-16T11:19:23.334494+0000","last_became_peered":"2023-10-16T11:19:23.334494+0000","last_unstale":"2023-10-16T11:19:24.836337+0000","last_undegraded":"2023-10-16T11:19:24.836337+0000","last_fullsized":"2023-10-16T11:19:24.836337+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835938+0000","last_change":"2023-10-16T11:19:22.332064+0000","last_active":"2023-10-16T11:19:24.835938+0000","last_peered":"2023-10-16T11:19:24.835938+0000","last_clean":"2023-10-16T11:19:24.835938+0000","last_became_active":"2023-10-16T11:19:22.331746+0000","last_became_peered":"2023-10-16T11:19:22.331746+0000","last_unstale":"2023-10-16T11:19:24.835938+0000","last_undegraded":"2023-10-16T11:19:24.835938+0000","last_fullsized":"2023-10-16T11:19:24.835938+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836560+0000","last_change":"2023-10-16T11:19:23.326944+0000","last_active":"2023-10-16T11:19:24.836560+0000","last_peered":"2023-10-16T11:19:24.836560+0000","last_clean":"2023-10-16T11:19:24.836560+0000","last_became_active":"2023-10-16T11:19:23.326680+0000","last_became_peered":"2023-10-16T11:19:23.326680+0000","last_unstale":"2023-10-16T11:19:24.836560+0000","last_undegraded":"2023-10-16T11:19:24.836560+0000","last_fullsized":"2023-10-16T11:19:24.836560+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:28.070345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319282+0000","last_change":"2023-10-16T11:19:22.328385+0000","last_active":"2023-10-16T11:19:24.319282+0000","last_peered":"2023-10-16T11:19:24.319282+0000","last_clean":"2023-10-16T11:19:24.319282+0000","last_became_active":"2023-10-16T11:19:22.328096+0000","last_became_peered":"2023-10-16T11:19:22.328096+0000","last_unstale":"2023-10-16T11:19:24.319282+0000","last_undegraded":"2023-10-16T11:19:24.319282+0000","last_fullsized":"2023-10-16T11:19:24.319282+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:40.312176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318309+0000","last_change":"2023-10-16T11:19:23.335733+0000","last_active":"2023-10-16T11:19:24.318309+0000","last_peered":"2023-10-16T11:19:24.318309+0000","last_clean":"2023-10-16T11:19:24.318309+0000","last_became_active":"2023-10-16T11:19:23.334936+0000","last_became_peered":"2023-10-16T11:19:23.334936+0000","last_unstale":"2023-10-16T11:19:24.318309+0000","last_undegraded":"2023-10-16T11:19:24.318309+0000","last_fullsized":"2023-10-16T11:19:24.318309+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:08.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834582+0000","last_change":"2023-10-16T11:19:22.329968+0000","last_active":"2023-10-16T11:19:24.834582+0000","last_peered":"2023-10-16T11:19:24.834582+0000","last_clean":"2023-10-16T11:19:24.834582+0000","last_became_active":"2023-10-16T11:19:22.326360+0000","last_became_peered":"2023-10-16T11:19:22.326360+0000","last_unstale":"2023-10-16T11:19:24.834582+0000","last_undegraded":"2023-10-16T11:19:24.834582+0000","last_fullsized":"2023-10-16T11:19:24.834582+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320333+0000","last_change":"2023-10-16T11:19:23.336335+0000","last_active":"2023-10-16T11:19:24.320333+0000","last_peered":"2023-10-16T11:19:24.320333+0000","last_clean":"2023-10-16T11:19:24.320333+0000","last_became_active":"2023-10-16T11:19:23.335852+0000","last_became_peered":"2023-10-16T11:19:23.335852+0000","last_unstale":"2023-10-16T11:19:24.320333+0000","last_undegraded":"2023-10-16T11:19:24.320333+0000","last_fullsized":"2023-10-16T11:19:24.320333+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836093+0000","last_change":"2023-10-16T11:19:22.328217+0000","last_active":"2023-10-16T11:19:24.836093+0000","last_peered":"2023-10-16T11:19:24.836093+0000","last_clean":"2023-10-16T11:19:24.836093+0000","last_became_active":"2023-10-16T11:19:22.327901+0000","last_became_peered":"2023-10-16T11:19:22.327901+0000","last_unstale":"2023-10-16T11:19:24.836093+0000","last_undegraded":"2023-10-16T11:19:24.836093+0000","last_fullsized":"2023-10-16T11:19:24.836093+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:48.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320160+0000","last_change":"2023-10-16T11:19:23.330616+0000","last_active":"2023-10-16T11:19:24.320160+0000","last_peered":"2023-10-16T11:19:24.320160+0000","last_clean":"2023-10-16T11:19:24.320160+0000","last_became_active":"2023-10-16T11:19:23.330317+0000","last_became_peered":"2023-10-16T11:19:23.330317+0000","last_unstale":"2023-10-16T11:19:24.320160+0000","last_undegraded":"2023-10-16T11:19:24.320160+0000","last_fullsized":"2023-10-16T11:19:24.320160+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835202+0000","last_change":"2023-10-16T11:19:22.328061+0000","last_active":"2023-10-16T11:19:24.835202+0000","last_peered":"2023-10-16T11:19:24.835202+0000","last_clean":"2023-10-16T11:19:24.835202+0000","last_became_active":"2023-10-16T11:19:22.327773+0000","last_became_peered":"2023-10-16T11:19:22.327773+0000","last_unstale":"2023-10-16T11:19:24.835202+0000","last_undegraded":"2023-10-16T11:19:24.835202+0000","last_fullsized":"2023-10-16T11:19:24.835202+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:39.077769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834923+0000","last_change":"2023-10-16T11:19:23.334130+0000","last_active":"2023-10-16T11:19:24.834923+0000","last_peered":"2023-10-16T11:19:24.834923+0000","last_clean":"2023-10-16T11:19:24.834923+0000","last_became_active":"2023-10-16T11:19:23.333073+0000","last_became_peered":"2023-10-16T11:19:23.333073+0000","last_unstale":"2023-10-16T11:19:24.834923+0000","last_undegraded":"2023-10-16T11:19:24.834923+0000","last_fullsized":"2023-10-16T11:19:24.834923+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:38.491191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834767+0000","last_change":"2023-10-16T11:19:22.330754+0000","last_active":"2023-10-16T11:19:24.834767+0000","last_peered":"2023-10-16T11:19:24.834767+0000","last_clean":"2023-10-16T11:19:24.834767+0000","last_became_active":"2023-10-16T11:19:22.330490+0000","last_became_peered":"2023-10-16T11:19:22.330490+0000","last_unstale":"2023-10-16T11:19:24.834767+0000","last_undegraded":"2023-10-16T11:19:24.834767+0000","last_fullsized":"2023-10-16T11:19:24.834767+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:17.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319916+0000","last_change":"2023-10-16T11:19:23.330327+0000","last_active":"2023-10-16T11:19:24.319916+0000","last_peered":"2023-10-16T11:19:24.319916+0000","last_clean":"2023-10-16T11:19:24.319916+0000","last_became_active":"2023-10-16T11:19:23.330040+0000","last_became_peered":"2023-10-16T11:19:23.330040+0000","last_unstale":"2023-10-16T11:19:24.319916+0000","last_undegraded":"2023-10-16T11:19:24.319916+0000","last_fullsized":"2023-10-16T11:19:24.319916+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318527+0000","last_change":"2023-10-16T11:19:22.332253+0000","last_active":"2023-10-16T11:19:24.318527+0000","last_peered":"2023-10-16T11:19:24.318527+0000","last_clean":"2023-10-16T11:19:24.318527+0000","last_became_active":"2023-10-16T11:19:22.331970+0000","last_became_peered":"2023-10-16T11:19:22.331970+0000","last_unstale":"2023-10-16T11:19:24.318527+0000","last_undegraded":"2023-10-16T11:19:24.318527+0000","last_fullsized":"2023-10-16T11:19:24.318527+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:19.740978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834824+0000","last_change":"2023-10-16T11:19:23.331144+0000","last_active":"2023-10-16T11:19:24.834824+0000","last_peered":"2023-10-16T11:19:24.834824+0000","last_clean":"2023-10-16T11:19:24.834824+0000","last_became_active":"2023-10-16T11:19:23.330824+0000","last_became_peered":"2023-10-16T11:19:23.330824+0000","last_unstale":"2023-10-16T11:19:24.834824+0000","last_undegraded":"2023-10-16T11:19:24.834824+0000","last_fullsized":"2023-10-16T11:19:24.834824+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:44:27.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836013+0000","last_change":"2023-10-16T11:19:22.325211+0000","last_active":"2023-10-16T11:19:24.836013+0000","last_peered":"2023-10-16T11:19:24.836013+0000","last_clean":"2023-10-16T11:19:24.836013+0000","last_became_active":"2023-10-16T11:19:22.324901+0000","last_became_peered":"2023-10-16T11:19:22.324901+0000","last_unstale":"2023-10-16T11:19:24.836013+0000","last_undegraded":"2023-10-16T11:19:24.836013+0000","last_fullsized":"2023-10-16T11:19:24.836013+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:46.432434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834454+0000","last_change":"2023-10-16T11:19:23.338568+0000","last_active":"2023-10-16T11:19:24.834454+0000","last_peered":"2023-10-16T11:19:24.834454+0000","last_clean":"2023-10-16T11:19:24.834454+0000","last_became_active":"2023-10-16T11:19:23.338280+0000","last_became_peered":"2023-10-16T11:19:23.338280+0000","last_unstale":"2023-10-16T11:19:24.834454+0000","last_undegraded":"2023-10-16T11:19:24.834454+0000","last_fullsized":"2023-10-16T11:19:24.834454+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:40.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"216'3","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834606+0000","last_change":"2023-10-16T11:19:22.331150+0000","last_active":"2023-10-16T11:19:24.834606+0000","last_peered":"2023-10-16T11:19:24.834606+0000","last_clean":"2023-10-16T11:19:24.834606+0000","last_became_active":"2023-10-16T11:19:22.330820+0000","last_became_peered":"2023-10-16T11:19:22.330820+0000","last_unstale":"2023-10-16T11:19:24.834606+0000","last_undegraded":"2023-10-16T11:19:24.834606+0000","last_fullsized":"2023-10-16T11:19:24.834606+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318394+0000","last_change":"2023-10-16T11:19:23.336196+0000","last_active":"2023-10-16T11:19:24.318394+0000","last_peered":"2023-10-16T11:19:24.318394+0000","last_clean":"2023-10-16T11:19:24.318394+0000","last_became_active":"2023-10-16T11:19:23.335924+0000","last_became_peered":"2023-10-16T11:19:23.335924+0000","last_unstale":"2023-10-16T11:19:24.318394+0000","last_undegraded":"2023-10-16T11:19:24.318394+0000","last_fullsized":"2023-10-16T11:19:24.318394+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:27:01.350215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"216'6","reported_seq":18,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836624+0000","last_change":"2023-10-16T11:19:22.329082+0000","last_active":"2023-10-16T11:19:24.836624+0000","last_peered":"2023-10-16T11:19:24.836624+0000","last_clean":"2023-10-16T11:19:24.836624+0000","last_became_active":"2023-10-16T11:19:22.328756+0000","last_became_peered":"2023-10-16T11:19:22.328756+0000","last_unstale":"2023-10-16T11:19:24.836624+0000","last_undegraded":"2023-10-16T11:19:24.836624+0000","last_fullsized":"2023-10-16T11:19:24.836624+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.384963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319334+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:24.319334+0000","last_peered":"2023-10-16T11:19:24.319334+0000","last_clean":"2023-10-16T11:19:24.319334+0000","last_became_active":"2023-10-16T11:19:23.331725+0000","last_became_peered":"2023-10-16T11:19:23.331725+0000","last_unstale":"2023-10-16T11:19:24.319334+0000","last_undegraded":"2023-10-16T11:19:24.319334+0000","last_fullsized":"2023-10-16T11:19:24.319334+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:45.043730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319327+0000","last_change":"2023-10-16T11:19:22.325030+0000","last_active":"2023-10-16T11:19:24.319327+0000","last_peered":"2023-10-16T11:19:24.319327+0000","last_clean":"2023-10-16T11:19:24.319327+0000","last_became_active":"2023-10-16T11:19:22.324733+0000","last_became_peered":"2023-10-16T11:19:22.324733+0000","last_unstale":"2023-10-16T11:19:24.319327+0000","last_undegraded":"2023-10-16T11:19:24.319327+0000","last_fullsized":"2023-10-16T11:19:24.319327+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:29.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835180+0000","last_change":"2023-10-16T11:19:23.337758+0000","last_active":"2023-10-16T11:19:24.835180+0000","last_peered":"2023-10-16T11:19:24.835180+0000","last_clean":"2023-10-16T11:19:24.835180+0000","last_became_active":"2023-10-16T11:19:23.337147+0000","last_became_peered":"2023-10-16T11:19:23.337147+0000","last_unstale":"2023-10-16T11:19:24.835180+0000","last_undegraded":"2023-10-16T11:19:24.835180+0000","last_fullsized":"2023-10-16T11:19:24.835180+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:06.136692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835090+0000","last_change":"2023-10-16T11:19:22.327307+0000","last_active":"2023-10-16T11:19:24.835090+0000","last_peered":"2023-10-16T11:19:24.835090+0000","last_clean":"2023-10-16T11:19:24.835090+0000","last_became_active":"2023-10-16T11:19:22.327036+0000","last_became_peered":"2023-10-16T11:19:22.327036+0000","last_unstale":"2023-10-16T11:19:24.835090+0000","last_undegraded":"2023-10-16T11:19:24.835090+0000","last_fullsized":"2023-10-16T11:19:24.835090+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:24.276671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834767+0000","last_change":"2023-10-16T11:19:23.333500+0000","last_active":"2023-10-16T11:19:24.834767+0000","last_peered":"2023-10-16T11:19:24.834767+0000","last_clean":"2023-10-16T11:19:24.834767+0000","last_became_active":"2023-10-16T11:19:23.333215+0000","last_became_peered":"2023-10-16T11:19:23.333215+0000","last_unstale":"2023-10-16T11:19:24.834767+0000","last_undegraded":"2023-10-16T11:19:24.834767+0000","last_fullsized":"2023-10-16T11:19:24.834767+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834354+0000","last_change":"2023-10-16T11:19:22.331044+0000","last_active":"2023-10-16T11:19:24.834354+0000","last_peered":"2023-10-16T11:19:24.834354+0000","last_clean":"2023-10-16T11:19:24.834354+0000","last_became_active":"2023-10-16T11:19:22.330757+0000","last_became_peered":"2023-10-16T11:19:22.330757+0000","last_unstale":"2023-10-16T11:19:24.834354+0000","last_undegraded":"2023-10-16T11:19:24.834354+0000","last_fullsized":"2023-10-16T11:19:24.834354+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835584+0000","last_change":"2023-10-16T11:19:23.336992+0000","last_active":"2023-10-16T11:19:24.835584+0000","last_peered":"2023-10-16T11:19:24.835584+0000","last_clean":"2023-10-16T11:19:24.835584+0000","last_became_active":"2023-10-16T11:19:23.336724+0000","last_became_peered":"2023-10-16T11:19:23.336724+0000","last_unstale":"2023-10-16T11:19:24.835584+0000","last_undegraded":"2023-10-16T11:19:24.835584+0000","last_fullsized":"2023-10-16T11:19:24.835584+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:01.326738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"216'8","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.317995+0000","last_change":"2023-10-16T11:19:22.332156+0000","last_active":"2023-10-16T11:19:24.317995+0000","last_peered":"2023-10-16T11:19:24.317995+0000","last_clean":"2023-10-16T11:19:24.317995+0000","last_became_active":"2023-10-16T11:19:22.331863+0000","last_became_peered":"2023-10-16T11:19:22.331863+0000","last_unstale":"2023-10-16T11:19:24.317995+0000","last_undegraded":"2023-10-16T11:19:24.317995+0000","last_fullsized":"2023-10-16T11:19:24.317995+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:36.175530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835297+0000","last_change":"2023-10-16T11:19:23.333018+0000","last_active":"2023-10-16T11:19:24.835297+0000","last_peered":"2023-10-16T11:19:24.835297+0000","last_clean":"2023-10-16T11:19:24.835297+0000","last_became_active":"2023-10-16T11:19:23.332725+0000","last_became_peered":"2023-10-16T11:19:23.332725+0000","last_unstale":"2023-10-16T11:19:24.835297+0000","last_undegraded":"2023-10-16T11:19:24.835297+0000","last_fullsized":"2023-10-16T11:19:24.835297+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:39.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836118+0000","last_change":"2023-10-16T11:19:22.331432+0000","last_active":"2023-10-16T11:19:24.836118+0000","last_peered":"2023-10-16T11:19:24.836118+0000","last_clean":"2023-10-16T11:19:24.836118+0000","last_became_active":"2023-10-16T11:19:22.331124+0000","last_became_peered":"2023-10-16T11:19:22.331124+0000","last_unstale":"2023-10-16T11:19:24.836118+0000","last_undegraded":"2023-10-16T11:19:24.836118+0000","last_fullsized":"2023-10-16T11:19:24.836118+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836283+0000","last_change":"2023-10-16T11:19:23.338701+0000","last_active":"2023-10-16T11:19:24.836283+0000","last_peered":"2023-10-16T11:19:24.836283+0000","last_clean":"2023-10-16T11:19:24.836283+0000","last_became_active":"2023-10-16T11:19:23.334033+0000","last_became_peered":"2023-10-16T11:19:23.334033+0000","last_unstale":"2023-10-16T11:19:24.836283+0000","last_undegraded":"2023-10-16T11:19:24.836283+0000","last_fullsized":"2023-10-16T11:19:24.836283+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836041+0000","last_change":"2023-10-16T11:19:22.330569+0000","last_active":"2023-10-16T11:19:24.836041+0000","last_peered":"2023-10-16T11:19:24.836041+0000","last_clean":"2023-10-16T11:19:24.836041+0000","last_became_active":"2023-10-16T11:19:22.329740+0000","last_became_peered":"2023-10-16T11:19:22.329740+0000","last_unstale":"2023-10-16T11:19:24.836041+0000","last_undegraded":"2023-10-16T11:19:24.836041+0000","last_fullsized":"2023-10-16T11:19:24.836041+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835745+0000","last_change":"2023-10-16T11:19:23.337274+0000","last_active":"2023-10-16T11:19:24.835745+0000","last_peered":"2023-10-16T11:19:24.835745+0000","last_clean":"2023-10-16T11:19:24.835745+0000","last_became_active":"2023-10-16T11:19:23.336889+0000","last_became_peered":"2023-10-16T11:19:23.336889+0000","last_unstale":"2023-10-16T11:19:24.835745+0000","last_undegraded":"2023-10-16T11:19:24.835745+0000","last_fullsized":"2023-10-16T11:19:24.835745+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319614+0000","last_change":"2023-10-16T11:19:22.318314+0000","last_active":"2023-10-16T11:19:24.319614+0000","last_peered":"2023-10-16T11:19:24.319614+0000","last_clean":"2023-10-16T11:19:24.319614+0000","last_became_active":"2023-10-16T11:19:22.318015+0000","last_became_peered":"2023-10-16T11:19:22.318015+0000","last_unstale":"2023-10-16T11:19:24.319614+0000","last_undegraded":"2023-10-16T11:19:24.319614+0000","last_fullsized":"2023-10-16T11:19:24.319614+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.421816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835880+0000","last_change":"2023-10-16T11:19:23.331171+0000","last_active":"2023-10-16T11:19:24.835880+0000","last_peered":"2023-10-16T11:19:24.835880+0000","last_clean":"2023-10-16T11:19:24.835880+0000","last_became_active":"2023-10-16T11:19:23.330836+0000","last_became_peered":"2023-10-16T11:19:23.330836+0000","last_unstale":"2023-10-16T11:19:24.835880+0000","last_undegraded":"2023-10-16T11:19:24.835880+0000","last_fullsized":"2023-10-16T11:19:24.835880+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:45.818409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318309+0000","last_change":"2023-10-16T11:19:22.332355+0000","last_active":"2023-10-16T11:19:24.318309+0000","last_peered":"2023-10-16T11:19:24.318309+0000","last_clean":"2023-10-16T11:19:24.318309+0000","last_became_active":"2023-10-16T11:19:22.331520+0000","last_became_peered":"2023-10-16T11:19:22.331520+0000","last_unstale":"2023-10-16T11:19:24.318309+0000","last_undegraded":"2023-10-16T11:19:24.318309+0000","last_fullsized":"2023-10-16T11:19:24.318309+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:10.682623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319436+0000","last_change":"2023-10-16T11:19:23.331460+0000","last_active":"2023-10-16T11:19:24.319436+0000","last_peered":"2023-10-16T11:19:24.319436+0000","last_clean":"2023-10-16T11:19:24.319436+0000","last_became_active":"2023-10-16T11:19:23.331178+0000","last_became_peered":"2023-10-16T11:19:23.331178+0000","last_unstale":"2023-10-16T11:19:24.319436+0000","last_undegraded":"2023-10-16T11:19:24.319436+0000","last_fullsized":"2023-10-16T11:19:24.319436+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:36.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318426+0000","last_change":"2023-10-16T11:19:22.329655+0000","last_active":"2023-10-16T11:19:24.318426+0000","last_peered":"2023-10-16T11:19:24.318426+0000","last_clean":"2023-10-16T11:19:24.318426+0000","last_became_active":"2023-10-16T11:19:22.329366+0000","last_became_peered":"2023-10-16T11:19:22.329366+0000","last_unstale":"2023-10-16T11:19:24.318426+0000","last_undegraded":"2023-10-16T11:19:24.318426+0000","last_fullsized":"2023-10-16T11:19:24.318426+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:47.331154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835742+0000","last_change":"2023-10-16T11:19:23.333742+0000","last_active":"2023-10-16T11:19:24.835742+0000","last_peered":"2023-10-16T11:19:24.835742+0000","last_clean":"2023-10-16T11:19:24.835742+0000","last_became_active":"2023-10-16T11:19:23.333424+0000","last_became_peered":"2023-10-16T11:19:23.333424+0000","last_unstale":"2023-10-16T11:19:24.835742+0000","last_undegraded":"2023-10-16T11:19:24.835742+0000","last_fullsized":"2023-10-16T11:19:24.835742+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.156569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.319530+0000","last_change":"2023-10-16T11:19:22.329858+0000","last_active":"2023-10-16T11:19:24.319530+0000","last_peered":"2023-10-16T11:19:24.319530+0000","last_clean":"2023-10-16T11:19:24.319530+0000","last_became_active":"2023-10-16T11:19:22.329586+0000","last_became_peered":"2023-10-16T11:19:22.329586+0000","last_unstale":"2023-10-16T11:19:24.319530+0000","last_undegraded":"2023-10-16T11:19:24.319530+0000","last_fullsized":"2023-10-16T11:19:24.319530+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:25.906324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834796+0000","last_change":"2023-10-16T11:19:23.333035+0000","last_active":"2023-10-16T11:19:24.834796+0000","last_peered":"2023-10-16T11:19:24.834796+0000","last_clean":"2023-10-16T11:19:24.834796+0000","last_became_active":"2023-10-16T11:19:23.332692+0000","last_became_peered":"2023-10-16T11:19:23.332692+0000","last_unstale":"2023-10-16T11:19:24.834796+0000","last_undegraded":"2023-10-16T11:19:24.834796+0000","last_fullsized":"2023-10-16T11:19:24.834796+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320007+0000","last_change":"2023-10-16T11:19:22.331457+0000","last_active":"2023-10-16T11:19:24.320007+0000","last_peered":"2023-10-16T11:19:24.320007+0000","last_clean":"2023-10-16T11:19:24.320007+0000","last_became_active":"2023-10-16T11:19:22.331010+0000","last_became_peered":"2023-10-16T11:19:22.331010+0000","last_unstale":"2023-10-16T11:19:24.320007+0000","last_undegraded":"2023-10-16T11:19:24.320007+0000","last_fullsized":"2023-10-16T11:19:24.320007+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:38.048633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836566+0000","last_change":"2023-10-16T11:19:23.336920+0000","last_active":"2023-10-16T11:19:24.836566+0000","last_peered":"2023-10-16T11:19:24.836566+0000","last_clean":"2023-10-16T11:19:24.836566+0000","last_became_active":"2023-10-16T11:19:23.336473+0000","last_became_peered":"2023-10-16T11:19:23.336473+0000","last_unstale":"2023-10-16T11:19:24.836566+0000","last_undegraded":"2023-10-16T11:19:24.836566+0000","last_fullsized":"2023-10-16T11:19:24.836566+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:22.421887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835914+0000","last_change":"2023-10-16T11:19:22.326294+0000","last_active":"2023-10-16T11:19:24.835914+0000","last_peered":"2023-10-16T11:19:24.835914+0000","last_clean":"2023-10-16T11:19:24.835914+0000","last_became_active":"2023-10-16T11:19:22.326012+0000","last_became_peered":"2023-10-16T11:19:22.326012+0000","last_unstale":"2023-10-16T11:19:24.835914+0000","last_undegraded":"2023-10-16T11:19:24.835914+0000","last_fullsized":"2023-10-16T11:19:24.835914+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835497+0000","last_change":"2023-10-16T11:19:23.333192+0000","last_active":"2023-10-16T11:19:24.835497+0000","last_peered":"2023-10-16T11:19:24.835497+0000","last_clean":"2023-10-16T11:19:24.835497+0000","last_became_active":"2023-10-16T11:19:23.332739+0000","last_became_peered":"2023-10-16T11:19:23.332739+0000","last_unstale":"2023-10-16T11:19:24.835497+0000","last_undegraded":"2023-10-16T11:19:24.835497+0000","last_fullsized":"2023-10-16T11:19:24.835497+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"216'2","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835009+0000","last_change":"2023-10-16T11:19:22.332950+0000","last_active":"2023-10-16T11:19:24.835009+0000","last_peered":"2023-10-16T11:19:24.835009+0000","last_clean":"2023-10-16T11:19:24.835009+0000","last_became_active":"2023-10-16T11:19:22.332650+0000","last_became_peered":"2023-10-16T11:19:22.332650+0000","last_unstale":"2023-10-16T11:19:24.835009+0000","last_undegraded":"2023-10-16T11:19:24.835009+0000","last_fullsized":"2023-10-16T11:19:24.835009+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834946+0000","last_change":"2023-10-16T11:19:23.325482+0000","last_active":"2023-10-16T11:19:24.834946+0000","last_peered":"2023-10-16T11:19:24.834946+0000","last_clean":"2023-10-16T11:19:24.834946+0000","last_became_active":"2023-10-16T11:19:23.325193+0000","last_became_peered":"2023-10-16T11:19:23.325193+0000","last_unstale":"2023-10-16T11:19:24.834946+0000","last_undegraded":"2023-10-16T11:19:24.834946+0000","last_fullsized":"2023-10-16T11:19:24.834946+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.191635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836563+0000","last_change":"2023-10-16T11:19:22.331182+0000","last_active":"2023-10-16T11:19:24.836563+0000","last_peered":"2023-10-16T11:19:24.836563+0000","last_clean":"2023-10-16T11:19:24.836563+0000","last_became_active":"2023-10-16T11:19:22.330860+0000","last_became_peered":"2023-10-16T11:19:22.330860+0000","last_unstale":"2023-10-16T11:19:24.836563+0000","last_undegraded":"2023-10-16T11:19:24.836563+0000","last_fullsized":"2023-10-16T11:19:24.836563+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834593+0000","last_change":"2023-10-16T11:19:23.331571+0000","last_active":"2023-10-16T11:19:24.834593+0000","last_peered":"2023-10-16T11:19:24.834593+0000","last_clean":"2023-10-16T11:19:24.834593+0000","last_became_active":"2023-10-16T11:19:23.331242+0000","last_became_peered":"2023-10-16T11:19:23.331242+0000","last_unstale":"2023-10-16T11:19:24.834593+0000","last_undegraded":"2023-10-16T11:19:24.834593+0000","last_fullsized":"2023-10-16T11:19:24.834593+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.320127+0000","last_change":"2023-10-16T11:19:22.326754+0000","last_active":"2023-10-16T11:19:24.320127+0000","last_peered":"2023-10-16T11:19:24.320127+0000","last_clean":"2023-10-16T11:19:24.320127+0000","last_became_active":"2023-10-16T11:19:22.326480+0000","last_became_peered":"2023-10-16T11:19:22.326480+0000","last_unstale":"2023-10-16T11:19:24.320127+0000","last_undegraded":"2023-10-16T11:19:24.320127+0000","last_fullsized":"2023-10-16T11:19:24.320127+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835594+0000","last_change":"2023-10-16T11:19:23.336160+0000","last_active":"2023-10-16T11:19:24.835594+0000","last_peered":"2023-10-16T11:19:24.835594+0000","last_clean":"2023-10-16T11:19:24.835594+0000","last_became_active":"2023-10-16T11:19:23.333920+0000","last_became_peered":"2023-10-16T11:19:23.333920+0000","last_unstale":"2023-10-16T11:19:24.835594+0000","last_undegraded":"2023-10-16T11:19:24.835594+0000","last_fullsized":"2023-10-16T11:19:24.835594+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836151+0000","last_change":"2023-10-16T11:19:22.335923+0000","last_active":"2023-10-16T11:19:24.836151+0000","last_peered":"2023-10-16T11:19:24.836151+0000","last_clean":"2023-10-16T11:19:24.836151+0000","last_became_active":"2023-10-16T11:19:22.335705+0000","last_became_peered":"2023-10-16T11:19:22.335705+0000","last_unstale":"2023-10-16T11:19:24.836151+0000","last_undegraded":"2023-10-16T11:19:24.836151+0000","last_fullsized":"2023-10-16T11:19:24.836151+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:40.803363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835646+0000","last_change":"2023-10-16T11:19:23.329220+0000","last_active":"2023-10-16T11:19:24.835646+0000","last_peered":"2023-10-16T11:19:24.835646+0000","last_clean":"2023-10-16T11:19:24.835646+0000","last_became_active":"2023-10-16T11:19:23.328923+0000","last_became_peered":"2023-10-16T11:19:23.328923+0000","last_unstale":"2023-10-16T11:19:24.835646+0000","last_undegraded":"2023-10-16T11:19:24.835646+0000","last_fullsized":"2023-10-16T11:19:24.835646+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835972+0000","last_change":"2023-10-16T11:19:22.333752+0000","last_active":"2023-10-16T11:19:24.835972+0000","last_peered":"2023-10-16T11:19:24.835972+0000","last_clean":"2023-10-16T11:19:24.835972+0000","last_became_active":"2023-10-16T11:19:22.333497+0000","last_became_peered":"2023-10-16T11:19:22.333497+0000","last_unstale":"2023-10-16T11:19:24.835972+0000","last_undegraded":"2023-10-16T11:19:24.835972+0000","last_fullsized":"2023-10-16T11:19:24.835972+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835880+0000","last_change":"2023-10-16T11:19:23.330624+0000","last_active":"2023-10-16T11:19:24.835880+0000","last_peered":"2023-10-16T11:19:24.835880+0000","last_clean":"2023-10-16T11:19:24.835880+0000","last_became_active":"2023-10-16T11:19:23.329588+0000","last_became_peered":"2023-10-16T11:19:23.329588+0000","last_unstale":"2023-10-16T11:19:24.835880+0000","last_undegraded":"2023-10-16T11:19:24.835880+0000","last_fullsized":"2023-10-16T11:19:24.835880+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:10.835410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834650+0000","last_change":"2023-10-16T11:19:22.329623+0000","last_active":"2023-10-16T11:19:24.834650+0000","last_peered":"2023-10-16T11:19:24.834650+0000","last_clean":"2023-10-16T11:19:24.834650+0000","last_became_active":"2023-10-16T11:19:22.329344+0000","last_became_peered":"2023-10-16T11:19:22.329344+0000","last_unstale":"2023-10-16T11:19:24.834650+0000","last_undegraded":"2023-10-16T11:19:24.834650+0000","last_fullsized":"2023-10-16T11:19:24.834650+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835841+0000","last_change":"2023-10-16T11:19:23.329115+0000","last_active":"2023-10-16T11:19:24.835841+0000","last_peered":"2023-10-16T11:19:24.835841+0000","last_clean":"2023-10-16T11:19:24.835841+0000","last_became_active":"2023-10-16T11:19:23.328830+0000","last_became_peered":"2023-10-16T11:19:23.328830+0000","last_unstale":"2023-10-16T11:19:24.835841+0000","last_undegraded":"2023-10-16T11:19:24.835841+0000","last_fullsized":"2023-10-16T11:19:24.835841+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:17.247488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836376+0000","last_change":"2023-10-16T11:19:22.332640+0000","last_active":"2023-10-16T11:19:24.836376+0000","last_peered":"2023-10-16T11:19:24.836376+0000","last_clean":"2023-10-16T11:19:24.836376+0000","last_became_active":"2023-10-16T11:19:22.331608+0000","last_became_peered":"2023-10-16T11:19:22.331608+0000","last_unstale":"2023-10-16T11:19:24.836376+0000","last_undegraded":"2023-10-16T11:19:24.836376+0000","last_fullsized":"2023-10-16T11:19:24.836376+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835155+0000","last_change":"2023-10-16T11:19:23.324999+0000","last_active":"2023-10-16T11:19:24.835155+0000","last_peered":"2023-10-16T11:19:24.835155+0000","last_clean":"2023-10-16T11:19:24.835155+0000","last_became_active":"2023-10-16T11:19:23.324686+0000","last_became_peered":"2023-10-16T11:19:23.324686+0000","last_unstale":"2023-10-16T11:19:24.835155+0000","last_undegraded":"2023-10-16T11:19:24.835155+0000","last_fullsized":"2023-10-16T11:19:24.835155+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:42.281964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.318104+0000","last_change":"2023-10-16T11:19:22.334152+0000","last_active":"2023-10-16T11:19:24.318104+0000","last_peered":"2023-10-16T11:19:24.318104+0000","last_clean":"2023-10-16T11:19:24.318104+0000","last_became_active":"2023-10-16T11:19:22.333859+0000","last_became_peered":"2023-10-16T11:19:22.333859+0000","last_unstale":"2023-10-16T11:19:24.318104+0000","last_undegraded":"2023-10-16T11:19:24.318104+0000","last_fullsized":"2023-10-16T11:19:24.318104+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:50.619981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.836232+0000","last_change":"2023-10-16T11:19:23.334041+0000","last_active":"2023-10-16T11:19:24.836232+0000","last_peered":"2023-10-16T11:19:24.836232+0000","last_clean":"2023-10-16T11:19:24.836232+0000","last_became_active":"2023-10-16T11:19:23.333752+0000","last_became_peered":"2023-10-16T11:19:23.333752+0000","last_unstale":"2023-10-16T11:19:24.836232+0000","last_undegraded":"2023-10-16T11:19:24.836232+0000","last_fullsized":"2023-10-16T11:19:24.836232+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"216'1","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.834559+0000","last_change":"2023-10-16T11:19:22.332936+0000","last_active":"2023-10-16T11:19:24.834559+0000","last_peered":"2023-10-16T11:19:24.834559+0000","last_clean":"2023-10-16T11:19:24.834559+0000","last_became_active":"2023-10-16T11:19:22.332598+0000","last_became_peered":"2023-10-16T11:19:22.332598+0000","last_unstale":"2023-10-16T11:19:24.834559+0000","last_undegraded":"2023-10-16T11:19:24.834559+0000","last_fullsized":"2023-10-16T11:19:24.834559+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:34.597859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"207'348","reported_seq":1208,"reported_epoch":216,"state":"active+clean","last_fresh":"2023-10-16T11:19:23.318091+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:19:23.318091+0000","last_peered":"2023-10-16T11:19:23.318091+0000","last_clean":"2023-10-16T11:19:23.318091+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:19:23.318091+0000","last_undegraded":"2023-10-16T11:19:23.318091+0000","last_fullsized":"2023-10-16T11:19:23.318091+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575989,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52476,"kb_used_data":3248,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319364,"statfs":{"total":96636764160,"available":96583028736,"internally_reserved":0,"allocated":3325952,"data_stored":2044330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.75700000000000001,"5min":0.84299999999999997,"15min":0.98499999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71499999999999997,"5min":0.79500000000000004,"15min":0.999},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.749,"5min":0.83999999999999997,"15min":1.0640000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63200000000000001,"15min":0.65100000000000002},"min":{"1min":0.40100000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.83099999999999996,"15min":1.0349999999999999},"last":0.621}]},{"osd":2,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":1.117},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.90400000000000003,"15min":1.05},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.442,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69799999999999995,"5min":0.83999999999999997,"15min":1.022},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.67200000000000004,"15min":0.68600000000000005},"min":{"1min":0.56399999999999995,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.083},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":0.77700000000000002,"5min":0.81599999999999995,"15min":0.94799999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"max":{"1min":0.78000000000000003,"5min":0.83299999999999996,"15min":0.96299999999999997},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.91200000000000003},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.80500000000000005,"5min":0.85299999999999998,"15min":0.92000000000000004},"last":0.73499999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.88500000000000001,"15min":0.92600000000000005},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59599999999999997,"15min":0.61099999999999999},"min":{"1min":0.43099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77600000000000002,"5min":0.88700000000000001,"15min":0.91300000000000003},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38020,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333820,"statfs":{"total":96636764160,"available":96597831680,"internally_reserved":0,"allocated":2744320,"data_stored":1461574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.40400000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.80600000000000005,"5min":0.81000000000000005,"15min":0.91000000000000003},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.73799999999999999,"5min":0.82599999999999996,"15min":0.88500000000000001},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61399999999999999},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59499999999999997,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.95499999999999996},"last":0.67900000000000005}]},{"osd":2,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.77400000000000002,"5min":0.83099999999999996,"15min":0.95599999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.621,"15min":0.63},"min":{"1min":0.435,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82899999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.69599999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61399999999999999,"15min":0.621},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":0.81699999999999995,"5min":0.84099999999999997,"15min":0.92700000000000005},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61599999999999999,"15min":0.629},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":0.97799999999999998},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34499999999999997,"5min":0.33900000000000002,"15min":0.30299999999999999},"max":{"1min":0.72899999999999998,"5min":0.87,"15min":0.87},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.75700000000000001,"5min":0.85499999999999998,"15min":0.89900000000000002},"last":0.443},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83299999999999996,"5min":0.92500000000000004,"15min":1.1599999999999999},"last":0.5}]},{"osd":7,"last update":"Mon Oct 16 11:18:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.42399999999999999,"15min":0.38300000000000001},"max":{"1min":0.80000000000000004,"5min":0.90700000000000003,"15min":0.93300000000000005},"last":0.378}]}]},{"osd":1,"up_from":13,"seq":55834575990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47720,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324120,"statfs":{"total":96636764160,"available":96587898880,"internally_reserved":0,"allocated":2715648,"data_stored":1452280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40899999999999997,"15min":0.41399999999999998},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.622,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.32300000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63900000000000001,"5min":0.64100000000000001,"15min":0.71699999999999997},"last":0.35799999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.46400000000000002,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.62,"5min":0.66600000000000004,"15min":0.70099999999999996},"last":0.308}]},{"osd":3,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54100000000000004,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.441,"15min":0.443},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53200000000000003,"5min":0.623,"15min":0.70499999999999996},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82099999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":1.0309999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88900000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90100000000000002,"5min":1.01,"15min":1.0760000000000001},"last":0.499}]},{"osd":6,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81999999999999995,"5min":0.93100000000000005,"15min":1.0149999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.68799999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.70299999999999996,"15min":0.70399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83399999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.81599999999999995,"5min":0.99399999999999999,"15min":1.016},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575990,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42740,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329100,"statfs":{"total":96636764160,"available":96592998400,"internally_reserved":0,"allocated":3313664,"data_stored":2042738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44400000000000001,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60699999999999998,"5min":1.272,"15min":1.272},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67700000000000005,"5min":1.351,"15min":1.351},"last":0.439}]},{"osd":4,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.46800000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.875,"5min":1.391,"15min":1.391},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80000000000000004,"5min":0.874,"15min":1.089},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.95599999999999996,"15min":1.216},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.69799999999999995},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90900000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.47199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89100000000000001,"5min":1.038,"15min":1.1180000000000001},"last":0.71499999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.621},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.72799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47744,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324096,"statfs":{"total":96636764160,"available":96587874304,"internally_reserved":0,"allocated":2732032,"data_stored":1459470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.63300000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.432}]},{"osd":3,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69799999999999995,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77800000000000002},"last":0.61099999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.95399999999999996,"15min":1.097},"last":0.626},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67200000000000004,"15min":0.68200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.91800000000000004,"5min":0.96199999999999997,"15min":1.016},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90500000000000003,"5min":1.022,"15min":1.0389999999999999},"last":0.74399999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91800000000000004,"5min":1.022,"15min":1.022},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91900000000000004,"5min":0.98999999999999999,"15min":1.004},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90400000000000003,"5min":1.0569999999999999,"15min":1.071},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.75600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575989,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51912,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319928,"statfs":{"total":96636764160,"available":96583606272,"internally_reserved":0,"allocated":2740224,"data_stored":1459506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38100000000000001,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.68700000000000006},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":1.0029999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.60899999999999999,"15min":0.67300000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.74299999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.58999999999999997,"5min":0.66800000000000004,"15min":0.96299999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64200000000000002,"5min":0.69899999999999995,"15min":1.024},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90600000000000003,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"last":0.749}]},{"osd":5,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.99099999999999999,"15min":1.026},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0640000000000001},"last":0.76300000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41799999999999998},"max":{"1min":0.85999999999999999,"5min":0.92900000000000005,"15min":1.0369999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.0620000000000001},"last":0.79000000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":1.002},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88400000000000001,"5min":1.006,"15min":1.1899999999999999},"last":0.77800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575989,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47736,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324104,"statfs":{"total":96636764160,"available":96587882496,"internally_reserved":0,"allocated":2732032,"data_stored":1459962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.625},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.623},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71599999999999997,"5min":0.82499999999999996,"15min":0.90700000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58799999999999997,"15min":0.623},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60599999999999998,"15min":0.63900000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.84399999999999997,"15min":0.88400000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.624},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.83799999999999997,"15min":0.995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.51500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.90500000000000003,"15min":1.115},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.35499999999999998,"15min":0.34200000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.83899999999999997},"last":0.622},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.64900000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575989,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64372,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307468,"statfs":{"total":96636764160,"available":96570847232,"internally_reserved":0,"allocated":2723840,"data_stored":1452336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.88600000000000001},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60299999999999998},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.86499999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.91400000000000003},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40899999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85899999999999999,"5min":0.878,"15min":0.878},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55300000000000005,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.68500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:19:28.111 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:19:28.111 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T11:19:28.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.271+0000 7f9156e48700 1 Processor -- start 2023-10-16T11:19:28.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.271+0000 7f9156e48700 1 -- start start 2023-10-16T11:19:28.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f9156e48700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f9150161e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f9156e48700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91500b3b60 0x7f91500724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f9156e48700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150071d70 con 0x7f9150072a20 2023-10-16T11:19:28.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f9156e48700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91500af9d0 con 0x7f91500b3b60 2023-10-16T11:19:28.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f9156e48700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91500b3970 con 0x7f91500af5f0 2023-10-16T11:19:28.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f914ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91500b3b60 0x7f91500724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f914ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91500b3b60 0x7f91500724e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38086/0 (socket says 172.21.15.158:38086) 2023-10-16T11:19:28.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.272+0000 7f914ffff700 1 -- 172.21.15.158:0/2827893350 learned_addr learned my addr 172.21.15.158:0/2827893350 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:28.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f9154be4700 1 --2- 172.21.15.158:0/2827893350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f9150161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914ffff700 1 -- 172.21.15.158:0/2827893350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 msgr2=0x7f9150161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:28.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914ffff700 1 --2- 172.21.15.158:0/2827893350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f9150161e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914ffff700 1 -- 172.21.15.158:0/2827893350 >> v1:172.21.15.172:6789/0 conn(0x7f9150072a20 legacy=0x7f915006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914ffff700 1 -- 172.21.15.158:0/2827893350 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150162370 con 0x7f91500b3b60 2023-10-16T11:19:28.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914ffff700 1 --2- 172.21.15.158:0/2827893350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500724e0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f913c0098f0 tx=0x7f913c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7407f64d9d8ffe4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:28.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914f7fe700 1 -- 172.21.15.158:0/2827893350 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c00b8a0 con 0x7f91500b3b60 2023-10-16T11:19:28.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914f7fe700 1 -- 172.21.15.158:0/2827893350 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913c00ba00 con 0x7f91500b3b60 2023-10-16T11:19:28.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f914f7fe700 1 -- 172.21.15.158:0/2827893350 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c00bd90 con 0x7f91500b3b60 2023-10-16T11:19:28.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f9156e48700 1 -- 172.21.15.158:0/2827893350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 msgr2=0x7f91500724e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.273+0000 7f9156e48700 1 --2- 172.21.15.158:0/2827893350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500724e0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f913c0098f0 tx=0x7f913c009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:28.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.274+0000 7f9156e48700 1 -- 172.21.15.158:0/2827893350 shutdown_connections 2023-10-16T11:19:28.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.274+0000 7f9156e48700 1 --2- 172.21.15.158:0/2827893350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f9150161e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.274+0000 7f9156e48700 1 --2- 172.21.15.158:0/2827893350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500724e0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.274+0000 7f9156e48700 1 -- 172.21.15.158:0/2827893350 >> 172.21.15.158:0/2827893350 conn(0x7f91500ccd30 msgr2=0x7f91500cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:28.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.274+0000 7f9156e48700 1 -- 172.21.15.158:0/2827893350 shutdown_connections 2023-10-16T11:19:28.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.274+0000 7f9156e48700 1 -- 172.21.15.158:0/2827893350 wait complete. 2023-10-16T11:19:28.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.275+0000 7f9156e48700 1 Processor -- start 2023-10-16T11:19:28.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.275+0000 7f9156e48700 1 -- start start 2023-10-16T11:19:28.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9150072a20 0x7f91500b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f91500c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91501f0be0 con 0x7f91500b3b60 2023-10-16T11:19:28.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915015d300 con 0x7f9150072a20 2023-10-16T11:19:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915015d5e0 con 0x7f91500af5f0 2023-10-16T11:19:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38096/0 (socket says 172.21.15.158:38096) 2023-10-16T11:19:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 -- 172.21.15.158:0/1177429474 learned_addr learned my addr 172.21.15.158:0/1177429474 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f914ffff700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f91500c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9154be4700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9150072a20 0x7f91500b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 -- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 msgr2=0x7f91500c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f91500c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 -- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9150072a20 msgr2=0x7f91500b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9150072a20 0x7f91500b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 -- 172.21.15.158:0/1177429474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c009550 con 0x7f91500b3b60 2023-10-16T11:19:28.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f91553e5700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500b2320 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f914800cc50 tx=0x7f914800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:28.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f914d7fa700 1 -- 172.21.15.158:0/1177429474 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148003c00 con 0x7f91500b3b60 2023-10-16T11:19:28.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f914d7fa700 1 -- 172.21.15.158:0/1177429474 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9148003d60 con 0x7f91500b3b60 2023-10-16T11:19:28.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915015d8c0 con 0x7f91500b3b60 2023-10-16T11:19:28.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f914d7fa700 1 -- 172.21.15.158:0/1177429474 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914800aab0 con 0x7f91500b3b60 2023-10-16T11:19:28.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.276+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91501f4010 con 0x7f91500b3b60 2023-10-16T11:19:28.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.277+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91500681b0 con 0x7f91500b3b60 2023-10-16T11:19:28.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.278+0000 7f914d7fa700 1 -- 172.21.15.158:0/1177429474 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f914800ac10 con 0x7f91500b3b60 2023-10-16T11:19:28.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.278+0000 7f914d7fa700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f913406b020 0x7f913406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.278+0000 7f914d7fa700 1 -- 172.21.15.158:0/1177429474 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f91480899c0 con 0x7f91500b3b60 2023-10-16T11:19:28.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.278+0000 7f9154be4700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f913406b020 0x7f913406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.279+0000 7f9154be4700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f913406b020 0x7f913406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9144006b20 tx=0x7f9144008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:28.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.283+0000 7f914d7fa700 1 -- 172.21.15.158:0/1177429474 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91480546d0 con 0x7f91500b3b60 2023-10-16T11:19:28.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.553+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f91500cd960 con 0x7f91500b3b60 2023-10-16T11:19:28.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.556+0000 7f914d7fa700 1 -- 172.21.15.158:0/1177429474 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9148057cf0 con 0x7f91500b3b60 2023-10-16T11:19:28.557 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T11:19:28.558 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T11:19:28.558 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T11:19:28.558 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T11:19:28.558 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T11:19:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f913406b020 msgr2=0x7f913406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f913406b020 0x7f913406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9144006b20 tx=0x7f9144008040 comp rx=0 tx=0).stop 2023-10-16T11:19:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 msgr2=0x7f91500b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500b2320 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f914800cc50 tx=0x7f914800cf60 comp rx=0 tx=0).stop 2023-10-16T11:19:28.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 shutdown_connections 2023-10-16T11:19:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f913406b020 0x7f913406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91500b3b60 0x7f91500b2320 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9150072a20 0x7f91500b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 --2- 172.21.15.158:0/1177429474 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91500af5f0 0x7f91500c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 >> 172.21.15.158:0/1177429474 conn(0x7f91500ccd30 msgr2=0x7f915015f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:28.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 shutdown_connections 2023-10-16T11:19:28.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.558+0000 7f9156e48700 1 -- 172.21.15.158:0/1177429474 wait complete. 2023-10-16T11:19:28.562 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T11:19:28.572 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:28.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.692+0000 7f5f6952c700 1 Processor -- start 2023-10-16T11:19:28.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.692+0000 7f5f6952c700 1 -- start start 2023-10-16T11:19:28.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f6952c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5f6415d7c0 0x7f5f640d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f6952c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 0x7f5f640ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f6952c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f640d3bd0 con 0x7f5f640ce950 2023-10-16T11:19:28.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f6952c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f640d4060 con 0x7f5f6415d7c0 2023-10-16T11:19:28.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f6952c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640cede0 con 0x7f5f640d1210 2023-10-16T11:19:28.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f637fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5f640ce950 0x7f5f640b50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38280/0 (socket says 172.21.15.158:38280) 2023-10-16T11:19:28.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f637fe700 1 -- 172.21.15.158:0/1869911848 learned_addr learned my addr 172.21.15.158:0/1869911848 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:28.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/1869911848 >> v2:172.21.15.172:3300/0 conn(0x7f5f6415d7c0 0x7f5f640d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f627fc700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 0x7f5f640ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450911259 0 0) 0x7f5f640d3bd0 con 0x7f5f640ce950 2023-10-16T11:19:28.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c0035b0 con 0x7f5f640ce950 2023-10-16T11:19:28.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052895935 0 0) 0x7f5f4c0035b0 con 0x7f5f640ce950 2023-10-16T11:19:28.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 msgr2=0x7f5f640ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:28.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.693+0000 7f5f61ffb700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 0x7f5f640ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 >> v2:172.21.15.172:3300/0 conn(0x7f5f6415d7c0 msgr2=0x7f5f640d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:28.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 --2- 172.21.15.158:0/1869911848 >> v2:172.21.15.172:3300/0 conn(0x7f5f6415d7c0 0x7f5f640d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640d2cf0 con 0x7f5f640ce950 2023-10-16T11:19:28.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f5f5c001a40 con 0x7f5f640ce950 2023-10-16T11:19:28.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/1869911848 >> v2:172.21.15.172:3300/0 conn(0x7f5f6415d7c0 0x7f5f640d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:28.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 >> v1:172.21.15.158:6789/0 conn(0x7f5f640ce950 legacy=0x7f5f640b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 0x7f5f4c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c004d90 0x7f5f4c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640d4060 con 0x7f5f4c003a00 2023-10-16T11:19:28.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640cede0 con 0x7f5f4c004d90 2023-10-16T11:19:28.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c00ba10 con 0x7f5f640d1210 2023-10-16T11:19:28.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f637fe700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c004d90 0x7f5f4c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f627fc700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 0x7f5f4c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 -- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 msgr2=0x7f5f4c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:28.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640d1210 0x7f5f4c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 -- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c004d90 msgr2=0x7f5f4c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:28.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f4c004d90 0x7f5f4c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 -- 172.21.15.158:0/1869911848 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0035b0 con 0x7f5f4c003a00 2023-10-16T11:19:28.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009d30 tx=0x7f5f4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de6b9ad35b6db92 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:28.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f58009e80 con 0x7f5f4c003a00 2023-10-16T11:19:28.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5800e9a0 con 0x7f5f4c003a00 2023-10-16T11:19:28.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.694+0000 7f5f61ffb700 1 -- 172.21.15.158:0/1869911848 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5800ed30 con 0x7f5f4c003a00 2023-10-16T11:19:28.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.695+0000 7f5f6952c700 1 -- 172.21.15.158:0/1869911848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 msgr2=0x7f5f4c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.695+0000 7f5f6952c700 1 --2- 172.21.15.158:0/1869911848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f4c003e20 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009d30 tx=0x7f5f4c005880 comp rx=0 tx=0).stop 2023-10-16T11:19:28.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.695+0000 7f5f637fe700 1 -- 172.21.15.158:0/1869911848 reap_dead start 2023-10-16T11:19:28.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.695+0000 7f5f6952c700 1 -- 172.21.15.158:0/1869911848 shutdown_connections 2023-10-16T11:19:28.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.695+0000 7f5f6952c700 1 -- 172.21.15.158:0/1869911848 >> 172.21.15.158:0/1869911848 conn(0x7f5f640669f0 msgr2=0x7f5f64066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:28.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.695+0000 7f5f6952c700 1 -- 172.21.15.158:0/1869911848 shutdown_connections 2023-10-16T11:19:28.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.695+0000 7f5f6952c700 1 -- 172.21.15.158:0/1869911848 wait complete. 2023-10-16T11:19:28.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.697+0000 7f5f6952c700 1 Processor -- start 2023-10-16T11:19:28.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.697+0000 7f5f6952c700 1 -- start start 2023-10-16T11:19:28.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f6952c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f64152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f6952c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640c9840 0x7f5f6415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f6952c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f640c9cc0 0x7f5f6415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f6952c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640c93b0 con 0x7f5f4c003a00 2023-10-16T11:19:28.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f6952c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640c87a0 con 0x7f5f640c9cc0 2023-10-16T11:19:28.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f6952c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640c8a80 con 0x7f5f640c9840 2023-10-16T11:19:28.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f64152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f64152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38112/0 (socket says 172.21.15.158:38112) 2023-10-16T11:19:28.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 -- 172.21.15.158:0/204253065 learned_addr learned my addr 172.21.15.158:0/204253065 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:28.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f637fe700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f640c9cc0 0x7f5f6415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f627fc700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640c9840 0x7f5f6415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 -- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640c9840 msgr2=0x7f5f6415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640c9840 0x7f5f6415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 -- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f640c9cc0 msgr2=0x7f5f6415bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f640c9cc0 0x7f5f6415bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 -- 172.21.15.158:0/204253065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640c8d60 con 0x7f5f4c003a00 2023-10-16T11:19:28.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f627fc700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640c9840 0x7f5f6415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:28.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f62ffd700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f64152450 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f5f58000c00 tx=0x7f5f580047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:28.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f4b7fe700 1 -- 172.21.15.158:0/204253065 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f58021070 con 0x7f5f4c003a00 2023-10-16T11:19:28.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f4b7fe700 1 -- 172.21.15.158:0/204253065 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f580036a0 con 0x7f5f4c003a00 2023-10-16T11:19:28.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f640c9040 con 0x7f5f4c003a00 2023-10-16T11:19:28.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.698+0000 7f5f4b7fe700 1 -- 172.21.15.158:0/204253065 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f58006e60 con 0x7f5f4c003a00 2023-10-16T11:19:28.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.699+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6415dba0 con 0x7f5f4c003a00 2023-10-16T11:19:28.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.699+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f640c9690 con 0x7f5f4c003a00 2023-10-16T11:19:28.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.700+0000 7f5f4b7fe700 1 -- 172.21.15.158:0/204253065 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f58025070 con 0x7f5f4c003a00 2023-10-16T11:19:28.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.700+0000 7f5f4b7fe700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f5f5006b020 0x7f5f5006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:28.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.700+0000 7f5f4b7fe700 1 -- 172.21.15.158:0/204253065 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f5f5808ae50 con 0x7f5f4c003a00 2023-10-16T11:19:28.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.700+0000 7f5f627fc700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f5f5006b020 0x7f5f5006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:28.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.701+0000 7f5f627fc700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f5f5006b020 0x7f5f5006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5f6415c380 tx=0x7f5f54008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:28.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.705+0000 7f5f4b7fe700 1 -- 172.21.15.158:0/204253065 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f58055ac0 con 0x7f5f4c003a00 2023-10-16T11:19:28.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.940+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f640681b0 con 0x7f5f4c003a00 2023-10-16T11:19:28.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.940+0000 7f5f4b7fe700 1 -- 172.21.15.158:0/204253065 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4938 (secure 0 0 0) 0x7f5f5802c090 con 0x7f5f4c003a00 2023-10-16T11:19:28.943 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:28.943 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":11586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T11:19:28.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f5f5006b020 msgr2=0x7f5f5006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f5f5006b020 0x7f5f5006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5f6415c380 tx=0x7f5f54008040 comp rx=0 tx=0).stop 2023-10-16T11:19:28.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 msgr2=0x7f5f64152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:28.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f64152450 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f5f58000c00 tx=0x7f5f580047c0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 shutdown_connections 2023-10-16T11:19:28.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f640c9cc0 0x7f5f6415bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f5f5006b020 0x7f5f5006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f4c003a00 0x7f5f64152450 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 --2- 172.21.15.158:0/204253065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f640c9840 0x7f5f6415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:28.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 >> 172.21.15.158:0/204253065 conn(0x7f5f640669f0 msgr2=0x7f5f6415a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:28.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 shutdown_connections 2023-10-16T11:19:28.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:28.943+0000 7f5f6952c700 1 -- 172.21.15.158:0/204253065 wait complete. 2023-10-16T11:19:28.947 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 112 2023-10-16T11:19:29.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:19:22.724992+0000', 'modified': '2023-10-16T11:19:25.742039+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 38819}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_38819': {'gid': 38819, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6834/972449219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6832', 'nonce': 972449219}, {'type': 'v1', 'addr': '172.21.15.172:6834', 'nonce': 972449219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:19:29.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:19:29.360 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T11:19:29.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.479+0000 7fe1b84b5700 1 Processor -- start 2023-10-16T11:19:29.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.479+0000 7fe1b84b5700 1 -- start start 2023-10-16T11:19:29.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.480+0000 7fe1b84b5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1b00cd360 0x7fe1b006ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.480+0000 7fe1b84b5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b015ec70 con 0x7fe1b015a6d0 2023-10-16T11:19:29.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.480+0000 7fe1b84b5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b015f1b0 con 0x7fe1b00cfe30 2023-10-16T11:19:29.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.480+0000 7fe1b84b5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b00afcd0 con 0x7fe1b00cd360 2023-10-16T11:19:29.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.480+0000 7fe1b6251700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe1b015a6d0 0x7fe1b00ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38296/0 (socket says 172.21.15.158:38296) 2023-10-16T11:19:29.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.480+0000 7fe1b6251700 1 -- 172.21.15.158:0/2886272950 learned_addr learned my addr 172.21.15.158:0/2886272950 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:29.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322475296 0 0) 0x7fe1b015ec70 con 0x7fe1b015a6d0 2023-10-16T11:19:29.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe19c0035b0 con 0x7fe1b015a6d0 2023-10-16T11:19:29.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/2886272950 >> v2:172.21.15.172:3300/0 conn(0x7fe1b00cd360 0x7fe1b006ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 355492642 0 0) 0x7fe19c0035b0 con 0x7fe1b015a6d0 2023-10-16T11:19:29.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 >> v2:172.21.15.172:3300/0 conn(0x7fe1b00cd360 msgr2=0x7fe1b006ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 --2- 172.21.15.158:0/2886272950 >> v2:172.21.15.172:3300/0 conn(0x7fe1b00cd360 0x7fe1b006ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 >> v1:172.21.15.172:6789/0 conn(0x7fe1b00cfe30 legacy=0x7fe1b00ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b00cab50 con 0x7fe1b015a6d0 2023-10-16T11:19:29.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe1a0001670 con 0x7fe1b015a6d0 2023-10-16T11:19:29.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 >> v1:172.21.15.158:6789/0 conn(0x7fe1b015a6d0 legacy=0x7fe1b00ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c003a00 0x7fe19c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe19c004850 0x7fe19c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe19c005210 0x7fe19c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00afcd0 con 0x7fe19c004850 2023-10-16T11:19:29.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c00c0b0 con 0x7fe19c003a00 2023-10-16T11:19:29.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c00c270 con 0x7fe19c005210 2023-10-16T11:19:29.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe19c004850 0x7fe19c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b6251700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c003a00 0x7fe19c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.481+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe19c005210 0x7fe19c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b5a50700 1 -- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe19c005210 msgr2=0x7fe19c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe19c005210 0x7fe19c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b5a50700 1 -- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c003a00 msgr2=0x7fe19c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c003a00 0x7fe19c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b6a52700 1 -- 172.21.15.158:0/2886272950 reap_dead start 2023-10-16T11:19:29.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b5a50700 1 -- 172.21.15.158:0/2886272950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0035b0 con 0x7fe19c004850 2023-10-16T11:19:29.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe19c004850 0x7fe19c004cd0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00d0270 tx=0x7fe19c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79ecb5495f67196 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0052d0 con 0x7fe19c004850 2023-10-16T11:19:29.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac005430 con 0x7fe19c004850 2023-10-16T11:19:29.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b524f700 1 -- 172.21.15.158:0/2886272950 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0038c0 con 0x7fe19c004850 2023-10-16T11:19:29.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b84b5700 1 -- 172.21.15.158:0/2886272950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe19c004850 msgr2=0x7fe19c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe19c004850 0x7fe19c004cd0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00d0270 tx=0x7fe19c005880 comp rx=0 tx=0).stop 2023-10-16T11:19:29.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b84b5700 1 -- 172.21.15.158:0/2886272950 shutdown_connections 2023-10-16T11:19:29.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/2886272950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe19c004850 0x7fe19c004cd0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b84b5700 1 -- 172.21.15.158:0/2886272950 >> 172.21.15.158:0/2886272950 conn(0x7fe1b006b9e0 msgr2=0x7fe1b006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.482+0000 7fe1b84b5700 1 -- 172.21.15.158:0/2886272950 shutdown_connections 2023-10-16T11:19:29.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.483+0000 7fe1b84b5700 1 -- 172.21.15.158:0/2886272950 wait complete. 2023-10-16T11:19:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.484+0000 7fe1b84b5700 1 Processor -- start 2023-10-16T11:19:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.484+0000 7fe1b84b5700 1 -- start start 2023-10-16T11:19:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b84b5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c004850 0x7fe1b006ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b84b5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1b014e4c0 0x7fe1b014e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b84b5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1b00c87c0 0x7fe1b00c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b84b5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00ca070 con 0x7fe1b00c87c0 2023-10-16T11:19:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b84b5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00b0ac0 con 0x7fe19c004850 2023-10-16T11:19:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b84b5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b014ee60 con 0x7fe1b014e4c0 2023-10-16T11:19:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1b00c87c0 0x7fe1b00c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1b00c87c0 0x7fe1b00c8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38136/0 (socket says 172.21.15.158:38136) 2023-10-16T11:19:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1704945454 learned_addr learned my addr 172.21.15.158:0/1704945454 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1b014e4c0 0x7fe1b014e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.485+0000 7fe1b6251700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c004850 0x7fe1b006ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1b014e4c0 msgr2=0x7fe1b014e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1b014e4c0 0x7fe1b014e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c004850 msgr2=0x7fe1b006ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c004850 0x7fe1b006ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1704945454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b00c9180 con 0x7fe1b00c87c0 2023-10-16T11:19:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b6251700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c004850 0x7fe1b006ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1b00c87c0 0x7fe1b00c8c40 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00cfa50 tx=0x7fe1a8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1a6ffd700 1 -- 172.21.15.158:0/1704945454 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800e040 con 0x7fe1b00c87c0 2023-10-16T11:19:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1a6ffd700 1 -- 172.21.15.158:0/1704945454 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a800f040 con 0x7fe1b00c87c0 2023-10-16T11:19:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b00c93f0 con 0x7fe1b00c87c0 2023-10-16T11:19:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1a6ffd700 1 -- 172.21.15.158:0/1704945454 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800fc70 con 0x7fe1b00c87c0 2023-10-16T11:19:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.486+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b00c99b0 con 0x7fe1b00c87c0 2023-10-16T11:19:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.487+0000 7fe1a6ffd700 1 -- 172.21.15.158:0/1704945454 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe1a8015430 con 0x7fe1b00c87c0 2023-10-16T11:19:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.488+0000 7fe1a6ffd700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe19406b020 0x7fe19406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.488+0000 7fe1a6ffd700 1 -- 172.21.15.158:0/1704945454 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe1a808a390 con 0x7fe1b00c87c0 2023-10-16T11:19:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.488+0000 7fe1b6251700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe19406b020 0x7fe19406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.488+0000 7fe1b6251700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe19406b020 0x7fe19406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0007fd0 tx=0x7fe1a0009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.490+0000 7fe1c469fb80 1 Processor -- start 2023-10-16T11:19:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.490+0000 7fe1c469fb80 1 -- start start 2023-10-16T11:19:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddf2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddf2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635eddc66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635edd3cc20 con 0x5635eddf0630 2023-10-16T11:19:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635edd21c10 con 0x5635eddc6010 2023-10-16T11:19:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635edd1f530 con 0x5635eddeb220 2023-10-16T11:19:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddf2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddf2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38142/0 (socket says 172.21.15.158:38142) 2023-10-16T11:19:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 -- 172.21.15.158:0/233129456 learned_addr learned my addr 172.21.15.158:0/233129456 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b6251700 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddf2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635eddc66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 -- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 msgr2=0x5635eddf2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddf2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 -- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 msgr2=0x5635eddc66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635eddc66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 -- 172.21.15.158:0/233129456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635edcd83a0 con 0x5635eddf0630 2023-10-16T11:19:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddf2fb0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac000c00 tx=0x7fe1ac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f517e9d815afcde server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe193fff700 1 -- 172.21.15.158:0/233129456 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0052d0 con 0x5635eddf0630 2023-10-16T11:19:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe193fff700 1 -- 172.21.15.158:0/233129456 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac022070 con 0x5635eddf0630 2023-10-16T11:19:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.491+0000 7fe193fff700 1 -- 172.21.15.158:0/233129456 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac006aa0 con 0x5635eddf0630 2023-10-16T11:19:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 -- 172.21.15.158:0/233129456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 msgr2=0x5635eddf2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddf2fb0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac000c00 tx=0x7fe1ac011000 comp rx=0 tx=0).stop 2023-10-16T11:19:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 -- 172.21.15.158:0/233129456 shutdown_connections 2023-10-16T11:19:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635eddc66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddf2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/233129456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddf2fb0 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 -- 172.21.15.158:0/233129456 >> 172.21.15.158:0/233129456 conn(0x5635edcf5d80 msgr2=0x5635ede10ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 -- 172.21.15.158:0/233129456 shutdown_connections 2023-10-16T11:19:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.492+0000 7fe1c469fb80 1 -- 172.21.15.158:0/233129456 wait complete. 2023-10-16T11:19:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.493+0000 7fe1c469fb80 1 -- start start 2023-10-16T11:19:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.493+0000 7fe1c469fb80 1 Processor -- start 2023-10-16T11:19:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635eddbcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635edd1ae20 con 0x5635eddf0630 2023-10-16T11:19:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635edcb76c0 con 0x5635eddeb220 2023-10-16T11:19:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635edcb7360 con 0x5635eddc6010 2023-10-16T11:19:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1c469fb80 10 client.0 _ll_register_callbacks cb 0x5635edb147b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:19:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38150/0 (socket says 172.21.15.158:38150) 2023-10-16T11:19:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 -- 172.21.15.158:0/4157541682 learned_addr learned my addr 172.21.15.158:0/4157541682 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6251700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635eddbcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 -- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 msgr2=0x5635eddee1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddee1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 -- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 msgr2=0x5635eddbcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635eddbcf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 -- 172.21.15.158:0/4157541682 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c0035b0 con 0x5635eddf0630 2023-10-16T11:19:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635eddbcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd480 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800f490 tx=0x7fe1a8060f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effcc5e8b85af702 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.494+0000 7fe1917fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:19:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1917fa700 1 -- 172.21.15.158:0/4157541682 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8059160 con 0x5635eddf0630 2023-10-16T11:19:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1917fa700 1 -- 172.21.15.158:0/4157541682 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe1a80581f0 con 0x5635eddf0630 2023-10-16T11:19:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635eddbb5b0 con 0x5635eddf0630 2023-10-16T11:19:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1917fa700 1 -- 172.21.15.158:0/4157541682 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1a8088120 con 0x5635eddf0630 2023-10-16T11:19:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 10 client.11706 fetch_fsmap learned FSMap version 112 2023-10-16T11:19:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x5635edb45d90 con 0x5635eddf0630 2023-10-16T11:19:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1917fa700 1 -- 172.21.15.158:0/4157541682 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5635eddbb5b0 con 0x5635eddf0630 2023-10-16T11:19:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1917fa700 1 -- 172.21.15.158:0/4157541682 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 112) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe1a8060ac0 con 0x5635eddf0630 2023-10-16T11:19:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 10 client.11706 fetch_fsmap finished waiting for FSMap version 112 2023-10-16T11:19:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 10 client.11706 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:19:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 20 client.11706 populate_metadata read hostname 'smithi158' 2023-10-16T11:19:29.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635eddf5ad0 0x5635eddf3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 4 client.11706 mds_command: new command op to 38819 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:19:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 --> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635edd13e00 con 0x5635eddf5ad0 2023-10-16T11:19:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.495+0000 7fe1b6251700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635eddf5ad0 0x5635eddf3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.496+0000 7fe1917fa700 10 client.11706 ms_handle_connect on v2:172.21.15.172:6832/972449219 2023-10-16T11:19:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.496+0000 7fe1b6251700 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635eddf5ad0 0x5635eddf3730 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0d230948ba1c12c server_cookie=6451b579cf7d50c4 in_seq=0 out_seq=0 2023-10-16T11:19:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.496+0000 7fe1917fa700 1 -- 172.21.15.158:0/4157541682 <== mds.0 v2:172.21.15.172:6832/972449219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5635edd13e00 con 0x5635eddf5ad0 2023-10-16T11:19:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.496+0000 7fe1917fa700 10 client.11706 handle_command_reply: tid=0 2023-10-16T11:19:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.496+0000 7fe1c469fb80 1 client.11706 shutdown 2023-10-16T11:19:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 msgr2=0x5635eddbd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd480 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800f490 tx=0x7fe1a8060f50 comp rx=0 tx=0).stop 2023-10-16T11:19:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 shutdown_connections 2023-10-16T11:19:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635eddf5ad0 0x5635eddf3730 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd480 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/4157541682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635eddbcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1917fa700 0 client.11706 ms_handle_reset on v2:172.21.15.172:6832/972449219 2023-10-16T11:19:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 >> 172.21.15.158:0/4157541682 conn(0x5635edcf5d80 msgr2=0x5635ede076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 shutdown_connections 2023-10-16T11:19:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 1 -- 172.21.15.158:0/4157541682 wait complete. 2023-10-16T11:19:29.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.497+0000 7fe1c469fb80 20 client.11706 trim_cache size 0 max 16384 2023-10-16T11:19:29.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 Processor -- start 2023-10-16T11:19:29.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 -- start start 2023-10-16T11:19:29.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddbd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635ede1ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635ede1f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635edb9b8d0 con 0x5635eddf0630 2023-10-16T11:19:29.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635eddf3380 con 0x5635eddeb220 2023-10-16T11:19:29.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.515+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635ede0ca40 con 0x5635eddc6010 2023-10-16T11:19:29.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635ede1f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635ede1f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38158/0 (socket says 172.21.15.158:38158) 2023-10-16T11:19:29.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1932895081 learned_addr learned my addr 172.21.15.158:0/1932895081 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:29.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6251700 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddbd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635ede1ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 msgr2=0x5635eddbd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddbd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 msgr2=0x5635ede1ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635ede1ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 -- 172.21.15.158:0/1932895081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635eddc4190 con 0x5635eddf0630 2023-10-16T11:19:29.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635ede1f450 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8019000 tx=0x7fe1a808a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2773406764c682 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1a4d82700 1 -- 172.21.15.158:0/1932895081 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8055040 con 0x5635eddf0630 2023-10-16T11:19:29.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1a4d82700 1 -- 172.21.15.158:0/1932895081 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a8088120 con 0x5635eddf0630 2023-10-16T11:19:29.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.516+0000 7fe1a4d82700 1 -- 172.21.15.158:0/1932895081 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80591a0 con 0x5635eddf0630 2023-10-16T11:19:29.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 -- 172.21.15.158:0/1932895081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 msgr2=0x5635ede1f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635ede1f450 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8019000 tx=0x7fe1a808a050 comp rx=0 tx=0).stop 2023-10-16T11:19:29.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 -- 172.21.15.158:0/1932895081 shutdown_connections 2023-10-16T11:19:29.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635ede1f450 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddc6010 0x5635eddbd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/1932895081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddeb220 0x5635ede1ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 -- 172.21.15.158:0/1932895081 >> 172.21.15.158:0/1932895081 conn(0x5635edcf5d80 msgr2=0x5635eddef890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 -- 172.21.15.158:0/1932895081 shutdown_connections 2023-10-16T11:19:29.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 -- 172.21.15.158:0/1932895081 wait complete. 2023-10-16T11:19:29.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 -- start start 2023-10-16T11:19:29.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.517+0000 7fe1c469fb80 1 Processor -- start 2023-10-16T11:19:29.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635ede6b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddbd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1c469fb80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635eddc6c00 con 0x5635eddf0630 2023-10-16T11:19:29.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635ede75c60 con 0x5635eddc6010 2023-10-16T11:19:29.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1c469fb80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635ede78250 con 0x5635eddeb220 2023-10-16T11:19:29.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1c469fb80 10 client.0 _ll_register_callbacks cb 0x7fe1a8058d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:19:29.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1b6a52700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38164/0 (socket says 172.21.15.158:38164) 2023-10-16T11:19:29.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1b6a52700 1 -- 172.21.15.158:0/341369572 learned_addr learned my addr 172.21.15.158:0/341369572 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:29.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1b6251700 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635ede6b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.518+0000 7fe1b5a50700 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddbd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1b6a52700 1 -- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 msgr2=0x5635eddbd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddbd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1b6a52700 1 -- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 msgr2=0x5635ede6b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635ede6b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1b6a52700 1 -- 172.21.15.158:0/341369572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635edb45d90 con 0x5635eddf0630 2023-10-16T11:19:29.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1b6a52700 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd670 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8058c90 tx=0x7fe1a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2540c412690e8cb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1917fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:19:29.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1917fa700 1 -- 172.21.15.158:0/341369572 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80583f0 con 0x5635eddf0630 2023-10-16T11:19:29.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1917fa700 1 -- 172.21.15.158:0/341369572 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe1a8077310 con 0x5635eddf0630 2023-10-16T11:19:29.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1ac022070 con 0x5635eddf0630 2023-10-16T11:19:29.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1917fa700 1 -- 172.21.15.158:0/341369572 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1a8055040 con 0x5635eddf0630 2023-10-16T11:19:29.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 10 client.11712 fetch_fsmap learned FSMap version 112 2023-10-16T11:19:29.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x5635edb2ec10 con 0x5635eddf0630 2023-10-16T11:19:29.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1917fa700 1 -- 172.21.15.158:0/341369572 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac022070 con 0x5635eddf0630 2023-10-16T11:19:29.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1917fa700 1 -- 172.21.15.158:0/341369572 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 112) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe1a805dae0 con 0x5635eddf0630 2023-10-16T11:19:29.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 10 client.11712 fetch_fsmap finished waiting for FSMap version 112 2023-10-16T11:19:29.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 10 client.11712 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:19:29.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 20 client.11712 populate_metadata read hostname 'smithi158' 2023-10-16T11:19:29.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635ede74c60 0x5635ede75040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 4 client.11712 mds_command: new command op to 38819 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:19:29.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 --> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5635edd13b20 con 0x5635ede74c60 2023-10-16T11:19:29.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.519+0000 7fe1b6251700 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635ede74c60 0x5635ede75040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1917fa700 10 client.11712 ms_handle_connect on v2:172.21.15.172:6832/972449219 2023-10-16T11:19:29.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1b6251700 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635ede74c60 0x5635ede75040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e6f210f0a0b9263 server_cookie=bc682c76c4e08dd9 in_seq=0 out_seq=0 2023-10-16T11:19:29.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1917fa700 1 -- 172.21.15.158:0/341369572 <== mds.0 v2:172.21.15.172:6832/972449219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x5635edd13b20 con 0x5635ede74c60 2023-10-16T11:19:29.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1917fa700 10 client.11712 handle_command_reply: tid=0 2023-10-16T11:19:29.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 client.11712 shutdown 2023-10-16T11:19:29.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 msgr2=0x5635eddbd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd670 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8058c90 tx=0x7fe1a8004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 shutdown_connections 2023-10-16T11:19:29.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:6832/972449219,v1:172.21.15.172:6834/972449219] conn(0x5635ede74c60 0x5635ede75040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635eddf0630 0x5635eddbd670 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635eddc6010 0x5635ede6b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1917fa700 0 client.11712 ms_handle_reset on v2:172.21.15.172:6832/972449219 2023-10-16T11:19:29.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 --2- 172.21.15.158:0/341369572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635eddeb220 0x5635eddbd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 >> 172.21.15.158:0/341369572 conn(0x5635edcf5d80 msgr2=0x5635eddf6140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 shutdown_connections 2023-10-16T11:19:29.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 1 -- 172.21.15.158:0/341369572 wait complete. 2023-10-16T11:19:29.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.520+0000 7fe1c469fb80 20 client.11712 trim_cache size 0 max 16384 2023-10-16T11:19:29.547 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:19:29.547 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:19:29.547 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:19:29.547 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 38819, 2023-10-16T11:19:29.547 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:19:29.548 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:19:29.548 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T11:19:29.548 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 6.1849181619999998, 2023-10-16T11:19:29.548 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 112, 2023-10-16T11:19:29.549 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 217, 2023-10-16T11:19:29.549 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 217, 2023-10-16T11:19:29.549 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 11.116227424 2023-10-16T11:19:29.549 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:19:29.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe19406b020 msgr2=0x7fe19406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe19406b020 0x7fe19406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0007fd0 tx=0x7fe1a0009040 comp rx=0 tx=0).stop 2023-10-16T11:19:29.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1b00c87c0 msgr2=0x7fe1b00c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1b00c87c0 0x7fe1b00c8c40 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00cfa50 tx=0x7fe1a8011e20 comp rx=0 tx=0).stop 2023-10-16T11:19:29.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 shutdown_connections 2023-10-16T11:19:29.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe19406b020 0x7fe19406d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1b00c87c0 0x7fe1b00c8c40 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe19c004850 0x7fe1b006ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 --2- 172.21.15.158:0/1704945454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1b014e4c0 0x7fe1b014e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 >> 172.21.15.158:0/1704945454 conn(0x7fe1b006b9e0 msgr2=0x7fe1b006ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 shutdown_connections 2023-10-16T11:19:29.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.521+0000 7fe1b84b5700 1 -- 172.21.15.158:0/1704945454 wait complete. 2023-10-16T11:19:29.553 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T11:19:29.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.675+0000 7fabd5e85700 1 Processor -- start 2023-10-16T11:19:29.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.675+0000 7fabd5e85700 1 -- start start 2023-10-16T11:19:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.675+0000 7fabd5e85700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabd006e5d0 0x7fabd006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.675+0000 7fabd5e85700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd0161270 con 0x7fabd006ab40 2023-10-16T11:19:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabd5e85700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd0161510 con 0x7fabd00729b0 2023-10-16T11:19:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabd5e85700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd01617f0 con 0x7fabd006e5d0 2023-10-16T11:19:29.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabceffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabd006e5d0 0x7fabd006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabcf7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fabd006ab40 0x7fabd00722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38344/0 (socket says 172.21.15.158:38344) 2023-10-16T11:19:29.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabcf7fe700 1 -- 172.21.15.158:0/1190519102 learned_addr learned my addr 172.21.15.158:0/1190519102 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:29.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabceffd700 1 -- 172.21.15.158:0/1190519102 >> v1:172.21.15.172:6789/0 conn(0x7fabd00729b0 legacy=0x7fabd015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:29.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabceffd700 1 -- 172.21.15.158:0/1190519102 >> v1:172.21.15.158:6789/0 conn(0x7fabd006ab40 legacy=0x7fabd00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabceffd700 1 -- 172.21.15.158:0/1190519102 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd0161ad0 con 0x7fabd006e5d0 2023-10-16T11:19:29.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabce7fc700 1 -- 172.21.15.158:0/1190519102 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506640743 0 0) 0x7fabd0161270 con 0x7fabd006ab40 2023-10-16T11:19:29.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabceffd700 1 --2- 172.21.15.158:0/1190519102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006e5d0 0x7fabd006d820 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009840 tx=0x7fabc4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef3ad63042d2c63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabce7fc700 1 -- 172.21.15.158:0/1190519102 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4004230 con 0x7fabd006e5d0 2023-10-16T11:19:29.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabce7fc700 1 -- 172.21.15.158:0/1190519102 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc4004390 con 0x7fabd006e5d0 2023-10-16T11:19:29.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.676+0000 7fabce7fc700 1 -- 172.21.15.158:0/1190519102 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc40036a0 con 0x7fabd006e5d0 2023-10-16T11:19:29.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.677+0000 7fabd5e85700 1 -- 172.21.15.158:0/1190519102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006e5d0 msgr2=0x7fabd006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.677+0000 7fabd5e85700 1 --2- 172.21.15.158:0/1190519102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006e5d0 0x7fabd006d820 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009840 tx=0x7fabc4009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:29.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.677+0000 7fabd5e85700 1 -- 172.21.15.158:0/1190519102 shutdown_connections 2023-10-16T11:19:29.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.677+0000 7fabd5e85700 1 --2- 172.21.15.158:0/1190519102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006e5d0 0x7fabd006d820 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.677+0000 7fabd5e85700 1 -- 172.21.15.158:0/1190519102 >> 172.21.15.158:0/1190519102 conn(0x7fabd00d02b0 msgr2=0x7fabd00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.677+0000 7fabd5e85700 1 -- 172.21.15.158:0/1190519102 shutdown_connections 2023-10-16T11:19:29.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.677+0000 7fabd5e85700 1 -- 172.21.15.158:0/1190519102 wait complete. 2023-10-16T11:19:29.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.678+0000 7fabd5e85700 1 Processor -- start 2023-10-16T11:19:29.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.678+0000 7fabd5e85700 1 -- start start 2023-10-16T11:19:29.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabd5e85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006ab40 0x7fabd00b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabd5e85700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabd006e5d0 0x7fabd00c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabd5e85700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabd00729b0 0x7fabd00c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabd5e85700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00b1060 con 0x7fabd006ab40 2023-10-16T11:19:29.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabd5e85700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01f3b60 con 0x7fabd00729b0 2023-10-16T11:19:29.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabd5e85700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01f0c60 con 0x7fabd006e5d0 2023-10-16T11:19:29.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006ab40 0x7fabd00b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006ab40 0x7fabd00b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38174/0 (socket says 172.21.15.158:38174) 2023-10-16T11:19:29.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 -- 172.21.15.158:0/2714529655 learned_addr learned my addr 172.21.15.158:0/2714529655 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:29.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabceffd700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabd006e5d0 0x7fabd00c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcffff700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabd00729b0 0x7fabd00c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 -- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabd006e5d0 msgr2=0x7fabd00c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabd006e5d0 0x7fabd00c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 -- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabd00729b0 msgr2=0x7fabd00c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabd00729b0 0x7fabd00c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 -- 172.21.15.158:0/2714529655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc40094a0 con 0x7fabd006ab40 2023-10-16T11:19:29.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcffff700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabd00729b0 0x7fabd00c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:29.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.679+0000 7fabcf7fe700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006ab40 0x7fabd00b0b20 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fabc00011e0 tx=0x7fabc000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.680+0000 7fabb7fff700 1 -- 172.21.15.158:0/2714529655 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0004c80 con 0x7fabd006ab40 2023-10-16T11:19:29.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.680+0000 7fabb7fff700 1 -- 172.21.15.158:0/2714529655 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc0004e00 con 0x7fabd006ab40 2023-10-16T11:19:29.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.680+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd01f0f40 con 0x7fabd006ab40 2023-10-16T11:19:29.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.680+0000 7fabb7fff700 1 -- 172.21.15.158:0/2714529655 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0007310 con 0x7fabd006ab40 2023-10-16T11:19:29.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.680+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd00b2850 con 0x7fabd006ab40 2023-10-16T11:19:29.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.681+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd00681b0 con 0x7fabd006ab40 2023-10-16T11:19:29.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.681+0000 7fabb7fff700 1 -- 172.21.15.158:0/2714529655 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fabc0007470 con 0x7fabd006ab40 2023-10-16T11:19:29.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.681+0000 7fabb7fff700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fabbc06b020 0x7fabbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:29.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.681+0000 7fabb7fff700 1 -- 172.21.15.158:0/2714529655 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fabc0089820 con 0x7fabd006ab40 2023-10-16T11:19:29.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.681+0000 7fabceffd700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fabbc06b020 0x7fabbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:29.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.682+0000 7fabceffd700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fabbc06b020 0x7fabbc06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009840 tx=0x7fabc401c040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:29.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.686+0000 7fabb7fff700 1 -- 172.21.15.158:0/2714529655 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fabc00544a0 con 0x7fabd006ab40 2023-10-16T11:19:29.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.922+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fabd00c6ec0 con 0x7fabd006ab40 2023-10-16T11:19:29.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.923+0000 7fabb7fff700 1 -- 172.21.15.158:0/2714529655 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v112) v1 ==== 53+0+73 (secure 0 0 0) 0x7fabc0057ac0 con 0x7fabd006ab40 2023-10-16T11:19:29.925 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T11:19:29.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fabbc06b020 msgr2=0x7fabbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fabbc06b020 0x7fabbc06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009840 tx=0x7fabc401c040 comp rx=0 tx=0).stop 2023-10-16T11:19:29.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006ab40 msgr2=0x7fabd00b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:29.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006ab40 0x7fabd00b0b20 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fabc00011e0 tx=0x7fabc000cae0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 shutdown_connections 2023-10-16T11:19:29.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabd00729b0 0x7fabd00c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fabbc06b020 0x7fabbc06d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd006ab40 0x7fabd00b0b20 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 --2- 172.21.15.158:0/2714529655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabd006e5d0 0x7fabd00c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:29.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 >> 172.21.15.158:0/2714529655 conn(0x7fabd00d02b0 msgr2=0x7fabd015d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:29.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 shutdown_connections 2023-10-16T11:19:29.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:29.925+0000 7fabd5e85700 1 -- 172.21.15.158:0/2714529655 wait complete. 2023-10-16T11:19:29.938 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T11:19:29.939 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T11:19:29.939 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T11:19:29.939 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T11:19:29.940 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T11:19:29.940 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T11:19:29.940 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T11:19:29.941 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T11:19:29.941 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:19:29.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:29.942 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T11:19:29.959 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T11:19:29.960 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T11:19:29.960 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T11:19:29.960 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:19:29.960 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T11:19:29.961 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:19:29.961 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:19:29.961 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:19:29.961 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:19:29.961 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:19:29.962 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:19:29.962 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T11:19:29.962 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T11:19:29.962 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T11:19:29.962 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T11:19:29.963 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 36663sec preferred_lft 36663sec 2023-10-16T11:19:29.963 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T11:19:29.963 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:19:29.963 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T11:19:29.963 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T11:19:29.963 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T11:19:29.964 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:19:29.964 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T11:19:29.964 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:19:29.964 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:19:29.964 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:19:29.965 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T11:19:29.965 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:19:29.965 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:19:29.965 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:19:29.965 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T11:19:29.966 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:19:29.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:29.966 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T11:19:30.024 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T11:19:30.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:30.026 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T11:19:30.084 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T11:19:30.085 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T11:19:30.085 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T11:19:30.085 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T11:19:30.085 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T11:19:30.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:30.087 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T11:19:30.087 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T11:19:30.087 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T11:19:30.088 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T11:19:30.088 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T11:19:30.088 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T11:19:30.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:30.192 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:19:30.209 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:19:30.209 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:19:30.209 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:19:30.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:30.210 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T11:19:30.276 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:19:30.277 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T11:19:30.333 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:19:30.334 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:19:30.334 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:19:30.390 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T11:19:30.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:30.390 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:19:30.475 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:19:30.512 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:19:30.513 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:19:30.513 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:19:30.513 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:19:30.513 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:19:30.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:19:30.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:19:30.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:19:30.514 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:19:30.581 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:19:30.582 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:19:30.582 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:19:30.582 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:19:30.582 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:19:30.583 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:19:30.583 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:19:30.583 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:19:30.583 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T11:19:30.583 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T11:19:30.583 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T11:19:30.584 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T11:19:30.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:30.585 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:19:30.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:30.612 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T11:19:30.701 INFO:teuthology.orchestra.run.smithi158.stdout:client38909 2023-10-16T11:19:30.701 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:30.702 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:30.719 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T11:19:30.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.882+0000 7fd0a7daf700 1 Processor -- start 2023-10-16T11:19:30.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.882+0000 7fd0a7daf700 1 -- start start 2023-10-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a7daf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd0a015a5b0 0x7fd0a0071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a7daf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd0a00cb900 0x7fd0a0071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a7daf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a00ce150 con 0x7fd0a00726a0 2023-10-16T11:19:30.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a7daf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a00cdf70 con 0x7fd0a00cb900 2023-10-16T11:19:30.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a7daf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a00ce430 con 0x7fd0a015a5b0 2023-10-16T11:19:30.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a534a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd0a00cb900 0x7fd0a0071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:30.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a634c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd0a00726a0 0x7fd0a006ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38366/0 (socket says 172.21.15.158:38366) 2023-10-16T11:19:30.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a634c700 1 -- 172.21.15.158:0/2104148555 learned_addr learned my addr 172.21.15.158:0/2104148555 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:30.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a534a700 1 -- 172.21.15.158:0/2104148555 >> v2:172.21.15.172:3300/0 conn(0x7fd0a015a5b0 msgr2=0x7fd0a0071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:30.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a5b4b700 1 --2- 172.21.15.158:0/2104148555 >> v2:172.21.15.172:3300/0 conn(0x7fd0a015a5b0 0x7fd0a0071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:30.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a534a700 1 --2- 172.21.15.158:0/2104148555 >> v2:172.21.15.172:3300/0 conn(0x7fd0a015a5b0 0x7fd0a0071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:30.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a534a700 1 -- 172.21.15.158:0/2104148555 >> v1:172.21.15.158:6789/0 conn(0x7fd0a00726a0 legacy=0x7fd0a006ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:30.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a534a700 1 -- 172.21.15.158:0/2104148555 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a00ce710 con 0x7fd0a00cb900 2023-10-16T11:19:30.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a4b49700 1 -- 172.21.15.158:0/2104148555 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185463137 0 0) 0x7fd0a00ce150 con 0x7fd0a00726a0 2023-10-16T11:19:30.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.883+0000 7fd0a5b4b700 1 --2- 172.21.15.158:0/2104148555 >> v2:172.21.15.172:3300/0 conn(0x7fd0a015a5b0 0x7fd0a0071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:30.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a534a700 1 --2- 172.21.15.158:0/2104148555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a00cb900 0x7fd0a0071ea0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00715d0 tx=0x7fd09c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fe629767fe19a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:30.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a4b49700 1 -- 172.21.15.158:0/2104148555 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c011830 con 0x7fd0a00cb900 2023-10-16T11:19:30.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a4b49700 1 -- 172.21.15.158:0/2104148555 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c011990 con 0x7fd0a00cb900 2023-10-16T11:19:30.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a4b49700 1 -- 172.21.15.158:0/2104148555 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c011d20 con 0x7fd0a00cb900 2023-10-16T11:19:30.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 -- 172.21.15.158:0/2104148555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a00cb900 msgr2=0x7fd0a0071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:30.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/2104148555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a00cb900 0x7fd0a0071ea0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00715d0 tx=0x7fd09c009b20 comp rx=0 tx=0).stop 2023-10-16T11:19:30.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 -- 172.21.15.158:0/2104148555 shutdown_connections 2023-10-16T11:19:30.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/2104148555 >> v2:172.21.15.172:3300/0 conn(0x7fd0a015a5b0 0x7fd0a0071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:30.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/2104148555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a00cb900 0x7fd0a0071ea0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:30.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 -- 172.21.15.158:0/2104148555 >> 172.21.15.158:0/2104148555 conn(0x7fd0a00b1e90 msgr2=0x7fd0a00b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:30.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 -- 172.21.15.158:0/2104148555 shutdown_connections 2023-10-16T11:19:30.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.884+0000 7fd0a7daf700 1 -- 172.21.15.158:0/2104148555 wait complete. 2023-10-16T11:19:30.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 Processor -- start 2023-10-16T11:19:30.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 -- start start 2023-10-16T11:19:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00726a0 0x7fd0a00c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a00cb900 0x7fd0a00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a015a5b0 0x7fd0a00c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:30.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a0156a80 con 0x7fd0a015a5b0 2023-10-16T11:19:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00c8cc0 con 0x7fd0a00cb900 2023-10-16T11:19:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a7daf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a0153110 con 0x7fd0a00726a0 2023-10-16T11:19:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a634c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a015a5b0 0x7fd0a00c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a634c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a015a5b0 0x7fd0a00c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38196/0 (socket says 172.21.15.158:38196) 2023-10-16T11:19:30.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.886+0000 7fd0a634c700 1 -- 172.21.15.158:0/1245911230 learned_addr learned my addr 172.21.15.158:0/1245911230 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:30.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a534a700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a00cb900 0x7fd0a00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:30.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a5b4b700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00726a0 0x7fd0a00c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:30.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a634c700 1 -- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00726a0 msgr2=0x7fd0a00c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:30.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a634c700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00726a0 0x7fd0a00c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:30.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a634c700 1 -- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a00cb900 msgr2=0x7fd0a00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:30.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a634c700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a00cb900 0x7fd0a00c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:30.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a634c700 1 -- 172.21.15.158:0/1245911230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0094a0 con 0x7fd0a015a5b0 2023-10-16T11:19:30.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a534a700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a00cb900 0x7fd0a00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:30.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a634c700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a015a5b0 0x7fd0a00c7b40 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7fd09800cc70 tx=0x7fd09800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:30.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0967fc700 1 -- 172.21.15.158:0/1245911230 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09800ad40 con 0x7fd0a015a5b0 2023-10-16T11:19:30.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0967fc700 1 -- 172.21.15.158:0/1245911230 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09800ce70 con 0x7fd0a015a5b0 2023-10-16T11:19:30.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a00c9490 con 0x7fd0a015a5b0 2023-10-16T11:19:30.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0967fc700 1 -- 172.21.15.158:0/1245911230 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0980072f0 con 0x7fd0a015a5b0 2023-10-16T11:19:30.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.887+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a00c8ed0 con 0x7fd0a015a5b0 2023-10-16T11:19:30.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.888+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a0156bc0 con 0x7fd0a015a5b0 2023-10-16T11:19:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.889+0000 7fd0967fc700 1 -- 172.21.15.158:0/1245911230 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd098007450 con 0x7fd0a015a5b0 2023-10-16T11:19:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.889+0000 7fd0967fc700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd08406b020 0x7fd08406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.889+0000 7fd0a5b4b700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd08406b020 0x7fd08406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.889+0000 7fd0967fc700 1 -- 172.21.15.158:0/1245911230 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fd098089890 con 0x7fd0a015a5b0 2023-10-16T11:19:30.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.889+0000 7fd0a5b4b700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd08406b020 0x7fd08406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd090006fd0 tx=0x7fd090008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:30.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:30.894+0000 7fd0967fc700 1 -- 172.21.15.158:0/1245911230 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd0a0156bc0 con 0x7fd0a015a5b0 2023-10-16T11:19:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.131+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd0a0156bc0 con 0x7fd0a015a5b0 2023-10-16T11:19:31.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.639+0000 7fd0967fc700 1 -- 172.21.15.158:0/1245911230 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v113) v1 ==== 136+0+0 (secure 0 0 0) 0x7fd0a0156bc0 con 0x7fd0a015a5b0 2023-10-16T11:19:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.641+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd08406b020 msgr2=0x7fd08406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.641+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd08406b020 0x7fd08406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd090006fd0 tx=0x7fd090008040 comp rx=0 tx=0).stop 2023-10-16T11:19:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.641+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a015a5b0 msgr2=0x7fd0a00c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.641+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a015a5b0 0x7fd0a00c7b40 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7fd09800cc70 tx=0x7fd09800a340 comp rx=0 tx=0).stop 2023-10-16T11:19:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 shutdown_connections 2023-10-16T11:19:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd08406b020 0x7fd08406d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0a015a5b0 0x7fd0a00c7b40 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00726a0 0x7fd0a00c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 --2- 172.21.15.158:0/1245911230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a00cb900 0x7fd0a00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 >> 172.21.15.158:0/1245911230 conn(0x7fd0a00b1e90 msgr2=0x7fd0a00b5200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 shutdown_connections 2023-10-16T11:19:31.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.642+0000 7fd0a7daf700 1 -- 172.21.15.158:0/1245911230 wait complete. 2023-10-16T11:19:31.656 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:31.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.775+0000 7f99ee9c7700 1 Processor -- start 2023-10-16T11:19:31.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99ee9c7700 1 -- start start 2023-10-16T11:19:31.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99ee9c7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99e806afe0 0x7f99e815df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99ee9c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 0x7f99e815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99ee9c7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e8161c80 con 0x7f99e815db30 2023-10-16T11:19:31.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99ee9c7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e806e090 con 0x7f99e806afe0 2023-10-16T11:19:31.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99ee9c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e806ddb0 con 0x7f99e8162110 2023-10-16T11:19:31.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99e7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f99e815db30 0x7f99e806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38376/0 (socket says 172.21.15.158:38376) 2023-10-16T11:19:31.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.776+0000 7f99e7fff700 1 -- 172.21.15.158:0/651310481 learned_addr learned my addr 172.21.15.158:0/651310481 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:31.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99ecf64700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 0x7f99e815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e77fe700 1 --2- 172.21.15.158:0/651310481 >> v2:172.21.15.172:3300/0 conn(0x7f99e806afe0 0x7f99e815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015213013 0 0) 0x7f99e8161c80 con 0x7f99e815db30 2023-10-16T11:19:31.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d40035b0 con 0x7f99e815db30 2023-10-16T11:19:31.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171709851 0 0) 0x7f99d40035b0 con 0x7f99e815db30 2023-10-16T11:19:31.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 msgr2=0x7f99e815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:31.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 0x7f99e815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 >> v2:172.21.15.172:3300/0 conn(0x7f99e806afe0 msgr2=0x7f99e815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:31.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 --2- 172.21.15.158:0/651310481 >> v2:172.21.15.172:3300/0 conn(0x7f99e806afe0 0x7f99e815df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8161630 con 0x7f99e815db30 2023-10-16T11:19:31.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f99d8001670 con 0x7f99e815db30 2023-10-16T11:19:31.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 >> v1:172.21.15.158:6789/0 conn(0x7f99e815db30 legacy=0x7f99e806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d4003a00 0x7f99d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 0x7f99d4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4004d90 0x7f99d4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e806e090 con 0x7f99d4003a00 2023-10-16T11:19:31.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e806ddb0 con 0x7f99d4004d90 2023-10-16T11:19:31.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d400baf0 con 0x7f99e8162110 2023-10-16T11:19:31.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e77fe700 1 --2- 172.21.15.158:0/651310481 >> v2:172.21.15.172:3300/0 conn(0x7f99e806afe0 0x7f99e815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:31.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e7fff700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d4003a00 0x7f99d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99ecf64700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4004d90 0x7f99d4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e77fe700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 0x7f99d4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e7fff700 1 -- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 msgr2=0x7f99d4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:31.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e7fff700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 0x7f99d4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e7fff700 1 -- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4004d90 msgr2=0x7f99d4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:31.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e7fff700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4004d90 0x7f99d4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e7fff700 1 -- 172.21.15.158:0/651310481 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40035b0 con 0x7f99d4003a00 2023-10-16T11:19:31.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99e77fe700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e8162110 0x7f99d4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:31.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.777+0000 7f99ecf64700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4004d90 0x7f99d4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:31.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99e7fff700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d4003a00 0x7f99d4003e20 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f99d800c2f0 tx=0x7f99d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7dff2f6a03d268 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:31.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d800ce30 con 0x7f99d4003a00 2023-10-16T11:19:31.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d8005960 con 0x7f99d4003a00 2023-10-16T11:19:31.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99e6ffd700 1 -- 172.21.15.158:0/651310481 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8005d10 con 0x7f99d4003a00 2023-10-16T11:19:31.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99ee9c7700 1 -- 172.21.15.158:0/651310481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d4003a00 msgr2=0x7f99d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99ee9c7700 1 --2- 172.21.15.158:0/651310481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99d4003a00 0x7f99d4003e20 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f99d800c2f0 tx=0x7f99d4005880 comp rx=0 tx=0).stop 2023-10-16T11:19:31.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99ecf64700 1 -- 172.21.15.158:0/651310481 reap_dead start 2023-10-16T11:19:31.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99ee9c7700 1 -- 172.21.15.158:0/651310481 shutdown_connections 2023-10-16T11:19:31.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99ee9c7700 1 -- 172.21.15.158:0/651310481 >> 172.21.15.158:0/651310481 conn(0x7f99e80ccdb0 msgr2=0x7f99e80cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:31.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99ee9c7700 1 -- 172.21.15.158:0/651310481 shutdown_connections 2023-10-16T11:19:31.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.778+0000 7f99ee9c7700 1 -- 172.21.15.158:0/651310481 wait complete. 2023-10-16T11:19:31.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.779+0000 7f99ee9c7700 1 Processor -- start 2023-10-16T11:19:31.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.780+0000 7f99ee9c7700 1 -- start start 2023-10-16T11:19:31.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.780+0000 7f99ee9c7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4003a00 0x7f99e80c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.780+0000 7f99ee9c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99e8159910 0x7f99e815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.780+0000 7f99ee9c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e815f4a0 0x7f99e815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.780+0000 7f99ee9c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80b36e0 con 0x7f99e8159910 2023-10-16T11:19:31.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.780+0000 7f99ee9c7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8151110 con 0x7f99d4003a00 2023-10-16T11:19:31.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.780+0000 7f99ee9c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80c6d80 con 0x7f99e815f4a0 2023-10-16T11:19:31.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99e8159910 0x7f99e815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99e8159910 0x7f99e815ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38214/0 (socket says 172.21.15.158:38214) 2023-10-16T11:19:31.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 -- 172.21.15.158:0/668673010 learned_addr learned my addr 172.21.15.158:0/668673010 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:31.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e7fff700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4003a00 0x7f99e80c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99ecf64700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e815f4a0 0x7f99e815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 -- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e815f4a0 msgr2=0x7f99e815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e815f4a0 0x7f99e815f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 -- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4003a00 msgr2=0x7f99e80c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4003a00 0x7f99e80c6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 -- 172.21.15.158:0/668673010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e815c4f0 con 0x7f99e8159910 2023-10-16T11:19:31.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e77fe700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99e8159910 0x7f99e815ef60 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f99d0010c00 tx=0x7f99d0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:31.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e4ff9700 1 -- 172.21.15.158:0/668673010 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000eb90 con 0x7f99e8159910 2023-10-16T11:19:31.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e4ff9700 1 -- 172.21.15.158:0/668673010 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d000ecf0 con 0x7f99e8159910 2023-10-16T11:19:31.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e815c7d0 con 0x7f99e8159910 2023-10-16T11:19:31.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99e4ff9700 1 -- 172.21.15.158:0/668673010 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d001c600 con 0x7f99e8159910 2023-10-16T11:19:31.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.781+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e80c3c70 con 0x7f99e8159910 2023-10-16T11:19:31.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.782+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e815ca10 con 0x7f99e8159910 2023-10-16T11:19:31.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.783+0000 7f99e4ff9700 1 -- 172.21.15.158:0/668673010 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f99d00143f0 con 0x7f99e8159910 2023-10-16T11:19:31.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.783+0000 7f99e4ff9700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99c806b020 0x7f99c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:31.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.783+0000 7f99e4ff9700 1 -- 172.21.15.158:0/668673010 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f99d0089fa0 con 0x7f99e8159910 2023-10-16T11:19:31.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.783+0000 7f99e7fff700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99c806b020 0x7f99c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:31.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.783+0000 7f99e7fff700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99c806b020 0x7f99c806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f99d8015040 tx=0x7f99d8007000 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:31.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.787+0000 7f99e4ff9700 1 -- 172.21.15.158:0/668673010 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99d00bcfe0 con 0x7f99e8159910 2023-10-16T11:19:31.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.981+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99e80681b0 con 0x7f99e8159910 2023-10-16T11:19:31.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.983+0000 7f99e4ff9700 1 -- 172.21.15.158:0/668673010 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+17531 (secure 0 0 0) 0x7f99d000ee60 con 0x7f99e8159910 2023-10-16T11:19:31.984 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:31.985 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":217,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:24.314678+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:31.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99c806b020 msgr2=0x7f99c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99c806b020 0x7f99c806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f99d8015040 tx=0x7f99d8007000 comp rx=0 tx=0).stop 2023-10-16T11:19:31.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99e8159910 msgr2=0x7f99e815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:31.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99e8159910 0x7f99e815ef60 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f99d0010c00 tx=0x7f99d0009ec0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 shutdown_connections 2023-10-16T11:19:31.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99e815f4a0 0x7f99e815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99d4003a00 0x7f99e80c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f99c806b020 0x7f99c806d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 --2- 172.21.15.158:0/668673010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99e8159910 0x7f99e815ef60 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:31.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 >> 172.21.15.158:0/668673010 conn(0x7f99e80ccdb0 msgr2=0x7f99e80aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:31.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 shutdown_connections 2023-10-16T11:19:31.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:31.985+0000 7f99ee9c7700 1 -- 172.21.15.158:0/668673010 wait complete. 2023-10-16T11:19:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:19:21.151561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '216', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:19:21.679054+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:19:32.000 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:19:32.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.121+0000 7fa627a71700 1 Processor -- start 2023-10-16T11:19:32.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.121+0000 7fa627a71700 1 -- start start 2023-10-16T11:19:32.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.122+0000 7fa627a71700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa62015da40 0x7fa6200caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.122+0000 7fa627a71700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6200cc590 0x7fa62006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.122+0000 7fa627a71700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6200cbd30 con 0x7fa62006ae10 2023-10-16T11:19:32.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.122+0000 7fa627a71700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62006b2c0 con 0x7fa62015da40 2023-10-16T11:19:32.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.122+0000 7fa627a71700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200cc1c0 con 0x7fa6200cc590 2023-10-16T11:19:32.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62500c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6200cc590 0x7fa62006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62580d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa62015da40 0x7fa6200caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62600e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa62006ae10 0x7fa6200ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:42304/0 (socket says 172.21.15.158:42304) 2023-10-16T11:19:32.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62500c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6200cc590 0x7fa62006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:49054/0 (socket says 172.21.15.158:49054) 2023-10-16T11:19:32.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62600e700 1 -- 172.21.15.158:0/906840277 learned_addr learned my addr 172.21.15.158:0/906840277 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:32.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62580d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa62015da40 0x7fa6200caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:39400/0 (socket says 172.21.15.158:39400) 2023-10-16T11:19:32.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62500c700 1 -- 172.21.15.158:0/906840277 >> v2:172.21.15.172:3300/0 conn(0x7fa62015da40 msgr2=0x7fa6200caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:32.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62500c700 1 --2- 172.21.15.158:0/906840277 >> v2:172.21.15.172:3300/0 conn(0x7fa62015da40 0x7fa6200caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62500c700 1 -- 172.21.15.158:0/906840277 >> v1:172.21.15.172:6789/0 conn(0x7fa62006ae10 legacy=0x7fa6200ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62500c700 1 -- 172.21.15.158:0/906840277 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62015de20 con 0x7fa6200cc590 2023-10-16T11:19:32.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62580d700 1 --2- 172.21.15.158:0/906840277 >> v2:172.21.15.172:3300/0 conn(0x7fa62015da40 0x7fa6200caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:32.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62500c700 1 --2- 172.21.15.158:0/906840277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6200cc590 0x7fa62006a8d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009840 tx=0x7fa61c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85901429d5f9650a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62480b700 1 -- 172.21.15.158:0/906840277 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c011830 con 0x7fa6200cc590 2023-10-16T11:19:32.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62480b700 1 -- 172.21.15.158:0/906840277 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c011990 con 0x7fa6200cc590 2023-10-16T11:19:32.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.123+0000 7fa62480b700 1 -- 172.21.15.158:0/906840277 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c011d20 con 0x7fa6200cc590 2023-10-16T11:19:32.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 -- 172.21.15.158:0/906840277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6200cc590 msgr2=0x7fa62006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 --2- 172.21.15.158:0/906840277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6200cc590 0x7fa62006a8d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009840 tx=0x7fa61c009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:32.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 -- 172.21.15.158:0/906840277 shutdown_connections 2023-10-16T11:19:32.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 --2- 172.21.15.158:0/906840277 >> v2:172.21.15.172:3300/0 conn(0x7fa62015da40 0x7fa6200caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 --2- 172.21.15.158:0/906840277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6200cc590 0x7fa62006a8d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 -- 172.21.15.158:0/906840277 >> 172.21.15.158:0/906840277 conn(0x7fa620099a10 msgr2=0x7fa6200d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:32.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 -- 172.21.15.158:0/906840277 shutdown_connections 2023-10-16T11:19:32.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.124+0000 7fa627a71700 1 -- 172.21.15.158:0/906840277 wait complete. 2023-10-16T11:19:32.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.125+0000 7fa627a71700 1 Processor -- start 2023-10-16T11:19:32.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.125+0000 7fa627a71700 1 -- start start 2023-10-16T11:19:32.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa627a71700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa62006ae10 0x7fa6201f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa627a71700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6200cc590 0x7fa6200b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa627a71700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa62015da40 0x7fa6201f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa627a71700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa620154150 con 0x7fa6200cc590 2023-10-16T11:19:32.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa627a71700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200ca330 con 0x7fa62006ae10 2023-10-16T11:19:32.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa627a71700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200c9d70 con 0x7fa62015da40 2023-10-16T11:19:32.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6200cc590 0x7fa6200b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6200cc590 0x7fa6200b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38228/0 (socket says 172.21.15.158:38228) 2023-10-16T11:19:32.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 -- 172.21.15.158:0/4040641410 learned_addr learned my addr 172.21.15.158:0/4040641410 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:32.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62600e700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa62015da40 0x7fa6201f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62580d700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa62006ae10 0x7fa6201f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 -- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa62015da40 msgr2=0x7fa6201f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa62015da40 0x7fa6201f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 -- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa62006ae10 msgr2=0x7fa6201f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa62006ae10 0x7fa6201f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 -- 172.21.15.158:0/4040641410 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6100094a0 con 0x7fa6200cc590 2023-10-16T11:19:32.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.126+0000 7fa62500c700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6200cc590 0x7fa6200b3f10 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fa61c000c00 tx=0x7fa61c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.127+0000 7fa6167fc700 1 -- 172.21.15.158:0/4040641410 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0045d0 con 0x7fa6200cc590 2023-10-16T11:19:32.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.127+0000 7fa6167fc700 1 -- 172.21.15.158:0/4040641410 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c01f070 con 0x7fa6200cc590 2023-10-16T11:19:32.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.127+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c0094a0 con 0x7fa6200cc590 2023-10-16T11:19:32.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.127+0000 7fa6167fc700 1 -- 172.21.15.158:0/4040641410 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0038e0 con 0x7fa6200cc590 2023-10-16T11:19:32.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.127+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6201e6300 con 0x7fa6200cc590 2023-10-16T11:19:32.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.128+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6200681b0 con 0x7fa6200cc590 2023-10-16T11:19:32.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.128+0000 7fa6167fc700 1 -- 172.21.15.158:0/4040641410 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa61c003a40 con 0x7fa6200cc590 2023-10-16T11:19:32.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.128+0000 7fa6167fc700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa60406b020 0x7fa60406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.129+0000 7fa62580d700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa60406b020 0x7fa60406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.129+0000 7fa6167fc700 1 -- 172.21.15.158:0/4040641410 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fa61c021070 con 0x7fa6200cc590 2023-10-16T11:19:32.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.129+0000 7fa62580d700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa60406b020 0x7fa60406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa610009cd0 tx=0x7fa610009210 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.133+0000 7fa6167fc700 1 -- 172.21.15.158:0/4040641410 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa61c055f50 con 0x7fa6200cc590 2023-10-16T11:19:32.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.322+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6200ce740 con 0x7fa6200cc590 2023-10-16T11:19:32.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.323+0000 7fa6167fc700 1 -- 172.21.15.158:0/4040641410 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v217) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa61c059570 con 0x7fa6200cc590 2023-10-16T11:19:32.325 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:19:32.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa60406b020 msgr2=0x7fa60406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa60406b020 0x7fa60406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa610009cd0 tx=0x7fa610009210 comp rx=0 tx=0).stop 2023-10-16T11:19:32.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6200cc590 msgr2=0x7fa6200b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6200cc590 0x7fa6200b3f10 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fa61c000c00 tx=0x7fa61c004750 comp rx=0 tx=0).stop 2023-10-16T11:19:32.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 shutdown_connections 2023-10-16T11:19:32.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa62015da40 0x7fa6201f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa62006ae10 0x7fa6201f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa60406b020 0x7fa60406d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 --2- 172.21.15.158:0/4040641410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6200cc590 0x7fa6200b3f10 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 >> 172.21.15.158:0/4040641410 conn(0x7fa620099a10 msgr2=0x7fa62015b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:32.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 shutdown_connections 2023-10-16T11:19:32.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.325+0000 7fa627a71700 1 -- 172.21.15.158:0/4040641410 wait complete. 2023-10-16T11:19:32.339 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:19:32.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.460+0000 7fc80a639700 1 Processor -- start 2023-10-16T11:19:32.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.460+0000 7fc80a639700 1 -- start start 2023-10-16T11:19:32.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc80a639700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc80415b6b0 0x7fc8040cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc80a639700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8040d0c60 0x7fc8040b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc80a639700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8040b3460 0x7fc804070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc80a639700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80415ba90 con 0x7fc8040b3460 2023-10-16T11:19:32.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc80a639700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8040b38f0 con 0x7fc80415b6b0 2023-10-16T11:19:32.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc80a639700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8040b4420 con 0x7fc8040d0c60 2023-10-16T11:19:32.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc808bd6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8040b3460 0x7fc804070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc808bd6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8040b3460 0x7fc804070300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38238/0 (socket says 172.21.15.158:38238) 2023-10-16T11:19:32.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc808bd6700 1 -- 172.21.15.158:0/2721198260 learned_addr learned my addr 172.21.15.158:0/2721198260 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:32.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.461+0000 7fc8037fe700 1 --2- 172.21.15.158:0/2721198260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8040d0c60 0x7fc8040b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc803fff700 1 --2- 172.21.15.158:0/2721198260 >> v2:172.21.15.172:3300/0 conn(0x7fc80415b6b0 0x7fc8040cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc808bd6700 1 -- 172.21.15.158:0/2721198260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8040d0c60 msgr2=0x7fc8040b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:32.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc808bd6700 1 --2- 172.21.15.158:0/2721198260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8040d0c60 0x7fc8040b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc808bd6700 1 -- 172.21.15.158:0/2721198260 >> v2:172.21.15.172:3300/0 conn(0x7fc80415b6b0 msgr2=0x7fc8040cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:32.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc808bd6700 1 --2- 172.21.15.158:0/2721198260 >> v2:172.21.15.172:3300/0 conn(0x7fc80415b6b0 0x7fc8040cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc808bd6700 1 -- 172.21.15.158:0/2721198260 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8040cbc20 con 0x7fc8040b3460 2023-10-16T11:19:32.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc8037fe700 1 --2- 172.21.15.158:0/2721198260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8040d0c60 0x7fc8040b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:32.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc808bd6700 1 --2- 172.21.15.158:0/2721198260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040b3460 0x7fc804070300 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fc80406ffb0 tx=0x7fc7fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36037134dfed9306 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc802ffd700 1 -- 172.21.15.158:0/2721198260 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00e070 con 0x7fc8040b3460 2023-10-16T11:19:32.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc802ffd700 1 -- 172.21.15.158:0/2721198260 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc00b9c0 con 0x7fc8040b3460 2023-10-16T11:19:32.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc802ffd700 1 -- 172.21.15.158:0/2721198260 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00bd70 con 0x7fc8040b3460 2023-10-16T11:19:32.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc80a639700 1 -- 172.21.15.158:0/2721198260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040b3460 msgr2=0x7fc804070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc80a639700 1 --2- 172.21.15.158:0/2721198260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040b3460 0x7fc804070300 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fc80406ffb0 tx=0x7fc7fc00b330 comp rx=0 tx=0).stop 2023-10-16T11:19:32.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc80a639700 1 -- 172.21.15.158:0/2721198260 shutdown_connections 2023-10-16T11:19:32.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc80a639700 1 --2- 172.21.15.158:0/2721198260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040b3460 0x7fc804070300 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc80a639700 1 --2- 172.21.15.158:0/2721198260 >> v2:172.21.15.172:3300/0 conn(0x7fc80415b6b0 0x7fc8040cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.462+0000 7fc80a639700 1 --2- 172.21.15.158:0/2721198260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8040d0c60 0x7fc8040b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.463+0000 7fc80a639700 1 -- 172.21.15.158:0/2721198260 >> 172.21.15.158:0/2721198260 conn(0x7fc8040cdf20 msgr2=0x7fc8040ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:32.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.463+0000 7fc80a639700 1 -- 172.21.15.158:0/2721198260 shutdown_connections 2023-10-16T11:19:32.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.463+0000 7fc80a639700 1 -- 172.21.15.158:0/2721198260 wait complete. 2023-10-16T11:19:32.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.464+0000 7fc80a639700 1 Processor -- start 2023-10-16T11:19:32.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.464+0000 7fc80a639700 1 -- start start 2023-10-16T11:19:32.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8040b3460 0x7fc8040a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040d0c60 0x7fc8040ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc80415b6b0 0x7fc8040cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8041572d0 con 0x7fc8040d0c60 2023-10-16T11:19:32.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc804152a40 con 0x7fc8040b3460 2023-10-16T11:19:32.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc804156ca0 con 0x7fc80415b6b0 2023-10-16T11:19:32.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040d0c60 0x7fc8040ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040d0c60 0x7fc8040ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38242/0 (socket says 172.21.15.158:38242) 2023-10-16T11:19:32.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 -- 172.21.15.158:0/3205448241 learned_addr learned my addr 172.21.15.158:0/3205448241 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:32.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc803fff700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8040b3460 0x7fc8040a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc808bd6700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc80415b6b0 0x7fc8040cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 -- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc80415b6b0 msgr2=0x7fc8040cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc80415b6b0 0x7fc8040cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 -- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8040b3460 msgr2=0x7fc8040a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8040b3460 0x7fc8040a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 -- 172.21.15.158:0/3205448241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc009b30 con 0x7fc8040d0c60 2023-10-16T11:19:32.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc8037fe700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040d0c60 0x7fc8040ca740 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400d800 tx=0x7fc7f400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc800ff9700 1 -- 172.21.15.158:0/3205448241 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4004030 con 0x7fc8040d0c60 2023-10-16T11:19:32.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc800ff9700 1 -- 172.21.15.158:0/3205448241 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f4004190 con 0x7fc8040d0c60 2023-10-16T11:19:32.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8041561a0 con 0x7fc8040d0c60 2023-10-16T11:19:32.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc800ff9700 1 -- 172.21.15.158:0/3205448241 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4004500 con 0x7fc8040d0c60 2023-10-16T11:19:32.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.465+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc804156ef0 con 0x7fc8040d0c60 2023-10-16T11:19:32.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.466+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc804157540 con 0x7fc8040d0c60 2023-10-16T11:19:32.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.467+0000 7fc800ff9700 1 -- 172.21.15.158:0/3205448241 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc7f40036a0 con 0x7fc8040d0c60 2023-10-16T11:19:32.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.467+0000 7fc800ff9700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc7e406b020 0x7fc7e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.467+0000 7fc800ff9700 1 -- 172.21.15.158:0/3205448241 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fc7f4088ec0 con 0x7fc8040d0c60 2023-10-16T11:19:32.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.467+0000 7fc803fff700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc7e406b020 0x7fc7e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.468+0000 7fc803fff700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc7e406b020 0x7fc7e406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc80415e750 tx=0x7fc7ec008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.472+0000 7fc800ff9700 1 -- 172.21.15.158:0/3205448241 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7f4053c80 con 0x7fc8040d0c60 2023-10-16T11:19:32.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.665+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc8040cc490 con 0x7fc8040d0c60 2023-10-16T11:19:32.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.665+0000 7fc800ff9700 1 -- 172.21.15.158:0/3205448241 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v217) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc7f40180a0 con 0x7fc8040d0c60 2023-10-16T11:19:32.668 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:19:32.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc7e406b020 msgr2=0x7fc7e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc7e406b020 0x7fc7e406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc80415e750 tx=0x7fc7ec008040 comp rx=0 tx=0).stop 2023-10-16T11:19:32.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040d0c60 msgr2=0x7fc8040ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040d0c60 0x7fc8040ca740 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400d800 tx=0x7fc7f400db10 comp rx=0 tx=0).stop 2023-10-16T11:19:32.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 shutdown_connections 2023-10-16T11:19:32.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc80415b6b0 0x7fc8040cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8040b3460 0x7fc8040a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc7e406b020 0x7fc7e406d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 --2- 172.21.15.158:0/3205448241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8040d0c60 0x7fc8040ca740 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.667+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 >> 172.21.15.158:0/3205448241 conn(0x7fc8040cdf20 msgr2=0x7fc80415c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:32.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.668+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 shutdown_connections 2023-10-16T11:19:32.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.668+0000 7fc80a639700 1 -- 172.21.15.158:0/3205448241 wait complete. 2023-10-16T11:19:32.682 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:19:32.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.803+0000 7f3d4bfe6700 1 Processor -- start 2023-10-16T11:19:32.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.804+0000 7f3d4bfe6700 1 -- start start 2023-10-16T11:19:32.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.804+0000 7f3d4bfe6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3d4406afe0 0x7f3d440caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.804+0000 7f3d4bfe6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d440afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.804+0000 7f3d4bfe6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440cbd30 con 0x7f3d4415da40 2023-10-16T11:19:32.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.804+0000 7f3d4bfe6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d440cc1c0 con 0x7f3d4406afe0 2023-10-16T11:19:32.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.804+0000 7f3d4bfe6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4415de20 con 0x7f3d440cc590 2023-10-16T11:19:32.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3d4406afe0 0x7f3d440caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3d4406afe0 0x7f3d440caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38250/0 (socket says 172.21.15.158:38250) 2023-10-16T11:19:32.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 -- 172.21.15.158:0/3620587042 learned_addr learned my addr 172.21.15.158:0/3620587042 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:32.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d4a583700 1 --2- 172.21.15.158:0/3620587042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d440afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 -- 172.21.15.158:0/3620587042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 msgr2=0x7f3d440afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:32.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 --2- 172.21.15.158:0/3620587042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d440afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 -- 172.21.15.158:0/3620587042 >> v1:172.21.15.172:6789/0 conn(0x7f3d4415da40 legacy=0x7f3d4406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 -- 172.21.15.158:0/3620587042 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d440ce260 con 0x7f3d4406afe0 2023-10-16T11:19:32.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d49581700 1 --2- 172.21.15.158:0/3620587042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4406afe0 0x7f3d440caa20 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f3d400098f0 tx=0x7f3d40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c245badfd99ea2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d48d80700 1 -- 172.21.15.158:0/3620587042 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40011830 con 0x7f3d4406afe0 2023-10-16T11:19:32.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d48d80700 1 -- 172.21.15.158:0/3620587042 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d40011990 con 0x7f3d4406afe0 2023-10-16T11:19:32.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.805+0000 7f3d48d80700 1 -- 172.21.15.158:0/3620587042 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40011d20 con 0x7f3d4406afe0 2023-10-16T11:19:32.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/3620587042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4406afe0 msgr2=0x7f3d440caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/3620587042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4406afe0 0x7f3d440caa20 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f3d400098f0 tx=0x7f3d40009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:32.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/3620587042 shutdown_connections 2023-10-16T11:19:32.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/3620587042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d440afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/3620587042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4406afe0 0x7f3d440caa20 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/3620587042 >> 172.21.15.158:0/3620587042 conn(0x7f3d44099a10 msgr2=0x7f3d440d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:32.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/3620587042 shutdown_connections 2023-10-16T11:19:32.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.806+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/3620587042 wait complete. 2023-10-16T11:19:32.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.807+0000 7f3d4bfe6700 1 Processor -- start 2023-10-16T11:19:32.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.807+0000 7f3d4bfe6700 1 -- start start 2023-10-16T11:19:32.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4bfe6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d4406afe0 0x7f3d4415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4bfe6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d441e7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4bfe6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4415da40 0x7f3d441e7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4bfe6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440ca950 con 0x7f3d4415da40 2023-10-16T11:19:32.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4bfe6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4415ba10 con 0x7f3d4406afe0 2023-10-16T11:19:32.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4bfe6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d441e5e30 con 0x7f3d440cc590 2023-10-16T11:19:32.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4415da40 0x7f3d441e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4415da40 0x7f3d441e7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38258/0 (socket says 172.21.15.158:38258) 2023-10-16T11:19:32.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 -- 172.21.15.158:0/2445460604 learned_addr learned my addr 172.21.15.158:0/2445460604 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:32.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d49581700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d441e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d49d82700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d4406afe0 0x7f3d4415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 -- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 msgr2=0x7f3d441e7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d441e7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 -- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d4406afe0 msgr2=0x7f3d4415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:32.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d4406afe0 0x7f3d4415b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 -- 172.21.15.158:0/2445460604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40009550 con 0x7f3d4415da40 2023-10-16T11:19:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d49d82700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d4406afe0 0x7f3d4415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d4a583700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4415da40 0x7f3d441e7740 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00efc0 tx=0x7f3d3c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.808+0000 7f3d3a7fc700 1 -- 172.21.15.158:0/2445460604 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c009ea0 con 0x7f3d4415da40 2023-10-16T11:19:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.809+0000 7f3d3a7fc700 1 -- 172.21.15.158:0/2445460604 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c00f040 con 0x7f3d4415da40 2023-10-16T11:19:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.809+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d441e6110 con 0x7f3d4415da40 2023-10-16T11:19:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.809+0000 7f3d3a7fc700 1 -- 172.21.15.158:0/2445460604 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c014480 con 0x7f3d4415da40 2023-10-16T11:19:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.809+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d441e66d0 con 0x7f3d4415da40 2023-10-16T11:19:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.809+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d440681b0 con 0x7f3d4415da40 2023-10-16T11:19:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.810+0000 7f3d3a7fc700 1 -- 172.21.15.158:0/2445460604 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3d3c0145e0 con 0x7f3d4415da40 2023-10-16T11:19:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.810+0000 7f3d3a7fc700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3d2806b020 0x7f3d2806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.810+0000 7f3d3a7fc700 1 -- 172.21.15.158:0/2445460604 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f3d3c08a670 con 0x7f3d4415da40 2023-10-16T11:19:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.810+0000 7f3d49d82700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3d2806b020 0x7f3d2806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.811+0000 7f3d49d82700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3d2806b020 0x7f3d2806d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d34000f60 tx=0x7f3d34009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:32.815+0000 7f3d3a7fc700 1 -- 172.21.15.158:0/2445460604 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3d3c055270 con 0x7f3d4415da40 2023-10-16T11:19:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.007+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d440d0e40 con 0x7f3d4415da40 2023-10-16T11:19:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.008+0000 7f3d3a7fc700 1 -- 172.21.15.158:0/2445460604 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v217) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3d3c058890 con 0x7f3d4415da40 2023-10-16T11:19:33.010 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:19:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3d2806b020 msgr2=0x7f3d2806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3d2806b020 0x7f3d2806d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d34000f60 tx=0x7f3d34009040 comp rx=0 tx=0).stop 2023-10-16T11:19:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4415da40 msgr2=0x7f3d441e7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4415da40 0x7f3d441e7740 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00efc0 tx=0x7f3d3c00c3a0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 shutdown_connections 2023-10-16T11:19:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f3d2806b020 0x7f3d2806d4e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d4415da40 0x7f3d441e7740 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d4406afe0 0x7f3d4415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 --2- 172.21.15.158:0/2445460604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d440cc590 0x7f3d441e7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 >> 172.21.15.158:0/2445460604 conn(0x7f3d44099a10 msgr2=0x7f3d4415c800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:33.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 shutdown_connections 2023-10-16T11:19:33.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.010+0000 7f3d4bfe6700 1 -- 172.21.15.158:0/2445460604 wait complete. 2023-10-16T11:19:33.024 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:19:33.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.143+0000 7fed06eb1700 1 Processor -- start 2023-10-16T11:19:33.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.143+0000 7fed06eb1700 1 -- start start 2023-10-16T11:19:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed06eb1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fed0015a5d0 0x7fed000ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed06eb1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0015e860 0x7fed000b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed06eb1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0015edd0 con 0x7fed000b1db0 2023-10-16T11:19:33.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed06eb1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed000b27a0 con 0x7fed0015a5d0 2023-10-16T11:19:33.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed06eb1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed000ce080 con 0x7fed0015e860 2023-10-16T11:19:33.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed0544e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0015e860 0x7fed000b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed04c4d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fed0015a5d0 0x7fed000ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed0544e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0015e860 0x7fed000b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:49100/0 (socket says 172.21.15.158:49100) 2023-10-16T11:19:33.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fed0544e700 1 -- 172.21.15.158:0/2286880304 learned_addr learned my addr 172.21.15.158:0/2286880304 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:33.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.144+0000 7fecfffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fed000b1db0 0x7fed0015e150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:42344/0 (socket says 172.21.15.158:42344) 2023-10-16T11:19:33.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fecff7fe700 1 -- 172.21.15.158:0/2286880304 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955806627 0 0) 0x7fed0015edd0 con 0x7fed000b1db0 2023-10-16T11:19:33.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fecff7fe700 1 -- 172.21.15.158:0/2286880304 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf00035b0 con 0x7fed000b1db0 2023-10-16T11:19:33.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fed0544e700 1 -- 172.21.15.158:0/2286880304 >> v2:172.21.15.172:3300/0 conn(0x7fed0015a5d0 msgr2=0x7fed000ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:33.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fed0544e700 1 --2- 172.21.15.158:0/2286880304 >> v2:172.21.15.172:3300/0 conn(0x7fed0015a5d0 0x7fed000ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fed0544e700 1 -- 172.21.15.158:0/2286880304 >> v1:172.21.15.172:6789/0 conn(0x7fed000b1db0 legacy=0x7fed0015e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fed0544e700 1 -- 172.21.15.158:0/2286880304 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed000ce360 con 0x7fed0015e860 2023-10-16T11:19:33.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fed04c4d700 1 --2- 172.21.15.158:0/2286880304 >> v2:172.21.15.172:3300/0 conn(0x7fed0015a5d0 0x7fed000ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:33.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fed0544e700 1 --2- 172.21.15.158:0/2286880304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0015e860 0x7fed000b2260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009e50 tx=0x7fecf800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7854c9a3abb2f915 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:33.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fecff7fe700 1 -- 172.21.15.158:0/2286880304 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800e070 con 0x7fed0015e860 2023-10-16T11:19:33.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fecff7fe700 1 -- 172.21.15.158:0/2286880304 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf800b910 con 0x7fed0015e860 2023-10-16T11:19:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.145+0000 7fecff7fe700 1 -- 172.21.15.158:0/2286880304 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800bcc0 con 0x7fed0015e860 2023-10-16T11:19:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 -- 172.21.15.158:0/2286880304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0015e860 msgr2=0x7fed000b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 --2- 172.21.15.158:0/2286880304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0015e860 0x7fed000b2260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009e50 tx=0x7fecf800b330 comp rx=0 tx=0).stop 2023-10-16T11:19:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 -- 172.21.15.158:0/2286880304 shutdown_connections 2023-10-16T11:19:33.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 --2- 172.21.15.158:0/2286880304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0015e860 0x7fed000b2260 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 --2- 172.21.15.158:0/2286880304 >> v2:172.21.15.172:3300/0 conn(0x7fed0015a5d0 0x7fed000ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 -- 172.21.15.158:0/2286880304 >> 172.21.15.158:0/2286880304 conn(0x7fed000ccdb0 msgr2=0x7fed000cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 -- 172.21.15.158:0/2286880304 shutdown_connections 2023-10-16T11:19:33.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.146+0000 7fed06eb1700 1 -- 172.21.15.158:0/2286880304 wait complete. 2023-10-16T11:19:33.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.147+0000 7fed06eb1700 1 Processor -- start 2023-10-16T11:19:33.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.147+0000 7fed06eb1700 1 -- start start 2023-10-16T11:19:33.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed06eb1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed000b1db0 0x7fed000af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed06eb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed0015a5d0 0x7fed00159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed06eb1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0015e860 0x7fed00159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed06eb1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed000af5c0 con 0x7fed0015a5d0 2023-10-16T11:19:33.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed06eb1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0014b580 con 0x7fed0015e860 2023-10-16T11:19:33.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed06eb1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0014b860 con 0x7fed000b1db0 2023-10-16T11:19:33.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed0015a5d0 0x7fed00159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed0015a5d0 0x7fed00159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38270/0 (socket says 172.21.15.158:38270) 2023-10-16T11:19:33.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 -- 172.21.15.158:0/1853006004 learned_addr learned my addr 172.21.15.158:0/1853006004 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:33.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed04c4d700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed000b1db0 0x7fed000af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fed0544e700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0015e860 0x7fed00159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 -- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed000b1db0 msgr2=0x7fed000af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed000b1db0 0x7fed000af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 -- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0015e860 msgr2=0x7fed00159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0015e860 0x7fed00159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 -- 172.21.15.158:0/1853006004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40094a0 con 0x7fed0015a5d0 2023-10-16T11:19:33.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.148+0000 7fecfffff700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed0015a5d0 0x7fed00159350 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7fecec00c700 tx=0x7fecec00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:33.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.149+0000 7fecfd7fa700 1 -- 172.21.15.158:0/1853006004 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec005cb0 con 0x7fed0015a5d0 2023-10-16T11:19:33.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.149+0000 7fecfd7fa700 1 -- 172.21.15.158:0/1853006004 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec005e10 con 0x7fed0015a5d0 2023-10-16T11:19:33.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.149+0000 7fecfd7fa700 1 -- 172.21.15.158:0/1853006004 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0072d0 con 0x7fed0015a5d0 2023-10-16T11:19:33.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.149+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf8009b30 con 0x7fed0015a5d0 2023-10-16T11:19:33.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.149+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0014be20 con 0x7fed0015a5d0 2023-10-16T11:19:33.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.150+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed000681b0 con 0x7fed0015a5d0 2023-10-16T11:19:33.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.150+0000 7fecfd7fa700 1 -- 172.21.15.158:0/1853006004 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fecec007430 con 0x7fed0015a5d0 2023-10-16T11:19:33.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.150+0000 7fecfd7fa700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fece406b020 0x7fece406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.150+0000 7fecfd7fa700 1 -- 172.21.15.158:0/1853006004 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fecec088e00 con 0x7fed0015a5d0 2023-10-16T11:19:33.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.150+0000 7fed04c4d700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fece406b020 0x7fece406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.151+0000 7fed04c4d700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fece406b020 0x7fece406d4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009d70 tx=0x7fecf4009210 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:33.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.155+0000 7fecfd7fa700 1 -- 172.21.15.158:0/1853006004 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fecec053ab0 con 0x7fed0015a5d0 2023-10-16T11:19:33.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.348+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fed000ca960 con 0x7fed0015a5d0 2023-10-16T11:19:33.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.349+0000 7fecfd7fa700 1 -- 172.21.15.158:0/1853006004 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v217) v1 ==== 84+0+209 (secure 0 0 0) 0x7fecec017090 con 0x7fed0015a5d0 2023-10-16T11:19:33.351 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:33.351 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:19:33.351 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:33.352 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:19:33.352 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:19:33.352 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:33.352 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:33.352 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 27, 2023-10-16T11:19:33.353 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": "cephfs_metadata" 2023-10-16T11:19:33.353 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:33.353 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:33.353 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 28, 2023-10-16T11:19:33.353 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": "cephfs_data" 2023-10-16T11:19:33.353 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:33.354 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:19:33.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fece406b020 msgr2=0x7fece406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fece406b020 0x7fece406d4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009d70 tx=0x7fecf4009210 comp rx=0 tx=0).stop 2023-10-16T11:19:33.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed0015a5d0 msgr2=0x7fed00159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed0015a5d0 0x7fed00159350 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7fecec00c700 tx=0x7fecec00ca10 comp rx=0 tx=0).stop 2023-10-16T11:19:33.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 shutdown_connections 2023-10-16T11:19:33.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0015e860 0x7fed00159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed000b1db0 0x7fed000af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fece406b020 0x7fece406d4e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 --2- 172.21.15.158:0/1853006004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed0015a5d0 0x7fed00159350 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 >> 172.21.15.158:0/1853006004 conn(0x7fed000ccdb0 msgr2=0x7fed00158a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:33.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 shutdown_connections 2023-10-16T11:19:33.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.351+0000 7fed06eb1700 1 -- 172.21.15.158:0/1853006004 wait complete. 2023-10-16T11:19:33.364 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T11:19:33.365 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T11:19:33.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.487+0000 7f97be05d700 1 Processor -- start 2023-10-16T11:19:33.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.487+0000 7f97be05d700 1 -- start start 2023-10-16T11:19:33.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97be05d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f97b815daa0 0x7f97b80cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97be05d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f97b80d0640 0x7f97b80cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97be05d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b80d1ee0 con 0x7f97b815d6c0 2023-10-16T11:19:33.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97be05d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b80d40a0 con 0x7f97b815daa0 2023-10-16T11:19:33.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97be05d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b80d12d0 con 0x7f97b80d0640 2023-10-16T11:19:33.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97b6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f97b815daa0 0x7f97b80cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97b6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f97b815daa0 0x7f97b80cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38278/0 (socket says 172.21.15.158:38278) 2023-10-16T11:19:33.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.488+0000 7f97b6ffd700 1 -- 172.21.15.158:0/2012897145 learned_addr learned my addr 172.21.15.158:0/2012897145 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:33.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b7fff700 1 --2- 172.21.15.158:0/2012897145 >> v2:172.21.15.172:3300/0 conn(0x7f97b80d0640 0x7f97b80cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b6ffd700 1 -- 172.21.15.158:0/2012897145 >> v2:172.21.15.172:3300/0 conn(0x7f97b80d0640 msgr2=0x7f97b80cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:33.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b6ffd700 1 --2- 172.21.15.158:0/2012897145 >> v2:172.21.15.172:3300/0 conn(0x7f97b80d0640 0x7f97b80cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b6ffd700 1 -- 172.21.15.158:0/2012897145 >> v1:172.21.15.172:6789/0 conn(0x7f97b815d6c0 legacy=0x7f97b80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b6ffd700 1 -- 172.21.15.158:0/2012897145 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80d15b0 con 0x7f97b815daa0 2023-10-16T11:19:33.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b7fff700 1 --2- 172.21.15.158:0/2012897145 >> v2:172.21.15.172:3300/0 conn(0x7f97b80d0640 0x7f97b80cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:33.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b6ffd700 1 --2- 172.21.15.158:0/2012897145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cec50 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f97b80b5050 tx=0x7f97a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b606cf601b34fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:33.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b67fc700 1 -- 172.21.15.158:0/2012897145 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a800b870 con 0x7f97b815daa0 2023-10-16T11:19:33.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b67fc700 1 -- 172.21.15.158:0/2012897145 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a800b9d0 con 0x7f97b815daa0 2023-10-16T11:19:33.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.489+0000 7f97b67fc700 1 -- 172.21.15.158:0/2012897145 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a800bd60 con 0x7f97b815daa0 2023-10-16T11:19:33.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 -- 172.21.15.158:0/2012897145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 msgr2=0x7f97b80cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 --2- 172.21.15.158:0/2012897145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cec50 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f97b80b5050 tx=0x7f97a8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 -- 172.21.15.158:0/2012897145 shutdown_connections 2023-10-16T11:19:33.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 --2- 172.21.15.158:0/2012897145 >> v2:172.21.15.172:3300/0 conn(0x7f97b80d0640 0x7f97b80cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 --2- 172.21.15.158:0/2012897145 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cec50 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 -- 172.21.15.158:0/2012897145 >> 172.21.15.158:0/2012897145 conn(0x7f97b809c940 msgr2=0x7f97b80d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:33.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 -- 172.21.15.158:0/2012897145 shutdown_connections 2023-10-16T11:19:33.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.490+0000 7f97be05d700 1 -- 172.21.15.158:0/2012897145 wait complete. 2023-10-16T11:19:33.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.491+0000 7f97be05d700 1 Processor -- start 2023-10-16T11:19:33.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.491+0000 7f97be05d700 1 -- start start 2023-10-16T11:19:33.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97be05d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b80d0640 0x7f97b80c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97be05d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b815d6c0 0x7f97b81609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97be05d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97be05d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80c90a0 con 0x7f97b815daa0 2023-10-16T11:19:33.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97be05d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b8160ef0 con 0x7f97b815d6c0 2023-10-16T11:19:33.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97be05d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80cc0b0 con 0x7f97b80d0640 2023-10-16T11:19:33.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38286/0 (socket says 172.21.15.158:38286) 2023-10-16T11:19:33.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 -- 172.21.15.158:0/3229708601 learned_addr learned my addr 172.21.15.158:0/3229708601 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:33.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b77fe700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b80d0640 0x7f97b80c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b6ffd700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b815d6c0 0x7f97b81609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 -- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b80d0640 msgr2=0x7f97b80c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b80d0640 0x7f97b80c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 -- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b815d6c0 msgr2=0x7f97b81609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:33.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b815d6c0 0x7f97b81609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:33.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 -- 172.21.15.158:0/3229708601 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a8009550 con 0x7f97b815daa0 2023-10-16T11:19:33.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f97b7fff700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cf0d0 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007ec0 tx=0x7f97b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.492+0000 7f979ffff700 1 -- 172.21.15.158:0/3229708601 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000dde0 con 0x7f97b815daa0 2023-10-16T11:19:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.493+0000 7f979ffff700 1 -- 172.21.15.158:0/3229708601 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b000f040 con 0x7f97b815daa0 2023-10-16T11:19:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.493+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b80cc390 con 0x7f97b815daa0 2023-10-16T11:19:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.493+0000 7f979ffff700 1 -- 172.21.15.158:0/3229708601 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0015640 con 0x7f97b815daa0 2023-10-16T11:19:33.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.493+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b80cc950 con 0x7f97b815daa0 2023-10-16T11:19:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.493+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b80681b0 con 0x7f97b815daa0 2023-10-16T11:19:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.494+0000 7f979ffff700 1 -- 172.21.15.158:0/3229708601 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97b001e3b0 con 0x7f97b815daa0 2023-10-16T11:19:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.494+0000 7f979ffff700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f97a406b020 0x7f97a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.495+0000 7f979ffff700 1 -- 172.21.15.158:0/3229708601 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f97b0089c30 con 0x7f97b815daa0 2023-10-16T11:19:33.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.495+0000 7f97b77fe700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f97a406b020 0x7f97a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:33.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.495+0000 7f97b77fe700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f97a406b020 0x7f97a406d4e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f97ac007fd0 tx=0x7f97ac009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:33.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.499+0000 7f979ffff700 1 -- 172.21.15.158:0/3229708601 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97b00547a0 con 0x7f97b815daa0 2023-10-16T11:19:33.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:33.693+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f97b80d0d30 con 0x7f97b815daa0 2023-10-16T11:19:34.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.650+0000 7f979ffff700 1 -- 172.21.15.158:0/3229708601 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v218) v1 ==== 144+0+0 (secure 0 0 0) 0x7f97b001a0b0 con 0x7f97b815daa0 2023-10-16T11:19:34.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f97a406b020 msgr2=0x7f97a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f97a406b020 0x7f97a406d4e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f97ac007fd0 tx=0x7f97ac009040 comp rx=0 tx=0).stop 2023-10-16T11:19:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 msgr2=0x7f97b80cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cf0d0 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007ec0 tx=0x7f97b000d3b0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 shutdown_connections 2023-10-16T11:19:34.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f97a406b020 0x7f97a406d4e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97b815daa0 0x7f97b80cf0d0 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97b80d0640 0x7f97b80c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 --2- 172.21.15.158:0/3229708601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97b815d6c0 0x7f97b81609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.654+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 >> 172.21.15.158:0/3229708601 conn(0x7f97b809c940 msgr2=0x7f97b80b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:34.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.655+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 shutdown_connections 2023-10-16T11:19:34.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.655+0000 7f97be05d700 1 -- 172.21.15.158:0/3229708601 wait complete. 2023-10-16T11:19:34.660 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_metadata' created 2023-10-16T11:19:34.675 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:19:34.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.806+0000 7fc459ca8700 1 Processor -- start 2023-10-16T11:19:34.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.806+0000 7fc459ca8700 1 -- start start 2023-10-16T11:19:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc459ca8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc45415a5e0 0x7fc4540cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc459ca8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 0x7fc45406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc459ca8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc454072c70 con 0x7fc45406f550 2023-10-16T11:19:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc459ca8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45415a9c0 con 0x7fc45415a5e0 2023-10-16T11:19:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc459ca8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540cbc00 con 0x7fc4540cb780 2023-10-16T11:19:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc453fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc45406f550 0x7fc4540cdea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38470/0 (socket says 172.21.15.158:38470) 2023-10-16T11:19:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc453fff700 1 -- 172.21.15.158:0/369183029 learned_addr learned my addr 172.21.15.158:0/369183029 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc4537fe700 1 --2- 172.21.15.158:0/369183029 >> v2:172.21.15.172:3300/0 conn(0x7fc45415a5e0 0x7fc4540cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064653460 0 0) 0x7fc44c001490 con 0x7fc45406f550 2023-10-16T11:19:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc452ffd700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 0x7fc45406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.807+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c0035b0 con 0x7fc45406f550 2023-10-16T11:19:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1612140825 0 0) 0x7fc43c0035b0 con 0x7fc45406f550 2023-10-16T11:19:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 msgr2=0x7fc45406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 0x7fc45406f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 >> v2:172.21.15.172:3300/0 conn(0x7fc45415a5e0 msgr2=0x7fc4540cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:34.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 --2- 172.21.15.158:0/369183029 >> v2:172.21.15.172:3300/0 conn(0x7fc45415a5e0 0x7fc4540cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540727e0 con 0x7fc45406f550 2023-10-16T11:19:34.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc44c001b80 con 0x7fc45406f550 2023-10-16T11:19:34.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 >> v1:172.21.15.158:6789/0 conn(0x7fc45406f550 legacy=0x7fc4540cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:34.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc43c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc43c004850 0x7fc43c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 0x7fc43c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540cbc00 con 0x7fc43c003a00 2023-10-16T11:19:34.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c00ba10 con 0x7fc43c004850 2023-10-16T11:19:34.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c00bb50 con 0x7fc4540cb780 2023-10-16T11:19:34.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4537fe700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc43c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc452ffd700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc43c004850 0x7fc43c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc453fff700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 0x7fc43c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4537fe700 1 -- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 msgr2=0x7fc43c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:34.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4537fe700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 0x7fc43c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4537fe700 1 -- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc43c004850 msgr2=0x7fc43c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:34.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4537fe700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc43c004850 0x7fc43c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4537fe700 1 -- 172.21.15.158:0/369183029 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c0035b0 con 0x7fc43c003a00 2023-10-16T11:19:34.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc453fff700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4540cb780 0x7fc43c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:34.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4537fe700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc43c003e20 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fc448009b60 tx=0x7fc43c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7785903d656ced1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:34.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44800e660 con 0x7fc43c003a00 2023-10-16T11:19:34.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44800e7c0 con 0x7fc43c003a00 2023-10-16T11:19:34.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.808+0000 7fc4527fc700 1 -- 172.21.15.158:0/369183029 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44800eb30 con 0x7fc43c003a00 2023-10-16T11:19:34.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.809+0000 7fc459ca8700 1 -- 172.21.15.158:0/369183029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 msgr2=0x7fc43c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:34.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.809+0000 7fc459ca8700 1 --2- 172.21.15.158:0/369183029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc43c003e20 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fc448009b60 tx=0x7fc43c005880 comp rx=0 tx=0).stop 2023-10-16T11:19:34.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.809+0000 7fc453fff700 1 -- 172.21.15.158:0/369183029 reap_dead start 2023-10-16T11:19:34.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.809+0000 7fc459ca8700 1 -- 172.21.15.158:0/369183029 shutdown_connections 2023-10-16T11:19:34.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.809+0000 7fc459ca8700 1 -- 172.21.15.158:0/369183029 >> 172.21.15.158:0/369183029 conn(0x7fc4540d02d0 msgr2=0x7fc4540d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:34.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.809+0000 7fc459ca8700 1 -- 172.21.15.158:0/369183029 shutdown_connections 2023-10-16T11:19:34.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.809+0000 7fc459ca8700 1 -- 172.21.15.158:0/369183029 wait complete. 2023-10-16T11:19:34.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.810+0000 7fc459ca8700 1 Processor -- start 2023-10-16T11:19:34.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.810+0000 7fc459ca8700 1 -- start start 2023-10-16T11:19:34.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc459ca8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc454154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc459ca8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45415d0e0 0x7fc454158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc459ca8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc45415d560 0x7fc4541594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc459ca8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc454165a10 con 0x7fc43c003a00 2023-10-16T11:19:34.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc459ca8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45415c040 con 0x7fc45415d0e0 2023-10-16T11:19:34.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc459ca8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45415c320 con 0x7fc45415d560 2023-10-16T11:19:34.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc4537fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc454154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc4537fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc454154820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38302/0 (socket says 172.21.15.158:38302) 2023-10-16T11:19:34.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc4537fe700 1 -- 172.21.15.158:0/2239506254 learned_addr learned my addr 172.21.15.158:0/2239506254 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:34.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc452ffd700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45415d0e0 0x7fc454158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.811+0000 7fc453fff700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc45415d560 0x7fc4541594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc4537fe700 1 -- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc45415d560 msgr2=0x7fc4541594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:34.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc4537fe700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc45415d560 0x7fc4541594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc4537fe700 1 -- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45415d0e0 msgr2=0x7fc454158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:34.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc4537fe700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45415d0e0 0x7fc454158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:34.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc4537fe700 1 -- 172.21.15.158:0/2239506254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45415c600 con 0x7fc43c003a00 2023-10-16T11:19:34.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc452ffd700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45415d0e0 0x7fc454158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:34.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc4537fe700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc454154820 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0046f0 tx=0x7fc4480047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:34.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc43bfff700 1 -- 172.21.15.158:0/2239506254 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44800e660 con 0x7fc43c003a00 2023-10-16T11:19:34.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc43bfff700 1 -- 172.21.15.158:0/2239506254 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448003ab0 con 0x7fc43c003a00 2023-10-16T11:19:34.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc454165a10 con 0x7fc43c003a00 2023-10-16T11:19:34.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc43bfff700 1 -- 172.21.15.158:0/2239506254 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448003e60 con 0x7fc43c003a00 2023-10-16T11:19:34.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.812+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45415cbc0 con 0x7fc43c003a00 2023-10-16T11:19:34.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.813+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45415cf30 con 0x7fc43c003a00 2023-10-16T11:19:34.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.813+0000 7fc43bfff700 1 -- 172.21.15.158:0/2239506254 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc448013470 con 0x7fc43c003a00 2023-10-16T11:19:34.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.814+0000 7fc43bfff700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc44006b020 0x7fc44006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:34.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.814+0000 7fc43bfff700 1 -- 172.21.15.158:0/2239506254 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6882+0+0 (secure 0 0 0) 0x7fc44808bba0 con 0x7fc43c003a00 2023-10-16T11:19:34.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.814+0000 7fc452ffd700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc44006b020 0x7fc44006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:34.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.814+0000 7fc452ffd700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc44006b020 0x7fc44006d4e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4540b2c00 tx=0x7fc444008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:34.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:34.818+0000 7fc43bfff700 1 -- 172.21.15.158:0/2239506254 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc448056810 con 0x7fc43c003a00 2023-10-16T11:19:35.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.010+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc454067500 con 0x7fc43c003a00 2023-10-16T11:19:35.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.656+0000 7fc43bfff700 1 -- 172.21.15.158:0/2239506254 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v219) v1 ==== 176+0+0 (secure 0 0 0) 0x7fc448017020 con 0x7fc43c003a00 2023-10-16T11:19:35.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc44006b020 msgr2=0x7fc44006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:35.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc44006b020 0x7fc44006d4e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4540b2c00 tx=0x7fc444008040 comp rx=0 tx=0).stop 2023-10-16T11:19:35.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 msgr2=0x7fc454154820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:35.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc454154820 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0046f0 tx=0x7fc4480047c0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 shutdown_connections 2023-10-16T11:19:35.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc45415d560 0x7fc4541594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc44006b020 0x7fc44006d4e0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4540b2c00 tx=0x7fc444008040 comp rx=0 tx=0).stop 2023-10-16T11:19:35.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc43c003a00 0x7fc454154820 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 --2- 172.21.15.158:0/2239506254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45415d0e0 0x7fc454158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.659+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 >> 172.21.15.158:0/2239506254 conn(0x7fc4540d02d0 msgr2=0x7fc4540cc490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:35.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.660+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 shutdown_connections 2023-10-16T11:19:35.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.660+0000 7fc459ca8700 1 -- 172.21.15.158:0/2239506254 wait complete. 2023-10-16T11:19:35.673 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_data' created 2023-10-16T11:19:35.685 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T11:19:35.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.808+0000 7f88cdf2e700 1 Processor -- start 2023-10-16T11:19:35.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.809+0000 7f88cdf2e700 1 -- start start 2023-10-16T11:19:35.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.809+0000 7f88cdf2e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 0x7f88c80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.809+0000 7f88cdf2e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80cbd30 con 0x7f88c806afe0 2023-10-16T11:19:35.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.809+0000 7f88cdf2e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80cc1c0 con 0x7f88c815da40 2023-10-16T11:19:35.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.809+0000 7f88cdf2e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c815de20 con 0x7f88c80cc590 2023-10-16T11:19:35.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.809+0000 7f88c6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f88c806afe0 0x7f88c80afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38488/0 (socket says 172.21.15.158:38488) 2023-10-16T11:19:35.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.809+0000 7f88c6ffd700 1 -- 172.21.15.158:0/534671064 learned_addr learned my addr 172.21.15.158:0/534671064 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:35.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639971520 0 0) 0x7f88c80cbd30 con 0x7f88c806afe0 2023-10-16T11:19:35.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b40035b0 con 0x7f88c806afe0 2023-10-16T11:19:35.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c7fff700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 0x7f88c80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582253534 0 0) 0x7f88b40035b0 con 0x7f88c806afe0 2023-10-16T11:19:35.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 msgr2=0x7f88c80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:35.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 0x7f88c80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 >> v1:172.21.15.172:6789/0 conn(0x7f88c815da40 legacy=0x7f88c806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:35.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80ce260 con 0x7f88c806afe0 2023-10-16T11:19:35.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f88b8001670 con 0x7f88c806afe0 2023-10-16T11:19:35.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 >> v1:172.21.15.158:6789/0 conn(0x7f88c806afe0 legacy=0x7f88c80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:35.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88b4003a30 0x7f88b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b40043e0 0x7f88b40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 0x7f88b4005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c815de20 con 0x7f88b4003a30 2023-10-16T11:19:35.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b400ba10 con 0x7f88b40043e0 2023-10-16T11:19:35.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b400bbd0 con 0x7f88c80cc590 2023-10-16T11:19:35.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c77fe700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88b4003a30 0x7f88b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c6ffd700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b40043e0 0x7f88b40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.810+0000 7f88c7fff700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 0x7f88b4005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c77fe700 1 -- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 msgr2=0x7f88b4005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:35.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c77fe700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80cc590 0x7f88b4005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c77fe700 1 -- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b40043e0 msgr2=0x7f88b40048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:35.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c77fe700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b40043e0 0x7f88b40048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c77fe700 1 -- 172.21.15.158:0/534671064 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40035b0 con 0x7f88b4003a30 2023-10-16T11:19:35.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c77fe700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88b4003a30 0x7f88b4003e10 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f88b00018d0 tx=0x7f88b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48a8cce561dc9b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:35.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b000c690 con 0x7f88b4003a30 2023-10-16T11:19:35.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b000c810 con 0x7f88b4003a30 2023-10-16T11:19:35.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c67fc700 1 -- 172.21.15.158:0/534671064 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b000cbc0 con 0x7f88b4003a30 2023-10-16T11:19:35.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88cdf2e700 1 -- 172.21.15.158:0/534671064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88b4003a30 msgr2=0x7f88b4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:35.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88cdf2e700 1 --2- 172.21.15.158:0/534671064 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88b4003a30 0x7f88b4003e10 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f88b00018d0 tx=0x7f88b4005880 comp rx=0 tx=0).stop 2023-10-16T11:19:35.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88c7fff700 1 -- 172.21.15.158:0/534671064 reap_dead start 2023-10-16T11:19:35.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88cdf2e700 1 -- 172.21.15.158:0/534671064 shutdown_connections 2023-10-16T11:19:35.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.811+0000 7f88cdf2e700 1 -- 172.21.15.158:0/534671064 >> 172.21.15.158:0/534671064 conn(0x7f88c8099a10 msgr2=0x7f88c80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:35.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.812+0000 7f88cdf2e700 1 -- 172.21.15.158:0/534671064 shutdown_connections 2023-10-16T11:19:35.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.812+0000 7f88cdf2e700 1 -- 172.21.15.158:0/534671064 wait complete. 2023-10-16T11:19:35.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.813+0000 7f88cdf2e700 1 Processor -- start 2023-10-16T11:19:35.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.813+0000 7f88cdf2e700 1 -- start start 2023-10-16T11:19:35.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88cdf2e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b4003a30 0x7f88c80b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88cdf2e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88c80b2b60 0x7f88c80b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88cdf2e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80b2fe0 0x7f88c80b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88cdf2e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c815b1e0 con 0x7f88c80b2b60 2023-10-16T11:19:35.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88cdf2e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b34d0 con 0x7f88b4003a30 2023-10-16T11:19:35.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88cdf2e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b37b0 con 0x7f88c80b2fe0 2023-10-16T11:19:35.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88c6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88c80b2b60 0x7f88c80b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88c6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88c80b2b60 0x7f88c80b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38324/0 (socket says 172.21.15.158:38324) 2023-10-16T11:19:35.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88c6ffd700 1 -- 172.21.15.158:0/2728211959 learned_addr learned my addr 172.21.15.158:0/2728211959 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:35.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88c77fe700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b4003a30 0x7f88c80b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.814+0000 7f88c7fff700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80b2fe0 0x7f88c80b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88c6ffd700 1 -- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80b2fe0 msgr2=0x7f88c80b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:35.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88c6ffd700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80b2fe0 0x7f88c80b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88c6ffd700 1 -- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b4003a30 msgr2=0x7f88c80b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:35.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88c6ffd700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b4003a30 0x7f88c80b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:35.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88c6ffd700 1 -- 172.21.15.158:0/2728211959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80b3a90 con 0x7f88c80b2b60 2023-10-16T11:19:35.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88c6ffd700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88c80b2b60 0x7f88c80b52a0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f88b80072c0 tx=0x7f88b80075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88bffff700 1 -- 172.21.15.158:0/2728211959 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8011580 con 0x7f88c80b2b60 2023-10-16T11:19:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88bffff700 1 -- 172.21.15.158:0/2728211959 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8011740 con 0x7f88c80b2b60 2023-10-16T11:19:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c80b3d70 con 0x7f88c80b2b60 2023-10-16T11:19:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88bffff700 1 -- 172.21.15.158:0/2728211959 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8011af0 con 0x7f88c80b2b60 2023-10-16T11:19:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.815+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c80b0f10 con 0x7f88c80b2b60 2023-10-16T11:19:35.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.816+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c80b4290 con 0x7f88c80b2b60 2023-10-16T11:19:35.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.816+0000 7f88bffff700 1 -- 172.21.15.158:0/2728211959 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f88b8011c50 con 0x7f88c80b2b60 2023-10-16T11:19:35.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.817+0000 7f88bffff700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f88a806b020 0x7f88a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:35.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.817+0000 7f88bffff700 1 -- 172.21.15.158:0/2728211959 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 7267+0+0 (secure 0 0 0) 0x7f88b8089c70 con 0x7f88c80b2b60 2023-10-16T11:19:35.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.817+0000 7f88c77fe700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f88a806b020 0x7f88a806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:35.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.817+0000 7f88c77fe700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f88a806b020 0x7f88a806d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88b0014040 tx=0x7f88b0005040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:35.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:35.822+0000 7f88bffff700 1 -- 172.21.15.158:0/2728211959 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88c80b4290 con 0x7f88c80b2b60 2023-10-16T11:19:36.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.060+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f88c80b4290 con 0x7f88c80b2b60 2023-10-16T11:19:36.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.658+0000 7f88bffff700 1 -- 172.21.15.158:0/2728211959 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 29 and data pool 30 v114) v1 ==== 183+0+0 (secure 0 0 0) 0x7f88c80b4290 con 0x7f88c80b2b60 2023-10-16T11:19:36.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.663+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f88a806b020 msgr2=0x7f88a806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:36.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.663+0000 7f88cdf2e700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f88a806b020 0x7f88a806d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88b0014040 tx=0x7f88b0005040 comp rx=0 tx=0).stop 2023-10-16T11:19:36.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.663+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88c80b2b60 msgr2=0x7f88c80b52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:36.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.663+0000 7f88cdf2e700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88c80b2b60 0x7f88c80b52a0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f88b80072c0 tx=0x7f88b80075d0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.664+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 shutdown_connections 2023-10-16T11:19:36.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.664+0000 7f88cdf2e700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88c80b2fe0 0x7f88c80b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.664+0000 7f88cdf2e700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88b4003a30 0x7f88c80b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.664+0000 7f88cdf2e700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f88a806b020 0x7f88a806d4e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.664+0000 7f88cdf2e700 1 --2- 172.21.15.158:0/2728211959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88c80b2b60 0x7f88c80b52a0 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.664+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 >> 172.21.15.158:0/2728211959 conn(0x7f88c8099a10 msgr2=0x7f88c80d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:36.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.665+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 shutdown_connections 2023-10-16T11:19:36.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.666+0000 7f88cdf2e700 1 -- 172.21.15.158:0/2728211959 wait complete. 2023-10-16T11:19:36.676 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 29 and data pool 30 2023-10-16T11:19:36.698 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:36.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.821+0000 7f909d54b700 1 Processor -- start 2023-10-16T11:19:36.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.821+0000 7f909d54b700 1 -- start start 2023-10-16T11:19:36.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f909d54b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90980d1210 0x7f90980ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:36.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f909d54b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90980d3bd0 con 0x7f90980ce950 2023-10-16T11:19:36.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f909d54b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90980d2fc0 con 0x7f909815d7c0 2023-10-16T11:19:36.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f909d54b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90980d4060 con 0x7f90980d1210 2023-10-16T11:19:36.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f90967fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90980d1210 0x7f90980ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:36.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f90977fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f90980ce950 0x7f90980b6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38510/0 (socket says 172.21.15.158:38510) 2023-10-16T11:19:36.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f90967fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f90980d1210 0x7f90980ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38332/0 (socket says 172.21.15.158:38332) 2023-10-16T11:19:36.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f90977fe700 1 -- 172.21.15.158:0/2202339599 learned_addr learned my addr 172.21.15.158:0/2202339599 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:36.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f90967fc700 1 -- 172.21.15.158:0/2202339599 >> v1:172.21.15.172:6789/0 conn(0x7f909815d7c0 legacy=0x7f90980d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:36.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f90967fc700 1 -- 172.21.15.158:0/2202339599 >> v1:172.21.15.158:6789/0 conn(0x7f90980ce950 legacy=0x7f90980b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:36.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f9095ffb700 1 -- 172.21.15.158:0/2202339599 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659908182 0 0) 0x7f90980d3bd0 con 0x7f90980ce950 2023-10-16T11:19:36.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.822+0000 7f90967fc700 1 -- 172.21.15.158:0/2202339599 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980d32a0 con 0x7f90980d1210 2023-10-16T11:19:36.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f90967fc700 1 --2- 172.21.15.158:0/2202339599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90980d1210 0x7f90980ce410 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f9088009840 tx=0x7f9088009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda0c7fa16d1cd9d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:36.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f9095ffb700 1 -- 172.21.15.158:0/2202339599 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088004230 con 0x7f90980d1210 2023-10-16T11:19:36.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f9095ffb700 1 -- 172.21.15.158:0/2202339599 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9088004390 con 0x7f90980d1210 2023-10-16T11:19:36.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f9095ffb700 1 -- 172.21.15.158:0/2202339599 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90880036a0 con 0x7f90980d1210 2023-10-16T11:19:36.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f909d54b700 1 -- 172.21.15.158:0/2202339599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90980d1210 msgr2=0x7f90980ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:36.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f909d54b700 1 --2- 172.21.15.158:0/2202339599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90980d1210 0x7f90980ce410 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f9088009840 tx=0x7f9088009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:36.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f909d54b700 1 -- 172.21.15.158:0/2202339599 shutdown_connections 2023-10-16T11:19:36.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f909d54b700 1 --2- 172.21.15.158:0/2202339599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90980d1210 0x7f90980ce410 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f909d54b700 1 -- 172.21.15.158:0/2202339599 >> 172.21.15.158:0/2202339599 conn(0x7f90980669f0 msgr2=0x7f9098066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:36.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f909d54b700 1 -- 172.21.15.158:0/2202339599 shutdown_connections 2023-10-16T11:19:36.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.823+0000 7f909d54b700 1 -- 172.21.15.158:0/2202339599 wait complete. 2023-10-16T11:19:36.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 Processor -- start 2023-10-16T11:19:36.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 -- start start 2023-10-16T11:19:36.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90980ce950 0x7f909815a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:36.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90980d1210 0x7f9098157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:36.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f909815d7c0 0x7f909815ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:36.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909815a6c0 con 0x7f909815d7c0 2023-10-16T11:19:36.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90981584b0 con 0x7f90980d1210 2023-10-16T11:19:36.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.825+0000 7f909d54b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909814ad30 con 0x7f90980ce950 2023-10-16T11:19:36.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f909815d7c0 0x7f909815ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:36.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f909815d7c0 0x7f909815ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38340/0 (socket says 172.21.15.158:38340) 2023-10-16T11:19:36.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 -- 172.21.15.158:0/1517843102 learned_addr learned my addr 172.21.15.158:0/1517843102 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:36.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90967fc700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90980d1210 0x7f9098157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:36.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f9096ffd700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90980ce950 0x7f909815a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:36.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 -- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90980ce950 msgr2=0x7f909815a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:36.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90980ce950 0x7f909815a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 -- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90980d1210 msgr2=0x7f9098157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:36.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90980d1210 0x7f9098157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:36.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 -- 172.21.15.158:0/1517843102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90880094a0 con 0x7f909815d7c0 2023-10-16T11:19:36.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f90977fe700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f909815d7c0 0x7f909815ade0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f909000caa0 tx=0x7f909000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:36.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f907f7fe700 1 -- 172.21.15.158:0/1517843102 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000a9e0 con 0x7f909815d7c0 2023-10-16T11:19:36.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f907f7fe700 1 -- 172.21.15.158:0/1517843102 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909000ab60 con 0x7f909815d7c0 2023-10-16T11:19:36.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f907f7fe700 1 -- 172.21.15.158:0/1517843102 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90900070d0 con 0x7f909815d7c0 2023-10-16T11:19:36.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90981581d0 con 0x7f909815d7c0 2023-10-16T11:19:36.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.826+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909814a9c0 con 0x7f909815d7c0 2023-10-16T11:19:36.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.827+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909814aea0 con 0x7f909815d7c0 2023-10-16T11:19:36.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.828+0000 7f907f7fe700 1 -- 172.21.15.158:0/1517843102 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9090007230 con 0x7f909815d7c0 2023-10-16T11:19:36.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.828+0000 7f907f7fe700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f908406b020 0x7f908406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:36.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.828+0000 7f907f7fe700 1 -- 172.21.15.158:0/1517843102 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f90900895b0 con 0x7f909815d7c0 2023-10-16T11:19:36.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.828+0000 7f9096ffd700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f908406b020 0x7f908406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:36.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.829+0000 7f9096ffd700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f908406b020 0x7f908406d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f908c007fd0 tx=0x7f908c007ec0 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:36.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:36.832+0000 7f907f7fe700 1 -- 172.21.15.158:0/1517843102 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9090053f80 con 0x7f909815d7c0 2023-10-16T11:19:37.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.025+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90980681b0 con 0x7f909815d7c0 2023-10-16T11:19:37.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.027+0000 7f907f7fe700 1 -- 172.21.15.158:0/1517843102 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v220) v1 ==== 74+0+21200 (secure 0 0 0) 0x7f90900575a0 con 0x7f909815d7c0 2023-10-16T11:19:37.029 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:37.029 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":220,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:36.061407+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:37.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f908406b020 msgr2=0x7f908406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f908406b020 0x7f908406d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f908c007fd0 tx=0x7f908c007ec0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f909815d7c0 msgr2=0x7f909815ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f909815d7c0 0x7f909815ade0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f909000caa0 tx=0x7f909000cdb0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 shutdown_connections 2023-10-16T11:19:37.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f908406b020 0x7f908406d4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f909815d7c0 0x7f909815ade0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90980ce950 0x7f909815a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 --2- 172.21.15.158:0/1517843102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90980d1210 0x7f9098157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 >> 172.21.15.158:0/1517843102 conn(0x7f90980669f0 msgr2=0x7f90980d2280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:37.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 shutdown_connections 2023-10-16T11:19:37.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.029+0000 7f909d54b700 1 -- 172.21.15.158:0/1517843102 wait complete. 2023-10-16T11:19:37.043 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:37.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.163+0000 7f8f218d0700 1 Processor -- start 2023-10-16T11:19:37.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.163+0000 7f8f218d0700 1 -- start start 2023-10-16T11:19:37.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f218d0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8f1c0d1210 0x7f8f1c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f218d0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0d3bd0 con 0x7f8f1c0ce950 2023-10-16T11:19:37.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f218d0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0d2fc0 con 0x7f8f1c15d7c0 2023-10-16T11:19:37.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f218d0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1c0d4060 con 0x7f8f1c0d1210 2023-10-16T11:19:37.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f1b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8f1c0ce950 0x7f8f1c0b6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38524/0 (socket says 172.21.15.158:38524) 2023-10-16T11:19:37.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f1b7fe700 1 -- 172.21.15.158:0/2253121114 learned_addr learned my addr 172.21.15.158:0/2253121114 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:37.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93784956 0 0) 0x7f8f1c0d3bd0 con 0x7f8f1c0ce950 2023-10-16T11:19:37.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/2253121114 >> v2:172.21.15.172:3300/0 conn(0x7f8f1c0d1210 0x7f8f1c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f040035b0 con 0x7f8f1c0ce950 2023-10-16T11:19:37.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450327110 0 0) 0x7f8f040035b0 con 0x7f8f1c0ce950 2023-10-16T11:19:37.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 >> v2:172.21.15.172:3300/0 conn(0x7f8f1c0d1210 msgr2=0x7f8f1c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:37.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 --2- 172.21.15.158:0/2253121114 >> v2:172.21.15.172:3300/0 conn(0x7f8f1c0d1210 0x7f8f1c0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 >> v1:172.21.15.172:6789/0 conn(0x7f8f1c15d7c0 legacy=0x7f8f1c0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0d32a0 con 0x7f8f1c0ce950 2023-10-16T11:19:37.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f8f14001a40 con 0x7f8f1c0ce950 2023-10-16T11:19:37.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/2253121114 >> v2:172.21.15.172:3300/0 conn(0x7f8f1c0d1210 0x7f8f1c0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:37.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 >> v1:172.21.15.158:6789/0 conn(0x7f8f1c0ce950 legacy=0x7f8f1c0b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f04003a00 0x7f8f04003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f04004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f04005210 0x7f8f0400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c0d4060 con 0x7f8f04004850 2023-10-16T11:19:37.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0400c160 con 0x7f8f04005210 2023-10-16T11:19:37.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.164+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0400c320 con 0x7f8f04003a00 2023-10-16T11:19:37.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f04004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1b7fe700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f04005210 0x7f8f0400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1affd700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f04003a00 0x7f8f04003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1a7fc700 1 -- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f04003a00 msgr2=0x7f8f04003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:37.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f04003a00 0x7f8f04003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1a7fc700 1 -- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f04005210 msgr2=0x7f8f0400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:37.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f04005210 0x7f8f0400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1a7fc700 1 -- 172.21.15.158:0/2253121114 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f040035b0 con 0x7f8f04004850 2023-10-16T11:19:37.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1b7fe700 1 -- 172.21.15.158:0/2253121114 reap_dead start 2023-10-16T11:19:37.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f04004cd0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009950 tx=0x7f8f04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9bc4982ea0e042a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:37.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c00c070 con 0x7f8f04004850 2023-10-16T11:19:37.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c013610 con 0x7f8f04004850 2023-10-16T11:19:37.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.165+0000 7f8f19ffb700 1 -- 172.21.15.158:0/2253121114 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0139a0 con 0x7f8f04004850 2023-10-16T11:19:37.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.166+0000 7f8f218d0700 1 -- 172.21.15.158:0/2253121114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 msgr2=0x7f8f04004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.166+0000 7f8f218d0700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f04004cd0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009950 tx=0x7f8f04005880 comp rx=0 tx=0).stop 2023-10-16T11:19:37.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.166+0000 7f8f218d0700 1 -- 172.21.15.158:0/2253121114 shutdown_connections 2023-10-16T11:19:37.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.166+0000 7f8f218d0700 1 --2- 172.21.15.158:0/2253121114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f04004cd0 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.166+0000 7f8f218d0700 1 -- 172.21.15.158:0/2253121114 >> 172.21.15.158:0/2253121114 conn(0x7f8f1c0669f0 msgr2=0x7f8f1c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:37.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.166+0000 7f8f218d0700 1 -- 172.21.15.158:0/2253121114 shutdown_connections 2023-10-16T11:19:37.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.166+0000 7f8f218d0700 1 -- 172.21.15.158:0/2253121114 wait complete. 2023-10-16T11:19:37.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.167+0000 7f8f218d0700 1 Processor -- start 2023-10-16T11:19:37.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.168+0000 7f8f218d0700 1 -- start start 2023-10-16T11:19:37.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.168+0000 7f8f218d0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f1c160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.168+0000 7f8f218d0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f1c15de80 0x7f8f1c15e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.168+0000 7f8f218d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f1c0b4690 0x7f8f1c0b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.168+0000 7f8f218d0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c0b5050 con 0x7f8f04004850 2023-10-16T11:19:37.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.168+0000 7f8f218d0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c14a630 con 0x7f8f1c15de80 2023-10-16T11:19:37.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.168+0000 7f8f218d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c14a8d0 con 0x7f8f1c0b4690 2023-10-16T11:19:37.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f1c160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f1c160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38356/0 (socket says 172.21.15.158:38356) 2023-10-16T11:19:37.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 -- 172.21.15.158:0/3672041731 learned_addr learned my addr 172.21.15.158:0/3672041731 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:37.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1b7fe700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f1c0b4690 0x7f8f1c0b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f1c15de80 0x7f8f1c15e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 -- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f1c0b4690 msgr2=0x7f8f1c0b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f1c0b4690 0x7f8f1c0b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 -- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f1c15de80 msgr2=0x7f8f1c15e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f1c15de80 0x7f8f1c15e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 -- 172.21.15.158:0/3672041731 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c14abb0 con 0x7f8f04004850 2023-10-16T11:19:37.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f1affd700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f1c160a60 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000f820 tx=0x7f8f1000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:37.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f03fff700 1 -- 172.21.15.158:0/3672041731 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f100052e0 con 0x7f8f04004850 2023-10-16T11:19:37.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f03fff700 1 -- 172.21.15.158:0/3672041731 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10005460 con 0x7f8f04004850 2023-10-16T11:19:37.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f03fff700 1 -- 172.21.15.158:0/3672041731 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f100038c0 con 0x7f8f04004850 2023-10-16T11:19:37.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c14ae90 con 0x7f8f04004850 2023-10-16T11:19:37.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.169+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c15e940 con 0x7f8f04004850 2023-10-16T11:19:37.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.170+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c15ead0 con 0x7f8f04004850 2023-10-16T11:19:37.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.171+0000 7f8f03fff700 1 -- 172.21.15.158:0/3672041731 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8f10003a20 con 0x7f8f04004850 2023-10-16T11:19:37.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.171+0000 7f8f03fff700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f8f0806b020 0x7f8f0806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.171+0000 7f8f03fff700 1 -- 172.21.15.158:0/3672041731 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f8f10089b90 con 0x7f8f04004850 2023-10-16T11:19:37.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.171+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f8f0806b020 0x7f8f0806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.171+0000 7f8f1a7fc700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f8f0806b020 0x7f8f0806d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c000c00 tx=0x7f8f0c00c000 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:37.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.175+0000 7f8f03fff700 1 -- 172.21.15.158:0/3672041731 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f100544b0 con 0x7f8f04004850 2023-10-16T11:19:37.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.367+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f1c14b200 con 0x7f8f04004850 2023-10-16T11:19:37.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.369+0000 7f8f03fff700 1 -- 172.21.15.158:0/3672041731 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v220) v1 ==== 74+0+21200 (secure 0 0 0) 0x7f8f10087e60 con 0x7f8f04004850 2023-10-16T11:19:37.371 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:37.371 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":220,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:36.061407+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:37.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f8f0806b020 msgr2=0x7f8f0806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f8f0806b020 0x7f8f0806d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c000c00 tx=0x7f8f0c00c000 comp rx=0 tx=0).stop 2023-10-16T11:19:37.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 msgr2=0x7f8f1c160a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f1c160a60 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000f820 tx=0x7f8f1000fbe0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 shutdown_connections 2023-10-16T11:19:37.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f1c0b4690 0x7f8f1c0b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f8f0806b020 0x7f8f0806d4e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f04004850 0x7f8f1c160a60 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 --2- 172.21.15.158:0/3672041731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f1c15de80 0x7f8f1c15e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 >> 172.21.15.158:0/3672041731 conn(0x7f8f1c0669f0 msgr2=0x7f8f1c0cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:37.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 shutdown_connections 2023-10-16T11:19:37.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.371+0000 7f8f218d0700 1 -- 172.21.15.158:0/3672041731 wait complete. 2023-10-16T11:19:37.385 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T11:19:37.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.506+0000 7fd5cfefd700 1 Processor -- start 2023-10-16T11:19:37.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.506+0000 7fd5cfefd700 1 -- start start 2023-10-16T11:19:37.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.507+0000 7fd5cfefd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 0x7fd5c80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.507+0000 7fd5cfefd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c80cbd30 con 0x7fd5c815da40 2023-10-16T11:19:37.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.507+0000 7fd5cfefd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c80cc1c0 con 0x7fd5c80cc590 2023-10-16T11:19:37.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.507+0000 7fd5cfefd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c815de20 con 0x7fd5c806afe0 2023-10-16T11:19:37.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.507+0000 7fd5cdc99700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd5c815da40 0x7fd5c806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38540/0 (socket says 172.21.15.158:38540) 2023-10-16T11:19:37.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.507+0000 7fd5cdc99700 1 -- 172.21.15.158:0/3714736204 learned_addr learned my addr 172.21.15.158:0/3714736204 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:37.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cd498700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 0x7fd5c80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846597926 0 0) 0x7fd5c80cbd30 con 0x7fd5c815da40 2023-10-16T11:19:37.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b40035b0 con 0x7fd5c815da40 2023-10-16T11:19:37.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991456152 0 0) 0x7fd5b40035b0 con 0x7fd5c815da40 2023-10-16T11:19:37.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 msgr2=0x7fd5c80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:37.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 0x7fd5c80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 >> v1:172.21.15.172:6789/0 conn(0x7fd5c80cc590 legacy=0x7fd5c80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80ce260 con 0x7fd5c815da40 2023-10-16T11:19:37.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd5b8001670 con 0x7fd5c815da40 2023-10-16T11:19:37.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 >> v1:172.21.15.158:6789/0 conn(0x7fd5c815da40 legacy=0x7fd5c806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5b4003a30 0x7fd5b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5b40043e0 0x7fd5b40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 0x7fd5b4005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c815de20 con 0x7fd5b4003a30 2023-10-16T11:19:37.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b400ba70 con 0x7fd5b40043e0 2023-10-16T11:19:37.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b400bc30 con 0x7fd5c806afe0 2023-10-16T11:19:37.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cdc99700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5b4003a30 0x7fd5b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cd498700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5b40043e0 0x7fd5b40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5ce49a700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 0x7fd5b4005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cdc99700 1 -- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 msgr2=0x7fd5b4005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:37.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cdc99700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5c806afe0 0x7fd5b4005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cdc99700 1 -- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5b40043e0 msgr2=0x7fd5b40048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:37.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cdc99700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5b40043e0 0x7fd5b40048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.508+0000 7fd5cdc99700 1 -- 172.21.15.158:0/3714736204 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40035b0 con 0x7fd5b4003a30 2023-10-16T11:19:37.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5cdc99700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5b4003a30 0x7fd5b4003e10 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8001eb0 tx=0x7fd5b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa3ddc4cddbaffd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:37.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b800ce00 con 0x7fd5b4003a30 2023-10-16T11:19:37.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b8005960 con 0x7fd5b4003a30 2023-10-16T11:19:37.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5ccc97700 1 -- 172.21.15.158:0/3714736204 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8005cf0 con 0x7fd5b4003a30 2023-10-16T11:19:37.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5cfefd700 1 -- 172.21.15.158:0/3714736204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5b4003a30 msgr2=0x7fd5b4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5cfefd700 1 --2- 172.21.15.158:0/3714736204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5b4003a30 0x7fd5b4003e10 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8001eb0 tx=0x7fd5b4005880 comp rx=0 tx=0).stop 2023-10-16T11:19:37.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5ce49a700 1 -- 172.21.15.158:0/3714736204 reap_dead start 2023-10-16T11:19:37.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5cfefd700 1 -- 172.21.15.158:0/3714736204 shutdown_connections 2023-10-16T11:19:37.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5cfefd700 1 -- 172.21.15.158:0/3714736204 >> 172.21.15.158:0/3714736204 conn(0x7fd5c8099a10 msgr2=0x7fd5c80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:37.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5cfefd700 1 -- 172.21.15.158:0/3714736204 shutdown_connections 2023-10-16T11:19:37.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.509+0000 7fd5cfefd700 1 -- 172.21.15.158:0/3714736204 wait complete. 2023-10-16T11:19:37.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.510+0000 7fd5cfefd700 1 Processor -- start 2023-10-16T11:19:37.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cfefd700 1 -- start start 2023-10-16T11:19:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cfefd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5b4003a30 0x7fd5c815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cfefd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5c809d9b0 0x7fd5c80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cfefd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5c809de30 0x7fd5c80cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cfefd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c815b8a0 con 0x7fd5c809d9b0 2023-10-16T11:19:37.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cfefd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80cac60 con 0x7fd5c809de30 2023-10-16T11:19:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cfefd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80c7620 con 0x7fd5b4003a30 2023-10-16T11:19:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cd498700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5c809d9b0 0x7fd5c80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cd498700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5c809d9b0 0x7fd5c80cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38378/0 (socket says 172.21.15.158:38378) 2023-10-16T11:19:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.511+0000 7fd5cd498700 1 -- 172.21.15.158:0/484710286 learned_addr learned my addr 172.21.15.158:0/484710286 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:37.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cdc99700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5b4003a30 0x7fd5c815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5ce49a700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5c809de30 0x7fd5c80cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cd498700 1 -- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5b4003a30 msgr2=0x7fd5c815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cd498700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5b4003a30 0x7fd5c815b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cd498700 1 -- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5c809de30 msgr2=0x7fd5c80cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:37.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cd498700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5c809de30 0x7fd5c80cfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:37.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cd498700 1 -- 172.21.15.158:0/484710286 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80c7900 con 0x7fd5c809d9b0 2023-10-16T11:19:37.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cd498700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5c809d9b0 0x7fd5c80cf980 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4011ad0 tx=0x7fd5c4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:37.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5be7fc700 1 -- 172.21.15.158:0/484710286 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400fa00 con 0x7fd5c809d9b0 2023-10-16T11:19:37.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5be7fc700 1 -- 172.21.15.158:0/484710286 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c400fb60 con 0x7fd5c809d9b0 2023-10-16T11:19:37.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c80c7be0 con 0x7fd5c809d9b0 2023-10-16T11:19:37.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5be7fc700 1 -- 172.21.15.158:0/484710286 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4014440 con 0x7fd5c809d9b0 2023-10-16T11:19:37.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.512+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c80c81a0 con 0x7fd5c809d9b0 2023-10-16T11:19:37.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.513+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c80cecf0 con 0x7fd5c809d9b0 2023-10-16T11:19:37.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.513+0000 7fd5be7fc700 1 -- 172.21.15.158:0/484710286 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd5c40145a0 con 0x7fd5c809d9b0 2023-10-16T11:19:37.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.514+0000 7fd5be7fc700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd5ac06b020 0x7fd5ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:37.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.514+0000 7fd5be7fc700 1 -- 172.21.15.158:0/484710286 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fd5c408a470 con 0x7fd5c809d9b0 2023-10-16T11:19:37.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.514+0000 7fd5cdc99700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd5ac06b020 0x7fd5ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:37.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.514+0000 7fd5cdc99700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd5ac06b020 0x7fd5ac06d4e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8015040 tx=0x7fd5b8007000 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:37.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.518+0000 7fd5be7fc700 1 -- 172.21.15.158:0/484710286 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5c4054e50 con 0x7fd5c809d9b0 2023-10-16T11:19:37.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:37.759+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd5c80681b0 con 0x7fd5c809d9b0 2023-10-16T11:19:38.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.691+0000 7fd5be7fc700 1 -- 172.21.15.158:0/484710286 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v117) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd5c4058470 con 0x7fd5c809d9b0 2023-10-16T11:19:38.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd5ac06b020 msgr2=0x7fd5ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:38.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd5ac06b020 0x7fd5ac06d4e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8015040 tx=0x7fd5b8007000 comp rx=0 tx=0).stop 2023-10-16T11:19:38.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5c809d9b0 msgr2=0x7fd5c80cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:38.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5c809d9b0 0x7fd5c80cf980 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4011ad0 tx=0x7fd5c4011e90 comp rx=0 tx=0).stop 2023-10-16T11:19:38.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 shutdown_connections 2023-10-16T11:19:38.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5c809de30 0x7fd5c80cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5b4003a30 0x7fd5c815b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fd5ac06b020 0x7fd5ac06d4e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 --2- 172.21.15.158:0/484710286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5c809d9b0 0x7fd5c80cf980 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 >> 172.21.15.158:0/484710286 conn(0x7fd5c8099a10 msgr2=0x7fd5c806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:38.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 shutdown_connections 2023-10-16T11:19:38.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.694+0000 7fd5cfefd700 1 -- 172.21.15.158:0/484710286 wait complete. 2023-10-16T11:19:38.710 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:19:38.711 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T11:19:38.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.839+0000 7fa06bed5700 1 Processor -- start 2023-10-16T11:19:38.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.839+0000 7fa06bed5700 1 -- start start 2023-10-16T11:19:38.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa06bed5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa06406afe0 0x7fa0640caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa06bed5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640cbd30 con 0x7fa06415da40 2023-10-16T11:19:38.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa06bed5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640cc1c0 con 0x7fa0640cc590 2023-10-16T11:19:38.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa06bed5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06415de20 con 0x7fa06406afe0 2023-10-16T11:19:38.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa069c71700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa06415da40 0x7fa06406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38560/0 (socket says 172.21.15.158:38560) 2023-10-16T11:19:38.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa069c71700 1 -- 172.21.15.158:0/1905845326 learned_addr learned my addr 172.21.15.158:0/1905845326 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:38.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799820451 0 0) 0x7fa0640cbd30 con 0x7fa06415da40 2023-10-16T11:19:38.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0500035b0 con 0x7fa06415da40 2023-10-16T11:19:38.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.840+0000 7fa069470700 1 --2- 172.21.15.158:0/1905845326 >> v2:172.21.15.172:3300/0 conn(0x7fa06406afe0 0x7fa0640caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2535731627 0 0) 0x7fa0500035b0 con 0x7fa06415da40 2023-10-16T11:19:38.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 >> v2:172.21.15.172:3300/0 conn(0x7fa06406afe0 msgr2=0x7fa0640caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:38.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 --2- 172.21.15.158:0/1905845326 >> v2:172.21.15.172:3300/0 conn(0x7fa06406afe0 0x7fa0640caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 >> v1:172.21.15.172:6789/0 conn(0x7fa0640cc590 legacy=0x7fa0640afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:38.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640ce260 con 0x7fa06415da40 2023-10-16T11:19:38.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fa060001670 con 0x7fa06415da40 2023-10-16T11:19:38.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 >> v1:172.21.15.158:6789/0 conn(0x7fa06415da40 legacy=0x7fa06406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:38.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa050003a00 0x7fa050003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050004850 0x7fa050004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa050005210 0x7fa05000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06415de20 con 0x7fa050003a00 2023-10-16T11:19:38.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05000c160 con 0x7fa050004850 2023-10-16T11:19:38.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05000c320 con 0x7fa050005210 2023-10-16T11:19:38.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069c71700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa050003a00 0x7fa050003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069470700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050004850 0x7fa050004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa06a472700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa050005210 0x7fa05000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069c71700 1 -- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa050005210 msgr2=0x7fa05000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:38.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069c71700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa050005210 0x7fa05000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069c71700 1 -- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050004850 msgr2=0x7fa050004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:38.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069c71700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050004850 0x7fa050004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069c71700 1 -- 172.21.15.158:0/1905845326 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0500035b0 con 0x7fa050003a00 2023-10-16T11:19:38.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa06a472700 1 -- 172.21.15.158:0/1905845326 reap_dead start 2023-10-16T11:19:38.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa069c71700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa050003a00 0x7fa050003e20 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7fa06000c240 tx=0x7fa050005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6f23374127af84 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:38.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060007c10 con 0x7fa050003a00 2023-10-16T11:19:38.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06000cde0 con 0x7fa050003a00 2023-10-16T11:19:38.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.841+0000 7fa068c6f700 1 -- 172.21.15.158:0/1905845326 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060005ba0 con 0x7fa050003a00 2023-10-16T11:19:38.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.842+0000 7fa06bed5700 1 -- 172.21.15.158:0/1905845326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa050003a00 msgr2=0x7fa050003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:38.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.842+0000 7fa06bed5700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa050003a00 0x7fa050003e20 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7fa06000c240 tx=0x7fa050005880 comp rx=0 tx=0).stop 2023-10-16T11:19:38.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.842+0000 7fa06bed5700 1 -- 172.21.15.158:0/1905845326 shutdown_connections 2023-10-16T11:19:38.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.842+0000 7fa06bed5700 1 --2- 172.21.15.158:0/1905845326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa050003a00 0x7fa050003e20 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.842+0000 7fa06bed5700 1 -- 172.21.15.158:0/1905845326 >> 172.21.15.158:0/1905845326 conn(0x7fa064099a10 msgr2=0x7fa0640d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:38.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.842+0000 7fa06bed5700 1 -- 172.21.15.158:0/1905845326 shutdown_connections 2023-10-16T11:19:38.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.842+0000 7fa06bed5700 1 -- 172.21.15.158:0/1905845326 wait complete. 2023-10-16T11:19:38.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.843+0000 7fa06bed5700 1 Processor -- start 2023-10-16T11:19:38.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.843+0000 7fa06bed5700 1 -- start start 2023-10-16T11:19:38.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06bed5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050003a00 0x7fa0640ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06bed5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0641f2940 0x7fa0641599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06bed5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa06415df60 0x7fa0640b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06bed5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c8180 con 0x7fa06415df60 2023-10-16T11:19:38.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06bed5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064159f10 con 0x7fa050003a00 2023-10-16T11:19:38.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06bed5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c7540 con 0x7fa0641f2940 2023-10-16T11:19:38.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa06415df60 0x7fa0640b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa06415df60 0x7fa0640b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38400/0 (socket says 172.21.15.158:38400) 2023-10-16T11:19:38.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 -- 172.21.15.158:0/2582092621 learned_addr learned my addr 172.21.15.158:0/2582092621 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:38.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa069c71700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050003a00 0x7fa0640ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa069470700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0641f2940 0x7fa0641599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 -- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0641f2940 msgr2=0x7fa0641599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0641f2940 0x7fa0641599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 -- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050003a00 msgr2=0x7fa0640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050003a00 0x7fa0640ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 -- 172.21.15.158:0/2582092621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640c77e0 con 0x7fa06415df60 2023-10-16T11:19:38.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa069c71700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050003a00 0x7fa0640ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:38.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.844+0000 7fa06a472700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa06415df60 0x7fa0640b2fc0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fa0640ccb90 tx=0x7fa05c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:38.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.845+0000 7fa05a7fc700 1 -- 172.21.15.158:0/2582092621 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c003c10 con 0x7fa06415df60 2023-10-16T11:19:38.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.845+0000 7fa05a7fc700 1 -- 172.21.15.158:0/2582092621 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c003d90 con 0x7fa06415df60 2023-10-16T11:19:38.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.845+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0640c7ac0 con 0x7fa06415df60 2023-10-16T11:19:38.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.845+0000 7fa05a7fc700 1 -- 172.21.15.158:0/2582092621 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c005310 con 0x7fa06415df60 2023-10-16T11:19:38.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.845+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06415e410 con 0x7fa06415df60 2023-10-16T11:19:38.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.846+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0640c82c0 con 0x7fa06415df60 2023-10-16T11:19:38.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.846+0000 7fa05a7fc700 1 -- 172.21.15.158:0/2582092621 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa05c005470 con 0x7fa06415df60 2023-10-16T11:19:38.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.846+0000 7fa05a7fc700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa04806b020 0x7fa04806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:38.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.846+0000 7fa05a7fc700 1 -- 172.21.15.158:0/2582092621 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fa05c089c70 con 0x7fa06415df60 2023-10-16T11:19:38.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.846+0000 7fa069c71700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa04806b020 0x7fa04806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:38.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.847+0000 7fa069c71700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa04806b020 0x7fa04806d4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa060000c00 tx=0x7fa060007000 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:38.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:38.851+0000 7fa05a7fc700 1 -- 172.21.15.158:0/2582092621 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0640c82c0 con 0x7fa06415df60 2023-10-16T11:19:39.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.088+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa0640c82c0 con 0x7fa06415df60 2023-10-16T11:19:39.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.692+0000 7fa05a7fc700 1 -- 172.21.15.158:0/2582092621 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v118) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa0640c82c0 con 0x7fa06415df60 2023-10-16T11:19:39.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa04806b020 msgr2=0x7fa04806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:39.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa04806b020 0x7fa04806d4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa060000c00 tx=0x7fa060007000 comp rx=0 tx=0).stop 2023-10-16T11:19:39.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa06415df60 msgr2=0x7fa0640b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:39.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa06415df60 0x7fa0640b2fc0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fa0640ccb90 tx=0x7fa05c00ee00 comp rx=0 tx=0).stop 2023-10-16T11:19:39.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 shutdown_connections 2023-10-16T11:19:39.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fa04806b020 0x7fa04806d4e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa06415df60 0x7fa0640b2fc0 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa050003a00 0x7fa0640ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 --2- 172.21.15.158:0/2582092621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa0641f2940 0x7fa0641599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 >> 172.21.15.158:0/2582092621 conn(0x7fa064099a10 msgr2=0x7fa0640b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:39.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 shutdown_connections 2023-10-16T11:19:39.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.695+0000 7fa06bed5700 1 -- 172.21.15.158:0/2582092621 wait complete. 2023-10-16T11:19:39.710 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T11:19:39.711 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:39.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.831+0000 7f1f66a24700 1 Processor -- start 2023-10-16T11:19:39.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.831+0000 7f1f66a24700 1 -- start start 2023-10-16T11:19:39.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f66a24700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cc5d0 0x7f1f6006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f66a24700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f6006f340 0x7f1f6006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f66a24700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6006fce0 con 0x7f1f6015dfb0 2023-10-16T11:19:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f66a24700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f600cf270 con 0x7f1f6006f340 2023-10-16T11:19:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f66a24700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6006e770 con 0x7f1f600cc5d0 2023-10-16T11:19:39.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f64fc1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f6006f340 0x7f1f6006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f5f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1f6015dfb0 0x7f1f600cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38580/0 (socket says 172.21.15.158:38580) 2023-10-16T11:19:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f5f7fe700 1 -- 172.21.15.158:0/2645959428 learned_addr learned my addr 172.21.15.158:0/2645959428 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f5ffff700 1 --2- 172.21.15.158:0/2645959428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cc5d0 0x7f1f6006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f64fc1700 1 -- 172.21.15.158:0/2645959428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cc5d0 msgr2=0x7f1f6006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:39.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f64fc1700 1 --2- 172.21.15.158:0/2645959428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cc5d0 0x7f1f6006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f64fc1700 1 -- 172.21.15.158:0/2645959428 >> v1:172.21.15.158:6789/0 conn(0x7f1f6015dfb0 legacy=0x7f1f600cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f5effd700 1 -- 172.21.15.158:0/2645959428 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631979070 0 0) 0x7f1f6006fce0 con 0x7f1f6015dfb0 2023-10-16T11:19:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f64fc1700 1 -- 172.21.15.158:0/2645959428 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600cbe00 con 0x7f1f6006f340 2023-10-16T11:19:39.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f64fc1700 1 --2- 172.21.15.158:0/2645959428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6006f340 0x7f1f6006f7a0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009d90 tx=0x7f1f5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0cc71b2395125d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.832+0000 7f1f5effd700 1 -- 172.21.15.158:0/2645959428 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5800e070 con 0x7f1f6006f340 2023-10-16T11:19:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f5effd700 1 -- 172.21.15.158:0/2645959428 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5800bbd0 con 0x7f1f6006f340 2023-10-16T11:19:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f5effd700 1 -- 172.21.15.158:0/2645959428 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58012430 con 0x7f1f6006f340 2023-10-16T11:19:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 -- 172.21.15.158:0/2645959428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6006f340 msgr2=0x7f1f6006f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:39.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 --2- 172.21.15.158:0/2645959428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6006f340 0x7f1f6006f7a0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009d90 tx=0x7f1f5800b330 comp rx=0 tx=0).stop 2023-10-16T11:19:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 -- 172.21.15.158:0/2645959428 shutdown_connections 2023-10-16T11:19:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 --2- 172.21.15.158:0/2645959428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6006f340 0x7f1f6006f7a0 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 --2- 172.21.15.158:0/2645959428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f600cc5d0 0x7f1f6006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 -- 172.21.15.158:0/2645959428 >> 172.21.15.158:0/2645959428 conn(0x7f1f600d02b0 msgr2=0x7f1f600d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:39.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 -- 172.21.15.158:0/2645959428 shutdown_connections 2023-10-16T11:19:39.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.833+0000 7f1f66a24700 1 -- 172.21.15.158:0/2645959428 wait complete. 2023-10-16T11:19:39.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.834+0000 7f1f66a24700 1 Processor -- start 2023-10-16T11:19:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f66a24700 1 -- start start 2023-10-16T11:19:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f66a24700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f6006f340 0x7f1f60159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f66a24700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600cc5d0 0x7f1f6006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:39.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f66a24700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f6015dfb0 0x7f1f6006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f66a24700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f601f2a60 con 0x7f1f600cc5d0 2023-10-16T11:19:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f66a24700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600c53e0 con 0x7f1f6006f340 2023-10-16T11:19:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f66a24700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600c56c0 con 0x7f1f6015dfb0 2023-10-16T11:19:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f5f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600cc5d0 0x7f1f6006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:39.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f5f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600cc5d0 0x7f1f6006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38412/0 (socket says 172.21.15.158:38412) 2023-10-16T11:19:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.835+0000 7f1f5f7fe700 1 -- 172.21.15.158:0/3955900807 learned_addr learned my addr 172.21.15.158:0/3955900807 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f64fc1700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f6015dfb0 0x7f1f6006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5ffff700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f6006f340 0x7f1f60159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5f7fe700 1 -- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f6015dfb0 msgr2=0x7f1f6006ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:39.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5f7fe700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f6015dfb0 0x7f1f6006ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5f7fe700 1 -- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f6006f340 msgr2=0x7f1f60159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5f7fe700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f6006f340 0x7f1f60159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5f7fe700 1 -- 172.21.15.158:0/3955900807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58009a70 con 0x7f1f600cc5d0 2023-10-16T11:19:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5f7fe700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600cc5d0 0x7f1f6006a8d0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400c540 tx=0x7f1f5400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:39.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5cff9700 1 -- 172.21.15.158:0/3955900807 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54004c80 con 0x7f1f600cc5d0 2023-10-16T11:19:39.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5cff9700 1 -- 172.21.15.158:0/3955900807 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f54004de0 con 0x7f1f600cc5d0 2023-10-16T11:19:39.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f600c59a0 con 0x7f1f600cc5d0 2023-10-16T11:19:39.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f5cff9700 1 -- 172.21.15.158:0/3955900807 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54007310 con 0x7f1f600cc5d0 2023-10-16T11:19:39.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.836+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f600c5c80 con 0x7f1f600cc5d0 2023-10-16T11:19:39.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.837+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f600681b0 con 0x7f1f600cc5d0 2023-10-16T11:19:39.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.838+0000 7f1f5cff9700 1 -- 172.21.15.158:0/3955900807 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f54007470 con 0x7f1f600cc5d0 2023-10-16T11:19:39.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.838+0000 7f1f5cff9700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f1f4c06b020 0x7f1f4c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:39.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.838+0000 7f1f5cff9700 1 -- 172.21.15.158:0/3955900807 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f1f54089bf0 con 0x7f1f600cc5d0 2023-10-16T11:19:39.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.838+0000 7f1f5ffff700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f1f4c06b020 0x7f1f4c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:39.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.838+0000 7f1f5ffff700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f1f4c06b020 0x7f1f4c06d4e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000b9b0 tx=0x7f1f50008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:39.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:39.842+0000 7f1f5cff9700 1 -- 172.21.15.158:0/3955900807 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f54054480 con 0x7f1f600cc5d0 2023-10-16T11:19:40.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.077+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f600b6230 con 0x7f1f600cc5d0 2023-10-16T11:19:40.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.078+0000 7f1f5cff9700 1 -- 172.21.15.158:0/3955900807 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6163 (secure 0 0 0) 0x7f1f54057aa0 con 0x7f1f600cc5d0 2023-10-16T11:19:40.080 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:40.080 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:19:40.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f1f4c06b020 msgr2=0x7f1f4c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f1f4c06b020 0x7f1f4c06d4e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000b9b0 tx=0x7f1f50008040 comp rx=0 tx=0).stop 2023-10-16T11:19:40.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600cc5d0 msgr2=0x7f1f6006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600cc5d0 0x7f1f6006a8d0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400c540 tx=0x7f1f5400c900 comp rx=0 tx=0).stop 2023-10-16T11:19:40.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 shutdown_connections 2023-10-16T11:19:40.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f6015dfb0 0x7f1f6006ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f6006f340 0x7f1f60159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f1f4c06b020 0x7f1f4c06d4e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 --2- 172.21.15.158:0/3955900807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f600cc5d0 0x7f1f6006a8d0 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 >> 172.21.15.158:0/3955900807 conn(0x7f1f600d02b0 msgr2=0x7f1f600c9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:40.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 shutdown_connections 2023-10-16T11:19:40.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.080+0000 7f1f66a24700 1 -- 172.21.15.158:0/3955900807 wait complete. 2023-10-16T11:19:40.085 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 118 2023-10-16T11:19:40.094 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:40.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.213+0000 7fb4a90e4700 1 Processor -- start 2023-10-16T11:19:40.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.213+0000 7fb4a90e4700 1 -- start start 2023-10-16T11:19:40.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.214+0000 7fb4a90e4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb4a406e5d0 0x7fb4a406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.214+0000 7fb4a90e4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a4161270 con 0x7fb4a406ab40 2023-10-16T11:19:40.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.214+0000 7fb4a90e4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a4161510 con 0x7fb4a40729b0 2023-10-16T11:19:40.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.214+0000 7fb4a90e4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a41617f0 con 0x7fb4a406e5d0 2023-10-16T11:19:40.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.214+0000 7fb4a259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb4a406e5d0 0x7fb4a406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.214+0000 7fb4a2d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb4a406ab40 0x7fb4a40722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38594/0 (socket says 172.21.15.158:38594) 2023-10-16T11:19:40.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.214+0000 7fb4a2d9d700 1 -- 172.21.15.158:0/697413447 learned_addr learned my addr 172.21.15.158:0/697413447 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:40.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a259c700 1 -- 172.21.15.158:0/697413447 >> v1:172.21.15.172:6789/0 conn(0x7fb4a40729b0 legacy=0x7fb4a415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:40.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a259c700 1 -- 172.21.15.158:0/697413447 >> v1:172.21.15.158:6789/0 conn(0x7fb4a406ab40 legacy=0x7fb4a40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a259c700 1 -- 172.21.15.158:0/697413447 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a4161ad0 con 0x7fb4a406e5d0 2023-10-16T11:19:40.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a259c700 1 --2- 172.21.15.158:0/697413447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a406d820 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fb494009840 tx=0x7fb494009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f1e2c6c0fcd8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a1d9b700 1 -- 172.21.15.158:0/697413447 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494004230 con 0x7fb4a406e5d0 2023-10-16T11:19:40.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a1d9b700 1 -- 172.21.15.158:0/697413447 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb494004390 con 0x7fb4a406e5d0 2023-10-16T11:19:40.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a1d9b700 1 -- 172.21.15.158:0/697413447 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4940036a0 con 0x7fb4a406e5d0 2023-10-16T11:19:40.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a90e4700 1 -- 172.21.15.158:0/697413447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 msgr2=0x7fb4a406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/697413447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a406d820 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fb494009840 tx=0x7fb494009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:40.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a90e4700 1 -- 172.21.15.158:0/697413447 shutdown_connections 2023-10-16T11:19:40.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/697413447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a406d820 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.215+0000 7fb4a90e4700 1 -- 172.21.15.158:0/697413447 >> 172.21.15.158:0/697413447 conn(0x7fb4a40d02b0 msgr2=0x7fb4a40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:40.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.216+0000 7fb4a90e4700 1 -- 172.21.15.158:0/697413447 shutdown_connections 2023-10-16T11:19:40.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.216+0000 7fb4a90e4700 1 -- 172.21.15.158:0/697413447 wait complete. 2023-10-16T11:19:40.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.217+0000 7fb4a90e4700 1 Processor -- start 2023-10-16T11:19:40.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.217+0000 7fb4a90e4700 1 -- start start 2023-10-16T11:19:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a90e4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a406ab40 0x7fb4a40c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a90e4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a40b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a90e4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4a40729b0 0x7fb4a40c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a90e4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40b0c90 con 0x7fb4a406e5d0 2023-10-16T11:19:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a90e4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40b0f30 con 0x7fb4a406ab40 2023-10-16T11:19:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a90e4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40b1210 con 0x7fb4a40729b0 2023-10-16T11:19:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a40b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a40b1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38426/0 (socket says 172.21.15.158:38426) 2023-10-16T11:19:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 -- 172.21.15.158:0/1497618933 learned_addr learned my addr 172.21.15.158:0/1497618933 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:40.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a359e700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4a40729b0 0x7fb4a40c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a2d9d700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a406ab40 0x7fb4a40c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 -- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4a40729b0 msgr2=0x7fb4a40c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4a40729b0 0x7fb4a40c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 -- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a406ab40 msgr2=0x7fb4a40c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a406ab40 0x7fb4a40c6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 -- 172.21.15.158:0/1497618933 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4940094a0 con 0x7fb4a406e5d0 2023-10-16T11:19:40.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a359e700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4a40729b0 0x7fb4a40c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:40.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a2d9d700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a406ab40 0x7fb4a40c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:40.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a259c700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a40b1cf0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fb4940090b0 tx=0x7fb494004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb48b7fe700 1 -- 172.21.15.158:0/1497618933 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494011830 con 0x7fb4a406e5d0 2023-10-16T11:19:40.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb48b7fe700 1 -- 172.21.15.158:0/1497618933 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49401e070 con 0x7fb4a406e5d0 2023-10-16T11:19:40.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.218+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a40b14f0 con 0x7fb4a406e5d0 2023-10-16T11:19:40.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.219+0000 7fb48b7fe700 1 -- 172.21.15.158:0/1497618933 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494011bf0 con 0x7fb4a406e5d0 2023-10-16T11:19:40.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.219+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a40c6750 con 0x7fb4a406e5d0 2023-10-16T11:19:40.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.219+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a40b1a10 con 0x7fb4a406e5d0 2023-10-16T11:19:40.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.220+0000 7fb48b7fe700 1 -- 172.21.15.158:0/1497618933 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb494011d50 con 0x7fb4a406e5d0 2023-10-16T11:19:40.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.220+0000 7fb48b7fe700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb49006b020 0x7fb49006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.220+0000 7fb48b7fe700 1 -- 172.21.15.158:0/1497618933 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fb49408b0b0 con 0x7fb4a406e5d0 2023-10-16T11:19:40.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.220+0000 7fb4a2d9d700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb49006b020 0x7fb49006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.221+0000 7fb4a2d9d700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb49006b020 0x7fb49006d4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb498005910 tx=0x7fb498009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.225+0000 7fb48b7fe700 1 -- 172.21.15.158:0/1497618933 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4a40b1a10 con 0x7fb4a406e5d0 2023-10-16T11:19:40.307 INFO:tasks.ceph.mds.d.smithi172.stderr:2023-10-16T11:19:40.306+0000 7fcfe2aca700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:19:40.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.422+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4a40b1a10 con 0x7fb4a406e5d0 2023-10-16T11:19:40.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.424+0000 7fb48b7fe700 1 -- 172.21.15.158:0/1497618933 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) v1 ==== 74+0+21187 (secure 0 0 0) 0x7fb4a40b1a10 con 0x7fb4a406e5d0 2023-10-16T11:19:40.426 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:40.426 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":221,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:37.659421+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb49006b020 msgr2=0x7fb49006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb49006b020 0x7fb49006d4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb498005910 tx=0x7fb498009040 comp rx=0 tx=0).stop 2023-10-16T11:19:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 msgr2=0x7fb4a40b1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a40b1cf0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fb4940090b0 tx=0x7fb494004750 comp rx=0 tx=0).stop 2023-10-16T11:19:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 shutdown_connections 2023-10-16T11:19:40.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4a40729b0 0x7fb4a40c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a406ab40 0x7fb4a40c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb49006b020 0x7fb49006d4e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 --2- 172.21.15.158:0/1497618933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4a406e5d0 0x7fb4a40b1cf0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 >> 172.21.15.158:0/1497618933 conn(0x7fb4a40d02b0 msgr2=0x7fb4a40aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:40.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 shutdown_connections 2023-10-16T11:19:40.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.426+0000 7fb4a90e4700 1 -- 172.21.15.158:0/1497618933 wait complete. 2023-10-16T11:19:40.440 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:19:40.441 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:19:40.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.559+0000 7f9310329700 1 Processor -- start 2023-10-16T11:19:40.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.559+0000 7f9310329700 1 -- start start 2023-10-16T11:19:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.560+0000 7f9310329700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f930806e5d0 0x7f930806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.560+0000 7f9310329700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93080729b0 0x7f9308071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.560+0000 7f9310329700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9308071e50 con 0x7f930806ab40 2023-10-16T11:19:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.560+0000 7f9310329700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9308161270 con 0x7f93080729b0 2023-10-16T11:19:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.560+0000 7f9310329700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9308161510 con 0x7f930806e5d0 2023-10-16T11:19:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f930806e5d0 0x7f930806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930e8c6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93080729b0 0x7f9308071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f930806e5d0 0x7f930806d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:49268/0 (socket says 172.21.15.158:49268) 2023-10-16T11:19:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930e0c5700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f930806ab40 0x7f93080722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:42512/0 (socket says 172.21.15.158:42512) 2023-10-16T11:19:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930e8c6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93080729b0 0x7f9308071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:39616/0 (socket says 172.21.15.158:39616) 2023-10-16T11:19:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 -- 172.21.15.158:0/3280918839 learned_addr learned my addr 172.21.15.158:0/3280918839 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 -- 172.21.15.158:0/3280918839 >> v2:172.21.15.172:3300/0 conn(0x7f93080729b0 msgr2=0x7f9308071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:40.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 --2- 172.21.15.158:0/3280918839 >> v2:172.21.15.172:3300/0 conn(0x7f93080729b0 0x7f9308071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 -- 172.21.15.158:0/3280918839 >> v1:172.21.15.172:6789/0 conn(0x7f930806ab40 legacy=0x7f93080722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d0c3700 1 -- 172.21.15.158:0/3280918839 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805329943 0 0) 0x7f9308071e50 con 0x7f930806ab40 2023-10-16T11:19:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 -- 172.21.15.158:0/3280918839 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93081617f0 con 0x7f930806e5d0 2023-10-16T11:19:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930e8c6700 1 --2- 172.21.15.158:0/3280918839 >> v2:172.21.15.172:3300/0 conn(0x7f93080729b0 0x7f9308071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:40.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d8c4700 1 --2- 172.21.15.158:0/3280918839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f930806e5d0 0x7f930806d820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f93040098f0 tx=0x7f9304009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bc397451cf3af51 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d0c3700 1 -- 172.21.15.158:0/3280918839 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304011830 con 0x7f930806e5d0 2023-10-16T11:19:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d0c3700 1 -- 172.21.15.158:0/3280918839 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9304011990 con 0x7f930806e5d0 2023-10-16T11:19:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.561+0000 7f930d0c3700 1 -- 172.21.15.158:0/3280918839 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304011d20 con 0x7f930806e5d0 2023-10-16T11:19:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 -- 172.21.15.158:0/3280918839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f930806e5d0 msgr2=0x7f930806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 --2- 172.21.15.158:0/3280918839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f930806e5d0 0x7f930806d820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f93040098f0 tx=0x7f9304009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:40.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 -- 172.21.15.158:0/3280918839 shutdown_connections 2023-10-16T11:19:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 --2- 172.21.15.158:0/3280918839 >> v2:172.21.15.172:3300/0 conn(0x7f93080729b0 0x7f9308071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 --2- 172.21.15.158:0/3280918839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f930806e5d0 0x7f930806d820 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 -- 172.21.15.158:0/3280918839 >> 172.21.15.158:0/3280918839 conn(0x7f93080d02b0 msgr2=0x7f93080d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 -- 172.21.15.158:0/3280918839 shutdown_connections 2023-10-16T11:19:40.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.562+0000 7f9310329700 1 -- 172.21.15.158:0/3280918839 wait complete. 2023-10-16T11:19:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.563+0000 7f9310329700 1 Processor -- start 2023-10-16T11:19:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.563+0000 7f9310329700 1 -- start start 2023-10-16T11:19:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f9310329700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f930806ab40 0x7f93080c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f9310329700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f930806e5d0 0x7f93080b1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f9310329700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93080729b0 0x7f93080afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f9310329700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93080b12e0 con 0x7f930806e5d0 2023-10-16T11:19:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f9310329700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93080c6a70 con 0x7f930806ab40 2023-10-16T11:19:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f9310329700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93080c5bc0 con 0x7f93080729b0 2023-10-16T11:19:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f930d8c4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f930806e5d0 0x7f93080b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f930d8c4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f930806e5d0 0x7f93080b1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38438/0 (socket says 172.21.15.158:38438) 2023-10-16T11:19:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f930d8c4700 1 -- 172.21.15.158:0/4093024716 learned_addr learned my addr 172.21.15.158:0/4093024716 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f930e0c5700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f930806ab40 0x7f93080c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.564+0000 7f930e8c6700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93080729b0 0x7f93080afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f930d8c4700 1 -- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93080729b0 msgr2=0x7f93080afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f930d8c4700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93080729b0 0x7f93080afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f930d8c4700 1 -- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f930806ab40 msgr2=0x7f93080c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f930d8c4700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f930806ab40 0x7f93080c6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f930d8c4700 1 -- 172.21.15.158:0/4093024716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9300009a70 con 0x7f930806e5d0 2023-10-16T11:19:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f930d8c4700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f930806e5d0 0x7f93080b1770 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f9304009c40 tx=0x7f9304004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f92feffd700 1 -- 172.21.15.158:0/4093024716 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93040045d0 con 0x7f930806e5d0 2023-10-16T11:19:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f92feffd700 1 -- 172.21.15.158:0/4093024716 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930401f070 con 0x7f930806e5d0 2023-10-16T11:19:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9304009550 con 0x7f930806e5d0 2023-10-16T11:19:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f92feffd700 1 -- 172.21.15.158:0/4093024716 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93040038a0 con 0x7f930806e5d0 2023-10-16T11:19:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.565+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93080b09b0 con 0x7f930806e5d0 2023-10-16T11:19:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.566+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93080ccb30 con 0x7f930806e5d0 2023-10-16T11:19:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.566+0000 7f92feffd700 1 -- 172.21.15.158:0/4093024716 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9304003a00 con 0x7f930806e5d0 2023-10-16T11:19:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.567+0000 7f92feffd700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f92ec06b020 0x7f92ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.567+0000 7f930e0c5700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f92ec06b020 0x7f92ec06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.567+0000 7f92feffd700 1 -- 172.21.15.158:0/4093024716 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f9304021070 con 0x7f930806e5d0 2023-10-16T11:19:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.567+0000 7f930e0c5700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f92ec06b020 0x7f92ec06d4e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92f8007fd0 tx=0x7f92f8007de0 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.571+0000 7f92feffd700 1 -- 172.21.15.158:0/4093024716 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9304055f50 con 0x7f930806e5d0 2023-10-16T11:19:40.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.761+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9308161db0 con 0x7f92ec06b020 2023-10-16T11:19:40.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.774+0000 7f92feffd700 1 -- 172.21.15.158:0/4093024716 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+516631 (secure 0 0 0) 0x7f92f8001370 con 0x7f92ec06b020 2023-10-16T11:19:40.781 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:40.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f92ec06b020 msgr2=0x7f92ec06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f92ec06b020 0x7f92ec06d4e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92f8007fd0 tx=0x7f92f8007de0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f930806e5d0 msgr2=0x7f93080b1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f930806e5d0 0x7f93080b1770 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f9304009c40 tx=0x7f9304004750 comp rx=0 tx=0).stop 2023-10-16T11:19:40.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 shutdown_connections 2023-10-16T11:19:40.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93080729b0 0x7f93080afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f930806ab40 0x7f93080c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f92ec06b020 0x7f92ec06d4e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 --2- 172.21.15.158:0/4093024716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f930806e5d0 0x7f93080b1770 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 >> 172.21.15.158:0/4093024716 conn(0x7f93080d02b0 msgr2=0x7f9308072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:40.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 shutdown_connections 2023-10-16T11:19:40.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.778+0000 7f9310329700 1 -- 172.21.15.158:0/4093024716 wait complete. 2023-10-16T11:19:40.789 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:19:40.792 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-16T11:19:39.276441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594025,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":626,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392820,"kb_used_data":22608,"kb_used_omap":12,"kb_used_meta":370099,"kb_avail":754581900,"statfs":{"total":773094113280,"available":772691865600,"internally_reserved":0,"allocated":23150592,"data_stored":12907940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":378981943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1247,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.624642"},"pg_stats":[{"pgid":"1.0","version":"207'348","reported_seq":1209,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835291+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:19:24.835291+0000","last_peered":"2023-10-16T11:19:24.835291+0000","last_clean":"2023-10-16T11:19:24.835291+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:19:24.835291+0000","last_undegraded":"2023-10-16T11:19:24.835291+0000","last_fullsized":"2023-10-16T11:19:24.835291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269008+0000","last_change":"2023-10-16T11:19:22.332936+0000","last_active":"2023-10-16T11:19:25.269008+0000","last_peered":"2023-10-16T11:19:25.269008+0000","last_clean":"2023-10-16T11:19:25.269008+0000","last_became_active":"2023-10-16T11:19:22.332598+0000","last_became_peered":"2023-10-16T11:19:22.332598+0000","last_unstale":"2023-10-16T11:19:25.269008+0000","last_undegraded":"2023-10-16T11:19:25.269008+0000","last_fullsized":"2023-10-16T11:19:25.269008+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:34.597859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272622+0000","last_change":"2023-10-16T11:19:23.334041+0000","last_active":"2023-10-16T11:19:25.272622+0000","last_peered":"2023-10-16T11:19:25.272622+0000","last_clean":"2023-10-16T11:19:25.272622+0000","last_became_active":"2023-10-16T11:19:23.333752+0000","last_became_peered":"2023-10-16T11:19:23.333752+0000","last_unstale":"2023-10-16T11:19:25.272622+0000","last_undegraded":"2023-10-16T11:19:25.272622+0000","last_fullsized":"2023-10-16T11:19:25.272622+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.656805+0000","last_change":"2023-10-16T11:19:34.656805+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.656805+0000","last_undegraded":"2023-10-16T11:19:34.656805+0000","last_fullsized":"2023-10-16T11:19:34.656805+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272403+0000","last_change":"2023-10-16T11:19:22.334152+0000","last_active":"2023-10-16T11:19:25.272403+0000","last_peered":"2023-10-16T11:19:25.272403+0000","last_clean":"2023-10-16T11:19:25.272403+0000","last_became_active":"2023-10-16T11:19:22.333859+0000","last_became_peered":"2023-10-16T11:19:22.333859+0000","last_unstale":"2023-10-16T11:19:25.272403+0000","last_undegraded":"2023-10-16T11:19:25.272403+0000","last_fullsized":"2023-10-16T11:19:25.272403+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:50.619981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273222+0000","last_change":"2023-10-16T11:19:23.324999+0000","last_active":"2023-10-16T11:19:25.273222+0000","last_peered":"2023-10-16T11:19:25.273222+0000","last_clean":"2023-10-16T11:19:25.273222+0000","last_became_active":"2023-10-16T11:19:23.324686+0000","last_became_peered":"2023-10-16T11:19:23.324686+0000","last_unstale":"2023-10-16T11:19:25.273222+0000","last_undegraded":"2023-10-16T11:19:25.273222+0000","last_fullsized":"2023-10-16T11:19:25.273222+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:42.281964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657250+0000","last_change":"2023-10-16T11:19:34.657250+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657250+0000","last_undegraded":"2023-10-16T11:19:34.657250+0000","last_fullsized":"2023-10-16T11:19:34.657250+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272562+0000","last_change":"2023-10-16T11:19:22.332640+0000","last_active":"2023-10-16T11:19:25.272562+0000","last_peered":"2023-10-16T11:19:25.272562+0000","last_clean":"2023-10-16T11:19:25.272562+0000","last_became_active":"2023-10-16T11:19:22.331608+0000","last_became_peered":"2023-10-16T11:19:22.331608+0000","last_unstale":"2023-10-16T11:19:25.272562+0000","last_undegraded":"2023-10-16T11:19:25.272562+0000","last_fullsized":"2023-10-16T11:19:25.272562+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272425+0000","last_change":"2023-10-16T11:19:23.329115+0000","last_active":"2023-10-16T11:19:25.272425+0000","last_peered":"2023-10-16T11:19:25.272425+0000","last_clean":"2023-10-16T11:19:25.272425+0000","last_became_active":"2023-10-16T11:19:23.328830+0000","last_became_peered":"2023-10-16T11:19:23.328830+0000","last_unstale":"2023-10-16T11:19:25.272425+0000","last_undegraded":"2023-10-16T11:19:25.272425+0000","last_fullsized":"2023-10-16T11:19:25.272425+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:17.247488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657464+0000","last_change":"2023-10-16T11:19:34.657464+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657464+0000","last_undegraded":"2023-10-16T11:19:34.657464+0000","last_fullsized":"2023-10-16T11:19:34.657464+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269019+0000","last_change":"2023-10-16T11:19:22.329623+0000","last_active":"2023-10-16T11:19:25.269019+0000","last_peered":"2023-10-16T11:19:25.269019+0000","last_clean":"2023-10-16T11:19:25.269019+0000","last_became_active":"2023-10-16T11:19:22.329344+0000","last_became_peered":"2023-10-16T11:19:22.329344+0000","last_unstale":"2023-10-16T11:19:25.269019+0000","last_undegraded":"2023-10-16T11:19:25.269019+0000","last_fullsized":"2023-10-16T11:19:25.269019+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272405+0000","last_change":"2023-10-16T11:19:23.330624+0000","last_active":"2023-10-16T11:19:25.272405+0000","last_peered":"2023-10-16T11:19:25.272405+0000","last_clean":"2023-10-16T11:19:25.272405+0000","last_became_active":"2023-10-16T11:19:23.329588+0000","last_became_peered":"2023-10-16T11:19:23.329588+0000","last_unstale":"2023-10-16T11:19:25.272405+0000","last_undegraded":"2023-10-16T11:19:25.272405+0000","last_fullsized":"2023-10-16T11:19:25.272405+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:10.835410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.658784+0000","last_change":"2023-10-16T11:19:34.658784+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.658784+0000","last_undegraded":"2023-10-16T11:19:34.658784+0000","last_fullsized":"2023-10-16T11:19:34.658784+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272255+0000","last_change":"2023-10-16T11:19:22.333752+0000","last_active":"2023-10-16T11:19:25.272255+0000","last_peered":"2023-10-16T11:19:25.272255+0000","last_clean":"2023-10-16T11:19:25.272255+0000","last_became_active":"2023-10-16T11:19:22.333497+0000","last_became_peered":"2023-10-16T11:19:22.333497+0000","last_unstale":"2023-10-16T11:19:25.272255+0000","last_undegraded":"2023-10-16T11:19:25.272255+0000","last_fullsized":"2023-10-16T11:19:25.272255+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272280+0000","last_change":"2023-10-16T11:19:23.329220+0000","last_active":"2023-10-16T11:19:25.272280+0000","last_peered":"2023-10-16T11:19:25.272280+0000","last_clean":"2023-10-16T11:19:25.272280+0000","last_became_active":"2023-10-16T11:19:23.328923+0000","last_became_peered":"2023-10-16T11:19:23.328923+0000","last_unstale":"2023-10-16T11:19:25.272280+0000","last_undegraded":"2023-10-16T11:19:25.272280+0000","last_fullsized":"2023-10-16T11:19:25.272280+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657189+0000","last_change":"2023-10-16T11:19:34.657189+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657189+0000","last_undegraded":"2023-10-16T11:19:34.657189+0000","last_fullsized":"2023-10-16T11:19:34.657189+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272600+0000","last_change":"2023-10-16T11:19:22.335923+0000","last_active":"2023-10-16T11:19:25.272600+0000","last_peered":"2023-10-16T11:19:25.272600+0000","last_clean":"2023-10-16T11:19:25.272600+0000","last_became_active":"2023-10-16T11:19:22.335705+0000","last_became_peered":"2023-10-16T11:19:22.335705+0000","last_unstale":"2023-10-16T11:19:25.272600+0000","last_undegraded":"2023-10-16T11:19:25.272600+0000","last_fullsized":"2023-10-16T11:19:25.272600+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:40.803363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272586+0000","last_change":"2023-10-16T11:19:23.336160+0000","last_active":"2023-10-16T11:19:25.272586+0000","last_peered":"2023-10-16T11:19:25.272586+0000","last_clean":"2023-10-16T11:19:25.272586+0000","last_became_active":"2023-10-16T11:19:23.333920+0000","last_became_peered":"2023-10-16T11:19:23.333920+0000","last_unstale":"2023-10-16T11:19:25.272586+0000","last_undegraded":"2023-10-16T11:19:25.272586+0000","last_fullsized":"2023-10-16T11:19:25.272586+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654775+0000","last_change":"2023-10-16T11:19:34.654775+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654775+0000","last_undegraded":"2023-10-16T11:19:34.654775+0000","last_fullsized":"2023-10-16T11:19:34.654775+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273044+0000","last_change":"2023-10-16T11:19:22.326754+0000","last_active":"2023-10-16T11:19:25.273044+0000","last_peered":"2023-10-16T11:19:25.273044+0000","last_clean":"2023-10-16T11:19:25.273044+0000","last_became_active":"2023-10-16T11:19:22.326480+0000","last_became_peered":"2023-10-16T11:19:22.326480+0000","last_unstale":"2023-10-16T11:19:25.273044+0000","last_undegraded":"2023-10-16T11:19:25.273044+0000","last_fullsized":"2023-10-16T11:19:25.273044+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.268994+0000","last_change":"2023-10-16T11:19:23.331571+0000","last_active":"2023-10-16T11:19:25.268994+0000","last_peered":"2023-10-16T11:19:25.268994+0000","last_clean":"2023-10-16T11:19:25.268994+0000","last_became_active":"2023-10-16T11:19:23.331242+0000","last_became_peered":"2023-10-16T11:19:23.331242+0000","last_unstale":"2023-10-16T11:19:25.268994+0000","last_undegraded":"2023-10-16T11:19:25.268994+0000","last_fullsized":"2023-10-16T11:19:25.268994+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.661222+0000","last_change":"2023-10-16T11:19:34.661222+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.661222+0000","last_undegraded":"2023-10-16T11:19:34.661222+0000","last_fullsized":"2023-10-16T11:19:34.661222+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"217'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:28.410245+0000","last_change":"2023-10-16T11:19:22.331182+0000","last_active":"2023-10-16T11:19:28.410245+0000","last_peered":"2023-10-16T11:19:28.410245+0000","last_clean":"2023-10-16T11:19:28.410245+0000","last_became_active":"2023-10-16T11:19:22.330860+0000","last_became_peered":"2023-10-16T11:19:22.330860+0000","last_unstale":"2023-10-16T11:19:28.410245+0000","last_undegraded":"2023-10-16T11:19:28.410245+0000","last_fullsized":"2023-10-16T11:19:28.410245+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273111+0000","last_change":"2023-10-16T11:19:23.325482+0000","last_active":"2023-10-16T11:19:25.273111+0000","last_peered":"2023-10-16T11:19:25.273111+0000","last_clean":"2023-10-16T11:19:25.273111+0000","last_became_active":"2023-10-16T11:19:23.325193+0000","last_became_peered":"2023-10-16T11:19:23.325193+0000","last_unstale":"2023-10-16T11:19:25.273111+0000","last_undegraded":"2023-10-16T11:19:25.273111+0000","last_fullsized":"2023-10-16T11:19:25.273111+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.191635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657303+0000","last_change":"2023-10-16T11:19:34.657303+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657303+0000","last_undegraded":"2023-10-16T11:19:34.657303+0000","last_fullsized":"2023-10-16T11:19:34.657303+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269133+0000","last_change":"2023-10-16T11:19:22.332950+0000","last_active":"2023-10-16T11:19:25.269133+0000","last_peered":"2023-10-16T11:19:25.269133+0000","last_clean":"2023-10-16T11:19:25.269133+0000","last_became_active":"2023-10-16T11:19:22.332650+0000","last_became_peered":"2023-10-16T11:19:22.332650+0000","last_unstale":"2023-10-16T11:19:25.269133+0000","last_undegraded":"2023-10-16T11:19:25.269133+0000","last_fullsized":"2023-10-16T11:19:25.269133+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272136+0000","last_change":"2023-10-16T11:19:23.333192+0000","last_active":"2023-10-16T11:19:25.272136+0000","last_peered":"2023-10-16T11:19:25.272136+0000","last_clean":"2023-10-16T11:19:25.272136+0000","last_became_active":"2023-10-16T11:19:23.332739+0000","last_became_peered":"2023-10-16T11:19:23.332739+0000","last_unstale":"2023-10-16T11:19:25.272136+0000","last_undegraded":"2023-10-16T11:19:25.272136+0000","last_fullsized":"2023-10-16T11:19:25.272136+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.655371+0000","last_change":"2023-10-16T11:19:34.655371+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.655371+0000","last_undegraded":"2023-10-16T11:19:34.655371+0000","last_fullsized":"2023-10-16T11:19:34.655371+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272508+0000","last_change":"2023-10-16T11:19:22.326294+0000","last_active":"2023-10-16T11:19:25.272508+0000","last_peered":"2023-10-16T11:19:25.272508+0000","last_clean":"2023-10-16T11:19:25.272508+0000","last_became_active":"2023-10-16T11:19:22.326012+0000","last_became_peered":"2023-10-16T11:19:22.326012+0000","last_unstale":"2023-10-16T11:19:25.272508+0000","last_undegraded":"2023-10-16T11:19:25.272508+0000","last_fullsized":"2023-10-16T11:19:25.272508+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272333+0000","last_change":"2023-10-16T11:19:23.336920+0000","last_active":"2023-10-16T11:19:25.272333+0000","last_peered":"2023-10-16T11:19:25.272333+0000","last_clean":"2023-10-16T11:19:25.272333+0000","last_became_active":"2023-10-16T11:19:23.336473+0000","last_became_peered":"2023-10-16T11:19:23.336473+0000","last_unstale":"2023-10-16T11:19:25.272333+0000","last_undegraded":"2023-10-16T11:19:25.272333+0000","last_fullsized":"2023-10-16T11:19:25.272333+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:22.421887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659575+0000","last_change":"2023-10-16T11:19:34.659575+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659575+0000","last_undegraded":"2023-10-16T11:19:34.659575+0000","last_fullsized":"2023-10-16T11:19:34.659575+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273005+0000","last_change":"2023-10-16T11:19:22.331457+0000","last_active":"2023-10-16T11:19:25.273005+0000","last_peered":"2023-10-16T11:19:25.273005+0000","last_clean":"2023-10-16T11:19:25.273005+0000","last_became_active":"2023-10-16T11:19:22.331010+0000","last_became_peered":"2023-10-16T11:19:22.331010+0000","last_unstale":"2023-10-16T11:19:25.273005+0000","last_undegraded":"2023-10-16T11:19:25.273005+0000","last_fullsized":"2023-10-16T11:19:25.273005+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:38.048633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269085+0000","last_change":"2023-10-16T11:19:23.333035+0000","last_active":"2023-10-16T11:19:25.269085+0000","last_peered":"2023-10-16T11:19:25.269085+0000","last_clean":"2023-10-16T11:19:25.269085+0000","last_became_active":"2023-10-16T11:19:23.332692+0000","last_became_peered":"2023-10-16T11:19:23.332692+0000","last_unstale":"2023-10-16T11:19:25.269085+0000","last_undegraded":"2023-10-16T11:19:25.269085+0000","last_fullsized":"2023-10-16T11:19:25.269085+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654916+0000","last_change":"2023-10-16T11:19:34.654916+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654916+0000","last_undegraded":"2023-10-16T11:19:34.654916+0000","last_fullsized":"2023-10-16T11:19:34.654916+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272667+0000","last_change":"2023-10-16T11:19:22.329858+0000","last_active":"2023-10-16T11:19:25.272667+0000","last_peered":"2023-10-16T11:19:25.272667+0000","last_clean":"2023-10-16T11:19:25.272667+0000","last_became_active":"2023-10-16T11:19:22.329586+0000","last_became_peered":"2023-10-16T11:19:22.329586+0000","last_unstale":"2023-10-16T11:19:25.272667+0000","last_undegraded":"2023-10-16T11:19:25.272667+0000","last_fullsized":"2023-10-16T11:19:25.272667+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:25.906324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272383+0000","last_change":"2023-10-16T11:19:23.333742+0000","last_active":"2023-10-16T11:19:25.272383+0000","last_peered":"2023-10-16T11:19:25.272383+0000","last_clean":"2023-10-16T11:19:25.272383+0000","last_became_active":"2023-10-16T11:19:23.333424+0000","last_became_peered":"2023-10-16T11:19:23.333424+0000","last_unstale":"2023-10-16T11:19:25.272383+0000","last_undegraded":"2023-10-16T11:19:25.272383+0000","last_fullsized":"2023-10-16T11:19:25.272383+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.156569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.658120+0000","last_change":"2023-10-16T11:19:34.658120+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.658120+0000","last_undegraded":"2023-10-16T11:19:34.658120+0000","last_fullsized":"2023-10-16T11:19:34.658120+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272502+0000","last_change":"2023-10-16T11:19:22.329655+0000","last_active":"2023-10-16T11:19:25.272502+0000","last_peered":"2023-10-16T11:19:25.272502+0000","last_clean":"2023-10-16T11:19:25.272502+0000","last_became_active":"2023-10-16T11:19:22.329366+0000","last_became_peered":"2023-10-16T11:19:22.329366+0000","last_unstale":"2023-10-16T11:19:25.272502+0000","last_undegraded":"2023-10-16T11:19:25.272502+0000","last_fullsized":"2023-10-16T11:19:25.272502+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:47.331154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272714+0000","last_change":"2023-10-16T11:19:23.331460+0000","last_active":"2023-10-16T11:19:25.272714+0000","last_peered":"2023-10-16T11:19:25.272714+0000","last_clean":"2023-10-16T11:19:25.272714+0000","last_became_active":"2023-10-16T11:19:23.331178+0000","last_became_peered":"2023-10-16T11:19:23.331178+0000","last_unstale":"2023-10-16T11:19:25.272714+0000","last_undegraded":"2023-10-16T11:19:25.272714+0000","last_fullsized":"2023-10-16T11:19:25.272714+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:36.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659234+0000","last_change":"2023-10-16T11:19:34.659234+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659234+0000","last_undegraded":"2023-10-16T11:19:34.659234+0000","last_fullsized":"2023-10-16T11:19:34.659234+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272555+0000","last_change":"2023-10-16T11:19:22.332355+0000","last_active":"2023-10-16T11:19:25.272555+0000","last_peered":"2023-10-16T11:19:25.272555+0000","last_clean":"2023-10-16T11:19:25.272555+0000","last_became_active":"2023-10-16T11:19:22.331520+0000","last_became_peered":"2023-10-16T11:19:22.331520+0000","last_unstale":"2023-10-16T11:19:25.272555+0000","last_undegraded":"2023-10-16T11:19:25.272555+0000","last_fullsized":"2023-10-16T11:19:25.272555+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:10.682623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272235+0000","last_change":"2023-10-16T11:19:23.331171+0000","last_active":"2023-10-16T11:19:25.272235+0000","last_peered":"2023-10-16T11:19:25.272235+0000","last_clean":"2023-10-16T11:19:25.272235+0000","last_became_active":"2023-10-16T11:19:23.330836+0000","last_became_peered":"2023-10-16T11:19:23.330836+0000","last_unstale":"2023-10-16T11:19:25.272235+0000","last_undegraded":"2023-10-16T11:19:25.272235+0000","last_fullsized":"2023-10-16T11:19:25.272235+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:45.818409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657687+0000","last_change":"2023-10-16T11:19:34.657687+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657687+0000","last_undegraded":"2023-10-16T11:19:34.657687+0000","last_fullsized":"2023-10-16T11:19:34.657687+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272738+0000","last_change":"2023-10-16T11:19:22.318314+0000","last_active":"2023-10-16T11:19:25.272738+0000","last_peered":"2023-10-16T11:19:25.272738+0000","last_clean":"2023-10-16T11:19:25.272738+0000","last_became_active":"2023-10-16T11:19:22.318015+0000","last_became_peered":"2023-10-16T11:19:22.318015+0000","last_unstale":"2023-10-16T11:19:25.272738+0000","last_undegraded":"2023-10-16T11:19:25.272738+0000","last_fullsized":"2023-10-16T11:19:25.272738+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.421816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272573+0000","last_change":"2023-10-16T11:19:23.337274+0000","last_active":"2023-10-16T11:19:25.272573+0000","last_peered":"2023-10-16T11:19:25.272573+0000","last_clean":"2023-10-16T11:19:25.272573+0000","last_became_active":"2023-10-16T11:19:23.336889+0000","last_became_peered":"2023-10-16T11:19:23.336889+0000","last_unstale":"2023-10-16T11:19:25.272573+0000","last_undegraded":"2023-10-16T11:19:25.272573+0000","last_fullsized":"2023-10-16T11:19:25.272573+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.658407+0000","last_change":"2023-10-16T11:19:34.658407+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.658407+0000","last_undegraded":"2023-10-16T11:19:34.658407+0000","last_fullsized":"2023-10-16T11:19:34.658407+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272663+0000","last_change":"2023-10-16T11:19:22.330569+0000","last_active":"2023-10-16T11:19:25.272663+0000","last_peered":"2023-10-16T11:19:25.272663+0000","last_clean":"2023-10-16T11:19:25.272663+0000","last_became_active":"2023-10-16T11:19:22.329740+0000","last_became_peered":"2023-10-16T11:19:22.329740+0000","last_unstale":"2023-10-16T11:19:25.272663+0000","last_undegraded":"2023-10-16T11:19:25.272663+0000","last_fullsized":"2023-10-16T11:19:25.272663+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272537+0000","last_change":"2023-10-16T11:19:23.338701+0000","last_active":"2023-10-16T11:19:25.272537+0000","last_peered":"2023-10-16T11:19:25.272537+0000","last_clean":"2023-10-16T11:19:25.272537+0000","last_became_active":"2023-10-16T11:19:23.334033+0000","last_became_peered":"2023-10-16T11:19:23.334033+0000","last_unstale":"2023-10-16T11:19:25.272537+0000","last_undegraded":"2023-10-16T11:19:25.272537+0000","last_fullsized":"2023-10-16T11:19:25.272537+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659360+0000","last_change":"2023-10-16T11:19:34.659360+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659360+0000","last_undegraded":"2023-10-16T11:19:34.659360+0000","last_fullsized":"2023-10-16T11:19:34.659360+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272687+0000","last_change":"2023-10-16T11:19:22.331432+0000","last_active":"2023-10-16T11:19:25.272687+0000","last_peered":"2023-10-16T11:19:25.272687+0000","last_clean":"2023-10-16T11:19:25.272687+0000","last_became_active":"2023-10-16T11:19:22.331124+0000","last_became_peered":"2023-10-16T11:19:22.331124+0000","last_unstale":"2023-10-16T11:19:25.272687+0000","last_undegraded":"2023-10-16T11:19:25.272687+0000","last_fullsized":"2023-10-16T11:19:25.272687+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272120+0000","last_change":"2023-10-16T11:19:23.333018+0000","last_active":"2023-10-16T11:19:25.272120+0000","last_peered":"2023-10-16T11:19:25.272120+0000","last_clean":"2023-10-16T11:19:25.272120+0000","last_became_active":"2023-10-16T11:19:23.332725+0000","last_became_peered":"2023-10-16T11:19:23.332725+0000","last_unstale":"2023-10-16T11:19:25.272120+0000","last_undegraded":"2023-10-16T11:19:25.272120+0000","last_fullsized":"2023-10-16T11:19:25.272120+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:39.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657167+0000","last_change":"2023-10-16T11:19:34.657167+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657167+0000","last_undegraded":"2023-10-16T11:19:34.657167+0000","last_fullsized":"2023-10-16T11:19:34.657167+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"216'8","reported_seq":21,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272378+0000","last_change":"2023-10-16T11:19:22.332156+0000","last_active":"2023-10-16T11:19:25.272378+0000","last_peered":"2023-10-16T11:19:25.272378+0000","last_clean":"2023-10-16T11:19:25.272378+0000","last_became_active":"2023-10-16T11:19:22.331863+0000","last_became_peered":"2023-10-16T11:19:22.331863+0000","last_unstale":"2023-10-16T11:19:25.272378+0000","last_undegraded":"2023-10-16T11:19:25.272378+0000","last_fullsized":"2023-10-16T11:19:25.272378+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:36.175530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272464+0000","last_change":"2023-10-16T11:19:23.336992+0000","last_active":"2023-10-16T11:19:25.272464+0000","last_peered":"2023-10-16T11:19:25.272464+0000","last_clean":"2023-10-16T11:19:25.272464+0000","last_became_active":"2023-10-16T11:19:23.336724+0000","last_became_peered":"2023-10-16T11:19:23.336724+0000","last_unstale":"2023-10-16T11:19:25.272464+0000","last_undegraded":"2023-10-16T11:19:25.272464+0000","last_fullsized":"2023-10-16T11:19:25.272464+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:01.326738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.655139+0000","last_change":"2023-10-16T11:19:34.655139+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.655139+0000","last_undegraded":"2023-10-16T11:19:34.655139+0000","last_fullsized":"2023-10-16T11:19:34.655139+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.268964+0000","last_change":"2023-10-16T11:19:22.331044+0000","last_active":"2023-10-16T11:19:25.268964+0000","last_peered":"2023-10-16T11:19:25.268964+0000","last_clean":"2023-10-16T11:19:25.268964+0000","last_became_active":"2023-10-16T11:19:22.330757+0000","last_became_peered":"2023-10-16T11:19:22.330757+0000","last_unstale":"2023-10-16T11:19:25.268964+0000","last_undegraded":"2023-10-16T11:19:25.268964+0000","last_fullsized":"2023-10-16T11:19:25.268964+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269076+0000","last_change":"2023-10-16T11:19:23.333500+0000","last_active":"2023-10-16T11:19:25.269076+0000","last_peered":"2023-10-16T11:19:25.269076+0000","last_clean":"2023-10-16T11:19:25.269076+0000","last_became_active":"2023-10-16T11:19:23.333215+0000","last_became_peered":"2023-10-16T11:19:23.333215+0000","last_unstale":"2023-10-16T11:19:25.269076+0000","last_undegraded":"2023-10-16T11:19:25.269076+0000","last_fullsized":"2023-10-16T11:19:25.269076+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.656476+0000","last_change":"2023-10-16T11:19:34.656476+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.656476+0000","last_undegraded":"2023-10-16T11:19:34.656476+0000","last_fullsized":"2023-10-16T11:19:34.656476+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273129+0000","last_change":"2023-10-16T11:19:22.327307+0000","last_active":"2023-10-16T11:19:25.273129+0000","last_peered":"2023-10-16T11:19:25.273129+0000","last_clean":"2023-10-16T11:19:25.273129+0000","last_became_active":"2023-10-16T11:19:22.327036+0000","last_became_peered":"2023-10-16T11:19:22.327036+0000","last_unstale":"2023-10-16T11:19:25.273129+0000","last_undegraded":"2023-10-16T11:19:25.273129+0000","last_fullsized":"2023-10-16T11:19:25.273129+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:24.276671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273204+0000","last_change":"2023-10-16T11:19:23.337758+0000","last_active":"2023-10-16T11:19:25.273204+0000","last_peered":"2023-10-16T11:19:25.273204+0000","last_clean":"2023-10-16T11:19:25.273204+0000","last_became_active":"2023-10-16T11:19:23.337147+0000","last_became_peered":"2023-10-16T11:19:23.337147+0000","last_unstale":"2023-10-16T11:19:25.273204+0000","last_undegraded":"2023-10-16T11:19:25.273204+0000","last_fullsized":"2023-10-16T11:19:25.273204+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:06.136692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.660031+0000","last_change":"2023-10-16T11:19:34.660031+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.660031+0000","last_undegraded":"2023-10-16T11:19:34.660031+0000","last_fullsized":"2023-10-16T11:19:34.660031+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272684+0000","last_change":"2023-10-16T11:19:22.325030+0000","last_active":"2023-10-16T11:19:25.272684+0000","last_peered":"2023-10-16T11:19:25.272684+0000","last_clean":"2023-10-16T11:19:25.272684+0000","last_became_active":"2023-10-16T11:19:22.324733+0000","last_became_peered":"2023-10-16T11:19:22.324733+0000","last_unstale":"2023-10-16T11:19:25.272684+0000","last_undegraded":"2023-10-16T11:19:25.272684+0000","last_fullsized":"2023-10-16T11:19:25.272684+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:29.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272703+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:25.272703+0000","last_peered":"2023-10-16T11:19:25.272703+0000","last_clean":"2023-10-16T11:19:25.272703+0000","last_became_active":"2023-10-16T11:19:23.331725+0000","last_became_peered":"2023-10-16T11:19:23.331725+0000","last_unstale":"2023-10-16T11:19:25.272703+0000","last_undegraded":"2023-10-16T11:19:25.272703+0000","last_fullsized":"2023-10-16T11:19:25.272703+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:45.043730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.655427+0000","last_change":"2023-10-16T11:19:34.655427+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.655427+0000","last_undegraded":"2023-10-16T11:19:34.655427+0000","last_fullsized":"2023-10-16T11:19:34.655427+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"217'8","reported_seq":21,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:33.409682+0000","last_change":"2023-10-16T11:19:22.329082+0000","last_active":"2023-10-16T11:19:33.409682+0000","last_peered":"2023-10-16T11:19:33.409682+0000","last_clean":"2023-10-16T11:19:33.409682+0000","last_became_active":"2023-10-16T11:19:22.328756+0000","last_became_peered":"2023-10-16T11:19:22.328756+0000","last_unstale":"2023-10-16T11:19:33.409682+0000","last_undegraded":"2023-10-16T11:19:33.409682+0000","last_fullsized":"2023-10-16T11:19:33.409682+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.384963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272472+0000","last_change":"2023-10-16T11:19:23.336196+0000","last_active":"2023-10-16T11:19:25.272472+0000","last_peered":"2023-10-16T11:19:25.272472+0000","last_clean":"2023-10-16T11:19:25.272472+0000","last_became_active":"2023-10-16T11:19:23.335924+0000","last_became_peered":"2023-10-16T11:19:23.335924+0000","last_unstale":"2023-10-16T11:19:25.272472+0000","last_undegraded":"2023-10-16T11:19:25.272472+0000","last_fullsized":"2023-10-16T11:19:25.272472+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:27:01.350215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657403+0000","last_change":"2023-10-16T11:19:34.657403+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657403+0000","last_undegraded":"2023-10-16T11:19:34.657403+0000","last_fullsized":"2023-10-16T11:19:34.657403+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"216'3","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269055+0000","last_change":"2023-10-16T11:19:22.331150+0000","last_active":"2023-10-16T11:19:25.269055+0000","last_peered":"2023-10-16T11:19:25.269055+0000","last_clean":"2023-10-16T11:19:25.269055+0000","last_became_active":"2023-10-16T11:19:22.330820+0000","last_became_peered":"2023-10-16T11:19:22.330820+0000","last_unstale":"2023-10-16T11:19:25.269055+0000","last_undegraded":"2023-10-16T11:19:25.269055+0000","last_fullsized":"2023-10-16T11:19:25.269055+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.268982+0000","last_change":"2023-10-16T11:19:23.338568+0000","last_active":"2023-10-16T11:19:25.268982+0000","last_peered":"2023-10-16T11:19:25.268982+0000","last_clean":"2023-10-16T11:19:25.268982+0000","last_became_active":"2023-10-16T11:19:23.338280+0000","last_became_peered":"2023-10-16T11:19:23.338280+0000","last_unstale":"2023-10-16T11:19:25.268982+0000","last_undegraded":"2023-10-16T11:19:25.268982+0000","last_fullsized":"2023-10-16T11:19:25.268982+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:40.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.661663+0000","last_change":"2023-10-16T11:19:34.661663+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.661663+0000","last_undegraded":"2023-10-16T11:19:34.661663+0000","last_fullsized":"2023-10-16T11:19:34.661663+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272445+0000","last_change":"2023-10-16T11:19:22.325211+0000","last_active":"2023-10-16T11:19:25.272445+0000","last_peered":"2023-10-16T11:19:25.272445+0000","last_clean":"2023-10-16T11:19:25.272445+0000","last_became_active":"2023-10-16T11:19:22.324901+0000","last_became_peered":"2023-10-16T11:19:22.324901+0000","last_unstale":"2023-10-16T11:19:25.272445+0000","last_undegraded":"2023-10-16T11:19:25.272445+0000","last_fullsized":"2023-10-16T11:19:25.272445+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:46.432434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273093+0000","last_change":"2023-10-16T11:19:23.331144+0000","last_active":"2023-10-16T11:19:25.273093+0000","last_peered":"2023-10-16T11:19:25.273093+0000","last_clean":"2023-10-16T11:19:25.273093+0000","last_became_active":"2023-10-16T11:19:23.330824+0000","last_became_peered":"2023-10-16T11:19:23.330824+0000","last_unstale":"2023-10-16T11:19:25.273093+0000","last_undegraded":"2023-10-16T11:19:25.273093+0000","last_fullsized":"2023-10-16T11:19:25.273093+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:44:27.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657865+0000","last_change":"2023-10-16T11:19:34.657865+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657865+0000","last_undegraded":"2023-10-16T11:19:34.657865+0000","last_fullsized":"2023-10-16T11:19:34.657865+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272566+0000","last_change":"2023-10-16T11:19:22.332253+0000","last_active":"2023-10-16T11:19:25.272566+0000","last_peered":"2023-10-16T11:19:25.272566+0000","last_clean":"2023-10-16T11:19:25.272566+0000","last_became_active":"2023-10-16T11:19:22.331970+0000","last_became_peered":"2023-10-16T11:19:22.331970+0000","last_unstale":"2023-10-16T11:19:25.272566+0000","last_undegraded":"2023-10-16T11:19:25.272566+0000","last_fullsized":"2023-10-16T11:19:25.272566+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:19.740978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272967+0000","last_change":"2023-10-16T11:19:23.330327+0000","last_active":"2023-10-16T11:19:25.272967+0000","last_peered":"2023-10-16T11:19:25.272967+0000","last_clean":"2023-10-16T11:19:25.272967+0000","last_became_active":"2023-10-16T11:19:23.330040+0000","last_became_peered":"2023-10-16T11:19:23.330040+0000","last_unstale":"2023-10-16T11:19:25.272967+0000","last_undegraded":"2023-10-16T11:19:25.272967+0000","last_fullsized":"2023-10-16T11:19:25.272967+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659906+0000","last_change":"2023-10-16T11:19:34.659906+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659906+0000","last_undegraded":"2023-10-16T11:19:34.659906+0000","last_fullsized":"2023-10-16T11:19:34.659906+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273074+0000","last_change":"2023-10-16T11:19:22.330754+0000","last_active":"2023-10-16T11:19:25.273074+0000","last_peered":"2023-10-16T11:19:25.273074+0000","last_clean":"2023-10-16T11:19:25.273074+0000","last_became_active":"2023-10-16T11:19:22.330490+0000","last_became_peered":"2023-10-16T11:19:22.330490+0000","last_unstale":"2023-10-16T11:19:25.273074+0000","last_undegraded":"2023-10-16T11:19:25.273074+0000","last_fullsized":"2023-10-16T11:19:25.273074+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:17.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269123+0000","last_change":"2023-10-16T11:19:23.334130+0000","last_active":"2023-10-16T11:19:25.269123+0000","last_peered":"2023-10-16T11:19:25.269123+0000","last_clean":"2023-10-16T11:19:25.269123+0000","last_became_active":"2023-10-16T11:19:23.333073+0000","last_became_peered":"2023-10-16T11:19:23.333073+0000","last_unstale":"2023-10-16T11:19:25.269123+0000","last_undegraded":"2023-10-16T11:19:25.269123+0000","last_fullsized":"2023-10-16T11:19:25.269123+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:38.491191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657803+0000","last_change":"2023-10-16T11:19:34.657803+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657803+0000","last_undegraded":"2023-10-16T11:19:34.657803+0000","last_fullsized":"2023-10-16T11:19:34.657803+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272260+0000","last_change":"2023-10-16T11:19:22.328061+0000","last_active":"2023-10-16T11:19:25.272260+0000","last_peered":"2023-10-16T11:19:25.272260+0000","last_clean":"2023-10-16T11:19:25.272260+0000","last_became_active":"2023-10-16T11:19:22.327773+0000","last_became_peered":"2023-10-16T11:19:22.327773+0000","last_unstale":"2023-10-16T11:19:25.272260+0000","last_undegraded":"2023-10-16T11:19:25.272260+0000","last_fullsized":"2023-10-16T11:19:25.272260+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:39.077769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273059+0000","last_change":"2023-10-16T11:19:23.330616+0000","last_active":"2023-10-16T11:19:25.273059+0000","last_peered":"2023-10-16T11:19:25.273059+0000","last_clean":"2023-10-16T11:19:25.273059+0000","last_became_active":"2023-10-16T11:19:23.330317+0000","last_became_peered":"2023-10-16T11:19:23.330317+0000","last_unstale":"2023-10-16T11:19:25.273059+0000","last_undegraded":"2023-10-16T11:19:25.273059+0000","last_fullsized":"2023-10-16T11:19:25.273059+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659497+0000","last_change":"2023-10-16T11:19:34.659497+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659497+0000","last_undegraded":"2023-10-16T11:19:34.659497+0000","last_fullsized":"2023-10-16T11:19:34.659497+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272319+0000","last_change":"2023-10-16T11:19:22.328217+0000","last_active":"2023-10-16T11:19:25.272319+0000","last_peered":"2023-10-16T11:19:25.272319+0000","last_clean":"2023-10-16T11:19:25.272319+0000","last_became_active":"2023-10-16T11:19:22.327901+0000","last_became_peered":"2023-10-16T11:19:22.327901+0000","last_unstale":"2023-10-16T11:19:25.272319+0000","last_undegraded":"2023-10-16T11:19:25.272319+0000","last_fullsized":"2023-10-16T11:19:25.272319+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:48.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273032+0000","last_change":"2023-10-16T11:19:23.336335+0000","last_active":"2023-10-16T11:19:25.273032+0000","last_peered":"2023-10-16T11:19:25.273032+0000","last_clean":"2023-10-16T11:19:25.273032+0000","last_became_active":"2023-10-16T11:19:23.335852+0000","last_became_peered":"2023-10-16T11:19:23.335852+0000","last_unstale":"2023-10-16T11:19:25.273032+0000","last_undegraded":"2023-10-16T11:19:25.273032+0000","last_fullsized":"2023-10-16T11:19:25.273032+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.660104+0000","last_change":"2023-10-16T11:19:34.660104+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.660104+0000","last_undegraded":"2023-10-16T11:19:34.660104+0000","last_fullsized":"2023-10-16T11:19:34.660104+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269042+0000","last_change":"2023-10-16T11:19:22.329968+0000","last_active":"2023-10-16T11:19:25.269042+0000","last_peered":"2023-10-16T11:19:25.269042+0000","last_clean":"2023-10-16T11:19:25.269042+0000","last_became_active":"2023-10-16T11:19:22.326360+0000","last_became_peered":"2023-10-16T11:19:22.326360+0000","last_unstale":"2023-10-16T11:19:25.269042+0000","last_undegraded":"2023-10-16T11:19:25.269042+0000","last_fullsized":"2023-10-16T11:19:25.269042+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272461+0000","last_change":"2023-10-16T11:19:23.335733+0000","last_active":"2023-10-16T11:19:25.272461+0000","last_peered":"2023-10-16T11:19:25.272461+0000","last_clean":"2023-10-16T11:19:25.272461+0000","last_became_active":"2023-10-16T11:19:23.334936+0000","last_became_peered":"2023-10-16T11:19:23.334936+0000","last_unstale":"2023-10-16T11:19:25.272461+0000","last_undegraded":"2023-10-16T11:19:25.272461+0000","last_fullsized":"2023-10-16T11:19:25.272461+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:08.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.656791+0000","last_change":"2023-10-16T11:19:34.656791+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.656791+0000","last_undegraded":"2023-10-16T11:19:34.656791+0000","last_fullsized":"2023-10-16T11:19:34.656791+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272725+0000","last_change":"2023-10-16T11:19:22.328385+0000","last_active":"2023-10-16T11:19:25.272725+0000","last_peered":"2023-10-16T11:19:25.272725+0000","last_clean":"2023-10-16T11:19:25.272725+0000","last_became_active":"2023-10-16T11:19:22.328096+0000","last_became_peered":"2023-10-16T11:19:22.328096+0000","last_unstale":"2023-10-16T11:19:25.272725+0000","last_undegraded":"2023-10-16T11:19:25.272725+0000","last_fullsized":"2023-10-16T11:19:25.272725+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:40.312176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272188+0000","last_change":"2023-10-16T11:19:23.326944+0000","last_active":"2023-10-16T11:19:25.272188+0000","last_peered":"2023-10-16T11:19:25.272188+0000","last_clean":"2023-10-16T11:19:25.272188+0000","last_became_active":"2023-10-16T11:19:23.326680+0000","last_became_peered":"2023-10-16T11:19:23.326680+0000","last_unstale":"2023-10-16T11:19:25.272188+0000","last_undegraded":"2023-10-16T11:19:25.272188+0000","last_fullsized":"2023-10-16T11:19:25.272188+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:28.070345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657531+0000","last_change":"2023-10-16T11:19:34.657531+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657531+0000","last_undegraded":"2023-10-16T11:19:34.657531+0000","last_fullsized":"2023-10-16T11:19:34.657531+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272648+0000","last_change":"2023-10-16T11:19:22.332064+0000","last_active":"2023-10-16T11:19:25.272648+0000","last_peered":"2023-10-16T11:19:25.272648+0000","last_clean":"2023-10-16T11:19:25.272648+0000","last_became_active":"2023-10-16T11:19:22.331746+0000","last_became_peered":"2023-10-16T11:19:22.331746+0000","last_unstale":"2023-10-16T11:19:25.272648+0000","last_undegraded":"2023-10-16T11:19:25.272648+0000","last_fullsized":"2023-10-16T11:19:25.272648+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272551+0000","last_change":"2023-10-16T11:19:23.342219+0000","last_active":"2023-10-16T11:19:25.272551+0000","last_peered":"2023-10-16T11:19:25.272551+0000","last_clean":"2023-10-16T11:19:25.272551+0000","last_became_active":"2023-10-16T11:19:23.334494+0000","last_became_peered":"2023-10-16T11:19:23.334494+0000","last_unstale":"2023-10-16T11:19:25.272551+0000","last_undegraded":"2023-10-16T11:19:25.272551+0000","last_fullsized":"2023-10-16T11:19:25.272551+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654954+0000","last_change":"2023-10-16T11:19:34.654954+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654954+0000","last_undegraded":"2023-10-16T11:19:34.654954+0000","last_fullsized":"2023-10-16T11:19:34.654954+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"216'4","reported_seq":17,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273074+0000","last_change":"2023-10-16T11:19:22.332308+0000","last_active":"2023-10-16T11:19:25.273074+0000","last_peered":"2023-10-16T11:19:25.273074+0000","last_clean":"2023-10-16T11:19:25.273074+0000","last_became_active":"2023-10-16T11:19:22.332005+0000","last_became_peered":"2023-10-16T11:19:22.332005+0000","last_unstale":"2023-10-16T11:19:25.273074+0000","last_undegraded":"2023-10-16T11:19:25.273074+0000","last_fullsized":"2023-10-16T11:19:25.273074+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272625+0000","last_change":"2023-10-16T11:19:23.341701+0000","last_active":"2023-10-16T11:19:25.272625+0000","last_peered":"2023-10-16T11:19:25.272625+0000","last_clean":"2023-10-16T11:19:25.272625+0000","last_became_active":"2023-10-16T11:19:23.341424+0000","last_became_peered":"2023-10-16T11:19:23.341424+0000","last_unstale":"2023-10-16T11:19:25.272625+0000","last_undegraded":"2023-10-16T11:19:25.272625+0000","last_fullsized":"2023-10-16T11:19:25.272625+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:25.448895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654310+0000","last_change":"2023-10-16T11:19:34.654310+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654310+0000","last_undegraded":"2023-10-16T11:19:34.654310+0000","last_fullsized":"2023-10-16T11:19:34.654310+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272205+0000","last_change":"2023-10-16T11:19:23.323955+0000","last_active":"2023-10-16T11:19:25.272205+0000","last_peered":"2023-10-16T11:19:25.272205+0000","last_clean":"2023-10-16T11:19:25.272205+0000","last_became_active":"2023-10-16T11:19:23.323706+0000","last_became_peered":"2023-10-16T11:19:23.323706+0000","last_unstale":"2023-10-16T11:19:25.272205+0000","last_undegraded":"2023-10-16T11:19:25.272205+0000","last_fullsized":"2023-10-16T11:19:25.272205+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.182127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272240+0000","last_change":"2023-10-16T11:19:23.332826+0000","last_active":"2023-10-16T11:19:25.272240+0000","last_peered":"2023-10-16T11:19:25.272240+0000","last_clean":"2023-10-16T11:19:25.272240+0000","last_became_active":"2023-10-16T11:19:23.332541+0000","last_became_peered":"2023-10-16T11:19:23.332541+0000","last_unstale":"2023-10-16T11:19:25.272240+0000","last_undegraded":"2023-10-16T11:19:25.272240+0000","last_fullsized":"2023-10-16T11:19:25.272240+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:12:33.418278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273019+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:25.273019+0000","last_peered":"2023-10-16T11:19:25.273019+0000","last_clean":"2023-10-16T11:19:25.273019+0000","last_became_active":"2023-10-16T11:19:23.335534+0000","last_became_peered":"2023-10-16T11:19:23.335534+0000","last_unstale":"2023-10-16T11:19:25.273019+0000","last_undegraded":"2023-10-16T11:19:25.273019+0000","last_fullsized":"2023-10-16T11:19:25.273019+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269029+0000","last_change":"2023-10-16T11:19:23.330876+0000","last_active":"2023-10-16T11:19:25.269029+0000","last_peered":"2023-10-16T11:19:25.269029+0000","last_clean":"2023-10-16T11:19:25.269029+0000","last_became_active":"2023-10-16T11:19:23.330597+0000","last_became_peered":"2023-10-16T11:19:23.330597+0000","last_unstale":"2023-10-16T11:19:25.269029+0000","last_undegraded":"2023-10-16T11:19:25.269029+0000","last_fullsized":"2023-10-16T11:19:25.269029+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272544+0000","last_change":"2023-10-16T11:19:23.327468+0000","last_active":"2023-10-16T11:19:25.272544+0000","last_peered":"2023-10-16T11:19:25.272544+0000","last_clean":"2023-10-16T11:19:25.272544+0000","last_became_active":"2023-10-16T11:19:23.327176+0000","last_became_peered":"2023-10-16T11:19:23.327176+0000","last_unstale":"2023-10-16T11:19:25.272544+0000","last_undegraded":"2023-10-16T11:19:25.272544+0000","last_fullsized":"2023-10-16T11:19:25.272544+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:02.654427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273186+0000","last_change":"2023-10-16T11:19:23.332619+0000","last_active":"2023-10-16T11:19:25.273186+0000","last_peered":"2023-10-16T11:19:25.273186+0000","last_clean":"2023-10-16T11:19:25.273186+0000","last_became_active":"2023-10-16T11:19:23.332037+0000","last_became_peered":"2023-10-16T11:19:23.332037+0000","last_unstale":"2023-10-16T11:19:25.273186+0000","last_undegraded":"2023-10-16T11:19:25.273186+0000","last_fullsized":"2023-10-16T11:19:25.273186+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:48.332974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272635+0000","last_change":"2023-10-16T11:19:23.341307+0000","last_active":"2023-10-16T11:19:25.272635+0000","last_peered":"2023-10-16T11:19:25.272635+0000","last_clean":"2023-10-16T11:19:25.272635+0000","last_became_active":"2023-10-16T11:19:23.334457+0000","last_became_peered":"2023-10-16T11:19:23.334457+0000","last_unstale":"2023-10-16T11:19:25.272635+0000","last_undegraded":"2023-10-16T11:19:25.272635+0000","last_fullsized":"2023-10-16T11:19:25.272635+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269114+0000","last_change":"2023-10-16T11:19:23.334306+0000","last_active":"2023-10-16T11:19:25.269114+0000","last_peered":"2023-10-16T11:19:25.269114+0000","last_clean":"2023-10-16T11:19:25.269114+0000","last_became_active":"2023-10-16T11:19:23.333460+0000","last_became_peered":"2023-10-16T11:19:23.333460+0000","last_unstale":"2023-10-16T11:19:25.269114+0000","last_undegraded":"2023-10-16T11:19:25.269114+0000","last_fullsized":"2023-10-16T11:19:25.269114+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:53.175386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272352+0000","last_change":"2023-10-16T11:19:23.341264+0000","last_active":"2023-10-16T11:19:25.272352+0000","last_peered":"2023-10-16T11:19:25.272352+0000","last_clean":"2023-10-16T11:19:25.272352+0000","last_became_active":"2023-10-16T11:19:23.340921+0000","last_became_peered":"2023-10-16T11:19:23.340921+0000","last_unstale":"2023-10-16T11:19:25.272352+0000","last_undegraded":"2023-10-16T11:19:25.272352+0000","last_fullsized":"2023-10-16T11:19:25.272352+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.710291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272645+0000","last_change":"2023-10-16T11:19:23.341165+0000","last_active":"2023-10-16T11:19:25.272645+0000","last_peered":"2023-10-16T11:19:25.272645+0000","last_clean":"2023-10-16T11:19:25.272645+0000","last_became_active":"2023-10-16T11:19:23.340853+0000","last_became_peered":"2023-10-16T11:19:23.340853+0000","last_unstale":"2023-10-16T11:19:25.272645+0000","last_undegraded":"2023-10-16T11:19:25.272645+0000","last_fullsized":"2023-10-16T11:19:25.272645+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:41.539293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272168+0000","last_change":"2023-10-16T11:19:23.341632+0000","last_active":"2023-10-16T11:19:25.272168+0000","last_peered":"2023-10-16T11:19:25.272168+0000","last_clean":"2023-10-16T11:19:25.272168+0000","last_became_active":"2023-10-16T11:19:23.341312+0000","last_became_peered":"2023-10-16T11:19:23.341312+0000","last_unstale":"2023-10-16T11:19:25.272168+0000","last_undegraded":"2023-10-16T11:19:25.272168+0000","last_fullsized":"2023-10-16T11:19:25.272168+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272212+0000","last_change":"2023-10-16T11:19:23.334105+0000","last_active":"2023-10-16T11:19:25.272212+0000","last_peered":"2023-10-16T11:19:25.272212+0000","last_clean":"2023-10-16T11:19:25.272212+0000","last_became_active":"2023-10-16T11:19:23.333836+0000","last_became_peered":"2023-10-16T11:19:23.333836+0000","last_unstale":"2023-10-16T11:19:25.272212+0000","last_undegraded":"2023-10-16T11:19:25.272212+0000","last_fullsized":"2023-10-16T11:19:25.272212+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.432829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272269+0000","last_change":"2023-10-16T11:19:23.331248+0000","last_active":"2023-10-16T11:19:25.272269+0000","last_peered":"2023-10-16T11:19:25.272269+0000","last_clean":"2023-10-16T11:19:25.272269+0000","last_became_active":"2023-10-16T11:19:23.330931+0000","last_became_peered":"2023-10-16T11:19:23.330931+0000","last_unstale":"2023-10-16T11:19:25.272269+0000","last_undegraded":"2023-10-16T11:19:25.272269+0000","last_fullsized":"2023-10-16T11:19:25.272269+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272294+0000","last_change":"2023-10-16T11:19:23.337766+0000","last_active":"2023-10-16T11:19:25.272294+0000","last_peered":"2023-10-16T11:19:25.272294+0000","last_clean":"2023-10-16T11:19:25.272294+0000","last_became_active":"2023-10-16T11:19:23.337503+0000","last_became_peered":"2023-10-16T11:19:23.337503+0000","last_unstale":"2023-10-16T11:19:25.272294+0000","last_undegraded":"2023-10-16T11:19:25.272294+0000","last_fullsized":"2023-10-16T11:19:25.272294+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269066+0000","last_change":"2023-10-16T11:19:23.333633+0000","last_active":"2023-10-16T11:19:25.269066+0000","last_peered":"2023-10-16T11:19:25.269066+0000","last_clean":"2023-10-16T11:19:25.269066+0000","last_became_active":"2023-10-16T11:19:23.333280+0000","last_became_peered":"2023-10-16T11:19:23.333280+0000","last_unstale":"2023-10-16T11:19:25.269066+0000","last_undegraded":"2023-10-16T11:19:25.269066+0000","last_fullsized":"2023-10-16T11:19:25.269066+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269104+0000","last_change":"2023-10-16T11:19:23.334483+0000","last_active":"2023-10-16T11:19:25.269104+0000","last_peered":"2023-10-16T11:19:25.269104+0000","last_clean":"2023-10-16T11:19:25.269104+0000","last_became_active":"2023-10-16T11:19:23.333819+0000","last_became_peered":"2023-10-16T11:19:23.333819+0000","last_unstale":"2023-10-16T11:19:25.269104+0000","last_undegraded":"2023-10-16T11:19:25.269104+0000","last_fullsized":"2023-10-16T11:19:25.269104+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273051+0000","last_change":"2023-10-16T11:19:23.332953+0000","last_active":"2023-10-16T11:19:25.273051+0000","last_peered":"2023-10-16T11:19:25.273051+0000","last_clean":"2023-10-16T11:19:25.273051+0000","last_became_active":"2023-10-16T11:19:23.332662+0000","last_became_peered":"2023-10-16T11:19:23.332662+0000","last_unstale":"2023-10-16T11:19:25.273051+0000","last_undegraded":"2023-10-16T11:19:25.273051+0000","last_fullsized":"2023-10-16T11:19:25.273051+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:27.611049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272989+0000","last_change":"2023-10-16T11:19:23.325666+0000","last_active":"2023-10-16T11:19:25.272989+0000","last_peered":"2023-10-16T11:19:25.272989+0000","last_clean":"2023-10-16T11:19:25.272989+0000","last_became_active":"2023-10-16T11:19:23.325400+0000","last_became_peered":"2023-10-16T11:19:23.325400+0000","last_unstale":"2023-10-16T11:19:25.272989+0000","last_undegraded":"2023-10-16T11:19:25.272989+0000","last_fullsized":"2023-10-16T11:19:25.272989+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272156+0000","last_change":"2023-10-16T11:19:23.342633+0000","last_active":"2023-10-16T11:19:25.272156+0000","last_peered":"2023-10-16T11:19:25.272156+0000","last_clean":"2023-10-16T11:19:25.272156+0000","last_became_active":"2023-10-16T11:19:23.342346+0000","last_became_peered":"2023-10-16T11:19:23.342346+0000","last_unstale":"2023-10-16T11:19:25.272156+0000","last_undegraded":"2023-10-16T11:19:25.272156+0000","last_fullsized":"2023-10-16T11:19:25.272156+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:12.087797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272298+0000","last_change":"2023-10-16T11:19:23.330180+0000","last_active":"2023-10-16T11:19:25.272298+0000","last_peered":"2023-10-16T11:19:25.272298+0000","last_clean":"2023-10-16T11:19:25.272298+0000","last_became_active":"2023-10-16T11:19:23.329598+0000","last_became_peered":"2023-10-16T11:19:23.329598+0000","last_unstale":"2023-10-16T11:19:25.272298+0000","last_undegraded":"2023-10-16T11:19:25.272298+0000","last_fullsized":"2023-10-16T11:19:25.272298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:59.613377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272530+0000","last_change":"2023-10-16T11:19:23.335413+0000","last_active":"2023-10-16T11:19:25.272530+0000","last_peered":"2023-10-16T11:19:25.272530+0000","last_clean":"2023-10-16T11:19:25.272530+0000","last_became_active":"2023-10-16T11:19:23.334994+0000","last_became_peered":"2023-10-16T11:19:23.334994+0000","last_unstale":"2023-10-16T11:19:25.272530+0000","last_undegraded":"2023-10-16T11:19:25.272530+0000","last_fullsized":"2023-10-16T11:19:25.272530+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:53.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273167+0000","last_change":"2023-10-16T11:19:23.333705+0000","last_active":"2023-10-16T11:19:25.273167+0000","last_peered":"2023-10-16T11:19:25.273167+0000","last_clean":"2023-10-16T11:19:25.273167+0000","last_became_active":"2023-10-16T11:19:23.333009+0000","last_became_peered":"2023-10-16T11:19:23.333009+0000","last_unstale":"2023-10-16T11:19:25.273167+0000","last_undegraded":"2023-10-16T11:19:25.273167+0000","last_fullsized":"2023-10-16T11:19:25.273167+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:41.811226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272308+0000","last_change":"2023-10-16T11:19:23.333382+0000","last_active":"2023-10-16T11:19:25.272308+0000","last_peered":"2023-10-16T11:19:25.272308+0000","last_clean":"2023-10-16T11:19:25.272308+0000","last_became_active":"2023-10-16T11:19:23.332643+0000","last_became_peered":"2023-10-16T11:19:23.332643+0000","last_unstale":"2023-10-16T11:19:25.272308+0000","last_undegraded":"2023-10-16T11:19:25.272308+0000","last_fullsized":"2023-10-16T11:19:25.272308+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:54.101849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269094+0000","last_change":"2023-10-16T11:19:23.336597+0000","last_active":"2023-10-16T11:19:25.269094+0000","last_peered":"2023-10-16T11:19:25.269094+0000","last_clean":"2023-10-16T11:19:25.269094+0000","last_became_active":"2023-10-16T11:19:23.336307+0000","last_became_peered":"2023-10-16T11:19:23.336307+0000","last_unstale":"2023-10-16T11:19:25.269094+0000","last_undegraded":"2023-10-16T11:19:25.269094+0000","last_fullsized":"2023-10-16T11:19:25.269094+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272100+0000","last_change":"2023-10-16T11:19:23.329954+0000","last_active":"2023-10-16T11:19:25.272100+0000","last_peered":"2023-10-16T11:19:25.272100+0000","last_clean":"2023-10-16T11:19:25.272100+0000","last_became_active":"2023-10-16T11:19:23.329688+0000","last_became_peered":"2023-10-16T11:19:23.329688+0000","last_unstale":"2023-10-16T11:19:25.272100+0000","last_undegraded":"2023-10-16T11:19:25.272100+0000","last_fullsized":"2023-10-16T11:19:25.272100+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:11.183201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272421+0000","last_change":"2023-10-16T11:19:23.329203+0000","last_active":"2023-10-16T11:19:25.272421+0000","last_peered":"2023-10-16T11:19:25.272421+0000","last_clean":"2023-10-16T11:19:25.272421+0000","last_became_active":"2023-10-16T11:19:23.328748+0000","last_became_peered":"2023-10-16T11:19:23.328748+0000","last_unstale":"2023-10-16T11:19:25.272421+0000","last_undegraded":"2023-10-16T11:19:25.272421+0000","last_fullsized":"2023-10-16T11:19:25.272421+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:09.942459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272279+0000","last_change":"2023-10-16T11:19:23.326663+0000","last_active":"2023-10-16T11:19:25.272279+0000","last_peered":"2023-10-16T11:19:25.272279+0000","last_clean":"2023-10-16T11:19:25.272279+0000","last_became_active":"2023-10-16T11:19:23.326378+0000","last_became_peered":"2023-10-16T11:19:23.326378+0000","last_unstale":"2023-10-16T11:19:25.272279+0000","last_undegraded":"2023-10-16T11:19:25.272279+0000","last_fullsized":"2023-10-16T11:19:25.272279+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:41.823015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272442+0000","last_change":"2023-10-16T11:19:23.330646+0000","last_active":"2023-10-16T11:19:25.272442+0000","last_peered":"2023-10-16T11:19:25.272442+0000","last_clean":"2023-10-16T11:19:25.272442+0000","last_became_active":"2023-10-16T11:19:23.330371+0000","last_became_peered":"2023-10-16T11:19:23.330371+0000","last_unstale":"2023-10-16T11:19:25.272442+0000","last_undegraded":"2023-10-16T11:19:25.272442+0000","last_fullsized":"2023-10-16T11:19:25.272442+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:55.845569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272341+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:25.272341+0000","last_peered":"2023-10-16T11:19:25.272341+0000","last_clean":"2023-10-16T11:19:25.272341+0000","last_became_active":"2023-10-16T11:19:23.335513+0000","last_became_peered":"2023-10-16T11:19:23.335513+0000","last_unstale":"2023-10-16T11:19:25.272341+0000","last_undegraded":"2023-10-16T11:19:25.272341+0000","last_fullsized":"2023-10-16T11:19:25.272341+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:30.545707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273148+0000","last_change":"2023-10-16T11:19:23.333266+0000","last_active":"2023-10-16T11:19:25.273148+0000","last_peered":"2023-10-16T11:19:25.273148+0000","last_clean":"2023-10-16T11:19:25.273148+0000","last_became_active":"2023-10-16T11:19:23.332412+0000","last_became_peered":"2023-10-16T11:19:23.332412+0000","last_unstale":"2023-10-16T11:19:25.273148+0000","last_undegraded":"2023-10-16T11:19:25.273148+0000","last_fullsized":"2023-10-16T11:19:25.273148+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:19.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272755+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:25.272755+0000","last_peered":"2023-10-16T11:19:25.272755+0000","last_clean":"2023-10-16T11:19:25.272755+0000","last_became_active":"2023-10-16T11:19:23.331671+0000","last_became_peered":"2023-10-16T11:19:23.331671+0000","last_unstale":"2023-10-16T11:19:25.272755+0000","last_undegraded":"2023-10-16T11:19:25.272755+0000","last_fullsized":"2023-10-16T11:19:25.272755+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:13.976802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272675+0000","last_change":"2023-10-16T11:19:23.334441+0000","last_active":"2023-10-16T11:19:25.272675+0000","last_peered":"2023-10-16T11:19:25.272675+0000","last_clean":"2023-10-16T11:19:25.272675+0000","last_became_active":"2023-10-16T11:19:23.334140+0000","last_became_peered":"2023-10-16T11:19:23.334140+0000","last_unstale":"2023-10-16T11:19:25.272675+0000","last_undegraded":"2023-10-16T11:19:25.272675+0000","last_fullsized":"2023-10-16T11:19:25.272675+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575991,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52500,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319340,"statfs":{"total":96636764160,"available":96583004160,"internally_reserved":0,"allocated":3350528,"data_stored":2060152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.67200000000000004,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.372},"max":{"1min":9.4359999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.68400000000000005,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":9.3059999999999992,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.71599999999999997,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":9.3610000000000007,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.119,"5min":0.71999999999999997,"15min":0.68100000000000005},"min":{"1min":0.49399999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.3960000000000008,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.72099999999999997,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":9.2889999999999997,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.753,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.71699999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.36099999999999999},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.442},{"interface":"front","average":{"1min":1.157,"5min":0.76500000000000001,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":9.4689999999999994,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.69899999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.61199999999999999,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.746},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.63100000000000001,"15min":0.56699999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.5399999999999991,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.64000000000000001,"15min":0.57399999999999995},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.69399999999999995,"15min":0.63100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.5690000000000008,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.625}]},{"osd":6,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.71199999999999997,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.155,"5min":0.69799999999999995,"15min":0.64500000000000002},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.58499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38016,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2744320,"data_stored":1462821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.91000000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.70599999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.69099999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64000000000000001,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.497},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.335,"5min":3.335,"15min":3.335},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.62}]}]},{"osd":1,"up_from":13,"seq":55834575992,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47744,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324096,"statfs":{"total":96636764160,"available":96587874304,"internally_reserved":0,"allocated":2740224,"data_stored":1466855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40899999999999997,"15min":0.41399999999999998},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.622,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.32300000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63900000000000001,"5min":0.64100000000000001,"15min":0.71699999999999997},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.46400000000000002,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.62,"5min":0.66600000000000004,"15min":0.70099999999999996},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54100000000000004,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.441,"15min":0.443},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53200000000000003,"5min":0.623,"15min":0.70499999999999996},"last":0.34000000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82099999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":1.0309999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88900000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90100000000000002,"5min":1.01,"15min":1.0760000000000001},"last":0.626}]},{"osd":6,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81999999999999995,"5min":0.93100000000000005,"15min":1.0149999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.68799999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.70299999999999996,"15min":0.70399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83399999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.81599999999999995,"5min":0.99399999999999999,"15min":1.016},"last":0.68000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575992,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42760,"kb_used_data":3260,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329080,"statfs":{"total":96636764160,"available":96592977920,"internally_reserved":0,"allocated":3338240,"data_stored":2057313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44400000000000001,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60699999999999998,"5min":1.272,"15min":1.272},"last":0.318},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67700000000000005,"5min":1.351,"15min":1.351},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.46800000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.875,"5min":1.391,"15min":1.391},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80000000000000004,"5min":0.874,"15min":1.089},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.95599999999999996,"15min":1.216},"last":0.56799999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.69799999999999995},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90900000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.47199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89100000000000001,"5min":1.038,"15min":1.1180000000000001},"last":0.64100000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.54600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575991,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47748,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324092,"statfs":{"total":96636764160,"available":96587870208,"internally_reserved":0,"allocated":2744320,"data_stored":1466945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.63300000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.29999999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69799999999999995,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77800000000000002},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.95399999999999996,"15min":1.097},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67200000000000004,"15min":0.68200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.91800000000000004,"5min":0.96199999999999997,"15min":1.016},"last":0.66100000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90500000000000003,"5min":1.022,"15min":1.0389999999999999},"last":0.629}]},{"osd":6,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91800000000000004,"5min":1.022,"15min":1.022},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91900000000000004,"5min":0.98999999999999999,"15min":1.004},"last":0.60999999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90400000000000003,"5min":1.0569999999999999,"15min":1.071},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.57899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51916,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319924,"statfs":{"total":96636764160,"available":96583602176,"internally_reserved":0,"allocated":2752512,"data_stored":1466981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38100000000000001,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.68700000000000006},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":1.0029999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.60899999999999999,"15min":0.67300000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.74299999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.58999999999999997,"5min":0.66800000000000004,"15min":0.96299999999999997},"last":0.308},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64200000000000002,"5min":0.69899999999999995,"15min":1.024},"last":0.375}]},{"osd":4,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90600000000000003,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.99099999999999999,"15min":1.026},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0640000000000001},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41799999999999998},"max":{"1min":0.85999999999999999,"5min":0.92900000000000005,"15min":1.0369999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.0620000000000001},"last":0.625}]},{"osd":7,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":1.002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88400000000000001,"5min":1.006,"15min":1.1899999999999999},"last":0.53900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575991,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47748,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324092,"statfs":{"total":96636764160,"available":96587870208,"internally_reserved":0,"allocated":2744320,"data_stored":1467437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.625},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.623},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71599999999999997,"5min":0.82499999999999996,"15min":0.90700000000000003},"last":0.441},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58799999999999997,"15min":0.623},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.624}]},{"osd":2,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60599999999999998,"15min":0.63900000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.84399999999999997,"15min":0.88400000000000001},"last":0.497}]},{"osd":3,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.624},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.83799999999999997,"15min":0.995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.51500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.41899999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.90500000000000003,"15min":1.115},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.35499999999999998,"15min":0.34200000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.83899999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.64200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575991,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64388,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307452,"statfs":{"total":96636764160,"available":96570830848,"internally_reserved":0,"allocated":2736128,"data_stored":1459436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.88600000000000001},"last":0.48099999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60299999999999998},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.86499999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.91400000000000003},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.498},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40899999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85899999999999999,"5min":0.878,"15min":0.878},"last":0.436}]},{"osd":7,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55300000000000005,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:19:40.803 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:19:40.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.951+0000 7f416d48f700 1 Processor -- start 2023-10-16T11:19:40.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.951+0000 7f416d48f700 1 -- start start 2023-10-16T11:19:40.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f416d48f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680cdee0 0x7f416815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f416d48f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f41680b6e10 0x7f41681521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f416d48f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416815c5d0 con 0x7f41680ce2c0 2023-10-16T11:19:40.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f416d48f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416815d2c0 con 0x7f41680b6e10 2023-10-16T11:19:40.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f416d48f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416815c8b0 con 0x7f41680cdee0 2023-10-16T11:19:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f41667fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f41680b6e10 0x7f41681521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f41667fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f41680b6e10 0x7f41681521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38448/0 (socket says 172.21.15.158:38448) 2023-10-16T11:19:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.952+0000 7f41667fc700 1 -- 172.21.15.158:0/4102636814 learned_addr learned my addr 172.21.15.158:0/4102636814 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f4166ffd700 1 --2- 172.21.15.158:0/4102636814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680cdee0 0x7f416815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f41667fc700 1 -- 172.21.15.158:0/4102636814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680cdee0 msgr2=0x7f416815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f41667fc700 1 --2- 172.21.15.158:0/4102636814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680cdee0 0x7f416815bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f41667fc700 1 -- 172.21.15.158:0/4102636814 >> v1:172.21.15.172:6789/0 conn(0x7f41680ce2c0 legacy=0x7f41680cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f41667fc700 1 -- 172.21.15.158:0/4102636814 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416815cb90 con 0x7f41680b6e10 2023-10-16T11:19:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f41667fc700 1 --2- 172.21.15.158:0/4102636814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680b6e10 0x7f41681521a0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f41580098f0 tx=0x7f4158009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e80a1d3a406141 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f4165ffb700 1 -- 172.21.15.158:0/4102636814 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158004230 con 0x7f41680b6e10 2023-10-16T11:19:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f4165ffb700 1 -- 172.21.15.158:0/4102636814 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4158004390 con 0x7f41680b6e10 2023-10-16T11:19:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f4165ffb700 1 -- 172.21.15.158:0/4102636814 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41580036a0 con 0x7f41680b6e10 2023-10-16T11:19:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f416d48f700 1 -- 172.21.15.158:0/4102636814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680b6e10 msgr2=0x7f41681521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.953+0000 7f416d48f700 1 --2- 172.21.15.158:0/4102636814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680b6e10 0x7f41681521a0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f41580098f0 tx=0x7f4158009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:40.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.954+0000 7f416d48f700 1 -- 172.21.15.158:0/4102636814 shutdown_connections 2023-10-16T11:19:40.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.954+0000 7f416d48f700 1 --2- 172.21.15.158:0/4102636814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680cdee0 0x7f416815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.954+0000 7f416d48f700 1 --2- 172.21.15.158:0/4102636814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680b6e10 0x7f41681521a0 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.954+0000 7f416d48f700 1 -- 172.21.15.158:0/4102636814 >> 172.21.15.158:0/4102636814 conn(0x7f416806a8d0 msgr2=0x7f416806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:40.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.954+0000 7f416d48f700 1 -- 172.21.15.158:0/4102636814 shutdown_connections 2023-10-16T11:19:40.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.954+0000 7f416d48f700 1 -- 172.21.15.158:0/4102636814 wait complete. 2023-10-16T11:19:40.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.955+0000 7f416d48f700 1 Processor -- start 2023-10-16T11:19:40.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.955+0000 7f416d48f700 1 -- start start 2023-10-16T11:19:40.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41680b6e10 0x7f41680c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680cdee0 0x7f41680c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680ce2c0 0x7f41680c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41681f1da0 con 0x7f41680cdee0 2023-10-16T11:19:40.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41680c7770 con 0x7f41680b6e10 2023-10-16T11:19:40.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41681f0ed0 con 0x7f41680ce2c0 2023-10-16T11:19:40.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680cdee0 0x7f41680c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680cdee0 0x7f41680c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38454/0 (socket says 172.21.15.158:38454) 2023-10-16T11:19:40.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 -- 172.21.15.158:0/912376985 learned_addr learned my addr 172.21.15.158:0/912376985 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:40.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41677fe700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680ce2c0 0x7f41680c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f4166ffd700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41680b6e10 0x7f41680c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 -- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680ce2c0 msgr2=0x7f41680c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680ce2c0 0x7f41680c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 -- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41680b6e10 msgr2=0x7f41680c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:40.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41680b6e10 0x7f41680c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:40.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 -- 172.21.15.158:0/912376985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158009550 con 0x7f41680cdee0 2023-10-16T11:19:40.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f41667fc700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680cdee0 0x7f41680c64f0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f415800b0e0 tx=0x7f4158004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f414f7fe700 1 -- 172.21.15.158:0/912376985 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158003e30 con 0x7f41680cdee0 2023-10-16T11:19:40.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f414f7fe700 1 -- 172.21.15.158:0/912376985 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415801f070 con 0x7f41680cdee0 2023-10-16T11:19:40.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41681f1190 con 0x7f41680cdee0 2023-10-16T11:19:40.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f414f7fe700 1 -- 172.21.15.158:0/912376985 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415800b8a0 con 0x7f41680cdee0 2023-10-16T11:19:40.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.956+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41681f1750 con 0x7f41680cdee0 2023-10-16T11:19:40.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.957+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41680cd170 con 0x7f41680cdee0 2023-10-16T11:19:40.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.958+0000 7f414f7fe700 1 -- 172.21.15.158:0/912376985 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4158021030 con 0x7f41680cdee0 2023-10-16T11:19:40.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.958+0000 7f414f7fe700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f415406b050 0x7f415406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:40.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.958+0000 7f4166ffd700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f415406b050 0x7f415406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:40.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.958+0000 7f414f7fe700 1 -- 172.21.15.158:0/912376985 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f415808b490 con 0x7f41680cdee0 2023-10-16T11:19:40.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.959+0000 7f4166ffd700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f415406b050 0x7f415406d510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f415c00a990 tx=0x7f415c008040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:40.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:40.963+0000 7f414f7fe700 1 -- 172.21.15.158:0/912376985 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4158055e40 con 0x7f41680cdee0 2023-10-16T11:19:41.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.152+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f416815c250 con 0x7f415406b050 2023-10-16T11:19:41.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.165+0000 7f414f7fe700 1 -- 172.21.15.158:0/912376985 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+516631 (secure 0 0 0) 0x7f415c00a660 con 0x7f415406b050 2023-10-16T11:19:41.173 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:41.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f415406b050 msgr2=0x7f415406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f415406b050 0x7f415406d510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f415c00a990 tx=0x7f415c008040 comp rx=0 tx=0).stop 2023-10-16T11:19:41.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680cdee0 msgr2=0x7f41680c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680cdee0 0x7f41680c64f0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f415800b0e0 tx=0x7f4158004750 comp rx=0 tx=0).stop 2023-10-16T11:19:41.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 shutdown_connections 2023-10-16T11:19:41.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41680ce2c0 0x7f41680c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41680b6e10 0x7f41680c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f415406b050 0x7f415406d510 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 --2- 172.21.15.158:0/912376985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41680cdee0 0x7f41680c64f0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.169+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 >> 172.21.15.158:0/912376985 conn(0x7f416806a8d0 msgr2=0x7f41680b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:41.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.170+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 shutdown_connections 2023-10-16T11:19:41.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.170+0000 7f416d48f700 1 -- 172.21.15.158:0/912376985 wait complete. 2023-10-16T11:19:41.180 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:19:41.183 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-16T11:19:39.276441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594025,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":626,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392820,"kb_used_data":22608,"kb_used_omap":12,"kb_used_meta":370099,"kb_avail":754581900,"statfs":{"total":773094113280,"available":772691865600,"internally_reserved":0,"allocated":23150592,"data_stored":12907940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":378981943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1247,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.624642"},"pg_stats":[{"pgid":"1.0","version":"207'348","reported_seq":1209,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:24.835291+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:19:24.835291+0000","last_peered":"2023-10-16T11:19:24.835291+0000","last_clean":"2023-10-16T11:19:24.835291+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:19:24.835291+0000","last_undegraded":"2023-10-16T11:19:24.835291+0000","last_fullsized":"2023-10-16T11:19:24.835291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269008+0000","last_change":"2023-10-16T11:19:22.332936+0000","last_active":"2023-10-16T11:19:25.269008+0000","last_peered":"2023-10-16T11:19:25.269008+0000","last_clean":"2023-10-16T11:19:25.269008+0000","last_became_active":"2023-10-16T11:19:22.332598+0000","last_became_peered":"2023-10-16T11:19:22.332598+0000","last_unstale":"2023-10-16T11:19:25.269008+0000","last_undegraded":"2023-10-16T11:19:25.269008+0000","last_fullsized":"2023-10-16T11:19:25.269008+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:34.597859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272622+0000","last_change":"2023-10-16T11:19:23.334041+0000","last_active":"2023-10-16T11:19:25.272622+0000","last_peered":"2023-10-16T11:19:25.272622+0000","last_clean":"2023-10-16T11:19:25.272622+0000","last_became_active":"2023-10-16T11:19:23.333752+0000","last_became_peered":"2023-10-16T11:19:23.333752+0000","last_unstale":"2023-10-16T11:19:25.272622+0000","last_undegraded":"2023-10-16T11:19:25.272622+0000","last_fullsized":"2023-10-16T11:19:25.272622+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.656805+0000","last_change":"2023-10-16T11:19:34.656805+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.656805+0000","last_undegraded":"2023-10-16T11:19:34.656805+0000","last_fullsized":"2023-10-16T11:19:34.656805+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272403+0000","last_change":"2023-10-16T11:19:22.334152+0000","last_active":"2023-10-16T11:19:25.272403+0000","last_peered":"2023-10-16T11:19:25.272403+0000","last_clean":"2023-10-16T11:19:25.272403+0000","last_became_active":"2023-10-16T11:19:22.333859+0000","last_became_peered":"2023-10-16T11:19:22.333859+0000","last_unstale":"2023-10-16T11:19:25.272403+0000","last_undegraded":"2023-10-16T11:19:25.272403+0000","last_fullsized":"2023-10-16T11:19:25.272403+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:50.619981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273222+0000","last_change":"2023-10-16T11:19:23.324999+0000","last_active":"2023-10-16T11:19:25.273222+0000","last_peered":"2023-10-16T11:19:25.273222+0000","last_clean":"2023-10-16T11:19:25.273222+0000","last_became_active":"2023-10-16T11:19:23.324686+0000","last_became_peered":"2023-10-16T11:19:23.324686+0000","last_unstale":"2023-10-16T11:19:25.273222+0000","last_undegraded":"2023-10-16T11:19:25.273222+0000","last_fullsized":"2023-10-16T11:19:25.273222+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:42.281964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657250+0000","last_change":"2023-10-16T11:19:34.657250+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657250+0000","last_undegraded":"2023-10-16T11:19:34.657250+0000","last_fullsized":"2023-10-16T11:19:34.657250+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272562+0000","last_change":"2023-10-16T11:19:22.332640+0000","last_active":"2023-10-16T11:19:25.272562+0000","last_peered":"2023-10-16T11:19:25.272562+0000","last_clean":"2023-10-16T11:19:25.272562+0000","last_became_active":"2023-10-16T11:19:22.331608+0000","last_became_peered":"2023-10-16T11:19:22.331608+0000","last_unstale":"2023-10-16T11:19:25.272562+0000","last_undegraded":"2023-10-16T11:19:25.272562+0000","last_fullsized":"2023-10-16T11:19:25.272562+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272425+0000","last_change":"2023-10-16T11:19:23.329115+0000","last_active":"2023-10-16T11:19:25.272425+0000","last_peered":"2023-10-16T11:19:25.272425+0000","last_clean":"2023-10-16T11:19:25.272425+0000","last_became_active":"2023-10-16T11:19:23.328830+0000","last_became_peered":"2023-10-16T11:19:23.328830+0000","last_unstale":"2023-10-16T11:19:25.272425+0000","last_undegraded":"2023-10-16T11:19:25.272425+0000","last_fullsized":"2023-10-16T11:19:25.272425+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:17.247488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657464+0000","last_change":"2023-10-16T11:19:34.657464+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657464+0000","last_undegraded":"2023-10-16T11:19:34.657464+0000","last_fullsized":"2023-10-16T11:19:34.657464+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269019+0000","last_change":"2023-10-16T11:19:22.329623+0000","last_active":"2023-10-16T11:19:25.269019+0000","last_peered":"2023-10-16T11:19:25.269019+0000","last_clean":"2023-10-16T11:19:25.269019+0000","last_became_active":"2023-10-16T11:19:22.329344+0000","last_became_peered":"2023-10-16T11:19:22.329344+0000","last_unstale":"2023-10-16T11:19:25.269019+0000","last_undegraded":"2023-10-16T11:19:25.269019+0000","last_fullsized":"2023-10-16T11:19:25.269019+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272405+0000","last_change":"2023-10-16T11:19:23.330624+0000","last_active":"2023-10-16T11:19:25.272405+0000","last_peered":"2023-10-16T11:19:25.272405+0000","last_clean":"2023-10-16T11:19:25.272405+0000","last_became_active":"2023-10-16T11:19:23.329588+0000","last_became_peered":"2023-10-16T11:19:23.329588+0000","last_unstale":"2023-10-16T11:19:25.272405+0000","last_undegraded":"2023-10-16T11:19:25.272405+0000","last_fullsized":"2023-10-16T11:19:25.272405+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:10.835410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.658784+0000","last_change":"2023-10-16T11:19:34.658784+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.658784+0000","last_undegraded":"2023-10-16T11:19:34.658784+0000","last_fullsized":"2023-10-16T11:19:34.658784+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272255+0000","last_change":"2023-10-16T11:19:22.333752+0000","last_active":"2023-10-16T11:19:25.272255+0000","last_peered":"2023-10-16T11:19:25.272255+0000","last_clean":"2023-10-16T11:19:25.272255+0000","last_became_active":"2023-10-16T11:19:22.333497+0000","last_became_peered":"2023-10-16T11:19:22.333497+0000","last_unstale":"2023-10-16T11:19:25.272255+0000","last_undegraded":"2023-10-16T11:19:25.272255+0000","last_fullsized":"2023-10-16T11:19:25.272255+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272280+0000","last_change":"2023-10-16T11:19:23.329220+0000","last_active":"2023-10-16T11:19:25.272280+0000","last_peered":"2023-10-16T11:19:25.272280+0000","last_clean":"2023-10-16T11:19:25.272280+0000","last_became_active":"2023-10-16T11:19:23.328923+0000","last_became_peered":"2023-10-16T11:19:23.328923+0000","last_unstale":"2023-10-16T11:19:25.272280+0000","last_undegraded":"2023-10-16T11:19:25.272280+0000","last_fullsized":"2023-10-16T11:19:25.272280+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657189+0000","last_change":"2023-10-16T11:19:34.657189+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657189+0000","last_undegraded":"2023-10-16T11:19:34.657189+0000","last_fullsized":"2023-10-16T11:19:34.657189+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272600+0000","last_change":"2023-10-16T11:19:22.335923+0000","last_active":"2023-10-16T11:19:25.272600+0000","last_peered":"2023-10-16T11:19:25.272600+0000","last_clean":"2023-10-16T11:19:25.272600+0000","last_became_active":"2023-10-16T11:19:22.335705+0000","last_became_peered":"2023-10-16T11:19:22.335705+0000","last_unstale":"2023-10-16T11:19:25.272600+0000","last_undegraded":"2023-10-16T11:19:25.272600+0000","last_fullsized":"2023-10-16T11:19:25.272600+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:40.803363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272586+0000","last_change":"2023-10-16T11:19:23.336160+0000","last_active":"2023-10-16T11:19:25.272586+0000","last_peered":"2023-10-16T11:19:25.272586+0000","last_clean":"2023-10-16T11:19:25.272586+0000","last_became_active":"2023-10-16T11:19:23.333920+0000","last_became_peered":"2023-10-16T11:19:23.333920+0000","last_unstale":"2023-10-16T11:19:25.272586+0000","last_undegraded":"2023-10-16T11:19:25.272586+0000","last_fullsized":"2023-10-16T11:19:25.272586+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654775+0000","last_change":"2023-10-16T11:19:34.654775+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654775+0000","last_undegraded":"2023-10-16T11:19:34.654775+0000","last_fullsized":"2023-10-16T11:19:34.654775+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273044+0000","last_change":"2023-10-16T11:19:22.326754+0000","last_active":"2023-10-16T11:19:25.273044+0000","last_peered":"2023-10-16T11:19:25.273044+0000","last_clean":"2023-10-16T11:19:25.273044+0000","last_became_active":"2023-10-16T11:19:22.326480+0000","last_became_peered":"2023-10-16T11:19:22.326480+0000","last_unstale":"2023-10-16T11:19:25.273044+0000","last_undegraded":"2023-10-16T11:19:25.273044+0000","last_fullsized":"2023-10-16T11:19:25.273044+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.268994+0000","last_change":"2023-10-16T11:19:23.331571+0000","last_active":"2023-10-16T11:19:25.268994+0000","last_peered":"2023-10-16T11:19:25.268994+0000","last_clean":"2023-10-16T11:19:25.268994+0000","last_became_active":"2023-10-16T11:19:23.331242+0000","last_became_peered":"2023-10-16T11:19:23.331242+0000","last_unstale":"2023-10-16T11:19:25.268994+0000","last_undegraded":"2023-10-16T11:19:25.268994+0000","last_fullsized":"2023-10-16T11:19:25.268994+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.661222+0000","last_change":"2023-10-16T11:19:34.661222+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.661222+0000","last_undegraded":"2023-10-16T11:19:34.661222+0000","last_fullsized":"2023-10-16T11:19:34.661222+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"217'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:28.410245+0000","last_change":"2023-10-16T11:19:22.331182+0000","last_active":"2023-10-16T11:19:28.410245+0000","last_peered":"2023-10-16T11:19:28.410245+0000","last_clean":"2023-10-16T11:19:28.410245+0000","last_became_active":"2023-10-16T11:19:22.330860+0000","last_became_peered":"2023-10-16T11:19:22.330860+0000","last_unstale":"2023-10-16T11:19:28.410245+0000","last_undegraded":"2023-10-16T11:19:28.410245+0000","last_fullsized":"2023-10-16T11:19:28.410245+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273111+0000","last_change":"2023-10-16T11:19:23.325482+0000","last_active":"2023-10-16T11:19:25.273111+0000","last_peered":"2023-10-16T11:19:25.273111+0000","last_clean":"2023-10-16T11:19:25.273111+0000","last_became_active":"2023-10-16T11:19:23.325193+0000","last_became_peered":"2023-10-16T11:19:23.325193+0000","last_unstale":"2023-10-16T11:19:25.273111+0000","last_undegraded":"2023-10-16T11:19:25.273111+0000","last_fullsized":"2023-10-16T11:19:25.273111+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.191635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657303+0000","last_change":"2023-10-16T11:19:34.657303+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657303+0000","last_undegraded":"2023-10-16T11:19:34.657303+0000","last_fullsized":"2023-10-16T11:19:34.657303+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269133+0000","last_change":"2023-10-16T11:19:22.332950+0000","last_active":"2023-10-16T11:19:25.269133+0000","last_peered":"2023-10-16T11:19:25.269133+0000","last_clean":"2023-10-16T11:19:25.269133+0000","last_became_active":"2023-10-16T11:19:22.332650+0000","last_became_peered":"2023-10-16T11:19:22.332650+0000","last_unstale":"2023-10-16T11:19:25.269133+0000","last_undegraded":"2023-10-16T11:19:25.269133+0000","last_fullsized":"2023-10-16T11:19:25.269133+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272136+0000","last_change":"2023-10-16T11:19:23.333192+0000","last_active":"2023-10-16T11:19:25.272136+0000","last_peered":"2023-10-16T11:19:25.272136+0000","last_clean":"2023-10-16T11:19:25.272136+0000","last_became_active":"2023-10-16T11:19:23.332739+0000","last_became_peered":"2023-10-16T11:19:23.332739+0000","last_unstale":"2023-10-16T11:19:25.272136+0000","last_undegraded":"2023-10-16T11:19:25.272136+0000","last_fullsized":"2023-10-16T11:19:25.272136+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.655371+0000","last_change":"2023-10-16T11:19:34.655371+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.655371+0000","last_undegraded":"2023-10-16T11:19:34.655371+0000","last_fullsized":"2023-10-16T11:19:34.655371+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272508+0000","last_change":"2023-10-16T11:19:22.326294+0000","last_active":"2023-10-16T11:19:25.272508+0000","last_peered":"2023-10-16T11:19:25.272508+0000","last_clean":"2023-10-16T11:19:25.272508+0000","last_became_active":"2023-10-16T11:19:22.326012+0000","last_became_peered":"2023-10-16T11:19:22.326012+0000","last_unstale":"2023-10-16T11:19:25.272508+0000","last_undegraded":"2023-10-16T11:19:25.272508+0000","last_fullsized":"2023-10-16T11:19:25.272508+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272333+0000","last_change":"2023-10-16T11:19:23.336920+0000","last_active":"2023-10-16T11:19:25.272333+0000","last_peered":"2023-10-16T11:19:25.272333+0000","last_clean":"2023-10-16T11:19:25.272333+0000","last_became_active":"2023-10-16T11:19:23.336473+0000","last_became_peered":"2023-10-16T11:19:23.336473+0000","last_unstale":"2023-10-16T11:19:25.272333+0000","last_undegraded":"2023-10-16T11:19:25.272333+0000","last_fullsized":"2023-10-16T11:19:25.272333+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:22.421887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659575+0000","last_change":"2023-10-16T11:19:34.659575+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659575+0000","last_undegraded":"2023-10-16T11:19:34.659575+0000","last_fullsized":"2023-10-16T11:19:34.659575+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273005+0000","last_change":"2023-10-16T11:19:22.331457+0000","last_active":"2023-10-16T11:19:25.273005+0000","last_peered":"2023-10-16T11:19:25.273005+0000","last_clean":"2023-10-16T11:19:25.273005+0000","last_became_active":"2023-10-16T11:19:22.331010+0000","last_became_peered":"2023-10-16T11:19:22.331010+0000","last_unstale":"2023-10-16T11:19:25.273005+0000","last_undegraded":"2023-10-16T11:19:25.273005+0000","last_fullsized":"2023-10-16T11:19:25.273005+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:38.048633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269085+0000","last_change":"2023-10-16T11:19:23.333035+0000","last_active":"2023-10-16T11:19:25.269085+0000","last_peered":"2023-10-16T11:19:25.269085+0000","last_clean":"2023-10-16T11:19:25.269085+0000","last_became_active":"2023-10-16T11:19:23.332692+0000","last_became_peered":"2023-10-16T11:19:23.332692+0000","last_unstale":"2023-10-16T11:19:25.269085+0000","last_undegraded":"2023-10-16T11:19:25.269085+0000","last_fullsized":"2023-10-16T11:19:25.269085+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654916+0000","last_change":"2023-10-16T11:19:34.654916+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654916+0000","last_undegraded":"2023-10-16T11:19:34.654916+0000","last_fullsized":"2023-10-16T11:19:34.654916+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272667+0000","last_change":"2023-10-16T11:19:22.329858+0000","last_active":"2023-10-16T11:19:25.272667+0000","last_peered":"2023-10-16T11:19:25.272667+0000","last_clean":"2023-10-16T11:19:25.272667+0000","last_became_active":"2023-10-16T11:19:22.329586+0000","last_became_peered":"2023-10-16T11:19:22.329586+0000","last_unstale":"2023-10-16T11:19:25.272667+0000","last_undegraded":"2023-10-16T11:19:25.272667+0000","last_fullsized":"2023-10-16T11:19:25.272667+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:25.906324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272383+0000","last_change":"2023-10-16T11:19:23.333742+0000","last_active":"2023-10-16T11:19:25.272383+0000","last_peered":"2023-10-16T11:19:25.272383+0000","last_clean":"2023-10-16T11:19:25.272383+0000","last_became_active":"2023-10-16T11:19:23.333424+0000","last_became_peered":"2023-10-16T11:19:23.333424+0000","last_unstale":"2023-10-16T11:19:25.272383+0000","last_undegraded":"2023-10-16T11:19:25.272383+0000","last_fullsized":"2023-10-16T11:19:25.272383+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.156569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.658120+0000","last_change":"2023-10-16T11:19:34.658120+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.658120+0000","last_undegraded":"2023-10-16T11:19:34.658120+0000","last_fullsized":"2023-10-16T11:19:34.658120+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272502+0000","last_change":"2023-10-16T11:19:22.329655+0000","last_active":"2023-10-16T11:19:25.272502+0000","last_peered":"2023-10-16T11:19:25.272502+0000","last_clean":"2023-10-16T11:19:25.272502+0000","last_became_active":"2023-10-16T11:19:22.329366+0000","last_became_peered":"2023-10-16T11:19:22.329366+0000","last_unstale":"2023-10-16T11:19:25.272502+0000","last_undegraded":"2023-10-16T11:19:25.272502+0000","last_fullsized":"2023-10-16T11:19:25.272502+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:47.331154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272714+0000","last_change":"2023-10-16T11:19:23.331460+0000","last_active":"2023-10-16T11:19:25.272714+0000","last_peered":"2023-10-16T11:19:25.272714+0000","last_clean":"2023-10-16T11:19:25.272714+0000","last_became_active":"2023-10-16T11:19:23.331178+0000","last_became_peered":"2023-10-16T11:19:23.331178+0000","last_unstale":"2023-10-16T11:19:25.272714+0000","last_undegraded":"2023-10-16T11:19:25.272714+0000","last_fullsized":"2023-10-16T11:19:25.272714+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:36.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659234+0000","last_change":"2023-10-16T11:19:34.659234+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659234+0000","last_undegraded":"2023-10-16T11:19:34.659234+0000","last_fullsized":"2023-10-16T11:19:34.659234+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272555+0000","last_change":"2023-10-16T11:19:22.332355+0000","last_active":"2023-10-16T11:19:25.272555+0000","last_peered":"2023-10-16T11:19:25.272555+0000","last_clean":"2023-10-16T11:19:25.272555+0000","last_became_active":"2023-10-16T11:19:22.331520+0000","last_became_peered":"2023-10-16T11:19:22.331520+0000","last_unstale":"2023-10-16T11:19:25.272555+0000","last_undegraded":"2023-10-16T11:19:25.272555+0000","last_fullsized":"2023-10-16T11:19:25.272555+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:10.682623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272235+0000","last_change":"2023-10-16T11:19:23.331171+0000","last_active":"2023-10-16T11:19:25.272235+0000","last_peered":"2023-10-16T11:19:25.272235+0000","last_clean":"2023-10-16T11:19:25.272235+0000","last_became_active":"2023-10-16T11:19:23.330836+0000","last_became_peered":"2023-10-16T11:19:23.330836+0000","last_unstale":"2023-10-16T11:19:25.272235+0000","last_undegraded":"2023-10-16T11:19:25.272235+0000","last_fullsized":"2023-10-16T11:19:25.272235+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:45.818409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657687+0000","last_change":"2023-10-16T11:19:34.657687+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657687+0000","last_undegraded":"2023-10-16T11:19:34.657687+0000","last_fullsized":"2023-10-16T11:19:34.657687+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272738+0000","last_change":"2023-10-16T11:19:22.318314+0000","last_active":"2023-10-16T11:19:25.272738+0000","last_peered":"2023-10-16T11:19:25.272738+0000","last_clean":"2023-10-16T11:19:25.272738+0000","last_became_active":"2023-10-16T11:19:22.318015+0000","last_became_peered":"2023-10-16T11:19:22.318015+0000","last_unstale":"2023-10-16T11:19:25.272738+0000","last_undegraded":"2023-10-16T11:19:25.272738+0000","last_fullsized":"2023-10-16T11:19:25.272738+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.421816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272573+0000","last_change":"2023-10-16T11:19:23.337274+0000","last_active":"2023-10-16T11:19:25.272573+0000","last_peered":"2023-10-16T11:19:25.272573+0000","last_clean":"2023-10-16T11:19:25.272573+0000","last_became_active":"2023-10-16T11:19:23.336889+0000","last_became_peered":"2023-10-16T11:19:23.336889+0000","last_unstale":"2023-10-16T11:19:25.272573+0000","last_undegraded":"2023-10-16T11:19:25.272573+0000","last_fullsized":"2023-10-16T11:19:25.272573+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.658407+0000","last_change":"2023-10-16T11:19:34.658407+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.658407+0000","last_undegraded":"2023-10-16T11:19:34.658407+0000","last_fullsized":"2023-10-16T11:19:34.658407+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272663+0000","last_change":"2023-10-16T11:19:22.330569+0000","last_active":"2023-10-16T11:19:25.272663+0000","last_peered":"2023-10-16T11:19:25.272663+0000","last_clean":"2023-10-16T11:19:25.272663+0000","last_became_active":"2023-10-16T11:19:22.329740+0000","last_became_peered":"2023-10-16T11:19:22.329740+0000","last_unstale":"2023-10-16T11:19:25.272663+0000","last_undegraded":"2023-10-16T11:19:25.272663+0000","last_fullsized":"2023-10-16T11:19:25.272663+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272537+0000","last_change":"2023-10-16T11:19:23.338701+0000","last_active":"2023-10-16T11:19:25.272537+0000","last_peered":"2023-10-16T11:19:25.272537+0000","last_clean":"2023-10-16T11:19:25.272537+0000","last_became_active":"2023-10-16T11:19:23.334033+0000","last_became_peered":"2023-10-16T11:19:23.334033+0000","last_unstale":"2023-10-16T11:19:25.272537+0000","last_undegraded":"2023-10-16T11:19:25.272537+0000","last_fullsized":"2023-10-16T11:19:25.272537+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659360+0000","last_change":"2023-10-16T11:19:34.659360+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659360+0000","last_undegraded":"2023-10-16T11:19:34.659360+0000","last_fullsized":"2023-10-16T11:19:34.659360+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272687+0000","last_change":"2023-10-16T11:19:22.331432+0000","last_active":"2023-10-16T11:19:25.272687+0000","last_peered":"2023-10-16T11:19:25.272687+0000","last_clean":"2023-10-16T11:19:25.272687+0000","last_became_active":"2023-10-16T11:19:22.331124+0000","last_became_peered":"2023-10-16T11:19:22.331124+0000","last_unstale":"2023-10-16T11:19:25.272687+0000","last_undegraded":"2023-10-16T11:19:25.272687+0000","last_fullsized":"2023-10-16T11:19:25.272687+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272120+0000","last_change":"2023-10-16T11:19:23.333018+0000","last_active":"2023-10-16T11:19:25.272120+0000","last_peered":"2023-10-16T11:19:25.272120+0000","last_clean":"2023-10-16T11:19:25.272120+0000","last_became_active":"2023-10-16T11:19:23.332725+0000","last_became_peered":"2023-10-16T11:19:23.332725+0000","last_unstale":"2023-10-16T11:19:25.272120+0000","last_undegraded":"2023-10-16T11:19:25.272120+0000","last_fullsized":"2023-10-16T11:19:25.272120+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:39.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657167+0000","last_change":"2023-10-16T11:19:34.657167+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657167+0000","last_undegraded":"2023-10-16T11:19:34.657167+0000","last_fullsized":"2023-10-16T11:19:34.657167+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"216'8","reported_seq":21,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272378+0000","last_change":"2023-10-16T11:19:22.332156+0000","last_active":"2023-10-16T11:19:25.272378+0000","last_peered":"2023-10-16T11:19:25.272378+0000","last_clean":"2023-10-16T11:19:25.272378+0000","last_became_active":"2023-10-16T11:19:22.331863+0000","last_became_peered":"2023-10-16T11:19:22.331863+0000","last_unstale":"2023-10-16T11:19:25.272378+0000","last_undegraded":"2023-10-16T11:19:25.272378+0000","last_fullsized":"2023-10-16T11:19:25.272378+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:36.175530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272464+0000","last_change":"2023-10-16T11:19:23.336992+0000","last_active":"2023-10-16T11:19:25.272464+0000","last_peered":"2023-10-16T11:19:25.272464+0000","last_clean":"2023-10-16T11:19:25.272464+0000","last_became_active":"2023-10-16T11:19:23.336724+0000","last_became_peered":"2023-10-16T11:19:23.336724+0000","last_unstale":"2023-10-16T11:19:25.272464+0000","last_undegraded":"2023-10-16T11:19:25.272464+0000","last_fullsized":"2023-10-16T11:19:25.272464+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:01.326738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.655139+0000","last_change":"2023-10-16T11:19:34.655139+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.655139+0000","last_undegraded":"2023-10-16T11:19:34.655139+0000","last_fullsized":"2023-10-16T11:19:34.655139+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.268964+0000","last_change":"2023-10-16T11:19:22.331044+0000","last_active":"2023-10-16T11:19:25.268964+0000","last_peered":"2023-10-16T11:19:25.268964+0000","last_clean":"2023-10-16T11:19:25.268964+0000","last_became_active":"2023-10-16T11:19:22.330757+0000","last_became_peered":"2023-10-16T11:19:22.330757+0000","last_unstale":"2023-10-16T11:19:25.268964+0000","last_undegraded":"2023-10-16T11:19:25.268964+0000","last_fullsized":"2023-10-16T11:19:25.268964+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269076+0000","last_change":"2023-10-16T11:19:23.333500+0000","last_active":"2023-10-16T11:19:25.269076+0000","last_peered":"2023-10-16T11:19:25.269076+0000","last_clean":"2023-10-16T11:19:25.269076+0000","last_became_active":"2023-10-16T11:19:23.333215+0000","last_became_peered":"2023-10-16T11:19:23.333215+0000","last_unstale":"2023-10-16T11:19:25.269076+0000","last_undegraded":"2023-10-16T11:19:25.269076+0000","last_fullsized":"2023-10-16T11:19:25.269076+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.656476+0000","last_change":"2023-10-16T11:19:34.656476+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.656476+0000","last_undegraded":"2023-10-16T11:19:34.656476+0000","last_fullsized":"2023-10-16T11:19:34.656476+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273129+0000","last_change":"2023-10-16T11:19:22.327307+0000","last_active":"2023-10-16T11:19:25.273129+0000","last_peered":"2023-10-16T11:19:25.273129+0000","last_clean":"2023-10-16T11:19:25.273129+0000","last_became_active":"2023-10-16T11:19:22.327036+0000","last_became_peered":"2023-10-16T11:19:22.327036+0000","last_unstale":"2023-10-16T11:19:25.273129+0000","last_undegraded":"2023-10-16T11:19:25.273129+0000","last_fullsized":"2023-10-16T11:19:25.273129+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:24.276671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273204+0000","last_change":"2023-10-16T11:19:23.337758+0000","last_active":"2023-10-16T11:19:25.273204+0000","last_peered":"2023-10-16T11:19:25.273204+0000","last_clean":"2023-10-16T11:19:25.273204+0000","last_became_active":"2023-10-16T11:19:23.337147+0000","last_became_peered":"2023-10-16T11:19:23.337147+0000","last_unstale":"2023-10-16T11:19:25.273204+0000","last_undegraded":"2023-10-16T11:19:25.273204+0000","last_fullsized":"2023-10-16T11:19:25.273204+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:06.136692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.660031+0000","last_change":"2023-10-16T11:19:34.660031+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.660031+0000","last_undegraded":"2023-10-16T11:19:34.660031+0000","last_fullsized":"2023-10-16T11:19:34.660031+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272684+0000","last_change":"2023-10-16T11:19:22.325030+0000","last_active":"2023-10-16T11:19:25.272684+0000","last_peered":"2023-10-16T11:19:25.272684+0000","last_clean":"2023-10-16T11:19:25.272684+0000","last_became_active":"2023-10-16T11:19:22.324733+0000","last_became_peered":"2023-10-16T11:19:22.324733+0000","last_unstale":"2023-10-16T11:19:25.272684+0000","last_undegraded":"2023-10-16T11:19:25.272684+0000","last_fullsized":"2023-10-16T11:19:25.272684+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:29.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272703+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:25.272703+0000","last_peered":"2023-10-16T11:19:25.272703+0000","last_clean":"2023-10-16T11:19:25.272703+0000","last_became_active":"2023-10-16T11:19:23.331725+0000","last_became_peered":"2023-10-16T11:19:23.331725+0000","last_unstale":"2023-10-16T11:19:25.272703+0000","last_undegraded":"2023-10-16T11:19:25.272703+0000","last_fullsized":"2023-10-16T11:19:25.272703+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:45.043730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.655427+0000","last_change":"2023-10-16T11:19:34.655427+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.655427+0000","last_undegraded":"2023-10-16T11:19:34.655427+0000","last_fullsized":"2023-10-16T11:19:34.655427+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"217'8","reported_seq":21,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:33.409682+0000","last_change":"2023-10-16T11:19:22.329082+0000","last_active":"2023-10-16T11:19:33.409682+0000","last_peered":"2023-10-16T11:19:33.409682+0000","last_clean":"2023-10-16T11:19:33.409682+0000","last_became_active":"2023-10-16T11:19:22.328756+0000","last_became_peered":"2023-10-16T11:19:22.328756+0000","last_unstale":"2023-10-16T11:19:33.409682+0000","last_undegraded":"2023-10-16T11:19:33.409682+0000","last_fullsized":"2023-10-16T11:19:33.409682+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.384963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272472+0000","last_change":"2023-10-16T11:19:23.336196+0000","last_active":"2023-10-16T11:19:25.272472+0000","last_peered":"2023-10-16T11:19:25.272472+0000","last_clean":"2023-10-16T11:19:25.272472+0000","last_became_active":"2023-10-16T11:19:23.335924+0000","last_became_peered":"2023-10-16T11:19:23.335924+0000","last_unstale":"2023-10-16T11:19:25.272472+0000","last_undegraded":"2023-10-16T11:19:25.272472+0000","last_fullsized":"2023-10-16T11:19:25.272472+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:27:01.350215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657403+0000","last_change":"2023-10-16T11:19:34.657403+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657403+0000","last_undegraded":"2023-10-16T11:19:34.657403+0000","last_fullsized":"2023-10-16T11:19:34.657403+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"216'3","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269055+0000","last_change":"2023-10-16T11:19:22.331150+0000","last_active":"2023-10-16T11:19:25.269055+0000","last_peered":"2023-10-16T11:19:25.269055+0000","last_clean":"2023-10-16T11:19:25.269055+0000","last_became_active":"2023-10-16T11:19:22.330820+0000","last_became_peered":"2023-10-16T11:19:22.330820+0000","last_unstale":"2023-10-16T11:19:25.269055+0000","last_undegraded":"2023-10-16T11:19:25.269055+0000","last_fullsized":"2023-10-16T11:19:25.269055+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.268982+0000","last_change":"2023-10-16T11:19:23.338568+0000","last_active":"2023-10-16T11:19:25.268982+0000","last_peered":"2023-10-16T11:19:25.268982+0000","last_clean":"2023-10-16T11:19:25.268982+0000","last_became_active":"2023-10-16T11:19:23.338280+0000","last_became_peered":"2023-10-16T11:19:23.338280+0000","last_unstale":"2023-10-16T11:19:25.268982+0000","last_undegraded":"2023-10-16T11:19:25.268982+0000","last_fullsized":"2023-10-16T11:19:25.268982+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:40.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.661663+0000","last_change":"2023-10-16T11:19:34.661663+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.661663+0000","last_undegraded":"2023-10-16T11:19:34.661663+0000","last_fullsized":"2023-10-16T11:19:34.661663+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272445+0000","last_change":"2023-10-16T11:19:22.325211+0000","last_active":"2023-10-16T11:19:25.272445+0000","last_peered":"2023-10-16T11:19:25.272445+0000","last_clean":"2023-10-16T11:19:25.272445+0000","last_became_active":"2023-10-16T11:19:22.324901+0000","last_became_peered":"2023-10-16T11:19:22.324901+0000","last_unstale":"2023-10-16T11:19:25.272445+0000","last_undegraded":"2023-10-16T11:19:25.272445+0000","last_fullsized":"2023-10-16T11:19:25.272445+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:46.432434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273093+0000","last_change":"2023-10-16T11:19:23.331144+0000","last_active":"2023-10-16T11:19:25.273093+0000","last_peered":"2023-10-16T11:19:25.273093+0000","last_clean":"2023-10-16T11:19:25.273093+0000","last_became_active":"2023-10-16T11:19:23.330824+0000","last_became_peered":"2023-10-16T11:19:23.330824+0000","last_unstale":"2023-10-16T11:19:25.273093+0000","last_undegraded":"2023-10-16T11:19:25.273093+0000","last_fullsized":"2023-10-16T11:19:25.273093+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:44:27.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657865+0000","last_change":"2023-10-16T11:19:34.657865+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657865+0000","last_undegraded":"2023-10-16T11:19:34.657865+0000","last_fullsized":"2023-10-16T11:19:34.657865+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272566+0000","last_change":"2023-10-16T11:19:22.332253+0000","last_active":"2023-10-16T11:19:25.272566+0000","last_peered":"2023-10-16T11:19:25.272566+0000","last_clean":"2023-10-16T11:19:25.272566+0000","last_became_active":"2023-10-16T11:19:22.331970+0000","last_became_peered":"2023-10-16T11:19:22.331970+0000","last_unstale":"2023-10-16T11:19:25.272566+0000","last_undegraded":"2023-10-16T11:19:25.272566+0000","last_fullsized":"2023-10-16T11:19:25.272566+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:19.740978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272967+0000","last_change":"2023-10-16T11:19:23.330327+0000","last_active":"2023-10-16T11:19:25.272967+0000","last_peered":"2023-10-16T11:19:25.272967+0000","last_clean":"2023-10-16T11:19:25.272967+0000","last_became_active":"2023-10-16T11:19:23.330040+0000","last_became_peered":"2023-10-16T11:19:23.330040+0000","last_unstale":"2023-10-16T11:19:25.272967+0000","last_undegraded":"2023-10-16T11:19:25.272967+0000","last_fullsized":"2023-10-16T11:19:25.272967+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659906+0000","last_change":"2023-10-16T11:19:34.659906+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659906+0000","last_undegraded":"2023-10-16T11:19:34.659906+0000","last_fullsized":"2023-10-16T11:19:34.659906+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273074+0000","last_change":"2023-10-16T11:19:22.330754+0000","last_active":"2023-10-16T11:19:25.273074+0000","last_peered":"2023-10-16T11:19:25.273074+0000","last_clean":"2023-10-16T11:19:25.273074+0000","last_became_active":"2023-10-16T11:19:22.330490+0000","last_became_peered":"2023-10-16T11:19:22.330490+0000","last_unstale":"2023-10-16T11:19:25.273074+0000","last_undegraded":"2023-10-16T11:19:25.273074+0000","last_fullsized":"2023-10-16T11:19:25.273074+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:17.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269123+0000","last_change":"2023-10-16T11:19:23.334130+0000","last_active":"2023-10-16T11:19:25.269123+0000","last_peered":"2023-10-16T11:19:25.269123+0000","last_clean":"2023-10-16T11:19:25.269123+0000","last_became_active":"2023-10-16T11:19:23.333073+0000","last_became_peered":"2023-10-16T11:19:23.333073+0000","last_unstale":"2023-10-16T11:19:25.269123+0000","last_undegraded":"2023-10-16T11:19:25.269123+0000","last_fullsized":"2023-10-16T11:19:25.269123+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:38.491191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657803+0000","last_change":"2023-10-16T11:19:34.657803+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657803+0000","last_undegraded":"2023-10-16T11:19:34.657803+0000","last_fullsized":"2023-10-16T11:19:34.657803+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272260+0000","last_change":"2023-10-16T11:19:22.328061+0000","last_active":"2023-10-16T11:19:25.272260+0000","last_peered":"2023-10-16T11:19:25.272260+0000","last_clean":"2023-10-16T11:19:25.272260+0000","last_became_active":"2023-10-16T11:19:22.327773+0000","last_became_peered":"2023-10-16T11:19:22.327773+0000","last_unstale":"2023-10-16T11:19:25.272260+0000","last_undegraded":"2023-10-16T11:19:25.272260+0000","last_fullsized":"2023-10-16T11:19:25.272260+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:39.077769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273059+0000","last_change":"2023-10-16T11:19:23.330616+0000","last_active":"2023-10-16T11:19:25.273059+0000","last_peered":"2023-10-16T11:19:25.273059+0000","last_clean":"2023-10-16T11:19:25.273059+0000","last_became_active":"2023-10-16T11:19:23.330317+0000","last_became_peered":"2023-10-16T11:19:23.330317+0000","last_unstale":"2023-10-16T11:19:25.273059+0000","last_undegraded":"2023-10-16T11:19:25.273059+0000","last_fullsized":"2023-10-16T11:19:25.273059+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.659497+0000","last_change":"2023-10-16T11:19:34.659497+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.659497+0000","last_undegraded":"2023-10-16T11:19:34.659497+0000","last_fullsized":"2023-10-16T11:19:34.659497+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272319+0000","last_change":"2023-10-16T11:19:22.328217+0000","last_active":"2023-10-16T11:19:25.272319+0000","last_peered":"2023-10-16T11:19:25.272319+0000","last_clean":"2023-10-16T11:19:25.272319+0000","last_became_active":"2023-10-16T11:19:22.327901+0000","last_became_peered":"2023-10-16T11:19:22.327901+0000","last_unstale":"2023-10-16T11:19:25.272319+0000","last_undegraded":"2023-10-16T11:19:25.272319+0000","last_fullsized":"2023-10-16T11:19:25.272319+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:48.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273032+0000","last_change":"2023-10-16T11:19:23.336335+0000","last_active":"2023-10-16T11:19:25.273032+0000","last_peered":"2023-10-16T11:19:25.273032+0000","last_clean":"2023-10-16T11:19:25.273032+0000","last_became_active":"2023-10-16T11:19:23.335852+0000","last_became_peered":"2023-10-16T11:19:23.335852+0000","last_unstale":"2023-10-16T11:19:25.273032+0000","last_undegraded":"2023-10-16T11:19:25.273032+0000","last_fullsized":"2023-10-16T11:19:25.273032+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.660104+0000","last_change":"2023-10-16T11:19:34.660104+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.660104+0000","last_undegraded":"2023-10-16T11:19:34.660104+0000","last_fullsized":"2023-10-16T11:19:34.660104+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269042+0000","last_change":"2023-10-16T11:19:22.329968+0000","last_active":"2023-10-16T11:19:25.269042+0000","last_peered":"2023-10-16T11:19:25.269042+0000","last_clean":"2023-10-16T11:19:25.269042+0000","last_became_active":"2023-10-16T11:19:22.326360+0000","last_became_peered":"2023-10-16T11:19:22.326360+0000","last_unstale":"2023-10-16T11:19:25.269042+0000","last_undegraded":"2023-10-16T11:19:25.269042+0000","last_fullsized":"2023-10-16T11:19:25.269042+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272461+0000","last_change":"2023-10-16T11:19:23.335733+0000","last_active":"2023-10-16T11:19:25.272461+0000","last_peered":"2023-10-16T11:19:25.272461+0000","last_clean":"2023-10-16T11:19:25.272461+0000","last_became_active":"2023-10-16T11:19:23.334936+0000","last_became_peered":"2023-10-16T11:19:23.334936+0000","last_unstale":"2023-10-16T11:19:25.272461+0000","last_undegraded":"2023-10-16T11:19:25.272461+0000","last_fullsized":"2023-10-16T11:19:25.272461+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:08.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.656791+0000","last_change":"2023-10-16T11:19:34.656791+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.656791+0000","last_undegraded":"2023-10-16T11:19:34.656791+0000","last_fullsized":"2023-10-16T11:19:34.656791+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"216'2","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272725+0000","last_change":"2023-10-16T11:19:22.328385+0000","last_active":"2023-10-16T11:19:25.272725+0000","last_peered":"2023-10-16T11:19:25.272725+0000","last_clean":"2023-10-16T11:19:25.272725+0000","last_became_active":"2023-10-16T11:19:22.328096+0000","last_became_peered":"2023-10-16T11:19:22.328096+0000","last_unstale":"2023-10-16T11:19:25.272725+0000","last_undegraded":"2023-10-16T11:19:25.272725+0000","last_fullsized":"2023-10-16T11:19:25.272725+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:40.312176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272188+0000","last_change":"2023-10-16T11:19:23.326944+0000","last_active":"2023-10-16T11:19:25.272188+0000","last_peered":"2023-10-16T11:19:25.272188+0000","last_clean":"2023-10-16T11:19:25.272188+0000","last_became_active":"2023-10-16T11:19:23.326680+0000","last_became_peered":"2023-10-16T11:19:23.326680+0000","last_unstale":"2023-10-16T11:19:25.272188+0000","last_undegraded":"2023-10-16T11:19:25.272188+0000","last_fullsized":"2023-10-16T11:19:25.272188+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:28.070345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.657531+0000","last_change":"2023-10-16T11:19:34.657531+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.657531+0000","last_undegraded":"2023-10-16T11:19:34.657531+0000","last_fullsized":"2023-10-16T11:19:34.657531+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"216'1","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272648+0000","last_change":"2023-10-16T11:19:22.332064+0000","last_active":"2023-10-16T11:19:25.272648+0000","last_peered":"2023-10-16T11:19:25.272648+0000","last_clean":"2023-10-16T11:19:25.272648+0000","last_became_active":"2023-10-16T11:19:22.331746+0000","last_became_peered":"2023-10-16T11:19:22.331746+0000","last_unstale":"2023-10-16T11:19:25.272648+0000","last_undegraded":"2023-10-16T11:19:25.272648+0000","last_fullsized":"2023-10-16T11:19:25.272648+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272551+0000","last_change":"2023-10-16T11:19:23.342219+0000","last_active":"2023-10-16T11:19:25.272551+0000","last_peered":"2023-10-16T11:19:25.272551+0000","last_clean":"2023-10-16T11:19:25.272551+0000","last_became_active":"2023-10-16T11:19:23.334494+0000","last_became_peered":"2023-10-16T11:19:23.334494+0000","last_unstale":"2023-10-16T11:19:25.272551+0000","last_undegraded":"2023-10-16T11:19:25.272551+0000","last_fullsized":"2023-10-16T11:19:25.272551+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654954+0000","last_change":"2023-10-16T11:19:34.654954+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654954+0000","last_undegraded":"2023-10-16T11:19:34.654954+0000","last_fullsized":"2023-10-16T11:19:34.654954+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"216'4","reported_seq":17,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273074+0000","last_change":"2023-10-16T11:19:22.332308+0000","last_active":"2023-10-16T11:19:25.273074+0000","last_peered":"2023-10-16T11:19:25.273074+0000","last_clean":"2023-10-16T11:19:25.273074+0000","last_became_active":"2023-10-16T11:19:22.332005+0000","last_became_peered":"2023-10-16T11:19:22.332005+0000","last_unstale":"2023-10-16T11:19:25.273074+0000","last_undegraded":"2023-10-16T11:19:25.273074+0000","last_fullsized":"2023-10-16T11:19:25.273074+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272625+0000","last_change":"2023-10-16T11:19:23.341701+0000","last_active":"2023-10-16T11:19:25.272625+0000","last_peered":"2023-10-16T11:19:25.272625+0000","last_clean":"2023-10-16T11:19:25.272625+0000","last_became_active":"2023-10-16T11:19:23.341424+0000","last_became_peered":"2023-10-16T11:19:23.341424+0000","last_unstale":"2023-10-16T11:19:25.272625+0000","last_undegraded":"2023-10-16T11:19:25.272625+0000","last_fullsized":"2023-10-16T11:19:25.272625+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:25.448895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":218,"state":"creating+peering","last_fresh":"2023-10-16T11:19:34.654310+0000","last_change":"2023-10-16T11:19:34.654310+0000","last_active":"2023-10-16T11:19:34.646267+0000","last_peered":"2023-10-16T11:19:34.646267+0000","last_clean":"2023-10-16T11:19:34.646267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:34.654310+0000","last_undegraded":"2023-10-16T11:19:34.654310+0000","last_fullsized":"2023-10-16T11:19:34.654310+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272205+0000","last_change":"2023-10-16T11:19:23.323955+0000","last_active":"2023-10-16T11:19:25.272205+0000","last_peered":"2023-10-16T11:19:25.272205+0000","last_clean":"2023-10-16T11:19:25.272205+0000","last_became_active":"2023-10-16T11:19:23.323706+0000","last_became_peered":"2023-10-16T11:19:23.323706+0000","last_unstale":"2023-10-16T11:19:25.272205+0000","last_undegraded":"2023-10-16T11:19:25.272205+0000","last_fullsized":"2023-10-16T11:19:25.272205+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.182127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272240+0000","last_change":"2023-10-16T11:19:23.332826+0000","last_active":"2023-10-16T11:19:25.272240+0000","last_peered":"2023-10-16T11:19:25.272240+0000","last_clean":"2023-10-16T11:19:25.272240+0000","last_became_active":"2023-10-16T11:19:23.332541+0000","last_became_peered":"2023-10-16T11:19:23.332541+0000","last_unstale":"2023-10-16T11:19:25.272240+0000","last_undegraded":"2023-10-16T11:19:25.272240+0000","last_fullsized":"2023-10-16T11:19:25.272240+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:12:33.418278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273019+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:25.273019+0000","last_peered":"2023-10-16T11:19:25.273019+0000","last_clean":"2023-10-16T11:19:25.273019+0000","last_became_active":"2023-10-16T11:19:23.335534+0000","last_became_peered":"2023-10-16T11:19:23.335534+0000","last_unstale":"2023-10-16T11:19:25.273019+0000","last_undegraded":"2023-10-16T11:19:25.273019+0000","last_fullsized":"2023-10-16T11:19:25.273019+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269029+0000","last_change":"2023-10-16T11:19:23.330876+0000","last_active":"2023-10-16T11:19:25.269029+0000","last_peered":"2023-10-16T11:19:25.269029+0000","last_clean":"2023-10-16T11:19:25.269029+0000","last_became_active":"2023-10-16T11:19:23.330597+0000","last_became_peered":"2023-10-16T11:19:23.330597+0000","last_unstale":"2023-10-16T11:19:25.269029+0000","last_undegraded":"2023-10-16T11:19:25.269029+0000","last_fullsized":"2023-10-16T11:19:25.269029+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272544+0000","last_change":"2023-10-16T11:19:23.327468+0000","last_active":"2023-10-16T11:19:25.272544+0000","last_peered":"2023-10-16T11:19:25.272544+0000","last_clean":"2023-10-16T11:19:25.272544+0000","last_became_active":"2023-10-16T11:19:23.327176+0000","last_became_peered":"2023-10-16T11:19:23.327176+0000","last_unstale":"2023-10-16T11:19:25.272544+0000","last_undegraded":"2023-10-16T11:19:25.272544+0000","last_fullsized":"2023-10-16T11:19:25.272544+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:02.654427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273186+0000","last_change":"2023-10-16T11:19:23.332619+0000","last_active":"2023-10-16T11:19:25.273186+0000","last_peered":"2023-10-16T11:19:25.273186+0000","last_clean":"2023-10-16T11:19:25.273186+0000","last_became_active":"2023-10-16T11:19:23.332037+0000","last_became_peered":"2023-10-16T11:19:23.332037+0000","last_unstale":"2023-10-16T11:19:25.273186+0000","last_undegraded":"2023-10-16T11:19:25.273186+0000","last_fullsized":"2023-10-16T11:19:25.273186+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:48.332974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272635+0000","last_change":"2023-10-16T11:19:23.341307+0000","last_active":"2023-10-16T11:19:25.272635+0000","last_peered":"2023-10-16T11:19:25.272635+0000","last_clean":"2023-10-16T11:19:25.272635+0000","last_became_active":"2023-10-16T11:19:23.334457+0000","last_became_peered":"2023-10-16T11:19:23.334457+0000","last_unstale":"2023-10-16T11:19:25.272635+0000","last_undegraded":"2023-10-16T11:19:25.272635+0000","last_fullsized":"2023-10-16T11:19:25.272635+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269114+0000","last_change":"2023-10-16T11:19:23.334306+0000","last_active":"2023-10-16T11:19:25.269114+0000","last_peered":"2023-10-16T11:19:25.269114+0000","last_clean":"2023-10-16T11:19:25.269114+0000","last_became_active":"2023-10-16T11:19:23.333460+0000","last_became_peered":"2023-10-16T11:19:23.333460+0000","last_unstale":"2023-10-16T11:19:25.269114+0000","last_undegraded":"2023-10-16T11:19:25.269114+0000","last_fullsized":"2023-10-16T11:19:25.269114+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:53.175386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272352+0000","last_change":"2023-10-16T11:19:23.341264+0000","last_active":"2023-10-16T11:19:25.272352+0000","last_peered":"2023-10-16T11:19:25.272352+0000","last_clean":"2023-10-16T11:19:25.272352+0000","last_became_active":"2023-10-16T11:19:23.340921+0000","last_became_peered":"2023-10-16T11:19:23.340921+0000","last_unstale":"2023-10-16T11:19:25.272352+0000","last_undegraded":"2023-10-16T11:19:25.272352+0000","last_fullsized":"2023-10-16T11:19:25.272352+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.710291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272645+0000","last_change":"2023-10-16T11:19:23.341165+0000","last_active":"2023-10-16T11:19:25.272645+0000","last_peered":"2023-10-16T11:19:25.272645+0000","last_clean":"2023-10-16T11:19:25.272645+0000","last_became_active":"2023-10-16T11:19:23.340853+0000","last_became_peered":"2023-10-16T11:19:23.340853+0000","last_unstale":"2023-10-16T11:19:25.272645+0000","last_undegraded":"2023-10-16T11:19:25.272645+0000","last_fullsized":"2023-10-16T11:19:25.272645+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:41.539293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272168+0000","last_change":"2023-10-16T11:19:23.341632+0000","last_active":"2023-10-16T11:19:25.272168+0000","last_peered":"2023-10-16T11:19:25.272168+0000","last_clean":"2023-10-16T11:19:25.272168+0000","last_became_active":"2023-10-16T11:19:23.341312+0000","last_became_peered":"2023-10-16T11:19:23.341312+0000","last_unstale":"2023-10-16T11:19:25.272168+0000","last_undegraded":"2023-10-16T11:19:25.272168+0000","last_fullsized":"2023-10-16T11:19:25.272168+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272212+0000","last_change":"2023-10-16T11:19:23.334105+0000","last_active":"2023-10-16T11:19:25.272212+0000","last_peered":"2023-10-16T11:19:25.272212+0000","last_clean":"2023-10-16T11:19:25.272212+0000","last_became_active":"2023-10-16T11:19:23.333836+0000","last_became_peered":"2023-10-16T11:19:23.333836+0000","last_unstale":"2023-10-16T11:19:25.272212+0000","last_undegraded":"2023-10-16T11:19:25.272212+0000","last_fullsized":"2023-10-16T11:19:25.272212+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.432829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272269+0000","last_change":"2023-10-16T11:19:23.331248+0000","last_active":"2023-10-16T11:19:25.272269+0000","last_peered":"2023-10-16T11:19:25.272269+0000","last_clean":"2023-10-16T11:19:25.272269+0000","last_became_active":"2023-10-16T11:19:23.330931+0000","last_became_peered":"2023-10-16T11:19:23.330931+0000","last_unstale":"2023-10-16T11:19:25.272269+0000","last_undegraded":"2023-10-16T11:19:25.272269+0000","last_fullsized":"2023-10-16T11:19:25.272269+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272294+0000","last_change":"2023-10-16T11:19:23.337766+0000","last_active":"2023-10-16T11:19:25.272294+0000","last_peered":"2023-10-16T11:19:25.272294+0000","last_clean":"2023-10-16T11:19:25.272294+0000","last_became_active":"2023-10-16T11:19:23.337503+0000","last_became_peered":"2023-10-16T11:19:23.337503+0000","last_unstale":"2023-10-16T11:19:25.272294+0000","last_undegraded":"2023-10-16T11:19:25.272294+0000","last_fullsized":"2023-10-16T11:19:25.272294+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269066+0000","last_change":"2023-10-16T11:19:23.333633+0000","last_active":"2023-10-16T11:19:25.269066+0000","last_peered":"2023-10-16T11:19:25.269066+0000","last_clean":"2023-10-16T11:19:25.269066+0000","last_became_active":"2023-10-16T11:19:23.333280+0000","last_became_peered":"2023-10-16T11:19:23.333280+0000","last_unstale":"2023-10-16T11:19:25.269066+0000","last_undegraded":"2023-10-16T11:19:25.269066+0000","last_fullsized":"2023-10-16T11:19:25.269066+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269104+0000","last_change":"2023-10-16T11:19:23.334483+0000","last_active":"2023-10-16T11:19:25.269104+0000","last_peered":"2023-10-16T11:19:25.269104+0000","last_clean":"2023-10-16T11:19:25.269104+0000","last_became_active":"2023-10-16T11:19:23.333819+0000","last_became_peered":"2023-10-16T11:19:23.333819+0000","last_unstale":"2023-10-16T11:19:25.269104+0000","last_undegraded":"2023-10-16T11:19:25.269104+0000","last_fullsized":"2023-10-16T11:19:25.269104+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273051+0000","last_change":"2023-10-16T11:19:23.332953+0000","last_active":"2023-10-16T11:19:25.273051+0000","last_peered":"2023-10-16T11:19:25.273051+0000","last_clean":"2023-10-16T11:19:25.273051+0000","last_became_active":"2023-10-16T11:19:23.332662+0000","last_became_peered":"2023-10-16T11:19:23.332662+0000","last_unstale":"2023-10-16T11:19:25.273051+0000","last_undegraded":"2023-10-16T11:19:25.273051+0000","last_fullsized":"2023-10-16T11:19:25.273051+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:27.611049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272989+0000","last_change":"2023-10-16T11:19:23.325666+0000","last_active":"2023-10-16T11:19:25.272989+0000","last_peered":"2023-10-16T11:19:25.272989+0000","last_clean":"2023-10-16T11:19:25.272989+0000","last_became_active":"2023-10-16T11:19:23.325400+0000","last_became_peered":"2023-10-16T11:19:23.325400+0000","last_unstale":"2023-10-16T11:19:25.272989+0000","last_undegraded":"2023-10-16T11:19:25.272989+0000","last_fullsized":"2023-10-16T11:19:25.272989+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272156+0000","last_change":"2023-10-16T11:19:23.342633+0000","last_active":"2023-10-16T11:19:25.272156+0000","last_peered":"2023-10-16T11:19:25.272156+0000","last_clean":"2023-10-16T11:19:25.272156+0000","last_became_active":"2023-10-16T11:19:23.342346+0000","last_became_peered":"2023-10-16T11:19:23.342346+0000","last_unstale":"2023-10-16T11:19:25.272156+0000","last_undegraded":"2023-10-16T11:19:25.272156+0000","last_fullsized":"2023-10-16T11:19:25.272156+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:12.087797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272298+0000","last_change":"2023-10-16T11:19:23.330180+0000","last_active":"2023-10-16T11:19:25.272298+0000","last_peered":"2023-10-16T11:19:25.272298+0000","last_clean":"2023-10-16T11:19:25.272298+0000","last_became_active":"2023-10-16T11:19:23.329598+0000","last_became_peered":"2023-10-16T11:19:23.329598+0000","last_unstale":"2023-10-16T11:19:25.272298+0000","last_undegraded":"2023-10-16T11:19:25.272298+0000","last_fullsized":"2023-10-16T11:19:25.272298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:59.613377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272530+0000","last_change":"2023-10-16T11:19:23.335413+0000","last_active":"2023-10-16T11:19:25.272530+0000","last_peered":"2023-10-16T11:19:25.272530+0000","last_clean":"2023-10-16T11:19:25.272530+0000","last_became_active":"2023-10-16T11:19:23.334994+0000","last_became_peered":"2023-10-16T11:19:23.334994+0000","last_unstale":"2023-10-16T11:19:25.272530+0000","last_undegraded":"2023-10-16T11:19:25.272530+0000","last_fullsized":"2023-10-16T11:19:25.272530+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:53.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273167+0000","last_change":"2023-10-16T11:19:23.333705+0000","last_active":"2023-10-16T11:19:25.273167+0000","last_peered":"2023-10-16T11:19:25.273167+0000","last_clean":"2023-10-16T11:19:25.273167+0000","last_became_active":"2023-10-16T11:19:23.333009+0000","last_became_peered":"2023-10-16T11:19:23.333009+0000","last_unstale":"2023-10-16T11:19:25.273167+0000","last_undegraded":"2023-10-16T11:19:25.273167+0000","last_fullsized":"2023-10-16T11:19:25.273167+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:41.811226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272308+0000","last_change":"2023-10-16T11:19:23.333382+0000","last_active":"2023-10-16T11:19:25.272308+0000","last_peered":"2023-10-16T11:19:25.272308+0000","last_clean":"2023-10-16T11:19:25.272308+0000","last_became_active":"2023-10-16T11:19:23.332643+0000","last_became_peered":"2023-10-16T11:19:23.332643+0000","last_unstale":"2023-10-16T11:19:25.272308+0000","last_undegraded":"2023-10-16T11:19:25.272308+0000","last_fullsized":"2023-10-16T11:19:25.272308+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:54.101849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.269094+0000","last_change":"2023-10-16T11:19:23.336597+0000","last_active":"2023-10-16T11:19:25.269094+0000","last_peered":"2023-10-16T11:19:25.269094+0000","last_clean":"2023-10-16T11:19:25.269094+0000","last_became_active":"2023-10-16T11:19:23.336307+0000","last_became_peered":"2023-10-16T11:19:23.336307+0000","last_unstale":"2023-10-16T11:19:25.269094+0000","last_undegraded":"2023-10-16T11:19:25.269094+0000","last_fullsized":"2023-10-16T11:19:25.269094+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272100+0000","last_change":"2023-10-16T11:19:23.329954+0000","last_active":"2023-10-16T11:19:25.272100+0000","last_peered":"2023-10-16T11:19:25.272100+0000","last_clean":"2023-10-16T11:19:25.272100+0000","last_became_active":"2023-10-16T11:19:23.329688+0000","last_became_peered":"2023-10-16T11:19:23.329688+0000","last_unstale":"2023-10-16T11:19:25.272100+0000","last_undegraded":"2023-10-16T11:19:25.272100+0000","last_fullsized":"2023-10-16T11:19:25.272100+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:11.183201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272421+0000","last_change":"2023-10-16T11:19:23.329203+0000","last_active":"2023-10-16T11:19:25.272421+0000","last_peered":"2023-10-16T11:19:25.272421+0000","last_clean":"2023-10-16T11:19:25.272421+0000","last_became_active":"2023-10-16T11:19:23.328748+0000","last_became_peered":"2023-10-16T11:19:23.328748+0000","last_unstale":"2023-10-16T11:19:25.272421+0000","last_undegraded":"2023-10-16T11:19:25.272421+0000","last_fullsized":"2023-10-16T11:19:25.272421+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:09.942459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272279+0000","last_change":"2023-10-16T11:19:23.326663+0000","last_active":"2023-10-16T11:19:25.272279+0000","last_peered":"2023-10-16T11:19:25.272279+0000","last_clean":"2023-10-16T11:19:25.272279+0000","last_became_active":"2023-10-16T11:19:23.326378+0000","last_became_peered":"2023-10-16T11:19:23.326378+0000","last_unstale":"2023-10-16T11:19:25.272279+0000","last_undegraded":"2023-10-16T11:19:25.272279+0000","last_fullsized":"2023-10-16T11:19:25.272279+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:41.823015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272442+0000","last_change":"2023-10-16T11:19:23.330646+0000","last_active":"2023-10-16T11:19:25.272442+0000","last_peered":"2023-10-16T11:19:25.272442+0000","last_clean":"2023-10-16T11:19:25.272442+0000","last_became_active":"2023-10-16T11:19:23.330371+0000","last_became_peered":"2023-10-16T11:19:23.330371+0000","last_unstale":"2023-10-16T11:19:25.272442+0000","last_undegraded":"2023-10-16T11:19:25.272442+0000","last_fullsized":"2023-10-16T11:19:25.272442+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:55.845569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272341+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:25.272341+0000","last_peered":"2023-10-16T11:19:25.272341+0000","last_clean":"2023-10-16T11:19:25.272341+0000","last_became_active":"2023-10-16T11:19:23.335513+0000","last_became_peered":"2023-10-16T11:19:23.335513+0000","last_unstale":"2023-10-16T11:19:25.272341+0000","last_undegraded":"2023-10-16T11:19:25.272341+0000","last_fullsized":"2023-10-16T11:19:25.272341+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:30.545707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.273148+0000","last_change":"2023-10-16T11:19:23.333266+0000","last_active":"2023-10-16T11:19:25.273148+0000","last_peered":"2023-10-16T11:19:25.273148+0000","last_clean":"2023-10-16T11:19:25.273148+0000","last_became_active":"2023-10-16T11:19:23.332412+0000","last_became_peered":"2023-10-16T11:19:23.332412+0000","last_unstale":"2023-10-16T11:19:25.273148+0000","last_undegraded":"2023-10-16T11:19:25.273148+0000","last_fullsized":"2023-10-16T11:19:25.273148+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:19.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272755+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:25.272755+0000","last_peered":"2023-10-16T11:19:25.272755+0000","last_clean":"2023-10-16T11:19:25.272755+0000","last_became_active":"2023-10-16T11:19:23.331671+0000","last_became_peered":"2023-10-16T11:19:23.331671+0000","last_unstale":"2023-10-16T11:19:25.272755+0000","last_undegraded":"2023-10-16T11:19:25.272755+0000","last_fullsized":"2023-10-16T11:19:25.272755+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:13.976802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T11:19:35.651650+0000","last_change":"2023-10-16T11:19:35.651650+0000","last_active":"2023-10-16T11:19:35.651650+0000","last_peered":"2023-10-16T11:19:35.651650+0000","last_clean":"2023-10-16T11:19:35.651650+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T11:19:35.651650+0000","last_undegraded":"2023-10-16T11:19:35.651650+0000","last_fullsized":"2023-10-16T11:19:35.651650+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-16T11:19:25.272675+0000","last_change":"2023-10-16T11:19:23.334441+0000","last_active":"2023-10-16T11:19:25.272675+0000","last_peered":"2023-10-16T11:19:25.272675+0000","last_clean":"2023-10-16T11:19:25.272675+0000","last_became_active":"2023-10-16T11:19:23.334140+0000","last_became_peered":"2023-10-16T11:19:23.334140+0000","last_unstale":"2023-10-16T11:19:25.272675+0000","last_undegraded":"2023-10-16T11:19:25.272675+0000","last_fullsized":"2023-10-16T11:19:25.272675+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575991,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52500,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319340,"statfs":{"total":96636764160,"available":96583004160,"internally_reserved":0,"allocated":3350528,"data_stored":2060152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.67200000000000004,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.372},"max":{"1min":9.4359999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.68400000000000005,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":9.3059999999999992,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.71599999999999997,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":9.3610000000000007,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.119,"5min":0.71999999999999997,"15min":0.68100000000000005},"min":{"1min":0.49399999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.3960000000000008,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.72099999999999997,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":9.2889999999999997,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.753,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.71699999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.36099999999999999},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.442},{"interface":"front","average":{"1min":1.157,"5min":0.76500000000000001,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":9.4689999999999994,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.69899999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.61199999999999999,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.746},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.63100000000000001,"15min":0.56699999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.5399999999999991,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.64000000000000001,"15min":0.57399999999999995},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.69399999999999995,"15min":0.63100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.5690000000000008,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.625}]},{"osd":6,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.71199999999999997,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.155,"5min":0.69799999999999995,"15min":0.64500000000000002},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.58499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38016,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2744320,"data_stored":1462821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.91000000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.70599999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.69099999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64000000000000001,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.497},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.335,"5min":3.335,"15min":3.335},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.62}]}]},{"osd":1,"up_from":13,"seq":55834575992,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47744,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324096,"statfs":{"total":96636764160,"available":96587874304,"internally_reserved":0,"allocated":2740224,"data_stored":1466855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40899999999999997,"15min":0.41399999999999998},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.622,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.32300000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63900000000000001,"5min":0.64100000000000001,"15min":0.71699999999999997},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.46400000000000002,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.62,"5min":0.66600000000000004,"15min":0.70099999999999996},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54100000000000004,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.441,"15min":0.443},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53200000000000003,"5min":0.623,"15min":0.70499999999999996},"last":0.34000000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82099999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":1.0309999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88900000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90100000000000002,"5min":1.01,"15min":1.0760000000000001},"last":0.626}]},{"osd":6,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81999999999999995,"5min":0.93100000000000005,"15min":1.0149999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.68799999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.70299999999999996,"15min":0.70399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83399999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.81599999999999995,"5min":0.99399999999999999,"15min":1.016},"last":0.68000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575992,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42760,"kb_used_data":3260,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329080,"statfs":{"total":96636764160,"available":96592977920,"internally_reserved":0,"allocated":3338240,"data_stored":2057313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44400000000000001,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60699999999999998,"5min":1.272,"15min":1.272},"last":0.318},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67700000000000005,"5min":1.351,"15min":1.351},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.46800000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.875,"5min":1.391,"15min":1.391},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80000000000000004,"5min":0.874,"15min":1.089},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.95599999999999996,"15min":1.216},"last":0.56799999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.69799999999999995},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90900000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.47199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89100000000000001,"5min":1.038,"15min":1.1180000000000001},"last":0.64100000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.54600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575991,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47748,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324092,"statfs":{"total":96636764160,"available":96587870208,"internally_reserved":0,"allocated":2744320,"data_stored":1466945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.63300000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.29999999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69799999999999995,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77800000000000002},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.95399999999999996,"15min":1.097},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67200000000000004,"15min":0.68200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.91800000000000004,"5min":0.96199999999999997,"15min":1.016},"last":0.66100000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90500000000000003,"5min":1.022,"15min":1.0389999999999999},"last":0.629}]},{"osd":6,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91800000000000004,"5min":1.022,"15min":1.022},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91900000000000004,"5min":0.98999999999999999,"15min":1.004},"last":0.60999999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90400000000000003,"5min":1.0569999999999999,"15min":1.071},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.57899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51916,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319924,"statfs":{"total":96636764160,"available":96583602176,"internally_reserved":0,"allocated":2752512,"data_stored":1466981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38100000000000001,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.68700000000000006},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":1.0029999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.60899999999999999,"15min":0.67300000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.74299999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.58999999999999997,"5min":0.66800000000000004,"15min":0.96299999999999997},"last":0.308},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64200000000000002,"5min":0.69899999999999995,"15min":1.024},"last":0.375}]},{"osd":4,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90600000000000003,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.99099999999999999,"15min":1.026},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0640000000000001},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41799999999999998},"max":{"1min":0.85999999999999999,"5min":0.92900000000000005,"15min":1.0369999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.0620000000000001},"last":0.625}]},{"osd":7,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":1.002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88400000000000001,"5min":1.006,"15min":1.1899999999999999},"last":0.53900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575991,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47748,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324092,"statfs":{"total":96636764160,"available":96587870208,"internally_reserved":0,"allocated":2744320,"data_stored":1467437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.625},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.623},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71599999999999997,"5min":0.82499999999999996,"15min":0.90700000000000003},"last":0.441},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58799999999999997,"15min":0.623},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.624}]},{"osd":2,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60599999999999998,"15min":0.63900000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.84399999999999997,"15min":0.88400000000000001},"last":0.497}]},{"osd":3,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.624},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.83799999999999997,"15min":0.995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.51500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.41899999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.90500000000000003,"15min":1.115},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.35499999999999998,"15min":0.34200000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.83899999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.64200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575991,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64388,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307452,"statfs":{"total":96636764160,"available":96570830848,"internally_reserved":0,"allocated":2736128,"data_stored":1459436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.88600000000000001},"last":0.48099999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60299999999999998},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.86499999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.91400000000000003},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.498},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40899999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85899999999999999,"5min":0.878,"15min":0.878},"last":0.436}]},{"osd":7,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55300000000000005,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:19:41.193 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:19:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.342+0000 7fe899128700 1 Processor -- start 2023-10-16T11:19:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.342+0000 7fe899128700 1 -- start start 2023-10-16T11:19:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe899128700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe89406ab40 0x7fe89406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe899128700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe899128700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe894071e50 con 0x7fe89406e5d0 2023-10-16T11:19:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe899128700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe894161270 con 0x7fe89406ab40 2023-10-16T11:19:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe899128700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe894161510 con 0x7fe8940729b0 2023-10-16T11:19:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe892d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe89406ab40 0x7fe89406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe892d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe89406ab40 0x7fe89406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38460/0 (socket says 172.21.15.158:38460) 2023-10-16T11:19:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.343+0000 7fe892d9d700 1 -- 172.21.15.158:0/3630933859 learned_addr learned my addr 172.21.15.158:0/3630933859 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe89359e700 1 --2- 172.21.15.158:0/3630933859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe892d9d700 1 -- 172.21.15.158:0/3630933859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 msgr2=0x7fe894071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe892d9d700 1 --2- 172.21.15.158:0/3630933859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe892d9d700 1 -- 172.21.15.158:0/3630933859 >> v1:172.21.15.172:6789/0 conn(0x7fe89406e5d0 legacy=0x7fe8940722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe892d9d700 1 -- 172.21.15.158:0/3630933859 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8941617f0 con 0x7fe89406ab40 2023-10-16T11:19:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe892d9d700 1 --2- 172.21.15.158:0/3630933859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe89406d820 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe8880098f0 tx=0x7fe888009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c4d4130f6cb513 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:41.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe891d9b700 1 -- 172.21.15.158:0/3630933859 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888004230 con 0x7fe89406ab40 2023-10-16T11:19:41.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe891d9b700 1 -- 172.21.15.158:0/3630933859 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888004390 con 0x7fe89406ab40 2023-10-16T11:19:41.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe891d9b700 1 -- 172.21.15.158:0/3630933859 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880036a0 con 0x7fe89406ab40 2023-10-16T11:19:41.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe899128700 1 -- 172.21.15.158:0/3630933859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 msgr2=0x7fe89406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.344+0000 7fe899128700 1 --2- 172.21.15.158:0/3630933859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe89406d820 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe8880098f0 tx=0x7fe888009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:41.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.345+0000 7fe899128700 1 -- 172.21.15.158:0/3630933859 shutdown_connections 2023-10-16T11:19:41.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.345+0000 7fe899128700 1 --2- 172.21.15.158:0/3630933859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.345+0000 7fe899128700 1 --2- 172.21.15.158:0/3630933859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe89406d820 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.345+0000 7fe899128700 1 -- 172.21.15.158:0/3630933859 >> 172.21.15.158:0/3630933859 conn(0x7fe8940d02b0 msgr2=0x7fe8940d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:41.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.345+0000 7fe899128700 1 -- 172.21.15.158:0/3630933859 shutdown_connections 2023-10-16T11:19:41.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.345+0000 7fe899128700 1 -- 172.21.15.158:0/3630933859 wait complete. 2023-10-16T11:19:41.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.346+0000 7fe899128700 1 Processor -- start 2023-10-16T11:19:41.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.346+0000 7fe899128700 1 -- start start 2023-10-16T11:19:41.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.347+0000 7fe899128700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe8940b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.347+0000 7fe899128700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe89406e5d0 0x7fe8941f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.347+0000 7fe899128700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.347+0000 7fe899128700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe894160e70 con 0x7fe89406ab40 2023-10-16T11:19:41.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.347+0000 7fe899128700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940c5860 con 0x7fe89406e5d0 2023-10-16T11:19:41.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.347+0000 7fe899128700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940c5b40 con 0x7fe8940729b0 2023-10-16T11:19:41.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe8940b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe8940b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38466/0 (socket says 172.21.15.158:38466) 2023-10-16T11:19:41.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 -- 172.21.15.158:0/2955214654 learned_addr learned my addr 172.21.15.158:0/2955214654 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:41.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe89259c700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe89406e5d0 0x7fe8941f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe89359e700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 -- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 msgr2=0x7fe894148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 -- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe89406e5d0 msgr2=0x7fe8941f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe89406e5d0 0x7fe8941f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 -- 172.21.15.158:0/2955214654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe888009550 con 0x7fe89406ab40 2023-10-16T11:19:41.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe89359e700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:41.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe892d9d700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe8940b23d0 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe894074ec0 tx=0x7fe888004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:41.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe87b7fe700 1 -- 172.21.15.158:0/2955214654 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888003dc0 con 0x7fe89406ab40 2023-10-16T11:19:41.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe87b7fe700 1 -- 172.21.15.158:0/2955214654 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88801f070 con 0x7fe89406ab40 2023-10-16T11:19:41.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe87b7fe700 1 -- 172.21.15.158:0/2955214654 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88800b8a0 con 0x7fe89406ab40 2023-10-16T11:19:41.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8940c5e20 con 0x7fe89406ab40 2023-10-16T11:19:41.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.348+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8941f2af0 con 0x7fe89406ab40 2023-10-16T11:19:41.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.349+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8940681b0 con 0x7fe89406ab40 2023-10-16T11:19:41.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.350+0000 7fe87b7fe700 1 -- 172.21.15.158:0/2955214654 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe888021030 con 0x7fe89406ab40 2023-10-16T11:19:41.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.350+0000 7fe87b7fe700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe88006b020 0x7fe88006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.350+0000 7fe87b7fe700 1 -- 172.21.15.158:0/2955214654 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fe88808b400 con 0x7fe89406ab40 2023-10-16T11:19:41.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.350+0000 7fe89259c700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe88006b020 0x7fe88006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.350+0000 7fe89259c700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe88006b020 0x7fe88006d4e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe8940cf0a0 tx=0x7fe884007ec0 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:41.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.355+0000 7fe87b7fe700 1 -- 172.21.15.158:0/2955214654 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe888055db0 con 0x7fe89406ab40 2023-10-16T11:19:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.543+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe8940c56e0 con 0x7fe88006b020 2023-10-16T11:19:41.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.556+0000 7fe87b7fe700 1 -- 172.21.15.158:0/2955214654 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528859 (secure 0 0 0) 0x7fe884003a90 con 0x7fe88006b020 2023-10-16T11:19:41.564 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:41.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe88006b020 msgr2=0x7fe88006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe88006b020 0x7fe88006d4e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe8940cf0a0 tx=0x7fe884007ec0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 msgr2=0x7fe8940b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe8940b23d0 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe894074ec0 tx=0x7fe888004750 comp rx=0 tx=0).stop 2023-10-16T11:19:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 shutdown_connections 2023-10-16T11:19:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8940729b0 0x7fe894148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fe88006b020 0x7fe88006d4e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89406ab40 0x7fe8940b23d0 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 --2- 172.21.15.158:0/2955214654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe89406e5d0 0x7fe8941f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 >> 172.21.15.158:0/2955214654 conn(0x7fe8940d02b0 msgr2=0x7fe8940ae7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 shutdown_connections 2023-10-16T11:19:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.561+0000 7fe899128700 1 -- 172.21.15.158:0/2955214654 wait complete. 2023-10-16T11:19:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:19:41.575 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-16T11:19:41.276814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":376,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":411268,"kb_used_data":22984,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754563452,"statfs":{"total":773094113280,"available":772672974848,"internally_reserved":0,"allocated":23535616,"data_stored":13122494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":397463095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001296"},"pg_stats":[{"pgid":"1.0","version":"207'348","reported_seq":1212,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666350+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:19:36.666350+0000","last_peered":"2023-10-16T11:19:36.666350+0000","last_clean":"2023-10-16T11:19:36.666350+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:19:36.666350+0000","last_undegraded":"2023-10-16T11:19:36.666350+0000","last_fullsized":"2023-10-16T11:19:36.666350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663250+0000","last_change":"2023-10-16T11:19:22.332936+0000","last_active":"2023-10-16T11:19:36.663250+0000","last_peered":"2023-10-16T11:19:36.663250+0000","last_clean":"2023-10-16T11:19:36.663250+0000","last_became_active":"2023-10-16T11:19:22.332598+0000","last_became_peered":"2023-10-16T11:19:22.332598+0000","last_unstale":"2023-10-16T11:19:36.663250+0000","last_undegraded":"2023-10-16T11:19:36.663250+0000","last_fullsized":"2023-10-16T11:19:36.663250+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:34.597859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666646+0000","last_change":"2023-10-16T11:19:23.334041+0000","last_active":"2023-10-16T11:19:36.666646+0000","last_peered":"2023-10-16T11:19:36.666646+0000","last_clean":"2023-10-16T11:19:36.666646+0000","last_became_active":"2023-10-16T11:19:23.333752+0000","last_became_peered":"2023-10-16T11:19:23.333752+0000","last_unstale":"2023-10-16T11:19:36.666646+0000","last_undegraded":"2023-10-16T11:19:36.666646+0000","last_fullsized":"2023-10-16T11:19:36.666646+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.685292+0000","last_change":"2023-10-16T11:19:35.677068+0000","last_active":"2023-10-16T11:19:36.685292+0000","last_peered":"2023-10-16T11:19:36.685292+0000","last_clean":"2023-10-16T11:19:36.685292+0000","last_became_active":"2023-10-16T11:19:35.676797+0000","last_became_peered":"2023-10-16T11:19:35.676797+0000","last_unstale":"2023-10-16T11:19:36.685292+0000","last_undegraded":"2023-10-16T11:19:36.685292+0000","last_fullsized":"2023-10-16T11:19:36.685292+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:43.621552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664039+0000","last_change":"2023-10-16T11:19:36.684566+0000","last_active":"2023-10-16T11:19:37.664039+0000","last_peered":"2023-10-16T11:19:37.664039+0000","last_clean":"2023-10-16T11:19:37.664039+0000","last_became_active":"2023-10-16T11:19:36.684276+0000","last_became_peered":"2023-10-16T11:19:36.684276+0000","last_unstale":"2023-10-16T11:19:37.664039+0000","last_undegraded":"2023-10-16T11:19:37.664039+0000","last_fullsized":"2023-10-16T11:19:37.664039+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:37.800770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672051+0000","last_change":"2023-10-16T11:19:22.334152+0000","last_active":"2023-10-16T11:19:36.672051+0000","last_peered":"2023-10-16T11:19:36.672051+0000","last_clean":"2023-10-16T11:19:36.672051+0000","last_became_active":"2023-10-16T11:19:22.333859+0000","last_became_peered":"2023-10-16T11:19:22.333859+0000","last_unstale":"2023-10-16T11:19:36.672051+0000","last_undegraded":"2023-10-16T11:19:36.672051+0000","last_fullsized":"2023-10-16T11:19:36.672051+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:50.619981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668272+0000","last_change":"2023-10-16T11:19:23.324999+0000","last_active":"2023-10-16T11:19:36.668272+0000","last_peered":"2023-10-16T11:19:36.668272+0000","last_clean":"2023-10-16T11:19:36.668272+0000","last_became_active":"2023-10-16T11:19:23.324686+0000","last_became_peered":"2023-10-16T11:19:23.324686+0000","last_unstale":"2023-10-16T11:19:36.668272+0000","last_undegraded":"2023-10-16T11:19:36.668272+0000","last_fullsized":"2023-10-16T11:19:36.668272+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:42.281964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"221'1","reported_seq":13,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994502+0000","last_change":"2023-10-16T11:19:35.674436+0000","last_active":"2023-10-16T11:19:37.994502+0000","last_peered":"2023-10-16T11:19:37.994502+0000","last_clean":"2023-10-16T11:19:37.994502+0000","last_became_active":"2023-10-16T11:19:35.674216+0000","last_became_peered":"2023-10-16T11:19:35.674216+0000","last_unstale":"2023-10-16T11:19:37.994502+0000","last_undegraded":"2023-10-16T11:19:37.994502+0000","last_fullsized":"2023-10-16T11:19:37.994502+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:05.514151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700186+0000","last_change":"2023-10-16T11:19:36.683592+0000","last_active":"2023-10-16T11:19:37.700186+0000","last_peered":"2023-10-16T11:19:37.700186+0000","last_clean":"2023-10-16T11:19:37.700186+0000","last_became_active":"2023-10-16T11:19:36.682898+0000","last_became_peered":"2023-10-16T11:19:36.682898+0000","last_unstale":"2023-10-16T11:19:37.700186+0000","last_undegraded":"2023-10-16T11:19:37.700186+0000","last_fullsized":"2023-10-16T11:19:37.700186+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:35.693733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663874+0000","last_change":"2023-10-16T11:19:22.332640+0000","last_active":"2023-10-16T11:19:36.663874+0000","last_peered":"2023-10-16T11:19:36.663874+0000","last_clean":"2023-10-16T11:19:36.663874+0000","last_became_active":"2023-10-16T11:19:22.331608+0000","last_became_peered":"2023-10-16T11:19:22.331608+0000","last_unstale":"2023-10-16T11:19:36.663874+0000","last_undegraded":"2023-10-16T11:19:36.663874+0000","last_fullsized":"2023-10-16T11:19:36.663874+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670102+0000","last_change":"2023-10-16T11:19:23.329115+0000","last_active":"2023-10-16T11:19:36.670102+0000","last_peered":"2023-10-16T11:19:36.670102+0000","last_clean":"2023-10-16T11:19:36.670102+0000","last_became_active":"2023-10-16T11:19:23.328830+0000","last_became_peered":"2023-10-16T11:19:23.328830+0000","last_unstale":"2023-10-16T11:19:36.670102+0000","last_undegraded":"2023-10-16T11:19:36.670102+0000","last_fullsized":"2023-10-16T11:19:36.670102+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:17.247488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699488+0000","last_change":"2023-10-16T11:19:35.687029+0000","last_active":"2023-10-16T11:19:36.699488+0000","last_peered":"2023-10-16T11:19:36.699488+0000","last_clean":"2023-10-16T11:19:36.699488+0000","last_became_active":"2023-10-16T11:19:35.686699+0000","last_became_peered":"2023-10-16T11:19:35.686699+0000","last_unstale":"2023-10-16T11:19:36.699488+0000","last_undegraded":"2023-10-16T11:19:36.699488+0000","last_fullsized":"2023-10-16T11:19:36.699488+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:52.450005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995162+0000","last_change":"2023-10-16T11:19:36.677702+0000","last_active":"2023-10-16T11:19:37.995162+0000","last_peered":"2023-10-16T11:19:37.995162+0000","last_clean":"2023-10-16T11:19:37.995162+0000","last_became_active":"2023-10-16T11:19:36.677422+0000","last_became_peered":"2023-10-16T11:19:36.677422+0000","last_unstale":"2023-10-16T11:19:37.995162+0000","last_undegraded":"2023-10-16T11:19:37.995162+0000","last_fullsized":"2023-10-16T11:19:37.995162+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:43.889855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663341+0000","last_change":"2023-10-16T11:19:22.329623+0000","last_active":"2023-10-16T11:19:36.663341+0000","last_peered":"2023-10-16T11:19:36.663341+0000","last_clean":"2023-10-16T11:19:36.663341+0000","last_became_active":"2023-10-16T11:19:22.329344+0000","last_became_peered":"2023-10-16T11:19:22.329344+0000","last_unstale":"2023-10-16T11:19:36.663341+0000","last_undegraded":"2023-10-16T11:19:36.663341+0000","last_fullsized":"2023-10-16T11:19:36.663341+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670592+0000","last_change":"2023-10-16T11:19:23.330624+0000","last_active":"2023-10-16T11:19:36.670592+0000","last_peered":"2023-10-16T11:19:36.670592+0000","last_clean":"2023-10-16T11:19:36.670592+0000","last_became_active":"2023-10-16T11:19:23.329588+0000","last_became_peered":"2023-10-16T11:19:23.329588+0000","last_unstale":"2023-10-16T11:19:36.670592+0000","last_undegraded":"2023-10-16T11:19:36.670592+0000","last_fullsized":"2023-10-16T11:19:36.670592+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:10.835410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.691778+0000","last_change":"2023-10-16T11:19:35.685879+0000","last_active":"2023-10-16T11:19:36.691778+0000","last_peered":"2023-10-16T11:19:36.691778+0000","last_clean":"2023-10-16T11:19:36.691778+0000","last_became_active":"2023-10-16T11:19:35.685381+0000","last_became_peered":"2023-10-16T11:19:35.685381+0000","last_unstale":"2023-10-16T11:19:36.691778+0000","last_undegraded":"2023-10-16T11:19:36.691778+0000","last_fullsized":"2023-10-16T11:19:36.691778+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:24.651235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664009+0000","last_change":"2023-10-16T11:19:36.692166+0000","last_active":"2023-10-16T11:19:37.664009+0000","last_peered":"2023-10-16T11:19:37.664009+0000","last_clean":"2023-10-16T11:19:37.664009+0000","last_became_active":"2023-10-16T11:19:36.691802+0000","last_became_peered":"2023-10-16T11:19:36.691802+0000","last_unstale":"2023-10-16T11:19:37.664009+0000","last_undegraded":"2023-10-16T11:19:37.664009+0000","last_fullsized":"2023-10-16T11:19:37.664009+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:31.108155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.683890+0000","last_change":"2023-10-16T11:19:22.333752+0000","last_active":"2023-10-16T11:19:36.683890+0000","last_peered":"2023-10-16T11:19:36.683890+0000","last_clean":"2023-10-16T11:19:36.683890+0000","last_became_active":"2023-10-16T11:19:22.333497+0000","last_became_peered":"2023-10-16T11:19:22.333497+0000","last_unstale":"2023-10-16T11:19:36.683890+0000","last_undegraded":"2023-10-16T11:19:36.683890+0000","last_fullsized":"2023-10-16T11:19:36.683890+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684460+0000","last_change":"2023-10-16T11:19:23.329220+0000","last_active":"2023-10-16T11:19:36.684460+0000","last_peered":"2023-10-16T11:19:36.684460+0000","last_clean":"2023-10-16T11:19:36.684460+0000","last_became_active":"2023-10-16T11:19:23.328923+0000","last_became_peered":"2023-10-16T11:19:23.328923+0000","last_unstale":"2023-10-16T11:19:36.684460+0000","last_undegraded":"2023-10-16T11:19:36.684460+0000","last_fullsized":"2023-10-16T11:19:36.684460+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667680+0000","last_change":"2023-10-16T11:19:35.674427+0000","last_active":"2023-10-16T11:19:36.667680+0000","last_peered":"2023-10-16T11:19:36.667680+0000","last_clean":"2023-10-16T11:19:36.667680+0000","last_became_active":"2023-10-16T11:19:35.674155+0000","last_became_peered":"2023-10-16T11:19:35.674155+0000","last_unstale":"2023-10-16T11:19:36.667680+0000","last_undegraded":"2023-10-16T11:19:36.667680+0000","last_fullsized":"2023-10-16T11:19:36.667680+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:25.009353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995237+0000","last_change":"2023-10-16T11:19:36.687286+0000","last_active":"2023-10-16T11:19:37.995237+0000","last_peered":"2023-10-16T11:19:37.995237+0000","last_clean":"2023-10-16T11:19:37.995237+0000","last_became_active":"2023-10-16T11:19:36.686995+0000","last_became_peered":"2023-10-16T11:19:36.686995+0000","last_unstale":"2023-10-16T11:19:37.995237+0000","last_undegraded":"2023-10-16T11:19:37.995237+0000","last_fullsized":"2023-10-16T11:19:37.995237+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:10.501316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664830+0000","last_change":"2023-10-16T11:19:22.335923+0000","last_active":"2023-10-16T11:19:36.664830+0000","last_peered":"2023-10-16T11:19:36.664830+0000","last_clean":"2023-10-16T11:19:36.664830+0000","last_became_active":"2023-10-16T11:19:22.335705+0000","last_became_peered":"2023-10-16T11:19:22.335705+0000","last_unstale":"2023-10-16T11:19:36.664830+0000","last_undegraded":"2023-10-16T11:19:36.664830+0000","last_fullsized":"2023-10-16T11:19:36.664830+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:40.803363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665902+0000","last_change":"2023-10-16T11:19:23.336160+0000","last_active":"2023-10-16T11:19:36.665902+0000","last_peered":"2023-10-16T11:19:36.665902+0000","last_clean":"2023-10-16T11:19:36.665902+0000","last_became_active":"2023-10-16T11:19:23.333920+0000","last_became_peered":"2023-10-16T11:19:23.333920+0000","last_unstale":"2023-10-16T11:19:36.665902+0000","last_undegraded":"2023-10-16T11:19:36.665902+0000","last_fullsized":"2023-10-16T11:19:36.665902+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671203+0000","last_change":"2023-10-16T11:19:35.671256+0000","last_active":"2023-10-16T11:19:36.671203+0000","last_peered":"2023-10-16T11:19:36.671203+0000","last_clean":"2023-10-16T11:19:36.671203+0000","last_became_active":"2023-10-16T11:19:35.670977+0000","last_became_peered":"2023-10-16T11:19:35.670977+0000","last_unstale":"2023-10-16T11:19:36.671203+0000","last_undegraded":"2023-10-16T11:19:36.671203+0000","last_fullsized":"2023-10-16T11:19:36.671203+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:00.113603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663985+0000","last_change":"2023-10-16T11:19:36.680876+0000","last_active":"2023-10-16T11:19:37.663985+0000","last_peered":"2023-10-16T11:19:37.663985+0000","last_clean":"2023-10-16T11:19:37.663985+0000","last_became_active":"2023-10-16T11:19:36.680576+0000","last_became_peered":"2023-10-16T11:19:36.680576+0000","last_unstale":"2023-10-16T11:19:37.663985+0000","last_undegraded":"2023-10-16T11:19:37.663985+0000","last_fullsized":"2023-10-16T11:19:37.663985+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:52.686759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666970+0000","last_change":"2023-10-16T11:19:22.326754+0000","last_active":"2023-10-16T11:19:36.666970+0000","last_peered":"2023-10-16T11:19:36.666970+0000","last_clean":"2023-10-16T11:19:36.666970+0000","last_became_active":"2023-10-16T11:19:22.326480+0000","last_became_peered":"2023-10-16T11:19:22.326480+0000","last_unstale":"2023-10-16T11:19:36.666970+0000","last_undegraded":"2023-10-16T11:19:36.666970+0000","last_fullsized":"2023-10-16T11:19:36.666970+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663142+0000","last_change":"2023-10-16T11:19:23.331571+0000","last_active":"2023-10-16T11:19:36.663142+0000","last_peered":"2023-10-16T11:19:36.663142+0000","last_clean":"2023-10-16T11:19:36.663142+0000","last_became_active":"2023-10-16T11:19:23.331242+0000","last_became_peered":"2023-10-16T11:19:23.331242+0000","last_unstale":"2023-10-16T11:19:36.663142+0000","last_undegraded":"2023-10-16T11:19:36.663142+0000","last_fullsized":"2023-10-16T11:19:36.663142+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684254+0000","last_change":"2023-10-16T11:19:35.679089+0000","last_active":"2023-10-16T11:19:36.684254+0000","last_peered":"2023-10-16T11:19:36.684254+0000","last_clean":"2023-10-16T11:19:36.684254+0000","last_became_active":"2023-10-16T11:19:35.678420+0000","last_became_peered":"2023-10-16T11:19:35.678420+0000","last_unstale":"2023-10-16T11:19:36.684254+0000","last_undegraded":"2023-10-16T11:19:36.684254+0000","last_fullsized":"2023-10-16T11:19:36.684254+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:30.151493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664447+0000","last_change":"2023-10-16T11:19:36.681755+0000","last_active":"2023-10-16T11:19:37.664447+0000","last_peered":"2023-10-16T11:19:37.664447+0000","last_clean":"2023-10-16T11:19:37.664447+0000","last_became_active":"2023-10-16T11:19:36.681399+0000","last_became_peered":"2023-10-16T11:19:36.681399+0000","last_unstale":"2023-10-16T11:19:37.664447+0000","last_undegraded":"2023-10-16T11:19:37.664447+0000","last_fullsized":"2023-10-16T11:19:37.664447+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:56.393185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"217'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689930+0000","last_change":"2023-10-16T11:19:22.331182+0000","last_active":"2023-10-16T11:19:36.689930+0000","last_peered":"2023-10-16T11:19:36.689930+0000","last_clean":"2023-10-16T11:19:36.689930+0000","last_became_active":"2023-10-16T11:19:22.330860+0000","last_became_peered":"2023-10-16T11:19:22.330860+0000","last_unstale":"2023-10-16T11:19:36.689930+0000","last_undegraded":"2023-10-16T11:19:36.689930+0000","last_fullsized":"2023-10-16T11:19:36.689930+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667572+0000","last_change":"2023-10-16T11:19:23.325482+0000","last_active":"2023-10-16T11:19:36.667572+0000","last_peered":"2023-10-16T11:19:36.667572+0000","last_clean":"2023-10-16T11:19:36.667572+0000","last_became_active":"2023-10-16T11:19:23.325193+0000","last_became_peered":"2023-10-16T11:19:23.325193+0000","last_unstale":"2023-10-16T11:19:36.667572+0000","last_undegraded":"2023-10-16T11:19:36.667572+0000","last_fullsized":"2023-10-16T11:19:36.667572+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.191635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678478+0000","last_change":"2023-10-16T11:19:35.681024+0000","last_active":"2023-10-16T11:19:36.678478+0000","last_peered":"2023-10-16T11:19:36.678478+0000","last_clean":"2023-10-16T11:19:36.678478+0000","last_became_active":"2023-10-16T11:19:35.677329+0000","last_became_peered":"2023-10-16T11:19:35.677329+0000","last_unstale":"2023-10-16T11:19:36.678478+0000","last_undegraded":"2023-10-16T11:19:36.678478+0000","last_fullsized":"2023-10-16T11:19:36.678478+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:36.304256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696725+0000","last_change":"2023-10-16T11:19:36.694524+0000","last_active":"2023-10-16T11:19:37.696725+0000","last_peered":"2023-10-16T11:19:37.696725+0000","last_clean":"2023-10-16T11:19:37.696725+0000","last_became_active":"2023-10-16T11:19:36.694188+0000","last_became_peered":"2023-10-16T11:19:36.694188+0000","last_unstale":"2023-10-16T11:19:37.696725+0000","last_undegraded":"2023-10-16T11:19:37.696725+0000","last_fullsized":"2023-10-16T11:19:37.696725+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:30.588015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662948+0000","last_change":"2023-10-16T11:19:22.332950+0000","last_active":"2023-10-16T11:19:36.662948+0000","last_peered":"2023-10-16T11:19:36.662948+0000","last_clean":"2023-10-16T11:19:36.662948+0000","last_became_active":"2023-10-16T11:19:22.332650+0000","last_became_peered":"2023-10-16T11:19:22.332650+0000","last_unstale":"2023-10-16T11:19:36.662948+0000","last_undegraded":"2023-10-16T11:19:36.662948+0000","last_fullsized":"2023-10-16T11:19:36.662948+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.680253+0000","last_change":"2023-10-16T11:19:23.333192+0000","last_active":"2023-10-16T11:19:36.680253+0000","last_peered":"2023-10-16T11:19:36.680253+0000","last_clean":"2023-10-16T11:19:36.680253+0000","last_became_active":"2023-10-16T11:19:23.332739+0000","last_became_peered":"2023-10-16T11:19:23.332739+0000","last_unstale":"2023-10-16T11:19:36.680253+0000","last_undegraded":"2023-10-16T11:19:36.680253+0000","last_fullsized":"2023-10-16T11:19:36.680253+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666051+0000","last_change":"2023-10-16T11:19:35.672187+0000","last_active":"2023-10-16T11:19:36.666051+0000","last_peered":"2023-10-16T11:19:36.666051+0000","last_clean":"2023-10-16T11:19:36.666051+0000","last_became_active":"2023-10-16T11:19:35.671883+0000","last_became_peered":"2023-10-16T11:19:35.671883+0000","last_unstale":"2023-10-16T11:19:36.666051+0000","last_undegraded":"2023-10-16T11:19:36.666051+0000","last_fullsized":"2023-10-16T11:19:36.666051+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:27.228484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662842+0000","last_change":"2023-10-16T11:19:36.680498+0000","last_active":"2023-10-16T11:19:37.662842+0000","last_peered":"2023-10-16T11:19:37.662842+0000","last_clean":"2023-10-16T11:19:37.662842+0000","last_became_active":"2023-10-16T11:19:36.680087+0000","last_became_peered":"2023-10-16T11:19:36.680087+0000","last_unstale":"2023-10-16T11:19:37.662842+0000","last_undegraded":"2023-10-16T11:19:37.662842+0000","last_fullsized":"2023-10-16T11:19:37.662842+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:30.982911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666212+0000","last_change":"2023-10-16T11:19:22.326294+0000","last_active":"2023-10-16T11:19:36.666212+0000","last_peered":"2023-10-16T11:19:36.666212+0000","last_clean":"2023-10-16T11:19:36.666212+0000","last_became_active":"2023-10-16T11:19:22.326012+0000","last_became_peered":"2023-10-16T11:19:22.326012+0000","last_unstale":"2023-10-16T11:19:36.666212+0000","last_undegraded":"2023-10-16T11:19:36.666212+0000","last_fullsized":"2023-10-16T11:19:36.666212+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.692071+0000","last_change":"2023-10-16T11:19:23.336920+0000","last_active":"2023-10-16T11:19:36.692071+0000","last_peered":"2023-10-16T11:19:36.692071+0000","last_clean":"2023-10-16T11:19:36.692071+0000","last_became_active":"2023-10-16T11:19:23.336473+0000","last_became_peered":"2023-10-16T11:19:23.336473+0000","last_unstale":"2023-10-16T11:19:36.692071+0000","last_undegraded":"2023-10-16T11:19:36.692071+0000","last_fullsized":"2023-10-16T11:19:36.692071+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:22.421887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.685513+0000","last_change":"2023-10-16T11:19:35.677026+0000","last_active":"2023-10-16T11:19:36.685513+0000","last_peered":"2023-10-16T11:19:36.685513+0000","last_clean":"2023-10-16T11:19:36.685513+0000","last_became_active":"2023-10-16T11:19:35.676697+0000","last_became_peered":"2023-10-16T11:19:35.676697+0000","last_unstale":"2023-10-16T11:19:36.685513+0000","last_undegraded":"2023-10-16T11:19:36.685513+0000","last_fullsized":"2023-10-16T11:19:36.685513+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:49.538337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.699997+0000","last_change":"2023-10-16T11:19:36.675909+0000","last_active":"2023-10-16T11:19:37.699997+0000","last_peered":"2023-10-16T11:19:37.699997+0000","last_clean":"2023-10-16T11:19:37.699997+0000","last_became_active":"2023-10-16T11:19:36.674282+0000","last_became_peered":"2023-10-16T11:19:36.674282+0000","last_unstale":"2023-10-16T11:19:37.699997+0000","last_undegraded":"2023-10-16T11:19:37.699997+0000","last_fullsized":"2023-10-16T11:19:37.699997+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:49.823406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668923+0000","last_change":"2023-10-16T11:19:22.331457+0000","last_active":"2023-10-16T11:19:36.668923+0000","last_peered":"2023-10-16T11:19:36.668923+0000","last_clean":"2023-10-16T11:19:36.668923+0000","last_became_active":"2023-10-16T11:19:22.331010+0000","last_became_peered":"2023-10-16T11:19:22.331010+0000","last_unstale":"2023-10-16T11:19:36.668923+0000","last_undegraded":"2023-10-16T11:19:36.668923+0000","last_fullsized":"2023-10-16T11:19:36.668923+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:38.048633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663202+0000","last_change":"2023-10-16T11:19:23.333035+0000","last_active":"2023-10-16T11:19:36.663202+0000","last_peered":"2023-10-16T11:19:36.663202+0000","last_clean":"2023-10-16T11:19:36.663202+0000","last_became_active":"2023-10-16T11:19:23.332692+0000","last_became_peered":"2023-10-16T11:19:23.332692+0000","last_unstale":"2023-10-16T11:19:36.663202+0000","last_undegraded":"2023-10-16T11:19:36.663202+0000","last_fullsized":"2023-10-16T11:19:36.663202+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.694775+0000","last_change":"2023-10-16T11:19:35.684312+0000","last_active":"2023-10-16T11:19:36.694775+0000","last_peered":"2023-10-16T11:19:36.694775+0000","last_clean":"2023-10-16T11:19:36.694775+0000","last_became_active":"2023-10-16T11:19:35.684055+0000","last_became_peered":"2023-10-16T11:19:35.684055+0000","last_unstale":"2023-10-16T11:19:36.694775+0000","last_undegraded":"2023-10-16T11:19:36.694775+0000","last_fullsized":"2023-10-16T11:19:36.694775+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:48.058006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995020+0000","last_change":"2023-10-16T11:19:36.691309+0000","last_active":"2023-10-16T11:19:37.995020+0000","last_peered":"2023-10-16T11:19:37.995020+0000","last_clean":"2023-10-16T11:19:37.995020+0000","last_became_active":"2023-10-16T11:19:36.690993+0000","last_became_peered":"2023-10-16T11:19:36.690993+0000","last_unstale":"2023-10-16T11:19:37.995020+0000","last_undegraded":"2023-10-16T11:19:37.995020+0000","last_fullsized":"2023-10-16T11:19:37.995020+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:12.957525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675071+0000","last_change":"2023-10-16T11:19:22.329858+0000","last_active":"2023-10-16T11:19:36.675071+0000","last_peered":"2023-10-16T11:19:36.675071+0000","last_clean":"2023-10-16T11:19:36.675071+0000","last_became_active":"2023-10-16T11:19:22.329586+0000","last_became_peered":"2023-10-16T11:19:22.329586+0000","last_unstale":"2023-10-16T11:19:36.675071+0000","last_undegraded":"2023-10-16T11:19:36.675071+0000","last_fullsized":"2023-10-16T11:19:36.675071+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:25.906324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670506+0000","last_change":"2023-10-16T11:19:23.333742+0000","last_active":"2023-10-16T11:19:36.670506+0000","last_peered":"2023-10-16T11:19:36.670506+0000","last_clean":"2023-10-16T11:19:36.670506+0000","last_became_active":"2023-10-16T11:19:23.333424+0000","last_became_peered":"2023-10-16T11:19:23.333424+0000","last_unstale":"2023-10-16T11:19:36.670506+0000","last_undegraded":"2023-10-16T11:19:36.670506+0000","last_fullsized":"2023-10-16T11:19:36.670506+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.156569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684622+0000","last_change":"2023-10-16T11:19:35.685893+0000","last_active":"2023-10-16T11:19:36.684622+0000","last_peered":"2023-10-16T11:19:36.684622+0000","last_clean":"2023-10-16T11:19:36.684622+0000","last_became_active":"2023-10-16T11:19:35.685647+0000","last_became_peered":"2023-10-16T11:19:35.685647+0000","last_unstale":"2023-10-16T11:19:36.684622+0000","last_undegraded":"2023-10-16T11:19:36.684622+0000","last_fullsized":"2023-10-16T11:19:36.684622+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:55.676783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697595+0000","last_change":"2023-10-16T11:19:36.694904+0000","last_active":"2023-10-16T11:19:37.697595+0000","last_peered":"2023-10-16T11:19:37.697595+0000","last_clean":"2023-10-16T11:19:37.697595+0000","last_became_active":"2023-10-16T11:19:36.690439+0000","last_became_peered":"2023-10-16T11:19:36.690439+0000","last_unstale":"2023-10-16T11:19:37.697595+0000","last_undegraded":"2023-10-16T11:19:37.697595+0000","last_fullsized":"2023-10-16T11:19:37.697595+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:37.294828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.673044+0000","last_change":"2023-10-16T11:19:22.329655+0000","last_active":"2023-10-16T11:19:36.673044+0000","last_peered":"2023-10-16T11:19:36.673044+0000","last_clean":"2023-10-16T11:19:36.673044+0000","last_became_active":"2023-10-16T11:19:22.329366+0000","last_became_peered":"2023-10-16T11:19:22.329366+0000","last_unstale":"2023-10-16T11:19:36.673044+0000","last_undegraded":"2023-10-16T11:19:36.673044+0000","last_fullsized":"2023-10-16T11:19:36.673044+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:47.331154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671563+0000","last_change":"2023-10-16T11:19:23.331460+0000","last_active":"2023-10-16T11:19:36.671563+0000","last_peered":"2023-10-16T11:19:36.671563+0000","last_clean":"2023-10-16T11:19:36.671563+0000","last_became_active":"2023-10-16T11:19:23.331178+0000","last_became_peered":"2023-10-16T11:19:23.331178+0000","last_unstale":"2023-10-16T11:19:36.671563+0000","last_undegraded":"2023-10-16T11:19:36.671563+0000","last_fullsized":"2023-10-16T11:19:36.671563+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:36.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699143+0000","last_change":"2023-10-16T11:19:35.683185+0000","last_active":"2023-10-16T11:19:36.699143+0000","last_peered":"2023-10-16T11:19:36.699143+0000","last_clean":"2023-10-16T11:19:36.699143+0000","last_became_active":"2023-10-16T11:19:35.681186+0000","last_became_peered":"2023-10-16T11:19:35.681186+0000","last_unstale":"2023-10-16T11:19:36.699143+0000","last_undegraded":"2023-10-16T11:19:36.699143+0000","last_fullsized":"2023-10-16T11:19:36.699143+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:26.084551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664927+0000","last_change":"2023-10-16T11:19:36.678756+0000","last_active":"2023-10-16T11:19:37.664927+0000","last_peered":"2023-10-16T11:19:37.664927+0000","last_clean":"2023-10-16T11:19:37.664927+0000","last_became_active":"2023-10-16T11:19:36.678452+0000","last_became_peered":"2023-10-16T11:19:36.678452+0000","last_unstale":"2023-10-16T11:19:37.664927+0000","last_undegraded":"2023-10-16T11:19:37.664927+0000","last_fullsized":"2023-10-16T11:19:37.664927+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:55.430561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675077+0000","last_change":"2023-10-16T11:19:22.332355+0000","last_active":"2023-10-16T11:19:36.675077+0000","last_peered":"2023-10-16T11:19:36.675077+0000","last_clean":"2023-10-16T11:19:36.675077+0000","last_became_active":"2023-10-16T11:19:22.331520+0000","last_became_peered":"2023-10-16T11:19:22.331520+0000","last_unstale":"2023-10-16T11:19:36.675077+0000","last_undegraded":"2023-10-16T11:19:36.675077+0000","last_fullsized":"2023-10-16T11:19:36.675077+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:10.682623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684099+0000","last_change":"2023-10-16T11:19:23.331171+0000","last_active":"2023-10-16T11:19:36.684099+0000","last_peered":"2023-10-16T11:19:36.684099+0000","last_clean":"2023-10-16T11:19:36.684099+0000","last_became_active":"2023-10-16T11:19:23.330836+0000","last_became_peered":"2023-10-16T11:19:23.330836+0000","last_unstale":"2023-10-16T11:19:36.684099+0000","last_undegraded":"2023-10-16T11:19:36.684099+0000","last_fullsized":"2023-10-16T11:19:36.684099+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:45.818409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667260+0000","last_change":"2023-10-16T11:19:35.674937+0000","last_active":"2023-10-16T11:19:36.667260+0000","last_peered":"2023-10-16T11:19:36.667260+0000","last_clean":"2023-10-16T11:19:36.667260+0000","last_became_active":"2023-10-16T11:19:35.674670+0000","last_became_peered":"2023-10-16T11:19:35.674670+0000","last_unstale":"2023-10-16T11:19:36.667260+0000","last_undegraded":"2023-10-16T11:19:36.667260+0000","last_fullsized":"2023-10-16T11:19:36.667260+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:50.829956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700060+0000","last_change":"2023-10-16T11:19:36.682803+0000","last_active":"2023-10-16T11:19:37.700060+0000","last_peered":"2023-10-16T11:19:37.700060+0000","last_clean":"2023-10-16T11:19:37.700060+0000","last_became_active":"2023-10-16T11:19:36.682480+0000","last_became_peered":"2023-10-16T11:19:36.682480+0000","last_unstale":"2023-10-16T11:19:37.700060+0000","last_undegraded":"2023-10-16T11:19:37.700060+0000","last_fullsized":"2023-10-16T11:19:37.700060+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:27.320950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678028+0000","last_change":"2023-10-16T11:19:22.318314+0000","last_active":"2023-10-16T11:19:36.678028+0000","last_peered":"2023-10-16T11:19:36.678028+0000","last_clean":"2023-10-16T11:19:36.678028+0000","last_became_active":"2023-10-16T11:19:22.318015+0000","last_became_peered":"2023-10-16T11:19:22.318015+0000","last_unstale":"2023-10-16T11:19:36.678028+0000","last_undegraded":"2023-10-16T11:19:36.678028+0000","last_fullsized":"2023-10-16T11:19:36.678028+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.421816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668042+0000","last_change":"2023-10-16T11:19:23.337274+0000","last_active":"2023-10-16T11:19:36.668042+0000","last_peered":"2023-10-16T11:19:36.668042+0000","last_clean":"2023-10-16T11:19:36.668042+0000","last_became_active":"2023-10-16T11:19:23.336889+0000","last_became_peered":"2023-10-16T11:19:23.336889+0000","last_unstale":"2023-10-16T11:19:36.668042+0000","last_undegraded":"2023-10-16T11:19:36.668042+0000","last_fullsized":"2023-10-16T11:19:36.668042+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699643+0000","last_change":"2023-10-16T11:19:35.679263+0000","last_active":"2023-10-16T11:19:36.699643+0000","last_peered":"2023-10-16T11:19:36.699643+0000","last_clean":"2023-10-16T11:19:36.699643+0000","last_became_active":"2023-10-16T11:19:35.678812+0000","last_became_peered":"2023-10-16T11:19:35.678812+0000","last_unstale":"2023-10-16T11:19:36.699643+0000","last_undegraded":"2023-10-16T11:19:36.699643+0000","last_fullsized":"2023-10-16T11:19:36.699643+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:57.878898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696290+0000","last_change":"2023-10-16T11:19:36.684169+0000","last_active":"2023-10-16T11:19:37.696290+0000","last_peered":"2023-10-16T11:19:37.696290+0000","last_clean":"2023-10-16T11:19:37.696290+0000","last_became_active":"2023-10-16T11:19:36.683898+0000","last_became_peered":"2023-10-16T11:19:36.683898+0000","last_unstale":"2023-10-16T11:19:37.696290+0000","last_undegraded":"2023-10-16T11:19:37.696290+0000","last_fullsized":"2023-10-16T11:19:37.696290+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:04:05.120700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665201+0000","last_change":"2023-10-16T11:19:22.330569+0000","last_active":"2023-10-16T11:19:36.665201+0000","last_peered":"2023-10-16T11:19:36.665201+0000","last_clean":"2023-10-16T11:19:36.665201+0000","last_became_active":"2023-10-16T11:19:22.329740+0000","last_became_peered":"2023-10-16T11:19:22.329740+0000","last_unstale":"2023-10-16T11:19:36.665201+0000","last_undegraded":"2023-10-16T11:19:36.665201+0000","last_fullsized":"2023-10-16T11:19:36.665201+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664845+0000","last_change":"2023-10-16T11:19:23.338701+0000","last_active":"2023-10-16T11:19:36.664845+0000","last_peered":"2023-10-16T11:19:36.664845+0000","last_clean":"2023-10-16T11:19:36.664845+0000","last_became_active":"2023-10-16T11:19:23.334033+0000","last_became_peered":"2023-10-16T11:19:23.334033+0000","last_unstale":"2023-10-16T11:19:36.664845+0000","last_undegraded":"2023-10-16T11:19:36.664845+0000","last_fullsized":"2023-10-16T11:19:36.664845+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672283+0000","last_change":"2023-10-16T11:19:35.677127+0000","last_active":"2023-10-16T11:19:36.672283+0000","last_peered":"2023-10-16T11:19:36.672283+0000","last_clean":"2023-10-16T11:19:36.672283+0000","last_became_active":"2023-10-16T11:19:35.676828+0000","last_became_peered":"2023-10-16T11:19:35.676828+0000","last_unstale":"2023-10-16T11:19:36.672283+0000","last_undegraded":"2023-10-16T11:19:36.672283+0000","last_fullsized":"2023-10-16T11:19:36.672283+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:15.101657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663965+0000","last_change":"2023-10-16T11:19:36.698526+0000","last_active":"2023-10-16T11:19:37.663965+0000","last_peered":"2023-10-16T11:19:37.663965+0000","last_clean":"2023-10-16T11:19:37.663965+0000","last_became_active":"2023-10-16T11:19:36.698271+0000","last_became_peered":"2023-10-16T11:19:36.698271+0000","last_unstale":"2023-10-16T11:19:37.663965+0000","last_undegraded":"2023-10-16T11:19:37.663965+0000","last_fullsized":"2023-10-16T11:19:37.663965+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:53.874459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665100+0000","last_change":"2023-10-16T11:19:22.331432+0000","last_active":"2023-10-16T11:19:36.665100+0000","last_peered":"2023-10-16T11:19:36.665100+0000","last_clean":"2023-10-16T11:19:36.665100+0000","last_became_active":"2023-10-16T11:19:22.331124+0000","last_became_peered":"2023-10-16T11:19:22.331124+0000","last_unstale":"2023-10-16T11:19:36.665100+0000","last_undegraded":"2023-10-16T11:19:36.665100+0000","last_fullsized":"2023-10-16T11:19:36.665100+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.680058+0000","last_change":"2023-10-16T11:19:23.333018+0000","last_active":"2023-10-16T11:19:36.680058+0000","last_peered":"2023-10-16T11:19:36.680058+0000","last_clean":"2023-10-16T11:19:36.680058+0000","last_became_active":"2023-10-16T11:19:23.332725+0000","last_became_peered":"2023-10-16T11:19:23.332725+0000","last_unstale":"2023-10-16T11:19:36.680058+0000","last_undegraded":"2023-10-16T11:19:36.680058+0000","last_fullsized":"2023-10-16T11:19:36.680058+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:39.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666647+0000","last_change":"2023-10-16T11:19:35.669741+0000","last_active":"2023-10-16T11:19:36.666647+0000","last_peered":"2023-10-16T11:19:36.666647+0000","last_clean":"2023-10-16T11:19:36.666647+0000","last_became_active":"2023-10-16T11:19:35.669477+0000","last_became_peered":"2023-10-16T11:19:35.669477+0000","last_unstale":"2023-10-16T11:19:36.666647+0000","last_undegraded":"2023-10-16T11:19:36.666647+0000","last_fullsized":"2023-10-16T11:19:36.666647+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:57.643373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994219+0000","last_change":"2023-10-16T11:19:36.690841+0000","last_active":"2023-10-16T11:19:37.994219+0000","last_peered":"2023-10-16T11:19:37.994219+0000","last_clean":"2023-10-16T11:19:37.994219+0000","last_became_active":"2023-10-16T11:19:36.690558+0000","last_became_peered":"2023-10-16T11:19:36.690558+0000","last_unstale":"2023-10-16T11:19:37.994219+0000","last_undegraded":"2023-10-16T11:19:37.994219+0000","last_fullsized":"2023-10-16T11:19:37.994219+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:30.591636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"216'8","reported_seq":24,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671923+0000","last_change":"2023-10-16T11:19:22.332156+0000","last_active":"2023-10-16T11:19:36.671923+0000","last_peered":"2023-10-16T11:19:36.671923+0000","last_clean":"2023-10-16T11:19:36.671923+0000","last_became_active":"2023-10-16T11:19:22.331863+0000","last_became_peered":"2023-10-16T11:19:22.331863+0000","last_unstale":"2023-10-16T11:19:36.671923+0000","last_undegraded":"2023-10-16T11:19:36.671923+0000","last_fullsized":"2023-10-16T11:19:36.671923+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:36.175530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670117+0000","last_change":"2023-10-16T11:19:23.336992+0000","last_active":"2023-10-16T11:19:36.670117+0000","last_peered":"2023-10-16T11:19:36.670117+0000","last_clean":"2023-10-16T11:19:36.670117+0000","last_became_active":"2023-10-16T11:19:23.336724+0000","last_became_peered":"2023-10-16T11:19:23.336724+0000","last_unstale":"2023-10-16T11:19:36.670117+0000","last_undegraded":"2023-10-16T11:19:36.670117+0000","last_fullsized":"2023-10-16T11:19:36.670117+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:01.326738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667579+0000","last_change":"2023-10-16T11:19:35.678016+0000","last_active":"2023-10-16T11:19:36.667579+0000","last_peered":"2023-10-16T11:19:36.667579+0000","last_clean":"2023-10-16T11:19:36.667579+0000","last_became_active":"2023-10-16T11:19:35.677757+0000","last_became_peered":"2023-10-16T11:19:35.677757+0000","last_unstale":"2023-10-16T11:19:36.667579+0000","last_undegraded":"2023-10-16T11:19:36.667579+0000","last_fullsized":"2023-10-16T11:19:36.667579+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:46.311559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994557+0000","last_change":"2023-10-16T11:19:36.677981+0000","last_active":"2023-10-16T11:19:37.994557+0000","last_peered":"2023-10-16T11:19:37.994557+0000","last_clean":"2023-10-16T11:19:37.994557+0000","last_became_active":"2023-10-16T11:19:36.677677+0000","last_became_peered":"2023-10-16T11:19:36.677677+0000","last_unstale":"2023-10-16T11:19:37.994557+0000","last_undegraded":"2023-10-16T11:19:37.994557+0000","last_fullsized":"2023-10-16T11:19:37.994557+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:31.283029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662222+0000","last_change":"2023-10-16T11:19:22.331044+0000","last_active":"2023-10-16T11:19:36.662222+0000","last_peered":"2023-10-16T11:19:36.662222+0000","last_clean":"2023-10-16T11:19:36.662222+0000","last_became_active":"2023-10-16T11:19:22.330757+0000","last_became_peered":"2023-10-16T11:19:22.330757+0000","last_unstale":"2023-10-16T11:19:36.662222+0000","last_undegraded":"2023-10-16T11:19:36.662222+0000","last_fullsized":"2023-10-16T11:19:36.662222+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662806+0000","last_change":"2023-10-16T11:19:23.333500+0000","last_active":"2023-10-16T11:19:36.662806+0000","last_peered":"2023-10-16T11:19:36.662806+0000","last_clean":"2023-10-16T11:19:36.662806+0000","last_became_active":"2023-10-16T11:19:23.333215+0000","last_became_peered":"2023-10-16T11:19:23.333215+0000","last_unstale":"2023-10-16T11:19:36.662806+0000","last_undegraded":"2023-10-16T11:19:36.662806+0000","last_fullsized":"2023-10-16T11:19:36.662806+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670009+0000","last_change":"2023-10-16T11:19:35.673910+0000","last_active":"2023-10-16T11:19:36.670009+0000","last_peered":"2023-10-16T11:19:36.670009+0000","last_clean":"2023-10-16T11:19:36.670009+0000","last_became_active":"2023-10-16T11:19:35.673640+0000","last_became_peered":"2023-10-16T11:19:35.673640+0000","last_unstale":"2023-10-16T11:19:36.670009+0000","last_undegraded":"2023-10-16T11:19:36.670009+0000","last_fullsized":"2023-10-16T11:19:36.670009+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:42.767262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994757+0000","last_change":"2023-10-16T11:19:36.678756+0000","last_active":"2023-10-16T11:19:37.994757+0000","last_peered":"2023-10-16T11:19:37.994757+0000","last_clean":"2023-10-16T11:19:37.994757+0000","last_became_active":"2023-10-16T11:19:36.678501+0000","last_became_peered":"2023-10-16T11:19:36.678501+0000","last_unstale":"2023-10-16T11:19:37.994757+0000","last_undegraded":"2023-10-16T11:19:37.994757+0000","last_fullsized":"2023-10-16T11:19:37.994757+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:00.784444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667556+0000","last_change":"2023-10-16T11:19:22.327307+0000","last_active":"2023-10-16T11:19:36.667556+0000","last_peered":"2023-10-16T11:19:36.667556+0000","last_clean":"2023-10-16T11:19:36.667556+0000","last_became_active":"2023-10-16T11:19:22.327036+0000","last_became_peered":"2023-10-16T11:19:22.327036+0000","last_unstale":"2023-10-16T11:19:36.667556+0000","last_undegraded":"2023-10-16T11:19:36.667556+0000","last_fullsized":"2023-10-16T11:19:36.667556+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:24.276671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666847+0000","last_change":"2023-10-16T11:19:23.337758+0000","last_active":"2023-10-16T11:19:36.666847+0000","last_peered":"2023-10-16T11:19:36.666847+0000","last_clean":"2023-10-16T11:19:36.666847+0000","last_became_active":"2023-10-16T11:19:23.337147+0000","last_became_peered":"2023-10-16T11:19:23.337147+0000","last_unstale":"2023-10-16T11:19:36.666847+0000","last_undegraded":"2023-10-16T11:19:36.666847+0000","last_fullsized":"2023-10-16T11:19:36.666847+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:06.136692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"220'6","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.700777+0000","last_change":"2023-10-16T11:19:35.682083+0000","last_active":"2023-10-16T11:19:36.700777+0000","last_peered":"2023-10-16T11:19:36.700777+0000","last_clean":"2023-10-16T11:19:36.700777+0000","last_became_active":"2023-10-16T11:19:35.681804+0000","last_became_peered":"2023-10-16T11:19:35.681804+0000","last_unstale":"2023-10-16T11:19:36.700777+0000","last_undegraded":"2023-10-16T11:19:36.700777+0000","last_fullsized":"2023-10-16T11:19:36.700777+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:54.161780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664448+0000","last_change":"2023-10-16T11:19:36.692085+0000","last_active":"2023-10-16T11:19:37.664448+0000","last_peered":"2023-10-16T11:19:37.664448+0000","last_clean":"2023-10-16T11:19:37.664448+0000","last_became_active":"2023-10-16T11:19:36.691313+0000","last_became_peered":"2023-10-16T11:19:36.691313+0000","last_unstale":"2023-10-16T11:19:37.664448+0000","last_undegraded":"2023-10-16T11:19:37.664448+0000","last_fullsized":"2023-10-16T11:19:37.664448+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:54.154198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675831+0000","last_change":"2023-10-16T11:19:22.325030+0000","last_active":"2023-10-16T11:19:36.675831+0000","last_peered":"2023-10-16T11:19:36.675831+0000","last_clean":"2023-10-16T11:19:36.675831+0000","last_became_active":"2023-10-16T11:19:22.324733+0000","last_became_peered":"2023-10-16T11:19:22.324733+0000","last_unstale":"2023-10-16T11:19:36.675831+0000","last_undegraded":"2023-10-16T11:19:36.675831+0000","last_fullsized":"2023-10-16T11:19:36.675831+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:29.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671514+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:36.671514+0000","last_peered":"2023-10-16T11:19:36.671514+0000","last_clean":"2023-10-16T11:19:36.671514+0000","last_became_active":"2023-10-16T11:19:23.331725+0000","last_became_peered":"2023-10-16T11:19:23.331725+0000","last_unstale":"2023-10-16T11:19:36.671514+0000","last_undegraded":"2023-10-16T11:19:36.671514+0000","last_fullsized":"2023-10-16T11:19:36.671514+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:45.043730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"220'3","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.682632+0000","last_change":"2023-10-16T11:19:35.672388+0000","last_active":"2023-10-16T11:19:36.682632+0000","last_peered":"2023-10-16T11:19:36.682632+0000","last_clean":"2023-10-16T11:19:36.682632+0000","last_became_active":"2023-10-16T11:19:35.672090+0000","last_became_peered":"2023-10-16T11:19:35.672090+0000","last_unstale":"2023-10-16T11:19:36.682632+0000","last_undegraded":"2023-10-16T11:19:36.682632+0000","last_fullsized":"2023-10-16T11:19:36.682632+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:15.533409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665100+0000","last_change":"2023-10-16T11:19:36.684006+0000","last_active":"2023-10-16T11:19:37.665100+0000","last_peered":"2023-10-16T11:19:37.665100+0000","last_clean":"2023-10-16T11:19:37.665100+0000","last_became_active":"2023-10-16T11:19:36.683679+0000","last_became_peered":"2023-10-16T11:19:36.683679+0000","last_unstale":"2023-10-16T11:19:37.665100+0000","last_undegraded":"2023-10-16T11:19:37.665100+0000","last_fullsized":"2023-10-16T11:19:37.665100+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:20.376959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"217'8","reported_seq":24,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666769+0000","last_change":"2023-10-16T11:19:22.329082+0000","last_active":"2023-10-16T11:19:36.666769+0000","last_peered":"2023-10-16T11:19:36.666769+0000","last_clean":"2023-10-16T11:19:36.666769+0000","last_became_active":"2023-10-16T11:19:22.328756+0000","last_became_peered":"2023-10-16T11:19:22.328756+0000","last_unstale":"2023-10-16T11:19:36.666769+0000","last_undegraded":"2023-10-16T11:19:36.666769+0000","last_fullsized":"2023-10-16T11:19:36.666769+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.384963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.673087+0000","last_change":"2023-10-16T11:19:23.336196+0000","last_active":"2023-10-16T11:19:36.673087+0000","last_peered":"2023-10-16T11:19:36.673087+0000","last_clean":"2023-10-16T11:19:36.673087+0000","last_became_active":"2023-10-16T11:19:23.335924+0000","last_became_peered":"2023-10-16T11:19:23.335924+0000","last_unstale":"2023-10-16T11:19:36.673087+0000","last_undegraded":"2023-10-16T11:19:36.673087+0000","last_fullsized":"2023-10-16T11:19:36.673087+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:27:01.350215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662408+0000","last_change":"2023-10-16T11:19:35.670005+0000","last_active":"2023-10-16T11:19:36.662408+0000","last_peered":"2023-10-16T11:19:36.662408+0000","last_clean":"2023-10-16T11:19:36.662408+0000","last_became_active":"2023-10-16T11:19:35.669743+0000","last_became_peered":"2023-10-16T11:19:35.669743+0000","last_unstale":"2023-10-16T11:19:36.662408+0000","last_undegraded":"2023-10-16T11:19:36.662408+0000","last_fullsized":"2023-10-16T11:19:36.662408+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:59.779061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696422+0000","last_change":"2023-10-16T11:19:36.697172+0000","last_active":"2023-10-16T11:19:37.696422+0000","last_peered":"2023-10-16T11:19:37.696422+0000","last_clean":"2023-10-16T11:19:37.696422+0000","last_became_active":"2023-10-16T11:19:36.696866+0000","last_became_peered":"2023-10-16T11:19:36.696866+0000","last_unstale":"2023-10-16T11:19:37.696422+0000","last_undegraded":"2023-10-16T11:19:37.696422+0000","last_fullsized":"2023-10-16T11:19:37.696422+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:18.714994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"216'3","reported_seq":19,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662822+0000","last_change":"2023-10-16T11:19:22.331150+0000","last_active":"2023-10-16T11:19:36.662822+0000","last_peered":"2023-10-16T11:19:36.662822+0000","last_clean":"2023-10-16T11:19:36.662822+0000","last_became_active":"2023-10-16T11:19:22.330820+0000","last_became_peered":"2023-10-16T11:19:22.330820+0000","last_unstale":"2023-10-16T11:19:36.662822+0000","last_undegraded":"2023-10-16T11:19:36.662822+0000","last_fullsized":"2023-10-16T11:19:36.662822+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662601+0000","last_change":"2023-10-16T11:19:23.338568+0000","last_active":"2023-10-16T11:19:36.662601+0000","last_peered":"2023-10-16T11:19:36.662601+0000","last_clean":"2023-10-16T11:19:36.662601+0000","last_became_active":"2023-10-16T11:19:23.338280+0000","last_became_peered":"2023-10-16T11:19:23.338280+0000","last_unstale":"2023-10-16T11:19:36.662601+0000","last_undegraded":"2023-10-16T11:19:36.662601+0000","last_fullsized":"2023-10-16T11:19:36.662601+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:40.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699722+0000","last_change":"2023-10-16T11:19:35.686423+0000","last_active":"2023-10-16T11:19:36.699722+0000","last_peered":"2023-10-16T11:19:36.699722+0000","last_clean":"2023-10-16T11:19:36.699722+0000","last_became_active":"2023-10-16T11:19:35.685384+0000","last_became_peered":"2023-10-16T11:19:35.685384+0000","last_unstale":"2023-10-16T11:19:36.699722+0000","last_undegraded":"2023-10-16T11:19:36.699722+0000","last_fullsized":"2023-10-16T11:19:36.699722+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:49.913941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994146+0000","last_change":"2023-10-16T11:19:36.672963+0000","last_active":"2023-10-16T11:19:37.994146+0000","last_peered":"2023-10-16T11:19:37.994146+0000","last_clean":"2023-10-16T11:19:37.994146+0000","last_became_active":"2023-10-16T11:19:36.672611+0000","last_became_peered":"2023-10-16T11:19:36.672611+0000","last_unstale":"2023-10-16T11:19:37.994146+0000","last_undegraded":"2023-10-16T11:19:37.994146+0000","last_fullsized":"2023-10-16T11:19:37.994146+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:17.500716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670020+0000","last_change":"2023-10-16T11:19:22.325211+0000","last_active":"2023-10-16T11:19:36.670020+0000","last_peered":"2023-10-16T11:19:36.670020+0000","last_clean":"2023-10-16T11:19:36.670020+0000","last_became_active":"2023-10-16T11:19:22.324901+0000","last_became_peered":"2023-10-16T11:19:22.324901+0000","last_unstale":"2023-10-16T11:19:36.670020+0000","last_undegraded":"2023-10-16T11:19:36.670020+0000","last_fullsized":"2023-10-16T11:19:36.670020+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:46.432434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666928+0000","last_change":"2023-10-16T11:19:23.331144+0000","last_active":"2023-10-16T11:19:36.666928+0000","last_peered":"2023-10-16T11:19:36.666928+0000","last_clean":"2023-10-16T11:19:36.666928+0000","last_became_active":"2023-10-16T11:19:23.330824+0000","last_became_peered":"2023-10-16T11:19:23.330824+0000","last_unstale":"2023-10-16T11:19:36.666928+0000","last_undegraded":"2023-10-16T11:19:36.666928+0000","last_fullsized":"2023-10-16T11:19:36.666928+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:44:27.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"220'8","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699249+0000","last_change":"2023-10-16T11:19:35.678999+0000","last_active":"2023-10-16T11:19:36.699249+0000","last_peered":"2023-10-16T11:19:36.699249+0000","last_clean":"2023-10-16T11:19:36.699249+0000","last_became_active":"2023-10-16T11:19:35.678522+0000","last_became_peered":"2023-10-16T11:19:35.678522+0000","last_unstale":"2023-10-16T11:19:36.699249+0000","last_undegraded":"2023-10-16T11:19:36.699249+0000","last_fullsized":"2023-10-16T11:19:36.699249+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:51.788613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994331+0000","last_change":"2023-10-16T11:19:36.674240+0000","last_active":"2023-10-16T11:19:37.994331+0000","last_peered":"2023-10-16T11:19:37.994331+0000","last_clean":"2023-10-16T11:19:37.994331+0000","last_became_active":"2023-10-16T11:19:36.673954+0000","last_became_peered":"2023-10-16T11:19:36.673954+0000","last_unstale":"2023-10-16T11:19:37.994331+0000","last_undegraded":"2023-10-16T11:19:37.994331+0000","last_fullsized":"2023-10-16T11:19:37.994331+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:15.742018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.676479+0000","last_change":"2023-10-16T11:19:22.332253+0000","last_active":"2023-10-16T11:19:36.676479+0000","last_peered":"2023-10-16T11:19:36.676479+0000","last_clean":"2023-10-16T11:19:36.676479+0000","last_became_active":"2023-10-16T11:19:22.331970+0000","last_became_peered":"2023-10-16T11:19:22.331970+0000","last_unstale":"2023-10-16T11:19:36.676479+0000","last_undegraded":"2023-10-16T11:19:36.676479+0000","last_fullsized":"2023-10-16T11:19:36.676479+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:19.740978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667271+0000","last_change":"2023-10-16T11:19:23.330327+0000","last_active":"2023-10-16T11:19:36.667271+0000","last_peered":"2023-10-16T11:19:36.667271+0000","last_clean":"2023-10-16T11:19:36.667271+0000","last_became_active":"2023-10-16T11:19:23.330040+0000","last_became_peered":"2023-10-16T11:19:23.330040+0000","last_unstale":"2023-10-16T11:19:36.667271+0000","last_undegraded":"2023-10-16T11:19:36.667271+0000","last_fullsized":"2023-10-16T11:19:36.667271+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664745+0000","last_change":"2023-10-16T11:19:35.677569+0000","last_active":"2023-10-16T11:19:36.664745+0000","last_peered":"2023-10-16T11:19:36.664745+0000","last_clean":"2023-10-16T11:19:36.664745+0000","last_became_active":"2023-10-16T11:19:35.677275+0000","last_became_peered":"2023-10-16T11:19:35.677275+0000","last_unstale":"2023-10-16T11:19:36.664745+0000","last_undegraded":"2023-10-16T11:19:36.664745+0000","last_fullsized":"2023-10-16T11:19:36.664745+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:56.520892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663873+0000","last_change":"2023-10-16T11:19:36.678766+0000","last_active":"2023-10-16T11:19:37.663873+0000","last_peered":"2023-10-16T11:19:37.663873+0000","last_clean":"2023-10-16T11:19:37.663873+0000","last_became_active":"2023-10-16T11:19:36.678481+0000","last_became_peered":"2023-10-16T11:19:36.678481+0000","last_unstale":"2023-10-16T11:19:37.663873+0000","last_undegraded":"2023-10-16T11:19:37.663873+0000","last_fullsized":"2023-10-16T11:19:37.663873+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:23.629013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666314+0000","last_change":"2023-10-16T11:19:22.330754+0000","last_active":"2023-10-16T11:19:36.666314+0000","last_peered":"2023-10-16T11:19:36.666314+0000","last_clean":"2023-10-16T11:19:36.666314+0000","last_became_active":"2023-10-16T11:19:22.330490+0000","last_became_peered":"2023-10-16T11:19:22.330490+0000","last_unstale":"2023-10-16T11:19:36.666314+0000","last_undegraded":"2023-10-16T11:19:36.666314+0000","last_fullsized":"2023-10-16T11:19:36.666314+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:17.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662829+0000","last_change":"2023-10-16T11:19:23.334130+0000","last_active":"2023-10-16T11:19:36.662829+0000","last_peered":"2023-10-16T11:19:36.662829+0000","last_clean":"2023-10-16T11:19:36.662829+0000","last_became_active":"2023-10-16T11:19:23.333073+0000","last_became_peered":"2023-10-16T11:19:23.333073+0000","last_unstale":"2023-10-16T11:19:36.662829+0000","last_undegraded":"2023-10-16T11:19:36.662829+0000","last_fullsized":"2023-10-16T11:19:36.662829+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:38.491191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.697295+0000","last_change":"2023-10-16T11:19:35.683271+0000","last_active":"2023-10-16T11:19:36.697295+0000","last_peered":"2023-10-16T11:19:36.697295+0000","last_clean":"2023-10-16T11:19:36.697295+0000","last_became_active":"2023-10-16T11:19:35.681021+0000","last_became_peered":"2023-10-16T11:19:35.681021+0000","last_unstale":"2023-10-16T11:19:36.697295+0000","last_undegraded":"2023-10-16T11:19:36.697295+0000","last_fullsized":"2023-10-16T11:19:36.697295+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:30.265858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664246+0000","last_change":"2023-10-16T11:19:36.679321+0000","last_active":"2023-10-16T11:19:37.664246+0000","last_peered":"2023-10-16T11:19:37.664246+0000","last_clean":"2023-10-16T11:19:37.664246+0000","last_became_active":"2023-10-16T11:19:36.679019+0000","last_became_peered":"2023-10-16T11:19:36.679019+0000","last_unstale":"2023-10-16T11:19:37.664246+0000","last_undegraded":"2023-10-16T11:19:37.664246+0000","last_fullsized":"2023-10-16T11:19:37.664246+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:00.210732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667231+0000","last_change":"2023-10-16T11:19:22.328061+0000","last_active":"2023-10-16T11:19:36.667231+0000","last_peered":"2023-10-16T11:19:36.667231+0000","last_clean":"2023-10-16T11:19:36.667231+0000","last_became_active":"2023-10-16T11:19:22.327773+0000","last_became_peered":"2023-10-16T11:19:22.327773+0000","last_unstale":"2023-10-16T11:19:36.667231+0000","last_undegraded":"2023-10-16T11:19:36.667231+0000","last_fullsized":"2023-10-16T11:19:36.667231+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:39.077769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668801+0000","last_change":"2023-10-16T11:19:23.330616+0000","last_active":"2023-10-16T11:19:36.668801+0000","last_peered":"2023-10-16T11:19:36.668801+0000","last_clean":"2023-10-16T11:19:36.668801+0000","last_became_active":"2023-10-16T11:19:23.330317+0000","last_became_peered":"2023-10-16T11:19:23.330317+0000","last_unstale":"2023-10-16T11:19:36.668801+0000","last_undegraded":"2023-10-16T11:19:36.668801+0000","last_fullsized":"2023-10-16T11:19:36.668801+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"220'4","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.686252+0000","last_change":"2023-10-16T11:19:35.671218+0000","last_active":"2023-10-16T11:19:36.686252+0000","last_peered":"2023-10-16T11:19:36.686252+0000","last_clean":"2023-10-16T11:19:36.686252+0000","last_became_active":"2023-10-16T11:19:35.670991+0000","last_became_peered":"2023-10-16T11:19:35.670991+0000","last_unstale":"2023-10-16T11:19:36.686252+0000","last_undegraded":"2023-10-16T11:19:36.686252+0000","last_fullsized":"2023-10-16T11:19:36.686252+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:11.399468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665905+0000","last_change":"2023-10-16T11:19:36.687869+0000","last_active":"2023-10-16T11:19:37.665905+0000","last_peered":"2023-10-16T11:19:37.665905+0000","last_clean":"2023-10-16T11:19:37.665905+0000","last_became_active":"2023-10-16T11:19:36.687472+0000","last_became_peered":"2023-10-16T11:19:36.687472+0000","last_unstale":"2023-10-16T11:19:37.665905+0000","last_undegraded":"2023-10-16T11:19:37.665905+0000","last_fullsized":"2023-10-16T11:19:37.665905+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:45.543759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666971+0000","last_change":"2023-10-16T11:19:22.328217+0000","last_active":"2023-10-16T11:19:36.666971+0000","last_peered":"2023-10-16T11:19:36.666971+0000","last_clean":"2023-10-16T11:19:36.666971+0000","last_became_active":"2023-10-16T11:19:22.327901+0000","last_became_peered":"2023-10-16T11:19:22.327901+0000","last_unstale":"2023-10-16T11:19:36.666971+0000","last_undegraded":"2023-10-16T11:19:36.666971+0000","last_fullsized":"2023-10-16T11:19:36.666971+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:48.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671972+0000","last_change":"2023-10-16T11:19:23.336335+0000","last_active":"2023-10-16T11:19:36.671972+0000","last_peered":"2023-10-16T11:19:36.671972+0000","last_clean":"2023-10-16T11:19:36.671972+0000","last_became_active":"2023-10-16T11:19:23.335852+0000","last_became_peered":"2023-10-16T11:19:23.335852+0000","last_unstale":"2023-10-16T11:19:36.671972+0000","last_undegraded":"2023-10-16T11:19:36.671972+0000","last_fullsized":"2023-10-16T11:19:36.671972+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.686520+0000","last_change":"2023-10-16T11:19:35.679257+0000","last_active":"2023-10-16T11:19:36.686520+0000","last_peered":"2023-10-16T11:19:36.686520+0000","last_clean":"2023-10-16T11:19:36.686520+0000","last_became_active":"2023-10-16T11:19:35.678977+0000","last_became_peered":"2023-10-16T11:19:35.678977+0000","last_unstale":"2023-10-16T11:19:36.686520+0000","last_undegraded":"2023-10-16T11:19:36.686520+0000","last_fullsized":"2023-10-16T11:19:36.686520+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:29.586253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697443+0000","last_change":"2023-10-16T11:19:36.683265+0000","last_active":"2023-10-16T11:19:37.697443+0000","last_peered":"2023-10-16T11:19:37.697443+0000","last_clean":"2023-10-16T11:19:37.697443+0000","last_became_active":"2023-10-16T11:19:36.681552+0000","last_became_peered":"2023-10-16T11:19:36.681552+0000","last_unstale":"2023-10-16T11:19:37.697443+0000","last_undegraded":"2023-10-16T11:19:37.697443+0000","last_fullsized":"2023-10-16T11:19:37.697443+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:08:32.673134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663673+0000","last_change":"2023-10-16T11:19:22.329968+0000","last_active":"2023-10-16T11:19:36.663673+0000","last_peered":"2023-10-16T11:19:36.663673+0000","last_clean":"2023-10-16T11:19:36.663673+0000","last_became_active":"2023-10-16T11:19:22.326360+0000","last_became_peered":"2023-10-16T11:19:22.326360+0000","last_unstale":"2023-10-16T11:19:36.663673+0000","last_undegraded":"2023-10-16T11:19:36.663673+0000","last_fullsized":"2023-10-16T11:19:36.663673+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675124+0000","last_change":"2023-10-16T11:19:23.335733+0000","last_active":"2023-10-16T11:19:36.675124+0000","last_peered":"2023-10-16T11:19:36.675124+0000","last_clean":"2023-10-16T11:19:36.675124+0000","last_became_active":"2023-10-16T11:19:23.334936+0000","last_became_peered":"2023-10-16T11:19:23.334936+0000","last_unstale":"2023-10-16T11:19:36.675124+0000","last_undegraded":"2023-10-16T11:19:36.675124+0000","last_fullsized":"2023-10-16T11:19:36.675124+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:08.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.690777+0000","last_change":"2023-10-16T11:19:35.684464+0000","last_active":"2023-10-16T11:19:36.690777+0000","last_peered":"2023-10-16T11:19:36.690777+0000","last_clean":"2023-10-16T11:19:36.690777+0000","last_became_active":"2023-10-16T11:19:35.684139+0000","last_became_peered":"2023-10-16T11:19:35.684139+0000","last_unstale":"2023-10-16T11:19:36.690777+0000","last_undegraded":"2023-10-16T11:19:36.690777+0000","last_fullsized":"2023-10-16T11:19:36.690777+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:52.513114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696386+0000","last_change":"2023-10-16T11:19:36.691931+0000","last_active":"2023-10-16T11:19:37.696386+0000","last_peered":"2023-10-16T11:19:37.696386+0000","last_clean":"2023-10-16T11:19:37.696386+0000","last_became_active":"2023-10-16T11:19:36.691594+0000","last_became_peered":"2023-10-16T11:19:36.691594+0000","last_unstale":"2023-10-16T11:19:37.696386+0000","last_undegraded":"2023-10-16T11:19:37.696386+0000","last_fullsized":"2023-10-16T11:19:37.696386+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:56.795176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.677928+0000","last_change":"2023-10-16T11:19:22.328385+0000","last_active":"2023-10-16T11:19:36.677928+0000","last_peered":"2023-10-16T11:19:36.677928+0000","last_clean":"2023-10-16T11:19:36.677928+0000","last_became_active":"2023-10-16T11:19:22.328096+0000","last_became_peered":"2023-10-16T11:19:22.328096+0000","last_unstale":"2023-10-16T11:19:36.677928+0000","last_undegraded":"2023-10-16T11:19:36.677928+0000","last_fullsized":"2023-10-16T11:19:36.677928+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:40.312176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678838+0000","last_change":"2023-10-16T11:19:23.326944+0000","last_active":"2023-10-16T11:19:36.678838+0000","last_peered":"2023-10-16T11:19:36.678838+0000","last_clean":"2023-10-16T11:19:36.678838+0000","last_became_active":"2023-10-16T11:19:23.326680+0000","last_became_peered":"2023-10-16T11:19:23.326680+0000","last_unstale":"2023-10-16T11:19:36.678838+0000","last_undegraded":"2023-10-16T11:19:36.678838+0000","last_fullsized":"2023-10-16T11:19:36.678838+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:28.070345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.693442+0000","last_change":"2023-10-16T11:19:35.685986+0000","last_active":"2023-10-16T11:19:36.693442+0000","last_peered":"2023-10-16T11:19:36.693442+0000","last_clean":"2023-10-16T11:19:36.693442+0000","last_became_active":"2023-10-16T11:19:35.685758+0000","last_became_peered":"2023-10-16T11:19:35.685758+0000","last_unstale":"2023-10-16T11:19:36.693442+0000","last_undegraded":"2023-10-16T11:19:36.693442+0000","last_fullsized":"2023-10-16T11:19:36.693442+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:12.481754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663581+0000","last_change":"2023-10-16T11:19:36.687085+0000","last_active":"2023-10-16T11:19:37.663581+0000","last_peered":"2023-10-16T11:19:37.663581+0000","last_clean":"2023-10-16T11:19:37.663581+0000","last_became_active":"2023-10-16T11:19:36.686804+0000","last_became_peered":"2023-10-16T11:19:36.686804+0000","last_unstale":"2023-10-16T11:19:37.663581+0000","last_undegraded":"2023-10-16T11:19:37.663581+0000","last_fullsized":"2023-10-16T11:19:37.663581+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:04.397243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667326+0000","last_change":"2023-10-16T11:19:22.332064+0000","last_active":"2023-10-16T11:19:36.667326+0000","last_peered":"2023-10-16T11:19:36.667326+0000","last_clean":"2023-10-16T11:19:36.667326+0000","last_became_active":"2023-10-16T11:19:22.331746+0000","last_became_peered":"2023-10-16T11:19:22.331746+0000","last_unstale":"2023-10-16T11:19:36.667326+0000","last_undegraded":"2023-10-16T11:19:36.667326+0000","last_fullsized":"2023-10-16T11:19:36.667326+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665063+0000","last_change":"2023-10-16T11:19:23.342219+0000","last_active":"2023-10-16T11:19:36.665063+0000","last_peered":"2023-10-16T11:19:36.665063+0000","last_clean":"2023-10-16T11:19:36.665063+0000","last_became_active":"2023-10-16T11:19:23.334494+0000","last_became_peered":"2023-10-16T11:19:23.334494+0000","last_unstale":"2023-10-16T11:19:36.665063+0000","last_undegraded":"2023-10-16T11:19:36.665063+0000","last_fullsized":"2023-10-16T11:19:36.665063+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670212+0000","last_change":"2023-10-16T11:19:35.674368+0000","last_active":"2023-10-16T11:19:36.670212+0000","last_peered":"2023-10-16T11:19:36.670212+0000","last_clean":"2023-10-16T11:19:36.670212+0000","last_became_active":"2023-10-16T11:19:35.674151+0000","last_became_peered":"2023-10-16T11:19:35.674151+0000","last_unstale":"2023-10-16T11:19:36.670212+0000","last_undegraded":"2023-10-16T11:19:36.670212+0000","last_fullsized":"2023-10-16T11:19:36.670212+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.707693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663231+0000","last_change":"2023-10-16T11:19:36.687042+0000","last_active":"2023-10-16T11:19:37.663231+0000","last_peered":"2023-10-16T11:19:37.663231+0000","last_clean":"2023-10-16T11:19:37.663231+0000","last_became_active":"2023-10-16T11:19:36.686749+0000","last_became_peered":"2023-10-16T11:19:36.686749+0000","last_unstale":"2023-10-16T11:19:37.663231+0000","last_undegraded":"2023-10-16T11:19:37.663231+0000","last_fullsized":"2023-10-16T11:19:37.663231+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:46.573220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"216'4","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668738+0000","last_change":"2023-10-16T11:19:22.332308+0000","last_active":"2023-10-16T11:19:36.668738+0000","last_peered":"2023-10-16T11:19:36.668738+0000","last_clean":"2023-10-16T11:19:36.668738+0000","last_became_active":"2023-10-16T11:19:22.332005+0000","last_became_peered":"2023-10-16T11:19:22.332005+0000","last_unstale":"2023-10-16T11:19:36.668738+0000","last_undegraded":"2023-10-16T11:19:36.668738+0000","last_fullsized":"2023-10-16T11:19:36.668738+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670591+0000","last_change":"2023-10-16T11:19:23.341701+0000","last_active":"2023-10-16T11:19:36.670591+0000","last_peered":"2023-10-16T11:19:36.670591+0000","last_clean":"2023-10-16T11:19:36.670591+0000","last_became_active":"2023-10-16T11:19:23.341424+0000","last_became_peered":"2023-10-16T11:19:23.341424+0000","last_unstale":"2023-10-16T11:19:36.670591+0000","last_undegraded":"2023-10-16T11:19:36.670591+0000","last_fullsized":"2023-10-16T11:19:36.670591+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:25.448895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667316+0000","last_change":"2023-10-16T11:19:35.684994+0000","last_active":"2023-10-16T11:19:36.667316+0000","last_peered":"2023-10-16T11:19:36.667316+0000","last_clean":"2023-10-16T11:19:36.667316+0000","last_became_active":"2023-10-16T11:19:35.684739+0000","last_became_peered":"2023-10-16T11:19:35.684739+0000","last_unstale":"2023-10-16T11:19:36.667316+0000","last_undegraded":"2023-10-16T11:19:36.667316+0000","last_fullsized":"2023-10-16T11:19:36.667316+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:46.895984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994879+0000","last_change":"2023-10-16T11:19:36.687519+0000","last_active":"2023-10-16T11:19:37.994879+0000","last_peered":"2023-10-16T11:19:37.994879+0000","last_clean":"2023-10-16T11:19:37.994879+0000","last_became_active":"2023-10-16T11:19:36.686851+0000","last_became_peered":"2023-10-16T11:19:36.686851+0000","last_unstale":"2023-10-16T11:19:37.994879+0000","last_undegraded":"2023-10-16T11:19:37.994879+0000","last_fullsized":"2023-10-16T11:19:37.994879+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:20.736974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666244+0000","last_change":"2023-10-16T11:19:23.323955+0000","last_active":"2023-10-16T11:19:36.666244+0000","last_peered":"2023-10-16T11:19:36.666244+0000","last_clean":"2023-10-16T11:19:36.666244+0000","last_became_active":"2023-10-16T11:19:23.323706+0000","last_became_peered":"2023-10-16T11:19:23.323706+0000","last_unstale":"2023-10-16T11:19:36.666244+0000","last_undegraded":"2023-10-16T11:19:36.666244+0000","last_fullsized":"2023-10-16T11:19:36.666244+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.182127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994845+0000","last_change":"2023-10-16T11:19:36.684069+0000","last_active":"2023-10-16T11:19:37.994845+0000","last_peered":"2023-10-16T11:19:37.994845+0000","last_clean":"2023-10-16T11:19:37.994845+0000","last_became_active":"2023-10-16T11:19:36.683792+0000","last_became_peered":"2023-10-16T11:19:36.683792+0000","last_unstale":"2023-10-16T11:19:37.994845+0000","last_undegraded":"2023-10-16T11:19:37.994845+0000","last_fullsized":"2023-10-16T11:19:37.994845+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:52.124440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666136+0000","last_change":"2023-10-16T11:19:23.332826+0000","last_active":"2023-10-16T11:19:36.666136+0000","last_peered":"2023-10-16T11:19:36.666136+0000","last_clean":"2023-10-16T11:19:36.666136+0000","last_became_active":"2023-10-16T11:19:23.332541+0000","last_became_peered":"2023-10-16T11:19:23.332541+0000","last_unstale":"2023-10-16T11:19:36.666136+0000","last_undegraded":"2023-10-16T11:19:36.666136+0000","last_fullsized":"2023-10-16T11:19:36.666136+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:12:33.418278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994582+0000","last_change":"2023-10-16T11:19:36.677032+0000","last_active":"2023-10-16T11:19:37.994582+0000","last_peered":"2023-10-16T11:19:37.994582+0000","last_clean":"2023-10-16T11:19:37.994582+0000","last_became_active":"2023-10-16T11:19:36.676736+0000","last_became_peered":"2023-10-16T11:19:36.676736+0000","last_unstale":"2023-10-16T11:19:37.994582+0000","last_undegraded":"2023-10-16T11:19:37.994582+0000","last_fullsized":"2023-10-16T11:19:37.994582+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:24.179558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671403+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:36.671403+0000","last_peered":"2023-10-16T11:19:36.671403+0000","last_clean":"2023-10-16T11:19:36.671403+0000","last_became_active":"2023-10-16T11:19:23.335534+0000","last_became_peered":"2023-10-16T11:19:23.335534+0000","last_unstale":"2023-10-16T11:19:36.671403+0000","last_undegraded":"2023-10-16T11:19:36.671403+0000","last_fullsized":"2023-10-16T11:19:36.671403+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994750+0000","last_change":"2023-10-16T11:19:36.684273+0000","last_active":"2023-10-16T11:19:37.994750+0000","last_peered":"2023-10-16T11:19:37.994750+0000","last_clean":"2023-10-16T11:19:37.994750+0000","last_became_active":"2023-10-16T11:19:36.683688+0000","last_became_peered":"2023-10-16T11:19:36.683688+0000","last_unstale":"2023-10-16T11:19:37.994750+0000","last_undegraded":"2023-10-16T11:19:37.994750+0000","last_fullsized":"2023-10-16T11:19:37.994750+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:54.831776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662964+0000","last_change":"2023-10-16T11:19:23.330876+0000","last_active":"2023-10-16T11:19:36.662964+0000","last_peered":"2023-10-16T11:19:36.662964+0000","last_clean":"2023-10-16T11:19:36.662964+0000","last_became_active":"2023-10-16T11:19:23.330597+0000","last_became_peered":"2023-10-16T11:19:23.330597+0000","last_unstale":"2023-10-16T11:19:36.662964+0000","last_undegraded":"2023-10-16T11:19:36.662964+0000","last_fullsized":"2023-10-16T11:19:36.662964+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662848+0000","last_change":"2023-10-16T11:19:36.691604+0000","last_active":"2023-10-16T11:19:37.662848+0000","last_peered":"2023-10-16T11:19:37.662848+0000","last_clean":"2023-10-16T11:19:37.662848+0000","last_became_active":"2023-10-16T11:19:36.690918+0000","last_became_peered":"2023-10-16T11:19:36.690918+0000","last_unstale":"2023-10-16T11:19:37.662848+0000","last_undegraded":"2023-10-16T11:19:37.662848+0000","last_fullsized":"2023-10-16T11:19:37.662848+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:49.453674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664799+0000","last_change":"2023-10-16T11:19:36.684752+0000","last_active":"2023-10-16T11:19:37.664799+0000","last_peered":"2023-10-16T11:19:37.664799+0000","last_clean":"2023-10-16T11:19:37.664799+0000","last_became_active":"2023-10-16T11:19:36.684491+0000","last_became_peered":"2023-10-16T11:19:36.684491+0000","last_unstale":"2023-10-16T11:19:37.664799+0000","last_undegraded":"2023-10-16T11:19:37.664799+0000","last_fullsized":"2023-10-16T11:19:37.664799+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:39.093443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.674996+0000","last_change":"2023-10-16T11:19:23.327468+0000","last_active":"2023-10-16T11:19:36.674996+0000","last_peered":"2023-10-16T11:19:36.674996+0000","last_clean":"2023-10-16T11:19:36.674996+0000","last_became_active":"2023-10-16T11:19:23.327176+0000","last_became_peered":"2023-10-16T11:19:23.327176+0000","last_unstale":"2023-10-16T11:19:36.674996+0000","last_undegraded":"2023-10-16T11:19:36.674996+0000","last_fullsized":"2023-10-16T11:19:36.674996+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:02.654427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700296+0000","last_change":"2023-10-16T11:19:36.685932+0000","last_active":"2023-10-16T11:19:37.700296+0000","last_peered":"2023-10-16T11:19:37.700296+0000","last_clean":"2023-10-16T11:19:37.700296+0000","last_became_active":"2023-10-16T11:19:36.685576+0000","last_became_peered":"2023-10-16T11:19:36.685576+0000","last_unstale":"2023-10-16T11:19:37.700296+0000","last_undegraded":"2023-10-16T11:19:37.700296+0000","last_fullsized":"2023-10-16T11:19:37.700296+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:58:51.519722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666950+0000","last_change":"2023-10-16T11:19:23.332619+0000","last_active":"2023-10-16T11:19:36.666950+0000","last_peered":"2023-10-16T11:19:36.666950+0000","last_clean":"2023-10-16T11:19:36.666950+0000","last_became_active":"2023-10-16T11:19:23.332037+0000","last_became_peered":"2023-10-16T11:19:23.332037+0000","last_unstale":"2023-10-16T11:19:36.666950+0000","last_undegraded":"2023-10-16T11:19:36.666950+0000","last_fullsized":"2023-10-16T11:19:36.666950+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:48.332974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665091+0000","last_change":"2023-10-16T11:19:23.341307+0000","last_active":"2023-10-16T11:19:36.665091+0000","last_peered":"2023-10-16T11:19:36.665091+0000","last_clean":"2023-10-16T11:19:36.665091+0000","last_became_active":"2023-10-16T11:19:23.334457+0000","last_became_peered":"2023-10-16T11:19:23.334457+0000","last_unstale":"2023-10-16T11:19:36.665091+0000","last_undegraded":"2023-10-16T11:19:36.665091+0000","last_fullsized":"2023-10-16T11:19:36.665091+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697007+0000","last_change":"2023-10-16T11:19:36.694609+0000","last_active":"2023-10-16T11:19:37.697007+0000","last_peered":"2023-10-16T11:19:37.697007+0000","last_clean":"2023-10-16T11:19:37.697007+0000","last_became_active":"2023-10-16T11:19:36.692033+0000","last_became_peered":"2023-10-16T11:19:36.692033+0000","last_unstale":"2023-10-16T11:19:37.697007+0000","last_undegraded":"2023-10-16T11:19:37.697007+0000","last_fullsized":"2023-10-16T11:19:37.697007+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:37.217522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664901+0000","last_change":"2023-10-16T11:19:36.678695+0000","last_active":"2023-10-16T11:19:37.664901+0000","last_peered":"2023-10-16T11:19:37.664901+0000","last_clean":"2023-10-16T11:19:37.664901+0000","last_became_active":"2023-10-16T11:19:36.678422+0000","last_became_peered":"2023-10-16T11:19:36.678422+0000","last_unstale":"2023-10-16T11:19:37.664901+0000","last_undegraded":"2023-10-16T11:19:37.664901+0000","last_fullsized":"2023-10-16T11:19:37.664901+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:08.320255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662807+0000","last_change":"2023-10-16T11:19:23.334306+0000","last_active":"2023-10-16T11:19:36.662807+0000","last_peered":"2023-10-16T11:19:36.662807+0000","last_clean":"2023-10-16T11:19:36.662807+0000","last_became_active":"2023-10-16T11:19:23.333460+0000","last_became_peered":"2023-10-16T11:19:23.333460+0000","last_unstale":"2023-10-16T11:19:36.662807+0000","last_undegraded":"2023-10-16T11:19:36.662807+0000","last_fullsized":"2023-10-16T11:19:36.662807+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:53.175386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994177+0000","last_change":"2023-10-16T11:19:36.687715+0000","last_active":"2023-10-16T11:19:37.994177+0000","last_peered":"2023-10-16T11:19:37.994177+0000","last_clean":"2023-10-16T11:19:37.994177+0000","last_became_active":"2023-10-16T11:19:36.687236+0000","last_became_peered":"2023-10-16T11:19:36.687236+0000","last_unstale":"2023-10-16T11:19:37.994177+0000","last_undegraded":"2023-10-16T11:19:37.994177+0000","last_fullsized":"2023-10-16T11:19:37.994177+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:55.480897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671643+0000","last_change":"2023-10-16T11:19:23.341264+0000","last_active":"2023-10-16T11:19:36.671643+0000","last_peered":"2023-10-16T11:19:36.671643+0000","last_clean":"2023-10-16T11:19:36.671643+0000","last_became_active":"2023-10-16T11:19:23.340921+0000","last_became_peered":"2023-10-16T11:19:23.340921+0000","last_unstale":"2023-10-16T11:19:36.671643+0000","last_undegraded":"2023-10-16T11:19:36.671643+0000","last_fullsized":"2023-10-16T11:19:36.671643+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.710291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663817+0000","last_change":"2023-10-16T11:19:36.681734+0000","last_active":"2023-10-16T11:19:37.663817+0000","last_peered":"2023-10-16T11:19:37.663817+0000","last_clean":"2023-10-16T11:19:37.663817+0000","last_became_active":"2023-10-16T11:19:36.680840+0000","last_became_peered":"2023-10-16T11:19:36.680840+0000","last_unstale":"2023-10-16T11:19:37.663817+0000","last_undegraded":"2023-10-16T11:19:37.663817+0000","last_fullsized":"2023-10-16T11:19:37.663817+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:50.733066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.679010+0000","last_change":"2023-10-16T11:19:23.341165+0000","last_active":"2023-10-16T11:19:36.679010+0000","last_peered":"2023-10-16T11:19:36.679010+0000","last_clean":"2023-10-16T11:19:36.679010+0000","last_became_active":"2023-10-16T11:19:23.340853+0000","last_became_peered":"2023-10-16T11:19:23.340853+0000","last_unstale":"2023-10-16T11:19:36.679010+0000","last_undegraded":"2023-10-16T11:19:36.679010+0000","last_fullsized":"2023-10-16T11:19:36.679010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:41.539293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700038+0000","last_change":"2023-10-16T11:19:36.676667+0000","last_active":"2023-10-16T11:19:37.700038+0000","last_peered":"2023-10-16T11:19:37.700038+0000","last_clean":"2023-10-16T11:19:37.700038+0000","last_became_active":"2023-10-16T11:19:36.676404+0000","last_became_peered":"2023-10-16T11:19:36.676404+0000","last_unstale":"2023-10-16T11:19:37.700038+0000","last_undegraded":"2023-10-16T11:19:37.700038+0000","last_fullsized":"2023-10-16T11:19:37.700038+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:38.793043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678960+0000","last_change":"2023-10-16T11:19:23.341632+0000","last_active":"2023-10-16T11:19:36.678960+0000","last_peered":"2023-10-16T11:19:36.678960+0000","last_clean":"2023-10-16T11:19:36.678960+0000","last_became_active":"2023-10-16T11:19:23.341312+0000","last_became_peered":"2023-10-16T11:19:23.341312+0000","last_unstale":"2023-10-16T11:19:36.678960+0000","last_undegraded":"2023-10-16T11:19:36.678960+0000","last_fullsized":"2023-10-16T11:19:36.678960+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662884+0000","last_change":"2023-10-16T11:19:36.677215+0000","last_active":"2023-10-16T11:19:37.662884+0000","last_peered":"2023-10-16T11:19:37.662884+0000","last_clean":"2023-10-16T11:19:37.662884+0000","last_became_active":"2023-10-16T11:19:36.676952+0000","last_became_peered":"2023-10-16T11:19:36.676952+0000","last_unstale":"2023-10-16T11:19:37.662884+0000","last_undegraded":"2023-10-16T11:19:37.662884+0000","last_fullsized":"2023-10-16T11:19:37.662884+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:41.725192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684353+0000","last_change":"2023-10-16T11:19:23.334105+0000","last_active":"2023-10-16T11:19:36.684353+0000","last_peered":"2023-10-16T11:19:36.684353+0000","last_clean":"2023-10-16T11:19:36.684353+0000","last_became_active":"2023-10-16T11:19:23.333836+0000","last_became_peered":"2023-10-16T11:19:23.333836+0000","last_unstale":"2023-10-16T11:19:36.684353+0000","last_undegraded":"2023-10-16T11:19:36.684353+0000","last_fullsized":"2023-10-16T11:19:36.684353+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.432829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664588+0000","last_change":"2023-10-16T11:19:36.683779+0000","last_active":"2023-10-16T11:19:37.664588+0000","last_peered":"2023-10-16T11:19:37.664588+0000","last_clean":"2023-10-16T11:19:37.664588+0000","last_became_active":"2023-10-16T11:19:36.681105+0000","last_became_peered":"2023-10-16T11:19:36.681105+0000","last_unstale":"2023-10-16T11:19:37.664588+0000","last_undegraded":"2023-10-16T11:19:37.664588+0000","last_fullsized":"2023-10-16T11:19:37.664588+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:08.596894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684713+0000","last_change":"2023-10-16T11:19:23.331248+0000","last_active":"2023-10-16T11:19:36.684713+0000","last_peered":"2023-10-16T11:19:36.684713+0000","last_clean":"2023-10-16T11:19:36.684713+0000","last_became_active":"2023-10-16T11:19:23.330931+0000","last_became_peered":"2023-10-16T11:19:23.330931+0000","last_unstale":"2023-10-16T11:19:36.684713+0000","last_undegraded":"2023-10-16T11:19:36.684713+0000","last_fullsized":"2023-10-16T11:19:36.684713+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697283+0000","last_change":"2023-10-16T11:19:36.690106+0000","last_active":"2023-10-16T11:19:37.697283+0000","last_peered":"2023-10-16T11:19:37.697283+0000","last_clean":"2023-10-16T11:19:37.697283+0000","last_became_active":"2023-10-16T11:19:36.683066+0000","last_became_peered":"2023-10-16T11:19:36.683066+0000","last_unstale":"2023-10-16T11:19:37.697283+0000","last_undegraded":"2023-10-16T11:19:37.697283+0000","last_fullsized":"2023-10-16T11:19:37.697283+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:06.675860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689742+0000","last_change":"2023-10-16T11:19:23.337766+0000","last_active":"2023-10-16T11:19:36.689742+0000","last_peered":"2023-10-16T11:19:36.689742+0000","last_clean":"2023-10-16T11:19:36.689742+0000","last_became_active":"2023-10-16T11:19:23.337503+0000","last_became_peered":"2023-10-16T11:19:23.337503+0000","last_unstale":"2023-10-16T11:19:36.689742+0000","last_undegraded":"2023-10-16T11:19:36.689742+0000","last_fullsized":"2023-10-16T11:19:36.689742+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995075+0000","last_change":"2023-10-16T11:19:36.685234+0000","last_active":"2023-10-16T11:19:37.995075+0000","last_peered":"2023-10-16T11:19:37.995075+0000","last_clean":"2023-10-16T11:19:37.995075+0000","last_became_active":"2023-10-16T11:19:36.679918+0000","last_became_peered":"2023-10-16T11:19:36.679918+0000","last_unstale":"2023-10-16T11:19:37.995075+0000","last_undegraded":"2023-10-16T11:19:37.995075+0000","last_fullsized":"2023-10-16T11:19:37.995075+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:31.563643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663100+0000","last_change":"2023-10-16T11:19:23.333633+0000","last_active":"2023-10-16T11:19:36.663100+0000","last_peered":"2023-10-16T11:19:36.663100+0000","last_clean":"2023-10-16T11:19:36.663100+0000","last_became_active":"2023-10-16T11:19:23.333280+0000","last_became_peered":"2023-10-16T11:19:23.333280+0000","last_unstale":"2023-10-16T11:19:36.663100+0000","last_undegraded":"2023-10-16T11:19:36.663100+0000","last_fullsized":"2023-10-16T11:19:36.663100+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700228+0000","last_change":"2023-10-16T11:19:36.683774+0000","last_active":"2023-10-16T11:19:37.700228+0000","last_peered":"2023-10-16T11:19:37.700228+0000","last_clean":"2023-10-16T11:19:37.700228+0000","last_became_active":"2023-10-16T11:19:36.683334+0000","last_became_peered":"2023-10-16T11:19:36.683334+0000","last_unstale":"2023-10-16T11:19:37.700228+0000","last_undegraded":"2023-10-16T11:19:37.700228+0000","last_fullsized":"2023-10-16T11:19:37.700228+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:58.061612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662748+0000","last_change":"2023-10-16T11:19:23.334483+0000","last_active":"2023-10-16T11:19:36.662748+0000","last_peered":"2023-10-16T11:19:36.662748+0000","last_clean":"2023-10-16T11:19:36.662748+0000","last_became_active":"2023-10-16T11:19:23.333819+0000","last_became_peered":"2023-10-16T11:19:23.333819+0000","last_unstale":"2023-10-16T11:19:36.662748+0000","last_undegraded":"2023-10-16T11:19:36.662748+0000","last_fullsized":"2023-10-16T11:19:36.662748+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696955+0000","last_change":"2023-10-16T11:19:36.692637+0000","last_active":"2023-10-16T11:19:37.696955+0000","last_peered":"2023-10-16T11:19:37.696955+0000","last_clean":"2023-10-16T11:19:37.696955+0000","last_became_active":"2023-10-16T11:19:36.692099+0000","last_became_peered":"2023-10-16T11:19:36.692099+0000","last_unstale":"2023-10-16T11:19:37.696955+0000","last_undegraded":"2023-10-16T11:19:37.696955+0000","last_fullsized":"2023-10-16T11:19:37.696955+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:34.007256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666671+0000","last_change":"2023-10-16T11:19:23.332953+0000","last_active":"2023-10-16T11:19:36.666671+0000","last_peered":"2023-10-16T11:19:36.666671+0000","last_clean":"2023-10-16T11:19:36.666671+0000","last_became_active":"2023-10-16T11:19:23.332662+0000","last_became_peered":"2023-10-16T11:19:23.332662+0000","last_unstale":"2023-10-16T11:19:36.666671+0000","last_undegraded":"2023-10-16T11:19:36.666671+0000","last_fullsized":"2023-10-16T11:19:36.666671+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:27.611049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994792+0000","last_change":"2023-10-16T11:19:36.679892+0000","last_active":"2023-10-16T11:19:37.994792+0000","last_peered":"2023-10-16T11:19:37.994792+0000","last_clean":"2023-10-16T11:19:37.994792+0000","last_became_active":"2023-10-16T11:19:36.679631+0000","last_became_peered":"2023-10-16T11:19:36.679631+0000","last_unstale":"2023-10-16T11:19:37.994792+0000","last_undegraded":"2023-10-16T11:19:37.994792+0000","last_fullsized":"2023-10-16T11:19:37.994792+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:51.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668684+0000","last_change":"2023-10-16T11:19:23.325666+0000","last_active":"2023-10-16T11:19:36.668684+0000","last_peered":"2023-10-16T11:19:36.668684+0000","last_clean":"2023-10-16T11:19:36.668684+0000","last_became_active":"2023-10-16T11:19:23.325400+0000","last_became_peered":"2023-10-16T11:19:23.325400+0000","last_unstale":"2023-10-16T11:19:36.668684+0000","last_undegraded":"2023-10-16T11:19:36.668684+0000","last_fullsized":"2023-10-16T11:19:36.668684+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994649+0000","last_change":"2023-10-16T11:19:36.684451+0000","last_active":"2023-10-16T11:19:37.994649+0000","last_peered":"2023-10-16T11:19:37.994649+0000","last_clean":"2023-10-16T11:19:37.994649+0000","last_became_active":"2023-10-16T11:19:36.684000+0000","last_became_peered":"2023-10-16T11:19:36.684000+0000","last_unstale":"2023-10-16T11:19:37.994649+0000","last_undegraded":"2023-10-16T11:19:37.994649+0000","last_fullsized":"2023-10-16T11:19:37.994649+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:43.889855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.679024+0000","last_change":"2023-10-16T11:19:23.342633+0000","last_active":"2023-10-16T11:19:36.679024+0000","last_peered":"2023-10-16T11:19:36.679024+0000","last_clean":"2023-10-16T11:19:36.679024+0000","last_became_active":"2023-10-16T11:19:23.342346+0000","last_became_peered":"2023-10-16T11:19:23.342346+0000","last_unstale":"2023-10-16T11:19:36.679024+0000","last_undegraded":"2023-10-16T11:19:36.679024+0000","last_fullsized":"2023-10-16T11:19:36.679024+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:12.087797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994149+0000","last_change":"2023-10-16T11:19:36.684843+0000","last_active":"2023-10-16T11:19:37.994149+0000","last_peered":"2023-10-16T11:19:37.994149+0000","last_clean":"2023-10-16T11:19:37.994149+0000","last_became_active":"2023-10-16T11:19:36.684561+0000","last_became_peered":"2023-10-16T11:19:36.684561+0000","last_unstale":"2023-10-16T11:19:37.994149+0000","last_undegraded":"2023-10-16T11:19:37.994149+0000","last_fullsized":"2023-10-16T11:19:37.994149+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:02.904122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666877+0000","last_change":"2023-10-16T11:19:23.330180+0000","last_active":"2023-10-16T11:19:36.666877+0000","last_peered":"2023-10-16T11:19:36.666877+0000","last_clean":"2023-10-16T11:19:36.666877+0000","last_became_active":"2023-10-16T11:19:23.329598+0000","last_became_peered":"2023-10-16T11:19:23.329598+0000","last_unstale":"2023-10-16T11:19:36.666877+0000","last_undegraded":"2023-10-16T11:19:36.666877+0000","last_fullsized":"2023-10-16T11:19:36.666877+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:59.613377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665085+0000","last_change":"2023-10-16T11:19:36.687617+0000","last_active":"2023-10-16T11:19:37.665085+0000","last_peered":"2023-10-16T11:19:37.665085+0000","last_clean":"2023-10-16T11:19:37.665085+0000","last_became_active":"2023-10-16T11:19:36.687333+0000","last_became_peered":"2023-10-16T11:19:36.687333+0000","last_unstale":"2023-10-16T11:19:37.665085+0000","last_undegraded":"2023-10-16T11:19:37.665085+0000","last_fullsized":"2023-10-16T11:19:37.665085+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:52.825820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.674910+0000","last_change":"2023-10-16T11:19:23.335413+0000","last_active":"2023-10-16T11:19:36.674910+0000","last_peered":"2023-10-16T11:19:36.674910+0000","last_clean":"2023-10-16T11:19:36.674910+0000","last_became_active":"2023-10-16T11:19:23.334994+0000","last_became_peered":"2023-10-16T11:19:23.334994+0000","last_unstale":"2023-10-16T11:19:36.674910+0000","last_undegraded":"2023-10-16T11:19:36.674910+0000","last_fullsized":"2023-10-16T11:19:36.674910+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:53.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663876+0000","last_change":"2023-10-16T11:19:36.676983+0000","last_active":"2023-10-16T11:19:37.663876+0000","last_peered":"2023-10-16T11:19:37.663876+0000","last_clean":"2023-10-16T11:19:37.663876+0000","last_became_active":"2023-10-16T11:19:36.676668+0000","last_became_peered":"2023-10-16T11:19:36.676668+0000","last_unstale":"2023-10-16T11:19:37.663876+0000","last_undegraded":"2023-10-16T11:19:37.663876+0000","last_fullsized":"2023-10-16T11:19:37.663876+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:25.390302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667223+0000","last_change":"2023-10-16T11:19:23.333705+0000","last_active":"2023-10-16T11:19:36.667223+0000","last_peered":"2023-10-16T11:19:36.667223+0000","last_clean":"2023-10-16T11:19:36.667223+0000","last_became_active":"2023-10-16T11:19:23.333009+0000","last_became_peered":"2023-10-16T11:19:23.333009+0000","last_unstale":"2023-10-16T11:19:36.667223+0000","last_undegraded":"2023-10-16T11:19:36.667223+0000","last_fullsized":"2023-10-16T11:19:36.667223+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:41.811226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697196+0000","last_change":"2023-10-16T11:19:36.695209+0000","last_active":"2023-10-16T11:19:37.697196+0000","last_peered":"2023-10-16T11:19:37.697196+0000","last_clean":"2023-10-16T11:19:37.697196+0000","last_became_active":"2023-10-16T11:19:36.694904+0000","last_became_peered":"2023-10-16T11:19:36.694904+0000","last_unstale":"2023-10-16T11:19:37.697196+0000","last_undegraded":"2023-10-16T11:19:37.697196+0000","last_fullsized":"2023-10-16T11:19:37.697196+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:47:45.793962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689827+0000","last_change":"2023-10-16T11:19:23.333382+0000","last_active":"2023-10-16T11:19:36.689827+0000","last_peered":"2023-10-16T11:19:36.689827+0000","last_clean":"2023-10-16T11:19:36.689827+0000","last_became_active":"2023-10-16T11:19:23.332643+0000","last_became_peered":"2023-10-16T11:19:23.332643+0000","last_unstale":"2023-10-16T11:19:36.689827+0000","last_undegraded":"2023-10-16T11:19:36.689827+0000","last_fullsized":"2023-10-16T11:19:36.689827+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:54.101849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664649+0000","last_change":"2023-10-16T11:19:36.683067+0000","last_active":"2023-10-16T11:19:37.664649+0000","last_peered":"2023-10-16T11:19:37.664649+0000","last_clean":"2023-10-16T11:19:37.664649+0000","last_became_active":"2023-10-16T11:19:36.682800+0000","last_became_peered":"2023-10-16T11:19:36.682800+0000","last_unstale":"2023-10-16T11:19:37.664649+0000","last_undegraded":"2023-10-16T11:19:37.664649+0000","last_fullsized":"2023-10-16T11:19:37.664649+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:50.857617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662651+0000","last_change":"2023-10-16T11:19:23.336597+0000","last_active":"2023-10-16T11:19:36.662651+0000","last_peered":"2023-10-16T11:19:36.662651+0000","last_clean":"2023-10-16T11:19:36.662651+0000","last_became_active":"2023-10-16T11:19:23.336307+0000","last_became_peered":"2023-10-16T11:19:23.336307+0000","last_unstale":"2023-10-16T11:19:36.662651+0000","last_undegraded":"2023-10-16T11:19:36.662651+0000","last_fullsized":"2023-10-16T11:19:36.662651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994870+0000","last_change":"2023-10-16T11:19:36.678840+0000","last_active":"2023-10-16T11:19:37.994870+0000","last_peered":"2023-10-16T11:19:37.994870+0000","last_clean":"2023-10-16T11:19:37.994870+0000","last_became_active":"2023-10-16T11:19:36.678438+0000","last_became_peered":"2023-10-16T11:19:36.678438+0000","last_unstale":"2023-10-16T11:19:37.994870+0000","last_undegraded":"2023-10-16T11:19:37.994870+0000","last_fullsized":"2023-10-16T11:19:37.994870+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:23.839599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678645+0000","last_change":"2023-10-16T11:19:23.329954+0000","last_active":"2023-10-16T11:19:36.678645+0000","last_peered":"2023-10-16T11:19:36.678645+0000","last_clean":"2023-10-16T11:19:36.678645+0000","last_became_active":"2023-10-16T11:19:23.329688+0000","last_became_peered":"2023-10-16T11:19:23.329688+0000","last_unstale":"2023-10-16T11:19:36.678645+0000","last_undegraded":"2023-10-16T11:19:36.678645+0000","last_fullsized":"2023-10-16T11:19:36.678645+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:11.183201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.699795+0000","last_change":"2023-10-16T11:19:36.676048+0000","last_active":"2023-10-16T11:19:37.699795+0000","last_peered":"2023-10-16T11:19:37.699795+0000","last_clean":"2023-10-16T11:19:37.699795+0000","last_became_active":"2023-10-16T11:19:36.675754+0000","last_became_peered":"2023-10-16T11:19:36.675754+0000","last_unstale":"2023-10-16T11:19:37.699795+0000","last_undegraded":"2023-10-16T11:19:37.699795+0000","last_fullsized":"2023-10-16T11:19:37.699795+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:58.387878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672717+0000","last_change":"2023-10-16T11:19:23.329203+0000","last_active":"2023-10-16T11:19:36.672717+0000","last_peered":"2023-10-16T11:19:36.672717+0000","last_clean":"2023-10-16T11:19:36.672717+0000","last_became_active":"2023-10-16T11:19:23.328748+0000","last_became_peered":"2023-10-16T11:19:23.328748+0000","last_unstale":"2023-10-16T11:19:36.672717+0000","last_undegraded":"2023-10-16T11:19:36.672717+0000","last_fullsized":"2023-10-16T11:19:36.672717+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:09.942459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994568+0000","last_change":"2023-10-16T11:19:36.684516+0000","last_active":"2023-10-16T11:19:37.994568+0000","last_peered":"2023-10-16T11:19:37.994568+0000","last_clean":"2023-10-16T11:19:37.994568+0000","last_became_active":"2023-10-16T11:19:36.684256+0000","last_became_peered":"2023-10-16T11:19:36.684256+0000","last_unstale":"2023-10-16T11:19:37.994568+0000","last_undegraded":"2023-10-16T11:19:37.994568+0000","last_fullsized":"2023-10-16T11:19:37.994568+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:12.957525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666433+0000","last_change":"2023-10-16T11:19:23.326663+0000","last_active":"2023-10-16T11:19:36.666433+0000","last_peered":"2023-10-16T11:19:36.666433+0000","last_clean":"2023-10-16T11:19:36.666433+0000","last_became_active":"2023-10-16T11:19:23.326378+0000","last_became_peered":"2023-10-16T11:19:23.326378+0000","last_unstale":"2023-10-16T11:19:36.666433+0000","last_undegraded":"2023-10-16T11:19:36.666433+0000","last_fullsized":"2023-10-16T11:19:36.666433+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:41.823015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663857+0000","last_change":"2023-10-16T11:19:36.684637+0000","last_active":"2023-10-16T11:19:37.663857+0000","last_peered":"2023-10-16T11:19:37.663857+0000","last_clean":"2023-10-16T11:19:37.663857+0000","last_became_active":"2023-10-16T11:19:36.684358+0000","last_became_peered":"2023-10-16T11:19:36.684358+0000","last_unstale":"2023-10-16T11:19:37.663857+0000","last_undegraded":"2023-10-16T11:19:37.663857+0000","last_fullsized":"2023-10-16T11:19:37.663857+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:29.950550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.676015+0000","last_change":"2023-10-16T11:19:23.330646+0000","last_active":"2023-10-16T11:19:36.676015+0000","last_peered":"2023-10-16T11:19:36.676015+0000","last_clean":"2023-10-16T11:19:36.676015+0000","last_became_active":"2023-10-16T11:19:23.330371+0000","last_became_peered":"2023-10-16T11:19:23.330371+0000","last_unstale":"2023-10-16T11:19:36.676015+0000","last_undegraded":"2023-10-16T11:19:36.676015+0000","last_fullsized":"2023-10-16T11:19:36.676015+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:55.845569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995300+0000","last_change":"2023-10-16T11:19:36.670851+0000","last_active":"2023-10-16T11:19:37.995300+0000","last_peered":"2023-10-16T11:19:37.995300+0000","last_clean":"2023-10-16T11:19:37.995300+0000","last_became_active":"2023-10-16T11:19:36.670512+0000","last_became_peered":"2023-10-16T11:19:36.670512+0000","last_unstale":"2023-10-16T11:19:37.995300+0000","last_undegraded":"2023-10-16T11:19:37.995300+0000","last_fullsized":"2023-10-16T11:19:37.995300+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:54.831776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666462+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:36.666462+0000","last_peered":"2023-10-16T11:19:36.666462+0000","last_clean":"2023-10-16T11:19:36.666462+0000","last_became_active":"2023-10-16T11:19:23.335513+0000","last_became_peered":"2023-10-16T11:19:23.335513+0000","last_unstale":"2023-10-16T11:19:36.666462+0000","last_undegraded":"2023-10-16T11:19:36.666462+0000","last_fullsized":"2023-10-16T11:19:36.666462+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:30.545707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700233+0000","last_change":"2023-10-16T11:19:36.686689+0000","last_active":"2023-10-16T11:19:37.700233+0000","last_peered":"2023-10-16T11:19:37.700233+0000","last_clean":"2023-10-16T11:19:37.700233+0000","last_became_active":"2023-10-16T11:19:36.686395+0000","last_became_peered":"2023-10-16T11:19:36.686395+0000","last_unstale":"2023-10-16T11:19:37.700233+0000","last_undegraded":"2023-10-16T11:19:37.700233+0000","last_fullsized":"2023-10-16T11:19:37.700233+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:46.501076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667038+0000","last_change":"2023-10-16T11:19:23.333266+0000","last_active":"2023-10-16T11:19:36.667038+0000","last_peered":"2023-10-16T11:19:36.667038+0000","last_clean":"2023-10-16T11:19:36.667038+0000","last_became_active":"2023-10-16T11:19:23.332412+0000","last_became_peered":"2023-10-16T11:19:23.332412+0000","last_unstale":"2023-10-16T11:19:36.667038+0000","last_undegraded":"2023-10-16T11:19:36.667038+0000","last_fullsized":"2023-10-16T11:19:36.667038+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:19.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664344+0000","last_change":"2023-10-16T11:19:36.687300+0000","last_active":"2023-10-16T11:19:37.664344+0000","last_peered":"2023-10-16T11:19:37.664344+0000","last_clean":"2023-10-16T11:19:37.664344+0000","last_became_active":"2023-10-16T11:19:36.687041+0000","last_became_peered":"2023-10-16T11:19:36.687041+0000","last_unstale":"2023-10-16T11:19:37.664344+0000","last_undegraded":"2023-10-16T11:19:37.664344+0000","last_fullsized":"2023-10-16T11:19:37.664344+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:12.863750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.681800+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:36.681800+0000","last_peered":"2023-10-16T11:19:36.681800+0000","last_clean":"2023-10-16T11:19:36.681800+0000","last_became_active":"2023-10-16T11:19:23.331671+0000","last_became_peered":"2023-10-16T11:19:23.331671+0000","last_unstale":"2023-10-16T11:19:36.681800+0000","last_undegraded":"2023-10-16T11:19:36.681800+0000","last_fullsized":"2023-10-16T11:19:36.681800+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:13.976802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664234+0000","last_change":"2023-10-16T11:19:36.682589+0000","last_active":"2023-10-16T11:19:37.664234+0000","last_peered":"2023-10-16T11:19:37.664234+0000","last_clean":"2023-10-16T11:19:37.664234+0000","last_became_active":"2023-10-16T11:19:36.682311+0000","last_became_peered":"2023-10-16T11:19:36.682311+0000","last_unstale":"2023-10-16T11:19:37.664234+0000","last_undegraded":"2023-10-16T11:19:37.664234+0000","last_fullsized":"2023-10-16T11:19:37.664234+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:52.728822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666733+0000","last_change":"2023-10-16T11:19:23.334441+0000","last_active":"2023-10-16T11:19:36.666733+0000","last_peered":"2023-10-16T11:19:36.666733+0000","last_clean":"2023-10-16T11:19:36.666733+0000","last_became_active":"2023-10-16T11:19:23.334140+0000","last_became_peered":"2023-10-16T11:19:23.334140+0000","last_unstale":"2023-10-16T11:19:36.666733+0000","last_undegraded":"2023-10-16T11:19:36.666733+0000","last_fullsized":"2023-10-16T11:19:36.666733+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3395584,"data_stored":2085172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.67200000000000004,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.372},"max":{"1min":9.4359999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.68400000000000005,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":9.3059999999999992,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.71599999999999997,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":9.3610000000000007,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.119,"5min":0.71999999999999997,"15min":0.68100000000000005},"min":{"1min":0.49399999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.3960000000000008,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.42299999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.72099999999999997,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":9.2889999999999997,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.753,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.57499999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.36099999999999999},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.157,"5min":0.76500000000000001,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":9.4689999999999994,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.61199999999999999,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.375},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.63100000000000001,"15min":0.56699999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.5399999999999991,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.503}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.64000000000000001,"15min":0.57399999999999995},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.69399999999999995,"15min":0.63100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.5690000000000008,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.71199999999999997,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.623},{"interface":"front","average":{"1min":1.155,"5min":0.69799999999999995,"15min":0.64500000000000002},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.44400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575992,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38096,"kb_used_data":2752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333744,"statfs":{"total":96636764160,"available":96597753856,"internally_reserved":0,"allocated":2818048,"data_stored":1496382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.91000000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.60499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.501},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.439}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.70599999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64000000000000001,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.34799999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.335,"5min":3.335,"15min":3.335},"last":0.374}]},{"osd":7,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.39800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575993,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47788,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324052,"statfs":{"total":96636764160,"available":96587829248,"internally_reserved":0,"allocated":2777088,"data_stored":1491203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.622,"15min":0.66700000000000004},"last":0.37},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.436,"15min":0.45200000000000001},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.52500000000000002,"5min":0.63900000000000001,"15min":0.71699999999999997},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.64000000000000001,"15min":0.70099999999999996},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.442},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":0.623,"15min":0.70499999999999996},"last":0.439}]},{"osd":4,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.48099999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65400000000000003,"15min":0.67200000000000004},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":0.89500000000000002,"15min":1.0149999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90000000000000002,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.90300000000000002,"15min":1.0620000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.016},"last":0.58499999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575993,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42796,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329044,"statfs":{"total":96636764160,"available":96592941056,"internally_reserved":0,"allocated":3375104,"data_stored":2081661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.28100000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44400000000000001,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60699999999999998,"5min":1.272,"15min":1.272},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.32200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.249},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67700000000000005,"5min":1.351,"15min":1.351},"last":0.36399999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.46800000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.875,"5min":1.391,"15min":1.391},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80000000000000004,"5min":0.874,"15min":1.089},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.95599999999999996,"15min":1.216},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.69799999999999995},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90900000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.47199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89100000000000001,"5min":1.038,"15min":1.1180000000000001},"last":0.70699999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.68700000000000006}]}]},{"osd":2,"up_from":13,"seq":55834575992,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2789376,"data_stored":1491909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.63300000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69799999999999995,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77800000000000002},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.95399999999999996,"15min":1.097},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67200000000000004,"15min":0.68200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.91800000000000004,"5min":0.96199999999999997,"15min":1.016},"last":0.77100000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.752},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90500000000000003,"5min":1.022,"15min":1.0389999999999999},"last":0.81899999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91800000000000004,"5min":1.022,"15min":1.022},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91900000000000004,"5min":0.98999999999999999,"15min":1.004},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90400000000000003,"5min":1.0569999999999999,"15min":1.071},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.80200000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575992,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51956,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319884,"statfs":{"total":96636764160,"available":96583561216,"internally_reserved":0,"allocated":2793472,"data_stored":1491351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38100000000000001,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.68700000000000006},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":1.0029999999999999},"last":0.34499999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.60899999999999999,"15min":0.67300000000000004},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.74299999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.58999999999999997,"5min":0.66800000000000004,"15min":0.96299999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64200000000000002,"5min":0.69899999999999995,"15min":1.024},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90600000000000003,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"last":0.623}]},{"osd":5,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.99099999999999999,"15min":1.026},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0640000000000001},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41799999999999998},"max":{"1min":0.85999999999999999,"5min":0.92900000000000005,"15min":1.0369999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.0620000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":1.002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88400000000000001,"5min":1.006,"15min":1.1899999999999999},"last":0.59599999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47792,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324048,"statfs":{"total":96636764160,"available":96587825152,"internally_reserved":0,"allocated":2785280,"data_stored":1491875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.502},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.625},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.623},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71599999999999997,"5min":0.82499999999999996,"15min":0.90700000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58799999999999997,"15min":0.623},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.65300000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60599999999999998,"15min":0.63900000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.84399999999999997,"15min":0.88400000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.624},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.83799999999999997,"15min":0.995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.51500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.71199999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.90500000000000003,"15min":1.115},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.35499999999999998,"15min":0.34200000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.83899999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.61599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82492,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94289348,"statfs":{"total":96636764160,"available":96552292352,"internally_reserved":0,"allocated":2801664,"data_stored":1492941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.88600000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60299999999999998},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.86499999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.91400000000000003},"last":0.499}]},{"osd":3,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40899999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85899999999999999,"5min":0.878,"15min":0.878},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55300000000000005,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:19:41.622 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-16T11:19:41.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.742+0000 7f9d12d64700 1 Processor -- start 2023-10-16T11:19:41.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.742+0000 7f9d12d64700 1 -- start start 2023-10-16T11:19:41.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.743+0000 7f9d12d64700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d0c0729b0 0x7f9d0c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.743+0000 7f9d12d64700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c161270 con 0x7f9d0c06ab40 2023-10-16T11:19:41.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.743+0000 7f9d12d64700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c161510 con 0x7f9d0c06e5d0 2023-10-16T11:19:41.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.743+0000 7f9d12d64700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0c1617f0 con 0x7f9d0c0729b0 2023-10-16T11:19:41.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.743+0000 7f9d11301700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d0c0729b0 0x7f9d0c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.743+0000 7f9d10b00700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9d0c06ab40 0x7f9d0c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38650/0 (socket says 172.21.15.158:38650) 2023-10-16T11:19:41.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.743+0000 7f9d10b00700 1 -- 172.21.15.158:0/291438014 learned_addr learned my addr 172.21.15.158:0/291438014 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:41.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d11301700 1 -- 172.21.15.158:0/291438014 >> v1:172.21.15.172:6789/0 conn(0x7f9d0c06e5d0 legacy=0x7f9d0c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:41.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d11301700 1 -- 172.21.15.158:0/291438014 >> v1:172.21.15.158:6789/0 conn(0x7f9d0c06ab40 legacy=0x7f9d0c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d11301700 1 -- 172.21.15.158:0/291438014 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c161ad0 con 0x7f9d0c0729b0 2023-10-16T11:19:41.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d0b7fe700 1 -- 172.21.15.158:0/291438014 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879472434 0 0) 0x7f9d0c161270 con 0x7f9d0c06ab40 2023-10-16T11:19:41.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d11301700 1 --2- 172.21.15.158:0/291438014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c0729b0 0x7f9d0c06d820 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009d90 tx=0x7f9d0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a746320645e64081 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:41.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d0b7fe700 1 -- 172.21.15.158:0/291438014 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0400e070 con 0x7f9d0c0729b0 2023-10-16T11:19:41.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d0b7fe700 1 -- 172.21.15.158:0/291438014 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0400bbd0 con 0x7f9d0c0729b0 2023-10-16T11:19:41.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d0b7fe700 1 -- 172.21.15.158:0/291438014 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04003a20 con 0x7f9d0c0729b0 2023-10-16T11:19:41.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d12d64700 1 -- 172.21.15.158:0/291438014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c0729b0 msgr2=0x7f9d0c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d12d64700 1 --2- 172.21.15.158:0/291438014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c0729b0 0x7f9d0c06d820 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009d90 tx=0x7f9d0400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:41.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d12d64700 1 -- 172.21.15.158:0/291438014 shutdown_connections 2023-10-16T11:19:41.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d12d64700 1 --2- 172.21.15.158:0/291438014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c0729b0 0x7f9d0c06d820 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.744+0000 7f9d12d64700 1 -- 172.21.15.158:0/291438014 >> 172.21.15.158:0/291438014 conn(0x7f9d0c0d02b0 msgr2=0x7f9d0c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:41.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.745+0000 7f9d12d64700 1 -- 172.21.15.158:0/291438014 shutdown_connections 2023-10-16T11:19:41.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.745+0000 7f9d12d64700 1 -- 172.21.15.158:0/291438014 wait complete. 2023-10-16T11:19:41.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.746+0000 7f9d12d64700 1 Processor -- start 2023-10-16T11:19:41.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.746+0000 7f9d12d64700 1 -- start start 2023-10-16T11:19:41.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d12d64700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c06ab40 0x7f9d0c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d12d64700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d0c06e5d0 0x7f9d0c0c4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d12d64700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d0c0729b0 0x7f9d0c0c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d12d64700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c1f7390 con 0x7f9d0c06ab40 2023-10-16T11:19:41.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d12d64700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c1f4790 con 0x7f9d0c0729b0 2023-10-16T11:19:41.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d12d64700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0c69d0 con 0x7f9d0c06e5d0 2023-10-16T11:19:41.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d10b00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c06ab40 0x7f9d0c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d10b00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c06ab40 0x7f9d0c154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38480/0 (socket says 172.21.15.158:38480) 2023-10-16T11:19:41.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.747+0000 7f9d10b00700 1 -- 172.21.15.158:0/4202926075 learned_addr learned my addr 172.21.15.158:0/4202926075 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:41.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d10b00700 1 -- 172.21.15.158:0/4202926075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d0c06e5d0 msgr2=0x7f9d0c0c4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d10b00700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d0c06e5d0 0x7f9d0c0c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d10b00700 1 -- 172.21.15.158:0/4202926075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d0c0729b0 msgr2=0x7f9d0c0c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:41.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d10b00700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d0c0729b0 0x7f9d0c0c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:41.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d10b00700 1 -- 172.21.15.158:0/4202926075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04009a70 con 0x7f9d0c06ab40 2023-10-16T11:19:41.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d10b00700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c06ab40 0x7f9d0c154b40 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000c750 tx=0x7f9d0000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:41.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d097fa700 1 -- 172.21.15.158:0/4202926075 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00004c80 con 0x7f9d0c06ab40 2023-10-16T11:19:41.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d097fa700 1 -- 172.21.15.158:0/4202926075 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d00004e00 con 0x7f9d0c06ab40 2023-10-16T11:19:41.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c155080 con 0x7f9d0c06ab40 2023-10-16T11:19:41.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d097fa700 1 -- 172.21.15.158:0/4202926075 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00007310 con 0x7f9d0c06ab40 2023-10-16T11:19:41.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.748+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c1f4140 con 0x7f9d0c06ab40 2023-10-16T11:19:41.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.749+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c0681b0 con 0x7f9d0c06ab40 2023-10-16T11:19:41.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.749+0000 7f9d097fa700 1 -- 172.21.15.158:0/4202926075 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9d00007470 con 0x7f9d0c06ab40 2023-10-16T11:19:41.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.750+0000 7f9d097fa700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9cf006b020 0x7f9cf006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:41.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.750+0000 7f9d097fa700 1 -- 172.21.15.158:0/4202926075 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f9d00089c00 con 0x7f9d0c06ab40 2023-10-16T11:19:41.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.750+0000 7f9d0bfff700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9cf006b020 0x7f9cf006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:41.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.750+0000 7f9d0bfff700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9cf006b020 0x7f9cf006d4e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0b6200 tx=0x7f9cf8007a20 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:41.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:41.754+0000 7f9d097fa700 1 -- 172.21.15.158:0/4202926075 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d00090080 con 0x7f9d0c06ab40 2023-10-16T11:19:42.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.029+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f9d0c0b6230 con 0x7f9d0c06ab40 2023-10-16T11:19:42.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.030+0000 7f9d097fa700 1 -- 172.21.15.158:0/4202926075 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1125 (secure 0 0 0) 0x7f9d00054520 con 0x7f9d0c06ab40 2023-10-16T11:19:42.032 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:42.032 INFO:teuthology.orchestra.run.smithi158.stdout:{"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5797,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":221,"num_osds":8,"num_up_osds":8,"osd_up_since":1697449394,"num_in_osds":8,"osd_in_since":1697449386,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":193}],"num_pgs":193,"num_pools":5,"num_objects":46,"data_bytes":596435,"bytes_used":421138432,"bytes_avail":772672974848,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":118,"id":13,"up":1,"in":1,"max":1,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"d","status":"up:active","gid":11586},{"filesystem_id":13,"rank":0,"name":"b","status":"up:active","gid":38819}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2023-10-16T11:18:55.955583+0000","services":{}},"progress_events":{}} 2023-10-16T11:19:42.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9cf006b020 msgr2=0x7f9cf006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9cf006b020 0x7f9cf006d4e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0b6200 tx=0x7f9cf8007a20 comp rx=0 tx=0).stop 2023-10-16T11:19:42.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c06ab40 msgr2=0x7f9d0c154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c06ab40 0x7f9d0c154b40 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000c750 tx=0x7f9d0000cb10 comp rx=0 tx=0).stop 2023-10-16T11:19:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 shutdown_connections 2023-10-16T11:19:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d0c0729b0 0x7f9d0c0c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f9cf006b020 0x7f9cf006d4e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d0c06ab40 0x7f9d0c154b40 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 --2- 172.21.15.158:0/4202926075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d0c06e5d0 0x7f9d0c0c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 >> 172.21.15.158:0/4202926075 conn(0x7f9d0c0d02b0 msgr2=0x7f9d0c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:42.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 shutdown_connections 2023-10-16T11:19:42.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.032+0000 7f9d12d64700 1 -- 172.21.15.158:0/4202926075 wait complete. 2023-10-16T11:19:42.046 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-16T11:19:42.046 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:19:42.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.165+0000 7f82a069d700 1 Processor -- start 2023-10-16T11:19:42.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.165+0000 7f82a069d700 1 -- start start 2023-10-16T11:19:42.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f82a069d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82980b24c0 0x7f82980ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:42.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f82a069d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829815edd0 con 0x7f829815a5d0 2023-10-16T11:19:42.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f82a069d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82980ce080 con 0x7f829815e150 2023-10-16T11:19:42.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f82a069d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82980ce360 con 0x7f82980b24c0 2023-10-16T11:19:42.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f829dc38700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82980b24c0 0x7f82980ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:42.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f829e439700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f829815a5d0 0x7f82980b1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38664/0 (socket says 172.21.15.158:38664) 2023-10-16T11:19:42.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f829e439700 1 -- 172.21.15.158:0/383926068 learned_addr learned my addr 172.21.15.158:0/383926068 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:42.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.166+0000 7f829dc38700 1 -- 172.21.15.158:0/383926068 >> v1:172.21.15.172:6789/0 conn(0x7f829815e150 legacy=0x7f829815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:42.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f829dc38700 1 -- 172.21.15.158:0/383926068 >> v1:172.21.15.158:6789/0 conn(0x7f829815a5d0 legacy=0x7f82980b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f829d437700 1 -- 172.21.15.158:0/383926068 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046986460 0 0) 0x7f829815edd0 con 0x7f829815a5d0 2023-10-16T11:19:42.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f829dc38700 1 -- 172.21.15.158:0/383926068 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82980ce640 con 0x7f82980b24c0 2023-10-16T11:19:42.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f829dc38700 1 --2- 172.21.15.158:0/383926068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82980b24c0 0x7f82980ceb60 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f829400bae0 tx=0x7f829400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30680a673b8d665 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:42.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f829d437700 1 -- 172.21.15.158:0/383926068 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829400dcf0 con 0x7f82980b24c0 2023-10-16T11:19:42.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f829d437700 1 -- 172.21.15.158:0/383926068 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829400de50 con 0x7f82980b24c0 2023-10-16T11:19:42.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f829d437700 1 -- 172.21.15.158:0/383926068 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294004250 con 0x7f82980b24c0 2023-10-16T11:19:42.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f82a069d700 1 -- 172.21.15.158:0/383926068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82980b24c0 msgr2=0x7f82980ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f82a069d700 1 --2- 172.21.15.158:0/383926068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82980b24c0 0x7f82980ceb60 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f829400bae0 tx=0x7f829400bdf0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f82a069d700 1 -- 172.21.15.158:0/383926068 shutdown_connections 2023-10-16T11:19:42.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f82a069d700 1 --2- 172.21.15.158:0/383926068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82980b24c0 0x7f82980ceb60 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f82a069d700 1 -- 172.21.15.158:0/383926068 >> 172.21.15.158:0/383926068 conn(0x7f82980ccdb0 msgr2=0x7f82980cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:42.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.167+0000 7f82a069d700 1 -- 172.21.15.158:0/383926068 shutdown_connections 2023-10-16T11:19:42.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.168+0000 7f82a069d700 1 -- 172.21.15.158:0/383926068 wait complete. 2023-10-16T11:19:42.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 Processor -- start 2023-10-16T11:19:42.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 -- start start 2023-10-16T11:19:42.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82980b24c0 0x7f8298157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:42.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829815a5d0 0x7f8298158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:42.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829815e150 0x7f82980d0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:42.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8298148280 con 0x7f829815a5d0 2023-10-16T11:19:42.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829815af10 con 0x7f829815e150 2023-10-16T11:19:42.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.169+0000 7f82a069d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8298147c30 con 0x7f82980b24c0 2023-10-16T11:19:42.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829815a5d0 0x7f8298158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:42.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829815a5d0 0x7f8298158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38496/0 (socket says 172.21.15.158:38496) 2023-10-16T11:19:42.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 -- 172.21.15.158:0/3494746307 learned_addr learned my addr 172.21.15.158:0/3494746307 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:42.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829e439700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82980b24c0 0x7f8298157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:42.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829ec3a700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829815e150 0x7f82980d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:42.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 -- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82980b24c0 msgr2=0x7f8298157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82980b24c0 0x7f8298157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 -- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829815e150 msgr2=0x7f82980d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829815e150 0x7f82980d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 -- 172.21.15.158:0/3494746307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829400b740 con 0x7f829815a5d0 2023-10-16T11:19:42.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f829dc38700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829815a5d0 0x7f8298158740 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f829806d510 tx=0x7f8294004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:42.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f828effd700 1 -- 172.21.15.158:0/3494746307 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294004230 con 0x7f829815a5d0 2023-10-16T11:19:42.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f828effd700 1 -- 172.21.15.158:0/3494746307 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829401f040 con 0x7f829815a5d0 2023-10-16T11:19:42.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f828effd700 1 -- 172.21.15.158:0/3494746307 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294010690 con 0x7f829815a5d0 2023-10-16T11:19:42.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8298147f10 con 0x7f829815a5d0 2023-10-16T11:19:42.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.170+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829815b1f0 con 0x7f829815a5d0 2023-10-16T11:19:42.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.171+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8298148430 con 0x7f829815a5d0 2023-10-16T11:19:42.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.172+0000 7f828effd700 1 -- 172.21.15.158:0/3494746307 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f829402c410 con 0x7f829815a5d0 2023-10-16T11:19:42.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.172+0000 7f828effd700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f827c06b020 0x7f827c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:42.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.172+0000 7f828effd700 1 -- 172.21.15.158:0/3494746307 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f829408dc90 con 0x7f829815a5d0 2023-10-16T11:19:42.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.172+0000 7f829e439700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f827c06b020 0x7f827c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:42.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.173+0000 7f829e439700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f827c06b020 0x7f827c06d4e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8288000f60 tx=0x7f8288007be0 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:42.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.176+0000 7f828effd700 1 -- 172.21.15.158:0/3494746307 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8298148430 con 0x7f829815a5d0 2023-10-16T11:19:42.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.364+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82980681b0 con 0x7f827c06b020 2023-10-16T11:19:42.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.377+0000 7f828effd700 1 -- 172.21.15.158:0/3494746307 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528859 (secure 0 0 0) 0x7f82980681b0 con 0x7f827c06b020 2023-10-16T11:19:42.385 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:42.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f827c06b020 msgr2=0x7f827c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f827c06b020 0x7f827c06d4e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8288000f60 tx=0x7f8288007be0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829815a5d0 msgr2=0x7f8298158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:42.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829815a5d0 0x7f8298158740 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f829806d510 tx=0x7f8294004750 comp rx=0 tx=0).stop 2023-10-16T11:19:42.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 shutdown_connections 2023-10-16T11:19:42.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f829815e150 0x7f82980d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82980b24c0 0x7f8298157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f827c06b020 0x7f827c06d4e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 --2- 172.21.15.158:0/3494746307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f829815a5d0 0x7f8298158740 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 >> 172.21.15.158:0/3494746307 conn(0x7f82980ccdb0 msgr2=0x7f829806ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 shutdown_connections 2023-10-16T11:19:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:42.382+0000 7f82a069d700 1 -- 172.21.15.158:0/3494746307 wait complete. 2023-10-16T11:19:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:19:42.396 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-16T11:19:41.276814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":376,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":411268,"kb_used_data":22984,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754563452,"statfs":{"total":773094113280,"available":772672974848,"internally_reserved":0,"allocated":23535616,"data_stored":13122494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":397463095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001296"},"pg_stats":[{"pgid":"1.0","version":"207'348","reported_seq":1212,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666350+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:19:36.666350+0000","last_peered":"2023-10-16T11:19:36.666350+0000","last_clean":"2023-10-16T11:19:36.666350+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:19:36.666350+0000","last_undegraded":"2023-10-16T11:19:36.666350+0000","last_fullsized":"2023-10-16T11:19:36.666350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663250+0000","last_change":"2023-10-16T11:19:22.332936+0000","last_active":"2023-10-16T11:19:36.663250+0000","last_peered":"2023-10-16T11:19:36.663250+0000","last_clean":"2023-10-16T11:19:36.663250+0000","last_became_active":"2023-10-16T11:19:22.332598+0000","last_became_peered":"2023-10-16T11:19:22.332598+0000","last_unstale":"2023-10-16T11:19:36.663250+0000","last_undegraded":"2023-10-16T11:19:36.663250+0000","last_fullsized":"2023-10-16T11:19:36.663250+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:34.597859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666646+0000","last_change":"2023-10-16T11:19:23.334041+0000","last_active":"2023-10-16T11:19:36.666646+0000","last_peered":"2023-10-16T11:19:36.666646+0000","last_clean":"2023-10-16T11:19:36.666646+0000","last_became_active":"2023-10-16T11:19:23.333752+0000","last_became_peered":"2023-10-16T11:19:23.333752+0000","last_unstale":"2023-10-16T11:19:36.666646+0000","last_undegraded":"2023-10-16T11:19:36.666646+0000","last_fullsized":"2023-10-16T11:19:36.666646+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.685292+0000","last_change":"2023-10-16T11:19:35.677068+0000","last_active":"2023-10-16T11:19:36.685292+0000","last_peered":"2023-10-16T11:19:36.685292+0000","last_clean":"2023-10-16T11:19:36.685292+0000","last_became_active":"2023-10-16T11:19:35.676797+0000","last_became_peered":"2023-10-16T11:19:35.676797+0000","last_unstale":"2023-10-16T11:19:36.685292+0000","last_undegraded":"2023-10-16T11:19:36.685292+0000","last_fullsized":"2023-10-16T11:19:36.685292+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:43.621552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664039+0000","last_change":"2023-10-16T11:19:36.684566+0000","last_active":"2023-10-16T11:19:37.664039+0000","last_peered":"2023-10-16T11:19:37.664039+0000","last_clean":"2023-10-16T11:19:37.664039+0000","last_became_active":"2023-10-16T11:19:36.684276+0000","last_became_peered":"2023-10-16T11:19:36.684276+0000","last_unstale":"2023-10-16T11:19:37.664039+0000","last_undegraded":"2023-10-16T11:19:37.664039+0000","last_fullsized":"2023-10-16T11:19:37.664039+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:37.800770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672051+0000","last_change":"2023-10-16T11:19:22.334152+0000","last_active":"2023-10-16T11:19:36.672051+0000","last_peered":"2023-10-16T11:19:36.672051+0000","last_clean":"2023-10-16T11:19:36.672051+0000","last_became_active":"2023-10-16T11:19:22.333859+0000","last_became_peered":"2023-10-16T11:19:22.333859+0000","last_unstale":"2023-10-16T11:19:36.672051+0000","last_undegraded":"2023-10-16T11:19:36.672051+0000","last_fullsized":"2023-10-16T11:19:36.672051+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:50.619981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668272+0000","last_change":"2023-10-16T11:19:23.324999+0000","last_active":"2023-10-16T11:19:36.668272+0000","last_peered":"2023-10-16T11:19:36.668272+0000","last_clean":"2023-10-16T11:19:36.668272+0000","last_became_active":"2023-10-16T11:19:23.324686+0000","last_became_peered":"2023-10-16T11:19:23.324686+0000","last_unstale":"2023-10-16T11:19:36.668272+0000","last_undegraded":"2023-10-16T11:19:36.668272+0000","last_fullsized":"2023-10-16T11:19:36.668272+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:42.281964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"221'1","reported_seq":13,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994502+0000","last_change":"2023-10-16T11:19:35.674436+0000","last_active":"2023-10-16T11:19:37.994502+0000","last_peered":"2023-10-16T11:19:37.994502+0000","last_clean":"2023-10-16T11:19:37.994502+0000","last_became_active":"2023-10-16T11:19:35.674216+0000","last_became_peered":"2023-10-16T11:19:35.674216+0000","last_unstale":"2023-10-16T11:19:37.994502+0000","last_undegraded":"2023-10-16T11:19:37.994502+0000","last_fullsized":"2023-10-16T11:19:37.994502+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:05.514151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700186+0000","last_change":"2023-10-16T11:19:36.683592+0000","last_active":"2023-10-16T11:19:37.700186+0000","last_peered":"2023-10-16T11:19:37.700186+0000","last_clean":"2023-10-16T11:19:37.700186+0000","last_became_active":"2023-10-16T11:19:36.682898+0000","last_became_peered":"2023-10-16T11:19:36.682898+0000","last_unstale":"2023-10-16T11:19:37.700186+0000","last_undegraded":"2023-10-16T11:19:37.700186+0000","last_fullsized":"2023-10-16T11:19:37.700186+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:35.693733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663874+0000","last_change":"2023-10-16T11:19:22.332640+0000","last_active":"2023-10-16T11:19:36.663874+0000","last_peered":"2023-10-16T11:19:36.663874+0000","last_clean":"2023-10-16T11:19:36.663874+0000","last_became_active":"2023-10-16T11:19:22.331608+0000","last_became_peered":"2023-10-16T11:19:22.331608+0000","last_unstale":"2023-10-16T11:19:36.663874+0000","last_undegraded":"2023-10-16T11:19:36.663874+0000","last_fullsized":"2023-10-16T11:19:36.663874+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670102+0000","last_change":"2023-10-16T11:19:23.329115+0000","last_active":"2023-10-16T11:19:36.670102+0000","last_peered":"2023-10-16T11:19:36.670102+0000","last_clean":"2023-10-16T11:19:36.670102+0000","last_became_active":"2023-10-16T11:19:23.328830+0000","last_became_peered":"2023-10-16T11:19:23.328830+0000","last_unstale":"2023-10-16T11:19:36.670102+0000","last_undegraded":"2023-10-16T11:19:36.670102+0000","last_fullsized":"2023-10-16T11:19:36.670102+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:17.247488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699488+0000","last_change":"2023-10-16T11:19:35.687029+0000","last_active":"2023-10-16T11:19:36.699488+0000","last_peered":"2023-10-16T11:19:36.699488+0000","last_clean":"2023-10-16T11:19:36.699488+0000","last_became_active":"2023-10-16T11:19:35.686699+0000","last_became_peered":"2023-10-16T11:19:35.686699+0000","last_unstale":"2023-10-16T11:19:36.699488+0000","last_undegraded":"2023-10-16T11:19:36.699488+0000","last_fullsized":"2023-10-16T11:19:36.699488+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:52.450005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995162+0000","last_change":"2023-10-16T11:19:36.677702+0000","last_active":"2023-10-16T11:19:37.995162+0000","last_peered":"2023-10-16T11:19:37.995162+0000","last_clean":"2023-10-16T11:19:37.995162+0000","last_became_active":"2023-10-16T11:19:36.677422+0000","last_became_peered":"2023-10-16T11:19:36.677422+0000","last_unstale":"2023-10-16T11:19:37.995162+0000","last_undegraded":"2023-10-16T11:19:37.995162+0000","last_fullsized":"2023-10-16T11:19:37.995162+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:43.889855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663341+0000","last_change":"2023-10-16T11:19:22.329623+0000","last_active":"2023-10-16T11:19:36.663341+0000","last_peered":"2023-10-16T11:19:36.663341+0000","last_clean":"2023-10-16T11:19:36.663341+0000","last_became_active":"2023-10-16T11:19:22.329344+0000","last_became_peered":"2023-10-16T11:19:22.329344+0000","last_unstale":"2023-10-16T11:19:36.663341+0000","last_undegraded":"2023-10-16T11:19:36.663341+0000","last_fullsized":"2023-10-16T11:19:36.663341+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670592+0000","last_change":"2023-10-16T11:19:23.330624+0000","last_active":"2023-10-16T11:19:36.670592+0000","last_peered":"2023-10-16T11:19:36.670592+0000","last_clean":"2023-10-16T11:19:36.670592+0000","last_became_active":"2023-10-16T11:19:23.329588+0000","last_became_peered":"2023-10-16T11:19:23.329588+0000","last_unstale":"2023-10-16T11:19:36.670592+0000","last_undegraded":"2023-10-16T11:19:36.670592+0000","last_fullsized":"2023-10-16T11:19:36.670592+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:10.835410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.691778+0000","last_change":"2023-10-16T11:19:35.685879+0000","last_active":"2023-10-16T11:19:36.691778+0000","last_peered":"2023-10-16T11:19:36.691778+0000","last_clean":"2023-10-16T11:19:36.691778+0000","last_became_active":"2023-10-16T11:19:35.685381+0000","last_became_peered":"2023-10-16T11:19:35.685381+0000","last_unstale":"2023-10-16T11:19:36.691778+0000","last_undegraded":"2023-10-16T11:19:36.691778+0000","last_fullsized":"2023-10-16T11:19:36.691778+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:24.651235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664009+0000","last_change":"2023-10-16T11:19:36.692166+0000","last_active":"2023-10-16T11:19:37.664009+0000","last_peered":"2023-10-16T11:19:37.664009+0000","last_clean":"2023-10-16T11:19:37.664009+0000","last_became_active":"2023-10-16T11:19:36.691802+0000","last_became_peered":"2023-10-16T11:19:36.691802+0000","last_unstale":"2023-10-16T11:19:37.664009+0000","last_undegraded":"2023-10-16T11:19:37.664009+0000","last_fullsized":"2023-10-16T11:19:37.664009+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:31.108155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.683890+0000","last_change":"2023-10-16T11:19:22.333752+0000","last_active":"2023-10-16T11:19:36.683890+0000","last_peered":"2023-10-16T11:19:36.683890+0000","last_clean":"2023-10-16T11:19:36.683890+0000","last_became_active":"2023-10-16T11:19:22.333497+0000","last_became_peered":"2023-10-16T11:19:22.333497+0000","last_unstale":"2023-10-16T11:19:36.683890+0000","last_undegraded":"2023-10-16T11:19:36.683890+0000","last_fullsized":"2023-10-16T11:19:36.683890+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684460+0000","last_change":"2023-10-16T11:19:23.329220+0000","last_active":"2023-10-16T11:19:36.684460+0000","last_peered":"2023-10-16T11:19:36.684460+0000","last_clean":"2023-10-16T11:19:36.684460+0000","last_became_active":"2023-10-16T11:19:23.328923+0000","last_became_peered":"2023-10-16T11:19:23.328923+0000","last_unstale":"2023-10-16T11:19:36.684460+0000","last_undegraded":"2023-10-16T11:19:36.684460+0000","last_fullsized":"2023-10-16T11:19:36.684460+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667680+0000","last_change":"2023-10-16T11:19:35.674427+0000","last_active":"2023-10-16T11:19:36.667680+0000","last_peered":"2023-10-16T11:19:36.667680+0000","last_clean":"2023-10-16T11:19:36.667680+0000","last_became_active":"2023-10-16T11:19:35.674155+0000","last_became_peered":"2023-10-16T11:19:35.674155+0000","last_unstale":"2023-10-16T11:19:36.667680+0000","last_undegraded":"2023-10-16T11:19:36.667680+0000","last_fullsized":"2023-10-16T11:19:36.667680+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:25.009353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995237+0000","last_change":"2023-10-16T11:19:36.687286+0000","last_active":"2023-10-16T11:19:37.995237+0000","last_peered":"2023-10-16T11:19:37.995237+0000","last_clean":"2023-10-16T11:19:37.995237+0000","last_became_active":"2023-10-16T11:19:36.686995+0000","last_became_peered":"2023-10-16T11:19:36.686995+0000","last_unstale":"2023-10-16T11:19:37.995237+0000","last_undegraded":"2023-10-16T11:19:37.995237+0000","last_fullsized":"2023-10-16T11:19:37.995237+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:10.501316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664830+0000","last_change":"2023-10-16T11:19:22.335923+0000","last_active":"2023-10-16T11:19:36.664830+0000","last_peered":"2023-10-16T11:19:36.664830+0000","last_clean":"2023-10-16T11:19:36.664830+0000","last_became_active":"2023-10-16T11:19:22.335705+0000","last_became_peered":"2023-10-16T11:19:22.335705+0000","last_unstale":"2023-10-16T11:19:36.664830+0000","last_undegraded":"2023-10-16T11:19:36.664830+0000","last_fullsized":"2023-10-16T11:19:36.664830+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:40.803363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665902+0000","last_change":"2023-10-16T11:19:23.336160+0000","last_active":"2023-10-16T11:19:36.665902+0000","last_peered":"2023-10-16T11:19:36.665902+0000","last_clean":"2023-10-16T11:19:36.665902+0000","last_became_active":"2023-10-16T11:19:23.333920+0000","last_became_peered":"2023-10-16T11:19:23.333920+0000","last_unstale":"2023-10-16T11:19:36.665902+0000","last_undegraded":"2023-10-16T11:19:36.665902+0000","last_fullsized":"2023-10-16T11:19:36.665902+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671203+0000","last_change":"2023-10-16T11:19:35.671256+0000","last_active":"2023-10-16T11:19:36.671203+0000","last_peered":"2023-10-16T11:19:36.671203+0000","last_clean":"2023-10-16T11:19:36.671203+0000","last_became_active":"2023-10-16T11:19:35.670977+0000","last_became_peered":"2023-10-16T11:19:35.670977+0000","last_unstale":"2023-10-16T11:19:36.671203+0000","last_undegraded":"2023-10-16T11:19:36.671203+0000","last_fullsized":"2023-10-16T11:19:36.671203+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:00.113603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663985+0000","last_change":"2023-10-16T11:19:36.680876+0000","last_active":"2023-10-16T11:19:37.663985+0000","last_peered":"2023-10-16T11:19:37.663985+0000","last_clean":"2023-10-16T11:19:37.663985+0000","last_became_active":"2023-10-16T11:19:36.680576+0000","last_became_peered":"2023-10-16T11:19:36.680576+0000","last_unstale":"2023-10-16T11:19:37.663985+0000","last_undegraded":"2023-10-16T11:19:37.663985+0000","last_fullsized":"2023-10-16T11:19:37.663985+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:52.686759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666970+0000","last_change":"2023-10-16T11:19:22.326754+0000","last_active":"2023-10-16T11:19:36.666970+0000","last_peered":"2023-10-16T11:19:36.666970+0000","last_clean":"2023-10-16T11:19:36.666970+0000","last_became_active":"2023-10-16T11:19:22.326480+0000","last_became_peered":"2023-10-16T11:19:22.326480+0000","last_unstale":"2023-10-16T11:19:36.666970+0000","last_undegraded":"2023-10-16T11:19:36.666970+0000","last_fullsized":"2023-10-16T11:19:36.666970+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663142+0000","last_change":"2023-10-16T11:19:23.331571+0000","last_active":"2023-10-16T11:19:36.663142+0000","last_peered":"2023-10-16T11:19:36.663142+0000","last_clean":"2023-10-16T11:19:36.663142+0000","last_became_active":"2023-10-16T11:19:23.331242+0000","last_became_peered":"2023-10-16T11:19:23.331242+0000","last_unstale":"2023-10-16T11:19:36.663142+0000","last_undegraded":"2023-10-16T11:19:36.663142+0000","last_fullsized":"2023-10-16T11:19:36.663142+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684254+0000","last_change":"2023-10-16T11:19:35.679089+0000","last_active":"2023-10-16T11:19:36.684254+0000","last_peered":"2023-10-16T11:19:36.684254+0000","last_clean":"2023-10-16T11:19:36.684254+0000","last_became_active":"2023-10-16T11:19:35.678420+0000","last_became_peered":"2023-10-16T11:19:35.678420+0000","last_unstale":"2023-10-16T11:19:36.684254+0000","last_undegraded":"2023-10-16T11:19:36.684254+0000","last_fullsized":"2023-10-16T11:19:36.684254+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:30.151493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664447+0000","last_change":"2023-10-16T11:19:36.681755+0000","last_active":"2023-10-16T11:19:37.664447+0000","last_peered":"2023-10-16T11:19:37.664447+0000","last_clean":"2023-10-16T11:19:37.664447+0000","last_became_active":"2023-10-16T11:19:36.681399+0000","last_became_peered":"2023-10-16T11:19:36.681399+0000","last_unstale":"2023-10-16T11:19:37.664447+0000","last_undegraded":"2023-10-16T11:19:37.664447+0000","last_fullsized":"2023-10-16T11:19:37.664447+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:56.393185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"217'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689930+0000","last_change":"2023-10-16T11:19:22.331182+0000","last_active":"2023-10-16T11:19:36.689930+0000","last_peered":"2023-10-16T11:19:36.689930+0000","last_clean":"2023-10-16T11:19:36.689930+0000","last_became_active":"2023-10-16T11:19:22.330860+0000","last_became_peered":"2023-10-16T11:19:22.330860+0000","last_unstale":"2023-10-16T11:19:36.689930+0000","last_undegraded":"2023-10-16T11:19:36.689930+0000","last_fullsized":"2023-10-16T11:19:36.689930+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667572+0000","last_change":"2023-10-16T11:19:23.325482+0000","last_active":"2023-10-16T11:19:36.667572+0000","last_peered":"2023-10-16T11:19:36.667572+0000","last_clean":"2023-10-16T11:19:36.667572+0000","last_became_active":"2023-10-16T11:19:23.325193+0000","last_became_peered":"2023-10-16T11:19:23.325193+0000","last_unstale":"2023-10-16T11:19:36.667572+0000","last_undegraded":"2023-10-16T11:19:36.667572+0000","last_fullsized":"2023-10-16T11:19:36.667572+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.191635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678478+0000","last_change":"2023-10-16T11:19:35.681024+0000","last_active":"2023-10-16T11:19:36.678478+0000","last_peered":"2023-10-16T11:19:36.678478+0000","last_clean":"2023-10-16T11:19:36.678478+0000","last_became_active":"2023-10-16T11:19:35.677329+0000","last_became_peered":"2023-10-16T11:19:35.677329+0000","last_unstale":"2023-10-16T11:19:36.678478+0000","last_undegraded":"2023-10-16T11:19:36.678478+0000","last_fullsized":"2023-10-16T11:19:36.678478+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:36.304256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696725+0000","last_change":"2023-10-16T11:19:36.694524+0000","last_active":"2023-10-16T11:19:37.696725+0000","last_peered":"2023-10-16T11:19:37.696725+0000","last_clean":"2023-10-16T11:19:37.696725+0000","last_became_active":"2023-10-16T11:19:36.694188+0000","last_became_peered":"2023-10-16T11:19:36.694188+0000","last_unstale":"2023-10-16T11:19:37.696725+0000","last_undegraded":"2023-10-16T11:19:37.696725+0000","last_fullsized":"2023-10-16T11:19:37.696725+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:30.588015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662948+0000","last_change":"2023-10-16T11:19:22.332950+0000","last_active":"2023-10-16T11:19:36.662948+0000","last_peered":"2023-10-16T11:19:36.662948+0000","last_clean":"2023-10-16T11:19:36.662948+0000","last_became_active":"2023-10-16T11:19:22.332650+0000","last_became_peered":"2023-10-16T11:19:22.332650+0000","last_unstale":"2023-10-16T11:19:36.662948+0000","last_undegraded":"2023-10-16T11:19:36.662948+0000","last_fullsized":"2023-10-16T11:19:36.662948+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.680253+0000","last_change":"2023-10-16T11:19:23.333192+0000","last_active":"2023-10-16T11:19:36.680253+0000","last_peered":"2023-10-16T11:19:36.680253+0000","last_clean":"2023-10-16T11:19:36.680253+0000","last_became_active":"2023-10-16T11:19:23.332739+0000","last_became_peered":"2023-10-16T11:19:23.332739+0000","last_unstale":"2023-10-16T11:19:36.680253+0000","last_undegraded":"2023-10-16T11:19:36.680253+0000","last_fullsized":"2023-10-16T11:19:36.680253+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666051+0000","last_change":"2023-10-16T11:19:35.672187+0000","last_active":"2023-10-16T11:19:36.666051+0000","last_peered":"2023-10-16T11:19:36.666051+0000","last_clean":"2023-10-16T11:19:36.666051+0000","last_became_active":"2023-10-16T11:19:35.671883+0000","last_became_peered":"2023-10-16T11:19:35.671883+0000","last_unstale":"2023-10-16T11:19:36.666051+0000","last_undegraded":"2023-10-16T11:19:36.666051+0000","last_fullsized":"2023-10-16T11:19:36.666051+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:27.228484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662842+0000","last_change":"2023-10-16T11:19:36.680498+0000","last_active":"2023-10-16T11:19:37.662842+0000","last_peered":"2023-10-16T11:19:37.662842+0000","last_clean":"2023-10-16T11:19:37.662842+0000","last_became_active":"2023-10-16T11:19:36.680087+0000","last_became_peered":"2023-10-16T11:19:36.680087+0000","last_unstale":"2023-10-16T11:19:37.662842+0000","last_undegraded":"2023-10-16T11:19:37.662842+0000","last_fullsized":"2023-10-16T11:19:37.662842+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:30.982911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666212+0000","last_change":"2023-10-16T11:19:22.326294+0000","last_active":"2023-10-16T11:19:36.666212+0000","last_peered":"2023-10-16T11:19:36.666212+0000","last_clean":"2023-10-16T11:19:36.666212+0000","last_became_active":"2023-10-16T11:19:22.326012+0000","last_became_peered":"2023-10-16T11:19:22.326012+0000","last_unstale":"2023-10-16T11:19:36.666212+0000","last_undegraded":"2023-10-16T11:19:36.666212+0000","last_fullsized":"2023-10-16T11:19:36.666212+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.692071+0000","last_change":"2023-10-16T11:19:23.336920+0000","last_active":"2023-10-16T11:19:36.692071+0000","last_peered":"2023-10-16T11:19:36.692071+0000","last_clean":"2023-10-16T11:19:36.692071+0000","last_became_active":"2023-10-16T11:19:23.336473+0000","last_became_peered":"2023-10-16T11:19:23.336473+0000","last_unstale":"2023-10-16T11:19:36.692071+0000","last_undegraded":"2023-10-16T11:19:36.692071+0000","last_fullsized":"2023-10-16T11:19:36.692071+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:22.421887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.685513+0000","last_change":"2023-10-16T11:19:35.677026+0000","last_active":"2023-10-16T11:19:36.685513+0000","last_peered":"2023-10-16T11:19:36.685513+0000","last_clean":"2023-10-16T11:19:36.685513+0000","last_became_active":"2023-10-16T11:19:35.676697+0000","last_became_peered":"2023-10-16T11:19:35.676697+0000","last_unstale":"2023-10-16T11:19:36.685513+0000","last_undegraded":"2023-10-16T11:19:36.685513+0000","last_fullsized":"2023-10-16T11:19:36.685513+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:49.538337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.699997+0000","last_change":"2023-10-16T11:19:36.675909+0000","last_active":"2023-10-16T11:19:37.699997+0000","last_peered":"2023-10-16T11:19:37.699997+0000","last_clean":"2023-10-16T11:19:37.699997+0000","last_became_active":"2023-10-16T11:19:36.674282+0000","last_became_peered":"2023-10-16T11:19:36.674282+0000","last_unstale":"2023-10-16T11:19:37.699997+0000","last_undegraded":"2023-10-16T11:19:37.699997+0000","last_fullsized":"2023-10-16T11:19:37.699997+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:49.823406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668923+0000","last_change":"2023-10-16T11:19:22.331457+0000","last_active":"2023-10-16T11:19:36.668923+0000","last_peered":"2023-10-16T11:19:36.668923+0000","last_clean":"2023-10-16T11:19:36.668923+0000","last_became_active":"2023-10-16T11:19:22.331010+0000","last_became_peered":"2023-10-16T11:19:22.331010+0000","last_unstale":"2023-10-16T11:19:36.668923+0000","last_undegraded":"2023-10-16T11:19:36.668923+0000","last_fullsized":"2023-10-16T11:19:36.668923+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:38.048633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663202+0000","last_change":"2023-10-16T11:19:23.333035+0000","last_active":"2023-10-16T11:19:36.663202+0000","last_peered":"2023-10-16T11:19:36.663202+0000","last_clean":"2023-10-16T11:19:36.663202+0000","last_became_active":"2023-10-16T11:19:23.332692+0000","last_became_peered":"2023-10-16T11:19:23.332692+0000","last_unstale":"2023-10-16T11:19:36.663202+0000","last_undegraded":"2023-10-16T11:19:36.663202+0000","last_fullsized":"2023-10-16T11:19:36.663202+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.694775+0000","last_change":"2023-10-16T11:19:35.684312+0000","last_active":"2023-10-16T11:19:36.694775+0000","last_peered":"2023-10-16T11:19:36.694775+0000","last_clean":"2023-10-16T11:19:36.694775+0000","last_became_active":"2023-10-16T11:19:35.684055+0000","last_became_peered":"2023-10-16T11:19:35.684055+0000","last_unstale":"2023-10-16T11:19:36.694775+0000","last_undegraded":"2023-10-16T11:19:36.694775+0000","last_fullsized":"2023-10-16T11:19:36.694775+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:48.058006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995020+0000","last_change":"2023-10-16T11:19:36.691309+0000","last_active":"2023-10-16T11:19:37.995020+0000","last_peered":"2023-10-16T11:19:37.995020+0000","last_clean":"2023-10-16T11:19:37.995020+0000","last_became_active":"2023-10-16T11:19:36.690993+0000","last_became_peered":"2023-10-16T11:19:36.690993+0000","last_unstale":"2023-10-16T11:19:37.995020+0000","last_undegraded":"2023-10-16T11:19:37.995020+0000","last_fullsized":"2023-10-16T11:19:37.995020+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:12.957525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675071+0000","last_change":"2023-10-16T11:19:22.329858+0000","last_active":"2023-10-16T11:19:36.675071+0000","last_peered":"2023-10-16T11:19:36.675071+0000","last_clean":"2023-10-16T11:19:36.675071+0000","last_became_active":"2023-10-16T11:19:22.329586+0000","last_became_peered":"2023-10-16T11:19:22.329586+0000","last_unstale":"2023-10-16T11:19:36.675071+0000","last_undegraded":"2023-10-16T11:19:36.675071+0000","last_fullsized":"2023-10-16T11:19:36.675071+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:25.906324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670506+0000","last_change":"2023-10-16T11:19:23.333742+0000","last_active":"2023-10-16T11:19:36.670506+0000","last_peered":"2023-10-16T11:19:36.670506+0000","last_clean":"2023-10-16T11:19:36.670506+0000","last_became_active":"2023-10-16T11:19:23.333424+0000","last_became_peered":"2023-10-16T11:19:23.333424+0000","last_unstale":"2023-10-16T11:19:36.670506+0000","last_undegraded":"2023-10-16T11:19:36.670506+0000","last_fullsized":"2023-10-16T11:19:36.670506+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.156569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684622+0000","last_change":"2023-10-16T11:19:35.685893+0000","last_active":"2023-10-16T11:19:36.684622+0000","last_peered":"2023-10-16T11:19:36.684622+0000","last_clean":"2023-10-16T11:19:36.684622+0000","last_became_active":"2023-10-16T11:19:35.685647+0000","last_became_peered":"2023-10-16T11:19:35.685647+0000","last_unstale":"2023-10-16T11:19:36.684622+0000","last_undegraded":"2023-10-16T11:19:36.684622+0000","last_fullsized":"2023-10-16T11:19:36.684622+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:55.676783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697595+0000","last_change":"2023-10-16T11:19:36.694904+0000","last_active":"2023-10-16T11:19:37.697595+0000","last_peered":"2023-10-16T11:19:37.697595+0000","last_clean":"2023-10-16T11:19:37.697595+0000","last_became_active":"2023-10-16T11:19:36.690439+0000","last_became_peered":"2023-10-16T11:19:36.690439+0000","last_unstale":"2023-10-16T11:19:37.697595+0000","last_undegraded":"2023-10-16T11:19:37.697595+0000","last_fullsized":"2023-10-16T11:19:37.697595+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:37.294828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.673044+0000","last_change":"2023-10-16T11:19:22.329655+0000","last_active":"2023-10-16T11:19:36.673044+0000","last_peered":"2023-10-16T11:19:36.673044+0000","last_clean":"2023-10-16T11:19:36.673044+0000","last_became_active":"2023-10-16T11:19:22.329366+0000","last_became_peered":"2023-10-16T11:19:22.329366+0000","last_unstale":"2023-10-16T11:19:36.673044+0000","last_undegraded":"2023-10-16T11:19:36.673044+0000","last_fullsized":"2023-10-16T11:19:36.673044+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:47.331154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671563+0000","last_change":"2023-10-16T11:19:23.331460+0000","last_active":"2023-10-16T11:19:36.671563+0000","last_peered":"2023-10-16T11:19:36.671563+0000","last_clean":"2023-10-16T11:19:36.671563+0000","last_became_active":"2023-10-16T11:19:23.331178+0000","last_became_peered":"2023-10-16T11:19:23.331178+0000","last_unstale":"2023-10-16T11:19:36.671563+0000","last_undegraded":"2023-10-16T11:19:36.671563+0000","last_fullsized":"2023-10-16T11:19:36.671563+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:36.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699143+0000","last_change":"2023-10-16T11:19:35.683185+0000","last_active":"2023-10-16T11:19:36.699143+0000","last_peered":"2023-10-16T11:19:36.699143+0000","last_clean":"2023-10-16T11:19:36.699143+0000","last_became_active":"2023-10-16T11:19:35.681186+0000","last_became_peered":"2023-10-16T11:19:35.681186+0000","last_unstale":"2023-10-16T11:19:36.699143+0000","last_undegraded":"2023-10-16T11:19:36.699143+0000","last_fullsized":"2023-10-16T11:19:36.699143+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:26.084551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664927+0000","last_change":"2023-10-16T11:19:36.678756+0000","last_active":"2023-10-16T11:19:37.664927+0000","last_peered":"2023-10-16T11:19:37.664927+0000","last_clean":"2023-10-16T11:19:37.664927+0000","last_became_active":"2023-10-16T11:19:36.678452+0000","last_became_peered":"2023-10-16T11:19:36.678452+0000","last_unstale":"2023-10-16T11:19:37.664927+0000","last_undegraded":"2023-10-16T11:19:37.664927+0000","last_fullsized":"2023-10-16T11:19:37.664927+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:55.430561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675077+0000","last_change":"2023-10-16T11:19:22.332355+0000","last_active":"2023-10-16T11:19:36.675077+0000","last_peered":"2023-10-16T11:19:36.675077+0000","last_clean":"2023-10-16T11:19:36.675077+0000","last_became_active":"2023-10-16T11:19:22.331520+0000","last_became_peered":"2023-10-16T11:19:22.331520+0000","last_unstale":"2023-10-16T11:19:36.675077+0000","last_undegraded":"2023-10-16T11:19:36.675077+0000","last_fullsized":"2023-10-16T11:19:36.675077+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:10.682623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684099+0000","last_change":"2023-10-16T11:19:23.331171+0000","last_active":"2023-10-16T11:19:36.684099+0000","last_peered":"2023-10-16T11:19:36.684099+0000","last_clean":"2023-10-16T11:19:36.684099+0000","last_became_active":"2023-10-16T11:19:23.330836+0000","last_became_peered":"2023-10-16T11:19:23.330836+0000","last_unstale":"2023-10-16T11:19:36.684099+0000","last_undegraded":"2023-10-16T11:19:36.684099+0000","last_fullsized":"2023-10-16T11:19:36.684099+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:45.818409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667260+0000","last_change":"2023-10-16T11:19:35.674937+0000","last_active":"2023-10-16T11:19:36.667260+0000","last_peered":"2023-10-16T11:19:36.667260+0000","last_clean":"2023-10-16T11:19:36.667260+0000","last_became_active":"2023-10-16T11:19:35.674670+0000","last_became_peered":"2023-10-16T11:19:35.674670+0000","last_unstale":"2023-10-16T11:19:36.667260+0000","last_undegraded":"2023-10-16T11:19:36.667260+0000","last_fullsized":"2023-10-16T11:19:36.667260+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:50.829956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700060+0000","last_change":"2023-10-16T11:19:36.682803+0000","last_active":"2023-10-16T11:19:37.700060+0000","last_peered":"2023-10-16T11:19:37.700060+0000","last_clean":"2023-10-16T11:19:37.700060+0000","last_became_active":"2023-10-16T11:19:36.682480+0000","last_became_peered":"2023-10-16T11:19:36.682480+0000","last_unstale":"2023-10-16T11:19:37.700060+0000","last_undegraded":"2023-10-16T11:19:37.700060+0000","last_fullsized":"2023-10-16T11:19:37.700060+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:27.320950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678028+0000","last_change":"2023-10-16T11:19:22.318314+0000","last_active":"2023-10-16T11:19:36.678028+0000","last_peered":"2023-10-16T11:19:36.678028+0000","last_clean":"2023-10-16T11:19:36.678028+0000","last_became_active":"2023-10-16T11:19:22.318015+0000","last_became_peered":"2023-10-16T11:19:22.318015+0000","last_unstale":"2023-10-16T11:19:36.678028+0000","last_undegraded":"2023-10-16T11:19:36.678028+0000","last_fullsized":"2023-10-16T11:19:36.678028+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.421816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668042+0000","last_change":"2023-10-16T11:19:23.337274+0000","last_active":"2023-10-16T11:19:36.668042+0000","last_peered":"2023-10-16T11:19:36.668042+0000","last_clean":"2023-10-16T11:19:36.668042+0000","last_became_active":"2023-10-16T11:19:23.336889+0000","last_became_peered":"2023-10-16T11:19:23.336889+0000","last_unstale":"2023-10-16T11:19:36.668042+0000","last_undegraded":"2023-10-16T11:19:36.668042+0000","last_fullsized":"2023-10-16T11:19:36.668042+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699643+0000","last_change":"2023-10-16T11:19:35.679263+0000","last_active":"2023-10-16T11:19:36.699643+0000","last_peered":"2023-10-16T11:19:36.699643+0000","last_clean":"2023-10-16T11:19:36.699643+0000","last_became_active":"2023-10-16T11:19:35.678812+0000","last_became_peered":"2023-10-16T11:19:35.678812+0000","last_unstale":"2023-10-16T11:19:36.699643+0000","last_undegraded":"2023-10-16T11:19:36.699643+0000","last_fullsized":"2023-10-16T11:19:36.699643+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:57.878898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696290+0000","last_change":"2023-10-16T11:19:36.684169+0000","last_active":"2023-10-16T11:19:37.696290+0000","last_peered":"2023-10-16T11:19:37.696290+0000","last_clean":"2023-10-16T11:19:37.696290+0000","last_became_active":"2023-10-16T11:19:36.683898+0000","last_became_peered":"2023-10-16T11:19:36.683898+0000","last_unstale":"2023-10-16T11:19:37.696290+0000","last_undegraded":"2023-10-16T11:19:37.696290+0000","last_fullsized":"2023-10-16T11:19:37.696290+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:04:05.120700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665201+0000","last_change":"2023-10-16T11:19:22.330569+0000","last_active":"2023-10-16T11:19:36.665201+0000","last_peered":"2023-10-16T11:19:36.665201+0000","last_clean":"2023-10-16T11:19:36.665201+0000","last_became_active":"2023-10-16T11:19:22.329740+0000","last_became_peered":"2023-10-16T11:19:22.329740+0000","last_unstale":"2023-10-16T11:19:36.665201+0000","last_undegraded":"2023-10-16T11:19:36.665201+0000","last_fullsized":"2023-10-16T11:19:36.665201+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664845+0000","last_change":"2023-10-16T11:19:23.338701+0000","last_active":"2023-10-16T11:19:36.664845+0000","last_peered":"2023-10-16T11:19:36.664845+0000","last_clean":"2023-10-16T11:19:36.664845+0000","last_became_active":"2023-10-16T11:19:23.334033+0000","last_became_peered":"2023-10-16T11:19:23.334033+0000","last_unstale":"2023-10-16T11:19:36.664845+0000","last_undegraded":"2023-10-16T11:19:36.664845+0000","last_fullsized":"2023-10-16T11:19:36.664845+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672283+0000","last_change":"2023-10-16T11:19:35.677127+0000","last_active":"2023-10-16T11:19:36.672283+0000","last_peered":"2023-10-16T11:19:36.672283+0000","last_clean":"2023-10-16T11:19:36.672283+0000","last_became_active":"2023-10-16T11:19:35.676828+0000","last_became_peered":"2023-10-16T11:19:35.676828+0000","last_unstale":"2023-10-16T11:19:36.672283+0000","last_undegraded":"2023-10-16T11:19:36.672283+0000","last_fullsized":"2023-10-16T11:19:36.672283+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:15.101657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663965+0000","last_change":"2023-10-16T11:19:36.698526+0000","last_active":"2023-10-16T11:19:37.663965+0000","last_peered":"2023-10-16T11:19:37.663965+0000","last_clean":"2023-10-16T11:19:37.663965+0000","last_became_active":"2023-10-16T11:19:36.698271+0000","last_became_peered":"2023-10-16T11:19:36.698271+0000","last_unstale":"2023-10-16T11:19:37.663965+0000","last_undegraded":"2023-10-16T11:19:37.663965+0000","last_fullsized":"2023-10-16T11:19:37.663965+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:53.874459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665100+0000","last_change":"2023-10-16T11:19:22.331432+0000","last_active":"2023-10-16T11:19:36.665100+0000","last_peered":"2023-10-16T11:19:36.665100+0000","last_clean":"2023-10-16T11:19:36.665100+0000","last_became_active":"2023-10-16T11:19:22.331124+0000","last_became_peered":"2023-10-16T11:19:22.331124+0000","last_unstale":"2023-10-16T11:19:36.665100+0000","last_undegraded":"2023-10-16T11:19:36.665100+0000","last_fullsized":"2023-10-16T11:19:36.665100+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.680058+0000","last_change":"2023-10-16T11:19:23.333018+0000","last_active":"2023-10-16T11:19:36.680058+0000","last_peered":"2023-10-16T11:19:36.680058+0000","last_clean":"2023-10-16T11:19:36.680058+0000","last_became_active":"2023-10-16T11:19:23.332725+0000","last_became_peered":"2023-10-16T11:19:23.332725+0000","last_unstale":"2023-10-16T11:19:36.680058+0000","last_undegraded":"2023-10-16T11:19:36.680058+0000","last_fullsized":"2023-10-16T11:19:36.680058+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:39.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666647+0000","last_change":"2023-10-16T11:19:35.669741+0000","last_active":"2023-10-16T11:19:36.666647+0000","last_peered":"2023-10-16T11:19:36.666647+0000","last_clean":"2023-10-16T11:19:36.666647+0000","last_became_active":"2023-10-16T11:19:35.669477+0000","last_became_peered":"2023-10-16T11:19:35.669477+0000","last_unstale":"2023-10-16T11:19:36.666647+0000","last_undegraded":"2023-10-16T11:19:36.666647+0000","last_fullsized":"2023-10-16T11:19:36.666647+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:57.643373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994219+0000","last_change":"2023-10-16T11:19:36.690841+0000","last_active":"2023-10-16T11:19:37.994219+0000","last_peered":"2023-10-16T11:19:37.994219+0000","last_clean":"2023-10-16T11:19:37.994219+0000","last_became_active":"2023-10-16T11:19:36.690558+0000","last_became_peered":"2023-10-16T11:19:36.690558+0000","last_unstale":"2023-10-16T11:19:37.994219+0000","last_undegraded":"2023-10-16T11:19:37.994219+0000","last_fullsized":"2023-10-16T11:19:37.994219+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:30.591636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"216'8","reported_seq":24,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671923+0000","last_change":"2023-10-16T11:19:22.332156+0000","last_active":"2023-10-16T11:19:36.671923+0000","last_peered":"2023-10-16T11:19:36.671923+0000","last_clean":"2023-10-16T11:19:36.671923+0000","last_became_active":"2023-10-16T11:19:22.331863+0000","last_became_peered":"2023-10-16T11:19:22.331863+0000","last_unstale":"2023-10-16T11:19:36.671923+0000","last_undegraded":"2023-10-16T11:19:36.671923+0000","last_fullsized":"2023-10-16T11:19:36.671923+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:36.175530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670117+0000","last_change":"2023-10-16T11:19:23.336992+0000","last_active":"2023-10-16T11:19:36.670117+0000","last_peered":"2023-10-16T11:19:36.670117+0000","last_clean":"2023-10-16T11:19:36.670117+0000","last_became_active":"2023-10-16T11:19:23.336724+0000","last_became_peered":"2023-10-16T11:19:23.336724+0000","last_unstale":"2023-10-16T11:19:36.670117+0000","last_undegraded":"2023-10-16T11:19:36.670117+0000","last_fullsized":"2023-10-16T11:19:36.670117+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:01.326738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667579+0000","last_change":"2023-10-16T11:19:35.678016+0000","last_active":"2023-10-16T11:19:36.667579+0000","last_peered":"2023-10-16T11:19:36.667579+0000","last_clean":"2023-10-16T11:19:36.667579+0000","last_became_active":"2023-10-16T11:19:35.677757+0000","last_became_peered":"2023-10-16T11:19:35.677757+0000","last_unstale":"2023-10-16T11:19:36.667579+0000","last_undegraded":"2023-10-16T11:19:36.667579+0000","last_fullsized":"2023-10-16T11:19:36.667579+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:46.311559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994557+0000","last_change":"2023-10-16T11:19:36.677981+0000","last_active":"2023-10-16T11:19:37.994557+0000","last_peered":"2023-10-16T11:19:37.994557+0000","last_clean":"2023-10-16T11:19:37.994557+0000","last_became_active":"2023-10-16T11:19:36.677677+0000","last_became_peered":"2023-10-16T11:19:36.677677+0000","last_unstale":"2023-10-16T11:19:37.994557+0000","last_undegraded":"2023-10-16T11:19:37.994557+0000","last_fullsized":"2023-10-16T11:19:37.994557+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:31.283029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662222+0000","last_change":"2023-10-16T11:19:22.331044+0000","last_active":"2023-10-16T11:19:36.662222+0000","last_peered":"2023-10-16T11:19:36.662222+0000","last_clean":"2023-10-16T11:19:36.662222+0000","last_became_active":"2023-10-16T11:19:22.330757+0000","last_became_peered":"2023-10-16T11:19:22.330757+0000","last_unstale":"2023-10-16T11:19:36.662222+0000","last_undegraded":"2023-10-16T11:19:36.662222+0000","last_fullsized":"2023-10-16T11:19:36.662222+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662806+0000","last_change":"2023-10-16T11:19:23.333500+0000","last_active":"2023-10-16T11:19:36.662806+0000","last_peered":"2023-10-16T11:19:36.662806+0000","last_clean":"2023-10-16T11:19:36.662806+0000","last_became_active":"2023-10-16T11:19:23.333215+0000","last_became_peered":"2023-10-16T11:19:23.333215+0000","last_unstale":"2023-10-16T11:19:36.662806+0000","last_undegraded":"2023-10-16T11:19:36.662806+0000","last_fullsized":"2023-10-16T11:19:36.662806+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670009+0000","last_change":"2023-10-16T11:19:35.673910+0000","last_active":"2023-10-16T11:19:36.670009+0000","last_peered":"2023-10-16T11:19:36.670009+0000","last_clean":"2023-10-16T11:19:36.670009+0000","last_became_active":"2023-10-16T11:19:35.673640+0000","last_became_peered":"2023-10-16T11:19:35.673640+0000","last_unstale":"2023-10-16T11:19:36.670009+0000","last_undegraded":"2023-10-16T11:19:36.670009+0000","last_fullsized":"2023-10-16T11:19:36.670009+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:42.767262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994757+0000","last_change":"2023-10-16T11:19:36.678756+0000","last_active":"2023-10-16T11:19:37.994757+0000","last_peered":"2023-10-16T11:19:37.994757+0000","last_clean":"2023-10-16T11:19:37.994757+0000","last_became_active":"2023-10-16T11:19:36.678501+0000","last_became_peered":"2023-10-16T11:19:36.678501+0000","last_unstale":"2023-10-16T11:19:37.994757+0000","last_undegraded":"2023-10-16T11:19:37.994757+0000","last_fullsized":"2023-10-16T11:19:37.994757+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:00.784444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667556+0000","last_change":"2023-10-16T11:19:22.327307+0000","last_active":"2023-10-16T11:19:36.667556+0000","last_peered":"2023-10-16T11:19:36.667556+0000","last_clean":"2023-10-16T11:19:36.667556+0000","last_became_active":"2023-10-16T11:19:22.327036+0000","last_became_peered":"2023-10-16T11:19:22.327036+0000","last_unstale":"2023-10-16T11:19:36.667556+0000","last_undegraded":"2023-10-16T11:19:36.667556+0000","last_fullsized":"2023-10-16T11:19:36.667556+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:24.276671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666847+0000","last_change":"2023-10-16T11:19:23.337758+0000","last_active":"2023-10-16T11:19:36.666847+0000","last_peered":"2023-10-16T11:19:36.666847+0000","last_clean":"2023-10-16T11:19:36.666847+0000","last_became_active":"2023-10-16T11:19:23.337147+0000","last_became_peered":"2023-10-16T11:19:23.337147+0000","last_unstale":"2023-10-16T11:19:36.666847+0000","last_undegraded":"2023-10-16T11:19:36.666847+0000","last_fullsized":"2023-10-16T11:19:36.666847+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:06.136692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"220'6","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.700777+0000","last_change":"2023-10-16T11:19:35.682083+0000","last_active":"2023-10-16T11:19:36.700777+0000","last_peered":"2023-10-16T11:19:36.700777+0000","last_clean":"2023-10-16T11:19:36.700777+0000","last_became_active":"2023-10-16T11:19:35.681804+0000","last_became_peered":"2023-10-16T11:19:35.681804+0000","last_unstale":"2023-10-16T11:19:36.700777+0000","last_undegraded":"2023-10-16T11:19:36.700777+0000","last_fullsized":"2023-10-16T11:19:36.700777+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:54.161780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664448+0000","last_change":"2023-10-16T11:19:36.692085+0000","last_active":"2023-10-16T11:19:37.664448+0000","last_peered":"2023-10-16T11:19:37.664448+0000","last_clean":"2023-10-16T11:19:37.664448+0000","last_became_active":"2023-10-16T11:19:36.691313+0000","last_became_peered":"2023-10-16T11:19:36.691313+0000","last_unstale":"2023-10-16T11:19:37.664448+0000","last_undegraded":"2023-10-16T11:19:37.664448+0000","last_fullsized":"2023-10-16T11:19:37.664448+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:54.154198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675831+0000","last_change":"2023-10-16T11:19:22.325030+0000","last_active":"2023-10-16T11:19:36.675831+0000","last_peered":"2023-10-16T11:19:36.675831+0000","last_clean":"2023-10-16T11:19:36.675831+0000","last_became_active":"2023-10-16T11:19:22.324733+0000","last_became_peered":"2023-10-16T11:19:22.324733+0000","last_unstale":"2023-10-16T11:19:36.675831+0000","last_undegraded":"2023-10-16T11:19:36.675831+0000","last_fullsized":"2023-10-16T11:19:36.675831+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:29.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671514+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:36.671514+0000","last_peered":"2023-10-16T11:19:36.671514+0000","last_clean":"2023-10-16T11:19:36.671514+0000","last_became_active":"2023-10-16T11:19:23.331725+0000","last_became_peered":"2023-10-16T11:19:23.331725+0000","last_unstale":"2023-10-16T11:19:36.671514+0000","last_undegraded":"2023-10-16T11:19:36.671514+0000","last_fullsized":"2023-10-16T11:19:36.671514+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:45.043730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"220'3","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.682632+0000","last_change":"2023-10-16T11:19:35.672388+0000","last_active":"2023-10-16T11:19:36.682632+0000","last_peered":"2023-10-16T11:19:36.682632+0000","last_clean":"2023-10-16T11:19:36.682632+0000","last_became_active":"2023-10-16T11:19:35.672090+0000","last_became_peered":"2023-10-16T11:19:35.672090+0000","last_unstale":"2023-10-16T11:19:36.682632+0000","last_undegraded":"2023-10-16T11:19:36.682632+0000","last_fullsized":"2023-10-16T11:19:36.682632+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:15.533409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665100+0000","last_change":"2023-10-16T11:19:36.684006+0000","last_active":"2023-10-16T11:19:37.665100+0000","last_peered":"2023-10-16T11:19:37.665100+0000","last_clean":"2023-10-16T11:19:37.665100+0000","last_became_active":"2023-10-16T11:19:36.683679+0000","last_became_peered":"2023-10-16T11:19:36.683679+0000","last_unstale":"2023-10-16T11:19:37.665100+0000","last_undegraded":"2023-10-16T11:19:37.665100+0000","last_fullsized":"2023-10-16T11:19:37.665100+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:20.376959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"217'8","reported_seq":24,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666769+0000","last_change":"2023-10-16T11:19:22.329082+0000","last_active":"2023-10-16T11:19:36.666769+0000","last_peered":"2023-10-16T11:19:36.666769+0000","last_clean":"2023-10-16T11:19:36.666769+0000","last_became_active":"2023-10-16T11:19:22.328756+0000","last_became_peered":"2023-10-16T11:19:22.328756+0000","last_unstale":"2023-10-16T11:19:36.666769+0000","last_undegraded":"2023-10-16T11:19:36.666769+0000","last_fullsized":"2023-10-16T11:19:36.666769+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.384963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.673087+0000","last_change":"2023-10-16T11:19:23.336196+0000","last_active":"2023-10-16T11:19:36.673087+0000","last_peered":"2023-10-16T11:19:36.673087+0000","last_clean":"2023-10-16T11:19:36.673087+0000","last_became_active":"2023-10-16T11:19:23.335924+0000","last_became_peered":"2023-10-16T11:19:23.335924+0000","last_unstale":"2023-10-16T11:19:36.673087+0000","last_undegraded":"2023-10-16T11:19:36.673087+0000","last_fullsized":"2023-10-16T11:19:36.673087+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:27:01.350215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662408+0000","last_change":"2023-10-16T11:19:35.670005+0000","last_active":"2023-10-16T11:19:36.662408+0000","last_peered":"2023-10-16T11:19:36.662408+0000","last_clean":"2023-10-16T11:19:36.662408+0000","last_became_active":"2023-10-16T11:19:35.669743+0000","last_became_peered":"2023-10-16T11:19:35.669743+0000","last_unstale":"2023-10-16T11:19:36.662408+0000","last_undegraded":"2023-10-16T11:19:36.662408+0000","last_fullsized":"2023-10-16T11:19:36.662408+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:59.779061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696422+0000","last_change":"2023-10-16T11:19:36.697172+0000","last_active":"2023-10-16T11:19:37.696422+0000","last_peered":"2023-10-16T11:19:37.696422+0000","last_clean":"2023-10-16T11:19:37.696422+0000","last_became_active":"2023-10-16T11:19:36.696866+0000","last_became_peered":"2023-10-16T11:19:36.696866+0000","last_unstale":"2023-10-16T11:19:37.696422+0000","last_undegraded":"2023-10-16T11:19:37.696422+0000","last_fullsized":"2023-10-16T11:19:37.696422+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:18.714994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"216'3","reported_seq":19,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662822+0000","last_change":"2023-10-16T11:19:22.331150+0000","last_active":"2023-10-16T11:19:36.662822+0000","last_peered":"2023-10-16T11:19:36.662822+0000","last_clean":"2023-10-16T11:19:36.662822+0000","last_became_active":"2023-10-16T11:19:22.330820+0000","last_became_peered":"2023-10-16T11:19:22.330820+0000","last_unstale":"2023-10-16T11:19:36.662822+0000","last_undegraded":"2023-10-16T11:19:36.662822+0000","last_fullsized":"2023-10-16T11:19:36.662822+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662601+0000","last_change":"2023-10-16T11:19:23.338568+0000","last_active":"2023-10-16T11:19:36.662601+0000","last_peered":"2023-10-16T11:19:36.662601+0000","last_clean":"2023-10-16T11:19:36.662601+0000","last_became_active":"2023-10-16T11:19:23.338280+0000","last_became_peered":"2023-10-16T11:19:23.338280+0000","last_unstale":"2023-10-16T11:19:36.662601+0000","last_undegraded":"2023-10-16T11:19:36.662601+0000","last_fullsized":"2023-10-16T11:19:36.662601+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:40.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699722+0000","last_change":"2023-10-16T11:19:35.686423+0000","last_active":"2023-10-16T11:19:36.699722+0000","last_peered":"2023-10-16T11:19:36.699722+0000","last_clean":"2023-10-16T11:19:36.699722+0000","last_became_active":"2023-10-16T11:19:35.685384+0000","last_became_peered":"2023-10-16T11:19:35.685384+0000","last_unstale":"2023-10-16T11:19:36.699722+0000","last_undegraded":"2023-10-16T11:19:36.699722+0000","last_fullsized":"2023-10-16T11:19:36.699722+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:49.913941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994146+0000","last_change":"2023-10-16T11:19:36.672963+0000","last_active":"2023-10-16T11:19:37.994146+0000","last_peered":"2023-10-16T11:19:37.994146+0000","last_clean":"2023-10-16T11:19:37.994146+0000","last_became_active":"2023-10-16T11:19:36.672611+0000","last_became_peered":"2023-10-16T11:19:36.672611+0000","last_unstale":"2023-10-16T11:19:37.994146+0000","last_undegraded":"2023-10-16T11:19:37.994146+0000","last_fullsized":"2023-10-16T11:19:37.994146+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:17.500716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670020+0000","last_change":"2023-10-16T11:19:22.325211+0000","last_active":"2023-10-16T11:19:36.670020+0000","last_peered":"2023-10-16T11:19:36.670020+0000","last_clean":"2023-10-16T11:19:36.670020+0000","last_became_active":"2023-10-16T11:19:22.324901+0000","last_became_peered":"2023-10-16T11:19:22.324901+0000","last_unstale":"2023-10-16T11:19:36.670020+0000","last_undegraded":"2023-10-16T11:19:36.670020+0000","last_fullsized":"2023-10-16T11:19:36.670020+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:46.432434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666928+0000","last_change":"2023-10-16T11:19:23.331144+0000","last_active":"2023-10-16T11:19:36.666928+0000","last_peered":"2023-10-16T11:19:36.666928+0000","last_clean":"2023-10-16T11:19:36.666928+0000","last_became_active":"2023-10-16T11:19:23.330824+0000","last_became_peered":"2023-10-16T11:19:23.330824+0000","last_unstale":"2023-10-16T11:19:36.666928+0000","last_undegraded":"2023-10-16T11:19:36.666928+0000","last_fullsized":"2023-10-16T11:19:36.666928+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:44:27.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"220'8","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699249+0000","last_change":"2023-10-16T11:19:35.678999+0000","last_active":"2023-10-16T11:19:36.699249+0000","last_peered":"2023-10-16T11:19:36.699249+0000","last_clean":"2023-10-16T11:19:36.699249+0000","last_became_active":"2023-10-16T11:19:35.678522+0000","last_became_peered":"2023-10-16T11:19:35.678522+0000","last_unstale":"2023-10-16T11:19:36.699249+0000","last_undegraded":"2023-10-16T11:19:36.699249+0000","last_fullsized":"2023-10-16T11:19:36.699249+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:51.788613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994331+0000","last_change":"2023-10-16T11:19:36.674240+0000","last_active":"2023-10-16T11:19:37.994331+0000","last_peered":"2023-10-16T11:19:37.994331+0000","last_clean":"2023-10-16T11:19:37.994331+0000","last_became_active":"2023-10-16T11:19:36.673954+0000","last_became_peered":"2023-10-16T11:19:36.673954+0000","last_unstale":"2023-10-16T11:19:37.994331+0000","last_undegraded":"2023-10-16T11:19:37.994331+0000","last_fullsized":"2023-10-16T11:19:37.994331+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:15.742018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.676479+0000","last_change":"2023-10-16T11:19:22.332253+0000","last_active":"2023-10-16T11:19:36.676479+0000","last_peered":"2023-10-16T11:19:36.676479+0000","last_clean":"2023-10-16T11:19:36.676479+0000","last_became_active":"2023-10-16T11:19:22.331970+0000","last_became_peered":"2023-10-16T11:19:22.331970+0000","last_unstale":"2023-10-16T11:19:36.676479+0000","last_undegraded":"2023-10-16T11:19:36.676479+0000","last_fullsized":"2023-10-16T11:19:36.676479+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:19.740978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667271+0000","last_change":"2023-10-16T11:19:23.330327+0000","last_active":"2023-10-16T11:19:36.667271+0000","last_peered":"2023-10-16T11:19:36.667271+0000","last_clean":"2023-10-16T11:19:36.667271+0000","last_became_active":"2023-10-16T11:19:23.330040+0000","last_became_peered":"2023-10-16T11:19:23.330040+0000","last_unstale":"2023-10-16T11:19:36.667271+0000","last_undegraded":"2023-10-16T11:19:36.667271+0000","last_fullsized":"2023-10-16T11:19:36.667271+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664745+0000","last_change":"2023-10-16T11:19:35.677569+0000","last_active":"2023-10-16T11:19:36.664745+0000","last_peered":"2023-10-16T11:19:36.664745+0000","last_clean":"2023-10-16T11:19:36.664745+0000","last_became_active":"2023-10-16T11:19:35.677275+0000","last_became_peered":"2023-10-16T11:19:35.677275+0000","last_unstale":"2023-10-16T11:19:36.664745+0000","last_undegraded":"2023-10-16T11:19:36.664745+0000","last_fullsized":"2023-10-16T11:19:36.664745+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:56.520892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663873+0000","last_change":"2023-10-16T11:19:36.678766+0000","last_active":"2023-10-16T11:19:37.663873+0000","last_peered":"2023-10-16T11:19:37.663873+0000","last_clean":"2023-10-16T11:19:37.663873+0000","last_became_active":"2023-10-16T11:19:36.678481+0000","last_became_peered":"2023-10-16T11:19:36.678481+0000","last_unstale":"2023-10-16T11:19:37.663873+0000","last_undegraded":"2023-10-16T11:19:37.663873+0000","last_fullsized":"2023-10-16T11:19:37.663873+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:23.629013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666314+0000","last_change":"2023-10-16T11:19:22.330754+0000","last_active":"2023-10-16T11:19:36.666314+0000","last_peered":"2023-10-16T11:19:36.666314+0000","last_clean":"2023-10-16T11:19:36.666314+0000","last_became_active":"2023-10-16T11:19:22.330490+0000","last_became_peered":"2023-10-16T11:19:22.330490+0000","last_unstale":"2023-10-16T11:19:36.666314+0000","last_undegraded":"2023-10-16T11:19:36.666314+0000","last_fullsized":"2023-10-16T11:19:36.666314+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:17.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662829+0000","last_change":"2023-10-16T11:19:23.334130+0000","last_active":"2023-10-16T11:19:36.662829+0000","last_peered":"2023-10-16T11:19:36.662829+0000","last_clean":"2023-10-16T11:19:36.662829+0000","last_became_active":"2023-10-16T11:19:23.333073+0000","last_became_peered":"2023-10-16T11:19:23.333073+0000","last_unstale":"2023-10-16T11:19:36.662829+0000","last_undegraded":"2023-10-16T11:19:36.662829+0000","last_fullsized":"2023-10-16T11:19:36.662829+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:38.491191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.697295+0000","last_change":"2023-10-16T11:19:35.683271+0000","last_active":"2023-10-16T11:19:36.697295+0000","last_peered":"2023-10-16T11:19:36.697295+0000","last_clean":"2023-10-16T11:19:36.697295+0000","last_became_active":"2023-10-16T11:19:35.681021+0000","last_became_peered":"2023-10-16T11:19:35.681021+0000","last_unstale":"2023-10-16T11:19:36.697295+0000","last_undegraded":"2023-10-16T11:19:36.697295+0000","last_fullsized":"2023-10-16T11:19:36.697295+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:30.265858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664246+0000","last_change":"2023-10-16T11:19:36.679321+0000","last_active":"2023-10-16T11:19:37.664246+0000","last_peered":"2023-10-16T11:19:37.664246+0000","last_clean":"2023-10-16T11:19:37.664246+0000","last_became_active":"2023-10-16T11:19:36.679019+0000","last_became_peered":"2023-10-16T11:19:36.679019+0000","last_unstale":"2023-10-16T11:19:37.664246+0000","last_undegraded":"2023-10-16T11:19:37.664246+0000","last_fullsized":"2023-10-16T11:19:37.664246+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:00.210732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667231+0000","last_change":"2023-10-16T11:19:22.328061+0000","last_active":"2023-10-16T11:19:36.667231+0000","last_peered":"2023-10-16T11:19:36.667231+0000","last_clean":"2023-10-16T11:19:36.667231+0000","last_became_active":"2023-10-16T11:19:22.327773+0000","last_became_peered":"2023-10-16T11:19:22.327773+0000","last_unstale":"2023-10-16T11:19:36.667231+0000","last_undegraded":"2023-10-16T11:19:36.667231+0000","last_fullsized":"2023-10-16T11:19:36.667231+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:39.077769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668801+0000","last_change":"2023-10-16T11:19:23.330616+0000","last_active":"2023-10-16T11:19:36.668801+0000","last_peered":"2023-10-16T11:19:36.668801+0000","last_clean":"2023-10-16T11:19:36.668801+0000","last_became_active":"2023-10-16T11:19:23.330317+0000","last_became_peered":"2023-10-16T11:19:23.330317+0000","last_unstale":"2023-10-16T11:19:36.668801+0000","last_undegraded":"2023-10-16T11:19:36.668801+0000","last_fullsized":"2023-10-16T11:19:36.668801+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"220'4","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.686252+0000","last_change":"2023-10-16T11:19:35.671218+0000","last_active":"2023-10-16T11:19:36.686252+0000","last_peered":"2023-10-16T11:19:36.686252+0000","last_clean":"2023-10-16T11:19:36.686252+0000","last_became_active":"2023-10-16T11:19:35.670991+0000","last_became_peered":"2023-10-16T11:19:35.670991+0000","last_unstale":"2023-10-16T11:19:36.686252+0000","last_undegraded":"2023-10-16T11:19:36.686252+0000","last_fullsized":"2023-10-16T11:19:36.686252+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:11.399468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665905+0000","last_change":"2023-10-16T11:19:36.687869+0000","last_active":"2023-10-16T11:19:37.665905+0000","last_peered":"2023-10-16T11:19:37.665905+0000","last_clean":"2023-10-16T11:19:37.665905+0000","last_became_active":"2023-10-16T11:19:36.687472+0000","last_became_peered":"2023-10-16T11:19:36.687472+0000","last_unstale":"2023-10-16T11:19:37.665905+0000","last_undegraded":"2023-10-16T11:19:37.665905+0000","last_fullsized":"2023-10-16T11:19:37.665905+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:45.543759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666971+0000","last_change":"2023-10-16T11:19:22.328217+0000","last_active":"2023-10-16T11:19:36.666971+0000","last_peered":"2023-10-16T11:19:36.666971+0000","last_clean":"2023-10-16T11:19:36.666971+0000","last_became_active":"2023-10-16T11:19:22.327901+0000","last_became_peered":"2023-10-16T11:19:22.327901+0000","last_unstale":"2023-10-16T11:19:36.666971+0000","last_undegraded":"2023-10-16T11:19:36.666971+0000","last_fullsized":"2023-10-16T11:19:36.666971+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:48.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671972+0000","last_change":"2023-10-16T11:19:23.336335+0000","last_active":"2023-10-16T11:19:36.671972+0000","last_peered":"2023-10-16T11:19:36.671972+0000","last_clean":"2023-10-16T11:19:36.671972+0000","last_became_active":"2023-10-16T11:19:23.335852+0000","last_became_peered":"2023-10-16T11:19:23.335852+0000","last_unstale":"2023-10-16T11:19:36.671972+0000","last_undegraded":"2023-10-16T11:19:36.671972+0000","last_fullsized":"2023-10-16T11:19:36.671972+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.686520+0000","last_change":"2023-10-16T11:19:35.679257+0000","last_active":"2023-10-16T11:19:36.686520+0000","last_peered":"2023-10-16T11:19:36.686520+0000","last_clean":"2023-10-16T11:19:36.686520+0000","last_became_active":"2023-10-16T11:19:35.678977+0000","last_became_peered":"2023-10-16T11:19:35.678977+0000","last_unstale":"2023-10-16T11:19:36.686520+0000","last_undegraded":"2023-10-16T11:19:36.686520+0000","last_fullsized":"2023-10-16T11:19:36.686520+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:29.586253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697443+0000","last_change":"2023-10-16T11:19:36.683265+0000","last_active":"2023-10-16T11:19:37.697443+0000","last_peered":"2023-10-16T11:19:37.697443+0000","last_clean":"2023-10-16T11:19:37.697443+0000","last_became_active":"2023-10-16T11:19:36.681552+0000","last_became_peered":"2023-10-16T11:19:36.681552+0000","last_unstale":"2023-10-16T11:19:37.697443+0000","last_undegraded":"2023-10-16T11:19:37.697443+0000","last_fullsized":"2023-10-16T11:19:37.697443+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:08:32.673134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663673+0000","last_change":"2023-10-16T11:19:22.329968+0000","last_active":"2023-10-16T11:19:36.663673+0000","last_peered":"2023-10-16T11:19:36.663673+0000","last_clean":"2023-10-16T11:19:36.663673+0000","last_became_active":"2023-10-16T11:19:22.326360+0000","last_became_peered":"2023-10-16T11:19:22.326360+0000","last_unstale":"2023-10-16T11:19:36.663673+0000","last_undegraded":"2023-10-16T11:19:36.663673+0000","last_fullsized":"2023-10-16T11:19:36.663673+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675124+0000","last_change":"2023-10-16T11:19:23.335733+0000","last_active":"2023-10-16T11:19:36.675124+0000","last_peered":"2023-10-16T11:19:36.675124+0000","last_clean":"2023-10-16T11:19:36.675124+0000","last_became_active":"2023-10-16T11:19:23.334936+0000","last_became_peered":"2023-10-16T11:19:23.334936+0000","last_unstale":"2023-10-16T11:19:36.675124+0000","last_undegraded":"2023-10-16T11:19:36.675124+0000","last_fullsized":"2023-10-16T11:19:36.675124+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:08.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.690777+0000","last_change":"2023-10-16T11:19:35.684464+0000","last_active":"2023-10-16T11:19:36.690777+0000","last_peered":"2023-10-16T11:19:36.690777+0000","last_clean":"2023-10-16T11:19:36.690777+0000","last_became_active":"2023-10-16T11:19:35.684139+0000","last_became_peered":"2023-10-16T11:19:35.684139+0000","last_unstale":"2023-10-16T11:19:36.690777+0000","last_undegraded":"2023-10-16T11:19:36.690777+0000","last_fullsized":"2023-10-16T11:19:36.690777+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:52.513114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696386+0000","last_change":"2023-10-16T11:19:36.691931+0000","last_active":"2023-10-16T11:19:37.696386+0000","last_peered":"2023-10-16T11:19:37.696386+0000","last_clean":"2023-10-16T11:19:37.696386+0000","last_became_active":"2023-10-16T11:19:36.691594+0000","last_became_peered":"2023-10-16T11:19:36.691594+0000","last_unstale":"2023-10-16T11:19:37.696386+0000","last_undegraded":"2023-10-16T11:19:37.696386+0000","last_fullsized":"2023-10-16T11:19:37.696386+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:56.795176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.677928+0000","last_change":"2023-10-16T11:19:22.328385+0000","last_active":"2023-10-16T11:19:36.677928+0000","last_peered":"2023-10-16T11:19:36.677928+0000","last_clean":"2023-10-16T11:19:36.677928+0000","last_became_active":"2023-10-16T11:19:22.328096+0000","last_became_peered":"2023-10-16T11:19:22.328096+0000","last_unstale":"2023-10-16T11:19:36.677928+0000","last_undegraded":"2023-10-16T11:19:36.677928+0000","last_fullsized":"2023-10-16T11:19:36.677928+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:40.312176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678838+0000","last_change":"2023-10-16T11:19:23.326944+0000","last_active":"2023-10-16T11:19:36.678838+0000","last_peered":"2023-10-16T11:19:36.678838+0000","last_clean":"2023-10-16T11:19:36.678838+0000","last_became_active":"2023-10-16T11:19:23.326680+0000","last_became_peered":"2023-10-16T11:19:23.326680+0000","last_unstale":"2023-10-16T11:19:36.678838+0000","last_undegraded":"2023-10-16T11:19:36.678838+0000","last_fullsized":"2023-10-16T11:19:36.678838+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:28.070345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.693442+0000","last_change":"2023-10-16T11:19:35.685986+0000","last_active":"2023-10-16T11:19:36.693442+0000","last_peered":"2023-10-16T11:19:36.693442+0000","last_clean":"2023-10-16T11:19:36.693442+0000","last_became_active":"2023-10-16T11:19:35.685758+0000","last_became_peered":"2023-10-16T11:19:35.685758+0000","last_unstale":"2023-10-16T11:19:36.693442+0000","last_undegraded":"2023-10-16T11:19:36.693442+0000","last_fullsized":"2023-10-16T11:19:36.693442+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:12.481754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663581+0000","last_change":"2023-10-16T11:19:36.687085+0000","last_active":"2023-10-16T11:19:37.663581+0000","last_peered":"2023-10-16T11:19:37.663581+0000","last_clean":"2023-10-16T11:19:37.663581+0000","last_became_active":"2023-10-16T11:19:36.686804+0000","last_became_peered":"2023-10-16T11:19:36.686804+0000","last_unstale":"2023-10-16T11:19:37.663581+0000","last_undegraded":"2023-10-16T11:19:37.663581+0000","last_fullsized":"2023-10-16T11:19:37.663581+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:04.397243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667326+0000","last_change":"2023-10-16T11:19:22.332064+0000","last_active":"2023-10-16T11:19:36.667326+0000","last_peered":"2023-10-16T11:19:36.667326+0000","last_clean":"2023-10-16T11:19:36.667326+0000","last_became_active":"2023-10-16T11:19:22.331746+0000","last_became_peered":"2023-10-16T11:19:22.331746+0000","last_unstale":"2023-10-16T11:19:36.667326+0000","last_undegraded":"2023-10-16T11:19:36.667326+0000","last_fullsized":"2023-10-16T11:19:36.667326+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665063+0000","last_change":"2023-10-16T11:19:23.342219+0000","last_active":"2023-10-16T11:19:36.665063+0000","last_peered":"2023-10-16T11:19:36.665063+0000","last_clean":"2023-10-16T11:19:36.665063+0000","last_became_active":"2023-10-16T11:19:23.334494+0000","last_became_peered":"2023-10-16T11:19:23.334494+0000","last_unstale":"2023-10-16T11:19:36.665063+0000","last_undegraded":"2023-10-16T11:19:36.665063+0000","last_fullsized":"2023-10-16T11:19:36.665063+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670212+0000","last_change":"2023-10-16T11:19:35.674368+0000","last_active":"2023-10-16T11:19:36.670212+0000","last_peered":"2023-10-16T11:19:36.670212+0000","last_clean":"2023-10-16T11:19:36.670212+0000","last_became_active":"2023-10-16T11:19:35.674151+0000","last_became_peered":"2023-10-16T11:19:35.674151+0000","last_unstale":"2023-10-16T11:19:36.670212+0000","last_undegraded":"2023-10-16T11:19:36.670212+0000","last_fullsized":"2023-10-16T11:19:36.670212+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.707693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663231+0000","last_change":"2023-10-16T11:19:36.687042+0000","last_active":"2023-10-16T11:19:37.663231+0000","last_peered":"2023-10-16T11:19:37.663231+0000","last_clean":"2023-10-16T11:19:37.663231+0000","last_became_active":"2023-10-16T11:19:36.686749+0000","last_became_peered":"2023-10-16T11:19:36.686749+0000","last_unstale":"2023-10-16T11:19:37.663231+0000","last_undegraded":"2023-10-16T11:19:37.663231+0000","last_fullsized":"2023-10-16T11:19:37.663231+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:46.573220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"216'4","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668738+0000","last_change":"2023-10-16T11:19:22.332308+0000","last_active":"2023-10-16T11:19:36.668738+0000","last_peered":"2023-10-16T11:19:36.668738+0000","last_clean":"2023-10-16T11:19:36.668738+0000","last_became_active":"2023-10-16T11:19:22.332005+0000","last_became_peered":"2023-10-16T11:19:22.332005+0000","last_unstale":"2023-10-16T11:19:36.668738+0000","last_undegraded":"2023-10-16T11:19:36.668738+0000","last_fullsized":"2023-10-16T11:19:36.668738+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670591+0000","last_change":"2023-10-16T11:19:23.341701+0000","last_active":"2023-10-16T11:19:36.670591+0000","last_peered":"2023-10-16T11:19:36.670591+0000","last_clean":"2023-10-16T11:19:36.670591+0000","last_became_active":"2023-10-16T11:19:23.341424+0000","last_became_peered":"2023-10-16T11:19:23.341424+0000","last_unstale":"2023-10-16T11:19:36.670591+0000","last_undegraded":"2023-10-16T11:19:36.670591+0000","last_fullsized":"2023-10-16T11:19:36.670591+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:25.448895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667316+0000","last_change":"2023-10-16T11:19:35.684994+0000","last_active":"2023-10-16T11:19:36.667316+0000","last_peered":"2023-10-16T11:19:36.667316+0000","last_clean":"2023-10-16T11:19:36.667316+0000","last_became_active":"2023-10-16T11:19:35.684739+0000","last_became_peered":"2023-10-16T11:19:35.684739+0000","last_unstale":"2023-10-16T11:19:36.667316+0000","last_undegraded":"2023-10-16T11:19:36.667316+0000","last_fullsized":"2023-10-16T11:19:36.667316+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:46.895984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994879+0000","last_change":"2023-10-16T11:19:36.687519+0000","last_active":"2023-10-16T11:19:37.994879+0000","last_peered":"2023-10-16T11:19:37.994879+0000","last_clean":"2023-10-16T11:19:37.994879+0000","last_became_active":"2023-10-16T11:19:36.686851+0000","last_became_peered":"2023-10-16T11:19:36.686851+0000","last_unstale":"2023-10-16T11:19:37.994879+0000","last_undegraded":"2023-10-16T11:19:37.994879+0000","last_fullsized":"2023-10-16T11:19:37.994879+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:20.736974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666244+0000","last_change":"2023-10-16T11:19:23.323955+0000","last_active":"2023-10-16T11:19:36.666244+0000","last_peered":"2023-10-16T11:19:36.666244+0000","last_clean":"2023-10-16T11:19:36.666244+0000","last_became_active":"2023-10-16T11:19:23.323706+0000","last_became_peered":"2023-10-16T11:19:23.323706+0000","last_unstale":"2023-10-16T11:19:36.666244+0000","last_undegraded":"2023-10-16T11:19:36.666244+0000","last_fullsized":"2023-10-16T11:19:36.666244+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.182127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994845+0000","last_change":"2023-10-16T11:19:36.684069+0000","last_active":"2023-10-16T11:19:37.994845+0000","last_peered":"2023-10-16T11:19:37.994845+0000","last_clean":"2023-10-16T11:19:37.994845+0000","last_became_active":"2023-10-16T11:19:36.683792+0000","last_became_peered":"2023-10-16T11:19:36.683792+0000","last_unstale":"2023-10-16T11:19:37.994845+0000","last_undegraded":"2023-10-16T11:19:37.994845+0000","last_fullsized":"2023-10-16T11:19:37.994845+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:52.124440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666136+0000","last_change":"2023-10-16T11:19:23.332826+0000","last_active":"2023-10-16T11:19:36.666136+0000","last_peered":"2023-10-16T11:19:36.666136+0000","last_clean":"2023-10-16T11:19:36.666136+0000","last_became_active":"2023-10-16T11:19:23.332541+0000","last_became_peered":"2023-10-16T11:19:23.332541+0000","last_unstale":"2023-10-16T11:19:36.666136+0000","last_undegraded":"2023-10-16T11:19:36.666136+0000","last_fullsized":"2023-10-16T11:19:36.666136+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:12:33.418278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994582+0000","last_change":"2023-10-16T11:19:36.677032+0000","last_active":"2023-10-16T11:19:37.994582+0000","last_peered":"2023-10-16T11:19:37.994582+0000","last_clean":"2023-10-16T11:19:37.994582+0000","last_became_active":"2023-10-16T11:19:36.676736+0000","last_became_peered":"2023-10-16T11:19:36.676736+0000","last_unstale":"2023-10-16T11:19:37.994582+0000","last_undegraded":"2023-10-16T11:19:37.994582+0000","last_fullsized":"2023-10-16T11:19:37.994582+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:24.179558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671403+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:36.671403+0000","last_peered":"2023-10-16T11:19:36.671403+0000","last_clean":"2023-10-16T11:19:36.671403+0000","last_became_active":"2023-10-16T11:19:23.335534+0000","last_became_peered":"2023-10-16T11:19:23.335534+0000","last_unstale":"2023-10-16T11:19:36.671403+0000","last_undegraded":"2023-10-16T11:19:36.671403+0000","last_fullsized":"2023-10-16T11:19:36.671403+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994750+0000","last_change":"2023-10-16T11:19:36.684273+0000","last_active":"2023-10-16T11:19:37.994750+0000","last_peered":"2023-10-16T11:19:37.994750+0000","last_clean":"2023-10-16T11:19:37.994750+0000","last_became_active":"2023-10-16T11:19:36.683688+0000","last_became_peered":"2023-10-16T11:19:36.683688+0000","last_unstale":"2023-10-16T11:19:37.994750+0000","last_undegraded":"2023-10-16T11:19:37.994750+0000","last_fullsized":"2023-10-16T11:19:37.994750+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:54.831776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662964+0000","last_change":"2023-10-16T11:19:23.330876+0000","last_active":"2023-10-16T11:19:36.662964+0000","last_peered":"2023-10-16T11:19:36.662964+0000","last_clean":"2023-10-16T11:19:36.662964+0000","last_became_active":"2023-10-16T11:19:23.330597+0000","last_became_peered":"2023-10-16T11:19:23.330597+0000","last_unstale":"2023-10-16T11:19:36.662964+0000","last_undegraded":"2023-10-16T11:19:36.662964+0000","last_fullsized":"2023-10-16T11:19:36.662964+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662848+0000","last_change":"2023-10-16T11:19:36.691604+0000","last_active":"2023-10-16T11:19:37.662848+0000","last_peered":"2023-10-16T11:19:37.662848+0000","last_clean":"2023-10-16T11:19:37.662848+0000","last_became_active":"2023-10-16T11:19:36.690918+0000","last_became_peered":"2023-10-16T11:19:36.690918+0000","last_unstale":"2023-10-16T11:19:37.662848+0000","last_undegraded":"2023-10-16T11:19:37.662848+0000","last_fullsized":"2023-10-16T11:19:37.662848+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:49.453674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664799+0000","last_change":"2023-10-16T11:19:36.684752+0000","last_active":"2023-10-16T11:19:37.664799+0000","last_peered":"2023-10-16T11:19:37.664799+0000","last_clean":"2023-10-16T11:19:37.664799+0000","last_became_active":"2023-10-16T11:19:36.684491+0000","last_became_peered":"2023-10-16T11:19:36.684491+0000","last_unstale":"2023-10-16T11:19:37.664799+0000","last_undegraded":"2023-10-16T11:19:37.664799+0000","last_fullsized":"2023-10-16T11:19:37.664799+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:39.093443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.674996+0000","last_change":"2023-10-16T11:19:23.327468+0000","last_active":"2023-10-16T11:19:36.674996+0000","last_peered":"2023-10-16T11:19:36.674996+0000","last_clean":"2023-10-16T11:19:36.674996+0000","last_became_active":"2023-10-16T11:19:23.327176+0000","last_became_peered":"2023-10-16T11:19:23.327176+0000","last_unstale":"2023-10-16T11:19:36.674996+0000","last_undegraded":"2023-10-16T11:19:36.674996+0000","last_fullsized":"2023-10-16T11:19:36.674996+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:02.654427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700296+0000","last_change":"2023-10-16T11:19:36.685932+0000","last_active":"2023-10-16T11:19:37.700296+0000","last_peered":"2023-10-16T11:19:37.700296+0000","last_clean":"2023-10-16T11:19:37.700296+0000","last_became_active":"2023-10-16T11:19:36.685576+0000","last_became_peered":"2023-10-16T11:19:36.685576+0000","last_unstale":"2023-10-16T11:19:37.700296+0000","last_undegraded":"2023-10-16T11:19:37.700296+0000","last_fullsized":"2023-10-16T11:19:37.700296+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:58:51.519722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666950+0000","last_change":"2023-10-16T11:19:23.332619+0000","last_active":"2023-10-16T11:19:36.666950+0000","last_peered":"2023-10-16T11:19:36.666950+0000","last_clean":"2023-10-16T11:19:36.666950+0000","last_became_active":"2023-10-16T11:19:23.332037+0000","last_became_peered":"2023-10-16T11:19:23.332037+0000","last_unstale":"2023-10-16T11:19:36.666950+0000","last_undegraded":"2023-10-16T11:19:36.666950+0000","last_fullsized":"2023-10-16T11:19:36.666950+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:48.332974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665091+0000","last_change":"2023-10-16T11:19:23.341307+0000","last_active":"2023-10-16T11:19:36.665091+0000","last_peered":"2023-10-16T11:19:36.665091+0000","last_clean":"2023-10-16T11:19:36.665091+0000","last_became_active":"2023-10-16T11:19:23.334457+0000","last_became_peered":"2023-10-16T11:19:23.334457+0000","last_unstale":"2023-10-16T11:19:36.665091+0000","last_undegraded":"2023-10-16T11:19:36.665091+0000","last_fullsized":"2023-10-16T11:19:36.665091+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697007+0000","last_change":"2023-10-16T11:19:36.694609+0000","last_active":"2023-10-16T11:19:37.697007+0000","last_peered":"2023-10-16T11:19:37.697007+0000","last_clean":"2023-10-16T11:19:37.697007+0000","last_became_active":"2023-10-16T11:19:36.692033+0000","last_became_peered":"2023-10-16T11:19:36.692033+0000","last_unstale":"2023-10-16T11:19:37.697007+0000","last_undegraded":"2023-10-16T11:19:37.697007+0000","last_fullsized":"2023-10-16T11:19:37.697007+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:37.217522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664901+0000","last_change":"2023-10-16T11:19:36.678695+0000","last_active":"2023-10-16T11:19:37.664901+0000","last_peered":"2023-10-16T11:19:37.664901+0000","last_clean":"2023-10-16T11:19:37.664901+0000","last_became_active":"2023-10-16T11:19:36.678422+0000","last_became_peered":"2023-10-16T11:19:36.678422+0000","last_unstale":"2023-10-16T11:19:37.664901+0000","last_undegraded":"2023-10-16T11:19:37.664901+0000","last_fullsized":"2023-10-16T11:19:37.664901+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:08.320255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662807+0000","last_change":"2023-10-16T11:19:23.334306+0000","last_active":"2023-10-16T11:19:36.662807+0000","last_peered":"2023-10-16T11:19:36.662807+0000","last_clean":"2023-10-16T11:19:36.662807+0000","last_became_active":"2023-10-16T11:19:23.333460+0000","last_became_peered":"2023-10-16T11:19:23.333460+0000","last_unstale":"2023-10-16T11:19:36.662807+0000","last_undegraded":"2023-10-16T11:19:36.662807+0000","last_fullsized":"2023-10-16T11:19:36.662807+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:53.175386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994177+0000","last_change":"2023-10-16T11:19:36.687715+0000","last_active":"2023-10-16T11:19:37.994177+0000","last_peered":"2023-10-16T11:19:37.994177+0000","last_clean":"2023-10-16T11:19:37.994177+0000","last_became_active":"2023-10-16T11:19:36.687236+0000","last_became_peered":"2023-10-16T11:19:36.687236+0000","last_unstale":"2023-10-16T11:19:37.994177+0000","last_undegraded":"2023-10-16T11:19:37.994177+0000","last_fullsized":"2023-10-16T11:19:37.994177+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:55.480897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671643+0000","last_change":"2023-10-16T11:19:23.341264+0000","last_active":"2023-10-16T11:19:36.671643+0000","last_peered":"2023-10-16T11:19:36.671643+0000","last_clean":"2023-10-16T11:19:36.671643+0000","last_became_active":"2023-10-16T11:19:23.340921+0000","last_became_peered":"2023-10-16T11:19:23.340921+0000","last_unstale":"2023-10-16T11:19:36.671643+0000","last_undegraded":"2023-10-16T11:19:36.671643+0000","last_fullsized":"2023-10-16T11:19:36.671643+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.710291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663817+0000","last_change":"2023-10-16T11:19:36.681734+0000","last_active":"2023-10-16T11:19:37.663817+0000","last_peered":"2023-10-16T11:19:37.663817+0000","last_clean":"2023-10-16T11:19:37.663817+0000","last_became_active":"2023-10-16T11:19:36.680840+0000","last_became_peered":"2023-10-16T11:19:36.680840+0000","last_unstale":"2023-10-16T11:19:37.663817+0000","last_undegraded":"2023-10-16T11:19:37.663817+0000","last_fullsized":"2023-10-16T11:19:37.663817+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:50.733066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.679010+0000","last_change":"2023-10-16T11:19:23.341165+0000","last_active":"2023-10-16T11:19:36.679010+0000","last_peered":"2023-10-16T11:19:36.679010+0000","last_clean":"2023-10-16T11:19:36.679010+0000","last_became_active":"2023-10-16T11:19:23.340853+0000","last_became_peered":"2023-10-16T11:19:23.340853+0000","last_unstale":"2023-10-16T11:19:36.679010+0000","last_undegraded":"2023-10-16T11:19:36.679010+0000","last_fullsized":"2023-10-16T11:19:36.679010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:41.539293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700038+0000","last_change":"2023-10-16T11:19:36.676667+0000","last_active":"2023-10-16T11:19:37.700038+0000","last_peered":"2023-10-16T11:19:37.700038+0000","last_clean":"2023-10-16T11:19:37.700038+0000","last_became_active":"2023-10-16T11:19:36.676404+0000","last_became_peered":"2023-10-16T11:19:36.676404+0000","last_unstale":"2023-10-16T11:19:37.700038+0000","last_undegraded":"2023-10-16T11:19:37.700038+0000","last_fullsized":"2023-10-16T11:19:37.700038+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:38.793043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678960+0000","last_change":"2023-10-16T11:19:23.341632+0000","last_active":"2023-10-16T11:19:36.678960+0000","last_peered":"2023-10-16T11:19:36.678960+0000","last_clean":"2023-10-16T11:19:36.678960+0000","last_became_active":"2023-10-16T11:19:23.341312+0000","last_became_peered":"2023-10-16T11:19:23.341312+0000","last_unstale":"2023-10-16T11:19:36.678960+0000","last_undegraded":"2023-10-16T11:19:36.678960+0000","last_fullsized":"2023-10-16T11:19:36.678960+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662884+0000","last_change":"2023-10-16T11:19:36.677215+0000","last_active":"2023-10-16T11:19:37.662884+0000","last_peered":"2023-10-16T11:19:37.662884+0000","last_clean":"2023-10-16T11:19:37.662884+0000","last_became_active":"2023-10-16T11:19:36.676952+0000","last_became_peered":"2023-10-16T11:19:36.676952+0000","last_unstale":"2023-10-16T11:19:37.662884+0000","last_undegraded":"2023-10-16T11:19:37.662884+0000","last_fullsized":"2023-10-16T11:19:37.662884+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:41.725192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684353+0000","last_change":"2023-10-16T11:19:23.334105+0000","last_active":"2023-10-16T11:19:36.684353+0000","last_peered":"2023-10-16T11:19:36.684353+0000","last_clean":"2023-10-16T11:19:36.684353+0000","last_became_active":"2023-10-16T11:19:23.333836+0000","last_became_peered":"2023-10-16T11:19:23.333836+0000","last_unstale":"2023-10-16T11:19:36.684353+0000","last_undegraded":"2023-10-16T11:19:36.684353+0000","last_fullsized":"2023-10-16T11:19:36.684353+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.432829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664588+0000","last_change":"2023-10-16T11:19:36.683779+0000","last_active":"2023-10-16T11:19:37.664588+0000","last_peered":"2023-10-16T11:19:37.664588+0000","last_clean":"2023-10-16T11:19:37.664588+0000","last_became_active":"2023-10-16T11:19:36.681105+0000","last_became_peered":"2023-10-16T11:19:36.681105+0000","last_unstale":"2023-10-16T11:19:37.664588+0000","last_undegraded":"2023-10-16T11:19:37.664588+0000","last_fullsized":"2023-10-16T11:19:37.664588+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:08.596894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684713+0000","last_change":"2023-10-16T11:19:23.331248+0000","last_active":"2023-10-16T11:19:36.684713+0000","last_peered":"2023-10-16T11:19:36.684713+0000","last_clean":"2023-10-16T11:19:36.684713+0000","last_became_active":"2023-10-16T11:19:23.330931+0000","last_became_peered":"2023-10-16T11:19:23.330931+0000","last_unstale":"2023-10-16T11:19:36.684713+0000","last_undegraded":"2023-10-16T11:19:36.684713+0000","last_fullsized":"2023-10-16T11:19:36.684713+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697283+0000","last_change":"2023-10-16T11:19:36.690106+0000","last_active":"2023-10-16T11:19:37.697283+0000","last_peered":"2023-10-16T11:19:37.697283+0000","last_clean":"2023-10-16T11:19:37.697283+0000","last_became_active":"2023-10-16T11:19:36.683066+0000","last_became_peered":"2023-10-16T11:19:36.683066+0000","last_unstale":"2023-10-16T11:19:37.697283+0000","last_undegraded":"2023-10-16T11:19:37.697283+0000","last_fullsized":"2023-10-16T11:19:37.697283+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:06.675860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689742+0000","last_change":"2023-10-16T11:19:23.337766+0000","last_active":"2023-10-16T11:19:36.689742+0000","last_peered":"2023-10-16T11:19:36.689742+0000","last_clean":"2023-10-16T11:19:36.689742+0000","last_became_active":"2023-10-16T11:19:23.337503+0000","last_became_peered":"2023-10-16T11:19:23.337503+0000","last_unstale":"2023-10-16T11:19:36.689742+0000","last_undegraded":"2023-10-16T11:19:36.689742+0000","last_fullsized":"2023-10-16T11:19:36.689742+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995075+0000","last_change":"2023-10-16T11:19:36.685234+0000","last_active":"2023-10-16T11:19:37.995075+0000","last_peered":"2023-10-16T11:19:37.995075+0000","last_clean":"2023-10-16T11:19:37.995075+0000","last_became_active":"2023-10-16T11:19:36.679918+0000","last_became_peered":"2023-10-16T11:19:36.679918+0000","last_unstale":"2023-10-16T11:19:37.995075+0000","last_undegraded":"2023-10-16T11:19:37.995075+0000","last_fullsized":"2023-10-16T11:19:37.995075+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:31.563643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663100+0000","last_change":"2023-10-16T11:19:23.333633+0000","last_active":"2023-10-16T11:19:36.663100+0000","last_peered":"2023-10-16T11:19:36.663100+0000","last_clean":"2023-10-16T11:19:36.663100+0000","last_became_active":"2023-10-16T11:19:23.333280+0000","last_became_peered":"2023-10-16T11:19:23.333280+0000","last_unstale":"2023-10-16T11:19:36.663100+0000","last_undegraded":"2023-10-16T11:19:36.663100+0000","last_fullsized":"2023-10-16T11:19:36.663100+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700228+0000","last_change":"2023-10-16T11:19:36.683774+0000","last_active":"2023-10-16T11:19:37.700228+0000","last_peered":"2023-10-16T11:19:37.700228+0000","last_clean":"2023-10-16T11:19:37.700228+0000","last_became_active":"2023-10-16T11:19:36.683334+0000","last_became_peered":"2023-10-16T11:19:36.683334+0000","last_unstale":"2023-10-16T11:19:37.700228+0000","last_undegraded":"2023-10-16T11:19:37.700228+0000","last_fullsized":"2023-10-16T11:19:37.700228+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:58.061612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662748+0000","last_change":"2023-10-16T11:19:23.334483+0000","last_active":"2023-10-16T11:19:36.662748+0000","last_peered":"2023-10-16T11:19:36.662748+0000","last_clean":"2023-10-16T11:19:36.662748+0000","last_became_active":"2023-10-16T11:19:23.333819+0000","last_became_peered":"2023-10-16T11:19:23.333819+0000","last_unstale":"2023-10-16T11:19:36.662748+0000","last_undegraded":"2023-10-16T11:19:36.662748+0000","last_fullsized":"2023-10-16T11:19:36.662748+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696955+0000","last_change":"2023-10-16T11:19:36.692637+0000","last_active":"2023-10-16T11:19:37.696955+0000","last_peered":"2023-10-16T11:19:37.696955+0000","last_clean":"2023-10-16T11:19:37.696955+0000","last_became_active":"2023-10-16T11:19:36.692099+0000","last_became_peered":"2023-10-16T11:19:36.692099+0000","last_unstale":"2023-10-16T11:19:37.696955+0000","last_undegraded":"2023-10-16T11:19:37.696955+0000","last_fullsized":"2023-10-16T11:19:37.696955+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:34.007256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666671+0000","last_change":"2023-10-16T11:19:23.332953+0000","last_active":"2023-10-16T11:19:36.666671+0000","last_peered":"2023-10-16T11:19:36.666671+0000","last_clean":"2023-10-16T11:19:36.666671+0000","last_became_active":"2023-10-16T11:19:23.332662+0000","last_became_peered":"2023-10-16T11:19:23.332662+0000","last_unstale":"2023-10-16T11:19:36.666671+0000","last_undegraded":"2023-10-16T11:19:36.666671+0000","last_fullsized":"2023-10-16T11:19:36.666671+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:27.611049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994792+0000","last_change":"2023-10-16T11:19:36.679892+0000","last_active":"2023-10-16T11:19:37.994792+0000","last_peered":"2023-10-16T11:19:37.994792+0000","last_clean":"2023-10-16T11:19:37.994792+0000","last_became_active":"2023-10-16T11:19:36.679631+0000","last_became_peered":"2023-10-16T11:19:36.679631+0000","last_unstale":"2023-10-16T11:19:37.994792+0000","last_undegraded":"2023-10-16T11:19:37.994792+0000","last_fullsized":"2023-10-16T11:19:37.994792+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:51.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668684+0000","last_change":"2023-10-16T11:19:23.325666+0000","last_active":"2023-10-16T11:19:36.668684+0000","last_peered":"2023-10-16T11:19:36.668684+0000","last_clean":"2023-10-16T11:19:36.668684+0000","last_became_active":"2023-10-16T11:19:23.325400+0000","last_became_peered":"2023-10-16T11:19:23.325400+0000","last_unstale":"2023-10-16T11:19:36.668684+0000","last_undegraded":"2023-10-16T11:19:36.668684+0000","last_fullsized":"2023-10-16T11:19:36.668684+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994649+0000","last_change":"2023-10-16T11:19:36.684451+0000","last_active":"2023-10-16T11:19:37.994649+0000","last_peered":"2023-10-16T11:19:37.994649+0000","last_clean":"2023-10-16T11:19:37.994649+0000","last_became_active":"2023-10-16T11:19:36.684000+0000","last_became_peered":"2023-10-16T11:19:36.684000+0000","last_unstale":"2023-10-16T11:19:37.994649+0000","last_undegraded":"2023-10-16T11:19:37.994649+0000","last_fullsized":"2023-10-16T11:19:37.994649+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:43.889855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.679024+0000","last_change":"2023-10-16T11:19:23.342633+0000","last_active":"2023-10-16T11:19:36.679024+0000","last_peered":"2023-10-16T11:19:36.679024+0000","last_clean":"2023-10-16T11:19:36.679024+0000","last_became_active":"2023-10-16T11:19:23.342346+0000","last_became_peered":"2023-10-16T11:19:23.342346+0000","last_unstale":"2023-10-16T11:19:36.679024+0000","last_undegraded":"2023-10-16T11:19:36.679024+0000","last_fullsized":"2023-10-16T11:19:36.679024+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:12.087797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994149+0000","last_change":"2023-10-16T11:19:36.684843+0000","last_active":"2023-10-16T11:19:37.994149+0000","last_peered":"2023-10-16T11:19:37.994149+0000","last_clean":"2023-10-16T11:19:37.994149+0000","last_became_active":"2023-10-16T11:19:36.684561+0000","last_became_peered":"2023-10-16T11:19:36.684561+0000","last_unstale":"2023-10-16T11:19:37.994149+0000","last_undegraded":"2023-10-16T11:19:37.994149+0000","last_fullsized":"2023-10-16T11:19:37.994149+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:02.904122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666877+0000","last_change":"2023-10-16T11:19:23.330180+0000","last_active":"2023-10-16T11:19:36.666877+0000","last_peered":"2023-10-16T11:19:36.666877+0000","last_clean":"2023-10-16T11:19:36.666877+0000","last_became_active":"2023-10-16T11:19:23.329598+0000","last_became_peered":"2023-10-16T11:19:23.329598+0000","last_unstale":"2023-10-16T11:19:36.666877+0000","last_undegraded":"2023-10-16T11:19:36.666877+0000","last_fullsized":"2023-10-16T11:19:36.666877+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:59.613377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665085+0000","last_change":"2023-10-16T11:19:36.687617+0000","last_active":"2023-10-16T11:19:37.665085+0000","last_peered":"2023-10-16T11:19:37.665085+0000","last_clean":"2023-10-16T11:19:37.665085+0000","last_became_active":"2023-10-16T11:19:36.687333+0000","last_became_peered":"2023-10-16T11:19:36.687333+0000","last_unstale":"2023-10-16T11:19:37.665085+0000","last_undegraded":"2023-10-16T11:19:37.665085+0000","last_fullsized":"2023-10-16T11:19:37.665085+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:52.825820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.674910+0000","last_change":"2023-10-16T11:19:23.335413+0000","last_active":"2023-10-16T11:19:36.674910+0000","last_peered":"2023-10-16T11:19:36.674910+0000","last_clean":"2023-10-16T11:19:36.674910+0000","last_became_active":"2023-10-16T11:19:23.334994+0000","last_became_peered":"2023-10-16T11:19:23.334994+0000","last_unstale":"2023-10-16T11:19:36.674910+0000","last_undegraded":"2023-10-16T11:19:36.674910+0000","last_fullsized":"2023-10-16T11:19:36.674910+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:53.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663876+0000","last_change":"2023-10-16T11:19:36.676983+0000","last_active":"2023-10-16T11:19:37.663876+0000","last_peered":"2023-10-16T11:19:37.663876+0000","last_clean":"2023-10-16T11:19:37.663876+0000","last_became_active":"2023-10-16T11:19:36.676668+0000","last_became_peered":"2023-10-16T11:19:36.676668+0000","last_unstale":"2023-10-16T11:19:37.663876+0000","last_undegraded":"2023-10-16T11:19:37.663876+0000","last_fullsized":"2023-10-16T11:19:37.663876+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:25.390302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667223+0000","last_change":"2023-10-16T11:19:23.333705+0000","last_active":"2023-10-16T11:19:36.667223+0000","last_peered":"2023-10-16T11:19:36.667223+0000","last_clean":"2023-10-16T11:19:36.667223+0000","last_became_active":"2023-10-16T11:19:23.333009+0000","last_became_peered":"2023-10-16T11:19:23.333009+0000","last_unstale":"2023-10-16T11:19:36.667223+0000","last_undegraded":"2023-10-16T11:19:36.667223+0000","last_fullsized":"2023-10-16T11:19:36.667223+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:41.811226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697196+0000","last_change":"2023-10-16T11:19:36.695209+0000","last_active":"2023-10-16T11:19:37.697196+0000","last_peered":"2023-10-16T11:19:37.697196+0000","last_clean":"2023-10-16T11:19:37.697196+0000","last_became_active":"2023-10-16T11:19:36.694904+0000","last_became_peered":"2023-10-16T11:19:36.694904+0000","last_unstale":"2023-10-16T11:19:37.697196+0000","last_undegraded":"2023-10-16T11:19:37.697196+0000","last_fullsized":"2023-10-16T11:19:37.697196+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:47:45.793962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689827+0000","last_change":"2023-10-16T11:19:23.333382+0000","last_active":"2023-10-16T11:19:36.689827+0000","last_peered":"2023-10-16T11:19:36.689827+0000","last_clean":"2023-10-16T11:19:36.689827+0000","last_became_active":"2023-10-16T11:19:23.332643+0000","last_became_peered":"2023-10-16T11:19:23.332643+0000","last_unstale":"2023-10-16T11:19:36.689827+0000","last_undegraded":"2023-10-16T11:19:36.689827+0000","last_fullsized":"2023-10-16T11:19:36.689827+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:54.101849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664649+0000","last_change":"2023-10-16T11:19:36.683067+0000","last_active":"2023-10-16T11:19:37.664649+0000","last_peered":"2023-10-16T11:19:37.664649+0000","last_clean":"2023-10-16T11:19:37.664649+0000","last_became_active":"2023-10-16T11:19:36.682800+0000","last_became_peered":"2023-10-16T11:19:36.682800+0000","last_unstale":"2023-10-16T11:19:37.664649+0000","last_undegraded":"2023-10-16T11:19:37.664649+0000","last_fullsized":"2023-10-16T11:19:37.664649+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:50.857617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662651+0000","last_change":"2023-10-16T11:19:23.336597+0000","last_active":"2023-10-16T11:19:36.662651+0000","last_peered":"2023-10-16T11:19:36.662651+0000","last_clean":"2023-10-16T11:19:36.662651+0000","last_became_active":"2023-10-16T11:19:23.336307+0000","last_became_peered":"2023-10-16T11:19:23.336307+0000","last_unstale":"2023-10-16T11:19:36.662651+0000","last_undegraded":"2023-10-16T11:19:36.662651+0000","last_fullsized":"2023-10-16T11:19:36.662651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994870+0000","last_change":"2023-10-16T11:19:36.678840+0000","last_active":"2023-10-16T11:19:37.994870+0000","last_peered":"2023-10-16T11:19:37.994870+0000","last_clean":"2023-10-16T11:19:37.994870+0000","last_became_active":"2023-10-16T11:19:36.678438+0000","last_became_peered":"2023-10-16T11:19:36.678438+0000","last_unstale":"2023-10-16T11:19:37.994870+0000","last_undegraded":"2023-10-16T11:19:37.994870+0000","last_fullsized":"2023-10-16T11:19:37.994870+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:23.839599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678645+0000","last_change":"2023-10-16T11:19:23.329954+0000","last_active":"2023-10-16T11:19:36.678645+0000","last_peered":"2023-10-16T11:19:36.678645+0000","last_clean":"2023-10-16T11:19:36.678645+0000","last_became_active":"2023-10-16T11:19:23.329688+0000","last_became_peered":"2023-10-16T11:19:23.329688+0000","last_unstale":"2023-10-16T11:19:36.678645+0000","last_undegraded":"2023-10-16T11:19:36.678645+0000","last_fullsized":"2023-10-16T11:19:36.678645+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:11.183201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.699795+0000","last_change":"2023-10-16T11:19:36.676048+0000","last_active":"2023-10-16T11:19:37.699795+0000","last_peered":"2023-10-16T11:19:37.699795+0000","last_clean":"2023-10-16T11:19:37.699795+0000","last_became_active":"2023-10-16T11:19:36.675754+0000","last_became_peered":"2023-10-16T11:19:36.675754+0000","last_unstale":"2023-10-16T11:19:37.699795+0000","last_undegraded":"2023-10-16T11:19:37.699795+0000","last_fullsized":"2023-10-16T11:19:37.699795+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:58.387878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672717+0000","last_change":"2023-10-16T11:19:23.329203+0000","last_active":"2023-10-16T11:19:36.672717+0000","last_peered":"2023-10-16T11:19:36.672717+0000","last_clean":"2023-10-16T11:19:36.672717+0000","last_became_active":"2023-10-16T11:19:23.328748+0000","last_became_peered":"2023-10-16T11:19:23.328748+0000","last_unstale":"2023-10-16T11:19:36.672717+0000","last_undegraded":"2023-10-16T11:19:36.672717+0000","last_fullsized":"2023-10-16T11:19:36.672717+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:09.942459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994568+0000","last_change":"2023-10-16T11:19:36.684516+0000","last_active":"2023-10-16T11:19:37.994568+0000","last_peered":"2023-10-16T11:19:37.994568+0000","last_clean":"2023-10-16T11:19:37.994568+0000","last_became_active":"2023-10-16T11:19:36.684256+0000","last_became_peered":"2023-10-16T11:19:36.684256+0000","last_unstale":"2023-10-16T11:19:37.994568+0000","last_undegraded":"2023-10-16T11:19:37.994568+0000","last_fullsized":"2023-10-16T11:19:37.994568+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:12.957525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666433+0000","last_change":"2023-10-16T11:19:23.326663+0000","last_active":"2023-10-16T11:19:36.666433+0000","last_peered":"2023-10-16T11:19:36.666433+0000","last_clean":"2023-10-16T11:19:36.666433+0000","last_became_active":"2023-10-16T11:19:23.326378+0000","last_became_peered":"2023-10-16T11:19:23.326378+0000","last_unstale":"2023-10-16T11:19:36.666433+0000","last_undegraded":"2023-10-16T11:19:36.666433+0000","last_fullsized":"2023-10-16T11:19:36.666433+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:41.823015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663857+0000","last_change":"2023-10-16T11:19:36.684637+0000","last_active":"2023-10-16T11:19:37.663857+0000","last_peered":"2023-10-16T11:19:37.663857+0000","last_clean":"2023-10-16T11:19:37.663857+0000","last_became_active":"2023-10-16T11:19:36.684358+0000","last_became_peered":"2023-10-16T11:19:36.684358+0000","last_unstale":"2023-10-16T11:19:37.663857+0000","last_undegraded":"2023-10-16T11:19:37.663857+0000","last_fullsized":"2023-10-16T11:19:37.663857+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:29.950550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.676015+0000","last_change":"2023-10-16T11:19:23.330646+0000","last_active":"2023-10-16T11:19:36.676015+0000","last_peered":"2023-10-16T11:19:36.676015+0000","last_clean":"2023-10-16T11:19:36.676015+0000","last_became_active":"2023-10-16T11:19:23.330371+0000","last_became_peered":"2023-10-16T11:19:23.330371+0000","last_unstale":"2023-10-16T11:19:36.676015+0000","last_undegraded":"2023-10-16T11:19:36.676015+0000","last_fullsized":"2023-10-16T11:19:36.676015+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:55.845569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995300+0000","last_change":"2023-10-16T11:19:36.670851+0000","last_active":"2023-10-16T11:19:37.995300+0000","last_peered":"2023-10-16T11:19:37.995300+0000","last_clean":"2023-10-16T11:19:37.995300+0000","last_became_active":"2023-10-16T11:19:36.670512+0000","last_became_peered":"2023-10-16T11:19:36.670512+0000","last_unstale":"2023-10-16T11:19:37.995300+0000","last_undegraded":"2023-10-16T11:19:37.995300+0000","last_fullsized":"2023-10-16T11:19:37.995300+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:54.831776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666462+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:36.666462+0000","last_peered":"2023-10-16T11:19:36.666462+0000","last_clean":"2023-10-16T11:19:36.666462+0000","last_became_active":"2023-10-16T11:19:23.335513+0000","last_became_peered":"2023-10-16T11:19:23.335513+0000","last_unstale":"2023-10-16T11:19:36.666462+0000","last_undegraded":"2023-10-16T11:19:36.666462+0000","last_fullsized":"2023-10-16T11:19:36.666462+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:30.545707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700233+0000","last_change":"2023-10-16T11:19:36.686689+0000","last_active":"2023-10-16T11:19:37.700233+0000","last_peered":"2023-10-16T11:19:37.700233+0000","last_clean":"2023-10-16T11:19:37.700233+0000","last_became_active":"2023-10-16T11:19:36.686395+0000","last_became_peered":"2023-10-16T11:19:36.686395+0000","last_unstale":"2023-10-16T11:19:37.700233+0000","last_undegraded":"2023-10-16T11:19:37.700233+0000","last_fullsized":"2023-10-16T11:19:37.700233+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:46.501076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667038+0000","last_change":"2023-10-16T11:19:23.333266+0000","last_active":"2023-10-16T11:19:36.667038+0000","last_peered":"2023-10-16T11:19:36.667038+0000","last_clean":"2023-10-16T11:19:36.667038+0000","last_became_active":"2023-10-16T11:19:23.332412+0000","last_became_peered":"2023-10-16T11:19:23.332412+0000","last_unstale":"2023-10-16T11:19:36.667038+0000","last_undegraded":"2023-10-16T11:19:36.667038+0000","last_fullsized":"2023-10-16T11:19:36.667038+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:19.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664344+0000","last_change":"2023-10-16T11:19:36.687300+0000","last_active":"2023-10-16T11:19:37.664344+0000","last_peered":"2023-10-16T11:19:37.664344+0000","last_clean":"2023-10-16T11:19:37.664344+0000","last_became_active":"2023-10-16T11:19:36.687041+0000","last_became_peered":"2023-10-16T11:19:36.687041+0000","last_unstale":"2023-10-16T11:19:37.664344+0000","last_undegraded":"2023-10-16T11:19:37.664344+0000","last_fullsized":"2023-10-16T11:19:37.664344+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:12.863750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.681800+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:36.681800+0000","last_peered":"2023-10-16T11:19:36.681800+0000","last_clean":"2023-10-16T11:19:36.681800+0000","last_became_active":"2023-10-16T11:19:23.331671+0000","last_became_peered":"2023-10-16T11:19:23.331671+0000","last_unstale":"2023-10-16T11:19:36.681800+0000","last_undegraded":"2023-10-16T11:19:36.681800+0000","last_fullsized":"2023-10-16T11:19:36.681800+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:13.976802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664234+0000","last_change":"2023-10-16T11:19:36.682589+0000","last_active":"2023-10-16T11:19:37.664234+0000","last_peered":"2023-10-16T11:19:37.664234+0000","last_clean":"2023-10-16T11:19:37.664234+0000","last_became_active":"2023-10-16T11:19:36.682311+0000","last_became_peered":"2023-10-16T11:19:36.682311+0000","last_unstale":"2023-10-16T11:19:37.664234+0000","last_undegraded":"2023-10-16T11:19:37.664234+0000","last_fullsized":"2023-10-16T11:19:37.664234+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:52.728822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666733+0000","last_change":"2023-10-16T11:19:23.334441+0000","last_active":"2023-10-16T11:19:36.666733+0000","last_peered":"2023-10-16T11:19:36.666733+0000","last_clean":"2023-10-16T11:19:36.666733+0000","last_became_active":"2023-10-16T11:19:23.334140+0000","last_became_peered":"2023-10-16T11:19:23.334140+0000","last_unstale":"2023-10-16T11:19:36.666733+0000","last_undegraded":"2023-10-16T11:19:36.666733+0000","last_fullsized":"2023-10-16T11:19:36.666733+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3395584,"data_stored":2085172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.67200000000000004,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.372},"max":{"1min":9.4359999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.68400000000000005,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":9.3059999999999992,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.71599999999999997,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":9.3610000000000007,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.119,"5min":0.71999999999999997,"15min":0.68100000000000005},"min":{"1min":0.49399999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.3960000000000008,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.42299999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.72099999999999997,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":9.2889999999999997,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.753,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.57499999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.36099999999999999},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.157,"5min":0.76500000000000001,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":9.4689999999999994,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.61199999999999999,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.375},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.63100000000000001,"15min":0.56699999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.5399999999999991,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.503}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.64000000000000001,"15min":0.57399999999999995},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.69399999999999995,"15min":0.63100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.5690000000000008,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.71199999999999997,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.623},{"interface":"front","average":{"1min":1.155,"5min":0.69799999999999995,"15min":0.64500000000000002},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.44400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575992,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38096,"kb_used_data":2752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333744,"statfs":{"total":96636764160,"available":96597753856,"internally_reserved":0,"allocated":2818048,"data_stored":1496382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.91000000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.60499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.501},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.439}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.70599999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64000000000000001,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.34799999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.335,"5min":3.335,"15min":3.335},"last":0.374}]},{"osd":7,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.39800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575993,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47788,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324052,"statfs":{"total":96636764160,"available":96587829248,"internally_reserved":0,"allocated":2777088,"data_stored":1491203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.622,"15min":0.66700000000000004},"last":0.37},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.436,"15min":0.45200000000000001},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.52500000000000002,"5min":0.63900000000000001,"15min":0.71699999999999997},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.64000000000000001,"15min":0.70099999999999996},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.442},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":0.623,"15min":0.70499999999999996},"last":0.439}]},{"osd":4,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.48099999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65400000000000003,"15min":0.67200000000000004},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":0.89500000000000002,"15min":1.0149999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90000000000000002,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.90300000000000002,"15min":1.0620000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.016},"last":0.58499999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575993,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42796,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329044,"statfs":{"total":96636764160,"available":96592941056,"internally_reserved":0,"allocated":3375104,"data_stored":2081661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.28100000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44400000000000001,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60699999999999998,"5min":1.272,"15min":1.272},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.32200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.249},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67700000000000005,"5min":1.351,"15min":1.351},"last":0.36399999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.46800000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.875,"5min":1.391,"15min":1.391},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80000000000000004,"5min":0.874,"15min":1.089},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.95599999999999996,"15min":1.216},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.69799999999999995},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90900000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.47199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89100000000000001,"5min":1.038,"15min":1.1180000000000001},"last":0.70699999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.68700000000000006}]}]},{"osd":2,"up_from":13,"seq":55834575992,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2789376,"data_stored":1491909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.63300000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69799999999999995,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77800000000000002},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.95399999999999996,"15min":1.097},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67200000000000004,"15min":0.68200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.91800000000000004,"5min":0.96199999999999997,"15min":1.016},"last":0.77100000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.752},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90500000000000003,"5min":1.022,"15min":1.0389999999999999},"last":0.81899999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91800000000000004,"5min":1.022,"15min":1.022},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91900000000000004,"5min":0.98999999999999999,"15min":1.004},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90400000000000003,"5min":1.0569999999999999,"15min":1.071},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.80200000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575992,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51956,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319884,"statfs":{"total":96636764160,"available":96583561216,"internally_reserved":0,"allocated":2793472,"data_stored":1491351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38100000000000001,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.68700000000000006},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":1.0029999999999999},"last":0.34499999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.60899999999999999,"15min":0.67300000000000004},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.74299999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.58999999999999997,"5min":0.66800000000000004,"15min":0.96299999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64200000000000002,"5min":0.69899999999999995,"15min":1.024},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90600000000000003,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"last":0.623}]},{"osd":5,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.99099999999999999,"15min":1.026},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0640000000000001},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41799999999999998},"max":{"1min":0.85999999999999999,"5min":0.92900000000000005,"15min":1.0369999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.0620000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":1.002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88400000000000001,"5min":1.006,"15min":1.1899999999999999},"last":0.59599999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47792,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324048,"statfs":{"total":96636764160,"available":96587825152,"internally_reserved":0,"allocated":2785280,"data_stored":1491875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.502},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.625},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.623},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71599999999999997,"5min":0.82499999999999996,"15min":0.90700000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58799999999999997,"15min":0.623},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.65300000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60599999999999998,"15min":0.63900000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.84399999999999997,"15min":0.88400000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.624},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.83799999999999997,"15min":0.995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.51500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.71199999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.90500000000000003,"15min":1.115},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.35499999999999998,"15min":0.34200000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.83899999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.61599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82492,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94289348,"statfs":{"total":96636764160,"available":96552292352,"internally_reserved":0,"allocated":2801664,"data_stored":1492941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.88600000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60299999999999998},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.86499999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.91400000000000003},"last":0.499}]},{"osd":3,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40899999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85899999999999999,"5min":0.878,"15min":0.878},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55300000000000005,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:19:45.407 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:19:45.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.541+0000 7fc105e6e700 1 Processor -- start 2023-10-16T11:19:45.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.541+0000 7fc105e6e700 1 -- start start 2023-10-16T11:19:45.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc105e6e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc1000cf4c0 0x7fc100165140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc105e6e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10015a990 con 0x7fc10006bfb0 2023-10-16T11:19:45.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc105e6e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10015ac00 con 0x7fc1000cc0c0 2023-10-16T11:19:45.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc105e6e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10015aee0 con 0x7fc1000cf4c0 2023-10-16T11:19:45.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0fffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc1000cf4c0 0x7fc100165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0f7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc10006bfb0 0x7fc1000cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38678/0 (socket says 172.21.15.158:38678) 2023-10-16T11:19:45.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0fffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc1000cf4c0 0x7fc100165140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38506/0 (socket says 172.21.15.158:38506) 2023-10-16T11:19:45.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0f7fff700 1 -- 172.21.15.158:0/1288070249 learned_addr learned my addr 172.21.15.158:0/1288070249 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:45.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0fffff700 1 -- 172.21.15.158:0/1288070249 >> v1:172.21.15.172:6789/0 conn(0x7fc1000cc0c0 legacy=0x7fc1000aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:45.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0fffff700 1 -- 172.21.15.158:0/1288070249 >> v1:172.21.15.158:6789/0 conn(0x7fc10006bfb0 legacy=0x7fc1000cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0feffd700 1 -- 172.21.15.158:0/1288070249 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567047823 0 0) 0x7fc10015a990 con 0x7fc10006bfb0 2023-10-16T11:19:45.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0fffff700 1 -- 172.21.15.158:0/1288070249 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10015b1c0 con 0x7fc1000cf4c0 2023-10-16T11:19:45.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0fffff700 1 --2- 172.21.15.158:0/1288070249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cf4c0 0x7fc100165140 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009d90 tx=0x7fc0f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ae90f4baa4a13b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:45.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0feffd700 1 -- 172.21.15.158:0/1288070249 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f800e070 con 0x7fc1000cf4c0 2023-10-16T11:19:45.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0feffd700 1 -- 172.21.15.158:0/1288070249 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f800bb90 con 0x7fc1000cf4c0 2023-10-16T11:19:45.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.542+0000 7fc0feffd700 1 -- 172.21.15.158:0/1288070249 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8003a20 con 0x7fc1000cf4c0 2023-10-16T11:19:45.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.543+0000 7fc105e6e700 1 -- 172.21.15.158:0/1288070249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cf4c0 msgr2=0x7fc100165140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.543+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1288070249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cf4c0 0x7fc100165140 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009d90 tx=0x7fc0f800b330 comp rx=0 tx=0).stop 2023-10-16T11:19:45.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.543+0000 7fc105e6e700 1 -- 172.21.15.158:0/1288070249 shutdown_connections 2023-10-16T11:19:45.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.543+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1288070249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cf4c0 0x7fc100165140 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.543+0000 7fc105e6e700 1 -- 172.21.15.158:0/1288070249 >> 172.21.15.158:0/1288070249 conn(0x7fc1000cc4b0 msgr2=0x7fc1000cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:45.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.543+0000 7fc105e6e700 1 -- 172.21.15.158:0/1288070249 shutdown_connections 2023-10-16T11:19:45.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.543+0000 7fc105e6e700 1 -- 172.21.15.158:0/1288070249 wait complete. 2023-10-16T11:19:45.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.544+0000 7fc105e6e700 1 Processor -- start 2023-10-16T11:19:45.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.544+0000 7fc105e6e700 1 -- start start 2023-10-16T11:19:45.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc105e6e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc10006bfb0 0x7fc100149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc105e6e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1000cf4c0 0x7fc10006d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc105e6e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cda40 0x7fc100159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc105e6e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc100149ac0 con 0x7fc1000cda40 2023-10-16T11:19:45.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc105e6e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1000b2730 con 0x7fc1000cf4c0 2023-10-16T11:19:45.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc105e6e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10006db00 con 0x7fc10006bfb0 2023-10-16T11:19:45.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc0fffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cda40 0x7fc100159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc0fffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cda40 0x7fc100159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38512/0 (socket says 172.21.15.158:38512) 2023-10-16T11:19:45.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc0fffff700 1 -- 172.21.15.158:0/1610347212 learned_addr learned my addr 172.21.15.158:0/1610347212 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:45.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc0f7fff700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1000cf4c0 0x7fc10006d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.545+0000 7fc0ff7fe700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc10006bfb0 0x7fc100149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fffff700 1 -- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc10006bfb0 msgr2=0x7fc100149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fffff700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc10006bfb0 0x7fc100149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fffff700 1 -- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1000cf4c0 msgr2=0x7fc10006d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fffff700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1000cf4c0 0x7fc10006d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fffff700 1 -- 172.21.15.158:0/1610347212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f8009a70 con 0x7fc1000cda40 2023-10-16T11:19:45.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0f7fff700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1000cf4c0 0x7fc10006d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:45.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fffff700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cda40 0x7fc100159260 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009470 tx=0x7fc0f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:45.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fcff9700 1 -- 172.21.15.158:0/1610347212 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8012630 con 0x7fc1000cda40 2023-10-16T11:19:45.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fcff9700 1 -- 172.21.15.158:0/1610347212 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f8012790 con 0x7fc1000cda40 2023-10-16T11:19:45.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc0fcff9700 1 -- 172.21.15.158:0/1610347212 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8012b20 con 0x7fc1000cda40 2023-10-16T11:19:45.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10006dde0 con 0x7fc1000cda40 2023-10-16T11:19:45.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.546+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10006e3a0 con 0x7fc1000cda40 2023-10-16T11:19:45.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.547+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1000681b0 con 0x7fc1000cda40 2023-10-16T11:19:45.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.547+0000 7fc0fcff9700 1 -- 172.21.15.158:0/1610347212 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc0f8012c80 con 0x7fc1000cda40 2023-10-16T11:19:45.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.548+0000 7fc0fcff9700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc0e006b050 0x7fc0e006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.548+0000 7fc0fcff9700 1 -- 172.21.15.158:0/1610347212 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fc0f808c070 con 0x7fc1000cda40 2023-10-16T11:19:45.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.548+0000 7fc0ff7fe700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc0e006b050 0x7fc0e006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.548+0000 7fc0ff7fe700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc0e006b050 0x7fc0e006d510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0004f30 tx=0x7fc0f0009040 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:45.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.552+0000 7fc0fcff9700 1 -- 172.21.15.158:0/1610347212 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0f8056910 con 0x7fc1000cda40 2023-10-16T11:19:45.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.740+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc10015b780 con 0x7fc0e006b050 2023-10-16T11:19:45.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.754+0000 7fc0fcff9700 1 -- 172.21.15.158:0/1610347212 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528689 (secure 0 0 0) 0x7fc0f0015430 con 0x7fc0e006b050 2023-10-16T11:19:45.762 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:45.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc0e006b050 msgr2=0x7fc0e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc0e006b050 0x7fc0e006d510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0004f30 tx=0x7fc0f0009040 comp rx=0 tx=0).stop 2023-10-16T11:19:45.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cda40 msgr2=0x7fc100159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cda40 0x7fc100159260 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009470 tx=0x7fc0f8004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 shutdown_connections 2023-10-16T11:19:45.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fc0e006b050 0x7fc0e006d510 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1000cda40 0x7fc100159260 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc10006bfb0 0x7fc100149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 --2- 172.21.15.158:0/1610347212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1000cf4c0 0x7fc10006d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 >> 172.21.15.158:0/1610347212 conn(0x7fc1000cc4b0 msgr2=0x7fc1000c5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:45.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 shutdown_connections 2023-10-16T11:19:45.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.758+0000 7fc105e6e700 1 -- 172.21.15.158:0/1610347212 wait complete. 2023-10-16T11:19:45.770 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:19:45.772 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-16T11:19:45.277920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":411268,"kb_used_data":22984,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754563452,"statfs":{"total":773094113280,"available":772672974848,"internally_reserved":0,"allocated":23535616,"data_stored":13122494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":397463095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.617530"},"pg_stats":[{"pgid":"1.0","version":"207'348","reported_seq":1212,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666350+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:19:36.666350+0000","last_peered":"2023-10-16T11:19:36.666350+0000","last_clean":"2023-10-16T11:19:36.666350+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:19:36.666350+0000","last_undegraded":"2023-10-16T11:19:36.666350+0000","last_fullsized":"2023-10-16T11:19:36.666350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663250+0000","last_change":"2023-10-16T11:19:22.332936+0000","last_active":"2023-10-16T11:19:36.663250+0000","last_peered":"2023-10-16T11:19:36.663250+0000","last_clean":"2023-10-16T11:19:36.663250+0000","last_became_active":"2023-10-16T11:19:22.332598+0000","last_became_peered":"2023-10-16T11:19:22.332598+0000","last_unstale":"2023-10-16T11:19:36.663250+0000","last_undegraded":"2023-10-16T11:19:36.663250+0000","last_fullsized":"2023-10-16T11:19:36.663250+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:34.597859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666646+0000","last_change":"2023-10-16T11:19:23.334041+0000","last_active":"2023-10-16T11:19:36.666646+0000","last_peered":"2023-10-16T11:19:36.666646+0000","last_clean":"2023-10-16T11:19:36.666646+0000","last_became_active":"2023-10-16T11:19:23.333752+0000","last_became_peered":"2023-10-16T11:19:23.333752+0000","last_unstale":"2023-10-16T11:19:36.666646+0000","last_undegraded":"2023-10-16T11:19:36.666646+0000","last_fullsized":"2023-10-16T11:19:36.666646+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.685292+0000","last_change":"2023-10-16T11:19:35.677068+0000","last_active":"2023-10-16T11:19:36.685292+0000","last_peered":"2023-10-16T11:19:36.685292+0000","last_clean":"2023-10-16T11:19:36.685292+0000","last_became_active":"2023-10-16T11:19:35.676797+0000","last_became_peered":"2023-10-16T11:19:35.676797+0000","last_unstale":"2023-10-16T11:19:36.685292+0000","last_undegraded":"2023-10-16T11:19:36.685292+0000","last_fullsized":"2023-10-16T11:19:36.685292+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:43.621552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664039+0000","last_change":"2023-10-16T11:19:36.684566+0000","last_active":"2023-10-16T11:19:37.664039+0000","last_peered":"2023-10-16T11:19:37.664039+0000","last_clean":"2023-10-16T11:19:37.664039+0000","last_became_active":"2023-10-16T11:19:36.684276+0000","last_became_peered":"2023-10-16T11:19:36.684276+0000","last_unstale":"2023-10-16T11:19:37.664039+0000","last_undegraded":"2023-10-16T11:19:37.664039+0000","last_fullsized":"2023-10-16T11:19:37.664039+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:37.800770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672051+0000","last_change":"2023-10-16T11:19:22.334152+0000","last_active":"2023-10-16T11:19:36.672051+0000","last_peered":"2023-10-16T11:19:36.672051+0000","last_clean":"2023-10-16T11:19:36.672051+0000","last_became_active":"2023-10-16T11:19:22.333859+0000","last_became_peered":"2023-10-16T11:19:22.333859+0000","last_unstale":"2023-10-16T11:19:36.672051+0000","last_undegraded":"2023-10-16T11:19:36.672051+0000","last_fullsized":"2023-10-16T11:19:36.672051+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:50.619981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668272+0000","last_change":"2023-10-16T11:19:23.324999+0000","last_active":"2023-10-16T11:19:36.668272+0000","last_peered":"2023-10-16T11:19:36.668272+0000","last_clean":"2023-10-16T11:19:36.668272+0000","last_became_active":"2023-10-16T11:19:23.324686+0000","last_became_peered":"2023-10-16T11:19:23.324686+0000","last_unstale":"2023-10-16T11:19:36.668272+0000","last_undegraded":"2023-10-16T11:19:36.668272+0000","last_fullsized":"2023-10-16T11:19:36.668272+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:42.281964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"221'1","reported_seq":13,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994502+0000","last_change":"2023-10-16T11:19:35.674436+0000","last_active":"2023-10-16T11:19:37.994502+0000","last_peered":"2023-10-16T11:19:37.994502+0000","last_clean":"2023-10-16T11:19:37.994502+0000","last_became_active":"2023-10-16T11:19:35.674216+0000","last_became_peered":"2023-10-16T11:19:35.674216+0000","last_unstale":"2023-10-16T11:19:37.994502+0000","last_undegraded":"2023-10-16T11:19:37.994502+0000","last_fullsized":"2023-10-16T11:19:37.994502+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:05.514151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700186+0000","last_change":"2023-10-16T11:19:36.683592+0000","last_active":"2023-10-16T11:19:37.700186+0000","last_peered":"2023-10-16T11:19:37.700186+0000","last_clean":"2023-10-16T11:19:37.700186+0000","last_became_active":"2023-10-16T11:19:36.682898+0000","last_became_peered":"2023-10-16T11:19:36.682898+0000","last_unstale":"2023-10-16T11:19:37.700186+0000","last_undegraded":"2023-10-16T11:19:37.700186+0000","last_fullsized":"2023-10-16T11:19:37.700186+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:35.693733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663874+0000","last_change":"2023-10-16T11:19:22.332640+0000","last_active":"2023-10-16T11:19:36.663874+0000","last_peered":"2023-10-16T11:19:36.663874+0000","last_clean":"2023-10-16T11:19:36.663874+0000","last_became_active":"2023-10-16T11:19:22.331608+0000","last_became_peered":"2023-10-16T11:19:22.331608+0000","last_unstale":"2023-10-16T11:19:36.663874+0000","last_undegraded":"2023-10-16T11:19:36.663874+0000","last_fullsized":"2023-10-16T11:19:36.663874+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670102+0000","last_change":"2023-10-16T11:19:23.329115+0000","last_active":"2023-10-16T11:19:36.670102+0000","last_peered":"2023-10-16T11:19:36.670102+0000","last_clean":"2023-10-16T11:19:36.670102+0000","last_became_active":"2023-10-16T11:19:23.328830+0000","last_became_peered":"2023-10-16T11:19:23.328830+0000","last_unstale":"2023-10-16T11:19:36.670102+0000","last_undegraded":"2023-10-16T11:19:36.670102+0000","last_fullsized":"2023-10-16T11:19:36.670102+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:17.247488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699488+0000","last_change":"2023-10-16T11:19:35.687029+0000","last_active":"2023-10-16T11:19:36.699488+0000","last_peered":"2023-10-16T11:19:36.699488+0000","last_clean":"2023-10-16T11:19:36.699488+0000","last_became_active":"2023-10-16T11:19:35.686699+0000","last_became_peered":"2023-10-16T11:19:35.686699+0000","last_unstale":"2023-10-16T11:19:36.699488+0000","last_undegraded":"2023-10-16T11:19:36.699488+0000","last_fullsized":"2023-10-16T11:19:36.699488+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:52.450005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995162+0000","last_change":"2023-10-16T11:19:36.677702+0000","last_active":"2023-10-16T11:19:37.995162+0000","last_peered":"2023-10-16T11:19:37.995162+0000","last_clean":"2023-10-16T11:19:37.995162+0000","last_became_active":"2023-10-16T11:19:36.677422+0000","last_became_peered":"2023-10-16T11:19:36.677422+0000","last_unstale":"2023-10-16T11:19:37.995162+0000","last_undegraded":"2023-10-16T11:19:37.995162+0000","last_fullsized":"2023-10-16T11:19:37.995162+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:43.889855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663341+0000","last_change":"2023-10-16T11:19:22.329623+0000","last_active":"2023-10-16T11:19:36.663341+0000","last_peered":"2023-10-16T11:19:36.663341+0000","last_clean":"2023-10-16T11:19:36.663341+0000","last_became_active":"2023-10-16T11:19:22.329344+0000","last_became_peered":"2023-10-16T11:19:22.329344+0000","last_unstale":"2023-10-16T11:19:36.663341+0000","last_undegraded":"2023-10-16T11:19:36.663341+0000","last_fullsized":"2023-10-16T11:19:36.663341+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670592+0000","last_change":"2023-10-16T11:19:23.330624+0000","last_active":"2023-10-16T11:19:36.670592+0000","last_peered":"2023-10-16T11:19:36.670592+0000","last_clean":"2023-10-16T11:19:36.670592+0000","last_became_active":"2023-10-16T11:19:23.329588+0000","last_became_peered":"2023-10-16T11:19:23.329588+0000","last_unstale":"2023-10-16T11:19:36.670592+0000","last_undegraded":"2023-10-16T11:19:36.670592+0000","last_fullsized":"2023-10-16T11:19:36.670592+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:10.835410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.691778+0000","last_change":"2023-10-16T11:19:35.685879+0000","last_active":"2023-10-16T11:19:36.691778+0000","last_peered":"2023-10-16T11:19:36.691778+0000","last_clean":"2023-10-16T11:19:36.691778+0000","last_became_active":"2023-10-16T11:19:35.685381+0000","last_became_peered":"2023-10-16T11:19:35.685381+0000","last_unstale":"2023-10-16T11:19:36.691778+0000","last_undegraded":"2023-10-16T11:19:36.691778+0000","last_fullsized":"2023-10-16T11:19:36.691778+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:24.651235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664009+0000","last_change":"2023-10-16T11:19:36.692166+0000","last_active":"2023-10-16T11:19:37.664009+0000","last_peered":"2023-10-16T11:19:37.664009+0000","last_clean":"2023-10-16T11:19:37.664009+0000","last_became_active":"2023-10-16T11:19:36.691802+0000","last_became_peered":"2023-10-16T11:19:36.691802+0000","last_unstale":"2023-10-16T11:19:37.664009+0000","last_undegraded":"2023-10-16T11:19:37.664009+0000","last_fullsized":"2023-10-16T11:19:37.664009+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:31.108155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.683890+0000","last_change":"2023-10-16T11:19:22.333752+0000","last_active":"2023-10-16T11:19:36.683890+0000","last_peered":"2023-10-16T11:19:36.683890+0000","last_clean":"2023-10-16T11:19:36.683890+0000","last_became_active":"2023-10-16T11:19:22.333497+0000","last_became_peered":"2023-10-16T11:19:22.333497+0000","last_unstale":"2023-10-16T11:19:36.683890+0000","last_undegraded":"2023-10-16T11:19:36.683890+0000","last_fullsized":"2023-10-16T11:19:36.683890+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684460+0000","last_change":"2023-10-16T11:19:23.329220+0000","last_active":"2023-10-16T11:19:36.684460+0000","last_peered":"2023-10-16T11:19:36.684460+0000","last_clean":"2023-10-16T11:19:36.684460+0000","last_became_active":"2023-10-16T11:19:23.328923+0000","last_became_peered":"2023-10-16T11:19:23.328923+0000","last_unstale":"2023-10-16T11:19:36.684460+0000","last_undegraded":"2023-10-16T11:19:36.684460+0000","last_fullsized":"2023-10-16T11:19:36.684460+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667680+0000","last_change":"2023-10-16T11:19:35.674427+0000","last_active":"2023-10-16T11:19:36.667680+0000","last_peered":"2023-10-16T11:19:36.667680+0000","last_clean":"2023-10-16T11:19:36.667680+0000","last_became_active":"2023-10-16T11:19:35.674155+0000","last_became_peered":"2023-10-16T11:19:35.674155+0000","last_unstale":"2023-10-16T11:19:36.667680+0000","last_undegraded":"2023-10-16T11:19:36.667680+0000","last_fullsized":"2023-10-16T11:19:36.667680+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:25.009353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995237+0000","last_change":"2023-10-16T11:19:36.687286+0000","last_active":"2023-10-16T11:19:37.995237+0000","last_peered":"2023-10-16T11:19:37.995237+0000","last_clean":"2023-10-16T11:19:37.995237+0000","last_became_active":"2023-10-16T11:19:36.686995+0000","last_became_peered":"2023-10-16T11:19:36.686995+0000","last_unstale":"2023-10-16T11:19:37.995237+0000","last_undegraded":"2023-10-16T11:19:37.995237+0000","last_fullsized":"2023-10-16T11:19:37.995237+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:10.501316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664830+0000","last_change":"2023-10-16T11:19:22.335923+0000","last_active":"2023-10-16T11:19:36.664830+0000","last_peered":"2023-10-16T11:19:36.664830+0000","last_clean":"2023-10-16T11:19:36.664830+0000","last_became_active":"2023-10-16T11:19:22.335705+0000","last_became_peered":"2023-10-16T11:19:22.335705+0000","last_unstale":"2023-10-16T11:19:36.664830+0000","last_undegraded":"2023-10-16T11:19:36.664830+0000","last_fullsized":"2023-10-16T11:19:36.664830+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:40.803363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665902+0000","last_change":"2023-10-16T11:19:23.336160+0000","last_active":"2023-10-16T11:19:36.665902+0000","last_peered":"2023-10-16T11:19:36.665902+0000","last_clean":"2023-10-16T11:19:36.665902+0000","last_became_active":"2023-10-16T11:19:23.333920+0000","last_became_peered":"2023-10-16T11:19:23.333920+0000","last_unstale":"2023-10-16T11:19:36.665902+0000","last_undegraded":"2023-10-16T11:19:36.665902+0000","last_fullsized":"2023-10-16T11:19:36.665902+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671203+0000","last_change":"2023-10-16T11:19:35.671256+0000","last_active":"2023-10-16T11:19:36.671203+0000","last_peered":"2023-10-16T11:19:36.671203+0000","last_clean":"2023-10-16T11:19:36.671203+0000","last_became_active":"2023-10-16T11:19:35.670977+0000","last_became_peered":"2023-10-16T11:19:35.670977+0000","last_unstale":"2023-10-16T11:19:36.671203+0000","last_undegraded":"2023-10-16T11:19:36.671203+0000","last_fullsized":"2023-10-16T11:19:36.671203+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:00.113603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663985+0000","last_change":"2023-10-16T11:19:36.680876+0000","last_active":"2023-10-16T11:19:37.663985+0000","last_peered":"2023-10-16T11:19:37.663985+0000","last_clean":"2023-10-16T11:19:37.663985+0000","last_became_active":"2023-10-16T11:19:36.680576+0000","last_became_peered":"2023-10-16T11:19:36.680576+0000","last_unstale":"2023-10-16T11:19:37.663985+0000","last_undegraded":"2023-10-16T11:19:37.663985+0000","last_fullsized":"2023-10-16T11:19:37.663985+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:52.686759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666970+0000","last_change":"2023-10-16T11:19:22.326754+0000","last_active":"2023-10-16T11:19:36.666970+0000","last_peered":"2023-10-16T11:19:36.666970+0000","last_clean":"2023-10-16T11:19:36.666970+0000","last_became_active":"2023-10-16T11:19:22.326480+0000","last_became_peered":"2023-10-16T11:19:22.326480+0000","last_unstale":"2023-10-16T11:19:36.666970+0000","last_undegraded":"2023-10-16T11:19:36.666970+0000","last_fullsized":"2023-10-16T11:19:36.666970+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663142+0000","last_change":"2023-10-16T11:19:23.331571+0000","last_active":"2023-10-16T11:19:36.663142+0000","last_peered":"2023-10-16T11:19:36.663142+0000","last_clean":"2023-10-16T11:19:36.663142+0000","last_became_active":"2023-10-16T11:19:23.331242+0000","last_became_peered":"2023-10-16T11:19:23.331242+0000","last_unstale":"2023-10-16T11:19:36.663142+0000","last_undegraded":"2023-10-16T11:19:36.663142+0000","last_fullsized":"2023-10-16T11:19:36.663142+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684254+0000","last_change":"2023-10-16T11:19:35.679089+0000","last_active":"2023-10-16T11:19:36.684254+0000","last_peered":"2023-10-16T11:19:36.684254+0000","last_clean":"2023-10-16T11:19:36.684254+0000","last_became_active":"2023-10-16T11:19:35.678420+0000","last_became_peered":"2023-10-16T11:19:35.678420+0000","last_unstale":"2023-10-16T11:19:36.684254+0000","last_undegraded":"2023-10-16T11:19:36.684254+0000","last_fullsized":"2023-10-16T11:19:36.684254+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:30.151493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664447+0000","last_change":"2023-10-16T11:19:36.681755+0000","last_active":"2023-10-16T11:19:37.664447+0000","last_peered":"2023-10-16T11:19:37.664447+0000","last_clean":"2023-10-16T11:19:37.664447+0000","last_became_active":"2023-10-16T11:19:36.681399+0000","last_became_peered":"2023-10-16T11:19:36.681399+0000","last_unstale":"2023-10-16T11:19:37.664447+0000","last_undegraded":"2023-10-16T11:19:37.664447+0000","last_fullsized":"2023-10-16T11:19:37.664447+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:56.393185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"217'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689930+0000","last_change":"2023-10-16T11:19:22.331182+0000","last_active":"2023-10-16T11:19:36.689930+0000","last_peered":"2023-10-16T11:19:36.689930+0000","last_clean":"2023-10-16T11:19:36.689930+0000","last_became_active":"2023-10-16T11:19:22.330860+0000","last_became_peered":"2023-10-16T11:19:22.330860+0000","last_unstale":"2023-10-16T11:19:36.689930+0000","last_undegraded":"2023-10-16T11:19:36.689930+0000","last_fullsized":"2023-10-16T11:19:36.689930+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:09.897505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667572+0000","last_change":"2023-10-16T11:19:23.325482+0000","last_active":"2023-10-16T11:19:36.667572+0000","last_peered":"2023-10-16T11:19:36.667572+0000","last_clean":"2023-10-16T11:19:36.667572+0000","last_became_active":"2023-10-16T11:19:23.325193+0000","last_became_peered":"2023-10-16T11:19:23.325193+0000","last_unstale":"2023-10-16T11:19:36.667572+0000","last_undegraded":"2023-10-16T11:19:36.667572+0000","last_fullsized":"2023-10-16T11:19:36.667572+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.191635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678478+0000","last_change":"2023-10-16T11:19:35.681024+0000","last_active":"2023-10-16T11:19:36.678478+0000","last_peered":"2023-10-16T11:19:36.678478+0000","last_clean":"2023-10-16T11:19:36.678478+0000","last_became_active":"2023-10-16T11:19:35.677329+0000","last_became_peered":"2023-10-16T11:19:35.677329+0000","last_unstale":"2023-10-16T11:19:36.678478+0000","last_undegraded":"2023-10-16T11:19:36.678478+0000","last_fullsized":"2023-10-16T11:19:36.678478+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:36.304256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696725+0000","last_change":"2023-10-16T11:19:36.694524+0000","last_active":"2023-10-16T11:19:37.696725+0000","last_peered":"2023-10-16T11:19:37.696725+0000","last_clean":"2023-10-16T11:19:37.696725+0000","last_became_active":"2023-10-16T11:19:36.694188+0000","last_became_peered":"2023-10-16T11:19:36.694188+0000","last_unstale":"2023-10-16T11:19:37.696725+0000","last_undegraded":"2023-10-16T11:19:37.696725+0000","last_fullsized":"2023-10-16T11:19:37.696725+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:30.588015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662948+0000","last_change":"2023-10-16T11:19:22.332950+0000","last_active":"2023-10-16T11:19:36.662948+0000","last_peered":"2023-10-16T11:19:36.662948+0000","last_clean":"2023-10-16T11:19:36.662948+0000","last_became_active":"2023-10-16T11:19:22.332650+0000","last_became_peered":"2023-10-16T11:19:22.332650+0000","last_unstale":"2023-10-16T11:19:36.662948+0000","last_undegraded":"2023-10-16T11:19:36.662948+0000","last_fullsized":"2023-10-16T11:19:36.662948+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:02:29.534204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.680253+0000","last_change":"2023-10-16T11:19:23.333192+0000","last_active":"2023-10-16T11:19:36.680253+0000","last_peered":"2023-10-16T11:19:36.680253+0000","last_clean":"2023-10-16T11:19:36.680253+0000","last_became_active":"2023-10-16T11:19:23.332739+0000","last_became_peered":"2023-10-16T11:19:23.332739+0000","last_unstale":"2023-10-16T11:19:36.680253+0000","last_undegraded":"2023-10-16T11:19:36.680253+0000","last_fullsized":"2023-10-16T11:19:36.680253+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666051+0000","last_change":"2023-10-16T11:19:35.672187+0000","last_active":"2023-10-16T11:19:36.666051+0000","last_peered":"2023-10-16T11:19:36.666051+0000","last_clean":"2023-10-16T11:19:36.666051+0000","last_became_active":"2023-10-16T11:19:35.671883+0000","last_became_peered":"2023-10-16T11:19:35.671883+0000","last_unstale":"2023-10-16T11:19:36.666051+0000","last_undegraded":"2023-10-16T11:19:36.666051+0000","last_fullsized":"2023-10-16T11:19:36.666051+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:27.228484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662842+0000","last_change":"2023-10-16T11:19:36.680498+0000","last_active":"2023-10-16T11:19:37.662842+0000","last_peered":"2023-10-16T11:19:37.662842+0000","last_clean":"2023-10-16T11:19:37.662842+0000","last_became_active":"2023-10-16T11:19:36.680087+0000","last_became_peered":"2023-10-16T11:19:36.680087+0000","last_unstale":"2023-10-16T11:19:37.662842+0000","last_undegraded":"2023-10-16T11:19:37.662842+0000","last_fullsized":"2023-10-16T11:19:37.662842+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:30.982911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666212+0000","last_change":"2023-10-16T11:19:22.326294+0000","last_active":"2023-10-16T11:19:36.666212+0000","last_peered":"2023-10-16T11:19:36.666212+0000","last_clean":"2023-10-16T11:19:36.666212+0000","last_became_active":"2023-10-16T11:19:22.326012+0000","last_became_peered":"2023-10-16T11:19:22.326012+0000","last_unstale":"2023-10-16T11:19:36.666212+0000","last_undegraded":"2023-10-16T11:19:36.666212+0000","last_fullsized":"2023-10-16T11:19:36.666212+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.692071+0000","last_change":"2023-10-16T11:19:23.336920+0000","last_active":"2023-10-16T11:19:36.692071+0000","last_peered":"2023-10-16T11:19:36.692071+0000","last_clean":"2023-10-16T11:19:36.692071+0000","last_became_active":"2023-10-16T11:19:23.336473+0000","last_became_peered":"2023-10-16T11:19:23.336473+0000","last_unstale":"2023-10-16T11:19:36.692071+0000","last_undegraded":"2023-10-16T11:19:36.692071+0000","last_fullsized":"2023-10-16T11:19:36.692071+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:22.421887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.685513+0000","last_change":"2023-10-16T11:19:35.677026+0000","last_active":"2023-10-16T11:19:36.685513+0000","last_peered":"2023-10-16T11:19:36.685513+0000","last_clean":"2023-10-16T11:19:36.685513+0000","last_became_active":"2023-10-16T11:19:35.676697+0000","last_became_peered":"2023-10-16T11:19:35.676697+0000","last_unstale":"2023-10-16T11:19:36.685513+0000","last_undegraded":"2023-10-16T11:19:36.685513+0000","last_fullsized":"2023-10-16T11:19:36.685513+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:49.538337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.699997+0000","last_change":"2023-10-16T11:19:36.675909+0000","last_active":"2023-10-16T11:19:37.699997+0000","last_peered":"2023-10-16T11:19:37.699997+0000","last_clean":"2023-10-16T11:19:37.699997+0000","last_became_active":"2023-10-16T11:19:36.674282+0000","last_became_peered":"2023-10-16T11:19:36.674282+0000","last_unstale":"2023-10-16T11:19:37.699997+0000","last_undegraded":"2023-10-16T11:19:37.699997+0000","last_fullsized":"2023-10-16T11:19:37.699997+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:49.823406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668923+0000","last_change":"2023-10-16T11:19:22.331457+0000","last_active":"2023-10-16T11:19:36.668923+0000","last_peered":"2023-10-16T11:19:36.668923+0000","last_clean":"2023-10-16T11:19:36.668923+0000","last_became_active":"2023-10-16T11:19:22.331010+0000","last_became_peered":"2023-10-16T11:19:22.331010+0000","last_unstale":"2023-10-16T11:19:36.668923+0000","last_undegraded":"2023-10-16T11:19:36.668923+0000","last_fullsized":"2023-10-16T11:19:36.668923+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:38.048633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663202+0000","last_change":"2023-10-16T11:19:23.333035+0000","last_active":"2023-10-16T11:19:36.663202+0000","last_peered":"2023-10-16T11:19:36.663202+0000","last_clean":"2023-10-16T11:19:36.663202+0000","last_became_active":"2023-10-16T11:19:23.332692+0000","last_became_peered":"2023-10-16T11:19:23.332692+0000","last_unstale":"2023-10-16T11:19:36.663202+0000","last_undegraded":"2023-10-16T11:19:36.663202+0000","last_fullsized":"2023-10-16T11:19:36.663202+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.694775+0000","last_change":"2023-10-16T11:19:35.684312+0000","last_active":"2023-10-16T11:19:36.694775+0000","last_peered":"2023-10-16T11:19:36.694775+0000","last_clean":"2023-10-16T11:19:36.694775+0000","last_became_active":"2023-10-16T11:19:35.684055+0000","last_became_peered":"2023-10-16T11:19:35.684055+0000","last_unstale":"2023-10-16T11:19:36.694775+0000","last_undegraded":"2023-10-16T11:19:36.694775+0000","last_fullsized":"2023-10-16T11:19:36.694775+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:48.058006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995020+0000","last_change":"2023-10-16T11:19:36.691309+0000","last_active":"2023-10-16T11:19:37.995020+0000","last_peered":"2023-10-16T11:19:37.995020+0000","last_clean":"2023-10-16T11:19:37.995020+0000","last_became_active":"2023-10-16T11:19:36.690993+0000","last_became_peered":"2023-10-16T11:19:36.690993+0000","last_unstale":"2023-10-16T11:19:37.995020+0000","last_undegraded":"2023-10-16T11:19:37.995020+0000","last_fullsized":"2023-10-16T11:19:37.995020+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:12.957525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675071+0000","last_change":"2023-10-16T11:19:22.329858+0000","last_active":"2023-10-16T11:19:36.675071+0000","last_peered":"2023-10-16T11:19:36.675071+0000","last_clean":"2023-10-16T11:19:36.675071+0000","last_became_active":"2023-10-16T11:19:22.329586+0000","last_became_peered":"2023-10-16T11:19:22.329586+0000","last_unstale":"2023-10-16T11:19:36.675071+0000","last_undegraded":"2023-10-16T11:19:36.675071+0000","last_fullsized":"2023-10-16T11:19:36.675071+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:25.906324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670506+0000","last_change":"2023-10-16T11:19:23.333742+0000","last_active":"2023-10-16T11:19:36.670506+0000","last_peered":"2023-10-16T11:19:36.670506+0000","last_clean":"2023-10-16T11:19:36.670506+0000","last_became_active":"2023-10-16T11:19:23.333424+0000","last_became_peered":"2023-10-16T11:19:23.333424+0000","last_unstale":"2023-10-16T11:19:36.670506+0000","last_undegraded":"2023-10-16T11:19:36.670506+0000","last_fullsized":"2023-10-16T11:19:36.670506+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.156569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684622+0000","last_change":"2023-10-16T11:19:35.685893+0000","last_active":"2023-10-16T11:19:36.684622+0000","last_peered":"2023-10-16T11:19:36.684622+0000","last_clean":"2023-10-16T11:19:36.684622+0000","last_became_active":"2023-10-16T11:19:35.685647+0000","last_became_peered":"2023-10-16T11:19:35.685647+0000","last_unstale":"2023-10-16T11:19:36.684622+0000","last_undegraded":"2023-10-16T11:19:36.684622+0000","last_fullsized":"2023-10-16T11:19:36.684622+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:55.676783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697595+0000","last_change":"2023-10-16T11:19:36.694904+0000","last_active":"2023-10-16T11:19:37.697595+0000","last_peered":"2023-10-16T11:19:37.697595+0000","last_clean":"2023-10-16T11:19:37.697595+0000","last_became_active":"2023-10-16T11:19:36.690439+0000","last_became_peered":"2023-10-16T11:19:36.690439+0000","last_unstale":"2023-10-16T11:19:37.697595+0000","last_undegraded":"2023-10-16T11:19:37.697595+0000","last_fullsized":"2023-10-16T11:19:37.697595+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:37.294828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.673044+0000","last_change":"2023-10-16T11:19:22.329655+0000","last_active":"2023-10-16T11:19:36.673044+0000","last_peered":"2023-10-16T11:19:36.673044+0000","last_clean":"2023-10-16T11:19:36.673044+0000","last_became_active":"2023-10-16T11:19:22.329366+0000","last_became_peered":"2023-10-16T11:19:22.329366+0000","last_unstale":"2023-10-16T11:19:36.673044+0000","last_undegraded":"2023-10-16T11:19:36.673044+0000","last_fullsized":"2023-10-16T11:19:36.673044+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:47.331154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671563+0000","last_change":"2023-10-16T11:19:23.331460+0000","last_active":"2023-10-16T11:19:36.671563+0000","last_peered":"2023-10-16T11:19:36.671563+0000","last_clean":"2023-10-16T11:19:36.671563+0000","last_became_active":"2023-10-16T11:19:23.331178+0000","last_became_peered":"2023-10-16T11:19:23.331178+0000","last_unstale":"2023-10-16T11:19:36.671563+0000","last_undegraded":"2023-10-16T11:19:36.671563+0000","last_fullsized":"2023-10-16T11:19:36.671563+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:36.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699143+0000","last_change":"2023-10-16T11:19:35.683185+0000","last_active":"2023-10-16T11:19:36.699143+0000","last_peered":"2023-10-16T11:19:36.699143+0000","last_clean":"2023-10-16T11:19:36.699143+0000","last_became_active":"2023-10-16T11:19:35.681186+0000","last_became_peered":"2023-10-16T11:19:35.681186+0000","last_unstale":"2023-10-16T11:19:36.699143+0000","last_undegraded":"2023-10-16T11:19:36.699143+0000","last_fullsized":"2023-10-16T11:19:36.699143+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:26.084551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664927+0000","last_change":"2023-10-16T11:19:36.678756+0000","last_active":"2023-10-16T11:19:37.664927+0000","last_peered":"2023-10-16T11:19:37.664927+0000","last_clean":"2023-10-16T11:19:37.664927+0000","last_became_active":"2023-10-16T11:19:36.678452+0000","last_became_peered":"2023-10-16T11:19:36.678452+0000","last_unstale":"2023-10-16T11:19:37.664927+0000","last_undegraded":"2023-10-16T11:19:37.664927+0000","last_fullsized":"2023-10-16T11:19:37.664927+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:55.430561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675077+0000","last_change":"2023-10-16T11:19:22.332355+0000","last_active":"2023-10-16T11:19:36.675077+0000","last_peered":"2023-10-16T11:19:36.675077+0000","last_clean":"2023-10-16T11:19:36.675077+0000","last_became_active":"2023-10-16T11:19:22.331520+0000","last_became_peered":"2023-10-16T11:19:22.331520+0000","last_unstale":"2023-10-16T11:19:36.675077+0000","last_undegraded":"2023-10-16T11:19:36.675077+0000","last_fullsized":"2023-10-16T11:19:36.675077+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:10.682623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684099+0000","last_change":"2023-10-16T11:19:23.331171+0000","last_active":"2023-10-16T11:19:36.684099+0000","last_peered":"2023-10-16T11:19:36.684099+0000","last_clean":"2023-10-16T11:19:36.684099+0000","last_became_active":"2023-10-16T11:19:23.330836+0000","last_became_peered":"2023-10-16T11:19:23.330836+0000","last_unstale":"2023-10-16T11:19:36.684099+0000","last_undegraded":"2023-10-16T11:19:36.684099+0000","last_fullsized":"2023-10-16T11:19:36.684099+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:45.818409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667260+0000","last_change":"2023-10-16T11:19:35.674937+0000","last_active":"2023-10-16T11:19:36.667260+0000","last_peered":"2023-10-16T11:19:36.667260+0000","last_clean":"2023-10-16T11:19:36.667260+0000","last_became_active":"2023-10-16T11:19:35.674670+0000","last_became_peered":"2023-10-16T11:19:35.674670+0000","last_unstale":"2023-10-16T11:19:36.667260+0000","last_undegraded":"2023-10-16T11:19:36.667260+0000","last_fullsized":"2023-10-16T11:19:36.667260+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:50.829956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700060+0000","last_change":"2023-10-16T11:19:36.682803+0000","last_active":"2023-10-16T11:19:37.700060+0000","last_peered":"2023-10-16T11:19:37.700060+0000","last_clean":"2023-10-16T11:19:37.700060+0000","last_became_active":"2023-10-16T11:19:36.682480+0000","last_became_peered":"2023-10-16T11:19:36.682480+0000","last_unstale":"2023-10-16T11:19:37.700060+0000","last_undegraded":"2023-10-16T11:19:37.700060+0000","last_fullsized":"2023-10-16T11:19:37.700060+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:27.320950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678028+0000","last_change":"2023-10-16T11:19:22.318314+0000","last_active":"2023-10-16T11:19:36.678028+0000","last_peered":"2023-10-16T11:19:36.678028+0000","last_clean":"2023-10-16T11:19:36.678028+0000","last_became_active":"2023-10-16T11:19:22.318015+0000","last_became_peered":"2023-10-16T11:19:22.318015+0000","last_unstale":"2023-10-16T11:19:36.678028+0000","last_undegraded":"2023-10-16T11:19:36.678028+0000","last_fullsized":"2023-10-16T11:19:36.678028+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:45.421816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668042+0000","last_change":"2023-10-16T11:19:23.337274+0000","last_active":"2023-10-16T11:19:36.668042+0000","last_peered":"2023-10-16T11:19:36.668042+0000","last_clean":"2023-10-16T11:19:36.668042+0000","last_became_active":"2023-10-16T11:19:23.336889+0000","last_became_peered":"2023-10-16T11:19:23.336889+0000","last_unstale":"2023-10-16T11:19:36.668042+0000","last_undegraded":"2023-10-16T11:19:36.668042+0000","last_fullsized":"2023-10-16T11:19:36.668042+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699643+0000","last_change":"2023-10-16T11:19:35.679263+0000","last_active":"2023-10-16T11:19:36.699643+0000","last_peered":"2023-10-16T11:19:36.699643+0000","last_clean":"2023-10-16T11:19:36.699643+0000","last_became_active":"2023-10-16T11:19:35.678812+0000","last_became_peered":"2023-10-16T11:19:35.678812+0000","last_unstale":"2023-10-16T11:19:36.699643+0000","last_undegraded":"2023-10-16T11:19:36.699643+0000","last_fullsized":"2023-10-16T11:19:36.699643+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:57.878898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696290+0000","last_change":"2023-10-16T11:19:36.684169+0000","last_active":"2023-10-16T11:19:37.696290+0000","last_peered":"2023-10-16T11:19:37.696290+0000","last_clean":"2023-10-16T11:19:37.696290+0000","last_became_active":"2023-10-16T11:19:36.683898+0000","last_became_peered":"2023-10-16T11:19:36.683898+0000","last_unstale":"2023-10-16T11:19:37.696290+0000","last_undegraded":"2023-10-16T11:19:37.696290+0000","last_fullsized":"2023-10-16T11:19:37.696290+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:04:05.120700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665201+0000","last_change":"2023-10-16T11:19:22.330569+0000","last_active":"2023-10-16T11:19:36.665201+0000","last_peered":"2023-10-16T11:19:36.665201+0000","last_clean":"2023-10-16T11:19:36.665201+0000","last_became_active":"2023-10-16T11:19:22.329740+0000","last_became_peered":"2023-10-16T11:19:22.329740+0000","last_unstale":"2023-10-16T11:19:36.665201+0000","last_undegraded":"2023-10-16T11:19:36.665201+0000","last_fullsized":"2023-10-16T11:19:36.665201+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664845+0000","last_change":"2023-10-16T11:19:23.338701+0000","last_active":"2023-10-16T11:19:36.664845+0000","last_peered":"2023-10-16T11:19:36.664845+0000","last_clean":"2023-10-16T11:19:36.664845+0000","last_became_active":"2023-10-16T11:19:23.334033+0000","last_became_peered":"2023-10-16T11:19:23.334033+0000","last_unstale":"2023-10-16T11:19:36.664845+0000","last_undegraded":"2023-10-16T11:19:36.664845+0000","last_fullsized":"2023-10-16T11:19:36.664845+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672283+0000","last_change":"2023-10-16T11:19:35.677127+0000","last_active":"2023-10-16T11:19:36.672283+0000","last_peered":"2023-10-16T11:19:36.672283+0000","last_clean":"2023-10-16T11:19:36.672283+0000","last_became_active":"2023-10-16T11:19:35.676828+0000","last_became_peered":"2023-10-16T11:19:35.676828+0000","last_unstale":"2023-10-16T11:19:36.672283+0000","last_undegraded":"2023-10-16T11:19:36.672283+0000","last_fullsized":"2023-10-16T11:19:36.672283+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:15.101657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663965+0000","last_change":"2023-10-16T11:19:36.698526+0000","last_active":"2023-10-16T11:19:37.663965+0000","last_peered":"2023-10-16T11:19:37.663965+0000","last_clean":"2023-10-16T11:19:37.663965+0000","last_became_active":"2023-10-16T11:19:36.698271+0000","last_became_peered":"2023-10-16T11:19:36.698271+0000","last_unstale":"2023-10-16T11:19:37.663965+0000","last_undegraded":"2023-10-16T11:19:37.663965+0000","last_fullsized":"2023-10-16T11:19:37.663965+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:53.874459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665100+0000","last_change":"2023-10-16T11:19:22.331432+0000","last_active":"2023-10-16T11:19:36.665100+0000","last_peered":"2023-10-16T11:19:36.665100+0000","last_clean":"2023-10-16T11:19:36.665100+0000","last_became_active":"2023-10-16T11:19:22.331124+0000","last_became_peered":"2023-10-16T11:19:22.331124+0000","last_unstale":"2023-10-16T11:19:36.665100+0000","last_undegraded":"2023-10-16T11:19:36.665100+0000","last_fullsized":"2023-10-16T11:19:36.665100+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.680058+0000","last_change":"2023-10-16T11:19:23.333018+0000","last_active":"2023-10-16T11:19:36.680058+0000","last_peered":"2023-10-16T11:19:36.680058+0000","last_clean":"2023-10-16T11:19:36.680058+0000","last_became_active":"2023-10-16T11:19:23.332725+0000","last_became_peered":"2023-10-16T11:19:23.332725+0000","last_unstale":"2023-10-16T11:19:36.680058+0000","last_undegraded":"2023-10-16T11:19:36.680058+0000","last_fullsized":"2023-10-16T11:19:36.680058+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:53:39.056496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666647+0000","last_change":"2023-10-16T11:19:35.669741+0000","last_active":"2023-10-16T11:19:36.666647+0000","last_peered":"2023-10-16T11:19:36.666647+0000","last_clean":"2023-10-16T11:19:36.666647+0000","last_became_active":"2023-10-16T11:19:35.669477+0000","last_became_peered":"2023-10-16T11:19:35.669477+0000","last_unstale":"2023-10-16T11:19:36.666647+0000","last_undegraded":"2023-10-16T11:19:36.666647+0000","last_fullsized":"2023-10-16T11:19:36.666647+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:57.643373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994219+0000","last_change":"2023-10-16T11:19:36.690841+0000","last_active":"2023-10-16T11:19:37.994219+0000","last_peered":"2023-10-16T11:19:37.994219+0000","last_clean":"2023-10-16T11:19:37.994219+0000","last_became_active":"2023-10-16T11:19:36.690558+0000","last_became_peered":"2023-10-16T11:19:36.690558+0000","last_unstale":"2023-10-16T11:19:37.994219+0000","last_undegraded":"2023-10-16T11:19:37.994219+0000","last_fullsized":"2023-10-16T11:19:37.994219+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:30.591636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"216'8","reported_seq":24,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671923+0000","last_change":"2023-10-16T11:19:22.332156+0000","last_active":"2023-10-16T11:19:36.671923+0000","last_peered":"2023-10-16T11:19:36.671923+0000","last_clean":"2023-10-16T11:19:36.671923+0000","last_became_active":"2023-10-16T11:19:22.331863+0000","last_became_peered":"2023-10-16T11:19:22.331863+0000","last_unstale":"2023-10-16T11:19:36.671923+0000","last_undegraded":"2023-10-16T11:19:36.671923+0000","last_fullsized":"2023-10-16T11:19:36.671923+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:36.175530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670117+0000","last_change":"2023-10-16T11:19:23.336992+0000","last_active":"2023-10-16T11:19:36.670117+0000","last_peered":"2023-10-16T11:19:36.670117+0000","last_clean":"2023-10-16T11:19:36.670117+0000","last_became_active":"2023-10-16T11:19:23.336724+0000","last_became_peered":"2023-10-16T11:19:23.336724+0000","last_unstale":"2023-10-16T11:19:36.670117+0000","last_undegraded":"2023-10-16T11:19:36.670117+0000","last_fullsized":"2023-10-16T11:19:36.670117+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:01.326738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667579+0000","last_change":"2023-10-16T11:19:35.678016+0000","last_active":"2023-10-16T11:19:36.667579+0000","last_peered":"2023-10-16T11:19:36.667579+0000","last_clean":"2023-10-16T11:19:36.667579+0000","last_became_active":"2023-10-16T11:19:35.677757+0000","last_became_peered":"2023-10-16T11:19:35.677757+0000","last_unstale":"2023-10-16T11:19:36.667579+0000","last_undegraded":"2023-10-16T11:19:36.667579+0000","last_fullsized":"2023-10-16T11:19:36.667579+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:46.311559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994557+0000","last_change":"2023-10-16T11:19:36.677981+0000","last_active":"2023-10-16T11:19:37.994557+0000","last_peered":"2023-10-16T11:19:37.994557+0000","last_clean":"2023-10-16T11:19:37.994557+0000","last_became_active":"2023-10-16T11:19:36.677677+0000","last_became_peered":"2023-10-16T11:19:36.677677+0000","last_unstale":"2023-10-16T11:19:37.994557+0000","last_undegraded":"2023-10-16T11:19:37.994557+0000","last_fullsized":"2023-10-16T11:19:37.994557+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:31.283029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662222+0000","last_change":"2023-10-16T11:19:22.331044+0000","last_active":"2023-10-16T11:19:36.662222+0000","last_peered":"2023-10-16T11:19:36.662222+0000","last_clean":"2023-10-16T11:19:36.662222+0000","last_became_active":"2023-10-16T11:19:22.330757+0000","last_became_peered":"2023-10-16T11:19:22.330757+0000","last_unstale":"2023-10-16T11:19:36.662222+0000","last_undegraded":"2023-10-16T11:19:36.662222+0000","last_fullsized":"2023-10-16T11:19:36.662222+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662806+0000","last_change":"2023-10-16T11:19:23.333500+0000","last_active":"2023-10-16T11:19:36.662806+0000","last_peered":"2023-10-16T11:19:36.662806+0000","last_clean":"2023-10-16T11:19:36.662806+0000","last_became_active":"2023-10-16T11:19:23.333215+0000","last_became_peered":"2023-10-16T11:19:23.333215+0000","last_unstale":"2023-10-16T11:19:36.662806+0000","last_undegraded":"2023-10-16T11:19:36.662806+0000","last_fullsized":"2023-10-16T11:19:36.662806+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670009+0000","last_change":"2023-10-16T11:19:35.673910+0000","last_active":"2023-10-16T11:19:36.670009+0000","last_peered":"2023-10-16T11:19:36.670009+0000","last_clean":"2023-10-16T11:19:36.670009+0000","last_became_active":"2023-10-16T11:19:35.673640+0000","last_became_peered":"2023-10-16T11:19:35.673640+0000","last_unstale":"2023-10-16T11:19:36.670009+0000","last_undegraded":"2023-10-16T11:19:36.670009+0000","last_fullsized":"2023-10-16T11:19:36.670009+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:42.767262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994757+0000","last_change":"2023-10-16T11:19:36.678756+0000","last_active":"2023-10-16T11:19:37.994757+0000","last_peered":"2023-10-16T11:19:37.994757+0000","last_clean":"2023-10-16T11:19:37.994757+0000","last_became_active":"2023-10-16T11:19:36.678501+0000","last_became_peered":"2023-10-16T11:19:36.678501+0000","last_unstale":"2023-10-16T11:19:37.994757+0000","last_undegraded":"2023-10-16T11:19:37.994757+0000","last_fullsized":"2023-10-16T11:19:37.994757+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:00.784444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667556+0000","last_change":"2023-10-16T11:19:22.327307+0000","last_active":"2023-10-16T11:19:36.667556+0000","last_peered":"2023-10-16T11:19:36.667556+0000","last_clean":"2023-10-16T11:19:36.667556+0000","last_became_active":"2023-10-16T11:19:22.327036+0000","last_became_peered":"2023-10-16T11:19:22.327036+0000","last_unstale":"2023-10-16T11:19:36.667556+0000","last_undegraded":"2023-10-16T11:19:36.667556+0000","last_fullsized":"2023-10-16T11:19:36.667556+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:24.276671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666847+0000","last_change":"2023-10-16T11:19:23.337758+0000","last_active":"2023-10-16T11:19:36.666847+0000","last_peered":"2023-10-16T11:19:36.666847+0000","last_clean":"2023-10-16T11:19:36.666847+0000","last_became_active":"2023-10-16T11:19:23.337147+0000","last_became_peered":"2023-10-16T11:19:23.337147+0000","last_unstale":"2023-10-16T11:19:36.666847+0000","last_undegraded":"2023-10-16T11:19:36.666847+0000","last_fullsized":"2023-10-16T11:19:36.666847+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:06.136692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"220'6","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.700777+0000","last_change":"2023-10-16T11:19:35.682083+0000","last_active":"2023-10-16T11:19:36.700777+0000","last_peered":"2023-10-16T11:19:36.700777+0000","last_clean":"2023-10-16T11:19:36.700777+0000","last_became_active":"2023-10-16T11:19:35.681804+0000","last_became_peered":"2023-10-16T11:19:35.681804+0000","last_unstale":"2023-10-16T11:19:36.700777+0000","last_undegraded":"2023-10-16T11:19:36.700777+0000","last_fullsized":"2023-10-16T11:19:36.700777+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:54.161780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664448+0000","last_change":"2023-10-16T11:19:36.692085+0000","last_active":"2023-10-16T11:19:37.664448+0000","last_peered":"2023-10-16T11:19:37.664448+0000","last_clean":"2023-10-16T11:19:37.664448+0000","last_became_active":"2023-10-16T11:19:36.691313+0000","last_became_peered":"2023-10-16T11:19:36.691313+0000","last_unstale":"2023-10-16T11:19:37.664448+0000","last_undegraded":"2023-10-16T11:19:37.664448+0000","last_fullsized":"2023-10-16T11:19:37.664448+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:54.154198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675831+0000","last_change":"2023-10-16T11:19:22.325030+0000","last_active":"2023-10-16T11:19:36.675831+0000","last_peered":"2023-10-16T11:19:36.675831+0000","last_clean":"2023-10-16T11:19:36.675831+0000","last_became_active":"2023-10-16T11:19:22.324733+0000","last_became_peered":"2023-10-16T11:19:22.324733+0000","last_unstale":"2023-10-16T11:19:36.675831+0000","last_undegraded":"2023-10-16T11:19:36.675831+0000","last_fullsized":"2023-10-16T11:19:36.675831+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:29.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671514+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:36.671514+0000","last_peered":"2023-10-16T11:19:36.671514+0000","last_clean":"2023-10-16T11:19:36.671514+0000","last_became_active":"2023-10-16T11:19:23.331725+0000","last_became_peered":"2023-10-16T11:19:23.331725+0000","last_unstale":"2023-10-16T11:19:36.671514+0000","last_undegraded":"2023-10-16T11:19:36.671514+0000","last_fullsized":"2023-10-16T11:19:36.671514+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:45.043730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"220'3","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.682632+0000","last_change":"2023-10-16T11:19:35.672388+0000","last_active":"2023-10-16T11:19:36.682632+0000","last_peered":"2023-10-16T11:19:36.682632+0000","last_clean":"2023-10-16T11:19:36.682632+0000","last_became_active":"2023-10-16T11:19:35.672090+0000","last_became_peered":"2023-10-16T11:19:35.672090+0000","last_unstale":"2023-10-16T11:19:36.682632+0000","last_undegraded":"2023-10-16T11:19:36.682632+0000","last_fullsized":"2023-10-16T11:19:36.682632+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:15.533409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665100+0000","last_change":"2023-10-16T11:19:36.684006+0000","last_active":"2023-10-16T11:19:37.665100+0000","last_peered":"2023-10-16T11:19:37.665100+0000","last_clean":"2023-10-16T11:19:37.665100+0000","last_became_active":"2023-10-16T11:19:36.683679+0000","last_became_peered":"2023-10-16T11:19:36.683679+0000","last_unstale":"2023-10-16T11:19:37.665100+0000","last_undegraded":"2023-10-16T11:19:37.665100+0000","last_fullsized":"2023-10-16T11:19:37.665100+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:20.376959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"217'8","reported_seq":24,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666769+0000","last_change":"2023-10-16T11:19:22.329082+0000","last_active":"2023-10-16T11:19:36.666769+0000","last_peered":"2023-10-16T11:19:36.666769+0000","last_clean":"2023-10-16T11:19:36.666769+0000","last_became_active":"2023-10-16T11:19:22.328756+0000","last_became_peered":"2023-10-16T11:19:22.328756+0000","last_unstale":"2023-10-16T11:19:36.666769+0000","last_undegraded":"2023-10-16T11:19:36.666769+0000","last_fullsized":"2023-10-16T11:19:36.666769+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.384963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.673087+0000","last_change":"2023-10-16T11:19:23.336196+0000","last_active":"2023-10-16T11:19:36.673087+0000","last_peered":"2023-10-16T11:19:36.673087+0000","last_clean":"2023-10-16T11:19:36.673087+0000","last_became_active":"2023-10-16T11:19:23.335924+0000","last_became_peered":"2023-10-16T11:19:23.335924+0000","last_unstale":"2023-10-16T11:19:36.673087+0000","last_undegraded":"2023-10-16T11:19:36.673087+0000","last_fullsized":"2023-10-16T11:19:36.673087+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:27:01.350215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662408+0000","last_change":"2023-10-16T11:19:35.670005+0000","last_active":"2023-10-16T11:19:36.662408+0000","last_peered":"2023-10-16T11:19:36.662408+0000","last_clean":"2023-10-16T11:19:36.662408+0000","last_became_active":"2023-10-16T11:19:35.669743+0000","last_became_peered":"2023-10-16T11:19:35.669743+0000","last_unstale":"2023-10-16T11:19:36.662408+0000","last_undegraded":"2023-10-16T11:19:36.662408+0000","last_fullsized":"2023-10-16T11:19:36.662408+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:59.779061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696422+0000","last_change":"2023-10-16T11:19:36.697172+0000","last_active":"2023-10-16T11:19:37.696422+0000","last_peered":"2023-10-16T11:19:37.696422+0000","last_clean":"2023-10-16T11:19:37.696422+0000","last_became_active":"2023-10-16T11:19:36.696866+0000","last_became_peered":"2023-10-16T11:19:36.696866+0000","last_unstale":"2023-10-16T11:19:37.696422+0000","last_undegraded":"2023-10-16T11:19:37.696422+0000","last_fullsized":"2023-10-16T11:19:37.696422+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:18.714994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"216'3","reported_seq":19,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662822+0000","last_change":"2023-10-16T11:19:22.331150+0000","last_active":"2023-10-16T11:19:36.662822+0000","last_peered":"2023-10-16T11:19:36.662822+0000","last_clean":"2023-10-16T11:19:36.662822+0000","last_became_active":"2023-10-16T11:19:22.330820+0000","last_became_peered":"2023-10-16T11:19:22.330820+0000","last_unstale":"2023-10-16T11:19:36.662822+0000","last_undegraded":"2023-10-16T11:19:36.662822+0000","last_fullsized":"2023-10-16T11:19:36.662822+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:50.307463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662601+0000","last_change":"2023-10-16T11:19:23.338568+0000","last_active":"2023-10-16T11:19:36.662601+0000","last_peered":"2023-10-16T11:19:36.662601+0000","last_clean":"2023-10-16T11:19:36.662601+0000","last_became_active":"2023-10-16T11:19:23.338280+0000","last_became_peered":"2023-10-16T11:19:23.338280+0000","last_unstale":"2023-10-16T11:19:36.662601+0000","last_undegraded":"2023-10-16T11:19:36.662601+0000","last_fullsized":"2023-10-16T11:19:36.662601+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:40.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699722+0000","last_change":"2023-10-16T11:19:35.686423+0000","last_active":"2023-10-16T11:19:36.699722+0000","last_peered":"2023-10-16T11:19:36.699722+0000","last_clean":"2023-10-16T11:19:36.699722+0000","last_became_active":"2023-10-16T11:19:35.685384+0000","last_became_peered":"2023-10-16T11:19:35.685384+0000","last_unstale":"2023-10-16T11:19:36.699722+0000","last_undegraded":"2023-10-16T11:19:36.699722+0000","last_fullsized":"2023-10-16T11:19:36.699722+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:49.913941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994146+0000","last_change":"2023-10-16T11:19:36.672963+0000","last_active":"2023-10-16T11:19:37.994146+0000","last_peered":"2023-10-16T11:19:37.994146+0000","last_clean":"2023-10-16T11:19:37.994146+0000","last_became_active":"2023-10-16T11:19:36.672611+0000","last_became_peered":"2023-10-16T11:19:36.672611+0000","last_unstale":"2023-10-16T11:19:37.994146+0000","last_undegraded":"2023-10-16T11:19:37.994146+0000","last_fullsized":"2023-10-16T11:19:37.994146+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:17.500716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670020+0000","last_change":"2023-10-16T11:19:22.325211+0000","last_active":"2023-10-16T11:19:36.670020+0000","last_peered":"2023-10-16T11:19:36.670020+0000","last_clean":"2023-10-16T11:19:36.670020+0000","last_became_active":"2023-10-16T11:19:22.324901+0000","last_became_peered":"2023-10-16T11:19:22.324901+0000","last_unstale":"2023-10-16T11:19:36.670020+0000","last_undegraded":"2023-10-16T11:19:36.670020+0000","last_fullsized":"2023-10-16T11:19:36.670020+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:46.432434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666928+0000","last_change":"2023-10-16T11:19:23.331144+0000","last_active":"2023-10-16T11:19:36.666928+0000","last_peered":"2023-10-16T11:19:36.666928+0000","last_clean":"2023-10-16T11:19:36.666928+0000","last_became_active":"2023-10-16T11:19:23.330824+0000","last_became_peered":"2023-10-16T11:19:23.330824+0000","last_unstale":"2023-10-16T11:19:36.666928+0000","last_undegraded":"2023-10-16T11:19:36.666928+0000","last_fullsized":"2023-10-16T11:19:36.666928+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:44:27.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"220'8","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.699249+0000","last_change":"2023-10-16T11:19:35.678999+0000","last_active":"2023-10-16T11:19:36.699249+0000","last_peered":"2023-10-16T11:19:36.699249+0000","last_clean":"2023-10-16T11:19:36.699249+0000","last_became_active":"2023-10-16T11:19:35.678522+0000","last_became_peered":"2023-10-16T11:19:35.678522+0000","last_unstale":"2023-10-16T11:19:36.699249+0000","last_undegraded":"2023-10-16T11:19:36.699249+0000","last_fullsized":"2023-10-16T11:19:36.699249+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:55:51.788613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994331+0000","last_change":"2023-10-16T11:19:36.674240+0000","last_active":"2023-10-16T11:19:37.994331+0000","last_peered":"2023-10-16T11:19:37.994331+0000","last_clean":"2023-10-16T11:19:37.994331+0000","last_became_active":"2023-10-16T11:19:36.673954+0000","last_became_peered":"2023-10-16T11:19:36.673954+0000","last_unstale":"2023-10-16T11:19:37.994331+0000","last_undegraded":"2023-10-16T11:19:37.994331+0000","last_fullsized":"2023-10-16T11:19:37.994331+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:15.742018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.676479+0000","last_change":"2023-10-16T11:19:22.332253+0000","last_active":"2023-10-16T11:19:36.676479+0000","last_peered":"2023-10-16T11:19:36.676479+0000","last_clean":"2023-10-16T11:19:36.676479+0000","last_became_active":"2023-10-16T11:19:22.331970+0000","last_became_peered":"2023-10-16T11:19:22.331970+0000","last_unstale":"2023-10-16T11:19:36.676479+0000","last_undegraded":"2023-10-16T11:19:36.676479+0000","last_fullsized":"2023-10-16T11:19:36.676479+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:19.740978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667271+0000","last_change":"2023-10-16T11:19:23.330327+0000","last_active":"2023-10-16T11:19:36.667271+0000","last_peered":"2023-10-16T11:19:36.667271+0000","last_clean":"2023-10-16T11:19:36.667271+0000","last_became_active":"2023-10-16T11:19:23.330040+0000","last_became_peered":"2023-10-16T11:19:23.330040+0000","last_unstale":"2023-10-16T11:19:36.667271+0000","last_undegraded":"2023-10-16T11:19:36.667271+0000","last_fullsized":"2023-10-16T11:19:36.667271+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.664745+0000","last_change":"2023-10-16T11:19:35.677569+0000","last_active":"2023-10-16T11:19:36.664745+0000","last_peered":"2023-10-16T11:19:36.664745+0000","last_clean":"2023-10-16T11:19:36.664745+0000","last_became_active":"2023-10-16T11:19:35.677275+0000","last_became_peered":"2023-10-16T11:19:35.677275+0000","last_unstale":"2023-10-16T11:19:36.664745+0000","last_undegraded":"2023-10-16T11:19:36.664745+0000","last_fullsized":"2023-10-16T11:19:36.664745+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:56.520892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663873+0000","last_change":"2023-10-16T11:19:36.678766+0000","last_active":"2023-10-16T11:19:37.663873+0000","last_peered":"2023-10-16T11:19:37.663873+0000","last_clean":"2023-10-16T11:19:37.663873+0000","last_became_active":"2023-10-16T11:19:36.678481+0000","last_became_peered":"2023-10-16T11:19:36.678481+0000","last_unstale":"2023-10-16T11:19:37.663873+0000","last_undegraded":"2023-10-16T11:19:37.663873+0000","last_fullsized":"2023-10-16T11:19:37.663873+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:23.629013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666314+0000","last_change":"2023-10-16T11:19:22.330754+0000","last_active":"2023-10-16T11:19:36.666314+0000","last_peered":"2023-10-16T11:19:36.666314+0000","last_clean":"2023-10-16T11:19:36.666314+0000","last_became_active":"2023-10-16T11:19:22.330490+0000","last_became_peered":"2023-10-16T11:19:22.330490+0000","last_unstale":"2023-10-16T11:19:36.666314+0000","last_undegraded":"2023-10-16T11:19:36.666314+0000","last_fullsized":"2023-10-16T11:19:36.666314+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:17.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662829+0000","last_change":"2023-10-16T11:19:23.334130+0000","last_active":"2023-10-16T11:19:36.662829+0000","last_peered":"2023-10-16T11:19:36.662829+0000","last_clean":"2023-10-16T11:19:36.662829+0000","last_became_active":"2023-10-16T11:19:23.333073+0000","last_became_peered":"2023-10-16T11:19:23.333073+0000","last_unstale":"2023-10-16T11:19:36.662829+0000","last_undegraded":"2023-10-16T11:19:36.662829+0000","last_fullsized":"2023-10-16T11:19:36.662829+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:38.491191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.697295+0000","last_change":"2023-10-16T11:19:35.683271+0000","last_active":"2023-10-16T11:19:36.697295+0000","last_peered":"2023-10-16T11:19:36.697295+0000","last_clean":"2023-10-16T11:19:36.697295+0000","last_became_active":"2023-10-16T11:19:35.681021+0000","last_became_peered":"2023-10-16T11:19:35.681021+0000","last_unstale":"2023-10-16T11:19:36.697295+0000","last_undegraded":"2023-10-16T11:19:36.697295+0000","last_fullsized":"2023-10-16T11:19:36.697295+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:30.265858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664246+0000","last_change":"2023-10-16T11:19:36.679321+0000","last_active":"2023-10-16T11:19:37.664246+0000","last_peered":"2023-10-16T11:19:37.664246+0000","last_clean":"2023-10-16T11:19:37.664246+0000","last_became_active":"2023-10-16T11:19:36.679019+0000","last_became_peered":"2023-10-16T11:19:36.679019+0000","last_unstale":"2023-10-16T11:19:37.664246+0000","last_undegraded":"2023-10-16T11:19:37.664246+0000","last_fullsized":"2023-10-16T11:19:37.664246+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:00.210732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667231+0000","last_change":"2023-10-16T11:19:22.328061+0000","last_active":"2023-10-16T11:19:36.667231+0000","last_peered":"2023-10-16T11:19:36.667231+0000","last_clean":"2023-10-16T11:19:36.667231+0000","last_became_active":"2023-10-16T11:19:22.327773+0000","last_became_peered":"2023-10-16T11:19:22.327773+0000","last_unstale":"2023-10-16T11:19:36.667231+0000","last_undegraded":"2023-10-16T11:19:36.667231+0000","last_fullsized":"2023-10-16T11:19:36.667231+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:39.077769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668801+0000","last_change":"2023-10-16T11:19:23.330616+0000","last_active":"2023-10-16T11:19:36.668801+0000","last_peered":"2023-10-16T11:19:36.668801+0000","last_clean":"2023-10-16T11:19:36.668801+0000","last_became_active":"2023-10-16T11:19:23.330317+0000","last_became_peered":"2023-10-16T11:19:23.330317+0000","last_unstale":"2023-10-16T11:19:36.668801+0000","last_undegraded":"2023-10-16T11:19:36.668801+0000","last_fullsized":"2023-10-16T11:19:36.668801+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:38.058918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"220'4","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.686252+0000","last_change":"2023-10-16T11:19:35.671218+0000","last_active":"2023-10-16T11:19:36.686252+0000","last_peered":"2023-10-16T11:19:36.686252+0000","last_clean":"2023-10-16T11:19:36.686252+0000","last_became_active":"2023-10-16T11:19:35.670991+0000","last_became_peered":"2023-10-16T11:19:35.670991+0000","last_unstale":"2023-10-16T11:19:36.686252+0000","last_undegraded":"2023-10-16T11:19:36.686252+0000","last_fullsized":"2023-10-16T11:19:36.686252+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:11.399468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665905+0000","last_change":"2023-10-16T11:19:36.687869+0000","last_active":"2023-10-16T11:19:37.665905+0000","last_peered":"2023-10-16T11:19:37.665905+0000","last_clean":"2023-10-16T11:19:37.665905+0000","last_became_active":"2023-10-16T11:19:36.687472+0000","last_became_peered":"2023-10-16T11:19:36.687472+0000","last_unstale":"2023-10-16T11:19:37.665905+0000","last_undegraded":"2023-10-16T11:19:37.665905+0000","last_fullsized":"2023-10-16T11:19:37.665905+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:45.543759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666971+0000","last_change":"2023-10-16T11:19:22.328217+0000","last_active":"2023-10-16T11:19:36.666971+0000","last_peered":"2023-10-16T11:19:36.666971+0000","last_clean":"2023-10-16T11:19:36.666971+0000","last_became_active":"2023-10-16T11:19:22.327901+0000","last_became_peered":"2023-10-16T11:19:22.327901+0000","last_unstale":"2023-10-16T11:19:36.666971+0000","last_undegraded":"2023-10-16T11:19:36.666971+0000","last_fullsized":"2023-10-16T11:19:36.666971+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:48.552111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671972+0000","last_change":"2023-10-16T11:19:23.336335+0000","last_active":"2023-10-16T11:19:36.671972+0000","last_peered":"2023-10-16T11:19:36.671972+0000","last_clean":"2023-10-16T11:19:36.671972+0000","last_became_active":"2023-10-16T11:19:23.335852+0000","last_became_peered":"2023-10-16T11:19:23.335852+0000","last_unstale":"2023-10-16T11:19:36.671972+0000","last_undegraded":"2023-10-16T11:19:36.671972+0000","last_fullsized":"2023-10-16T11:19:36.671972+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:13.506516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"220'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.686520+0000","last_change":"2023-10-16T11:19:35.679257+0000","last_active":"2023-10-16T11:19:36.686520+0000","last_peered":"2023-10-16T11:19:36.686520+0000","last_clean":"2023-10-16T11:19:36.686520+0000","last_became_active":"2023-10-16T11:19:35.678977+0000","last_became_peered":"2023-10-16T11:19:35.678977+0000","last_unstale":"2023-10-16T11:19:36.686520+0000","last_undegraded":"2023-10-16T11:19:36.686520+0000","last_fullsized":"2023-10-16T11:19:36.686520+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:29.586253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697443+0000","last_change":"2023-10-16T11:19:36.683265+0000","last_active":"2023-10-16T11:19:37.697443+0000","last_peered":"2023-10-16T11:19:37.697443+0000","last_clean":"2023-10-16T11:19:37.697443+0000","last_became_active":"2023-10-16T11:19:36.681552+0000","last_became_peered":"2023-10-16T11:19:36.681552+0000","last_unstale":"2023-10-16T11:19:37.697443+0000","last_undegraded":"2023-10-16T11:19:37.697443+0000","last_fullsized":"2023-10-16T11:19:37.697443+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:08:32.673134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663673+0000","last_change":"2023-10-16T11:19:22.329968+0000","last_active":"2023-10-16T11:19:36.663673+0000","last_peered":"2023-10-16T11:19:36.663673+0000","last_clean":"2023-10-16T11:19:36.663673+0000","last_became_active":"2023-10-16T11:19:22.326360+0000","last_became_peered":"2023-10-16T11:19:22.326360+0000","last_unstale":"2023-10-16T11:19:36.663673+0000","last_undegraded":"2023-10-16T11:19:36.663673+0000","last_fullsized":"2023-10-16T11:19:36.663673+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:01.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.675124+0000","last_change":"2023-10-16T11:19:23.335733+0000","last_active":"2023-10-16T11:19:36.675124+0000","last_peered":"2023-10-16T11:19:36.675124+0000","last_clean":"2023-10-16T11:19:36.675124+0000","last_became_active":"2023-10-16T11:19:23.334936+0000","last_became_peered":"2023-10-16T11:19:23.334936+0000","last_unstale":"2023-10-16T11:19:36.675124+0000","last_undegraded":"2023-10-16T11:19:36.675124+0000","last_fullsized":"2023-10-16T11:19:36.675124+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:08.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.690777+0000","last_change":"2023-10-16T11:19:35.684464+0000","last_active":"2023-10-16T11:19:36.690777+0000","last_peered":"2023-10-16T11:19:36.690777+0000","last_clean":"2023-10-16T11:19:36.690777+0000","last_became_active":"2023-10-16T11:19:35.684139+0000","last_became_peered":"2023-10-16T11:19:35.684139+0000","last_unstale":"2023-10-16T11:19:36.690777+0000","last_undegraded":"2023-10-16T11:19:36.690777+0000","last_fullsized":"2023-10-16T11:19:36.690777+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:52.513114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696386+0000","last_change":"2023-10-16T11:19:36.691931+0000","last_active":"2023-10-16T11:19:37.696386+0000","last_peered":"2023-10-16T11:19:37.696386+0000","last_clean":"2023-10-16T11:19:37.696386+0000","last_became_active":"2023-10-16T11:19:36.691594+0000","last_became_peered":"2023-10-16T11:19:36.691594+0000","last_unstale":"2023-10-16T11:19:37.696386+0000","last_undegraded":"2023-10-16T11:19:37.696386+0000","last_fullsized":"2023-10-16T11:19:37.696386+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:56.795176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"216'2","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.677928+0000","last_change":"2023-10-16T11:19:22.328385+0000","last_active":"2023-10-16T11:19:36.677928+0000","last_peered":"2023-10-16T11:19:36.677928+0000","last_clean":"2023-10-16T11:19:36.677928+0000","last_became_active":"2023-10-16T11:19:22.328096+0000","last_became_peered":"2023-10-16T11:19:22.328096+0000","last_unstale":"2023-10-16T11:19:36.677928+0000","last_undegraded":"2023-10-16T11:19:36.677928+0000","last_fullsized":"2023-10-16T11:19:36.677928+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:40.312176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678838+0000","last_change":"2023-10-16T11:19:23.326944+0000","last_active":"2023-10-16T11:19:36.678838+0000","last_peered":"2023-10-16T11:19:36.678838+0000","last_clean":"2023-10-16T11:19:36.678838+0000","last_became_active":"2023-10-16T11:19:23.326680+0000","last_became_peered":"2023-10-16T11:19:23.326680+0000","last_unstale":"2023-10-16T11:19:36.678838+0000","last_undegraded":"2023-10-16T11:19:36.678838+0000","last_fullsized":"2023-10-16T11:19:36.678838+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:56:28.070345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"220'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.693442+0000","last_change":"2023-10-16T11:19:35.685986+0000","last_active":"2023-10-16T11:19:36.693442+0000","last_peered":"2023-10-16T11:19:36.693442+0000","last_clean":"2023-10-16T11:19:36.693442+0000","last_became_active":"2023-10-16T11:19:35.685758+0000","last_became_peered":"2023-10-16T11:19:35.685758+0000","last_unstale":"2023-10-16T11:19:36.693442+0000","last_undegraded":"2023-10-16T11:19:36.693442+0000","last_fullsized":"2023-10-16T11:19:36.693442+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:12.481754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663581+0000","last_change":"2023-10-16T11:19:36.687085+0000","last_active":"2023-10-16T11:19:37.663581+0000","last_peered":"2023-10-16T11:19:37.663581+0000","last_clean":"2023-10-16T11:19:37.663581+0000","last_became_active":"2023-10-16T11:19:36.686804+0000","last_became_peered":"2023-10-16T11:19:36.686804+0000","last_unstale":"2023-10-16T11:19:37.663581+0000","last_undegraded":"2023-10-16T11:19:37.663581+0000","last_fullsized":"2023-10-16T11:19:37.663581+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:04.397243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"216'1","reported_seq":17,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667326+0000","last_change":"2023-10-16T11:19:22.332064+0000","last_active":"2023-10-16T11:19:36.667326+0000","last_peered":"2023-10-16T11:19:36.667326+0000","last_clean":"2023-10-16T11:19:36.667326+0000","last_became_active":"2023-10-16T11:19:22.331746+0000","last_became_peered":"2023-10-16T11:19:22.331746+0000","last_unstale":"2023-10-16T11:19:36.667326+0000","last_undegraded":"2023-10-16T11:19:36.667326+0000","last_fullsized":"2023-10-16T11:19:36.667326+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:51.951857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665063+0000","last_change":"2023-10-16T11:19:23.342219+0000","last_active":"2023-10-16T11:19:36.665063+0000","last_peered":"2023-10-16T11:19:36.665063+0000","last_clean":"2023-10-16T11:19:36.665063+0000","last_became_active":"2023-10-16T11:19:23.334494+0000","last_became_peered":"2023-10-16T11:19:23.334494+0000","last_unstale":"2023-10-16T11:19:36.665063+0000","last_undegraded":"2023-10-16T11:19:36.665063+0000","last_fullsized":"2023-10-16T11:19:36.665063+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:40.085632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670212+0000","last_change":"2023-10-16T11:19:35.674368+0000","last_active":"2023-10-16T11:19:36.670212+0000","last_peered":"2023-10-16T11:19:36.670212+0000","last_clean":"2023-10-16T11:19:36.670212+0000","last_became_active":"2023-10-16T11:19:35.674151+0000","last_became_peered":"2023-10-16T11:19:35.674151+0000","last_unstale":"2023-10-16T11:19:36.670212+0000","last_undegraded":"2023-10-16T11:19:36.670212+0000","last_fullsized":"2023-10-16T11:19:36.670212+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.707693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663231+0000","last_change":"2023-10-16T11:19:36.687042+0000","last_active":"2023-10-16T11:19:37.663231+0000","last_peered":"2023-10-16T11:19:37.663231+0000","last_clean":"2023-10-16T11:19:37.663231+0000","last_became_active":"2023-10-16T11:19:36.686749+0000","last_became_peered":"2023-10-16T11:19:36.686749+0000","last_unstale":"2023-10-16T11:19:37.663231+0000","last_undegraded":"2023-10-16T11:19:37.663231+0000","last_fullsized":"2023-10-16T11:19:37.663231+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:46.573220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"216'4","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668738+0000","last_change":"2023-10-16T11:19:22.332308+0000","last_active":"2023-10-16T11:19:36.668738+0000","last_peered":"2023-10-16T11:19:36.668738+0000","last_clean":"2023-10-16T11:19:36.668738+0000","last_became_active":"2023-10-16T11:19:22.332005+0000","last_became_peered":"2023-10-16T11:19:22.332005+0000","last_unstale":"2023-10-16T11:19:36.668738+0000","last_undegraded":"2023-10-16T11:19:36.668738+0000","last_fullsized":"2023-10-16T11:19:36.668738+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:21.299639+0000","last_clean_scrub_stamp":"2023-10-16T11:19:21.299639+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:50.413697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.670591+0000","last_change":"2023-10-16T11:19:23.341701+0000","last_active":"2023-10-16T11:19:36.670591+0000","last_peered":"2023-10-16T11:19:36.670591+0000","last_clean":"2023-10-16T11:19:36.670591+0000","last_became_active":"2023-10-16T11:19:23.341424+0000","last_became_peered":"2023-10-16T11:19:23.341424+0000","last_unstale":"2023-10-16T11:19:36.670591+0000","last_undegraded":"2023-10-16T11:19:36.670591+0000","last_fullsized":"2023-10-16T11:19:36.670591+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:25.448895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667316+0000","last_change":"2023-10-16T11:19:35.684994+0000","last_active":"2023-10-16T11:19:36.667316+0000","last_peered":"2023-10-16T11:19:36.667316+0000","last_clean":"2023-10-16T11:19:36.667316+0000","last_became_active":"2023-10-16T11:19:35.684739+0000","last_became_peered":"2023-10-16T11:19:35.684739+0000","last_unstale":"2023-10-16T11:19:36.667316+0000","last_undegraded":"2023-10-16T11:19:36.667316+0000","last_fullsized":"2023-10-16T11:19:36.667316+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:34.646267+0000","last_clean_scrub_stamp":"2023-10-16T11:19:34.646267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:46.895984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994879+0000","last_change":"2023-10-16T11:19:36.687519+0000","last_active":"2023-10-16T11:19:37.994879+0000","last_peered":"2023-10-16T11:19:37.994879+0000","last_clean":"2023-10-16T11:19:37.994879+0000","last_became_active":"2023-10-16T11:19:36.686851+0000","last_became_peered":"2023-10-16T11:19:36.686851+0000","last_unstale":"2023-10-16T11:19:37.994879+0000","last_undegraded":"2023-10-16T11:19:37.994879+0000","last_fullsized":"2023-10-16T11:19:37.994879+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:20.736974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666244+0000","last_change":"2023-10-16T11:19:23.323955+0000","last_active":"2023-10-16T11:19:36.666244+0000","last_peered":"2023-10-16T11:19:36.666244+0000","last_clean":"2023-10-16T11:19:36.666244+0000","last_became_active":"2023-10-16T11:19:23.323706+0000","last_became_peered":"2023-10-16T11:19:23.323706+0000","last_unstale":"2023-10-16T11:19:36.666244+0000","last_undegraded":"2023-10-16T11:19:36.666244+0000","last_fullsized":"2023-10-16T11:19:36.666244+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.182127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994845+0000","last_change":"2023-10-16T11:19:36.684069+0000","last_active":"2023-10-16T11:19:37.994845+0000","last_peered":"2023-10-16T11:19:37.994845+0000","last_clean":"2023-10-16T11:19:37.994845+0000","last_became_active":"2023-10-16T11:19:36.683792+0000","last_became_peered":"2023-10-16T11:19:36.683792+0000","last_unstale":"2023-10-16T11:19:37.994845+0000","last_undegraded":"2023-10-16T11:19:37.994845+0000","last_fullsized":"2023-10-16T11:19:37.994845+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:52.124440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666136+0000","last_change":"2023-10-16T11:19:23.332826+0000","last_active":"2023-10-16T11:19:36.666136+0000","last_peered":"2023-10-16T11:19:36.666136+0000","last_clean":"2023-10-16T11:19:36.666136+0000","last_became_active":"2023-10-16T11:19:23.332541+0000","last_became_peered":"2023-10-16T11:19:23.332541+0000","last_unstale":"2023-10-16T11:19:36.666136+0000","last_undegraded":"2023-10-16T11:19:36.666136+0000","last_fullsized":"2023-10-16T11:19:36.666136+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:12:33.418278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994582+0000","last_change":"2023-10-16T11:19:36.677032+0000","last_active":"2023-10-16T11:19:37.994582+0000","last_peered":"2023-10-16T11:19:37.994582+0000","last_clean":"2023-10-16T11:19:37.994582+0000","last_became_active":"2023-10-16T11:19:36.676736+0000","last_became_peered":"2023-10-16T11:19:36.676736+0000","last_unstale":"2023-10-16T11:19:37.994582+0000","last_undegraded":"2023-10-16T11:19:37.994582+0000","last_fullsized":"2023-10-16T11:19:37.994582+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:24.179558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671403+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:36.671403+0000","last_peered":"2023-10-16T11:19:36.671403+0000","last_clean":"2023-10-16T11:19:36.671403+0000","last_became_active":"2023-10-16T11:19:23.335534+0000","last_became_peered":"2023-10-16T11:19:23.335534+0000","last_unstale":"2023-10-16T11:19:36.671403+0000","last_undegraded":"2023-10-16T11:19:36.671403+0000","last_fullsized":"2023-10-16T11:19:36.671403+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994750+0000","last_change":"2023-10-16T11:19:36.684273+0000","last_active":"2023-10-16T11:19:37.994750+0000","last_peered":"2023-10-16T11:19:37.994750+0000","last_clean":"2023-10-16T11:19:37.994750+0000","last_became_active":"2023-10-16T11:19:36.683688+0000","last_became_peered":"2023-10-16T11:19:36.683688+0000","last_unstale":"2023-10-16T11:19:37.994750+0000","last_undegraded":"2023-10-16T11:19:37.994750+0000","last_fullsized":"2023-10-16T11:19:37.994750+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:54.831776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662964+0000","last_change":"2023-10-16T11:19:23.330876+0000","last_active":"2023-10-16T11:19:36.662964+0000","last_peered":"2023-10-16T11:19:36.662964+0000","last_clean":"2023-10-16T11:19:36.662964+0000","last_became_active":"2023-10-16T11:19:23.330597+0000","last_became_peered":"2023-10-16T11:19:23.330597+0000","last_unstale":"2023-10-16T11:19:36.662964+0000","last_undegraded":"2023-10-16T11:19:36.662964+0000","last_fullsized":"2023-10-16T11:19:36.662964+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:59.414971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662848+0000","last_change":"2023-10-16T11:19:36.691604+0000","last_active":"2023-10-16T11:19:37.662848+0000","last_peered":"2023-10-16T11:19:37.662848+0000","last_clean":"2023-10-16T11:19:37.662848+0000","last_became_active":"2023-10-16T11:19:36.690918+0000","last_became_peered":"2023-10-16T11:19:36.690918+0000","last_unstale":"2023-10-16T11:19:37.662848+0000","last_undegraded":"2023-10-16T11:19:37.662848+0000","last_fullsized":"2023-10-16T11:19:37.662848+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:49.453674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664799+0000","last_change":"2023-10-16T11:19:36.684752+0000","last_active":"2023-10-16T11:19:37.664799+0000","last_peered":"2023-10-16T11:19:37.664799+0000","last_clean":"2023-10-16T11:19:37.664799+0000","last_became_active":"2023-10-16T11:19:36.684491+0000","last_became_peered":"2023-10-16T11:19:36.684491+0000","last_unstale":"2023-10-16T11:19:37.664799+0000","last_undegraded":"2023-10-16T11:19:37.664799+0000","last_fullsized":"2023-10-16T11:19:37.664799+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:39.093443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.674996+0000","last_change":"2023-10-16T11:19:23.327468+0000","last_active":"2023-10-16T11:19:36.674996+0000","last_peered":"2023-10-16T11:19:36.674996+0000","last_clean":"2023-10-16T11:19:36.674996+0000","last_became_active":"2023-10-16T11:19:23.327176+0000","last_became_peered":"2023-10-16T11:19:23.327176+0000","last_unstale":"2023-10-16T11:19:36.674996+0000","last_undegraded":"2023-10-16T11:19:36.674996+0000","last_fullsized":"2023-10-16T11:19:36.674996+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:02.654427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700296+0000","last_change":"2023-10-16T11:19:36.685932+0000","last_active":"2023-10-16T11:19:37.700296+0000","last_peered":"2023-10-16T11:19:37.700296+0000","last_clean":"2023-10-16T11:19:37.700296+0000","last_became_active":"2023-10-16T11:19:36.685576+0000","last_became_peered":"2023-10-16T11:19:36.685576+0000","last_unstale":"2023-10-16T11:19:37.700296+0000","last_undegraded":"2023-10-16T11:19:37.700296+0000","last_fullsized":"2023-10-16T11:19:37.700296+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:58:51.519722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666950+0000","last_change":"2023-10-16T11:19:23.332619+0000","last_active":"2023-10-16T11:19:36.666950+0000","last_peered":"2023-10-16T11:19:36.666950+0000","last_clean":"2023-10-16T11:19:36.666950+0000","last_became_active":"2023-10-16T11:19:23.332037+0000","last_became_peered":"2023-10-16T11:19:23.332037+0000","last_unstale":"2023-10-16T11:19:36.666950+0000","last_undegraded":"2023-10-16T11:19:36.666950+0000","last_fullsized":"2023-10-16T11:19:36.666950+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:48.332974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.665091+0000","last_change":"2023-10-16T11:19:23.341307+0000","last_active":"2023-10-16T11:19:36.665091+0000","last_peered":"2023-10-16T11:19:36.665091+0000","last_clean":"2023-10-16T11:19:36.665091+0000","last_became_active":"2023-10-16T11:19:23.334457+0000","last_became_peered":"2023-10-16T11:19:23.334457+0000","last_unstale":"2023-10-16T11:19:36.665091+0000","last_undegraded":"2023-10-16T11:19:36.665091+0000","last_fullsized":"2023-10-16T11:19:36.665091+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:45.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697007+0000","last_change":"2023-10-16T11:19:36.694609+0000","last_active":"2023-10-16T11:19:37.697007+0000","last_peered":"2023-10-16T11:19:37.697007+0000","last_clean":"2023-10-16T11:19:37.697007+0000","last_became_active":"2023-10-16T11:19:36.692033+0000","last_became_peered":"2023-10-16T11:19:36.692033+0000","last_unstale":"2023-10-16T11:19:37.697007+0000","last_undegraded":"2023-10-16T11:19:37.697007+0000","last_fullsized":"2023-10-16T11:19:37.697007+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:37.217522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664901+0000","last_change":"2023-10-16T11:19:36.678695+0000","last_active":"2023-10-16T11:19:37.664901+0000","last_peered":"2023-10-16T11:19:37.664901+0000","last_clean":"2023-10-16T11:19:37.664901+0000","last_became_active":"2023-10-16T11:19:36.678422+0000","last_became_peered":"2023-10-16T11:19:36.678422+0000","last_unstale":"2023-10-16T11:19:37.664901+0000","last_undegraded":"2023-10-16T11:19:37.664901+0000","last_fullsized":"2023-10-16T11:19:37.664901+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:08.320255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662807+0000","last_change":"2023-10-16T11:19:23.334306+0000","last_active":"2023-10-16T11:19:36.662807+0000","last_peered":"2023-10-16T11:19:36.662807+0000","last_clean":"2023-10-16T11:19:36.662807+0000","last_became_active":"2023-10-16T11:19:23.333460+0000","last_became_peered":"2023-10-16T11:19:23.333460+0000","last_unstale":"2023-10-16T11:19:36.662807+0000","last_undegraded":"2023-10-16T11:19:36.662807+0000","last_fullsized":"2023-10-16T11:19:36.662807+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:53.175386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994177+0000","last_change":"2023-10-16T11:19:36.687715+0000","last_active":"2023-10-16T11:19:37.994177+0000","last_peered":"2023-10-16T11:19:37.994177+0000","last_clean":"2023-10-16T11:19:37.994177+0000","last_became_active":"2023-10-16T11:19:36.687236+0000","last_became_peered":"2023-10-16T11:19:36.687236+0000","last_unstale":"2023-10-16T11:19:37.994177+0000","last_undegraded":"2023-10-16T11:19:37.994177+0000","last_fullsized":"2023-10-16T11:19:37.994177+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:55.480897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.671643+0000","last_change":"2023-10-16T11:19:23.341264+0000","last_active":"2023-10-16T11:19:36.671643+0000","last_peered":"2023-10-16T11:19:36.671643+0000","last_clean":"2023-10-16T11:19:36.671643+0000","last_became_active":"2023-10-16T11:19:23.340921+0000","last_became_peered":"2023-10-16T11:19:23.340921+0000","last_unstale":"2023-10-16T11:19:36.671643+0000","last_undegraded":"2023-10-16T11:19:36.671643+0000","last_fullsized":"2023-10-16T11:19:36.671643+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.710291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663817+0000","last_change":"2023-10-16T11:19:36.681734+0000","last_active":"2023-10-16T11:19:37.663817+0000","last_peered":"2023-10-16T11:19:37.663817+0000","last_clean":"2023-10-16T11:19:37.663817+0000","last_became_active":"2023-10-16T11:19:36.680840+0000","last_became_peered":"2023-10-16T11:19:36.680840+0000","last_unstale":"2023-10-16T11:19:37.663817+0000","last_undegraded":"2023-10-16T11:19:37.663817+0000","last_fullsized":"2023-10-16T11:19:37.663817+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:50.733066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.679010+0000","last_change":"2023-10-16T11:19:23.341165+0000","last_active":"2023-10-16T11:19:36.679010+0000","last_peered":"2023-10-16T11:19:36.679010+0000","last_clean":"2023-10-16T11:19:36.679010+0000","last_became_active":"2023-10-16T11:19:23.340853+0000","last_became_peered":"2023-10-16T11:19:23.340853+0000","last_unstale":"2023-10-16T11:19:36.679010+0000","last_undegraded":"2023-10-16T11:19:36.679010+0000","last_fullsized":"2023-10-16T11:19:36.679010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:41.539293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700038+0000","last_change":"2023-10-16T11:19:36.676667+0000","last_active":"2023-10-16T11:19:37.700038+0000","last_peered":"2023-10-16T11:19:37.700038+0000","last_clean":"2023-10-16T11:19:37.700038+0000","last_became_active":"2023-10-16T11:19:36.676404+0000","last_became_peered":"2023-10-16T11:19:36.676404+0000","last_unstale":"2023-10-16T11:19:37.700038+0000","last_undegraded":"2023-10-16T11:19:37.700038+0000","last_fullsized":"2023-10-16T11:19:37.700038+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:38.793043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678960+0000","last_change":"2023-10-16T11:19:23.341632+0000","last_active":"2023-10-16T11:19:36.678960+0000","last_peered":"2023-10-16T11:19:36.678960+0000","last_clean":"2023-10-16T11:19:36.678960+0000","last_became_active":"2023-10-16T11:19:23.341312+0000","last_became_peered":"2023-10-16T11:19:23.341312+0000","last_unstale":"2023-10-16T11:19:36.678960+0000","last_undegraded":"2023-10-16T11:19:36.678960+0000","last_fullsized":"2023-10-16T11:19:36.678960+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:33.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.662884+0000","last_change":"2023-10-16T11:19:36.677215+0000","last_active":"2023-10-16T11:19:37.662884+0000","last_peered":"2023-10-16T11:19:37.662884+0000","last_clean":"2023-10-16T11:19:37.662884+0000","last_became_active":"2023-10-16T11:19:36.676952+0000","last_became_peered":"2023-10-16T11:19:36.676952+0000","last_unstale":"2023-10-16T11:19:37.662884+0000","last_undegraded":"2023-10-16T11:19:37.662884+0000","last_fullsized":"2023-10-16T11:19:37.662884+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:41.725192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684353+0000","last_change":"2023-10-16T11:19:23.334105+0000","last_active":"2023-10-16T11:19:36.684353+0000","last_peered":"2023-10-16T11:19:36.684353+0000","last_clean":"2023-10-16T11:19:36.684353+0000","last_became_active":"2023-10-16T11:19:23.333836+0000","last_became_peered":"2023-10-16T11:19:23.333836+0000","last_unstale":"2023-10-16T11:19:36.684353+0000","last_undegraded":"2023-10-16T11:19:36.684353+0000","last_fullsized":"2023-10-16T11:19:36.684353+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.432829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664588+0000","last_change":"2023-10-16T11:19:36.683779+0000","last_active":"2023-10-16T11:19:37.664588+0000","last_peered":"2023-10-16T11:19:37.664588+0000","last_clean":"2023-10-16T11:19:37.664588+0000","last_became_active":"2023-10-16T11:19:36.681105+0000","last_became_peered":"2023-10-16T11:19:36.681105+0000","last_unstale":"2023-10-16T11:19:37.664588+0000","last_undegraded":"2023-10-16T11:19:37.664588+0000","last_fullsized":"2023-10-16T11:19:37.664588+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:08.596894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.684713+0000","last_change":"2023-10-16T11:19:23.331248+0000","last_active":"2023-10-16T11:19:36.684713+0000","last_peered":"2023-10-16T11:19:36.684713+0000","last_clean":"2023-10-16T11:19:36.684713+0000","last_became_active":"2023-10-16T11:19:23.330931+0000","last_became_peered":"2023-10-16T11:19:23.330931+0000","last_unstale":"2023-10-16T11:19:36.684713+0000","last_undegraded":"2023-10-16T11:19:36.684713+0000","last_fullsized":"2023-10-16T11:19:36.684713+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697283+0000","last_change":"2023-10-16T11:19:36.690106+0000","last_active":"2023-10-16T11:19:37.697283+0000","last_peered":"2023-10-16T11:19:37.697283+0000","last_clean":"2023-10-16T11:19:37.697283+0000","last_became_active":"2023-10-16T11:19:36.683066+0000","last_became_peered":"2023-10-16T11:19:36.683066+0000","last_unstale":"2023-10-16T11:19:37.697283+0000","last_undegraded":"2023-10-16T11:19:37.697283+0000","last_fullsized":"2023-10-16T11:19:37.697283+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:06.675860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689742+0000","last_change":"2023-10-16T11:19:23.337766+0000","last_active":"2023-10-16T11:19:36.689742+0000","last_peered":"2023-10-16T11:19:36.689742+0000","last_clean":"2023-10-16T11:19:36.689742+0000","last_became_active":"2023-10-16T11:19:23.337503+0000","last_became_peered":"2023-10-16T11:19:23.337503+0000","last_unstale":"2023-10-16T11:19:36.689742+0000","last_undegraded":"2023-10-16T11:19:36.689742+0000","last_fullsized":"2023-10-16T11:19:36.689742+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:50.949629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995075+0000","last_change":"2023-10-16T11:19:36.685234+0000","last_active":"2023-10-16T11:19:37.995075+0000","last_peered":"2023-10-16T11:19:37.995075+0000","last_clean":"2023-10-16T11:19:37.995075+0000","last_became_active":"2023-10-16T11:19:36.679918+0000","last_became_peered":"2023-10-16T11:19:36.679918+0000","last_unstale":"2023-10-16T11:19:37.995075+0000","last_undegraded":"2023-10-16T11:19:37.995075+0000","last_fullsized":"2023-10-16T11:19:37.995075+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:31.563643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.663100+0000","last_change":"2023-10-16T11:19:23.333633+0000","last_active":"2023-10-16T11:19:36.663100+0000","last_peered":"2023-10-16T11:19:36.663100+0000","last_clean":"2023-10-16T11:19:36.663100+0000","last_became_active":"2023-10-16T11:19:23.333280+0000","last_became_peered":"2023-10-16T11:19:23.333280+0000","last_unstale":"2023-10-16T11:19:36.663100+0000","last_undegraded":"2023-10-16T11:19:36.663100+0000","last_fullsized":"2023-10-16T11:19:36.663100+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:08.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700228+0000","last_change":"2023-10-16T11:19:36.683774+0000","last_active":"2023-10-16T11:19:37.700228+0000","last_peered":"2023-10-16T11:19:37.700228+0000","last_clean":"2023-10-16T11:19:37.700228+0000","last_became_active":"2023-10-16T11:19:36.683334+0000","last_became_peered":"2023-10-16T11:19:36.683334+0000","last_unstale":"2023-10-16T11:19:37.700228+0000","last_undegraded":"2023-10-16T11:19:37.700228+0000","last_fullsized":"2023-10-16T11:19:37.700228+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:58.061612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662748+0000","last_change":"2023-10-16T11:19:23.334483+0000","last_active":"2023-10-16T11:19:36.662748+0000","last_peered":"2023-10-16T11:19:36.662748+0000","last_clean":"2023-10-16T11:19:36.662748+0000","last_became_active":"2023-10-16T11:19:23.333819+0000","last_became_peered":"2023-10-16T11:19:23.333819+0000","last_unstale":"2023-10-16T11:19:36.662748+0000","last_undegraded":"2023-10-16T11:19:36.662748+0000","last_fullsized":"2023-10-16T11:19:36.662748+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:00.142989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.696955+0000","last_change":"2023-10-16T11:19:36.692637+0000","last_active":"2023-10-16T11:19:37.696955+0000","last_peered":"2023-10-16T11:19:37.696955+0000","last_clean":"2023-10-16T11:19:37.696955+0000","last_became_active":"2023-10-16T11:19:36.692099+0000","last_became_peered":"2023-10-16T11:19:36.692099+0000","last_unstale":"2023-10-16T11:19:37.696955+0000","last_undegraded":"2023-10-16T11:19:37.696955+0000","last_fullsized":"2023-10-16T11:19:37.696955+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:34.007256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666671+0000","last_change":"2023-10-16T11:19:23.332953+0000","last_active":"2023-10-16T11:19:36.666671+0000","last_peered":"2023-10-16T11:19:36.666671+0000","last_clean":"2023-10-16T11:19:36.666671+0000","last_became_active":"2023-10-16T11:19:23.332662+0000","last_became_peered":"2023-10-16T11:19:23.332662+0000","last_unstale":"2023-10-16T11:19:36.666671+0000","last_undegraded":"2023-10-16T11:19:36.666671+0000","last_fullsized":"2023-10-16T11:19:36.666671+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:27.611049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994792+0000","last_change":"2023-10-16T11:19:36.679892+0000","last_active":"2023-10-16T11:19:37.994792+0000","last_peered":"2023-10-16T11:19:37.994792+0000","last_clean":"2023-10-16T11:19:37.994792+0000","last_became_active":"2023-10-16T11:19:36.679631+0000","last_became_peered":"2023-10-16T11:19:36.679631+0000","last_unstale":"2023-10-16T11:19:37.994792+0000","last_undegraded":"2023-10-16T11:19:37.994792+0000","last_fullsized":"2023-10-16T11:19:37.994792+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:51.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.668684+0000","last_change":"2023-10-16T11:19:23.325666+0000","last_active":"2023-10-16T11:19:36.668684+0000","last_peered":"2023-10-16T11:19:36.668684+0000","last_clean":"2023-10-16T11:19:36.668684+0000","last_became_active":"2023-10-16T11:19:23.325400+0000","last_became_peered":"2023-10-16T11:19:23.325400+0000","last_unstale":"2023-10-16T11:19:36.668684+0000","last_undegraded":"2023-10-16T11:19:36.668684+0000","last_fullsized":"2023-10-16T11:19:36.668684+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:42.556945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994649+0000","last_change":"2023-10-16T11:19:36.684451+0000","last_active":"2023-10-16T11:19:37.994649+0000","last_peered":"2023-10-16T11:19:37.994649+0000","last_clean":"2023-10-16T11:19:37.994649+0000","last_became_active":"2023-10-16T11:19:36.684000+0000","last_became_peered":"2023-10-16T11:19:36.684000+0000","last_unstale":"2023-10-16T11:19:37.994649+0000","last_undegraded":"2023-10-16T11:19:37.994649+0000","last_fullsized":"2023-10-16T11:19:37.994649+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:43.889855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.679024+0000","last_change":"2023-10-16T11:19:23.342633+0000","last_active":"2023-10-16T11:19:36.679024+0000","last_peered":"2023-10-16T11:19:36.679024+0000","last_clean":"2023-10-16T11:19:36.679024+0000","last_became_active":"2023-10-16T11:19:23.342346+0000","last_became_peered":"2023-10-16T11:19:23.342346+0000","last_unstale":"2023-10-16T11:19:36.679024+0000","last_undegraded":"2023-10-16T11:19:36.679024+0000","last_fullsized":"2023-10-16T11:19:36.679024+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:32:12.087797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994149+0000","last_change":"2023-10-16T11:19:36.684843+0000","last_active":"2023-10-16T11:19:37.994149+0000","last_peered":"2023-10-16T11:19:37.994149+0000","last_clean":"2023-10-16T11:19:37.994149+0000","last_became_active":"2023-10-16T11:19:36.684561+0000","last_became_peered":"2023-10-16T11:19:36.684561+0000","last_unstale":"2023-10-16T11:19:37.994149+0000","last_undegraded":"2023-10-16T11:19:37.994149+0000","last_fullsized":"2023-10-16T11:19:37.994149+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:02.904122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666877+0000","last_change":"2023-10-16T11:19:23.330180+0000","last_active":"2023-10-16T11:19:36.666877+0000","last_peered":"2023-10-16T11:19:36.666877+0000","last_clean":"2023-10-16T11:19:36.666877+0000","last_became_active":"2023-10-16T11:19:23.329598+0000","last_became_peered":"2023-10-16T11:19:23.329598+0000","last_unstale":"2023-10-16T11:19:36.666877+0000","last_undegraded":"2023-10-16T11:19:36.666877+0000","last_fullsized":"2023-10-16T11:19:36.666877+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:59.613377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.665085+0000","last_change":"2023-10-16T11:19:36.687617+0000","last_active":"2023-10-16T11:19:37.665085+0000","last_peered":"2023-10-16T11:19:37.665085+0000","last_clean":"2023-10-16T11:19:37.665085+0000","last_became_active":"2023-10-16T11:19:36.687333+0000","last_became_peered":"2023-10-16T11:19:36.687333+0000","last_unstale":"2023-10-16T11:19:37.665085+0000","last_undegraded":"2023-10-16T11:19:37.665085+0000","last_fullsized":"2023-10-16T11:19:37.665085+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:52.825820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.674910+0000","last_change":"2023-10-16T11:19:23.335413+0000","last_active":"2023-10-16T11:19:36.674910+0000","last_peered":"2023-10-16T11:19:36.674910+0000","last_clean":"2023-10-16T11:19:36.674910+0000","last_became_active":"2023-10-16T11:19:23.334994+0000","last_became_peered":"2023-10-16T11:19:23.334994+0000","last_unstale":"2023-10-16T11:19:36.674910+0000","last_undegraded":"2023-10-16T11:19:36.674910+0000","last_fullsized":"2023-10-16T11:19:36.674910+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:53.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663876+0000","last_change":"2023-10-16T11:19:36.676983+0000","last_active":"2023-10-16T11:19:37.663876+0000","last_peered":"2023-10-16T11:19:37.663876+0000","last_clean":"2023-10-16T11:19:37.663876+0000","last_became_active":"2023-10-16T11:19:36.676668+0000","last_became_peered":"2023-10-16T11:19:36.676668+0000","last_unstale":"2023-10-16T11:19:37.663876+0000","last_undegraded":"2023-10-16T11:19:37.663876+0000","last_fullsized":"2023-10-16T11:19:37.663876+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:25.390302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667223+0000","last_change":"2023-10-16T11:19:23.333705+0000","last_active":"2023-10-16T11:19:36.667223+0000","last_peered":"2023-10-16T11:19:36.667223+0000","last_clean":"2023-10-16T11:19:36.667223+0000","last_became_active":"2023-10-16T11:19:23.333009+0000","last_became_peered":"2023-10-16T11:19:23.333009+0000","last_unstale":"2023-10-16T11:19:36.667223+0000","last_undegraded":"2023-10-16T11:19:36.667223+0000","last_fullsized":"2023-10-16T11:19:36.667223+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:25:41.811226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.697196+0000","last_change":"2023-10-16T11:19:36.695209+0000","last_active":"2023-10-16T11:19:37.697196+0000","last_peered":"2023-10-16T11:19:37.697196+0000","last_clean":"2023-10-16T11:19:37.697196+0000","last_became_active":"2023-10-16T11:19:36.694904+0000","last_became_peered":"2023-10-16T11:19:36.694904+0000","last_unstale":"2023-10-16T11:19:37.697196+0000","last_undegraded":"2023-10-16T11:19:37.697196+0000","last_fullsized":"2023-10-16T11:19:37.697196+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:47:45.793962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.689827+0000","last_change":"2023-10-16T11:19:23.333382+0000","last_active":"2023-10-16T11:19:36.689827+0000","last_peered":"2023-10-16T11:19:36.689827+0000","last_clean":"2023-10-16T11:19:36.689827+0000","last_became_active":"2023-10-16T11:19:23.332643+0000","last_became_peered":"2023-10-16T11:19:23.332643+0000","last_unstale":"2023-10-16T11:19:36.689827+0000","last_undegraded":"2023-10-16T11:19:36.689827+0000","last_fullsized":"2023-10-16T11:19:36.689827+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:54.101849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664649+0000","last_change":"2023-10-16T11:19:36.683067+0000","last_active":"2023-10-16T11:19:37.664649+0000","last_peered":"2023-10-16T11:19:37.664649+0000","last_clean":"2023-10-16T11:19:37.664649+0000","last_became_active":"2023-10-16T11:19:36.682800+0000","last_became_peered":"2023-10-16T11:19:36.682800+0000","last_unstale":"2023-10-16T11:19:37.664649+0000","last_undegraded":"2023-10-16T11:19:37.664649+0000","last_fullsized":"2023-10-16T11:19:37.664649+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:50.857617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.662651+0000","last_change":"2023-10-16T11:19:23.336597+0000","last_active":"2023-10-16T11:19:36.662651+0000","last_peered":"2023-10-16T11:19:36.662651+0000","last_clean":"2023-10-16T11:19:36.662651+0000","last_became_active":"2023-10-16T11:19:23.336307+0000","last_became_peered":"2023-10-16T11:19:23.336307+0000","last_unstale":"2023-10-16T11:19:36.662651+0000","last_undegraded":"2023-10-16T11:19:36.662651+0000","last_fullsized":"2023-10-16T11:19:36.662651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.339536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994870+0000","last_change":"2023-10-16T11:19:36.678840+0000","last_active":"2023-10-16T11:19:37.994870+0000","last_peered":"2023-10-16T11:19:37.994870+0000","last_clean":"2023-10-16T11:19:37.994870+0000","last_became_active":"2023-10-16T11:19:36.678438+0000","last_became_peered":"2023-10-16T11:19:36.678438+0000","last_unstale":"2023-10-16T11:19:37.994870+0000","last_undegraded":"2023-10-16T11:19:37.994870+0000","last_fullsized":"2023-10-16T11:19:37.994870+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:23.839599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.678645+0000","last_change":"2023-10-16T11:19:23.329954+0000","last_active":"2023-10-16T11:19:36.678645+0000","last_peered":"2023-10-16T11:19:36.678645+0000","last_clean":"2023-10-16T11:19:36.678645+0000","last_became_active":"2023-10-16T11:19:23.329688+0000","last_became_peered":"2023-10-16T11:19:23.329688+0000","last_unstale":"2023-10-16T11:19:36.678645+0000","last_undegraded":"2023-10-16T11:19:36.678645+0000","last_fullsized":"2023-10-16T11:19:36.678645+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:11.183201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.699795+0000","last_change":"2023-10-16T11:19:36.676048+0000","last_active":"2023-10-16T11:19:37.699795+0000","last_peered":"2023-10-16T11:19:37.699795+0000","last_clean":"2023-10-16T11:19:37.699795+0000","last_became_active":"2023-10-16T11:19:36.675754+0000","last_became_peered":"2023-10-16T11:19:36.675754+0000","last_unstale":"2023-10-16T11:19:37.699795+0000","last_undegraded":"2023-10-16T11:19:37.699795+0000","last_fullsized":"2023-10-16T11:19:37.699795+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:58.387878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.672717+0000","last_change":"2023-10-16T11:19:23.329203+0000","last_active":"2023-10-16T11:19:36.672717+0000","last_peered":"2023-10-16T11:19:36.672717+0000","last_clean":"2023-10-16T11:19:36.672717+0000","last_became_active":"2023-10-16T11:19:23.328748+0000","last_became_peered":"2023-10-16T11:19:23.328748+0000","last_unstale":"2023-10-16T11:19:36.672717+0000","last_undegraded":"2023-10-16T11:19:36.672717+0000","last_fullsized":"2023-10-16T11:19:36.672717+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:09.942459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.994568+0000","last_change":"2023-10-16T11:19:36.684516+0000","last_active":"2023-10-16T11:19:37.994568+0000","last_peered":"2023-10-16T11:19:37.994568+0000","last_clean":"2023-10-16T11:19:37.994568+0000","last_became_active":"2023-10-16T11:19:36.684256+0000","last_became_peered":"2023-10-16T11:19:36.684256+0000","last_unstale":"2023-10-16T11:19:37.994568+0000","last_undegraded":"2023-10-16T11:19:37.994568+0000","last_fullsized":"2023-10-16T11:19:37.994568+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:12.957525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666433+0000","last_change":"2023-10-16T11:19:23.326663+0000","last_active":"2023-10-16T11:19:36.666433+0000","last_peered":"2023-10-16T11:19:36.666433+0000","last_clean":"2023-10-16T11:19:36.666433+0000","last_became_active":"2023-10-16T11:19:23.326378+0000","last_became_peered":"2023-10-16T11:19:23.326378+0000","last_unstale":"2023-10-16T11:19:36.666433+0000","last_undegraded":"2023-10-16T11:19:36.666433+0000","last_fullsized":"2023-10-16T11:19:36.666433+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:41.823015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.663857+0000","last_change":"2023-10-16T11:19:36.684637+0000","last_active":"2023-10-16T11:19:37.663857+0000","last_peered":"2023-10-16T11:19:37.663857+0000","last_clean":"2023-10-16T11:19:37.663857+0000","last_became_active":"2023-10-16T11:19:36.684358+0000","last_became_peered":"2023-10-16T11:19:36.684358+0000","last_unstale":"2023-10-16T11:19:37.663857+0000","last_undegraded":"2023-10-16T11:19:37.663857+0000","last_fullsized":"2023-10-16T11:19:37.663857+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:29.950550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.676015+0000","last_change":"2023-10-16T11:19:23.330646+0000","last_active":"2023-10-16T11:19:36.676015+0000","last_peered":"2023-10-16T11:19:36.676015+0000","last_clean":"2023-10-16T11:19:36.676015+0000","last_became_active":"2023-10-16T11:19:23.330371+0000","last_became_peered":"2023-10-16T11:19:23.330371+0000","last_unstale":"2023-10-16T11:19:36.676015+0000","last_undegraded":"2023-10-16T11:19:36.676015+0000","last_fullsized":"2023-10-16T11:19:36.676015+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:55.845569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.995300+0000","last_change":"2023-10-16T11:19:36.670851+0000","last_active":"2023-10-16T11:19:37.995300+0000","last_peered":"2023-10-16T11:19:37.995300+0000","last_clean":"2023-10-16T11:19:37.995300+0000","last_became_active":"2023-10-16T11:19:36.670512+0000","last_became_peered":"2023-10-16T11:19:36.670512+0000","last_unstale":"2023-10-16T11:19:37.995300+0000","last_undegraded":"2023-10-16T11:19:37.995300+0000","last_fullsized":"2023-10-16T11:19:37.995300+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:54.831776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666462+0000","last_change":"2023-10-16T11:19:23.335811+0000","last_active":"2023-10-16T11:19:36.666462+0000","last_peered":"2023-10-16T11:19:36.666462+0000","last_clean":"2023-10-16T11:19:36.666462+0000","last_became_active":"2023-10-16T11:19:23.335513+0000","last_became_peered":"2023-10-16T11:19:23.335513+0000","last_unstale":"2023-10-16T11:19:36.666462+0000","last_undegraded":"2023-10-16T11:19:36.666462+0000","last_fullsized":"2023-10-16T11:19:36.666462+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:30.545707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.700233+0000","last_change":"2023-10-16T11:19:36.686689+0000","last_active":"2023-10-16T11:19:37.700233+0000","last_peered":"2023-10-16T11:19:37.700233+0000","last_clean":"2023-10-16T11:19:37.700233+0000","last_became_active":"2023-10-16T11:19:36.686395+0000","last_became_peered":"2023-10-16T11:19:36.686395+0000","last_unstale":"2023-10-16T11:19:37.700233+0000","last_undegraded":"2023-10-16T11:19:37.700233+0000","last_fullsized":"2023-10-16T11:19:37.700233+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:46.501076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.667038+0000","last_change":"2023-10-16T11:19:23.333266+0000","last_active":"2023-10-16T11:19:36.667038+0000","last_peered":"2023-10-16T11:19:36.667038+0000","last_clean":"2023-10-16T11:19:36.667038+0000","last_became_active":"2023-10-16T11:19:23.332412+0000","last_became_peered":"2023-10-16T11:19:23.332412+0000","last_unstale":"2023-10-16T11:19:36.667038+0000","last_undegraded":"2023-10-16T11:19:36.667038+0000","last_fullsized":"2023-10-16T11:19:36.667038+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:19.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664344+0000","last_change":"2023-10-16T11:19:36.687300+0000","last_active":"2023-10-16T11:19:37.664344+0000","last_peered":"2023-10-16T11:19:37.664344+0000","last_clean":"2023-10-16T11:19:37.664344+0000","last_became_active":"2023-10-16T11:19:36.687041+0000","last_became_peered":"2023-10-16T11:19:36.687041+0000","last_unstale":"2023-10-16T11:19:37.664344+0000","last_undegraded":"2023-10-16T11:19:37.664344+0000","last_fullsized":"2023-10-16T11:19:37.664344+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:12.863750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.681800+0000","last_change":"2023-10-16T11:19:23.332069+0000","last_active":"2023-10-16T11:19:36.681800+0000","last_peered":"2023-10-16T11:19:36.681800+0000","last_clean":"2023-10-16T11:19:36.681800+0000","last_became_active":"2023-10-16T11:19:23.331671+0000","last_became_peered":"2023-10-16T11:19:23.331671+0000","last_unstale":"2023-10-16T11:19:36.681800+0000","last_undegraded":"2023-10-16T11:19:36.681800+0000","last_fullsized":"2023-10-16T11:19:36.681800+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:13.976802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":11,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-16T11:19:37.664234+0000","last_change":"2023-10-16T11:19:36.682589+0000","last_active":"2023-10-16T11:19:37.664234+0000","last_peered":"2023-10-16T11:19:37.664234+0000","last_clean":"2023-10-16T11:19:37.664234+0000","last_became_active":"2023-10-16T11:19:36.682311+0000","last_became_peered":"2023-10-16T11:19:36.682311+0000","last_unstale":"2023-10-16T11:19:37.664234+0000","last_undegraded":"2023-10-16T11:19:37.664234+0000","last_fullsized":"2023-10-16T11:19:37.664234+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:35.651650+0000","last_clean_scrub_stamp":"2023-10-16T11:19:35.651650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:45:52.728822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-10-16T11:19:36.666733+0000","last_change":"2023-10-16T11:19:23.334441+0000","last_active":"2023-10-16T11:19:36.666733+0000","last_peered":"2023-10-16T11:19:36.666733+0000","last_clean":"2023-10-16T11:19:36.666733+0000","last_became_active":"2023-10-16T11:19:23.334140+0000","last_became_peered":"2023-10-16T11:19:23.334140+0000","last_unstale":"2023-10-16T11:19:36.666733+0000","last_undegraded":"2023-10-16T11:19:36.666733+0000","last_fullsized":"2023-10-16T11:19:36.666733+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:19:22.307502+0000","last_clean_scrub_stamp":"2023-10-16T11:19:22.307502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:42.327577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3395584,"data_stored":2085172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.67200000000000004,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.372},"max":{"1min":9.4359999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.68400000000000005,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":9.3059999999999992,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.71599999999999997,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":9.3610000000000007,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.621},{"interface":"front","average":{"1min":1.119,"5min":0.71999999999999997,"15min":0.68100000000000005},"min":{"1min":0.49399999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.3960000000000008,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.72099999999999997,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":9.2889999999999997,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.753,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.36099999999999999},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.157,"5min":0.76500000000000001,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":9.4689999999999994,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.63700000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.61199999999999999,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.63100000000000001,"15min":0.56699999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.5399999999999991,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.47599999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.64000000000000001,"15min":0.57399999999999995},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.69399999999999995,"15min":0.63100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.5690000000000008,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.71199999999999997,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.155,"5min":0.69799999999999995,"15min":0.64500000000000002},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.442}]}]},{"osd":6,"up_from":13,"seq":55834575993,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38096,"kb_used_data":2752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333744,"statfs":{"total":96636764160,"available":96597753856,"internally_reserved":0,"allocated":2818048,"data_stored":1496382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.91000000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.499},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.52300000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.70599999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64000000000000001,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.71599999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.335,"5min":3.335,"15min":3.335},"last":0.621}]},{"osd":7,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.752}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47788,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324052,"statfs":{"total":96636764160,"available":96587829248,"internally_reserved":0,"allocated":2777088,"data_stored":1491203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.622,"15min":0.66700000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.436,"15min":0.45200000000000001},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.52500000000000002,"5min":0.63900000000000001,"15min":0.71699999999999997},"last":0.376}]},{"osd":2,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.64000000000000001,"15min":0.70099999999999996},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.442},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":0.623,"15min":0.70499999999999996},"last":0.33000000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.48099999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65400000000000003,"15min":0.67200000000000004},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":0.89500000000000002,"15min":1.0149999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90000000000000002,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.90300000000000002,"15min":1.0620000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.016},"last":0.52000000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575994,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42796,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329044,"statfs":{"total":96636764160,"available":96592941056,"internally_reserved":0,"allocated":3375104,"data_stored":2081661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.318}]},{"osd":2,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44400000000000001,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60699999999999998,"5min":1.272,"15min":1.272},"last":0.379},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.83199999999999996},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.254},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67700000000000005,"5min":1.351,"15min":1.351},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.46800000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.875,"5min":1.391,"15min":1.391},"last":0.71999999999999997}]},{"osd":5,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80000000000000004,"5min":0.874,"15min":1.089},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.95599999999999996,"15min":1.216},"last":0.66600000000000004}]},{"osd":6,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.69799999999999995},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90900000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.47199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89100000000000001,"5min":1.038,"15min":1.1180000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94599999999999995,"15min":1.0780000000000001},"last":0.70199999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575993,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2789376,"data_stored":1491909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.308},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.63300000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.29099999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.255}]},{"osd":3,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69799999999999995,"5min":0.72099999999999997,"15min":0.73499999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77800000000000002},"last":0.37}]},{"osd":4,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.95399999999999996,"15min":1.097},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67200000000000004,"15min":0.68200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.91800000000000004,"5min":0.96199999999999997,"15min":1.016},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.748},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90500000000000003,"5min":1.022,"15min":1.0389999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91800000000000004,"5min":1.022,"15min":1.022},"last":0.62},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91900000000000004,"5min":0.98999999999999999,"15min":1.004},"last":0.79900000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90400000000000003,"5min":1.0569999999999999,"15min":1.071},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97799999999999998,"15min":1.077},"last":0.79100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575993,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51956,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319884,"statfs":{"total":96636764160,"available":96583561216,"internally_reserved":0,"allocated":2793472,"data_stored":1491351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38100000000000001,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.68700000000000006},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":1.0029999999999999},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.60899999999999999,"15min":0.67300000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.74299999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.58999999999999997,"5min":0.66800000000000004,"15min":0.96299999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64200000000000002,"5min":0.69899999999999995,"15min":1.024},"last":0.35399999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90600000000000003,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.501},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"last":0.57699999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.99099999999999999,"15min":1.026},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0640000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41799999999999998},"max":{"1min":0.85999999999999999,"5min":0.92900000000000005,"15min":1.0369999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.0620000000000001},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":1.002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88400000000000001,"5min":1.006,"15min":1.1899999999999999},"last":0.628}]}]},{"osd":4,"up_from":13,"seq":55834575993,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47792,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324048,"statfs":{"total":96636764160,"available":96587825152,"internally_reserved":0,"allocated":2785280,"data_stored":1491875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.625},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.623},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71599999999999997,"5min":0.82499999999999996,"15min":0.90700000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58799999999999997,"15min":0.623},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60599999999999998,"15min":0.63900000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.84399999999999997,"15min":0.88400000000000001},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.624},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.83799999999999997,"15min":0.995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.629,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.51500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.73199999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.85799999999999998,"15min":0.86199999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.63400000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.90500000000000003,"15min":1.115},"last":0.745}]},{"osd":7,"last update":"Mon Oct 16 11:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.35499999999999998,"15min":0.34200000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.83899999999999997},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.93200000000000005},"last":0.79000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575993,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82492,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94289348,"statfs":{"total":96636764160,"available":96552292352,"internally_reserved":0,"allocated":2801664,"data_stored":1492941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.88600000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60299999999999998},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.67600000000000005}]},{"osd":2,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.86499999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.91400000000000003},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40899999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.79500000000000004}]},{"osd":6,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.73399999999999999,"15min":0.82499999999999996},"last":0.75},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85899999999999999,"5min":0.878,"15min":0.878},"last":0.84499999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55300000000000005,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.78300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:19:45.783 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:19:45.783 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:19:45.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.931+0000 7fdc75f9f700 1 Processor -- start 2023-10-16T11:19:45.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.931+0000 7fdc75f9f700 1 -- start start 2023-10-16T11:19:45.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.932+0000 7fdc75f9f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc7015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.932+0000 7fdc75f9f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7015c5d0 con 0x7fdc700b6e10 2023-10-16T11:19:45.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.932+0000 7fdc75f9f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7015c8b0 con 0x7fdc700ce2c0 2023-10-16T11:19:45.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.932+0000 7fdc75f9f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7015d2c0 con 0x7fdc700cdee0 2023-10-16T11:19:45.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.932+0000 7fdc6effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdc700b6e10 0x7fdc701521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38694/0 (socket says 172.21.15.158:38694) 2023-10-16T11:19:45.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.932+0000 7fdc6effd700 1 -- 172.21.15.158:0/82787373 learned_addr learned my addr 172.21.15.158:0/82787373 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:45.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.932+0000 7fdc6f7fe700 1 --2- 172.21.15.158:0/82787373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc7015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895589120 0 0) 0x7fdc7015c5d0 con 0x7fdc700b6e10 2023-10-16T11:19:45.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc580035b0 con 0x7fdc700b6e10 2023-10-16T11:19:45.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105255421 0 0) 0x7fdc7015c8b0 con 0x7fdc700ce2c0 2023-10-16T11:19:45.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc7015c5d0 con 0x7fdc700ce2c0 2023-10-16T11:19:45.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 880415393 0 0) 0x7fdc580035b0 con 0x7fdc700b6e10 2023-10-16T11:19:45.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6f7fe700 1 -- 172.21.15.158:0/82787373 >> v1:172.21.15.172:6789/0 conn(0x7fdc700ce2c0 legacy=0x7fdc700cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6f7fe700 1 -- 172.21.15.158:0/82787373 >> v1:172.21.15.158:6789/0 conn(0x7fdc700b6e10 legacy=0x7fdc701521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6f7fe700 1 -- 172.21.15.158:0/82787373 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7015cb90 con 0x7fdc700cdee0 2023-10-16T11:19:45.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6f7fe700 1 --2- 172.21.15.158:0/82787373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc7015bd10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009870 tx=0x7fdc64009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a61a2e97696e04c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:45.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60001670 con 0x7fdc700cdee0 2023-10-16T11:19:45.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc64004030 con 0x7fdc700cdee0 2023-10-16T11:19:45.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.933+0000 7fdc6e7fc700 1 -- 172.21.15.158:0/82787373 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc640043c0 con 0x7fdc700cdee0 2023-10-16T11:19:45.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.934+0000 7fdc75f9f700 1 -- 172.21.15.158:0/82787373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 msgr2=0x7fdc7015bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.934+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/82787373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc7015bd10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009870 tx=0x7fdc64009b80 comp rx=0 tx=0).stop 2023-10-16T11:19:45.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.934+0000 7fdc75f9f700 1 -- 172.21.15.158:0/82787373 shutdown_connections 2023-10-16T11:19:45.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.934+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/82787373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc7015bd10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.934+0000 7fdc75f9f700 1 -- 172.21.15.158:0/82787373 >> 172.21.15.158:0/82787373 conn(0x7fdc7006a8d0 msgr2=0x7fdc7006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:45.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.934+0000 7fdc75f9f700 1 -- 172.21.15.158:0/82787373 shutdown_connections 2023-10-16T11:19:45.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.934+0000 7fdc75f9f700 1 -- 172.21.15.158:0/82787373 wait complete. 2023-10-16T11:19:45.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.935+0000 7fdc75f9f700 1 Processor -- start 2023-10-16T11:19:45.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.935+0000 7fdc75f9f700 1 -- start start 2023-10-16T11:19:45.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc75f9f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc700b6e10 0x7fdc7014e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc75f9f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc700c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc75f9f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc700ce2c0 0x7fdc700c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc75f9f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc700c4570 con 0x7fdc700ce2c0 2023-10-16T11:19:45.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc75f9f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc700c6950 con 0x7fdc700b6e10 2023-10-16T11:19:45.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc75f9f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc700cfd30 con 0x7fdc700cdee0 2023-10-16T11:19:45.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc700ce2c0 0x7fdc700c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc700ce2c0 0x7fdc700c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38526/0 (socket says 172.21.15.158:38526) 2023-10-16T11:19:45.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 -- 172.21.15.158:0/2444125778 learned_addr learned my addr 172.21.15.158:0/2444125778 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:45.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6effd700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc700c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6f7fe700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc700b6e10 0x7fdc7014e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 -- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 msgr2=0x7fdc700c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc700c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 -- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc700b6e10 msgr2=0x7fdc7014e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:45.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc700b6e10 0x7fdc7014e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:45.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.936+0000 7fdc6ffff700 1 -- 172.21.15.158:0/2444125778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc640094a0 con 0x7fdc700ce2c0 2023-10-16T11:19:45.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.937+0000 7fdc6ffff700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc700ce2c0 0x7fdc700c6410 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800cbf0 tx=0x7fdc6800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:45.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.937+0000 7fdc57fff700 1 -- 172.21.15.158:0/2444125778 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68003c00 con 0x7fdc700ce2c0 2023-10-16T11:19:45.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.937+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc700cf120 con 0x7fdc700ce2c0 2023-10-16T11:19:45.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.937+0000 7fdc57fff700 1 -- 172.21.15.158:0/2444125778 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc68003d60 con 0x7fdc700ce2c0 2023-10-16T11:19:45.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.937+0000 7fdc57fff700 1 -- 172.21.15.158:0/2444125778 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6800aa60 con 0x7fdc700ce2c0 2023-10-16T11:19:45.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.937+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc700cf6e0 con 0x7fdc700ce2c0 2023-10-16T11:19:45.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.938+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc700cfec0 con 0x7fdc700ce2c0 2023-10-16T11:19:45.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.938+0000 7fdc57fff700 1 -- 172.21.15.158:0/2444125778 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdc6800abc0 con 0x7fdc700ce2c0 2023-10-16T11:19:45.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.939+0000 7fdc57fff700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdc5c06b020 0x7fdc5c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:45.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.939+0000 7fdc57fff700 1 -- 172.21.15.158:0/2444125778 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fdc68089e00 con 0x7fdc700ce2c0 2023-10-16T11:19:45.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.939+0000 7fdc6f7fe700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdc5c06b020 0x7fdc5c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:45.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.939+0000 7fdc6f7fe700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdc5c06b020 0x7fdc5c06d4e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc640097c0 tx=0x7fdc64008030 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:45.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:45.943+0000 7fdc57fff700 1 -- 172.21.15.158:0/2444125778 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc680546a0 con 0x7fdc700ce2c0 2023-10-16T11:19:46.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.178+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc700681b0 con 0x7fdc700ce2c0 2023-10-16T11:19:46.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.178+0000 7fdc57fff700 1 -- 172.21.15.158:0/2444125778 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6163 (secure 0 0 0) 0x7fdc68057cc0 con 0x7fdc700ce2c0 2023-10-16T11:19:46.180 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:46.181 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:19:46.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.180+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdc5c06b020 msgr2=0x7fdc5c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.180+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdc5c06b020 0x7fdc5c06d4e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc640097c0 tx=0x7fdc64008030 comp rx=0 tx=0).stop 2023-10-16T11:19:46.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.180+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc700ce2c0 msgr2=0x7fdc700c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.180+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc700ce2c0 0x7fdc700c6410 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800cbf0 tx=0x7fdc6800cf00 comp rx=0 tx=0).stop 2023-10-16T11:19:46.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 shutdown_connections 2023-10-16T11:19:46.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdc5c06b020 0x7fdc5c06d4e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc700ce2c0 0x7fdc700c6410 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc700b6e10 0x7fdc7014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 --2- 172.21.15.158:0/2444125778 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc700cdee0 0x7fdc700c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 >> 172.21.15.158:0/2444125778 conn(0x7fdc7006a8d0 msgr2=0x7fdc70154350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 shutdown_connections 2023-10-16T11:19:46.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.181+0000 7fdc75f9f700 1 -- 172.21.15.158:0/2444125778 wait complete. 2023-10-16T11:19:46.185 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 118 2023-10-16T11:19:46.408 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:19:46.411 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:19:46.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 118, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:19:36.061742+0000', 'modified': '2023-10-16T11:19:39.089874+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11586}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11586': {'gid': 11586, 'name': 'd', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.172:6835/2771711373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 2771711373}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 2771711373}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:19:46.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:19:46.598 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T11:19:46.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.719+0000 7ff0bb9fe700 1 Processor -- start 2023-10-16T11:19:46.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.719+0000 7ff0bb9fe700 1 -- start start 2023-10-16T11:19:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0bb9fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 0x7ff0b406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0bb9fe700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b40cb2d0 con 0x7ff0b4158280 2023-10-16T11:19:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0bb9fe700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b4158660 con 0x7ff0b40cf180 2023-10-16T11:19:46.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0bb9fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b415bc40 con 0x7ff0b40722e0 2023-10-16T11:19:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0b979a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff0b4158280 0x7ff0b40cea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38706/0 (socket says 172.21.15.158:38706) 2023-10-16T11:19:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0b979a700 1 -- 172.21.15.158:0/268475372 learned_addr learned my addr 172.21.15.158:0/268475372 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 0x7ff0b406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277036034 0 0) 0x7ff0b40cb2d0 con 0x7ff0b4158280 2023-10-16T11:19:46.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a00035b0 con 0x7ff0b4158280 2023-10-16T11:19:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142141941 0 0) 0x7ff0b4158660 con 0x7ff0b40cf180 2023-10-16T11:19:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b40cb2d0 con 0x7ff0b40cf180 2023-10-16T11:19:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2401830481 0 0) 0x7ff0a00035b0 con 0x7ff0b4158280 2023-10-16T11:19:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 msgr2=0x7ff0b406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 0x7ff0b406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 >> v1:172.21.15.172:6789/0 conn(0x7ff0b40cf180 legacy=0x7ff0b40cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b415b030 con 0x7ff0b4158280 2023-10-16T11:19:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff0a4001670 con 0x7ff0b4158280 2023-10-16T11:19:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.720+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 >> v1:172.21.15.158:6789/0 conn(0x7ff0b4158280 legacy=0x7ff0b40cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a0003bc0 0x7ff0a0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0a0004850 0x7ff0a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 0x7ff0a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b415bc40 con 0x7ff0a0003bc0 2023-10-16T11:19:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a000bac0 con 0x7ff0a0004850 2023-10-16T11:19:46.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a000bc50 con 0x7ff0b40722e0 2023-10-16T11:19:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b979a700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a0003bc0 0x7ff0a0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0a0004850 0x7ff0a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 0x7ff0a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b979a700 1 -- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 msgr2=0x7ff0a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b979a700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b40722e0 0x7ff0a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b979a700 1 -- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0a0004850 msgr2=0x7ff0a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b979a700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0a0004850 0x7ff0a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b979a700 1 -- 172.21.15.158:0/268475372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00035b0 con 0x7ff0a0003bc0 2023-10-16T11:19:46.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0b979a700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a0003bc0 0x7ff0a0003fc0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400c210 tx=0x7ff0a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263402cfa87d81c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400cd10 con 0x7ff0a0003bc0 2023-10-16T11:19:46.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a400ce90 con 0x7ff0a0003bc0 2023-10-16T11:19:46.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.721+0000 7ff0abfff700 1 -- 172.21.15.158:0/268475372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4005ba0 con 0x7ff0a0003bc0 2023-10-16T11:19:46.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.722+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/268475372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a0003bc0 msgr2=0x7ff0a0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.722+0000 7ff0bb9fe700 1 --2- 172.21.15.158:0/268475372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a0003bc0 0x7ff0a0003fc0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400c210 tx=0x7ff0a0005880 comp rx=0 tx=0).stop 2023-10-16T11:19:46.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.722+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/268475372 reap_dead start 2023-10-16T11:19:46.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.722+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/268475372 shutdown_connections 2023-10-16T11:19:46.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.722+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/268475372 >> 172.21.15.158:0/268475372 conn(0x7ff0b40cc700 msgr2=0x7ff0b40ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.722+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/268475372 shutdown_connections 2023-10-16T11:19:46.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.722+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/268475372 wait complete. 2023-10-16T11:19:46.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.723+0000 7ff0bb9fe700 1 Processor -- start 2023-10-16T11:19:46.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.723+0000 7ff0bb9fe700 1 -- start start 2023-10-16T11:19:46.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0bb9fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0a0003bc0 0x7ff0b4154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0bb9fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b4153f50 0x7ff0b41e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0bb9fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0b41e5710 0x7ff0b41526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0bb9fe700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b41e5bc0 con 0x7ff0b41e5710 2023-10-16T11:19:46.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0bb9fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b415c830 con 0x7ff0b4153f50 2023-10-16T11:19:46.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0bb9fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b415cb10 con 0x7ff0a0003bc0 2023-10-16T11:19:46.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0b41e5710 0x7ff0b41526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0b41e5710 0x7ff0b41526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38546/0 (socket says 172.21.15.158:38546) 2023-10-16T11:19:46.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/3018155512 learned_addr learned my addr 172.21.15.158:0/3018155512 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:46.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b979a700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0a0003bc0 0x7ff0b4154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b4153f50 0x7ff0b41e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0a0003bc0 msgr2=0x7ff0b4154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0a0003bc0 0x7ff0b4154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b4153f50 msgr2=0x7ff0b41e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b4153f50 0x7ff0b41e51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/3018155512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b415d980 con 0x7ff0b41e5710 2023-10-16T11:19:46.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b4153f50 0x7ff0b41e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:46.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0b41e5710 0x7ff0b41526d0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4071180 tx=0x7ff0ac00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0a9ffb700 1 -- 172.21.15.158:0/3018155512 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00e040 con 0x7ff0b41e5710 2023-10-16T11:19:46.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.724+0000 7ff0a9ffb700 1 -- 172.21.15.158:0/3018155512 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00f040 con 0x7ff0b41e5710 2023-10-16T11:19:46.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.725+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b415d580 con 0x7ff0b41e5710 2023-10-16T11:19:46.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.725+0000 7ff0a9ffb700 1 -- 172.21.15.158:0/3018155512 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00fc30 con 0x7ff0b41e5710 2023-10-16T11:19:46.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.725+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b415d130 con 0x7ff0b41e5710 2023-10-16T11:19:46.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.726+0000 7ff0a9ffb700 1 -- 172.21.15.158:0/3018155512 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff0ac00fd90 con 0x7ff0b41e5710 2023-10-16T11:19:46.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.726+0000 7ff0a9ffb700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff09806b050 0x7ff09806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.727+0000 7ff0a9ffb700 1 -- 172.21.15.158:0/3018155512 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7ff0ac08a5f0 con 0x7ff0b41e5710 2023-10-16T11:19:46.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.727+0000 7ff0b979a700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff09806b050 0x7ff09806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.727+0000 7ff0b979a700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff09806b050 0x7ff09806d510 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4015040 tx=0x7ff0a4007f90 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.729+0000 7ff0c7be8b80 1 Processor -- start 2023-10-16T11:19:46.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.729+0000 7ff0c7be8b80 1 -- start start 2023-10-16T11:19:46.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636c87a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c87fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636c5b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636bd1c20 con 0x561636c5b010 2023-10-16T11:19:46.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636bb6c10 con 0x561636c85630 2023-10-16T11:19:46.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x561636bb4530 con 0x561636c80220 2023-10-16T11:19:46.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636c5b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636c5b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38554/0 (socket says 172.21.15.158:38554) 2023-10-16T11:19:46.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/163038125 learned_addr learned my addr 172.21.15.158:0/163038125 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:46.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b979a700 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636c87a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c87fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 msgr2=0x561636c87a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636c87a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 msgr2=0x561636c87fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c87fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/163038125 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561636b6d3a0 con 0x561636c5b010 2023-10-16T11:19:46.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b979a700 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636c87a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:46.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c87fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:46.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636c5b6c0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac088300 tx=0x7ff0ac003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9431b224483635 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff09effd700 1 -- 172.21.15.158:0/163038125 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac05cca0 con 0x561636c5b010 2023-10-16T11:19:46.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff09effd700 1 -- 172.21.15.158:0/163038125 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac060e50 con 0x561636c5b010 2023-10-16T11:19:46.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.730+0000 7ff09effd700 1 -- 172.21.15.158:0/163038125 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac058ff0 con 0x561636c5b010 2023-10-16T11:19:46.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/163038125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 msgr2=0x561636c5b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636c5b6c0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac088300 tx=0x7ff0ac003f00 comp rx=0 tx=0).stop 2023-10-16T11:19:46.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/163038125 shutdown_connections 2023-10-16T11:19:46.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636c5b6c0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636c87a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/163038125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c87fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/163038125 >> 172.21.15.158:0/163038125 conn(0x561636b8ad80 msgr2=0x561636ca5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/163038125 shutdown_connections 2023-10-16T11:19:46.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/163038125 wait complete. 2023-10-16T11:19:46.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.731+0000 7ff0c7be8b80 1 -- start start 2023-10-16T11:19:46.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.732+0000 7ff0c7be8b80 1 Processor -- start 2023-10-16T11:19:46.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.732+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c5b010 0x561636c87440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c80220 0x561636c80600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c8ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636bafe20 con 0x561636c80220 2023-10-16T11:19:46.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636b4c6c0 con 0x561636c85630 2023-10-16T11:19:46.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x561636b4c360 con 0x561636c5b010 2023-10-16T11:19:46.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 10 client.0 _ll_register_callbacks cb 0x5616369a97b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:19:46.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c80220 0x561636c80600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c80220 0x561636c80600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38558/0 (socket says 172.21.15.158:38558) 2023-10-16T11:19:46.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 -- 172.21.15.158:0/2126800376 learned_addr learned my addr 172.21.15.158:0/2126800376 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:46.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b979a700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c5b010 0x561636c87440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c8ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 -- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c5b010 msgr2=0x561636c87440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c5b010 0x561636c87440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 -- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 msgr2=0x561636c8ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c8ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 -- 172.21.15.158:0/2126800376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b415d130 con 0x561636c80220 2023-10-16T11:19:46.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c8ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:46.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c80220 0x561636c80600 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7ff0b41e5eb0 tx=0x7ff0b0003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3a4f03c94a42ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff093fff700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:19:46.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff093fff700 1 -- 172.21.15.158:0/2126800376 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0001b70 con 0x561636c80220 2023-10-16T11:19:46.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff093fff700 1 -- 172.21.15.158:0/2126800376 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7ff0b0001cd0 con 0x561636c80220 2023-10-16T11:19:46.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561636c505b0 con 0x561636c80220 2023-10-16T11:19:46.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff093fff700 1 -- 172.21.15.158:0/2126800376 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0b0016470 con 0x561636c80220 2023-10-16T11:19:46.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 10 client.11877 fetch_fsmap learned FSMap version 118 2023-10-16T11:19:46.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.733+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) v3 -- 0x5616369dad90 con 0x561636c80220 2023-10-16T11:19:46.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff093fff700 1 -- 172.21.15.158:0/2126800376 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561636c505b0 con 0x561636c80220 2023-10-16T11:19:46.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff093fff700 1 -- 172.21.15.158:0/2126800376 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 118) v1 ==== 3891+0+0 (secure 0 0 0) 0x7ff0b00165d0 con 0x561636c80220 2023-10-16T11:19:46.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0c7be8b80 10 client.11877 fetch_fsmap finished waiting for FSMap version 118 2023-10-16T11:19:46.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0c7be8b80 10 client.11877 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:19:46.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0c7be8b80 20 client.11877 populate_metadata read hostname 'smithi158' 2023-10-16T11:19:46.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636cb23f0 0x561636cb27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0c7be8b80 4 client.11877 mds_command: new command op to 11586 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:19:46.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 --> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561636ba8e00 con 0x561636cb23f0 2023-10-16T11:19:46.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0b979a700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636cb23f0 0x561636cb27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff093fff700 10 client.11877 ms_handle_connect on v2:172.21.15.172:6833/2771711373 2023-10-16T11:19:46.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.734+0000 7ff0b979a700 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636cb23f0 0x561636cb27d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4abc5895fbc7b09e server_cookie=d1ea560b97c829a2 in_seq=0 out_seq=0 2023-10-16T11:19:46.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff093fff700 1 -- 172.21.15.158:0/2126800376 <== mds.0 v2:172.21.15.172:6833/2771711373 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561636ba8e00 con 0x561636cb23f0 2023-10-16T11:19:46.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff093fff700 10 client.11877 handle_command_reply: tid=0 2023-10-16T11:19:46.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 client.11877 shutdown 2023-10-16T11:19:46.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c80220 msgr2=0x561636c80600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c80220 0x561636c80600 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7ff0b41e5eb0 tx=0x7ff0b0003e50 comp rx=0 tx=0).stop 2023-10-16T11:19:46.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 shutdown_connections 2023-10-16T11:19:46.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636cb23f0 0x561636cb27d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c85630 0x561636c8ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c5b010 0x561636c87440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff093fff700 0 client.11877 ms_handle_reset on v2:172.21.15.172:6833/2771711373 2023-10-16T11:19:46.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2126800376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c80220 0x561636c80600 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 >> 172.21.15.158:0/2126800376 conn(0x561636b8ad80 msgr2=0x561636ca34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 shutdown_connections 2023-10-16T11:19:46.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.735+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2126800376 wait complete. 2023-10-16T11:19:46.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.736+0000 7ff0c7be8b80 20 client.11877 trim_cache size 0 max 16384 2023-10-16T11:19:46.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.753+0000 7ff0c7be8b80 1 Processor -- start 2023-10-16T11:19:46.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.753+0000 7ff0c7be8b80 1 -- start start 2023-10-16T11:19:46.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636cb2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c80220 0x561636c88f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c85630 0x561636c87ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636a308d0 con 0x561636c5b010 2023-10-16T11:19:46.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636cb5500 con 0x561636c80220 2023-10-16T11:19:46.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x561636c84b10 con 0x561636c85630 2023-10-16T11:19:46.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636cb2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636cb2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38564/0 (socket says 172.21.15.158:38564) 2023-10-16T11:19:46.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 -- 172.21.15.158:0/134777285 learned_addr learned my addr 172.21.15.158:0/134777285 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:46.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c80220 0x561636c88f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c85630 0x561636c87ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 -- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c85630 msgr2=0x561636c87ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c85630 0x561636c87ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 -- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c80220 msgr2=0x561636c88f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c80220 0x561636c88f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 -- 172.21.15.158:0/134777285 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616369c3c10 con 0x561636c5b010 2023-10-16T11:19:46.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b8f99700 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c80220 0x561636c88f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:46.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c85630 0x561636c87ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:46.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff0b979a700 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636cb2910 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4007040 tx=0x7ff0a4009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a83dab08306db8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff09f7fe700 1 -- 172.21.15.158:0/134777285 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4030040 con 0x561636c5b010 2023-10-16T11:19:46.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff09f7fe700 1 -- 172.21.15.158:0/134777285 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a400ccf0 con 0x561636c5b010 2023-10-16T11:19:46.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.754+0000 7ff09f7fe700 1 -- 172.21.15.158:0/134777285 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4033530 con 0x561636c5b010 2023-10-16T11:19:46.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/134777285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 msgr2=0x561636cb2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636cb2910 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4007040 tx=0x7ff0a4009050 comp rx=0 tx=0).stop 2023-10-16T11:19:46.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/134777285 shutdown_connections 2023-10-16T11:19:46.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c85630 0x561636c87ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c5b010 0x561636cb2910 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/134777285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c80220 0x561636c88f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/134777285 >> 172.21.15.158:0/134777285 conn(0x561636b8ad80 msgr2=0x561636c5d400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/134777285 shutdown_connections 2023-10-16T11:19:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.755+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/134777285 wait complete. 2023-10-16T11:19:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.756+0000 7ff0c7be8b80 1 -- start start 2023-10-16T11:19:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.756+0000 7ff0c7be8b80 1 Processor -- start 2023-10-16T11:19:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.756+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c5b010 0x561636d0d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.756+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636cb1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.756+0000 7ff0c7be8b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c85630 0x561636cb1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.756+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636c895a0 con 0x561636c85630 2023-10-16T11:19:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x561636d0df90 con 0x561636c5b010 2023-10-16T11:19:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0c7be8b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x561636c85e60 con 0x561636c80220 2023-10-16T11:19:46.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0c7be8b80 10 client.0 _ll_register_callbacks cb 0x5616369a97b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:19:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c85630 0x561636cb1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c85630 0x561636cb1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38574/0 (socket says 172.21.15.158:38574) 2023-10-16T11:19:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b8f99700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636cb1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/2745605759 learned_addr learned my addr 172.21.15.158:0/2745605759 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b979a700 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c5b010 0x561636d0d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 msgr2=0x561636cb1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636cb1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c5b010 msgr2=0x561636d0d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c5b010 0x561636d0d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 -- 172.21.15.158:0/2745605759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a00035b0 con 0x561636c85630 2023-10-16T11:19:46.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0b9f9b700 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c85630 0x561636cb1660 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac088350 tx=0x7ff0ac021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c7151ecb5c66f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff093fff700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:19:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff093fff700 1 -- 172.21.15.158:0/2745605759 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac058fb0 con 0x561636c85630 2023-10-16T11:19:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff093fff700 1 -- 172.21.15.158:0/2745605759 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7ff0ac05cca0 con 0x561636c85630 2023-10-16T11:19:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0ac060e50 con 0x561636c85630 2023-10-16T11:19:46.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff093fff700 1 -- 172.21.15.158:0/2745605759 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0ac0609c0 con 0x561636c85630 2023-10-16T11:19:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0c7be8b80 10 client.11883 fetch_fsmap learned FSMap version 118 2023-10-16T11:19:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) v3 -- 0x561636a01200 con 0x561636c85630 2023-10-16T11:19:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.757+0000 7ff093fff700 1 -- 172.21.15.158:0/2745605759 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac060e50 con 0x561636c85630 2023-10-16T11:19:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff093fff700 1 -- 172.21.15.158:0/2745605759 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 118) v1 ==== 3891+0+0 (secure 0 0 0) 0x7ff0ac070130 con 0x561636c85630 2023-10-16T11:19:46.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0c7be8b80 10 client.11883 fetch_fsmap finished waiting for FSMap version 118 2023-10-16T11:19:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0c7be8b80 10 client.11883 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:19:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0c7be8b80 20 client.11883 populate_metadata read hostname 'smithi158' 2023-10-16T11:19:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636c862f0 0x561636c866d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0c7be8b80 4 client.11883 mds_command: new command op to 11586 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:19:46.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 --> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561636ba8b20 con 0x561636c862f0 2023-10-16T11:19:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0b979a700 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636c862f0 0x561636c866d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff093fff700 10 client.11883 ms_handle_connect on v2:172.21.15.172:6833/2771711373 2023-10-16T11:19:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0b979a700 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636c862f0 0x561636c866d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35beb1d51e2020ff server_cookie=35de83066f5eedef in_seq=0 out_seq=0 2023-10-16T11:19:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff093fff700 1 -- 172.21.15.158:0/2745605759 <== mds.0 v2:172.21.15.172:6833/2771711373 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x561636ba8b20 con 0x561636c862f0 2023-10-16T11:19:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff093fff700 10 client.11883 handle_command_reply: tid=0 2023-10-16T11:19:46.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.758+0000 7ff0c7be8b80 1 client.11883 shutdown 2023-10-16T11:19:46.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c85630 msgr2=0x561636cb1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c85630 0x561636cb1660 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac088350 tx=0x7ff0ac021000 comp rx=0 tx=0).stop 2023-10-16T11:19:46.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 shutdown_connections 2023-10-16T11:19:46.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:6833/2771711373,v1:172.21.15.172:6835/2771711373] conn(0x561636c862f0 0x561636c866d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561636c85630 0x561636cb1660 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff093fff700 0 client.11883 ms_handle_reset on v2:172.21.15.172:6833/2771711373 2023-10-16T11:19:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561636c5b010 0x561636d0d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 --2- 172.21.15.158:0/2745605759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x561636c80220 0x561636cb1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 >> 172.21.15.158:0/2745605759 conn(0x561636b8ad80 msgr2=0x561636c82660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 shutdown_connections 2023-10-16T11:19:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 1 -- 172.21.15.158:0/2745605759 wait complete. 2023-10-16T11:19:46.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0c7be8b80 20 client.11883 trim_cache size 0 max 16384 2023-10-16T11:19:46.786 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:19:46.787 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:19:46.787 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:19:46.787 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 11586, 2023-10-16T11:19:46.787 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:19:46.787 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:19:46.788 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "backup_fs", 2023-10-16T11:19:46.788 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 10.09946965, 2023-10-16T11:19:46.788 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 118, 2023-10-16T11:19:46.788 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 221, 2023-10-16T11:19:46.788 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 221, 2023-10-16T11:19:46.789 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 28.354509320999998 2023-10-16T11:19:46.789 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:19:46.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff09806b050 msgr2=0x7ff09806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff09806b050 0x7ff09806d510 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4015040 tx=0x7ff0a4007f90 comp rx=0 tx=0).stop 2023-10-16T11:19:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0b41e5710 msgr2=0x7ff0b41526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0b41e5710 0x7ff0b41526d0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4071180 tx=0x7ff0ac00f490 comp rx=0 tx=0).stop 2023-10-16T11:19:46.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 shutdown_connections 2023-10-16T11:19:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff09806b050 0x7ff09806d510 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0b41e5710 0x7ff0b41526d0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0a0003bc0 0x7ff0b4154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 --2- 172.21.15.158:0/3018155512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b4153f50 0x7ff0b41e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 >> 172.21.15.158:0/3018155512 conn(0x7ff0b40cc700 msgr2=0x7ff0b41561a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 shutdown_connections 2023-10-16T11:19:46.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.759+0000 7ff0bb9fe700 1 -- 172.21.15.158:0/3018155512 wait complete. 2023-10-16T11:19:46.793 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T11:19:46.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.913+0000 7ff1647f5700 1 Processor -- start 2023-10-16T11:19:46.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.913+0000 7ff1647f5700 1 -- start start 2023-10-16T11:19:46.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff1647f5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c06bf40 0x7ff15c161e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff1647f5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff15c162370 0x7ff15c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff1647f5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c0720f0 con 0x7ff15c0cdac0 2023-10-16T11:19:46.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff1647f5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c0cc370 con 0x7ff15c162370 2023-10-16T11:19:46.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff1647f5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c162800 con 0x7ff15c06bf40 2023-10-16T11:19:46.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff162d92700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff15c162370 0x7ff15c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff161d90700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c06bf40 0x7ff15c161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff162d92700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff15c162370 0x7ff15c0ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:39760/0 (socket says 172.21.15.158:39760) 2023-10-16T11:19:46.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff161d90700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c06bf40 0x7ff15c161e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:49416/0 (socket says 172.21.15.158:49416) 2023-10-16T11:19:46.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff162d92700 1 -- 172.21.15.158:0/2864573903 learned_addr learned my addr 172.21.15.158:0/2864573903 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:46.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.914+0000 7ff162591700 1 --1- 172.21.15.158:0/2864573903 >> v1:172.21.15.172:6789/0 conn(0x7ff15c0cdac0 0x7ff15c0cdea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:42658/0 (socket says 172.21.15.158:42658) 2023-10-16T11:19:46.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff162d92700 1 -- 172.21.15.158:0/2864573903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c06bf40 msgr2=0x7ff15c161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:46.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff162d92700 1 --2- 172.21.15.158:0/2864573903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c06bf40 0x7ff15c161e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff162d92700 1 -- 172.21.15.158:0/2864573903 >> v1:172.21.15.172:6789/0 conn(0x7ff15c0cdac0 legacy=0x7ff15c0cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff162d92700 1 -- 172.21.15.158:0/2864573903 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c0726b0 con 0x7ff15c162370 2023-10-16T11:19:46.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff162d92700 1 --2- 172.21.15.158:0/2864573903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c162370 0x7ff15c0ca580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff154009e50 tx=0x7ff15400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=639992fae09c5a51 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff16158f700 1 -- 172.21.15.158:0/2864573903 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15400e070 con 0x7ff15c162370 2023-10-16T11:19:46.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff16158f700 1 -- 172.21.15.158:0/2864573903 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15400b910 con 0x7ff15c162370 2023-10-16T11:19:46.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.915+0000 7ff16158f700 1 -- 172.21.15.158:0/2864573903 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15400bcc0 con 0x7ff15c162370 2023-10-16T11:19:46.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 -- 172.21.15.158:0/2864573903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c162370 msgr2=0x7ff15c0ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 --2- 172.21.15.158:0/2864573903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c162370 0x7ff15c0ca580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff154009e50 tx=0x7ff15400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:46.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 -- 172.21.15.158:0/2864573903 shutdown_connections 2023-10-16T11:19:46.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 --2- 172.21.15.158:0/2864573903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c162370 0x7ff15c0ca580 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 --2- 172.21.15.158:0/2864573903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c06bf40 0x7ff15c161e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 -- 172.21.15.158:0/2864573903 >> 172.21.15.158:0/2864573903 conn(0x7ff15c0cc700 msgr2=0x7ff15c0ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:46.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 -- 172.21.15.158:0/2864573903 shutdown_connections 2023-10-16T11:19:46.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.916+0000 7ff1647f5700 1 -- 172.21.15.158:0/2864573903 wait complete. 2023-10-16T11:19:46.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.917+0000 7ff1647f5700 1 Processor -- start 2023-10-16T11:19:46.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.917+0000 7ff1647f5700 1 -- start start 2023-10-16T11:19:46.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff1647f5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c06bf40 0x7ff15c0c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff1647f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff15c0cdac0 0x7ff15c0c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff1647f5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c162370 0x7ff15c159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff1647f5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c0c7ad0 con 0x7ff15c0cdac0 2023-10-16T11:19:46.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff1647f5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c0c7290 con 0x7ff15c06bf40 2023-10-16T11:19:46.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff1647f5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c159760 con 0x7ff15c162370 2023-10-16T11:19:46.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff15c0cdac0 0x7ff15c0c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff162591700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c06bf40 0x7ff15c0c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff15c0cdac0 0x7ff15c0c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38586/0 (socket says 172.21.15.158:38586) 2023-10-16T11:19:46.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 -- 172.21.15.158:0/3518784256 learned_addr learned my addr 172.21.15.158:0/3518784256 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:46.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff162d92700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c162370 0x7ff15c159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 -- 172.21.15.158:0/3518784256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c162370 msgr2=0x7ff15c159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c162370 0x7ff15c159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 -- 172.21.15.158:0/3518784256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c06bf40 msgr2=0x7ff15c0c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:46.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c06bf40 0x7ff15c0c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:46.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 -- 172.21.15.158:0/3518784256 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff154009b30 con 0x7ff15c0cdac0 2023-10-16T11:19:46.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff161d90700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff15c0cdac0 0x7ff15c0c6d50 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7ff15800e870 tx=0x7ff15800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff152ffd700 1 -- 172.21.15.158:0/3518784256 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158009590 con 0x7ff15c0cdac0 2023-10-16T11:19:46.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.918+0000 7ff152ffd700 1 -- 172.21.15.158:0/3518784256 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff158013680 con 0x7ff15c0cdac0 2023-10-16T11:19:46.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.919+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c0c7510 con 0x7ff15c0cdac0 2023-10-16T11:19:46.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.919+0000 7ff152ffd700 1 -- 172.21.15.158:0/3518784256 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580139f0 con 0x7ff15c0cdac0 2023-10-16T11:19:46.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.919+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c0c4140 con 0x7ff15c0cdac0 2023-10-16T11:19:46.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.920+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c0c3fa0 con 0x7ff15c0cdac0 2023-10-16T11:19:46.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.920+0000 7ff152ffd700 1 -- 172.21.15.158:0/3518784256 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff158013b50 con 0x7ff15c0cdac0 2023-10-16T11:19:46.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.920+0000 7ff152ffd700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff14006b020 0x7ff14006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:46.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.920+0000 7ff152ffd700 1 -- 172.21.15.158:0/3518784256 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7ff158089f90 con 0x7ff15c0cdac0 2023-10-16T11:19:46.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.920+0000 7ff162591700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff14006b020 0x7ff14006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:46.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.921+0000 7ff162591700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff14006b020 0x7ff14006d4e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff14c000f60 tx=0x7ff14c007f10 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:46.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:46.925+0000 7ff152ffd700 1 -- 172.21.15.158:0/3518784256 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1580bcfe0 con 0x7ff15c0cdac0 2023-10-16T11:19:47.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.144+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff15c072340 con 0x7ff14006b020 2023-10-16T11:19:47.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.146+0000 7ff152ffd700 1 -- 172.21.15.158:0/3518784256 <== mgr.11331 v2:172.21.15.158:6832/1944509827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7ff14c0133f0 con 0x7ff14006b020 2023-10-16T11:19:47.148 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:19:47.148 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:47.148 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T11:19:47.148 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:19:47.148 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:19:47.149 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "backup_fs" 2023-10-16T11:19:47.149 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:19:47.149 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:19:47.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff14006b020 msgr2=0x7ff14006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff14006b020 0x7ff14006d4e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff14c000f60 tx=0x7ff14c007f10 comp rx=0 tx=0).stop 2023-10-16T11:19:47.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff15c0cdac0 msgr2=0x7ff15c0c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff15c0cdac0 0x7ff15c0c6d50 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7ff15800e870 tx=0x7ff15800eb80 comp rx=0 tx=0).stop 2023-10-16T11:19:47.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 shutdown_connections 2023-10-16T11:19:47.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff15c162370 0x7ff15c159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff15c06bf40 0x7ff15c0c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7ff14006b020 0x7ff14006d4e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 --2- 172.21.15.158:0/3518784256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff15c0cdac0 0x7ff15c0c6d50 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 >> 172.21.15.158:0/3518784256 conn(0x7ff15c0cc700 msgr2=0x7ff15c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:47.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 shutdown_connections 2023-10-16T11:19:47.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.148+0000 7ff1647f5700 1 -- 172.21.15.158:0/3518784256 wait complete. 2023-10-16T11:19:47.162 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T11:19:47.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.283+0000 7fdaf6153700 1 Processor -- start 2023-10-16T11:19:47.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaf6153700 1 -- start start 2023-10-16T11:19:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaf6153700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 0x7fdaf00cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaf6153700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdaf006ef60 0x7fdaf00ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaf6153700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf006b540 con 0x7fdaf0072af0 2023-10-16T11:19:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaf6153700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf015b780 con 0x7fdaf006ef60 2023-10-16T11:19:47.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaf6153700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf006b260 con 0x7fdaf015b3a0 2023-10-16T11:19:47.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaeeffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdaf0072af0 0x7fdaf00718d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:38770/0 (socket says 172.21.15.158:38770) 2023-10-16T11:19:47.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.284+0000 7fdaeeffd700 1 -- 172.21.15.158:0/2943340662 learned_addr learned my addr 172.21.15.158:0/2943340662 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaeffff700 1 --2- 172.21.15.158:0/2943340662 >> v2:172.21.15.172:3300/0 conn(0x7fdaf006ef60 0x7fdaf00ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaef7fe700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 0x7fdaf00cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243994773 0 0) 0x7fdaf006b540 con 0x7fdaf0072af0 2023-10-16T11:19:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdadc0035b0 con 0x7fdaf0072af0 2023-10-16T11:19:47.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728453402 0 0) 0x7fdadc0035b0 con 0x7fdaf0072af0 2023-10-16T11:19:47.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 msgr2=0x7fdaf00cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:47.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 0x7fdaf00cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 >> v2:172.21.15.172:3300/0 conn(0x7fdaf006ef60 msgr2=0x7fdaf00ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:47.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 --2- 172.21.15.158:0/2943340662 >> v2:172.21.15.172:3300/0 conn(0x7fdaf006ef60 0x7fdaf00ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf015eca0 con 0x7fdaf0072af0 2023-10-16T11:19:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fdad8001670 con 0x7fdaf0072af0 2023-10-16T11:19:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 >> v1:172.21.15.158:6789/0 conn(0x7fdaf0072af0 legacy=0x7fdaf00718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 0x7fdadc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdadc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdadc004d90 0x7fdadc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf015b780 con 0x7fdadc003f80 2023-10-16T11:19:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf006b260 con 0x7fdadc004d90 2023-10-16T11:19:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc00ba10 con 0x7fdaf015b3a0 2023-10-16T11:19:47.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaeeffd700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdadc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaef7fe700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 0x7fdadc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaeffff700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdadc004d90 0x7fdadc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.285+0000 7fdaeeffd700 1 -- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 msgr2=0x7fdadc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaeeffd700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf015b3a0 0x7fdadc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaeeffd700 1 -- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdadc004d90 msgr2=0x7fdadc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaeeffd700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdadc004d90 0x7fdadc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaeeffd700 1 -- 172.21.15.158:0/2943340662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc0035b0 con 0x7fdadc003f80 2023-10-16T11:19:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaeeffd700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdadc004850 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c320 tx=0x7fdadc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55d1a1d91496c40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad800ce00 con 0x7fdadc003f80 2023-10-16T11:19:47.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad8005960 con 0x7fdadc003f80 2023-10-16T11:19:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaee7fc700 1 -- 172.21.15.158:0/2943340662 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8005d10 con 0x7fdadc003f80 2023-10-16T11:19:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaf6153700 1 -- 172.21.15.158:0/2943340662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 msgr2=0x7fdadc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaf6153700 1 --2- 172.21.15.158:0/2943340662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdadc004850 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c320 tx=0x7fdadc005880 comp rx=0 tx=0).stop 2023-10-16T11:19:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaeffff700 1 -- 172.21.15.158:0/2943340662 reap_dead start 2023-10-16T11:19:47.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaf6153700 1 -- 172.21.15.158:0/2943340662 shutdown_connections 2023-10-16T11:19:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaf6153700 1 -- 172.21.15.158:0/2943340662 >> 172.21.15.158:0/2943340662 conn(0x7fdaf00b1d90 msgr2=0x7fdaf00b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaf6153700 1 -- 172.21.15.158:0/2943340662 shutdown_connections 2023-10-16T11:19:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.286+0000 7fdaf6153700 1 -- 172.21.15.158:0/2943340662 wait complete. 2023-10-16T11:19:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.288+0000 7fdaf6153700 1 Processor -- start 2023-10-16T11:19:47.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.288+0000 7fdaf6153700 1 -- start start 2023-10-16T11:19:47.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaf6153700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdaf015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaf6153700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf01550d0 0x7fdaf00ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaf6153700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf00ae740 0x7fdaf015fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaf6153700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf015cff0 con 0x7fdadc003f80 2023-10-16T11:19:47.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaf6153700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf015c3b0 con 0x7fdaf01550d0 2023-10-16T11:19:47.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaf6153700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf015c650 con 0x7fdaf00ae740 2023-10-16T11:19:47.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaef7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdaf015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaef7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdaf015d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38604/0 (socket says 172.21.15.158:38604) 2023-10-16T11:19:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaef7fe700 1 -- 172.21.15.158:0/4001703661 learned_addr learned my addr 172.21.15.158:0/4001703661 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaeeffd700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf01550d0 0x7fdaf00ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.289+0000 7fdaeffff700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf00ae740 0x7fdaf015fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaef7fe700 1 -- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf00ae740 msgr2=0x7fdaf015fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaef7fe700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf00ae740 0x7fdaf015fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaef7fe700 1 -- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf01550d0 msgr2=0x7fdaf00ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaef7fe700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf01550d0 0x7fdaf00ae200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaef7fe700 1 -- 172.21.15.158:0/4001703661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf015c930 con 0x7fdadc003f80 2023-10-16T11:19:47.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaeffff700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf00ae740 0x7fdaf015fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaef7fe700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdaf015d410 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fdae0011aa0 tx=0x7fdae0011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdae7fff700 1 -- 172.21.15.158:0/4001703661 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae000f940 con 0x7fdadc003f80 2023-10-16T11:19:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdae7fff700 1 -- 172.21.15.158:0/4001703661 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae000faa0 con 0x7fdadc003f80 2023-10-16T11:19:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf015cc10 con 0x7fdadc003f80 2023-10-16T11:19:47.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdae7fff700 1 -- 172.21.15.158:0/4001703661 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae000fe50 con 0x7fdadc003f80 2023-10-16T11:19:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.290+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf0155550 con 0x7fdadc003f80 2023-10-16T11:19:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.291+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf015d260 con 0x7fdadc003f80 2023-10-16T11:19:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.291+0000 7fdae7fff700 1 -- 172.21.15.158:0/4001703661 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 137) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdae0014440 con 0x7fdadc003f80 2023-10-16T11:19:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.292+0000 7fdae7fff700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdad006b020 0x7fdad006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.292+0000 7fdae7fff700 1 -- 172.21.15.158:0/4001703661 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fdae008a3c0 con 0x7fdadc003f80 2023-10-16T11:19:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.292+0000 7fdaeeffd700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdad006b020 0x7fdad006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.292+0000 7fdaeeffd700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdad006b020 0x7fdad006d4e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdad8000c00 tx=0x7fdad800a000 comp rx=0 tx=0).ready entity=mgr.11331 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:47.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.296+0000 7fdae7fff700 1 -- 172.21.15.158:0/4001703661 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdae0054e30 con 0x7fdadc003f80 2023-10-16T11:19:47.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.548+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fdaf00681b0 con 0x7fdadc003f80 2023-10-16T11:19:47.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.719+0000 7fdae7fff700 1 -- 172.21.15.158:0/4001703661 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v138) v1 ==== 92+0+0 (secure 0 0 0) 0x7fdae0058450 con 0x7fdadc003f80 2023-10-16T11:19:47.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdad006b020 msgr2=0x7fdad006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdad006b020 0x7fdad006d4e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdad8000c00 tx=0x7fdad800a000 comp rx=0 tx=0).stop 2023-10-16T11:19:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 msgr2=0x7fdaf015d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdaf015d410 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fdae0011aa0 tx=0x7fdae0011db0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 shutdown_connections 2023-10-16T11:19:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaf00ae740 0x7fdaf015fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fdad006b020 0x7fdad006d4e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdadc003f80 0x7fdaf015d410 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 --2- 172.21.15.158:0/4001703661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaf01550d0 0x7fdaf00ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.723+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 >> 172.21.15.158:0/4001703661 conn(0x7fdaf00b1d90 msgr2=0x7fdaf00afc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:47.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.724+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 shutdown_connections 2023-10-16T11:19:47.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.724+0000 7fdaf6153700 1 -- 172.21.15.158:0/4001703661 wait complete. 2023-10-16T11:19:47.740 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T11:19:47.864 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:47.863+0000 7facee434200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:19:47.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.865+0000 7f6888aed700 1 Processor -- start 2023-10-16T11:19:47.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.866+0000 7f6888aed700 1 -- start start 2023-10-16T11:19:47.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.866+0000 7f6888aed700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68840d0c60 0x7f68840b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.866+0000 7f6888aed700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f68840b3460 0x7f68840b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6888aed700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68840b4420 con 0x7f688415b6b0 2023-10-16T11:19:47.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6888aed700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688415ba90 con 0x7f68840d0c60 2023-10-16T11:19:47.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6888aed700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68840b38d0 con 0x7f68840b3460 2023-10-16T11:19:47.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68840d0c60 0x7f68840b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f68840d0c60 0x7f68840b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38640/0 (socket says 172.21.15.158:38640) 2023-10-16T11:19:47.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 -- 172.21.15.158:0/3114934191 learned_addr learned my addr 172.21.15.158:0/3114934191 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:47.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6882d9d700 1 --2- 172.21.15.158:0/3114934191 >> v2:172.21.15.172:3300/0 conn(0x7f68840b3460 0x7f68840b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 -- 172.21.15.158:0/3114934191 >> v2:172.21.15.172:3300/0 conn(0x7f68840b3460 msgr2=0x7f68840b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:47.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 --2- 172.21.15.158:0/3114934191 >> v2:172.21.15.172:3300/0 conn(0x7f68840b3460 0x7f68840b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 -- 172.21.15.158:0/3114934191 >> v1:172.21.15.172:6789/0 conn(0x7f688415b6b0 legacy=0x7f68840cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 -- 172.21.15.158:0/3114934191 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68840cbc20 con 0x7f68840d0c60 2023-10-16T11:19:47.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f6881d9b700 1 --2- 172.21.15.158:0/3114934191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840d0c60 0x7f68840b2f20 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f68840b0cb0 tx=0x7f6878009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54dd8a7de637d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:47.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f688159a700 1 -- 172.21.15.158:0/3114934191 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878011830 con 0x7f68840d0c60 2023-10-16T11:19:47.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f688159a700 1 -- 172.21.15.158:0/3114934191 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6878011990 con 0x7f68840d0c60 2023-10-16T11:19:47.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.867+0000 7f688159a700 1 -- 172.21.15.158:0/3114934191 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878011d20 con 0x7f68840d0c60 2023-10-16T11:19:47.873 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f16d2b7f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:19:47.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 -- 172.21.15.158:0/3114934191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840d0c60 msgr2=0x7f68840b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 --2- 172.21.15.158:0/3114934191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840d0c60 0x7f68840b2f20 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f68840b0cb0 tx=0x7f6878009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:47.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 -- 172.21.15.158:0/3114934191 shutdown_connections 2023-10-16T11:19:47.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 --2- 172.21.15.158:0/3114934191 >> v2:172.21.15.172:3300/0 conn(0x7f68840b3460 0x7f68840b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 --2- 172.21.15.158:0/3114934191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840d0c60 0x7f68840b2f20 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 -- 172.21.15.158:0/3114934191 >> 172.21.15.158:0/3114934191 conn(0x7f68840cdf20 msgr2=0x7f68840ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:47.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 -- 172.21.15.158:0/3114934191 shutdown_connections 2023-10-16T11:19:47.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.868+0000 7f6888aed700 1 -- 172.21.15.158:0/3114934191 wait complete. 2023-10-16T11:19:47.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.869+0000 7f6888aed700 1 Processor -- start 2023-10-16T11:19:47.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.869+0000 7f6888aed700 1 -- start start 2023-10-16T11:19:47.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6888aed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840b3460 0x7f688414e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6888aed700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68840d0c60 0x7f68840a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6888aed700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f688415b6b0 0x7f68840add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6888aed700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68840ac990 con 0x7f68840b3460 2023-10-16T11:19:47.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6888aed700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688414e7d0 con 0x7f68840d0c60 2023-10-16T11:19:47.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6888aed700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68840ae830 con 0x7f688415b6b0 2023-10-16T11:19:47.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f688259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840b3460 0x7f688414e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f688259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840b3460 0x7f688414e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38644/0 (socket says 172.21.15.158:38644) 2023-10-16T11:19:47.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f688259c700 1 -- 172.21.15.158:0/3115627151 learned_addr learned my addr 172.21.15.158:0/3115627151 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:47.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6881d9b700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68840d0c60 0x7f68840a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.870+0000 7f6882d9d700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f688415b6b0 0x7f68840add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:47.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f688259c700 1 -- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f688415b6b0 msgr2=0x7f68840add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f688259c700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f688415b6b0 0x7f68840add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f688259c700 1 -- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68840d0c60 msgr2=0x7f68840a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:47.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f688259c700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68840d0c60 0x7f68840a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:47.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f688259c700 1 -- 172.21.15.158:0/3115627151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6878009550 con 0x7f68840b3460 2023-10-16T11:19:47.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f688259c700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840b3460 0x7f688414e290 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f687400c4e0 tx=0x7f687400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:47.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f6872ffd700 1 -- 172.21.15.158:0/3115627151 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68740052d0 con 0x7f68840b3460 2023-10-16T11:19:47.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f6872ffd700 1 -- 172.21.15.158:0/3115627151 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687400ce90 con 0x7f68840b3460 2023-10-16T11:19:47.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68840aeb10 con 0x7f68840b3460 2023-10-16T11:19:47.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f6872ffd700 1 -- 172.21.15.158:0/3115627151 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6874003680 con 0x7f68840b3460 2023-10-16T11:19:47.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.871+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68840af0d0 con 0x7f68840b3460 2023-10-16T11:19:47.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.872+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68840681b0 con 0x7f68840b3460 2023-10-16T11:19:47.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.872+0000 7f6872ffd700 1 -- 172.21.15.158:0/3115627151 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 138) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f68740037e0 con 0x7f68840b3460 2023-10-16T11:19:47.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.873+0000 7f6872ffd700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 0x7f686406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:47.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.873+0000 7f6881d9b700 1 -- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 msgr2=0x7f686406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:47.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.873+0000 7f6872ffd700 1 -- 172.21.15.158:0/3115627151 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f6874089170 con 0x7f68840b3460 2023-10-16T11:19:47.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.873+0000 7f6881d9b700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 0x7f686406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:47.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:47.877+0000 7f6872ffd700 1 -- 172.21.15.158:0/3115627151 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6874053a90 con 0x7f68840b3460 2023-10-16T11:19:47.931 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:47.930+0000 7facee434200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:47.938 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:47.937+0000 7f16d2b7f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:48.004 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:48.002+0000 7facee434200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:19:48.012 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:48.010+0000 7f16d2b7f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:19:48.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.074+0000 7f6881d9b700 1 -- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 msgr2=0x7f686406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:48.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.074+0000 7f6881d9b700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 0x7f686406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:48.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.079+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f688415bcf0 con 0x7f68840b3460 2023-10-16T11:19:48.082 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:48.081+0000 7facee434200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:19:48.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.082+0000 7f6872ffd700 1 -- 172.21.15.158:0/3115627151 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f68740570b0 con 0x7f68840b3460 2023-10-16T11:19:48.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 msgr2=0x7f686406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:48.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 0x7f686406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840b3460 msgr2=0x7f688414e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840b3460 0x7f688414e290 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f687400c4e0 tx=0x7f687400c7f0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 shutdown_connections 2023-10-16T11:19:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f688415b6b0 0x7f68840add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f686406b070 0x7f686406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68840b3460 0x7f688414e290 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 --2- 172.21.15.158:0/3115627151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68840d0c60 0x7f68840a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 >> 172.21.15.158:0/3115627151 conn(0x7f68840cdf20 msgr2=0x7f68840d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:48.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 shutdown_connections 2023-10-16T11:19:48.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.086+0000 7f6888aed700 1 -- 172.21.15.158:0/3115627151 wait complete. 2023-10-16T11:19:48.096 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:48.095+0000 7f16d2b7f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:19:48.105 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T11:19:48.223 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:48.222+0000 7facee434200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:19:48.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.232+0000 7fb0d20cd700 1 Processor -- start 2023-10-16T11:19:48.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.232+0000 7fb0d20cd700 1 -- start start 2023-10-16T11:19:48.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0d20cd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb0cc071920 0x7fb0cc06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0d20cd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071d00 0x7fb0cc15f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0d20cd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0cbb20 con 0x7fb0cc0cf6c0 2023-10-16T11:19:48.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0d20cd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc06a8d0 con 0x7fb0cc071920 2023-10-16T11:19:48.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0d20cd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc06abb0 con 0x7fb0cc071d00 2023-10-16T11:19:48.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb0cc071920 0x7fb0cc06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071d00 0x7fb0cc15f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb0cc071920 0x7fb0cc06d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:39878/0 (socket says 172.21.15.158:39878) 2023-10-16T11:19:48.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071d00 0x7fb0cc15f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:49534/0 (socket says 172.21.15.158:49534) 2023-10-16T11:19:48.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/1546665628 learned_addr learned my addr 172.21.15.158:0/1546665628 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:48.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0caffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb0cc0cf6c0 0x7fb0cc15f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:42778/0 (socket says 172.21.15.158:42778) 2023-10-16T11:19:48.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071d00 msgr2=0x7fb0cc15f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:48.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cb7fe700 1 --2- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071d00 0x7fb0cc15f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/1546665628 >> v1:172.21.15.172:6789/0 conn(0x7fb0cc0cf6c0 legacy=0x7fb0cc15f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/1546665628 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc06dcb0 con 0x7fb0cc071920 2023-10-16T11:19:48.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0ca7fc700 1 -- 172.21.15.158:0/1546665628 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403036398 0 0) 0x7fb0cc0cbb20 con 0x7fb0cc0cf6c0 2023-10-16T11:19:48.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.233+0000 7fb0cbfff700 1 --2- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071d00 0x7fb0cc15f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:48.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.234+0000 7fb0cb7fe700 1 --2- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc071920 0x7fb0cc06d3f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009e80 tx=0x7fb0c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ef6e5a3aae369f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:48.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.234+0000 7fb0ca7fc700 1 -- 172.21.15.158:0/1546665628 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c400e070 con 0x7fb0cc071920 2023-10-16T11:19:48.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.234+0000 7fb0ca7fc700 1 -- 172.21.15.158:0/1546665628 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c400b8d0 con 0x7fb0cc071920 2023-10-16T11:19:48.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.234+0000 7fb0ca7fc700 1 -- 172.21.15.158:0/1546665628 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c400bc80 con 0x7fb0cc071920 2023-10-16T11:19:48.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 -- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc071920 msgr2=0x7fb0cc06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc071920 0x7fb0cc06d3f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009e80 tx=0x7fb0c400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:48.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 -- 172.21.15.158:0/1546665628 shutdown_connections 2023-10-16T11:19:48.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071d00 0x7fb0cc15f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/1546665628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc071920 0x7fb0cc06d3f0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 -- 172.21.15.158:0/1546665628 >> 172.21.15.158:0/1546665628 conn(0x7fb0cc0b1bd0 msgr2=0x7fb0cc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:48.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 -- 172.21.15.158:0/1546665628 shutdown_connections 2023-10-16T11:19:48.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.235+0000 7fb0d20cd700 1 -- 172.21.15.158:0/1546665628 wait complete. 2023-10-16T11:19:48.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.236+0000 7fb0d20cd700 1 Processor -- start 2023-10-16T11:19:48.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.236+0000 7fb0d20cd700 1 -- start start 2023-10-16T11:19:48.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.237+0000 7fb0d20cd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071920 0x7fb0cc13b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.237+0000 7fb0d20cd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0cc071d00 0x7fb0cc1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.237+0000 7fb0d20cd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc0cf6c0 0x7fb0cc15bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.237+0000 7fb0d20cd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc13b950 con 0x7fb0cc071d00 2023-10-16T11:19:48.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.237+0000 7fb0d20cd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc15aea0 con 0x7fb0cc0cf6c0 2023-10-16T11:19:48.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.237+0000 7fb0d20cd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc15a290 con 0x7fb0cc071920 2023-10-16T11:19:48.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071920 0x7fb0cc13b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0caffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0cc071d00 0x7fb0cc1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071920 0x7fb0cc13b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:49536/0 (socket says 172.21.15.158:49536) 2023-10-16T11:19:48.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/4093334564 learned_addr learned my addr 172.21.15.158:0/4093334564 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:48.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cbfff700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc0cf6c0 0x7fb0cc15bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc0cf6c0 msgr2=0x7fb0cc15bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc0cf6c0 0x7fb0cc15bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0cc071d00 msgr2=0x7fb0cc1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0cc071d00 0x7fb0cc1598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 -- 172.21.15.158:0/4093334564 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc0094a0 con 0x7fb0cc071920 2023-10-16T11:19:48.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0cb7fe700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071920 0x7fb0cc13b410 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400b680 tx=0x7fb0c4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:48.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0c3fff700 1 -- 172.21.15.158:0/4093334564 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40048f0 con 0x7fb0cc071920 2023-10-16T11:19:48.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0c3fff700 1 -- 172.21.15.158:0/4093334564 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c40099b0 con 0x7fb0cc071920 2023-10-16T11:19:48.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c4009b30 con 0x7fb0cc071920 2023-10-16T11:19:48.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0c3fff700 1 -- 172.21.15.158:0/4093334564 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c401b630 con 0x7fb0cc071920 2023-10-16T11:19:48.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.238+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0cc15a850 con 0x7fb0cc071920 2023-10-16T11:19:48.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.240+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0cc0681b0 con 0x7fb0cc071920 2023-10-16T11:19:48.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.240+0000 7fb0c3fff700 1 -- 172.21.15.158:0/4093334564 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 138) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fb0c401b790 con 0x7fb0cc071920 2023-10-16T11:19:48.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.240+0000 7fb0c3fff700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 0x7fb0ac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.240+0000 7fb0caffd700 1 -- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 msgr2=0x7fb0ac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:48.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.240+0000 7fb0c3fff700 1 -- 172.21.15.158:0/4093334564 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fb0c408b4c0 con 0x7fb0cc071920 2023-10-16T11:19:48.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.240+0000 7fb0caffd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 0x7fb0ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:48.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.245+0000 7fb0c3fff700 1 -- 172.21.15.158:0/4093334564 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0c4055d60 con 0x7fb0cc071920 2023-10-16T11:19:48.251 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:48.249+0000 7f16d2b7f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:19:48.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.441+0000 7fb0caffd700 1 -- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 msgr2=0x7fb0ac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:48.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.441+0000 7fb0caffd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 0x7fb0ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:48.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.443+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb0cc15b180 con 0x7fb0cc071920 2023-10-16T11:19:48.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.450+0000 7fb0c3fff700 1 -- 172.21.15.158:0/4093334564 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb0c4059380 con 0x7fb0cc071920 2023-10-16T11:19:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 msgr2=0x7fb0ac06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 0x7fb0ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071920 msgr2=0x7fb0cc13b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071920 0x7fb0cc13b410 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400b680 tx=0x7fb0c4004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 shutdown_connections 2023-10-16T11:19:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0cc0cf6c0 0x7fb0cc15bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0cc071920 0x7fb0cc13b410 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7fb0ac06b070 0x7fb0ac06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 --2- 172.21.15.158:0/4093334564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0cc071d00 0x7fb0cc1598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 >> 172.21.15.158:0/4093334564 conn(0x7fb0cc0b1bd0 msgr2=0x7fb0cc157bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 shutdown_connections 2023-10-16T11:19:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.452+0000 7fb0d20cd700 1 -- 172.21.15.158:0/4093334564 wait complete. 2023-10-16T11:19:48.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:48.467 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T11:19:48.487 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T11:19:48.550 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:48.548+0000 7facee434200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:19:48.587 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:48.586+0000 7f16d2b7f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:19:48.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.656+0000 7f4dd1e5b700 1 Processor -- start 2023-10-16T11:19:48.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.656+0000 7f4dd1e5b700 1 -- start start 2023-10-16T11:19:48.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.656+0000 7f4dd1e5b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15d6c0 0x7f4dcc0cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.656+0000 7f4dd1e5b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4dcc15daa0 0x7f4dcc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.656+0000 7f4dd1e5b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc0d1ee0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.656+0000 7f4dd1e5b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dcc0d40a0 con 0x7f4dcc15daa0 2023-10-16T11:19:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.656+0000 7f4dd1e5b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0d12d0 con 0x7f4dcc15d6c0 2023-10-16T11:19:48.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15d6c0 0x7f4dcc0cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcaffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4dcc15daa0 0x7f4dcc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcbfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4dcc0d2370 0x7f4dcc0cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:42838/0 (socket says 172.21.15.158:42838) 2023-10-16T11:19:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcbfff700 1 -- 172.21.15.158:0/3669585670 learned_addr learned my addr 172.21.15.158:0/3669585670 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcaffd700 1 -- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15d6c0 msgr2=0x7f4dcc0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15d6c0 0x7f4dcc0cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcaffd700 1 -- 172.21.15.158:0/3669585670 >> v1:172.21.15.172:6789/0 conn(0x7f4dcc0d2370 legacy=0x7f4dcc0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcaffd700 1 -- 172.21.15.158:0/3669585670 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0d15b0 con 0x7f4dcc15daa0 2023-10-16T11:19:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dca7fc700 1 -- 172.21.15.158:0/3669585670 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949180777 0 0) 0x7f4dcc0d1ee0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcb7fe700 1 --2- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15d6c0 0x7f4dcc0cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15daa0 0x7f4dcc0d0470 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009840 tx=0x7f4db4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd16b6f3c8339446 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.657+0000 7f4dca7fc700 1 -- 172.21.15.158:0/3669585670 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4011830 con 0x7f4dcc15daa0 2023-10-16T11:19:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dca7fc700 1 -- 172.21.15.158:0/3669585670 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4011990 con 0x7f4dcc15daa0 2023-10-16T11:19:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dca7fc700 1 -- 172.21.15.158:0/3669585670 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4011d20 con 0x7f4dcc15daa0 2023-10-16T11:19:48.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15daa0 msgr2=0x7f4dcc0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 --2- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15daa0 0x7f4dcc0d0470 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009840 tx=0x7f4db4009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:48.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/3669585670 shutdown_connections 2023-10-16T11:19:48.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 --2- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15d6c0 0x7f4dcc0cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 --2- 172.21.15.158:0/3669585670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15daa0 0x7f4dcc0d0470 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/3669585670 >> 172.21.15.158:0/3669585670 conn(0x7f4dcc09c940 msgr2=0x7f4dcc0d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:48.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/3669585670 shutdown_connections 2023-10-16T11:19:48.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.658+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/3669585670 wait complete. 2023-10-16T11:19:48.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.660+0000 7f4dd1e5b700 1 Processor -- start 2023-10-16T11:19:48.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.660+0000 7f4dd1e5b700 1 -- start start 2023-10-16T11:19:48.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.660+0000 7f4dd1e5b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dcc0d2370 0x7f4dcc158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.660+0000 7f4dd1e5b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15d6c0 0x7f4dcc1606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.660+0000 7f4dd1e5b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15daa0 0x7f4dcc06c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dd1e5b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc1f7350 con 0x7f4dcc0d2370 2023-10-16T11:19:48.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dd1e5b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc159390 con 0x7f4dcc15d6c0 2023-10-16T11:19:48.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dd1e5b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0c7100 con 0x7f4dcc15daa0 2023-10-16T11:19:48.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dcc0d2370 0x7f4dcc158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dcc0d2370 0x7f4dcc158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38760/0 (socket says 172.21.15.158:38760) 2023-10-16T11:19:48.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 -- 172.21.15.158:0/983084308 learned_addr learned my addr 172.21.15.158:0/983084308 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:48.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcbfff700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15daa0 0x7f4dcc06c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15d6c0 0x7f4dcc1606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:48.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15daa0 msgr2=0x7f4dcc06c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4dcc15daa0 0x7f4dcc06c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15d6c0 msgr2=0x7f4dcc1606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:48.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dcc15d6c0 0x7f4dcc1606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:48.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 -- 172.21.15.158:0/983084308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009a70 con 0x7f4dcc0d2370 2023-10-16T11:19:48.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dcb7fe700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dcc0d2370 0x7f4dcc158e50 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009f60 tx=0x7f4dbc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:48.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc00cdc0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc00f040 con 0x7f4dcc0d2370 2023-10-16T11:19:48.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db40094a0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0146a0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.661+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dcc1f63f0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.662+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dcc0681b0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.663+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 138) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4dbc014800 con 0x7f4dcc0d2370 2023-10-16T11:19:48.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.663+0000 7f4dc3fff700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 0x7f4dac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:48.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.663+0000 7f4dcaffd700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 msgr2=0x7f4dac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:48.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.663+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f4dbc08abc0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.663+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 0x7f4dac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:48.674 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:48.668+0000 7facee434200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:19:48.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.671+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4dbc0553d0 con 0x7f4dcc0d2370 2023-10-16T11:19:48.711 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:48.710+0000 7f16d2b7f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:19:48.744 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:48.743+0000 7facee434200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:19:48.788 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:48.787+0000 7f16d2b7f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:19:48.823 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:48.822+0000 7facee434200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:19:48.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.864+0000 7f4dcaffd700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 msgr2=0x7f4dac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:48.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.864+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 0x7f4dac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:48.870 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:48.868+0000 7f16d2b7f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:19:48.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:48.897+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 --> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dcc0d1b70 con 0x7f4dac06b070 2023-10-16T11:19:49.190 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:49.189+0000 7facee434200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:19:49.239 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:49.237+0000 7f16d2b7f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:19:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:49.265+0000 7f4dcaffd700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 msgr2=0x7f4dac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:49.265+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 0x7f4dac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T11:19:49.761 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:49.759+0000 7facee434200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:19:49.808 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:49.807+0000 7f16d2b7f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:19:49.829 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:49.828+0000 7facee434200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:19:49.878 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:49.877+0000 7f16d2b7f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:19:50.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:50.066+0000 7f4dcaffd700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 msgr2=0x7f4dac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:50.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:50.066+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 0x7f4dac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T11:19:50.456 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:50.454+0000 7facee434200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:50.508 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:50.507+0000 7f16d2b7f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:50.533 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:50.532+0000 7facee434200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:19:50.586 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:50.584+0000 7f16d2b7f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:19:50.677 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:50.675+0000 7facee434200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:50.730 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:50.729+0000 7f16d2b7f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:19:50.905 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:50.903+0000 7facee434200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:19:50.961 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:50.960+0000 7f16d2b7f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:19:50.991 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:50.977+0000 7facee434200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:19:51.036 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:51.034+0000 7f16d2b7f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:19:51.244 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:51.242+0000 7facee434200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:19:51.302 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:51.301+0000 7f16d2b7f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:19:51.466 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:51.464+0000 7facee434200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:19:51.525 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:51.524+0000 7f16d2b7f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:19:51.615 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:51.613+0000 7facee434200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:19:51.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:51.667+0000 7f4dcaffd700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 msgr2=0x7f4dac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1944509827 2023-10-16T11:19:51.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:51.667+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 0x7f4dac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T11:19:51.676 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:51.675+0000 7f16d2b7f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:19:51.730 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:51.728+0000 7facee434200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:19:51.791 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:51.790+0000 7f16d2b7f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:19:51.886 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:51.884+0000 7facee434200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:19:51.948 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:51.947+0000 7f16d2b7f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:19:52.150 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:52.148+0000 7facee434200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:52.214 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:52.212+0000 7f16d2b7f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:52.311 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:52.309+0000 7facee434200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:19:52.376 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:52.374+0000 7f16d2b7f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:19:52.840 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:52.838+0000 7facee434200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:19:52.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:52.909+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 139) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4dbc088a60 con 0x7f4dcc0d2370 2023-10-16T11:19:52.913 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:52.911+0000 7f16d2b7f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:19:52.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:52.933+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 140) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f4dbc0557a0 con 0x7f4dcc0d2370 2023-10-16T11:19:52.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:52.933+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 msgr2=0x7f4dac06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:52.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:52.933+0000 7f4dc3fff700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1944509827,v1:172.21.15.158:6833/1944509827] conn(0x7f4dac06b070 0x7f4dac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:53.158 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.156+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.159 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.156+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.159 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.156+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.159 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.156+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.159 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.156+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.160 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.157+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.160 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.157+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.160 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.157+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.160 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.157+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.160 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.157+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.168 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.168 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.169 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.169 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.169 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f164835f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.169 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.169 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.170 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.170 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.170 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:53.166+0000 7f1645359700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:19:53.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:53.937+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f4dbc056070 con 0x7f4dcc0d2370 2023-10-16T11:19:53.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:53.937+0000 7f4dc3fff700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f4dac0279b0 0x7f4dac027d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:53.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:53.937+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dac071fd0 con 0x7f4dac0279b0 2023-10-16T11:19:53.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:53.939+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f4dac0279b0 0x7f4dac027d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:53.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:53.942+0000 7f4dcaffd700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f4dac0279b0 0x7f4dac027d90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc15de80 tx=0x7f4db401c040 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:54.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.009+0000 7f4dc3fff700 1 -- 172.21.15.158:0/983084308 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7f4dac071fd0 con 0x7f4dac0279b0 2023-10-16T11:19:54.011 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T11:19:54.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f4dac0279b0 msgr2=0x7f4dac027d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f4dac0279b0 0x7f4dac027d90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc15de80 tx=0x7f4db401c040 comp rx=0 tx=0).stop 2023-10-16T11:19:54.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dcc0d2370 msgr2=0x7f4dcc158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 --2- 172.21.15.158:0/983084308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4dcc0d2370 0x7f4dcc158e50 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009f60 tx=0x7f4dbc009f90 comp rx=0 tx=0).stop 2023-10-16T11:19:54.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dcbfff700 1 -- 172.21.15.158:0/983084308 reap_dead start 2023-10-16T11:19:54.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 shutdown_connections 2023-10-16T11:19:54.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 >> 172.21.15.158:0/983084308 conn(0x7f4dcc09c940 msgr2=0x7f4dcc0d3340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:54.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 shutdown_connections 2023-10-16T11:19:54.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.012+0000 7f4dd1e5b700 1 -- 172.21.15.158:0/983084308 wait complete. 2023-10-16T11:19:54.027 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T11:19:54.027 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T11:19:54.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.152+0000 7f2e7e3d6700 1 Processor -- start 2023-10-16T11:19:54.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.152+0000 7f2e7e3d6700 1 -- start start 2023-10-16T11:19:54.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e7e3d6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7815da40 0x7f2e780caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e7e3d6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e780cc590 0x7f2e7806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e7e3d6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e7806ae10 0x7f2e780ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e7e3d6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7806b280 con 0x7f2e780cc590 2023-10-16T11:19:54.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e7e3d6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e780cbd30 con 0x7f2e7806ae10 2023-10-16T11:19:54.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e7e3d6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780cc1c0 con 0x7f2e7815da40 2023-10-16T11:19:54.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e777fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e780cc590 0x7f2e7806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e777fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e780cc590 0x7f2e7806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38978/0 (socket says 172.21.15.158:38978) 2023-10-16T11:19:54.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.153+0000 7f2e777fe700 1 -- 172.21.15.158:0/2145258734 learned_addr learned my addr 172.21.15.158:0/2145258734 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:54.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e777fe700 1 -- 172.21.15.158:0/2145258734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7815da40 msgr2=0x7f2e780caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:54.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e77fff700 1 --2- 172.21.15.158:0/2145258734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7815da40 0x7f2e780caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e7c973700 1 --2- 172.21.15.158:0/2145258734 >> v2:172.21.15.172:3300/0 conn(0x7f2e7806ae10 0x7f2e780ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e777fe700 1 --2- 172.21.15.158:0/2145258734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7815da40 0x7f2e780caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e777fe700 1 -- 172.21.15.158:0/2145258734 >> v2:172.21.15.172:3300/0 conn(0x7f2e7806ae10 msgr2=0x7f2e780ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:54.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e777fe700 1 --2- 172.21.15.158:0/2145258734 >> v2:172.21.15.172:3300/0 conn(0x7f2e7806ae10 0x7f2e780ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e777fe700 1 -- 172.21.15.158:0/2145258734 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7815de20 con 0x7f2e780cc590 2023-10-16T11:19:54.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e777fe700 1 --2- 172.21.15.158:0/2145258734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e780cc590 0x7f2e7806a8d0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f2e780d1380 tx=0x7f2e68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6713a29ad811679 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:54.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e76ffd700 1 -- 172.21.15.158:0/2145258734 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800b870 con 0x7f2e780cc590 2023-10-16T11:19:54.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e76ffd700 1 -- 172.21.15.158:0/2145258734 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6800b9d0 con 0x7f2e780cc590 2023-10-16T11:19:54.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.154+0000 7f2e76ffd700 1 -- 172.21.15.158:0/2145258734 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800bd60 con 0x7f2e780cc590 2023-10-16T11:19:54.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/2145258734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e780cc590 msgr2=0x7f2e7806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/2145258734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e780cc590 0x7f2e7806a8d0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f2e780d1380 tx=0x7f2e68009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/2145258734 shutdown_connections 2023-10-16T11:19:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/2145258734 >> v2:172.21.15.172:3300/0 conn(0x7f2e7806ae10 0x7f2e780ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/2145258734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e7815da40 0x7f2e780caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/2145258734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e780cc590 0x7f2e7806a8d0 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/2145258734 >> 172.21.15.158:0/2145258734 conn(0x7f2e78099a10 msgr2=0x7f2e780d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/2145258734 shutdown_connections 2023-10-16T11:19:54.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.155+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/2145258734 wait complete. 2023-10-16T11:19:54.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.156+0000 7f2e7e3d6700 1 Processor -- start 2023-10-16T11:19:54.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.156+0000 7f2e7e3d6700 1 -- start start 2023-10-16T11:19:54.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7e3d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7806ae10 0x7f2e781e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7e3d6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cc590 0x7f2e780c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7e3d6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7815da40 0x7f2e7815b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7e3d6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e781e5c80 con 0x7f2e7806ae10 2023-10-16T11:19:54.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7e3d6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e781e33f0 con 0x7f2e7815da40 2023-10-16T11:19:54.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7e3d6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e781e36d0 con 0x7f2e780cc590 2023-10-16T11:19:54.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7806ae10 0x7f2e781e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7806ae10 0x7f2e781e2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38982/0 (socket says 172.21.15.158:38982) 2023-10-16T11:19:54.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 -- 172.21.15.158:0/729555742 learned_addr learned my addr 172.21.15.158:0/729555742 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:54.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e777fe700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cc590 0x7f2e780c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7c973700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7815da40 0x7f2e7815b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 -- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cc590 msgr2=0x7f2e780c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cc590 0x7f2e780c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 -- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7815da40 msgr2=0x7f2e7815b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7815da40 0x7f2e7815b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 -- 172.21.15.158:0/729555742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009550 con 0x7f2e7806ae10 2023-10-16T11:19:54.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e77fff700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7806ae10 0x7f2e781e2cf0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00c7f0 tx=0x7f2e6c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:54.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e74ff9700 1 -- 172.21.15.158:0/729555742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c011800 con 0x7f2e7806ae10 2023-10-16T11:19:54.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e74ff9700 1 -- 172.21.15.158:0/729555742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c011960 con 0x7f2e7806ae10 2023-10-16T11:19:54.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e74ff9700 1 -- 172.21.15.158:0/729555742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c011cd0 con 0x7f2e7806ae10 2023-10-16T11:19:54.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.157+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e781581c0 con 0x7f2e7806ae10 2023-10-16T11:19:54.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.158+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e781e39b0 con 0x7f2e7806ae10 2023-10-16T11:19:54.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.158+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e780681b0 con 0x7f2e7806ae10 2023-10-16T11:19:54.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.159+0000 7f2e74ff9700 1 -- 172.21.15.158:0/729555742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2e6c004020 con 0x7f2e7806ae10 2023-10-16T11:19:54.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.159+0000 7f2e74ff9700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f2e6406ada0 0x7f2e6406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.159+0000 7f2e74ff9700 1 -- 172.21.15.158:0/729555742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f2e6c0894a0 con 0x7f2e7806ae10 2023-10-16T11:19:54.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.159+0000 7f2e777fe700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f2e6406ada0 0x7f2e6406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.160+0000 7f2e777fe700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f2e6406ada0 0x7f2e6406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e68000c00 tx=0x7f2e68008030 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:54.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.164+0000 7f2e74ff9700 1 -- 172.21.15.158:0/729555742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e6c0540b0 con 0x7f2e7806ae10 2023-10-16T11:19:54.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.386+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e780cb120 con 0x7f2e6406ada0 2023-10-16T11:19:54.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.433+0000 7f2e74ff9700 1 -- 172.21.15.158:0/729555742 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f2e68006c60 con 0x7f2e6406ada0 2023-10-16T11:19:54.435 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T11:19:54.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f2e6406ada0 msgr2=0x7f2e6406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f2e6406ada0 0x7f2e6406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e68000c00 tx=0x7f2e68008030 comp rx=0 tx=0).stop 2023-10-16T11:19:54.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7806ae10 msgr2=0x7f2e781e2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7806ae10 0x7f2e781e2cf0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00c7f0 tx=0x7f2e6c00cb00 comp rx=0 tx=0).stop 2023-10-16T11:19:54.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 shutdown_connections 2023-10-16T11:19:54.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f2e6406ada0 0x7f2e6406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e7815da40 0x7f2e7815b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e7806ae10 0x7f2e781e2cf0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 --2- 172.21.15.158:0/729555742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cc590 0x7f2e780c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 >> 172.21.15.158:0/729555742 conn(0x7f2e78099a10 msgr2=0x7f2e780b5740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:54.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.435+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 shutdown_connections 2023-10-16T11:19:54.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.436+0000 7f2e7e3d6700 1 -- 172.21.15.158:0/729555742 wait complete. 2023-10-16T11:19:54.449 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T11:19:54.450 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T11:19:54.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.575+0000 7f2303c60700 1 Processor -- start 2023-10-16T11:19:54.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.575+0000 7f2303c60700 1 -- start start 2023-10-16T11:19:54.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f2303c60700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22fc15d6c0 0x7f22fc0cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f2303c60700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f22fc0cf360 0x7f22fc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f2303c60700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0d1ee0 con 0x7f22fc15daa0 2023-10-16T11:19:54.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f2303c60700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc0d40a0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f2303c60700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc0d12d0 con 0x7f22fc0cf360 2023-10-16T11:19:54.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f23019fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22fc15d6c0 0x7f22fc0cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f23019fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22fc15d6c0 0x7f22fc0cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38990/0 (socket says 172.21.15.158:38990) 2023-10-16T11:19:54.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.576+0000 7f23019fc700 1 -- 172.21.15.158:0/4224273213 learned_addr learned my addr 172.21.15.158:0/4224273213 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:54.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23021fd700 1 --2- 172.21.15.158:0/4224273213 >> v2:172.21.15.172:3300/0 conn(0x7f22fc0cf360 0x7f22fc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23019fc700 1 -- 172.21.15.158:0/4224273213 >> v2:172.21.15.172:3300/0 conn(0x7f22fc0cf360 msgr2=0x7f22fc0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:54.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23019fc700 1 --2- 172.21.15.158:0/4224273213 >> v2:172.21.15.172:3300/0 conn(0x7f22fc0cf360 0x7f22fc0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23019fc700 1 -- 172.21.15.158:0/4224273213 >> v1:172.21.15.172:6789/0 conn(0x7f22fc15daa0 legacy=0x7f22fc0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23019fc700 1 -- 172.21.15.158:0/4224273213 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0d15b0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23019fc700 1 --2- 172.21.15.158:0/4224273213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0cff30 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0b5050 tx=0x7f22ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c61eb4fefb8a5f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:54.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23009fa700 1 -- 172.21.15.158:0/4224273213 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec00b870 con 0x7f22fc15d6c0 2023-10-16T11:19:54.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23009fa700 1 -- 172.21.15.158:0/4224273213 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec00b9d0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.577+0000 7f23009fa700 1 -- 172.21.15.158:0/4224273213 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec00bd60 con 0x7f22fc15d6c0 2023-10-16T11:19:54.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 -- 172.21.15.158:0/4224273213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 msgr2=0x7f22fc0cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 --2- 172.21.15.158:0/4224273213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0cff30 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0b5050 tx=0x7f22ec009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 -- 172.21.15.158:0/4224273213 shutdown_connections 2023-10-16T11:19:54.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 --2- 172.21.15.158:0/4224273213 >> v2:172.21.15.172:3300/0 conn(0x7f22fc0cf360 0x7f22fc0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 --2- 172.21.15.158:0/4224273213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0cff30 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 -- 172.21.15.158:0/4224273213 >> 172.21.15.158:0/4224273213 conn(0x7f22fc09c940 msgr2=0x7f22fc0d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:54.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 -- 172.21.15.158:0/4224273213 shutdown_connections 2023-10-16T11:19:54.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.578+0000 7f2303c60700 1 -- 172.21.15.158:0/4224273213 wait complete. 2023-10-16T11:19:54.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.579+0000 7f2303c60700 1 Processor -- start 2023-10-16T11:19:54.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.579+0000 7f2303c60700 1 -- start start 2023-10-16T11:19:54.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc0cf360 0x7f22fc151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc15daa0 0x7f22fc0b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0b67a0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0c70e0 con 0x7f22fc0cf360 2023-10-16T11:19:54.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0b8710 con 0x7f22fc15daa0 2023-10-16T11:19:54.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0b4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38998/0 (socket says 172.21.15.158:38998) 2023-10-16T11:19:54.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 -- 172.21.15.158:0/3896895541 learned_addr learned my addr 172.21.15.158:0/3896895541 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:54.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23019fc700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc0cf360 0x7f22fc151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23021fd700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc15daa0 0x7f22fc0b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 -- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc15daa0 msgr2=0x7f22fc0b6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc15daa0 0x7f22fc0b6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 -- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc0cf360 msgr2=0x7f22fc151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc0cf360 0x7f22fc151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 -- 172.21.15.158:0/3896895541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec009550 con 0x7f22fc15d6c0 2023-10-16T11:19:54.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23021fd700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc15daa0 0x7f22fc0b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f23011fb700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0b4750 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c420 tx=0x7f22f800c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f22f27fc700 1 -- 172.21.15.158:0/3896895541 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f80052d0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f22f27fc700 1 -- 172.21.15.158:0/3896895541 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8005430 con 0x7f22fc15d6c0 2023-10-16T11:19:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc0b89f0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f22f27fc700 1 -- 172.21.15.158:0/3896895541 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f80038c0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.580+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc0b8fb0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.581+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc0d2350 con 0x7f22fc15d6c0 2023-10-16T11:19:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.582+0000 7f22f27fc700 1 -- 172.21.15.158:0/3896895541 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f22f8003a20 con 0x7f22fc15d6c0 2023-10-16T11:19:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.582+0000 7f22f27fc700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f22e006add0 0x7f22e006d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.582+0000 7f22f27fc700 1 -- 172.21.15.158:0/3896895541 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f22f8088dc0 con 0x7f22fc15d6c0 2023-10-16T11:19:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.582+0000 7f23019fc700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f22e006add0 0x7f22e006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:54.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.582+0000 7f23019fc700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f22e006add0 0x7f22e006d290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22ec000c00 tx=0x7f22ec008030 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.586+0000 7f22f27fc700 1 -- 172.21.15.158:0/3896895541 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22fc0d2350 con 0x7f22fc15d6c0 2023-10-16T11:19:54.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.805+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f22fc0d1b70 con 0x7f22e006add0 2023-10-16T11:19:54.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.856+0000 7f22f27fc700 1 -- 172.21.15.158:0/3896895541 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f22ec00bb40 con 0x7f22e006add0 2023-10-16T11:19:54.858 INFO:teuthology.orchestra.run.smithi158.stdout:Retention added to path mandatory_fs_argument_test_dir 2023-10-16T11:19:54.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f22e006add0 msgr2=0x7f22e006d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f22e006add0 0x7f22e006d290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22ec000c00 tx=0x7f22ec008030 comp rx=0 tx=0).stop 2023-10-16T11:19:54.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 msgr2=0x7f22fc0b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:54.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0b4750 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c420 tx=0x7f22f800c730 comp rx=0 tx=0).stop 2023-10-16T11:19:54.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 shutdown_connections 2023-10-16T11:19:54.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f22e006add0 0x7f22e006d290 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22fc15daa0 0x7f22fc0b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22fc0cf360 0x7f22fc151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 --2- 172.21.15.158:0/3896895541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22fc15d6c0 0x7f22fc0b4750 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:54.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 >> 172.21.15.158:0/3896895541 conn(0x7f22fc09c940 msgr2=0x7f22fc15bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:54.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 shutdown_connections 2023-10-16T11:19:54.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:54.859+0000 7f2303c60700 1 -- 172.21.15.158:0/3896895541 wait complete. 2023-10-16T11:19:54.873 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2023-10-16T11:19:54.873 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T11:19:55.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.002+0000 7fc9af51b700 1 Processor -- start 2023-10-16T11:19:55.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.002+0000 7fc9af51b700 1 -- start start 2023-10-16T11:19:55.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9af51b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a80b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9af51b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9a80b3460 0x7fc9a80b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9af51b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80b4420 con 0x7fc9a815b6b0 2023-10-16T11:19:55.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9af51b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a815ba90 con 0x7fc9a80b3460 2023-10-16T11:19:55.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9af51b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a80b38f0 con 0x7fc9a80d0c60 2023-10-16T11:19:55.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9acab6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a80b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9adab8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9a80b3460 0x7fc9a80b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9acab6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a80b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:49854/0 (socket says 172.21.15.158:49854) 2023-10-16T11:19:55.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9ad2b7700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc9a815b6b0 0x7fc9a80cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:43098/0 (socket says 172.21.15.158:43098) 2023-10-16T11:19:55.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.003+0000 7fc9acab6700 1 -- 172.21.15.158:0/4214089040 learned_addr learned my addr 172.21.15.158:0/4214089040 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:55.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc9adab8700 1 -- 172.21.15.158:0/4214089040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 msgr2=0x7fc9a80b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:55.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc9adab8700 1 --2- 172.21.15.158:0/4214089040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a80b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc9adab8700 1 -- 172.21.15.158:0/4214089040 >> v1:172.21.15.172:6789/0 conn(0x7fc9a815b6b0 legacy=0x7fc9a80cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc9adab8700 1 -- 172.21.15.158:0/4214089040 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80cbc20 con 0x7fc9a80b3460 2023-10-16T11:19:55.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc99ffff700 1 -- 172.21.15.158:0/4214089040 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742395689 0 0) 0x7fc9a80b4420 con 0x7fc9a815b6b0 2023-10-16T11:19:55.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc9adab8700 1 --2- 172.21.15.158:0/4214089040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 0x7fc9a80b48b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009d90 tx=0x7fc9a000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26354020cb35d562 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc99ffff700 1 -- 172.21.15.158:0/4214089040 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000e070 con 0x7fc9a80b3460 2023-10-16T11:19:55.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc99ffff700 1 -- 172.21.15.158:0/4214089040 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a000bb20 con 0x7fc9a80b3460 2023-10-16T11:19:55.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.004+0000 7fc99ffff700 1 -- 172.21.15.158:0/4214089040 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0012430 con 0x7fc9a80b3460 2023-10-16T11:19:55.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 -- 172.21.15.158:0/4214089040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 msgr2=0x7fc9a80b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 --2- 172.21.15.158:0/4214089040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 0x7fc9a80b48b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009d90 tx=0x7fc9a000b330 comp rx=0 tx=0).stop 2023-10-16T11:19:55.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 -- 172.21.15.158:0/4214089040 shutdown_connections 2023-10-16T11:19:55.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 --2- 172.21.15.158:0/4214089040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 0x7fc9a80b48b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 --2- 172.21.15.158:0/4214089040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a80b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 -- 172.21.15.158:0/4214089040 >> 172.21.15.158:0/4214089040 conn(0x7fc9a80cdf20 msgr2=0x7fc9a80ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:55.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 -- 172.21.15.158:0/4214089040 shutdown_connections 2023-10-16T11:19:55.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.005+0000 7fc9af51b700 1 -- 172.21.15.158:0/4214089040 wait complete. 2023-10-16T11:19:55.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.006+0000 7fc9af51b700 1 Processor -- start 2023-10-16T11:19:55.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.006+0000 7fc9af51b700 1 -- start start 2023-10-16T11:19:55.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.007+0000 7fc9af51b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 0x7fc9a80aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.007+0000 7fc9af51b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a8158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.007+0000 7fc9af51b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9a815b6b0 0x7fc9a8158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.007+0000 7fc9af51b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8158e90 con 0x7fc9a815b6b0 2023-10-16T11:19:55.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.007+0000 7fc9af51b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8159340 con 0x7fc9a80b3460 2023-10-16T11:19:55.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.007+0000 7fc9af51b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a80a9270 con 0x7fc9a80d0c60 2023-10-16T11:19:55.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.007+0000 7fc9adab8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9a815b6b0 0x7fc9a8158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9a815b6b0 0x7fc9a8158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39028/0 (socket says 172.21.15.158:39028) 2023-10-16T11:19:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 -- 172.21.15.158:0/1108315712 learned_addr learned my addr 172.21.15.158:0/1108315712 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9acab6700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a8158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9ad2b7700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 0x7fc9a80aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 -- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 msgr2=0x7fc9a8158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a8158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 -- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 msgr2=0x7fc9a80aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 0x7fc9a80aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 -- 172.21.15.158:0/1108315712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0009a70 con 0x7fc9a815b6b0 2023-10-16T11:19:55.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9acab6700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a8158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:55.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9adab8700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9a815b6b0 0x7fc9a8158950 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fc9a80675e0 tx=0x7fc9a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc99dffb700 1 -- 172.21.15.158:0/1108315712 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0012430 con 0x7fc9a815b6b0 2023-10-16T11:19:55.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc99dffb700 1 -- 172.21.15.158:0/1108315712 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a0020070 con 0x7fc9a815b6b0 2023-10-16T11:19:55.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a80a9550 con 0x7fc9a815b6b0 2023-10-16T11:19:55.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc99dffb700 1 -- 172.21.15.158:0/1108315712 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0004280 con 0x7fc9a815b6b0 2023-10-16T11:19:55.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.008+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a80a9b10 con 0x7fc9a815b6b0 2023-10-16T11:19:55.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.009+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a80a9eb0 con 0x7fc9a815b6b0 2023-10-16T11:19:55.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.010+0000 7fc99dffb700 1 -- 172.21.15.158:0/1108315712 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc9a00043e0 con 0x7fc9a815b6b0 2023-10-16T11:19:55.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.010+0000 7fc99dffb700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc98c06ada0 0x7fc98c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.011+0000 7fc99dffb700 1 -- 172.21.15.158:0/1108315712 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fc9a0022070 con 0x7fc9a815b6b0 2023-10-16T11:19:55.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.011+0000 7fc9ad2b7700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc98c06ada0 0x7fc98c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.011+0000 7fc9ad2b7700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc98c06ada0 0x7fc98c06d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4000f60 tx=0x7fc9a4007c90 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.015+0000 7fc99dffb700 1 -- 172.21.15.158:0/1108315712 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9a0056820 con 0x7fc9a815b6b0 2023-10-16T11:19:55.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.231+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9a80cc1e0 con 0x7fc98c06ada0 2023-10-16T11:19:55.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.236+0000 7fc99dffb700 1 -- 172.21.15.158:0/1108315712 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7fc9a4013420 con 0x7fc98c06ada0 2023-10-16T11:19:55.238 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:55.238 INFO:teuthology.orchestra.run.smithi158.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T11:19:55.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc98c06ada0 msgr2=0x7fc98c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc98c06ada0 0x7fc98c06d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4000f60 tx=0x7fc9a4007c90 comp rx=0 tx=0).stop 2023-10-16T11:19:55.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9a815b6b0 msgr2=0x7fc9a8158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9a815b6b0 0x7fc9a8158950 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fc9a80675e0 tx=0x7fc9a0004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 shutdown_connections 2023-10-16T11:19:55.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc98c06ada0 0x7fc98c06d260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9a815b6b0 0x7fc9a8158950 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9a80b3460 0x7fc9a80aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 --2- 172.21.15.158:0/1108315712 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9a80d0c60 0x7fc9a8158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 >> 172.21.15.158:0/1108315712 conn(0x7fc9a80cdf20 msgr2=0x7fc9a80d1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:55.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 shutdown_connections 2023-10-16T11:19:55.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.239+0000 7fc9af51b700 1 -- 172.21.15.158:0/1108315712 wait complete. 2023-10-16T11:19:55.252 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T11:19:55.253 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T11:19:55.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.378+0000 7f629010d700 1 Processor -- start 2023-10-16T11:19:55.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.379+0000 7f629010d700 1 -- start start 2023-10-16T11:19:55.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f629010d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6288163380 0x7f6288165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f629010d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62880d0ae0 con 0x7f628815deb0 2023-10-16T11:19:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f629010d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62880d12d0 con 0x7f628815dad0 2023-10-16T11:19:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f629010d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62880cbdf0 con 0x7f6288163380 2023-10-16T11:19:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628d6a8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f628815deb0 0x7f62880ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39210/0 (socket says 172.21.15.158:39210) 2023-10-16T11:19:55.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628d6a8700 1 -- 172.21.15.158:0/3931156423 learned_addr learned my addr 172.21.15.158:0/3931156423 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628e6aa700 1 --2- 172.21.15.158:0/3931156423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6288163380 0x7f6288165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628cea7700 1 -- 172.21.15.158:0/3931156423 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566128301 0 0) 0x7f62880d0ae0 con 0x7f628815deb0 2023-10-16T11:19:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628cea7700 1 -- 172.21.15.158:0/3931156423 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62740035b0 con 0x7f628815deb0 2023-10-16T11:19:55.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628cea7700 1 -- 172.21.15.158:0/3931156423 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953345237 0 0) 0x7f62880d12d0 con 0x7f628815dad0 2023-10-16T11:19:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628cea7700 1 -- 172.21.15.158:0/3931156423 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62880d0ae0 con 0x7f628815dad0 2023-10-16T11:19:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628e6aa700 1 -- 172.21.15.158:0/3931156423 >> v1:172.21.15.172:6789/0 conn(0x7f628815dad0 legacy=0x7f62880cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628e6aa700 1 -- 172.21.15.158:0/3931156423 >> v1:172.21.15.158:6789/0 conn(0x7f628815deb0 legacy=0x7f62880ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628e6aa700 1 -- 172.21.15.158:0/3931156423 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62880cc280 con 0x7f6288163380 2023-10-16T11:19:55.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.380+0000 7f628e6aa700 1 --2- 172.21.15.158:0/3931156423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6288163380 0x7f6288165770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6280009dc0 tx=0x7f628000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9581c1e79dbde851 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f628cea7700 1 -- 172.21.15.158:0/3931156423 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628000e070 con 0x7f6288163380 2023-10-16T11:19:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f628cea7700 1 -- 172.21.15.158:0/3931156423 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628000bb20 con 0x7f6288163380 2023-10-16T11:19:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f628cea7700 1 -- 172.21.15.158:0/3931156423 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280012430 con 0x7f6288163380 2023-10-16T11:19:55.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f629010d700 1 -- 172.21.15.158:0/3931156423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6288163380 msgr2=0x7f6288165770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f629010d700 1 --2- 172.21.15.158:0/3931156423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6288163380 0x7f6288165770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6280009dc0 tx=0x7f628000b330 comp rx=0 tx=0).stop 2023-10-16T11:19:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f629010d700 1 -- 172.21.15.158:0/3931156423 shutdown_connections 2023-10-16T11:19:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f629010d700 1 --2- 172.21.15.158:0/3931156423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6288163380 0x7f6288165770 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f629010d700 1 -- 172.21.15.158:0/3931156423 >> 172.21.15.158:0/3931156423 conn(0x7f6288099a10 msgr2=0x7f62880d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:55.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f629010d700 1 -- 172.21.15.158:0/3931156423 shutdown_connections 2023-10-16T11:19:55.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.381+0000 7f629010d700 1 -- 172.21.15.158:0/3931156423 wait complete. 2023-10-16T11:19:55.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.382+0000 7f629010d700 1 Processor -- start 2023-10-16T11:19:55.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.382+0000 7f629010d700 1 -- start start 2023-10-16T11:19:55.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f629010d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f628815dad0 0x7f6288149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f629010d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f628815deb0 0x7f62880c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f629010d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6288163380 0x7f62880c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f629010d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62880c9890 con 0x7f6288163380 2023-10-16T11:19:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f629010d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62880ca6c0 con 0x7f628815deb0 2023-10-16T11:19:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f629010d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62880c9ab0 con 0x7f628815dad0 2023-10-16T11:19:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f628e6aa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6288163380 0x7f62880c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f628e6aa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6288163380 0x7f62880c9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39042/0 (socket says 172.21.15.158:39042) 2023-10-16T11:19:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f628e6aa700 1 -- 172.21.15.158:0/1240317930 learned_addr learned my addr 172.21.15.158:0/1240317930 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f628dea9700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f628815dad0 0x7f6288149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.383+0000 7f628d6a8700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f628815deb0 0x7f62880c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f628e6aa700 1 -- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f628815dad0 msgr2=0x7f6288149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f628e6aa700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f628815dad0 0x7f6288149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f628e6aa700 1 -- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f628815deb0 msgr2=0x7f62880c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f628e6aa700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f628815deb0 0x7f62880c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f628e6aa700 1 -- 172.21.15.158:0/1240317930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280009a70 con 0x7f6288163380 2023-10-16T11:19:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f628e6aa700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6288163380 0x7f62880c9350 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f6280014040 tx=0x7f6280004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f627e7fc700 1 -- 172.21.15.158:0/1240317930 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628001c810 con 0x7f6288163380 2023-10-16T11:19:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f627e7fc700 1 -- 172.21.15.158:0/1240317930 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6280020070 con 0x7f6288163380 2023-10-16T11:19:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f627e7fc700 1 -- 172.21.15.158:0/1240317930 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628001cbb0 con 0x7f6288163380 2023-10-16T11:19:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62880c9d90 con 0x7f6288163380 2023-10-16T11:19:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.384+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62880ca350 con 0x7f6288163380 2023-10-16T11:19:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.385+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62880ca4e0 con 0x7f6288163380 2023-10-16T11:19:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.385+0000 7f627e7fc700 1 -- 172.21.15.158:0/1240317930 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f628001cd10 con 0x7f6288163380 2023-10-16T11:19:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.386+0000 7f627e7fc700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f626c06ada0 0x7f626c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.386+0000 7f627e7fc700 1 -- 172.21.15.158:0/1240317930 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f6280022070 con 0x7f6288163380 2023-10-16T11:19:55.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.386+0000 7f628dea9700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f626c06ada0 0x7f626c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.386+0000 7f628dea9700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f626c06ada0 0x7f626c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f62780011e0 tx=0x7f6278009040 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.390+0000 7f627e7fc700 1 -- 172.21.15.158:0/1240317930 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6280056580 con 0x7f6288163380 2023-10-16T11:19:55.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.605+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62880d0d10 con 0x7f626c06ada0 2023-10-16T11:19:55.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.611+0000 7f627e7fc700 1 -- 172.21.15.158:0/1240317930 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7f6278001d50 con 0x7f626c06ada0 2023-10-16T11:19:55.613 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:55.613 INFO:teuthology.orchestra.run.smithi158.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T11:19:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T11:19:55.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f626c06ada0 msgr2=0x7f626c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f626c06ada0 0x7f626c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f62780011e0 tx=0x7f6278009040 comp rx=0 tx=0).stop 2023-10-16T11:19:55.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6288163380 msgr2=0x7f62880c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6288163380 0x7f62880c9350 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f6280014040 tx=0x7f6280004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 shutdown_connections 2023-10-16T11:19:55.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f626c06ada0 0x7f626c06d260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6288163380 0x7f62880c9350 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f628815dad0 0x7f6288149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 --2- 172.21.15.158:0/1240317930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f628815deb0 0x7f62880c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 >> 172.21.15.158:0/1240317930 conn(0x7f6288099a10 msgr2=0x7f62880cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:55.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 shutdown_connections 2023-10-16T11:19:55.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.614+0000 7f629010d700 1 -- 172.21.15.158:0/1240317930 wait complete. 2023-10-16T11:19:55.627 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T11:19:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T11:19:55.628 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T11:19:55.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.753+0000 7f9df95f5700 1 Processor -- start 2023-10-16T11:19:55.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.753+0000 7f9df95f5700 1 -- start start 2023-10-16T11:19:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df95f5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 0x7f9df40b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df95f5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df40b7a20 con 0x7f9df40b7590 2023-10-16T11:19:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df95f5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df40b7c90 con 0x7f9df40d0530 2023-10-16T11:19:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df95f5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40b8910 con 0x7f9df4160d30 2023-10-16T11:19:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df37fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9df40b7590 0x7f9df40d20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39226/0 (socket says 172.21.15.158:39226) 2023-10-16T11:19:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df37fe700 1 -- 172.21.15.158:0/1681496377 learned_addr learned my addr 172.21.15.158:0/1681496377 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698212620 0 0) 0x7f9df40b7a20 con 0x7f9df40b7590 2023-10-16T11:19:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ddc0035b0 con 0x7f9df40b7590 2023-10-16T11:19:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 0x7f9df40b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304668122 0 0) 0x7f9ddc0035b0 con 0x7f9df40b7590 2023-10-16T11:19:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 msgr2=0x7f9df40b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 0x7f9df40b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 >> v1:172.21.15.172:6789/0 conn(0x7f9df40d0530 legacy=0x7f9df40d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df40b7f70 con 0x7f9df40b7590 2023-10-16T11:19:55.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.754+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f9dec0019f0 con 0x7f9df40b7590 2023-10-16T11:19:55.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 >> v1:172.21.15.158:6789/0 conn(0x7f9df40b7590 legacy=0x7f9df40d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ddc003a30 0x7f9ddc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc004850 0x7f9ddc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 0x7f9ddc005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40b8910 con 0x7f9ddc003a30 2023-10-16T11:19:55.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc00bb40 con 0x7f9ddc004850 2023-10-16T11:19:55.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc00bd00 con 0x7f9df4160d30 2023-10-16T11:19:55.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ddc003a30 0x7f9ddc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df27fc700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc004850 0x7f9ddc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df37fe700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 0x7f9ddc005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df2ffd700 1 -- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 msgr2=0x7f9ddc005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:55.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df4160d30 0x7f9ddc005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df2ffd700 1 -- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc004850 msgr2=0x7f9ddc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:55.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc004850 0x7f9ddc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df2ffd700 1 -- 172.21.15.158:0/1681496377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc0035b0 con 0x7f9ddc003a30 2023-10-16T11:19:55.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ddc003a30 0x7f9ddc003e50 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009b60 tx=0x7f9ddc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa8ae3725eb3d24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de800e6c0 con 0x7f9ddc003a30 2023-10-16T11:19:55.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de800e820 con 0x7f9ddc003a30 2023-10-16T11:19:55.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.755+0000 7f9df1ffb700 1 -- 172.21.15.158:0/1681496377 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de800eb90 con 0x7f9ddc003a30 2023-10-16T11:19:55.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.756+0000 7f9df95f5700 1 -- 172.21.15.158:0/1681496377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ddc003a30 msgr2=0x7f9ddc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.756+0000 7f9df95f5700 1 --2- 172.21.15.158:0/1681496377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ddc003a30 0x7f9ddc003e50 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009b60 tx=0x7f9ddc005880 comp rx=0 tx=0).stop 2023-10-16T11:19:55.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.756+0000 7f9df37fe700 1 -- 172.21.15.158:0/1681496377 reap_dead start 2023-10-16T11:19:55.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.756+0000 7f9df95f5700 1 -- 172.21.15.158:0/1681496377 shutdown_connections 2023-10-16T11:19:55.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.756+0000 7f9df95f5700 1 -- 172.21.15.158:0/1681496377 >> 172.21.15.158:0/1681496377 conn(0x7f9df40669f0 msgr2=0x7f9df4066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:55.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.756+0000 7f9df95f5700 1 -- 172.21.15.158:0/1681496377 shutdown_connections 2023-10-16T11:19:55.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.756+0000 7f9df95f5700 1 -- 172.21.15.158:0/1681496377 wait complete. 2023-10-16T11:19:55.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.757+0000 7f9df95f5700 1 Processor -- start 2023-10-16T11:19:55.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.757+0000 7f9df95f5700 1 -- start start 2023-10-16T11:19:55.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df95f5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc003a30 0x7f9df414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df95f5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df40c7cc0 0x7f9df40c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df95f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df4159300 0x7f9df4159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df95f5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40c8680 con 0x7f9df4159300 2023-10-16T11:19:55.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df95f5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4159cc0 con 0x7f9ddc003a30 2023-10-16T11:19:55.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df95f5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41561a0 con 0x7f9df40c7cc0 2023-10-16T11:19:55.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df4159300 0x7f9df4159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df4159300 0x7f9df4159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39060/0 (socket says 172.21.15.158:39060) 2023-10-16T11:19:55.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 -- 172.21.15.158:0/2061786525 learned_addr learned my addr 172.21.15.158:0/2061786525 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:55.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df27fc700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df40c7cc0 0x7f9df40c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc003a30 0x7f9df414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 -- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df40c7cc0 msgr2=0x7f9df40c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df40c7cc0 0x7f9df40c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 -- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc003a30 msgr2=0x7f9df414fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:55.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc003a30 0x7f9df414fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:55.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 -- 172.21.15.158:0/2061786525 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df40c94b0 con 0x7f9df4159300 2023-10-16T11:19:55.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.758+0000 7f9df37fe700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df4159300 0x7f9df4159780 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0047d0 tx=0x7f9dec00a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.759+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00adf0 con 0x7f9df4159300 2023-10-16T11:19:55.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.759+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec0133f0 con 0x7f9df4159300 2023-10-16T11:19:55.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.759+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df40c88a0 con 0x7f9df4159300 2023-10-16T11:19:55.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.759+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec013760 con 0x7f9df4159300 2023-10-16T11:19:55.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.759+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df40c8e60 con 0x7f9df4159300 2023-10-16T11:19:55.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.760+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df40c9640 con 0x7f9df4159300 2023-10-16T11:19:55.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.760+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9dec0138c0 con 0x7f9df4159300 2023-10-16T11:19:55.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.761+0000 7f9ddb7fe700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f9de006ada0 0x7f9de006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:55.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.761+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f9dec08a3d0 con 0x7f9df4159300 2023-10-16T11:19:55.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.761+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f9de006ada0 0x7f9de006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:55.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.761+0000 7f9df2ffd700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f9de006ada0 0x7f9de006d260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9de80090b0 tx=0x7f9de8008030 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:55.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.765+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9dec054fe0 con 0x7f9df4159300 2023-10-16T11:19:55.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.941+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 142) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f9dec011d90 con 0x7f9df4159300 2023-10-16T11:19:55.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:55.990+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9df40b8250 con 0x7f9de006ada0 2023-10-16T11:19:56.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.050+0000 7f9ddb7fe700 1 -- 172.21.15.158:0/2061786525 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9de800e990 con 0x7f9de006ada0 2023-10-16T11:19:56.053 INFO:teuthology.orchestra.run.smithi158.stdout:Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T11:19:56.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f9de006ada0 msgr2=0x7f9de006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f9de006ada0 0x7f9de006d260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9de80090b0 tx=0x7f9de8008030 comp rx=0 tx=0).stop 2023-10-16T11:19:56.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df4159300 msgr2=0x7f9df4159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df4159300 0x7f9df4159780 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0047d0 tx=0x7f9dec00a5f0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 shutdown_connections 2023-10-16T11:19:56.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f9de006ada0 0x7f9de006d260 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df4159300 0x7f9df4159780 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ddc003a30 0x7f9df414fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 --2- 172.21.15.158:0/2061786525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9df40c7cc0 0x7f9df40c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 >> 172.21.15.158:0/2061786525 conn(0x7f9df40669f0 msgr2=0x7f9df40b6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:56.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 shutdown_connections 2023-10-16T11:19:56.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.053+0000 7f9df95f5700 1 -- 172.21.15.158:0/2061786525 wait complete. 2023-10-16T11:19:56.067 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T11:19:56.067 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T11:19:56.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.192+0000 7fe6ab314700 1 Processor -- start 2023-10-16T11:19:56.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.192+0000 7fe6ab314700 1 -- start start 2023-10-16T11:19:56.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.193+0000 7fe6ab314700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 0x7fe6a40cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.193+0000 7fe6ab314700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe6a40ccc00 0x7fe6a40cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.193+0000 7fe6ab314700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a40d0ae0 con 0x7fe6a415deb0 2023-10-16T11:19:56.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.193+0000 7fe6ab314700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a40d12d0 con 0x7fe6a40ccc00 2023-10-16T11:19:56.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.193+0000 7fe6ab314700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40cbdf0 con 0x7fe6a415dad0 2023-10-16T11:19:56.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe6a88af700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe6a415deb0 0x7fe6a40ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39244/0 (socket says 172.21.15.158:39244) 2023-10-16T11:19:56.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe6a88af700 1 -- 172.21.15.158:0/3697199988 learned_addr learned my addr 172.21.15.158:0/3697199988 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:56.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 0x7fe6a40cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe6a98b1700 1 --2- 172.21.15.158:0/3697199988 >> v2:172.21.15.172:3300/0 conn(0x7fe6a40ccc00 0x7fe6a40cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699720908 0 0) 0x7fe6a40d0ae0 con 0x7fe6a415deb0 2023-10-16T11:19:56.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6900035b0 con 0x7fe6a415deb0 2023-10-16T11:19:56.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431190720 0 0) 0x7fe6900035b0 con 0x7fe6a415deb0 2023-10-16T11:19:56.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 msgr2=0x7fe6a40cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:56.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 0x7fe6a40cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 >> v2:172.21.15.172:3300/0 conn(0x7fe6a40ccc00 msgr2=0x7fe6a40cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:56.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 --2- 172.21.15.158:0/3697199988 >> v2:172.21.15.172:3300/0 conn(0x7fe6a40ccc00 0x7fe6a40cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a40cc280 con 0x7fe6a415deb0 2023-10-16T11:19:56.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe694001670 con 0x7fe6a415deb0 2023-10-16T11:19:56.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 >> v1:172.21.15.158:6789/0 conn(0x7fe6a415deb0 legacy=0x7fe6a40ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe690003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 0x7fe690004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe690004d90 0x7fe690005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40d12d0 con 0x7fe690004d90 2023-10-16T11:19:56.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40cbdf0 con 0x7fe690003a00 2023-10-16T11:19:56.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69000ba10 con 0x7fe6a415dad0 2023-10-16T11:19:56.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe690003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe6a98b1700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe690004d90 0x7fe690005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.194+0000 7fe6a88af700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 0x7fe690004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a90b0700 1 -- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 msgr2=0x7fe690004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:56.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a415dad0 0x7fe690004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a90b0700 1 -- 172.21.15.158:0/3697199988 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe690004d90 msgr2=0x7fe690005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:56.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe690004d90 0x7fe690005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a90b0700 1 -- 172.21.15.158:0/3697199988 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900035b0 con 0x7fe690003a00 2023-10-16T11:19:56.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a98b1700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe690004d90 0x7fe690005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:56.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe690003e20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009b60 tx=0x7fe690005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93c22df113a5fdc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:56.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a000e6c0 con 0x7fe690003a00 2023-10-16T11:19:56.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a000e820 con 0x7fe690003a00 2023-10-16T11:19:56.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe69bfff700 1 -- 172.21.15.158:0/3697199988 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a000eb90 con 0x7fe690003a00 2023-10-16T11:19:56.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6ab314700 1 -- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 msgr2=0x7fe690003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6ab314700 1 --2- 172.21.15.158:0/3697199988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe690003e20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009b60 tx=0x7fe690005880 comp rx=0 tx=0).stop 2023-10-16T11:19:56.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6a98b1700 1 -- 172.21.15.158:0/3697199988 reap_dead start 2023-10-16T11:19:56.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6ab314700 1 -- 172.21.15.158:0/3697199988 shutdown_connections 2023-10-16T11:19:56.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.195+0000 7fe6ab314700 1 -- 172.21.15.158:0/3697199988 >> 172.21.15.158:0/3697199988 conn(0x7fe6a4099a10 msgr2=0x7fe6a40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:56.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.196+0000 7fe6ab314700 1 -- 172.21.15.158:0/3697199988 shutdown_connections 2023-10-16T11:19:56.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.196+0000 7fe6ab314700 1 -- 172.21.15.158:0/3697199988 wait complete. 2023-10-16T11:19:56.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.197+0000 7fe6ab314700 1 Processor -- start 2023-10-16T11:19:56.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.197+0000 7fe6ab314700 1 -- start start 2023-10-16T11:19:56.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6ab314700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe6a415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6ab314700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6a40afd10 0x7fe6a40b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6ab314700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a40addf0 0x7fe6a40ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6ab314700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40af6f0 con 0x7fe6a40afd10 2023-10-16T11:19:56.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6ab314700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40c8d20 con 0x7fe690003a00 2023-10-16T11:19:56.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6ab314700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40b06d0 con 0x7fe6a40addf0 2023-10-16T11:19:56.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6a40afd10 0x7fe6a40b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6a40afd10 0x7fe6a40b0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39080/0 (socket says 172.21.15.158:39080) 2023-10-16T11:19:56.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 -- 172.21.15.158:0/1805076910 learned_addr learned my addr 172.21.15.158:0/1805076910 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:56.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a98b1700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a40addf0 0x7fe6a40ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe6a415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 -- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a40addf0 msgr2=0x7fe6a40ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a40addf0 0x7fe6a40ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 -- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 msgr2=0x7fe6a415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe6a415aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 -- 172.21.15.158:0/1805076910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a40ae7c0 con 0x7fe6a40afd10 2023-10-16T11:19:56.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6a88af700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6a40afd10 0x7fe6a40b0190 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7fe694007340 tx=0x7fe694007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:56.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe699ffb700 1 -- 172.21.15.158:0/1805076910 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69400f6c0 con 0x7fe6a40afd10 2023-10-16T11:19:56.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe699ffb700 1 -- 172.21.15.158:0/1805076910 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69400f820 con 0x7fe6a40afd10 2023-10-16T11:19:56.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a40aea70 con 0x7fe6a40afd10 2023-10-16T11:19:56.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.198+0000 7fe699ffb700 1 -- 172.21.15.158:0/1805076910 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69400fbd0 con 0x7fe6a40afd10 2023-10-16T11:19:56.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.199+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a40af030 con 0x7fe6a40afd10 2023-10-16T11:19:56.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.200+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a40681b0 con 0x7fe6a40afd10 2023-10-16T11:19:56.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.200+0000 7fe699ffb700 1 -- 172.21.15.158:0/1805076910 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 142) v1 ==== 89451+0+0 (secure 0 0 0) 0x7fe69400fd30 con 0x7fe6a40afd10 2023-10-16T11:19:56.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.200+0000 7fe699ffb700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe68806b190 0x7fe68806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.200+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe68806b190 0x7fe68806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.200+0000 7fe699ffb700 1 -- 172.21.15.158:0/1805076910 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fe69408a190 con 0x7fe6a40afd10 2023-10-16T11:19:56.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.201+0000 7fe6a90b0700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe68806b190 0x7fe68806d650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe6a00090b0 tx=0x7fe6a001f040 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:56.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.205+0000 7fe699ffb700 1 -- 172.21.15.158:0/1805076910 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6940547b0 con 0x7fe6a40afd10 2023-10-16T11:19:56.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.427+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6a40d0ff0 con 0x7fe68806b190 2023-10-16T11:19:56.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.489+0000 7fe699ffb700 1 -- 172.21.15.158:0/1805076910 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7fe6a000ea90 con 0x7fe68806b190 2023-10-16T11:19:56.491 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T11:19:56.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe68806b190 msgr2=0x7fe68806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe68806b190 0x7fe68806d650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe6a00090b0 tx=0x7fe6a001f040 comp rx=0 tx=0).stop 2023-10-16T11:19:56.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6a40afd10 msgr2=0x7fe6a40b0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6a40afd10 0x7fe6a40b0190 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7fe694007340 tx=0x7fe694007650 comp rx=0 tx=0).stop 2023-10-16T11:19:56.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 shutdown_connections 2023-10-16T11:19:56.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe68806b190 0x7fe68806d650 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe6a40addf0 0x7fe6a40ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe690003a00 0x7fe6a415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 --2- 172.21.15.158:0/1805076910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6a40afd10 0x7fe6a40b0190 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 >> 172.21.15.158:0/1805076910 conn(0x7fe6a4099a10 msgr2=0x7fe6a415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:56.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 shutdown_connections 2023-10-16T11:19:56.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.491+0000 7fe6ab314700 1 -- 172.21.15.158:0/1805076910 wait complete. 2023-10-16T11:19:56.504 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T11:19:56.505 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T11:19:56.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.628+0000 7f2193f43700 1 Processor -- start 2023-10-16T11:19:56.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.629+0000 7f2193f43700 1 -- start start 2023-10-16T11:19:56.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.629+0000 7f2193f43700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f218c160d40 0x7f218c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.629+0000 7f2193f43700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f218c0c50c0 0x7f218c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.629+0000 7f2193f43700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c0ce110 con 0x7f218c0cf420 2023-10-16T11:19:56.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.629+0000 7f2193f43700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218c0c61e0 con 0x7f218c0c50c0 2023-10-16T11:19:56.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.629+0000 7f2193f43700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218c0ce3f0 con 0x7f218c160d40 2023-10-16T11:19:56.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21924e0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f218c0c50c0 0x7f218c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21914de700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f218c0cf420 0x7f218c0c49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39262/0 (socket says 172.21.15.158:39262) 2023-10-16T11:19:56.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21914de700 1 -- 172.21.15.158:0/2490844302 learned_addr learned my addr 172.21.15.158:0/2490844302 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:56.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f2191cdf700 1 --2- 172.21.15.158:0/2490844302 >> v2:172.21.15.172:3300/0 conn(0x7f218c160d40 0x7f218c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21924e0700 1 -- 172.21.15.158:0/2490844302 >> v2:172.21.15.172:3300/0 conn(0x7f218c160d40 msgr2=0x7f218c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:56.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21924e0700 1 --2- 172.21.15.158:0/2490844302 >> v2:172.21.15.172:3300/0 conn(0x7f218c160d40 0x7f218c0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21924e0700 1 -- 172.21.15.158:0/2490844302 >> v1:172.21.15.158:6789/0 conn(0x7f218c0cf420 legacy=0x7f218c0c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21924e0700 1 -- 172.21.15.158:0/2490844302 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0ce6d0 con 0x7f218c0c50c0 2023-10-16T11:19:56.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f2190cdd700 1 -- 172.21.15.158:0/2490844302 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749051776 0 0) 0x7f218c0ce110 con 0x7f218c0cf420 2023-10-16T11:19:56.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f2191cdf700 1 --2- 172.21.15.158:0/2490844302 >> v2:172.21.15.172:3300/0 conn(0x7f218c160d40 0x7f218c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:19:56.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f21924e0700 1 --2- 172.21.15.158:0/2490844302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c0c50c0 0x7f218c06abb0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f218c0d0b80 tx=0x7f218400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c082a9e402b81ddd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:56.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f2190cdd700 1 -- 172.21.15.158:0/2490844302 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218400e070 con 0x7f218c0c50c0 2023-10-16T11:19:56.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f2190cdd700 1 -- 172.21.15.158:0/2490844302 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218400bbd0 con 0x7f218c0c50c0 2023-10-16T11:19:56.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.630+0000 7f2190cdd700 1 -- 172.21.15.158:0/2490844302 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184012430 con 0x7f218c0c50c0 2023-10-16T11:19:56.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 -- 172.21.15.158:0/2490844302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c0c50c0 msgr2=0x7f218c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 --2- 172.21.15.158:0/2490844302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c0c50c0 0x7f218c06abb0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f218c0d0b80 tx=0x7f218400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:56.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 -- 172.21.15.158:0/2490844302 shutdown_connections 2023-10-16T11:19:56.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 --2- 172.21.15.158:0/2490844302 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c0c50c0 0x7f218c06abb0 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 --2- 172.21.15.158:0/2490844302 >> v2:172.21.15.172:3300/0 conn(0x7f218c160d40 0x7f218c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 -- 172.21.15.158:0/2490844302 >> 172.21.15.158:0/2490844302 conn(0x7f218c0cc5b0 msgr2=0x7f218c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:56.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 -- 172.21.15.158:0/2490844302 shutdown_connections 2023-10-16T11:19:56.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.631+0000 7f2193f43700 1 -- 172.21.15.158:0/2490844302 wait complete. 2023-10-16T11:19:56.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.632+0000 7f2193f43700 1 Processor -- start 2023-10-16T11:19:56.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.632+0000 7f2193f43700 1 -- start start 2023-10-16T11:19:56.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f2193f43700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f218c0c50c0 0x7f218c1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f2193f43700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f218c0cf420 0x7f218c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f2193f43700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c160d40 0x7f218c157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f2193f43700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0adc80 con 0x7f218c160d40 2023-10-16T11:19:56.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f2193f43700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b2bb0 con 0x7f218c0c50c0 2023-10-16T11:19:56.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f2193f43700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b2e90 con 0x7f218c0cf420 2023-10-16T11:19:56.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c160d40 0x7f218c157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c160d40 0x7f218c157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39094/0 (socket says 172.21.15.158:39094) 2023-10-16T11:19:56.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 -- 172.21.15.158:0/2230621552 learned_addr learned my addr 172.21.15.158:0/2230621552 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:56.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f2191cdf700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f218c0c50c0 0x7f218c1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21914de700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f218c0cf420 0x7f218c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 -- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f218c0cf420 msgr2=0x7f218c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f218c0cf420 0x7f218c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 -- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f218c0c50c0 msgr2=0x7f218c1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f218c0c50c0 0x7f218c1556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 -- 172.21.15.158:0/2230621552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2184009a70 con 0x7f218c160d40 2023-10-16T11:19:56.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21924e0700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c160d40 0x7f218c157510 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f218400b370 tx=0x7f2184004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:56.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.633+0000 7f21827fc700 1 -- 172.21.15.158:0/2230621552 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184012430 con 0x7f218c160d40 2023-10-16T11:19:56.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.634+0000 7f21827fc700 1 -- 172.21.15.158:0/2230621552 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2184020070 con 0x7f218c160d40 2023-10-16T11:19:56.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.634+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c1612a0 con 0x7f218c160d40 2023-10-16T11:19:56.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.634+0000 7f21827fc700 1 -- 172.21.15.158:0/2230621552 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184004280 con 0x7f218c160d40 2023-10-16T11:19:56.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.634+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218c0b3450 con 0x7f218c160d40 2023-10-16T11:19:56.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.634+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218c066db0 con 0x7f218c160d40 2023-10-16T11:19:56.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.635+0000 7f21827fc700 1 -- 172.21.15.158:0/2230621552 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 142) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f21840043e0 con 0x7f218c160d40 2023-10-16T11:19:56.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.635+0000 7f21827fc700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f217006b190 0x7f217006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:56.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.635+0000 7f21827fc700 1 -- 172.21.15.158:0/2230621552 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f2184022070 con 0x7f218c160d40 2023-10-16T11:19:56.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.635+0000 7f2191cdf700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f217006b190 0x7f217006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:56.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.636+0000 7f2191cdf700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f217006b190 0x7f217006d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f217c006fd0 tx=0x7f217c008040 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:56.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.640+0000 7f21827fc700 1 -- 172.21.15.158:0/2230621552 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21840566f0 con 0x7f218c160d40 2023-10-16T11:19:56.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.859+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 --> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f218c0ce9b0 con 0x7f217006b190 2023-10-16T11:19:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.923+0000 7f21827fc700 1 -- 172.21.15.158:0/2230621552 <== mgr.39029 v2:172.21.15.158:6832/1511764298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f217c00a700 con 0x7f217006b190 2023-10-16T11:19:56.925 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T11:19:56.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f217006b190 msgr2=0x7f217006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f217006b190 0x7f217006d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f217c006fd0 tx=0x7f217c008040 comp rx=0 tx=0).stop 2023-10-16T11:19:56.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c160d40 msgr2=0x7f218c157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:56.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c160d40 0x7f218c157510 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f218400b370 tx=0x7f2184004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 shutdown_connections 2023-10-16T11:19:56.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f217006b190 0x7f217006d650 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c160d40 0x7f218c157510 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f218c0c50c0 0x7f218c1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 --2- 172.21.15.158:0/2230621552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f218c0cf420 0x7f218c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 >> 172.21.15.158:0/2230621552 conn(0x7f218c0cc5b0 msgr2=0x7f218c0ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 shutdown_connections 2023-10-16T11:19:56.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:56.925+0000 7f2193f43700 1 -- 172.21.15.158:0/2230621552 wait complete. 2023-10-16T11:19:56.939 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T11:19:56.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:19:56.939 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T11:19:56.958 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:19:57.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.119+0000 7f9753349700 1 Processor -- start 2023-10-16T11:19:57.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.120+0000 7f9753349700 1 -- start start 2023-10-16T11:19:57.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.120+0000 7f9753349700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f974c0d0c60 0x7f974c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:57.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.120+0000 7f9753349700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:57.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.120+0000 7f9753349700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c0b4420 con 0x7f974c15b6b0 2023-10-16T11:19:57.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.120+0000 7f9753349700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c15ba90 con 0x7f974c0d0c60 2023-10-16T11:19:57.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.120+0000 7f9753349700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0b38f0 con 0x7f974c0b3460 2023-10-16T11:19:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f974c0d0c60 0x7f974c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f974c0d0c60 0x7f974c0b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39100/0 (socket says 172.21.15.158:39100) 2023-10-16T11:19:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 -- 172.21.15.158:0/1699747766 learned_addr learned my addr 172.21.15.158:0/1699747766 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97518e6700 1 --2- 172.21.15.158:0/1699747766 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 -- 172.21.15.158:0/1699747766 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 msgr2=0x7f974c0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 --2- 172.21.15.158:0/1699747766 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c0b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 -- 172.21.15.158:0/1699747766 >> v1:172.21.15.172:6789/0 conn(0x7f974c15b6b0 legacy=0x7f974c0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 -- 172.21.15.158:0/1699747766 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0cbc20 con 0x7f974c0d0c60 2023-10-16T11:19:57.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f97508e4700 1 --2- 172.21.15.158:0/1699747766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c0d0c60 0x7f974c0b2f20 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f97480098f0 tx=0x7f9748009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2c8dadef63bf2c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:57.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f9743fff700 1 -- 172.21.15.158:0/1699747766 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974800b8a0 con 0x7f974c0d0c60 2023-10-16T11:19:57.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f9743fff700 1 -- 172.21.15.158:0/1699747766 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974800ba00 con 0x7f974c0d0c60 2023-10-16T11:19:57.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.121+0000 7f9743fff700 1 -- 172.21.15.158:0/1699747766 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974800bd90 con 0x7f974c0d0c60 2023-10-16T11:19:57.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 -- 172.21.15.158:0/1699747766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c0d0c60 msgr2=0x7f974c0b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:57.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 --2- 172.21.15.158:0/1699747766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c0d0c60 0x7f974c0b2f20 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f97480098f0 tx=0x7f9748009c00 comp rx=0 tx=0).stop 2023-10-16T11:19:57.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 -- 172.21.15.158:0/1699747766 shutdown_connections 2023-10-16T11:19:57.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 --2- 172.21.15.158:0/1699747766 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c0b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 --2- 172.21.15.158:0/1699747766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c0d0c60 0x7f974c0b2f20 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 -- 172.21.15.158:0/1699747766 >> 172.21.15.158:0/1699747766 conn(0x7f974c0cdf20 msgr2=0x7f974c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:57.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 -- 172.21.15.158:0/1699747766 shutdown_connections 2023-10-16T11:19:57.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.122+0000 7f9753349700 1 -- 172.21.15.158:0/1699747766 wait complete. 2023-10-16T11:19:57.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.123+0000 7f9753349700 1 Processor -- start 2023-10-16T11:19:57.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.123+0000 7f9753349700 1 -- start start 2023-10-16T11:19:57.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9753349700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c1528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:57.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9753349700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f974c0d0c60 0x7f974c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:57.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9753349700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c15b6b0 0x7f974c159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9753349700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0a84f0 con 0x7f974c15b6b0 2023-10-16T11:19:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9753349700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c153ac0 con 0x7f974c0d0c60 2023-10-16T11:19:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9753349700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0cb520 con 0x7f974c0b3460 2023-10-16T11:19:57.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c15b6b0 0x7f974c159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c15b6b0 0x7f974c159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39108/0 (socket says 172.21.15.158:39108) 2023-10-16T11:19:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 -- 172.21.15.158:0/1766659040 learned_addr learned my addr 172.21.15.158:0/1766659040 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97510e5700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c1528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97508e4700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f974c0d0c60 0x7f974c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:57.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 -- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 msgr2=0x7f974c1528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:57.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c1528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 -- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f974c0d0c60 msgr2=0x7f974c159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:57.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f974c0d0c60 0x7f974c159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 -- 172.21.15.158:0/1766659040 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9748009550 con 0x7f974c15b6b0 2023-10-16T11:19:57.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f97518e6700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c15b6b0 0x7f974c159cd0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f9744007e60 tx=0x7f974400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:57.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9741ffb700 1 -- 172.21.15.158:0/1766659040 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974400ed30 con 0x7f974c15b6b0 2023-10-16T11:19:57.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9741ffb700 1 -- 172.21.15.158:0/1766659040 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9744014420 con 0x7f974c15b6b0 2023-10-16T11:19:57.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c153190 con 0x7f974c15b6b0 2023-10-16T11:19:57.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.124+0000 7f9741ffb700 1 -- 172.21.15.158:0/1766659040 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97440147d0 con 0x7f974c15b6b0 2023-10-16T11:19:57.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.125+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c153750 con 0x7f974c15b6b0 2023-10-16T11:19:57.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.126+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c153c70 con 0x7f974c15b6b0 2023-10-16T11:19:57.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.126+0000 7f9741ffb700 1 -- 172.21.15.158:0/1766659040 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 142) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f9744014930 con 0x7f974c15b6b0 2023-10-16T11:19:57.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.126+0000 7f9741ffb700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f973006b190 0x7f973006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:57.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.127+0000 7f97510e5700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f973006b190 0x7f973006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:57.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.127+0000 7f9741ffb700 1 -- 172.21.15.158:0/1766659040 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f974408a6e0 con 0x7f974c15b6b0 2023-10-16T11:19:57.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.127+0000 7f97510e5700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f973006b190 0x7f973006d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f973c007fd0 tx=0x7f973c009040 comp rx=0 tx=0).ready entity=mgr.39029 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:57.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.131+0000 7f9741ffb700 1 -- 172.21.15.158:0/1766659040 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9744054d20 con 0x7f974c15b6b0 2023-10-16T11:19:57.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.384+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f974c0681b0 con 0x7f974c15b6b0 2023-10-16T11:19:57.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.951+0000 7f9741ffb700 1 -- 172.21.15.158:0/1766659040 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v143) v1 ==== 93+0+0 (secure 0 0 0) 0x7f9744058340 con 0x7f974c15b6b0 2023-10-16T11:19:57.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.954+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f973006b190 msgr2=0x7f973006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:57.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.954+0000 7f9753349700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f973006b190 0x7f973006d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f973c007fd0 tx=0x7f973c009040 comp rx=0 tx=0).stop 2023-10-16T11:19:57.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.954+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c15b6b0 msgr2=0x7f974c159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:57.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.954+0000 7f9753349700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c15b6b0 0x7f974c159cd0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f9744007e60 tx=0x7f974400e3c0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.955+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 shutdown_connections 2023-10-16T11:19:57.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.955+0000 7f9753349700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f973006b190 0x7f973006d650 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.955+0000 7f9753349700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f974c15b6b0 0x7f974c159cd0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.955+0000 7f9753349700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f974c0b3460 0x7f974c1528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.955+0000 7f9753349700 1 --2- 172.21.15.158:0/1766659040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f974c0d0c60 0x7f974c159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:57.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.955+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 >> 172.21.15.158:0/1766659040 conn(0x7f974c0cdf20 msgr2=0x7f974c0cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:57.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.956+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 shutdown_connections 2023-10-16T11:19:57.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:57.957+0000 7f9753349700 1 -- 172.21.15.158:0/1766659040 wait complete. 2023-10-16T11:19:57.976 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:19:57.977 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:19:58.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:19:58.068 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:19:58.069 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:58.069 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:58.087 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:19:58.088 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:19:58.103 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.101+0000 7f89884a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:19:58.109 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:58.108+0000 7fa523dc8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:19:58.127 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:19:58.169 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.168+0000 7f89884a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:58.177 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:58.176+0000 7fa523dc8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:19:58.243 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.241+0000 7f89884a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:19:58.253 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:58.251+0000 7fa523dc8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:19:58.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.290+0000 7fc95ffff700 1 Processor -- start 2023-10-16T11:19:58.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.290+0000 7fc95ffff700 1 -- start start 2023-10-16T11:19:58.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9600715c0 0x7fc96015de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95ffff700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9600ca820 con 0x7fc96015e3d0 2023-10-16T11:19:58.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9600cab00 con 0x7fc9600719a0 2023-10-16T11:19:58.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95ffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96006b8e0 con 0x7fc9600715c0 2023-10-16T11:19:58.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95e7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc96015e3d0 0x7fc9600cd5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39348/0 (socket says 172.21.15.158:39348) 2023-10-16T11:19:58.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95e7fc700 1 -- 172.21.15.158:0/803996329 learned_addr learned my addr 172.21.15.158:0/803996329 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:58.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95effd700 1 --2- 172.21.15.158:0/803996329 >> v2:172.21.15.172:3300/0 conn(0x7fc9600715c0 0x7fc96015de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95dffb700 1 -- 172.21.15.158:0/803996329 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258537872 0 0) 0x7fc9600ca820 con 0x7fc96015e3d0 2023-10-16T11:19:58.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.291+0000 7fc95dffb700 1 -- 172.21.15.158:0/803996329 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9480035b0 con 0x7fc96015e3d0 2023-10-16T11:19:58.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95effd700 1 -- 172.21.15.158:0/803996329 >> v1:172.21.15.172:6789/0 conn(0x7fc9600719a0 legacy=0x7fc9600cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95dffb700 1 -- 172.21.15.158:0/803996329 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956918797 0 0) 0x7fc9600cab00 con 0x7fc9600719a0 2023-10-16T11:19:58.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95effd700 1 -- 172.21.15.158:0/803996329 >> v1:172.21.15.158:6789/0 conn(0x7fc96015e3d0 legacy=0x7fc9600cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95effd700 1 -- 172.21.15.158:0/803996329 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600cade0 con 0x7fc9600715c0 2023-10-16T11:19:58.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95effd700 1 --2- 172.21.15.158:0/803996329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600715c0 0x7fc96015de90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc95000a9e0 tx=0x7fc95000acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d624353f6dba4f00 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:58.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95dffb700 1 -- 172.21.15.158:0/803996329 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000cb70 con 0x7fc9600715c0 2023-10-16T11:19:58.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95dffb700 1 -- 172.21.15.158:0/803996329 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95000ccd0 con 0x7fc9600715c0 2023-10-16T11:19:58.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95dffb700 1 -- 172.21.15.158:0/803996329 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000ae80 con 0x7fc9600715c0 2023-10-16T11:19:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95ffff700 1 -- 172.21.15.158:0/803996329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600715c0 msgr2=0x7fc96015de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.292+0000 7fc95ffff700 1 --2- 172.21.15.158:0/803996329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600715c0 0x7fc96015de90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc95000a9e0 tx=0x7fc95000acf0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.293+0000 7fc95ffff700 1 -- 172.21.15.158:0/803996329 shutdown_connections 2023-10-16T11:19:58.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.293+0000 7fc95ffff700 1 --2- 172.21.15.158:0/803996329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600715c0 0x7fc96015de90 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.293+0000 7fc95ffff700 1 -- 172.21.15.158:0/803996329 >> 172.21.15.158:0/803996329 conn(0x7fc9600cfe00 msgr2=0x7fc9600d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.293+0000 7fc95ffff700 1 -- 172.21.15.158:0/803996329 shutdown_connections 2023-10-16T11:19:58.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.293+0000 7fc95ffff700 1 -- 172.21.15.158:0/803996329 wait complete. 2023-10-16T11:19:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.294+0000 7fc95ffff700 1 Processor -- start 2023-10-16T11:19:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.294+0000 7fc95ffff700 1 -- start start 2023-10-16T11:19:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.295+0000 7fc95ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9600715c0 0x7fc9601f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.295+0000 7fc95ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600719a0 0x7fc9600c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.295+0000 7fc95ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96015e3d0 0x7fc9600c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.295+0000 7fc95ffff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601f3090 con 0x7fc9600715c0 2023-10-16T11:19:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.295+0000 7fc95ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601f3370 con 0x7fc9600719a0 2023-10-16T11:19:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.295+0000 7fc95ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601f3650 con 0x7fc96015e3d0 2023-10-16T11:19:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9600715c0 0x7fc9601f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9600715c0 0x7fc9601f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39176/0 (socket says 172.21.15.158:39176) 2023-10-16T11:19:58.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 -- 172.21.15.158:0/4141275508 learned_addr learned my addr 172.21.15.158:0/4141275508 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:58.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95e7fc700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600719a0 0x7fc9600c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95f7fe700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96015e3d0 0x7fc9600c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 -- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96015e3d0 msgr2=0x7fc9600c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96015e3d0 0x7fc9600c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 -- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600719a0 msgr2=0x7fc9600c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600719a0 0x7fc9600c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 -- 172.21.15.158:0/4141275508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95000a640 con 0x7fc9600715c0 2023-10-16T11:19:58.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95effd700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9600715c0 0x7fc9601f2a70 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fc95000a1c0 tx=0x7fc9500047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:58.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc947fff700 1 -- 172.21.15.158:0/4141275508 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000ae80 con 0x7fc9600715c0 2023-10-16T11:19:58.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc947fff700 1 -- 172.21.15.158:0/4141275508 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95000c970 con 0x7fc9600715c0 2023-10-16T11:19:58.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9601f3930 con 0x7fc9600715c0 2023-10-16T11:19:58.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc947fff700 1 -- 172.21.15.158:0/4141275508 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950023660 con 0x7fc9600715c0 2023-10-16T11:19:58.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.296+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9600c2630 con 0x7fc9600715c0 2023-10-16T11:19:58.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.297+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc960073f90 con 0x7fc9600715c0 2023-10-16T11:19:58.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.298+0000 7fc947fff700 1 -- 172.21.15.158:0/4141275508 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fc950021070 con 0x7fc9600715c0 2023-10-16T11:19:58.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.298+0000 7fc947fff700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 0x7fc94c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.298+0000 7fc95e7fc700 1 -- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 msgr2=0x7fc94c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:58.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.298+0000 7fc947fff700 1 -- 172.21.15.158:0/4141275508 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fc95008b380 con 0x7fc9600715c0 2023-10-16T11:19:58.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.298+0000 7fc95e7fc700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 0x7fc94c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:58.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.302+0000 7fc947fff700 1 -- 172.21.15.158:0/4141275508 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc950055880 con 0x7fc9600715c0 2023-10-16T11:19:58.322 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.320+0000 7f89884a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:19:58.337 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:58.335+0000 7fa523dc8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:19:58.464 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.463+0000 7f89884a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:19:58.480 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:58.478+0000 7fa523dc8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:19:58.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.499+0000 7fc95e7fc700 1 -- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 msgr2=0x7fc94c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:58.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.499+0000 7fc95e7fc700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 0x7fc94c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:58.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.499+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc96004ec20 con 0x7fc9600715c0 2023-10-16T11:19:58.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.501+0000 7fc947fff700 1 -- 172.21.15.158:0/4141275508 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+21694 (secure 0 0 0) 0x7fc950058ea0 con 0x7fc9600715c0 2023-10-16T11:19:58.503 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:19:58.503 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":222,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:52.926692+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:19:58.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 msgr2=0x7fc94c06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:58.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 0x7fc94c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9600715c0 msgr2=0x7fc9601f2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9600715c0 0x7fc9601f2a70 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fc95000a1c0 tx=0x7fc9500047c0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 shutdown_connections 2023-10-16T11:19:58.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fc94c06b1d0 0x7fc94c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc96015e3d0 0x7fc9600c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9600715c0 0x7fc9601f2a70 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 --2- 172.21.15.158:0/4141275508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9600719a0 0x7fc9600c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 >> 172.21.15.158:0/4141275508 conn(0x7fc9600cfe00 msgr2=0x7fc96015d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:58.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.503+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 shutdown_connections 2023-10-16T11:19:58.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.504+0000 7fc95ffff700 1 -- 172.21.15.158:0/4141275508 wait complete. 2023-10-16T11:19:58.518 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:19:21.151561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '216', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:19:21.679054+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T11:19:33.715104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '220', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T11:19:35.032906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:19:58.519 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:19:58.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.681+0000 7f7aa32cf700 1 Processor -- start 2023-10-16T11:19:58.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.681+0000 7f7aa32cf700 1 -- start start 2023-10-16T11:19:58.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.682+0000 7f7aa32cf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7a9c15edf0 0x7f7a9c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.682+0000 7f7aa32cf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c072760 0x7f7a9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.682+0000 7f7aa32cf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a9c06df00 0x7f7a9c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.682+0000 7f7aa32cf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a9c15f1d0 con 0x7f7a9c06df00 2023-10-16T11:19:58.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.682+0000 7f7aa32cf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a9c06e960 con 0x7f7a9c15edf0 2023-10-16T11:19:58.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.682+0000 7f7aa32cf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c06e3a0 con 0x7f7a9c072760 2023-10-16T11:19:58.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.682+0000 7f7aa186c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a9c06df00 0x7f7a9c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a9c06df00 0x7f7a9c0ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39232/0 (socket says 172.21.15.158:39232) 2023-10-16T11:19:58.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 -- 172.21.15.158:0/2158404423 learned_addr learned my addr 172.21.15.158:0/2158404423 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:58.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa086a700 1 --2- 172.21.15.158:0/2158404423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c072760 0x7f7a9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa106b700 1 --2- 172.21.15.158:0/2158404423 >> v2:172.21.15.172:3300/0 conn(0x7f7a9c15edf0 0x7f7a9c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 -- 172.21.15.158:0/2158404423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c072760 msgr2=0x7f7a9c06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:58.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 --2- 172.21.15.158:0/2158404423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c072760 0x7f7a9c06d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 -- 172.21.15.158:0/2158404423 >> v2:172.21.15.172:3300/0 conn(0x7f7a9c15edf0 msgr2=0x7f7a9c072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:58.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 --2- 172.21.15.158:0/2158404423 >> v2:172.21.15.172:3300/0 conn(0x7f7a9c15edf0 0x7f7a9c072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 -- 172.21.15.158:0/2158404423 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c06e680 con 0x7f7a9c06df00 2023-10-16T11:19:58.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa086a700 1 --2- 172.21.15.158:0/2158404423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c072760 0x7f7a9c06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:19:58.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7aa186c700 1 --2- 172.21.15.158:0/2158404423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c06df00 0x7f7a9c0ce2b0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c074530 tx=0x7f7a9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be80d8cb0042492 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:58.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7a93fff700 1 -- 172.21.15.158:0/2158404423 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9400e070 con 0x7f7a9c06df00 2023-10-16T11:19:58.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7a93fff700 1 -- 172.21.15.158:0/2158404423 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9400b910 con 0x7f7a9c06df00 2023-10-16T11:19:58.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.683+0000 7f7a93fff700 1 -- 172.21.15.158:0/2158404423 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9400bcc0 con 0x7f7a9c06df00 2023-10-16T11:19:58.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 -- 172.21.15.158:0/2158404423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c06df00 msgr2=0x7f7a9c0ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/2158404423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c06df00 0x7f7a9c0ce2b0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c074530 tx=0x7f7a9400b330 comp rx=0 tx=0).stop 2023-10-16T11:19:58.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 -- 172.21.15.158:0/2158404423 shutdown_connections 2023-10-16T11:19:58.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/2158404423 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c06df00 0x7f7a9c0ce2b0 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/2158404423 >> v2:172.21.15.172:3300/0 conn(0x7f7a9c15edf0 0x7f7a9c072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/2158404423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c072760 0x7f7a9c06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 -- 172.21.15.158:0/2158404423 >> 172.21.15.158:0/2158404423 conn(0x7f7a9c0d03b0 msgr2=0x7f7a9c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:58.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 -- 172.21.15.158:0/2158404423 shutdown_connections 2023-10-16T11:19:58.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.684+0000 7f7aa32cf700 1 -- 172.21.15.158:0/2158404423 wait complete. 2023-10-16T11:19:58.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.685+0000 7f7aa32cf700 1 Processor -- start 2023-10-16T11:19:58.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.685+0000 7f7aa32cf700 1 -- start start 2023-10-16T11:19:58.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa32cf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c06df00 0x7f7a9c0c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa32cf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15edf0 0x7f7a9c1f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa32cf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c1f3100 0x7f7a9c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa32cf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1f3bb0 con 0x7f7a9c1f3100 2023-10-16T11:19:58.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa32cf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1f35f0 con 0x7f7a9c06df00 2023-10-16T11:19:58.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa32cf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1f38d0 con 0x7f7a9c15edf0 2023-10-16T11:19:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c1f3100 0x7f7a9c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c1f3100 0x7f7a9c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39238/0 (socket says 172.21.15.158:39238) 2023-10-16T11:19:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 -- 172.21.15.158:0/1910455745 learned_addr learned my addr 172.21.15.158:0/1910455745 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa086a700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15edf0 0x7f7a9c1f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa106b700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c06df00 0x7f7a9c0c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:58.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 -- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15edf0 msgr2=0x7f7a9c1f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15edf0 0x7f7a9c1f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 -- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c06df00 msgr2=0x7f7a9c0c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c06df00 0x7f7a9c0c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.686+0000 7f7aa186c700 1 -- 172.21.15.158:0/1910455745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94009b30 con 0x7f7a9c1f3100 2023-10-16T11:19:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.687+0000 7f7aa186c700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c1f3100 0x7f7a9c0ac890 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f7a94012000 tx=0x7f7a94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.687+0000 7f7a91ffb700 1 -- 172.21.15.158:0/1910455745 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a940099b0 con 0x7f7a9c1f3100 2023-10-16T11:19:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.687+0000 7f7a91ffb700 1 -- 172.21.15.158:0/1910455745 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a940092e0 con 0x7f7a9c1f3100 2023-10-16T11:19:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.687+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c0cab30 con 0x7f7a9c1f3100 2023-10-16T11:19:58.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.687+0000 7f7a91ffb700 1 -- 172.21.15.158:0/1910455745 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94004470 con 0x7f7a9c1f3100 2023-10-16T11:19:58.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.687+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c15bce0 con 0x7f7a9c1f3100 2023-10-16T11:19:58.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.688+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9c0cb050 con 0x7f7a9c1f3100 2023-10-16T11:19:58.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.688+0000 7f7a91ffb700 1 -- 172.21.15.158:0/1910455745 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f7a940045d0 con 0x7f7a9c1f3100 2023-10-16T11:19:58.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.689+0000 7f7a91ffb700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 0x7f7a8006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:58.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.689+0000 7f7aa106b700 1 -- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 msgr2=0x7f7a8006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:58.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.689+0000 7f7a91ffb700 1 -- 172.21.15.158:0/1910455745 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f7a9408b380 con 0x7f7a9c1f3100 2023-10-16T11:19:58.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.689+0000 7f7aa106b700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 0x7f7a8006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:58.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.693+0000 7f7a91ffb700 1 -- 172.21.15.158:0/1910455745 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a94055930 con 0x7f7a9c1f3100 2023-10-16T11:19:58.789 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.788+0000 7f89884a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:19:58.828 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:58.826+0000 7fa523dc8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:19:58.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.890+0000 7f7aa106b700 1 -- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 msgr2=0x7f7a8006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:58.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.890+0000 7f7aa106b700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 0x7f7a8006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:58.904 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.903+0000 7f89884a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:19:58.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.905+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a9c0681b0 con 0x7f7a9c1f3100 2023-10-16T11:19:58.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.906+0000 7f7a91ffb700 1 -- 172.21.15.158:0/1910455745 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v222) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a9401f020 con 0x7f7a9c1f3100 2023-10-16T11:19:58.908 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:19:58.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 msgr2=0x7f7a8006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:58.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 0x7f7a8006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c1f3100 msgr2=0x7f7a9c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:58.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c1f3100 0x7f7a9c0ac890 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f7a94012000 tx=0x7f7a94004ad0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 shutdown_connections 2023-10-16T11:19:58.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f7a8006b1d0 0x7f7a8006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a9c1f3100 0x7f7a9c0ac890 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a9c06df00 0x7f7a9c0c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 --2- 172.21.15.158:0/1910455745 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a9c15edf0 0x7f7a9c1f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:58.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 >> 172.21.15.158:0/1910455745 conn(0x7f7a9c0d03b0 msgr2=0x7f7a9c0c9070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:58.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 shutdown_connections 2023-10-16T11:19:58.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:58.908+0000 7f7aa32cf700 1 -- 172.21.15.158:0/1910455745 wait complete. 2023-10-16T11:19:58.924 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:19:58.949 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:58.948+0000 7fa523dc8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:19:58.978 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:58.976+0000 7f89884a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:19:59.027 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:59.025+0000 7fa523dc8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:19:59.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.050+0000 7f5103ea0700 1 Processor -- start 2023-10-16T11:19:59.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.050+0000 7f5103ea0700 1 -- start start 2023-10-16T11:19:59.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5103ea0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc15b6b0 0x7f50fc0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5103ea0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f50fc0d0c60 0x7f50fc0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5103ea0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc0b4420 con 0x7f50fc0b3460 2023-10-16T11:19:59.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5103ea0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50fc0b3910 con 0x7f50fc0d0c60 2023-10-16T11:19:59.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5103ea0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc15ba90 con 0x7f50fc15b6b0 2023-10-16T11:19:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f510243d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f50fc0b3460 0x7f50fc070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39424/0 (socket says 172.21.15.158:39424) 2023-10-16T11:19:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f510243d700 1 -- 172.21.15.158:0/1586947790 learned_addr learned my addr 172.21.15.158:0/1586947790 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5101c3c700 1 --2- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc15b6b0 0x7f50fc0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f510143b700 1 --2- 172.21.15.158:0/1586947790 >> v2:172.21.15.172:3300/0 conn(0x7f50fc0d0c60 0x7f50fc0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5100c3a700 1 -- 172.21.15.158:0/1586947790 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686515846 0 0) 0x7f50fc0b4420 con 0x7f50fc0b3460 2023-10-16T11:19:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f5100c3a700 1 -- 172.21.15.158:0/1586947790 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e80035b0 con 0x7f50fc0b3460 2023-10-16T11:19:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f510143b700 1 -- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc15b6b0 msgr2=0x7f50fc0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f510143b700 1 --2- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc15b6b0 0x7f50fc0cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f510143b700 1 -- 172.21.15.158:0/1586947790 >> v1:172.21.15.158:6789/0 conn(0x7f50fc0b3460 legacy=0x7f50fc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.051+0000 7f510143b700 1 -- 172.21.15.158:0/1586947790 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc0cbc20 con 0x7f50fc0d0c60 2023-10-16T11:19:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5101c3c700 1 --2- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc15b6b0 0x7f50fc0cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f510143b700 1 --2- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0d0c60 0x7f50fc0b2f20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009840 tx=0x7f50ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70dcda4da38e7eea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5100c3a700 1 -- 172.21.15.158:0/1586947790 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec011830 con 0x7f50fc0d0c60 2023-10-16T11:19:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5100c3a700 1 -- 172.21.15.158:0/1586947790 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ec011990 con 0x7f50fc0d0c60 2023-10-16T11:19:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5100c3a700 1 -- 172.21.15.158:0/1586947790 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec011d20 con 0x7f50fc0d0c60 2023-10-16T11:19:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5103ea0700 1 -- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0d0c60 msgr2=0x7f50fc0b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0d0c60 0x7f50fc0b2f20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009840 tx=0x7f50ec009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5103ea0700 1 -- 172.21.15.158:0/1586947790 shutdown_connections 2023-10-16T11:19:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc15b6b0 0x7f50fc0cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1586947790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0d0c60 0x7f50fc0b2f20 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.052+0000 7f5103ea0700 1 -- 172.21.15.158:0/1586947790 >> 172.21.15.158:0/1586947790 conn(0x7f50fc0cdf20 msgr2=0x7f50fc0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:59.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.053+0000 7f5103ea0700 1 -- 172.21.15.158:0/1586947790 shutdown_connections 2023-10-16T11:19:59.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.053+0000 7f5103ea0700 1 -- 172.21.15.158:0/1586947790 wait complete. 2023-10-16T11:19:59.061 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:59.058+0000 7f89884a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:19:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.054+0000 7f5103ea0700 1 Processor -- start 2023-10-16T11:19:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.054+0000 7f5103ea0700 1 -- start start 2023-10-16T11:19:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.054+0000 7f5103ea0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0b3460 0x7f50fc0caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.054+0000 7f5103ea0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc0d0c60 0x7f50fc0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.054+0000 7f5103ea0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50fc15b6b0 0x7f50fc0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f5103ea0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc1f6410 con 0x7f50fc15b6b0 2023-10-16T11:19:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f5103ea0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc0c91f0 con 0x7f50fc0b3460 2023-10-16T11:19:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f5103ea0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc0c94d0 con 0x7f50fc0d0c60 2023-10-16T11:19:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50fc15b6b0 0x7f50fc0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50fc15b6b0 0x7f50fc0c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39254/0 (socket says 172.21.15.158:39254) 2023-10-16T11:19:59.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 -- 172.21.15.158:0/1794364537 learned_addr learned my addr 172.21.15.158:0/1794364537 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:59.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510143b700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc0d0c60 0x7f50fc0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f5101c3c700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0b3460 0x7f50fc0caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 -- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc0d0c60 msgr2=0x7f50fc0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc0d0c60 0x7f50fc0cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 -- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0b3460 msgr2=0x7f50fc0caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0b3460 0x7f50fc0caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 -- 172.21.15.158:0/1794364537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f80094a0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f510243d700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50fc15b6b0 0x7f50fc0c8570 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f50f400caa0 tx=0x7f50f400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f50f27fc700 1 -- 172.21.15.158:0/1794364537 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f400a9e0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f50f27fc700 1 -- 172.21.15.158:0/1794364537 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f400ab40 con 0x7f50fc15b6b0 2023-10-16T11:19:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ec0094a0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f50f27fc700 1 -- 172.21.15.158:0/1794364537 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4012400 con 0x7f50fc15b6b0 2023-10-16T11:19:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.055+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50fc0c9a90 con 0x7f50fc15b6b0 2023-10-16T11:19:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.056+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50fc0681b0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.057+0000 7f50f27fc700 1 -- 172.21.15.158:0/1794364537 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f50f401b3b0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.057+0000 7f50f27fc700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 0x7f50e006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.057+0000 7f5101c3c700 1 -- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 msgr2=0x7f50e006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.057+0000 7f50f27fc700 1 -- 172.21.15.158:0/1794364537 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f50f4089dd0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.057+0000 7f5101c3c700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 0x7f50e006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:59.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.062+0000 7f50f27fc700 1 -- 172.21.15.158:0/1794364537 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50f40544a0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.114 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:59.112+0000 7fa523dc8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:19:59.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.258+0000 7f5101c3c700 1 -- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 msgr2=0x7f50e006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:59.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.258+0000 7f5101c3c700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 0x7f50e006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:59.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.264+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50fc0ce510 con 0x7f50fc15b6b0 2023-10-16T11:19:59.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.264+0000 7f50f27fc700 1 -- 172.21.15.158:0/1794364537 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v222) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50f4057ac0 con 0x7f50fc15b6b0 2023-10-16T11:19:59.266 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:19:59.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 msgr2=0x7f50e006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:59.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 0x7f50e006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50fc15b6b0 msgr2=0x7f50fc0c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50fc15b6b0 0x7f50fc0c8570 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f50f400caa0 tx=0x7f50f400cdb0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 shutdown_connections 2023-10-16T11:19:59.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f50e006b1d0 0x7f50e006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50fc15b6b0 0x7f50fc0c8570 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50fc0b3460 0x7f50fc0caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 --2- 172.21.15.158:0/1794364537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50fc0d0c60 0x7f50fc0cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 >> 172.21.15.158:0/1794364537 conn(0x7f50fc0cdf20 msgr2=0x7f50fc0d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:59.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 shutdown_connections 2023-10-16T11:19:59.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.266+0000 7f5103ea0700 1 -- 172.21.15.158:0/1794364537 wait complete. 2023-10-16T11:19:59.280 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:19:59.439 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:19:59.438+0000 7f89884a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:19:59.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.447+0000 7f1573d7e700 1 Processor -- start 2023-10-16T11:19:59.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.447+0000 7f1573d7e700 1 -- start start 2023-10-16T11:19:59.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1573d7e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f156c06ab40 0x7f156c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1573d7e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f156c06e3a0 0x7f156c160a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1573d7e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c0cee50 con 0x7f156c06af20 2023-10-16T11:19:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1573d7e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c06d9c0 con 0x7f156c06ab40 2023-10-16T11:19:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1573d7e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c0ce890 con 0x7f156c06e3a0 2023-10-16T11:19:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f156c06ab40 0x7f156c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f156c06ab40 0x7f156c0ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39288/0 (socket says 172.21.15.158:39288) 2023-10-16T11:19:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 -- 172.21.15.158:0/1949531225 learned_addr learned my addr 172.21.15.158:0/1949531225 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f157231b700 1 --2- 172.21.15.158:0/1949531225 >> v2:172.21.15.172:3300/0 conn(0x7f156c06e3a0 0x7f156c160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 -- 172.21.15.158:0/1949531225 >> v2:172.21.15.172:3300/0 conn(0x7f156c06e3a0 msgr2=0x7f156c160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:19:59.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 --2- 172.21.15.158:0/1949531225 >> v2:172.21.15.172:3300/0 conn(0x7f156c06e3a0 0x7f156c160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 -- 172.21.15.158:0/1949531225 >> v1:172.21.15.172:6789/0 conn(0x7f156c06af20 legacy=0x7f156c1620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 -- 172.21.15.158:0/1949531225 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c0ceb70 con 0x7f156c06ab40 2023-10-16T11:19:59.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.448+0000 7f1571b1a700 1 --2- 172.21.15.158:0/1949531225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06ab40 0x7f156c0ce2b0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f156c0d0070 tx=0x7f1568009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1afd76c584ab0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1570b18700 1 -- 172.21.15.158:0/1949531225 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568011830 con 0x7f156c06ab40 2023-10-16T11:19:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1570b18700 1 -- 172.21.15.158:0/1949531225 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1568011990 con 0x7f156c06ab40 2023-10-16T11:19:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1570b18700 1 -- 172.21.15.158:0/1949531225 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568011d20 con 0x7f156c06ab40 2023-10-16T11:19:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 -- 172.21.15.158:0/1949531225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06ab40 msgr2=0x7f156c0ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1949531225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06ab40 0x7f156c0ce2b0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f156c0d0070 tx=0x7f1568009bd0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 -- 172.21.15.158:0/1949531225 shutdown_connections 2023-10-16T11:19:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1949531225 >> v2:172.21.15.172:3300/0 conn(0x7f156c06e3a0 0x7f156c160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1949531225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06ab40 0x7f156c0ce2b0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 -- 172.21.15.158:0/1949531225 >> 172.21.15.158:0/1949531225 conn(0x7f156c0d03b0 msgr2=0x7f156c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 -- 172.21.15.158:0/1949531225 shutdown_connections 2023-10-16T11:19:59.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.449+0000 7f1573d7e700 1 -- 172.21.15.158:0/1949531225 wait complete. 2023-10-16T11:19:59.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.451+0000 7f1573d7e700 1 Processor -- start 2023-10-16T11:19:59.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.451+0000 7f1573d7e700 1 -- start start 2023-10-16T11:19:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1573d7e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f156c06ab40 0x7f156c1f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1573d7e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156c06af20 0x7f156c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1573d7e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06e3a0 0x7f156c0aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1573d7e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c1f7a60 con 0x7f156c06e3a0 2023-10-16T11:19:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1573d7e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c0b5300 con 0x7f156c06af20 2023-10-16T11:19:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1573d7e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c0b4450 con 0x7f156c06ab40 2023-10-16T11:19:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06e3a0 0x7f156c0aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1571319700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156c06af20 0x7f156c1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1571b1a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f156c06ab40 0x7f156c1f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06e3a0 0x7f156c0aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39296/0 (socket says 172.21.15.158:39296) 2023-10-16T11:19:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 -- 172.21.15.158:0/1850638886 learned_addr learned my addr 172.21.15.158:0/1850638886 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 -- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f156c06ab40 msgr2=0x7f156c1f7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f156c06ab40 0x7f156c1f7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 -- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156c06af20 msgr2=0x7f156c1559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156c06af20 0x7f156c1559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 -- 172.21.15.158:0/1850638886 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568009550 con 0x7f156c06e3a0 2023-10-16T11:19:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1571b1a700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f156c06ab40 0x7f156c1f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f1571319700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156c06af20 0x7f156c1559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.452+0000 7f157231b700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06e3a0 0x7f156c0aed50 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f156400efc0 tx=0x7f156400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.453+0000 7f15627fc700 1 -- 172.21.15.158:0/1850638886 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156400cad0 con 0x7f156c06e3a0 2023-10-16T11:19:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.453+0000 7f15627fc700 1 -- 172.21.15.158:0/1850638886 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156400f040 con 0x7f156c06e3a0 2023-10-16T11:19:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.453+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156c0b4730 con 0x7f156c06e3a0 2023-10-16T11:19:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.453+0000 7f15627fc700 1 -- 172.21.15.158:0/1850638886 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156400ce70 con 0x7f156c06e3a0 2023-10-16T11:19:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.453+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156c157160 con 0x7f156c06e3a0 2023-10-16T11:19:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.454+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156c0681b0 con 0x7f156c06e3a0 2023-10-16T11:19:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.454+0000 7f15627fc700 1 -- 172.21.15.158:0/1850638886 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f1564014480 con 0x7f156c06e3a0 2023-10-16T11:19:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.455+0000 7f15627fc700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 0x7f155006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.455+0000 7f1571b1a700 1 -- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 msgr2=0x7f155006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:59.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.455+0000 7f15627fc700 1 -- 172.21.15.158:0/1850638886 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f156408b3d0 con 0x7f156c06e3a0 2023-10-16T11:19:59.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.455+0000 7f1571b1a700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 0x7f155006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:59.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.459+0000 7f15627fc700 1 -- 172.21.15.158:0/1850638886 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f15640559d0 con 0x7f156c06e3a0 2023-10-16T11:19:59.504 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:19:59.502+0000 7fa523dc8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:19:59.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.656+0000 7f1571b1a700 1 -- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 msgr2=0x7f155006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:59.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.656+0000 7f1571b1a700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 0x7f155006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:19:59.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.659+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f156c0b6330 con 0x7f156c06e3a0 2023-10-16T11:19:59.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.659+0000 7f15627fc700 1 -- 172.21.15.158:0/1850638886 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v222) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1564018070 con 0x7f156c06e3a0 2023-10-16T11:19:59.661 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:19:59.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 msgr2=0x7f155006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:19:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 0x7f155006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06e3a0 msgr2=0x7f156c0aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06e3a0 0x7f156c0aed50 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f156400efc0 tx=0x7f156400c0c0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 shutdown_connections 2023-10-16T11:19:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f155006b1d0 0x7f155006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f156c06e3a0 0x7f156c0aed50 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f156c06ab40 0x7f156c1f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 --2- 172.21.15.158:0/1850638886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156c06af20 0x7f156c1559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 >> 172.21.15.158:0/1850638886 conn(0x7f156c0d03b0 msgr2=0x7f156c15ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 shutdown_connections 2023-10-16T11:19:59.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.662+0000 7f1573d7e700 1 -- 172.21.15.158:0/1850638886 wait complete. 2023-10-16T11:19:59.676 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T11:19:59.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.801+0000 7ff9a04e2700 1 Processor -- start 2023-10-16T11:19:59.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.801+0000 7ff9a04e2700 1 -- start start 2023-10-16T11:19:59.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff9a04e2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff99806afe0 0x7ff9980caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff9a04e2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9980cbd30 con 0x7ff99815da40 2023-10-16T11:19:59.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff9a04e2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9980cc1c0 con 0x7ff9980cc590 2023-10-16T11:19:59.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff9a04e2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99815de20 con 0x7ff99806afe0 2023-10-16T11:19:59.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99da7d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff99806afe0 0x7ff9980caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99e27e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff99815da40 0x7ff99806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39478/0 (socket says 172.21.15.158:39478) 2023-10-16T11:19:59.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99e27e700 1 -- 172.21.15.158:0/2186672985 learned_addr learned my addr 172.21.15.158:0/2186672985 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:19:59.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99da7d700 1 --2- 172.21.15.158:0/2186672985 >> v2:172.21.15.158:3300/0 conn(0x7ff99806afe0 0x7ff9980caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39304/0 (socket says 172.21.15.158:39304) 2023-10-16T11:19:59.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99da7d700 1 -- 172.21.15.158:0/2186672985 >> v1:172.21.15.172:6789/0 conn(0x7ff9980cc590 legacy=0x7ff9980afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99da7d700 1 -- 172.21.15.158:0/2186672985 >> v1:172.21.15.158:6789/0 conn(0x7ff99815da40 legacy=0x7ff99806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99da7d700 1 -- 172.21.15.158:0/2186672985 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9980ce260 con 0x7ff99806afe0 2023-10-16T11:19:59.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.802+0000 7ff99d27c700 1 -- 172.21.15.158:0/2186672985 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035791238 0 0) 0x7ff9980cbd30 con 0x7ff99815da40 2023-10-16T11:19:59.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.803+0000 7ff99da7d700 1 --2- 172.21.15.158:0/2186672985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff99806afe0 0x7ff9980caa20 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7ff988009840 tx=0x7ff988009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b30fe4eae4cafda server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:59.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.803+0000 7ff99d27c700 1 -- 172.21.15.158:0/2186672985 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff988011830 con 0x7ff99806afe0 2023-10-16T11:19:59.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.803+0000 7ff99d27c700 1 -- 172.21.15.158:0/2186672985 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff988011990 con 0x7ff99806afe0 2023-10-16T11:19:59.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.803+0000 7ff99d27c700 1 -- 172.21.15.158:0/2186672985 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff988011d20 con 0x7ff99806afe0 2023-10-16T11:19:59.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.803+0000 7ff9a04e2700 1 -- 172.21.15.158:0/2186672985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff99806afe0 msgr2=0x7ff9980caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.803+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/2186672985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff99806afe0 0x7ff9980caa20 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7ff988009840 tx=0x7ff988009b50 comp rx=0 tx=0).stop 2023-10-16T11:19:59.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.804+0000 7ff9a04e2700 1 -- 172.21.15.158:0/2186672985 shutdown_connections 2023-10-16T11:19:59.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.804+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/2186672985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff99806afe0 0x7ff9980caa20 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.804+0000 7ff9a04e2700 1 -- 172.21.15.158:0/2186672985 >> 172.21.15.158:0/2186672985 conn(0x7ff998099a10 msgr2=0x7ff9980d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:19:59.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.804+0000 7ff9a04e2700 1 -- 172.21.15.158:0/2186672985 shutdown_connections 2023-10-16T11:19:59.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.804+0000 7ff9a04e2700 1 -- 172.21.15.158:0/2186672985 wait complete. 2023-10-16T11:19:59.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.805+0000 7ff9a04e2700 1 Processor -- start 2023-10-16T11:19:59.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.805+0000 7ff9a04e2700 1 -- start start 2023-10-16T11:19:59.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff9a04e2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff99806afe0 0x7ff99815af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff9a04e2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9980cc590 0x7ff9980c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff9a04e2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff99815da40 0x7ff9980c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff9a04e2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99815cf80 con 0x7ff9980cc590 2023-10-16T11:19:59.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff9a04e2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99815b4c0 con 0x7ff99806afe0 2023-10-16T11:19:59.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff9a04e2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9980c4060 con 0x7ff99815da40 2023-10-16T11:19:59.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9980cc590 0x7ff9980c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9980cc590 0x7ff9980c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39310/0 (socket says 172.21.15.158:39310) 2023-10-16T11:19:59.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99e27e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff99806afe0 0x7ff99815af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 -- 172.21.15.158:0/1739562342 learned_addr learned my addr 172.21.15.158:0/1739562342 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:19:59.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99ea7f700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff99815da40 0x7ff9980c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:19:59.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 -- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff99815da40 msgr2=0x7ff9980c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff99815da40 0x7ff9980c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 -- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff99806afe0 msgr2=0x7ff99815af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:19:59.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff99806afe0 0x7ff99815af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:19:59.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 -- 172.21.15.158:0/1739562342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880094a0 con 0x7ff9980cc590 2023-10-16T11:19:59.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99e27e700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff99806afe0 0x7ff99815af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:19:59.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff99da7d700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9980cc590 0x7ff9980c6960 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7ff9880090b0 tx=0x7ff988004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:19:59.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff98effd700 1 -- 172.21.15.158:0/1739562342 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9880045d0 con 0x7ff9980cc590 2023-10-16T11:19:59.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff98effd700 1 -- 172.21.15.158:0/1739562342 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98801f070 con 0x7ff9980cc590 2023-10-16T11:19:59.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99815c6e0 con 0x7ff9980cc590 2023-10-16T11:19:59.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.806+0000 7ff98effd700 1 -- 172.21.15.158:0/1739562342 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9880038a0 con 0x7ff9980cc590 2023-10-16T11:19:59.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.807+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99815cca0 con 0x7ff9980cc590 2023-10-16T11:19:59.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.807+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9980c3e30 con 0x7ff9980cc590 2023-10-16T11:19:59.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.808+0000 7ff98effd700 1 -- 172.21.15.158:0/1739562342 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7ff988003a00 con 0x7ff9980cc590 2023-10-16T11:19:59.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.808+0000 7ff98effd700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 0x7ff97c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:19:59.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.808+0000 7ff99e27e700 1 -- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 msgr2=0x7ff97c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:19:59.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.809+0000 7ff98effd700 1 -- 172.21.15.158:0/1739562342 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7ff988021070 con 0x7ff9980cc590 2023-10-16T11:19:59.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.809+0000 7ff99e27e700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 0x7ff97c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:19:59.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:19:59.812+0000 7ff98effd700 1 -- 172.21.15.158:0/1739562342 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9880561c0 con 0x7ff9980cc590 2023-10-16T11:20:00.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.010+0000 7ff99e27e700 1 -- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 msgr2=0x7ff97c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:00.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.010+0000 7ff99e27e700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 0x7ff97c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:00.014 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:00.013+0000 7f89884a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:20:00.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.045+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9980c3e30 con 0x7ff9980cc590 2023-10-16T11:20:00.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.045+0000 7ff98effd700 1 -- 172.21.15.158:0/1739562342 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v222) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff98802b020 con 0x7ff9980cc590 2023-10-16T11:20:00.047 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:00.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 msgr2=0x7ff97c06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:00.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 0x7ff97c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9980cc590 msgr2=0x7ff9980c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9980cc590 0x7ff9980c6960 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7ff9880090b0 tx=0x7ff988004750 comp rx=0 tx=0).stop 2023-10-16T11:20:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 shutdown_connections 2023-10-16T11:20:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7ff97c06b1d0 0x7ff97c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff99815da40 0x7ff9980c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff99806afe0 0x7ff99815af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 --2- 172.21.15.158:0/1739562342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9980cc590 0x7ff9980c6960 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 >> 172.21.15.158:0/1739562342 conn(0x7ff998099a10 msgr2=0x7ff99806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 shutdown_connections 2023-10-16T11:20:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.047+0000 7ff9a04e2700 1 -- 172.21.15.158:0/1739562342 wait complete. 2023-10-16T11:20:00.063 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T11:20:00.086 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:00.084+0000 7f89884a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:20:00.104 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:00.103+0000 7fa523dc8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:20:00.179 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:00.178+0000 7fa523dc8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:20:00.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.193+0000 7fe94479a700 1 Processor -- start 2023-10-16T11:20:00.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.193+0000 7fe94479a700 1 -- start start 2023-10-16T11:20:00.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe94479a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe93c159af0 0x7fe93c0ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe94479a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0ce3e0 0x7fe93c15de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe94479a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe93c0b0c00 0x7fe93c0b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe94479a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c0b15d0 con 0x7fe93c159af0 2023-10-16T11:20:00.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe94479a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c0cd3d0 con 0x7fe93c0b0c00 2023-10-16T11:20:00.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe94479a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0cdf50 con 0x7fe93c0ce3e0 2023-10-16T11:20:00.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe93c159af0 0x7fe93c0ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe93c159af0 0x7fe93c0ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39318/0 (socket says 172.21.15.158:39318) 2023-10-16T11:20:00.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 -- 172.21.15.158:0/244746087 learned_addr learned my addr 172.21.15.158:0/244746087 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:00.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe941d35700 1 --2- 172.21.15.158:0/244746087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0ce3e0 0x7fe93c15de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942d37700 1 --2- 172.21.15.158:0/244746087 >> v2:172.21.15.172:3300/0 conn(0x7fe93c0b0c00 0x7fe93c0b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 -- 172.21.15.158:0/244746087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0ce3e0 msgr2=0x7fe93c15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:00.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 --2- 172.21.15.158:0/244746087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0ce3e0 0x7fe93c15de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 -- 172.21.15.158:0/244746087 >> v2:172.21.15.172:3300/0 conn(0x7fe93c0b0c00 msgr2=0x7fe93c0b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:00.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 --2- 172.21.15.158:0/244746087 >> v2:172.21.15.172:3300/0 conn(0x7fe93c0b0c00 0x7fe93c0b1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 -- 172.21.15.158:0/244746087 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0cd620 con 0x7fe93c159af0 2023-10-16T11:20:00.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.194+0000 7fe942536700 1 --2- 172.21.15.158:0/244746087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c0ca8b0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fe93c06ec20 tx=0x7fe92c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61b8c18fcaab656 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:00.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe941534700 1 -- 172.21.15.158:0/244746087 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c00c900 con 0x7fe93c159af0 2023-10-16T11:20:00.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe941534700 1 -- 172.21.15.158:0/244746087 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c00ca60 con 0x7fe93c159af0 2023-10-16T11:20:00.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe941534700 1 -- 172.21.15.158:0/244746087 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c00cdf0 con 0x7fe93c159af0 2023-10-16T11:20:00.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 -- 172.21.15.158:0/244746087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 msgr2=0x7fe93c0ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 --2- 172.21.15.158:0/244746087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c0ca8b0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fe93c06ec20 tx=0x7fe92c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 -- 172.21.15.158:0/244746087 shutdown_connections 2023-10-16T11:20:00.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 --2- 172.21.15.158:0/244746087 >> v2:172.21.15.172:3300/0 conn(0x7fe93c0b0c00 0x7fe93c0b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 --2- 172.21.15.158:0/244746087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c0ca8b0 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 --2- 172.21.15.158:0/244746087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0ce3e0 0x7fe93c15de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 -- 172.21.15.158:0/244746087 >> 172.21.15.158:0/244746087 conn(0x7fe93c06af00 msgr2=0x7fe93c0cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:00.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 -- 172.21.15.158:0/244746087 shutdown_connections 2023-10-16T11:20:00.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.195+0000 7fe94479a700 1 -- 172.21.15.158:0/244746087 wait complete. 2023-10-16T11:20:00.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.197+0000 7fe94479a700 1 Processor -- start 2023-10-16T11:20:00.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.197+0000 7fe94479a700 1 -- start start 2023-10-16T11:20:00.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe94479a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0b0c00 0x7fe93c0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe94479a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe93c0ce3e0 0x7fe93c0c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe94479a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe94479a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c152680 con 0x7fe93c159af0 2023-10-16T11:20:00.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe94479a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0c6a20 con 0x7fe93c0ce3e0 2023-10-16T11:20:00.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe94479a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c145cf0 con 0x7fe93c0b0c00 2023-10-16T11:20:00.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c1517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39328/0 (socket says 172.21.15.158:39328) 2023-10-16T11:20:00.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 -- 172.21.15.158:0/1230280387 learned_addr learned my addr 172.21.15.158:0/1230280387 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:00.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942536700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0b0c00 0x7fe93c0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe941d35700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe93c0ce3e0 0x7fe93c0c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 -- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0b0c00 msgr2=0x7fe93c0c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0b0c00 0x7fe93c0c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 -- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe93c0ce3e0 msgr2=0x7fe93c0c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe93c0ce3e0 0x7fe93c0c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 -- 172.21.15.158:0/1230280387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c009550 con 0x7fe93c159af0 2023-10-16T11:20:00.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942536700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0b0c00 0x7fe93c0c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:00.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe942d37700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c1517c0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fe934009dd0 tx=0x7fe93400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:00.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe932ffd700 1 -- 172.21.15.158:0/1230280387 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934009c40 con 0x7fe93c159af0 2023-10-16T11:20:00.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe932ffd700 1 -- 172.21.15.158:0/1230280387 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93400f040 con 0x7fe93c159af0 2023-10-16T11:20:00.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.198+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c0c7380 con 0x7fe93c159af0 2023-10-16T11:20:00.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.199+0000 7fe932ffd700 1 -- 172.21.15.158:0/1230280387 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400ceb0 con 0x7fe93c159af0 2023-10-16T11:20:00.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.199+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c0c6360 con 0x7fe93c159af0 2023-10-16T11:20:00.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.200+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c0c61c0 con 0x7fe93c159af0 2023-10-16T11:20:00.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.200+0000 7fe932ffd700 1 -- 172.21.15.158:0/1230280387 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fe934014480 con 0x7fe93c159af0 2023-10-16T11:20:00.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.200+0000 7fe932ffd700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 0x7fe92006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.200+0000 7fe942536700 1 -- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 msgr2=0x7fe92006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:00.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.200+0000 7fe932ffd700 1 -- 172.21.15.158:0/1230280387 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fe93408ad10 con 0x7fe93c159af0 2023-10-16T11:20:00.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.200+0000 7fe942536700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 0x7fe92006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:00.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.205+0000 7fe932ffd700 1 -- 172.21.15.158:0/1230280387 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe93c0c61c0 con 0x7fe93c159af0 2023-10-16T11:20:00.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.401+0000 7fe942536700 1 -- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 msgr2=0x7fe92006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:00.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.401+0000 7fe942536700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 0x7fe92006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:00.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.406+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fe93c0c6b60 con 0x7fe93c159af0 2023-10-16T11:20:00.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.406+0000 7fe932ffd700 1 -- 172.21.15.158:0/1230280387 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v222) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe934018020 con 0x7fe93c159af0 2023-10-16T11:20:00.408 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:00.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 msgr2=0x7fe92006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:00.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 0x7fe92006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 msgr2=0x7fe93c1517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c1517c0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fe934009dd0 tx=0x7fe93400c3a0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 shutdown_connections 2023-10-16T11:20:00.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe92006b1d0 0x7fe92006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe93c159af0 0x7fe93c1517c0 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe93c0b0c00 0x7fe93c0c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 --2- 172.21.15.158:0/1230280387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe93c0ce3e0 0x7fe93c0c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.408+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 >> 172.21.15.158:0/1230280387 conn(0x7fe93c06af00 msgr2=0x7fe93c157410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:00.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.409+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 shutdown_connections 2023-10-16T11:20:00.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.409+0000 7fe94479a700 1 -- 172.21.15.158:0/1230280387 wait complete. 2023-10-16T11:20:00.423 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:20:00.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.585+0000 7fd882d8c700 1 Processor -- start 2023-10-16T11:20:00.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.586+0000 7fd882d8c700 1 -- start start 2023-10-16T11:20:00.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd882d8c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c15a6d0 0x7fd87c0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd882d8c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd87c0cfe30 0x7fd87c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd882d8c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c15ec70 con 0x7fd87c0cd190 2023-10-16T11:20:00.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd882d8c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87c0afcd0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd882d8c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c0cd620 con 0x7fd87c15a6d0 2023-10-16T11:20:00.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd87c0cfe30 0x7fd87c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd881329700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd87c0cd190 0x7fd87c06ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39514/0 (socket says 172.21.15.158:39514) 2023-10-16T11:20:00.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd881329700 1 -- 172.21.15.158:0/1923872103 learned_addr learned my addr 172.21.15.158:0/1923872103 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:00.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87bfff700 1 --2- 172.21.15.158:0/1923872103 >> v2:172.21.15.158:3300/0 conn(0x7fd87c0cfe30 0x7fd87c0ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39336/0 (socket says 172.21.15.158:39336) 2023-10-16T11:20:00.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd880b28700 1 --2- 172.21.15.158:0/1923872103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c15a6d0 0x7fd87c0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87bfff700 1 -- 172.21.15.158:0/1923872103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c15a6d0 msgr2=0x7fd87c0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:00.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87bfff700 1 --2- 172.21.15.158:0/1923872103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c15a6d0 0x7fd87c0ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87b7fe700 1 -- 172.21.15.158:0/1923872103 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997065162 0 0) 0x7fd87c15ec70 con 0x7fd87c0cd190 2023-10-16T11:20:00.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87bfff700 1 -- 172.21.15.158:0/1923872103 >> v1:172.21.15.158:6789/0 conn(0x7fd87c0cd190 legacy=0x7fd87c06ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87bfff700 1 -- 172.21.15.158:0/1923872103 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c15f1b0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.587+0000 7fd87bfff700 1 --2- 172.21.15.158:0/1923872103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0ccc50 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7fd868009840 tx=0x7fd868009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d68e2308c34ed1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:00.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd87b7fe700 1 -- 172.21.15.158:0/1923872103 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868011830 con 0x7fd87c0cfe30 2023-10-16T11:20:00.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd87b7fe700 1 -- 172.21.15.158:0/1923872103 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd868011990 con 0x7fd87c0cfe30 2023-10-16T11:20:00.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd87b7fe700 1 -- 172.21.15.158:0/1923872103 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868011d20 con 0x7fd87c0cfe30 2023-10-16T11:20:00.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 -- 172.21.15.158:0/1923872103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 msgr2=0x7fd87c0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1923872103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0ccc50 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7fd868009840 tx=0x7fd868009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:00.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 -- 172.21.15.158:0/1923872103 shutdown_connections 2023-10-16T11:20:00.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1923872103 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c15a6d0 0x7fd87c0ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1923872103 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0ccc50 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 -- 172.21.15.158:0/1923872103 >> 172.21.15.158:0/1923872103 conn(0x7fd87c06b9e0 msgr2=0x7fd87c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:00.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 -- 172.21.15.158:0/1923872103 shutdown_connections 2023-10-16T11:20:00.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.588+0000 7fd882d8c700 1 -- 172.21.15.158:0/1923872103 wait complete. 2023-10-16T11:20:00.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.589+0000 7fd882d8c700 1 Processor -- start 2023-10-16T11:20:00.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.590+0000 7fd882d8c700 1 -- start start 2023-10-16T11:20:00.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.590+0000 7fd882d8c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c0cd190 0x7fd87c1f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.590+0000 7fd882d8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.590+0000 7fd882d8c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c15a6d0 0x7fd87c1f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.590+0000 7fd882d8c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c0b2100 con 0x7fd87c0cfe30 2023-10-16T11:20:00.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.590+0000 7fd882d8c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c0aeca0 con 0x7fd87c15a6d0 2023-10-16T11:20:00.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.590+0000 7fd882d8c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c1f4070 con 0x7fd87c0cd190 2023-10-16T11:20:00.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39342/0 (socket says 172.21.15.158:39342) 2023-10-16T11:20:00.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 -- 172.21.15.158:0/1596776271 learned_addr learned my addr 172.21.15.158:0/1596776271 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:00.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd880b28700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c0cd190 0x7fd87c1f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd881329700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c15a6d0 0x7fd87c1f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:00.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 -- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c0cd190 msgr2=0x7fd87c1f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c0cd190 0x7fd87c1f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 -- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c15a6d0 msgr2=0x7fd87c1f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c15a6d0 0x7fd87c1f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 -- 172.21.15.158:0/1596776271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680094a0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd87bfff700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0b2590 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fd868009fd0 tx=0x7fd868004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:00.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd8797fa700 1 -- 172.21.15.158:0/1596776271 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8680036a0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd8797fa700 1 -- 172.21.15.158:0/1596776271 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86801f070 con 0x7fd87c0cfe30 2023-10-16T11:20:00.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c0b14f0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd8797fa700 1 -- 172.21.15.158:0/1596776271 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868003a60 con 0x7fd87c0cfe30 2023-10-16T11:20:00.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.591+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c0b1880 con 0x7fd87c0cfe30 2023-10-16T11:20:00.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.592+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c0681b0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.593+0000 7fd8797fa700 1 -- 172.21.15.158:0/1596776271 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fd868003bc0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.593+0000 7fd8797fa700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 0x7fd86006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:00.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.593+0000 7fd880b28700 1 -- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 msgr2=0x7fd86006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:00.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.593+0000 7fd8797fa700 1 -- 172.21.15.158:0/1596776271 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fd868021070 con 0x7fd87c0cfe30 2023-10-16T11:20:00.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.593+0000 7fd880b28700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 0x7fd86006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:00.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.598+0000 7fd8797fa700 1 -- 172.21.15.158:0/1596776271 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd868055f60 con 0x7fd87c0cfe30 2023-10-16T11:20:00.726 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:00.724+0000 7f89884a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.794+0000 7fd880b28700 1 -- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 msgr2=0x7fd86006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.794+0000 7fd880b28700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 0x7fd86006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:00.803 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:00.802+0000 7f89884a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:20:00.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.818+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd87c0720b0 con 0x7fd87c0cfe30 2023-10-16T11:20:00.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.819+0000 7fd8797fa700 1 -- 172.21.15.158:0/1596776271 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v222) v1 ==== 80+0+476 (secure 0 0 0) 0x7fd868015020 con 0x7fd87c0cfe30 2023-10-16T11:20:00.821 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2949922272 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.821 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/1944509827 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.821 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/1944509827 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.821 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3674079859 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.822 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1611448416 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.822 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/305366342 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.822 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1831692059 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.822 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/520781827 2023-10-17T11:19:52.926663+0000 2023-10-16T11:20:00.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 msgr2=0x7fd86006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:00.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 0x7fd86006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 msgr2=0x7fd87c0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:00.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0b2590 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fd868009fd0 tx=0x7fd868004750 comp rx=0 tx=0).stop 2023-10-16T11:20:00.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 shutdown_connections 2023-10-16T11:20:00.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fd86006b1d0 0x7fd86006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c15a6d0 0x7fd87c1f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c0cd190 0x7fd87c1f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 --2- 172.21.15.158:0/1596776271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd87c0cfe30 0x7fd87c0b2590 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:00.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 >> 172.21.15.158:0/1596776271 conn(0x7fd87c06b9e0 msgr2=0x7fd87c06eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:00.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 shutdown_connections 2023-10-16T11:20:00.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:00.821+0000 7fd882d8c700 1 -- 172.21.15.158:0/1596776271 wait complete. 2023-10-16T11:20:00.826 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T11:20:00.835 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:20:00.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:00.836 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:20:00.847 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:00.845+0000 7fa523dc8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:00.880 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:20:00.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:00.881 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:20:00.932 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:00.931+0000 7fa523dc8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:20:00.937 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:20:00.937 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:00.951 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:00.949+0000 7f89884a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:00.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:20:00.955 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:20:00.956 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:00.994 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:20:00.995 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:20:00.995 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:01.013 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:20:01.013 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:01.089 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:01.088+0000 7fa523dc8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:01.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.157+0000 7f527912d700 1 Processor -- start 2023-10-16T11:20:01.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.157+0000 7f527912d700 1 -- start start 2023-10-16T11:20:01.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.157+0000 7f527912d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f52740cc5d0 0x7f527406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.157+0000 7f527912d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f527415dfb0 0x7f52740cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f527912d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527406e770 con 0x7f527406f340 2023-10-16T11:20:01.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f527912d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52740cf270 con 0x7f52740cc5d0 2023-10-16T11:20:01.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f527912d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527415e4e0 con 0x7f527415dfb0 2023-10-16T11:20:01.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f52740cc5d0 0x7f527406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f52740cc5d0 0x7f527406e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39366/0 (socket says 172.21.15.158:39366) 2023-10-16T11:20:01.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 -- 172.21.15.158:0/1720092273 learned_addr learned my addr 172.21.15.158:0/1720092273 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:01.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f527259c700 1 --2- 172.21.15.158:0/1720092273 >> v2:172.21.15.172:3300/0 conn(0x7f527415dfb0 0x7f52740cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 -- 172.21.15.158:0/1720092273 >> v2:172.21.15.172:3300/0 conn(0x7f527415dfb0 msgr2=0x7f52740cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:01.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 --2- 172.21.15.158:0/1720092273 >> v2:172.21.15.172:3300/0 conn(0x7f527415dfb0 0x7f52740cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 -- 172.21.15.158:0/1720092273 >> v1:172.21.15.172:6789/0 conn(0x7f527406f340 legacy=0x7f5274071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 -- 172.21.15.158:0/1720092273 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527415e7c0 con 0x7f52740cc5d0 2023-10-16T11:20:01.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5272d9d700 1 --2- 172.21.15.158:0/1720092273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52740cc5d0 0x7f527406e230 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f5274071460 tx=0x7f5264009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae76a8dd26df1e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5271d9b700 1 -- 172.21.15.158:0/1720092273 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264011830 con 0x7f52740cc5d0 2023-10-16T11:20:01.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5271d9b700 1 -- 172.21.15.158:0/1720092273 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5264011990 con 0x7f52740cc5d0 2023-10-16T11:20:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.158+0000 7f5271d9b700 1 -- 172.21.15.158:0/1720092273 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264011d20 con 0x7f52740cc5d0 2023-10-16T11:20:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 -- 172.21.15.158:0/1720092273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52740cc5d0 msgr2=0x7f527406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 --2- 172.21.15.158:0/1720092273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52740cc5d0 0x7f527406e230 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f5274071460 tx=0x7f5264009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 -- 172.21.15.158:0/1720092273 shutdown_connections 2023-10-16T11:20:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 --2- 172.21.15.158:0/1720092273 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52740cc5d0 0x7f527406e230 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 --2- 172.21.15.158:0/1720092273 >> v2:172.21.15.172:3300/0 conn(0x7f527415dfb0 0x7f52740cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 -- 172.21.15.158:0/1720092273 >> 172.21.15.158:0/1720092273 conn(0x7f52740d02b0 msgr2=0x7f52740d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:01.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 -- 172.21.15.158:0/1720092273 shutdown_connections 2023-10-16T11:20:01.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.159+0000 7f527912d700 1 -- 172.21.15.158:0/1720092273 wait complete. 2023-10-16T11:20:01.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.160+0000 7f527912d700 1 Processor -- start 2023-10-16T11:20:01.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.160+0000 7f527912d700 1 -- start start 2023-10-16T11:20:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527912d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f527406f340 0x7f527414a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527912d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52740cc5d0 0x7f527415d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527912d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f527415dfb0 0x7f52741f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527912d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52740ac890 con 0x7f527406f340 2023-10-16T11:20:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527912d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52740adb80 con 0x7f527415dfb0 2023-10-16T11:20:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527912d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527415d840 con 0x7f52740cc5d0 2023-10-16T11:20:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f5272d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f527406f340 0x7f527414a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f5272d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f527406f340 0x7f527414a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39372/0 (socket says 172.21.15.158:39372) 2023-10-16T11:20:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f5272d9d700 1 -- 172.21.15.158:0/3412993257 learned_addr learned my addr 172.21.15.158:0/3412993257 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527259c700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52740cc5d0 0x7f527415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.161+0000 7f527359e700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f527415dfb0 0x7f52741f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f5272d9d700 1 -- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52740cc5d0 msgr2=0x7f527415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f5272d9d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52740cc5d0 0x7f527415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f5272d9d700 1 -- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f527415dfb0 msgr2=0x7f52741f6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f5272d9d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f527415dfb0 0x7f52741f6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f5272d9d700 1 -- 172.21.15.158:0/3412993257 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5264009550 con 0x7f527406f340 2023-10-16T11:20:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f5272d9d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f527406f340 0x7f527414a880 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f5264009c10 tx=0x7f5264004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:01.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f525b7fe700 1 -- 172.21.15.158:0/3412993257 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264004510 con 0x7f527406f340 2023-10-16T11:20:01.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f525b7fe700 1 -- 172.21.15.158:0/3412993257 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526401e070 con 0x7f527406f340 2023-10-16T11:20:01.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52740acae0 con 0x7f527406f340 2023-10-16T11:20:01.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f525b7fe700 1 -- 172.21.15.158:0/3412993257 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52640038e0 con 0x7f527406f340 2023-10-16T11:20:01.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.162+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52740ad0a0 con 0x7f527406f340 2023-10-16T11:20:01.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.163+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52740681b0 con 0x7f527406f340 2023-10-16T11:20:01.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.163+0000 7f525b7fe700 1 -- 172.21.15.158:0/3412993257 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f5264003a40 con 0x7f527406f340 2023-10-16T11:20:01.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.164+0000 7f525b7fe700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 0x7f526006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.164+0000 7f527259c700 1 -- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 msgr2=0x7f526006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:01.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.164+0000 7f525b7fe700 1 -- 172.21.15.158:0/3412993257 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f526408afb0 con 0x7f527406f340 2023-10-16T11:20:01.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.164+0000 7f527259c700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 0x7f526006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:01.186 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:01.182+0000 7f89884a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:20:01.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.168+0000 7f525b7fe700 1 -- 172.21.15.158:0/3412993257 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52640bdfe0 con 0x7f527406f340 2023-10-16T11:20:01.258 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:01.256+0000 7f89884a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:20:01.341 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:01.339+0000 7fa523dc8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:20:01.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.365+0000 7f527259c700 1 -- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 msgr2=0x7f526006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:01.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.365+0000 7f527259c700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 0x7f526006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:01.417 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:01.416+0000 7fa523dc8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:20:01.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.424+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5274072050 con 0x7f527406f340 2023-10-16T11:20:01.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.425+0000 7f525b7fe700 1 -- 172.21.15.158:0/3412993257 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6163 (secure 0 0 0) 0x7f5264015090 con 0x7f527406f340 2023-10-16T11:20:01.427 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:01.427 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:20:01.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 msgr2=0x7f526006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:01.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 0x7f526006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f527406f340 msgr2=0x7f527414a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f527406f340 0x7f527414a880 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f5264009c10 tx=0x7f5264004750 comp rx=0 tx=0).stop 2023-10-16T11:20:01.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 shutdown_connections 2023-10-16T11:20:01.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f526006b1d0 0x7f526006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f527415dfb0 0x7f52741f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f527406f340 0x7f527414a880 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 --2- 172.21.15.158:0/3412993257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52740cc5d0 0x7f527415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 >> 172.21.15.158:0/3412993257 conn(0x7f52740d02b0 msgr2=0x7f527406f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:01.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 shutdown_connections 2023-10-16T11:20:01.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.428+0000 7f527912d700 1 -- 172.21.15.158:0/3412993257 wait complete. 2023-10-16T11:20:01.432 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 118 2023-10-16T11:20:01.443 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:01.522 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:01.521+0000 7f89884a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:20:01.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.570+0000 7f8871819700 1 Processor -- start 2023-10-16T11:20:01.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.570+0000 7f8871819700 1 -- start start 2023-10-16T11:20:01.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.570+0000 7f8871819700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f886c0cf8a0 0x7f886c161270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.570+0000 7f8871819700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c06bd10 0x7f886c15e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.570+0000 7f8871819700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886c15de20 con 0x7f886c0cf4c0 2023-10-16T11:20:01.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.570+0000 7f8871819700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f886c162180 con 0x7f886c0cf8a0 2023-10-16T11:20:01.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.570+0000 7f8871819700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c1618e0 con 0x7f886c06bd10 2023-10-16T11:20:01.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f886c0cf8a0 0x7f886c161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f886c0cf4c0 0x7f886c06d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39558/0 (socket says 172.21.15.158:39558) 2023-10-16T11:20:01.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886affd700 1 -- 172.21.15.158:0/2528910349 learned_addr learned my addr 172.21.15.158:0/2528910349 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:01.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886b7fe700 1 --2- 172.21.15.158:0/2528910349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c06bd10 0x7f886c15e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886a7fc700 1 -- 172.21.15.158:0/2528910349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c06bd10 msgr2=0x7f886c15e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:01.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886a7fc700 1 --2- 172.21.15.158:0/2528910349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c06bd10 0x7f886c15e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886a7fc700 1 -- 172.21.15.158:0/2528910349 >> v1:172.21.15.158:6789/0 conn(0x7f886c0cf4c0 legacy=0x7f886c06d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886a7fc700 1 -- 172.21.15.158:0/2528910349 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c161bc0 con 0x7f886c0cf8a0 2023-10-16T11:20:01.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f886a7fc700 1 --2- 172.21.15.158:0/2528910349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf8a0 0x7f886c161270 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f8860009840 tx=0x7f8860009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199261fe4c75eaf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:01.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f8869ffb700 1 -- 172.21.15.158:0/2528910349 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860011830 con 0x7f886c0cf8a0 2023-10-16T11:20:01.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f8869ffb700 1 -- 172.21.15.158:0/2528910349 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8860011990 con 0x7f886c0cf8a0 2023-10-16T11:20:01.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.571+0000 7f8869ffb700 1 -- 172.21.15.158:0/2528910349 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860011d20 con 0x7f886c0cf8a0 2023-10-16T11:20:01.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 -- 172.21.15.158:0/2528910349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf8a0 msgr2=0x7f886c161270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 --2- 172.21.15.158:0/2528910349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf8a0 0x7f886c161270 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f8860009840 tx=0x7f8860009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:01.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 -- 172.21.15.158:0/2528910349 shutdown_connections 2023-10-16T11:20:01.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 --2- 172.21.15.158:0/2528910349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c06bd10 0x7f886c15e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 --2- 172.21.15.158:0/2528910349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf8a0 0x7f886c161270 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 -- 172.21.15.158:0/2528910349 >> 172.21.15.158:0/2528910349 conn(0x7f886c06a8d0 msgr2=0x7f886c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:01.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 -- 172.21.15.158:0/2528910349 shutdown_connections 2023-10-16T11:20:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.572+0000 7f8871819700 1 -- 172.21.15.158:0/2528910349 wait complete. 2023-10-16T11:20:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.573+0000 7f8871819700 1 Processor -- start 2023-10-16T11:20:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.573+0000 7f8871819700 1 -- start start 2023-10-16T11:20:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f8871819700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f886c06bd10 0x7f886c152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f8871819700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf4c0 0x7f886c1549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f8871819700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c0cf8a0 0x7f886c0b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f8871819700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0b4750 con 0x7f886c0cf4c0 2023-10-16T11:20:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f8871819700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0b3b40 con 0x7f886c06bd10 2023-10-16T11:20:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f8871819700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0b3e20 con 0x7f886c0cf8a0 2023-10-16T11:20:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf4c0 0x7f886c1549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf4c0 0x7f886c1549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39390/0 (socket says 172.21.15.158:39390) 2023-10-16T11:20:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 -- 172.21.15.158:0/2774669969 learned_addr learned my addr 172.21.15.158:0/2774669969 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886affd700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f886c06bd10 0x7f886c152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886b7fe700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c0cf8a0 0x7f886c0b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 -- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c0cf8a0 msgr2=0x7f886c0b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c0cf8a0 0x7f886c0b6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 -- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f886c06bd10 msgr2=0x7f886c152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f886c06bd10 0x7f886c152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.574+0000 7f886a7fc700 1 -- 172.21.15.158:0/2774669969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88600094a0 con 0x7f886c0cf4c0 2023-10-16T11:20:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.575+0000 7f886a7fc700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf4c0 0x7f886c1549d0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f8860000c00 tx=0x7f8860004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.575+0000 7f8853fff700 1 -- 172.21.15.158:0/2774669969 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88600036a0 con 0x7f886c0cf4c0 2023-10-16T11:20:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.575+0000 7f8853fff700 1 -- 172.21.15.158:0/2774669969 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886001e070 con 0x7f886c0cf4c0 2023-10-16T11:20:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.575+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886c0b4100 con 0x7f886c0cf4c0 2023-10-16T11:20:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.575+0000 7f8853fff700 1 -- 172.21.15.158:0/2774669969 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860003a60 con 0x7f886c0cf4c0 2023-10-16T11:20:01.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.575+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886c153330 con 0x7f886c0cf4c0 2023-10-16T11:20:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.576+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886c161e80 con 0x7f886c0cf4c0 2023-10-16T11:20:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.576+0000 7f8853fff700 1 -- 172.21.15.158:0/2774669969 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f8860003bc0 con 0x7f886c0cf4c0 2023-10-16T11:20:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.577+0000 7f8853fff700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 0x7f885806d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.577+0000 7f886affd700 1 -- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 msgr2=0x7f885806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:01.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.577+0000 7f8853fff700 1 -- 172.21.15.158:0/2774669969 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f886008b120 con 0x7f886c0cf4c0 2023-10-16T11:20:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.577+0000 7f886affd700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 0x7f885806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:01.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.581+0000 7f8853fff700 1 -- 172.21.15.158:0/2774669969 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f886c161e80 con 0x7f886c0cf4c0 2023-10-16T11:20:01.699 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:01.697+0000 7fa523dc8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:20:01.745 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:01.744+0000 7f89884a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:20:01.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.778+0000 7f886affd700 1 -- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 msgr2=0x7f885806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:01.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.778+0000 7f886affd700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 0x7f885806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:01.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.780+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f886c161e80 con 0x7f886c0cf4c0 2023-10-16T11:20:01.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.782+0000 7f8853fff700 1 -- 172.21.15.158:0/2774669969 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+21694 (secure 0 0 0) 0x7f886c161e80 con 0x7f886c0cf4c0 2023-10-16T11:20:01.784 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:01.784 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":222,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:19:52.926692+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 msgr2=0x7f885806d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 0x7f885806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf4c0 msgr2=0x7f886c1549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf4c0 0x7f886c1549d0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f8860000c00 tx=0x7f8860004750 comp rx=0 tx=0).stop 2023-10-16T11:20:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 shutdown_connections 2023-10-16T11:20:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f885806b1d0 0x7f885806d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f886c0cf8a0 0x7f886c0b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f886c06bd10 0x7f886c152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 --2- 172.21.15.158:0/2774669969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c0cf4c0 0x7f886c1549d0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 >> 172.21.15.158:0/2774669969 conn(0x7f886c06a8d0 msgr2=0x7f886c15e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:01.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 shutdown_connections 2023-10-16T11:20:01.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.784+0000 7f8871819700 1 -- 172.21.15.158:0/2774669969 wait complete. 2023-10-16T11:20:01.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:19:21.151561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '216', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:19:21.679054+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T11:19:33.715104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '220', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T11:19:35.032906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:20:01.800 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:20:01.896 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:01.894+0000 7f89884a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:20:01.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.924+0000 7fbf006c7700 1 Processor -- start 2023-10-16T11:20:01.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.924+0000 7fbf006c7700 1 -- start start 2023-10-16T11:20:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.925+0000 7fbf006c7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbef815da40 0x7fbef80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.925+0000 7fbf006c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef80afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.925+0000 7fbf006c7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef80cbd30 con 0x7fbef80cc590 2023-10-16T11:20:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.925+0000 7fbf006c7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef80cc1c0 con 0x7fbef815da40 2023-10-16T11:20:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.925+0000 7fbf006c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef815de20 con 0x7fbef806afe0 2023-10-16T11:20:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.925+0000 7fbefdc62700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbef80cc590 0x7fbef806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39574/0 (socket says 172.21.15.158:39574) 2023-10-16T11:20:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.925+0000 7fbefdc62700 1 -- 172.21.15.158:0/3113184460 learned_addr learned my addr 172.21.15.158:0/3113184460 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefe463700 1 --2- 172.21.15.158:0/3113184460 >> v2:172.21.15.172:3300/0 conn(0x7fbef815da40 0x7fbef80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefec64700 1 --2- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef80afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefd461700 1 -- 172.21.15.158:0/3113184460 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483852319 0 0) 0x7fbef80cbd30 con 0x7fbef80cc590 2023-10-16T11:20:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefd461700 1 -- 172.21.15.158:0/3113184460 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee40035b0 con 0x7fbef80cc590 2023-10-16T11:20:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefe463700 1 -- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 msgr2=0x7fbef80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefe463700 1 --2- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef80afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefe463700 1 -- 172.21.15.158:0/3113184460 >> v1:172.21.15.158:6789/0 conn(0x7fbef80cc590 legacy=0x7fbef806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefe463700 1 -- 172.21.15.158:0/3113184460 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef80ce260 con 0x7fbef815da40 2023-10-16T11:20:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefe463700 1 --2- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef815da40 0x7fbef80caa20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009840 tx=0x7fbef4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=233143c3e1eb4868 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefd461700 1 -- 172.21.15.158:0/3113184460 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4011830 con 0x7fbef815da40 2023-10-16T11:20:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefd461700 1 -- 172.21.15.158:0/3113184460 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef4011990 con 0x7fbef815da40 2023-10-16T11:20:01.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.926+0000 7fbefd461700 1 -- 172.21.15.158:0/3113184460 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4011d00 con 0x7fbef815da40 2023-10-16T11:20:01.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 -- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef815da40 msgr2=0x7fbef80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 --2- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef815da40 0x7fbef80caa20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009840 tx=0x7fbef4009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:01.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 -- 172.21.15.158:0/3113184460 shutdown_connections 2023-10-16T11:20:01.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 --2- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef815da40 0x7fbef80caa20 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 --2- 172.21.15.158:0/3113184460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef80afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 -- 172.21.15.158:0/3113184460 >> 172.21.15.158:0/3113184460 conn(0x7fbef8099a10 msgr2=0x7fbef80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:01.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 -- 172.21.15.158:0/3113184460 shutdown_connections 2023-10-16T11:20:01.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.927+0000 7fbf006c7700 1 -- 172.21.15.158:0/3113184460 wait complete. 2023-10-16T11:20:01.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.928+0000 7fbf006c7700 1 Processor -- start 2023-10-16T11:20:01.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.928+0000 7fbf006c7700 1 -- start start 2023-10-16T11:20:01.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbf006c7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbf006c7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef80cc590 0x7fbef80d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbf006c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbef815da40 0x7fbef80d0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbf006c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c74a0 con 0x7fbef815da40 2023-10-16T11:20:01.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbf006c7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c82c0 con 0x7fbef80cc590 2023-10-16T11:20:01.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbf006c7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c76b0 con 0x7fbef806afe0 2023-10-16T11:20:01.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefec64700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbef815da40 0x7fbef80d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefec64700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbef815da40 0x7fbef80d0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39406/0 (socket says 172.21.15.158:39406) 2023-10-16T11:20:01.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefec64700 1 -- 172.21.15.158:0/847863722 learned_addr learned my addr 172.21.15.158:0/847863722 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:01.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefe463700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefdc62700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef80cc590 0x7fbef80d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:01.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefdc62700 1 -- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 msgr2=0x7fbef815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefdc62700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef815b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefdc62700 1 -- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbef815da40 msgr2=0x7fbef80d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:01.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefdc62700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbef815da40 0x7fbef80d0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:01.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefdc62700 1 -- 172.21.15.158:0/847863722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef40094a0 con 0x7fbef80cc590 2023-10-16T11:20:01.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.929+0000 7fbefec64700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbef815da40 0x7fbef80d0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:01.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.930+0000 7fbefdc62700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef80cc590 0x7fbef80d0240 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fbef80cca40 tx=0x7fbee800c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:01.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.930+0000 7fbeeeffd700 1 -- 172.21.15.158:0/847863722 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8005b70 con 0x7fbef80cc590 2023-10-16T11:20:01.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.930+0000 7fbeeeffd700 1 -- 172.21.15.158:0/847863722 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee8005cd0 con 0x7fbef80cc590 2023-10-16T11:20:01.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.930+0000 7fbeeeffd700 1 -- 172.21.15.158:0/847863722 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee80072f0 con 0x7fbef80cc590 2023-10-16T11:20:01.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.930+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef80c7990 con 0x7fbef80cc590 2023-10-16T11:20:01.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.930+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef80c7f50 con 0x7fbef80cc590 2023-10-16T11:20:01.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.931+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef80681b0 con 0x7fbef80cc590 2023-10-16T11:20:01.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.931+0000 7fbeeeffd700 1 -- 172.21.15.158:0/847863722 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fbee8007450 con 0x7fbef80cc590 2023-10-16T11:20:01.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.932+0000 7fbeeeffd700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 0x7fbedc06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:01.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.932+0000 7fbefe463700 1 -- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 msgr2=0x7fbedc06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:01.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.932+0000 7fbeeeffd700 1 -- 172.21.15.158:0/847863722 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fbee80894b0 con 0x7fbef80cc590 2023-10-16T11:20:01.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.932+0000 7fbefe463700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 0x7fbedc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:01.943 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:01.935+0000 7fa523dc8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:20:01.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:01.940+0000 7fbeeeffd700 1 -- 172.21.15.158:0/847863722 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbee8053a30 con 0x7fbef80cc590 2023-10-16T11:20:02.014 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:02.013+0000 7f89884a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:20:02.104 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:02.102+0000 7fa523dc8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:20:02.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.133+0000 7fbefe463700 1 -- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 msgr2=0x7fbedc06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:02.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.133+0000 7fbefe463700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 0x7fbedc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:02.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.155+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbef80c8470 con 0x7fbef80cc590 2023-10-16T11:20:02.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.156+0000 7fbeeeffd700 1 -- 172.21.15.158:0/847863722 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v222) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbee80170a0 con 0x7fbef80cc590 2023-10-16T11:20:02.158 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:20:02.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 msgr2=0x7fbedc06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:02.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 0x7fbedc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef80cc590 msgr2=0x7fbef80d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef80cc590 0x7fbef80d0240 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fbef80cca40 tx=0x7fbee800c700 comp rx=0 tx=0).stop 2023-10-16T11:20:02.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 shutdown_connections 2023-10-16T11:20:02.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fbedc06b1d0 0x7fbedc06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbef815da40 0x7fbef80d0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbef806afe0 0x7fbef815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 --2- 172.21.15.158:0/847863722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbef80cc590 0x7fbef80d0240 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 >> 172.21.15.158:0/847863722 conn(0x7fbef8099a10 msgr2=0x7fbef806f330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:02.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 shutdown_connections 2023-10-16T11:20:02.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.158+0000 7fbf006c7700 1 -- 172.21.15.158:0/847863722 wait complete. 2023-10-16T11:20:02.173 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:20:02.173 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:02.171+0000 7f89884a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:20:02.226 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:02.224+0000 7fa523dc8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:20:02.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.298+0000 7f6df466c700 1 Processor -- start 2023-10-16T11:20:02.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.298+0000 7f6df466c700 1 -- start start 2023-10-16T11:20:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.298+0000 7f6df466c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec141920 0x7f6dec06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df466c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6dec144840 0x7f6dec0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df466c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6dec0cdc40 0x7f6dec15f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df466c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dec0ce0d0 con 0x7f6dec144840 2023-10-16T11:20:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df466c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dec06c210 con 0x7f6dec0cdc40 2023-10-16T11:20:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df466c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec141d00 con 0x7f6dec141920 2023-10-16T11:20:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6dec144840 0x7f6dec0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6dec144840 0x7f6dec0cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39414/0 (socket says 172.21.15.158:39414) 2023-10-16T11:20:02.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 -- 172.21.15.158:0/3356021936 learned_addr learned my addr 172.21.15.158:0/3356021936 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df2408700 1 --2- 172.21.15.158:0/3356021936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec141920 0x7f6dec06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df2c09700 1 --2- 172.21.15.158:0/3356021936 >> v2:172.21.15.172:3300/0 conn(0x7f6dec0cdc40 0x7f6dec15f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 -- 172.21.15.158:0/3356021936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec141920 msgr2=0x7f6dec06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 --2- 172.21.15.158:0/3356021936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec141920 0x7f6dec06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 -- 172.21.15.158:0/3356021936 >> v2:172.21.15.172:3300/0 conn(0x7f6dec0cdc40 msgr2=0x7f6dec15f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:02.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 --2- 172.21.15.158:0/3356021936 >> v2:172.21.15.172:3300/0 conn(0x7f6dec0cdc40 0x7f6dec15f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 -- 172.21.15.158:0/3356021936 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec06b970 con 0x7f6dec144840 2023-10-16T11:20:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1c07700 1 --2- 172.21.15.158:0/3356021936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec144840 0x7f6dec0cd700 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0ca110 tx=0x7f6ddc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37dc088ebdfc62ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1406700 1 -- 172.21.15.158:0/3356021936 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc00b870 con 0x7f6dec144840 2023-10-16T11:20:02.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1406700 1 -- 172.21.15.158:0/3356021936 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ddc00b9d0 con 0x7f6dec144840 2023-10-16T11:20:02.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.299+0000 7f6df1406700 1 -- 172.21.15.158:0/3356021936 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc00bd60 con 0x7f6dec144840 2023-10-16T11:20:02.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 -- 172.21.15.158:0/3356021936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec144840 msgr2=0x7f6dec0cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 --2- 172.21.15.158:0/3356021936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec144840 0x7f6dec0cd700 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0ca110 tx=0x7f6ddc009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 -- 172.21.15.158:0/3356021936 shutdown_connections 2023-10-16T11:20:02.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 --2- 172.21.15.158:0/3356021936 >> v2:172.21.15.172:3300/0 conn(0x7f6dec0cdc40 0x7f6dec15f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 --2- 172.21.15.158:0/3356021936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec141920 0x7f6dec06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 --2- 172.21.15.158:0/3356021936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec144840 0x7f6dec0cd700 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 -- 172.21.15.158:0/3356021936 >> 172.21.15.158:0/3356021936 conn(0x7f6dec0cc5b0 msgr2=0x7f6dec0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:02.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 -- 172.21.15.158:0/3356021936 shutdown_connections 2023-10-16T11:20:02.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.300+0000 7f6df466c700 1 -- 172.21.15.158:0/3356021936 wait complete. 2023-10-16T11:20:02.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.302+0000 7f6df466c700 1 Processor -- start 2023-10-16T11:20:02.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.302+0000 7f6df466c700 1 -- start start 2023-10-16T11:20:02.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df466c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec0cdc40 0x7f6dec1e4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df466c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec141920 0x7f6dec140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df466c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dec144840 0x7f6dec1f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df466c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec1e5f10 con 0x7f6dec141920 2023-10-16T11:20:02.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df466c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec140650 con 0x7f6dec144840 2023-10-16T11:20:02.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df466c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec1f4790 con 0x7f6dec0cdc40 2023-10-16T11:20:02.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec141920 0x7f6dec140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec141920 0x7f6dec140110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39420/0 (socket says 172.21.15.158:39420) 2023-10-16T11:20:02.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 -- 172.21.15.158:0/4247858408 learned_addr learned my addr 172.21.15.158:0/4247858408 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:02.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df2408700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec0cdc40 0x7f6dec1e4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df2c09700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dec144840 0x7f6dec1f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 -- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec0cdc40 msgr2=0x7f6dec1e4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec0cdc40 0x7f6dec1e4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 -- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dec144840 msgr2=0x7f6dec1f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dec144840 0x7f6dec1f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 -- 172.21.15.158:0/4247858408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc009550 con 0x7f6dec141920 2023-10-16T11:20:02.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df1c07700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec141920 0x7f6dec140110 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc000c00 tx=0x7f6ddc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:02.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6de2ffd700 1 -- 172.21.15.158:0/4247858408 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc004310 con 0x7f6dec141920 2023-10-16T11:20:02.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6de2ffd700 1 -- 172.21.15.158:0/4247858408 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ddc01e070 con 0x7f6dec141920 2023-10-16T11:20:02.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6de2ffd700 1 -- 172.21.15.158:0/4247858408 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc0036a0 con 0x7f6dec141920 2023-10-16T11:20:02.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.303+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dec1f49c0 con 0x7f6dec141920 2023-10-16T11:20:02.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.304+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dec1f4f80 con 0x7f6dec141920 2023-10-16T11:20:02.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.305+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dec0681b0 con 0x7f6dec141920 2023-10-16T11:20:02.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.305+0000 7f6de2ffd700 1 -- 172.21.15.158:0/4247858408 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f6ddc020030 con 0x7f6dec141920 2023-10-16T11:20:02.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.305+0000 7f6de2ffd700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 0x7f6dd006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.305+0000 7f6df2408700 1 -- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 msgr2=0x7f6dd006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:02.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.305+0000 7f6de2ffd700 1 -- 172.21.15.158:0/4247858408 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7f6ddc08b080 con 0x7f6dec141920 2023-10-16T11:20:02.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.305+0000 7f6df2408700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 0x7f6dd006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:02.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.310+0000 7f6de2ffd700 1 -- 172.21.15.158:0/4247858408 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ddc055690 con 0x7f6dec141920 2023-10-16T11:20:02.393 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:02.392+0000 7fa523dc8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:20:02.438 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:02.436+0000 7f89884a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:02.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.506+0000 7f6df2408700 1 -- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 msgr2=0x7f6dd006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:02.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.506+0000 7f6df2408700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 0x7f6dd006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:02.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.510+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6dec0725f0 con 0x7f6dec141920 2023-10-16T11:20:02.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.510+0000 7f6de2ffd700 1 -- 172.21.15.158:0/4247858408 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v222) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ddc003800 con 0x7f6dec141920 2023-10-16T11:20:02.512 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:02.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.512+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 msgr2=0x7f6dd006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:02.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.512+0000 7f6df466c700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 0x7f6dd006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec141920 msgr2=0x7f6dec140110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec141920 0x7f6dec140110 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc000c00 tx=0x7f6ddc004750 comp rx=0 tx=0).stop 2023-10-16T11:20:02.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 shutdown_connections 2023-10-16T11:20:02.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7f6dd006b1d0 0x7f6dd006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dec144840 0x7f6dec1f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dec0cdc40 0x7f6dec1e4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 --2- 172.21.15.158:0/4247858408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6dec141920 0x7f6dec140110 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 >> 172.21.15.158:0/4247858408 conn(0x7f6dec0cc5b0 msgr2=0x7f6dec1e33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:02.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 shutdown_connections 2023-10-16T11:20:02.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.513+0000 7f6df466c700 1 -- 172.21.15.158:0/4247858408 wait complete. 2023-10-16T11:20:02.530 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:20:02.600 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:02.598+0000 7f89884a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:20:02.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.656+0000 7fee5e89d700 1 Processor -- start 2023-10-16T11:20:02.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.656+0000 7fee5e89d700 1 -- start start 2023-10-16T11:20:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.656+0000 7fee5e89d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee5815da40 0x7fee580caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.656+0000 7fee5e89d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5806afe0 0x7fee580afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.656+0000 7fee5e89d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee580cbd30 con 0x7fee580cc590 2023-10-16T11:20:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.656+0000 7fee5e89d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee580cc1c0 con 0x7fee5815da40 2023-10-16T11:20:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.656+0000 7fee5e89d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5815de20 con 0x7fee5806afe0 2023-10-16T11:20:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee5815da40 0x7fee580caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee5815da40 0x7fee580caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39428/0 (socket says 172.21.15.158:39428) 2023-10-16T11:20:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee577fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fee580cc590 0x7fee5806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39606/0 (socket says 172.21.15.158:39606) 2023-10-16T11:20:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 -- 172.21.15.158:0/2113848197 learned_addr learned my addr 172.21.15.158:0/2113848197 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee5ce3a700 1 --2- 172.21.15.158:0/2113848197 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5806afe0 0x7fee580afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 -- 172.21.15.158:0/2113848197 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5806afe0 msgr2=0x7fee580afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 --2- 172.21.15.158:0/2113848197 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5806afe0 0x7fee580afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 -- 172.21.15.158:0/2113848197 >> v1:172.21.15.158:6789/0 conn(0x7fee580cc590 legacy=0x7fee5806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 -- 172.21.15.158:0/2113848197 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee580ce260 con 0x7fee5815da40 2023-10-16T11:20:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee5ce3a700 1 --2- 172.21.15.158:0/2113848197 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5806afe0 0x7fee580afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee56ffd700 1 -- 172.21.15.158:0/2113848197 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584218920 0 0) 0x7fee580cbd30 con 0x7fee580cc590 2023-10-16T11:20:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee57fff700 1 --2- 172.21.15.158:0/2113848197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5815da40 0x7fee580caa20 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fee480098f0 tx=0x7fee48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b8071a7cd765c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee56ffd700 1 -- 172.21.15.158:0/2113848197 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4800b8a0 con 0x7fee5815da40 2023-10-16T11:20:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee56ffd700 1 -- 172.21.15.158:0/2113848197 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4800ba00 con 0x7fee5815da40 2023-10-16T11:20:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.657+0000 7fee56ffd700 1 -- 172.21.15.158:0/2113848197 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4800bd90 con 0x7fee5815da40 2023-10-16T11:20:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 -- 172.21.15.158:0/2113848197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5815da40 msgr2=0x7fee580caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 --2- 172.21.15.158:0/2113848197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5815da40 0x7fee580caa20 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fee480098f0 tx=0x7fee48009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 -- 172.21.15.158:0/2113848197 shutdown_connections 2023-10-16T11:20:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 --2- 172.21.15.158:0/2113848197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5815da40 0x7fee580caa20 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 --2- 172.21.15.158:0/2113848197 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5806afe0 0x7fee580afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 -- 172.21.15.158:0/2113848197 >> 172.21.15.158:0/2113848197 conn(0x7fee58099a10 msgr2=0x7fee580d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 -- 172.21.15.158:0/2113848197 shutdown_connections 2023-10-16T11:20:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.658+0000 7fee5e89d700 1 -- 172.21.15.158:0/2113848197 wait complete. 2023-10-16T11:20:02.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.660+0000 7fee5e89d700 1 Processor -- start 2023-10-16T11:20:02.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.660+0000 7fee5e89d700 1 -- start start 2023-10-16T11:20:02.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5806afe0 0x7fee580c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee580cc590 0x7fee581e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5815da40 0x7fee580b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580c5b70 con 0x7fee5806afe0 2023-10-16T11:20:02.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580c52a0 con 0x7fee580cc590 2023-10-16T11:20:02.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee581e5390 con 0x7fee5815da40 2023-10-16T11:20:02.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5806afe0 0x7fee580c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5806afe0 0x7fee580c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39434/0 (socket says 172.21.15.158:39434) 2023-10-16T11:20:02.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 -- 172.21.15.158:0/927590824 learned_addr learned my addr 172.21.15.158:0/927590824 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:02.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5ce3a700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5815da40 0x7fee580b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee577fe700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee580cc590 0x7fee581e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 -- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5815da40 msgr2=0x7fee580b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5815da40 0x7fee580b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 -- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee580cc590 msgr2=0x7fee581e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee580cc590 0x7fee581e7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 -- 172.21.15.158:0/927590824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee48009550 con 0x7fee5806afe0 2023-10-16T11:20:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee57fff700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5806afe0 0x7fee580c74a0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fee48009c40 tx=0x7fee48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee54ff9700 1 -- 172.21.15.158:0/927590824 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee480044d0 con 0x7fee5806afe0 2023-10-16T11:20:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee54ff9700 1 -- 172.21.15.158:0/927590824 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4801f070 con 0x7fee5806afe0 2023-10-16T11:20:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee580c54b0 con 0x7fee5806afe0 2023-10-16T11:20:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee54ff9700 1 -- 172.21.15.158:0/927590824 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee480038e0 con 0x7fee5806afe0 2023-10-16T11:20:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.662+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee580cab50 con 0x7fee5806afe0 2023-10-16T11:20:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.664+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee580681b0 con 0x7fee5806afe0 2023-10-16T11:20:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.664+0000 7fee54ff9700 1 -- 172.21.15.158:0/927590824 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fee48003a40 con 0x7fee5806afe0 2023-10-16T11:20:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.664+0000 7fee54ff9700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 0x7fee4406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.665+0000 7fee577fe700 1 -- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 msgr2=0x7fee4406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.665+0000 7fee54ff9700 1 -- 172.21.15.158:0/927590824 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fee48021070 con 0x7fee5806afe0 2023-10-16T11:20:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.665+0000 7fee577fe700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 0x7fee4406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:02.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.669+0000 7fee54ff9700 1 -- 172.21.15.158:0/927590824 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee48056000 con 0x7fee5806afe0 2023-10-16T11:20:02.676 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:02.675+0000 7fa523dc8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:02.846 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:02.844+0000 7fa523dc8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:20:02.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.866+0000 7fee577fe700 1 -- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 msgr2=0x7fee4406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:02.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.866+0000 7fee577fe700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 0x7fee4406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:02.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.866+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee5806f4e0 con 0x7fee5806afe0 2023-10-16T11:20:02.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.867+0000 7fee54ff9700 1 -- 172.21.15.158:0/927590824 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v222) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee480150a0 con 0x7fee5806afe0 2023-10-16T11:20:02.869 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 msgr2=0x7fee4406d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 0x7fee4406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5806afe0 msgr2=0x7fee580c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:02.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5806afe0 0x7fee580c74a0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fee48009c40 tx=0x7fee48004750 comp rx=0 tx=0).stop 2023-10-16T11:20:02.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 shutdown_connections 2023-10-16T11:20:02.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fee4406b1d0 0x7fee4406d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee5815da40 0x7fee580b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee5806afe0 0x7fee580c74a0 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 --2- 172.21.15.158:0/927590824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee580cc590 0x7fee581e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 >> 172.21.15.158:0/927590824 conn(0x7fee58099a10 msgr2=0x7fee5815ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 shutdown_connections 2023-10-16T11:20:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:02.869+0000 7fee5e89d700 1 -- 172.21.15.158:0/927590824 wait complete. 2023-10-16T11:20:02.883 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T11:20:03.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.009+0000 7fe06fba7700 1 Processor -- start 2023-10-16T11:20:03.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.009+0000 7fe06fba7700 1 -- start start 2023-10-16T11:20:03.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06fba7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe0680b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06fba7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe0680b3460 0x7fe0680b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06fba7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680b4420 con 0x7fe06815b6b0 2023-10-16T11:20:03.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06fba7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06815ba90 con 0x7fe0680b3460 2023-10-16T11:20:03.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06fba7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680b38f0 con 0x7fe0680d0c60 2023-10-16T11:20:03.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06d943700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe06815b6b0 0x7fe0680cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39620/0 (socket says 172.21.15.158:39620) 2023-10-16T11:20:03.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06d943700 1 -- 172.21.15.158:0/3175446003 learned_addr learned my addr 172.21.15.158:0/3175446003 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:03.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06e144700 1 --2- 172.21.15.158:0/3175446003 >> v2:172.21.15.158:3300/0 conn(0x7fe0680b3460 0x7fe0680b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06d142700 1 --2- 172.21.15.158:0/3175446003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe0680b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06c941700 1 -- 172.21.15.158:0/3175446003 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433087808 0 0) 0x7fe0680b4420 con 0x7fe06815b6b0 2023-10-16T11:20:03.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06c941700 1 -- 172.21.15.158:0/3175446003 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0540035b0 con 0x7fe06815b6b0 2023-10-16T11:20:03.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06e144700 1 -- 172.21.15.158:0/3175446003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 msgr2=0x7fe0680b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:03.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06e144700 1 --2- 172.21.15.158:0/3175446003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe0680b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06e144700 1 -- 172.21.15.158:0/3175446003 >> v1:172.21.15.158:6789/0 conn(0x7fe06815b6b0 legacy=0x7fe0680cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.010+0000 7fe06e144700 1 -- 172.21.15.158:0/3175446003 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680cbc20 con 0x7fe0680b3460 2023-10-16T11:20:03.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06e144700 1 --2- 172.21.15.158:0/3175446003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680b48b0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fe060009d90 tx=0x7fe06000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb084d8b5028451 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:03.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06c941700 1 -- 172.21.15.158:0/3175446003 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06000e070 con 0x7fe0680b3460 2023-10-16T11:20:03.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06c941700 1 -- 172.21.15.158:0/3175446003 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06000bb20 con 0x7fe0680b3460 2023-10-16T11:20:03.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06c941700 1 -- 172.21.15.158:0/3175446003 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06000be90 con 0x7fe0680b3460 2023-10-16T11:20:03.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06fba7700 1 -- 172.21.15.158:0/3175446003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 msgr2=0x7fe0680b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3175446003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680b48b0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fe060009d90 tx=0x7fe06000b330 comp rx=0 tx=0).stop 2023-10-16T11:20:03.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06fba7700 1 -- 172.21.15.158:0/3175446003 shutdown_connections 2023-10-16T11:20:03.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3175446003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680b48b0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3175446003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe0680b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.011+0000 7fe06fba7700 1 -- 172.21.15.158:0/3175446003 >> 172.21.15.158:0/3175446003 conn(0x7fe0680cdf20 msgr2=0x7fe0680ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:03.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.012+0000 7fe06fba7700 1 -- 172.21.15.158:0/3175446003 shutdown_connections 2023-10-16T11:20:03.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.012+0000 7fe06fba7700 1 -- 172.21.15.158:0/3175446003 wait complete. 2023-10-16T11:20:03.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 Processor -- start 2023-10-16T11:20:03.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 -- start start 2023-10-16T11:20:03.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe068151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06815b6b0 0x7fe0681f07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0681f1690 con 0x7fe0680b3460 2023-10-16T11:20:03.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680c9650 con 0x7fe06815b6b0 2023-10-16T11:20:03.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.013+0000 7fe06fba7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe068154670 con 0x7fe0680d0c60 2023-10-16T11:20:03.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680c71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39450/0 (socket says 172.21.15.158:39450) 2023-10-16T11:20:03.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 -- 172.21.15.158:0/3389408361 learned_addr learned my addr 172.21.15.158:0/3389408361 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:03.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d142700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe068151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06e144700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06815b6b0 0x7fe0681f07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 -- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 msgr2=0x7fe068151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe068151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 -- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06815b6b0 msgr2=0x7fe0681f07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06815b6b0 0x7fe0681f07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 -- 172.21.15.158:0/3389408361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe060009a70 con 0x7fe0680b3460 2023-10-16T11:20:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06d943700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680c71c0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c420 tx=0x7fe06400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:03.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe05e7fc700 1 -- 172.21.15.158:0/3389408361 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064005d70 con 0x7fe0680b3460 2023-10-16T11:20:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe05e7fc700 1 -- 172.21.15.158:0/3389408361 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06400ce30 con 0x7fe0680b3460 2023-10-16T11:20:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe068154950 con 0x7fe0680b3460 2023-10-16T11:20:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe05e7fc700 1 -- 172.21.15.158:0/3389408361 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640072d0 con 0x7fe0680b3460 2023-10-16T11:20:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.014+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe068154f10 con 0x7fe0680b3460 2023-10-16T11:20:03.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.015+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0680681b0 con 0x7fe0680b3460 2023-10-16T11:20:03.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.016+0000 7fe05e7fc700 1 -- 172.21.15.158:0/3389408361 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fe064007430 con 0x7fe0680b3460 2023-10-16T11:20:03.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.016+0000 7fe05e7fc700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 0x7fe04c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.016+0000 7fe06d142700 1 -- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 msgr2=0x7fe04c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:03.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.016+0000 7fe05e7fc700 1 -- 172.21.15.158:0/3389408361 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fe0640893b0 con 0x7fe0680b3460 2023-10-16T11:20:03.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.016+0000 7fe06d142700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 0x7fe04c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:03.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.020+0000 7fe05e7fc700 1 -- 172.21.15.158:0/3389408361 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe064053a70 con 0x7fe0680b3460 2023-10-16T11:20:03.132 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:03.131+0000 7f89884a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:20:03.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.217+0000 7fe06d142700 1 -- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 msgr2=0x7fe04c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:03.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.217+0000 7fe06d142700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 0x7fe04c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:03.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.218+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0680b3a30 con 0x7fe0680b3460 2023-10-16T11:20:03.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.218+0000 7fe05e7fc700 1 -- 172.21.15.158:0/3389408361 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v222) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe064057090 con 0x7fe0680b3460 2023-10-16T11:20:03.220 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:03.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 msgr2=0x7fe04c06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:03.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 0x7fe04c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 msgr2=0x7fe0680c71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680c71c0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c420 tx=0x7fe06400c730 comp rx=0 tx=0).stop 2023-10-16T11:20:03.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 shutdown_connections 2023-10-16T11:20:03.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7fe04c06b1d0 0x7fe04c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06815b6b0 0x7fe0681f07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0680b3460 0x7fe0680c71c0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 --2- 172.21.15.158:0/3389408361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0680d0c60 0x7fe068151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 >> 172.21.15.158:0/3389408361 conn(0x7fe0680cdf20 msgr2=0x7fe0680d1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:03.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 shutdown_connections 2023-10-16T11:20:03.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.221+0000 7fe06fba7700 1 -- 172.21.15.158:0/3389408361 wait complete. 2023-10-16T11:20:03.235 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T11:20:03.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.363+0000 7fad1e076700 1 Processor -- start 2023-10-16T11:20:03.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.363+0000 7fad1e076700 1 -- start start 2023-10-16T11:20:03.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad1e076700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad180d0c60 0x7fad180b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad1e076700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad180b3460 0x7fad180b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad1e076700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad180b4420 con 0x7fad1815b6b0 2023-10-16T11:20:03.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad1e076700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1815ba90 con 0x7fad180b3460 2023-10-16T11:20:03.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad1e076700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad180b38f0 con 0x7fad180d0c60 2023-10-16T11:20:03.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad17fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad180b3460 0x7fad180b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad177fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fad1815b6b0 0x7fad180cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39636/0 (socket says 172.21.15.158:39636) 2023-10-16T11:20:03.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad177fe700 1 -- 172.21.15.158:0/171962216 learned_addr learned my addr 172.21.15.158:0/171962216 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:03.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad16ffd700 1 --2- 172.21.15.158:0/171962216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad180d0c60 0x7fad180b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad167fc700 1 -- 172.21.15.158:0/171962216 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250797367 0 0) 0x7fad180b4420 con 0x7fad1815b6b0 2023-10-16T11:20:03.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad167fc700 1 -- 172.21.15.158:0/171962216 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad040035b0 con 0x7fad1815b6b0 2023-10-16T11:20:03.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad17fff700 1 -- 172.21.15.158:0/171962216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad180d0c60 msgr2=0x7fad180b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:03.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad17fff700 1 --2- 172.21.15.158:0/171962216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad180d0c60 0x7fad180b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad17fff700 1 -- 172.21.15.158:0/171962216 >> v1:172.21.15.158:6789/0 conn(0x7fad1815b6b0 legacy=0x7fad180cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.364+0000 7fad17fff700 1 -- 172.21.15.158:0/171962216 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad180cbc20 con 0x7fad180b3460 2023-10-16T11:20:03.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad17fff700 1 --2- 172.21.15.158:0/171962216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad180b48b0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fad10009840 tx=0x7fad10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9d78f8a46ed032 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:03.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad167fc700 1 -- 172.21.15.158:0/171962216 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10011830 con 0x7fad180b3460 2023-10-16T11:20:03.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad167fc700 1 -- 172.21.15.158:0/171962216 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad10011990 con 0x7fad180b3460 2023-10-16T11:20:03.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad167fc700 1 -- 172.21.15.158:0/171962216 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10011d20 con 0x7fad180b3460 2023-10-16T11:20:03.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 -- 172.21.15.158:0/171962216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 msgr2=0x7fad180b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 --2- 172.21.15.158:0/171962216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad180b48b0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fad10009840 tx=0x7fad10009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:03.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 -- 172.21.15.158:0/171962216 shutdown_connections 2023-10-16T11:20:03.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 --2- 172.21.15.158:0/171962216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad180b48b0 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 --2- 172.21.15.158:0/171962216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad180d0c60 0x7fad180b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 -- 172.21.15.158:0/171962216 >> 172.21.15.158:0/171962216 conn(0x7fad180cdf20 msgr2=0x7fad180ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 -- 172.21.15.158:0/171962216 shutdown_connections 2023-10-16T11:20:03.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.365+0000 7fad1e076700 1 -- 172.21.15.158:0/171962216 wait complete. 2023-10-16T11:20:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.367+0000 7fad1e076700 1 Processor -- start 2023-10-16T11:20:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.367+0000 7fad1e076700 1 -- start start 2023-10-16T11:20:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad1e076700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad181f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad1e076700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad180d0c60 0x7fad181f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad1e076700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad1815b6b0 0x7fad180ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad1e076700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad181f06f0 con 0x7fad180b3460 2023-10-16T11:20:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad1e076700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad181f3bf0 con 0x7fad180d0c60 2023-10-16T11:20:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad1e076700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad180ac890 con 0x7fad1815b6b0 2023-10-16T11:20:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad181f01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad181f01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39466/0 (socket says 172.21.15.158:39466) 2023-10-16T11:20:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 -- 172.21.15.158:0/1772971088 learned_addr learned my addr 172.21.15.158:0/1772971088 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad16ffd700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad180d0c60 0x7fad181f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 -- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad1815b6b0 msgr2=0x7fad180ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad1815b6b0 0x7fad180ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 -- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad180d0c60 msgr2=0x7fad181f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad180d0c60 0x7fad181f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 -- 172.21.15.158:0/1772971088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad100094a0 con 0x7fad180b3460 2023-10-16T11:20:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad16ffd700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad180d0c60 0x7fad181f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:03.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad177fe700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad181f01b0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fad0800cbc0 tx=0x7fad0800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0800abb0 con 0x7fad180b3460 2023-10-16T11:20:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0800ad10 con 0x7fad180b3460 2023-10-16T11:20:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad180acb30 con 0x7fad180b3460 2023-10-16T11:20:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.368+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08012620 con 0x7fad180b3460 2023-10-16T11:20:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.369+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad180ad0f0 con 0x7fad180b3460 2023-10-16T11:20:03.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.369+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad180681b0 con 0x7fad180b3460 2023-10-16T11:20:03.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.370+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fad080113a0 con 0x7fad180b3460 2023-10-16T11:20:03.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.370+0000 7fad0ffff700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7facf806b1d0 0x7facf806d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.370+0000 7fad16ffd700 1 -- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7facf806b1d0 msgr2=0x7facf806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1511764298 2023-10-16T11:20:03.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.370+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 7717+0+0 (secure 0 0 0) 0x7fad0808a1a0 con 0x7fad180b3460 2023-10-16T11:20:03.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.370+0000 7fad16ffd700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7facf806b1d0 0x7facf806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:03.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.378+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fad08054600 con 0x7fad180b3460 2023-10-16T11:20:03.416 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.415+0000 7fa523dc8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:20:03.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.434+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 144) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fad080549d0 con 0x7fad180b3460 2023-10-16T11:20:03.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.435+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7facf806b1d0 msgr2=0x7facf806d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:03.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.435+0000 7fad0ffff700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7facf806b1d0 0x7facf806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.503 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.503 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.503 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.504 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.504 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.504 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.504 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.504 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.505 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.505 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.502+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.512 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.513 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.513 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.513 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.513 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49f5b4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.514 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.514 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.514 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.514 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.514 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:03.511+0000 7fa49c5ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:03.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.594+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fad180cb5a0 con 0x7fad180b3460 2023-10-16T11:20:03.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.594+0000 7fad0ffff700 1 -- 172.21.15.158:0/1772971088 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v223) v1 ==== 103+0+11 (secure 0 0 0) 0x7fad08017020 con 0x7fad180b3460 2023-10-16T11:20:03.596 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:03.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.598+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 msgr2=0x7fad181f01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.598+0000 7fad1e076700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad181f01b0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fad0800cbc0 tx=0x7fad0800cf80 comp rx=0 tx=0).stop 2023-10-16T11:20:03.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 shutdown_connections 2023-10-16T11:20:03.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:6832/1511764298,v1:172.21.15.158:6833/1511764298] conn(0x7facf806b1d0 0x7facf806d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad1815b6b0 0x7fad180ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad180b3460 0x7fad181f01b0 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 --2- 172.21.15.158:0/1772971088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad180d0c60 0x7fad181f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 >> 172.21.15.158:0/1772971088 conn(0x7fad180cdf20 msgr2=0x7fad1806cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:03.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 shutdown_connections 2023-10-16T11:20:03.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.599+0000 7fad1e076700 1 -- 172.21.15.158:0/1772971088 wait complete. 2023-10-16T11:20:03.613 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:03.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.732+0000 7f6a0aa5d700 1 Processor -- start 2023-10-16T11:20:03.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.732+0000 7f6a0aa5d700 1 -- start start 2023-10-16T11:20:03.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.732+0000 7f6a0aa5d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6a0415daa0 0x7f6a0406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.732+0000 7f6a0aa5d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.732+0000 7f6a0aa5d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040ce4d0 con 0x7f6a040d0bd0 2023-10-16T11:20:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a0aa5d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a040d1c20 con 0x7f6a0415daa0 2023-10-16T11:20:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a0aa5d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040d3f00 con 0x7f6a040d3570 2023-10-16T11:20:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a037fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6a040d0bd0 0x7f6a040d1510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39754/0 (socket says 172.21.15.158:39754) 2023-10-16T11:20:03.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a037fe700 1 -- 172.21.15.158:0/2929566915 learned_addr learned my addr 172.21.15.158:0/2929566915 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a03fff700 1 --2- 172.21.15.158:0/2929566915 >> v2:172.21.15.172:3300/0 conn(0x7f6a0415daa0 0x7f6a0406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a08ffa700 1 --2- 172.21.15.158:0/2929566915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a02ffd700 1 -- 172.21.15.158:0/2929566915 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728653104 0 0) 0x7f6a040ce4d0 con 0x7f6a040d0bd0 2023-10-16T11:20:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a02ffd700 1 -- 172.21.15.158:0/2929566915 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec0035b0 con 0x7f6a040d0bd0 2023-10-16T11:20:03.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a08ffa700 1 -- 172.21.15.158:0/2929566915 >> v2:172.21.15.172:3300/0 conn(0x7f6a0415daa0 msgr2=0x7f6a0406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a08ffa700 1 --2- 172.21.15.158:0/2929566915 >> v2:172.21.15.172:3300/0 conn(0x7f6a0415daa0 0x7f6a0406b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a08ffa700 1 -- 172.21.15.158:0/2929566915 >> v1:172.21.15.158:6789/0 conn(0x7f6a040d0bd0 legacy=0x7f6a040d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a08ffa700 1 -- 172.21.15.158:0/2929566915 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0406c1b0 con 0x7f6a040d3570 2023-10-16T11:20:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a08ffa700 1 --2- 172.21.15.158:0/2929566915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040d39c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f69fc009d90 tx=0x7f69fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9a92d835d34d1db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:03.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a02ffd700 1 -- 172.21.15.158:0/2929566915 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc00e070 con 0x7f6a040d3570 2023-10-16T11:20:03.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a02ffd700 1 -- 172.21.15.158:0/2929566915 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc00bb20 con 0x7f6a040d3570 2023-10-16T11:20:03.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.733+0000 7f6a02ffd700 1 -- 172.21.15.158:0/2929566915 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc012430 con 0x7f6a040d3570 2023-10-16T11:20:03.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/2929566915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 msgr2=0x7f6a040d39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 --2- 172.21.15.158:0/2929566915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040d39c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f69fc009d90 tx=0x7f69fc00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:03.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/2929566915 shutdown_connections 2023-10-16T11:20:03.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 --2- 172.21.15.158:0/2929566915 >> v2:172.21.15.172:3300/0 conn(0x7f6a0415daa0 0x7f6a0406b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 --2- 172.21.15.158:0/2929566915 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040d39c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/2929566915 >> 172.21.15.158:0/2929566915 conn(0x7f6a0408d040 msgr2=0x7f6a0409c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:03.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/2929566915 shutdown_connections 2023-10-16T11:20:03.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.734+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/2929566915 wait complete. 2023-10-16T11:20:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.735+0000 7f6a0aa5d700 1 Processor -- start 2023-10-16T11:20:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.735+0000 7f6a0aa5d700 1 -- start start 2023-10-16T11:20:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a0aa5d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a040d0bd0 0x7f6a04152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a0aa5d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a0aa5d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a0415daa0 0x7f6a040cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a0aa5d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040cba90 con 0x7f6a0415daa0 2023-10-16T11:20:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a0aa5d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040c86f0 con 0x7f6a040d0bd0 2023-10-16T11:20:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a0aa5d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0415f060 con 0x7f6a040d3570 2023-10-16T11:20:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a08ffa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a0415daa0 0x7f6a040cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a08ffa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a0415daa0 0x7f6a040cc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39586/0 (socket says 172.21.15.158:39586) 2023-10-16T11:20:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a08ffa700 1 -- 172.21.15.158:0/3078779785 learned_addr learned my addr 172.21.15.158:0/3078779785 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a03fff700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a040d0bd0 0x7f6a04152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a037fe700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:03.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.736+0000 7f6a08ffa700 1 -- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 msgr2=0x7f6a040cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a08ffa700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a08ffa700 1 -- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a040d0bd0 msgr2=0x7f6a04152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a08ffa700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a040d0bd0 0x7f6a04152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a08ffa700 1 -- 172.21.15.158:0/3078779785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc009a70 con 0x7f6a0415daa0 2023-10-16T11:20:03.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a08ffa700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a0415daa0 0x7f6a040cc720 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f69fc014040 tx=0x7f69fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:03.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a00ff9700 1 -- 172.21.15.158:0/3078779785 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc012430 con 0x7f6a0415daa0 2023-10-16T11:20:03.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a00ff9700 1 -- 172.21.15.158:0/3078779785 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc020070 con 0x7f6a0415daa0 2023-10-16T11:20:03.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a00ff9700 1 -- 172.21.15.158:0/3078779785 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc004280 con 0x7f6a0415daa0 2023-10-16T11:20:03.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a040cb4d0 con 0x7f6a0415daa0 2023-10-16T11:20:03.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.737+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a040c8310 con 0x7f6a0415daa0 2023-10-16T11:20:03.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.738+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a040c8170 con 0x7f6a0415daa0 2023-10-16T11:20:03.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.738+0000 7f6a00ff9700 1 -- 172.21.15.158:0/3078779785 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 144) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f69fc0043e0 con 0x7f6a0415daa0 2023-10-16T11:20:03.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.739+0000 7f6a00ff9700 1 -- 172.21.15.158:0/3078779785 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f69fc022070 con 0x7f6a0415daa0 2023-10-16T11:20:03.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.743+0000 7f6a00ff9700 1 -- 172.21.15.158:0/3078779785 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a040c8170 con 0x7f6a0415daa0 2023-10-16T11:20:03.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.981+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a040c8830 con 0x7f6a0415daa0 2023-10-16T11:20:03.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.982+0000 7f6a00ff9700 1 -- 172.21.15.158:0/3078779785 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6163 (secure 0 0 0) 0x7f6a040c8830 con 0x7f6a0415daa0 2023-10-16T11:20:03.984 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:03.984 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:20:03.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a0415daa0 msgr2=0x7f6a040cc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:03.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a0415daa0 0x7f6a040cc720 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f69fc014040 tx=0x7f69fc004ad0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 shutdown_connections 2023-10-16T11:20:03.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a0415daa0 0x7f6a040cc720 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a040d0bd0 0x7f6a04152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 --2- 172.21.15.158:0/3078779785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a040d3570 0x7f6a040cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:03.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 >> 172.21.15.158:0/3078779785 conn(0x7f6a0408d040 msgr2=0x7f6a040b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:03.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 shutdown_connections 2023-10-16T11:20:03.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:03.984+0000 7f6a0aa5d700 1 -- 172.21.15.158:0/3078779785 wait complete. 2023-10-16T11:20:03.988 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 118 2023-10-16T11:20:03.998 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:04.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.120+0000 7fac74966700 1 Processor -- start 2023-10-16T11:20:04.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.120+0000 7fac74966700 1 -- start start 2023-10-16T11:20:04.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac74966700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fac7006e5d0 0x7fac7006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac74966700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70161270 con 0x7fac700729b0 2023-10-16T11:20:04.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac74966700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70161510 con 0x7fac7006ab40 2023-10-16T11:20:04.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac74966700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac701617f0 con 0x7fac7006e5d0 2023-10-16T11:20:04.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6ed9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fac700729b0 0x7fac7015df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39768/0 (socket says 172.21.15.158:39768) 2023-10-16T11:20:04.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6ed9d700 1 -- 172.21.15.158:0/3183024378 learned_addr learned my addr 172.21.15.158:0/3183024378 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:04.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6dd9b700 1 --2- 172.21.15.158:0/3183024378 >> v2:172.21.15.172:3300/0 conn(0x7fac7006e5d0 0x7fac7006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682202599 0 0) 0x7fac70161270 con 0x7fac700729b0 2023-10-16T11:20:04.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac580035b0 con 0x7fac700729b0 2023-10-16T11:20:04.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1675230197 0 0) 0x7fac580035b0 con 0x7fac700729b0 2023-10-16T11:20:04.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 >> v2:172.21.15.172:3300/0 conn(0x7fac7006e5d0 msgr2=0x7fac7006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 --2- 172.21.15.158:0/3183024378 >> v2:172.21.15.172:3300/0 conn(0x7fac7006e5d0 0x7fac7006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 >> v1:172.21.15.172:6789/0 conn(0x7fac7006ab40 legacy=0x7fac700722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70161ad0 con 0x7fac700729b0 2023-10-16T11:20:04.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fac680019f0 con 0x7fac700729b0 2023-10-16T11:20:04.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 >> v1:172.21.15.158:6789/0 conn(0x7fac700729b0 legacy=0x7fac7015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac58003a00 0x7fac58003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac58004400 0x7fac580048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac580050d0 0x7fac5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac701617f0 con 0x7fac58003a00 2023-10-16T11:20:04.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5800c030 con 0x7fac58004400 2023-10-16T11:20:04.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5800c1f0 con 0x7fac580050d0 2023-10-16T11:20:04.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.121+0000 7fac6e59c700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac58003a00 0x7fac58003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6dd9b700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac58004400 0x7fac580048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6ed9d700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac580050d0 0x7fac5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6e59c700 1 -- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac580050d0 msgr2=0x7fac5800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6e59c700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac580050d0 0x7fac5800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6e59c700 1 -- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac58004400 msgr2=0x7fac580048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6e59c700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac58004400 0x7fac580048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6e59c700 1 -- 172.21.15.158:0/3183024378 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580035b0 con 0x7fac58003a00 2023-10-16T11:20:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6ed9d700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac580050d0 0x7fac5800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6ed9d700 1 -- 172.21.15.158:0/3183024378 reap_dead start 2023-10-16T11:20:04.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6e59c700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac58003a00 0x7fac58003de0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fac600018b0 tx=0x7fac58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c46dc44ef56be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60001370 con 0x7fac58003a00 2023-10-16T11:20:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac600052f0 con 0x7fac58003a00 2023-10-16T11:20:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac6d59a700 1 -- 172.21.15.158:0/3183024378 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60003680 con 0x7fac58003a00 2023-10-16T11:20:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac74966700 1 -- 172.21.15.158:0/3183024378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac58003a00 msgr2=0x7fac58003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.122+0000 7fac74966700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac58003a00 0x7fac58003de0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fac600018b0 tx=0x7fac58005880 comp rx=0 tx=0).stop 2023-10-16T11:20:04.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.123+0000 7fac74966700 1 -- 172.21.15.158:0/3183024378 shutdown_connections 2023-10-16T11:20:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.123+0000 7fac74966700 1 --2- 172.21.15.158:0/3183024378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac58003a00 0x7fac58003de0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.123+0000 7fac74966700 1 -- 172.21.15.158:0/3183024378 >> 172.21.15.158:0/3183024378 conn(0x7fac700d02b0 msgr2=0x7fac700d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.123+0000 7fac74966700 1 -- 172.21.15.158:0/3183024378 shutdown_connections 2023-10-16T11:20:04.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.123+0000 7fac74966700 1 -- 172.21.15.158:0/3183024378 wait complete. 2023-10-16T11:20:04.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.124+0000 7fac74966700 1 Processor -- start 2023-10-16T11:20:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.124+0000 7fac74966700 1 -- start start 2023-10-16T11:20:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac74966700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac58003a00 0x7fac701f5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac74966700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac70148740 0x7fac701f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac74966700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac700ae830 0x7fac700aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac74966700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac700af220 con 0x7fac700ae830 2023-10-16T11:20:04.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac74966700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac701f3790 con 0x7fac70148740 2023-10-16T11:20:04.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac74966700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac700b1450 con 0x7fac58003a00 2023-10-16T11:20:04.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac6ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac700ae830 0x7fac700aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac6ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac700ae830 0x7fac700aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39604/0 (socket says 172.21.15.158:39604) 2023-10-16T11:20:04.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.125+0000 7fac6ed9d700 1 -- 172.21.15.158:0/3540111022 learned_addr learned my addr 172.21.15.158:0/3540111022 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:04.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6e59c700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac58003a00 0x7fac701f5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6dd9b700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac70148740 0x7fac701f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6ed9d700 1 -- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac58003a00 msgr2=0x7fac701f5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6ed9d700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac58003a00 0x7fac701f5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6ed9d700 1 -- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac70148740 msgr2=0x7fac701f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6ed9d700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac70148740 0x7fac701f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6ed9d700 1 -- 172.21.15.158:0/3540111022 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac701f6ce0 con 0x7fac700ae830 2023-10-16T11:20:04.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6e59c700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac58003a00 0x7fac701f5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:04.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac6ed9d700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac700ae830 0x7fac700aece0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fac70072db0 tx=0x7fac68007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac5effd700 1 -- 172.21.15.158:0/3540111022 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68017070 con 0x7fac700ae830 2023-10-16T11:20:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac5effd700 1 -- 172.21.15.158:0/3540111022 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6800ee50 con 0x7fac700ae830 2023-10-16T11:20:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac70148bc0 con 0x7fac700ae830 2023-10-16T11:20:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac5effd700 1 -- 172.21.15.158:0/3540111022 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68013420 con 0x7fac700ae830 2023-10-16T11:20:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.126+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac701f6720 con 0x7fac700ae830 2023-10-16T11:20:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.127+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac700681b0 con 0x7fac700ae830 2023-10-16T11:20:04.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.127+0000 7fac5effd700 1 -- 172.21.15.158:0/3540111022 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 144) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fac68013580 con 0x7fac700ae830 2023-10-16T11:20:04.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.128+0000 7fac5effd700 1 -- 172.21.15.158:0/3540111022 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fac68089880 con 0x7fac700ae830 2023-10-16T11:20:04.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.132+0000 7fac5effd700 1 -- 172.21.15.158:0/3540111022 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fac68054330 con 0x7fac700ae830 2023-10-16T11:20:04.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.330+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac700b6230 con 0x7fac700ae830 2023-10-16T11:20:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.332+0000 7fac5effd700 1 -- 172.21.15.158:0/3540111022 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+22388 (secure 0 0 0) 0x7fac68057950 con 0x7fac700ae830 2023-10-16T11:20:04.334 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:04.334 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":223,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:03.428922+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/931576556":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2043584975":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/139807143":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/572486792":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1251861851":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6832/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2783528774":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1862821458":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/417671612":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/534116803":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac700ae830 msgr2=0x7fac700aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac700ae830 0x7fac700aece0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fac70072db0 tx=0x7fac68007c90 comp rx=0 tx=0).stop 2023-10-16T11:20:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 shutdown_connections 2023-10-16T11:20:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac700ae830 0x7fac700aece0 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac58003a00 0x7fac701f5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 --2- 172.21.15.158:0/3540111022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac70148740 0x7fac701f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 >> 172.21.15.158:0/3540111022 conn(0x7fac700d02b0 msgr2=0x7fac700ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.334+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 shutdown_connections 2023-10-16T11:20:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.335+0000 7fac74966700 1 -- 172.21.15.158:0/3540111022 wait complete. 2023-10-16T11:20:04.348 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:20:04.349 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:04.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.474+0000 7fda8c091700 1 Processor -- start 2023-10-16T11:20:04.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.474+0000 7fda8c091700 1 -- start start 2023-10-16T11:20:04.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.475+0000 7fda8c091700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 0x7fda8406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.475+0000 7fda8c091700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda840729b0 0x7fda84071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.475+0000 7fda8c091700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda84071e50 con 0x7fda8406ab40 2023-10-16T11:20:04.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.475+0000 7fda8c091700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda84161270 con 0x7fda840729b0 2023-10-16T11:20:04.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.475+0000 7fda8c091700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda84161510 con 0x7fda8406e5d0 2023-10-16T11:20:04.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.475+0000 7fda89e2d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fda8406ab40 0x7fda840722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39804/0 (socket says 172.21.15.158:39804) 2023-10-16T11:20:04.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.475+0000 7fda89e2d700 1 -- 172.21.15.158:0/1329652732 learned_addr learned my addr 172.21.15.158:0/1329652732 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:04.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979976454 0 0) 0x7fda84071e50 con 0x7fda8406ab40 2023-10-16T11:20:04.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8a62e700 1 --2- 172.21.15.158:0/1329652732 >> v2:172.21.15.172:3300/0 conn(0x7fda840729b0 0x7fda84071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6c0035b0 con 0x7fda8406ab40 2023-10-16T11:20:04.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8962c700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 0x7fda8406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 530377667 0 0) 0x7fda6c0035b0 con 0x7fda8406ab40 2023-10-16T11:20:04.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 msgr2=0x7fda8406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 0x7fda8406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 >> v2:172.21.15.172:3300/0 conn(0x7fda840729b0 msgr2=0x7fda84071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 --2- 172.21.15.158:0/1329652732 >> v2:172.21.15.172:3300/0 conn(0x7fda840729b0 0x7fda84071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda841617f0 con 0x7fda8406ab40 2023-10-16T11:20:04.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fda7c001670 con 0x7fda8406ab40 2023-10-16T11:20:04.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 >> v1:172.21.15.158:6789/0 conn(0x7fda8406ab40 legacy=0x7fda840722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 0x7fda6c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda6c003f80 0x7fda6c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84161270 con 0x7fda6c004d90 2023-10-16T11:20:04.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84161510 con 0x7fda6c003f80 2023-10-16T11:20:04.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00ba60 con 0x7fda8406e5d0 2023-10-16T11:20:04.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8a62e700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8962c700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda6c003f80 0x7fda6c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda89e2d700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 0x7fda6c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8a62e700 1 -- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 msgr2=0x7fda6c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8a62e700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda8406e5d0 0x7fda6c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8a62e700 1 -- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda6c003f80 msgr2=0x7fda6c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8a62e700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda6c003f80 0x7fda6c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.476+0000 7fda8a62e700 1 -- 172.21.15.158:0/1329652732 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0035b0 con 0x7fda6c004d90 2023-10-16T11:20:04.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8a62e700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda6c005210 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fda74009f90 tx=0x7fda6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f607390350471403 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7400ec30 con 0x7fda6c004d90 2023-10-16T11:20:04.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7400f040 con 0x7fda6c004d90 2023-10-16T11:20:04.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda88e2b700 1 -- 172.21.15.158:0/1329652732 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74014480 con 0x7fda6c004d90 2023-10-16T11:20:04.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8c091700 1 -- 172.21.15.158:0/1329652732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 msgr2=0x7fda6c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8c091700 1 --2- 172.21.15.158:0/1329652732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda6c005210 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fda74009f90 tx=0x7fda6c005880 comp rx=0 tx=0).stop 2023-10-16T11:20:04.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8a62e700 1 -- 172.21.15.158:0/1329652732 reap_dead start 2023-10-16T11:20:04.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8c091700 1 -- 172.21.15.158:0/1329652732 shutdown_connections 2023-10-16T11:20:04.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8c091700 1 -- 172.21.15.158:0/1329652732 >> 172.21.15.158:0/1329652732 conn(0x7fda840d02b0 msgr2=0x7fda840d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:04.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8c091700 1 -- 172.21.15.158:0/1329652732 shutdown_connections 2023-10-16T11:20:04.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.477+0000 7fda8c091700 1 -- 172.21.15.158:0/1329652732 wait complete. 2023-10-16T11:20:04.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 Processor -- start 2023-10-16T11:20:04.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 -- start start 2023-10-16T11:20:04.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda840c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda84155b30 0x7fda840c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda840c78c0 0x7fda841f5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda840c7d50 con 0x7fda6c004d90 2023-10-16T11:20:04.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84156380 con 0x7fda840c78c0 2023-10-16T11:20:04.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda8c091700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda841f6940 con 0x7fda84155b30 2023-10-16T11:20:04.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda89e2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda840c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda89e2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda840c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39640/0 (socket says 172.21.15.158:39640) 2023-10-16T11:20:04.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.479+0000 7fda89e2d700 1 -- 172.21.15.158:0/2608679065 learned_addr learned my addr 172.21.15.158:0/2608679065 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:04.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda8962c700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda84155b30 0x7fda840c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda8a62e700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda840c78c0 0x7fda841f5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda89e2d700 1 -- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda84155b30 msgr2=0x7fda840c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda89e2d700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda84155b30 0x7fda840c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda89e2d700 1 -- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda840c78c0 msgr2=0x7fda841f5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda89e2d700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda840c78c0 0x7fda841f5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda89e2d700 1 -- 172.21.15.158:0/2608679065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840c8b90 con 0x7fda6c004d90 2023-10-16T11:20:04.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda89e2d700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda840c9670 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fda7c001aa0 tx=0x7fda7c001760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda827fc700 1 -- 172.21.15.158:0/2608679065 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c00f660 con 0x7fda6c004d90 2023-10-16T11:20:04.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda827fc700 1 -- 172.21.15.158:0/2608679065 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c00f7c0 con 0x7fda6c004d90 2023-10-16T11:20:04.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda827fc700 1 -- 172.21.15.158:0/2608679065 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c00fb70 con 0x7fda6c004d90 2023-10-16T11:20:04.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda840c8e70 con 0x7fda6c004d90 2023-10-16T11:20:04.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.480+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda841f6c20 con 0x7fda6c004d90 2023-10-16T11:20:04.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.481+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda840681b0 con 0x7fda6c004d90 2023-10-16T11:20:04.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.481+0000 7fda827fc700 1 -- 172.21.15.158:0/2608679065 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fda7c00fcd0 con 0x7fda6c004d90 2023-10-16T11:20:04.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.482+0000 7fda827fc700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fda7006ad80 0x7fda7006d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.482+0000 7fda8962c700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fda7006ad80 0x7fda7006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.482+0000 7fda827fc700 1 -- 172.21.15.158:0/2608679065 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fda7c089c70 con 0x7fda6c004d90 2023-10-16T11:20:04.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.482+0000 7fda8962c700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fda7006ad80 0x7fda7006d240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda840c3c10 tx=0x7fda78008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.486+0000 7fda827fc700 1 -- 172.21.15.158:0/2608679065 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda7c054610 con 0x7fda6c004d90 2023-10-16T11:20:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.722+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda840b6230 con 0x7fda6c004d90 2023-10-16T11:20:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.722+0000 7fda827fc700 1 -- 172.21.15.158:0/2608679065 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6163 (secure 0 0 0) 0x7fda7c057c30 con 0x7fda6c004d90 2023-10-16T11:20:04.724 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:04.725 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:20:04.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fda7006ad80 msgr2=0x7fda7006d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fda7006ad80 0x7fda7006d240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda840c3c10 tx=0x7fda78008040 comp rx=0 tx=0).stop 2023-10-16T11:20:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 msgr2=0x7fda840c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda840c9670 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fda7c001aa0 tx=0x7fda7c001760 comp rx=0 tx=0).stop 2023-10-16T11:20:04.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 shutdown_connections 2023-10-16T11:20:04.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fda7006ad80 0x7fda7006d240 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda840c78c0 0x7fda841f5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c004d90 0x7fda840c9670 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 --2- 172.21.15.158:0/2608679065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda84155b30 0x7fda840c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.724+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 >> 172.21.15.158:0/2608679065 conn(0x7fda840d02b0 msgr2=0x7fda840b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.725+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 shutdown_connections 2023-10-16T11:20:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.725+0000 7fda8c091700 1 -- 172.21.15.158:0/2608679065 wait complete. 2023-10-16T11:20:04.729 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 118 2023-10-16T11:20:04.738 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:04.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.857+0000 7ff3c28d0700 1 Processor -- start 2023-10-16T11:20:04.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.857+0000 7ff3c28d0700 1 -- start start 2023-10-16T11:20:04.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c28d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0cdee0 0x7ff3bc15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c28d0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3bc0ce2c0 0x7ff3bc0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c28d0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc15c5d0 con 0x7ff3bc0b6e10 2023-10-16T11:20:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c28d0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3bc15d2c0 con 0x7ff3bc0ce2c0 2023-10-16T11:20:04.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c28d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc15c8b0 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c0e6d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3bc0ce2c0 0x7ff3bc0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3bb7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff3bc0b6e10 0x7ff3bc1521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39828/0 (socket says 172.21.15.158:39828) 2023-10-16T11:20:04.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3bb7fe700 1 -- 172.21.15.158:0/3984504813 learned_addr learned my addr 172.21.15.158:0/3984504813 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:04.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c0e6d700 1 -- 172.21.15.158:0/3984504813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0cdee0 msgr2=0x7ff3bc15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:04.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c0e6d700 1 --2- 172.21.15.158:0/3984504813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0cdee0 0x7ff3bc15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c0e6d700 1 -- 172.21.15.158:0/3984504813 >> v1:172.21.15.158:6789/0 conn(0x7ff3bc0b6e10 legacy=0x7ff3bc1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3c0e6d700 1 -- 172.21.15.158:0/3984504813 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc15cb90 con 0x7ff3bc0ce2c0 2023-10-16T11:20:04.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.858+0000 7ff3baffd700 1 -- 172.21.15.158:0/3984504813 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893305249 0 0) 0x7ff3bc15c5d0 con 0x7ff3bc0b6e10 2023-10-16T11:20:04.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c0e6d700 1 --2- 172.21.15.158:0/3984504813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc0cba20 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009d90 tx=0x7ff3b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80c11682068efcf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3baffd700 1 -- 172.21.15.158:0/3984504813 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b400e070 con 0x7ff3bc0ce2c0 2023-10-16T11:20:04.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3baffd700 1 -- 172.21.15.158:0/3984504813 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b400bbd0 con 0x7ff3bc0ce2c0 2023-10-16T11:20:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3baffd700 1 -- 172.21.15.158:0/3984504813 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4012430 con 0x7ff3bc0ce2c0 2023-10-16T11:20:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 -- 172.21.15.158:0/3984504813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0ce2c0 msgr2=0x7ff3bc0cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/3984504813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc0cba20 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009d90 tx=0x7ff3b400b330 comp rx=0 tx=0).stop 2023-10-16T11:20:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 -- 172.21.15.158:0/3984504813 shutdown_connections 2023-10-16T11:20:04.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/3984504813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc0cba20 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/3984504813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0cdee0 0x7ff3bc15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 -- 172.21.15.158:0/3984504813 >> 172.21.15.158:0/3984504813 conn(0x7ff3bc06a8d0 msgr2=0x7ff3bc06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 -- 172.21.15.158:0/3984504813 shutdown_connections 2023-10-16T11:20:04.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.859+0000 7ff3c28d0700 1 -- 172.21.15.158:0/3984504813 wait complete. 2023-10-16T11:20:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.861+0000 7ff3c28d0700 1 Processor -- start 2023-10-16T11:20:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.861+0000 7ff3c28d0700 1 -- start start 2023-10-16T11:20:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3c28d0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0b6e10 0x7ff3bc0d0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3c28d0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0cdee0 0x7ff3bc146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3c28d0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3c28d0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc0d00c0 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3c28d0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc146970 con 0x7ff3bc0ce2c0 2023-10-16T11:20:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3c28d0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc0cf4b0 con 0x7ff3bc0b6e10 2023-10-16T11:20:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0cdee0 0x7ff3bc146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0cdee0 0x7ff3bc146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39656/0 (socket says 172.21.15.158:39656) 2023-10-16T11:20:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 -- 172.21.15.158:0/4229862135 learned_addr learned my addr 172.21.15.158:0/4229862135 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3c0e6d700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 -- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0b6e10 msgr2=0x7ff3bc0d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bbfff700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0b6e10 0x7ff3bc0d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0b6e10 0x7ff3bc0d0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 -- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3bc0ce2c0 msgr2=0x7ff3bc153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:04.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc153920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:04.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.862+0000 7ff3bb7fe700 1 -- 172.21.15.158:0/4229862135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4009a70 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.863+0000 7ff3c0e6d700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:04.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.863+0000 7ff3bb7fe700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0cdee0 0x7ff3bc146430 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0011e0 tx=0x7ff3ac00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.863+0000 7ff3b8ff9700 1 -- 172.21.15.158:0/4229862135 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac004c80 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.863+0000 7ff3b8ff9700 1 -- 172.21.15.158:0/4229862135 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac004e00 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.863+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc0cf790 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.863+0000 7ff3b8ff9700 1 -- 172.21.15.158:0/4229862135 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac007310 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.863+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3bc0cfd50 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.864+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3bc0d0230 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.864+0000 7ff3b8ff9700 1 -- 172.21.15.158:0/4229862135 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff3ac007470 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.865+0000 7ff3b8ff9700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff3a806ad50 0x7ff3a806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:04.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.865+0000 7ff3b8ff9700 1 -- 172.21.15.158:0/4229862135 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 8190+0+0 (secure 0 0 0) 0x7ff3ac089a60 con 0x7ff3bc0cdee0 2023-10-16T11:20:04.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.865+0000 7ff3bbfff700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff3a806ad50 0x7ff3a806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:04.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.865+0000 7ff3bbfff700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff3a806ad50 0x7ff3a806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0006fd0 tx=0x7ff3b0008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:04.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:04.869+0000 7ff3b8ff9700 1 -- 172.21.15.158:0/4229862135 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3ac054400 con 0x7ff3bc0cdee0 2023-10-16T11:20:05.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.110+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3bc0681b0 con 0x7ff3bc0cdee0 2023-10-16T11:20:05.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.110+0000 7ff3b8ff9700 1 -- 172.21.15.158:0/4229862135 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6163 (secure 0 0 0) 0x7ff3ac057a20 con 0x7ff3bc0cdee0 2023-10-16T11:20:05.112 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:05.112 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:22.724992+0000","modified":"2023-10-16T11:19:25.742039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38819":{"gid":38819,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/972449219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":972449219},{"type":"v1","addr":"172.21.15.172:6834","nonce":972449219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:20:05.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff3a806ad50 msgr2=0x7ff3a806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff3a806ad50 0x7ff3a806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0006fd0 tx=0x7ff3b0008040 comp rx=0 tx=0).stop 2023-10-16T11:20:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0cdee0 msgr2=0x7ff3bc146430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0cdee0 0x7ff3bc146430 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0011e0 tx=0x7ff3ac00cae0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 shutdown_connections 2023-10-16T11:20:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff3a806ad50 0x7ff3a806d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3bc0ce2c0 0x7ff3bc153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3bc0b6e10 0x7ff3bc0d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 --2- 172.21.15.158:0/4229862135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3bc0cdee0 0x7ff3bc146430 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 >> 172.21.15.158:0/4229862135 conn(0x7ff3bc06a8d0 msgr2=0x7ff3bc15f100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 shutdown_connections 2023-10-16T11:20:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.112+0000 7ff3c28d0700 1 -- 172.21.15.158:0/4229862135 wait complete. 2023-10-16T11:20:05.117 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 118 2023-10-16T11:20:05.126 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:05.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.245+0000 7f5a13e91700 1 Processor -- start 2023-10-16T11:20:05.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.245+0000 7f5a13e91700 1 -- start start 2023-10-16T11:20:05.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a13e91700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5a0c0cc5d0 0x7f5a0c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a13e91700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5a0c15dfb0 0x7f5a0c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a13e91700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c06e770 con 0x7f5a0c06f340 2023-10-16T11:20:05.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a13e91700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c0cf270 con 0x7f5a0c15dfb0 2023-10-16T11:20:05.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a13e91700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c15e4e0 con 0x7f5a0c0cc5d0 2023-10-16T11:20:05.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a1142c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5a0c15dfb0 0x7f5a0c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a1142c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5a0c15dfb0 0x7f5a0c0cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39662/0 (socket says 172.21.15.158:39662) 2023-10-16T11:20:05.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a1142c700 1 -- 172.21.15.158:0/1594340200 learned_addr learned my addr 172.21.15.158:0/1594340200 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:05.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a11c2d700 1 --2- 172.21.15.158:0/1594340200 >> v2:172.21.15.172:3300/0 conn(0x7f5a0c0cc5d0 0x7f5a0c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a1142c700 1 -- 172.21.15.158:0/1594340200 >> v2:172.21.15.172:3300/0 conn(0x7f5a0c0cc5d0 msgr2=0x7f5a0c06e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:05.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a1142c700 1 --2- 172.21.15.158:0/1594340200 >> v2:172.21.15.172:3300/0 conn(0x7f5a0c0cc5d0 0x7f5a0c06e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.246+0000 7f5a1142c700 1 -- 172.21.15.158:0/1594340200 >> v1:172.21.15.172:6789/0 conn(0x7f5a0c06f340 legacy=0x7f5a0c071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a1142c700 1 -- 172.21.15.158:0/1594340200 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c15e7c0 con 0x7f5a0c15dfb0 2023-10-16T11:20:05.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a1142c700 1 --2- 172.21.15.158:0/1594340200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c15dfb0 0x7f5a0c0cb6f0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0d1390 tx=0x7f5a08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5694527eed4d08c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:05.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a10c2b700 1 -- 172.21.15.158:0/1594340200 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011830 con 0x7f5a0c15dfb0 2023-10-16T11:20:05.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a10c2b700 1 -- 172.21.15.158:0/1594340200 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08011990 con 0x7f5a0c15dfb0 2023-10-16T11:20:05.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a10c2b700 1 -- 172.21.15.158:0/1594340200 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011d20 con 0x7f5a0c15dfb0 2023-10-16T11:20:05.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 -- 172.21.15.158:0/1594340200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c15dfb0 msgr2=0x7f5a0c0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 --2- 172.21.15.158:0/1594340200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c15dfb0 0x7f5a0c0cb6f0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0d1390 tx=0x7f5a08009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 -- 172.21.15.158:0/1594340200 shutdown_connections 2023-10-16T11:20:05.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 --2- 172.21.15.158:0/1594340200 >> v2:172.21.15.172:3300/0 conn(0x7f5a0c0cc5d0 0x7f5a0c06e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 --2- 172.21.15.158:0/1594340200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c15dfb0 0x7f5a0c0cb6f0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 -- 172.21.15.158:0/1594340200 >> 172.21.15.158:0/1594340200 conn(0x7f5a0c0d02b0 msgr2=0x7f5a0c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:05.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 -- 172.21.15.158:0/1594340200 shutdown_connections 2023-10-16T11:20:05.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.247+0000 7f5a13e91700 1 -- 172.21.15.158:0/1594340200 wait complete. 2023-10-16T11:20:05.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 Processor -- start 2023-10-16T11:20:05.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 -- start start 2023-10-16T11:20:05.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c06f340 0x7f5a0c0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c0cc5d0 0x7f5a0c149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a0c15dfb0 0x7f5a0c14bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0c7380 con 0x7f5a0c06f340 2023-10-16T11:20:05.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0c6a90 con 0x7f5a0c0cc5d0 2023-10-16T11:20:05.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a13e91700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0c75d0 con 0x7f5a0c15dfb0 2023-10-16T11:20:05.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.249+0000 7f5a11c2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c06f340 0x7f5a0c0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c06f340 0x7f5a0c0c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39668/0 (socket says 172.21.15.158:39668) 2023-10-16T11:20:05.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 -- 172.21.15.158:0/371658279 learned_addr learned my addr 172.21.15.158:0/371658279 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:05.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a1142c700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c0cc5d0 0x7f5a0c149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a1242e700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a0c15dfb0 0x7f5a0c14bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 -- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a0c15dfb0 msgr2=0x7f5a0c14bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a0c15dfb0 0x7f5a0c14bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 -- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c0cc5d0 msgr2=0x7f5a0c149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c0cc5d0 0x7f5a0c149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 -- 172.21.15.158:0/371658279 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a08009550 con 0x7f5a0c06f340 2023-10-16T11:20:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a1142c700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c0cc5d0 0x7f5a0c149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a11c2d700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c06f340 0x7f5a0c0c6550 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e870 tx=0x7f59fc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a027fc700 1 -- 172.21.15.158:0/371658279 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc009590 con 0x7f5a0c06f340 2023-10-16T11:20:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a027fc700 1 -- 172.21.15.158:0/371658279 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc00ee70 con 0x7f5a0c06f340 2023-10-16T11:20:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c0c78b0 con 0x7f5a0c06f340 2023-10-16T11:20:05.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a027fc700 1 -- 172.21.15.158:0/371658279 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc00ca50 con 0x7f5a0c06f340 2023-10-16T11:20:05.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.250+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c0c7e70 con 0x7f5a0c06f340 2023-10-16T11:20:05.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.252+0000 7f5a027fc700 1 -- 172.21.15.158:0/371658279 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f59fc00cbb0 con 0x7f5a0c06f340 2023-10-16T11:20:05.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.252+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c0681b0 con 0x7f5a0c06f340 2023-10-16T11:20:05.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.252+0000 7f5a027fc700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f59f006ad50 0x7f59f006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.252+0000 7f5a027fc700 1 -- 172.21.15.158:0/371658279 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f59fc089c20 con 0x7f5a0c06f340 2023-10-16T11:20:05.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.252+0000 7f5a1142c700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f59f006ad50 0x7f59f006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.252+0000 7f5a1142c700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f59f006ad50 0x7f59f006d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c15e410 tx=0x7f5a08008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:05.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.256+0000 7f5a027fc700 1 -- 172.21.15.158:0/371658279 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59fc0545c0 con 0x7f5a0c06f340 2023-10-16T11:20:05.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.450+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a0c0b6230 con 0x7f5a0c06f340 2023-10-16T11:20:05.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.451+0000 7f5a027fc700 1 -- 172.21.15.158:0/371658279 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+22388 (secure 0 0 0) 0x7f59fc021020 con 0x7f5a0c06f340 2023-10-16T11:20:05.453 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:05.454 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":223,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:03.428922+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:19:21.151561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T11:19:21.679054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/931576556":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2043584975":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/139807143":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/572486792":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1251861851":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6832/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2783528774":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1862821458":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/417671612":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/534116803":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:05.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f59f006ad50 msgr2=0x7f59f006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f59f006ad50 0x7f59f006d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c15e410 tx=0x7f5a08008030 comp rx=0 tx=0).stop 2023-10-16T11:20:05.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c06f340 msgr2=0x7f5a0c0c6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c06f340 0x7f5a0c0c6550 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e870 tx=0x7f59fc00ec30 comp rx=0 tx=0).stop 2023-10-16T11:20:05.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 shutdown_connections 2023-10-16T11:20:05.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f59f006ad50 0x7f59f006d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a0c15dfb0 0x7f5a0c14bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a0c06f340 0x7f5a0c0c6550 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 --2- 172.21.15.158:0/371658279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c0cc5d0 0x7f5a0c149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 >> 172.21.15.158:0/371658279 conn(0x7f5a0c0d02b0 msgr2=0x7f5a0c1590c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:05.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 shutdown_connections 2023-10-16T11:20:05.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.454+0000 7f5a13e91700 1 -- 172.21.15.158:0/371658279 wait complete. 2023-10-16T11:20:05.468 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:20:05.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.592+0000 7f45d8695700 1 Processor -- start 2023-10-16T11:20:05.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.592+0000 7f45d8695700 1 -- start start 2023-10-16T11:20:05.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.592+0000 7f45d8695700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f45d00caaa0 0x7f45d00cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d8695700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00af510 con 0x7f45d006bbc0 2023-10-16T11:20:05.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d8695700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d006a8d0 con 0x7f45d006d740 2023-10-16T11:20:05.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d8695700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d00cb3c0 con 0x7f45d00caaa0 2023-10-16T11:20:05.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d6431700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f45d00caaa0 0x7f45d00cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d5c30700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f45d006bbc0 0x7f45d00cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39870/0 (socket says 172.21.15.158:39870) 2023-10-16T11:20:05.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d5c30700 1 -- 172.21.15.158:0/3819781205 learned_addr learned my addr 172.21.15.158:0/3819781205 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:05.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d6431700 1 -- 172.21.15.158:0/3819781205 >> v1:172.21.15.172:6789/0 conn(0x7f45d006d740 legacy=0x7f45d006db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d6431700 1 -- 172.21.15.158:0/3819781205 >> v1:172.21.15.158:6789/0 conn(0x7f45d006bbc0 legacy=0x7f45d00cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d6431700 1 -- 172.21.15.158:0/3819781205 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d006e2a0 con 0x7f45d00caaa0 2023-10-16T11:20:05.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d542f700 1 -- 172.21.15.158:0/3819781205 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021425774 0 0) 0x7f45cc001110 con 0x7f45d006bbc0 2023-10-16T11:20:05.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d6431700 1 --2- 172.21.15.158:0/3819781205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d00caaa0 0x7f45d00cae80 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009840 tx=0x7f45c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f72a220a04fec4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:05.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d542f700 1 -- 172.21.15.158:0/3819781205 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0011830 con 0x7f45d00caaa0 2023-10-16T11:20:05.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d542f700 1 -- 172.21.15.158:0/3819781205 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c0011990 con 0x7f45d00caaa0 2023-10-16T11:20:05.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.593+0000 7f45d542f700 1 -- 172.21.15.158:0/3819781205 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0011d20 con 0x7f45d00caaa0 2023-10-16T11:20:05.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.594+0000 7f45d8695700 1 -- 172.21.15.158:0/3819781205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d00caaa0 msgr2=0x7f45d00cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.594+0000 7f45d8695700 1 --2- 172.21.15.158:0/3819781205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d00caaa0 0x7f45d00cae80 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009840 tx=0x7f45c0009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:05.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.594+0000 7f45d8695700 1 -- 172.21.15.158:0/3819781205 shutdown_connections 2023-10-16T11:20:05.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.594+0000 7f45d8695700 1 --2- 172.21.15.158:0/3819781205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d00caaa0 0x7f45d00cae80 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.594+0000 7f45d8695700 1 -- 172.21.15.158:0/3819781205 >> 172.21.15.158:0/3819781205 conn(0x7f45d00cc650 msgr2=0x7f45d00cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:05.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.594+0000 7f45d8695700 1 -- 172.21.15.158:0/3819781205 shutdown_connections 2023-10-16T11:20:05.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.594+0000 7f45d8695700 1 -- 172.21.15.158:0/3819781205 wait complete. 2023-10-16T11:20:05.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.595+0000 7f45d8695700 1 Processor -- start 2023-10-16T11:20:05.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.596+0000 7f45d8695700 1 -- start start 2023-10-16T11:20:05.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.596+0000 7f45d8695700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d006bbc0 0x7f45d00c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.596+0000 7f45d8695700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45d006d740 0x7f45d0158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.596+0000 7f45d8695700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45d00caaa0 0x7f45d00c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.596+0000 7f45d8695700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00c7bf0 con 0x7f45d006bbc0 2023-10-16T11:20:05.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.596+0000 7f45d8695700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00c7e90 con 0x7f45d006d740 2023-10-16T11:20:05.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.596+0000 7f45d8695700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00c8170 con 0x7f45d00caaa0 2023-10-16T11:20:05.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d006bbc0 0x7f45d00c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d006bbc0 0x7f45d00c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39700/0 (socket says 172.21.15.158:39700) 2023-10-16T11:20:05.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 -- 172.21.15.158:0/3961331947 learned_addr learned my addr 172.21.15.158:0/3961331947 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:05.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d5c30700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45d006d740 0x7f45d0158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 -- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45d00caaa0 msgr2=0x7f45d00c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6c32700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45d00caaa0 0x7f45d00c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45d00caaa0 0x7f45d00c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 -- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45d006d740 msgr2=0x7f45d0158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45d006d740 0x7f45d0158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 -- 172.21.15.158:0/3961331947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c00094a0 con 0x7f45d006bbc0 2023-10-16T11:20:05.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d5c30700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45d006d740 0x7f45d0158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:05.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6c32700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45d00caaa0 0x7f45d00c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:05.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d6431700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d006bbc0 0x7f45d00c62d0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f45c0014040 tx=0x7f45c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:05.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45c6ffd700 1 -- 172.21.15.158:0/3961331947 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c001a900 con 0x7f45d006bbc0 2023-10-16T11:20:05.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45c6ffd700 1 -- 172.21.15.158:0/3961331947 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c001aa60 con 0x7f45d006bbc0 2023-10-16T11:20:05.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45c6ffd700 1 -- 172.21.15.158:0/3961331947 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c001add0 con 0x7f45d006bbc0 2023-10-16T11:20:05.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d00c8450 con 0x7f45d006bbc0 2023-10-16T11:20:05.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.597+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d00c8a10 con 0x7f45d006bbc0 2023-10-16T11:20:05.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.598+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d00681b0 con 0x7f45d006bbc0 2023-10-16T11:20:05.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.599+0000 7f45c6ffd700 1 -- 172.21.15.158:0/3961331947 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f45c0022070 con 0x7f45d006bbc0 2023-10-16T11:20:05.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.599+0000 7f45c6ffd700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f45b406ad50 0x7f45b406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.599+0000 7f45c6ffd700 1 -- 172.21.15.158:0/3961331947 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f45c008ab60 con 0x7f45d006bbc0 2023-10-16T11:20:05.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.599+0000 7f45d5c30700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f45b406ad50 0x7f45b406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.599+0000 7f45d5c30700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f45b406ad50 0x7f45b406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45cc007fd0 tx=0x7f45cc009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:05.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.603+0000 7f45c6ffd700 1 -- 172.21.15.158:0/3961331947 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f45c00555b0 con 0x7f45d006bbc0 2023-10-16T11:20:05.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.841+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f45d00ce360 con 0x7f45d006bbc0 2023-10-16T11:20:05.843 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:20:05.842+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:20:05.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.843+0000 7f45c6ffd700 1 -- 172.21.15.158:0/3961331947 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v119) v1 ==== 161+0+0 (secure 0 0 0) 0x7f45c0029090 con 0x7f45d006bbc0 2023-10-16T11:20:05.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f45b406ad50 msgr2=0x7f45b406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f45b406ad50 0x7f45b406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45cc007fd0 tx=0x7f45cc009040 comp rx=0 tx=0).stop 2023-10-16T11:20:05.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d006bbc0 msgr2=0x7f45d00c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d006bbc0 0x7f45d00c62d0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f45c0014040 tx=0x7f45c0004750 comp rx=0 tx=0).stop 2023-10-16T11:20:05.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 shutdown_connections 2023-10-16T11:20:05.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f45b406ad50 0x7f45b406d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45d00caaa0 0x7f45d00c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45d006bbc0 0x7f45d00c62d0 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 --2- 172.21.15.158:0/3961331947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45d006d740 0x7f45d0158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.846+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 >> 172.21.15.158:0/3961331947 conn(0x7f45d00cc650 msgr2=0x7f45d00af840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:05.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.847+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 shutdown_connections 2023-10-16T11:20:05.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.847+0000 7f45d8695700 1 -- 172.21.15.158:0/3961331947 wait complete. 2023-10-16T11:20:05.851 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:20:05.853 INFO:tasks.ceph.mds.b.smithi172.stderr: -5789> 2023-10-16T11:19:25.307+0000 7f8dbaac2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:20:05.860 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:20:05.884 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:20:05.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.983+0000 7f2a340a2700 1 Processor -- start 2023-10-16T11:20:05.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.984+0000 7f2a340a2700 1 -- start start 2023-10-16T11:20:05.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.984+0000 7f2a340a2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a2c06afe0 0x7f2a2c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.984+0000 7f2a340a2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a2c0cc590 0x7f2a2c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.984+0000 7f2a340a2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2c0cbd30 con 0x7f2a2c15da40 2023-10-16T11:20:05.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.984+0000 7f2a340a2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2c0cc1c0 con 0x7f2a2c06afe0 2023-10-16T11:20:05.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.984+0000 7f2a340a2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2c15de20 con 0x7f2a2c0cc590 2023-10-16T11:20:05.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3163d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a2c06afe0 0x7f2a2c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a31e3e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2a2c15da40 0x7f2a2c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39884/0 (socket says 172.21.15.158:39884) 2023-10-16T11:20:05.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a31e3e700 1 -- 172.21.15.158:0/4182699237 learned_addr learned my addr 172.21.15.158:0/4182699237 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:05.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3263f700 1 --2- 172.21.15.158:0/4182699237 >> v2:172.21.15.172:3300/0 conn(0x7f2a2c0cc590 0x7f2a2c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3163d700 1 -- 172.21.15.158:0/4182699237 >> v2:172.21.15.172:3300/0 conn(0x7f2a2c0cc590 msgr2=0x7f2a2c0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:05.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3163d700 1 --2- 172.21.15.158:0/4182699237 >> v2:172.21.15.172:3300/0 conn(0x7f2a2c0cc590 0x7f2a2c0afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3163d700 1 -- 172.21.15.158:0/4182699237 >> v1:172.21.15.158:6789/0 conn(0x7f2a2c15da40 legacy=0x7f2a2c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3163d700 1 -- 172.21.15.158:0/4182699237 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2c0ce260 con 0x7f2a2c06afe0 2023-10-16T11:20:05.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a30e3c700 1 -- 172.21.15.158:0/4182699237 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977375889 0 0) 0x7f2a2c0cbd30 con 0x7f2a2c15da40 2023-10-16T11:20:05.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3263f700 1 --2- 172.21.15.158:0/4182699237 >> v2:172.21.15.172:3300/0 conn(0x7f2a2c0cc590 0x7f2a2c0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:05.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a3163d700 1 --2- 172.21.15.158:0/4182699237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c0caa20 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0cdd70 tx=0x7f2a28009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1704dc6c24613b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:05.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a30e3c700 1 -- 172.21.15.158:0/4182699237 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28011830 con 0x7f2a2c06afe0 2023-10-16T11:20:05.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a30e3c700 1 -- 172.21.15.158:0/4182699237 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a28011990 con 0x7f2a2c06afe0 2023-10-16T11:20:05.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.985+0000 7f2a30e3c700 1 -- 172.21.15.158:0/4182699237 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28011d20 con 0x7f2a2c06afe0 2023-10-16T11:20:05.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 -- 172.21.15.158:0/4182699237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c06afe0 msgr2=0x7f2a2c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 --2- 172.21.15.158:0/4182699237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c0caa20 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0cdd70 tx=0x7f2a28009b20 comp rx=0 tx=0).stop 2023-10-16T11:20:05.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 -- 172.21.15.158:0/4182699237 shutdown_connections 2023-10-16T11:20:05.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 --2- 172.21.15.158:0/4182699237 >> v2:172.21.15.172:3300/0 conn(0x7f2a2c0cc590 0x7f2a2c0afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 --2- 172.21.15.158:0/4182699237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c0caa20 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 -- 172.21.15.158:0/4182699237 >> 172.21.15.158:0/4182699237 conn(0x7f2a2c099a10 msgr2=0x7f2a2c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:05.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 -- 172.21.15.158:0/4182699237 shutdown_connections 2023-10-16T11:20:05.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.986+0000 7f2a340a2700 1 -- 172.21.15.158:0/4182699237 wait complete. 2023-10-16T11:20:05.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.987+0000 7f2a340a2700 1 Processor -- start 2023-10-16T11:20:05.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.987+0000 7f2a340a2700 1 -- start start 2023-10-16T11:20:05.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a340a2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a340a2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a2c0cc590 0x7f2a2c0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a340a2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c15da40 0x7f2a2c0cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:05.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a340a2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c09af40 con 0x7f2a2c15da40 2023-10-16T11:20:05.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a340a2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c15bef0 con 0x7f2a2c06afe0 2023-10-16T11:20:05.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a340a2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c09a330 con 0x7f2a2c0cc590 2023-10-16T11:20:05.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c15da40 0x7f2a2c0cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c15da40 0x7f2a2c0cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39716/0 (socket says 172.21.15.158:39716) 2023-10-16T11:20:05.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 -- 172.21.15.158:0/3282685366 learned_addr learned my addr 172.21.15.158:0/3282685366 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:05.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a31e3e700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 -- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a2c0cc590 msgr2=0x7f2a2c0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3163d700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a2c0cc590 0x7f2a2c0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:05.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a2c0cc590 0x7f2a2c0ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 -- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c06afe0 msgr2=0x7f2a2c15b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:05.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c15b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:05.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 -- 172.21.15.158:0/3282685366 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a280094a0 con 0x7f2a2c15da40 2023-10-16T11:20:05.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a31e3e700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c15b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:06.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.988+0000 7f2a3263f700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c15da40 0x7f2a2c0cd1b0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400cef0 tx=0x7f2a2400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:06.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.989+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400ae50 con 0x7f2a2c15da40 2023-10-16T11:20:06.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.989+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a24015440 con 0x7f2a2c15da40 2023-10-16T11:20:06.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.989+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a240157f0 con 0x7f2a2c15da40 2023-10-16T11:20:06.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.989+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c09a610 con 0x7f2a2c15da40 2023-10-16T11:20:06.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.989+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c09abd0 con 0x7f2a2c15da40 2023-10-16T11:20:06.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.989+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c0681b0 con 0x7f2a2c15da40 2023-10-16T11:20:06.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.990+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2a2400d430 con 0x7f2a2c15da40 2023-10-16T11:20:06.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.990+0000 7f2a227fc700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a1006ad50 0x7f2a1006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:06.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.991+0000 7f2a31e3e700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a1006ad50 0x7f2a1006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:06.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.991+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f2a2408a2c0 con 0x7f2a2c15da40 2023-10-16T11:20:06.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.991+0000 7f2a31e3e700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a1006ad50 0x7f2a1006d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c007fd0 tx=0x7f2a1c009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:06.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:05.995+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a24054c00 con 0x7f2a2c15da40 2023-10-16T11:20:06.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.232+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2a2c06ed70 con 0x7f2a2c15da40 2023-10-16T11:20:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.445+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 146) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f2a24054fd0 con 0x7f2a2c15da40 2023-10-16T11:20:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.845+0000 7f2a227fc700 1 -- 172.21.15.158:0/3282685366 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v120) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2a24071530 con 0x7f2a2c15da40 2023-10-16T11:20:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a1006ad50 msgr2=0x7f2a1006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a1006ad50 0x7f2a1006d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c007fd0 tx=0x7f2a1c009040 comp rx=0 tx=0).stop 2023-10-16T11:20:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c15da40 msgr2=0x7f2a2c0cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:06.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c15da40 0x7f2a2c0cd1b0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400cef0 tx=0x7f2a2400a340 comp rx=0 tx=0).stop 2023-10-16T11:20:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 shutdown_connections 2023-10-16T11:20:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a1006ad50 0x7f2a1006d210 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a2c15da40 0x7f2a2c0cd1b0 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c06afe0 0x7f2a2c15b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:06.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 --2- 172.21.15.158:0/3282685366 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a2c0cc590 0x7f2a2c0ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 >> 172.21.15.158:0/3282685366 conn(0x7f2a2c099a10 msgr2=0x7f2a2c06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 shutdown_connections 2023-10-16T11:20:06.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.848+0000 7f2a340a2700 1 -- 172.21.15.158:0/3282685366 wait complete. 2023-10-16T11:20:06.863 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:20:06.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.988+0000 7f020e947700 1 Processor -- start 2023-10-16T11:20:06.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.989+0000 7f020e947700 1 -- start start 2023-10-16T11:20:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f020e947700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0208159e20 0x7f02080cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f020e947700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f020806ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:06.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f020e947700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02080ce110 con 0x7f020806bc30 2023-10-16T11:20:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f020e947700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020806b2c0 con 0x7f0208159e20 2023-10-16T11:20:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f020e947700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02080cb8f0 con 0x7f020806a8d0 2023-10-16T11:20:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0207fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0208159e20 0x7f02080cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f02077fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f020806bc30 0x7f02080ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39898/0 (socket says 172.21.15.158:39898) 2023-10-16T11:20:06.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f02077fe700 1 -- 172.21.15.158:0/3550879432 learned_addr learned my addr 172.21.15.158:0/3550879432 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f020cee4700 1 --2- 172.21.15.158:0/3550879432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f020806ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0207fff700 1 -- 172.21.15.158:0/3550879432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 msgr2=0x7f020806ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0207fff700 1 --2- 172.21.15.158:0/3550879432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f020806ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0206ffd700 1 -- 172.21.15.158:0/3550879432 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987221758 0 0) 0x7f02080ce110 con 0x7f020806bc30 2023-10-16T11:20:06.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0207fff700 1 -- 172.21.15.158:0/3550879432 >> v1:172.21.15.158:6789/0 conn(0x7f020806bc30 legacy=0x7f02080ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:06.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0207fff700 1 -- 172.21.15.158:0/3550879432 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02080cbbc0 con 0x7f0208159e20 2023-10-16T11:20:06.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0207fff700 1 --2- 172.21.15.158:0/3550879432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080cb3b0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009840 tx=0x7f01fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdd69d4e32af5af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:06.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0206ffd700 1 -- 172.21.15.158:0/3550879432 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc011830 con 0x7f0208159e20 2023-10-16T11:20:06.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0206ffd700 1 -- 172.21.15.158:0/3550879432 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc011990 con 0x7f0208159e20 2023-10-16T11:20:06.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.990+0000 7f0206ffd700 1 -- 172.21.15.158:0/3550879432 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc011d20 con 0x7f0208159e20 2023-10-16T11:20:06.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 -- 172.21.15.158:0/3550879432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 msgr2=0x7f02080cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:06.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 --2- 172.21.15.158:0/3550879432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080cb3b0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009840 tx=0x7f01fc009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:06.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 -- 172.21.15.158:0/3550879432 shutdown_connections 2023-10-16T11:20:06.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 --2- 172.21.15.158:0/3550879432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080cb3b0 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:06.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 --2- 172.21.15.158:0/3550879432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f020806ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:06.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 -- 172.21.15.158:0/3550879432 >> 172.21.15.158:0/3550879432 conn(0x7f02080cc690 msgr2=0x7f02080cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:06.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 -- 172.21.15.158:0/3550879432 shutdown_connections 2023-10-16T11:20:06.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.991+0000 7f020e947700 1 -- 172.21.15.158:0/3550879432 wait complete. 2023-10-16T11:20:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.992+0000 7f020e947700 1 Processor -- start 2023-10-16T11:20:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.992+0000 7f020e947700 1 -- start start 2023-10-16T11:20:07.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020e947700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f02080c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020e947700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f020806bc30 0x7f0208149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020e947700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020e947700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02080c3af0 con 0x7f0208159e20 2023-10-16T11:20:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020e947700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02080c40b0 con 0x7f020806bc30 2023-10-16T11:20:07.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020e947700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02080c6a50 con 0x7f020806a8d0 2023-10-16T11:20:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020cee4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020cee4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39730/0 (socket says 172.21.15.158:39730) 2023-10-16T11:20:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020cee4700 1 -- 172.21.15.158:0/1867214057 learned_addr learned my addr 172.21.15.158:0/1867214057 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f0207fff700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f02080c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:07.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f020cee4700 1 -- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 msgr2=0x7f02080c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.993+0000 7f02077fe700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f020806bc30 0x7f0208149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f020cee4700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f02080c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f020cee4700 1 -- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f020806bc30 msgr2=0x7f0208149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f020cee4700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f020806bc30 0x7f0208149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:07.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f020cee4700 1 -- 172.21.15.158:0/1867214057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc0094a0 con 0x7f0208159e20 2023-10-16T11:20:07.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f0207fff700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f02080c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:07.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f02077fe700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f020806bc30 0x7f0208149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:07.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f020cee4700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080c6510 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f020000cda0 tx=0x7f020000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:07.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f0204ff9700 1 -- 172.21.15.158:0/1867214057 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200003a20 con 0x7f0208159e20 2023-10-16T11:20:07.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f0204ff9700 1 -- 172.21.15.158:0/1867214057 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200003b80 con 0x7f0208159e20 2023-10-16T11:20:07.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0208149660 con 0x7f0208159e20 2023-10-16T11:20:07.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f0204ff9700 1 -- 172.21.15.158:0/1867214057 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200004020 con 0x7f0208159e20 2023-10-16T11:20:07.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.994+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02080c3d40 con 0x7f0208159e20 2023-10-16T11:20:07.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.995+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02080681b0 con 0x7f0208159e20 2023-10-16T11:20:07.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.995+0000 7f0204ff9700 1 -- 172.21.15.158:0/1867214057 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 146) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f0200004180 con 0x7f0208159e20 2023-10-16T11:20:07.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.996+0000 7f0204ff9700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f01f406b0b0 0x7f01f406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:07.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.996+0000 7f0204ff9700 1 -- 172.21.15.158:0/1867214057 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f0200012070 con 0x7f0208159e20 2023-10-16T11:20:07.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.996+0000 7f0207fff700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f01f406b0b0 0x7f01f406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:07.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:06.996+0000 7f0207fff700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f01f406b0b0 0x7f01f406d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0090b0 tx=0x7f01fc008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:07.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:07.000+0000 7f0204ff9700 1 -- 172.21.15.158:0/1867214057 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0200054f70 con 0x7f0208159e20 2023-10-16T11:20:07.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:07.194+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f02080c41f0 con 0x7f0208159e20 2023-10-16T11:20:08.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.054+0000 7f0204ff9700 1 -- 172.21.15.158:0/1867214057 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v225) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0200058590 con 0x7f0208159e20 2023-10-16T11:20:08.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.058+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f01f406b0b0 msgr2=0x7f01f406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:08.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.058+0000 7f020e947700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f01f406b0b0 0x7f01f406d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0090b0 tx=0x7f01fc008030 comp rx=0 tx=0).stop 2023-10-16T11:20:08.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.058+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 msgr2=0x7f02080c6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:08.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.058+0000 7f020e947700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080c6510 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f020000cda0 tx=0x7f020000a570 comp rx=0 tx=0).stop 2023-10-16T11:20:08.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 shutdown_connections 2023-10-16T11:20:08.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f01f406b0b0 0x7f01f406d570 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0208159e20 0x7f02080c6510 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f020806a8d0 0x7f02080c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 --2- 172.21.15.158:0/1867214057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f020806bc30 0x7f0208149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 >> 172.21.15.158:0/1867214057 conn(0x7f02080cc690 msgr2=0x7f020806dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:08.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 shutdown_connections 2023-10-16T11:20:08.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.060+0000 7f020e947700 1 -- 172.21.15.158:0/1867214057 wait complete. 2023-10-16T11:20:08.063 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:20:08.077 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:20:08.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.209+0000 7fba1d366700 1 Processor -- start 2023-10-16T11:20:08.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.209+0000 7fba1d366700 1 -- start start 2023-10-16T11:20:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba1d366700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 0x7fba180b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba1d366700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba180b7c90 con 0x7fba180d29b0 2023-10-16T11:20:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba1d366700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba180b7f70 con 0x7fba18160d30 2023-10-16T11:20:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba1d366700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b8910 con 0x7fba180d0c40 2023-10-16T11:20:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba167fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fba180d29b0 0x7fba180d0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39912/0 (socket says 172.21.15.158:39912) 2023-10-16T11:20:08.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba167fc700 1 -- 172.21.15.158:0/3418772537 learned_addr learned my addr 172.21.15.158:0/3418772537 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:08.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317951024 0 0) 0x7fba180b7c90 con 0x7fba180d29b0 2023-10-16T11:20:08.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba000035b0 con 0x7fba180d29b0 2023-10-16T11:20:08.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba177fe700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 0x7fba180b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3331767729 0 0) 0x7fba000035b0 con 0x7fba180d29b0 2023-10-16T11:20:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 msgr2=0x7fba180b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 0x7fba180b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 >> v1:172.21.15.172:6789/0 conn(0x7fba18160d30 legacy=0x7fba180b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba180b8250 con 0x7fba180d29b0 2023-10-16T11:20:08.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fba08001670 con 0x7fba180d29b0 2023-10-16T11:20:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 >> v1:172.21.15.158:6789/0 conn(0x7fba180d29b0 legacy=0x7fba180d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba00003a30 0x7fba00003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba00004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 0x7fba0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b8910 con 0x7fba00004850 2023-10-16T11:20:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000055a0 con 0x7fba00003a30 2023-10-16T11:20:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0000bf50 con 0x7fba180d0c40 2023-10-16T11:20:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.210+0000 7fba167fc700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba00004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba16ffd700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba00003a30 0x7fba00003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba177fe700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 0x7fba0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba167fc700 1 -- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 msgr2=0x7fba0000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba167fc700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 0x7fba0000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba167fc700 1 -- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba00003a30 msgr2=0x7fba00003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba167fc700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba00003a30 0x7fba00003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba167fc700 1 -- 172.21.15.158:0/3418772537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba000035b0 con 0x7fba00004850 2023-10-16T11:20:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba177fe700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba180d0c40 0x7fba0000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:08.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba167fc700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba00004cd0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7fba0800c240 tx=0x7fba00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7dbd8a0a5a3bfe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08007c10 con 0x7fba00004850 2023-10-16T11:20:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0800cde0 con 0x7fba00004850 2023-10-16T11:20:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba15ffb700 1 -- 172.21.15.158:0/3418772537 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08005ba0 con 0x7fba00004850 2023-10-16T11:20:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba1d366700 1 -- 172.21.15.158:0/3418772537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 msgr2=0x7fba00004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba1d366700 1 --2- 172.21.15.158:0/3418772537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba00004cd0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7fba0800c240 tx=0x7fba00005880 comp rx=0 tx=0).stop 2023-10-16T11:20:08.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.211+0000 7fba177fe700 1 -- 172.21.15.158:0/3418772537 reap_dead start 2023-10-16T11:20:08.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.212+0000 7fba1d366700 1 -- 172.21.15.158:0/3418772537 shutdown_connections 2023-10-16T11:20:08.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.212+0000 7fba1d366700 1 -- 172.21.15.158:0/3418772537 >> 172.21.15.158:0/3418772537 conn(0x7fba180669f0 msgr2=0x7fba18066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:08.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.212+0000 7fba1d366700 1 -- 172.21.15.158:0/3418772537 shutdown_connections 2023-10-16T11:20:08.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.212+0000 7fba1d366700 1 -- 172.21.15.158:0/3418772537 wait complete. 2023-10-16T11:20:08.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.213+0000 7fba1d366700 1 Processor -- start 2023-10-16T11:20:08.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.213+0000 7fba1d366700 1 -- start start 2023-10-16T11:20:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba1814b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1815a730 0x7fba1815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba18158830 0x7fba18158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1815b0f0 con 0x7fba00004850 2023-10-16T11:20:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba181594f0 con 0x7fba18158830 2023-10-16T11:20:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba181597d0 con 0x7fba1815a730 2023-10-16T11:20:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba1814b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba1814b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39746/0 (socket says 172.21.15.158:39746) 2023-10-16T11:20:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 -- 172.21.15.158:0/2238062155 learned_addr learned my addr 172.21.15.158:0/2238062155 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba177fe700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba18158830 0x7fba18158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba167fc700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1815a730 0x7fba1815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 -- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1815a730 msgr2=0x7fba1815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1815a730 0x7fba1815abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 -- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba18158830 msgr2=0x7fba18158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba18158830 0x7fba18158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 -- 172.21.15.158:0/2238062155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba18159ab0 con 0x7fba00004850 2023-10-16T11:20:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba16ffd700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba1814b150 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00f7f0 tx=0x7fba0c00fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fb9ff7fe700 1 -- 172.21.15.158:0/2238062155 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0052e0 con 0x7fba00004850 2023-10-16T11:20:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fb9ff7fe700 1 -- 172.21.15.158:0/2238062155 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c005440 con 0x7fba00004850 2023-10-16T11:20:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba18159d90 con 0x7fba00004850 2023-10-16T11:20:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fb9ff7fe700 1 -- 172.21.15.158:0/2238062155 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0038c0 con 0x7fba00004850 2023-10-16T11:20:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.214+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba181591f0 con 0x7fba00004850 2023-10-16T11:20:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.215+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1815a3e0 con 0x7fba00004850 2023-10-16T11:20:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.216+0000 7fb9ff7fe700 1 -- 172.21.15.158:0/2238062155 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fba0c003a20 con 0x7fba00004850 2023-10-16T11:20:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.216+0000 7fb9ff7fe700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fba0406b140 0x7fba0406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:08.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.216+0000 7fb9ff7fe700 1 -- 172.21.15.158:0/2238062155 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 7838+0+0 (secure 0 0 0) 0x7fba0c089f40 con 0x7fba00004850 2023-10-16T11:20:08.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.216+0000 7fba167fc700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fba0406b140 0x7fba0406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:08.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.217+0000 7fba167fc700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fba0406b140 0x7fba0406d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba180c9d10 tx=0x7fba08007000 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:08.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.221+0000 7fb9ff7fe700 1 -- 172.21.15.158:0/2238062155 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba0c054440 con 0x7fba00004850 2023-10-16T11:20:08.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:08.418+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fba180681b0 con 0x7fba00004850 2023-10-16T11:20:09.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.056+0000 7fb9ff7fe700 1 -- 172.21.15.158:0/2238062155 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v226) v1 ==== 169+0+0 (secure 0 0 0) 0x7fba0c057a60 con 0x7fba00004850 2023-10-16T11:20:09.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.058+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fba0406b140 msgr2=0x7fba0406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.058+0000 7fba1d366700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fba0406b140 0x7fba0406d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba180c9d10 tx=0x7fba08007000 comp rx=0 tx=0).stop 2023-10-16T11:20:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.058+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 msgr2=0x7fba1814b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.058+0000 7fba1d366700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba1814b150 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00f7f0 tx=0x7fba0c00fb00 comp rx=0 tx=0).stop 2023-10-16T11:20:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 shutdown_connections 2023-10-16T11:20:09.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fba0406b140 0x7fba0406d600 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba18158830 0x7fba18158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004850 0x7fba1814b150 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 --2- 172.21.15.158:0/2238062155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba1815a730 0x7fba1815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 >> 172.21.15.158:0/2238062155 conn(0x7fba180669f0 msgr2=0x7fba180ca690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:09.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 shutdown_connections 2023-10-16T11:20:09.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.059+0000 7fba1d366700 1 -- 172.21.15.158:0/2238062155 wait complete. 2023-10-16T11:20:09.063 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:20:09.075 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:09.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.196+0000 7fcf126b9700 1 Processor -- start 2023-10-16T11:20:09.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.196+0000 7fcf126b9700 1 -- start start 2023-10-16T11:20:09.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf126b9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf0c15d7c0 0x7fcf0c0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf126b9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf126b9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0c0d3bd0 con 0x7fcf0c0d1210 2023-10-16T11:20:09.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf126b9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0c0d4060 con 0x7fcf0c15d7c0 2023-10-16T11:20:09.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf126b9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0d2cf0 con 0x7fcf0c0ceb20 2023-10-16T11:20:09.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf0bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf0c15d7c0 0x7fcf0c0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf0bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf0c15d7c0 0x7fcf0c0d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39754/0 (socket says 172.21.15.158:39754) 2023-10-16T11:20:09.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf0bfff700 1 -- 172.21.15.158:0/1594874256 learned_addr learned my addr 172.21.15.158:0/1594874256 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:09.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.197+0000 7fcf10c56700 1 --2- 172.21.15.158:0/1594874256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0bfff700 1 -- 172.21.15.158:0/1594874256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 msgr2=0x7fcf0c0b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:09.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0bfff700 1 --2- 172.21.15.158:0/1594874256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0bfff700 1 -- 172.21.15.158:0/1594874256 >> v1:172.21.15.172:6789/0 conn(0x7fcf0c0d1210 legacy=0x7fcf0c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0bfff700 1 -- 172.21.15.158:0/1594874256 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c0d2fc0 con 0x7fcf0c15d7c0 2023-10-16T11:20:09.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0bfff700 1 --2- 172.21.15.158:0/1594874256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c15d7c0 0x7fcf0c0d27b0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0098f0 tx=0x7fcefc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a71edd0b1a170aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0affd700 1 -- 172.21.15.158:0/1594874256 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc011830 con 0x7fcf0c15d7c0 2023-10-16T11:20:09.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0affd700 1 -- 172.21.15.158:0/1594874256 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc011990 con 0x7fcf0c15d7c0 2023-10-16T11:20:09.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf0affd700 1 -- 172.21.15.158:0/1594874256 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc011d20 con 0x7fcf0c15d7c0 2023-10-16T11:20:09.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf126b9700 1 -- 172.21.15.158:0/1594874256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c15d7c0 msgr2=0x7fcf0c0d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.198+0000 7fcf126b9700 1 --2- 172.21.15.158:0/1594874256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c15d7c0 0x7fcf0c0d27b0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0098f0 tx=0x7fcefc009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:09.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.199+0000 7fcf126b9700 1 -- 172.21.15.158:0/1594874256 shutdown_connections 2023-10-16T11:20:09.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.199+0000 7fcf126b9700 1 --2- 172.21.15.158:0/1594874256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.199+0000 7fcf126b9700 1 --2- 172.21.15.158:0/1594874256 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c15d7c0 0x7fcf0c0d27b0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.199+0000 7fcf126b9700 1 -- 172.21.15.158:0/1594874256 >> 172.21.15.158:0/1594874256 conn(0x7fcf0c0669f0 msgr2=0x7fcf0c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:09.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.199+0000 7fcf126b9700 1 -- 172.21.15.158:0/1594874256 shutdown_connections 2023-10-16T11:20:09.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.199+0000 7fcf126b9700 1 -- 172.21.15.158:0/1594874256 wait complete. 2023-10-16T11:20:09.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.200+0000 7fcf126b9700 1 Processor -- start 2023-10-16T11:20:09.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.200+0000 7fcf126b9700 1 -- start start 2023-10-16T11:20:09.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c0d1210 0x7fcf0c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf0c15d7c0 0x7fcf0c0b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c1f6c10 con 0x7fcf0c0d1210 2023-10-16T11:20:09.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c1f6270 con 0x7fcf0c15d7c0 2023-10-16T11:20:09.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c1f6550 con 0x7fcf0c0ceb20 2023-10-16T11:20:09.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c0d1210 0x7fcf0c0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c0d1210 0x7fcf0c0b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39762/0 (socket says 172.21.15.158:39762) 2023-10-16T11:20:09.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 -- 172.21.15.158:0/3000894063 learned_addr learned my addr 172.21.15.158:0/3000894063 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:09.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf10c56700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf0c15d7c0 0x7fcf0c0b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0bfff700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 -- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 msgr2=0x7fcf0c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 -- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf0c15d7c0 msgr2=0x7fcf0c0b8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf0c15d7c0 0x7fcf0c0b8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 -- 172.21.15.158:0/3000894063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc009550 con 0x7fcf0c0d1210 2023-10-16T11:20:09.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf0b7fe700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c0d1210 0x7fcf0c0b8810 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000c4e0 tx=0x7fcf0000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf08ff9700 1 -- 172.21.15.158:0/3000894063 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf000052d0 con 0x7fcf0c0d1210 2023-10-16T11:20:09.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf08ff9700 1 -- 172.21.15.158:0/3000894063 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf00005430 con 0x7fcf0c0d1210 2023-10-16T11:20:09.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf08ff9700 1 -- 172.21.15.158:0/3000894063 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf000038c0 con 0x7fcf0c0d1210 2023-10-16T11:20:09.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c1f6830 con 0x7fcf0c0d1210 2023-10-16T11:20:09.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.201+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c1f7030 con 0x7fcf0c0d1210 2023-10-16T11:20:09.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.202+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c0d3840 con 0x7fcf0c0d1210 2023-10-16T11:20:09.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.203+0000 7fcf08ff9700 1 -- 172.21.15.158:0/3000894063 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcf00003a20 con 0x7fcf0c0d1210 2023-10-16T11:20:09.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.203+0000 7fcf08ff9700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fcef806b140 0x7fcef806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.203+0000 7fcf08ff9700 1 -- 172.21.15.158:0/3000894063 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7fcf00089430 con 0x7fcf0c0d1210 2023-10-16T11:20:09.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.203+0000 7fcf0bfff700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fcef806b140 0x7fcef806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.204+0000 7fcf0bfff700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fcef806b140 0x7fcef806d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcefc014040 tx=0x7fcefc008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.208+0000 7fcf08ff9700 1 -- 172.21.15.158:0/3000894063 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf0c0d3840 con 0x7fcf0c0d1210 2023-10-16T11:20:09.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.403+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf0c1f6d50 con 0x7fcf0c0d1210 2023-10-16T11:20:09.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.405+0000 7fcf08ff9700 1 -- 172.21.15.158:0/3000894063 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+18806 (secure 0 0 0) 0x7fcf0c1f6d50 con 0x7fcf0c0d1210 2023-10-16T11:20:09.408 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:09.408 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":226,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:09.055255+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/931576556":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2043584975":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/139807143":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6834/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:0/572486792":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1251861851":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6832/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:6832/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2783528774":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1862821458":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/417671612":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/534116803":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fcef806b140 msgr2=0x7fcef806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fcef806b140 0x7fcef806d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcefc014040 tx=0x7fcefc008030 comp rx=0 tx=0).stop 2023-10-16T11:20:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c0d1210 msgr2=0x7fcf0c0b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c0d1210 0x7fcf0c0b8810 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000c4e0 tx=0x7fcf0000c8a0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 shutdown_connections 2023-10-16T11:20:09.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fcef806b140 0x7fcef806d600 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf0c15d7c0 0x7fcf0c0b8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcf0c0ceb20 0x7fcf0c0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 --2- 172.21.15.158:0/3000894063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0c0d1210 0x7fcf0c0b8810 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 >> 172.21.15.158:0/3000894063 conn(0x7fcf0c0669f0 msgr2=0x7fcf0c15caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:09.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 shutdown_connections 2023-10-16T11:20:09.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.407+0000 7fcf126b9700 1 -- 172.21.15.158:0/3000894063 wait complete. 2023-10-16T11:20:09.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T11:19:33.715104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '220', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T11:19:35.032906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:20:09.423 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:20:09.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.545+0000 7fb0ace24700 1 Processor -- start 2023-10-16T11:20:09.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.545+0000 7fb0ace24700 1 -- start start 2023-10-16T11:20:09.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0ace24700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0a806afe0 0x7fb0a80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0ace24700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0ace24700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a80cbd30 con 0x7fb0a815da40 2023-10-16T11:20:09.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0ace24700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a80cc1c0 con 0x7fb0a806afe0 2023-10-16T11:20:09.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0ace24700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a815de20 con 0x7fb0a80cc590 2023-10-16T11:20:09.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0a806afe0 0x7fb0a80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0a806afe0 0x7fb0a80caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39770/0 (socket says 172.21.15.158:39770) 2023-10-16T11:20:09.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 -- 172.21.15.158:0/2656250227 learned_addr learned my addr 172.21.15.158:0/2656250227 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:09.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a6d9d700 1 --2- 172.21.15.158:0/2656250227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 -- 172.21.15.158:0/2656250227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 msgr2=0x7fb0a80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:09.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 --2- 172.21.15.158:0/2656250227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 -- 172.21.15.158:0/2656250227 >> v1:172.21.15.172:6789/0 conn(0x7fb0a815da40 legacy=0x7fb0a806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:09.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 -- 172.21.15.158:0/2656250227 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a80ce260 con 0x7fb0a806afe0 2023-10-16T11:20:09.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a5d9b700 1 --2- 172.21.15.158:0/2656250227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a806afe0 0x7fb0a80caa20 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0098f0 tx=0x7fb09c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78707f61e1b5bc1d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a559a700 1 -- 172.21.15.158:0/2656250227 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c004230 con 0x7fb0a806afe0 2023-10-16T11:20:09.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a559a700 1 -- 172.21.15.158:0/2656250227 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c004390 con 0x7fb0a806afe0 2023-10-16T11:20:09.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.546+0000 7fb0a559a700 1 -- 172.21.15.158:0/2656250227 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0036a0 con 0x7fb0a806afe0 2023-10-16T11:20:09.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 -- 172.21.15.158:0/2656250227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a806afe0 msgr2=0x7fb0a80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 --2- 172.21.15.158:0/2656250227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a806afe0 0x7fb0a80caa20 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0098f0 tx=0x7fb09c009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:09.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 -- 172.21.15.158:0/2656250227 shutdown_connections 2023-10-16T11:20:09.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 --2- 172.21.15.158:0/2656250227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 --2- 172.21.15.158:0/2656250227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a806afe0 0x7fb0a80caa20 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 -- 172.21.15.158:0/2656250227 >> 172.21.15.158:0/2656250227 conn(0x7fb0a8099a10 msgr2=0x7fb0a80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:09.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 -- 172.21.15.158:0/2656250227 shutdown_connections 2023-10-16T11:20:09.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.547+0000 7fb0ace24700 1 -- 172.21.15.158:0/2656250227 wait complete. 2023-10-16T11:20:09.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.548+0000 7fb0ace24700 1 Processor -- start 2023-10-16T11:20:09.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.548+0000 7fb0ace24700 1 -- start start 2023-10-16T11:20:09.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0ace24700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0a806afe0 0x7fb0a814d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0ace24700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0ace24700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a815da40 0x7fb0a814e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0ace24700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a80ae570 con 0x7fb0a815da40 2023-10-16T11:20:09.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0ace24700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a80ad960 con 0x7fb0a806afe0 2023-10-16T11:20:09.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0ace24700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a80adc40 con 0x7fb0a80cc590 2023-10-16T11:20:09.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0a6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a815da40 0x7fb0a814e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0a6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a815da40 0x7fb0a814e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39776/0 (socket says 172.21.15.158:39776) 2023-10-16T11:20:09.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.549+0000 7fb0a6d9d700 1 -- 172.21.15.158:0/3394192093 learned_addr learned my addr 172.21.15.158:0/3394192093 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:09.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a659c700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0a806afe0 0x7fb0a814d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a5d9b700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a6d9d700 1 -- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 msgr2=0x7fb0a80ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a6d9d700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a6d9d700 1 -- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0a806afe0 msgr2=0x7fb0a814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a6d9d700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0a806afe0 0x7fb0a814d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a6d9d700 1 -- 172.21.15.158:0/3394192093 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c009550 con 0x7fb0a815da40 2023-10-16T11:20:09.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0a6d9d700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a815da40 0x7fb0a814e290 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007e40 tx=0x7fb0a000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb096ffd700 1 -- 172.21.15.158:0/3394192093 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a000eea0 con 0x7fb0a815da40 2023-10-16T11:20:09.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb096ffd700 1 -- 172.21.15.158:0/3394192093 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a0014420 con 0x7fb0a815da40 2023-10-16T11:20:09.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a80adf20 con 0x7fb0a815da40 2023-10-16T11:20:09.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb096ffd700 1 -- 172.21.15.158:0/3394192093 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00147b0 con 0x7fb0a815da40 2023-10-16T11:20:09.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.550+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a814dd10 con 0x7fb0a815da40 2023-10-16T11:20:09.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.551+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a80ae6e0 con 0x7fb0a815da40 2023-10-16T11:20:09.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.551+0000 7fb096ffd700 1 -- 172.21.15.158:0/3394192093 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb0a0014910 con 0x7fb0a815da40 2023-10-16T11:20:09.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.552+0000 7fb096ffd700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fb08806b140 0x7fb08806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.552+0000 7fb0a659c700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fb08806b140 0x7fb08806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.552+0000 7fb096ffd700 1 -- 172.21.15.158:0/3394192093 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7fb0a008a8c0 con 0x7fb0a815da40 2023-10-16T11:20:09.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.552+0000 7fb0a659c700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fb08806b140 0x7fb08806d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb098004f30 tx=0x7fb098009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.556+0000 7fb096ffd700 1 -- 172.21.15.158:0/3394192093 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0a0054fa0 con 0x7fb0a815da40 2023-10-16T11:20:09.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.755+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0a80681b0 con 0x7fb0a815da40 2023-10-16T11:20:09.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.755+0000 7fb096ffd700 1 -- 172.21.15.158:0/3394192093 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v226) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0a00585c0 con 0x7fb0a815da40 2023-10-16T11:20:09.758 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:20:09.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fb08806b140 msgr2=0x7fb08806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fb08806b140 0x7fb08806d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb098004f30 tx=0x7fb098009040 comp rx=0 tx=0).stop 2023-10-16T11:20:09.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a815da40 msgr2=0x7fb0a814e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a815da40 0x7fb0a814e290 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007e40 tx=0x7fb0a000e3c0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 shutdown_connections 2023-10-16T11:20:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fb08806b140 0x7fb08806d600 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0a815da40 0x7fb0a814e290 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0a806afe0 0x7fb0a814d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 --2- 172.21.15.158:0/3394192093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0a80cc590 0x7fb0a80ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 >> 172.21.15.158:0/3394192093 conn(0x7fb0a8099a10 msgr2=0x7fb0a80c5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 shutdown_connections 2023-10-16T11:20:09.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.758+0000 7fb0ace24700 1 -- 172.21.15.158:0/3394192093 wait complete. 2023-10-16T11:20:09.772 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T11:20:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.893+0000 7fd987d6e700 1 Processor -- start 2023-10-16T11:20:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.893+0000 7fd987d6e700 1 -- start start 2023-10-16T11:20:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd987d6e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 0x7fd9800caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd987d6e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9800cbd30 con 0x7fd98015da40 2023-10-16T11:20:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd987d6e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9800cc1c0 con 0x7fd9800cc590 2023-10-16T11:20:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd987d6e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98015de20 con 0x7fd98006afe0 2023-10-16T11:20:09.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd985b0a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd98015da40 0x7fd98006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39958/0 (socket says 172.21.15.158:39958) 2023-10-16T11:20:09.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd985b0a700 1 -- 172.21.15.158:0/3947482426 learned_addr learned my addr 172.21.15.158:0/3947482426 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:09.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756046003 0 0) 0x7fd9800cbd30 con 0x7fd98015da40 2023-10-16T11:20:09.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd96c0035b0 con 0x7fd98015da40 2023-10-16T11:20:09.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.894+0000 7fd985309700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 0x7fd9800caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2020736500 0 0) 0x7fd96c0035b0 con 0x7fd98015da40 2023-10-16T11:20:09.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 msgr2=0x7fd9800caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:09.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 0x7fd9800caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 >> v1:172.21.15.172:6789/0 conn(0x7fd9800cc590 legacy=0x7fd9800afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9800ce260 con 0x7fd98015da40 2023-10-16T11:20:09.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd970001670 con 0x7fd98015da40 2023-10-16T11:20:09.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 >> v1:172.21.15.158:6789/0 conn(0x7fd98015da40 legacy=0x7fd98006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 0x7fd96c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd96c0040b0 0x7fd96c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c004d90 0x7fd96c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98015de20 con 0x7fd96c0040b0 2023-10-16T11:20:09.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c00bc50 con 0x7fd96c004d90 2023-10-16T11:20:09.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c00be10 con 0x7fd98006afe0 2023-10-16T11:20:09.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985309700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd96c0040b0 0x7fd96c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985b0a700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 0x7fd96c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd98630b700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c004d90 0x7fd96c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985309700 1 -- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 msgr2=0x7fd96c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:09.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985309700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd98006afe0 0x7fd96c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985309700 1 -- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c004d90 msgr2=0x7fd96c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:09.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985309700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c004d90 0x7fd96c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985309700 1 -- 172.21.15.158:0/3947482426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0035b0 con 0x7fd96c0040b0 2023-10-16T11:20:09.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd98630b700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c004d90 0x7fd96c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:09.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd985309700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd96c0040b0 0x7fd96c004850 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009890 tx=0x7fd96c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6275846a98f5618 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00c070 con 0x7fd96c0040b0 2023-10-16T11:20:09.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c013610 con 0x7fd96c0040b0 2023-10-16T11:20:09.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.895+0000 7fd984b08700 1 -- 172.21.15.158:0/3947482426 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c0139a0 con 0x7fd96c0040b0 2023-10-16T11:20:09.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.896+0000 7fd987d6e700 1 -- 172.21.15.158:0/3947482426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd96c0040b0 msgr2=0x7fd96c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.896+0000 7fd987d6e700 1 --2- 172.21.15.158:0/3947482426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd96c0040b0 0x7fd96c004850 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009890 tx=0x7fd96c005880 comp rx=0 tx=0).stop 2023-10-16T11:20:09.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.896+0000 7fd98630b700 1 -- 172.21.15.158:0/3947482426 reap_dead start 2023-10-16T11:20:09.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.896+0000 7fd987d6e700 1 -- 172.21.15.158:0/3947482426 shutdown_connections 2023-10-16T11:20:09.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.896+0000 7fd987d6e700 1 -- 172.21.15.158:0/3947482426 >> 172.21.15.158:0/3947482426 conn(0x7fd980099a10 msgr2=0x7fd9800d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:09.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.896+0000 7fd987d6e700 1 -- 172.21.15.158:0/3947482426 shutdown_connections 2023-10-16T11:20:09.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.896+0000 7fd987d6e700 1 -- 172.21.15.158:0/3947482426 wait complete. 2023-10-16T11:20:09.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.897+0000 7fd987d6e700 1 Processor -- start 2023-10-16T11:20:09.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.897+0000 7fd987d6e700 1 -- start start 2023-10-16T11:20:09.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd987d6e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd96c0040b0 0x7fd9800c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd987d6e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd98014fcc0 0x7fd9800ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd987d6e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9800ca700 0x7fd9800c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd987d6e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800c5070 con 0x7fd9800ca700 2023-10-16T11:20:09.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd987d6e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800c43f0 con 0x7fd98014fcc0 2023-10-16T11:20:09.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd987d6e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800c46d0 con 0x7fd96c0040b0 2023-10-16T11:20:09.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9800ca700 0x7fd9800c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9800ca700 0x7fd9800c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39798/0 (socket says 172.21.15.158:39798) 2023-10-16T11:20:09.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 -- 172.21.15.158:0/1507616785 learned_addr learned my addr 172.21.15.158:0/1507616785 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:09.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd985309700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd98014fcc0 0x7fd9800ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 -- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd96c0040b0 msgr2=0x7fd9800c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd985b0a700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd96c0040b0 0x7fd9800c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd96c0040b0 0x7fd9800c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 -- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd98014fcc0 msgr2=0x7fd9800ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:09.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd98014fcc0 0x7fd9800ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:09.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd98630b700 1 -- 172.21.15.158:0/1507616785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9800c49b0 con 0x7fd9800ca700 2023-10-16T11:20:09.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.898+0000 7fd985309700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd98014fcc0 0x7fd9800ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:09.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.899+0000 7fd98630b700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9800ca700 0x7fd9800c60c0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fd978000f80 tx=0x7fd97800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.899+0000 7fd9767fc700 1 -- 172.21.15.158:0/1507616785 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978003c10 con 0x7fd9800ca700 2023-10-16T11:20:09.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.899+0000 7fd9767fc700 1 -- 172.21.15.158:0/1507616785 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd978003d70 con 0x7fd9800ca700 2023-10-16T11:20:09.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.899+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9800c4c90 con 0x7fd9800ca700 2023-10-16T11:20:09.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.899+0000 7fd9767fc700 1 -- 172.21.15.158:0/1507616785 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978005310 con 0x7fd9800ca700 2023-10-16T11:20:09.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.899+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9800c59d0 con 0x7fd9800ca700 2023-10-16T11:20:09.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.900+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9800c51b0 con 0x7fd9800ca700 2023-10-16T11:20:09.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.900+0000 7fd9767fc700 1 -- 172.21.15.158:0/1507616785 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd978005470 con 0x7fd9800ca700 2023-10-16T11:20:09.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.900+0000 7fd9767fc700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd96406b140 0x7fd96406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:09.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.901+0000 7fd9767fc700 1 -- 172.21.15.158:0/1507616785 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7fd97808a0d0 con 0x7fd9800ca700 2023-10-16T11:20:09.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.901+0000 7fd985b0a700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd96406b140 0x7fd96406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:09.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.901+0000 7fd985b0a700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd96406b140 0x7fd96406d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd970001eb0 tx=0x7fd970000ed0 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:09.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:09.905+0000 7fd9767fc700 1 -- 172.21.15.158:0/1507616785 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9800c51b0 con 0x7fd9800ca700 2023-10-16T11:20:10.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.098+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9800c51b0 con 0x7fd9800ca700 2023-10-16T11:20:10.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.098+0000 7fd9767fc700 1 -- 172.21.15.158:0/1507616785 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v226) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd9800c51b0 con 0x7fd9800ca700 2023-10-16T11:20:10.102 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:10.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.100+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd96406b140 msgr2=0x7fd96406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.100+0000 7fd987d6e700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd96406b140 0x7fd96406d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd970001eb0 tx=0x7fd970000ed0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.100+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9800ca700 msgr2=0x7fd9800c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.100+0000 7fd987d6e700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9800ca700 0x7fd9800c60c0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fd978000f80 tx=0x7fd97800ee00 comp rx=0 tx=0).stop 2023-10-16T11:20:10.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 shutdown_connections 2023-10-16T11:20:10.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd96406b140 0x7fd96406d600 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9800ca700 0x7fd9800c60c0 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd96c0040b0 0x7fd9800c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 --2- 172.21.15.158:0/1507616785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd98014fcc0 0x7fd9800ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 >> 172.21.15.158:0/1507616785 conn(0x7fd980099a10 msgr2=0x7fd9800b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:10.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 shutdown_connections 2023-10-16T11:20:10.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.101+0000 7fd987d6e700 1 -- 172.21.15.158:0/1507616785 wait complete. 2023-10-16T11:20:10.115 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T11:20:10.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.236+0000 7f032832f700 1 Processor -- start 2023-10-16T11:20:10.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.236+0000 7f032832f700 1 -- start start 2023-10-16T11:20:10.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f032832f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f03200c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f032832f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f032006e540 0x7f032006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f032832f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032006c220 con 0x7f0320143790 2023-10-16T11:20:10.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f032832f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0320143b70 con 0x7f032006e540 2023-10-16T11:20:10.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f032832f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03200c5a60 con 0x7f03200c50a0 2023-10-16T11:20:10.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f03258ca700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f03200c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f03268cc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f032006e540 0x7f032006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f03260cb700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0320143790 0x7f032006bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:43882/0 (socket says 172.21.15.158:43882) 2023-10-16T11:20:10.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f03268cc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f032006e540 0x7f032006e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:40986/0 (socket says 172.21.15.158:40986) 2023-10-16T11:20:10.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f03258ca700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f03200c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:50638/0 (socket says 172.21.15.158:50638) 2023-10-16T11:20:10.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.237+0000 7f03260cb700 1 -- 172.21.15.158:0/2807253597 learned_addr learned my addr 172.21.15.158:0/2807253597 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:10.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03268cc700 1 -- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 msgr2=0x7f03200c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:10.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03268cc700 1 --2- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f03200c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03268cc700 1 -- 172.21.15.158:0/2807253597 >> v1:172.21.15.172:6789/0 conn(0x7f0320143790 legacy=0x7f032006bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03268cc700 1 -- 172.21.15.158:0/2807253597 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032006ef10 con 0x7f032006e540 2023-10-16T11:20:10.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03250c9700 1 -- 172.21.15.158:0/2807253597 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098300615 0 0) 0x7f032006c220 con 0x7f0320143790 2023-10-16T11:20:10.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03258ca700 1 --2- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f03200c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03268cc700 1 --2- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f032006e540 0x7f032006e9d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0318009d90 tx=0x7f031800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18df27ef61f174e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03250c9700 1 -- 172.21.15.158:0/2807253597 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031800e070 con 0x7f032006e540 2023-10-16T11:20:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03250c9700 1 -- 172.21.15.158:0/2807253597 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031800bb20 con 0x7f032006e540 2023-10-16T11:20:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f03250c9700 1 -- 172.21.15.158:0/2807253597 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318012430 con 0x7f032006e540 2023-10-16T11:20:10.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f032832f700 1 -- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f032006e540 msgr2=0x7f032006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.238+0000 7f032832f700 1 --2- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f032006e540 0x7f032006e9d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0318009d90 tx=0x7f031800b330 comp rx=0 tx=0).stop 2023-10-16T11:20:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.239+0000 7f032832f700 1 -- 172.21.15.158:0/2807253597 shutdown_connections 2023-10-16T11:20:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.239+0000 7f032832f700 1 --2- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f032006e540 0x7f032006e9d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.239+0000 7f032832f700 1 --2- 172.21.15.158:0/2807253597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f03200c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.239+0000 7f032832f700 1 -- 172.21.15.158:0/2807253597 >> 172.21.15.158:0/2807253597 conn(0x7f03200cc5b0 msgr2=0x7f03200cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:10.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.239+0000 7f032832f700 1 -- 172.21.15.158:0/2807253597 shutdown_connections 2023-10-16T11:20:10.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.239+0000 7f032832f700 1 -- 172.21.15.158:0/2807253597 wait complete. 2023-10-16T11:20:10.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.240+0000 7f032832f700 1 Processor -- start 2023-10-16T11:20:10.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.240+0000 7f032832f700 1 -- start start 2023-10-16T11:20:10.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.240+0000 7f032832f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f032006e540 0x7f0320157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.240+0000 7f032832f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f032013f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.240+0000 7f032832f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0320143790 0x7f03200cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.240+0000 7f032832f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320158770 con 0x7f032006e540 2023-10-16T11:20:10.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.240+0000 7f032832f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320158a50 con 0x7f0320143790 2023-10-16T11:20:10.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f032832f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0320158d30 con 0x7f03200c50a0 2023-10-16T11:20:10.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f032006e540 0x7f0320157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f032006e540 0x7f0320157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39808/0 (socket says 172.21.15.158:39808) 2023-10-16T11:20:10.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 -- 172.21.15.158:0/3434918104 learned_addr learned my addr 172.21.15.158:0/3434918104 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:10.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03258ca700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f032013f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03268cc700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0320143790 0x7f03200cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 -- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 msgr2=0x7f032013f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f032013f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 -- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0320143790 msgr2=0x7f03200cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0320143790 0x7f03200cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 -- 172.21.15.158:0/3434918104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c0094a0 con 0x7f032006e540 2023-10-16T11:20:10.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f03260cb700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f032006e540 0x7f0320157fd0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f031000c550 tx=0x7f031000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f0316ffd700 1 -- 172.21.15.158:0/3434918104 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310004c80 con 0x7f032006e540 2023-10-16T11:20:10.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f0316ffd700 1 -- 172.21.15.158:0/3434918104 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0310004e00 con 0x7f032006e540 2023-10-16T11:20:10.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0318009a70 con 0x7f032006e540 2023-10-16T11:20:10.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f0316ffd700 1 -- 172.21.15.158:0/3434918104 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310007310 con 0x7f032006e540 2023-10-16T11:20:10.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.241+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03201592f0 con 0x7f032006e540 2023-10-16T11:20:10.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.242+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0320159510 con 0x7f032006e540 2023-10-16T11:20:10.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.243+0000 7f0316ffd700 1 -- 172.21.15.158:0/3434918104 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0310007470 con 0x7f032006e540 2023-10-16T11:20:10.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.243+0000 7f0316ffd700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f030406b140 0x7f030406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.243+0000 7f0316ffd700 1 -- 172.21.15.158:0/3434918104 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7f0310089d40 con 0x7f032006e540 2023-10-16T11:20:10.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.243+0000 7f03258ca700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f030406b140 0x7f030406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.244+0000 7f03258ca700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f030406b140 0x7f030406d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f03200cedf0 tx=0x7f031c008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.248+0000 7f0316ffd700 1 -- 172.21.15.158:0/3434918104 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0310054470 con 0x7f032006e540 2023-10-16T11:20:10.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.441+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f03200681b0 con 0x7f032006e540 2023-10-16T11:20:10.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.441+0000 7f0316ffd700 1 -- 172.21.15.158:0/3434918104 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v226) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0310057a90 con 0x7f032006e540 2023-10-16T11:20:10.444 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:10.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f030406b140 msgr2=0x7f030406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f030406b140 0x7f030406d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f03200cedf0 tx=0x7f031c008040 comp rx=0 tx=0).stop 2023-10-16T11:20:10.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f032006e540 msgr2=0x7f0320157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f032006e540 0x7f0320157fd0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f031000c550 tx=0x7f031000c910 comp rx=0 tx=0).stop 2023-10-16T11:20:10.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 shutdown_connections 2023-10-16T11:20:10.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f030406b140 0x7f030406d600 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0320143790 0x7f03200cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f032006e540 0x7f0320157fd0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 --2- 172.21.15.158:0/3434918104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03200c50a0 0x7f032013f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 >> 172.21.15.158:0/3434918104 conn(0x7f03200cc5b0 msgr2=0x7f0320164e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:10.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 shutdown_connections 2023-10-16T11:20:10.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.443+0000 7f032832f700 1 -- 172.21.15.158:0/3434918104 wait complete. 2023-10-16T11:20:10.458 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:10.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.577+0000 7f1c5ae16700 1 Processor -- start 2023-10-16T11:20:10.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.577+0000 7f1c5ae16700 1 -- start start 2023-10-16T11:20:10.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.577+0000 7f1c5ae16700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c540d1d50 0x7f1c54161540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c5ae16700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c5406ba90 0x7f1c54161a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c5ae16700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c540d21e0 con 0x7f1c5415dbc0 2023-10-16T11:20:10.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c5ae16700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5415f320 con 0x7f1c5406ba90 2023-10-16T11:20:10.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c5ae16700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5415ef10 con 0x7f1c540d1d50 2023-10-16T11:20:10.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c58bb2700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1c5415dbc0 0x7f1c540cefa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:39992/0 (socket says 172.21.15.158:39992) 2023-10-16T11:20:10.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c593b3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c5406ba90 0x7f1c54161a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c58bb2700 1 -- 172.21.15.158:0/1509697185 learned_addr learned my addr 172.21.15.158:0/1509697185 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:10.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c53fff700 1 --2- 172.21.15.158:0/1509697185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c540d1d50 0x7f1c54161540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c593b3700 1 -- 172.21.15.158:0/1509697185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c540d1d50 msgr2=0x7f1c54161540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:10.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c593b3700 1 --2- 172.21.15.158:0/1509697185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c540d1d50 0x7f1c54161540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c593b3700 1 -- 172.21.15.158:0/1509697185 >> v1:172.21.15.158:6789/0 conn(0x7f1c5415dbc0 legacy=0x7f1c540cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c593b3700 1 -- 172.21.15.158:0/1509697185 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5415e150 con 0x7f1c5406ba90 2023-10-16T11:20:10.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c537fe700 1 -- 172.21.15.158:0/1509697185 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116371380 0 0) 0x7f1c540d21e0 con 0x7f1c5415dbc0 2023-10-16T11:20:10.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c593b3700 1 --2- 172.21.15.158:0/1509697185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c5406ba90 0x7f1c54161a80 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009d90 tx=0x7f1c4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e098f1472fc3012 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c537fe700 1 -- 172.21.15.158:0/1509697185 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c00e070 con 0x7f1c5406ba90 2023-10-16T11:20:10.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c537fe700 1 -- 172.21.15.158:0/1509697185 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c00bbd0 con 0x7f1c5406ba90 2023-10-16T11:20:10.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.578+0000 7f1c537fe700 1 -- 172.21.15.158:0/1509697185 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c012430 con 0x7f1c5406ba90 2023-10-16T11:20:10.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 -- 172.21.15.158:0/1509697185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c5406ba90 msgr2=0x7f1c54161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/1509697185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c5406ba90 0x7f1c54161a80 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009d90 tx=0x7f1c4c00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:10.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 -- 172.21.15.158:0/1509697185 shutdown_connections 2023-10-16T11:20:10.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/1509697185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c5406ba90 0x7f1c54161a80 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/1509697185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c540d1d50 0x7f1c54161540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 -- 172.21.15.158:0/1509697185 >> 172.21.15.158:0/1509697185 conn(0x7f1c5406a8d0 msgr2=0x7f1c5406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 -- 172.21.15.158:0/1509697185 shutdown_connections 2023-10-16T11:20:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.579+0000 7f1c5ae16700 1 -- 172.21.15.158:0/1509697185 wait complete. 2023-10-16T11:20:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.580+0000 7f1c5ae16700 1 Processor -- start 2023-10-16T11:20:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.580+0000 7f1c5ae16700 1 -- start start 2023-10-16T11:20:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c5ae16700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c5406ba90 0x7f1c541f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c5ae16700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c540c6f10 0x7f1c540c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c5ae16700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c541f35e0 0x7f1c541f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c5ae16700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540c78d0 con 0x7f1c540c6f10 2023-10-16T11:20:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c5ae16700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c541f6810 con 0x7f1c541f35e0 2023-10-16T11:20:10.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c5ae16700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c541f6af0 con 0x7f1c5406ba90 2023-10-16T11:20:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c540c6f10 0x7f1c540c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c540c6f10 0x7f1c540c7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39824/0 (socket says 172.21.15.158:39824) 2023-10-16T11:20:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 -- 172.21.15.158:0/3435163023 learned_addr learned my addr 172.21.15.158:0/3435163023 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 -- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c5406ba90 msgr2=0x7f1c541f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c58bb2700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c5406ba90 0x7f1c541f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c593b3700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c541f35e0 0x7f1c541f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c5406ba90 0x7f1c541f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 -- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c541f35e0 msgr2=0x7f1c541f3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c541f35e0 0x7f1c541f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.581+0000 7f1c53fff700 1 -- 172.21.15.158:0/3435163023 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c009a70 con 0x7f1c540c6f10 2023-10-16T11:20:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.582+0000 7f1c53fff700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c540c6f10 0x7f1c540c7390 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000d800 tx=0x7f1c4000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.582+0000 7f1c517fa700 1 -- 172.21.15.158:0/3435163023 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c400125f0 con 0x7f1c540c6f10 2023-10-16T11:20:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.582+0000 7f1c517fa700 1 -- 172.21.15.158:0/3435163023 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40012750 con 0x7f1c540c6f10 2023-10-16T11:20:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.582+0000 7f1c517fa700 1 -- 172.21.15.158:0/3435163023 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40012b00 con 0x7f1c540c6f10 2023-10-16T11:20:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.582+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c541f6dd0 con 0x7f1c540c6f10 2023-10-16T11:20:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.582+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c541f7390 con 0x7f1c540c6f10 2023-10-16T11:20:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.583+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c541f7550 con 0x7f1c540c6f10 2023-10-16T11:20:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.583+0000 7f1c517fa700 1 -- 172.21.15.158:0/3435163023 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1c40012c60 con 0x7f1c540c6f10 2023-10-16T11:20:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.584+0000 7f1c517fa700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f1c3806b140 0x7f1c3806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.584+0000 7f1c58bb2700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f1c3806b140 0x7f1c3806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.584+0000 7f1c517fa700 1 -- 172.21.15.158:0/3435163023 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7f1c4008a220 con 0x7f1c540c6f10 2023-10-16T11:20:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.584+0000 7f1c58bb2700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f1c3806b140 0x7f1c3806d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c48000f60 tx=0x7f1c48009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.588+0000 7f1c517fa700 1 -- 172.21.15.158:0/3435163023 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c400548d0 con 0x7f1c540c6f10 2023-10-16T11:20:10.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.822+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c541f7700 con 0x7f1c540c6f10 2023-10-16T11:20:10.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.823+0000 7f1c517fa700 1 -- 172.21.15.158:0/3435163023 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4938 (secure 0 0 0) 0x7f1c40057ef0 con 0x7f1c540c6f10 2023-10-16T11:20:10.826 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:10.826 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":120,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":39275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/37744519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":37744519},{"type":"v1","addr":"172.21.15.172:6834","nonce":37744519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:20:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f1c3806b140 msgr2=0x7f1c3806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f1c3806b140 0x7f1c3806d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c48000f60 tx=0x7f1c48009040 comp rx=0 tx=0).stop 2023-10-16T11:20:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c540c6f10 msgr2=0x7f1c540c7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c540c6f10 0x7f1c540c7390 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000d800 tx=0x7f1c4000dbc0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 shutdown_connections 2023-10-16T11:20:10.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f1c3806b140 0x7f1c3806d600 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c541f35e0 0x7f1c541f3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c5406ba90 0x7f1c541f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 --2- 172.21.15.158:0/3435163023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c540c6f10 0x7f1c540c7390 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 >> 172.21.15.158:0/3435163023 conn(0x7f1c5406a8d0 msgr2=0x7f1c540cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:10.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 shutdown_connections 2023-10-16T11:20:10.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.825+0000 7f1c5ae16700 1 -- 172.21.15.158:0/3435163023 wait complete. 2023-10-16T11:20:10.831 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 120 2023-10-16T11:20:10.840 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:10.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.958+0000 7fed6fffa700 1 Processor -- start 2023-10-16T11:20:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.958+0000 7fed6fffa700 1 -- start start 2023-10-16T11:20:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6fffa700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fed680cf4c0 0x7fed680ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6fffa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fed680cb660 0x7fed680cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6fffa700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed680ceb10 con 0x7fed6815a340 2023-10-16T11:20:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6fffa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed680cd600 con 0x7fed680cb660 2023-10-16T11:20:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6fffa700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed680ce140 con 0x7fed680cf4c0 2023-10-16T11:20:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6e597700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fed680cb660 0x7fed680cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6dd96700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fed6815a340 0x7fed680cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40006/0 (socket says 172.21.15.158:40006) 2023-10-16T11:20:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6dd96700 1 -- 172.21.15.158:0/538939288 learned_addr learned my addr 172.21.15.158:0/538939288 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:10.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6d595700 1 --2- 172.21.15.158:0/538939288 >> v2:172.21.15.172:3300/0 conn(0x7fed680cf4c0 0x7fed680ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6e597700 1 -- 172.21.15.158:0/538939288 >> v2:172.21.15.172:3300/0 conn(0x7fed680cf4c0 msgr2=0x7fed680ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:10.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6e597700 1 --2- 172.21.15.158:0/538939288 >> v2:172.21.15.172:3300/0 conn(0x7fed680cf4c0 0x7fed680ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6e597700 1 -- 172.21.15.158:0/538939288 >> v1:172.21.15.158:6789/0 conn(0x7fed6815a340 legacy=0x7fed680cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6e597700 1 -- 172.21.15.158:0/538939288 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed680cd810 con 0x7fed680cb660 2023-10-16T11:20:10.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6cd94700 1 -- 172.21.15.158:0/538939288 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377096168 0 0) 0x7fed64001110 con 0x7fed6815a340 2023-10-16T11:20:10.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6e597700 1 --2- 172.21.15.158:0/538939288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cb660 0x7fed680cbad0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7fed680b1610 tx=0x7fed6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b36203670228700 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.959+0000 7fed6cd94700 1 -- 172.21.15.158:0/538939288 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6000e070 con 0x7fed680cb660 2023-10-16T11:20:10.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6cd94700 1 -- 172.21.15.158:0/538939288 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6000bb20 con 0x7fed680cb660 2023-10-16T11:20:10.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6cd94700 1 -- 172.21.15.158:0/538939288 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60012430 con 0x7fed680cb660 2023-10-16T11:20:10.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 -- 172.21.15.158:0/538939288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cb660 msgr2=0x7fed680cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 --2- 172.21.15.158:0/538939288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cb660 0x7fed680cbad0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7fed680b1610 tx=0x7fed6000b330 comp rx=0 tx=0).stop 2023-10-16T11:20:10.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 -- 172.21.15.158:0/538939288 shutdown_connections 2023-10-16T11:20:10.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 --2- 172.21.15.158:0/538939288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cb660 0x7fed680cbad0 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 --2- 172.21.15.158:0/538939288 >> v2:172.21.15.172:3300/0 conn(0x7fed680cf4c0 0x7fed680ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 -- 172.21.15.158:0/538939288 >> 172.21.15.158:0/538939288 conn(0x7fed680cc4b0 msgr2=0x7fed680cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 -- 172.21.15.158:0/538939288 shutdown_connections 2023-10-16T11:20:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.960+0000 7fed6fffa700 1 -- 172.21.15.158:0/538939288 wait complete. 2023-10-16T11:20:10.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.961+0000 7fed6fffa700 1 Processor -- start 2023-10-16T11:20:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.961+0000 7fed6fffa700 1 -- start start 2023-10-16T11:20:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6fffa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed680cb660 0x7fed680c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6fffa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cf4c0 0x7fed680c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6fffa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed6815a340 0x7fed68156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6fffa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed680c9200 con 0x7fed680cf4c0 2023-10-16T11:20:10.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6fffa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed680c85f0 con 0x7fed680cb660 2023-10-16T11:20:10.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6fffa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed680c88d0 con 0x7fed6815a340 2023-10-16T11:20:10.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cf4c0 0x7fed680c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cf4c0 0x7fed680c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39838/0 (socket says 172.21.15.158:39838) 2023-10-16T11:20:10.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 -- 172.21.15.158:0/2235838133 learned_addr learned my addr 172.21.15.158:0/2235838133 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:10.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6dd96700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed680cb660 0x7fed680c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6e597700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed6815a340 0x7fed68156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 -- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed6815a340 msgr2=0x7fed68156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed6815a340 0x7fed68156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 -- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed680cb660 msgr2=0x7fed680c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:10.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed680cb660 0x7fed680c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:10.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.962+0000 7fed6d595700 1 -- 172.21.15.158:0/2235838133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60009a70 con 0x7fed680cf4c0 2023-10-16T11:20:10.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed6dd96700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed680cb660 0x7fed680c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:10.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed6e597700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed6815a340 0x7fed68156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:10.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed6d595700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cf4c0 0x7fed680c7630 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7fed5800e9b0 tx=0x7fed5800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed5e7fc700 1 -- 172.21.15.158:0/2235838133 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed58009710 con 0x7fed680cf4c0 2023-10-16T11:20:10.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed5e7fc700 1 -- 172.21.15.158:0/2235838133 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5800c850 con 0x7fed680cf4c0 2023-10-16T11:20:10.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed680c8bb0 con 0x7fed680cf4c0 2023-10-16T11:20:10.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed5e7fc700 1 -- 172.21.15.158:0/2235838133 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5800cbc0 con 0x7fed680cf4c0 2023-10-16T11:20:10.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.963+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed680c9690 con 0x7fed680cf4c0 2023-10-16T11:20:10.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.964+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed680c93b0 con 0x7fed680cf4c0 2023-10-16T11:20:10.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.964+0000 7fed5e7fc700 1 -- 172.21.15.158:0/2235838133 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fed5800cd20 con 0x7fed680cf4c0 2023-10-16T11:20:10.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.965+0000 7fed5e7fc700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fed4c06b140 0x7fed4c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:10.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.965+0000 7fed6dd96700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fed4c06b140 0x7fed4c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:10.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.965+0000 7fed5e7fc700 1 -- 172.21.15.158:0/2235838133 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7fed5808a1a0 con 0x7fed680cf4c0 2023-10-16T11:20:10.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.965+0000 7fed6dd96700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fed4c06b140 0x7fed4c06d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fed64007fd0 tx=0x7fed64009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:10.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:10.969+0000 7fed5e7fc700 1 -- 172.21.15.158:0/2235838133 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed680c93b0 con 0x7fed680cf4c0 2023-10-16T11:20:11.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.161+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed680681b0 con 0x7fed680cf4c0 2023-10-16T11:20:11.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.163+0000 7fed5e7fc700 1 -- 172.21.15.158:0/2235838133 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+18806 (secure 0 0 0) 0x7fed58054850 con 0x7fed680cf4c0 2023-10-16T11:20:11.166 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:11.166 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":226,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:09.055255+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/931576556":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2043584975":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/139807143":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6834/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:0/572486792":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1251861851":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6832/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:6832/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2783528774":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1862821458":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/417671612":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/534116803":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:11.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fed4c06b140 msgr2=0x7fed4c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fed4c06b140 0x7fed4c06d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fed64007fd0 tx=0x7fed64009040 comp rx=0 tx=0).stop 2023-10-16T11:20:11.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cf4c0 msgr2=0x7fed680c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cf4c0 0x7fed680c7630 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7fed5800e9b0 tx=0x7fed5800ed70 comp rx=0 tx=0).stop 2023-10-16T11:20:11.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 shutdown_connections 2023-10-16T11:20:11.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fed4c06b140 0x7fed4c06d600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed6815a340 0x7fed68156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed680cb660 0x7fed680c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 --2- 172.21.15.158:0/2235838133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed680cf4c0 0x7fed680c7630 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 >> 172.21.15.158:0/2235838133 conn(0x7fed680cc4b0 msgr2=0x7fed6814f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:11.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 shutdown_connections 2023-10-16T11:20:11.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.165+0000 7fed6fffa700 1 -- 172.21.15.158:0/2235838133 wait complete. 2023-10-16T11:20:11.180 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T11:20:11.181 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:11.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.299+0000 7f713d36e700 1 Processor -- start 2023-10-16T11:20:11.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.299+0000 7f713d36e700 1 -- start start 2023-10-16T11:20:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.299+0000 7f713d36e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71381599b0 0x7f71380ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.299+0000 7f713d36e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7138072250 con 0x7f7138072430 2023-10-16T11:20:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.299+0000 7f713d36e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713806d370 con 0x7f713815d7b0 2023-10-16T11:20:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.299+0000 7f713d36e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713815dc90 con 0x7f71381599b0 2023-10-16T11:20:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7136ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71381599b0 0x7f71380ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f71377fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7138072430 0x7f713806cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40022/0 (socket says 172.21.15.158:40022) 2023-10-16T11:20:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f71377fe700 1 -- 172.21.15.158:0/2138772785 learned_addr learned my addr 172.21.15.158:0/2138772785 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7136ffd700 1 -- 172.21.15.158:0/2138772785 >> v1:172.21.15.172:6789/0 conn(0x7f713815d7b0 legacy=0x7f713806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7136ffd700 1 -- 172.21.15.158:0/2138772785 >> v1:172.21.15.158:6789/0 conn(0x7f7138072430 legacy=0x7f713806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7135ffb700 1 -- 172.21.15.158:0/2138772785 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824953549 0 0) 0x7f7138072250 con 0x7f7138072430 2023-10-16T11:20:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7136ffd700 1 -- 172.21.15.158:0/2138772785 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713815df70 con 0x7f71381599b0 2023-10-16T11:20:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7136ffd700 1 --2- 172.21.15.158:0/2138772785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71381599b0 0x7f71380ca0c0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f7128009840 tx=0x7f7128009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bacaefbdd75a53b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7135ffb700 1 -- 172.21.15.158:0/2138772785 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128004230 con 0x7f71381599b0 2023-10-16T11:20:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7135ffb700 1 -- 172.21.15.158:0/2138772785 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7128004390 con 0x7f71381599b0 2023-10-16T11:20:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.300+0000 7f7135ffb700 1 -- 172.21.15.158:0/2138772785 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71280036a0 con 0x7f71381599b0 2023-10-16T11:20:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.301+0000 7f713d36e700 1 -- 172.21.15.158:0/2138772785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71381599b0 msgr2=0x7f71380ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.301+0000 7f713d36e700 1 --2- 172.21.15.158:0/2138772785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71381599b0 0x7f71380ca0c0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f7128009840 tx=0x7f7128009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.301+0000 7f713d36e700 1 -- 172.21.15.158:0/2138772785 shutdown_connections 2023-10-16T11:20:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.301+0000 7f713d36e700 1 --2- 172.21.15.158:0/2138772785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71381599b0 0x7f71380ca0c0 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.301+0000 7f713d36e700 1 -- 172.21.15.158:0/2138772785 >> 172.21.15.158:0/2138772785 conn(0x7f713806af00 msgr2=0x7f71380cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:11.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.301+0000 7f713d36e700 1 -- 172.21.15.158:0/2138772785 shutdown_connections 2023-10-16T11:20:11.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.301+0000 7f713d36e700 1 -- 172.21.15.158:0/2138772785 wait complete. 2023-10-16T11:20:11.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.302+0000 7f713d36e700 1 Processor -- start 2023-10-16T11:20:11.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.302+0000 7f713d36e700 1 -- start start 2023-10-16T11:20:11.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f713d36e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7138072430 0x7f71380b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f713d36e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71381599b0 0x7f71380b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f713d36e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f713815d7b0 0x7f713814cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f713d36e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713814e050 con 0x7f713815d7b0 2023-10-16T11:20:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f713d36e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713815bd40 con 0x7f7138072430 2023-10-16T11:20:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f713d36e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713815c020 con 0x7f71381599b0 2023-10-16T11:20:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f71377fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f713815d7b0 0x7f713814cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f71377fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f713815d7b0 0x7f713814cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39852/0 (socket says 172.21.15.158:39852) 2023-10-16T11:20:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f71377fe700 1 -- 172.21.15.158:0/3722489005 learned_addr learned my addr 172.21.15.158:0/3722489005 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f7136ffd700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7138072430 0x7f71380b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f71367fc700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71381599b0 0x7f71380b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.303+0000 7f71377fe700 1 -- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71381599b0 msgr2=0x7f71380b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f71377fe700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71381599b0 0x7f71380b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f71377fe700 1 -- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7138072430 msgr2=0x7f71380b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f71377fe700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7138072430 0x7f71380b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f71377fe700 1 -- 172.21.15.158:0/3722489005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280094a0 con 0x7f713815d7b0 2023-10-16T11:20:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f71367fc700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71381599b0 0x7f71380b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f71377fe700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f713815d7b0 0x7f713814cfc0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f713000cb60 tx=0x7f713000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f711f7fe700 1 -- 172.21.15.158:0/3722489005 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713000ab50 con 0x7f713815d7b0 2023-10-16T11:20:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f711f7fe700 1 -- 172.21.15.158:0/3722489005 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713000acb0 con 0x7f713815d7b0 2023-10-16T11:20:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713814e230 con 0x7f713815d7b0 2023-10-16T11:20:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f711f7fe700 1 -- 172.21.15.158:0/3722489005 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130007310 con 0x7f713815d7b0 2023-10-16T11:20:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.304+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713815cc20 con 0x7f713815d7b0 2023-10-16T11:20:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.305+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713815e510 con 0x7f713815d7b0 2023-10-16T11:20:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.305+0000 7f711f7fe700 1 -- 172.21.15.158:0/3722489005 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7130007470 con 0x7f713815d7b0 2023-10-16T11:20:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.306+0000 7f711f7fe700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f712406b140 0x7f712406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.306+0000 7f711f7fe700 1 -- 172.21.15.158:0/3722489005 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7f7130089e00 con 0x7f713815d7b0 2023-10-16T11:20:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.306+0000 7f7136ffd700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f712406b140 0x7f712406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.306+0000 7f7136ffd700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f712406b140 0x7f712406d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7128000c00 tx=0x7f712801c040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:11.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.310+0000 7f711f7fe700 1 -- 172.21.15.158:0/3722489005 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f713815e510 con 0x7f713815d7b0 2023-10-16T11:20:11.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.548+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f713814e750 con 0x7f713815d7b0 2023-10-16T11:20:11.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.548+0000 7f711f7fe700 1 -- 172.21.15.158:0/3722489005 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4938 (secure 0 0 0) 0x7f713814e750 con 0x7f713815d7b0 2023-10-16T11:20:11.551 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:11.551 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":120,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":39275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/37744519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":37744519},{"type":"v1","addr":"172.21.15.172:6834","nonce":37744519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:20:11.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.550+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f712406b140 msgr2=0x7f712406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.550+0000 7f713d36e700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f712406b140 0x7f712406d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7128000c00 tx=0x7f712801c040 comp rx=0 tx=0).stop 2023-10-16T11:20:11.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.550+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f713815d7b0 msgr2=0x7f713814cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.550+0000 7f713d36e700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f713815d7b0 0x7f713814cfc0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f713000cb60 tx=0x7f713000cf20 comp rx=0 tx=0).stop 2023-10-16T11:20:11.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.550+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 shutdown_connections 2023-10-16T11:20:11.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.551+0000 7f713d36e700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f712406b140 0x7f712406d600 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.551+0000 7f713d36e700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f713815d7b0 0x7f713814cfc0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.551+0000 7f713d36e700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7138072430 0x7f71380b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.551+0000 7f713d36e700 1 --2- 172.21.15.158:0/3722489005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71381599b0 0x7f71380b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.551+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 >> 172.21.15.158:0/3722489005 conn(0x7f713806af00 msgr2=0x7f7138071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:11.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.551+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 shutdown_connections 2023-10-16T11:20:11.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.551+0000 7f713d36e700 1 -- 172.21.15.158:0/3722489005 wait complete. 2023-10-16T11:20:11.556 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 120 2023-10-16T11:20:11.565 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:11.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.683+0000 7fbbbd2c9700 1 Processor -- start 2023-10-16T11:20:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.683+0000 7fbbbd2c9700 1 -- start start 2023-10-16T11:20:11.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.684+0000 7fbbbd2c9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbbb80729b0 0x7fbbb806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.684+0000 7fbbbd2c9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb8161270 con 0x7fbbb806ab40 2023-10-16T11:20:11.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.684+0000 7fbbbd2c9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb8161510 con 0x7fbbb806e5d0 2023-10-16T11:20:11.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.684+0000 7fbbbd2c9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb81617f0 con 0x7fbbb80729b0 2023-10-16T11:20:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.684+0000 7fbbb6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbbb806ab40 0x7fbbb80722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40032/0 (socket says 172.21.15.158:40032) 2023-10-16T11:20:11.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.684+0000 7fbbb6ffd700 1 -- 172.21.15.158:0/1717762024 learned_addr learned my addr 172.21.15.158:0/1717762024 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:11.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb77fe700 1 --2- 172.21.15.158:0/1717762024 >> v2:172.21.15.172:3300/0 conn(0x7fbbb80729b0 0x7fbbb806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002457205 0 0) 0x7fbbb8161270 con 0x7fbbb806ab40 2023-10-16T11:20:11.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba00035b0 con 0x7fbbb806ab40 2023-10-16T11:20:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2740123486 0 0) 0x7fbba00035b0 con 0x7fbbb806ab40 2023-10-16T11:20:11.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 >> v2:172.21.15.172:3300/0 conn(0x7fbbb80729b0 msgr2=0x7fbbb806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 --2- 172.21.15.158:0/1717762024 >> v2:172.21.15.172:3300/0 conn(0x7fbbb80729b0 0x7fbbb806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 >> v1:172.21.15.172:6789/0 conn(0x7fbbb806e5d0 legacy=0x7fbbb815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8161ad0 con 0x7fbbb806ab40 2023-10-16T11:20:11.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fbbac001670 con 0x7fbbb806ab40 2023-10-16T11:20:11.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 >> v1:172.21.15.158:6789/0 conn(0x7fbbb806ab40 legacy=0x7fbbb80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba0003a00 0x7fbba0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0004850 0x7fbba0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba0005210 0x7fbba000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb81617f0 con 0x7fbba0003a00 2023-10-16T11:20:11.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba000c060 con 0x7fbba0005210 2023-10-16T11:20:11.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba000c220 con 0x7fbba0004850 2023-10-16T11:20:11.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba0003a00 0x7fbba0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb77fe700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba0005210 0x7fbba000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb67fc700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0004850 0x7fbba0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb6ffd700 1 -- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0004850 msgr2=0x7fbba0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0004850 0x7fbba0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb6ffd700 1 -- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba0005210 msgr2=0x7fbba000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba0005210 0x7fbba000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb6ffd700 1 -- 172.21.15.158:0/1717762024 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba00035b0 con 0x7fbba0003a00 2023-10-16T11:20:11.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.685+0000 7fbbb77fe700 1 -- 172.21.15.158:0/1717762024 reap_dead start 2023-10-16T11:20:11.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba0003a00 0x7fbba0003de0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00c210 tx=0x7fbba0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef30118bbce33c4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac007b70 con 0x7fbba0003a00 2023-10-16T11:20:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac00cd70 con 0x7fbba0003a00 2023-10-16T11:20:11.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbb5ffb700 1 -- 172.21.15.158:0/1717762024 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac005ba0 con 0x7fbba0003a00 2023-10-16T11:20:11.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/1717762024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba0003a00 msgr2=0x7fbba0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba0003a00 0x7fbba0003de0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00c210 tx=0x7fbba0005880 comp rx=0 tx=0).stop 2023-10-16T11:20:11.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/1717762024 shutdown_connections 2023-10-16T11:20:11.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/1717762024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba0003a00 0x7fbba0003de0 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/1717762024 >> 172.21.15.158:0/1717762024 conn(0x7fbbb80d02b0 msgr2=0x7fbbb80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:11.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/1717762024 shutdown_connections 2023-10-16T11:20:11.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.686+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/1717762024 wait complete. 2023-10-16T11:20:11.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 Processor -- start 2023-10-16T11:20:11.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 -- start start 2023-10-16T11:20:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0003a00 0x7fbbb8154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbb80b5530 0x7fbbb80b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbb80c3c80 0x7fbbb80c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb815efd0 con 0x7fbbb80b5530 2023-10-16T11:20:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80b4490 con 0x7fbbb80c3c80 2023-10-16T11:20:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbbd2c9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80b4770 con 0x7fbba0003a00 2023-10-16T11:20:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.688+0000 7fbbb67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbb80b5530 0x7fbbb80b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbb80b5530 0x7fbbb80b59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39870/0 (socket says 172.21.15.158:39870) 2023-10-16T11:20:11.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 -- 172.21.15.158:0/3715885344 learned_addr learned my addr 172.21.15.158:0/3715885344 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:11.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb77fe700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbb80c3c80 0x7fbbb80c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0003a00 0x7fbbb8154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 -- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0003a00 msgr2=0x7fbbb8154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0003a00 0x7fbbb8154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 -- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbb80c3c80 msgr2=0x7fbbb80c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbb80c3c80 0x7fbbb80c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 -- 172.21.15.158:0/3715885344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb80b4a50 con 0x7fbbb80b5530 2023-10-16T11:20:11.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0003a00 0x7fbbb8154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:11.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbb67fc700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbb80b5530 0x7fbbb80b59b0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7fbba800e7b0 tx=0x7fbba800eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:11.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/3715885344 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80052e0 con 0x7fbbb80b5530 2023-10-16T11:20:11.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/3715885344 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba8005460 con 0x7fbbb80b5530 2023-10-16T11:20:11.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb80b4d30 con 0x7fbbb80b5530 2023-10-16T11:20:11.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/3715885344 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80038c0 con 0x7fbbb80b5530 2023-10-16T11:20:11.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.689+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb80b4030 con 0x7fbbb80b5530 2023-10-16T11:20:11.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.690+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb80b5250 con 0x7fbbb80b5530 2023-10-16T11:20:11.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.691+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/3715885344 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbba8003a20 con 0x7fbbb80b5530 2023-10-16T11:20:11.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.691+0000 7fbb9f7fe700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fbba406b140 0x7fbba406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:11.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.691+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/3715885344 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7fbba8089970 con 0x7fbbb80b5530 2023-10-16T11:20:11.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.691+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fbba406b140 0x7fbba406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:11.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.692+0000 7fbbb6ffd700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fbba406b140 0x7fbba406d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80cab60 tx=0x7fbbac007000 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:11.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.696+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/3715885344 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbb80b5250 con 0x7fbbb80b5530 2023-10-16T11:20:11.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.932+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbb80681b0 con 0x7fbbb80b5530 2023-10-16T11:20:11.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.933+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/3715885344 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4938 (secure 0 0 0) 0x7fbba8019090 con 0x7fbbb80b5530 2023-10-16T11:20:11.936 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:11.936 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":120,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":39275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/37744519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":37744519},{"type":"v1","addr":"172.21.15.172:6834","nonce":37744519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:19:36.061742+0000","modified":"2023-10-16T11:19:39.089874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11586":{"gid":11586,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/2771711373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":2771711373},{"type":"v1","addr":"172.21.15.172:6835","nonce":2771711373}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T11:20:11.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fbba406b140 msgr2=0x7fbba406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fbba406b140 0x7fbba406d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80cab60 tx=0x7fbbac007000 comp rx=0 tx=0).stop 2023-10-16T11:20:11.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbb80b5530 msgr2=0x7fbbb80b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:11.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbb80b5530 0x7fbbb80b59b0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7fbba800e7b0 tx=0x7fbba800eb70 comp rx=0 tx=0).stop 2023-10-16T11:20:11.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 shutdown_connections 2023-10-16T11:20:11.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fbba406b140 0x7fbba406d600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbb80c3c80 0x7fbbb80c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbba0003a00 0x7fbbb8154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 --2- 172.21.15.158:0/3715885344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbbb80b5530 0x7fbbb80b59b0 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:11.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 >> 172.21.15.158:0/3715885344 conn(0x7fbbb80d02b0 msgr2=0x7fbbb80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:11.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 shutdown_connections 2023-10-16T11:20:11.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:11.935+0000 7fbbbd2c9700 1 -- 172.21.15.158:0/3715885344 wait complete. 2023-10-16T11:20:11.941 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 120 2023-10-16T11:20:11.950 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:12.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.069+0000 7f2732f94700 1 Processor -- start 2023-10-16T11:20:12.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.069+0000 7f2732f94700 1 -- start start 2023-10-16T11:20:12.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.069+0000 7f2732f94700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.069+0000 7f2732f94700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c06e770 con 0x7f272c15dfb0 2023-10-16T11:20:12.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.069+0000 7f2732f94700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c0cbe00 con 0x7f272c0cc5d0 2023-10-16T11:20:12.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.069+0000 7f2732f94700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272c0cf270 con 0x7f272c06fa50 2023-10-16T11:20:12.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f272bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f272c15dfb0 0x7f272c06f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40052/0 (socket says 172.21.15.158:40052) 2023-10-16T11:20:12.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f272bfff700 1 -- 172.21.15.158:0/192671467 learned_addr learned my addr 172.21.15.158:0/192671467 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:12.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f2731531700 1 --2- 172.21.15.158:0/192671467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f272b7fe700 1 -- 172.21.15.158:0/192671467 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523380608 0 0) 0x7f2718001110 con 0x7f272c15dfb0 2023-10-16T11:20:12.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f272b7fe700 1 -- 172.21.15.158:0/192671467 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f271c0035b0 con 0x7f272c15dfb0 2023-10-16T11:20:12.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f2731531700 1 -- 172.21.15.158:0/192671467 >> v1:172.21.15.172:6789/0 conn(0x7f272c0cc5d0 legacy=0x7f272c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f2731531700 1 -- 172.21.15.158:0/192671467 >> v1:172.21.15.158:6789/0 conn(0x7f272c15dfb0 legacy=0x7f272c06f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f2731531700 1 -- 172.21.15.158:0/192671467 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c15e4e0 con 0x7f272c06fa50 2023-10-16T11:20:12.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.070+0000 7f2731531700 1 --2- 172.21.15.158:0/192671467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c06e230 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2724009dc0 tx=0x7f272400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=772bce525d3f9dd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f272b7fe700 1 -- 172.21.15.158:0/192671467 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272400e070 con 0x7f272c06fa50 2023-10-16T11:20:12.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f272b7fe700 1 -- 172.21.15.158:0/192671467 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272400bb20 con 0x7f272c06fa50 2023-10-16T11:20:12.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f272b7fe700 1 -- 172.21.15.158:0/192671467 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724012430 con 0x7f272c06fa50 2023-10-16T11:20:12.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f2732f94700 1 -- 172.21.15.158:0/192671467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 msgr2=0x7f272c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f2732f94700 1 --2- 172.21.15.158:0/192671467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c06e230 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2724009dc0 tx=0x7f272400b330 comp rx=0 tx=0).stop 2023-10-16T11:20:12.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f2732f94700 1 -- 172.21.15.158:0/192671467 shutdown_connections 2023-10-16T11:20:12.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f2732f94700 1 --2- 172.21.15.158:0/192671467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c06e230 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f2732f94700 1 -- 172.21.15.158:0/192671467 >> 172.21.15.158:0/192671467 conn(0x7f272c0d02b0 msgr2=0x7f272c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:12.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f2732f94700 1 -- 172.21.15.158:0/192671467 shutdown_connections 2023-10-16T11:20:12.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.071+0000 7f2732f94700 1 -- 172.21.15.158:0/192671467 wait complete. 2023-10-16T11:20:12.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.072+0000 7f2732f94700 1 Processor -- start 2023-10-16T11:20:12.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2732f94700 1 -- start start 2023-10-16T11:20:12.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2732f94700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c0b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2732f94700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c0cc5d0 0x7f272c06af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2732f94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15dfb0 0x7f272c0b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2732f94700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c15d8d0 con 0x7f272c15dfb0 2023-10-16T11:20:12.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2732f94700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c06aa90 con 0x7f272c0cc5d0 2023-10-16T11:20:12.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2732f94700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272c0b1340 con 0x7f272c06fa50 2023-10-16T11:20:12.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.073+0000 7f2731531700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15dfb0 0x7f272c0b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15dfb0 0x7f272c0b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39886/0 (socket says 172.21.15.158:39886) 2023-10-16T11:20:12.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 -- 172.21.15.158:0/2838651694 learned_addr learned my addr 172.21.15.158:0/2838651694 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f272bfff700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c0cc5d0 0x7f272c06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2730d30700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c0b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 -- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 msgr2=0x7f272c0b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c0b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 -- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c0cc5d0 msgr2=0x7f272c06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c0cc5d0 0x7f272c06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 -- 172.21.15.158:0/2838651694 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724009a70 con 0x7f272c15dfb0 2023-10-16T11:20:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2731531700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15dfb0 0x7f272c0b5070 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f272400b680 tx=0x7f2724004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f27297fa700 1 -- 172.21.15.158:0/2838651694 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724012430 con 0x7f272c15dfb0 2023-10-16T11:20:12.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c1504b0 con 0x7f272c15dfb0 2023-10-16T11:20:12.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c15d310 con 0x7f272c15dfb0 2023-10-16T11:20:12.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f27297fa700 1 -- 172.21.15.158:0/2838651694 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2724020070 con 0x7f272c15dfb0 2023-10-16T11:20:12.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.074+0000 7f27297fa700 1 -- 172.21.15.158:0/2838651694 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27240098f0 con 0x7f272c15dfb0 2023-10-16T11:20:12.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.075+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272c0681b0 con 0x7f272c15dfb0 2023-10-16T11:20:12.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.076+0000 7f27297fa700 1 -- 172.21.15.158:0/2838651694 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2724004040 con 0x7f272c15dfb0 2023-10-16T11:20:12.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.076+0000 7f27297fa700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f271006b140 0x7f271006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.076+0000 7f2730d30700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f271006b140 0x7f271006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.076+0000 7f2730d30700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f271006b140 0x7f271006d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2720000f60 tx=0x7f2720009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.077+0000 7f27297fa700 1 -- 172.21.15.158:0/2838651694 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7f2724022070 con 0x7f272c15dfb0 2023-10-16T11:20:12.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.081+0000 7f27297fa700 1 -- 172.21.15.158:0/2838651694 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2724056b70 con 0x7f272c15dfb0 2023-10-16T11:20:12.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.276+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f272c06abd0 con 0x7f272c15dfb0 2023-10-16T11:20:12.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.277+0000 7f27297fa700 1 -- 172.21.15.158:0/2838651694 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+18806 (secure 0 0 0) 0x7f272405a190 con 0x7f272c15dfb0 2023-10-16T11:20:12.280 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:12.280 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":226,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:09.055255+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:19:33.715104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:19:35.032906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/931576556":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2043584975":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/139807143":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6834/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:0/572486792":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1251861851":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6832/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:6832/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2783528774":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1862821458":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/417671612":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/534116803":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:12.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.279+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f271006b140 msgr2=0x7f271006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.279+0000 7f2732f94700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f271006b140 0x7f271006d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2720000f60 tx=0x7f2720009040 comp rx=0 tx=0).stop 2023-10-16T11:20:12.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.279+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15dfb0 msgr2=0x7f272c0b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.279+0000 7f2732f94700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15dfb0 0x7f272c0b5070 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f272400b680 tx=0x7f2724004ad0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 shutdown_connections 2023-10-16T11:20:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f271006b140 0x7f271006d600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f272c15dfb0 0x7f272c0b5070 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f272c06fa50 0x7f272c0b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 --2- 172.21.15.158:0/2838651694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f272c0cc5d0 0x7f272c06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 >> 172.21.15.158:0/2838651694 conn(0x7f272c0d02b0 msgr2=0x7f272c0c6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 shutdown_connections 2023-10-16T11:20:12.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.280+0000 7f2732f94700 1 -- 172.21.15.158:0/2838651694 wait complete. 2023-10-16T11:20:12.295 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T11:20:12.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.414+0000 7fa22a20e700 1 Processor -- start 2023-10-16T11:20:12.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.414+0000 7fa22a20e700 1 -- start start 2023-10-16T11:20:12.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.414+0000 7fa22a20e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa22406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.414+0000 7fa22a20e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa2240722e0 0x7fa2240cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.414+0000 7fa22a20e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240cb2d0 con 0x7fa22415e210 2023-10-16T11:20:12.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.414+0000 7fa22a20e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22415bc40 con 0x7fa2240722e0 2023-10-16T11:20:12.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.414+0000 7fa22a20e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa224158660 con 0x7fa224158280 2023-10-16T11:20:12.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa2237fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa22406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa222ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa2240722e0 0x7fa2240cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa223fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa22415e210 0x7fa2240cb740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:43974/0 (socket says 172.21.15.158:43974) 2023-10-16T11:20:12.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa223fff700 1 -- 172.21.15.158:0/3328948213 learned_addr learned my addr 172.21.15.158:0/3328948213 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:12.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa2227fc700 1 -- 172.21.15.158:0/3328948213 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455518214 0 0) 0x7fa2240cb2d0 con 0x7fa22415e210 2023-10-16T11:20:12.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa2237fe700 1 -- 172.21.15.158:0/3328948213 >> v2:172.21.15.172:3300/0 conn(0x7fa2240722e0 msgr2=0x7fa2240cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:12.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa2237fe700 1 --2- 172.21.15.158:0/3328948213 >> v2:172.21.15.172:3300/0 conn(0x7fa2240722e0 0x7fa2240cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa2237fe700 1 -- 172.21.15.158:0/3328948213 >> v1:172.21.15.172:6789/0 conn(0x7fa22415e210 legacy=0x7fa2240cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa2237fe700 1 -- 172.21.15.158:0/3328948213 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22415b030 con 0x7fa224158280 2023-10-16T11:20:12.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.415+0000 7fa2237fe700 1 --2- 172.21.15.158:0/3328948213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa22406a8d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa214009840 tx=0x7fa214009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da5d593be3a8efd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa2227fc700 1 -- 172.21.15.158:0/3328948213 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214011830 con 0x7fa224158280 2023-10-16T11:20:12.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa2227fc700 1 -- 172.21.15.158:0/3328948213 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa214011990 con 0x7fa224158280 2023-10-16T11:20:12.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa2227fc700 1 -- 172.21.15.158:0/3328948213 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214011d20 con 0x7fa224158280 2023-10-16T11:20:12.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 -- 172.21.15.158:0/3328948213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 msgr2=0x7fa22406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 --2- 172.21.15.158:0/3328948213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa22406a8d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa214009840 tx=0x7fa214009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:12.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 -- 172.21.15.158:0/3328948213 shutdown_connections 2023-10-16T11:20:12.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 --2- 172.21.15.158:0/3328948213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa22406a8d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 --2- 172.21.15.158:0/3328948213 >> v2:172.21.15.172:3300/0 conn(0x7fa2240722e0 0x7fa2240cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 -- 172.21.15.158:0/3328948213 >> 172.21.15.158:0/3328948213 conn(0x7fa2240cc700 msgr2=0x7fa2240ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:12.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 -- 172.21.15.158:0/3328948213 shutdown_connections 2023-10-16T11:20:12.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.416+0000 7fa22a20e700 1 -- 172.21.15.158:0/3328948213 wait complete. 2023-10-16T11:20:12.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.417+0000 7fa22a20e700 1 Processor -- start 2023-10-16T11:20:12.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa22a20e700 1 -- start start 2023-10-16T11:20:12.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa22a20e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2240722e0 0x7fa2240aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa22a20e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa2240cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa22a20e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22415e210 0x7fa2240cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa22a20e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241f3720 con 0x7fa22415e210 2023-10-16T11:20:12.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa22a20e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241f3a00 con 0x7fa2240722e0 2023-10-16T11:20:12.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa22a20e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241f3ce0 con 0x7fa224158280 2023-10-16T11:20:12.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.418+0000 7fa223fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22415e210 0x7fa2240cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22415e210 0x7fa2240cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39900/0 (socket says 172.21.15.158:39900) 2023-10-16T11:20:12.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 -- 172.21.15.158:0/20621263 learned_addr learned my addr 172.21.15.158:0/20621263 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:12.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa2237fe700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2240722e0 0x7fa2240aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa222ffd700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa2240cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 -- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 msgr2=0x7fa2240cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa2240cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 -- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2240722e0 msgr2=0x7fa2240aa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2240722e0 0x7fa2240aa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 -- 172.21.15.158:0/20621263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140094a0 con 0x7fa22415e210 2023-10-16T11:20:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa223fff700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22415e210 0x7fa2240cdcd0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00cb30 tx=0x7fa21c00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa20bfff700 1 -- 172.21.15.158:0/20621263 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c00ab50 con 0x7fa22415e210 2023-10-16T11:20:12.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa20bfff700 1 -- 172.21.15.158:0/20621263 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c00acb0 con 0x7fa22415e210 2023-10-16T11:20:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2241f3fc0 con 0x7fa22415e210 2023-10-16T11:20:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa20bfff700 1 -- 172.21.15.158:0/20621263 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c007310 con 0x7fa22415e210 2023-10-16T11:20:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.419+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2241f34b0 con 0x7fa22415e210 2023-10-16T11:20:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.420+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2240681b0 con 0x7fa22415e210 2023-10-16T11:20:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.421+0000 7fa20bfff700 1 -- 172.21.15.158:0/20621263 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa21c007470 con 0x7fa22415e210 2023-10-16T11:20:12.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.421+0000 7fa20bfff700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa21006b140 0x7fa21006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.421+0000 7fa20bfff700 1 -- 172.21.15.158:0/20621263 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7424+0+0 (secure 0 0 0) 0x7fa21c089b40 con 0x7fa22415e210 2023-10-16T11:20:12.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.421+0000 7fa2237fe700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa21006b140 0x7fa21006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.422+0000 7fa2237fe700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa21006b140 0x7fa21006d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa214000c00 tx=0x7fa214008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.426+0000 7fa20bfff700 1 -- 172.21.15.158:0/20621263 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa21c054160 con 0x7fa22415e210 2023-10-16T11:20:12.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.665+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fa224159910 con 0x7fa22415e210 2023-10-16T11:20:12.668 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:20:12.665+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:20:12.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.667+0000 7fa20bfff700 1 -- 172.21.15.158:0/20621263 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v121) v1 ==== 167+0+0 (secure 0 0 0) 0x7fa21c057780 con 0x7fa22415e210 2023-10-16T11:20:12.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa21006b140 msgr2=0x7fa21006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa21006b140 0x7fa21006d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa214000c00 tx=0x7fa214008030 comp rx=0 tx=0).stop 2023-10-16T11:20:12.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22415e210 msgr2=0x7fa2240cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22415e210 0x7fa2240cdcd0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00cb30 tx=0x7fa21c00cef0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 shutdown_connections 2023-10-16T11:20:12.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa21006b140 0x7fa21006d600 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22415e210 0x7fa2240cdcd0 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2240722e0 0x7fa2240aa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 --2- 172.21.15.158:0/20621263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa224158280 0x7fa2240cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.669+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 >> 172.21.15.158:0/20621263 conn(0x7fa2240cc700 msgr2=0x7fa2241587d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:12.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.670+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 shutdown_connections 2023-10-16T11:20:12.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.670+0000 7fa22a20e700 1 -- 172.21.15.158:0/20621263 wait complete. 2023-10-16T11:20:12.675 INFO:tasks.ceph.mds.d.smithi172.stderr: -1801> 2023-10-16T11:19:40.306+0000 7fcfe2aca700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:20:12.675 INFO:teuthology.orchestra.run.smithi158.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:20:12.684 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T11:20:12.702 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:20:12.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.807+0000 7f17ae7a0700 1 Processor -- start 2023-10-16T11:20:12.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.807+0000 7f17ae7a0700 1 -- start start 2023-10-16T11:20:12.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17ae7a0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17a80cf010 0x7f17a815e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17ae7a0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a815ecc0 con 0x7f17a806d430 2023-10-16T11:20:12.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17ae7a0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a8161400 con 0x7f17a815eec0 2023-10-16T11:20:12.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17ae7a0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a81616e0 con 0x7f17a80cf010 2023-10-16T11:20:12.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17a7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17a80cf010 0x7f17a815e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17acd3d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f17a806d430 0x7f17a80cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40080/0 (socket says 172.21.15.158:40080) 2023-10-16T11:20:12.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17a7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17a80cf010 0x7f17a815e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39904/0 (socket says 172.21.15.158:39904) 2023-10-16T11:20:12.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17acd3d700 1 -- 172.21.15.158:0/351162175 learned_addr learned my addr 172.21.15.158:0/351162175 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:12.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17a7fff700 1 -- 172.21.15.158:0/351162175 >> v1:172.21.15.172:6789/0 conn(0x7f17a815eec0 legacy=0x7f17a80cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17a7fff700 1 -- 172.21.15.158:0/351162175 >> v1:172.21.15.158:6789/0 conn(0x7f17a806d430 legacy=0x7f17a80cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17a7fff700 1 -- 172.21.15.158:0/351162175 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a81619c0 con 0x7f17a80cf010 2023-10-16T11:20:12.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.808+0000 7f17a6ffd700 1 -- 172.21.15.158:0/351162175 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592864160 0 0) 0x7f17a815ecc0 con 0x7f17a806d430 2023-10-16T11:20:12.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17a7fff700 1 --2- 172.21.15.158:0/351162175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a80cf010 0x7f17a815e780 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f1798009870 tx=0x7f1798009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9a9dfbf98bf996 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17a6ffd700 1 -- 172.21.15.158:0/351162175 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798011830 con 0x7f17a80cf010 2023-10-16T11:20:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17a6ffd700 1 -- 172.21.15.158:0/351162175 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1798011990 con 0x7f17a80cf010 2023-10-16T11:20:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17a6ffd700 1 -- 172.21.15.158:0/351162175 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798011d20 con 0x7f17a80cf010 2023-10-16T11:20:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17ae7a0700 1 -- 172.21.15.158:0/351162175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a80cf010 msgr2=0x7f17a815e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/351162175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a80cf010 0x7f17a815e780 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f1798009870 tx=0x7f1798009b80 comp rx=0 tx=0).stop 2023-10-16T11:20:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17ae7a0700 1 -- 172.21.15.158:0/351162175 shutdown_connections 2023-10-16T11:20:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/351162175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a80cf010 0x7f17a815e780 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17ae7a0700 1 -- 172.21.15.158:0/351162175 >> 172.21.15.158:0/351162175 conn(0x7f17a806a8d0 msgr2=0x7f17a806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17ae7a0700 1 -- 172.21.15.158:0/351162175 shutdown_connections 2023-10-16T11:20:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.809+0000 7f17ae7a0700 1 -- 172.21.15.158:0/351162175 wait complete. 2023-10-16T11:20:12.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.810+0000 7f17ae7a0700 1 Processor -- start 2023-10-16T11:20:12.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.811+0000 7f17ae7a0700 1 -- start start 2023-10-16T11:20:12.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.811+0000 7f17ae7a0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a806d430 0x7f17a80b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.811+0000 7f17ae7a0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a80cf010 0x7f17a8143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.811+0000 7f17ae7a0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a815eec0 0x7f17a81438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.811+0000 7f17ae7a0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80b5e20 con 0x7f17a815eec0 2023-10-16T11:20:12.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.811+0000 7f17ae7a0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80b5250 con 0x7f17a806d430 2023-10-16T11:20:12.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.811+0000 7f17ae7a0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80b54f0 con 0x7f17a80cf010 2023-10-16T11:20:12.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a815eec0 0x7f17a81438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a815eec0 0x7f17a81438a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39912/0 (socket says 172.21.15.158:39912) 2023-10-16T11:20:12.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 -- 172.21.15.158:0/1348573176 learned_addr learned my addr 172.21.15.158:0/1348573176 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:12.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17a7fff700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a806d430 0x7f17a80b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17a77fe700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a80cf010 0x7f17a8143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 -- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a80cf010 msgr2=0x7f17a8143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a80cf010 0x7f17a8143360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 -- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a806d430 msgr2=0x7f17a80b62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:12.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a806d430 0x7f17a80b62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:12.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 -- 172.21.15.158:0/1348573176 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17980094a0 con 0x7f17a815eec0 2023-10-16T11:20:12.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17a77fe700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a80cf010 0x7f17a8143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:12.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17acd3d700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a815eec0 0x7f17a81438a0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f17a000cba0 tx=0x7f17a000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17a4ff9700 1 -- 172.21.15.158:0/1348573176 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a000aad0 con 0x7f17a815eec0 2023-10-16T11:20:12.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17a4ff9700 1 -- 172.21.15.158:0/1348573176 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a000ac50 con 0x7f17a815eec0 2023-10-16T11:20:12.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a80b57d0 con 0x7f17a815eec0 2023-10-16T11:20:12.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17a4ff9700 1 -- 172.21.15.158:0/1348573176 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00070d0 con 0x7f17a815eec0 2023-10-16T11:20:12.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.812+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a80b67f0 con 0x7f17a815eec0 2023-10-16T11:20:12.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.813+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a80b5fd0 con 0x7f17a815eec0 2023-10-16T11:20:12.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.814+0000 7f17a4ff9700 1 -- 172.21.15.158:0/1348573176 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f17a0007230 con 0x7f17a815eec0 2023-10-16T11:20:12.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.814+0000 7f17a4ff9700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f179406b140 0x7f179406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:12.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.814+0000 7f17a4ff9700 1 -- 172.21.15.158:0/1348573176 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7510+0+0 (secure 0 0 0) 0x7f17a0089e20 con 0x7f17a815eec0 2023-10-16T11:20:12.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.814+0000 7f17a7fff700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f179406b140 0x7f179406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:12.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.815+0000 7f17a7fff700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f179406b140 0x7f179406d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17980090b0 tx=0x7f179801c040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:12.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:12.819+0000 7f17a4ff9700 1 -- 172.21.15.158:0/1348573176 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17a80b5fd0 con 0x7f17a815eec0 2023-10-16T11:20:13.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.057+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f17a80b6a10 con 0x7f17a815eec0 2023-10-16T11:20:13.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.668+0000 7f17a4ff9700 1 -- 172.21.15.158:0/1348573176 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v122) v1 ==== 107+0+0 (secure 0 0 0) 0x7f17a0018020 con 0x7f17a815eec0 2023-10-16T11:20:13.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f179406b140 msgr2=0x7f179406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f179406b140 0x7f179406d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17980090b0 tx=0x7f179801c040 comp rx=0 tx=0).stop 2023-10-16T11:20:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a815eec0 msgr2=0x7f17a81438a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a815eec0 0x7f17a81438a0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f17a000cba0 tx=0x7f17a000ceb0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 shutdown_connections 2023-10-16T11:20:13.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f179406b140 0x7f179406d600 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17a815eec0 0x7f17a81438a0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a806d430 0x7f17a80b62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 --2- 172.21.15.158:0/1348573176 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a80cf010 0x7f17a8143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 >> 172.21.15.158:0/1348573176 conn(0x7f17a806a8d0 msgr2=0x7f17a806c830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:13.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 shutdown_connections 2023-10-16T11:20:13.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.671+0000 7f17ae7a0700 1 -- 172.21.15.158:0/1348573176 wait complete. 2023-10-16T11:20:13.687 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T11:20:13.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.813+0000 7fe4f6500700 1 Processor -- start 2023-10-16T11:20:13.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.813+0000 7fe4f6500700 1 -- start start 2023-10-16T11:20:13.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4f6500700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4f00ce510 0x7fe4f015f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:13.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4f6500700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe4f00cbaf0 0x7fe4f00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:13.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4f6500700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00cf4c0 con 0x7fe4f015b590 2023-10-16T11:20:13.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4f6500700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f00cc4c0 con 0x7fe4f00ce510 2023-10-16T11:20:13.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4f6500700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f00b49e0 con 0x7fe4f00cbaf0 2023-10-16T11:20:13.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4f00ce510 0x7fe4f015f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:13.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4f00ce510 0x7fe4f015f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39920/0 (socket says 172.21.15.158:39920) 2023-10-16T11:20:13.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/401257751 learned_addr learned my addr 172.21.15.158:0/401257751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:13.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4f4a9d700 1 --2- 172.21.15.158:0/401257751 >> v2:172.21.15.172:3300/0 conn(0x7fe4f00cbaf0 0x7fe4f00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:13.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/401257751 >> v2:172.21.15.172:3300/0 conn(0x7fe4f00cbaf0 msgr2=0x7fe4f00cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:13.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 --2- 172.21.15.158:0/401257751 >> v2:172.21.15.172:3300/0 conn(0x7fe4f00cbaf0 0x7fe4f00cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/401257751 >> v1:172.21.15.172:6789/0 conn(0x7fe4f015b590 legacy=0x7fe4f00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:13.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/401257751 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f015f710 con 0x7fe4f00ce510 2023-10-16T11:20:13.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.814+0000 7fe4ef7fe700 1 --2- 172.21.15.158:0/401257751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f015f1d0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00cf430 tx=0x7fe4e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e2af1f9328d44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:13.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4eeffd700 1 -- 172.21.15.158:0/401257751 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e000b860 con 0x7fe4f00ce510 2023-10-16T11:20:13.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4eeffd700 1 -- 172.21.15.158:0/401257751 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e000b9c0 con 0x7fe4f00ce510 2023-10-16T11:20:13.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4eeffd700 1 -- 172.21.15.158:0/401257751 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e000bd50 con 0x7fe4f00ce510 2023-10-16T11:20:13.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 -- 172.21.15.158:0/401257751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 msgr2=0x7fe4f015f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:13.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 --2- 172.21.15.158:0/401257751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f015f1d0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00cf430 tx=0x7fe4e0009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:13.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 -- 172.21.15.158:0/401257751 shutdown_connections 2023-10-16T11:20:13.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 --2- 172.21.15.158:0/401257751 >> v2:172.21.15.172:3300/0 conn(0x7fe4f00cbaf0 0x7fe4f00cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 --2- 172.21.15.158:0/401257751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f015f1d0 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 -- 172.21.15.158:0/401257751 >> 172.21.15.158:0/401257751 conn(0x7fe4f006a8d0 msgr2=0x7fe4f006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:13.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 -- 172.21.15.158:0/401257751 shutdown_connections 2023-10-16T11:20:13.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.815+0000 7fe4f6500700 1 -- 172.21.15.158:0/401257751 wait complete. 2023-10-16T11:20:13.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.816+0000 7fe4f6500700 1 Processor -- start 2023-10-16T11:20:13.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4f6500700 1 -- start start 2023-10-16T11:20:13.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4f6500700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4f00cbaf0 0x7fe4f0150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:13.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4f6500700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:13.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4f6500700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4f015b590 0x7fe4f00c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:13.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4f6500700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f006d4b0 con 0x7fe4f00ce510 2023-10-16T11:20:13.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4f6500700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f006c410 con 0x7fe4f015b590 2023-10-16T11:20:13.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4f6500700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f006c6f0 con 0x7fe4f00cbaf0 2023-10-16T11:20:13.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4ef7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:13.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4ef7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f006b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39926/0 (socket says 172.21.15.158:39926) 2023-10-16T11:20:13.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.817+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/3541490252 learned_addr learned my addr 172.21.15.158:0/3541490252 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:13.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4f4a9d700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4f015b590 0x7fe4f00c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:13.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4effff700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4f00cbaf0 0x7fe4f0150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:13.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4f00cbaf0 msgr2=0x7fe4f0150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:13.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ef7fe700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4f00cbaf0 0x7fe4f0150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4f015b590 msgr2=0x7fe4f00c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:13.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ef7fe700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4f015b590 0x7fe4f00c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:13.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ef7fe700 1 -- 172.21.15.158:0/3541490252 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0009550 con 0x7fe4f00ce510 2023-10-16T11:20:13.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ef7fe700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f006b9e0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009c40 tx=0x7fe4e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:13.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ecff9700 1 -- 172.21.15.158:0/3541490252 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0004480 con 0x7fe4f00ce510 2023-10-16T11:20:13.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ecff9700 1 -- 172.21.15.158:0/3541490252 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e001f070 con 0x7fe4f00ce510 2023-10-16T11:20:13.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4ecff9700 1 -- 172.21.15.158:0/3541490252 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e00038e0 con 0x7fe4f00ce510 2023-10-16T11:20:13.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f006c9d0 con 0x7fe4f00ce510 2023-10-16T11:20:13.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.818+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f006cf90 con 0x7fe4f00ce510 2023-10-16T11:20:13.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.819+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f00681b0 con 0x7fe4f00ce510 2023-10-16T11:20:13.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.819+0000 7fe4ecff9700 1 -- 172.21.15.158:0/3541490252 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe4e0021030 con 0x7fe4f00ce510 2023-10-16T11:20:13.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.820+0000 7fe4ecff9700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe4dc06b140 0x7fe4dc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:13.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.820+0000 7fe4ecff9700 1 -- 172.21.15.158:0/3541490252 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7510+0+0 (secure 0 0 0) 0x7fe4e008b500 con 0x7fe4f00ce510 2023-10-16T11:20:13.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.820+0000 7fe4effff700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe4dc06b140 0x7fe4dc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:13.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.820+0000 7fe4effff700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe4dc06b140 0x7fe4dc06d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4000f60 tx=0x7fe4e4009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:13.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:13.824+0000 7fe4ecff9700 1 -- 172.21.15.158:0/3541490252 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4e0055bd0 con 0x7fe4f00ce510 2023-10-16T11:20:14.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.015+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe4f00cc180 con 0x7fe4f00ce510 2023-10-16T11:20:14.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.669+0000 7fe4ecff9700 1 -- 172.21.15.158:0/3541490252 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v228) v1 ==== 190+0+0 (secure 0 0 0) 0x7fe4e00591f0 con 0x7fe4f00ce510 2023-10-16T11:20:14.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe4dc06b140 msgr2=0x7fe4dc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:14.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe4dc06b140 0x7fe4dc06d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4000f60 tx=0x7fe4e4009040 comp rx=0 tx=0).stop 2023-10-16T11:20:14.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 msgr2=0x7fe4f006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:14.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f006b9e0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009c40 tx=0x7fe4e0004750 comp rx=0 tx=0).stop 2023-10-16T11:20:14.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 shutdown_connections 2023-10-16T11:20:14.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe4dc06b140 0x7fe4dc06d600 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4f015b590 0x7fe4f00c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4f00cbaf0 0x7fe4f0150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 --2- 172.21.15.158:0/3541490252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f00ce510 0x7fe4f006b9e0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 >> 172.21.15.158:0/3541490252 conn(0x7fe4f006a8d0 msgr2=0x7fe4f006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:14.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.672+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 shutdown_connections 2023-10-16T11:20:14.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.673+0000 7fe4f6500700 1 -- 172.21.15.158:0/3541490252 wait complete. 2023-10-16T11:20:14.677 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_metadata' removed 2023-10-16T11:20:14.691 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T11:20:14.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.814+0000 7fa2c6f58700 1 Processor -- start 2023-10-16T11:20:14.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.814+0000 7fa2c6f58700 1 -- start start 2023-10-16T11:20:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c6f58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c0143790 0x7fa2c006bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c6f58700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2c00c50a0 0x7fa2c00c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c6f58700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa2c006e540 0x7fa2c006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c6f58700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c0143b70 con 0x7fa2c00c50a0 2023-10-16T11:20:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c6f58700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c00c5a60 con 0x7fa2c006e540 2023-10-16T11:20:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c6f58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c006ef10 con 0x7fa2c0143790 2023-10-16T11:20:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2c00c50a0 0x7fa2c00c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2c00c50a0 0x7fa2c00c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39932/0 (socket says 172.21.15.158:39932) 2023-10-16T11:20:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 -- 172.21.15.158:0/1361722758 learned_addr learned my addr 172.21.15.158:0/1361722758 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c54f5700 1 --2- 172.21.15.158:0/1361722758 >> v2:172.21.15.172:3300/0 conn(0x7fa2c006e540 0x7fa2c006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c4cf4700 1 --2- 172.21.15.158:0/1361722758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c0143790 0x7fa2c006bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:14.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 -- 172.21.15.158:0/1361722758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c0143790 msgr2=0x7fa2c006bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 --2- 172.21.15.158:0/1361722758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c0143790 0x7fa2c006bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 -- 172.21.15.158:0/1361722758 >> v2:172.21.15.172:3300/0 conn(0x7fa2c006e540 msgr2=0x7fa2c006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 --2- 172.21.15.158:0/1361722758 >> v2:172.21.15.172:3300/0 conn(0x7fa2c006e540 0x7fa2c006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2bffff700 1 -- 172.21.15.158:0/1361722758 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c006aa70 con 0x7fa2c00c50a0 2023-10-16T11:20:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c54f5700 1 --2- 172.21.15.158:0/1361722758 >> v2:172.21.15.172:3300/0 conn(0x7fa2c006e540 0x7fa2c006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.815+0000 7fa2c4cf4700 1 --2- 172.21.15.158:0/1361722758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c0143790 0x7fa2c006bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2bffff700 1 --2- 172.21.15.158:0/1361722758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c00c50a0 0x7fa2c00c5520 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00cd250 tx=0x7fa2ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c93e26a0f9c61aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:14.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2bf7fe700 1 -- 172.21.15.158:0/1361722758 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac00b870 con 0x7fa2c00c50a0 2023-10-16T11:20:14.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2bf7fe700 1 -- 172.21.15.158:0/1361722758 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac00b9d0 con 0x7fa2c00c50a0 2023-10-16T11:20:14.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2bf7fe700 1 -- 172.21.15.158:0/1361722758 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac00bd60 con 0x7fa2c00c50a0 2023-10-16T11:20:14.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 -- 172.21.15.158:0/1361722758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c00c50a0 msgr2=0x7fa2c00c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:14.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/1361722758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c00c50a0 0x7fa2c00c5520 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00cd250 tx=0x7fa2ac009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 -- 172.21.15.158:0/1361722758 shutdown_connections 2023-10-16T11:20:14.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/1361722758 >> v2:172.21.15.172:3300/0 conn(0x7fa2c006e540 0x7fa2c006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/1361722758 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c0143790 0x7fa2c006bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/1361722758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c00c50a0 0x7fa2c00c5520 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 -- 172.21.15.158:0/1361722758 >> 172.21.15.158:0/1361722758 conn(0x7fa2c00cc5b0 msgr2=0x7fa2c00cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:14.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 -- 172.21.15.158:0/1361722758 shutdown_connections 2023-10-16T11:20:14.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.816+0000 7fa2c6f58700 1 -- 172.21.15.158:0/1361722758 wait complete. 2023-10-16T11:20:14.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 Processor -- start 2023-10-16T11:20:14.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 -- start start 2023-10-16T11:20:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2c006e540 0x7fa2c0152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c00c50a0 0x7fa2c00b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:14.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c0143790 0x7fa2c01568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00b37d0 con 0x7fa2c0143790 2023-10-16T11:20:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0158530 con 0x7fa2c006e540 2023-10-16T11:20:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.818+0000 7fa2c6f58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00b3210 con 0x7fa2c00c50a0 2023-10-16T11:20:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c0143790 0x7fa2c01568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c0143790 0x7fa2c01568c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39940/0 (socket says 172.21.15.158:39940) 2023-10-16T11:20:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 -- 172.21.15.158:0/3314198508 learned_addr learned my addr 172.21.15.158:0/3314198508 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c4cf4700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2c006e540 0x7fa2c0152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2bffff700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c00c50a0 0x7fa2c00b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 -- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c00c50a0 msgr2=0x7fa2c00b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c00c50a0 0x7fa2c00b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 -- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2c006e540 msgr2=0x7fa2c0152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2c006e540 0x7fa2c0152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 -- 172.21.15.158:0/3314198508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac009550 con 0x7fa2c0143790 2023-10-16T11:20:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c4cf4700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2c006e540 0x7fa2c0152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c54f5700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c0143790 0x7fa2c01568c0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800cef0 tx=0x7fa2b800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2bd7fa700 1 -- 172.21.15.158:0/3314198508 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b800ae50 con 0x7fa2c0143790 2023-10-16T11:20:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2bd7fa700 1 -- 172.21.15.158:0/3314198508 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b8015440 con 0x7fa2c0143790 2023-10-16T11:20:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c00b34f0 con 0x7fa2c0143790 2023-10-16T11:20:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2bd7fa700 1 -- 172.21.15.158:0/3314198508 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b80157f0 con 0x7fa2c0143790 2023-10-16T11:20:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.819+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c0158a20 con 0x7fa2c0143790 2023-10-16T11:20:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.820+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0158c40 con 0x7fa2c0143790 2023-10-16T11:20:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.821+0000 7fa2bd7fa700 1 -- 172.21.15.158:0/3314198508 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa2b800d430 con 0x7fa2c0143790 2023-10-16T11:20:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.821+0000 7fa2bd7fa700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a406b140 0x7fa2a406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.821+0000 7fa2bd7fa700 1 -- 172.21.15.158:0/3314198508 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7066+0+0 (secure 0 0 0) 0x7fa2b808acd0 con 0x7fa2c0143790 2023-10-16T11:20:14.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.821+0000 7fa2c4cf4700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a406b140 0x7fa2a406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:14.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.822+0000 7fa2c4cf4700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a406b140 0x7fa2a406d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4006fd0 tx=0x7fa2b4008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:14.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:14.825+0000 7fa2bd7fa700 1 -- 172.21.15.158:0/3314198508 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa2b80554d0 con 0x7fa2c0143790 2023-10-16T11:20:15.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.016+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa2c00681b0 con 0x7fa2c0143790 2023-10-16T11:20:15.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.674+0000 7fa2bd7fa700 1 -- 172.21.15.158:0/3314198508 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v229) v1 ==== 178+0+0 (secure 0 0 0) 0x7fa2b801a0d0 con 0x7fa2c0143790 2023-10-16T11:20:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a406b140 msgr2=0x7fa2a406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a406b140 0x7fa2a406d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4006fd0 tx=0x7fa2b4008040 comp rx=0 tx=0).stop 2023-10-16T11:20:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c0143790 msgr2=0x7fa2c01568c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c0143790 0x7fa2c01568c0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800cef0 tx=0x7fa2b800a340 comp rx=0 tx=0).stop 2023-10-16T11:20:15.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 shutdown_connections 2023-10-16T11:20:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a406b140 0x7fa2a406d600 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2c0143790 0x7fa2c01568c0 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2c006e540 0x7fa2c0152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 --2- 172.21.15.158:0/3314198508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2c00c50a0 0x7fa2c00b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 >> 172.21.15.158:0/3314198508 conn(0x7fa2c00cc5b0 msgr2=0x7fa2c00af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 shutdown_connections 2023-10-16T11:20:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.677+0000 7fa2c6f58700 1 -- 172.21.15.158:0/3314198508 wait complete. 2023-10-16T11:20:15.682 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_data' removed 2023-10-16T11:20:15.696 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:20:15.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.825+0000 7f56e66b9700 1 Processor -- start 2023-10-16T11:20:15.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.825+0000 7f56e66b9700 1 -- start start 2023-10-16T11:20:15.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.826+0000 7f56e66b9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:15.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.826+0000 7f56e66b9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f56e00ccc00 0x7f56e00cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:15.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.826+0000 7f56e66b9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e00d0ae0 con 0x7f56e015deb0 2023-10-16T11:20:15.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.826+0000 7f56e66b9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e00d12d0 con 0x7f56e00ccc00 2023-10-16T11:20:15.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.826+0000 7f56e66b9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00cbdf0 con 0x7f56e015dad0 2023-10-16T11:20:15.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56dffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:15.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56e4c56700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f56e00ccc00 0x7f56e00cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:15.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56df7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f56e015deb0 0x7f56e00ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:44024/0 (socket says 172.21.15.158:44024) 2023-10-16T11:20:15.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56df7fe700 1 -- 172.21.15.158:0/932304467 learned_addr learned my addr 172.21.15.158:0/932304467 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:15.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56e4c56700 1 -- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 msgr2=0x7f56e00cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:15.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56e4c56700 1 --2- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56e4c56700 1 -- 172.21.15.158:0/932304467 >> v1:172.21.15.172:6789/0 conn(0x7f56e015deb0 legacy=0x7f56e00ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:15.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56e4c56700 1 -- 172.21.15.158:0/932304467 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e00cc280 con 0x7f56e00ccc00 2023-10-16T11:20:15.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56dffff700 1 --2- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:15.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56deffd700 1 -- 172.21.15.158:0/932304467 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92201218 0 0) 0x7f56e00d0ae0 con 0x7f56e015deb0 2023-10-16T11:20:15.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56e4c56700 1 --2- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e00ccc00 0x7f56e00cd0b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009d90 tx=0x7f56d800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f2d9fe15c1aa23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:15.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56deffd700 1 -- 172.21.15.158:0/932304467 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d800e070 con 0x7f56e00ccc00 2023-10-16T11:20:15.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56deffd700 1 -- 172.21.15.158:0/932304467 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d800bb20 con 0x7f56e00ccc00 2023-10-16T11:20:15.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.827+0000 7f56deffd700 1 -- 172.21.15.158:0/932304467 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8012430 con 0x7f56e00ccc00 2023-10-16T11:20:15.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 -- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e00ccc00 msgr2=0x7f56e00cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:15.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 --2- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e00ccc00 0x7f56e00cd0b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009d90 tx=0x7f56d800b330 comp rx=0 tx=0).stop 2023-10-16T11:20:15.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 -- 172.21.15.158:0/932304467 shutdown_connections 2023-10-16T11:20:15.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 --2- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e00ccc00 0x7f56e00cd0b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 --2- 172.21.15.158:0/932304467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 -- 172.21.15.158:0/932304467 >> 172.21.15.158:0/932304467 conn(0x7f56e0099a10 msgr2=0x7f56e00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:15.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 -- 172.21.15.158:0/932304467 shutdown_connections 2023-10-16T11:20:15.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.828+0000 7f56e66b9700 1 -- 172.21.15.158:0/932304467 wait complete. 2023-10-16T11:20:15.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.829+0000 7f56e66b9700 1 Processor -- start 2023-10-16T11:20:15.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.830+0000 7f56e66b9700 1 -- start start 2023-10-16T11:20:15.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56e00ccc00 0x7f56e00ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:15.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:15.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e015deb0 0x7f56e00b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:15.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00c83b0 con 0x7f56e00ccc00 2023-10-16T11:20:15.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00c8690 con 0x7f56e015deb0 2023-10-16T11:20:15.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00c8970 con 0x7f56e015dad0 2023-10-16T11:20:15.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56e00ccc00 0x7f56e00ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:15.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56e00ccc00 0x7f56e00ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39950/0 (socket says 172.21.15.158:39950) 2023-10-16T11:20:15.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 -- 172.21.15.158:0/1321455152 learned_addr learned my addr 172.21.15.158:0/1321455152 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:15.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e4c56700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e015deb0 0x7f56e00b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:15.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56df7fe700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:15.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 -- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 msgr2=0x7f56e00b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:15.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 -- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e015deb0 msgr2=0x7f56e00b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:15.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e015deb0 0x7f56e00b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:15.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 -- 172.21.15.158:0/1321455152 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d40094a0 con 0x7f56e00ccc00 2023-10-16T11:20:15.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dffff700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56e00ccc00 0x7f56e00ca060 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f56d400ebe0 tx=0x7f56d400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:15.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dcff9700 1 -- 172.21.15.158:0/1321455152 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d400c850 con 0x7f56e00ccc00 2023-10-16T11:20:15.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dcff9700 1 -- 172.21.15.158:0/1321455152 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d400c9b0 con 0x7f56e00ccc00 2023-10-16T11:20:15.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d8009a70 con 0x7f56e00ccc00 2023-10-16T11:20:15.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56dcff9700 1 -- 172.21.15.158:0/1321455152 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d400cd40 con 0x7f56e00ccc00 2023-10-16T11:20:15.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.831+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e00c8040 con 0x7f56e00ccc00 2023-10-16T11:20:15.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.833+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e00681b0 con 0x7f56e00ccc00 2023-10-16T11:20:15.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.833+0000 7f56dcff9700 1 -- 172.21.15.158:0/1321455152 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f56d4013470 con 0x7f56e00ccc00 2023-10-16T11:20:15.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.833+0000 7f56dcff9700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f56cc06b140 0x7f56cc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:15.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.833+0000 7f56dcff9700 1 -- 172.21.15.158:0/1321455152 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7f56d408a2c0 con 0x7f56e00ccc00 2023-10-16T11:20:15.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.833+0000 7f56df7fe700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f56cc06b140 0x7f56cc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:15.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.834+0000 7f56df7fe700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f56cc06b140 0x7f56cc06d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f56e00c7600 tx=0x7f56d0007e70 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:15.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:15.838+0000 7f56dcff9700 1 -- 172.21.15.158:0/1321455152 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56d4054b40 con 0x7f56e00ccc00 2023-10-16T11:20:16.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.031+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f56e00c8fc0 con 0x7f56e00ccc00 2023-10-16T11:20:16.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.035+0000 7f56dcff9700 1 -- 172.21.15.158:0/1321455152 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) v1 ==== 115+0+0 (secure 0 0 0) 0x7f56d400cea0 con 0x7f56e00ccc00 2023-10-16T11:20:16.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f56cc06b140 msgr2=0x7f56cc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f56cc06b140 0x7f56cc06d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f56e00c7600 tx=0x7f56d0007e70 comp rx=0 tx=0).stop 2023-10-16T11:20:16.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56e00ccc00 msgr2=0x7f56e00ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56e00ccc00 0x7f56e00ca060 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f56d400ebe0 tx=0x7f56d400eef0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 shutdown_connections 2023-10-16T11:20:16.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f56cc06b140 0x7f56cc06d600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56e015deb0 0x7f56e00b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56e00ccc00 0x7f56e00ca060 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 --2- 172.21.15.158:0/1321455152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56e015dad0 0x7f56e00b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 >> 172.21.15.158:0/1321455152 conn(0x7f56e0099a10 msgr2=0x7f56e0152710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:16.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 shutdown_connections 2023-10-16T11:20:16.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.037+0000 7f56e66b9700 1 -- 172.21.15.158:0/1321455152 wait complete. 2023-10-16T11:20:16.053 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:20:16.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.172+0000 7febb0897700 1 Processor -- start 2023-10-16T11:20:16.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.172+0000 7febb0897700 1 -- start start 2023-10-16T11:20:16.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.173+0000 7febb0897700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febac06afe0 0x7febac0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.173+0000 7febb0897700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac0cbd30 con 0x7febac15da40 2023-10-16T11:20:16.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.173+0000 7febb0897700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac0cc1c0 con 0x7febac0cc590 2023-10-16T11:20:16.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.173+0000 7febb0897700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac15de20 con 0x7febac06afe0 2023-10-16T11:20:16.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba9d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febac06afe0 0x7febac0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7febaa59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7febac15da40 0x7febac06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40134/0 (socket says 172.21.15.158:40134) 2023-10-16T11:20:16.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7febaa59c700 1 -- 172.21.15.158:0/276917981 learned_addr learned my addr 172.21.15.158:0/276917981 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:16.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba9d9b700 1 -- 172.21.15.158:0/276917981 >> v1:172.21.15.172:6789/0 conn(0x7febac0cc590 legacy=0x7febac0afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba9d9b700 1 -- 172.21.15.158:0/276917981 >> v1:172.21.15.158:6789/0 conn(0x7febac15da40 legacy=0x7febac06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba959a700 1 -- 172.21.15.158:0/276917981 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099271515 0 0) 0x7febac0cbd30 con 0x7febac15da40 2023-10-16T11:20:16.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba9d9b700 1 -- 172.21.15.158:0/276917981 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac0ce260 con 0x7febac06afe0 2023-10-16T11:20:16.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba9d9b700 1 --2- 172.21.15.158:0/276917981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac06afe0 0x7febac0caa20 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009840 tx=0x7feb9c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51a38c924a7023d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba959a700 1 -- 172.21.15.158:0/276917981 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c011830 con 0x7febac06afe0 2023-10-16T11:20:16.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba959a700 1 -- 172.21.15.158:0/276917981 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c011990 con 0x7febac06afe0 2023-10-16T11:20:16.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.174+0000 7feba959a700 1 -- 172.21.15.158:0/276917981 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c011d20 con 0x7febac06afe0 2023-10-16T11:20:16.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.175+0000 7febb0897700 1 -- 172.21.15.158:0/276917981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac06afe0 msgr2=0x7febac0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.175+0000 7febb0897700 1 --2- 172.21.15.158:0/276917981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac06afe0 0x7febac0caa20 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009840 tx=0x7feb9c009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:16.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.175+0000 7febb0897700 1 -- 172.21.15.158:0/276917981 shutdown_connections 2023-10-16T11:20:16.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.175+0000 7febb0897700 1 --2- 172.21.15.158:0/276917981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac06afe0 0x7febac0caa20 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.175+0000 7febb0897700 1 -- 172.21.15.158:0/276917981 >> 172.21.15.158:0/276917981 conn(0x7febac099a10 msgr2=0x7febac0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:16.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.175+0000 7febb0897700 1 -- 172.21.15.158:0/276917981 shutdown_connections 2023-10-16T11:20:16.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.175+0000 7febb0897700 1 -- 172.21.15.158:0/276917981 wait complete. 2023-10-16T11:20:16.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.176+0000 7febb0897700 1 Processor -- start 2023-10-16T11:20:16.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.176+0000 7febb0897700 1 -- start start 2023-10-16T11:20:16.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7febb0897700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febac06afe0 0x7febac0b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7febb0897700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac0cc590 0x7febac0c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7febb0897700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febac15da40 0x7febac09d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7febb0897700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac14c0d0 con 0x7febac0cc590 2023-10-16T11:20:16.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7febb0897700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac0cae30 con 0x7febac06afe0 2023-10-16T11:20:16.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7febb0897700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac0ca870 con 0x7febac15da40 2023-10-16T11:20:16.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac0cc590 0x7febac0c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac0cc590 0x7febac0c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39966/0 (socket says 172.21.15.158:39966) 2023-10-16T11:20:16.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 -- 172.21.15.158:0/3220146938 learned_addr learned my addr 172.21.15.158:0/3220146938 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:16.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7febaa59c700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febac06afe0 0x7febac0b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 -- 172.21.15.158:0/3220146938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febac15da40 msgr2=0x7febac09d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febac15da40 0x7febac09d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 -- 172.21.15.158:0/3220146938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febac06afe0 msgr2=0x7febac0b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febac06afe0 0x7febac0b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 -- 172.21.15.158:0/3220146938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c0094a0 con 0x7febac0cc590 2023-10-16T11:20:16.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.177+0000 7feba9d9b700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac0cc590 0x7febac0c9f00 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0090b0 tx=0x7feb9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.178+0000 7feb9affd700 1 -- 172.21.15.158:0/3220146938 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c01a900 con 0x7febac0cc590 2023-10-16T11:20:16.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.178+0000 7feb9affd700 1 -- 172.21.15.158:0/3220146938 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c01aa60 con 0x7febac0cc590 2023-10-16T11:20:16.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.178+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac0cab50 con 0x7febac0cc590 2023-10-16T11:20:16.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.178+0000 7feb9affd700 1 -- 172.21.15.158:0/3220146938 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c01add0 con 0x7febac0cc590 2023-10-16T11:20:16.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.178+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac0c7820 con 0x7febac0cc590 2023-10-16T11:20:16.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.179+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac0681b0 con 0x7febac0cc590 2023-10-16T11:20:16.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.179+0000 7feb9affd700 1 -- 172.21.15.158:0/3220146938 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7feb9c022070 con 0x7febac0cc590 2023-10-16T11:20:16.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.180+0000 7feb9affd700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7feb8c06b140 0x7feb8c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.180+0000 7feb9affd700 1 -- 172.21.15.158:0/3220146938 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7feb9c08b000 con 0x7febac0cc590 2023-10-16T11:20:16.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.180+0000 7febaa59c700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7feb8c06b140 0x7feb8c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.180+0000 7febaa59c700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7feb8c06b140 0x7feb8c06d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feba0007fd0 tx=0x7feba0009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.185+0000 7feb9affd700 1 -- 172.21.15.158:0/3220146938 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb9c055ae0 con 0x7febac0cc590 2023-10-16T11:20:16.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.377+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7febac0b31d0 con 0x7febac0cc590 2023-10-16T11:20:16.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.382+0000 7feb9affd700 1 -- 172.21.15.158:0/3220146938 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7feb9c059100 con 0x7febac0cc590 2023-10-16T11:20:16.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7feb8c06b140 msgr2=0x7feb8c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7feb8c06b140 0x7feb8c06d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feba0007fd0 tx=0x7feba0009040 comp rx=0 tx=0).stop 2023-10-16T11:20:16.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac0cc590 msgr2=0x7febac0c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac0cc590 0x7febac0c9f00 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0090b0 tx=0x7feb9c004750 comp rx=0 tx=0).stop 2023-10-16T11:20:16.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 shutdown_connections 2023-10-16T11:20:16.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7feb8c06b140 0x7feb8c06d600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febac15da40 0x7febac09d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febac06afe0 0x7febac0b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 --2- 172.21.15.158:0/3220146938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febac0cc590 0x7febac0c9f00 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 >> 172.21.15.158:0/3220146938 conn(0x7febac099a10 msgr2=0x7febac0b3620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:16.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 shutdown_connections 2023-10-16T11:20:16.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.383+0000 7febb0897700 1 -- 172.21.15.158:0/3220146938 wait complete. 2023-10-16T11:20:16.398 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:20:16.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:20:16.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:20:16.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.521+0000 7f4199794700 1 Processor -- start 2023-10-16T11:20:16.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.521+0000 7f4199794700 1 -- start start 2023-10-16T11:20:16.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f4199794700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f41940ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f4199794700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f41940cbaf0 0x7f41940cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f4199794700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41940cf4c0 con 0x7f419415f1d0 2023-10-16T11:20:16.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f4199794700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41940cc4c0 con 0x7f41940cbaf0 2023-10-16T11:20:16.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f4199794700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940b49e0 con 0x7f419415b590 2023-10-16T11:20:16.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f41937fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f41940cbaf0 0x7f41940cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f41937fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f41940cbaf0 0x7f41940cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39976/0 (socket says 172.21.15.158:39976) 2023-10-16T11:20:16.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.522+0000 7f41937fe700 1 -- 172.21.15.158:0/1560642108 learned_addr learned my addr 172.21.15.158:0/1560642108 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:16.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f4192ffd700 1 --2- 172.21.15.158:0/1560642108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f41940ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f41937fe700 1 -- 172.21.15.158:0/1560642108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 msgr2=0x7f41940ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:16.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f41937fe700 1 --2- 172.21.15.158:0/1560642108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f41940ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f41937fe700 1 -- 172.21.15.158:0/1560642108 >> v1:172.21.15.172:6789/0 conn(0x7f419415f1d0 legacy=0x7f41940b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f41937fe700 1 -- 172.21.15.158:0/1560642108 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419415f970 con 0x7f41940cbaf0 2023-10-16T11:20:16.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f41937fe700 1 --2- 172.21.15.158:0/1560642108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41940cbf80 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f418c009e80 tx=0x7f418c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f948cd86d03b4fdd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f4191ffb700 1 -- 172.21.15.158:0/1560642108 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c00e070 con 0x7f41940cbaf0 2023-10-16T11:20:16.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f4191ffb700 1 -- 172.21.15.158:0/1560642108 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c00b980 con 0x7f41940cbaf0 2023-10-16T11:20:16.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f4191ffb700 1 -- 172.21.15.158:0/1560642108 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c00bd30 con 0x7f41940cbaf0 2023-10-16T11:20:16.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f4199794700 1 -- 172.21.15.158:0/1560642108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 msgr2=0x7f41940cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.523+0000 7f4199794700 1 --2- 172.21.15.158:0/1560642108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41940cbf80 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f418c009e80 tx=0x7f418c00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:16.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.524+0000 7f4199794700 1 -- 172.21.15.158:0/1560642108 shutdown_connections 2023-10-16T11:20:16.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.524+0000 7f4199794700 1 --2- 172.21.15.158:0/1560642108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41940cbf80 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.524+0000 7f4199794700 1 --2- 172.21.15.158:0/1560642108 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f41940ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.524+0000 7f4199794700 1 -- 172.21.15.158:0/1560642108 >> 172.21.15.158:0/1560642108 conn(0x7f419406a8d0 msgr2=0x7f419406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:16.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.524+0000 7f4199794700 1 -- 172.21.15.158:0/1560642108 shutdown_connections 2023-10-16T11:20:16.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.524+0000 7f4199794700 1 -- 172.21.15.158:0/1560642108 wait complete. 2023-10-16T11:20:16.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.525+0000 7f4199794700 1 Processor -- start 2023-10-16T11:20:16.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.525+0000 7f4199794700 1 -- start start 2023-10-16T11:20:16.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41941f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f419406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419415f1d0 0x7f41941578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419406c700 con 0x7f41940cbaf0 2023-10-16T11:20:16.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419406cb20 con 0x7f419415f1d0 2023-10-16T11:20:16.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419406ba80 con 0x7f419415b590 2023-10-16T11:20:16.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41941f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41941f28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39978/0 (socket says 172.21.15.158:39978) 2023-10-16T11:20:16.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 -- 172.21.15.158:0/2819310842 learned_addr learned my addr 172.21.15.158:0/2819310842 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:16.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f41927fc700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f419406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f41937fe700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419415f1d0 0x7f41941578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 -- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 msgr2=0x7f419406d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f419406d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 -- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419415f1d0 msgr2=0x7f41941578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419415f1d0 0x7f41941578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 -- 172.21.15.158:0/2819310842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c009b30 con 0x7f41940cbaf0 2023-10-16T11:20:16.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f41937fe700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419415f1d0 0x7f41941578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:16.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4192ffd700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41941f28e0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f418800e810 tx=0x7f418800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f417b7fe700 1 -- 172.21.15.158:0/2819310842 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188013610 con 0x7f41940cbaf0 2023-10-16T11:20:16.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f417b7fe700 1 -- 172.21.15.158:0/2819310842 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4188013770 con 0x7f41940cbaf0 2023-10-16T11:20:16.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419406bd60 con 0x7f41940cbaf0 2023-10-16T11:20:16.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f417b7fe700 1 -- 172.21.15.158:0/2819310842 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188013b20 con 0x7f41940cbaf0 2023-10-16T11:20:16.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.526+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419406c320 con 0x7f41940cbaf0 2023-10-16T11:20:16.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.527+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419406c970 con 0x7f41940cbaf0 2023-10-16T11:20:16.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.528+0000 7f417b7fe700 1 -- 172.21.15.158:0/2819310842 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4188013c80 con 0x7f41940cbaf0 2023-10-16T11:20:16.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.528+0000 7f417b7fe700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f418006b140 0x7f418006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.528+0000 7f417b7fe700 1 -- 172.21.15.158:0/2819310842 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7f4188089ba0 con 0x7f41940cbaf0 2023-10-16T11:20:16.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.528+0000 7f41927fc700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f418006b140 0x7f418006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.529+0000 7f41927fc700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f418006b140 0x7f418006d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41940b0560 tx=0x7f4184009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.534+0000 7f417b7fe700 1 -- 172.21.15.158:0/2819310842 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4188054440 con 0x7f41940cbaf0 2023-10-16T11:20:16.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.728+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f41940681b0 con 0x7f41940cbaf0 2023-10-16T11:20:16.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.731+0000 7f417b7fe700 1 -- 172.21.15.158:0/2819310842 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v81) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4188057a60 con 0x7f41940cbaf0 2023-10-16T11:20:16.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f418006b140 msgr2=0x7f418006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f418006b140 0x7f418006d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41940b0560 tx=0x7f4184009040 comp rx=0 tx=0).stop 2023-10-16T11:20:16.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 msgr2=0x7f41941f28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41941f28e0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f418800e810 tx=0x7f418800eb20 comp rx=0 tx=0).stop 2023-10-16T11:20:16.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 shutdown_connections 2023-10-16T11:20:16.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f418006b140 0x7f418006d600 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f419415f1d0 0x7f41941578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41940cbaf0 0x7f41941f28e0 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 --2- 172.21.15.158:0/2819310842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f419415b590 0x7f419406d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 >> 172.21.15.158:0/2819310842 conn(0x7f419406a8d0 msgr2=0x7f41940cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:16.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 shutdown_connections 2023-10-16T11:20:16.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.733+0000 7f4199794700 1 -- 172.21.15.158:0/2819310842 wait complete. 2023-10-16T11:20:16.748 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T11:20:16.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.868+0000 7f59b00e6700 1 Processor -- start 2023-10-16T11:20:16.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.868+0000 7f59b00e6700 1 -- start start 2023-10-16T11:20:16.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59b00e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59a80cdee0 0x7f59a815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59b00e6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a815c5d0 con 0x7f59a80b6e10 2023-10-16T11:20:16.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59b00e6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a815c8b0 con 0x7f59a80ce2c0 2023-10-16T11:20:16.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59b00e6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a815d2c0 con 0x7f59a80cdee0 2023-10-16T11:20:16.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59ade82700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59a80cdee0 0x7f59a815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59ad681700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f59a80b6e10 0x7f59a81521a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40162/0 (socket says 172.21.15.158:40162) 2023-10-16T11:20:16.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59ade82700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59a80cdee0 0x7f59a815bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39986/0 (socket says 172.21.15.158:39986) 2023-10-16T11:20:16.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.869+0000 7f59ad681700 1 -- 172.21.15.158:0/3629921830 learned_addr learned my addr 172.21.15.158:0/3629921830 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:16.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ade82700 1 -- 172.21.15.158:0/3629921830 >> v1:172.21.15.172:6789/0 conn(0x7f59a80ce2c0 legacy=0x7f59a80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:16.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ade82700 1 -- 172.21.15.158:0/3629921830 >> v1:172.21.15.158:6789/0 conn(0x7f59a80b6e10 legacy=0x7f59a81521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ade82700 1 -- 172.21.15.158:0/3629921830 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a815cb90 con 0x7f59a80cdee0 2023-10-16T11:20:16.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ace80700 1 -- 172.21.15.158:0/3629921830 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291424113 0 0) 0x7f59a815c5d0 con 0x7f59a80b6e10 2023-10-16T11:20:16.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ade82700 1 --2- 172.21.15.158:0/3629921830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80cdee0 0x7f59a815bd10 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f5998009840 tx=0x7f5998009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d650334cfd7be2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ace80700 1 -- 172.21.15.158:0/3629921830 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998004230 con 0x7f59a80cdee0 2023-10-16T11:20:16.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ace80700 1 -- 172.21.15.158:0/3629921830 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998004390 con 0x7f59a80cdee0 2023-10-16T11:20:16.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59ace80700 1 -- 172.21.15.158:0/3629921830 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59980036a0 con 0x7f59a80cdee0 2023-10-16T11:20:16.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59b00e6700 1 -- 172.21.15.158:0/3629921830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80cdee0 msgr2=0x7f59a815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.870+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3629921830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80cdee0 0x7f59a815bd10 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f5998009840 tx=0x7f5998009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:16.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.871+0000 7f59b00e6700 1 -- 172.21.15.158:0/3629921830 shutdown_connections 2023-10-16T11:20:16.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.871+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3629921830 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80cdee0 0x7f59a815bd10 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.871+0000 7f59b00e6700 1 -- 172.21.15.158:0/3629921830 >> 172.21.15.158:0/3629921830 conn(0x7f59a806a8d0 msgr2=0x7f59a806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:16.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.871+0000 7f59b00e6700 1 -- 172.21.15.158:0/3629921830 shutdown_connections 2023-10-16T11:20:16.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.871+0000 7f59b00e6700 1 -- 172.21.15.158:0/3629921830 wait complete. 2023-10-16T11:20:16.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 Processor -- start 2023-10-16T11:20:16.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 -- start start 2023-10-16T11:20:16.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80b6e10 0x7f59a814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59a80cdee0 0x7f59a80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59a80ce2c0 0x7f59a80b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8150350 con 0x7f59a80b6e10 2023-10-16T11:20:16.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80d01d0 con 0x7f59a80ce2c0 2023-10-16T11:20:16.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.872+0000 7f59b00e6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80cf930 con 0x7f59a80cdee0 2023-10-16T11:20:16.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80b6e10 0x7f59a814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80b6e10 0x7f59a814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39994/0 (socket says 172.21.15.158:39994) 2023-10-16T11:20:16.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 -- 172.21.15.158:0/3458946277 learned_addr learned my addr 172.21.15.158:0/3458946277 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:16.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ae683700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59a80ce2c0 0x7f59a80b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ad681700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59a80cdee0 0x7f59a80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 -- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59a80cdee0 msgr2=0x7f59a80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59a80cdee0 0x7f59a80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 -- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59a80ce2c0 msgr2=0x7f59a80b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:16.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59a80ce2c0 0x7f59a80b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:16.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 -- 172.21.15.158:0/3458946277 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980094a0 con 0x7f59a80b6e10 2023-10-16T11:20:16.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ae683700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59a80ce2c0 0x7f59a80b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:16.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59ade82700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80b6e10 0x7f59a814fe10 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f59980090b0 tx=0x7f5998004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f599e7fc700 1 -- 172.21.15.158:0/3458946277 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998011830 con 0x7f59a80b6e10 2023-10-16T11:20:16.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f599e7fc700 1 -- 172.21.15.158:0/3458946277 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599801e070 con 0x7f59a80b6e10 2023-10-16T11:20:16.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a80cfc10 con 0x7f59a80b6e10 2023-10-16T11:20:16.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f599e7fc700 1 -- 172.21.15.158:0/3458946277 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998011bb0 con 0x7f59a80b6e10 2023-10-16T11:20:16.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.873+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a80d06b0 con 0x7f59a80b6e10 2023-10-16T11:20:16.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.874+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a80681b0 con 0x7f59a80b6e10 2023-10-16T11:20:16.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.875+0000 7f599e7fc700 1 -- 172.21.15.158:0/3458946277 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5998011d10 con 0x7f59a80b6e10 2023-10-16T11:20:16.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.875+0000 7f599e7fc700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f598c06b140 0x7f598c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:16.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.875+0000 7f59ad681700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f598c06b140 0x7f598c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:16.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.875+0000 7f599e7fc700 1 -- 172.21.15.158:0/3458946277 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7f599808b110 con 0x7f59a80b6e10 2023-10-16T11:20:16.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.876+0000 7f59ad681700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f598c06b140 0x7f598c06d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59a8154b10 tx=0x7f59a4007be0 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:16.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:16.881+0000 7f599e7fc700 1 -- 172.21.15.158:0/3458946277 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5998055b40 con 0x7f59a80b6e10 2023-10-16T11:20:17.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.153+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f59a80b3bf0 con 0x7f59a80b6e10 2023-10-16T11:20:17.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.676+0000 7f599e7fc700 1 -- 172.21.15.158:0/3458946277 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3169) v1 ==== 186+0+0 (secure 0 0 0) 0x7f5998020100 con 0x7f59a80b6e10 2023-10-16T11:20:17.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f598c06b140 msgr2=0x7f598c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:17.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f598c06b140 0x7f598c06d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59a8154b10 tx=0x7f59a4007be0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80b6e10 msgr2=0x7f59a814fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:17.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80b6e10 0x7f59a814fe10 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f59980090b0 tx=0x7f5998004750 comp rx=0 tx=0).stop 2023-10-16T11:20:17.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 shutdown_connections 2023-10-16T11:20:17.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f598c06b140 0x7f598c06d600 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59a80ce2c0 0x7f59a80b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59a80b6e10 0x7f59a814fe10 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 --2- 172.21.15.158:0/3458946277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59a80cdee0 0x7f59a80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 >> 172.21.15.158:0/3458946277 conn(0x7f59a806a8d0 msgr2=0x7f59a80b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:17.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 shutdown_connections 2023-10-16T11:20:17.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.679+0000 7f59b00e6700 1 -- 172.21.15.158:0/3458946277 wait complete. 2023-10-16T11:20:17.694 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2023-10-16T11:20:17.695 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2023-10-16T11:20:17.695 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T11:20:17.696 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:17.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.816+0000 7f19c410c700 1 Processor -- start 2023-10-16T11:20:17.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.816+0000 7f19c410c700 1 -- start start 2023-10-16T11:20:17.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c410c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:17.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c410c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f19bc0729b0 0x7f19bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:17.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c410c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc071e50 con 0x7f19bc06ab40 2023-10-16T11:20:17.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c410c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19bc161270 con 0x7f19bc0729b0 2023-10-16T11:20:17.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c410c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc161510 con 0x7f19bc06e5d0 2023-10-16T11:20:17.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c16a7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:17.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c26a9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f19bc0729b0 0x7f19bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c1ea8700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f19bc06ab40 0x7f19bc0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:44078/0 (socket says 172.21.15.158:44078) 2023-10-16T11:20:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c1ea8700 1 -- 172.21.15.158:0/1079607346 learned_addr learned my addr 172.21.15.158:0/1079607346 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c26a9700 1 -- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 msgr2=0x7f19bc06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c26a9700 1 --2- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c26a9700 1 -- 172.21.15.158:0/1079607346 >> v1:172.21.15.172:6789/0 conn(0x7f19bc06ab40 legacy=0x7f19bc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c26a9700 1 -- 172.21.15.158:0/1079607346 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc1617f0 con 0x7f19bc0729b0 2023-10-16T11:20:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c0ea6700 1 -- 172.21.15.158:0/1079607346 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445412534 0 0) 0x7f19bc071e50 con 0x7f19bc06ab40 2023-10-16T11:20:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c16a7700 1 --2- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c26a9700 1 --2- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 0x7f19bc071910 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009d90 tx=0x7f19b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62a2bf17cad5f0b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c0ea6700 1 -- 172.21.15.158:0/1079607346 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b400e070 con 0x7f19bc0729b0 2023-10-16T11:20:17.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c0ea6700 1 -- 172.21.15.158:0/1079607346 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b400bb20 con 0x7f19bc0729b0 2023-10-16T11:20:17.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c0ea6700 1 -- 172.21.15.158:0/1079607346 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4012430 con 0x7f19bc0729b0 2023-10-16T11:20:17.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c410c700 1 -- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 msgr2=0x7f19bc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:17.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.817+0000 7f19c410c700 1 --2- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 0x7f19bc071910 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009d90 tx=0x7f19b400b330 comp rx=0 tx=0).stop 2023-10-16T11:20:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 -- 172.21.15.158:0/1079607346 shutdown_connections 2023-10-16T11:20:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 --2- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 0x7f19bc071910 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 --2- 172.21.15.158:0/1079607346 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 -- 172.21.15.158:0/1079607346 >> 172.21.15.158:0/1079607346 conn(0x7f19bc0d02b0 msgr2=0x7f19bc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 -- 172.21.15.158:0/1079607346 shutdown_connections 2023-10-16T11:20:17.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 -- 172.21.15.158:0/1079607346 wait complete. 2023-10-16T11:20:17.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 Processor -- start 2023-10-16T11:20:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.819+0000 7f19c410c700 1 -- start start 2023-10-16T11:20:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc06ab40 0x7f19bc155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc15e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 0x7f19bc0c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc160230 con 0x7f19bc06ab40 2023-10-16T11:20:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc15f5b0 con 0x7f19bc0729b0 2023-10-16T11:20:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc15f890 con 0x7f19bc06e5d0 2023-10-16T11:20:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc06ab40 0x7f19bc155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc06ab40 0x7f19bc155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40002/0 (socket says 172.21.15.158:40002) 2023-10-16T11:20:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 -- 172.21.15.158:0/4214225135 learned_addr learned my addr 172.21.15.158:0/4214225135 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:17.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c26a9700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 0x7f19bc0c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:17.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c16a7700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc15e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:17.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 -- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 msgr2=0x7f19bc15e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:17.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc15e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 -- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 msgr2=0x7f19bc0c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:17.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 0x7f19bc0c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:17.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 -- 172.21.15.158:0/4214225135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac0094a0 con 0x7f19bc06ab40 2023-10-16T11:20:17.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c1ea8700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc06ab40 0x7f19bc155120 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f19b80011e0 tx=0x7f19b800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:17.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19b27fc700 1 -- 172.21.15.158:0/4214225135 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8004c80 con 0x7f19bc06ab40 2023-10-16T11:20:17.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19b27fc700 1 -- 172.21.15.158:0/4214225135 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b8004e00 con 0x7f19bc06ab40 2023-10-16T11:20:17.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b4009a70 con 0x7f19bc06ab40 2023-10-16T11:20:17.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19b27fc700 1 -- 172.21.15.158:0/4214225135 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8007310 con 0x7f19bc06ab40 2023-10-16T11:20:17.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc15fe50 con 0x7f19bc06ab40 2023-10-16T11:20:17.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.821+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc0681b0 con 0x7f19bc06ab40 2023-10-16T11:20:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.822+0000 7f19b27fc700 1 -- 172.21.15.158:0/4214225135 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f19b8007470 con 0x7f19bc06ab40 2023-10-16T11:20:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.822+0000 7f19b27fc700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f19a006b140 0x7f19a006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.822+0000 7f19b27fc700 1 -- 172.21.15.158:0/4214225135 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7f19b8089ad0 con 0x7f19bc06ab40 2023-10-16T11:20:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.822+0000 7f19c16a7700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f19a006b140 0x7f19a006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.822+0000 7f19c16a7700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f19a006b140 0x7f19a006d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0ca580 tx=0x7f19ac009210 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:17.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:17.826+0000 7f19b27fc700 1 -- 172.21.15.158:0/4214225135 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19b80543f0 con 0x7f19bc06ab40 2023-10-16T11:20:18.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.020+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19bc0b6230 con 0x7f19bc06ab40 2023-10-16T11:20:18.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.021+0000 7f19b27fc700 1 -- 172.21.15.158:0/4214225135 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+15282 (secure 0 0 0) 0x7f19b8057a10 con 0x7f19bc06ab40 2023-10-16T11:20:18.024 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:18.024 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":229,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:15.671931+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6833/2771711373":"2023-10-17T11:20:12.667460+0000","172.21.15.172:6835/2771711373":"2023-10-17T11:20:12.667460+0000","172.21.15.158:0/931576556":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2043584975":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/139807143":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6834/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:0/572486792":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1251861851":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.172:6832/972449219":"2023-10-17T11:20:05.843085+0000","172.21.15.158:6832/1511764298":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2783528774":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/2949922272":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1862821458":"2023-10-17T11:20:03.428853+0000","172.21.15.158:6833/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:6832/1944509827":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/3674079859":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/417671612":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/1611448416":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/534116803":"2023-10-17T11:20:03.428853+0000","172.21.15.158:0/305366342":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/1831692059":"2023-10-17T11:19:52.926663+0000","172.21.15.158:0/520781827":"2023-10-17T11:19:52.926663+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:18.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f19a006b140 msgr2=0x7f19a006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f19a006b140 0x7f19a006d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0ca580 tx=0x7f19ac009210 comp rx=0 tx=0).stop 2023-10-16T11:20:18.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc06ab40 msgr2=0x7f19bc155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc06ab40 0x7f19bc155120 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f19b80011e0 tx=0x7f19b800ca30 comp rx=0 tx=0).stop 2023-10-16T11:20:18.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 shutdown_connections 2023-10-16T11:20:18.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f19a006b140 0x7f19a006d600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19bc0729b0 0x7f19bc0c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19bc06ab40 0x7f19bc155120 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 --2- 172.21.15.158:0/4214225135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19bc06e5d0 0x7f19bc15e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 >> 172.21.15.158:0/4214225135 conn(0x7f19bc0d02b0 msgr2=0x7f19bc157540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 shutdown_connections 2023-10-16T11:20:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.024+0000 7f19c410c700 1 -- 172.21.15.158:0/4214225135 wait complete. 2023-10-16T11:20:18.038 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:20:18.039 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:20:18.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.160+0000 7fc29fc29700 1 Processor -- start 2023-10-16T11:20:18.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.160+0000 7fc29fc29700 1 -- start start 2023-10-16T11:20:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.160+0000 7fc29fc29700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc29815b6b0 0x7fc2980cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.160+0000 7fc29fc29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 0x7fc2980b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.160+0000 7fc29fc29700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2980b4420 con 0x7fc2980d0c60 2023-10-16T11:20:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.160+0000 7fc29fc29700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29815ba90 con 0x7fc29815b6b0 2023-10-16T11:20:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29fc29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2980cbc20 con 0x7fc2980b3630 2023-10-16T11:20:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29d1c4700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc2980d0c60 0x7fc2980b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40190/0 (socket says 172.21.15.158:40190) 2023-10-16T11:20:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29d1c4700 1 -- 172.21.15.158:0/2736194071 learned_addr learned my addr 172.21.15.158:0/2736194071 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/2736194071 >> v2:172.21.15.172:3300/0 conn(0x7fc29815b6b0 0x7fc2980cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29e1c6700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 0x7fc2980b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189688322 0 0) 0x7fc2980b4420 con 0x7fc2980d0c60 2023-10-16T11:20:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2840035b0 con 0x7fc2980d0c60 2023-10-16T11:20:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1112167313 0 0) 0x7fc2840035b0 con 0x7fc2980d0c60 2023-10-16T11:20:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 msgr2=0x7fc2980b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 0x7fc2980b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 >> v2:172.21.15.172:3300/0 conn(0x7fc29815b6b0 msgr2=0x7fc2980cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 --2- 172.21.15.158:0/2736194071 >> v2:172.21.15.172:3300/0 conn(0x7fc29815b6b0 0x7fc2980cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2980cbf00 con 0x7fc2980d0c60 2023-10-16T11:20:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc294001670 con 0x7fc2980d0c60 2023-10-16T11:20:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 >> v1:172.21.15.158:6789/0 conn(0x7fc2980d0c60 legacy=0x7fc2980b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc284003a00 0x7fc284003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 0x7fc284004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284004d90 0x7fc284005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2980cbc20 con 0x7fc284003a00 2023-10-16T11:20:18.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28400ba10 con 0x7fc284004d90 2023-10-16T11:20:18.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28400bb80 con 0x7fc2980b3630 2023-10-16T11:20:18.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/2736194071 >> v2:172.21.15.172:3300/0 conn(0x7fc29815b6b0 0x7fc2980cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:18.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.161+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc284003a00 0x7fc284003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29d1c4700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 0x7fc284004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29e1c6700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284004d90 0x7fc284005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29d9c5700 1 -- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 msgr2=0x7fc284004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:18.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980b3630 0x7fc284004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29d9c5700 1 -- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284004d90 msgr2=0x7fc284005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:18.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284004d90 0x7fc284005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29d9c5700 1 -- 172.21.15.158:0/2736194071 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2840035b0 con 0x7fc284003a00 2023-10-16T11:20:18.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc284003a00 0x7fc284003e20 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fc288009d10 tx=0x7fc284005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7869621e3badbc13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288009e60 con 0x7fc284003a00 2023-10-16T11:20:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28800e9a0 con 0x7fc284003a00 2023-10-16T11:20:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29c9c3700 1 -- 172.21.15.158:0/2736194071 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28800ed10 con 0x7fc284003a00 2023-10-16T11:20:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29fc29700 1 -- 172.21.15.158:0/2736194071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc284003a00 msgr2=0x7fc284003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.162+0000 7fc29fc29700 1 --2- 172.21.15.158:0/2736194071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc284003a00 0x7fc284003e20 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fc288009d10 tx=0x7fc284005880 comp rx=0 tx=0).stop 2023-10-16T11:20:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.163+0000 7fc29e1c6700 1 -- 172.21.15.158:0/2736194071 reap_dead start 2023-10-16T11:20:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.163+0000 7fc29fc29700 1 -- 172.21.15.158:0/2736194071 shutdown_connections 2023-10-16T11:20:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.163+0000 7fc29fc29700 1 -- 172.21.15.158:0/2736194071 >> 172.21.15.158:0/2736194071 conn(0x7fc2980cdf20 msgr2=0x7fc2980ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.163+0000 7fc29fc29700 1 -- 172.21.15.158:0/2736194071 shutdown_connections 2023-10-16T11:20:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.163+0000 7fc29fc29700 1 -- 172.21.15.158:0/2736194071 wait complete. 2023-10-16T11:20:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.164+0000 7fc29fc29700 1 Processor -- start 2023-10-16T11:20:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.164+0000 7fc29fc29700 1 -- start start 2023-10-16T11:20:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29fc29700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284003a00 0x7fc2980c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29fc29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2980c9490 0x7fc298153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29fc29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980c9910 0x7fc2981540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29fc29700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc298152170 con 0x7fc2980c9490 2023-10-16T11:20:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29fc29700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2980c9ee0 con 0x7fc284003a00 2023-10-16T11:20:18.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29fc29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc298155c30 con 0x7fc2980c9910 2023-10-16T11:20:18.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2980c9490 0x7fc298153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2980c9490 0x7fc298153b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40024/0 (socket says 172.21.15.158:40024) 2023-10-16T11:20:18.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 -- 172.21.15.158:0/680446027 learned_addr learned my addr 172.21.15.158:0/680446027 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:18.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284003a00 0x7fc2980c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29e1c6700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980c9910 0x7fc2981540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 -- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980c9910 msgr2=0x7fc2981540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980c9910 0x7fc2981540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 -- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284003a00 msgr2=0x7fc2980c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284003a00 0x7fc2980c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 -- 172.21.15.158:0/680446027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2980c7890 con 0x7fc2980c9490 2023-10-16T11:20:18.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29d1c4700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2980c9490 0x7fc298153b60 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fc2940073a0 tx=0x7fc2940076b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:18.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc28e7fc700 1 -- 172.21.15.158:0/680446027 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29400f6c0 con 0x7fc2980c9490 2023-10-16T11:20:18.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc28e7fc700 1 -- 172.21.15.158:0/680446027 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29400f820 con 0x7fc2980c9490 2023-10-16T11:20:18.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc298151840 con 0x7fc2980c9490 2023-10-16T11:20:18.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.165+0000 7fc28e7fc700 1 -- 172.21.15.158:0/680446027 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29400fbd0 con 0x7fc2980c9490 2023-10-16T11:20:18.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.166+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc298151e00 con 0x7fc2980c9490 2023-10-16T11:20:18.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.167+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2980681b0 con 0x7fc2980c9490 2023-10-16T11:20:18.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.167+0000 7fc28e7fc700 1 -- 172.21.15.158:0/680446027 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc29400fd30 con 0x7fc2980c9490 2023-10-16T11:20:18.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.167+0000 7fc28e7fc700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc27c06b140 0x7fc27c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.167+0000 7fc28e7fc700 1 -- 172.21.15.158:0/680446027 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7fc294089500 con 0x7fc2980c9490 2023-10-16T11:20:18.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.167+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc27c06b140 0x7fc27c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.168+0000 7fc29d9c5700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc27c06b140 0x7fc27c06d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2880090b0 tx=0x7fc288008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:18.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.173+0000 7fc28e7fc700 1 -- 172.21.15.158:0/680446027 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc294053f60 con 0x7fc2980c9490 2023-10-16T11:20:18.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.362+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc2980cd680 con 0x7fc2980c9490 2023-10-16T11:20:18.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.362+0000 7fc28e7fc700 1 -- 172.21.15.158:0/680446027 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v229) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc29401a0a0 con 0x7fc2980c9490 2023-10-16T11:20:18.365 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:20:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc27c06b140 msgr2=0x7fc27c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc27c06b140 0x7fc27c06d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2880090b0 tx=0x7fc288008030 comp rx=0 tx=0).stop 2023-10-16T11:20:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2980c9490 msgr2=0x7fc298153b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2980c9490 0x7fc298153b60 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fc2940073a0 tx=0x7fc2940076b0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 shutdown_connections 2023-10-16T11:20:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc27c06b140 0x7fc27c06d600 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2980c9910 0x7fc2981540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc284003a00 0x7fc2980c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 --2- 172.21.15.158:0/680446027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2980c9490 0x7fc298153b60 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 >> 172.21.15.158:0/680446027 conn(0x7fc2980cdf20 msgr2=0x7fc2980b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 shutdown_connections 2023-10-16T11:20:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.365+0000 7fc29fc29700 1 -- 172.21.15.158:0/680446027 wait complete. 2023-10-16T11:20:18.379 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T11:20:18.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.499+0000 7efc60e5a700 1 Processor -- start 2023-10-16T11:20:18.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.500+0000 7efc60e5a700 1 -- start start 2023-10-16T11:20:18.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.500+0000 7efc60e5a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efc5c072360 0x7efc5c072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.500+0000 7efc60e5a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc5c06a8d0 0x7efc5c06ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.500+0000 7efc60e5a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5c06f850 con 0x7efc5c06f470 2023-10-16T11:20:18.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.500+0000 7efc60e5a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5c072d50 con 0x7efc5c06a8d0 2023-10-16T11:20:18.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.500+0000 7efc60e5a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5c06b220 con 0x7efc5c072360 2023-10-16T11:20:18.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc5c06a8d0 0x7efc5c06ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc5c06a8d0 0x7efc5c06ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40032/0 (socket says 172.21.15.158:40032) 2023-10-16T11:20:18.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1652778531 learned_addr learned my addr 172.21.15.158:0/1652778531 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:18.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1652778531 >> v2:172.21.15.172:3300/0 conn(0x7efc5c072360 msgr2=0x7efc5c072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:18.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 --2- 172.21.15.158:0/1652778531 >> v2:172.21.15.172:3300/0 conn(0x7efc5c072360 0x7efc5c072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1652778531 >> v1:172.21.15.172:6789/0 conn(0x7efc5c06f470 legacy=0x7efc5c06e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:18.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1652778531 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c0b4260 con 0x7efc5c06a8d0 2023-10-16T11:20:18.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5ad9d700 1 --2- 172.21.15.158:0/1652778531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c06a8d0 0x7efc5c06ace0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0b5320 tx=0x7efc5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6749eeb5b0879328 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:18.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5959a700 1 -- 172.21.15.158:0/1652778531 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5400e070 con 0x7efc5c06a8d0 2023-10-16T11:20:18.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5959a700 1 -- 172.21.15.158:0/1652778531 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc54003a20 con 0x7efc5c06a8d0 2023-10-16T11:20:18.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.501+0000 7efc5959a700 1 -- 172.21.15.158:0/1652778531 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54003dd0 con 0x7efc5c06a8d0 2023-10-16T11:20:18.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 -- 172.21.15.158:0/1652778531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c06a8d0 msgr2=0x7efc5c06ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1652778531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c06a8d0 0x7efc5c06ace0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0b5320 tx=0x7efc5400b330 comp rx=0 tx=0).stop 2023-10-16T11:20:18.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 -- 172.21.15.158:0/1652778531 shutdown_connections 2023-10-16T11:20:18.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1652778531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c06a8d0 0x7efc5c06ace0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1652778531 >> v2:172.21.15.172:3300/0 conn(0x7efc5c072360 0x7efc5c072810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 -- 172.21.15.158:0/1652778531 >> 172.21.15.158:0/1652778531 conn(0x7efc5c0ccda0 msgr2=0x7efc5c0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:18.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 -- 172.21.15.158:0/1652778531 shutdown_connections 2023-10-16T11:20:18.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.502+0000 7efc60e5a700 1 -- 172.21.15.158:0/1652778531 wait complete. 2023-10-16T11:20:18.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.503+0000 7efc60e5a700 1 Processor -- start 2023-10-16T11:20:18.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.503+0000 7efc60e5a700 1 -- start start 2023-10-16T11:20:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc60e5a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc5c06a8d0 0x7efc5c1f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc60e5a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc5c06f470 0x7efc5c1e7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc60e5a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c072360 0x7efc5c0adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc60e5a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1e8810 con 0x7efc5c072360 2023-10-16T11:20:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc60e5a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1e7c00 con 0x7efc5c06f470 2023-10-16T11:20:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc60e5a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1e7ee0 con 0x7efc5c06a8d0 2023-10-16T11:20:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c072360 0x7efc5c0adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c072360 0x7efc5c0adda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40038/0 (socket says 172.21.15.158:40038) 2023-10-16T11:20:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1476053379 learned_addr learned my addr 172.21.15.158:0/1476053379 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc59d9b700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc5c06f470 0x7efc5c1e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5a59c700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc5c06a8d0 0x7efc5c1f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc5c06a8d0 msgr2=0x7efc5c1f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc5c06a8d0 0x7efc5c1f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc5c06f470 msgr2=0x7efc5c1e7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:18.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc5c06f470 0x7efc5c1e7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:18.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 -- 172.21.15.158:0/1476053379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54009b30 con 0x7efc5c072360 2023-10-16T11:20:18.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.504+0000 7efc5ad9d700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c072360 0x7efc5c0adda0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7efc54015040 tx=0x7efc54004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:18.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.505+0000 7efc52ffd700 1 -- 172.21.15.158:0/1476053379 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54004240 con 0x7efc5c072360 2023-10-16T11:20:18.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.505+0000 7efc52ffd700 1 -- 172.21.15.158:0/1476053379 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc540043a0 con 0x7efc5c072360 2023-10-16T11:20:18.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.505+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c1e81c0 con 0x7efc5c072360 2023-10-16T11:20:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.505+0000 7efc52ffd700 1 -- 172.21.15.158:0/1476053379 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54004750 con 0x7efc5c072360 2023-10-16T11:20:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.505+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5c1e77d0 con 0x7efc5c072360 2023-10-16T11:20:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.506+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5c0c5630 con 0x7efc5c072360 2023-10-16T11:20:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.506+0000 7efc52ffd700 1 -- 172.21.15.158:0/1476053379 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efc54019070 con 0x7efc5c072360 2023-10-16T11:20:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.507+0000 7efc52ffd700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc3c06b140 0x7efc3c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.507+0000 7efc52ffd700 1 -- 172.21.15.158:0/1476053379 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7efc5408aef0 con 0x7efc5c072360 2023-10-16T11:20:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.507+0000 7efc5a59c700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc3c06b140 0x7efc3c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.507+0000 7efc5a59c700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc3c06b140 0x7efc3c06d600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efc440016a0 tx=0x7efc44009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.511+0000 7efc52ffd700 1 -- 172.21.15.158:0/1476053379 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc54055790 con 0x7efc5c072360 2023-10-16T11:20:18.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:18.786+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7efc5c1e8950 con 0x7efc5c072360 2023-10-16T11:20:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.688+0000 7efc52ffd700 1 -- 172.21.15.158:0/1476053379 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3171) v1 ==== 190+0+0 (secure 0 0 0) 0x7efc5c1e8950 con 0x7efc5c072360 2023-10-16T11:20:19.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc3c06b140 msgr2=0x7efc3c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:19.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc3c06b140 0x7efc3c06d600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efc440016a0 tx=0x7efc44009040 comp rx=0 tx=0).stop 2023-10-16T11:20:19.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c072360 msgr2=0x7efc5c0adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:19.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c072360 0x7efc5c0adda0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7efc54015040 tx=0x7efc54004ad0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 shutdown_connections 2023-10-16T11:20:19.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc3c06b140 0x7efc3c06d600 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc5c072360 0x7efc5c0adda0 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc5c06a8d0 0x7efc5c1f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 --2- 172.21.15.158:0/1476053379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc5c06f470 0x7efc5c1e7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 >> 172.21.15.158:0/1476053379 conn(0x7efc5c0ccda0 msgr2=0x7efc5c0b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:19.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 shutdown_connections 2023-10-16T11:20:19.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.692+0000 7efc60e5a700 1 -- 172.21.15.158:0/1476053379 wait complete. 2023-10-16T11:20:19.710 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T11:20:19.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.840+0000 7f2a8895f700 1 Processor -- start 2023-10-16T11:20:19.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.840+0000 7f2a8895f700 1 -- start start 2023-10-16T11:20:19.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a8895f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:19.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a8895f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a8406afe0 0x7f2a840afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:19.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a8895f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a840cbd30 con 0x7f2a840cc590 2023-10-16T11:20:19.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a8895f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a840cc1c0 con 0x7f2a8406afe0 2023-10-16T11:20:19.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a8895f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8415de20 con 0x7f2a8415da40 2023-10-16T11:20:19.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a82d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a8406afe0 0x7f2a840afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:19.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a82d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a8406afe0 0x7f2a840afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40046/0 (socket says 172.21.15.158:40046) 2023-10-16T11:20:19.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a82d9d700 1 -- 172.21.15.158:0/1263396549 learned_addr learned my addr 172.21.15.158:0/1263396549 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:19.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a8259c700 1 --2- 172.21.15.158:0/1263396549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:19.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a82d9d700 1 -- 172.21.15.158:0/1263396549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 msgr2=0x7f2a840caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:19.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a82d9d700 1 --2- 172.21.15.158:0/1263396549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a82d9d700 1 -- 172.21.15.158:0/1263396549 >> v1:172.21.15.172:6789/0 conn(0x7f2a840cc590 legacy=0x7f2a8406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:19.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.841+0000 7f2a82d9d700 1 -- 172.21.15.158:0/1263396549 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a840ce260 con 0x7f2a8406afe0 2023-10-16T11:20:19.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a82d9d700 1 --2- 172.21.15.158:0/1263396549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840afa40 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009e50 tx=0x7f2a7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7babeeb4badb666c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:19.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8159a700 1 -- 172.21.15.158:0/1263396549 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c00e070 con 0x7f2a8406afe0 2023-10-16T11:20:19.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8159a700 1 -- 172.21.15.158:0/1263396549 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c00b9c0 con 0x7f2a8406afe0 2023-10-16T11:20:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8159a700 1 -- 172.21.15.158:0/1263396549 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c00bd70 con 0x7f2a8406afe0 2023-10-16T11:20:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 -- 172.21.15.158:0/1263396549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 msgr2=0x7f2a840afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1263396549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840afa40 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009e50 tx=0x7f2a7c00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 -- 172.21.15.158:0/1263396549 shutdown_connections 2023-10-16T11:20:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1263396549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840afa40 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1263396549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 -- 172.21.15.158:0/1263396549 >> 172.21.15.158:0/1263396549 conn(0x7f2a84099a10 msgr2=0x7f2a840d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 -- 172.21.15.158:0/1263396549 shutdown_connections 2023-10-16T11:20:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.842+0000 7f2a8895f700 1 -- 172.21.15.158:0/1263396549 wait complete. 2023-10-16T11:20:19.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.844+0000 7f2a8895f700 1 Processor -- start 2023-10-16T11:20:19.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.844+0000 7f2a8895f700 1 -- start start 2023-10-16T11:20:19.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8895f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:19.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8895f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a840cc590 0x7f2a840b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:19.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8895f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:19.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8895f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a841f2a70 con 0x7f2a8406afe0 2023-10-16T11:20:19.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8895f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a841f2ce0 con 0x7f2a840cc590 2023-10-16T11:20:19.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8895f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a841f2fc0 con 0x7f2a8415da40 2023-10-16T11:20:19.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:19.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40048/0 (socket says 172.21.15.158:40048) 2023-10-16T11:20:19.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8259c700 1 -- 172.21.15.158:0/1956754597 learned_addr learned my addr 172.21.15.158:0/1956754597 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:19.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a82d9d700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:19.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a81d9b700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a840cc590 0x7f2a840b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:19.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.845+0000 7f2a8259c700 1 -- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 msgr2=0x7f2a840b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:19.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a8259c700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a8259c700 1 -- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a840cc590 msgr2=0x7f2a840b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:19.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a8259c700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a840cc590 0x7f2a840b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:19.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a8259c700 1 -- 172.21.15.158:0/1956754597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c009b30 con 0x7f2a8406afe0 2023-10-16T11:20:19.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a8259c700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840c74a0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800d800 tx=0x7f2a7800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:19.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a72ffd700 1 -- 172.21.15.158:0/1956754597 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a780125f0 con 0x7f2a8406afe0 2023-10-16T11:20:19.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a72ffd700 1 -- 172.21.15.158:0/1956754597 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78012750 con 0x7f2a8406afe0 2023-10-16T11:20:19.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a841f32a0 con 0x7f2a8406afe0 2023-10-16T11:20:19.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a72ffd700 1 -- 172.21.15.158:0/1956754597 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78012b00 con 0x7f2a8406afe0 2023-10-16T11:20:19.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.846+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a841f3860 con 0x7f2a8406afe0 2023-10-16T11:20:19.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.847+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a840681b0 con 0x7f2a8406afe0 2023-10-16T11:20:19.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.847+0000 7f2a72ffd700 1 -- 172.21.15.158:0/1956754597 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2a78012c60 con 0x7f2a8406afe0 2023-10-16T11:20:19.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.848+0000 7f2a72ffd700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a6406b140 0x7f2a6406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:19.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.848+0000 7f2a72ffd700 1 -- 172.21.15.158:0/1956754597 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7f2a7808a180 con 0x7f2a8406afe0 2023-10-16T11:20:19.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.848+0000 7f2a81d9b700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a6406b140 0x7f2a6406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:19.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.848+0000 7f2a81d9b700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a6406b140 0x7f2a6406d600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2a840ca8d0 tx=0x7f2a74009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:19.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:19.852+0000 7f2a72ffd700 1 -- 172.21.15.158:0/1956754597 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a78054b30 con 0x7f2a8406afe0 2023-10-16T11:20:20.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.045+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2a840b5ac0 con 0x7f2a8406afe0 2023-10-16T11:20:20.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.048+0000 7f2a72ffd700 1 -- 172.21.15.158:0/1956754597 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2a78058150 con 0x7f2a8406afe0 2023-10-16T11:20:20.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a6406b140 msgr2=0x7f2a6406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a6406b140 0x7f2a6406d600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2a840ca8d0 tx=0x7f2a74009040 comp rx=0 tx=0).stop 2023-10-16T11:20:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 msgr2=0x7f2a840c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840c74a0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800d800 tx=0x7f2a7800dbc0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 shutdown_connections 2023-10-16T11:20:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2a6406b140 0x7f2a6406d600 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8415da40 0x7f2a840b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8406afe0 0x7f2a840c74a0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 --2- 172.21.15.158:0/1956754597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a840cc590 0x7f2a840b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 >> 172.21.15.158:0/1956754597 conn(0x7f2a84099a10 msgr2=0x7f2a8415c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:20.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 shutdown_connections 2023-10-16T11:20:20.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.051+0000 7f2a8895f700 1 -- 172.21.15.158:0/1956754597 wait complete. 2023-10-16T11:20:20.064 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:20:20.065 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:20:20.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:20:20.164 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:20:20.165 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:20:20.165 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:20.183 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:20:20.183 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:20:20.184 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:20.203 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:20:20.204 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:20:20.204 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:20.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:20:20.241 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:20:20.241 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:20.262 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:20:20.262 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T11:20:20.263 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:20.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.398+0000 7f58b682b700 1 Processor -- start 2023-10-16T11:20:20.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.399+0000 7f58b682b700 1 -- start start 2023-10-16T11:20:20.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.399+0000 7f58b682b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58b00cdee0 0x7f58b015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.399+0000 7f58b682b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00b6e10 0x7f58b01521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.399+0000 7f58b682b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b015c5d0 con 0x7f58b00ce2c0 2023-10-16T11:20:20.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.399+0000 7f58b682b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b015d2c0 con 0x7f58b00cdee0 2023-10-16T11:20:20.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.399+0000 7f58b682b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b015c8b0 con 0x7f58b00b6e10 2023-10-16T11:20:20.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.399+0000 7f58b4dc8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f58b00ce2c0 0x7f58b00cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40236/0 (socket says 172.21.15.158:40236) 2023-10-16T11:20:20.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58b4dc8700 1 -- 172.21.15.158:0/2290625261 learned_addr learned my addr 172.21.15.158:0/2290625261 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:20.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58af7fe700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00b6e10 0x7f58b01521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58affff700 1 --2- 172.21.15.158:0/2290625261 >> v2:172.21.15.172:3300/0 conn(0x7f58b00cdee0 0x7f58b015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022273711 0 0) 0x7f58b015c5d0 con 0x7f58b00ce2c0 2023-10-16T11:20:20.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f589c0035b0 con 0x7f58b00ce2c0 2023-10-16T11:20:20.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019358950 0 0) 0x7f589c0035b0 con 0x7f58b00ce2c0 2023-10-16T11:20:20.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00b6e10 msgr2=0x7f58b01521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:20.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00b6e10 0x7f58b01521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 >> v2:172.21.15.172:3300/0 conn(0x7f58b00cdee0 msgr2=0x7f58b015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:20.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 --2- 172.21.15.158:0/2290625261 >> v2:172.21.15.172:3300/0 conn(0x7f58b00cdee0 0x7f58b015bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b015cb90 con 0x7f58b00ce2c0 2023-10-16T11:20:20.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f58a80019f0 con 0x7f58b00ce2c0 2023-10-16T11:20:20.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 >> v1:172.21.15.158:6789/0 conn(0x7f58b00ce2c0 legacy=0x7f58b00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f589c003a00 0x7f589c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f589c0043f0 0x7f589c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f589c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b015d2c0 con 0x7f589c004fe0 2023-10-16T11:20:20.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b015c8b0 con 0x7f589c0043f0 2023-10-16T11:20:20.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c00bf80 con 0x7f589c003a00 2023-10-16T11:20:20.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58affff700 1 --2- 172.21.15.158:0/2290625261 >> v2:172.21.15.172:3300/0 conn(0x7f58b00cdee0 0x7f58b015bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:20.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58af7fe700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00b6e10 0x7f58b01521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:20.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58b4dc8700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f589c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58af7fe700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f589c0043f0 0x7f589c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.400+0000 7f58affff700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f589c003a00 0x7f589c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b4dc8700 1 -- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f589c003a00 msgr2=0x7f589c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:20.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b4dc8700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f589c003a00 0x7f589c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b4dc8700 1 -- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f589c0043f0 msgr2=0x7f589c0048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:20.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b4dc8700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f589c0043f0 0x7f589c0048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b4dc8700 1 -- 172.21.15.158:0/2290625261 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c0035b0 con 0x7f589c004fe0 2023-10-16T11:20:20.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b4dc8700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f589c00ba10 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f58a800a860 tx=0x7f589c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc6a67eb8a2f322 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:20.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8012600 con 0x7f589c004fe0 2023-10-16T11:20:20.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a8012780 con 0x7f589c004fe0 2023-10-16T11:20:20.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58aeffd700 1 -- 172.21.15.158:0/2290625261 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8012b30 con 0x7f589c004fe0 2023-10-16T11:20:20.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b682b700 1 -- 172.21.15.158:0/2290625261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 msgr2=0x7f589c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b682b700 1 --2- 172.21.15.158:0/2290625261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f589c00ba10 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f58a800a860 tx=0x7f589c005880 comp rx=0 tx=0).stop 2023-10-16T11:20:20.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.401+0000 7f58b4dc8700 1 -- 172.21.15.158:0/2290625261 reap_dead start 2023-10-16T11:20:20.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.402+0000 7f58b682b700 1 -- 172.21.15.158:0/2290625261 shutdown_connections 2023-10-16T11:20:20.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.402+0000 7f58b682b700 1 -- 172.21.15.158:0/2290625261 >> 172.21.15.158:0/2290625261 conn(0x7f58b006a8d0 msgr2=0x7f58b006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:20.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.402+0000 7f58b682b700 1 -- 172.21.15.158:0/2290625261 shutdown_connections 2023-10-16T11:20:20.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.402+0000 7f58b682b700 1 -- 172.21.15.158:0/2290625261 wait complete. 2023-10-16T11:20:20.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.403+0000 7f58b682b700 1 Processor -- start 2023-10-16T11:20:20.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.403+0000 7f58b682b700 1 -- start start 2023-10-16T11:20:20.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b682b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f58b01f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b682b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b00d08b0 0x7f58b00c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b682b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00c7630 0x7f58b0155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b682b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00c7ab0 con 0x7f589c004fe0 2023-10-16T11:20:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b682b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00d1a60 con 0x7f58b00d08b0 2023-10-16T11:20:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b682b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b01f72e0 con 0x7f58b00c7630 2023-10-16T11:20:20.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f58b01f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f58b01f6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40068/0 (socket says 172.21.15.158:40068) 2023-10-16T11:20:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 -- 172.21.15.158:0/3838654025 learned_addr learned my addr 172.21.15.158:0/3838654025 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b4dc8700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00c7630 0x7f58b0155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58af7fe700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b00d08b0 0x7f58b00c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 -- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00c7630 msgr2=0x7f58b0155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00c7630 0x7f58b0155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 -- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b00d08b0 msgr2=0x7f58b00c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b00d08b0 0x7f58b00c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 -- 172.21.15.158:0/3838654025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b00d0e50 con 0x7f589c004fe0 2023-10-16T11:20:20.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58affff700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f58b01f6da0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f5898011930 tx=0x7f5898011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:20.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58acff9700 1 -- 172.21.15.158:0/3838654025 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898015870 con 0x7f589c004fe0 2023-10-16T11:20:20.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58acff9700 1 -- 172.21.15.158:0/3838654025 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58980159d0 con 0x7f589c004fe0 2023-10-16T11:20:20.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b00d1130 con 0x7f589c004fe0 2023-10-16T11:20:20.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.404+0000 7f58acff9700 1 -- 172.21.15.158:0/3838654025 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898015d40 con 0x7f589c004fe0 2023-10-16T11:20:20.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.405+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b00d16f0 con 0x7f589c004fe0 2023-10-16T11:20:20.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.405+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b015c5d0 con 0x7f589c004fe0 2023-10-16T11:20:20.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.406+0000 7f58acff9700 1 -- 172.21.15.158:0/3838654025 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5898013400 con 0x7f589c004fe0 2023-10-16T11:20:20.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.406+0000 7f58acff9700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f589006b140 0x7f589006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.406+0000 7f58acff9700 1 -- 172.21.15.158:0/3838654025 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7f589808a5e0 con 0x7f589c004fe0 2023-10-16T11:20:20.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.406+0000 7f58af7fe700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f589006b140 0x7f589006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.407+0000 7f58af7fe700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f589006b140 0x7f589006d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58b00c7ec0 tx=0x7f58a0009210 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:20.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.411+0000 7f58acff9700 1 -- 172.21.15.158:0/3838654025 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5898054f90 con 0x7f589c004fe0 2023-10-16T11:20:20.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.646+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58b00681b0 con 0x7f589c004fe0 2023-10-16T11:20:20.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.646+0000 7f58acff9700 1 -- 172.21.15.158:0/3838654025 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3714 (secure 0 0 0) 0x7f58980585b0 con 0x7f589c004fe0 2023-10-16T11:20:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:20.649 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":122,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2326288796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2326288796},{"type":"v1","addr":"172.21.15.158:6836","nonce":2326288796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1756274310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1756274310},{"type":"v1","addr":"172.21.15.158:6837","nonce":1756274310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":39275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/37744519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":37744519},{"type":"v1","addr":"172.21.15.172:6834","nonce":37744519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":39323,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1674267500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1674267500},{"type":"v1","addr":"172.21.15.172:6835","nonce":1674267500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[]} 2023-10-16T11:20:20.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f589006b140 msgr2=0x7f589006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f589006b140 0x7f589006d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58b00c7ec0 tx=0x7f58a0009210 comp rx=0 tx=0).stop 2023-10-16T11:20:20.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 msgr2=0x7f58b01f6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f58b01f6da0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f5898011930 tx=0x7f5898011c40 comp rx=0 tx=0).stop 2023-10-16T11:20:20.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 shutdown_connections 2023-10-16T11:20:20.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f589006b140 0x7f589006d600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58b00c7630 0x7f58b0155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f589c004fe0 0x7f58b01f6da0 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 --2- 172.21.15.158:0/3838654025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b00d08b0 0x7f58b00c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 >> 172.21.15.158:0/3838654025 conn(0x7f58b006a8d0 msgr2=0x7f58b015d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:20.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 shutdown_connections 2023-10-16T11:20:20.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.648+0000 7f58b682b700 1 -- 172.21.15.158:0/3838654025 wait complete. 2023-10-16T11:20:20.653 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 122 2023-10-16T11:20:20.662 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T11:20:20.662 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T11:20:20.663 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:20:20.663 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:20:20.663 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T11:20:20.664 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T11:20:20.664 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:20:20.664 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:20:20.665 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T11:20:20.665 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T11:20:20.665 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:20:20.665 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:20:20.666 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T11:20:20.666 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T11:20:20.666 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:20:20.666 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:20:20.671 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:20:20.672 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T11:20:20.673 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:20:20.673 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T11:20:20.679 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:20:20.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T11:20:20.680 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:20:20.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T11:20:20.715 INFO:tasks.ceph.mds.a:Started 2023-10-16T11:20:20.716 DEBUG:teuthology.parallel:result is None 2023-10-16T11:20:20.717 INFO:tasks.ceph.mds.c:Started 2023-10-16T11:20:20.717 DEBUG:teuthology.parallel:result is None 2023-10-16T11:20:20.722 INFO:tasks.ceph.mds.b:Started 2023-10-16T11:20:20.722 DEBUG:teuthology.parallel:result is None 2023-10-16T11:20:20.723 INFO:tasks.ceph.mds.d:Started 2023-10-16T11:20:20.723 DEBUG:teuthology.parallel:result is None 2023-10-16T11:20:20.724 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T11:20:20.794 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:20:20.794 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T11:20:20.810 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:20:20.811 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:20:20.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.852+0000 7fd2beddc700 1 Processor -- start 2023-10-16T11:20:20.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.852+0000 7fd2beddc700 1 -- start start 2023-10-16T11:20:20.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2beddc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2b815da10 0x7fd2b815eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2beddc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2b80665e0 0x7fd2b80cb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2beddc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b80ce7b0 con 0x7fd2b80cbe70 2023-10-16T11:20:20.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2beddc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b80cc2e0 con 0x7fd2b80665e0 2023-10-16T11:20:20.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2beddc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b815f070 con 0x7fd2b815da10 2023-10-16T11:20:20.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2b80665e0 0x7fd2b80cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2bd379700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd2b80cbe70 0x7fd2b80ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40278/0 (socket says 172.21.15.158:40278) 2023-10-16T11:20:20.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2b80665e0 0x7fd2b80cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40102/0 (socket says 172.21.15.158:40102) 2023-10-16T11:20:20.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2bd379700 1 -- 172.21.15.158:0/3197414556 learned_addr learned my addr 172.21.15.158:0/3197414556 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:20.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2bcb78700 1 --2- 172.21.15.158:0/3197414556 >> v2:172.21.15.172:3300/0 conn(0x7fd2b815da10 0x7fd2b815eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b7fff700 1 -- 172.21.15.158:0/3197414556 >> v2:172.21.15.172:3300/0 conn(0x7fd2b815da10 msgr2=0x7fd2b815eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:20.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b7fff700 1 --2- 172.21.15.158:0/3197414556 >> v2:172.21.15.172:3300/0 conn(0x7fd2b815da10 0x7fd2b815eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b7fff700 1 -- 172.21.15.158:0/3197414556 >> v1:172.21.15.158:6789/0 conn(0x7fd2b80cbe70 legacy=0x7fd2b80ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b7fff700 1 -- 172.21.15.158:0/3197414556 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b806e910 con 0x7fd2b80665e0 2023-10-16T11:20:20.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b77fe700 1 -- 172.21.15.158:0/3197414556 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933357656 0 0) 0x7fd2b80ce7b0 con 0x7fd2b80cbe70 2023-10-16T11:20:20.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b7fff700 1 --2- 172.21.15.158:0/3197414556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80665e0 0x7fd2b80cb930 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7fd2b806fd50 tx=0x7fd2a4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e28eb90089ce6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:20.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b77fe700 1 -- 172.21.15.158:0/3197414556 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4011830 con 0x7fd2b80665e0 2023-10-16T11:20:20.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b77fe700 1 -- 172.21.15.158:0/3197414556 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a4011990 con 0x7fd2b80665e0 2023-10-16T11:20:20.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.853+0000 7fd2b77fe700 1 -- 172.21.15.158:0/3197414556 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4011d20 con 0x7fd2b80665e0 2023-10-16T11:20:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 -- 172.21.15.158:0/3197414556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80665e0 msgr2=0x7fd2b80cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 --2- 172.21.15.158:0/3197414556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80665e0 0x7fd2b80cb930 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7fd2b806fd50 tx=0x7fd2a4009b20 comp rx=0 tx=0).stop 2023-10-16T11:20:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 -- 172.21.15.158:0/3197414556 shutdown_connections 2023-10-16T11:20:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 --2- 172.21.15.158:0/3197414556 >> v2:172.21.15.172:3300/0 conn(0x7fd2b815da10 0x7fd2b815eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 --2- 172.21.15.158:0/3197414556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80665e0 0x7fd2b80cb930 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 -- 172.21.15.158:0/3197414556 >> 172.21.15.158:0/3197414556 conn(0x7fd2b80cc650 msgr2=0x7fd2b80cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 -- 172.21.15.158:0/3197414556 shutdown_connections 2023-10-16T11:20:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.854+0000 7fd2beddc700 1 -- 172.21.15.158:0/3197414556 wait complete. 2023-10-16T11:20:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.855+0000 7fd2beddc700 1 Processor -- start 2023-10-16T11:20:20.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.855+0000 7fd2beddc700 1 -- start start 2023-10-16T11:20:20.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2beddc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2b80665e0 0x7fd2b80b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2beddc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80cbe70 0x7fd2b81f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2beddc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2b815da10 0x7fd2b80c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2beddc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b81f2360 con 0x7fd2b80cbe70 2023-10-16T11:20:20.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2beddc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b81f2600 con 0x7fd2b80665e0 2023-10-16T11:20:20.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2beddc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b81f28e0 con 0x7fd2b815da10 2023-10-16T11:20:20.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80cbe70 0x7fd2b81f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80cbe70 0x7fd2b81f1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40108/0 (socket says 172.21.15.158:40108) 2023-10-16T11:20:20.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2b7fff700 1 -- 172.21.15.158:0/2719269359 learned_addr learned my addr 172.21.15.158:0/2719269359 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2bcb78700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2b80665e0 0x7fd2b80b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.856+0000 7fd2bd379700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2b815da10 0x7fd2b80c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b7fff700 1 -- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2b815da10 msgr2=0x7fd2b80c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b7fff700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2b815da10 0x7fd2b80c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b7fff700 1 -- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2b80665e0 msgr2=0x7fd2b80b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b7fff700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2b80665e0 0x7fd2b80b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b7fff700 1 -- 172.21.15.158:0/2719269359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40094a0 con 0x7fd2b80cbe70 2023-10-16T11:20:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2bcb78700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2b80665e0 0x7fd2b80b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:20.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b7fff700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80cbe70 0x7fd2b81f1e20 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4014040 tx=0x7fd2a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b57fa700 1 -- 172.21.15.158:0/2719269359 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40038a0 con 0x7fd2b80cbe70 2023-10-16T11:20:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b57fa700 1 -- 172.21.15.158:0/2719269359 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a401e070 con 0x7fd2b80cbe70 2023-10-16T11:20:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2b57fa700 1 -- 172.21.15.158:0/2719269359 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4003c20 con 0x7fd2b80cbe70 2023-10-16T11:20:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b81f2bc0 con 0x7fd2b80cbe70 2023-10-16T11:20:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.857+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b81f3180 con 0x7fd2b80cbe70 2023-10-16T11:20:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.858+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b81f3370 con 0x7fd2b80cbe70 2023-10-16T11:20:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.858+0000 7fd2b57fa700 1 -- 172.21.15.158:0/2719269359 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd2a402d450 con 0x7fd2b80cbe70 2023-10-16T11:20:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.859+0000 7fd2b57fa700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd29c06b190 0x7fd29c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.859+0000 7fd2b57fa700 1 -- 172.21.15.158:0/2719269359 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6646+0+0 (secure 0 0 0) 0x7fd2a408ac90 con 0x7fd2b80cbe70 2023-10-16T11:20:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.859+0000 7fd2bcb78700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd29c06b190 0x7fd29c06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:20.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.859+0000 7fd2bcb78700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd29c06b190 0x7fd29c06d650 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2b815c780 tx=0x7fd2ac008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:20.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:20.863+0000 7fd2b57fa700 1 -- 172.21.15.158:0/2719269359 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2a40557a0 con 0x7fd2b80cbe70 2023-10-16T11:20:21.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.072+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd2b80681b0 con 0x7fd2b80cbe70 2023-10-16T11:20:21.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.705+0000 7fd2b57fa700 1 -- 172.21.15.158:0/2719269359 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v231) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd2a4015020 con 0x7fd2b80cbe70 2023-10-16T11:20:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd29c06b190 msgr2=0x7fd29c06d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd29c06b190 0x7fd29c06d650 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2b815c780 tx=0x7fd2ac008040 comp rx=0 tx=0).stop 2023-10-16T11:20:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80cbe70 msgr2=0x7fd2b81f1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80cbe70 0x7fd2b81f1e20 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4014040 tx=0x7fd2a4004750 comp rx=0 tx=0).stop 2023-10-16T11:20:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 shutdown_connections 2023-10-16T11:20:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd29c06b190 0x7fd29c06d650 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2b815da10 0x7fd2b80c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2b80665e0 0x7fd2b80b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 --2- 172.21.15.158:0/2719269359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2b80cbe70 0x7fd2b81f1e20 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 >> 172.21.15.158:0/2719269359 conn(0x7fd2b80cc650 msgr2=0x7fd2b80c8510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 shutdown_connections 2023-10-16T11:20:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.708+0000 7fd2beddc700 1 -- 172.21.15.158:0/2719269359 wait complete. 2023-10-16T11:20:21.712 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T11:20:21.723 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T11:20:21.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.842+0000 7f015948d700 1 Processor -- start 2023-10-16T11:20:21.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.842+0000 7f015948d700 1 -- start start 2023-10-16T11:20:21.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f015948d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 0x7f01540ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f015948d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f01540ce950 0x7f01540b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f015948d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01540d3bd0 con 0x7f015415d7c0 2023-10-16T11:20:21.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f015948d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01540d4060 con 0x7f01540ce950 2023-10-16T11:20:21.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f015948d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01540cede0 con 0x7f01540d1210 2023-10-16T11:20:21.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0152ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f015415d7c0 0x7f01540d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40292/0 (socket says 172.21.15.158:40292) 2023-10-16T11:20:21.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0152ffd700 1 -- 172.21.15.158:0/546735164 learned_addr learned my addr 172.21.15.158:0/546735164 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:21.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f01537fe700 1 --2- 172.21.15.158:0/546735164 >> v2:172.21.15.172:3300/0 conn(0x7f01540ce950 0x7f01540b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f01527fc700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 0x7f01540ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149763042 0 0) 0x7f01540d3bd0 con 0x7f015415d7c0 2023-10-16T11:20:21.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c0035b0 con 0x7f015415d7c0 2023-10-16T11:20:21.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3832971707 0 0) 0x7f013c0035b0 con 0x7f015415d7c0 2023-10-16T11:20:21.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 msgr2=0x7f01540ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:21.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 0x7f01540ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 >> v2:172.21.15.172:3300/0 conn(0x7f01540ce950 msgr2=0x7f01540b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:21.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 --2- 172.21.15.158:0/546735164 >> v2:172.21.15.172:3300/0 conn(0x7f01540ce950 0x7f01540b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01540d2fc0 con 0x7f015415d7c0 2023-10-16T11:20:21.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.843+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0148001670 con 0x7f015415d7c0 2023-10-16T11:20:21.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 >> v1:172.21.15.158:6789/0 conn(0x7f015415d7c0 legacy=0x7f01540d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f013c003a00 0x7f013c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f013c004850 0x7f013c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 0x7f013c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01540d4060 con 0x7f013c004850 2023-10-16T11:20:21.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01540cede0 con 0x7f013c003a00 2023-10-16T11:20:21.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013c00ba10 con 0x7f01540d1210 2023-10-16T11:20:21.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01527fc700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f013c004850 0x7f013c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0152ffd700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f013c003a00 0x7f013c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01537fe700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 0x7f013c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01527fc700 1 -- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 msgr2=0x7f013c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:21.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01527fc700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01540d1210 0x7f013c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01527fc700 1 -- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f013c003a00 msgr2=0x7f013c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:21.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01527fc700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f013c003a00 0x7f013c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01527fc700 1 -- 172.21.15.158:0/546735164 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c0035b0 con 0x7f013c004850 2023-10-16T11:20:21.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f01527fc700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f013c004850 0x7f013c004cd0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f0144009b60 tx=0x7f013c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c51bf16048b751 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:21.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014400e6c0 con 0x7f013c004850 2023-10-16T11:20:21.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014400e820 con 0x7f013c004850 2023-10-16T11:20:21.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.844+0000 7f0151ffb700 1 -- 172.21.15.158:0/546735164 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014400eb90 con 0x7f013c004850 2023-10-16T11:20:21.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.845+0000 7f015948d700 1 -- 172.21.15.158:0/546735164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f013c004850 msgr2=0x7f013c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:21.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.845+0000 7f015948d700 1 --2- 172.21.15.158:0/546735164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f013c004850 0x7f013c004cd0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f0144009b60 tx=0x7f013c005880 comp rx=0 tx=0).stop 2023-10-16T11:20:21.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.845+0000 7f01537fe700 1 -- 172.21.15.158:0/546735164 reap_dead start 2023-10-16T11:20:21.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.845+0000 7f015948d700 1 -- 172.21.15.158:0/546735164 shutdown_connections 2023-10-16T11:20:21.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.845+0000 7f015948d700 1 -- 172.21.15.158:0/546735164 >> 172.21.15.158:0/546735164 conn(0x7f01540669f0 msgr2=0x7f0154066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:21.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.845+0000 7f015948d700 1 -- 172.21.15.158:0/546735164 shutdown_connections 2023-10-16T11:20:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.845+0000 7f015948d700 1 -- 172.21.15.158:0/546735164 wait complete. 2023-10-16T11:20:21.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.847+0000 7f015948d700 1 Processor -- start 2023-10-16T11:20:21.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.847+0000 7f015948d700 1 -- start start 2023-10-16T11:20:21.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.847+0000 7f015948d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f013c004850 0x7f01540c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f015948d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01540b5860 0x7f01540b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f015948d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01540b5cc0 0x7f01540b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f015948d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01541f3920 con 0x7f01540b5cc0 2023-10-16T11:20:21.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f015948d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01540c9da0 con 0x7f01540b5860 2023-10-16T11:20:21.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f015948d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01541f2da0 con 0x7f013c004850 2023-10-16T11:20:21.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01540b5cc0 0x7f01540b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01540b5cc0 0x7f01540b8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40132/0 (socket says 172.21.15.158:40132) 2023-10-16T11:20:21.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 -- 172.21.15.158:0/3136049998 learned_addr learned my addr 172.21.15.158:0/3136049998 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:21.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f0152ffd700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f013c004850 0x7f01540c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01527fc700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01540b5860 0x7f01540b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 -- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f013c004850 msgr2=0x7f01540c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:21.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f013c004850 0x7f01540c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 -- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01540b5860 msgr2=0x7f01540b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:21.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01540b5860 0x7f01540b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:21.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 -- 172.21.15.158:0/3136049998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01541f3080 con 0x7f01540b5cc0 2023-10-16T11:20:21.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01527fc700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01540b5860 0x7f01540b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:21.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f01537fe700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01540b5cc0 0x7f01540b8d10 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f01540b6190 tx=0x7f014c011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:21.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f013b7fe700 1 -- 172.21.15.158:0/3136049998 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00e040 con 0x7f01540b5cc0 2023-10-16T11:20:21.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f013b7fe700 1 -- 172.21.15.158:0/3136049998 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c00f040 con 0x7f01540b5cc0 2023-10-16T11:20:21.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01541f3360 con 0x7f01540b5cc0 2023-10-16T11:20:21.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f013b7fe700 1 -- 172.21.15.158:0/3136049998 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c015430 con 0x7f01540b5cc0 2023-10-16T11:20:21.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.848+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01541f29e0 con 0x7f01540b5cc0 2023-10-16T11:20:21.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.849+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01540681b0 con 0x7f01540b5cc0 2023-10-16T11:20:21.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.850+0000 7f013b7fe700 1 -- 172.21.15.158:0/3136049998 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f014c015590 con 0x7f01540b5cc0 2023-10-16T11:20:21.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.850+0000 7f013b7fe700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f014006b170 0x7f014006d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:21.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.850+0000 7f013b7fe700 1 -- 172.21.15.158:0/3136049998 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f014c08a050 con 0x7f01540b5cc0 2023-10-16T11:20:21.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.850+0000 7f0152ffd700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f014006b170 0x7f014006d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:21.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.851+0000 7f0152ffd700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f014006b170 0x7f014006d630 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01540b57b0 tx=0x7f0148007980 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:21.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:21.854+0000 7f013b7fe700 1 -- 172.21.15.158:0/3136049998 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f014c054cc0 con 0x7f01540b5cc0 2023-10-16T11:20:22.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.125+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f01541f2b70 con 0x7f01540b5cc0 2023-10-16T11:20:22.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.125+0000 7f013b7fe700 1 -- 172.21.15.158:0/3136049998 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v54) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f014c0582e0 con 0x7f01540b5cc0 2023-10-16T11:20:22.127 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T11:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T11:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T11:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.130 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.130 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.130 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.130 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T11:20:22.130 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T11:20:22.131 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.131 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.131 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:20:22.131 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:20:22.131 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.131 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.132 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.132 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.132 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T11:20:22.132 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T11:20:22.132 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.132 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:20:22.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:20:22.133 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.133 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.133 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.134 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T11:20:22.134 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T11:20:22.134 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:20:22.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:20:22.135 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.135 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.136 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.136 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T11:20:22.136 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T11:20:22.136 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.137 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.137 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.137 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.137 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.137 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.138 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T11:20:22.138 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T11:20:22.138 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.138 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.138 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.139 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.139 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.139 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.139 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.139 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T11:20:22.140 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T11:20:22.140 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.140 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.140 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.140 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.141 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.141 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.141 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.141 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T11:20:22.141 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T11:20:22.141 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.142 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.142 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.142 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.142 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.143 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.143 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T11:20:22.143 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T11:20:22.143 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.143 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.144 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.144 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.144 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.144 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.144 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T11:20:22.145 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T11:20:22.145 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.145 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.145 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.145 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.146 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.146 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.146 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T11:20:22.146 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T11:20:22.146 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.147 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.147 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.147 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.147 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.147 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.148 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.148 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T11:20:22.148 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T11:20:22.148 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.148 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:20:22.148 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:20:22.149 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.149 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.149 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.149 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.149 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T11:20:22.149 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T11:20:22.150 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.150 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.150 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T11:20:22.150 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T11:20:22.150 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.151 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.151 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.151 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T11:20:22.151 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T11:20:22.151 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.151 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.152 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:20:22.152 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:20:22.152 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:20:22.152 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.152 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.152 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.153 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T11:20:22.153 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T11:20:22.153 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.153 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.153 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:20:22.154 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:20:22.154 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:20:22.154 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.154 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.154 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.154 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T11:20:22.155 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T11:20:22.155 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.155 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:20:22.155 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:20:22.155 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:20:22.156 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:20:22.156 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.156 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.156 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.156 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T11:20:22.156 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T11:20:22.157 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.157 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:20:22.157 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:20:22.157 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:20:22.157 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.157 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.158 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.158 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T11:20:22.158 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T11:20:22.158 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.159 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:20:22.159 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:20:22.159 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.159 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.159 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:22.159 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:22.160 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T11:20:22.160 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T11:20:22.160 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:20:22.160 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:20:22.160 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:20:22.160 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:20:22.161 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.161 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:22.161 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T11:20:22.161 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:20:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f014006b170 msgr2=0x7f014006d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f014006b170 0x7f014006d630 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01540b57b0 tx=0x7f0148007980 comp rx=0 tx=0).stop 2023-10-16T11:20:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01540b5cc0 msgr2=0x7f01540b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01540b5cc0 0x7f01540b8d10 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f01540b6190 tx=0x7f014c011f60 comp rx=0 tx=0).stop 2023-10-16T11:20:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 shutdown_connections 2023-10-16T11:20:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f014006b170 0x7f014006d630 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01540b5cc0 0x7f01540b8d10 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f013c004850 0x7f01540c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 --2- 172.21.15.158:0/3136049998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01540b5860 0x7f01540b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 >> 172.21.15.158:0/3136049998 conn(0x7f01540669f0 msgr2=0x7f015415bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.127+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 shutdown_connections 2023-10-16T11:20:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.128+0000 7f015948d700 1 -- 172.21.15.158:0/3136049998 wait complete. 2023-10-16T11:20:22.165 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:22.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.284+0000 7f5abfb2b700 1 Processor -- start 2023-10-16T11:20:22.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.284+0000 7f5abfb2b700 1 -- start start 2023-10-16T11:20:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abfb2b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abfb2b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab8072910 con 0x7f5ab80af590 2023-10-16T11:20:22.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abfb2b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80b1e40 con 0x7f5ab806e090 2023-10-16T11:20:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abfb2b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80ca7e0 con 0x7f5ab815a6c0 2023-10-16T11:20:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abd0c6700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5ab80af590 0x7f5ab806d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40314/0 (socket says 172.21.15.158:40314) 2023-10-16T11:20:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abd0c6700 1 -- 172.21.15.158:0/429776140 learned_addr learned my addr 172.21.15.158:0/429776140 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:22.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abd8c7700 1 --2- 172.21.15.158:0/429776140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abc8c5700 1 -- 172.21.15.158:0/429776140 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831976614 0 0) 0x7f5ab8072910 con 0x7f5ab80af590 2023-10-16T11:20:22.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abc8c5700 1 -- 172.21.15.158:0/429776140 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa40035b0 con 0x7f5ab80af590 2023-10-16T11:20:22.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abc8c5700 1 -- 172.21.15.158:0/429776140 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585494194 0 0) 0x7f5ab80b1e40 con 0x7f5ab806e090 2023-10-16T11:20:22.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abd8c7700 1 -- 172.21.15.158:0/429776140 >> v1:172.21.15.172:6789/0 conn(0x7f5ab806e090 legacy=0x7f5ab80ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abd8c7700 1 -- 172.21.15.158:0/429776140 >> v1:172.21.15.158:6789/0 conn(0x7f5ab80af590 legacy=0x7f5ab806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abd8c7700 1 -- 172.21.15.158:0/429776140 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80b2120 con 0x7f5ab815a6c0 2023-10-16T11:20:22.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.285+0000 7f5abd8c7700 1 --2- 172.21.15.158:0/429776140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80723d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009850 tx=0x7f5ab4009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2f2f1e1ec030404 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abc8c5700 1 -- 172.21.15.158:0/429776140 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8001670 con 0x7f5ab815a6c0 2023-10-16T11:20:22.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abc8c5700 1 -- 172.21.15.158:0/429776140 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab4005c30 con 0x7f5ab815a6c0 2023-10-16T11:20:22.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abc8c5700 1 -- 172.21.15.158:0/429776140 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4006030 con 0x7f5ab815a6c0 2023-10-16T11:20:22.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abfb2b700 1 -- 172.21.15.158:0/429776140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 msgr2=0x7f5ab80723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/429776140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80723d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009850 tx=0x7f5ab4009b60 comp rx=0 tx=0).stop 2023-10-16T11:20:22.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abfb2b700 1 -- 172.21.15.158:0/429776140 shutdown_connections 2023-10-16T11:20:22.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/429776140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80723d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abfb2b700 1 -- 172.21.15.158:0/429776140 >> 172.21.15.158:0/429776140 conn(0x7f5ab80ccda0 msgr2=0x7f5ab80cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:22.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abfb2b700 1 -- 172.21.15.158:0/429776140 shutdown_connections 2023-10-16T11:20:22.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.286+0000 7f5abfb2b700 1 -- 172.21.15.158:0/429776140 wait complete. 2023-10-16T11:20:22.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.288+0000 7f5abfb2b700 1 Processor -- start 2023-10-16T11:20:22.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.288+0000 7f5abfb2b700 1 -- start start 2023-10-16T11:20:22.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abfb2b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ab806e090 0x7f5ab80c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abfb2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab80af590 0x7f5ab81587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abfb2b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abfb2b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80c81e0 con 0x7f5ab80af590 2023-10-16T11:20:22.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abfb2b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80c84c0 con 0x7f5ab806e090 2023-10-16T11:20:22.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abfb2b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80c87a0 con 0x7f5ab815a6c0 2023-10-16T11:20:22.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab80af590 0x7f5ab81587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab80af590 0x7f5ab81587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40142/0 (socket says 172.21.15.158:40142) 2023-10-16T11:20:22.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 -- 172.21.15.158:0/2349340882 learned_addr learned my addr 172.21.15.158:0/2349340882 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:22.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd8c7700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ab806e090 0x7f5ab80c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abe0c8700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 -- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 msgr2=0x7f5ab80c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 -- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ab806e090 msgr2=0x7f5ab80c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ab806e090 0x7f5ab80c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 -- 172.21.15.158:0/2349340882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4009430 con 0x7f5ab80af590 2023-10-16T11:20:22.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5abd0c6700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab80af590 0x7f5ab81587d0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c6d0 tx=0x7f5aa800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.289+0000 7f5aae7fc700 1 -- 172.21.15.158:0/2349340882 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8005d70 con 0x7f5ab80af590 2023-10-16T11:20:22.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.290+0000 7f5aae7fc700 1 -- 172.21.15.158:0/2349340882 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa8007070 con 0x7f5ab80af590 2023-10-16T11:20:22.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.290+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab80c8a80 con 0x7f5ab80af590 2023-10-16T11:20:22.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.290+0000 7f5aae7fc700 1 -- 172.21.15.158:0/2349340882 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8007400 con 0x7f5ab80af590 2023-10-16T11:20:22.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.290+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab8158d10 con 0x7f5ab80af590 2023-10-16T11:20:22.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.291+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab80681b0 con 0x7f5ab80af590 2023-10-16T11:20:22.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.291+0000 7f5aae7fc700 1 -- 172.21.15.158:0/2349340882 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5aa8007560 con 0x7f5ab80af590 2023-10-16T11:20:22.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.291+0000 7f5aae7fc700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5a9c06b140 0x7f5a9c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.291+0000 7f5aae7fc700 1 -- 172.21.15.158:0/2349340882 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f5aa80895d0 con 0x7f5ab80af590 2023-10-16T11:20:22.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.291+0000 7f5abd8c7700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5a9c06b140 0x7f5a9c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.292+0000 7f5abd8c7700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5a9c06b140 0x7f5a9c06d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40097a0 tx=0x7f5ab4007000 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.296+0000 7f5aae7fc700 1 -- 172.21.15.158:0/2349340882 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5aa8054360 con 0x7f5ab80af590 2023-10-16T11:20:22.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.487+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab80b2cc0 con 0x7f5ab80af590 2023-10-16T11:20:22.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.488+0000 7f5aae7fc700 1 -- 172.21.15.158:0/2349340882 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+13819 (secure 0 0 0) 0x7f5aa8057980 con 0x7f5ab80af590 2023-10-16T11:20:22.490 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:22.491 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":233,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:21.715099+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:22.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5a9c06b140 msgr2=0x7f5a9c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5a9c06b140 0x7f5a9c06d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40097a0 tx=0x7f5ab4007000 comp rx=0 tx=0).stop 2023-10-16T11:20:22.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab80af590 msgr2=0x7f5ab81587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab80af590 0x7f5ab81587d0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c6d0 tx=0x7f5aa800ca90 comp rx=0 tx=0).stop 2023-10-16T11:20:22.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 shutdown_connections 2023-10-16T11:20:22.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5a9c06b140 0x7f5a9c06d600 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ab815a6c0 0x7f5ab80c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ab806e090 0x7f5ab80c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 --2- 172.21.15.158:0/2349340882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab80af590 0x7f5ab81587d0 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 >> 172.21.15.158:0/2349340882 conn(0x7f5ab80ccda0 msgr2=0x7f5ab8159990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:22.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 shutdown_connections 2023-10-16T11:20:22.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.490+0000 7f5abfb2b700 1 -- 172.21.15.158:0/2349340882 wait complete. 2023-10-16T11:20:22.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:20:22.505 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:20:22.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.627+0000 7f2569a71700 1 Processor -- start 2023-10-16T11:20:22.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.627+0000 7f2569a71700 1 -- start start 2023-10-16T11:20:22.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f2569a71700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 0x7f25640cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f2569a71700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25640cfc50 con 0x7f256415f0a0 2023-10-16T11:20:22.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f2569a71700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256415c8f0 con 0x7f25640b36c0 2023-10-16T11:20:22.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f2569a71700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256415ffd0 con 0x7f25640cf2f0 2023-10-16T11:20:22.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f2562ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f256415f0a0 0x7f256415c1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40328/0 (socket says 172.21.15.158:40328) 2023-10-16T11:20:22.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f2562ffd700 1 -- 172.21.15.158:0/3771597072 learned_addr learned my addr 172.21.15.158:0/3771597072 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:22.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f2563fff700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 0x7f25640cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745580196 0 0) 0x7f2554001110 con 0x7f256415f0a0 2023-10-16T11:20:22.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f254c0035b0 con 0x7f256415f0a0 2023-10-16T11:20:22.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49929305 0 0) 0x7f254c0035b0 con 0x7f256415f0a0 2023-10-16T11:20:22.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 msgr2=0x7f25640cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:22.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 0x7f25640cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 >> v1:172.21.15.172:6789/0 conn(0x7f25640b36c0 legacy=0x7f25640d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256415f730 con 0x7f256415f0a0 2023-10-16T11:20:22.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f2554001800 con 0x7f256415f0a0 2023-10-16T11:20:22.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 >> v1:172.21.15.158:6789/0 conn(0x7f256415f0a0 legacy=0x7f256415c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f254c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254c0043e0 0x7f254c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.628+0000 7f25627fc700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 0x7f254c005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256415ffd0 con 0x7f254c003a30 2023-10-16T11:20:22.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c00ba40 con 0x7f254c0043e0 2023-10-16T11:20:22.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c00bc00 con 0x7f25640cf2f0 2023-10-16T11:20:22.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25637fe700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f254c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f2563fff700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 0x7f254c005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f2562ffd700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254c0043e0 0x7f254c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25637fe700 1 -- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 msgr2=0x7f254c005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:22.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25637fe700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25640cf2f0 0x7f254c005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25637fe700 1 -- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254c0043e0 msgr2=0x7f254c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:22.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25637fe700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254c0043e0 0x7f254c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25637fe700 1 -- 172.21.15.158:0/3771597072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c0035b0 con 0x7f254c003a30 2023-10-16T11:20:22.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25637fe700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f254c003e10 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f25580018e0 tx=0x7f254c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e59cda8ba57772 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800c660 con 0x7f254c003a30 2023-10-16T11:20:22.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255800c7e0 con 0x7f254c003a30 2023-10-16T11:20:22.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.629+0000 7f25627fc700 1 -- 172.21.15.158:0/3771597072 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800cb90 con 0x7f254c003a30 2023-10-16T11:20:22.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.630+0000 7f2569a71700 1 -- 172.21.15.158:0/3771597072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 msgr2=0x7f254c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.630+0000 7f2569a71700 1 --2- 172.21.15.158:0/3771597072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f254c003e10 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f25580018e0 tx=0x7f254c005880 comp rx=0 tx=0).stop 2023-10-16T11:20:22.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.630+0000 7f2563fff700 1 -- 172.21.15.158:0/3771597072 reap_dead start 2023-10-16T11:20:22.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.630+0000 7f2569a71700 1 -- 172.21.15.158:0/3771597072 shutdown_connections 2023-10-16T11:20:22.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.630+0000 7f2569a71700 1 -- 172.21.15.158:0/3771597072 >> 172.21.15.158:0/3771597072 conn(0x7f25640cdf90 msgr2=0x7f25640ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:22.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.630+0000 7f2569a71700 1 -- 172.21.15.158:0/3771597072 shutdown_connections 2023-10-16T11:20:22.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.630+0000 7f2569a71700 1 -- 172.21.15.158:0/3771597072 wait complete. 2023-10-16T11:20:22.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.631+0000 7f2569a71700 1 Processor -- start 2023-10-16T11:20:22.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.631+0000 7f2569a71700 1 -- start start 2023-10-16T11:20:22.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2569a71700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f2564154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2569a71700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2564146480 0x7f25641546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2569a71700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2564142400 0x7f2564142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2569a71700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256415f550 con 0x7f254c003a30 2023-10-16T11:20:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2569a71700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25641578e0 con 0x7f2564146480 2023-10-16T11:20:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2569a71700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25640c4810 con 0x7f2564142400 2023-10-16T11:20:22.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f2564154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f2564154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40162/0 (socket says 172.21.15.158:40162) 2023-10-16T11:20:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 -- 172.21.15.158:0/3149811744 learned_addr learned my addr 172.21.15.158:0/3149811744 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2563fff700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2564142400 0x7f2564142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f2562ffd700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2564146480 0x7f25641546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 -- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2564142400 msgr2=0x7f2564142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2564142400 0x7f2564142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 -- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2564146480 msgr2=0x7f25641546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2564146480 0x7f25641546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 -- 172.21.15.158:0/3149811744 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2564156c60 con 0x7f254c003a30 2023-10-16T11:20:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.632+0000 7f25637fe700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f2564154170 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f25580011f0 tx=0x7f2558001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.633+0000 7f254bfff700 1 -- 172.21.15.158:0/3149811744 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800cee0 con 0x7f254c003a30 2023-10-16T11:20:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.633+0000 7f254bfff700 1 -- 172.21.15.158:0/3149811744 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25580054e0 con 0x7f254c003a30 2023-10-16T11:20:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.633+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2564156f40 con 0x7f254c003a30 2023-10-16T11:20:22.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.633+0000 7f254bfff700 1 -- 172.21.15.158:0/3149811744 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25580038f0 con 0x7f254c003a30 2023-10-16T11:20:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.633+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2564157500 con 0x7f254c003a30 2023-10-16T11:20:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.633+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25641576c0 con 0x7f254c003a30 2023-10-16T11:20:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.634+0000 7f254bfff700 1 -- 172.21.15.158:0/3149811744 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2558022070 con 0x7f254c003a30 2023-10-16T11:20:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.634+0000 7f254bfff700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f255006b140 0x7f255006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.634+0000 7f254bfff700 1 -- 172.21.15.158:0/3149811744 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f255808ac60 con 0x7f254c003a30 2023-10-16T11:20:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.634+0000 7f2562ffd700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f255006b140 0x7f255006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.635+0000 7f2562ffd700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f255006b140 0x7f255006d600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f25640c8590 tx=0x7f2554009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.640+0000 7f254bfff700 1 -- 172.21.15.158:0/3149811744 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2558055c60 con 0x7f254c003a30 2023-10-16T11:20:22.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.836+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25640681b0 con 0x7f254c003a30 2023-10-16T11:20:22.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.837+0000 7f254bfff700 1 -- 172.21.15.158:0/3149811744 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v233) v1 ==== 93+0+10 (secure 0 0 0) 0x7f255801ae60 con 0x7f254c003a30 2023-10-16T11:20:22.840 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:20:22.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f255006b140 msgr2=0x7f255006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f255006b140 0x7f255006d600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f25640c8590 tx=0x7f2554009040 comp rx=0 tx=0).stop 2023-10-16T11:20:22.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 msgr2=0x7f2564154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f2564154170 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f25580011f0 tx=0x7f2558001160 comp rx=0 tx=0).stop 2023-10-16T11:20:22.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 shutdown_connections 2023-10-16T11:20:22.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f255006b140 0x7f255006d600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2564142400 0x7f2564142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254c003a30 0x7f2564154170 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 --2- 172.21.15.158:0/3149811744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2564146480 0x7f25641546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 >> 172.21.15.158:0/3149811744 conn(0x7f25640cdf90 msgr2=0x7f256415e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:22.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.839+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 shutdown_connections 2023-10-16T11:20:22.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.840+0000 7f2569a71700 1 -- 172.21.15.158:0/3149811744 wait complete. 2023-10-16T11:20:22.854 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:20:22.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.974+0000 7fe1d4da0700 1 Processor -- start 2023-10-16T11:20:22.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.974+0000 7fe1d4da0700 1 -- start start 2023-10-16T11:20:22.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1d4da0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00715c0 0x7fe1d015de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1d4da0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1d015e3d0 0x7fe1d00cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1d4da0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe1d00719a0 0x7fe1d00cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1d4da0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d00ca820 con 0x7fe1d00719a0 2023-10-16T11:20:22.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1d4da0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d00cab00 con 0x7fe1d015e3d0 2023-10-16T11:20:22.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1d4da0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d00cade0 con 0x7fe1d00715c0 2023-10-16T11:20:22.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1cffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe1d00719a0 0x7fe1d00cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1cffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe1d00719a0 0x7fe1d00cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40176/0 (socket says 172.21.15.158:40176) 2023-10-16T11:20:22.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.975+0000 7fe1cffff700 1 -- 172.21.15.158:0/3811855709 learned_addr learned my addr 172.21.15.158:0/3811855709 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:22.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1cf7fe700 1 --2- 172.21.15.158:0/3811855709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00715c0 0x7fe1d015de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1ceffd700 1 --2- 172.21.15.158:0/3811855709 >> v2:172.21.15.172:3300/0 conn(0x7fe1d015e3d0 0x7fe1d00cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1cffff700 1 -- 172.21.15.158:0/3811855709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00715c0 msgr2=0x7fe1d015de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:22.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1cffff700 1 --2- 172.21.15.158:0/3811855709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00715c0 0x7fe1d015de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1cffff700 1 -- 172.21.15.158:0/3811855709 >> v2:172.21.15.172:3300/0 conn(0x7fe1d015e3d0 msgr2=0x7fe1d00cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:22.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1cffff700 1 --2- 172.21.15.158:0/3811855709 >> v2:172.21.15.172:3300/0 conn(0x7fe1d015e3d0 0x7fe1d00cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1cffff700 1 -- 172.21.15.158:0/3811855709 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00cb0c0 con 0x7fe1d00719a0 2023-10-16T11:20:22.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1cffff700 1 --2- 172.21.15.158:0/3811855709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d00719a0 0x7fe1d00cdaf0 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00b2590 tx=0x7fe1c800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205ef6a8f3ac2ab5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1ce7fc700 1 -- 172.21.15.158:0/3811855709 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8010070 con 0x7fe1d00719a0 2023-10-16T11:20:22.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1ce7fc700 1 -- 172.21.15.158:0/3811855709 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c800db80 con 0x7fe1d00719a0 2023-10-16T11:20:22.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1ce7fc700 1 -- 172.21.15.158:0/3811855709 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8016400 con 0x7fe1d00719a0 2023-10-16T11:20:22.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1d4da0700 1 -- 172.21.15.158:0/3811855709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d00719a0 msgr2=0x7fe1d00cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.976+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/3811855709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d00719a0 0x7fe1d00cdaf0 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00b2590 tx=0x7fe1c800d430 comp rx=0 tx=0).stop 2023-10-16T11:20:22.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.977+0000 7fe1d4da0700 1 -- 172.21.15.158:0/3811855709 shutdown_connections 2023-10-16T11:20:22.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.977+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/3811855709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d00719a0 0x7fe1d00cdaf0 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.977+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/3811855709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00715c0 0x7fe1d015de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.977+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/3811855709 >> v2:172.21.15.172:3300/0 conn(0x7fe1d015e3d0 0x7fe1d00cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.977+0000 7fe1d4da0700 1 -- 172.21.15.158:0/3811855709 >> 172.21.15.158:0/3811855709 conn(0x7fe1d00cfe00 msgr2=0x7fe1d00d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:22.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.977+0000 7fe1d4da0700 1 -- 172.21.15.158:0/3811855709 shutdown_connections 2023-10-16T11:20:22.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.977+0000 7fe1d4da0700 1 -- 172.21.15.158:0/3811855709 wait complete. 2023-10-16T11:20:22.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.978+0000 7fe1d4da0700 1 Processor -- start 2023-10-16T11:20:22.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.978+0000 7fe1d4da0700 1 -- start start 2023-10-16T11:20:22.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1d4da0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1d00715c0 0x7fe1d00b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1d4da0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00719a0 0x7fe1d00c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1d4da0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d015e3d0 0x7fe1d00c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1d4da0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d00af540 con 0x7fe1d015e3d0 2023-10-16T11:20:22.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1d4da0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d00ad670 con 0x7fe1d00715c0 2023-10-16T11:20:22.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1d4da0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d00ad950 con 0x7fe1d00719a0 2023-10-16T11:20:22.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d015e3d0 0x7fe1d00c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cf7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1d00715c0 0x7fe1d00b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d015e3d0 0x7fe1d00c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40182/0 (socket says 172.21.15.158:40182) 2023-10-16T11:20:22.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 -- 172.21.15.158:0/1821631520 learned_addr learned my addr 172.21.15.158:0/1821631520 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:22.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1ceffd700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00719a0 0x7fe1d00c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 -- 172.21.15.158:0/1821631520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00719a0 msgr2=0x7fe1d00c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00719a0 0x7fe1d00c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 -- 172.21.15.158:0/1821631520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1d00715c0 msgr2=0x7fe1d00b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:22.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1d00715c0 0x7fe1d00b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:22.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 -- 172.21.15.158:0/1821631520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c800bd40 con 0x7fe1d015e3d0 2023-10-16T11:20:22.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1cffff700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d015e3d0 0x7fe1d00c8bd0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0072290 tx=0x7fe1c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1b7fff700 1 -- 172.21.15.158:0/1821631520 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8004430 con 0x7fe1d015e3d0 2023-10-16T11:20:22.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.979+0000 7fe1b7fff700 1 -- 172.21.15.158:0/1821631520 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c8004590 con 0x7fe1d015e3d0 2023-10-16T11:20:22.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.980+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d00adc30 con 0x7fe1d015e3d0 2023-10-16T11:20:22.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.980+0000 7fe1b7fff700 1 -- 172.21.15.158:0/1821631520 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8004940 con 0x7fe1d015e3d0 2023-10-16T11:20:22.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.980+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d00ae1f0 con 0x7fe1d015e3d0 2023-10-16T11:20:22.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.980+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d00ae410 con 0x7fe1d015e3d0 2023-10-16T11:20:22.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.981+0000 7fe1b7fff700 1 -- 172.21.15.158:0/1821631520 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe1c8029460 con 0x7fe1d015e3d0 2023-10-16T11:20:22.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.981+0000 7fe1b7fff700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:22.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.981+0000 7fe1b7fff700 1 -- 172.21.15.158:0/1821631520 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5657+0+0 (secure 0 0 0) 0x7fe1c808b2f0 con 0x7fe1d015e3d0 2023-10-16T11:20:22.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.981+0000 7fe1cf7fe700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:22.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.982+0000 7fe1cf7fe700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe1bc06b140 0x7fe1bc06d600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4006b20 tx=0x7fe1c4008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:22.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:22.986+0000 7fe1b7fff700 1 -- 172.21.15.158:0/1821631520 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1c8056110 con 0x7fe1d015e3d0 2023-10-16T11:20:23.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.180+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe1d004ec20 con 0x7fe1d015e3d0 2023-10-16T11:20:23.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.181+0000 7fe1b7fff700 1 -- 172.21.15.158:0/1821631520 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v233) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe1c8059730 con 0x7fe1d015e3d0 2023-10-16T11:20:23.183 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:23.184 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:20:23.184 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:23.184 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:20:23.185 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:20:23.185 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:23.185 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:20:23.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe1bc06b140 msgr2=0x7fe1bc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe1bc06b140 0x7fe1bc06d600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4006b20 tx=0x7fe1c4008040 comp rx=0 tx=0).stop 2023-10-16T11:20:23.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d015e3d0 msgr2=0x7fe1d00c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d015e3d0 0x7fe1d00c8bd0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0072290 tx=0x7fe1c8004ad0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 shutdown_connections 2023-10-16T11:20:23.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d015e3d0 0x7fe1d00c8bd0 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1d00715c0 0x7fe1d00b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 --2- 172.21.15.158:0/1821631520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1d00719a0 0x7fe1d00c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 >> 172.21.15.158:0/1821631520 conn(0x7fe1d00cfe00 msgr2=0x7fe1d00c7830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:23.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 shutdown_connections 2023-10-16T11:20:23.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.183+0000 7fe1d4da0700 1 -- 172.21.15.158:0/1821631520 wait complete. 2023-10-16T11:20:23.197 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T11:20:23.197 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T11:20:23.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.321+0000 7f8e9805a700 1 Processor -- start 2023-10-16T11:20:23.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.321+0000 7f8e9805a700 1 -- start start 2023-10-16T11:20:23.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e9805a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015daa0 0x7f8e900cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e9805a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8e900d0640 0x7f8e900cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e9805a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e900d1ee0 con 0x7f8e9015d6c0 2023-10-16T11:20:23.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e9805a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e900d40a0 con 0x7f8e900d0640 2023-10-16T11:20:23.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e9805a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e900d12d0 con 0x7f8e9015daa0 2023-10-16T11:20:23.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e965f7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8e900d0640 0x7f8e900cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e955f5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015daa0 0x7f8e900cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e965f7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8e900d0640 0x7f8e900cf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:41370/0 (socket says 172.21.15.158:41370) 2023-10-16T11:20:23.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e95df6700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8e9015d6c0 0x7f8e900cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:44266/0 (socket says 172.21.15.158:44266) 2023-10-16T11:20:23.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e965f7700 1 -- 172.21.15.158:0/1915682437 learned_addr learned my addr 172.21.15.158:0/1915682437 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:23.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.322+0000 7f8e955f5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015daa0 0x7f8e900cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:51022/0 (socket says 172.21.15.158:51022) 2023-10-16T11:20:23.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e955f5700 1 -- 172.21.15.158:0/1915682437 >> v2:172.21.15.172:3300/0 conn(0x7f8e900d0640 msgr2=0x7f8e900cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:23.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e955f5700 1 --2- 172.21.15.158:0/1915682437 >> v2:172.21.15.172:3300/0 conn(0x7f8e900d0640 0x7f8e900cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e955f5700 1 -- 172.21.15.158:0/1915682437 >> v1:172.21.15.172:6789/0 conn(0x7f8e9015d6c0 legacy=0x7f8e900cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e955f5700 1 -- 172.21.15.158:0/1915682437 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e900d15b0 con 0x7f8e9015daa0 2023-10-16T11:20:23.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e965f7700 1 --2- 172.21.15.158:0/1915682437 >> v2:172.21.15.172:3300/0 conn(0x7f8e900d0640 0x7f8e900cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:23.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e955f5700 1 --2- 172.21.15.158:0/1915682437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015daa0 0x7f8e900cec50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009840 tx=0x7f8e80009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=797d855059827fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:23.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e94df4700 1 -- 172.21.15.158:0/1915682437 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80011830 con 0x7f8e9015daa0 2023-10-16T11:20:23.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e94df4700 1 -- 172.21.15.158:0/1915682437 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e80011990 con 0x7f8e9015daa0 2023-10-16T11:20:23.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e94df4700 1 -- 172.21.15.158:0/1915682437 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80011d20 con 0x7f8e9015daa0 2023-10-16T11:20:23.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e9805a700 1 -- 172.21.15.158:0/1915682437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015daa0 msgr2=0x7f8e900cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.323+0000 7f8e9805a700 1 --2- 172.21.15.158:0/1915682437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015daa0 0x7f8e900cec50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009840 tx=0x7f8e80009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:23.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.324+0000 7f8e9805a700 1 -- 172.21.15.158:0/1915682437 shutdown_connections 2023-10-16T11:20:23.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.324+0000 7f8e9805a700 1 --2- 172.21.15.158:0/1915682437 >> v2:172.21.15.172:3300/0 conn(0x7f8e900d0640 0x7f8e900cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.324+0000 7f8e9805a700 1 --2- 172.21.15.158:0/1915682437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015daa0 0x7f8e900cec50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.324+0000 7f8e9805a700 1 -- 172.21.15.158:0/1915682437 >> 172.21.15.158:0/1915682437 conn(0x7f8e9009c940 msgr2=0x7f8e900d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:23.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.324+0000 7f8e9805a700 1 -- 172.21.15.158:0/1915682437 shutdown_connections 2023-10-16T11:20:23.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.324+0000 7f8e9805a700 1 -- 172.21.15.158:0/1915682437 wait complete. 2023-10-16T11:20:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.325+0000 7f8e9805a700 1 Processor -- start 2023-10-16T11:20:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.325+0000 7f8e9805a700 1 -- start start 2023-10-16T11:20:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e900d0640 0x7f8e900b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015d6c0 0x7f8e901f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9015daa0 0x7f8e900c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e900b11f0 con 0x7f8e9015daa0 2023-10-16T11:20:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e900b17b0 con 0x7f8e900d0640 2023-10-16T11:20:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e900c9a70 con 0x7f8e9015d6c0 2023-10-16T11:20:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9015daa0 0x7f8e900c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9015daa0 0x7f8e900c9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40196/0 (socket says 172.21.15.158:40196) 2023-10-16T11:20:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 -- 172.21.15.158:0/2598202368 learned_addr learned my addr 172.21.15.158:0/2598202368 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e95df6700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e900d0640 0x7f8e900b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e955f5700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015d6c0 0x7f8e901f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 -- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015d6c0 msgr2=0x7f8e901f2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015d6c0 0x7f8e901f2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 -- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e900d0640 msgr2=0x7f8e900b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e900d0640 0x7f8e900b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 -- 172.21.15.158:0/2598202368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e88009b30 con 0x7f8e9015daa0 2023-10-16T11:20:23.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e955f5700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015d6c0 0x7f8e901f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:23.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e965f7700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9015daa0 0x7f8e900c9020 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009f10 tx=0x7f8e8800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:23.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e867fc700 1 -- 172.21.15.158:0/2598202368 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88018070 con 0x7f8e9015daa0 2023-10-16T11:20:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e867fc700 1 -- 172.21.15.158:0/2598202368 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8800f040 con 0x7f8e9015daa0 2023-10-16T11:20:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e800094a0 con 0x7f8e9015daa0 2023-10-16T11:20:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e867fc700 1 -- 172.21.15.158:0/2598202368 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88003c40 con 0x7f8e9015daa0 2023-10-16T11:20:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.326+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e900b1440 con 0x7f8e9015daa0 2023-10-16T11:20:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.327+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e900681b0 con 0x7f8e9015daa0 2023-10-16T11:20:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.328+0000 7f8e867fc700 1 -- 172.21.15.158:0/2598202368 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8e8800c9f0 con 0x7f8e9015daa0 2023-10-16T11:20:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.328+0000 7f8e867fc700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f8e7406b140 0x7f8e7406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.328+0000 7f8e867fc700 1 -- 172.21.15.158:0/2598202368 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f8e88089d70 con 0x7f8e9015daa0 2023-10-16T11:20:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.328+0000 7f8e95df6700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f8e7406b140 0x7f8e7406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.329+0000 7f8e95df6700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f8e7406b140 0x7f8e7406d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c000f60 tx=0x7f8e8c007ed0 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:23.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.333+0000 7f8e867fc700 1 -- 172.21.15.158:0/2598202368 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e880549f0 con 0x7f8e9015daa0 2023-10-16T11:20:23.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.526+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8e900b18f0 con 0x7f8e9015daa0 2023-10-16T11:20:23.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.709+0000 7f8e867fc700 1 -- 172.21.15.158:0/2598202368 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v234) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8e88058010 con 0x7f8e9015daa0 2023-10-16T11:20:23.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f8e7406b140 msgr2=0x7f8e7406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f8e7406b140 0x7f8e7406d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c000f60 tx=0x7f8e8c007ed0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9015daa0 msgr2=0x7f8e900c9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9015daa0 0x7f8e900c9020 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009f10 tx=0x7f8e8800c3a0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 shutdown_connections 2023-10-16T11:20:23.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f8e7406b140 0x7f8e7406d600 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9015daa0 0x7f8e900c9020 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e900d0640 0x7f8e900b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 --2- 172.21.15.158:0/2598202368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e9015d6c0 0x7f8e901f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 >> 172.21.15.158:0/2598202368 conn(0x7f8e9009c940 msgr2=0x7f8e9006b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:23.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 shutdown_connections 2023-10-16T11:20:23.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.712+0000 7f8e9805a700 1 -- 172.21.15.158:0/2598202368 wait complete. 2023-10-16T11:20:23.716 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T11:20:23.729 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:20:23.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.855+0000 7efc47ac3700 1 Processor -- start 2023-10-16T11:20:23.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.856+0000 7efc47ac3700 1 -- start start 2023-10-16T11:20:23.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.856+0000 7efc47ac3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efc400b30a0 0x7efc400cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.856+0000 7efc47ac3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc400b3480 0x7efc4015ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.856+0000 7efc47ac3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4015f060 con 0x7efc4015f820 2023-10-16T11:20:23.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.856+0000 7efc47ac3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4015c2a0 con 0x7efc400b3480 2023-10-16T11:20:23.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.856+0000 7efc47ac3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4015c580 con 0x7efc400b30a0 2023-10-16T11:20:23.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc400b3480 0x7efc4015ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc400b3480 0x7efc4015ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40202/0 (socket says 172.21.15.158:40202) 2023-10-16T11:20:23.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 -- 172.21.15.158:0/1909531325 learned_addr learned my addr 172.21.15.158:0/1909531325 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:23.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4585f700 1 --2- 172.21.15.158:0/1909531325 >> v2:172.21.15.172:3300/0 conn(0x7efc400b30a0 0x7efc400cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 -- 172.21.15.158:0/1909531325 >> v2:172.21.15.172:3300/0 conn(0x7efc400b30a0 msgr2=0x7efc400cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:23.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 --2- 172.21.15.158:0/1909531325 >> v2:172.21.15.172:3300/0 conn(0x7efc400b30a0 0x7efc400cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 -- 172.21.15.158:0/1909531325 >> v1:172.21.15.172:6789/0 conn(0x7efc4015f820 legacy=0x7efc40151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:23.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 -- 172.21.15.158:0/1909531325 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4015c860 con 0x7efc400b3480 2023-10-16T11:20:23.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4505e700 1 --2- 172.21.15.158:0/1909531325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc400b3480 0x7efc4015ba70 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7efc400cef10 tx=0x7efc3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9c38e1b6478a1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:23.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4485d700 1 -- 172.21.15.158:0/1909531325 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c004230 con 0x7efc400b3480 2023-10-16T11:20:23.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4485d700 1 -- 172.21.15.158:0/1909531325 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c004390 con 0x7efc400b3480 2023-10-16T11:20:23.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.857+0000 7efc4485d700 1 -- 172.21.15.158:0/1909531325 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0036a0 con 0x7efc400b3480 2023-10-16T11:20:23.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 -- 172.21.15.158:0/1909531325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc400b3480 msgr2=0x7efc4015ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1909531325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc400b3480 0x7efc4015ba70 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7efc400cef10 tx=0x7efc3c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 -- 172.21.15.158:0/1909531325 shutdown_connections 2023-10-16T11:20:23.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1909531325 >> v2:172.21.15.172:3300/0 conn(0x7efc400b30a0 0x7efc400cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1909531325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc400b3480 0x7efc4015ba70 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 -- 172.21.15.158:0/1909531325 >> 172.21.15.158:0/1909531325 conn(0x7efc4006ec50 msgr2=0x7efc4006f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:23.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 -- 172.21.15.158:0/1909531325 shutdown_connections 2023-10-16T11:20:23.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.858+0000 7efc47ac3700 1 -- 172.21.15.158:0/1909531325 wait complete. 2023-10-16T11:20:23.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.859+0000 7efc47ac3700 1 Processor -- start 2023-10-16T11:20:23.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.859+0000 7efc47ac3700 1 -- start start 2023-10-16T11:20:23.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc47ac3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc400b30a0 0x7efc400b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc47ac3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc400b3480 0x7efc400b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc47ac3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc4015f820 0x7efc400c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc47ac3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc400c94d0 con 0x7efc4015f820 2023-10-16T11:20:23.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc47ac3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc400c51b0 con 0x7efc400b30a0 2023-10-16T11:20:23.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc47ac3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc400c7e80 con 0x7efc400b3480 2023-10-16T11:20:23.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc4015f820 0x7efc400c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc4015f820 0x7efc400c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40210/0 (socket says 172.21.15.158:40210) 2023-10-16T11:20:23.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 -- 172.21.15.158:0/1624518051 learned_addr learned my addr 172.21.15.158:0/1624518051 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:23.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc4585f700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc400b30a0 0x7efc400b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc4505e700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc400b3480 0x7efc400b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 -- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc400b3480 msgr2=0x7efc400b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc400b3480 0x7efc400b6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 -- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc400b30a0 msgr2=0x7efc400b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:23.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc400b30a0 0x7efc400b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:23.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.860+0000 7efc46060700 1 -- 172.21.15.158:0/1624518051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c009550 con 0x7efc4015f820 2023-10-16T11:20:23.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.861+0000 7efc46060700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc4015f820 0x7efc400c7260 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7efc3800b970 tx=0x7efc3800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:23.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.861+0000 7efc367fc700 1 -- 172.21.15.158:0/1624518051 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3800d860 con 0x7efc4015f820 2023-10-16T11:20:23.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.861+0000 7efc367fc700 1 -- 172.21.15.158:0/1624518051 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3800d9c0 con 0x7efc4015f820 2023-10-16T11:20:23.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.861+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc400c96e0 con 0x7efc4015f820 2023-10-16T11:20:23.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.861+0000 7efc367fc700 1 -- 172.21.15.158:0/1624518051 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3800dd70 con 0x7efc4015f820 2023-10-16T11:20:23.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.861+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc400c9ca0 con 0x7efc4015f820 2023-10-16T11:20:23.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.862+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc400706b0 con 0x7efc4015f820 2023-10-16T11:20:23.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.862+0000 7efc367fc700 1 -- 172.21.15.158:0/1624518051 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efc3800e430 con 0x7efc4015f820 2023-10-16T11:20:23.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.863+0000 7efc367fc700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc2406b140 0x7efc2406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:23.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.863+0000 7efc367fc700 1 -- 172.21.15.158:0/1624518051 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6027+0+0 (secure 0 0 0) 0x7efc38089e10 con 0x7efc4015f820 2023-10-16T11:20:23.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.863+0000 7efc4585f700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc2406b140 0x7efc2406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:23.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.863+0000 7efc4585f700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc2406b140 0x7efc2406d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc30006b20 tx=0x7efc30008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:23.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:23.867+0000 7efc367fc700 1 -- 172.21.15.158:0/1624518051 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc3808d030 con 0x7efc4015f820 2023-10-16T11:20:24.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.063+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efc400681b0 con 0x7efc4015f820 2023-10-16T11:20:24.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.714+0000 7efc367fc700 1 -- 172.21.15.158:0/1624518051 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v235) v1 ==== 170+0+0 (secure 0 0 0) 0x7efc38054870 con 0x7efc4015f820 2023-10-16T11:20:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc2406b140 msgr2=0x7efc2406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc2406b140 0x7efc2406d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc30006b20 tx=0x7efc30008040 comp rx=0 tx=0).stop 2023-10-16T11:20:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc4015f820 msgr2=0x7efc400c7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:24.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc4015f820 0x7efc400c7260 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7efc3800b970 tx=0x7efc3800bc80 comp rx=0 tx=0).stop 2023-10-16T11:20:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 shutdown_connections 2023-10-16T11:20:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7efc2406b140 0x7efc2406d600 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc4015f820 0x7efc400c7260 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc400b30a0 0x7efc400b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:24.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 --2- 172.21.15.158:0/1624518051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc400b3480 0x7efc400b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.721+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 >> 172.21.15.158:0/1624518051 conn(0x7efc4006ec50 msgr2=0x7efc400cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.723+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 shutdown_connections 2023-10-16T11:20:24.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.723+0000 7efc47ac3700 1 -- 172.21.15.158:0/1624518051 wait complete. 2023-10-16T11:20:24.734 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T11:20:24.748 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T11:20:24.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.873+0000 7f27a8cbc700 1 Processor -- start 2023-10-16T11:20:24.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.874+0000 7f27a8cbc700 1 -- start start 2023-10-16T11:20:24.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.874+0000 7f27a8cbc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f27a415b660 0x7f27a40718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:24.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a8cbc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a415f250 con 0x7f27a40ce2b0 2023-10-16T11:20:24.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a8cbc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a4072140 con 0x7f27a406fbc0 2023-10-16T11:20:24.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a8cbc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a4072420 con 0x7f27a415b660 2023-10-16T11:20:24.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f27a415b660 0x7f27a40718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:24.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a259c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f27a415b660 0x7f27a40718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40214/0 (socket says 172.21.15.158:40214) 2023-10-16T11:20:24.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a2d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f27a40ce2b0 0x7f27a40b3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40394/0 (socket says 172.21.15.158:40394) 2023-10-16T11:20:24.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a259c700 1 -- 172.21.15.158:0/1004870813 learned_addr learned my addr 172.21.15.158:0/1004870813 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:24.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a259c700 1 -- 172.21.15.158:0/1004870813 >> v1:172.21.15.172:6789/0 conn(0x7f27a406fbc0 legacy=0x7f27a415fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:24.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a259c700 1 -- 172.21.15.158:0/1004870813 >> v1:172.21.15.158:6789/0 conn(0x7f27a40ce2b0 legacy=0x7f27a40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:24.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a259c700 1 -- 172.21.15.158:0/1004870813 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4072700 con 0x7f27a415b660 2023-10-16T11:20:24.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a259c700 1 --2- 172.21.15.158:0/1004870813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a415b660 0x7f27a40718f0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f27980098f0 tx=0x7f2798009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7754c68b601c8979 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:24.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a159a700 1 -- 172.21.15.158:0/1004870813 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2798004230 con 0x7f27a415b660 2023-10-16T11:20:24.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a159a700 1 -- 172.21.15.158:0/1004870813 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2798004390 con 0x7f27a415b660 2023-10-16T11:20:24.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.875+0000 7f27a159a700 1 -- 172.21.15.158:0/1004870813 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27980036a0 con 0x7f27a415b660 2023-10-16T11:20:24.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.876+0000 7f27a8cbc700 1 -- 172.21.15.158:0/1004870813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a415b660 msgr2=0x7f27a40718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:24.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.876+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/1004870813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a415b660 0x7f27a40718f0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f27980098f0 tx=0x7f2798009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:24.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.876+0000 7f27a8cbc700 1 -- 172.21.15.158:0/1004870813 shutdown_connections 2023-10-16T11:20:24.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.876+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/1004870813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a415b660 0x7f27a40718f0 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:24.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.876+0000 7f27a8cbc700 1 -- 172.21.15.158:0/1004870813 >> 172.21.15.158:0/1004870813 conn(0x7f27a40d03b0 msgr2=0x7f27a40d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:24.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.876+0000 7f27a8cbc700 1 -- 172.21.15.158:0/1004870813 shutdown_connections 2023-10-16T11:20:24.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.876+0000 7f27a8cbc700 1 -- 172.21.15.158:0/1004870813 wait complete. 2023-10-16T11:20:24.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.877+0000 7f27a8cbc700 1 Processor -- start 2023-10-16T11:20:24.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.877+0000 7f27a8cbc700 1 -- start start 2023-10-16T11:20:24.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a8cbc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a406fbc0 0x7f27a40c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:24.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a8cbc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27a40ce2b0 0x7f27a40c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:24.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a8cbc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27a415b660 0x7f27a40b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:24.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a8cbc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a40ca260 con 0x7f27a406fbc0 2023-10-16T11:20:24.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a8cbc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a40c95e0 con 0x7f27a415b660 2023-10-16T11:20:24.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a8cbc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a40c98c0 con 0x7f27a40ce2b0 2023-10-16T11:20:24.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a406fbc0 0x7f27a40c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:24.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a406fbc0 0x7f27a40c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40220/0 (socket says 172.21.15.158:40220) 2023-10-16T11:20:24.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 -- 172.21.15.158:0/858625788 learned_addr learned my addr 172.21.15.158:0/858625788 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:24.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a2d9d700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27a415b660 0x7f27a40b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:24.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a1d9b700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27a40ce2b0 0x7f27a40c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:24.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 -- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27a40ce2b0 msgr2=0x7f27a40c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:24.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27a40ce2b0 0x7f27a40c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:24.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 -- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27a415b660 msgr2=0x7f27a40b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:24.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27a415b660 0x7f27a40b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:24.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.878+0000 7f27a259c700 1 -- 172.21.15.158:0/858625788 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798009550 con 0x7f27a406fbc0 2023-10-16T11:20:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.879+0000 7f27a259c700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a406fbc0 0x7f27a40c9000 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f27a40ccd00 tx=0x7f2798004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.879+0000 7f2792ffd700 1 -- 172.21.15.158:0/858625788 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279800b660 con 0x7f27a406fbc0 2023-10-16T11:20:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.879+0000 7f2792ffd700 1 -- 172.21.15.158:0/858625788 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279800b7e0 con 0x7f27a406fbc0 2023-10-16T11:20:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.879+0000 7f2792ffd700 1 -- 172.21.15.158:0/858625788 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279800bb90 con 0x7f27a406fbc0 2023-10-16T11:20:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.879+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a40c9ba0 con 0x7f27a406fbc0 2023-10-16T11:20:24.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.879+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a40c8520 con 0x7f27a406fbc0 2023-10-16T11:20:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.880+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a40ca3a0 con 0x7f27a406fbc0 2023-10-16T11:20:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.880+0000 7f2792ffd700 1 -- 172.21.15.158:0/858625788 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f279800bcf0 con 0x7f27a406fbc0 2023-10-16T11:20:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.881+0000 7f2792ffd700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f278406b140 0x7f278406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.881+0000 7f2792ffd700 1 -- 172.21.15.158:0/858625788 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f279808b740 con 0x7f27a406fbc0 2023-10-16T11:20:24.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.881+0000 7f27a1d9b700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f278406b140 0x7f278406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:24.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.881+0000 7f27a1d9b700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f278406b140 0x7f278406d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27a41f3fd0 tx=0x7f2794009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:24.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:24.885+0000 7f2792ffd700 1 -- 172.21.15.158:0/858625788 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2798056530 con 0x7f27a406fbc0 2023-10-16T11:20:25.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.119+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f27a40ca3a0 con 0x7f27a406fbc0 2023-10-16T11:20:25.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.734+0000 7f2792ffd700 1 -- 172.21.15.158:0/858625788 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v125) v1 ==== 174+0+0 (secure 0 0 0) 0x7f27a40ca3a0 con 0x7f27a406fbc0 2023-10-16T11:20:25.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.745+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f278406b140 msgr2=0x7f278406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:25.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.745+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f278406b140 0x7f278406d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27a41f3fd0 tx=0x7f2794009040 comp rx=0 tx=0).stop 2023-10-16T11:20:25.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.745+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a406fbc0 msgr2=0x7f27a40c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:25.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.745+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a406fbc0 0x7f27a40c9000 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f27a40ccd00 tx=0x7f2798004750 comp rx=0 tx=0).stop 2023-10-16T11:20:25.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.748+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 shutdown_connections 2023-10-16T11:20:25.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.748+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f278406b140 0x7f278406d600 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.749+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27a415b660 0x7f27a40b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.749+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27a406fbc0 0x7f27a40c9000 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.749+0000 7f27a8cbc700 1 --2- 172.21.15.158:0/858625788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27a40ce2b0 0x7f27a40c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.749+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 >> 172.21.15.158:0/858625788 conn(0x7f27a40d03b0 msgr2=0x7f27a40d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:25.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.749+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 shutdown_connections 2023-10-16T11:20:25.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.750+0000 7f27a8cbc700 1 -- 172.21.15.158:0/858625788 wait complete. 2023-10-16T11:20:25.755 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 31 and data pool 32 2023-10-16T11:20:25.771 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:25.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.892+0000 7fd258ca8700 1 Processor -- start 2023-10-16T11:20:25.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.892+0000 7fd258ca8700 1 -- start start 2023-10-16T11:20:25.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd258ca8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25406e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:25.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd258ca8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd25406e810 0x7fd254072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:25.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd258ca8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2540728a0 0x7fd25406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:25.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd258ca8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd254072cf0 con 0x7fd2540728a0 2023-10-16T11:20:25.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd258ca8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25406f850 con 0x7fd25406e810 2023-10-16T11:20:25.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd258ca8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2540b4260 con 0x7fd25406f470 2023-10-16T11:20:25.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2540728a0 0x7fd25406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:25.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2540728a0 0x7fd25406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40230/0 (socket says 172.21.15.158:40230) 2023-10-16T11:20:25.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 -- 172.21.15.158:0/1709730221 learned_addr learned my addr 172.21.15.158:0/1709730221 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:25.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd251d9b700 1 --2- 172.21.15.158:0/1709730221 >> v2:172.21.15.172:3300/0 conn(0x7fd25406e810 0x7fd254072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:25.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd25259c700 1 --2- 172.21.15.158:0/1709730221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25406e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:25.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 -- 172.21.15.158:0/1709730221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 msgr2=0x7fd25406e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:25.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 --2- 172.21.15.158:0/1709730221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25406e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 -- 172.21.15.158:0/1709730221 >> v2:172.21.15.172:3300/0 conn(0x7fd25406e810 msgr2=0x7fd254072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:25.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 --2- 172.21.15.158:0/1709730221 >> v2:172.21.15.172:3300/0 conn(0x7fd25406e810 0x7fd254072360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd252d9d700 1 -- 172.21.15.158:0/1709730221 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2540b4540 con 0x7fd2540728a0 2023-10-16T11:20:25.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.893+0000 7fd251d9b700 1 --2- 172.21.15.158:0/1709730221 >> v2:172.21.15.172:3300/0 conn(0x7fd25406e810 0x7fd254072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:25.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd252d9d700 1 --2- 172.21.15.158:0/1709730221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2540728a0 0x7fd25406a8d0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7fd2540d1420 tx=0x7fd24c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b59fccaf467c7d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:25.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd25159a700 1 -- 172.21.15.158:0/1709730221 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c00e070 con 0x7fd2540728a0 2023-10-16T11:20:25.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd25159a700 1 -- 172.21.15.158:0/1709730221 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24c00b9c0 con 0x7fd2540728a0 2023-10-16T11:20:25.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd25159a700 1 -- 172.21.15.158:0/1709730221 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c00bd70 con 0x7fd2540728a0 2023-10-16T11:20:25.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 -- 172.21.15.158:0/1709730221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2540728a0 msgr2=0x7fd25406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:25.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 --2- 172.21.15.158:0/1709730221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2540728a0 0x7fd25406a8d0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7fd2540d1420 tx=0x7fd24c00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:25.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 -- 172.21.15.158:0/1709730221 shutdown_connections 2023-10-16T11:20:25.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 --2- 172.21.15.158:0/1709730221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2540728a0 0x7fd25406a8d0 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 --2- 172.21.15.158:0/1709730221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25406e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 --2- 172.21.15.158:0/1709730221 >> v2:172.21.15.172:3300/0 conn(0x7fd25406e810 0x7fd254072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 -- 172.21.15.158:0/1709730221 >> 172.21.15.158:0/1709730221 conn(0x7fd2540ccda0 msgr2=0x7fd2540cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:25.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 -- 172.21.15.158:0/1709730221 shutdown_connections 2023-10-16T11:20:25.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.894+0000 7fd258ca8700 1 -- 172.21.15.158:0/1709730221 wait complete. 2023-10-16T11:20:25.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.896+0000 7fd258ca8700 1 Processor -- start 2023-10-16T11:20:25.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.896+0000 7fd258ca8700 1 -- start start 2023-10-16T11:20:25.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd25406e810 0x7fd2541e9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:25.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25415d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:25.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2540728a0 0x7fd2540b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:25.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2540ce000 con 0x7fd25406e810 2023-10-16T11:20:25.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2541f2ce0 con 0x7fd2540728a0 2023-10-16T11:20:25.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2541e91f0 con 0x7fd25406f470 2023-10-16T11:20:25.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd25406e810 0x7fd2541e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:25.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd25406e810 0x7fd2541e9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40232/0 (socket says 172.21.15.158:40232) 2023-10-16T11:20:25.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 -- 172.21.15.158:0/2134769958 learned_addr learned my addr 172.21.15.158:0/2134769958 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:25.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd252d9d700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2540728a0 0x7fd2540b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:25.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd251d9b700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25415d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:25.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 -- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 msgr2=0x7fd25415d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:25.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25415d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 -- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2540728a0 msgr2=0x7fd2540b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:25.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2540728a0 0x7fd2540b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:25.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 -- 172.21.15.158:0/2134769958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c009b30 con 0x7fd25406e810 2023-10-16T11:20:25.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd25259c700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd25406e810 0x7fd2541e9440 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fd24800e730 tx=0x7fd24800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:25.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd242ffd700 1 -- 172.21.15.158:0/2134769958 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24800c070 con 0x7fd25406e810 2023-10-16T11:20:25.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd242ffd700 1 -- 172.21.15.158:0/2134769958 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd248013610 con 0x7fd25406e810 2023-10-16T11:20:25.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2541e8c30 con 0x7fd25406e810 2023-10-16T11:20:25.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd242ffd700 1 -- 172.21.15.158:0/2134769958 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2480139c0 con 0x7fd25406e810 2023-10-16T11:20:25.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.897+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2541e9980 con 0x7fd25406e810 2023-10-16T11:20:25.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.898+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2540ce2e0 con 0x7fd25406e810 2023-10-16T11:20:25.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.899+0000 7fd242ffd700 1 -- 172.21.15.158:0/2134769958 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd248013b20 con 0x7fd25406e810 2023-10-16T11:20:25.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.899+0000 7fd242ffd700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd23406b140 0x7fd23406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:25.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.899+0000 7fd242ffd700 1 -- 172.21.15.158:0/2134769958 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fd248089950 con 0x7fd25406e810 2023-10-16T11:20:25.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.899+0000 7fd251d9b700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd23406b140 0x7fd23406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:25.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.900+0000 7fd251d9b700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd23406b140 0x7fd23406d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2540d1060 tx=0x7fd244008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:25.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:25.904+0000 7fd242ffd700 1 -- 172.21.15.158:0/2134769958 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2480541d0 con 0x7fd25406e810 2023-10-16T11:20:26.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.097+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2540681b0 con 0x7fd25406e810 2023-10-16T11:20:26.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.098+0000 7fd242ffd700 1 -- 172.21.15.158:0/2134769958 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17544 (secure 0 0 0) 0x7fd2480577f0 con 0x7fd25406e810 2023-10-16T11:20:26.100 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:26.101 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":236,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:25.120456+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd23406b140 msgr2=0x7fd23406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd23406b140 0x7fd23406d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2540d1060 tx=0x7fd244008040 comp rx=0 tx=0).stop 2023-10-16T11:20:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd25406e810 msgr2=0x7fd2541e9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd25406e810 0x7fd2541e9440 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fd24800e730 tx=0x7fd24800ea40 comp rx=0 tx=0).stop 2023-10-16T11:20:26.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 shutdown_connections 2023-10-16T11:20:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd23406b140 0x7fd23406d600 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2540728a0 0x7fd2540b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd25406e810 0x7fd2541e9440 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 --2- 172.21.15.158:0/2134769958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd25406f470 0x7fd25415d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 >> 172.21.15.158:0/2134769958 conn(0x7fd2540ccda0 msgr2=0x7fd2540ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:26.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 shutdown_connections 2023-10-16T11:20:26.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.101+0000 7fd258ca8700 1 -- 172.21.15.158:0/2134769958 wait complete. 2023-10-16T11:20:26.115 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:26.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.235+0000 7f7776237700 1 Processor -- start 2023-10-16T11:20:26.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.235+0000 7f7776237700 1 -- start start 2023-10-16T11:20:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.236+0000 7f7776237700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f777006ab40 0x7f777006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.236+0000 7f7776237700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7770161270 con 0x7f777006e5d0 2023-10-16T11:20:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.236+0000 7f7776237700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7770161510 con 0x7f77700729b0 2023-10-16T11:20:26.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.236+0000 7f7776237700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77701617f0 con 0x7f777006ab40 2023-10-16T11:20:26.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.236+0000 7f776f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f777006ab40 0x7f777006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.236+0000 7f776effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f777006e5d0 0x7f77700722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40416/0 (socket says 172.21.15.158:40416) 2023-10-16T11:20:26.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.236+0000 7f776effd700 1 -- 172.21.15.158:0/1417474978 learned_addr learned my addr 172.21.15.158:0/1417474978 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:26.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776f7fe700 1 -- 172.21.15.158:0/1417474978 >> v1:172.21.15.172:6789/0 conn(0x7f77700729b0 legacy=0x7f777015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:26.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776f7fe700 1 -- 172.21.15.158:0/1417474978 >> v1:172.21.15.158:6789/0 conn(0x7f777006e5d0 legacy=0x7f77700722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776e7fc700 1 -- 172.21.15.158:0/1417474978 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619870884 0 0) 0x7f7770161270 con 0x7f777006e5d0 2023-10-16T11:20:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776f7fe700 1 -- 172.21.15.158:0/1417474978 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7770161ad0 con 0x7f777006ab40 2023-10-16T11:20:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776f7fe700 1 --2- 172.21.15.158:0/1417474978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f777006ab40 0x7f777006d820 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f7758009840 tx=0x7f7758009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d89ec561dcf77d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776e7fc700 1 -- 172.21.15.158:0/1417474978 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758004230 con 0x7f777006ab40 2023-10-16T11:20:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776e7fc700 1 -- 172.21.15.158:0/1417474978 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758004390 con 0x7f777006ab40 2023-10-16T11:20:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f776e7fc700 1 -- 172.21.15.158:0/1417474978 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77580036a0 con 0x7f777006ab40 2023-10-16T11:20:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f7776237700 1 -- 172.21.15.158:0/1417474978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f777006ab40 msgr2=0x7f777006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f7776237700 1 --2- 172.21.15.158:0/1417474978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f777006ab40 0x7f777006d820 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f7758009840 tx=0x7f7758009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f7776237700 1 -- 172.21.15.158:0/1417474978 shutdown_connections 2023-10-16T11:20:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f7776237700 1 --2- 172.21.15.158:0/1417474978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f777006ab40 0x7f777006d820 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f7776237700 1 -- 172.21.15.158:0/1417474978 >> 172.21.15.158:0/1417474978 conn(0x7f77700d02b0 msgr2=0x7f77700d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.237+0000 7f7776237700 1 -- 172.21.15.158:0/1417474978 shutdown_connections 2023-10-16T11:20:26.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.238+0000 7f7776237700 1 -- 172.21.15.158:0/1417474978 wait complete. 2023-10-16T11:20:26.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.239+0000 7f7776237700 1 Processor -- start 2023-10-16T11:20:26.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.239+0000 7f7776237700 1 -- start start 2023-10-16T11:20:26.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f7776237700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f777006ab40 0x7f777014e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f7776237700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f777006e5d0 0x7f77700cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f7776237700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77700729b0 0x7f77701f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f7776237700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77701f3e90 con 0x7f77700729b0 2023-10-16T11:20:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f7776237700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777015e720 con 0x7f777006ab40 2023-10-16T11:20:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f7776237700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777014e3a0 con 0x7f777006e5d0 2023-10-16T11:20:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f776ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77700729b0 0x7f77701f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f776ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77700729b0 0x7f77701f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40250/0 (socket says 172.21.15.158:40250) 2023-10-16T11:20:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f776ffff700 1 -- 172.21.15.158:0/1930661703 learned_addr learned my addr 172.21.15.158:0/1930661703 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f776f7fe700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f777006ab40 0x7f777014e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.240+0000 7f776effd700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f777006e5d0 0x7f77700cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f776ffff700 1 -- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f777006e5d0 msgr2=0x7f77700cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f776ffff700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f777006e5d0 0x7f77700cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f776ffff700 1 -- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f777006ab40 msgr2=0x7f777014e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f776ffff700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f777006ab40 0x7f777014e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f776ffff700 1 -- 172.21.15.158:0/1930661703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77580094a0 con 0x7f77700729b0 2023-10-16T11:20:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f776ffff700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77700729b0 0x7f77701f2ea0 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f776800b9e0 tx=0x7f776800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:26.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f7767fff700 1 -- 172.21.15.158:0/1930661703 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776800c9b0 con 0x7f77700729b0 2023-10-16T11:20:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f7767fff700 1 -- 172.21.15.158:0/1930661703 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776800cb50 con 0x7f77700729b0 2023-10-16T11:20:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777014ed70 con 0x7f77700729b0 2023-10-16T11:20:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f7767fff700 1 -- 172.21.15.158:0/1930661703 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768014440 con 0x7f77700729b0 2023-10-16T11:20:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.241+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77701f3bb0 con 0x7f77700729b0 2023-10-16T11:20:26.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.242+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777014e530 con 0x7f77700729b0 2023-10-16T11:20:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.243+0000 7f7767fff700 1 -- 172.21.15.158:0/1930661703 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f77680145a0 con 0x7f77700729b0 2023-10-16T11:20:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.243+0000 7f7767fff700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f775006b140 0x7f775006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.243+0000 7f7767fff700 1 -- 172.21.15.158:0/1930661703 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f7768089fe0 con 0x7f77700729b0 2023-10-16T11:20:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.243+0000 7f776f7fe700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f775006b140 0x7f775006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.244+0000 7f776f7fe700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f775006b140 0x7f775006d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f77700cba50 tx=0x7f775801c040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:26.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.247+0000 7f7767fff700 1 -- 172.21.15.158:0/1930661703 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7768054b70 con 0x7f77700729b0 2023-10-16T11:20:26.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.439+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77700681b0 con 0x7f77700729b0 2023-10-16T11:20:26.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.440+0000 7f7767fff700 1 -- 172.21.15.158:0/1930661703 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17544 (secure 0 0 0) 0x7f7768081e60 con 0x7f77700729b0 2023-10-16T11:20:26.442 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:26.442 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":236,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:25.120456+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:26.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f775006b140 msgr2=0x7f775006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f775006b140 0x7f775006d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f77700cba50 tx=0x7f775801c040 comp rx=0 tx=0).stop 2023-10-16T11:20:26.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77700729b0 msgr2=0x7f77701f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77700729b0 0x7f77701f2ea0 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f776800b9e0 tx=0x7f776800bda0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 shutdown_connections 2023-10-16T11:20:26.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f775006b140 0x7f775006d600 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77700729b0 0x7f77701f2ea0 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f777006ab40 0x7f777014e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 --2- 172.21.15.158:0/1930661703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f777006e5d0 0x7f77700cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 >> 172.21.15.158:0/1930661703 conn(0x7f77700d02b0 msgr2=0x7f77700722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:26.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 shutdown_connections 2023-10-16T11:20:26.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.442+0000 7f7776237700 1 -- 172.21.15.158:0/1930661703 wait complete. 2023-10-16T11:20:26.457 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T11:20:26.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.579+0000 7f0bc8caf700 1 Processor -- start 2023-10-16T11:20:26.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.579+0000 7f0bc8caf700 1 -- start start 2023-10-16T11:20:26.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.580+0000 7f0bc8caf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc406d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.580+0000 7f0bc8caf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0bc406d950 0x7f0bc40c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.580+0000 7f0bc8caf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc40cf930 con 0x7f0bc4151220 2023-10-16T11:20:26.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.580+0000 7f0bc8caf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc40cf650 con 0x7f0bc406d950 2023-10-16T11:20:26.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.580+0000 7f0bc8caf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40c51b0 con 0x7f0bc40cf190 2023-10-16T11:20:26.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0bc406d950 0x7f0bc40c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0bc406d950 0x7f0bc40c5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40258/0 (socket says 172.21.15.158:40258) 2023-10-16T11:20:26.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 -- 172.21.15.158:0/3135709264 learned_addr learned my addr 172.21.15.158:0/3135709264 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:26.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc1d9b700 1 --2- 172.21.15.158:0/3135709264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 -- 172.21.15.158:0/3135709264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 msgr2=0x7f0bc406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 --2- 172.21.15.158:0/3135709264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc406d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 -- 172.21.15.158:0/3135709264 >> v1:172.21.15.172:6789/0 conn(0x7f0bc4151220 legacy=0x7f0bc40665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 -- 172.21.15.158:0/3135709264 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40c5490 con 0x7f0bc406d950 2023-10-16T11:20:26.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc2d9d700 1 --2- 172.21.15.158:0/3135709264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40c5c90 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00d150 tx=0x7f0bbc00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e7da8005d657b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc159a700 1 -- 172.21.15.158:0/3135709264 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc010070 con 0x7f0bc406d950 2023-10-16T11:20:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc159a700 1 -- 172.21.15.158:0/3135709264 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc00dd80 con 0x7f0bc406d950 2023-10-16T11:20:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.581+0000 7f0bc159a700 1 -- 172.21.15.158:0/3135709264 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00b310 con 0x7f0bc406d950 2023-10-16T11:20:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 -- 172.21.15.158:0/3135709264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 msgr2=0x7f0bc40c5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/3135709264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40c5c90 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00d150 tx=0x7f0bbc00d460 comp rx=0 tx=0).stop 2023-10-16T11:20:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 -- 172.21.15.158:0/3135709264 shutdown_connections 2023-10-16T11:20:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/3135709264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40c5c90 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/3135709264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc406d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 -- 172.21.15.158:0/3135709264 >> 172.21.15.158:0/3135709264 conn(0x7f0bc40cc5b0 msgr2=0x7f0bc40cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 -- 172.21.15.158:0/3135709264 shutdown_connections 2023-10-16T11:20:26.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.582+0000 7f0bc8caf700 1 -- 172.21.15.158:0/3135709264 wait complete. 2023-10-16T11:20:26.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.584+0000 7f0bc8caf700 1 Processor -- start 2023-10-16T11:20:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.584+0000 7f0bc8caf700 1 -- start start 2023-10-16T11:20:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.584+0000 7f0bc8caf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.584+0000 7f0bc8caf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc414c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc8caf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bc4151220 0x7f0bc414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc8caf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b27d0 con 0x7f0bc406d950 2023-10-16T11:20:26.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc8caf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40cb970 con 0x7f0bc4151220 2023-10-16T11:20:26.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc8caf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b29e0 con 0x7f0bc40cf190 2023-10-16T11:20:26.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40262/0 (socket says 172.21.15.158:40262) 2023-10-16T11:20:26.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 -- 172.21.15.158:0/1385095503 learned_addr learned my addr 172.21.15.158:0/1385095503 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:26.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc1d9b700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc414c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc2d9d700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bc4151220 0x7f0bc414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 -- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 msgr2=0x7f0bc414c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc414c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 -- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bc4151220 msgr2=0x7f0bc414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bc4151220 0x7f0bc414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 -- 172.21.15.158:0/1385095503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc00bd40 con 0x7f0bc406d950 2023-10-16T11:20:26.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc2d9d700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bc4151220 0x7f0bc414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc259c700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40cb430 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400b4a0 tx=0x7f0bb400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bb2ffd700 1 -- 172.21.15.158:0/1385095503 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4011830 con 0x7f0bc406d950 2023-10-16T11:20:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bb2ffd700 1 -- 172.21.15.158:0/1385095503 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb4011990 con 0x7f0bc406d950 2023-10-16T11:20:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc40b2cc0 con 0x7f0bc406d950 2023-10-16T11:20:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bb2ffd700 1 -- 172.21.15.158:0/1385095503 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4011d40 con 0x7f0bc406d950 2023-10-16T11:20:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.585+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc40b3280 con 0x7f0bc406d950 2023-10-16T11:20:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.586+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc40681b0 con 0x7f0bc406d950 2023-10-16T11:20:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.587+0000 7f0bb2ffd700 1 -- 172.21.15.158:0/1385095503 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0bb401a400 con 0x7f0bc406d950 2023-10-16T11:20:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.587+0000 7f0bb2ffd700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0ba406b140 0x7f0ba406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.587+0000 7f0bc1d9b700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0ba406b140 0x7f0ba406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.587+0000 7f0bb2ffd700 1 -- 172.21.15.158:0/1385095503 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f0bb4089a60 con 0x7f0bc406d950 2023-10-16T11:20:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.588+0000 7f0bc1d9b700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0ba406b140 0x7f0ba406d600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009930 tx=0x7f0bb8008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:26.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.592+0000 7f0bb2ffd700 1 -- 172.21.15.158:0/1385095503 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bb4054490 con 0x7f0bc406d950 2023-10-16T11:20:26.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:26.834+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0bc40723f0 con 0x7f0bc406d950 2023-10-16T11:20:27.711 INFO:tasks.ceph.mds.d.smithi172.stderr:2023-10-16T11:20:27.709+0000 7f00bcdbf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:20:27.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.759+0000 7f0bb2ffd700 1 -- 172.21.15.158:0/1385095503 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v128) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0bb4057ab0 con 0x7f0bc406d950 2023-10-16T11:20:27.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0ba406b140 msgr2=0x7f0ba406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:27.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0ba406b140 0x7f0ba406d600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009930 tx=0x7f0bb8008040 comp rx=0 tx=0).stop 2023-10-16T11:20:27.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 msgr2=0x7f0bc40cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:27.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40cb430 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400b4a0 tx=0x7f0bb400b860 comp rx=0 tx=0).stop 2023-10-16T11:20:27.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 shutdown_connections 2023-10-16T11:20:27.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0ba406b140 0x7f0ba406d600 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0bc4151220 0x7f0bc414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0bc406d950 0x7f0bc40cb430 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 --2- 172.21.15.158:0/1385095503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0bc40cf190 0x7f0bc414c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 >> 172.21.15.158:0/1385095503 conn(0x7f0bc40cc5b0 msgr2=0x7f0bc40665a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:27.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 shutdown_connections 2023-10-16T11:20:27.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.763+0000 7f0bc8caf700 1 -- 172.21.15.158:0/1385095503 wait complete. 2023-10-16T11:20:27.779 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:20:27.780 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T11:20:27.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.907+0000 7f675dacc700 1 Processor -- start 2023-10-16T11:20:27.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.907+0000 7f675dacc700 1 -- start start 2023-10-16T11:20:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f675dacc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f675815b590 0x7f67580ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f675dacc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f675815f1d0 0x7f67580b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f675dacc700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67580cf4c0 con 0x7f67580cbaf0 2023-10-16T11:20:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f675dacc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675815f970 con 0x7f675815f1d0 2023-10-16T11:20:27.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f675dacc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675815fc50 con 0x7f675815b590 2023-10-16T11:20:27.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f6756ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f675815f1d0 0x7f67580b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:27.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f6757fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f67580cbaf0 0x7f67580cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40448/0 (socket says 172.21.15.158:40448) 2023-10-16T11:20:27.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f6756ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f675815f1d0 0x7f67580b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40268/0 (socket says 172.21.15.158:40268) 2023-10-16T11:20:27.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.908+0000 7f6757fff700 1 -- 172.21.15.158:0/1516724491 learned_addr learned my addr 172.21.15.158:0/1516724491 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:27.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f67577fe700 1 --2- 172.21.15.158:0/1516724491 >> v2:172.21.15.172:3300/0 conn(0x7f675815b590 0x7f67580ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:27.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f6756ffd700 1 -- 172.21.15.158:0/1516724491 >> v2:172.21.15.172:3300/0 conn(0x7f675815b590 msgr2=0x7f67580ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:27.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f6756ffd700 1 --2- 172.21.15.158:0/1516724491 >> v2:172.21.15.172:3300/0 conn(0x7f675815b590 0x7f67580ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f6756ffd700 1 -- 172.21.15.158:0/1516724491 >> v1:172.21.15.158:6789/0 conn(0x7f67580cbaf0 legacy=0x7f67580cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:27.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f6756ffd700 1 -- 172.21.15.158:0/1516724491 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675815ff30 con 0x7f675815f1d0 2023-10-16T11:20:27.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f67567fc700 1 -- 172.21.15.158:0/1516724491 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559775965 0 0) 0x7f67580cf4c0 con 0x7f67580cbaf0 2023-10-16T11:20:27.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f6756ffd700 1 --2- 172.21.15.158:0/1516724491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f675815f1d0 0x7f67580b42d0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f67580b3520 tx=0x7f6748009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c88960f04476908 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:27.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f67567fc700 1 -- 172.21.15.158:0/1516724491 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748011830 con 0x7f675815f1d0 2023-10-16T11:20:27.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f67567fc700 1 -- 172.21.15.158:0/1516724491 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6748011990 con 0x7f675815f1d0 2023-10-16T11:20:27.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f67567fc700 1 -- 172.21.15.158:0/1516724491 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748011d20 con 0x7f675815f1d0 2023-10-16T11:20:27.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f675dacc700 1 -- 172.21.15.158:0/1516724491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f675815f1d0 msgr2=0x7f67580b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:27.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f675dacc700 1 --2- 172.21.15.158:0/1516724491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f675815f1d0 0x7f67580b42d0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f67580b3520 tx=0x7f6748009b20 comp rx=0 tx=0).stop 2023-10-16T11:20:27.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f675dacc700 1 -- 172.21.15.158:0/1516724491 shutdown_connections 2023-10-16T11:20:27.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f675dacc700 1 --2- 172.21.15.158:0/1516724491 >> v2:172.21.15.172:3300/0 conn(0x7f675815b590 0x7f67580ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f675dacc700 1 --2- 172.21.15.158:0/1516724491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f675815f1d0 0x7f67580b42d0 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.909+0000 7f675dacc700 1 -- 172.21.15.158:0/1516724491 >> 172.21.15.158:0/1516724491 conn(0x7f675806a8d0 msgr2=0x7f675806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:27.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.910+0000 7f675dacc700 1 -- 172.21.15.158:0/1516724491 shutdown_connections 2023-10-16T11:20:27.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.910+0000 7f675dacc700 1 -- 172.21.15.158:0/1516724491 wait complete. 2023-10-16T11:20:27.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.911+0000 7f675dacc700 1 Processor -- start 2023-10-16T11:20:27.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.911+0000 7f675dacc700 1 -- start start 2023-10-16T11:20:27.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67580cbaf0 0x7f675814d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:27.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f675815b590 0x7f67580b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:27.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f675815f1d0 0x7f67580b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:27.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675814dde0 con 0x7f67580cbaf0 2023-10-16T11:20:27.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675815d110 con 0x7f675815f1d0 2023-10-16T11:20:27.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675815d3f0 con 0x7f675815b590 2023-10-16T11:20:27.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67580cbaf0 0x7f675814d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:27.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67580cbaf0 0x7f675814d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40274/0 (socket says 172.21.15.158:40274) 2023-10-16T11:20:27.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 -- 172.21.15.158:0/3119609311 learned_addr learned my addr 172.21.15.158:0/3119609311 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:27.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f6757fff700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f675815f1d0 0x7f67580b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:27.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f6756ffd700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f675815b590 0x7f67580b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:27.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 -- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f675815b590 msgr2=0x7f67580b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:27.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f675815b590 0x7f67580b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 -- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f675815f1d0 msgr2=0x7f67580b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:27.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f675815f1d0 0x7f67580b61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:27.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 -- 172.21.15.158:0/3119609311 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67480094a0 con 0x7f67580cbaf0 2023-10-16T11:20:27.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f67577fe700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67580cbaf0 0x7f675814d010 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f674c00e810 tx=0x7f674c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:27.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f673ffff700 1 -- 172.21.15.158:0/3119609311 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c013610 con 0x7f67580cbaf0 2023-10-16T11:20:27.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f673ffff700 1 -- 172.21.15.158:0/3119609311 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c013770 con 0x7f67580cbaf0 2023-10-16T11:20:27.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f673ffff700 1 -- 172.21.15.158:0/3119609311 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c013b20 con 0x7f67580cbaf0 2023-10-16T11:20:27.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675815d6d0 con 0x7f67580cbaf0 2023-10-16T11:20:27.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.912+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675806ba80 con 0x7f67580cbaf0 2023-10-16T11:20:27.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.913+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67580681b0 con 0x7f67580cbaf0 2023-10-16T11:20:27.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.914+0000 7f673ffff700 1 -- 172.21.15.158:0/3119609311 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f674c013c80 con 0x7f67580cbaf0 2023-10-16T11:20:27.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.914+0000 7f673ffff700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f674406b140 0x7f674406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:27.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.914+0000 7f673ffff700 1 -- 172.21.15.158:0/3119609311 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f674c089ab0 con 0x7f67580cbaf0 2023-10-16T11:20:27.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.914+0000 7f6756ffd700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f674406b140 0x7f674406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:27.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.915+0000 7f6756ffd700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f674406b140 0x7f674406d600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f67480090b0 tx=0x7f674801c040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:27.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:27.919+0000 7f673ffff700 1 -- 172.21.15.158:0/3119609311 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f674c054340 con 0x7f67580cbaf0 2023-10-16T11:20:28.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.156+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f67580b07c0 con 0x7f67580cbaf0 2023-10-16T11:20:28.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.761+0000 7f673ffff700 1 -- 172.21.15.158:0/3119609311 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v129) v1 ==== 122+0+0 (secure 0 0 0) 0x7f674c057960 con 0x7f67580cbaf0 2023-10-16T11:20:28.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f674406b140 msgr2=0x7f674406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:28.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f674406b140 0x7f674406d600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f67480090b0 tx=0x7f674801c040 comp rx=0 tx=0).stop 2023-10-16T11:20:28.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67580cbaf0 msgr2=0x7f675814d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:28.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67580cbaf0 0x7f675814d010 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f674c00e810 tx=0x7f674c00eb20 comp rx=0 tx=0).stop 2023-10-16T11:20:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 shutdown_connections 2023-10-16T11:20:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f674406b140 0x7f674406d600 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f675815f1d0 0x7f67580b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67580cbaf0 0x7f675814d010 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 --2- 172.21.15.158:0/3119609311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f675815b590 0x7f67580b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 >> 172.21.15.158:0/3119609311 conn(0x7f675806a8d0 msgr2=0x7f67580b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 shutdown_connections 2023-10-16T11:20:28.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.764+0000 7f675dacc700 1 -- 172.21.15.158:0/3119609311 wait complete. 2023-10-16T11:20:28.779 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T11:20:28.780 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.901+0000 7f6ac2fbd700 1 Processor -- start 2023-10-16T11:20:28.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.901+0000 7f6ac2fbd700 1 -- start start 2023-10-16T11:20:28.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6ac2fbd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6abc06e5d0 0x7f6abc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6ac2fbd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 0x7f6abc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6ac2fbd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc071e50 con 0x7f6abc06ab40 2023-10-16T11:20:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6ac2fbd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc161270 con 0x7f6abc06e5d0 2023-10-16T11:20:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6ac2fbd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc161510 con 0x7f6abc0729b0 2023-10-16T11:20:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6abc06e5d0 0x7f6abc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:28.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6abc06e5d0 0x7f6abc06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40284/0 (socket says 172.21.15.158:40284) 2023-10-16T11:20:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6ac0d59700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6abc06ab40 0x7f6abc0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40458/0 (socket says 172.21.15.158:40458) 2023-10-16T11:20:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 -- 172.21.15.158:0/2047543120 learned_addr learned my addr 172.21.15.158:0/2047543120 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 -- 172.21.15.158:0/2047543120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 msgr2=0x7f6abc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 --2- 172.21.15.158:0/2047543120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 0x7f6abc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 -- 172.21.15.158:0/2047543120 >> v1:172.21.15.158:6789/0 conn(0x7f6abc06ab40 legacy=0x7f6abc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abb7fe700 1 -- 172.21.15.158:0/2047543120 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839590152 0 0) 0x7f6abc071e50 con 0x7f6abc06ab40 2023-10-16T11:20:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 -- 172.21.15.158:0/2047543120 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc1617f0 con 0x7f6abc06e5d0 2023-10-16T11:20:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abbfff700 1 --2- 172.21.15.158:0/2047543120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06e5d0 0x7f6abc06d820 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80098f0 tx=0x7f6aa8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8e70dcb0ef2ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.902+0000 7f6abb7fe700 1 -- 172.21.15.158:0/2047543120 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa800b8a0 con 0x7f6abc06e5d0 2023-10-16T11:20:28.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6abb7fe700 1 -- 172.21.15.158:0/2047543120 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa800ba00 con 0x7f6abc06e5d0 2023-10-16T11:20:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6abb7fe700 1 -- 172.21.15.158:0/2047543120 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa800bd90 con 0x7f6abc06e5d0 2023-10-16T11:20:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/2047543120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06e5d0 msgr2=0x7f6abc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/2047543120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06e5d0 0x7f6abc06d820 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80098f0 tx=0x7f6aa8009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/2047543120 shutdown_connections 2023-10-16T11:20:28.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/2047543120 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 0x7f6abc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/2047543120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06e5d0 0x7f6abc06d820 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/2047543120 >> 172.21.15.158:0/2047543120 conn(0x7f6abc0d02b0 msgr2=0x7f6abc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/2047543120 shutdown_connections 2023-10-16T11:20:28.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.903+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/2047543120 wait complete. 2023-10-16T11:20:28.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.904+0000 7f6ac2fbd700 1 Processor -- start 2023-10-16T11:20:28.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.904+0000 7f6ac2fbd700 1 -- start start 2023-10-16T11:20:28.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac2fbd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06ab40 0x7f6abc155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:28.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac2fbd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc06e5d0 0x7f6abc0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:28.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac2fbd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 0x7f6abc0c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:28.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac2fbd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0b5db0 con 0x7f6abc06ab40 2023-10-16T11:20:28.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac2fbd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0c9450 con 0x7f6abc06e5d0 2023-10-16T11:20:28.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac2fbd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0c9730 con 0x7f6abc0729b0 2023-10-16T11:20:28.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac0d59700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06ab40 0x7f6abc155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:28.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac0d59700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06ab40 0x7f6abc155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40290/0 (socket says 172.21.15.158:40290) 2023-10-16T11:20:28.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.905+0000 7f6ac0d59700 1 -- 172.21.15.158:0/681501956 learned_addr learned my addr 172.21.15.158:0/681501956 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:28.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6abbfff700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc06e5d0 0x7f6abc0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:28.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac155a700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 0x7f6abc0c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:28.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac0d59700 1 -- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 msgr2=0x7f6abc0c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:28.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac0d59700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 0x7f6abc0c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac0d59700 1 -- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc06e5d0 msgr2=0x7f6abc0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:28.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac0d59700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc06e5d0 0x7f6abc0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:28.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac0d59700 1 -- 172.21.15.158:0/681501956 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa8009550 con 0x7f6abc06ab40 2023-10-16T11:20:28.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac0d59700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06ab40 0x7f6abc155120 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00011e0 tx=0x7f6ab000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:28.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ab97fa700 1 -- 172.21.15.158:0/681501956 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0004c80 con 0x7f6abc06ab40 2023-10-16T11:20:28.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ab97fa700 1 -- 172.21.15.158:0/681501956 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab0004e00 con 0x7f6abc06ab40 2023-10-16T11:20:28.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc0c9a10 con 0x7f6abc06ab40 2023-10-16T11:20:28.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ab97fa700 1 -- 172.21.15.158:0/681501956 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0007310 con 0x7f6abc06ab40 2023-10-16T11:20:28.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.906+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc0c9fd0 con 0x7f6abc06ab40 2023-10-16T11:20:28.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.907+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc0681b0 con 0x7f6abc06ab40 2023-10-16T11:20:28.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.908+0000 7f6ab97fa700 1 -- 172.21.15.158:0/681501956 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6ab0007470 con 0x7f6abc06ab40 2023-10-16T11:20:28.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.908+0000 7f6ab97fa700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6aa006b140 0x7f6aa006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:28.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.908+0000 7f6ab97fa700 1 -- 172.21.15.158:0/681501956 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f6ab0089900 con 0x7f6abc06ab40 2023-10-16T11:20:28.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.908+0000 7f6abbfff700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6aa006b140 0x7f6aa006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:28.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.908+0000 7f6abbfff700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6aa006b140 0x7f6aa006d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80098f0 tx=0x7f6aa8008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:28.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:28.912+0000 7f6ab97fa700 1 -- 172.21.15.158:0/681501956 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ab00543c0 con 0x7f6abc06ab40 2023-10-16T11:20:29.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.147+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6abc161aa0 con 0x7f6abc06ab40 2023-10-16T11:20:29.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.147+0000 7f6ab97fa700 1 -- 172.21.15.158:0/681501956 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f6ab00579e0 con 0x7f6abc06ab40 2023-10-16T11:20:29.149 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:29.150 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":12432,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T11:20:29.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.149+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6aa006b140 msgr2=0x7f6aa006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.149+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6aa006b140 0x7f6aa006d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80098f0 tx=0x7f6aa8008030 comp rx=0 tx=0).stop 2023-10-16T11:20:29.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.149+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06ab40 msgr2=0x7f6abc155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.149+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06ab40 0x7f6abc155120 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00011e0 tx=0x7f6ab000ca30 comp rx=0 tx=0).stop 2023-10-16T11:20:29.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 shutdown_connections 2023-10-16T11:20:29.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6aa006b140 0x7f6aa006d600 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6abc0729b0 0x7f6abc0c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6abc06ab40 0x7f6abc155120 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 --2- 172.21.15.158:0/681501956 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6abc06e5d0 0x7f6abc0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 >> 172.21.15.158:0/681501956 conn(0x7f6abc0d02b0 msgr2=0x7f6abc1576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:29.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 shutdown_connections 2023-10-16T11:20:29.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.150+0000 7f6ac2fbd700 1 -- 172.21.15.158:0/681501956 wait complete. 2023-10-16T11:20:29.154 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 129 2023-10-16T11:20:29.164 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:29.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.284+0000 7f076c64f700 1 Processor -- start 2023-10-16T11:20:29.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.284+0000 7f076c64f700 1 -- start start 2023-10-16T11:20:29.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076c64f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f076406d820 0x7f076415f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076c64f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0764072700 con 0x7f07640cb010 2023-10-16T11:20:29.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076c64f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07640cca90 con 0x7f076406b0a0 2023-10-16T11:20:29.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076c64f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0764071910 con 0x7f076406d820 2023-10-16T11:20:29.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076abec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f076406d820 0x7f076415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f0769bea700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f07640cb010 0x7f076415bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40472/0 (socket says 172.21.15.158:40472) 2023-10-16T11:20:29.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076abec700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f076406d820 0x7f076415f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40300/0 (socket says 172.21.15.158:40300) 2023-10-16T11:20:29.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f0769bea700 1 -- 172.21.15.158:0/1166745028 learned_addr learned my addr 172.21.15.158:0/1166745028 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:29.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076abec700 1 -- 172.21.15.158:0/1166745028 >> v1:172.21.15.172:6789/0 conn(0x7f076406b0a0 legacy=0x7f07640ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:29.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076abec700 1 -- 172.21.15.158:0/1166745028 >> v1:172.21.15.158:6789/0 conn(0x7f07640cb010 legacy=0x7f076415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076abec700 1 -- 172.21.15.158:0/1166745028 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0764071af0 con 0x7f076406d820 2023-10-16T11:20:29.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f07693e9700 1 -- 172.21.15.158:0/1166745028 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358419995 0 0) 0x7f0764072700 con 0x7f07640cb010 2023-10-16T11:20:29.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f076abec700 1 --2- 172.21.15.158:0/1166745028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f076415f010 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f075c009d90 tx=0x7f075c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbef5da9d054224 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:29.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f07693e9700 1 -- 172.21.15.158:0/1166745028 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00e070 con 0x7f076406d820 2023-10-16T11:20:29.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f07693e9700 1 -- 172.21.15.158:0/1166745028 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c00bb90 con 0x7f076406d820 2023-10-16T11:20:29.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.285+0000 7f07693e9700 1 -- 172.21.15.158:0/1166745028 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c003a20 con 0x7f076406d820 2023-10-16T11:20:29.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.286+0000 7f076c64f700 1 -- 172.21.15.158:0/1166745028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 msgr2=0x7f076415f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.286+0000 7f076c64f700 1 --2- 172.21.15.158:0/1166745028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f076415f010 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f075c009d90 tx=0x7f075c00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:29.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.286+0000 7f076c64f700 1 -- 172.21.15.158:0/1166745028 shutdown_connections 2023-10-16T11:20:29.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.286+0000 7f076c64f700 1 --2- 172.21.15.158:0/1166745028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f076415f010 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.286+0000 7f076c64f700 1 -- 172.21.15.158:0/1166745028 >> 172.21.15.158:0/1166745028 conn(0x7f07640d02b0 msgr2=0x7f07640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:29.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.286+0000 7f076c64f700 1 -- 172.21.15.158:0/1166745028 shutdown_connections 2023-10-16T11:20:29.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.286+0000 7f076c64f700 1 -- 172.21.15.158:0/1166745028 wait complete. 2023-10-16T11:20:29.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.287+0000 7f076c64f700 1 Processor -- start 2023-10-16T11:20:29.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.287+0000 7f076c64f700 1 -- start start 2023-10-16T11:20:29.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076c64f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f076406b0a0 0x7f076415cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076c64f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f07641f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076c64f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cb010 0x7f0764153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076c64f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f1d30 con 0x7f076406d820 2023-10-16T11:20:29.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076c64f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f22d0 con 0x7f076406b0a0 2023-10-16T11:20:29.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076c64f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f25b0 con 0x7f07640cb010 2023-10-16T11:20:29.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f07641f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f07641f17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40304/0 (socket says 172.21.15.158:40304) 2023-10-16T11:20:29.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 -- 172.21.15.158:0/2101364027 learned_addr learned my addr 172.21.15.158:0/2101364027 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:29.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076abec700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cb010 0x7f0764153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f076a3eb700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f076406b0a0 0x7f076415cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 -- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cb010 msgr2=0x7f0764153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cb010 0x7f0764153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 -- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f076406b0a0 msgr2=0x7f076415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f076406b0a0 0x7f076415cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 -- 172.21.15.158:0/2101364027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c009a70 con 0x7f076406d820 2023-10-16T11:20:29.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.288+0000 7f0769bea700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f07641f17f0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f07540011e0 tx=0x7f075400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:29.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.289+0000 7f075affd700 1 -- 172.21.15.158:0/2101364027 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754004c80 con 0x7f076406d820 2023-10-16T11:20:29.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.289+0000 7f075affd700 1 -- 172.21.15.158:0/2101364027 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754004e00 con 0x7f076406d820 2023-10-16T11:20:29.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.289+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07641f2890 con 0x7f076406d820 2023-10-16T11:20:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.289+0000 7f075affd700 1 -- 172.21.15.158:0/2101364027 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754007310 con 0x7f076406d820 2023-10-16T11:20:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.289+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07641f2e50 con 0x7f076406d820 2023-10-16T11:20:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.289+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07641f3070 con 0x7f076406d820 2023-10-16T11:20:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.290+0000 7f075affd700 1 -- 172.21.15.158:0/2101364027 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0754007470 con 0x7f076406d820 2023-10-16T11:20:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.290+0000 7f075affd700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f074806b140 0x7f074806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.291+0000 7f075affd700 1 -- 172.21.15.158:0/2101364027 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f0754089b40 con 0x7f076406d820 2023-10-16T11:20:29.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.291+0000 7f076a3eb700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f074806b140 0x7f074806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.291+0000 7f076a3eb700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f074806b140 0x7f074806d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0760009870 tx=0x7f0760008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:29.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.295+0000 7f075affd700 1 -- 172.21.15.158:0/2101364027 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0754054570 con 0x7f076406d820 2023-10-16T11:20:29.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.489+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07640681b0 con 0x7f076406d820 2023-10-16T11:20:29.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.490+0000 7f075affd700 1 -- 172.21.15.158:0/2101364027 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+17531 (secure 0 0 0) 0x7f0754089030 con 0x7f076406d820 2023-10-16T11:20:29.492 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:29.492 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":237,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:26.734730+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:29.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f074806b140 msgr2=0x7f074806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f074806b140 0x7f074806d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0760009870 tx=0x7f0760008040 comp rx=0 tx=0).stop 2023-10-16T11:20:29.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 msgr2=0x7f07641f17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f07641f17f0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f07540011e0 tx=0x7f075400cae0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 shutdown_connections 2023-10-16T11:20:29.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f074806b140 0x7f074806d600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cb010 0x7f0764153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f076406b0a0 0x7f076415cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 --2- 172.21.15.158:0/2101364027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076406d820 0x7f07641f17f0 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 >> 172.21.15.158:0/2101364027 conn(0x7f07640d02b0 msgr2=0x7f076415bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:29.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 shutdown_connections 2023-10-16T11:20:29.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.492+0000 7f076c64f700 1 -- 172.21.15.158:0/2101364027 wait complete. 2023-10-16T11:20:29.506 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:20:29.507 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:20:29.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.627+0000 7f584641a700 1 Processor -- start 2023-10-16T11:20:29.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.627+0000 7f584641a700 1 -- start start 2023-10-16T11:20:29.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.627+0000 7f584641a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58400af590 0x7f584006d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f584641a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 0x7f58400ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f584641a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400b1e40 con 0x7f584015a6c0 2023-10-16T11:20:29.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f584641a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584006e370 con 0x7f58400af590 2023-10-16T11:20:29.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f584641a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400ca7e0 con 0x7f584006dec0 2023-10-16T11:20:29.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583ffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f584015a6c0 0x7f58400723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40486/0 (socket says 172.21.15.158:40486) 2023-10-16T11:20:29.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583ffff700 1 -- 172.21.15.158:0/320884764 learned_addr learned my addr 172.21.15.158:0/320884764 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:29.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583f7fe700 1 --2- 172.21.15.158:0/320884764 >> v2:172.21.15.172:3300/0 conn(0x7f58400af590 0x7f584006d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f58449b7700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 0x7f58400ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57464085 0 0) 0x7f58400b1e40 con 0x7f584015a6c0 2023-10-16T11:20:29.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58280035b0 con 0x7f584015a6c0 2023-10-16T11:20:29.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3129087628 0 0) 0x7f58280035b0 con 0x7f584015a6c0 2023-10-16T11:20:29.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 msgr2=0x7f58400ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:29.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 0x7f58400ad430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 >> v2:172.21.15.172:3300/0 conn(0x7f58400af590 msgr2=0x7f584006d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:29.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 --2- 172.21.15.158:0/320884764 >> v2:172.21.15.172:3300/0 conn(0x7f58400af590 0x7f584006d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400b2120 con 0x7f584015a6c0 2023-10-16T11:20:29.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f58300017f0 con 0x7f584015a6c0 2023-10-16T11:20:29.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 >> v1:172.21.15.158:6789/0 conn(0x7f584015a6c0 legacy=0x7f58400723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828003a00 0x7f5828003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5828004850 0x7f5828004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 0x7f5828005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584006e370 con 0x7f5828004850 2023-10-16T11:20:29.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400ca7e0 con 0x7f5828003a00 2023-10-16T11:20:29.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58280079f0 con 0x7f584006dec0 2023-10-16T11:20:29.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583f7fe700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5828004850 0x7f5828004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f583ffff700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828003a00 0x7f5828003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.628+0000 7f58449b7700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 0x7f5828005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583f7fe700 1 -- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 msgr2=0x7f5828005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:29.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583f7fe700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f584006dec0 0x7f5828005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583f7fe700 1 -- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828003a00 msgr2=0x7f5828003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:29.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583f7fe700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828003a00 0x7f5828003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583f7fe700 1 -- 172.21.15.158:0/320884764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58280035b0 con 0x7f5828004850 2023-10-16T11:20:29.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583f7fe700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5828004850 0x7f5828004cd0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f5834009b30 tx=0x7f5828005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f749b972358f2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:29.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400e6c0 con 0x7f5828004850 2023-10-16T11:20:29.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583400e820 con 0x7f5828004850 2023-10-16T11:20:29.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.629+0000 7f583effd700 1 -- 172.21.15.158:0/320884764 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400eb90 con 0x7f5828004850 2023-10-16T11:20:29.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.630+0000 7f584641a700 1 -- 172.21.15.158:0/320884764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5828004850 msgr2=0x7f5828004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.630+0000 7f584641a700 1 --2- 172.21.15.158:0/320884764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5828004850 0x7f5828004cd0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f5834009b30 tx=0x7f5828005880 comp rx=0 tx=0).stop 2023-10-16T11:20:29.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.630+0000 7f58449b7700 1 -- 172.21.15.158:0/320884764 reap_dead start 2023-10-16T11:20:29.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.630+0000 7f584641a700 1 -- 172.21.15.158:0/320884764 shutdown_connections 2023-10-16T11:20:29.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.630+0000 7f584641a700 1 -- 172.21.15.158:0/320884764 >> 172.21.15.158:0/320884764 conn(0x7f58400ccda0 msgr2=0x7f58400cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:29.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.630+0000 7f584641a700 1 -- 172.21.15.158:0/320884764 shutdown_connections 2023-10-16T11:20:29.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.630+0000 7f584641a700 1 -- 172.21.15.158:0/320884764 wait complete. 2023-10-16T11:20:29.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.631+0000 7f584641a700 1 Processor -- start 2023-10-16T11:20:29.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.631+0000 7f584641a700 1 -- start start 2023-10-16T11:20:29.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f584641a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828004850 0x7f58401577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f584641a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58400c73f0 0x7f58400c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f584641a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58400c7db0 0x7f58400c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f584641a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400c87a0 con 0x7f58400c73f0 2023-10-16T11:20:29.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f584641a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400c8230 con 0x7f5828004850 2023-10-16T11:20:29.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f584641a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400c6b70 con 0x7f58400c7db0 2023-10-16T11:20:29.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58400c73f0 0x7f58400c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58400c73f0 0x7f58400c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40322/0 (socket says 172.21.15.158:40322) 2023-10-16T11:20:29.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 -- 172.21.15.158:0/2004985187 learned_addr learned my addr 172.21.15.158:0/2004985187 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:29.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583ffff700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828004850 0x7f58401577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f58449b7700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58400c7db0 0x7f58400c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 -- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58400c7db0 msgr2=0x7f58400c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58400c7db0 0x7f58400c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 -- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828004850 msgr2=0x7f58401577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828004850 0x7f58401577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.632+0000 7f583f7fe700 1 -- 172.21.15.158:0/2004985187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5840157ce0 con 0x7f58400c73f0 2023-10-16T11:20:29.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.633+0000 7f583f7fe700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58400c73f0 0x7f58400c7870 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f58280046d0 tx=0x7f58340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:29.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.633+0000 7f583cff9700 1 -- 172.21.15.158:0/2004985187 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400e6c0 con 0x7f58400c73f0 2023-10-16T11:20:29.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.633+0000 7f583cff9700 1 -- 172.21.15.158:0/2004985187 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58340038a0 con 0x7f58400c73f0 2023-10-16T11:20:29.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.633+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58400c5ff0 con 0x7f58400c73f0 2023-10-16T11:20:29.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.633+0000 7f583cff9700 1 -- 172.21.15.158:0/2004985187 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834003c50 con 0x7f58400c73f0 2023-10-16T11:20:29.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.633+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58400c65b0 con 0x7f58400c73f0 2023-10-16T11:20:29.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.634+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58400681b0 con 0x7f58400c73f0 2023-10-16T11:20:29.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.634+0000 7f583cff9700 1 -- 172.21.15.158:0/2004985187 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5834013470 con 0x7f58400c73f0 2023-10-16T11:20:29.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.635+0000 7f583cff9700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f582c06b140 0x7f582c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:29.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.635+0000 7f583cff9700 1 -- 172.21.15.158:0/2004985187 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f583408b810 con 0x7f58400c73f0 2023-10-16T11:20:29.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.635+0000 7f583ffff700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f582c06b140 0x7f582c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:29.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.635+0000 7f583ffff700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f582c06b140 0x7f582c06d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5830000ef0 tx=0x7f58300079d0 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:29.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.639+0000 7f583cff9700 1 -- 172.21.15.158:0/2004985187 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5834059e80 con 0x7f58400c73f0 2023-10-16T11:20:29.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.829+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 --> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58400c87a0 con 0x7f582c06b140 2023-10-16T11:20:29.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.837+0000 7f583cff9700 1 -- 172.21.15.158:0/2004985187 <== mgr.12030 v2:172.21.15.158:6832/3408161846 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289944 (secure 0 0 0) 0x7f58400c87a0 con 0x7f582c06b140 2023-10-16T11:20:29.843 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:29.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f582c06b140 msgr2=0x7f582c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f582c06b140 0x7f582c06d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5830000ef0 tx=0x7f58300079d0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58400c73f0 msgr2=0x7f58400c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:29.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58400c73f0 0x7f58400c7870 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f58280046d0 tx=0x7f58340047c0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 shutdown_connections 2023-10-16T11:20:29.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f582c06b140 0x7f582c06d600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58400c7db0 0x7f58400c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5828004850 0x7f58401577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 --2- 172.21.15.158:0/2004985187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58400c73f0 0x7f58400c7870 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:29.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 >> 172.21.15.158:0/2004985187 conn(0x7f58400ccda0 msgr2=0x7f58400afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:29.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 shutdown_connections 2023-10-16T11:20:29.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:29.841+0000 7f584641a700 1 -- 172.21.15.158:0/2004985187 wait complete. 2023-10-16T11:20:29.849 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:20:29.855 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T11:20:29.448594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436528,"kb_used_data":24616,"kb_used_omap":12,"kb_used_meta":411763,"kb_avail":754538192,"statfs":{"total":773094113280,"available":772647108608,"internally_reserved":0,"allocated":25206784,"data_stored":14063924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":421645879},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.734617"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802700+0000","last_change":"2023-10-16T11:20:25.748352+0000","last_active":"2023-10-16T11:20:26.802700+0000","last_peered":"2023-10-16T11:20:26.802700+0000","last_clean":"2023-10-16T11:20:26.802700+0000","last_became_active":"2023-10-16T11:20:25.747848+0000","last_became_peered":"2023-10-16T11:20:25.747848+0000","last_unstale":"2023-10-16T11:20:26.802700+0000","last_undegraded":"2023-10-16T11:20:26.802700+0000","last_fullsized":"2023-10-16T11:20:26.802700+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:48.171042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801655+0000","last_change":"2023-10-16T11:20:25.749395+0000","last_active":"2023-10-16T11:20:26.801655+0000","last_peered":"2023-10-16T11:20:26.801655+0000","last_clean":"2023-10-16T11:20:26.801655+0000","last_became_active":"2023-10-16T11:20:25.749044+0000","last_became_peered":"2023-10-16T11:20:25.749044+0000","last_unstale":"2023-10-16T11:20:26.801655+0000","last_undegraded":"2023-10-16T11:20:26.801655+0000","last_fullsized":"2023-10-16T11:20:26.801655+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:16.211507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803081+0000","last_change":"2023-10-16T11:20:25.739817+0000","last_active":"2023-10-16T11:20:26.803081+0000","last_peered":"2023-10-16T11:20:26.803081+0000","last_clean":"2023-10-16T11:20:26.803081+0000","last_became_active":"2023-10-16T11:20:25.739601+0000","last_became_peered":"2023-10-16T11:20:25.739601+0000","last_unstale":"2023-10-16T11:20:26.803081+0000","last_undegraded":"2023-10-16T11:20:26.803081+0000","last_fullsized":"2023-10-16T11:20:26.803081+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:38.443038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802907+0000","last_change":"2023-10-16T11:20:25.751269+0000","last_active":"2023-10-16T11:20:26.802907+0000","last_peered":"2023-10-16T11:20:26.802907+0000","last_clean":"2023-10-16T11:20:26.802907+0000","last_became_active":"2023-10-16T11:20:25.750973+0000","last_became_peered":"2023-10-16T11:20:25.750973+0000","last_unstale":"2023-10-16T11:20:26.802907+0000","last_undegraded":"2023-10-16T11:20:26.802907+0000","last_fullsized":"2023-10-16T11:20:26.802907+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:36.490706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741753+0000","last_change":"2023-10-16T11:20:25.750599+0000","last_active":"2023-10-16T11:20:26.741753+0000","last_peered":"2023-10-16T11:20:26.741753+0000","last_clean":"2023-10-16T11:20:26.741753+0000","last_became_active":"2023-10-16T11:20:25.749962+0000","last_became_peered":"2023-10-16T11:20:25.749962+0000","last_unstale":"2023-10-16T11:20:26.741753+0000","last_undegraded":"2023-10-16T11:20:26.741753+0000","last_fullsized":"2023-10-16T11:20:26.741753+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:15.073727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802717+0000","last_change":"2023-10-16T11:20:25.745973+0000","last_active":"2023-10-16T11:20:26.802717+0000","last_peered":"2023-10-16T11:20:26.802717+0000","last_clean":"2023-10-16T11:20:26.802717+0000","last_became_active":"2023-10-16T11:20:25.745706+0000","last_became_peered":"2023-10-16T11:20:25.745706+0000","last_unstale":"2023-10-16T11:20:26.802717+0000","last_undegraded":"2023-10-16T11:20:26.802717+0000","last_fullsized":"2023-10-16T11:20:26.802717+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:08.628698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801829+0000","last_change":"2023-10-16T11:20:25.748211+0000","last_active":"2023-10-16T11:20:26.801829+0000","last_peered":"2023-10-16T11:20:26.801829+0000","last_clean":"2023-10-16T11:20:26.801829+0000","last_became_active":"2023-10-16T11:20:25.747956+0000","last_became_peered":"2023-10-16T11:20:25.747956+0000","last_unstale":"2023-10-16T11:20:26.801829+0000","last_undegraded":"2023-10-16T11:20:26.801829+0000","last_fullsized":"2023-10-16T11:20:26.801829+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741324+0000","last_change":"2023-10-16T11:20:25.750901+0000","last_active":"2023-10-16T11:20:26.741324+0000","last_peered":"2023-10-16T11:20:26.741324+0000","last_clean":"2023-10-16T11:20:26.741324+0000","last_became_active":"2023-10-16T11:20:25.750582+0000","last_became_peered":"2023-10-16T11:20:25.750582+0000","last_unstale":"2023-10-16T11:20:26.741324+0000","last_undegraded":"2023-10-16T11:20:26.741324+0000","last_fullsized":"2023-10-16T11:20:26.741324+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:47.707747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802565+0000","last_change":"2023-10-16T11:20:25.750204+0000","last_active":"2023-10-16T11:20:26.802565+0000","last_peered":"2023-10-16T11:20:26.802565+0000","last_clean":"2023-10-16T11:20:26.802565+0000","last_became_active":"2023-10-16T11:20:25.749903+0000","last_became_peered":"2023-10-16T11:20:25.749903+0000","last_unstale":"2023-10-16T11:20:26.802565+0000","last_undegraded":"2023-10-16T11:20:26.802565+0000","last_fullsized":"2023-10-16T11:20:26.802565+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:48.734101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803228+0000","last_change":"2023-10-16T11:20:25.747726+0000","last_active":"2023-10-16T11:20:26.803228+0000","last_peered":"2023-10-16T11:20:26.803228+0000","last_clean":"2023-10-16T11:20:26.803228+0000","last_became_active":"2023-10-16T11:20:25.747314+0000","last_became_peered":"2023-10-16T11:20:25.747314+0000","last_unstale":"2023-10-16T11:20:26.803228+0000","last_undegraded":"2023-10-16T11:20:26.803228+0000","last_fullsized":"2023-10-16T11:20:26.803228+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803230+0000","last_change":"2023-10-16T11:20:25.747416+0000","last_active":"2023-10-16T11:20:26.803230+0000","last_peered":"2023-10-16T11:20:26.803230+0000","last_clean":"2023-10-16T11:20:26.803230+0000","last_became_active":"2023-10-16T11:20:25.746950+0000","last_became_peered":"2023-10-16T11:20:25.746950+0000","last_unstale":"2023-10-16T11:20:26.803230+0000","last_undegraded":"2023-10-16T11:20:26.803230+0000","last_fullsized":"2023-10-16T11:20:26.803230+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803775+0000","last_change":"2023-10-16T11:20:25.743709+0000","last_active":"2023-10-16T11:20:26.803775+0000","last_peered":"2023-10-16T11:20:26.803775+0000","last_clean":"2023-10-16T11:20:26.803775+0000","last_became_active":"2023-10-16T11:20:25.743444+0000","last_became_peered":"2023-10-16T11:20:25.743444+0000","last_unstale":"2023-10-16T11:20:26.803775+0000","last_undegraded":"2023-10-16T11:20:26.803775+0000","last_fullsized":"2023-10-16T11:20:26.803775+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:28.662644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746334+0000","last_change":"2023-10-16T11:20:25.737922+0000","last_active":"2023-10-16T11:20:26.746334+0000","last_peered":"2023-10-16T11:20:26.746334+0000","last_clean":"2023-10-16T11:20:26.746334+0000","last_became_active":"2023-10-16T11:20:25.737086+0000","last_became_peered":"2023-10-16T11:20:25.737086+0000","last_unstale":"2023-10-16T11:20:26.746334+0000","last_undegraded":"2023-10-16T11:20:26.746334+0000","last_fullsized":"2023-10-16T11:20:26.746334+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:17.897224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803610+0000","last_change":"2023-10-16T11:20:25.745458+0000","last_active":"2023-10-16T11:20:26.803610+0000","last_peered":"2023-10-16T11:20:26.803610+0000","last_clean":"2023-10-16T11:20:26.803610+0000","last_became_active":"2023-10-16T11:20:25.745257+0000","last_became_peered":"2023-10-16T11:20:25.745257+0000","last_unstale":"2023-10-16T11:20:26.803610+0000","last_undegraded":"2023-10-16T11:20:26.803610+0000","last_fullsized":"2023-10-16T11:20:26.803610+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740363+0000","last_change":"2023-10-16T11:20:25.741269+0000","last_active":"2023-10-16T11:20:26.740363+0000","last_peered":"2023-10-16T11:20:26.740363+0000","last_clean":"2023-10-16T11:20:26.740363+0000","last_became_active":"2023-10-16T11:20:25.740995+0000","last_became_peered":"2023-10-16T11:20:25.740995+0000","last_unstale":"2023-10-16T11:20:26.740363+0000","last_undegraded":"2023-10-16T11:20:26.740363+0000","last_fullsized":"2023-10-16T11:20:26.740363+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:45.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802182+0000","last_change":"2023-10-16T11:20:25.742299+0000","last_active":"2023-10-16T11:20:26.802182+0000","last_peered":"2023-10-16T11:20:26.802182+0000","last_clean":"2023-10-16T11:20:26.802182+0000","last_became_active":"2023-10-16T11:20:25.741394+0000","last_became_peered":"2023-10-16T11:20:25.741394+0000","last_unstale":"2023-10-16T11:20:26.802182+0000","last_undegraded":"2023-10-16T11:20:26.802182+0000","last_fullsized":"2023-10-16T11:20:26.802182+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:35.560067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746702+0000","last_change":"2023-10-16T11:20:25.744534+0000","last_active":"2023-10-16T11:20:26.746702+0000","last_peered":"2023-10-16T11:20:26.746702+0000","last_clean":"2023-10-16T11:20:26.746702+0000","last_became_active":"2023-10-16T11:20:25.744224+0000","last_became_peered":"2023-10-16T11:20:25.744224+0000","last_unstale":"2023-10-16T11:20:26.746702+0000","last_undegraded":"2023-10-16T11:20:26.746702+0000","last_fullsized":"2023-10-16T11:20:26.746702+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:14.437943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803456+0000","last_change":"2023-10-16T11:20:25.747551+0000","last_active":"2023-10-16T11:20:26.803456+0000","last_peered":"2023-10-16T11:20:26.803456+0000","last_clean":"2023-10-16T11:20:26.803456+0000","last_became_active":"2023-10-16T11:20:25.746989+0000","last_became_peered":"2023-10-16T11:20:25.746989+0000","last_unstale":"2023-10-16T11:20:26.803456+0000","last_undegraded":"2023-10-16T11:20:26.803456+0000","last_fullsized":"2023-10-16T11:20:26.803456+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740647+0000","last_change":"2023-10-16T11:20:25.738568+0000","last_active":"2023-10-16T11:20:26.740647+0000","last_peered":"2023-10-16T11:20:26.740647+0000","last_clean":"2023-10-16T11:20:26.740647+0000","last_became_active":"2023-10-16T11:20:25.738313+0000","last_became_peered":"2023-10-16T11:20:25.738313+0000","last_unstale":"2023-10-16T11:20:26.740647+0000","last_undegraded":"2023-10-16T11:20:26.740647+0000","last_fullsized":"2023-10-16T11:20:26.740647+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:28.152434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802035+0000","last_change":"2023-10-16T11:20:25.750312+0000","last_active":"2023-10-16T11:20:26.802035+0000","last_peered":"2023-10-16T11:20:26.802035+0000","last_clean":"2023-10-16T11:20:26.802035+0000","last_became_active":"2023-10-16T11:20:25.749320+0000","last_became_peered":"2023-10-16T11:20:25.749320+0000","last_unstale":"2023-10-16T11:20:26.802035+0000","last_undegraded":"2023-10-16T11:20:26.802035+0000","last_fullsized":"2023-10-16T11:20:26.802035+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:26.044494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802039+0000","last_change":"2023-10-16T11:20:25.744924+0000","last_active":"2023-10-16T11:20:26.802039+0000","last_peered":"2023-10-16T11:20:26.802039+0000","last_clean":"2023-10-16T11:20:26.802039+0000","last_became_active":"2023-10-16T11:20:25.744659+0000","last_became_peered":"2023-10-16T11:20:25.744659+0000","last_unstale":"2023-10-16T11:20:26.802039+0000","last_undegraded":"2023-10-16T11:20:26.802039+0000","last_fullsized":"2023-10-16T11:20:26.802039+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746563+0000","last_change":"2023-10-16T11:20:25.750717+0000","last_active":"2023-10-16T11:20:26.746563+0000","last_peered":"2023-10-16T11:20:26.746563+0000","last_clean":"2023-10-16T11:20:26.746563+0000","last_became_active":"2023-10-16T11:20:25.750419+0000","last_became_peered":"2023-10-16T11:20:25.750419+0000","last_unstale":"2023-10-16T11:20:26.746563+0000","last_undegraded":"2023-10-16T11:20:26.746563+0000","last_fullsized":"2023-10-16T11:20:26.746563+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:21.095229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740584+0000","last_change":"2023-10-16T11:20:25.738444+0000","last_active":"2023-10-16T11:20:26.740584+0000","last_peered":"2023-10-16T11:20:26.740584+0000","last_clean":"2023-10-16T11:20:26.740584+0000","last_became_active":"2023-10-16T11:20:25.738166+0000","last_became_peered":"2023-10-16T11:20:25.738166+0000","last_unstale":"2023-10-16T11:20:26.740584+0000","last_undegraded":"2023-10-16T11:20:26.740584+0000","last_fullsized":"2023-10-16T11:20:26.740584+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:57.655885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746045+0000","last_change":"2023-10-16T11:20:25.741211+0000","last_active":"2023-10-16T11:20:26.746045+0000","last_peered":"2023-10-16T11:20:26.746045+0000","last_clean":"2023-10-16T11:20:26.746045+0000","last_became_active":"2023-10-16T11:20:25.740922+0000","last_became_peered":"2023-10-16T11:20:25.740922+0000","last_unstale":"2023-10-16T11:20:26.746045+0000","last_undegraded":"2023-10-16T11:20:26.746045+0000","last_fullsized":"2023-10-16T11:20:26.746045+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:55.904667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801586+0000","last_change":"2023-10-16T11:20:25.741581+0000","last_active":"2023-10-16T11:20:26.801586+0000","last_peered":"2023-10-16T11:20:26.801586+0000","last_clean":"2023-10-16T11:20:26.801586+0000","last_became_active":"2023-10-16T11:20:25.740490+0000","last_became_peered":"2023-10-16T11:20:25.740490+0000","last_unstale":"2023-10-16T11:20:26.801586+0000","last_undegraded":"2023-10-16T11:20:26.801586+0000","last_fullsized":"2023-10-16T11:20:26.801586+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:20.166253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802066+0000","last_change":"2023-10-16T11:20:25.747818+0000","last_active":"2023-10-16T11:20:26.802066+0000","last_peered":"2023-10-16T11:20:26.802066+0000","last_clean":"2023-10-16T11:20:26.802066+0000","last_became_active":"2023-10-16T11:20:25.747533+0000","last_became_peered":"2023-10-16T11:20:25.747533+0000","last_unstale":"2023-10-16T11:20:26.802066+0000","last_undegraded":"2023-10-16T11:20:26.802066+0000","last_fullsized":"2023-10-16T11:20:26.802066+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741725+0000","last_change":"2023-10-16T11:20:25.746406+0000","last_active":"2023-10-16T11:20:26.741725+0000","last_peered":"2023-10-16T11:20:26.741725+0000","last_clean":"2023-10-16T11:20:26.741725+0000","last_became_active":"2023-10-16T11:20:25.746140+0000","last_became_peered":"2023-10-16T11:20:25.746140+0000","last_unstale":"2023-10-16T11:20:26.741725+0000","last_undegraded":"2023-10-16T11:20:26.741725+0000","last_fullsized":"2023-10-16T11:20:26.741725+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:42.488771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802723+0000","last_change":"2023-10-16T11:20:25.750643+0000","last_active":"2023-10-16T11:20:26.802723+0000","last_peered":"2023-10-16T11:20:26.802723+0000","last_clean":"2023-10-16T11:20:26.802723+0000","last_became_active":"2023-10-16T11:20:25.749980+0000","last_became_peered":"2023-10-16T11:20:25.749980+0000","last_unstale":"2023-10-16T11:20:26.802723+0000","last_undegraded":"2023-10-16T11:20:26.802723+0000","last_fullsized":"2023-10-16T11:20:26.802723+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:41.498772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803323+0000","last_change":"2023-10-16T11:20:25.744400+0000","last_active":"2023-10-16T11:20:26.803323+0000","last_peered":"2023-10-16T11:20:26.803323+0000","last_clean":"2023-10-16T11:20:26.803323+0000","last_became_active":"2023-10-16T11:20:25.744118+0000","last_became_peered":"2023-10-16T11:20:25.744118+0000","last_unstale":"2023-10-16T11:20:26.803323+0000","last_undegraded":"2023-10-16T11:20:26.803323+0000","last_fullsized":"2023-10-16T11:20:26.803323+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803736+0000","last_change":"2023-10-16T11:20:25.749096+0000","last_active":"2023-10-16T11:20:26.803736+0000","last_peered":"2023-10-16T11:20:26.803736+0000","last_clean":"2023-10-16T11:20:26.803736+0000","last_became_active":"2023-10-16T11:20:25.748841+0000","last_became_peered":"2023-10-16T11:20:25.748841+0000","last_unstale":"2023-10-16T11:20:26.803736+0000","last_undegraded":"2023-10-16T11:20:26.803736+0000","last_fullsized":"2023-10-16T11:20:26.803736+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801386+0000","last_change":"2023-10-16T11:20:25.741998+0000","last_active":"2023-10-16T11:20:26.801386+0000","last_peered":"2023-10-16T11:20:26.801386+0000","last_clean":"2023-10-16T11:20:26.801386+0000","last_became_active":"2023-10-16T11:20:25.741482+0000","last_became_peered":"2023-10-16T11:20:25.741482+0000","last_unstale":"2023-10-16T11:20:26.801386+0000","last_undegraded":"2023-10-16T11:20:26.801386+0000","last_fullsized":"2023-10-16T11:20:26.801386+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:01.922741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802196+0000","last_change":"2023-10-16T11:20:25.750764+0000","last_active":"2023-10-16T11:20:26.802196+0000","last_peered":"2023-10-16T11:20:26.802196+0000","last_clean":"2023-10-16T11:20:26.802196+0000","last_became_active":"2023-10-16T11:20:25.750127+0000","last_became_peered":"2023-10-16T11:20:25.750127+0000","last_unstale":"2023-10-16T11:20:26.802196+0000","last_undegraded":"2023-10-16T11:20:26.802196+0000","last_fullsized":"2023-10-16T11:20:26.802196+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.429724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801485+0000","last_change":"2023-10-16T11:20:25.752205+0000","last_active":"2023-10-16T11:20:26.801485+0000","last_peered":"2023-10-16T11:20:26.801485+0000","last_clean":"2023-10-16T11:20:26.801485+0000","last_became_active":"2023-10-16T11:20:25.751915+0000","last_became_peered":"2023-10-16T11:20:25.751915+0000","last_unstale":"2023-10-16T11:20:26.801485+0000","last_undegraded":"2023-10-16T11:20:26.801485+0000","last_fullsized":"2023-10-16T11:20:26.801485+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:43.213188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"236'4","reported_seq":16,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.763173+0000","last_change":"2023-10-16T11:20:24.733695+0000","last_active":"2023-10-16T11:20:25.763173+0000","last_peered":"2023-10-16T11:20:25.763173+0000","last_clean":"2023-10-16T11:20:25.763173+0000","last_became_active":"2023-10-16T11:20:24.733413+0000","last_became_peered":"2023-10-16T11:20:24.733413+0000","last_unstale":"2023-10-16T11:20:25.763173+0000","last_undegraded":"2023-10-16T11:20:25.763173+0000","last_fullsized":"2023-10-16T11:20:25.763173+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:34.326531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740347+0000","last_change":"2023-10-16T11:20:25.747992+0000","last_active":"2023-10-16T11:20:26.740347+0000","last_peered":"2023-10-16T11:20:26.740347+0000","last_clean":"2023-10-16T11:20:26.740347+0000","last_became_active":"2023-10-16T11:20:25.747735+0000","last_became_peered":"2023-10-16T11:20:25.747735+0000","last_unstale":"2023-10-16T11:20:26.740347+0000","last_undegraded":"2023-10-16T11:20:26.740347+0000","last_fullsized":"2023-10-16T11:20:26.740347+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:49.269723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757593+0000","last_change":"2023-10-16T11:20:24.739822+0000","last_active":"2023-10-16T11:20:25.757593+0000","last_peered":"2023-10-16T11:20:25.757593+0000","last_clean":"2023-10-16T11:20:25.757593+0000","last_became_active":"2023-10-16T11:20:24.739470+0000","last_became_peered":"2023-10-16T11:20:24.739470+0000","last_unstale":"2023-10-16T11:20:25.757593+0000","last_undegraded":"2023-10-16T11:20:25.757593+0000","last_fullsized":"2023-10-16T11:20:25.757593+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:18.641488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802991+0000","last_change":"2023-10-16T11:20:25.746049+0000","last_active":"2023-10-16T11:20:26.802991+0000","last_peered":"2023-10-16T11:20:26.802991+0000","last_clean":"2023-10-16T11:20:26.802991+0000","last_became_active":"2023-10-16T11:20:25.745759+0000","last_became_peered":"2023-10-16T11:20:25.745759+0000","last_unstale":"2023-10-16T11:20:26.802991+0000","last_undegraded":"2023-10-16T11:20:26.802991+0000","last_fullsized":"2023-10-16T11:20:26.802991+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:40:29.501267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.765321+0000","last_change":"2023-10-16T11:20:24.727708+0000","last_active":"2023-10-16T11:20:25.765321+0000","last_peered":"2023-10-16T11:20:25.765321+0000","last_clean":"2023-10-16T11:20:25.765321+0000","last_became_active":"2023-10-16T11:20:24.727357+0000","last_became_peered":"2023-10-16T11:20:24.727357+0000","last_unstale":"2023-10-16T11:20:25.765321+0000","last_undegraded":"2023-10-16T11:20:25.765321+0000","last_fullsized":"2023-10-16T11:20:25.765321+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:32.307241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803875+0000","last_change":"2023-10-16T11:20:25.742133+0000","last_active":"2023-10-16T11:20:26.803875+0000","last_peered":"2023-10-16T11:20:26.803875+0000","last_clean":"2023-10-16T11:20:26.803875+0000","last_became_active":"2023-10-16T11:20:25.741865+0000","last_became_peered":"2023-10-16T11:20:25.741865+0000","last_unstale":"2023-10-16T11:20:26.803875+0000","last_undegraded":"2023-10-16T11:20:26.803875+0000","last_fullsized":"2023-10-16T11:20:26.803875+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:25.230100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757467+0000","last_change":"2023-10-16T11:20:24.722283+0000","last_active":"2023-10-16T11:20:25.757467+0000","last_peered":"2023-10-16T11:20:25.757467+0000","last_clean":"2023-10-16T11:20:25.757467+0000","last_became_active":"2023-10-16T11:20:24.721789+0000","last_became_peered":"2023-10-16T11:20:24.721789+0000","last_unstale":"2023-10-16T11:20:25.757467+0000","last_undegraded":"2023-10-16T11:20:25.757467+0000","last_fullsized":"2023-10-16T11:20:25.757467+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:37.116862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802129+0000","last_change":"2023-10-16T11:20:25.748913+0000","last_active":"2023-10-16T11:20:26.802129+0000","last_peered":"2023-10-16T11:20:26.802129+0000","last_clean":"2023-10-16T11:20:26.802129+0000","last_became_active":"2023-10-16T11:20:25.748588+0000","last_became_peered":"2023-10-16T11:20:25.748588+0000","last_unstale":"2023-10-16T11:20:26.802129+0000","last_undegraded":"2023-10-16T11:20:26.802129+0000","last_fullsized":"2023-10-16T11:20:26.802129+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.765029+0000","last_change":"2023-10-16T11:20:24.733622+0000","last_active":"2023-10-16T11:20:25.765029+0000","last_peered":"2023-10-16T11:20:25.765029+0000","last_clean":"2023-10-16T11:20:25.765029+0000","last_became_active":"2023-10-16T11:20:24.733342+0000","last_became_peered":"2023-10-16T11:20:24.733342+0000","last_unstale":"2023-10-16T11:20:25.765029+0000","last_undegraded":"2023-10-16T11:20:25.765029+0000","last_fullsized":"2023-10-16T11:20:25.765029+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:10.962482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803222+0000","last_change":"2023-10-16T11:20:25.751310+0000","last_active":"2023-10-16T11:20:26.803222+0000","last_peered":"2023-10-16T11:20:26.803222+0000","last_clean":"2023-10-16T11:20:26.803222+0000","last_became_active":"2023-10-16T11:20:25.751042+0000","last_became_peered":"2023-10-16T11:20:25.751042+0000","last_unstale":"2023-10-16T11:20:26.803222+0000","last_undegraded":"2023-10-16T11:20:26.803222+0000","last_fullsized":"2023-10-16T11:20:26.803222+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736174+0000","last_change":"2023-10-16T11:20:24.739609+0000","last_active":"2023-10-16T11:20:25.736174+0000","last_peered":"2023-10-16T11:20:25.736174+0000","last_clean":"2023-10-16T11:20:25.736174+0000","last_became_active":"2023-10-16T11:20:24.739275+0000","last_became_peered":"2023-10-16T11:20:24.739275+0000","last_unstale":"2023-10-16T11:20:25.736174+0000","last_undegraded":"2023-10-16T11:20:25.736174+0000","last_fullsized":"2023-10-16T11:20:25.736174+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:47.827300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741853+0000","last_change":"2023-10-16T11:20:25.744793+0000","last_active":"2023-10-16T11:20:26.741853+0000","last_peered":"2023-10-16T11:20:26.741853+0000","last_clean":"2023-10-16T11:20:26.741853+0000","last_became_active":"2023-10-16T11:20:25.744521+0000","last_became_peered":"2023-10-16T11:20:25.744521+0000","last_unstale":"2023-10-16T11:20:26.741853+0000","last_undegraded":"2023-10-16T11:20:26.741853+0000","last_fullsized":"2023-10-16T11:20:26.741853+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:44.539888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.747287+0000","last_change":"2023-10-16T11:20:24.723659+0000","last_active":"2023-10-16T11:20:25.747287+0000","last_peered":"2023-10-16T11:20:25.747287+0000","last_clean":"2023-10-16T11:20:25.747287+0000","last_became_active":"2023-10-16T11:20:24.723342+0000","last_became_peered":"2023-10-16T11:20:24.723342+0000","last_unstale":"2023-10-16T11:20:25.747287+0000","last_undegraded":"2023-10-16T11:20:25.747287+0000","last_fullsized":"2023-10-16T11:20:25.747287+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:29.707065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802133+0000","last_change":"2023-10-16T11:20:25.748568+0000","last_active":"2023-10-16T11:20:26.802133+0000","last_peered":"2023-10-16T11:20:26.802133+0000","last_clean":"2023-10-16T11:20:26.802133+0000","last_became_active":"2023-10-16T11:20:25.748318+0000","last_became_peered":"2023-10-16T11:20:25.748318+0000","last_unstale":"2023-10-16T11:20:26.802133+0000","last_undegraded":"2023-10-16T11:20:26.802133+0000","last_fullsized":"2023-10-16T11:20:26.802133+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732300+0000","last_change":"2023-10-16T11:20:24.736095+0000","last_active":"2023-10-16T11:20:25.732300+0000","last_peered":"2023-10-16T11:20:25.732300+0000","last_clean":"2023-10-16T11:20:25.732300+0000","last_became_active":"2023-10-16T11:20:24.735775+0000","last_became_peered":"2023-10-16T11:20:24.735775+0000","last_unstale":"2023-10-16T11:20:25.732300+0000","last_undegraded":"2023-10-16T11:20:25.732300+0000","last_fullsized":"2023-10-16T11:20:25.732300+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:35.485188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746270+0000","last_change":"2023-10-16T11:20:25.745589+0000","last_active":"2023-10-16T11:20:26.746270+0000","last_peered":"2023-10-16T11:20:26.746270+0000","last_clean":"2023-10-16T11:20:26.746270+0000","last_became_active":"2023-10-16T11:20:25.745297+0000","last_became_peered":"2023-10-16T11:20:25.745297+0000","last_unstale":"2023-10-16T11:20:26.746270+0000","last_undegraded":"2023-10-16T11:20:26.746270+0000","last_fullsized":"2023-10-16T11:20:26.746270+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:17.743037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.737188+0000","last_change":"2023-10-16T11:20:24.738634+0000","last_active":"2023-10-16T11:20:25.737188+0000","last_peered":"2023-10-16T11:20:25.737188+0000","last_clean":"2023-10-16T11:20:25.737188+0000","last_became_active":"2023-10-16T11:20:24.738372+0000","last_became_peered":"2023-10-16T11:20:24.738372+0000","last_unstale":"2023-10-16T11:20:25.737188+0000","last_undegraded":"2023-10-16T11:20:25.737188+0000","last_fullsized":"2023-10-16T11:20:25.737188+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:23.747207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803537+0000","last_change":"2023-10-16T11:20:25.748579+0000","last_active":"2023-10-16T11:20:26.803537+0000","last_peered":"2023-10-16T11:20:26.803537+0000","last_clean":"2023-10-16T11:20:26.803537+0000","last_became_active":"2023-10-16T11:20:25.748347+0000","last_became_peered":"2023-10-16T11:20:25.748347+0000","last_unstale":"2023-10-16T11:20:26.803537+0000","last_undegraded":"2023-10-16T11:20:26.803537+0000","last_fullsized":"2023-10-16T11:20:26.803537+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"236'3","reported_seq":15,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.758189+0000","last_change":"2023-10-16T11:20:24.728938+0000","last_active":"2023-10-16T11:20:25.758189+0000","last_peered":"2023-10-16T11:20:25.758189+0000","last_clean":"2023-10-16T11:20:25.758189+0000","last_became_active":"2023-10-16T11:20:24.728638+0000","last_became_peered":"2023-10-16T11:20:24.728638+0000","last_unstale":"2023-10-16T11:20:25.758189+0000","last_undegraded":"2023-10-16T11:20:25.758189+0000","last_fullsized":"2023-10-16T11:20:25.758189+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:42.082000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801991+0000","last_change":"2023-10-16T11:20:25.751420+0000","last_active":"2023-10-16T11:20:26.801991+0000","last_peered":"2023-10-16T11:20:26.801991+0000","last_clean":"2023-10-16T11:20:26.801991+0000","last_became_active":"2023-10-16T11:20:25.751161+0000","last_became_peered":"2023-10-16T11:20:25.751161+0000","last_unstale":"2023-10-16T11:20:26.801991+0000","last_undegraded":"2023-10-16T11:20:26.801991+0000","last_fullsized":"2023-10-16T11:20:26.801991+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"236'6","reported_seq":18,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.766988+0000","last_change":"2023-10-16T11:20:24.724201+0000","last_active":"2023-10-16T11:20:25.766988+0000","last_peered":"2023-10-16T11:20:25.766988+0000","last_clean":"2023-10-16T11:20:25.766988+0000","last_became_active":"2023-10-16T11:20:24.723918+0000","last_became_peered":"2023-10-16T11:20:24.723918+0000","last_unstale":"2023-10-16T11:20:25.766988+0000","last_undegraded":"2023-10-16T11:20:25.766988+0000","last_fullsized":"2023-10-16T11:20:25.766988+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:55.483353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801992+0000","last_change":"2023-10-16T11:20:25.742201+0000","last_active":"2023-10-16T11:20:26.801992+0000","last_peered":"2023-10-16T11:20:26.801992+0000","last_clean":"2023-10-16T11:20:26.801992+0000","last_became_active":"2023-10-16T11:20:25.741871+0000","last_became_peered":"2023-10-16T11:20:25.741871+0000","last_unstale":"2023-10-16T11:20:26.801992+0000","last_undegraded":"2023-10-16T11:20:26.801992+0000","last_fullsized":"2023-10-16T11:20:26.801992+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:18:28.417174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.761652+0000","last_change":"2023-10-16T11:20:24.725162+0000","last_active":"2023-10-16T11:20:25.761652+0000","last_peered":"2023-10-16T11:20:25.761652+0000","last_clean":"2023-10-16T11:20:25.761652+0000","last_became_active":"2023-10-16T11:20:24.724795+0000","last_became_peered":"2023-10-16T11:20:24.724795+0000","last_unstale":"2023-10-16T11:20:25.761652+0000","last_undegraded":"2023-10-16T11:20:25.761652+0000","last_fullsized":"2023-10-16T11:20:25.761652+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:25.854243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801952+0000","last_change":"2023-10-16T11:20:25.739623+0000","last_active":"2023-10-16T11:20:26.801952+0000","last_peered":"2023-10-16T11:20:26.801952+0000","last_clean":"2023-10-16T11:20:26.801952+0000","last_became_active":"2023-10-16T11:20:25.739321+0000","last_became_peered":"2023-10-16T11:20:25.739321+0000","last_unstale":"2023-10-16T11:20:26.801952+0000","last_undegraded":"2023-10-16T11:20:26.801952+0000","last_fullsized":"2023-10-16T11:20:26.801952+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736599+0000","last_change":"2023-10-16T11:20:24.735120+0000","last_active":"2023-10-16T11:20:25.736599+0000","last_peered":"2023-10-16T11:20:25.736599+0000","last_clean":"2023-10-16T11:20:25.736599+0000","last_became_active":"2023-10-16T11:20:24.734857+0000","last_became_peered":"2023-10-16T11:20:24.734857+0000","last_unstale":"2023-10-16T11:20:25.736599+0000","last_undegraded":"2023-10-16T11:20:25.736599+0000","last_fullsized":"2023-10-16T11:20:25.736599+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:03.791128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.742231+0000","last_change":"2023-10-16T11:20:25.737997+0000","last_active":"2023-10-16T11:20:26.742231+0000","last_peered":"2023-10-16T11:20:26.742231+0000","last_clean":"2023-10-16T11:20:26.742231+0000","last_became_active":"2023-10-16T11:20:25.737314+0000","last_became_peered":"2023-10-16T11:20:25.737314+0000","last_unstale":"2023-10-16T11:20:26.742231+0000","last_undegraded":"2023-10-16T11:20:26.742231+0000","last_fullsized":"2023-10-16T11:20:26.742231+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:44.489552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736247+0000","last_change":"2023-10-16T11:20:24.733587+0000","last_active":"2023-10-16T11:20:25.736247+0000","last_peered":"2023-10-16T11:20:25.736247+0000","last_clean":"2023-10-16T11:20:25.736247+0000","last_became_active":"2023-10-16T11:20:24.733160+0000","last_became_peered":"2023-10-16T11:20:24.733160+0000","last_unstale":"2023-10-16T11:20:25.736247+0000","last_undegraded":"2023-10-16T11:20:25.736247+0000","last_fullsized":"2023-10-16T11:20:25.736247+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:31.939688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741730+0000","last_change":"2023-10-16T11:20:25.748924+0000","last_active":"2023-10-16T11:20:26.741730+0000","last_peered":"2023-10-16T11:20:26.741730+0000","last_clean":"2023-10-16T11:20:26.741730+0000","last_became_active":"2023-10-16T11:20:25.748637+0000","last_became_peered":"2023-10-16T11:20:25.748637+0000","last_unstale":"2023-10-16T11:20:26.741730+0000","last_undegraded":"2023-10-16T11:20:26.741730+0000","last_fullsized":"2023-10-16T11:20:26.741730+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:40.894330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"236'8","reported_seq":20,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.759908+0000","last_change":"2023-10-16T11:20:24.740384+0000","last_active":"2023-10-16T11:20:25.759908+0000","last_peered":"2023-10-16T11:20:25.759908+0000","last_clean":"2023-10-16T11:20:25.759908+0000","last_became_active":"2023-10-16T11:20:24.739999+0000","last_became_peered":"2023-10-16T11:20:24.739999+0000","last_unstale":"2023-10-16T11:20:25.759908+0000","last_undegraded":"2023-10-16T11:20:25.759908+0000","last_fullsized":"2023-10-16T11:20:25.759908+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:31.286100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741872+0000","last_change":"2023-10-16T11:20:25.749009+0000","last_active":"2023-10-16T11:20:26.741872+0000","last_peered":"2023-10-16T11:20:26.741872+0000","last_clean":"2023-10-16T11:20:26.741872+0000","last_became_active":"2023-10-16T11:20:25.748708+0000","last_became_peered":"2023-10-16T11:20:25.748708+0000","last_unstale":"2023-10-16T11:20:26.741872+0000","last_undegraded":"2023-10-16T11:20:26.741872+0000","last_fullsized":"2023-10-16T11:20:26.741872+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:55.578524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.762990+0000","last_change":"2023-10-16T11:20:24.731614+0000","last_active":"2023-10-16T11:20:25.762990+0000","last_peered":"2023-10-16T11:20:25.762990+0000","last_clean":"2023-10-16T11:20:25.762990+0000","last_became_active":"2023-10-16T11:20:24.731329+0000","last_became_peered":"2023-10-16T11:20:24.731329+0000","last_unstale":"2023-10-16T11:20:25.762990+0000","last_undegraded":"2023-10-16T11:20:25.762990+0000","last_fullsized":"2023-10-16T11:20:25.762990+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:39.887304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741287+0000","last_change":"2023-10-16T11:20:25.750514+0000","last_active":"2023-10-16T11:20:26.741287+0000","last_peered":"2023-10-16T11:20:26.741287+0000","last_clean":"2023-10-16T11:20:26.741287+0000","last_became_active":"2023-10-16T11:20:25.750226+0000","last_became_peered":"2023-10-16T11:20:25.750226+0000","last_unstale":"2023-10-16T11:20:26.741287+0000","last_undegraded":"2023-10-16T11:20:26.741287+0000","last_fullsized":"2023-10-16T11:20:26.741287+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:02.546128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757505+0000","last_change":"2023-10-16T11:20:24.740869+0000","last_active":"2023-10-16T11:20:25.757505+0000","last_peered":"2023-10-16T11:20:25.757505+0000","last_clean":"2023-10-16T11:20:25.757505+0000","last_became_active":"2023-10-16T11:20:24.740527+0000","last_became_peered":"2023-10-16T11:20:24.740527+0000","last_unstale":"2023-10-16T11:20:25.757505+0000","last_undegraded":"2023-10-16T11:20:25.757505+0000","last_fullsized":"2023-10-16T11:20:25.757505+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:49.172459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801513+0000","last_change":"2023-10-16T11:20:25.749628+0000","last_active":"2023-10-16T11:20:26.801513+0000","last_peered":"2023-10-16T11:20:26.801513+0000","last_clean":"2023-10-16T11:20:26.801513+0000","last_became_active":"2023-10-16T11:20:25.749358+0000","last_became_peered":"2023-10-16T11:20:25.749358+0000","last_unstale":"2023-10-16T11:20:26.801513+0000","last_undegraded":"2023-10-16T11:20:26.801513+0000","last_fullsized":"2023-10-16T11:20:26.801513+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:39.916608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757931+0000","last_change":"2023-10-16T11:20:24.735496+0000","last_active":"2023-10-16T11:20:25.757931+0000","last_peered":"2023-10-16T11:20:25.757931+0000","last_clean":"2023-10-16T11:20:25.757931+0000","last_became_active":"2023-10-16T11:20:24.734988+0000","last_became_peered":"2023-10-16T11:20:24.734988+0000","last_unstale":"2023-10-16T11:20:25.757931+0000","last_undegraded":"2023-10-16T11:20:25.757931+0000","last_fullsized":"2023-10-16T11:20:25.757931+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:22.060729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802742+0000","last_change":"2023-10-16T11:20:25.746364+0000","last_active":"2023-10-16T11:20:26.802742+0000","last_peered":"2023-10-16T11:20:26.802742+0000","last_clean":"2023-10-16T11:20:26.802742+0000","last_became_active":"2023-10-16T11:20:25.746108+0000","last_became_peered":"2023-10-16T11:20:25.746108+0000","last_unstale":"2023-10-16T11:20:26.802742+0000","last_undegraded":"2023-10-16T11:20:26.802742+0000","last_fullsized":"2023-10-16T11:20:26.802742+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:03.593372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.729604+0000","last_change":"2023-10-16T11:20:24.727424+0000","last_active":"2023-10-16T11:20:25.729604+0000","last_peered":"2023-10-16T11:20:25.729604+0000","last_clean":"2023-10-16T11:20:25.729604+0000","last_became_active":"2023-10-16T11:20:24.727122+0000","last_became_peered":"2023-10-16T11:20:24.727122+0000","last_unstale":"2023-10-16T11:20:25.729604+0000","last_undegraded":"2023-10-16T11:20:25.729604+0000","last_fullsized":"2023-10-16T11:20:25.729604+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:38.786539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802766+0000","last_change":"2023-10-16T11:20:25.741687+0000","last_active":"2023-10-16T11:20:26.802766+0000","last_peered":"2023-10-16T11:20:26.802766+0000","last_clean":"2023-10-16T11:20:26.802766+0000","last_became_active":"2023-10-16T11:20:25.741022+0000","last_became_peered":"2023-10-16T11:20:25.741022+0000","last_unstale":"2023-10-16T11:20:26.802766+0000","last_undegraded":"2023-10-16T11:20:26.802766+0000","last_fullsized":"2023-10-16T11:20:26.802766+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:23.748016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757355+0000","last_change":"2023-10-16T11:20:24.738500+0000","last_active":"2023-10-16T11:20:25.757355+0000","last_peered":"2023-10-16T11:20:25.757355+0000","last_clean":"2023-10-16T11:20:25.757355+0000","last_became_active":"2023-10-16T11:20:24.737980+0000","last_became_peered":"2023-10-16T11:20:24.737980+0000","last_unstale":"2023-10-16T11:20:25.757355+0000","last_undegraded":"2023-10-16T11:20:25.757355+0000","last_fullsized":"2023-10-16T11:20:25.757355+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:15.374424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801726+0000","last_change":"2023-10-16T11:20:25.752050+0000","last_active":"2023-10-16T11:20:26.801726+0000","last_peered":"2023-10-16T11:20:26.801726+0000","last_clean":"2023-10-16T11:20:26.801726+0000","last_became_active":"2023-10-16T11:20:25.751760+0000","last_became_peered":"2023-10-16T11:20:25.751760+0000","last_unstale":"2023-10-16T11:20:26.801726+0000","last_undegraded":"2023-10-16T11:20:26.801726+0000","last_fullsized":"2023-10-16T11:20:26.801726+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:40.238520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.731915+0000","last_change":"2023-10-16T11:20:24.727961+0000","last_active":"2023-10-16T11:20:25.731915+0000","last_peered":"2023-10-16T11:20:25.731915+0000","last_clean":"2023-10-16T11:20:25.731915+0000","last_became_active":"2023-10-16T11:20:24.727467+0000","last_became_peered":"2023-10-16T11:20:24.727467+0000","last_unstale":"2023-10-16T11:20:25.731915+0000","last_undegraded":"2023-10-16T11:20:25.731915+0000","last_fullsized":"2023-10-16T11:20:25.731915+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:33.847435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746862+0000","last_change":"2023-10-16T11:20:25.743908+0000","last_active":"2023-10-16T11:20:26.746862+0000","last_peered":"2023-10-16T11:20:26.746862+0000","last_clean":"2023-10-16T11:20:26.746862+0000","last_became_active":"2023-10-16T11:20:25.743570+0000","last_became_peered":"2023-10-16T11:20:25.743570+0000","last_unstale":"2023-10-16T11:20:26.746862+0000","last_undegraded":"2023-10-16T11:20:26.746862+0000","last_fullsized":"2023-10-16T11:20:26.746862+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:53.089548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732816+0000","last_change":"2023-10-16T11:20:24.735291+0000","last_active":"2023-10-16T11:20:25.732816+0000","last_peered":"2023-10-16T11:20:25.732816+0000","last_clean":"2023-10-16T11:20:25.732816+0000","last_became_active":"2023-10-16T11:20:24.734998+0000","last_became_peered":"2023-10-16T11:20:24.734998+0000","last_unstale":"2023-10-16T11:20:25.732816+0000","last_undegraded":"2023-10-16T11:20:25.732816+0000","last_fullsized":"2023-10-16T11:20:25.732816+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:26.682154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746666+0000","last_change":"2023-10-16T11:20:25.736861+0000","last_active":"2023-10-16T11:20:26.746666+0000","last_peered":"2023-10-16T11:20:26.746666+0000","last_clean":"2023-10-16T11:20:26.746666+0000","last_became_active":"2023-10-16T11:20:25.736595+0000","last_became_peered":"2023-10-16T11:20:25.736595+0000","last_unstale":"2023-10-16T11:20:26.746666+0000","last_undegraded":"2023-10-16T11:20:26.746666+0000","last_fullsized":"2023-10-16T11:20:26.746666+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:28.356199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732030+0000","last_change":"2023-10-16T11:20:24.729252+0000","last_active":"2023-10-16T11:20:25.732030+0000","last_peered":"2023-10-16T11:20:25.732030+0000","last_clean":"2023-10-16T11:20:25.732030+0000","last_became_active":"2023-10-16T11:20:24.729048+0000","last_became_peered":"2023-10-16T11:20:24.729048+0000","last_unstale":"2023-10-16T11:20:25.732030+0000","last_undegraded":"2023-10-16T11:20:25.732030+0000","last_fullsized":"2023-10-16T11:20:25.732030+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.587854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801850+0000","last_change":"2023-10-16T11:20:25.748002+0000","last_active":"2023-10-16T11:20:26.801850+0000","last_peered":"2023-10-16T11:20:26.801850+0000","last_clean":"2023-10-16T11:20:26.801850+0000","last_became_active":"2023-10-16T11:20:25.747521+0000","last_became_peered":"2023-10-16T11:20:25.747521+0000","last_unstale":"2023-10-16T11:20:26.801850+0000","last_undegraded":"2023-10-16T11:20:26.801850+0000","last_fullsized":"2023-10-16T11:20:26.801850+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.756762+0000","last_change":"2023-10-16T11:20:24.738296+0000","last_active":"2023-10-16T11:20:25.756762+0000","last_peered":"2023-10-16T11:20:25.756762+0000","last_clean":"2023-10-16T11:20:25.756762+0000","last_became_active":"2023-10-16T11:20:24.737992+0000","last_became_peered":"2023-10-16T11:20:24.737992+0000","last_unstale":"2023-10-16T11:20:25.756762+0000","last_undegraded":"2023-10-16T11:20:25.756762+0000","last_fullsized":"2023-10-16T11:20:25.756762+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:42.936914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740224+0000","last_change":"2023-10-16T11:20:25.736801+0000","last_active":"2023-10-16T11:20:26.740224+0000","last_peered":"2023-10-16T11:20:26.740224+0000","last_clean":"2023-10-16T11:20:26.740224+0000","last_became_active":"2023-10-16T11:20:25.736501+0000","last_became_peered":"2023-10-16T11:20:25.736501+0000","last_unstale":"2023-10-16T11:20:26.740224+0000","last_undegraded":"2023-10-16T11:20:26.740224+0000","last_fullsized":"2023-10-16T11:20:26.740224+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:41.787813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.737361+0000","last_change":"2023-10-16T11:20:24.734520+0000","last_active":"2023-10-16T11:20:25.737361+0000","last_peered":"2023-10-16T11:20:25.737361+0000","last_clean":"2023-10-16T11:20:25.737361+0000","last_became_active":"2023-10-16T11:20:24.733902+0000","last_became_peered":"2023-10-16T11:20:24.733902+0000","last_unstale":"2023-10-16T11:20:25.737361+0000","last_undegraded":"2023-10-16T11:20:25.737361+0000","last_fullsized":"2023-10-16T11:20:25.737361+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:43.725198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802492+0000","last_change":"2023-10-16T11:20:25.739360+0000","last_active":"2023-10-16T11:20:26.802492+0000","last_peered":"2023-10-16T11:20:26.802492+0000","last_clean":"2023-10-16T11:20:26.802492+0000","last_became_active":"2023-10-16T11:20:25.739110+0000","last_became_peered":"2023-10-16T11:20:25.739110+0000","last_unstale":"2023-10-16T11:20:26.802492+0000","last_undegraded":"2023-10-16T11:20:26.802492+0000","last_fullsized":"2023-10-16T11:20:26.802492+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:40.578713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732104+0000","last_change":"2023-10-16T11:20:24.736139+0000","last_active":"2023-10-16T11:20:25.732104+0000","last_peered":"2023-10-16T11:20:25.732104+0000","last_clean":"2023-10-16T11:20:25.732104+0000","last_became_active":"2023-10-16T11:20:24.735807+0000","last_became_peered":"2023-10-16T11:20:24.735807+0000","last_unstale":"2023-10-16T11:20:25.732104+0000","last_undegraded":"2023-10-16T11:20:25.732104+0000","last_fullsized":"2023-10-16T11:20:25.732104+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741838+0000","last_change":"2023-10-16T11:20:25.748753+0000","last_active":"2023-10-16T11:20:26.741838+0000","last_peered":"2023-10-16T11:20:26.741838+0000","last_clean":"2023-10-16T11:20:26.741838+0000","last_became_active":"2023-10-16T11:20:25.748480+0000","last_became_peered":"2023-10-16T11:20:25.748480+0000","last_unstale":"2023-10-16T11:20:26.741838+0000","last_undegraded":"2023-10-16T11:20:26.741838+0000","last_fullsized":"2023-10-16T11:20:26.741838+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:51.963113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.733403+0000","last_change":"2023-10-16T11:20:24.729644+0000","last_active":"2023-10-16T11:20:25.733403+0000","last_peered":"2023-10-16T11:20:25.733403+0000","last_clean":"2023-10-16T11:20:25.733403+0000","last_became_active":"2023-10-16T11:20:24.729487+0000","last_became_peered":"2023-10-16T11:20:24.729487+0000","last_unstale":"2023-10-16T11:20:25.733403+0000","last_undegraded":"2023-10-16T11:20:25.733403+0000","last_fullsized":"2023-10-16T11:20:25.733403+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:45.766549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803475+0000","last_change":"2023-10-16T11:20:25.740829+0000","last_active":"2023-10-16T11:20:26.803475+0000","last_peered":"2023-10-16T11:20:26.803475+0000","last_clean":"2023-10-16T11:20:26.803475+0000","last_became_active":"2023-10-16T11:20:25.740552+0000","last_became_peered":"2023-10-16T11:20:25.740552+0000","last_unstale":"2023-10-16T11:20:26.803475+0000","last_undegraded":"2023-10-16T11:20:26.803475+0000","last_fullsized":"2023-10-16T11:20:26.803475+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732759+0000","last_change":"2023-10-16T11:20:24.727846+0000","last_active":"2023-10-16T11:20:25.732759+0000","last_peered":"2023-10-16T11:20:25.732759+0000","last_clean":"2023-10-16T11:20:25.732759+0000","last_became_active":"2023-10-16T11:20:24.727551+0000","last_became_peered":"2023-10-16T11:20:24.727551+0000","last_unstale":"2023-10-16T11:20:25.732759+0000","last_undegraded":"2023-10-16T11:20:25.732759+0000","last_fullsized":"2023-10-16T11:20:25.732759+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:10.257213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802994+0000","last_change":"2023-10-16T11:20:25.748408+0000","last_active":"2023-10-16T11:20:26.802994+0000","last_peered":"2023-10-16T11:20:26.802994+0000","last_clean":"2023-10-16T11:20:26.802994+0000","last_became_active":"2023-10-16T11:20:25.747574+0000","last_became_peered":"2023-10-16T11:20:25.747574+0000","last_unstale":"2023-10-16T11:20:26.802994+0000","last_undegraded":"2023-10-16T11:20:26.802994+0000","last_fullsized":"2023-10-16T11:20:26.802994+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:57.537856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736743+0000","last_change":"2023-10-16T11:20:24.728924+0000","last_active":"2023-10-16T11:20:25.736743+0000","last_peered":"2023-10-16T11:20:25.736743+0000","last_clean":"2023-10-16T11:20:25.736743+0000","last_became_active":"2023-10-16T11:20:24.728614+0000","last_became_peered":"2023-10-16T11:20:24.728614+0000","last_unstale":"2023-10-16T11:20:25.736743+0000","last_undegraded":"2023-10-16T11:20:25.736743+0000","last_fullsized":"2023-10-16T11:20:25.736743+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:23.737157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801568+0000","last_change":"2023-10-16T11:20:25.751879+0000","last_active":"2023-10-16T11:20:26.801568+0000","last_peered":"2023-10-16T11:20:26.801568+0000","last_clean":"2023-10-16T11:20:26.801568+0000","last_became_active":"2023-10-16T11:20:25.750440+0000","last_became_peered":"2023-10-16T11:20:25.750440+0000","last_unstale":"2023-10-16T11:20:26.801568+0000","last_undegraded":"2023-10-16T11:20:26.801568+0000","last_fullsized":"2023-10-16T11:20:26.801568+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:57.379246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736961+0000","last_change":"2023-10-16T11:20:24.732437+0000","last_active":"2023-10-16T11:20:25.736961+0000","last_peered":"2023-10-16T11:20:25.736961+0000","last_clean":"2023-10-16T11:20:25.736961+0000","last_became_active":"2023-10-16T11:20:24.732111+0000","last_became_peered":"2023-10-16T11:20:24.732111+0000","last_unstale":"2023-10-16T11:20:25.736961+0000","last_undegraded":"2023-10-16T11:20:25.736961+0000","last_fullsized":"2023-10-16T11:20:25.736961+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:26.846516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746538+0000","last_change":"2023-10-16T11:20:25.750621+0000","last_active":"2023-10-16T11:20:26.746538+0000","last_peered":"2023-10-16T11:20:26.746538+0000","last_clean":"2023-10-16T11:20:26.746538+0000","last_became_active":"2023-10-16T11:20:25.750352+0000","last_became_peered":"2023-10-16T11:20:25.750352+0000","last_unstale":"2023-10-16T11:20:26.746538+0000","last_undegraded":"2023-10-16T11:20:26.746538+0000","last_fullsized":"2023-10-16T11:20:26.746538+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:57.295597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.731291+0000","last_change":"2023-10-16T11:20:24.733888+0000","last_active":"2023-10-16T11:20:25.731291+0000","last_peered":"2023-10-16T11:20:25.731291+0000","last_clean":"2023-10-16T11:20:25.731291+0000","last_became_active":"2023-10-16T11:20:24.733588+0000","last_became_peered":"2023-10-16T11:20:24.733588+0000","last_unstale":"2023-10-16T11:20:25.731291+0000","last_undegraded":"2023-10-16T11:20:25.731291+0000","last_fullsized":"2023-10-16T11:20:25.731291+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:06.768468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802430+0000","last_change":"2023-10-16T11:20:25.748051+0000","last_active":"2023-10-16T11:20:26.802430+0000","last_peered":"2023-10-16T11:20:26.802430+0000","last_clean":"2023-10-16T11:20:26.802430+0000","last_became_active":"2023-10-16T11:20:25.747647+0000","last_became_peered":"2023-10-16T11:20:25.747647+0000","last_unstale":"2023-10-16T11:20:26.802430+0000","last_undegraded":"2023-10-16T11:20:26.802430+0000","last_fullsized":"2023-10-16T11:20:26.802430+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:06.072095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.761389+0000","last_change":"2023-10-16T11:20:24.736168+0000","last_active":"2023-10-16T11:20:25.761389+0000","last_peered":"2023-10-16T11:20:25.761389+0000","last_clean":"2023-10-16T11:20:25.761389+0000","last_became_active":"2023-10-16T11:20:24.735886+0000","last_became_peered":"2023-10-16T11:20:24.735886+0000","last_unstale":"2023-10-16T11:20:25.761389+0000","last_undegraded":"2023-10-16T11:20:25.761389+0000","last_fullsized":"2023-10-16T11:20:25.761389+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"223'370","reported_seq":1306,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.729253+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:20:25.729253+0000","last_peered":"2023-10-16T11:20:25.729253+0000","last_clean":"2023-10-16T11:20:25.729253+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:20:25.729253+0000","last_undegraded":"2023-10-16T11:20:25.729253+0000","last_fullsized":"2023-10-16T11:20:25.729253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576000,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52748,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319092,"statfs":{"total":96636764160,"available":96582750208,"internally_reserved":0,"allocated":3596288,"data_stored":2200288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.67200000000000004,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.372},"max":{"1min":9.4359999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.68400000000000005,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":9.3059999999999992,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.71599999999999997,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":9.3610000000000007,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.119,"5min":0.71999999999999997,"15min":0.68100000000000005},"min":{"1min":0.49399999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.3960000000000008,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.72099999999999997,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":9.2889999999999997,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.753,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.36099999999999999},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.157,"5min":0.76500000000000001,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":9.4689999999999994,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.71999999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.61199999999999999,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.747},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.63100000000000001,"15min":0.56699999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.5399999999999991,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.64000000000000001,"15min":0.57399999999999995},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.69399999999999995,"15min":0.63100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.5690000000000008,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.71199999999999997,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.155,"5min":0.69799999999999995,"15min":0.64500000000000002},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576000,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56328,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315512,"statfs":{"total":96636764160,"available":96579084288,"internally_reserved":0,"allocated":3002368,"data_stored":1609908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.91000000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.53500000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.70599999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64000000000000001,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.625},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.335,"5min":3.335,"15min":3.335},"last":0.64300000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576001,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47992,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323848,"statfs":{"total":96636764160,"available":96587620352,"internally_reserved":0,"allocated":2994176,"data_stored":1609796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.622,"15min":0.66700000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.436,"15min":0.45200000000000001},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.52500000000000002,"5min":0.63900000000000001,"15min":0.71699999999999997},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.64000000000000001,"15min":0.70099999999999996},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.442},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":0.623,"15min":0.70499999999999996},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.48099999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65400000000000003,"15min":0.67200000000000004},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":0.89500000000000002,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90000000000000002,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.90300000000000002,"15min":1.0620000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.016},"last":0.57299999999999995}]}]},{"osd":0,"up_from":13,"seq":55834576001,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61068,"kb_used_data":3516,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310772,"statfs":{"total":96636764160,"available":96574230528,"internally_reserved":0,"allocated":3600384,"data_stored":2201776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.45000000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.26700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52800000000000002,"5min":0.66100000000000003,"15min":0.77200000000000002},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":1.272,"15min":1.272},"last":0.309},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40899999999999997,"15min":0.433},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.374},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54200000000000004,"5min":1.351,"15min":1.351},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83099999999999996,"5min":1.391,"15min":1.391},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76000000000000001,"5min":0.874,"15min":1.089},"last":0.62},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.624,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.95599999999999996,"15min":1.216},"last":0.496}]},{"osd":6,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.47199999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.1180000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.67200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80300000000000005,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80500000000000005,"5min":0.89600000000000002,"15min":1.0780000000000001},"last":0.59999999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576000,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48004,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323836,"statfs":{"total":96636764160,"available":96587608064,"internally_reserved":0,"allocated":3002368,"data_stored":1610448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.371,"15min":0.39500000000000002},"min":{"1min":0.253,"5min":0.188,"15min":0.188},"max":{"1min":0.51800000000000002,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.42799999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.193,"15min":0.193},"max":{"1min":0.53600000000000003,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.44400000000000001,"15min":0.46500000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59499999999999997,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39000000000000001,"15min":0.41599999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.45500000000000002,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.436}]},{"osd":3,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.442},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56699999999999995,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.499},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48799999999999999,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.95399999999999996,"15min":1.097},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64700000000000002,"15min":0.67300000000000004},"min":{"1min":0.438,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77100000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63700000000000001,"15min":0.66700000000000004},"min":{"1min":0.45500000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82499999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.69199999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81899999999999995,"5min":1.0169999999999999,"15min":1.0389999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64500000000000002,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76700000000000002,"5min":0.97799999999999998,"15min":1.022},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67700000000000005,"15min":0.70299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84999999999999998,"5min":0.98999999999999999,"15min":1.004},"last":0.79900000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.69199999999999995,"15min":0.70599999999999996},"min":{"1min":0.48599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81000000000000005,"5min":0.99299999999999999,"15min":1.071},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.67100000000000004,"15min":0.69799999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81200000000000006,"5min":0.97799999999999998,"15min":1.077},"last":0.78400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576000,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52176,"kb_used_data":2948,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319664,"statfs":{"total":96636764160,"available":96583335936,"internally_reserved":0,"allocated":3018752,"data_stored":1612046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.376,"15min":0.39500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38300000000000001,"15min":0.40699999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":1.0029999999999999},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42499999999999999,"15min":0.441},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46400000000000002,"15min":0.48299999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.50800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.024},"last":0.377}]},{"osd":4,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.68000000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.026},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.41799999999999998},"max":{"1min":5.8399999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.622},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69899999999999995,"15min":0.70499999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.4409999999999998,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.66600000000000004,"15min":0.69999999999999996},"min":{"1min":0.45300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77800000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.58199999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576000,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52164,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319676,"statfs":{"total":96636764160,"available":96583348224,"internally_reserved":0,"allocated":2998272,"data_stored":1609866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56999999999999995,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67700000000000005,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57599999999999996,"15min":0.61799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.61499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74099999999999999,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.61399999999999999},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.622},"min":{"1min":0.40400000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60799999999999998,"15min":0.63700000000000001},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.67000000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.61599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67900000000000005,"5min":0.83799999999999997,"15min":0.995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48199999999999998,"15min":0.5},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71499999999999997,"5min":0.73599999999999999,"15min":0.86199999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76300000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.749}]},{"osd":6,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.76200000000000001,"5min":0.878,"15min":1.0609999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.745,"5min":0.83099999999999996,"15min":1.115},"last":0.79500000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.83999999999999997,"5min":0.84499999999999997,"15min":0.93200000000000005},"last":0.77200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576000,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66048,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305792,"statfs":{"total":96636764160,"available":96569131008,"internally_reserved":0,"allocated":2994176,"data_stored":1609796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61199999999999999,"15min":0.63200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.28999999999999998},"max":{"1min":0.78000000000000003,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.62,"15min":0.65000000000000002},"min":{"1min":0.39700000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72999999999999998,"5min":0.871,"15min":0.88200000000000001},"last":0.67900000000000005}]},{"osd":1,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.57699999999999996,"15min":0.59499999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.752,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.501,"5min":0.56399999999999995,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.59099999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73399999999999999,"5min":0.80300000000000005,"15min":0.86499999999999999},"last":0.623},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58399999999999996,"15min":0.61699999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.752,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.61099999999999999},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57999999999999996,"15min":0.60399999999999998},"min":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.70299999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80900000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.33200000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.82499999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84499999999999997,"5min":0.878,"15min":0.878},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.57299999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78300000000000003,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:20:29.862 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:20:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.015+0000 7fc48e179700 1 Processor -- start 2023-10-16T11:20:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.015+0000 7fc48e179700 1 -- start start 2023-10-16T11:20:30.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.015+0000 7fc48e179700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc488072360 0x7fc488072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.015+0000 7fc48e179700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806a8d0 0x7fc48806ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.015+0000 7fc48e179700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48806b240 con 0x7fc48806f470 2023-10-16T11:20:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.015+0000 7fc48e179700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc488072d50 con 0x7fc488072360 2023-10-16T11:20:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.015+0000 7fc48e179700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48806f850 con 0x7fc48806a8d0 2023-10-16T11:20:30.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc488072360 0x7fc488072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc488072360 0x7fc488072810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40332/0 (socket says 172.21.15.158:40332) 2023-10-16T11:20:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 -- 172.21.15.158:0/1456389414 learned_addr learned my addr 172.21.15.158:0/1456389414 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 -- 172.21.15.158:0/1456389414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806a8d0 msgr2=0x7fc48806ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 --2- 172.21.15.158:0/1456389414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806a8d0 0x7fc48806ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 -- 172.21.15.158:0/1456389414 >> v1:172.21.15.172:6789/0 conn(0x7fc48806f470 legacy=0x7fc48806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 -- 172.21.15.158:0/1456389414 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4880b4260 con 0x7fc488072360 2023-10-16T11:20:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc486ffd700 1 --2- 172.21.15.158:0/1456389414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc488072810 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7fc4780098f0 tx=0x7fc478009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bc10acb0e8e382 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc4867fc700 1 -- 172.21.15.158:0/1456389414 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc478004230 con 0x7fc488072360 2023-10-16T11:20:30.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc4867fc700 1 -- 172.21.15.158:0/1456389414 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc478004390 con 0x7fc488072360 2023-10-16T11:20:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.016+0000 7fc4867fc700 1 -- 172.21.15.158:0/1456389414 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4780036a0 con 0x7fc488072360 2023-10-16T11:20:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 -- 172.21.15.158:0/1456389414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 msgr2=0x7fc488072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 --2- 172.21.15.158:0/1456389414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc488072810 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7fc4780098f0 tx=0x7fc478009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 -- 172.21.15.158:0/1456389414 shutdown_connections 2023-10-16T11:20:30.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 --2- 172.21.15.158:0/1456389414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806a8d0 0x7fc48806ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 --2- 172.21.15.158:0/1456389414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc488072810 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 -- 172.21.15.158:0/1456389414 >> 172.21.15.158:0/1456389414 conn(0x7fc4880ccda0 msgr2=0x7fc4880cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 -- 172.21.15.158:0/1456389414 shutdown_connections 2023-10-16T11:20:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.017+0000 7fc48e179700 1 -- 172.21.15.158:0/1456389414 wait complete. 2023-10-16T11:20:30.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.018+0000 7fc48e179700 1 Processor -- start 2023-10-16T11:20:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.018+0000 7fc48e179700 1 -- start start 2023-10-16T11:20:30.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc48e179700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc48806a8d0 0x7fc4880c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc48e179700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806f470 0x7fc4880c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc48e179700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc48815a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc48e179700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881e8430 con 0x7fc488072360 2023-10-16T11:20:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc48e179700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881e7fa0 con 0x7fc48806a8d0 2023-10-16T11:20:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc48e179700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881e7390 con 0x7fc48806f470 2023-10-16T11:20:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc48815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc48815a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40340/0 (socket says 172.21.15.158:40340) 2023-10-16T11:20:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 -- 172.21.15.158:0/1923490348 learned_addr learned my addr 172.21.15.158:0/1923490348 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc4877fe700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc48806a8d0 0x7fc4880c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc486ffd700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806f470 0x7fc4880c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 -- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806f470 msgr2=0x7fc4880c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806f470 0x7fc4880c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 -- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc48806a8d0 msgr2=0x7fc4880c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc48806a8d0 0x7fc4880c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 -- 172.21.15.158:0/1923490348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc478009550 con 0x7fc488072360 2023-10-16T11:20:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc487fff700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc48815a810 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7fc48000b970 tx=0x7fc48000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.019+0000 7fc46ffff700 1 -- 172.21.15.158:0/1923490348 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000c9b0 con 0x7fc488072360 2023-10-16T11:20:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.020+0000 7fc46ffff700 1 -- 172.21.15.158:0/1923490348 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48000cb30 con 0x7fc488072360 2023-10-16T11:20:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.020+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4881e7670 con 0x7fc488072360 2023-10-16T11:20:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.020+0000 7fc46ffff700 1 -- 172.21.15.158:0/1923490348 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480014440 con 0x7fc488072360 2023-10-16T11:20:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.020+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4881e7c30 con 0x7fc488072360 2023-10-16T11:20:30.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.020+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4880ad620 con 0x7fc488072360 2023-10-16T11:20:30.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.021+0000 7fc46ffff700 1 -- 172.21.15.158:0/1923490348 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc48001d3b0 con 0x7fc488072360 2023-10-16T11:20:30.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.021+0000 7fc46ffff700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc47406b140 0x7fc47406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.021+0000 7fc46ffff700 1 -- 172.21.15.158:0/1923490348 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fc480006e40 con 0x7fc488072360 2023-10-16T11:20:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.021+0000 7fc4877fe700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc47406b140 0x7fc47406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.022+0000 7fc4877fe700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc47406b140 0x7fc47406d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc47c004f80 tx=0x7fc47c009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.026+0000 7fc46ffff700 1 -- 172.21.15.158:0/1923490348 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4800544f0 con 0x7fc488072360 2023-10-16T11:20:30.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.214+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 --> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc488071b60 con 0x7fc47406b140 2023-10-16T11:20:30.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.222+0000 7fc46ffff700 1 -- 172.21.15.158:0/1923490348 <== mgr.12030 v2:172.21.15.158:6832/3408161846 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289944 (secure 0 0 0) 0x7fc47c0038e0 con 0x7fc47406b140 2023-10-16T11:20:30.228 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:30.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc47406b140 msgr2=0x7fc47406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc47406b140 0x7fc47406d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc47c004f80 tx=0x7fc47c009040 comp rx=0 tx=0).stop 2023-10-16T11:20:30.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 msgr2=0x7fc48815a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc48815a810 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7fc48000b970 tx=0x7fc48000bd30 comp rx=0 tx=0).stop 2023-10-16T11:20:30.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 shutdown_connections 2023-10-16T11:20:30.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fc47406b140 0x7fc47406d600 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc488072360 0x7fc48815a810 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc48806a8d0 0x7fc4880c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 --2- 172.21.15.158:0/1923490348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc48806f470 0x7fc4880c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 >> 172.21.15.158:0/1923490348 conn(0x7fc4880ccda0 msgr2=0x7fc4880acf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:30.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 shutdown_connections 2023-10-16T11:20:30.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.225+0000 7fc48e179700 1 -- 172.21.15.158:0/1923490348 wait complete. 2023-10-16T11:20:30.232 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:20:30.239 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T11:20:29.448594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436528,"kb_used_data":24616,"kb_used_omap":12,"kb_used_meta":411763,"kb_avail":754538192,"statfs":{"total":773094113280,"available":772647108608,"internally_reserved":0,"allocated":25206784,"data_stored":14063924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":421645879},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.734617"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802700+0000","last_change":"2023-10-16T11:20:25.748352+0000","last_active":"2023-10-16T11:20:26.802700+0000","last_peered":"2023-10-16T11:20:26.802700+0000","last_clean":"2023-10-16T11:20:26.802700+0000","last_became_active":"2023-10-16T11:20:25.747848+0000","last_became_peered":"2023-10-16T11:20:25.747848+0000","last_unstale":"2023-10-16T11:20:26.802700+0000","last_undegraded":"2023-10-16T11:20:26.802700+0000","last_fullsized":"2023-10-16T11:20:26.802700+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:48.171042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801655+0000","last_change":"2023-10-16T11:20:25.749395+0000","last_active":"2023-10-16T11:20:26.801655+0000","last_peered":"2023-10-16T11:20:26.801655+0000","last_clean":"2023-10-16T11:20:26.801655+0000","last_became_active":"2023-10-16T11:20:25.749044+0000","last_became_peered":"2023-10-16T11:20:25.749044+0000","last_unstale":"2023-10-16T11:20:26.801655+0000","last_undegraded":"2023-10-16T11:20:26.801655+0000","last_fullsized":"2023-10-16T11:20:26.801655+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:16.211507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803081+0000","last_change":"2023-10-16T11:20:25.739817+0000","last_active":"2023-10-16T11:20:26.803081+0000","last_peered":"2023-10-16T11:20:26.803081+0000","last_clean":"2023-10-16T11:20:26.803081+0000","last_became_active":"2023-10-16T11:20:25.739601+0000","last_became_peered":"2023-10-16T11:20:25.739601+0000","last_unstale":"2023-10-16T11:20:26.803081+0000","last_undegraded":"2023-10-16T11:20:26.803081+0000","last_fullsized":"2023-10-16T11:20:26.803081+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:38.443038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802907+0000","last_change":"2023-10-16T11:20:25.751269+0000","last_active":"2023-10-16T11:20:26.802907+0000","last_peered":"2023-10-16T11:20:26.802907+0000","last_clean":"2023-10-16T11:20:26.802907+0000","last_became_active":"2023-10-16T11:20:25.750973+0000","last_became_peered":"2023-10-16T11:20:25.750973+0000","last_unstale":"2023-10-16T11:20:26.802907+0000","last_undegraded":"2023-10-16T11:20:26.802907+0000","last_fullsized":"2023-10-16T11:20:26.802907+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:36.490706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741753+0000","last_change":"2023-10-16T11:20:25.750599+0000","last_active":"2023-10-16T11:20:26.741753+0000","last_peered":"2023-10-16T11:20:26.741753+0000","last_clean":"2023-10-16T11:20:26.741753+0000","last_became_active":"2023-10-16T11:20:25.749962+0000","last_became_peered":"2023-10-16T11:20:25.749962+0000","last_unstale":"2023-10-16T11:20:26.741753+0000","last_undegraded":"2023-10-16T11:20:26.741753+0000","last_fullsized":"2023-10-16T11:20:26.741753+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:15.073727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802717+0000","last_change":"2023-10-16T11:20:25.745973+0000","last_active":"2023-10-16T11:20:26.802717+0000","last_peered":"2023-10-16T11:20:26.802717+0000","last_clean":"2023-10-16T11:20:26.802717+0000","last_became_active":"2023-10-16T11:20:25.745706+0000","last_became_peered":"2023-10-16T11:20:25.745706+0000","last_unstale":"2023-10-16T11:20:26.802717+0000","last_undegraded":"2023-10-16T11:20:26.802717+0000","last_fullsized":"2023-10-16T11:20:26.802717+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:08.628698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801829+0000","last_change":"2023-10-16T11:20:25.748211+0000","last_active":"2023-10-16T11:20:26.801829+0000","last_peered":"2023-10-16T11:20:26.801829+0000","last_clean":"2023-10-16T11:20:26.801829+0000","last_became_active":"2023-10-16T11:20:25.747956+0000","last_became_peered":"2023-10-16T11:20:25.747956+0000","last_unstale":"2023-10-16T11:20:26.801829+0000","last_undegraded":"2023-10-16T11:20:26.801829+0000","last_fullsized":"2023-10-16T11:20:26.801829+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741324+0000","last_change":"2023-10-16T11:20:25.750901+0000","last_active":"2023-10-16T11:20:26.741324+0000","last_peered":"2023-10-16T11:20:26.741324+0000","last_clean":"2023-10-16T11:20:26.741324+0000","last_became_active":"2023-10-16T11:20:25.750582+0000","last_became_peered":"2023-10-16T11:20:25.750582+0000","last_unstale":"2023-10-16T11:20:26.741324+0000","last_undegraded":"2023-10-16T11:20:26.741324+0000","last_fullsized":"2023-10-16T11:20:26.741324+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:47.707747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802565+0000","last_change":"2023-10-16T11:20:25.750204+0000","last_active":"2023-10-16T11:20:26.802565+0000","last_peered":"2023-10-16T11:20:26.802565+0000","last_clean":"2023-10-16T11:20:26.802565+0000","last_became_active":"2023-10-16T11:20:25.749903+0000","last_became_peered":"2023-10-16T11:20:25.749903+0000","last_unstale":"2023-10-16T11:20:26.802565+0000","last_undegraded":"2023-10-16T11:20:26.802565+0000","last_fullsized":"2023-10-16T11:20:26.802565+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:48.734101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803228+0000","last_change":"2023-10-16T11:20:25.747726+0000","last_active":"2023-10-16T11:20:26.803228+0000","last_peered":"2023-10-16T11:20:26.803228+0000","last_clean":"2023-10-16T11:20:26.803228+0000","last_became_active":"2023-10-16T11:20:25.747314+0000","last_became_peered":"2023-10-16T11:20:25.747314+0000","last_unstale":"2023-10-16T11:20:26.803228+0000","last_undegraded":"2023-10-16T11:20:26.803228+0000","last_fullsized":"2023-10-16T11:20:26.803228+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803230+0000","last_change":"2023-10-16T11:20:25.747416+0000","last_active":"2023-10-16T11:20:26.803230+0000","last_peered":"2023-10-16T11:20:26.803230+0000","last_clean":"2023-10-16T11:20:26.803230+0000","last_became_active":"2023-10-16T11:20:25.746950+0000","last_became_peered":"2023-10-16T11:20:25.746950+0000","last_unstale":"2023-10-16T11:20:26.803230+0000","last_undegraded":"2023-10-16T11:20:26.803230+0000","last_fullsized":"2023-10-16T11:20:26.803230+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803775+0000","last_change":"2023-10-16T11:20:25.743709+0000","last_active":"2023-10-16T11:20:26.803775+0000","last_peered":"2023-10-16T11:20:26.803775+0000","last_clean":"2023-10-16T11:20:26.803775+0000","last_became_active":"2023-10-16T11:20:25.743444+0000","last_became_peered":"2023-10-16T11:20:25.743444+0000","last_unstale":"2023-10-16T11:20:26.803775+0000","last_undegraded":"2023-10-16T11:20:26.803775+0000","last_fullsized":"2023-10-16T11:20:26.803775+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:28.662644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746334+0000","last_change":"2023-10-16T11:20:25.737922+0000","last_active":"2023-10-16T11:20:26.746334+0000","last_peered":"2023-10-16T11:20:26.746334+0000","last_clean":"2023-10-16T11:20:26.746334+0000","last_became_active":"2023-10-16T11:20:25.737086+0000","last_became_peered":"2023-10-16T11:20:25.737086+0000","last_unstale":"2023-10-16T11:20:26.746334+0000","last_undegraded":"2023-10-16T11:20:26.746334+0000","last_fullsized":"2023-10-16T11:20:26.746334+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:17.897224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803610+0000","last_change":"2023-10-16T11:20:25.745458+0000","last_active":"2023-10-16T11:20:26.803610+0000","last_peered":"2023-10-16T11:20:26.803610+0000","last_clean":"2023-10-16T11:20:26.803610+0000","last_became_active":"2023-10-16T11:20:25.745257+0000","last_became_peered":"2023-10-16T11:20:25.745257+0000","last_unstale":"2023-10-16T11:20:26.803610+0000","last_undegraded":"2023-10-16T11:20:26.803610+0000","last_fullsized":"2023-10-16T11:20:26.803610+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740363+0000","last_change":"2023-10-16T11:20:25.741269+0000","last_active":"2023-10-16T11:20:26.740363+0000","last_peered":"2023-10-16T11:20:26.740363+0000","last_clean":"2023-10-16T11:20:26.740363+0000","last_became_active":"2023-10-16T11:20:25.740995+0000","last_became_peered":"2023-10-16T11:20:25.740995+0000","last_unstale":"2023-10-16T11:20:26.740363+0000","last_undegraded":"2023-10-16T11:20:26.740363+0000","last_fullsized":"2023-10-16T11:20:26.740363+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:45.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802182+0000","last_change":"2023-10-16T11:20:25.742299+0000","last_active":"2023-10-16T11:20:26.802182+0000","last_peered":"2023-10-16T11:20:26.802182+0000","last_clean":"2023-10-16T11:20:26.802182+0000","last_became_active":"2023-10-16T11:20:25.741394+0000","last_became_peered":"2023-10-16T11:20:25.741394+0000","last_unstale":"2023-10-16T11:20:26.802182+0000","last_undegraded":"2023-10-16T11:20:26.802182+0000","last_fullsized":"2023-10-16T11:20:26.802182+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:35.560067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746702+0000","last_change":"2023-10-16T11:20:25.744534+0000","last_active":"2023-10-16T11:20:26.746702+0000","last_peered":"2023-10-16T11:20:26.746702+0000","last_clean":"2023-10-16T11:20:26.746702+0000","last_became_active":"2023-10-16T11:20:25.744224+0000","last_became_peered":"2023-10-16T11:20:25.744224+0000","last_unstale":"2023-10-16T11:20:26.746702+0000","last_undegraded":"2023-10-16T11:20:26.746702+0000","last_fullsized":"2023-10-16T11:20:26.746702+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:14.437943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803456+0000","last_change":"2023-10-16T11:20:25.747551+0000","last_active":"2023-10-16T11:20:26.803456+0000","last_peered":"2023-10-16T11:20:26.803456+0000","last_clean":"2023-10-16T11:20:26.803456+0000","last_became_active":"2023-10-16T11:20:25.746989+0000","last_became_peered":"2023-10-16T11:20:25.746989+0000","last_unstale":"2023-10-16T11:20:26.803456+0000","last_undegraded":"2023-10-16T11:20:26.803456+0000","last_fullsized":"2023-10-16T11:20:26.803456+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740647+0000","last_change":"2023-10-16T11:20:25.738568+0000","last_active":"2023-10-16T11:20:26.740647+0000","last_peered":"2023-10-16T11:20:26.740647+0000","last_clean":"2023-10-16T11:20:26.740647+0000","last_became_active":"2023-10-16T11:20:25.738313+0000","last_became_peered":"2023-10-16T11:20:25.738313+0000","last_unstale":"2023-10-16T11:20:26.740647+0000","last_undegraded":"2023-10-16T11:20:26.740647+0000","last_fullsized":"2023-10-16T11:20:26.740647+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:28.152434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802035+0000","last_change":"2023-10-16T11:20:25.750312+0000","last_active":"2023-10-16T11:20:26.802035+0000","last_peered":"2023-10-16T11:20:26.802035+0000","last_clean":"2023-10-16T11:20:26.802035+0000","last_became_active":"2023-10-16T11:20:25.749320+0000","last_became_peered":"2023-10-16T11:20:25.749320+0000","last_unstale":"2023-10-16T11:20:26.802035+0000","last_undegraded":"2023-10-16T11:20:26.802035+0000","last_fullsized":"2023-10-16T11:20:26.802035+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:26.044494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802039+0000","last_change":"2023-10-16T11:20:25.744924+0000","last_active":"2023-10-16T11:20:26.802039+0000","last_peered":"2023-10-16T11:20:26.802039+0000","last_clean":"2023-10-16T11:20:26.802039+0000","last_became_active":"2023-10-16T11:20:25.744659+0000","last_became_peered":"2023-10-16T11:20:25.744659+0000","last_unstale":"2023-10-16T11:20:26.802039+0000","last_undegraded":"2023-10-16T11:20:26.802039+0000","last_fullsized":"2023-10-16T11:20:26.802039+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746563+0000","last_change":"2023-10-16T11:20:25.750717+0000","last_active":"2023-10-16T11:20:26.746563+0000","last_peered":"2023-10-16T11:20:26.746563+0000","last_clean":"2023-10-16T11:20:26.746563+0000","last_became_active":"2023-10-16T11:20:25.750419+0000","last_became_peered":"2023-10-16T11:20:25.750419+0000","last_unstale":"2023-10-16T11:20:26.746563+0000","last_undegraded":"2023-10-16T11:20:26.746563+0000","last_fullsized":"2023-10-16T11:20:26.746563+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:21.095229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740584+0000","last_change":"2023-10-16T11:20:25.738444+0000","last_active":"2023-10-16T11:20:26.740584+0000","last_peered":"2023-10-16T11:20:26.740584+0000","last_clean":"2023-10-16T11:20:26.740584+0000","last_became_active":"2023-10-16T11:20:25.738166+0000","last_became_peered":"2023-10-16T11:20:25.738166+0000","last_unstale":"2023-10-16T11:20:26.740584+0000","last_undegraded":"2023-10-16T11:20:26.740584+0000","last_fullsized":"2023-10-16T11:20:26.740584+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:57.655885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746045+0000","last_change":"2023-10-16T11:20:25.741211+0000","last_active":"2023-10-16T11:20:26.746045+0000","last_peered":"2023-10-16T11:20:26.746045+0000","last_clean":"2023-10-16T11:20:26.746045+0000","last_became_active":"2023-10-16T11:20:25.740922+0000","last_became_peered":"2023-10-16T11:20:25.740922+0000","last_unstale":"2023-10-16T11:20:26.746045+0000","last_undegraded":"2023-10-16T11:20:26.746045+0000","last_fullsized":"2023-10-16T11:20:26.746045+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:55.904667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801586+0000","last_change":"2023-10-16T11:20:25.741581+0000","last_active":"2023-10-16T11:20:26.801586+0000","last_peered":"2023-10-16T11:20:26.801586+0000","last_clean":"2023-10-16T11:20:26.801586+0000","last_became_active":"2023-10-16T11:20:25.740490+0000","last_became_peered":"2023-10-16T11:20:25.740490+0000","last_unstale":"2023-10-16T11:20:26.801586+0000","last_undegraded":"2023-10-16T11:20:26.801586+0000","last_fullsized":"2023-10-16T11:20:26.801586+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:20.166253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802066+0000","last_change":"2023-10-16T11:20:25.747818+0000","last_active":"2023-10-16T11:20:26.802066+0000","last_peered":"2023-10-16T11:20:26.802066+0000","last_clean":"2023-10-16T11:20:26.802066+0000","last_became_active":"2023-10-16T11:20:25.747533+0000","last_became_peered":"2023-10-16T11:20:25.747533+0000","last_unstale":"2023-10-16T11:20:26.802066+0000","last_undegraded":"2023-10-16T11:20:26.802066+0000","last_fullsized":"2023-10-16T11:20:26.802066+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741725+0000","last_change":"2023-10-16T11:20:25.746406+0000","last_active":"2023-10-16T11:20:26.741725+0000","last_peered":"2023-10-16T11:20:26.741725+0000","last_clean":"2023-10-16T11:20:26.741725+0000","last_became_active":"2023-10-16T11:20:25.746140+0000","last_became_peered":"2023-10-16T11:20:25.746140+0000","last_unstale":"2023-10-16T11:20:26.741725+0000","last_undegraded":"2023-10-16T11:20:26.741725+0000","last_fullsized":"2023-10-16T11:20:26.741725+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:42.488771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802723+0000","last_change":"2023-10-16T11:20:25.750643+0000","last_active":"2023-10-16T11:20:26.802723+0000","last_peered":"2023-10-16T11:20:26.802723+0000","last_clean":"2023-10-16T11:20:26.802723+0000","last_became_active":"2023-10-16T11:20:25.749980+0000","last_became_peered":"2023-10-16T11:20:25.749980+0000","last_unstale":"2023-10-16T11:20:26.802723+0000","last_undegraded":"2023-10-16T11:20:26.802723+0000","last_fullsized":"2023-10-16T11:20:26.802723+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:41.498772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803323+0000","last_change":"2023-10-16T11:20:25.744400+0000","last_active":"2023-10-16T11:20:26.803323+0000","last_peered":"2023-10-16T11:20:26.803323+0000","last_clean":"2023-10-16T11:20:26.803323+0000","last_became_active":"2023-10-16T11:20:25.744118+0000","last_became_peered":"2023-10-16T11:20:25.744118+0000","last_unstale":"2023-10-16T11:20:26.803323+0000","last_undegraded":"2023-10-16T11:20:26.803323+0000","last_fullsized":"2023-10-16T11:20:26.803323+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803736+0000","last_change":"2023-10-16T11:20:25.749096+0000","last_active":"2023-10-16T11:20:26.803736+0000","last_peered":"2023-10-16T11:20:26.803736+0000","last_clean":"2023-10-16T11:20:26.803736+0000","last_became_active":"2023-10-16T11:20:25.748841+0000","last_became_peered":"2023-10-16T11:20:25.748841+0000","last_unstale":"2023-10-16T11:20:26.803736+0000","last_undegraded":"2023-10-16T11:20:26.803736+0000","last_fullsized":"2023-10-16T11:20:26.803736+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801386+0000","last_change":"2023-10-16T11:20:25.741998+0000","last_active":"2023-10-16T11:20:26.801386+0000","last_peered":"2023-10-16T11:20:26.801386+0000","last_clean":"2023-10-16T11:20:26.801386+0000","last_became_active":"2023-10-16T11:20:25.741482+0000","last_became_peered":"2023-10-16T11:20:25.741482+0000","last_unstale":"2023-10-16T11:20:26.801386+0000","last_undegraded":"2023-10-16T11:20:26.801386+0000","last_fullsized":"2023-10-16T11:20:26.801386+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:01.922741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802196+0000","last_change":"2023-10-16T11:20:25.750764+0000","last_active":"2023-10-16T11:20:26.802196+0000","last_peered":"2023-10-16T11:20:26.802196+0000","last_clean":"2023-10-16T11:20:26.802196+0000","last_became_active":"2023-10-16T11:20:25.750127+0000","last_became_peered":"2023-10-16T11:20:25.750127+0000","last_unstale":"2023-10-16T11:20:26.802196+0000","last_undegraded":"2023-10-16T11:20:26.802196+0000","last_fullsized":"2023-10-16T11:20:26.802196+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.429724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801485+0000","last_change":"2023-10-16T11:20:25.752205+0000","last_active":"2023-10-16T11:20:26.801485+0000","last_peered":"2023-10-16T11:20:26.801485+0000","last_clean":"2023-10-16T11:20:26.801485+0000","last_became_active":"2023-10-16T11:20:25.751915+0000","last_became_peered":"2023-10-16T11:20:25.751915+0000","last_unstale":"2023-10-16T11:20:26.801485+0000","last_undegraded":"2023-10-16T11:20:26.801485+0000","last_fullsized":"2023-10-16T11:20:26.801485+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:43.213188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"236'4","reported_seq":16,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.763173+0000","last_change":"2023-10-16T11:20:24.733695+0000","last_active":"2023-10-16T11:20:25.763173+0000","last_peered":"2023-10-16T11:20:25.763173+0000","last_clean":"2023-10-16T11:20:25.763173+0000","last_became_active":"2023-10-16T11:20:24.733413+0000","last_became_peered":"2023-10-16T11:20:24.733413+0000","last_unstale":"2023-10-16T11:20:25.763173+0000","last_undegraded":"2023-10-16T11:20:25.763173+0000","last_fullsized":"2023-10-16T11:20:25.763173+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:34.326531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740347+0000","last_change":"2023-10-16T11:20:25.747992+0000","last_active":"2023-10-16T11:20:26.740347+0000","last_peered":"2023-10-16T11:20:26.740347+0000","last_clean":"2023-10-16T11:20:26.740347+0000","last_became_active":"2023-10-16T11:20:25.747735+0000","last_became_peered":"2023-10-16T11:20:25.747735+0000","last_unstale":"2023-10-16T11:20:26.740347+0000","last_undegraded":"2023-10-16T11:20:26.740347+0000","last_fullsized":"2023-10-16T11:20:26.740347+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:49.269723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757593+0000","last_change":"2023-10-16T11:20:24.739822+0000","last_active":"2023-10-16T11:20:25.757593+0000","last_peered":"2023-10-16T11:20:25.757593+0000","last_clean":"2023-10-16T11:20:25.757593+0000","last_became_active":"2023-10-16T11:20:24.739470+0000","last_became_peered":"2023-10-16T11:20:24.739470+0000","last_unstale":"2023-10-16T11:20:25.757593+0000","last_undegraded":"2023-10-16T11:20:25.757593+0000","last_fullsized":"2023-10-16T11:20:25.757593+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:18.641488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802991+0000","last_change":"2023-10-16T11:20:25.746049+0000","last_active":"2023-10-16T11:20:26.802991+0000","last_peered":"2023-10-16T11:20:26.802991+0000","last_clean":"2023-10-16T11:20:26.802991+0000","last_became_active":"2023-10-16T11:20:25.745759+0000","last_became_peered":"2023-10-16T11:20:25.745759+0000","last_unstale":"2023-10-16T11:20:26.802991+0000","last_undegraded":"2023-10-16T11:20:26.802991+0000","last_fullsized":"2023-10-16T11:20:26.802991+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:40:29.501267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.765321+0000","last_change":"2023-10-16T11:20:24.727708+0000","last_active":"2023-10-16T11:20:25.765321+0000","last_peered":"2023-10-16T11:20:25.765321+0000","last_clean":"2023-10-16T11:20:25.765321+0000","last_became_active":"2023-10-16T11:20:24.727357+0000","last_became_peered":"2023-10-16T11:20:24.727357+0000","last_unstale":"2023-10-16T11:20:25.765321+0000","last_undegraded":"2023-10-16T11:20:25.765321+0000","last_fullsized":"2023-10-16T11:20:25.765321+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:32.307241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803875+0000","last_change":"2023-10-16T11:20:25.742133+0000","last_active":"2023-10-16T11:20:26.803875+0000","last_peered":"2023-10-16T11:20:26.803875+0000","last_clean":"2023-10-16T11:20:26.803875+0000","last_became_active":"2023-10-16T11:20:25.741865+0000","last_became_peered":"2023-10-16T11:20:25.741865+0000","last_unstale":"2023-10-16T11:20:26.803875+0000","last_undegraded":"2023-10-16T11:20:26.803875+0000","last_fullsized":"2023-10-16T11:20:26.803875+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:25.230100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757467+0000","last_change":"2023-10-16T11:20:24.722283+0000","last_active":"2023-10-16T11:20:25.757467+0000","last_peered":"2023-10-16T11:20:25.757467+0000","last_clean":"2023-10-16T11:20:25.757467+0000","last_became_active":"2023-10-16T11:20:24.721789+0000","last_became_peered":"2023-10-16T11:20:24.721789+0000","last_unstale":"2023-10-16T11:20:25.757467+0000","last_undegraded":"2023-10-16T11:20:25.757467+0000","last_fullsized":"2023-10-16T11:20:25.757467+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:37.116862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802129+0000","last_change":"2023-10-16T11:20:25.748913+0000","last_active":"2023-10-16T11:20:26.802129+0000","last_peered":"2023-10-16T11:20:26.802129+0000","last_clean":"2023-10-16T11:20:26.802129+0000","last_became_active":"2023-10-16T11:20:25.748588+0000","last_became_peered":"2023-10-16T11:20:25.748588+0000","last_unstale":"2023-10-16T11:20:26.802129+0000","last_undegraded":"2023-10-16T11:20:26.802129+0000","last_fullsized":"2023-10-16T11:20:26.802129+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.765029+0000","last_change":"2023-10-16T11:20:24.733622+0000","last_active":"2023-10-16T11:20:25.765029+0000","last_peered":"2023-10-16T11:20:25.765029+0000","last_clean":"2023-10-16T11:20:25.765029+0000","last_became_active":"2023-10-16T11:20:24.733342+0000","last_became_peered":"2023-10-16T11:20:24.733342+0000","last_unstale":"2023-10-16T11:20:25.765029+0000","last_undegraded":"2023-10-16T11:20:25.765029+0000","last_fullsized":"2023-10-16T11:20:25.765029+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:10.962482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803222+0000","last_change":"2023-10-16T11:20:25.751310+0000","last_active":"2023-10-16T11:20:26.803222+0000","last_peered":"2023-10-16T11:20:26.803222+0000","last_clean":"2023-10-16T11:20:26.803222+0000","last_became_active":"2023-10-16T11:20:25.751042+0000","last_became_peered":"2023-10-16T11:20:25.751042+0000","last_unstale":"2023-10-16T11:20:26.803222+0000","last_undegraded":"2023-10-16T11:20:26.803222+0000","last_fullsized":"2023-10-16T11:20:26.803222+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736174+0000","last_change":"2023-10-16T11:20:24.739609+0000","last_active":"2023-10-16T11:20:25.736174+0000","last_peered":"2023-10-16T11:20:25.736174+0000","last_clean":"2023-10-16T11:20:25.736174+0000","last_became_active":"2023-10-16T11:20:24.739275+0000","last_became_peered":"2023-10-16T11:20:24.739275+0000","last_unstale":"2023-10-16T11:20:25.736174+0000","last_undegraded":"2023-10-16T11:20:25.736174+0000","last_fullsized":"2023-10-16T11:20:25.736174+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:47.827300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741853+0000","last_change":"2023-10-16T11:20:25.744793+0000","last_active":"2023-10-16T11:20:26.741853+0000","last_peered":"2023-10-16T11:20:26.741853+0000","last_clean":"2023-10-16T11:20:26.741853+0000","last_became_active":"2023-10-16T11:20:25.744521+0000","last_became_peered":"2023-10-16T11:20:25.744521+0000","last_unstale":"2023-10-16T11:20:26.741853+0000","last_undegraded":"2023-10-16T11:20:26.741853+0000","last_fullsized":"2023-10-16T11:20:26.741853+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:44.539888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.747287+0000","last_change":"2023-10-16T11:20:24.723659+0000","last_active":"2023-10-16T11:20:25.747287+0000","last_peered":"2023-10-16T11:20:25.747287+0000","last_clean":"2023-10-16T11:20:25.747287+0000","last_became_active":"2023-10-16T11:20:24.723342+0000","last_became_peered":"2023-10-16T11:20:24.723342+0000","last_unstale":"2023-10-16T11:20:25.747287+0000","last_undegraded":"2023-10-16T11:20:25.747287+0000","last_fullsized":"2023-10-16T11:20:25.747287+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:29.707065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802133+0000","last_change":"2023-10-16T11:20:25.748568+0000","last_active":"2023-10-16T11:20:26.802133+0000","last_peered":"2023-10-16T11:20:26.802133+0000","last_clean":"2023-10-16T11:20:26.802133+0000","last_became_active":"2023-10-16T11:20:25.748318+0000","last_became_peered":"2023-10-16T11:20:25.748318+0000","last_unstale":"2023-10-16T11:20:26.802133+0000","last_undegraded":"2023-10-16T11:20:26.802133+0000","last_fullsized":"2023-10-16T11:20:26.802133+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732300+0000","last_change":"2023-10-16T11:20:24.736095+0000","last_active":"2023-10-16T11:20:25.732300+0000","last_peered":"2023-10-16T11:20:25.732300+0000","last_clean":"2023-10-16T11:20:25.732300+0000","last_became_active":"2023-10-16T11:20:24.735775+0000","last_became_peered":"2023-10-16T11:20:24.735775+0000","last_unstale":"2023-10-16T11:20:25.732300+0000","last_undegraded":"2023-10-16T11:20:25.732300+0000","last_fullsized":"2023-10-16T11:20:25.732300+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:35.485188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746270+0000","last_change":"2023-10-16T11:20:25.745589+0000","last_active":"2023-10-16T11:20:26.746270+0000","last_peered":"2023-10-16T11:20:26.746270+0000","last_clean":"2023-10-16T11:20:26.746270+0000","last_became_active":"2023-10-16T11:20:25.745297+0000","last_became_peered":"2023-10-16T11:20:25.745297+0000","last_unstale":"2023-10-16T11:20:26.746270+0000","last_undegraded":"2023-10-16T11:20:26.746270+0000","last_fullsized":"2023-10-16T11:20:26.746270+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:17.743037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.737188+0000","last_change":"2023-10-16T11:20:24.738634+0000","last_active":"2023-10-16T11:20:25.737188+0000","last_peered":"2023-10-16T11:20:25.737188+0000","last_clean":"2023-10-16T11:20:25.737188+0000","last_became_active":"2023-10-16T11:20:24.738372+0000","last_became_peered":"2023-10-16T11:20:24.738372+0000","last_unstale":"2023-10-16T11:20:25.737188+0000","last_undegraded":"2023-10-16T11:20:25.737188+0000","last_fullsized":"2023-10-16T11:20:25.737188+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:23.747207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803537+0000","last_change":"2023-10-16T11:20:25.748579+0000","last_active":"2023-10-16T11:20:26.803537+0000","last_peered":"2023-10-16T11:20:26.803537+0000","last_clean":"2023-10-16T11:20:26.803537+0000","last_became_active":"2023-10-16T11:20:25.748347+0000","last_became_peered":"2023-10-16T11:20:25.748347+0000","last_unstale":"2023-10-16T11:20:26.803537+0000","last_undegraded":"2023-10-16T11:20:26.803537+0000","last_fullsized":"2023-10-16T11:20:26.803537+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"236'3","reported_seq":15,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.758189+0000","last_change":"2023-10-16T11:20:24.728938+0000","last_active":"2023-10-16T11:20:25.758189+0000","last_peered":"2023-10-16T11:20:25.758189+0000","last_clean":"2023-10-16T11:20:25.758189+0000","last_became_active":"2023-10-16T11:20:24.728638+0000","last_became_peered":"2023-10-16T11:20:24.728638+0000","last_unstale":"2023-10-16T11:20:25.758189+0000","last_undegraded":"2023-10-16T11:20:25.758189+0000","last_fullsized":"2023-10-16T11:20:25.758189+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:42.082000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801991+0000","last_change":"2023-10-16T11:20:25.751420+0000","last_active":"2023-10-16T11:20:26.801991+0000","last_peered":"2023-10-16T11:20:26.801991+0000","last_clean":"2023-10-16T11:20:26.801991+0000","last_became_active":"2023-10-16T11:20:25.751161+0000","last_became_peered":"2023-10-16T11:20:25.751161+0000","last_unstale":"2023-10-16T11:20:26.801991+0000","last_undegraded":"2023-10-16T11:20:26.801991+0000","last_fullsized":"2023-10-16T11:20:26.801991+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"236'6","reported_seq":18,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.766988+0000","last_change":"2023-10-16T11:20:24.724201+0000","last_active":"2023-10-16T11:20:25.766988+0000","last_peered":"2023-10-16T11:20:25.766988+0000","last_clean":"2023-10-16T11:20:25.766988+0000","last_became_active":"2023-10-16T11:20:24.723918+0000","last_became_peered":"2023-10-16T11:20:24.723918+0000","last_unstale":"2023-10-16T11:20:25.766988+0000","last_undegraded":"2023-10-16T11:20:25.766988+0000","last_fullsized":"2023-10-16T11:20:25.766988+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:55.483353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801992+0000","last_change":"2023-10-16T11:20:25.742201+0000","last_active":"2023-10-16T11:20:26.801992+0000","last_peered":"2023-10-16T11:20:26.801992+0000","last_clean":"2023-10-16T11:20:26.801992+0000","last_became_active":"2023-10-16T11:20:25.741871+0000","last_became_peered":"2023-10-16T11:20:25.741871+0000","last_unstale":"2023-10-16T11:20:26.801992+0000","last_undegraded":"2023-10-16T11:20:26.801992+0000","last_fullsized":"2023-10-16T11:20:26.801992+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:18:28.417174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.761652+0000","last_change":"2023-10-16T11:20:24.725162+0000","last_active":"2023-10-16T11:20:25.761652+0000","last_peered":"2023-10-16T11:20:25.761652+0000","last_clean":"2023-10-16T11:20:25.761652+0000","last_became_active":"2023-10-16T11:20:24.724795+0000","last_became_peered":"2023-10-16T11:20:24.724795+0000","last_unstale":"2023-10-16T11:20:25.761652+0000","last_undegraded":"2023-10-16T11:20:25.761652+0000","last_fullsized":"2023-10-16T11:20:25.761652+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:25.854243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801952+0000","last_change":"2023-10-16T11:20:25.739623+0000","last_active":"2023-10-16T11:20:26.801952+0000","last_peered":"2023-10-16T11:20:26.801952+0000","last_clean":"2023-10-16T11:20:26.801952+0000","last_became_active":"2023-10-16T11:20:25.739321+0000","last_became_peered":"2023-10-16T11:20:25.739321+0000","last_unstale":"2023-10-16T11:20:26.801952+0000","last_undegraded":"2023-10-16T11:20:26.801952+0000","last_fullsized":"2023-10-16T11:20:26.801952+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736599+0000","last_change":"2023-10-16T11:20:24.735120+0000","last_active":"2023-10-16T11:20:25.736599+0000","last_peered":"2023-10-16T11:20:25.736599+0000","last_clean":"2023-10-16T11:20:25.736599+0000","last_became_active":"2023-10-16T11:20:24.734857+0000","last_became_peered":"2023-10-16T11:20:24.734857+0000","last_unstale":"2023-10-16T11:20:25.736599+0000","last_undegraded":"2023-10-16T11:20:25.736599+0000","last_fullsized":"2023-10-16T11:20:25.736599+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:03.791128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.742231+0000","last_change":"2023-10-16T11:20:25.737997+0000","last_active":"2023-10-16T11:20:26.742231+0000","last_peered":"2023-10-16T11:20:26.742231+0000","last_clean":"2023-10-16T11:20:26.742231+0000","last_became_active":"2023-10-16T11:20:25.737314+0000","last_became_peered":"2023-10-16T11:20:25.737314+0000","last_unstale":"2023-10-16T11:20:26.742231+0000","last_undegraded":"2023-10-16T11:20:26.742231+0000","last_fullsized":"2023-10-16T11:20:26.742231+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:44.489552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736247+0000","last_change":"2023-10-16T11:20:24.733587+0000","last_active":"2023-10-16T11:20:25.736247+0000","last_peered":"2023-10-16T11:20:25.736247+0000","last_clean":"2023-10-16T11:20:25.736247+0000","last_became_active":"2023-10-16T11:20:24.733160+0000","last_became_peered":"2023-10-16T11:20:24.733160+0000","last_unstale":"2023-10-16T11:20:25.736247+0000","last_undegraded":"2023-10-16T11:20:25.736247+0000","last_fullsized":"2023-10-16T11:20:25.736247+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:31.939688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741730+0000","last_change":"2023-10-16T11:20:25.748924+0000","last_active":"2023-10-16T11:20:26.741730+0000","last_peered":"2023-10-16T11:20:26.741730+0000","last_clean":"2023-10-16T11:20:26.741730+0000","last_became_active":"2023-10-16T11:20:25.748637+0000","last_became_peered":"2023-10-16T11:20:25.748637+0000","last_unstale":"2023-10-16T11:20:26.741730+0000","last_undegraded":"2023-10-16T11:20:26.741730+0000","last_fullsized":"2023-10-16T11:20:26.741730+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:40.894330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"236'8","reported_seq":20,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.759908+0000","last_change":"2023-10-16T11:20:24.740384+0000","last_active":"2023-10-16T11:20:25.759908+0000","last_peered":"2023-10-16T11:20:25.759908+0000","last_clean":"2023-10-16T11:20:25.759908+0000","last_became_active":"2023-10-16T11:20:24.739999+0000","last_became_peered":"2023-10-16T11:20:24.739999+0000","last_unstale":"2023-10-16T11:20:25.759908+0000","last_undegraded":"2023-10-16T11:20:25.759908+0000","last_fullsized":"2023-10-16T11:20:25.759908+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:31.286100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741872+0000","last_change":"2023-10-16T11:20:25.749009+0000","last_active":"2023-10-16T11:20:26.741872+0000","last_peered":"2023-10-16T11:20:26.741872+0000","last_clean":"2023-10-16T11:20:26.741872+0000","last_became_active":"2023-10-16T11:20:25.748708+0000","last_became_peered":"2023-10-16T11:20:25.748708+0000","last_unstale":"2023-10-16T11:20:26.741872+0000","last_undegraded":"2023-10-16T11:20:26.741872+0000","last_fullsized":"2023-10-16T11:20:26.741872+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:55.578524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.762990+0000","last_change":"2023-10-16T11:20:24.731614+0000","last_active":"2023-10-16T11:20:25.762990+0000","last_peered":"2023-10-16T11:20:25.762990+0000","last_clean":"2023-10-16T11:20:25.762990+0000","last_became_active":"2023-10-16T11:20:24.731329+0000","last_became_peered":"2023-10-16T11:20:24.731329+0000","last_unstale":"2023-10-16T11:20:25.762990+0000","last_undegraded":"2023-10-16T11:20:25.762990+0000","last_fullsized":"2023-10-16T11:20:25.762990+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:39.887304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741287+0000","last_change":"2023-10-16T11:20:25.750514+0000","last_active":"2023-10-16T11:20:26.741287+0000","last_peered":"2023-10-16T11:20:26.741287+0000","last_clean":"2023-10-16T11:20:26.741287+0000","last_became_active":"2023-10-16T11:20:25.750226+0000","last_became_peered":"2023-10-16T11:20:25.750226+0000","last_unstale":"2023-10-16T11:20:26.741287+0000","last_undegraded":"2023-10-16T11:20:26.741287+0000","last_fullsized":"2023-10-16T11:20:26.741287+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:02.546128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757505+0000","last_change":"2023-10-16T11:20:24.740869+0000","last_active":"2023-10-16T11:20:25.757505+0000","last_peered":"2023-10-16T11:20:25.757505+0000","last_clean":"2023-10-16T11:20:25.757505+0000","last_became_active":"2023-10-16T11:20:24.740527+0000","last_became_peered":"2023-10-16T11:20:24.740527+0000","last_unstale":"2023-10-16T11:20:25.757505+0000","last_undegraded":"2023-10-16T11:20:25.757505+0000","last_fullsized":"2023-10-16T11:20:25.757505+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:49.172459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801513+0000","last_change":"2023-10-16T11:20:25.749628+0000","last_active":"2023-10-16T11:20:26.801513+0000","last_peered":"2023-10-16T11:20:26.801513+0000","last_clean":"2023-10-16T11:20:26.801513+0000","last_became_active":"2023-10-16T11:20:25.749358+0000","last_became_peered":"2023-10-16T11:20:25.749358+0000","last_unstale":"2023-10-16T11:20:26.801513+0000","last_undegraded":"2023-10-16T11:20:26.801513+0000","last_fullsized":"2023-10-16T11:20:26.801513+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:39.916608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757931+0000","last_change":"2023-10-16T11:20:24.735496+0000","last_active":"2023-10-16T11:20:25.757931+0000","last_peered":"2023-10-16T11:20:25.757931+0000","last_clean":"2023-10-16T11:20:25.757931+0000","last_became_active":"2023-10-16T11:20:24.734988+0000","last_became_peered":"2023-10-16T11:20:24.734988+0000","last_unstale":"2023-10-16T11:20:25.757931+0000","last_undegraded":"2023-10-16T11:20:25.757931+0000","last_fullsized":"2023-10-16T11:20:25.757931+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:22.060729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802742+0000","last_change":"2023-10-16T11:20:25.746364+0000","last_active":"2023-10-16T11:20:26.802742+0000","last_peered":"2023-10-16T11:20:26.802742+0000","last_clean":"2023-10-16T11:20:26.802742+0000","last_became_active":"2023-10-16T11:20:25.746108+0000","last_became_peered":"2023-10-16T11:20:25.746108+0000","last_unstale":"2023-10-16T11:20:26.802742+0000","last_undegraded":"2023-10-16T11:20:26.802742+0000","last_fullsized":"2023-10-16T11:20:26.802742+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:03.593372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.729604+0000","last_change":"2023-10-16T11:20:24.727424+0000","last_active":"2023-10-16T11:20:25.729604+0000","last_peered":"2023-10-16T11:20:25.729604+0000","last_clean":"2023-10-16T11:20:25.729604+0000","last_became_active":"2023-10-16T11:20:24.727122+0000","last_became_peered":"2023-10-16T11:20:24.727122+0000","last_unstale":"2023-10-16T11:20:25.729604+0000","last_undegraded":"2023-10-16T11:20:25.729604+0000","last_fullsized":"2023-10-16T11:20:25.729604+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:38.786539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802766+0000","last_change":"2023-10-16T11:20:25.741687+0000","last_active":"2023-10-16T11:20:26.802766+0000","last_peered":"2023-10-16T11:20:26.802766+0000","last_clean":"2023-10-16T11:20:26.802766+0000","last_became_active":"2023-10-16T11:20:25.741022+0000","last_became_peered":"2023-10-16T11:20:25.741022+0000","last_unstale":"2023-10-16T11:20:26.802766+0000","last_undegraded":"2023-10-16T11:20:26.802766+0000","last_fullsized":"2023-10-16T11:20:26.802766+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:23.748016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.757355+0000","last_change":"2023-10-16T11:20:24.738500+0000","last_active":"2023-10-16T11:20:25.757355+0000","last_peered":"2023-10-16T11:20:25.757355+0000","last_clean":"2023-10-16T11:20:25.757355+0000","last_became_active":"2023-10-16T11:20:24.737980+0000","last_became_peered":"2023-10-16T11:20:24.737980+0000","last_unstale":"2023-10-16T11:20:25.757355+0000","last_undegraded":"2023-10-16T11:20:25.757355+0000","last_fullsized":"2023-10-16T11:20:25.757355+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:15.374424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801726+0000","last_change":"2023-10-16T11:20:25.752050+0000","last_active":"2023-10-16T11:20:26.801726+0000","last_peered":"2023-10-16T11:20:26.801726+0000","last_clean":"2023-10-16T11:20:26.801726+0000","last_became_active":"2023-10-16T11:20:25.751760+0000","last_became_peered":"2023-10-16T11:20:25.751760+0000","last_unstale":"2023-10-16T11:20:26.801726+0000","last_undegraded":"2023-10-16T11:20:26.801726+0000","last_fullsized":"2023-10-16T11:20:26.801726+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:40.238520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.731915+0000","last_change":"2023-10-16T11:20:24.727961+0000","last_active":"2023-10-16T11:20:25.731915+0000","last_peered":"2023-10-16T11:20:25.731915+0000","last_clean":"2023-10-16T11:20:25.731915+0000","last_became_active":"2023-10-16T11:20:24.727467+0000","last_became_peered":"2023-10-16T11:20:24.727467+0000","last_unstale":"2023-10-16T11:20:25.731915+0000","last_undegraded":"2023-10-16T11:20:25.731915+0000","last_fullsized":"2023-10-16T11:20:25.731915+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:33.847435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746862+0000","last_change":"2023-10-16T11:20:25.743908+0000","last_active":"2023-10-16T11:20:26.746862+0000","last_peered":"2023-10-16T11:20:26.746862+0000","last_clean":"2023-10-16T11:20:26.746862+0000","last_became_active":"2023-10-16T11:20:25.743570+0000","last_became_peered":"2023-10-16T11:20:25.743570+0000","last_unstale":"2023-10-16T11:20:26.746862+0000","last_undegraded":"2023-10-16T11:20:26.746862+0000","last_fullsized":"2023-10-16T11:20:26.746862+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:53.089548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732816+0000","last_change":"2023-10-16T11:20:24.735291+0000","last_active":"2023-10-16T11:20:25.732816+0000","last_peered":"2023-10-16T11:20:25.732816+0000","last_clean":"2023-10-16T11:20:25.732816+0000","last_became_active":"2023-10-16T11:20:24.734998+0000","last_became_peered":"2023-10-16T11:20:24.734998+0000","last_unstale":"2023-10-16T11:20:25.732816+0000","last_undegraded":"2023-10-16T11:20:25.732816+0000","last_fullsized":"2023-10-16T11:20:25.732816+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:26.682154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746666+0000","last_change":"2023-10-16T11:20:25.736861+0000","last_active":"2023-10-16T11:20:26.746666+0000","last_peered":"2023-10-16T11:20:26.746666+0000","last_clean":"2023-10-16T11:20:26.746666+0000","last_became_active":"2023-10-16T11:20:25.736595+0000","last_became_peered":"2023-10-16T11:20:25.736595+0000","last_unstale":"2023-10-16T11:20:26.746666+0000","last_undegraded":"2023-10-16T11:20:26.746666+0000","last_fullsized":"2023-10-16T11:20:26.746666+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:28.356199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732030+0000","last_change":"2023-10-16T11:20:24.729252+0000","last_active":"2023-10-16T11:20:25.732030+0000","last_peered":"2023-10-16T11:20:25.732030+0000","last_clean":"2023-10-16T11:20:25.732030+0000","last_became_active":"2023-10-16T11:20:24.729048+0000","last_became_peered":"2023-10-16T11:20:24.729048+0000","last_unstale":"2023-10-16T11:20:25.732030+0000","last_undegraded":"2023-10-16T11:20:25.732030+0000","last_fullsized":"2023-10-16T11:20:25.732030+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.587854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801850+0000","last_change":"2023-10-16T11:20:25.748002+0000","last_active":"2023-10-16T11:20:26.801850+0000","last_peered":"2023-10-16T11:20:26.801850+0000","last_clean":"2023-10-16T11:20:26.801850+0000","last_became_active":"2023-10-16T11:20:25.747521+0000","last_became_peered":"2023-10-16T11:20:25.747521+0000","last_unstale":"2023-10-16T11:20:26.801850+0000","last_undegraded":"2023-10-16T11:20:26.801850+0000","last_fullsized":"2023-10-16T11:20:26.801850+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"236'2","reported_seq":14,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.756762+0000","last_change":"2023-10-16T11:20:24.738296+0000","last_active":"2023-10-16T11:20:25.756762+0000","last_peered":"2023-10-16T11:20:25.756762+0000","last_clean":"2023-10-16T11:20:25.756762+0000","last_became_active":"2023-10-16T11:20:24.737992+0000","last_became_peered":"2023-10-16T11:20:24.737992+0000","last_unstale":"2023-10-16T11:20:25.756762+0000","last_undegraded":"2023-10-16T11:20:25.756762+0000","last_fullsized":"2023-10-16T11:20:25.756762+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:42.936914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.740224+0000","last_change":"2023-10-16T11:20:25.736801+0000","last_active":"2023-10-16T11:20:26.740224+0000","last_peered":"2023-10-16T11:20:26.740224+0000","last_clean":"2023-10-16T11:20:26.740224+0000","last_became_active":"2023-10-16T11:20:25.736501+0000","last_became_peered":"2023-10-16T11:20:25.736501+0000","last_unstale":"2023-10-16T11:20:26.740224+0000","last_undegraded":"2023-10-16T11:20:26.740224+0000","last_fullsized":"2023-10-16T11:20:26.740224+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:41.787813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.737361+0000","last_change":"2023-10-16T11:20:24.734520+0000","last_active":"2023-10-16T11:20:25.737361+0000","last_peered":"2023-10-16T11:20:25.737361+0000","last_clean":"2023-10-16T11:20:25.737361+0000","last_became_active":"2023-10-16T11:20:24.733902+0000","last_became_peered":"2023-10-16T11:20:24.733902+0000","last_unstale":"2023-10-16T11:20:25.737361+0000","last_undegraded":"2023-10-16T11:20:25.737361+0000","last_fullsized":"2023-10-16T11:20:25.737361+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:43.725198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802492+0000","last_change":"2023-10-16T11:20:25.739360+0000","last_active":"2023-10-16T11:20:26.802492+0000","last_peered":"2023-10-16T11:20:26.802492+0000","last_clean":"2023-10-16T11:20:26.802492+0000","last_became_active":"2023-10-16T11:20:25.739110+0000","last_became_peered":"2023-10-16T11:20:25.739110+0000","last_unstale":"2023-10-16T11:20:26.802492+0000","last_undegraded":"2023-10-16T11:20:26.802492+0000","last_fullsized":"2023-10-16T11:20:26.802492+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:40.578713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732104+0000","last_change":"2023-10-16T11:20:24.736139+0000","last_active":"2023-10-16T11:20:25.732104+0000","last_peered":"2023-10-16T11:20:25.732104+0000","last_clean":"2023-10-16T11:20:25.732104+0000","last_became_active":"2023-10-16T11:20:24.735807+0000","last_became_peered":"2023-10-16T11:20:24.735807+0000","last_unstale":"2023-10-16T11:20:25.732104+0000","last_undegraded":"2023-10-16T11:20:25.732104+0000","last_fullsized":"2023-10-16T11:20:25.732104+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.741838+0000","last_change":"2023-10-16T11:20:25.748753+0000","last_active":"2023-10-16T11:20:26.741838+0000","last_peered":"2023-10-16T11:20:26.741838+0000","last_clean":"2023-10-16T11:20:26.741838+0000","last_became_active":"2023-10-16T11:20:25.748480+0000","last_became_peered":"2023-10-16T11:20:25.748480+0000","last_unstale":"2023-10-16T11:20:26.741838+0000","last_undegraded":"2023-10-16T11:20:26.741838+0000","last_fullsized":"2023-10-16T11:20:26.741838+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:51.963113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.733403+0000","last_change":"2023-10-16T11:20:24.729644+0000","last_active":"2023-10-16T11:20:25.733403+0000","last_peered":"2023-10-16T11:20:25.733403+0000","last_clean":"2023-10-16T11:20:25.733403+0000","last_became_active":"2023-10-16T11:20:24.729487+0000","last_became_peered":"2023-10-16T11:20:24.729487+0000","last_unstale":"2023-10-16T11:20:25.733403+0000","last_undegraded":"2023-10-16T11:20:25.733403+0000","last_fullsized":"2023-10-16T11:20:25.733403+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:45.766549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.803475+0000","last_change":"2023-10-16T11:20:25.740829+0000","last_active":"2023-10-16T11:20:26.803475+0000","last_peered":"2023-10-16T11:20:26.803475+0000","last_clean":"2023-10-16T11:20:26.803475+0000","last_became_active":"2023-10-16T11:20:25.740552+0000","last_became_peered":"2023-10-16T11:20:25.740552+0000","last_unstale":"2023-10-16T11:20:26.803475+0000","last_undegraded":"2023-10-16T11:20:26.803475+0000","last_fullsized":"2023-10-16T11:20:26.803475+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.732759+0000","last_change":"2023-10-16T11:20:24.727846+0000","last_active":"2023-10-16T11:20:25.732759+0000","last_peered":"2023-10-16T11:20:25.732759+0000","last_clean":"2023-10-16T11:20:25.732759+0000","last_became_active":"2023-10-16T11:20:24.727551+0000","last_became_peered":"2023-10-16T11:20:24.727551+0000","last_unstale":"2023-10-16T11:20:25.732759+0000","last_undegraded":"2023-10-16T11:20:25.732759+0000","last_fullsized":"2023-10-16T11:20:25.732759+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:10.257213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802994+0000","last_change":"2023-10-16T11:20:25.748408+0000","last_active":"2023-10-16T11:20:26.802994+0000","last_peered":"2023-10-16T11:20:26.802994+0000","last_clean":"2023-10-16T11:20:26.802994+0000","last_became_active":"2023-10-16T11:20:25.747574+0000","last_became_peered":"2023-10-16T11:20:25.747574+0000","last_unstale":"2023-10-16T11:20:26.802994+0000","last_undegraded":"2023-10-16T11:20:26.802994+0000","last_fullsized":"2023-10-16T11:20:26.802994+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:57.537856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736743+0000","last_change":"2023-10-16T11:20:24.728924+0000","last_active":"2023-10-16T11:20:25.736743+0000","last_peered":"2023-10-16T11:20:25.736743+0000","last_clean":"2023-10-16T11:20:25.736743+0000","last_became_active":"2023-10-16T11:20:24.728614+0000","last_became_peered":"2023-10-16T11:20:24.728614+0000","last_unstale":"2023-10-16T11:20:25.736743+0000","last_undegraded":"2023-10-16T11:20:25.736743+0000","last_fullsized":"2023-10-16T11:20:25.736743+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:23.737157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.801568+0000","last_change":"2023-10-16T11:20:25.751879+0000","last_active":"2023-10-16T11:20:26.801568+0000","last_peered":"2023-10-16T11:20:26.801568+0000","last_clean":"2023-10-16T11:20:26.801568+0000","last_became_active":"2023-10-16T11:20:25.750440+0000","last_became_peered":"2023-10-16T11:20:25.750440+0000","last_unstale":"2023-10-16T11:20:26.801568+0000","last_undegraded":"2023-10-16T11:20:26.801568+0000","last_fullsized":"2023-10-16T11:20:26.801568+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:57.379246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.736961+0000","last_change":"2023-10-16T11:20:24.732437+0000","last_active":"2023-10-16T11:20:25.736961+0000","last_peered":"2023-10-16T11:20:25.736961+0000","last_clean":"2023-10-16T11:20:25.736961+0000","last_became_active":"2023-10-16T11:20:24.732111+0000","last_became_peered":"2023-10-16T11:20:24.732111+0000","last_unstale":"2023-10-16T11:20:25.736961+0000","last_undegraded":"2023-10-16T11:20:25.736961+0000","last_fullsized":"2023-10-16T11:20:25.736961+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:26.846516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.746538+0000","last_change":"2023-10-16T11:20:25.750621+0000","last_active":"2023-10-16T11:20:26.746538+0000","last_peered":"2023-10-16T11:20:26.746538+0000","last_clean":"2023-10-16T11:20:26.746538+0000","last_became_active":"2023-10-16T11:20:25.750352+0000","last_became_peered":"2023-10-16T11:20:25.750352+0000","last_unstale":"2023-10-16T11:20:26.746538+0000","last_undegraded":"2023-10-16T11:20:26.746538+0000","last_fullsized":"2023-10-16T11:20:26.746538+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:57.295597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.731291+0000","last_change":"2023-10-16T11:20:24.733888+0000","last_active":"2023-10-16T11:20:25.731291+0000","last_peered":"2023-10-16T11:20:25.731291+0000","last_clean":"2023-10-16T11:20:25.731291+0000","last_became_active":"2023-10-16T11:20:24.733588+0000","last_became_peered":"2023-10-16T11:20:24.733588+0000","last_unstale":"2023-10-16T11:20:25.731291+0000","last_undegraded":"2023-10-16T11:20:25.731291+0000","last_fullsized":"2023-10-16T11:20:25.731291+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:06.768468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":237,"state":"active+clean","last_fresh":"2023-10-16T11:20:26.802430+0000","last_change":"2023-10-16T11:20:25.748051+0000","last_active":"2023-10-16T11:20:26.802430+0000","last_peered":"2023-10-16T11:20:26.802430+0000","last_clean":"2023-10-16T11:20:26.802430+0000","last_became_active":"2023-10-16T11:20:25.747647+0000","last_became_peered":"2023-10-16T11:20:25.747647+0000","last_unstale":"2023-10-16T11:20:26.802430+0000","last_undegraded":"2023-10-16T11:20:26.802430+0000","last_fullsized":"2023-10-16T11:20:26.802430+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:06.072095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"236'1","reported_seq":13,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.761389+0000","last_change":"2023-10-16T11:20:24.736168+0000","last_active":"2023-10-16T11:20:25.761389+0000","last_peered":"2023-10-16T11:20:25.761389+0000","last_clean":"2023-10-16T11:20:25.761389+0000","last_became_active":"2023-10-16T11:20:24.735886+0000","last_became_peered":"2023-10-16T11:20:24.735886+0000","last_unstale":"2023-10-16T11:20:25.761389+0000","last_undegraded":"2023-10-16T11:20:25.761389+0000","last_fullsized":"2023-10-16T11:20:25.761389+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"223'370","reported_seq":1306,"reported_epoch":236,"state":"active+clean","last_fresh":"2023-10-16T11:20:25.729253+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:20:25.729253+0000","last_peered":"2023-10-16T11:20:25.729253+0000","last_clean":"2023-10-16T11:20:25.729253+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:20:25.729253+0000","last_undegraded":"2023-10-16T11:20:25.729253+0000","last_fullsized":"2023-10-16T11:20:25.729253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576000,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52748,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319092,"statfs":{"total":96636764160,"available":96582750208,"internally_reserved":0,"allocated":3596288,"data_stored":2200288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.67200000000000004,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.372},"max":{"1min":9.4359999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.68400000000000005,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":9.3059999999999992,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.71599999999999997,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":9.3610000000000007,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.119,"5min":0.71999999999999997,"15min":0.68100000000000005},"min":{"1min":0.49399999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.3960000000000008,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.72099999999999997,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":9.2889999999999997,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.753,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.36099999999999999},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.157,"5min":0.76500000000000001,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":9.4689999999999994,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.71999999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.61199999999999999,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.747},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.63100000000000001,"15min":0.56699999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.5399999999999991,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.64000000000000001,"15min":0.57399999999999995},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.69399999999999995,"15min":0.63100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.5690000000000008,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.71199999999999997,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.155,"5min":0.69799999999999995,"15min":0.64500000000000002},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576000,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56328,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315512,"statfs":{"total":96636764160,"available":96579084288,"internally_reserved":0,"allocated":3002368,"data_stored":1609908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.91000000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.53500000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.70599999999999996,"5min":0.85899999999999999,"15min":0.88200000000000001},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64000000000000001,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.32300000000000001},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.625},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.335,"5min":3.335,"15min":3.335},"last":0.64300000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576001,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47992,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323848,"statfs":{"total":96636764160,"available":96587620352,"internally_reserved":0,"allocated":2994176,"data_stored":1609796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.622,"15min":0.66700000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.436,"15min":0.45200000000000001},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.52500000000000002,"5min":0.63900000000000001,"15min":0.71699999999999997},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.64000000000000001,"15min":0.70099999999999996},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.442},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":0.623,"15min":0.70499999999999996},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.48099999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65400000000000003,"15min":0.67200000000000004},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":0.89500000000000002,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90000000000000002,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.90300000000000002,"15min":1.0620000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.016},"last":0.57299999999999995}]}]},{"osd":0,"up_from":13,"seq":55834576001,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61068,"kb_used_data":3516,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310772,"statfs":{"total":96636764160,"available":96574230528,"internally_reserved":0,"allocated":3600384,"data_stored":2201776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.45000000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.26700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52800000000000002,"5min":0.66100000000000003,"15min":0.77200000000000002},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":1.272,"15min":1.272},"last":0.309},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40899999999999997,"15min":0.433},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.374},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54200000000000004,"5min":1.351,"15min":1.351},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83099999999999996,"5min":1.391,"15min":1.391},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76000000000000001,"5min":0.874,"15min":1.089},"last":0.62},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.624,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.95599999999999996,"15min":1.216},"last":0.496}]},{"osd":6,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.47199999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.1180000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.67200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80300000000000005,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80500000000000005,"5min":0.89600000000000002,"15min":1.0780000000000001},"last":0.59999999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576000,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48004,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323836,"statfs":{"total":96636764160,"available":96587608064,"internally_reserved":0,"allocated":3002368,"data_stored":1610448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.371,"15min":0.39500000000000002},"min":{"1min":0.253,"5min":0.188,"15min":0.188},"max":{"1min":0.51800000000000002,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.42799999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.193,"15min":0.193},"max":{"1min":0.53600000000000003,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.44400000000000001,"15min":0.46500000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59499999999999997,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39000000000000001,"15min":0.41599999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.45500000000000002,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.436}]},{"osd":3,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.442},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56699999999999995,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.499},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48799999999999999,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.95399999999999996,"15min":1.097},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64700000000000002,"15min":0.67300000000000004},"min":{"1min":0.438,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77100000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63700000000000001,"15min":0.66700000000000004},"min":{"1min":0.45500000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82499999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.69199999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81899999999999995,"5min":1.0169999999999999,"15min":1.0389999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64500000000000002,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76700000000000002,"5min":0.97799999999999998,"15min":1.022},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67700000000000005,"15min":0.70299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84999999999999998,"5min":0.98999999999999999,"15min":1.004},"last":0.79900000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.69199999999999995,"15min":0.70599999999999996},"min":{"1min":0.48599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81000000000000005,"5min":0.99299999999999999,"15min":1.071},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.67100000000000004,"15min":0.69799999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81200000000000006,"5min":0.97799999999999998,"15min":1.077},"last":0.78400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576000,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52176,"kb_used_data":2948,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319664,"statfs":{"total":96636764160,"available":96583335936,"internally_reserved":0,"allocated":3018752,"data_stored":1612046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.376,"15min":0.39500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38300000000000001,"15min":0.40699999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":1.0029999999999999},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42499999999999999,"15min":0.441},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46400000000000002,"15min":0.48299999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.50800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.024},"last":0.377}]},{"osd":4,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.68000000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.026},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.41799999999999998},"max":{"1min":5.8399999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.622},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69899999999999995,"15min":0.70499999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.4409999999999998,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.66600000000000004,"15min":0.69999999999999996},"min":{"1min":0.45300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77800000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.58199999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576000,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52164,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319676,"statfs":{"total":96636764160,"available":96583348224,"internally_reserved":0,"allocated":2998272,"data_stored":1609866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56999999999999995,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67700000000000005,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57599999999999996,"15min":0.61799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.61499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74099999999999999,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.61399999999999999},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.622},"min":{"1min":0.40400000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60799999999999998,"15min":0.63700000000000001},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.67000000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.61599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67900000000000005,"5min":0.83799999999999997,"15min":0.995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48199999999999998,"15min":0.5},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71499999999999997,"5min":0.73599999999999999,"15min":0.86199999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76300000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.749}]},{"osd":6,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.76200000000000001,"5min":0.878,"15min":1.0609999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.745,"5min":0.83099999999999996,"15min":1.115},"last":0.79500000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.83999999999999997,"5min":0.84499999999999997,"15min":0.93200000000000005},"last":0.77200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576000,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66048,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305792,"statfs":{"total":96636764160,"available":96569131008,"internally_reserved":0,"allocated":2994176,"data_stored":1609796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61199999999999999,"15min":0.63200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.28999999999999998},"max":{"1min":0.78000000000000003,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.62,"15min":0.65000000000000002},"min":{"1min":0.39700000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72999999999999998,"5min":0.871,"15min":0.88200000000000001},"last":0.67900000000000005}]},{"osd":1,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.57699999999999996,"15min":0.59499999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.752,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.501,"5min":0.56399999999999995,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.59099999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73399999999999999,"5min":0.80300000000000005,"15min":0.86499999999999999},"last":0.623},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58399999999999996,"15min":0.61699999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.752,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.61099999999999999},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57999999999999996,"15min":0.60399999999999998},"min":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.70299999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80900000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.33200000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.82499999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84499999999999997,"5min":0.878,"15min":0.878},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.57299999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78300000000000003,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:20:30.245 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:20:30.245 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T11:20:30.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.405+0000 7fa2c3545700 1 Processor -- start 2023-10-16T11:20:30.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.406+0000 7fa2c3545700 1 -- start start 2023-10-16T11:20:30.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.406+0000 7fa2c3545700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2bc15a780 0x7fa2bc0b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.406+0000 7fa2c3545700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc06db30 0x7fa2bc0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.406+0000 7fa2c3545700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc15e190 con 0x7fa2bc072b60 2023-10-16T11:20:30.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c3545700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2bc15e470 con 0x7fa2bc15a780 2023-10-16T11:20:30.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c3545700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc15e750 con 0x7fa2bc06db30 2023-10-16T11:20:30.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c12e1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2bc15a780 0x7fa2bc0b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c0ae0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa2bc072b60 0x7fa2bc06d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40522/0 (socket says 172.21.15.158:40522) 2023-10-16T11:20:30.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c12e1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2bc15a780 0x7fa2bc0b4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40344/0 (socket says 172.21.15.158:40344) 2023-10-16T11:20:30.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c0ae0700 1 -- 172.21.15.158:0/1807839880 learned_addr learned my addr 172.21.15.158:0/1807839880 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:30.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c1ae2700 1 --2- 172.21.15.158:0/1807839880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc06db30 0x7fa2bc0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c12e1700 1 -- 172.21.15.158:0/1807839880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc06db30 msgr2=0x7fa2bc0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:30.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c12e1700 1 --2- 172.21.15.158:0/1807839880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc06db30 0x7fa2bc0b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c12e1700 1 -- 172.21.15.158:0/1807839880 >> v1:172.21.15.158:6789/0 conn(0x7fa2bc072b60 legacy=0x7fa2bc06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c12e1700 1 -- 172.21.15.158:0/1807839880 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc15ea30 con 0x7fa2bc15a780 2023-10-16T11:20:30.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2c12e1700 1 --2- 172.21.15.158:0/1807839880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc15a780 0x7fa2bc0b4930 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00bae0 tx=0x7fa2ac00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42f4e866ade9e75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2b3fff700 1 -- 172.21.15.158:0/1807839880 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac00dd30 con 0x7fa2bc15a780 2023-10-16T11:20:30.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2b3fff700 1 -- 172.21.15.158:0/1807839880 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac00de90 con 0x7fa2bc15a780 2023-10-16T11:20:30.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.407+0000 7fa2b3fff700 1 -- 172.21.15.158:0/1807839880 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac010690 con 0x7fa2bc15a780 2023-10-16T11:20:30.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 -- 172.21.15.158:0/1807839880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc15a780 msgr2=0x7fa2bc0b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1807839880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc15a780 0x7fa2bc0b4930 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00bae0 tx=0x7fa2ac00bdf0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 -- 172.21.15.158:0/1807839880 shutdown_connections 2023-10-16T11:20:30.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1807839880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc15a780 0x7fa2bc0b4930 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1807839880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc06db30 0x7fa2bc0b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 -- 172.21.15.158:0/1807839880 >> 172.21.15.158:0/1807839880 conn(0x7fa2bc0ccd30 msgr2=0x7fa2bc0cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:30.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 -- 172.21.15.158:0/1807839880 shutdown_connections 2023-10-16T11:20:30.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.408+0000 7fa2c3545700 1 -- 172.21.15.158:0/1807839880 wait complete. 2023-10-16T11:20:30.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.409+0000 7fa2c3545700 1 Processor -- start 2023-10-16T11:20:30.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.409+0000 7fa2c3545700 1 -- start start 2023-10-16T11:20:30.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c3545700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc06db30 0x7fa2bc0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c3545700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc072b60 0x7fa2bc15d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c3545700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc15a780 0x7fa2bc0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c3545700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0c6d60 con 0x7fa2bc072b60 2023-10-16T11:20:30.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c3545700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0ce390 con 0x7fa2bc06db30 2023-10-16T11:20:30.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c3545700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0b1b60 con 0x7fa2bc15a780 2023-10-16T11:20:30.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc072b60 0x7fa2bc15d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc072b60 0x7fa2bc15d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40352/0 (socket says 172.21.15.158:40352) 2023-10-16T11:20:30.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 -- 172.21.15.158:0/1439566466 learned_addr learned my addr 172.21.15.158:0/1439566466 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:30.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c12e1700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc06db30 0x7fa2bc0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c1ae2700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc15a780 0x7fa2bc0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 -- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc15a780 msgr2=0x7fa2bc0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc15a780 0x7fa2bc0cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 -- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc06db30 msgr2=0x7fa2bc0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc06db30 0x7fa2bc0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 -- 172.21.15.158:0/1439566466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac00b740 con 0x7fa2bc072b60 2023-10-16T11:20:30.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c1ae2700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc15a780 0x7fa2bc0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:30.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c0ae0700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc072b60 0x7fa2bc15d450 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800c500 tx=0x7fa2b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2b1ffb700 1 -- 172.21.15.158:0/1439566466 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8005d70 con 0x7fa2bc072b60 2023-10-16T11:20:30.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2b1ffb700 1 -- 172.21.15.158:0/1439566466 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b80070d0 con 0x7fa2bc072b60 2023-10-16T11:20:30.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc0b1e40 con 0x7fa2bc072b60 2023-10-16T11:20:30.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.410+0000 7fa2b1ffb700 1 -- 172.21.15.158:0/1439566466 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8007440 con 0x7fa2bc072b60 2023-10-16T11:20:30.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.411+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc0b1830 con 0x7fa2bc072b60 2023-10-16T11:20:30.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.412+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc0b2360 con 0x7fa2bc072b60 2023-10-16T11:20:30.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.412+0000 7fa2b1ffb700 1 -- 172.21.15.158:0/1439566466 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa2b80075a0 con 0x7fa2bc072b60 2023-10-16T11:20:30.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.412+0000 7fa2b1ffb700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a006b140 0x7fa2a006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.412+0000 7fa2b1ffb700 1 -- 172.21.15.158:0/1439566466 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fa2b8089110 con 0x7fa2bc072b60 2023-10-16T11:20:30.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.412+0000 7fa2c12e1700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a006b140 0x7fa2a006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.413+0000 7fa2c12e1700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a006b140 0x7fa2a006d600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00ba30 tx=0x7fa2ac013000 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.417+0000 7fa2b1ffb700 1 -- 172.21.15.158:0/1439566466 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa2bc0b2360 con 0x7fa2bc072b60 2023-10-16T11:20:30.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.687+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa2bc0b2360 con 0x7fa2bc072b60 2023-10-16T11:20:30.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.689+0000 7fa2b1ffb700 1 -- 172.21.15.158:0/1439566466 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v55) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa2bc0b2360 con 0x7fa2bc072b60 2023-10-16T11:20:30.691 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T11:20:30.691 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T11:20:30.691 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T11:20:30.691 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T11:20:30.692 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T11:20:30.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.691+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a006b140 msgr2=0x7fa2a006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.691+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a006b140 0x7fa2a006d600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00ba30 tx=0x7fa2ac013000 comp rx=0 tx=0).stop 2023-10-16T11:20:30.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.691+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc072b60 msgr2=0x7fa2bc15d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.691+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc072b60 0x7fa2bc15d450 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800c500 tx=0x7fa2b800c8c0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 shutdown_connections 2023-10-16T11:20:30.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa2a006b140 0x7fa2a006d600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc15a780 0x7fa2bc0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc06db30 0x7fa2bc0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 --2- 172.21.15.158:0/1439566466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2bc072b60 0x7fa2bc15d450 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 >> 172.21.15.158:0/1439566466 conn(0x7fa2bc0ccd30 msgr2=0x7fa2bc06d420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:30.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 shutdown_connections 2023-10-16T11:20:30.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.692+0000 7fa2c3545700 1 -- 172.21.15.158:0/1439566466 wait complete. 2023-10-16T11:20:30.696 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T11:20:30.705 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:30.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.826+0000 7f254e882700 1 Processor -- start 2023-10-16T11:20:30.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.826+0000 7f254e882700 1 -- start start 2023-10-16T11:20:30.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f254e882700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25480cc0c0 0x7f25480aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f254e882700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f254806bfb0 0x7f25480cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f254e882700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254815b810 con 0x7f25480cf2f0 2023-10-16T11:20:30.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f254e882700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25480cf780 con 0x7f254806bfb0 2023-10-16T11:20:30.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f254e882700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254815a990 con 0x7f25480cc0c0 2023-10-16T11:20:30.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f25477fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f254806bfb0 0x7f25480cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f25477fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f254806bfb0 0x7f25480cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40358/0 (socket says 172.21.15.158:40358) 2023-10-16T11:20:30.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f25477fe700 1 -- 172.21.15.158:0/3226626264 learned_addr learned my addr 172.21.15.158:0/3226626264 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:30.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f254ce1f700 1 --1- 172.21.15.158:0/3226626264 >> v1:172.21.15.158:6789/0 conn(0x7f25480cf2f0 0x7f254815dfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40538/0 (socket says 172.21.15.158:40538) 2023-10-16T11:20:30.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.827+0000 7f2547fff700 1 --2- 172.21.15.158:0/3226626264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25480cc0c0 0x7f25480aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f25477fe700 1 -- 172.21.15.158:0/3226626264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25480cc0c0 msgr2=0x7f25480aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:30.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f25477fe700 1 --2- 172.21.15.158:0/3226626264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25480cc0c0 0x7f25480aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f25477fe700 1 -- 172.21.15.158:0/3226626264 >> v1:172.21.15.158:6789/0 conn(0x7f25480cf2f0 legacy=0x7f254815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f2546ffd700 1 -- 172.21.15.158:0/3226626264 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270210789 0 0) 0x7f254815b810 con 0x7f25480cf2f0 2023-10-16T11:20:30.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f25477fe700 1 -- 172.21.15.158:0/3226626264 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254815ac00 con 0x7f254806bfb0 2023-10-16T11:20:30.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f25477fe700 1 --2- 172.21.15.158:0/3226626264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f25480cedb0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f25380098f0 tx=0x7f2538009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f0bacf132f9b98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f2546ffd700 1 -- 172.21.15.158:0/3226626264 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800b8a0 con 0x7f254806bfb0 2023-10-16T11:20:30.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f2546ffd700 1 -- 172.21.15.158:0/3226626264 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253800ba00 con 0x7f254806bfb0 2023-10-16T11:20:30.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f2546ffd700 1 -- 172.21.15.158:0/3226626264 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800bd90 con 0x7f254806bfb0 2023-10-16T11:20:30.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 -- 172.21.15.158:0/3226626264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 msgr2=0x7f25480cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 --2- 172.21.15.158:0/3226626264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f25480cedb0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f25380098f0 tx=0x7f2538009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:30.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 -- 172.21.15.158:0/3226626264 shutdown_connections 2023-10-16T11:20:30.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 --2- 172.21.15.158:0/3226626264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25480cc0c0 0x7f25480aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 --2- 172.21.15.158:0/3226626264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f25480cedb0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 -- 172.21.15.158:0/3226626264 >> 172.21.15.158:0/3226626264 conn(0x7f25480cc4b0 msgr2=0x7f25480cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:30.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 -- 172.21.15.158:0/3226626264 shutdown_connections 2023-10-16T11:20:30.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.828+0000 7f254e882700 1 -- 172.21.15.158:0/3226626264 wait complete. 2023-10-16T11:20:30.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.830+0000 7f254e882700 1 Processor -- start 2023-10-16T11:20:30.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.830+0000 7f254e882700 1 -- start start 2023-10-16T11:20:30.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.830+0000 7f254e882700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f254813fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.830+0000 7f254e882700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25480cf2f0 0x7f254815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.830+0000 7f254e882700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2548140320 0x7f254815c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f254e882700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2548141440 con 0x7f254806bfb0 2023-10-16T11:20:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f254e882700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2548140830 con 0x7f25480cf2f0 2023-10-16T11:20:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f254e882700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2548140b10 con 0x7f2548140320 2023-10-16T11:20:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f254813fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f254813fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40364/0 (socket says 172.21.15.158:40364) 2023-10-16T11:20:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 -- 172.21.15.158:0/1600761115 learned_addr learned my addr 172.21.15.158:0/1600761115 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f254ce1f700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2548140320 0x7f254815c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f25477fe700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25480cf2f0 0x7f254815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 -- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2548140320 msgr2=0x7f254815c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2548140320 0x7f254815c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 -- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25480cf2f0 msgr2=0x7f254815c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25480cf2f0 0x7f254815c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 -- 172.21.15.158:0/1600761115 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2538009550 con 0x7f254806bfb0 2023-10-16T11:20:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2547fff700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f254813fde0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f253c00e810 tx=0x7f253c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2544ff9700 1 -- 172.21.15.158:0/1600761115 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c013610 con 0x7f254806bfb0 2023-10-16T11:20:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2544ff9700 1 -- 172.21.15.158:0/1600761115 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c013770 con 0x7f254806bfb0 2023-10-16T11:20:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2548140df0 con 0x7f254806bfb0 2023-10-16T11:20:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f2544ff9700 1 -- 172.21.15.158:0/1600761115 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c013b20 con 0x7f254806bfb0 2023-10-16T11:20:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.831+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25480c97d0 con 0x7f254806bfb0 2023-10-16T11:20:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.832+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25480681b0 con 0x7f254806bfb0 2023-10-16T11:20:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.833+0000 7f2544ff9700 1 -- 172.21.15.158:0/1600761115 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f253c013c80 con 0x7f254806bfb0 2023-10-16T11:20:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.833+0000 7f2544ff9700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f253406b140 0x7f253406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.833+0000 7f2544ff9700 1 -- 172.21.15.158:0/1600761115 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f253c089b20 con 0x7f254806bfb0 2023-10-16T11:20:30.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.833+0000 7f25477fe700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f253406b140 0x7f253406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:30.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.834+0000 7f25477fe700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f253406b140 0x7f253406d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25480c9410 tx=0x7f2538008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:30.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:30.838+0000 7f2544ff9700 1 -- 172.21.15.158:0/1600761115 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f253c054450 con 0x7f254806bfb0 2023-10-16T11:20:31.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.074+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f254806bc90 con 0x7f254806bfb0 2023-10-16T11:20:31.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.075+0000 7f2544ff9700 1 -- 172.21.15.158:0/1600761115 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f253c057a70 con 0x7f254806bfb0 2023-10-16T11:20:31.077 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:31.077 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":12432,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T11:20:31.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f253406b140 msgr2=0x7f253406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f253406b140 0x7f253406d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25480c9410 tx=0x7f2538008030 comp rx=0 tx=0).stop 2023-10-16T11:20:31.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 msgr2=0x7f254813fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f254813fde0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f253c00e810 tx=0x7f253c00ebd0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 shutdown_connections 2023-10-16T11:20:31.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f253406b140 0x7f253406d600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2548140320 0x7f254815c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f254806bfb0 0x7f254813fde0 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 --2- 172.21.15.158:0/1600761115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25480cf2f0 0x7f254815c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 >> 172.21.15.158:0/1600761115 conn(0x7f25480cc4b0 msgr2=0x7f254806f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 shutdown_connections 2023-10-16T11:20:31.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.077+0000 7f254e882700 1 -- 172.21.15.158:0/1600761115 wait complete. 2023-10-16T11:20:31.081 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 129 2023-10-16T11:20:31.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:20:25.120788+0000', 'modified': '2023-10-16T11:20:28.157424+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 39383}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_39383': {'gid': 39383, 'name': 'd', 'rank': 0, 'incarnation': 125, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6834/468116497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6832', 'nonce': 468116497}, {'type': 'v1', 'addr': '172.21.15.172:6834', 'nonce': 468116497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:20:31.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:20:31.494 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T11:20:31.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.615+0000 7f3e370a6700 1 Processor -- start 2023-10-16T11:20:31.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.615+0000 7f3e370a6700 1 -- start start 2023-10-16T11:20:31.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e370a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 0x7f3e300cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e370a6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3e3015ed20 0x7f3e300cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e370a6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e30161200 con 0x7f3e300ce3f0 2023-10-16T11:20:31.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e370a6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e300ce7d0 con 0x7f3e3015ed20 2023-10-16T11:20:31.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e370a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e30162060 con 0x7f3e3015e8e0 2023-10-16T11:20:31.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e34e42700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3e300ce3f0 0x7f3e3006d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40548/0 (socket says 172.21.15.158:40548) 2023-10-16T11:20:31.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e34e42700 1 -- 172.21.15.158:0/3957450871 learned_addr learned my addr 172.21.15.158:0/3957450871 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:31.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e35643700 1 --2- 172.21.15.158:0/3957450871 >> v2:172.21.15.172:3300/0 conn(0x7f3e3015ed20 0x7f3e300cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949994082 0 0) 0x7f3e30161200 con 0x7f3e300ce3f0 2023-10-16T11:20:31.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e200035b0 con 0x7f3e300ce3f0 2023-10-16T11:20:31.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 0x7f3e300cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789069879 0 0) 0x7f3e200035b0 con 0x7f3e300ce3f0 2023-10-16T11:20:31.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 msgr2=0x7f3e300cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 0x7f3e300cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 >> v2:172.21.15.172:3300/0 conn(0x7f3e3015ed20 msgr2=0x7f3e300cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 --2- 172.21.15.158:0/3957450871 >> v2:172.21.15.172:3300/0 conn(0x7f3e3015ed20 0x7f3e300cf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e30161450 con 0x7f3e300ce3f0 2023-10-16T11:20:31.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3e24001670 con 0x7f3e300ce3f0 2023-10-16T11:20:31.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 >> v1:172.21.15.158:6789/0 conn(0x7f3e300ce3f0 legacy=0x7f3e3006d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e20003a00 0x7f3e20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e20004850 0x7f3e20004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.616+0000 7f3e2f7fe700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 0x7f3e20005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300ce7d0 con 0x7f3e20003a00 2023-10-16T11:20:31.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e30162060 con 0x7f3e20004850 2023-10-16T11:20:31.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2000ba10 con 0x7f3e3015e8e0 2023-10-16T11:20:31.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e20003a00 0x7f3e20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e20004850 0x7f3e20004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e35643700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 0x7f3e20005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e34e42700 1 -- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 msgr2=0x7f3e20005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e3015e8e0 0x7f3e20005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e34e42700 1 -- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e20004850 msgr2=0x7f3e20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e20004850 0x7f3e20004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e34e42700 1 -- 172.21.15.158:0/3957450871 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e200035b0 con 0x7f3e20003a00 2023-10-16T11:20:31.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e20004850 0x7f3e20004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:31.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e20003a00 0x7f3e20003e20 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c370 tx=0x7f3e20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12f611b16663974 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24007c60 con 0x7f3e20003a00 2023-10-16T11:20:31.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e24005960 con 0x7f3e20003a00 2023-10-16T11:20:31.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.617+0000 7f3e2f7fe700 1 -- 172.21.15.158:0/3957450871 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24005d10 con 0x7f3e20003a00 2023-10-16T11:20:31.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.618+0000 7f3e370a6700 1 -- 172.21.15.158:0/3957450871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e20003a00 msgr2=0x7f3e20003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.618+0000 7f3e370a6700 1 --2- 172.21.15.158:0/3957450871 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e20003a00 0x7f3e20003e20 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c370 tx=0x7f3e20005880 comp rx=0 tx=0).stop 2023-10-16T11:20:31.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.618+0000 7f3e35643700 1 -- 172.21.15.158:0/3957450871 reap_dead start 2023-10-16T11:20:31.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.618+0000 7f3e370a6700 1 -- 172.21.15.158:0/3957450871 shutdown_connections 2023-10-16T11:20:31.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.618+0000 7f3e370a6700 1 -- 172.21.15.158:0/3957450871 >> 172.21.15.158:0/3957450871 conn(0x7f3e3006a8d0 msgr2=0x7f3e3006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.618+0000 7f3e370a6700 1 -- 172.21.15.158:0/3957450871 shutdown_connections 2023-10-16T11:20:31.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.618+0000 7f3e370a6700 1 -- 172.21.15.158:0/3957450871 wait complete. 2023-10-16T11:20:31.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.619+0000 7f3e370a6700 1 Processor -- start 2023-10-16T11:20:31.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.619+0000 7f3e370a6700 1 -- start start 2023-10-16T11:20:31.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e370a6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e20003a00 0x7f3e30154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e370a6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e30155310 0x7f3e300b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e370a6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e301e9660 0x7f3e301e9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e370a6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300b6550 con 0x7f3e301e9660 2023-10-16T11:20:31.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e370a6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e301e91d0 con 0x7f3e30155310 2023-10-16T11:20:31.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e370a6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300b6f80 con 0x7f3e20003a00 2023-10-16T11:20:31.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e301e9660 0x7f3e301e9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e301e9660 0x7f3e301e9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40388/0 (socket says 172.21.15.158:40388) 2023-10-16T11:20:31.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 -- 172.21.15.158:0/255314431 learned_addr learned my addr 172.21.15.158:0/255314431 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:31.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e30155310 0x7f3e300b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e34e42700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e20003a00 0x7f3e30154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 -- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e20003a00 msgr2=0x7f3e30154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e20003a00 0x7f3e30154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 -- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e30155310 msgr2=0x7f3e300b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e30155310 0x7f3e300b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 -- 172.21.15.158:0/255314431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e301e85c0 con 0x7f3e301e9660 2023-10-16T11:20:31.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e35643700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e301e9660 0x7f3e301e9ae0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f3e3015b770 tx=0x7f3e28011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.620+0000 7f3e2d7fa700 1 -- 172.21.15.158:0/255314431 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800e040 con 0x7f3e301e9660 2023-10-16T11:20:31.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.621+0000 7f3e2d7fa700 1 -- 172.21.15.158:0/255314431 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2800f040 con 0x7f3e301e9660 2023-10-16T11:20:31.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.621+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e301e88a0 con 0x7f3e301e9660 2023-10-16T11:20:31.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.621+0000 7f3e2d7fa700 1 -- 172.21.15.158:0/255314431 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800fcc0 con 0x7f3e301e9660 2023-10-16T11:20:31.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.621+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e301e8e60 con 0x7f3e301e9660 2023-10-16T11:20:31.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.622+0000 7f3e2d7fa700 1 -- 172.21.15.158:0/255314431 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3e28022460 con 0x7f3e301e9660 2023-10-16T11:20:31.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.622+0000 7f3e2d7fa700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e1406b140 0x7f3e1406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.622+0000 7f3e2d7fa700 1 -- 172.21.15.158:0/255314431 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f3e2808a150 con 0x7f3e301e9660 2023-10-16T11:20:31.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.622+0000 7f3e34e42700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e1406b140 0x7f3e1406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.623+0000 7f3e34e42700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e1406b140 0x7f3e1406d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e24015040 tx=0x7f3e2400a000 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.624+0000 7f3e43290b80 1 Processor -- start 2023-10-16T11:20:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.624+0000 7f3e43290b80 1 -- start start 2023-10-16T11:20:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.625+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b00706ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.625+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 0x55b00706cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.625+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070406c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.625+0000 7f3e43290b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b006fb6c20 con 0x55b007040010 2023-10-16T11:20:31.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.625+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b006f9bc10 con 0x55b007065220 2023-10-16T11:20:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.625+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b006f99530 con 0x55b00706a630 2023-10-16T11:20:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.625+0000 7f3e35643700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070406c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40396/0 (socket says 172.21.15.158:40396) 2023-10-16T11:20:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 -- 172.21.15.158:0/1655991443 learned_addr learned my addr 172.21.15.158:0/1655991443 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:31.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e34e42700 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b00706ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 -- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 msgr2=0x55b00706cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 0x55b00706cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 -- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 msgr2=0x55b00706ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b00706ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 -- 172.21.15.158:0/1655991443 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b006f523a0 con 0x55b007040010 2023-10-16T11:20:31.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e34e42700 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b00706ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:31.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e35643700 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070406c0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800f540 tx=0x7f3e28058100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c19c35f0e930d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e1a7fc700 1 -- 172.21.15.158:0/1655991443 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28056a30 con 0x55b007040010 2023-10-16T11:20:31.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e1a7fc700 1 -- 172.21.15.158:0/1655991443 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e280588f0 con 0x55b007040010 2023-10-16T11:20:31.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.626+0000 7f3e1a7fc700 1 -- 172.21.15.158:0/1655991443 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28057b80 con 0x55b007040010 2023-10-16T11:20:31.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 -- 172.21.15.158:0/1655991443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 msgr2=0x55b0070406c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070406c0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800f540 tx=0x7f3e28058100 comp rx=0 tx=0).stop 2023-10-16T11:20:31.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 -- 172.21.15.158:0/1655991443 shutdown_connections 2023-10-16T11:20:31.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070406c0 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b00706ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1655991443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 0x55b00706cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 -- 172.21.15.158:0/1655991443 >> 172.21.15.158:0/1655991443 conn(0x55b006f6fd80 msgr2=0x55b00708aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 -- 172.21.15.158:0/1655991443 shutdown_connections 2023-10-16T11:20:31.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 -- 172.21.15.158:0/1655991443 wait complete. 2023-10-16T11:20:31.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.627+0000 7f3e43290b80 1 -- start start 2023-10-16T11:20:31.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 1 Processor -- start 2023-10-16T11:20:31.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070644d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b007068d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 0x55b00706fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b006f94e20 con 0x55b007040010 2023-10-16T11:20:31.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b006f316c0 con 0x55b007065220 2023-10-16T11:20:31.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b006f31360 con 0x55b00706a630 2023-10-16T11:20:31.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.628+0000 7f3e43290b80 10 client.0 _ll_register_callbacks cb 0x55b006d8e7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:20:31.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070644d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40398/0 (socket says 172.21.15.158:40398) 2023-10-16T11:20:31.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 -- 172.21.15.158:0/3282588665 learned_addr learned my addr 172.21.15.158:0/3282588665 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:31.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b007068d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e35643700 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 0x55b00706fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 -- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 msgr2=0x55b00706fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 0x55b00706fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 -- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 msgr2=0x55b007068d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b007068d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 -- 172.21.15.158:0/3282588665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e301e8e60 con 0x55b007040010 2023-10-16T11:20:31.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070644d0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f3e200046a0 tx=0x7f3e24001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f56aea68a6f6a14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e0f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:20:31.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3282588665 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004c80 con 0x55b007040010 2023-10-16T11:20:31.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3282588665 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f3e24004de0 con 0x55b007040010 2023-10-16T11:20:31.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0070355b0 con 0x55b007040010 2023-10-16T11:20:31.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3282588665 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e240016a0 con 0x55b007040010 2023-10-16T11:20:31.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e43290b80 10 client.12573 fetch_fsmap learned FSMap version 129 2023-10-16T11:20:31.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x55b006dbfd90 con 0x55b007040010 2023-10-16T11:20:31.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.629+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3282588665 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b0070355b0 con 0x55b007040010 2023-10-16T11:20:31.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3282588665 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 129) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3e24007070 con 0x55b007040010 2023-10-16T11:20:31.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e43290b80 10 client.12573 fetch_fsmap finished waiting for FSMap version 129 2023-10-16T11:20:31.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e43290b80 10 client.12573 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:20:31.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e43290b80 20 client.12573 populate_metadata read hostname 'smithi158' 2023-10-16T11:20:31.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b00709a8f0 0x55b00709acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e43290b80 4 client.12573 mds_command: new command op to 39383 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:20:31.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 --> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b006f8de00 con 0x55b00709a8f0 2023-10-16T11:20:31.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b00709a8f0 0x55b00709acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e0f7fe700 10 client.12573 ms_handle_connect on v2:172.21.15.172:6832/468116497 2023-10-16T11:20:31.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.630+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b00709a8f0 0x55b00709acd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1338f32fb30eea3 server_cookie=4d509e2d1a4015e9 in_seq=0 out_seq=0 2023-10-16T11:20:31.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3282588665 <== mds.0 v2:172.21.15.172:6832/468116497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b006f8de00 con 0x55b00709a8f0 2023-10-16T11:20:31.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e0f7fe700 10 client.12573 handle_command_reply: tid=0 2023-10-16T11:20:31.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 client.12573 shutdown 2023-10-16T11:20:31.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 msgr2=0x55b0070644d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070644d0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f3e200046a0 tx=0x7f3e24001a00 comp rx=0 tx=0).stop 2023-10-16T11:20:31.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 shutdown_connections 2023-10-16T11:20:31.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b00709a8f0 0x55b00709acd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b00706a630 0x55b00706fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b0070644d0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e0f7fe700 0 client.12573 ms_handle_reset on v2:172.21.15.172:6832/468116497 2023-10-16T11:20:31.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3282588665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b007065220 0x55b007068d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 >> 172.21.15.158:0/3282588665 conn(0x55b006f6fd80 msgr2=0x55b0070419f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 shutdown_connections 2023-10-16T11:20:31.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.631+0000 7f3e43290b80 1 -- 172.21.15.158:0/3282588665 wait complete. 2023-10-16T11:20:31.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.632+0000 7f3e43290b80 20 client.12573 trim_cache size 0 max 16384 2023-10-16T11:20:31.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.649+0000 7f3e43290b80 1 Processor -- start 2023-10-16T11:20:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.649+0000 7f3e43290b80 1 -- start start 2023-10-16T11:20:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007040010 0x55b0070698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007065220 0x55b007088220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e43290b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b006e158d0 con 0x55b007065220 2023-10-16T11:20:31.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b007039630 con 0x55b00706a630 2023-10-16T11:20:31.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b00701f4c0 con 0x55b007040010 2023-10-16T11:20:31.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007065220 0x55b007088220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007065220 0x55b007088220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40408/0 (socket says 172.21.15.158:40408) 2023-10-16T11:20:31.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 -- 172.21.15.158:0/1110268714 learned_addr learned my addr 172.21.15.158:0/1110268714 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:31.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 -- 172.21.15.158:0/1110268714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007040010 msgr2=0x55b0070698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e35643700 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007040010 0x55b0070698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 -- 172.21.15.158:0/1110268714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 msgr2=0x55b007088760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 -- 172.21.15.158:0/1110268714 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b006da8c10 con 0x55b007065220 2023-10-16T11:20:31.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.650+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007065220 0x55b007088220 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0097b0 tx=0x7f3e1c009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccf34d43bf123da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e1affd700 1 -- 172.21.15.158:0/1110268714 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c01a040 con 0x55b007065220 2023-10-16T11:20:31.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e1affd700 1 -- 172.21.15.158:0/1110268714 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c01c040 con 0x55b007065220 2023-10-16T11:20:31.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e1affd700 1 -- 172.21.15.158:0/1110268714 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c021430 con 0x55b007065220 2023-10-16T11:20:31.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 -- 172.21.15.158:0/1110268714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007065220 msgr2=0x55b007088220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007065220 0x55b007088220 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0097b0 tx=0x7f3e1c009d70 comp rx=0 tx=0).stop 2023-10-16T11:20:31.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 -- 172.21.15.158:0/1110268714 shutdown_connections 2023-10-16T11:20:31.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007040010 0x55b0070698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 --2- 172.21.15.158:0/1110268714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007065220 0x55b007088220 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 -- 172.21.15.158:0/1110268714 >> 172.21.15.158:0/1110268714 conn(0x55b006f6fd80 msgr2=0x55b0070a8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 -- 172.21.15.158:0/1110268714 shutdown_connections 2023-10-16T11:20:31.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.651+0000 7f3e43290b80 1 -- 172.21.15.158:0/1110268714 wait complete. 2023-10-16T11:20:31.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.652+0000 7f3e43290b80 1 -- start start 2023-10-16T11:20:31.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.652+0000 7f3e43290b80 1 Processor -- start 2023-10-16T11:20:31.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b007099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007065220 0x55b0070699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b007042690 con 0x55b007040010 2023-10-16T11:20:31.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0070eec40 con 0x55b00706a630 2023-10-16T11:20:31.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b007041aa0 con 0x55b007065220 2023-10-16T11:20:31.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 10 client.0 _ll_register_callbacks cb 0x7f3e200045d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:20:31.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b007099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b007099a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40412/0 (socket says 172.21.15.158:40412) 2023-10-16T11:20:31.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 -- 172.21.15.158:0/3981057918 learned_addr learned my addr 172.21.15.158:0/3981057918 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:31.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007065220 0x55b0070699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 -- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007065220 msgr2=0x55b0070699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007065220 0x55b0070699a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e35643700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 -- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 msgr2=0x55b007088200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:31.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 -- 172.21.15.158:0/3981057918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1c009a70 con 0x55b007040010 2023-10-16T11:20:31.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e35643700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:31.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e34e42700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b007099a70 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f3e240075b0 tx=0x7f3e24007980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cb4a9771f0da71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e0f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:20:31.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3981057918 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004c80 con 0x55b007040010 2023-10-16T11:20:31.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3981057918 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f3e24004de0 con 0x55b007040010 2023-10-16T11:20:31.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.653+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e280588f0 con 0x55b007040010 2023-10-16T11:20:31.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3981057918 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e24001740 con 0x55b007040010 2023-10-16T11:20:31.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 10 client.12579 fetch_fsmap learned FSMap version 129 2023-10-16T11:20:31.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x55b006de6200 con 0x55b007040010 2023-10-16T11:20:31.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3981057918 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e280588f0 con 0x55b007040010 2023-10-16T11:20:31.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3981057918 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 129) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3e24005040 con 0x55b007040010 2023-10-16T11:20:31.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 10 client.12579 fetch_fsmap finished waiting for FSMap version 129 2023-10-16T11:20:31.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 10 client.12579 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:20:31.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 20 client.12579 populate_metadata read hostname 'smithi158' 2023-10-16T11:20:31.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b007085530 0x55b007088740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 4 client.12579 mds_command: new command op to 39383 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:20:31.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 --> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b006f8db20 con 0x55b007085530 2023-10-16T11:20:31.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b007085530 0x55b007088740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e0f7fe700 10 client.12579 ms_handle_connect on v2:172.21.15.172:6832/468116497 2023-10-16T11:20:31.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.654+0000 7f3e2ffff700 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b007085530 0x55b007088740 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5505e4619793467e server_cookie=21a212232821ccea in_seq=0 out_seq=0 2023-10-16T11:20:31.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e0f7fe700 1 -- 172.21.15.158:0/3981057918 <== mds.0 v2:172.21.15.172:6832/468116497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x55b006f8db20 con 0x55b007085530 2023-10-16T11:20:31.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e0f7fe700 10 client.12579 handle_command_reply: tid=0 2023-10-16T11:20:31.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 client.12579 shutdown 2023-10-16T11:20:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 msgr2=0x55b007099a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b007099a70 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f3e240075b0 tx=0x7f3e24007980 comp rx=0 tx=0).stop 2023-10-16T11:20:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 shutdown_connections 2023-10-16T11:20:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:6832/468116497,v1:172.21.15.172:6834/468116497] conn(0x55b007085530 0x55b007088740 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b00706a630 0x55b007088200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b007040010 0x55b007099a70 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e0f7fe700 0 client.12579 ms_handle_reset on v2:172.21.15.172:6832/468116497 2023-10-16T11:20:31.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 --2- 172.21.15.158:0/3981057918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b007065220 0x55b0070699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 >> 172.21.15.158:0/3981057918 conn(0x55b006f6fd80 msgr2=0x55b007067a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 shutdown_connections 2023-10-16T11:20:31.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 1 -- 172.21.15.158:0/3981057918 wait complete. 2023-10-16T11:20:31.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e43290b80 20 client.12579 trim_cache size 0 max 16384 2023-10-16T11:20:31.682 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:20:31.682 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:20:31.682 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:20:31.683 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 39383, 2023-10-16T11:20:31.683 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:20:31.683 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:20:31.683 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T11:20:31.683 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9142987170000003, 2023-10-16T11:20:31.684 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 129, 2023-10-16T11:20:31.684 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 237, 2023-10-16T11:20:31.684 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 237, 2023-10-16T11:20:31.684 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.844321372 2023-10-16T11:20:31.684 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:20:31.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e1406b140 msgr2=0x7f3e1406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e370a6700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e1406b140 0x7f3e1406d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e24015040 tx=0x7f3e2400a000 comp rx=0 tx=0).stop 2023-10-16T11:20:31.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e301e9660 msgr2=0x7f3e301e9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.655+0000 7f3e370a6700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e301e9660 0x7f3e301e9ae0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f3e3015b770 tx=0x7f3e28011f80 comp rx=0 tx=0).stop 2023-10-16T11:20:31.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 shutdown_connections 2023-10-16T11:20:31.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e1406b140 0x7f3e1406d600 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e301e9660 0x7f3e301e9ae0 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e20003a00 0x7f3e30154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 --2- 172.21.15.158:0/255314431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e30155310 0x7f3e300b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 >> 172.21.15.158:0/255314431 conn(0x7f3e3006a8d0 msgr2=0x7f3e301562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 shutdown_connections 2023-10-16T11:20:31.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.656+0000 7f3e370a6700 1 -- 172.21.15.158:0/255314431 wait complete. 2023-10-16T11:20:31.689 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T11:20:31.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.810+0000 7f04bbe14700 1 Processor -- start 2023-10-16T11:20:31.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.810+0000 7f04bbe14700 1 -- start start 2023-10-16T11:20:31.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04bbe14700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f04b40729b0 0x7f04b406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04bbe14700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b4161270 con 0x7f04b406ab40 2023-10-16T11:20:31.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04bbe14700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b4161510 con 0x7f04b406e5d0 2023-10-16T11:20:31.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04bbe14700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b41617f0 con 0x7f04b40729b0 2023-10-16T11:20:31.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04ba3b1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f04b40729b0 0x7f04b406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04ba3b1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f04b40729b0 0x7f04b406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40424/0 (socket says 172.21.15.158:40424) 2023-10-16T11:20:31.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04ba3b1700 1 -- 172.21.15.158:0/3979368584 learned_addr learned my addr 172.21.15.158:0/3979368584 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:31.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04b9bb0700 1 --1- 172.21.15.158:0/3979368584 >> v1:172.21.15.158:6789/0 conn(0x7f04b406ab40 0x7f04b40722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40596/0 (socket says 172.21.15.158:40596) 2023-10-16T11:20:31.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04ba3b1700 1 -- 172.21.15.158:0/3979368584 >> v1:172.21.15.172:6789/0 conn(0x7f04b406e5d0 legacy=0x7f04b415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04ba3b1700 1 -- 172.21.15.158:0/3979368584 >> v1:172.21.15.158:6789/0 conn(0x7f04b406ab40 legacy=0x7f04b40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04ba3b1700 1 -- 172.21.15.158:0/3979368584 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4161ad0 con 0x7f04b40729b0 2023-10-16T11:20:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04b8bae700 1 -- 172.21.15.158:0/3979368584 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415887810 0 0) 0x7f04b4161270 con 0x7f04b406ab40 2023-10-16T11:20:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04ba3b1700 1 --2- 172.21.15.158:0/3979368584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b40729b0 0x7f04b406d820 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009e50 tx=0x7f04ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0778546cbd6aec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04b8bae700 1 -- 172.21.15.158:0/3979368584 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00e070 con 0x7f04b40729b0 2023-10-16T11:20:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04b8bae700 1 -- 172.21.15.158:0/3979368584 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac00b9c0 con 0x7f04b40729b0 2023-10-16T11:20:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.811+0000 7f04b8bae700 1 -- 172.21.15.158:0/3979368584 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00bd70 con 0x7f04b40729b0 2023-10-16T11:20:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.812+0000 7f04bbe14700 1 -- 172.21.15.158:0/3979368584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b40729b0 msgr2=0x7f04b406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.812+0000 7f04bbe14700 1 --2- 172.21.15.158:0/3979368584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b40729b0 0x7f04b406d820 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009e50 tx=0x7f04ac00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.812+0000 7f04bbe14700 1 -- 172.21.15.158:0/3979368584 shutdown_connections 2023-10-16T11:20:31.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.812+0000 7f04bbe14700 1 --2- 172.21.15.158:0/3979368584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b40729b0 0x7f04b406d820 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.812+0000 7f04bbe14700 1 -- 172.21.15.158:0/3979368584 >> 172.21.15.158:0/3979368584 conn(0x7f04b40d02b0 msgr2=0x7f04b40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.812+0000 7f04bbe14700 1 -- 172.21.15.158:0/3979368584 shutdown_connections 2023-10-16T11:20:31.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.812+0000 7f04bbe14700 1 -- 172.21.15.158:0/3979368584 wait complete. 2023-10-16T11:20:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.813+0000 7f04bbe14700 1 Processor -- start 2023-10-16T11:20:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.813+0000 7f04bbe14700 1 -- start start 2023-10-16T11:20:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04bbe14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04b406ab40 0x7f04b40c3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04bbe14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b406e5d0 0x7f04b40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04bbe14700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40729b0 0x7f04b40c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04bbe14700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b41f6380 con 0x7f04b406e5d0 2023-10-16T11:20:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04bbe14700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b41f6620 con 0x7f04b40729b0 2023-10-16T11:20:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04bbe14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b41f6900 con 0x7f04b406ab40 2023-10-16T11:20:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b406e5d0 0x7f04b40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b406e5d0 0x7f04b40c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40428/0 (socket says 172.21.15.158:40428) 2023-10-16T11:20:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 -- 172.21.15.158:0/1052395686 learned_addr learned my addr 172.21.15.158:0/1052395686 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b9bb0700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04b406ab40 0x7f04b40c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04ba3b1700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40729b0 0x7f04b40c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 -- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04b406ab40 msgr2=0x7f04b40c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04b406ab40 0x7f04b40c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 -- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40729b0 msgr2=0x7f04b40c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40729b0 0x7f04b40c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 -- 172.21.15.158:0/1052395686 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac009b30 con 0x7f04b406e5d0 2023-10-16T11:20:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04ba3b1700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40729b0 0x7f04b40c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b9bb0700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04b406ab40 0x7f04b40c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:31.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.814+0000 7f04b93af700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b406e5d0 0x7f04b40c88b0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f04b000c500 tx=0x7f04b000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.815+0000 7f04aa7fc700 1 -- 172.21.15.158:0/1052395686 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b00052d0 con 0x7f04b406e5d0 2023-10-16T11:20:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.815+0000 7f04aa7fc700 1 -- 172.21.15.158:0/1052395686 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b0005430 con 0x7f04b406e5d0 2023-10-16T11:20:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.815+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b41f6be0 con 0x7f04b406e5d0 2023-10-16T11:20:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.815+0000 7f04aa7fc700 1 -- 172.21.15.158:0/1052395686 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b00038c0 con 0x7f04b406e5d0 2023-10-16T11:20:31.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.815+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b40c9600 con 0x7f04b406e5d0 2023-10-16T11:20:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.815+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b41f7100 con 0x7f04b406e5d0 2023-10-16T11:20:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.816+0000 7f04aa7fc700 1 -- 172.21.15.158:0/1052395686 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f04b0003a20 con 0x7f04b406e5d0 2023-10-16T11:20:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.816+0000 7f04aa7fc700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f049806b140 0x7f049806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.817+0000 7f04aa7fc700 1 -- 172.21.15.158:0/1052395686 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f04b00890c0 con 0x7f04b406e5d0 2023-10-16T11:20:31.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.817+0000 7f04b9bb0700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f049806b140 0x7f049806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.817+0000 7f04b9bb0700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f049806b140 0x7f049806d600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000f60 tx=0x7f04a4007ce0 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:31.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:31.821+0000 7f04aa7fc700 1 -- 172.21.15.158:0/1052395686 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f04b41f7100 con 0x7f04b406e5d0 2023-10-16T11:20:32.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.056+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f04b41f7100 con 0x7f04b406e5d0 2023-10-16T11:20:32.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.056+0000 7f04aa7fc700 1 -- 172.21.15.158:0/1052395686 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v129) v1 ==== 53+0+73 (secure 0 0 0) 0x7f04b41f7100 con 0x7f04b406e5d0 2023-10-16T11:20:32.059 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T11:20:32.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f049806b140 msgr2=0x7f049806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:32.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f049806b140 0x7f049806d600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000f60 tx=0x7f04a4007ce0 comp rx=0 tx=0).stop 2023-10-16T11:20:32.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b406e5d0 msgr2=0x7f04b40c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:32.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b406e5d0 0x7f04b40c88b0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f04b000c500 tx=0x7f04b000c8c0 comp rx=0 tx=0).stop 2023-10-16T11:20:32.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 shutdown_connections 2023-10-16T11:20:32.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f049806b140 0x7f049806d600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:32.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40729b0 0x7f04b40c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:32.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04b406ab40 0x7f04b40c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:32.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 --2- 172.21.15.158:0/1052395686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b406e5d0 0x7f04b40c88b0 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:32.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 >> 172.21.15.158:0/1052395686 conn(0x7f04b40d02b0 msgr2=0x7f04b40ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:32.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 shutdown_connections 2023-10-16T11:20:32.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:32.059+0000 7f04bbe14700 1 -- 172.21.15.158:0/1052395686 wait complete. 2023-10-16T11:20:32.072 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T11:20:32.073 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T11:20:32.073 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T11:20:32.073 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T11:20:32.074 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T11:20:32.074 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T11:20:32.074 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T11:20:32.075 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T11:20:32.075 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:20:32.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.076 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T11:20:32.093 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T11:20:32.093 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T11:20:32.093 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T11:20:32.094 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:20:32.094 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T11:20:32.094 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:20:32.094 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:20:32.094 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:20:32.095 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:20:32.095 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:20:32.095 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:20:32.095 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T11:20:32.096 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T11:20:32.096 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T11:20:32.096 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T11:20:32.096 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 36601sec preferred_lft 36601sec 2023-10-16T11:20:32.096 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T11:20:32.096 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:20:32.097 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T11:20:32.097 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T11:20:32.097 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T11:20:32.097 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:20:32.097 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T11:20:32.097 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:20:32.098 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:20:32.098 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:20:32.098 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T11:20:32.098 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:20:32.098 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:20:32.099 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:20:32.099 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T11:20:32.099 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:20:32.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.100 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T11:20:32.158 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T11:20:32.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.160 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T11:20:32.219 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T11:20:32.219 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T11:20:32.219 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T11:20:32.219 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T11:20:32.219 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T11:20:32.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.220 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T11:20:32.221 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T11:20:32.221 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T11:20:32.221 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T11:20:32.222 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T11:20:32.222 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T11:20:32.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.324 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:20:32.340 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:20:32.340 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:20:32.341 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:20:32.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.341 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T11:20:32.407 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:20:32.408 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T11:20:32.464 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:20:32.464 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:20:32.465 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:20:32.520 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T11:20:32.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.521 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:20:32.605 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:20:32.605 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:20:32.629 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:20:32.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:20:32.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:20:32.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:20:32.630 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:20:32.631 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:20:32.631 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:20:32.631 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:20:32.704 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:20:32.704 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:20:32.704 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:20:32.704 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:20:32.704 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:20:32.705 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:20:32.705 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:20:32.705 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:20:32.705 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T11:20:32.705 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T11:20:32.706 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T11:20:32.707 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T11:20:32.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.708 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:20:32.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:32.736 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T11:20:32.823 INFO:teuthology.orchestra.run.smithi158.stdout:client39473 2023-10-16T11:20:32.824 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:20:32.824 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:32.842 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T11:20:33.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.004+0000 7ff3579e7700 1 Processor -- start 2023-10-16T11:20:33.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff3579e7700 1 -- start start 2023-10-16T11:20:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff3579e7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff350159280 0x7ff35006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff3579e7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff3500d0e10 0x7ff35015c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff3579e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3500cbef0 0x7ff3500cc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff3579e7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3500cb710 con 0x7ff350159280 2023-10-16T11:20:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff3579e7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff350159660 con 0x7ff3500d0e10 2023-10-16T11:20:33.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff3579e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35006b120 con 0x7ff3500cbef0 2023-10-16T11:20:33.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.005+0000 7ff355783700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff350159280 0x7ff35006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff350159280 0x7ff35006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40438/0 (socket says 172.21.15.158:40438) 2023-10-16T11:20:33.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 -- 172.21.15.158:0/4122141582 learned_addr learned my addr 172.21.15.158:0/4122141582 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:33.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff354f82700 1 --2- 172.21.15.158:0/4122141582 >> v2:172.21.15.172:3300/0 conn(0x7ff3500d0e10 0x7ff35015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355f84700 1 --2- 172.21.15.158:0/4122141582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3500cbef0 0x7ff3500cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 -- 172.21.15.158:0/4122141582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3500cbef0 msgr2=0x7ff3500cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:33.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 --2- 172.21.15.158:0/4122141582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3500cbef0 0x7ff3500cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 -- 172.21.15.158:0/4122141582 >> v2:172.21.15.172:3300/0 conn(0x7ff3500d0e10 msgr2=0x7ff35015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:33.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 --2- 172.21.15.158:0/4122141582 >> v2:172.21.15.172:3300/0 conn(0x7ff3500d0e10 0x7ff35015c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 -- 172.21.15.158:0/4122141582 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35006b400 con 0x7ff350159280 2023-10-16T11:20:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff354f82700 1 --2- 172.21.15.158:0/4122141582 >> v2:172.21.15.172:3300/0 conn(0x7ff3500d0e10 0x7ff35015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff355783700 1 --2- 172.21.15.158:0/4122141582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff350159280 0x7ff35006a8d0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7ff3500c7400 tx=0x7ff340009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d4938bbef4ad36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff347fff700 1 -- 172.21.15.158:0/4122141582 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340011830 con 0x7ff350159280 2023-10-16T11:20:33.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff347fff700 1 -- 172.21.15.158:0/4122141582 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff340011990 con 0x7ff350159280 2023-10-16T11:20:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.006+0000 7ff347fff700 1 -- 172.21.15.158:0/4122141582 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340011d20 con 0x7ff350159280 2023-10-16T11:20:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 -- 172.21.15.158:0/4122141582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff350159280 msgr2=0x7ff35006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 --2- 172.21.15.158:0/4122141582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff350159280 0x7ff35006a8d0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7ff3500c7400 tx=0x7ff340009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 -- 172.21.15.158:0/4122141582 shutdown_connections 2023-10-16T11:20:33.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 --2- 172.21.15.158:0/4122141582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3500cbef0 0x7ff3500cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 --2- 172.21.15.158:0/4122141582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff350159280 0x7ff35006a8d0 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 --2- 172.21.15.158:0/4122141582 >> v2:172.21.15.172:3300/0 conn(0x7ff3500d0e10 0x7ff35015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 -- 172.21.15.158:0/4122141582 >> 172.21.15.158:0/4122141582 conn(0x7ff350099a10 msgr2=0x7ff3500d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:33.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 -- 172.21.15.158:0/4122141582 shutdown_connections 2023-10-16T11:20:33.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.007+0000 7ff3579e7700 1 -- 172.21.15.158:0/4122141582 wait complete. 2023-10-16T11:20:33.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.008+0000 7ff3579e7700 1 Processor -- start 2023-10-16T11:20:33.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.008+0000 7ff3579e7700 1 -- start start 2023-10-16T11:20:33.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff3579e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3500cbef0 0x7ff350155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff3579e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3500d0e10 0x7ff3500af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff3579e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff350159280 0x7ff3500afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff3579e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500b0430 con 0x7ff3500cbef0 2023-10-16T11:20:33.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff3579e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500c49d0 con 0x7ff3500d0e10 2023-10-16T11:20:33.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff3579e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500c40a0 con 0x7ff350159280 2023-10-16T11:20:33.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3500cbef0 0x7ff350155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3500cbef0 0x7ff350155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40444/0 (socket says 172.21.15.158:40444) 2023-10-16T11:20:33.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 -- 172.21.15.158:0/1860953819 learned_addr learned my addr 172.21.15.158:0/1860953819 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:33.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff354f82700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3500d0e10 0x7ff3500af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355f84700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff350159280 0x7ff3500afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 -- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff350159280 msgr2=0x7ff3500afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff350159280 0x7ff3500afdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 -- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3500d0e10 msgr2=0x7ff3500af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3500d0e10 0x7ff3500af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 -- 172.21.15.158:0/1860953819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff340009550 con 0x7ff3500cbef0 2023-10-16T11:20:33.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff355783700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3500cbef0 0x7ff350155600 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7ff340009fd0 tx=0x7ff340004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:33.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff345ffb700 1 -- 172.21.15.158:0/1860953819 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340004550 con 0x7ff3500cbef0 2023-10-16T11:20:33.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff345ffb700 1 -- 172.21.15.158:0/1860953819 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34001f070 con 0x7ff3500cbef0 2023-10-16T11:20:33.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3500c4380 con 0x7ff3500cbef0 2023-10-16T11:20:33.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.009+0000 7ff345ffb700 1 -- 172.21.15.158:0/1860953819 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3400038e0 con 0x7ff3500cbef0 2023-10-16T11:20:33.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.010+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3500c6e50 con 0x7ff3500cbef0 2023-10-16T11:20:33.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.011+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3500681b0 con 0x7ff3500cbef0 2023-10-16T11:20:33.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.011+0000 7ff345ffb700 1 -- 172.21.15.158:0/1860953819 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff340003a40 con 0x7ff3500cbef0 2023-10-16T11:20:33.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.011+0000 7ff345ffb700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff33406b140 0x7ff33406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.012+0000 7ff345ffb700 1 -- 172.21.15.158:0/1860953819 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff340021070 con 0x7ff3500cbef0 2023-10-16T11:20:33.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.012+0000 7ff354f82700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff33406b140 0x7ff33406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.012+0000 7ff354f82700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff33406b140 0x7ff33406d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff350159c20 tx=0x7ff34c008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:33.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.016+0000 7ff345ffb700 1 -- 172.21.15.158:0/1860953819 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff340055f90 con 0x7ff3500cbef0 2023-10-16T11:20:33.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.254+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff35006c6d0 con 0x7ff3500cbef0 2023-10-16T11:20:33.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.782+0000 7ff345ffb700 1 -- 172.21.15.158:0/1860953819 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v130) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff3400595b0 con 0x7ff3500cbef0 2023-10-16T11:20:33.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff33406b140 msgr2=0x7ff33406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff33406b140 0x7ff33406d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff350159c20 tx=0x7ff34c008040 comp rx=0 tx=0).stop 2023-10-16T11:20:33.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3500cbef0 msgr2=0x7ff350155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3500cbef0 0x7ff350155600 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7ff340009fd0 tx=0x7ff340004750 comp rx=0 tx=0).stop 2023-10-16T11:20:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 shutdown_connections 2023-10-16T11:20:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7ff33406b140 0x7ff33406d600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff350159280 0x7ff3500afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3500cbef0 0x7ff350155600 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 --2- 172.21.15.158:0/1860953819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3500d0e10 0x7ff3500af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 >> 172.21.15.158:0/1860953819 conn(0x7ff350099a10 msgr2=0x7ff350073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 shutdown_connections 2023-10-16T11:20:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.785+0000 7ff3579e7700 1 -- 172.21.15.158:0/1860953819 wait complete. 2023-10-16T11:20:33.800 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:33.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.920+0000 7f97ff640700 1 Processor -- start 2023-10-16T11:20:33.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.920+0000 7f97ff640700 1 -- start start 2023-10-16T11:20:33.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97ff640700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f97f806e5d0 0x7f97f806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97ff640700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 0x7f97f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97ff640700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f8071e50 con 0x7f97f806ab40 2023-10-16T11:20:33.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97ff640700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f8161270 con 0x7f97f806e5d0 2023-10-16T11:20:33.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97ff640700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8161510 con 0x7f97f80729b0 2023-10-16T11:20:33.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97fd3dc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f97f806ab40 0x7f97f80722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40628/0 (socket says 172.21.15.158:40628) 2023-10-16T11:20:33.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97fd3dc700 1 -- 172.21.15.158:0/857950662 learned_addr learned my addr 172.21.15.158:0/857950662 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:33.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/857950662 >> v2:172.21.15.172:3300/0 conn(0x7f97f806e5d0 0x7f97f806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.921+0000 7f97fdbdd700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 0x7f97f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496698974 0 0) 0x7f97f8071e50 con 0x7f97f806ab40 2023-10-16T11:20:33.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e40035b0 con 0x7f97f806ab40 2023-10-16T11:20:33.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512063151 0 0) 0x7f97e40035b0 con 0x7f97f806ab40 2023-10-16T11:20:33.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 msgr2=0x7f97f8071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:33.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 0x7f97f8071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 >> v2:172.21.15.172:3300/0 conn(0x7f97f806e5d0 msgr2=0x7f97f806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:33.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 --2- 172.21.15.158:0/857950662 >> v2:172.21.15.172:3300/0 conn(0x7f97f806e5d0 0x7f97f806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f81617f0 con 0x7f97f806ab40 2023-10-16T11:20:33.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f97e8001670 con 0x7f97f806ab40 2023-10-16T11:20:33.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 >> v1:172.21.15.158:6789/0 conn(0x7f97f806ab40 legacy=0x7f97f80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 0x7f97e4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97e4004d90 0x7f97e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8161270 con 0x7f97e4003f80 2023-10-16T11:20:33.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8161510 con 0x7f97e4004d90 2023-10-16T11:20:33.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e400ba10 con 0x7f97f80729b0 2023-10-16T11:20:33.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fd3dc700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 0x7f97e4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fdbdd700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97e4004d90 0x7f97e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fcbdb700 1 -- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 msgr2=0x7f97e4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:33.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80729b0 0x7f97e4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fcbdb700 1 -- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97e4004d90 msgr2=0x7f97e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:33.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97e4004d90 0x7f97e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.922+0000 7f97fcbdb700 1 -- 172.21.15.158:0/857950662 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40035b0 con 0x7f97e4003f80 2023-10-16T11:20:33.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97e4004850 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009b30 tx=0x7f97e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3c1e1faacc2b7a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:33.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f400e660 con 0x7f97e4003f80 2023-10-16T11:20:33.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f400e7c0 con 0x7f97e4003f80 2023-10-16T11:20:33.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97effff700 1 -- 172.21.15.158:0/857950662 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f400eb30 con 0x7f97e4003f80 2023-10-16T11:20:33.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97ff640700 1 -- 172.21.15.158:0/857950662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 msgr2=0x7f97e4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97ff640700 1 --2- 172.21.15.158:0/857950662 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97e4004850 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009b30 tx=0x7f97e4005880 comp rx=0 tx=0).stop 2023-10-16T11:20:33.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97fdbdd700 1 -- 172.21.15.158:0/857950662 reap_dead start 2023-10-16T11:20:33.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97ff640700 1 -- 172.21.15.158:0/857950662 shutdown_connections 2023-10-16T11:20:33.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97ff640700 1 -- 172.21.15.158:0/857950662 >> 172.21.15.158:0/857950662 conn(0x7f97f80d02b0 msgr2=0x7f97f80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:33.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97ff640700 1 -- 172.21.15.158:0/857950662 shutdown_connections 2023-10-16T11:20:33.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.923+0000 7f97ff640700 1 -- 172.21.15.158:0/857950662 wait complete. 2023-10-16T11:20:33.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 Processor -- start 2023-10-16T11:20:33.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 -- start start 2023-10-16T11:20:33.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97f8154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80c3c80 0x7f97f80ad3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97f80c40e0 0x7f97f80ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8155220 con 0x7f97e4003f80 2023-10-16T11:20:33.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f80c8600 con 0x7f97f80c40e0 2023-10-16T11:20:33.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97ff640700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f80c8dc0 con 0x7f97f80c3c80 2023-10-16T11:20:33.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97fd3dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97f8154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97fd3dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97f8154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40464/0 (socket says 172.21.15.158:40464) 2023-10-16T11:20:33.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.925+0000 7f97fd3dc700 1 -- 172.21.15.158:0/163839063 learned_addr learned my addr 172.21.15.158:0/163839063 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:33.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fdbdd700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97f80c40e0 0x7f97f80ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80c3c80 0x7f97f80ad3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fd3dc700 1 -- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80c3c80 msgr2=0x7f97f80ad3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fd3dc700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80c3c80 0x7f97f80ad3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fd3dc700 1 -- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97f80c40e0 msgr2=0x7f97f80ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:33.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fd3dc700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97f80c40e0 0x7f97f80ad910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:33.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fd3dc700 1 -- 172.21.15.158:0/163839063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f80c8320 con 0x7f97e4003f80 2023-10-16T11:20:33.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fdbdd700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97f80c40e0 0x7f97f80ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:33.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97fd3dc700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97f8154ce0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f97e8007310 tx=0x7f97e8007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:33.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97edffb700 1 -- 172.21.15.158:0/163839063 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8007ca0 con 0x7f97e4003f80 2023-10-16T11:20:33.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97edffb700 1 -- 172.21.15.158:0/163839063 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e8007800 con 0x7f97e4003f80 2023-10-16T11:20:33.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f80c9060 con 0x7f97e4003f80 2023-10-16T11:20:33.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97edffb700 1 -- 172.21.15.158:0/163839063 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e800fb10 con 0x7f97e4003f80 2023-10-16T11:20:33.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.926+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f80c9860 con 0x7f97e4003f80 2023-10-16T11:20:33.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.927+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f80b2d10 con 0x7f97e4003f80 2023-10-16T11:20:33.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.927+0000 7f97edffb700 1 -- 172.21.15.158:0/163839063 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f97e800fc70 con 0x7f97e4003f80 2023-10-16T11:20:33.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.928+0000 7f97edffb700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f97dc06b140 0x7f97dc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:33.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.928+0000 7f97edffb700 1 -- 172.21.15.158:0/163839063 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f97e8089d70 con 0x7f97e4003f80 2023-10-16T11:20:33.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.928+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f97dc06b140 0x7f97dc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:33.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.928+0000 7f97fcbdb700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f97dc06b140 0x7f97dc06d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f97f40090b0 tx=0x7f97f401e040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:33.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:33.932+0000 7f97edffb700 1 -- 172.21.15.158:0/163839063 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97e8054830 con 0x7f97e4003f80 2023-10-16T11:20:34.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.125+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97f80c9580 con 0x7f97e4003f80 2023-10-16T11:20:34.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.126+0000 7f97edffb700 1 -- 172.21.15.158:0/163839063 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+17531 (secure 0 0 0) 0x7f97f80c9580 con 0x7f97e4003f80 2023-10-16T11:20:34.129 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:34.130 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":237,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:26.734730+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:34.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f97dc06b140 msgr2=0x7f97dc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f97dc06b140 0x7f97dc06d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f97f40090b0 tx=0x7f97f401e040 comp rx=0 tx=0).stop 2023-10-16T11:20:34.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 msgr2=0x7f97f8154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97f8154ce0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f97e8007310 tx=0x7f97e8007620 comp rx=0 tx=0).stop 2023-10-16T11:20:34.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 shutdown_connections 2023-10-16T11:20:34.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f97dc06b140 0x7f97dc06d600 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97f80c40e0 0x7f97f80ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97e4003f80 0x7f97f8154ce0 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 --2- 172.21.15.158:0/163839063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97f80c3c80 0x7f97f80ad3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 >> 172.21.15.158:0/163839063 conn(0x7f97f80d02b0 msgr2=0x7f97f81576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:34.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 shutdown_connections 2023-10-16T11:20:34.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.128+0000 7f97ff640700 1 -- 172.21.15.158:0/163839063 wait complete. 2023-10-16T11:20:34.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:20:23.549102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:20:24.086127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '237', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:20:34.143 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:20:34.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.266+0000 7f113546f700 1 Processor -- start 2023-10-16T11:20:34.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.266+0000 7f113546f700 1 -- start start 2023-10-16T11:20:34.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.266+0000 7f113546f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11300ca8b0 0x7f113015de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.266+0000 7f113546f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11300cd3d0 con 0x7f11300b0c00 2023-10-16T11:20:34.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.266+0000 7f113546f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11300cdf50 con 0x7f1130159af0 2023-10-16T11:20:34.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.266+0000 7f113546f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11300cd620 con 0x7f11300ca8b0 2023-10-16T11:20:34.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f11300b0c00 0x7f1130070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40652/0 (socket says 172.21.15.158:40652) 2023-10-16T11:20:34.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112f7fe700 1 -- 172.21.15.158:0/306949581 learned_addr learned my addr 172.21.15.158:0/306949581 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:34.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496114782 0 0) 0x7f11300cd3d0 con 0x7f11300b0c00 2023-10-16T11:20:34.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112e7fc700 1 --2- 172.21.15.158:0/306949581 >> v2:172.21.15.172:3300/0 conn(0x7f11300ca8b0 0x7f113015de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111c0035b0 con 0x7f11300b0c00 2023-10-16T11:20:34.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058570941 0 0) 0x7f111c0035b0 con 0x7f11300b0c00 2023-10-16T11:20:34.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 >> v2:172.21.15.172:3300/0 conn(0x7f11300ca8b0 msgr2=0x7f113015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:34.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 --2- 172.21.15.158:0/306949581 >> v2:172.21.15.172:3300/0 conn(0x7f11300ca8b0 0x7f113015de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 >> v1:172.21.15.172:6789/0 conn(0x7f1130159af0 legacy=0x7f11300ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11300cd900 con 0x7f11300b0c00 2023-10-16T11:20:34.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f11280019f0 con 0x7f11300b0c00 2023-10-16T11:20:34.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 >> v1:172.21.15.158:6789/0 conn(0x7f11300b0c00 legacy=0x7f1130070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f111c003a00 0x7f111c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c004400 0x7f111c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c0050d0 0x7f111c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11300cd620 con 0x7f111c004400 2023-10-16T11:20:34.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c00c030 con 0x7f111c003a00 2023-10-16T11:20:34.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c00c1f0 con 0x7f111c0050d0 2023-10-16T11:20:34.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112e7fc700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c004400 0x7f111c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112effd700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f111c003a00 0x7f111c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.267+0000 7f112f7fe700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c0050d0 0x7f111c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112e7fc700 1 -- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c0050d0 msgr2=0x7f111c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:34.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112e7fc700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c0050d0 0x7f111c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112e7fc700 1 -- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f111c003a00 msgr2=0x7f111c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:34.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112e7fc700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f111c003a00 0x7f111c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112e7fc700 1 -- 172.21.15.158:0/306949581 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0035b0 con 0x7f111c004400 2023-10-16T11:20:34.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112effd700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f111c003a00 0x7f111c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:34.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112f7fe700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c0050d0 0x7f111c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:34.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112f7fe700 1 -- 172.21.15.158:0/306949581 reap_dead start 2023-10-16T11:20:34.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112e7fc700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c004400 0x7f111c0048d0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f1120009b60 tx=0x7f111c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a624667c26f9cb74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112000e6c0 con 0x7f111c004400 2023-10-16T11:20:34.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112000e820 con 0x7f111c004400 2023-10-16T11:20:34.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f112dffb700 1 -- 172.21.15.158:0/306949581 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112000eb90 con 0x7f111c004400 2023-10-16T11:20:34.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f113546f700 1 -- 172.21.15.158:0/306949581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c004400 msgr2=0x7f111c0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f113546f700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c004400 0x7f111c0048d0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f1120009b60 tx=0x7f111c005880 comp rx=0 tx=0).stop 2023-10-16T11:20:34.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f113546f700 1 -- 172.21.15.158:0/306949581 shutdown_connections 2023-10-16T11:20:34.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f113546f700 1 --2- 172.21.15.158:0/306949581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c004400 0x7f111c0048d0 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f113546f700 1 -- 172.21.15.158:0/306949581 >> 172.21.15.158:0/306949581 conn(0x7f113006af00 msgr2=0x7f11300cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:34.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f113546f700 1 -- 172.21.15.158:0/306949581 shutdown_connections 2023-10-16T11:20:34.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.268+0000 7f113546f700 1 -- 172.21.15.158:0/306949581 wait complete. 2023-10-16T11:20:34.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.270+0000 7f113546f700 1 Processor -- start 2023-10-16T11:20:34.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.270+0000 7f113546f700 1 -- start start 2023-10-16T11:20:34.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f113546f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c004400 0x7f11301f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f113546f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113015c930 0x7f113015cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f113546f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11300c83d0 0x7f11300c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f113546f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11301f2490 con 0x7f113015c930 2023-10-16T11:20:34.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f113546f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11300c8dc0 con 0x7f11300c83d0 2023-10-16T11:20:34.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f113546f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11301f5a90 con 0x7f111c004400 2023-10-16T11:20:34.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f112e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113015c930 0x7f113015cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f112e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113015c930 0x7f113015cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40488/0 (socket says 172.21.15.158:40488) 2023-10-16T11:20:34.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f112e7fc700 1 -- 172.21.15.158:0/790743822 learned_addr learned my addr 172.21.15.158:0/790743822 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:34.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f112f7fe700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11300c83d0 0x7f11300c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.271+0000 7f112effd700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c004400 0x7f11301f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f112e7fc700 1 -- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c004400 msgr2=0x7f11301f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f112e7fc700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c004400 0x7f11301f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f112e7fc700 1 -- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11300c83d0 msgr2=0x7f11300c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f112e7fc700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11300c83d0 0x7f11300c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f112e7fc700 1 -- 172.21.15.158:0/790743822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1130154120 con 0x7f113015c930 2023-10-16T11:20:34.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f112f7fe700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11300c83d0 0x7f11300c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:34.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f112e7fc700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113015c930 0x7f113015cdb0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f112000e4b0 tx=0x7f11200047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f11277fe700 1 -- 172.21.15.158:0/790743822 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112000e6c0 con 0x7f113015c930 2023-10-16T11:20:34.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f11277fe700 1 -- 172.21.15.158:0/790743822 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120003ad0 con 0x7f113015c930 2023-10-16T11:20:34.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11301f1810 con 0x7f113015c930 2023-10-16T11:20:34.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f11277fe700 1 -- 172.21.15.158:0/790743822 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120003e80 con 0x7f113015c930 2023-10-16T11:20:34.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.272+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11301f1dd0 con 0x7f113015c930 2023-10-16T11:20:34.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.273+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11300681b0 con 0x7f113015c930 2023-10-16T11:20:34.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.273+0000 7f11277fe700 1 -- 172.21.15.158:0/790743822 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1120013470 con 0x7f113015c930 2023-10-16T11:20:34.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.274+0000 7f11277fe700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f111006b140 0x7f111006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.274+0000 7f112effd700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f111006b140 0x7f111006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.274+0000 7f11277fe700 1 -- 172.21.15.158:0/790743822 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f112008b480 con 0x7f113015c930 2023-10-16T11:20:34.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.274+0000 7f112effd700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f111006b140 0x7f111006d600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1118007bd0 tx=0x7f1118009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.278+0000 7f11277fe700 1 -- 172.21.15.158:0/790743822 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1120056060 con 0x7f113015c930 2023-10-16T11:20:34.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.468+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1130154120 con 0x7f113015c930 2023-10-16T11:20:34.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.468+0000 7f11277fe700 1 -- 172.21.15.158:0/790743822 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v237) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1130154120 con 0x7f113015c930 2023-10-16T11:20:34.471 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:20:34.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f111006b140 msgr2=0x7f111006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f111006b140 0x7f111006d600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1118007bd0 tx=0x7f1118009040 comp rx=0 tx=0).stop 2023-10-16T11:20:34.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113015c930 msgr2=0x7f113015cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113015c930 0x7f113015cdb0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f112000e4b0 tx=0x7f11200047c0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 shutdown_connections 2023-10-16T11:20:34.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f111006b140 0x7f111006d600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11300c83d0 0x7f11300c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f111c004400 0x7f11301f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 --2- 172.21.15.158:0/790743822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113015c930 0x7f113015cdb0 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 >> 172.21.15.158:0/790743822 conn(0x7f113006af00 msgr2=0x7f11301574a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:34.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 shutdown_connections 2023-10-16T11:20:34.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.471+0000 7f113546f700 1 -- 172.21.15.158:0/790743822 wait complete. 2023-10-16T11:20:34.485 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:20:34.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.607+0000 7fd68e548700 1 Processor -- start 2023-10-16T11:20:34.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.607+0000 7fd68e548700 1 -- start start 2023-10-16T11:20:34.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd68e548700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd68815b6b0 0x7fd6880cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd68e548700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6880d0c60 0x7fd6880b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd68e548700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6880b4420 con 0x7fd6880b3460 2023-10-16T11:20:34.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd68e548700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68815ba90 con 0x7fd6880d0c60 2023-10-16T11:20:34.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd68e548700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6880b38d0 con 0x7fd68815b6b0 2023-10-16T11:20:34.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6880d0c60 0x7fd6880b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6880d0c60 0x7fd6880b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40494/0 (socket says 172.21.15.158:40494) 2023-10-16T11:20:34.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 -- 172.21.15.158:0/721712842 learned_addr learned my addr 172.21.15.158:0/721712842 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:34.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd687fff700 1 --2- 172.21.15.158:0/721712842 >> v2:172.21.15.172:3300/0 conn(0x7fd68815b6b0 0x7fd6880cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 -- 172.21.15.158:0/721712842 >> v2:172.21.15.172:3300/0 conn(0x7fd68815b6b0 msgr2=0x7fd6880cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:34.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 --2- 172.21.15.158:0/721712842 >> v2:172.21.15.172:3300/0 conn(0x7fd68815b6b0 0x7fd6880cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 -- 172.21.15.158:0/721712842 >> v1:172.21.15.172:6789/0 conn(0x7fd6880b3460 legacy=0x7fd688070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:34.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 -- 172.21.15.158:0/721712842 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6880cbc20 con 0x7fd6880d0c60 2023-10-16T11:20:34.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.608+0000 7fd6877fe700 1 --2- 172.21.15.158:0/721712842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6880d0c60 0x7fd6880b2f20 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7fd68806ffb0 tx=0x7fd678009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1727b6b603fa4c03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd686ffd700 1 -- 172.21.15.158:0/721712842 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67800b860 con 0x7fd6880d0c60 2023-10-16T11:20:34.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd686ffd700 1 -- 172.21.15.158:0/721712842 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67800b9c0 con 0x7fd6880d0c60 2023-10-16T11:20:34.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd686ffd700 1 -- 172.21.15.158:0/721712842 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67800bd50 con 0x7fd6880d0c60 2023-10-16T11:20:34.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 -- 172.21.15.158:0/721712842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6880d0c60 msgr2=0x7fd6880b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 --2- 172.21.15.158:0/721712842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6880d0c60 0x7fd6880b2f20 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7fd68806ffb0 tx=0x7fd678009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:34.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 -- 172.21.15.158:0/721712842 shutdown_connections 2023-10-16T11:20:34.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 --2- 172.21.15.158:0/721712842 >> v2:172.21.15.172:3300/0 conn(0x7fd68815b6b0 0x7fd6880cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 --2- 172.21.15.158:0/721712842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6880d0c60 0x7fd6880b2f20 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 -- 172.21.15.158:0/721712842 >> 172.21.15.158:0/721712842 conn(0x7fd6880cdf20 msgr2=0x7fd6880ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:34.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 -- 172.21.15.158:0/721712842 shutdown_connections 2023-10-16T11:20:34.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.609+0000 7fd68e548700 1 -- 172.21.15.158:0/721712842 wait complete. 2023-10-16T11:20:34.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.610+0000 7fd68e548700 1 Processor -- start 2023-10-16T11:20:34.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.610+0000 7fd68e548700 1 -- start start 2023-10-16T11:20:34.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68e548700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6880b3460 0x7fd688150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68e548700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6880d0c60 0x7fd6880c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68e548700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd68815b6b0 0x7fd6881f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68e548700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd688151e60 con 0x7fd68815b6b0 2023-10-16T11:20:34.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68e548700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd688151830 con 0x7fd6880b3460 2023-10-16T11:20:34.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68e548700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd688151070 con 0x7fd6880d0c60 2023-10-16T11:20:34.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68cae5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd68815b6b0 0x7fd6881f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68cae5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd68815b6b0 0x7fd6881f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40502/0 (socket says 172.21.15.158:40502) 2023-10-16T11:20:34.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd68cae5700 1 -- 172.21.15.158:0/1575558288 learned_addr learned my addr 172.21.15.158:0/1575558288 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:34.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.611+0000 7fd687fff700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6880b3460 0x7fd688150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd6877fe700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6880d0c60 0x7fd6880c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68cae5700 1 -- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6880d0c60 msgr2=0x7fd6880c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68cae5700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6880d0c60 0x7fd6880c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68cae5700 1 -- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6880b3460 msgr2=0x7fd688150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68cae5700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6880b3460 0x7fd688150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68cae5700 1 -- 172.21.15.158:0/1575558288 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678009550 con 0x7fd68815b6b0 2023-10-16T11:20:34.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd6877fe700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6880d0c60 0x7fd6880c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:34.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68cae5700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd68815b6b0 0x7fd6881f2c90 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7fd68000cb50 tx=0x7fd68000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd684ff9700 1 -- 172.21.15.158:0/1575558288 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680003c00 con 0x7fd68815b6b0 2023-10-16T11:20:34.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd684ff9700 1 -- 172.21.15.158:0/1575558288 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd680003d60 con 0x7fd68815b6b0 2023-10-16T11:20:34.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68806ca70 con 0x7fd68815b6b0 2023-10-16T11:20:34.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd684ff9700 1 -- 172.21.15.158:0/1575558288 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68000ab70 con 0x7fd68815b6b0 2023-10-16T11:20:34.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.612+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd688151a80 con 0x7fd68815b6b0 2023-10-16T11:20:34.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.613+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6880681b0 con 0x7fd68815b6b0 2023-10-16T11:20:34.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.613+0000 7fd684ff9700 1 -- 172.21.15.158:0/1575558288 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd68000acd0 con 0x7fd68815b6b0 2023-10-16T11:20:34.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.614+0000 7fd684ff9700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd67406b140 0x7fd67406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.614+0000 7fd684ff9700 1 -- 172.21.15.158:0/1575558288 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fd680089cf0 con 0x7fd68815b6b0 2023-10-16T11:20:34.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.614+0000 7fd687fff700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd67406b140 0x7fd67406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.614+0000 7fd687fff700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd67406b140 0x7fd67406d600 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009930 tx=0x7fd67c008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.618+0000 7fd684ff9700 1 -- 172.21.15.158:0/1575558288 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd68008e070 con 0x7fd68815b6b0 2023-10-16T11:20:34.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.809+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6880cc1b0 con 0x7fd68815b6b0 2023-10-16T11:20:34.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.809+0000 7fd684ff9700 1 -- 172.21.15.158:0/1575558288 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v237) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6800547b0 con 0x7fd68815b6b0 2023-10-16T11:20:34.811 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:34.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd67406b140 msgr2=0x7fd67406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd67406b140 0x7fd67406d600 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009930 tx=0x7fd67c008040 comp rx=0 tx=0).stop 2023-10-16T11:20:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd68815b6b0 msgr2=0x7fd6881f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd68815b6b0 0x7fd6881f2c90 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7fd68000cb50 tx=0x7fd68000cf10 comp rx=0 tx=0).stop 2023-10-16T11:20:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 shutdown_connections 2023-10-16T11:20:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd67406b140 0x7fd67406d600 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd68815b6b0 0x7fd6881f2c90 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6880b3460 0x7fd688150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 --2- 172.21.15.158:0/1575558288 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6880d0c60 0x7fd6880c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 >> 172.21.15.158:0/1575558288 conn(0x7fd6880cdf20 msgr2=0x7fd68815c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 shutdown_connections 2023-10-16T11:20:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.811+0000 7fd68e548700 1 -- 172.21.15.158:0/1575558288 wait complete. 2023-10-16T11:20:34.825 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:20:34.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.948+0000 7f2bb59f6700 1 Processor -- start 2023-10-16T11:20:34.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.948+0000 7f2bb59f6700 1 -- start start 2023-10-16T11:20:34.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.948+0000 7f2bb59f6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2bb00b38f0 0x7f2bb00d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.948+0000 7f2bb59f6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb015e3e0 con 0x7f2bb015d840 2023-10-16T11:20:34.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.948+0000 7f2bb59f6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb015e6c0 con 0x7f2bb00d2010 2023-10-16T11:20:34.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.948+0000 7f2bb59f6700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb015ec80 con 0x7f2bb00b38f0 2023-10-16T11:20:34.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2bae7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2bb00b38f0 0x7f2bb00d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2baeffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2bb015d840 0x7f2bb015dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40682/0 (socket says 172.21.15.158:40682) 2023-10-16T11:20:34.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2baeffd700 1 -- 172.21.15.158:0/456289891 learned_addr learned my addr 172.21.15.158:0/456289891 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:34.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2bae7fc700 1 -- 172.21.15.158:0/456289891 >> v1:172.21.15.172:6789/0 conn(0x7f2bb00d2010 legacy=0x7f2bb006bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:34.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2bae7fc700 1 -- 172.21.15.158:0/456289891 >> v1:172.21.15.158:6789/0 conn(0x7f2bb015d840 legacy=0x7f2bb015dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2bae7fc700 1 -- 172.21.15.158:0/456289891 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb015e9a0 con 0x7f2bb00b38f0 2023-10-16T11:20:34.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2badffb700 1 -- 172.21.15.158:0/456289891 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064321238 0 0) 0x7f2bb015e3e0 con 0x7f2bb015d840 2023-10-16T11:20:34.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2bae7fc700 1 --2- 172.21.15.158:0/456289891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00b38f0 0x7f2bb00d1ad0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009840 tx=0x7f2ba4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6874ba7ff0f104 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2badffb700 1 -- 172.21.15.158:0/456289891 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4004230 con 0x7f2bb00b38f0 2023-10-16T11:20:34.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2badffb700 1 -- 172.21.15.158:0/456289891 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4004390 con 0x7f2bb00b38f0 2023-10-16T11:20:34.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.949+0000 7f2badffb700 1 -- 172.21.15.158:0/456289891 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba40036a0 con 0x7f2bb00b38f0 2023-10-16T11:20:34.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.950+0000 7f2bb59f6700 1 -- 172.21.15.158:0/456289891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00b38f0 msgr2=0x7f2bb00d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.950+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/456289891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00b38f0 0x7f2bb00d1ad0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009840 tx=0x7f2ba4009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:34.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.950+0000 7f2bb59f6700 1 -- 172.21.15.158:0/456289891 shutdown_connections 2023-10-16T11:20:34.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.950+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/456289891 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00b38f0 0x7f2bb00d1ad0 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.950+0000 7f2bb59f6700 1 -- 172.21.15.158:0/456289891 >> 172.21.15.158:0/456289891 conn(0x7f2bb006ec00 msgr2=0x7f2bb006efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:34.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.950+0000 7f2bb59f6700 1 -- 172.21.15.158:0/456289891 shutdown_connections 2023-10-16T11:20:34.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.950+0000 7f2bb59f6700 1 -- 172.21.15.158:0/456289891 wait complete. 2023-10-16T11:20:34.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.951+0000 7f2bb59f6700 1 Processor -- start 2023-10-16T11:20:34.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.951+0000 7f2bb59f6700 1 -- start start 2023-10-16T11:20:34.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bb59f6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bb00b38f0 0x7f2bb0156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bb59f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00d2010 0x7f2bb00c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bb59f6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bb015d840 0x7f2bb00aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bb59f6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb00b5b30 con 0x7f2bb00d2010 2023-10-16T11:20:34.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bb59f6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb00c7210 con 0x7f2bb015d840 2023-10-16T11:20:34.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bb59f6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb00b5190 con 0x7f2bb00b38f0 2023-10-16T11:20:34.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00d2010 0x7f2bb00c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00d2010 0x7f2bb00c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40514/0 (socket says 172.21.15.158:40514) 2023-10-16T11:20:34.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 -- 172.21.15.158:0/3895866779 learned_addr learned my addr 172.21.15.158:0/3895866779 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:34.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2baf7fe700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bb015d840 0x7f2bb00aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2baeffd700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bb00b38f0 0x7f2bb0156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 -- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bb00b38f0 msgr2=0x7f2bb0156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bb00b38f0 0x7f2bb0156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 -- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bb015d840 msgr2=0x7f2bb00aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:34.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bb015d840 0x7f2bb00aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:34.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 -- 172.21.15.158:0/3895866779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40094a0 con 0x7f2bb00d2010 2023-10-16T11:20:34.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2bae7fc700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00d2010 0x7f2bb00c68a0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40090b0 tx=0x7f2ba4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.952+0000 7f2b97fff700 1 -- 172.21.15.158:0/3895866779 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011630 con 0x7f2bb00d2010 2023-10-16T11:20:34.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.953+0000 7f2b97fff700 1 -- 172.21.15.158:0/3895866779 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba401e070 con 0x7f2bb00d2010 2023-10-16T11:20:34.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.953+0000 7f2b97fff700 1 -- 172.21.15.158:0/3895866779 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011990 con 0x7f2bb00d2010 2023-10-16T11:20:34.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.953+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb00b5470 con 0x7f2bb00d2010 2023-10-16T11:20:34.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.953+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb0157500 con 0x7f2bb00d2010 2023-10-16T11:20:34.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.954+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb00b5c70 con 0x7f2bb00d2010 2023-10-16T11:20:34.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.954+0000 7f2b97fff700 1 -- 172.21.15.158:0/3895866779 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2ba4011af0 con 0x7f2bb00d2010 2023-10-16T11:20:34.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.954+0000 7f2b97fff700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2b9c06b140 0x7f2b9c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:34.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.954+0000 7f2b97fff700 1 -- 172.21.15.158:0/3895866779 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f2ba408b080 con 0x7f2bb00d2010 2023-10-16T11:20:34.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.955+0000 7f2baeffd700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2b9c06b140 0x7f2b9c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:34.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.955+0000 7f2baeffd700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2b9c06b140 0x7f2b9c06d600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0000f60 tx=0x7f2ba0007c90 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:34.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:34.959+0000 7f2b97fff700 1 -- 172.21.15.158:0/3895866779 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2bb00b5c70 con 0x7f2bb00d2010 2023-10-16T11:20:35.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.152+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2bb00b5c70 con 0x7f2bb00d2010 2023-10-16T11:20:35.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.152+0000 7f2b97fff700 1 -- 172.21.15.158:0/3895866779 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v237) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2bb00b5c70 con 0x7f2bb00d2010 2023-10-16T11:20:35.154 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:35.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2b9c06b140 msgr2=0x7f2b9c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2b9c06b140 0x7f2b9c06d600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0000f60 tx=0x7f2ba0007c90 comp rx=0 tx=0).stop 2023-10-16T11:20:35.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00d2010 msgr2=0x7f2bb00c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00d2010 0x7f2bb00c68a0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40090b0 tx=0x7f2ba4004750 comp rx=0 tx=0).stop 2023-10-16T11:20:35.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 shutdown_connections 2023-10-16T11:20:35.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f2b9c06b140 0x7f2b9c06d600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bb015d840 0x7f2bb00aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bb00b38f0 0x7f2bb0156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 --2- 172.21.15.158:0/3895866779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bb00d2010 0x7f2bb00c68a0 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 >> 172.21.15.158:0/3895866779 conn(0x7f2bb006ec00 msgr2=0x7f2bb006efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:35.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 shutdown_connections 2023-10-16T11:20:35.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.154+0000 7f2bb59f6700 1 -- 172.21.15.158:0/3895866779 wait complete. 2023-10-16T11:20:35.168 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:20:35.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.286+0000 7f4a66a21700 1 Processor -- start 2023-10-16T11:20:35.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.287+0000 7f4a66a21700 1 -- start start 2023-10-16T11:20:35.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a66a21700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a6006e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a66a21700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4a6006f340 0x7f4a6006f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a66a21700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6006e770 con 0x7f4a600cc5d0 2023-10-16T11:20:35.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a66a21700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6006fd30 con 0x7f4a6006f340 2023-10-16T11:20:35.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a66a21700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600cf270 con 0x7f4a6015dfb0 2023-10-16T11:20:35.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a6006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a64fbe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4a6006f340 0x7f4a6006f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a6006e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:51356/0 (socket says 172.21.15.158:51356) 2023-10-16T11:20:35.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 -- 172.21.15.158:0/2229453517 learned_addr learned my addr 172.21.15.158:0/2229453517 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:35.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5ffff700 1 --1- 172.21.15.158:0/2229453517 >> v1:172.21.15.172:6789/0 conn(0x7f4a600cc5d0 0x7f4a600cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:44600/0 (socket says 172.21.15.158:44600) 2023-10-16T11:20:35.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 -- 172.21.15.158:0/2229453517 >> v2:172.21.15.172:3300/0 conn(0x7f4a6006f340 msgr2=0x7f4a6006f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:35.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 --2- 172.21.15.158:0/2229453517 >> v2:172.21.15.172:3300/0 conn(0x7f4a6006f340 0x7f4a6006f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 -- 172.21.15.158:0/2229453517 >> v1:172.21.15.172:6789/0 conn(0x7f4a600cc5d0 legacy=0x7f4a600cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 -- 172.21.15.158:0/2229453517 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a600cbe00 con 0x7f4a6015dfb0 2023-10-16T11:20:35.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.288+0000 7f4a5f7fe700 1 --2- 172.21.15.158:0/2229453517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a6006e230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098f0 tx=0x7f4a54009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41987bfcfd879804 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:35.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a5effd700 1 -- 172.21.15.158:0/2229453517 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54011830 con 0x7f4a6015dfb0 2023-10-16T11:20:35.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a5effd700 1 -- 172.21.15.158:0/2229453517 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a54011990 con 0x7f4a6015dfb0 2023-10-16T11:20:35.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a5effd700 1 -- 172.21.15.158:0/2229453517 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54011d20 con 0x7f4a6015dfb0 2023-10-16T11:20:35.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 -- 172.21.15.158:0/2229453517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 msgr2=0x7f4a6006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 --2- 172.21.15.158:0/2229453517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a6006e230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098f0 tx=0x7f4a54009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:35.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 -- 172.21.15.158:0/2229453517 shutdown_connections 2023-10-16T11:20:35.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 --2- 172.21.15.158:0/2229453517 >> v2:172.21.15.172:3300/0 conn(0x7f4a6006f340 0x7f4a6006f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 --2- 172.21.15.158:0/2229453517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a6006e230 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 -- 172.21.15.158:0/2229453517 >> 172.21.15.158:0/2229453517 conn(0x7f4a600d02b0 msgr2=0x7f4a600d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:35.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 -- 172.21.15.158:0/2229453517 shutdown_connections 2023-10-16T11:20:35.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.289+0000 7f4a66a21700 1 -- 172.21.15.158:0/2229453517 wait complete. 2023-10-16T11:20:35.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.290+0000 7f4a66a21700 1 Processor -- start 2023-10-16T11:20:35.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.291+0000 7f4a66a21700 1 -- start start 2023-10-16T11:20:35.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.291+0000 7f4a66a21700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a6006f340 0x7f4a600b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.291+0000 7f4a66a21700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a600cc5d0 0x7f4a600b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.291+0000 7f4a66a21700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a600aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.291+0000 7f4a66a21700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a601f4ec0 con 0x7f4a6006f340 2023-10-16T11:20:35.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.291+0000 7f4a66a21700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600b1580 con 0x7f4a600cc5d0 2023-10-16T11:20:35.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.291+0000 7f4a66a21700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600af1f0 con 0x7f4a6015dfb0 2023-10-16T11:20:35.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a6006f340 0x7f4a600b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a6006f340 0x7f4a600b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40526/0 (socket says 172.21.15.158:40526) 2023-10-16T11:20:35.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 -- 172.21.15.158:0/3168787721 learned_addr learned my addr 172.21.15.158:0/3168787721 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:35.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5f7fe700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a600cc5d0 0x7f4a600b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a64fbe700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a600aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 -- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 msgr2=0x7f4a600aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a600aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 -- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a600cc5d0 msgr2=0x7f4a600b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a600cc5d0 0x7f4a600b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 -- 172.21.15.158:0/3168787721 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a54009550 con 0x7f4a6006f340 2023-10-16T11:20:35.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5ffff700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a6006f340 0x7f4a600b0890 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c500 tx=0x7f4a5000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:35.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5cff9700 1 -- 172.21.15.158:0/3168787721 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50005cb0 con 0x7f4a6006f340 2023-10-16T11:20:35.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5cff9700 1 -- 172.21.15.158:0/3168787721 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50005e10 con 0x7f4a6006f340 2023-10-16T11:20:35.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a601f0d00 con 0x7f4a6006f340 2023-10-16T11:20:35.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a5cff9700 1 -- 172.21.15.158:0/3168787721 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a500072d0 con 0x7f4a6006f340 2023-10-16T11:20:35.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.292+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a601f4590 con 0x7f4a6006f340 2023-10-16T11:20:35.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.293+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a601f51a0 con 0x7f4a6006f340 2023-10-16T11:20:35.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.294+0000 7f4a5cff9700 1 -- 172.21.15.158:0/3168787721 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4a50007430 con 0x7f4a6006f340 2023-10-16T11:20:35.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.295+0000 7f4a5cff9700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f4a4c06b140 0x7f4a4c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.295+0000 7f4a5cff9700 1 -- 172.21.15.158:0/3168787721 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f4a500891b0 con 0x7f4a6006f340 2023-10-16T11:20:35.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.295+0000 7f4a5f7fe700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f4a4c06b140 0x7f4a4c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.295+0000 7f4a5f7fe700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f4a4c06b140 0x7f4a4c06d600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098f0 tx=0x7f4a5401d040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:35.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.299+0000 7f4a5cff9700 1 -- 172.21.15.158:0/3168787721 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a50053d00 con 0x7f4a6006f340 2023-10-16T11:20:35.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.493+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4a600af360 con 0x7f4a6006f340 2023-10-16T11:20:35.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.493+0000 7f4a5cff9700 1 -- 172.21.15.158:0/3168787721 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v237) v1 ==== 84+0+209 (secure 0 0 0) 0x7f4a50017020 con 0x7f4a6006f340 2023-10-16T11:20:35.495 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:35.495 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:20:35.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:35.496 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:20:35.496 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:20:35.496 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:35.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:35.497 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 31, 2023-10-16T11:20:35.497 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": "cephfs_metadata" 2023-10-16T11:20:35.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:35.497 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:35.497 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 32, 2023-10-16T11:20:35.497 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": "cephfs_data" 2023-10-16T11:20:35.498 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:35.498 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:20:35.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f4a4c06b140 msgr2=0x7f4a4c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f4a4c06b140 0x7f4a4c06d600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098f0 tx=0x7f4a5401d040 comp rx=0 tx=0).stop 2023-10-16T11:20:35.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a6006f340 msgr2=0x7f4a600b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a6006f340 0x7f4a600b0890 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c500 tx=0x7f4a5000c810 comp rx=0 tx=0).stop 2023-10-16T11:20:35.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 shutdown_connections 2023-10-16T11:20:35.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f4a4c06b140 0x7f4a4c06d600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a6015dfb0 0x7f4a600aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a6006f340 0x7f4a600b0890 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 --2- 172.21.15.158:0/3168787721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a600cc5d0 0x7f4a600b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.495+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 >> 172.21.15.158:0/3168787721 conn(0x7f4a600d02b0 msgr2=0x7f4a600c6700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:35.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.496+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 shutdown_connections 2023-10-16T11:20:35.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.496+0000 7f4a66a21700 1 -- 172.21.15.158:0/3168787721 wait complete. 2023-10-16T11:20:35.509 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T11:20:35.510 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T11:20:35.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.633+0000 7fe87cd17700 1 Processor -- start 2023-10-16T11:20:35.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.634+0000 7fe87cd17700 1 -- start start 2023-10-16T11:20:35.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.634+0000 7fe87cd17700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8780ce510 0x7fe87815f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.634+0000 7fe87cd17700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780cbaf0 0x7fe8780cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.634+0000 7fe87cd17700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8780cf4c0 con 0x7fe87815b590 2023-10-16T11:20:35.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.634+0000 7fe87cd17700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8780cc4c0 con 0x7fe8780ce510 2023-10-16T11:20:35.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.634+0000 7fe87cd17700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8780b49e0 con 0x7fe8780cbaf0 2023-10-16T11:20:35.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8780ce510 0x7fe87815f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8780ce510 0x7fe87815f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40536/0 (socket says 172.21.15.158:40536) 2023-10-16T11:20:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 -- 172.21.15.158:0/3881278199 learned_addr learned my addr 172.21.15.158:0/3881278199 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe87659c700 1 --1- 172.21.15.158:0/3881278199 >> v1:172.21.15.158:6789/0 conn(0x7fe87815b590 0x7fe8780b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40710/0 (socket says 172.21.15.158:40710) 2023-10-16T11:20:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe876d9d700 1 --2- 172.21.15.158:0/3881278199 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780cbaf0 0x7fe8780cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 -- 172.21.15.158:0/3881278199 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780cbaf0 msgr2=0x7fe8780cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:35.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 --2- 172.21.15.158:0/3881278199 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780cbaf0 0x7fe8780cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 -- 172.21.15.158:0/3881278199 >> v1:172.21.15.158:6789/0 conn(0x7fe87815b590 legacy=0x7fe8780b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 -- 172.21.15.158:0/3881278199 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87815f710 con 0x7fe8780ce510 2023-10-16T11:20:35.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe87559a700 1 -- 172.21.15.158:0/3881278199 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770111882 0 0) 0x7fe8780cf4c0 con 0x7fe87815b590 2023-10-16T11:20:35.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe875d9b700 1 --2- 172.21.15.158:0/3881278199 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780ce510 0x7fe87815f1d0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7fe868009920 tx=0x7fe868009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e8a5a24a02c72c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:35.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe87559a700 1 -- 172.21.15.158:0/3881278199 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86800ba20 con 0x7fe8780ce510 2023-10-16T11:20:35.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe87559a700 1 -- 172.21.15.158:0/3881278199 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86800bb80 con 0x7fe8780ce510 2023-10-16T11:20:35.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.635+0000 7fe87559a700 1 -- 172.21.15.158:0/3881278199 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868009e70 con 0x7fe8780ce510 2023-10-16T11:20:35.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 -- 172.21.15.158:0/3881278199 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780ce510 msgr2=0x7fe87815f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 --2- 172.21.15.158:0/3881278199 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780ce510 0x7fe87815f1d0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7fe868009920 tx=0x7fe868009c30 comp rx=0 tx=0).stop 2023-10-16T11:20:35.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 -- 172.21.15.158:0/3881278199 shutdown_connections 2023-10-16T11:20:35.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 --2- 172.21.15.158:0/3881278199 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780cbaf0 0x7fe8780cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 --2- 172.21.15.158:0/3881278199 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780ce510 0x7fe87815f1d0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 -- 172.21.15.158:0/3881278199 >> 172.21.15.158:0/3881278199 conn(0x7fe87806a8d0 msgr2=0x7fe87806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:35.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 -- 172.21.15.158:0/3881278199 shutdown_connections 2023-10-16T11:20:35.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.636+0000 7fe87cd17700 1 -- 172.21.15.158:0/3881278199 wait complete. 2023-10-16T11:20:35.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.637+0000 7fe87cd17700 1 Processor -- start 2023-10-16T11:20:35.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.637+0000 7fe87cd17700 1 -- start start 2023-10-16T11:20:35.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87cd17700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780cbaf0 0x7fe8781f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87cd17700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780ce510 0x7fe87815e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87cd17700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe87815b590 0x7fe87815e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87cd17700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87814df60 con 0x7fe8780cbaf0 2023-10-16T11:20:35.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87cd17700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8780cb060 con 0x7fe87815b590 2023-10-16T11:20:35.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87cd17700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8780cad80 con 0x7fe8780ce510 2023-10-16T11:20:35.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780cbaf0 0x7fe8781f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780cbaf0 0x7fe8781f0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40540/0 (socket says 172.21.15.158:40540) 2023-10-16T11:20:35.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 -- 172.21.15.158:0/2373889512 learned_addr learned my addr 172.21.15.158:0/2373889512 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:35.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe875d9b700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780ce510 0x7fe87815e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 -- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780ce510 msgr2=0x7fe87815e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780ce510 0x7fe87815e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 -- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe87815b590 msgr2=0x7fe87815e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:35.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe876d9d700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe87815b590 0x7fe87815e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe87815b590 0x7fe87815e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:35.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 -- 172.21.15.158:0/2373889512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe868009550 con 0x7fe8780cbaf0 2023-10-16T11:20:35.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe875d9b700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780ce510 0x7fe87815e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:35.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.638+0000 7fe87659c700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780cbaf0 0x7fe8781f0640 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00c550 tx=0x7fe86c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:35.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.639+0000 7fe866ffd700 1 -- 172.21.15.158:0/2373889512 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c004c80 con 0x7fe8780cbaf0 2023-10-16T11:20:35.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.639+0000 7fe866ffd700 1 -- 172.21.15.158:0/2373889512 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c004e00 con 0x7fe8780cbaf0 2023-10-16T11:20:35.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.639+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87815ba80 con 0x7fe8780cbaf0 2023-10-16T11:20:35.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.639+0000 7fe866ffd700 1 -- 172.21.15.158:0/2373889512 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c007310 con 0x7fe8780cbaf0 2023-10-16T11:20:35.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.639+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87815c040 con 0x7fe8780cbaf0 2023-10-16T11:20:35.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.640+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8780681b0 con 0x7fe8780cbaf0 2023-10-16T11:20:35.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.640+0000 7fe866ffd700 1 -- 172.21.15.158:0/2373889512 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe86c007470 con 0x7fe8780cbaf0 2023-10-16T11:20:35.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.640+0000 7fe866ffd700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe85806b140 0x7fe85806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:35.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.641+0000 7fe866ffd700 1 -- 172.21.15.158:0/2373889512 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe86c089a20 con 0x7fe8780cbaf0 2023-10-16T11:20:35.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.641+0000 7fe875d9b700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe85806b140 0x7fe85806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:35.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.641+0000 7fe875d9b700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe85806b140 0x7fe85806d600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe87815dca0 tx=0x7fe868008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:35.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.645+0000 7fe866ffd700 1 -- 172.21.15.158:0/2373889512 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe86c054450 con 0x7fe8780cbaf0 2023-10-16T11:20:35.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:35.839+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe8780b07c0 con 0x7fe8780cbaf0 2023-10-16T11:20:36.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.798+0000 7fe866ffd700 1 -- 172.21.15.158:0/2373889512 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v238) v1 ==== 144+0+0 (secure 0 0 0) 0x7fe86c057a70 con 0x7fe8780cbaf0 2023-10-16T11:20:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe85806b140 msgr2=0x7fe85806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe85806b140 0x7fe85806d600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe87815dca0 tx=0x7fe868008030 comp rx=0 tx=0).stop 2023-10-16T11:20:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780cbaf0 msgr2=0x7fe8781f0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:36.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780cbaf0 0x7fe8781f0640 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00c550 tx=0x7fe86c00c910 comp rx=0 tx=0).stop 2023-10-16T11:20:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 shutdown_connections 2023-10-16T11:20:36.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe85806b140 0x7fe85806d600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe87815b590 0x7fe87815e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8780cbaf0 0x7fe8781f0640 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 --2- 172.21.15.158:0/2373889512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8780ce510 0x7fe87815e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.802+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 >> 172.21.15.158:0/2373889512 conn(0x7fe87806a8d0 msgr2=0x7fe8780b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.804+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 shutdown_connections 2023-10-16T11:20:36.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.805+0000 7fe87cd17700 1 -- 172.21.15.158:0/2373889512 wait complete. 2023-10-16T11:20:36.810 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_metadata' created 2023-10-16T11:20:36.824 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:20:36.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.956+0000 7f5880556700 1 Processor -- start 2023-10-16T11:20:36.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.956+0000 7f5880556700 1 -- start start 2023-10-16T11:20:36.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.957+0000 7f5880556700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 0x7f58780b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.957+0000 7f5880556700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f587806db30 0x7f58780b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.957+0000 7f5880556700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587815e190 con 0x7f5878072b60 2023-10-16T11:20:36.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.957+0000 7f5880556700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587815e470 con 0x7f587806db30 2023-10-16T11:20:36.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.957+0000 7f5880556700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587815e750 con 0x7f587815a780 2023-10-16T11:20:36.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587daf1700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5878072b60 0x7f587806d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40724/0 (socket says 172.21.15.158:40724) 2023-10-16T11:20:36.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587daf1700 1 -- 172.21.15.158:0/1991795345 learned_addr learned my addr 172.21.15.158:0/1991795345 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:36.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587eaf3700 1 --2- 172.21.15.158:0/1991795345 >> v2:172.21.15.172:3300/0 conn(0x7f587806db30 0x7f58780b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587e2f2700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 0x7f58780b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18168288 0 0) 0x7f587815e190 con 0x7f5878072b60 2023-10-16T11:20:36.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58640035b0 con 0x7f5878072b60 2023-10-16T11:20:36.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2425075727 0 0) 0x7f58640035b0 con 0x7f5878072b60 2023-10-16T11:20:36.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 msgr2=0x7f58780b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:36.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 0x7f58780b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 >> v2:172.21.15.172:3300/0 conn(0x7f587806db30 msgr2=0x7f58780b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:36.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 --2- 172.21.15.158:0/1991795345 >> v2:172.21.15.172:3300/0 conn(0x7f587806db30 0x7f58780b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587815ea30 con 0x7f5878072b60 2023-10-16T11:20:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f5874001670 con 0x7f5878072b60 2023-10-16T11:20:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 >> v1:172.21.15.158:6789/0 conn(0x7f5878072b60 legacy=0x7f587806d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 0x7f5864003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5864003f80 0x7f5864004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f5864005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587815e470 con 0x7f5864004d90 2023-10-16T11:20:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587815e750 con 0x7f5864003f80 2023-10-16T11:20:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586400ba40 con 0x7f587815a780 2023-10-16T11:20:36.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587eaf3700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f5864005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587e2f2700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 0x7f5864003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.958+0000 7f587daf1700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5864003f80 0x7f5864004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587eaf3700 1 -- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 msgr2=0x7f5864003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:36.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587eaf3700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f587815a780 0x7f5864003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587eaf3700 1 -- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5864003f80 msgr2=0x7f5864004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:36.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587eaf3700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5864003f80 0x7f5864004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587eaf3700 1 -- 172.21.15.158:0/1991795345 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58640035b0 con 0x7f5864004d90 2023-10-16T11:20:36.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587eaf3700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f5864005210 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f5870009fc0 tx=0x7f5864005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e923ec6cc8ce34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:36.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587000ec30 con 0x7f5864004d90 2023-10-16T11:20:36.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587000f040 con 0x7f5864004d90 2023-10-16T11:20:36.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587d2f0700 1 -- 172.21.15.158:0/1991795345 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870014480 con 0x7f5864004d90 2023-10-16T11:20:36.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f5880556700 1 -- 172.21.15.158:0/1991795345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 msgr2=0x7f5864005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:36.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f5880556700 1 --2- 172.21.15.158:0/1991795345 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f5864005210 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f5870009fc0 tx=0x7f5864005880 comp rx=0 tx=0).stop 2023-10-16T11:20:36.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.959+0000 7f587eaf3700 1 -- 172.21.15.158:0/1991795345 reap_dead start 2023-10-16T11:20:36.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.960+0000 7f5880556700 1 -- 172.21.15.158:0/1991795345 shutdown_connections 2023-10-16T11:20:36.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.960+0000 7f5880556700 1 -- 172.21.15.158:0/1991795345 >> 172.21.15.158:0/1991795345 conn(0x7f58780ccd30 msgr2=0x7f58780cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:36.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.960+0000 7f5880556700 1 -- 172.21.15.158:0/1991795345 shutdown_connections 2023-10-16T11:20:36.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.960+0000 7f5880556700 1 -- 172.21.15.158:0/1991795345 wait complete. 2023-10-16T11:20:36.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 Processor -- start 2023-10-16T11:20:36.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 -- start start 2023-10-16T11:20:36.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f58780c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58781f2cc0 0x7f58780ce940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58780cee80 0x7f58781e7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587815b9c0 con 0x7f5864004d90 2023-10-16T11:20:36.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587815c170 con 0x7f58781f2cc0 2023-10-16T11:20:36.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.961+0000 7f5880556700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587815c450 con 0x7f58780cee80 2023-10-16T11:20:36.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f58780c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f58780c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40562/0 (socket says 172.21.15.158:40562) 2023-10-16T11:20:36.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 -- 172.21.15.158:0/4257828168 learned_addr learned my addr 172.21.15.158:0/4257828168 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:36.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587eaf3700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58780cee80 0x7f58781e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587daf1700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58781f2cc0 0x7f58780ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 -- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58780cee80 msgr2=0x7f58781e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:36.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58780cee80 0x7f58781e7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 -- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58781f2cc0 msgr2=0x7f58780ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:36.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58781f2cc0 0x7f58780ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:36.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 -- 172.21.15.158:0/4257828168 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587815c730 con 0x7f5864004d90 2023-10-16T11:20:36.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f587e2f2700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f58780c7380 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f5868011aa0 tx=0x7f5868011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:36.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f586effd700 1 -- 172.21.15.158:0/4257828168 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5868011ee0 con 0x7f5864004d90 2023-10-16T11:20:36.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f586effd700 1 -- 172.21.15.158:0/4257828168 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586800f940 con 0x7f5864004d90 2023-10-16T11:20:36.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f586effd700 1 -- 172.21.15.158:0/4257828168 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586800fcf0 con 0x7f5864004d90 2023-10-16T11:20:36.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587815ca10 con 0x7f5864004d90 2023-10-16T11:20:36.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.962+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58780cf4b0 con 0x7f5864004d90 2023-10-16T11:20:36.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.963+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587815cd80 con 0x7f5864004d90 2023-10-16T11:20:36.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.964+0000 7f586effd700 1 -- 172.21.15.158:0/4257828168 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5868014440 con 0x7f5864004d90 2023-10-16T11:20:36.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.964+0000 7f586effd700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f585c06b140 0x7f585c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:36.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.964+0000 7f586effd700 1 -- 172.21.15.158:0/4257828168 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6882+0+0 (secure 0 0 0) 0x7f586808a4d0 con 0x7f5864004d90 2023-10-16T11:20:36.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.964+0000 7f587daf1700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f585c06b140 0x7f585c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:36.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.964+0000 7f587daf1700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f585c06b140 0x7f585c06d600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f58781f6960 tx=0x7f5874007980 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:36.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:36.969+0000 7f586effd700 1 -- 172.21.15.158:0/4257828168 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5868054e40 con 0x7f5864004d90 2023-10-16T11:20:37.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.160+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f58780cf260 con 0x7f5864004d90 2023-10-16T11:20:37.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.800+0000 7f586effd700 1 -- 172.21.15.158:0/4257828168 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v239) v1 ==== 176+0+0 (secure 0 0 0) 0x7f5868058460 con 0x7f5864004d90 2023-10-16T11:20:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.809+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f585c06b140 msgr2=0x7f585c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.809+0000 7f5880556700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f585c06b140 0x7f585c06d600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f58781f6960 tx=0x7f5874007980 comp rx=0 tx=0).stop 2023-10-16T11:20:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.809+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 msgr2=0x7f58780c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:37.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.809+0000 7f5880556700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f58780c7380 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f5868011aa0 tx=0x7f5868011db0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.814+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 shutdown_connections 2023-10-16T11:20:37.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.814+0000 7f5880556700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f585c06b140 0x7f585c06d600 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.814+0000 7f5880556700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58780cee80 0x7f58781e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.814+0000 7f5880556700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5864004d90 0x7f58780c7380 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.814+0000 7f5880556700 1 --2- 172.21.15.158:0/4257828168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58781f2cc0 0x7f58780ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.814+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 >> 172.21.15.158:0/4257828168 conn(0x7f58780ccd30 msgr2=0x7f587806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:37.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.815+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 shutdown_connections 2023-10-16T11:20:37.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.816+0000 7f5880556700 1 -- 172.21.15.158:0/4257828168 wait complete. 2023-10-16T11:20:37.820 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_data' created 2023-10-16T11:20:37.835 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T11:20:37.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.959+0000 7f7e53f6b700 1 Processor -- start 2023-10-16T11:20:37.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.959+0000 7f7e53f6b700 1 -- start start 2023-10-16T11:20:37.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e53f6b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7e4c15d990 0x7f7e4c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:37.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e53f6b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7e4c070300 0x7f7e4c161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:37.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e53f6b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c15dd70 con 0x7f7e4c162010 2023-10-16T11:20:37.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e53f6b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c1616b0 con 0x7f7e4c070300 2023-10-16T11:20:37.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e53f6b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c0ced70 con 0x7f7e4c15d990 2023-10-16T11:20:37.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7e4c070300 0x7f7e4c161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:37.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7e4c070300 0x7f7e4c161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40570/0 (socket says 172.21.15.158:40570) 2023-10-16T11:20:37.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 -- 172.21.15.158:0/3503858832 learned_addr learned my addr 172.21.15.158:0/3503858832 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:37.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51d07700 1 --2- 172.21.15.158:0/3503858832 >> v2:172.21.15.172:3300/0 conn(0x7f7e4c15d990 0x7f7e4c06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:37.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 -- 172.21.15.158:0/3503858832 >> v2:172.21.15.172:3300/0 conn(0x7f7e4c15d990 msgr2=0x7f7e4c06d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:37.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 --2- 172.21.15.158:0/3503858832 >> v2:172.21.15.172:3300/0 conn(0x7f7e4c15d990 0x7f7e4c06d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 -- 172.21.15.158:0/3503858832 >> v1:172.21.15.172:6789/0 conn(0x7f7e4c162010 legacy=0x7f7e4c0cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:37.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 -- 172.21.15.158:0/3503858832 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c070780 con 0x7f7e4c070300 2023-10-16T11:20:37.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.960+0000 7f7e51506700 1 --2- 172.21.15.158:0/3503858832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c161ad0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0d25a0 tx=0x7f7e3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9c3731e169570f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:37.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e50d05700 1 -- 172.21.15.158:0/3503858832 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c004230 con 0x7f7e4c070300 2023-10-16T11:20:37.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e50d05700 1 -- 172.21.15.158:0/3503858832 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c004390 con 0x7f7e4c070300 2023-10-16T11:20:37.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e50d05700 1 -- 172.21.15.158:0/3503858832 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c0036a0 con 0x7f7e4c070300 2023-10-16T11:20:37.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 -- 172.21.15.158:0/3503858832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 msgr2=0x7f7e4c161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:37.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/3503858832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c161ad0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0d25a0 tx=0x7f7e3c009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:37.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 -- 172.21.15.158:0/3503858832 shutdown_connections 2023-10-16T11:20:37.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/3503858832 >> v2:172.21.15.172:3300/0 conn(0x7f7e4c15d990 0x7f7e4c06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/3503858832 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c161ad0 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 -- 172.21.15.158:0/3503858832 >> 172.21.15.158:0/3503858832 conn(0x7f7e4c0cdf20 msgr2=0x7f7e4c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:37.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 -- 172.21.15.158:0/3503858832 shutdown_connections 2023-10-16T11:20:37.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.961+0000 7f7e53f6b700 1 -- 172.21.15.158:0/3503858832 wait complete. 2023-10-16T11:20:37.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.962+0000 7f7e53f6b700 1 Processor -- start 2023-10-16T11:20:37.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.962+0000 7f7e53f6b700 1 -- start start 2023-10-16T11:20:37.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e53f6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:37.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e53f6b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e4c15d990 0x7f7e4c0c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:37.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e53f6b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e4c162010 0x7f7e4c0b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:37.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e53f6b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0b5ef0 con 0x7f7e4c070300 2023-10-16T11:20:37.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e53f6b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0c5ed0 con 0x7f7e4c162010 2023-10-16T11:20:37.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e53f6b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c158850 con 0x7f7e4c15d990 2023-10-16T11:20:37.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e51d07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:37.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e51d07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40574/0 (socket says 172.21.15.158:40574) 2023-10-16T11:20:37.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e51d07700 1 -- 172.21.15.158:0/2936929264 learned_addr learned my addr 172.21.15.158:0/2936929264 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:37.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e51506700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e4c15d990 0x7f7e4c0c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:37.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.963+0000 7f7e52508700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e4c162010 0x7f7e4c0b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:37.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e51d07700 1 -- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e4c15d990 msgr2=0x7f7e4c0c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:37.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e51d07700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e4c15d990 0x7f7e4c0c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e51d07700 1 -- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e4c162010 msgr2=0x7f7e4c0b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:37.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e51d07700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e4c162010 0x7f7e4c0b6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:37.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e51d07700 1 -- 172.21.15.158:0/2936929264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009550 con 0x7f7e4c070300 2023-10-16T11:20:37.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e51d07700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c158310 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800d800 tx=0x7f7e4800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:37.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e427fc700 1 -- 172.21.15.158:0/2936929264 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e480125f0 con 0x7f7e4c070300 2023-10-16T11:20:37.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e427fc700 1 -- 172.21.15.158:0/2936929264 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48012750 con 0x7f7e4c070300 2023-10-16T11:20:37.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c0c5910 con 0x7f7e4c070300 2023-10-16T11:20:37.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e427fc700 1 -- 172.21.15.158:0/2936929264 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48012b00 con 0x7f7e4c070300 2023-10-16T11:20:37.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.964+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c0b5b10 con 0x7f7e4c070300 2023-10-16T11:20:37.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.965+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4c0681b0 con 0x7f7e4c070300 2023-10-16T11:20:37.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.965+0000 7f7e427fc700 1 -- 172.21.15.158:0/2936929264 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7e48012c60 con 0x7f7e4c070300 2023-10-16T11:20:37.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.966+0000 7f7e427fc700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f7e3006b140 0x7f7e3006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:37.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.966+0000 7f7e51506700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f7e3006b140 0x7f7e3006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:37.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.966+0000 7f7e427fc700 1 -- 172.21.15.158:0/2936929264 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 7267+0+0 (secure 0 0 0) 0x7f7e4808a470 con 0x7f7e4c070300 2023-10-16T11:20:37.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.966+0000 7f7e51506700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f7e3006b140 0x7f7e3006d600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009870 tx=0x7f7e3c008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:37.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:37.971+0000 7f7e427fc700 1 -- 172.21.15.158:0/2936929264 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e48054b00 con 0x7f7e4c070300 2023-10-16T11:20:38.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.208+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f7e4c0b5ca0 con 0x7f7e4c070300 2023-10-16T11:20:38.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.809+0000 7f7e427fc700 1 -- 172.21.15.158:0/2936929264 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 33 and data pool 34 v131) v1 ==== 183+0+0 (secure 0 0 0) 0x7f7e48058120 con 0x7f7e4c070300 2023-10-16T11:20:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.815+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f7e3006b140 msgr2=0x7f7e3006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.815+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f7e3006b140 0x7f7e3006d600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009870 tx=0x7f7e3c008030 comp rx=0 tx=0).stop 2023-10-16T11:20:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.815+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 msgr2=0x7f7e4c158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:38.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.815+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c158310 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800d800 tx=0x7f7e4800dbc0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.816+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 shutdown_connections 2023-10-16T11:20:38.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.816+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f7e3006b140 0x7f7e3006d600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.816+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e4c162010 0x7f7e4c0b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.816+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e4c070300 0x7f7e4c158310 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.816+0000 7f7e53f6b700 1 --2- 172.21.15.158:0/2936929264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e4c15d990 0x7f7e4c0c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.816+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 >> 172.21.15.158:0/2936929264 conn(0x7f7e4c0cdf20 msgr2=0x7f7e4c15e010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:38.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.821+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 shutdown_connections 2023-10-16T11:20:38.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.823+0000 7f7e53f6b700 1 -- 172.21.15.158:0/2936929264 wait complete. 2023-10-16T11:20:38.828 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 33 and data pool 34 2023-10-16T11:20:38.851 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:38.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.971+0000 7f3e48411700 1 Processor -- start 2023-10-16T11:20:38.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.971+0000 7f3e48411700 1 -- start start 2023-10-16T11:20:38.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e48411700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 0x7f3e4006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e48411700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e40161270 con 0x7f3e4006e5d0 2023-10-16T11:20:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e48411700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e40161510 con 0x7f3e400729b0 2023-10-16T11:20:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e48411700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e401617f0 con 0x7f3e4006ab40 2023-10-16T11:20:38.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e459ac700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3e4006e5d0 0x7f3e400722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40758/0 (socket says 172.21.15.158:40758) 2023-10-16T11:20:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e459ac700 1 -- 172.21.15.158:0/1261878320 learned_addr learned my addr 172.21.15.158:0/1261878320 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e461ad700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 0x7f3e4006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137369123 0 0) 0x7f3e40161270 con 0x7f3e4006e5d0 2023-10-16T11:20:38.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c0035b0 con 0x7f3e4006e5d0 2023-10-16T11:20:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568111942 0 0) 0x7f3e2c0035b0 con 0x7f3e4006e5d0 2023-10-16T11:20:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 msgr2=0x7f3e4006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 0x7f3e4006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 >> v1:172.21.15.172:6789/0 conn(0x7f3e400729b0 legacy=0x7f3e4015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40161ad0 con 0x7f3e4006e5d0 2023-10-16T11:20:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3e30001670 con 0x7f3e4006e5d0 2023-10-16T11:20:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 >> v1:172.21.15.158:6789/0 conn(0x7f3e4006e5d0 legacy=0x7f3e400722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e2c003a30 0x7f3e2c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e2c004850 0x7f3e2c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 0x7f3e2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e401617f0 con 0x7f3e2c003a30 2023-10-16T11:20:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c00bac0 con 0x7f3e2c004850 2023-10-16T11:20:38.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c00bc80 con 0x7f3e4006ab40 2023-10-16T11:20:38.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.972+0000 7f3e461ad700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e2c003a30 0x7f3e2c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e459ac700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e2c004850 0x7f3e2c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e469ae700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 0x7f3e2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e461ad700 1 -- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 msgr2=0x7f3e2c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:38.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e461ad700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e4006ab40 0x7f3e2c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e461ad700 1 -- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e2c004850 msgr2=0x7f3e2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:38.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e461ad700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e2c004850 0x7f3e2c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e461ad700 1 -- 172.21.15.158:0/1261878320 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c0035b0 con 0x7f3e2c003a30 2023-10-16T11:20:38.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e461ad700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e2c003a30 0x7f3e2c003e50 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009b30 tx=0x7f3e2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8921c837ef7575 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:38.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00e6c0 con 0x7f3e2c003a30 2023-10-16T11:20:38.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c00e820 con 0x7f3e2c003a30 2023-10-16T11:20:38.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e451ab700 1 -- 172.21.15.158:0/1261878320 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00eb90 con 0x7f3e2c003a30 2023-10-16T11:20:38.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e48411700 1 -- 172.21.15.158:0/1261878320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e2c003a30 msgr2=0x7f3e2c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:38.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.973+0000 7f3e48411700 1 --2- 172.21.15.158:0/1261878320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e2c003a30 0x7f3e2c003e50 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009b30 tx=0x7f3e2c005880 comp rx=0 tx=0).stop 2023-10-16T11:20:38.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.974+0000 7f3e469ae700 1 -- 172.21.15.158:0/1261878320 reap_dead start 2023-10-16T11:20:38.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.974+0000 7f3e48411700 1 -- 172.21.15.158:0/1261878320 shutdown_connections 2023-10-16T11:20:38.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.974+0000 7f3e48411700 1 -- 172.21.15.158:0/1261878320 >> 172.21.15.158:0/1261878320 conn(0x7f3e400d02b0 msgr2=0x7f3e400d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:38.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.974+0000 7f3e48411700 1 -- 172.21.15.158:0/1261878320 shutdown_connections 2023-10-16T11:20:38.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.974+0000 7f3e48411700 1 -- 172.21.15.158:0/1261878320 wait complete. 2023-10-16T11:20:38.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.975+0000 7f3e48411700 1 Processor -- start 2023-10-16T11:20:38.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.975+0000 7f3e48411700 1 -- start start 2023-10-16T11:20:38.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e2c003a30 0x7f3e400b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e400b2e60 0x7f3e400b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e401f6420 0x7f3e40154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400c66d0 con 0x7f3e401f6420 2023-10-16T11:20:38.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400b3320 con 0x7f3e400b2e60 2023-10-16T11:20:38.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400b3600 con 0x7f3e2c003a30 2023-10-16T11:20:38.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e401f6420 0x7f3e40154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e401f6420 0x7f3e40154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40598/0 (socket says 172.21.15.158:40598) 2023-10-16T11:20:38.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 -- 172.21.15.158:0/3718707361 learned_addr learned my addr 172.21.15.158:0/3718707361 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:38.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e459ac700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e400b2e60 0x7f3e400b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e461ad700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e2c003a30 0x7f3e400b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 -- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e2c003a30 msgr2=0x7f3e400b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:38.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e2c003a30 0x7f3e400b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 -- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e400b2e60 msgr2=0x7f3e400b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:38.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e400b2e60 0x7f3e400b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:38.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 -- 172.21.15.158:0/3718707361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400b38e0 con 0x7f3e401f6420 2023-10-16T11:20:38.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e461ad700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e2c003a30 0x7f3e400b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:38.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e459ac700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e400b2e60 0x7f3e400b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:38.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e469ae700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e401f6420 0x7f3e40154790 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f3e38000f80 tx=0x7f3e3800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:38.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e36ffd700 1 -- 172.21.15.158:0/3718707361 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38003c10 con 0x7f3e401f6420 2023-10-16T11:20:38.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e36ffd700 1 -- 172.21.15.158:0/3718707361 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e38003d90 con 0x7f3e401f6420 2023-10-16T11:20:38.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e400b3bc0 con 0x7f3e401f6420 2023-10-16T11:20:38.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e36ffd700 1 -- 172.21.15.158:0/3718707361 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38005310 con 0x7f3e401f6420 2023-10-16T11:20:38.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.976+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e400b5fb0 con 0x7f3e401f6420 2023-10-16T11:20:38.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.977+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e400681b0 con 0x7f3e401f6420 2023-10-16T11:20:38.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.978+0000 7f3e36ffd700 1 -- 172.21.15.158:0/3718707361 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3e38005470 con 0x7f3e401f6420 2023-10-16T11:20:38.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.978+0000 7f3e36ffd700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e2406b140 0x7f3e2406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:38.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.978+0000 7f3e36ffd700 1 -- 172.21.15.158:0/3718707361 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f3e3808a130 con 0x7f3e401f6420 2023-10-16T11:20:38.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.978+0000 7f3e461ad700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e2406b140 0x7f3e2406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:38.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.979+0000 7f3e461ad700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e2406b140 0x7f3e2406d600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0090b0 tx=0x7f3e3c008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:38.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:38.983+0000 7f3e36ffd700 1 -- 172.21.15.158:0/3718707361 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e38054800 con 0x7f3e401f6420 2023-10-16T11:20:39.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.174+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e400cc680 con 0x7f3e401f6420 2023-10-16T11:20:39.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.176+0000 7f3e36ffd700 1 -- 172.21.15.158:0/3718707361 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+21204 (secure 0 0 0) 0x7f3e38057e20 con 0x7f3e401f6420 2023-10-16T11:20:39.178 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:39.178 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":240,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:38.209653+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:39.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e2406b140 msgr2=0x7f3e2406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e2406b140 0x7f3e2406d600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0090b0 tx=0x7f3e3c008030 comp rx=0 tx=0).stop 2023-10-16T11:20:39.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e401f6420 msgr2=0x7f3e40154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e401f6420 0x7f3e40154790 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f3e38000f80 tx=0x7f3e3800ee80 comp rx=0 tx=0).stop 2023-10-16T11:20:39.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 shutdown_connections 2023-10-16T11:20:39.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f3e2406b140 0x7f3e2406d600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e401f6420 0x7f3e40154790 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e2c003a30 0x7f3e400b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 --2- 172.21.15.158:0/3718707361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e400b2e60 0x7f3e400b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 >> 172.21.15.158:0/3718707361 conn(0x7f3e400d02b0 msgr2=0x7f3e40073c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:39.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 shutdown_connections 2023-10-16T11:20:39.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.178+0000 7f3e48411700 1 -- 172.21.15.158:0/3718707361 wait complete. 2023-10-16T11:20:39.192 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:39.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.310+0000 7f2831264700 1 Processor -- start 2023-10-16T11:20:39.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.311+0000 7f2831264700 1 -- start start 2023-10-16T11:20:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.311+0000 7f2831264700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c06ab40 0x7f282c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.311+0000 7f2831264700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f282c0729b0 0x7f282c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.311+0000 7f2831264700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c071e50 con 0x7f282c06e5d0 2023-10-16T11:20:39.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.311+0000 7f2831264700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282c161270 con 0x7f282c0729b0 2023-10-16T11:20:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.311+0000 7f2831264700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282c161510 con 0x7f282c06ab40 2023-10-16T11:20:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.311+0000 7f282b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f282c0729b0 0x7f282c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f282c0729b0 0x7f282c071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40604/0 (socket says 172.21.15.158:40604) 2023-10-16T11:20:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282b7fe700 1 -- 172.21.15.158:0/33935079 learned_addr learned my addr 172.21.15.158:0/33935079 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:39.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282affd700 1 --2- 172.21.15.158:0/33935079 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c06ab40 0x7f282c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282b7fe700 1 -- 172.21.15.158:0/33935079 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c06ab40 msgr2=0x7f282c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282b7fe700 1 --2- 172.21.15.158:0/33935079 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c06ab40 0x7f282c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282b7fe700 1 -- 172.21.15.158:0/33935079 >> v1:172.21.15.172:6789/0 conn(0x7f282c06e5d0 legacy=0x7f282c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282b7fe700 1 -- 172.21.15.158:0/33935079 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c1617f0 con 0x7f282c0729b0 2023-10-16T11:20:39.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f282b7fe700 1 --2- 172.21.15.158:0/33935079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c0729b0 0x7f282c071910 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f2824009e50 tx=0x7f282400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5772f0bc186b2aa4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f2829ffb700 1 -- 172.21.15.158:0/33935079 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282400e070 con 0x7f282c0729b0 2023-10-16T11:20:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f2829ffb700 1 -- 172.21.15.158:0/33935079 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282400b9c0 con 0x7f282c0729b0 2023-10-16T11:20:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.312+0000 7f2829ffb700 1 -- 172.21.15.158:0/33935079 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282400bd70 con 0x7f282c0729b0 2023-10-16T11:20:39.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 -- 172.21.15.158:0/33935079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c0729b0 msgr2=0x7f282c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 --2- 172.21.15.158:0/33935079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c0729b0 0x7f282c071910 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f2824009e50 tx=0x7f282400b330 comp rx=0 tx=0).stop 2023-10-16T11:20:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 -- 172.21.15.158:0/33935079 shutdown_connections 2023-10-16T11:20:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 --2- 172.21.15.158:0/33935079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c0729b0 0x7f282c071910 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 --2- 172.21.15.158:0/33935079 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c06ab40 0x7f282c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 -- 172.21.15.158:0/33935079 >> 172.21.15.158:0/33935079 conn(0x7f282c0d02b0 msgr2=0x7f282c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:39.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 -- 172.21.15.158:0/33935079 shutdown_connections 2023-10-16T11:20:39.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.313+0000 7f2831264700 1 -- 172.21.15.158:0/33935079 wait complete. 2023-10-16T11:20:39.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.314+0000 7f2831264700 1 Processor -- start 2023-10-16T11:20:39.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.314+0000 7f2831264700 1 -- start start 2023-10-16T11:20:39.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f2831264700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c06ab40 0x7f282c1f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f2831264700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f282c06e5d0 0x7f282c0c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f2831264700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c0729b0 0x7f282c0b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f2831264700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0b5210 con 0x7f282c06ab40 2023-10-16T11:20:39.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f2831264700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c1f3950 con 0x7f282c06e5d0 2023-10-16T11:20:39.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f2831264700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282c1f3c30 con 0x7f282c0729b0 2023-10-16T11:20:39.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f282affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c06ab40 0x7f282c1f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f282affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c06ab40 0x7f282c1f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40606/0 (socket says 172.21.15.158:40606) 2023-10-16T11:20:39.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f282affd700 1 -- 172.21.15.158:0/1626560983 learned_addr learned my addr 172.21.15.158:0/1626560983 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:39.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f282a7fc700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f282c06e5d0 0x7f282c0c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.315+0000 7f282b7fe700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c0729b0 0x7f282c0b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f282affd700 1 -- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c0729b0 msgr2=0x7f282c0b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f282affd700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c0729b0 0x7f282c0b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f282affd700 1 -- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f282c06e5d0 msgr2=0x7f282c0c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f282affd700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f282c06e5d0 0x7f282c0c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f282affd700 1 -- 172.21.15.158:0/1626560983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824009b30 con 0x7f282c06ab40 2023-10-16T11:20:39.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f282a7fc700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f282c06e5d0 0x7f282c0c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:20:39.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f282affd700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c06ab40 0x7f282c1f3040 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7f282000e810 tx=0x7f282000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:39.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f28137fe700 1 -- 172.21.15.158:0/1626560983 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820013610 con 0x7f282c06ab40 2023-10-16T11:20:39.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f28137fe700 1 -- 172.21.15.158:0/1626560983 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820013770 con 0x7f282c06ab40 2023-10-16T11:20:39.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282c1f3f10 con 0x7f282c06ab40 2023-10-16T11:20:39.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f28137fe700 1 -- 172.21.15.158:0/1626560983 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820013b20 con 0x7f282c06ab40 2023-10-16T11:20:39.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.316+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282c1f3580 con 0x7f282c06ab40 2023-10-16T11:20:39.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.317+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282c0b4fc0 con 0x7f282c06ab40 2023-10-16T11:20:39.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.317+0000 7f28137fe700 1 -- 172.21.15.158:0/1626560983 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2820013c80 con 0x7f282c06ab40 2023-10-16T11:20:39.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.318+0000 7f28137fe700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f281806b140 0x7f281806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.318+0000 7f28137fe700 1 -- 172.21.15.158:0/1626560983 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f2820089f00 con 0x7f282c06ab40 2023-10-16T11:20:39.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.318+0000 7f282a7fc700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f281806b140 0x7f281806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.318+0000 7f282a7fc700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f281806b140 0x7f281806d600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f282c0b2540 tx=0x7f281c009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:39.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.322+0000 7f28137fe700 1 -- 172.21.15.158:0/1626560983 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f28200544d0 con 0x7f282c06ab40 2023-10-16T11:20:39.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.517+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f282c0681b0 con 0x7f282c06ab40 2023-10-16T11:20:39.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.518+0000 7f28137fe700 1 -- 172.21.15.158:0/1626560983 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+21204 (secure 0 0 0) 0x7f2820057af0 con 0x7f282c06ab40 2023-10-16T11:20:39.520 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:39.521 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":240,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:38.209653+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:39.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.520+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f281806b140 msgr2=0x7f281806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.520+0000 7f2831264700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f281806b140 0x7f281806d600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f282c0b2540 tx=0x7f281c009040 comp rx=0 tx=0).stop 2023-10-16T11:20:39.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.520+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c06ab40 msgr2=0x7f282c1f3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.520+0000 7f2831264700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c06ab40 0x7f282c1f3040 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7f282000e810 tx=0x7f282000ebd0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.520+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 shutdown_connections 2023-10-16T11:20:39.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.520+0000 7f2831264700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f281806b140 0x7f281806d600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.521+0000 7f2831264700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f282c0729b0 0x7f282c0b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.521+0000 7f2831264700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f282c06ab40 0x7f282c1f3040 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.521+0000 7f2831264700 1 --2- 172.21.15.158:0/1626560983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f282c06e5d0 0x7f282c0c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.521+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 >> 172.21.15.158:0/1626560983 conn(0x7f282c0d02b0 msgr2=0x7f282c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:39.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.521+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 shutdown_connections 2023-10-16T11:20:39.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.521+0000 7f2831264700 1 -- 172.21.15.158:0/1626560983 wait complete. 2023-10-16T11:20:39.535 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T11:20:39.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.658+0000 7f25899fd700 1 Processor -- start 2023-10-16T11:20:39.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.658+0000 7f25899fd700 1 -- start start 2023-10-16T11:20:39.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f25899fd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f258415b590 0x7f25840ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f25899fd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25840cf4c0 con 0x7f25840cbaf0 2023-10-16T11:20:39.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f25899fd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258415f970 con 0x7f258415f1d0 2023-10-16T11:20:39.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f25899fd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25840b49e0 con 0x7f258415b590 2023-10-16T11:20:39.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f2582ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f258415b590 0x7f25840ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f2582ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f258415b590 0x7f25840ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40614/0 (socket says 172.21.15.158:40614) 2023-10-16T11:20:39.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f25837fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f25840cbaf0 0x7f25840cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40794/0 (socket says 172.21.15.158:40794) 2023-10-16T11:20:39.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.659+0000 7f2582ffd700 1 -- 172.21.15.158:0/1469902456 learned_addr learned my addr 172.21.15.158:0/1469902456 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:39.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2582ffd700 1 -- 172.21.15.158:0/1469902456 >> v1:172.21.15.172:6789/0 conn(0x7f258415f1d0 legacy=0x7f25840b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:39.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2582ffd700 1 -- 172.21.15.158:0/1469902456 >> v1:172.21.15.158:6789/0 conn(0x7f25840cbaf0 legacy=0x7f25840cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2582ffd700 1 -- 172.21.15.158:0/1469902456 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258415fc50 con 0x7f258415b590 2023-10-16T11:20:39.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2581ffb700 1 -- 172.21.15.158:0/1469902456 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236857872 0 0) 0x7f25840cf4c0 con 0x7f25840cbaf0 2023-10-16T11:20:39.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2582ffd700 1 --2- 172.21.15.158:0/1469902456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415b590 0x7f25840ce510 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f25780098f0 tx=0x7f2578009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558503f3c97a5eda server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:39.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2581ffb700 1 -- 172.21.15.158:0/1469902456 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578004230 con 0x7f258415b590 2023-10-16T11:20:39.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2581ffb700 1 -- 172.21.15.158:0/1469902456 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578004390 con 0x7f258415b590 2023-10-16T11:20:39.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f2581ffb700 1 -- 172.21.15.158:0/1469902456 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25780036a0 con 0x7f258415b590 2023-10-16T11:20:39.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f25899fd700 1 -- 172.21.15.158:0/1469902456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415b590 msgr2=0x7f25840ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f25899fd700 1 --2- 172.21.15.158:0/1469902456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415b590 0x7f25840ce510 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f25780098f0 tx=0x7f2578009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:39.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f25899fd700 1 -- 172.21.15.158:0/1469902456 shutdown_connections 2023-10-16T11:20:39.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.660+0000 7f25899fd700 1 --2- 172.21.15.158:0/1469902456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415b590 0x7f25840ce510 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.661+0000 7f25899fd700 1 -- 172.21.15.158:0/1469902456 >> 172.21.15.158:0/1469902456 conn(0x7f258406a8d0 msgr2=0x7f258406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:39.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.661+0000 7f25899fd700 1 -- 172.21.15.158:0/1469902456 shutdown_connections 2023-10-16T11:20:39.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.661+0000 7f25899fd700 1 -- 172.21.15.158:0/1469902456 wait complete. 2023-10-16T11:20:39.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.662+0000 7f25899fd700 1 Processor -- start 2023-10-16T11:20:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.662+0000 7f25899fd700 1 -- start start 2023-10-16T11:20:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25840cbaf0 0x7f2584153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f258415b590 0x7f258406cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415f1d0 0x7f258406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258415ddc0 con 0x7f258415f1d0 2023-10-16T11:20:39.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2584154130 con 0x7f258415b590 2023-10-16T11:20:39.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258415dfd0 con 0x7f25840cbaf0 2023-10-16T11:20:39.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415f1d0 0x7f258406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415f1d0 0x7f258406d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40624/0 (socket says 172.21.15.158:40624) 2023-10-16T11:20:39.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 -- 172.21.15.158:0/2850824746 learned_addr learned my addr 172.21.15.158:0/2850824746 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:39.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f2582ffd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25840cbaf0 0x7f2584153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25827fc700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f258415b590 0x7f258406cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 -- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25840cbaf0 msgr2=0x7f2584153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25840cbaf0 0x7f2584153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 -- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f258415b590 msgr2=0x7f258406cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f258415b590 0x7f258406cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 -- 172.21.15.158:0/2850824746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578009550 con 0x7f258415f1d0 2023-10-16T11:20:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25837fe700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415f1d0 0x7f258406d500 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f257c00cb60 tx=0x7f257c00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f256bfff700 1 -- 172.21.15.158:0/2850824746 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c00a9e0 con 0x7f258415f1d0 2023-10-16T11:20:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f256bfff700 1 -- 172.21.15.158:0/2850824746 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c00ab40 con 0x7f258415f1d0 2023-10-16T11:20:39.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f256bfff700 1 -- 172.21.15.158:0/2850824746 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c0070d0 con 0x7f258415f1d0 2023-10-16T11:20:39.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258415e2b0 con 0x7f258415f1d0 2023-10-16T11:20:39.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.663+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258415e870 con 0x7f258415f1d0 2023-10-16T11:20:39.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.664+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258415f630 con 0x7f258415f1d0 2023-10-16T11:20:39.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.665+0000 7f256bfff700 1 -- 172.21.15.158:0/2850824746 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f257c007230 con 0x7f258415f1d0 2023-10-16T11:20:39.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.666+0000 7f256bfff700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f257006b140 0x7f257006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:39.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.666+0000 7f256bfff700 1 -- 172.21.15.158:0/2850824746 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f257c089d60 con 0x7f258415f1d0 2023-10-16T11:20:39.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.666+0000 7f2582ffd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f257006b140 0x7f257006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:39.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.666+0000 7f2582ffd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f257006b140 0x7f257006d600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2578009130 tx=0x7f2578008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:39.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.669+0000 7f256bfff700 1 -- 172.21.15.158:0/2850824746 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f257c054410 con 0x7f258415f1d0 2023-10-16T11:20:39.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:39.911+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f258415f630 con 0x7f258415f1d0 2023-10-16T11:20:40.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.838+0000 7f256bfff700 1 -- 172.21.15.158:0/2850824746 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v133) v1 ==== 121+0+0 (secure 0 0 0) 0x7f257c018070 con 0x7f258415f1d0 2023-10-16T11:20:40.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.842+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f257006b140 msgr2=0x7f257006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:40.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.842+0000 7f25899fd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f257006b140 0x7f257006d600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2578009130 tx=0x7f2578008030 comp rx=0 tx=0).stop 2023-10-16T11:20:40.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.842+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415f1d0 msgr2=0x7f258406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:40.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.842+0000 7f25899fd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415f1d0 0x7f258406d500 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f257c00cb60 tx=0x7f257c00ce70 comp rx=0 tx=0).stop 2023-10-16T11:20:40.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 shutdown_connections 2023-10-16T11:20:40.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f257006b140 0x7f257006d600 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:40.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f258415f1d0 0x7f258406d500 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:40.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25840cbaf0 0x7f2584153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:40.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 --2- 172.21.15.158:0/2850824746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f258415b590 0x7f258406cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:40.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 >> 172.21.15.158:0/2850824746 conn(0x7f258406a8d0 msgr2=0x7f25840b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:40.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 shutdown_connections 2023-10-16T11:20:40.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.843+0000 7f25899fd700 1 -- 172.21.15.158:0/2850824746 wait complete. 2023-10-16T11:20:40.860 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:20:40.860 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T11:20:40.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.987+0000 7f354ffc5700 1 Processor -- start 2023-10-16T11:20:40.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.988+0000 7f354ffc5700 1 -- start start 2023-10-16T11:20:40.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.988+0000 7f354ffc5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f354806d5d0 0x7f3548070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:40.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.988+0000 7f354ffc5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354815dd70 con 0x7f35480cbaf0 2023-10-16T11:20:40.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.988+0000 7f354ffc5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3548070a00 con 0x7f354815d990 2023-10-16T11:20:40.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354ffc5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35481616b0 con 0x7f354806d5d0 2023-10-16T11:20:40.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354d560700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f354806d5d0 0x7f3548070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:40.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354e562700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f35480cbaf0 0x7f35480b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40808/0 (socket says 172.21.15.158:40808) 2023-10-16T11:20:40.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354e562700 1 -- 172.21.15.158:0/3107302425 learned_addr learned my addr 172.21.15.158:0/3107302425 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:40.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354d560700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f354806d5d0 0x7f3548070300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40630/0 (socket says 172.21.15.158:40630) 2023-10-16T11:20:40.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354d560700 1 -- 172.21.15.158:0/3107302425 >> v1:172.21.15.172:6789/0 conn(0x7f354815d990 legacy=0x7f3548161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:40.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354cd5f700 1 -- 172.21.15.158:0/3107302425 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255055110 0 0) 0x7f3540001490 con 0x7f35480cbaf0 2023-10-16T11:20:40.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354d560700 1 -- 172.21.15.158:0/3107302425 >> v1:172.21.15.158:6789/0 conn(0x7f35480cbaf0 legacy=0x7f35480b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:40.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354d560700 1 -- 172.21.15.158:0/3107302425 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548070ce0 con 0x7f354806d5d0 2023-10-16T11:20:40.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354d560700 1 --2- 172.21.15.158:0/3107302425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354806d5d0 0x7f3548070300 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f3544009840 tx=0x7f3544009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b850aa108e5d36b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:40.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354cd5f700 1 -- 172.21.15.158:0/3107302425 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544011830 con 0x7f354806d5d0 2023-10-16T11:20:40.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354cd5f700 1 -- 172.21.15.158:0/3107302425 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3544011990 con 0x7f354806d5d0 2023-10-16T11:20:40.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.989+0000 7f354cd5f700 1 -- 172.21.15.158:0/3107302425 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544011d20 con 0x7f354806d5d0 2023-10-16T11:20:40.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.990+0000 7f354ffc5700 1 -- 172.21.15.158:0/3107302425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354806d5d0 msgr2=0x7f3548070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:40.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.990+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3107302425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354806d5d0 0x7f3548070300 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f3544009840 tx=0x7f3544009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:40.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.990+0000 7f354ffc5700 1 -- 172.21.15.158:0/3107302425 shutdown_connections 2023-10-16T11:20:40.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.990+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3107302425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354806d5d0 0x7f3548070300 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:40.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.990+0000 7f354ffc5700 1 -- 172.21.15.158:0/3107302425 >> 172.21.15.158:0/3107302425 conn(0x7f35480cdf20 msgr2=0x7f35480ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:40.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.990+0000 7f354ffc5700 1 -- 172.21.15.158:0/3107302425 shutdown_connections 2023-10-16T11:20:40.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.990+0000 7f354ffc5700 1 -- 172.21.15.158:0/3107302425 wait complete. 2023-10-16T11:20:40.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.991+0000 7f354ffc5700 1 Processor -- start 2023-10-16T11:20:40.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.991+0000 7f354ffc5700 1 -- start start 2023-10-16T11:20:40.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354ffc5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f354806d5d0 0x7f35480b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:40.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354ffc5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35480cbaf0 0x7f35480ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:41.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354ffc5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354815d990 0x7f35480cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:41.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354ffc5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35480cb690 con 0x7f354815d990 2023-10-16T11:20:41.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354ffc5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354814b900 con 0x7f35480cbaf0 2023-10-16T11:20:41.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354ffc5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35480cb3b0 con 0x7f354806d5d0 2023-10-16T11:20:41.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354815d990 0x7f35480cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:41.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354815d990 0x7f35480cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40638/0 (socket says 172.21.15.158:40638) 2023-10-16T11:20:41.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 -- 172.21.15.158:0/3624682953 learned_addr learned my addr 172.21.15.158:0/3624682953 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:41.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354d560700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35480cbaf0 0x7f35480ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:41.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354dd61700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f354806d5d0 0x7f35480b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:41.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 -- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f354806d5d0 msgr2=0x7f35480b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:41.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f354806d5d0 0x7f35480b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:41.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 -- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35480cbaf0 msgr2=0x7f35480ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:41.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35480cbaf0 0x7f35480ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:41.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 -- 172.21.15.158:0/3624682953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35440094a0 con 0x7f354815d990 2023-10-16T11:20:41.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354e562700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354815d990 0x7f35480cab90 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f354000cca0 tx=0x7f354000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:41.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f353e7fc700 1 -- 172.21.15.158:0/3624682953 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354000abc0 con 0x7f354815d990 2023-10-16T11:20:41.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f353e7fc700 1 -- 172.21.15.158:0/3624682953 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354000ad40 con 0x7f354815d990 2023-10-16T11:20:41.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35480c5ff0 con 0x7f354815d990 2023-10-16T11:20:41.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.992+0000 7f353e7fc700 1 -- 172.21.15.158:0/3624682953 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540007310 con 0x7f354815d990 2023-10-16T11:20:41.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.993+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35480c65b0 con 0x7f354815d990 2023-10-16T11:20:41.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.993+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35480c67a0 con 0x7f354815d990 2023-10-16T11:20:41.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.994+0000 7f353e7fc700 1 -- 172.21.15.158:0/3624682953 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3540007470 con 0x7f354815d990 2023-10-16T11:20:41.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.994+0000 7f353e7fc700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f352c06b140 0x7f352c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:41.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.994+0000 7f353e7fc700 1 -- 172.21.15.158:0/3624682953 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f3540089d70 con 0x7f354815d990 2023-10-16T11:20:41.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.994+0000 7f354dd61700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f352c06b140 0x7f352c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:41.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.995+0000 7f354dd61700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f352c06b140 0x7f352c06d600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3538000f60 tx=0x7f3538009040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:41.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:40.999+0000 7f353e7fc700 1 -- 172.21.15.158:0/3624682953 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3540054430 con 0x7f354815d990 2023-10-16T11:20:41.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.236+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f354815dd70 con 0x7f354815d990 2023-10-16T11:20:41.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.851+0000 7f353e7fc700 1 -- 172.21.15.158:0/3624682953 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v134) v1 ==== 125+0+0 (secure 0 0 0) 0x7f354815dd70 con 0x7f354815d990 2023-10-16T11:20:41.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.853+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f352c06b140 msgr2=0x7f352c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.853+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f352c06b140 0x7f352c06d600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3538000f60 tx=0x7f3538009040 comp rx=0 tx=0).stop 2023-10-16T11:20:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.853+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354815d990 msgr2=0x7f35480cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354815d990 0x7f35480cab90 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f354000cca0 tx=0x7f354000a060 comp rx=0 tx=0).stop 2023-10-16T11:20:41.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 shutdown_connections 2023-10-16T11:20:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f352c06b140 0x7f352c06d600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f354815d990 0x7f35480cab90 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f354806d5d0 0x7f35480b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 --2- 172.21.15.158:0/3624682953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35480cbaf0 0x7f35480ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:41.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 >> 172.21.15.158:0/3624682953 conn(0x7f35480cdf20 msgr2=0x7f35480ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 shutdown_connections 2023-10-16T11:20:41.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.854+0000 7f354ffc5700 1 -- 172.21.15.158:0/3624682953 wait complete. 2023-10-16T11:20:41.870 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T11:20:41.871 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:41.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 Processor -- start 2023-10-16T11:20:41.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 -- start start 2023-10-16T11:20:41.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:41.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5c8c06e5d0 0x7f5c8c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5c8c06af20 0x7f5c8c0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c071e50 con 0x7f5c8c06e5d0 2023-10-16T11:20:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c161270 con 0x7f5c8c06af20 2023-10-16T11:20:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.995+0000 7f5c91dc1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c161510 con 0x7f5c8c06ab40 2023-10-16T11:20:42.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5c8c06e5d0 0x7f5c8c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5c8c06e5d0 0x7f5c8c071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40642/0 (socket says 172.21.15.158:40642) 2023-10-16T11:20:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 -- 172.21.15.158:0/3941012493 learned_addr learned my addr 172.21.15.158:0/3941012493 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 -- 172.21.15.158:0/3941012493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 msgr2=0x7f5c8c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8bfff700 1 --2- 172.21.15.158:0/3941012493 >> v2:172.21.15.172:3300/0 conn(0x7f5c8c06af20 0x7f5c8c0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8b7fe700 1 --2- 172.21.15.158:0/3941012493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 --2- 172.21.15.158:0/3941012493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 -- 172.21.15.158:0/3941012493 >> v2:172.21.15.172:3300/0 conn(0x7f5c8c06af20 msgr2=0x7f5c8c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 --2- 172.21.15.158:0/3941012493 >> v2:172.21.15.172:3300/0 conn(0x7f5c8c06af20 0x7f5c8c0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 -- 172.21.15.158:0/3941012493 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c1617f0 con 0x7f5c8c06e5d0 2023-10-16T11:20:42.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8bfff700 1 --2- 172.21.15.158:0/3941012493 >> v2:172.21.15.172:3300/0 conn(0x7f5c8c06af20 0x7f5c8c0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8b7fe700 1 --2- 172.21.15.158:0/3941012493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8affd700 1 --2- 172.21.15.158:0/3941012493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06e5d0 0x7f5c8c071910 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06eaf0 tx=0x7f5c74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6785dc66ec7c1d35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8a7fc700 1 -- 172.21.15.158:0/3941012493 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400b870 con 0x7f5c8c06e5d0 2023-10-16T11:20:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8a7fc700 1 -- 172.21.15.158:0/3941012493 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7400b9d0 con 0x7f5c8c06e5d0 2023-10-16T11:20:42.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.996+0000 7f5c8a7fc700 1 -- 172.21.15.158:0/3941012493 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400bd60 con 0x7f5c8c06e5d0 2023-10-16T11:20:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 -- 172.21.15.158:0/3941012493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06e5d0 msgr2=0x7f5c8c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/3941012493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06e5d0 0x7f5c8c071910 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06eaf0 tx=0x7f5c74009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 -- 172.21.15.158:0/3941012493 shutdown_connections 2023-10-16T11:20:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/3941012493 >> v2:172.21.15.172:3300/0 conn(0x7f5c8c06af20 0x7f5c8c0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/3941012493 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/3941012493 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06e5d0 0x7f5c8c071910 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 -- 172.21.15.158:0/3941012493 >> 172.21.15.158:0/3941012493 conn(0x7f5c8c0d02b0 msgr2=0x7f5c8c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:42.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 -- 172.21.15.158:0/3941012493 shutdown_connections 2023-10-16T11:20:42.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.997+0000 7f5c91dc1700 1 -- 172.21.15.158:0/3941012493 wait complete. 2023-10-16T11:20:42.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.998+0000 7f5c91dc1700 1 Processor -- start 2023-10-16T11:20:42.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.998+0000 7f5c91dc1700 1 -- start start 2023-10-16T11:20:42.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c91dc1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c0c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c91dc1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06af20 0x7f5c8c0b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c91dc1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c8c06e5d0 0x7f5c8c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c91dc1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0b0e50 con 0x7f5c8c06af20 2023-10-16T11:20:42.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c91dc1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0c3c80 con 0x7f5c8c06e5d0 2023-10-16T11:20:42.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c91dc1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0c3f20 con 0x7f5c8c06ab40 2023-10-16T11:20:42.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c8affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06af20 0x7f5c8c0b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c8affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06af20 0x7f5c8c0b1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40650/0 (socket says 172.21.15.158:40650) 2023-10-16T11:20:42.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:41.999+0000 7f5c8affd700 1 -- 172.21.15.158:0/1032192884 learned_addr learned my addr 172.21.15.158:0/1032192884 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:42.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c8affd700 1 -- 172.21.15.158:0/1032192884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 msgr2=0x7f5c8c0c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c8affd700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c0c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c8affd700 1 -- 172.21.15.158:0/1032192884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c8c06e5d0 msgr2=0x7f5c8c0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c8bfff700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c8c06e5d0 0x7f5c8c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c8affd700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c8c06e5d0 0x7f5c8c0b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c8affd700 1 -- 172.21.15.158:0/1032192884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c74009550 con 0x7f5c8c06af20 2023-10-16T11:20:42.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c8affd700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06af20 0x7f5c8c0b1d10 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f5c74014040 tx=0x7f5c74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c83fff700 1 -- 172.21.15.158:0/1032192884 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74004520 con 0x7f5c8c06af20 2023-10-16T11:20:42.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c83fff700 1 -- 172.21.15.158:0/1032192884 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7401e070 con 0x7f5c8c06af20 2023-10-16T11:20:42.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c0c4200 con 0x7f5c8c06af20 2023-10-16T11:20:42.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c83fff700 1 -- 172.21.15.158:0/1032192884 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c740038e0 con 0x7f5c8c06af20 2023-10-16T11:20:42.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.000+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c0c47c0 con 0x7f5c8c06af20 2023-10-16T11:20:42.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.001+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c0681b0 con 0x7f5c8c06af20 2023-10-16T11:20:42.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.002+0000 7f5c83fff700 1 -- 172.21.15.158:0/1032192884 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5c74003a40 con 0x7f5c8c06af20 2023-10-16T11:20:42.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.002+0000 7f5c83fff700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5c6c06b140 0x7f5c6c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.002+0000 7f5c83fff700 1 -- 172.21.15.158:0/1032192884 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f5c74020070 con 0x7f5c8c06af20 2023-10-16T11:20:42.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.002+0000 7f5c8b7fe700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5c6c06b140 0x7f5c6c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.003+0000 7f5c8b7fe700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5c6c06b140 0x7f5c6c06d600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c007920 tx=0x7f5c7c008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.007+0000 7f5c83fff700 1 -- 172.21.15.158:0/1032192884 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5c74055d80 con 0x7f5c8c06af20 2023-10-16T11:20:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.243+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c8c161aa0 con 0x7f5c8c06af20 2023-10-16T11:20:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.243+0000 7f5c83fff700 1 -- 172.21.15.158:0/1032192884 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6160 (secure 0 0 0) 0x7f5c7402ee60 con 0x7f5c8c06af20 2023-10-16T11:20:42.245 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:42.246 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:20:42.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5c6c06b140 msgr2=0x7f5c6c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5c6c06b140 0x7f5c6c06d600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c007920 tx=0x7f5c7c008040 comp rx=0 tx=0).stop 2023-10-16T11:20:42.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06af20 msgr2=0x7f5c8c0b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06af20 0x7f5c8c0b1d10 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f5c74014040 tx=0x7f5c74004750 comp rx=0 tx=0).stop 2023-10-16T11:20:42.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 shutdown_connections 2023-10-16T11:20:42.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5c6c06b140 0x7f5c6c06d600 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c8c06e5d0 0x7f5c8c0b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c8c06ab40 0x7f5c8c0c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 --2- 172.21.15.158:0/1032192884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c8c06af20 0x7f5c8c0b1d10 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.245+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 >> 172.21.15.158:0/1032192884 conn(0x7f5c8c0d02b0 msgr2=0x7f5c8c0ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:42.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.246+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 shutdown_connections 2023-10-16T11:20:42.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.246+0000 7f5c91dc1700 1 -- 172.21.15.158:0/1032192884 wait complete. 2023-10-16T11:20:42.250 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 134 2023-10-16T11:20:42.259 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:42.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.379+0000 7f2269e1a700 1 Processor -- start 2023-10-16T11:20:42.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.380+0000 7f2269e1a700 1 -- start start 2023-10-16T11:20:42.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.380+0000 7f2269e1a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f226406ab40 0x7f226406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.380+0000 7f2269e1a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f226406e5d0 0x7f2264071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.380+0000 7f2269e1a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406af20 0x7f22640722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.380+0000 7f2269e1a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2264071e50 con 0x7f226406e5d0 2023-10-16T11:20:42.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.380+0000 7f2269e1a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2264161270 con 0x7f226406ab40 2023-10-16T11:20:42.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.380+0000 7f2269e1a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2264161510 con 0x7f226406af20 2023-10-16T11:20:42.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f226406e5d0 0x7f2264071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f226406e5d0 0x7f2264071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40656/0 (socket says 172.21.15.158:40656) 2023-10-16T11:20:42.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 -- 172.21.15.158:0/2911296991 learned_addr learned my addr 172.21.15.158:0/2911296991 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:42.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f22637fe700 1 --2- 172.21.15.158:0/2911296991 >> v2:172.21.15.172:3300/0 conn(0x7f226406ab40 0x7f226406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2263fff700 1 --2- 172.21.15.158:0/2911296991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406af20 0x7f22640722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 -- 172.21.15.158:0/2911296991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406af20 msgr2=0x7f22640722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:42.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 --2- 172.21.15.158:0/2911296991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406af20 0x7f22640722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 -- 172.21.15.158:0/2911296991 >> v2:172.21.15.172:3300/0 conn(0x7f226406ab40 msgr2=0x7f226406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:42.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 --2- 172.21.15.158:0/2911296991 >> v2:172.21.15.172:3300/0 conn(0x7f226406ab40 0x7f226406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 -- 172.21.15.158:0/2911296991 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22641617f0 con 0x7f226406e5d0 2023-10-16T11:20:42.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f2262ffd700 1 --2- 172.21.15.158:0/2911296991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406e5d0 0x7f2264071910 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f2264074d70 tx=0x7f224c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0ee9d1dbf7d747 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f22627fc700 1 -- 172.21.15.158:0/2911296991 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c011830 con 0x7f226406e5d0 2023-10-16T11:20:42.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f22627fc700 1 -- 172.21.15.158:0/2911296991 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c011990 con 0x7f226406e5d0 2023-10-16T11:20:42.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.381+0000 7f22627fc700 1 -- 172.21.15.158:0/2911296991 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c011d20 con 0x7f226406e5d0 2023-10-16T11:20:42.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 -- 172.21.15.158:0/2911296991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406e5d0 msgr2=0x7f2264071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 --2- 172.21.15.158:0/2911296991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406e5d0 0x7f2264071910 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f2264074d70 tx=0x7f224c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 -- 172.21.15.158:0/2911296991 shutdown_connections 2023-10-16T11:20:42.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 --2- 172.21.15.158:0/2911296991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406af20 0x7f22640722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 --2- 172.21.15.158:0/2911296991 >> v2:172.21.15.172:3300/0 conn(0x7f226406ab40 0x7f226406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 --2- 172.21.15.158:0/2911296991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406e5d0 0x7f2264071910 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 -- 172.21.15.158:0/2911296991 >> 172.21.15.158:0/2911296991 conn(0x7f22640d02b0 msgr2=0x7f22640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:42.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 -- 172.21.15.158:0/2911296991 shutdown_connections 2023-10-16T11:20:42.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.382+0000 7f2269e1a700 1 -- 172.21.15.158:0/2911296991 wait complete. 2023-10-16T11:20:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.383+0000 7f2269e1a700 1 Processor -- start 2023-10-16T11:20:42.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.383+0000 7f2269e1a700 1 -- start start 2023-10-16T11:20:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2269e1a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406ab40 0x7f2264155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2269e1a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406af20 0x7f22640b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2269e1a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f226406e5d0 0x7f22640b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2269e1a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22640b0a00 con 0x7f226406af20 2023-10-16T11:20:42.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2269e1a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22640b0ce0 con 0x7f226406e5d0 2023-10-16T11:20:42.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2269e1a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22640b0fc0 con 0x7f226406ab40 2023-10-16T11:20:42.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406af20 0x7f22640b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406af20 0x7f22640b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40662/0 (socket says 172.21.15.158:40662) 2023-10-16T11:20:42.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 -- 172.21.15.158:0/717114647 learned_addr learned my addr 172.21.15.158:0/717114647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:42.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2263fff700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f226406e5d0 0x7f22640b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f22637fe700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406ab40 0x7f2264155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 -- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406ab40 msgr2=0x7f2264155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406ab40 0x7f2264155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 -- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f226406e5d0 msgr2=0x7f22640b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f226406e5d0 0x7f22640b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.384+0000 7f2262ffd700 1 -- 172.21.15.158:0/717114647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c009550 con 0x7f226406af20 2023-10-16T11:20:42.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.385+0000 7f2262ffd700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406af20 0x7f22640b2410 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f224c009c10 tx=0x7f224c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.385+0000 7f225bfff700 1 -- 172.21.15.158:0/717114647 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c004510 con 0x7f226406af20 2023-10-16T11:20:42.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.385+0000 7f225bfff700 1 -- 172.21.15.158:0/717114647 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c01e070 con 0x7f226406af20 2023-10-16T11:20:42.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.385+0000 7f225bfff700 1 -- 172.21.15.158:0/717114647 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0038e0 con 0x7f226406af20 2023-10-16T11:20:42.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.385+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22640b12a0 con 0x7f226406af20 2023-10-16T11:20:42.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.385+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2264155a40 con 0x7f226406af20 2023-10-16T11:20:42.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.386+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22640b17c0 con 0x7f226406af20 2023-10-16T11:20:42.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.386+0000 7f225bfff700 1 -- 172.21.15.158:0/717114647 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f224c003a40 con 0x7f226406af20 2023-10-16T11:20:42.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.386+0000 7f225bfff700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f224406b140 0x7f224406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.387+0000 7f225bfff700 1 -- 172.21.15.158:0/717114647 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f224c08ad60 con 0x7f226406af20 2023-10-16T11:20:42.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.387+0000 7f22637fe700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f224406b140 0x7f224406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.387+0000 7f22637fe700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f224406b140 0x7f224406d600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2254009ab0 tx=0x7f2254008040 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.391+0000 7f225bfff700 1 -- 172.21.15.158:0/717114647 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22640b17c0 con 0x7f226406af20 2023-10-16T11:20:42.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.586+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22640b17c0 con 0x7f226406af20 2023-10-16T11:20:42.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.588+0000 7f225bfff700 1 -- 172.21.15.158:0/717114647 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) v1 ==== 74+0+21191 (secure 0 0 0) 0x7f22640b17c0 con 0x7f226406af20 2023-10-16T11:20:42.590 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:42.590 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":241,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:39.810562+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:42.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f224406b140 msgr2=0x7f224406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f224406b140 0x7f224406d600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2254009ab0 tx=0x7f2254008040 comp rx=0 tx=0).stop 2023-10-16T11:20:42.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406af20 msgr2=0x7f22640b2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406af20 0x7f22640b2410 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f224c009c10 tx=0x7f224c004750 comp rx=0 tx=0).stop 2023-10-16T11:20:42.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 shutdown_connections 2023-10-16T11:20:42.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f224406b140 0x7f224406d600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f226406e5d0 0x7f22640b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f226406ab40 0x7f2264155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 --2- 172.21.15.158:0/717114647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f226406af20 0x7f22640b2410 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 >> 172.21.15.158:0/717114647 conn(0x7f22640d02b0 msgr2=0x7f22641576b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:42.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 shutdown_connections 2023-10-16T11:20:42.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.590+0000 7f2269e1a700 1 -- 172.21.15.158:0/717114647 wait complete. 2023-10-16T11:20:42.605 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:20:42.605 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:20:42.711 INFO:tasks.ceph.mds.b.smithi172.stderr:2023-10-16T11:20:42.710+0000 7f5424b2b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:20:42.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.723+0000 7f22d2563700 1 Processor -- start 2023-10-16T11:20:42.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.723+0000 7f22d2563700 1 -- start start 2023-10-16T11:20:42.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22d2563700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22cc06ab40 0x7f22cc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22d2563700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f22cc0729b0 0x7f22cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22d2563700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc071e50 con 0x7f22cc06e5d0 2023-10-16T11:20:42.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22d2563700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22cc161270 con 0x7f22cc06ab40 2023-10-16T11:20:42.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22d2563700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22cc161510 con 0x7f22cc0729b0 2023-10-16T11:20:42.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22cbfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22cc06ab40 0x7f22cc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22cb7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f22cc06e5d0 0x7f22cc0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40844/0 (socket says 172.21.15.158:40844) 2023-10-16T11:20:42.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.724+0000 7f22cb7fe700 1 -- 172.21.15.158:0/4159946253 learned_addr learned my addr 172.21.15.158:0/4159946253 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:42.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22d0b00700 1 --2- 172.21.15.158:0/4159946253 >> v2:172.21.15.172:3300/0 conn(0x7f22cc0729b0 0x7f22cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22cbfff700 1 -- 172.21.15.158:0/4159946253 >> v2:172.21.15.172:3300/0 conn(0x7f22cc0729b0 msgr2=0x7f22cc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:42.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22cbfff700 1 --2- 172.21.15.158:0/4159946253 >> v2:172.21.15.172:3300/0 conn(0x7f22cc0729b0 0x7f22cc071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22cbfff700 1 -- 172.21.15.158:0/4159946253 >> v1:172.21.15.158:6789/0 conn(0x7f22cc06e5d0 legacy=0x7f22cc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22caffd700 1 -- 172.21.15.158:0/4159946253 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513927701 0 0) 0x7f22cc071e50 con 0x7f22cc06e5d0 2023-10-16T11:20:42.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22cbfff700 1 -- 172.21.15.158:0/4159946253 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc1617f0 con 0x7f22cc06ab40 2023-10-16T11:20:42.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22cbfff700 1 --2- 172.21.15.158:0/4159946253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc06ab40 0x7f22cc06d820 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f22cc158240 tx=0x7f22bc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5208e72a47754f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22caffd700 1 -- 172.21.15.158:0/4159946253 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc011830 con 0x7f22cc06ab40 2023-10-16T11:20:42.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22caffd700 1 -- 172.21.15.158:0/4159946253 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc011990 con 0x7f22cc06ab40 2023-10-16T11:20:42.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.725+0000 7f22caffd700 1 -- 172.21.15.158:0/4159946253 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc011d20 con 0x7f22cc06ab40 2023-10-16T11:20:42.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 -- 172.21.15.158:0/4159946253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc06ab40 msgr2=0x7f22cc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 --2- 172.21.15.158:0/4159946253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc06ab40 0x7f22cc06d820 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f22cc158240 tx=0x7f22bc009b20 comp rx=0 tx=0).stop 2023-10-16T11:20:42.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 -- 172.21.15.158:0/4159946253 shutdown_connections 2023-10-16T11:20:42.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 --2- 172.21.15.158:0/4159946253 >> v2:172.21.15.172:3300/0 conn(0x7f22cc0729b0 0x7f22cc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 --2- 172.21.15.158:0/4159946253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc06ab40 0x7f22cc06d820 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 -- 172.21.15.158:0/4159946253 >> 172.21.15.158:0/4159946253 conn(0x7f22cc0d02b0 msgr2=0x7f22cc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:42.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 -- 172.21.15.158:0/4159946253 shutdown_connections 2023-10-16T11:20:42.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.726+0000 7f22d2563700 1 -- 172.21.15.158:0/4159946253 wait complete. 2023-10-16T11:20:42.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.727+0000 7f22d2563700 1 Processor -- start 2023-10-16T11:20:42.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.727+0000 7f22d2563700 1 -- start start 2023-10-16T11:20:42.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d2563700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22cc06ab40 0x7f22cc14c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d2563700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22cc06e5d0 0x7f22cc15ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d2563700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc0729b0 0x7f22cc0c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d2563700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc0b3960 con 0x7f22cc0729b0 2023-10-16T11:20:42.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d2563700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc0b3c40 con 0x7f22cc06ab40 2023-10-16T11:20:42.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d2563700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc0b3f20 con 0x7f22cc06e5d0 2023-10-16T11:20:42.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc0729b0 0x7f22cc0c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc0729b0 0x7f22cc0c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40678/0 (socket says 172.21.15.158:40678) 2023-10-16T11:20:42.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 -- 172.21.15.158:0/3773335237 learned_addr learned my addr 172.21.15.158:0/3773335237 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:42.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22cbfff700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22cc06ab40 0x7f22cc14c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22cb7fe700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22cc06e5d0 0x7f22cc15ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 -- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22cc06e5d0 msgr2=0x7f22cc15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22cc06e5d0 0x7f22cc15ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 -- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22cc06ab40 msgr2=0x7f22cc14c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22cc06ab40 0x7f22cc14c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 -- 172.21.15.158:0/3773335237 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc0094a0 con 0x7f22cc0729b0 2023-10-16T11:20:42.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22cbfff700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22cc06ab40 0x7f22cc14c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:42.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22cb7fe700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22cc06e5d0 0x7f22cc15ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:42.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d0b00700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc0729b0 0x7f22cc0c3c40 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f22c400efc0 tx=0x7f22c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22c8ff9700 1 -- 172.21.15.158:0/3773335237 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c4009ca0 con 0x7f22cc0729b0 2023-10-16T11:20:42.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22c8ff9700 1 -- 172.21.15.158:0/3773335237 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c400f040 con 0x7f22cc0729b0 2023-10-16T11:20:42.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22cc0b4200 con 0x7f22cc0729b0 2023-10-16T11:20:42.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.728+0000 7f22c8ff9700 1 -- 172.21.15.158:0/3773335237 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c4014480 con 0x7f22cc0729b0 2023-10-16T11:20:42.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.729+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22cc0b3550 con 0x7f22cc0729b0 2023-10-16T11:20:42.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.730+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22cc0681b0 con 0x7f22cc0729b0 2023-10-16T11:20:42.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.730+0000 7f22c8ff9700 1 -- 172.21.15.158:0/3773335237 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f22c40145e0 con 0x7f22cc0729b0 2023-10-16T11:20:42.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.730+0000 7f22c8ff9700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f22b806b140 0x7f22b806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:42.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.731+0000 7f22c8ff9700 1 -- 172.21.15.158:0/3773335237 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f22c408ae40 con 0x7f22cc0729b0 2023-10-16T11:20:42.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.731+0000 7f22cbfff700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f22b806b140 0x7f22b806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:42.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.731+0000 7f22cbfff700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f22b806b140 0x7f22b806d600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f22bc000c00 tx=0x7f22bc008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:42.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.735+0000 7f22c8ff9700 1 -- 172.21.15.158:0/3773335237 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22c4055490 con 0x7f22cc0729b0 2023-10-16T11:20:42.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.923+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 --> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f22cc161db0 con 0x7f22b806b140 2023-10-16T11:20:42.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.937+0000 7f22c8ff9700 1 -- 172.21.15.158:0/3773335237 <== mgr.12030 v2:172.21.15.158:6832/3408161846 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528937 (secure 0 0 0) 0x7f22bc011b00 con 0x7f22b806b140 2023-10-16T11:20:42.945 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:42.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f22b806b140 msgr2=0x7f22b806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f22b806b140 0x7f22b806d600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f22bc000c00 tx=0x7f22bc008030 comp rx=0 tx=0).stop 2023-10-16T11:20:42.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc0729b0 msgr2=0x7f22cc0c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:42.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc0729b0 0x7f22cc0c3c40 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f22c400efc0 tx=0x7f22c400c3a0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 shutdown_connections 2023-10-16T11:20:42.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f22b806b140 0x7f22b806d600 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc0729b0 0x7f22cc0c3c40 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22cc06ab40 0x7f22cc14c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 --2- 172.21.15.158:0/3773335237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22cc06e5d0 0x7f22cc15ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:42.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.941+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 >> 172.21.15.158:0/3773335237 conn(0x7f22cc0d02b0 msgr2=0x7f22cc0adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:42.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.942+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 shutdown_connections 2023-10-16T11:20:42.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:42.942+0000 7f22d2563700 1 -- 172.21.15.158:0/3773335237 wait complete. 2023-10-16T11:20:42.953 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:20:42.955 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-10-16T11:20:41.451108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":240,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436864,"kb_used_data":24948,"kb_used_omap":12,"kb_used_meta":411763,"kb_avail":754537856,"statfs":{"total":773094113280,"available":772646764544,"internally_reserved":0,"allocated":25546752,"data_stored":14253517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":421645879},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.649993"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840571+0000","last_change":"2023-10-16T11:20:38.839544+0000","last_active":"2023-10-16T11:20:38.840571+0000","last_peered":"2023-10-16T11:20:38.840571+0000","last_clean":"2023-10-16T11:20:38.840571+0000","last_became_active":"2023-10-16T11:20:38.839288+0000","last_became_peered":"2023-10-16T11:20:38.839288+0000","last_unstale":"2023-10-16T11:20:38.840571+0000","last_undegraded":"2023-10-16T11:20:38.840571+0000","last_fullsized":"2023-10-16T11:20:38.840571+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:36.154489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834896+0000","last_change":"2023-10-16T11:20:37.821795+0000","last_active":"2023-10-16T11:20:38.834896+0000","last_peered":"2023-10-16T11:20:38.834896+0000","last_clean":"2023-10-16T11:20:38.834896+0000","last_became_active":"2023-10-16T11:20:37.821171+0000","last_became_peered":"2023-10-16T11:20:37.821171+0000","last_unstale":"2023-10-16T11:20:38.834896+0000","last_undegraded":"2023-10-16T11:20:38.834896+0000","last_fullsized":"2023-10-16T11:20:38.834896+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:52.616578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804659+0000","last_change":"2023-10-16T11:20:25.748352+0000","last_active":"2023-10-16T11:20:37.804659+0000","last_peered":"2023-10-16T11:20:37.804659+0000","last_clean":"2023-10-16T11:20:37.804659+0000","last_became_active":"2023-10-16T11:20:25.747848+0000","last_became_peered":"2023-10-16T11:20:25.747848+0000","last_unstale":"2023-10-16T11:20:37.804659+0000","last_undegraded":"2023-10-16T11:20:37.804659+0000","last_fullsized":"2023-10-16T11:20:37.804659+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:48.171042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.841363+0000","last_change":"2023-10-16T11:20:38.840167+0000","last_active":"2023-10-16T11:20:38.841363+0000","last_peered":"2023-10-16T11:20:38.841363+0000","last_clean":"2023-10-16T11:20:38.841363+0000","last_became_active":"2023-10-16T11:20:38.839889+0000","last_became_peered":"2023-10-16T11:20:38.839889+0000","last_unstale":"2023-10-16T11:20:38.841363+0000","last_undegraded":"2023-10-16T11:20:38.841363+0000","last_fullsized":"2023-10-16T11:20:38.841363+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:32.738869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"240'4","reported_seq":15,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842724+0000","last_change":"2023-10-16T11:20:37.825024+0000","last_active":"2023-10-16T11:20:38.842724+0000","last_peered":"2023-10-16T11:20:38.842724+0000","last_clean":"2023-10-16T11:20:38.842724+0000","last_became_active":"2023-10-16T11:20:37.824652+0000","last_became_peered":"2023-10-16T11:20:37.824652+0000","last_unstale":"2023-10-16T11:20:38.842724+0000","last_undegraded":"2023-10-16T11:20:38.842724+0000","last_fullsized":"2023-10-16T11:20:38.842724+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:31.726955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.821043+0000","last_change":"2023-10-16T11:20:25.749395+0000","last_active":"2023-10-16T11:20:37.821043+0000","last_peered":"2023-10-16T11:20:37.821043+0000","last_clean":"2023-10-16T11:20:37.821043+0000","last_became_active":"2023-10-16T11:20:25.749044+0000","last_became_peered":"2023-10-16T11:20:25.749044+0000","last_unstale":"2023-10-16T11:20:37.821043+0000","last_undegraded":"2023-10-16T11:20:37.821043+0000","last_fullsized":"2023-10-16T11:20:37.821043+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:16.211507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.841804+0000","last_change":"2023-10-16T11:20:38.833450+0000","last_active":"2023-10-16T11:20:38.841804+0000","last_peered":"2023-10-16T11:20:38.841804+0000","last_clean":"2023-10-16T11:20:38.841804+0000","last_became_active":"2023-10-16T11:20:38.833202+0000","last_became_peered":"2023-10-16T11:20:38.833202+0000","last_unstale":"2023-10-16T11:20:38.841804+0000","last_undegraded":"2023-10-16T11:20:38.841804+0000","last_fullsized":"2023-10-16T11:20:38.841804+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:10.744127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837684+0000","last_change":"2023-10-16T11:20:37.821444+0000","last_active":"2023-10-16T11:20:38.837684+0000","last_peered":"2023-10-16T11:20:38.837684+0000","last_clean":"2023-10-16T11:20:38.837684+0000","last_became_active":"2023-10-16T11:20:37.821183+0000","last_became_peered":"2023-10-16T11:20:37.821183+0000","last_unstale":"2023-10-16T11:20:38.837684+0000","last_undegraded":"2023-10-16T11:20:38.837684+0000","last_fullsized":"2023-10-16T11:20:38.837684+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:24.978539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.809011+0000","last_change":"2023-10-16T11:20:25.739817+0000","last_active":"2023-10-16T11:20:37.809011+0000","last_peered":"2023-10-16T11:20:37.809011+0000","last_clean":"2023-10-16T11:20:37.809011+0000","last_became_active":"2023-10-16T11:20:25.739601+0000","last_became_peered":"2023-10-16T11:20:25.739601+0000","last_unstale":"2023-10-16T11:20:37.809011+0000","last_undegraded":"2023-10-16T11:20:37.809011+0000","last_fullsized":"2023-10-16T11:20:37.809011+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:38.443038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838632+0000","last_change":"2023-10-16T11:20:38.822788+0000","last_active":"2023-10-16T11:20:38.838632+0000","last_peered":"2023-10-16T11:20:38.838632+0000","last_clean":"2023-10-16T11:20:38.838632+0000","last_became_active":"2023-10-16T11:20:38.822510+0000","last_became_peered":"2023-10-16T11:20:38.822510+0000","last_unstale":"2023-10-16T11:20:38.838632+0000","last_undegraded":"2023-10-16T11:20:38.838632+0000","last_fullsized":"2023-10-16T11:20:38.838632+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:13.281230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844121+0000","last_change":"2023-10-16T11:20:37.816446+0000","last_active":"2023-10-16T11:20:38.844121+0000","last_peered":"2023-10-16T11:20:38.844121+0000","last_clean":"2023-10-16T11:20:38.844121+0000","last_became_active":"2023-10-16T11:20:37.815968+0000","last_became_peered":"2023-10-16T11:20:37.815968+0000","last_unstale":"2023-10-16T11:20:38.844121+0000","last_undegraded":"2023-10-16T11:20:38.844121+0000","last_fullsized":"2023-10-16T11:20:38.844121+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:36.822624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806723+0000","last_change":"2023-10-16T11:20:25.751269+0000","last_active":"2023-10-16T11:20:37.806723+0000","last_peered":"2023-10-16T11:20:37.806723+0000","last_clean":"2023-10-16T11:20:37.806723+0000","last_became_active":"2023-10-16T11:20:25.750973+0000","last_became_peered":"2023-10-16T11:20:25.750973+0000","last_unstale":"2023-10-16T11:20:37.806723+0000","last_undegraded":"2023-10-16T11:20:37.806723+0000","last_fullsized":"2023-10-16T11:20:37.806723+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:36.490706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843350+0000","last_change":"2023-10-16T11:20:38.840537+0000","last_active":"2023-10-16T11:20:38.843350+0000","last_peered":"2023-10-16T11:20:38.843350+0000","last_clean":"2023-10-16T11:20:38.843350+0000","last_became_active":"2023-10-16T11:20:38.838185+0000","last_became_peered":"2023-10-16T11:20:38.838185+0000","last_unstale":"2023-10-16T11:20:38.843350+0000","last_undegraded":"2023-10-16T11:20:38.843350+0000","last_fullsized":"2023-10-16T11:20:38.843350+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:39.948003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844724+0000","last_change":"2023-10-16T11:20:37.820445+0000","last_active":"2023-10-16T11:20:38.844724+0000","last_peered":"2023-10-16T11:20:38.844724+0000","last_clean":"2023-10-16T11:20:38.844724+0000","last_became_active":"2023-10-16T11:20:37.820143+0000","last_became_peered":"2023-10-16T11:20:37.820143+0000","last_unstale":"2023-10-16T11:20:38.844724+0000","last_undegraded":"2023-10-16T11:20:38.844724+0000","last_fullsized":"2023-10-16T11:20:38.844724+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:21.515898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813297+0000","last_change":"2023-10-16T11:20:25.750599+0000","last_active":"2023-10-16T11:20:37.813297+0000","last_peered":"2023-10-16T11:20:37.813297+0000","last_clean":"2023-10-16T11:20:37.813297+0000","last_became_active":"2023-10-16T11:20:25.749962+0000","last_became_peered":"2023-10-16T11:20:25.749962+0000","last_unstale":"2023-10-16T11:20:37.813297+0000","last_undegraded":"2023-10-16T11:20:37.813297+0000","last_fullsized":"2023-10-16T11:20:37.813297+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:15.073727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840065+0000","last_change":"2023-10-16T11:20:38.833354+0000","last_active":"2023-10-16T11:20:38.840065+0000","last_peered":"2023-10-16T11:20:38.840065+0000","last_clean":"2023-10-16T11:20:38.840065+0000","last_became_active":"2023-10-16T11:20:38.832945+0000","last_became_peered":"2023-10-16T11:20:38.832945+0000","last_unstale":"2023-10-16T11:20:38.840065+0000","last_undegraded":"2023-10-16T11:20:38.840065+0000","last_fullsized":"2023-10-16T11:20:38.840065+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:25.776247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840318+0000","last_change":"2023-10-16T11:20:37.818148+0000","last_active":"2023-10-16T11:20:38.840318+0000","last_peered":"2023-10-16T11:20:38.840318+0000","last_clean":"2023-10-16T11:20:38.840318+0000","last_became_active":"2023-10-16T11:20:37.817676+0000","last_became_peered":"2023-10-16T11:20:37.817676+0000","last_unstale":"2023-10-16T11:20:38.840318+0000","last_undegraded":"2023-10-16T11:20:38.840318+0000","last_fullsized":"2023-10-16T11:20:38.840318+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.825698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804881+0000","last_change":"2023-10-16T11:20:25.745973+0000","last_active":"2023-10-16T11:20:37.804881+0000","last_peered":"2023-10-16T11:20:37.804881+0000","last_clean":"2023-10-16T11:20:37.804881+0000","last_became_active":"2023-10-16T11:20:25.745706+0000","last_became_peered":"2023-10-16T11:20:25.745706+0000","last_unstale":"2023-10-16T11:20:37.804881+0000","last_undegraded":"2023-10-16T11:20:37.804881+0000","last_fullsized":"2023-10-16T11:20:37.804881+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:08.628698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844221+0000","last_change":"2023-10-16T11:20:38.838232+0000","last_active":"2023-10-16T11:20:38.844221+0000","last_peered":"2023-10-16T11:20:38.844221+0000","last_clean":"2023-10-16T11:20:38.844221+0000","last_became_active":"2023-10-16T11:20:38.837628+0000","last_became_peered":"2023-10-16T11:20:38.837628+0000","last_unstale":"2023-10-16T11:20:38.844221+0000","last_undegraded":"2023-10-16T11:20:38.844221+0000","last_fullsized":"2023-10-16T11:20:38.844221+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.963194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.823216+0000","last_change":"2023-10-16T11:20:37.814903+0000","last_active":"2023-10-16T11:20:38.823216+0000","last_peered":"2023-10-16T11:20:38.823216+0000","last_clean":"2023-10-16T11:20:38.823216+0000","last_became_active":"2023-10-16T11:20:37.814642+0000","last_became_peered":"2023-10-16T11:20:37.814642+0000","last_unstale":"2023-10-16T11:20:38.823216+0000","last_undegraded":"2023-10-16T11:20:38.823216+0000","last_fullsized":"2023-10-16T11:20:38.823216+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:55.013398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816744+0000","last_change":"2023-10-16T11:20:25.748211+0000","last_active":"2023-10-16T11:20:37.816744+0000","last_peered":"2023-10-16T11:20:37.816744+0000","last_clean":"2023-10-16T11:20:37.816744+0000","last_became_active":"2023-10-16T11:20:25.747956+0000","last_became_peered":"2023-10-16T11:20:25.747956+0000","last_unstale":"2023-10-16T11:20:37.816744+0000","last_undegraded":"2023-10-16T11:20:37.816744+0000","last_fullsized":"2023-10-16T11:20:37.816744+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838813+0000","last_change":"2023-10-16T11:20:38.838166+0000","last_active":"2023-10-16T11:20:38.838813+0000","last_peered":"2023-10-16T11:20:38.838813+0000","last_clean":"2023-10-16T11:20:38.838813+0000","last_became_active":"2023-10-16T11:20:38.837889+0000","last_became_peered":"2023-10-16T11:20:38.837889+0000","last_unstale":"2023-10-16T11:20:38.838813+0000","last_undegraded":"2023-10-16T11:20:38.838813+0000","last_fullsized":"2023-10-16T11:20:38.838813+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:52.880299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.824373+0000","last_change":"2023-10-16T11:20:37.819002+0000","last_active":"2023-10-16T11:20:38.824373+0000","last_peered":"2023-10-16T11:20:38.824373+0000","last_clean":"2023-10-16T11:20:38.824373+0000","last_became_active":"2023-10-16T11:20:37.818578+0000","last_became_peered":"2023-10-16T11:20:37.818578+0000","last_unstale":"2023-10-16T11:20:38.824373+0000","last_undegraded":"2023-10-16T11:20:38.824373+0000","last_fullsized":"2023-10-16T11:20:38.824373+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:55.970715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812779+0000","last_change":"2023-10-16T11:20:25.750901+0000","last_active":"2023-10-16T11:20:37.812779+0000","last_peered":"2023-10-16T11:20:37.812779+0000","last_clean":"2023-10-16T11:20:37.812779+0000","last_became_active":"2023-10-16T11:20:25.750582+0000","last_became_peered":"2023-10-16T11:20:25.750582+0000","last_unstale":"2023-10-16T11:20:37.812779+0000","last_undegraded":"2023-10-16T11:20:37.812779+0000","last_fullsized":"2023-10-16T11:20:37.812779+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:47.707747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840552+0000","last_change":"2023-10-16T11:20:38.835941+0000","last_active":"2023-10-16T11:20:38.840552+0000","last_peered":"2023-10-16T11:20:38.840552+0000","last_clean":"2023-10-16T11:20:38.840552+0000","last_became_active":"2023-10-16T11:20:38.835620+0000","last_became_peered":"2023-10-16T11:20:38.835620+0000","last_unstale":"2023-10-16T11:20:38.840552+0000","last_undegraded":"2023-10-16T11:20:38.840552+0000","last_fullsized":"2023-10-16T11:20:38.840552+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:23:58.942409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.824574+0000","last_change":"2023-10-16T11:20:37.818876+0000","last_active":"2023-10-16T11:20:38.824574+0000","last_peered":"2023-10-16T11:20:38.824574+0000","last_clean":"2023-10-16T11:20:38.824574+0000","last_became_active":"2023-10-16T11:20:37.818610+0000","last_became_peered":"2023-10-16T11:20:37.818610+0000","last_unstale":"2023-10-16T11:20:38.824574+0000","last_undegraded":"2023-10-16T11:20:38.824574+0000","last_fullsized":"2023-10-16T11:20:38.824574+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:53.263379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804712+0000","last_change":"2023-10-16T11:20:25.750204+0000","last_active":"2023-10-16T11:20:37.804712+0000","last_peered":"2023-10-16T11:20:37.804712+0000","last_clean":"2023-10-16T11:20:37.804712+0000","last_became_active":"2023-10-16T11:20:25.749903+0000","last_became_peered":"2023-10-16T11:20:25.749903+0000","last_unstale":"2023-10-16T11:20:37.804712+0000","last_undegraded":"2023-10-16T11:20:37.804712+0000","last_fullsized":"2023-10-16T11:20:37.804712+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:48.734101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.830225+0000","last_change":"2023-10-16T11:20:38.826174+0000","last_active":"2023-10-16T11:20:38.830225+0000","last_peered":"2023-10-16T11:20:38.830225+0000","last_clean":"2023-10-16T11:20:38.830225+0000","last_became_active":"2023-10-16T11:20:38.825902+0000","last_became_peered":"2023-10-16T11:20:38.825902+0000","last_unstale":"2023-10-16T11:20:38.830225+0000","last_undegraded":"2023-10-16T11:20:38.830225+0000","last_fullsized":"2023-10-16T11:20:38.830225+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:59.413642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.816526+0000","last_change":"2023-10-16T11:20:37.819397+0000","last_active":"2023-10-16T11:20:38.816526+0000","last_peered":"2023-10-16T11:20:38.816526+0000","last_clean":"2023-10-16T11:20:38.816526+0000","last_became_active":"2023-10-16T11:20:37.819133+0000","last_became_peered":"2023-10-16T11:20:37.819133+0000","last_unstale":"2023-10-16T11:20:38.816526+0000","last_undegraded":"2023-10-16T11:20:38.816526+0000","last_fullsized":"2023-10-16T11:20:38.816526+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:21.860649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806760+0000","last_change":"2023-10-16T11:20:25.747726+0000","last_active":"2023-10-16T11:20:37.806760+0000","last_peered":"2023-10-16T11:20:37.806760+0000","last_clean":"2023-10-16T11:20:37.806760+0000","last_became_active":"2023-10-16T11:20:25.747314+0000","last_became_peered":"2023-10-16T11:20:25.747314+0000","last_unstale":"2023-10-16T11:20:37.806760+0000","last_undegraded":"2023-10-16T11:20:37.806760+0000","last_fullsized":"2023-10-16T11:20:37.806760+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837048+0000","last_change":"2023-10-16T11:20:38.834016+0000","last_active":"2023-10-16T11:20:38.837048+0000","last_peered":"2023-10-16T11:20:38.837048+0000","last_clean":"2023-10-16T11:20:38.837048+0000","last_became_active":"2023-10-16T11:20:38.832409+0000","last_became_peered":"2023-10-16T11:20:38.832409+0000","last_unstale":"2023-10-16T11:20:38.837048+0000","last_undegraded":"2023-10-16T11:20:38.837048+0000","last_fullsized":"2023-10-16T11:20:38.837048+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:47.307575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"240'6","reported_seq":17,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.846260+0000","last_change":"2023-10-16T11:20:37.822490+0000","last_active":"2023-10-16T11:20:38.846260+0000","last_peered":"2023-10-16T11:20:38.846260+0000","last_clean":"2023-10-16T11:20:38.846260+0000","last_became_active":"2023-10-16T11:20:37.822219+0000","last_became_peered":"2023-10-16T11:20:37.822219+0000","last_unstale":"2023-10-16T11:20:38.846260+0000","last_undegraded":"2023-10-16T11:20:38.846260+0000","last_fullsized":"2023-10-16T11:20:38.846260+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:00.023220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.809179+0000","last_change":"2023-10-16T11:20:25.747416+0000","last_active":"2023-10-16T11:20:37.809179+0000","last_peered":"2023-10-16T11:20:37.809179+0000","last_clean":"2023-10-16T11:20:37.809179+0000","last_became_active":"2023-10-16T11:20:25.746950+0000","last_became_peered":"2023-10-16T11:20:25.746950+0000","last_unstale":"2023-10-16T11:20:37.809179+0000","last_undegraded":"2023-10-16T11:20:37.809179+0000","last_fullsized":"2023-10-16T11:20:37.809179+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843591+0000","last_change":"2023-10-16T11:20:38.842102+0000","last_active":"2023-10-16T11:20:38.843591+0000","last_peered":"2023-10-16T11:20:38.843591+0000","last_clean":"2023-10-16T11:20:38.843591+0000","last_became_active":"2023-10-16T11:20:38.841601+0000","last_became_peered":"2023-10-16T11:20:38.841601+0000","last_unstale":"2023-10-16T11:20:38.843591+0000","last_undegraded":"2023-10-16T11:20:38.843591+0000","last_fullsized":"2023-10-16T11:20:38.843591+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:26.708062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"240'3","reported_seq":14,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842625+0000","last_change":"2023-10-16T11:20:37.818796+0000","last_active":"2023-10-16T11:20:38.842625+0000","last_peered":"2023-10-16T11:20:38.842625+0000","last_clean":"2023-10-16T11:20:38.842625+0000","last_became_active":"2023-10-16T11:20:37.818545+0000","last_became_peered":"2023-10-16T11:20:37.818545+0000","last_unstale":"2023-10-16T11:20:38.842625+0000","last_undegraded":"2023-10-16T11:20:38.842625+0000","last_fullsized":"2023-10-16T11:20:38.842625+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:05.678488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807590+0000","last_change":"2023-10-16T11:20:25.743709+0000","last_active":"2023-10-16T11:20:37.807590+0000","last_peered":"2023-10-16T11:20:37.807590+0000","last_clean":"2023-10-16T11:20:37.807590+0000","last_became_active":"2023-10-16T11:20:25.743444+0000","last_became_peered":"2023-10-16T11:20:25.743444+0000","last_unstale":"2023-10-16T11:20:37.807590+0000","last_undegraded":"2023-10-16T11:20:37.807590+0000","last_fullsized":"2023-10-16T11:20:37.807590+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:28.662644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836955+0000","last_change":"2023-10-16T11:20:38.834327+0000","last_active":"2023-10-16T11:20:38.836955+0000","last_peered":"2023-10-16T11:20:38.836955+0000","last_clean":"2023-10-16T11:20:38.836955+0000","last_became_active":"2023-10-16T11:20:38.833988+0000","last_became_peered":"2023-10-16T11:20:38.833988+0000","last_unstale":"2023-10-16T11:20:38.836955+0000","last_undegraded":"2023-10-16T11:20:38.836955+0000","last_fullsized":"2023-10-16T11:20:38.836955+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:17.884888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.818487+0000","last_change":"2023-10-16T11:20:37.816942+0000","last_active":"2023-10-16T11:20:38.818487+0000","last_peered":"2023-10-16T11:20:38.818487+0000","last_clean":"2023-10-16T11:20:38.818487+0000","last_became_active":"2023-10-16T11:20:37.815666+0000","last_became_peered":"2023-10-16T11:20:37.815666+0000","last_unstale":"2023-10-16T11:20:38.818487+0000","last_undegraded":"2023-10-16T11:20:38.818487+0000","last_fullsized":"2023-10-16T11:20:38.818487+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:03.222103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810201+0000","last_change":"2023-10-16T11:20:25.737922+0000","last_active":"2023-10-16T11:20:37.810201+0000","last_peered":"2023-10-16T11:20:37.810201+0000","last_clean":"2023-10-16T11:20:37.810201+0000","last_became_active":"2023-10-16T11:20:25.737086+0000","last_became_peered":"2023-10-16T11:20:25.737086+0000","last_unstale":"2023-10-16T11:20:37.810201+0000","last_undegraded":"2023-10-16T11:20:37.810201+0000","last_fullsized":"2023-10-16T11:20:37.810201+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:17.897224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839810+0000","last_change":"2023-10-16T11:20:38.833495+0000","last_active":"2023-10-16T11:20:38.839810+0000","last_peered":"2023-10-16T11:20:38.839810+0000","last_clean":"2023-10-16T11:20:38.839810+0000","last_became_active":"2023-10-16T11:20:38.833247+0000","last_became_peered":"2023-10-16T11:20:38.833247+0000","last_unstale":"2023-10-16T11:20:38.839810+0000","last_undegraded":"2023-10-16T11:20:38.839810+0000","last_fullsized":"2023-10-16T11:20:38.839810+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:32.097784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840955+0000","last_change":"2023-10-16T11:20:37.818788+0000","last_active":"2023-10-16T11:20:38.840955+0000","last_peered":"2023-10-16T11:20:38.840955+0000","last_clean":"2023-10-16T11:20:38.840955+0000","last_became_active":"2023-10-16T11:20:37.818537+0000","last_became_peered":"2023-10-16T11:20:37.818537+0000","last_unstale":"2023-10-16T11:20:38.840955+0000","last_undegraded":"2023-10-16T11:20:38.840955+0000","last_fullsized":"2023-10-16T11:20:38.840955+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:46.682699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813692+0000","last_change":"2023-10-16T11:20:25.745458+0000","last_active":"2023-10-16T11:20:37.813692+0000","last_peered":"2023-10-16T11:20:37.813692+0000","last_clean":"2023-10-16T11:20:37.813692+0000","last_became_active":"2023-10-16T11:20:25.745257+0000","last_became_peered":"2023-10-16T11:20:25.745257+0000","last_unstale":"2023-10-16T11:20:37.813692+0000","last_undegraded":"2023-10-16T11:20:37.813692+0000","last_fullsized":"2023-10-16T11:20:37.813692+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832698+0000","last_change":"2023-10-16T11:20:38.830760+0000","last_active":"2023-10-16T11:20:38.832698+0000","last_peered":"2023-10-16T11:20:38.832698+0000","last_clean":"2023-10-16T11:20:38.832698+0000","last_became_active":"2023-10-16T11:20:38.830505+0000","last_became_peered":"2023-10-16T11:20:38.830505+0000","last_unstale":"2023-10-16T11:20:38.832698+0000","last_undegraded":"2023-10-16T11:20:38.832698+0000","last_fullsized":"2023-10-16T11:20:38.832698+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:51.210622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"240'8","reported_seq":19,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843703+0000","last_change":"2023-10-16T11:20:37.823469+0000","last_active":"2023-10-16T11:20:38.843703+0000","last_peered":"2023-10-16T11:20:38.843703+0000","last_clean":"2023-10-16T11:20:38.843703+0000","last_became_active":"2023-10-16T11:20:37.823231+0000","last_became_peered":"2023-10-16T11:20:37.823231+0000","last_unstale":"2023-10-16T11:20:38.843703+0000","last_undegraded":"2023-10-16T11:20:38.843703+0000","last_fullsized":"2023-10-16T11:20:38.843703+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:47.140140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803677+0000","last_change":"2023-10-16T11:20:25.741269+0000","last_active":"2023-10-16T11:20:37.803677+0000","last_peered":"2023-10-16T11:20:37.803677+0000","last_clean":"2023-10-16T11:20:37.803677+0000","last_became_active":"2023-10-16T11:20:25.740995+0000","last_became_peered":"2023-10-16T11:20:25.740995+0000","last_unstale":"2023-10-16T11:20:37.803677+0000","last_undegraded":"2023-10-16T11:20:37.803677+0000","last_fullsized":"2023-10-16T11:20:37.803677+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:45.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839783+0000","last_change":"2023-10-16T11:20:38.834668+0000","last_active":"2023-10-16T11:20:38.839783+0000","last_peered":"2023-10-16T11:20:38.839783+0000","last_clean":"2023-10-16T11:20:38.839783+0000","last_became_active":"2023-10-16T11:20:38.834405+0000","last_became_peered":"2023-10-16T11:20:38.834405+0000","last_unstale":"2023-10-16T11:20:38.839783+0000","last_undegraded":"2023-10-16T11:20:38.839783+0000","last_fullsized":"2023-10-16T11:20:38.839783+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:06.544476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.816217+0000","last_change":"2023-10-16T11:20:37.822881+0000","last_active":"2023-10-16T11:20:38.816217+0000","last_peered":"2023-10-16T11:20:38.816217+0000","last_clean":"2023-10-16T11:20:38.816217+0000","last_became_active":"2023-10-16T11:20:37.822663+0000","last_became_peered":"2023-10-16T11:20:37.822663+0000","last_unstale":"2023-10-16T11:20:38.816217+0000","last_undegraded":"2023-10-16T11:20:38.816217+0000","last_fullsized":"2023-10-16T11:20:38.816217+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:08.064858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813287+0000","last_change":"2023-10-16T11:20:25.742299+0000","last_active":"2023-10-16T11:20:37.813287+0000","last_peered":"2023-10-16T11:20:37.813287+0000","last_clean":"2023-10-16T11:20:37.813287+0000","last_became_active":"2023-10-16T11:20:25.741394+0000","last_became_peered":"2023-10-16T11:20:25.741394+0000","last_unstale":"2023-10-16T11:20:37.813287+0000","last_undegraded":"2023-10-16T11:20:37.813287+0000","last_fullsized":"2023-10-16T11:20:37.813287+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:35.560067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839006+0000","last_change":"2023-10-16T11:20:38.835643+0000","last_active":"2023-10-16T11:20:38.839006+0000","last_peered":"2023-10-16T11:20:38.839006+0000","last_clean":"2023-10-16T11:20:38.839006+0000","last_became_active":"2023-10-16T11:20:38.835197+0000","last_became_peered":"2023-10-16T11:20:38.835197+0000","last_unstale":"2023-10-16T11:20:38.839006+0000","last_undegraded":"2023-10-16T11:20:38.839006+0000","last_fullsized":"2023-10-16T11:20:38.839006+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:08.451101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844172+0000","last_change":"2023-10-16T11:20:37.824771+0000","last_active":"2023-10-16T11:20:38.844172+0000","last_peered":"2023-10-16T11:20:38.844172+0000","last_clean":"2023-10-16T11:20:38.844172+0000","last_became_active":"2023-10-16T11:20:37.823889+0000","last_became_peered":"2023-10-16T11:20:37.823889+0000","last_unstale":"2023-10-16T11:20:38.844172+0000","last_undegraded":"2023-10-16T11:20:38.844172+0000","last_fullsized":"2023-10-16T11:20:38.844172+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:19.853934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811254+0000","last_change":"2023-10-16T11:20:25.744534+0000","last_active":"2023-10-16T11:20:37.811254+0000","last_peered":"2023-10-16T11:20:37.811254+0000","last_clean":"2023-10-16T11:20:37.811254+0000","last_became_active":"2023-10-16T11:20:25.744224+0000","last_became_peered":"2023-10-16T11:20:25.744224+0000","last_unstale":"2023-10-16T11:20:37.811254+0000","last_undegraded":"2023-10-16T11:20:37.811254+0000","last_fullsized":"2023-10-16T11:20:37.811254+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:14.437943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837184+0000","last_change":"2023-10-16T11:20:38.833376+0000","last_active":"2023-10-16T11:20:38.837184+0000","last_peered":"2023-10-16T11:20:38.837184+0000","last_clean":"2023-10-16T11:20:38.837184+0000","last_became_active":"2023-10-16T11:20:38.833081+0000","last_became_peered":"2023-10-16T11:20:38.833081+0000","last_unstale":"2023-10-16T11:20:38.837184+0000","last_undegraded":"2023-10-16T11:20:38.837184+0000","last_fullsized":"2023-10-16T11:20:38.837184+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:56.175759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844934+0000","last_change":"2023-10-16T11:20:37.824665+0000","last_active":"2023-10-16T11:20:38.844934+0000","last_peered":"2023-10-16T11:20:38.844934+0000","last_clean":"2023-10-16T11:20:38.844934+0000","last_became_active":"2023-10-16T11:20:37.824328+0000","last_became_peered":"2023-10-16T11:20:37.824328+0000","last_unstale":"2023-10-16T11:20:38.844934+0000","last_undegraded":"2023-10-16T11:20:38.844934+0000","last_fullsized":"2023-10-16T11:20:38.844934+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:29.555280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806853+0000","last_change":"2023-10-16T11:20:25.747551+0000","last_active":"2023-10-16T11:20:37.806853+0000","last_peered":"2023-10-16T11:20:37.806853+0000","last_clean":"2023-10-16T11:20:37.806853+0000","last_became_active":"2023-10-16T11:20:25.746989+0000","last_became_peered":"2023-10-16T11:20:25.746989+0000","last_unstale":"2023-10-16T11:20:37.806853+0000","last_undegraded":"2023-10-16T11:20:37.806853+0000","last_fullsized":"2023-10-16T11:20:37.806853+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837089+0000","last_change":"2023-10-16T11:20:38.834474+0000","last_active":"2023-10-16T11:20:38.837089+0000","last_peered":"2023-10-16T11:20:38.837089+0000","last_clean":"2023-10-16T11:20:38.837089+0000","last_became_active":"2023-10-16T11:20:38.833723+0000","last_became_peered":"2023-10-16T11:20:38.833723+0000","last_unstale":"2023-10-16T11:20:38.837089+0000","last_undegraded":"2023-10-16T11:20:38.837089+0000","last_fullsized":"2023-10-16T11:20:38.837089+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:06.806707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.814098+0000","last_change":"2023-10-16T11:20:37.824288+0000","last_active":"2023-10-16T11:20:38.814098+0000","last_peered":"2023-10-16T11:20:38.814098+0000","last_clean":"2023-10-16T11:20:38.814098+0000","last_became_active":"2023-10-16T11:20:37.824006+0000","last_became_peered":"2023-10-16T11:20:37.824006+0000","last_unstale":"2023-10-16T11:20:38.814098+0000","last_undegraded":"2023-10-16T11:20:38.814098+0000","last_fullsized":"2023-10-16T11:20:38.814098+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:26.795558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804059+0000","last_change":"2023-10-16T11:20:25.738568+0000","last_active":"2023-10-16T11:20:37.804059+0000","last_peered":"2023-10-16T11:20:37.804059+0000","last_clean":"2023-10-16T11:20:37.804059+0000","last_became_active":"2023-10-16T11:20:25.738313+0000","last_became_peered":"2023-10-16T11:20:25.738313+0000","last_unstale":"2023-10-16T11:20:37.804059+0000","last_undegraded":"2023-10-16T11:20:37.804059+0000","last_fullsized":"2023-10-16T11:20:37.804059+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:28.152434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.835474+0000","last_change":"2023-10-16T11:20:38.828989+0000","last_active":"2023-10-16T11:20:38.835474+0000","last_peered":"2023-10-16T11:20:38.835474+0000","last_clean":"2023-10-16T11:20:38.835474+0000","last_became_active":"2023-10-16T11:20:38.828709+0000","last_became_peered":"2023-10-16T11:20:38.828709+0000","last_unstale":"2023-10-16T11:20:38.835474+0000","last_undegraded":"2023-10-16T11:20:38.835474+0000","last_fullsized":"2023-10-16T11:20:38.835474+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:20.029437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843092+0000","last_change":"2023-10-16T11:20:37.819719+0000","last_active":"2023-10-16T11:20:38.843092+0000","last_peered":"2023-10-16T11:20:38.843092+0000","last_clean":"2023-10-16T11:20:38.843092+0000","last_became_active":"2023-10-16T11:20:37.819410+0000","last_became_peered":"2023-10-16T11:20:37.819410+0000","last_unstale":"2023-10-16T11:20:38.843092+0000","last_undegraded":"2023-10-16T11:20:38.843092+0000","last_fullsized":"2023-10-16T11:20:38.843092+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:38.939710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.819400+0000","last_change":"2023-10-16T11:20:25.750312+0000","last_active":"2023-10-16T11:20:37.819400+0000","last_peered":"2023-10-16T11:20:37.819400+0000","last_clean":"2023-10-16T11:20:37.819400+0000","last_became_active":"2023-10-16T11:20:25.749320+0000","last_became_peered":"2023-10-16T11:20:25.749320+0000","last_unstale":"2023-10-16T11:20:37.819400+0000","last_undegraded":"2023-10-16T11:20:37.819400+0000","last_fullsized":"2023-10-16T11:20:37.819400+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:26.044494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834192+0000","last_change":"2023-10-16T11:20:38.824994+0000","last_active":"2023-10-16T11:20:38.834192+0000","last_peered":"2023-10-16T11:20:38.834192+0000","last_clean":"2023-10-16T11:20:38.834192+0000","last_became_active":"2023-10-16T11:20:38.824683+0000","last_became_peered":"2023-10-16T11:20:38.824683+0000","last_unstale":"2023-10-16T11:20:38.834192+0000","last_undegraded":"2023-10-16T11:20:38.834192+0000","last_fullsized":"2023-10-16T11:20:38.834192+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:16.319016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.823180+0000","last_change":"2023-10-16T11:20:37.814633+0000","last_active":"2023-10-16T11:20:38.823180+0000","last_peered":"2023-10-16T11:20:38.823180+0000","last_clean":"2023-10-16T11:20:38.823180+0000","last_became_active":"2023-10-16T11:20:37.814347+0000","last_became_peered":"2023-10-16T11:20:37.814347+0000","last_unstale":"2023-10-16T11:20:38.823180+0000","last_undegraded":"2023-10-16T11:20:38.823180+0000","last_fullsized":"2023-10-16T11:20:38.823180+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:24.767953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816618+0000","last_change":"2023-10-16T11:20:25.744924+0000","last_active":"2023-10-16T11:20:37.816618+0000","last_peered":"2023-10-16T11:20:37.816618+0000","last_clean":"2023-10-16T11:20:37.816618+0000","last_became_active":"2023-10-16T11:20:25.744659+0000","last_became_peered":"2023-10-16T11:20:25.744659+0000","last_unstale":"2023-10-16T11:20:37.816618+0000","last_undegraded":"2023-10-16T11:20:37.816618+0000","last_fullsized":"2023-10-16T11:20:37.816618+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.833124+0000","last_change":"2023-10-16T11:20:38.824167+0000","last_active":"2023-10-16T11:20:38.833124+0000","last_peered":"2023-10-16T11:20:38.833124+0000","last_clean":"2023-10-16T11:20:38.833124+0000","last_became_active":"2023-10-16T11:20:38.823749+0000","last_became_peered":"2023-10-16T11:20:38.823749+0000","last_unstale":"2023-10-16T11:20:38.833124+0000","last_undegraded":"2023-10-16T11:20:38.833124+0000","last_fullsized":"2023-10-16T11:20:38.833124+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:06.544476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844428+0000","last_change":"2023-10-16T11:20:37.818380+0000","last_active":"2023-10-16T11:20:38.844428+0000","last_peered":"2023-10-16T11:20:38.844428+0000","last_clean":"2023-10-16T11:20:38.844428+0000","last_became_active":"2023-10-16T11:20:37.818042+0000","last_became_peered":"2023-10-16T11:20:37.818042+0000","last_unstale":"2023-10-16T11:20:38.844428+0000","last_undegraded":"2023-10-16T11:20:38.844428+0000","last_fullsized":"2023-10-16T11:20:38.844428+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:38.939710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810348+0000","last_change":"2023-10-16T11:20:25.750717+0000","last_active":"2023-10-16T11:20:37.810348+0000","last_peered":"2023-10-16T11:20:37.810348+0000","last_clean":"2023-10-16T11:20:37.810348+0000","last_became_active":"2023-10-16T11:20:25.750419+0000","last_became_peered":"2023-10-16T11:20:25.750419+0000","last_unstale":"2023-10-16T11:20:37.810348+0000","last_undegraded":"2023-10-16T11:20:37.810348+0000","last_fullsized":"2023-10-16T11:20:37.810348+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:21.095229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.841828+0000","last_change":"2023-10-16T11:20:38.833965+0000","last_active":"2023-10-16T11:20:38.841828+0000","last_peered":"2023-10-16T11:20:38.841828+0000","last_clean":"2023-10-16T11:20:38.841828+0000","last_became_active":"2023-10-16T11:20:38.833701+0000","last_became_peered":"2023-10-16T11:20:38.833701+0000","last_unstale":"2023-10-16T11:20:38.841828+0000","last_undegraded":"2023-10-16T11:20:38.841828+0000","last_fullsized":"2023-10-16T11:20:38.841828+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:54.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.819135+0000","last_change":"2023-10-16T11:20:37.812983+0000","last_active":"2023-10-16T11:20:38.819135+0000","last_peered":"2023-10-16T11:20:38.819135+0000","last_clean":"2023-10-16T11:20:38.819135+0000","last_became_active":"2023-10-16T11:20:37.812653+0000","last_became_peered":"2023-10-16T11:20:37.812653+0000","last_unstale":"2023-10-16T11:20:38.819135+0000","last_undegraded":"2023-10-16T11:20:38.819135+0000","last_fullsized":"2023-10-16T11:20:38.819135+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:29.758492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803783+0000","last_change":"2023-10-16T11:20:25.738444+0000","last_active":"2023-10-16T11:20:37.803783+0000","last_peered":"2023-10-16T11:20:37.803783+0000","last_clean":"2023-10-16T11:20:37.803783+0000","last_became_active":"2023-10-16T11:20:25.738166+0000","last_became_peered":"2023-10-16T11:20:25.738166+0000","last_unstale":"2023-10-16T11:20:37.803783+0000","last_undegraded":"2023-10-16T11:20:37.803783+0000","last_fullsized":"2023-10-16T11:20:37.803783+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:57.655885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836065+0000","last_change":"2023-10-16T11:20:38.825862+0000","last_active":"2023-10-16T11:20:38.836065+0000","last_peered":"2023-10-16T11:20:38.836065+0000","last_clean":"2023-10-16T11:20:38.836065+0000","last_became_active":"2023-10-16T11:20:38.825651+0000","last_became_peered":"2023-10-16T11:20:38.825651+0000","last_unstale":"2023-10-16T11:20:38.836065+0000","last_undegraded":"2023-10-16T11:20:38.836065+0000","last_fullsized":"2023-10-16T11:20:38.836065+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:40.135422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.822987+0000","last_change":"2023-10-16T11:20:37.824585+0000","last_active":"2023-10-16T11:20:38.822987+0000","last_peered":"2023-10-16T11:20:38.822987+0000","last_clean":"2023-10-16T11:20:38.822987+0000","last_became_active":"2023-10-16T11:20:37.824308+0000","last_became_peered":"2023-10-16T11:20:37.824308+0000","last_unstale":"2023-10-16T11:20:38.822987+0000","last_undegraded":"2023-10-16T11:20:38.822987+0000","last_fullsized":"2023-10-16T11:20:38.822987+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:53.964759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810288+0000","last_change":"2023-10-16T11:20:25.741211+0000","last_active":"2023-10-16T11:20:37.810288+0000","last_peered":"2023-10-16T11:20:37.810288+0000","last_clean":"2023-10-16T11:20:37.810288+0000","last_became_active":"2023-10-16T11:20:25.740922+0000","last_became_peered":"2023-10-16T11:20:25.740922+0000","last_unstale":"2023-10-16T11:20:37.810288+0000","last_undegraded":"2023-10-16T11:20:37.810288+0000","last_fullsized":"2023-10-16T11:20:37.810288+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:55.904667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843615+0000","last_change":"2023-10-16T11:20:38.840850+0000","last_active":"2023-10-16T11:20:38.843615+0000","last_peered":"2023-10-16T11:20:38.843615+0000","last_clean":"2023-10-16T11:20:38.843615+0000","last_became_active":"2023-10-16T11:20:38.840587+0000","last_became_peered":"2023-10-16T11:20:38.840587+0000","last_unstale":"2023-10-16T11:20:38.843615+0000","last_undegraded":"2023-10-16T11:20:38.843615+0000","last_fullsized":"2023-10-16T11:20:38.843615+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:06.896559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.814558+0000","last_change":"2023-10-16T11:20:37.824584+0000","last_active":"2023-10-16T11:20:38.814558+0000","last_peered":"2023-10-16T11:20:38.814558+0000","last_clean":"2023-10-16T11:20:38.814558+0000","last_became_active":"2023-10-16T11:20:37.824309+0000","last_became_peered":"2023-10-16T11:20:37.824309+0000","last_unstale":"2023-10-16T11:20:38.814558+0000","last_undegraded":"2023-10-16T11:20:38.814558+0000","last_fullsized":"2023-10-16T11:20:38.814558+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:06.259639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.819373+0000","last_change":"2023-10-16T11:20:25.741581+0000","last_active":"2023-10-16T11:20:37.819373+0000","last_peered":"2023-10-16T11:20:37.819373+0000","last_clean":"2023-10-16T11:20:37.819373+0000","last_became_active":"2023-10-16T11:20:25.740490+0000","last_became_peered":"2023-10-16T11:20:25.740490+0000","last_unstale":"2023-10-16T11:20:37.819373+0000","last_undegraded":"2023-10-16T11:20:37.819373+0000","last_fullsized":"2023-10-16T11:20:37.819373+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:20.166253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842614+0000","last_change":"2023-10-16T11:20:38.833860+0000","last_active":"2023-10-16T11:20:38.842614+0000","last_peered":"2023-10-16T11:20:38.842614+0000","last_clean":"2023-10-16T11:20:38.842614+0000","last_became_active":"2023-10-16T11:20:38.833570+0000","last_became_peered":"2023-10-16T11:20:38.833570+0000","last_unstale":"2023-10-16T11:20:38.842614+0000","last_undegraded":"2023-10-16T11:20:38.842614+0000","last_fullsized":"2023-10-16T11:20:38.842614+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:21.540202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837386+0000","last_change":"2023-10-16T11:20:37.812869+0000","last_active":"2023-10-16T11:20:38.837386+0000","last_peered":"2023-10-16T11:20:38.837386+0000","last_clean":"2023-10-16T11:20:38.837386+0000","last_became_active":"2023-10-16T11:20:37.812556+0000","last_became_peered":"2023-10-16T11:20:37.812556+0000","last_unstale":"2023-10-16T11:20:38.837386+0000","last_undegraded":"2023-10-16T11:20:38.837386+0000","last_fullsized":"2023-10-16T11:20:38.837386+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:21.865730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816718+0000","last_change":"2023-10-16T11:20:25.747818+0000","last_active":"2023-10-16T11:20:37.816718+0000","last_peered":"2023-10-16T11:20:37.816718+0000","last_clean":"2023-10-16T11:20:37.816718+0000","last_became_active":"2023-10-16T11:20:25.747533+0000","last_became_peered":"2023-10-16T11:20:25.747533+0000","last_unstale":"2023-10-16T11:20:37.816718+0000","last_undegraded":"2023-10-16T11:20:37.816718+0000","last_fullsized":"2023-10-16T11:20:37.816718+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843593+0000","last_change":"2023-10-16T11:20:38.841998+0000","last_active":"2023-10-16T11:20:38.843593+0000","last_peered":"2023-10-16T11:20:38.843593+0000","last_clean":"2023-10-16T11:20:38.843593+0000","last_became_active":"2023-10-16T11:20:38.838632+0000","last_became_peered":"2023-10-16T11:20:38.838632+0000","last_unstale":"2023-10-16T11:20:38.843593+0000","last_undegraded":"2023-10-16T11:20:38.843593+0000","last_fullsized":"2023-10-16T11:20:38.843593+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:30.044114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.828808+0000","last_change":"2023-10-16T11:20:37.819685+0000","last_active":"2023-10-16T11:20:38.828808+0000","last_peered":"2023-10-16T11:20:38.828808+0000","last_clean":"2023-10-16T11:20:38.828808+0000","last_became_active":"2023-10-16T11:20:37.819375+0000","last_became_peered":"2023-10-16T11:20:37.819375+0000","last_unstale":"2023-10-16T11:20:38.828808+0000","last_undegraded":"2023-10-16T11:20:38.828808+0000","last_fullsized":"2023-10-16T11:20:38.828808+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:53.867762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814009+0000","last_change":"2023-10-16T11:20:25.746406+0000","last_active":"2023-10-16T11:20:37.814009+0000","last_peered":"2023-10-16T11:20:37.814009+0000","last_clean":"2023-10-16T11:20:37.814009+0000","last_became_active":"2023-10-16T11:20:25.746140+0000","last_became_peered":"2023-10-16T11:20:25.746140+0000","last_unstale":"2023-10-16T11:20:37.814009+0000","last_undegraded":"2023-10-16T11:20:37.814009+0000","last_fullsized":"2023-10-16T11:20:37.814009+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:42.488771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836035+0000","last_change":"2023-10-16T11:20:38.834839+0000","last_active":"2023-10-16T11:20:38.836035+0000","last_peered":"2023-10-16T11:20:38.836035+0000","last_clean":"2023-10-16T11:20:38.836035+0000","last_became_active":"2023-10-16T11:20:38.833518+0000","last_became_peered":"2023-10-16T11:20:38.833518+0000","last_unstale":"2023-10-16T11:20:38.836035+0000","last_undegraded":"2023-10-16T11:20:38.836035+0000","last_fullsized":"2023-10-16T11:20:38.836035+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:32.097784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.827001+0000","last_change":"2023-10-16T11:20:37.820664+0000","last_active":"2023-10-16T11:20:38.827001+0000","last_peered":"2023-10-16T11:20:38.827001+0000","last_clean":"2023-10-16T11:20:38.827001+0000","last_became_active":"2023-10-16T11:20:37.820394+0000","last_became_peered":"2023-10-16T11:20:37.820394+0000","last_unstale":"2023-10-16T11:20:38.827001+0000","last_undegraded":"2023-10-16T11:20:38.827001+0000","last_fullsized":"2023-10-16T11:20:38.827001+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:46.682699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807677+0000","last_change":"2023-10-16T11:20:25.750643+0000","last_active":"2023-10-16T11:20:37.807677+0000","last_peered":"2023-10-16T11:20:37.807677+0000","last_clean":"2023-10-16T11:20:37.807677+0000","last_became_active":"2023-10-16T11:20:25.749980+0000","last_became_peered":"2023-10-16T11:20:25.749980+0000","last_unstale":"2023-10-16T11:20:37.807677+0000","last_undegraded":"2023-10-16T11:20:37.807677+0000","last_fullsized":"2023-10-16T11:20:37.807677+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:41.498772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843389+0000","last_change":"2023-10-16T11:20:38.841575+0000","last_active":"2023-10-16T11:20:38.843389+0000","last_peered":"2023-10-16T11:20:38.843389+0000","last_clean":"2023-10-16T11:20:38.843389+0000","last_became_active":"2023-10-16T11:20:38.841319+0000","last_became_peered":"2023-10-16T11:20:38.841319+0000","last_unstale":"2023-10-16T11:20:38.843389+0000","last_undegraded":"2023-10-16T11:20:38.843389+0000","last_fullsized":"2023-10-16T11:20:38.843389+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:47.690992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.813543+0000","last_change":"2023-10-16T11:20:37.825703+0000","last_active":"2023-10-16T11:20:38.813543+0000","last_peered":"2023-10-16T11:20:38.813543+0000","last_clean":"2023-10-16T11:20:38.813543+0000","last_became_active":"2023-10-16T11:20:37.825425+0000","last_became_peered":"2023-10-16T11:20:37.825425+0000","last_unstale":"2023-10-16T11:20:38.813543+0000","last_undegraded":"2023-10-16T11:20:38.813543+0000","last_fullsized":"2023-10-16T11:20:38.813543+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:42:28.228873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813543+0000","last_change":"2023-10-16T11:20:25.744400+0000","last_active":"2023-10-16T11:20:37.813543+0000","last_peered":"2023-10-16T11:20:37.813543+0000","last_clean":"2023-10-16T11:20:37.813543+0000","last_became_active":"2023-10-16T11:20:25.744118+0000","last_became_peered":"2023-10-16T11:20:25.744118+0000","last_unstale":"2023-10-16T11:20:37.813543+0000","last_undegraded":"2023-10-16T11:20:37.813543+0000","last_fullsized":"2023-10-16T11:20:37.813543+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839580+0000","last_change":"2023-10-16T11:20:38.835993+0000","last_active":"2023-10-16T11:20:38.839580+0000","last_peered":"2023-10-16T11:20:38.839580+0000","last_clean":"2023-10-16T11:20:38.839580+0000","last_became_active":"2023-10-16T11:20:38.835750+0000","last_became_peered":"2023-10-16T11:20:38.835750+0000","last_unstale":"2023-10-16T11:20:38.839580+0000","last_undegraded":"2023-10-16T11:20:38.839580+0000","last_fullsized":"2023-10-16T11:20:38.839580+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:18.097381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.813626+0000","last_change":"2023-10-16T11:20:37.824421+0000","last_active":"2023-10-16T11:20:38.813626+0000","last_peered":"2023-10-16T11:20:38.813626+0000","last_clean":"2023-10-16T11:20:38.813626+0000","last_became_active":"2023-10-16T11:20:37.824028+0000","last_became_peered":"2023-10-16T11:20:37.824028+0000","last_unstale":"2023-10-16T11:20:38.813626+0000","last_undegraded":"2023-10-16T11:20:38.813626+0000","last_fullsized":"2023-10-16T11:20:38.813626+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:57.821106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813784+0000","last_change":"2023-10-16T11:20:25.749096+0000","last_active":"2023-10-16T11:20:37.813784+0000","last_peered":"2023-10-16T11:20:37.813784+0000","last_clean":"2023-10-16T11:20:37.813784+0000","last_became_active":"2023-10-16T11:20:25.748841+0000","last_became_peered":"2023-10-16T11:20:25.748841+0000","last_unstale":"2023-10-16T11:20:37.813784+0000","last_undegraded":"2023-10-16T11:20:37.813784+0000","last_fullsized":"2023-10-16T11:20:37.813784+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836886+0000","last_change":"2023-10-16T11:20:38.830851+0000","last_active":"2023-10-16T11:20:38.836886+0000","last_peered":"2023-10-16T11:20:38.836886+0000","last_clean":"2023-10-16T11:20:38.836886+0000","last_became_active":"2023-10-16T11:20:38.830453+0000","last_became_peered":"2023-10-16T11:20:38.830453+0000","last_unstale":"2023-10-16T11:20:38.836886+0000","last_undegraded":"2023-10-16T11:20:38.836886+0000","last_fullsized":"2023-10-16T11:20:38.836886+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:19.647950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.813660+0000","last_change":"2023-10-16T11:20:37.823025+0000","last_active":"2023-10-16T11:20:38.813660+0000","last_peered":"2023-10-16T11:20:38.813660+0000","last_clean":"2023-10-16T11:20:38.813660+0000","last_became_active":"2023-10-16T11:20:37.822440+0000","last_became_peered":"2023-10-16T11:20:37.822440+0000","last_unstale":"2023-10-16T11:20:38.813660+0000","last_undegraded":"2023-10-16T11:20:38.813660+0000","last_fullsized":"2023-10-16T11:20:38.813660+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:32.410180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.817446+0000","last_change":"2023-10-16T11:20:25.741998+0000","last_active":"2023-10-16T11:20:37.817446+0000","last_peered":"2023-10-16T11:20:37.817446+0000","last_clean":"2023-10-16T11:20:37.817446+0000","last_became_active":"2023-10-16T11:20:25.741482+0000","last_became_peered":"2023-10-16T11:20:25.741482+0000","last_unstale":"2023-10-16T11:20:37.817446+0000","last_undegraded":"2023-10-16T11:20:37.817446+0000","last_fullsized":"2023-10-16T11:20:37.817446+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:01.922741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842595+0000","last_change":"2023-10-16T11:20:38.838960+0000","last_active":"2023-10-16T11:20:38.842595+0000","last_peered":"2023-10-16T11:20:38.842595+0000","last_clean":"2023-10-16T11:20:38.842595+0000","last_became_active":"2023-10-16T11:20:38.838636+0000","last_became_peered":"2023-10-16T11:20:38.838636+0000","last_unstale":"2023-10-16T11:20:38.842595+0000","last_undegraded":"2023-10-16T11:20:38.842595+0000","last_fullsized":"2023-10-16T11:20:38.842595+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:41.240676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.825361+0000","last_change":"2023-10-16T11:20:37.824717+0000","last_active":"2023-10-16T11:20:38.825361+0000","last_peered":"2023-10-16T11:20:38.825361+0000","last_clean":"2023-10-16T11:20:38.825361+0000","last_became_active":"2023-10-16T11:20:37.824431+0000","last_became_peered":"2023-10-16T11:20:37.824431+0000","last_unstale":"2023-10-16T11:20:38.825361+0000","last_undegraded":"2023-10-16T11:20:38.825361+0000","last_fullsized":"2023-10-16T11:20:38.825361+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.825698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807071+0000","last_change":"2023-10-16T11:20:25.750764+0000","last_active":"2023-10-16T11:20:37.807071+0000","last_peered":"2023-10-16T11:20:37.807071+0000","last_clean":"2023-10-16T11:20:37.807071+0000","last_became_active":"2023-10-16T11:20:25.750127+0000","last_became_peered":"2023-10-16T11:20:25.750127+0000","last_unstale":"2023-10-16T11:20:37.807071+0000","last_undegraded":"2023-10-16T11:20:37.807071+0000","last_fullsized":"2023-10-16T11:20:37.807071+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.429724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843537+0000","last_change":"2023-10-16T11:20:38.841800+0000","last_active":"2023-10-16T11:20:38.843537+0000","last_peered":"2023-10-16T11:20:38.843537+0000","last_clean":"2023-10-16T11:20:38.843537+0000","last_became_active":"2023-10-16T11:20:38.841534+0000","last_became_peered":"2023-10-16T11:20:38.841534+0000","last_unstale":"2023-10-16T11:20:38.843537+0000","last_undegraded":"2023-10-16T11:20:38.843537+0000","last_fullsized":"2023-10-16T11:20:38.843537+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:03.575170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.818279+0000","last_change":"2023-10-16T11:20:25.752205+0000","last_active":"2023-10-16T11:20:37.818279+0000","last_peered":"2023-10-16T11:20:37.818279+0000","last_clean":"2023-10-16T11:20:37.818279+0000","last_became_active":"2023-10-16T11:20:25.751915+0000","last_became_peered":"2023-10-16T11:20:25.751915+0000","last_unstale":"2023-10-16T11:20:37.818279+0000","last_undegraded":"2023-10-16T11:20:37.818279+0000","last_fullsized":"2023-10-16T11:20:37.818279+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:43.213188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"236'4","reported_seq":19,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804563+0000","last_change":"2023-10-16T11:20:24.733695+0000","last_active":"2023-10-16T11:20:37.804563+0000","last_peered":"2023-10-16T11:20:37.804563+0000","last_clean":"2023-10-16T11:20:37.804563+0000","last_became_active":"2023-10-16T11:20:24.733413+0000","last_became_peered":"2023-10-16T11:20:24.733413+0000","last_unstale":"2023-10-16T11:20:37.804563+0000","last_undegraded":"2023-10-16T11:20:37.804563+0000","last_fullsized":"2023-10-16T11:20:37.804563+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:34.326531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.833144+0000","last_change":"2023-10-16T11:20:38.824981+0000","last_active":"2023-10-16T11:20:38.833144+0000","last_peered":"2023-10-16T11:20:38.833144+0000","last_clean":"2023-10-16T11:20:38.833144+0000","last_became_active":"2023-10-16T11:20:38.824705+0000","last_became_peered":"2023-10-16T11:20:38.824705+0000","last_unstale":"2023-10-16T11:20:38.833144+0000","last_undegraded":"2023-10-16T11:20:38.833144+0000","last_fullsized":"2023-10-16T11:20:38.833144+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:54.833992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804216+0000","last_change":"2023-10-16T11:20:25.747992+0000","last_active":"2023-10-16T11:20:37.804216+0000","last_peered":"2023-10-16T11:20:37.804216+0000","last_clean":"2023-10-16T11:20:37.804216+0000","last_became_active":"2023-10-16T11:20:25.747735+0000","last_became_peered":"2023-10-16T11:20:25.747735+0000","last_unstale":"2023-10-16T11:20:37.804216+0000","last_undegraded":"2023-10-16T11:20:37.804216+0000","last_fullsized":"2023-10-16T11:20:37.804216+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:49.269723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811335+0000","last_change":"2023-10-16T11:20:24.739822+0000","last_active":"2023-10-16T11:20:37.811335+0000","last_peered":"2023-10-16T11:20:37.811335+0000","last_clean":"2023-10-16T11:20:37.811335+0000","last_became_active":"2023-10-16T11:20:24.739470+0000","last_became_peered":"2023-10-16T11:20:24.739470+0000","last_unstale":"2023-10-16T11:20:37.811335+0000","last_undegraded":"2023-10-16T11:20:37.811335+0000","last_fullsized":"2023-10-16T11:20:37.811335+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:18.641488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843370+0000","last_change":"2023-10-16T11:20:38.840727+0000","last_active":"2023-10-16T11:20:38.843370+0000","last_peered":"2023-10-16T11:20:38.843370+0000","last_clean":"2023-10-16T11:20:38.843370+0000","last_became_active":"2023-10-16T11:20:38.840265+0000","last_became_peered":"2023-10-16T11:20:38.840265+0000","last_unstale":"2023-10-16T11:20:38.843370+0000","last_undegraded":"2023-10-16T11:20:38.843370+0000","last_fullsized":"2023-10-16T11:20:38.843370+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:56.021692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.808200+0000","last_change":"2023-10-16T11:20:25.746049+0000","last_active":"2023-10-16T11:20:37.808200+0000","last_peered":"2023-10-16T11:20:37.808200+0000","last_clean":"2023-10-16T11:20:37.808200+0000","last_became_active":"2023-10-16T11:20:25.745759+0000","last_became_peered":"2023-10-16T11:20:25.745759+0000","last_unstale":"2023-10-16T11:20:37.808200+0000","last_undegraded":"2023-10-16T11:20:37.808200+0000","last_fullsized":"2023-10-16T11:20:37.808200+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:40:29.501267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814266+0000","last_change":"2023-10-16T11:20:24.727708+0000","last_active":"2023-10-16T11:20:37.814266+0000","last_peered":"2023-10-16T11:20:37.814266+0000","last_clean":"2023-10-16T11:20:37.814266+0000","last_became_active":"2023-10-16T11:20:24.727357+0000","last_became_peered":"2023-10-16T11:20:24.727357+0000","last_unstale":"2023-10-16T11:20:37.814266+0000","last_undegraded":"2023-10-16T11:20:37.814266+0000","last_fullsized":"2023-10-16T11:20:37.814266+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:32.307241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834217+0000","last_change":"2023-10-16T11:20:38.830140+0000","last_active":"2023-10-16T11:20:38.834217+0000","last_peered":"2023-10-16T11:20:38.834217+0000","last_clean":"2023-10-16T11:20:38.834217+0000","last_became_active":"2023-10-16T11:20:38.829866+0000","last_became_peered":"2023-10-16T11:20:38.829866+0000","last_unstale":"2023-10-16T11:20:38.834217+0000","last_undegraded":"2023-10-16T11:20:38.834217+0000","last_fullsized":"2023-10-16T11:20:38.834217+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:57.577794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813997+0000","last_change":"2023-10-16T11:20:25.742133+0000","last_active":"2023-10-16T11:20:37.813997+0000","last_peered":"2023-10-16T11:20:37.813997+0000","last_clean":"2023-10-16T11:20:37.813997+0000","last_became_active":"2023-10-16T11:20:25.741865+0000","last_became_peered":"2023-10-16T11:20:25.741865+0000","last_unstale":"2023-10-16T11:20:37.813997+0000","last_undegraded":"2023-10-16T11:20:37.813997+0000","last_fullsized":"2023-10-16T11:20:37.813997+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:25.230100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804262+0000","last_change":"2023-10-16T11:20:24.722283+0000","last_active":"2023-10-16T11:20:37.804262+0000","last_peered":"2023-10-16T11:20:37.804262+0000","last_clean":"2023-10-16T11:20:37.804262+0000","last_became_active":"2023-10-16T11:20:24.721789+0000","last_became_peered":"2023-10-16T11:20:24.721789+0000","last_unstale":"2023-10-16T11:20:37.804262+0000","last_undegraded":"2023-10-16T11:20:37.804262+0000","last_fullsized":"2023-10-16T11:20:37.804262+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:37.116862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840495+0000","last_change":"2023-10-16T11:20:38.839286+0000","last_active":"2023-10-16T11:20:38.840495+0000","last_peered":"2023-10-16T11:20:38.840495+0000","last_clean":"2023-10-16T11:20:38.840495+0000","last_became_active":"2023-10-16T11:20:38.839008+0000","last_became_peered":"2023-10-16T11:20:38.839008+0000","last_unstale":"2023-10-16T11:20:38.840495+0000","last_undegraded":"2023-10-16T11:20:38.840495+0000","last_fullsized":"2023-10-16T11:20:38.840495+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:25.776247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816721+0000","last_change":"2023-10-16T11:20:25.748913+0000","last_active":"2023-10-16T11:20:37.816721+0000","last_peered":"2023-10-16T11:20:37.816721+0000","last_clean":"2023-10-16T11:20:37.816721+0000","last_became_active":"2023-10-16T11:20:25.748588+0000","last_became_peered":"2023-10-16T11:20:25.748588+0000","last_unstale":"2023-10-16T11:20:37.816721+0000","last_undegraded":"2023-10-16T11:20:37.816721+0000","last_fullsized":"2023-10-16T11:20:37.816721+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813877+0000","last_change":"2023-10-16T11:20:24.733622+0000","last_active":"2023-10-16T11:20:37.813877+0000","last_peered":"2023-10-16T11:20:37.813877+0000","last_clean":"2023-10-16T11:20:37.813877+0000","last_became_active":"2023-10-16T11:20:24.733342+0000","last_became_peered":"2023-10-16T11:20:24.733342+0000","last_unstale":"2023-10-16T11:20:37.813877+0000","last_undegraded":"2023-10-16T11:20:37.813877+0000","last_fullsized":"2023-10-16T11:20:37.813877+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:10.962482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837163+0000","last_change":"2023-10-16T11:20:38.832530+0000","last_active":"2023-10-16T11:20:38.837163+0000","last_peered":"2023-10-16T11:20:38.837163+0000","last_clean":"2023-10-16T11:20:38.837163+0000","last_became_active":"2023-10-16T11:20:38.832247+0000","last_became_peered":"2023-10-16T11:20:38.832247+0000","last_unstale":"2023-10-16T11:20:38.837163+0000","last_undegraded":"2023-10-16T11:20:38.837163+0000","last_fullsized":"2023-10-16T11:20:38.837163+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:51.097103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806850+0000","last_change":"2023-10-16T11:20:25.751310+0000","last_active":"2023-10-16T11:20:37.806850+0000","last_peered":"2023-10-16T11:20:37.806850+0000","last_clean":"2023-10-16T11:20:37.806850+0000","last_became_active":"2023-10-16T11:20:25.751042+0000","last_became_peered":"2023-10-16T11:20:25.751042+0000","last_unstale":"2023-10-16T11:20:37.806850+0000","last_undegraded":"2023-10-16T11:20:37.806850+0000","last_fullsized":"2023-10-16T11:20:37.806850+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806543+0000","last_change":"2023-10-16T11:20:24.739609+0000","last_active":"2023-10-16T11:20:37.806543+0000","last_peered":"2023-10-16T11:20:37.806543+0000","last_clean":"2023-10-16T11:20:37.806543+0000","last_became_active":"2023-10-16T11:20:24.739275+0000","last_became_peered":"2023-10-16T11:20:24.739275+0000","last_unstale":"2023-10-16T11:20:37.806543+0000","last_undegraded":"2023-10-16T11:20:37.806543+0000","last_fullsized":"2023-10-16T11:20:37.806543+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:47.827300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837974+0000","last_change":"2023-10-16T11:20:38.835827+0000","last_active":"2023-10-16T11:20:38.837974+0000","last_peered":"2023-10-16T11:20:38.837974+0000","last_clean":"2023-10-16T11:20:38.837974+0000","last_became_active":"2023-10-16T11:20:38.835538+0000","last_became_peered":"2023-10-16T11:20:38.835538+0000","last_unstale":"2023-10-16T11:20:38.837974+0000","last_undegraded":"2023-10-16T11:20:38.837974+0000","last_fullsized":"2023-10-16T11:20:38.837974+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:21.628073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814773+0000","last_change":"2023-10-16T11:20:25.744793+0000","last_active":"2023-10-16T11:20:37.814773+0000","last_peered":"2023-10-16T11:20:37.814773+0000","last_clean":"2023-10-16T11:20:37.814773+0000","last_became_active":"2023-10-16T11:20:25.744521+0000","last_became_peered":"2023-10-16T11:20:25.744521+0000","last_unstale":"2023-10-16T11:20:37.814773+0000","last_undegraded":"2023-10-16T11:20:37.814773+0000","last_fullsized":"2023-10-16T11:20:37.814773+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:44.539888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807124+0000","last_change":"2023-10-16T11:20:24.723659+0000","last_active":"2023-10-16T11:20:37.807124+0000","last_peered":"2023-10-16T11:20:37.807124+0000","last_clean":"2023-10-16T11:20:37.807124+0000","last_became_active":"2023-10-16T11:20:24.723342+0000","last_became_peered":"2023-10-16T11:20:24.723342+0000","last_unstale":"2023-10-16T11:20:37.807124+0000","last_undegraded":"2023-10-16T11:20:37.807124+0000","last_fullsized":"2023-10-16T11:20:37.807124+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:29.707065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836788+0000","last_change":"2023-10-16T11:20:38.834173+0000","last_active":"2023-10-16T11:20:38.836788+0000","last_peered":"2023-10-16T11:20:38.836788+0000","last_clean":"2023-10-16T11:20:38.836788+0000","last_became_active":"2023-10-16T11:20:38.833870+0000","last_became_peered":"2023-10-16T11:20:38.833870+0000","last_unstale":"2023-10-16T11:20:38.836788+0000","last_undegraded":"2023-10-16T11:20:38.836788+0000","last_fullsized":"2023-10-16T11:20:38.836788+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:57.302607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.815911+0000","last_change":"2023-10-16T11:20:25.748568+0000","last_active":"2023-10-16T11:20:37.815911+0000","last_peered":"2023-10-16T11:20:37.815911+0000","last_clean":"2023-10-16T11:20:37.815911+0000","last_became_active":"2023-10-16T11:20:25.748318+0000","last_became_peered":"2023-10-16T11:20:25.748318+0000","last_unstale":"2023-10-16T11:20:37.815911+0000","last_undegraded":"2023-10-16T11:20:37.815911+0000","last_fullsized":"2023-10-16T11:20:37.815911+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.815289+0000","last_change":"2023-10-16T11:20:24.736095+0000","last_active":"2023-10-16T11:20:37.815289+0000","last_peered":"2023-10-16T11:20:37.815289+0000","last_clean":"2023-10-16T11:20:37.815289+0000","last_became_active":"2023-10-16T11:20:24.735775+0000","last_became_peered":"2023-10-16T11:20:24.735775+0000","last_unstale":"2023-10-16T11:20:37.815289+0000","last_undegraded":"2023-10-16T11:20:37.815289+0000","last_fullsized":"2023-10-16T11:20:37.815289+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:35.485188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842751+0000","last_change":"2023-10-16T11:20:38.840186+0000","last_active":"2023-10-16T11:20:38.842751+0000","last_peered":"2023-10-16T11:20:38.842751+0000","last_clean":"2023-10-16T11:20:38.842751+0000","last_became_active":"2023-10-16T11:20:38.838073+0000","last_became_peered":"2023-10-16T11:20:38.838073+0000","last_unstale":"2023-10-16T11:20:38.842751+0000","last_undegraded":"2023-10-16T11:20:38.842751+0000","last_fullsized":"2023-10-16T11:20:38.842751+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:52.880299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810525+0000","last_change":"2023-10-16T11:20:25.745589+0000","last_active":"2023-10-16T11:20:37.810525+0000","last_peered":"2023-10-16T11:20:37.810525+0000","last_clean":"2023-10-16T11:20:37.810525+0000","last_became_active":"2023-10-16T11:20:25.745297+0000","last_became_peered":"2023-10-16T11:20:25.745297+0000","last_unstale":"2023-10-16T11:20:37.810525+0000","last_undegraded":"2023-10-16T11:20:37.810525+0000","last_fullsized":"2023-10-16T11:20:37.810525+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:17.743037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806958+0000","last_change":"2023-10-16T11:20:24.738634+0000","last_active":"2023-10-16T11:20:37.806958+0000","last_peered":"2023-10-16T11:20:37.806958+0000","last_clean":"2023-10-16T11:20:37.806958+0000","last_became_active":"2023-10-16T11:20:24.738372+0000","last_became_peered":"2023-10-16T11:20:24.738372+0000","last_unstale":"2023-10-16T11:20:37.806958+0000","last_undegraded":"2023-10-16T11:20:37.806958+0000","last_fullsized":"2023-10-16T11:20:37.806958+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:23.747207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832314+0000","last_change":"2023-10-16T11:20:38.820038+0000","last_active":"2023-10-16T11:20:38.832314+0000","last_peered":"2023-10-16T11:20:38.832314+0000","last_clean":"2023-10-16T11:20:38.832314+0000","last_became_active":"2023-10-16T11:20:38.819796+0000","last_became_peered":"2023-10-16T11:20:38.819796+0000","last_unstale":"2023-10-16T11:20:38.832314+0000","last_undegraded":"2023-10-16T11:20:38.832314+0000","last_fullsized":"2023-10-16T11:20:38.832314+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:57.773345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806800+0000","last_change":"2023-10-16T11:20:25.748579+0000","last_active":"2023-10-16T11:20:37.806800+0000","last_peered":"2023-10-16T11:20:37.806800+0000","last_clean":"2023-10-16T11:20:37.806800+0000","last_became_active":"2023-10-16T11:20:25.748347+0000","last_became_peered":"2023-10-16T11:20:25.748347+0000","last_unstale":"2023-10-16T11:20:37.806800+0000","last_undegraded":"2023-10-16T11:20:37.806800+0000","last_fullsized":"2023-10-16T11:20:37.806800+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"236'3","reported_seq":18,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814108+0000","last_change":"2023-10-16T11:20:24.728938+0000","last_active":"2023-10-16T11:20:37.814108+0000","last_peered":"2023-10-16T11:20:37.814108+0000","last_clean":"2023-10-16T11:20:37.814108+0000","last_became_active":"2023-10-16T11:20:24.728638+0000","last_became_peered":"2023-10-16T11:20:24.728638+0000","last_unstale":"2023-10-16T11:20:37.814108+0000","last_undegraded":"2023-10-16T11:20:37.814108+0000","last_fullsized":"2023-10-16T11:20:37.814108+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:42.082000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842632+0000","last_change":"2023-10-16T11:20:38.836478+0000","last_active":"2023-10-16T11:20:38.842632+0000","last_peered":"2023-10-16T11:20:38.842632+0000","last_clean":"2023-10-16T11:20:38.842632+0000","last_became_active":"2023-10-16T11:20:38.836190+0000","last_became_peered":"2023-10-16T11:20:38.836190+0000","last_unstale":"2023-10-16T11:20:38.842632+0000","last_undegraded":"2023-10-16T11:20:38.842632+0000","last_fullsized":"2023-10-16T11:20:38.842632+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:52.880299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816526+0000","last_change":"2023-10-16T11:20:25.751420+0000","last_active":"2023-10-16T11:20:37.816526+0000","last_peered":"2023-10-16T11:20:37.816526+0000","last_clean":"2023-10-16T11:20:37.816526+0000","last_became_active":"2023-10-16T11:20:25.751161+0000","last_became_peered":"2023-10-16T11:20:25.751161+0000","last_unstale":"2023-10-16T11:20:37.816526+0000","last_undegraded":"2023-10-16T11:20:37.816526+0000","last_fullsized":"2023-10-16T11:20:37.816526+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"237'8","reported_seq":23,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807800+0000","last_change":"2023-10-16T11:20:24.724201+0000","last_active":"2023-10-16T11:20:37.807800+0000","last_peered":"2023-10-16T11:20:37.807800+0000","last_clean":"2023-10-16T11:20:37.807800+0000","last_became_active":"2023-10-16T11:20:24.723918+0000","last_became_peered":"2023-10-16T11:20:24.723918+0000","last_unstale":"2023-10-16T11:20:37.807800+0000","last_undegraded":"2023-10-16T11:20:37.807800+0000","last_fullsized":"2023-10-16T11:20:37.807800+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:55.483353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840202+0000","last_change":"2023-10-16T11:20:38.835082+0000","last_active":"2023-10-16T11:20:38.840202+0000","last_peered":"2023-10-16T11:20:38.840202+0000","last_clean":"2023-10-16T11:20:38.840202+0000","last_became_active":"2023-10-16T11:20:38.834792+0000","last_became_peered":"2023-10-16T11:20:38.834792+0000","last_unstale":"2023-10-16T11:20:38.840202+0000","last_undegraded":"2023-10-16T11:20:38.840202+0000","last_fullsized":"2023-10-16T11:20:38.840202+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:55.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.819328+0000","last_change":"2023-10-16T11:20:25.742201+0000","last_active":"2023-10-16T11:20:37.819328+0000","last_peered":"2023-10-16T11:20:37.819328+0000","last_clean":"2023-10-16T11:20:37.819328+0000","last_became_active":"2023-10-16T11:20:25.741871+0000","last_became_peered":"2023-10-16T11:20:25.741871+0000","last_unstale":"2023-10-16T11:20:37.819328+0000","last_undegraded":"2023-10-16T11:20:37.819328+0000","last_fullsized":"2023-10-16T11:20:37.819328+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:18:28.417174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804356+0000","last_change":"2023-10-16T11:20:24.725162+0000","last_active":"2023-10-16T11:20:37.804356+0000","last_peered":"2023-10-16T11:20:37.804356+0000","last_clean":"2023-10-16T11:20:37.804356+0000","last_became_active":"2023-10-16T11:20:24.724795+0000","last_became_peered":"2023-10-16T11:20:24.724795+0000","last_unstale":"2023-10-16T11:20:37.804356+0000","last_undegraded":"2023-10-16T11:20:37.804356+0000","last_fullsized":"2023-10-16T11:20:37.804356+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:25.854243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832471+0000","last_change":"2023-10-16T11:20:38.830717+0000","last_active":"2023-10-16T11:20:38.832471+0000","last_peered":"2023-10-16T11:20:38.832471+0000","last_clean":"2023-10-16T11:20:38.832471+0000","last_became_active":"2023-10-16T11:20:38.830464+0000","last_became_peered":"2023-10-16T11:20:38.830464+0000","last_unstale":"2023-10-16T11:20:38.832471+0000","last_undegraded":"2023-10-16T11:20:38.832471+0000","last_fullsized":"2023-10-16T11:20:38.832471+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:52:15.104758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816809+0000","last_change":"2023-10-16T11:20:25.739623+0000","last_active":"2023-10-16T11:20:37.816809+0000","last_peered":"2023-10-16T11:20:37.816809+0000","last_clean":"2023-10-16T11:20:37.816809+0000","last_became_active":"2023-10-16T11:20:25.739321+0000","last_became_peered":"2023-10-16T11:20:25.739321+0000","last_unstale":"2023-10-16T11:20:37.816809+0000","last_undegraded":"2023-10-16T11:20:37.816809+0000","last_fullsized":"2023-10-16T11:20:37.816809+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813068+0000","last_change":"2023-10-16T11:20:24.735120+0000","last_active":"2023-10-16T11:20:37.813068+0000","last_peered":"2023-10-16T11:20:37.813068+0000","last_clean":"2023-10-16T11:20:37.813068+0000","last_became_active":"2023-10-16T11:20:24.734857+0000","last_became_peered":"2023-10-16T11:20:24.734857+0000","last_unstale":"2023-10-16T11:20:37.813068+0000","last_undegraded":"2023-10-16T11:20:37.813068+0000","last_fullsized":"2023-10-16T11:20:37.813068+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:03.791128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843414+0000","last_change":"2023-10-16T11:20:38.837581+0000","last_active":"2023-10-16T11:20:38.843414+0000","last_peered":"2023-10-16T11:20:38.843414+0000","last_clean":"2023-10-16T11:20:38.843414+0000","last_became_active":"2023-10-16T11:20:38.836911+0000","last_became_peered":"2023-10-16T11:20:38.836911+0000","last_unstale":"2023-10-16T11:20:38.843414+0000","last_undegraded":"2023-10-16T11:20:38.843414+0000","last_fullsized":"2023-10-16T11:20:38.843414+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:40.355466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804303+0000","last_change":"2023-10-16T11:20:25.737997+0000","last_active":"2023-10-16T11:20:37.804303+0000","last_peered":"2023-10-16T11:20:37.804303+0000","last_clean":"2023-10-16T11:20:37.804303+0000","last_became_active":"2023-10-16T11:20:25.737314+0000","last_became_peered":"2023-10-16T11:20:25.737314+0000","last_unstale":"2023-10-16T11:20:37.804303+0000","last_undegraded":"2023-10-16T11:20:37.804303+0000","last_fullsized":"2023-10-16T11:20:37.804303+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:44.489552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813483+0000","last_change":"2023-10-16T11:20:24.733587+0000","last_active":"2023-10-16T11:20:37.813483+0000","last_peered":"2023-10-16T11:20:37.813483+0000","last_clean":"2023-10-16T11:20:37.813483+0000","last_became_active":"2023-10-16T11:20:24.733160+0000","last_became_peered":"2023-10-16T11:20:24.733160+0000","last_unstale":"2023-10-16T11:20:37.813483+0000","last_undegraded":"2023-10-16T11:20:37.813483+0000","last_fullsized":"2023-10-16T11:20:37.813483+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:31.939688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840303+0000","last_change":"2023-10-16T11:20:38.836459+0000","last_active":"2023-10-16T11:20:38.840303+0000","last_peered":"2023-10-16T11:20:38.840303+0000","last_clean":"2023-10-16T11:20:38.840303+0000","last_became_active":"2023-10-16T11:20:38.836106+0000","last_became_peered":"2023-10-16T11:20:38.836106+0000","last_unstale":"2023-10-16T11:20:38.840303+0000","last_undegraded":"2023-10-16T11:20:38.840303+0000","last_fullsized":"2023-10-16T11:20:38.840303+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:46.037088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814690+0000","last_change":"2023-10-16T11:20:25.748924+0000","last_active":"2023-10-16T11:20:37.814690+0000","last_peered":"2023-10-16T11:20:37.814690+0000","last_clean":"2023-10-16T11:20:37.814690+0000","last_became_active":"2023-10-16T11:20:25.748637+0000","last_became_peered":"2023-10-16T11:20:25.748637+0000","last_unstale":"2023-10-16T11:20:37.814690+0000","last_undegraded":"2023-10-16T11:20:37.814690+0000","last_fullsized":"2023-10-16T11:20:37.814690+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:40.894330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"236'8","reported_seq":23,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.818188+0000","last_change":"2023-10-16T11:20:24.740384+0000","last_active":"2023-10-16T11:20:37.818188+0000","last_peered":"2023-10-16T11:20:37.818188+0000","last_clean":"2023-10-16T11:20:37.818188+0000","last_became_active":"2023-10-16T11:20:24.739999+0000","last_became_peered":"2023-10-16T11:20:24.739999+0000","last_unstale":"2023-10-16T11:20:37.818188+0000","last_undegraded":"2023-10-16T11:20:37.818188+0000","last_fullsized":"2023-10-16T11:20:37.818188+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:31.286100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842575+0000","last_change":"2023-10-16T11:20:38.838947+0000","last_active":"2023-10-16T11:20:38.842575+0000","last_peered":"2023-10-16T11:20:38.842575+0000","last_clean":"2023-10-16T11:20:38.842575+0000","last_became_active":"2023-10-16T11:20:38.838463+0000","last_became_peered":"2023-10-16T11:20:38.838463+0000","last_unstale":"2023-10-16T11:20:38.842575+0000","last_undegraded":"2023-10-16T11:20:38.842575+0000","last_fullsized":"2023-10-16T11:20:38.842575+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:32.097784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812735+0000","last_change":"2023-10-16T11:20:25.749009+0000","last_active":"2023-10-16T11:20:37.812735+0000","last_peered":"2023-10-16T11:20:37.812735+0000","last_clean":"2023-10-16T11:20:37.812735+0000","last_became_active":"2023-10-16T11:20:25.748708+0000","last_became_peered":"2023-10-16T11:20:25.748708+0000","last_unstale":"2023-10-16T11:20:37.812735+0000","last_undegraded":"2023-10-16T11:20:37.812735+0000","last_fullsized":"2023-10-16T11:20:37.812735+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:55.578524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804549+0000","last_change":"2023-10-16T11:20:24.731614+0000","last_active":"2023-10-16T11:20:37.804549+0000","last_peered":"2023-10-16T11:20:37.804549+0000","last_clean":"2023-10-16T11:20:37.804549+0000","last_became_active":"2023-10-16T11:20:24.731329+0000","last_became_peered":"2023-10-16T11:20:24.731329+0000","last_unstale":"2023-10-16T11:20:37.804549+0000","last_undegraded":"2023-10-16T11:20:37.804549+0000","last_fullsized":"2023-10-16T11:20:37.804549+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:39.887304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844250+0000","last_change":"2023-10-16T11:20:38.838636+0000","last_active":"2023-10-16T11:20:38.844250+0000","last_peered":"2023-10-16T11:20:38.844250+0000","last_clean":"2023-10-16T11:20:38.844250+0000","last_became_active":"2023-10-16T11:20:38.837985+0000","last_became_peered":"2023-10-16T11:20:38.837985+0000","last_unstale":"2023-10-16T11:20:38.844250+0000","last_undegraded":"2023-10-16T11:20:38.844250+0000","last_fullsized":"2023-10-16T11:20:38.844250+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:06.743345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813408+0000","last_change":"2023-10-16T11:20:25.750514+0000","last_active":"2023-10-16T11:20:37.813408+0000","last_peered":"2023-10-16T11:20:37.813408+0000","last_clean":"2023-10-16T11:20:37.813408+0000","last_became_active":"2023-10-16T11:20:25.750226+0000","last_became_peered":"2023-10-16T11:20:25.750226+0000","last_unstale":"2023-10-16T11:20:37.813408+0000","last_undegraded":"2023-10-16T11:20:37.813408+0000","last_fullsized":"2023-10-16T11:20:37.813408+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:02.546128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.820950+0000","last_change":"2023-10-16T11:20:24.740869+0000","last_active":"2023-10-16T11:20:37.820950+0000","last_peered":"2023-10-16T11:20:37.820950+0000","last_clean":"2023-10-16T11:20:37.820950+0000","last_became_active":"2023-10-16T11:20:24.740527+0000","last_became_peered":"2023-10-16T11:20:24.740527+0000","last_unstale":"2023-10-16T11:20:37.820950+0000","last_undegraded":"2023-10-16T11:20:37.820950+0000","last_fullsized":"2023-10-16T11:20:37.820950+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:49.172459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838096+0000","last_change":"2023-10-16T11:20:38.831925+0000","last_active":"2023-10-16T11:20:38.838096+0000","last_peered":"2023-10-16T11:20:38.838096+0000","last_clean":"2023-10-16T11:20:38.838096+0000","last_became_active":"2023-10-16T11:20:38.831611+0000","last_became_peered":"2023-10-16T11:20:38.831611+0000","last_unstale":"2023-10-16T11:20:38.838096+0000","last_undegraded":"2023-10-16T11:20:38.838096+0000","last_fullsized":"2023-10-16T11:20:38.838096+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:16.818119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.817592+0000","last_change":"2023-10-16T11:20:25.749628+0000","last_active":"2023-10-16T11:20:37.817592+0000","last_peered":"2023-10-16T11:20:37.817592+0000","last_clean":"2023-10-16T11:20:37.817592+0000","last_became_active":"2023-10-16T11:20:25.749358+0000","last_became_peered":"2023-10-16T11:20:25.749358+0000","last_unstale":"2023-10-16T11:20:37.817592+0000","last_undegraded":"2023-10-16T11:20:37.817592+0000","last_fullsized":"2023-10-16T11:20:37.817592+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:39.916608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811438+0000","last_change":"2023-10-16T11:20:24.735496+0000","last_active":"2023-10-16T11:20:37.811438+0000","last_peered":"2023-10-16T11:20:37.811438+0000","last_clean":"2023-10-16T11:20:37.811438+0000","last_became_active":"2023-10-16T11:20:24.734988+0000","last_became_peered":"2023-10-16T11:20:24.734988+0000","last_unstale":"2023-10-16T11:20:37.811438+0000","last_undegraded":"2023-10-16T11:20:37.811438+0000","last_fullsized":"2023-10-16T11:20:37.811438+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:22.060729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834542+0000","last_change":"2023-10-16T11:20:38.830069+0000","last_active":"2023-10-16T11:20:38.834542+0000","last_peered":"2023-10-16T11:20:38.834542+0000","last_clean":"2023-10-16T11:20:38.834542+0000","last_became_active":"2023-10-16T11:20:38.829778+0000","last_became_peered":"2023-10-16T11:20:38.829778+0000","last_unstale":"2023-10-16T11:20:38.834542+0000","last_undegraded":"2023-10-16T11:20:38.834542+0000","last_fullsized":"2023-10-16T11:20:38.834542+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:02.357965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812039+0000","last_change":"2023-10-16T11:20:25.746364+0000","last_active":"2023-10-16T11:20:37.812039+0000","last_peered":"2023-10-16T11:20:37.812039+0000","last_clean":"2023-10-16T11:20:37.812039+0000","last_became_active":"2023-10-16T11:20:25.746108+0000","last_became_peered":"2023-10-16T11:20:25.746108+0000","last_unstale":"2023-10-16T11:20:37.812039+0000","last_undegraded":"2023-10-16T11:20:37.812039+0000","last_fullsized":"2023-10-16T11:20:37.812039+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:03.593372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804264+0000","last_change":"2023-10-16T11:20:24.727424+0000","last_active":"2023-10-16T11:20:37.804264+0000","last_peered":"2023-10-16T11:20:37.804264+0000","last_clean":"2023-10-16T11:20:37.804264+0000","last_became_active":"2023-10-16T11:20:24.727122+0000","last_became_peered":"2023-10-16T11:20:24.727122+0000","last_unstale":"2023-10-16T11:20:37.804264+0000","last_undegraded":"2023-10-16T11:20:37.804264+0000","last_fullsized":"2023-10-16T11:20:37.804264+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:38.786539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832268+0000","last_change":"2023-10-16T11:20:38.830412+0000","last_active":"2023-10-16T11:20:38.832268+0000","last_peered":"2023-10-16T11:20:38.832268+0000","last_clean":"2023-10-16T11:20:38.832268+0000","last_became_active":"2023-10-16T11:20:38.830113+0000","last_became_peered":"2023-10-16T11:20:38.830113+0000","last_unstale":"2023-10-16T11:20:38.832268+0000","last_undegraded":"2023-10-16T11:20:38.832268+0000","last_fullsized":"2023-10-16T11:20:38.832268+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:26.326791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806635+0000","last_change":"2023-10-16T11:20:25.741687+0000","last_active":"2023-10-16T11:20:37.806635+0000","last_peered":"2023-10-16T11:20:37.806635+0000","last_clean":"2023-10-16T11:20:37.806635+0000","last_became_active":"2023-10-16T11:20:25.741022+0000","last_became_peered":"2023-10-16T11:20:25.741022+0000","last_unstale":"2023-10-16T11:20:37.806635+0000","last_undegraded":"2023-10-16T11:20:37.806635+0000","last_fullsized":"2023-10-16T11:20:37.806635+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:23.748016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812127+0000","last_change":"2023-10-16T11:20:24.738500+0000","last_active":"2023-10-16T11:20:37.812127+0000","last_peered":"2023-10-16T11:20:37.812127+0000","last_clean":"2023-10-16T11:20:37.812127+0000","last_became_active":"2023-10-16T11:20:24.737980+0000","last_became_peered":"2023-10-16T11:20:24.737980+0000","last_unstale":"2023-10-16T11:20:37.812127+0000","last_undegraded":"2023-10-16T11:20:37.812127+0000","last_fullsized":"2023-10-16T11:20:37.812127+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:15.374424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838491+0000","last_change":"2023-10-16T11:20:38.833307+0000","last_active":"2023-10-16T11:20:38.838491+0000","last_peered":"2023-10-16T11:20:38.838491+0000","last_clean":"2023-10-16T11:20:38.838491+0000","last_became_active":"2023-10-16T11:20:38.832896+0000","last_became_peered":"2023-10-16T11:20:38.832896+0000","last_unstale":"2023-10-16T11:20:38.838491+0000","last_undegraded":"2023-10-16T11:20:38.838491+0000","last_fullsized":"2023-10-16T11:20:38.838491+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:02.978176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.818864+0000","last_change":"2023-10-16T11:20:25.752050+0000","last_active":"2023-10-16T11:20:37.818864+0000","last_peered":"2023-10-16T11:20:37.818864+0000","last_clean":"2023-10-16T11:20:37.818864+0000","last_became_active":"2023-10-16T11:20:25.751760+0000","last_became_peered":"2023-10-16T11:20:25.751760+0000","last_unstale":"2023-10-16T11:20:37.818864+0000","last_undegraded":"2023-10-16T11:20:37.818864+0000","last_fullsized":"2023-10-16T11:20:37.818864+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:40.238520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810113+0000","last_change":"2023-10-16T11:20:24.727961+0000","last_active":"2023-10-16T11:20:37.810113+0000","last_peered":"2023-10-16T11:20:37.810113+0000","last_clean":"2023-10-16T11:20:37.810113+0000","last_became_active":"2023-10-16T11:20:24.727467+0000","last_became_peered":"2023-10-16T11:20:24.727467+0000","last_unstale":"2023-10-16T11:20:37.810113+0000","last_undegraded":"2023-10-16T11:20:37.810113+0000","last_fullsized":"2023-10-16T11:20:37.810113+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:33.847435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843397+0000","last_change":"2023-10-16T11:20:38.837408+0000","last_active":"2023-10-16T11:20:38.843397+0000","last_peered":"2023-10-16T11:20:38.843397+0000","last_clean":"2023-10-16T11:20:38.843397+0000","last_became_active":"2023-10-16T11:20:38.836556+0000","last_became_peered":"2023-10-16T11:20:38.836556+0000","last_unstale":"2023-10-16T11:20:38.843397+0000","last_undegraded":"2023-10-16T11:20:38.843397+0000","last_fullsized":"2023-10-16T11:20:38.843397+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:19.129128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811569+0000","last_change":"2023-10-16T11:20:25.743908+0000","last_active":"2023-10-16T11:20:37.811569+0000","last_peered":"2023-10-16T11:20:37.811569+0000","last_clean":"2023-10-16T11:20:37.811569+0000","last_became_active":"2023-10-16T11:20:25.743570+0000","last_became_peered":"2023-10-16T11:20:25.743570+0000","last_unstale":"2023-10-16T11:20:37.811569+0000","last_undegraded":"2023-10-16T11:20:37.811569+0000","last_fullsized":"2023-10-16T11:20:37.811569+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:53.089548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810602+0000","last_change":"2023-10-16T11:20:24.735291+0000","last_active":"2023-10-16T11:20:37.810602+0000","last_peered":"2023-10-16T11:20:37.810602+0000","last_clean":"2023-10-16T11:20:37.810602+0000","last_became_active":"2023-10-16T11:20:24.734998+0000","last_became_peered":"2023-10-16T11:20:24.734998+0000","last_unstale":"2023-10-16T11:20:37.810602+0000","last_undegraded":"2023-10-16T11:20:37.810602+0000","last_fullsized":"2023-10-16T11:20:37.810602+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:26.682154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837950+0000","last_change":"2023-10-16T11:20:38.829921+0000","last_active":"2023-10-16T11:20:38.837950+0000","last_peered":"2023-10-16T11:20:38.837950+0000","last_clean":"2023-10-16T11:20:38.837950+0000","last_became_active":"2023-10-16T11:20:38.829615+0000","last_became_peered":"2023-10-16T11:20:38.829615+0000","last_unstale":"2023-10-16T11:20:38.837950+0000","last_undegraded":"2023-10-16T11:20:38.837950+0000","last_fullsized":"2023-10-16T11:20:38.837950+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:33.255692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811289+0000","last_change":"2023-10-16T11:20:25.736861+0000","last_active":"2023-10-16T11:20:37.811289+0000","last_peered":"2023-10-16T11:20:37.811289+0000","last_clean":"2023-10-16T11:20:37.811289+0000","last_became_active":"2023-10-16T11:20:25.736595+0000","last_became_peered":"2023-10-16T11:20:25.736595+0000","last_unstale":"2023-10-16T11:20:37.811289+0000","last_undegraded":"2023-10-16T11:20:37.811289+0000","last_fullsized":"2023-10-16T11:20:37.811289+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:28.356199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811658+0000","last_change":"2023-10-16T11:20:24.729252+0000","last_active":"2023-10-16T11:20:37.811658+0000","last_peered":"2023-10-16T11:20:37.811658+0000","last_clean":"2023-10-16T11:20:37.811658+0000","last_became_active":"2023-10-16T11:20:24.729048+0000","last_became_peered":"2023-10-16T11:20:24.729048+0000","last_unstale":"2023-10-16T11:20:37.811658+0000","last_undegraded":"2023-10-16T11:20:37.811658+0000","last_fullsized":"2023-10-16T11:20:37.811658+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.587854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837698+0000","last_change":"2023-10-16T11:20:38.832394+0000","last_active":"2023-10-16T11:20:38.837698+0000","last_peered":"2023-10-16T11:20:38.837698+0000","last_clean":"2023-10-16T11:20:38.837698+0000","last_became_active":"2023-10-16T11:20:38.829399+0000","last_became_peered":"2023-10-16T11:20:38.829399+0000","last_unstale":"2023-10-16T11:20:38.837698+0000","last_undegraded":"2023-10-16T11:20:38.837698+0000","last_fullsized":"2023-10-16T11:20:38.837698+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:07.323719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816137+0000","last_change":"2023-10-16T11:20:25.748002+0000","last_active":"2023-10-16T11:20:37.816137+0000","last_peered":"2023-10-16T11:20:37.816137+0000","last_clean":"2023-10-16T11:20:37.816137+0000","last_became_active":"2023-10-16T11:20:25.747521+0000","last_became_peered":"2023-10-16T11:20:25.747521+0000","last_unstale":"2023-10-16T11:20:37.816137+0000","last_undegraded":"2023-10-16T11:20:37.816137+0000","last_fullsized":"2023-10-16T11:20:37.816137+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810940+0000","last_change":"2023-10-16T11:20:24.738296+0000","last_active":"2023-10-16T11:20:37.810940+0000","last_peered":"2023-10-16T11:20:37.810940+0000","last_clean":"2023-10-16T11:20:37.810940+0000","last_became_active":"2023-10-16T11:20:24.737992+0000","last_became_peered":"2023-10-16T11:20:24.737992+0000","last_unstale":"2023-10-16T11:20:37.810940+0000","last_undegraded":"2023-10-16T11:20:37.810940+0000","last_fullsized":"2023-10-16T11:20:37.810940+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:42.936914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840885+0000","last_change":"2023-10-16T11:20:38.832751+0000","last_active":"2023-10-16T11:20:38.840885+0000","last_peered":"2023-10-16T11:20:38.840885+0000","last_clean":"2023-10-16T11:20:38.840885+0000","last_became_active":"2023-10-16T11:20:38.832449+0000","last_became_peered":"2023-10-16T11:20:38.832449+0000","last_unstale":"2023-10-16T11:20:38.840885+0000","last_undegraded":"2023-10-16T11:20:38.840885+0000","last_fullsized":"2023-10-16T11:20:38.840885+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:09:28.758123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803637+0000","last_change":"2023-10-16T11:20:25.736801+0000","last_active":"2023-10-16T11:20:37.803637+0000","last_peered":"2023-10-16T11:20:37.803637+0000","last_clean":"2023-10-16T11:20:37.803637+0000","last_became_active":"2023-10-16T11:20:25.736501+0000","last_became_peered":"2023-10-16T11:20:25.736501+0000","last_unstale":"2023-10-16T11:20:37.803637+0000","last_undegraded":"2023-10-16T11:20:37.803637+0000","last_fullsized":"2023-10-16T11:20:37.803637+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:41.787813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"237'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814415+0000","last_change":"2023-10-16T11:20:24.734520+0000","last_active":"2023-10-16T11:20:37.814415+0000","last_peered":"2023-10-16T11:20:37.814415+0000","last_clean":"2023-10-16T11:20:37.814415+0000","last_became_active":"2023-10-16T11:20:24.733902+0000","last_became_peered":"2023-10-16T11:20:24.733902+0000","last_unstale":"2023-10-16T11:20:37.814415+0000","last_undegraded":"2023-10-16T11:20:37.814415+0000","last_fullsized":"2023-10-16T11:20:37.814415+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:43.725198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839552+0000","last_change":"2023-10-16T11:20:38.830629+0000","last_active":"2023-10-16T11:20:38.839552+0000","last_peered":"2023-10-16T11:20:38.839552+0000","last_clean":"2023-10-16T11:20:38.839552+0000","last_became_active":"2023-10-16T11:20:38.830370+0000","last_became_peered":"2023-10-16T11:20:38.830370+0000","last_unstale":"2023-10-16T11:20:38.839552+0000","last_undegraded":"2023-10-16T11:20:38.839552+0000","last_fullsized":"2023-10-16T11:20:38.839552+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:36:19.694559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806708+0000","last_change":"2023-10-16T11:20:25.739360+0000","last_active":"2023-10-16T11:20:37.806708+0000","last_peered":"2023-10-16T11:20:37.806708+0000","last_clean":"2023-10-16T11:20:37.806708+0000","last_became_active":"2023-10-16T11:20:25.739110+0000","last_became_peered":"2023-10-16T11:20:25.739110+0000","last_unstale":"2023-10-16T11:20:37.806708+0000","last_undegraded":"2023-10-16T11:20:37.806708+0000","last_fullsized":"2023-10-16T11:20:37.806708+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:40.578713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814270+0000","last_change":"2023-10-16T11:20:24.736139+0000","last_active":"2023-10-16T11:20:37.814270+0000","last_peered":"2023-10-16T11:20:37.814270+0000","last_clean":"2023-10-16T11:20:37.814270+0000","last_became_active":"2023-10-16T11:20:24.735807+0000","last_became_peered":"2023-10-16T11:20:24.735807+0000","last_unstale":"2023-10-16T11:20:37.814270+0000","last_undegraded":"2023-10-16T11:20:37.814270+0000","last_fullsized":"2023-10-16T11:20:37.814270+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.831725+0000","last_change":"2023-10-16T11:20:38.830621+0000","last_active":"2023-10-16T11:20:38.831725+0000","last_peered":"2023-10-16T11:20:38.831725+0000","last_clean":"2023-10-16T11:20:38.831725+0000","last_became_active":"2023-10-16T11:20:38.830345+0000","last_became_peered":"2023-10-16T11:20:38.830345+0000","last_unstale":"2023-10-16T11:20:38.831725+0000","last_undegraded":"2023-10-16T11:20:38.831725+0000","last_fullsized":"2023-10-16T11:20:38.831725+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:06.544476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813206+0000","last_change":"2023-10-16T11:20:25.748753+0000","last_active":"2023-10-16T11:20:37.813206+0000","last_peered":"2023-10-16T11:20:37.813206+0000","last_clean":"2023-10-16T11:20:37.813206+0000","last_became_active":"2023-10-16T11:20:25.748480+0000","last_became_peered":"2023-10-16T11:20:25.748480+0000","last_unstale":"2023-10-16T11:20:37.813206+0000","last_undegraded":"2023-10-16T11:20:37.813206+0000","last_fullsized":"2023-10-16T11:20:37.813206+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:51.963113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814359+0000","last_change":"2023-10-16T11:20:24.729644+0000","last_active":"2023-10-16T11:20:37.814359+0000","last_peered":"2023-10-16T11:20:37.814359+0000","last_clean":"2023-10-16T11:20:37.814359+0000","last_became_active":"2023-10-16T11:20:24.729487+0000","last_became_peered":"2023-10-16T11:20:24.729487+0000","last_unstale":"2023-10-16T11:20:37.814359+0000","last_undegraded":"2023-10-16T11:20:37.814359+0000","last_fullsized":"2023-10-16T11:20:37.814359+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:45.766549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842068+0000","last_change":"2023-10-16T11:20:38.834447+0000","last_active":"2023-10-16T11:20:38.842068+0000","last_peered":"2023-10-16T11:20:38.842068+0000","last_clean":"2023-10-16T11:20:38.842068+0000","last_became_active":"2023-10-16T11:20:38.833694+0000","last_became_peered":"2023-10-16T11:20:38.833694+0000","last_unstale":"2023-10-16T11:20:38.842068+0000","last_undegraded":"2023-10-16T11:20:38.842068+0000","last_fullsized":"2023-10-16T11:20:38.842068+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:58.540418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814091+0000","last_change":"2023-10-16T11:20:25.740829+0000","last_active":"2023-10-16T11:20:37.814091+0000","last_peered":"2023-10-16T11:20:37.814091+0000","last_clean":"2023-10-16T11:20:37.814091+0000","last_became_active":"2023-10-16T11:20:25.740552+0000","last_became_peered":"2023-10-16T11:20:25.740552+0000","last_unstale":"2023-10-16T11:20:37.814091+0000","last_undegraded":"2023-10-16T11:20:37.814091+0000","last_fullsized":"2023-10-16T11:20:37.814091+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810287+0000","last_change":"2023-10-16T11:20:24.727846+0000","last_active":"2023-10-16T11:20:37.810287+0000","last_peered":"2023-10-16T11:20:37.810287+0000","last_clean":"2023-10-16T11:20:37.810287+0000","last_became_active":"2023-10-16T11:20:24.727551+0000","last_became_peered":"2023-10-16T11:20:24.727551+0000","last_unstale":"2023-10-16T11:20:37.810287+0000","last_undegraded":"2023-10-16T11:20:37.810287+0000","last_fullsized":"2023-10-16T11:20:37.810287+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:10.257213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837069+0000","last_change":"2023-10-16T11:20:38.834306+0000","last_active":"2023-10-16T11:20:38.837069+0000","last_peered":"2023-10-16T11:20:38.837069+0000","last_clean":"2023-10-16T11:20:38.837069+0000","last_became_active":"2023-10-16T11:20:38.834024+0000","last_became_peered":"2023-10-16T11:20:38.834024+0000","last_unstale":"2023-10-16T11:20:38.837069+0000","last_undegraded":"2023-10-16T11:20:38.837069+0000","last_fullsized":"2023-10-16T11:20:38.837069+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:46.033448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.805657+0000","last_change":"2023-10-16T11:20:25.748408+0000","last_active":"2023-10-16T11:20:37.805657+0000","last_peered":"2023-10-16T11:20:37.805657+0000","last_clean":"2023-10-16T11:20:37.805657+0000","last_became_active":"2023-10-16T11:20:25.747574+0000","last_became_peered":"2023-10-16T11:20:25.747574+0000","last_unstale":"2023-10-16T11:20:37.805657+0000","last_undegraded":"2023-10-16T11:20:37.805657+0000","last_fullsized":"2023-10-16T11:20:37.805657+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:57.537856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813161+0000","last_change":"2023-10-16T11:20:24.728924+0000","last_active":"2023-10-16T11:20:37.813161+0000","last_peered":"2023-10-16T11:20:37.813161+0000","last_clean":"2023-10-16T11:20:37.813161+0000","last_became_active":"2023-10-16T11:20:24.728614+0000","last_became_peered":"2023-10-16T11:20:24.728614+0000","last_unstale":"2023-10-16T11:20:37.813161+0000","last_undegraded":"2023-10-16T11:20:37.813161+0000","last_fullsized":"2023-10-16T11:20:37.813161+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:23.737157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839653+0000","last_change":"2023-10-16T11:20:38.835181+0000","last_active":"2023-10-16T11:20:38.839653+0000","last_peered":"2023-10-16T11:20:38.839653+0000","last_clean":"2023-10-16T11:20:38.839653+0000","last_became_active":"2023-10-16T11:20:38.834922+0000","last_became_peered":"2023-10-16T11:20:38.834922+0000","last_unstale":"2023-10-16T11:20:38.839653+0000","last_undegraded":"2023-10-16T11:20:38.839653+0000","last_fullsized":"2023-10-16T11:20:38.839653+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:05.487994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.821141+0000","last_change":"2023-10-16T11:20:25.751879+0000","last_active":"2023-10-16T11:20:37.821141+0000","last_peered":"2023-10-16T11:20:37.821141+0000","last_clean":"2023-10-16T11:20:37.821141+0000","last_became_active":"2023-10-16T11:20:25.750440+0000","last_became_peered":"2023-10-16T11:20:25.750440+0000","last_unstale":"2023-10-16T11:20:37.821141+0000","last_undegraded":"2023-10-16T11:20:37.821141+0000","last_fullsized":"2023-10-16T11:20:37.821141+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:57.379246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806639+0000","last_change":"2023-10-16T11:20:24.732437+0000","last_active":"2023-10-16T11:20:37.806639+0000","last_peered":"2023-10-16T11:20:37.806639+0000","last_clean":"2023-10-16T11:20:37.806639+0000","last_became_active":"2023-10-16T11:20:24.732111+0000","last_became_peered":"2023-10-16T11:20:24.732111+0000","last_unstale":"2023-10-16T11:20:37.806639+0000","last_undegraded":"2023-10-16T11:20:37.806639+0000","last_fullsized":"2023-10-16T11:20:37.806639+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:26.846516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838494+0000","last_change":"2023-10-16T11:20:38.835643+0000","last_active":"2023-10-16T11:20:38.838494+0000","last_peered":"2023-10-16T11:20:38.838494+0000","last_clean":"2023-10-16T11:20:38.838494+0000","last_became_active":"2023-10-16T11:20:38.835326+0000","last_became_peered":"2023-10-16T11:20:38.835326+0000","last_unstale":"2023-10-16T11:20:38.838494+0000","last_undegraded":"2023-10-16T11:20:38.838494+0000","last_fullsized":"2023-10-16T11:20:38.838494+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:56.021692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811359+0000","last_change":"2023-10-16T11:20:25.750621+0000","last_active":"2023-10-16T11:20:37.811359+0000","last_peered":"2023-10-16T11:20:37.811359+0000","last_clean":"2023-10-16T11:20:37.811359+0000","last_became_active":"2023-10-16T11:20:25.750352+0000","last_became_peered":"2023-10-16T11:20:25.750352+0000","last_unstale":"2023-10-16T11:20:37.811359+0000","last_undegraded":"2023-10-16T11:20:37.811359+0000","last_fullsized":"2023-10-16T11:20:37.811359+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:57.295597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810371+0000","last_change":"2023-10-16T11:20:24.733888+0000","last_active":"2023-10-16T11:20:37.810371+0000","last_peered":"2023-10-16T11:20:37.810371+0000","last_clean":"2023-10-16T11:20:37.810371+0000","last_became_active":"2023-10-16T11:20:24.733588+0000","last_became_peered":"2023-10-16T11:20:24.733588+0000","last_unstale":"2023-10-16T11:20:37.810371+0000","last_undegraded":"2023-10-16T11:20:37.810371+0000","last_fullsized":"2023-10-16T11:20:37.810371+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:06.768468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844237+0000","last_change":"2023-10-16T11:20:38.838396+0000","last_active":"2023-10-16T11:20:38.844237+0000","last_peered":"2023-10-16T11:20:38.844237+0000","last_clean":"2023-10-16T11:20:38.844237+0000","last_became_active":"2023-10-16T11:20:38.837632+0000","last_became_peered":"2023-10-16T11:20:38.837632+0000","last_unstale":"2023-10-16T11:20:38.844237+0000","last_undegraded":"2023-10-16T11:20:38.844237+0000","last_fullsized":"2023-10-16T11:20:38.844237+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:34.965136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.805321+0000","last_change":"2023-10-16T11:20:25.748051+0000","last_active":"2023-10-16T11:20:37.805321+0000","last_peered":"2023-10-16T11:20:37.805321+0000","last_clean":"2023-10-16T11:20:37.805321+0000","last_became_active":"2023-10-16T11:20:25.747647+0000","last_became_peered":"2023-10-16T11:20:25.747647+0000","last_unstale":"2023-10-16T11:20:37.805321+0000","last_undegraded":"2023-10-16T11:20:37.805321+0000","last_fullsized":"2023-10-16T11:20:37.805321+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:06.072095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816886+0000","last_change":"2023-10-16T11:20:24.736168+0000","last_active":"2023-10-16T11:20:37.816886+0000","last_peered":"2023-10-16T11:20:37.816886+0000","last_clean":"2023-10-16T11:20:37.816886+0000","last_became_active":"2023-10-16T11:20:24.735886+0000","last_became_peered":"2023-10-16T11:20:24.735886+0000","last_unstale":"2023-10-16T11:20:37.816886+0000","last_undegraded":"2023-10-16T11:20:37.816886+0000","last_fullsized":"2023-10-16T11:20:37.816886+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"223'370","reported_seq":1309,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803645+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:20:37.803645+0000","last_peered":"2023-10-16T11:20:37.803645+0000","last_clean":"2023-10-16T11:20:37.803645+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:20:37.803645+0000","last_undegraded":"2023-10-16T11:20:37.803645+0000","last_fullsized":"2023-10-16T11:20:37.803645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52792,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319048,"statfs":{"total":96636764160,"available":96582705152,"internally_reserved":0,"allocated":3641344,"data_stored":2224748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64600000000000002,"15min":0.60699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68999999999999995,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.66400000000000003,"15min":0.628},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.68799999999999994,"15min":0.66200000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.622,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.70099999999999996,"15min":0.67400000000000004},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.71199999999999997,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.436}]},{"osd":2,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.71399999999999997,"15min":0.67300000000000004},"min":{"1min":0.48099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78900000000000003,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.73299999999999998,"15min":0.69099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":9.516,"15min":9.516},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.68300000000000005,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.68799999999999994,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.75900000000000001,"15min":0.71199999999999997},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.54100000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32300000000000001},"max":{"1min":0.747,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.63200000000000001,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.33500000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65300000000000002,"15min":0.57799999999999996},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":0.98399999999999999,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69699999999999995,"15min":0.63},"min":{"1min":0.41499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79500000000000004,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.71599999999999997,"15min":0.64600000000000002},"min":{"1min":0.41599999999999998,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.84199999999999997,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.69699999999999995,"15min":0.64200000000000002},"min":{"1min":0.442,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76600000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.54500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56368,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315472,"statfs":{"total":96636764160,"available":96579043328,"internally_reserved":0,"allocated":3043328,"data_stored":1633798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75800000000000001,"5min":0.80700000000000005,"15min":0.86299999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.59199999999999997,"15min":0.61399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69399999999999995,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58099999999999996,"15min":0.60599999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73499999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":2.387,"15min":2.387},"last":0.80000000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.57199999999999995,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.314},"max":{"1min":0.58699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":0.62},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.871}]},{"osd":3,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.622,"15min":0.621},"min":{"1min":0.36499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.65300000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.36399999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71599999999999997,"5min":1.02,"15min":1.02},"last":0.81000000000000005}]},{"osd":4,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.32900000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.55500000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.77800000000000002,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.45400000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76200000000000001,"5min":3.335,"15min":3.335},"last":0.81999999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":2.524,"15min":2.524},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.752,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.85599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576003,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48020,"kb_used_data":2948,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323820,"statfs":{"total":96636764160,"available":96587591680,"internally_reserved":0,"allocated":3018752,"data_stored":1625207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.622,"15min":0.66700000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.436,"15min":0.45200000000000001},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.52500000000000002,"5min":0.63900000000000001,"15min":0.71699999999999997},"last":0.38200000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.64000000000000001,"15min":0.70099999999999996},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.442},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":0.623,"15min":0.70499999999999996},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.48099999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.64700000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.69199999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65400000000000003,"15min":0.67200000000000004},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":0.89500000000000002,"15min":1.0149999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90000000000000002,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.60499999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.90300000000000002,"15min":1.0620000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.016},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61108,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310732,"statfs":{"total":96636764160,"available":96574189568,"internally_reserved":0,"allocated":3641344,"data_stored":2226949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.45000000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.26700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52800000000000002,"5min":0.66100000000000003,"15min":0.77200000000000002},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":1.272,"15min":1.272},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40899999999999997,"15min":0.433},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.313},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54200000000000004,"5min":1.351,"15min":1.351},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83099999999999996,"5min":1.391,"15min":1.391},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76000000000000001,"5min":0.874,"15min":1.089},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.624,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.95599999999999996,"15min":1.216},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.47199999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.1180000000000001},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.67200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80300000000000005,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80500000000000005,"5min":0.89600000000000002,"15min":1.0780000000000001},"last":0.70599999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48064,"kb_used_data":2988,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323776,"statfs":{"total":96636764160,"available":96587546624,"internally_reserved":0,"allocated":3059712,"data_stored":1636520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.371,"15min":0.39500000000000002},"min":{"1min":0.253,"5min":0.188,"15min":0.188},"max":{"1min":0.51800000000000002,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.42799999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.193,"15min":0.193},"max":{"1min":0.53600000000000003,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.313}]},{"osd":1,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.44400000000000001,"15min":0.46500000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59499999999999997,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39000000000000001,"15min":0.41599999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.45500000000000002,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.29199999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.442},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56699999999999995,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48799999999999999,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.95399999999999996,"15min":1.097},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64700000000000002,"15min":0.67300000000000004},"min":{"1min":0.438,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77100000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63700000000000001,"15min":0.66700000000000004},"min":{"1min":0.45500000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82499999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.69199999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81899999999999995,"5min":1.0169999999999999,"15min":1.0389999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64500000000000002,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76700000000000002,"5min":0.97799999999999998,"15min":1.022},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67700000000000005,"15min":0.70299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84999999999999998,"5min":0.98999999999999999,"15min":1.004},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.69199999999999995,"15min":0.70599999999999996},"min":{"1min":0.48599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81000000000000005,"5min":0.99299999999999999,"15min":1.071},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.67100000000000004,"15min":0.69799999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81200000000000006,"5min":0.97799999999999998,"15min":1.077},"last":0.505}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52212,"kb_used_data":2988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319628,"statfs":{"total":96636764160,"available":96583299072,"internally_reserved":0,"allocated":3059712,"data_stored":1637239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.376,"15min":0.39500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38300000000000001,"15min":0.40699999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":1.0029999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42499999999999999,"15min":0.441},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.443}]},{"osd":2,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46400000000000002,"15min":0.48299999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.50800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.024},"last":0.496}]},{"osd":4,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.68000000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.026},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.77600000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.41799999999999998},"max":{"1min":5.8399999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69899999999999995,"15min":0.70499999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.4409999999999998,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.66600000000000004,"15min":0.69999999999999996},"min":{"1min":0.45300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77800000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.71299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52212,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319628,"statfs":{"total":96636764160,"available":96583299072,"internally_reserved":0,"allocated":3047424,"data_stored":1635334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56999999999999995,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67700000000000005,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.503},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57599999999999996,"15min":0.61799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.61499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74099999999999999,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.61399999999999999},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.626}]},{"osd":2,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.622},"min":{"1min":0.40400000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60799999999999998,"15min":0.63700000000000001},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.61599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67900000000000005,"5min":0.83799999999999997,"15min":0.995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48199999999999998,"15min":0.5},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71499999999999997,"5min":0.73599999999999999,"15min":0.86199999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76300000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.38300000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.76200000000000001,"5min":0.878,"15min":1.0609999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.745,"5min":0.83099999999999996,"15min":1.115},"last":0.44}]},{"osd":7,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.83999999999999997,"5min":0.84499999999999997,"15min":0.93200000000000005},"last":0.58099999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66088,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305752,"statfs":{"total":96636764160,"available":96569090048,"internally_reserved":0,"allocated":3035136,"data_stored":1633722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61199999999999999,"15min":0.63200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.28999999999999998},"max":{"1min":0.78000000000000003,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.62,"15min":0.65000000000000002},"min":{"1min":0.39700000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72999999999999998,"5min":0.871,"15min":0.88200000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.57699999999999996,"15min":0.59499999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.752,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.56399999999999995,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.59099999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73399999999999999,"5min":0.80300000000000005,"15min":0.86499999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58399999999999996,"15min":0.61699999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.752,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.61099999999999999},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57999999999999996,"15min":0.60399999999999998},"min":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.70299999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80900000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.82499999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84499999999999997,"5min":0.878,"15min":0.878},"last":0.496}]},{"osd":7,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.57299999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78300000000000003,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:20:42.967 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:20:43.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.115+0000 7f0a719e7700 1 Processor -- start 2023-10-16T11:20:43.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.115+0000 7f0a719e7700 1 -- start start 2023-10-16T11:20:43.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.115+0000 7f0a719e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 0x7f0a6c0ace30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.115+0000 7f0a719e7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c0cf500 con 0x7f0a6c0caf70 2023-10-16T11:20:43.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.115+0000 7f0a719e7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c06a970 con 0x7f0a6c0ce070 2023-10-16T11:20:43.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.115+0000 7f0a719e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c06ac50 con 0x7f0a6c06f660 2023-10-16T11:20:43.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a6b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0a6c0caf70 0x7f0a6c06b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40862/0 (socket says 172.21.15.158:40862) 2023-10-16T11:20:43.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a6b7fe700 1 -- 172.21.15.158:0/419130581 learned_addr learned my addr 172.21.15.158:0/419130581 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:43.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a6affd700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 0x7f0a6c0ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763109331 0 0) 0x7f0a6c0cf500 con 0x7f0a6c0caf70 2023-10-16T11:20:43.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a540035b0 con 0x7f0a6c0caf70 2023-10-16T11:20:43.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633388581 0 0) 0x7f0a6c06a970 con 0x7f0a6c0ce070 2023-10-16T11:20:43.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a6c0cf500 con 0x7f0a6c0ce070 2023-10-16T11:20:43.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3055035314 0 0) 0x7f0a540035b0 con 0x7f0a6c0caf70 2023-10-16T11:20:43.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 msgr2=0x7f0a6c0ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:43.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 0x7f0a6c0ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 >> v1:172.21.15.172:6789/0 conn(0x7f0a6c0ce070 legacy=0x7f0a6c0ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c06af30 con 0x7f0a6c0caf70 2023-10-16T11:20:43.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0a64001a10 con 0x7f0a6c0caf70 2023-10-16T11:20:43.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 >> v1:172.21.15.158:6789/0 conn(0x7f0a6c0caf70 legacy=0x7f0a6c06b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 0x7f0a54003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a54004170 0x7f0a54004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004d90 0x7f0a54005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c06ac50 con 0x7f0a54004170 2023-10-16T11:20:43.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5400bb50 con 0x7f0a54004d90 2023-10-16T11:20:43.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5400bd10 con 0x7f0a6c06f660 2023-10-16T11:20:43.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a6a7fc700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a54004170 0x7f0a54004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a6affd700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 0x7f0a54003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.116+0000 7f0a6b7fe700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004d90 0x7f0a54005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a6a7fc700 1 -- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 msgr2=0x7f0a54003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:43.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a6a7fc700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c06f660 0x7f0a54003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a6a7fc700 1 -- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004d90 msgr2=0x7f0a54005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:43.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a6a7fc700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004d90 0x7f0a54005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a6a7fc700 1 -- 172.21.15.158:0/419130581 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540035b0 con 0x7f0a54004170 2023-10-16T11:20:43.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a6a7fc700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a54004170 0x7f0a54004850 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c001940 tx=0x7f0a54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159f476cc197c457 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:43.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c001430 con 0x7f0a54004170 2023-10-16T11:20:43.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c0050d0 con 0x7f0a54004170 2023-10-16T11:20:43.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a69ffb700 1 -- 172.21.15.158:0/419130581 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c005480 con 0x7f0a54004170 2023-10-16T11:20:43.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a719e7700 1 -- 172.21.15.158:0/419130581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a54004170 msgr2=0x7f0a54004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a719e7700 1 --2- 172.21.15.158:0/419130581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a54004170 0x7f0a54004850 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c001940 tx=0x7f0a54005880 comp rx=0 tx=0).stop 2023-10-16T11:20:43.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a6b7fe700 1 -- 172.21.15.158:0/419130581 reap_dead start 2023-10-16T11:20:43.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.117+0000 7f0a719e7700 1 -- 172.21.15.158:0/419130581 shutdown_connections 2023-10-16T11:20:43.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.118+0000 7f0a719e7700 1 -- 172.21.15.158:0/419130581 >> 172.21.15.158:0/419130581 conn(0x7f0a6c0b1510 msgr2=0x7f0a6c0b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:43.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.118+0000 7f0a719e7700 1 -- 172.21.15.158:0/419130581 shutdown_connections 2023-10-16T11:20:43.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.118+0000 7f0a719e7700 1 -- 172.21.15.158:0/419130581 wait complete. 2023-10-16T11:20:43.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.119+0000 7f0a719e7700 1 Processor -- start 2023-10-16T11:20:43.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.119+0000 7f0a719e7700 1 -- start start 2023-10-16T11:20:43.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004170 0x7f0a6c152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c15dcf0 0x7f0a6c1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6c1f6b10 0x7f0a6c1f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0b4ee0 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0c4240 con 0x7f0a54004170 2023-10-16T11:20:43.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0c4520 con 0x7f0a6c15dcf0 2023-10-16T11:20:43.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6c1f6b10 0x7f0a6c1f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6c1f6b10 0x7f0a6c1f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40698/0 (socket says 172.21.15.158:40698) 2023-10-16T11:20:43.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 -- 172.21.15.158:0/257315707 learned_addr learned my addr 172.21.15.158:0/257315707 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:43.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6a7fc700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c15dcf0 0x7f0a6c1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6affd700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004170 0x7f0a6c152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 -- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c15dcf0 msgr2=0x7f0a6c1f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c15dcf0 0x7f0a6c1f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 -- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004170 msgr2=0x7f0a6c152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004170 0x7f0a6c152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 -- 172.21.15.158:0/257315707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0c4800 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a6b7fe700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6c1f6b10 0x7f0a6c1f6f90 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0af060 tx=0x7f0a6400a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:43.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a53fff700 1 -- 172.21.15.158:0/257315707 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6400ac20 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a53fff700 1 -- 172.21.15.158:0/257315707 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6400ada0 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c0c3f30 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a53fff700 1 -- 172.21.15.158:0/257315707 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64013630 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.120+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c0c4d20 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.121+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c0c49f0 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.122+0000 7f0a53fff700 1 -- 172.21.15.158:0/257315707 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0a64013790 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.122+0000 7f0a53fff700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0a5806b140 0x7f0a5806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.122+0000 7f0a53fff700 1 -- 172.21.15.158:0/257315707 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f0a6408a760 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.122+0000 7f0a6affd700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0a5806b140 0x7f0a5806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.123+0000 7f0a6affd700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0a5806b140 0x7f0a5806d600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009b80 tx=0x7f0a60009210 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:43.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.126+0000 7f0a53fff700 1 -- 172.21.15.158:0/257315707 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a64054f00 con 0x7f0a6c1f6b10 2023-10-16T11:20:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.313+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 --> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a6c1535a0 con 0x7f0a5806b140 2023-10-16T11:20:43.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.327+0000 7f0a53fff700 1 -- 172.21.15.158:0/257315707 <== mgr.12030 v2:172.21.15.158:6832/3408161846 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528937 (secure 0 0 0) 0x7f0a6c1535a0 con 0x7f0a5806b140 2023-10-16T11:20:43.335 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:43.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0a5806b140 msgr2=0x7f0a5806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0a5806b140 0x7f0a5806d600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009b80 tx=0x7f0a60009210 comp rx=0 tx=0).stop 2023-10-16T11:20:43.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6c1f6b10 msgr2=0x7f0a6c1f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6c1f6b10 0x7f0a6c1f6f90 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0af060 tx=0x7f0a6400a510 comp rx=0 tx=0).stop 2023-10-16T11:20:43.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 shutdown_connections 2023-10-16T11:20:43.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f0a5806b140 0x7f0a5806d600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a6c1f6b10 0x7f0a6c1f6f90 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a54004170 0x7f0a6c152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 --2- 172.21.15.158:0/257315707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a6c15dcf0 0x7f0a6c1f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 >> 172.21.15.158:0/257315707 conn(0x7f0a6c0b1510 msgr2=0x7f0a6c0cea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:43.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 shutdown_connections 2023-10-16T11:20:43.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.331+0000 7f0a719e7700 1 -- 172.21.15.158:0/257315707 wait complete. 2023-10-16T11:20:43.342 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:20:43.345 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-10-16T11:20:41.451108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":240,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436864,"kb_used_data":24948,"kb_used_omap":12,"kb_used_meta":411763,"kb_avail":754537856,"statfs":{"total":773094113280,"available":772646764544,"internally_reserved":0,"allocated":25546752,"data_stored":14253517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":421645879},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.649993"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840571+0000","last_change":"2023-10-16T11:20:38.839544+0000","last_active":"2023-10-16T11:20:38.840571+0000","last_peered":"2023-10-16T11:20:38.840571+0000","last_clean":"2023-10-16T11:20:38.840571+0000","last_became_active":"2023-10-16T11:20:38.839288+0000","last_became_peered":"2023-10-16T11:20:38.839288+0000","last_unstale":"2023-10-16T11:20:38.840571+0000","last_undegraded":"2023-10-16T11:20:38.840571+0000","last_fullsized":"2023-10-16T11:20:38.840571+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:36.154489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834896+0000","last_change":"2023-10-16T11:20:37.821795+0000","last_active":"2023-10-16T11:20:38.834896+0000","last_peered":"2023-10-16T11:20:38.834896+0000","last_clean":"2023-10-16T11:20:38.834896+0000","last_became_active":"2023-10-16T11:20:37.821171+0000","last_became_peered":"2023-10-16T11:20:37.821171+0000","last_unstale":"2023-10-16T11:20:38.834896+0000","last_undegraded":"2023-10-16T11:20:38.834896+0000","last_fullsized":"2023-10-16T11:20:38.834896+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:52.616578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804659+0000","last_change":"2023-10-16T11:20:25.748352+0000","last_active":"2023-10-16T11:20:37.804659+0000","last_peered":"2023-10-16T11:20:37.804659+0000","last_clean":"2023-10-16T11:20:37.804659+0000","last_became_active":"2023-10-16T11:20:25.747848+0000","last_became_peered":"2023-10-16T11:20:25.747848+0000","last_unstale":"2023-10-16T11:20:37.804659+0000","last_undegraded":"2023-10-16T11:20:37.804659+0000","last_fullsized":"2023-10-16T11:20:37.804659+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:48.171042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.841363+0000","last_change":"2023-10-16T11:20:38.840167+0000","last_active":"2023-10-16T11:20:38.841363+0000","last_peered":"2023-10-16T11:20:38.841363+0000","last_clean":"2023-10-16T11:20:38.841363+0000","last_became_active":"2023-10-16T11:20:38.839889+0000","last_became_peered":"2023-10-16T11:20:38.839889+0000","last_unstale":"2023-10-16T11:20:38.841363+0000","last_undegraded":"2023-10-16T11:20:38.841363+0000","last_fullsized":"2023-10-16T11:20:38.841363+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:32.738869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"240'4","reported_seq":15,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842724+0000","last_change":"2023-10-16T11:20:37.825024+0000","last_active":"2023-10-16T11:20:38.842724+0000","last_peered":"2023-10-16T11:20:38.842724+0000","last_clean":"2023-10-16T11:20:38.842724+0000","last_became_active":"2023-10-16T11:20:37.824652+0000","last_became_peered":"2023-10-16T11:20:37.824652+0000","last_unstale":"2023-10-16T11:20:38.842724+0000","last_undegraded":"2023-10-16T11:20:38.842724+0000","last_fullsized":"2023-10-16T11:20:38.842724+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:31.726955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.821043+0000","last_change":"2023-10-16T11:20:25.749395+0000","last_active":"2023-10-16T11:20:37.821043+0000","last_peered":"2023-10-16T11:20:37.821043+0000","last_clean":"2023-10-16T11:20:37.821043+0000","last_became_active":"2023-10-16T11:20:25.749044+0000","last_became_peered":"2023-10-16T11:20:25.749044+0000","last_unstale":"2023-10-16T11:20:37.821043+0000","last_undegraded":"2023-10-16T11:20:37.821043+0000","last_fullsized":"2023-10-16T11:20:37.821043+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:16.211507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.841804+0000","last_change":"2023-10-16T11:20:38.833450+0000","last_active":"2023-10-16T11:20:38.841804+0000","last_peered":"2023-10-16T11:20:38.841804+0000","last_clean":"2023-10-16T11:20:38.841804+0000","last_became_active":"2023-10-16T11:20:38.833202+0000","last_became_peered":"2023-10-16T11:20:38.833202+0000","last_unstale":"2023-10-16T11:20:38.841804+0000","last_undegraded":"2023-10-16T11:20:38.841804+0000","last_fullsized":"2023-10-16T11:20:38.841804+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:10.744127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837684+0000","last_change":"2023-10-16T11:20:37.821444+0000","last_active":"2023-10-16T11:20:38.837684+0000","last_peered":"2023-10-16T11:20:38.837684+0000","last_clean":"2023-10-16T11:20:38.837684+0000","last_became_active":"2023-10-16T11:20:37.821183+0000","last_became_peered":"2023-10-16T11:20:37.821183+0000","last_unstale":"2023-10-16T11:20:38.837684+0000","last_undegraded":"2023-10-16T11:20:38.837684+0000","last_fullsized":"2023-10-16T11:20:38.837684+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:24.978539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.809011+0000","last_change":"2023-10-16T11:20:25.739817+0000","last_active":"2023-10-16T11:20:37.809011+0000","last_peered":"2023-10-16T11:20:37.809011+0000","last_clean":"2023-10-16T11:20:37.809011+0000","last_became_active":"2023-10-16T11:20:25.739601+0000","last_became_peered":"2023-10-16T11:20:25.739601+0000","last_unstale":"2023-10-16T11:20:37.809011+0000","last_undegraded":"2023-10-16T11:20:37.809011+0000","last_fullsized":"2023-10-16T11:20:37.809011+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:38.443038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838632+0000","last_change":"2023-10-16T11:20:38.822788+0000","last_active":"2023-10-16T11:20:38.838632+0000","last_peered":"2023-10-16T11:20:38.838632+0000","last_clean":"2023-10-16T11:20:38.838632+0000","last_became_active":"2023-10-16T11:20:38.822510+0000","last_became_peered":"2023-10-16T11:20:38.822510+0000","last_unstale":"2023-10-16T11:20:38.838632+0000","last_undegraded":"2023-10-16T11:20:38.838632+0000","last_fullsized":"2023-10-16T11:20:38.838632+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:13.281230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844121+0000","last_change":"2023-10-16T11:20:37.816446+0000","last_active":"2023-10-16T11:20:38.844121+0000","last_peered":"2023-10-16T11:20:38.844121+0000","last_clean":"2023-10-16T11:20:38.844121+0000","last_became_active":"2023-10-16T11:20:37.815968+0000","last_became_peered":"2023-10-16T11:20:37.815968+0000","last_unstale":"2023-10-16T11:20:38.844121+0000","last_undegraded":"2023-10-16T11:20:38.844121+0000","last_fullsized":"2023-10-16T11:20:38.844121+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:36.822624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806723+0000","last_change":"2023-10-16T11:20:25.751269+0000","last_active":"2023-10-16T11:20:37.806723+0000","last_peered":"2023-10-16T11:20:37.806723+0000","last_clean":"2023-10-16T11:20:37.806723+0000","last_became_active":"2023-10-16T11:20:25.750973+0000","last_became_peered":"2023-10-16T11:20:25.750973+0000","last_unstale":"2023-10-16T11:20:37.806723+0000","last_undegraded":"2023-10-16T11:20:37.806723+0000","last_fullsized":"2023-10-16T11:20:37.806723+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:36.490706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843350+0000","last_change":"2023-10-16T11:20:38.840537+0000","last_active":"2023-10-16T11:20:38.843350+0000","last_peered":"2023-10-16T11:20:38.843350+0000","last_clean":"2023-10-16T11:20:38.843350+0000","last_became_active":"2023-10-16T11:20:38.838185+0000","last_became_peered":"2023-10-16T11:20:38.838185+0000","last_unstale":"2023-10-16T11:20:38.843350+0000","last_undegraded":"2023-10-16T11:20:38.843350+0000","last_fullsized":"2023-10-16T11:20:38.843350+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:39.948003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844724+0000","last_change":"2023-10-16T11:20:37.820445+0000","last_active":"2023-10-16T11:20:38.844724+0000","last_peered":"2023-10-16T11:20:38.844724+0000","last_clean":"2023-10-16T11:20:38.844724+0000","last_became_active":"2023-10-16T11:20:37.820143+0000","last_became_peered":"2023-10-16T11:20:37.820143+0000","last_unstale":"2023-10-16T11:20:38.844724+0000","last_undegraded":"2023-10-16T11:20:38.844724+0000","last_fullsized":"2023-10-16T11:20:38.844724+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:21.515898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813297+0000","last_change":"2023-10-16T11:20:25.750599+0000","last_active":"2023-10-16T11:20:37.813297+0000","last_peered":"2023-10-16T11:20:37.813297+0000","last_clean":"2023-10-16T11:20:37.813297+0000","last_became_active":"2023-10-16T11:20:25.749962+0000","last_became_peered":"2023-10-16T11:20:25.749962+0000","last_unstale":"2023-10-16T11:20:37.813297+0000","last_undegraded":"2023-10-16T11:20:37.813297+0000","last_fullsized":"2023-10-16T11:20:37.813297+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:15.073727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840065+0000","last_change":"2023-10-16T11:20:38.833354+0000","last_active":"2023-10-16T11:20:38.840065+0000","last_peered":"2023-10-16T11:20:38.840065+0000","last_clean":"2023-10-16T11:20:38.840065+0000","last_became_active":"2023-10-16T11:20:38.832945+0000","last_became_peered":"2023-10-16T11:20:38.832945+0000","last_unstale":"2023-10-16T11:20:38.840065+0000","last_undegraded":"2023-10-16T11:20:38.840065+0000","last_fullsized":"2023-10-16T11:20:38.840065+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:25.776247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840318+0000","last_change":"2023-10-16T11:20:37.818148+0000","last_active":"2023-10-16T11:20:38.840318+0000","last_peered":"2023-10-16T11:20:38.840318+0000","last_clean":"2023-10-16T11:20:38.840318+0000","last_became_active":"2023-10-16T11:20:37.817676+0000","last_became_peered":"2023-10-16T11:20:37.817676+0000","last_unstale":"2023-10-16T11:20:38.840318+0000","last_undegraded":"2023-10-16T11:20:38.840318+0000","last_fullsized":"2023-10-16T11:20:38.840318+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.825698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804881+0000","last_change":"2023-10-16T11:20:25.745973+0000","last_active":"2023-10-16T11:20:37.804881+0000","last_peered":"2023-10-16T11:20:37.804881+0000","last_clean":"2023-10-16T11:20:37.804881+0000","last_became_active":"2023-10-16T11:20:25.745706+0000","last_became_peered":"2023-10-16T11:20:25.745706+0000","last_unstale":"2023-10-16T11:20:37.804881+0000","last_undegraded":"2023-10-16T11:20:37.804881+0000","last_fullsized":"2023-10-16T11:20:37.804881+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:08.628698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844221+0000","last_change":"2023-10-16T11:20:38.838232+0000","last_active":"2023-10-16T11:20:38.844221+0000","last_peered":"2023-10-16T11:20:38.844221+0000","last_clean":"2023-10-16T11:20:38.844221+0000","last_became_active":"2023-10-16T11:20:38.837628+0000","last_became_peered":"2023-10-16T11:20:38.837628+0000","last_unstale":"2023-10-16T11:20:38.844221+0000","last_undegraded":"2023-10-16T11:20:38.844221+0000","last_fullsized":"2023-10-16T11:20:38.844221+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.963194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.823216+0000","last_change":"2023-10-16T11:20:37.814903+0000","last_active":"2023-10-16T11:20:38.823216+0000","last_peered":"2023-10-16T11:20:38.823216+0000","last_clean":"2023-10-16T11:20:38.823216+0000","last_became_active":"2023-10-16T11:20:37.814642+0000","last_became_peered":"2023-10-16T11:20:37.814642+0000","last_unstale":"2023-10-16T11:20:38.823216+0000","last_undegraded":"2023-10-16T11:20:38.823216+0000","last_fullsized":"2023-10-16T11:20:38.823216+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:55.013398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816744+0000","last_change":"2023-10-16T11:20:25.748211+0000","last_active":"2023-10-16T11:20:37.816744+0000","last_peered":"2023-10-16T11:20:37.816744+0000","last_clean":"2023-10-16T11:20:37.816744+0000","last_became_active":"2023-10-16T11:20:25.747956+0000","last_became_peered":"2023-10-16T11:20:25.747956+0000","last_unstale":"2023-10-16T11:20:37.816744+0000","last_undegraded":"2023-10-16T11:20:37.816744+0000","last_fullsized":"2023-10-16T11:20:37.816744+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838813+0000","last_change":"2023-10-16T11:20:38.838166+0000","last_active":"2023-10-16T11:20:38.838813+0000","last_peered":"2023-10-16T11:20:38.838813+0000","last_clean":"2023-10-16T11:20:38.838813+0000","last_became_active":"2023-10-16T11:20:38.837889+0000","last_became_peered":"2023-10-16T11:20:38.837889+0000","last_unstale":"2023-10-16T11:20:38.838813+0000","last_undegraded":"2023-10-16T11:20:38.838813+0000","last_fullsized":"2023-10-16T11:20:38.838813+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:52.880299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.824373+0000","last_change":"2023-10-16T11:20:37.819002+0000","last_active":"2023-10-16T11:20:38.824373+0000","last_peered":"2023-10-16T11:20:38.824373+0000","last_clean":"2023-10-16T11:20:38.824373+0000","last_became_active":"2023-10-16T11:20:37.818578+0000","last_became_peered":"2023-10-16T11:20:37.818578+0000","last_unstale":"2023-10-16T11:20:38.824373+0000","last_undegraded":"2023-10-16T11:20:38.824373+0000","last_fullsized":"2023-10-16T11:20:38.824373+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:55.970715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812779+0000","last_change":"2023-10-16T11:20:25.750901+0000","last_active":"2023-10-16T11:20:37.812779+0000","last_peered":"2023-10-16T11:20:37.812779+0000","last_clean":"2023-10-16T11:20:37.812779+0000","last_became_active":"2023-10-16T11:20:25.750582+0000","last_became_peered":"2023-10-16T11:20:25.750582+0000","last_unstale":"2023-10-16T11:20:37.812779+0000","last_undegraded":"2023-10-16T11:20:37.812779+0000","last_fullsized":"2023-10-16T11:20:37.812779+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:47.707747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840552+0000","last_change":"2023-10-16T11:20:38.835941+0000","last_active":"2023-10-16T11:20:38.840552+0000","last_peered":"2023-10-16T11:20:38.840552+0000","last_clean":"2023-10-16T11:20:38.840552+0000","last_became_active":"2023-10-16T11:20:38.835620+0000","last_became_peered":"2023-10-16T11:20:38.835620+0000","last_unstale":"2023-10-16T11:20:38.840552+0000","last_undegraded":"2023-10-16T11:20:38.840552+0000","last_fullsized":"2023-10-16T11:20:38.840552+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:23:58.942409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.824574+0000","last_change":"2023-10-16T11:20:37.818876+0000","last_active":"2023-10-16T11:20:38.824574+0000","last_peered":"2023-10-16T11:20:38.824574+0000","last_clean":"2023-10-16T11:20:38.824574+0000","last_became_active":"2023-10-16T11:20:37.818610+0000","last_became_peered":"2023-10-16T11:20:37.818610+0000","last_unstale":"2023-10-16T11:20:38.824574+0000","last_undegraded":"2023-10-16T11:20:38.824574+0000","last_fullsized":"2023-10-16T11:20:38.824574+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:53.263379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804712+0000","last_change":"2023-10-16T11:20:25.750204+0000","last_active":"2023-10-16T11:20:37.804712+0000","last_peered":"2023-10-16T11:20:37.804712+0000","last_clean":"2023-10-16T11:20:37.804712+0000","last_became_active":"2023-10-16T11:20:25.749903+0000","last_became_peered":"2023-10-16T11:20:25.749903+0000","last_unstale":"2023-10-16T11:20:37.804712+0000","last_undegraded":"2023-10-16T11:20:37.804712+0000","last_fullsized":"2023-10-16T11:20:37.804712+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:48.734101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.830225+0000","last_change":"2023-10-16T11:20:38.826174+0000","last_active":"2023-10-16T11:20:38.830225+0000","last_peered":"2023-10-16T11:20:38.830225+0000","last_clean":"2023-10-16T11:20:38.830225+0000","last_became_active":"2023-10-16T11:20:38.825902+0000","last_became_peered":"2023-10-16T11:20:38.825902+0000","last_unstale":"2023-10-16T11:20:38.830225+0000","last_undegraded":"2023-10-16T11:20:38.830225+0000","last_fullsized":"2023-10-16T11:20:38.830225+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:59.413642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.816526+0000","last_change":"2023-10-16T11:20:37.819397+0000","last_active":"2023-10-16T11:20:38.816526+0000","last_peered":"2023-10-16T11:20:38.816526+0000","last_clean":"2023-10-16T11:20:38.816526+0000","last_became_active":"2023-10-16T11:20:37.819133+0000","last_became_peered":"2023-10-16T11:20:37.819133+0000","last_unstale":"2023-10-16T11:20:38.816526+0000","last_undegraded":"2023-10-16T11:20:38.816526+0000","last_fullsized":"2023-10-16T11:20:38.816526+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:21.860649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806760+0000","last_change":"2023-10-16T11:20:25.747726+0000","last_active":"2023-10-16T11:20:37.806760+0000","last_peered":"2023-10-16T11:20:37.806760+0000","last_clean":"2023-10-16T11:20:37.806760+0000","last_became_active":"2023-10-16T11:20:25.747314+0000","last_became_peered":"2023-10-16T11:20:25.747314+0000","last_unstale":"2023-10-16T11:20:37.806760+0000","last_undegraded":"2023-10-16T11:20:37.806760+0000","last_fullsized":"2023-10-16T11:20:37.806760+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837048+0000","last_change":"2023-10-16T11:20:38.834016+0000","last_active":"2023-10-16T11:20:38.837048+0000","last_peered":"2023-10-16T11:20:38.837048+0000","last_clean":"2023-10-16T11:20:38.837048+0000","last_became_active":"2023-10-16T11:20:38.832409+0000","last_became_peered":"2023-10-16T11:20:38.832409+0000","last_unstale":"2023-10-16T11:20:38.837048+0000","last_undegraded":"2023-10-16T11:20:38.837048+0000","last_fullsized":"2023-10-16T11:20:38.837048+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:47.307575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"240'6","reported_seq":17,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.846260+0000","last_change":"2023-10-16T11:20:37.822490+0000","last_active":"2023-10-16T11:20:38.846260+0000","last_peered":"2023-10-16T11:20:38.846260+0000","last_clean":"2023-10-16T11:20:38.846260+0000","last_became_active":"2023-10-16T11:20:37.822219+0000","last_became_peered":"2023-10-16T11:20:37.822219+0000","last_unstale":"2023-10-16T11:20:38.846260+0000","last_undegraded":"2023-10-16T11:20:38.846260+0000","last_fullsized":"2023-10-16T11:20:38.846260+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:00.023220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.809179+0000","last_change":"2023-10-16T11:20:25.747416+0000","last_active":"2023-10-16T11:20:37.809179+0000","last_peered":"2023-10-16T11:20:37.809179+0000","last_clean":"2023-10-16T11:20:37.809179+0000","last_became_active":"2023-10-16T11:20:25.746950+0000","last_became_peered":"2023-10-16T11:20:25.746950+0000","last_unstale":"2023-10-16T11:20:37.809179+0000","last_undegraded":"2023-10-16T11:20:37.809179+0000","last_fullsized":"2023-10-16T11:20:37.809179+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843591+0000","last_change":"2023-10-16T11:20:38.842102+0000","last_active":"2023-10-16T11:20:38.843591+0000","last_peered":"2023-10-16T11:20:38.843591+0000","last_clean":"2023-10-16T11:20:38.843591+0000","last_became_active":"2023-10-16T11:20:38.841601+0000","last_became_peered":"2023-10-16T11:20:38.841601+0000","last_unstale":"2023-10-16T11:20:38.843591+0000","last_undegraded":"2023-10-16T11:20:38.843591+0000","last_fullsized":"2023-10-16T11:20:38.843591+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:26.708062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"240'3","reported_seq":14,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842625+0000","last_change":"2023-10-16T11:20:37.818796+0000","last_active":"2023-10-16T11:20:38.842625+0000","last_peered":"2023-10-16T11:20:38.842625+0000","last_clean":"2023-10-16T11:20:38.842625+0000","last_became_active":"2023-10-16T11:20:37.818545+0000","last_became_peered":"2023-10-16T11:20:37.818545+0000","last_unstale":"2023-10-16T11:20:38.842625+0000","last_undegraded":"2023-10-16T11:20:38.842625+0000","last_fullsized":"2023-10-16T11:20:38.842625+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:05.678488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807590+0000","last_change":"2023-10-16T11:20:25.743709+0000","last_active":"2023-10-16T11:20:37.807590+0000","last_peered":"2023-10-16T11:20:37.807590+0000","last_clean":"2023-10-16T11:20:37.807590+0000","last_became_active":"2023-10-16T11:20:25.743444+0000","last_became_peered":"2023-10-16T11:20:25.743444+0000","last_unstale":"2023-10-16T11:20:37.807590+0000","last_undegraded":"2023-10-16T11:20:37.807590+0000","last_fullsized":"2023-10-16T11:20:37.807590+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:28.662644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836955+0000","last_change":"2023-10-16T11:20:38.834327+0000","last_active":"2023-10-16T11:20:38.836955+0000","last_peered":"2023-10-16T11:20:38.836955+0000","last_clean":"2023-10-16T11:20:38.836955+0000","last_became_active":"2023-10-16T11:20:38.833988+0000","last_became_peered":"2023-10-16T11:20:38.833988+0000","last_unstale":"2023-10-16T11:20:38.836955+0000","last_undegraded":"2023-10-16T11:20:38.836955+0000","last_fullsized":"2023-10-16T11:20:38.836955+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:17.884888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.818487+0000","last_change":"2023-10-16T11:20:37.816942+0000","last_active":"2023-10-16T11:20:38.818487+0000","last_peered":"2023-10-16T11:20:38.818487+0000","last_clean":"2023-10-16T11:20:38.818487+0000","last_became_active":"2023-10-16T11:20:37.815666+0000","last_became_peered":"2023-10-16T11:20:37.815666+0000","last_unstale":"2023-10-16T11:20:38.818487+0000","last_undegraded":"2023-10-16T11:20:38.818487+0000","last_fullsized":"2023-10-16T11:20:38.818487+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:03.222103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810201+0000","last_change":"2023-10-16T11:20:25.737922+0000","last_active":"2023-10-16T11:20:37.810201+0000","last_peered":"2023-10-16T11:20:37.810201+0000","last_clean":"2023-10-16T11:20:37.810201+0000","last_became_active":"2023-10-16T11:20:25.737086+0000","last_became_peered":"2023-10-16T11:20:25.737086+0000","last_unstale":"2023-10-16T11:20:37.810201+0000","last_undegraded":"2023-10-16T11:20:37.810201+0000","last_fullsized":"2023-10-16T11:20:37.810201+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:17.897224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839810+0000","last_change":"2023-10-16T11:20:38.833495+0000","last_active":"2023-10-16T11:20:38.839810+0000","last_peered":"2023-10-16T11:20:38.839810+0000","last_clean":"2023-10-16T11:20:38.839810+0000","last_became_active":"2023-10-16T11:20:38.833247+0000","last_became_peered":"2023-10-16T11:20:38.833247+0000","last_unstale":"2023-10-16T11:20:38.839810+0000","last_undegraded":"2023-10-16T11:20:38.839810+0000","last_fullsized":"2023-10-16T11:20:38.839810+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:32.097784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840955+0000","last_change":"2023-10-16T11:20:37.818788+0000","last_active":"2023-10-16T11:20:38.840955+0000","last_peered":"2023-10-16T11:20:38.840955+0000","last_clean":"2023-10-16T11:20:38.840955+0000","last_became_active":"2023-10-16T11:20:37.818537+0000","last_became_peered":"2023-10-16T11:20:37.818537+0000","last_unstale":"2023-10-16T11:20:38.840955+0000","last_undegraded":"2023-10-16T11:20:38.840955+0000","last_fullsized":"2023-10-16T11:20:38.840955+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:46.682699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813692+0000","last_change":"2023-10-16T11:20:25.745458+0000","last_active":"2023-10-16T11:20:37.813692+0000","last_peered":"2023-10-16T11:20:37.813692+0000","last_clean":"2023-10-16T11:20:37.813692+0000","last_became_active":"2023-10-16T11:20:25.745257+0000","last_became_peered":"2023-10-16T11:20:25.745257+0000","last_unstale":"2023-10-16T11:20:37.813692+0000","last_undegraded":"2023-10-16T11:20:37.813692+0000","last_fullsized":"2023-10-16T11:20:37.813692+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832698+0000","last_change":"2023-10-16T11:20:38.830760+0000","last_active":"2023-10-16T11:20:38.832698+0000","last_peered":"2023-10-16T11:20:38.832698+0000","last_clean":"2023-10-16T11:20:38.832698+0000","last_became_active":"2023-10-16T11:20:38.830505+0000","last_became_peered":"2023-10-16T11:20:38.830505+0000","last_unstale":"2023-10-16T11:20:38.832698+0000","last_undegraded":"2023-10-16T11:20:38.832698+0000","last_fullsized":"2023-10-16T11:20:38.832698+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:51.210622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"240'8","reported_seq":19,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843703+0000","last_change":"2023-10-16T11:20:37.823469+0000","last_active":"2023-10-16T11:20:38.843703+0000","last_peered":"2023-10-16T11:20:38.843703+0000","last_clean":"2023-10-16T11:20:38.843703+0000","last_became_active":"2023-10-16T11:20:37.823231+0000","last_became_peered":"2023-10-16T11:20:37.823231+0000","last_unstale":"2023-10-16T11:20:38.843703+0000","last_undegraded":"2023-10-16T11:20:38.843703+0000","last_fullsized":"2023-10-16T11:20:38.843703+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:47.140140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803677+0000","last_change":"2023-10-16T11:20:25.741269+0000","last_active":"2023-10-16T11:20:37.803677+0000","last_peered":"2023-10-16T11:20:37.803677+0000","last_clean":"2023-10-16T11:20:37.803677+0000","last_became_active":"2023-10-16T11:20:25.740995+0000","last_became_peered":"2023-10-16T11:20:25.740995+0000","last_unstale":"2023-10-16T11:20:37.803677+0000","last_undegraded":"2023-10-16T11:20:37.803677+0000","last_fullsized":"2023-10-16T11:20:37.803677+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:45.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839783+0000","last_change":"2023-10-16T11:20:38.834668+0000","last_active":"2023-10-16T11:20:38.839783+0000","last_peered":"2023-10-16T11:20:38.839783+0000","last_clean":"2023-10-16T11:20:38.839783+0000","last_became_active":"2023-10-16T11:20:38.834405+0000","last_became_peered":"2023-10-16T11:20:38.834405+0000","last_unstale":"2023-10-16T11:20:38.839783+0000","last_undegraded":"2023-10-16T11:20:38.839783+0000","last_fullsized":"2023-10-16T11:20:38.839783+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:06.544476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.816217+0000","last_change":"2023-10-16T11:20:37.822881+0000","last_active":"2023-10-16T11:20:38.816217+0000","last_peered":"2023-10-16T11:20:38.816217+0000","last_clean":"2023-10-16T11:20:38.816217+0000","last_became_active":"2023-10-16T11:20:37.822663+0000","last_became_peered":"2023-10-16T11:20:37.822663+0000","last_unstale":"2023-10-16T11:20:38.816217+0000","last_undegraded":"2023-10-16T11:20:38.816217+0000","last_fullsized":"2023-10-16T11:20:38.816217+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:08.064858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813287+0000","last_change":"2023-10-16T11:20:25.742299+0000","last_active":"2023-10-16T11:20:37.813287+0000","last_peered":"2023-10-16T11:20:37.813287+0000","last_clean":"2023-10-16T11:20:37.813287+0000","last_became_active":"2023-10-16T11:20:25.741394+0000","last_became_peered":"2023-10-16T11:20:25.741394+0000","last_unstale":"2023-10-16T11:20:37.813287+0000","last_undegraded":"2023-10-16T11:20:37.813287+0000","last_fullsized":"2023-10-16T11:20:37.813287+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:35.560067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839006+0000","last_change":"2023-10-16T11:20:38.835643+0000","last_active":"2023-10-16T11:20:38.839006+0000","last_peered":"2023-10-16T11:20:38.839006+0000","last_clean":"2023-10-16T11:20:38.839006+0000","last_became_active":"2023-10-16T11:20:38.835197+0000","last_became_peered":"2023-10-16T11:20:38.835197+0000","last_unstale":"2023-10-16T11:20:38.839006+0000","last_undegraded":"2023-10-16T11:20:38.839006+0000","last_fullsized":"2023-10-16T11:20:38.839006+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:08.451101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844172+0000","last_change":"2023-10-16T11:20:37.824771+0000","last_active":"2023-10-16T11:20:38.844172+0000","last_peered":"2023-10-16T11:20:38.844172+0000","last_clean":"2023-10-16T11:20:38.844172+0000","last_became_active":"2023-10-16T11:20:37.823889+0000","last_became_peered":"2023-10-16T11:20:37.823889+0000","last_unstale":"2023-10-16T11:20:38.844172+0000","last_undegraded":"2023-10-16T11:20:38.844172+0000","last_fullsized":"2023-10-16T11:20:38.844172+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:19.853934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811254+0000","last_change":"2023-10-16T11:20:25.744534+0000","last_active":"2023-10-16T11:20:37.811254+0000","last_peered":"2023-10-16T11:20:37.811254+0000","last_clean":"2023-10-16T11:20:37.811254+0000","last_became_active":"2023-10-16T11:20:25.744224+0000","last_became_peered":"2023-10-16T11:20:25.744224+0000","last_unstale":"2023-10-16T11:20:37.811254+0000","last_undegraded":"2023-10-16T11:20:37.811254+0000","last_fullsized":"2023-10-16T11:20:37.811254+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:14.437943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837184+0000","last_change":"2023-10-16T11:20:38.833376+0000","last_active":"2023-10-16T11:20:38.837184+0000","last_peered":"2023-10-16T11:20:38.837184+0000","last_clean":"2023-10-16T11:20:38.837184+0000","last_became_active":"2023-10-16T11:20:38.833081+0000","last_became_peered":"2023-10-16T11:20:38.833081+0000","last_unstale":"2023-10-16T11:20:38.837184+0000","last_undegraded":"2023-10-16T11:20:38.837184+0000","last_fullsized":"2023-10-16T11:20:38.837184+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:56.175759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844934+0000","last_change":"2023-10-16T11:20:37.824665+0000","last_active":"2023-10-16T11:20:38.844934+0000","last_peered":"2023-10-16T11:20:38.844934+0000","last_clean":"2023-10-16T11:20:38.844934+0000","last_became_active":"2023-10-16T11:20:37.824328+0000","last_became_peered":"2023-10-16T11:20:37.824328+0000","last_unstale":"2023-10-16T11:20:38.844934+0000","last_undegraded":"2023-10-16T11:20:38.844934+0000","last_fullsized":"2023-10-16T11:20:38.844934+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:29.555280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806853+0000","last_change":"2023-10-16T11:20:25.747551+0000","last_active":"2023-10-16T11:20:37.806853+0000","last_peered":"2023-10-16T11:20:37.806853+0000","last_clean":"2023-10-16T11:20:37.806853+0000","last_became_active":"2023-10-16T11:20:25.746989+0000","last_became_peered":"2023-10-16T11:20:25.746989+0000","last_unstale":"2023-10-16T11:20:37.806853+0000","last_undegraded":"2023-10-16T11:20:37.806853+0000","last_fullsized":"2023-10-16T11:20:37.806853+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837089+0000","last_change":"2023-10-16T11:20:38.834474+0000","last_active":"2023-10-16T11:20:38.837089+0000","last_peered":"2023-10-16T11:20:38.837089+0000","last_clean":"2023-10-16T11:20:38.837089+0000","last_became_active":"2023-10-16T11:20:38.833723+0000","last_became_peered":"2023-10-16T11:20:38.833723+0000","last_unstale":"2023-10-16T11:20:38.837089+0000","last_undegraded":"2023-10-16T11:20:38.837089+0000","last_fullsized":"2023-10-16T11:20:38.837089+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:06.806707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.814098+0000","last_change":"2023-10-16T11:20:37.824288+0000","last_active":"2023-10-16T11:20:38.814098+0000","last_peered":"2023-10-16T11:20:38.814098+0000","last_clean":"2023-10-16T11:20:38.814098+0000","last_became_active":"2023-10-16T11:20:37.824006+0000","last_became_peered":"2023-10-16T11:20:37.824006+0000","last_unstale":"2023-10-16T11:20:38.814098+0000","last_undegraded":"2023-10-16T11:20:38.814098+0000","last_fullsized":"2023-10-16T11:20:38.814098+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:26.795558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804059+0000","last_change":"2023-10-16T11:20:25.738568+0000","last_active":"2023-10-16T11:20:37.804059+0000","last_peered":"2023-10-16T11:20:37.804059+0000","last_clean":"2023-10-16T11:20:37.804059+0000","last_became_active":"2023-10-16T11:20:25.738313+0000","last_became_peered":"2023-10-16T11:20:25.738313+0000","last_unstale":"2023-10-16T11:20:37.804059+0000","last_undegraded":"2023-10-16T11:20:37.804059+0000","last_fullsized":"2023-10-16T11:20:37.804059+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:28.152434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.835474+0000","last_change":"2023-10-16T11:20:38.828989+0000","last_active":"2023-10-16T11:20:38.835474+0000","last_peered":"2023-10-16T11:20:38.835474+0000","last_clean":"2023-10-16T11:20:38.835474+0000","last_became_active":"2023-10-16T11:20:38.828709+0000","last_became_peered":"2023-10-16T11:20:38.828709+0000","last_unstale":"2023-10-16T11:20:38.835474+0000","last_undegraded":"2023-10-16T11:20:38.835474+0000","last_fullsized":"2023-10-16T11:20:38.835474+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:20.029437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"240'1","reported_seq":12,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843092+0000","last_change":"2023-10-16T11:20:37.819719+0000","last_active":"2023-10-16T11:20:38.843092+0000","last_peered":"2023-10-16T11:20:38.843092+0000","last_clean":"2023-10-16T11:20:38.843092+0000","last_became_active":"2023-10-16T11:20:37.819410+0000","last_became_peered":"2023-10-16T11:20:37.819410+0000","last_unstale":"2023-10-16T11:20:38.843092+0000","last_undegraded":"2023-10-16T11:20:38.843092+0000","last_fullsized":"2023-10-16T11:20:38.843092+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:38.939710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.819400+0000","last_change":"2023-10-16T11:20:25.750312+0000","last_active":"2023-10-16T11:20:37.819400+0000","last_peered":"2023-10-16T11:20:37.819400+0000","last_clean":"2023-10-16T11:20:37.819400+0000","last_became_active":"2023-10-16T11:20:25.749320+0000","last_became_peered":"2023-10-16T11:20:25.749320+0000","last_unstale":"2023-10-16T11:20:37.819400+0000","last_undegraded":"2023-10-16T11:20:37.819400+0000","last_fullsized":"2023-10-16T11:20:37.819400+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:26.044494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834192+0000","last_change":"2023-10-16T11:20:38.824994+0000","last_active":"2023-10-16T11:20:38.834192+0000","last_peered":"2023-10-16T11:20:38.834192+0000","last_clean":"2023-10-16T11:20:38.834192+0000","last_became_active":"2023-10-16T11:20:38.824683+0000","last_became_peered":"2023-10-16T11:20:38.824683+0000","last_unstale":"2023-10-16T11:20:38.834192+0000","last_undegraded":"2023-10-16T11:20:38.834192+0000","last_fullsized":"2023-10-16T11:20:38.834192+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:16.319016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.823180+0000","last_change":"2023-10-16T11:20:37.814633+0000","last_active":"2023-10-16T11:20:38.823180+0000","last_peered":"2023-10-16T11:20:38.823180+0000","last_clean":"2023-10-16T11:20:38.823180+0000","last_became_active":"2023-10-16T11:20:37.814347+0000","last_became_peered":"2023-10-16T11:20:37.814347+0000","last_unstale":"2023-10-16T11:20:38.823180+0000","last_undegraded":"2023-10-16T11:20:38.823180+0000","last_fullsized":"2023-10-16T11:20:38.823180+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:24.767953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816618+0000","last_change":"2023-10-16T11:20:25.744924+0000","last_active":"2023-10-16T11:20:37.816618+0000","last_peered":"2023-10-16T11:20:37.816618+0000","last_clean":"2023-10-16T11:20:37.816618+0000","last_became_active":"2023-10-16T11:20:25.744659+0000","last_became_peered":"2023-10-16T11:20:25.744659+0000","last_unstale":"2023-10-16T11:20:37.816618+0000","last_undegraded":"2023-10-16T11:20:37.816618+0000","last_fullsized":"2023-10-16T11:20:37.816618+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.833124+0000","last_change":"2023-10-16T11:20:38.824167+0000","last_active":"2023-10-16T11:20:38.833124+0000","last_peered":"2023-10-16T11:20:38.833124+0000","last_clean":"2023-10-16T11:20:38.833124+0000","last_became_active":"2023-10-16T11:20:38.823749+0000","last_became_peered":"2023-10-16T11:20:38.823749+0000","last_unstale":"2023-10-16T11:20:38.833124+0000","last_undegraded":"2023-10-16T11:20:38.833124+0000","last_fullsized":"2023-10-16T11:20:38.833124+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:06.544476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844428+0000","last_change":"2023-10-16T11:20:37.818380+0000","last_active":"2023-10-16T11:20:38.844428+0000","last_peered":"2023-10-16T11:20:38.844428+0000","last_clean":"2023-10-16T11:20:38.844428+0000","last_became_active":"2023-10-16T11:20:37.818042+0000","last_became_peered":"2023-10-16T11:20:37.818042+0000","last_unstale":"2023-10-16T11:20:38.844428+0000","last_undegraded":"2023-10-16T11:20:38.844428+0000","last_fullsized":"2023-10-16T11:20:38.844428+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:38.939710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810348+0000","last_change":"2023-10-16T11:20:25.750717+0000","last_active":"2023-10-16T11:20:37.810348+0000","last_peered":"2023-10-16T11:20:37.810348+0000","last_clean":"2023-10-16T11:20:37.810348+0000","last_became_active":"2023-10-16T11:20:25.750419+0000","last_became_peered":"2023-10-16T11:20:25.750419+0000","last_unstale":"2023-10-16T11:20:37.810348+0000","last_undegraded":"2023-10-16T11:20:37.810348+0000","last_fullsized":"2023-10-16T11:20:37.810348+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:21.095229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.841828+0000","last_change":"2023-10-16T11:20:38.833965+0000","last_active":"2023-10-16T11:20:38.841828+0000","last_peered":"2023-10-16T11:20:38.841828+0000","last_clean":"2023-10-16T11:20:38.841828+0000","last_became_active":"2023-10-16T11:20:38.833701+0000","last_became_peered":"2023-10-16T11:20:38.833701+0000","last_unstale":"2023-10-16T11:20:38.841828+0000","last_undegraded":"2023-10-16T11:20:38.841828+0000","last_fullsized":"2023-10-16T11:20:38.841828+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:54.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.819135+0000","last_change":"2023-10-16T11:20:37.812983+0000","last_active":"2023-10-16T11:20:38.819135+0000","last_peered":"2023-10-16T11:20:38.819135+0000","last_clean":"2023-10-16T11:20:38.819135+0000","last_became_active":"2023-10-16T11:20:37.812653+0000","last_became_peered":"2023-10-16T11:20:37.812653+0000","last_unstale":"2023-10-16T11:20:38.819135+0000","last_undegraded":"2023-10-16T11:20:38.819135+0000","last_fullsized":"2023-10-16T11:20:38.819135+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:29.758492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803783+0000","last_change":"2023-10-16T11:20:25.738444+0000","last_active":"2023-10-16T11:20:37.803783+0000","last_peered":"2023-10-16T11:20:37.803783+0000","last_clean":"2023-10-16T11:20:37.803783+0000","last_became_active":"2023-10-16T11:20:25.738166+0000","last_became_peered":"2023-10-16T11:20:25.738166+0000","last_unstale":"2023-10-16T11:20:37.803783+0000","last_undegraded":"2023-10-16T11:20:37.803783+0000","last_fullsized":"2023-10-16T11:20:37.803783+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:57.655885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836065+0000","last_change":"2023-10-16T11:20:38.825862+0000","last_active":"2023-10-16T11:20:38.836065+0000","last_peered":"2023-10-16T11:20:38.836065+0000","last_clean":"2023-10-16T11:20:38.836065+0000","last_became_active":"2023-10-16T11:20:38.825651+0000","last_became_peered":"2023-10-16T11:20:38.825651+0000","last_unstale":"2023-10-16T11:20:38.836065+0000","last_undegraded":"2023-10-16T11:20:38.836065+0000","last_fullsized":"2023-10-16T11:20:38.836065+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:40.135422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.822987+0000","last_change":"2023-10-16T11:20:37.824585+0000","last_active":"2023-10-16T11:20:38.822987+0000","last_peered":"2023-10-16T11:20:38.822987+0000","last_clean":"2023-10-16T11:20:38.822987+0000","last_became_active":"2023-10-16T11:20:37.824308+0000","last_became_peered":"2023-10-16T11:20:37.824308+0000","last_unstale":"2023-10-16T11:20:38.822987+0000","last_undegraded":"2023-10-16T11:20:38.822987+0000","last_fullsized":"2023-10-16T11:20:38.822987+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:53.964759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810288+0000","last_change":"2023-10-16T11:20:25.741211+0000","last_active":"2023-10-16T11:20:37.810288+0000","last_peered":"2023-10-16T11:20:37.810288+0000","last_clean":"2023-10-16T11:20:37.810288+0000","last_became_active":"2023-10-16T11:20:25.740922+0000","last_became_peered":"2023-10-16T11:20:25.740922+0000","last_unstale":"2023-10-16T11:20:37.810288+0000","last_undegraded":"2023-10-16T11:20:37.810288+0000","last_fullsized":"2023-10-16T11:20:37.810288+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:55.904667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843615+0000","last_change":"2023-10-16T11:20:38.840850+0000","last_active":"2023-10-16T11:20:38.843615+0000","last_peered":"2023-10-16T11:20:38.843615+0000","last_clean":"2023-10-16T11:20:38.843615+0000","last_became_active":"2023-10-16T11:20:38.840587+0000","last_became_peered":"2023-10-16T11:20:38.840587+0000","last_unstale":"2023-10-16T11:20:38.843615+0000","last_undegraded":"2023-10-16T11:20:38.843615+0000","last_fullsized":"2023-10-16T11:20:38.843615+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:06.896559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.814558+0000","last_change":"2023-10-16T11:20:37.824584+0000","last_active":"2023-10-16T11:20:38.814558+0000","last_peered":"2023-10-16T11:20:38.814558+0000","last_clean":"2023-10-16T11:20:38.814558+0000","last_became_active":"2023-10-16T11:20:37.824309+0000","last_became_peered":"2023-10-16T11:20:37.824309+0000","last_unstale":"2023-10-16T11:20:38.814558+0000","last_undegraded":"2023-10-16T11:20:38.814558+0000","last_fullsized":"2023-10-16T11:20:38.814558+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:06.259639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.819373+0000","last_change":"2023-10-16T11:20:25.741581+0000","last_active":"2023-10-16T11:20:37.819373+0000","last_peered":"2023-10-16T11:20:37.819373+0000","last_clean":"2023-10-16T11:20:37.819373+0000","last_became_active":"2023-10-16T11:20:25.740490+0000","last_became_peered":"2023-10-16T11:20:25.740490+0000","last_unstale":"2023-10-16T11:20:37.819373+0000","last_undegraded":"2023-10-16T11:20:37.819373+0000","last_fullsized":"2023-10-16T11:20:37.819373+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:20.166253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842614+0000","last_change":"2023-10-16T11:20:38.833860+0000","last_active":"2023-10-16T11:20:38.842614+0000","last_peered":"2023-10-16T11:20:38.842614+0000","last_clean":"2023-10-16T11:20:38.842614+0000","last_became_active":"2023-10-16T11:20:38.833570+0000","last_became_peered":"2023-10-16T11:20:38.833570+0000","last_unstale":"2023-10-16T11:20:38.842614+0000","last_undegraded":"2023-10-16T11:20:38.842614+0000","last_fullsized":"2023-10-16T11:20:38.842614+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:21.540202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"240'2","reported_seq":13,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837386+0000","last_change":"2023-10-16T11:20:37.812869+0000","last_active":"2023-10-16T11:20:38.837386+0000","last_peered":"2023-10-16T11:20:38.837386+0000","last_clean":"2023-10-16T11:20:38.837386+0000","last_became_active":"2023-10-16T11:20:37.812556+0000","last_became_peered":"2023-10-16T11:20:37.812556+0000","last_unstale":"2023-10-16T11:20:38.837386+0000","last_undegraded":"2023-10-16T11:20:38.837386+0000","last_fullsized":"2023-10-16T11:20:38.837386+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:21.865730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816718+0000","last_change":"2023-10-16T11:20:25.747818+0000","last_active":"2023-10-16T11:20:37.816718+0000","last_peered":"2023-10-16T11:20:37.816718+0000","last_clean":"2023-10-16T11:20:37.816718+0000","last_became_active":"2023-10-16T11:20:25.747533+0000","last_became_peered":"2023-10-16T11:20:25.747533+0000","last_unstale":"2023-10-16T11:20:37.816718+0000","last_undegraded":"2023-10-16T11:20:37.816718+0000","last_fullsized":"2023-10-16T11:20:37.816718+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843593+0000","last_change":"2023-10-16T11:20:38.841998+0000","last_active":"2023-10-16T11:20:38.843593+0000","last_peered":"2023-10-16T11:20:38.843593+0000","last_clean":"2023-10-16T11:20:38.843593+0000","last_became_active":"2023-10-16T11:20:38.838632+0000","last_became_peered":"2023-10-16T11:20:38.838632+0000","last_unstale":"2023-10-16T11:20:38.843593+0000","last_undegraded":"2023-10-16T11:20:38.843593+0000","last_fullsized":"2023-10-16T11:20:38.843593+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:30.044114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.828808+0000","last_change":"2023-10-16T11:20:37.819685+0000","last_active":"2023-10-16T11:20:38.828808+0000","last_peered":"2023-10-16T11:20:38.828808+0000","last_clean":"2023-10-16T11:20:38.828808+0000","last_became_active":"2023-10-16T11:20:37.819375+0000","last_became_peered":"2023-10-16T11:20:37.819375+0000","last_unstale":"2023-10-16T11:20:38.828808+0000","last_undegraded":"2023-10-16T11:20:38.828808+0000","last_fullsized":"2023-10-16T11:20:38.828808+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:53.867762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814009+0000","last_change":"2023-10-16T11:20:25.746406+0000","last_active":"2023-10-16T11:20:37.814009+0000","last_peered":"2023-10-16T11:20:37.814009+0000","last_clean":"2023-10-16T11:20:37.814009+0000","last_became_active":"2023-10-16T11:20:25.746140+0000","last_became_peered":"2023-10-16T11:20:25.746140+0000","last_unstale":"2023-10-16T11:20:37.814009+0000","last_undegraded":"2023-10-16T11:20:37.814009+0000","last_fullsized":"2023-10-16T11:20:37.814009+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:42.488771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836035+0000","last_change":"2023-10-16T11:20:38.834839+0000","last_active":"2023-10-16T11:20:38.836035+0000","last_peered":"2023-10-16T11:20:38.836035+0000","last_clean":"2023-10-16T11:20:38.836035+0000","last_became_active":"2023-10-16T11:20:38.833518+0000","last_became_peered":"2023-10-16T11:20:38.833518+0000","last_unstale":"2023-10-16T11:20:38.836035+0000","last_undegraded":"2023-10-16T11:20:38.836035+0000","last_fullsized":"2023-10-16T11:20:38.836035+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:32.097784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.827001+0000","last_change":"2023-10-16T11:20:37.820664+0000","last_active":"2023-10-16T11:20:38.827001+0000","last_peered":"2023-10-16T11:20:38.827001+0000","last_clean":"2023-10-16T11:20:38.827001+0000","last_became_active":"2023-10-16T11:20:37.820394+0000","last_became_peered":"2023-10-16T11:20:37.820394+0000","last_unstale":"2023-10-16T11:20:38.827001+0000","last_undegraded":"2023-10-16T11:20:38.827001+0000","last_fullsized":"2023-10-16T11:20:38.827001+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:46.682699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807677+0000","last_change":"2023-10-16T11:20:25.750643+0000","last_active":"2023-10-16T11:20:37.807677+0000","last_peered":"2023-10-16T11:20:37.807677+0000","last_clean":"2023-10-16T11:20:37.807677+0000","last_became_active":"2023-10-16T11:20:25.749980+0000","last_became_peered":"2023-10-16T11:20:25.749980+0000","last_unstale":"2023-10-16T11:20:37.807677+0000","last_undegraded":"2023-10-16T11:20:37.807677+0000","last_fullsized":"2023-10-16T11:20:37.807677+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:41.498772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843389+0000","last_change":"2023-10-16T11:20:38.841575+0000","last_active":"2023-10-16T11:20:38.843389+0000","last_peered":"2023-10-16T11:20:38.843389+0000","last_clean":"2023-10-16T11:20:38.843389+0000","last_became_active":"2023-10-16T11:20:38.841319+0000","last_became_peered":"2023-10-16T11:20:38.841319+0000","last_unstale":"2023-10-16T11:20:38.843389+0000","last_undegraded":"2023-10-16T11:20:38.843389+0000","last_fullsized":"2023-10-16T11:20:38.843389+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:47.690992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.813543+0000","last_change":"2023-10-16T11:20:37.825703+0000","last_active":"2023-10-16T11:20:38.813543+0000","last_peered":"2023-10-16T11:20:38.813543+0000","last_clean":"2023-10-16T11:20:38.813543+0000","last_became_active":"2023-10-16T11:20:37.825425+0000","last_became_peered":"2023-10-16T11:20:37.825425+0000","last_unstale":"2023-10-16T11:20:38.813543+0000","last_undegraded":"2023-10-16T11:20:38.813543+0000","last_fullsized":"2023-10-16T11:20:38.813543+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:42:28.228873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813543+0000","last_change":"2023-10-16T11:20:25.744400+0000","last_active":"2023-10-16T11:20:37.813543+0000","last_peered":"2023-10-16T11:20:37.813543+0000","last_clean":"2023-10-16T11:20:37.813543+0000","last_became_active":"2023-10-16T11:20:25.744118+0000","last_became_peered":"2023-10-16T11:20:25.744118+0000","last_unstale":"2023-10-16T11:20:37.813543+0000","last_undegraded":"2023-10-16T11:20:37.813543+0000","last_fullsized":"2023-10-16T11:20:37.813543+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839580+0000","last_change":"2023-10-16T11:20:38.835993+0000","last_active":"2023-10-16T11:20:38.839580+0000","last_peered":"2023-10-16T11:20:38.839580+0000","last_clean":"2023-10-16T11:20:38.839580+0000","last_became_active":"2023-10-16T11:20:38.835750+0000","last_became_peered":"2023-10-16T11:20:38.835750+0000","last_unstale":"2023-10-16T11:20:38.839580+0000","last_undegraded":"2023-10-16T11:20:38.839580+0000","last_fullsized":"2023-10-16T11:20:38.839580+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:18.097381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.813626+0000","last_change":"2023-10-16T11:20:37.824421+0000","last_active":"2023-10-16T11:20:38.813626+0000","last_peered":"2023-10-16T11:20:38.813626+0000","last_clean":"2023-10-16T11:20:38.813626+0000","last_became_active":"2023-10-16T11:20:37.824028+0000","last_became_peered":"2023-10-16T11:20:37.824028+0000","last_unstale":"2023-10-16T11:20:38.813626+0000","last_undegraded":"2023-10-16T11:20:38.813626+0000","last_fullsized":"2023-10-16T11:20:38.813626+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:57.821106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813784+0000","last_change":"2023-10-16T11:20:25.749096+0000","last_active":"2023-10-16T11:20:37.813784+0000","last_peered":"2023-10-16T11:20:37.813784+0000","last_clean":"2023-10-16T11:20:37.813784+0000","last_became_active":"2023-10-16T11:20:25.748841+0000","last_became_peered":"2023-10-16T11:20:25.748841+0000","last_unstale":"2023-10-16T11:20:37.813784+0000","last_undegraded":"2023-10-16T11:20:37.813784+0000","last_fullsized":"2023-10-16T11:20:37.813784+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836886+0000","last_change":"2023-10-16T11:20:38.830851+0000","last_active":"2023-10-16T11:20:38.836886+0000","last_peered":"2023-10-16T11:20:38.836886+0000","last_clean":"2023-10-16T11:20:38.836886+0000","last_became_active":"2023-10-16T11:20:38.830453+0000","last_became_peered":"2023-10-16T11:20:38.830453+0000","last_unstale":"2023-10-16T11:20:38.836886+0000","last_undegraded":"2023-10-16T11:20:38.836886+0000","last_fullsized":"2023-10-16T11:20:38.836886+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:19.647950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.813660+0000","last_change":"2023-10-16T11:20:37.823025+0000","last_active":"2023-10-16T11:20:38.813660+0000","last_peered":"2023-10-16T11:20:38.813660+0000","last_clean":"2023-10-16T11:20:38.813660+0000","last_became_active":"2023-10-16T11:20:37.822440+0000","last_became_peered":"2023-10-16T11:20:37.822440+0000","last_unstale":"2023-10-16T11:20:38.813660+0000","last_undegraded":"2023-10-16T11:20:38.813660+0000","last_fullsized":"2023-10-16T11:20:38.813660+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:32.410180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.817446+0000","last_change":"2023-10-16T11:20:25.741998+0000","last_active":"2023-10-16T11:20:37.817446+0000","last_peered":"2023-10-16T11:20:37.817446+0000","last_clean":"2023-10-16T11:20:37.817446+0000","last_became_active":"2023-10-16T11:20:25.741482+0000","last_became_peered":"2023-10-16T11:20:25.741482+0000","last_unstale":"2023-10-16T11:20:37.817446+0000","last_undegraded":"2023-10-16T11:20:37.817446+0000","last_fullsized":"2023-10-16T11:20:37.817446+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:01.922741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842595+0000","last_change":"2023-10-16T11:20:38.838960+0000","last_active":"2023-10-16T11:20:38.842595+0000","last_peered":"2023-10-16T11:20:38.842595+0000","last_clean":"2023-10-16T11:20:38.842595+0000","last_became_active":"2023-10-16T11:20:38.838636+0000","last_became_peered":"2023-10-16T11:20:38.838636+0000","last_unstale":"2023-10-16T11:20:38.842595+0000","last_undegraded":"2023-10-16T11:20:38.842595+0000","last_fullsized":"2023-10-16T11:20:38.842595+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:41.240676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.825361+0000","last_change":"2023-10-16T11:20:37.824717+0000","last_active":"2023-10-16T11:20:38.825361+0000","last_peered":"2023-10-16T11:20:38.825361+0000","last_clean":"2023-10-16T11:20:38.825361+0000","last_became_active":"2023-10-16T11:20:37.824431+0000","last_became_peered":"2023-10-16T11:20:37.824431+0000","last_unstale":"2023-10-16T11:20:38.825361+0000","last_undegraded":"2023-10-16T11:20:38.825361+0000","last_fullsized":"2023-10-16T11:20:38.825361+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":238,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:36.790590+0000","last_clean_scrub_stamp":"2023-10-16T11:20:36.790590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.825698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807071+0000","last_change":"2023-10-16T11:20:25.750764+0000","last_active":"2023-10-16T11:20:37.807071+0000","last_peered":"2023-10-16T11:20:37.807071+0000","last_clean":"2023-10-16T11:20:37.807071+0000","last_became_active":"2023-10-16T11:20:25.750127+0000","last_became_peered":"2023-10-16T11:20:25.750127+0000","last_unstale":"2023-10-16T11:20:37.807071+0000","last_undegraded":"2023-10-16T11:20:37.807071+0000","last_fullsized":"2023-10-16T11:20:37.807071+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:04.429724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843537+0000","last_change":"2023-10-16T11:20:38.841800+0000","last_active":"2023-10-16T11:20:38.843537+0000","last_peered":"2023-10-16T11:20:38.843537+0000","last_clean":"2023-10-16T11:20:38.843537+0000","last_became_active":"2023-10-16T11:20:38.841534+0000","last_became_peered":"2023-10-16T11:20:38.841534+0000","last_unstale":"2023-10-16T11:20:38.843537+0000","last_undegraded":"2023-10-16T11:20:38.843537+0000","last_fullsized":"2023-10-16T11:20:38.843537+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:03.575170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.818279+0000","last_change":"2023-10-16T11:20:25.752205+0000","last_active":"2023-10-16T11:20:37.818279+0000","last_peered":"2023-10-16T11:20:37.818279+0000","last_clean":"2023-10-16T11:20:37.818279+0000","last_became_active":"2023-10-16T11:20:25.751915+0000","last_became_peered":"2023-10-16T11:20:25.751915+0000","last_unstale":"2023-10-16T11:20:37.818279+0000","last_undegraded":"2023-10-16T11:20:37.818279+0000","last_fullsized":"2023-10-16T11:20:37.818279+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:43.213188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"236'4","reported_seq":19,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804563+0000","last_change":"2023-10-16T11:20:24.733695+0000","last_active":"2023-10-16T11:20:37.804563+0000","last_peered":"2023-10-16T11:20:37.804563+0000","last_clean":"2023-10-16T11:20:37.804563+0000","last_became_active":"2023-10-16T11:20:24.733413+0000","last_became_peered":"2023-10-16T11:20:24.733413+0000","last_unstale":"2023-10-16T11:20:37.804563+0000","last_undegraded":"2023-10-16T11:20:37.804563+0000","last_fullsized":"2023-10-16T11:20:37.804563+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:34.326531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.833144+0000","last_change":"2023-10-16T11:20:38.824981+0000","last_active":"2023-10-16T11:20:38.833144+0000","last_peered":"2023-10-16T11:20:38.833144+0000","last_clean":"2023-10-16T11:20:38.833144+0000","last_became_active":"2023-10-16T11:20:38.824705+0000","last_became_peered":"2023-10-16T11:20:38.824705+0000","last_unstale":"2023-10-16T11:20:38.833144+0000","last_undegraded":"2023-10-16T11:20:38.833144+0000","last_fullsized":"2023-10-16T11:20:38.833144+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:54.833992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804216+0000","last_change":"2023-10-16T11:20:25.747992+0000","last_active":"2023-10-16T11:20:37.804216+0000","last_peered":"2023-10-16T11:20:37.804216+0000","last_clean":"2023-10-16T11:20:37.804216+0000","last_became_active":"2023-10-16T11:20:25.747735+0000","last_became_peered":"2023-10-16T11:20:25.747735+0000","last_unstale":"2023-10-16T11:20:37.804216+0000","last_undegraded":"2023-10-16T11:20:37.804216+0000","last_fullsized":"2023-10-16T11:20:37.804216+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:49.269723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811335+0000","last_change":"2023-10-16T11:20:24.739822+0000","last_active":"2023-10-16T11:20:37.811335+0000","last_peered":"2023-10-16T11:20:37.811335+0000","last_clean":"2023-10-16T11:20:37.811335+0000","last_became_active":"2023-10-16T11:20:24.739470+0000","last_became_peered":"2023-10-16T11:20:24.739470+0000","last_unstale":"2023-10-16T11:20:37.811335+0000","last_undegraded":"2023-10-16T11:20:37.811335+0000","last_fullsized":"2023-10-16T11:20:37.811335+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:18.641488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843370+0000","last_change":"2023-10-16T11:20:38.840727+0000","last_active":"2023-10-16T11:20:38.843370+0000","last_peered":"2023-10-16T11:20:38.843370+0000","last_clean":"2023-10-16T11:20:38.843370+0000","last_became_active":"2023-10-16T11:20:38.840265+0000","last_became_peered":"2023-10-16T11:20:38.840265+0000","last_unstale":"2023-10-16T11:20:38.843370+0000","last_undegraded":"2023-10-16T11:20:38.843370+0000","last_fullsized":"2023-10-16T11:20:38.843370+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:56.021692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.808200+0000","last_change":"2023-10-16T11:20:25.746049+0000","last_active":"2023-10-16T11:20:37.808200+0000","last_peered":"2023-10-16T11:20:37.808200+0000","last_clean":"2023-10-16T11:20:37.808200+0000","last_became_active":"2023-10-16T11:20:25.745759+0000","last_became_peered":"2023-10-16T11:20:25.745759+0000","last_unstale":"2023-10-16T11:20:37.808200+0000","last_undegraded":"2023-10-16T11:20:37.808200+0000","last_fullsized":"2023-10-16T11:20:37.808200+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:40:29.501267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814266+0000","last_change":"2023-10-16T11:20:24.727708+0000","last_active":"2023-10-16T11:20:37.814266+0000","last_peered":"2023-10-16T11:20:37.814266+0000","last_clean":"2023-10-16T11:20:37.814266+0000","last_became_active":"2023-10-16T11:20:24.727357+0000","last_became_peered":"2023-10-16T11:20:24.727357+0000","last_unstale":"2023-10-16T11:20:37.814266+0000","last_undegraded":"2023-10-16T11:20:37.814266+0000","last_fullsized":"2023-10-16T11:20:37.814266+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:32.307241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834217+0000","last_change":"2023-10-16T11:20:38.830140+0000","last_active":"2023-10-16T11:20:38.834217+0000","last_peered":"2023-10-16T11:20:38.834217+0000","last_clean":"2023-10-16T11:20:38.834217+0000","last_became_active":"2023-10-16T11:20:38.829866+0000","last_became_peered":"2023-10-16T11:20:38.829866+0000","last_unstale":"2023-10-16T11:20:38.834217+0000","last_undegraded":"2023-10-16T11:20:38.834217+0000","last_fullsized":"2023-10-16T11:20:38.834217+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:57.577794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813997+0000","last_change":"2023-10-16T11:20:25.742133+0000","last_active":"2023-10-16T11:20:37.813997+0000","last_peered":"2023-10-16T11:20:37.813997+0000","last_clean":"2023-10-16T11:20:37.813997+0000","last_became_active":"2023-10-16T11:20:25.741865+0000","last_became_peered":"2023-10-16T11:20:25.741865+0000","last_unstale":"2023-10-16T11:20:37.813997+0000","last_undegraded":"2023-10-16T11:20:37.813997+0000","last_fullsized":"2023-10-16T11:20:37.813997+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:25.230100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804262+0000","last_change":"2023-10-16T11:20:24.722283+0000","last_active":"2023-10-16T11:20:37.804262+0000","last_peered":"2023-10-16T11:20:37.804262+0000","last_clean":"2023-10-16T11:20:37.804262+0000","last_became_active":"2023-10-16T11:20:24.721789+0000","last_became_peered":"2023-10-16T11:20:24.721789+0000","last_unstale":"2023-10-16T11:20:37.804262+0000","last_undegraded":"2023-10-16T11:20:37.804262+0000","last_fullsized":"2023-10-16T11:20:37.804262+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:37.116862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840495+0000","last_change":"2023-10-16T11:20:38.839286+0000","last_active":"2023-10-16T11:20:38.840495+0000","last_peered":"2023-10-16T11:20:38.840495+0000","last_clean":"2023-10-16T11:20:38.840495+0000","last_became_active":"2023-10-16T11:20:38.839008+0000","last_became_peered":"2023-10-16T11:20:38.839008+0000","last_unstale":"2023-10-16T11:20:38.840495+0000","last_undegraded":"2023-10-16T11:20:38.840495+0000","last_fullsized":"2023-10-16T11:20:38.840495+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:25.776247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816721+0000","last_change":"2023-10-16T11:20:25.748913+0000","last_active":"2023-10-16T11:20:37.816721+0000","last_peered":"2023-10-16T11:20:37.816721+0000","last_clean":"2023-10-16T11:20:37.816721+0000","last_became_active":"2023-10-16T11:20:25.748588+0000","last_became_peered":"2023-10-16T11:20:25.748588+0000","last_unstale":"2023-10-16T11:20:37.816721+0000","last_undegraded":"2023-10-16T11:20:37.816721+0000","last_fullsized":"2023-10-16T11:20:37.816721+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.087951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813877+0000","last_change":"2023-10-16T11:20:24.733622+0000","last_active":"2023-10-16T11:20:37.813877+0000","last_peered":"2023-10-16T11:20:37.813877+0000","last_clean":"2023-10-16T11:20:37.813877+0000","last_became_active":"2023-10-16T11:20:24.733342+0000","last_became_peered":"2023-10-16T11:20:24.733342+0000","last_unstale":"2023-10-16T11:20:37.813877+0000","last_undegraded":"2023-10-16T11:20:37.813877+0000","last_fullsized":"2023-10-16T11:20:37.813877+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:10.962482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837163+0000","last_change":"2023-10-16T11:20:38.832530+0000","last_active":"2023-10-16T11:20:38.837163+0000","last_peered":"2023-10-16T11:20:38.837163+0000","last_clean":"2023-10-16T11:20:38.837163+0000","last_became_active":"2023-10-16T11:20:38.832247+0000","last_became_peered":"2023-10-16T11:20:38.832247+0000","last_unstale":"2023-10-16T11:20:38.837163+0000","last_undegraded":"2023-10-16T11:20:38.837163+0000","last_fullsized":"2023-10-16T11:20:38.837163+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:51.097103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806850+0000","last_change":"2023-10-16T11:20:25.751310+0000","last_active":"2023-10-16T11:20:37.806850+0000","last_peered":"2023-10-16T11:20:37.806850+0000","last_clean":"2023-10-16T11:20:37.806850+0000","last_became_active":"2023-10-16T11:20:25.751042+0000","last_became_peered":"2023-10-16T11:20:25.751042+0000","last_unstale":"2023-10-16T11:20:37.806850+0000","last_undegraded":"2023-10-16T11:20:37.806850+0000","last_fullsized":"2023-10-16T11:20:37.806850+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:11.968000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806543+0000","last_change":"2023-10-16T11:20:24.739609+0000","last_active":"2023-10-16T11:20:37.806543+0000","last_peered":"2023-10-16T11:20:37.806543+0000","last_clean":"2023-10-16T11:20:37.806543+0000","last_became_active":"2023-10-16T11:20:24.739275+0000","last_became_peered":"2023-10-16T11:20:24.739275+0000","last_unstale":"2023-10-16T11:20:37.806543+0000","last_undegraded":"2023-10-16T11:20:37.806543+0000","last_fullsized":"2023-10-16T11:20:37.806543+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:47.827300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837974+0000","last_change":"2023-10-16T11:20:38.835827+0000","last_active":"2023-10-16T11:20:38.837974+0000","last_peered":"2023-10-16T11:20:38.837974+0000","last_clean":"2023-10-16T11:20:38.837974+0000","last_became_active":"2023-10-16T11:20:38.835538+0000","last_became_peered":"2023-10-16T11:20:38.835538+0000","last_unstale":"2023-10-16T11:20:38.837974+0000","last_undegraded":"2023-10-16T11:20:38.837974+0000","last_fullsized":"2023-10-16T11:20:38.837974+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:21.628073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814773+0000","last_change":"2023-10-16T11:20:25.744793+0000","last_active":"2023-10-16T11:20:37.814773+0000","last_peered":"2023-10-16T11:20:37.814773+0000","last_clean":"2023-10-16T11:20:37.814773+0000","last_became_active":"2023-10-16T11:20:25.744521+0000","last_became_peered":"2023-10-16T11:20:25.744521+0000","last_unstale":"2023-10-16T11:20:37.814773+0000","last_undegraded":"2023-10-16T11:20:37.814773+0000","last_fullsized":"2023-10-16T11:20:37.814773+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:44.539888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807124+0000","last_change":"2023-10-16T11:20:24.723659+0000","last_active":"2023-10-16T11:20:37.807124+0000","last_peered":"2023-10-16T11:20:37.807124+0000","last_clean":"2023-10-16T11:20:37.807124+0000","last_became_active":"2023-10-16T11:20:24.723342+0000","last_became_peered":"2023-10-16T11:20:24.723342+0000","last_unstale":"2023-10-16T11:20:37.807124+0000","last_undegraded":"2023-10-16T11:20:37.807124+0000","last_fullsized":"2023-10-16T11:20:37.807124+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:29.707065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.836788+0000","last_change":"2023-10-16T11:20:38.834173+0000","last_active":"2023-10-16T11:20:38.836788+0000","last_peered":"2023-10-16T11:20:38.836788+0000","last_clean":"2023-10-16T11:20:38.836788+0000","last_became_active":"2023-10-16T11:20:38.833870+0000","last_became_peered":"2023-10-16T11:20:38.833870+0000","last_unstale":"2023-10-16T11:20:38.836788+0000","last_undegraded":"2023-10-16T11:20:38.836788+0000","last_fullsized":"2023-10-16T11:20:38.836788+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:26:57.302607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.815911+0000","last_change":"2023-10-16T11:20:25.748568+0000","last_active":"2023-10-16T11:20:37.815911+0000","last_peered":"2023-10-16T11:20:37.815911+0000","last_clean":"2023-10-16T11:20:37.815911+0000","last_became_active":"2023-10-16T11:20:25.748318+0000","last_became_peered":"2023-10-16T11:20:25.748318+0000","last_unstale":"2023-10-16T11:20:37.815911+0000","last_undegraded":"2023-10-16T11:20:37.815911+0000","last_fullsized":"2023-10-16T11:20:37.815911+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.354284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.815289+0000","last_change":"2023-10-16T11:20:24.736095+0000","last_active":"2023-10-16T11:20:37.815289+0000","last_peered":"2023-10-16T11:20:37.815289+0000","last_clean":"2023-10-16T11:20:37.815289+0000","last_became_active":"2023-10-16T11:20:24.735775+0000","last_became_peered":"2023-10-16T11:20:24.735775+0000","last_unstale":"2023-10-16T11:20:37.815289+0000","last_undegraded":"2023-10-16T11:20:37.815289+0000","last_fullsized":"2023-10-16T11:20:37.815289+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:35.485188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842751+0000","last_change":"2023-10-16T11:20:38.840186+0000","last_active":"2023-10-16T11:20:38.842751+0000","last_peered":"2023-10-16T11:20:38.842751+0000","last_clean":"2023-10-16T11:20:38.842751+0000","last_became_active":"2023-10-16T11:20:38.838073+0000","last_became_peered":"2023-10-16T11:20:38.838073+0000","last_unstale":"2023-10-16T11:20:38.842751+0000","last_undegraded":"2023-10-16T11:20:38.842751+0000","last_fullsized":"2023-10-16T11:20:38.842751+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:52.880299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810525+0000","last_change":"2023-10-16T11:20:25.745589+0000","last_active":"2023-10-16T11:20:37.810525+0000","last_peered":"2023-10-16T11:20:37.810525+0000","last_clean":"2023-10-16T11:20:37.810525+0000","last_became_active":"2023-10-16T11:20:25.745297+0000","last_became_peered":"2023-10-16T11:20:25.745297+0000","last_unstale":"2023-10-16T11:20:37.810525+0000","last_undegraded":"2023-10-16T11:20:37.810525+0000","last_fullsized":"2023-10-16T11:20:37.810525+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:17.743037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806958+0000","last_change":"2023-10-16T11:20:24.738634+0000","last_active":"2023-10-16T11:20:37.806958+0000","last_peered":"2023-10-16T11:20:37.806958+0000","last_clean":"2023-10-16T11:20:37.806958+0000","last_became_active":"2023-10-16T11:20:24.738372+0000","last_became_peered":"2023-10-16T11:20:24.738372+0000","last_unstale":"2023-10-16T11:20:37.806958+0000","last_undegraded":"2023-10-16T11:20:37.806958+0000","last_fullsized":"2023-10-16T11:20:37.806958+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:23.747207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832314+0000","last_change":"2023-10-16T11:20:38.820038+0000","last_active":"2023-10-16T11:20:38.832314+0000","last_peered":"2023-10-16T11:20:38.832314+0000","last_clean":"2023-10-16T11:20:38.832314+0000","last_became_active":"2023-10-16T11:20:38.819796+0000","last_became_peered":"2023-10-16T11:20:38.819796+0000","last_unstale":"2023-10-16T11:20:38.832314+0000","last_undegraded":"2023-10-16T11:20:38.832314+0000","last_fullsized":"2023-10-16T11:20:38.832314+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:57.773345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806800+0000","last_change":"2023-10-16T11:20:25.748579+0000","last_active":"2023-10-16T11:20:37.806800+0000","last_peered":"2023-10-16T11:20:37.806800+0000","last_clean":"2023-10-16T11:20:37.806800+0000","last_became_active":"2023-10-16T11:20:25.748347+0000","last_became_peered":"2023-10-16T11:20:25.748347+0000","last_unstale":"2023-10-16T11:20:37.806800+0000","last_undegraded":"2023-10-16T11:20:37.806800+0000","last_fullsized":"2023-10-16T11:20:37.806800+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:06:20.365098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"236'3","reported_seq":18,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814108+0000","last_change":"2023-10-16T11:20:24.728938+0000","last_active":"2023-10-16T11:20:37.814108+0000","last_peered":"2023-10-16T11:20:37.814108+0000","last_clean":"2023-10-16T11:20:37.814108+0000","last_became_active":"2023-10-16T11:20:24.728638+0000","last_became_peered":"2023-10-16T11:20:24.728638+0000","last_unstale":"2023-10-16T11:20:37.814108+0000","last_undegraded":"2023-10-16T11:20:37.814108+0000","last_fullsized":"2023-10-16T11:20:37.814108+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:42.082000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842632+0000","last_change":"2023-10-16T11:20:38.836478+0000","last_active":"2023-10-16T11:20:38.842632+0000","last_peered":"2023-10-16T11:20:38.842632+0000","last_clean":"2023-10-16T11:20:38.842632+0000","last_became_active":"2023-10-16T11:20:38.836190+0000","last_became_peered":"2023-10-16T11:20:38.836190+0000","last_unstale":"2023-10-16T11:20:38.842632+0000","last_undegraded":"2023-10-16T11:20:38.842632+0000","last_fullsized":"2023-10-16T11:20:38.842632+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:52.880299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816526+0000","last_change":"2023-10-16T11:20:25.751420+0000","last_active":"2023-10-16T11:20:37.816526+0000","last_peered":"2023-10-16T11:20:37.816526+0000","last_clean":"2023-10-16T11:20:37.816526+0000","last_became_active":"2023-10-16T11:20:25.751161+0000","last_became_peered":"2023-10-16T11:20:25.751161+0000","last_unstale":"2023-10-16T11:20:37.816526+0000","last_undegraded":"2023-10-16T11:20:37.816526+0000","last_fullsized":"2023-10-16T11:20:37.816526+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:33.312759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"237'8","reported_seq":23,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.807800+0000","last_change":"2023-10-16T11:20:24.724201+0000","last_active":"2023-10-16T11:20:37.807800+0000","last_peered":"2023-10-16T11:20:37.807800+0000","last_clean":"2023-10-16T11:20:37.807800+0000","last_became_active":"2023-10-16T11:20:24.723918+0000","last_became_peered":"2023-10-16T11:20:24.723918+0000","last_unstale":"2023-10-16T11:20:37.807800+0000","last_undegraded":"2023-10-16T11:20:37.807800+0000","last_fullsized":"2023-10-16T11:20:37.807800+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:55.483353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840202+0000","last_change":"2023-10-16T11:20:38.835082+0000","last_active":"2023-10-16T11:20:38.840202+0000","last_peered":"2023-10-16T11:20:38.840202+0000","last_clean":"2023-10-16T11:20:38.840202+0000","last_became_active":"2023-10-16T11:20:38.834792+0000","last_became_peered":"2023-10-16T11:20:38.834792+0000","last_unstale":"2023-10-16T11:20:38.840202+0000","last_undegraded":"2023-10-16T11:20:38.840202+0000","last_fullsized":"2023-10-16T11:20:38.840202+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:55.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.819328+0000","last_change":"2023-10-16T11:20:25.742201+0000","last_active":"2023-10-16T11:20:37.819328+0000","last_peered":"2023-10-16T11:20:37.819328+0000","last_clean":"2023-10-16T11:20:37.819328+0000","last_became_active":"2023-10-16T11:20:25.741871+0000","last_became_peered":"2023-10-16T11:20:25.741871+0000","last_unstale":"2023-10-16T11:20:37.819328+0000","last_undegraded":"2023-10-16T11:20:37.819328+0000","last_fullsized":"2023-10-16T11:20:37.819328+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:18:28.417174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804356+0000","last_change":"2023-10-16T11:20:24.725162+0000","last_active":"2023-10-16T11:20:37.804356+0000","last_peered":"2023-10-16T11:20:37.804356+0000","last_clean":"2023-10-16T11:20:37.804356+0000","last_became_active":"2023-10-16T11:20:24.724795+0000","last_became_peered":"2023-10-16T11:20:24.724795+0000","last_unstale":"2023-10-16T11:20:37.804356+0000","last_undegraded":"2023-10-16T11:20:37.804356+0000","last_fullsized":"2023-10-16T11:20:37.804356+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:25.854243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832471+0000","last_change":"2023-10-16T11:20:38.830717+0000","last_active":"2023-10-16T11:20:38.832471+0000","last_peered":"2023-10-16T11:20:38.832471+0000","last_clean":"2023-10-16T11:20:38.832471+0000","last_became_active":"2023-10-16T11:20:38.830464+0000","last_became_peered":"2023-10-16T11:20:38.830464+0000","last_unstale":"2023-10-16T11:20:38.832471+0000","last_undegraded":"2023-10-16T11:20:38.832471+0000","last_fullsized":"2023-10-16T11:20:38.832471+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:52:15.104758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816809+0000","last_change":"2023-10-16T11:20:25.739623+0000","last_active":"2023-10-16T11:20:37.816809+0000","last_peered":"2023-10-16T11:20:37.816809+0000","last_clean":"2023-10-16T11:20:37.816809+0000","last_became_active":"2023-10-16T11:20:25.739321+0000","last_became_peered":"2023-10-16T11:20:25.739321+0000","last_unstale":"2023-10-16T11:20:37.816809+0000","last_undegraded":"2023-10-16T11:20:37.816809+0000","last_fullsized":"2023-10-16T11:20:37.816809+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:46.772067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813068+0000","last_change":"2023-10-16T11:20:24.735120+0000","last_active":"2023-10-16T11:20:37.813068+0000","last_peered":"2023-10-16T11:20:37.813068+0000","last_clean":"2023-10-16T11:20:37.813068+0000","last_became_active":"2023-10-16T11:20:24.734857+0000","last_became_peered":"2023-10-16T11:20:24.734857+0000","last_unstale":"2023-10-16T11:20:37.813068+0000","last_undegraded":"2023-10-16T11:20:37.813068+0000","last_fullsized":"2023-10-16T11:20:37.813068+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:03.791128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843414+0000","last_change":"2023-10-16T11:20:38.837581+0000","last_active":"2023-10-16T11:20:38.843414+0000","last_peered":"2023-10-16T11:20:38.843414+0000","last_clean":"2023-10-16T11:20:38.843414+0000","last_became_active":"2023-10-16T11:20:38.836911+0000","last_became_peered":"2023-10-16T11:20:38.836911+0000","last_unstale":"2023-10-16T11:20:38.843414+0000","last_undegraded":"2023-10-16T11:20:38.843414+0000","last_fullsized":"2023-10-16T11:20:38.843414+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:40.355466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804303+0000","last_change":"2023-10-16T11:20:25.737997+0000","last_active":"2023-10-16T11:20:37.804303+0000","last_peered":"2023-10-16T11:20:37.804303+0000","last_clean":"2023-10-16T11:20:37.804303+0000","last_became_active":"2023-10-16T11:20:25.737314+0000","last_became_peered":"2023-10-16T11:20:25.737314+0000","last_unstale":"2023-10-16T11:20:37.804303+0000","last_undegraded":"2023-10-16T11:20:37.804303+0000","last_fullsized":"2023-10-16T11:20:37.804303+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:44.489552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813483+0000","last_change":"2023-10-16T11:20:24.733587+0000","last_active":"2023-10-16T11:20:37.813483+0000","last_peered":"2023-10-16T11:20:37.813483+0000","last_clean":"2023-10-16T11:20:37.813483+0000","last_became_active":"2023-10-16T11:20:24.733160+0000","last_became_peered":"2023-10-16T11:20:24.733160+0000","last_unstale":"2023-10-16T11:20:37.813483+0000","last_undegraded":"2023-10-16T11:20:37.813483+0000","last_fullsized":"2023-10-16T11:20:37.813483+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:31.939688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840303+0000","last_change":"2023-10-16T11:20:38.836459+0000","last_active":"2023-10-16T11:20:38.840303+0000","last_peered":"2023-10-16T11:20:38.840303+0000","last_clean":"2023-10-16T11:20:38.840303+0000","last_became_active":"2023-10-16T11:20:38.836106+0000","last_became_peered":"2023-10-16T11:20:38.836106+0000","last_unstale":"2023-10-16T11:20:38.840303+0000","last_undegraded":"2023-10-16T11:20:38.840303+0000","last_fullsized":"2023-10-16T11:20:38.840303+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:46.037088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814690+0000","last_change":"2023-10-16T11:20:25.748924+0000","last_active":"2023-10-16T11:20:37.814690+0000","last_peered":"2023-10-16T11:20:37.814690+0000","last_clean":"2023-10-16T11:20:37.814690+0000","last_became_active":"2023-10-16T11:20:25.748637+0000","last_became_peered":"2023-10-16T11:20:25.748637+0000","last_unstale":"2023-10-16T11:20:37.814690+0000","last_undegraded":"2023-10-16T11:20:37.814690+0000","last_fullsized":"2023-10-16T11:20:37.814690+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:40.894330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"236'8","reported_seq":23,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.818188+0000","last_change":"2023-10-16T11:20:24.740384+0000","last_active":"2023-10-16T11:20:37.818188+0000","last_peered":"2023-10-16T11:20:37.818188+0000","last_clean":"2023-10-16T11:20:37.818188+0000","last_became_active":"2023-10-16T11:20:24.739999+0000","last_became_peered":"2023-10-16T11:20:24.739999+0000","last_unstale":"2023-10-16T11:20:37.818188+0000","last_undegraded":"2023-10-16T11:20:37.818188+0000","last_fullsized":"2023-10-16T11:20:37.818188+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:31.286100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842575+0000","last_change":"2023-10-16T11:20:38.838947+0000","last_active":"2023-10-16T11:20:38.842575+0000","last_peered":"2023-10-16T11:20:38.842575+0000","last_clean":"2023-10-16T11:20:38.842575+0000","last_became_active":"2023-10-16T11:20:38.838463+0000","last_became_peered":"2023-10-16T11:20:38.838463+0000","last_unstale":"2023-10-16T11:20:38.842575+0000","last_undegraded":"2023-10-16T11:20:38.842575+0000","last_fullsized":"2023-10-16T11:20:38.842575+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:32.097784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812735+0000","last_change":"2023-10-16T11:20:25.749009+0000","last_active":"2023-10-16T11:20:37.812735+0000","last_peered":"2023-10-16T11:20:37.812735+0000","last_clean":"2023-10-16T11:20:37.812735+0000","last_became_active":"2023-10-16T11:20:25.748708+0000","last_became_peered":"2023-10-16T11:20:25.748708+0000","last_unstale":"2023-10-16T11:20:37.812735+0000","last_undegraded":"2023-10-16T11:20:37.812735+0000","last_fullsized":"2023-10-16T11:20:37.812735+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:55.578524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804549+0000","last_change":"2023-10-16T11:20:24.731614+0000","last_active":"2023-10-16T11:20:37.804549+0000","last_peered":"2023-10-16T11:20:37.804549+0000","last_clean":"2023-10-16T11:20:37.804549+0000","last_became_active":"2023-10-16T11:20:24.731329+0000","last_became_peered":"2023-10-16T11:20:24.731329+0000","last_unstale":"2023-10-16T11:20:37.804549+0000","last_undegraded":"2023-10-16T11:20:37.804549+0000","last_fullsized":"2023-10-16T11:20:37.804549+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:39.887304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844250+0000","last_change":"2023-10-16T11:20:38.838636+0000","last_active":"2023-10-16T11:20:38.844250+0000","last_peered":"2023-10-16T11:20:38.844250+0000","last_clean":"2023-10-16T11:20:38.844250+0000","last_became_active":"2023-10-16T11:20:38.837985+0000","last_became_peered":"2023-10-16T11:20:38.837985+0000","last_unstale":"2023-10-16T11:20:38.844250+0000","last_undegraded":"2023-10-16T11:20:38.844250+0000","last_fullsized":"2023-10-16T11:20:38.844250+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:06.743345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813408+0000","last_change":"2023-10-16T11:20:25.750514+0000","last_active":"2023-10-16T11:20:37.813408+0000","last_peered":"2023-10-16T11:20:37.813408+0000","last_clean":"2023-10-16T11:20:37.813408+0000","last_became_active":"2023-10-16T11:20:25.750226+0000","last_became_peered":"2023-10-16T11:20:25.750226+0000","last_unstale":"2023-10-16T11:20:37.813408+0000","last_undegraded":"2023-10-16T11:20:37.813408+0000","last_fullsized":"2023-10-16T11:20:37.813408+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:02.546128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.820950+0000","last_change":"2023-10-16T11:20:24.740869+0000","last_active":"2023-10-16T11:20:37.820950+0000","last_peered":"2023-10-16T11:20:37.820950+0000","last_clean":"2023-10-16T11:20:37.820950+0000","last_became_active":"2023-10-16T11:20:24.740527+0000","last_became_peered":"2023-10-16T11:20:24.740527+0000","last_unstale":"2023-10-16T11:20:37.820950+0000","last_undegraded":"2023-10-16T11:20:37.820950+0000","last_fullsized":"2023-10-16T11:20:37.820950+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:49.172459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838096+0000","last_change":"2023-10-16T11:20:38.831925+0000","last_active":"2023-10-16T11:20:38.838096+0000","last_peered":"2023-10-16T11:20:38.838096+0000","last_clean":"2023-10-16T11:20:38.838096+0000","last_became_active":"2023-10-16T11:20:38.831611+0000","last_became_peered":"2023-10-16T11:20:38.831611+0000","last_unstale":"2023-10-16T11:20:38.838096+0000","last_undegraded":"2023-10-16T11:20:38.838096+0000","last_fullsized":"2023-10-16T11:20:38.838096+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:16.818119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.817592+0000","last_change":"2023-10-16T11:20:25.749628+0000","last_active":"2023-10-16T11:20:37.817592+0000","last_peered":"2023-10-16T11:20:37.817592+0000","last_clean":"2023-10-16T11:20:37.817592+0000","last_became_active":"2023-10-16T11:20:25.749358+0000","last_became_peered":"2023-10-16T11:20:25.749358+0000","last_unstale":"2023-10-16T11:20:37.817592+0000","last_undegraded":"2023-10-16T11:20:37.817592+0000","last_fullsized":"2023-10-16T11:20:37.817592+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:39.916608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811438+0000","last_change":"2023-10-16T11:20:24.735496+0000","last_active":"2023-10-16T11:20:37.811438+0000","last_peered":"2023-10-16T11:20:37.811438+0000","last_clean":"2023-10-16T11:20:37.811438+0000","last_became_active":"2023-10-16T11:20:24.734988+0000","last_became_peered":"2023-10-16T11:20:24.734988+0000","last_unstale":"2023-10-16T11:20:37.811438+0000","last_undegraded":"2023-10-16T11:20:37.811438+0000","last_fullsized":"2023-10-16T11:20:37.811438+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:22.060729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.834542+0000","last_change":"2023-10-16T11:20:38.830069+0000","last_active":"2023-10-16T11:20:38.834542+0000","last_peered":"2023-10-16T11:20:38.834542+0000","last_clean":"2023-10-16T11:20:38.834542+0000","last_became_active":"2023-10-16T11:20:38.829778+0000","last_became_peered":"2023-10-16T11:20:38.829778+0000","last_unstale":"2023-10-16T11:20:38.834542+0000","last_undegraded":"2023-10-16T11:20:38.834542+0000","last_fullsized":"2023-10-16T11:20:38.834542+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:02.357965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812039+0000","last_change":"2023-10-16T11:20:25.746364+0000","last_active":"2023-10-16T11:20:37.812039+0000","last_peered":"2023-10-16T11:20:37.812039+0000","last_clean":"2023-10-16T11:20:37.812039+0000","last_became_active":"2023-10-16T11:20:25.746108+0000","last_became_peered":"2023-10-16T11:20:25.746108+0000","last_unstale":"2023-10-16T11:20:37.812039+0000","last_undegraded":"2023-10-16T11:20:37.812039+0000","last_fullsized":"2023-10-16T11:20:37.812039+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:03.593372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.804264+0000","last_change":"2023-10-16T11:20:24.727424+0000","last_active":"2023-10-16T11:20:37.804264+0000","last_peered":"2023-10-16T11:20:37.804264+0000","last_clean":"2023-10-16T11:20:37.804264+0000","last_became_active":"2023-10-16T11:20:24.727122+0000","last_became_peered":"2023-10-16T11:20:24.727122+0000","last_unstale":"2023-10-16T11:20:37.804264+0000","last_undegraded":"2023-10-16T11:20:37.804264+0000","last_fullsized":"2023-10-16T11:20:37.804264+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:38.786539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.832268+0000","last_change":"2023-10-16T11:20:38.830412+0000","last_active":"2023-10-16T11:20:38.832268+0000","last_peered":"2023-10-16T11:20:38.832268+0000","last_clean":"2023-10-16T11:20:38.832268+0000","last_became_active":"2023-10-16T11:20:38.830113+0000","last_became_peered":"2023-10-16T11:20:38.830113+0000","last_unstale":"2023-10-16T11:20:38.832268+0000","last_undegraded":"2023-10-16T11:20:38.832268+0000","last_fullsized":"2023-10-16T11:20:38.832268+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:51:26.326791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806635+0000","last_change":"2023-10-16T11:20:25.741687+0000","last_active":"2023-10-16T11:20:37.806635+0000","last_peered":"2023-10-16T11:20:37.806635+0000","last_clean":"2023-10-16T11:20:37.806635+0000","last_became_active":"2023-10-16T11:20:25.741022+0000","last_became_peered":"2023-10-16T11:20:25.741022+0000","last_unstale":"2023-10-16T11:20:37.806635+0000","last_undegraded":"2023-10-16T11:20:37.806635+0000","last_fullsized":"2023-10-16T11:20:37.806635+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:23.748016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.812127+0000","last_change":"2023-10-16T11:20:24.738500+0000","last_active":"2023-10-16T11:20:37.812127+0000","last_peered":"2023-10-16T11:20:37.812127+0000","last_clean":"2023-10-16T11:20:37.812127+0000","last_became_active":"2023-10-16T11:20:24.737980+0000","last_became_peered":"2023-10-16T11:20:24.737980+0000","last_unstale":"2023-10-16T11:20:37.812127+0000","last_undegraded":"2023-10-16T11:20:37.812127+0000","last_fullsized":"2023-10-16T11:20:37.812127+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:15.374424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838491+0000","last_change":"2023-10-16T11:20:38.833307+0000","last_active":"2023-10-16T11:20:38.838491+0000","last_peered":"2023-10-16T11:20:38.838491+0000","last_clean":"2023-10-16T11:20:38.838491+0000","last_became_active":"2023-10-16T11:20:38.832896+0000","last_became_peered":"2023-10-16T11:20:38.832896+0000","last_unstale":"2023-10-16T11:20:38.838491+0000","last_undegraded":"2023-10-16T11:20:38.838491+0000","last_fullsized":"2023-10-16T11:20:38.838491+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:02.978176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.818864+0000","last_change":"2023-10-16T11:20:25.752050+0000","last_active":"2023-10-16T11:20:37.818864+0000","last_peered":"2023-10-16T11:20:37.818864+0000","last_clean":"2023-10-16T11:20:37.818864+0000","last_became_active":"2023-10-16T11:20:25.751760+0000","last_became_peered":"2023-10-16T11:20:25.751760+0000","last_unstale":"2023-10-16T11:20:37.818864+0000","last_undegraded":"2023-10-16T11:20:37.818864+0000","last_fullsized":"2023-10-16T11:20:37.818864+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:40.238520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810113+0000","last_change":"2023-10-16T11:20:24.727961+0000","last_active":"2023-10-16T11:20:37.810113+0000","last_peered":"2023-10-16T11:20:37.810113+0000","last_clean":"2023-10-16T11:20:37.810113+0000","last_became_active":"2023-10-16T11:20:24.727467+0000","last_became_peered":"2023-10-16T11:20:24.727467+0000","last_unstale":"2023-10-16T11:20:37.810113+0000","last_undegraded":"2023-10-16T11:20:37.810113+0000","last_fullsized":"2023-10-16T11:20:37.810113+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:48:33.847435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.843397+0000","last_change":"2023-10-16T11:20:38.837408+0000","last_active":"2023-10-16T11:20:38.843397+0000","last_peered":"2023-10-16T11:20:38.843397+0000","last_clean":"2023-10-16T11:20:38.843397+0000","last_became_active":"2023-10-16T11:20:38.836556+0000","last_became_peered":"2023-10-16T11:20:38.836556+0000","last_unstale":"2023-10-16T11:20:38.843397+0000","last_undegraded":"2023-10-16T11:20:38.843397+0000","last_fullsized":"2023-10-16T11:20:38.843397+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:19.129128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811569+0000","last_change":"2023-10-16T11:20:25.743908+0000","last_active":"2023-10-16T11:20:37.811569+0000","last_peered":"2023-10-16T11:20:37.811569+0000","last_clean":"2023-10-16T11:20:37.811569+0000","last_became_active":"2023-10-16T11:20:25.743570+0000","last_became_peered":"2023-10-16T11:20:25.743570+0000","last_unstale":"2023-10-16T11:20:37.811569+0000","last_undegraded":"2023-10-16T11:20:37.811569+0000","last_fullsized":"2023-10-16T11:20:37.811569+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:53.089548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810602+0000","last_change":"2023-10-16T11:20:24.735291+0000","last_active":"2023-10-16T11:20:37.810602+0000","last_peered":"2023-10-16T11:20:37.810602+0000","last_clean":"2023-10-16T11:20:37.810602+0000","last_became_active":"2023-10-16T11:20:24.734998+0000","last_became_peered":"2023-10-16T11:20:24.734998+0000","last_unstale":"2023-10-16T11:20:37.810602+0000","last_undegraded":"2023-10-16T11:20:37.810602+0000","last_fullsized":"2023-10-16T11:20:37.810602+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:26.682154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837950+0000","last_change":"2023-10-16T11:20:38.829921+0000","last_active":"2023-10-16T11:20:38.837950+0000","last_peered":"2023-10-16T11:20:38.837950+0000","last_clean":"2023-10-16T11:20:38.837950+0000","last_became_active":"2023-10-16T11:20:38.829615+0000","last_became_peered":"2023-10-16T11:20:38.829615+0000","last_unstale":"2023-10-16T11:20:38.837950+0000","last_undegraded":"2023-10-16T11:20:38.837950+0000","last_fullsized":"2023-10-16T11:20:38.837950+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:33.255692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811289+0000","last_change":"2023-10-16T11:20:25.736861+0000","last_active":"2023-10-16T11:20:37.811289+0000","last_peered":"2023-10-16T11:20:37.811289+0000","last_clean":"2023-10-16T11:20:37.811289+0000","last_became_active":"2023-10-16T11:20:25.736595+0000","last_became_peered":"2023-10-16T11:20:25.736595+0000","last_unstale":"2023-10-16T11:20:37.811289+0000","last_undegraded":"2023-10-16T11:20:37.811289+0000","last_fullsized":"2023-10-16T11:20:37.811289+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:28.356199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811658+0000","last_change":"2023-10-16T11:20:24.729252+0000","last_active":"2023-10-16T11:20:37.811658+0000","last_peered":"2023-10-16T11:20:37.811658+0000","last_clean":"2023-10-16T11:20:37.811658+0000","last_became_active":"2023-10-16T11:20:24.729048+0000","last_became_peered":"2023-10-16T11:20:24.729048+0000","last_unstale":"2023-10-16T11:20:37.811658+0000","last_undegraded":"2023-10-16T11:20:37.811658+0000","last_fullsized":"2023-10-16T11:20:37.811658+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.587854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837698+0000","last_change":"2023-10-16T11:20:38.832394+0000","last_active":"2023-10-16T11:20:38.837698+0000","last_peered":"2023-10-16T11:20:38.837698+0000","last_clean":"2023-10-16T11:20:38.837698+0000","last_became_active":"2023-10-16T11:20:38.829399+0000","last_became_peered":"2023-10-16T11:20:38.829399+0000","last_unstale":"2023-10-16T11:20:38.837698+0000","last_undegraded":"2023-10-16T11:20:38.837698+0000","last_fullsized":"2023-10-16T11:20:38.837698+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:07.323719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816137+0000","last_change":"2023-10-16T11:20:25.748002+0000","last_active":"2023-10-16T11:20:37.816137+0000","last_peered":"2023-10-16T11:20:37.816137+0000","last_clean":"2023-10-16T11:20:37.816137+0000","last_became_active":"2023-10-16T11:20:25.747521+0000","last_became_peered":"2023-10-16T11:20:25.747521+0000","last_unstale":"2023-10-16T11:20:37.816137+0000","last_undegraded":"2023-10-16T11:20:37.816137+0000","last_fullsized":"2023-10-16T11:20:37.816137+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:09.510168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"236'2","reported_seq":17,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810940+0000","last_change":"2023-10-16T11:20:24.738296+0000","last_active":"2023-10-16T11:20:37.810940+0000","last_peered":"2023-10-16T11:20:37.810940+0000","last_clean":"2023-10-16T11:20:37.810940+0000","last_became_active":"2023-10-16T11:20:24.737992+0000","last_became_peered":"2023-10-16T11:20:24.737992+0000","last_unstale":"2023-10-16T11:20:37.810940+0000","last_undegraded":"2023-10-16T11:20:37.810940+0000","last_fullsized":"2023-10-16T11:20:37.810940+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:42.936914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.840885+0000","last_change":"2023-10-16T11:20:38.832751+0000","last_active":"2023-10-16T11:20:38.840885+0000","last_peered":"2023-10-16T11:20:38.840885+0000","last_clean":"2023-10-16T11:20:38.840885+0000","last_became_active":"2023-10-16T11:20:38.832449+0000","last_became_peered":"2023-10-16T11:20:38.832449+0000","last_unstale":"2023-10-16T11:20:38.840885+0000","last_undegraded":"2023-10-16T11:20:38.840885+0000","last_fullsized":"2023-10-16T11:20:38.840885+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:09:28.758123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803637+0000","last_change":"2023-10-16T11:20:25.736801+0000","last_active":"2023-10-16T11:20:37.803637+0000","last_peered":"2023-10-16T11:20:37.803637+0000","last_clean":"2023-10-16T11:20:37.803637+0000","last_became_active":"2023-10-16T11:20:25.736501+0000","last_became_peered":"2023-10-16T11:20:25.736501+0000","last_unstale":"2023-10-16T11:20:37.803637+0000","last_undegraded":"2023-10-16T11:20:37.803637+0000","last_fullsized":"2023-10-16T11:20:37.803637+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:41.787813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"237'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814415+0000","last_change":"2023-10-16T11:20:24.734520+0000","last_active":"2023-10-16T11:20:37.814415+0000","last_peered":"2023-10-16T11:20:37.814415+0000","last_clean":"2023-10-16T11:20:37.814415+0000","last_became_active":"2023-10-16T11:20:24.733902+0000","last_became_peered":"2023-10-16T11:20:24.733902+0000","last_unstale":"2023-10-16T11:20:37.814415+0000","last_undegraded":"2023-10-16T11:20:37.814415+0000","last_fullsized":"2023-10-16T11:20:37.814415+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:43.725198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839552+0000","last_change":"2023-10-16T11:20:38.830629+0000","last_active":"2023-10-16T11:20:38.839552+0000","last_peered":"2023-10-16T11:20:38.839552+0000","last_clean":"2023-10-16T11:20:38.839552+0000","last_became_active":"2023-10-16T11:20:38.830370+0000","last_became_peered":"2023-10-16T11:20:38.830370+0000","last_unstale":"2023-10-16T11:20:38.839552+0000","last_undegraded":"2023-10-16T11:20:38.839552+0000","last_fullsized":"2023-10-16T11:20:38.839552+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:36:19.694559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806708+0000","last_change":"2023-10-16T11:20:25.739360+0000","last_active":"2023-10-16T11:20:37.806708+0000","last_peered":"2023-10-16T11:20:37.806708+0000","last_clean":"2023-10-16T11:20:37.806708+0000","last_became_active":"2023-10-16T11:20:25.739110+0000","last_became_peered":"2023-10-16T11:20:25.739110+0000","last_unstale":"2023-10-16T11:20:37.806708+0000","last_undegraded":"2023-10-16T11:20:37.806708+0000","last_fullsized":"2023-10-16T11:20:37.806708+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:59:40.578713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814270+0000","last_change":"2023-10-16T11:20:24.736139+0000","last_active":"2023-10-16T11:20:37.814270+0000","last_peered":"2023-10-16T11:20:37.814270+0000","last_clean":"2023-10-16T11:20:37.814270+0000","last_became_active":"2023-10-16T11:20:24.735807+0000","last_became_peered":"2023-10-16T11:20:24.735807+0000","last_unstale":"2023-10-16T11:20:37.814270+0000","last_undegraded":"2023-10-16T11:20:37.814270+0000","last_fullsized":"2023-10-16T11:20:37.814270+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.831725+0000","last_change":"2023-10-16T11:20:38.830621+0000","last_active":"2023-10-16T11:20:38.831725+0000","last_peered":"2023-10-16T11:20:38.831725+0000","last_clean":"2023-10-16T11:20:38.831725+0000","last_became_active":"2023-10-16T11:20:38.830345+0000","last_became_peered":"2023-10-16T11:20:38.830345+0000","last_unstale":"2023-10-16T11:20:38.831725+0000","last_undegraded":"2023-10-16T11:20:38.831725+0000","last_fullsized":"2023-10-16T11:20:38.831725+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:06.544476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813206+0000","last_change":"2023-10-16T11:20:25.748753+0000","last_active":"2023-10-16T11:20:37.813206+0000","last_peered":"2023-10-16T11:20:37.813206+0000","last_clean":"2023-10-16T11:20:37.813206+0000","last_became_active":"2023-10-16T11:20:25.748480+0000","last_became_peered":"2023-10-16T11:20:25.748480+0000","last_unstale":"2023-10-16T11:20:37.813206+0000","last_undegraded":"2023-10-16T11:20:37.813206+0000","last_fullsized":"2023-10-16T11:20:37.813206+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:51.963113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814359+0000","last_change":"2023-10-16T11:20:24.729644+0000","last_active":"2023-10-16T11:20:37.814359+0000","last_peered":"2023-10-16T11:20:37.814359+0000","last_clean":"2023-10-16T11:20:37.814359+0000","last_became_active":"2023-10-16T11:20:24.729487+0000","last_became_peered":"2023-10-16T11:20:24.729487+0000","last_unstale":"2023-10-16T11:20:37.814359+0000","last_undegraded":"2023-10-16T11:20:37.814359+0000","last_fullsized":"2023-10-16T11:20:37.814359+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:45.766549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.842068+0000","last_change":"2023-10-16T11:20:38.834447+0000","last_active":"2023-10-16T11:20:38.842068+0000","last_peered":"2023-10-16T11:20:38.842068+0000","last_clean":"2023-10-16T11:20:38.842068+0000","last_became_active":"2023-10-16T11:20:38.833694+0000","last_became_peered":"2023-10-16T11:20:38.833694+0000","last_unstale":"2023-10-16T11:20:38.842068+0000","last_undegraded":"2023-10-16T11:20:38.842068+0000","last_fullsized":"2023-10-16T11:20:38.842068+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:58.540418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.814091+0000","last_change":"2023-10-16T11:20:25.740829+0000","last_active":"2023-10-16T11:20:37.814091+0000","last_peered":"2023-10-16T11:20:37.814091+0000","last_clean":"2023-10-16T11:20:37.814091+0000","last_became_active":"2023-10-16T11:20:25.740552+0000","last_became_peered":"2023-10-16T11:20:25.740552+0000","last_unstale":"2023-10-16T11:20:37.814091+0000","last_undegraded":"2023-10-16T11:20:37.814091+0000","last_fullsized":"2023-10-16T11:20:37.814091+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:51.179585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810287+0000","last_change":"2023-10-16T11:20:24.727846+0000","last_active":"2023-10-16T11:20:37.810287+0000","last_peered":"2023-10-16T11:20:37.810287+0000","last_clean":"2023-10-16T11:20:37.810287+0000","last_became_active":"2023-10-16T11:20:24.727551+0000","last_became_peered":"2023-10-16T11:20:24.727551+0000","last_unstale":"2023-10-16T11:20:37.810287+0000","last_undegraded":"2023-10-16T11:20:37.810287+0000","last_fullsized":"2023-10-16T11:20:37.810287+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:10.257213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.837069+0000","last_change":"2023-10-16T11:20:38.834306+0000","last_active":"2023-10-16T11:20:38.837069+0000","last_peered":"2023-10-16T11:20:38.837069+0000","last_clean":"2023-10-16T11:20:38.837069+0000","last_became_active":"2023-10-16T11:20:38.834024+0000","last_became_peered":"2023-10-16T11:20:38.834024+0000","last_unstale":"2023-10-16T11:20:38.837069+0000","last_undegraded":"2023-10-16T11:20:38.837069+0000","last_fullsized":"2023-10-16T11:20:38.837069+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:03:46.033448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.805657+0000","last_change":"2023-10-16T11:20:25.748408+0000","last_active":"2023-10-16T11:20:37.805657+0000","last_peered":"2023-10-16T11:20:37.805657+0000","last_clean":"2023-10-16T11:20:37.805657+0000","last_became_active":"2023-10-16T11:20:25.747574+0000","last_became_peered":"2023-10-16T11:20:25.747574+0000","last_unstale":"2023-10-16T11:20:37.805657+0000","last_undegraded":"2023-10-16T11:20:37.805657+0000","last_fullsized":"2023-10-16T11:20:37.805657+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:57.537856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.813161+0000","last_change":"2023-10-16T11:20:24.728924+0000","last_active":"2023-10-16T11:20:37.813161+0000","last_peered":"2023-10-16T11:20:37.813161+0000","last_clean":"2023-10-16T11:20:37.813161+0000","last_became_active":"2023-10-16T11:20:24.728614+0000","last_became_peered":"2023-10-16T11:20:24.728614+0000","last_unstale":"2023-10-16T11:20:37.813161+0000","last_undegraded":"2023-10-16T11:20:37.813161+0000","last_fullsized":"2023-10-16T11:20:37.813161+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:23.737157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.839653+0000","last_change":"2023-10-16T11:20:38.835181+0000","last_active":"2023-10-16T11:20:38.839653+0000","last_peered":"2023-10-16T11:20:38.839653+0000","last_clean":"2023-10-16T11:20:38.839653+0000","last_became_active":"2023-10-16T11:20:38.834922+0000","last_became_peered":"2023-10-16T11:20:38.834922+0000","last_unstale":"2023-10-16T11:20:38.839653+0000","last_undegraded":"2023-10-16T11:20:38.839653+0000","last_fullsized":"2023-10-16T11:20:38.839653+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:05.487994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.821141+0000","last_change":"2023-10-16T11:20:25.751879+0000","last_active":"2023-10-16T11:20:37.821141+0000","last_peered":"2023-10-16T11:20:37.821141+0000","last_clean":"2023-10-16T11:20:37.821141+0000","last_became_active":"2023-10-16T11:20:25.750440+0000","last_became_peered":"2023-10-16T11:20:25.750440+0000","last_unstale":"2023-10-16T11:20:37.821141+0000","last_undegraded":"2023-10-16T11:20:37.821141+0000","last_fullsized":"2023-10-16T11:20:37.821141+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:57.379246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.806639+0000","last_change":"2023-10-16T11:20:24.732437+0000","last_active":"2023-10-16T11:20:37.806639+0000","last_peered":"2023-10-16T11:20:37.806639+0000","last_clean":"2023-10-16T11:20:37.806639+0000","last_became_active":"2023-10-16T11:20:24.732111+0000","last_became_peered":"2023-10-16T11:20:24.732111+0000","last_unstale":"2023-10-16T11:20:37.806639+0000","last_undegraded":"2023-10-16T11:20:37.806639+0000","last_fullsized":"2023-10-16T11:20:37.806639+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:26.846516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.838494+0000","last_change":"2023-10-16T11:20:38.835643+0000","last_active":"2023-10-16T11:20:38.838494+0000","last_peered":"2023-10-16T11:20:38.838494+0000","last_clean":"2023-10-16T11:20:38.838494+0000","last_became_active":"2023-10-16T11:20:38.835326+0000","last_became_peered":"2023-10-16T11:20:38.835326+0000","last_unstale":"2023-10-16T11:20:38.838494+0000","last_undegraded":"2023-10-16T11:20:38.838494+0000","last_fullsized":"2023-10-16T11:20:38.838494+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:56.021692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.811359+0000","last_change":"2023-10-16T11:20:25.750621+0000","last_active":"2023-10-16T11:20:37.811359+0000","last_peered":"2023-10-16T11:20:37.811359+0000","last_clean":"2023-10-16T11:20:37.811359+0000","last_became_active":"2023-10-16T11:20:25.750352+0000","last_became_peered":"2023-10-16T11:20:25.750352+0000","last_unstale":"2023-10-16T11:20:37.811359+0000","last_undegraded":"2023-10-16T11:20:37.811359+0000","last_fullsized":"2023-10-16T11:20:37.811359+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:57.295597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.810371+0000","last_change":"2023-10-16T11:20:24.733888+0000","last_active":"2023-10-16T11:20:37.810371+0000","last_peered":"2023-10-16T11:20:37.810371+0000","last_clean":"2023-10-16T11:20:37.810371+0000","last_became_active":"2023-10-16T11:20:24.733588+0000","last_became_peered":"2023-10-16T11:20:24.733588+0000","last_unstale":"2023-10-16T11:20:37.810371+0000","last_undegraded":"2023-10-16T11:20:37.810371+0000","last_fullsized":"2023-10-16T11:20:37.810371+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:06.768468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-16T11:20:38.844237+0000","last_change":"2023-10-16T11:20:38.838396+0000","last_active":"2023-10-16T11:20:38.844237+0000","last_peered":"2023-10-16T11:20:38.844237+0000","last_clean":"2023-10-16T11:20:38.844237+0000","last_became_active":"2023-10-16T11:20:38.837632+0000","last_became_peered":"2023-10-16T11:20:38.837632+0000","last_unstale":"2023-10-16T11:20:38.844237+0000","last_undegraded":"2023-10-16T11:20:38.844237+0000","last_fullsized":"2023-10-16T11:20:38.844237+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:37.798883+0000","last_clean_scrub_stamp":"2023-10-16T11:20:37.798883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:34.965136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":14,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.805321+0000","last_change":"2023-10-16T11:20:25.748051+0000","last_active":"2023-10-16T11:20:37.805321+0000","last_peered":"2023-10-16T11:20:37.805321+0000","last_clean":"2023-10-16T11:20:37.805321+0000","last_became_active":"2023-10-16T11:20:25.747647+0000","last_became_peered":"2023-10-16T11:20:25.747647+0000","last_unstale":"2023-10-16T11:20:37.805321+0000","last_undegraded":"2023-10-16T11:20:37.805321+0000","last_fullsized":"2023-10-16T11:20:37.805321+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:24.710641+0000","last_clean_scrub_stamp":"2023-10-16T11:20:24.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:06.072095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"236'1","reported_seq":16,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.816886+0000","last_change":"2023-10-16T11:20:24.736168+0000","last_active":"2023-10-16T11:20:37.816886+0000","last_peered":"2023-10-16T11:20:37.816886+0000","last_clean":"2023-10-16T11:20:37.816886+0000","last_became_active":"2023-10-16T11:20:24.735886+0000","last_became_peered":"2023-10-16T11:20:24.735886+0000","last_unstale":"2023-10-16T11:20:37.816886+0000","last_undegraded":"2023-10-16T11:20:37.816886+0000","last_fullsized":"2023-10-16T11:20:37.816886+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:20:23.705123+0000","last_clean_scrub_stamp":"2023-10-16T11:20:23.705123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:22.742499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"223'370","reported_seq":1309,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-16T11:20:37.803645+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:20:37.803645+0000","last_peered":"2023-10-16T11:20:37.803645+0000","last_clean":"2023-10-16T11:20:37.803645+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:20:37.803645+0000","last_undegraded":"2023-10-16T11:20:37.803645+0000","last_fullsized":"2023-10-16T11:20:37.803645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52792,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319048,"statfs":{"total":96636764160,"available":96582705152,"internally_reserved":0,"allocated":3641344,"data_stored":2224748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64600000000000002,"15min":0.60699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68999999999999995,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.66400000000000003,"15min":0.628},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.68799999999999994,"15min":0.66200000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.622,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.70099999999999996,"15min":0.67400000000000004},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.71199999999999997,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.436}]},{"osd":2,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.71399999999999997,"15min":0.67300000000000004},"min":{"1min":0.48099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78900000000000003,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.73299999999999998,"15min":0.69099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":9.516,"15min":9.516},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.68300000000000005,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.68799999999999994,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.75900000000000001,"15min":0.71199999999999997},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.54100000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32300000000000001},"max":{"1min":0.747,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.63200000000000001,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.33500000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65300000000000002,"15min":0.57799999999999996},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":0.98399999999999999,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69699999999999995,"15min":0.63},"min":{"1min":0.41499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79500000000000004,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.71599999999999997,"15min":0.64600000000000002},"min":{"1min":0.41599999999999998,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.84199999999999997,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.69699999999999995,"15min":0.64200000000000002},"min":{"1min":0.442,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76600000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.54500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56368,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315472,"statfs":{"total":96636764160,"available":96579043328,"internally_reserved":0,"allocated":3043328,"data_stored":1633798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75800000000000001,"5min":0.80700000000000005,"15min":0.86299999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.59199999999999997,"15min":0.61399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69399999999999995,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58099999999999996,"15min":0.60599999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73499999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":2.387,"15min":2.387},"last":0.80000000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.57199999999999995,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.314},"max":{"1min":0.58699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":0.62},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.871}]},{"osd":3,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.622,"15min":0.621},"min":{"1min":0.36499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.65300000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.36399999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71599999999999997,"5min":1.02,"15min":1.02},"last":0.81000000000000005}]},{"osd":4,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.32900000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.55500000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.77800000000000002,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.45400000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76200000000000001,"5min":3.335,"15min":3.335},"last":0.81999999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":2.524,"15min":2.524},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.752,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.85599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576003,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48020,"kb_used_data":2948,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323820,"statfs":{"total":96636764160,"available":96587591680,"internally_reserved":0,"allocated":3018752,"data_stored":1625207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.622,"15min":0.66700000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.436,"15min":0.45200000000000001},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.52500000000000002,"5min":0.63900000000000001,"15min":0.71699999999999997},"last":0.38200000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55400000000000005,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.434,"15min":0.44900000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.64000000000000001,"15min":0.70099999999999996},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.442},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":0.623,"15min":0.70499999999999996},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.48099999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.64700000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.69199999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65400000000000003,"15min":0.67200000000000004},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":0.89500000000000002,"15min":1.0149999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90000000000000002,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.60499999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.90300000000000002,"15min":1.0620000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.016},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61108,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310732,"statfs":{"total":96636764160,"available":96574189568,"internally_reserved":0,"allocated":3641344,"data_stored":2226949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.45000000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.26700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52800000000000002,"5min":0.66100000000000003,"15min":0.77200000000000002},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":1.272,"15min":1.272},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40899999999999997,"15min":0.433},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.313},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54200000000000004,"5min":1.351,"15min":1.351},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83099999999999996,"5min":1.391,"15min":1.391},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76000000000000001,"5min":0.874,"15min":1.089},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.624,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.95599999999999996,"15min":1.216},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.47199999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.1180000000000001},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.67200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80300000000000005,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80500000000000005,"5min":0.89600000000000002,"15min":1.0780000000000001},"last":0.70599999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48064,"kb_used_data":2988,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323776,"statfs":{"total":96636764160,"available":96587546624,"internally_reserved":0,"allocated":3059712,"data_stored":1636520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.371,"15min":0.39500000000000002},"min":{"1min":0.253,"5min":0.188,"15min":0.188},"max":{"1min":0.51800000000000002,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.42799999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.193,"15min":0.193},"max":{"1min":0.53600000000000003,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.313}]},{"osd":1,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.44400000000000001,"15min":0.46500000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59499999999999997,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39000000000000001,"15min":0.41599999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.45500000000000002,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.29199999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.442},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56699999999999995,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48799999999999999,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.95399999999999996,"15min":1.097},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64700000000000002,"15min":0.67300000000000004},"min":{"1min":0.438,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77100000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63700000000000001,"15min":0.66700000000000004},"min":{"1min":0.45500000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82499999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.69199999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81899999999999995,"5min":1.0169999999999999,"15min":1.0389999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64500000000000002,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76700000000000002,"5min":0.97799999999999998,"15min":1.022},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67700000000000005,"15min":0.70299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84999999999999998,"5min":0.98999999999999999,"15min":1.004},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.69199999999999995,"15min":0.70599999999999996},"min":{"1min":0.48599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81000000000000005,"5min":0.99299999999999999,"15min":1.071},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.67100000000000004,"15min":0.69799999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81200000000000006,"5min":0.97799999999999998,"15min":1.077},"last":0.505}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52212,"kb_used_data":2988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319628,"statfs":{"total":96636764160,"available":96583299072,"internally_reserved":0,"allocated":3059712,"data_stored":1637239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.376,"15min":0.39500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38300000000000001,"15min":0.40699999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":1.0029999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.22600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42499999999999999,"15min":0.441},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.443}]},{"osd":2,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46400000000000002,"15min":0.48299999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.50800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.024},"last":0.496}]},{"osd":4,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.68000000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.026},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.77600000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.41799999999999998},"max":{"1min":5.8399999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69899999999999995,"15min":0.70499999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.4409999999999998,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.69899999999999995,"15min":0.71199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.66600000000000004,"15min":0.69999999999999996},"min":{"1min":0.45300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77800000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.71299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52212,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319628,"statfs":{"total":96636764160,"available":96583299072,"internally_reserved":0,"allocated":3047424,"data_stored":1635334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56999999999999995,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67700000000000005,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.503},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57599999999999996,"15min":0.61799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.61499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74099999999999999,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.61399999999999999},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.95499999999999996},"last":0.626}]},{"osd":2,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.622},"min":{"1min":0.40400000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60799999999999998,"15min":0.63700000000000001},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.61599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67900000000000005,"5min":0.83799999999999997,"15min":0.995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48199999999999998,"15min":0.5},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71499999999999997,"5min":0.73599999999999999,"15min":0.86199999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76300000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.38300000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.76200000000000001,"5min":0.878,"15min":1.0609999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.745,"5min":0.83099999999999996,"15min":1.115},"last":0.44}]},{"osd":7,"last update":"Mon Oct 16 11:20:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.83999999999999997,"5min":0.84499999999999997,"15min":0.93200000000000005},"last":0.58099999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66088,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305752,"statfs":{"total":96636764160,"available":96569090048,"internally_reserved":0,"allocated":3035136,"data_stored":1633722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61199999999999999,"15min":0.63200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.28999999999999998},"max":{"1min":0.78000000000000003,"5min":0.82799999999999996,"15min":0.89300000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.62,"15min":0.65000000000000002},"min":{"1min":0.39700000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72999999999999998,"5min":0.871,"15min":0.88200000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.57699999999999996,"15min":0.59499999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.752,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.56399999999999995,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.59099999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73399999999999999,"5min":0.80300000000000005,"15min":0.86499999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58399999999999996,"15min":0.61699999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.752,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.61099999999999999},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57999999999999996,"15min":0.60399999999999998},"min":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.39700000000000002},"max":{"1min":0.70299999999999996,"5min":0.82899999999999996,"15min":0.90500000000000003},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80900000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.82499999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84499999999999997,"5min":0.878,"15min":0.878},"last":0.496}]},{"osd":7,"last update":"Mon Oct 16 11:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54800000000000004,"15min":0.57299999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73599999999999999,"5min":0.871,"15min":0.874},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78300000000000003,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:20:43.356 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:20:43.356 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:43.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.505+0000 7faf9e72f700 1 Processor -- start 2023-10-16T11:20:43.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.505+0000 7faf9e72f700 1 -- start start 2023-10-16T11:20:43.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.505+0000 7faf9e72f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 0x7faf980ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.505+0000 7faf9e72f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faf9815e150 0x7faf9815e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.505+0000 7faf9e72f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9815eb40 con 0x7faf9815a5d0 2023-10-16T11:20:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.505+0000 7faf9e72f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9815edd0 con 0x7faf9815e150 2023-10-16T11:20:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.505+0000 7faf9e72f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf980ce080 con 0x7faf980b24c0 2023-10-16T11:20:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf97fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7faf9815a5d0 0x7faf980b1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40878/0 (socket says 172.21.15.158:40878) 2023-10-16T11:20:43.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf97fff700 1 -- 172.21.15.158:0/2264172421 learned_addr learned my addr 172.21.15.158:0/2264172421 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:43.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf9cccc700 1 --2- 172.21.15.158:0/2264172421 >> v2:172.21.15.172:3300/0 conn(0x7faf9815e150 0x7faf9815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf977fe700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 0x7faf980ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073948604 0 0) 0x7faf9815eb40 con 0x7faf9815a5d0 2023-10-16T11:20:43.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf800035b0 con 0x7faf9815a5d0 2023-10-16T11:20:43.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3162307558 0 0) 0x7faf800035b0 con 0x7faf9815a5d0 2023-10-16T11:20:43.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 msgr2=0x7faf980ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:43.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 0x7faf980ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 >> v2:172.21.15.172:3300/0 conn(0x7faf9815e150 msgr2=0x7faf9815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:43.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 --2- 172.21.15.158:0/2264172421 >> v2:172.21.15.172:3300/0 conn(0x7faf9815e150 0x7faf9815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980ce360 con 0x7faf9815a5d0 2023-10-16T11:20:43.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7faf88001670 con 0x7faf9815a5d0 2023-10-16T11:20:43.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 >> v1:172.21.15.158:6789/0 conn(0x7faf9815a5d0 legacy=0x7faf980b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 0x7faf80003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80003f80 0x7faf80004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf80004d90 0x7faf80005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9815edd0 con 0x7faf80004d90 2023-10-16T11:20:43.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf980ce080 con 0x7faf80003f80 2023-10-16T11:20:43.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8000ba10 con 0x7faf980b24c0 2023-10-16T11:20:43.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf977fe700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80003f80 0x7faf80004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf97fff700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 0x7faf80003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.506+0000 7faf9cccc700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf80004d90 0x7faf80005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9cccc700 1 -- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 msgr2=0x7faf80003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9cccc700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf980b24c0 0x7faf80003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9cccc700 1 -- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80003f80 msgr2=0x7faf80004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9cccc700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80003f80 0x7faf80004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9cccc700 1 -- 172.21.15.158:0/2264172421 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf800035b0 con 0x7faf80004d90 2023-10-16T11:20:43.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf977fe700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80003f80 0x7faf80004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:43.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9cccc700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf80004d90 0x7faf80005210 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7faf90009fc0 tx=0x7faf80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d269495e32fdb77d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:43.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000ebd0 con 0x7faf80004d90 2023-10-16T11:20:43.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9000f040 con 0x7faf80004d90 2023-10-16T11:20:43.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf96ffd700 1 -- 172.21.15.158:0/2264172421 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90014480 con 0x7faf80004d90 2023-10-16T11:20:43.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9e72f700 1 -- 172.21.15.158:0/2264172421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf80004d90 msgr2=0x7faf80005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9e72f700 1 --2- 172.21.15.158:0/2264172421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf80004d90 0x7faf80005210 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7faf90009fc0 tx=0x7faf80005880 comp rx=0 tx=0).stop 2023-10-16T11:20:43.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9cccc700 1 -- 172.21.15.158:0/2264172421 reap_dead start 2023-10-16T11:20:43.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9e72f700 1 -- 172.21.15.158:0/2264172421 shutdown_connections 2023-10-16T11:20:43.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9e72f700 1 -- 172.21.15.158:0/2264172421 >> 172.21.15.158:0/2264172421 conn(0x7faf980ccdb0 msgr2=0x7faf980cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:43.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9e72f700 1 -- 172.21.15.158:0/2264172421 shutdown_connections 2023-10-16T11:20:43.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.507+0000 7faf9e72f700 1 -- 172.21.15.158:0/2264172421 wait complete. 2023-10-16T11:20:43.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.509+0000 7faf9e72f700 1 Processor -- start 2023-10-16T11:20:43.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.509+0000 7faf9e72f700 1 -- start start 2023-10-16T11:20:43.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80004d90 0x7faf9815d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf98153e40 0x7faf980b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf981553e0 0x7faf980c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf980c76a0 con 0x7faf981553e0 2023-10-16T11:20:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9815d690 con 0x7faf80004d90 2023-10-16T11:20:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9815d970 con 0x7faf98153e40 2023-10-16T11:20:43.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf981553e0 0x7faf980c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf981553e0 0x7faf980c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40718/0 (socket says 172.21.15.158:40718) 2023-10-16T11:20:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 -- 172.21.15.158:0/3023651513 learned_addr learned my addr 172.21.15.158:0/3023651513 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 -- 172.21.15.158:0/3023651513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf98153e40 msgr2=0x7faf980b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf98153e40 0x7faf980b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 -- 172.21.15.158:0/3023651513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80004d90 msgr2=0x7faf9815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80004d90 0x7faf9815d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 -- 172.21.15.158:0/3023651513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9815dc50 con 0x7faf981553e0 2023-10-16T11:20:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9cccc700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf981553e0 0x7faf980c7ac0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7faf90017040 tx=0x7faf90009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf94ff9700 1 -- 172.21.15.158:0/3023651513 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90007000 con 0x7faf981553e0 2023-10-16T11:20:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf94ff9700 1 -- 172.21.15.158:0/3023651513 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf90014d60 con 0x7faf981553e0 2023-10-16T11:20:43.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf980c97c0 con 0x7faf981553e0 2023-10-16T11:20:43.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf94ff9700 1 -- 172.21.15.158:0/3023651513 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf900276b0 con 0x7faf981553e0 2023-10-16T11:20:43.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.510+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf980b1580 con 0x7faf981553e0 2023-10-16T11:20:43.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.511+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf980c73e0 con 0x7faf981553e0 2023-10-16T11:20:43.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.512+0000 7faf94ff9700 1 -- 172.21.15.158:0/3023651513 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faf90024070 con 0x7faf981553e0 2023-10-16T11:20:43.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.512+0000 7faf94ff9700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7faf8406b140 0x7faf8406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.512+0000 7faf94ff9700 1 -- 172.21.15.158:0/3023651513 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7faf9008ba80 con 0x7faf981553e0 2023-10-16T11:20:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.512+0000 7faf97fff700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7faf8406b140 0x7faf8406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.512+0000 7faf97fff700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7faf8406b140 0x7faf8406d600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf88001cc0 tx=0x7faf88001b90 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:43.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.516+0000 7faf94ff9700 1 -- 172.21.15.158:0/3023651513 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf90056300 con 0x7faf981553e0 2023-10-16T11:20:43.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.755+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf980b1580 con 0x7faf981553e0 2023-10-16T11:20:43.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.755+0000 7faf94ff9700 1 -- 172.21.15.158:0/3023651513 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6160 (secure 0 0 0) 0x7faf980b1580 con 0x7faf981553e0 2023-10-16T11:20:43.757 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:43.757 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:20:43.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.757+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7faf8406b140 msgr2=0x7faf8406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.757+0000 7faf9e72f700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7faf8406b140 0x7faf8406d600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf88001cc0 tx=0x7faf88001b90 comp rx=0 tx=0).stop 2023-10-16T11:20:43.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.757+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf981553e0 msgr2=0x7faf980c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:43.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.757+0000 7faf9e72f700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf981553e0 0x7faf980c7ac0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7faf90017040 tx=0x7faf90009750 comp rx=0 tx=0).stop 2023-10-16T11:20:43.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.757+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 shutdown_connections 2023-10-16T11:20:43.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.758+0000 7faf9e72f700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7faf8406b140 0x7faf8406d600 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.758+0000 7faf9e72f700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf981553e0 0x7faf980c7ac0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.758+0000 7faf9e72f700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf80004d90 0x7faf9815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.758+0000 7faf9e72f700 1 --2- 172.21.15.158:0/3023651513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf98153e40 0x7faf980b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:43.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.758+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 >> 172.21.15.158:0/3023651513 conn(0x7faf980ccdb0 msgr2=0x7faf9806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:43.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.758+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 shutdown_connections 2023-10-16T11:20:43.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:43.758+0000 7faf9e72f700 1 -- 172.21.15.158:0/3023651513 wait complete. 2023-10-16T11:20:43.762 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 134 2023-10-16T11:20:44.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 134, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:20:38.210020+0000', 'modified': '2023-10-16T11:20:41.237425+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12432}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12432': {'gid': 12432, 'name': 'b', 'rank': 0, 'incarnation': 131, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.172:6835/1061053278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 1061053278}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 1061053278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:20:44.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:20:44.175 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T11:20:44.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.297+0000 7fe433ef0700 1 Processor -- start 2023-10-16T11:20:44.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.297+0000 7fe433ef0700 1 -- start start 2023-10-16T11:20:44.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe433ef0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe42c15b6e0 0x7fe42c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe433ef0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe42c06afe0 0x7fe42c071f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe433ef0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c071af0 con 0x7fe42c15f4d0 2023-10-16T11:20:44.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe433ef0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42c0afb60 con 0x7fe42c15b6e0 2023-10-16T11:20:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe433ef0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42c15fa40 con 0x7fe42c06afe0 2023-10-16T11:20:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe431c8c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe42c15b6e0 0x7fe42c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe43148b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe42c15f4d0 0x7fe42c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40898/0 (socket says 172.21.15.158:40898) 2023-10-16T11:20:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe43148b700 1 -- 172.21.15.158:0/2467841359 learned_addr learned my addr 172.21.15.158:0/2467841359 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe43248d700 1 --2- 172.21.15.158:0/2467841359 >> v2:172.21.15.172:3300/0 conn(0x7fe42c06afe0 0x7fe42c071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe431c8c700 1 -- 172.21.15.158:0/2467841359 >> v2:172.21.15.172:3300/0 conn(0x7fe42c06afe0 msgr2=0x7fe42c071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe431c8c700 1 --2- 172.21.15.158:0/2467841359 >> v2:172.21.15.172:3300/0 conn(0x7fe42c06afe0 0x7fe42c071f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe431c8c700 1 -- 172.21.15.158:0/2467841359 >> v1:172.21.15.158:6789/0 conn(0x7fe42c15f4d0 legacy=0x7fe42c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.298+0000 7fe431c8c700 1 -- 172.21.15.158:0/2467841359 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c15fd20 con 0x7fe42c15b6e0 2023-10-16T11:20:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe431c8c700 1 --2- 172.21.15.158:0/2467841359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c06dcd0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fe42c06b990 tx=0x7fe41c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183f2154d04e7c18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe430c8a700 1 -- 172.21.15.158:0/2467841359 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c011830 con 0x7fe42c15b6e0 2023-10-16T11:20:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe430c8a700 1 -- 172.21.15.158:0/2467841359 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c011990 con 0x7fe42c15b6e0 2023-10-16T11:20:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe430c8a700 1 -- 172.21.15.158:0/2467841359 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c011d20 con 0x7fe42c15b6e0 2023-10-16T11:20:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 -- 172.21.15.158:0/2467841359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 msgr2=0x7fe42c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 --2- 172.21.15.158:0/2467841359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c06dcd0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fe42c06b990 tx=0x7fe41c009b20 comp rx=0 tx=0).stop 2023-10-16T11:20:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 -- 172.21.15.158:0/2467841359 shutdown_connections 2023-10-16T11:20:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 --2- 172.21.15.158:0/2467841359 >> v2:172.21.15.172:3300/0 conn(0x7fe42c06afe0 0x7fe42c071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 --2- 172.21.15.158:0/2467841359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c06dcd0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 -- 172.21.15.158:0/2467841359 >> 172.21.15.158:0/2467841359 conn(0x7fe42c06b9e0 msgr2=0x7fe42c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 -- 172.21.15.158:0/2467841359 shutdown_connections 2023-10-16T11:20:44.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.299+0000 7fe433ef0700 1 -- 172.21.15.158:0/2467841359 wait complete. 2023-10-16T11:20:44.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 Processor -- start 2023-10-16T11:20:44.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 -- start start 2023-10-16T11:20:44.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42c06afe0 0x7fe42c0c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c1f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42c15f4d0 0x7fe42c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c0c6dd0 con 0x7fe42c15b6e0 2023-10-16T11:20:44.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c0c8070 con 0x7fe42c06afe0 2023-10-16T11:20:44.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe433ef0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c0c6150 con 0x7fe42c15f4d0 2023-10-16T11:20:44.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c1f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c1f3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40728/0 (socket says 172.21.15.158:40728) 2023-10-16T11:20:44.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.301+0000 7fe43148b700 1 -- 172.21.15.158:0/367224807 learned_addr learned my addr 172.21.15.158:0/367224807 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe43148b700 1 -- 172.21.15.158:0/367224807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42c15f4d0 msgr2=0x7fe42c0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe43148b700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42c15f4d0 0x7fe42c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe43148b700 1 -- 172.21.15.158:0/367224807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42c06afe0 msgr2=0x7fe42c0c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe43148b700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42c06afe0 0x7fe42c0c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe43148b700 1 -- 172.21.15.158:0/367224807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0094a0 con 0x7fe42c15b6e0 2023-10-16T11:20:44.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe43148b700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c1f3a20 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7fe42800c480 tx=0x7fe42800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe4227fc700 1 -- 172.21.15.158:0/367224807 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428005d70 con 0x7fe42c15b6e0 2023-10-16T11:20:44.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe4227fc700 1 -- 172.21.15.158:0/367224807 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4280070d0 con 0x7fe42c15b6e0 2023-10-16T11:20:44.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42c0c6430 con 0x7fe42c15b6e0 2023-10-16T11:20:44.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe4227fc700 1 -- 172.21.15.158:0/367224807 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428007440 con 0x7fe42c15b6e0 2023-10-16T11:20:44.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.302+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42c0c69f0 con 0x7fe42c15b6e0 2023-10-16T11:20:44.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.303+0000 7fe4227fc700 1 -- 172.21.15.158:0/367224807 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe4280075a0 con 0x7fe42c15b6e0 2023-10-16T11:20:44.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.304+0000 7fe4227fc700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe41006b140 0x7fe41006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.304+0000 7fe4227fc700 1 -- 172.21.15.158:0/367224807 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fe428089330 con 0x7fe42c15b6e0 2023-10-16T11:20:44.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.304+0000 7fe431c8c700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe41006b140 0x7fe41006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.304+0000 7fe431c8c700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe41006b140 0x7fe41006d600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0090b0 tx=0x7fe41c010000 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.306+0000 7fe4400dab80 1 Processor -- start 2023-10-16T11:20:44.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.306+0000 7fe4400dab80 1 -- start start 2023-10-16T11:20:44.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c6117220 0x55b6c611ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c611c630 0x55b6c611efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe4400dab80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c6068c20 con 0x55b6c611c630 2023-10-16T11:20:44.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c604dc10 con 0x55b6c6117220 2023-10-16T11:20:44.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6c604b530 con 0x55b6c60f2010 2023-10-16T11:20:44.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c611c630 0x55b6c611efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c611c630 0x55b6c611efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40740/0 (socket says 172.21.15.158:40740) 2023-10-16T11:20:44.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 -- 172.21.15.158:0/372231963 learned_addr learned my addr 172.21.15.158:0/372231963 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe431c8c700 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c6117220 0x55b6c611ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43248d700 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 -- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 msgr2=0x55b6c60f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 -- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c6117220 msgr2=0x55b6c611ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c6117220 0x55b6c611ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 -- 172.21.15.158:0/372231963 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6c60043a0 con 0x55b6c611c630 2023-10-16T11:20:44.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe43148b700 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c611c630 0x55b6c611efb0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fe42805f5f0 tx=0x7fe428057950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85e415766b163b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.307+0000 7fe40f7fe700 1 -- 172.21.15.158:0/372231963 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42806f170 con 0x55b6c611c630 2023-10-16T11:20:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe40f7fe700 1 -- 172.21.15.158:0/372231963 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe428057b30 con 0x55b6c611c630 2023-10-16T11:20:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe40f7fe700 1 -- 172.21.15.158:0/372231963 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428056dc0 con 0x55b6c611c630 2023-10-16T11:20:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 -- 172.21.15.158:0/372231963 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c611c630 msgr2=0x55b6c611efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c611c630 0x55b6c611efb0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fe42805f5f0 tx=0x7fe428057950 comp rx=0 tx=0).stop 2023-10-16T11:20:44.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 -- 172.21.15.158:0/372231963 shutdown_connections 2023-10-16T11:20:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c6117220 0x55b6c611ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 --2- 172.21.15.158:0/372231963 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c611c630 0x55b6c611efb0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 -- 172.21.15.158:0/372231963 >> 172.21.15.158:0/372231963 conn(0x55b6c6021d80 msgr2=0x55b6c613cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 -- 172.21.15.158:0/372231963 shutdown_connections 2023-10-16T11:20:44.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.308+0000 7fe4400dab80 1 -- 172.21.15.158:0/372231963 wait complete. 2023-10-16T11:20:44.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.309+0000 7fe4400dab80 1 -- start start 2023-10-16T11:20:44.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.309+0000 7fe4400dab80 1 Processor -- start 2023-10-16T11:20:44.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c60f2010 0x55b6c61162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c6117220 0x55b6c613c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c613ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe4400dab80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c6046e20 con 0x55b6c60f2010 2023-10-16T11:20:44.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c5fe36c0 con 0x55b6c611c630 2023-10-16T11:20:44.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6c5fe3360 con 0x55b6c6117220 2023-10-16T11:20:44.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe4400dab80 10 client.0 _ll_register_callbacks cb 0x55b6c5e407b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:20:44.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c60f2010 0x55b6c61162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c60f2010 0x55b6c61162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40744/0 (socket says 172.21.15.158:40744) 2023-10-16T11:20:44.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 -- 172.21.15.158:0/311980983 learned_addr learned my addr 172.21.15.158:0/311980983 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe43248d700 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c613ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe43148b700 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c6117220 0x55b6c613c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 -- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c6117220 msgr2=0x55b6c613c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c6117220 0x55b6c613c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 -- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 msgr2=0x55b6c613ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c613ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 -- 172.21.15.158:0/311980983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42c0c69f0 con 0x55b6c60f2010 2023-10-16T11:20:44.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe431c8c700 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c60f2010 0x55b6c61162b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0cdf50 tx=0x7fe41c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b04f57ad90e4a84 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.310+0000 7fe40cff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:20:44.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe40cff9700 1 -- 172.21.15.158:0/311980983 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c003b70 con 0x55b6c60f2010 2023-10-16T11:20:44.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b6c60e75b0 con 0x55b6c60f2010 2023-10-16T11:20:44.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe40cff9700 1 -- 172.21.15.158:0/311980983 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fe41c003cd0 con 0x55b6c60f2010 2023-10-16T11:20:44.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe40cff9700 1 -- 172.21.15.158:0/311980983 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe41c003e90 con 0x55b6c60f2010 2023-10-16T11:20:44.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 10 client.12723 fetch_fsmap learned FSMap version 134 2023-10-16T11:20:44.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x55b6c5e71d90 con 0x55b6c60f2010 2023-10-16T11:20:44.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe40cff9700 1 -- 172.21.15.158:0/311980983 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b6c60e75b0 con 0x55b6c60f2010 2023-10-16T11:20:44.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe40cff9700 1 -- 172.21.15.158:0/311980983 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 134) v1 ==== 3891+0+0 (secure 0 0 0) 0x7fe41c031440 con 0x55b6c60f2010 2023-10-16T11:20:44.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 10 client.12723 fetch_fsmap finished waiting for FSMap version 134 2023-10-16T11:20:44.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 10 client.12723 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:20:44.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 20 client.12723 populate_metadata read hostname 'smithi158' 2023-10-16T11:20:44.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c611acf0 0x55b6c61565d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 4 client.12723 mds_command: new command op to 12432 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:20:44.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 --> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6c603fe00 con 0x55b6c611acf0 2023-10-16T11:20:44.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.311+0000 7fe43148b700 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c611acf0 0x55b6c61565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.312+0000 7fe40cff9700 10 client.12723 ms_handle_connect on v2:172.21.15.172:6833/1061053278 2023-10-16T11:20:44.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.312+0000 7fe43148b700 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c611acf0 0x55b6c61565d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c4d9cf925c3cdc7 server_cookie=18735a7698a3247c in_seq=0 out_seq=0 2023-10-16T11:20:44.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.312+0000 7fe40cff9700 1 -- 172.21.15.158:0/311980983 <== mds.0 v2:172.21.15.172:6833/1061053278 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b6c603fe00 con 0x55b6c611acf0 2023-10-16T11:20:44.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.312+0000 7fe40cff9700 10 client.12723 handle_command_reply: tid=0 2023-10-16T11:20:44.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.312+0000 7fe4400dab80 1 client.12723 shutdown 2023-10-16T11:20:44.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c60f2010 msgr2=0x55b6c61162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c60f2010 0x55b6c61162b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0cdf50 tx=0x7fe41c0047c0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 shutdown_connections 2023-10-16T11:20:44.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c611acf0 0x55b6c61565d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c613ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe40cff9700 0 client.12723 ms_handle_reset on v2:172.21.15.172:6833/1061053278 2023-10-16T11:20:44.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c60f2010 0x55b6c61162b0 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 --2- 172.21.15.158:0/311980983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c6117220 0x55b6c613c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 >> 172.21.15.158:0/311980983 conn(0x55b6c6021d80 msgr2=0x55b6c60e9640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 shutdown_connections 2023-10-16T11:20:44.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 1 -- 172.21.15.158:0/311980983 wait complete. 2023-10-16T11:20:44.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.313+0000 7fe4400dab80 20 client.12723 trim_cache size 0 max 16384 2023-10-16T11:20:44.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.330+0000 7fe4400dab80 1 Processor -- start 2023-10-16T11:20:44.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.330+0000 7fe4400dab80 1 -- start start 2023-10-16T11:20:44.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c60f2010 0x55b6c6117600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c6117db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c611c630 0x55b6c60f2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe4400dab80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c5ec78d0 con 0x55b6c6117220 2023-10-16T11:20:44.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c615a8d0 con 0x55b6c60f2010 2023-10-16T11:20:44.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6c613ac90 con 0x55b6c611c630 2023-10-16T11:20:44.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c6117db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c6117db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40750/0 (socket says 172.21.15.158:40750) 2023-10-16T11:20:44.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 -- 172.21.15.158:0/1432860240 learned_addr learned my addr 172.21.15.158:0/1432860240 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 -- 172.21.15.158:0/1432860240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c611c630 msgr2=0x55b6c60f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 --2- 172.21.15.158:0/1432860240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c611c630 0x55b6c60f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 -- 172.21.15.158:0/1432860240 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c60f2010 msgr2=0x55b6c6117600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 --2- 172.21.15.158:0/1432860240 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c60f2010 0x55b6c6117600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.331+0000 7fe43148b700 1 -- 172.21.15.158:0/1432860240 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6c5e5ac10 con 0x55b6c6117220 2023-10-16T11:20:44.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe43148b700 1 --2- 172.21.15.158:0/1432860240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c6117db0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fe428055c20 tx=0x7fe42801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556329bded07a91b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe40ffff700 1 -- 172.21.15.158:0/1432860240 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42806f170 con 0x55b6c6117220 2023-10-16T11:20:44.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe40ffff700 1 -- 172.21.15.158:0/1432860240 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe428057ca0 con 0x55b6c6117220 2023-10-16T11:20:44.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe40ffff700 1 -- 172.21.15.158:0/1432860240 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428057000 con 0x55b6c6117220 2023-10-16T11:20:44.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 -- 172.21.15.158:0/1432860240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 msgr2=0x55b6c6117db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 --2- 172.21.15.158:0/1432860240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c6117db0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fe428055c20 tx=0x7fe42801e000 comp rx=0 tx=0).stop 2023-10-16T11:20:44.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 -- 172.21.15.158:0/1432860240 shutdown_connections 2023-10-16T11:20:44.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 --2- 172.21.15.158:0/1432860240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c611c630 0x55b6c60f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 --2- 172.21.15.158:0/1432860240 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c60f2010 0x55b6c6117600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 --2- 172.21.15.158:0/1432860240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c6117db0 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 -- 172.21.15.158:0/1432860240 >> 172.21.15.158:0/1432860240 conn(0x55b6c6021d80 msgr2=0x55b6c613d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 -- 172.21.15.158:0/1432860240 shutdown_connections 2023-10-16T11:20:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.332+0000 7fe4400dab80 1 -- 172.21.15.158:0/1432860240 wait complete. 2023-10-16T11:20:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.333+0000 7fe4400dab80 1 -- start start 2023-10-16T11:20:44.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.333+0000 7fe4400dab80 1 Processor -- start 2023-10-16T11:20:44.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c60f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe4400dab80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c60f2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe4400dab80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c60e8f40 con 0x55b6c6117220 2023-10-16T11:20:44.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6c6185c50 con 0x55b6c611c630 2023-10-16T11:20:44.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe4400dab80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6c6117600 con 0x55b6c60f2010 2023-10-16T11:20:44.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe4400dab80 10 client.0 _ll_register_callbacks cb 0x7fe41c011d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:20:44.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c60f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c60f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40758/0 (socket says 172.21.15.158:40758) 2023-10-16T11:20:44.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 -- 172.21.15.158:0/3456053496 learned_addr learned my addr 172.21.15.158:0/3456053496 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe431c8c700 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43248d700 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c60f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 -- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 msgr2=0x55b6c60f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 -- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 msgr2=0x55b6c60f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c60f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 -- 172.21.15.158:0/3456053496 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6c60043a0 con 0x55b6c6117220 2023-10-16T11:20:44.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe43148b700 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c60f5f60 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7fe428053dd0 tx=0x7fe42800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43861b2f24ea363 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.334+0000 7fe40cff9700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:20:44.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe40cff9700 1 -- 172.21.15.158:0/3456053496 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428055d30 con 0x55b6c6117220 2023-10-16T11:20:44.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe40cff9700 1 -- 172.21.15.158:0/3456053496 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fe428083e40 con 0x55b6c6117220 2023-10-16T11:20:44.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe428057b30 con 0x55b6c6117220 2023-10-16T11:20:44.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe40cff9700 1 -- 172.21.15.158:0/3456053496 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe42805f690 con 0x55b6c6117220 2023-10-16T11:20:44.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 10 client.12729 fetch_fsmap learned FSMap version 134 2023-10-16T11:20:44.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x55b6c5e98200 con 0x55b6c6117220 2023-10-16T11:20:44.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe40cff9700 1 -- 172.21.15.158:0/3456053496 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe428057b30 con 0x55b6c6117220 2023-10-16T11:20:44.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe40cff9700 1 -- 172.21.15.158:0/3456053496 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 134) v1 ==== 3891+0+0 (secure 0 0 0) 0x7fe428087080 con 0x55b6c6117220 2023-10-16T11:20:44.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 10 client.12729 fetch_fsmap finished waiting for FSMap version 134 2023-10-16T11:20:44.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 10 client.12729 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:20:44.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 20 client.12729 populate_metadata read hostname 'smithi158' 2023-10-16T11:20:44.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c6186510 0x55b6c60f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 4 client.12729 mds_command: new command op to 12432 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:20:44.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 --> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b6c603fb20 con 0x55b6c6186510 2023-10-16T11:20:44.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.335+0000 7fe431c8c700 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c6186510 0x55b6c60f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe40cff9700 10 client.12729 ms_handle_connect on v2:172.21.15.172:6833/1061053278 2023-10-16T11:20:44.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe431c8c700 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c6186510 0x55b6c60f29c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95c83124bd0688e1 server_cookie=9a0cc8af1265f925 in_seq=0 out_seq=0 2023-10-16T11:20:44.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe40cff9700 1 -- 172.21.15.158:0/3456053496 <== mds.0 v2:172.21.15.172:6833/1061053278 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x55b6c603fb20 con 0x55b6c6186510 2023-10-16T11:20:44.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe40cff9700 10 client.12729 handle_command_reply: tid=0 2023-10-16T11:20:44.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 client.12729 shutdown 2023-10-16T11:20:44.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 msgr2=0x55b6c60f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c60f5f60 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7fe428053dd0 tx=0x7fe42800ce70 comp rx=0 tx=0).stop 2023-10-16T11:20:44.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 shutdown_connections 2023-10-16T11:20:44.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:6833/1061053278,v1:172.21.15.172:6835/1061053278] conn(0x55b6c6186510 0x55b6c60f29c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b6c611c630 0x55b6c60f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55b6c60f2010 0x55b6c60f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 --2- 172.21.15.158:0/3456053496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b6c6117220 0x55b6c60f5f60 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe40cff9700 0 client.12729 ms_handle_reset on v2:172.21.15.172:6833/1061053278 2023-10-16T11:20:44.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 >> 172.21.15.158:0/3456053496 conn(0x55b6c6021d80 msgr2=0x55b6c611f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 shutdown_connections 2023-10-16T11:20:44.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 1 -- 172.21.15.158:0/3456053496 wait complete. 2023-10-16T11:20:44.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.336+0000 7fe4400dab80 20 client.12729 trim_cache size 0 max 16384 2023-10-16T11:20:44.358 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:20:44.359 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:20:44.359 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:20:44.359 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 12432, 2023-10-16T11:20:44.359 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:20:44.359 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:20:44.360 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "backup_fs", 2023-10-16T11:20:44.360 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.5256218989999999, 2023-10-16T11:20:44.360 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 134, 2023-10-16T11:20:44.360 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 241, 2023-10-16T11:20:44.360 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 241, 2023-10-16T11:20:44.360 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 23.525276439999999 2023-10-16T11:20:44.361 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:20:44.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe41006b140 msgr2=0x7fe41006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe41006b140 0x7fe41006d600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0090b0 tx=0x7fe41c010000 comp rx=0 tx=0).stop 2023-10-16T11:20:44.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 msgr2=0x7fe42c1f3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c1f3a20 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7fe42800c480 tx=0x7fe42800c840 comp rx=0 tx=0).stop 2023-10-16T11:20:44.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 shutdown_connections 2023-10-16T11:20:44.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fe41006b140 0x7fe41006d600 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe42c15f4d0 0x7fe42c0b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe42c06afe0 0x7fe42c0c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 --2- 172.21.15.158:0/367224807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe42c15b6e0 0x7fe42c1f3a20 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 >> 172.21.15.158:0/367224807 conn(0x7fe42c06b9e0 msgr2=0x7fe42c0b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 shutdown_connections 2023-10-16T11:20:44.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.337+0000 7fe433ef0700 1 -- 172.21.15.158:0/367224807 wait complete. 2023-10-16T11:20:44.365 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T11:20:44.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.486+0000 7fa9a6f71700 1 Processor -- start 2023-10-16T11:20:44.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.486+0000 7fa9a6f71700 1 -- start start 2023-10-16T11:20:44.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a6f71700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa9a0158280 0x7fa9a006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a6f71700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9a00cf180 0x7fa9a00cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a6f71700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a00cb2d0 con 0x7fa9a00722e0 2023-10-16T11:20:44.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a6f71700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a015bc40 con 0x7fa9a0158280 2023-10-16T11:20:44.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a6f71700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a0158660 con 0x7fa9a00cf180 2023-10-16T11:20:44.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a4d0d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa9a0158280 0x7fa9a006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a4d0d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa9a0158280 0x7fa9a006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40764/0 (socket says 172.21.15.158:40764) 2023-10-16T11:20:44.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a4d0d700 1 -- 172.21.15.158:0/4075711162 learned_addr learned my addr 172.21.15.158:0/4075711162 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a550e700 1 --2- 172.21.15.158:0/4075711162 >> v2:172.21.15.172:3300/0 conn(0x7fa9a00cf180 0x7fa9a00cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a4d0d700 1 -- 172.21.15.158:0/4075711162 >> v2:172.21.15.172:3300/0 conn(0x7fa9a00cf180 msgr2=0x7fa9a00cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a4d0d700 1 --2- 172.21.15.158:0/4075711162 >> v2:172.21.15.172:3300/0 conn(0x7fa9a00cf180 0x7fa9a00cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a4d0d700 1 -- 172.21.15.158:0/4075711162 >> v1:172.21.15.172:6789/0 conn(0x7fa9a00722e0 legacy=0x7fa9a00cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.487+0000 7fa9a4d0d700 1 -- 172.21.15.158:0/4075711162 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a015b030 con 0x7fa9a0158280 2023-10-16T11:20:44.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a4d0d700 1 --2- 172.21.15.158:0/4075711162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a0158280 0x7fa9a006a8d0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0071160 tx=0x7fa994009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919f92b4c035683d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa99f7fe700 1 -- 172.21.15.158:0/4075711162 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99400b870 con 0x7fa9a0158280 2023-10-16T11:20:44.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa99f7fe700 1 -- 172.21.15.158:0/4075711162 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99400b9d0 con 0x7fa9a0158280 2023-10-16T11:20:44.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa99f7fe700 1 -- 172.21.15.158:0/4075711162 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99400bd60 con 0x7fa9a0158280 2023-10-16T11:20:44.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 -- 172.21.15.158:0/4075711162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a0158280 msgr2=0x7fa9a006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/4075711162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a0158280 0x7fa9a006a8d0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0071160 tx=0x7fa994009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 -- 172.21.15.158:0/4075711162 shutdown_connections 2023-10-16T11:20:44.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/4075711162 >> v2:172.21.15.172:3300/0 conn(0x7fa9a00cf180 0x7fa9a00cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/4075711162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a0158280 0x7fa9a006a8d0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 -- 172.21.15.158:0/4075711162 >> 172.21.15.158:0/4075711162 conn(0x7fa9a00cc700 msgr2=0x7fa9a00ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 -- 172.21.15.158:0/4075711162 shutdown_connections 2023-10-16T11:20:44.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.488+0000 7fa9a6f71700 1 -- 172.21.15.158:0/4075711162 wait complete. 2023-10-16T11:20:44.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.489+0000 7fa9a6f71700 1 Processor -- start 2023-10-16T11:20:44.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa9a6f71700 1 -- start start 2023-10-16T11:20:44.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa9a6f71700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00722e0 0x7fa9a014dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa9a6f71700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a00cf180 0x7fa9a00b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa9a6f71700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a0158280 0x7fa9a01f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa9a6f71700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00b2ad0 con 0x7fa9a00cf180 2023-10-16T11:20:44.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa9a6f71700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00b53c0 con 0x7fa9a0158280 2023-10-16T11:20:44.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa9a6f71700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00b47b0 con 0x7fa9a00722e0 2023-10-16T11:20:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.490+0000 7fa99ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a00cf180 0x7fa9a00b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a00cf180 0x7fa9a00b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40774/0 (socket says 172.21.15.158:40774) 2023-10-16T11:20:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 -- 172.21.15.158:0/3305646127 learned_addr learned my addr 172.21.15.158:0/3305646127 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa9a550e700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a0158280 0x7fa9a01f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa9a4d0d700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00722e0 0x7fa9a014dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 -- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00722e0 msgr2=0x7fa9a014dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00722e0 0x7fa9a014dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 -- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a0158280 msgr2=0x7fa9a01f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a0158280 0x7fa9a01f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 -- 172.21.15.158:0/3305646127 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa994009550 con 0x7fa9a00cf180 2023-10-16T11:20:44.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99ffff700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a00cf180 0x7fa9a00b36a0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00c4b0 tx=0x7fa98c00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99d7fa700 1 -- 172.21.15.158:0/3305646127 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0052d0 con 0x7fa9a00cf180 2023-10-16T11:20:44.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99d7fa700 1 -- 172.21.15.158:0/3305646127 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c005430 con 0x7fa9a00cf180 2023-10-16T11:20:44.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a00b4a90 con 0x7fa9a00cf180 2023-10-16T11:20:44.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa99d7fa700 1 -- 172.21.15.158:0/3305646127 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0038c0 con 0x7fa9a00cf180 2023-10-16T11:20:44.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.491+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a00b5050 con 0x7fa9a00cf180 2023-10-16T11:20:44.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.492+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a00681b0 con 0x7fa9a00cf180 2023-10-16T11:20:44.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.493+0000 7fa99d7fa700 1 -- 172.21.15.158:0/3305646127 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa98c003a20 con 0x7fa9a00cf180 2023-10-16T11:20:44.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.493+0000 7fa99d7fa700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa98406b140 0x7fa98406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.493+0000 7fa99d7fa700 1 -- 172.21.15.158:0/3305646127 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fa98c089470 con 0x7fa9a00cf180 2023-10-16T11:20:44.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.493+0000 7fa9a4d0d700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa98406b140 0x7fa98406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.493+0000 7fa9a4d0d700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa98406b140 0x7fa98406d600 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa994014040 tx=0x7fa994008030 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.497+0000 7fa99d7fa700 1 -- 172.21.15.158:0/3305646127 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa98c053c10 con 0x7fa9a00cf180 2023-10-16T11:20:44.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.718+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 --> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9a015b5f0 con 0x7fa98406b140 2023-10-16T11:20:44.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.719+0000 7fa99d7fa700 1 -- 172.21.15.158:0/3305646127 <== mgr.12030 v2:172.21.15.158:6832/3408161846 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fa99400bb40 con 0x7fa98406b140 2023-10-16T11:20:44.721 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:20:44.721 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:44.721 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T11:20:44.721 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:20:44.722 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:20:44.722 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "backup_fs" 2023-10-16T11:20:44.722 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:20:44.722 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:20:44.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.721+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa98406b140 msgr2=0x7fa98406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.721+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa98406b140 0x7fa98406d600 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa994014040 tx=0x7fa994008030 comp rx=0 tx=0).stop 2023-10-16T11:20:44.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.721+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a00cf180 msgr2=0x7fa9a00b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.721+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a00cf180 0x7fa9a00b36a0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00c4b0 tx=0x7fa98c00c870 comp rx=0 tx=0).stop 2023-10-16T11:20:44.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 shutdown_connections 2023-10-16T11:20:44.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fa98406b140 0x7fa98406d600 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a0158280 0x7fa9a01f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00722e0 0x7fa9a014dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 --2- 172.21.15.158:0/3305646127 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9a00cf180 0x7fa9a00b36a0 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 >> 172.21.15.158:0/3305646127 conn(0x7fa9a00cc700 msgr2=0x7fa9a015bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 shutdown_connections 2023-10-16T11:20:44.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.722+0000 7fa9a6f71700 1 -- 172.21.15.158:0/3305646127 wait complete. 2023-10-16T11:20:44.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T11:20:44.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.856+0000 7f6c53493700 1 Processor -- start 2023-10-16T11:20:44.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.857+0000 7f6c53493700 1 -- start start 2023-10-16T11:20:44.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.857+0000 7f6c53493700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c4c15d910 0x7f6c4c0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.857+0000 7f6c53493700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c06a8d0 con 0x7f6c4c0700a0 2023-10-16T11:20:44.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.857+0000 7f6c53493700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c161e50 con 0x7f6c4c0cc9e0 2023-10-16T11:20:44.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.857+0000 7f6c53493700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c072970 con 0x7f6c4c15d910 2023-10-16T11:20:44.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.857+0000 7f6c51a30700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6c4c0700a0 0x7f6c4c06f590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:40958/0 (socket says 172.21.15.158:40958) 2023-10-16T11:20:44.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c51a30700 1 -- 172.21.15.158:0/2416232459 learned_addr learned my addr 172.21.15.158:0/2416232459 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:44.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c5122f700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c4c15d910 0x7f6c4c0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216840850 0 0) 0x7f6c4c06a8d0 con 0x7f6c4c0700a0 2023-10-16T11:20:44.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c380035b0 con 0x7f6c4c0700a0 2023-10-16T11:20:44.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510549647 0 0) 0x7f6c380035b0 con 0x7f6c4c0700a0 2023-10-16T11:20:44.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c4c15d910 msgr2=0x7f6c4c0cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c4c15d910 0x7f6c4c0cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 >> v1:172.21.15.172:6789/0 conn(0x7f6c4c0cc9e0 legacy=0x7f6c4c06ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c06fca0 con 0x7f6c4c0700a0 2023-10-16T11:20:44.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f6c440019f0 con 0x7f6c4c0700a0 2023-10-16T11:20:44.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 >> v1:172.21.15.158:6789/0 conn(0x7f6c4c0700a0 legacy=0x7f6c4c06f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c38003a30 0x7f6c38003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38004420 0x7f6c380048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c380050f0 0x7f6c3800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c072970 con 0x7f6c380050f0 2023-10-16T11:20:44.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3800c000 con 0x7f6c38004420 2023-10-16T11:20:44.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3800c1c0 con 0x7f6c38003a30 2023-10-16T11:20:44.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c5122f700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c4c15d910 0x7f6c4c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:44.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c51a30700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c380050f0 0x7f6c3800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c50a2e700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38004420 0x7f6c380048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.858+0000 7f6c5122f700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c38003a30 0x7f6c38003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.859+0000 7f6c51a30700 1 -- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c38003a30 msgr2=0x7f6c38003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.859+0000 7f6c51a30700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c38003a30 0x7f6c38003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.859+0000 7f6c51a30700 1 -- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38004420 msgr2=0x7f6c380048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:44.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.859+0000 7f6c51a30700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38004420 0x7f6c380048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.859+0000 7f6c51a30700 1 -- 172.21.15.158:0/2416232459 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c380035b0 con 0x7f6c380050f0 2023-10-16T11:20:44.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.859+0000 7f6c51a30700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c380050f0 0x7f6c3800ba10 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400a800 tx=0x7f6c38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e43ce9412502589 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4400a2f0 con 0x7f6c380050f0 2023-10-16T11:20:44.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c44012620 con 0x7f6c380050f0 2023-10-16T11:20:44.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c43fff700 1 -- 172.21.15.158:0/2416232459 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440129d0 con 0x7f6c380050f0 2023-10-16T11:20:44.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c53493700 1 -- 172.21.15.158:0/2416232459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c380050f0 msgr2=0x7f6c3800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c53493700 1 --2- 172.21.15.158:0/2416232459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c380050f0 0x7f6c3800ba10 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400a800 tx=0x7f6c38005880 comp rx=0 tx=0).stop 2023-10-16T11:20:44.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c51a30700 1 -- 172.21.15.158:0/2416232459 reap_dead start 2023-10-16T11:20:44.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c53493700 1 -- 172.21.15.158:0/2416232459 shutdown_connections 2023-10-16T11:20:44.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c53493700 1 -- 172.21.15.158:0/2416232459 >> 172.21.15.158:0/2416232459 conn(0x7f6c4c0b1e90 msgr2=0x7f6c4c0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:44.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c53493700 1 -- 172.21.15.158:0/2416232459 shutdown_connections 2023-10-16T11:20:44.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.861+0000 7f6c53493700 1 -- 172.21.15.158:0/2416232459 wait complete. 2023-10-16T11:20:44.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.863+0000 7f6c53493700 1 Processor -- start 2023-10-16T11:20:44.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.863+0000 7f6c53493700 1 -- start start 2023-10-16T11:20:44.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c53493700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380050f0 0x7f6c4c15ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c53493700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c4c15cd20 0x7f6c4c1f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c53493700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c4c1f3500 0x7f6c4c0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c53493700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c15d1a0 con 0x7f6c4c1f3500 2023-10-16T11:20:44.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c53493700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c1f3ab0 con 0x7f6c4c15cd20 2023-10-16T11:20:44.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c53493700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c1f3d90 con 0x7f6c380050f0 2023-10-16T11:20:44.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c4c1f3500 0x7f6c4c0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c4c1f3500 0x7f6c4c0b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40794/0 (socket says 172.21.15.158:40794) 2023-10-16T11:20:44.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 -- 172.21.15.158:0/291056786 learned_addr learned my addr 172.21.15.158:0/291056786 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:44.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c5122f700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380050f0 0x7f6c4c15ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c50a2e700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c4c15cd20 0x7f6c4c1f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 -- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380050f0 msgr2=0x7f6c4c15ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380050f0 0x7f6c4c15ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 -- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c4c15cd20 msgr2=0x7f6c4c1f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:44.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c4c15cd20 0x7f6c4c1f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:44.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.864+0000 7f6c51a30700 1 -- 172.21.15.158:0/291056786 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c0c7f80 con 0x7f6c4c1f3500 2023-10-16T11:20:44.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.865+0000 7f6c51a30700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c4c1f3500 0x7f6c4c0b4f70 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f6c44001cb0 tx=0x7f6c44001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.865+0000 7f6c41ffb700 1 -- 172.21.15.158:0/291056786 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44023070 con 0x7f6c4c1f3500 2023-10-16T11:20:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.865+0000 7f6c41ffb700 1 -- 172.21.15.158:0/291056786 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4400a210 con 0x7f6c4c1f3500 2023-10-16T11:20:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.865+0000 7f6c41ffb700 1 -- 172.21.15.158:0/291056786 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4401e430 con 0x7f6c4c1f3500 2023-10-16T11:20:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.865+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c0c8260 con 0x7f6c4c1f3500 2023-10-16T11:20:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.865+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4c0b2f00 con 0x7f6c4c1f3500 2023-10-16T11:20:44.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.866+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4c0c8780 con 0x7f6c4c1f3500 2023-10-16T11:20:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.866+0000 7f6c41ffb700 1 -- 172.21.15.158:0/291056786 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6c4401e7a0 con 0x7f6c4c1f3500 2023-10-16T11:20:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.867+0000 7f6c41ffb700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6c3006b140 0x7f6c3006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.867+0000 7f6c5122f700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6c3006b140 0x7f6c3006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.867+0000 7f6c41ffb700 1 -- 172.21.15.158:0/291056786 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f6c4408bf40 con 0x7f6c4c1f3500 2023-10-16T11:20:44.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.867+0000 7f6c5122f700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6c3006b140 0x7f6c3006d600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c160d80 tx=0x7f6c3c009210 comp rx=0 tx=0).ready entity=mgr.12030 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:44.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:44.871+0000 7f6c41ffb700 1 -- 172.21.15.158:0/291056786 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c4c0c8780 con 0x7f6c4c1f3500 2023-10-16T11:20:45.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.119+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6c4c0c8780 con 0x7f6c4c1f3500 2023-10-16T11:20:45.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.868+0000 7f6c41ffb700 1 -- 172.21.15.158:0/291056786 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v148) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6c4c0c8780 con 0x7f6c4c1f3500 2023-10-16T11:20:45.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6c3006b140 msgr2=0x7f6c3006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:45.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6c3006b140 0x7f6c3006d600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c160d80 tx=0x7f6c3c009210 comp rx=0 tx=0).stop 2023-10-16T11:20:45.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c4c1f3500 msgr2=0x7f6c4c0b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:45.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c4c1f3500 0x7f6c4c0b4f70 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f6c44001cb0 tx=0x7f6c44001d80 comp rx=0 tx=0).stop 2023-10-16T11:20:45.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 shutdown_connections 2023-10-16T11:20:45.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f6c3006b140 0x7f6c3006d600 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:45.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c4c1f3500 0x7f6c4c0b4f70 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:45.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380050f0 0x7f6c4c15ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:45.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 --2- 172.21.15.158:0/291056786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c4c15cd20 0x7f6c4c1f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:45.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 >> 172.21.15.158:0/291056786 conn(0x7f6c4c0b1e90 msgr2=0x7f6c4c0afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:45.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.871+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 shutdown_connections 2023-10-16T11:20:45.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:45.872+0000 7f6c53493700 1 -- 172.21.15.158:0/291056786 wait complete. 2023-10-16T11:20:45.892 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T11:20:46.018 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.017+0000 7f1b19634200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:20:46.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.025+0000 7f03e4b93700 1 Processor -- start 2023-10-16T11:20:46.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.025+0000 7f03e4b93700 1 -- start start 2023-10-16T11:20:46.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.026+0000 7f03e4b93700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f03e006afe0 0x7f03e00b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.026+0000 7f03e4b93700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03e00af700 0x7f03e00b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.026+0000 7f03e4b93700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e006ef20 con 0x7f03e015b4f0 2023-10-16T11:20:46.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.026+0000 7f03e4b93700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e00c3d70 con 0x7f03e00af700 2023-10-16T11:20:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.026+0000 7f03e4b93700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e006f200 con 0x7f03e006afe0 2023-10-16T11:20:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.026+0000 7f03ded9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03e00af700 0x7f03e00b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ded9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03e00af700 0x7f03e00b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40828/0 (socket says 172.21.15.158:40828) 2023-10-16T11:20:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ded9d700 1 -- 172.21.15.158:0/2962147665 learned_addr learned my addr 172.21.15.158:0/2962147665 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:46.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ddd9b700 1 --2- 172.21.15.158:0/2962147665 >> v2:172.21.15.172:3300/0 conn(0x7f03e006afe0 0x7f03e00b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ded9d700 1 -- 172.21.15.158:0/2962147665 >> v2:172.21.15.172:3300/0 conn(0x7f03e006afe0 msgr2=0x7f03e00b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ded9d700 1 --2- 172.21.15.158:0/2962147665 >> v2:172.21.15.172:3300/0 conn(0x7f03e006afe0 0x7f03e00b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ded9d700 1 -- 172.21.15.158:0/2962147665 >> v1:172.21.15.172:6789/0 conn(0x7f03e015b4f0 legacy=0x7f03e006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ded9d700 1 -- 172.21.15.158:0/2962147665 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e006f4e0 con 0x7f03e00af700 2023-10-16T11:20:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03ded9d700 1 --2- 172.21.15.158:0/2962147665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e00b3730 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009e50 tx=0x7f03d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8484ee9654860cd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03dd59a700 1 -- 172.21.15.158:0/2962147665 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d800e070 con 0x7f03e00af700 2023-10-16T11:20:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03dd59a700 1 -- 172.21.15.158:0/2962147665 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d800b8d0 con 0x7f03e00af700 2023-10-16T11:20:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.027+0000 7f03dd59a700 1 -- 172.21.15.158:0/2962147665 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d800bc80 con 0x7f03e00af700 2023-10-16T11:20:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 -- 172.21.15.158:0/2962147665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 msgr2=0x7f03e00b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 --2- 172.21.15.158:0/2962147665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e00b3730 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009e50 tx=0x7f03d800b330 comp rx=0 tx=0).stop 2023-10-16T11:20:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 -- 172.21.15.158:0/2962147665 shutdown_connections 2023-10-16T11:20:46.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 --2- 172.21.15.158:0/2962147665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e00b3730 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 --2- 172.21.15.158:0/2962147665 >> v2:172.21.15.172:3300/0 conn(0x7f03e006afe0 0x7f03e00b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 -- 172.21.15.158:0/2962147665 >> 172.21.15.158:0/2962147665 conn(0x7f03e00b1e90 msgr2=0x7f03e00b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 -- 172.21.15.158:0/2962147665 shutdown_connections 2023-10-16T11:20:46.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.028+0000 7f03e4b93700 1 -- 172.21.15.158:0/2962147665 wait complete. 2023-10-16T11:20:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.029+0000 7f03e4b93700 1 Processor -- start 2023-10-16T11:20:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.029+0000 7f03e4b93700 1 -- start start 2023-10-16T11:20:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03e4b93700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03e006afe0 0x7f03e01f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03e4b93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e0158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03e4b93700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e015b4f0 0x7f03e0158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03e4b93700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00b5fc0 con 0x7f03e00af700 2023-10-16T11:20:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03e4b93700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00b3de0 con 0x7f03e015b4f0 2023-10-16T11:20:46.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03e4b93700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00b40c0 con 0x7f03e006afe0 2023-10-16T11:20:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e0158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e0158680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40832/0 (socket says 172.21.15.158:40832) 2023-10-16T11:20:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 -- 172.21.15.158:0/623585668 learned_addr learned my addr 172.21.15.158:0/623585668 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:46.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ded9d700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e015b4f0 0x7f03e0158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03de59c700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03e006afe0 0x7f03e01f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 -- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03e006afe0 msgr2=0x7f03e01f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03e006afe0 0x7f03e01f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 -- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e015b4f0 msgr2=0x7f03e0158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e015b4f0 0x7f03e0158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.030+0000 7f03ddd9b700 1 -- 172.21.15.158:0/623585668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8009b30 con 0x7f03e00af700 2023-10-16T11:20:46.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.031+0000 7f03ddd9b700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e0158680 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f03d400e980 tx=0x7f03d400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:46.044 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.031+0000 7f4d00bf9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.031+0000 7f03ceffd700 1 -- 172.21.15.158:0/623585668 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4013680 con 0x7f03e00af700 2023-10-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.031+0000 7f03ceffd700 1 -- 172.21.15.158:0/623585668 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d40137e0 con 0x7f03e00af700 2023-10-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.031+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e00b43a0 con 0x7f03e00af700 2023-10-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.031+0000 7f03ceffd700 1 -- 172.21.15.158:0/623585668 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4013b50 con 0x7f03e00af700 2023-10-16T11:20:46.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.031+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e00c8410 con 0x7f03e00af700 2023-10-16T11:20:46.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.032+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e00c85d0 con 0x7f03e00af700 2023-10-16T11:20:46.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.032+0000 7f03ceffd700 1 -- 172.21.15.158:0/623585668 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 148) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f03d4013cb0 con 0x7f03e00af700 2023-10-16T11:20:46.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.033+0000 7f03ceffd700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 0x7f03c006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.033+0000 7f03de59c700 1 -- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 msgr2=0x7f03c006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:46.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.033+0000 7f03ceffd700 1 -- 172.21.15.158:0/623585668 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f03d408a850 con 0x7f03e00af700 2023-10-16T11:20:46.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.033+0000 7f03de59c700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 0x7f03c006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:46.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.037+0000 7f03ceffd700 1 -- 172.21.15.158:0/623585668 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03d4054f00 con 0x7f03e00af700 2023-10-16T11:20:46.085 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.084+0000 7f1b19634200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:46.101 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.100+0000 7f4d00bf9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:46.157 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.156+0000 7f1b19634200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:20:46.174 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.173+0000 7f4d00bf9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:20:46.234 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.233+0000 7f1b19634200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:20:46.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.234+0000 7f03de59c700 1 -- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 msgr2=0x7f03c006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:46.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.234+0000 7f03de59c700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 0x7f03c006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:46.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.240+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f03e00681b0 con 0x7f03e00af700 2023-10-16T11:20:46.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.246+0000 7f03ceffd700 1 -- 172.21.15.158:0/623585668 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f03d40210d0 con 0x7f03e00af700 2023-10-16T11:20:46.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 msgr2=0x7f03c006d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:46.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 0x7f03c006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 msgr2=0x7f03e0158680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e0158680 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f03d400e980 tx=0x7f03d400ec90 comp rx=0 tx=0).stop 2023-10-16T11:20:46.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 shutdown_connections 2023-10-16T11:20:46.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f03c006b190 0x7f03c006d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e015b4f0 0x7f03e0158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03e006afe0 0x7f03e01f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 --2- 172.21.15.158:0/623585668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e00af700 0x7f03e0158680 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 >> 172.21.15.158:0/623585668 conn(0x7f03e00b1e90 msgr2=0x7f03e006d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:46.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 shutdown_connections 2023-10-16T11:20:46.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.248+0000 7f03e4b93700 1 -- 172.21.15.158:0/623585668 wait complete. 2023-10-16T11:20:46.255 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.254+0000 7f4d00bf9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:20:46.262 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T11:20:46.376 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.374+0000 7f1b19634200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:20:46.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.387+0000 7f5efcf86700 1 Processor -- start 2023-10-16T11:20:46.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.387+0000 7f5efcf86700 1 -- start start 2023-10-16T11:20:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5efcf86700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef815a5b0 0x7f5ef8071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5efcf86700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ef80cb900 0x7f5ef8071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5efcf86700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ef80726a0 0x7f5ef806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5efcf86700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef80cdf70 con 0x7f5ef80cb900 2023-10-16T11:20:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5efcf86700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef80ce150 con 0x7f5ef80726a0 2023-10-16T11:20:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5efcf86700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef80ce430 con 0x7f5ef815a5b0 2023-10-16T11:20:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5ef5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ef80cb900 0x7f5ef8071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5ef5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ef80cb900 0x7f5ef8071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40876/0 (socket says 172.21.15.158:40876) 2023-10-16T11:20:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/542546631 learned_addr learned my addr 172.21.15.158:0/542546631 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5ef659c700 1 --2- 172.21.15.158:0/542546631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef815a5b0 0x7f5ef8071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.388+0000 7f5ef6d9d700 1 --2- 172.21.15.158:0/542546631 >> v2:172.21.15.172:3300/0 conn(0x7f5ef80726a0 0x7f5ef806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/542546631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef815a5b0 msgr2=0x7f5ef8071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef5d9b700 1 --2- 172.21.15.158:0/542546631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef815a5b0 0x7f5ef8071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/542546631 >> v2:172.21.15.172:3300/0 conn(0x7f5ef80726a0 msgr2=0x7f5ef806ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:46.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef5d9b700 1 --2- 172.21.15.158:0/542546631 >> v2:172.21.15.172:3300/0 conn(0x7f5ef80726a0 0x7f5ef806ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/542546631 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef80ce710 con 0x7f5ef80cb900 2023-10-16T11:20:46.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef5d9b700 1 --2- 172.21.15.158:0/542546631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef8071ea0 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f5ef806eba0 tx=0x7f5ee8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2487c616099d2d21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:46.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef559a700 1 -- 172.21.15.158:0/542546631 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8011830 con 0x7f5ef80cb900 2023-10-16T11:20:46.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef559a700 1 -- 172.21.15.158:0/542546631 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee8011990 con 0x7f5ef80cb900 2023-10-16T11:20:46.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.389+0000 7f5ef559a700 1 -- 172.21.15.158:0/542546631 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8011d20 con 0x7f5ef80cb900 2023-10-16T11:20:46.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 -- 172.21.15.158:0/542546631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 msgr2=0x7f5ef8071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 --2- 172.21.15.158:0/542546631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef8071ea0 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f5ef806eba0 tx=0x7f5ee8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 -- 172.21.15.158:0/542546631 shutdown_connections 2023-10-16T11:20:46.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 --2- 172.21.15.158:0/542546631 >> v2:172.21.15.172:3300/0 conn(0x7f5ef80726a0 0x7f5ef806ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 --2- 172.21.15.158:0/542546631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef815a5b0 0x7f5ef8071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 --2- 172.21.15.158:0/542546631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef8071ea0 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 -- 172.21.15.158:0/542546631 >> 172.21.15.158:0/542546631 conn(0x7f5ef80b1e90 msgr2=0x7f5ef80b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:46.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 -- 172.21.15.158:0/542546631 shutdown_connections 2023-10-16T11:20:46.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.390+0000 7f5efcf86700 1 -- 172.21.15.158:0/542546631 wait complete. 2023-10-16T11:20:46.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.391+0000 7f5efcf86700 1 Processor -- start 2023-10-16T11:20:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.391+0000 7f5efcf86700 1 -- start start 2023-10-16T11:20:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5efcf86700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef80726a0 0x7f5ef814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5efcf86700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef80c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5efcf86700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef815a5b0 0x7f5ef81f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5efcf86700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef80c8720 con 0x7f5ef80cb900 2023-10-16T11:20:46.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5efcf86700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef80c8a00 con 0x7f5ef815a5b0 2023-10-16T11:20:46.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5efcf86700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81f3340 con 0x7f5ef80726a0 2023-10-16T11:20:46.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef80c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef80726a0 0x7f5ef814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef80c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40884/0 (socket says 172.21.15.158:40884) 2023-10-16T11:20:46.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/2018136822 learned_addr learned my addr 172.21.15.158:0/2018136822 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:46.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef6d9d700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef815a5b0 0x7f5ef81f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/2018136822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef80726a0 msgr2=0x7f5ef814b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef80726a0 0x7f5ef814b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/2018136822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef815a5b0 msgr2=0x7f5ef81f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef815a5b0 0x7f5ef81f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 -- 172.21.15.158:0/2018136822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8009550 con 0x7f5ef80cb900 2023-10-16T11:20:46.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ef5d9b700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef80c7520 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8000c00 tx=0x7f5ee8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:46.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.392+0000 7f5ee6ffd700 1 -- 172.21.15.158:0/2018136822 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8004240 con 0x7f5ef80cb900 2023-10-16T11:20:46.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.393+0000 7f5ee6ffd700 1 -- 172.21.15.158:0/2018136822 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee801f070 con 0x7f5ef80cb900 2023-10-16T11:20:46.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.393+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef81f3620 con 0x7f5ef80cb900 2023-10-16T11:20:46.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.393+0000 7f5ee6ffd700 1 -- 172.21.15.158:0/2018136822 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee80036a0 con 0x7f5ef80cb900 2023-10-16T11:20:46.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.393+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef80c8ce0 con 0x7f5ef80cb900 2023-10-16T11:20:46.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.393+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef8074a90 con 0x7f5ef80cb900 2023-10-16T11:20:46.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.394+0000 7f5ee6ffd700 1 -- 172.21.15.158:0/2018136822 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 148) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f5ee8021030 con 0x7f5ef80cb900 2023-10-16T11:20:46.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.394+0000 7f5ee6ffd700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 0x7f5ed806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.394+0000 7f5ef659c700 1 -- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 msgr2=0x7f5ed806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:46.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.394+0000 7f5ee6ffd700 1 -- 172.21.15.158:0/2018136822 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f5ee808b1b0 con 0x7f5ef80cb900 2023-10-16T11:20:46.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.394+0000 7f5ef659c700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 0x7f5ed806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:46.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.399+0000 7f5ee6ffd700 1 -- 172.21.15.158:0/2018136822 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ee8055910 con 0x7f5ef80cb900 2023-10-16T11:20:46.409 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.408+0000 7f4d00bf9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:20:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.595+0000 7f5ef659c700 1 -- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 msgr2=0x7f5ed806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.595+0000 7f5ef659c700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 0x7f5ed806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:46.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.600+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5ef80681b0 con 0x7f5ef80cb900 2023-10-16T11:20:46.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.603+0000 7f5ee6ffd700 1 -- 172.21.15.158:0/2018136822 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5ee8058f30 con 0x7f5ef80cb900 2023-10-16T11:20:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 msgr2=0x7f5ed806d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 0x7f5ed806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 msgr2=0x7f5ef80c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef80c7520 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8000c00 tx=0x7f5ee8004750 comp rx=0 tx=0).stop 2023-10-16T11:20:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 shutdown_connections 2023-10-16T11:20:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7f5ed806b190 0x7f5ed806d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef815a5b0 0x7f5ef81f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ef80726a0 0x7f5ef814b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 --2- 172.21.15.158:0/2018136822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ef80cb900 0x7f5ef80c7520 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 >> 172.21.15.158:0/2018136822 conn(0x7f5ef80b1e90 msgr2=0x7f5ef806e250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 shutdown_connections 2023-10-16T11:20:46.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.605+0000 7f5efcf86700 1 -- 172.21.15.158:0/2018136822 wait complete. 2023-10-16T11:20:46.619 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:20:46.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:46.620 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T11:20:46.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:20:46.682 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2023-10-16T11:20:46.705 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.703+0000 7f1b19634200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:20:46.748 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.746+0000 7f4d00bf9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:20:46.823 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.822+0000 7f1b19634200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:20:46.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.852+0000 7fd73cd5a700 1 Processor -- start 2023-10-16T11:20:46.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.852+0000 7fd73cd5a700 1 -- start start 2023-10-16T11:20:46.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd73cd5a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7380cc590 0x7fd7380caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd73cd5a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7380cbd30 con 0x7fd73806afe0 2023-10-16T11:20:46.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd73cd5a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7380cc1c0 con 0x7fd73815da40 2023-10-16T11:20:46.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd73cd5a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73815de20 con 0x7fd7380cc590 2023-10-16T11:20:46.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd736d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7380cc590 0x7fd7380caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd736d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd7380cc590 0x7fd7380caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40914/0 (socket says 172.21.15.158:40914) 2023-10-16T11:20:46.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd736d9d700 1 -- 172.21.15.158:0/572332248 learned_addr learned my addr 172.21.15.158:0/572332248 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:46.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.853+0000 7fd736d9d700 1 -- 172.21.15.158:0/572332248 >> v1:172.21.15.172:6789/0 conn(0x7fd73815da40 legacy=0x7fd73806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.854+0000 7fd736d9d700 1 -- 172.21.15.158:0/572332248 >> v1:172.21.15.158:6789/0 conn(0x7fd73806afe0 legacy=0x7fd7380afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.854+0000 7fd736d9d700 1 -- 172.21.15.158:0/572332248 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7380ce260 con 0x7fd7380cc590 2023-10-16T11:20:46.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.854+0000 7fd736d9d700 1 --2- 172.21.15.158:0/572332248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7380cc590 0x7fd7380caa20 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7fd730009e50 tx=0x7fd73000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a3c0bf56f80ebb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:46.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.854+0000 7fd73559a700 1 -- 172.21.15.158:0/572332248 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73000e070 con 0x7fd7380cc590 2023-10-16T11:20:46.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.854+0000 7fd73559a700 1 -- 172.21.15.158:0/572332248 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73000b9c0 con 0x7fd7380cc590 2023-10-16T11:20:46.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.854+0000 7fd73559a700 1 -- 172.21.15.158:0/572332248 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73000bd70 con 0x7fd7380cc590 2023-10-16T11:20:46.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.855+0000 7fd73cd5a700 1 -- 172.21.15.158:0/572332248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7380cc590 msgr2=0x7fd7380caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.855+0000 7fd73cd5a700 1 --2- 172.21.15.158:0/572332248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7380cc590 0x7fd7380caa20 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7fd730009e50 tx=0x7fd73000b330 comp rx=0 tx=0).stop 2023-10-16T11:20:46.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.855+0000 7fd73cd5a700 1 -- 172.21.15.158:0/572332248 shutdown_connections 2023-10-16T11:20:46.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.855+0000 7fd73cd5a700 1 --2- 172.21.15.158:0/572332248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7380cc590 0x7fd7380caa20 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.855+0000 7fd73cd5a700 1 -- 172.21.15.158:0/572332248 >> 172.21.15.158:0/572332248 conn(0x7fd738099a10 msgr2=0x7fd7380d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:46.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.855+0000 7fd73cd5a700 1 -- 172.21.15.158:0/572332248 shutdown_connections 2023-10-16T11:20:46.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.855+0000 7fd73cd5a700 1 -- 172.21.15.158:0/572332248 wait complete. 2023-10-16T11:20:46.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.856+0000 7fd73cd5a700 1 Processor -- start 2023-10-16T11:20:46.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.856+0000 7fd73cd5a700 1 -- start start 2023-10-16T11:20:46.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd73cd5a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73806afe0 0x7fd738159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd73cd5a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7380cc590 0x7fd73815c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd73cd5a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd73815da40 0x7fd7380c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd73cd5a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73815a870 con 0x7fd73815da40 2023-10-16T11:20:46.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd73cd5a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7381e6480 con 0x7fd73806afe0 2023-10-16T11:20:46.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd73cd5a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73815aa80 con 0x7fd7380cc590 2023-10-16T11:20:46.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd73815da40 0x7fd7380c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd73815da40 0x7fd7380c92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:40920/0 (socket says 172.21.15.158:40920) 2023-10-16T11:20:46.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 -- 172.21.15.158:0/1781935711 learned_addr learned my addr 172.21.15.158:0/1781935711 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:46.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73806afe0 0x7fd738159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd735d9b700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7380cc590 0x7fd73815c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:46.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7380cc590 msgr2=0x7fd73815c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7380cc590 0x7fd73815c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73806afe0 msgr2=0x7fd738159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:46.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73806afe0 0x7fd738159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:46.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 -- 172.21.15.158:0/1781935711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd730009b30 con 0x7fd73815da40 2023-10-16T11:20:46.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd736d9d700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd73815da40 0x7fd7380c92b0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7fd7380702f0 tx=0x7fd730004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:46.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.857+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730004800 con 0x7fd73815da40 2023-10-16T11:20:46.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.858+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730004960 con 0x7fd73815da40 2023-10-16T11:20:46.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.858+0000 7fd73cd5a700 1 -- 172.21.15.158:0/1781935711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73815ad60 con 0x7fd73815da40 2023-10-16T11:20:46.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.858+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73001c680 con 0x7fd73815da40 2023-10-16T11:20:46.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.858+0000 7fd73cd5a700 1 -- 172.21.15.158:0/1781935711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73815c930 con 0x7fd73815da40 2023-10-16T11:20:46.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.859+0000 7fd73cd5a700 1 -- 172.21.15.158:0/1781935711 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7381e6630 con 0x7fd73815da40 2023-10-16T11:20:46.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.859+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 148) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fd73001c7e0 con 0x7fd73815da40 2023-10-16T11:20:46.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.860+0000 7fd726ffd700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 0x7fd71806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:46.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.860+0000 7fd73659c700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 msgr2=0x7fd71806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:46.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.860+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fd73008be90 con 0x7fd73815da40 2023-10-16T11:20:46.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.860+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 0x7fd71806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:46.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:46.864+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7381e6630 con 0x7fd73815da40 2023-10-16T11:20:46.874 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.873+0000 7f4d00bf9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:20:46.902 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.900+0000 7f1b19634200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:20:46.951 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:46.949+0000 7f4d00bf9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:20:46.981 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:46.979+0000 7f1b19634200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:20:47.032 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:47.031+0000 7f4d00bf9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:20:47.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:47.061+0000 7fd73659c700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 msgr2=0x7fd71806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:47.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:47.061+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 0x7fd71806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:47.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:47.089+0000 7fd73cd5a700 1 -- 172.21.15.158:0/1781935711 --> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd73815abf0 con 0x7fd71806b190 2023-10-16T11:20:47.351 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:47.350+0000 7f1b19634200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:20:47.405 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:47.404+0000 7f4d00bf9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:20:47.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:47.462+0000 7fd73659c700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 msgr2=0x7fd71806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:47.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:47.462+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 0x7fd71806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T11:20:47.921 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:47.919+0000 7f1b19634200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:20:47.987 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:47.985+0000 7f4d00bf9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:20:47.990 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:47.989+0000 7f1b19634200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:20:48.057 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:48.055+0000 7f4d00bf9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:20:48.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:48.263+0000 7fd73659c700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 msgr2=0x7fd71806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:48.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:48.263+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 0x7fd71806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T11:20:48.624 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:48.623+0000 7f1b19634200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:48.694 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:48.692+0000 7f4d00bf9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:48.701 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:48.699+0000 7f1b19634200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:20:48.774 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:48.773+0000 7f4d00bf9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:20:48.849 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:48.848+0000 7f1b19634200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:48.920 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:48.919+0000 7f4d00bf9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:49.078 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:49.077+0000 7f1b19634200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:20:49.149 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:49.148+0000 7f4d00bf9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:20:49.152 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:49.151+0000 7f1b19634200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:20:49.223 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:49.222+0000 7f4d00bf9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:20:49.418 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:49.417+0000 7f1b19634200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:20:49.499 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:49.497+0000 7f4d00bf9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:20:49.640 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:49.639+0000 7f1b19634200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:20:49.723 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:49.721+0000 7f4d00bf9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:20:49.792 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:49.792+0000 7f1b19634200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:20:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:49.864+0000 7fd73659c700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 msgr2=0x7fd71806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3408161846 2023-10-16T11:20:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:49.864+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 0x7fd71806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T11:20:49.877 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:49.875+0000 7f4d00bf9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:20:49.906 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:49.906+0000 7f1b19634200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:20:49.991 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:49.989+0000 7f4d00bf9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:20:50.063 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:50.063+0000 7f1b19634200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:20:50.149 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:50.147+0000 7f4d00bf9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:20:50.327 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:50.326+0000 7f1b19634200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:50.414 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:50.413+0000 7f4d00bf9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:50.488 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:50.488+0000 7f1b19634200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:20:50.576 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:50.575+0000 7f4d00bf9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:20:51.017 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:51.016+0000 7f1b19634200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:20:51.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:51.085+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 149) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fd7300568b0 con 0x7fd73815da40 2023-10-16T11:20:51.112 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.110+0000 7f4d00bf9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:20:51.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:51.135+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 150) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fd730023360 con 0x7fd73815da40 2023-10-16T11:20:51.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:51.135+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 msgr2=0x7fd71806d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:51.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:51.135+0000 7fd726ffd700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3408161846,v1:172.21.15.158:6833/3408161846] conn(0x7fd71806b190 0x7fd71806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:51.371 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.372 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.372 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.372 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.372 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.373 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.373 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.373 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.373 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.373 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.369+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.381 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.379+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.382 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.379+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.382 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.379+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.382 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.379+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.382 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.379+0000 7f4c763d9700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.382 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.380+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.383 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.380+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.383 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.380+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.383 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.380+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:51.383 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:51.380+0000 7f4c733d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:52.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.140+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 151) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd730057180 con 0x7fd73815da40 2023-10-16T11:20:52.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.140+0000 7fd726ffd700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fd71801cc80 0x7fd71801d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.140+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 --> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd73815abf0 con 0x7fd71801cc80 2023-10-16T11:20:52.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.143+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fd71801cc80 0x7fd71801d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.143+0000 7fd73659c700 1 --2- 172.21.15.158:0/1781935711 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fd71801cc80 0x7fd71801d060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd728007fd0 tx=0x7fd728007ec0 comp rx=0 tx=0).ready entity=mgr.39569 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:52.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.146+0000 7fd726ffd700 1 -- 172.21.15.158:0/1781935711 <== mgr.39569 v2:172.21.15.158:6832/3687571003 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7fd73815abf0 con 0x7fd71801cc80 2023-10-16T11:20:52.148 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:52.146+0000 7f4c7d3e7700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2023-10-16T11:20:52.150 INFO:teuthology.orchestra.run.smithi158.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2023-10-16T11:20:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T11:20:52.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:52.156 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T11:20:52.177 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:20:52.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.342+0000 7ff8c4635700 1 Processor -- start 2023-10-16T11:20:52.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.342+0000 7ff8c4635700 1 -- start start 2023-10-16T11:20:52.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.343+0000 7ff8c4635700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff8bc143790 0x7ff8bc0665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.343+0000 7ff8c4635700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc06aa70 con 0x7ff8bc06bb10 2023-10-16T11:20:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.343+0000 7ff8c4635700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc06ad50 con 0x7ff8bc0c57b0 2023-10-16T11:20:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.343+0000 7ff8c4635700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8bc143b70 con 0x7ff8bc143790 2023-10-16T11:20:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.343+0000 7ff8c1bd0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff8bc06bb10 0x7ff8bc0c50a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41324/0 (socket says 172.21.15.158:41324) 2023-10-16T11:20:52.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.343+0000 7ff8c1bd0700 1 -- 172.21.15.158:0/2404704406 learned_addr learned my addr 172.21.15.158:0/2404704406 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c23d1700 1 --2- 172.21.15.158:0/2404704406 >> v2:172.21.15.172:3300/0 conn(0x7ff8bc143790 0x7ff8bc0665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363273897 0 0) 0x7ff8bc06aa70 con 0x7ff8bc06bb10 2023-10-16T11:20:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a80035b0 con 0x7ff8bc06bb10 2023-10-16T11:20:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834097354 0 0) 0x7ff8a80035b0 con 0x7ff8bc06bb10 2023-10-16T11:20:52.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 >> v2:172.21.15.172:3300/0 conn(0x7ff8bc143790 msgr2=0x7ff8bc0665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:52.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 --2- 172.21.15.158:0/2404704406 >> v2:172.21.15.172:3300/0 conn(0x7ff8bc143790 0x7ff8bc0665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:52.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 >> v1:172.21.15.172:6789/0 conn(0x7ff8bc0c57b0 legacy=0x7ff8bc06e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:52.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc06b030 con 0x7ff8bc06bb10 2023-10-16T11:20:52.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff8b8001670 con 0x7ff8bc06bb10 2023-10-16T11:20:52.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 >> v1:172.21.15.158:6789/0 conn(0x7ff8bc06bb10 legacy=0x7ff8bc0c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:52.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a8003a00 0x7ff8a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a8004400 0x7ff8a80048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a80050d0 0x7ff8a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc143b70 con 0x7ff8a80050d0 2023-10-16T11:20:52.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a800c030 con 0x7ff8a8003a00 2023-10-16T11:20:52.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a800c1f0 con 0x7ff8a8004400 2023-10-16T11:20:52.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c2bd2700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a80050d0 0x7ff8a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c23d1700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a8003a00 0x7ff8a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c1bd0700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a8004400 0x7ff8a80048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c2bd2700 1 -- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a8004400 msgr2=0x7ff8a80048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:52.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c2bd2700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a8004400 0x7ff8a80048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:52.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c2bd2700 1 -- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a8003a00 msgr2=0x7ff8a8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:52.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c2bd2700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a8003a00 0x7ff8a8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:52.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c2bd2700 1 -- 172.21.15.158:0/2404704406 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a80035b0 con 0x7ff8a80050d0 2023-10-16T11:20:52.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.344+0000 7ff8c2bd2700 1 -- 172.21.15.158:0/2404704406 reap_dead start 2023-10-16T11:20:52.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c2bd2700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a80050d0 0x7ff8a800ba10 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4002220 tx=0x7ff8a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4b83e5fc54285e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:52.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400cd70 con 0x7ff8a80050d0 2023-10-16T11:20:52.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b4003a00 con 0x7ff8a80050d0 2023-10-16T11:20:52.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c13cf700 1 -- 172.21.15.158:0/2404704406 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4003db0 con 0x7ff8a80050d0 2023-10-16T11:20:52.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c4635700 1 -- 172.21.15.158:0/2404704406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a80050d0 msgr2=0x7ff8a800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:52.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a80050d0 0x7ff8a800ba10 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4002220 tx=0x7ff8a8005880 comp rx=0 tx=0).stop 2023-10-16T11:20:52.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c4635700 1 -- 172.21.15.158:0/2404704406 shutdown_connections 2023-10-16T11:20:52.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2404704406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a80050d0 0x7ff8a800ba10 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:52.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c4635700 1 -- 172.21.15.158:0/2404704406 >> 172.21.15.158:0/2404704406 conn(0x7ff8bc0cc5b0 msgr2=0x7ff8bc0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:52.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c4635700 1 -- 172.21.15.158:0/2404704406 shutdown_connections 2023-10-16T11:20:52.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.345+0000 7ff8c4635700 1 -- 172.21.15.158:0/2404704406 wait complete. 2023-10-16T11:20:52.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 Processor -- start 2023-10-16T11:20:52.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 -- start start 2023-10-16T11:20:52.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a80050d0 0x7ff8bc13e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc15c810 0x7ff8bc0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0b4a70 0x7ff8bc0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0c7df0 con 0x7ff8bc15c810 2023-10-16T11:20:52.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0b5430 con 0x7ff8bc0b4a70 2023-10-16T11:20:52.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c4635700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0b2a00 con 0x7ff8a80050d0 2023-10-16T11:20:52.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c1bd0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc15c810 0x7ff8bc0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c1bd0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc15c810 0x7ff8bc0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41162/0 (socket says 172.21.15.158:41162) 2023-10-16T11:20:52.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.347+0000 7ff8c1bd0700 1 -- 172.21.15.158:0/2604103167 learned_addr learned my addr 172.21.15.158:0/2604103167 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:52.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c2bd2700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0b4a70 0x7ff8bc0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c23d1700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a80050d0 0x7ff8bc13e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c1bd0700 1 -- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a80050d0 msgr2=0x7ff8bc13e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:52.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c1bd0700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a80050d0 0x7ff8bc13e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:52.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c1bd0700 1 -- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0b4a70 msgr2=0x7ff8bc0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:52.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c1bd0700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0b4a70 0x7ff8bc0b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:52.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c1bd0700 1 -- 172.21.15.158:0/2604103167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc0c7550 con 0x7ff8bc15c810 2023-10-16T11:20:52.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c1bd0700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc15c810 0x7ff8bc0c6c00 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7ff8b80072f0 tx=0x7ff8b8007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:52.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8b2ffd700 1 -- 172.21.15.158:0/2604103167 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b800f5e0 con 0x7ff8bc15c810 2023-10-16T11:20:52.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8b2ffd700 1 -- 172.21.15.158:0/2604103167 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b800f760 con 0x7ff8bc15c810 2023-10-16T11:20:52.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8b2ffd700 1 -- 172.21.15.158:0/2604103167 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b800fb10 con 0x7ff8bc15c810 2023-10-16T11:20:52.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc0c7830 con 0x7ff8bc15c810 2023-10-16T11:20:52.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.348+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc13ee30 con 0x7ff8bc15c810 2023-10-16T11:20:52.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.349+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8bc0681b0 con 0x7ff8bc15c810 2023-10-16T11:20:52.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.349+0000 7ff8b2ffd700 1 -- 172.21.15.158:0/2604103167 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 151) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff8b800fc70 con 0x7ff8bc15c810 2023-10-16T11:20:52.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.350+0000 7ff8b2ffd700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7ff8a006ada0 0x7ff8a006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:52.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.350+0000 7ff8c23d1700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7ff8a006ada0 0x7ff8a006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:52.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.350+0000 7ff8b2ffd700 1 -- 172.21.15.158:0/2604103167 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7ff8b8089030 con 0x7ff8bc15c810 2023-10-16T11:20:52.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.350+0000 7ff8c23d1700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7ff8a006ada0 0x7ff8a006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009b20 tx=0x7ff8ac008040 comp rx=0 tx=0).ready entity=mgr.39569 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:52.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.355+0000 7ff8b2ffd700 1 -- 172.21.15.158:0/2604103167 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8b8053bf0 con 0x7ff8bc15c810 2023-10-16T11:20:52.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:52.606+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff8bc0b27d0 con 0x7ff8bc15c810 2023-10-16T11:20:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.140+0000 7ff8b2ffd700 1 -- 172.21.15.158:0/2604103167 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v152) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff8b801a070 con 0x7ff8bc15c810 2023-10-16T11:20:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7ff8a006ada0 msgr2=0x7ff8a006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7ff8a006ada0 0x7ff8a006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009b20 tx=0x7ff8ac008040 comp rx=0 tx=0).stop 2023-10-16T11:20:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc15c810 msgr2=0x7ff8bc0c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc15c810 0x7ff8bc0c6c00 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7ff8b80072f0 tx=0x7ff8b8007600 comp rx=0 tx=0).stop 2023-10-16T11:20:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 shutdown_connections 2023-10-16T11:20:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7ff8a006ada0 0x7ff8a006d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8bc0b4a70 0x7ff8bc0b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a80050d0 0x7ff8bc13e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 --2- 172.21.15.158:0/2604103167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc15c810 0x7ff8bc0c6c00 unknown :-1 s=CLOSED pgs=2959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 >> 172.21.15.158:0/2604103167 conn(0x7ff8bc0cc5b0 msgr2=0x7ff8bc0af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.144+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 shutdown_connections 2023-10-16T11:20:53.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.145+0000 7ff8c4635700 1 -- 172.21.15.158:0/2604103167 wait complete. 2023-10-16T11:20:53.163 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:20:53.163 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:20:53.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:20:53.252 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:20:53.252 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:20:53.253 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:53.270 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:20:53.270 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:53.290 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:53.290+0000 7f40a087f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:20:53.300 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:53.299+0000 7f8c33cac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:20:53.318 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:53.357 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:53.356+0000 7f40a087f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:53.368 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:53.367+0000 7f8c33cac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:53.429 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:53.428+0000 7f40a087f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:20:53.443 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:53.442+0000 7f8c33cac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:20:53.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.480+0000 7fccbebc2700 1 Processor -- start 2023-10-16T11:20:53.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.480+0000 7fccbebc2700 1 -- start start 2023-10-16T11:20:53.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbebc2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80ce530 0x7fccb80b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbebc2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fccb815f0f0 0x7fccb80cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbebc2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb815b930 con 0x7fccb80cbfe0 2023-10-16T11:20:53.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbebc2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb80cc490 con 0x7fccb815f0f0 2023-10-16T11:20:53.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbebc2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80cf020 con 0x7fccb80ce530 2023-10-16T11:20:53.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccb7fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fccb815f0f0 0x7fccb80cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbc95e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80ce530 0x7fccb80b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbd15f700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fccb80cbfe0 0x7fccb80d13e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:45294/0 (socket says 172.21.15.158:45294) 2023-10-16T11:20:53.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbd15f700 1 -- 172.21.15.158:0/2862324365 learned_addr learned my addr 172.21.15.158:0/2862324365 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:53.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbc95e700 1 -- 172.21.15.158:0/2862324365 >> v2:172.21.15.172:3300/0 conn(0x7fccb815f0f0 msgr2=0x7fccb80cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:53.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbc95e700 1 --2- 172.21.15.158:0/2862324365 >> v2:172.21.15.172:3300/0 conn(0x7fccb815f0f0 0x7fccb80cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbc95e700 1 -- 172.21.15.158:0/2862324365 >> v1:172.21.15.172:6789/0 conn(0x7fccb80cbfe0 legacy=0x7fccb80d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccbc95e700 1 -- 172.21.15.158:0/2862324365 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb815f5f0 con 0x7fccb80ce530 2023-10-16T11:20:53.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.481+0000 7fccb7fff700 1 --2- 172.21.15.158:0/2862324365 >> v2:172.21.15.172:3300/0 conn(0x7fccb815f0f0 0x7fccb80cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:53.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbc95e700 1 --2- 172.21.15.158:0/2862324365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80ce530 0x7fccb80b3b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fccac009840 tx=0x7fccac009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=444e0dbb2f97f5b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:53.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccb77fe700 1 -- 172.21.15.158:0/2862324365 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac011830 con 0x7fccb80ce530 2023-10-16T11:20:53.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccb77fe700 1 -- 172.21.15.158:0/2862324365 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac011990 con 0x7fccb80ce530 2023-10-16T11:20:53.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccb77fe700 1 -- 172.21.15.158:0/2862324365 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac011d20 con 0x7fccb80ce530 2023-10-16T11:20:53.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 -- 172.21.15.158:0/2862324365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80ce530 msgr2=0x7fccb80b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 --2- 172.21.15.158:0/2862324365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80ce530 0x7fccb80b3b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fccac009840 tx=0x7fccac009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:53.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 -- 172.21.15.158:0/2862324365 shutdown_connections 2023-10-16T11:20:53.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 --2- 172.21.15.158:0/2862324365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80ce530 0x7fccb80b3b20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 --2- 172.21.15.158:0/2862324365 >> v2:172.21.15.172:3300/0 conn(0x7fccb815f0f0 0x7fccb80cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 -- 172.21.15.158:0/2862324365 >> 172.21.15.158:0/2862324365 conn(0x7fccb806a8d0 msgr2=0x7fccb806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:53.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 -- 172.21.15.158:0/2862324365 shutdown_connections 2023-10-16T11:20:53.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.482+0000 7fccbebc2700 1 -- 172.21.15.158:0/2862324365 wait complete. 2023-10-16T11:20:53.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.484+0000 7fccbebc2700 1 Processor -- start 2023-10-16T11:20:53.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.484+0000 7fccbebc2700 1 -- start start 2023-10-16T11:20:53.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.484+0000 7fccbebc2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80cbfe0 0x7fccb80c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.484+0000 7fccbebc2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccb80ce530 0x7fccb8158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.484+0000 7fccbebc2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb815f0f0 0x7fccb80c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbebc2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80c5b90 con 0x7fccb80ce530 2023-10-16T11:20:53.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbebc2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb81590a0 con 0x7fccb815f0f0 2023-10-16T11:20:53.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbebc2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80cb210 con 0x7fccb80cbfe0 2023-10-16T11:20:53.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccb80ce530 0x7fccb8158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccb80ce530 0x7fccb8158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41216/0 (socket says 172.21.15.158:41216) 2023-10-16T11:20:53.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 -- 172.21.15.158:0/1044677073 learned_addr learned my addr 172.21.15.158:0/1044677073 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:53.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbd15f700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb815f0f0 0x7fccb80c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbc95e700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80cbfe0 0x7fccb80c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 -- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80cbfe0 msgr2=0x7fccb80c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80cbfe0 0x7fccb80c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 -- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb815f0f0 msgr2=0x7fccb80c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb815f0f0 0x7fccb80c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 -- 172.21.15.158:0/1044677073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40094a0 con 0x7fccb80ce530 2023-10-16T11:20:53.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbd15f700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb815f0f0 0x7fccb80c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:53.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb7fff700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccb80ce530 0x7fccb8158b60 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7fcca400ebe0 tx=0x7fcca400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:53.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb57fa700 1 -- 172.21.15.158:0/1044677073 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca400c850 con 0x7fccb80ce530 2023-10-16T11:20:53.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb57fa700 1 -- 172.21.15.158:0/1044677073 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca400c9b0 con 0x7fccb80ce530 2023-10-16T11:20:53.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac0094a0 con 0x7fccb80ce530 2023-10-16T11:20:53.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccb57fa700 1 -- 172.21.15.158:0/1044677073 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca400cd40 con 0x7fccb80ce530 2023-10-16T11:20:53.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.485+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb80cb7d0 con 0x7fccb80ce530 2023-10-16T11:20:53.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.486+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb80cb9f0 con 0x7fccb80ce530 2023-10-16T11:20:53.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.487+0000 7fccb57fa700 1 -- 172.21.15.158:0/1044677073 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 152) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fcca4013470 con 0x7fccb80ce530 2023-10-16T11:20:53.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.488+0000 7fccb57fa700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 0x7fcc9c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.488+0000 7fccbc95e700 1 -- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 msgr2=0x7fcc9c06d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:53.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.488+0000 7fccb57fa700 1 -- 172.21.15.158:0/1044677073 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7fcca4089a70 con 0x7fccb80ce530 2023-10-16T11:20:53.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.488+0000 7fccbc95e700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 0x7fcc9c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:53.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.493+0000 7fccb57fa700 1 -- 172.21.15.158:0/1044677073 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcca4054630 con 0x7fccb80ce530 2023-10-16T11:20:53.507 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:53.507+0000 7f40a087f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:20:53.528 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:53.526+0000 7f8c33cac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:20:53.649 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:53.649+0000 7f40a087f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:20:53.672 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:53.670+0000 7f8c33cac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:20:53.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.689+0000 7fccbc95e700 1 -- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 msgr2=0x7fcc9c06d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:53.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.689+0000 7fccbc95e700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 0x7fcc9c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:53.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.692+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb80681b0 con 0x7fccb80ce530 2023-10-16T11:20:53.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.693+0000 7fccb57fa700 1 -- 172.21.15.158:0/1044677073 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) v1 ==== 74+0+21822 (secure 0 0 0) 0x7fcca4017020 con 0x7fccb80ce530 2023-10-16T11:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":242,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:51.125732+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:53.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 msgr2=0x7fcc9c06d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:53.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 0x7fcc9c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccb80ce530 msgr2=0x7fccb8158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccb80ce530 0x7fccb8158b60 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7fcca400ebe0 tx=0x7fcca400eef0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 shutdown_connections 2023-10-16T11:20:53.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fcc9c06ad50 0x7fcc9c06d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb815f0f0 0x7fccb80c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb80cbfe0 0x7fccb80c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.695+0000 7fccbebc2700 1 --2- 172.21.15.158:0/1044677073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccb80ce530 0x7fccb8158b60 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.696+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 >> 172.21.15.158:0/1044677073 conn(0x7fccb806a8d0 msgr2=0x7fccb815bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:53.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.696+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 shutdown_connections 2023-10-16T11:20:53.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.696+0000 7fccbebc2700 1 -- 172.21.15.158:0/1044677073 wait complete. 2023-10-16T11:20:53.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:20:23.549102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:20:24.086127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '237', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T11:20:35.862082+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '240', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T11:20:37.182868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '241', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:20:53.711 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:20:53.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.876+0000 7f3850850700 1 Processor -- start 2023-10-16T11:20:53.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.876+0000 7f3850850700 1 -- start start 2023-10-16T11:20:53.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.877+0000 7f3850850700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f384c0d0c40 0x7f384c0b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.877+0000 7f3850850700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0b7c90 con 0x7f384c160d30 2023-10-16T11:20:53.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.877+0000 7f3850850700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0b7f70 con 0x7f384c0d29b0 2023-10-16T11:20:53.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.877+0000 7f3850850700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384c0b8910 con 0x7f384c0d0c40 2023-10-16T11:20:53.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.877+0000 7f384a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f384c160d30 0x7f384c0b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41418/0 (socket says 172.21.15.158:41418) 2023-10-16T11:20:53.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.877+0000 7f384a59c700 1 -- 172.21.15.158:0/2896942119 learned_addr learned my addr 172.21.15.158:0/2896942119 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:53.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.877+0000 7f384ad9d700 1 --2- 172.21.15.158:0/2896942119 >> v2:172.21.15.172:3300/0 conn(0x7f384c0d0c40 0x7f384c0b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698629439 0 0) 0x7f384c0b7c90 con 0x7f384c160d30 2023-10-16T11:20:53.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38340035b0 con 0x7f384c160d30 2023-10-16T11:20:53.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559517966 0 0) 0x7f384c0b7f70 con 0x7f384c0d29b0 2023-10-16T11:20:53.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f384c0b7c90 con 0x7f384c0d29b0 2023-10-16T11:20:53.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384ad9d700 1 -- 172.21.15.158:0/2896942119 >> v1:172.21.15.172:6789/0 conn(0x7f384c0d29b0 legacy=0x7f384c0d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384ad9d700 1 -- 172.21.15.158:0/2896942119 >> v1:172.21.15.158:6789/0 conn(0x7f384c160d30 legacy=0x7f384c0b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912213563 0 0) 0x7f38340035b0 con 0x7f384c160d30 2023-10-16T11:20:53.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384ad9d700 1 -- 172.21.15.158:0/2896942119 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c0b8250 con 0x7f384c0d0c40 2023-10-16T11:20:53.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384ad9d700 1 --2- 172.21.15.158:0/2896942119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d0c40 0x7f384c0b7590 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3844009d90 tx=0x7f384400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62cff8867c4c57fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:53.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384400e070 con 0x7f384c0d0c40 2023-10-16T11:20:53.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3844003a20 con 0x7f384c0d0c40 2023-10-16T11:20:53.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.878+0000 7f384959a700 1 -- 172.21.15.158:0/2896942119 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c001670 con 0x7f384c0d0c40 2023-10-16T11:20:53.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.879+0000 7f3850850700 1 -- 172.21.15.158:0/2896942119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d0c40 msgr2=0x7f384c0b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.879+0000 7f3850850700 1 --2- 172.21.15.158:0/2896942119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d0c40 0x7f384c0b7590 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3844009d90 tx=0x7f384400b330 comp rx=0 tx=0).stop 2023-10-16T11:20:53.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.879+0000 7f3850850700 1 -- 172.21.15.158:0/2896942119 shutdown_connections 2023-10-16T11:20:53.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.879+0000 7f3850850700 1 --2- 172.21.15.158:0/2896942119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d0c40 0x7f384c0b7590 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.879+0000 7f3850850700 1 -- 172.21.15.158:0/2896942119 >> 172.21.15.158:0/2896942119 conn(0x7f384c0669f0 msgr2=0x7f384c066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:53.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.879+0000 7f3850850700 1 -- 172.21.15.158:0/2896942119 shutdown_connections 2023-10-16T11:20:53.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.879+0000 7f3850850700 1 -- 172.21.15.158:0/2896942119 wait complete. 2023-10-16T11:20:53.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.880+0000 7f3850850700 1 Processor -- start 2023-10-16T11:20:53.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.880+0000 7f3850850700 1 -- start start 2023-10-16T11:20:53.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f3850850700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f384c0d0c40 0x7f384c0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f3850850700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d29b0 0x7f384c14bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f3850850700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f384c160d30 0x7f384c153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f3850850700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c0c7f30 con 0x7f384c160d30 2023-10-16T11:20:53.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f3850850700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c0c7970 con 0x7f384c0d29b0 2023-10-16T11:20:53.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f3850850700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384c0c7c50 con 0x7f384c0d0c40 2023-10-16T11:20:53.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f384c160d30 0x7f384c153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f384c160d30 0x7f384c153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41252/0 (socket says 172.21.15.158:41252) 2023-10-16T11:20:53.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 -- 172.21.15.158:0/3642447187 learned_addr learned my addr 172.21.15.158:0/3642447187 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:53.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384a59c700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f384c0d0c40 0x7f384c0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f3849d9b700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d29b0 0x7f384c14bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:53.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 -- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f384c0d0c40 msgr2=0x7f384c0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f384c0d0c40 0x7f384c0ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 -- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d29b0 msgr2=0x7f384c14bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:53.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d29b0 0x7f384c14bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:53.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.881+0000 7f384ad9d700 1 -- 172.21.15.158:0/3642447187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3844009a70 con 0x7f384c160d30 2023-10-16T11:20:53.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.882+0000 7f384ad9d700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f384c160d30 0x7f384c153180 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f384400b370 tx=0x7f3844004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:53.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.882+0000 7f383affd700 1 -- 172.21.15.158:0/3642447187 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844004440 con 0x7f384c160d30 2023-10-16T11:20:53.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.882+0000 7f383affd700 1 -- 172.21.15.158:0/3642447187 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38440045a0 con 0x7f384c160d30 2023-10-16T11:20:53.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.882+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384c1529a0 con 0x7f384c160d30 2023-10-16T11:20:53.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.882+0000 7f383affd700 1 -- 172.21.15.158:0/3642447187 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844004950 con 0x7f384c160d30 2023-10-16T11:20:53.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.882+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384c14c910 con 0x7f384c160d30 2023-10-16T11:20:53.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.883+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384c152ff0 con 0x7f384c160d30 2023-10-16T11:20:53.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.883+0000 7f383affd700 1 -- 172.21.15.158:0/3642447187 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 152) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f384401d470 con 0x7f384c160d30 2023-10-16T11:20:53.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.884+0000 7f383affd700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 0x7f382c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:53.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.884+0000 7f384a59c700 1 -- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 msgr2=0x7f382c06d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:53.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.884+0000 7f383affd700 1 -- 172.21.15.158:0/3642447187 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f384408b670 con 0x7f384c160d30 2023-10-16T11:20:53.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.884+0000 7f384a59c700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 0x7f382c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:53.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:53.888+0000 7f383affd700 1 -- 172.21.15.158:0/3642447187 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3844059c80 con 0x7f384c160d30 2023-10-16T11:20:53.978 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:53.977+0000 7f40a087f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:20:54.016 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:54.014+0000 7f8c33cac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:20:54.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.085+0000 7f384a59c700 1 -- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 msgr2=0x7f382c06d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:54.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.085+0000 7f384a59c700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 0x7f382c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:54.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.087+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f384c0681b0 con 0x7f384c160d30 2023-10-16T11:20:54.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.088+0000 7f383affd700 1 -- 172.21.15.158:0/3642447187 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v242) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3844021020 con 0x7f384c160d30 2023-10-16T11:20:54.090 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:20:54.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 msgr2=0x7f382c06d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:54.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 0x7f382c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f384c160d30 msgr2=0x7f384c153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f384c160d30 0x7f384c153180 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f384400b370 tx=0x7f3844004ad0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 shutdown_connections 2023-10-16T11:20:54.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f382c06ad50 0x7f382c06d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f384c160d30 0x7f384c153180 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f384c0d0c40 0x7f384c0ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 --2- 172.21.15.158:0/3642447187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f384c0d29b0 0x7f384c14bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 >> 172.21.15.158:0/3642447187 conn(0x7f384c0669f0 msgr2=0x7f384c0d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:54.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 shutdown_connections 2023-10-16T11:20:54.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.091+0000 7f3850850700 1 -- 172.21.15.158:0/3642447187 wait complete. 2023-10-16T11:20:54.095 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:54.092+0000 7f40a087f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:20:54.106 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:20:54.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:54.136+0000 7f8c33cac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:20:54.171 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:54.171+0000 7f40a087f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:20:54.217 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:54.216+0000 7f8c33cac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:20:54.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.235+0000 7fa20afee700 1 Processor -- start 2023-10-16T11:20:54.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.235+0000 7fa20afee700 1 -- start start 2023-10-16T11:20:54.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa20afee700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa20406afe0 0x7fa204161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa20afee700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20415ee60 con 0x7fa20415b540 2023-10-16T11:20:54.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa20afee700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20415e2e0 con 0x7fa20406f180 2023-10-16T11:20:54.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa20afee700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20415b920 con 0x7fa20406afe0 2023-10-16T11:20:54.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa20958b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa20406afe0 0x7fa204161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa208d8a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa20415b540 0x7fa20406d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41434/0 (socket says 172.21.15.158:41434) 2023-10-16T11:20:54.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa208d8a700 1 -- 172.21.15.158:0/1966174414 learned_addr learned my addr 172.21.15.158:0/1966174414 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:54.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.236+0000 7fa20958b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa20406afe0 0x7fa204161860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41262/0 (socket says 172.21.15.158:41262) 2023-10-16T11:20:54.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa20958b700 1 -- 172.21.15.158:0/1966174414 >> v1:172.21.15.172:6789/0 conn(0x7fa20406f180 legacy=0x7fa20406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa20958b700 1 -- 172.21.15.158:0/1966174414 >> v1:172.21.15.158:6789/0 conn(0x7fa20415b540 legacy=0x7fa20406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa20958b700 1 -- 172.21.15.158:0/1966174414 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20415e5c0 con 0x7fa20406afe0 2023-10-16T11:20:54.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa2037fe700 1 -- 172.21.15.158:0/1966174414 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799464381 0 0) 0x7fa20415ee60 con 0x7fa20415b540 2023-10-16T11:20:54.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa20958b700 1 --2- 172.21.15.158:0/1966174414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa204161860 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009d90 tx=0x7fa1fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58f6715dd36d5c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:54.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa2037fe700 1 -- 172.21.15.158:0/1966174414 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc00e070 con 0x7fa20406afe0 2023-10-16T11:20:54.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa2037fe700 1 -- 172.21.15.158:0/1966174414 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc00bbd0 con 0x7fa20406afe0 2023-10-16T11:20:54.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa2037fe700 1 -- 172.21.15.158:0/1966174414 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc012430 con 0x7fa20406afe0 2023-10-16T11:20:54.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa20afee700 1 -- 172.21.15.158:0/1966174414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 msgr2=0x7fa204161860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.237+0000 7fa20afee700 1 --2- 172.21.15.158:0/1966174414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa204161860 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009d90 tx=0x7fa1fc00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.238+0000 7fa20afee700 1 -- 172.21.15.158:0/1966174414 shutdown_connections 2023-10-16T11:20:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.238+0000 7fa20afee700 1 --2- 172.21.15.158:0/1966174414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa204161860 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.238+0000 7fa20afee700 1 -- 172.21.15.158:0/1966174414 >> 172.21.15.158:0/1966174414 conn(0x7fa2040d03b0 msgr2=0x7fa2040d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.238+0000 7fa20afee700 1 -- 172.21.15.158:0/1966174414 shutdown_connections 2023-10-16T11:20:54.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.238+0000 7fa20afee700 1 -- 172.21.15.158:0/1966174414 wait complete. 2023-10-16T11:20:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.239+0000 7fa20afee700 1 Processor -- start 2023-10-16T11:20:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.239+0000 7fa20afee700 1 -- start start 2023-10-16T11:20:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20afee700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa20414ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20afee700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa20406f180 0x7fa20415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20afee700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa20415b540 0x7fa20415ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20afee700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20415d340 con 0x7fa20406afe0 2023-10-16T11:20:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20afee700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040ac890 con 0x7fa20415b540 2023-10-16T11:20:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20afee700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040ad410 con 0x7fa20406f180 2023-10-16T11:20:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa20414ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa20414ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41264/0 (socket says 172.21.15.158:41264) 2023-10-16T11:20:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 -- 172.21.15.158:0/2824552362 learned_addr learned my addr 172.21.15.158:0/2824552362 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20958b700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa20415b540 0x7fa20415ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa203fff700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa20406f180 0x7fa20415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 -- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa20406f180 msgr2=0x7fa20415c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa20406f180 0x7fa20415c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 -- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa20415b540 msgr2=0x7fa20415ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa20415b540 0x7fa20415ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 -- 172.21.15.158:0/2824552362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc009a70 con 0x7fa20406afe0 2023-10-16T11:20:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa208d8a700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa20414ae40 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800c520 tx=0x7fa1f800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa2017fa700 1 -- 172.21.15.158:0/2824552362 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8004c80 con 0x7fa20406afe0 2023-10-16T11:20:54.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa2017fa700 1 -- 172.21.15.158:0/2824552362 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f8004de0 con 0x7fa20406afe0 2023-10-16T11:20:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2040acae0 con 0x7fa20406afe0 2023-10-16T11:20:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.240+0000 7fa2017fa700 1 -- 172.21.15.158:0/2824552362 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8007310 con 0x7fa20406afe0 2023-10-16T11:20:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.241+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2040ad0a0 con 0x7fa20406afe0 2023-10-16T11:20:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.241+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2040ce310 con 0x7fa20406afe0 2023-10-16T11:20:54.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.242+0000 7fa2017fa700 1 -- 172.21.15.158:0/2824552362 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fa1f8007470 con 0x7fa20406afe0 2023-10-16T11:20:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.242+0000 7fa2017fa700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 0x7fa1e806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.242+0000 7fa203fff700 1 -- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 msgr2=0x7fa1e806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.242+0000 7fa2017fa700 1 -- 172.21.15.158:0/2824552362 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7fa1f8089d70 con 0x7fa20406afe0 2023-10-16T11:20:54.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.242+0000 7fa203fff700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 0x7fa1e806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:54.252 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:54.251+0000 7f40a087f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:20:54.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.247+0000 7fa2017fa700 1 -- 172.21.15.158:0/2824552362 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1f80542a0 con 0x7fa20406afe0 2023-10-16T11:20:54.303 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:54.301+0000 7f8c33cac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:20:54.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.443+0000 7fa203fff700 1 -- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 msgr2=0x7fa1e806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:54.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.443+0000 7fa203fff700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 0x7fa1e806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:54.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.446+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa2040681b0 con 0x7fa20406afe0 2023-10-16T11:20:54.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.447+0000 7fa2017fa700 1 -- 172.21.15.158:0/2824552362 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v242) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1f80578c0 con 0x7fa20406afe0 2023-10-16T11:20:54.449 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:54.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 msgr2=0x7fa1e806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:54.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 0x7fa1e806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 msgr2=0x7fa20414ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa20414ae40 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800c520 tx=0x7fa1f800c830 comp rx=0 tx=0).stop 2023-10-16T11:20:54.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 shutdown_connections 2023-10-16T11:20:54.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fa1e806b140 0x7fa1e806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa20415b540 0x7fa20415ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa20406afe0 0x7fa20414ae40 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 --2- 172.21.15.158:0/2824552362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa20406f180 0x7fa20415c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 >> 172.21.15.158:0/2824552362 conn(0x7fa2040d03b0 msgr2=0x7fa20406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:54.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 shutdown_connections 2023-10-16T11:20:54.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.449+0000 7fa20afee700 1 -- 172.21.15.158:0/2824552362 wait complete. 2023-10-16T11:20:54.463 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:20:54.622 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:54.621+0000 7f40a087f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:20:54.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.629+0000 7f02fcae5700 1 Processor -- start 2023-10-16T11:20:54.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.629+0000 7f02fcae5700 1 -- start start 2023-10-16T11:20:54.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02fcae5700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f02f8160d30 0x7f02f80b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02fcae5700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02f80d0530 0x7f02f80d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02fcae5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f80d19a0 0x7f02f80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02fcae5700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f80b8910 con 0x7f02f80d0530 2023-10-16T11:20:54.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02fcae5700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f80d0ef0 con 0x7f02f8160d30 2023-10-16T11:20:54.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02fcae5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80b7c90 con 0x7f02f80d19a0 2023-10-16T11:20:54.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02f5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02f80d0530 0x7f02f80d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02f5d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f02f80d0530 0x7f02f80d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41296/0 (socket says 172.21.15.158:41296) 2023-10-16T11:20:54.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.630+0000 7f02f5d9b700 1 -- 172.21.15.158:0/1756020186 learned_addr learned my addr 172.21.15.158:0/1756020186 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:54.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f659c700 1 --2- 172.21.15.158:0/1756020186 >> v2:172.21.15.172:3300/0 conn(0x7f02f8160d30 0x7f02f80b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f6d9d700 1 --2- 172.21.15.158:0/1756020186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f80d19a0 0x7f02f80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f5d9b700 1 -- 172.21.15.158:0/1756020186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f80d19a0 msgr2=0x7f02f80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:54.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f5d9b700 1 --2- 172.21.15.158:0/1756020186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f80d19a0 0x7f02f80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f5d9b700 1 -- 172.21.15.158:0/1756020186 >> v2:172.21.15.172:3300/0 conn(0x7f02f8160d30 msgr2=0x7f02f80b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:54.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f5d9b700 1 --2- 172.21.15.158:0/1756020186 >> v2:172.21.15.172:3300/0 conn(0x7f02f8160d30 0x7f02f80b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f5d9b700 1 -- 172.21.15.158:0/1756020186 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f80b7f70 con 0x7f02f80d0530 2023-10-16T11:20:54.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f659c700 1 --2- 172.21.15.158:0/1756020186 >> v2:172.21.15.172:3300/0 conn(0x7f02f8160d30 0x7f02f80b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:54.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f5d9b700 1 --2- 172.21.15.158:0/1756020186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f80d09b0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7f02f80d4690 tx=0x7f02ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adab0b74ba50ac81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:54.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f559a700 1 -- 172.21.15.158:0/1756020186 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec011830 con 0x7f02f80d0530 2023-10-16T11:20:54.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f559a700 1 -- 172.21.15.158:0/1756020186 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ec011990 con 0x7f02f80d0530 2023-10-16T11:20:54.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.631+0000 7f02f559a700 1 -- 172.21.15.158:0/1756020186 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec011d20 con 0x7f02f80d0530 2023-10-16T11:20:54.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 -- 172.21.15.158:0/1756020186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 msgr2=0x7f02f80d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 --2- 172.21.15.158:0/1756020186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f80d09b0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7f02f80d4690 tx=0x7f02ec009bd0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 -- 172.21.15.158:0/1756020186 shutdown_connections 2023-10-16T11:20:54.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 --2- 172.21.15.158:0/1756020186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f80d19a0 0x7f02f80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 --2- 172.21.15.158:0/1756020186 >> v2:172.21.15.172:3300/0 conn(0x7f02f8160d30 0x7f02f80b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 --2- 172.21.15.158:0/1756020186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f80d09b0 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 -- 172.21.15.158:0/1756020186 >> 172.21.15.158:0/1756020186 conn(0x7f02f80669f0 msgr2=0x7f02f8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:54.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 -- 172.21.15.158:0/1756020186 shutdown_connections 2023-10-16T11:20:54.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.632+0000 7f02fcae5700 1 -- 172.21.15.158:0/1756020186 wait complete. 2023-10-16T11:20:54.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.633+0000 7f02fcae5700 1 Processor -- start 2023-10-16T11:20:54.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.633+0000 7f02fcae5700 1 -- start start 2023-10-16T11:20:54.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02fcae5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f8159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02fcae5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f80d19a0 0x7f02f814a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02fcae5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8160d30 0x7f02f80cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02fcae5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80c8c80 con 0x7f02f80d0530 2023-10-16T11:20:54.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02fcae5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80c8e90 con 0x7f02f80d19a0 2023-10-16T11:20:54.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02fcae5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f8159fc0 con 0x7f02f8160d30 2023-10-16T11:20:54.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02f659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f8159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02f659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f8159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41300/0 (socket says 172.21.15.158:41300) 2023-10-16T11:20:54.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02f659c700 1 -- 172.21.15.158:0/4058486927 learned_addr learned my addr 172.21.15.158:0/4058486927 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:54.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02f5d9b700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f80d19a0 0x7f02f814a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.634+0000 7f02f6d9d700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8160d30 0x7f02f80cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02f659c700 1 -- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8160d30 msgr2=0x7f02f80cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02f659c700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8160d30 0x7f02f80cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02f659c700 1 -- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f80d19a0 msgr2=0x7f02f814a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02f659c700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f80d19a0 0x7f02f814a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02f659c700 1 -- 172.21.15.158:0/4058486927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e80094a0 con 0x7f02f80d0530 2023-10-16T11:20:54.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02f659c700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f8159770 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f02e800ebe0 tx=0x7f02e800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:54.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02e6ffd700 1 -- 172.21.15.158:0/4058486927 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8009760 con 0x7f02f80d0530 2023-10-16T11:20:54.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02e6ffd700 1 -- 172.21.15.158:0/4058486927 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e800c850 con 0x7f02f80d0530 2023-10-16T11:20:54.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ec009550 con 0x7f02f80d0530 2023-10-16T11:20:54.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02e6ffd700 1 -- 172.21.15.158:0/4058486927 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e800cbe0 con 0x7f02f80d0530 2023-10-16T11:20:54.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.635+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f80c9410 con 0x7f02f80d0530 2023-10-16T11:20:54.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.636+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f80681b0 con 0x7f02f80d0530 2023-10-16T11:20:54.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.638+0000 7f02e6ffd700 1 -- 172.21.15.158:0/4058486927 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f02e800cd40 con 0x7f02f80d0530 2023-10-16T11:20:54.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.639+0000 7f02e6ffd700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 0x7f02d806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.639+0000 7f02f5d9b700 1 -- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 msgr2=0x7f02d806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:54.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.639+0000 7f02e6ffd700 1 -- 172.21.15.158:0/4058486927 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f02e808aac0 con 0x7f02f80d0530 2023-10-16T11:20:54.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.639+0000 7f02f5d9b700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 0x7f02d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:54.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.642+0000 7f02e6ffd700 1 -- 172.21.15.158:0/4058486927 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02e8054f40 con 0x7f02f80d0530 2023-10-16T11:20:54.696 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:54.695+0000 7f8c33cac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:20:54.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.840+0000 7f02f5d9b700 1 -- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 msgr2=0x7f02d806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:54.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.840+0000 7f02f5d9b700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 0x7f02d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:54.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.843+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02f81585d0 con 0x7f02f80d0530 2023-10-16T11:20:54.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.844+0000 7f02e6ffd700 1 -- 172.21.15.158:0/4058486927 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v242) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02e80210a0 con 0x7f02f80d0530 2023-10-16T11:20:54.846 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:54.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 msgr2=0x7f02d806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:54.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 0x7f02d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 msgr2=0x7f02f8159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f8159770 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f02e800ebe0 tx=0x7f02e800eef0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 shutdown_connections 2023-10-16T11:20:54.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f02d806b140 0x7f02d806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02f8160d30 0x7f02f80cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02f80d0530 0x7f02f8159770 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 --2- 172.21.15.158:0/4058486927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02f80d19a0 0x7f02f814a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 >> 172.21.15.158:0/4058486927 conn(0x7f02f80669f0 msgr2=0x7f02f80d3780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:54.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 shutdown_connections 2023-10-16T11:20:54.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.846+0000 7f02fcae5700 1 -- 172.21.15.158:0/4058486927 wait complete. 2023-10-16T11:20:54.860 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T11:20:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.983+0000 7f1cdbfe7700 1 Processor -- start 2023-10-16T11:20:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.984+0000 7f1cdbfe7700 1 -- start start 2023-10-16T11:20:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.984+0000 7f1cdbfe7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1cd415d990 0x7f1cd406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.984+0000 7f1cdbfe7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cd40cbaf0 0x7f1cd40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.984+0000 7f1cdbfe7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd415dd70 con 0x7f1cd4070300 2023-10-16T11:20:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.984+0000 7f1cdbfe7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd41616b0 con 0x7f1cd40cbaf0 2023-10-16T11:20:54.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.984+0000 7f1cdbfe7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40ced70 con 0x7f1cd415d990 2023-10-16T11:20:54.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.984+0000 7f1cda584700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cd40cbaf0 0x7f1cd40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cda584700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cd40cbaf0 0x7f1cd40cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41314/0 (socket says 172.21.15.158:41314) 2023-10-16T11:20:54.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cda584700 1 -- 172.21.15.158:0/1918049475 learned_addr learned my addr 172.21.15.158:0/1918049475 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:54.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cd9d83700 1 --2- 172.21.15.158:0/1918049475 >> v2:172.21.15.172:3300/0 conn(0x7f1cd415d990 0x7f1cd406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cda584700 1 -- 172.21.15.158:0/1918049475 >> v2:172.21.15.172:3300/0 conn(0x7f1cd415d990 msgr2=0x7f1cd406d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:54.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cda584700 1 --2- 172.21.15.158:0/1918049475 >> v2:172.21.15.172:3300/0 conn(0x7f1cd415d990 0x7f1cd406d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cda584700 1 -- 172.21.15.158:0/1918049475 >> v1:172.21.15.172:6789/0 conn(0x7f1cd4070300 legacy=0x7f1cd4161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cda584700 1 -- 172.21.15.158:0/1918049475 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4070780 con 0x7f1cd40cbaf0 2023-10-16T11:20:54.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cda584700 1 --2- 172.21.15.158:0/1918049475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cbaf0 0x7f1cd40cbf80 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40d25a0 tx=0x7f1ccc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991f2bdeb6783412 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:54.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cd8d81700 1 -- 172.21.15.158:0/1918049475 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00e070 con 0x7f1cd40cbaf0 2023-10-16T11:20:54.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cd8d81700 1 -- 172.21.15.158:0/1918049475 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc00b8d0 con 0x7f1cd40cbaf0 2023-10-16T11:20:54.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.985+0000 7f1cd8d81700 1 -- 172.21.15.158:0/1918049475 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00bc80 con 0x7f1cd40cbaf0 2023-10-16T11:20:54.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/1918049475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cbaf0 msgr2=0x7f1cd40cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/1918049475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cbaf0 0x7f1cd40cbf80 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40d25a0 tx=0x7f1ccc00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:54.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/1918049475 shutdown_connections 2023-10-16T11:20:54.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/1918049475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cbaf0 0x7f1cd40cbf80 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/1918049475 >> v2:172.21.15.172:3300/0 conn(0x7f1cd415d990 0x7f1cd406d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/1918049475 >> 172.21.15.158:0/1918049475 conn(0x7f1cd40cdf20 msgr2=0x7f1cd40ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:54.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/1918049475 shutdown_connections 2023-10-16T11:20:54.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.986+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/1918049475 wait complete. 2023-10-16T11:20:54.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.987+0000 7f1cdbfe7700 1 Processor -- start 2023-10-16T11:20:54.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.988+0000 7f1cdbfe7700 1 -- start start 2023-10-16T11:20:54.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.988+0000 7f1cdbfe7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd4070300 0x7f1cd415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.988+0000 7f1cdbfe7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cbaf0 0x7f1cd40ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.988+0000 7f1cdbfe7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd415d990 0x7f1cd40cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:54.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.988+0000 7f1cdbfe7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40b6770 con 0x7f1cd4070300 2023-10-16T11:20:54.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.988+0000 7f1cdbfe7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40b6f10 con 0x7f1cd415d990 2023-10-16T11:20:54.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.988+0000 7f1cdbfe7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd415ace0 con 0x7f1cd40cbaf0 2023-10-16T11:20:54.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd4070300 0x7f1cd415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd4070300 0x7f1cd415a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41316/0 (socket says 172.21.15.158:41316) 2023-10-16T11:20:54.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 -- 172.21.15.158:0/69103205 learned_addr learned my addr 172.21.15.158:0/69103205 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:54.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9582700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cbaf0 0x7f1cd40ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cda584700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd415d990 0x7f1cd40cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:54.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 -- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cbaf0 msgr2=0x7f1cd40ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cbaf0 0x7f1cd40ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 -- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd415d990 msgr2=0x7f1cd40cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:54.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd415d990 0x7f1cd40cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:54.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 -- 172.21.15.158:0/69103205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc009b30 con 0x7f1cd4070300 2023-10-16T11:20:54.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cd9d83700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd4070300 0x7f1cd415a7a0 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000e8a0 tx=0x7f1cd000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:54.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cca7fc700 1 -- 172.21.15.158:0/69103205 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0009590 con 0x7f1cd4070300 2023-10-16T11:20:54.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cca7fc700 1 -- 172.21.15.158:0/69103205 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd0013680 con 0x7f1cd4070300 2023-10-16T11:20:54.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd415e810 con 0x7f1cd4070300 2023-10-16T11:20:54.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cca7fc700 1 -- 172.21.15.158:0/69103205 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00139f0 con 0x7f1cd4070300 2023-10-16T11:20:55.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.989+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd40cb1a0 con 0x7f1cd4070300 2023-10-16T11:20:55.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.990+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd40681b0 con 0x7f1cd4070300 2023-10-16T11:20:55.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.991+0000 7f1cca7fc700 1 -- 172.21.15.158:0/69103205 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f1cd0013b50 con 0x7f1cd4070300 2023-10-16T11:20:55.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.991+0000 7f1cca7fc700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 0x7f1cb806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.991+0000 7f1cd9582700 1 -- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 msgr2=0x7f1cb806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:55.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.991+0000 7f1cca7fc700 1 -- 172.21.15.158:0/69103205 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f1cd008a010 con 0x7f1cd4070300 2023-10-16T11:20:55.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.991+0000 7f1cd9582700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 0x7f1cb806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:55.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:54.996+0000 7f1cca7fc700 1 -- 172.21.15.158:0/69103205 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cd00545c0 con 0x7f1cd4070300 2023-10-16T11:20:55.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.192+0000 7f1cd9582700 1 -- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 msgr2=0x7f1cb806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:55.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.192+0000 7f1cd9582700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 0x7f1cb806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:55.199 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:55.197+0000 7f40a087f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:20:55.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.244+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1cd40b7420 con 0x7f1cd4070300 2023-10-16T11:20:55.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.244+0000 7f1cca7fc700 1 -- 172.21.15.158:0/69103205 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v242) v1 ==== 107+0+11 (secure 0 0 0) 0x7f1cd0021020 con 0x7f1cd4070300 2023-10-16T11:20:55.246 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:55.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 msgr2=0x7f1cb806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:55.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 0x7f1cb806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd4070300 msgr2=0x7f1cd415a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd4070300 0x7f1cd415a7a0 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000e8a0 tx=0x7f1cd000ebb0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 shutdown_connections 2023-10-16T11:20:55.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cb806b140 0x7f1cb806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd415d990 0x7f1cd40cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd4070300 0x7f1cd415a7a0 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 --2- 172.21.15.158:0/69103205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cbaf0 0x7f1cd40ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 >> 172.21.15.158:0/69103205 conn(0x7f1cd40cdf20 msgr2=0x7f1cd40b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:55.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 shutdown_connections 2023-10-16T11:20:55.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.247+0000 7f1cdbfe7700 1 -- 172.21.15.158:0/69103205 wait complete. 2023-10-16T11:20:55.263 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T11:20:55.269 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:55.268+0000 7f40a087f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:20:55.322 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:55.321+0000 7f8c33cac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:20:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.388+0000 7fef84f8b700 1 Processor -- start 2023-10-16T11:20:55.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.388+0000 7fef84f8b700 1 -- start start 2023-10-16T11:20:55.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef84f8b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fef8015d990 0x7fef8006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef84f8b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fef80070300 0x7fef80161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef84f8b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8015dd70 con 0x7fef80162010 2023-10-16T11:20:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef84f8b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef801616b0 con 0x7fef8015d990 2023-10-16T11:20:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef84f8b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef800ced70 con 0x7fef80070300 2023-10-16T11:20:55.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef7e59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fef8015d990 0x7fef8006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef7e59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fef8015d990 0x7fef8006d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41326/0 (socket says 172.21.15.158:41326) 2023-10-16T11:20:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.389+0000 7fef7e59c700 1 -- 172.21.15.158:0/670613021 learned_addr learned my addr 172.21.15.158:0/670613021 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7dd9b700 1 --2- 172.21.15.158:0/670613021 >> v2:172.21.15.172:3300/0 conn(0x7fef80070300 0x7fef80161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7e59c700 1 -- 172.21.15.158:0/670613021 >> v2:172.21.15.172:3300/0 conn(0x7fef80070300 msgr2=0x7fef80161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:55.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7e59c700 1 --2- 172.21.15.158:0/670613021 >> v2:172.21.15.172:3300/0 conn(0x7fef80070300 0x7fef80161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7e59c700 1 -- 172.21.15.158:0/670613021 >> v1:172.21.15.172:6789/0 conn(0x7fef80162010 legacy=0x7fef800cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7e59c700 1 -- 172.21.15.158:0/670613021 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef80070780 con 0x7fef8015d990 2023-10-16T11:20:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7e59c700 1 --2- 172.21.15.158:0/670613021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef8015d990 0x7fef8006d5d0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7fef800c4bb0 tx=0x7fef70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9bfab3c860ae4d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7d59a700 1 -- 172.21.15.158:0/670613021 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70004230 con 0x7fef8015d990 2023-10-16T11:20:55.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7d59a700 1 -- 172.21.15.158:0/670613021 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef70004390 con 0x7fef8015d990 2023-10-16T11:20:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.390+0000 7fef7d59a700 1 -- 172.21.15.158:0/670613021 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700036a0 con 0x7fef8015d990 2023-10-16T11:20:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 -- 172.21.15.158:0/670613021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef8015d990 msgr2=0x7fef8006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 --2- 172.21.15.158:0/670613021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef8015d990 0x7fef8006d5d0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7fef800c4bb0 tx=0x7fef70009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 -- 172.21.15.158:0/670613021 shutdown_connections 2023-10-16T11:20:55.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 --2- 172.21.15.158:0/670613021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef8015d990 0x7fef8006d5d0 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 --2- 172.21.15.158:0/670613021 >> v2:172.21.15.172:3300/0 conn(0x7fef80070300 0x7fef80161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 -- 172.21.15.158:0/670613021 >> 172.21.15.158:0/670613021 conn(0x7fef800cdf20 msgr2=0x7fef800ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 -- 172.21.15.158:0/670613021 shutdown_connections 2023-10-16T11:20:55.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.391+0000 7fef84f8b700 1 -- 172.21.15.158:0/670613021 wait complete. 2023-10-16T11:20:55.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.392+0000 7fef84f8b700 1 Processor -- start 2023-10-16T11:20:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.392+0000 7fef84f8b700 1 -- start start 2023-10-16T11:20:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef84f8b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef80070300 0x7fef800b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef84f8b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef8015d990 0x7fef800b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef84f8b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef80162010 0x7fef80152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef84f8b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef800b5ca0 con 0x7fef80070300 2023-10-16T11:20:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef84f8b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef800b5650 con 0x7fef80162010 2023-10-16T11:20:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef84f8b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef800b1960 con 0x7fef8015d990 2023-10-16T11:20:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef80070300 0x7fef800b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef80070300 0x7fef800b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41332/0 (socket says 172.21.15.158:41332) 2023-10-16T11:20:55.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 -- 172.21.15.158:0/1098765010 learned_addr learned my addr 172.21.15.158:0/1098765010 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7dd9b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef8015d990 0x7fef800b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7ed9d700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef80162010 0x7fef80152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 -- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef8015d990 msgr2=0x7fef800b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef8015d990 0x7fef800b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 -- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef80162010 msgr2=0x7fef80152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef80162010 0x7fef80152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 -- 172.21.15.158:0/1098765010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef70009550 con 0x7fef80070300 2023-10-16T11:20:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.393+0000 7fef7e59c700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef80070300 0x7fef800b60c0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fef800675e0 tx=0x7fef70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.394+0000 7fef6effd700 1 -- 172.21.15.158:0/1098765010 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70003da0 con 0x7fef80070300 2023-10-16T11:20:55.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.394+0000 7fef6effd700 1 -- 172.21.15.158:0/1098765010 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef70011630 con 0x7fef80070300 2023-10-16T11:20:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.394+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef800b58c0 con 0x7fef80070300 2023-10-16T11:20:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.394+0000 7fef6effd700 1 -- 172.21.15.158:0/1098765010 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700119e0 con 0x7fef80070300 2023-10-16T11:20:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.394+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef800b1c40 con 0x7fef80070300 2023-10-16T11:20:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.394+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef800b5de0 con 0x7fef80070300 2023-10-16T11:20:55.405 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:55.395+0000 7f8c33cac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:20:55.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.395+0000 7fef6effd700 1 -- 172.21.15.158:0/1098765010 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fef7001b440 con 0x7fef80070300 2023-10-16T11:20:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.396+0000 7fef6effd700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 0x7fef6006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.396+0000 7fef7dd9b700 1 -- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 msgr2=0x7fef6006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.396+0000 7fef6effd700 1 -- 172.21.15.158:0/1098765010 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7fef7008b9e0 con 0x7fef80070300 2023-10-16T11:20:55.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.396+0000 7fef7dd9b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 0x7fef6006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:55.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.400+0000 7fef6effd700 1 -- 172.21.15.158:0/1098765010 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef70056180 con 0x7fef80070300 2023-10-16T11:20:55.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.597+0000 7fef7dd9b700 1 -- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 msgr2=0x7fef6006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:55.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.597+0000 7fef7dd9b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 0x7fef6006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:55.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.601+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fef8015dd70 con 0x7fef80070300 2023-10-16T11:20:55.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.601+0000 7fef6effd700 1 -- 172.21.15.158:0/1098765010 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v242) v1 ==== 103+0+11 (secure 0 0 0) 0x7fef8015dd70 con 0x7fef80070300 2023-10-16T11:20:55.603 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:55.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 msgr2=0x7fef6006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:55.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 0x7fef6006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef80070300 msgr2=0x7fef800b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef80070300 0x7fef800b60c0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fef800675e0 tx=0x7fef70004750 comp rx=0 tx=0).stop 2023-10-16T11:20:55.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 shutdown_connections 2023-10-16T11:20:55.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fef6006b140 0x7fef6006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef80162010 0x7fef80152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef80070300 0x7fef800b60c0 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 --2- 172.21.15.158:0/1098765010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fef8015d990 0x7fef800b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 >> 172.21.15.158:0/1098765010 conn(0x7fef800cdf20 msgr2=0x7fef8015ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:55.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 shutdown_connections 2023-10-16T11:20:55.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.605+0000 7fef84f8b700 1 -- 172.21.15.158:0/1098765010 wait complete. 2023-10-16T11:20:55.624 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:20:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.745+0000 7f7579117700 1 Processor -- start 2023-10-16T11:20:55.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.745+0000 7f7579117700 1 -- start start 2023-10-16T11:20:55.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7579117700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757415b650 0x7f757415fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7579117700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f75740b4320 0x7f75740b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7579117700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740cf550 con 0x7f75740b47a0 2023-10-16T11:20:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7579117700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75740cc6d0 con 0x7f75740b4320 2023-10-16T11:20:55.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7579117700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75740cf2b0 con 0x7f757415b650 2023-10-16T11:20:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7572d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757415b650 0x7f757415fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f757259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f75740b4320 0x7f75740b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7572d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757415b650 0x7f757415fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:52176/0 (socket says 172.21.15.158:52176) 2023-10-16T11:20:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.746+0000 7f7572d9d700 1 -- 172.21.15.158:0/773583751 learned_addr learned my addr 172.21.15.158:0/773583751 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:55.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7572d9d700 1 -- 172.21.15.158:0/773583751 >> v2:172.21.15.172:3300/0 conn(0x7f75740b4320 msgr2=0x7f75740b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:55.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7572d9d700 1 --2- 172.21.15.158:0/773583751 >> v2:172.21.15.172:3300/0 conn(0x7f75740b4320 0x7f75740b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7572d9d700 1 -- 172.21.15.158:0/773583751 >> v1:172.21.15.172:6789/0 conn(0x7f75740b47a0 legacy=0x7f75740cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7572d9d700 1 -- 172.21.15.158:0/773583751 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75741600f0 con 0x7f757415b650 2023-10-16T11:20:55.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f757259c700 1 --2- 172.21.15.158:0/773583751 >> v2:172.21.15.172:3300/0 conn(0x7f75740b4320 0x7f75740b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:55.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7572d9d700 1 --2- 172.21.15.158:0/773583751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757415b650 0x7f757415fbb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f75640098f0 tx=0x7f7564009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1aa3312f3594684 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:55.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7571d9b700 1 -- 172.21.15.158:0/773583751 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7564011830 con 0x7f757415b650 2023-10-16T11:20:55.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7571d9b700 1 -- 172.21.15.158:0/773583751 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7564011990 con 0x7f757415b650 2023-10-16T11:20:55.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.747+0000 7f7571d9b700 1 -- 172.21.15.158:0/773583751 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7564011d20 con 0x7f757415b650 2023-10-16T11:20:55.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 -- 172.21.15.158:0/773583751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757415b650 msgr2=0x7f757415fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 --2- 172.21.15.158:0/773583751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757415b650 0x7f757415fbb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f75640098f0 tx=0x7f7564009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:55.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 -- 172.21.15.158:0/773583751 shutdown_connections 2023-10-16T11:20:55.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 --2- 172.21.15.158:0/773583751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f757415b650 0x7f757415fbb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 --2- 172.21.15.158:0/773583751 >> v2:172.21.15.172:3300/0 conn(0x7f75740b4320 0x7f75740b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 -- 172.21.15.158:0/773583751 >> 172.21.15.158:0/773583751 conn(0x7f75740cdf50 msgr2=0x7f75740ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:55.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 -- 172.21.15.158:0/773583751 shutdown_connections 2023-10-16T11:20:55.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.748+0000 7f7579117700 1 -- 172.21.15.158:0/773583751 wait complete. 2023-10-16T11:20:55.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.750+0000 7f7579117700 1 Processor -- start 2023-10-16T11:20:55.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.750+0000 7f7579117700 1 -- start start 2023-10-16T11:20:55.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.750+0000 7f7579117700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75740b4320 0x7f757406c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.750+0000 7f7579117700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740b47a0 0x7f757414c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f7579117700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415b650 0x7f757414c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f7579117700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757406da10 con 0x7f757415b650 2023-10-16T11:20:55.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f7579117700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757406ce00 con 0x7f75740b4320 2023-10-16T11:20:55.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f7579117700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757406d0e0 con 0x7f75740b47a0 2023-10-16T11:20:55.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415b650 0x7f757414c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415b650 0x7f757414c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41352/0 (socket says 172.21.15.158:41352) 2023-10-16T11:20:55.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 -- 172.21.15.158:0/1280605001 learned_addr learned my addr 172.21.15.158:0/1280605001 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:55.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f7572d9d700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75740b4320 0x7f757406c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757259c700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740b47a0 0x7f757414c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:55.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 -- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740b47a0 msgr2=0x7f757414c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740b47a0 0x7f757414c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 -- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75740b4320 msgr2=0x7f757406c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75740b4320 0x7f757406c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 -- 172.21.15.158:0/1280605001 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75680094a0 con 0x7f757415b650 2023-10-16T11:20:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f7572d9d700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75740b4320 0x7f757406c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:55.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757259c700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740b47a0 0x7f757414c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f757359e700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415b650 0x7f757414c980 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f756c00cc50 tx=0x7f756c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f755b7fe700 1 -- 172.21.15.158:0/1280605001 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c003c00 con 0x7f757415b650 2023-10-16T11:20:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f755b7fe700 1 -- 172.21.15.158:0/1280605001 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c003d60 con 0x7f757415b650 2023-10-16T11:20:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.751+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7564009550 con 0x7f757415b650 2023-10-16T11:20:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.752+0000 7f755b7fe700 1 -- 172.21.15.158:0/1280605001 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c00aab0 con 0x7f757415b650 2023-10-16T11:20:55.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.752+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757406d6a0 con 0x7f757415b650 2023-10-16T11:20:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.753+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75740681b0 con 0x7f757415b650 2023-10-16T11:20:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.753+0000 7f755b7fe700 1 -- 172.21.15.158:0/1280605001 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f756c00ac10 con 0x7f757415b650 2023-10-16T11:20:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.754+0000 7f755b7fe700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 0x7f756006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.754+0000 7f7572d9d700 1 -- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 msgr2=0x7f756006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:55.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.754+0000 7f755b7fe700 1 -- 172.21.15.158:0/1280605001 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f756c08a210 con 0x7f757415b650 2023-10-16T11:20:55.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.754+0000 7f7572d9d700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 0x7f756006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:55.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.758+0000 7f755b7fe700 1 -- 172.21.15.158:0/1280605001 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f756c054730 con 0x7f757415b650 2023-10-16T11:20:55.904 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:55.902+0000 7f40a087f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:55.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.955+0000 7f7572d9d700 1 -- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 msgr2=0x7f756006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:55.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.955+0000 7f7572d9d700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 0x7f756006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:55.981 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:55.980+0000 7f40a087f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:20:55.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.989+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f75740cc3e0 con 0x7f757415b650 2023-10-16T11:20:55.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.989+0000 7f755b7fe700 1 -- 172.21.15.158:0/1280605001 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v242) v1 ==== 81+0+592 (secure 0 0 0) 0x7f756c057d50 con 0x7f757415b650 2023-10-16T11:20:55.991 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2550010748 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.991 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2248613235 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.991 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1739296604 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.991 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3563325503 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.992 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/940033591 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.992 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/3408161846 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.992 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/3408161846 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.992 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3480048181 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.992 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/689546029 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.992 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/67584808 2023-10-17T11:20:51.125668+0000 2023-10-16T11:20:55.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 msgr2=0x7f756006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:55.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 0x7f756006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415b650 msgr2=0x7f757414c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:55.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415b650 0x7f757414c980 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f756c00cc50 tx=0x7f756c00cf60 comp rx=0 tx=0).stop 2023-10-16T11:20:55.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 shutdown_connections 2023-10-16T11:20:55.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f756006b140 0x7f756006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f757415b650 0x7f757414c980 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75740b4320 0x7f757406c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 --2- 172.21.15.158:0/1280605001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75740b47a0 0x7f757414c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:55.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 >> 172.21.15.158:0/1280605001 conn(0x7f75740cdf50 msgr2=0x7f757415e120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:55.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 shutdown_connections 2023-10-16T11:20:55.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:55.991+0000 7f7579117700 1 -- 172.21.15.158:0/1280605001 wait complete. 2023-10-16T11:20:55.996 INFO:teuthology.orchestra.run.smithi158.stderr:listed 10 entries 2023-10-16T11:20:56.006 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:20:56.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:56.006 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:20:56.053 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:20:56.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:20:56.054 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:20:56.070 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:20:56.071 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:56.085 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:56.084+0000 7f8c33cac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:56.089 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:20:56.089 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:20:56.089 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:56.106 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:20:56.107 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:20:56.107 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:20:56.131 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:56.130+0000 7f40a087f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:56.147 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:20:56.147 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:56.168 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:56.167+0000 7f8c33cac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:20:56.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.269+0000 7f76af940700 1 Processor -- start 2023-10-16T11:20:56.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.270+0000 7f76af940700 1 -- start start 2023-10-16T11:20:56.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.270+0000 7f76af940700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f76a80d1210 0x7f76a80ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.270+0000 7f76af940700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f76a80ce950 0x7f76a80b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.270+0000 7f76af940700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a80d3bd0 con 0x7f76a815d7c0 2023-10-16T11:20:56.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.270+0000 7f76af940700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a80d4060 con 0x7f76a80ce950 2023-10-16T11:20:56.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.270+0000 7f76af940700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a80d2fc0 con 0x7f76a80d1210 2023-10-16T11:20:56.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f76a80ce950 0x7f76a80b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f76a80ce950 0x7f76a80b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41378/0 (socket says 172.21.15.158:41378) 2023-10-16T11:20:56.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 -- 172.21.15.158:0/2290405285 learned_addr learned my addr 172.21.15.158:0/2290405285 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:56.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76ad6dc700 1 --1- 172.21.15.158:0/2290405285 >> v1:172.21.15.158:6789/0 conn(0x7f76a815d7c0 0x7f76a80d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41550/0 (socket says 172.21.15.158:41550) 2023-10-16T11:20:56.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76acedb700 1 --2- 172.21.15.158:0/2290405285 >> v2:172.21.15.172:3300/0 conn(0x7f76a80d1210 0x7f76a80ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 -- 172.21.15.158:0/2290405285 >> v2:172.21.15.172:3300/0 conn(0x7f76a80d1210 msgr2=0x7f76a80ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:56.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 --2- 172.21.15.158:0/2290405285 >> v2:172.21.15.172:3300/0 conn(0x7f76a80d1210 0x7f76a80ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 -- 172.21.15.158:0/2290405285 >> v1:172.21.15.158:6789/0 conn(0x7f76a815d7c0 legacy=0x7f76a80d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f769ffff700 1 -- 172.21.15.158:0/2290405285 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244447574 0 0) 0x7f76a80d3bd0 con 0x7f76a815d7c0 2023-10-16T11:20:56.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 -- 172.21.15.158:0/2290405285 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a80d32a0 con 0x7f76a80ce950 2023-10-16T11:20:56.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f76adedd700 1 --2- 172.21.15.158:0/2290405285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80b6800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f76a80b9750 tx=0x7f76a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f324c1e917de13e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:56.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f769ffff700 1 -- 172.21.15.158:0/2290405285 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a000e070 con 0x7f76a80ce950 2023-10-16T11:20:56.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f769ffff700 1 -- 172.21.15.158:0/2290405285 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a000b910 con 0x7f76a80ce950 2023-10-16T11:20:56.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.271+0000 7f769ffff700 1 -- 172.21.15.158:0/2290405285 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a000bcc0 con 0x7f76a80ce950 2023-10-16T11:20:56.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 -- 172.21.15.158:0/2290405285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 msgr2=0x7f76a80b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 --2- 172.21.15.158:0/2290405285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80b6800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f76a80b9750 tx=0x7f76a000b330 comp rx=0 tx=0).stop 2023-10-16T11:20:56.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 -- 172.21.15.158:0/2290405285 shutdown_connections 2023-10-16T11:20:56.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 --2- 172.21.15.158:0/2290405285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80b6800 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 --2- 172.21.15.158:0/2290405285 >> v2:172.21.15.172:3300/0 conn(0x7f76a80d1210 0x7f76a80ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 -- 172.21.15.158:0/2290405285 >> 172.21.15.158:0/2290405285 conn(0x7f76a80669f0 msgr2=0x7f76a8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:56.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 -- 172.21.15.158:0/2290405285 shutdown_connections 2023-10-16T11:20:56.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.272+0000 7f76af940700 1 -- 172.21.15.158:0/2290405285 wait complete. 2023-10-16T11:20:56.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.273+0000 7f76af940700 1 Processor -- start 2023-10-16T11:20:56.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.273+0000 7f76af940700 1 -- start start 2023-10-16T11:20:56.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.274+0000 7f76af940700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.274+0000 7f76af940700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76a80d1210 0x7f76a81606c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76af940700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76a815d7c0 0x7f76a8159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76af940700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a8160c00 con 0x7f76a80ce950 2023-10-16T11:20:56.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76af940700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a8159790 con 0x7f76a815d7c0 2023-10-16T11:20:56.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76af940700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a815ac80 con 0x7f76a80d1210 2023-10-16T11:20:56.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41380/0 (socket says 172.21.15.158:41380) 2023-10-16T11:20:56.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 -- 172.21.15.158:0/1709396931 learned_addr learned my addr 172.21.15.158:0/1709396931 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:56.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76adedd700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76a815d7c0 0x7f76a8159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76acedb700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76a80d1210 0x7f76a81606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 -- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76a80d1210 msgr2=0x7f76a81606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76a80d1210 0x7f76a81606c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 -- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76a815d7c0 msgr2=0x7f76a8159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76a815d7c0 0x7f76a8159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 -- 172.21.15.158:0/1709396931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a0009b30 con 0x7f76a80ce950 2023-10-16T11:20:56.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76ad6dc700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80c80f0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f76a400c550 tx=0x7f76a400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:56.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f769dffb700 1 -- 172.21.15.158:0/1709396931 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4004c80 con 0x7f76a80ce950 2023-10-16T11:20:56.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f769dffb700 1 -- 172.21.15.158:0/1709396931 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4004e00 con 0x7f76a80ce950 2023-10-16T11:20:56.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a80b4c20 con 0x7f76a80ce950 2023-10-16T11:20:56.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.275+0000 7f769dffb700 1 -- 172.21.15.158:0/1709396931 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4007310 con 0x7f76a80ce950 2023-10-16T11:20:56.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.276+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a815a6c0 con 0x7f76a80ce950 2023-10-16T11:20:56.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.277+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a80681b0 con 0x7f76a80ce950 2023-10-16T11:20:56.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.277+0000 7f769dffb700 1 -- 172.21.15.158:0/1709396931 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f76a4007470 con 0x7f76a80ce950 2023-10-16T11:20:56.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.277+0000 7f769dffb700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 0x7f768c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.277+0000 7f76acedb700 1 -- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 msgr2=0x7f768c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:56.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.277+0000 7f769dffb700 1 -- 172.21.15.158:0/1709396931 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f76a4089d60 con 0x7f76a80ce950 2023-10-16T11:20:56.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.277+0000 7f76acedb700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 0x7f768c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:56.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.282+0000 7f769dffb700 1 -- 172.21.15.158:0/1709396931 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76a4054310 con 0x7f76a80ce950 2023-10-16T11:20:56.334 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:56.333+0000 7f8c33cac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:20:56.366 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:56.364+0000 7f40a087f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:20:56.440 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:56.438+0000 7f40a087f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:20:56.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.478+0000 7f76acedb700 1 -- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 msgr2=0x7f768c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:56.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.478+0000 7f76acedb700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 0x7f768c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:56.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.541+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76a815aa60 con 0x7f76a80ce950 2023-10-16T11:20:56.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.541+0000 7f769dffb700 1 -- 172.21.15.158:0/1709396931 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6160 (secure 0 0 0) 0x7f76a405ce60 con 0x7f76a80ce950 2023-10-16T11:20:56.543 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:56.544 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:20:56.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 msgr2=0x7f768c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:56.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 0x7f768c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 msgr2=0x7f76a80c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80c80f0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f76a400c550 tx=0x7f76a400c860 comp rx=0 tx=0).stop 2023-10-16T11:20:56.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 shutdown_connections 2023-10-16T11:20:56.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f768c06b140 0x7f768c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76a815d7c0 0x7f76a8159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76a80ce950 0x7f76a80c80f0 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 --2- 172.21.15.158:0/1709396931 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76a80d1210 0x7f76a81606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 >> 172.21.15.158:0/1709396931 conn(0x7f76a80669f0 msgr2=0x7f76a80d26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:56.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 shutdown_connections 2023-10-16T11:20:56.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.544+0000 7f76af940700 1 -- 172.21.15.158:0/1709396931 wait complete. 2023-10-16T11:20:56.548 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 134 2023-10-16T11:20:56.558 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:56.568 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:56.567+0000 7f8c33cac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:20:56.650 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:56.648+0000 7f8c33cac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:20:56.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.690+0000 7f372dd0e700 1 Processor -- start 2023-10-16T11:20:56.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.691+0000 7f372dd0e700 1 -- start start 2023-10-16T11:20:56.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.691+0000 7f372dd0e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37280d1210 0x7f37280ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.691+0000 7f372dd0e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.691+0000 7f372dd0e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280d3bd0 con 0x7f372815d7c0 2023-10-16T11:20:56.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.691+0000 7f372dd0e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37280d4060 con 0x7f37280d1210 2023-10-16T11:20:56.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.691+0000 7f372dd0e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37280cedc0 con 0x7f37280ce950 2023-10-16T11:20:56.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3726ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37280d1210 0x7f37280ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3727fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f37277fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f372815d7c0 0x7f37280d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:45470/0 (socket says 172.21.15.158:45470) 2023-10-16T11:20:56.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f37277fe700 1 -- 172.21.15.158:0/4078001832 learned_addr learned my addr 172.21.15.158:0/4078001832 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:56.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3726ffd700 1 -- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 msgr2=0x7f37280b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:56.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3726ffd700 1 --2- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3726ffd700 1 -- 172.21.15.158:0/4078001832 >> v1:172.21.15.172:6789/0 conn(0x7f372815d7c0 legacy=0x7f37280d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3726ffd700 1 -- 172.21.15.158:0/4078001832 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280d2fc0 con 0x7f37280d1210 2023-10-16T11:20:56.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f37267fc700 1 -- 172.21.15.158:0/4078001832 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623780068 0 0) 0x7f37280d3bd0 con 0x7f372815d7c0 2023-10-16T11:20:56.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3727fff700 1 --2- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:56.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f3726ffd700 1 --2- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37280d1210 0x7f37280ce410 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f371c009840 tx=0x7f371c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a940d6a69a97fa6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:56.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f37267fc700 1 -- 172.21.15.158:0/4078001832 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c011830 con 0x7f37280d1210 2023-10-16T11:20:56.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f37267fc700 1 -- 172.21.15.158:0/4078001832 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c011990 con 0x7f37280d1210 2023-10-16T11:20:56.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.692+0000 7f37267fc700 1 -- 172.21.15.158:0/4078001832 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c011d20 con 0x7f37280d1210 2023-10-16T11:20:56.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 -- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37280d1210 msgr2=0x7f37280ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 --2- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37280d1210 0x7f37280ce410 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f371c009840 tx=0x7f371c009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:56.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 -- 172.21.15.158:0/4078001832 shutdown_connections 2023-10-16T11:20:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 --2- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 --2- 172.21.15.158:0/4078001832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37280d1210 0x7f37280ce410 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 -- 172.21.15.158:0/4078001832 >> 172.21.15.158:0/4078001832 conn(0x7f37280669f0 msgr2=0x7f3728066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 -- 172.21.15.158:0/4078001832 shutdown_connections 2023-10-16T11:20:56.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.693+0000 7f372dd0e700 1 -- 172.21.15.158:0/4078001832 wait complete. 2023-10-16T11:20:56.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.694+0000 7f372dd0e700 1 Processor -- start 2023-10-16T11:20:56.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.694+0000 7f372dd0e700 1 -- start start 2023-10-16T11:20:56.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f372dd0e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f372dd0e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37280d1210 0x7f37280aa120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f372dd0e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f372815d7c0 0x7f37280aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f372dd0e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372814d0c0 con 0x7f37280d1210 2023-10-16T11:20:56.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f372dd0e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372814c440 con 0x7f372815d7c0 2023-10-16T11:20:56.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f372dd0e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372814c720 con 0x7f37280ce950 2023-10-16T11:20:56.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f3726ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37280d1210 0x7f37280aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f3726ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37280d1210 0x7f37280aa120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41398/0 (socket says 172.21.15.158:41398) 2023-10-16T11:20:56.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f3726ffd700 1 -- 172.21.15.158:0/339032851 learned_addr learned my addr 172.21.15.158:0/339032851 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:56.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f37277fe700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.695+0000 7f3727fff700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f372815d7c0 0x7f37280aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:56.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f3726ffd700 1 -- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 msgr2=0x7f37280cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f3726ffd700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f3726ffd700 1 -- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f372815d7c0 msgr2=0x7f37280aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f3726ffd700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f372815d7c0 0x7f37280aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f3726ffd700 1 -- 172.21.15.158:0/339032851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3720009480 con 0x7f37280d1210 2023-10-16T11:20:56.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f3726ffd700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37280d1210 0x7f37280aa120 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f371c009fd0 tx=0x7f371c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:56.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f370ffff700 1 -- 172.21.15.158:0/339032851 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c0045d0 con 0x7f37280d1210 2023-10-16T11:20:56.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f370ffff700 1 -- 172.21.15.158:0/339032851 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c01e070 con 0x7f37280d1210 2023-10-16T11:20:56.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f370ffff700 1 -- 172.21.15.158:0/339032851 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c0038e0 con 0x7f37280d1210 2023-10-16T11:20:56.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371c0094a0 con 0x7f37280d1210 2023-10-16T11:20:56.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.696+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372814cce0 con 0x7f37280d1210 2023-10-16T11:20:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.697+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37280681b0 con 0x7f37280d1210 2023-10-16T11:20:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.698+0000 7f370ffff700 1 -- 172.21.15.158:0/339032851 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f371c003a40 con 0x7f37280d1210 2023-10-16T11:20:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.698+0000 7f370ffff700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.698+0000 7f37277fe700 1 -- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 msgr2=0x7f371406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.698+0000 7f370ffff700 1 -- 172.21.15.158:0/339032851 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f371c08b090 con 0x7f37280d1210 2023-10-16T11:20:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.698+0000 7f37277fe700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:56.709 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:56.705+0000 7f40a087f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:20:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.703+0000 7f370ffff700 1 -- 172.21.15.158:0/339032851 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f371c055640 con 0x7f37280d1210 2023-10-16T11:20:56.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.899+0000 7f37277fe700 1 -- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 msgr2=0x7f371406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:56.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.899+0000 7f37277fe700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:56.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.908+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37280cb7c0 con 0x7f37280d1210 2023-10-16T11:20:56.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.909+0000 7f370ffff700 1 -- 172.21.15.158:0/339032851 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) v1 ==== 74+0+21822 (secure 0 0 0) 0x7f371c015090 con 0x7f37280d1210 2023-10-16T11:20:56.911 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:56.912 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":242,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:51.125732+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:56.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.911+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 msgr2=0x7f371406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:56.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.911+0000 7f372dd0e700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.911+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37280d1210 msgr2=0x7f37280aa120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:56.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.911+0000 7f372dd0e700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37280d1210 0x7f37280aa120 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f371c009fd0 tx=0x7f371c004750 comp rx=0 tx=0).stop 2023-10-16T11:20:56.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 shutdown_connections 2023-10-16T11:20:56.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f372815d7c0 0x7f37280aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37280ce950 0x7f37280cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 --2- 172.21.15.158:0/339032851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37280d1210 0x7f37280aa120 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:56.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 >> 172.21.15.158:0/339032851 conn(0x7f37280669f0 msgr2=0x7f37280b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:56.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 shutdown_connections 2023-10-16T11:20:56.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:56.912+0000 7f372dd0e700 1 -- 172.21.15.158:0/339032851 wait complete. 2023-10-16T11:20:56.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:20:23.549102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:20:24.086127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '237', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T11:20:35.862082+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '240', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T11:20:37.182868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '241', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:20:56.927 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:20:56.932 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:56.930+0000 7f8c33cac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:20:56.932 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:56.931+0000 7f40a087f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:20:57.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.052+0000 7feea255c700 1 Processor -- start 2023-10-16T11:20:57.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7feea255c700 1 -- start start 2023-10-16T11:20:57.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7feea255c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee9c15b6b0 0x7fee9c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7feea255c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee9c0b3630 0x7fee9c0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7feea255c700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c0b4420 con 0x7fee9c0d0c60 2023-10-16T11:20:57.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7feea255c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c15ba90 con 0x7fee9c0b3630 2023-10-16T11:20:57.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7feea255c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c0cbc20 con 0x7fee9c15b6b0 2023-10-16T11:20:57.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7feea0af9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee9c0b3630 0x7fee9c0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.053+0000 7fee9b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fee9c0d0c60 0x7fee9c0b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41584/0 (socket says 172.21.15.158:41584) 2023-10-16T11:20:57.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7fee9b7fe700 1 -- 172.21.15.158:0/4054481372 learned_addr learned my addr 172.21.15.158:0/4054481372 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:57.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7fee9bfff700 1 --2- 172.21.15.158:0/4054481372 >> v2:172.21.15.172:3300/0 conn(0x7fee9c15b6b0 0x7fee9c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7feea0af9700 1 -- 172.21.15.158:0/4054481372 >> v2:172.21.15.172:3300/0 conn(0x7fee9c15b6b0 msgr2=0x7fee9c0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:57.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7feea0af9700 1 --2- 172.21.15.158:0/4054481372 >> v2:172.21.15.172:3300/0 conn(0x7fee9c15b6b0 0x7fee9c0cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7feea0af9700 1 -- 172.21.15.158:0/4054481372 >> v1:172.21.15.158:6789/0 conn(0x7fee9c0d0c60 legacy=0x7fee9c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7feea0af9700 1 -- 172.21.15.158:0/4054481372 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c0cbf00 con 0x7fee9c0b3630 2023-10-16T11:20:57.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7fee9affd700 1 -- 172.21.15.158:0/4054481372 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756240020 0 0) 0x7fee9c0b4420 con 0x7fee9c0d0c60 2023-10-16T11:20:57.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7feea0af9700 1 --2- 172.21.15.158:0/4054481372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c0b3630 0x7fee9c0b48b0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fee9c06ffb0 tx=0x7fee9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3747e5b1c5533775 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:57.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7fee9affd700 1 -- 172.21.15.158:0/4054481372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9400e070 con 0x7fee9c0b3630 2023-10-16T11:20:57.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7fee9affd700 1 -- 172.21.15.158:0/4054481372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9400bb20 con 0x7fee9c0b3630 2023-10-16T11:20:57.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.054+0000 7fee9affd700 1 -- 172.21.15.158:0/4054481372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94012430 con 0x7fee9c0b3630 2023-10-16T11:20:57.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 -- 172.21.15.158:0/4054481372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c0b3630 msgr2=0x7fee9c0b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 --2- 172.21.15.158:0/4054481372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c0b3630 0x7fee9c0b48b0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fee9c06ffb0 tx=0x7fee9400b330 comp rx=0 tx=0).stop 2023-10-16T11:20:57.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 -- 172.21.15.158:0/4054481372 shutdown_connections 2023-10-16T11:20:57.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 --2- 172.21.15.158:0/4054481372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c0b3630 0x7fee9c0b48b0 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 --2- 172.21.15.158:0/4054481372 >> v2:172.21.15.172:3300/0 conn(0x7fee9c15b6b0 0x7fee9c0cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 -- 172.21.15.158:0/4054481372 >> 172.21.15.158:0/4054481372 conn(0x7fee9c0cdf20 msgr2=0x7fee9c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:57.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 -- 172.21.15.158:0/4054481372 shutdown_connections 2023-10-16T11:20:57.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.055+0000 7feea255c700 1 -- 172.21.15.158:0/4054481372 wait complete. 2023-10-16T11:20:57.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.056+0000 7feea255c700 1 Processor -- start 2023-10-16T11:20:57.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.056+0000 7feea255c700 1 -- start start 2023-10-16T11:20:57.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea255c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee9c0b3630 0x7fee9c0c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea255c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee9c0d0c60 0x7fee9c0caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea255c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c15b6b0 0x7fee9c1508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea255c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0c9ac0 con 0x7fee9c15b6b0 2023-10-16T11:20:57.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea255c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c155230 con 0x7fee9c0b3630 2023-10-16T11:20:57.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea255c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0c97b0 con 0x7fee9c0d0c60 2023-10-16T11:20:57.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea0af9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c15b6b0 0x7fee9c1508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea0af9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c15b6b0 0x7fee9c1508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41416/0 (socket says 172.21.15.158:41416) 2023-10-16T11:20:57.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea0af9700 1 -- 172.21.15.158:0/192716338 learned_addr learned my addr 172.21.15.158:0/192716338 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:57.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7fee9bfff700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee9c0b3630 0x7fee9c0c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7fee9b7fe700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee9c0d0c60 0x7fee9c0caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea0af9700 1 -- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee9c0d0c60 msgr2=0x7fee9c0caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea0af9700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee9c0d0c60 0x7fee9c0caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea0af9700 1 -- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee9c0b3630 msgr2=0x7fee9c0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.057+0000 7feea0af9700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee9c0b3630 0x7fee9c0c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.058+0000 7feea0af9700 1 -- 172.21.15.158:0/192716338 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee94009a70 con 0x7fee9c15b6b0 2023-10-16T11:20:57.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.058+0000 7feea0af9700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c15b6b0 0x7fee9c1508a0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7fee94013000 tx=0x7fee94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:57.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.058+0000 7fee98ff9700 1 -- 172.21.15.158:0/192716338 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94012430 con 0x7fee9c15b6b0 2023-10-16T11:20:57.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.058+0000 7fee98ff9700 1 -- 172.21.15.158:0/192716338 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee94020070 con 0x7fee9c15b6b0 2023-10-16T11:20:57.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.058+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c0ca420 con 0x7fee9c15b6b0 2023-10-16T11:20:57.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.058+0000 7fee98ff9700 1 -- 172.21.15.158:0/192716338 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94004280 con 0x7fee9c15b6b0 2023-10-16T11:20:57.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.058+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c154c70 con 0x7fee9c15b6b0 2023-10-16T11:20:57.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.059+0000 7fee98ff9700 1 -- 172.21.15.158:0/192716338 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fee940043e0 con 0x7fee9c15b6b0 2023-10-16T11:20:57.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.060+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c0c9c00 con 0x7fee9c15b6b0 2023-10-16T11:20:57.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.060+0000 7fee98ff9700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 0x7fee8806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.060+0000 7fee9bfff700 1 -- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 msgr2=0x7fee8806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:57.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.060+0000 7fee98ff9700 1 -- 172.21.15.158:0/192716338 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7fee94022070 con 0x7fee9c15b6b0 2023-10-16T11:20:57.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.060+0000 7fee9bfff700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 0x7fee8806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:57.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.064+0000 7fee98ff9700 1 -- 172.21.15.158:0/192716338 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee9c0c9c00 con 0x7fee9c15b6b0 2023-10-16T11:20:57.083 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:57.082+0000 7f40a087f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:20:57.172 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:57.170+0000 7f8c33cac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:20:57.199 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:57.197+0000 7f40a087f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:20:57.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.261+0000 7fee9bfff700 1 -- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 msgr2=0x7fee8806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:57.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.261+0000 7fee9bfff700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 0x7fee8806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:57.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.265+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee9c0ca420 con 0x7fee9c15b6b0 2023-10-16T11:20:57.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.265+0000 7fee98ff9700 1 -- 172.21.15.158:0/192716338 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v242) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee9c0ca420 con 0x7fee9c15b6b0 2023-10-16T11:20:57.267 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:20:57.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.267+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 msgr2=0x7fee8806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:57.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.267+0000 7feea255c700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 0x7fee8806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.267+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c15b6b0 msgr2=0x7fee9c1508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.267+0000 7feea255c700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c15b6b0 0x7fee9c1508a0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7fee94013000 tx=0x7fee94004ad0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 shutdown_connections 2023-10-16T11:20:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fee8806b140 0x7fee8806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee9c15b6b0 0x7fee9c1508a0 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee9c0b3630 0x7fee9c0c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 --2- 172.21.15.158:0/192716338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee9c0d0c60 0x7fee9c0caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 >> 172.21.15.158:0/192716338 conn(0x7fee9c0cdf20 msgr2=0x7fee9c0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 shutdown_connections 2023-10-16T11:20:57.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.268+0000 7feea255c700 1 -- 172.21.15.158:0/192716338 wait complete. 2023-10-16T11:20:57.282 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:20:57.331 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:57.330+0000 7f8c33cac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:20:57.356 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:57.354+0000 7f40a087f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:20:57.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.448+0000 7f1cd93f4700 1 Processor -- start 2023-10-16T11:20:57.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.448+0000 7f1cd93f4700 1 -- start start 2023-10-16T11:20:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd93f4700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cd40cf070 0x7f1cd40cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd93f4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd415dc20 con 0x7f1cd415d840 2023-10-16T11:20:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd93f4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd406c0e0 con 0x7f1cd40cbdd0 2023-10-16T11:20:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd93f4700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40cfa40 con 0x7f1cd40cf070 2023-10-16T11:20:57.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd27fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cd40cf070 0x7f1cd40cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd2ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1cd415d840 0x7f1cd4161d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41598/0 (socket says 172.21.15.158:41598) 2023-10-16T11:20:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd2ffd700 1 -- 172.21.15.158:0/621127617 learned_addr learned my addr 172.21.15.158:0/621127617 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.449+0000 7f1cd27fc700 1 -- 172.21.15.158:0/621127617 >> v1:172.21.15.172:6789/0 conn(0x7f1cd40cbdd0 legacy=0x7f1cd40cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.450+0000 7f1cd27fc700 1 -- 172.21.15.158:0/621127617 >> v1:172.21.15.158:6789/0 conn(0x7f1cd415d840 legacy=0x7f1cd4161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.450+0000 7f1cd27fc700 1 -- 172.21.15.158:0/621127617 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd406c3c0 con 0x7f1cd40cf070 2023-10-16T11:20:57.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.450+0000 7f1cd1ffb700 1 -- 172.21.15.158:0/621127617 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463809744 0 0) 0x7f1cc4001110 con 0x7f1cd415d840 2023-10-16T11:20:57.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.450+0000 7f1cd27fc700 1 --2- 172.21.15.158:0/621127617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cf070 0x7f1cd40cf500 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009840 tx=0x7f1cc8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b71d9c2319148d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:57.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.450+0000 7f1cd1ffb700 1 -- 172.21.15.158:0/621127617 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8011830 con 0x7f1cd40cf070 2023-10-16T11:20:57.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.450+0000 7f1cd1ffb700 1 -- 172.21.15.158:0/621127617 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc8011990 con 0x7f1cd40cf070 2023-10-16T11:20:57.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.450+0000 7f1cd1ffb700 1 -- 172.21.15.158:0/621127617 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8011d20 con 0x7f1cd40cf070 2023-10-16T11:20:57.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.451+0000 7f1cd93f4700 1 -- 172.21.15.158:0/621127617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cf070 msgr2=0x7f1cd40cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.451+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/621127617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cf070 0x7f1cd40cf500 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009840 tx=0x7f1cc8009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:57.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.451+0000 7f1cd93f4700 1 -- 172.21.15.158:0/621127617 shutdown_connections 2023-10-16T11:20:57.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.451+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/621127617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd40cf070 0x7f1cd40cf500 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.451+0000 7f1cd93f4700 1 -- 172.21.15.158:0/621127617 >> 172.21.15.158:0/621127617 conn(0x7f1cd406a8d0 msgr2=0x7f1cd406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:57.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.451+0000 7f1cd93f4700 1 -- 172.21.15.158:0/621127617 shutdown_connections 2023-10-16T11:20:57.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.451+0000 7f1cd93f4700 1 -- 172.21.15.158:0/621127617 wait complete. 2023-10-16T11:20:57.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.452+0000 7f1cd93f4700 1 Processor -- start 2023-10-16T11:20:57.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.452+0000 7f1cd93f4700 1 -- start start 2023-10-16T11:20:57.458 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f8c33cac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:20:57.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd93f4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd40cbdd0 0x7f1cd414f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd93f4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cf070 0x7f1cd40c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd93f4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd415d840 0x7f1cd40d1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd93f4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd414fd80 con 0x7f1cd415d840 2023-10-16T11:20:57.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd93f4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41f3260 con 0x7f1cd40cbdd0 2023-10-16T11:20:57.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd93f4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41f3540 con 0x7f1cd40cf070 2023-10-16T11:20:57.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd415d840 0x7f1cd40d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd415d840 0x7f1cd40d1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41432/0 (socket says 172.21.15.158:41432) 2023-10-16T11:20:57.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 -- 172.21.15.158:0/1886093967 learned_addr learned my addr 172.21.15.158:0/1886093967 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:57.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd2ffd700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd40cbdd0 0x7f1cd414f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd27fc700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cf070 0x7f1cd40c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 -- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cf070 msgr2=0x7f1cd40c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cf070 0x7f1cd40c57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 -- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd40cbdd0 msgr2=0x7f1cd414f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd40cbdd0 0x7f1cd414f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.453+0000 7f1cd37fe700 1 -- 172.21.15.158:0/1886093967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80094a0 con 0x7f1cd415d840 2023-10-16T11:20:57.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.454+0000 7f1cd37fe700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd415d840 0x7f1cd40d1430 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00cc50 tx=0x7f1ccc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:57.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.454+0000 7f1cbb7fe700 1 -- 172.21.15.158:0/1886093967 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc003c00 con 0x7f1cd415d840 2023-10-16T11:20:57.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.454+0000 7f1cbb7fe700 1 -- 172.21.15.158:0/1886093967 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc003d60 con 0x7f1cd415d840 2023-10-16T11:20:57.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.454+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd41f3820 con 0x7f1cd415d840 2023-10-16T11:20:57.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.454+0000 7f1cbb7fe700 1 -- 172.21.15.158:0/1886093967 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00aab0 con 0x7f1cd415d840 2023-10-16T11:20:57.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.454+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd41f3de0 con 0x7f1cd415d840 2023-10-16T11:20:57.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.455+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd40681b0 con 0x7f1cd415d840 2023-10-16T11:20:57.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.455+0000 7f1cbb7fe700 1 -- 172.21.15.158:0/1886093967 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f1ccc00ac10 con 0x7f1cd415d840 2023-10-16T11:20:57.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.456+0000 7f1cbb7fe700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 0x7f1cc006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.456+0000 7f1cd2ffd700 1 -- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 msgr2=0x7f1cc006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:57.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.456+0000 7f1cbb7fe700 1 -- 172.21.15.158:0/1886093967 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f1ccc08a210 con 0x7f1cd415d840 2023-10-16T11:20:57.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.456+0000 7f1cd2ffd700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 0x7f1cc006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:57.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.460+0000 7f1cbb7fe700 1 -- 172.21.15.158:0/1886093967 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ccc054730 con 0x7f1cd415d840 2023-10-16T11:20:57.620 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:57.618+0000 7f8c33cac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:20:57.621 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:57.619+0000 7f40a087f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:57.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.657+0000 7f1cd2ffd700 1 -- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 msgr2=0x7f1cc006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:57.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.657+0000 7f1cd2ffd700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 0x7f1cc006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:57.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.658+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1cd40b3a30 con 0x7f1cd415d840 2023-10-16T11:20:57.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.658+0000 7f1cbb7fe700 1 -- 172.21.15.158:0/1886093967 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v242) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1ccc057d50 con 0x7f1cd415d840 2023-10-16T11:20:57.660 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:57.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 msgr2=0x7f1cc006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:57.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 0x7f1cc006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd415d840 msgr2=0x7f1cd40d1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd415d840 0x7f1cd40d1430 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00cc50 tx=0x7f1ccc00cf60 comp rx=0 tx=0).stop 2023-10-16T11:20:57.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 shutdown_connections 2023-10-16T11:20:57.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f1cc006b140 0x7f1cc006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cd415d840 0x7f1cd40d1430 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cd40cbdd0 0x7f1cd414f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 --2- 172.21.15.158:0/1886093967 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cd40cf070 0x7f1cd40c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 >> 172.21.15.158:0/1886093967 conn(0x7f1cd406a8d0 msgr2=0x7f1cd406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:57.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 shutdown_connections 2023-10-16T11:20:57.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.661+0000 7f1cd93f4700 1 -- 172.21.15.158:0/1886093967 wait complete. 2023-10-16T11:20:57.677 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:20:57.784 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:57.782+0000 7f40a087f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:20:57.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.847+0000 7f0895525700 1 Processor -- start 2023-10-16T11:20:57.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.847+0000 7f0895525700 1 -- start start 2023-10-16T11:20:57.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f0895525700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f089015d990 0x7f089006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f0895525700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08900cbaf0 0x7f08900cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f0895525700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089015dd70 con 0x7f0890070300 2023-10-16T11:20:57.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f0895525700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08901616b0 con 0x7f08900cbaf0 2023-10-16T11:20:57.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f0895525700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08900ced70 con 0x7f089015d990 2023-10-16T11:20:57.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08900cbaf0 0x7f08900cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08900cbaf0 0x7f08900cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41440/0 (socket says 172.21.15.158:41440) 2023-10-16T11:20:57.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088e7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0890070300 0x7f0890161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41614/0 (socket says 172.21.15.158:41614) 2023-10-16T11:20:57.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 -- 172.21.15.158:0/1643781915 learned_addr learned my addr 172.21.15.158:0/1643781915 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:57.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088effd700 1 --2- 172.21.15.158:0/1643781915 >> v2:172.21.15.172:3300/0 conn(0x7f089015d990 0x7f089006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 -- 172.21.15.158:0/1643781915 >> v2:172.21.15.172:3300/0 conn(0x7f089015d990 msgr2=0x7f089006d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:57.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 --2- 172.21.15.158:0/1643781915 >> v2:172.21.15.172:3300/0 conn(0x7f089015d990 0x7f089006d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 -- 172.21.15.158:0/1643781915 >> v1:172.21.15.158:6789/0 conn(0x7f0890070300 legacy=0x7f0890161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088dffb700 1 -- 172.21.15.158:0/1643781915 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344754490 0 0) 0x7f0884001110 con 0x7f0890070300 2023-10-16T11:20:57.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 -- 172.21.15.158:0/1643781915 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0890070780 con 0x7f08900cbaf0 2023-10-16T11:20:57.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.848+0000 7f088f7fe700 1 --2- 172.21.15.158:0/1643781915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08900cbaf0 0x7f08900cbf80 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f08900d25a0 tx=0x7f088800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad0c102f31eadcd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:57.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f088dffb700 1 -- 172.21.15.158:0/1643781915 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088800e070 con 0x7f08900cbaf0 2023-10-16T11:20:57.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f088dffb700 1 -- 172.21.15.158:0/1643781915 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088800b8d0 con 0x7f08900cbaf0 2023-10-16T11:20:57.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f088dffb700 1 -- 172.21.15.158:0/1643781915 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088800bc80 con 0x7f08900cbaf0 2023-10-16T11:20:57.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 -- 172.21.15.158:0/1643781915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08900cbaf0 msgr2=0x7f08900cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 --2- 172.21.15.158:0/1643781915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08900cbaf0 0x7f08900cbf80 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f08900d25a0 tx=0x7f088800b330 comp rx=0 tx=0).stop 2023-10-16T11:20:57.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 -- 172.21.15.158:0/1643781915 shutdown_connections 2023-10-16T11:20:57.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 --2- 172.21.15.158:0/1643781915 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08900cbaf0 0x7f08900cbf80 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 --2- 172.21.15.158:0/1643781915 >> v2:172.21.15.172:3300/0 conn(0x7f089015d990 0x7f089006d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 -- 172.21.15.158:0/1643781915 >> 172.21.15.158:0/1643781915 conn(0x7f08900cdf20 msgr2=0x7f08900ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:57.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 -- 172.21.15.158:0/1643781915 shutdown_connections 2023-10-16T11:20:57.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.849+0000 7f0895525700 1 -- 172.21.15.158:0/1643781915 wait complete. 2023-10-16T11:20:57.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 Processor -- start 2023-10-16T11:20:57.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 -- start start 2023-10-16T11:20:57.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0890070300 0x7f0890148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08900cbaf0 0x7f0890160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f089015d990 0x7f08900cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089015ef60 con 0x7f0890070300 2023-10-16T11:20:57.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089015fdc0 con 0x7f08900cbaf0 2023-10-16T11:20:57.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f0895525700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089015f1b0 con 0x7f089015d990 2023-10-16T11:20:57.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f088effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0890070300 0x7f0890148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f088effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0890070300 0x7f0890148460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41442/0 (socket says 172.21.15.158:41442) 2023-10-16T11:20:57.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.851+0000 7f088effd700 1 -- 172.21.15.158:0/2800090706 learned_addr learned my addr 172.21.15.158:0/2800090706 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:57.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088f7fe700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f089015d990 0x7f08900cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088e7fc700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08900cbaf0 0x7f0890160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:57.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088effd700 1 -- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f089015d990 msgr2=0x7f08900cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088effd700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f089015d990 0x7f08900cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088effd700 1 -- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08900cbaf0 msgr2=0x7f0890160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:57.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088effd700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08900cbaf0 0x7f0890160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:57.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088effd700 1 -- 172.21.15.158:0/2800090706 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888009b30 con 0x7f0890070300 2023-10-16T11:20:57.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f088effd700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0890070300 0x7f0890148460 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f088000e850 tx=0x7f088000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:57.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f08777fe700 1 -- 172.21.15.158:0/2800090706 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800095b0 con 0x7f0890070300 2023-10-16T11:20:57.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f08777fe700 1 -- 172.21.15.158:0/2800090706 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0880013680 con 0x7f0890070300 2023-10-16T11:20:57.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f08777fe700 1 -- 172.21.15.158:0/2800090706 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880013a10 con 0x7f0890070300 2023-10-16T11:20:57.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089015f490 con 0x7f0890070300 2023-10-16T11:20:57.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.852+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089015fa50 con 0x7f0890070300 2023-10-16T11:20:57.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.853+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08900681b0 con 0x7f0890070300 2023-10-16T11:20:57.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.854+0000 7f08777fe700 1 -- 172.21.15.158:0/2800090706 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f0880013b70 con 0x7f0890070300 2023-10-16T11:20:57.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.854+0000 7f08777fe700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 0x7f087c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:57.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.854+0000 7f088e7fc700 1 -- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 msgr2=0x7f087c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:57.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.854+0000 7f08777fe700 1 -- 172.21.15.158:0/2800090706 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f088008a460 con 0x7f0890070300 2023-10-16T11:20:57.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.854+0000 7f088e7fc700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 0x7f087c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:57.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:57.859+0000 7f08777fe700 1 -- 172.21.15.158:0/2800090706 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0880054990 con 0x7f0890070300 2023-10-16T11:20:57.904 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:57.902+0000 7f8c33cac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:20:58.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.055+0000 7f088e7fc700 1 -- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 msgr2=0x7f087c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:58.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.055+0000 7f088e7fc700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 0x7f087c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:58.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.058+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0890070f90 con 0x7f0890070300 2023-10-16T11:20:58.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.059+0000 7f08777fe700 1 -- 172.21.15.158:0/2800090706 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v242) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0880057fb0 con 0x7f0890070300 2023-10-16T11:20:58.061 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:58.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 msgr2=0x7f087c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:58.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 0x7f087c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0890070300 msgr2=0x7f0890148460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0890070300 0x7f0890148460 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f088000e850 tx=0x7f088000eb60 comp rx=0 tx=0).stop 2023-10-16T11:20:58.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 shutdown_connections 2023-10-16T11:20:58.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7f087c06b140 0x7f087c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f089015d990 0x7f08900cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0890070300 0x7f0890148460 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 --2- 172.21.15.158:0/2800090706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08900cbaf0 0x7f0890160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 >> 172.21.15.158:0/2800090706 conn(0x7f08900cdf20 msgr2=0x7f08900b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:58.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 shutdown_connections 2023-10-16T11:20:58.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.061+0000 7f0895525700 1 -- 172.21.15.158:0/2800090706 wait complete. 2023-10-16T11:20:58.073 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.072+0000 7f8c33cac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:20:58.076 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T11:20:58.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.201+0000 7fba5690d700 1 Processor -- start 2023-10-16T11:20:58.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.201+0000 7fba5690d700 1 -- start start 2023-10-16T11:20:58.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.201+0000 7fba5690d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fba5015dbd0 0x7fba5006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.201+0000 7fba5690d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5015dfc0 0x7fba500718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba5690d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5015e510 con 0x7fba5006a8d0 2023-10-16T11:20:58.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba5690d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba500ccbb0 con 0x7fba5015dbd0 2023-10-16T11:20:58.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba5690d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5015e7f0 con 0x7fba5015dfc0 2023-10-16T11:20:58.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba54eaa700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fba5006a8d0 0x7fba500ccfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41632/0 (socket says 172.21.15.158:41632) 2023-10-16T11:20:58.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba54eaa700 1 -- 172.21.15.158:0/4169539607 learned_addr learned my addr 172.21.15.158:0/4169539607 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:58.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4f7fe700 1 --2- 172.21.15.158:0/4169539607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5015dfc0 0x7fba500718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4ffff700 1 --2- 172.21.15.158:0/4169539607 >> v2:172.21.15.172:3300/0 conn(0x7fba5015dbd0 0x7fba5006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4effd700 1 -- 172.21.15.158:0/4169539607 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592594324 0 0) 0x7fba5015e510 con 0x7fba5006a8d0 2023-10-16T11:20:58.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4effd700 1 -- 172.21.15.158:0/4169539607 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba380035b0 con 0x7fba5006a8d0 2023-10-16T11:20:58.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4f7fe700 1 -- 172.21.15.158:0/4169539607 >> v2:172.21.15.172:3300/0 conn(0x7fba5015dbd0 msgr2=0x7fba5006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:58.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4f7fe700 1 --2- 172.21.15.158:0/4169539607 >> v2:172.21.15.172:3300/0 conn(0x7fba5015dbd0 0x7fba5006df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4f7fe700 1 -- 172.21.15.158:0/4169539607 >> v1:172.21.15.158:6789/0 conn(0x7fba5006a8d0 legacy=0x7fba500ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4f7fe700 1 -- 172.21.15.158:0/4169539607 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5015ead0 con 0x7fba5015dfc0 2023-10-16T11:20:58.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4ffff700 1 --2- 172.21.15.158:0/4169539607 >> v2:172.21.15.172:3300/0 conn(0x7fba5015dbd0 0x7fba5006df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:58.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4f7fe700 1 --2- 172.21.15.158:0/4169539607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5015dfc0 0x7fba500718f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fba44009840 tx=0x7fba44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e0affc3100fbbf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:58.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4effd700 1 -- 172.21.15.158:0/4169539607 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44011830 con 0x7fba5015dfc0 2023-10-16T11:20:58.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4effd700 1 -- 172.21.15.158:0/4169539607 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba44011990 con 0x7fba5015dfc0 2023-10-16T11:20:58.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.202+0000 7fba4effd700 1 -- 172.21.15.158:0/4169539607 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44011d20 con 0x7fba5015dfc0 2023-10-16T11:20:58.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 -- 172.21.15.158:0/4169539607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5015dfc0 msgr2=0x7fba500718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 --2- 172.21.15.158:0/4169539607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5015dfc0 0x7fba500718f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fba44009840 tx=0x7fba44009b50 comp rx=0 tx=0).stop 2023-10-16T11:20:58.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 -- 172.21.15.158:0/4169539607 shutdown_connections 2023-10-16T11:20:58.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 --2- 172.21.15.158:0/4169539607 >> v2:172.21.15.172:3300/0 conn(0x7fba5015dbd0 0x7fba5006df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 --2- 172.21.15.158:0/4169539607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5015dfc0 0x7fba500718f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 -- 172.21.15.158:0/4169539607 >> 172.21.15.158:0/4169539607 conn(0x7fba500b1f00 msgr2=0x7fba500b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:58.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 -- 172.21.15.158:0/4169539607 shutdown_connections 2023-10-16T11:20:58.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.203+0000 7fba5690d700 1 -- 172.21.15.158:0/4169539607 wait complete. 2023-10-16T11:20:58.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.204+0000 7fba5690d700 1 Processor -- start 2023-10-16T11:20:58.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.204+0000 7fba5690d700 1 -- start start 2023-10-16T11:20:58.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba5690d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5006a8d0 0x7fba501f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba5690d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5015dfc0 0x7fba50151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba5690d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5015b1e0 0x7fba500c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba5690d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba500c4130 con 0x7fba5015b1e0 2023-10-16T11:20:58.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba5690d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba501f7520 con 0x7fba5015dfc0 2023-10-16T11:20:58.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba5690d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba501f6c80 con 0x7fba5006a8d0 2023-10-16T11:20:58.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba54eaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5015b1e0 0x7fba500c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba54eaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5015b1e0 0x7fba500c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41462/0 (socket says 172.21.15.158:41462) 2023-10-16T11:20:58.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba54eaa700 1 -- 172.21.15.158:0/3373573979 learned_addr learned my addr 172.21.15.158:0/3373573979 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:58.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba4ffff700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5006a8d0 0x7fba501f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.205+0000 7fba4f7fe700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5015dfc0 0x7fba50151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba54eaa700 1 -- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5006a8d0 msgr2=0x7fba501f65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba54eaa700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5006a8d0 0x7fba501f65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba54eaa700 1 -- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5015dfc0 msgr2=0x7fba50151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba54eaa700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5015dfc0 0x7fba50151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba54eaa700 1 -- 172.21.15.158:0/3373573979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba400094a0 con 0x7fba5015b1e0 2023-10-16T11:20:58.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba54eaa700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5015b1e0 0x7fba500c7d20 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7fba4800cb70 tx=0x7fba4800ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:58.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba4cff9700 1 -- 172.21.15.158:0/3373573979 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4800aa00 con 0x7fba5015b1e0 2023-10-16T11:20:58.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba4cff9700 1 -- 172.21.15.158:0/3373573979 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4800ab60 con 0x7fba5015b1e0 2023-10-16T11:20:58.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba4cff9700 1 -- 172.21.15.158:0/3373573979 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48012400 con 0x7fba5015b1e0 2023-10-16T11:20:58.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba440094a0 con 0x7fba5015b1e0 2023-10-16T11:20:58.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.206+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba501f7240 con 0x7fba5015b1e0 2023-10-16T11:20:58.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.207+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba500c4270 con 0x7fba5015b1e0 2023-10-16T11:20:58.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.208+0000 7fba4cff9700 1 -- 172.21.15.158:0/3373573979 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fba4801b3b0 con 0x7fba5015b1e0 2023-10-16T11:20:58.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.208+0000 7fba4cff9700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 0x7fba3c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.208+0000 7fba4ffff700 1 -- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 msgr2=0x7fba3c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:58.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.208+0000 7fba4cff9700 1 -- 172.21.15.158:0/3373573979 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7fba48089ea0 con 0x7fba5015b1e0 2023-10-16T11:20:58.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.208+0000 7fba4ffff700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 0x7fba3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:58.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.212+0000 7fba4cff9700 1 -- 172.21.15.158:0/3373573979 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba500c4270 con 0x7fba5015b1e0 2023-10-16T11:20:58.319 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:20:58.317+0000 7f40a087f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:20:58.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.409+0000 7fba4ffff700 1 -- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 msgr2=0x7fba3c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:58.411 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.409+0000 7fba4ffff700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 0x7fba3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:20:58.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.413+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba500c4270 con 0x7fba5015b1e0 2023-10-16T11:20:58.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.413+0000 7fba4cff9700 1 -- 172.21.15.158:0/3373573979 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v242) v1 ==== 107+0+11 (secure 0 0 0) 0x7fba48017020 con 0x7fba5015b1e0 2023-10-16T11:20:58.415 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:20:58.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 msgr2=0x7fba3c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:58.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 0x7fba3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5015b1e0 msgr2=0x7fba500c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5015b1e0 0x7fba500c7d20 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7fba4800cb70 tx=0x7fba4800ce80 comp rx=0 tx=0).stop 2023-10-16T11:20:58.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 shutdown_connections 2023-10-16T11:20:58.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fba3c06b140 0x7fba3c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba5015b1e0 0x7fba500c7d20 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba5006a8d0 0x7fba501f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.415+0000 7fba5690d700 1 --2- 172.21.15.158:0/3373573979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba5015dfc0 0x7fba50151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.416+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 >> 172.21.15.158:0/3373573979 conn(0x7fba500b1f00 msgr2=0x7fba50072340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:58.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.416+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 shutdown_connections 2023-10-16T11:20:58.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.416+0000 7fba5690d700 1 -- 172.21.15.158:0/3373573979 wait complete. 2023-10-16T11:20:58.430 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T11:20:58.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.555+0000 7fc97f59e700 1 Processor -- start 2023-10-16T11:20:58.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.555+0000 7fc97f59e700 1 -- start start 2023-10-16T11:20:58.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc98015d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97f59e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc98006ca00 0x7fc98006bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97f59e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9800cc6e0 con 0x7fc9800ce720 2023-10-16T11:20:58.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97f59e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9800ceba0 con 0x7fc98006ca00 2023-10-16T11:20:58.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97f59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98006c130 con 0x7fc98015d510 2023-10-16T11:20:58.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc98006ca00 0x7fc98006bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc98006ca00 0x7fc98006bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41468/0 (socket says 172.21.15.158:41468) 2023-10-16T11:20:58.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 -- 172.21.15.158:0/2357364349 learned_addr learned my addr 172.21.15.158:0/2357364349 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:58.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97dd9b700 1 --1- 172.21.15.158:0/2357364349 >> v1:172.21.15.158:6789/0 conn(0x7fc9800ce720 0x7fc98006c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41644/0 (socket says 172.21.15.158:41644) 2023-10-16T11:20:58.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97e59c700 1 --2- 172.21.15.158:0/2357364349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc98015d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 -- 172.21.15.158:0/2357364349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 msgr2=0x7fc98015d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:58.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 --2- 172.21.15.158:0/2357364349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc98015d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 -- 172.21.15.158:0/2357364349 >> v1:172.21.15.158:6789/0 conn(0x7fc9800ce720 legacy=0x7fc98006c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 -- 172.21.15.158:0/2357364349 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9800cbad0 con 0x7fc98006ca00 2023-10-16T11:20:58.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97d59a700 1 -- 172.21.15.158:0/2357364349 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593120654 0 0) 0x7fc9800cc6e0 con 0x7fc9800ce720 2023-10-16T11:20:58.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.556+0000 7fc97ed9d700 1 --2- 172.21.15.158:0/2357364349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc98006bbf0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fc978009e80 tx=0x7fc97800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cb2a82a37e6c99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:58.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97d59a700 1 -- 172.21.15.158:0/2357364349 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97800e070 con 0x7fc98006ca00 2023-10-16T11:20:58.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97d59a700 1 -- 172.21.15.158:0/2357364349 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97800b980 con 0x7fc98006ca00 2023-10-16T11:20:58.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97d59a700 1 -- 172.21.15.158:0/2357364349 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97800bd30 con 0x7fc98006ca00 2023-10-16T11:20:58.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 -- 172.21.15.158:0/2357364349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 msgr2=0x7fc98006bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2357364349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc98006bbf0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fc978009e80 tx=0x7fc97800b330 comp rx=0 tx=0).stop 2023-10-16T11:20:58.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 -- 172.21.15.158:0/2357364349 shutdown_connections 2023-10-16T11:20:58.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2357364349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc98006bbf0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2357364349 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc98015d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 -- 172.21.15.158:0/2357364349 >> 172.21.15.158:0/2357364349 conn(0x7fc9800d12e0 msgr2=0x7fc9800d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:58.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 -- 172.21.15.158:0/2357364349 shutdown_connections 2023-10-16T11:20:58.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.557+0000 7fc97f59e700 1 -- 172.21.15.158:0/2357364349 wait complete. 2023-10-16T11:20:58.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.559+0000 7fc97f59e700 1 Processor -- start 2023-10-16T11:20:58.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.559+0000 7fc97f59e700 1 -- start start 2023-10-16T11:20:58.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc980150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9800ce720 0x7fc9800c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc9800ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9800701d0 con 0x7fc98006ca00 2023-10-16T11:20:58.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc980150ed0 con 0x7fc9800ce720 2023-10-16T11:20:58.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98006fc10 con 0x7fc98015d510 2023-10-16T11:20:58.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc980150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc980150990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41470/0 (socket says 172.21.15.158:41470) 2023-10-16T11:20:58.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 -- 172.21.15.158:0/2411946105 learned_addr learned my addr 172.21.15.158:0/2411946105 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:58.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97dd9b700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9800ce720 0x7fc9800c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97ed9d700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc9800ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 -- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 msgr2=0x7fc9800ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc9800ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 -- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9800ce720 msgr2=0x7fc9800c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9800ce720 0x7fc9800c83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 -- 172.21.15.158:0/2411946105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978009b30 con 0x7fc98006ca00 2023-10-16T11:20:58.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97e59c700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc980150990 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fc97400d800 tx=0x7fc97400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:58.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9740125f0 con 0x7fc98006ca00 2023-10-16T11:20:58.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc974012750 con 0x7fc98006ca00 2023-10-16T11:20:58.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974012b00 con 0x7fc98006ca00 2023-10-16T11:20:58.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98006fef0 con 0x7fc98006ca00 2023-10-16T11:20:58.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.560+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9800caf00 con 0x7fc98006ca00 2023-10-16T11:20:58.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.561+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98004ebb0 con 0x7fc98006ca00 2023-10-16T11:20:58.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.562+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fc974012c60 con 0x7fc98006ca00 2023-10-16T11:20:58.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.562+0000 7fc96effd700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fc96006b140 0x7fc96006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.562+0000 7fc97dd9b700 1 -- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fc96006b140 msgr2=0x7fc96006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3687571003 2023-10-16T11:20:58.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.562+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7803+0+0 (secure 0 0 0) 0x7fc97408a2b0 con 0x7fc98006ca00 2023-10-16T11:20:58.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.562+0000 7fc97dd9b700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fc96006b140 0x7fc96006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:20:58.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.567+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9740547d0 con 0x7fc98006ca00 2023-10-16T11:20:58.649 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.648+0000 7f8c33cac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:20:58.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.668+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 154) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc974054ba0 con 0x7fc98006ca00 2023-10-16T11:20:58.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.668+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fc96006b140 msgr2=0x7fc96006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:20:58.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.668+0000 7fc96effd700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fc96006b140 0x7fc96006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.734 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.734 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.734 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.734 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.735 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.736 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.732+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.743 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.741+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.743 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.741+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.743 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.741+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.744 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.741+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.744 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.741+0000 7f8bac492700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.744 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.742+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.744 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.742+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.745 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.742+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.745 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.742+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.745 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:20:58.742+0000 7f8baf498700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:20:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.788+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fc9800caa10 con 0x7fc98006ca00 2023-10-16T11:20:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.788+0000 7fc96effd700 1 -- 172.21.15.158:0/2411946105 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v243) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc9740878a0 con 0x7fc98006ca00 2023-10-16T11:20:58.790 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:20:58.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 msgr2=0x7fc980150990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc980150990 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fc97400d800 tx=0x7fc97400db10 comp rx=0 tx=0).stop 2023-10-16T11:20:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 shutdown_connections 2023-10-16T11:20:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:6832/3687571003,v1:172.21.15.158:6833/3687571003] conn(0x7fc96006b140 0x7fc96006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc98015d510 0x7fc9800ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98006ca00 0x7fc980150990 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 --2- 172.21.15.158:0/2411946105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9800ce720 0x7fc9800c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 >> 172.21.15.158:0/2411946105 conn(0x7fc9800d12e0 msgr2=0x7fc9800b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:58.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.790+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 shutdown_connections 2023-10-16T11:20:58.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.791+0000 7fc97f59e700 1 -- 172.21.15.158:0/2411946105 wait complete. 2023-10-16T11:20:58.804 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:58.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.923+0000 7f5d79c3b700 1 Processor -- start 2023-10-16T11:20:58.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.923+0000 7f5d79c3b700 1 -- start start 2023-10-16T11:20:58.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d79c3b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5d740ce640 0x7f5d740cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d79c3b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5d740c65b0 0x7f5d7406f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d79c3b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d740cef60 con 0x7f5d740cd600 2023-10-16T11:20:58.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d79c3b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7415e480 con 0x7f5d740ce640 2023-10-16T11:20:58.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d79c3b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7415d800 con 0x7f5d740c65b0 2023-10-16T11:20:58.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d737fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5d740ce640 0x7f5d740cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d73fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5d740cd600 0x7f5d740cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41758/0 (socket says 172.21.15.158:41758) 2023-10-16T11:20:58.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d73fff700 1 -- 172.21.15.158:0/3827798241 learned_addr learned my addr 172.21.15.158:0/3827798241 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:20:58.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d72ffd700 1 --2- 172.21.15.158:0/3827798241 >> v2:172.21.15.172:3300/0 conn(0x7f5d740c65b0 0x7f5d7406f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d737fe700 1 -- 172.21.15.158:0/3827798241 >> v2:172.21.15.172:3300/0 conn(0x7f5d740c65b0 msgr2=0x7f5d7406f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:58.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d737fe700 1 --2- 172.21.15.158:0/3827798241 >> v2:172.21.15.172:3300/0 conn(0x7f5d740c65b0 0x7f5d7406f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.924+0000 7f5d727fc700 1 -- 172.21.15.158:0/3827798241 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649982116 0 0) 0x7f5d740cef60 con 0x7f5d740cd600 2023-10-16T11:20:58.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d737fe700 1 -- 172.21.15.158:0/3827798241 >> v1:172.21.15.158:6789/0 conn(0x7f5d740cd600 legacy=0x7f5d740cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d737fe700 1 -- 172.21.15.158:0/3827798241 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7415dae0 con 0x7f5d740ce640 2023-10-16T11:20:58.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d737fe700 1 --2- 172.21.15.158:0/3827798241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cea20 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7f5d740aeb40 tx=0x7f5d5c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df46aedcb5ed1866 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:58.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d727fc700 1 -- 172.21.15.158:0/3827798241 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c011830 con 0x7f5d740ce640 2023-10-16T11:20:58.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d727fc700 1 -- 172.21.15.158:0/3827798241 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c011990 con 0x7f5d740ce640 2023-10-16T11:20:58.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d727fc700 1 -- 172.21.15.158:0/3827798241 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c011d20 con 0x7f5d740ce640 2023-10-16T11:20:58.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3827798241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 msgr2=0x7f5d740cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.925+0000 7f5d79c3b700 1 --2- 172.21.15.158:0/3827798241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cea20 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7f5d740aeb40 tx=0x7f5d5c009b20 comp rx=0 tx=0).stop 2023-10-16T11:20:58.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.926+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3827798241 shutdown_connections 2023-10-16T11:20:58.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.926+0000 7f5d79c3b700 1 --2- 172.21.15.158:0/3827798241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cea20 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.926+0000 7f5d79c3b700 1 --2- 172.21.15.158:0/3827798241 >> v2:172.21.15.172:3300/0 conn(0x7f5d740c65b0 0x7f5d7406f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.926+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3827798241 >> 172.21.15.158:0/3827798241 conn(0x7f5d740cc4b0 msgr2=0x7f5d740cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:58.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.926+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3827798241 shutdown_connections 2023-10-16T11:20:58.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.926+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3827798241 wait complete. 2023-10-16T11:20:58.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.927+0000 7f5d79c3b700 1 Processor -- start 2023-10-16T11:20:58.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.927+0000 7f5d79c3b700 1 -- start start 2023-10-16T11:20:58.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d79c3b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d740c65b0 0x7f5d740c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d79c3b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d740cd600 0x7f5d740cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d79c3b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:58.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d79c3b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740c9af0 con 0x7f5d740ce640 2023-10-16T11:20:58.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d79c3b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740b58f0 con 0x7f5d740cd600 2023-10-16T11:20:58.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d79c3b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740c8ae0 con 0x7f5d740c65b0 2023-10-16T11:20:58.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d73fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d73fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41588/0 (socket says 172.21.15.158:41588) 2023-10-16T11:20:58.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.928+0000 7f5d73fff700 1 -- 172.21.15.158:0/3982093136 learned_addr learned my addr 172.21.15.158:0/3982093136 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:58.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d72ffd700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d740cd600 0x7f5d740cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d737fe700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d740c65b0 0x7f5d740c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:58.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d73fff700 1 -- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d740c65b0 msgr2=0x7f5d740c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d73fff700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d740c65b0 0x7f5d740c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d73fff700 1 -- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d740cd600 msgr2=0x7f5d740cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:58.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d73fff700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d740cd600 0x7f5d740cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:58.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d73fff700 1 -- 172.21.15.158:0/3982093136 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c0094a0 con 0x7f5d740ce640 2023-10-16T11:20:58.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d737fe700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d740c65b0 0x7f5d740c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:58.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d73fff700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cbeb0 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00cbd0 tx=0x7f5d6c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:58.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d6bfff700 1 -- 172.21.15.158:0/3982093136 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c00ab10 con 0x7f5d740ce640 2023-10-16T11:20:58.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d6bfff700 1 -- 172.21.15.158:0/3982093136 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c00ac90 con 0x7f5d740ce640 2023-10-16T11:20:58.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d740c8d30 con 0x7f5d740ce640 2023-10-16T11:20:58.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d6bfff700 1 -- 172.21.15.158:0/3982093136 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0070d0 con 0x7f5d740ce640 2023-10-16T11:20:58.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.929+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d740c92f0 con 0x7f5d740ce640 2023-10-16T11:20:58.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.930+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d740681b0 con 0x7f5d740ce640 2023-10-16T11:20:58.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.930+0000 7f5d6bfff700 1 -- 172.21.15.158:0/3982093136 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 154) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f5d6c007230 con 0x7f5d740ce640 2023-10-16T11:20:58.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.931+0000 7f5d6bfff700 1 -- 172.21.15.158:0/3982093136 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 8233+0+0 (secure 0 0 0) 0x7f5d6c0898b0 con 0x7f5d740ce640 2023-10-16T11:20:58.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:58.935+0000 7f5d6bfff700 1 -- 172.21.15.158:0/3982093136 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d6c054330 con 0x7f5d740ce640 2023-10-16T11:20:59.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.173+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d7415e0a0 con 0x7f5d740ce640 2023-10-16T11:20:59.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.174+0000 7f5d6bfff700 1 -- 172.21.15.158:0/3982093136 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6160 (secure 0 0 0) 0x7f5d6c018020 con 0x7f5d740ce640 2023-10-16T11:20:59.176 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:59.176 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:20:59.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 msgr2=0x7f5d740cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cbeb0 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00cbd0 tx=0x7f5d6c00cf90 comp rx=0 tx=0).stop 2023-10-16T11:20:59.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 shutdown_connections 2023-10-16T11:20:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d740ce640 0x7f5d740cbeb0 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d740c65b0 0x7f5d740c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 --2- 172.21.15.158:0/3982093136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d740cd600 0x7f5d740cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 >> 172.21.15.158:0/3982093136 conn(0x7f5d740cc4b0 msgr2=0x7f5d740cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 shutdown_connections 2023-10-16T11:20:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.176+0000 7f5d79c3b700 1 -- 172.21.15.158:0/3982093136 wait complete. 2023-10-16T11:20:59.180 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 134 2023-10-16T11:20:59.190 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:20:59.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.309+0000 7fec0b31e700 1 Processor -- start 2023-10-16T11:20:59.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.309+0000 7fec0b31e700 1 -- start start 2023-10-16T11:20:59.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec0b31e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec0406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec0b31e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fec0406e5d0 0x7fec04071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec0b31e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec04071e50 con 0x7fec0406af20 2023-10-16T11:20:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec0b31e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec04161270 con 0x7fec0406e5d0 2023-10-16T11:20:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec0b31e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec04161510 con 0x7fec0406ab40 2023-10-16T11:20:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec088b9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fec0406e5d0 0x7fec04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec090ba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec0406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec090ba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec0406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:52424/0 (socket says 172.21.15.158:52424) 2023-10-16T11:20:59.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec098bb700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fec0406af20 0x7fec040722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:45674/0 (socket says 172.21.15.158:45674) 2023-10-16T11:20:59.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec088b9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fec0406e5d0 0x7fec04071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:42770/0 (socket says 172.21.15.158:42770) 2023-10-16T11:20:59.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec090ba700 1 -- 172.21.15.158:0/3632324409 learned_addr learned my addr 172.21.15.158:0/3632324409 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec090ba700 1 -- 172.21.15.158:0/3632324409 >> v2:172.21.15.172:3300/0 conn(0x7fec0406e5d0 msgr2=0x7fec04071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec090ba700 1 --2- 172.21.15.158:0/3632324409 >> v2:172.21.15.172:3300/0 conn(0x7fec0406e5d0 0x7fec04071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec090ba700 1 -- 172.21.15.158:0/3632324409 >> v1:172.21.15.172:6789/0 conn(0x7fec0406af20 legacy=0x7fec040722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec090ba700 1 -- 172.21.15.158:0/3632324409 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec041617f0 con 0x7fec0406ab40 2023-10-16T11:20:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7febfbfff700 1 -- 172.21.15.158:0/3632324409 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933444674 0 0) 0x7fec04071e50 con 0x7fec0406af20 2023-10-16T11:20:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.310+0000 7fec088b9700 1 --2- 172.21.15.158:0/3632324409 >> v2:172.21.15.172:3300/0 conn(0x7fec0406e5d0 0x7fec04071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:20:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec090ba700 1 --2- 172.21.15.158:0/3632324409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec0406d820 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fec000098f0 tx=0x7fec00009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b72bea366b8e6055 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7febfbfff700 1 -- 172.21.15.158:0/3632324409 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00011830 con 0x7fec0406ab40 2023-10-16T11:20:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7febfbfff700 1 -- 172.21.15.158:0/3632324409 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec00011990 con 0x7fec0406ab40 2023-10-16T11:20:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7febfbfff700 1 -- 172.21.15.158:0/3632324409 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00011d20 con 0x7fec0406ab40 2023-10-16T11:20:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 -- 172.21.15.158:0/3632324409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 msgr2=0x7fec0406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 --2- 172.21.15.158:0/3632324409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec0406d820 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fec000098f0 tx=0x7fec00009c00 comp rx=0 tx=0).stop 2023-10-16T11:20:59.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 -- 172.21.15.158:0/3632324409 shutdown_connections 2023-10-16T11:20:59.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 --2- 172.21.15.158:0/3632324409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec0406d820 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 --2- 172.21.15.158:0/3632324409 >> v2:172.21.15.172:3300/0 conn(0x7fec0406e5d0 0x7fec04071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 -- 172.21.15.158:0/3632324409 >> 172.21.15.158:0/3632324409 conn(0x7fec040d02b0 msgr2=0x7fec040d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:59.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 -- 172.21.15.158:0/3632324409 shutdown_connections 2023-10-16T11:20:59.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.311+0000 7fec0b31e700 1 -- 172.21.15.158:0/3632324409 wait complete. 2023-10-16T11:20:59.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 Processor -- start 2023-10-16T11:20:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 -- start start 2023-10-16T11:20:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec040c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec0406af20 0x7fec04156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec0406e5d0 0x7fec040b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec040c6d50 con 0x7fec0406e5d0 2023-10-16T11:20:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec041f3fc0 con 0x7fec0406af20 2023-10-16T11:20:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec0b31e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec041f3ce0 con 0x7fec0406ab40 2023-10-16T11:20:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.313+0000 7fec098bb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec0406e5d0 0x7fec040b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec0406e5d0 0x7fec040b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41600/0 (socket says 172.21.15.158:41600) 2023-10-16T11:20:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 -- 172.21.15.158:0/718303372 learned_addr learned my addr 172.21.15.158:0/718303372 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:59.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec090ba700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec040c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec088b9700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec0406af20 0x7fec04156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 -- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 msgr2=0x7fec040c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec040c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 -- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec0406af20 msgr2=0x7fec04156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec0406af20 0x7fec04156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 -- 172.21.15.158:0/718303372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40094a0 con 0x7fec0406e5d0 2023-10-16T11:20:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec088b9700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec0406af20 0x7fec04156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec098bb700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec0406e5d0 0x7fec040b3ed0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7febfc002230 tx=0x7febfc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7febf9ffb700 1 -- 172.21.15.158:0/718303372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00ac20 con 0x7fec0406e5d0 2023-10-16T11:20:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7febf9ffb700 1 -- 172.21.15.158:0/718303372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00ada0 con 0x7fec0406e5d0 2023-10-16T11:20:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec00009550 con 0x7fec0406e5d0 2023-10-16T11:20:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7febf9ffb700 1 -- 172.21.15.158:0/718303372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc007310 con 0x7fec0406e5d0 2023-10-16T11:20:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.314+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec040b36d0 con 0x7fec0406e5d0 2023-10-16T11:20:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.315+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec040681b0 con 0x7fec0406e5d0 2023-10-16T11:20:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.316+0000 7febf9ffb700 1 -- 172.21.15.158:0/718303372 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 154) v1 ==== 88940+0+0 (secure 0 0 0) 0x7febfc007470 con 0x7fec0406e5d0 2023-10-16T11:20:59.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.316+0000 7febf9ffb700 1 -- 172.21.15.158:0/718303372 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 8233+0+0 (secure 0 0 0) 0x7febfc089670 con 0x7fec0406e5d0 2023-10-16T11:20:59.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.321+0000 7febf9ffb700 1 -- 172.21.15.158:0/718303372 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febfc054070 con 0x7fec0406e5d0 2023-10-16T11:20:59.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.515+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec040b6230 con 0x7fec0406e5d0 2023-10-16T11:20:59.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.516+0000 7febf9ffb700 1 -- 172.21.15.158:0/718303372 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+22453 (secure 0 0 0) 0x7febfc018090 con 0x7fec0406e5d0 2023-10-16T11:20:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:59.519 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":243,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:58.661904+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/937741801":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3443938198":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/872331855":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/1271192867":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/331460130":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/387969810":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/750584278":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2035506903":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:20:59.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec0406e5d0 msgr2=0x7fec040b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec0406e5d0 0x7fec040b3ed0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7febfc002230 tx=0x7febfc00a060 comp rx=0 tx=0).stop 2023-10-16T11:20:59.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 shutdown_connections 2023-10-16T11:20:59.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec0406e5d0 0x7fec040b3ed0 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec0406ab40 0x7fec040c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 --2- 172.21.15.158:0/718303372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec0406af20 0x7fec04156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 >> 172.21.15.158:0/718303372 conn(0x7fec040d02b0 msgr2=0x7fec04072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:59.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 shutdown_connections 2023-10-16T11:20:59.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.519+0000 7fec0b31e700 1 -- 172.21.15.158:0/718303372 wait complete. 2023-10-16T11:20:59.533 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:20:59.533 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:20:59.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.651+0000 7f289c6a3700 1 Processor -- start 2023-10-16T11:20:59.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.652+0000 7f289c6a3700 1 -- start start 2023-10-16T11:20:59.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.652+0000 7f289c6a3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2894157890 0x7f289415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.652+0000 7f289c6a3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f289406b9f0 0x7f289406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.652+0000 7f289c6a3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289415b4d0 con 0x7f289415dc30 2023-10-16T11:20:59.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.652+0000 7f289c6a3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f289415e5d0 con 0x7f289406b9f0 2023-10-16T11:20:59.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289c6a3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f289415e0c0 con 0x7f2894157890 2023-10-16T11:20:59.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f289406b9f0 0x7f289406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f289406b9f0 0x7f289406df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41606/0 (socket says 172.21.15.158:41606) 2023-10-16T11:20:59.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 -- 172.21.15.158:0/3714428537 learned_addr learned my addr 172.21.15.158:0/3714428537 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:59.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289a43f700 1 --2- 172.21.15.158:0/3714428537 >> v2:172.21.15.172:3300/0 conn(0x7f2894157890 0x7f289415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 -- 172.21.15.158:0/3714428537 >> v2:172.21.15.172:3300/0 conn(0x7f2894157890 msgr2=0x7f289415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:59.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 --2- 172.21.15.158:0/3714428537 >> v2:172.21.15.172:3300/0 conn(0x7f2894157890 0x7f289415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 -- 172.21.15.158:0/3714428537 >> v1:172.21.15.172:6789/0 conn(0x7f289415dc30 legacy=0x7f289406b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:20:59.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 -- 172.21.15.158:0/3714428537 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289406bef0 con 0x7f289406b9f0 2023-10-16T11:20:59.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289a43f700 1 --2- 172.21.15.158:0/3714428537 >> v2:172.21.15.172:3300/0 conn(0x7f2894157890 0x7f289415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:20:59.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289ac40700 1 --2- 172.21.15.158:0/3714428537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289406b9f0 0x7f289406df30 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f28940b2bb0 tx=0x7f288c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd470dfd92b5d4c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:59.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289943d700 1 -- 172.21.15.158:0/3714428537 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c00e070 con 0x7f289406b9f0 2023-10-16T11:20:59.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289943d700 1 -- 172.21.15.158:0/3714428537 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288c003a20 con 0x7f289406b9f0 2023-10-16T11:20:59.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.653+0000 7f289943d700 1 -- 172.21.15.158:0/3714428537 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c003dd0 con 0x7f289406b9f0 2023-10-16T11:20:59.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 -- 172.21.15.158:0/3714428537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289406b9f0 msgr2=0x7f289406df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3714428537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289406b9f0 0x7f289406df30 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f28940b2bb0 tx=0x7f288c00b330 comp rx=0 tx=0).stop 2023-10-16T11:20:59.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 -- 172.21.15.158:0/3714428537 shutdown_connections 2023-10-16T11:20:59.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3714428537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289406b9f0 0x7f289406df30 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3714428537 >> v2:172.21.15.172:3300/0 conn(0x7f2894157890 0x7f289415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 -- 172.21.15.158:0/3714428537 >> 172.21.15.158:0/3714428537 conn(0x7f28940cfef0 msgr2=0x7f28940d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:59.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 -- 172.21.15.158:0/3714428537 shutdown_connections 2023-10-16T11:20:59.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.654+0000 7f289c6a3700 1 -- 172.21.15.158:0/3714428537 wait complete. 2023-10-16T11:20:59.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.655+0000 7f289c6a3700 1 Processor -- start 2023-10-16T11:20:59.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.655+0000 7f289c6a3700 1 -- start start 2023-10-16T11:20:59.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.656+0000 7f289c6a3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f289406b9f0 0x7f2894156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.656+0000 7f289c6a3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2894157890 0x7f2894158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.656+0000 7f289c6a3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289415dc30 0x7f2894158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.656+0000 7f289c6a3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289414deb0 con 0x7f289415dc30 2023-10-16T11:20:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.656+0000 7f289c6a3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28941541d0 con 0x7f2894157890 2023-10-16T11:20:59.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.656+0000 7f289c6a3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2894153370 con 0x7f289406b9f0 2023-10-16T11:20:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289415dc30 0x7f2894158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289415dc30 0x7f2894158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41612/0 (socket says 172.21.15.158:41612) 2023-10-16T11:20:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 -- 172.21.15.158:0/3206289141 learned_addr learned my addr 172.21.15.158:0/3206289141 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:20:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f2899c3e700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2894157890 0x7f2894158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289a43f700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f289406b9f0 0x7f2894156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 -- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f289406b9f0 msgr2=0x7f2894156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f289406b9f0 0x7f2894156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 -- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2894157890 msgr2=0x7f2894158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2894157890 0x7f2894158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 -- 172.21.15.158:0/3206289141 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c009b30 con 0x7f289415dc30 2023-10-16T11:20:59.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289ac40700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289415dc30 0x7f2894158cd0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f2894072820 tx=0x7f288c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:59.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c004440 con 0x7f289415dc30 2023-10-16T11:20:59.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288c0045c0 con 0x7f289415dc30 2023-10-16T11:20:59.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2894153650 con 0x7f289415dc30 2023-10-16T11:20:59.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c004970 con 0x7f289415dc30 2023-10-16T11:20:59.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.657+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2894153c10 con 0x7f289415dc30 2023-10-16T11:20:59.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.658+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289414e040 con 0x7f289415dc30 2023-10-16T11:20:59.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.659+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 154) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f288c012430 con 0x7f289415dc30 2023-10-16T11:20:59.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.659+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 8233+0+0 (secure 0 0 0) 0x7f288c08aef0 con 0x7f289415dc30 2023-10-16T11:20:59.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.663+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f289414e040 con 0x7f289415dc30 2023-10-16T11:20:59.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.671+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f288c0093a0 con 0x7f289415dc30 2023-10-16T11:20:59.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.671+0000 7f288affd700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2878073b00 0x7f2878075ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:20:59.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.674+0000 7f289a43f700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2878073b00 0x7f2878075ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:20:59.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.675+0000 7f289a43f700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2878073b00 0x7f2878075ef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2884006fd0 tx=0x7f2884008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:20:59.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.906+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f289414dff0 con 0x7f289415dc30 2023-10-16T11:20:59.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.907+0000 7f288affd700 1 -- 172.21.15.158:0/3206289141 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6160 (secure 0 0 0) 0x7f288c06f570 con 0x7f289415dc30 2023-10-16T11:20:59.909 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:20:59.909 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:20:59.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2878073b00 msgr2=0x7f2878075ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2878073b00 0x7f2878075ef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2884006fd0 tx=0x7f2884008040 comp rx=0 tx=0).stop 2023-10-16T11:20:59.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289415dc30 msgr2=0x7f2894158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:20:59.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289415dc30 0x7f2894158cd0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f2894072820 tx=0x7f288c004ad0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 shutdown_connections 2023-10-16T11:20:59.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2878073b00 0x7f2878075ef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f289415dc30 0x7f2894158cd0 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f289406b9f0 0x7f2894156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 --2- 172.21.15.158:0/3206289141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2894157890 0x7f2894158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:20:59.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 >> 172.21.15.158:0/3206289141 conn(0x7f28940cfef0 msgr2=0x7f2894159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:20:59.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 shutdown_connections 2023-10-16T11:20:59.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:20:59.909+0000 7f289c6a3700 1 -- 172.21.15.158:0/3206289141 wait complete. 2023-10-16T11:20:59.913 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 134 2023-10-16T11:20:59.923 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:00.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.041+0000 7f2abd89e700 1 Processor -- start 2023-10-16T11:21:00.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.041+0000 7f2abd89e700 1 -- start start 2023-10-16T11:21:00.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2abd89e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2ab80ce640 0x7f2ab80cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2abd89e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2ab80cae30 0x7f2ab80cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2abd89e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80cef60 con 0x7f2ab80c65b0 2023-10-16T11:21:00.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2abd89e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab815e480 con 0x7f2ab80cae30 2023-10-16T11:21:00.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2abd89e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab815d800 con 0x7f2ab80ce640 2023-10-16T11:21:00.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2ab77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2ab80cae30 0x7f2ab80cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2ab77fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2ab80cae30 0x7f2ab80cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41642/0 (socket says 172.21.15.158:41642) 2023-10-16T11:21:00.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.042+0000 7f2ab77fe700 1 -- 172.21.15.158:0/774643743 learned_addr learned my addr 172.21.15.158:0/774643743 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:00.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab6ffd700 1 --2- 172.21.15.158:0/774643743 >> v2:172.21.15.172:3300/0 conn(0x7f2ab80ce640 0x7f2ab80cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab77fe700 1 -- 172.21.15.158:0/774643743 >> v2:172.21.15.172:3300/0 conn(0x7f2ab80ce640 msgr2=0x7f2ab80cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:00.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab77fe700 1 --2- 172.21.15.158:0/774643743 >> v2:172.21.15.172:3300/0 conn(0x7f2ab80ce640 0x7f2ab80cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab77fe700 1 -- 172.21.15.158:0/774643743 >> v1:172.21.15.172:6789/0 conn(0x7f2ab80c65b0 legacy=0x7f2ab806f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab77fe700 1 -- 172.21.15.158:0/774643743 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab815dae0 con 0x7f2ab80cae30 2023-10-16T11:21:00.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab77fe700 1 --2- 172.21.15.158:0/774643743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80cae30 0x7f2ab80cb2c0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80b2670 tx=0x7f2ab000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f92af5bf023891a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab5ffb700 1 -- 172.21.15.158:0/774643743 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000b910 con 0x7f2ab80cae30 2023-10-16T11:21:00.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab5ffb700 1 -- 172.21.15.158:0/774643743 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab000ba70 con 0x7f2ab80cae30 2023-10-16T11:21:00.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2ab5ffb700 1 -- 172.21.15.158:0/774643743 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000be20 con 0x7f2ab80cae30 2023-10-16T11:21:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2abd89e700 1 -- 172.21.15.158:0/774643743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80cae30 msgr2=0x7f2ab80cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.043+0000 7f2abd89e700 1 --2- 172.21.15.158:0/774643743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80cae30 0x7f2ab80cb2c0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80b2670 tx=0x7f2ab000b330 comp rx=0 tx=0).stop 2023-10-16T11:21:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.044+0000 7f2abd89e700 1 -- 172.21.15.158:0/774643743 shutdown_connections 2023-10-16T11:21:00.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.044+0000 7f2abd89e700 1 --2- 172.21.15.158:0/774643743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80cae30 0x7f2ab80cb2c0 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.044+0000 7f2abd89e700 1 --2- 172.21.15.158:0/774643743 >> v2:172.21.15.172:3300/0 conn(0x7f2ab80ce640 0x7f2ab80cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.044+0000 7f2abd89e700 1 -- 172.21.15.158:0/774643743 >> 172.21.15.158:0/774643743 conn(0x7f2ab80cc4b0 msgr2=0x7f2ab80cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.044+0000 7f2abd89e700 1 -- 172.21.15.158:0/774643743 shutdown_connections 2023-10-16T11:21:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.044+0000 7f2abd89e700 1 -- 172.21.15.158:0/774643743 wait complete. 2023-10-16T11:21:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.045+0000 7f2abd89e700 1 Processor -- start 2023-10-16T11:21:00.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.045+0000 7f2abd89e700 1 -- start start 2023-10-16T11:21:00.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80c65b0 0x7f2ab81422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ab80cae30 0x7f2ab81427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab80ce640 0x7f2ab80af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80c7a00 con 0x7f2ab80c65b0 2023-10-16T11:21:00.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80c6e10 con 0x7f2ab80ce640 2023-10-16T11:21:00.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80c70d0 con 0x7f2ab80cae30 2023-10-16T11:21:00.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80c65b0 0x7f2ab81422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80c65b0 0x7f2ab81422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41644/0 (socket says 172.21.15.158:41644) 2023-10-16T11:21:00.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 -- 172.21.15.158:0/3231256105 learned_addr learned my addr 172.21.15.158:0/3231256105 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:00.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab77fe700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab80ce640 0x7f2ab80af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab67fc700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ab80cae30 0x7f2ab81427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 -- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ab80cae30 msgr2=0x7f2ab81427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ab80cae30 0x7f2ab81427e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 -- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab80ce640 msgr2=0x7f2ab80af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab80ce640 0x7f2ab80af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 -- 172.21.15.158:0/3231256105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0009b30 con 0x7f2ab80c65b0 2023-10-16T11:21:00.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab77fe700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab80ce640 0x7f2ab80af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:00.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2ab6ffd700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80c65b0 0x7f2ab81422a0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00e9b0 tx=0x7f2aac00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2a9ffff700 1 -- 172.21.15.158:0/3231256105 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac009710 con 0x7f2ab80c65b0 2023-10-16T11:21:00.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2a9ffff700 1 -- 172.21.15.158:0/3231256105 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac00c850 con 0x7f2ab80c65b0 2023-10-16T11:21:00.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab80c73b0 con 0x7f2ab80c65b0 2023-10-16T11:21:00.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2a9ffff700 1 -- 172.21.15.158:0/3231256105 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac00cbc0 con 0x7f2ab80c65b0 2023-10-16T11:21:00.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.046+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab80afa10 con 0x7f2ab80c65b0 2023-10-16T11:21:00.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.048+0000 7f2a9ffff700 1 -- 172.21.15.158:0/3231256105 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2aac00cd20 con 0x7f2ab80c65b0 2023-10-16T11:21:00.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.048+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab80681b0 con 0x7f2ab80c65b0 2023-10-16T11:21:00.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.048+0000 7f2a9ffff700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2aa406ad50 0x7f2aa406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.048+0000 7f2ab67fc700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2aa406ad50 0x7f2aa406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.048+0000 7f2a9ffff700 1 -- 172.21.15.158:0/3231256105 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 8233+0+0 (secure 0 0 0) 0x7f2aac089dd0 con 0x7f2ab80c65b0 2023-10-16T11:21:00.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.049+0000 7f2ab67fc700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2aa406ad50 0x7f2aa406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2ab81f26f0 tx=0x7f2aa8009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.053+0000 7f2a9ffff700 1 -- 172.21.15.158:0/3231256105 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2aac08d030 con 0x7f2ab80c65b0 2023-10-16T11:21:00.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.294+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ab8143060 con 0x7f2ab80c65b0 2023-10-16T11:21:00.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.294+0000 7f2a9ffff700 1 -- 172.21.15.158:0/3231256105 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6160 (secure 0 0 0) 0x7f2aac021020 con 0x7f2ab80c65b0 2023-10-16T11:21:00.296 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:00.297 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:25.120788+0000","modified":"2023-10-16T11:20:28.157424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39383},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39383":{"gid":39383,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/468116497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":468116497},{"type":"v1","addr":"172.21.15.172:6834","nonce":468116497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:21:00.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2aa406ad50 msgr2=0x7f2aa406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2aa406ad50 0x7f2aa406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2ab81f26f0 tx=0x7f2aa8009040 comp rx=0 tx=0).stop 2023-10-16T11:21:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80c65b0 msgr2=0x7f2ab81422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80c65b0 0x7f2ab81422a0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00e9b0 tx=0x7f2aac00ed70 comp rx=0 tx=0).stop 2023-10-16T11:21:00.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 shutdown_connections 2023-10-16T11:21:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2aa406ad50 0x7f2aa406d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab80ce640 0x7f2ab80af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ab80c65b0 0x7f2ab81422a0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 --2- 172.21.15.158:0/3231256105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ab80cae30 0x7f2ab81427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 >> 172.21.15.158:0/3231256105 conn(0x7f2ab80cc4b0 msgr2=0x7f2ab81411d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:00.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 shutdown_connections 2023-10-16T11:21:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.297+0000 7f2abd89e700 1 -- 172.21.15.158:0/3231256105 wait complete. 2023-10-16T11:21:00.301 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 134 2023-10-16T11:21:00.311 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:00.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.431+0000 7fb58799a700 1 Processor -- start 2023-10-16T11:21:00.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.431+0000 7fb58799a700 1 -- start start 2023-10-16T11:21:00.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb58799a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb58006ab40 0x7fb58006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb58799a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb580071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb58799a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb580071e50 con 0x7fb58006af20 2023-10-16T11:21:00.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb58799a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb580161270 con 0x7fb58006ab40 2023-10-16T11:21:00.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb58799a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb580161510 con 0x7fb58006e5d0 2023-10-16T11:21:00.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb58006ab40 0x7fb58006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb58006ab40 0x7fb58006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41652/0 (socket says 172.21.15.158:41652) 2023-10-16T11:21:00.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 -- 172.21.15.158:0/1507506755 learned_addr learned my addr 172.21.15.158:0/1507506755 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:00.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb584f35700 1 --2- 172.21.15.158:0/1507506755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb580071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 -- 172.21.15.158:0/1507506755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 msgr2=0x7fb580071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:00.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 --2- 172.21.15.158:0/1507506755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb580071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 -- 172.21.15.158:0/1507506755 >> v1:172.21.15.158:6789/0 conn(0x7fb58006af20 legacy=0x7fb5800722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 -- 172.21.15.158:0/1507506755 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5801617f0 con 0x7fb58006ab40 2023-10-16T11:21:00.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb577fff700 1 -- 172.21.15.158:0/1507506755 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125854233 0 0) 0x7fb580071e50 con 0x7fb58006af20 2023-10-16T11:21:00.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb584f35700 1 --2- 172.21.15.158:0/1507506755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb580071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:00.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.432+0000 7fb585736700 1 --2- 172.21.15.158:0/1507506755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb58006d820 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0098f0 tx=0x7fb57c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55063abcc413f04 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb577fff700 1 -- 172.21.15.158:0/1507506755 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00b8a0 con 0x7fb58006ab40 2023-10-16T11:21:00.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb577fff700 1 -- 172.21.15.158:0/1507506755 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c00ba00 con 0x7fb58006ab40 2023-10-16T11:21:00.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb577fff700 1 -- 172.21.15.158:0/1507506755 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00bd90 con 0x7fb58006ab40 2023-10-16T11:21:00.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 -- 172.21.15.158:0/1507506755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 msgr2=0x7fb58006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 --2- 172.21.15.158:0/1507506755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb58006d820 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0098f0 tx=0x7fb57c009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:00.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 -- 172.21.15.158:0/1507506755 shutdown_connections 2023-10-16T11:21:00.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 --2- 172.21.15.158:0/1507506755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb58006d820 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 --2- 172.21.15.158:0/1507506755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb580071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 -- 172.21.15.158:0/1507506755 >> 172.21.15.158:0/1507506755 conn(0x7fb5800d02b0 msgr2=0x7fb5800d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:00.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 -- 172.21.15.158:0/1507506755 shutdown_connections 2023-10-16T11:21:00.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.433+0000 7fb58799a700 1 -- 172.21.15.158:0/1507506755 wait complete. 2023-10-16T11:21:00.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.434+0000 7fb58799a700 1 Processor -- start 2023-10-16T11:21:00.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.434+0000 7fb58799a700 1 -- start start 2023-10-16T11:21:00.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb58799a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb5800cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb58799a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb58006af20 0x7fb580153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb58799a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb58015de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb58799a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580153640 con 0x7fb58006ab40 2023-10-16T11:21:00.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb58799a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58015f1e0 con 0x7fb58006af20 2023-10-16T11:21:00.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb58799a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800cba40 con 0x7fb58006e5d0 2023-10-16T11:21:00.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb585736700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb5800cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb585736700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb5800cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41658/0 (socket says 172.21.15.158:41658) 2023-10-16T11:21:00.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb585736700 1 -- 172.21.15.158:0/1695077748 learned_addr learned my addr 172.21.15.158:0/1695077748 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:00.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb585736700 1 -- 172.21.15.158:0/1695077748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 msgr2=0x7fb58015de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb585736700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb58015de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb585736700 1 -- 172.21.15.158:0/1695077748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb58006af20 msgr2=0x7fb580153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.435+0000 7fb585736700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb58006af20 0x7fb580153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.436+0000 7fb585736700 1 -- 172.21.15.158:0/1695077748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c009550 con 0x7fb58006ab40 2023-10-16T11:21:00.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.436+0000 7fb585736700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb5800cc420 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009c40 tx=0x7fb57c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.436+0000 7fb575ffb700 1 -- 172.21.15.158:0/1695077748 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0036a0 con 0x7fb58006ab40 2023-10-16T11:21:00.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.436+0000 7fb575ffb700 1 -- 172.21.15.158:0/1695077748 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c003800 con 0x7fb58006ab40 2023-10-16T11:21:00.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.436+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb580160c70 con 0x7fb58006ab40 2023-10-16T11:21:00.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.436+0000 7fb575ffb700 1 -- 172.21.15.158:0/1695077748 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c003b70 con 0x7fb58006ab40 2023-10-16T11:21:00.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.436+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5800cb3f0 con 0x7fb58006ab40 2023-10-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.437+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5800681b0 con 0x7fb58006ab40 2023-10-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.437+0000 7fb575ffb700 1 -- 172.21.15.158:0/1695077748 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb57c003cd0 con 0x7fb58006ab40 2023-10-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.438+0000 7fb575ffb700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb56406ad50 0x7fb56406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.438+0000 7fb575ffb700 1 -- 172.21.15.158:0/1695077748 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 8233+0+0 (secure 0 0 0) 0x7fb57c021030 con 0x7fb58006ab40 2023-10-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.438+0000 7fb584f35700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb56406ad50 0x7fb56406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.438+0000 7fb584f35700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb56406ad50 0x7fb56406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb58006d7d0 tx=0x7fb570008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.442+0000 7fb575ffb700 1 -- 172.21.15.158:0/1695077748 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb57c055e10 con 0x7fb58006ab40 2023-10-16T11:21:00.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.634+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5800b6230 con 0x7fb58006ab40 2023-10-16T11:21:00.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.636+0000 7fb575ffb700 1 -- 172.21.15.158:0/1695077748 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+22453 (secure 0 0 0) 0x7fb57c02b070 con 0x7fb58006ab40 2023-10-16T11:21:00.638 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:00.638 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":243,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:20:58.661904+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:20:23.549102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T11:20:24.086127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/937741801":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3443938198":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/872331855":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/1271192867":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/331460130":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/387969810":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/750584278":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2035506903":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:00.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb56406ad50 msgr2=0x7fb56406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb56406ad50 0x7fb56406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb58006d7d0 tx=0x7fb570008040 comp rx=0 tx=0).stop 2023-10-16T11:21:00.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 msgr2=0x7fb5800cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb5800cc420 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009c40 tx=0x7fb57c004750 comp rx=0 tx=0).stop 2023-10-16T11:21:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 shutdown_connections 2023-10-16T11:21:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb56406ad50 0x7fb56406d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb58006e5d0 0x7fb58015de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb58006ab40 0x7fb5800cc420 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 --2- 172.21.15.158:0/1695077748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb58006af20 0x7fb580153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 >> 172.21.15.158:0/1695077748 conn(0x7fb5800d02b0 msgr2=0x7fb5800aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 shutdown_connections 2023-10-16T11:21:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.638+0000 7fb58799a700 1 -- 172.21.15.158:0/1695077748 wait complete. 2023-10-16T11:21:00.653 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:21:00.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.784+0000 7f9e61da3700 1 Processor -- start 2023-10-16T11:21:00.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.784+0000 7f9e61da3700 1 -- start start 2023-10-16T11:21:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e61da3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e5c159a10 0x7f9e5c071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e61da3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c15e220 0x7f9e5c06c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e61da3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9e5c06ccc0 0x7f9e5c0c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e61da3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c06d100 con 0x7f9e5c159a10 2023-10-16T11:21:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e61da3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c0716b0 con 0x7f9e5c06ccc0 2023-10-16T11:21:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e61da3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c071990 con 0x7f9e5c15e220 2023-10-16T11:21:00.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e5b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e5c159a10 0x7f9e5c071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e5b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e5c159a10 0x7f9e5c071010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41684/0 (socket says 172.21.15.158:41684) 2023-10-16T11:21:00.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.785+0000 7f9e5b7fe700 1 -- 172.21.15.158:0/343407321 learned_addr learned my addr 172.21.15.158:0/343407321 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:00.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5affd700 1 --2- 172.21.15.158:0/343407321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c15e220 0x7f9e5c06c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5b7fe700 1 -- 172.21.15.158:0/343407321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c15e220 msgr2=0x7f9e5c06c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:00.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5bfff700 1 --2- 172.21.15.158:0/343407321 >> v2:172.21.15.172:3300/0 conn(0x7f9e5c06ccc0 0x7f9e5c0c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5b7fe700 1 --2- 172.21.15.158:0/343407321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c15e220 0x7f9e5c06c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5b7fe700 1 -- 172.21.15.158:0/343407321 >> v2:172.21.15.172:3300/0 conn(0x7f9e5c06ccc0 msgr2=0x7f9e5c0c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:00.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5b7fe700 1 --2- 172.21.15.158:0/343407321 >> v2:172.21.15.172:3300/0 conn(0x7f9e5c06ccc0 0x7f9e5c0c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5b7fe700 1 -- 172.21.15.158:0/343407321 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c071c70 con 0x7f9e5c159a10 2023-10-16T11:21:00.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5bfff700 1 --2- 172.21.15.158:0/343407321 >> v2:172.21.15.172:3300/0 conn(0x7f9e5c06ccc0 0x7f9e5c0c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5b7fe700 1 --2- 172.21.15.158:0/343407321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c071010 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c15d740 tx=0x7f9e50009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fa73d771afd795 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5a7fc700 1 -- 172.21.15.158:0/343407321 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5000b870 con 0x7f9e5c159a10 2023-10-16T11:21:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5a7fc700 1 -- 172.21.15.158:0/343407321 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5000b9d0 con 0x7f9e5c159a10 2023-10-16T11:21:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.786+0000 7f9e5a7fc700 1 -- 172.21.15.158:0/343407321 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5000bd60 con 0x7f9e5c159a10 2023-10-16T11:21:00.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 -- 172.21.15.158:0/343407321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 msgr2=0x7f9e5c071010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 --2- 172.21.15.158:0/343407321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c071010 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c15d740 tx=0x7f9e50009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 -- 172.21.15.158:0/343407321 shutdown_connections 2023-10-16T11:21:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 --2- 172.21.15.158:0/343407321 >> v2:172.21.15.172:3300/0 conn(0x7f9e5c06ccc0 0x7f9e5c0c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 --2- 172.21.15.158:0/343407321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c071010 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 --2- 172.21.15.158:0/343407321 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c15e220 0x7f9e5c06c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 -- 172.21.15.158:0/343407321 >> 172.21.15.158:0/343407321 conn(0x7f9e5c0cbd30 msgr2=0x7f9e5c0cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 -- 172.21.15.158:0/343407321 shutdown_connections 2023-10-16T11:21:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.787+0000 7f9e61da3700 1 -- 172.21.15.158:0/343407321 wait complete. 2023-10-16T11:21:00.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.788+0000 7f9e61da3700 1 Processor -- start 2023-10-16T11:21:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.788+0000 7f9e61da3700 1 -- start start 2023-10-16T11:21:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e61da3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c06ccc0 0x7f9e5c0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e61da3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c1e3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e61da3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e5c15e220 0x7f9e5c158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e61da3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c1e3b40 con 0x7f9e5c159a10 2023-10-16T11:21:00.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e61da3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c1e4980 con 0x7f9e5c15e220 2023-10-16T11:21:00.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e61da3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c1e3d70 con 0x7f9e5c06ccc0 2023-10-16T11:21:00.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c1e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c1e3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41692/0 (socket says 172.21.15.158:41692) 2023-10-16T11:21:00.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 -- 172.21.15.158:0/1396533517 learned_addr learned my addr 172.21.15.158:0/1396533517 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:00.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5b7fe700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c06ccc0 0x7f9e5c0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5bfff700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e5c15e220 0x7f9e5c158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 -- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c06ccc0 msgr2=0x7f9e5c0b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c06ccc0 0x7f9e5c0b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 -- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e5c15e220 msgr2=0x7f9e5c158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:00.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e5c15e220 0x7f9e5c158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:00.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 -- 172.21.15.158:0/1396533517 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50009550 con 0x7f9e5c159a10 2023-10-16T11:21:00.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e5affd700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c1e3600 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00eb40 tx=0x7f9e4c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e43fff700 1 -- 172.21.15.158:0/1396533517 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c007980 con 0x7f9e5c159a10 2023-10-16T11:21:00.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e43fff700 1 -- 172.21.15.158:0/1396533517 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c007ae0 con 0x7f9e5c159a10 2023-10-16T11:21:00.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c1e4050 con 0x7f9e5c159a10 2023-10-16T11:21:00.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.789+0000 7f9e43fff700 1 -- 172.21.15.158:0/1396533517 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c007e50 con 0x7f9e5c159a10 2023-10-16T11:21:00.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.790+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c1e4610 con 0x7f9e5c159a10 2023-10-16T11:21:00.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.790+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c0681b0 con 0x7f9e5c159a10 2023-10-16T11:21:00.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.791+0000 7f9e43fff700 1 -- 172.21.15.158:0/1396533517 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9e4c010440 con 0x7f9e5c159a10 2023-10-16T11:21:00.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.791+0000 7f9e43fff700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e4806ad50 0x7f9e4806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:00.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.791+0000 7f9e43fff700 1 -- 172.21.15.158:0/1396533517 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 8233+0+0 (secure 0 0 0) 0x7f9e4c014070 con 0x7f9e5c159a10 2023-10-16T11:21:00.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.791+0000 7f9e5b7fe700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e4806ad50 0x7f9e4806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:00.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.792+0000 7f9e5b7fe700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e4806ad50 0x7f9e4806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9e50000c00 tx=0x7f9e50008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:00.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:00.796+0000 7f9e43fff700 1 -- 172.21.15.158:0/1396533517 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e4c0568d0 con 0x7f9e5c159a10 2023-10-16T11:21:01.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.031+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9e5c0b5230 con 0x7f9e5c159a10 2023-10-16T11:21:01.034 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:21:01.032+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:21:01.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.034+0000 7f9e43fff700 1 -- 172.21.15.158:0/1396533517 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v135) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9e4c059ef0 con 0x7f9e5c159a10 2023-10-16T11:21:01.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e4806ad50 msgr2=0x7f9e4806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:01.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e4806ad50 0x7f9e4806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9e50000c00 tx=0x7f9e50008030 comp rx=0 tx=0).stop 2023-10-16T11:21:01.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 msgr2=0x7f9e5c1e3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:01.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c1e3600 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00eb40 tx=0x7f9e4c00ee50 comp rx=0 tx=0).stop 2023-10-16T11:21:01.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 shutdown_connections 2023-10-16T11:21:01.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e4806ad50 0x7f9e4806d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e5c15e220 0x7f9e5c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e5c06ccc0 0x7f9e5c0b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 --2- 172.21.15.158:0/1396533517 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e5c159a10 0x7f9e5c1e3600 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 >> 172.21.15.158:0/1396533517 conn(0x7f9e5c0cbd30 msgr2=0x7f9e5c0c2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:01.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 shutdown_connections 2023-10-16T11:21:01.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.037+0000 7f9e61da3700 1 -- 172.21.15.158:0/1396533517 wait complete. 2023-10-16T11:21:01.041 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:21:01.042 INFO:tasks.ceph.mds.d.smithi172.stderr: -5449> 2023-10-16T11:20:27.709+0000 7f00bcdbf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:21:01.051 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:21:01.074 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:21:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.172+0000 7f30857a7700 1 Processor -- start 2023-10-16T11:21:01.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.172+0000 7f30857a7700 1 -- start start 2023-10-16T11:21:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f30857a7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3080160d30 0x7f30800b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f30857a7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800b7590 0x7f30800d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f30857a7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30800b7a20 con 0x7f30800d0530 2023-10-16T11:21:01.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f30857a7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30800b8910 con 0x7f3080160d30 2023-10-16T11:21:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f30857a7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30800d0ef0 con 0x7f30800b7590 2023-10-16T11:21:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3080160d30 0x7f30800b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307e7fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f30800d0530 0x7f30800d19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41876/0 (socket says 172.21.15.158:41876) 2023-10-16T11:21:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3080160d30 0x7f30800b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41698/0 (socket says 172.21.15.158:41698) 2023-10-16T11:21:01.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307e7fc700 1 -- 172.21.15.158:0/917465400 learned_addr learned my addr 172.21.15.158:0/917465400 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307f7fe700 1 --2- 172.21.15.158:0/917465400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800b7590 0x7f30800d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307effd700 1 -- 172.21.15.158:0/917465400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800b7590 msgr2=0x7f30800d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307effd700 1 --2- 172.21.15.158:0/917465400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800b7590 0x7f30800d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307effd700 1 -- 172.21.15.158:0/917465400 >> v1:172.21.15.158:6789/0 conn(0x7f30800d0530 legacy=0x7f30800d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:01.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307effd700 1 -- 172.21.15.158:0/917465400 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30800b7c90 con 0x7f3080160d30 2023-10-16T11:21:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307dffb700 1 -- 172.21.15.158:0/917465400 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797219320 0 0) 0x7f30800b7a20 con 0x7f30800d0530 2023-10-16T11:21:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307f7fe700 1 --2- 172.21.15.158:0/917465400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800b7590 0x7f30800d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.173+0000 7f307effd700 1 --2- 172.21.15.158:0/917465400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f30800b8f50 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f3074009840 tx=0x7f3074009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0c8ba085143174 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f307dffb700 1 -- 172.21.15.158:0/917465400 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074011830 con 0x7f3080160d30 2023-10-16T11:21:01.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f307dffb700 1 -- 172.21.15.158:0/917465400 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3074011990 con 0x7f3080160d30 2023-10-16T11:21:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f307dffb700 1 -- 172.21.15.158:0/917465400 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074011d20 con 0x7f3080160d30 2023-10-16T11:21:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 -- 172.21.15.158:0/917465400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 msgr2=0x7f30800b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 --2- 172.21.15.158:0/917465400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f30800b8f50 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f3074009840 tx=0x7f3074009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 -- 172.21.15.158:0/917465400 shutdown_connections 2023-10-16T11:21:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 --2- 172.21.15.158:0/917465400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f30800b8f50 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 --2- 172.21.15.158:0/917465400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800b7590 0x7f30800d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 -- 172.21.15.158:0/917465400 >> 172.21.15.158:0/917465400 conn(0x7f30800669f0 msgr2=0x7f3080066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 -- 172.21.15.158:0/917465400 shutdown_connections 2023-10-16T11:21:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.174+0000 7f30857a7700 1 -- 172.21.15.158:0/917465400 wait complete. 2023-10-16T11:21:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.175+0000 7f30857a7700 1 Processor -- start 2023-10-16T11:21:01.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.175+0000 7f30857a7700 1 -- start start 2023-10-16T11:21:01.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f30857a7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30800b7590 0x7f30801f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:01.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f30857a7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800d0530 0x7f30800d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:01.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f30857a7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f308015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:01.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f30857a7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308015cd90 con 0x7f3080160d30 2023-10-16T11:21:01.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f30857a7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30801f33d0 con 0x7f30800b7590 2023-10-16T11:21:01.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f30857a7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30801f36b0 con 0x7f30800d0530 2023-10-16T11:21:01.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f307f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f308015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:01.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f307f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f308015a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41708/0 (socket says 172.21.15.158:41708) 2023-10-16T11:21:01.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f307f7fe700 1 -- 172.21.15.158:0/1414299216 learned_addr learned my addr 172.21.15.158:0/1414299216 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:01.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f307effd700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30800b7590 0x7f30801f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:01.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.176+0000 7f307e7fc700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800d0530 0x7f30800d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:01.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f307f7fe700 1 -- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800d0530 msgr2=0x7f30800d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:01.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f307f7fe700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800d0530 0x7f30800d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f307f7fe700 1 -- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30800b7590 msgr2=0x7f30801f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f307f7fe700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30800b7590 0x7f30801f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f307f7fe700 1 -- 172.21.15.158:0/1414299216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30740094a0 con 0x7f3080160d30 2023-10-16T11:21:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f307f7fe700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f308015a9c0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f3078007ef0 tx=0x7f307800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307800dd80 con 0x7f3080160d30 2023-10-16T11:21:01.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307800f040 con 0x7f3080160d30 2023-10-16T11:21:01.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30801f3990 con 0x7f3080160d30 2023-10-16T11:21:01.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078003c60 con 0x7f3080160d30 2023-10-16T11:21:01.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.177+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30801f3050 con 0x7f3080160d30 2023-10-16T11:21:01.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.178+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308015d270 con 0x7f3080160d30 2023-10-16T11:21:01.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.178+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3078004030 con 0x7f3080160d30 2023-10-16T11:21:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.179+0000 7f30677fe700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f306c06ad50 0x7f306c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.179+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 8319+0+0 (secure 0 0 0) 0x7f3078089d80 con 0x7f3080160d30 2023-10-16T11:21:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.179+0000 7f307effd700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f306c06ad50 0x7f306c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.179+0000 7f307effd700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f306c06ad50 0x7f306c06d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f30740090b0 tx=0x7f307401c040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:01.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.183+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f308015d270 con 0x7f3080160d30 2023-10-16T11:21:01.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.425+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f308015d270 con 0x7f3080160d30 2023-10-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:01.680+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 156) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f3078054a70 con 0x7f3080160d30 2023-10-16T11:21:02.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.035+0000 7f30677fe700 1 -- 172.21.15.158:0/1414299216 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v136) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3078070fd0 con 0x7f3080160d30 2023-10-16T11:21:02.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f306c06ad50 msgr2=0x7f306c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:02.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f306c06ad50 0x7f306c06d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f30740090b0 tx=0x7f307401c040 comp rx=0 tx=0).stop 2023-10-16T11:21:02.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 msgr2=0x7f308015a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:02.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f308015a9c0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f3078007ef0 tx=0x7f307800d3b0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 shutdown_connections 2023-10-16T11:21:02.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f306c06ad50 0x7f306c06d210 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3080160d30 0x7f308015a9c0 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30800b7590 0x7f30801f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 --2- 172.21.15.158:0/1414299216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30800d0530 0x7f30800d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 >> 172.21.15.158:0/1414299216 conn(0x7f30800669f0 msgr2=0x7f308006c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:02.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 shutdown_connections 2023-10-16T11:21:02.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.039+0000 7f30857a7700 1 -- 172.21.15.158:0/1414299216 wait complete. 2023-10-16T11:21:02.054 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:21:02.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.179+0000 7f6713d80700 1 Processor -- start 2023-10-16T11:21:02.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f6713d80700 1 -- start start 2023-10-16T11:21:02.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f6713d80700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15dad0 0x7f670c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:02.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f6713d80700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f670c15deb0 0x7f670c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:02.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f6713d80700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f670c0cea20 0x7f670c0c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:02.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f6713d80700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c0d0ae0 con 0x7f670c0cea20 2023-10-16T11:21:02.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f6713d80700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c0d12d0 con 0x7f670c15deb0 2023-10-16T11:21:02.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f6713d80700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0cbdf0 con 0x7f670c15dad0 2023-10-16T11:21:02.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.180+0000 7f671231d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f670c0cea20 0x7f670c0c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:02.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f670c0cea20 0x7f670c0c3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41716/0 (socket says 172.21.15.158:41716) 2023-10-16T11:21:02.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 -- 172.21.15.158:0/1183691405 learned_addr learned my addr 172.21.15.158:0/1183691405 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:02.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 -- 172.21.15.158:0/1183691405 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15dad0 msgr2=0x7f670c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:02.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 --2- 172.21.15.158:0/1183691405 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15dad0 0x7f670c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 -- 172.21.15.158:0/1183691405 >> v2:172.21.15.172:3300/0 conn(0x7f670c15deb0 msgr2=0x7f670c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:02.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671131b700 1 --2- 172.21.15.158:0/1183691405 >> v2:172.21.15.172:3300/0 conn(0x7f670c15deb0 0x7f670c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:02.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 --2- 172.21.15.158:0/1183691405 >> v2:172.21.15.172:3300/0 conn(0x7f670c15deb0 0x7f670c0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 -- 172.21.15.158:0/1183691405 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c0cc280 con 0x7f670c0cea20 2023-10-16T11:21:02.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f671231d700 1 --2- 172.21.15.158:0/1183691405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c0c3f20 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f670c071580 tx=0x7f66fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181d5a79fdc8efa7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:02.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f6710b1a700 1 -- 172.21.15.158:0/1183691405 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc00b870 con 0x7f670c0cea20 2023-10-16T11:21:02.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f6710b1a700 1 -- 172.21.15.158:0/1183691405 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66fc00b9d0 con 0x7f670c0cea20 2023-10-16T11:21:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.181+0000 7f6710b1a700 1 -- 172.21.15.158:0/1183691405 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc00bd60 con 0x7f670c0cea20 2023-10-16T11:21:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 -- 172.21.15.158:0/1183691405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 msgr2=0x7f670c0c3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 --2- 172.21.15.158:0/1183691405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c0c3f20 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f670c071580 tx=0x7f66fc009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 -- 172.21.15.158:0/1183691405 shutdown_connections 2023-10-16T11:21:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 --2- 172.21.15.158:0/1183691405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c0c3f20 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 --2- 172.21.15.158:0/1183691405 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15dad0 0x7f670c0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 --2- 172.21.15.158:0/1183691405 >> v2:172.21.15.172:3300/0 conn(0x7f670c15deb0 0x7f670c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 -- 172.21.15.158:0/1183691405 >> 172.21.15.158:0/1183691405 conn(0x7f670c099a10 msgr2=0x7f670c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 -- 172.21.15.158:0/1183691405 shutdown_connections 2023-10-16T11:21:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.182+0000 7f6713d80700 1 -- 172.21.15.158:0/1183691405 wait complete. 2023-10-16T11:21:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.183+0000 7f6713d80700 1 Processor -- start 2023-10-16T11:21:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.183+0000 7f6713d80700 1 -- start start 2023-10-16T11:21:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f670c15dad0 0x7f670c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15deb0 0x7f670c153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c155280 con 0x7f670c0cea20 2023-10-16T11:21:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c1543c0 con 0x7f670c15dad0 2023-10-16T11:21:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0c5f20 con 0x7f670c15deb0 2023-10-16T11:21:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41718/0 (socket says 172.21.15.158:41718) 2023-10-16T11:21:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 -- 172.21.15.158:0/3600234771 learned_addr learned my addr 172.21.15.158:0/3600234771 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f671231d700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15deb0 0x7f670c153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f671131b700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f670c15dad0 0x7f670c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 -- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15deb0 msgr2=0x7f670c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15deb0 0x7f670c153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 -- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f670c15dad0 msgr2=0x7f670c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f670c15dad0 0x7f670c0c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 -- 172.21.15.158:0/3600234771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc009550 con 0x7f670c0cea20 2023-10-16T11:21:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f671231d700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15deb0 0x7f670c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f671131b700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f670c15dad0 0x7f670c0c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6711b1c700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c157100 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f670400ba50 tx=0x7f670400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f670a7fc700 1 -- 172.21.15.158:0/3600234771 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400cab0 con 0x7f670c0cea20 2023-10-16T11:21:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f670a7fc700 1 -- 172.21.15.158:0/3600234771 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400cc30 con 0x7f670c0cea20 2023-10-16T11:21:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670c0c6200 con 0x7f670c0cea20 2023-10-16T11:21:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f670a7fc700 1 -- 172.21.15.158:0/3600234771 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704014440 con 0x7f670c0cea20 2023-10-16T11:21:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.184+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670c155490 con 0x7f670c0cea20 2023-10-16T11:21:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.185+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670c0d0fb0 con 0x7f670c0cea20 2023-10-16T11:21:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.186+0000 7f670a7fc700 1 -- 172.21.15.158:0/3600234771 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 156) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f670401d3b0 con 0x7f670c0cea20 2023-10-16T11:21:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.186+0000 7f670a7fc700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f66f806b0b0 0x7f66f806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.186+0000 7f670a7fc700 1 -- 172.21.15.158:0/3600234771 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 8319+0+0 (secure 0 0 0) 0x7f6704006e40 con 0x7f670c0cea20 2023-10-16T11:21:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.186+0000 7f671131b700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f66f806b0b0 0x7f66f806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.187+0000 7f671131b700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f66f806b0b0 0x7f66f806d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6700006fd0 tx=0x7f6700008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.190+0000 7f670a7fc700 1 -- 172.21.15.158:0/3600234771 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6704054530 con 0x7f670c0cea20 2023-10-16T11:21:02.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:02.383+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f670c0681b0 con 0x7f670c0cea20 2023-10-16T11:21:03.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.161+0000 7f670a7fc700 1 -- 172.21.15.158:0/3600234771 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v245) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6704019070 con 0x7f670c0cea20 2023-10-16T11:21:03.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.166+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f66f806b0b0 msgr2=0x7f66f806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:03.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.166+0000 7f6713d80700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f66f806b0b0 0x7f66f806d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6700006fd0 tx=0x7f6700008040 comp rx=0 tx=0).stop 2023-10-16T11:21:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.166+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 msgr2=0x7f670c157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.166+0000 7f6713d80700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c157100 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f670400ba50 tx=0x7f670400be10 comp rx=0 tx=0).stop 2023-10-16T11:21:03.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.168+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 shutdown_connections 2023-10-16T11:21:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.168+0000 7f6713d80700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f66f806b0b0 0x7f66f806d570 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.168+0000 7f6713d80700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f670c15deb0 0x7f670c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.168+0000 7f6713d80700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f670c0cea20 0x7f670c157100 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.168+0000 7f6713d80700 1 --2- 172.21.15.158:0/3600234771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f670c15dad0 0x7f670c0c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.168+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 >> 172.21.15.158:0/3600234771 conn(0x7f670c099a10 msgr2=0x7f670c06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:03.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.170+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 shutdown_connections 2023-10-16T11:21:03.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.170+0000 7f6713d80700 1 -- 172.21.15.158:0/3600234771 wait complete. 2023-10-16T11:21:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:21:03.188 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:21:03.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.315+0000 7f411fe5d700 1 Processor -- start 2023-10-16T11:21:03.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.316+0000 7f411fe5d700 1 -- start start 2023-10-16T11:21:03.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.316+0000 7f411fe5d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f411815d6c0 0x7f41180cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:03.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.316+0000 7f411fe5d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41180cf360 0x7f41180d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:03.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.316+0000 7f411fe5d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180d1ee0 con 0x7f411815daa0 2023-10-16T11:21:03.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.316+0000 7f411fe5d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41180d40a0 con 0x7f411815d6c0 2023-10-16T11:21:03.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.316+0000 7f411fe5d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41180d12d0 con 0x7f41180cf360 2023-10-16T11:21:03.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f411815d6c0 0x7f41180cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:03.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f411815d6c0 0x7f41180cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41726/0 (socket says 172.21.15.158:41726) 2023-10-16T11:21:03.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 -- 172.21.15.158:0/594626498 learned_addr learned my addr 172.21.15.158:0/594626498 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:03.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 -- 172.21.15.158:0/594626498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41180cf360 msgr2=0x7f41180d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:03.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 --2- 172.21.15.158:0/594626498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41180cf360 0x7f41180d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 -- 172.21.15.158:0/594626498 >> v1:172.21.15.172:6789/0 conn(0x7f411815daa0 legacy=0x7f41180cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:03.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 -- 172.21.15.158:0/594626498 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180d15b0 con 0x7f411815d6c0 2023-10-16T11:21:03.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411dbf9700 1 --2- 172.21.15.158:0/594626498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411815d6c0 0x7f41180cff30 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f41080098f0 tx=0x7f4108009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467652786a68cf35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:03.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411cbf7700 1 -- 172.21.15.158:0/594626498 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4108004230 con 0x7f411815d6c0 2023-10-16T11:21:03.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411cbf7700 1 -- 172.21.15.158:0/594626498 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4108004390 con 0x7f411815d6c0 2023-10-16T11:21:03.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.317+0000 7f411cbf7700 1 -- 172.21.15.158:0/594626498 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41080036a0 con 0x7f411815d6c0 2023-10-16T11:21:03.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 -- 172.21.15.158:0/594626498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411815d6c0 msgr2=0x7f41180cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:03.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 --2- 172.21.15.158:0/594626498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411815d6c0 0x7f41180cff30 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f41080098f0 tx=0x7f4108009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:03.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 -- 172.21.15.158:0/594626498 shutdown_connections 2023-10-16T11:21:03.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 --2- 172.21.15.158:0/594626498 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f41180cf360 0x7f41180d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 --2- 172.21.15.158:0/594626498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411815d6c0 0x7f41180cff30 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 -- 172.21.15.158:0/594626498 >> 172.21.15.158:0/594626498 conn(0x7f411809c940 msgr2=0x7f41180d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:03.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 -- 172.21.15.158:0/594626498 shutdown_connections 2023-10-16T11:21:03.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.318+0000 7f411fe5d700 1 -- 172.21.15.158:0/594626498 wait complete. 2023-10-16T11:21:03.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.319+0000 7f411fe5d700 1 Processor -- start 2023-10-16T11:21:03.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.319+0000 7f411fe5d700 1 -- start start 2023-10-16T11:21:03.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411fe5d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41180cf360 0x7f41180b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:03.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411fe5d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f411815d6c0 0x7f4118157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:03.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411fe5d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f411815daa0 0x7f41180b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:03.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411fe5d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41181f3de0 con 0x7f41180cf360 2023-10-16T11:21:03.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411fe5d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41181f3260 con 0x7f411815daa0 2023-10-16T11:21:03.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411fe5d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41181f3540 con 0x7f411815d6c0 2023-10-16T11:21:03.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41180cf360 0x7f41180b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:03.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41180cf360 0x7f41180b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41732/0 (socket says 172.21.15.158:41732) 2023-10-16T11:21:03.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 -- 172.21.15.158:0/2781712314 learned_addr learned my addr 172.21.15.158:0/2781712314 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:03.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411e3fa700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f411815daa0 0x7f41180b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:03.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411d3f8700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f411815d6c0 0x7f4118157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:03.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 -- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f411815d6c0 msgr2=0x7f4118157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:03.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f411815d6c0 0x7f4118157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 -- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f411815daa0 msgr2=0x7f41180b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:03.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f411815daa0 0x7f41180b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:03.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 -- 172.21.15.158:0/2781712314 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108009550 con 0x7f41180cf360 2023-10-16T11:21:03.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411d3f8700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f411815d6c0 0x7f4118157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:03.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411dbf9700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41180cf360 0x7f41180b0bb0 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f411808d7d0 tx=0x7f4108004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:03.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f410e7fc700 1 -- 172.21.15.158:0/2781712314 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4108003e70 con 0x7f41180cf360 2023-10-16T11:21:03.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f410e7fc700 1 -- 172.21.15.158:0/2781712314 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410801f070 con 0x7f41180cf360 2023-10-16T11:21:03.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41181f3820 con 0x7f41180cf360 2023-10-16T11:21:03.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.320+0000 7f410e7fc700 1 -- 172.21.15.158:0/2781712314 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410800b8a0 con 0x7f41180cf360 2023-10-16T11:21:03.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.321+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4118157ff0 con 0x7f41180cf360 2023-10-16T11:21:03.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.321+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41180cbe30 con 0x7f41180cf360 2023-10-16T11:21:03.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.322+0000 7f410e7fc700 1 -- 172.21.15.158:0/2781712314 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4108021030 con 0x7f41180cf360 2023-10-16T11:21:03.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.322+0000 7f410e7fc700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f40fc06b140 0x7f40fc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:03.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.322+0000 7f410e7fc700 1 -- 172.21.15.158:0/2781712314 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 7881+0+0 (secure 0 0 0) 0x7f410808baa0 con 0x7f41180cf360 2023-10-16T11:21:03.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.322+0000 7f411d3f8700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f40fc06b140 0x7f40fc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:03.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.323+0000 7f411d3f8700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f40fc06b140 0x7f40fc06d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4114006fd0 tx=0x7f4114008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:03.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.327+0000 7f410e7fc700 1 -- 172.21.15.158:0/2781712314 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f41180cbe30 con 0x7f41180cf360 2023-10-16T11:21:03.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:03.525+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f41180681b0 con 0x7f41180cf360 2023-10-16T11:21:04.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.168+0000 7f410e7fc700 1 -- 172.21.15.158:0/2781712314 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v246) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4108056030 con 0x7f41180cf360 2023-10-16T11:21:04.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f40fc06b140 msgr2=0x7f40fc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f40fc06b140 0x7f40fc06d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4114006fd0 tx=0x7f4114008040 comp rx=0 tx=0).stop 2023-10-16T11:21:04.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41180cf360 msgr2=0x7f41180b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41180cf360 0x7f41180b0bb0 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f411808d7d0 tx=0x7f4108004750 comp rx=0 tx=0).stop 2023-10-16T11:21:04.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 shutdown_connections 2023-10-16T11:21:04.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f40fc06b140 0x7f40fc06d600 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f411815daa0 0x7f41180b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41180cf360 0x7f41180b0bb0 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 --2- 172.21.15.158:0/2781712314 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f411815d6c0 0x7f4118157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 >> 172.21.15.158:0/2781712314 conn(0x7f411809c940 msgr2=0x7f41180cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:04.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 shutdown_connections 2023-10-16T11:21:04.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.172+0000 7f411fe5d700 1 -- 172.21.15.158:0/2781712314 wait complete. 2023-10-16T11:21:04.176 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:21:04.189 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.317+0000 7f2f1daeb700 1 Processor -- start 2023-10-16T11:21:04.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.317+0000 7f2f1daeb700 1 -- start start 2023-10-16T11:21:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.317+0000 7f2f1daeb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f180cc5d0 0x7f2f1806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.317+0000 7f2f1daeb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 0x7f2f180cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.317+0000 7f2f1daeb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1806e770 con 0x7f2f1806f340 2023-10-16T11:21:04.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.317+0000 7f2f1daeb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f180cf270 con 0x7f2f180cc5d0 2023-10-16T11:21:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.317+0000 7f2f1daeb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1815e4e0 con 0x7f2f1815dfb0 2023-10-16T11:21:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f17fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2f1806f340 0x7f2f18071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41920/0 (socket says 172.21.15.158:41920) 2023-10-16T11:21:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f17fff700 1 -- 172.21.15.158:0/3898431668 learned_addr learned my addr 172.21.15.158:0/3898431668 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558914214 0 0) 0x7f2f10001490 con 0x7f2f1806f340 2023-10-16T11:21:04.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f177fe700 1 --2- 172.21.15.158:0/3898431668 >> v2:172.21.15.172:3300/0 conn(0x7f2f180cc5d0 0x7f2f1806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f16ffd700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 0x7f2f180cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f000035b0 con 0x7f2f1806f340 2023-10-16T11:21:04.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476209555 0 0) 0x7f2f000035b0 con 0x7f2f1806f340 2023-10-16T11:21:04.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 msgr2=0x7f2f180cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:04.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 0x7f2f180cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 >> v2:172.21.15.172:3300/0 conn(0x7f2f180cc5d0 msgr2=0x7f2f1806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:04.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 --2- 172.21.15.158:0/3898431668 >> v2:172.21.15.172:3300/0 conn(0x7f2f180cc5d0 0x7f2f1806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1815e7c0 con 0x7f2f1806f340 2023-10-16T11:21:04.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f2f10001b80 con 0x7f2f1806f340 2023-10-16T11:21:04.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 >> v1:172.21.15.158:6789/0 conn(0x7f2f1806f340 legacy=0x7f2f18071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f00003a00 0x7f2f00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 0x7f2f00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f00004d90 0x7f2f00005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180cf270 con 0x7f2f00004d90 2023-10-16T11:21:04.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1815e4e0 con 0x7f2f00003a00 2023-10-16T11:21:04.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0000ba10 con 0x7f2f1815dfb0 2023-10-16T11:21:04.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f17fff700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f00004d90 0x7f2f00005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f177fe700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f00003a00 0x7f2f00003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.318+0000 7f2f16ffd700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 0x7f2f00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f17fff700 1 -- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 msgr2=0x7f2f00004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:04.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f17fff700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1815dfb0 0x7f2f00004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f17fff700 1 -- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f00003a00 msgr2=0x7f2f00003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:04.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f17fff700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f00003a00 0x7f2f00003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f17fff700 1 -- 172.21.15.158:0/3898431668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000035b0 con 0x7f2f00004d90 2023-10-16T11:21:04.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f17fff700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f00004d90 0x7f2f00005210 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000a830 tx=0x7f2f00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8c38bc21935886 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:04.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10012600 con 0x7f2f00004d90 2023-10-16T11:21:04.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f100127a0 con 0x7f2f00004d90 2023-10-16T11:21:04.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f167fc700 1 -- 172.21.15.158:0/3898431668 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10012b50 con 0x7f2f00004d90 2023-10-16T11:21:04.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f1daeb700 1 -- 172.21.15.158:0/3898431668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f00004d90 msgr2=0x7f2f00005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f1daeb700 1 --2- 172.21.15.158:0/3898431668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f00004d90 0x7f2f00005210 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000a830 tx=0x7f2f00005880 comp rx=0 tx=0).stop 2023-10-16T11:21:04.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f17fff700 1 -- 172.21.15.158:0/3898431668 reap_dead start 2023-10-16T11:21:04.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.319+0000 7f2f1daeb700 1 -- 172.21.15.158:0/3898431668 shutdown_connections 2023-10-16T11:21:04.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.320+0000 7f2f1daeb700 1 -- 172.21.15.158:0/3898431668 >> 172.21.15.158:0/3898431668 conn(0x7f2f180d02b0 msgr2=0x7f2f180d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:04.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.320+0000 7f2f1daeb700 1 -- 172.21.15.158:0/3898431668 shutdown_connections 2023-10-16T11:21:04.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.320+0000 7f2f1daeb700 1 -- 172.21.15.158:0/3898431668 wait complete. 2023-10-16T11:21:04.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.321+0000 7f2f1daeb700 1 Processor -- start 2023-10-16T11:21:04.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.321+0000 7f2f1daeb700 1 -- start start 2023-10-16T11:21:04.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f1daeb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1814f5f0 0x7f2f180c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f1daeb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f180b2ea0 0x7f2f180c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f1daeb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f180d06c0 0x7f2f180d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f1daeb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180b3320 con 0x7f2f180b2ea0 2023-10-16T11:21:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f1daeb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180d1080 con 0x7f2f180d06c0 2023-10-16T11:21:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f1daeb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180c78c0 con 0x7f2f1814f5f0 2023-10-16T11:21:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f180b2ea0 0x7f2f180c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f180b2ea0 0x7f2f180c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41754/0 (socket says 172.21.15.158:41754) 2023-10-16T11:21:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 -- 172.21.15.158:0/998098119 learned_addr learned my addr 172.21.15.158:0/998098119 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f177fe700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1814f5f0 0x7f2f180c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f17fff700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f180d06c0 0x7f2f180d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 -- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1814f5f0 msgr2=0x7f2f180c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1814f5f0 0x7f2f180c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 -- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f180d06c0 msgr2=0x7f2f180d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f180d06c0 0x7f2f180d0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 -- 172.21.15.158:0/998098119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f180c7af0 con 0x7f2f180b2ea0 2023-10-16T11:21:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f16ffd700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f180b2ea0 0x7f2f180c7380 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f2f080119c0 tx=0x7f2f08011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2efffff700 1 -- 172.21.15.158:0/998098119 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08011e50 con 0x7f2f180b2ea0 2023-10-16T11:21:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2efffff700 1 -- 172.21.15.158:0/998098119 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08004240 con 0x7f2f180b2ea0 2023-10-16T11:21:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f180c7dd0 con 0x7f2f180b2ea0 2023-10-16T11:21:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.322+0000 7f2efffff700 1 -- 172.21.15.158:0/998098119 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0800f730 con 0x7f2f180b2ea0 2023-10-16T11:21:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.323+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f180c4580 con 0x7f2f180b2ea0 2023-10-16T11:21:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.323+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f180681b0 con 0x7f2f180b2ea0 2023-10-16T11:21:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.324+0000 7f2efffff700 1 -- 172.21.15.158:0/998098119 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2f0800f890 con 0x7f2f180b2ea0 2023-10-16T11:21:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.324+0000 7f2efffff700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2f0406b140 0x7f2f0406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.325+0000 7f2f177fe700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2f0406b140 0x7f2f0406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.325+0000 7f2efffff700 1 -- 172.21.15.158:0/998098119 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f2f0808a510 con 0x7f2f180b2ea0 2023-10-16T11:21:04.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.325+0000 7f2f177fe700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2f0406b140 0x7f2f0406d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009c60 tx=0x7f2f0c009210 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:04.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.329+0000 7f2efffff700 1 -- 172.21.15.158:0/998098119 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2f08054b80 con 0x7f2f180b2ea0 2023-10-16T11:21:04.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.521+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f180b6230 con 0x7f2f180b2ea0 2023-10-16T11:21:04.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.523+0000 7f2efffff700 1 -- 172.21.15.158:0/998098119 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+18871 (secure 0 0 0) 0x7f2f080581a0 con 0x7f2f180b2ea0 2023-10-16T11:21:04.525 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:04.525 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":246,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:04.162755+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6834/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/937741801":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3443938198":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/872331855":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/1271192867":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/331460130":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/387969810":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.172:6832/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/750584278":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2035506903":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:04.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2f0406b140 msgr2=0x7f2f0406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2f0406b140 0x7f2f0406d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009c60 tx=0x7f2f0c009210 comp rx=0 tx=0).stop 2023-10-16T11:21:04.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f180b2ea0 msgr2=0x7f2f180c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f180b2ea0 0x7f2f180c7380 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f2f080119c0 tx=0x7f2f08011cd0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 shutdown_connections 2023-10-16T11:21:04.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2f0406b140 0x7f2f0406d600 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f180d06c0 0x7f2f180d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f1814f5f0 0x7f2f180c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 --2- 172.21.15.158:0/998098119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f180b2ea0 0x7f2f180c7380 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 >> 172.21.15.158:0/998098119 conn(0x7f2f180d02b0 msgr2=0x7f2f18150200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:04.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 shutdown_connections 2023-10-16T11:21:04.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.525+0000 7f2f1daeb700 1 -- 172.21.15.158:0/998098119 wait complete. 2023-10-16T11:21:04.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T11:20:35.862082+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '240', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T11:20:37.182868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '241', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:21:04.540 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:21:04.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.662+0000 7f92d103f700 1 Processor -- start 2023-10-16T11:21:04.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.662+0000 7f92d103f700 1 -- start start 2023-10-16T11:21:04.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92d103f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92cc15dad0 0x7f92cc0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92d103f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15deb0 0x7f92cc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92d103f700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0d0ae0 con 0x7f92cc0cea20 2023-10-16T11:21:04.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92d103f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc0d12d0 con 0x7f92cc15dad0 2023-10-16T11:21:04.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92d103f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0cbdf0 con 0x7f92cc15deb0 2023-10-16T11:21:04.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92cad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92cc15dad0 0x7f92cc0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92cad9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92cc15dad0 0x7f92cc0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41760/0 (socket says 172.21.15.158:41760) 2023-10-16T11:21:04.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92cad9d700 1 -- 172.21.15.158:0/3536562496 learned_addr learned my addr 172.21.15.158:0/3536562496 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:04.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92cad9d700 1 -- 172.21.15.158:0/3536562496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15deb0 msgr2=0x7f92cc0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:04.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92cad9d700 1 --2- 172.21.15.158:0/3536562496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15deb0 0x7f92cc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92cad9d700 1 -- 172.21.15.158:0/3536562496 >> v1:172.21.15.172:6789/0 conn(0x7f92cc0cea20 legacy=0x7f92cc0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:04.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.663+0000 7f92cad9d700 1 -- 172.21.15.158:0/3536562496 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0cc280 con 0x7f92cc15dad0 2023-10-16T11:21:04.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92cad9d700 1 --2- 172.21.15.158:0/3536562496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc15dad0 0x7f92cc0cdfa0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0098f0 tx=0x7f92bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b2d092666ebdf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:04.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92c9d9b700 1 -- 172.21.15.158:0/3536562496 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc004230 con 0x7f92cc15dad0 2023-10-16T11:21:04.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92c9d9b700 1 -- 172.21.15.158:0/3536562496 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc004390 con 0x7f92cc15dad0 2023-10-16T11:21:04.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92c9d9b700 1 -- 172.21.15.158:0/3536562496 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc0036a0 con 0x7f92cc15dad0 2023-10-16T11:21:04.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 -- 172.21.15.158:0/3536562496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc15dad0 msgr2=0x7f92cc0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 --2- 172.21.15.158:0/3536562496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc15dad0 0x7f92cc0cdfa0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0098f0 tx=0x7f92bc009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:04.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 -- 172.21.15.158:0/3536562496 shutdown_connections 2023-10-16T11:21:04.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 --2- 172.21.15.158:0/3536562496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc15dad0 0x7f92cc0cdfa0 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 --2- 172.21.15.158:0/3536562496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15deb0 0x7f92cc0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 -- 172.21.15.158:0/3536562496 >> 172.21.15.158:0/3536562496 conn(0x7f92cc099a10 msgr2=0x7f92cc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:04.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 -- 172.21.15.158:0/3536562496 shutdown_connections 2023-10-16T11:21:04.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.664+0000 7f92d103f700 1 -- 172.21.15.158:0/3536562496 wait complete. 2023-10-16T11:21:04.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.666+0000 7f92d103f700 1 Processor -- start 2023-10-16T11:21:04.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.666+0000 7f92d103f700 1 -- start start 2023-10-16T11:21:04.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc0cea20 0x7f92cc159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15dad0 0x7f92cc0c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92cc15deb0 0x7f92cc14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc14fdc0 con 0x7f92cc0cea20 2023-10-16T11:21:04.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc15b5c0 con 0x7f92cc15deb0 2023-10-16T11:21:04.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc15a470 con 0x7f92cc15dad0 2023-10-16T11:21:04.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc0cea20 0x7f92cc159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc0cea20 0x7f92cc159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41766/0 (socket says 172.21.15.158:41766) 2023-10-16T11:21:04.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 -- 172.21.15.158:0/1973260616 learned_addr learned my addr 172.21.15.158:0/1973260616 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:04.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cb59e700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92cc15deb0 0x7f92cc14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92ca59c700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15dad0 0x7f92cc0c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 -- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15dad0 msgr2=0x7f92cc0c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15dad0 0x7f92cc0c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 -- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92cc15deb0 msgr2=0x7f92cc14c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92cc15deb0 0x7f92cc14c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 -- 172.21.15.158:0/1973260616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc009550 con 0x7f92cc0cea20 2023-10-16T11:21:04.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cb59e700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92cc15deb0 0x7f92cc14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:04.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92cad9d700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc0cea20 0x7f92cc159e90 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f92cc070430 tx=0x7f92bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:04.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92b37fe700 1 -- 172.21.15.158:0/1973260616 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc003e30 con 0x7f92cc0cea20 2023-10-16T11:21:04.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92b37fe700 1 -- 172.21.15.158:0/1973260616 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc01f070 con 0x7f92cc0cea20 2023-10-16T11:21:04.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc15a750 con 0x7f92cc0cea20 2023-10-16T11:21:04.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92b37fe700 1 -- 172.21.15.158:0/1973260616 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc00b8a0 con 0x7f92cc0cea20 2023-10-16T11:21:04.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.667+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc15ad10 con 0x7f92cc0cea20 2023-10-16T11:21:04.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.668+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc0681b0 con 0x7f92cc0cea20 2023-10-16T11:21:04.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.669+0000 7f92b37fe700 1 -- 172.21.15.158:0/1973260616 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f92bc021030 con 0x7f92cc0cea20 2023-10-16T11:21:04.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.669+0000 7f92b37fe700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f92b806b140 0x7f92b806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:04.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.669+0000 7f92b37fe700 1 -- 172.21.15.158:0/1973260616 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f92bc08b760 con 0x7f92cc0cea20 2023-10-16T11:21:04.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.669+0000 7f92ca59c700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f92b806b140 0x7f92b806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:04.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.670+0000 7f92ca59c700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f92b806b140 0x7f92b806d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92c0006fd0 tx=0x7f92c0008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:04.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.674+0000 7f92b37fe700 1 -- 172.21.15.158:0/1973260616 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f92bc055eb0 con 0x7f92cc0cea20 2023-10-16T11:21:04.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.867+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f92cc0681b0 con 0x7f92cc0cea20 2023-10-16T11:21:04.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.867+0000 7f92b37fe700 1 -- 172.21.15.158:0/1973260616 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v246) v1 ==== 93+0+10 (secure 0 0 0) 0x7f92bc0594d0 con 0x7f92cc0cea20 2023-10-16T11:21:04.869 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:21:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f92b806b140 msgr2=0x7f92b806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f92b806b140 0x7f92b806d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92c0006fd0 tx=0x7f92c0008040 comp rx=0 tx=0).stop 2023-10-16T11:21:04.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc0cea20 msgr2=0x7f92cc159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc0cea20 0x7f92cc159e90 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f92cc070430 tx=0x7f92bc004750 comp rx=0 tx=0).stop 2023-10-16T11:21:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 shutdown_connections 2023-10-16T11:21:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f92b806b140 0x7f92b806d600 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92cc15deb0 0x7f92cc14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92cc0cea20 0x7f92cc159e90 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 --2- 172.21.15.158:0/1973260616 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92cc15dad0 0x7f92cc0c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 >> 172.21.15.158:0/1973260616 conn(0x7f92cc099a10 msgr2=0x7f92cc152b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 shutdown_connections 2023-10-16T11:21:04.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:04.869+0000 7f92d103f700 1 -- 172.21.15.158:0/1973260616 wait complete. 2023-10-16T11:21:04.884 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T11:21:05.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.009+0000 7fb7adaa7700 1 Processor -- start 2023-10-16T11:21:05.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb7adaa7700 1 -- start start 2023-10-16T11:21:05.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb7adaa7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb7a806afe0 0x7fb7a806df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb7adaa7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb7a815dfc0 0x7fb7a80718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb7adaa7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a815e510 con 0x7fb7a815dbd0 2023-10-16T11:21:05.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb7adaa7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a80ccbb0 con 0x7fb7a806afe0 2023-10-16T11:21:05.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb7adaa7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a815e7f0 con 0x7fb7a815dfc0 2023-10-16T11:21:05.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb79ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb7a806afe0 0x7fb7a806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.010+0000 7fb7a77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb7a815dbd0 0x7fb7a806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41950/0 (socket says 172.21.15.158:41950) 2023-10-16T11:21:05.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb79ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb7a806afe0 0x7fb7a806df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41776/0 (socket says 172.21.15.158:41776) 2023-10-16T11:21:05.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb7a77fe700 1 -- 172.21.15.158:0/120647469 learned_addr learned my addr 172.21.15.158:0/120647469 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:05.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb7a7fff700 1 --2- 172.21.15.158:0/120647469 >> v2:172.21.15.172:3300/0 conn(0x7fb7a815dfc0 0x7fb7a80718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb79ffff700 1 -- 172.21.15.158:0/120647469 >> v2:172.21.15.172:3300/0 conn(0x7fb7a815dfc0 msgr2=0x7fb7a80718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:05.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb79ffff700 1 --2- 172.21.15.158:0/120647469 >> v2:172.21.15.172:3300/0 conn(0x7fb7a815dfc0 0x7fb7a80718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb79ffff700 1 -- 172.21.15.158:0/120647469 >> v1:172.21.15.158:6789/0 conn(0x7fb7a815dbd0 legacy=0x7fb7a806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb79ffff700 1 -- 172.21.15.158:0/120647469 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a815ead0 con 0x7fb7a806afe0 2023-10-16T11:21:05.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb7a6ffd700 1 -- 172.21.15.158:0/120647469 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660684631 0 0) 0x7fb7a815e510 con 0x7fb7a815dbd0 2023-10-16T11:21:05.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb79ffff700 1 --2- 172.21.15.158:0/120647469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a806df00 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fb7a806d9b0 tx=0x7fb790009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e867ef585fabf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb7a6ffd700 1 -- 172.21.15.158:0/120647469 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790011830 con 0x7fb7a806afe0 2023-10-16T11:21:05.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb7a6ffd700 1 -- 172.21.15.158:0/120647469 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb790011990 con 0x7fb7a806afe0 2023-10-16T11:21:05.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.011+0000 7fb7a6ffd700 1 -- 172.21.15.158:0/120647469 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790011d20 con 0x7fb7a806afe0 2023-10-16T11:21:05.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 -- 172.21.15.158:0/120647469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 msgr2=0x7fb7a806df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/120647469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a806df00 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fb7a806d9b0 tx=0x7fb790009b20 comp rx=0 tx=0).stop 2023-10-16T11:21:05.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 -- 172.21.15.158:0/120647469 shutdown_connections 2023-10-16T11:21:05.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/120647469 >> v2:172.21.15.172:3300/0 conn(0x7fb7a815dfc0 0x7fb7a80718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/120647469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a806df00 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 -- 172.21.15.158:0/120647469 >> 172.21.15.158:0/120647469 conn(0x7fb7a80b1f00 msgr2=0x7fb7a80b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:05.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 -- 172.21.15.158:0/120647469 shutdown_connections 2023-10-16T11:21:05.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.012+0000 7fb7adaa7700 1 -- 172.21.15.158:0/120647469 wait complete. 2023-10-16T11:21:05.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.013+0000 7fb7adaa7700 1 Processor -- start 2023-10-16T11:21:05.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.013+0000 7fb7adaa7700 1 -- start start 2023-10-16T11:21:05.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a81f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7a815dfc0 0x7fb7a80acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7a80c9780 0x7fb7a80c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80aeb10 con 0x7fb7a806afe0 2023-10-16T11:21:05.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80adf00 con 0x7fb7a80c9780 2023-10-16T11:21:05.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80ae1e0 con 0x7fb7a815dfc0 2023-10-16T11:21:05.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a81f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a81f3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41780/0 (socket says 172.21.15.158:41780) 2023-10-16T11:21:05.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 -- 172.21.15.158:0/2377878100 learned_addr learned my addr 172.21.15.158:0/2377878100 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:05.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a7fff700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7a80c9780 0x7fb7a80c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb79ffff700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7a815dfc0 0x7fb7a80acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 -- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7a815dfc0 msgr2=0x7fb7a80acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7a815dfc0 0x7fb7a80acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 -- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7a80c9780 msgr2=0x7fb7a80c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7a80c9780 0x7fb7a80c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 -- 172.21.15.158:0/2377878100 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7900094a0 con 0x7fb7a806afe0 2023-10-16T11:21:05.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb79ffff700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7a815dfc0 0x7fb7a80acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:05.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a77fe700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a81f3880 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fb7980011e0 tx=0x7fb79800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a4ff9700 1 -- 172.21.15.158:0/2377878100 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798004c80 con 0x7fb7a806afe0 2023-10-16T11:21:05.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a4ff9700 1 -- 172.21.15.158:0/2377878100 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb798004de0 con 0x7fb7a806afe0 2023-10-16T11:21:05.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a80ae4c0 con 0x7fb7a806afe0 2023-10-16T11:21:05.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7a4ff9700 1 -- 172.21.15.158:0/2377878100 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798007310 con 0x7fb7a806afe0 2023-10-16T11:21:05.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.014+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a81f3dc0 con 0x7fb7a806afe0 2023-10-16T11:21:05.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.015+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a80ccfb0 con 0x7fb7a806afe0 2023-10-16T11:21:05.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.016+0000 7fb7a4ff9700 1 -- 172.21.15.158:0/2377878100 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb798007470 con 0x7fb7a806afe0 2023-10-16T11:21:05.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.017+0000 7fb7a4ff9700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb78806b140 0x7fb78806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.017+0000 7fb79ffff700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb78806b140 0x7fb78806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.017+0000 7fb7a4ff9700 1 -- 172.21.15.158:0/2377878100 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7fb798089d30 con 0x7fb7a806afe0 2023-10-16T11:21:05.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.017+0000 7fb79ffff700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb78806b140 0x7fb78806d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb7900090b0 tx=0x7fb790008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.021+0000 7fb7a4ff9700 1 -- 172.21.15.158:0/2377878100 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb7a80ccfb0 con 0x7fb7a806afe0 2023-10-16T11:21:05.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.215+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb7a80ccfb0 con 0x7fb7a806afe0 2023-10-16T11:21:05.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.215+0000 7fb7a4ff9700 1 -- 172.21.15.158:0/2377878100 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v246) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb7a80ccfb0 con 0x7fb7a806afe0 2023-10-16T11:21:05.217 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:21:05.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb78806b140 msgr2=0x7fb78806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb78806b140 0x7fb78806d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb7900090b0 tx=0x7fb790008030 comp rx=0 tx=0).stop 2023-10-16T11:21:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 msgr2=0x7fb7a81f3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a81f3880 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fb7980011e0 tx=0x7fb79800ca30 comp rx=0 tx=0).stop 2023-10-16T11:21:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 shutdown_connections 2023-10-16T11:21:05.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb78806b140 0x7fb78806d600 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7a80c9780 0x7fb7a80c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7a806afe0 0x7fb7a81f3880 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.217+0000 7fb7adaa7700 1 --2- 172.21.15.158:0/2377878100 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7a815dfc0 0x7fb7a80acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.218+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 >> 172.21.15.158:0/2377878100 conn(0x7fb7a80b1f00 msgr2=0x7fb7a80ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:05.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.218+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 shutdown_connections 2023-10-16T11:21:05.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.218+0000 7fb7adaa7700 1 -- 172.21.15.158:0/2377878100 wait complete. 2023-10-16T11:21:05.232 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T11:21:05.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.354+0000 7f9b32601700 1 Processor -- start 2023-10-16T11:21:05.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.354+0000 7f9b32601700 1 -- start start 2023-10-16T11:21:05.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b32601700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b2c15db50 0x7f9b2c0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b32601700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0caa80 con 0x7f9b2c0cf680 2023-10-16T11:21:05.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b32601700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0cad60 con 0x7f9b2c15d770 2023-10-16T11:21:05.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b32601700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2c0cb040 con 0x7f9b2c15db50 2023-10-16T11:21:05.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b2c15db50 0x7f9b2c0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b30b9e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9b2c0cf680 0x7f9b2c0cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41968/0 (socket says 172.21.15.158:41968) 2023-10-16T11:21:05.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b2c15db50 0x7f9b2c0cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41790/0 (socket says 172.21.15.158:41790) 2023-10-16T11:21:05.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b30b9e700 1 -- 172.21.15.158:0/806242228 learned_addr learned my addr 172.21.15.158:0/806242228 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:05.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2b7fe700 1 -- 172.21.15.158:0/806242228 >> v1:172.21.15.172:6789/0 conn(0x7f9b2c15d770 legacy=0x7f9b2c0cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:05.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2b7fe700 1 -- 172.21.15.158:0/806242228 >> v1:172.21.15.158:6789/0 conn(0x7f9b2c0cf680 legacy=0x7f9b2c0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2b7fe700 1 -- 172.21.15.158:0/806242228 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0cb320 con 0x7f9b2c15db50 2023-10-16T11:21:05.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2affd700 1 -- 172.21.15.158:0/806242228 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969877793 0 0) 0x7f9b2c0caa80 con 0x7f9b2c0cf680 2023-10-16T11:21:05.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2b7fe700 1 --2- 172.21.15.158:0/806242228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15db50 0x7f9b2c0cbe00 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009840 tx=0x7f9b1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6753d72b4558e5a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.355+0000 7f9b2affd700 1 -- 172.21.15.158:0/806242228 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c004230 con 0x7f9b2c15db50 2023-10-16T11:21:05.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b2affd700 1 -- 172.21.15.158:0/806242228 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c004390 con 0x7f9b2c15db50 2023-10-16T11:21:05.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b2affd700 1 -- 172.21.15.158:0/806242228 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c0036a0 con 0x7f9b2c15db50 2023-10-16T11:21:05.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b32601700 1 -- 172.21.15.158:0/806242228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15db50 msgr2=0x7f9b2c0cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b32601700 1 --2- 172.21.15.158:0/806242228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15db50 0x7f9b2c0cbe00 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009840 tx=0x7f9b1c009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b32601700 1 -- 172.21.15.158:0/806242228 shutdown_connections 2023-10-16T11:21:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b32601700 1 --2- 172.21.15.158:0/806242228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15db50 0x7f9b2c0cbe00 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b32601700 1 -- 172.21.15.158:0/806242228 >> 172.21.15.158:0/806242228 conn(0x7f9b2c099a10 msgr2=0x7f9b2c0d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b32601700 1 -- 172.21.15.158:0/806242228 shutdown_connections 2023-10-16T11:21:05.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.356+0000 7f9b32601700 1 -- 172.21.15.158:0/806242228 wait complete. 2023-10-16T11:21:05.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.357+0000 7f9b32601700 1 Processor -- start 2023-10-16T11:21:05.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b32601700 1 -- start start 2023-10-16T11:21:05.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b32601700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b2c0cf680 0x7f9b2c153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b32601700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15d770 0x7f9b2c0ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b32601700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b2c15db50 0x7f9b2c152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b32601700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c0ca7a0 con 0x7f9b2c15d770 2023-10-16T11:21:05.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b32601700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c0c8ed0 con 0x7f9b2c0cf680 2023-10-16T11:21:05.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b32601700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c0c8320 con 0x7f9b2c15db50 2023-10-16T11:21:05.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b2b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15d770 0x7f9b2c0ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b2b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15d770 0x7f9b2c0ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41798/0 (socket says 172.21.15.158:41798) 2023-10-16T11:21:05.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.358+0000 7f9b2b7fe700 1 -- 172.21.15.158:0/3232142661 learned_addr learned my addr 172.21.15.158:0/3232142661 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:05.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b30b9e700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b2c15db50 0x7f9b2c152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2bfff700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b2c0cf680 0x7f9b2c153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2b7fe700 1 -- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b2c15db50 msgr2=0x7f9b2c152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2b7fe700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b2c15db50 0x7f9b2c152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2b7fe700 1 -- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b2c0cf680 msgr2=0x7f9b2c153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2b7fe700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b2c0cf680 0x7f9b2c153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2b7fe700 1 -- 172.21.15.158:0/3232142661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c0094a0 con 0x7f9b2c15d770 2023-10-16T11:21:05.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b30b9e700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b2c15db50 0x7f9b2c152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:05.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2bfff700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b2c0cf680 0x7f9b2c153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:05.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b2b7fe700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15d770 0x7f9b2c0ca260 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0090b0 tx=0x7f9b1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b28ff9700 1 -- 172.21.15.158:0/3232142661 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c011630 con 0x7f9b2c15d770 2023-10-16T11:21:05.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b28ff9700 1 -- 172.21.15.158:0/3232142661 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c01f070 con 0x7f9b2c15d770 2023-10-16T11:21:05.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c0c85a0 con 0x7f9b2c15d770 2023-10-16T11:21:05.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b28ff9700 1 -- 172.21.15.158:0/3232142661 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c0119d0 con 0x7f9b2c15d770 2023-10-16T11:21:05.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.359+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c0c8b60 con 0x7f9b2c15d770 2023-10-16T11:21:05.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.360+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c0681b0 con 0x7f9b2c15d770 2023-10-16T11:21:05.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.361+0000 7f9b28ff9700 1 -- 172.21.15.158:0/3232142661 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9b1c011b30 con 0x7f9b2c15d770 2023-10-16T11:21:05.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.361+0000 7f9b28ff9700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9b0c06b140 0x7f9b0c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.361+0000 7f9b28ff9700 1 -- 172.21.15.158:0/3232142661 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f9b1c021070 con 0x7f9b2c15d770 2023-10-16T11:21:05.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.361+0000 7f9b2bfff700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9b0c06b140 0x7f9b0c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.362+0000 7f9b2bfff700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9b0c06b140 0x7f9b0c06d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9b14003fc0 tx=0x7f9b14009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.366+0000 7f9b28ff9700 1 -- 172.21.15.158:0/3232142661 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b1c056440 con 0x7f9b2c15d770 2023-10-16T11:21:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.557+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f9b2c0ce420 con 0x7f9b2c15d770 2023-10-16T11:21:05.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.557+0000 7f9b28ff9700 1 -- 172.21.15.158:0/3232142661 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v246) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9b1c059a60 con 0x7f9b2c15d770 2023-10-16T11:21:05.560 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:21:05.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9b0c06b140 msgr2=0x7f9b0c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9b0c06b140 0x7f9b0c06d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9b14003fc0 tx=0x7f9b14009040 comp rx=0 tx=0).stop 2023-10-16T11:21:05.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15d770 msgr2=0x7f9b2c0ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15d770 0x7f9b2c0ca260 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0090b0 tx=0x7f9b1c004750 comp rx=0 tx=0).stop 2023-10-16T11:21:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 shutdown_connections 2023-10-16T11:21:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9b0c06b140 0x7f9b0c06d600 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9b2c15db50 0x7f9b2c152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b2c0cf680 0x7f9b2c153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 --2- 172.21.15.158:0/3232142661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b2c15d770 0x7f9b2c0ca260 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 >> 172.21.15.158:0/3232142661 conn(0x7f9b2c099a10 msgr2=0x7f9b2c0d0290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:05.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 shutdown_connections 2023-10-16T11:21:05.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.560+0000 7f9b32601700 1 -- 172.21.15.158:0/3232142661 wait complete. 2023-10-16T11:21:05.574 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:05.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.695+0000 7f7332a5e700 1 Processor -- start 2023-10-16T11:21:05.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.695+0000 7f7332a5e700 1 -- start start 2023-10-16T11:21:05.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.695+0000 7f7332a5e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f732c15daa0 0x7f732c06b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.695+0000 7f7332a5e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c0ce4d0 con 0x7f732c0d3570 2023-10-16T11:21:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.695+0000 7f7332a5e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c06c1b0 con 0x7f732c0d0bd0 2023-10-16T11:21:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.695+0000 7f7332a5e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732c0d1c20 con 0x7f732c15daa0 2023-10-16T11:21:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f732c15daa0 0x7f732c06b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f7330ffb700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f732c0d3570 0x7f732c0d3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41980/0 (socket says 172.21.15.158:41980) 2023-10-16T11:21:05.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f7330ffb700 1 -- 172.21.15.158:0/1305624196 learned_addr learned my addr 172.21.15.158:0/1305624196 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732bfff700 1 -- 172.21.15.158:0/1305624196 >> v1:172.21.15.172:6789/0 conn(0x7f732c0d0bd0 legacy=0x7f732c0d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732bfff700 1 -- 172.21.15.158:0/1305624196 >> v1:172.21.15.158:6789/0 conn(0x7f732c0d3570 legacy=0x7f732c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732affd700 1 -- 172.21.15.158:0/1305624196 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845096823 0 0) 0x7f732c0ce4d0 con 0x7f732c0d3570 2023-10-16T11:21:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732bfff700 1 -- 172.21.15.158:0/1305624196 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c06c490 con 0x7f732c15daa0 2023-10-16T11:21:05.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732bfff700 1 --2- 172.21.15.158:0/1305624196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c15daa0 0x7f732c06b8d0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f7320009820 tx=0x7f7320009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210cb3fe8dc7e844 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732affd700 1 -- 172.21.15.158:0/1305624196 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320005e30 con 0x7f732c15daa0 2023-10-16T11:21:05.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732affd700 1 -- 172.21.15.158:0/1305624196 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7320004030 con 0x7f732c15daa0 2023-10-16T11:21:05.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.696+0000 7f732affd700 1 -- 172.21.15.158:0/1305624196 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73200043c0 con 0x7f732c15daa0 2023-10-16T11:21:05.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.697+0000 7f7332a5e700 1 -- 172.21.15.158:0/1305624196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c15daa0 msgr2=0x7f732c06b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.697+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1305624196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c15daa0 0x7f732c06b8d0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f7320009820 tx=0x7f7320009b30 comp rx=0 tx=0).stop 2023-10-16T11:21:05.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.697+0000 7f7332a5e700 1 -- 172.21.15.158:0/1305624196 shutdown_connections 2023-10-16T11:21:05.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.697+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1305624196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c15daa0 0x7f732c06b8d0 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.697+0000 7f7332a5e700 1 -- 172.21.15.158:0/1305624196 >> 172.21.15.158:0/1305624196 conn(0x7f732c08d040 msgr2=0x7f732c09c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:05.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.697+0000 7f7332a5e700 1 -- 172.21.15.158:0/1305624196 shutdown_connections 2023-10-16T11:21:05.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.697+0000 7f7332a5e700 1 -- 172.21.15.158:0/1305624196 wait complete. 2023-10-16T11:21:05.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.698+0000 7f7332a5e700 1 Processor -- start 2023-10-16T11:21:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.698+0000 7f7332a5e700 1 -- start start 2023-10-16T11:21:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f7332a5e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f732c0d0bd0 0x7f732c0b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f7332a5e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c0d3570 0x7f732c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f7332a5e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f732c15daa0 0x7f732c0b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f7332a5e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c15fb50 con 0x7f732c0d3570 2023-10-16T11:21:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f7332a5e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c15fd60 con 0x7f732c0d0bd0 2023-10-16T11:21:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f7332a5e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c160040 con 0x7f732c15daa0 2023-10-16T11:21:05.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c0d3570 0x7f732c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c0d3570 0x7f732c0b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41810/0 (socket says 172.21.15.158:41810) 2023-10-16T11:21:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 -- 172.21.15.158:0/1611856743 learned_addr learned my addr 172.21.15.158:0/1611856743 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f7330ffb700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f732c15daa0 0x7f732c0b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732bfff700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f732c0d0bd0 0x7f732c0b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 -- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f732c15daa0 msgr2=0x7f732c0b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f732c15daa0 0x7f732c0b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 -- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f732c0d0bd0 msgr2=0x7f732c0b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f732c0d0bd0 0x7f732c0b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 -- 172.21.15.158:0/1611856743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7320009430 con 0x7f732c0d3570 2023-10-16T11:21:05.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.699+0000 7f732b7fe700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c0d3570 0x7f732c0b1ea0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f731c00b4a0 tx=0x7f731c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.700+0000 7f7328ff9700 1 -- 172.21.15.158:0/1611856743 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c011830 con 0x7f732c0d3570 2023-10-16T11:21:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.700+0000 7f7328ff9700 1 -- 172.21.15.158:0/1611856743 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c011990 con 0x7f732c0d3570 2023-10-16T11:21:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.700+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c160320 con 0x7f732c0d3570 2023-10-16T11:21:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.700+0000 7f7328ff9700 1 -- 172.21.15.158:0/1611856743 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c011d40 con 0x7f732c0d3570 2023-10-16T11:21:05.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.700+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c0b8f80 con 0x7f732c0d3570 2023-10-16T11:21:05.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.700+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c0c7360 con 0x7f732c0d3570 2023-10-16T11:21:05.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.701+0000 7f7328ff9700 1 -- 172.21.15.158:0/1611856743 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f731c01a400 con 0x7f732c0d3570 2023-10-16T11:21:05.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.702+0000 7f7328ff9700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f731806b140 0x7f731806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:05.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.702+0000 7f732bfff700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f731806b140 0x7f731806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:05.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.702+0000 7f7328ff9700 1 -- 172.21.15.158:0/1611856743 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f731c089e00 con 0x7f732c0d3570 2023-10-16T11:21:05.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.702+0000 7f732bfff700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f731806b140 0x7f731806d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7320009b70 tx=0x7f7320008070 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:05.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.706+0000 7f7328ff9700 1 -- 172.21.15.158:0/1611856743 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f732c0c7360 con 0x7f732c0d3570 2023-10-16T11:21:05.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.946+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f732c0c7360 con 0x7f732c0d3570 2023-10-16T11:21:05.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.946+0000 7f7328ff9700 1 -- 172.21.15.158:0/1611856743 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f732c0c7360 con 0x7f732c0d3570 2023-10-16T11:21:05.949 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:05.949 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":39755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4215831623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4215831623},{"type":"v1","addr":"172.21.15.172:6834","nonce":4215831623}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:21:05.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.948+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f731806b140 msgr2=0x7f731806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.948+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f731806b140 0x7f731806d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7320009b70 tx=0x7f7320008070 comp rx=0 tx=0).stop 2023-10-16T11:21:05.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c0d3570 msgr2=0x7f732c0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:05.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c0d3570 0x7f732c0b1ea0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f731c00b4a0 tx=0x7f731c00b860 comp rx=0 tx=0).stop 2023-10-16T11:21:05.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 shutdown_connections 2023-10-16T11:21:05.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f731806b140 0x7f731806d600 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f732c15daa0 0x7f732c0b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f732c0d0bd0 0x7f732c0b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 --2- 172.21.15.158:0/1611856743 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f732c0d3570 0x7f732c0b1ea0 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:05.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 >> 172.21.15.158:0/1611856743 conn(0x7f732c08d040 msgr2=0x7f732c0ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:05.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 shutdown_connections 2023-10-16T11:21:05.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:05.949+0000 7f7332a5e700 1 -- 172.21.15.158:0/1611856743 wait complete. 2023-10-16T11:21:05.959 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 136 2023-10-16T11:21:05.966 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:06.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.085+0000 7f7a3fae2700 1 Processor -- start 2023-10-16T11:21:06.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.085+0000 7f7a3fae2700 1 -- start start 2023-10-16T11:21:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3fae2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 0x7f7a3815f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3fae2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7a3806d820 0x7f7a380ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3fae2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a38072700 con 0x7f7a3806e2b0 2023-10-16T11:21:06.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3fae2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a38071910 con 0x7f7a3806d820 2023-10-16T11:21:06.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3fae2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3806b480 con 0x7f7a3806b0a0 2023-10-16T11:21:06.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3e07f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7a3806e2b0 0x7f7a3815bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:41996/0 (socket says 172.21.15.158:41996) 2023-10-16T11:21:06.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3e07f700 1 -- 172.21.15.158:0/3674433183 learned_addr learned my addr 172.21.15.158:0/3674433183 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:06.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3d07d700 1 --2- 172.21.15.158:0/3674433183 >> v2:172.21.15.172:3300/0 conn(0x7f7a3806d820 0x7f7a380ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3d87e700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 0x7f7a3815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430784577 0 0) 0x7f7a38072700 con 0x7f7a3806e2b0 2023-10-16T11:21:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a240035b0 con 0x7f7a3806e2b0 2023-10-16T11:21:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246918253 0 0) 0x7f7a240035b0 con 0x7f7a3806e2b0 2023-10-16T11:21:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 msgr2=0x7f7a3815f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 0x7f7a3815f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 >> v2:172.21.15.172:3300/0 conn(0x7f7a3806d820 msgr2=0x7f7a380ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 --2- 172.21.15.158:0/3674433183 >> v2:172.21.15.172:3300/0 conn(0x7f7a3806d820 0x7f7a380ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380cca90 con 0x7f7a3806e2b0 2023-10-16T11:21:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f7a300019f0 con 0x7f7a3806e2b0 2023-10-16T11:21:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 >> v1:172.21.15.158:6789/0 conn(0x7f7a3806e2b0 legacy=0x7f7a3815bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a24003a30 0x7f7a24003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 0x7f7a24004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a24004d90 0x7f7a24005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38071910 con 0x7f7a24004d90 2023-10-16T11:21:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3806b480 con 0x7f7a24003a30 2023-10-16T11:21:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.086+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2400ba10 con 0x7f7a3806b0a0 2023-10-16T11:21:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3e07f700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a24004d90 0x7f7a24005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3d07d700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 0x7f7a24004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3d87e700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a24003a30 0x7f7a24003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3e07f700 1 -- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 msgr2=0x7f7a24004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3e07f700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3806b0a0 0x7f7a24004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3e07f700 1 -- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a24003a30 msgr2=0x7f7a24003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3e07f700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a24003a30 0x7f7a24003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3e07f700 1 -- 172.21.15.158:0/3674433183 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240035b0 con 0x7f7a24004d90 2023-10-16T11:21:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3e07f700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a24004d90 0x7f7a24005210 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000a860 tx=0x7f7a24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee59b0b8cb6673a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3000a2f0 con 0x7f7a24004d90 2023-10-16T11:21:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30012620 con 0x7f7a24004d90 2023-10-16T11:21:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.087+0000 7f7a3c87c700 1 -- 172.21.15.158:0/3674433183 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300129d0 con 0x7f7a24004d90 2023-10-16T11:21:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.088+0000 7f7a3fae2700 1 -- 172.21.15.158:0/3674433183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a24004d90 msgr2=0x7f7a24005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.088+0000 7f7a3fae2700 1 --2- 172.21.15.158:0/3674433183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a24004d90 0x7f7a24005210 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000a860 tx=0x7f7a24005880 comp rx=0 tx=0).stop 2023-10-16T11:21:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.088+0000 7f7a3e07f700 1 -- 172.21.15.158:0/3674433183 reap_dead start 2023-10-16T11:21:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.088+0000 7f7a3fae2700 1 -- 172.21.15.158:0/3674433183 shutdown_connections 2023-10-16T11:21:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.088+0000 7f7a3fae2700 1 -- 172.21.15.158:0/3674433183 >> 172.21.15.158:0/3674433183 conn(0x7f7a380d02b0 msgr2=0x7f7a380d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.088+0000 7f7a3fae2700 1 -- 172.21.15.158:0/3674433183 shutdown_connections 2023-10-16T11:21:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.088+0000 7f7a3fae2700 1 -- 172.21.15.158:0/3674433183 wait complete. 2023-10-16T11:21:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.089+0000 7f7a3fae2700 1 Processor -- start 2023-10-16T11:21:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.089+0000 7f7a3fae2700 1 -- start start 2023-10-16T11:21:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3fae2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a38152610 0x7f7a380c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3fae2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3815f2e0 0x7f7a38140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3fae2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a38140d40 0x7f7a3813e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3fae2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3815e5b0 con 0x7f7a38152610 2023-10-16T11:21:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3fae2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3815ee50 con 0x7f7a38140d40 2023-10-16T11:21:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3fae2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38141180 con 0x7f7a3815f2e0 2023-10-16T11:21:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a38152610 0x7f7a380c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a38152610 0x7f7a380c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41828/0 (socket says 172.21.15.158:41828) 2023-10-16T11:21:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 -- 172.21.15.158:0/219202291 learned_addr learned my addr 172.21.15.158:0/219202291 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d07d700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3815f2e0 0x7f7a38140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3e07f700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a38140d40 0x7f7a3813e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 -- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3815f2e0 msgr2=0x7f7a38140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3815f2e0 0x7f7a38140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 -- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a38140d40 msgr2=0x7f7a3813e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a38140d40 0x7f7a3813e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 -- 172.21.15.158:0/219202291 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380ca4a0 con 0x7f7a38152610 2023-10-16T11:21:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3d87e700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a38152610 0x7f7a380c3de0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f7a34011ad0 tx=0x7f7a34011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a2e7fc700 1 -- 172.21.15.158:0/219202291 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400f940 con 0x7f7a38152610 2023-10-16T11:21:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a2e7fc700 1 -- 172.21.15.158:0/219202291 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3400faa0 con 0x7f7a38152610 2023-10-16T11:21:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a381530d0 con 0x7f7a38152610 2023-10-16T11:21:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.090+0000 7f7a2e7fc700 1 -- 172.21.15.158:0/219202291 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400fe50 con 0x7f7a38152610 2023-10-16T11:21:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.091+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3815e800 con 0x7f7a38152610 2023-10-16T11:21:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.091+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3815f130 con 0x7f7a38152610 2023-10-16T11:21:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.092+0000 7f7a2e7fc700 1 -- 172.21.15.158:0/219202291 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7a34014440 con 0x7f7a38152610 2023-10-16T11:21:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.092+0000 7f7a2e7fc700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f7a1c06b140 0x7f7a1c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.092+0000 7f7a2e7fc700 1 -- 172.21.15.158:0/219202291 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f7a3408a7f0 con 0x7f7a38152610 2023-10-16T11:21:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.092+0000 7f7a3d07d700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f7a1c06b140 0x7f7a1c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.093+0000 7f7a3d07d700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f7a1c06b140 0x7f7a1c06d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7a380c76a0 tx=0x7f7a28009210 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.097+0000 7f7a2e7fc700 1 -- 172.21.15.158:0/219202291 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a34054e80 con 0x7f7a38152610 2023-10-16T11:21:06.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.301+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a380681b0 con 0x7f7a38152610 2023-10-16T11:21:06.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.302+0000 7f7a2e7fc700 1 -- 172.21.15.158:0/219202291 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+18871 (secure 0 0 0) 0x7f7a340584a0 con 0x7f7a38152610 2023-10-16T11:21:06.304 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:06.305 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":246,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:04.162755+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6834/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/937741801":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3443938198":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/872331855":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/1271192867":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/331460130":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/387969810":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.172:6832/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/750584278":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2035506903":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:06.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f7a1c06b140 msgr2=0x7f7a1c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f7a1c06b140 0x7f7a1c06d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7a380c76a0 tx=0x7f7a28009210 comp rx=0 tx=0).stop 2023-10-16T11:21:06.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a38152610 msgr2=0x7f7a380c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a38152610 0x7f7a380c3de0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f7a34011ad0 tx=0x7f7a34011de0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 shutdown_connections 2023-10-16T11:21:06.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f7a1c06b140 0x7f7a1c06d600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a38140d40 0x7f7a3813e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a38152610 0x7f7a380c3de0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 --2- 172.21.15.158:0/219202291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a3815f2e0 0x7f7a38140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 >> 172.21.15.158:0/219202291 conn(0x7f7a380d02b0 msgr2=0x7f7a380b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:06.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 shutdown_connections 2023-10-16T11:21:06.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.305+0000 7f7a3fae2700 1 -- 172.21.15.158:0/219202291 wait complete. 2023-10-16T11:21:06.319 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T11:21:06.319 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:06.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.438+0000 7f5b9a871700 1 Processor -- start 2023-10-16T11:21:06.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.438+0000 7f5b9a871700 1 -- start start 2023-10-16T11:21:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b9a871700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 0x7f5b9406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b9a871700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b94161270 con 0x7f5b9406ab40 2023-10-16T11:21:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b9a871700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b94161510 con 0x7f5b940729b0 2023-10-16T11:21:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b9a871700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b941617f0 con 0x7f5b9406e5d0 2023-10-16T11:21:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b93fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5b9406ab40 0x7f5b940722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42012/0 (socket says 172.21.15.158:42012) 2023-10-16T11:21:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b93fff700 1 -- 172.21.15.158:0/4161467542 learned_addr learned my addr 172.21.15.158:0/4161467542 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b937fe700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 0x7f5b9406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902392147 0 0) 0x7f5b94161270 con 0x7f5b9406ab40 2023-10-16T11:21:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b800035b0 con 0x7f5b9406ab40 2023-10-16T11:21:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459358717 0 0) 0x7f5b800035b0 con 0x7f5b9406ab40 2023-10-16T11:21:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 msgr2=0x7f5b9406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 0x7f5b9406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 >> v1:172.21.15.172:6789/0 conn(0x7f5b940729b0 legacy=0x7f5b9415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94161ad0 con 0x7f5b9406ab40 2023-10-16T11:21:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f5b7c001670 con 0x7f5b9406ab40 2023-10-16T11:21:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.439+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 >> v1:172.21.15.158:6789/0 conn(0x7f5b9406ab40 legacy=0x7f5b940722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 0x7f5b80003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b80003f70 0x7f5b80004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80004d90 0x7f5b80005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b941617f0 con 0x7f5b80003f70 2023-10-16T11:21:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8000ba70 con 0x7f5b80004d90 2023-10-16T11:21:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8000bc30 con 0x7f5b9406e5d0 2023-10-16T11:21:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b937fe700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b80003f70 0x7f5b80004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b93fff700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 0x7f5b80003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b98e0e700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80004d90 0x7f5b80005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b937fe700 1 -- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 msgr2=0x7f5b80003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b937fe700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b9406e5d0 0x7f5b80003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b937fe700 1 -- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80004d90 msgr2=0x7f5b80005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b937fe700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80004d90 0x7f5b80005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b937fe700 1 -- 172.21.15.158:0/4161467542 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b800035b0 con 0x7f5b80003f70 2023-10-16T11:21:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b937fe700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b80003f70 0x7f5b80004850 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f5b940d0fc0 tx=0x7f5b80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b35654a5044d502 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84009e70 con 0x7f5b80003f70 2023-10-16T11:21:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8400e6c0 con 0x7f5b80003f70 2023-10-16T11:21:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.440+0000 7f5b92ffd700 1 -- 172.21.15.158:0/4161467542 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400ea30 con 0x7f5b80003f70 2023-10-16T11:21:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.441+0000 7f5b9a871700 1 -- 172.21.15.158:0/4161467542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b80003f70 msgr2=0x7f5b80004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.441+0000 7f5b9a871700 1 --2- 172.21.15.158:0/4161467542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b80003f70 0x7f5b80004850 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f5b940d0fc0 tx=0x7f5b80005880 comp rx=0 tx=0).stop 2023-10-16T11:21:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.441+0000 7f5b98e0e700 1 -- 172.21.15.158:0/4161467542 reap_dead start 2023-10-16T11:21:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.441+0000 7f5b9a871700 1 -- 172.21.15.158:0/4161467542 shutdown_connections 2023-10-16T11:21:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.441+0000 7f5b9a871700 1 -- 172.21.15.158:0/4161467542 >> 172.21.15.158:0/4161467542 conn(0x7f5b940d02b0 msgr2=0x7f5b940d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.441+0000 7f5b9a871700 1 -- 172.21.15.158:0/4161467542 shutdown_connections 2023-10-16T11:21:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.441+0000 7f5b9a871700 1 -- 172.21.15.158:0/4161467542 wait complete. 2023-10-16T11:21:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.442+0000 7f5b9a871700 1 Processor -- start 2023-10-16T11:21:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.442+0000 7f5b9a871700 1 -- start start 2023-10-16T11:21:06.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b9a871700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80003f70 0x7f5b9414a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b9a871700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b940b42c0 0x7f5b940b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b9a871700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b940c8970 0x7f5b940c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b9a871700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940c9330 con 0x7f5b940c8970 2023-10-16T11:21:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b9a871700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940b0610 con 0x7f5b80003f70 2023-10-16T11:21:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b9a871700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940b4df0 con 0x7f5b940b42c0 2023-10-16T11:21:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b940c8970 0x7f5b940c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b940c8970 0x7f5b940c8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41852/0 (socket says 172.21.15.158:41852) 2023-10-16T11:21:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 -- 172.21.15.158:0/1214239715 learned_addr learned my addr 172.21.15.158:0/1214239715 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b937fe700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b940b42c0 0x7f5b940b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b93fff700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80003f70 0x7f5b9414a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 -- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b940b42c0 msgr2=0x7f5b940b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b940b42c0 0x7f5b940b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 -- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80003f70 msgr2=0x7f5b9414a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80003f70 0x7f5b9414a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b98e0e700 1 -- 172.21.15.158:0/1214239715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940b50d0 con 0x7f5b940c8970 2023-10-16T11:21:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.443+0000 7f5b93fff700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80003f70 0x7f5b9414a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.444+0000 7f5b98e0e700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b940c8970 0x7f5b940c8df0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c001520 tx=0x7f5b8c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.444+0000 7f5b90ff9700 1 -- 172.21.15.158:0/1214239715 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c003cd0 con 0x7f5b940c8970 2023-10-16T11:21:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.444+0000 7f5b90ff9700 1 -- 172.21.15.158:0/1214239715 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c003e30 con 0x7f5b940c8970 2023-10-16T11:21:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.444+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b940b53b0 con 0x7f5b940c8970 2023-10-16T11:21:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.444+0000 7f5b90ff9700 1 -- 172.21.15.158:0/1214239715 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c00dbd0 con 0x7f5b940c8970 2023-10-16T11:21:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.444+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b940b5970 con 0x7f5b940c8970 2023-10-16T11:21:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.445+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b940681b0 con 0x7f5b940c8970 2023-10-16T11:21:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.445+0000 7f5b90ff9700 1 -- 172.21.15.158:0/1214239715 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5b8c00dd30 con 0x7f5b940c8970 2023-10-16T11:21:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.445+0000 7f5b90ff9700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f5b7406b140 0x7f5b7406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.446+0000 7f5b90ff9700 1 -- 172.21.15.158:0/1214239715 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f5b8c08a510 con 0x7f5b940c8970 2023-10-16T11:21:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.446+0000 7f5b93fff700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f5b7406b140 0x7f5b7406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.446+0000 7f5b93fff700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f5b7406b140 0x7f5b7406d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c001eb0 tx=0x7f5b7c009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.450+0000 7f5b90ff9700 1 -- 172.21.15.158:0/1214239715 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b8c054c50 con 0x7f5b940c8970 2023-10-16T11:21:06.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.687+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b94157d30 con 0x7f5b940c8970 2023-10-16T11:21:06.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.687+0000 7f5b90ff9700 1 -- 172.21.15.158:0/1214239715 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f5b8c058270 con 0x7f5b940c8970 2023-10-16T11:21:06.690 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:06.690 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":39755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4215831623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4215831623},{"type":"v1","addr":"172.21.15.172:6834","nonce":4215831623}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:21:06.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f5b7406b140 msgr2=0x7f5b7406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f5b7406b140 0x7f5b7406d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c001eb0 tx=0x7f5b7c009040 comp rx=0 tx=0).stop 2023-10-16T11:21:06.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b940c8970 msgr2=0x7f5b940c8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b940c8970 0x7f5b940c8df0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c001520 tx=0x7f5b8c00d430 comp rx=0 tx=0).stop 2023-10-16T11:21:06.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 shutdown_connections 2023-10-16T11:21:06.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f5b7406b140 0x7f5b7406d600 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b940c8970 0x7f5b940c8df0 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b80003f70 0x7f5b9414a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 --2- 172.21.15.158:0/1214239715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b940b42c0 0x7f5b940b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 >> 172.21.15.158:0/1214239715 conn(0x7f5b940d02b0 msgr2=0x7f5b940ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:06.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 shutdown_connections 2023-10-16T11:21:06.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.690+0000 7f5b9a871700 1 -- 172.21.15.158:0/1214239715 wait complete. 2023-10-16T11:21:06.694 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 136 2023-10-16T11:21:06.704 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.823+0000 7f6f58014700 1 Processor -- start 2023-10-16T11:21:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.823+0000 7f6f58014700 1 -- start start 2023-10-16T11:21:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f58014700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f500d1210 0x7f6f500ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f58014700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6f500ce950 0x7f6f500b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f58014700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f500d3bd0 con 0x7f6f5015d7c0 2023-10-16T11:21:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f58014700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f500d4060 con 0x7f6f500ce950 2023-10-16T11:21:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f58014700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500cede0 con 0x7f6f500d1210 2023-10-16T11:21:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f565b1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6f500ce950 0x7f6f500b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f55db0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6f5015d7c0 0x7f6f500d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42032/0 (socket says 172.21.15.158:42032) 2023-10-16T11:21:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f55db0700 1 -- 172.21.15.158:0/34603582 learned_addr learned my addr 172.21.15.158:0/34603582 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f565b1700 1 -- 172.21.15.158:0/34603582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f500d1210 msgr2=0x7f6f500ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f565b1700 1 --2- 172.21.15.158:0/34603582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f500d1210 0x7f6f500ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f565b1700 1 -- 172.21.15.158:0/34603582 >> v1:172.21.15.158:6789/0 conn(0x7f6f5015d7c0 legacy=0x7f6f500d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f565b1700 1 -- 172.21.15.158:0/34603582 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f500d2fc0 con 0x7f6f500ce950 2023-10-16T11:21:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f54dae700 1 -- 172.21.15.158:0/34603582 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911477407 0 0) 0x7f6f500d3bd0 con 0x7f6f5015d7c0 2023-10-16T11:21:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.824+0000 7f6f565b1700 1 --2- 172.21.15.158:0/34603582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f500b6800 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009d90 tx=0x7f6f4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b4ff885116136b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f54dae700 1 -- 172.21.15.158:0/34603582 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800e070 con 0x7f6f500ce950 2023-10-16T11:21:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f54dae700 1 -- 172.21.15.158:0/34603582 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4800bbd0 con 0x7f6f500ce950 2023-10-16T11:21:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f54dae700 1 -- 172.21.15.158:0/34603582 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48012430 con 0x7f6f500ce950 2023-10-16T11:21:06.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 -- 172.21.15.158:0/34603582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 msgr2=0x7f6f500b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 --2- 172.21.15.158:0/34603582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f500b6800 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009d90 tx=0x7f6f4800b330 comp rx=0 tx=0).stop 2023-10-16T11:21:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 -- 172.21.15.158:0/34603582 shutdown_connections 2023-10-16T11:21:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 --2- 172.21.15.158:0/34603582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f500b6800 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 --2- 172.21.15.158:0/34603582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f500d1210 0x7f6f500ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 -- 172.21.15.158:0/34603582 >> 172.21.15.158:0/34603582 conn(0x7f6f500669f0 msgr2=0x7f6f50066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:06.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 -- 172.21.15.158:0/34603582 shutdown_connections 2023-10-16T11:21:06.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.825+0000 7f6f58014700 1 -- 172.21.15.158:0/34603582 wait complete. 2023-10-16T11:21:06.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.826+0000 7f6f58014700 1 Processor -- start 2023-10-16T11:21:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.826+0000 7f6f58014700 1 -- start start 2023-10-16T11:21:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f58014700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f50151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f58014700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f500d1210 0x7f6f5015b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f58014700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f5015d7c0 0x7f6f5015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f58014700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500c86c0 con 0x7f6f500ce950 2023-10-16T11:21:06.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f58014700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50152bc0 con 0x7f6f500d1210 2023-10-16T11:21:06.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f58014700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50152ea0 con 0x7f6f5015d7c0 2023-10-16T11:21:06.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f55db0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f50151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f55db0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f50151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41862/0 (socket says 172.21.15.158:41862) 2023-10-16T11:21:06.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f55db0700 1 -- 172.21.15.158:0/4231846169 learned_addr learned my addr 172.21.15.158:0/4231846169 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:06.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f55db0700 1 -- 172.21.15.158:0/4231846169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f5015d7c0 msgr2=0x7f6f5015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f55db0700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f5015d7c0 0x7f6f5015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f55db0700 1 -- 172.21.15.158:0/4231846169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f500d1210 msgr2=0x7f6f5015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:06.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.827+0000 7f6f55db0700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f500d1210 0x7f6f5015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:06.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.828+0000 7f6f55db0700 1 -- 172.21.15.158:0/4231846169 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48009a70 con 0x7f6f500ce950 2023-10-16T11:21:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.828+0000 7f6f55db0700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f50151550 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0011e0 tx=0x7f6f4c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.828+0000 7f6f467fc700 1 -- 172.21.15.158:0/4231846169 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c004c80 con 0x7f6f500ce950 2023-10-16T11:21:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.828+0000 7f6f467fc700 1 -- 172.21.15.158:0/4231846169 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4c004e00 con 0x7f6f500ce950 2023-10-16T11:21:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.828+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f500c8910 con 0x7f6f500ce950 2023-10-16T11:21:06.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.828+0000 7f6f467fc700 1 -- 172.21.15.158:0/4231846169 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c007310 con 0x7f6f500ce950 2023-10-16T11:21:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.828+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f50151a90 con 0x7f6f500ce950 2023-10-16T11:21:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.829+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f500d3560 con 0x7f6f500ce950 2023-10-16T11:21:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.829+0000 7f6f467fc700 1 -- 172.21.15.158:0/4231846169 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6f4c007470 con 0x7f6f500ce950 2023-10-16T11:21:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.830+0000 7f6f467fc700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f6f3406b140 0x7f6f3406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:06.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.830+0000 7f6f467fc700 1 -- 172.21.15.158:0/4231846169 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f6f4c089f40 con 0x7f6f500ce950 2023-10-16T11:21:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.830+0000 7f6f555af700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f6f3406b140 0x7f6f3406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.830+0000 7f6f555af700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f6f3406b140 0x7f6f3406d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f40006fd0 tx=0x7f6f40006d40 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:06.834+0000 7f6f467fc700 1 -- 172.21.15.158:0/4231846169 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f500d3560 con 0x7f6f500ce950 2023-10-16T11:21:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.070+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f500d3560 con 0x7f6f500ce950 2023-10-16T11:21:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.071+0000 7f6f467fc700 1 -- 172.21.15.158:0/4231846169 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f6f500d3560 con 0x7f6f500ce950 2023-10-16T11:21:07.073 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:07.074 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":39755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4215831623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4215831623},{"type":"v1","addr":"172.21.15.172:6834","nonce":4215831623}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:20:38.210020+0000","modified":"2023-10-16T11:20:41.237425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12432":{"gid":12432,"name":"b","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.172:6835/1061053278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1061053278},{"type":"v1","addr":"172.21.15.172:6835","nonce":1061053278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T11:21:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f6f3406b140 msgr2=0x7f6f3406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f6f3406b140 0x7f6f3406d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f40006fd0 tx=0x7f6f40006d40 comp rx=0 tx=0).stop 2023-10-16T11:21:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 msgr2=0x7f6f50151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f50151550 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0011e0 tx=0x7f6f4c00cae0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 shutdown_connections 2023-10-16T11:21:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f6f3406b140 0x7f6f3406d600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f5015d7c0 0x7f6f5015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f500ce950 0x7f6f50151550 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 --2- 172.21.15.158:0/4231846169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f500d1210 0x7f6f5015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 >> 172.21.15.158:0/4231846169 conn(0x7f6f500669f0 msgr2=0x7f6f500b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 shutdown_connections 2023-10-16T11:21:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.073+0000 7f6f58014700 1 -- 172.21.15.158:0/4231846169 wait complete. 2023-10-16T11:21:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 136 2023-10-16T11:21:07.087 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:07.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.208+0000 7f9c259bf700 1 Processor -- start 2023-10-16T11:21:07.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.208+0000 7f9c259bf700 1 -- start start 2023-10-16T11:21:07.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c259bf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c2015d7c0 0x7f9c200d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c259bf700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c200d1210 0x7f9c200ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c259bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c200b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c259bf700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c200d4060 con 0x7f9c200d1210 2023-10-16T11:21:07.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c259bf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c200d3bd0 con 0x7f9c2015d7c0 2023-10-16T11:21:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c259bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200cedc0 con 0x7f9c200ce950 2023-10-16T11:21:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c200d1210 0x7f9c200ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c200d1210 0x7f9c200ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41870/0 (socket says 172.21.15.158:41870) 2023-10-16T11:21:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/782514762 learned_addr learned my addr 172.21.15.158:0/782514762 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1effd700 1 --2- 172.21.15.158:0/782514762 >> v2:172.21.15.172:3300/0 conn(0x7f9c2015d7c0 0x7f9c200d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/782514762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c200b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/782514762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 msgr2=0x7f9c200b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 --2- 172.21.15.158:0/782514762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c200b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/782514762 >> v2:172.21.15.172:3300/0 conn(0x7f9c2015d7c0 msgr2=0x7f9c200d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 --2- 172.21.15.158:0/782514762 >> v2:172.21.15.172:3300/0 conn(0x7f9c2015d7c0 0x7f9c200d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/782514762 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c200d2cf0 con 0x7f9c200d1210 2023-10-16T11:21:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.209+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/782514762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c200b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c1e7fc700 1 --2- 172.21.15.158:0/782514762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200ce410 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f9c200d0bc0 tx=0x7f9c10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f62de668a95101 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c1dffb700 1 -- 172.21.15.158:0/782514762 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000b870 con 0x7f9c200d1210 2023-10-16T11:21:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c1dffb700 1 -- 172.21.15.158:0/782514762 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1000b9d0 con 0x7f9c200d1210 2023-10-16T11:21:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c1dffb700 1 -- 172.21.15.158:0/782514762 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000bd60 con 0x7f9c200d1210 2023-10-16T11:21:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 -- 172.21.15.158:0/782514762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 msgr2=0x7f9c200ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 --2- 172.21.15.158:0/782514762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200ce410 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f9c200d0bc0 tx=0x7f9c10009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 -- 172.21.15.158:0/782514762 shutdown_connections 2023-10-16T11:21:07.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 --2- 172.21.15.158:0/782514762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c200b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 --2- 172.21.15.158:0/782514762 >> v2:172.21.15.172:3300/0 conn(0x7f9c2015d7c0 0x7f9c200d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 --2- 172.21.15.158:0/782514762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200ce410 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 -- 172.21.15.158:0/782514762 >> 172.21.15.158:0/782514762 conn(0x7f9c200669f0 msgr2=0x7f9c20066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:07.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 -- 172.21.15.158:0/782514762 shutdown_connections 2023-10-16T11:21:07.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.210+0000 7f9c259bf700 1 -- 172.21.15.158:0/782514762 wait complete. 2023-10-16T11:21:07.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.212+0000 7f9c259bf700 1 Processor -- start 2023-10-16T11:21:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.212+0000 7f9c259bf700 1 -- start start 2023-10-16T11:21:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c259bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c2015f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c259bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c259bf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2015d7c0 0x7f9c20151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c259bf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200c7c20 con 0x7f9c200d1210 2023-10-16T11:21:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c259bf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c20152430 con 0x7f9c2015d7c0 2023-10-16T11:21:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c259bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200c7e50 con 0x7f9c200ce950 2023-10-16T11:21:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200c76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41876/0 (socket says 172.21.15.158:41876) 2023-10-16T11:21:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/2855611688 learned_addr learned my addr 172.21.15.158:0/2855611688 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 msgr2=0x7f9c2015f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1effd700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c2015f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1e7fc700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c2015f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2015d7c0 msgr2=0x7f9c20151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2015d7c0 0x7f9c20151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.213+0000 7f9c1e7fc700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2015d7c0 0x7f9c20151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c1e7fc700 1 -- 172.21.15.158:0/2855611688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10009550 con 0x7f9c200d1210 2023-10-16T11:21:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c1f7fe700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2015d7c0 0x7f9c20151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c1e7fc700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200c76e0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f9c10000c00 tx=0x7f9c10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c07fff700 1 -- 172.21.15.158:0/2855611688 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10004540 con 0x7f9c200d1210 2023-10-16T11:21:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c07fff700 1 -- 172.21.15.158:0/2855611688 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c100036a0 con 0x7f9c200d1210 2023-10-16T11:21:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c200c8130 con 0x7f9c200d1210 2023-10-16T11:21:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c07fff700 1 -- 172.21.15.158:0/2855611688 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10003a50 con 0x7f9c200d1210 2023-10-16T11:21:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.214+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c200c86f0 con 0x7f9c200d1210 2023-10-16T11:21:07.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.215+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2014f3c0 con 0x7f9c200d1210 2023-10-16T11:21:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.215+0000 7f9c07fff700 1 -- 172.21.15.158:0/2855611688 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9c10021070 con 0x7f9c200d1210 2023-10-16T11:21:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.215+0000 7f9c07fff700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9c0c06b140 0x7f9c0c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.216+0000 7f9c07fff700 1 -- 172.21.15.158:0/2855611688 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f9c1008b780 con 0x7f9c200d1210 2023-10-16T11:21:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.216+0000 7f9c1effd700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9c0c06b140 0x7f9c0c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.216+0000 7f9c1effd700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9c0c06b140 0x7f9c0c06d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c2015c2e0 tx=0x7f9c14008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.220+0000 7f9c07fff700 1 -- 172.21.15.158:0/2855611688 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c10055e80 con 0x7f9c200d1210 2023-10-16T11:21:07.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.415+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c20160fb0 con 0x7f9c200d1210 2023-10-16T11:21:07.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.416+0000 7f9c07fff700 1 -- 172.21.15.158:0/2855611688 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+18871 (secure 0 0 0) 0x7f9c10019e60 con 0x7f9c200d1210 2023-10-16T11:21:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:07.419 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":246,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:04.162755+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T11:20:35.862082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"240","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T11:20:37.182868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6834/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/937741801":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3443938198":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/872331855":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/1271192867":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/331460130":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/387969810":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.172:6832/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/750584278":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2035506903":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:07.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9c0c06b140 msgr2=0x7f9c0c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9c0c06b140 0x7f9c0c06d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c2015c2e0 tx=0x7f9c14008040 comp rx=0 tx=0).stop 2023-10-16T11:21:07.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 msgr2=0x7f9c200c76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200c76e0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f9c10000c00 tx=0x7f9c10004750 comp rx=0 tx=0).stop 2023-10-16T11:21:07.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 shutdown_connections 2023-10-16T11:21:07.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9c0c06b140 0x7f9c0c06d600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c2015d7c0 0x7f9c20151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c200ce950 0x7f9c2015f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 --2- 172.21.15.158:0/2855611688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c200d1210 0x7f9c200c76e0 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 >> 172.21.15.158:0/2855611688 conn(0x7f9c200669f0 msgr2=0x7f9c200b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:07.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 shutdown_connections 2023-10-16T11:21:07.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.419+0000 7f9c259bf700 1 -- 172.21.15.158:0/2855611688 wait complete. 2023-10-16T11:21:07.433 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T11:21:07.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.553+0000 7f57621db700 1 Processor -- start 2023-10-16T11:21:07.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.554+0000 7f57621db700 1 -- start start 2023-10-16T11:21:07.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.554+0000 7f57621db700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f575c0d0ae0 0x7f575c0b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.554+0000 7f57621db700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c0b4950 0x7f575c0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.554+0000 7f57621db700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c15f990 con 0x7f575c15eb10 2023-10-16T11:21:07.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.554+0000 7f57621db700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0cdef0 con 0x7f575c0d0ae0 2023-10-16T11:21:07.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.554+0000 7f57621db700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0cd070 con 0x7f575c0b4950 2023-10-16T11:21:07.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575affd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f575c0d0ae0 0x7f575c0b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f575c15eb10 0x7f575c15f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42058/0 (socket says 172.21.15.158:42058) 2023-10-16T11:21:07.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575b7fe700 1 -- 172.21.15.158:0/450494005 learned_addr learned my addr 172.21.15.158:0/450494005 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:07.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575bfff700 1 --2- 172.21.15.158:0/450494005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c0b4950 0x7f575c0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575affd700 1 -- 172.21.15.158:0/450494005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c0b4950 msgr2=0x7f575c0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:07.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575affd700 1 --2- 172.21.15.158:0/450494005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c0b4950 0x7f575c0cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575affd700 1 -- 172.21.15.158:0/450494005 >> v1:172.21.15.158:6789/0 conn(0x7f575c15eb10 legacy=0x7f575c15f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575affd700 1 -- 172.21.15.158:0/450494005 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c15fc70 con 0x7f575c0d0ae0 2023-10-16T11:21:07.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575bfff700 1 --2- 172.21.15.158:0/450494005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c0b4950 0x7f575c0cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:07.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575affd700 1 --2- 172.21.15.158:0/450494005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0d0ae0 0x7f575c0b4410 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f5750009840 tx=0x7f5750009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46502edfdb2b39d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575a7fc700 1 -- 172.21.15.158:0/450494005 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750011830 con 0x7f575c0d0ae0 2023-10-16T11:21:07.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575a7fc700 1 -- 172.21.15.158:0/450494005 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5750011990 con 0x7f575c0d0ae0 2023-10-16T11:21:07.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.555+0000 7f575a7fc700 1 -- 172.21.15.158:0/450494005 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750011d20 con 0x7f575c0d0ae0 2023-10-16T11:21:07.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 -- 172.21.15.158:0/450494005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0d0ae0 msgr2=0x7f575c0b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 --2- 172.21.15.158:0/450494005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0d0ae0 0x7f575c0b4410 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f5750009840 tx=0x7f5750009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:07.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 -- 172.21.15.158:0/450494005 shutdown_connections 2023-10-16T11:21:07.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 --2- 172.21.15.158:0/450494005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c0b4950 0x7f575c0cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.562 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 --2- 172.21.15.158:0/450494005 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0d0ae0 0x7f575c0b4410 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 -- 172.21.15.158:0/450494005 >> 172.21.15.158:0/450494005 conn(0x7f575c0ce040 msgr2=0x7f575c0ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 -- 172.21.15.158:0/450494005 shutdown_connections 2023-10-16T11:21:07.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.556+0000 7f57621db700 1 -- 172.21.15.158:0/450494005 wait complete. 2023-10-16T11:21:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.557+0000 7f57621db700 1 Processor -- start 2023-10-16T11:21:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.557+0000 7f57621db700 1 -- start start 2023-10-16T11:21:07.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f57621db700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0b4950 0x7f575c1f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f57621db700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f575c0d0ae0 0x7f575c1f1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f57621db700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c15eb10 0x7f575c1f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f57621db700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0cad30 con 0x7f575c0b4950 2023-10-16T11:21:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f57621db700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1f1c30 con 0x7f575c0d0ae0 2023-10-16T11:21:07.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f57621db700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1f1f10 con 0x7f575c15eb10 2023-10-16T11:21:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0b4950 0x7f575c1f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0b4950 0x7f575c1f3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41884/0 (socket says 172.21.15.158:41884) 2023-10-16T11:21:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 -- 172.21.15.158:0/2612373926 learned_addr learned my addr 172.21.15.158:0/2612373926 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575bfff700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c15eb10 0x7f575c1f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575affd700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f575c0d0ae0 0x7f575c1f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 -- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c15eb10 msgr2=0x7f575c1f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c15eb10 0x7f575c1f1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 -- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f575c0d0ae0 msgr2=0x7f575c1f1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f575c0d0ae0 0x7f575c1f1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 -- 172.21.15.158:0/2612373926 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57500094a0 con 0x7f575c0b4950 2023-10-16T11:21:07.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f575b7fe700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0b4950 0x7f575c1f3ee0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c480 tx=0x7f574c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f5743fff700 1 -- 172.21.15.158:0/2612373926 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c005d70 con 0x7f575c0b4950 2023-10-16T11:21:07.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f5743fff700 1 -- 172.21.15.158:0/2612373926 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c0070d0 con 0x7f575c0b4950 2023-10-16T11:21:07.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0caf40 con 0x7f575c0b4950 2023-10-16T11:21:07.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.558+0000 7f5743fff700 1 -- 172.21.15.158:0/2612373926 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c007440 con 0x7f575c0b4950 2023-10-16T11:21:07.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.559+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c1f4420 con 0x7f575c0b4950 2023-10-16T11:21:07.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.560+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c0681b0 con 0x7f575c0b4950 2023-10-16T11:21:07.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.560+0000 7f5743fff700 1 -- 172.21.15.158:0/2612373926 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f574c0075a0 con 0x7f575c0b4950 2023-10-16T11:21:07.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.560+0000 7f5743fff700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f574806b140 0x7f574806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.560+0000 7f5743fff700 1 -- 172.21.15.158:0/2612373926 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7467+0+0 (secure 0 0 0) 0x7f574c0894a0 con 0x7f575c0b4950 2023-10-16T11:21:07.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.560+0000 7f575affd700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f574806b140 0x7f574806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.561+0000 7f575affd700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f574806b140 0x7f574806d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5750009840 tx=0x7f5750008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.565+0000 7f5743fff700 1 -- 172.21.15.158:0/2612373926 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f574c053c90 con 0x7f575c0b4950 2023-10-16T11:21:07.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.802+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f575c14d180 con 0x7f575c0b4950 2023-10-16T11:21:07.804 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:21:07.802+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:21:07.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.803+0000 7f5743fff700 1 -- 172.21.15.158:0/2612373926 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v137) v1 ==== 167+0+0 (secure 0 0 0) 0x7f574c0572b0 con 0x7f575c0b4950 2023-10-16T11:21:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f574806b140 msgr2=0x7f574806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f574806b140 0x7f574806d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5750009840 tx=0x7f5750008030 comp rx=0 tx=0).stop 2023-10-16T11:21:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0b4950 msgr2=0x7f575c1f3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0b4950 0x7f575c1f3ee0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c480 tx=0x7f574c00c840 comp rx=0 tx=0).stop 2023-10-16T11:21:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 shutdown_connections 2023-10-16T11:21:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f574806b140 0x7f574806d600 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f575c15eb10 0x7f575c1f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c0b4950 0x7f575c1f3ee0 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 --2- 172.21.15.158:0/2612373926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f575c0d0ae0 0x7f575c1f1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 >> 172.21.15.158:0/2612373926 conn(0x7f575c0ce040 msgr2=0x7f575c0c8600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 shutdown_connections 2023-10-16T11:21:07.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.807+0000 7f57621db700 1 -- 172.21.15.158:0/2612373926 wait complete. 2023-10-16T11:21:07.812 INFO:tasks.ceph.mds.b.smithi172.stderr: -1571> 2023-10-16T11:20:42.710+0000 7f5424b2b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:21:07.812 INFO:teuthology.orchestra.run.smithi158.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:21:07.821 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T11:21:07.837 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:21:07.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.943+0000 7f9a6aea1700 1 Processor -- start 2023-10-16T11:21:07.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.943+0000 7f9a6aea1700 1 -- start start 2023-10-16T11:21:07.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a6aea1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a64161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a6aea1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a640cf070 0x7f9a640cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a6aea1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a640d1430 0x7f9a640d1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a6aea1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a640cfa10 con 0x7f9a640cf070 2023-10-16T11:21:07.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a6aea1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6415dc20 con 0x7f9a640d1430 2023-10-16T11:21:07.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a6aea1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6406c0e0 con 0x7f9a6415d840 2023-10-16T11:21:07.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a640cf070 0x7f9a640cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a640cf070 0x7f9a640cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41894/0 (socket says 172.21.15.158:41894) 2023-10-16T11:21:07.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 -- 172.21.15.158:0/1160451261 learned_addr learned my addr 172.21.15.158:0/1160451261 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:07.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a6943e700 1 --2- 172.21.15.158:0/1160451261 >> v2:172.21.15.172:3300/0 conn(0x7f9a640d1430 0x7f9a640d1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a68c3d700 1 --2- 172.21.15.158:0/1160451261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a64161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 -- 172.21.15.158:0/1160451261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 msgr2=0x7f9a64161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:07.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 --2- 172.21.15.158:0/1160451261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a64161d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 -- 172.21.15.158:0/1160451261 >> v2:172.21.15.172:3300/0 conn(0x7f9a640d1430 msgr2=0x7f9a640d1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:07.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 --2- 172.21.15.158:0/1160451261 >> v2:172.21.15.172:3300/0 conn(0x7f9a640d1430 0x7f9a640d1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 -- 172.21.15.158:0/1160451261 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6406c3c0 con 0x7f9a640cf070 2023-10-16T11:21:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a63fff700 1 --2- 172.21.15.158:0/1160451261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640cf070 0x7f9a640cf4d0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cf000 tx=0x7f9a50009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57cfb044293d61f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a637fe700 1 -- 172.21.15.158:0/1160451261 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000b870 con 0x7f9a640cf070 2023-10-16T11:21:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.944+0000 7f9a637fe700 1 -- 172.21.15.158:0/1160451261 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5000b9d0 con 0x7f9a640cf070 2023-10-16T11:21:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a637fe700 1 -- 172.21.15.158:0/1160451261 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000bd60 con 0x7f9a640cf070 2023-10-16T11:21:07.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 -- 172.21.15.158:0/1160451261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640cf070 msgr2=0x7f9a640cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/1160451261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640cf070 0x7f9a640cf4d0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cf000 tx=0x7f9a50009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 -- 172.21.15.158:0/1160451261 shutdown_connections 2023-10-16T11:21:07.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/1160451261 >> v2:172.21.15.172:3300/0 conn(0x7f9a640d1430 0x7f9a640d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/1160451261 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a64161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/1160451261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640cf070 0x7f9a640cf4d0 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 -- 172.21.15.158:0/1160451261 >> 172.21.15.158:0/1160451261 conn(0x7f9a6406a8d0 msgr2=0x7f9a6406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:07.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 -- 172.21.15.158:0/1160451261 shutdown_connections 2023-10-16T11:21:07.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.945+0000 7f9a6aea1700 1 -- 172.21.15.158:0/1160451261 wait complete. 2023-10-16T11:21:07.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.946+0000 7f9a6aea1700 1 Processor -- start 2023-10-16T11:21:07.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.946+0000 7f9a6aea1700 1 -- start start 2023-10-16T11:21:07.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a6aea1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640cf070 0x7f9a640b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a6aea1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640d1430 0x7f9a640b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a6aea1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a640b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a6aea1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640b4b30 con 0x7f9a640d1430 2023-10-16T11:21:07.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a6aea1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640b4e10 con 0x7f9a640cf070 2023-10-16T11:21:07.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a6aea1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640b50f0 con 0x7f9a6415d840 2023-10-16T11:21:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640d1430 0x7f9a640b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640d1430 0x7f9a640b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41900/0 (socket says 172.21.15.158:41900) 2023-10-16T11:21:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a63fff700 1 -- 172.21.15.158:0/596645541 learned_addr learned my addr 172.21.15.158:0/596645541 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:07.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a6943e700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a640b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.947+0000 7f9a68c3d700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640cf070 0x7f9a640b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a63fff700 1 -- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 msgr2=0x7f9a640b6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a63fff700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a640b6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a63fff700 1 -- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640cf070 msgr2=0x7f9a640b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:07.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a63fff700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640cf070 0x7f9a640b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:07.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a63fff700 1 -- 172.21.15.158:0/596645541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a50009550 con 0x7f9a640d1430 2023-10-16T11:21:07.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a68c3d700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640cf070 0x7f9a640b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:07.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a63fff700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640d1430 0x7f9a640b5bd0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f9a50000c00 tx=0x7f9a50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a617fa700 1 -- 172.21.15.158:0/596645541 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50004540 con 0x7f9a640d1430 2023-10-16T11:21:07.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a617fa700 1 -- 172.21.15.158:0/596645541 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a500036a0 con 0x7f9a640d1430 2023-10-16T11:21:07.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a640b53d0 con 0x7f9a640d1430 2023-10-16T11:21:07.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a617fa700 1 -- 172.21.15.158:0/596645541 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50003a50 con 0x7f9a640d1430 2023-10-16T11:21:07.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.948+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a640c97b0 con 0x7f9a640d1430 2023-10-16T11:21:07.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.949+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a640b58f0 con 0x7f9a640d1430 2023-10-16T11:21:07.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.949+0000 7f9a617fa700 1 -- 172.21.15.158:0/596645541 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9a50021070 con 0x7f9a640d1430 2023-10-16T11:21:07.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.950+0000 7f9a617fa700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9a4806b140 0x7f9a4806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:07.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.950+0000 7f9a617fa700 1 -- 172.21.15.158:0/596645541 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7553+0+0 (secure 0 0 0) 0x7f9a5008b890 con 0x7f9a640d1430 2023-10-16T11:21:07.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.950+0000 7f9a68c3d700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9a4806b140 0x7f9a4806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:07.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.950+0000 7f9a68c3d700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9a4806b140 0x7f9a4806d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9a58007a50 tx=0x7f9a58008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:07.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:07.954+0000 7f9a617fa700 1 -- 172.21.15.158:0/596645541 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a640b58f0 con 0x7f9a640d1430 2023-10-16T11:21:08.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.194+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9a640b58f0 con 0x7f9a640d1430 2023-10-16T11:21:08.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.805+0000 7f9a617fa700 1 -- 172.21.15.158:0/596645541 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v138) v1 ==== 107+0+0 (secure 0 0 0) 0x7f9a640b58f0 con 0x7f9a640d1430 2023-10-16T11:21:08.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.807+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9a4806b140 msgr2=0x7f9a4806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:08.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.807+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9a4806b140 0x7f9a4806d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9a58007a50 tx=0x7f9a58008040 comp rx=0 tx=0).stop 2023-10-16T11:21:08.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.807+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640d1430 msgr2=0x7f9a640b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:08.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.807+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640d1430 0x7f9a640b5bd0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f9a50000c00 tx=0x7f9a50004750 comp rx=0 tx=0).stop 2023-10-16T11:21:08.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 shutdown_connections 2023-10-16T11:21:08.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9a4806b140 0x7f9a4806d600 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:08.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a6415d840 0x7f9a640b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:08.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640cf070 0x7f9a640b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:08.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 --2- 172.21.15.158:0/596645541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a640d1430 0x7f9a640b5bd0 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:08.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 >> 172.21.15.158:0/596645541 conn(0x7f9a6406a8d0 msgr2=0x7f9a640ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:08.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 shutdown_connections 2023-10-16T11:21:08.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.808+0000 7f9a6aea1700 1 -- 172.21.15.158:0/596645541 wait complete. 2023-10-16T11:21:08.824 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T11:21:08.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.951+0000 7ff048131700 1 Processor -- start 2023-10-16T11:21:08.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.952+0000 7ff048131700 1 -- start start 2023-10-16T11:21:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.952+0000 7ff048131700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff04015da40 0x7ff0400caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.952+0000 7ff048131700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400cbd30 con 0x7ff0400cc590 2023-10-16T11:21:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.952+0000 7ff048131700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400cc1c0 con 0x7ff04006afe0 2023-10-16T11:21:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.952+0000 7ff048131700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04015de20 con 0x7ff04015da40 2023-10-16T11:21:08.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff045ecd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff04015da40 0x7ff0400caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff0456cc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff0400cc590 0x7ff04006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42082/0 (socket says 172.21.15.158:42082) 2023-10-16T11:21:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff0456cc700 1 -- 172.21.15.158:0/1278999485 learned_addr learned my addr 172.21.15.158:0/1278999485 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff045ecd700 1 -- 172.21.15.158:0/1278999485 >> v1:172.21.15.172:6789/0 conn(0x7ff04006afe0 legacy=0x7ff0400afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff045ecd700 1 -- 172.21.15.158:0/1278999485 >> v1:172.21.15.158:6789/0 conn(0x7ff0400cc590 legacy=0x7ff04006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:08.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff044ecb700 1 -- 172.21.15.158:0/1278999485 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946967554 0 0) 0x7ff0400cbd30 con 0x7ff0400cc590 2023-10-16T11:21:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff045ecd700 1 -- 172.21.15.158:0/1278999485 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400ce260 con 0x7ff04015da40 2023-10-16T11:21:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff045ecd700 1 --2- 172.21.15.158:0/1278999485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04015da40 0x7ff0400caa20 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7ff030009840 tx=0x7ff030009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913dc464975c932 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff044ecb700 1 -- 172.21.15.158:0/1278999485 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030004230 con 0x7ff04015da40 2023-10-16T11:21:08.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff044ecb700 1 -- 172.21.15.158:0/1278999485 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff030004390 con 0x7ff04015da40 2023-10-16T11:21:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.953+0000 7ff044ecb700 1 -- 172.21.15.158:0/1278999485 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0300036a0 con 0x7ff04015da40 2023-10-16T11:21:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.954+0000 7ff048131700 1 -- 172.21.15.158:0/1278999485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04015da40 msgr2=0x7ff0400caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.954+0000 7ff048131700 1 --2- 172.21.15.158:0/1278999485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04015da40 0x7ff0400caa20 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7ff030009840 tx=0x7ff030009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.954+0000 7ff048131700 1 -- 172.21.15.158:0/1278999485 shutdown_connections 2023-10-16T11:21:08.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.954+0000 7ff048131700 1 --2- 172.21.15.158:0/1278999485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04015da40 0x7ff0400caa20 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:08.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.954+0000 7ff048131700 1 -- 172.21.15.158:0/1278999485 >> 172.21.15.158:0/1278999485 conn(0x7ff040099a10 msgr2=0x7ff0400d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:08.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.954+0000 7ff048131700 1 -- 172.21.15.158:0/1278999485 shutdown_connections 2023-10-16T11:21:08.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.954+0000 7ff048131700 1 -- 172.21.15.158:0/1278999485 wait complete. 2023-10-16T11:21:08.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.955+0000 7ff048131700 1 Processor -- start 2023-10-16T11:21:08.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.955+0000 7ff048131700 1 -- start start 2023-10-16T11:21:08.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff048131700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04006afe0 0x7ff0400ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:08.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff048131700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0400cc590 0x7ff0401e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:08.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff048131700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04015da40 0x7ff0401e6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:08.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff048131700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400cabb0 con 0x7ff0400cc590 2023-10-16T11:21:08.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff048131700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0401e5250 con 0x7ff04006afe0 2023-10-16T11:21:08.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff048131700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0401e5530 con 0x7ff04015da40 2023-10-16T11:21:08.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0400cc590 0x7ff0401e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:08.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0400cc590 0x7ff0401e6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41912/0 (socket says 172.21.15.158:41912) 2023-10-16T11:21:08.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 -- 172.21.15.158:0/1468643895 learned_addr learned my addr 172.21.15.158:0/1468643895 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:08.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 -- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04015da40 msgr2=0x7ff0401e6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:08.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0466ce700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04015da40 0x7ff0401e6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:08.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff045ecd700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04006afe0 0x7ff0400ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:08.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04015da40 0x7ff0401e6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:08.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 -- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04006afe0 msgr2=0x7ff0400ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:08.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04006afe0 0x7ff0400ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:08.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 -- 172.21.15.158:0/1468643895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300094a0 con 0x7ff0400cc590 2023-10-16T11:21:08.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0466ce700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04015da40 0x7ff0401e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:08.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff045ecd700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04006afe0 0x7ff0400ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:08.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0456cc700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0400cc590 0x7ff0401e6890 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00c540 tx=0x7ff03c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:08.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0367fc700 1 -- 172.21.15.158:0/1468643895 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c004c80 con 0x7ff0400cc590 2023-10-16T11:21:08.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0367fc700 1 -- 172.21.15.158:0/1468643895 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c004de0 con 0x7ff0400cc590 2023-10-16T11:21:08.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0401e5810 con 0x7ff0400cc590 2023-10-16T11:21:08.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.956+0000 7ff0367fc700 1 -- 172.21.15.158:0/1468643895 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c007310 con 0x7ff0400cc590 2023-10-16T11:21:08.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.957+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0401e5dd0 con 0x7ff0400cc590 2023-10-16T11:21:08.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.957+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0401e5ff0 con 0x7ff0400cc590 2023-10-16T11:21:08.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.958+0000 7ff0367fc700 1 -- 172.21.15.158:0/1468643895 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff03c007470 con 0x7ff0400cc590 2023-10-16T11:21:08.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.958+0000 7ff0367fc700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff02406b140 0x7ff02406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:08.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.958+0000 7ff0367fc700 1 -- 172.21.15.158:0/1468643895 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7553+0+0 (secure 0 0 0) 0x7ff03c089e50 con 0x7ff0400cc590 2023-10-16T11:21:08.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.958+0000 7ff045ecd700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff02406b140 0x7ff02406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:08.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.959+0000 7ff045ecd700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff02406b140 0x7ff02406d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0300090b0 tx=0x7ff030013040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:08.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:08.963+0000 7ff0367fc700 1 -- 172.21.15.158:0/1468643895 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff03c0544f0 con 0x7ff0400cc590 2023-10-16T11:21:09.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.155+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff0400681b0 con 0x7ff0400cc590 2023-10-16T11:21:09.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.806+0000 7ff0367fc700 1 -- 172.21.15.158:0/1468643895 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v248) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff03c057b10 con 0x7ff0400cc590 2023-10-16T11:21:09.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff02406b140 msgr2=0x7ff02406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:09.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff02406b140 0x7ff02406d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0300090b0 tx=0x7ff030013040 comp rx=0 tx=0).stop 2023-10-16T11:21:09.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0400cc590 msgr2=0x7ff0401e6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:09.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0400cc590 0x7ff0401e6890 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00c540 tx=0x7ff03c00c900 comp rx=0 tx=0).stop 2023-10-16T11:21:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 shutdown_connections 2023-10-16T11:21:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff02406b140 0x7ff02406d600 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04015da40 0x7ff0401e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04006afe0 0x7ff0400ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 --2- 172.21.15.158:0/1468643895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0400cc590 0x7ff0401e6890 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 >> 172.21.15.158:0/1468643895 conn(0x7ff040099a10 msgr2=0x7ff0400ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 shutdown_connections 2023-10-16T11:21:09.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.811+0000 7ff048131700 1 -- 172.21.15.158:0/1468643895 wait complete. 2023-10-16T11:21:09.817 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_metadata' removed 2023-10-16T11:21:09.829 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T11:21:09.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.959+0000 7fa61d7b1700 1 Processor -- start 2023-10-16T11:21:09.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.959+0000 7fa61d7b1700 1 -- start start 2023-10-16T11:21:09.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.959+0000 7fa61d7b1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa61815a490 0x7fa61815e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:09.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.959+0000 7fa61d7b1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa61806cc60 0x7fa6180cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:09.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.959+0000 7fa61d7b1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61815a870 con 0x7fa618071600 2023-10-16T11:21:09.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.959+0000 7fa61d7b1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61815eed0 con 0x7fa61806cc60 2023-10-16T11:21:09.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.959+0000 7fa61d7b1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180cab90 con 0x7fa61815a490 2023-10-16T11:21:09.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6177fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa61806cc60 0x7fa6180cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:09.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6167fc700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa618071600 0x7fa61806c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42096/0 (socket says 172.21.15.158:42096) 2023-10-16T11:21:09.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6167fc700 1 -- 172.21.15.158:0/3750762305 learned_addr learned my addr 172.21.15.158:0/3750762305 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:09.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa616ffd700 1 --2- 172.21.15.158:0/3750762305 >> v2:172.21.15.172:3300/0 conn(0x7fa61815a490 0x7fa61815e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:09.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6177fe700 1 -- 172.21.15.158:0/3750762305 >> v2:172.21.15.172:3300/0 conn(0x7fa61815a490 msgr2=0x7fa61815e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:09.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6177fe700 1 --2- 172.21.15.158:0/3750762305 >> v2:172.21.15.172:3300/0 conn(0x7fa61815a490 0x7fa61815e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6177fe700 1 -- 172.21.15.158:0/3750762305 >> v1:172.21.15.158:6789/0 conn(0x7fa618071600 legacy=0x7fa61806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:09.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6177fe700 1 -- 172.21.15.158:0/3750762305 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180cae70 con 0x7fa61806cc60 2023-10-16T11:21:09.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa615ffb700 1 -- 172.21.15.158:0/3750762305 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122765561 0 0) 0x7fa61815a870 con 0x7fa618071600 2023-10-16T11:21:09.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa6177fe700 1 --2- 172.21.15.158:0/3750762305 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61806cc60 0x7fa6180cd280 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fa6180d00f0 tx=0x7fa61000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e871cfb56e46a59 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:09.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa615ffb700 1 -- 172.21.15.158:0/3750762305 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000e070 con 0x7fa61806cc60 2023-10-16T11:21:09.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa615ffb700 1 -- 172.21.15.158:0/3750762305 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61000bbd0 con 0x7fa61806cc60 2023-10-16T11:21:09.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.960+0000 7fa615ffb700 1 -- 172.21.15.158:0/3750762305 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610012430 con 0x7fa61806cc60 2023-10-16T11:21:09.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3750762305 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61806cc60 msgr2=0x7fa6180cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:09.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3750762305 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61806cc60 0x7fa6180cd280 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fa6180d00f0 tx=0x7fa61000b330 comp rx=0 tx=0).stop 2023-10-16T11:21:09.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3750762305 shutdown_connections 2023-10-16T11:21:09.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3750762305 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61806cc60 0x7fa6180cd280 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3750762305 >> v2:172.21.15.172:3300/0 conn(0x7fa61815a490 0x7fa61815e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3750762305 >> 172.21.15.158:0/3750762305 conn(0x7fa61806af00 msgr2=0x7fa6180cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:09.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3750762305 shutdown_connections 2023-10-16T11:21:09.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.961+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3750762305 wait complete. 2023-10-16T11:21:09.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.962+0000 7fa61d7b1700 1 Processor -- start 2023-10-16T11:21:09.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.962+0000 7fa61d7b1700 1 -- start start 2023-10-16T11:21:09.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa61d7b1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa61806cc60 0x7fa618159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:09.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa61d7b1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa618071600 0x7fa6180c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:09.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa61d7b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61815a490 0x7fa6180c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:09.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa61d7b1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61815bdd0 con 0x7fa61815a490 2023-10-16T11:21:09.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa61d7b1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61815c2f0 con 0x7fa618071600 2023-10-16T11:21:09.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa61d7b1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c8eb0 con 0x7fa61806cc60 2023-10-16T11:21:09.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61815a490 0x7fa6180c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:09.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61815a490 0x7fa6180c4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41928/0 (socket says 172.21.15.158:41928) 2023-10-16T11:21:09.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 -- 172.21.15.158:0/3558967174 learned_addr learned my addr 172.21.15.158:0/3558967174 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:09.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa616ffd700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa61806cc60 0x7fa618159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:09.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6167fc700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa618071600 0x7fa6180c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:09.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 -- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa61806cc60 msgr2=0x7fa618159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:09.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa61806cc60 0x7fa618159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 -- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa618071600 msgr2=0x7fa6180c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:09.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa618071600 0x7fa6180c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:09.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.963+0000 7fa6177fe700 1 -- 172.21.15.158:0/3558967174 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610009a70 con 0x7fa61815a490 2023-10-16T11:21:09.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.964+0000 7fa6177fe700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61815a490 0x7fa6180c4e10 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fa6180cf8f0 tx=0x7fa610004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:09.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.964+0000 7fa5ff7fe700 1 -- 172.21.15.158:0/3558967174 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610012430 con 0x7fa61815a490 2023-10-16T11:21:09.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.964+0000 7fa5ff7fe700 1 -- 172.21.15.158:0/3558967174 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610020070 con 0x7fa61815a490 2023-10-16T11:21:09.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.964+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61815ad30 con 0x7fa61815a490 2023-10-16T11:21:09.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.964+0000 7fa5ff7fe700 1 -- 172.21.15.158:0/3558967174 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610004280 con 0x7fa61815a490 2023-10-16T11:21:09.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.964+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61815b2f0 con 0x7fa61815a490 2023-10-16T11:21:09.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.965+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6180ca7e0 con 0x7fa61815a490 2023-10-16T11:21:09.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.965+0000 7fa5ff7fe700 1 -- 172.21.15.158:0/3558967174 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa6100043e0 con 0x7fa61815a490 2023-10-16T11:21:09.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.966+0000 7fa5ff7fe700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa60406b140 0x7fa60406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:09.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.966+0000 7fa5ff7fe700 1 -- 172.21.15.158:0/3558967174 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7109+0+0 (secure 0 0 0) 0x7fa610022070 con 0x7fa61815a490 2023-10-16T11:21:09.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.966+0000 7fa616ffd700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa60406b140 0x7fa60406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:09.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.966+0000 7fa616ffd700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa60406b140 0x7fa60406d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00b9b0 tx=0x7fa60c008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:09.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:09.970+0000 7fa5ff7fe700 1 -- 172.21.15.158:0/3558967174 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa610056570 con 0x7fa61815a490 2023-10-16T11:21:10.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.159+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa6180681b0 con 0x7fa61815a490 2023-10-16T11:21:10.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.808+0000 7fa5ff7fe700 1 -- 172.21.15.158:0/3558967174 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v249) v1 ==== 178+0+0 (secure 0 0 0) 0x7fa61002c0d0 con 0x7fa61815a490 2023-10-16T11:21:10.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa60406b140 msgr2=0x7fa60406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:10.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa60406b140 0x7fa60406d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00b9b0 tx=0x7fa60c008040 comp rx=0 tx=0).stop 2023-10-16T11:21:10.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61815a490 msgr2=0x7fa6180c4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:10.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61815a490 0x7fa6180c4e10 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fa6180cf8f0 tx=0x7fa610004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 shutdown_connections 2023-10-16T11:21:10.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa60406b140 0x7fa60406d600 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa61815a490 0x7fa6180c4e10 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa61806cc60 0x7fa618159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 --2- 172.21.15.158:0/3558967174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa618071600 0x7fa6180c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 >> 172.21.15.158:0/3558967174 conn(0x7fa61806af00 msgr2=0x7fa6180ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:10.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 shutdown_connections 2023-10-16T11:21:10.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.811+0000 7fa61d7b1700 1 -- 172.21.15.158:0/3558967174 wait complete. 2023-10-16T11:21:10.816 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'backup_fs_data' removed 2023-10-16T11:21:10.826 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:21:10.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.951+0000 7f502b4ef700 1 Processor -- start 2023-10-16T11:21:10.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.951+0000 7f502b4ef700 1 -- start start 2023-10-16T11:21:10.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f502b4ef700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f502415da40 0x7f50240caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f502b4ef700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50240cbd30 con 0x7f50240cc590 2023-10-16T11:21:10.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f502b4ef700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50240cc1c0 con 0x7f502406afe0 2023-10-16T11:21:10.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f502b4ef700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f502415de20 con 0x7f502415da40 2023-10-16T11:21:10.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f5028a8a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f50240cc590 0x7f502406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42108/0 (socket says 172.21.15.158:42108) 2023-10-16T11:21:10.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f5028a8a700 1 -- 172.21.15.158:0/1228835507 learned_addr learned my addr 172.21.15.158:0/1228835507 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:10.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046056017 0 0) 0x7f50240cbd30 con 0x7f50240cc590 2023-10-16T11:21:10.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50100035b0 con 0x7f50240cc590 2023-10-16T11:21:10.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f502928b700 1 --2- 172.21.15.158:0/1228835507 >> v2:172.21.15.172:3300/0 conn(0x7f502415da40 0x7f50240caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2185897369 0 0) 0x7f50100035b0 con 0x7f50240cc590 2023-10-16T11:21:10.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 >> v2:172.21.15.172:3300/0 conn(0x7f502415da40 msgr2=0x7f50240caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:10.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 --2- 172.21.15.158:0/1228835507 >> v2:172.21.15.172:3300/0 conn(0x7f502415da40 0x7f50240caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 >> v1:172.21.15.172:6789/0 conn(0x7f502406afe0 legacy=0x7f50240afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:10.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50240ce260 con 0x7f50240cc590 2023-10-16T11:21:10.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f5020001670 con 0x7f50240cc590 2023-10-16T11:21:10.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.952+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 >> v1:172.21.15.158:6789/0 conn(0x7f50240cc590 legacy=0x7f502406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:10.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5010003a00 0x7f5010003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5010004850 0x7f5010004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010005210 0x7f501000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502415de20 con 0x7f5010003a00 2023-10-16T11:21:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501000c160 con 0x7f5010004850 2023-10-16T11:21:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501000c320 con 0x7f5010005210 2023-10-16T11:21:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f502928b700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5010003a00 0x7f5010003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f5028a8a700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5010004850 0x7f5010004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f5029a8c700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010005210 0x7f501000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f502928b700 1 -- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010005210 msgr2=0x7f501000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:10.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f502928b700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010005210 0x7f501000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f502928b700 1 -- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5010004850 msgr2=0x7f5010004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:10.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f502928b700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5010004850 0x7f5010004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f5029a8c700 1 -- 172.21.15.158:0/1228835507 reap_dead start 2023-10-16T11:21:10.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f502928b700 1 -- 172.21.15.158:0/1228835507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50100035b0 con 0x7f5010003a00 2023-10-16T11:21:10.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f502928b700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5010003a00 0x7f5010003e20 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f5014009890 tx=0x7f5010005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccaf8a56fe8be498 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501400c070 con 0x7f5010003a00 2023-10-16T11:21:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014013610 con 0x7f5010003a00 2023-10-16T11:21:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.953+0000 7f501bfff700 1 -- 172.21.15.158:0/1228835507 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50140139c0 con 0x7f5010003a00 2023-10-16T11:21:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.954+0000 7f502b4ef700 1 -- 172.21.15.158:0/1228835507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5010003a00 msgr2=0x7f5010003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:10.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.954+0000 7f502b4ef700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5010003a00 0x7f5010003e20 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f5014009890 tx=0x7f5010005880 comp rx=0 tx=0).stop 2023-10-16T11:21:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.954+0000 7f502b4ef700 1 -- 172.21.15.158:0/1228835507 shutdown_connections 2023-10-16T11:21:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.954+0000 7f502b4ef700 1 --2- 172.21.15.158:0/1228835507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5010003a00 0x7f5010003e20 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.954+0000 7f502b4ef700 1 -- 172.21.15.158:0/1228835507 >> 172.21.15.158:0/1228835507 conn(0x7f5024099a10 msgr2=0x7f50240d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.954+0000 7f502b4ef700 1 -- 172.21.15.158:0/1228835507 shutdown_connections 2023-10-16T11:21:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.954+0000 7f502b4ef700 1 -- 172.21.15.158:0/1228835507 wait complete. 2023-10-16T11:21:10.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.955+0000 7f502b4ef700 1 Processor -- start 2023-10-16T11:21:10.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.955+0000 7f502b4ef700 1 -- start start 2023-10-16T11:21:10.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502b4ef700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010003a00 0x7f50240c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502b4ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50240c3e30 0x7f50241f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502b4ef700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f502414d710 0x7f50240b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502b4ef700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5024159df0 con 0x7f50240c3e30 2023-10-16T11:21:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502b4ef700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50241f7220 con 0x7f502414d710 2023-10-16T11:21:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502b4ef700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50240c5ac0 con 0x7f5010003a00 2023-10-16T11:21:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50240c3e30 0x7f50241f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50240c3e30 0x7f50241f6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41944/0 (socket says 172.21.15.158:41944) 2023-10-16T11:21:10.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 -- 172.21.15.158:0/3035264993 learned_addr learned my addr 172.21.15.158:0/3035264993 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:10.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502928b700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010003a00 0x7f50240c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5029a8c700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f502414d710 0x7f50240b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 -- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010003a00 msgr2=0x7f50240c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:10.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010003a00 0x7f50240c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 -- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f502414d710 msgr2=0x7f50240b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:10.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f502414d710 0x7f50240b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:10.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 -- 172.21.15.158:0/3035264993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50240c5d60 con 0x7f50240c3e30 2023-10-16T11:21:10.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5028a8a700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50240c3e30 0x7f50241f6ce0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f5020007290 tx=0x7f50200075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:10.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5019ffb700 1 -- 172.21.15.158:0/3035264993 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5020011580 con 0x7f50240c3e30 2023-10-16T11:21:10.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5019ffb700 1 -- 172.21.15.158:0/3035264993 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5020011700 con 0x7f50240c3e30 2023-10-16T11:21:10.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50240c6040 con 0x7f50240c3e30 2023-10-16T11:21:10.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.956+0000 7f5019ffb700 1 -- 172.21.15.158:0/3035264993 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5020011ab0 con 0x7f50240c3e30 2023-10-16T11:21:10.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.957+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50240c4380 con 0x7f50240c3e30 2023-10-16T11:21:10.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.958+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50240c4510 con 0x7f50240c3e30 2023-10-16T11:21:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.958+0000 7f5019ffb700 1 -- 172.21.15.158:0/3035264993 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5020011c10 con 0x7f50240c3e30 2023-10-16T11:21:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.958+0000 7f5019ffb700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f500806b140 0x7f500806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.959+0000 7f5019ffb700 1 -- 172.21.15.158:0/3035264993 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f5020089c60 con 0x7f50240c3e30 2023-10-16T11:21:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.959+0000 7f502928b700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f500806b140 0x7f500806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:10.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.959+0000 7f502928b700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f500806b140 0x7f500806d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5014016040 tx=0x7f501400c000 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:10.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:10.963+0000 7f5019ffb700 1 -- 172.21.15.158:0/3035264993 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50200546b0 con 0x7f50240c3e30 2023-10-16T11:21:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.155+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f50240681b0 con 0x7f50240c3e30 2023-10-16T11:21:11.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.159+0000 7f5019ffb700 1 -- 172.21.15.158:0/3035264993 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5020057cd0 con 0x7f50240c3e30 2023-10-16T11:21:11.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f500806b140 msgr2=0x7f500806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f500806b140 0x7f500806d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5014016040 tx=0x7f501400c000 comp rx=0 tx=0).stop 2023-10-16T11:21:11.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50240c3e30 msgr2=0x7f50241f6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50240c3e30 0x7f50241f6ce0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f5020007290 tx=0x7f50200075a0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 shutdown_connections 2023-10-16T11:21:11.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f500806b140 0x7f500806d600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f502414d710 0x7f50240b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5010003a00 0x7f50240c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 --2- 172.21.15.158:0/3035264993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50240c3e30 0x7f50241f6ce0 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 >> 172.21.15.158:0/3035264993 conn(0x7f5024099a10 msgr2=0x7f50240d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:11.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.161+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 shutdown_connections 2023-10-16T11:21:11.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.162+0000 7f502b4ef700 1 -- 172.21.15.158:0/3035264993 wait complete. 2023-10-16T11:21:11.176 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:21:11.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.298+0000 7f933519d700 1 Processor -- start 2023-10-16T11:21:11.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.298+0000 7f933519d700 1 -- start start 2023-10-16T11:21:11.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f933519d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f933015d910 0x7f93300cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f933519d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93300cc9e0 0x7f93300ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f933519d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933006a8d0 con 0x7f93300700a0 2023-10-16T11:21:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f933519d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9330161e50 con 0x7f933015d910 2023-10-16T11:21:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f933519d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9330072970 con 0x7f93300cc9e0 2023-10-16T11:21:11.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f933015d910 0x7f93300cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932ed9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f933015d910 0x7f93300cdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41950/0 (socket says 172.21.15.158:41950) 2023-10-16T11:21:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932ed9d700 1 -- 172.21.15.158:0/1926922837 learned_addr learned my addr 172.21.15.158:0/1926922837 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932ed9d700 1 -- 172.21.15.158:0/1926922837 >> v2:172.21.15.172:3300/0 conn(0x7f93300cc9e0 msgr2=0x7f93300ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932e59c700 1 --2- 172.21.15.158:0/1926922837 >> v2:172.21.15.172:3300/0 conn(0x7f93300cc9e0 0x7f93300ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932ed9d700 1 --2- 172.21.15.158:0/1926922837 >> v2:172.21.15.172:3300/0 conn(0x7f93300cc9e0 0x7f93300ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932ed9d700 1 -- 172.21.15.158:0/1926922837 >> v1:172.21.15.172:6789/0 conn(0x7f93300700a0 legacy=0x7f933006ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.299+0000 7f932ed9d700 1 -- 172.21.15.158:0/1926922837 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933015dcf0 con 0x7f933015d910 2023-10-16T11:21:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f932ed9d700 1 --2- 172.21.15.158:0/1926922837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f93300cdf70 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f93301613e0 tx=0x7f9324009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ca07f9ae7da175 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:11.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f932dd9b700 1 -- 172.21.15.158:0/1926922837 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324004230 con 0x7f933015d910 2023-10-16T11:21:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f932dd9b700 1 -- 172.21.15.158:0/1926922837 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324004390 con 0x7f933015d910 2023-10-16T11:21:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f932dd9b700 1 -- 172.21.15.158:0/1926922837 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93240036a0 con 0x7f933015d910 2023-10-16T11:21:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 -- 172.21.15.158:0/1926922837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 msgr2=0x7f93300cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 --2- 172.21.15.158:0/1926922837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f93300cdf70 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f93301613e0 tx=0x7f9324009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 -- 172.21.15.158:0/1926922837 shutdown_connections 2023-10-16T11:21:11.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 --2- 172.21.15.158:0/1926922837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f93300cdf70 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 --2- 172.21.15.158:0/1926922837 >> v2:172.21.15.172:3300/0 conn(0x7f93300cc9e0 0x7f93300ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 -- 172.21.15.158:0/1926922837 >> 172.21.15.158:0/1926922837 conn(0x7f93300b1e90 msgr2=0x7f93300b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:11.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 -- 172.21.15.158:0/1926922837 shutdown_connections 2023-10-16T11:21:11.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.300+0000 7f933519d700 1 -- 172.21.15.158:0/1926922837 wait complete. 2023-10-16T11:21:11.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.302+0000 7f933519d700 1 Processor -- start 2023-10-16T11:21:11.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.302+0000 7f933519d700 1 -- start start 2023-10-16T11:21:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f933519d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93300700a0 0x7f93300ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f933519d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93300cc9e0 0x7f93300af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f933519d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f9330154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f933519d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93301f7570 con 0x7f933015d910 2023-10-16T11:21:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f933519d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93300af6e0 con 0x7f93300cc9e0 2023-10-16T11:21:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f933519d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93300b3bf0 con 0x7f93300700a0 2023-10-16T11:21:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f9330154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f9330154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41960/0 (socket says 172.21.15.158:41960) 2023-10-16T11:21:11.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 -- 172.21.15.158:0/1160283856 learned_addr learned my addr 172.21.15.158:0/1160283856 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932ed9d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93300700a0 0x7f93300ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932e59c700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93300cc9e0 0x7f93300af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 -- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93300700a0 msgr2=0x7f93300ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93300700a0 0x7f93300ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 -- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93300cc9e0 msgr2=0x7f93300af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93300cc9e0 0x7f93300af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 -- 172.21.15.158:0/1160283856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324009550 con 0x7f933015d910 2023-10-16T11:21:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f932f59e700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f9330154d60 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7f932800b990 tx=0x7f932800bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f93177fe700 1 -- 172.21.15.158:0/1160283856 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800d890 con 0x7f933015d910 2023-10-16T11:21:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f93177fe700 1 -- 172.21.15.158:0/1160283856 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932800d9f0 con 0x7f933015d910 2023-10-16T11:21:11.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93300b3ed0 con 0x7f933015d910 2023-10-16T11:21:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.303+0000 7f93177fe700 1 -- 172.21.15.158:0/1160283856 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800dda0 con 0x7f933015d910 2023-10-16T11:21:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.304+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933015f820 con 0x7f933015d910 2023-10-16T11:21:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.304+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93300681b0 con 0x7f933015d910 2023-10-16T11:21:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.305+0000 7f93177fe700 1 -- 172.21.15.158:0/1160283856 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9328006d50 con 0x7f933015d910 2023-10-16T11:21:11.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.305+0000 7f93177fe700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f931c06b140 0x7f931c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.305+0000 7f93177fe700 1 -- 172.21.15.158:0/1160283856 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f9328089e30 con 0x7f933015d910 2023-10-16T11:21:11.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.305+0000 7f932ed9d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f931c06b140 0x7f931c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.306+0000 7f932ed9d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f931c06b140 0x7f931c06d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9324009130 tx=0x7f9324008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:11.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.310+0000 7f93177fe700 1 -- 172.21.15.158:0/1160283856 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9328054690 con 0x7f933015d910 2023-10-16T11:21:11.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.509+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f93300c71e0 con 0x7f933015d910 2023-10-16T11:21:11.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.515+0000 7f93177fe700 1 -- 172.21.15.158:0/1160283856 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9328057cb0 con 0x7f933015d910 2023-10-16T11:21:11.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.519+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f931c06b140 msgr2=0x7f931c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.519+0000 7f933519d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f931c06b140 0x7f931c06d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9324009130 tx=0x7f9324008030 comp rx=0 tx=0).stop 2023-10-16T11:21:11.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.519+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 msgr2=0x7f9330154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.519+0000 7f933519d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f9330154d60 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7f932800b990 tx=0x7f932800bca0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 shutdown_connections 2023-10-16T11:21:11.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f931c06b140 0x7f931c06d600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f933015d910 0x7f9330154d60 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93300700a0 0x7f93300ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 --2- 172.21.15.158:0/1160283856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93300cc9e0 0x7f93300af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 >> 172.21.15.158:0/1160283856 conn(0x7f93300b1e90 msgr2=0x7f93300b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:11.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 shutdown_connections 2023-10-16T11:21:11.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.520+0000 7f933519d700 1 -- 172.21.15.158:0/1160283856 wait complete. 2023-10-16T11:21:11.534 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:21:11.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.654+0000 7f72837ef700 1 Processor -- start 2023-10-16T11:21:11.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.654+0000 7f72837ef700 1 -- start start 2023-10-16T11:21:11.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.655+0000 7f72837ef700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f727c0cc200 0x7f727c15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.655+0000 7f72837ef700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c0cf4c0 con 0x7f727c0ce510 2023-10-16T11:21:11.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.655+0000 7f72837ef700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c15f710 con 0x7f727c15b590 2023-10-16T11:21:11.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.655+0000 7f72837ef700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727c0b49e0 con 0x7f727c0cc200 2023-10-16T11:21:11.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.655+0000 7f7281d8c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f727c0cc200 0x7f727c15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.655+0000 7f7280d8a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f727c0ce510 0x7f727c0cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42142/0 (socket says 172.21.15.158:42142) 2023-10-16T11:21:11.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.655+0000 7f7280d8a700 1 -- 172.21.15.158:0/3310278444 learned_addr learned my addr 172.21.15.158:0/3310278444 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:11.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7281d8c700 1 -- 172.21.15.158:0/3310278444 >> v1:172.21.15.172:6789/0 conn(0x7f727c15b590 legacy=0x7f727c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:11.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7281d8c700 1 -- 172.21.15.158:0/3310278444 >> v1:172.21.15.158:6789/0 conn(0x7f727c0ce510 legacy=0x7f727c0cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7273fff700 1 -- 172.21.15.158:0/3310278444 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953388575 0 0) 0x7f727c0cf4c0 con 0x7f727c0ce510 2023-10-16T11:21:11.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7281d8c700 1 -- 172.21.15.158:0/3310278444 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c15f970 con 0x7f727c0cc200 2023-10-16T11:21:11.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7281d8c700 1 --2- 172.21.15.158:0/3310278444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c0cc200 0x7f727c15f1d0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f7274009dc0 tx=0x7f727400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4eb3b754bc073 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:11.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7273fff700 1 -- 172.21.15.158:0/3310278444 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727400e070 con 0x7f727c0cc200 2023-10-16T11:21:11.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7273fff700 1 -- 172.21.15.158:0/3310278444 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727400bb90 con 0x7f727c0cc200 2023-10-16T11:21:11.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f7273fff700 1 -- 172.21.15.158:0/3310278444 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274012430 con 0x7f727c0cc200 2023-10-16T11:21:11.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f72837ef700 1 -- 172.21.15.158:0/3310278444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c0cc200 msgr2=0x7f727c15f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f72837ef700 1 --2- 172.21.15.158:0/3310278444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c0cc200 0x7f727c15f1d0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f7274009dc0 tx=0x7f727400b330 comp rx=0 tx=0).stop 2023-10-16T11:21:11.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f72837ef700 1 -- 172.21.15.158:0/3310278444 shutdown_connections 2023-10-16T11:21:11.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f72837ef700 1 --2- 172.21.15.158:0/3310278444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c0cc200 0x7f727c15f1d0 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.656+0000 7f72837ef700 1 -- 172.21.15.158:0/3310278444 >> 172.21.15.158:0/3310278444 conn(0x7f727c06a8d0 msgr2=0x7f727c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:11.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.657+0000 7f72837ef700 1 -- 172.21.15.158:0/3310278444 shutdown_connections 2023-10-16T11:21:11.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.657+0000 7f72837ef700 1 -- 172.21.15.158:0/3310278444 wait complete. 2023-10-16T11:21:11.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 Processor -- start 2023-10-16T11:21:11.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 -- start start 2023-10-16T11:21:11.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f727c0cc200 0x7f727c1f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f727c0ce510 0x7f727c15d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c15b590 0x7f727c15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c15d2c0 con 0x7f727c15b590 2023-10-16T11:21:11.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c15ca20 con 0x7f727c0cc200 2023-10-16T11:21:11.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.658+0000 7f72837ef700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727c15cd00 con 0x7f727c0ce510 2023-10-16T11:21:11.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c15b590 0x7f727c15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c15b590 0x7f727c15df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41974/0 (socket says 172.21.15.158:41974) 2023-10-16T11:21:11.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 -- 172.21.15.158:0/2076782671 learned_addr learned my addr 172.21.15.158:0/2076782671 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:11.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 -- 172.21.15.158:0/2076782671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f727c0ce510 msgr2=0x7f727c15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f727c0ce510 0x7f727c15d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 -- 172.21.15.158:0/2076782671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f727c0cc200 msgr2=0x7f727c1f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f727c0cc200 0x7f727c1f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 -- 172.21.15.158:0/2076782671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7274009a70 con 0x7f727c15b590 2023-10-16T11:21:11.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7281d8c700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c15b590 0x7f727c15df30 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f7274013000 tx=0x7f7274004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:11.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7271ffb700 1 -- 172.21.15.158:0/2076782671 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727401b8e0 con 0x7f727c15b590 2023-10-16T11:21:11.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7271ffb700 1 -- 172.21.15.158:0/2076782671 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7274020070 con 0x7f727c15b590 2023-10-16T11:21:11.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f7271ffb700 1 -- 172.21.15.158:0/2076782671 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727401bca0 con 0x7f727c15b590 2023-10-16T11:21:11.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727c15cfe0 con 0x7f727c15b590 2023-10-16T11:21:11.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.659+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727c1f0c70 con 0x7f727c15b590 2023-10-16T11:21:11.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.660+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727c0c8570 con 0x7f727c15b590 2023-10-16T11:21:11.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.661+0000 7f7271ffb700 1 -- 172.21.15.158:0/2076782671 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7274026400 con 0x7f727c15b590 2023-10-16T11:21:11.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.661+0000 7f7271ffb700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f726006b140 0x7f726006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:11.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.661+0000 7f7271ffb700 1 -- 172.21.15.158:0/2076782671 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f727408bca0 con 0x7f727c15b590 2023-10-16T11:21:11.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.661+0000 7f728158b700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f726006b140 0x7f726006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:11.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.661+0000 7f728158b700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f726006b140 0x7f726006d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f726c000f60 tx=0x7f726c007a40 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:11.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.665+0000 7f7271ffb700 1 -- 172.21.15.158:0/2076782671 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f727c0c8570 con 0x7f727c15b590 2023-10-16T11:21:11.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.862+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f727c0c8950 con 0x7f727c15b590 2023-10-16T11:21:11.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.864+0000 7f7271ffb700 1 -- 172.21.15.158:0/2076782671 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f727c0c8950 con 0x7f727c15b590 2023-10-16T11:21:11.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f726006b140 msgr2=0x7f726006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f726006b140 0x7f726006d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f726c000f60 tx=0x7f726c007a40 comp rx=0 tx=0).stop 2023-10-16T11:21:11.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c15b590 msgr2=0x7f727c15df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:11.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c15b590 0x7f727c15df30 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f7274013000 tx=0x7f7274004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 shutdown_connections 2023-10-16T11:21:11.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f726006b140 0x7f726006d600 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f727c15b590 0x7f727c15df30 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f727c0cc200 0x7f727c1f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 --2- 172.21.15.158:0/2076782671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f727c0ce510 0x7f727c15d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:11.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 >> 172.21.15.158:0/2076782671 conn(0x7f727c06a8d0 msgr2=0x7f727c0b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:11.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 shutdown_connections 2023-10-16T11:21:11.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:11.867+0000 7f72837ef700 1 -- 172.21.15.158:0/2076782671 wait complete. 2023-10-16T11:21:11.882 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T11:21:12.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3795c8700 1 Processor -- start 2023-10-16T11:21:12.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3795c8700 1 -- start start 2023-10-16T11:21:12.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3795c8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb37415db90 0x7fb37406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3795c8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3740cddb0 0x7fb3740ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3795c8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3740d0fb0 con 0x7fb37415d7b0 2023-10-16T11:21:12.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3795c8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37415e100 con 0x7fb37415db90 2023-10-16T11:21:12.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3795c8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37415e3e0 con 0x7fb3740cddb0 2023-10-16T11:21:12.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.001+0000 7fb3727fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb37415db90 0x7fb37406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3727fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb37415db90 0x7fb37406b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41982/0 (socket says 172.21.15.158:41982) 2023-10-16T11:21:12.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3727fc700 1 -- 172.21.15.158:0/2271635071 learned_addr learned my addr 172.21.15.158:0/2271635071 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3737fe700 1 --2- 172.21.15.158:0/2271635071 >> v2:172.21.15.172:3300/0 conn(0x7fb3740cddb0 0x7fb3740ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3727fc700 1 -- 172.21.15.158:0/2271635071 >> v2:172.21.15.172:3300/0 conn(0x7fb3740cddb0 msgr2=0x7fb3740ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3727fc700 1 --2- 172.21.15.158:0/2271635071 >> v2:172.21.15.172:3300/0 conn(0x7fb3740cddb0 0x7fb3740ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3727fc700 1 -- 172.21.15.158:0/2271635071 >> v1:172.21.15.172:6789/0 conn(0x7fb37415d7b0 legacy=0x7fb3740b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3727fc700 1 -- 172.21.15.158:0/2271635071 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37415e6c0 con 0x7fb37415db90 2023-10-16T11:21:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb3727fc700 1 --2- 172.21.15.158:0/2271635071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb37415db90 0x7fb37406b9e0 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b0e10 tx=0x7fb364009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cb777db4f7640d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb371ffb700 1 -- 172.21.15.158:0/2271635071 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364004230 con 0x7fb37415db90 2023-10-16T11:21:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb371ffb700 1 -- 172.21.15.158:0/2271635071 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364004390 con 0x7fb37415db90 2023-10-16T11:21:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.002+0000 7fb371ffb700 1 -- 172.21.15.158:0/2271635071 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640036a0 con 0x7fb37415db90 2023-10-16T11:21:12.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 -- 172.21.15.158:0/2271635071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb37415db90 msgr2=0x7fb37406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:12.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 --2- 172.21.15.158:0/2271635071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb37415db90 0x7fb37406b9e0 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b0e10 tx=0x7fb364009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 -- 172.21.15.158:0/2271635071 shutdown_connections 2023-10-16T11:21:12.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 --2- 172.21.15.158:0/2271635071 >> v2:172.21.15.172:3300/0 conn(0x7fb3740cddb0 0x7fb3740ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 --2- 172.21.15.158:0/2271635071 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb37415db90 0x7fb37406b9e0 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 -- 172.21.15.158:0/2271635071 >> 172.21.15.158:0/2271635071 conn(0x7fb374071220 msgr2=0x7fb37406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:12.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 -- 172.21.15.158:0/2271635071 shutdown_connections 2023-10-16T11:21:12.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.003+0000 7fb3795c8700 1 -- 172.21.15.158:0/2271635071 wait complete. 2023-10-16T11:21:12.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.004+0000 7fb3795c8700 1 Processor -- start 2023-10-16T11:21:12.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.004+0000 7fb3795c8700 1 -- start start 2023-10-16T11:21:12.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3795c8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3740cddb0 0x7fb3740aff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3795c8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb37415d7b0 0x7fb3741f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3795c8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb37415db90 0x7fb374160f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3795c8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741618d0 con 0x7fb3740cddb0 2023-10-16T11:21:12.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3795c8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374161bb0 con 0x7fb37415db90 2023-10-16T11:21:12.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3795c8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740c5e40 con 0x7fb37415d7b0 2023-10-16T11:21:12.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3740cddb0 0x7fb3740aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3740cddb0 0x7fb3740aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41984/0 (socket says 172.21.15.158:41984) 2023-10-16T11:21:12.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 -- 172.21.15.158:0/1000032057 learned_addr learned my addr 172.21.15.158:0/1000032057 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:12.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3727fc700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb37415d7b0 0x7fb3741f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 -- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb37415d7b0 msgr2=0x7fb3741f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:12.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb37415d7b0 0x7fb3741f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 -- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb37415db90 msgr2=0x7fb374160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:12.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3737fe700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb37415db90 0x7fb374160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb37415db90 0x7fb374160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 -- 172.21.15.158:0/1000032057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364009550 con 0x7fb3740cddb0 2023-10-16T11:21:12.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3727fc700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb37415d7b0 0x7fb3741f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:12.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb372ffd700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3740cddb0 0x7fb3740aff80 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7fb36800c4d0 tx=0x7fb36800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:12.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb35b7fe700 1 -- 172.21.15.158:0/1000032057 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011790 con 0x7fb3740cddb0 2023-10-16T11:21:12.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb35b7fe700 1 -- 172.21.15.158:0/1000032057 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3680118f0 con 0x7fb3740cddb0 2023-10-16T11:21:12.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3740c6120 con 0x7fb3740cddb0 2023-10-16T11:21:12.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.005+0000 7fb35b7fe700 1 -- 172.21.15.158:0/1000032057 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011ca0 con 0x7fb3740cddb0 2023-10-16T11:21:12.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.006+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3741f3880 con 0x7fb3740cddb0 2023-10-16T11:21:12.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.007+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3740c6340 con 0x7fb3740cddb0 2023-10-16T11:21:12.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.007+0000 7fb35b7fe700 1 -- 172.21.15.158:0/1000032057 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb36801c460 con 0x7fb3740cddb0 2023-10-16T11:21:12.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.007+0000 7fb35b7fe700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb36006b140 0x7fb36006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.007+0000 7fb35b7fe700 1 -- 172.21.15.158:0/1000032057 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fb368089bf0 con 0x7fb3740cddb0 2023-10-16T11:21:12.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.007+0000 7fb3727fc700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb36006b140 0x7fb36006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.008+0000 7fb3727fc700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb36006b140 0x7fb36006d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb3740c8230 tx=0x7fb364008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:12.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.012+0000 7fb35b7fe700 1 -- 172.21.15.158:0/1000032057 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb368054630 con 0x7fb3740cddb0 2023-10-16T11:21:12.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.284+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fb3740681b0 con 0x7fb3740cddb0 2023-10-16T11:21:12.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.811+0000 7fb35b7fe700 1 -- 172.21.15.158:0/1000032057 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3220) v1 ==== 187+0+0 (secure 0 0 0) 0x7fb368057c50 con 0x7fb3740cddb0 2023-10-16T11:21:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb36006b140 msgr2=0x7fb36006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb36006b140 0x7fb36006d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb3740c8230 tx=0x7fb364008030 comp rx=0 tx=0).stop 2023-10-16T11:21:12.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3740cddb0 msgr2=0x7fb3740aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3740cddb0 0x7fb3740aff80 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7fb36800c4d0 tx=0x7fb36800c890 comp rx=0 tx=0).stop 2023-10-16T11:21:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 shutdown_connections 2023-10-16T11:21:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb36006b140 0x7fb36006d600 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb37415db90 0x7fb374160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3740cddb0 0x7fb3740aff80 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 --2- 172.21.15.158:0/1000032057 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb37415d7b0 0x7fb3741f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 >> 172.21.15.158:0/1000032057 conn(0x7fb374071220 msgr2=0x7fb3740d1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:12.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 shutdown_connections 2023-10-16T11:21:12.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.814+0000 7fb3795c8700 1 -- 172.21.15.158:0/1000032057 wait complete. 2023-10-16T11:21:12.830 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2023-10-16T11:21:12.870 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-16T11:21:12.871 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-16T11:21:12.871 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:12.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.992+0000 7f53556ac700 1 Processor -- start 2023-10-16T11:21:12.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.992+0000 7f53556ac700 1 -- start start 2023-10-16T11:21:12.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f53556ac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f535015d7c0 0x7f53500d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f53556ac700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53500d1210 0x7f53500ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f53556ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53500ce950 0x7f53500b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:12.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f53556ac700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53500d4060 con 0x7f53500d1210 2023-10-16T11:21:12.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f53556ac700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53500d3bd0 con 0x7f535015d7c0 2023-10-16T11:21:12.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f53556ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53500cedc0 con 0x7f53500ce950 2023-10-16T11:21:12.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53500d1210 0x7f53500ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53500d1210 0x7f53500ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41992/0 (socket says 172.21.15.158:41992) 2023-10-16T11:21:12.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 -- 172.21.15.158:0/856712626 learned_addr learned my addr 172.21.15.158:0/856712626 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:12.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534f7fe700 1 --2- 172.21.15.158:0/856712626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53500ce950 0x7f53500b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534effd700 1 --2- 172.21.15.158:0/856712626 >> v2:172.21.15.172:3300/0 conn(0x7f535015d7c0 0x7f53500d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:12.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 -- 172.21.15.158:0/856712626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53500ce950 msgr2=0x7f53500b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:12.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 --2- 172.21.15.158:0/856712626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53500ce950 0x7f53500b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 -- 172.21.15.158:0/856712626 >> v2:172.21.15.172:3300/0 conn(0x7f535015d7c0 msgr2=0x7f53500d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:12.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 --2- 172.21.15.158:0/856712626 >> v2:172.21.15.172:3300/0 conn(0x7f535015d7c0 0x7f53500d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:12.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 -- 172.21.15.158:0/856712626 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500d2cf0 con 0x7f53500d1210 2023-10-16T11:21:13.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534e7fc700 1 --2- 172.21.15.158:0/856712626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53500ce410 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f53500d0bc0 tx=0x7f5340009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3642a762caecc693 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534dffb700 1 -- 172.21.15.158:0/856712626 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534000b870 con 0x7f53500d1210 2023-10-16T11:21:13.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534dffb700 1 -- 172.21.15.158:0/856712626 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534000b9d0 con 0x7f53500d1210 2023-10-16T11:21:13.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.993+0000 7f534dffb700 1 -- 172.21.15.158:0/856712626 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534000bd60 con 0x7f53500d1210 2023-10-16T11:21:13.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 -- 172.21.15.158:0/856712626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 msgr2=0x7f53500ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 --2- 172.21.15.158:0/856712626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53500ce410 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f53500d0bc0 tx=0x7f5340009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 -- 172.21.15.158:0/856712626 shutdown_connections 2023-10-16T11:21:13.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 --2- 172.21.15.158:0/856712626 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53500ce950 0x7f53500b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 --2- 172.21.15.158:0/856712626 >> v2:172.21.15.172:3300/0 conn(0x7f535015d7c0 0x7f53500d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 --2- 172.21.15.158:0/856712626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53500ce410 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 -- 172.21.15.158:0/856712626 >> 172.21.15.158:0/856712626 conn(0x7f53500669f0 msgr2=0x7f5350066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:13.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 -- 172.21.15.158:0/856712626 shutdown_connections 2023-10-16T11:21:13.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.994+0000 7f53556ac700 1 -- 172.21.15.158:0/856712626 wait complete. 2023-10-16T11:21:13.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.996+0000 7f53556ac700 1 Processor -- start 2023-10-16T11:21:13.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.996+0000 7f53556ac700 1 -- start start 2023-10-16T11:21:13.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53556ac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53500ce950 0x7f53501603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53556ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53501f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53556ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f535015d7c0 0x7f53501f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53556ac700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53501521c0 con 0x7f53500d1210 2023-10-16T11:21:13.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53556ac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53501f71c0 con 0x7f53500ce950 2023-10-16T11:21:13.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53556ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5350160a10 con 0x7f535015d7c0 2023-10-16T11:21:13.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53501f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53501f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42000/0 (socket says 172.21.15.158:42000) 2023-10-16T11:21:13.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 -- 172.21.15.158:0/469887659 learned_addr learned my addr 172.21.15.158:0/469887659 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:13.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534f7fe700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f535015d7c0 0x7f53501f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534effd700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53500ce950 0x7f53501603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 -- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f535015d7c0 msgr2=0x7f53501f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f535015d7c0 0x7f53501f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 -- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53500ce950 msgr2=0x7f53501603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53500ce950 0x7f53501603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 -- 172.21.15.158:0/469887659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340009550 con 0x7f53500d1210 2023-10-16T11:21:13.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f534e7fc700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53501f6c80 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f5340000c00 tx=0x7f5340004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53377fe700 1 -- 172.21.15.158:0/469887659 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340004480 con 0x7f53500d1210 2023-10-16T11:21:13.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53377fe700 1 -- 172.21.15.158:0/469887659 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53400045e0 con 0x7f53500d1210 2023-10-16T11:21:13.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535014f200 con 0x7f53500d1210 2023-10-16T11:21:13.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.997+0000 7f53377fe700 1 -- 172.21.15.158:0/469887659 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53400038e0 con 0x7f53500d1210 2023-10-16T11:21:13.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.998+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535014e720 con 0x7f53500d1210 2023-10-16T11:21:13.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.998+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53500681b0 con 0x7f53500d1210 2023-10-16T11:21:13.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.999+0000 7f53377fe700 1 -- 172.21.15.158:0/469887659 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5340021070 con 0x7f53500d1210 2023-10-16T11:21:13.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.999+0000 7f53377fe700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f533c06b140 0x7f533c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.999+0000 7f53377fe700 1 -- 172.21.15.158:0/469887659 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f534008b250 con 0x7f53500d1210 2023-10-16T11:21:13.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:12.999+0000 7f534effd700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f533c06b140 0x7f533c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.000+0000 7f534effd700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f533c06b140 0x7f533c06d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f534400aae0 tx=0x7f5344008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.004+0000 7f53377fe700 1 -- 172.21.15.158:0/469887659 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5340055c50 con 0x7f53500d1210 2023-10-16T11:21:13.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.197+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53500b6380 con 0x7f53500d1210 2023-10-16T11:21:13.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.198+0000 7f53377fe700 1 -- 172.21.15.158:0/469887659 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+15343 (secure 0 0 0) 0x7f5340019e60 con 0x7f53500d1210 2023-10-16T11:21:13.200 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:13.200 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":249,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:10.807578+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6835/1061053278":"2023-10-17T11:21:07.803498+0000","172.21.15.172:6833/1061053278":"2023-10-17T11:21:07.803498+0000","172.21.15.172:6834/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/937741801":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3443938198":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/872331855":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/1271192867":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/331460130":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/387969810":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/2550010748":"2023-10-17T11:20:51.125668+0000","172.21.15.172:6832/468116497":"2023-10-17T11:21:01.033145+0000","172.21.15.158:0/2248613235":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/1739296604":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/3563325503":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/940033591":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3687571003":"2023-10-17T11:20:58.661853+0000","172.21.15.158:6833/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:6832/3408161846":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/750584278":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/3480048181":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/689546029":"2023-10-17T11:20:51.125668+0000","172.21.15.158:0/2035506903":"2023-10-17T11:20:58.661853+0000","172.21.15.158:0/67584808":"2023-10-17T11:20:51.125668+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:13.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f533c06b140 msgr2=0x7f533c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f533c06b140 0x7f533c06d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f534400aae0 tx=0x7f5344008040 comp rx=0 tx=0).stop 2023-10-16T11:21:13.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 msgr2=0x7f53501f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53501f6c80 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f5340000c00 tx=0x7f5340004750 comp rx=0 tx=0).stop 2023-10-16T11:21:13.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 shutdown_connections 2023-10-16T11:21:13.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f533c06b140 0x7f533c06d600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f535015d7c0 0x7f53501f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53500ce950 0x7f53501603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 --2- 172.21.15.158:0/469887659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53500d1210 0x7f53501f6c80 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.200+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 >> 172.21.15.158:0/469887659 conn(0x7f53500669f0 msgr2=0x7f53500cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:13.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.201+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 shutdown_connections 2023-10-16T11:21:13.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.201+0000 7f53556ac700 1 -- 172.21.15.158:0/469887659 wait complete. 2023-10-16T11:21:13.214 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:21:13.215 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:21:13.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.337+0000 7f3be5d9f700 1 Processor -- start 2023-10-16T11:21:13.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.337+0000 7f3be5d9f700 1 -- start start 2023-10-16T11:21:13.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3be5d9f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 0x7f3be00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3be5d9f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3be00cc590 0x7f3be006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3be5d9f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be00cbd30 con 0x7f3be006ae10 2023-10-16T11:21:13.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3be5d9f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be006b280 con 0x7f3be00cc590 2023-10-16T11:21:13.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3be5d9f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00cc1c0 con 0x7f3be015da40 2023-10-16T11:21:13.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bdffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3be006ae10 0x7f3be00ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42186/0 (socket says 172.21.15.158:42186) 2023-10-16T11:21:13.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bdffff700 1 -- 172.21.15.158:0/3064149993 learned_addr learned my addr 172.21.15.158:0/3064149993 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:13.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938911406 0 0) 0x7f3be00cbd30 con 0x7f3be006ae10 2023-10-16T11:21:13.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc80035b0 con 0x7f3be006ae10 2023-10-16T11:21:13.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bdeffd700 1 --2- 172.21.15.158:0/3064149993 >> v2:172.21.15.172:3300/0 conn(0x7f3be00cc590 0x7f3be006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 0x7f3be00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446508420 0 0) 0x7f3bc80035b0 con 0x7f3be006ae10 2023-10-16T11:21:13.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 msgr2=0x7f3be00caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:13.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 0x7f3be00caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 >> v2:172.21.15.172:3300/0 conn(0x7f3be00cc590 msgr2=0x7f3be006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:13.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 --2- 172.21.15.158:0/3064149993 >> v2:172.21.15.172:3300/0 conn(0x7f3be00cc590 0x7f3be006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be015de20 con 0x7f3be006ae10 2023-10-16T11:21:13.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.338+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f3bd80019f0 con 0x7f3be006ae10 2023-10-16T11:21:13.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 >> v1:172.21.15.158:6789/0 conn(0x7f3be006ae10 legacy=0x7f3be00ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3bc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 0x7f3bc8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bc8004d90 0x7f3bc8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be006b280 con 0x7f3bc8003a00 2023-10-16T11:21:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00cc1c0 con 0x7f3bc8004d90 2023-10-16T11:21:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc800bba0 con 0x7f3be015da40 2023-10-16T11:21:13.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3bc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdffff700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bc8004d90 0x7f3bc8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdeffd700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 0x7f3bc8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdf7fe700 1 -- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 msgr2=0x7f3bc8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:13.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 0x7f3bc8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdf7fe700 1 -- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bc8004d90 msgr2=0x7f3bc8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:13.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bc8004d90 0x7f3bc8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdf7fe700 1 -- 172.21.15.158:0/3064149993 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc80035b0 con 0x7f3bc8003a00 2023-10-16T11:21:13.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdeffd700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be015da40 0x7f3bc8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:13.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3bc8003e20 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400aa10 tx=0x7f3bc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99da8b674e873c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400c600 con 0x7f3bc8003a00 2023-10-16T11:21:13.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd400c760 con 0x7f3bc8003a00 2023-10-16T11:21:13.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.339+0000 7f3bde7fc700 1 -- 172.21.15.158:0/3064149993 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400cb10 con 0x7f3bc8003a00 2023-10-16T11:21:13.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.340+0000 7f3be5d9f700 1 -- 172.21.15.158:0/3064149993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 msgr2=0x7f3bc8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.340+0000 7f3be5d9f700 1 --2- 172.21.15.158:0/3064149993 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3bc8003e20 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400aa10 tx=0x7f3bc8005880 comp rx=0 tx=0).stop 2023-10-16T11:21:13.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.340+0000 7f3bdffff700 1 -- 172.21.15.158:0/3064149993 reap_dead start 2023-10-16T11:21:13.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.340+0000 7f3be5d9f700 1 -- 172.21.15.158:0/3064149993 shutdown_connections 2023-10-16T11:21:13.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.340+0000 7f3be5d9f700 1 -- 172.21.15.158:0/3064149993 >> 172.21.15.158:0/3064149993 conn(0x7f3be0099a10 msgr2=0x7f3be00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:13.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.340+0000 7f3be5d9f700 1 -- 172.21.15.158:0/3064149993 shutdown_connections 2023-10-16T11:21:13.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.340+0000 7f3be5d9f700 1 -- 172.21.15.158:0/3064149993 wait complete. 2023-10-16T11:21:13.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.341+0000 7f3be5d9f700 1 Processor -- start 2023-10-16T11:21:13.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.341+0000 7f3be5d9f700 1 -- start start 2023-10-16T11:21:13.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3be5d9f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3be00c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3be5d9f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0154460 0x7f3be00c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3be5d9f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be00c9940 0x7f3be00c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3be5d9f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0157ec0 con 0x7f3bc8003a00 2023-10-16T11:21:13.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3be5d9f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00ca310 con 0x7f3be0154460 2023-10-16T11:21:13.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3be5d9f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00c9060 con 0x7f3be00c9940 2023-10-16T11:21:13.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3be00c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3be00c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42018/0 (socket says 172.21.15.158:42018) 2023-10-16T11:21:13.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 -- 172.21.15.158:0/961767395 learned_addr learned my addr 172.21.15.158:0/961767395 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:13.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdeffd700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0154460 0x7f3be00c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdffff700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be00c9940 0x7f3be00c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 -- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be00c9940 msgr2=0x7f3be00c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be00c9940 0x7f3be00c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 -- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0154460 msgr2=0x7f3be00c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0154460 0x7f3be00c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.342+0000 7f3bdf7fe700 1 -- 172.21.15.158:0/961767395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be01f2d90 con 0x7f3bc8003a00 2023-10-16T11:21:13.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.343+0000 7f3bdf7fe700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3be00c8210 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800baf0 tx=0x7f3bd40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.343+0000 7f3bc7fff700 1 -- 172.21.15.158:0/961767395 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd401f070 con 0x7f3bc8003a00 2023-10-16T11:21:13.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.343+0000 7f3bc7fff700 1 -- 172.21.15.158:0/961767395 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd400a050 con 0x7f3bc8003a00 2023-10-16T11:21:13.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.343+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be0157840 con 0x7f3bc8003a00 2023-10-16T11:21:13.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.343+0000 7f3bc7fff700 1 -- 172.21.15.158:0/961767395 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400cee0 con 0x7f3bc8003a00 2023-10-16T11:21:13.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.343+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be0157ae0 con 0x7f3bc8003a00 2023-10-16T11:21:13.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.344+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be01576a0 con 0x7f3bc8003a00 2023-10-16T11:21:13.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.344+0000 7f3bc7fff700 1 -- 172.21.15.158:0/961767395 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3bd40038b0 con 0x7f3bc8003a00 2023-10-16T11:21:13.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.344+0000 7f3bc7fff700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f3bcc06b140 0x7f3bcc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.345+0000 7f3bc7fff700 1 -- 172.21.15.158:0/961767395 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f3bd408b050 con 0x7f3bc8003a00 2023-10-16T11:21:13.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.345+0000 7f3bdeffd700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f3bcc06b140 0x7f3bcc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.345+0000 7f3bdeffd700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f3bcc06b140 0x7f3bcc06d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3be0154880 tx=0x7f3bd0008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.349+0000 7f3bc7fff700 1 -- 172.21.15.158:0/961767395 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3be01576a0 con 0x7f3bc8003a00 2023-10-16T11:21:13.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.539+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3be01576a0 con 0x7f3bc8003a00 2023-10-16T11:21:13.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.539+0000 7f3bc7fff700 1 -- 172.21.15.158:0/961767395 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v249) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3be01576a0 con 0x7f3bc8003a00 2023-10-16T11:21:13.541 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:21:13.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f3bcc06b140 msgr2=0x7f3bcc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f3bcc06b140 0x7f3bcc06d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3be0154880 tx=0x7f3bd0008040 comp rx=0 tx=0).stop 2023-10-16T11:21:13.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 msgr2=0x7f3be00c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3be00c8210 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800baf0 tx=0x7f3bd40047c0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 shutdown_connections 2023-10-16T11:21:13.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f3bcc06b140 0x7f3bcc06d600 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3be00c9940 0x7f3be00c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bc8003a00 0x7f3be00c8210 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 --2- 172.21.15.158:0/961767395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0154460 0x7f3be00c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 >> 172.21.15.158:0/961767395 conn(0x7f3be0099a10 msgr2=0x7f3be00b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:13.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 shutdown_connections 2023-10-16T11:21:13.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.541+0000 7f3be5d9f700 1 -- 172.21.15.158:0/961767395 wait complete. 2023-10-16T11:21:13.555 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T11:21:13.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.675+0000 7ff5caa69700 1 Processor -- start 2023-10-16T11:21:13.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.675+0000 7ff5caa69700 1 -- start start 2023-10-16T11:21:13.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5caa69700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff5c406ab40 0x7ff5c406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5caa69700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff5c406e5d0 0x7ff5c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5caa69700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c40722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5caa69700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c4071e50 con 0x7ff5c406ab40 2023-10-16T11:21:13.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5caa69700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c4161270 con 0x7ff5c406e5d0 2023-10-16T11:21:13.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5caa69700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4161510 con 0x7ff5c406af20 2023-10-16T11:21:13.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5c8805700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff5c406ab40 0x7ff5c406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5c8805700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff5c406ab40 0x7ff5c406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42026/0 (socket says 172.21.15.158:42026) 2023-10-16T11:21:13.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.676+0000 7ff5c8805700 1 -- 172.21.15.158:0/8569983 learned_addr learned my addr 172.21.15.158:0/8569983 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:13.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c9006700 1 --2- 172.21.15.158:0/8569983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c40722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c3fff700 1 --2- 172.21.15.158:0/8569983 >> v2:172.21.15.172:3300/0 conn(0x7ff5c406e5d0 0x7ff5c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c8805700 1 -- 172.21.15.158:0/8569983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 msgr2=0x7ff5c40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:13.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c8805700 1 --2- 172.21.15.158:0/8569983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c40722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c8805700 1 -- 172.21.15.158:0/8569983 >> v2:172.21.15.172:3300/0 conn(0x7ff5c406e5d0 msgr2=0x7ff5c4071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:13.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c8805700 1 --2- 172.21.15.158:0/8569983 >> v2:172.21.15.172:3300/0 conn(0x7ff5c406e5d0 0x7ff5c4071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c8805700 1 -- 172.21.15.158:0/8569983 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c41617f0 con 0x7ff5c406ab40 2023-10-16T11:21:13.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c3fff700 1 --2- 172.21.15.158:0/8569983 >> v2:172.21.15.172:3300/0 conn(0x7ff5c406e5d0 0x7ff5c4071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:13.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c9006700 1 --2- 172.21.15.158:0/8569983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c40722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:13.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c8805700 1 --2- 172.21.15.158:0/8569983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406ab40 0x7ff5c406d820 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7ff5c40c3bf0 tx=0x7ff5b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b789b79106c10c5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c37fe700 1 -- 172.21.15.158:0/8569983 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800b870 con 0x7ff5c406ab40 2023-10-16T11:21:13.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c37fe700 1 -- 172.21.15.158:0/8569983 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b800b9d0 con 0x7ff5c406ab40 2023-10-16T11:21:13.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.677+0000 7ff5c37fe700 1 -- 172.21.15.158:0/8569983 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800bd60 con 0x7ff5c406ab40 2023-10-16T11:21:13.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 -- 172.21.15.158:0/8569983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406ab40 msgr2=0x7ff5c406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 --2- 172.21.15.158:0/8569983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406ab40 0x7ff5c406d820 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7ff5c40c3bf0 tx=0x7ff5b8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 -- 172.21.15.158:0/8569983 shutdown_connections 2023-10-16T11:21:13.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 --2- 172.21.15.158:0/8569983 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c40722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 --2- 172.21.15.158:0/8569983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406ab40 0x7ff5c406d820 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 --2- 172.21.15.158:0/8569983 >> v2:172.21.15.172:3300/0 conn(0x7ff5c406e5d0 0x7ff5c4071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 -- 172.21.15.158:0/8569983 >> 172.21.15.158:0/8569983 conn(0x7ff5c40d02b0 msgr2=0x7ff5c40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 -- 172.21.15.158:0/8569983 shutdown_connections 2023-10-16T11:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.678+0000 7ff5caa69700 1 -- 172.21.15.158:0/8569983 wait complete. 2023-10-16T11:21:13.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.679+0000 7ff5caa69700 1 Processor -- start 2023-10-16T11:21:13.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.679+0000 7ff5caa69700 1 -- start start 2023-10-16T11:21:13.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5caa69700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c406ab40 0x7ff5c41507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5caa69700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c4155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5caa69700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406e5d0 0x7ff5c4155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5caa69700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4150d20 con 0x7ff5c406e5d0 2023-10-16T11:21:13.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5caa69700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4152df0 con 0x7ff5c406ab40 2023-10-16T11:21:13.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5caa69700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c41524c0 con 0x7ff5c406af20 2023-10-16T11:21:13.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5c9006700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406e5d0 0x7ff5c4155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5c9006700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406e5d0 0x7ff5c4155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42032/0 (socket says 172.21.15.158:42032) 2023-10-16T11:21:13.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.680+0000 7ff5c9006700 1 -- 172.21.15.158:0/2169636316 learned_addr learned my addr 172.21.15.158:0/2169636316 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:13.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c8805700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c406ab40 0x7ff5c41507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c3fff700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c4155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c9006700 1 -- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 msgr2=0x7ff5c4155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c9006700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c4155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c9006700 1 -- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c406ab40 msgr2=0x7ff5c41507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:13.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c9006700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c406ab40 0x7ff5c41507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:13.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c9006700 1 -- 172.21.15.158:0/2169636316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8009550 con 0x7ff5c406e5d0 2023-10-16T11:21:13.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c3fff700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c4155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:13.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c8805700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c406ab40 0x7ff5c41507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:13.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c9006700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406e5d0 0x7ff5c4155680 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009fa0 tx=0x7ff5bc00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c17fa700 1 -- 172.21.15.158:0/2169636316 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc003a20 con 0x7ff5c406e5d0 2023-10-16T11:21:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c17fa700 1 -- 172.21.15.158:0/2169636316 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc00f040 con 0x7ff5c406e5d0 2023-10-16T11:21:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c41527a0 con 0x7ff5c406e5d0 2023-10-16T11:21:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5c17fa700 1 -- 172.21.15.158:0/2169636316 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc003de0 con 0x7ff5c406e5d0 2023-10-16T11:21:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.681+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c41f6ec0 con 0x7ff5c406e5d0 2023-10-16T11:21:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.682+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c40681b0 con 0x7ff5c406e5d0 2023-10-16T11:21:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.683+0000 7ff5c17fa700 1 -- 172.21.15.158:0/2169636316 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff5bc004030 con 0x7ff5c406e5d0 2023-10-16T11:21:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.683+0000 7ff5c17fa700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff5a806b170 0x7ff5a806d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.683+0000 7ff5c17fa700 1 -- 172.21.15.158:0/2169636316 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7ff5bc08a060 con 0x7ff5c406e5d0 2023-10-16T11:21:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.683+0000 7ff5c8805700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff5a806b170 0x7ff5a806d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.683+0000 7ff5c8805700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff5a806b170 0x7ff5a806d630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009130 tx=0x7ff5b8008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.688+0000 7ff5c17fa700 1 -- 172.21.15.158:0/2169636316 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5bc0548c0 con 0x7ff5c406e5d0 2023-10-16T11:21:13.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:13.961+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7ff5c4152f30 con 0x7ff5c406e5d0 2023-10-16T11:21:14.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.813+0000 7ff5c17fa700 1 -- 172.21.15.158:0/2169636316 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3222) v1 ==== 158+0+0 (secure 0 0 0) 0x7ff5bc057ee0 con 0x7ff5c406e5d0 2023-10-16T11:21:14.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff5a806b170 msgr2=0x7ff5a806d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:14.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff5a806b170 0x7ff5a806d630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009130 tx=0x7ff5b8008030 comp rx=0 tx=0).stop 2023-10-16T11:21:14.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406e5d0 msgr2=0x7ff5c4155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:14.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406e5d0 0x7ff5c4155680 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009fa0 tx=0x7ff5bc00d460 comp rx=0 tx=0).stop 2023-10-16T11:21:14.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 shutdown_connections 2023-10-16T11:21:14.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff5a806b170 0x7ff5a806d630 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5c406e5d0 0x7ff5c4155680 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5c406ab40 0x7ff5c41507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 --2- 172.21.15.158:0/2169636316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5c406af20 0x7ff5c4155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 >> 172.21.15.158:0/2169636316 conn(0x7ff5c40d02b0 msgr2=0x7ff5c4072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:14.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 shutdown_connections 2023-10-16T11:21:14.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.816+0000 7ff5caa69700 1 -- 172.21.15.158:0/2169636316 wait complete. 2023-10-16T11:21:14.831 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T11:21:14.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.956+0000 7fa2038f7700 1 Processor -- start 2023-10-16T11:21:14.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.956+0000 7fa2038f7700 1 -- start start 2023-10-16T11:21:14.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa2038f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 0x7fa1fc0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa2038f7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1fc0cf4c0 con 0x7fa1fc0cbaf0 2023-10-16T11:21:14.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa2038f7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1fc15f970 con 0x7fa1fc15f1d0 2023-10-16T11:21:14.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa2038f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0b49e0 con 0x7fa1fc15b590 2023-10-16T11:21:14.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa201e94700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa1fc0cbaf0 0x7fa1fc0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42216/0 (socket says 172.21.15.158:42216) 2023-10-16T11:21:14.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa201e94700 1 -- 172.21.15.158:0/1805212560 learned_addr learned my addr 172.21.15.158:0/1805212560 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:14.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa201693700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 0x7fa1fc0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597919036 0 0) 0x7fa1fc0cf4c0 con 0x7fa1fc0cbaf0 2023-10-16T11:21:14.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e80035b0 con 0x7fa1fc0cbaf0 2023-10-16T11:21:14.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1941510385 0 0) 0x7fa1e80035b0 con 0x7fa1fc0cbaf0 2023-10-16T11:21:14.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 msgr2=0x7fa1fc0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:14.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 0x7fa1fc0ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 >> v1:172.21.15.172:6789/0 conn(0x7fa1fc15f1d0 legacy=0x7fa1fc0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:14.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc15fc50 con 0x7fa1fc0cbaf0 2023-10-16T11:21:14.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.957+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fa1f40019f0 con 0x7fa1fc0cbaf0 2023-10-16T11:21:14.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 >> v1:172.21.15.158:6789/0 conn(0x7fa1fc0cbaf0 legacy=0x7fa1fc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:14.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 0x7fa1e8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e8003f70 0x7fa1e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1e8004d90 0x7fa1e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0b49e0 con 0x7fa1e8004d90 2023-10-16T11:21:14.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e800bb20 con 0x7fa1e8003f70 2023-10-16T11:21:14.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e800bce0 con 0x7fa1fc15b590 2023-10-16T11:21:14.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201e94700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1e8004d90 0x7fa1e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201693700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 0x7fa1e8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa200e92700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e8003f70 0x7fa1e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201e94700 1 -- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 msgr2=0x7fa1e8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:14.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201e94700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc15b590 0x7fa1e8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201e94700 1 -- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e8003f70 msgr2=0x7fa1e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:14.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201e94700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e8003f70 0x7fa1e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201e94700 1 -- 172.21.15.158:0/1805212560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80035b0 con 0x7fa1e8004d90 2023-10-16T11:21:14.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa201e94700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1e8004d90 0x7fa1e8005210 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400a840 tx=0x7fa1e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0928e93e9664fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:14.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4012600 con 0x7fa1e8004d90 2023-10-16T11:21:14.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f4012760 con 0x7fa1e8004d90 2023-10-16T11:21:14.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.958+0000 7fa1f3fff700 1 -- 172.21.15.158:0/1805212560 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4012af0 con 0x7fa1e8004d90 2023-10-16T11:21:14.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.959+0000 7fa2038f7700 1 -- 172.21.15.158:0/1805212560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1e8004d90 msgr2=0x7fa1e8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:14.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.959+0000 7fa2038f7700 1 --2- 172.21.15.158:0/1805212560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1e8004d90 0x7fa1e8005210 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400a840 tx=0x7fa1e8005880 comp rx=0 tx=0).stop 2023-10-16T11:21:14.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.959+0000 7fa201e94700 1 -- 172.21.15.158:0/1805212560 reap_dead start 2023-10-16T11:21:14.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.959+0000 7fa2038f7700 1 -- 172.21.15.158:0/1805212560 shutdown_connections 2023-10-16T11:21:14.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.959+0000 7fa2038f7700 1 -- 172.21.15.158:0/1805212560 >> 172.21.15.158:0/1805212560 conn(0x7fa1fc06a8d0 msgr2=0x7fa1fc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:14.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.959+0000 7fa2038f7700 1 -- 172.21.15.158:0/1805212560 shutdown_connections 2023-10-16T11:21:14.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.959+0000 7fa2038f7700 1 -- 172.21.15.158:0/1805212560 wait complete. 2023-10-16T11:21:14.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.960+0000 7fa2038f7700 1 Processor -- start 2023-10-16T11:21:14.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.960+0000 7fa2038f7700 1 -- start start 2023-10-16T11:21:14.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa2038f7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc0c6ef0 0x7fa1fc1577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa2038f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc0c72d0 0x7fa1fc157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa2038f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc15c5a0 0x7fa1fc15ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa2038f7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc15ea50 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa2038f7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc155f00 con 0x7fa1fc0c6ef0 2023-10-16T11:21:14.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa2038f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc15cf90 con 0x7fa1fc0c72d0 2023-10-16T11:21:14.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc15c5a0 0x7fa1fc15ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc15c5a0 0x7fa1fc15ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42052/0 (socket says 172.21.15.158:42052) 2023-10-16T11:21:14.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 -- 172.21.15.158:0/2590835010 learned_addr learned my addr 172.21.15.158:0/2590835010 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:14.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa200e92700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc0c72d0 0x7fa1fc157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201693700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc0c6ef0 0x7fa1fc1577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 -- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc0c72d0 msgr2=0x7fa1fc157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:14.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc0c72d0 0x7fa1fc157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 -- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc0c6ef0 msgr2=0x7fa1fc1577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:14.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc0c6ef0 0x7fa1fc1577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:14.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 -- 172.21.15.158:0/2590835010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc15d200 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa201e94700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc15c5a0 0x7fa1fc15ca50 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400ae30 tx=0x7fa1fc0cf430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:14.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa1f1ffb700 1 -- 172.21.15.158:0/2590835010 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4012ce0 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.961+0000 7fa1f1ffb700 1 -- 172.21.15.158:0/2590835010 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f4012e40 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.962+0000 7fa1f1ffb700 1 -- 172.21.15.158:0/2590835010 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f401e630 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.962+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc15d4e0 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.962+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc155b90 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.963+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc1560b0 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.963+0000 7fa1f1ffb700 1 -- 172.21.15.158:0/2590835010 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa1f401e790 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.963+0000 7fa1f1ffb700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa1e006b140 0x7fa1e006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:14.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.964+0000 7fa1f1ffb700 1 -- 172.21.15.158:0/2590835010 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fa1f408baf0 con 0x7fa1fc15c5a0 2023-10-16T11:21:14.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.964+0000 7fa201693700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa1e006b140 0x7fa1e006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:14.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.964+0000 7fa201693700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa1e006b140 0x7fa1e006d600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc15e630 tx=0x7fa1f8009210 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:14.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:14.968+0000 7fa1f1ffb700 1 -- 172.21.15.158:0/2590835010 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1fc1560b0 con 0x7fa1fc15c5a0 2023-10-16T11:21:15.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.159+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa1fc15f610 con 0x7fa1fc15c5a0 2023-10-16T11:21:15.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.162+0000 7fa1f1ffb700 1 -- 172.21.15.158:0/2590835010 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa1f4056650 con 0x7fa1fc15c5a0 2023-10-16T11:21:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa1e006b140 msgr2=0x7fa1e006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa1e006b140 0x7fa1e006d600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc15e630 tx=0x7fa1f8009210 comp rx=0 tx=0).stop 2023-10-16T11:21:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc15c5a0 msgr2=0x7fa1fc15ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc15c5a0 0x7fa1fc15ca50 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400ae30 tx=0x7fa1fc0cf430 comp rx=0 tx=0).stop 2023-10-16T11:21:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 shutdown_connections 2023-10-16T11:21:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fa1e006b140 0x7fa1e006d600 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1fc15c5a0 0x7fa1fc15ca50 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1fc0c6ef0 0x7fa1fc1577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 --2- 172.21.15.158:0/2590835010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1fc0c72d0 0x7fa1fc157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 >> 172.21.15.158:0/2590835010 conn(0x7fa1fc06a8d0 msgr2=0x7fa1fc0b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 shutdown_connections 2023-10-16T11:21:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.165+0000 7fa2038f7700 1 -- 172.21.15.158:0/2590835010 wait complete. 2023-10-16T11:21:15.179 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:21:15.179 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:21:15.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:21:15.268 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:21:15.269 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:15.269 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:15.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:21:15.287 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:21:15.288 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:15.307 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:21:15.308 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:15.308 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:15.345 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:21:15.345 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:21:15.346 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:15.366 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:21:15.366 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T11:21:15.366 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:15.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.502+0000 7fda006c4700 1 Processor -- start 2023-10-16T11:21:15.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.502+0000 7fda006c4700 1 -- start start 2023-10-16T11:21:15.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.503+0000 7fda006c4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd9f80cf730 0x7fd9f81602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.503+0000 7fda006c4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f81613b0 con 0x7fd9f815d2f0 2023-10-16T11:21:15.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.503+0000 7fda006c4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f8161690 con 0x7fd9f815dde0 2023-10-16T11:21:15.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.503+0000 7fda006c4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f8161970 con 0x7fd9f80cf730 2023-10-16T11:21:15.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fe460700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd9f815d2f0 0x7fd9f815d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42232/0 (socket says 172.21.15.158:42232) 2023-10-16T11:21:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fe460700 1 -- 172.21.15.158:0/797410883 learned_addr learned my addr 172.21.15.158:0/797410883 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fec61700 1 --2- 172.21.15.158:0/797410883 >> v2:172.21.15.172:3300/0 conn(0x7fd9f80cf730 0x7fd9f81602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085723933 0 0) 0x7fd9f81613b0 con 0x7fd9f815d2f0 2023-10-16T11:21:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e40035b0 con 0x7fd9f815d2f0 2023-10-16T11:21:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3054855744 0 0) 0x7fd9e40035b0 con 0x7fd9f815d2f0 2023-10-16T11:21:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 >> v2:172.21.15.172:3300/0 conn(0x7fd9f80cf730 msgr2=0x7fd9f81602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 --2- 172.21.15.158:0/797410883 >> v2:172.21.15.172:3300/0 conn(0x7fd9f80cf730 0x7fd9f81602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 >> v1:172.21.15.172:6789/0 conn(0x7fd9f815dde0 legacy=0x7fd9f80cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f8161c50 con 0x7fd9f815d2f0 2023-10-16T11:21:15.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd9e8001670 con 0x7fd9f815d2f0 2023-10-16T11:21:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 >> v1:172.21.15.158:6789/0 conn(0x7fd9f815d2f0 legacy=0x7fd9f815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e4003a00 0x7fd9e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e4004850 0x7fd9e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f8161970 con 0x7fd9e4005330 2023-10-16T11:21:15.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e400c200 con 0x7fd9e4003a00 2023-10-16T11:21:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e400c3c0 con 0x7fd9e4004850 2023-10-16T11:21:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fec61700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.504+0000 7fd9fe460700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e4003a00 0x7fd9e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fdc5f700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e4004850 0x7fd9e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fec61700 1 -- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e4004850 msgr2=0x7fd9e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fec61700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e4004850 0x7fd9e4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fec61700 1 -- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e4003a00 msgr2=0x7fd9e4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fec61700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e4003a00 0x7fd9e4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fec61700 1 -- 172.21.15.158:0/797410883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40035b0 con 0x7fd9e4005330 2023-10-16T11:21:15.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fdc5f700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e4004850 0x7fd9e4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:15.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fe460700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e4003a00 0x7fd9e4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:15.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fec61700 1 -- 172.21.15.158:0/797410883 reap_dead start 2023-10-16T11:21:15.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fec61700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9e400ba10 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009f90 tx=0x7fd9e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a32eb00012ed2b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:15.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0003a20 con 0x7fd9e4005330 2023-10-16T11:21:15.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f000f040 con 0x7fd9e4005330 2023-10-16T11:21:15.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fd9fd45e700 1 -- 172.21.15.158:0/797410883 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0003dc0 con 0x7fd9e4005330 2023-10-16T11:21:15.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fda006c4700 1 -- 172.21.15.158:0/797410883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 msgr2=0x7fd9e400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fda006c4700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9e400ba10 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009f90 tx=0x7fd9e4005880 comp rx=0 tx=0).stop 2023-10-16T11:21:15.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fda006c4700 1 -- 172.21.15.158:0/797410883 shutdown_connections 2023-10-16T11:21:15.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fda006c4700 1 --2- 172.21.15.158:0/797410883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9e400ba10 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.505+0000 7fda006c4700 1 -- 172.21.15.158:0/797410883 >> 172.21.15.158:0/797410883 conn(0x7fd9f806a8d0 msgr2=0x7fd9f806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:15.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.506+0000 7fda006c4700 1 -- 172.21.15.158:0/797410883 shutdown_connections 2023-10-16T11:21:15.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.506+0000 7fda006c4700 1 -- 172.21.15.158:0/797410883 wait complete. 2023-10-16T11:21:15.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 Processor -- start 2023-10-16T11:21:15.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 -- start start 2023-10-16T11:21:15.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9f80c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f80c5fd0 0x7fd9f80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f80ad9b0 0x7fd9f80ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f80c64d0 con 0x7fd9e4005330 2023-10-16T11:21:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f80c67b0 con 0x7fd9f80ad9b0 2023-10-16T11:21:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.507+0000 7fda006c4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f80c6a90 con 0x7fd9f80c5fd0 2023-10-16T11:21:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9f80c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9f80c7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42070/0 (socket says 172.21.15.158:42070) 2023-10-16T11:21:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 -- 172.21.15.158:0/1416974789 learned_addr learned my addr 172.21.15.158:0/1416974789 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fdc5f700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f80c5fd0 0x7fd9f80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fec61700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f80ad9b0 0x7fd9f80ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 -- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f80c5fd0 msgr2=0x7fd9f80b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f80c5fd0 0x7fd9f80b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 -- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f80ad9b0 msgr2=0x7fd9f80ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f80ad9b0 0x7fd9f80ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 -- 172.21.15.158:0/1416974789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f80c6d70 con 0x7fd9e4005330 2023-10-16T11:21:15.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fdc5f700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f80c5fd0 0x7fd9f80b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9fe460700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9f80c7570 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800e760 tx=0x7fd9e800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9eeffd700 1 -- 172.21.15.158:0/1416974789 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8010510 con 0x7fd9e4005330 2023-10-16T11:21:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9eeffd700 1 -- 172.21.15.158:0/1416974789 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e8010670 con 0x7fd9e4005330 2023-10-16T11:21:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f80c7050 con 0x7fd9e4005330 2023-10-16T11:21:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fd9eeffd700 1 -- 172.21.15.158:0/1416974789 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80109e0 con 0x7fd9e4005330 2023-10-16T11:21:15.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.508+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f80b52d0 con 0x7fd9e4005330 2023-10-16T11:21:15.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.509+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f80c7270 con 0x7fd9e4005330 2023-10-16T11:21:15.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.510+0000 7fd9eeffd700 1 -- 172.21.15.158:0/1416974789 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd9e8010b40 con 0x7fd9e4005330 2023-10-16T11:21:15.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.510+0000 7fd9eeffd700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd9dc06b140 0x7fd9dc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.510+0000 7fd9eeffd700 1 -- 172.21.15.158:0/1416974789 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fd9e8089310 con 0x7fd9e4005330 2023-10-16T11:21:15.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.510+0000 7fd9fdc5f700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd9dc06b140 0x7fd9dc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.511+0000 7fd9fdc5f700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd9dc06b140 0x7fd9dc06d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4000f60 tx=0x7fd9f4009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:15.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.515+0000 7fd9eeffd700 1 -- 172.21.15.158:0/1416974789 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9e8053e90 con 0x7fd9e4005330 2023-10-16T11:21:15.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.750+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9f80b38c0 con 0x7fd9e4005330 2023-10-16T11:21:15.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.751+0000 7fd9eeffd700 1 -- 172.21.15.158:0/1416974789 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+3714 (secure 0 0 0) 0x7fd9e8019090 con 0x7fd9e4005330 2023-10-16T11:21:15.753 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:15.753 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":138,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/965686132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":965686132},{"type":"v1","addr":"172.21.15.158:6836","nonce":965686132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1496914785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1496914785},{"type":"v1","addr":"172.21.15.158:6837","nonce":1496914785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":39755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4215831623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4215831623},{"type":"v1","addr":"172.21.15.172:6834","nonce":4215831623}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":39794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/963699601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":963699601},{"type":"v1","addr":"172.21.15.172:6835","nonce":963699601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":138}],"filesystems":[]} 2023-10-16T11:21:15.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd9dc06b140 msgr2=0x7fd9dc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd9dc06b140 0x7fd9dc06d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4000f60 tx=0x7fd9f4009040 comp rx=0 tx=0).stop 2023-10-16T11:21:15.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 msgr2=0x7fd9f80c7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9f80c7570 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800e760 tx=0x7fd9e800ea70 comp rx=0 tx=0).stop 2023-10-16T11:21:15.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 shutdown_connections 2023-10-16T11:21:15.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd9dc06b140 0x7fd9dc06d600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f80ad9b0 0x7fd9f80ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9e4005330 0x7fd9f80c7570 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 --2- 172.21.15.158:0/1416974789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f80c5fd0 0x7fd9f80b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 >> 172.21.15.158:0/1416974789 conn(0x7fd9f806a8d0 msgr2=0x7fd9f806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:15.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 shutdown_connections 2023-10-16T11:21:15.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.753+0000 7fda006c4700 1 -- 172.21.15.158:0/1416974789 wait complete. 2023-10-16T11:21:15.758 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 138 2023-10-16T11:21:15.767 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T11:21:15.767 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T11:21:15.767 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:21:15.768 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:15.768 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T11:21:15.768 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T11:21:15.769 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:21:15.769 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:15.769 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T11:21:15.770 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T11:21:15.770 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:21:15.770 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:15.771 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T11:21:15.771 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T11:21:15.771 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:21:15.771 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:15.777 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:21:15.777 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T11:21:15.779 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:21:15.779 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T11:21:15.784 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:21:15.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T11:21:15.785 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:21:15.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T11:21:15.820 INFO:tasks.ceph.mds.a:Started 2023-10-16T11:21:15.821 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:15.822 INFO:tasks.ceph.mds.c:Started 2023-10-16T11:21:15.822 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:15.827 INFO:tasks.ceph.mds.b:Started 2023-10-16T11:21:15.827 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:15.828 INFO:tasks.ceph.mds.d:Started 2023-10-16T11:21:15.828 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:15.829 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T11:21:15.900 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T11:21:15.901 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:21:15.913 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:21:15.914 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:21:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.956+0000 7f37557cc700 1 Processor -- start 2023-10-16T11:21:15.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.956+0000 7f37557cc700 1 -- start start 2023-10-16T11:21:15.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.956+0000 7f37557cc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f375015b670 0x7f375015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.956+0000 7f37557cc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500b1440 0x7f375015bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.956+0000 7f37557cc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3750070870 con 0x7f37500cb710 2023-10-16T11:21:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f37557cc700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375015ba50 con 0x7f375015b670 2023-10-16T11:21:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f37557cc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3750070b10 con 0x7f37500b1440 2023-10-16T11:21:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f375015b670 0x7f375015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f375015b670 0x7f375015f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42100/0 (socket says 172.21.15.158:42100) 2023-10-16T11:21:15.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 -- 172.21.15.158:0/1705919086 learned_addr learned my addr 172.21.15.158:0/1705919086 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:15.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374f7fe700 1 --2- 172.21.15.158:0/1705919086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500b1440 0x7f375015bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 -- 172.21.15.158:0/1705919086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500b1440 msgr2=0x7f375015bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:15.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 --2- 172.21.15.158:0/1705919086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500b1440 0x7f375015bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 -- 172.21.15.158:0/1705919086 >> v1:172.21.15.172:6789/0 conn(0x7f37500cb710 legacy=0x7f37500b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 -- 172.21.15.158:0/1705919086 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750070df0 con 0x7f375015b670 2023-10-16T11:21:15.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374effd700 1 --2- 172.21.15.158:0/1705919086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f375015b670 0x7f375015f1c0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f37400098f0 tx=0x7f3740009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e578b8ad1b69a686 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:15.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374dffb700 1 -- 172.21.15.158:0/1705919086 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740011830 con 0x7f375015b670 2023-10-16T11:21:15.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374dffb700 1 -- 172.21.15.158:0/1705919086 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3740011990 con 0x7f375015b670 2023-10-16T11:21:15.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.957+0000 7f374dffb700 1 -- 172.21.15.158:0/1705919086 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740011d20 con 0x7f375015b670 2023-10-16T11:21:15.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 -- 172.21.15.158:0/1705919086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f375015b670 msgr2=0x7f375015f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 --2- 172.21.15.158:0/1705919086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f375015b670 0x7f375015f1c0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f37400098f0 tx=0x7f3740009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:15.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 -- 172.21.15.158:0/1705919086 shutdown_connections 2023-10-16T11:21:15.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 --2- 172.21.15.158:0/1705919086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500b1440 0x7f375015bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 --2- 172.21.15.158:0/1705919086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f375015b670 0x7f375015f1c0 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 -- 172.21.15.158:0/1705919086 >> 172.21.15.158:0/1705919086 conn(0x7f37500cbd30 msgr2=0x7f37500cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:15.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 -- 172.21.15.158:0/1705919086 shutdown_connections 2023-10-16T11:21:15.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.958+0000 7f37557cc700 1 -- 172.21.15.158:0/1705919086 wait complete. 2023-10-16T11:21:15.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.959+0000 7f37557cc700 1 Processor -- start 2023-10-16T11:21:15.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.959+0000 7f37557cc700 1 -- start start 2023-10-16T11:21:15.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f37557cc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37500b1440 0x7f37501566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f37557cc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500cb710 0x7f37500cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f37557cc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375015b670 0x7f37500cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f37557cc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37501e56e0 con 0x7f37500b1440 2023-10-16T11:21:15.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f37557cc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37501e4db0 con 0x7f375015b670 2023-10-16T11:21:15.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f37557cc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37501e5090 con 0x7f37500cb710 2023-10-16T11:21:15.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37500b1440 0x7f37501566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37500b1440 0x7f37501566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42104/0 (socket says 172.21.15.158:42104) 2023-10-16T11:21:15.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 -- 172.21.15.158:0/2658952836 learned_addr learned my addr 172.21.15.158:0/2658952836 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:15.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374e7fc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500cb710 0x7f37500cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374f7fe700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375015b670 0x7f37500cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 -- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500cb710 msgr2=0x7f37500cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500cb710 0x7f37500cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 -- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375015b670 msgr2=0x7f37500cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:15.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375015b670 0x7f37500cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:15.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.960+0000 7f374effd700 1 -- 172.21.15.158:0/2658952836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3740009550 con 0x7f37500b1440 2023-10-16T11:21:15.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.961+0000 7f374effd700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37500b1440 0x7f37501566c0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f3740009c40 tx=0x7f3740004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:15.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.961+0000 7f37377fe700 1 -- 172.21.15.158:0/2658952836 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740004510 con 0x7f37500b1440 2023-10-16T11:21:15.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.961+0000 7f37377fe700 1 -- 172.21.15.158:0/2658952836 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374001e070 con 0x7f37500b1440 2023-10-16T11:21:15.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.961+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37501e5370 con 0x7f37500b1440 2023-10-16T11:21:15.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.961+0000 7f37377fe700 1 -- 172.21.15.158:0/2658952836 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37400038e0 con 0x7f37500b1440 2023-10-16T11:21:15.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.961+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3750156c00 con 0x7f37500b1440 2023-10-16T11:21:15.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.962+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37500681b0 con 0x7f37500b1440 2023-10-16T11:21:15.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.962+0000 7f37377fe700 1 -- 172.21.15.158:0/2658952836 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3740003a40 con 0x7f37500b1440 2023-10-16T11:21:15.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.963+0000 7f37377fe700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f373c06b140 0x7f373c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:15.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.963+0000 7f37377fe700 1 -- 172.21.15.158:0/2658952836 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f374008ab70 con 0x7f37500b1440 2023-10-16T11:21:15.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.963+0000 7f374e7fc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f373c06b140 0x7f373c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:15.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.963+0000 7f374e7fc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f373c06b140 0x7f373c06d600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37501f6700 tx=0x7f3744009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:15.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:15.967+0000 7f37377fe700 1 -- 172.21.15.158:0/2658952836 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f37400555c0 con 0x7f37500b1440 2023-10-16T11:21:16.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.175+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f375015ed20 con 0x7f37500b1440 2023-10-16T11:21:16.799 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:21:16.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:21:16.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.825+0000 7f37377fe700 1 -- 172.21.15.158:0/2658952836 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v251) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3740058be0 con 0x7f37500b1440 2023-10-16T11:21:16.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f373c06b140 msgr2=0x7f373c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:16.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f373c06b140 0x7f373c06d600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37501f6700 tx=0x7f3744009040 comp rx=0 tx=0).stop 2023-10-16T11:21:16.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37500b1440 msgr2=0x7f37501566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:16.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37500b1440 0x7f37501566c0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f3740009c40 tx=0x7f3740004750 comp rx=0 tx=0).stop 2023-10-16T11:21:16.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 shutdown_connections 2023-10-16T11:21:16.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f373c06b140 0x7f373c06d600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:16.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375015b670 0x7f37500cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:16.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37500b1440 0x7f37501566c0 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:16.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 --2- 172.21.15.158:0/2658952836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37500cb710 0x7f37500cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:16.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 >> 172.21.15.158:0/2658952836 conn(0x7f37500cbd30 msgr2=0x7f37500c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:16.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 shutdown_connections 2023-10-16T11:21:16.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:16.829+0000 7f37557cc700 1 -- 172.21.15.158:0/2658952836 wait complete. 2023-10-16T11:21:16.833 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T11:21:16.847 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T11:21:17.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.031+0000 7f91d7397700 1 Processor -- start 2023-10-16T11:21:17.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.031+0000 7f91d7397700 1 -- start start 2023-10-16T11:21:17.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d7397700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91d015b730 0x7f91d00ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d7397700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00ca780 con 0x7f91d015b350 2023-10-16T11:21:17.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d7397700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d006aa90 con 0x7f91d00ce630 2023-10-16T11:21:17.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d7397700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d00cf820 con 0x7f91d015b730 2023-10-16T11:21:17.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d4932700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91d015b730 0x7f91d00ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d5133700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f91d015b350 0x7f91d00cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42290/0 (socket says 172.21.15.158:42290) 2023-10-16T11:21:17.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d4932700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91d015b730 0x7f91d00ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42116/0 (socket says 172.21.15.158:42116) 2023-10-16T11:21:17.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.032+0000 7f91d5133700 1 -- 172.21.15.158:0/434926460 learned_addr learned my addr 172.21.15.158:0/434926460 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:17.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d4932700 1 -- 172.21.15.158:0/434926460 >> v1:172.21.15.172:6789/0 conn(0x7f91d00ce630 legacy=0x7f91d006af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d4932700 1 -- 172.21.15.158:0/434926460 >> v1:172.21.15.158:6789/0 conn(0x7f91d015b350 legacy=0x7f91d00cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d4932700 1 -- 172.21.15.158:0/434926460 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00cf260 con 0x7f91d015b730 2023-10-16T11:21:17.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d4932700 1 --2- 172.21.15.158:0/434926460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d015b730 0x7f91d00ca080 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009840 tx=0x7f91cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b69a11e19fb9c80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91c7fff700 1 -- 172.21.15.158:0/434926460 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc011830 con 0x7f91d015b730 2023-10-16T11:21:17.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91c7fff700 1 -- 172.21.15.158:0/434926460 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc011990 con 0x7f91d015b730 2023-10-16T11:21:17.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91c7fff700 1 -- 172.21.15.158:0/434926460 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc011d20 con 0x7f91d015b730 2023-10-16T11:21:17.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d7397700 1 -- 172.21.15.158:0/434926460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d015b730 msgr2=0x7f91d00ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d7397700 1 --2- 172.21.15.158:0/434926460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d015b730 0x7f91d00ca080 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009840 tx=0x7f91cc009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:17.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d7397700 1 -- 172.21.15.158:0/434926460 shutdown_connections 2023-10-16T11:21:17.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d7397700 1 --2- 172.21.15.158:0/434926460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d015b730 0x7f91d00ca080 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.033+0000 7f91d7397700 1 -- 172.21.15.158:0/434926460 >> 172.21.15.158:0/434926460 conn(0x7f91d00cc4b0 msgr2=0x7f91d00cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:17.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.034+0000 7f91d7397700 1 -- 172.21.15.158:0/434926460 shutdown_connections 2023-10-16T11:21:17.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.034+0000 7f91d7397700 1 -- 172.21.15.158:0/434926460 wait complete. 2023-10-16T11:21:17.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.035+0000 7f91d7397700 1 Processor -- start 2023-10-16T11:21:17.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.035+0000 7f91d7397700 1 -- start start 2023-10-16T11:21:17.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.035+0000 7f91d7397700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d00ce630 0x7f91d013def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.035+0000 7f91d7397700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91d015b350 0x7f91d00b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.035+0000 7f91d7397700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91d015b730 0x7f91d0159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d7397700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00b3940 con 0x7f91d00ce630 2023-10-16T11:21:17.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d7397700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00b3c20 con 0x7f91d015b730 2023-10-16T11:21:17.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d7397700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00b3f00 con 0x7f91d015b350 2023-10-16T11:21:17.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d00ce630 0x7f91d013def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d00ce630 0x7f91d013def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42120/0 (socket says 172.21.15.158:42120) 2023-10-16T11:21:17.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 -- 172.21.15.158:0/3815539824 learned_addr learned my addr 172.21.15.158:0/3815539824 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:17.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d4932700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91d015b350 0x7f91d00b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5934700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91d015b730 0x7f91d0159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 -- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91d015b350 msgr2=0x7f91d00b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91d015b350 0x7f91d00b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 -- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91d015b730 msgr2=0x7f91d0159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91d015b730 0x7f91d0159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 -- 172.21.15.158:0/3815539824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91cc0094a0 con 0x7f91d00ce630 2023-10-16T11:21:17.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d5133700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d00ce630 0x7f91d013def0 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f91c000c520 tx=0x7f91c000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91c5ffb700 1 -- 172.21.15.158:0/3815539824 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0005d70 con 0x7f91d00ce630 2023-10-16T11:21:17.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91c5ffb700 1 -- 172.21.15.158:0/3815539824 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c00070d0 con 0x7f91d00ce630 2023-10-16T11:21:17.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d00b41e0 con 0x7f91d00ce630 2023-10-16T11:21:17.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.036+0000 7f91c5ffb700 1 -- 172.21.15.158:0/3815539824 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0007440 con 0x7f91d00ce630 2023-10-16T11:21:17.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.037+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d0141f00 con 0x7f91d00ce630 2023-10-16T11:21:17.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.038+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d00681b0 con 0x7f91d00ce630 2023-10-16T11:21:17.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.038+0000 7f91c5ffb700 1 -- 172.21.15.158:0/3815539824 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f91c00075a0 con 0x7f91d00ce630 2023-10-16T11:21:17.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.038+0000 7f91c5ffb700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f91b406b140 0x7f91b406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.038+0000 7f91c5ffb700 1 -- 172.21.15.158:0/3815539824 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f91c0088d70 con 0x7f91d00ce630 2023-10-16T11:21:17.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.038+0000 7f91d4932700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f91b406b140 0x7f91b406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.039+0000 7f91d4932700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f91b406b140 0x7f91b406d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009840 tx=0x7f91cc008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.043+0000 7f91c5ffb700 1 -- 172.21.15.158:0/3815539824 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91c0053a90 con 0x7f91d00ce630 2023-10-16T11:21:17.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.312+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f91d00cd820 con 0x7f91d00ce630 2023-10-16T11:21:17.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.313+0000 7f91c5ffb700 1 -- 172.21.15.158:0/3815539824 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v55) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f91c00570b0 con 0x7f91d00ce630 2023-10-16T11:21:17.315 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:17.315 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:21:17.315 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T11:21:17.315 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.316 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T11:21:17.316 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T11:21:17.316 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.316 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.316 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:17.317 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:17.317 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.317 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.317 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.317 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.317 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T11:21:17.318 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T11:21:17.318 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.318 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.318 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:17.318 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:17.319 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.319 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.319 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.319 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.319 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T11:21:17.319 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T11:21:17.320 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.320 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.320 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:17.320 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:17.320 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.321 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.321 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.321 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.321 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T11:21:17.321 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T11:21:17.322 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.322 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.322 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:17.322 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:17.322 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.322 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.323 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.323 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.323 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T11:21:17.323 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T11:21:17.323 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.324 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.324 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.324 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.324 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.324 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.324 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.325 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T11:21:17.325 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T11:21:17.325 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.325 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.325 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.326 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.326 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.326 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.326 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.326 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T11:21:17.327 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T11:21:17.327 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.327 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.327 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.327 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.328 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.328 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.328 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.328 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T11:21:17.328 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T11:21:17.329 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.329 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.329 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.329 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.329 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.330 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.330 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.330 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T11:21:17.330 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T11:21:17.330 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.331 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.331 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.331 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.331 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.331 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.331 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.332 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T11:21:17.332 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T11:21:17.332 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.332 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.332 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.333 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.333 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.333 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.333 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.333 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T11:21:17.334 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T11:21:17.334 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.334 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.334 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.334 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.335 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.335 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.337 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.338 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T11:21:17.338 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T11:21:17.338 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.338 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:17.338 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:17.339 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.339 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.339 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.339 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.339 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T11:21:17.340 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T11:21:17.340 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.340 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.340 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T11:21:17.340 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T11:21:17.341 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.341 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.341 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.341 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T11:21:17.341 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T11:21:17.342 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.342 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.342 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:21:17.342 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:21:17.342 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:21:17.342 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.343 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.343 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.343 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T11:21:17.343 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T11:21:17.343 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.344 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.344 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:21:17.344 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:21:17.344 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:21:17.344 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.345 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.345 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.345 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T11:21:17.345 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T11:21:17.345 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.345 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:17.346 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:21:17.346 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:21:17.346 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:21:17.346 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.346 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.347 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.347 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T11:21:17.347 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T11:21:17.347 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.347 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:21:17.347 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:17.348 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:17.348 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.348 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.348 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.348 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.348 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T11:21:17.349 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T11:21:17.349 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.349 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:21:17.349 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:21:17.349 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.349 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.350 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:17.350 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:17.350 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T11:21:17.350 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T11:21:17.350 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:17.351 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:21:17.351 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:21:17.351 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:17.351 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.351 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:17.351 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T11:21:17.352 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:21:17.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f91b406b140 msgr2=0x7f91b406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f91b406b140 0x7f91b406d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009840 tx=0x7f91cc008030 comp rx=0 tx=0).stop 2023-10-16T11:21:17.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d00ce630 msgr2=0x7f91d013def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d00ce630 0x7f91d013def0 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f91c000c520 tx=0x7f91c000c830 comp rx=0 tx=0).stop 2023-10-16T11:21:17.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 shutdown_connections 2023-10-16T11:21:17.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f91b406b140 0x7f91b406d600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91d015b730 0x7f91d0159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91d00ce630 0x7f91d013def0 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 --2- 172.21.15.158:0/3815539824 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f91d015b350 0x7f91d00b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 >> 172.21.15.158:0/3815539824 conn(0x7f91d00cc4b0 msgr2=0x7f91d006d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:17.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 shutdown_connections 2023-10-16T11:21:17.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.315+0000 7f91d7397700 1 -- 172.21.15.158:0/3815539824 wait complete. 2023-10-16T11:21:17.356 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:17.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.475+0000 7f1c0abb0700 1 Processor -- start 2023-10-16T11:21:17.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.476+0000 7f1c0abb0700 1 -- start start 2023-10-16T11:21:17.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.476+0000 7f1c0abb0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c0406e5d0 0x7f1c0406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.476+0000 7f1c0abb0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c04071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.476+0000 7f1c0abb0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c04071e50 con 0x7f1c0406ab40 2023-10-16T11:21:17.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.476+0000 7f1c0abb0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c04161270 con 0x7f1c0406e5d0 2023-10-16T11:21:17.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.476+0000 7f1c0abb0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04161510 con 0x7f1c040729b0 2023-10-16T11:21:17.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0914d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c03fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c0406e5d0 0x7f1c0406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0894c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1c0406ab40 0x7f1c040722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:46208/0 (socket says 172.21.15.158:46208) 2023-10-16T11:21:17.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0894c700 1 -- 172.21.15.158:0/2962802817 learned_addr learned my addr 172.21.15.158:0/2962802817 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:17.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c037fe700 1 -- 172.21.15.158:0/2962802817 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570503467 0 0) 0x7f1c04071e50 con 0x7f1c0406ab40 2023-10-16T11:21:17.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c037fe700 1 -- 172.21.15.158:0/2962802817 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf40035b0 con 0x7f1c0406ab40 2023-10-16T11:21:17.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0914d700 1 -- 172.21.15.158:0/2962802817 >> v2:172.21.15.172:3300/0 conn(0x7f1c0406e5d0 msgr2=0x7f1c0406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:17.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0914d700 1 --2- 172.21.15.158:0/2962802817 >> v2:172.21.15.172:3300/0 conn(0x7f1c0406e5d0 0x7f1c0406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0914d700 1 -- 172.21.15.158:0/2962802817 >> v1:172.21.15.172:6789/0 conn(0x7f1c0406ab40 legacy=0x7f1c040722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0914d700 1 -- 172.21.15.158:0/2962802817 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c041617f0 con 0x7f1c040729b0 2023-10-16T11:21:17.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c0914d700 1 --2- 172.21.15.158:0/2962802817 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c04071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009d90 tx=0x7f1bfc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96a892844fb68de2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.477+0000 7f1c037fe700 1 -- 172.21.15.158:0/2962802817 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc00e070 con 0x7f1c040729b0 2023-10-16T11:21:17.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c037fe700 1 -- 172.21.15.158:0/2962802817 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc00bb20 con 0x7f1c040729b0 2023-10-16T11:21:17.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c037fe700 1 -- 172.21.15.158:0/2962802817 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc00be90 con 0x7f1c040729b0 2023-10-16T11:21:17.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 -- 172.21.15.158:0/2962802817 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 msgr2=0x7f1c04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/2962802817 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c04071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009d90 tx=0x7f1bfc00b330 comp rx=0 tx=0).stop 2023-10-16T11:21:17.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 -- 172.21.15.158:0/2962802817 shutdown_connections 2023-10-16T11:21:17.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/2962802817 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c04071910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/2962802817 >> v2:172.21.15.172:3300/0 conn(0x7f1c0406e5d0 0x7f1c0406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 -- 172.21.15.158:0/2962802817 >> 172.21.15.158:0/2962802817 conn(0x7f1c040d02b0 msgr2=0x7f1c040d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:17.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 -- 172.21.15.158:0/2962802817 shutdown_connections 2023-10-16T11:21:17.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.478+0000 7f1c0abb0700 1 -- 172.21.15.158:0/2962802817 wait complete. 2023-10-16T11:21:17.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.479+0000 7f1c0abb0700 1 Processor -- start 2023-10-16T11:21:17.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.479+0000 7f1c0abb0700 1 -- start start 2023-10-16T11:21:17.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0abb0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c0406ab40 0x7f1c040c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0abb0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c0406e5d0 0x7f1c04152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0abb0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c040b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0abb0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0415de20 con 0x7f1c0406ab40 2023-10-16T11:21:17.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0abb0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040b20e0 con 0x7f1c0406e5d0 2023-10-16T11:21:17.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0abb0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040b1560 con 0x7f1c040729b0 2023-10-16T11:21:17.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0894c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c0406ab40 0x7f1c040c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0894c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c0406ab40 0x7f1c040c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42134/0 (socket says 172.21.15.158:42134) 2023-10-16T11:21:17.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.480+0000 7f1c0894c700 1 -- 172.21.15.158:0/3925554004 learned_addr learned my addr 172.21.15.158:0/3925554004 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:17.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c03fff700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c0406e5d0 0x7f1c04152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0914d700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c040b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0894c700 1 -- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 msgr2=0x7f1c040b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0894c700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c040b0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0894c700 1 -- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c0406e5d0 msgr2=0x7f1c04152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0894c700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c0406e5d0 0x7f1c04152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0894c700 1 -- 172.21.15.158:0/3925554004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc009a70 con 0x7f1c0406ab40 2023-10-16T11:21:17.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0894c700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c0406ab40 0x7f1c040c9080 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800c4d0 tx=0x7f1bf800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c017fa700 1 -- 172.21.15.158:0/3925554004 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8005d70 con 0x7f1c0406ab40 2023-10-16T11:21:17.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c017fa700 1 -- 172.21.15.158:0/3925554004 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf80070d0 con 0x7f1c0406ab40 2023-10-16T11:21:17.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c017fa700 1 -- 172.21.15.158:0/3925554004 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8007440 con 0x7f1c0406ab40 2023-10-16T11:21:17.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c040b1840 con 0x7f1c0406ab40 2023-10-16T11:21:17.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.481+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c040b1e00 con 0x7f1c0406ab40 2023-10-16T11:21:17.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.482+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0415f880 con 0x7f1c0406ab40 2023-10-16T11:21:17.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.482+0000 7f1c017fa700 1 -- 172.21.15.158:0/3925554004 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1bf80075a0 con 0x7f1c0406ab40 2023-10-16T11:21:17.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.483+0000 7f1c017fa700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f1be806b140 0x7f1be806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.483+0000 7f1c03fff700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f1be806b140 0x7f1be806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.483+0000 7f1c017fa700 1 -- 172.21.15.158:0/3925554004 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f1bf8088de0 con 0x7f1c0406ab40 2023-10-16T11:21:17.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.483+0000 7f1c03fff700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f1be806b140 0x7f1be806d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c040b2440 tx=0x7f1bf0008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.487+0000 7f1c017fa700 1 -- 172.21.15.158:0/3925554004 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c0415f880 con 0x7f1c0406ab40 2023-10-16T11:21:17.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.681+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c0415f880 con 0x7f1c0406ab40 2023-10-16T11:21:17.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.682+0000 7f1c017fa700 1 -- 172.21.15.158:0/3925554004 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+13819 (secure 0 0 0) 0x7f1c0415f880 con 0x7f1c0406ab40 2023-10-16T11:21:17.683 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:17.684 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":253,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:16.834186+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:17.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f1be806b140 msgr2=0x7f1be806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f1be806b140 0x7f1be806d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c040b2440 tx=0x7f1bf0008040 comp rx=0 tx=0).stop 2023-10-16T11:21:17.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c0406ab40 msgr2=0x7f1c040c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c0406ab40 0x7f1c040c9080 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800c4d0 tx=0x7f1bf800c890 comp rx=0 tx=0).stop 2023-10-16T11:21:17.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 shutdown_connections 2023-10-16T11:21:17.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f1be806b140 0x7f1be806d600 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c040729b0 0x7f1c040b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c0406ab40 0x7f1c040c9080 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 --2- 172.21.15.158:0/3925554004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c0406e5d0 0x7f1c04152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 >> 172.21.15.158:0/3925554004 conn(0x7f1c040d02b0 msgr2=0x7f1c04072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:17.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 shutdown_connections 2023-10-16T11:21:17.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.684+0000 7f1c0abb0700 1 -- 172.21.15.158:0/3925554004 wait complete. 2023-10-16T11:21:17.698 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:21:17.699 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:21:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.820+0000 7fdc242e3700 1 Processor -- start 2023-10-16T11:21:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.820+0000 7fdc242e3700 1 -- start start 2023-10-16T11:21:17.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.821+0000 7fdc242e3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc1c0d0c60 0x7fdc1c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.821+0000 7fdc242e3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdc1c0b3460 0x7fdc1c0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.821+0000 7fdc242e3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c0b4420 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.821+0000 7fdc242e3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c15ba90 con 0x7fdc1c0d0c60 2023-10-16T11:21:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.821+0000 7fdc242e3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b38d0 con 0x7fdc1c0b3460 2023-10-16T11:21:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc1c0d0c60 0x7fdc1c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc1c0d0c60 0x7fdc1c0b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42144/0 (socket says 172.21.15.158:42144) 2023-10-16T11:21:17.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 -- 172.21.15.158:0/2315709704 learned_addr learned my addr 172.21.15.158:0/2315709704 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:17.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc22880700 1 --2- 172.21.15.158:0/2315709704 >> v2:172.21.15.172:3300/0 conn(0x7fdc1c0b3460 0x7fdc1c0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 -- 172.21.15.158:0/2315709704 >> v2:172.21.15.172:3300/0 conn(0x7fdc1c0b3460 msgr2=0x7fdc1c0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:17.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 --2- 172.21.15.158:0/2315709704 >> v2:172.21.15.172:3300/0 conn(0x7fdc1c0b3460 0x7fdc1c0b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 -- 172.21.15.158:0/2315709704 >> v1:172.21.15.172:6789/0 conn(0x7fdc1c15b6b0 legacy=0x7fdc1c0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 -- 172.21.15.158:0/2315709704 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0cbc20 con 0x7fdc1c0d0c60 2023-10-16T11:21:17.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.822+0000 7fdc2187e700 1 --2- 172.21.15.158:0/2315709704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c0d0c60 0x7fdc1c0b2f20 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0b0cb0 tx=0x7fdc0c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf9af1e44c9aa1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.824+0000 7fdc2107d700 1 -- 172.21.15.158:0/2315709704 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c011830 con 0x7fdc1c0d0c60 2023-10-16T11:21:17.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.825+0000 7fdc2107d700 1 -- 172.21.15.158:0/2315709704 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c011990 con 0x7fdc1c0d0c60 2023-10-16T11:21:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.825+0000 7fdc2107d700 1 -- 172.21.15.158:0/2315709704 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c011d20 con 0x7fdc1c0d0c60 2023-10-16T11:21:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 -- 172.21.15.158:0/2315709704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c0d0c60 msgr2=0x7fdc1c0b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 --2- 172.21.15.158:0/2315709704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c0d0c60 0x7fdc1c0b2f20 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0b0cb0 tx=0x7fdc0c009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:17.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 -- 172.21.15.158:0/2315709704 shutdown_connections 2023-10-16T11:21:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 --2- 172.21.15.158:0/2315709704 >> v2:172.21.15.172:3300/0 conn(0x7fdc1c0b3460 0x7fdc1c0b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 --2- 172.21.15.158:0/2315709704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c0d0c60 0x7fdc1c0b2f20 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 -- 172.21.15.158:0/2315709704 >> 172.21.15.158:0/2315709704 conn(0x7fdc1c0cdf20 msgr2=0x7fdc1c0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 -- 172.21.15.158:0/2315709704 shutdown_connections 2023-10-16T11:21:17.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.826+0000 7fdc242e3700 1 -- 172.21.15.158:0/2315709704 wait complete. 2023-10-16T11:21:17.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.827+0000 7fdc242e3700 1 Processor -- start 2023-10-16T11:21:17.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.827+0000 7fdc242e3700 1 -- start start 2023-10-16T11:21:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc242e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc1c0b3460 0x7fdc1c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc242e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc1c0d0c60 0x7fdc1c1f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc242e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c15b6b0 0x7fdc1c0c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc242e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c06c960 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc242e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c1f4040 con 0x7fdc1c0d0c60 2023-10-16T11:21:17.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc242e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c1f3d10 con 0x7fdc1c0b3460 2023-10-16T11:21:17.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc22880700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c15b6b0 0x7fdc1c0c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc2207f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc1c0b3460 0x7fdc1c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc22880700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c15b6b0 0x7fdc1c0c7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42152/0 (socket says 172.21.15.158:42152) 2023-10-16T11:21:17.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.828+0000 7fdc22880700 1 -- 172.21.15.158:0/3118207865 learned_addr learned my addr 172.21.15.158:0/3118207865 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:17.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc2187e700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc1c0d0c60 0x7fdc1c1f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc22880700 1 -- 172.21.15.158:0/3118207865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc1c0b3460 msgr2=0x7fdc1c06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc22880700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc1c0b3460 0x7fdc1c06b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc22880700 1 -- 172.21.15.158:0/3118207865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc1c0d0c60 msgr2=0x7fdc1c1f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:17.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc22880700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc1c0d0c60 0x7fdc1c1f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:17.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc22880700 1 -- 172.21.15.158:0/3118207865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c009550 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc22880700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c15b6b0 0x7fdc1c0c7c10 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009d30 tx=0x7fdc1400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc12ffd700 1 -- 172.21.15.158:0/3118207865 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1400cc90 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc12ffd700 1 -- 172.21.15.158:0/3118207865 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1400f040 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c06c0c0 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc12ffd700 1 -- 172.21.15.158:0/3118207865 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14009e80 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.829+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c06c680 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.830+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1c0681b0 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.830+0000 7fdc12ffd700 1 -- 172.21.15.158:0/3118207865 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdc14014480 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.831+0000 7fdc12ffd700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdc0006b140 0x7fdc0006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:17.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.831+0000 7fdc12ffd700 1 -- 172.21.15.158:0/3118207865 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 5657+0+0 (secure 0 0 0) 0x7fdc1408a330 con 0x7fdc1c15b6b0 2023-10-16T11:21:17.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.831+0000 7fdc2207f700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdc0006b140 0x7fdc0006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:17.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.831+0000 7fdc2207f700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdc0006b140 0x7fdc0006d600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc180018a0 tx=0x7fdc18009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:17.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:17.835+0000 7fdc12ffd700 1 -- 172.21.15.158:0/3118207865 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc14054f20 con 0x7fdc1c15b6b0 2023-10-16T11:21:18.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.026+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc1c1f4180 con 0x7fdc1c15b6b0 2023-10-16T11:21:18.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.027+0000 7fdc12ffd700 1 -- 172.21.15.158:0/3118207865 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v253) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc14058540 con 0x7fdc1c15b6b0 2023-10-16T11:21:18.029 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:21:18.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdc0006b140 msgr2=0x7fdc0006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdc0006b140 0x7fdc0006d600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc180018a0 tx=0x7fdc18009040 comp rx=0 tx=0).stop 2023-10-16T11:21:18.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c15b6b0 msgr2=0x7fdc1c0c7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c15b6b0 0x7fdc1c0c7c10 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009d30 tx=0x7fdc1400c3a0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 shutdown_connections 2023-10-16T11:21:18.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdc0006b140 0x7fdc0006d600 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c15b6b0 0x7fdc1c0c7c10 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc1c0b3460 0x7fdc1c06b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 --2- 172.21.15.158:0/3118207865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc1c0d0c60 0x7fdc1c1f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 >> 172.21.15.158:0/3118207865 conn(0x7fdc1c0cdf20 msgr2=0x7fdc1c15e370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:18.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 shutdown_connections 2023-10-16T11:21:18.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.029+0000 7fdc242e3700 1 -- 172.21.15.158:0/3118207865 wait complete. 2023-10-16T11:21:18.043 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:21:18.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.162+0000 7facf391b700 1 Processor -- start 2023-10-16T11:21:18.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.162+0000 7facf391b700 1 -- start start 2023-10-16T11:21:18.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf391b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7facec1524e0 0x7facec0cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf391b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec0cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf391b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec0cfe80 con 0x7facec0d0310 2023-10-16T11:21:18.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf391b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec0cf270 con 0x7facec1524e0 2023-10-16T11:21:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf391b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0cf550 con 0x7facec15c530 2023-10-16T11:21:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7facec1524e0 0x7facec0cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7facec1524e0 0x7facec0cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42158/0 (socket says 172.21.15.158:42158) 2023-10-16T11:21:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 -- 172.21.15.158:0/1573472386 learned_addr learned my addr 172.21.15.158:0/1573472386 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:18.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf1eb8700 1 --2- 172.21.15.158:0/1573472386 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec0cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 -- 172.21.15.158:0/1573472386 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 msgr2=0x7facec0cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 --2- 172.21.15.158:0/1573472386 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec0cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 -- 172.21.15.158:0/1573472386 >> v1:172.21.15.172:6789/0 conn(0x7facec0d0310 legacy=0x7facec15be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 -- 172.21.15.158:0/1573472386 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec0cf830 con 0x7facec1524e0 2023-10-16T11:21:18.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7facf16b7700 1 --2- 172.21.15.158:0/1573472386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec1524e0 0x7facec0cbad0 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7facdc0098f0 tx=0x7facdc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c224b4eee94c5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7face3fff700 1 -- 172.21.15.158:0/1573472386 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc004230 con 0x7facec1524e0 2023-10-16T11:21:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7face3fff700 1 -- 172.21.15.158:0/1573472386 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc004390 con 0x7facec1524e0 2023-10-16T11:21:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.163+0000 7face3fff700 1 -- 172.21.15.158:0/1573472386 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0036a0 con 0x7facec1524e0 2023-10-16T11:21:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 -- 172.21.15.158:0/1573472386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec1524e0 msgr2=0x7facec0cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 --2- 172.21.15.158:0/1573472386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec1524e0 0x7facec0cbad0 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7facdc0098f0 tx=0x7facdc009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 -- 172.21.15.158:0/1573472386 shutdown_connections 2023-10-16T11:21:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 --2- 172.21.15.158:0/1573472386 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec0cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 --2- 172.21.15.158:0/1573472386 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec1524e0 0x7facec0cbad0 unknown :-1 s=CLOSED pgs=3105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 -- 172.21.15.158:0/1573472386 >> 172.21.15.158:0/1573472386 conn(0x7facec06a8d0 msgr2=0x7facec06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:18.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 -- 172.21.15.158:0/1573472386 shutdown_connections 2023-10-16T11:21:18.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.164+0000 7facf391b700 1 -- 172.21.15.158:0/1573472386 wait complete. 2023-10-16T11:21:18.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.165+0000 7facf391b700 1 Processor -- start 2023-10-16T11:21:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.165+0000 7facf391b700 1 -- start start 2023-10-16T11:21:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf391b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec0d0310 0x7facec0afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf391b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facec1524e0 0x7facec151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf391b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec06c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf391b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec1560d0 con 0x7facec0d0310 2023-10-16T11:21:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf391b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec152af0 con 0x7facec1524e0 2023-10-16T11:21:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf391b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0c7ce0 con 0x7facec15c530 2023-10-16T11:21:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec0d0310 0x7facec0afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec0d0310 0x7facec0afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42162/0 (socket says 172.21.15.158:42162) 2023-10-16T11:21:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 -- 172.21.15.158:0/929672238 learned_addr learned my addr 172.21.15.158:0/929672238 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf1eb8700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec06c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf0eb6700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facec1524e0 0x7facec151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 -- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 msgr2=0x7facec06c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec06c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 -- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facec1524e0 msgr2=0x7facec151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facec1524e0 0x7facec151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf16b7700 1 -- 172.21.15.158:0/929672238 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc009550 con 0x7facec0d0310 2023-10-16T11:21:18.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.166+0000 7facf0eb6700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facec1524e0 0x7facec151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:18.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.167+0000 7facf16b7700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec0d0310 0x7facec0afa70 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7facdc009130 tx=0x7facdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.167+0000 7face1ffb700 1 -- 172.21.15.158:0/929672238 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc003c00 con 0x7facec0d0310 2023-10-16T11:21:18.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.167+0000 7face1ffb700 1 -- 172.21.15.158:0/929672238 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc01f070 con 0x7facec0d0310 2023-10-16T11:21:18.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.167+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec156320 con 0x7facec0d0310 2023-10-16T11:21:18.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.167+0000 7face1ffb700 1 -- 172.21.15.158:0/929672238 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00b660 con 0x7facec0d0310 2023-10-16T11:21:18.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.167+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec1568e0 con 0x7facec0d0310 2023-10-16T11:21:18.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.168+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec0b7490 con 0x7facec0d0310 2023-10-16T11:21:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.168+0000 7face1ffb700 1 -- 172.21.15.158:0/929672238 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7facdc021030 con 0x7facec0d0310 2023-10-16T11:21:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.169+0000 7face1ffb700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7facd006b140 0x7facd006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.169+0000 7face1ffb700 1 -- 172.21.15.158:0/929672238 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 5657+0+0 (secure 0 0 0) 0x7facdc08ae80 con 0x7facec0d0310 2023-10-16T11:21:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.169+0000 7facf0eb6700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7facd006b140 0x7facd006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.169+0000 7facf0eb6700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7facd006b140 0x7facd006d600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7facec06dbc0 tx=0x7face8009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.173+0000 7face1ffb700 1 -- 172.21.15.158:0/929672238 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7facdc055d60 con 0x7facec0d0310 2023-10-16T11:21:18.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.369+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7facec0b7490 con 0x7facec0d0310 2023-10-16T11:21:18.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.369+0000 7face1ffb700 1 -- 172.21.15.158:0/929672238 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v253) v1 ==== 84+0+65 (secure 0 0 0) 0x7facdc059380 con 0x7facec0d0310 2023-10-16T11:21:18.371 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:18.371 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:21:18.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:18.372 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:21:18.372 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:21:18.372 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:18.372 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:21:18.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7facd006b140 msgr2=0x7facd006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7facd006b140 0x7facd006d600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7facec06dbc0 tx=0x7face8009040 comp rx=0 tx=0).stop 2023-10-16T11:21:18.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec0d0310 msgr2=0x7facec0afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec0d0310 0x7facec0afa70 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7facdc009130 tx=0x7facdc004750 comp rx=0 tx=0).stop 2023-10-16T11:21:18.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 shutdown_connections 2023-10-16T11:21:18.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7facd006b140 0x7facd006d600 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7facec15c530 0x7facec06c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7facec0d0310 0x7facec0afa70 unknown :-1 s=CLOSED pgs=3106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 --2- 172.21.15.158:0/929672238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facec1524e0 0x7facec151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 >> 172.21.15.158:0/929672238 conn(0x7facec06a8d0 msgr2=0x7facec15f690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 shutdown_connections 2023-10-16T11:21:18.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.371+0000 7facf391b700 1 -- 172.21.15.158:0/929672238 wait complete. 2023-10-16T11:21:18.385 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T11:21:18.385 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T11:21:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.511+0000 7fd086b55700 1 Processor -- start 2023-10-16T11:21:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.511+0000 7fd086b55700 1 -- start start 2023-10-16T11:21:18.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.511+0000 7fd086b55700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0800d0c60 0x7fd0800b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.511+0000 7fd086b55700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd0800b3460 0x7fd0800b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.511+0000 7fd086b55700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0800b4420 con 0x7fd08015b6b0 2023-10-16T11:21:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.511+0000 7fd086b55700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08015ba90 con 0x7fd0800b3460 2023-10-16T11:21:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.511+0000 7fd086b55700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800b38f0 con 0x7fd0800d0c60 2023-10-16T11:21:18.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd07ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0800d0c60 0x7fd0800b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd0850f2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd0800b3460 0x7fd0800b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd0850f2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd0800b3460 0x7fd0800b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:43354/0 (socket says 172.21.15.158:43354) 2023-10-16T11:21:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd0850f2700 1 -- 172.21.15.158:0/3209683019 learned_addr learned my addr 172.21.15.158:0/3209683019 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd07ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0800d0c60 0x7fd0800b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:53006/0 (socket says 172.21.15.158:53006) 2023-10-16T11:21:18.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd07ffff700 1 -- 172.21.15.158:0/3209683019 >> v2:172.21.15.172:3300/0 conn(0x7fd0800b3460 msgr2=0x7fd0800b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:18.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd07ffff700 1 --2- 172.21.15.158:0/3209683019 >> v2:172.21.15.172:3300/0 conn(0x7fd0800b3460 0x7fd0800b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd07ffff700 1 -- 172.21.15.158:0/3209683019 >> v1:172.21.15.172:6789/0 conn(0x7fd08015b6b0 legacy=0x7fd0800cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd07ffff700 1 -- 172.21.15.158:0/3209683019 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800cbc20 con 0x7fd0800d0c60 2023-10-16T11:21:18.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd0850f2700 1 --2- 172.21.15.158:0/3209683019 >> v2:172.21.15.172:3300/0 conn(0x7fd0800b3460 0x7fd0800b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:18.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.512+0000 7fd07ffff700 1 --2- 172.21.15.158:0/3209683019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0800d0c60 0x7fd0800b2f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009840 tx=0x7fd06c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d09faeae0679c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd07f7fe700 1 -- 172.21.15.158:0/3209683019 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c011830 con 0x7fd0800d0c60 2023-10-16T11:21:18.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd07f7fe700 1 -- 172.21.15.158:0/3209683019 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c011990 con 0x7fd0800d0c60 2023-10-16T11:21:18.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd07f7fe700 1 -- 172.21.15.158:0/3209683019 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c011d20 con 0x7fd0800d0c60 2023-10-16T11:21:18.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 -- 172.21.15.158:0/3209683019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0800d0c60 msgr2=0x7fd0800b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 --2- 172.21.15.158:0/3209683019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0800d0c60 0x7fd0800b2f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009840 tx=0x7fd06c009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:18.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 -- 172.21.15.158:0/3209683019 shutdown_connections 2023-10-16T11:21:18.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 --2- 172.21.15.158:0/3209683019 >> v2:172.21.15.172:3300/0 conn(0x7fd0800b3460 0x7fd0800b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 --2- 172.21.15.158:0/3209683019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0800d0c60 0x7fd0800b2f20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 -- 172.21.15.158:0/3209683019 >> 172.21.15.158:0/3209683019 conn(0x7fd0800cdf20 msgr2=0x7fd0800ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:18.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 -- 172.21.15.158:0/3209683019 shutdown_connections 2023-10-16T11:21:18.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.513+0000 7fd086b55700 1 -- 172.21.15.158:0/3209683019 wait complete. 2023-10-16T11:21:18.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.515+0000 7fd086b55700 1 Processor -- start 2023-10-16T11:21:18.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.515+0000 7fd086b55700 1 -- start start 2023-10-16T11:21:18.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd086b55700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0800b3460 0x7fd0800cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd086b55700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0800d0c60 0x7fd0801f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd086b55700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd08015b6b0 0x7fd0800a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd086b55700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800cb360 con 0x7fd0800b3460 2023-10-16T11:21:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd086b55700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c8610 con 0x7fd0800d0c60 2023-10-16T11:21:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd086b55700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c88f0 con 0x7fd08015b6b0 2023-10-16T11:21:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0800b3460 0x7fd0800cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0800b3460 0x7fd0800cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42176/0 (socket says 172.21.15.158:42176) 2023-10-16T11:21:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 -- 172.21.15.158:0/3110857683 learned_addr learned my addr 172.21.15.158:0/3110857683 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0850f2700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd08015b6b0 0x7fd0800a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd07ffff700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0800d0c60 0x7fd0801f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 -- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd08015b6b0 msgr2=0x7fd0800a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd08015b6b0 0x7fd0800a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 -- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0800d0c60 msgr2=0x7fd0801f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0800d0c60 0x7fd0801f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd0848f1700 1 -- 172.21.15.158:0/3110857683 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078009b30 con 0x7fd0800b3460 2023-10-16T11:21:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.516+0000 7fd07ffff700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0800d0c60 0x7fd0801f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.517+0000 7fd0848f1700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0800b3460 0x7fd0800cae20 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fd07400c430 tx=0x7fd07400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.517+0000 7fd07d7fa700 1 -- 172.21.15.158:0/3110857683 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740052d0 con 0x7fd0800b3460 2023-10-16T11:21:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.517+0000 7fd07d7fa700 1 -- 172.21.15.158:0/3110857683 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074005450 con 0x7fd0800b3460 2023-10-16T11:21:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.517+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c0094a0 con 0x7fd0800b3460 2023-10-16T11:21:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.517+0000 7fd07d7fa700 1 -- 172.21.15.158:0/3110857683 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074006aa0 con 0x7fd0800b3460 2023-10-16T11:21:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.517+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0800c8eb0 con 0x7fd0800b3460 2023-10-16T11:21:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.518+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0800ce620 con 0x7fd0800b3460 2023-10-16T11:21:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.518+0000 7fd07d7fa700 1 -- 172.21.15.158:0/3110857683 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd074003680 con 0x7fd0800b3460 2023-10-16T11:21:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.518+0000 7fd07d7fa700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd06406b140 0x7fd06406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.519+0000 7fd07d7fa700 1 -- 172.21.15.158:0/3110857683 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 5657+0+0 (secure 0 0 0) 0x7fd074088a90 con 0x7fd0800b3460 2023-10-16T11:21:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.519+0000 7fd07ffff700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd06406b140 0x7fd06406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.519+0000 7fd07ffff700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd06406b140 0x7fd06406d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd0800d1060 tx=0x7fd06c008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.523+0000 7fd07d7fa700 1 -- 172.21.15.158:0/3110857683 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd0740538e0 con 0x7fd0800b3460 2023-10-16T11:21:18.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.713+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd0800681b0 con 0x7fd0800b3460 2023-10-16T11:21:18.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.831+0000 7fd07d7fa700 1 -- 172.21.15.158:0/3110857683 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v254) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd074056f00 con 0x7fd0800b3460 2023-10-16T11:21:18.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.834+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd06406b140 msgr2=0x7fd06406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.834+0000 7fd086b55700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd06406b140 0x7fd06406d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd0800d1060 tx=0x7fd06c008030 comp rx=0 tx=0).stop 2023-10-16T11:21:18.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.834+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0800b3460 msgr2=0x7fd0800cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.834+0000 7fd086b55700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0800b3460 0x7fd0800cae20 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fd07400c430 tx=0x7fd07400c740 comp rx=0 tx=0).stop 2023-10-16T11:21:18.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 shutdown_connections 2023-10-16T11:21:18.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fd06406b140 0x7fd06406d600 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd08015b6b0 0x7fd0800a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0800b3460 0x7fd0800cae20 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 --2- 172.21.15.158:0/3110857683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0800d0c60 0x7fd0801f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 >> 172.21.15.158:0/3110857683 conn(0x7fd0800cdf20 msgr2=0x7fd0800b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:18.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 shutdown_connections 2023-10-16T11:21:18.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.835+0000 7fd086b55700 1 -- 172.21.15.158:0/3110857683 wait complete. 2023-10-16T11:21:18.841 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T11:21:18.852 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:21:18.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.978+0000 7fb63d773700 1 Processor -- start 2023-10-16T11:21:18.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.978+0000 7fb63d773700 1 -- start start 2023-10-16T11:21:18.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb63d773700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb63806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb63d773700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6380ce4f0 0x7fb6380d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb63d773700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6380d1100 con 0x7fb6380b3690 2023-10-16T11:21:18.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb63d773700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63806bf20 con 0x7fb6380ce4f0 2023-10-16T11:21:18.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb63d773700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380d0f70 con 0x7fb63815d700 2023-10-16T11:21:18.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6380ce4f0 0x7fb6380d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6380ce4f0 0x7fb6380d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42188/0 (socket says 172.21.15.158:42188) 2023-10-16T11:21:18.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 -- 172.21.15.158:0/1226116099 learned_addr learned my addr 172.21.15.158:0/1226116099 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:18.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb636ffd700 1 --2- 172.21.15.158:0/1226116099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb63806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 -- 172.21.15.158:0/1226116099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 msgr2=0x7fb63806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:18.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 --2- 172.21.15.158:0/1226116099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb63806b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 -- 172.21.15.158:0/1226116099 >> v1:172.21.15.172:6789/0 conn(0x7fb6380b3690 legacy=0x7fb6380cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 -- 172.21.15.158:0/1226116099 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63815eaf0 con 0x7fb6380ce4f0 2023-10-16T11:21:18.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.979+0000 7fb6377fe700 1 --2- 172.21.15.158:0/1226116099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380ce4f0 0x7fb6380d1590 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7fb630009e50 tx=0x7fb63000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d402440871944dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb635ffb700 1 -- 172.21.15.158:0/1226116099 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63000e070 con 0x7fb6380ce4f0 2023-10-16T11:21:18.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb635ffb700 1 -- 172.21.15.158:0/1226116099 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63000b9c0 con 0x7fb6380ce4f0 2023-10-16T11:21:18.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb635ffb700 1 -- 172.21.15.158:0/1226116099 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63000bd70 con 0x7fb6380ce4f0 2023-10-16T11:21:18.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 -- 172.21.15.158:0/1226116099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380ce4f0 msgr2=0x7fb6380d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 --2- 172.21.15.158:0/1226116099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380ce4f0 0x7fb6380d1590 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7fb630009e50 tx=0x7fb63000b330 comp rx=0 tx=0).stop 2023-10-16T11:21:18.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 -- 172.21.15.158:0/1226116099 shutdown_connections 2023-10-16T11:21:18.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 --2- 172.21.15.158:0/1226116099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380ce4f0 0x7fb6380d1590 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 --2- 172.21.15.158:0/1226116099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb63806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 -- 172.21.15.158:0/1226116099 >> 172.21.15.158:0/1226116099 conn(0x7fb638071220 msgr2=0x7fb63806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:18.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 -- 172.21.15.158:0/1226116099 shutdown_connections 2023-10-16T11:21:18.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.980+0000 7fb63d773700 1 -- 172.21.15.158:0/1226116099 wait complete. 2023-10-16T11:21:18.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.981+0000 7fb63d773700 1 Processor -- start 2023-10-16T11:21:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.981+0000 7fb63d773700 1 -- start start 2023-10-16T11:21:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb63d773700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380b3690 0x7fb638155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb63d773700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6380ce4f0 0x7fb6380c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb63d773700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb638152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb63d773700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380c8250 con 0x7fb6380b3690 2023-10-16T11:21:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb63d773700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6381566a0 con 0x7fb6380ce4f0 2023-10-16T11:21:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb63d773700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb638156980 con 0x7fb63815d700 2023-10-16T11:21:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb636ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380b3690 0x7fb638155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb636ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380b3690 0x7fb638155340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42190/0 (socket says 172.21.15.158:42190) 2023-10-16T11:21:18.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb636ffd700 1 -- 172.21.15.158:0/1073270080 learned_addr learned my addr 172.21.15.158:0/1073270080 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:18.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb6377fe700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb638152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.982+0000 7fb6367fc700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6380ce4f0 0x7fb6380c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb636ffd700 1 -- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 msgr2=0x7fb638152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb636ffd700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb638152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb636ffd700 1 -- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6380ce4f0 msgr2=0x7fb6380c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:18.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb636ffd700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6380ce4f0 0x7fb6380c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:18.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb636ffd700 1 -- 172.21.15.158:0/1073270080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630009b30 con 0x7fb6380b3690 2023-10-16T11:21:18.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb636ffd700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380b3690 0x7fb638155340 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00d800 tx=0x7fb62c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb61f7fe700 1 -- 172.21.15.158:0/1073270080 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0125f0 con 0x7fb6380b3690 2023-10-16T11:21:18.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb61f7fe700 1 -- 172.21.15.158:0/1073270080 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c012750 con 0x7fb6380b3690 2023-10-16T11:21:18.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6381523f0 con 0x7fb6380b3690 2023-10-16T11:21:18.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb61f7fe700 1 -- 172.21.15.158:0/1073270080 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c012b00 con 0x7fb6380b3690 2023-10-16T11:21:18.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.983+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb638156c60 con 0x7fb6380b3690 2023-10-16T11:21:18.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.984+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb638152a40 con 0x7fb6380b3690 2023-10-16T11:21:18.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.985+0000 7fb61f7fe700 1 -- 172.21.15.158:0/1073270080 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb62c012c60 con 0x7fb6380b3690 2023-10-16T11:21:18.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.985+0000 7fb61f7fe700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb62406b140 0x7fb62406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:18.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.985+0000 7fb61f7fe700 1 -- 172.21.15.158:0/1073270080 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6027+0+0 (secure 0 0 0) 0x7fb62c089ab0 con 0x7fb6380b3690 2023-10-16T11:21:18.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.985+0000 7fb6367fc700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb62406b140 0x7fb62406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:18.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.986+0000 7fb6367fc700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb62406b140 0x7fb62406d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb638155880 tx=0x7fb628009040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:18.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:18.990+0000 7fb61f7fe700 1 -- 172.21.15.158:0/1073270080 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb62c054750 con 0x7fb6380b3690 2023-10-16T11:21:19.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.183+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb638067870 con 0x7fb6380b3690 2023-10-16T11:21:19.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.833+0000 7fb61f7fe700 1 -- 172.21.15.158:0/1073270080 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v255) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb62c057d70 con 0x7fb6380b3690 2023-10-16T11:21:19.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb62406b140 msgr2=0x7fb62406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:19.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb62406b140 0x7fb62406d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb638155880 tx=0x7fb628009040 comp rx=0 tx=0).stop 2023-10-16T11:21:19.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380b3690 msgr2=0x7fb638155340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:19.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380b3690 0x7fb638155340 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00d800 tx=0x7fb62c00db10 comp rx=0 tx=0).stop 2023-10-16T11:21:19.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 shutdown_connections 2023-10-16T11:21:19.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb62406b140 0x7fb62406d600 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:19.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb63815d700 0x7fb638152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:19.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6380b3690 0x7fb638155340 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:19.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 --2- 172.21.15.158:0/1073270080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6380ce4f0 0x7fb6380c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:19.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.837+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 >> 172.21.15.158:0/1073270080 conn(0x7fb638071220 msgr2=0x7fb63815ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:19.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.838+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 shutdown_connections 2023-10-16T11:21:19.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.840+0000 7fb63d773700 1 -- 172.21.15.158:0/1073270080 wait complete. 2023-10-16T11:21:19.845 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T11:21:19.866 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T11:21:20.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.993+0000 7fdb2e33b700 1 Processor -- start 2023-10-16T11:21:20.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.993+0000 7fdb2e33b700 1 -- start start 2023-10-16T11:21:20.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2e33b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdb2815b590 0x7fdb280ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:20.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2e33b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815f1d0 0x7fdb280b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:20.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2e33b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdb280cbaf0 0x7fdb280cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:20.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2e33b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb280cc4c0 con 0x7fdb2815b590 2023-10-16T11:21:20.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2e33b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb280cf4c0 con 0x7fdb280cbaf0 2023-10-16T11:21:20.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2e33b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2815f970 con 0x7fdb2815f1d0 2023-10-16T11:21:20.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdb2815b590 0x7fdb280ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:20.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdb2815b590 0x7fdb280ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42198/0 (socket says 172.21.15.158:42198) 2023-10-16T11:21:20.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 -- 172.21.15.158:0/4155191588 learned_addr learned my addr 172.21.15.158:0/4155191588 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:20.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2c8d8700 1 --2- 172.21.15.158:0/4155191588 >> v2:172.21.15.172:3300/0 conn(0x7fdb280cbaf0 0x7fdb280cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:20.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb277fe700 1 --2- 172.21.15.158:0/4155191588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815f1d0 0x7fdb280b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:20.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 -- 172.21.15.158:0/4155191588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815f1d0 msgr2=0x7fdb280b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:20.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 --2- 172.21.15.158:0/4155191588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815f1d0 0x7fdb280b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 -- 172.21.15.158:0/4155191588 >> v2:172.21.15.172:3300/0 conn(0x7fdb280cbaf0 msgr2=0x7fdb280cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:20.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 --2- 172.21.15.158:0/4155191588 >> v2:172.21.15.172:3300/0 conn(0x7fdb280cbaf0 0x7fdb280cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 -- 172.21.15.158:0/4155191588 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2815fc50 con 0x7fdb2815b590 2023-10-16T11:21:20.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb2c8d8700 1 --2- 172.21.15.158:0/4155191588 >> v2:172.21.15.172:3300/0 conn(0x7fdb280cbaf0 0x7fdb280cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:20.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb277fe700 1 --2- 172.21.15.158:0/4155191588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815f1d0 0x7fdb280b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:20.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.994+0000 7fdb27fff700 1 --2- 172.21.15.158:0/4155191588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb2815b590 0x7fdb280ce510 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fdb280b3640 tx=0x7fdb1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78288eacfcbd8881 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:20.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb26ffd700 1 -- 172.21.15.158:0/4155191588 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c00b870 con 0x7fdb2815b590 2023-10-16T11:21:20.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb26ffd700 1 -- 172.21.15.158:0/4155191588 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c00b9d0 con 0x7fdb2815b590 2023-10-16T11:21:20.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb26ffd700 1 -- 172.21.15.158:0/4155191588 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c00bd60 con 0x7fdb2815b590 2023-10-16T11:21:20.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 -- 172.21.15.158:0/4155191588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb2815b590 msgr2=0x7fdb280ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:20.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/4155191588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb2815b590 0x7fdb280ce510 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fdb280b3640 tx=0x7fdb1c009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 -- 172.21.15.158:0/4155191588 shutdown_connections 2023-10-16T11:21:20.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/4155191588 >> v2:172.21.15.172:3300/0 conn(0x7fdb280cbaf0 0x7fdb280cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/4155191588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb2815b590 0x7fdb280ce510 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/4155191588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815f1d0 0x7fdb280b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 -- 172.21.15.158:0/4155191588 >> 172.21.15.158:0/4155191588 conn(0x7fdb2806a8d0 msgr2=0x7fdb2806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:20.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 -- 172.21.15.158:0/4155191588 shutdown_connections 2023-10-16T11:21:20.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.995+0000 7fdb2e33b700 1 -- 172.21.15.158:0/4155191588 wait complete. 2023-10-16T11:21:20.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.996+0000 7fdb2e33b700 1 Processor -- start 2023-10-16T11:21:20.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb2e33b700 1 -- start start 2023-10-16T11:21:20.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb2e33b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb280cbaf0 0x7fdb2815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:20.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb2e33b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815b590 0x7fdb28155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:20.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb2e33b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb2815f1d0 0x7fdb28155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:20.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb2e33b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2815c120 con 0x7fdb280cbaf0 2023-10-16T11:21:20.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb2e33b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2815cbf0 con 0x7fdb2815f1d0 2023-10-16T11:21:20.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb2e33b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2815c350 con 0x7fdb2815b590 2023-10-16T11:21:20.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb27fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb280cbaf0 0x7fdb2815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:20.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb27fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb280cbaf0 0x7fdb2815bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42206/0 (socket says 172.21.15.158:42206) 2023-10-16T11:21:20.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.997+0000 7fdb27fff700 1 -- 172.21.15.158:0/1262697159 learned_addr learned my addr 172.21.15.158:0/1262697159 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:20.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb277fe700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815b590 0x7fdb28155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:20.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb2c8d8700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb2815f1d0 0x7fdb28155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:20.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb27fff700 1 -- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815b590 msgr2=0x7fdb28155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:20.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb27fff700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815b590 0x7fdb28155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb27fff700 1 -- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb2815f1d0 msgr2=0x7fdb28155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:20.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb27fff700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb2815f1d0 0x7fdb28155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb27fff700 1 -- 172.21.15.158:0/1262697159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c009550 con 0x7fdb280cbaf0 2023-10-16T11:21:20.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb277fe700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815b590 0x7fdb28155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:20.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb2c8d8700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb2815f1d0 0x7fdb28155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:20.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb27fff700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb280cbaf0 0x7fdb2815bbe0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c000c00 tx=0x7fdb1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:20.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb24ff9700 1 -- 172.21.15.158:0/1262697159 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c004540 con 0x7fdb280cbaf0 2023-10-16T11:21:20.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb24ff9700 1 -- 172.21.15.158:0/1262697159 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c0036a0 con 0x7fdb280cbaf0 2023-10-16T11:21:20.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2815c630 con 0x7fdb280cbaf0 2023-10-16T11:21:20.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb24ff9700 1 -- 172.21.15.158:0/1262697159 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c003a50 con 0x7fdb280cbaf0 2023-10-16T11:21:20.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.998+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb280c9060 con 0x7fdb280cbaf0 2023-10-16T11:21:20.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:19.999+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb280c8ec0 con 0x7fdb280cbaf0 2023-10-16T11:21:20.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.000+0000 7fdb24ff9700 1 -- 172.21.15.158:0/1262697159 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdb1c021070 con 0x7fdb280cbaf0 2023-10-16T11:21:20.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.000+0000 7fdb24ff9700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdb1406b140 0x7fdb1406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:20.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.000+0000 7fdb24ff9700 1 -- 172.21.15.158:0/1262697159 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6409+0+0 (secure 0 0 0) 0x7fdb1c08b310 con 0x7fdb280cbaf0 2023-10-16T11:21:20.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.000+0000 7fdb277fe700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdb1406b140 0x7fdb1406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:20.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.001+0000 7fdb277fe700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdb1406b140 0x7fdb1406d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdb280b62b0 tx=0x7fdb18008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:20.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.004+0000 7fdb24ff9700 1 -- 172.21.15.158:0/1262697159 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb1c055f10 con 0x7fdb280cbaf0 2023-10-16T11:21:20.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.239+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fdb280c8ec0 con 0x7fdb280cbaf0 2023-10-16T11:21:20.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.867+0000 7fdb24ff9700 1 -- 172.21.15.158:0/1262697159 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v141) v1 ==== 174+0+0 (secure 0 0 0) 0x7fdb280c8ec0 con 0x7fdb280cbaf0 2023-10-16T11:21:20.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.874+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdb1406b140 msgr2=0x7fdb1406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:20.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.874+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdb1406b140 0x7fdb1406d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdb280b62b0 tx=0x7fdb18008040 comp rx=0 tx=0).stop 2023-10-16T11:21:20.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.874+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb280cbaf0 msgr2=0x7fdb2815bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:20.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.874+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb280cbaf0 0x7fdb2815bbe0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c000c00 tx=0x7fdb1c004750 comp rx=0 tx=0).stop 2023-10-16T11:21:20.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.876+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 shutdown_connections 2023-10-16T11:21:20.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.876+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fdb1406b140 0x7fdb1406d600 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.876+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb2815f1d0 0x7fdb28155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.876+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb280cbaf0 0x7fdb2815bbe0 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.876+0000 7fdb2e33b700 1 --2- 172.21.15.158:0/1262697159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb2815b590 0x7fdb28155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:20.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.876+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 >> 172.21.15.158:0/1262697159 conn(0x7fdb2806a8d0 msgr2=0x7fdb281596b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:20.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.877+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 shutdown_connections 2023-10-16T11:21:20.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:20.877+0000 7fdb2e33b700 1 -- 172.21.15.158:0/1262697159 wait complete. 2023-10-16T11:21:20.882 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 35 and data pool 36 2023-10-16T11:21:20.896 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:21.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.015+0000 7fc75f330700 1 Processor -- start 2023-10-16T11:21:21.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.016+0000 7fc75f330700 1 -- start start 2023-10-16T11:21:21.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.016+0000 7fc75f330700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580b6e10 0x7fc7581521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.016+0000 7fc75f330700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7580ce2c0 0x7fc7581526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.016+0000 7fc75f330700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75815c5d0 con 0x7fc7580cdee0 2023-10-16T11:21:21.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.016+0000 7fc75f330700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75815d2c0 con 0x7fc7580ce2c0 2023-10-16T11:21:21.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.016+0000 7fc75f330700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc758152c20 con 0x7fc7580b6e10 2023-10-16T11:21:21.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7580ce2c0 0x7fc7581526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7580ce2c0 0x7fc7581526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42216/0 (socket says 172.21.15.158:42216) 2023-10-16T11:21:21.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 -- 172.21.15.158:0/3361947529 learned_addr learned my addr 172.21.15.158:0/3361947529 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:21.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 -- 172.21.15.158:0/3361947529 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580b6e10 msgr2=0x7fc7581521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 --2- 172.21.15.158:0/3361947529 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580b6e10 0x7fc7581521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 -- 172.21.15.158:0/3361947529 >> v1:172.21.15.172:6789/0 conn(0x7fc7580cdee0 legacy=0x7fc75815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 -- 172.21.15.158:0/3361947529 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75815c8b0 con 0x7fc7580ce2c0 2023-10-16T11:21:21.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc75d8cd700 1 --2- 172.21.15.158:0/3361947529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580ce2c0 0x7fc7581526e0 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fc750009e50 tx=0x7fc75000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e6f364f951d0f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc74ffff700 1 -- 172.21.15.158:0/3361947529 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75000e070 con 0x7fc7580ce2c0 2023-10-16T11:21:21.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc74ffff700 1 -- 172.21.15.158:0/3361947529 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750003a20 con 0x7fc7580ce2c0 2023-10-16T11:21:21.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.017+0000 7fc74ffff700 1 -- 172.21.15.158:0/3361947529 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750003dd0 con 0x7fc7580ce2c0 2023-10-16T11:21:21.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 -- 172.21.15.158:0/3361947529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580ce2c0 msgr2=0x7fc7581526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 --2- 172.21.15.158:0/3361947529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580ce2c0 0x7fc7581526e0 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fc750009e50 tx=0x7fc75000b330 comp rx=0 tx=0).stop 2023-10-16T11:21:21.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 -- 172.21.15.158:0/3361947529 shutdown_connections 2023-10-16T11:21:21.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 --2- 172.21.15.158:0/3361947529 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580ce2c0 0x7fc7581526e0 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 --2- 172.21.15.158:0/3361947529 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580b6e10 0x7fc7581521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 -- 172.21.15.158:0/3361947529 >> 172.21.15.158:0/3361947529 conn(0x7fc75806a8d0 msgr2=0x7fc75806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:21.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 -- 172.21.15.158:0/3361947529 shutdown_connections 2023-10-16T11:21:21.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.018+0000 7fc75f330700 1 -- 172.21.15.158:0/3361947529 wait complete. 2023-10-16T11:21:21.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.019+0000 7fc75f330700 1 Processor -- start 2023-10-16T11:21:21.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.019+0000 7fc75f330700 1 -- start start 2023-10-16T11:21:21.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75f330700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580b6e10 0x7fc7581f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75f330700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580cdee0 0x7fc75806bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75f330700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7580ce2c0 0x7fc75815a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75f330700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806c640 con 0x7fc7580b6e10 2023-10-16T11:21:21.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75f330700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806c920 con 0x7fc7580ce2c0 2023-10-16T11:21:21.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75f330700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806cc00 con 0x7fc7580cdee0 2023-10-16T11:21:21.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580b6e10 0x7fc7581f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580b6e10 0x7fc7581f6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42218/0 (socket says 172.21.15.158:42218) 2023-10-16T11:21:21.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 -- 172.21.15.158:0/1540935896 learned_addr learned my addr 172.21.15.158:0/1540935896 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:21.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d8cd700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7580ce2c0 0x7fc75815a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75c8cb700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580cdee0 0x7fc75806bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 -- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580cdee0 msgr2=0x7fc75806bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580cdee0 0x7fc75806bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 -- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7580ce2c0 msgr2=0x7fc75815a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7580ce2c0 0x7fc75815a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 -- 172.21.15.158:0/1540935896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750009b30 con 0x7fc7580b6e10 2023-10-16T11:21:21.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.020+0000 7fc75d0cc700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580b6e10 0x7fc7581f6480 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fc74800c4d0 tx=0x7fc74800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.021+0000 7fc74dffb700 1 -- 172.21.15.158:0/1540935896 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748011790 con 0x7fc7580b6e10 2023-10-16T11:21:21.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.021+0000 7fc74dffb700 1 -- 172.21.15.158:0/1540935896 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7480118f0 con 0x7fc7580b6e10 2023-10-16T11:21:21.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.021+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75806cee0 con 0x7fc7580b6e10 2023-10-16T11:21:21.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.021+0000 7fc74dffb700 1 -- 172.21.15.158:0/1540935896 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748011ca0 con 0x7fc7580b6e10 2023-10-16T11:21:21.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.021+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7581f69c0 con 0x7fc7580b6e10 2023-10-16T11:21:21.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.022+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc758074a90 con 0x7fc7580b6e10 2023-10-16T11:21:21.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.022+0000 7fc74dffb700 1 -- 172.21.15.158:0/1540935896 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc74801c460 con 0x7fc7580b6e10 2023-10-16T11:21:21.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.022+0000 7fc74dffb700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc73c06b140 0x7fc73c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.023+0000 7fc74dffb700 1 -- 172.21.15.158:0/1540935896 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fc748089c20 con 0x7fc7580b6e10 2023-10-16T11:21:21.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.023+0000 7fc75c8cb700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc73c06b140 0x7fc73c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.023+0000 7fc75c8cb700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc73c06b140 0x7fc73c06d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc754006fd0 tx=0x7fc754008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.027+0000 7fc74dffb700 1 -- 172.21.15.158:0/1540935896 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc748054690 con 0x7fc7580b6e10 2023-10-16T11:21:21.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.218+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7580681b0 con 0x7fc7580b6e10 2023-10-16T11:21:21.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.220+0000 7fc74dffb700 1 -- 172.21.15.158:0/1540935896 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+17480 (secure 0 0 0) 0x7fc748057cb0 con 0x7fc7580b6e10 2023-10-16T11:21:21.222 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:21.222 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":256,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:20.241108+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:18.736213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:19.205576+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:21.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc73c06b140 msgr2=0x7fc73c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc73c06b140 0x7fc73c06d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc754006fd0 tx=0x7fc754008040 comp rx=0 tx=0).stop 2023-10-16T11:21:21.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580b6e10 msgr2=0x7fc7581f6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580b6e10 0x7fc7581f6480 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fc74800c4d0 tx=0x7fc74800c890 comp rx=0 tx=0).stop 2023-10-16T11:21:21.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 shutdown_connections 2023-10-16T11:21:21.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc73c06b140 0x7fc73c06d600 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7580ce2c0 0x7fc75815a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7580b6e10 0x7fc7581f6480 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 --2- 172.21.15.158:0/1540935896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7580cdee0 0x7fc75806bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 >> 172.21.15.158:0/1540935896 conn(0x7fc75806a8d0 msgr2=0x7fc7580cbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:21.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 shutdown_connections 2023-10-16T11:21:21.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.222+0000 7fc75f330700 1 -- 172.21.15.158:0/1540935896 wait complete. 2023-10-16T11:21:21.236 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:21.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.355+0000 7f8483962700 1 Processor -- start 2023-10-16T11:21:21.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.356+0000 7f8483962700 1 -- start start 2023-10-16T11:21:21.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.356+0000 7f8483962700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 0x7f847c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.356+0000 7f8483962700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c161270 con 0x7f847c06e5d0 2023-10-16T11:21:21.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.356+0000 7f8483962700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c161510 con 0x7f847c0729b0 2023-10-16T11:21:21.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.356+0000 7f8483962700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1617f0 con 0x7f847c06ab40 2023-10-16T11:21:21.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8480efd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f847c06e5d0 0x7f847c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42402/0 (socket says 172.21.15.158:42402) 2023-10-16T11:21:21.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8480efd700 1 -- 172.21.15.158:0/2041943138 learned_addr learned my addr 172.21.15.158:0/2041943138 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:21.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f84816fe700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 0x7f847c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660227823 0 0) 0x7f847c161270 con 0x7f847c06e5d0 2023-10-16T11:21:21.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84680035b0 con 0x7f847c06e5d0 2023-10-16T11:21:21.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225084496 0 0) 0x7f847c161510 con 0x7f847c0729b0 2023-10-16T11:21:21.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f847c161270 con 0x7f847c0729b0 2023-10-16T11:21:21.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568292382 0 0) 0x7f84680035b0 con 0x7f847c06e5d0 2023-10-16T11:21:21.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 msgr2=0x7f847c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 0x7f847c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 >> v1:172.21.15.172:6789/0 conn(0x7f847c0729b0 legacy=0x7f847c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c161ad0 con 0x7f847c06e5d0 2023-10-16T11:21:21.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f846c001670 con 0x7f847c06e5d0 2023-10-16T11:21:21.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 >> v1:172.21.15.158:6789/0 conn(0x7f847c06e5d0 legacy=0x7f847c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468003bc0 0x7f8468003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8468004850 0x7f8468004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 0x7f8468005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1617f0 con 0x7f8468004850 2023-10-16T11:21:21.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846800bac0 con 0x7f8468003bc0 2023-10-16T11:21:21.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846800bc80 con 0x7f847c06ab40 2023-10-16T11:21:21.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8480efd700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8468004850 0x7f8468004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f84816fe700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468003bc0 0x7f8468003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.357+0000 7f8481eff700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 0x7f8468005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8480efd700 1 -- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 msgr2=0x7f8468005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8480efd700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c06ab40 0x7f8468005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8480efd700 1 -- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468003bc0 msgr2=0x7f8468003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8480efd700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468003bc0 0x7f8468003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8480efd700 1 -- 172.21.15.158:0/2041943138 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680035b0 con 0x7f8468004850 2023-10-16T11:21:21.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8480efd700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8468004850 0x7f8468004cd0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c3c10 tx=0x7f8468005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21024ab1b71d473 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00cc10 con 0x7f8468004850 2023-10-16T11:21:21.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c00cd70 con 0x7f8468004850 2023-10-16T11:21:21.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8473fff700 1 -- 172.21.15.158:0/2041943138 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c005b80 con 0x7f8468004850 2023-10-16T11:21:21.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8483962700 1 -- 172.21.15.158:0/2041943138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8468004850 msgr2=0x7f8468004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8483962700 1 --2- 172.21.15.158:0/2041943138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8468004850 0x7f8468004cd0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c3c10 tx=0x7f8468005880 comp rx=0 tx=0).stop 2023-10-16T11:21:21.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8481eff700 1 -- 172.21.15.158:0/2041943138 reap_dead start 2023-10-16T11:21:21.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8483962700 1 -- 172.21.15.158:0/2041943138 shutdown_connections 2023-10-16T11:21:21.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.358+0000 7f8483962700 1 -- 172.21.15.158:0/2041943138 >> 172.21.15.158:0/2041943138 conn(0x7f847c0d02b0 msgr2=0x7f847c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:21.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.359+0000 7f8483962700 1 -- 172.21.15.158:0/2041943138 shutdown_connections 2023-10-16T11:21:21.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.359+0000 7f8483962700 1 -- 172.21.15.158:0/2041943138 wait complete. 2023-10-16T11:21:21.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 Processor -- start 2023-10-16T11:21:21.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 -- start start 2023-10-16T11:21:21.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468004850 0x7f847c1605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c15df00 0x7f847c15e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f847c14c050 0x7f847c14c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1f2d20 con 0x7f847c14c050 2023-10-16T11:21:21.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c15f7f0 con 0x7f8468004850 2023-10-16T11:21:21.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.360+0000 7f8483962700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c15ec70 con 0x7f847c15df00 2023-10-16T11:21:21.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f847c14c050 0x7f847c14c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f847c14c050 0x7f847c14c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42240/0 (socket says 172.21.15.158:42240) 2023-10-16T11:21:21.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 -- 172.21.15.158:0/2440344541 learned_addr learned my addr 172.21.15.158:0/2440344541 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:21.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f84816fe700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468004850 0x7f847c1605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8480efd700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c15df00 0x7f847c15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 -- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c15df00 msgr2=0x7f847c15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c15df00 0x7f847c15e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 -- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468004850 msgr2=0x7f847c1605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468004850 0x7f847c1605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 -- 172.21.15.158:0/2440344541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c15ef50 con 0x7f847c14c050 2023-10-16T11:21:21.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f84816fe700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468004850 0x7f847c1605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:21.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8480efd700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c15df00 0x7f847c15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:21.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8481eff700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f847c14c050 0x7f847c14c500 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c7310 tx=0x7f847400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8471ffb700 1 -- 172.21.15.158:0/2440344541 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847400dc70 con 0x7f847c14c050 2023-10-16T11:21:21.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8471ffb700 1 -- 172.21.15.158:0/2440344541 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847400ddf0 con 0x7f847c14c050 2023-10-16T11:21:21.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c15f230 con 0x7f847c14c050 2023-10-16T11:21:21.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8471ffb700 1 -- 172.21.15.158:0/2440344541 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474003c40 con 0x7f847c14c050 2023-10-16T11:21:21.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.361+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c15e8c0 con 0x7f847c14c050 2023-10-16T11:21:21.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.362+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c1f2e60 con 0x7f847c14c050 2023-10-16T11:21:21.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.363+0000 7f8471ffb700 1 -- 172.21.15.158:0/2440344541 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f84740050d0 con 0x7f847c14c050 2023-10-16T11:21:21.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.363+0000 7f8471ffb700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f846006b140 0x7f846006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.363+0000 7f8471ffb700 1 -- 172.21.15.158:0/2440344541 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f847408a0d0 con 0x7f847c14c050 2023-10-16T11:21:21.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.363+0000 7f84816fe700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f846006b140 0x7f846006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.363+0000 7f84816fe700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f846006b140 0x7f846006d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8478006dd0 tx=0x7f8478008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.367+0000 7f8471ffb700 1 -- 172.21.15.158:0/2440344541 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f847c1f2e60 con 0x7f847c14c050 2023-10-16T11:21:21.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.563+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f847c1f2e60 con 0x7f847c14c050 2023-10-16T11:21:21.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.564+0000 7f8471ffb700 1 -- 172.21.15.158:0/2440344541 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+17480 (secure 0 0 0) 0x7f847c1f2e60 con 0x7f847c14c050 2023-10-16T11:21:21.566 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:21.566 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":256,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:20.241108+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:18.736213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:19.205576+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:21.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f846006b140 msgr2=0x7f846006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f846006b140 0x7f846006d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8478006dd0 tx=0x7f8478008040 comp rx=0 tx=0).stop 2023-10-16T11:21:21.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f847c14c050 msgr2=0x7f847c14c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f847c14c050 0x7f847c14c500 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c7310 tx=0x7f847400d710 comp rx=0 tx=0).stop 2023-10-16T11:21:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 shutdown_connections 2023-10-16T11:21:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f846006b140 0x7f846006d600 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f847c14c050 0x7f847c14c500 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8468004850 0x7f847c1605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 --2- 172.21.15.158:0/2440344541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f847c15df00 0x7f847c15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 >> 172.21.15.158:0/2440344541 conn(0x7f847c0d02b0 msgr2=0x7f847c0ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:21.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 shutdown_connections 2023-10-16T11:21:21.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.566+0000 7f8483962700 1 -- 172.21.15.158:0/2440344541 wait complete. 2023-10-16T11:21:21.581 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T11:21:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.702+0000 7fc3c7bd5700 1 Processor -- start 2023-10-16T11:21:21.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.702+0000 7fc3c7bd5700 1 -- start start 2023-10-16T11:21:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c7bd5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 0x7fc3c00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c7bd5700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00cbd30 con 0x7fc3c015da40 2023-10-16T11:21:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c7bd5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00cc1c0 con 0x7fc3c00cc590 2023-10-16T11:21:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c7bd5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c015de20 con 0x7fc3c006afe0 2023-10-16T11:21:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c5971700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc3c015da40 0x7fc3c006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42420/0 (socket says 172.21.15.158:42420) 2023-10-16T11:21:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c5971700 1 -- 172.21.15.158:0/2089393541 learned_addr learned my addr 172.21.15.158:0/2089393541 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c5170700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 0x7fc3c00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014800083 0 0) 0x7fc3c00cbd30 con 0x7fc3c015da40 2023-10-16T11:21:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac0035b0 con 0x7fc3c015da40 2023-10-16T11:21:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2097101079 0 0) 0x7fc3ac0035b0 con 0x7fc3c015da40 2023-10-16T11:21:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 msgr2=0x7fc3c00caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c496f700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 0x7fc3c00caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.703+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 >> v1:172.21.15.172:6789/0 conn(0x7fc3c00cc590 legacy=0x7fc3c00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c00ce260 con 0x7fc3c015da40 2023-10-16T11:21:21.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc3bc001670 con 0x7fc3c015da40 2023-10-16T11:21:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 >> v1:172.21.15.158:6789/0 conn(0x7fc3c015da40 legacy=0x7fc3c006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3ac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3ac0043e0 0x7fc3ac0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 0x7fc3ac005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c015de20 con 0x7fc3ac003a30 2023-10-16T11:21:21.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac00ba70 con 0x7fc3ac0043e0 2023-10-16T11:21:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac00bc30 con 0x7fc3c006afe0 2023-10-16T11:21:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5971700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3ac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5170700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3ac0043e0 0x7fc3ac0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c6172700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 0x7fc3ac005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5971700 1 -- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 msgr2=0x7fc3ac005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5971700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c006afe0 0x7fc3ac005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5971700 1 -- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3ac0043e0 msgr2=0x7fc3ac0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5971700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3ac0043e0 0x7fc3ac0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5971700 1 -- 172.21.15.158:0/2089393541 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac0035b0 con 0x7fc3ac003a30 2023-10-16T11:21:21.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5170700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3ac0043e0 0x7fc3ac0048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c5971700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3ac003e10 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001eb0 tx=0x7fc3ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1ba3554e94c662 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc00ce00 con 0x7fc3ac003a30 2023-10-16T11:21:21.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3bc005960 con 0x7fc3ac003a30 2023-10-16T11:21:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.704+0000 7fc3c496f700 1 -- 172.21.15.158:0/2089393541 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc005cf0 con 0x7fc3ac003a30 2023-10-16T11:21:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.705+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/2089393541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 msgr2=0x7fc3ac003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.705+0000 7fc3c7bd5700 1 --2- 172.21.15.158:0/2089393541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3ac003e10 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001eb0 tx=0x7fc3ac005880 comp rx=0 tx=0).stop 2023-10-16T11:21:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.705+0000 7fc3c6172700 1 -- 172.21.15.158:0/2089393541 reap_dead start 2023-10-16T11:21:21.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.705+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/2089393541 shutdown_connections 2023-10-16T11:21:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.705+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/2089393541 >> 172.21.15.158:0/2089393541 conn(0x7fc3c0099a10 msgr2=0x7fc3c00d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.705+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/2089393541 shutdown_connections 2023-10-16T11:21:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.705+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/2089393541 wait complete. 2023-10-16T11:21:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.706+0000 7fc3c7bd5700 1 Processor -- start 2023-10-16T11:21:21.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.706+0000 7fc3c7bd5700 1 -- start start 2023-10-16T11:21:21.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c7bd5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3c01f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c7bd5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c00c3e30 0x7fc3c00c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c7bd5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3c00b3900 0x7fc3c00b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c7bd5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00b42d0 con 0x7fc3ac003a30 2023-10-16T11:21:21.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c7bd5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00c4ed0 con 0x7fc3c00b3900 2023-10-16T11:21:21.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c7bd5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00c6700 con 0x7fc3c00c3e30 2023-10-16T11:21:21.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3c01f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3c01f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42256/0 (socket says 172.21.15.158:42256) 2023-10-16T11:21:21.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 -- 172.21.15.158:0/3066322803 learned_addr learned my addr 172.21.15.158:0/3066322803 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:21.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c6172700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3c00b3900 0x7fc3c00b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5170700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c00c3e30 0x7fc3c00c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 -- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c00c3e30 msgr2=0x7fc3c00c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c00c3e30 0x7fc3c00c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 -- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3c00b3900 msgr2=0x7fc3c00b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:21.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3c00b3900 0x7fc3c00b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:21.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 -- 172.21.15.158:0/3066322803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c00c60d0 con 0x7fc3ac003a30 2023-10-16T11:21:21.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c6172700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3c00b3900 0x7fc3c00b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:21.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.707+0000 7fc3c5971700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3c01f5920 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001930 tx=0x7fc3bc001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.708+0000 7fc3b67fc700 1 -- 172.21.15.158:0/3066322803 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc004a80 con 0x7fc3ac003a30 2023-10-16T11:21:21.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.708+0000 7fc3b67fc700 1 -- 172.21.15.158:0/3066322803 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3bc004be0 con 0x7fc3ac003a30 2023-10-16T11:21:21.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.708+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c00c42c0 con 0x7fc3ac003a30 2023-10-16T11:21:21.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.708+0000 7fc3b67fc700 1 -- 172.21.15.158:0/3066322803 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0070d0 con 0x7fc3ac003a30 2023-10-16T11:21:21.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.708+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c00c4880 con 0x7fc3ac003a30 2023-10-16T11:21:21.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.708+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c00c5240 con 0x7fc3ac003a30 2023-10-16T11:21:21.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.709+0000 7fc3b67fc700 1 -- 172.21.15.158:0/3066322803 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc3bc007230 con 0x7fc3ac003a30 2023-10-16T11:21:21.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.709+0000 7fc3b67fc700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc3a406b190 0x7fc3a406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:21.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.709+0000 7fc3c5170700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc3a406b190 0x7fc3a406d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:21.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.709+0000 7fc3b67fc700 1 -- 172.21.15.158:0/3066322803 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fc3bc08afa0 con 0x7fc3ac003a30 2023-10-16T11:21:21.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.710+0000 7fc3c5170700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc3a406b190 0x7fc3a406d650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3c015b070 tx=0x7fc3b0008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:21.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.714+0000 7fc3b67fc700 1 -- 172.21.15.158:0/3066322803 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3bc0bdfe0 con 0x7fc3ac003a30 2023-10-16T11:21:21.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:21.957+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc3c00b42d0 con 0x7fc3ac003a30 2023-10-16T11:21:22.831 INFO:tasks.ceph.mds.d.smithi172.stderr:2023-10-16T11:21:22.829+0000 7fedebc3a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:21:22.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.872+0000 7fc3b67fc700 1 -- 172.21.15.158:0/3066322803 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v144) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc3bc055b20 con 0x7fc3ac003a30 2023-10-16T11:21:22.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.876+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc3a406b190 msgr2=0x7fc3a406d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:22.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.876+0000 7fc3c7bd5700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc3a406b190 0x7fc3a406d650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3c015b070 tx=0x7fc3b0008040 comp rx=0 tx=0).stop 2023-10-16T11:21:22.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.876+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 msgr2=0x7fc3c01f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:22.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.876+0000 7fc3c7bd5700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3c01f5920 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001930 tx=0x7fc3bc001a00 comp rx=0 tx=0).stop 2023-10-16T11:21:22.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 shutdown_connections 2023-10-16T11:21:22.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fc3a406b190 0x7fc3a406d650 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:22.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3c00b3900 0x7fc3c00b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:22.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3ac003a30 0x7fc3c01f5920 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:22.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 --2- 172.21.15.158:0/3066322803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3c00c3e30 0x7fc3c00c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:22.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 >> 172.21.15.158:0/3066322803 conn(0x7fc3c0099a10 msgr2=0x7fc3c015c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:22.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 shutdown_connections 2023-10-16T11:21:22.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:22.877+0000 7fc3c7bd5700 1 -- 172.21.15.158:0/3066322803 wait complete. 2023-10-16T11:21:22.894 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:21:22.894 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T11:21:23.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.024+0000 7fe3e9697700 1 Processor -- start 2023-10-16T11:21:23.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e9697700 1 -- start start 2023-10-16T11:21:23.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e9697700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe3e4159af0 0x7fe3e40ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:23.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e9697700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe3e40ce3e0 0x7fe3e415de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e9697700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40b0c00 0x7fe3e40b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e9697700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e40b15d0 con 0x7fe3e4159af0 2023-10-16T11:21:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e9697700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e40cd3d0 con 0x7fe3e40ce3e0 2023-10-16T11:21:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e9697700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40cdf50 con 0x7fe3e40b0c00 2023-10-16T11:21:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.025+0000 7fe3e2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe3e4159af0 0x7fe3e40ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:23.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe3e4159af0 0x7fe3e40ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42264/0 (socket says 172.21.15.158:42264) 2023-10-16T11:21:23.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1591413378 learned_addr learned my addr 172.21.15.158:0/1591413378 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:23.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e37fe700 1 --2- 172.21.15.158:0/1591413378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40b0c00 0x7fe3e40b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:23.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e27fc700 1 --2- 172.21.15.158:0/1591413378 >> v2:172.21.15.172:3300/0 conn(0x7fe3e40ce3e0 0x7fe3e415de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:23.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1591413378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40b0c00 msgr2=0x7fe3e40b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:23.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 --2- 172.21.15.158:0/1591413378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40b0c00 0x7fe3e40b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1591413378 >> v2:172.21.15.172:3300/0 conn(0x7fe3e40ce3e0 msgr2=0x7fe3e415de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:23.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 --2- 172.21.15.158:0/1591413378 >> v2:172.21.15.172:3300/0 conn(0x7fe3e40ce3e0 0x7fe3e415de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1591413378 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e40cd620 con 0x7fe3e4159af0 2023-10-16T11:21:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e27fc700 1 --2- 172.21.15.158:0/1591413378 >> v2:172.21.15.172:3300/0 conn(0x7fe3e40ce3e0 0x7fe3e415de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e2ffd700 1 --2- 172.21.15.158:0/1591413378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e4159af0 0x7fe3e40ca8b0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40cbb60 tx=0x7fe3d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc15b465999977ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e1ffb700 1 -- 172.21.15.158:0/1591413378 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d800b870 con 0x7fe3e4159af0 2023-10-16T11:21:23.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e1ffb700 1 -- 172.21.15.158:0/1591413378 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d800b9d0 con 0x7fe3e4159af0 2023-10-16T11:21:23.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.026+0000 7fe3e1ffb700 1 -- 172.21.15.158:0/1591413378 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d800bd60 con 0x7fe3e4159af0 2023-10-16T11:21:23.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 -- 172.21.15.158:0/1591413378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e4159af0 msgr2=0x7fe3e40ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:23.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1591413378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e4159af0 0x7fe3e40ca8b0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40cbb60 tx=0x7fe3d8009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 -- 172.21.15.158:0/1591413378 shutdown_connections 2023-10-16T11:21:23.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1591413378 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40b0c00 0x7fe3e40b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1591413378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e4159af0 0x7fe3e40ca8b0 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1591413378 >> v2:172.21.15.172:3300/0 conn(0x7fe3e40ce3e0 0x7fe3e415de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 -- 172.21.15.158:0/1591413378 >> 172.21.15.158:0/1591413378 conn(0x7fe3e406af00 msgr2=0x7fe3e40cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:23.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 -- 172.21.15.158:0/1591413378 shutdown_connections 2023-10-16T11:21:23.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.027+0000 7fe3e9697700 1 -- 172.21.15.158:0/1591413378 wait complete. 2023-10-16T11:21:23.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.028+0000 7fe3e9697700 1 Processor -- start 2023-10-16T11:21:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.028+0000 7fe3e9697700 1 -- start start 2023-10-16T11:21:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e9697700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e40b0c00 0x7fe3e41453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e9697700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40ce3e0 0x7fe3e41489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e9697700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3e4159af0 0x7fe3e4148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:23.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e9697700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e4145d00 con 0x7fe3e40b0c00 2023-10-16T11:21:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e9697700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e414c030 con 0x7fe3e4159af0 2023-10-16T11:21:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e9697700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e414c600 con 0x7fe3e40ce3e0 2023-10-16T11:21:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e40b0c00 0x7fe3e41453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:23.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e40b0c00 0x7fe3e41453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42270/0 (socket says 172.21.15.158:42270) 2023-10-16T11:21:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1397371245 learned_addr learned my addr 172.21.15.158:0/1397371245 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e27fc700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40ce3e0 0x7fe3e41489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e37fe700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3e4159af0 0x7fe3e4148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40ce3e0 msgr2=0x7fe3e41489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:23.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40ce3e0 0x7fe3e41489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3e4159af0 msgr2=0x7fe3e4148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3e4159af0 0x7fe3e4148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 -- 172.21.15.158:0/1397371245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8009550 con 0x7fe3e40b0c00 2023-10-16T11:21:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e37fe700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3e4159af0 0x7fe3e4148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:23.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.029+0000 7fe3e2ffd700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e40b0c00 0x7fe3e41453d0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8014040 tx=0x7fe3d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.030+0000 7fe3cb7fe700 1 -- 172.21.15.158:0/1397371245 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8004520 con 0x7fe3e40b0c00 2023-10-16T11:21:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.030+0000 7fe3cb7fe700 1 -- 172.21.15.158:0/1397371245 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d801b6c0 con 0x7fe3e40b0c00 2023-10-16T11:21:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.030+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e414c8e0 con 0x7fe3e40b0c00 2023-10-16T11:21:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.030+0000 7fe3cb7fe700 1 -- 172.21.15.158:0/1397371245 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d801ba70 con 0x7fe3e40b0c00 2023-10-16T11:21:23.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.030+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e414b350 con 0x7fe3e40b0c00 2023-10-16T11:21:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.031+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e414c460 con 0x7fe3e40b0c00 2023-10-16T11:21:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.031+0000 7fe3cb7fe700 1 -- 172.21.15.158:0/1397371245 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe3d801bbd0 con 0x7fe3e40b0c00 2023-10-16T11:21:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.031+0000 7fe3cb7fe700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fe3d006b140 0x7fe3d006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.032+0000 7fe3cb7fe700 1 -- 172.21.15.158:0/1397371245 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe3d808cae0 con 0x7fe3e40b0c00 2023-10-16T11:21:23.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.032+0000 7fe3e27fc700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fe3d006b140 0x7fe3d006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.032+0000 7fe3e27fc700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fe3d006b140 0x7fe3d006d600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4006fd0 tx=0x7fe3d4008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:23.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.036+0000 7fe3cb7fe700 1 -- 172.21.15.158:0/1397371245 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3e414c460 con 0x7fe3e40b0c00 2023-10-16T11:21:23.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.274+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe3e414ce00 con 0x7fe3e40b0c00 2023-10-16T11:21:23.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.874+0000 7fe3cb7fe700 1 -- 172.21.15.158:0/1397371245 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v145) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe3e414ce00 con 0x7fe3e40b0c00 2023-10-16T11:21:23.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fe3d006b140 msgr2=0x7fe3d006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:23.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fe3d006b140 0x7fe3d006d600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4006fd0 tx=0x7fe3d4008040 comp rx=0 tx=0).stop 2023-10-16T11:21:23.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e40b0c00 msgr2=0x7fe3e41453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:23.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e40b0c00 0x7fe3e41453d0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8014040 tx=0x7fe3d8004750 comp rx=0 tx=0).stop 2023-10-16T11:21:23.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 shutdown_connections 2023-10-16T11:21:23.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fe3d006b140 0x7fe3d006d600 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3e4159af0 0x7fe3e4148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3e40b0c00 0x7fe3e41453d0 unknown :-1 s=CLOSED pgs=3121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.876+0000 7fe3e9697700 1 --2- 172.21.15.158:0/1397371245 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3e40ce3e0 0x7fe3e41489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:23.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.877+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 >> 172.21.15.158:0/1397371245 conn(0x7fe3e406af00 msgr2=0x7fe3e41580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:23.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.877+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 shutdown_connections 2023-10-16T11:21:23.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:23.877+0000 7fe3e9697700 1 -- 172.21.15.158:0/1397371245 wait complete. 2023-10-16T11:21:23.892 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T11:21:23.892 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:24.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 Processor -- start 2023-10-16T11:21:24.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 -- start start 2023-10-16T11:21:24.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffae006ab40 0x7ffae006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffae006e5d0 0x7ffae0071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006af20 0x7ffae00722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae0071e50 con 0x7ffae006ab40 2023-10-16T11:21:24.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae0161270 con 0x7ffae006e5d0 2023-10-16T11:21:24.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.015+0000 7ffae711c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae0161510 con 0x7ffae006af20 2023-10-16T11:21:24.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffae006ab40 0x7ffae006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffae006ab40 0x7ffae006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42278/0 (socket says 172.21.15.158:42278) 2023-10-16T11:21:24.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 -- 172.21.15.158:0/1995612889 learned_addr learned my addr 172.21.15.158:0/1995612889 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:24.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffadffff700 1 --2- 172.21.15.158:0/1995612889 >> v2:172.21.15.172:3300/0 conn(0x7ffae006e5d0 0x7ffae0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae56b9700 1 --2- 172.21.15.158:0/1995612889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006af20 0x7ffae00722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 -- 172.21.15.158:0/1995612889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006af20 msgr2=0x7ffae00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:24.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 --2- 172.21.15.158:0/1995612889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006af20 0x7ffae00722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 -- 172.21.15.158:0/1995612889 >> v2:172.21.15.172:3300/0 conn(0x7ffae006e5d0 msgr2=0x7ffae0071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:24.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 --2- 172.21.15.158:0/1995612889 >> v2:172.21.15.172:3300/0 conn(0x7ffae006e5d0 0x7ffae0071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 -- 172.21.15.158:0/1995612889 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae01617f0 con 0x7ffae006ab40 2023-10-16T11:21:24.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffae4eb8700 1 --2- 172.21.15.158:0/1995612889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae006d820 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7ffae00c3bf0 tx=0x7ffad4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7dec84406e8cbb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffadf7fe700 1 -- 172.21.15.158:0/1995612889 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad400b870 con 0x7ffae006ab40 2023-10-16T11:21:24.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffadf7fe700 1 -- 172.21.15.158:0/1995612889 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad400b9d0 con 0x7ffae006ab40 2023-10-16T11:21:24.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.016+0000 7ffadf7fe700 1 -- 172.21.15.158:0/1995612889 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad400bd60 con 0x7ffae006ab40 2023-10-16T11:21:24.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 -- 172.21.15.158:0/1995612889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 msgr2=0x7ffae006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 --2- 172.21.15.158:0/1995612889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae006d820 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7ffae00c3bf0 tx=0x7ffad4009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 -- 172.21.15.158:0/1995612889 shutdown_connections 2023-10-16T11:21:24.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 --2- 172.21.15.158:0/1995612889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006af20 0x7ffae00722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 --2- 172.21.15.158:0/1995612889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae006d820 unknown :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 --2- 172.21.15.158:0/1995612889 >> v2:172.21.15.172:3300/0 conn(0x7ffae006e5d0 0x7ffae0071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 -- 172.21.15.158:0/1995612889 >> 172.21.15.158:0/1995612889 conn(0x7ffae00d02b0 msgr2=0x7ffae00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:24.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 -- 172.21.15.158:0/1995612889 shutdown_connections 2023-10-16T11:21:24.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.017+0000 7ffae711c700 1 -- 172.21.15.158:0/1995612889 wait complete. 2023-10-16T11:21:24.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.018+0000 7ffae711c700 1 Processor -- start 2023-10-16T11:21:24.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.018+0000 7ffae711c700 1 -- start start 2023-10-16T11:21:24.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.019+0000 7ffae711c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae01f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.019+0000 7ffae711c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae006af20 0x7ffae00cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.019+0000 7ffae711c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006e5d0 0x7ffae00cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.019+0000 7ffae711c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00c74b0 con 0x7ffae006ab40 2023-10-16T11:21:24.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.019+0000 7ffae711c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00c7790 con 0x7ffae006af20 2023-10-16T11:21:24.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.019+0000 7ffae711c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00c7a70 con 0x7ffae006e5d0 2023-10-16T11:21:24.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.019+0000 7ffae4eb8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae01f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae01f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42284/0 (socket says 172.21.15.158:42284) 2023-10-16T11:21:24.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 -- 172.21.15.158:0/839519154 learned_addr learned my addr 172.21.15.158:0/839519154 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:24.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffadffff700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae006af20 0x7ffae00cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae56b9700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006e5d0 0x7ffae00cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 -- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006e5d0 msgr2=0x7ffae00cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006e5d0 0x7ffae00cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 -- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae006af20 msgr2=0x7ffae00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae006af20 0x7ffae00cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 -- 172.21.15.158:0/839519154 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad4009550 con 0x7ffae006ab40 2023-10-16T11:21:24.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae4eb8700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae01f29e0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7ffad4000c00 tx=0x7ffad4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffadd7fa700 1 -- 172.21.15.158:0/839519154 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4004540 con 0x7ffae006ab40 2023-10-16T11:21:24.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffadd7fa700 1 -- 172.21.15.158:0/839519154 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad40036a0 con 0x7ffae006ab40 2023-10-16T11:21:24.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae00c7d50 con 0x7ffae006ab40 2023-10-16T11:21:24.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffadd7fa700 1 -- 172.21.15.158:0/839519154 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4003a50 con 0x7ffae006ab40 2023-10-16T11:21:24.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.020+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae00c4060 con 0x7ffae006ab40 2023-10-16T11:21:24.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.021+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae00681b0 con 0x7ffae006ab40 2023-10-16T11:21:24.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.022+0000 7ffadd7fa700 1 -- 172.21.15.158:0/839519154 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ffad4021070 con 0x7ffae006ab40 2023-10-16T11:21:24.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.022+0000 7ffadd7fa700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ffac406b140 0x7ffac406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.022+0000 7ffadd7fa700 1 -- 172.21.15.158:0/839519154 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ffad408b3e0 con 0x7ffae006ab40 2023-10-16T11:21:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.022+0000 7ffadffff700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ffac406b140 0x7ffac406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.023+0000 7ffadffff700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ffac406b140 0x7ffac406d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffae00c99d0 tx=0x7ffacc008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.027+0000 7ffadd7fa700 1 -- 172.21.15.158:0/839519154 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffad4055ea0 con 0x7ffae006ab40 2023-10-16T11:21:24.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.262+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffae00b6230 con 0x7ffae006ab40 2023-10-16T11:21:24.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.263+0000 7ffadd7fa700 1 -- 172.21.15.158:0/839519154 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4935 (secure 0 0 0) 0x7ffad40594c0 con 0x7ffae006ab40 2023-10-16T11:21:24.265 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:24.265 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":145,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1010484950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1010484950},{"type":"v1","addr":"172.21.15.158:6836","nonce":1010484950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":13224,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/54641557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":54641557},{"type":"v1","addr":"172.21.15.158:6837","nonce":54641557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":13239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3388825539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3388825539},{"type":"v1","addr":"172.21.15.172:6835","nonce":3388825539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:20.241439+0000","modified":"2023-10-16T11:21:23.275699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39845":{"gid":39845,"name":"d","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3986459990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3986459990},{"type":"v1","addr":"172.21.15.172:6834","nonce":3986459990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T11:21:24.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ffac406b140 msgr2=0x7ffac406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ffac406b140 0x7ffac406d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffae00c99d0 tx=0x7ffacc008040 comp rx=0 tx=0).stop 2023-10-16T11:21:24.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 msgr2=0x7ffae01f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae01f29e0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7ffad4000c00 tx=0x7ffad4004750 comp rx=0 tx=0).stop 2023-10-16T11:21:24.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 shutdown_connections 2023-10-16T11:21:24.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ffac406b140 0x7ffac406d600 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae006e5d0 0x7ffae00cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffae006ab40 0x7ffae01f29e0 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 --2- 172.21.15.158:0/839519154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae006af20 0x7ffae00cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.265+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 >> 172.21.15.158:0/839519154 conn(0x7ffae00d02b0 msgr2=0x7ffae015d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:24.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.266+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 shutdown_connections 2023-10-16T11:21:24.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.266+0000 7ffae711c700 1 -- 172.21.15.158:0/839519154 wait complete. 2023-10-16T11:21:24.270 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 145 2023-10-16T11:21:24.279 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:24.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.398+0000 7f8f88a0e700 1 Processor -- start 2023-10-16T11:21:24.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.398+0000 7f8f88a0e700 1 -- start start 2023-10-16T11:21:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f88a0e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8f8415b5c0 0x7f8f840718d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f88a0e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f8406d770 0x7f8f840ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f88a0e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f84072b70 con 0x7f8f840cae30 2023-10-16T11:21:24.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f88a0e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840cb2e0 con 0x7f8f8406d770 2023-10-16T11:21:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f88a0e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8415b9a0 con 0x7f8f8415b5c0 2023-10-16T11:21:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f81d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f8406d770 0x7f8f840ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f82d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8f840cae30 0x7f8f8406f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42472/0 (socket says 172.21.15.158:42472) 2023-10-16T11:21:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f82d9d700 1 -- 172.21.15.158:0/3443260472 learned_addr learned my addr 172.21.15.158:0/3443260472 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:24.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.399+0000 7f8f8259c700 1 --2- 172.21.15.158:0/3443260472 >> v2:172.21.15.172:3300/0 conn(0x7f8f8415b5c0 0x7f8f840718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f81d9b700 1 -- 172.21.15.158:0/3443260472 >> v2:172.21.15.172:3300/0 conn(0x7f8f8415b5c0 msgr2=0x7f8f840718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f81d9b700 1 --2- 172.21.15.158:0/3443260472 >> v2:172.21.15.172:3300/0 conn(0x7f8f8415b5c0 0x7f8f840718d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f81d9b700 1 -- 172.21.15.158:0/3443260472 >> v1:172.21.15.158:6789/0 conn(0x7f8f840cae30 legacy=0x7f8f8406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f81d9b700 1 -- 172.21.15.158:0/3443260472 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f84071f60 con 0x7f8f8406d770 2023-10-16T11:21:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f8159a700 1 -- 172.21.15.158:0/3443260472 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610183986 0 0) 0x7f8f84072b70 con 0x7f8f840cae30 2023-10-16T11:21:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f81d9b700 1 --2- 172.21.15.158:0/3443260472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8406d770 0x7f8f840ca8f0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f8f8406c9a0 tx=0x7f8f74009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8041a4daaa57c5c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f8159a700 1 -- 172.21.15.158:0/3443260472 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74011830 con 0x7f8f8406d770 2023-10-16T11:21:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f8159a700 1 -- 172.21.15.158:0/3443260472 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74011990 con 0x7f8f8406d770 2023-10-16T11:21:24.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f8159a700 1 -- 172.21.15.158:0/3443260472 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74011d20 con 0x7f8f8406d770 2023-10-16T11:21:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3443260472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8406d770 msgr2=0x7f8f840ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.400+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3443260472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8406d770 0x7f8f840ca8f0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f8f8406c9a0 tx=0x7f8f74009b20 comp rx=0 tx=0).stop 2023-10-16T11:21:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.401+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3443260472 shutdown_connections 2023-10-16T11:21:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.401+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3443260472 >> v2:172.21.15.172:3300/0 conn(0x7f8f8415b5c0 0x7f8f840718d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.401+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3443260472 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f8406d770 0x7f8f840ca8f0 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.401+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3443260472 >> 172.21.15.158:0/3443260472 conn(0x7f8f840d02a0 msgr2=0x7f8f840d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:24.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.401+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3443260472 shutdown_connections 2023-10-16T11:21:24.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.401+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3443260472 wait complete. 2023-10-16T11:21:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.402+0000 7f8f88a0e700 1 Processor -- start 2023-10-16T11:21:24.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.402+0000 7f8f88a0e700 1 -- start start 2023-10-16T11:21:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f88a0e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8406d770 0x7f8f841e2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f88a0e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f840cae30 0x7f8f840c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f88a0e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8415b5c0 0x7f8f840c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f88a0e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f840b2670 con 0x7f8f840cae30 2023-10-16T11:21:24.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f88a0e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f841e3030 con 0x7f8f8415b5c0 2023-10-16T11:21:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f88a0e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f841e3260 con 0x7f8f8406d770 2023-10-16T11:21:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f840cae30 0x7f8f840c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f840cae30 0x7f8f840c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42300/0 (socket says 172.21.15.158:42300) 2023-10-16T11:21:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 -- 172.21.15.158:0/3778841372 learned_addr learned my addr 172.21.15.158:0/3778841372 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:24.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f8259c700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8406d770 0x7f8f841e2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f82d9d700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8415b5c0 0x7f8f840c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 -- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8406d770 msgr2=0x7f8f841e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8406d770 0x7f8f841e2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 -- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8415b5c0 msgr2=0x7f8f840c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8415b5c0 0x7f8f840c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 -- 172.21.15.158:0/3778841372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740094a0 con 0x7f8f840cae30 2023-10-16T11:21:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.403+0000 7f8f81d9b700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f840cae30 0x7f8f840c7370 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f8f84070340 tx=0x7f8f74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.404+0000 7f8f72ffd700 1 -- 172.21.15.158:0/3778841372 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f740036a0 con 0x7f8f840cae30 2023-10-16T11:21:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.404+0000 7f8f72ffd700 1 -- 172.21.15.158:0/3778841372 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7401f070 con 0x7f8f840cae30 2023-10-16T11:21:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.404+0000 7f8f72ffd700 1 -- 172.21.15.158:0/3778841372 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74003a60 con 0x7f8f840cae30 2023-10-16T11:21:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.404+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f841e3540 con 0x7f8f840cae30 2023-10-16T11:21:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.404+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f841e3b00 con 0x7f8f840cae30 2023-10-16T11:21:24.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.404+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f840681b0 con 0x7f8f840cae30 2023-10-16T11:21:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.405+0000 7f8f72ffd700 1 -- 172.21.15.158:0/3778841372 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8f74003bc0 con 0x7f8f840cae30 2023-10-16T11:21:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.405+0000 7f8f72ffd700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f8f6406b140 0x7f8f6406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.405+0000 7f8f72ffd700 1 -- 172.21.15.158:0/3778841372 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f8f74021070 con 0x7f8f840cae30 2023-10-16T11:21:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.405+0000 7f8f8259c700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f8f6406b140 0x7f8f6406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.406+0000 7f8f8259c700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f8f6406b140 0x7f8f6406d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f841e2350 tx=0x7f8f78009210 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.410+0000 7f8f72ffd700 1 -- 172.21.15.158:0/3778841372 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f74055f10 con 0x7f8f840cae30 2023-10-16T11:21:24.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.602+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f841e16e0 con 0x7f8f840cae30 2023-10-16T11:21:24.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.603+0000 7f8f72ffd700 1 -- 172.21.15.158:0/3778841372 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+17467 (secure 0 0 0) 0x7f8f74059530 con 0x7f8f840cae30 2023-10-16T11:21:24.605 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:24.605 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":257,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:21.853347+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:18.736213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:19.205576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:24.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f8f6406b140 msgr2=0x7f8f6406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f8f6406b140 0x7f8f6406d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f841e2350 tx=0x7f8f78009210 comp rx=0 tx=0).stop 2023-10-16T11:21:24.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f840cae30 msgr2=0x7f8f840c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f840cae30 0x7f8f840c7370 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f8f84070340 tx=0x7f8f74004750 comp rx=0 tx=0).stop 2023-10-16T11:21:24.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 shutdown_connections 2023-10-16T11:21:24.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f8f6406b140 0x7f8f6406d600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8415b5c0 0x7f8f840c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f8406d770 0x7f8f841e2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 --2- 172.21.15.158:0/3778841372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f840cae30 0x7f8f840c7370 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 >> 172.21.15.158:0/3778841372 conn(0x7f8f840d02a0 msgr2=0x7f8f840b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:24.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.605+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 shutdown_connections 2023-10-16T11:21:24.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.606+0000 7f8f88a0e700 1 -- 172.21.15.158:0/3778841372 wait complete. 2023-10-16T11:21:24.620 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:21:24.620 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:21:24.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.741+0000 7fed3b7b8700 1 Processor -- start 2023-10-16T11:21:24.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.741+0000 7fed3b7b8700 1 -- start start 2023-10-16T11:21:24.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed3b7b8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fed3415d7c0 0x7fed34161290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed3b7b8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fed340ca5b0 0x7fed341617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed3b7b8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 0x7fed3406b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed3b7b8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed340c4d10 con 0x7fed340ca5b0 2023-10-16T11:21:24.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed3b7b8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed340c55b0 con 0x7fed3415d7c0 2023-10-16T11:21:24.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed3b7b8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3415dba0 con 0x7fed3406b300 2023-10-16T11:21:24.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed38d53700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fed340ca5b0 0x7fed341617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed38d53700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fed340ca5b0 0x7fed341617d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42306/0 (socket says 172.21.15.158:42306) 2023-10-16T11:21:24.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.742+0000 7fed38d53700 1 -- 172.21.15.158:0/683823445 learned_addr learned my addr 172.21.15.158:0/683823445 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:24.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed38d53700 1 -- 172.21.15.158:0/683823445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 msgr2=0x7fed3406b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:24.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed38d53700 1 --2- 172.21.15.158:0/683823445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 0x7fed3406b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed38d53700 1 -- 172.21.15.158:0/683823445 >> v2:172.21.15.172:3300/0 conn(0x7fed3415d7c0 msgr2=0x7fed34161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:24.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed38d53700 1 --2- 172.21.15.158:0/683823445 >> v2:172.21.15.172:3300/0 conn(0x7fed3415d7c0 0x7fed34161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed38d53700 1 -- 172.21.15.158:0/683823445 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3406c0c0 con 0x7fed340ca5b0 2023-10-16T11:21:24.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed38d53700 1 --2- 172.21.15.158:0/683823445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed341617d0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7fed340cf970 tx=0x7fed30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1ec1e5dc1e78b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed2bfff700 1 -- 172.21.15.158:0/683823445 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3000b870 con 0x7fed340ca5b0 2023-10-16T11:21:24.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed2bfff700 1 -- 172.21.15.158:0/683823445 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3000b9d0 con 0x7fed340ca5b0 2023-10-16T11:21:24.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed2bfff700 1 -- 172.21.15.158:0/683823445 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3000bd60 con 0x7fed340ca5b0 2023-10-16T11:21:24.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed3b7b8700 1 -- 172.21.15.158:0/683823445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 msgr2=0x7fed341617d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.743+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/683823445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed341617d0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7fed340cf970 tx=0x7fed30009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.744+0000 7fed3b7b8700 1 -- 172.21.15.158:0/683823445 shutdown_connections 2023-10-16T11:21:24.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.744+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/683823445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 0x7fed3406b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.744+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/683823445 >> v2:172.21.15.172:3300/0 conn(0x7fed3415d7c0 0x7fed34161290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.744+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/683823445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed341617d0 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.744+0000 7fed3b7b8700 1 -- 172.21.15.158:0/683823445 >> 172.21.15.158:0/683823445 conn(0x7fed340cc4b0 msgr2=0x7fed340cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:24.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.744+0000 7fed3b7b8700 1 -- 172.21.15.158:0/683823445 shutdown_connections 2023-10-16T11:21:24.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.744+0000 7fed3b7b8700 1 -- 172.21.15.158:0/683823445 wait complete. 2023-10-16T11:21:24.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.745+0000 7fed3b7b8700 1 Processor -- start 2023-10-16T11:21:24.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.745+0000 7fed3b7b8700 1 -- start start 2023-10-16T11:21:24.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed3b7b8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 0x7fed340b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed3b7b8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed340c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed3b7b8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed3415d7c0 0x7fed340c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed3b7b8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34152080 con 0x7fed340ca5b0 2023-10-16T11:21:24.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed3b7b8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340b0020 con 0x7fed3415d7c0 2023-10-16T11:21:24.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed3b7b8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed340ada80 con 0x7fed3406b300 2023-10-16T11:21:24.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed340c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed340c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42314/0 (socket says 172.21.15.158:42314) 2023-10-16T11:21:24.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 -- 172.21.15.158:0/4187830746 learned_addr learned my addr 172.21.15.158:0/4187830746 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:24.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed39554700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 0x7fed340b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed39d55700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed3415d7c0 0x7fed340c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 -- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 msgr2=0x7fed340b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 0x7fed340b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 -- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed3415d7c0 msgr2=0x7fed340c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed3415d7c0 0x7fed340c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 -- 172.21.15.158:0/4187830746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed30009550 con 0x7fed340ca5b0 2023-10-16T11:21:24.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed38d53700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed340c76d0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7fed340cd160 tx=0x7fed30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed29ffb700 1 -- 172.21.15.158:0/4187830746 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30004310 con 0x7fed340ca5b0 2023-10-16T11:21:24.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed29ffb700 1 -- 172.21.15.158:0/4187830746 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3001f070 con 0x7fed340ca5b0 2023-10-16T11:21:24.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed340af710 con 0x7fed340ca5b0 2023-10-16T11:21:24.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.746+0000 7fed29ffb700 1 -- 172.21.15.158:0/4187830746 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed300036a0 con 0x7fed340ca5b0 2023-10-16T11:21:24.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.747+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed340af9d0 con 0x7fed340ca5b0 2023-10-16T11:21:24.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.748+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed340b01d0 con 0x7fed340ca5b0 2023-10-16T11:21:24.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.748+0000 7fed29ffb700 1 -- 172.21.15.158:0/4187830746 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fed30021030 con 0x7fed340ca5b0 2023-10-16T11:21:24.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.748+0000 7fed29ffb700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fed1806b140 0x7fed1806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:24.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.748+0000 7fed29ffb700 1 -- 172.21.15.158:0/4187830746 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fed3008b000 con 0x7fed340ca5b0 2023-10-16T11:21:24.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.748+0000 7fed39554700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fed1806b140 0x7fed1806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:24.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.749+0000 7fed39554700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fed1806b140 0x7fed1806d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed24009970 tx=0x7fed24009110 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:24.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.753+0000 7fed29ffb700 1 -- 172.21.15.158:0/4187830746 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed340b01d0 con 0x7fed340ca5b0 2023-10-16T11:21:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.944+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 --> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed340c5240 con 0x7fed1806b140 2023-10-16T11:21:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.952+0000 7fed29ffb700 1 -- 172.21.15.158:0/4187830746 <== mgr.12843 v2:172.21.15.158:6832/3125181137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290088 (secure 0 0 0) 0x7fed240095c0 con 0x7fed1806b140 2023-10-16T11:21:24.957 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:24.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fed1806b140 msgr2=0x7fed1806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fed1806b140 0x7fed1806d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed24009970 tx=0x7fed24009110 comp rx=0 tx=0).stop 2023-10-16T11:21:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 msgr2=0x7fed340c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed340c76d0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7fed340cd160 tx=0x7fed30004750 comp rx=0 tx=0).stop 2023-10-16T11:21:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 shutdown_connections 2023-10-16T11:21:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fed1806b140 0x7fed1806d600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed3415d7c0 0x7fed340c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed3406b300 0x7fed340b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 --2- 172.21.15.158:0/4187830746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed340ca5b0 0x7fed340c76d0 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 >> 172.21.15.158:0/4187830746 conn(0x7fed340cc4b0 msgr2=0x7fed340cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 shutdown_connections 2023-10-16T11:21:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:24.956+0000 7fed3b7b8700 1 -- 172.21.15.158:0/4187830746 wait complete. 2023-10-16T11:21:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:21:24.969 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T11:21:24.682563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":451076,"kb_used_data":26696,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754523644,"statfs":{"total":773094113280,"available":772632211456,"internally_reserved":0,"allocated":27336704,"data_stored":15271316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":434425399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.850015"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953392+0000","last_change":"2023-10-16T11:21:20.873354+0000","last_active":"2023-10-16T11:21:22.953392+0000","last_peered":"2023-10-16T11:21:22.953392+0000","last_clean":"2023-10-16T11:21:22.953392+0000","last_became_active":"2023-10-16T11:21:20.873059+0000","last_became_peered":"2023-10-16T11:21:20.873059+0000","last_unstale":"2023-10-16T11:21:22.953392+0000","last_undegraded":"2023-10-16T11:21:22.953392+0000","last_fullsized":"2023-10-16T11:21:22.953392+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:16:40.456737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953428+0000","last_change":"2023-10-16T11:21:20.878799+0000","last_active":"2023-10-16T11:21:22.953428+0000","last_peered":"2023-10-16T11:21:22.953428+0000","last_clean":"2023-10-16T11:21:22.953428+0000","last_became_active":"2023-10-16T11:21:20.878509+0000","last_became_peered":"2023-10-16T11:21:20.878509+0000","last_unstale":"2023-10-16T11:21:22.953428+0000","last_undegraded":"2023-10-16T11:21:22.953428+0000","last_fullsized":"2023-10-16T11:21:22.953428+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:14.231361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953628+0000","last_change":"2023-10-16T11:21:20.877276+0000","last_active":"2023-10-16T11:21:22.953628+0000","last_peered":"2023-10-16T11:21:22.953628+0000","last_clean":"2023-10-16T11:21:22.953628+0000","last_became_active":"2023-10-16T11:21:20.877002+0000","last_became_peered":"2023-10-16T11:21:20.877002+0000","last_unstale":"2023-10-16T11:21:22.953628+0000","last_undegraded":"2023-10-16T11:21:22.953628+0000","last_fullsized":"2023-10-16T11:21:22.953628+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:41.706764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953474+0000","last_change":"2023-10-16T11:21:20.863524+0000","last_active":"2023-10-16T11:21:22.953474+0000","last_peered":"2023-10-16T11:21:22.953474+0000","last_clean":"2023-10-16T11:21:22.953474+0000","last_became_active":"2023-10-16T11:21:20.863265+0000","last_became_peered":"2023-10-16T11:21:20.863265+0000","last_unstale":"2023-10-16T11:21:22.953474+0000","last_undegraded":"2023-10-16T11:21:22.953474+0000","last_fullsized":"2023-10-16T11:21:22.953474+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:44.964933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952776+0000","last_change":"2023-10-16T11:21:20.879735+0000","last_active":"2023-10-16T11:21:22.952776+0000","last_peered":"2023-10-16T11:21:22.952776+0000","last_clean":"2023-10-16T11:21:22.952776+0000","last_became_active":"2023-10-16T11:21:20.879472+0000","last_became_peered":"2023-10-16T11:21:20.879472+0000","last_unstale":"2023-10-16T11:21:22.952776+0000","last_undegraded":"2023-10-16T11:21:22.952776+0000","last_fullsized":"2023-10-16T11:21:22.952776+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:25.834081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952610+0000","last_change":"2023-10-16T11:21:20.867649+0000","last_active":"2023-10-16T11:21:22.952610+0000","last_peered":"2023-10-16T11:21:22.952610+0000","last_clean":"2023-10-16T11:21:22.952610+0000","last_became_active":"2023-10-16T11:21:20.867349+0000","last_became_peered":"2023-10-16T11:21:20.867349+0000","last_unstale":"2023-10-16T11:21:22.952610+0000","last_undegraded":"2023-10-16T11:21:22.952610+0000","last_fullsized":"2023-10-16T11:21:22.952610+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:21.764175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952926+0000","last_change":"2023-10-16T11:21:20.869639+0000","last_active":"2023-10-16T11:21:22.952926+0000","last_peered":"2023-10-16T11:21:22.952926+0000","last_clean":"2023-10-16T11:21:22.952926+0000","last_became_active":"2023-10-16T11:21:20.869374+0000","last_became_peered":"2023-10-16T11:21:20.869374+0000","last_unstale":"2023-10-16T11:21:22.952926+0000","last_undegraded":"2023-10-16T11:21:22.952926+0000","last_fullsized":"2023-10-16T11:21:22.952926+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:57.278022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952644+0000","last_change":"2023-10-16T11:21:20.867908+0000","last_active":"2023-10-16T11:21:22.952644+0000","last_peered":"2023-10-16T11:21:22.952644+0000","last_clean":"2023-10-16T11:21:22.952644+0000","last_became_active":"2023-10-16T11:21:20.867652+0000","last_became_peered":"2023-10-16T11:21:20.867652+0000","last_unstale":"2023-10-16T11:21:22.952644+0000","last_undegraded":"2023-10-16T11:21:22.952644+0000","last_fullsized":"2023-10-16T11:21:22.952644+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:57.667626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953340+0000","last_change":"2023-10-16T11:21:20.871129+0000","last_active":"2023-10-16T11:21:22.953340+0000","last_peered":"2023-10-16T11:21:22.953340+0000","last_clean":"2023-10-16T11:21:22.953340+0000","last_became_active":"2023-10-16T11:21:20.870868+0000","last_became_peered":"2023-10-16T11:21:20.870868+0000","last_unstale":"2023-10-16T11:21:22.953340+0000","last_undegraded":"2023-10-16T11:21:22.953340+0000","last_fullsized":"2023-10-16T11:21:22.953340+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:31.143865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953378+0000","last_change":"2023-10-16T11:21:20.872430+0000","last_active":"2023-10-16T11:21:22.953378+0000","last_peered":"2023-10-16T11:21:22.953378+0000","last_clean":"2023-10-16T11:21:22.953378+0000","last_became_active":"2023-10-16T11:21:20.872135+0000","last_became_peered":"2023-10-16T11:21:20.872135+0000","last_unstale":"2023-10-16T11:21:22.953378+0000","last_undegraded":"2023-10-16T11:21:22.953378+0000","last_fullsized":"2023-10-16T11:21:22.953378+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:36.782718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857284+0000","last_change":"2023-10-16T11:21:20.876425+0000","last_active":"2023-10-16T11:21:21.857284+0000","last_peered":"2023-10-16T11:21:21.857284+0000","last_clean":"2023-10-16T11:21:21.857284+0000","last_became_active":"2023-10-16T11:21:20.876167+0000","last_became_peered":"2023-10-16T11:21:20.876167+0000","last_unstale":"2023-10-16T11:21:21.857284+0000","last_undegraded":"2023-10-16T11:21:21.857284+0000","last_fullsized":"2023-10-16T11:21:21.857284+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:29.298975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953552+0000","last_change":"2023-10-16T11:21:20.871087+0000","last_active":"2023-10-16T11:21:22.953552+0000","last_peered":"2023-10-16T11:21:22.953552+0000","last_clean":"2023-10-16T11:21:22.953552+0000","last_became_active":"2023-10-16T11:21:20.870664+0000","last_became_peered":"2023-10-16T11:21:20.870664+0000","last_unstale":"2023-10-16T11:21:22.953552+0000","last_undegraded":"2023-10-16T11:21:22.953552+0000","last_fullsized":"2023-10-16T11:21:22.953552+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:10:10.791379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953348+0000","last_change":"2023-10-16T11:21:20.874457+0000","last_active":"2023-10-16T11:21:22.953348+0000","last_peered":"2023-10-16T11:21:22.953348+0000","last_clean":"2023-10-16T11:21:22.953348+0000","last_became_active":"2023-10-16T11:21:20.874188+0000","last_became_peered":"2023-10-16T11:21:20.874188+0000","last_unstale":"2023-10-16T11:21:22.953348+0000","last_undegraded":"2023-10-16T11:21:22.953348+0000","last_fullsized":"2023-10-16T11:21:22.953348+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:39.661386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953105+0000","last_change":"2023-10-16T11:21:20.870673+0000","last_active":"2023-10-16T11:21:22.953105+0000","last_peered":"2023-10-16T11:21:22.953105+0000","last_clean":"2023-10-16T11:21:22.953105+0000","last_became_active":"2023-10-16T11:21:20.870401+0000","last_became_peered":"2023-10-16T11:21:20.870401+0000","last_unstale":"2023-10-16T11:21:22.953105+0000","last_undegraded":"2023-10-16T11:21:22.953105+0000","last_fullsized":"2023-10-16T11:21:22.953105+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:51:15.955800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953064+0000","last_change":"2023-10-16T11:21:20.862779+0000","last_active":"2023-10-16T11:21:22.953064+0000","last_peered":"2023-10-16T11:21:22.953064+0000","last_clean":"2023-10-16T11:21:22.953064+0000","last_became_active":"2023-10-16T11:21:20.862507+0000","last_became_peered":"2023-10-16T11:21:20.862507+0000","last_unstale":"2023-10-16T11:21:22.953064+0000","last_undegraded":"2023-10-16T11:21:22.953064+0000","last_fullsized":"2023-10-16T11:21:22.953064+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:10.195225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857516+0000","last_change":"2023-10-16T11:21:20.874345+0000","last_active":"2023-10-16T11:21:21.857516+0000","last_peered":"2023-10-16T11:21:21.857516+0000","last_clean":"2023-10-16T11:21:21.857516+0000","last_became_active":"2023-10-16T11:21:20.874042+0000","last_became_peered":"2023-10-16T11:21:20.874042+0000","last_unstale":"2023-10-16T11:21:21.857516+0000","last_undegraded":"2023-10-16T11:21:21.857516+0000","last_fullsized":"2023-10-16T11:21:21.857516+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:45:42.867811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858220+0000","last_change":"2023-10-16T11:21:20.864051+0000","last_active":"2023-10-16T11:21:21.858220+0000","last_peered":"2023-10-16T11:21:21.858220+0000","last_clean":"2023-10-16T11:21:21.858220+0000","last_became_active":"2023-10-16T11:21:20.863769+0000","last_became_peered":"2023-10-16T11:21:20.863769+0000","last_unstale":"2023-10-16T11:21:21.858220+0000","last_undegraded":"2023-10-16T11:21:21.858220+0000","last_fullsized":"2023-10-16T11:21:21.858220+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:00.719281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953637+0000","last_change":"2023-10-16T11:21:20.871882+0000","last_active":"2023-10-16T11:21:22.953637+0000","last_peered":"2023-10-16T11:21:22.953637+0000","last_clean":"2023-10-16T11:21:22.953637+0000","last_became_active":"2023-10-16T11:21:20.871592+0000","last_became_peered":"2023-10-16T11:21:20.871592+0000","last_unstale":"2023-10-16T11:21:22.953637+0000","last_undegraded":"2023-10-16T11:21:22.953637+0000","last_fullsized":"2023-10-16T11:21:22.953637+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:39.347652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858184+0000","last_change":"2023-10-16T11:21:20.869240+0000","last_active":"2023-10-16T11:21:21.858184+0000","last_peered":"2023-10-16T11:21:21.858184+0000","last_clean":"2023-10-16T11:21:21.858184+0000","last_became_active":"2023-10-16T11:21:20.868972+0000","last_became_peered":"2023-10-16T11:21:20.868972+0000","last_unstale":"2023-10-16T11:21:21.858184+0000","last_undegraded":"2023-10-16T11:21:21.858184+0000","last_fullsized":"2023-10-16T11:21:21.858184+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:27.953134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858329+0000","last_change":"2023-10-16T11:21:20.875009+0000","last_active":"2023-10-16T11:21:21.858329+0000","last_peered":"2023-10-16T11:21:21.858329+0000","last_clean":"2023-10-16T11:21:21.858329+0000","last_became_active":"2023-10-16T11:21:20.874720+0000","last_became_peered":"2023-10-16T11:21:20.874720+0000","last_unstale":"2023-10-16T11:21:21.858329+0000","last_undegraded":"2023-10-16T11:21:21.858329+0000","last_fullsized":"2023-10-16T11:21:21.858329+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:39.063930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857087+0000","last_change":"2023-10-16T11:21:20.869041+0000","last_active":"2023-10-16T11:21:21.857087+0000","last_peered":"2023-10-16T11:21:21.857087+0000","last_clean":"2023-10-16T11:21:21.857087+0000","last_became_active":"2023-10-16T11:21:20.868785+0000","last_became_peered":"2023-10-16T11:21:20.868785+0000","last_unstale":"2023-10-16T11:21:21.857087+0000","last_undegraded":"2023-10-16T11:21:21.857087+0000","last_fullsized":"2023-10-16T11:21:21.857087+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:02.049634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856675+0000","last_change":"2023-10-16T11:21:20.869713+0000","last_active":"2023-10-16T11:21:21.856675+0000","last_peered":"2023-10-16T11:21:21.856675+0000","last_clean":"2023-10-16T11:21:21.856675+0000","last_became_active":"2023-10-16T11:21:20.869088+0000","last_became_peered":"2023-10-16T11:21:20.869088+0000","last_unstale":"2023-10-16T11:21:21.856675+0000","last_undegraded":"2023-10-16T11:21:21.856675+0000","last_fullsized":"2023-10-16T11:21:21.856675+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:40.650230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953414+0000","last_change":"2023-10-16T11:21:20.867354+0000","last_active":"2023-10-16T11:21:22.953414+0000","last_peered":"2023-10-16T11:21:22.953414+0000","last_clean":"2023-10-16T11:21:22.953414+0000","last_became_active":"2023-10-16T11:21:20.867105+0000","last_became_peered":"2023-10-16T11:21:20.867105+0000","last_unstale":"2023-10-16T11:21:22.953414+0000","last_undegraded":"2023-10-16T11:21:22.953414+0000","last_fullsized":"2023-10-16T11:21:22.953414+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:19.234265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856031+0000","last_change":"2023-10-16T11:21:20.874563+0000","last_active":"2023-10-16T11:21:21.856031+0000","last_peered":"2023-10-16T11:21:21.856031+0000","last_clean":"2023-10-16T11:21:21.856031+0000","last_became_active":"2023-10-16T11:21:20.874357+0000","last_became_peered":"2023-10-16T11:21:20.874357+0000","last_unstale":"2023-10-16T11:21:21.856031+0000","last_undegraded":"2023-10-16T11:21:21.856031+0000","last_fullsized":"2023-10-16T11:21:21.856031+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:52.659355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953611+0000","last_change":"2023-10-16T11:21:20.869158+0000","last_active":"2023-10-16T11:21:22.953611+0000","last_peered":"2023-10-16T11:21:22.953611+0000","last_clean":"2023-10-16T11:21:22.953611+0000","last_became_active":"2023-10-16T11:21:20.868856+0000","last_became_peered":"2023-10-16T11:21:20.868856+0000","last_unstale":"2023-10-16T11:21:22.953611+0000","last_undegraded":"2023-10-16T11:21:22.953611+0000","last_fullsized":"2023-10-16T11:21:22.953611+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:48.577732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952644+0000","last_change":"2023-10-16T11:21:20.873998+0000","last_active":"2023-10-16T11:21:22.952644+0000","last_peered":"2023-10-16T11:21:22.952644+0000","last_clean":"2023-10-16T11:21:22.952644+0000","last_became_active":"2023-10-16T11:21:20.873713+0000","last_became_peered":"2023-10-16T11:21:20.873713+0000","last_unstale":"2023-10-16T11:21:22.952644+0000","last_undegraded":"2023-10-16T11:21:22.952644+0000","last_fullsized":"2023-10-16T11:21:22.952644+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:08.659912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952677+0000","last_change":"2023-10-16T11:21:20.879227+0000","last_active":"2023-10-16T11:21:22.952677+0000","last_peered":"2023-10-16T11:21:22.952677+0000","last_clean":"2023-10-16T11:21:22.952677+0000","last_became_active":"2023-10-16T11:21:20.878922+0000","last_became_peered":"2023-10-16T11:21:20.878922+0000","last_unstale":"2023-10-16T11:21:22.952677+0000","last_undegraded":"2023-10-16T11:21:22.952677+0000","last_fullsized":"2023-10-16T11:21:22.952677+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:23.417317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856448+0000","last_change":"2023-10-16T11:21:20.869603+0000","last_active":"2023-10-16T11:21:21.856448+0000","last_peered":"2023-10-16T11:21:21.856448+0000","last_clean":"2023-10-16T11:21:21.856448+0000","last_became_active":"2023-10-16T11:21:20.869231+0000","last_became_peered":"2023-10-16T11:21:20.869231+0000","last_unstale":"2023-10-16T11:21:21.856448+0000","last_undegraded":"2023-10-16T11:21:21.856448+0000","last_fullsized":"2023-10-16T11:21:21.856448+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:27.477587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.412439+0000","last_change":"2023-10-16T11:21:20.874243+0000","last_active":"2023-10-16T11:21:22.412439+0000","last_peered":"2023-10-16T11:21:22.412439+0000","last_clean":"2023-10-16T11:21:22.412439+0000","last_became_active":"2023-10-16T11:21:20.873980+0000","last_became_peered":"2023-10-16T11:21:20.873980+0000","last_unstale":"2023-10-16T11:21:22.412439+0000","last_undegraded":"2023-10-16T11:21:22.412439+0000","last_fullsized":"2023-10-16T11:21:22.412439+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:36.014320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856230+0000","last_change":"2023-10-16T11:21:20.869783+0000","last_active":"2023-10-16T11:21:21.856230+0000","last_peered":"2023-10-16T11:21:21.856230+0000","last_clean":"2023-10-16T11:21:21.856230+0000","last_became_active":"2023-10-16T11:21:20.869508+0000","last_became_peered":"2023-10-16T11:21:20.869508+0000","last_unstale":"2023-10-16T11:21:21.856230+0000","last_undegraded":"2023-10-16T11:21:21.856230+0000","last_fullsized":"2023-10-16T11:21:21.856230+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:32.527753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952781+0000","last_change":"2023-10-16T11:21:20.875868+0000","last_active":"2023-10-16T11:21:22.952781+0000","last_peered":"2023-10-16T11:21:22.952781+0000","last_clean":"2023-10-16T11:21:22.952781+0000","last_became_active":"2023-10-16T11:21:20.875159+0000","last_became_peered":"2023-10-16T11:21:20.875159+0000","last_unstale":"2023-10-16T11:21:22.952781+0000","last_undegraded":"2023-10-16T11:21:22.952781+0000","last_fullsized":"2023-10-16T11:21:22.952781+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:42.829245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856106+0000","last_change":"2023-10-16T11:21:20.873972+0000","last_active":"2023-10-16T11:21:21.856106+0000","last_peered":"2023-10-16T11:21:21.856106+0000","last_clean":"2023-10-16T11:21:21.856106+0000","last_became_active":"2023-10-16T11:21:20.873285+0000","last_became_peered":"2023-10-16T11:21:20.873285+0000","last_unstale":"2023-10-16T11:21:21.856106+0000","last_undegraded":"2023-10-16T11:21:21.856106+0000","last_fullsized":"2023-10-16T11:21:21.856106+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:23.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"256'4","reported_seq":16,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.885240+0000","last_change":"2023-10-16T11:21:19.853470+0000","last_active":"2023-10-16T11:21:20.885240+0000","last_peered":"2023-10-16T11:21:20.885240+0000","last_clean":"2023-10-16T11:21:20.885240+0000","last_became_active":"2023-10-16T11:21:19.853198+0000","last_became_peered":"2023-10-16T11:21:19.853198+0000","last_unstale":"2023-10-16T11:21:20.885240+0000","last_undegraded":"2023-10-16T11:21:20.885240+0000","last_fullsized":"2023-10-16T11:21:20.885240+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:35.615415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953472+0000","last_change":"2023-10-16T11:21:20.872590+0000","last_active":"2023-10-16T11:21:22.953472+0000","last_peered":"2023-10-16T11:21:22.953472+0000","last_clean":"2023-10-16T11:21:22.953472+0000","last_became_active":"2023-10-16T11:21:20.872308+0000","last_became_peered":"2023-10-16T11:21:20.872308+0000","last_unstale":"2023-10-16T11:21:22.953472+0000","last_undegraded":"2023-10-16T11:21:22.953472+0000","last_fullsized":"2023-10-16T11:21:22.953472+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:03.714962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882397+0000","last_change":"2023-10-16T11:21:19.855092+0000","last_active":"2023-10-16T11:21:20.882397+0000","last_peered":"2023-10-16T11:21:20.882397+0000","last_clean":"2023-10-16T11:21:20.882397+0000","last_became_active":"2023-10-16T11:21:19.854771+0000","last_became_peered":"2023-10-16T11:21:19.854771+0000","last_unstale":"2023-10-16T11:21:20.882397+0000","last_undegraded":"2023-10-16T11:21:20.882397+0000","last_fullsized":"2023-10-16T11:21:20.882397+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:23.617910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953664+0000","last_change":"2023-10-16T11:21:20.877402+0000","last_active":"2023-10-16T11:21:22.953664+0000","last_peered":"2023-10-16T11:21:22.953664+0000","last_clean":"2023-10-16T11:21:22.953664+0000","last_became_active":"2023-10-16T11:21:20.877109+0000","last_became_peered":"2023-10-16T11:21:20.877109+0000","last_unstale":"2023-10-16T11:21:22.953664+0000","last_undegraded":"2023-10-16T11:21:22.953664+0000","last_fullsized":"2023-10-16T11:21:22.953664+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:14.131040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883470+0000","last_change":"2023-10-16T11:21:19.854161+0000","last_active":"2023-10-16T11:21:20.883470+0000","last_peered":"2023-10-16T11:21:20.883470+0000","last_clean":"2023-10-16T11:21:20.883470+0000","last_became_active":"2023-10-16T11:21:19.853159+0000","last_became_peered":"2023-10-16T11:21:19.853159+0000","last_unstale":"2023-10-16T11:21:20.883470+0000","last_undegraded":"2023-10-16T11:21:20.883470+0000","last_fullsized":"2023-10-16T11:21:20.883470+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:29.519798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953024+0000","last_change":"2023-10-16T11:21:20.870583+0000","last_active":"2023-10-16T11:21:22.953024+0000","last_peered":"2023-10-16T11:21:22.953024+0000","last_clean":"2023-10-16T11:21:22.953024+0000","last_became_active":"2023-10-16T11:21:20.870295+0000","last_became_peered":"2023-10-16T11:21:20.870295+0000","last_unstale":"2023-10-16T11:21:22.953024+0000","last_undegraded":"2023-10-16T11:21:22.953024+0000","last_fullsized":"2023-10-16T11:21:22.953024+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:27.477587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883433+0000","last_change":"2023-10-16T11:21:19.853555+0000","last_active":"2023-10-16T11:21:20.883433+0000","last_peered":"2023-10-16T11:21:20.883433+0000","last_clean":"2023-10-16T11:21:20.883433+0000","last_became_active":"2023-10-16T11:21:19.853179+0000","last_became_peered":"2023-10-16T11:21:19.853179+0000","last_unstale":"2023-10-16T11:21:20.883433+0000","last_undegraded":"2023-10-16T11:21:20.883433+0000","last_fullsized":"2023-10-16T11:21:20.883433+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:56.625722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953381+0000","last_change":"2023-10-16T11:21:20.876740+0000","last_active":"2023-10-16T11:21:22.953381+0000","last_peered":"2023-10-16T11:21:22.953381+0000","last_clean":"2023-10-16T11:21:22.953381+0000","last_became_active":"2023-10-16T11:21:20.876264+0000","last_became_peered":"2023-10-16T11:21:20.876264+0000","last_unstale":"2023-10-16T11:21:22.953381+0000","last_undegraded":"2023-10-16T11:21:22.953381+0000","last_fullsized":"2023-10-16T11:21:22.953381+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:59.922507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883454+0000","last_change":"2023-10-16T11:21:19.857352+0000","last_active":"2023-10-16T11:21:20.883454+0000","last_peered":"2023-10-16T11:21:20.883454+0000","last_clean":"2023-10-16T11:21:20.883454+0000","last_became_active":"2023-10-16T11:21:19.857078+0000","last_became_peered":"2023-10-16T11:21:19.857078+0000","last_unstale":"2023-10-16T11:21:20.883454+0000","last_undegraded":"2023-10-16T11:21:20.883454+0000","last_fullsized":"2023-10-16T11:21:20.883454+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:06.084643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856199+0000","last_change":"2023-10-16T11:21:20.876206+0000","last_active":"2023-10-16T11:21:21.856199+0000","last_peered":"2023-10-16T11:21:21.856199+0000","last_clean":"2023-10-16T11:21:21.856199+0000","last_became_active":"2023-10-16T11:21:20.875903+0000","last_became_peered":"2023-10-16T11:21:20.875903+0000","last_unstale":"2023-10-16T11:21:21.856199+0000","last_undegraded":"2023-10-16T11:21:21.856199+0000","last_fullsized":"2023-10-16T11:21:21.856199+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:26.577607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882006+0000","last_change":"2023-10-16T11:21:19.848262+0000","last_active":"2023-10-16T11:21:20.882006+0000","last_peered":"2023-10-16T11:21:20.882006+0000","last_clean":"2023-10-16T11:21:20.882006+0000","last_became_active":"2023-10-16T11:21:19.847989+0000","last_became_peered":"2023-10-16T11:21:19.847989+0000","last_unstale":"2023-10-16T11:21:20.882006+0000","last_undegraded":"2023-10-16T11:21:20.882006+0000","last_fullsized":"2023-10-16T11:21:20.882006+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:37.505082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856231+0000","last_change":"2023-10-16T11:21:20.875941+0000","last_active":"2023-10-16T11:21:21.856231+0000","last_peered":"2023-10-16T11:21:21.856231+0000","last_clean":"2023-10-16T11:21:21.856231+0000","last_became_active":"2023-10-16T11:21:20.874949+0000","last_became_peered":"2023-10-16T11:21:20.874949+0000","last_unstale":"2023-10-16T11:21:21.856231+0000","last_undegraded":"2023-10-16T11:21:21.856231+0000","last_fullsized":"2023-10-16T11:21:21.856231+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:24.359753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863017+0000","last_change":"2023-10-16T11:21:19.854706+0000","last_active":"2023-10-16T11:21:20.863017+0000","last_peered":"2023-10-16T11:21:20.863017+0000","last_clean":"2023-10-16T11:21:20.863017+0000","last_became_active":"2023-10-16T11:21:19.854183+0000","last_became_peered":"2023-10-16T11:21:19.854183+0000","last_unstale":"2023-10-16T11:21:20.863017+0000","last_undegraded":"2023-10-16T11:21:20.863017+0000","last_fullsized":"2023-10-16T11:21:20.863017+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:39.654182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858431+0000","last_change":"2023-10-16T11:21:20.864001+0000","last_active":"2023-10-16T11:21:21.858431+0000","last_peered":"2023-10-16T11:21:21.858431+0000","last_clean":"2023-10-16T11:21:21.858431+0000","last_became_active":"2023-10-16T11:21:20.863710+0000","last_became_peered":"2023-10-16T11:21:20.863710+0000","last_unstale":"2023-10-16T11:21:21.858431+0000","last_undegraded":"2023-10-16T11:21:21.858431+0000","last_fullsized":"2023-10-16T11:21:21.858431+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:42.568368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.856666+0000","last_change":"2023-10-16T11:21:19.854075+0000","last_active":"2023-10-16T11:21:20.856666+0000","last_peered":"2023-10-16T11:21:20.856666+0000","last_clean":"2023-10-16T11:21:20.856666+0000","last_became_active":"2023-10-16T11:21:19.853748+0000","last_became_peered":"2023-10-16T11:21:19.853748+0000","last_unstale":"2023-10-16T11:21:20.856666+0000","last_undegraded":"2023-10-16T11:21:20.856666+0000","last_fullsized":"2023-10-16T11:21:20.856666+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:32.125504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952922+0000","last_change":"2023-10-16T11:21:20.876041+0000","last_active":"2023-10-16T11:21:22.952922+0000","last_peered":"2023-10-16T11:21:22.952922+0000","last_clean":"2023-10-16T11:21:22.952922+0000","last_became_active":"2023-10-16T11:21:20.875530+0000","last_became_peered":"2023-10-16T11:21:20.875530+0000","last_unstale":"2023-10-16T11:21:22.952922+0000","last_undegraded":"2023-10-16T11:21:22.952922+0000","last_fullsized":"2023-10-16T11:21:22.952922+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:37.610269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.858487+0000","last_change":"2023-10-16T11:21:19.855733+0000","last_active":"2023-10-16T11:21:20.858487+0000","last_peered":"2023-10-16T11:21:20.858487+0000","last_clean":"2023-10-16T11:21:20.858487+0000","last_became_active":"2023-10-16T11:21:19.855458+0000","last_became_peered":"2023-10-16T11:21:19.855458+0000","last_unstale":"2023-10-16T11:21:20.858487+0000","last_undegraded":"2023-10-16T11:21:20.858487+0000","last_fullsized":"2023-10-16T11:21:20.858487+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:12.707047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953620+0000","last_change":"2023-10-16T11:21:20.873318+0000","last_active":"2023-10-16T11:21:22.953620+0000","last_peered":"2023-10-16T11:21:22.953620+0000","last_clean":"2023-10-16T11:21:22.953620+0000","last_became_active":"2023-10-16T11:21:20.873009+0000","last_became_peered":"2023-10-16T11:21:20.873009+0000","last_unstale":"2023-10-16T11:21:22.953620+0000","last_undegraded":"2023-10-16T11:21:22.953620+0000","last_fullsized":"2023-10-16T11:21:22.953620+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:34.913555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863998+0000","last_change":"2023-10-16T11:21:19.854559+0000","last_active":"2023-10-16T11:21:20.863998+0000","last_peered":"2023-10-16T11:21:20.863998+0000","last_clean":"2023-10-16T11:21:20.863998+0000","last_became_active":"2023-10-16T11:21:19.854175+0000","last_became_peered":"2023-10-16T11:21:19.854175+0000","last_unstale":"2023-10-16T11:21:20.863998+0000","last_undegraded":"2023-10-16T11:21:20.863998+0000","last_fullsized":"2023-10-16T11:21:20.863998+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:20:47.814076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953153+0000","last_change":"2023-10-16T11:21:20.860947+0000","last_active":"2023-10-16T11:21:22.953153+0000","last_peered":"2023-10-16T11:21:22.953153+0000","last_clean":"2023-10-16T11:21:22.953153+0000","last_became_active":"2023-10-16T11:21:20.860550+0000","last_became_peered":"2023-10-16T11:21:20.860550+0000","last_unstale":"2023-10-16T11:21:22.953153+0000","last_undegraded":"2023-10-16T11:21:22.953153+0000","last_fullsized":"2023-10-16T11:21:22.953153+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:47.084611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"256'3","reported_seq":15,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883631+0000","last_change":"2023-10-16T11:21:19.852566+0000","last_active":"2023-10-16T11:21:20.883631+0000","last_peered":"2023-10-16T11:21:20.883631+0000","last_clean":"2023-10-16T11:21:20.883631+0000","last_became_active":"2023-10-16T11:21:19.852230+0000","last_became_peered":"2023-10-16T11:21:19.852230+0000","last_unstale":"2023-10-16T11:21:20.883631+0000","last_undegraded":"2023-10-16T11:21:20.883631+0000","last_fullsized":"2023-10-16T11:21:20.883631+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:27.802569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952933+0000","last_change":"2023-10-16T11:21:20.879825+0000","last_active":"2023-10-16T11:21:22.952933+0000","last_peered":"2023-10-16T11:21:22.952933+0000","last_clean":"2023-10-16T11:21:22.952933+0000","last_became_active":"2023-10-16T11:21:20.879516+0000","last_became_peered":"2023-10-16T11:21:20.879516+0000","last_unstale":"2023-10-16T11:21:22.952933+0000","last_undegraded":"2023-10-16T11:21:22.952933+0000","last_fullsized":"2023-10-16T11:21:22.952933+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:36.014320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"256'6","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.888143+0000","last_change":"2023-10-16T11:21:19.852735+0000","last_active":"2023-10-16T11:21:20.888143+0000","last_peered":"2023-10-16T11:21:20.888143+0000","last_clean":"2023-10-16T11:21:20.888143+0000","last_became_active":"2023-10-16T11:21:19.852491+0000","last_became_peered":"2023-10-16T11:21:19.852491+0000","last_unstale":"2023-10-16T11:21:20.888143+0000","last_undegraded":"2023-10-16T11:21:20.888143+0000","last_fullsized":"2023-10-16T11:21:20.888143+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:25.572752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953659+0000","last_change":"2023-10-16T11:21:20.870915+0000","last_active":"2023-10-16T11:21:22.953659+0000","last_peered":"2023-10-16T11:21:22.953659+0000","last_clean":"2023-10-16T11:21:22.953659+0000","last_became_active":"2023-10-16T11:21:20.870368+0000","last_became_peered":"2023-10-16T11:21:20.870368+0000","last_unstale":"2023-10-16T11:21:22.953659+0000","last_undegraded":"2023-10-16T11:21:22.953659+0000","last_fullsized":"2023-10-16T11:21:22.953659+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:33.243878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.884015+0000","last_change":"2023-10-16T11:21:19.856454+0000","last_active":"2023-10-16T11:21:20.884015+0000","last_peered":"2023-10-16T11:21:20.884015+0000","last_clean":"2023-10-16T11:21:20.884015+0000","last_became_active":"2023-10-16T11:21:19.856175+0000","last_became_peered":"2023-10-16T11:21:19.856175+0000","last_unstale":"2023-10-16T11:21:20.884015+0000","last_undegraded":"2023-10-16T11:21:20.884015+0000","last_fullsized":"2023-10-16T11:21:20.884015+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:20.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953611+0000","last_change":"2023-10-16T11:21:20.871038+0000","last_active":"2023-10-16T11:21:22.953611+0000","last_peered":"2023-10-16T11:21:22.953611+0000","last_clean":"2023-10-16T11:21:22.953611+0000","last_became_active":"2023-10-16T11:21:20.870582+0000","last_became_peered":"2023-10-16T11:21:20.870582+0000","last_unstale":"2023-10-16T11:21:22.953611+0000","last_undegraded":"2023-10-16T11:21:22.953611+0000","last_fullsized":"2023-10-16T11:21:22.953611+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:04.917463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.866777+0000","last_change":"2023-10-16T11:21:19.858731+0000","last_active":"2023-10-16T11:21:20.866777+0000","last_peered":"2023-10-16T11:21:20.866777+0000","last_clean":"2023-10-16T11:21:20.866777+0000","last_became_active":"2023-10-16T11:21:19.858433+0000","last_became_peered":"2023-10-16T11:21:19.858433+0000","last_unstale":"2023-10-16T11:21:20.866777+0000","last_undegraded":"2023-10-16T11:21:20.866777+0000","last_fullsized":"2023-10-16T11:21:20.866777+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:45.296228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858059+0000","last_change":"2023-10-16T11:21:20.864542+0000","last_active":"2023-10-16T11:21:21.858059+0000","last_peered":"2023-10-16T11:21:21.858059+0000","last_clean":"2023-10-16T11:21:21.858059+0000","last_became_active":"2023-10-16T11:21:20.864117+0000","last_became_peered":"2023-10-16T11:21:20.864117+0000","last_unstale":"2023-10-16T11:21:21.858059+0000","last_undegraded":"2023-10-16T11:21:21.858059+0000","last_fullsized":"2023-10-16T11:21:21.858059+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:38.844676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.868419+0000","last_change":"2023-10-16T11:21:19.855209+0000","last_active":"2023-10-16T11:21:20.868419+0000","last_peered":"2023-10-16T11:21:20.868419+0000","last_clean":"2023-10-16T11:21:20.868419+0000","last_became_active":"2023-10-16T11:21:19.853190+0000","last_became_peered":"2023-10-16T11:21:19.853190+0000","last_unstale":"2023-10-16T11:21:20.868419+0000","last_undegraded":"2023-10-16T11:21:20.868419+0000","last_fullsized":"2023-10-16T11:21:20.868419+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:16.416230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858408+0000","last_change":"2023-10-16T11:21:20.860460+0000","last_active":"2023-10-16T11:21:21.858408+0000","last_peered":"2023-10-16T11:21:21.858408+0000","last_clean":"2023-10-16T11:21:21.858408+0000","last_became_active":"2023-10-16T11:21:20.860108+0000","last_became_peered":"2023-10-16T11:21:20.860108+0000","last_unstale":"2023-10-16T11:21:21.858408+0000","last_undegraded":"2023-10-16T11:21:21.858408+0000","last_fullsized":"2023-10-16T11:21:21.858408+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:22.973532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"256'8","reported_seq":20,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.887160+0000","last_change":"2023-10-16T11:21:19.855640+0000","last_active":"2023-10-16T11:21:20.887160+0000","last_peered":"2023-10-16T11:21:20.887160+0000","last_clean":"2023-10-16T11:21:20.887160+0000","last_became_active":"2023-10-16T11:21:19.854814+0000","last_became_peered":"2023-10-16T11:21:19.854814+0000","last_unstale":"2023-10-16T11:21:20.887160+0000","last_undegraded":"2023-10-16T11:21:20.887160+0000","last_fullsized":"2023-10-16T11:21:20.887160+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:58.546367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952411+0000","last_change":"2023-10-16T11:21:20.868828+0000","last_active":"2023-10-16T11:21:22.952411+0000","last_peered":"2023-10-16T11:21:22.952411+0000","last_clean":"2023-10-16T11:21:22.952411+0000","last_became_active":"2023-10-16T11:21:20.868519+0000","last_became_peered":"2023-10-16T11:21:20.868519+0000","last_unstale":"2023-10-16T11:21:22.952411+0000","last_undegraded":"2023-10-16T11:21:22.952411+0000","last_fullsized":"2023-10-16T11:21:22.952411+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:05.218119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882022+0000","last_change":"2023-10-16T11:21:19.856057+0000","last_active":"2023-10-16T11:21:20.882022+0000","last_peered":"2023-10-16T11:21:20.882022+0000","last_clean":"2023-10-16T11:21:20.882022+0000","last_became_active":"2023-10-16T11:21:19.855623+0000","last_became_peered":"2023-10-16T11:21:19.855623+0000","last_unstale":"2023-10-16T11:21:20.882022+0000","last_undegraded":"2023-10-16T11:21:20.882022+0000","last_fullsized":"2023-10-16T11:21:20.882022+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:07.905060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856513+0000","last_change":"2023-10-16T11:21:20.873473+0000","last_active":"2023-10-16T11:21:21.856513+0000","last_peered":"2023-10-16T11:21:21.856513+0000","last_clean":"2023-10-16T11:21:21.856513+0000","last_became_active":"2023-10-16T11:21:20.873205+0000","last_became_peered":"2023-10-16T11:21:20.873205+0000","last_unstale":"2023-10-16T11:21:21.856513+0000","last_undegraded":"2023-10-16T11:21:21.856513+0000","last_fullsized":"2023-10-16T11:21:21.856513+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:51:15.955800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882535+0000","last_change":"2023-10-16T11:21:19.857447+0000","last_active":"2023-10-16T11:21:20.882535+0000","last_peered":"2023-10-16T11:21:20.882535+0000","last_clean":"2023-10-16T11:21:20.882535+0000","last_became_active":"2023-10-16T11:21:19.857115+0000","last_became_peered":"2023-10-16T11:21:19.857115+0000","last_unstale":"2023-10-16T11:21:20.882535+0000","last_undegraded":"2023-10-16T11:21:20.882535+0000","last_fullsized":"2023-10-16T11:21:20.882535+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:20:47.814076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952731+0000","last_change":"2023-10-16T11:21:20.871844+0000","last_active":"2023-10-16T11:21:22.952731+0000","last_peered":"2023-10-16T11:21:22.952731+0000","last_clean":"2023-10-16T11:21:22.952731+0000","last_became_active":"2023-10-16T11:21:20.871521+0000","last_became_peered":"2023-10-16T11:21:20.871521+0000","last_unstale":"2023-10-16T11:21:22.952731+0000","last_undegraded":"2023-10-16T11:21:22.952731+0000","last_fullsized":"2023-10-16T11:21:22.952731+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:06.489484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883580+0000","last_change":"2023-10-16T11:21:19.854671+0000","last_active":"2023-10-16T11:21:20.883580+0000","last_peered":"2023-10-16T11:21:20.883580+0000","last_clean":"2023-10-16T11:21:20.883580+0000","last_became_active":"2023-10-16T11:21:19.854334+0000","last_became_peered":"2023-10-16T11:21:19.854334+0000","last_unstale":"2023-10-16T11:21:20.883580+0000","last_undegraded":"2023-10-16T11:21:20.883580+0000","last_fullsized":"2023-10-16T11:21:20.883580+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:50.695229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953386+0000","last_change":"2023-10-16T11:21:20.869590+0000","last_active":"2023-10-16T11:21:22.953386+0000","last_peered":"2023-10-16T11:21:22.953386+0000","last_clean":"2023-10-16T11:21:22.953386+0000","last_became_active":"2023-10-16T11:21:20.869076+0000","last_became_peered":"2023-10-16T11:21:20.869076+0000","last_unstale":"2023-10-16T11:21:22.953386+0000","last_undegraded":"2023-10-16T11:21:22.953386+0000","last_fullsized":"2023-10-16T11:21:22.953386+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:15.463518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863025+0000","last_change":"2023-10-16T11:21:19.859330+0000","last_active":"2023-10-16T11:21:20.863025+0000","last_peered":"2023-10-16T11:21:20.863025+0000","last_clean":"2023-10-16T11:21:20.863025+0000","last_became_active":"2023-10-16T11:21:19.859011+0000","last_became_peered":"2023-10-16T11:21:19.859011+0000","last_unstale":"2023-10-16T11:21:20.863025+0000","last_undegraded":"2023-10-16T11:21:20.863025+0000","last_fullsized":"2023-10-16T11:21:20.863025+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:41.892036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856451+0000","last_change":"2023-10-16T11:21:20.873499+0000","last_active":"2023-10-16T11:21:21.856451+0000","last_peered":"2023-10-16T11:21:21.856451+0000","last_clean":"2023-10-16T11:21:21.856451+0000","last_became_active":"2023-10-16T11:21:20.873180+0000","last_became_peered":"2023-10-16T11:21:20.873180+0000","last_unstale":"2023-10-16T11:21:21.856451+0000","last_undegraded":"2023-10-16T11:21:21.856451+0000","last_fullsized":"2023-10-16T11:21:21.856451+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:19.234265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.884623+0000","last_change":"2023-10-16T11:21:19.858156+0000","last_active":"2023-10-16T11:21:20.884623+0000","last_peered":"2023-10-16T11:21:20.884623+0000","last_clean":"2023-10-16T11:21:20.884623+0000","last_became_active":"2023-10-16T11:21:19.857859+0000","last_became_peered":"2023-10-16T11:21:19.857859+0000","last_unstale":"2023-10-16T11:21:20.884623+0000","last_undegraded":"2023-10-16T11:21:20.884623+0000","last_fullsized":"2023-10-16T11:21:20.884623+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:22.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856345+0000","last_change":"2023-10-16T11:21:20.867387+0000","last_active":"2023-10-16T11:21:21.856345+0000","last_peered":"2023-10-16T11:21:21.856345+0000","last_clean":"2023-10-16T11:21:21.856345+0000","last_became_active":"2023-10-16T11:21:20.867151+0000","last_became_peered":"2023-10-16T11:21:20.867151+0000","last_unstale":"2023-10-16T11:21:21.856345+0000","last_undegraded":"2023-10-16T11:21:21.856345+0000","last_fullsized":"2023-10-16T11:21:21.856345+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:57.278022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.855333+0000","last_change":"2023-10-16T11:21:19.853584+0000","last_active":"2023-10-16T11:21:20.855333+0000","last_peered":"2023-10-16T11:21:20.855333+0000","last_clean":"2023-10-16T11:21:20.855333+0000","last_became_active":"2023-10-16T11:21:19.853315+0000","last_became_peered":"2023-10-16T11:21:19.853315+0000","last_unstale":"2023-10-16T11:21:20.855333+0000","last_undegraded":"2023-10-16T11:21:20.855333+0000","last_fullsized":"2023-10-16T11:21:20.855333+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:44.852756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858206+0000","last_change":"2023-10-16T11:21:20.864407+0000","last_active":"2023-10-16T11:21:21.858206+0000","last_peered":"2023-10-16T11:21:21.858206+0000","last_clean":"2023-10-16T11:21:21.858206+0000","last_became_active":"2023-10-16T11:21:20.863739+0000","last_became_peered":"2023-10-16T11:21:20.863739+0000","last_unstale":"2023-10-16T11:21:21.858206+0000","last_undegraded":"2023-10-16T11:21:21.858206+0000","last_fullsized":"2023-10-16T11:21:21.858206+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:43.954316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.868010+0000","last_change":"2023-10-16T11:21:19.853083+0000","last_active":"2023-10-16T11:21:20.868010+0000","last_peered":"2023-10-16T11:21:20.868010+0000","last_clean":"2023-10-16T11:21:20.868010+0000","last_became_active":"2023-10-16T11:21:19.852799+0000","last_became_peered":"2023-10-16T11:21:19.852799+0000","last_unstale":"2023-10-16T11:21:20.868010+0000","last_undegraded":"2023-10-16T11:21:20.868010+0000","last_fullsized":"2023-10-16T11:21:20.868010+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:35.615415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952450+0000","last_change":"2023-10-16T11:21:20.870505+0000","last_active":"2023-10-16T11:21:22.952450+0000","last_peered":"2023-10-16T11:21:22.952450+0000","last_clean":"2023-10-16T11:21:22.952450+0000","last_became_active":"2023-10-16T11:21:20.870248+0000","last_became_peered":"2023-10-16T11:21:20.870248+0000","last_unstale":"2023-10-16T11:21:22.952450+0000","last_undegraded":"2023-10-16T11:21:22.952450+0000","last_fullsized":"2023-10-16T11:21:22.952450+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:03.638407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.864897+0000","last_change":"2023-10-16T11:21:19.852581+0000","last_active":"2023-10-16T11:21:20.864897+0000","last_peered":"2023-10-16T11:21:20.864897+0000","last_clean":"2023-10-16T11:21:20.864897+0000","last_became_active":"2023-10-16T11:21:19.852113+0000","last_became_peered":"2023-10-16T11:21:19.852113+0000","last_unstale":"2023-10-16T11:21:20.864897+0000","last_undegraded":"2023-10-16T11:21:20.864897+0000","last_fullsized":"2023-10-16T11:21:20.864897+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:07.655056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953057+0000","last_change":"2023-10-16T11:21:20.876890+0000","last_active":"2023-10-16T11:21:22.953057+0000","last_peered":"2023-10-16T11:21:22.953057+0000","last_clean":"2023-10-16T11:21:22.953057+0000","last_became_active":"2023-10-16T11:21:20.876617+0000","last_became_peered":"2023-10-16T11:21:20.876617+0000","last_unstale":"2023-10-16T11:21:22.953057+0000","last_undegraded":"2023-10-16T11:21:22.953057+0000","last_fullsized":"2023-10-16T11:21:22.953057+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:36.015828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.884482+0000","last_change":"2023-10-16T11:21:19.858179+0000","last_active":"2023-10-16T11:21:20.884482+0000","last_peered":"2023-10-16T11:21:20.884482+0000","last_clean":"2023-10-16T11:21:20.884482+0000","last_became_active":"2023-10-16T11:21:19.857905+0000","last_became_peered":"2023-10-16T11:21:19.857905+0000","last_unstale":"2023-10-16T11:21:20.884482+0000","last_undegraded":"2023-10-16T11:21:20.884482+0000","last_fullsized":"2023-10-16T11:21:20.884482+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:04.213264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857637+0000","last_change":"2023-10-16T11:21:20.866902+0000","last_active":"2023-10-16T11:21:21.857637+0000","last_peered":"2023-10-16T11:21:21.857637+0000","last_clean":"2023-10-16T11:21:21.857637+0000","last_became_active":"2023-10-16T11:21:20.866629+0000","last_became_peered":"2023-10-16T11:21:20.866629+0000","last_unstale":"2023-10-16T11:21:21.857637+0000","last_undegraded":"2023-10-16T11:21:21.857637+0000","last_fullsized":"2023-10-16T11:21:21.857637+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:24.438824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.867180+0000","last_change":"2023-10-16T11:21:19.858912+0000","last_active":"2023-10-16T11:21:20.867180+0000","last_peered":"2023-10-16T11:21:20.867180+0000","last_clean":"2023-10-16T11:21:20.867180+0000","last_became_active":"2023-10-16T11:21:19.858654+0000","last_became_peered":"2023-10-16T11:21:19.858654+0000","last_unstale":"2023-10-16T11:21:20.867180+0000","last_undegraded":"2023-10-16T11:21:20.867180+0000","last_fullsized":"2023-10-16T11:21:20.867180+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:56.204594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953229+0000","last_change":"2023-10-16T11:21:20.874240+0000","last_active":"2023-10-16T11:21:22.953229+0000","last_peered":"2023-10-16T11:21:22.953229+0000","last_clean":"2023-10-16T11:21:22.953229+0000","last_became_active":"2023-10-16T11:21:20.873584+0000","last_became_peered":"2023-10-16T11:21:20.873584+0000","last_unstale":"2023-10-16T11:21:22.953229+0000","last_undegraded":"2023-10-16T11:21:22.953229+0000","last_fullsized":"2023-10-16T11:21:22.953229+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:50.700023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.858382+0000","last_change":"2023-10-16T11:21:19.854345+0000","last_active":"2023-10-16T11:21:20.858382+0000","last_peered":"2023-10-16T11:21:20.858382+0000","last_clean":"2023-10-16T11:21:20.858382+0000","last_became_active":"2023-10-16T11:21:19.853927+0000","last_became_peered":"2023-10-16T11:21:19.853927+0000","last_unstale":"2023-10-16T11:21:20.858382+0000","last_undegraded":"2023-10-16T11:21:20.858382+0000","last_fullsized":"2023-10-16T11:21:20.858382+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:18.538107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953600+0000","last_change":"2023-10-16T11:21:20.864934+0000","last_active":"2023-10-16T11:21:22.953600+0000","last_peered":"2023-10-16T11:21:22.953600+0000","last_clean":"2023-10-16T11:21:22.953600+0000","last_became_active":"2023-10-16T11:21:20.864623+0000","last_became_peered":"2023-10-16T11:21:20.864623+0000","last_unstale":"2023-10-16T11:21:22.953600+0000","last_undegraded":"2023-10-16T11:21:22.953600+0000","last_fullsized":"2023-10-16T11:21:22.953600+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:00.130637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863335+0000","last_change":"2023-10-16T11:21:19.856230+0000","last_active":"2023-10-16T11:21:20.863335+0000","last_peered":"2023-10-16T11:21:20.863335+0000","last_clean":"2023-10-16T11:21:20.863335+0000","last_became_active":"2023-10-16T11:21:19.855798+0000","last_became_peered":"2023-10-16T11:21:19.855798+0000","last_unstale":"2023-10-16T11:21:20.863335+0000","last_undegraded":"2023-10-16T11:21:20.863335+0000","last_fullsized":"2023-10-16T11:21:20.863335+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:03.626811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953221+0000","last_change":"2023-10-16T11:21:20.864277+0000","last_active":"2023-10-16T11:21:22.953221+0000","last_peered":"2023-10-16T11:21:22.953221+0000","last_clean":"2023-10-16T11:21:22.953221+0000","last_became_active":"2023-10-16T11:21:20.864001+0000","last_became_peered":"2023-10-16T11:21:20.864001+0000","last_unstale":"2023-10-16T11:21:22.953221+0000","last_undegraded":"2023-10-16T11:21:22.953221+0000","last_fullsized":"2023-10-16T11:21:22.953221+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:40.650230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.865172+0000","last_change":"2023-10-16T11:21:19.856119+0000","last_active":"2023-10-16T11:21:20.865172+0000","last_peered":"2023-10-16T11:21:20.865172+0000","last_clean":"2023-10-16T11:21:20.865172+0000","last_became_active":"2023-10-16T11:21:19.855791+0000","last_became_peered":"2023-10-16T11:21:19.855791+0000","last_unstale":"2023-10-16T11:21:20.865172+0000","last_undegraded":"2023-10-16T11:21:20.865172+0000","last_fullsized":"2023-10-16T11:21:20.865172+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:24.829226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953750+0000","last_change":"2023-10-16T11:21:20.863082+0000","last_active":"2023-10-16T11:21:22.953750+0000","last_peered":"2023-10-16T11:21:22.953750+0000","last_clean":"2023-10-16T11:21:22.953750+0000","last_became_active":"2023-10-16T11:21:20.862820+0000","last_became_peered":"2023-10-16T11:21:20.862820+0000","last_unstale":"2023-10-16T11:21:22.953750+0000","last_undegraded":"2023-10-16T11:21:22.953750+0000","last_fullsized":"2023-10-16T11:21:22.953750+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:58.851375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.856182+0000","last_change":"2023-10-16T11:21:19.853507+0000","last_active":"2023-10-16T11:21:20.856182+0000","last_peered":"2023-10-16T11:21:20.856182+0000","last_clean":"2023-10-16T11:21:20.856182+0000","last_became_active":"2023-10-16T11:21:19.853225+0000","last_became_peered":"2023-10-16T11:21:19.853225+0000","last_unstale":"2023-10-16T11:21:20.856182+0000","last_undegraded":"2023-10-16T11:21:20.856182+0000","last_fullsized":"2023-10-16T11:21:20.856182+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:02.656474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952782+0000","last_change":"2023-10-16T11:21:20.868118+0000","last_active":"2023-10-16T11:21:22.952782+0000","last_peered":"2023-10-16T11:21:22.952782+0000","last_clean":"2023-10-16T11:21:22.952782+0000","last_became_active":"2023-10-16T11:21:20.867840+0000","last_became_peered":"2023-10-16T11:21:20.867840+0000","last_unstale":"2023-10-16T11:21:22.952782+0000","last_undegraded":"2023-10-16T11:21:22.952782+0000","last_fullsized":"2023-10-16T11:21:22.952782+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:30.453547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.858004+0000","last_change":"2023-10-16T11:21:19.854495+0000","last_active":"2023-10-16T11:21:20.858004+0000","last_peered":"2023-10-16T11:21:20.858004+0000","last_clean":"2023-10-16T11:21:20.858004+0000","last_became_active":"2023-10-16T11:21:19.854178+0000","last_became_peered":"2023-10-16T11:21:19.854178+0000","last_unstale":"2023-10-16T11:21:20.858004+0000","last_undegraded":"2023-10-16T11:21:20.858004+0000","last_fullsized":"2023-10-16T11:21:20.858004+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:00.550117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953714+0000","last_change":"2023-10-16T11:21:20.870231+0000","last_active":"2023-10-16T11:21:22.953714+0000","last_peered":"2023-10-16T11:21:22.953714+0000","last_clean":"2023-10-16T11:21:22.953714+0000","last_became_active":"2023-10-16T11:21:20.869973+0000","last_became_peered":"2023-10-16T11:21:20.869973+0000","last_unstale":"2023-10-16T11:21:22.953714+0000","last_undegraded":"2023-10-16T11:21:22.953714+0000","last_fullsized":"2023-10-16T11:21:22.953714+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:14:30.942915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.865043+0000","last_change":"2023-10-16T11:21:19.855368+0000","last_active":"2023-10-16T11:21:20.865043+0000","last_peered":"2023-10-16T11:21:20.865043+0000","last_clean":"2023-10-16T11:21:20.865043+0000","last_became_active":"2023-10-16T11:21:19.854895+0000","last_became_peered":"2023-10-16T11:21:19.854895+0000","last_unstale":"2023-10-16T11:21:20.865043+0000","last_undegraded":"2023-10-16T11:21:20.865043+0000","last_fullsized":"2023-10-16T11:21:20.865043+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:05.484629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.412651+0000","last_change":"2023-10-16T11:21:20.874328+0000","last_active":"2023-10-16T11:21:22.412651+0000","last_peered":"2023-10-16T11:21:22.412651+0000","last_clean":"2023-10-16T11:21:22.412651+0000","last_became_active":"2023-10-16T11:21:20.874076+0000","last_became_peered":"2023-10-16T11:21:20.874076+0000","last_unstale":"2023-10-16T11:21:22.412651+0000","last_undegraded":"2023-10-16T11:21:22.412651+0000","last_fullsized":"2023-10-16T11:21:22.412651+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:51.610368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"243'381","reported_seq":1365,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.859644+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:21:20.859644+0000","last_peered":"2023-10-16T11:21:20.859644+0000","last_clean":"2023-10-16T11:21:20.859644+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:21:20.859644+0000","last_undegraded":"2023-10-16T11:21:20.859644+0000","last_fullsized":"2023-10-16T11:21:20.859644+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53012,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318828,"statfs":{"total":96636764160,"available":96582479872,"internally_reserved":0,"allocated":3874816,"data_stored":2353352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64600000000000002,"15min":0.60699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68999999999999995,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.66400000000000003,"15min":0.628},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.68799999999999994,"15min":0.66200000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.622,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.70099999999999996,"15min":0.67400000000000004},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.71199999999999997,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.71399999999999997,"15min":0.67300000000000004},"min":{"1min":0.48099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78900000000000003,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.73299999999999998,"15min":0.69099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":9.516,"15min":9.516},"last":0.42399999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.68300000000000005,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.68799999999999994,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.75900000000000001,"15min":0.71199999999999997},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.61099999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.54100000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32300000000000001},"max":{"1min":0.747,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.37},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.63200000000000001,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.623}]},{"osd":5,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65300000000000002,"15min":0.57799999999999996},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":0.98399999999999999,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69699999999999995,"15min":0.63},"min":{"1min":0.41499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79500000000000004,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.71599999999999997,"15min":0.64600000000000002},"min":{"1min":0.41599999999999998,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.84199999999999997,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.502},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.69699999999999995,"15min":0.64200000000000002},"min":{"1min":0.442,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76600000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.65000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56572,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315268,"statfs":{"total":96636764160,"available":96578834432,"internally_reserved":0,"allocated":3260416,"data_stored":1760720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75800000000000001,"5min":0.80700000000000005,"15min":0.86299999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.59199999999999997,"15min":0.61399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69399999999999995,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.71699999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58099999999999996,"15min":0.60599999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73499999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":2.387,"15min":2.387},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.57199999999999995,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.314},"max":{"1min":0.58699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":0.62},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.622,"15min":0.621},"min":{"1min":0.36499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.65300000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.36399999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71599999999999997,"5min":1.02,"15min":1.02},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.32900000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.55500000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.77800000000000002,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.45400000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76200000000000001,"5min":3.335,"15min":3.335},"last":0.79200000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":2.524,"15min":2.524},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.752,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.70399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52420,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319420,"statfs":{"total":96636764160,"available":96583086080,"internally_reserved":0,"allocated":3264512,"data_stored":1760810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.38700000000000001,"15min":0.40799999999999997},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51000000000000001,"5min":0.622,"15min":0.66700000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53300000000000003,"5min":0.63900000000000001,"15min":0.68799999999999994},"last":0.37}]},{"osd":2,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.375,"15min":0.40100000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.439,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.41099999999999998,"15min":0.44400000000000001},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48899999999999999,"5min":0.623,"15min":0.70099999999999996},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.441,"15min":0.46600000000000003},"min":{"1min":0.30299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.53200000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41399999999999998,"15min":0.437},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.61799999999999999,"5min":0.623,"15min":0.65600000000000003},"last":0.39800000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.624,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73399999999999999,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.64300000000000002,"15min":0.68700000000000006},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.86299999999999999,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.70599999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89100000000000001,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64100000000000001,"15min":0.66700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":1.0149999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.64400000000000002,"15min":0.67800000000000005},"min":{"1min":0.43099999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79300000000000004,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.69899999999999995},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89300000000000002,"5min":0.89800000000000002,"15min":1.0620000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.44},"max":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":1.016},"last":0.67200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61328,"kb_used_data":3772,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310512,"statfs":{"total":96636764160,"available":96573964288,"internally_reserved":0,"allocated":3862528,"data_stored":2351704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.45000000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.26700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52800000000000002,"5min":0.66100000000000003,"15min":0.77200000000000002},"last":0.19500000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":1.272,"15min":1.272},"last":0.315},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.28299999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40899999999999997,"15min":0.433},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.216},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54200000000000004,"5min":1.351,"15min":1.351},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83099999999999996,"5min":1.391,"15min":1.391},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76000000000000001,"5min":0.874,"15min":1.089},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.624,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.95599999999999996,"15min":1.216},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.47199999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.1180000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.67200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80300000000000005,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80500000000000005,"5min":0.89600000000000002,"15min":1.0780000000000001},"last":0.52800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834576010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52412,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319428,"statfs":{"total":96636764160,"available":96583094272,"internally_reserved":0,"allocated":3260416,"data_stored":1760720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.245,"5min":0.188,"15min":0.188},"max":{"1min":0.47699999999999998,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.437},"min":{"1min":0.255,"5min":0.193,"15min":0.193},"max":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.436,"15min":0.46400000000000002},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.41099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45300000000000001,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.36499999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":1.097},"last":0.627},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86499999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.67100000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83099999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92700000000000005,"5min":1.0169999999999999,"15min":1.028},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.498,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85899999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.70399999999999996},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":0.99099999999999999,"15min":1.071},"last":0.62},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.97799999999999998,"15min":1.077},"last":0.69799999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576010,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56580,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315260,"statfs":{"total":96636764160,"available":96578826240,"internally_reserved":0,"allocated":3268608,"data_stored":1760812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36499999999999999,"15min":0.39000000000000001},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.621,"5min":1.032,"15min":1.032},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.36799999999999999,"15min":0.40500000000000003},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.52200000000000002,"5min":1.341,"15min":1.341},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40899999999999997,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55000000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.44400000000000001,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.51300000000000001,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.313},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45900000000000002,"15min":0.503},"min":{"1min":0.34799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54600000000000004,"5min":0.96599999999999997,"15min":1.024},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.623,"15min":0.65300000000000002},"min":{"1min":0.45500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3220000000000001,"5min":2.629,"15min":2.629},"last":0.76400000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.66700000000000004},"min":{"1min":0.443,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":1.0149999999999999,"15min":1.026},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.746}]},{"osd":6,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81100000000000005,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.80400000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":0.71399999999999997},"min":{"1min":0.49099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.304,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65800000000000003,"15min":0.69899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.86399999999999999,"5min":1.006,"15min":1.1899999999999999},"last":0.72199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52436,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319404,"statfs":{"total":96636764160,"available":96583069696,"internally_reserved":0,"allocated":3276800,"data_stored":1762422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.60899999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.61399999999999999},"min":{"1min":0.41899999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81899999999999995,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.60799999999999998},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.95499999999999996},"last":0.441}]},{"osd":2,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57199999999999995,"15min":0.61699999999999999},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81100000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.73599999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60899999999999999,"15min":0.63200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.86199999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76000000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.71599999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.82899999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.751},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.35799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":1.115},"last":0.81499999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.80000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66316,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305524,"statfs":{"total":96636764160,"available":96568856576,"internally_reserved":0,"allocated":3268608,"data_stored":1760776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58599999999999997,"15min":0.623},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28999999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.64300000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.88200000000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67900000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74299999999999999,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53700000000000003,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.623,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81899999999999995,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.747}]},{"osd":3,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79300000000000004,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.59899999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.498},{"interface":"front","average":{"1min":0.503,"5min":0.496,"15min":0.51600000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83599999999999997,"5min":0.878,"15min":0.878},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.874},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.623,"15min":0.63700000000000001},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:21:24.975 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:21:25.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.128+0000 7fcb0a198700 1 Processor -- start 2023-10-16T11:21:25.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.128+0000 7fcb0a198700 1 -- start start 2023-10-16T11:21:25.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.128+0000 7fcb0a198700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcb04071980 0x7fcb0415f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.128+0000 7fcb0a198700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0415f1a0 con 0x7fcb040ca980 2023-10-16T11:21:25.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb0a198700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb04071e80 con 0x7fcb040cc910 2023-10-16T11:21:25.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb0a198700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb04072160 con 0x7fcb04071980 2023-10-16T11:21:25.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb037fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcb040ca980 0x7fcb040cad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42496/0 (socket says 172.21.15.158:42496) 2023-10-16T11:21:25.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb037fe700 1 -- 172.21.15.158:0/2734447446 learned_addr learned my addr 172.21.15.158:0/2734447446 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:25.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713167680 0 0) 0x7fcb0415f1a0 con 0x7fcb040ca980 2023-10-16T11:21:25.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaec0035b0 con 0x7fcb040ca980 2023-10-16T11:21:25.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb03fff700 1 --2- 172.21.15.158:0/2734447446 >> v2:172.21.15.172:3300/0 conn(0x7fcb04071980 0x7fcb0415f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058669915 0 0) 0x7fcaec0035b0 con 0x7fcb040ca980 2023-10-16T11:21:25.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 >> v2:172.21.15.172:3300/0 conn(0x7fcb04071980 msgr2=0x7fcb0415f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:25.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 --2- 172.21.15.158:0/2734447446 >> v2:172.21.15.172:3300/0 conn(0x7fcb04071980 0x7fcb0415f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 >> v1:172.21.15.172:6789/0 conn(0x7fcb040cc910 legacy=0x7fcb0415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:25.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04072440 con 0x7fcb040ca980 2023-10-16T11:21:25.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcaf4001670 con 0x7fcb040ca980 2023-10-16T11:21:25.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb03fff700 1 --2- 172.21.15.158:0/2734447446 >> v2:172.21.15.172:3300/0 conn(0x7fcb04071980 0x7fcb0415f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:25.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 >> v1:172.21.15.158:6789/0 conn(0x7fcb040ca980 legacy=0x7fcb040cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaec003a00 0x7fcaec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec004850 0x7fcaec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaec005330 0x7fcaec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb04072160 con 0x7fcaec005330 2023-10-16T11:21:25.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec00c200 con 0x7fcaec004850 2023-10-16T11:21:25.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec00c3c0 con 0x7fcaec003a00 2023-10-16T11:21:25.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb03fff700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaec005330 0x7fcaec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb02ffd700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec004850 0x7fcaec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.129+0000 7fcb037fe700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaec003a00 0x7fcaec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb03fff700 1 -- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaec003a00 msgr2=0x7fcaec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:25.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb03fff700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaec003a00 0x7fcaec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb03fff700 1 -- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec004850 msgr2=0x7fcaec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:25.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb03fff700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec004850 0x7fcaec004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb03fff700 1 -- 172.21.15.158:0/2734447446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaec0035b0 con 0x7fcaec005330 2023-10-16T11:21:25.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb037fe700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaec003a00 0x7fcaec003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:25.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb03fff700 1 -- 172.21.15.158:0/2734447446 reap_dead start 2023-10-16T11:21:25.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb03fff700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaec005330 0x7fcaec00ba10 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7fcb040d02e0 tx=0x7fcaec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabe98b4433ff28b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc009490 con 0x7fcaec005330 2023-10-16T11:21:25.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc007e90 con 0x7fcaec005330 2023-10-16T11:21:25.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb027fc700 1 -- 172.21.15.158:0/2734447446 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc014640 con 0x7fcaec005330 2023-10-16T11:21:25.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb0a198700 1 -- 172.21.15.158:0/2734447446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaec005330 msgr2=0x7fcaec00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb0a198700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaec005330 0x7fcaec00ba10 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7fcb040d02e0 tx=0x7fcaec005880 comp rx=0 tx=0).stop 2023-10-16T11:21:25.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb0a198700 1 -- 172.21.15.158:0/2734447446 shutdown_connections 2023-10-16T11:21:25.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb0a198700 1 --2- 172.21.15.158:0/2734447446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaec005330 0x7fcaec00ba10 unknown :-1 s=CLOSED pgs=3130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.130+0000 7fcb0a198700 1 -- 172.21.15.158:0/2734447446 >> 172.21.15.158:0/2734447446 conn(0x7fcb040ccdb0 msgr2=0x7fcb040cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:25.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.131+0000 7fcb0a198700 1 -- 172.21.15.158:0/2734447446 shutdown_connections 2023-10-16T11:21:25.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.131+0000 7fcb0a198700 1 -- 172.21.15.158:0/2734447446 wait complete. 2023-10-16T11:21:25.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.132+0000 7fcb0a198700 1 Processor -- start 2023-10-16T11:21:25.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.132+0000 7fcb0a198700 1 -- start start 2023-10-16T11:21:25.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec005330 0x7fcb04158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb041589e0 0x7fcb04157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb041e59e0 0x7fcb040b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb041581a0 con 0x7fcb041e59e0 2023-10-16T11:21:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb041f4030 con 0x7fcaec005330 2023-10-16T11:21:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb041f4280 con 0x7fcb041589e0 2023-10-16T11:21:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb041e59e0 0x7fcb040b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb041e59e0 0x7fcb040b1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42336/0 (socket says 172.21.15.158:42336) 2023-10-16T11:21:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 -- 172.21.15.158:0/319024112 learned_addr learned my addr 172.21.15.158:0/319024112 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb02ffd700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb041589e0 0x7fcb04157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb037fe700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec005330 0x7fcb04158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 -- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb041589e0 msgr2=0x7fcb04157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb041589e0 0x7fcb04157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 -- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec005330 msgr2=0x7fcb04158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec005330 0x7fcb04158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 -- 172.21.15.158:0/319024112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb041f4560 con 0x7fcb041e59e0 2023-10-16T11:21:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb03fff700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb041e59e0 0x7fcb040b1980 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7fcb040af780 tx=0x7fcafc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcaebfff700 1 -- 172.21.15.158:0/319024112 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc007000 con 0x7fcb041e59e0 2023-10-16T11:21:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcaebfff700 1 -- 172.21.15.158:0/319024112 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc004840 con 0x7fcb041e59e0 2023-10-16T11:21:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb041f4840 con 0x7fcb041e59e0 2023-10-16T11:21:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcaebfff700 1 -- 172.21.15.158:0/319024112 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc01d640 con 0x7fcb041e59e0 2023-10-16T11:21:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.133+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb040b1ec0 con 0x7fcb041e59e0 2023-10-16T11:21:25.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.134+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb041f4d60 con 0x7fcb041e59e0 2023-10-16T11:21:25.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.135+0000 7fcaebfff700 1 -- 172.21.15.158:0/319024112 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcafc01b040 con 0x7fcb041e59e0 2023-10-16T11:21:25.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.135+0000 7fcaebfff700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fcaf006b140 0x7fcaf006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.135+0000 7fcaebfff700 1 -- 172.21.15.158:0/319024112 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fcafc08a9d0 con 0x7fcb041e59e0 2023-10-16T11:21:25.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.135+0000 7fcb037fe700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fcaf006b140 0x7fcaf006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.136+0000 7fcb037fe700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fcaf006b140 0x7fcaf006d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4000c00 tx=0x7fcaf4007040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.140+0000 7fcaebfff700 1 -- 172.21.15.158:0/319024112 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb041f4d60 con 0x7fcb041e59e0 2023-10-16T11:21:25.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.327+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 --> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb04072720 con 0x7fcaf006b140 2023-10-16T11:21:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.335+0000 7fcaebfff700 1 -- 172.21.15.158:0/319024112 <== mgr.12843 v2:172.21.15.158:6832/3125181137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290088 (secure 0 0 0) 0x7fcaf40117c0 con 0x7fcaf006b140 2023-10-16T11:21:25.340 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:25.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fcaf006b140 msgr2=0x7fcaf006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fcaf006b140 0x7fcaf006d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4000c00 tx=0x7fcaf4007040 comp rx=0 tx=0).stop 2023-10-16T11:21:25.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb041e59e0 msgr2=0x7fcb040b1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb041e59e0 0x7fcb040b1980 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7fcb040af780 tx=0x7fcafc004b40 comp rx=0 tx=0).stop 2023-10-16T11:21:25.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 shutdown_connections 2023-10-16T11:21:25.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fcaf006b140 0x7fcaf006d600 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb041e59e0 0x7fcb040b1980 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaec005330 0x7fcb04158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 --2- 172.21.15.158:0/319024112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb041589e0 0x7fcb04157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 >> 172.21.15.158:0/319024112 conn(0x7fcb040ccdb0 msgr2=0x7fcb04159770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:25.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 shutdown_connections 2023-10-16T11:21:25.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.338+0000 7fcb0a198700 1 -- 172.21.15.158:0/319024112 wait complete. 2023-10-16T11:21:25.345 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:21:25.352 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T11:21:24.682563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":451076,"kb_used_data":26696,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754523644,"statfs":{"total":773094113280,"available":772632211456,"internally_reserved":0,"allocated":27336704,"data_stored":15271316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":434425399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.850015"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953392+0000","last_change":"2023-10-16T11:21:20.873354+0000","last_active":"2023-10-16T11:21:22.953392+0000","last_peered":"2023-10-16T11:21:22.953392+0000","last_clean":"2023-10-16T11:21:22.953392+0000","last_became_active":"2023-10-16T11:21:20.873059+0000","last_became_peered":"2023-10-16T11:21:20.873059+0000","last_unstale":"2023-10-16T11:21:22.953392+0000","last_undegraded":"2023-10-16T11:21:22.953392+0000","last_fullsized":"2023-10-16T11:21:22.953392+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:16:40.456737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953428+0000","last_change":"2023-10-16T11:21:20.878799+0000","last_active":"2023-10-16T11:21:22.953428+0000","last_peered":"2023-10-16T11:21:22.953428+0000","last_clean":"2023-10-16T11:21:22.953428+0000","last_became_active":"2023-10-16T11:21:20.878509+0000","last_became_peered":"2023-10-16T11:21:20.878509+0000","last_unstale":"2023-10-16T11:21:22.953428+0000","last_undegraded":"2023-10-16T11:21:22.953428+0000","last_fullsized":"2023-10-16T11:21:22.953428+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:14.231361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953628+0000","last_change":"2023-10-16T11:21:20.877276+0000","last_active":"2023-10-16T11:21:22.953628+0000","last_peered":"2023-10-16T11:21:22.953628+0000","last_clean":"2023-10-16T11:21:22.953628+0000","last_became_active":"2023-10-16T11:21:20.877002+0000","last_became_peered":"2023-10-16T11:21:20.877002+0000","last_unstale":"2023-10-16T11:21:22.953628+0000","last_undegraded":"2023-10-16T11:21:22.953628+0000","last_fullsized":"2023-10-16T11:21:22.953628+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:41.706764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953474+0000","last_change":"2023-10-16T11:21:20.863524+0000","last_active":"2023-10-16T11:21:22.953474+0000","last_peered":"2023-10-16T11:21:22.953474+0000","last_clean":"2023-10-16T11:21:22.953474+0000","last_became_active":"2023-10-16T11:21:20.863265+0000","last_became_peered":"2023-10-16T11:21:20.863265+0000","last_unstale":"2023-10-16T11:21:22.953474+0000","last_undegraded":"2023-10-16T11:21:22.953474+0000","last_fullsized":"2023-10-16T11:21:22.953474+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:44.964933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952776+0000","last_change":"2023-10-16T11:21:20.879735+0000","last_active":"2023-10-16T11:21:22.952776+0000","last_peered":"2023-10-16T11:21:22.952776+0000","last_clean":"2023-10-16T11:21:22.952776+0000","last_became_active":"2023-10-16T11:21:20.879472+0000","last_became_peered":"2023-10-16T11:21:20.879472+0000","last_unstale":"2023-10-16T11:21:22.952776+0000","last_undegraded":"2023-10-16T11:21:22.952776+0000","last_fullsized":"2023-10-16T11:21:22.952776+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:25.834081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952610+0000","last_change":"2023-10-16T11:21:20.867649+0000","last_active":"2023-10-16T11:21:22.952610+0000","last_peered":"2023-10-16T11:21:22.952610+0000","last_clean":"2023-10-16T11:21:22.952610+0000","last_became_active":"2023-10-16T11:21:20.867349+0000","last_became_peered":"2023-10-16T11:21:20.867349+0000","last_unstale":"2023-10-16T11:21:22.952610+0000","last_undegraded":"2023-10-16T11:21:22.952610+0000","last_fullsized":"2023-10-16T11:21:22.952610+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:21.764175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952926+0000","last_change":"2023-10-16T11:21:20.869639+0000","last_active":"2023-10-16T11:21:22.952926+0000","last_peered":"2023-10-16T11:21:22.952926+0000","last_clean":"2023-10-16T11:21:22.952926+0000","last_became_active":"2023-10-16T11:21:20.869374+0000","last_became_peered":"2023-10-16T11:21:20.869374+0000","last_unstale":"2023-10-16T11:21:22.952926+0000","last_undegraded":"2023-10-16T11:21:22.952926+0000","last_fullsized":"2023-10-16T11:21:22.952926+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:57.278022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952644+0000","last_change":"2023-10-16T11:21:20.867908+0000","last_active":"2023-10-16T11:21:22.952644+0000","last_peered":"2023-10-16T11:21:22.952644+0000","last_clean":"2023-10-16T11:21:22.952644+0000","last_became_active":"2023-10-16T11:21:20.867652+0000","last_became_peered":"2023-10-16T11:21:20.867652+0000","last_unstale":"2023-10-16T11:21:22.952644+0000","last_undegraded":"2023-10-16T11:21:22.952644+0000","last_fullsized":"2023-10-16T11:21:22.952644+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:57.667626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953340+0000","last_change":"2023-10-16T11:21:20.871129+0000","last_active":"2023-10-16T11:21:22.953340+0000","last_peered":"2023-10-16T11:21:22.953340+0000","last_clean":"2023-10-16T11:21:22.953340+0000","last_became_active":"2023-10-16T11:21:20.870868+0000","last_became_peered":"2023-10-16T11:21:20.870868+0000","last_unstale":"2023-10-16T11:21:22.953340+0000","last_undegraded":"2023-10-16T11:21:22.953340+0000","last_fullsized":"2023-10-16T11:21:22.953340+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:31.143865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953378+0000","last_change":"2023-10-16T11:21:20.872430+0000","last_active":"2023-10-16T11:21:22.953378+0000","last_peered":"2023-10-16T11:21:22.953378+0000","last_clean":"2023-10-16T11:21:22.953378+0000","last_became_active":"2023-10-16T11:21:20.872135+0000","last_became_peered":"2023-10-16T11:21:20.872135+0000","last_unstale":"2023-10-16T11:21:22.953378+0000","last_undegraded":"2023-10-16T11:21:22.953378+0000","last_fullsized":"2023-10-16T11:21:22.953378+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:36.782718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857284+0000","last_change":"2023-10-16T11:21:20.876425+0000","last_active":"2023-10-16T11:21:21.857284+0000","last_peered":"2023-10-16T11:21:21.857284+0000","last_clean":"2023-10-16T11:21:21.857284+0000","last_became_active":"2023-10-16T11:21:20.876167+0000","last_became_peered":"2023-10-16T11:21:20.876167+0000","last_unstale":"2023-10-16T11:21:21.857284+0000","last_undegraded":"2023-10-16T11:21:21.857284+0000","last_fullsized":"2023-10-16T11:21:21.857284+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:29.298975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953552+0000","last_change":"2023-10-16T11:21:20.871087+0000","last_active":"2023-10-16T11:21:22.953552+0000","last_peered":"2023-10-16T11:21:22.953552+0000","last_clean":"2023-10-16T11:21:22.953552+0000","last_became_active":"2023-10-16T11:21:20.870664+0000","last_became_peered":"2023-10-16T11:21:20.870664+0000","last_unstale":"2023-10-16T11:21:22.953552+0000","last_undegraded":"2023-10-16T11:21:22.953552+0000","last_fullsized":"2023-10-16T11:21:22.953552+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:10:10.791379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953348+0000","last_change":"2023-10-16T11:21:20.874457+0000","last_active":"2023-10-16T11:21:22.953348+0000","last_peered":"2023-10-16T11:21:22.953348+0000","last_clean":"2023-10-16T11:21:22.953348+0000","last_became_active":"2023-10-16T11:21:20.874188+0000","last_became_peered":"2023-10-16T11:21:20.874188+0000","last_unstale":"2023-10-16T11:21:22.953348+0000","last_undegraded":"2023-10-16T11:21:22.953348+0000","last_fullsized":"2023-10-16T11:21:22.953348+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:39.661386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953105+0000","last_change":"2023-10-16T11:21:20.870673+0000","last_active":"2023-10-16T11:21:22.953105+0000","last_peered":"2023-10-16T11:21:22.953105+0000","last_clean":"2023-10-16T11:21:22.953105+0000","last_became_active":"2023-10-16T11:21:20.870401+0000","last_became_peered":"2023-10-16T11:21:20.870401+0000","last_unstale":"2023-10-16T11:21:22.953105+0000","last_undegraded":"2023-10-16T11:21:22.953105+0000","last_fullsized":"2023-10-16T11:21:22.953105+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:51:15.955800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953064+0000","last_change":"2023-10-16T11:21:20.862779+0000","last_active":"2023-10-16T11:21:22.953064+0000","last_peered":"2023-10-16T11:21:22.953064+0000","last_clean":"2023-10-16T11:21:22.953064+0000","last_became_active":"2023-10-16T11:21:20.862507+0000","last_became_peered":"2023-10-16T11:21:20.862507+0000","last_unstale":"2023-10-16T11:21:22.953064+0000","last_undegraded":"2023-10-16T11:21:22.953064+0000","last_fullsized":"2023-10-16T11:21:22.953064+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:10.195225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857516+0000","last_change":"2023-10-16T11:21:20.874345+0000","last_active":"2023-10-16T11:21:21.857516+0000","last_peered":"2023-10-16T11:21:21.857516+0000","last_clean":"2023-10-16T11:21:21.857516+0000","last_became_active":"2023-10-16T11:21:20.874042+0000","last_became_peered":"2023-10-16T11:21:20.874042+0000","last_unstale":"2023-10-16T11:21:21.857516+0000","last_undegraded":"2023-10-16T11:21:21.857516+0000","last_fullsized":"2023-10-16T11:21:21.857516+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:45:42.867811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858220+0000","last_change":"2023-10-16T11:21:20.864051+0000","last_active":"2023-10-16T11:21:21.858220+0000","last_peered":"2023-10-16T11:21:21.858220+0000","last_clean":"2023-10-16T11:21:21.858220+0000","last_became_active":"2023-10-16T11:21:20.863769+0000","last_became_peered":"2023-10-16T11:21:20.863769+0000","last_unstale":"2023-10-16T11:21:21.858220+0000","last_undegraded":"2023-10-16T11:21:21.858220+0000","last_fullsized":"2023-10-16T11:21:21.858220+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:00.719281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953637+0000","last_change":"2023-10-16T11:21:20.871882+0000","last_active":"2023-10-16T11:21:22.953637+0000","last_peered":"2023-10-16T11:21:22.953637+0000","last_clean":"2023-10-16T11:21:22.953637+0000","last_became_active":"2023-10-16T11:21:20.871592+0000","last_became_peered":"2023-10-16T11:21:20.871592+0000","last_unstale":"2023-10-16T11:21:22.953637+0000","last_undegraded":"2023-10-16T11:21:22.953637+0000","last_fullsized":"2023-10-16T11:21:22.953637+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:39.347652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858184+0000","last_change":"2023-10-16T11:21:20.869240+0000","last_active":"2023-10-16T11:21:21.858184+0000","last_peered":"2023-10-16T11:21:21.858184+0000","last_clean":"2023-10-16T11:21:21.858184+0000","last_became_active":"2023-10-16T11:21:20.868972+0000","last_became_peered":"2023-10-16T11:21:20.868972+0000","last_unstale":"2023-10-16T11:21:21.858184+0000","last_undegraded":"2023-10-16T11:21:21.858184+0000","last_fullsized":"2023-10-16T11:21:21.858184+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:27.953134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858329+0000","last_change":"2023-10-16T11:21:20.875009+0000","last_active":"2023-10-16T11:21:21.858329+0000","last_peered":"2023-10-16T11:21:21.858329+0000","last_clean":"2023-10-16T11:21:21.858329+0000","last_became_active":"2023-10-16T11:21:20.874720+0000","last_became_peered":"2023-10-16T11:21:20.874720+0000","last_unstale":"2023-10-16T11:21:21.858329+0000","last_undegraded":"2023-10-16T11:21:21.858329+0000","last_fullsized":"2023-10-16T11:21:21.858329+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:39.063930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857087+0000","last_change":"2023-10-16T11:21:20.869041+0000","last_active":"2023-10-16T11:21:21.857087+0000","last_peered":"2023-10-16T11:21:21.857087+0000","last_clean":"2023-10-16T11:21:21.857087+0000","last_became_active":"2023-10-16T11:21:20.868785+0000","last_became_peered":"2023-10-16T11:21:20.868785+0000","last_unstale":"2023-10-16T11:21:21.857087+0000","last_undegraded":"2023-10-16T11:21:21.857087+0000","last_fullsized":"2023-10-16T11:21:21.857087+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:02.049634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856675+0000","last_change":"2023-10-16T11:21:20.869713+0000","last_active":"2023-10-16T11:21:21.856675+0000","last_peered":"2023-10-16T11:21:21.856675+0000","last_clean":"2023-10-16T11:21:21.856675+0000","last_became_active":"2023-10-16T11:21:20.869088+0000","last_became_peered":"2023-10-16T11:21:20.869088+0000","last_unstale":"2023-10-16T11:21:21.856675+0000","last_undegraded":"2023-10-16T11:21:21.856675+0000","last_fullsized":"2023-10-16T11:21:21.856675+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:40.650230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953414+0000","last_change":"2023-10-16T11:21:20.867354+0000","last_active":"2023-10-16T11:21:22.953414+0000","last_peered":"2023-10-16T11:21:22.953414+0000","last_clean":"2023-10-16T11:21:22.953414+0000","last_became_active":"2023-10-16T11:21:20.867105+0000","last_became_peered":"2023-10-16T11:21:20.867105+0000","last_unstale":"2023-10-16T11:21:22.953414+0000","last_undegraded":"2023-10-16T11:21:22.953414+0000","last_fullsized":"2023-10-16T11:21:22.953414+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:19.234265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856031+0000","last_change":"2023-10-16T11:21:20.874563+0000","last_active":"2023-10-16T11:21:21.856031+0000","last_peered":"2023-10-16T11:21:21.856031+0000","last_clean":"2023-10-16T11:21:21.856031+0000","last_became_active":"2023-10-16T11:21:20.874357+0000","last_became_peered":"2023-10-16T11:21:20.874357+0000","last_unstale":"2023-10-16T11:21:21.856031+0000","last_undegraded":"2023-10-16T11:21:21.856031+0000","last_fullsized":"2023-10-16T11:21:21.856031+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:52.659355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953611+0000","last_change":"2023-10-16T11:21:20.869158+0000","last_active":"2023-10-16T11:21:22.953611+0000","last_peered":"2023-10-16T11:21:22.953611+0000","last_clean":"2023-10-16T11:21:22.953611+0000","last_became_active":"2023-10-16T11:21:20.868856+0000","last_became_peered":"2023-10-16T11:21:20.868856+0000","last_unstale":"2023-10-16T11:21:22.953611+0000","last_undegraded":"2023-10-16T11:21:22.953611+0000","last_fullsized":"2023-10-16T11:21:22.953611+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:48.577732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952644+0000","last_change":"2023-10-16T11:21:20.873998+0000","last_active":"2023-10-16T11:21:22.952644+0000","last_peered":"2023-10-16T11:21:22.952644+0000","last_clean":"2023-10-16T11:21:22.952644+0000","last_became_active":"2023-10-16T11:21:20.873713+0000","last_became_peered":"2023-10-16T11:21:20.873713+0000","last_unstale":"2023-10-16T11:21:22.952644+0000","last_undegraded":"2023-10-16T11:21:22.952644+0000","last_fullsized":"2023-10-16T11:21:22.952644+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:08.659912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952677+0000","last_change":"2023-10-16T11:21:20.879227+0000","last_active":"2023-10-16T11:21:22.952677+0000","last_peered":"2023-10-16T11:21:22.952677+0000","last_clean":"2023-10-16T11:21:22.952677+0000","last_became_active":"2023-10-16T11:21:20.878922+0000","last_became_peered":"2023-10-16T11:21:20.878922+0000","last_unstale":"2023-10-16T11:21:22.952677+0000","last_undegraded":"2023-10-16T11:21:22.952677+0000","last_fullsized":"2023-10-16T11:21:22.952677+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:23.417317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856448+0000","last_change":"2023-10-16T11:21:20.869603+0000","last_active":"2023-10-16T11:21:21.856448+0000","last_peered":"2023-10-16T11:21:21.856448+0000","last_clean":"2023-10-16T11:21:21.856448+0000","last_became_active":"2023-10-16T11:21:20.869231+0000","last_became_peered":"2023-10-16T11:21:20.869231+0000","last_unstale":"2023-10-16T11:21:21.856448+0000","last_undegraded":"2023-10-16T11:21:21.856448+0000","last_fullsized":"2023-10-16T11:21:21.856448+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:27.477587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.412439+0000","last_change":"2023-10-16T11:21:20.874243+0000","last_active":"2023-10-16T11:21:22.412439+0000","last_peered":"2023-10-16T11:21:22.412439+0000","last_clean":"2023-10-16T11:21:22.412439+0000","last_became_active":"2023-10-16T11:21:20.873980+0000","last_became_peered":"2023-10-16T11:21:20.873980+0000","last_unstale":"2023-10-16T11:21:22.412439+0000","last_undegraded":"2023-10-16T11:21:22.412439+0000","last_fullsized":"2023-10-16T11:21:22.412439+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:36.014320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856230+0000","last_change":"2023-10-16T11:21:20.869783+0000","last_active":"2023-10-16T11:21:21.856230+0000","last_peered":"2023-10-16T11:21:21.856230+0000","last_clean":"2023-10-16T11:21:21.856230+0000","last_became_active":"2023-10-16T11:21:20.869508+0000","last_became_peered":"2023-10-16T11:21:20.869508+0000","last_unstale":"2023-10-16T11:21:21.856230+0000","last_undegraded":"2023-10-16T11:21:21.856230+0000","last_fullsized":"2023-10-16T11:21:21.856230+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:32.527753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952781+0000","last_change":"2023-10-16T11:21:20.875868+0000","last_active":"2023-10-16T11:21:22.952781+0000","last_peered":"2023-10-16T11:21:22.952781+0000","last_clean":"2023-10-16T11:21:22.952781+0000","last_became_active":"2023-10-16T11:21:20.875159+0000","last_became_peered":"2023-10-16T11:21:20.875159+0000","last_unstale":"2023-10-16T11:21:22.952781+0000","last_undegraded":"2023-10-16T11:21:22.952781+0000","last_fullsized":"2023-10-16T11:21:22.952781+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:42.829245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856106+0000","last_change":"2023-10-16T11:21:20.873972+0000","last_active":"2023-10-16T11:21:21.856106+0000","last_peered":"2023-10-16T11:21:21.856106+0000","last_clean":"2023-10-16T11:21:21.856106+0000","last_became_active":"2023-10-16T11:21:20.873285+0000","last_became_peered":"2023-10-16T11:21:20.873285+0000","last_unstale":"2023-10-16T11:21:21.856106+0000","last_undegraded":"2023-10-16T11:21:21.856106+0000","last_fullsized":"2023-10-16T11:21:21.856106+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:23.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"256'4","reported_seq":16,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.885240+0000","last_change":"2023-10-16T11:21:19.853470+0000","last_active":"2023-10-16T11:21:20.885240+0000","last_peered":"2023-10-16T11:21:20.885240+0000","last_clean":"2023-10-16T11:21:20.885240+0000","last_became_active":"2023-10-16T11:21:19.853198+0000","last_became_peered":"2023-10-16T11:21:19.853198+0000","last_unstale":"2023-10-16T11:21:20.885240+0000","last_undegraded":"2023-10-16T11:21:20.885240+0000","last_fullsized":"2023-10-16T11:21:20.885240+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:35.615415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953472+0000","last_change":"2023-10-16T11:21:20.872590+0000","last_active":"2023-10-16T11:21:22.953472+0000","last_peered":"2023-10-16T11:21:22.953472+0000","last_clean":"2023-10-16T11:21:22.953472+0000","last_became_active":"2023-10-16T11:21:20.872308+0000","last_became_peered":"2023-10-16T11:21:20.872308+0000","last_unstale":"2023-10-16T11:21:22.953472+0000","last_undegraded":"2023-10-16T11:21:22.953472+0000","last_fullsized":"2023-10-16T11:21:22.953472+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:03.714962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882397+0000","last_change":"2023-10-16T11:21:19.855092+0000","last_active":"2023-10-16T11:21:20.882397+0000","last_peered":"2023-10-16T11:21:20.882397+0000","last_clean":"2023-10-16T11:21:20.882397+0000","last_became_active":"2023-10-16T11:21:19.854771+0000","last_became_peered":"2023-10-16T11:21:19.854771+0000","last_unstale":"2023-10-16T11:21:20.882397+0000","last_undegraded":"2023-10-16T11:21:20.882397+0000","last_fullsized":"2023-10-16T11:21:20.882397+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:23.617910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953664+0000","last_change":"2023-10-16T11:21:20.877402+0000","last_active":"2023-10-16T11:21:22.953664+0000","last_peered":"2023-10-16T11:21:22.953664+0000","last_clean":"2023-10-16T11:21:22.953664+0000","last_became_active":"2023-10-16T11:21:20.877109+0000","last_became_peered":"2023-10-16T11:21:20.877109+0000","last_unstale":"2023-10-16T11:21:22.953664+0000","last_undegraded":"2023-10-16T11:21:22.953664+0000","last_fullsized":"2023-10-16T11:21:22.953664+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:14.131040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883470+0000","last_change":"2023-10-16T11:21:19.854161+0000","last_active":"2023-10-16T11:21:20.883470+0000","last_peered":"2023-10-16T11:21:20.883470+0000","last_clean":"2023-10-16T11:21:20.883470+0000","last_became_active":"2023-10-16T11:21:19.853159+0000","last_became_peered":"2023-10-16T11:21:19.853159+0000","last_unstale":"2023-10-16T11:21:20.883470+0000","last_undegraded":"2023-10-16T11:21:20.883470+0000","last_fullsized":"2023-10-16T11:21:20.883470+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:29.519798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953024+0000","last_change":"2023-10-16T11:21:20.870583+0000","last_active":"2023-10-16T11:21:22.953024+0000","last_peered":"2023-10-16T11:21:22.953024+0000","last_clean":"2023-10-16T11:21:22.953024+0000","last_became_active":"2023-10-16T11:21:20.870295+0000","last_became_peered":"2023-10-16T11:21:20.870295+0000","last_unstale":"2023-10-16T11:21:22.953024+0000","last_undegraded":"2023-10-16T11:21:22.953024+0000","last_fullsized":"2023-10-16T11:21:22.953024+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:27.477587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883433+0000","last_change":"2023-10-16T11:21:19.853555+0000","last_active":"2023-10-16T11:21:20.883433+0000","last_peered":"2023-10-16T11:21:20.883433+0000","last_clean":"2023-10-16T11:21:20.883433+0000","last_became_active":"2023-10-16T11:21:19.853179+0000","last_became_peered":"2023-10-16T11:21:19.853179+0000","last_unstale":"2023-10-16T11:21:20.883433+0000","last_undegraded":"2023-10-16T11:21:20.883433+0000","last_fullsized":"2023-10-16T11:21:20.883433+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:56.625722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953381+0000","last_change":"2023-10-16T11:21:20.876740+0000","last_active":"2023-10-16T11:21:22.953381+0000","last_peered":"2023-10-16T11:21:22.953381+0000","last_clean":"2023-10-16T11:21:22.953381+0000","last_became_active":"2023-10-16T11:21:20.876264+0000","last_became_peered":"2023-10-16T11:21:20.876264+0000","last_unstale":"2023-10-16T11:21:22.953381+0000","last_undegraded":"2023-10-16T11:21:22.953381+0000","last_fullsized":"2023-10-16T11:21:22.953381+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:59.922507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883454+0000","last_change":"2023-10-16T11:21:19.857352+0000","last_active":"2023-10-16T11:21:20.883454+0000","last_peered":"2023-10-16T11:21:20.883454+0000","last_clean":"2023-10-16T11:21:20.883454+0000","last_became_active":"2023-10-16T11:21:19.857078+0000","last_became_peered":"2023-10-16T11:21:19.857078+0000","last_unstale":"2023-10-16T11:21:20.883454+0000","last_undegraded":"2023-10-16T11:21:20.883454+0000","last_fullsized":"2023-10-16T11:21:20.883454+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:06.084643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856199+0000","last_change":"2023-10-16T11:21:20.876206+0000","last_active":"2023-10-16T11:21:21.856199+0000","last_peered":"2023-10-16T11:21:21.856199+0000","last_clean":"2023-10-16T11:21:21.856199+0000","last_became_active":"2023-10-16T11:21:20.875903+0000","last_became_peered":"2023-10-16T11:21:20.875903+0000","last_unstale":"2023-10-16T11:21:21.856199+0000","last_undegraded":"2023-10-16T11:21:21.856199+0000","last_fullsized":"2023-10-16T11:21:21.856199+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:26.577607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882006+0000","last_change":"2023-10-16T11:21:19.848262+0000","last_active":"2023-10-16T11:21:20.882006+0000","last_peered":"2023-10-16T11:21:20.882006+0000","last_clean":"2023-10-16T11:21:20.882006+0000","last_became_active":"2023-10-16T11:21:19.847989+0000","last_became_peered":"2023-10-16T11:21:19.847989+0000","last_unstale":"2023-10-16T11:21:20.882006+0000","last_undegraded":"2023-10-16T11:21:20.882006+0000","last_fullsized":"2023-10-16T11:21:20.882006+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:37.505082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856231+0000","last_change":"2023-10-16T11:21:20.875941+0000","last_active":"2023-10-16T11:21:21.856231+0000","last_peered":"2023-10-16T11:21:21.856231+0000","last_clean":"2023-10-16T11:21:21.856231+0000","last_became_active":"2023-10-16T11:21:20.874949+0000","last_became_peered":"2023-10-16T11:21:20.874949+0000","last_unstale":"2023-10-16T11:21:21.856231+0000","last_undegraded":"2023-10-16T11:21:21.856231+0000","last_fullsized":"2023-10-16T11:21:21.856231+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:24.359753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863017+0000","last_change":"2023-10-16T11:21:19.854706+0000","last_active":"2023-10-16T11:21:20.863017+0000","last_peered":"2023-10-16T11:21:20.863017+0000","last_clean":"2023-10-16T11:21:20.863017+0000","last_became_active":"2023-10-16T11:21:19.854183+0000","last_became_peered":"2023-10-16T11:21:19.854183+0000","last_unstale":"2023-10-16T11:21:20.863017+0000","last_undegraded":"2023-10-16T11:21:20.863017+0000","last_fullsized":"2023-10-16T11:21:20.863017+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:39.654182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858431+0000","last_change":"2023-10-16T11:21:20.864001+0000","last_active":"2023-10-16T11:21:21.858431+0000","last_peered":"2023-10-16T11:21:21.858431+0000","last_clean":"2023-10-16T11:21:21.858431+0000","last_became_active":"2023-10-16T11:21:20.863710+0000","last_became_peered":"2023-10-16T11:21:20.863710+0000","last_unstale":"2023-10-16T11:21:21.858431+0000","last_undegraded":"2023-10-16T11:21:21.858431+0000","last_fullsized":"2023-10-16T11:21:21.858431+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:42.568368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.856666+0000","last_change":"2023-10-16T11:21:19.854075+0000","last_active":"2023-10-16T11:21:20.856666+0000","last_peered":"2023-10-16T11:21:20.856666+0000","last_clean":"2023-10-16T11:21:20.856666+0000","last_became_active":"2023-10-16T11:21:19.853748+0000","last_became_peered":"2023-10-16T11:21:19.853748+0000","last_unstale":"2023-10-16T11:21:20.856666+0000","last_undegraded":"2023-10-16T11:21:20.856666+0000","last_fullsized":"2023-10-16T11:21:20.856666+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:32.125504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952922+0000","last_change":"2023-10-16T11:21:20.876041+0000","last_active":"2023-10-16T11:21:22.952922+0000","last_peered":"2023-10-16T11:21:22.952922+0000","last_clean":"2023-10-16T11:21:22.952922+0000","last_became_active":"2023-10-16T11:21:20.875530+0000","last_became_peered":"2023-10-16T11:21:20.875530+0000","last_unstale":"2023-10-16T11:21:22.952922+0000","last_undegraded":"2023-10-16T11:21:22.952922+0000","last_fullsized":"2023-10-16T11:21:22.952922+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:37.610269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.858487+0000","last_change":"2023-10-16T11:21:19.855733+0000","last_active":"2023-10-16T11:21:20.858487+0000","last_peered":"2023-10-16T11:21:20.858487+0000","last_clean":"2023-10-16T11:21:20.858487+0000","last_became_active":"2023-10-16T11:21:19.855458+0000","last_became_peered":"2023-10-16T11:21:19.855458+0000","last_unstale":"2023-10-16T11:21:20.858487+0000","last_undegraded":"2023-10-16T11:21:20.858487+0000","last_fullsized":"2023-10-16T11:21:20.858487+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:12.707047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953620+0000","last_change":"2023-10-16T11:21:20.873318+0000","last_active":"2023-10-16T11:21:22.953620+0000","last_peered":"2023-10-16T11:21:22.953620+0000","last_clean":"2023-10-16T11:21:22.953620+0000","last_became_active":"2023-10-16T11:21:20.873009+0000","last_became_peered":"2023-10-16T11:21:20.873009+0000","last_unstale":"2023-10-16T11:21:22.953620+0000","last_undegraded":"2023-10-16T11:21:22.953620+0000","last_fullsized":"2023-10-16T11:21:22.953620+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:34.913555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863998+0000","last_change":"2023-10-16T11:21:19.854559+0000","last_active":"2023-10-16T11:21:20.863998+0000","last_peered":"2023-10-16T11:21:20.863998+0000","last_clean":"2023-10-16T11:21:20.863998+0000","last_became_active":"2023-10-16T11:21:19.854175+0000","last_became_peered":"2023-10-16T11:21:19.854175+0000","last_unstale":"2023-10-16T11:21:20.863998+0000","last_undegraded":"2023-10-16T11:21:20.863998+0000","last_fullsized":"2023-10-16T11:21:20.863998+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:20:47.814076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953153+0000","last_change":"2023-10-16T11:21:20.860947+0000","last_active":"2023-10-16T11:21:22.953153+0000","last_peered":"2023-10-16T11:21:22.953153+0000","last_clean":"2023-10-16T11:21:22.953153+0000","last_became_active":"2023-10-16T11:21:20.860550+0000","last_became_peered":"2023-10-16T11:21:20.860550+0000","last_unstale":"2023-10-16T11:21:22.953153+0000","last_undegraded":"2023-10-16T11:21:22.953153+0000","last_fullsized":"2023-10-16T11:21:22.953153+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:47.084611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"256'3","reported_seq":15,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883631+0000","last_change":"2023-10-16T11:21:19.852566+0000","last_active":"2023-10-16T11:21:20.883631+0000","last_peered":"2023-10-16T11:21:20.883631+0000","last_clean":"2023-10-16T11:21:20.883631+0000","last_became_active":"2023-10-16T11:21:19.852230+0000","last_became_peered":"2023-10-16T11:21:19.852230+0000","last_unstale":"2023-10-16T11:21:20.883631+0000","last_undegraded":"2023-10-16T11:21:20.883631+0000","last_fullsized":"2023-10-16T11:21:20.883631+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:27.802569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952933+0000","last_change":"2023-10-16T11:21:20.879825+0000","last_active":"2023-10-16T11:21:22.952933+0000","last_peered":"2023-10-16T11:21:22.952933+0000","last_clean":"2023-10-16T11:21:22.952933+0000","last_became_active":"2023-10-16T11:21:20.879516+0000","last_became_peered":"2023-10-16T11:21:20.879516+0000","last_unstale":"2023-10-16T11:21:22.952933+0000","last_undegraded":"2023-10-16T11:21:22.952933+0000","last_fullsized":"2023-10-16T11:21:22.952933+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:36.014320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"256'6","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.888143+0000","last_change":"2023-10-16T11:21:19.852735+0000","last_active":"2023-10-16T11:21:20.888143+0000","last_peered":"2023-10-16T11:21:20.888143+0000","last_clean":"2023-10-16T11:21:20.888143+0000","last_became_active":"2023-10-16T11:21:19.852491+0000","last_became_peered":"2023-10-16T11:21:19.852491+0000","last_unstale":"2023-10-16T11:21:20.888143+0000","last_undegraded":"2023-10-16T11:21:20.888143+0000","last_fullsized":"2023-10-16T11:21:20.888143+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:25.572752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953659+0000","last_change":"2023-10-16T11:21:20.870915+0000","last_active":"2023-10-16T11:21:22.953659+0000","last_peered":"2023-10-16T11:21:22.953659+0000","last_clean":"2023-10-16T11:21:22.953659+0000","last_became_active":"2023-10-16T11:21:20.870368+0000","last_became_peered":"2023-10-16T11:21:20.870368+0000","last_unstale":"2023-10-16T11:21:22.953659+0000","last_undegraded":"2023-10-16T11:21:22.953659+0000","last_fullsized":"2023-10-16T11:21:22.953659+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:33.243878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.884015+0000","last_change":"2023-10-16T11:21:19.856454+0000","last_active":"2023-10-16T11:21:20.884015+0000","last_peered":"2023-10-16T11:21:20.884015+0000","last_clean":"2023-10-16T11:21:20.884015+0000","last_became_active":"2023-10-16T11:21:19.856175+0000","last_became_peered":"2023-10-16T11:21:19.856175+0000","last_unstale":"2023-10-16T11:21:20.884015+0000","last_undegraded":"2023-10-16T11:21:20.884015+0000","last_fullsized":"2023-10-16T11:21:20.884015+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:20.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953611+0000","last_change":"2023-10-16T11:21:20.871038+0000","last_active":"2023-10-16T11:21:22.953611+0000","last_peered":"2023-10-16T11:21:22.953611+0000","last_clean":"2023-10-16T11:21:22.953611+0000","last_became_active":"2023-10-16T11:21:20.870582+0000","last_became_peered":"2023-10-16T11:21:20.870582+0000","last_unstale":"2023-10-16T11:21:22.953611+0000","last_undegraded":"2023-10-16T11:21:22.953611+0000","last_fullsized":"2023-10-16T11:21:22.953611+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:04.917463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.866777+0000","last_change":"2023-10-16T11:21:19.858731+0000","last_active":"2023-10-16T11:21:20.866777+0000","last_peered":"2023-10-16T11:21:20.866777+0000","last_clean":"2023-10-16T11:21:20.866777+0000","last_became_active":"2023-10-16T11:21:19.858433+0000","last_became_peered":"2023-10-16T11:21:19.858433+0000","last_unstale":"2023-10-16T11:21:20.866777+0000","last_undegraded":"2023-10-16T11:21:20.866777+0000","last_fullsized":"2023-10-16T11:21:20.866777+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:45.296228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858059+0000","last_change":"2023-10-16T11:21:20.864542+0000","last_active":"2023-10-16T11:21:21.858059+0000","last_peered":"2023-10-16T11:21:21.858059+0000","last_clean":"2023-10-16T11:21:21.858059+0000","last_became_active":"2023-10-16T11:21:20.864117+0000","last_became_peered":"2023-10-16T11:21:20.864117+0000","last_unstale":"2023-10-16T11:21:21.858059+0000","last_undegraded":"2023-10-16T11:21:21.858059+0000","last_fullsized":"2023-10-16T11:21:21.858059+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:38.844676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.868419+0000","last_change":"2023-10-16T11:21:19.855209+0000","last_active":"2023-10-16T11:21:20.868419+0000","last_peered":"2023-10-16T11:21:20.868419+0000","last_clean":"2023-10-16T11:21:20.868419+0000","last_became_active":"2023-10-16T11:21:19.853190+0000","last_became_peered":"2023-10-16T11:21:19.853190+0000","last_unstale":"2023-10-16T11:21:20.868419+0000","last_undegraded":"2023-10-16T11:21:20.868419+0000","last_fullsized":"2023-10-16T11:21:20.868419+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:16.416230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858408+0000","last_change":"2023-10-16T11:21:20.860460+0000","last_active":"2023-10-16T11:21:21.858408+0000","last_peered":"2023-10-16T11:21:21.858408+0000","last_clean":"2023-10-16T11:21:21.858408+0000","last_became_active":"2023-10-16T11:21:20.860108+0000","last_became_peered":"2023-10-16T11:21:20.860108+0000","last_unstale":"2023-10-16T11:21:21.858408+0000","last_undegraded":"2023-10-16T11:21:21.858408+0000","last_fullsized":"2023-10-16T11:21:21.858408+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:32:22.973532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"256'8","reported_seq":20,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.887160+0000","last_change":"2023-10-16T11:21:19.855640+0000","last_active":"2023-10-16T11:21:20.887160+0000","last_peered":"2023-10-16T11:21:20.887160+0000","last_clean":"2023-10-16T11:21:20.887160+0000","last_became_active":"2023-10-16T11:21:19.854814+0000","last_became_peered":"2023-10-16T11:21:19.854814+0000","last_unstale":"2023-10-16T11:21:20.887160+0000","last_undegraded":"2023-10-16T11:21:20.887160+0000","last_fullsized":"2023-10-16T11:21:20.887160+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:58.546367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952411+0000","last_change":"2023-10-16T11:21:20.868828+0000","last_active":"2023-10-16T11:21:22.952411+0000","last_peered":"2023-10-16T11:21:22.952411+0000","last_clean":"2023-10-16T11:21:22.952411+0000","last_became_active":"2023-10-16T11:21:20.868519+0000","last_became_peered":"2023-10-16T11:21:20.868519+0000","last_unstale":"2023-10-16T11:21:22.952411+0000","last_undegraded":"2023-10-16T11:21:22.952411+0000","last_fullsized":"2023-10-16T11:21:22.952411+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:05.218119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882022+0000","last_change":"2023-10-16T11:21:19.856057+0000","last_active":"2023-10-16T11:21:20.882022+0000","last_peered":"2023-10-16T11:21:20.882022+0000","last_clean":"2023-10-16T11:21:20.882022+0000","last_became_active":"2023-10-16T11:21:19.855623+0000","last_became_peered":"2023-10-16T11:21:19.855623+0000","last_unstale":"2023-10-16T11:21:20.882022+0000","last_undegraded":"2023-10-16T11:21:20.882022+0000","last_fullsized":"2023-10-16T11:21:20.882022+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:07.905060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856513+0000","last_change":"2023-10-16T11:21:20.873473+0000","last_active":"2023-10-16T11:21:21.856513+0000","last_peered":"2023-10-16T11:21:21.856513+0000","last_clean":"2023-10-16T11:21:21.856513+0000","last_became_active":"2023-10-16T11:21:20.873205+0000","last_became_peered":"2023-10-16T11:21:20.873205+0000","last_unstale":"2023-10-16T11:21:21.856513+0000","last_undegraded":"2023-10-16T11:21:21.856513+0000","last_fullsized":"2023-10-16T11:21:21.856513+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:51:15.955800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.882535+0000","last_change":"2023-10-16T11:21:19.857447+0000","last_active":"2023-10-16T11:21:20.882535+0000","last_peered":"2023-10-16T11:21:20.882535+0000","last_clean":"2023-10-16T11:21:20.882535+0000","last_became_active":"2023-10-16T11:21:19.857115+0000","last_became_peered":"2023-10-16T11:21:19.857115+0000","last_unstale":"2023-10-16T11:21:20.882535+0000","last_undegraded":"2023-10-16T11:21:20.882535+0000","last_fullsized":"2023-10-16T11:21:20.882535+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:20:47.814076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952731+0000","last_change":"2023-10-16T11:21:20.871844+0000","last_active":"2023-10-16T11:21:22.952731+0000","last_peered":"2023-10-16T11:21:22.952731+0000","last_clean":"2023-10-16T11:21:22.952731+0000","last_became_active":"2023-10-16T11:21:20.871521+0000","last_became_peered":"2023-10-16T11:21:20.871521+0000","last_unstale":"2023-10-16T11:21:22.952731+0000","last_undegraded":"2023-10-16T11:21:22.952731+0000","last_fullsized":"2023-10-16T11:21:22.952731+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:06.489484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"256'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.883580+0000","last_change":"2023-10-16T11:21:19.854671+0000","last_active":"2023-10-16T11:21:20.883580+0000","last_peered":"2023-10-16T11:21:20.883580+0000","last_clean":"2023-10-16T11:21:20.883580+0000","last_became_active":"2023-10-16T11:21:19.854334+0000","last_became_peered":"2023-10-16T11:21:19.854334+0000","last_unstale":"2023-10-16T11:21:20.883580+0000","last_undegraded":"2023-10-16T11:21:20.883580+0000","last_fullsized":"2023-10-16T11:21:20.883580+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:50.695229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953386+0000","last_change":"2023-10-16T11:21:20.869590+0000","last_active":"2023-10-16T11:21:22.953386+0000","last_peered":"2023-10-16T11:21:22.953386+0000","last_clean":"2023-10-16T11:21:22.953386+0000","last_became_active":"2023-10-16T11:21:20.869076+0000","last_became_peered":"2023-10-16T11:21:20.869076+0000","last_unstale":"2023-10-16T11:21:22.953386+0000","last_undegraded":"2023-10-16T11:21:22.953386+0000","last_fullsized":"2023-10-16T11:21:22.953386+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:15.463518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863025+0000","last_change":"2023-10-16T11:21:19.859330+0000","last_active":"2023-10-16T11:21:20.863025+0000","last_peered":"2023-10-16T11:21:20.863025+0000","last_clean":"2023-10-16T11:21:20.863025+0000","last_became_active":"2023-10-16T11:21:19.859011+0000","last_became_peered":"2023-10-16T11:21:19.859011+0000","last_unstale":"2023-10-16T11:21:20.863025+0000","last_undegraded":"2023-10-16T11:21:20.863025+0000","last_fullsized":"2023-10-16T11:21:20.863025+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:41.892036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856451+0000","last_change":"2023-10-16T11:21:20.873499+0000","last_active":"2023-10-16T11:21:21.856451+0000","last_peered":"2023-10-16T11:21:21.856451+0000","last_clean":"2023-10-16T11:21:21.856451+0000","last_became_active":"2023-10-16T11:21:20.873180+0000","last_became_peered":"2023-10-16T11:21:20.873180+0000","last_unstale":"2023-10-16T11:21:21.856451+0000","last_undegraded":"2023-10-16T11:21:21.856451+0000","last_fullsized":"2023-10-16T11:21:21.856451+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:19.234265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.884623+0000","last_change":"2023-10-16T11:21:19.858156+0000","last_active":"2023-10-16T11:21:20.884623+0000","last_peered":"2023-10-16T11:21:20.884623+0000","last_clean":"2023-10-16T11:21:20.884623+0000","last_became_active":"2023-10-16T11:21:19.857859+0000","last_became_peered":"2023-10-16T11:21:19.857859+0000","last_unstale":"2023-10-16T11:21:20.884623+0000","last_undegraded":"2023-10-16T11:21:20.884623+0000","last_fullsized":"2023-10-16T11:21:20.884623+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:22.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.856345+0000","last_change":"2023-10-16T11:21:20.867387+0000","last_active":"2023-10-16T11:21:21.856345+0000","last_peered":"2023-10-16T11:21:21.856345+0000","last_clean":"2023-10-16T11:21:21.856345+0000","last_became_active":"2023-10-16T11:21:20.867151+0000","last_became_peered":"2023-10-16T11:21:20.867151+0000","last_unstale":"2023-10-16T11:21:21.856345+0000","last_undegraded":"2023-10-16T11:21:21.856345+0000","last_fullsized":"2023-10-16T11:21:21.856345+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:57.278022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.855333+0000","last_change":"2023-10-16T11:21:19.853584+0000","last_active":"2023-10-16T11:21:20.855333+0000","last_peered":"2023-10-16T11:21:20.855333+0000","last_clean":"2023-10-16T11:21:20.855333+0000","last_became_active":"2023-10-16T11:21:19.853315+0000","last_became_peered":"2023-10-16T11:21:19.853315+0000","last_unstale":"2023-10-16T11:21:20.855333+0000","last_undegraded":"2023-10-16T11:21:20.855333+0000","last_fullsized":"2023-10-16T11:21:20.855333+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:44.852756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.858206+0000","last_change":"2023-10-16T11:21:20.864407+0000","last_active":"2023-10-16T11:21:21.858206+0000","last_peered":"2023-10-16T11:21:21.858206+0000","last_clean":"2023-10-16T11:21:21.858206+0000","last_became_active":"2023-10-16T11:21:20.863739+0000","last_became_peered":"2023-10-16T11:21:20.863739+0000","last_unstale":"2023-10-16T11:21:21.858206+0000","last_undegraded":"2023-10-16T11:21:21.858206+0000","last_fullsized":"2023-10-16T11:21:21.858206+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:43.954316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.868010+0000","last_change":"2023-10-16T11:21:19.853083+0000","last_active":"2023-10-16T11:21:20.868010+0000","last_peered":"2023-10-16T11:21:20.868010+0000","last_clean":"2023-10-16T11:21:20.868010+0000","last_became_active":"2023-10-16T11:21:19.852799+0000","last_became_peered":"2023-10-16T11:21:19.852799+0000","last_unstale":"2023-10-16T11:21:20.868010+0000","last_undegraded":"2023-10-16T11:21:20.868010+0000","last_fullsized":"2023-10-16T11:21:20.868010+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:54:35.615415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952450+0000","last_change":"2023-10-16T11:21:20.870505+0000","last_active":"2023-10-16T11:21:22.952450+0000","last_peered":"2023-10-16T11:21:22.952450+0000","last_clean":"2023-10-16T11:21:22.952450+0000","last_became_active":"2023-10-16T11:21:20.870248+0000","last_became_peered":"2023-10-16T11:21:20.870248+0000","last_unstale":"2023-10-16T11:21:22.952450+0000","last_undegraded":"2023-10-16T11:21:22.952450+0000","last_fullsized":"2023-10-16T11:21:22.952450+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:03.638407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.864897+0000","last_change":"2023-10-16T11:21:19.852581+0000","last_active":"2023-10-16T11:21:20.864897+0000","last_peered":"2023-10-16T11:21:20.864897+0000","last_clean":"2023-10-16T11:21:20.864897+0000","last_became_active":"2023-10-16T11:21:19.852113+0000","last_became_peered":"2023-10-16T11:21:19.852113+0000","last_unstale":"2023-10-16T11:21:20.864897+0000","last_undegraded":"2023-10-16T11:21:20.864897+0000","last_fullsized":"2023-10-16T11:21:20.864897+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:07.655056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953057+0000","last_change":"2023-10-16T11:21:20.876890+0000","last_active":"2023-10-16T11:21:22.953057+0000","last_peered":"2023-10-16T11:21:22.953057+0000","last_clean":"2023-10-16T11:21:22.953057+0000","last_became_active":"2023-10-16T11:21:20.876617+0000","last_became_peered":"2023-10-16T11:21:20.876617+0000","last_unstale":"2023-10-16T11:21:22.953057+0000","last_undegraded":"2023-10-16T11:21:22.953057+0000","last_fullsized":"2023-10-16T11:21:22.953057+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:36.015828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"256'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.884482+0000","last_change":"2023-10-16T11:21:19.858179+0000","last_active":"2023-10-16T11:21:20.884482+0000","last_peered":"2023-10-16T11:21:20.884482+0000","last_clean":"2023-10-16T11:21:20.884482+0000","last_became_active":"2023-10-16T11:21:19.857905+0000","last_became_peered":"2023-10-16T11:21:19.857905+0000","last_unstale":"2023-10-16T11:21:20.884482+0000","last_undegraded":"2023-10-16T11:21:20.884482+0000","last_fullsized":"2023-10-16T11:21:20.884482+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:04.213264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:21.857637+0000","last_change":"2023-10-16T11:21:20.866902+0000","last_active":"2023-10-16T11:21:21.857637+0000","last_peered":"2023-10-16T11:21:21.857637+0000","last_clean":"2023-10-16T11:21:21.857637+0000","last_became_active":"2023-10-16T11:21:20.866629+0000","last_became_peered":"2023-10-16T11:21:20.866629+0000","last_unstale":"2023-10-16T11:21:21.857637+0000","last_undegraded":"2023-10-16T11:21:21.857637+0000","last_fullsized":"2023-10-16T11:21:21.857637+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:24.438824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.867180+0000","last_change":"2023-10-16T11:21:19.858912+0000","last_active":"2023-10-16T11:21:20.867180+0000","last_peered":"2023-10-16T11:21:20.867180+0000","last_clean":"2023-10-16T11:21:20.867180+0000","last_became_active":"2023-10-16T11:21:19.858654+0000","last_became_peered":"2023-10-16T11:21:19.858654+0000","last_unstale":"2023-10-16T11:21:20.867180+0000","last_undegraded":"2023-10-16T11:21:20.867180+0000","last_fullsized":"2023-10-16T11:21:20.867180+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:56.204594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953229+0000","last_change":"2023-10-16T11:21:20.874240+0000","last_active":"2023-10-16T11:21:22.953229+0000","last_peered":"2023-10-16T11:21:22.953229+0000","last_clean":"2023-10-16T11:21:22.953229+0000","last_became_active":"2023-10-16T11:21:20.873584+0000","last_became_peered":"2023-10-16T11:21:20.873584+0000","last_unstale":"2023-10-16T11:21:22.953229+0000","last_undegraded":"2023-10-16T11:21:22.953229+0000","last_fullsized":"2023-10-16T11:21:22.953229+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:50.700023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.858382+0000","last_change":"2023-10-16T11:21:19.854345+0000","last_active":"2023-10-16T11:21:20.858382+0000","last_peered":"2023-10-16T11:21:20.858382+0000","last_clean":"2023-10-16T11:21:20.858382+0000","last_became_active":"2023-10-16T11:21:19.853927+0000","last_became_peered":"2023-10-16T11:21:19.853927+0000","last_unstale":"2023-10-16T11:21:20.858382+0000","last_undegraded":"2023-10-16T11:21:20.858382+0000","last_fullsized":"2023-10-16T11:21:20.858382+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:18.538107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953600+0000","last_change":"2023-10-16T11:21:20.864934+0000","last_active":"2023-10-16T11:21:22.953600+0000","last_peered":"2023-10-16T11:21:22.953600+0000","last_clean":"2023-10-16T11:21:22.953600+0000","last_became_active":"2023-10-16T11:21:20.864623+0000","last_became_peered":"2023-10-16T11:21:20.864623+0000","last_unstale":"2023-10-16T11:21:22.953600+0000","last_undegraded":"2023-10-16T11:21:22.953600+0000","last_fullsized":"2023-10-16T11:21:22.953600+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:00.130637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.863335+0000","last_change":"2023-10-16T11:21:19.856230+0000","last_active":"2023-10-16T11:21:20.863335+0000","last_peered":"2023-10-16T11:21:20.863335+0000","last_clean":"2023-10-16T11:21:20.863335+0000","last_became_active":"2023-10-16T11:21:19.855798+0000","last_became_peered":"2023-10-16T11:21:19.855798+0000","last_unstale":"2023-10-16T11:21:20.863335+0000","last_undegraded":"2023-10-16T11:21:20.863335+0000","last_fullsized":"2023-10-16T11:21:20.863335+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:03.626811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953221+0000","last_change":"2023-10-16T11:21:20.864277+0000","last_active":"2023-10-16T11:21:22.953221+0000","last_peered":"2023-10-16T11:21:22.953221+0000","last_clean":"2023-10-16T11:21:22.953221+0000","last_became_active":"2023-10-16T11:21:20.864001+0000","last_became_peered":"2023-10-16T11:21:20.864001+0000","last_unstale":"2023-10-16T11:21:22.953221+0000","last_undegraded":"2023-10-16T11:21:22.953221+0000","last_fullsized":"2023-10-16T11:21:22.953221+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:40.650230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.865172+0000","last_change":"2023-10-16T11:21:19.856119+0000","last_active":"2023-10-16T11:21:20.865172+0000","last_peered":"2023-10-16T11:21:20.865172+0000","last_clean":"2023-10-16T11:21:20.865172+0000","last_became_active":"2023-10-16T11:21:19.855791+0000","last_became_peered":"2023-10-16T11:21:19.855791+0000","last_unstale":"2023-10-16T11:21:20.865172+0000","last_undegraded":"2023-10-16T11:21:20.865172+0000","last_fullsized":"2023-10-16T11:21:20.865172+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:24.829226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953750+0000","last_change":"2023-10-16T11:21:20.863082+0000","last_active":"2023-10-16T11:21:22.953750+0000","last_peered":"2023-10-16T11:21:22.953750+0000","last_clean":"2023-10-16T11:21:22.953750+0000","last_became_active":"2023-10-16T11:21:20.862820+0000","last_became_peered":"2023-10-16T11:21:20.862820+0000","last_unstale":"2023-10-16T11:21:22.953750+0000","last_undegraded":"2023-10-16T11:21:22.953750+0000","last_fullsized":"2023-10-16T11:21:22.953750+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:58.851375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.856182+0000","last_change":"2023-10-16T11:21:19.853507+0000","last_active":"2023-10-16T11:21:20.856182+0000","last_peered":"2023-10-16T11:21:20.856182+0000","last_clean":"2023-10-16T11:21:20.856182+0000","last_became_active":"2023-10-16T11:21:19.853225+0000","last_became_peered":"2023-10-16T11:21:19.853225+0000","last_unstale":"2023-10-16T11:21:20.856182+0000","last_undegraded":"2023-10-16T11:21:20.856182+0000","last_fullsized":"2023-10-16T11:21:20.856182+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:02.656474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.952782+0000","last_change":"2023-10-16T11:21:20.868118+0000","last_active":"2023-10-16T11:21:22.952782+0000","last_peered":"2023-10-16T11:21:22.952782+0000","last_clean":"2023-10-16T11:21:22.952782+0000","last_became_active":"2023-10-16T11:21:20.867840+0000","last_became_peered":"2023-10-16T11:21:20.867840+0000","last_unstale":"2023-10-16T11:21:22.952782+0000","last_undegraded":"2023-10-16T11:21:22.952782+0000","last_fullsized":"2023-10-16T11:21:22.952782+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:30.453547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.858004+0000","last_change":"2023-10-16T11:21:19.854495+0000","last_active":"2023-10-16T11:21:20.858004+0000","last_peered":"2023-10-16T11:21:20.858004+0000","last_clean":"2023-10-16T11:21:20.858004+0000","last_became_active":"2023-10-16T11:21:19.854178+0000","last_became_peered":"2023-10-16T11:21:19.854178+0000","last_unstale":"2023-10-16T11:21:20.858004+0000","last_undegraded":"2023-10-16T11:21:20.858004+0000","last_fullsized":"2023-10-16T11:21:20.858004+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:00.550117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.953714+0000","last_change":"2023-10-16T11:21:20.870231+0000","last_active":"2023-10-16T11:21:22.953714+0000","last_peered":"2023-10-16T11:21:22.953714+0000","last_clean":"2023-10-16T11:21:22.953714+0000","last_became_active":"2023-10-16T11:21:20.869973+0000","last_became_peered":"2023-10-16T11:21:20.869973+0000","last_unstale":"2023-10-16T11:21:22.953714+0000","last_undegraded":"2023-10-16T11:21:22.953714+0000","last_fullsized":"2023-10-16T11:21:22.953714+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:14:30.942915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.865043+0000","last_change":"2023-10-16T11:21:19.855368+0000","last_active":"2023-10-16T11:21:20.865043+0000","last_peered":"2023-10-16T11:21:20.865043+0000","last_clean":"2023-10-16T11:21:20.865043+0000","last_became_active":"2023-10-16T11:21:19.854895+0000","last_became_peered":"2023-10-16T11:21:19.854895+0000","last_unstale":"2023-10-16T11:21:20.865043+0000","last_undegraded":"2023-10-16T11:21:20.865043+0000","last_fullsized":"2023-10-16T11:21:20.865043+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:18.827284+0000","last_clean_scrub_stamp":"2023-10-16T11:21:18.827284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:05.484629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":11,"reported_epoch":257,"state":"active+clean","last_fresh":"2023-10-16T11:21:22.412651+0000","last_change":"2023-10-16T11:21:20.874328+0000","last_active":"2023-10-16T11:21:22.412651+0000","last_peered":"2023-10-16T11:21:22.412651+0000","last_clean":"2023-10-16T11:21:22.412651+0000","last_became_active":"2023-10-16T11:21:20.874076+0000","last_became_peered":"2023-10-16T11:21:20.874076+0000","last_unstale":"2023-10-16T11:21:22.412651+0000","last_undegraded":"2023-10-16T11:21:22.412651+0000","last_fullsized":"2023-10-16T11:21:22.412651+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:19.832139+0000","last_clean_scrub_stamp":"2023-10-16T11:21:19.832139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:51.610368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"243'381","reported_seq":1365,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T11:21:20.859644+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:21:20.859644+0000","last_peered":"2023-10-16T11:21:20.859644+0000","last_clean":"2023-10-16T11:21:20.859644+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:21:20.859644+0000","last_undegraded":"2023-10-16T11:21:20.859644+0000","last_fullsized":"2023-10-16T11:21:20.859644+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53012,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318828,"statfs":{"total":96636764160,"available":96582479872,"internally_reserved":0,"allocated":3874816,"data_stored":2353352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64600000000000002,"15min":0.60699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68999999999999995,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.66400000000000003,"15min":0.628},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.68799999999999994,"15min":0.66200000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.622,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.70099999999999996,"15min":0.67400000000000004},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.71199999999999997,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.71399999999999997,"15min":0.67300000000000004},"min":{"1min":0.48099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78900000000000003,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.73299999999999998,"15min":0.69099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":9.516,"15min":9.516},"last":0.42399999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.68300000000000005,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.68799999999999994,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.75900000000000001,"15min":0.71199999999999997},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.61099999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.54100000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32300000000000001},"max":{"1min":0.747,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.37},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.63200000000000001,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.623}]},{"osd":5,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65300000000000002,"15min":0.57799999999999996},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.33700000000000002},"max":{"1min":0.98399999999999999,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69699999999999995,"15min":0.63},"min":{"1min":0.41499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79500000000000004,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:20:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.71599999999999997,"15min":0.64600000000000002},"min":{"1min":0.41599999999999998,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.84199999999999997,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.502},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.69699999999999995,"15min":0.64200000000000002},"min":{"1min":0.442,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76600000000000001,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.65000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56572,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315268,"statfs":{"total":96636764160,"available":96578834432,"internally_reserved":0,"allocated":3260416,"data_stored":1760720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75800000000000001,"5min":0.80700000000000005,"15min":0.86299999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.59199999999999997,"15min":0.61399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69399999999999995,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.71699999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58099999999999996,"15min":0.60599999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73499999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":2.387,"15min":2.387},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.57199999999999995,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.314},"max":{"1min":0.58699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":0.62},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.622,"15min":0.621},"min":{"1min":0.36499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.65300000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.36399999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71599999999999997,"5min":1.02,"15min":1.02},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.32900000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.55500000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.77800000000000002,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.45400000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76200000000000001,"5min":3.335,"15min":3.335},"last":0.79200000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":2.524,"15min":2.524},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.752,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.70399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52420,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319420,"statfs":{"total":96636764160,"available":96583086080,"internally_reserved":0,"allocated":3264512,"data_stored":1760810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.38700000000000001,"15min":0.40799999999999997},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51000000000000001,"5min":0.622,"15min":0.66700000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53300000000000003,"5min":0.63900000000000001,"15min":0.68799999999999994},"last":0.37}]},{"osd":2,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.375,"15min":0.40100000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.439,"5min":0.57299999999999995,"15min":0.67300000000000004},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.41099999999999998,"15min":0.44400000000000001},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48899999999999999,"5min":0.623,"15min":0.70099999999999996},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.441,"15min":0.46600000000000003},"min":{"1min":0.30299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.53200000000000003,"5min":0.66100000000000003,"15min":0.73399999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41399999999999998,"15min":0.437},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.61799999999999999,"5min":0.623,"15min":0.65600000000000003},"last":0.39800000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.624,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73399999999999999,"5min":0.91300000000000003,"15min":1.0309999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.64300000000000002,"15min":0.68700000000000006},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.86299999999999999,"5min":0.91600000000000004,"15min":1.0389999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.70599999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89100000000000001,"5min":0.95299999999999996,"15min":1.0760000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64100000000000001,"15min":0.66700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":1.0149999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.64400000000000002,"15min":0.67800000000000005},"min":{"1min":0.43099999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79300000000000004,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.69899999999999995},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89300000000000002,"5min":0.89800000000000002,"15min":1.0620000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.44},"max":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":1.016},"last":0.67200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61328,"kb_used_data":3772,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310512,"statfs":{"total":96636764160,"available":96573964288,"internally_reserved":0,"allocated":3862528,"data_stored":2351704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.45000000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.26700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52800000000000002,"5min":0.66100000000000003,"15min":0.77200000000000002},"last":0.19500000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":1.272,"15min":1.272},"last":0.315},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.28299999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40899999999999997,"15min":0.433},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.216},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54200000000000004,"5min":1.351,"15min":1.351},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83099999999999996,"5min":1.391,"15min":1.391},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76000000000000001,"5min":0.874,"15min":1.089},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.624,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.95599999999999996,"15min":1.216},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.47199999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.1180000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.67200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80300000000000005,"5min":0.97399999999999998,"15min":1.0249999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80500000000000005,"5min":0.89600000000000002,"15min":1.0780000000000001},"last":0.52800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834576010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52412,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319428,"statfs":{"total":96636764160,"available":96583094272,"internally_reserved":0,"allocated":3260416,"data_stored":1760720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.245,"5min":0.188,"15min":0.188},"max":{"1min":0.47699999999999998,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.437},"min":{"1min":0.255,"5min":0.193,"15min":0.193},"max":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.436,"15min":0.46400000000000002},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.41099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45300000000000001,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.36499999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":1.097},"last":0.627},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86499999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.67100000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83099999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92700000000000005,"5min":1.0169999999999999,"15min":1.028},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.498,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85899999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.70399999999999996},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":0.99099999999999999,"15min":1.071},"last":0.62},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.97799999999999998,"15min":1.077},"last":0.69799999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576010,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56580,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315260,"statfs":{"total":96636764160,"available":96578826240,"internally_reserved":0,"allocated":3268608,"data_stored":1760812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36499999999999999,"15min":0.39000000000000001},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.621,"5min":1.032,"15min":1.032},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.36799999999999999,"15min":0.40500000000000003},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.52200000000000002,"5min":1.341,"15min":1.341},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40899999999999997,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55000000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.44400000000000001,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.51300000000000001,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.313},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45900000000000002,"15min":0.503},"min":{"1min":0.34799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54600000000000004,"5min":0.96599999999999997,"15min":1.024},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.623,"15min":0.65300000000000002},"min":{"1min":0.45500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3220000000000001,"5min":2.629,"15min":2.629},"last":0.76400000000000001}]},{"osd":5,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.66700000000000004},"min":{"1min":0.443,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":1.0149999999999999,"15min":1.026},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.746}]},{"osd":6,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81100000000000005,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.80400000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":0.71399999999999997},"min":{"1min":0.49099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.304,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65800000000000003,"15min":0.69899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.86399999999999999,"5min":1.006,"15min":1.1899999999999999},"last":0.72199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52436,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319404,"statfs":{"total":96636764160,"available":96583069696,"internally_reserved":0,"allocated":3276800,"data_stored":1762422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.60899999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.61399999999999999},"min":{"1min":0.41899999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81899999999999995,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.60799999999999998},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.95499999999999996},"last":0.441}]},{"osd":2,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57199999999999995,"15min":0.61699999999999999},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81100000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.73599999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60899999999999999,"15min":0.63200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.86199999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76000000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.71599999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.82899999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.751},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.35799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":1.115},"last":0.81499999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.80000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66316,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305524,"statfs":{"total":96636764160,"available":96568856576,"internally_reserved":0,"allocated":3268608,"data_stored":1760776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58599999999999997,"15min":0.623},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28999999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.64300000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.88200000000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67900000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74299999999999999,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53700000000000003,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.623,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81899999999999995,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.747}]},{"osd":3,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79300000000000004,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.59899999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.498},{"interface":"front","average":{"1min":0.503,"5min":0.496,"15min":0.51600000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83599999999999997,"5min":0.878,"15min":0.878},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.874},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.623,"15min":0.63700000000000001},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:21:25.358 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:21:25.359 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T11:21:25.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.519+0000 7ff6f1565700 1 Processor -- start 2023-10-16T11:21:25.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.519+0000 7ff6f1565700 1 -- start start 2023-10-16T11:21:25.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6f1565700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6ec06b9e0 0x7ff6ec06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6f1565700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff6ec070530 0x7ff6ec070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6f1565700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec161440 con 0x7ff6ec06ec50 2023-10-16T11:21:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6f1565700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ec06c300 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6f1565700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ec161670 con 0x7ff6ec070530 2023-10-16T11:21:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6eaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6ec06b9e0 0x7ff6ec06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6eaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff6ec06b9e0 0x7ff6ec06bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42340/0 (socket says 172.21.15.158:42340) 2023-10-16T11:21:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.520+0000 7ff6eaffd700 1 -- 172.21.15.158:0/3512620592 learned_addr learned my addr 172.21.15.158:0/3512620592 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6eaffd700 1 -- 172.21.15.158:0/3512620592 >> v2:172.21.15.172:3300/0 conn(0x7ff6ec070530 msgr2=0x7ff6ec070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6eaffd700 1 --2- 172.21.15.158:0/3512620592 >> v2:172.21.15.172:3300/0 conn(0x7ff6ec070530 0x7ff6ec070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6eaffd700 1 -- 172.21.15.158:0/3512620592 >> v1:172.21.15.172:6789/0 conn(0x7ff6ec06ec50 legacy=0x7ff6ec06f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6eaffd700 1 -- 172.21.15.158:0/3512620592 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec070ed0 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6eaffd700 1 --2- 172.21.15.158:0/3512620592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec06bdc0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec06ded0 tx=0x7ff6dc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35f04b3f8e23d36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6e9ffb700 1 -- 172.21.15.158:0/3512620592 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc004230 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6e9ffb700 1 -- 172.21.15.158:0/3512620592 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc004390 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6e9ffb700 1 -- 172.21.15.158:0/3512620592 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0036a0 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6f1565700 1 -- 172.21.15.158:0/3512620592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 msgr2=0x7ff6ec06bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6f1565700 1 --2- 172.21.15.158:0/3512620592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec06bdc0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec06ded0 tx=0x7ff6dc009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6f1565700 1 -- 172.21.15.158:0/3512620592 shutdown_connections 2023-10-16T11:21:25.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.521+0000 7ff6f1565700 1 --2- 172.21.15.158:0/3512620592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec06bdc0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.522+0000 7ff6f1565700 1 --2- 172.21.15.158:0/3512620592 >> v2:172.21.15.172:3300/0 conn(0x7ff6ec070530 0x7ff6ec070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.522+0000 7ff6f1565700 1 -- 172.21.15.158:0/3512620592 >> 172.21.15.158:0/3512620592 conn(0x7ff6ec0d1410 msgr2=0x7ff6ec0d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.522+0000 7ff6f1565700 1 -- 172.21.15.158:0/3512620592 shutdown_connections 2023-10-16T11:21:25.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.522+0000 7ff6f1565700 1 -- 172.21.15.158:0/3512620592 wait complete. 2023-10-16T11:21:25.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.523+0000 7ff6f1565700 1 Processor -- start 2023-10-16T11:21:25.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.523+0000 7ff6f1565700 1 -- start start 2023-10-16T11:21:25.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6f1565700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6f1565700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6ec06ec50 0x7ff6ec15d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6f1565700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6ec070530 0x7ff6ec0c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6f1565700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec15d680 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6f1565700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0c4bb0 con 0x7ff6ec06ec50 2023-10-16T11:21:25.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6f1565700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec15d870 con 0x7ff6ec070530 2023-10-16T11:21:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6eaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6eaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42348/0 (socket says 172.21.15.158:42348) 2023-10-16T11:21:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.524+0000 7ff6eaffd700 1 -- 172.21.15.158:0/2464382953 learned_addr learned my addr 172.21.15.158:0/2464382953 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6eb7fe700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6ec070530 0x7ff6ec0c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6ea7fc700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6ec06ec50 0x7ff6ec15d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6eaffd700 1 -- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6ec070530 msgr2=0x7ff6ec0c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6eaffd700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6ec070530 0x7ff6ec0c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6eaffd700 1 -- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6ec06ec50 msgr2=0x7ff6ec15d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6eaffd700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6ec06ec50 0x7ff6ec15d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6eaffd700 1 -- 172.21.15.158:0/2464382953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc009550 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6ea7fc700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6ec06ec50 0x7ff6ec15d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6eaffd700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec15cc00 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0b4000 tx=0x7ff6dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6d37fe700 1 -- 172.21.15.158:0/2464382953 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc003dc0 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6d37fe700 1 -- 172.21.15.158:0/2464382953 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc01f070 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec15db50 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6d37fe700 1 -- 172.21.15.158:0/2464382953 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc011870 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.525+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec15e110 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.526+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec0681b0 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.526+0000 7ff6d37fe700 1 -- 172.21.15.158:0/2464382953 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff6dc021030 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.527+0000 7ff6d37fe700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.527+0000 7ff6d37fe700 1 -- 172.21.15.158:0/2464382953 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff6dc08b2e0 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.527+0000 7ff6ea7fc700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.527+0000 7ff6ea7fc700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff6d806b140 0x7ff6d806d600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0c4eb0 tx=0x7ff6e0008040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.531+0000 7ff6d37fe700 1 -- 172.21.15.158:0/2464382953 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6dc055e70 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.803+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff6ec0d2300 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.804+0000 7ff6d37fe700 1 -- 172.21.15.158:0/2464382953 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v56) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff6dc059490 con 0x7ff6ec06b9e0 2023-10-16T11:21:25.806 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T11:21:25.806 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T11:21:25.807 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T11:21:25.807 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T11:21:25.807 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T11:21:25.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff6d806b140 msgr2=0x7ff6d806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff6d806b140 0x7ff6d806d600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0c4eb0 tx=0x7ff6e0008040 comp rx=0 tx=0).stop 2023-10-16T11:21:25.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 msgr2=0x7ff6ec15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec15cc00 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0b4000 tx=0x7ff6dc004750 comp rx=0 tx=0).stop 2023-10-16T11:21:25.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 shutdown_connections 2023-10-16T11:21:25.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6ec070530 0x7ff6ec0c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6ec06b9e0 0x7ff6ec15cc00 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 --2- 172.21.15.158:0/2464382953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6ec06ec50 0x7ff6ec15d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 >> 172.21.15.158:0/2464382953 conn(0x7ff6ec0d1410 msgr2=0x7ff6ec15ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:25.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 shutdown_connections 2023-10-16T11:21:25.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.807+0000 7ff6f1565700 1 -- 172.21.15.158:0/2464382953 wait complete. 2023-10-16T11:21:25.811 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T11:21:25.821 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:25.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.941+0000 7fb0fc9d6700 1 Processor -- start 2023-10-16T11:21:25.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.941+0000 7fb0fc9d6700 1 -- start start 2023-10-16T11:21:25.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0fc9d6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f806c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0fc9d6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb0f815d910 0x7fb0f8161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0fc9d6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f806bba0 con 0x7fb0f806d590 2023-10-16T11:21:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0fc9d6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f806be80 con 0x7fb0f815d910 2023-10-16T11:21:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0fc9d6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f806c160 con 0x7fb0f815d530 2023-10-16T11:21:25.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0f659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0f5d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb0f815d910 0x7fb0f8161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0f659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f806c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:53188/0 (socket says 172.21.15.158:53188) 2023-10-16T11:21:25.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.942+0000 7fb0f659c700 1 -- 172.21.15.158:0/1402321602 learned_addr learned my addr 172.21.15.158:0/1402321602 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:25.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f659c700 1 -- 172.21.15.158:0/1402321602 >> v2:172.21.15.172:3300/0 conn(0x7fb0f815d910 msgr2=0x7fb0f8161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f659c700 1 --2- 172.21.15.158:0/1402321602 >> v2:172.21.15.172:3300/0 conn(0x7fb0f815d910 0x7fb0f8161c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f659c700 1 -- 172.21.15.158:0/1402321602 >> v1:172.21.15.172:6789/0 conn(0x7fb0f806d590 legacy=0x7fb0f80cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f659c700 1 -- 172.21.15.158:0/1402321602 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f806c440 con 0x7fb0f815d530 2023-10-16T11:21:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f5d9b700 1 --2- 172.21.15.158:0/1402321602 >> v2:172.21.15.172:3300/0 conn(0x7fb0f815d910 0x7fb0f8161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f659c700 1 --2- 172.21.15.158:0/1402321602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f806c940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80098f0 tx=0x7fb0e8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3d8c4dbc967ef34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f559a700 1 -- 172.21.15.158:0/1402321602 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8011830 con 0x7fb0f815d530 2023-10-16T11:21:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f559a700 1 -- 172.21.15.158:0/1402321602 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e8011990 con 0x7fb0f815d530 2023-10-16T11:21:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0f559a700 1 -- 172.21.15.158:0/1402321602 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8011d20 con 0x7fb0f815d530 2023-10-16T11:21:25.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1402321602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 msgr2=0x7fb0f806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.943+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1402321602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f806c940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80098f0 tx=0x7fb0e8009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.944+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1402321602 shutdown_connections 2023-10-16T11:21:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.944+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1402321602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f806c940 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.944+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1402321602 >> v2:172.21.15.172:3300/0 conn(0x7fb0f815d910 0x7fb0f8161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.944+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1402321602 >> 172.21.15.158:0/1402321602 conn(0x7fb0f8071180 msgr2=0x7fb0f806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:25.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.944+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1402321602 shutdown_connections 2023-10-16T11:21:25.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.944+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1402321602 wait complete. 2023-10-16T11:21:25.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.945+0000 7fb0fc9d6700 1 Processor -- start 2023-10-16T11:21:25.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.945+0000 7fb0fc9d6700 1 -- start start 2023-10-16T11:21:25.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0fc9d6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0f806d590 0x7fb0f81582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0fc9d6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f80c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0fc9d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0f815d910 0x7fb0f80b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0fc9d6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f815e5c0 con 0x7fb0f815d910 2023-10-16T11:21:25.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0fc9d6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f80c6d50 con 0x7fb0f806d590 2023-10-16T11:21:25.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0fc9d6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f815dcf0 con 0x7fb0f815d530 2023-10-16T11:21:25.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0f815d910 0x7fb0f80b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0f815d910 0x7fb0f80b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42364/0 (socket says 172.21.15.158:42364) 2023-10-16T11:21:25.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 -- 172.21.15.158:0/1708136426 learned_addr learned my addr 172.21.15.158:0/1708136426 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:25.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f5d9b700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f80c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f659c700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0f806d590 0x7fb0f81582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 -- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 msgr2=0x7fb0f80c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f80c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 -- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0f806d590 msgr2=0x7fb0f81582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:25.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0f806d590 0x7fb0f81582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:25.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 -- 172.21.15.158:0/1708136426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec0094a0 con 0x7fb0f815d910 2023-10-16T11:21:25.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0f6d9d700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0f815d910 0x7fb0f80b6200 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000cc50 tx=0x7fb0f000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0e6ffd700 1 -- 172.21.15.158:0/1708136426 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0003c00 con 0x7fb0f815d910 2023-10-16T11:21:25.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0e6ffd700 1 -- 172.21.15.158:0/1708136426 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f0003d60 con 0x7fb0f815d910 2023-10-16T11:21:25.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8009550 con 0x7fb0f815d910 2023-10-16T11:21:25.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.946+0000 7fb0e6ffd700 1 -- 172.21.15.158:0/1708136426 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f000aab0 con 0x7fb0f815d910 2023-10-16T11:21:25.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.947+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f80c6b20 con 0x7fb0f815d910 2023-10-16T11:21:25.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.947+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f80d0340 con 0x7fb0f815d910 2023-10-16T11:21:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.948+0000 7fb0e6ffd700 1 -- 172.21.15.158:0/1708136426 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb0f000ac10 con 0x7fb0f815d910 2023-10-16T11:21:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.949+0000 7fb0e6ffd700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb0d806b140 0x7fb0d806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.949+0000 7fb0e6ffd700 1 -- 172.21.15.158:0/1708136426 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fb0f008a760 con 0x7fb0f815d910 2023-10-16T11:21:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.949+0000 7fb0f659c700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb0d806b140 0x7fb0d806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.949+0000 7fb0f659c700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb0d806b140 0x7fb0d806d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009c40 tx=0x7fb0e8008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:25.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:25.952+0000 7fb0e6ffd700 1 -- 172.21.15.158:0/1708136426 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0f0055220 con 0x7fb0f815d910 2023-10-16T11:21:26.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.190+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0f80681b0 con 0x7fb0f815d910 2023-10-16T11:21:26.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.191+0000 7fb0e6ffd700 1 -- 172.21.15.158:0/1708136426 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fb0f0058840 con 0x7fb0f815d910 2023-10-16T11:21:26.193 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:26.193 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":145,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1010484950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1010484950},{"type":"v1","addr":"172.21.15.158:6836","nonce":1010484950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":13224,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/54641557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":54641557},{"type":"v1","addr":"172.21.15.158:6837","nonce":54641557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":13239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3388825539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3388825539},{"type":"v1","addr":"172.21.15.172:6835","nonce":3388825539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:20.241439+0000","modified":"2023-10-16T11:21:23.275699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39845":{"gid":39845,"name":"d","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3986459990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3986459990},{"type":"v1","addr":"172.21.15.172:6834","nonce":3986459990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T11:21:26.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb0d806b140 msgr2=0x7fb0d806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb0d806b140 0x7fb0d806d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009c40 tx=0x7fb0e8008030 comp rx=0 tx=0).stop 2023-10-16T11:21:26.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0f815d910 msgr2=0x7fb0f80b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0f815d910 0x7fb0f80b6200 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000cc50 tx=0x7fb0f000cf60 comp rx=0 tx=0).stop 2023-10-16T11:21:26.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 shutdown_connections 2023-10-16T11:21:26.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7fb0d806b140 0x7fb0d806d600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0f815d910 0x7fb0f80b6200 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0f806d590 0x7fb0f81582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 --2- 172.21.15.158:0/1708136426 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb0f815d530 0x7fb0f80c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 >> 172.21.15.158:0/1708136426 conn(0x7fb0f8071180 msgr2=0x7fb0f80b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 shutdown_connections 2023-10-16T11:21:26.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.193+0000 7fb0fc9d6700 1 -- 172.21.15.158:0/1708136426 wait complete. 2023-10-16T11:21:26.198 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 145 2023-10-16T11:21:26.610 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 145, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:21:20.241439+0000', 'modified': '2023-10-16T11:21:23.275699+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 39845}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_39845': {'gid': 39845, 'name': 'd', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6834/3986459990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6832', 'nonce': 3986459990}, {'type': 'v1', 'addr': '172.21.15.172:6834', 'nonce': 3986459990}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:21:26.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:21:26.611 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T11:21:26.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.733+0000 7f12a908d700 1 Processor -- start 2023-10-16T11:21:26.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.734+0000 7f12a908d700 1 -- start start 2023-10-16T11:21:26.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.734+0000 7f12a908d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12a415d990 0x7f12a40b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.734+0000 7f12a908d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a406e480 0x7f12a40b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.734+0000 7f12a908d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a40af600 con 0x7f12a40b3760 2023-10-16T11:21:26.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.734+0000 7f12a908d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a40b2c90 con 0x7f12a415d990 2023-10-16T11:21:26.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.734+0000 7f12a908d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a406e9b0 con 0x7f12a406e480 2023-10-16T11:21:26.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a406e480 0x7f12a40b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a2d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12a415d990 0x7f12a40b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a359e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f12a40b3760 0x7f12a40cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:46452/0 (socket says 172.21.15.158:46452) 2023-10-16T11:21:26.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a406e480 0x7f12a40b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:53204/0 (socket says 172.21.15.158:53204) 2023-10-16T11:21:26.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a2d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12a415d990 0x7f12a40b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:43548/0 (socket says 172.21.15.158:43548) 2023-10-16T11:21:26.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a359e700 1 -- 172.21.15.158:0/2860434347 learned_addr learned my addr 172.21.15.158:0/2860434347 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:26.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a259c700 1 -- 172.21.15.158:0/2860434347 >> v2:172.21.15.172:3300/0 conn(0x7f12a415d990 msgr2=0x7f12a40b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a259c700 1 --2- 172.21.15.158:0/2860434347 >> v2:172.21.15.172:3300/0 conn(0x7f12a415d990 0x7f12a40b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a259c700 1 -- 172.21.15.158:0/2860434347 >> v1:172.21.15.172:6789/0 conn(0x7f12a40b3760 legacy=0x7f12a40cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a1d9b700 1 -- 172.21.15.158:0/2860434347 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072956660 0 0) 0x7f12a40af600 con 0x7f12a40b3760 2023-10-16T11:21:26.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a259c700 1 -- 172.21.15.158:0/2860434347 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a40ce890 con 0x7f12a406e480 2023-10-16T11:21:26.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/2860434347 >> v2:172.21.15.172:3300/0 conn(0x7f12a415d990 0x7f12a40b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:26.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.735+0000 7f12a259c700 1 --2- 172.21.15.158:0/2860434347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a406e480 0x7f12a40b3220 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1294009840 tx=0x7f1294009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=250ee4ef55752365 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a1d9b700 1 -- 172.21.15.158:0/2860434347 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294011830 con 0x7f12a406e480 2023-10-16T11:21:26.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a1d9b700 1 -- 172.21.15.158:0/2860434347 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294011990 con 0x7f12a406e480 2023-10-16T11:21:26.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a1d9b700 1 -- 172.21.15.158:0/2860434347 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294011d20 con 0x7f12a406e480 2023-10-16T11:21:26.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 -- 172.21.15.158:0/2860434347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a406e480 msgr2=0x7f12a40b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 --2- 172.21.15.158:0/2860434347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a406e480 0x7f12a40b3220 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1294009840 tx=0x7f1294009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:26.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 -- 172.21.15.158:0/2860434347 shutdown_connections 2023-10-16T11:21:26.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 --2- 172.21.15.158:0/2860434347 >> v2:172.21.15.172:3300/0 conn(0x7f12a415d990 0x7f12a40b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 --2- 172.21.15.158:0/2860434347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a406e480 0x7f12a40b3220 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 -- 172.21.15.158:0/2860434347 >> 172.21.15.158:0/2860434347 conn(0x7f12a406b9e0 msgr2=0x7f12a406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 -- 172.21.15.158:0/2860434347 shutdown_connections 2023-10-16T11:21:26.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.736+0000 7f12a908d700 1 -- 172.21.15.158:0/2860434347 wait complete. 2023-10-16T11:21:26.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 Processor -- start 2023-10-16T11:21:26.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 -- start start 2023-10-16T11:21:26.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a406e480 0x7f12a41534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a40b3760 0x7f12a415f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a415d990 0x7f12a40b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a41f6500 con 0x7f12a406e480 2023-10-16T11:21:26.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a41f3450 con 0x7f12a40b3760 2023-10-16T11:21:26.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.738+0000 7f12a908d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a41f3730 con 0x7f12a415d990 2023-10-16T11:21:26.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a406e480 0x7f12a41534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a406e480 0x7f12a41534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42374/0 (socket says 172.21.15.158:42374) 2023-10-16T11:21:26.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 -- 172.21.15.158:0/2299989205 learned_addr learned my addr 172.21.15.158:0/2299989205 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:26.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a259c700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a40b3760 0x7f12a415f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a359e700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a415d990 0x7f12a40b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 -- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a415d990 msgr2=0x7f12a40b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a415d990 0x7f12a40b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 -- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a40b3760 msgr2=0x7f12a415f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a40b3760 0x7f12a415f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 -- 172.21.15.158:0/2299989205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12980094a0 con 0x7f12a406e480 2023-10-16T11:21:26.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a359e700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a415d990 0x7f12a40b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:26.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a406e480 0x7f12a41534e0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f129800ebe0 tx=0x7f129800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f128b7fe700 1 -- 172.21.15.158:0/2299989205 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129800c850 con 0x7f12a406e480 2023-10-16T11:21:26.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f128b7fe700 1 -- 172.21.15.158:0/2299989205 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129800c9b0 con 0x7f12a406e480 2023-10-16T11:21:26.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12940094a0 con 0x7f12a406e480 2023-10-16T11:21:26.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f128b7fe700 1 -- 172.21.15.158:0/2299989205 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129800cd40 con 0x7f12a406e480 2023-10-16T11:21:26.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.739+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a41f3cf0 con 0x7f12a406e480 2023-10-16T11:21:26.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.741+0000 7f128b7fe700 1 -- 172.21.15.158:0/2299989205 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1298013470 con 0x7f12a406e480 2023-10-16T11:21:26.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.741+0000 7f128b7fe700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f129006b140 0x7f129006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.741+0000 7f128b7fe700 1 -- 172.21.15.158:0/2299989205 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f1298089e30 con 0x7f12a406e480 2023-10-16T11:21:26.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.741+0000 7f12a259c700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f129006b140 0x7f129006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.742+0000 7f12a259c700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f129006b140 0x7f129006d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1294009840 tx=0x7f1294008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.743+0000 7f12b5277b80 1 Processor -- start 2023-10-16T11:21:26.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.743+0000 7f12b5277b80 1 -- start start 2023-10-16T11:21:26.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb1220 0x560eaafb8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb6630 0x560eaafb8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 0x560eaaf8c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12b5277b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eaaf02c20 con 0x560eaafb1220 2023-10-16T11:21:26.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eaaee7c10 con 0x560eaafb6630 2023-10-16T11:21:26.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x560eaaee5530 con 0x560eaaf8c010 2023-10-16T11:21:26.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb1220 0x560eaafb8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb1220 0x560eaafb8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42382/0 (socket says 172.21.15.158:42382) 2023-10-16T11:21:26.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3575602809 learned_addr learned my addr 172.21.15.158:0/3575602809 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:26.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a359e700 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 0x560eaaf8c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a259c700 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb6630 0x560eaafb8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 msgr2=0x560eaaf8c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 0x560eaaf8c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb6630 msgr2=0x560eaafb8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb6630 0x560eaafb8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3575602809 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eaae9e3a0 con 0x560eaafb1220 2023-10-16T11:21:26.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.744+0000 7f12a259c700 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb6630 0x560eaafb8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:26.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.745+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb1220 0x560eaafb8a70 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f12980602e0 tx=0x7f1298058180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec03f349bea6b3f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.745+0000 7f1288ff9700 1 -- 172.21.15.158:0/3575602809 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129805c660 con 0x560eaafb1220 2023-10-16T11:21:26.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.745+0000 7f1288ff9700 1 -- 172.21.15.158:0/3575602809 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1298060810 con 0x560eaafb1220 2023-10-16T11:21:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.745+0000 7f1288ff9700 1 -- 172.21.15.158:0/3575602809 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298058990 con 0x560eaafb1220 2023-10-16T11:21:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.745+0000 7f12b5277b80 1 -- 172.21.15.158:0/3575602809 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb1220 msgr2=0x560eaafb8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.745+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb1220 0x560eaafb8a70 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f12980602e0 tx=0x7f1298058180 comp rx=0 tx=0).stop 2023-10-16T11:21:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 -- 172.21.15.158:0/3575602809 shutdown_connections 2023-10-16T11:21:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 0x560eaaf8c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb1220 0x560eaafb8a70 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3575602809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb6630 0x560eaafb8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 -- 172.21.15.158:0/3575602809 >> 172.21.15.158:0/3575602809 conn(0x560eaaebbd80 msgr2=0x560eaafd6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 -- 172.21.15.158:0/3575602809 shutdown_connections 2023-10-16T11:21:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 -- 172.21.15.158:0/3575602809 wait complete. 2023-10-16T11:21:26.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 -- start start 2023-10-16T11:21:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.746+0000 7f12b5277b80 1 Processor -- start 2023-10-16T11:21:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaaf8c010 0x560eaafb9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafb1220 0x560eaaf8e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb6630 0x560eaaf8ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12b5277b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eaaee0e20 con 0x560eaafb6630 2023-10-16T11:21:26.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eaae7d6c0 con 0x560eaaf8c010 2023-10-16T11:21:26.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x560eaae7d360 con 0x560eaafb1220 2023-10-16T11:21:26.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12b5277b80 10 client.0 _ll_register_callbacks cb 0x560eaacda7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:21:26.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb6630 0x560eaaf8ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb6630 0x560eaaf8ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42392/0 (socket says 172.21.15.158:42392) 2023-10-16T11:21:26.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 -- 172.21.15.158:0/2383376917 learned_addr learned my addr 172.21.15.158:0/2383376917 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:26.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaaf8c010 0x560eaafb9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a259c700 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafb1220 0x560eaaf8e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 -- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafb1220 msgr2=0x560eaaf8e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafb1220 0x560eaaf8e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 -- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaaf8c010 msgr2=0x560eaafb9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaaf8c010 0x560eaafb9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.747+0000 7f12a359e700 1 -- 172.21.15.158:0/2383376917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a41f3cf0 con 0x560eaafb6630 2023-10-16T11:21:26.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12a359e700 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb6630 0x560eaaf8ea40 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f12a40b2ee0 tx=0x7f129c0104c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcfe45593f16bbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f1281ffb700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:21:26.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f1281ffb700 1 -- 172.21.15.158:0/2383376917 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c010cd0 con 0x560eaafb6630 2023-10-16T11:21:26.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f1281ffb700 1 -- 172.21.15.158:0/2383376917 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f129c010e30 con 0x560eaafb6630 2023-10-16T11:21:26.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560eaaf815b0 con 0x560eaafb6630 2023-10-16T11:21:26.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f1281ffb700 1 -- 172.21.15.158:0/2383376917 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) v2 ==== 24+0+0 (secure 0 0 0) 0x7f129c0020e0 con 0x560eaafb6630 2023-10-16T11:21:26.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 10 client.13353 fetch_fsmap learned FSMap version 145 2023-10-16T11:21:26.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=2+}) v3 -- 0x560eaad0bd90 con 0x560eaafb6630 2023-10-16T11:21:26.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f1281ffb700 1 -- 172.21.15.158:0/2383376917 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560eaaf815b0 con 0x560eaafb6630 2023-10-16T11:21:26.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f1281ffb700 1 -- 172.21.15.158:0/2383376917 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 145) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f129c01d3f0 con 0x560eaafb6630 2023-10-16T11:21:26.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 10 client.13353 fetch_fsmap finished waiting for FSMap version 145 2023-10-16T11:21:26.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 10 client.13353 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:21:26.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 20 client.13353 populate_metadata read hostname 'smithi158' 2023-10-16T11:21:26.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafafcd0 0x560eaafb00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 4 client.13353 mds_command: new command op to 39845 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:21:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 --> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560eaaed9e00 con 0x560eaafafcd0 2023-10-16T11:21:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.748+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafafcd0 0x560eaafb00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.749+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafafcd0 0x560eaafb00b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36a0f4cd27cca73d server_cookie=ca1ed848b59afe45 in_seq=0 out_seq=0 2023-10-16T11:21:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.749+0000 7f1281ffb700 10 client.13353 ms_handle_connect on v2:172.21.15.172:6832/3986459990 2023-10-16T11:21:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.749+0000 7f1281ffb700 1 -- 172.21.15.158:0/2383376917 <== mds.0 v2:172.21.15.172:6832/3986459990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560eaaed9e00 con 0x560eaafafcd0 2023-10-16T11:21:26.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.749+0000 7f1281ffb700 10 client.13353 handle_command_reply: tid=0 2023-10-16T11:21:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 client.13353 shutdown 2023-10-16T11:21:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb6630 msgr2=0x560eaaf8ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb6630 0x560eaaf8ea40 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f12a40b2ee0 tx=0x7f129c0104c0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 shutdown_connections 2023-10-16T11:21:26.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafafcd0 0x560eaafb00b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafb6630 0x560eaaf8ea40 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaaf8c010 0x560eaafb9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f1281ffb700 0 client.13353 ms_handle_reset on v2:172.21.15.172:6832/3986459990 2023-10-16T11:21:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 --2- 172.21.15.158:0/2383376917 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafb1220 0x560eaaf8e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 >> 172.21.15.158:0/2383376917 conn(0x560eaaebbd80 msgr2=0x560eaafd6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 shutdown_connections 2023-10-16T11:21:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 1 -- 172.21.15.158:0/2383376917 wait complete. 2023-10-16T11:21:26.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.750+0000 7f12b5277b80 20 client.13353 trim_cache size 0 max 16384 2023-10-16T11:21:26.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.767+0000 7f12b5277b80 1 Processor -- start 2023-10-16T11:21:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.767+0000 7f12b5277b80 1 -- start start 2023-10-16T11:21:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 0x560eaafd38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafafcd0 0x560eaafd3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eaafb16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12b5277b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eaad618d0 con 0x560eaafafcd0 2023-10-16T11:21:26.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eaaff5fb0 con 0x560eaafb1220 2023-10-16T11:21:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x560eaafb2e30 con 0x560eaaf8c010 2023-10-16T11:21:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafafcd0 0x560eaafd3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12a259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafafcd0 0x560eaafd3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42398/0 (socket says 172.21.15.158:42398) 2023-10-16T11:21:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.768+0000 7f12a259c700 1 -- 172.21.15.158:0/3658271585 learned_addr learned my addr 172.21.15.158:0/3658271585 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:26.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12a259c700 1 -- 172.21.15.158:0/3658271585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 msgr2=0x560eaafd38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12a259c700 1 --2- 172.21.15.158:0/3658271585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 0x560eaafd38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12a259c700 1 -- 172.21.15.158:0/3658271585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 msgr2=0x560eaafb16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12a259c700 1 --2- 172.21.15.158:0/3658271585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eaafb16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12a259c700 1 -- 172.21.15.158:0/3658271585 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eaacf4c10 con 0x560eaafafcd0 2023-10-16T11:21:26.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12a259c700 1 --2- 172.21.15.158:0/3658271585 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafafcd0 0x560eaafd3df0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f1294000c00 tx=0x7f1294011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ea64939b3a5062 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12897fa700 1 -- 172.21.15.158:0/3658271585 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129402b050 con 0x560eaafafcd0 2023-10-16T11:21:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12897fa700 1 -- 172.21.15.158:0/3658271585 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129402c040 con 0x560eaafafcd0 2023-10-16T11:21:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12897fa700 1 -- 172.21.15.158:0/3658271585 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294020070 con 0x560eaafafcd0 2023-10-16T11:21:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 -- 172.21.15.158:0/3658271585 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafafcd0 msgr2=0x560eaafd3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3658271585 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafafcd0 0x560eaafd3df0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f1294000c00 tx=0x7f1294011000 comp rx=0 tx=0).stop 2023-10-16T11:21:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 -- 172.21.15.158:0/3658271585 shutdown_connections 2023-10-16T11:21:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3658271585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eaafb16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3658271585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaaf8c010 0x560eaafd38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3658271585 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaafafcd0 0x560eaafd3df0 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 -- 172.21.15.158:0/3658271585 >> 172.21.15.158:0/3658271585 conn(0x560eaaebbd80 msgr2=0x560eaaf8fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 -- 172.21.15.158:0/3658271585 shutdown_connections 2023-10-16T11:21:26.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.769+0000 7f12b5277b80 1 -- 172.21.15.158:0/3658271585 wait complete. 2023-10-16T11:21:26.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.770+0000 7f12b5277b80 1 -- start start 2023-10-16T11:21:26.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.770+0000 7f12b5277b80 1 Processor -- start 2023-10-16T11:21:26.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaaf8c010 0x560eaafb22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafafcd0 0x560eab03c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eab02ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eab03ce90 con 0x560eaaf8c010 2023-10-16T11:21:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eaafe1d70 con 0x560eaafb1220 2023-10-16T11:21:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x560eaafb1810 con 0x560eaafafcd0 2023-10-16T11:21:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 10 client.0 _ll_register_callbacks cb 0x7f129c007530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:21:26.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaaf8c010 0x560eaafb22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaaf8c010 0x560eaafb22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42402/0 (socket says 172.21.15.158:42402) 2023-10-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3104372051 learned_addr learned my addr 172.21.15.158:0/3104372051 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a359e700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eab02ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a259c700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafafcd0 0x560eab03c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafafcd0 msgr2=0x560eab03c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafafcd0 0x560eab03c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 msgr2=0x560eab02ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eab02ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 -- 172.21.15.158:0/3104372051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a40ce890 con 0x560eaaf8c010 2023-10-16T11:21:26.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a359e700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eab02ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:26.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a259c700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafafcd0 0x560eab03c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:26.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12a2d9d700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaaf8c010 0x560eaafb22b0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f12980584a0 tx=0x7f1298000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a37b273746d0935 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f1281ffb700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:21:26.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f1281ffb700 1 -- 172.21.15.158:0/3104372051 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298057c00 con 0x560eaaf8c010 2023-10-16T11:21:26.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f1281ffb700 1 -- 172.21.15.158:0/3104372051 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f1298058770 con 0x560eaaf8c010 2023-10-16T11:21:26.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.771+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1298060810 con 0x560eaaf8c010 2023-10-16T11:21:26.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f1281ffb700 1 -- 172.21.15.158:0/3104372051 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1298058930 con 0x560eaaf8c010 2023-10-16T11:21:26.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 10 client.13359 fetch_fsmap learned FSMap version 145 2023-10-16T11:21:26.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=2+}) v3 -- 0x560eaad32200 con 0x560eaaf8c010 2023-10-16T11:21:26.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f1281ffb700 1 -- 172.21.15.158:0/3104372051 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1298060810 con 0x560eaaf8c010 2023-10-16T11:21:26.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f1281ffb700 1 -- 172.21.15.158:0/3104372051 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 145) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1298060380 con 0x560eaaf8c010 2023-10-16T11:21:26.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 10 client.13359 fetch_fsmap finished waiting for FSMap version 145 2023-10-16T11:21:26.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 10 client.13359 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T11:21:26.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 20 client.13359 populate_metadata read hostname 'smithi158' 2023-10-16T11:21:26.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafb6630 0x560eab030500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 4 client.13359 mds_command: new command op to 39845 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:21:26.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 --> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560eaaed9b20 con 0x560eaafb6630 2023-10-16T11:21:26.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12a259c700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafb6630 0x560eab030500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f1281ffb700 10 client.13359 ms_handle_connect on v2:172.21.15.172:6832/3986459990 2023-10-16T11:21:26.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.772+0000 7f12a259c700 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafb6630 0x560eab030500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2319e9a0680b0f8d server_cookie=159916f595009b45 in_seq=0 out_seq=0 2023-10-16T11:21:26.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f1281ffb700 1 -- 172.21.15.158:0/3104372051 <== mds.0 v2:172.21.15.172:6832/3986459990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x560eaaed9b20 con 0x560eaafb6630 2023-10-16T11:21:26.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f1281ffb700 10 client.13359 handle_command_reply: tid=0 2023-10-16T11:21:26.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 client.13359 shutdown 2023-10-16T11:21:26.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaaf8c010 msgr2=0x560eaafb22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaaf8c010 0x560eaafb22b0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f12980584a0 tx=0x7f1298000bc0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 shutdown_connections 2023-10-16T11:21:26.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560eaafb1220 0x560eab02ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:6832/3986459990,v1:172.21.15.172:6834/3986459990] conn(0x560eaafb6630 0x560eab030500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x560eaaf8c010 0x560eaafb22b0 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 --2- 172.21.15.158:0/3104372051 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x560eaafafcd0 0x560eab03c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f1281ffb700 0 client.13359 ms_handle_reset on v2:172.21.15.172:6832/3986459990 2023-10-16T11:21:26.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 >> 172.21.15.158:0/3104372051 conn(0x560eaaebbd80 msgr2=0x560eaafb2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 shutdown_connections 2023-10-16T11:21:26.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 1 -- 172.21.15.158:0/3104372051 wait complete. 2023-10-16T11:21:26.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12b5277b80 20 client.13359 trim_cache size 0 max 16384 2023-10-16T11:21:26.798 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:21:26.798 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:21:26.798 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:21:26.798 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 39845, 2023-10-16T11:21:26.798 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:21:26.799 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:21:26.799 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T11:21:26.799 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9054890169999998, 2023-10-16T11:21:26.799 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 145, 2023-10-16T11:21:26.799 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 257, 2023-10-16T11:21:26.800 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 257, 2023-10-16T11:21:26.800 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.859212017999999 2023-10-16T11:21:26.800 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:21:26.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f129006b140 msgr2=0x7f129006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f129006b140 0x7f129006d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1294009840 tx=0x7f1294008030 comp rx=0 tx=0).stop 2023-10-16T11:21:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a406e480 msgr2=0x7f12a41534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a406e480 0x7f12a41534e0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f129800ebe0 tx=0x7f129800eef0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 shutdown_connections 2023-10-16T11:21:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f129006b140 0x7f129006d600 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12a415d990 0x7f12a40b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12a406e480 0x7f12a41534e0 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 --2- 172.21.15.158:0/2299989205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12a40b3760 0x7f12a415f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.773+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 >> 172.21.15.158:0/2299989205 conn(0x7f12a406b9e0 msgr2=0x7f12a4071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.774+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 shutdown_connections 2023-10-16T11:21:26.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.774+0000 7f12a908d700 1 -- 172.21.15.158:0/2299989205 wait complete. 2023-10-16T11:21:26.804 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T11:21:26.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.923+0000 7f3a185f7700 1 Processor -- start 2023-10-16T11:21:26.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.923+0000 7f3a185f7700 1 -- start start 2023-10-16T11:21:26.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a185f7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a1015db70 0x7f3a1015ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a185f7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3a100cb7e0 0x7f3a100cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a185f7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1015efc0 con 0x7f3a10143ba0 2023-10-16T11:21:26.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a185f7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a100cc130 con 0x7f3a1015db70 2023-10-16T11:21:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a185f7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1006fbc0 con 0x7f3a100cb7e0 2023-10-16T11:21:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a16393700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a1015db70 0x7f3a1015ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a16393700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a1015db70 0x7f3a1015ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42410/0 (socket says 172.21.15.158:42410) 2023-10-16T11:21:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a16393700 1 -- 172.21.15.158:0/1016974814 learned_addr learned my addr 172.21.15.158:0/1016974814 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:26.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.924+0000 7f3a16b94700 1 --2- 172.21.15.158:0/1016974814 >> v2:172.21.15.172:3300/0 conn(0x7f3a100cb7e0 0x7f3a100cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a16393700 1 -- 172.21.15.158:0/1016974814 >> v2:172.21.15.172:3300/0 conn(0x7f3a100cb7e0 msgr2=0x7f3a100cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a16393700 1 --2- 172.21.15.158:0/1016974814 >> v2:172.21.15.172:3300/0 conn(0x7f3a100cb7e0 0x7f3a100cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a16393700 1 -- 172.21.15.158:0/1016974814 >> v1:172.21.15.172:6789/0 conn(0x7f3a10143ba0 legacy=0x7f3a1006f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a16393700 1 -- 172.21.15.158:0/1016974814 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1006e9f0 con 0x7f3a1015db70 2023-10-16T11:21:26.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a16393700 1 --2- 172.21.15.158:0/1016974814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a1015ea80 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f3a100726c0 tx=0x7f3a0000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90316fd5fa87354f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a15391700 1 -- 172.21.15.158:0/1016974814 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000dcf0 con 0x7f3a1015db70 2023-10-16T11:21:26.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a15391700 1 -- 172.21.15.158:0/1016974814 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0000de50 con 0x7f3a1015db70 2023-10-16T11:21:26.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a15391700 1 -- 172.21.15.158:0/1016974814 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00004250 con 0x7f3a1015db70 2023-10-16T11:21:26.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a185f7700 1 -- 172.21.15.158:0/1016974814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 msgr2=0x7f3a1015ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a185f7700 1 --2- 172.21.15.158:0/1016974814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a1015ea80 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f3a100726c0 tx=0x7f3a0000be70 comp rx=0 tx=0).stop 2023-10-16T11:21:26.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.925+0000 7f3a185f7700 1 -- 172.21.15.158:0/1016974814 shutdown_connections 2023-10-16T11:21:26.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.926+0000 7f3a185f7700 1 --2- 172.21.15.158:0/1016974814 >> v2:172.21.15.172:3300/0 conn(0x7f3a100cb7e0 0x7f3a100cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.926+0000 7f3a185f7700 1 --2- 172.21.15.158:0/1016974814 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a1015ea80 unknown :-1 s=CLOSED pgs=3140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.926+0000 7f3a185f7700 1 -- 172.21.15.158:0/1016974814 >> 172.21.15.158:0/1016974814 conn(0x7f3a100cc5a0 msgr2=0x7f3a100cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:26.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.926+0000 7f3a185f7700 1 -- 172.21.15.158:0/1016974814 shutdown_connections 2023-10-16T11:21:26.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.926+0000 7f3a185f7700 1 -- 172.21.15.158:0/1016974814 wait complete. 2023-10-16T11:21:26.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 Processor -- start 2023-10-16T11:21:26.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 -- start start 2023-10-16T11:21:26.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a100cb7e0 0x7f3a10158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a10143ba0 0x7f3a1015af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a101eea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1015d160 con 0x7f3a1015db70 2023-10-16T11:21:26.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a101eef50 con 0x7f3a100cb7e0 2023-10-16T11:21:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a185f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100a8590 con 0x7f3a10143ba0 2023-10-16T11:21:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.927+0000 7f3a16b94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a101eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a101eea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42418/0 (socket says 172.21.15.158:42418) 2023-10-16T11:21:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 -- 172.21.15.158:0/2536234195 learned_addr learned my addr 172.21.15.158:0/2536234195 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:26.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16393700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a100cb7e0 0x7f3a10158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a15b92700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a10143ba0 0x7f3a1015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 -- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a10143ba0 msgr2=0x7f3a1015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a10143ba0 0x7f3a1015af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 -- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a100cb7e0 msgr2=0x7f3a10158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a100cb7e0 0x7f3a10158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 -- 172.21.15.158:0/2536234195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0000b7f0 con 0x7f3a1015db70 2023-10-16T11:21:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a16b94700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a101eea10 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f3a08007e60 tx=0x7f3a0800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a06ffd700 1 -- 172.21.15.158:0/2536234195 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0800ed30 con 0x7f3a1015db70 2023-10-16T11:21:26.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a06ffd700 1 -- 172.21.15.158:0/2536234195 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a08014420 con 0x7f3a1015db70 2023-10-16T11:21:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a101ef9e0 con 0x7f3a1015db70 2023-10-16T11:21:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a06ffd700 1 -- 172.21.15.158:0/2536234195 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080147d0 con 0x7f3a1015db70 2023-10-16T11:21:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.928+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a101ef420 con 0x7f3a1015db70 2023-10-16T11:21:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.929+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a10066db0 con 0x7f3a1015db70 2023-10-16T11:21:26.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.930+0000 7f3a06ffd700 1 -- 172.21.15.158:0/2536234195 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3a08014930 con 0x7f3a1015db70 2023-10-16T11:21:26.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.930+0000 7f3a06ffd700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f39f406b170 0x7f39f406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:26.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.930+0000 7f3a16393700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f39f406b170 0x7f39f406d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:26.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.930+0000 7f3a06ffd700 1 -- 172.21.15.158:0/2536234195 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f3a0808a1a0 con 0x7f3a1015db70 2023-10-16T11:21:26.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.931+0000 7f3a16393700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f39f406b170 0x7f39f406d630 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000b340 tx=0x7f3a0001d040 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:26.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:26.935+0000 7f3a06ffd700 1 -- 172.21.15.158:0/2536234195 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a08054de0 con 0x7f3a1015db70 2023-10-16T11:21:27.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.170+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3a100681b0 con 0x7f3a1015db70 2023-10-16T11:21:27.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.170+0000 7f3a06ffd700 1 -- 172.21.15.158:0/2536234195 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v145) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3a08058400 con 0x7f3a1015db70 2023-10-16T11:21:27.172 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T11:21:27.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f39f406b170 msgr2=0x7f39f406d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:27.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f39f406b170 0x7f39f406d630 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000b340 tx=0x7f3a0001d040 comp rx=0 tx=0).stop 2023-10-16T11:21:27.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 msgr2=0x7f3a101eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:27.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a101eea10 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f3a08007e60 tx=0x7f3a0800e3c0 comp rx=0 tx=0).stop 2023-10-16T11:21:27.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 shutdown_connections 2023-10-16T11:21:27.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f39f406b170 0x7f39f406d630 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:27.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a1015db70 0x7f3a101eea10 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:27.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a100cb7e0 0x7f3a10158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:27.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 --2- 172.21.15.158:0/2536234195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a10143ba0 0x7f3a1015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:27.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 >> 172.21.15.158:0/2536234195 conn(0x7f3a100cc5a0 msgr2=0x7f3a100b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:27.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 shutdown_connections 2023-10-16T11:21:27.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:27.172+0000 7f3a185f7700 1 -- 172.21.15.158:0/2536234195 wait complete. 2023-10-16T11:21:27.186 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T11:21:27.186 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T11:21:27.186 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T11:21:27.187 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T11:21:27.187 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T11:21:27.187 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T11:21:27.188 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T11:21:27.188 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T11:21:27.188 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:21:27.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.189 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T11:21:27.206 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T11:21:27.206 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T11:21:27.207 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T11:21:27.207 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:21:27.207 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T11:21:27.207 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:21:27.207 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:21:27.208 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:21:27.208 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:21:27.208 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:21:27.208 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:21:27.208 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T11:21:27.209 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T11:21:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T11:21:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T11:21:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 36545sec preferred_lft 36545sec 2023-10-16T11:21:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T11:21:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:21:27.210 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T11:21:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T11:21:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T11:21:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:21:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T11:21:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:21:27.211 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:21:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:21:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T11:21:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:21:27.211 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:21:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:21:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T11:21:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:21:27.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.213 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T11:21:27.272 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T11:21:27.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.274 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T11:21:27.332 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T11:21:27.332 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T11:21:27.333 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T11:21:27.333 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T11:21:27.333 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T11:21:27.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.334 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T11:21:27.334 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T11:21:27.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T11:21:27.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T11:21:27.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T11:21:27.336 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T11:21:27.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.437 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:21:27.453 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:21:27.453 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:21:27.454 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:21:27.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.455 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T11:21:27.520 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:21:27.520 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T11:21:27.577 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:21:27.578 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:21:27.578 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:21:27.633 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T11:21:27.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.634 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:21:27.718 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:21:27.719 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:21:27.743 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:21:27.743 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:21:27.743 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:21:27.744 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:21:27.744 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:21:27.744 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:21:27.744 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:21:27.744 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:21:27.819 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:21:27.820 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:21:27.820 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:21:27.820 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:21:27.820 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:21:27.821 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:21:27.821 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:21:27.821 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:21:27.821 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T11:21:27.821 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T11:21:27.822 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T11:21:27.823 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T11:21:27.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.823 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:21:27.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:27.851 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T11:21:27.938 INFO:teuthology.orchestra.run.smithi158.stdout:client11683 2023-10-16T11:21:27.939 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:27.939 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:27.956 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T11:21:28.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.117+0000 7f98b4940700 1 Processor -- start 2023-10-16T11:21:28.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.117+0000 7f98b4940700 1 -- start start 2023-10-16T11:21:28.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98b4940700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98b015d950 0x7f98b015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98b4940700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98b00cec30 0x7f98b006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98b4940700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b00ce320 con 0x7f98b00ce7b0 2023-10-16T11:21:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98b4940700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b00725c0 con 0x7f98b015d950 2023-10-16T11:21:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98b4940700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b015e270 con 0x7f98b00cec30 2023-10-16T11:21:28.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98ae59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98b015d950 0x7f98b015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98ae59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98b015d950 0x7f98b015dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42426/0 (socket says 172.21.15.158:42426) 2023-10-16T11:21:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.118+0000 7f98ae59c700 1 -- 172.21.15.158:0/2051528855 learned_addr learned my addr 172.21.15.158:0/2051528855 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98aed9d700 1 --2- 172.21.15.158:0/2051528855 >> v2:172.21.15.172:3300/0 conn(0x7f98b00cec30 0x7f98b006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ae59c700 1 -- 172.21.15.158:0/2051528855 >> v2:172.21.15.172:3300/0 conn(0x7f98b00cec30 msgr2=0x7f98b006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:28.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ae59c700 1 --2- 172.21.15.158:0/2051528855 >> v2:172.21.15.172:3300/0 conn(0x7f98b00cec30 0x7f98b006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ae59c700 1 -- 172.21.15.158:0/2051528855 >> v1:172.21.15.172:6789/0 conn(0x7f98b00ce7b0 legacy=0x7f98b0071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ae59c700 1 -- 172.21.15.158:0/2051528855 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b00722f0 con 0x7f98b015d950 2023-10-16T11:21:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ae59c700 1 --2- 172.21.15.158:0/2051528855 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b015d950 0x7f98b015dd30 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f98b00cc690 tx=0x7f98a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f7ea8cc0b3c339 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ad59a700 1 -- 172.21.15.158:0/2051528855 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4004230 con 0x7f98b015d950 2023-10-16T11:21:28.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ad59a700 1 -- 172.21.15.158:0/2051528855 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a4004390 con 0x7f98b015d950 2023-10-16T11:21:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98ad59a700 1 -- 172.21.15.158:0/2051528855 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a40036a0 con 0x7f98b015d950 2023-10-16T11:21:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98b4940700 1 -- 172.21.15.158:0/2051528855 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b015d950 msgr2=0x7f98b015dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.119+0000 7f98b4940700 1 --2- 172.21.15.158:0/2051528855 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b015d950 0x7f98b015dd30 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f98b00cc690 tx=0x7f98a4009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.120+0000 7f98b4940700 1 -- 172.21.15.158:0/2051528855 shutdown_connections 2023-10-16T11:21:28.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.120+0000 7f98b4940700 1 --2- 172.21.15.158:0/2051528855 >> v2:172.21.15.172:3300/0 conn(0x7f98b00cec30 0x7f98b006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.120+0000 7f98b4940700 1 --2- 172.21.15.158:0/2051528855 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b015d950 0x7f98b015dd30 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.120+0000 7f98b4940700 1 -- 172.21.15.158:0/2051528855 >> 172.21.15.158:0/2051528855 conn(0x7f98b00cc700 msgr2=0x7f98b00ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.120+0000 7f98b4940700 1 -- 172.21.15.158:0/2051528855 shutdown_connections 2023-10-16T11:21:28.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.120+0000 7f98b4940700 1 -- 172.21.15.158:0/2051528855 wait complete. 2023-10-16T11:21:28.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.121+0000 7f98b4940700 1 Processor -- start 2023-10-16T11:21:28.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.121+0000 7f98b4940700 1 -- start start 2023-10-16T11:21:28.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98b4940700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b00ce7b0 0x7f98b015ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98b4940700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b00cec30 0x7f98b00c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98b4940700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b015d950 0x7f98b00c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98b4940700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b015bdb0 con 0x7f98b00cec30 2023-10-16T11:21:28.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98b4940700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b015b7f0 con 0x7f98b00ce7b0 2023-10-16T11:21:28.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98b4940700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b015bad0 con 0x7f98b015d950 2023-10-16T11:21:28.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b00cec30 0x7f98b00c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b00cec30 0x7f98b00c7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42434/0 (socket says 172.21.15.158:42434) 2023-10-16T11:21:28.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 -- 172.21.15.158:0/1998770224 learned_addr learned my addr 172.21.15.158:0/1998770224 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:28.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98ae59c700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b00ce7b0 0x7f98b015ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98aed9d700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b015d950 0x7f98b00c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 -- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b015d950 msgr2=0x7f98b00c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b015d950 0x7f98b00c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 -- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b00ce7b0 msgr2=0x7f98b015ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b00ce7b0 0x7f98b015ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 -- 172.21.15.158:0/1998770224 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4009550 con 0x7f98b00cec30 2023-10-16T11:21:28.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98aed9d700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b015d950 0x7f98b00c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:28.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98add9b700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b00cec30 0x7f98b00c7740 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f98a000c4d0 tx=0x7f98a000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:28.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f989effd700 1 -- 172.21.15.158:0/1998770224 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0011790 con 0x7f98b00cec30 2023-10-16T11:21:28.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f989effd700 1 -- 172.21.15.158:0/1998770224 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a00118f0 con 0x7f98b00cec30 2023-10-16T11:21:28.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f989effd700 1 -- 172.21.15.158:0/1998770224 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0011ca0 con 0x7f98b00cec30 2023-10-16T11:21:28.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.122+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b014f660 con 0x7f98b00cec30 2023-10-16T11:21:28.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.123+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b014d5e0 con 0x7f98b00cec30 2023-10-16T11:21:28.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.123+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b014f4c0 con 0x7f98b00cec30 2023-10-16T11:21:28.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.124+0000 7f989effd700 1 -- 172.21.15.158:0/1998770224 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f98a001c460 con 0x7f98b00cec30 2023-10-16T11:21:28.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.124+0000 7f989effd700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f989006b140 0x7f989006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.124+0000 7f989effd700 1 -- 172.21.15.158:0/1998770224 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f98a0089a40 con 0x7f98b00cec30 2023-10-16T11:21:28.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.124+0000 7f98ae59c700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f989006b140 0x7f989006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.125+0000 7f98ae59c700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f989006b140 0x7f989006d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009130 tx=0x7f98a4008030 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:28.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.129+0000 7f989effd700 1 -- 172.21.15.158:0/1998770224 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98b014f4c0 con 0x7f98b00cec30 2023-10-16T11:21:28.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.353+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 --> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f98b015bc40 con 0x7f989006b140 2023-10-16T11:21:28.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.354+0000 7f989effd700 1 -- 172.21.15.158:0/1998770224 <== mgr.12843 v2:172.21.15.158:6832/3125181137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f98b015bc40 con 0x7f989006b140 2023-10-16T11:21:28.356 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:21:28.356 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:28.357 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T11:21:28.357 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:28.357 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:21:28.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f989006b140 msgr2=0x7f989006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f989006b140 0x7f989006d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009130 tx=0x7f98a4008030 comp rx=0 tx=0).stop 2023-10-16T11:21:28.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b00cec30 msgr2=0x7f98b00c7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b00cec30 0x7f98b00c7740 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f98a000c4d0 tx=0x7f98a000c7e0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 shutdown_connections 2023-10-16T11:21:28.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f989006b140 0x7f989006d600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98b015d950 0x7f98b00c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98b00ce7b0 0x7f98b015ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 --2- 172.21.15.158:0/1998770224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98b00cec30 0x7f98b00c7740 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 >> 172.21.15.158:0/1998770224 conn(0x7f98b00cc700 msgr2=0x7f98b00b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:28.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 shutdown_connections 2023-10-16T11:21:28.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.357+0000 7f98b4940700 1 -- 172.21.15.158:0/1998770224 wait complete. 2023-10-16T11:21:28.373 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T11:21:28.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.494+0000 7feb0baee700 1 Processor -- start 2023-10-16T11:21:28.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.494+0000 7feb0baee700 1 -- start start 2023-10-16T11:21:28.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.495+0000 7feb0baee700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb04151220 0x7feb040665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.495+0000 7feb0baee700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151600 0x7feb040c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.495+0000 7feb0baee700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb040cf930 con 0x7feb040cf190 2023-10-16T11:21:28.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.495+0000 7feb0baee700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb040cf650 con 0x7feb04151220 2023-10-16T11:21:28.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.495+0000 7feb0baee700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb040c51b0 con 0x7feb04151600 2023-10-16T11:21:28.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0a08b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151600 0x7feb040c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0988a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb04151220 0x7feb040665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb09089700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7feb040cf190 0x7feb0406d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:46520/0 (socket says 172.21.15.158:46520) 2023-10-16T11:21:28.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb09089700 1 -- 172.21.15.158:0/3788546113 learned_addr learned my addr 172.21.15.158:0/3788546113 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0a08b700 1 -- 172.21.15.158:0/3788546113 >> v2:172.21.15.172:3300/0 conn(0x7feb04151220 msgr2=0x7feb040665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0a08b700 1 --2- 172.21.15.158:0/3788546113 >> v2:172.21.15.172:3300/0 conn(0x7feb04151220 0x7feb040665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0a08b700 1 -- 172.21.15.158:0/3788546113 >> v1:172.21.15.172:6789/0 conn(0x7feb040cf190 legacy=0x7feb0406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0a08b700 1 -- 172.21.15.158:0/3788546113 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb040c5490 con 0x7feb04151600 2023-10-16T11:21:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0988a700 1 --2- 172.21.15.158:0/3788546113 >> v2:172.21.15.172:3300/0 conn(0x7feb04151220 0x7feb040665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb0a08b700 1 --2- 172.21.15.158:0/3788546113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151600 0x7feb040c5c90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feafc009dc0 tx=0x7feafc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e69ed24def9d2ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:28.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb08888700 1 -- 172.21.15.158:0/3788546113 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc00e070 con 0x7feb04151600 2023-10-16T11:21:28.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb08888700 1 -- 172.21.15.158:0/3788546113 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc00bae0 con 0x7feb04151600 2023-10-16T11:21:28.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.496+0000 7feb08888700 1 -- 172.21.15.158:0/3788546113 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc00be90 con 0x7feb04151600 2023-10-16T11:21:28.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 -- 172.21.15.158:0/3788546113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151600 msgr2=0x7feb040c5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 --2- 172.21.15.158:0/3788546113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151600 0x7feb040c5c90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feafc009dc0 tx=0x7feafc00b330 comp rx=0 tx=0).stop 2023-10-16T11:21:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 -- 172.21.15.158:0/3788546113 shutdown_connections 2023-10-16T11:21:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 --2- 172.21.15.158:0/3788546113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151600 0x7feb040c5c90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 --2- 172.21.15.158:0/3788546113 >> v2:172.21.15.172:3300/0 conn(0x7feb04151220 0x7feb040665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 -- 172.21.15.158:0/3788546113 >> 172.21.15.158:0/3788546113 conn(0x7feb040cc5b0 msgr2=0x7feb040cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 -- 172.21.15.158:0/3788546113 shutdown_connections 2023-10-16T11:21:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.497+0000 7feb0baee700 1 -- 172.21.15.158:0/3788546113 wait complete. 2023-10-16T11:21:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.498+0000 7feb0baee700 1 Processor -- start 2023-10-16T11:21:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.498+0000 7feb0baee700 1 -- start start 2023-10-16T11:21:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0baee700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040cf190 0x7feb0414e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0baee700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151220 0x7feb041598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0baee700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb04151600 0x7feb0415da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0baee700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0414ec60 con 0x7feb04151600 2023-10-16T11:21:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0baee700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb041560d0 con 0x7feb040cf190 2023-10-16T11:21:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0baee700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb041554c0 con 0x7feb04151220 2023-10-16T11:21:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0a08b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb04151600 0x7feb0415da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0a08b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb04151600 0x7feb0415da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42450/0 (socket says 172.21.15.158:42450) 2023-10-16T11:21:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0a08b700 1 -- 172.21.15.158:0/733761818 learned_addr learned my addr 172.21.15.158:0/733761818 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb09089700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151220 0x7feb041598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0988a700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040cf190 0x7feb0414e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0a08b700 1 -- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151220 msgr2=0x7feb041598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0a08b700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151220 0x7feb041598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.499+0000 7feb0a08b700 1 -- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040cf190 msgr2=0x7feb0414e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feb0a08b700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040cf190 0x7feb0414e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feb0a08b700 1 -- 172.21.15.158:0/733761818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf40094a0 con 0x7feb04151600 2023-10-16T11:21:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feb0a08b700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb04151600 0x7feb0415da40 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7feafc014040 tx=0x7feafc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feafa7fc700 1 -- 172.21.15.158:0/733761818 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc0045d0 con 0x7feb04151600 2023-10-16T11:21:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feafa7fc700 1 -- 172.21.15.158:0/733761818 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc020070 con 0x7feb04151600 2023-10-16T11:21:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feafa7fc700 1 -- 172.21.15.158:0/733761818 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc027480 con 0x7feb04151600 2023-10-16T11:21:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feb0baee700 1 -- 172.21.15.158:0/733761818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc009a70 con 0x7feb04151600 2023-10-16T11:21:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.500+0000 7feb0baee700 1 -- 172.21.15.158:0/733761818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb04155a80 con 0x7feb04151600 2023-10-16T11:21:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.501+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae00052f0 con 0x7feb04151600 2023-10-16T11:21:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.502+0000 7feafa7fc700 1 -- 172.21.15.158:0/733761818 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7feafc012430 con 0x7feb04151600 2023-10-16T11:21:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.502+0000 7feafa7fc700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7feae806b140 0x7feae806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:28.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.502+0000 7feafa7fc700 1 -- 172.21.15.158:0/733761818 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7feafc022070 con 0x7feb04151600 2023-10-16T11:21:28.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.502+0000 7feb0988a700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7feae806b140 0x7feae806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:28.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.503+0000 7feb0988a700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7feae806b140 0x7feae806d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb0406d480 tx=0x7feaf4009210 comp rx=0 tx=0).ready entity=mgr.12843 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:28.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.506+0000 7feafa7fc700 1 -- 172.21.15.158:0/733761818 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feafc056490 con 0x7feb04151600 2023-10-16T11:21:28.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.755+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7feae0005b20 con 0x7feb04151600 2023-10-16T11:21:28.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.888+0000 7feafa7fc700 1 -- 172.21.15.158:0/733761818 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v158) v1 ==== 92+0+0 (secure 0 0 0) 0x7feafc059ab0 con 0x7feb04151600 2023-10-16T11:21:28.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7feae806b140 msgr2=0x7feae806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7feae806b140 0x7feae806d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb0406d480 tx=0x7feaf4009210 comp rx=0 tx=0).stop 2023-10-16T11:21:28.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb04151600 msgr2=0x7feb0415da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:28.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb04151600 0x7feb0415da40 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7feafc014040 tx=0x7feafc004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 shutdown_connections 2023-10-16T11:21:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7feae806b140 0x7feae806d600 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb04151600 0x7feb0415da40 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb040cf190 0x7feb0414e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 --2- 172.21.15.158:0/733761818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb04151220 0x7feb041598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 >> 172.21.15.158:0/733761818 conn(0x7feb040cc5b0 msgr2=0x7feb0406dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 shutdown_connections 2023-10-16T11:21:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:28.893+0000 7feae7fff700 1 -- 172.21.15.158:0/733761818 wait complete. 2023-10-16T11:21:28.908 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T11:21:29.032 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.030+0000 7fa739dd6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:21:29.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.033+0000 7f9e8653b700 1 Processor -- start 2023-10-16T11:21:29.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.033+0000 7f9e8653b700 1 -- start start 2023-10-16T11:21:29.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e8653b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8015da40 0x7f9e800caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e8653b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e800cc590 0x7f9e8006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e8653b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9e8006ae10 0x7f9e800ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e8653b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8006b280 con 0x7f9e800cc590 2023-10-16T11:21:29.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e8653b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e800cbd30 con 0x7f9e8006ae10 2023-10-16T11:21:29.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e8653b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800cc1c0 con 0x7f9e8015da40 2023-10-16T11:21:29.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e7f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e800cc590 0x7f9e8006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e7f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e800cc590 0x7f9e8006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42482/0 (socket says 172.21.15.158:42482) 2023-10-16T11:21:29.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e7f7fe700 1 -- 172.21.15.158:0/4262585707 learned_addr learned my addr 172.21.15.158:0/4262585707 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:29.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.034+0000 7f9e7ffff700 1 --2- 172.21.15.158:0/4262585707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8015da40 0x7f9e800caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e84ad8700 1 --2- 172.21.15.158:0/4262585707 >> v2:172.21.15.172:3300/0 conn(0x7f9e8006ae10 0x7f9e800ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7f7fe700 1 -- 172.21.15.158:0/4262585707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8015da40 msgr2=0x7f9e800caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:29.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7f7fe700 1 --2- 172.21.15.158:0/4262585707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8015da40 0x7f9e800caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7f7fe700 1 -- 172.21.15.158:0/4262585707 >> v2:172.21.15.172:3300/0 conn(0x7f9e8006ae10 msgr2=0x7f9e800ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:29.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7f7fe700 1 --2- 172.21.15.158:0/4262585707 >> v2:172.21.15.172:3300/0 conn(0x7f9e8006ae10 0x7f9e800ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7f7fe700 1 -- 172.21.15.158:0/4262585707 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8015de20 con 0x7f9e800cc590 2023-10-16T11:21:29.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7f7fe700 1 --2- 172.21.15.158:0/4262585707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e8006a8d0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f9e800d1380 tx=0x7f9e74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81dbaddef37b6e8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:29.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7effd700 1 -- 172.21.15.158:0/4262585707 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74011830 con 0x7f9e800cc590 2023-10-16T11:21:29.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7effd700 1 -- 172.21.15.158:0/4262585707 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e74011990 con 0x7f9e800cc590 2023-10-16T11:21:29.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.035+0000 7f9e7effd700 1 -- 172.21.15.158:0/4262585707 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74011d20 con 0x7f9e800cc590 2023-10-16T11:21:29.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 -- 172.21.15.158:0/4262585707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 msgr2=0x7f9e8006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 --2- 172.21.15.158:0/4262585707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e8006a8d0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f9e800d1380 tx=0x7f9e74009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 -- 172.21.15.158:0/4262585707 shutdown_connections 2023-10-16T11:21:29.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 --2- 172.21.15.158:0/4262585707 >> v2:172.21.15.172:3300/0 conn(0x7f9e8006ae10 0x7f9e800ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 --2- 172.21.15.158:0/4262585707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8015da40 0x7f9e800caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 --2- 172.21.15.158:0/4262585707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e8006a8d0 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 -- 172.21.15.158:0/4262585707 >> 172.21.15.158:0/4262585707 conn(0x7f9e80099a10 msgr2=0x7f9e800d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:29.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 -- 172.21.15.158:0/4262585707 shutdown_connections 2023-10-16T11:21:29.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.036+0000 7f9e8653b700 1 -- 172.21.15.158:0/4262585707 wait complete. 2023-10-16T11:21:29.043 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.036+0000 7ffa80ac7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:21:29.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.037+0000 7f9e8653b700 1 Processor -- start 2023-10-16T11:21:29.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.037+0000 7f9e8653b700 1 -- start start 2023-10-16T11:21:29.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e8653b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8006ae10 0x7f9e801e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e8653b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e800b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e8653b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e8015da40 0x7f9e801567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e8653b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e801e7550 con 0x7f9e800cc590 2023-10-16T11:21:29.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e8653b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e801e68d0 con 0x7f9e8015da40 2023-10-16T11:21:29.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e8653b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e801e6bb0 con 0x7f9e8006ae10 2023-10-16T11:21:29.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e7f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e800b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e7f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e800b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42488/0 (socket says 172.21.15.158:42488) 2023-10-16T11:21:29.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e7f7fe700 1 -- 172.21.15.158:0/3123331653 learned_addr learned my addr 172.21.15.158:0/3123331653 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:29.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e7ffff700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8006ae10 0x7f9e801e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e84ad8700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e8015da40 0x7f9e801567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e84ad8700 1 -- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8006ae10 msgr2=0x7f9e801e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e84ad8700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8006ae10 0x7f9e801e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e84ad8700 1 -- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 msgr2=0x7f9e800b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e84ad8700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e800b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e84ad8700 1 -- 172.21.15.158:0/3123331653 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e74009550 con 0x7f9e8015da40 2023-10-16T11:21:29.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e7ffff700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8006ae10 0x7f9e801e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:29.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.038+0000 7f9e7f7fe700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e800b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:29.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.039+0000 7f9e84ad8700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e8015da40 0x7f9e801567f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800eed0 tx=0x7f9e7800b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:29.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.039+0000 7f9e7cff9700 1 -- 172.21.15.158:0/3123331653 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78010040 con 0x7f9e8015da40 2023-10-16T11:21:29.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.039+0000 7f9e7cff9700 1 -- 172.21.15.158:0/3123331653 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7800ba80 con 0x7f9e8015da40 2023-10-16T11:21:29.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.039+0000 7f9e7cff9700 1 -- 172.21.15.158:0/3123331653 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800be30 con 0x7f9e8015da40 2023-10-16T11:21:29.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.039+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e801e6e90 con 0x7f9e8015da40 2023-10-16T11:21:29.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.039+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e800b3f10 con 0x7f9e8015da40 2023-10-16T11:21:29.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.040+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e800681b0 con 0x7f9e8015da40 2023-10-16T11:21:29.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.040+0000 7f9e7cff9700 1 -- 172.21.15.158:0/3123331653 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 158) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f9e7801d400 con 0x7f9e8015da40 2023-10-16T11:21:29.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.040+0000 7f9e7cff9700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 0x7f9e6c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.041+0000 7f9e7ffff700 1 -- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 msgr2=0x7f9e6c06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:29.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.041+0000 7f9e7cff9700 1 -- 172.21.15.158:0/3123331653 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f9e7808a5e0 con 0x7f9e8015da40 2023-10-16T11:21:29.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.041+0000 7f9e7ffff700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 0x7f9e6c06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:29.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.045+0000 7f9e7cff9700 1 -- 172.21.15.158:0/3123331653 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e78054e80 con 0x7f9e8015da40 2023-10-16T11:21:29.099 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.097+0000 7fa739dd6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:29.107 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.105+0000 7ffa80ac7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:29.171 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.169+0000 7fa739dd6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:21:29.180 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.179+0000 7ffa80ac7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:21:29.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.242+0000 7f9e7ffff700 1 -- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 msgr2=0x7f9e6c06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:29.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.242+0000 7f9e7ffff700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 0x7f9e6c06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:29.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.244+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9e8006f7b0 con 0x7f9e8015da40 2023-10-16T11:21:29.248 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.247+0000 7fa739dd6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:21:29.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.249+0000 7f9e7cff9700 1 -- 172.21.15.158:0/3123331653 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9e780584a0 con 0x7f9e8015da40 2023-10-16T11:21:29.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 msgr2=0x7f9e6c06d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:29.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 0x7f9e6c06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e8015da40 msgr2=0x7f9e801567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e8015da40 0x7f9e801567f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800eed0 tx=0x7f9e7800b390 comp rx=0 tx=0).stop 2023-10-16T11:21:29.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 shutdown_connections 2023-10-16T11:21:29.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f9e6c06b190 0x7f9e6c06d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e8015da40 0x7f9e801567f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e8006ae10 0x7f9e801e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 --2- 172.21.15.158:0/3123331653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e800cc590 0x7f9e800b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.251+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 >> 172.21.15.158:0/3123331653 conn(0x7f9e80099a10 msgr2=0x7f9e8006ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:29.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.252+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 shutdown_connections 2023-10-16T11:21:29.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.252+0000 7f9e8653b700 1 -- 172.21.15.158:0/3123331653 wait complete. 2023-10-16T11:21:29.261 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.260+0000 7ffa80ac7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:21:29.266 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T11:21:29.390 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.389+0000 7fa739dd6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:21:29.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.391+0000 7febd9159700 1 Processor -- start 2023-10-16T11:21:29.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.391+0000 7febd9159700 1 -- start start 2023-10-16T11:21:29.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.392+0000 7febd9159700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd415dc40 0x7febd415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.392+0000 7febd9159700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd415f380 con 0x7febd406d9c0 2023-10-16T11:21:29.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.392+0000 7febd9159700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd406ffa0 con 0x7febd406de40 2023-10-16T11:21:29.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.392+0000 7febd9159700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd406e2d0 con 0x7febd415dc40 2023-10-16T11:21:29.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.392+0000 7febd259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7febd406d9c0 0x7febd406f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42690/0 (socket says 172.21.15.158:42690) 2023-10-16T11:21:29.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.392+0000 7febd259c700 1 -- 172.21.15.158:0/2093599593 learned_addr learned my addr 172.21.15.158:0/2093599593 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:29.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.392+0000 7febd2d9d700 1 --2- 172.21.15.158:0/2093599593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd415dc40 0x7febd415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd1d9b700 1 -- 172.21.15.158:0/2093599593 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669610054 0 0) 0x7febc8001110 con 0x7febd406d9c0 2023-10-16T11:21:29.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd1d9b700 1 -- 172.21.15.158:0/2093599593 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc0035b0 con 0x7febd406d9c0 2023-10-16T11:21:29.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd1d9b700 1 -- 172.21.15.158:0/2093599593 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544196014 0 0) 0x7febd406ffa0 con 0x7febd406de40 2023-10-16T11:21:29.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd2d9d700 1 -- 172.21.15.158:0/2093599593 >> v1:172.21.15.172:6789/0 conn(0x7febd406de40 legacy=0x7febd406f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd2d9d700 1 -- 172.21.15.158:0/2093599593 >> v1:172.21.15.158:6789/0 conn(0x7febd406d9c0 legacy=0x7febd406f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd2d9d700 1 -- 172.21.15.158:0/2093599593 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4071e70 con 0x7febd415dc40 2023-10-16T11:21:29.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd2d9d700 1 --2- 172.21.15.158:0/2093599593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd415dc40 0x7febd415ee40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7febc4009870 tx=0x7febc4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43b21c7c3610fbe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:29.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd1d9b700 1 -- 172.21.15.158:0/2093599593 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc4011830 con 0x7febd415dc40 2023-10-16T11:21:29.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd1d9b700 1 -- 172.21.15.158:0/2093599593 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc4011990 con 0x7febd415dc40 2023-10-16T11:21:29.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.393+0000 7febd1d9b700 1 -- 172.21.15.158:0/2093599593 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc4011d20 con 0x7febd415dc40 2023-10-16T11:21:29.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.394+0000 7febd9159700 1 -- 172.21.15.158:0/2093599593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd415dc40 msgr2=0x7febd415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.394+0000 7febd9159700 1 --2- 172.21.15.158:0/2093599593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd415dc40 0x7febd415ee40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7febc4009870 tx=0x7febc4009b80 comp rx=0 tx=0).stop 2023-10-16T11:21:29.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.394+0000 7febd9159700 1 -- 172.21.15.158:0/2093599593 shutdown_connections 2023-10-16T11:21:29.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.394+0000 7febd9159700 1 --2- 172.21.15.158:0/2093599593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd415dc40 0x7febd415ee40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.394+0000 7febd9159700 1 -- 172.21.15.158:0/2093599593 >> 172.21.15.158:0/2093599593 conn(0x7febd40d03b0 msgr2=0x7febd40d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:29.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.394+0000 7febd9159700 1 -- 172.21.15.158:0/2093599593 shutdown_connections 2023-10-16T11:21:29.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.394+0000 7febd9159700 1 -- 172.21.15.158:0/2093599593 wait complete. 2023-10-16T11:21:29.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.395+0000 7febd9159700 1 Processor -- start 2023-10-16T11:21:29.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.395+0000 7febd9159700 1 -- start start 2023-10-16T11:21:29.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd9159700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd406d9c0 0x7febd40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd9159700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febd406de40 0x7febd40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd9159700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd415dc40 0x7febd4158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd9159700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd40cdd80 con 0x7febd415dc40 2023-10-16T11:21:29.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd9159700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd40c87a0 con 0x7febd406de40 2023-10-16T11:21:29.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd9159700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd40acf50 con 0x7febd406d9c0 2023-10-16T11:21:29.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd415dc40 0x7febd4158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd415dc40 0x7febd4158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42522/0 (socket says 172.21.15.158:42522) 2023-10-16T11:21:29.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd359e700 1 -- 172.21.15.158:0/2212288350 learned_addr learned my addr 172.21.15.158:0/2212288350 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:29.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd2d9d700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd406d9c0 0x7febd40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.396+0000 7febd259c700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febd406de40 0x7febd40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd359e700 1 -- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd406d9c0 msgr2=0x7febd40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd359e700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd406d9c0 0x7febd40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd359e700 1 -- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febd406de40 msgr2=0x7febd40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd359e700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febd406de40 0x7febd40cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd359e700 1 -- 172.21.15.158:0/2212288350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40094a0 con 0x7febd415dc40 2023-10-16T11:21:29.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd359e700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd415dc40 0x7febd4158e90 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7febcc001560 tx=0x7febcc00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:29.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febbb7fe700 1 -- 172.21.15.158:0/2212288350 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc003e30 con 0x7febd415dc40 2023-10-16T11:21:29.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febbb7fe700 1 -- 172.21.15.158:0/2212288350 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc005070 con 0x7febd415dc40 2023-10-16T11:21:29.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febbb7fe700 1 -- 172.21.15.158:0/2212288350 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc005420 con 0x7febd415dc40 2023-10-16T11:21:29.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd40ad230 con 0x7febd415dc40 2023-10-16T11:21:29.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.397+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd40ad7f0 con 0x7febd415dc40 2023-10-16T11:21:29.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.398+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd40c89a0 con 0x7febd415dc40 2023-10-16T11:21:29.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.399+0000 7febbb7fe700 1 -- 172.21.15.158:0/2212288350 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 158) v1 ==== 89450+0+0 (secure 0 0 0) 0x7febcc005580 con 0x7febd415dc40 2023-10-16T11:21:29.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.399+0000 7febbb7fe700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 0x7febc006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.399+0000 7febd2d9d700 1 -- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 msgr2=0x7febc006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:29.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.399+0000 7febbb7fe700 1 -- 172.21.15.158:0/2212288350 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7febcc089650 con 0x7febd415dc40 2023-10-16T11:21:29.407 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.399+0000 7febd2d9d700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 0x7febc006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:29.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.403+0000 7febbb7fe700 1 -- 172.21.15.158:0/2212288350 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febcc0540a0 con 0x7febd415dc40 2023-10-16T11:21:29.413 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.412+0000 7ffa80ac7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:21:29.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.600+0000 7febd2d9d700 1 -- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 msgr2=0x7febc006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:29.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.600+0000 7febd2d9d700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 0x7febc006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:29.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.604+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7febd40ada10 con 0x7febd415dc40 2023-10-16T11:21:29.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.608+0000 7febbb7fe700 1 -- 172.21.15.158:0/2212288350 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7febcc0576c0 con 0x7febd415dc40 2023-10-16T11:21:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 msgr2=0x7febc006d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 0x7febc006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd415dc40 msgr2=0x7febd4158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd415dc40 0x7febd4158e90 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7febcc001560 tx=0x7febcc00cb50 comp rx=0 tx=0).stop 2023-10-16T11:21:29.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 shutdown_connections 2023-10-16T11:21:29.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7febc006b190 0x7febc006d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd415dc40 0x7febd4158e90 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febd406d9c0 0x7febd40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 --2- 172.21.15.158:0/2212288350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febd406de40 0x7febd40cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 >> 172.21.15.158:0/2212288350 conn(0x7febd40d03b0 msgr2=0x7febd40b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:29.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 shutdown_connections 2023-10-16T11:21:29.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.611+0000 7febd9159700 1 -- 172.21.15.158:0/2212288350 wait complete. 2023-10-16T11:21:29.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:29.625 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T11:21:29.649 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T11:21:29.719 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.718+0000 7fa739dd6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:21:29.755 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.754+0000 7ffa80ac7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:21:29.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.814+0000 7f2e23afd700 1 Processor -- start 2023-10-16T11:21:29.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.814+0000 7f2e23afd700 1 -- start start 2023-10-16T11:21:29.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e23afd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e1c15dad0 0x7f2e1c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e23afd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 0x7f2e1c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e23afd700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1c0d0ae0 con 0x7f2e1c15deb0 2023-10-16T11:21:29.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e23afd700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1c0d12d0 con 0x7f2e1c15dad0 2023-10-16T11:21:29.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e23afd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c0cbdf0 con 0x7f2e1c0ccc00 2023-10-16T11:21:29.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21899700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e1c15dad0 0x7f2e1c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21899700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e1c15dad0 0x7f2e1c0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42544/0 (socket says 172.21.15.158:42544) 2023-10-16T11:21:29.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21899700 1 -- 172.21.15.158:0/1049014395 learned_addr learned my addr 172.21.15.158:0/1049014395 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:29.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21098700 1 --1- 172.21.15.158:0/1049014395 >> v1:172.21.15.158:6789/0 conn(0x7f2e1c15deb0 0x7f2e1c0ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42722/0 (socket says 172.21.15.158:42722) 2023-10-16T11:21:29.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e2209a700 1 --2- 172.21.15.158:0/1049014395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 0x7f2e1c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21899700 1 -- 172.21.15.158:0/1049014395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 msgr2=0x7f2e1c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:29.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21899700 1 --2- 172.21.15.158:0/1049014395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 0x7f2e1c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21899700 1 -- 172.21.15.158:0/1049014395 >> v1:172.21.15.158:6789/0 conn(0x7f2e1c15deb0 legacy=0x7f2e1c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e20897700 1 -- 172.21.15.158:0/1049014395 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971326875 0 0) 0x7f2e1c0d0ae0 con 0x7f2e1c15deb0 2023-10-16T11:21:29.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.815+0000 7f2e21899700 1 -- 172.21.15.158:0/1049014395 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c0cc280 con 0x7f2e1c15dad0 2023-10-16T11:21:29.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e21899700 1 --2- 172.21.15.158:0/1049014395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15dad0 0x7f2e1c0cdfa0 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f2e180098f0 tx=0x7f2e18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be61f324a7bf082c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:29.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e20897700 1 -- 172.21.15.158:0/1049014395 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18011830 con 0x7f2e1c15dad0 2023-10-16T11:21:29.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e20897700 1 -- 172.21.15.158:0/1049014395 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e18011990 con 0x7f2e1c15dad0 2023-10-16T11:21:29.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e20897700 1 -- 172.21.15.158:0/1049014395 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18011d20 con 0x7f2e1c15dad0 2023-10-16T11:21:29.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e23afd700 1 -- 172.21.15.158:0/1049014395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15dad0 msgr2=0x7f2e1c0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e23afd700 1 --2- 172.21.15.158:0/1049014395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15dad0 0x7f2e1c0cdfa0 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f2e180098f0 tx=0x7f2e18009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:29.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e23afd700 1 -- 172.21.15.158:0/1049014395 shutdown_connections 2023-10-16T11:21:29.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e23afd700 1 --2- 172.21.15.158:0/1049014395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15dad0 0x7f2e1c0cdfa0 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e23afd700 1 --2- 172.21.15.158:0/1049014395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 0x7f2e1c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.816+0000 7f2e23afd700 1 -- 172.21.15.158:0/1049014395 >> 172.21.15.158:0/1049014395 conn(0x7f2e1c099a10 msgr2=0x7f2e1c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:29.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.817+0000 7f2e23afd700 1 -- 172.21.15.158:0/1049014395 shutdown_connections 2023-10-16T11:21:29.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.817+0000 7f2e23afd700 1 -- 172.21.15.158:0/1049014395 wait complete. 2023-10-16T11:21:29.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.818+0000 7f2e23afd700 1 Processor -- start 2023-10-16T11:21:29.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.818+0000 7f2e23afd700 1 -- start start 2023-10-16T11:21:29.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e23afd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 0x7f2e1c0b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e23afd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1c15dad0 0x7f2e1c154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e23afd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15deb0 0x7f2e1c0c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e23afd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c154e80 con 0x7f2e1c15deb0 2023-10-16T11:21:29.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e23afd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c0b49c0 con 0x7f2e1c15dad0 2023-10-16T11:21:29.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e23afd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c0b4ca0 con 0x7f2e1c0ccc00 2023-10-16T11:21:29.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e2209a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15deb0 0x7f2e1c0c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e2209a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15deb0 0x7f2e1c0c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42554/0 (socket says 172.21.15.158:42554) 2023-10-16T11:21:29.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e2209a700 1 -- 172.21.15.158:0/307468640 learned_addr learned my addr 172.21.15.158:0/307468640 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:29.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 0x7f2e1c0b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.819+0000 7f2e21098700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1c15dad0 0x7f2e1c154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:29.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e2209a700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 msgr2=0x7f2e1c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e2209a700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e1c0ccc00 0x7f2e1c0b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e2209a700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1c15dad0 msgr2=0x7f2e1c154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:29.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e2209a700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1c15dad0 0x7f2e1c154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:29.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e2209a700 1 -- 172.21.15.158:0/307468640 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009550 con 0x7f2e1c15deb0 2023-10-16T11:21:29.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e2209a700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15deb0 0x7f2e1c0c9880 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400efc0 tx=0x7f2e1400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:29.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e14009ea0 con 0x7f2e1c15deb0 2023-10-16T11:21:29.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1400f040 con 0x7f2e1c15deb0 2023-10-16T11:21:29.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c0b4f80 con 0x7f2e1c15deb0 2023-10-16T11:21:29.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400ced0 con 0x7f2e1c15deb0 2023-10-16T11:21:29.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.820+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c0b5540 con 0x7f2e1c15deb0 2023-10-16T11:21:29.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.821+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1c0b5760 con 0x7f2e1c15deb0 2023-10-16T11:21:29.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.822+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 158) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f2e14014480 con 0x7f2e1c15deb0 2023-10-16T11:21:29.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.822+0000 7f2e127fc700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 0x7f2e0006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:29.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.822+0000 7f2e21899700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 msgr2=0x7f2e0006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:29.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.822+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f2e1408ad30 con 0x7f2e1c15deb0 2023-10-16T11:21:29.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.822+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 0x7f2e0006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:29.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:29.826+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e140557c0 con 0x7f2e1c15deb0 2023-10-16T11:21:29.836 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.835+0000 7fa739dd6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:21:29.876 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.875+0000 7ffa80ac7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:21:29.910 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.909+0000 7fa739dd6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:21:29.952 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:29.950+0000 7ffa80ac7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:21:29.991 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:29.989+0000 7fa739dd6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:21:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:30.023+0000 7f2e21899700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 msgr2=0x7f2e0006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:30.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:30.023+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 0x7f2e0006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:30.033 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:30.031+0000 7ffa80ac7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:21:30.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:30.052+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 --> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e1c0b4e10 con 0x7f2e0006b190 2023-10-16T11:21:30.360 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:30.359+0000 7fa739dd6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:21:30.410 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:30.409+0000 7ffa80ac7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:21:30.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:30.424+0000 7f2e21899700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 msgr2=0x7f2e0006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:30.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:30.424+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 0x7f2e0006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T11:21:30.930 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:30.929+0000 7fa739dd6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:21:30.980 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:30.979+0000 7ffa80ac7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:21:30.999 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:30.999+0000 7fa739dd6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:21:31.050 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:31.048+0000 7ffa80ac7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:21:31.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:31.225+0000 7f2e21899700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 msgr2=0x7f2e0006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:31.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:31.225+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 0x7f2e0006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T11:21:31.629 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:31.628+0000 7fa739dd6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:31.684 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:31.682+0000 7ffa80ac7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:31.705 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:31.705+0000 7fa739dd6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:21:31.763 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:31.762+0000 7ffa80ac7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:21:31.850 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:31.849+0000 7fa739dd6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:31.909 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:31.907+0000 7ffa80ac7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:32.079 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:32.079+0000 7fa739dd6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:21:32.139 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:32.137+0000 7ffa80ac7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:21:32.153 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:32.152+0000 7fa739dd6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:21:32.213 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:32.211+0000 7ffa80ac7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:21:32.417 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:32.416+0000 7fa739dd6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:21:32.496 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:32.494+0000 7ffa80ac7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:21:32.639 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:32.639+0000 7fa739dd6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:21:32.719 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:32.718+0000 7ffa80ac7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:21:32.790 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:32.790+0000 7fa739dd6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:21:32.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:32.826+0000 7f2e21899700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 msgr2=0x7f2e0006d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/3125181137 2023-10-16T11:21:32.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:32.826+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 0x7f2e0006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T11:21:32.873 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:32.871+0000 7ffa80ac7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:21:32.904 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:32.904+0000 7fa739dd6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:21:32.988 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:32.987+0000 7ffa80ac7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:21:33.062 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:33.061+0000 7fa739dd6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:21:33.146 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:33.145+0000 7ffa80ac7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:21:33.325 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:33.324+0000 7fa739dd6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:33.412 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:33.410+0000 7ffa80ac7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:33.490 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:33.489+0000 7fa739dd6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:21:33.576 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:33.575+0000 7ffa80ac7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:21:34.017 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:34.016+0000 7fa739dd6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:21:34.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:34.086+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 159) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f2e14055b90 con 0x7f2e1c15deb0 2023-10-16T11:21:34.107 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.105+0000 7ffa80ac7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:21:34.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:34.126+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 160) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f2e14044410 con 0x7f2e1c15deb0 2023-10-16T11:21:34.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:34.127+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 msgr2=0x7f2e0006d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:34.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:34.127+0000 7f2e127fc700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/3125181137,v1:172.21.15.158:6833/3125181137] conn(0x7f2e0006b190 0x7f2e0006d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:34.271 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.268+0000 7ff9f3aa2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.271 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.268+0000 7ff9f3aa2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.271 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f3aa2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.271 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f3aa2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.272 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f3aa2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.272 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f6aa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.272 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f6aa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.272 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f6aa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.272 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f6aa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:34.272 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:34.269+0000 7ff9f6aa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:35.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.133+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mon.0 v2:172.21.15.158:3300/0 9 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2e14056460 con 0x7f2e1c15deb0 2023-10-16T11:21:35.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.133+0000 7f2e127fc700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f2e0001cc80 0x7f2e0001d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.133+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 --> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e1c0b4e10 con 0x7f2e0001cc80 2023-10-16T11:21:35.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.135+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f2e0001cc80 0x7f2e0001d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.136+0000 7f2e21899700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f2e0001cc80 0x7f2e0001d060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009c40 tx=0x7f2e18008030 comp rx=0 tx=0).ready entity=mgr.13392 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:35.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.195+0000 7f2e127fc700 1 -- 172.21.15.158:0/307468640 <== mgr.13392 v2:172.21.15.158:6832/1196176990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2e1c0b4e10 con 0x7f2e0001cc80 2023-10-16T11:21:35.197 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule set for path snap_test_dir1 2023-10-16T11:21:35.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.197+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f2e0001cc80 msgr2=0x7f2e0001d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.197+0000 7f2e23afd700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f2e0001cc80 0x7f2e0001d060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009c40 tx=0x7f2e18008030 comp rx=0 tx=0).stop 2023-10-16T11:21:35.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.197+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15deb0 msgr2=0x7f2e1c0c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.197+0000 7f2e23afd700 1 --2- 172.21.15.158:0/307468640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e1c15deb0 0x7f2e1c0c9880 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400efc0 tx=0x7f2e1400c3a0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.197+0000 7f2e2209a700 1 -- 172.21.15.158:0/307468640 reap_dead start 2023-10-16T11:21:35.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.198+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 shutdown_connections 2023-10-16T11:21:35.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.198+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 >> 172.21.15.158:0/307468640 conn(0x7f2e1c099a10 msgr2=0x7f2e1c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:35.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.198+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 shutdown_connections 2023-10-16T11:21:35.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.198+0000 7f2e23afd700 1 -- 172.21.15.158:0/307468640 wait complete. 2023-10-16T11:21:35.212 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T11:21:35.213 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T11:21:35.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.338+0000 7fbf65ff0700 1 Processor -- start 2023-10-16T11:21:35.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.338+0000 7fbf65ff0700 1 -- start start 2023-10-16T11:21:35.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf65ff0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf6015dfc0 0x7fbf6006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf65ff0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6015e510 con 0x7fbf6015dbd0 2023-10-16T11:21:35.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf65ff0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6015e7f0 con 0x7fbf6006afe0 2023-10-16T11:21:35.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf65ff0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf600ccbb0 con 0x7fbf6015dfc0 2023-10-16T11:21:35.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbf6015dbd0 0x7fbf6006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42898/0 (socket says 172.21.15.158:42898) 2023-10-16T11:21:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5f7fe700 1 -- 172.21.15.158:0/1467255756 learned_addr learned my addr 172.21.15.158:0/1467255756 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5ffff700 1 --2- 172.21.15.158:0/1467255756 >> v2:172.21.15.172:3300/0 conn(0x7fbf6015dfc0 0x7fbf6006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467707811 0 0) 0x7fbf6015e510 con 0x7fbf6015dbd0 2023-10-16T11:21:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf480035b0 con 0x7fbf6015dbd0 2023-10-16T11:21:35.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1062444155 0 0) 0x7fbf480035b0 con 0x7fbf6015dbd0 2023-10-16T11:21:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 >> v2:172.21.15.172:3300/0 conn(0x7fbf6015dfc0 msgr2=0x7fbf6006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 --2- 172.21.15.158:0/1467255756 >> v2:172.21.15.172:3300/0 conn(0x7fbf6015dfc0 0x7fbf6006df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 >> v1:172.21.15.172:6789/0 conn(0x7fbf6006afe0 legacy=0x7fbf600ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6015ead0 con 0x7fbf6015dbd0 2023-10-16T11:21:35.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fbf50001670 con 0x7fbf6015dbd0 2023-10-16T11:21:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 >> v1:172.21.15.158:6789/0 conn(0x7fbf6015dbd0 legacy=0x7fbf6006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf48003a00 0x7fbf48003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf48004850 0x7fbf48004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48005210 0x7fbf4800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600ccbb0 con 0x7fbf48003a00 2023-10-16T11:21:35.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4800c1b0 con 0x7fbf48005210 2023-10-16T11:21:35.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4800c370 con 0x7fbf48004850 2023-10-16T11:21:35.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5f7fe700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf48003a00 0x7fbf48003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5ffff700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48005210 0x7fbf4800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.339+0000 7fbf5effd700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf48004850 0x7fbf48004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5f7fe700 1 -- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf48004850 msgr2=0x7fbf48004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:35.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5f7fe700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf48004850 0x7fbf48004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5f7fe700 1 -- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48005210 msgr2=0x7fbf4800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:35.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5f7fe700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48005210 0x7fbf4800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5f7fe700 1 -- 172.21.15.158:0/1467255756 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480035b0 con 0x7fbf48003a00 2023-10-16T11:21:35.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5ffff700 1 -- 172.21.15.158:0/1467255756 reap_dead start 2023-10-16T11:21:35.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5f7fe700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf48003a00 0x7fbf48003e20 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c240 tx=0x7fbf48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca24a79ab05fe1ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:35.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50007c10 con 0x7fbf48003a00 2023-10-16T11:21:35.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5000cd80 con 0x7fbf48003a00 2023-10-16T11:21:35.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf5e7fc700 1 -- 172.21.15.158:0/1467255756 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50005ba0 con 0x7fbf48003a00 2023-10-16T11:21:35.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf65ff0700 1 -- 172.21.15.158:0/1467255756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf48003a00 msgr2=0x7fbf48003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf48003a00 0x7fbf48003e20 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c240 tx=0x7fbf48005880 comp rx=0 tx=0).stop 2023-10-16T11:21:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.340+0000 7fbf65ff0700 1 -- 172.21.15.158:0/1467255756 shutdown_connections 2023-10-16T11:21:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.341+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/1467255756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf48003a00 0x7fbf48003e20 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.341+0000 7fbf65ff0700 1 -- 172.21.15.158:0/1467255756 >> 172.21.15.158:0/1467255756 conn(0x7fbf600b1f00 msgr2=0x7fbf600b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.341+0000 7fbf65ff0700 1 -- 172.21.15.158:0/1467255756 shutdown_connections 2023-10-16T11:21:35.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.341+0000 7fbf65ff0700 1 -- 172.21.15.158:0/1467255756 wait complete. 2023-10-16T11:21:35.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.342+0000 7fbf65ff0700 1 Processor -- start 2023-10-16T11:21:35.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.342+0000 7fbf65ff0700 1 -- start start 2023-10-16T11:21:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48003a00 0x7fbf60151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf600cae00 0x7fbf600cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf600b4a80 0x7fbf600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600cb7c0 con 0x7fbf600b4a80 2023-10-16T11:21:35.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600c8110 con 0x7fbf48003a00 2023-10-16T11:21:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600c7540 con 0x7fbf600cae00 2023-10-16T11:21:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf600b4a80 0x7fbf600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48003a00 0x7fbf60151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf600b4a80 0x7fbf600b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42738/0 (socket says 172.21.15.158:42738) 2023-10-16T11:21:35.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2668210874 learned_addr learned my addr 172.21.15.158:0/2668210874 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:35.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5effd700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf600cae00 0x7fbf600cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2668210874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf600cae00 msgr2=0x7fbf600cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf600cae00 0x7fbf600cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2668210874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48003a00 msgr2=0x7fbf60151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48003a00 0x7fbf60151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 -- 172.21.15.158:0/2668210874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf600c77e0 con 0x7fbf600b4a80 2023-10-16T11:21:35.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf5ffff700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf600b4a80 0x7fbf600b4f00 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7fbf600ccad0 tx=0x7fbf58011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:35.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf47fff700 1 -- 172.21.15.158:0/2668210874 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5800e040 con 0x7fbf600b4a80 2023-10-16T11:21:35.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf47fff700 1 -- 172.21.15.158:0/2668210874 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5800f040 con 0x7fbf600b4a80 2023-10-16T11:21:35.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf600c7ac0 con 0x7fbf600b4a80 2023-10-16T11:21:35.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf47fff700 1 -- 172.21.15.158:0/2668210874 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5800fc10 con 0x7fbf600b4a80 2023-10-16T11:21:35.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.343+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf60151ec0 con 0x7fbf600b4a80 2023-10-16T11:21:35.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.344+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf60074ad0 con 0x7fbf600b4a80 2023-10-16T11:21:35.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.345+0000 7fbf47fff700 1 -- 172.21.15.158:0/2668210874 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fbf5800fd70 con 0x7fbf600b4a80 2023-10-16T11:21:35.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.345+0000 7fbf47fff700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbf4c06ada0 0x7fbf4c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.345+0000 7fbf47fff700 1 -- 172.21.15.158:0/2668210874 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fbf58089d10 con 0x7fbf600b4a80 2023-10-16T11:21:35.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.345+0000 7fbf5f7fe700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbf4c06ada0 0x7fbf4c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.346+0000 7fbf5f7fe700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbf4c06ada0 0x7fbf4c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbf50000c00 tx=0x7fbf50007000 comp rx=0 tx=0).ready entity=mgr.13392 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:35.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.350+0000 7fbf47fff700 1 -- 172.21.15.158:0/2668210874 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf58054c30 con 0x7fbf600b4a80 2023-10-16T11:21:35.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.569+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 --> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf6015edb0 con 0x7fbf4c06ada0 2023-10-16T11:21:35.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.620+0000 7fbf47fff700 1 -- 172.21.15.158:0/2668210874 <== mgr.13392 v2:172.21.15.158:6832/1196176990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fbf50007b30 con 0x7fbf4c06ada0 2023-10-16T11:21:35.623 INFO:teuthology.orchestra.run.smithi158.stdout:Retention added to path snap_test_dir1 2023-10-16T11:21:35.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbf4c06ada0 msgr2=0x7fbf4c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbf4c06ada0 0x7fbf4c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbf50000c00 tx=0x7fbf50007000 comp rx=0 tx=0).stop 2023-10-16T11:21:35.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf600b4a80 msgr2=0x7fbf600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf600b4a80 0x7fbf600b4f00 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7fbf600ccad0 tx=0x7fbf58011ed0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 shutdown_connections 2023-10-16T11:21:35.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf600b4a80 0x7fbf600b4f00 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbf4c06ada0 0x7fbf4c06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf48003a00 0x7fbf60151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 --2- 172.21.15.158:0/2668210874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf600cae00 0x7fbf600cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 >> 172.21.15.158:0/2668210874 conn(0x7fbf600b1f00 msgr2=0x7fbf6006a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:35.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 shutdown_connections 2023-10-16T11:21:35.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.623+0000 7fbf65ff0700 1 -- 172.21.15.158:0/2668210874 wait complete. 2023-10-16T11:21:35.637 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T11:21:35.637 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-11_22_00 in ~25s... 2023-10-16T11:21:35.638 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T11:21:35.638 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T11:21:35.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.759+0000 7fad52c78700 1 Processor -- start 2023-10-16T11:21:35.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.759+0000 7fad52c78700 1 -- start start 2023-10-16T11:21:35.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad52c78700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad4c163380 0x7fad4c165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad52c78700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0d0ae0 con 0x7fad4c15deb0 2023-10-16T11:21:35.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad52c78700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0d12d0 con 0x7fad4c15dad0 2023-10-16T11:21:35.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad52c78700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c0cbdf0 con 0x7fad4c163380 2023-10-16T11:21:35.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad51215700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad4c163380 0x7fad4c165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad51215700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad4c163380 0x7fad4c165770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42746/0 (socket says 172.21.15.158:42746) 2023-10-16T11:21:35.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad4bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fad4c15deb0 0x7fad4c0ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42918/0 (socket says 172.21.15.158:42918) 2023-10-16T11:21:35.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.760+0000 7fad51215700 1 -- 172.21.15.158:0/684928942 learned_addr learned my addr 172.21.15.158:0/684928942 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:35.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad51215700 1 -- 172.21.15.158:0/684928942 >> v1:172.21.15.172:6789/0 conn(0x7fad4c15dad0 legacy=0x7fad4c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:35.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad51215700 1 -- 172.21.15.158:0/684928942 >> v1:172.21.15.158:6789/0 conn(0x7fad4c15deb0 legacy=0x7fad4c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad51215700 1 -- 172.21.15.158:0/684928942 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c0cc280 con 0x7fad4c163380 2023-10-16T11:21:35.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad51215700 1 --2- 172.21.15.158:0/684928942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c165770 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7fad44009e50 tx=0x7fad4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f116198e01d11d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:35.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad4b7fe700 1 -- 172.21.15.158:0/684928942 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4400e070 con 0x7fad4c163380 2023-10-16T11:21:35.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad4b7fe700 1 -- 172.21.15.158:0/684928942 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44003a20 con 0x7fad4c163380 2023-10-16T11:21:35.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad4b7fe700 1 -- 172.21.15.158:0/684928942 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44003dd0 con 0x7fad4c163380 2023-10-16T11:21:35.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad52c78700 1 -- 172.21.15.158:0/684928942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 msgr2=0x7fad4c165770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad52c78700 1 --2- 172.21.15.158:0/684928942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c165770 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7fad44009e50 tx=0x7fad4400b330 comp rx=0 tx=0).stop 2023-10-16T11:21:35.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad52c78700 1 -- 172.21.15.158:0/684928942 shutdown_connections 2023-10-16T11:21:35.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad52c78700 1 --2- 172.21.15.158:0/684928942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c165770 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad52c78700 1 -- 172.21.15.158:0/684928942 >> 172.21.15.158:0/684928942 conn(0x7fad4c099a10 msgr2=0x7fad4c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:35.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad52c78700 1 -- 172.21.15.158:0/684928942 shutdown_connections 2023-10-16T11:21:35.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.761+0000 7fad52c78700 1 -- 172.21.15.158:0/684928942 wait complete. 2023-10-16T11:21:35.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 Processor -- start 2023-10-16T11:21:35.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 -- start start 2023-10-16T11:21:35.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad4c15dad0 0x7fad4c152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad4c15deb0 0x7fad4c151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c154760 con 0x7fad4c163380 2023-10-16T11:21:35.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c153d10 con 0x7fad4c15dad0 2023-10-16T11:21:35.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.763+0000 7fad52c78700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c153a30 con 0x7fad4c15deb0 2023-10-16T11:21:35.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42752/0 (socket says 172.21.15.158:42752) 2023-10-16T11:21:35.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 -- 172.21.15.158:0/2935569982 learned_addr learned my addr 172.21.15.158:0/2935569982 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:35.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad50a14700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad4c15dad0 0x7fad4c152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad4bfff700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad4c15deb0 0x7fad4c151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 -- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad4c15deb0 msgr2=0x7fad4c151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad4c15deb0 0x7fad4c151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 -- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad4c15dad0 msgr2=0x7fad4c152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:35.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad4c15dad0 0x7fad4c152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:35.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 -- 172.21.15.158:0/2935569982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44009b30 con 0x7fad4c163380 2023-10-16T11:21:35.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad51215700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c154bf0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fad4c070430 tx=0x7fad44004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:35.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad497fa700 1 -- 172.21.15.158:0/2935569982 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44004440 con 0x7fad4c163380 2023-10-16T11:21:35.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad497fa700 1 -- 172.21.15.158:0/2935569982 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad440092e0 con 0x7fad4c163380 2023-10-16T11:21:35.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c154110 con 0x7fad4c163380 2023-10-16T11:21:35.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad497fa700 1 -- 172.21.15.158:0/2935569982 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad440047e0 con 0x7fad4c163380 2023-10-16T11:21:35.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.764+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c152280 con 0x7fad4c163380 2023-10-16T11:21:35.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.765+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c154910 con 0x7fad4c163380 2023-10-16T11:21:35.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.766+0000 7fad497fa700 1 -- 172.21.15.158:0/2935569982 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fad44025450 con 0x7fad4c163380 2023-10-16T11:21:35.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.766+0000 7fad497fa700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fad3006ada0 0x7fad3006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:35.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.766+0000 7fad497fa700 1 -- 172.21.15.158:0/2935569982 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fad4408afe0 con 0x7fad4c163380 2023-10-16T11:21:35.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.766+0000 7fad50a14700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fad3006ada0 0x7fad3006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:35.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.766+0000 7fad50a14700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fad3006ada0 0x7fad3006d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad4c1f6d00 tx=0x7fad40009040 comp rx=0 tx=0).ready entity=mgr.13392 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:35.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.771+0000 7fad497fa700 1 -- 172.21.15.158:0/2935569982 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fad44055e80 con 0x7fad4c163380 2023-10-16T11:21:35.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:35.991+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 --> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad4c0d0d10 con 0x7fad3006ada0 2023-10-16T11:21:36.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.001+0000 7fad497fa700 1 -- 172.21.15.158:0/2935569982 <== mgr.13392 v2:172.21.15.158:6832/1196176990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fad40007e20 con 0x7fad3006ada0 2023-10-16T11:21:36.003 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:36.003 INFO:teuthology.orchestra.run.smithi158.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T11:21:36.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fad3006ada0 msgr2=0x7fad3006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fad3006ada0 0x7fad3006d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad4c1f6d00 tx=0x7fad40009040 comp rx=0 tx=0).stop 2023-10-16T11:21:36.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 msgr2=0x7fad4c154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c154bf0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fad4c070430 tx=0x7fad44004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 shutdown_connections 2023-10-16T11:21:36.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad4c163380 0x7fad4c154bf0 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fad3006ada0 0x7fad3006d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad4c15dad0 0x7fad4c152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 --2- 172.21.15.158:0/2935569982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad4c15deb0 0x7fad4c151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.003+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 >> 172.21.15.158:0/2935569982 conn(0x7fad4c099a10 msgr2=0x7fad4c156650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:36.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.004+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 shutdown_connections 2023-10-16T11:21:36.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.004+0000 7fad52c78700 1 -- 172.21.15.158:0/2935569982 wait complete. 2023-10-16T11:21:36.017 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T11:21:36.018 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T11:21:36.018 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T11:21:36.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.140+0000 7f994f527700 1 Processor -- start 2023-10-16T11:21:36.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.140+0000 7f994f527700 1 -- start start 2023-10-16T11:21:36.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994f527700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9948160d40 0x7f99480ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994f527700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480ce110 con 0x7f99480cf420 2023-10-16T11:21:36.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994f527700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480ce3f0 con 0x7f99480c50c0 2023-10-16T11:21:36.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994f527700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99480c61e0 con 0x7f9948160d40 2023-10-16T11:21:36.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994d2c3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9948160d40 0x7f99480ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994d2c3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9948160d40 0x7f99480ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42756/0 (socket says 172.21.15.158:42756) 2023-10-16T11:21:36.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994d2c3700 1 -- 172.21.15.158:0/1927728682 learned_addr learned my addr 172.21.15.158:0/1927728682 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:36.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994d2c3700 1 -- 172.21.15.158:0/1927728682 >> v1:172.21.15.172:6789/0 conn(0x7f99480c50c0 legacy=0x7f994806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:36.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994d2c3700 1 -- 172.21.15.158:0/1927728682 >> v1:172.21.15.158:6789/0 conn(0x7f99480cf420 legacy=0x7f99480c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994d2c3700 1 -- 172.21.15.158:0/1927728682 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480ce6d0 con 0x7f9948160d40 2023-10-16T11:21:36.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f994d2c3700 1 --2- 172.21.15.158:0/1927728682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9948160d40 0x7f99480ceee0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7f99440098f0 tx=0x7f9944009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d10cf796142f32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:36.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f993ffff700 1 -- 172.21.15.158:0/1927728682 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944004230 con 0x7f9948160d40 2023-10-16T11:21:36.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f993ffff700 1 -- 172.21.15.158:0/1927728682 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9944004390 con 0x7f9948160d40 2023-10-16T11:21:36.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.141+0000 7f993ffff700 1 -- 172.21.15.158:0/1927728682 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99440036a0 con 0x7f9948160d40 2023-10-16T11:21:36.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.142+0000 7f994f527700 1 -- 172.21.15.158:0/1927728682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9948160d40 msgr2=0x7f99480ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.142+0000 7f994f527700 1 --2- 172.21.15.158:0/1927728682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9948160d40 0x7f99480ceee0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7f99440098f0 tx=0x7f9944009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:36.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.142+0000 7f994f527700 1 -- 172.21.15.158:0/1927728682 shutdown_connections 2023-10-16T11:21:36.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.142+0000 7f994f527700 1 --2- 172.21.15.158:0/1927728682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9948160d40 0x7f99480ceee0 unknown :-1 s=CLOSED pgs=3174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.142+0000 7f994f527700 1 -- 172.21.15.158:0/1927728682 >> 172.21.15.158:0/1927728682 conn(0x7f99480cc5b0 msgr2=0x7f99480cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:36.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.142+0000 7f994f527700 1 -- 172.21.15.158:0/1927728682 shutdown_connections 2023-10-16T11:21:36.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.142+0000 7f994f527700 1 -- 172.21.15.158:0/1927728682 wait complete. 2023-10-16T11:21:36.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.143+0000 7f994f527700 1 Processor -- start 2023-10-16T11:21:36.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.143+0000 7f994f527700 1 -- start start 2023-10-16T11:21:36.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994f527700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99480c50c0 0x7f994814d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994f527700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99480cf420 0x7f99480b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994f527700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9948160d40 0x7f99480aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994f527700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480ad9d0 con 0x7f99480c50c0 2023-10-16T11:21:36.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994f527700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480adcb0 con 0x7f9948160d40 2023-10-16T11:21:36.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994f527700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480adf90 con 0x7f99480cf420 2023-10-16T11:21:36.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99480c50c0 0x7f994814d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99480c50c0 0x7f994814d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42762/0 (socket says 172.21.15.158:42762) 2023-10-16T11:21:36.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 -- 172.21.15.158:0/620069251 learned_addr learned my addr 172.21.15.158:0/620069251 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:36.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 -- 172.21.15.158:0/620069251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99480cf420 msgr2=0x7f99480b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99480cf420 0x7f99480b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 -- 172.21.15.158:0/620069251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9948160d40 msgr2=0x7f99480aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9948160d40 0x7f99480aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.144+0000 7f994d2c3700 1 -- 172.21.15.158:0/620069251 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944009550 con 0x7f99480c50c0 2023-10-16T11:21:36.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.145+0000 7f994d2c3700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99480c50c0 0x7f994814d530 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f9948070e90 tx=0x7f9944004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:36.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.145+0000 7f993dffb700 1 -- 172.21.15.158:0/620069251 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994400b660 con 0x7f99480c50c0 2023-10-16T11:21:36.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.145+0000 7f993dffb700 1 -- 172.21.15.158:0/620069251 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994400b7e0 con 0x7f99480c50c0 2023-10-16T11:21:36.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.145+0000 7f993dffb700 1 -- 172.21.15.158:0/620069251 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994400bb90 con 0x7f99480c50c0 2023-10-16T11:21:36.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.145+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99481612a0 con 0x7f99480c50c0 2023-10-16T11:21:36.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.145+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99480ae550 con 0x7f99480c50c0 2023-10-16T11:21:36.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.146+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948066db0 con 0x7f99480c50c0 2023-10-16T11:21:36.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.146+0000 7f993dffb700 1 -- 172.21.15.158:0/620069251 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f994400bcf0 con 0x7f99480c50c0 2023-10-16T11:21:36.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.147+0000 7f993dffb700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f992c06add0 0x7f992c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.147+0000 7f993dffb700 1 -- 172.21.15.158:0/620069251 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f994408b0f0 con 0x7f99480c50c0 2023-10-16T11:21:36.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.147+0000 7f994cac2700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f992c06add0 0x7f992c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.147+0000 7f994cac2700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f992c06add0 0x7f992c06d290 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9938000f60 tx=0x7f9938007a60 comp rx=0 tx=0).ready entity=mgr.13392 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:36.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.151+0000 7f993dffb700 1 -- 172.21.15.158:0/620069251 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9944056010 con 0x7f99480c50c0 2023-10-16T11:21:36.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.378+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 --> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9948161120 con 0x7f992c06add0 2023-10-16T11:21:36.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.436+0000 7f993dffb700 1 -- 172.21.15.158:0/620069251 <== mgr.13392 v2:172.21.15.158:6832/1196176990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9938005080 con 0x7f992c06add0 2023-10-16T11:21:36.438 INFO:teuthology.orchestra.run.smithi158.stdout:Schedule removed for path snap_test_dir1 2023-10-16T11:21:36.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f992c06add0 msgr2=0x7f992c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f992c06add0 0x7f992c06d290 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9938000f60 tx=0x7f9938007a60 comp rx=0 tx=0).stop 2023-10-16T11:21:36.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99480c50c0 msgr2=0x7f994814d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99480c50c0 0x7f994814d530 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f9948070e90 tx=0x7f9944004750 comp rx=0 tx=0).stop 2023-10-16T11:21:36.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 shutdown_connections 2023-10-16T11:21:36.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f992c06add0 0x7f992c06d290 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9948160d40 0x7f99480aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99480c50c0 0x7f994814d530 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 --2- 172.21.15.158:0/620069251 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99480cf420 0x7f99480b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 >> 172.21.15.158:0/620069251 conn(0x7f99480cc5b0 msgr2=0x7f99480cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:36.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 shutdown_connections 2023-10-16T11:21:36.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.439+0000 7f994f527700 1 -- 172.21.15.158:0/620069251 wait complete. 2023-10-16T11:21:36.453 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T11:21:36.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:36.453 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T11:21:36.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:36.471 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T11:21:36.530 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:21:36.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.691+0000 7ffb4e722700 1 Processor -- start 2023-10-16T11:21:36.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.691+0000 7ffb4e722700 1 -- start start 2023-10-16T11:21:36.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb4e722700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 0x7ffb480caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb4e722700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb480cbd30 con 0x7ffb480cc590 2023-10-16T11:21:36.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb4e722700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb480cc1c0 con 0x7ffb4815da40 2023-10-16T11:21:36.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb4e722700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4815de20 con 0x7ffb4806afe0 2023-10-16T11:21:36.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb4ccbf700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffb480cc590 0x7ffb480afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:42966/0 (socket says 172.21.15.158:42966) 2023-10-16T11:21:36.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb4ccbf700 1 -- 172.21.15.158:0/2982301533 learned_addr learned my addr 172.21.15.158:0/2982301533 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:36.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb477fe700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 0x7ffb480caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.692+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502471155 0 0) 0x7ffb480cbd30 con 0x7ffb480cc590 2023-10-16T11:21:36.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300035b0 con 0x7ffb480cc590 2023-10-16T11:21:36.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3714002496 0 0) 0x7ffb300035b0 con 0x7ffb480cc590 2023-10-16T11:21:36.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 msgr2=0x7ffb480caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:36.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 0x7ffb480caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 >> v1:172.21.15.172:6789/0 conn(0x7ffb4815da40 legacy=0x7ffb4806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb480ce260 con 0x7ffb480cc590 2023-10-16T11:21:36.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ffb400019f0 con 0x7ffb480cc590 2023-10-16T11:21:36.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 >> v1:172.21.15.158:6789/0 conn(0x7ffb480cc590 legacy=0x7ffb480afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb30003a30 0x7ffb30003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb30004850 0x7ffb30004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 0x7ffb30005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4815de20 con 0x7ffb30003a30 2023-10-16T11:21:36.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3000baf0 con 0x7ffb30004850 2023-10-16T11:21:36.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3000bcb0 con 0x7ffb4806afe0 2023-10-16T11:21:36.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb47fff700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb30003a30 0x7ffb30003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb477fe700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb30004850 0x7ffb30004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb4ccbf700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 0x7ffb30005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb47fff700 1 -- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 msgr2=0x7ffb30005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:36.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb47fff700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb4806afe0 0x7ffb30005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb47fff700 1 -- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb30004850 msgr2=0x7ffb30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:36.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb47fff700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb30004850 0x7ffb30004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb47fff700 1 -- 172.21.15.158:0/2982301533 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300035b0 con 0x7ffb30003a30 2023-10-16T11:21:36.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.693+0000 7ffb47fff700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb30003a30 0x7ffb30003e50 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c001800 tx=0x7ffb30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0589794ef50325 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:36.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c0012f0 con 0x7ffb30003a30 2023-10-16T11:21:36.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c0052f0 con 0x7ffb30003a30 2023-10-16T11:21:36.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb46ffd700 1 -- 172.21.15.158:0/2982301533 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c006860 con 0x7ffb30003a30 2023-10-16T11:21:36.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb4e722700 1 -- 172.21.15.158:0/2982301533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb30003a30 msgr2=0x7ffb30003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb4e722700 1 --2- 172.21.15.158:0/2982301533 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb30003a30 0x7ffb30003e50 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c001800 tx=0x7ffb30005880 comp rx=0 tx=0).stop 2023-10-16T11:21:36.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb4ccbf700 1 -- 172.21.15.158:0/2982301533 reap_dead start 2023-10-16T11:21:36.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb4e722700 1 -- 172.21.15.158:0/2982301533 shutdown_connections 2023-10-16T11:21:36.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb4e722700 1 -- 172.21.15.158:0/2982301533 >> 172.21.15.158:0/2982301533 conn(0x7ffb48099a10 msgr2=0x7ffb480d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:36.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb4e722700 1 -- 172.21.15.158:0/2982301533 shutdown_connections 2023-10-16T11:21:36.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.694+0000 7ffb4e722700 1 -- 172.21.15.158:0/2982301533 wait complete. 2023-10-16T11:21:36.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.696+0000 7ffb4e722700 1 Processor -- start 2023-10-16T11:21:36.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.696+0000 7ffb4e722700 1 -- start start 2023-10-16T11:21:36.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb4e722700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb30003a30 0x7ffb480b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb4e722700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb481f39c0 0x7ffb48159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb4e722700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4806ee10 0x7ffb4806f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb4e722700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4806f7e0 con 0x7ffb481f39c0 2023-10-16T11:21:36.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb4e722700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb481f62a0 con 0x7ffb4806ee10 2023-10-16T11:21:36.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb4e722700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb48159990 con 0x7ffb30003a30 2023-10-16T11:21:36.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb481f39c0 0x7ffb48159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb481f39c0 0x7ffb48159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42800/0 (socket says 172.21.15.158:42800) 2023-10-16T11:21:36.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 -- 172.21.15.158:0/3035571161 learned_addr learned my addr 172.21.15.158:0/3035571161 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:36.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb47fff700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb30003a30 0x7ffb480b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb4ccbf700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4806ee10 0x7ffb4806f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 -- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb30003a30 msgr2=0x7ffb480b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb30003a30 0x7ffb480b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 -- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4806ee10 msgr2=0x7ffb4806f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:36.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4806ee10 0x7ffb4806f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:36.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 -- 172.21.15.158:0/3035571161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb481f64f0 con 0x7ffb481f39c0 2023-10-16T11:21:36.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb477fe700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb481f39c0 0x7ffb48159020 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7ffb380119a0 tx=0x7ffb38011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:36.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.697+0000 7ffb44ff9700 1 -- 172.21.15.158:0/3035571161 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3800f700 con 0x7ffb481f39c0 2023-10-16T11:21:36.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.698+0000 7ffb44ff9700 1 -- 172.21.15.158:0/3035571161 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3800f860 con 0x7ffb481f39c0 2023-10-16T11:21:36.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.698+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb481f67d0 con 0x7ffb481f39c0 2023-10-16T11:21:36.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.698+0000 7ffb44ff9700 1 -- 172.21.15.158:0/3035571161 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3800fbf0 con 0x7ffb481f39c0 2023-10-16T11:21:36.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.698+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb480b0e60 con 0x7ffb481f39c0 2023-10-16T11:21:36.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.698+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb480681b0 con 0x7ffb481f39c0 2023-10-16T11:21:36.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.699+0000 7ffb44ff9700 1 -- 172.21.15.158:0/3035571161 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ffb3800fd50 con 0x7ffb481f39c0 2023-10-16T11:21:36.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.699+0000 7ffb44ff9700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7ffb3406ada0 0x7ffb3406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:36.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.699+0000 7ffb44ff9700 1 -- 172.21.15.158:0/3035571161 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7ffb38089b10 con 0x7ffb481f39c0 2023-10-16T11:21:36.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.699+0000 7ffb47fff700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7ffb3406ada0 0x7ffb3406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:36.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.700+0000 7ffb47fff700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7ffb3406ada0 0x7ffb3406d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c014040 tx=0x7ffb3c009030 comp rx=0 tx=0).ready entity=mgr.13392 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:36.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.704+0000 7ffb44ff9700 1 -- 172.21.15.158:0/3035571161 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb38054a30 con 0x7ffb481f39c0 2023-10-16T11:21:36.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:36.954+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ffb4806c6d0 con 0x7ffb481f39c0 2023-10-16T11:21:37.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.129+0000 7ffb44ff9700 1 -- 172.21.15.158:0/3035571161 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v162) v1 ==== 93+0+0 (secure 0 0 0) 0x7ffb38058050 con 0x7ffb481f39c0 2023-10-16T11:21:37.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.132+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7ffb3406ada0 msgr2=0x7ffb3406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.132+0000 7ffb4e722700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7ffb3406ada0 0x7ffb3406d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c014040 tx=0x7ffb3c009030 comp rx=0 tx=0).stop 2023-10-16T11:21:37.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.132+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb481f39c0 msgr2=0x7ffb48159020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.132+0000 7ffb4e722700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb481f39c0 0x7ffb48159020 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7ffb380119a0 tx=0x7ffb38011cb0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.134+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 shutdown_connections 2023-10-16T11:21:37.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.134+0000 7ffb4e722700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7ffb3406ada0 0x7ffb3406d260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.134+0000 7ffb4e722700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb4806ee10 0x7ffb4806f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.134+0000 7ffb4e722700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb30003a30 0x7ffb480b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.134+0000 7ffb4e722700 1 --2- 172.21.15.158:0/3035571161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb481f39c0 0x7ffb48159020 unknown :-1 s=CLOSED pgs=3180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.134+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 >> 172.21.15.158:0/3035571161 conn(0x7ffb48099a10 msgr2=0x7ffb4806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:37.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.134+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 shutdown_connections 2023-10-16T11:21:37.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.135+0000 7ffb4e722700 1 -- 172.21.15.158:0/3035571161 wait complete. 2023-10-16T11:21:37.152 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:21:37.153 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:21:37.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:21:37.231 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:21:37.231 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:37.232 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:37.250 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:37.250 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:37.283 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:37.281+0000 7f76dfdfc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:21:37.284 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:37.283+0000 7f2f9e404200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:21:37.300 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:37.350 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:37.349+0000 7f2f9e404200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:37.352 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:37.350+0000 7f76dfdfc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:37.423 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:37.421+0000 7f2f9e404200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:21:37.428 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:37.425+0000 7f76dfdfc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:21:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.461+0000 7fc9f4040700 1 Processor -- start 2023-10-16T11:21:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f4040700 1 -- start start 2023-10-16T11:21:37.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f4040700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9ec0cf480 0x7fc9ec161c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f4040700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf860 0x7fc9ec0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f4040700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec15e480 con 0x7fc9ec06d540 2023-10-16T11:21:37.467 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f4040700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ec15e6e0 con 0x7fc9ec0cf480 2023-10-16T11:21:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f4040700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec15e9c0 con 0x7fc9ec0cf860 2023-10-16T11:21:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f15db700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9ec06d540 0x7fc9ec15dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43044/0 (socket says 172.21.15.158:43044) 2023-10-16T11:21:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.462+0000 7fc9f15db700 1 -- 172.21.15.158:0/956053320 learned_addr learned my addr 172.21.15.158:0/956053320 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f25dd700 1 --2- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf860 0x7fc9ec0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f1ddc700 1 --2- 172.21.15.158:0/956053320 >> v2:172.21.15.172:3300/0 conn(0x7fc9ec0cf480 0x7fc9ec161c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f0dda700 1 -- 172.21.15.158:0/956053320 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935211448 0 0) 0x7fc9ec15e480 con 0x7fc9ec06d540 2023-10-16T11:21:37.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f0dda700 1 -- 172.21.15.158:0/956053320 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d80035b0 con 0x7fc9ec06d540 2023-10-16T11:21:37.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f1ddc700 1 -- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf860 msgr2=0x7fc9ec0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:37.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f1ddc700 1 --2- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf860 0x7fc9ec0cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f1ddc700 1 -- 172.21.15.158:0/956053320 >> v1:172.21.15.158:6789/0 conn(0x7fc9ec06d540 legacy=0x7fc9ec15dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f1ddc700 1 -- 172.21.15.158:0/956053320 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec15eca0 con 0x7fc9ec0cf480 2023-10-16T11:21:37.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f25dd700 1 --2- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf860 0x7fc9ec0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:37.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f1ddc700 1 --2- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec0cf480 0x7fc9ec161c00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009870 tx=0x7fc9e8009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d523a194d809a12 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f0dda700 1 -- 172.21.15.158:0/956053320 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8011830 con 0x7fc9ec0cf480 2023-10-16T11:21:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f0dda700 1 -- 172.21.15.158:0/956053320 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e8011990 con 0x7fc9ec0cf480 2023-10-16T11:21:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.463+0000 7fc9f0dda700 1 -- 172.21.15.158:0/956053320 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8011d20 con 0x7fc9ec0cf480 2023-10-16T11:21:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 -- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec0cf480 msgr2=0x7fc9ec161c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 --2- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec0cf480 0x7fc9ec161c00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009870 tx=0x7fc9e8009b80 comp rx=0 tx=0).stop 2023-10-16T11:21:37.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 -- 172.21.15.158:0/956053320 shutdown_connections 2023-10-16T11:21:37.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 --2- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec0cf480 0x7fc9ec161c00 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 --2- 172.21.15.158:0/956053320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf860 0x7fc9ec0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 -- 172.21.15.158:0/956053320 >> 172.21.15.158:0/956053320 conn(0x7fc9ec06a8d0 msgr2=0x7fc9ec06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:37.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 -- 172.21.15.158:0/956053320 shutdown_connections 2023-10-16T11:21:37.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.464+0000 7fc9f4040700 1 -- 172.21.15.158:0/956053320 wait complete. 2023-10-16T11:21:37.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.465+0000 7fc9f4040700 1 Processor -- start 2023-10-16T11:21:37.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.465+0000 7fc9f4040700 1 -- start start 2023-10-16T11:21:37.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f4040700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec06d540 0x7fc9ec15d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f4040700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf480 0x7fc9ec1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f4040700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ec0cf860 0x7fc9ec15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f4040700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec15d0d0 con 0x7fc9ec0cf860 2023-10-16T11:21:37.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f4040700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec15b1e0 con 0x7fc9ec06d540 2023-10-16T11:21:37.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f4040700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec15af00 con 0x7fc9ec0cf480 2023-10-16T11:21:37.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f25dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ec0cf860 0x7fc9ec15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f25dd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ec0cf860 0x7fc9ec15a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42878/0 (socket says 172.21.15.158:42878) 2023-10-16T11:21:37.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f25dd700 1 -- 172.21.15.158:0/2970014050 learned_addr learned my addr 172.21.15.158:0/2970014050 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:37.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f15db700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf480 0x7fc9ec1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.466+0000 7fc9f1ddc700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec06d540 0x7fc9ec15d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f15db700 1 -- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec06d540 msgr2=0x7fc9ec15d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f15db700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec06d540 0x7fc9ec15d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f15db700 1 -- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ec0cf860 msgr2=0x7fc9ec15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f15db700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ec0cf860 0x7fc9ec15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f15db700 1 -- 172.21.15.158:0/2970014050 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e4009a70 con 0x7fc9ec0cf480 2023-10-16T11:21:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f1ddc700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec06d540 0x7fc9ec15d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f25dd700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ec0cf860 0x7fc9ec15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f15db700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf480 0x7fc9ec1602d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00c430 tx=0x7fc9dc00c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9e27fc700 1 -- 172.21.15.158:0/2970014050 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc005d70 con 0x7fc9ec0cf480 2023-10-16T11:21:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9e27fc700 1 -- 172.21.15.158:0/2970014050 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9dc0070d0 con 0x7fc9ec0cf480 2023-10-16T11:21:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e80094a0 con 0x7fc9ec0cf480 2023-10-16T11:21:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9e27fc700 1 -- 172.21.15.158:0/2970014050 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc007460 con 0x7fc9ec0cf480 2023-10-16T11:21:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.467+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec15cd60 con 0x7fc9ec0cf480 2023-10-16T11:21:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.468+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ec15d280 con 0x7fc9ec0cf480 2023-10-16T11:21:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.468+0000 7fc9e27fc700 1 -- 172.21.15.158:0/2970014050 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc9dc0075c0 con 0x7fc9ec0cf480 2023-10-16T11:21:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.469+0000 7fc9e27fc700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 0x7fc9d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.469+0000 7fc9f1ddc700 1 -- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 msgr2=0x7fc9d006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.469+0000 7fc9e27fc700 1 -- 172.21.15.158:0/2970014050 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fc9dc088db0 con 0x7fc9ec0cf480 2023-10-16T11:21:37.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.469+0000 7fc9f1ddc700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 0x7fc9d006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:37.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.473+0000 7fc9e27fc700 1 -- 172.21.15.158:0/2970014050 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9ec15d280 con 0x7fc9ec0cf480 2023-10-16T11:21:37.503 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:37.501+0000 7f2f9e404200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:21:37.512 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:37.509+0000 7f76dfdfc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:21:37.644 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:37.642+0000 7f2f9e404200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:21:37.658 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:37.656+0000 7f76dfdfc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:21:37.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.670+0000 7fc9f1ddc700 1 -- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 msgr2=0x7fc9d006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:37.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.670+0000 7fc9f1ddc700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 0x7fc9d006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:37.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.678+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9ec15d240 con 0x7fc9ec0cf480 2023-10-16T11:21:37.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.679+0000 7fc9e27fc700 1 -- 172.21.15.158:0/2970014050 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+18100 (secure 0 0 0) 0x7fc9dc053880 con 0x7fc9ec0cf480 2023-10-16T11:21:37.682 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:37.682 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":258,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:34.118784+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:18.736213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:19.205576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/595141618":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1384560093":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/385046482":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6833/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6832/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/4272085425":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1184494648":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/2593348127":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1448525844":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/3641755142":"2023-10-17T11:21:34.118742+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:37.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 msgr2=0x7fc9d006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:37.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 0x7fc9d006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf480 msgr2=0x7fc9ec1602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf480 0x7fc9ec1602d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00c430 tx=0x7fc9dc00c740 comp rx=0 tx=0).stop 2023-10-16T11:21:37.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 shutdown_connections 2023-10-16T11:21:37.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9ec0cf860 0x7fc9ec15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9d006b020 0x7fc9d006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9ec06d540 0x7fc9ec15d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 --2- 172.21.15.158:0/2970014050 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9ec0cf480 0x7fc9ec1602d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 >> 172.21.15.158:0/2970014050 conn(0x7fc9ec06a8d0 msgr2=0x7fc9ec0cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:37.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 shutdown_connections 2023-10-16T11:21:37.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.681+0000 7fc9f4040700 1 -- 172.21.15.158:0/2970014050 wait complete. 2023-10-16T11:21:37.697 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:21:18.736213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:21:19.205576+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '257', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:21:37.698 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:21:37.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.862+0000 7fc9e2b9e700 1 Processor -- start 2023-10-16T11:21:37.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.862+0000 7fc9e2b9e700 1 -- start start 2023-10-16T11:21:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e2b9e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9dc0d2030 0x7fc9dc0cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e2b9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0cfe00 0x7fc9dc0b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e2b9e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0cd010 con 0x7fc9dc0b27e0 2023-10-16T11:21:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e2b9e700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9dc0cd250 con 0x7fc9dc0d2030 2023-10-16T11:21:37.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e2b9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0d0230 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e139b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9dc0d2030 0x7fc9dc0cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e139b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9dc0d2030 0x7fc9dc0cf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42920/0 (socket says 172.21.15.158:42920) 2023-10-16T11:21:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.863+0000 7fc9e139b700 1 -- 172.21.15.158:0/237476066 learned_addr learned my addr 172.21.15.158:0/237476066 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e239d700 1 --2- 172.21.15.158:0/237476066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0cfe00 0x7fc9dc0b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e139b700 1 -- 172.21.15.158:0/237476066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0cfe00 msgr2=0x7fc9dc0b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e139b700 1 --2- 172.21.15.158:0/237476066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0cfe00 0x7fc9dc0b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e139b700 1 -- 172.21.15.158:0/237476066 >> v1:172.21.15.172:6789/0 conn(0x7fc9dc0b27e0 legacy=0x7fc9dc0cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e139b700 1 -- 172.21.15.158:0/237476066 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc0b2bc0 con 0x7fc9dc0d2030 2023-10-16T11:21:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e139b700 1 --2- 172.21.15.158:0/237476066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc0cf8c0 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0098f0 tx=0x7fc9cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6f26eb6d0c85ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:37.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e0b9a700 1 -- 172.21.15.158:0/237476066 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc011830 con 0x7fc9dc0d2030 2023-10-16T11:21:37.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e0b9a700 1 -- 172.21.15.158:0/237476066 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc011990 con 0x7fc9dc0d2030 2023-10-16T11:21:37.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.864+0000 7fc9e0b9a700 1 -- 172.21.15.158:0/237476066 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc011d20 con 0x7fc9dc0d2030 2023-10-16T11:21:37.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.865+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/237476066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 msgr2=0x7fc9dc0cf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.865+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/237476066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc0cf8c0 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0098f0 tx=0x7fc9cc009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:37.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.865+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/237476066 shutdown_connections 2023-10-16T11:21:37.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.865+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/237476066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0cfe00 0x7fc9dc0b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.865+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/237476066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc0cf8c0 unknown :-1 s=CLOSED pgs=3186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.865+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/237476066 >> 172.21.15.158:0/237476066 conn(0x7fc9dc0d12e0 msgr2=0x7fc9dc0d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.866+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/237476066 shutdown_connections 2023-10-16T11:21:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.866+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/237476066 wait complete. 2023-10-16T11:21:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.867+0000 7fc9e2b9e700 1 Processor -- start 2023-10-16T11:21:37.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.867+0000 7fc9e2b9e700 1 -- start start 2023-10-16T11:21:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e2b9e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0b27e0 0x7fc9dc1f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e2b9e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9dc0cfe00 0x7fc9dc1f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e2b9e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc15b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e2b9e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0c6380 con 0x7fc9dc0d2030 2023-10-16T11:21:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e2b9e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0c6f00 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e2b9e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0c65d0 con 0x7fc9dc0b27e0 2023-10-16T11:21:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e239d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc15b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e139b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9dc0cfe00 0x7fc9dc1f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e239d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc15b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42928/0 (socket says 172.21.15.158:42928) 2023-10-16T11:21:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e239d700 1 -- 172.21.15.158:0/1830361168 learned_addr learned my addr 172.21.15.158:0/1830361168 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:37.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e1b9c700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0b27e0 0x7fc9dc1f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e139b700 1 -- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0b27e0 msgr2=0x7fc9dc1f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e139b700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0b27e0 0x7fc9dc1f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e139b700 1 -- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 msgr2=0x7fc9dc15b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e139b700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc15b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:37.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e139b700 1 -- 172.21.15.158:0/1830361168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc009550 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e239d700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc15b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.868+0000 7fc9e139b700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9dc0cfe00 0x7fc9dc1f3a60 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc016000 tx=0x7fc9cc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.869+0000 7fc9d27fc700 1 -- 172.21.15.158:0/1830361168 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc004400 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.869+0000 7fc9d27fc700 1 -- 172.21.15.158:0/1830361168 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc01f070 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.869+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9dc0c68b0 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.869+0000 7fc9d27fc700 1 -- 172.21.15.158:0/1830361168 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc0038e0 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.869+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc15b9e0 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.870+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc04ebb0 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.870+0000 7fc9d27fc700 1 -- 172.21.15.158:0/1830361168 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc9cc021030 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.870+0000 7fc9d27fc700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 0x7fc9c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:37.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.870+0000 7fc9e1b9c700 1 -- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 msgr2=0x7fc9c006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:37.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.871+0000 7fc9d27fc700 1 -- 172.21.15.158:0/1830361168 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fc9cc08af50 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.871+0000 7fc9e1b9c700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 0x7fc9c006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:37.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:37.875+0000 7fc9d27fc700 1 -- 172.21.15.158:0/1830361168 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9cc055b50 con 0x7fc9dc0cfe00 2023-10-16T11:21:37.970 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:37.968+0000 7f2f9e404200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:21:38.003 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:38.000+0000 7f76dfdfc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:21:38.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.072+0000 7fc9e1b9c700 1 -- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 msgr2=0x7fc9c006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:38.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.072+0000 7fc9e1b9c700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 0x7fc9c006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:38.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.075+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc9dc0b4860 con 0x7fc9dc0cfe00 2023-10-16T11:21:38.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.075+0000 7fc9d27fc700 1 -- 172.21.15.158:0/1830361168 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9cc0150a0 con 0x7fc9dc0cfe00 2023-10-16T11:21:38.078 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:21:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 msgr2=0x7fc9c006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 0x7fc9c006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9dc0cfe00 msgr2=0x7fc9dc1f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9dc0cfe00 0x7fc9dc1f3a60 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc016000 tx=0x7fc9cc004750 comp rx=0 tx=0).stop 2023-10-16T11:21:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 shutdown_connections 2023-10-16T11:21:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9dc0d2030 0x7fc9dc15b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9dc0b27e0 0x7fc9dc1f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc9c006b020 0x7fc9c006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 --2- 172.21.15.158:0/1830361168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9dc0cfe00 0x7fc9dc1f3a60 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 >> 172.21.15.158:0/1830361168 conn(0x7fc9dc0d12e0 msgr2=0x7fc9dc06ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:38.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 shutdown_connections 2023-10-16T11:21:38.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.078+0000 7fc9e2b9e700 1 -- 172.21.15.158:0/1830361168 wait complete. 2023-10-16T11:21:38.085 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:38.083+0000 7f2f9e404200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:21:38.095 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:21:38.127 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:38.124+0000 7f76dfdfc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:21:38.159 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:38.157+0000 7f2f9e404200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:21:38.204 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:38.202+0000 7f76dfdfc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:21:38.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.219+0000 7f3b4a376700 1 Processor -- start 2023-10-16T11:21:38.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.219+0000 7f3b4a376700 1 -- start start 2023-10-16T11:21:38.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.220+0000 7f3b4a376700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3b440b3630 0x7f3b44166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.220+0000 7f3b4a376700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b440b4420 con 0x7f3b440d0c60 2023-10-16T11:21:38.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.220+0000 7f3b4a376700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b440cbc20 con 0x7f3b4415b6b0 2023-10-16T11:21:38.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b4a376700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4415ba90 con 0x7f3b440b3630 2023-10-16T11:21:38.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b437fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3b440d0c60 0x7f3b440b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43112/0 (socket says 172.21.15.158:43112) 2023-10-16T11:21:38.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b437fe700 1 -- 172.21.15.158:0/2046431237 learned_addr learned my addr 172.21.15.158:0/2046431237 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:38.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b48913700 1 --2- 172.21.15.158:0/2046431237 >> v2:172.21.15.172:3300/0 conn(0x7f3b440b3630 0x7f3b44166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b42ffd700 1 -- 172.21.15.158:0/2046431237 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774804957 0 0) 0x7f3b440b4420 con 0x7f3b440d0c60 2023-10-16T11:21:38.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b42ffd700 1 -- 172.21.15.158:0/2046431237 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c0035b0 con 0x7f3b440d0c60 2023-10-16T11:21:38.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b42ffd700 1 -- 172.21.15.158:0/2046431237 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971460018 0 0) 0x7f3b2c0035b0 con 0x7f3b440d0c60 2023-10-16T11:21:38.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b48913700 1 -- 172.21.15.158:0/2046431237 >> v1:172.21.15.172:6789/0 conn(0x7f3b4415b6b0 legacy=0x7f3b440cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b48913700 1 -- 172.21.15.158:0/2046431237 >> v1:172.21.15.158:6789/0 conn(0x7f3b440d0c60 legacy=0x7f3b440b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b48913700 1 -- 172.21.15.158:0/2046431237 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b440cbf00 con 0x7f3b440b3630 2023-10-16T11:21:38.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.221+0000 7f3b48913700 1 --2- 172.21.15.158:0/2046431237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440b3630 0x7f3b44166120 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009d90 tx=0x7f3b3c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616be4d55f4ad0be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:38.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.222+0000 7f3b42ffd700 1 -- 172.21.15.158:0/2046431237 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c00e070 con 0x7f3b440b3630 2023-10-16T11:21:38.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.222+0000 7f3b42ffd700 1 -- 172.21.15.158:0/2046431237 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c003a20 con 0x7f3b440b3630 2023-10-16T11:21:38.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.222+0000 7f3b42ffd700 1 -- 172.21.15.158:0/2046431237 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38001670 con 0x7f3b440b3630 2023-10-16T11:21:38.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.222+0000 7f3b4a376700 1 -- 172.21.15.158:0/2046431237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440b3630 msgr2=0x7f3b44166120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.222+0000 7f3b4a376700 1 --2- 172.21.15.158:0/2046431237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440b3630 0x7f3b44166120 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009d90 tx=0x7f3b3c00b330 comp rx=0 tx=0).stop 2023-10-16T11:21:38.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.222+0000 7f3b4a376700 1 -- 172.21.15.158:0/2046431237 shutdown_connections 2023-10-16T11:21:38.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.223+0000 7f3b4a376700 1 --2- 172.21.15.158:0/2046431237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440b3630 0x7f3b44166120 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.223+0000 7f3b4a376700 1 -- 172.21.15.158:0/2046431237 >> 172.21.15.158:0/2046431237 conn(0x7f3b440cdf20 msgr2=0x7f3b440ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:38.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.223+0000 7f3b4a376700 1 -- 172.21.15.158:0/2046431237 shutdown_connections 2023-10-16T11:21:38.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.223+0000 7f3b4a376700 1 -- 172.21.15.158:0/2046431237 wait complete. 2023-10-16T11:21:38.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.224+0000 7f3b4a376700 1 Processor -- start 2023-10-16T11:21:38.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.225+0000 7f3b4a376700 1 -- start start 2023-10-16T11:21:38.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b4a376700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b440b3630 0x7f3b441e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b4a376700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440d0c60 0x7f3b44158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b4a376700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b4415b6b0 0x7f3b440c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b4a376700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b441f0d60 con 0x7f3b440b3630 2023-10-16T11:21:38.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b4a376700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b441f1040 con 0x7f3b440d0c60 2023-10-16T11:21:38.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b4a376700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b441f1320 con 0x7f3b4415b6b0 2023-10-16T11:21:38.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b440b3630 0x7f3b441e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b440b3630 0x7f3b441e3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:42940/0 (socket says 172.21.15.158:42940) 2023-10-16T11:21:38.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 -- 172.21.15.158:0/3494626307 learned_addr learned my addr 172.21.15.158:0/3494626307 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:38.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b48913700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b4415b6b0 0x7f3b440c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b437fe700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440d0c60 0x7f3b44158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 -- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b4415b6b0 msgr2=0x7f3b440c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b4415b6b0 0x7f3b440c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 -- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440d0c60 msgr2=0x7f3b44158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440d0c60 0x7f3b44158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.226+0000 7f3b43fff700 1 -- 172.21.15.158:0/3494626307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c009a70 con 0x7f3b440b3630 2023-10-16T11:21:38.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.227+0000 7f3b43fff700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b440b3630 0x7f3b441e3b30 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400c4e0 tx=0x7f3b3400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:38.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.227+0000 7f3b40ff9700 1 -- 172.21.15.158:0/3494626307 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b340054e0 con 0x7f3b440b3630 2023-10-16T11:21:38.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.227+0000 7f3b40ff9700 1 -- 172.21.15.158:0/3494626307 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b34003680 con 0x7f3b440b3630 2023-10-16T11:21:38.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.227+0000 7f3b40ff9700 1 -- 172.21.15.158:0/3494626307 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b34003a30 con 0x7f3b440b3630 2023-10-16T11:21:38.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.227+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b441f1600 con 0x7f3b440b3630 2023-10-16T11:21:38.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.227+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b44158e40 con 0x7f3b440b3630 2023-10-16T11:21:38.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.228+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b440681b0 con 0x7f3b440b3630 2023-10-16T11:21:38.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.229+0000 7f3b40ff9700 1 -- 172.21.15.158:0/3494626307 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f3b34003b90 con 0x7f3b440b3630 2023-10-16T11:21:38.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.229+0000 7f3b40ff9700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 0x7f3b3006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.229+0000 7f3b437fe700 1 -- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 msgr2=0x7f3b3006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:38.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.229+0000 7f3b40ff9700 1 -- 172.21.15.158:0/3494626307 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f3b34089000 con 0x7f3b440b3630 2023-10-16T11:21:38.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.229+0000 7f3b437fe700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 0x7f3b3006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:38.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.234+0000 7f3b40ff9700 1 -- 172.21.15.158:0/3494626307 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b34053ad0 con 0x7f3b440b3630 2023-10-16T11:21:38.239 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:38.238+0000 7f2f9e404200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:21:38.293 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:38.290+0000 7f76dfdfc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:21:38.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.430+0000 7f3b437fe700 1 -- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 msgr2=0x7f3b3006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:38.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.430+0000 7f3b437fe700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 0x7f3b3006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:38.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.431+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3b440cf910 con 0x7f3b440b3630 2023-10-16T11:21:38.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.432+0000 7f3b40ff9700 1 -- 172.21.15.158:0/3494626307 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3b34017020 con 0x7f3b440b3630 2023-10-16T11:21:38.480 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:21:38.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 msgr2=0x7f3b3006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:38.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 0x7f3b3006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b440b3630 msgr2=0x7f3b441e3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b440b3630 0x7f3b441e3b30 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400c4e0 tx=0x7f3b3400c7f0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 shutdown_connections 2023-10-16T11:21:38.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3b4415b6b0 0x7f3b440c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b440b3630 0x7f3b441e3b30 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f3b3006b020 0x7f3b3006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 --2- 172.21.15.158:0/3494626307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b440d0c60 0x7f3b44158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 >> 172.21.15.158:0/3494626307 conn(0x7f3b440cdf20 msgr2=0x7f3b440b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:38.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 shutdown_connections 2023-10-16T11:21:38.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.434+0000 7f3b4a376700 1 -- 172.21.15.158:0/3494626307 wait complete. 2023-10-16T11:21:38.485 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:21:38.611 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:38.609+0000 7f2f9e404200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:21:38.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.647+0000 7f22ceef3700 1 Processor -- start 2023-10-16T11:21:38.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.647+0000 7f22ceef3700 1 -- start start 2023-10-16T11:21:38.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.647+0000 7f22ceef3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815daa0 0x7f22c80cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.647+0000 7f22ceef3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22c80d0640 0x7f22c80cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22ceef3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c80d1ee0 con 0x7f22c815d6c0 2023-10-16T11:21:38.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22ceef3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c80d40a0 con 0x7f22c80d0640 2023-10-16T11:21:38.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22ceef3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80d12d0 con 0x7f22c815daa0 2023-10-16T11:21:38.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22cd490700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22c80d0640 0x7f22c80cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22ccc8f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f22c815d6c0 0x7f22c80cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43170/0 (socket says 172.21.15.158:43170) 2023-10-16T11:21:38.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22ccc8f700 1 -- 172.21.15.158:0/2858873683 learned_addr learned my addr 172.21.15.158:0/2858873683 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:38.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22c7fff700 1 --2- 172.21.15.158:0/2858873683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815daa0 0x7f22c80cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22cd490700 1 -- 172.21.15.158:0/2858873683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815daa0 msgr2=0x7f22c80cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:38.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22cd490700 1 --2- 172.21.15.158:0/2858873683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815daa0 0x7f22c80cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22cd490700 1 -- 172.21.15.158:0/2858873683 >> v1:172.21.15.158:6789/0 conn(0x7f22c815d6c0 legacy=0x7f22c80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22c77fe700 1 -- 172.21.15.158:0/2858873683 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400233317 0 0) 0x7f22c80d1ee0 con 0x7f22c815d6c0 2023-10-16T11:21:38.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22cd490700 1 -- 172.21.15.158:0/2858873683 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80d15b0 con 0x7f22c80d0640 2023-10-16T11:21:38.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22cd490700 1 --2- 172.21.15.158:0/2858873683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c80d0640 0x7f22c80cf190 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009d90 tx=0x7f22c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fff1fd20d1b67b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:38.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22c77fe700 1 -- 172.21.15.158:0/2858873683 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c000e070 con 0x7f22c80d0640 2023-10-16T11:21:38.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22c77fe700 1 -- 172.21.15.158:0/2858873683 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c000bb20 con 0x7f22c80d0640 2023-10-16T11:21:38.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.648+0000 7f22c77fe700 1 -- 172.21.15.158:0/2858873683 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0012430 con 0x7f22c80d0640 2023-10-16T11:21:38.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 -- 172.21.15.158:0/2858873683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c80d0640 msgr2=0x7f22c80cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2858873683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c80d0640 0x7f22c80cf190 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009d90 tx=0x7f22c000b330 comp rx=0 tx=0).stop 2023-10-16T11:21:38.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 -- 172.21.15.158:0/2858873683 shutdown_connections 2023-10-16T11:21:38.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2858873683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c80d0640 0x7f22c80cf190 unknown :-1 s=CLOSED pgs=3190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2858873683 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815daa0 0x7f22c80cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 -- 172.21.15.158:0/2858873683 >> 172.21.15.158:0/2858873683 conn(0x7f22c809c940 msgr2=0x7f22c80d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:38.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 -- 172.21.15.158:0/2858873683 shutdown_connections 2023-10-16T11:21:38.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.649+0000 7f22ceef3700 1 -- 172.21.15.158:0/2858873683 wait complete. 2023-10-16T11:21:38.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.650+0000 7f22ceef3700 1 Processor -- start 2023-10-16T11:21:38.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.650+0000 7f22ceef3700 1 -- start start 2023-10-16T11:21:38.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22ceef3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22c80d0640 0x7f22c815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22ceef3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815d6c0 0x7f22c815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22ceef3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c815daa0 0x7f22c80b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22ceef3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80cb2b0 con 0x7f22c815daa0 2023-10-16T11:21:38.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22ceef3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80ca6a0 con 0x7f22c80d0640 2023-10-16T11:21:38.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22ceef3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80ca980 con 0x7f22c815d6c0 2023-10-16T11:21:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22cd490700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c815daa0 0x7f22c80b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22cd490700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c815daa0 0x7f22c80b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43004/0 (socket says 172.21.15.158:43004) 2023-10-16T11:21:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22cd490700 1 -- 172.21.15.158:0/2295400255 learned_addr learned my addr 172.21.15.158:0/2295400255 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22c7fff700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815d6c0 0x7f22c815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.651+0000 7f22ccc8f700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22c80d0640 0x7f22c815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22cd490700 1 -- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815d6c0 msgr2=0x7f22c815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22cd490700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815d6c0 0x7f22c815a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22cd490700 1 -- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22c80d0640 msgr2=0x7f22c815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22cd490700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22c80d0640 0x7f22c815aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22cd490700 1 -- 172.21.15.158:0/2295400255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c0009a70 con 0x7f22c815daa0 2023-10-16T11:21:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22c7fff700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815d6c0 0x7f22c815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22ccc8f700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22c80d0640 0x7f22c815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22cd490700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c815daa0 0x7f22c80b21b0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f22c0014040 tx=0x7f22c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22c57fa700 1 -- 172.21.15.158:0/2295400255 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0012430 con 0x7f22c815daa0 2023-10-16T11:21:38.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22c57fa700 1 -- 172.21.15.158:0/2295400255 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c0020070 con 0x7f22c815daa0 2023-10-16T11:21:38.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c80cac60 con 0x7f22c815daa0 2023-10-16T11:21:38.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22c57fa700 1 -- 172.21.15.158:0/2295400255 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0004280 con 0x7f22c815daa0 2023-10-16T11:21:38.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.652+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c815b3f0 con 0x7f22c815daa0 2023-10-16T11:21:38.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.653+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c80d2350 con 0x7f22c815daa0 2023-10-16T11:21:38.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.654+0000 7f22c57fa700 1 -- 172.21.15.158:0/2295400255 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f22c00043e0 con 0x7f22c815daa0 2023-10-16T11:21:38.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.654+0000 7f22c57fa700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 0x7f22ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:38.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.654+0000 7f22ccc8f700 1 -- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 msgr2=0x7f22ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:38.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.654+0000 7f22c57fa700 1 -- 172.21.15.158:0/2295400255 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f22c0022070 con 0x7f22c815daa0 2023-10-16T11:21:38.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.654+0000 7f22ccc8f700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 0x7f22ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:38.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.658+0000 7f22c57fa700 1 -- 172.21.15.158:0/2295400255 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22c80d2350 con 0x7f22c815daa0 2023-10-16T11:21:38.683 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:38.681+0000 7f76dfdfc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:21:38.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.855+0000 7f22ccc8f700 1 -- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 msgr2=0x7f22ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.855+0000 7f22ccc8f700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 0x7f22ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:38.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.861+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f22c80d2350 con 0x7f22c815daa0 2023-10-16T11:21:38.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.861+0000 7f22c57fa700 1 -- 172.21.15.158:0/2295400255 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v258) v1 ==== 100+0+11 (secure 0 0 0) 0x7f22c80d2350 con 0x7f22c815daa0 2023-10-16T11:21:38.864 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:21:38.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.863+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 msgr2=0x7f22ac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:38.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.863+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 0x7f22ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.863+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c815daa0 msgr2=0x7f22c80b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:38.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.863+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c815daa0 0x7f22c80b21b0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f22c0014040 tx=0x7f22c0004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 shutdown_connections 2023-10-16T11:21:38.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c815daa0 0x7f22c80b21b0 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f22ac06b020 0x7f22ac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22c80d0640 0x7f22c815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 --2- 172.21.15.158:0/2295400255 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22c815d6c0 0x7f22c815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:38.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 >> 172.21.15.158:0/2295400255 conn(0x7f22c809c940 msgr2=0x7f22c815c880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:38.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 shutdown_connections 2023-10-16T11:21:38.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:38.864+0000 7f22ceef3700 1 -- 172.21.15.158:0/2295400255 wait complete. 2023-10-16T11:21:38.879 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:21:39.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.000+0000 7f05ac290700 1 Processor -- start 2023-10-16T11:21:39.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.000+0000 7f05ac290700 1 -- start start 2023-10-16T11:21:39.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.001+0000 7f05ac290700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 0x7f05a4161390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.001+0000 7f05ac290700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a41622c0 con 0x7f05a415dab0 2023-10-16T11:21:39.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.001+0000 7f05ac290700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a4161d00 con 0x7f05a406d640 2023-10-16T11:21:39.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.001+0000 7f05ac290700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4161fe0 con 0x7f05a415d6d0 2023-10-16T11:21:39.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a982b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f05a415dab0 0x7f05a406c490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43186/0 (socket says 172.21.15.158:43186) 2023-10-16T11:21:39.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a982b700 1 -- 172.21.15.158:0/2737112126 learned_addr learned my addr 172.21.15.158:0/2737112126 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:39.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05aa02c700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 0x7f05a4161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571866383 0 0) 0x7f05a41622c0 con 0x7f05a415dab0 2023-10-16T11:21:39.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05900035b0 con 0x7f05a415dab0 2023-10-16T11:21:39.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786067022 0 0) 0x7f05900035b0 con 0x7f05a415dab0 2023-10-16T11:21:39.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 msgr2=0x7f05a4161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:39.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 0x7f05a4161390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 >> v1:172.21.15.172:6789/0 conn(0x7f05a406d640 legacy=0x7f05a40cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a406bc90 con 0x7f05a415dab0 2023-10-16T11:21:39.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f0594001670 con 0x7f05a415dab0 2023-10-16T11:21:39.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 >> v1:172.21.15.158:6789/0 conn(0x7f05a415dab0 legacy=0x7f05a406c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590003a30 0x7f0590003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 0x7f0590004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0590004d90 0x7f0590005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4161fe0 con 0x7f0590004d90 2023-10-16T11:21:39.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059000bac0 con 0x7f0590003a30 2023-10-16T11:21:39.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059000bc80 con 0x7f05a415d6d0 2023-10-16T11:21:39.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.002+0000 7f05aa82d700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0590004d90 0x7f0590005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05aa02c700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590003a30 0x7f0590003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05aa82d700 1 -- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 msgr2=0x7f0590004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:39.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05a982b700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 0x7f0590004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05aa82d700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 0x7f0590004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05aa82d700 1 -- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590003a30 msgr2=0x7f0590003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:39.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05aa82d700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590003a30 0x7f0590003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05aa82d700 1 -- 172.21.15.158:0/2737112126 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900035b0 con 0x7f0590004d90 2023-10-16T11:21:39.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05a982b700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a415d6d0 0x7f0590004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:39.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05aa82d700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0590004d90 0x7f0590005210 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f059c00c160 tx=0x7f0590005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddb8c347a36cb8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:39.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c001db0 con 0x7f0590004d90 2023-10-16T11:21:39.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c00cca0 con 0x7f0590004d90 2023-10-16T11:21:39.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.003+0000 7f05a902a700 1 -- 172.21.15.158:0/2737112126 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c003c40 con 0x7f0590004d90 2023-10-16T11:21:39.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.004+0000 7f05ac290700 1 -- 172.21.15.158:0/2737112126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0590004d90 msgr2=0x7f0590005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.004+0000 7f05ac290700 1 --2- 172.21.15.158:0/2737112126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0590004d90 0x7f0590005210 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f059c00c160 tx=0x7f0590005880 comp rx=0 tx=0).stop 2023-10-16T11:21:39.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.004+0000 7f05aa82d700 1 -- 172.21.15.158:0/2737112126 reap_dead start 2023-10-16T11:21:39.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.004+0000 7f05ac290700 1 -- 172.21.15.158:0/2737112126 shutdown_connections 2023-10-16T11:21:39.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.004+0000 7f05ac290700 1 -- 172.21.15.158:0/2737112126 >> 172.21.15.158:0/2737112126 conn(0x7f05a40ce040 msgr2=0x7f05a40ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:39.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.004+0000 7f05ac290700 1 -- 172.21.15.158:0/2737112126 shutdown_connections 2023-10-16T11:21:39.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.004+0000 7f05ac290700 1 -- 172.21.15.158:0/2737112126 wait complete. 2023-10-16T11:21:39.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.005+0000 7f05ac290700 1 Processor -- start 2023-10-16T11:21:39.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.005+0000 7f05ac290700 1 -- start start 2023-10-16T11:21:39.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590004d90 0x7f05a41f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a4154ec0 0x7f05a4157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a4157660 0x7f05a4151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4157ae0 con 0x7f05a4157660 2023-10-16T11:21:39.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a41f62c0 con 0x7f0590004d90 2023-10-16T11:21:39.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4153f00 con 0x7f05a4154ec0 2023-10-16T11:21:39.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a4157660 0x7f05a4151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a4157660 0x7f05a4151170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43024/0 (socket says 172.21.15.158:43024) 2023-10-16T11:21:39.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 -- 172.21.15.158:0/1204577540 learned_addr learned my addr 172.21.15.158:0/1204577540 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:39.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa02c700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590004d90 0x7f05a41f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05a982b700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a4154ec0 0x7f05a4157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 -- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a4154ec0 msgr2=0x7f05a4157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a4154ec0 0x7f05a4157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 -- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590004d90 msgr2=0x7f05a41f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590004d90 0x7f05a41f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 -- 172.21.15.158:0/1204577540 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a4153050 con 0x7f05a4157660 2023-10-16T11:21:39.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05aa82d700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a4157660 0x7f05a4151170 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f05900047b0 tx=0x7f059c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:39.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f059affd700 1 -- 172.21.15.158:0/1204577540 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c003c10 con 0x7f05a4157660 2023-10-16T11:21:39.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f059affd700 1 -- 172.21.15.158:0/1204577540 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c003d90 con 0x7f05a4157660 2023-10-16T11:21:39.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f059affd700 1 -- 172.21.15.158:0/1204577540 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c005310 con 0x7f05a4157660 2023-10-16T11:21:39.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a41532f0 con 0x7f05a4157660 2023-10-16T11:21:39.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.006+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a41538b0 con 0x7f05a4157660 2023-10-16T11:21:39.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.007+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a40681b0 con 0x7f05a4157660 2023-10-16T11:21:39.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.008+0000 7f059affd700 1 -- 172.21.15.158:0/1204577540 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f059c019070 con 0x7f05a4157660 2023-10-16T11:21:39.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.008+0000 7f059affd700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 0x7f058806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.008+0000 7f05aa02c700 1 -- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 msgr2=0x7f058806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:39.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.008+0000 7f059affd700 1 -- 172.21.15.158:0/1204577540 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f059c08a540 con 0x7f05a4157660 2023-10-16T11:21:39.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.008+0000 7f05aa02c700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 0x7f058806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:39.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.017+0000 7f059affd700 1 -- 172.21.15.158:0/1204577540 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f059c055150 con 0x7f05a4157660 2023-10-16T11:21:39.186 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:39.184+0000 7f2f9e404200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:21:39.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.209+0000 7f05aa02c700 1 -- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 msgr2=0x7f058806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:39.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.209+0000 7f05aa02c700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 0x7f058806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:39.256 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:39.255+0000 7f2f9e404200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:21:39.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.262+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f05a40b37a0 con 0x7f05a4157660 2023-10-16T11:21:39.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.262+0000 7f059affd700 1 -- 172.21.15.158:0/1204577540 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v258) v1 ==== 81+0+594 (secure 0 0 0) 0x7f059c058770 con 0x7f05a4157660 2023-10-16T11:21:39.265 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/595141618 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.265 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1384560093 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.266 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/385046482 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.266 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/3125181137 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.266 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/3125181137 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.266 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/4272085425 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.266 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1184494648 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.267 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/2593348127 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.267 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1448525844 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.267 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3641755142 2023-10-17T11:21:34.118742+0000 2023-10-16T11:21:39.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.264+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 msgr2=0x7f058806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:39.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.264+0000 7f05ac290700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 0x7f058806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.264+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a4157660 msgr2=0x7f05a4151170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.264+0000 7f05ac290700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a4157660 0x7f05a4151170 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f05900047b0 tx=0x7f059c0014a0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 shutdown_connections 2023-10-16T11:21:39.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05a4157660 0x7f05a4151170 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f058806b020 0x7f058806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0590004d90 0x7f05a41f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 --2- 172.21.15.158:0/1204577540 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05a4154ec0 0x7f05a4157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 >> 172.21.15.158:0/1204577540 conn(0x7f05a40ce040 msgr2=0x7f05a415f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:39.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 shutdown_connections 2023-10-16T11:21:39.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.265+0000 7f05ac290700 1 -- 172.21.15.158:0/1204577540 wait complete. 2023-10-16T11:21:39.271 INFO:teuthology.orchestra.run.smithi158.stderr:listed 10 entries 2023-10-16T11:21:39.280 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:21:39.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:39.281 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:21:39.289 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:39.286+0000 7f76dfdfc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:21:39.323 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:21:39.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:21:39.324 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:21:39.364 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:39.362+0000 7f76dfdfc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:21:39.380 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:21:39.381 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:39.398 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:21:39.398 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:39.398 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:39.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:21:39.438 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:21:39.439 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:39.456 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:21:39.456 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.599+0000 7f569c9e6700 1 Processor -- start 2023-10-16T11:21:39.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.599+0000 7f569c9e6700 1 -- start start 2023-10-16T11:21:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569c9e6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 0x7f56981683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569c9e6700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56980d3bd0 con 0x7f56980d1210 2023-10-16T11:21:39.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569c9e6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56980d2fc0 con 0x7f569815d7c0 2023-10-16T11:21:39.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569c9e6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56980d4060 con 0x7f56980b7810 2023-10-16T11:21:39.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f5695d9b700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f56980d1210 0x7f56980ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43208/0 (socket says 172.21.15.158:43208) 2023-10-16T11:21:39.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f5695d9b700 1 -- 172.21.15.158:0/84249831 learned_addr learned my addr 172.21.15.158:0/84249831 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:39.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f5696d9d700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 0x7f56981683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625590960 0 0) 0x7f56980d3bd0 con 0x7f56980d1210 2023-10-16T11:21:39.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56800035b0 con 0x7f56980d1210 2023-10-16T11:21:39.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168502892 0 0) 0x7f56980d2fc0 con 0x7f569815d7c0 2023-10-16T11:21:39.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56980d3bd0 con 0x7f569815d7c0 2023-10-16T11:21:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867942145 0 0) 0x7f56800035b0 con 0x7f56980d1210 2023-10-16T11:21:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 msgr2=0x7f56981683f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:39.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 0x7f56981683f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 >> v1:172.21.15.172:6789/0 conn(0x7f569815d7c0 legacy=0x7f56980d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56980d32a0 con 0x7f56980d1210 2023-10-16T11:21:39.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f568c001670 con 0x7f56980d1210 2023-10-16T11:21:39.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f5696d9d700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 0x7f56981683f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:39.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 >> v1:172.21.15.158:6789/0 conn(0x7f56980d1210 legacy=0x7f56980ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.601+0000 7f569559a700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f5680003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5680004850 0x7f5680004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 0x7f5680005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980d4060 con 0x7f5680003bc0 2023-10-16T11:21:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568000ba70 con 0x7f5680004850 2023-10-16T11:21:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568000bc30 con 0x7f56980b7810 2023-10-16T11:21:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569659c700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f5680003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f5695d9b700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5680004850 0x7f5680004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f5696d9d700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 0x7f5680005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569659c700 1 -- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 msgr2=0x7f5680005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:39.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569659c700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56980b7810 0x7f5680005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569659c700 1 -- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5680004850 msgr2=0x7f5680004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:39.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569659c700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5680004850 0x7f5680004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569659c700 1 -- 172.21.15.158:0/84249831 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690009a70 con 0x7f5680003bc0 2023-10-16T11:21:39.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569659c700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f5680003fc0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f5688007fd0 tx=0x7f5680005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1e8bee40730435 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:39.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688001430 con 0x7f5680003bc0 2023-10-16T11:21:39.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56880052d0 con 0x7f5680003bc0 2023-10-16T11:21:39.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.602+0000 7f569559a700 1 -- 172.21.15.158:0/84249831 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688001d40 con 0x7f5680003bc0 2023-10-16T11:21:39.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.603+0000 7f569c9e6700 1 -- 172.21.15.158:0/84249831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 msgr2=0x7f5680003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.603+0000 7f569c9e6700 1 --2- 172.21.15.158:0/84249831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f5680003fc0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f5688007fd0 tx=0x7f5680005880 comp rx=0 tx=0).stop 2023-10-16T11:21:39.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.603+0000 7f5696d9d700 1 -- 172.21.15.158:0/84249831 reap_dead start 2023-10-16T11:21:39.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.603+0000 7f569c9e6700 1 -- 172.21.15.158:0/84249831 shutdown_connections 2023-10-16T11:21:39.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.603+0000 7f569c9e6700 1 -- 172.21.15.158:0/84249831 >> 172.21.15.158:0/84249831 conn(0x7f56980669f0 msgr2=0x7f5698066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:39.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.603+0000 7f569c9e6700 1 -- 172.21.15.158:0/84249831 shutdown_connections 2023-10-16T11:21:39.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.603+0000 7f569c9e6700 1 -- 172.21.15.158:0/84249831 wait complete. 2023-10-16T11:21:39.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.604+0000 7f569c9e6700 1 Processor -- start 2023-10-16T11:21:39.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.604+0000 7f569c9e6700 1 -- start start 2023-10-16T11:21:39.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569c9e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f56981f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569c9e6700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56981f3940 0x7f56980aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569c9e6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56980aa5f0 0x7f56981f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569c9e6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56981f2e10 con 0x7f5680003bc0 2023-10-16T11:21:39.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569c9e6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980aaa70 con 0x7f56980aa5f0 2023-10-16T11:21:39.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569c9e6700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56980c9640 con 0x7f56981f3940 2023-10-16T11:21:39.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f56981f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f56981f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43042/0 (socket says 172.21.15.158:43042) 2023-10-16T11:21:39.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f569659c700 1 -- 172.21.15.158:0/1926688409 learned_addr learned my addr 172.21.15.158:0/1926688409 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:39.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f5695d9b700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56981f3940 0x7f56980aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.605+0000 7f5696d9d700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56980aa5f0 0x7f56981f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:39.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569659c700 1 -- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56981f3940 msgr2=0x7f56980aa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569659c700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56981f3940 0x7f56980aa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569659c700 1 -- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56980aa5f0 msgr2=0x7f56981f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569659c700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56980aa5f0 0x7f56981f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569659c700 1 -- 172.21.15.158:0/1926688409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56980c8a30 con 0x7f5680003bc0 2023-10-16T11:21:39.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f5696d9d700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56980aa5f0 0x7f56981f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:39.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569659c700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f56981f28d0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f568800c3c0 tx=0x7f56880012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:39.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f5686ffd700 1 -- 172.21.15.158:0/1926688409 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688001d40 con 0x7f5680003bc0 2023-10-16T11:21:39.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f5686ffd700 1 -- 172.21.15.158:0/1926688409 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56880013d0 con 0x7f5680003bc0 2023-10-16T11:21:39.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56980c8d10 con 0x7f5680003bc0 2023-10-16T11:21:39.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f5686ffd700 1 -- 172.21.15.158:0/1926688409 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688007280 con 0x7f5680003bc0 2023-10-16T11:21:39.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.606+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56980c92d0 con 0x7f5680003bc0 2023-10-16T11:21:39.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.607+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56980c9920 con 0x7f5680003bc0 2023-10-16T11:21:39.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.607+0000 7f5686ffd700 1 -- 172.21.15.158:0/1926688409 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f56880073e0 con 0x7f5680003bc0 2023-10-16T11:21:39.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.608+0000 7f5686ffd700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 0x7f567806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:39.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.608+0000 7f5695d9b700 1 -- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 msgr2=0x7f567806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:39.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.608+0000 7f5686ffd700 1 -- 172.21.15.158:0/1926688409 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f568808ac30 con 0x7f5680003bc0 2023-10-16T11:21:39.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.608+0000 7f5695d9b700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 0x7f567806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:39.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.612+0000 7f5686ffd700 1 -- 172.21.15.158:0/1926688409 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5688055790 con 0x7f5680003bc0 2023-10-16T11:21:39.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.809+0000 7f5695d9b700 1 -- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 msgr2=0x7f567806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:39.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.809+0000 7f5695d9b700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 0x7f567806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:39.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.859+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56980c9460 con 0x7f5680003bc0 2023-10-16T11:21:39.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.859+0000 7f5686ffd700 1 -- 172.21.15.158:0/1926688409 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f5688017090 con 0x7f5680003bc0 2023-10-16T11:21:39.862 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:39.863 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":145,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1010484950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1010484950},{"type":"v1","addr":"172.21.15.158:6836","nonce":1010484950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":13224,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/54641557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":54641557},{"type":"v1","addr":"172.21.15.158:6837","nonce":54641557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":13239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3388825539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3388825539},{"type":"v1","addr":"172.21.15.172:6835","nonce":3388825539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:20.241439+0000","modified":"2023-10-16T11:21:23.275699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39845":{"gid":39845,"name":"d","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3986459990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3986459990},{"type":"v1","addr":"172.21.15.172:6834","nonce":3986459990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T11:21:39.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.861+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 msgr2=0x7f567806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:39.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.861+0000 7f569c9e6700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 0x7f567806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.861+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 msgr2=0x7f56981f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:39.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.861+0000 7f569c9e6700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f56981f28d0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f568800c3c0 tx=0x7f56880012d0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 shutdown_connections 2023-10-16T11:21:39.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f567806b020 0x7f567806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56980aa5f0 0x7f56981f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5680003bc0 0x7f56981f28d0 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 --2- 172.21.15.158:0/1926688409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f56981f3940 0x7f56980aa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:39.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 >> 172.21.15.158:0/1926688409 conn(0x7f56980669f0 msgr2=0x7f56980ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:39.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 shutdown_connections 2023-10-16T11:21:39.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:39.862+0000 7f569c9e6700 1 -- 172.21.15.158:0/1926688409 wait complete. 2023-10-16T11:21:39.867 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 145 2023-10-16T11:21:39.878 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:39.905 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:39.904+0000 7f2f9e404200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:39.984 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:39.982+0000 7f2f9e404200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:21:40.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.002+0000 7feb9fae1700 1 Processor -- start 2023-10-16T11:21:40.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.002+0000 7feb9fae1700 1 -- start start 2023-10-16T11:21:40.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.003+0000 7feb9fae1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb9815dde0 0x7feb980cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.003+0000 7feb9fae1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb981613b0 con 0x7feb9815d2f0 2023-10-16T11:21:40.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.003+0000 7feb9fae1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb98161690 con 0x7feb9815ee50 2023-10-16T11:21:40.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.003+0000 7feb9fae1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb98161970 con 0x7feb9815dde0 2023-10-16T11:21:40.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.003+0000 7feb9d87d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7feb9815d2f0 0x7feb9815d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43228/0 (socket says 172.21.15.158:43228) 2023-10-16T11:21:40.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.003+0000 7feb9d87d700 1 -- 172.21.15.158:0/2337483518 learned_addr learned my addr 172.21.15.158:0/2337483518 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:40.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.003+0000 7feb9d07c700 1 --2- 172.21.15.158:0/2337483518 >> v2:172.21.15.172:3300/0 conn(0x7feb9815dde0 0x7feb980cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9c87b700 1 -- 172.21.15.158:0/2337483518 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423830746 0 0) 0x7feb981613b0 con 0x7feb9815d2f0 2023-10-16T11:21:40.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9c87b700 1 -- 172.21.15.158:0/2337483518 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb840035b0 con 0x7feb9815d2f0 2023-10-16T11:21:40.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9d07c700 1 -- 172.21.15.158:0/2337483518 >> v1:172.21.15.172:6789/0 conn(0x7feb9815ee50 legacy=0x7feb981602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9c87b700 1 -- 172.21.15.158:0/2337483518 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467552319 0 0) 0x7feb98161690 con 0x7feb9815ee50 2023-10-16T11:21:40.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9d07c700 1 -- 172.21.15.158:0/2337483518 >> v1:172.21.15.158:6789/0 conn(0x7feb9815d2f0 legacy=0x7feb9815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9d07c700 1 -- 172.21.15.158:0/2337483518 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98161c50 con 0x7feb9815dde0 2023-10-16T11:21:40.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9d07c700 1 --2- 172.21.15.158:0/2337483518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb9815dde0 0x7feb980cf020 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7feb88009840 tx=0x7feb88009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f774fe4c65ffb2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:40.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9c87b700 1 -- 172.21.15.158:0/2337483518 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88011830 con 0x7feb9815dde0 2023-10-16T11:21:40.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9c87b700 1 -- 172.21.15.158:0/2337483518 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb88011990 con 0x7feb9815dde0 2023-10-16T11:21:40.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.004+0000 7feb9c87b700 1 -- 172.21.15.158:0/2337483518 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88011d20 con 0x7feb9815dde0 2023-10-16T11:21:40.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.005+0000 7feb9fae1700 1 -- 172.21.15.158:0/2337483518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb9815dde0 msgr2=0x7feb980cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.005+0000 7feb9fae1700 1 --2- 172.21.15.158:0/2337483518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb9815dde0 0x7feb980cf020 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7feb88009840 tx=0x7feb88009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:40.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.005+0000 7feb9fae1700 1 -- 172.21.15.158:0/2337483518 shutdown_connections 2023-10-16T11:21:40.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.005+0000 7feb9fae1700 1 --2- 172.21.15.158:0/2337483518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb9815dde0 0x7feb980cf020 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.005+0000 7feb9fae1700 1 -- 172.21.15.158:0/2337483518 >> 172.21.15.158:0/2337483518 conn(0x7feb9806a8d0 msgr2=0x7feb9806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:40.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.005+0000 7feb9fae1700 1 -- 172.21.15.158:0/2337483518 shutdown_connections 2023-10-16T11:21:40.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.005+0000 7feb9fae1700 1 -- 172.21.15.158:0/2337483518 wait complete. 2023-10-16T11:21:40.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.006+0000 7feb9fae1700 1 Processor -- start 2023-10-16T11:21:40.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.006+0000 7feb9fae1700 1 -- start start 2023-10-16T11:21:40.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9fae1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb9815d2f0 0x7feb981f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9fae1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb9815ee50 0x7feb980c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9fae1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb98157650 0x7feb9815f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9fae1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98157f10 con 0x7feb9815d2f0 2023-10-16T11:21:40.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9fae1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb980c7730 con 0x7feb98157650 2023-10-16T11:21:40.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9fae1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb980c7a10 con 0x7feb9815ee50 2023-10-16T11:21:40.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb9815d2f0 0x7feb981f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb9815d2f0 0x7feb981f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43058/0 (socket says 172.21.15.158:43058) 2023-10-16T11:21:40.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 -- 172.21.15.158:0/1024679949 learned_addr learned my addr 172.21.15.158:0/1024679949 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:40.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9e07e700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb98157650 0x7feb9815f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d07c700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb9815ee50 0x7feb980c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 -- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb9815ee50 msgr2=0x7feb980c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb9815ee50 0x7feb980c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 -- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb98157650 msgr2=0x7feb9815f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb98157650 0x7feb9815f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 -- 172.21.15.158:0/1024679949 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880094a0 con 0x7feb9815d2f0 2023-10-16T11:21:40.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9e07e700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb98157650 0x7feb9815f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:40.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9d87d700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb9815d2f0 0x7feb981f33f0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c420 tx=0x7feb9400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:40.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb8e7fc700 1 -- 172.21.15.158:0/1024679949 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94005d70 con 0x7feb9815d2f0 2023-10-16T11:21:40.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb8e7fc700 1 -- 172.21.15.158:0/1024679949 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9400ce30 con 0x7feb9815d2f0 2023-10-16T11:21:40.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.007+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb980c7cf0 con 0x7feb9815d2f0 2023-10-16T11:21:40.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.008+0000 7feb8e7fc700 1 -- 172.21.15.158:0/1024679949 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940072d0 con 0x7feb9815d2f0 2023-10-16T11:21:40.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.008+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb980c5fe0 con 0x7feb9815d2f0 2023-10-16T11:21:40.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.009+0000 7feb8e7fc700 1 -- 172.21.15.158:0/1024679949 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7feb94007430 con 0x7feb9815d2f0 2023-10-16T11:21:40.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.009+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb980c6170 con 0x7feb9815d2f0 2023-10-16T11:21:40.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.009+0000 7feb8e7fc700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 0x7feb7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.010+0000 7feb9d07c700 1 -- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 msgr2=0x7feb7c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:40.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.010+0000 7feb8e7fc700 1 -- 172.21.15.158:0/1024679949 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7feb94088ec0 con 0x7feb9815d2f0 2023-10-16T11:21:40.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.010+0000 7feb9d07c700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 0x7feb7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:40.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.014+0000 7feb8e7fc700 1 -- 172.21.15.158:0/1024679949 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb980c6170 con 0x7feb9815d2f0 2023-10-16T11:21:40.044 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:40.042+0000 7f76dfdfc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:40.125 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:40.122+0000 7f76dfdfc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:21:40.131 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:40.130+0000 7f2f9e404200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:40.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.211+0000 7feb9d07c700 1 -- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 msgr2=0x7feb7c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:40.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.211+0000 7feb9d07c700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 0x7feb7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:40.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.217+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb980c7870 con 0x7feb9815d2f0 2023-10-16T11:21:40.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.218+0000 7feb8e7fc700 1 -- 172.21.15.158:0/1024679949 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+18100 (secure 0 0 0) 0x7feb980c7870 con 0x7feb9815d2f0 2023-10-16T11:21:40.221 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:40.222 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":258,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:34.118784+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:18.736213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:19.205576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/595141618":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1384560093":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/385046482":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6833/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6832/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/4272085425":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1184494648":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/2593348127":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1448525844":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/3641755142":"2023-10-17T11:21:34.118742+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:40.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 msgr2=0x7feb7c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:40.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 0x7feb7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb9815d2f0 msgr2=0x7feb981f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb9815d2f0 0x7feb981f33f0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c420 tx=0x7feb9400c730 comp rx=0 tx=0).stop 2023-10-16T11:21:40.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 shutdown_connections 2023-10-16T11:21:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7feb7c06b020 0x7feb7c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb98157650 0x7feb9815f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb9815d2f0 0x7feb981f33f0 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 --2- 172.21.15.158:0/1024679949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb9815ee50 0x7feb980c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.220+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 >> 172.21.15.158:0/1024679949 conn(0x7feb9806a8d0 msgr2=0x7feb9806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.221+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 shutdown_connections 2023-10-16T11:21:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.221+0000 7feb9fae1700 1 -- 172.21.15.158:0/1024679949 wait complete. 2023-10-16T11:21:40.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:21:18.736213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:21:19.205576+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '257', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:21:40.237 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:21:40.280 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:40.278+0000 7f76dfdfc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:21:40.362 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:40.361+0000 7f2f9e404200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:21:40.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.361+0000 7f2865e9a700 1 Processor -- start 2023-10-16T11:21:40.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f2865e9a700 1 -- start start 2023-10-16T11:21:40.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f2865e9a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28600d29b0 0x7f28600d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f2865e9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28600d0a70 0x7f28600b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f2865e9a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600b7c90 con 0x7f2860160d30 2023-10-16T11:21:40.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f2865e9a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28600b8910 con 0x7f28600d29b0 2023-10-16T11:21:40.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f2865e9a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28600d0f00 con 0x7f28600d0a70 2023-10-16T11:21:40.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f285effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28600d29b0 0x7f28600d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.362+0000 7f285f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2860160d30 0x7f28600b8f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43288/0 (socket says 172.21.15.158:43288) 2023-10-16T11:21:40.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f28600d29b0 0x7f28600d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43114/0 (socket says 172.21.15.158:43114) 2023-10-16T11:21:40.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285f7fe700 1 -- 172.21.15.158:0/3411960179 learned_addr learned my addr 172.21.15.158:0/3411960179 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:40.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285ffff700 1 --2- 172.21.15.158:0/3411960179 >> v2:172.21.15.172:3300/0 conn(0x7f28600d0a70 0x7f28600b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285effd700 1 -- 172.21.15.158:0/3411960179 >> v2:172.21.15.172:3300/0 conn(0x7f28600d0a70 msgr2=0x7f28600b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:40.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285effd700 1 --2- 172.21.15.158:0/3411960179 >> v2:172.21.15.172:3300/0 conn(0x7f28600d0a70 0x7f28600b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285effd700 1 -- 172.21.15.158:0/3411960179 >> v1:172.21.15.158:6789/0 conn(0x7f2860160d30 legacy=0x7f28600b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285e7fc700 1 -- 172.21.15.158:0/3411960179 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566289594 0 0) 0x7f28600b7c90 con 0x7f2860160d30 2023-10-16T11:21:40.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285effd700 1 -- 172.21.15.158:0/3411960179 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600b7f70 con 0x7f28600d29b0 2023-10-16T11:21:40.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285effd700 1 --2- 172.21.15.158:0/3411960179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28600d29b0 0x7f28600d0530 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f2854009840 tx=0x7f2854009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52553f116beb8a9c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:40.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285e7fc700 1 -- 172.21.15.158:0/3411960179 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854011830 con 0x7f28600d29b0 2023-10-16T11:21:40.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285e7fc700 1 -- 172.21.15.158:0/3411960179 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2854011990 con 0x7f28600d29b0 2023-10-16T11:21:40.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.363+0000 7f285e7fc700 1 -- 172.21.15.158:0/3411960179 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854011d20 con 0x7f28600d29b0 2023-10-16T11:21:40.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 -- 172.21.15.158:0/3411960179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28600d29b0 msgr2=0x7f28600d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 --2- 172.21.15.158:0/3411960179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28600d29b0 0x7f28600d0530 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f2854009840 tx=0x7f2854009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:40.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 -- 172.21.15.158:0/3411960179 shutdown_connections 2023-10-16T11:21:40.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 --2- 172.21.15.158:0/3411960179 >> v2:172.21.15.172:3300/0 conn(0x7f28600d0a70 0x7f28600b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 --2- 172.21.15.158:0/3411960179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28600d29b0 0x7f28600d0530 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 -- 172.21.15.158:0/3411960179 >> 172.21.15.158:0/3411960179 conn(0x7f28600669f0 msgr2=0x7f2860066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:40.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 -- 172.21.15.158:0/3411960179 shutdown_connections 2023-10-16T11:21:40.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.364+0000 7f2865e9a700 1 -- 172.21.15.158:0/3411960179 wait complete. 2023-10-16T11:21:40.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.365+0000 7f2865e9a700 1 Processor -- start 2023-10-16T11:21:40.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.365+0000 7f2865e9a700 1 -- start start 2023-10-16T11:21:40.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f2865e9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28600d0a70 0x7f28601f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f2865e9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28600d29b0 0x7f28600ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f2865e9a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2860160d30 0x7f28601f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f2865e9a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28600c7cc0 con 0x7f2860160d30 2023-10-16T11:21:40.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f2865e9a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601f71d0 con 0x7f28600d29b0 2023-10-16T11:21:40.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f2865e9a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28600c7ed0 con 0x7f28600d0a70 2023-10-16T11:21:40.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2860160d30 0x7f28601f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2860160d30 0x7f28601f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43122/0 (socket says 172.21.15.158:43122) 2023-10-16T11:21:40.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 -- 172.21.15.158:0/2749279635 learned_addr learned my addr 172.21.15.158:0/2749279635 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:40.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285f7fe700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28600d0a70 0x7f28601f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285effd700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28600d29b0 0x7f28600ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 -- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28600d0a70 msgr2=0x7f28601f6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28600d0a70 0x7f28601f6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 -- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28600d29b0 msgr2=0x7f28600ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28600d29b0 0x7f28600ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.366+0000 7f285ffff700 1 -- 172.21.15.158:0/2749279635 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540094a0 con 0x7f2860160d30 2023-10-16T11:21:40.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.367+0000 7f285ffff700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2860160d30 0x7f28601f21b0 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f285800efc0 tx=0x7f285800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:40.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.367+0000 7f2847fff700 1 -- 172.21.15.158:0/2749279635 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858009ea0 con 0x7f2860160d30 2023-10-16T11:21:40.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.367+0000 7f2847fff700 1 -- 172.21.15.158:0/2749279635 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285800f040 con 0x7f2860160d30 2023-10-16T11:21:40.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.367+0000 7f2847fff700 1 -- 172.21.15.158:0/2749279635 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800ced0 con 0x7f2860160d30 2023-10-16T11:21:40.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.367+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28600c81b0 con 0x7f2860160d30 2023-10-16T11:21:40.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.367+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28600c8770 con 0x7f2860160d30 2023-10-16T11:21:40.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.368+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2860159c90 con 0x7f2860160d30 2023-10-16T11:21:40.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.368+0000 7f2847fff700 1 -- 172.21.15.158:0/2749279635 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2858014480 con 0x7f2860160d30 2023-10-16T11:21:40.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.369+0000 7f2847fff700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 0x7f284c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.369+0000 7f285f7fe700 1 -- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 msgr2=0x7f284c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:40.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.369+0000 7f2847fff700 1 -- 172.21.15.158:0/2749279635 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f285808a510 con 0x7f2860160d30 2023-10-16T11:21:40.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.369+0000 7f285f7fe700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 0x7f284c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:40.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.373+0000 7f2847fff700 1 -- 172.21.15.158:0/2749279635 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2858054f60 con 0x7f2860160d30 2023-10-16T11:21:40.439 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:40.437+0000 7f2f9e404200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:21:40.523 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:40.521+0000 7f76dfdfc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:21:40.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.570+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28600681b0 con 0x7f2860160d30 2023-10-16T11:21:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.570+0000 7f285f7fe700 1 -- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 msgr2=0x7f284c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.570+0000 7f285f7fe700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 0x7f284c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:40.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.570+0000 7f2847fff700 1 -- 172.21.15.158:0/2749279635 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2858058580 con 0x7f2860160d30 2023-10-16T11:21:40.574 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:21:40.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 msgr2=0x7f284c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 0x7f284c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2860160d30 msgr2=0x7f28601f21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2860160d30 0x7f28601f21b0 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f285800efc0 tx=0x7f285800c3a0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 shutdown_connections 2023-10-16T11:21:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2860160d30 0x7f28601f21b0 unknown :-1 s=CLOSED pgs=3202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28600d0a70 0x7f28601f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f284c06b020 0x7f284c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 --2- 172.21.15.158:0/2749279635 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28600d29b0 0x7f28600ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 >> 172.21.15.158:0/2749279635 conn(0x7f28600669f0 msgr2=0x7f28600cc730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.572+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 shutdown_connections 2023-10-16T11:21:40.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.573+0000 7f2865e9a700 1 -- 172.21.15.158:0/2749279635 wait complete. 2023-10-16T11:21:40.588 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:21:40.602 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:40.600+0000 7f76dfdfc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:21:40.704 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:40.702+0000 7f2f9e404200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:21:40.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.715+0000 7fbe06475700 1 Processor -- start 2023-10-16T11:21:40.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.715+0000 7fbe06475700 1 -- start start 2023-10-16T11:21:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbe06475700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 0x7fbe00165770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbe06475700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000d0ae0 con 0x7fbe0015deb0 2023-10-16T11:21:40.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbe06475700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000d12d0 con 0x7fbe0015dad0 2023-10-16T11:21:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbe06475700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000cbdf0 con 0x7fbe00163380 2023-10-16T11:21:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdff7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbe0015deb0 0x7fbe000ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43306/0 (socket says 172.21.15.158:43306) 2023-10-16T11:21:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdff7fe700 1 -- 172.21.15.158:0/2251753201 learned_addr learned my addr 172.21.15.158:0/2251753201 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbe04a12700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 0x7fbe00165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254975906 0 0) 0x7fbe000d0ae0 con 0x7fbe0015deb0 2023-10-16T11:21:40.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde80035b0 con 0x7fbe0015deb0 2023-10-16T11:21:40.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421535648 0 0) 0x7fbe000d12d0 con 0x7fbe0015dad0 2023-10-16T11:21:40.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe000d0ae0 con 0x7fbe0015dad0 2023-10-16T11:21:40.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146326467 0 0) 0x7fbde80035b0 con 0x7fbe0015deb0 2023-10-16T11:21:40.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 msgr2=0x7fbe00165770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:40.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 0x7fbe00165770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 >> v1:172.21.15.172:6789/0 conn(0x7fbe0015dad0 legacy=0x7fbe000cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe000cc280 con 0x7fbe0015deb0 2023-10-16T11:21:40.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fbdf0001670 con 0x7fbe0015deb0 2023-10-16T11:21:40.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.716+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 >> v1:172.21.15.158:6789/0 conn(0x7fbe0015deb0 legacy=0x7fbe000ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 0x7fbde8003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde8004120 0x7fbde8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde8004d90 0x7fbde8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000cbdf0 con 0x7fbde8004120 2023-10-16T11:21:40.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde800bba0 con 0x7fbde8004d90 2023-10-16T11:21:40.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde800bd30 con 0x7fbe00163380 2023-10-16T11:21:40.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdff7fe700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde8004120 0x7fbde8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfffff700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 0x7fbde8003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbe04a12700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde8004d90 0x7fbde8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdff7fe700 1 -- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 msgr2=0x7fbde8003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:40.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdff7fe700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe00163380 0x7fbde8003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdff7fe700 1 -- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde8004d90 msgr2=0x7fbde8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:40.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdff7fe700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbde8004d90 0x7fbde8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdff7fe700 1 -- 172.21.15.158:0/2251753201 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80035b0 con 0x7fbde8004120 2023-10-16T11:21:40.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdff7fe700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde8004120 0x7fbde8004850 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000c210 tx=0x7fbde8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e251bf23409995c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:40.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf000cd50 con 0x7fbde8004120 2023-10-16T11:21:40.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf0005960 con 0x7fbde8004120 2023-10-16T11:21:40.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.717+0000 7fbdfeffd700 1 -- 172.21.15.158:0/2251753201 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0005d10 con 0x7fbde8004120 2023-10-16T11:21:40.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.718+0000 7fbe06475700 1 -- 172.21.15.158:0/2251753201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde8004120 msgr2=0x7fbde8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.718+0000 7fbe06475700 1 --2- 172.21.15.158:0/2251753201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbde8004120 0x7fbde8004850 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000c210 tx=0x7fbde8005880 comp rx=0 tx=0).stop 2023-10-16T11:21:40.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.718+0000 7fbe04a12700 1 -- 172.21.15.158:0/2251753201 reap_dead start 2023-10-16T11:21:40.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.718+0000 7fbe06475700 1 -- 172.21.15.158:0/2251753201 shutdown_connections 2023-10-16T11:21:40.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.718+0000 7fbe06475700 1 -- 172.21.15.158:0/2251753201 >> 172.21.15.158:0/2251753201 conn(0x7fbe00099a10 msgr2=0x7fbe000d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:40.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.718+0000 7fbe06475700 1 -- 172.21.15.158:0/2251753201 shutdown_connections 2023-10-16T11:21:40.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.718+0000 7fbe06475700 1 -- 172.21.15.158:0/2251753201 wait complete. 2023-10-16T11:21:40.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.719+0000 7fbe06475700 1 Processor -- start 2023-10-16T11:21:40.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbe06475700 1 -- start start 2023-10-16T11:21:40.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbe06475700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe001f6db0 0x7fbe001549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbe06475700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe000ce640 0x7fbe00155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbe06475700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe000b33e0 0x7fbe000b3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbe06475700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe001550f0 con 0x7fbe001f6db0 2023-10-16T11:21:40.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbe06475700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0014d170 con 0x7fbe000ce640 2023-10-16T11:21:40.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbe06475700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000ce1b0 con 0x7fbe000b33e0 2023-10-16T11:21:40.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbdfffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe001f6db0 0x7fbe001549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbdfffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe001f6db0 0x7fbe001549f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43140/0 (socket says 172.21.15.158:43140) 2023-10-16T11:21:40.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.720+0000 7fbdfffff700 1 -- 172.21.15.158:0/3508223507 learned_addr learned my addr 172.21.15.158:0/3508223507 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:40.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbe04a12700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe000b33e0 0x7fbe000b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdff7fe700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe000ce640 0x7fbe00155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:40.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfffff700 1 -- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe000b33e0 msgr2=0x7fbe000b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfffff700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe000b33e0 0x7fbe000b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfffff700 1 -- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe000ce640 msgr2=0x7fbe00155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfffff700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe000ce640 0x7fbe00155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfffff700 1 -- 172.21.15.158:0/3508223507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe000b3d60 con 0x7fbe001f6db0 2023-10-16T11:21:40.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfffff700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe001f6db0 0x7fbe001549f0 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400f810 tx=0x7fbdf400fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:40.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfcff9700 1 -- 172.21.15.158:0/3508223507 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40052e0 con 0x7fbe001f6db0 2023-10-16T11:21:40.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfcff9700 1 -- 172.21.15.158:0/3508223507 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf4005440 con 0x7fbe001f6db0 2023-10-16T11:21:40.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe00156200 con 0x7fbe001f6db0 2023-10-16T11:21:40.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbdfcff9700 1 -- 172.21.15.158:0/3508223507 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40038c0 con 0x7fbe001f6db0 2023-10-16T11:21:40.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.721+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe000cd8c0 con 0x7fbe001f6db0 2023-10-16T11:21:40.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.722+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe000681b0 con 0x7fbe001f6db0 2023-10-16T11:21:40.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.723+0000 7fbdfcff9700 1 -- 172.21.15.158:0/3508223507 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fbdf4003a20 con 0x7fbe001f6db0 2023-10-16T11:21:40.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.723+0000 7fbdfcff9700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 0x7fbdec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:40.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.723+0000 7fbdff7fe700 1 -- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 msgr2=0x7fbdec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:40.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.723+0000 7fbdfcff9700 1 -- 172.21.15.158:0/3508223507 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fbdf40899a0 con 0x7fbe001f6db0 2023-10-16T11:21:40.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.723+0000 7fbdff7fe700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 0x7fbdec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:40.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.728+0000 7fbdfcff9700 1 -- 172.21.15.158:0/3508223507 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbdf4054470 con 0x7fbe001f6db0 2023-10-16T11:21:40.888 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:40.885+0000 7f76dfdfc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:21:40.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.924+0000 7fbdff7fe700 1 -- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 msgr2=0x7fbdec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:40.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.924+0000 7fbdff7fe700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 0x7fbdec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:40.928 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:40.927+0000 7f2f9e404200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:21:40.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.939+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe000b4b80 con 0x7fbe001f6db0 2023-10-16T11:21:40.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.939+0000 7fbdfcff9700 1 -- 172.21.15.158:0/3508223507 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbdf4057a90 con 0x7fbe001f6db0 2023-10-16T11:21:40.942 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:21:40.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 msgr2=0x7fbdec06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:40.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 0x7fbdec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe001f6db0 msgr2=0x7fbe001549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:40.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe001f6db0 0x7fbe001549f0 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400f810 tx=0x7fbdf400fb20 comp rx=0 tx=0).stop 2023-10-16T11:21:40.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 shutdown_connections 2023-10-16T11:21:40.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe000b33e0 0x7fbe000b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe001f6db0 0x7fbe001549f0 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fbdec06b020 0x7fbdec06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 --2- 172.21.15.158:0/3508223507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe000ce640 0x7fbe00155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:40.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 >> 172.21.15.158:0/3508223507 conn(0x7fbe00099a10 msgr2=0x7fbe00156e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:40.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 shutdown_connections 2023-10-16T11:21:40.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:40.942+0000 7fbe06475700 1 -- 172.21.15.158:0/3508223507 wait complete. 2023-10-16T11:21:40.957 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:21:41.079 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:41.077+0000 7f2f9e404200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:21:41.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.121+0000 7fc765525700 1 Processor -- start 2023-10-16T11:21:41.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.121+0000 7fc765525700 1 -- start start 2023-10-16T11:21:41.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc765525700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc76015d990 0x7fc76006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc765525700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7600cbaf0 0x7fc7600cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc765525700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76015dd70 con 0x7fc760070300 2023-10-16T11:21:41.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc765525700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7601616b0 con 0x7fc76015d990 2023-10-16T11:21:41.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc765525700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600ced70 con 0x7fc7600cbaf0 2023-10-16T11:21:41.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc76015d990 0x7fc76006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75effd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc76015d990 0x7fc76006d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43150/0 (socket says 172.21.15.158:43150) 2023-10-16T11:21:41.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75effd700 1 -- 172.21.15.158:0/3906757514 learned_addr learned my addr 172.21.15.158:0/3906757514 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:41.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75f7fe700 1 --2- 172.21.15.158:0/3906757514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7600cbaf0 0x7fc7600cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75effd700 1 -- 172.21.15.158:0/3906757514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7600cbaf0 msgr2=0x7fc7600cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:41.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75effd700 1 --2- 172.21.15.158:0/3906757514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7600cbaf0 0x7fc7600cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75effd700 1 -- 172.21.15.158:0/3906757514 >> v1:172.21.15.172:6789/0 conn(0x7fc760070300 legacy=0x7fc760161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.122+0000 7fc75effd700 1 -- 172.21.15.158:0/3906757514 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760070780 con 0x7fc76015d990 2023-10-16T11:21:41.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc75effd700 1 --2- 172.21.15.158:0/3906757514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76015d990 0x7fc76006d5d0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7fc750009920 tx=0x7fc750009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f508c40c5103cc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:41.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc75dffb700 1 -- 172.21.15.158:0/3906757514 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750011830 con 0x7fc76015d990 2023-10-16T11:21:41.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc75dffb700 1 -- 172.21.15.158:0/3906757514 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750011990 con 0x7fc76015d990 2023-10-16T11:21:41.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc75dffb700 1 -- 172.21.15.158:0/3906757514 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750011d20 con 0x7fc76015d990 2023-10-16T11:21:41.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 -- 172.21.15.158:0/3906757514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76015d990 msgr2=0x7fc76006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 --2- 172.21.15.158:0/3906757514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76015d990 0x7fc76006d5d0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7fc750009920 tx=0x7fc750009c30 comp rx=0 tx=0).stop 2023-10-16T11:21:41.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 -- 172.21.15.158:0/3906757514 shutdown_connections 2023-10-16T11:21:41.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 --2- 172.21.15.158:0/3906757514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7600cbaf0 0x7fc7600cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 --2- 172.21.15.158:0/3906757514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc76015d990 0x7fc76006d5d0 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 -- 172.21.15.158:0/3906757514 >> 172.21.15.158:0/3906757514 conn(0x7fc7600cdf20 msgr2=0x7fc7600ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:41.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 -- 172.21.15.158:0/3906757514 shutdown_connections 2023-10-16T11:21:41.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.123+0000 7fc765525700 1 -- 172.21.15.158:0/3906757514 wait complete. 2023-10-16T11:21:41.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 Processor -- start 2023-10-16T11:21:41.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 -- start start 2023-10-16T11:21:41.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc760070300 0x7fc7601e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7600cbaf0 0x7fc76014b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc76015d990 0x7fc7600c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601e67e0 con 0x7fc7600cbaf0 2023-10-16T11:21:41.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601e6a80 con 0x7fc76015d990 2023-10-16T11:21:41.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.125+0000 7fc765525700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601e6d60 con 0x7fc760070300 2023-10-16T11:21:41.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7600cbaf0 0x7fc76014b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7600cbaf0 0x7fc76014b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43158/0 (socket says 172.21.15.158:43158) 2023-10-16T11:21:41.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 -- 172.21.15.158:0/1701190362 learned_addr learned my addr 172.21.15.158:0/1701190362 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:41.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75effd700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc760070300 0x7fc7601e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75f7fe700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc76015d990 0x7fc7600c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 -- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc760070300 msgr2=0x7fc7601e7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc760070300 0x7fc7601e7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 -- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc76015d990 msgr2=0x7fc7600c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc76015d990 0x7fc7600c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 -- 172.21.15.158:0/1701190362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750009550 con 0x7fc7600cbaf0 2023-10-16T11:21:41.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc75e7fc700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7600cbaf0 0x7fc76014b0e0 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c430 tx=0x7fc75400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:41.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc7477fe700 1 -- 172.21.15.158:0/1701190362 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540050d0 con 0x7fc7600cbaf0 2023-10-16T11:21:41.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc7477fe700 1 -- 172.21.15.158:0/1701190362 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc754005250 con 0x7fc7600cbaf0 2023-10-16T11:21:41.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7601e7040 con 0x7fc7600cbaf0 2023-10-16T11:21:41.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc7477fe700 1 -- 172.21.15.158:0/1701190362 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754006860 con 0x7fc7600cbaf0 2023-10-16T11:21:41.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.126+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7601e7600 con 0x7fc7600cbaf0 2023-10-16T11:21:41.139 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:41.127+0000 7f76dfdfc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:21:41.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.127+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7600681b0 con 0x7fc7600cbaf0 2023-10-16T11:21:41.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.128+0000 7fc7477fe700 1 -- 172.21.15.158:0/1701190362 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc754003680 con 0x7fc7600cbaf0 2023-10-16T11:21:41.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.128+0000 7fc7477fe700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.128+0000 7fc75effd700 1 -- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 msgr2=0x7fc74c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:41.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.128+0000 7fc7477fe700 1 -- 172.21.15.158:0/1701190362 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fc754088af0 con 0x7fc7600cbaf0 2023-10-16T11:21:41.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.128+0000 7fc75effd700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:41.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.133+0000 7fc7477fe700 1 -- 172.21.15.158:0/1701190362 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc754053650 con 0x7fc7600cbaf0 2023-10-16T11:21:41.196 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:41.194+0000 7f2f9e404200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:21:41.289 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:41.287+0000 7f76dfdfc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:21:41.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.329+0000 7fc75effd700 1 -- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 msgr2=0x7fc74c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:41.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.329+0000 7fc75effd700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:41.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.334+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc76015dd70 con 0x7fc7600cbaf0 2023-10-16T11:21:41.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.335+0000 7fc7477fe700 1 -- 172.21.15.158:0/1701190362 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v258) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc76015dd70 con 0x7fc7600cbaf0 2023-10-16T11:21:41.338 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:21:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 msgr2=0x7fc74c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7600cbaf0 msgr2=0x7fc76014b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7600cbaf0 0x7fc76014b0e0 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c430 tx=0x7fc75400c740 comp rx=0 tx=0).stop 2023-10-16T11:21:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 shutdown_connections 2023-10-16T11:21:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fc74c06b020 0x7fc74c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc76015d990 0x7fc7600c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc760070300 0x7fc7601e7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 --2- 172.21.15.158:0/1701190362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7600cbaf0 0x7fc76014b0e0 unknown :-1 s=CLOSED pgs=3207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 >> 172.21.15.158:0/1701190362 conn(0x7fc7600cdf20 msgr2=0x7fc76015b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:41.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 shutdown_connections 2023-10-16T11:21:41.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.337+0000 7fc765525700 1 -- 172.21.15.158:0/1701190362 wait complete. 2023-10-16T11:21:41.352 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:41.351+0000 7f2f9e404200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:21:41.354 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:41.412 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:41.410+0000 7f76dfdfc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:21:41.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.474+0000 7fa5d3263700 1 Processor -- start 2023-10-16T11:21:41.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.474+0000 7fa5d3263700 1 -- start start 2023-10-16T11:21:41.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.475+0000 7fa5d3263700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc15ed60 0x7fa5cc0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.475+0000 7fa5d3263700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa5cc0b3480 0x7fa5cc0b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.475+0000 7fa5d3263700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc0b3910 con 0x7fa5cc06a8d0 2023-10-16T11:21:41.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.475+0000 7fa5d3263700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc0b3f30 con 0x7fa5cc0b3480 2023-10-16T11:21:41.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.475+0000 7fa5d3263700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0afb90 con 0x7fa5cc15ed60 2023-10-16T11:21:41.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.475+0000 7fa5d1800700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fa5cc06a8d0 0x7fa5cc06e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43344/0 (socket says 172.21.15.158:43344) 2023-10-16T11:21:41.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.475+0000 7fa5d1800700 1 -- 172.21.15.158:0/1119957904 learned_addr learned my addr 172.21.15.158:0/1119957904 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:41.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5d0fff700 1 --2- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc15ed60 0x7fa5cc0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cbfff700 1 --2- 172.21.15.158:0/1119957904 >> v2:172.21.15.172:3300/0 conn(0x7fa5cc0b3480 0x7fa5cc0b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cb7fe700 1 -- 172.21.15.158:0/1119957904 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897242227 0 0) 0x7fa5c4001490 con 0x7fa5cc06a8d0 2023-10-16T11:21:41.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cb7fe700 1 -- 172.21.15.158:0/1119957904 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5bc0035b0 con 0x7fa5cc06a8d0 2023-10-16T11:21:41.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cbfff700 1 -- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc15ed60 msgr2=0x7fa5cc0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:41.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cbfff700 1 --2- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc15ed60 0x7fa5cc0cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cbfff700 1 -- 172.21.15.158:0/1119957904 >> v1:172.21.15.158:6789/0 conn(0x7fa5cc06a8d0 legacy=0x7fa5cc06e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cbfff700 1 -- 172.21.15.158:0/1119957904 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0b3c00 con 0x7fa5cc0b3480 2023-10-16T11:21:41.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5d0fff700 1 --2- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc15ed60 0x7fa5cc0cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:41.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cbfff700 1 --2- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc0b3480 0x7fa5cc0b43c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009840 tx=0x7fa5b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff3a5ff3a5fd7078 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:41.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cb7fe700 1 -- 172.21.15.158:0/1119957904 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8011830 con 0x7fa5cc0b3480 2023-10-16T11:21:41.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cb7fe700 1 -- 172.21.15.158:0/1119957904 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8011990 con 0x7fa5cc0b3480 2023-10-16T11:21:41.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.476+0000 7fa5cb7fe700 1 -- 172.21.15.158:0/1119957904 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8011d00 con 0x7fa5cc0b3480 2023-10-16T11:21:41.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 -- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc0b3480 msgr2=0x7fa5cc0b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 --2- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc0b3480 0x7fa5cc0b43c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009840 tx=0x7fa5b8009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:41.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 -- 172.21.15.158:0/1119957904 shutdown_connections 2023-10-16T11:21:41.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 --2- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc15ed60 0x7fa5cc0cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 --2- 172.21.15.158:0/1119957904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc0b3480 0x7fa5cc0b43c0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 -- 172.21.15.158:0/1119957904 >> 172.21.15.158:0/1119957904 conn(0x7fa5cc0ccda0 msgr2=0x7fa5cc0cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:41.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 -- 172.21.15.158:0/1119957904 shutdown_connections 2023-10-16T11:21:41.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.477+0000 7fa5d3263700 1 -- 172.21.15.158:0/1119957904 wait complete. 2023-10-16T11:21:41.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.478+0000 7fa5d3263700 1 Processor -- start 2023-10-16T11:21:41.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.478+0000 7fa5d3263700 1 -- start start 2023-10-16T11:21:41.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d3263700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc06a8d0 0x7fa5cc1f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d3263700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc15ed60 0x7fa5cc1f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d3263700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5cc15a330 0x7fa5cc15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d3263700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc15ad20 con 0x7fa5cc15a330 2023-10-16T11:21:41.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d3263700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1f56c0 con 0x7fa5cc15ed60 2023-10-16T11:21:41.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d3263700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc15d480 con 0x7fa5cc06a8d0 2023-10-16T11:21:41.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d1800700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5cc15a330 0x7fa5cc15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d1800700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5cc15a330 0x7fa5cc15a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43174/0 (socket says 172.21.15.158:43174) 2023-10-16T11:21:41.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d1800700 1 -- 172.21.15.158:0/731735449 learned_addr learned my addr 172.21.15.158:0/731735449 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:41.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5cbfff700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc15ed60 0x7fa5cc1f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.479+0000 7fa5d0fff700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc06a8d0 0x7fa5cc1f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5cbfff700 1 -- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc06a8d0 msgr2=0x7fa5cc1f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5cbfff700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc06a8d0 0x7fa5cc1f5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5cbfff700 1 -- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5cc15a330 msgr2=0x7fa5cc15a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5cbfff700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5cc15a330 0x7fa5cc15a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5cbfff700 1 -- 172.21.15.158:0/731735449 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0009430 con 0x7fa5cc15ed60 2023-10-16T11:21:41.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5d1800700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5cc15a330 0x7fa5cc15a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:41.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5cbfff700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc15ed60 0x7fa5cc1f2290 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0cca20 tx=0x7fa5b8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:41.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5c97fa700 1 -- 172.21.15.158:0/731735449 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80044d0 con 0x7fa5cc15ed60 2023-10-16T11:21:41.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5c97fa700 1 -- 172.21.15.158:0/731735449 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b801f070 con 0x7fa5cc15ed60 2023-10-16T11:21:41.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5c97fa700 1 -- 172.21.15.158:0/731735449 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80038c0 con 0x7fa5cc15ed60 2023-10-16T11:21:41.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b80094a0 con 0x7fa5cc15ed60 2023-10-16T11:21:41.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.480+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc15b100 con 0x7fa5cc15ed60 2023-10-16T11:21:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.481+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc0681b0 con 0x7fa5cc15ed60 2023-10-16T11:21:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.482+0000 7fa5c97fa700 1 -- 172.21.15.158:0/731735449 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa5b8003a20 con 0x7fa5cc15ed60 2023-10-16T11:21:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.482+0000 7fa5c97fa700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 0x7fa5b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.482+0000 7fa5d0fff700 1 -- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 msgr2=0x7fa5b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:41.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.482+0000 7fa5c97fa700 1 -- 172.21.15.158:0/731735449 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fa5b8021070 con 0x7fa5cc15ed60 2023-10-16T11:21:41.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.482+0000 7fa5d0fff700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 0x7fa5b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:41.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.487+0000 7fa5c97fa700 1 -- 172.21.15.158:0/731735449 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5b8055db0 con 0x7fa5cc15ed60 2023-10-16T11:21:41.582 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:41.579+0000 7f76dfdfc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:21:41.616 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:41.614+0000 7f2f9e404200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:41.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.683+0000 7fa5d0fff700 1 -- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 msgr2=0x7fa5b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:41.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.683+0000 7fa5d0fff700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 0x7fa5b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:41.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.732+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5cc071a40 con 0x7fa5cc15ed60 2023-10-16T11:21:41.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.733+0000 7fa5c97fa700 1 -- 172.21.15.158:0/731735449 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fa5b80593d0 con 0x7fa5cc15ed60 2023-10-16T11:21:41.736 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:41.736 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":145,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1010484950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1010484950},{"type":"v1","addr":"172.21.15.158:6836","nonce":1010484950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":13224,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/54641557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":54641557},{"type":"v1","addr":"172.21.15.158:6837","nonce":54641557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":13239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3388825539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3388825539},{"type":"v1","addr":"172.21.15.172:6835","nonce":3388825539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:20.241439+0000","modified":"2023-10-16T11:21:23.275699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39845":{"gid":39845,"name":"d","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3986459990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3986459990},{"type":"v1","addr":"172.21.15.172:6834","nonce":3986459990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T11:21:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 msgr2=0x7fa5b006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 0x7fa5b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc15ed60 msgr2=0x7fa5cc1f2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc15ed60 0x7fa5cc1f2290 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0cca20 tx=0x7fa5b8004750 comp rx=0 tx=0).stop 2023-10-16T11:21:41.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 shutdown_connections 2023-10-16T11:21:41.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5cc15a330 0x7fa5cc15a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5cc06a8d0 0x7fa5cc1f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7fa5b006b020 0x7fa5b006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 --2- 172.21.15.158:0/731735449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5cc15ed60 0x7fa5cc1f2290 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 >> 172.21.15.158:0/731735449 conn(0x7fa5cc0ccda0 msgr2=0x7fa5cc06ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:41.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 shutdown_connections 2023-10-16T11:21:41.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.736+0000 7fa5d3263700 1 -- 172.21.15.158:0/731735449 wait complete. 2023-10-16T11:21:41.741 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 145 2023-10-16T11:21:41.751 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:41.779 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:41.777+0000 7f2f9e404200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:21:41.861 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:41.858+0000 7f76dfdfc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:21:41.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.871+0000 7f0ef6e49700 1 Processor -- start 2023-10-16T11:21:41.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.872+0000 7f0ef6e49700 1 -- start start 2023-10-16T11:21:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.872+0000 7f0ef6e49700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef015b800 0x7f0ef00d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.872+0000 7f0ef6e49700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ef00b4290 0x7f0ef00b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.872+0000 7f0ef6e49700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef015f2f0 con 0x7f0ef00d0b20 2023-10-16T11:21:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.872+0000 7f0ef6e49700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef00d1930 con 0x7f0ef00b4290 2023-10-16T11:21:41.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.872+0000 7f0ef6e49700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef015f5d0 con 0x7f0ef015b800 2023-10-16T11:21:41.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ef00b4290 0x7f0ef00b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ef00b4290 0x7f0ef00b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43184/0 (socket says 172.21.15.158:43184) 2023-10-16T11:21:41.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 -- 172.21.15.158:0/15244156 learned_addr learned my addr 172.21.15.158:0/15244156 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef4be5700 1 --2- 172.21.15.158:0/15244156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef015b800 0x7f0ef00d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 -- 172.21.15.158:0/15244156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef015b800 msgr2=0x7f0ef00d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 --2- 172.21.15.158:0/15244156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef015b800 0x7f0ef00d13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 -- 172.21.15.158:0/15244156 >> v1:172.21.15.172:6789/0 conn(0x7f0ef00d0b20 legacy=0x7f0ef015fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 -- 172.21.15.158:0/15244156 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef00b4c60 con 0x7f0ef00b4290 2023-10-16T11:21:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef4be5700 1 --2- 172.21.15.158:0/15244156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef015b800 0x7f0ef00d13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0ef53e6700 1 --2- 172.21.15.158:0/15244156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef00b4290 0x7f0ef00b4720 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009e50 tx=0x7f0ee800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ee47969d83090c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0eef7fe700 1 -- 172.21.15.158:0/15244156 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee800e070 con 0x7f0ef00b4290 2023-10-16T11:21:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0eef7fe700 1 -- 172.21.15.158:0/15244156 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee800b9c0 con 0x7f0ef00b4290 2023-10-16T11:21:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.873+0000 7f0eef7fe700 1 -- 172.21.15.158:0/15244156 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee800bd70 con 0x7f0ef00b4290 2023-10-16T11:21:41.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 -- 172.21.15.158:0/15244156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef00b4290 msgr2=0x7f0ef00b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/15244156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef00b4290 0x7f0ef00b4720 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009e50 tx=0x7f0ee800b330 comp rx=0 tx=0).stop 2023-10-16T11:21:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 -- 172.21.15.158:0/15244156 shutdown_connections 2023-10-16T11:21:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/15244156 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef00b4290 0x7f0ef00b4720 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/15244156 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef015b800 0x7f0ef00d13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 -- 172.21.15.158:0/15244156 >> 172.21.15.158:0/15244156 conn(0x7f0ef006a8d0 msgr2=0x7f0ef006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:41.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 -- 172.21.15.158:0/15244156 shutdown_connections 2023-10-16T11:21:41.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.874+0000 7f0ef6e49700 1 -- 172.21.15.158:0/15244156 wait complete. 2023-10-16T11:21:41.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.876+0000 7f0ef6e49700 1 Processor -- start 2023-10-16T11:21:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.876+0000 7f0ef6e49700 1 -- start start 2023-10-16T11:21:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef6e49700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ef00b4290 0x7f0ef00aad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef6e49700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef00d0b20 0x7f0ef015a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef6e49700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef015b800 0x7f0ef01f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef6e49700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef00a9d40 con 0x7f0ef015b800 2023-10-16T11:21:41.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef6e49700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef01f5400 con 0x7f0ef00b4290 2023-10-16T11:21:41.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef6e49700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef00ab290 con 0x7f0ef00d0b20 2023-10-16T11:21:41.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef53e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef015b800 0x7f0ef01f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef53e6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef015b800 0x7f0ef01f45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43190/0 (socket says 172.21.15.158:43190) 2023-10-16T11:21:41.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef4be5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ef00b4290 0x7f0ef00aad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0ef53e6700 1 -- 172.21.15.158:0/4193662032 learned_addr learned my addr 172.21.15.158:0/4193662032 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:41.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.877+0000 7f0eeffff700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef00d0b20 0x7f0ef015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:41.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef53e6700 1 -- 172.21.15.158:0/4193662032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef00d0b20 msgr2=0x7f0ef015a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef53e6700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef00d0b20 0x7f0ef015a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef53e6700 1 -- 172.21.15.158:0/4193662032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ef00b4290 msgr2=0x7f0ef00aad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef53e6700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ef00b4290 0x7f0ef00aad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef53e6700 1 -- 172.21.15.158:0/4193662032 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8009b30 con 0x7f0ef015b800 2023-10-16T11:21:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef53e6700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef015b800 0x7f0ef01f45b0 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00cedb0 tx=0x7f0ee8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:41.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0eed7fa700 1 -- 172.21.15.158:0/4193662032 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8004840 con 0x7f0ef015b800 2023-10-16T11:21:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0eed7fa700 1 -- 172.21.15.158:0/4193662032 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee80099b0 con 0x7f0ef015b800 2023-10-16T11:21:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef00a9f90 con 0x7f0ef015b800 2023-10-16T11:21:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0eed7fa700 1 -- 172.21.15.158:0/4193662032 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee801c680 con 0x7f0ef015b800 2023-10-16T11:21:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.878+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef00aa550 con 0x7f0ef015b800 2023-10-16T11:21:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.879+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef00681b0 con 0x7f0ef015b800 2023-10-16T11:21:41.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.880+0000 7f0eed7fa700 1 -- 172.21.15.158:0/4193662032 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0ee801c7e0 con 0x7f0ef015b800 2023-10-16T11:21:41.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.880+0000 7f0eed7fa700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 0x7f0ed406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:41.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.880+0000 7f0ef4be5700 1 -- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 msgr2=0x7f0ed406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:41.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.880+0000 7f0eed7fa700 1 -- 172.21.15.158:0/4193662032 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f0ee808b860 con 0x7f0ef015b800 2023-10-16T11:21:41.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.880+0000 7f0ef4be5700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 0x7f0ed406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:41.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:41.885+0000 7f0eed7fa700 1 -- 172.21.15.158:0/4193662032 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ee80562b0 con 0x7f0ef015b800 2023-10-16T11:21:42.034 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.032+0000 7f76dfdfc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:21:42.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.081+0000 7f0ef4be5700 1 -- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 msgr2=0x7f0ed406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:42.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.081+0000 7f0ef4be5700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 0x7f0ed406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:42.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.085+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ef00b3c80 con 0x7f0ef015b800 2023-10-16T11:21:42.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.086+0000 7f0eed7fa700 1 -- 172.21.15.158:0/4193662032 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+18100 (secure 0 0 0) 0x7f0ee80598d0 con 0x7f0ef015b800 2023-10-16T11:21:42.089 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:42.090 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":258,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:34.118784+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:18.736213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:19.205576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/595141618":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1384560093":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/385046482":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6833/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6832/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/4272085425":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1184494648":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/2593348127":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1448525844":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/3641755142":"2023-10-17T11:21:34.118742+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:42.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 msgr2=0x7f0ed406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:42.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 0x7f0ed406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef015b800 msgr2=0x7f0ef01f45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef015b800 0x7f0ef01f45b0 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00cedb0 tx=0x7f0ee8004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 shutdown_connections 2023-10-16T11:21:42.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ef015b800 0x7f0ef01f45b0 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f0ed406b020 0x7f0ed406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ef00b4290 0x7f0ef00aad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 --2- 172.21.15.158:0/4193662032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0ef00d0b20 0x7f0ef015a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 >> 172.21.15.158:0/4193662032 conn(0x7f0ef006a8d0 msgr2=0x7f0ef00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:42.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 shutdown_connections 2023-10-16T11:21:42.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.090+0000 7f0ef6e49700 1 -- 172.21.15.158:0/4193662032 wait complete. 2023-10-16T11:21:42.107 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:21:42.108 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:42.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.227+0000 7f344bfaa700 1 Processor -- start 2023-10-16T11:21:42.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.227+0000 7f344bfaa700 1 -- start start 2023-10-16T11:21:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.228+0000 7f344bfaa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f344406f470 0x7f344406e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.228+0000 7f344bfaa700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f344406e810 0x7f3444072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.228+0000 7f344bfaa700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444072cf0 con 0x7f34440728a0 2023-10-16T11:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.228+0000 7f344bfaa700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344406f850 con 0x7f344406e810 2023-10-16T11:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.228+0000 7f344bfaa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34440b4260 con 0x7f344406f470 2023-10-16T11:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.228+0000 7f3449545700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f344406e810 0x7f3444072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449545700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f344406e810 0x7f3444072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43198/0 (socket says 172.21.15.158:43198) 2023-10-16T11:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449545700 1 -- 172.21.15.158:0/421806324 learned_addr learned my addr 172.21.15.158:0/421806324 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449d46700 1 --2- 172.21.15.158:0/421806324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f344406f470 0x7f344406e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449545700 1 -- 172.21.15.158:0/421806324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f344406f470 msgr2=0x7f344406e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449545700 1 --2- 172.21.15.158:0/421806324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f344406f470 0x7f344406e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449545700 1 -- 172.21.15.158:0/421806324 >> v1:172.21.15.172:6789/0 conn(0x7f34440728a0 legacy=0x7f344406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449545700 1 -- 172.21.15.158:0/421806324 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34440b4540 con 0x7f344406e810 2023-10-16T11:21:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3449545700 1 --2- 172.21.15.158:0/421806324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406e810 0x7f3444072360 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f34340098f0 tx=0x7f3434009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d70276d669adb30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3448d44700 1 -- 172.21.15.158:0/421806324 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434011830 con 0x7f344406e810 2023-10-16T11:21:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3448d44700 1 -- 172.21.15.158:0/421806324 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3434011990 con 0x7f344406e810 2023-10-16T11:21:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.229+0000 7f3448d44700 1 -- 172.21.15.158:0/421806324 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434011d20 con 0x7f344406e810 2023-10-16T11:21:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 -- 172.21.15.158:0/421806324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406e810 msgr2=0x7f3444072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 --2- 172.21.15.158:0/421806324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406e810 0x7f3444072360 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f34340098f0 tx=0x7f3434009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:42.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 -- 172.21.15.158:0/421806324 shutdown_connections 2023-10-16T11:21:42.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 --2- 172.21.15.158:0/421806324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f344406f470 0x7f344406e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 --2- 172.21.15.158:0/421806324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406e810 0x7f3444072360 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 -- 172.21.15.158:0/421806324 >> 172.21.15.158:0/421806324 conn(0x7f34440ccda0 msgr2=0x7f34440cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:42.237 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 -- 172.21.15.158:0/421806324 shutdown_connections 2023-10-16T11:21:42.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.230+0000 7f344bfaa700 1 -- 172.21.15.158:0/421806324 wait complete. 2023-10-16T11:21:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.231+0000 7f344bfaa700 1 Processor -- start 2023-10-16T11:21:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.231+0000 7f344bfaa700 1 -- start start 2023-10-16T11:21:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f344bfaa700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f344406e810 0x7f34440d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f344bfaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406f470 0x7f344415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f344bfaa700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34440728a0 0x7f34440cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f344bfaa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3444160520 con 0x7f344406f470 2023-10-16T11:21:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f344bfaa700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34441e7a50 con 0x7f344406e810 2023-10-16T11:21:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f344bfaa700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34441e6fb0 con 0x7f34440728a0 2023-10-16T11:21:42.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406f470 0x7f344415ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406f470 0x7f344415ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43204/0 (socket says 172.21.15.158:43204) 2023-10-16T11:21:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 -- 172.21.15.158:0/3327471537 learned_addr learned my addr 172.21.15.158:0/3327471537 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f344a547700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34440728a0 0x7f34440cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449d46700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f344406e810 0x7f34440d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 -- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34440728a0 msgr2=0x7f34440cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34440728a0 0x7f34440cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 -- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f344406e810 msgr2=0x7f34440d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f344406e810 0x7f34440d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 -- 172.21.15.158:0/3327471537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434009550 con 0x7f344406f470 2023-10-16T11:21:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f3449545700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406f470 0x7f344415ffe0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f3434000c00 tx=0x7f3434004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.232+0000 7f343a7fc700 1 -- 172.21.15.158:0/3327471537 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434004400 con 0x7f344406f470 2023-10-16T11:21:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.233+0000 7f343a7fc700 1 -- 172.21.15.158:0/3327471537 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343401f070 con 0x7f344406f470 2023-10-16T11:21:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.233+0000 7f344bfaa700 1 -- 172.21.15.158:0/3327471537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34441e7290 con 0x7f344406f470 2023-10-16T11:21:42.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.233+0000 7f343a7fc700 1 -- 172.21.15.158:0/3327471537 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34340038c0 con 0x7f344406f470 2023-10-16T11:21:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.233+0000 7f344bfaa700 1 -- 172.21.15.158:0/3327471537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3444160950 con 0x7f344406f470 2023-10-16T11:21:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.234+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34200052f0 con 0x7f344406f470 2023-10-16T11:21:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.235+0000 7f343a7fc700 1 -- 172.21.15.158:0/3327471537 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f3434021030 con 0x7f344406f470 2023-10-16T11:21:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.235+0000 7f343a7fc700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 0x7f342806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.235+0000 7f3449d46700 1 -- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 msgr2=0x7f342806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:42.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.235+0000 7f343a7fc700 1 -- 172.21.15.158:0/3327471537 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f343408af10 con 0x7f344406f470 2023-10-16T11:21:42.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.235+0000 7f3449d46700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 0x7f342806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:21:42.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.239+0000 7f343a7fc700 1 -- 172.21.15.158:0/3327471537 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34340559f0 con 0x7f344406f470 2023-10-16T11:21:42.313 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:21:42.311+0000 7f2f9e404200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:21:42.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.436+0000 7f3449d46700 1 -- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 msgr2=0x7f342806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/1196176990 2023-10-16T11:21:42.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.436+0000 7f3449d46700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 0x7f342806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:21:42.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.497+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34200055a0 con 0x7f344406f470 2023-10-16T11:21:42.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.498+0000 7f343a7fc700 1 -- 172.21.15.158:0/3327471537 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f3434059010 con 0x7f344406f470 2023-10-16T11:21:42.501 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:42.501 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":145,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1010484950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1010484950},{"type":"v1","addr":"172.21.15.158:6836","nonce":1010484950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":13224,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/54641557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":54641557},{"type":"v1","addr":"172.21.15.158:6837","nonce":54641557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":13239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3388825539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3388825539},{"type":"v1","addr":"172.21.15.172:6835","nonce":3388825539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:20.241439+0000","modified":"2023-10-16T11:21:23.275699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39845":{"gid":39845,"name":"d","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3986459990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3986459990},{"type":"v1","addr":"172.21.15.172:6834","nonce":3986459990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T11:21:42.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 msgr2=0x7f342806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:21:42.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 0x7f342806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406f470 msgr2=0x7f344415ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406f470 0x7f344415ffe0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f3434000c00 tx=0x7f3434004750 comp rx=0 tx=0).stop 2023-10-16T11:21:42.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 shutdown_connections 2023-10-16T11:21:42.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34440728a0 0x7f34440cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:6832/1196176990,v1:172.21.15.158:6833/1196176990] conn(0x7f342806b020 0x7f342806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f344406e810 0x7f34440d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 --2- 172.21.15.158:0/3327471537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f344406f470 0x7f344415ffe0 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 >> 172.21.15.158:0/3327471537 conn(0x7f34440ccda0 msgr2=0x7f344406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:42.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 shutdown_connections 2023-10-16T11:21:42.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.500+0000 7f3427fff700 1 -- 172.21.15.158:0/3327471537 wait complete. 2023-10-16T11:21:42.505 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 145 2023-10-16T11:21:42.515 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:42.615 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.612+0000 7f76dfdfc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:21:42.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.640+0000 7f0737b5b700 1 Processor -- start 2023-10-16T11:21:42.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.640+0000 7f0737b5b700 1 -- start start 2023-10-16T11:21:42.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.641+0000 7f0737b5b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f073015b730 0x7f07300ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f0737b5b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07300ca780 con 0x7f07300ce630 2023-10-16T11:21:42.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f0737b5b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073006aa90 con 0x7f073015b350 2023-10-16T11:21:42.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f0737b5b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07300cf820 con 0x7f073015b730 2023-10-16T11:21:42.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07350f6700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f073015b730 0x7f07300ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07360f8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f07300ce630 0x7f073006af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43406/0 (socket says 172.21.15.158:43406) 2023-10-16T11:21:42.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07360f8700 1 -- 172.21.15.158:0/824432820 learned_addr learned my addr 172.21.15.158:0/824432820 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:42.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07350f6700 1 -- 172.21.15.158:0/824432820 >> v1:172.21.15.172:6789/0 conn(0x7f073015b350 legacy=0x7f07300cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07350f6700 1 -- 172.21.15.158:0/824432820 >> v1:172.21.15.158:6789/0 conn(0x7f07300ce630 legacy=0x7f073006af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07350f6700 1 -- 172.21.15.158:0/824432820 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07300cf260 con 0x7f073015b730 2023-10-16T11:21:42.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07350f6700 1 --2- 172.21.15.158:0/824432820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b730 0x7f07300ca080 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f0720009840 tx=0x7f0720009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a333998a147eb47c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:42.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07348f5700 1 -- 172.21.15.158:0/824432820 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720011830 con 0x7f073015b730 2023-10-16T11:21:42.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07348f5700 1 -- 172.21.15.158:0/824432820 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0720011990 con 0x7f073015b730 2023-10-16T11:21:42.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.642+0000 7f07348f5700 1 -- 172.21.15.158:0/824432820 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720011d20 con 0x7f073015b730 2023-10-16T11:21:42.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.643+0000 7f0737b5b700 1 -- 172.21.15.158:0/824432820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b730 msgr2=0x7f07300ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.643+0000 7f0737b5b700 1 --2- 172.21.15.158:0/824432820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b730 0x7f07300ca080 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f0720009840 tx=0x7f0720009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:42.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.643+0000 7f0737b5b700 1 -- 172.21.15.158:0/824432820 shutdown_connections 2023-10-16T11:21:42.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.643+0000 7f0737b5b700 1 --2- 172.21.15.158:0/824432820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b730 0x7f07300ca080 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.643+0000 7f0737b5b700 1 -- 172.21.15.158:0/824432820 >> 172.21.15.158:0/824432820 conn(0x7f07300cc4b0 msgr2=0x7f07300cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:42.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.643+0000 7f0737b5b700 1 -- 172.21.15.158:0/824432820 shutdown_connections 2023-10-16T11:21:42.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.643+0000 7f0737b5b700 1 -- 172.21.15.158:0/824432820 wait complete. 2023-10-16T11:21:42.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.645+0000 7f0737b5b700 1 Processor -- start 2023-10-16T11:21:42.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.645+0000 7f0737b5b700 1 -- start start 2023-10-16T11:21:42.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f0737b5b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07300ce630 0x7f07300c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f0737b5b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b350 0x7f07300c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f0737b5b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f073015b730 0x7f07300c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:42.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f0737b5b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07300c6930 con 0x7f073015b350 2023-10-16T11:21:42.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f0737b5b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07300c9b10 con 0x7f07300ce630 2023-10-16T11:21:42.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f0737b5b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07300ad9b0 con 0x7f073015b730 2023-10-16T11:21:42.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b350 0x7f07300c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b350 0x7f07300c5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43240/0 (socket says 172.21.15.158:43240) 2023-10-16T11:21:42.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 -- 172.21.15.158:0/2304355596 learned_addr learned my addr 172.21.15.158:0/2304355596 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:42.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07360f8700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f073015b730 0x7f07300c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07358f7700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07300ce630 0x7f07300c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:42.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 -- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f073015b730 msgr2=0x7f07300c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f073015b730 0x7f07300c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 -- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07300ce630 msgr2=0x7f07300c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07300ce630 0x7f07300c8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 -- 172.21.15.158:0/2304355596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200094a0 con 0x7f073015b350 2023-10-16T11:21:42.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07350f6700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b350 0x7f07300c5af0 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f0720013000 tx=0x7f0720004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:42.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07267fc700 1 -- 172.21.15.158:0/2304355596 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07200036a0 con 0x7f073015b350 2023-10-16T11:21:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f07267fc700 1 -- 172.21.15.158:0/2304355596 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072001f070 con 0x7f073015b350 2023-10-16T11:21:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.646+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07300c8a70 con 0x7f073015b350 2023-10-16T11:21:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.647+0000 7f07267fc700 1 -- 172.21.15.158:0/2304355596 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720003a60 con 0x7f073015b350 2023-10-16T11:21:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.647+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07300c9030 con 0x7f073015b350 2023-10-16T11:21:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.647+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07300681b0 con 0x7f073015b350 2023-10-16T11:21:42.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.648+0000 7f07267fc700 1 -- 172.21.15.158:0/2304355596 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 163) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0720003bc0 con 0x7f073015b350 2023-10-16T11:21:42.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.648+0000 7f07267fc700 1 -- 172.21.15.158:0/2304355596 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f0720021070 con 0x7f073015b350 2023-10-16T11:21:42.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.653+0000 7f07267fc700 1 -- 172.21.15.158:0/2304355596 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0720055e70 con 0x7f073015b350 2023-10-16T11:21:42.694 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.692+0000 7f76585e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.695 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.692+0000 7f76585e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.695 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.692+0000 7f76585e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.695 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.692+0000 7f76585e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.695 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.692+0000 7f76585e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.698 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.693+0000 7f765b628700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.698 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.693+0000 7f765b628700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.698 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.693+0000 7f765b628700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.699 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.693+0000 7f765b628700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.699 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:21:42.693+0000 7f765b628700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:21:42.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.898+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07300cd820 con 0x7f073015b350 2023-10-16T11:21:42.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.899+0000 7f07267fc700 1 -- 172.21.15.158:0/2304355596 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f0720015020 con 0x7f073015b350 2023-10-16T11:21:42.902 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:42.902 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":145,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1010484950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1010484950},{"type":"v1","addr":"172.21.15.158:6836","nonce":1010484950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":13224,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/54641557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":54641557},{"type":"v1","addr":"172.21.15.158:6837","nonce":54641557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":13239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3388825539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3388825539},{"type":"v1","addr":"172.21.15.172:6835","nonce":3388825539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:20.241439+0000","modified":"2023-10-16T11:21:23.275699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":39845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_39845":{"gid":39845,"name":"d","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6834/3986459990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3986459990},{"type":"v1","addr":"172.21.15.172:6834","nonce":3986459990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T11:21:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b350 msgr2=0x7f07300c5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:42.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b350 0x7f07300c5af0 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f0720013000 tx=0x7f0720004750 comp rx=0 tx=0).stop 2023-10-16T11:21:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 shutdown_connections 2023-10-16T11:21:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f073015b730 0x7f07300c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07300ce630 0x7f07300c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 --2- 172.21.15.158:0/2304355596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f073015b350 0x7f07300c5af0 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 >> 172.21.15.158:0/2304355596 conn(0x7f07300cc4b0 msgr2=0x7f073006ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:42.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 shutdown_connections 2023-10-16T11:21:42.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:42.901+0000 7f0737b5b700 1 -- 172.21.15.158:0/2304355596 wait complete. 2023-10-16T11:21:42.906 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 145 2023-10-16T11:21:42.916 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:43.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.035+0000 7fcc47bba700 1 Processor -- start 2023-10-16T11:21:43.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.035+0000 7fcc47bba700 1 -- start start 2023-10-16T11:21:43.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.036+0000 7fcc47bba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 0x7fcc4006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.036+0000 7fcc47bba700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc40161270 con 0x7fcc4006ab40 2023-10-16T11:21:43.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.036+0000 7fcc47bba700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc40161510 con 0x7fcc400729b0 2023-10-16T11:21:43.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.036+0000 7fcc47bba700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc401617f0 con 0x7fcc4006e5d0 2023-10-16T11:21:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.036+0000 7fcc45956700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcc4006ab40 0x7fcc400722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43472/0 (socket says 172.21.15.158:43472) 2023-10-16T11:21:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.036+0000 7fcc45956700 1 -- 172.21.15.158:0/4265606714 learned_addr learned my addr 172.21.15.158:0/4265606714 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487829966 0 0) 0x7fcc40161270 con 0x7fcc4006ab40 2023-10-16T11:21:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c0035b0 con 0x7fcc4006ab40 2023-10-16T11:21:43.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45155700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 0x7fcc4006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899998623 0 0) 0x7fcc40161510 con 0x7fcc400729b0 2023-10-16T11:21:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc40161270 con 0x7fcc400729b0 2023-10-16T11:21:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1479677136 0 0) 0x7fcc2c0035b0 con 0x7fcc4006ab40 2023-10-16T11:21:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 msgr2=0x7fcc4006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 0x7fcc4006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 >> v1:172.21.15.172:6789/0 conn(0x7fcc400729b0 legacy=0x7fcc4015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc40161ad0 con 0x7fcc4006ab40 2023-10-16T11:21:43.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fcc30001670 con 0x7fcc4006ab40 2023-10-16T11:21:43.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 >> v1:172.21.15.158:6789/0 conn(0x7fcc4006ab40 legacy=0x7fcc400722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc2c003bc0 0x7fcc2c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 0x7fcc2c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c004d90 0x7fcc2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc401617f0 con 0x7fcc2c003bc0 2023-10-16T11:21:43.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c00bac0 con 0x7fcc2c004d90 2023-10-16T11:21:43.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c00bc80 con 0x7fcc4006e5d0 2023-10-16T11:21:43.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45956700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc2c003bc0 0x7fcc2c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45155700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 0x7fcc2c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc46157700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c004d90 0x7fcc2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45956700 1 -- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 msgr2=0x7fcc2c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:43.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45956700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc4006e5d0 0x7fcc2c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45956700 1 -- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c004d90 msgr2=0x7fcc2c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:43.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45956700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c004d90 0x7fcc2c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.037+0000 7fcc45956700 1 -- 172.21.15.158:0/4265606714 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0035b0 con 0x7fcc2c003bc0 2023-10-16T11:21:43.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc45956700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc2c003bc0 0x7fcc2c003fc0 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000c110 tx=0x7fcc2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360fdea83f8750f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3000ccb0 con 0x7fcc2c003bc0 2023-10-16T11:21:43.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3000ce10 con 0x7fcc2c003bc0 2023-10-16T11:21:43.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc44954700 1 -- 172.21.15.158:0/4265606714 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30005ba0 con 0x7fcc2c003bc0 2023-10-16T11:21:43.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc47bba700 1 -- 172.21.15.158:0/4265606714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc2c003bc0 msgr2=0x7fcc2c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc47bba700 1 --2- 172.21.15.158:0/4265606714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc2c003bc0 0x7fcc2c003fc0 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000c110 tx=0x7fcc2c005880 comp rx=0 tx=0).stop 2023-10-16T11:21:43.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc46157700 1 -- 172.21.15.158:0/4265606714 reap_dead start 2023-10-16T11:21:43.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc47bba700 1 -- 172.21.15.158:0/4265606714 shutdown_connections 2023-10-16T11:21:43.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc47bba700 1 -- 172.21.15.158:0/4265606714 >> 172.21.15.158:0/4265606714 conn(0x7fcc400d02b0 msgr2=0x7fcc400d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:43.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc47bba700 1 -- 172.21.15.158:0/4265606714 shutdown_connections 2023-10-16T11:21:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.038+0000 7fcc47bba700 1 -- 172.21.15.158:0/4265606714 wait complete. 2023-10-16T11:21:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.039+0000 7fcc47bba700 1 Processor -- start 2023-10-16T11:21:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc47bba700 1 -- start start 2023-10-16T11:21:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc47bba700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c003bc0 0x7fcc401487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc47bba700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc400c4690 0x7fcc400c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc47bba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc400b07e0 0x7fcc400b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc47bba700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400c5680 con 0x7fcc400b07e0 2023-10-16T11:21:43.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc47bba700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc40148d10 con 0x7fcc2c003bc0 2023-10-16T11:21:43.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc47bba700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400b0350 con 0x7fcc400c4690 2023-10-16T11:21:43.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc46157700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc400b07e0 0x7fcc400b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc46157700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc400b07e0 0x7fcc400b0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43312/0 (socket says 172.21.15.158:43312) 2023-10-16T11:21:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc46157700 1 -- 172.21.15.158:0/467141893 learned_addr learned my addr 172.21.15.158:0/467141893 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc45956700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c003bc0 0x7fcc401487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.040+0000 7fcc45155700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc400c4690 0x7fcc400c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc46157700 1 -- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc400c4690 msgr2=0x7fcc400c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc46157700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc400c4690 0x7fcc400c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc46157700 1 -- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c003bc0 msgr2=0x7fcc401487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc46157700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c003bc0 0x7fcc401487d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc46157700 1 -- 172.21.15.158:0/467141893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400c53a0 con 0x7fcc400b07e0 2023-10-16T11:21:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc45155700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc400c4690 0x7fcc400c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc46157700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc400b07e0 0x7fcc400b0c90 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7fcc4006eb20 tx=0x7fcc3800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc367fc700 1 -- 172.21.15.158:0/467141893 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38003c10 con 0x7fcc400b07e0 2023-10-16T11:21:43.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc367fc700 1 -- 172.21.15.158:0/467141893 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc38003d90 con 0x7fcc400b07e0 2023-10-16T11:21:43.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc400afd00 con 0x7fcc400b07e0 2023-10-16T11:21:43.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc367fc700 1 -- 172.21.15.158:0/467141893 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3800dbd0 con 0x7fcc400b07e0 2023-10-16T11:21:43.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.041+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc400c5050 con 0x7fcc400b07e0 2023-10-16T11:21:43.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.042+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc400afb60 con 0x7fcc400b07e0 2023-10-16T11:21:43.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.042+0000 7fcc367fc700 1 -- 172.21.15.158:0/467141893 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 163) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fcc3800dd30 con 0x7fcc400b07e0 2023-10-16T11:21:43.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.043+0000 7fcc367fc700 1 -- 172.21.15.158:0/467141893 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fcc38089990 con 0x7fcc400b07e0 2023-10-16T11:21:43.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.047+0000 7fcc367fc700 1 -- 172.21.15.158:0/467141893 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc400afb60 con 0x7fcc400b07e0 2023-10-16T11:21:43.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.243+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc400afb60 con 0x7fcc400b07e0 2023-10-16T11:21:43.247 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.244+0000 7fcc367fc700 1 -- 172.21.15.158:0/467141893 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+18610 (secure 0 0 0) 0x7fcc400afb60 con 0x7fcc400b07e0 2023-10-16T11:21:43.248 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:43.248 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":259,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:42.628033+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:18.736213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:19.205576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6832/1196176990":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/2368789841":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/1642562285":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/1105433381":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/2407391765":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/595141618":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1477193149":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/1384560093":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/385046482":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6833/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6832/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/4272085425":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1184494648":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/2564736313":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/2593348127":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1448525844":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6833/1196176990":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/3641755142":"2023-10-17T11:21:34.118742+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc400b07e0 msgr2=0x7fcc400b0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc400b07e0 0x7fcc400b0c90 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7fcc4006eb20 tx=0x7fcc3800d430 comp rx=0 tx=0).stop 2023-10-16T11:21:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 shutdown_connections 2023-10-16T11:21:43.250 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc400b07e0 0x7fcc400b0c90 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc2c003bc0 0x7fcc401487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 --2- 172.21.15.158:0/467141893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc400c4690 0x7fcc400c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 >> 172.21.15.158:0/467141893 conn(0x7fcc400d02b0 msgr2=0x7fcc40157740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 shutdown_connections 2023-10-16T11:21:43.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.247+0000 7fcc47bba700 1 -- 172.21.15.158:0/467141893 wait complete. 2023-10-16T11:21:43.262 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:21:43.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.381+0000 7f8510a08700 1 Processor -- start 2023-10-16T11:21:43.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.381+0000 7f8510a08700 1 -- start start 2023-10-16T11:21:43.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8510a08700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15d990 0x7f850c0b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8510a08700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f850c06e480 0x7f850c0b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8510a08700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f850c0b3760 0x7f850c0cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8510a08700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0b2c90 con 0x7f850c06e480 2023-10-16T11:21:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8510a08700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0af600 con 0x7f850c0b3760 2023-10-16T11:21:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8510a08700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c06e9b0 con 0x7f850c15d990 2023-10-16T11:21:43.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f850c06e480 0x7f850c0b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f850c06e480 0x7f850c0b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43314/0 (socket says 172.21.15.158:43314) 2023-10-16T11:21:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 -- 172.21.15.158:0/1295315310 learned_addr learned my addr 172.21.15.158:0/1295315310 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f850ad9d700 1 --2- 172.21.15.158:0/1295315310 >> v2:172.21.15.172:3300/0 conn(0x7f850c0b3760 0x7f850c0cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f850a59c700 1 --2- 172.21.15.158:0/1295315310 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15d990 0x7f850c0b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 -- 172.21.15.158:0/1295315310 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15d990 msgr2=0x7f850c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 --2- 172.21.15.158:0/1295315310 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15d990 0x7f850c0b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 -- 172.21.15.158:0/1295315310 >> v2:172.21.15.172:3300/0 conn(0x7f850c0b3760 msgr2=0x7f850c0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 --2- 172.21.15.158:0/1295315310 >> v2:172.21.15.172:3300/0 conn(0x7f850c0b3760 0x7f850c0cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 -- 172.21.15.158:0/1295315310 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0ce890 con 0x7f850c06e480 2023-10-16T11:21:43.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f850a59c700 1 --2- 172.21.15.158:0/1295315310 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15d990 0x7f850c0b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:43.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f850ad9d700 1 --2- 172.21.15.158:0/1295315310 >> v2:172.21.15.172:3300/0 conn(0x7f850c0b3760 0x7f850c0cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:43.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f8509d9b700 1 --2- 172.21.15.158:0/1295315310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c0b3220 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 crypto rx=0x7f850c0cdec0 tx=0x7f84fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf56e67b6713375e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.382+0000 7f850959a700 1 -- 172.21.15.158:0/1295315310 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00b870 con 0x7f850c06e480 2023-10-16T11:21:43.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f850959a700 1 -- 172.21.15.158:0/1295315310 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc00b9d0 con 0x7f850c06e480 2023-10-16T11:21:43.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f850959a700 1 -- 172.21.15.158:0/1295315310 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00bd60 con 0x7f850c06e480 2023-10-16T11:21:43.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 -- 172.21.15.158:0/1295315310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 msgr2=0x7f850c0b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 --2- 172.21.15.158:0/1295315310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c0b3220 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 crypto rx=0x7f850c0cdec0 tx=0x7f84fc009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 -- 172.21.15.158:0/1295315310 shutdown_connections 2023-10-16T11:21:43.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 --2- 172.21.15.158:0/1295315310 >> v2:172.21.15.172:3300/0 conn(0x7f850c0b3760 0x7f850c0cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 --2- 172.21.15.158:0/1295315310 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c15d990 0x7f850c0b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 --2- 172.21.15.158:0/1295315310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c0b3220 unknown :-1 s=CLOSED pgs=3229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 -- 172.21.15.158:0/1295315310 >> 172.21.15.158:0/1295315310 conn(0x7f850c06b9e0 msgr2=0x7f850c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 -- 172.21.15.158:0/1295315310 shutdown_connections 2023-10-16T11:21:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.383+0000 7f8510a08700 1 -- 172.21.15.158:0/1295315310 wait complete. 2023-10-16T11:21:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.384+0000 7f8510a08700 1 Processor -- start 2023-10-16T11:21:43.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.384+0000 7f8510a08700 1 -- start start 2023-10-16T11:21:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f8510a08700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f8510a08700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c0b3760 0x7f850c0718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f8510a08700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15d990 0x7f850c071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f8510a08700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0c50f0 con 0x7f850c06e480 2023-10-16T11:21:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f8510a08700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0c53d0 con 0x7f850c15d990 2023-10-16T11:21:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f8510a08700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0c56b0 con 0x7f850c0b3760 2023-10-16T11:21:43.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f850a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f850a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43320/0 (socket says 172.21.15.158:43320) 2023-10-16T11:21:43.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f850a59c700 1 -- 172.21.15.158:0/3962216431 learned_addr learned my addr 172.21.15.158:0/3962216431 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:43.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f8509d9b700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c0b3760 0x7f850c0718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.385+0000 7f850ad9d700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15d990 0x7f850c071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f850a59c700 1 -- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c0b3760 msgr2=0x7f850c0718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f850a59c700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c0b3760 0x7f850c0718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f850a59c700 1 -- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15d990 msgr2=0x7f850c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f850a59c700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15d990 0x7f850c071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f850a59c700 1 -- 172.21.15.158:0/3962216431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc009550 con 0x7f850c06e480 2023-10-16T11:21:43.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f850ad9d700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15d990 0x7f850c071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:43.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f850a59c700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c159aa0 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7f850000c7f0 tx=0x7f850000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500011800 con 0x7f850c06e480 2023-10-16T11:21:43.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500011960 con 0x7f850c06e480 2023-10-16T11:21:43.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500011cf0 con 0x7f850c06e480 2023-10-16T11:21:43.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c0c5990 con 0x7f850c06e480 2023-10-16T11:21:43.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.386+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c0c6300 con 0x7f850c06e480 2023-10-16T11:21:43.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.387+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c0cf550 con 0x7f850c06e480 2023-10-16T11:21:43.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.387+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 163) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f8500018360 con 0x7f850c06e480 2023-10-16T11:21:43.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.388+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f85000895c0 con 0x7f850c06e480 2023-10-16T11:21:43.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.392+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8500054410 con 0x7f850c06e480 2023-10-16T11:21:43.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.628+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f850c0681b0 con 0x7f850c06e480 2023-10-16T11:21:43.632 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:21:43.629+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:21:43.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.637+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 164) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f8500053ec0 con 0x7f850c06e480 2023-10-16T11:21:43.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.637+0000 7f84faffd700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f84ec071390 0x7f84ec073780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.638+0000 7f8509d9b700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f84ec071390 0x7f84ec073780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.639+0000 7f8509d9b700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f84ec071390 0x7f84ec073780 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f850c0cdec0 tx=0x7f84fc008030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.641+0000 7f84faffd700 1 -- 172.21.15.158:0/3962216431 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v146) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8500075050 con 0x7f850c06e480 2023-10-16T11:21:43.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.644+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f84ec071390 msgr2=0x7f84ec073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f84ec071390 0x7f84ec073780 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f850c0cdec0 tx=0x7f84fc008030 comp rx=0 tx=0).stop 2023-10-16T11:21:43.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 msgr2=0x7f850c159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c159aa0 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7f850000c7f0 tx=0x7f850000cb00 comp rx=0 tx=0).stop 2023-10-16T11:21:43.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 shutdown_connections 2023-10-16T11:21:43.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c15d990 0x7f850c071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f84ec071390 0x7f84ec073780 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f850c06e480 0x7f850c159aa0 unknown :-1 s=CLOSED pgs=3230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.645+0000 7f8510a08700 1 --2- 172.21.15.158:0/3962216431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f850c0b3760 0x7f850c0718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.646+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 >> 172.21.15.158:0/3962216431 conn(0x7f850c06b9e0 msgr2=0x7f850c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:43.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.646+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 shutdown_connections 2023-10-16T11:21:43.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.646+0000 7f8510a08700 1 -- 172.21.15.158:0/3962216431 wait complete. 2023-10-16T11:21:43.651 INFO:tasks.ceph.mds.d.smithi172.stderr: -5187> 2023-10-16T11:21:22.829+0000 7fedebc3a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T11:21:43.651 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:21:43.661 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:21:43.679 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:21:43.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.782+0000 7f39aeb15700 1 Processor -- start 2023-10-16T11:21:43.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.782+0000 7f39aeb15700 1 -- start start 2023-10-16T11:21:43.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39aeb15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a80cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39aeb15700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a80d0ae0 con 0x7f39a815deb0 2023-10-16T11:21:43.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39aeb15700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a80d12d0 con 0x7f39a80ccc00 2023-10-16T11:21:43.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39aeb15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80cbdf0 con 0x7f39a815dad0 2023-10-16T11:21:43.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39a7fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f39a815deb0 0x7f39a80ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43528/0 (socket says 172.21.15.158:43528) 2023-10-16T11:21:43.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39a7fff700 1 -- 172.21.15.158:0/4094279066 learned_addr learned my addr 172.21.15.158:0/4094279066 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:43.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39ac8b1700 1 --2- 172.21.15.158:0/4094279066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a80cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39a77fe700 1 -- 172.21.15.158:0/4094279066 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718241681 0 0) 0x7f39a80d0ae0 con 0x7f39a815deb0 2023-10-16T11:21:43.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.783+0000 7f39a77fe700 1 -- 172.21.15.158:0/4094279066 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39980035b0 con 0x7f39a815deb0 2023-10-16T11:21:43.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39ac8b1700 1 -- 172.21.15.158:0/4094279066 >> v1:172.21.15.172:6789/0 conn(0x7f39a80ccc00 legacy=0x7f39a80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39a77fe700 1 -- 172.21.15.158:0/4094279066 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528654664 0 0) 0x7f39a80d12d0 con 0x7f39a80ccc00 2023-10-16T11:21:43.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39ac8b1700 1 -- 172.21.15.158:0/4094279066 >> v1:172.21.15.158:6789/0 conn(0x7f39a815deb0 legacy=0x7f39a80ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39ac8b1700 1 -- 172.21.15.158:0/4094279066 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80cc280 con 0x7f39a815dad0 2023-10-16T11:21:43.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39ac8b1700 1 --2- 172.21.15.158:0/4094279066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a80cdfa0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f399c009870 tx=0x7f399c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c86858ab5392a81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39a77fe700 1 -- 172.21.15.158:0/4094279066 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c004230 con 0x7f39a815dad0 2023-10-16T11:21:43.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39a77fe700 1 -- 172.21.15.158:0/4094279066 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c004390 con 0x7f39a815dad0 2023-10-16T11:21:43.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.784+0000 7f39a77fe700 1 -- 172.21.15.158:0/4094279066 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c0036a0 con 0x7f39a815dad0 2023-10-16T11:21:43.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.785+0000 7f39aeb15700 1 -- 172.21.15.158:0/4094279066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 msgr2=0x7f39a80cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.785+0000 7f39aeb15700 1 --2- 172.21.15.158:0/4094279066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a80cdfa0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f399c009870 tx=0x7f399c009b80 comp rx=0 tx=0).stop 2023-10-16T11:21:43.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.785+0000 7f39aeb15700 1 -- 172.21.15.158:0/4094279066 shutdown_connections 2023-10-16T11:21:43.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.785+0000 7f39aeb15700 1 --2- 172.21.15.158:0/4094279066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a80cdfa0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.785+0000 7f39aeb15700 1 -- 172.21.15.158:0/4094279066 >> 172.21.15.158:0/4094279066 conn(0x7f39a8099a10 msgr2=0x7f39a80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:43.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.785+0000 7f39aeb15700 1 -- 172.21.15.158:0/4094279066 shutdown_connections 2023-10-16T11:21:43.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.785+0000 7f39aeb15700 1 -- 172.21.15.158:0/4094279066 wait complete. 2023-10-16T11:21:43.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.786+0000 7f39aeb15700 1 Processor -- start 2023-10-16T11:21:43.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.786+0000 7f39aeb15700 1 -- start start 2023-10-16T11:21:43.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39aeb15700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39a80ccc00 0x7f39a81533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39aeb15700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a81f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39aeb15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39a815deb0 0x7f39a815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39aeb15700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a8153900 con 0x7f39a815deb0 2023-10-16T11:21:43.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39aeb15700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a81f2d10 con 0x7f39a80ccc00 2023-10-16T11:21:43.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39aeb15700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a8152ce0 con 0x7f39a815dad0 2023-10-16T11:21:43.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ad0b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39a815deb0 0x7f39a815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39a80ccc00 0x7f39a81533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39a80ccc00 0x7f39a81533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:44536/0 (socket says 172.21.15.158:44536) 2023-10-16T11:21:43.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 -- 172.21.15.158:0/1715599836 learned_addr learned my addr 172.21.15.158:0/1715599836 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:43.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39a7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a81f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ad0b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39a815deb0 0x7f39a815ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43360/0 (socket says 172.21.15.158:43360) 2023-10-16T11:21:43.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 -- 172.21.15.158:0/1715599836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 msgr2=0x7f39a81f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a81f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 -- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39a815deb0 msgr2=0x7f39a815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:43.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39a815deb0 0x7f39a815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:43.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 -- 172.21.15.158:0/1715599836 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0094a0 con 0x7f39a80ccc00 2023-10-16T11:21:43.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ad0b2700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39a815deb0 0x7f39a815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:43.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.787+0000 7f39ac8b1700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39a80ccc00 0x7f39a81533c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097c0 tx=0x7f399c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.788+0000 7f39a57fa700 1 -- 172.21.15.158:0/1715599836 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c01f070 con 0x7f39a80ccc00 2023-10-16T11:21:43.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.788+0000 7f39a57fa700 1 -- 172.21.15.158:0/1715599836 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c003990 con 0x7f39a80ccc00 2023-10-16T11:21:43.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.788+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a8154010 con 0x7f39a80ccc00 2023-10-16T11:21:43.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.788+0000 7f39a57fa700 1 -- 172.21.15.158:0/1715599836 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c003d40 con 0x7f39a80ccc00 2023-10-16T11:21:43.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.788+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a81f29a0 con 0x7f39a80ccc00 2023-10-16T11:21:43.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.789+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a81f2ff0 con 0x7f39a80ccc00 2023-10-16T11:21:43.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.789+0000 7f39a57fa700 1 -- 172.21.15.158:0/1715599836 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 164) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f399c023070 con 0x7f39a80ccc00 2023-10-16T11:21:43.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.789+0000 7f39a57fa700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f398c06ad50 0x7f398c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:43.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.789+0000 7f39a57fa700 1 -- 172.21.15.158:0/1715599836 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f399c08c220 con 0x7f39a80ccc00 2023-10-16T11:21:43.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.789+0000 7f39a7fff700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f398c06ad50 0x7f398c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:43.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.790+0000 7f39a7fff700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f398c06ad50 0x7f398c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f39a815bf90 tx=0x7f3994000bc0 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:43.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:43.795+0000 7f39a57fa700 1 -- 172.21.15.158:0/1715599836 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f399c057030 con 0x7f39a80ccc00 2023-10-16T11:21:44.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.036+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f39a81f2b30 con 0x7f39a80ccc00 2023-10-16T11:21:44.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.649+0000 7f39a57fa700 1 -- 172.21.15.158:0/1715599836 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v147) v1 ==== 104+0+0 (secure 0 0 0) 0x7f399c05a650 con 0x7f39a80ccc00 2023-10-16T11:21:44.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f398c06ad50 msgr2=0x7f398c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:44.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f398c06ad50 0x7f398c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f39a815bf90 tx=0x7f3994000bc0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39a80ccc00 msgr2=0x7f39a81533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:44.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39a80ccc00 0x7f39a81533c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097c0 tx=0x7f399c004750 comp rx=0 tx=0).stop 2023-10-16T11:21:44.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 shutdown_connections 2023-10-16T11:21:44.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f398c06ad50 0x7f398c06d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39a815deb0 0x7f39a815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39a80ccc00 0x7f39a81533c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 --2- 172.21.15.158:0/1715599836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39a815dad0 0x7f39a81f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 >> 172.21.15.158:0/1715599836 conn(0x7f39a8099a10 msgr2=0x7f39a80c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:44.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 shutdown_connections 2023-10-16T11:21:44.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.652+0000 7f39aeb15700 1 -- 172.21.15.158:0/1715599836 wait complete. 2023-10-16T11:21:44.667 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:21:44.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.792+0000 7f664f834700 1 Processor -- start 2023-10-16T11:21:44.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.792+0000 7f664f834700 1 -- start start 2023-10-16T11:21:44.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66480c50a0 0x7f66480c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:44.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664806c220 con 0x7f6648143790 2023-10-16T11:21:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664806aa70 con 0x7f664806e540 2023-10-16T11:21:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6648143b70 con 0x7f66480c50a0 2023-10-16T11:21:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664d5d0700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6648143790 0x7f664806bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43558/0 (socket says 172.21.15.158:43558) 2023-10-16T11:21:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664d5d0700 1 -- 172.21.15.158:0/2500966719 learned_addr learned my addr 172.21.15.158:0/2500966719 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:44.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664cdcf700 1 --2- 172.21.15.158:0/2500966719 >> v2:172.21.15.172:3300/0 conn(0x7f66480c50a0 0x7f66480c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f663ffff700 1 -- 172.21.15.158:0/2500966719 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456786294 0 0) 0x7f664806c220 con 0x7f6648143790 2023-10-16T11:21:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f663ffff700 1 -- 172.21.15.158:0/2500966719 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66340034d0 con 0x7f6648143790 2023-10-16T11:21:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f663ffff700 1 -- 172.21.15.158:0/2500966719 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 605516343 0 0) 0x7f66340034d0 con 0x7f6648143790 2023-10-16T11:21:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664cdcf700 1 -- 172.21.15.158:0/2500966719 >> v1:172.21.15.172:6789/0 conn(0x7f664806e540 legacy=0x7f664806e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:44.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664cdcf700 1 -- 172.21.15.158:0/2500966719 >> v1:172.21.15.158:6789/0 conn(0x7f6648143790 legacy=0x7f664806bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:44.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664cdcf700 1 -- 172.21.15.158:0/2500966719 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664806ad50 con 0x7f66480c50a0 2023-10-16T11:21:44.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664cdcf700 1 --2- 172.21.15.158:0/2500966719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66480c50a0 0x7f66480c5520 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f6638009840 tx=0x7f6638009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20504633eb2149d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:44.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f663ffff700 1 -- 172.21.15.158:0/2500966719 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6644001670 con 0x7f66480c50a0 2023-10-16T11:21:44.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f663ffff700 1 -- 172.21.15.158:0/2500966719 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6638004030 con 0x7f66480c50a0 2023-10-16T11:21:44.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f663ffff700 1 -- 172.21.15.158:0/2500966719 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380043a0 con 0x7f66480c50a0 2023-10-16T11:21:44.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- 172.21.15.158:0/2500966719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66480c50a0 msgr2=0x7f66480c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:44.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 --2- 172.21.15.158:0/2500966719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66480c50a0 0x7f66480c5520 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f6638009840 tx=0x7f6638009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:44.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- 172.21.15.158:0/2500966719 shutdown_connections 2023-10-16T11:21:44.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 --2- 172.21.15.158:0/2500966719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66480c50a0 0x7f66480c5520 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- 172.21.15.158:0/2500966719 >> 172.21.15.158:0/2500966719 conn(0x7f66480cc5b0 msgr2=0x7f66480cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:44.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- 172.21.15.158:0/2500966719 shutdown_connections 2023-10-16T11:21:44.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.794+0000 7f664f834700 1 -- 172.21.15.158:0/2500966719 wait complete. 2023-10-16T11:21:44.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 Processor -- start 2023-10-16T11:21:44.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 -- start start 2023-10-16T11:21:44.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f664806e540 0x7f6648137ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:44.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66480c50a0 0x7f66481e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:44.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6648143790 0x7f6648136f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:44.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481e3df0 con 0x7f6648143790 2023-10-16T11:21:44.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481e40d0 con 0x7f664806e540 2023-10-16T11:21:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664f834700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66481e43b0 con 0x7f66480c50a0 2023-10-16T11:21:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664ddd1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6648143790 0x7f6648136f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664ddd1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6648143790 0x7f6648136f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43392/0 (socket says 172.21.15.158:43392) 2023-10-16T11:21:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.796+0000 7f664ddd1700 1 -- 172.21.15.158:0/2619528207 learned_addr learned my addr 172.21.15.158:0/2619528207 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664d5d0700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f664806e540 0x7f6648137ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664cdcf700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66480c50a0 0x7f66481e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664ddd1700 1 -- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66480c50a0 msgr2=0x7f66481e35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664ddd1700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66480c50a0 0x7f66481e35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664ddd1700 1 -- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f664806e540 msgr2=0x7f6648137ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664ddd1700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f664806e540 0x7f6648137ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664ddd1700 1 -- 172.21.15.158:0/2619528207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380094a0 con 0x7f6648143790 2023-10-16T11:21:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664d5d0700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f664806e540 0x7f6648137ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664ddd1700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6648143790 0x7f6648136f10 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7f664000c890 tx=0x7f664000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640003e30 con 0x7f6648143790 2023-10-16T11:21:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664000ce90 con 0x7f6648143790 2023-10-16T11:21:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66481e4690 con 0x7f6648143790 2023-10-16T11:21:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400052b0 con 0x7f6648143790 2023-10-16T11:21:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.797+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66481e3ae0 con 0x7f6648143790 2023-10-16T11:21:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.798+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66481e4bb0 con 0x7f6648143790 2023-10-16T11:21:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.799+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 164) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6640005410 con 0x7f6648143790 2023-10-16T11:21:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.799+0000 7f663dffb700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f662c06ad80 0x7f662c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:44.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.799+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f66400895e0 con 0x7f6648143790 2023-10-16T11:21:44.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.799+0000 7f664d5d0700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f662c06ad80 0x7f662c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:44.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.799+0000 7f664d5d0700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f662c06ad80 0x7f662c06d240 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6644001eb0 tx=0x7f6644009040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:44.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:44.804+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66481e4bb0 con 0x7f6648143790 2023-10-16T11:21:45.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.000+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f66481e4bb0 con 0x7f6648143790 2023-10-16T11:21:45.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.156+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6640054720 con 0x7f6648143790 2023-10-16T11:21:45.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.652+0000 7f663dffb700 1 -- 172.21.15.158:0/2619528207 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v261) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6640057de0 con 0x7f6648143790 2023-10-16T11:21:45.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f662c06ad80 msgr2=0x7f662c06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:45.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f662c06ad80 0x7f662c06d240 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6644001eb0 tx=0x7f6644009040 comp rx=0 tx=0).stop 2023-10-16T11:21:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6648143790 msgr2=0x7f6648136f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6648143790 0x7f6648136f10 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7f664000c890 tx=0x7f664000cc50 comp rx=0 tx=0).stop 2023-10-16T11:21:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 shutdown_connections 2023-10-16T11:21:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f662c06ad80 0x7f662c06d240 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6648143790 0x7f6648136f10 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f664806e540 0x7f6648137ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 --2- 172.21.15.158:0/2619528207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66480c50a0 0x7f66481e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.655+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 >> 172.21.15.158:0/2619528207 conn(0x7f66480cc5b0 msgr2=0x7f66480cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:45.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.656+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 shutdown_connections 2023-10-16T11:21:45.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.656+0000 7f664f834700 1 -- 172.21.15.158:0/2619528207 wait complete. 2023-10-16T11:21:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:21:45.672 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:21:45.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.796+0000 7f2ae0c65700 1 Processor -- start 2023-10-16T11:21:45.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.796+0000 7f2ae0c65700 1 -- start start 2023-10-16T11:21:45.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ae0c65700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2adc15dad0 0x7f2adc0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ae0c65700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2adc15deb0 0x7f2adc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ae0c65700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2adc0d0ae0 con 0x7f2adc0cea20 2023-10-16T11:21:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ae0c65700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2adc0cee50 con 0x7f2adc15deb0 2023-10-16T11:21:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ae0c65700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2adc0d12d0 con 0x7f2adc15dad0 2023-10-16T11:21:45.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ad9d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2adc15deb0 0x7f2adc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2adad9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2adc0cea20 0x7f2adc0c3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43574/0 (socket says 172.21.15.158:43574) 2023-10-16T11:21:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2adad9d700 1 -- 172.21.15.158:0/674427833 learned_addr learned my addr 172.21.15.158:0/674427833 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ad9d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2adc15deb0 0x7f2adc0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43398/0 (socket says 172.21.15.158:43398) 2023-10-16T11:21:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ada59c700 1 --2- 172.21.15.158:0/674427833 >> v2:172.21.15.172:3300/0 conn(0x7f2adc15dad0 0x7f2adc0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:45.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ad9d9b700 1 -- 172.21.15.158:0/674427833 >> v2:172.21.15.172:3300/0 conn(0x7f2adc15dad0 msgr2=0x7f2adc0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ad9d9b700 1 --2- 172.21.15.158:0/674427833 >> v2:172.21.15.172:3300/0 conn(0x7f2adc15dad0 0x7f2adc0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ad9d9b700 1 -- 172.21.15.158:0/674427833 >> v1:172.21.15.158:6789/0 conn(0x7f2adc0cea20 legacy=0x7f2adc0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ad959a700 1 -- 172.21.15.158:0/674427833 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728303581 0 0) 0x7f2adc0d0ae0 con 0x7f2adc0cea20 2023-10-16T11:21:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.797+0000 7f2ad9d9b700 1 -- 172.21.15.158:0/674427833 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc0cbdf0 con 0x7f2adc15deb0 2023-10-16T11:21:45.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ad9d9b700 1 --2- 172.21.15.158:0/674427833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc0ce4e0 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009840 tx=0x7f2acc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4759533f5570342 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:45.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ad959a700 1 -- 172.21.15.158:0/674427833 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc011830 con 0x7f2adc15deb0 2023-10-16T11:21:45.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ad959a700 1 -- 172.21.15.158:0/674427833 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc011990 con 0x7f2adc15deb0 2023-10-16T11:21:45.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ad959a700 1 -- 172.21.15.158:0/674427833 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc011d20 con 0x7f2adc15deb0 2023-10-16T11:21:45.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 -- 172.21.15.158:0/674427833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 msgr2=0x7f2adc0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:45.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/674427833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc0ce4e0 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009840 tx=0x7f2acc009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:45.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 -- 172.21.15.158:0/674427833 shutdown_connections 2023-10-16T11:21:45.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/674427833 >> v2:172.21.15.172:3300/0 conn(0x7f2adc15dad0 0x7f2adc0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/674427833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc0ce4e0 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 -- 172.21.15.158:0/674427833 >> 172.21.15.158:0/674427833 conn(0x7f2adc099a10 msgr2=0x7f2adc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:45.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 -- 172.21.15.158:0/674427833 shutdown_connections 2023-10-16T11:21:45.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.798+0000 7f2ae0c65700 1 -- 172.21.15.158:0/674427833 wait complete. 2023-10-16T11:21:45.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.800+0000 7f2ae0c65700 1 Processor -- start 2023-10-16T11:21:45.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.800+0000 7f2ae0c65700 1 -- start start 2023-10-16T11:21:45.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ae0c65700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2adc0cea20 0x7f2adc14fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:45.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ae0c65700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2adc15dad0 0x7f2adc0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:45.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ae0c65700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc14e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:45.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ae0c65700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc06f060 con 0x7f2adc15deb0 2023-10-16T11:21:45.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ae0c65700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc14e690 con 0x7f2adc0cea20 2023-10-16T11:21:45.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ae0c65700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0c5630 con 0x7f2adc15dad0 2023-10-16T11:21:45.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2adad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc14e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:45.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2adad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc14e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43404/0 (socket says 172.21.15.158:43404) 2023-10-16T11:21:45.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2adad9d700 1 -- 172.21.15.158:0/586770813 learned_addr learned my addr 172.21.15.158:0/586770813 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:45.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ada59c700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2adc0cea20 0x7f2adc14fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:45.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2adad9d700 1 -- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2adc15dad0 msgr2=0x7f2adc0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:45.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2ad9d9b700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2adc15dad0 0x7f2adc0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:45.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2adad9d700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2adc15dad0 0x7f2adc0c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2adad9d700 1 -- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2adc0cea20 msgr2=0x7f2adc14fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:45.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.801+0000 7f2adad9d700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2adc0cea20 0x7f2adc14fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:45.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2adad9d700 1 -- 172.21.15.158:0/586770813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0094a0 con 0x7f2adc15deb0 2023-10-16T11:21:45.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2ada59c700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2adc0cea20 0x7f2adc14fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:45.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2adad9d700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc14e150 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4002230 tx=0x7f2ad400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:45.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2acaffd700 1 -- 172.21.15.158:0/586770813 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad400ac20 con 0x7f2adc15deb0 2023-10-16T11:21:45.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2acaffd700 1 -- 172.21.15.158:0/586770813 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad400ada0 con 0x7f2adc15deb0 2023-10-16T11:21:45.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2acaffd700 1 -- 172.21.15.158:0/586770813 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4007310 con 0x7f2adc15deb0 2023-10-16T11:21:45.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2adc0c5910 con 0x7f2adc15deb0 2023-10-16T11:21:45.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.802+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2adc0c5ed0 con 0x7f2adc15deb0 2023-10-16T11:21:45.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.803+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2adc0c60c0 con 0x7f2adc15deb0 2023-10-16T11:21:45.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.803+0000 7f2acaffd700 1 -- 172.21.15.158:0/586770813 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2ad4007470 con 0x7f2adc15deb0 2023-10-16T11:21:45.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.804+0000 7f2acaffd700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2abc06af90 0x7f2abc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:45.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.804+0000 7f2acaffd700 1 -- 172.21.15.158:0/586770813 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6931+0+0 (secure 0 0 0) 0x7f2ad4089660 con 0x7f2adc15deb0 2023-10-16T11:21:45.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.804+0000 7f2ada59c700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2abc06af90 0x7f2abc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:45.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.804+0000 7f2ada59c700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2abc06af90 0x7f2abc06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009a00 tx=0x7f2ad0008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:45.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:45.809+0000 7f2acaffd700 1 -- 172.21.15.158:0/586770813 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ad4054250 con 0x7f2adc15deb0 2023-10-16T11:21:46.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.000+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2adc0681b0 con 0x7f2adc15deb0 2023-10-16T11:21:46.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.654+0000 7f2acaffd700 1 -- 172.21.15.158:0/586770813 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v262) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2ad4057870 con 0x7f2adc15deb0 2023-10-16T11:21:46.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2abc06af90 msgr2=0x7f2abc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:46.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2abc06af90 0x7f2abc06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009a00 tx=0x7f2ad0008040 comp rx=0 tx=0).stop 2023-10-16T11:21:46.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 msgr2=0x7f2adc14e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:46.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc14e150 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4002230 tx=0x7f2ad400a060 comp rx=0 tx=0).stop 2023-10-16T11:21:46.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 shutdown_connections 2023-10-16T11:21:46.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2abc06af90 0x7f2abc06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2adc15deb0 0x7f2adc14e150 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2adc0cea20 0x7f2adc14fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 --2- 172.21.15.158:0/586770813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2adc15dad0 0x7f2adc0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.657+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 >> 172.21.15.158:0/586770813 conn(0x7f2adc099a10 msgr2=0x7f2adc15cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:46.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.658+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 shutdown_connections 2023-10-16T11:21:46.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.658+0000 7f2ae0c65700 1 -- 172.21.15.158:0/586770813 wait complete. 2023-10-16T11:21:46.662 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:21:46.677 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:21:46.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.808+0000 7f2a91090700 1 Processor -- start 2023-10-16T11:21:46.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.808+0000 7f2a91090700 1 -- start start 2023-10-16T11:21:46.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a91090700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a8c15d910 0x7f2a8c0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:46.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a91090700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2a8c0700a0 0x7f2a8c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:46.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a91090700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c06a8d0 con 0x7f2a8c0cc9e0 2023-10-16T11:21:46.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a91090700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c161e50 con 0x7f2a8c0700a0 2023-10-16T11:21:46.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a91090700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c072970 con 0x7f2a8c15d910 2023-10-16T11:21:46.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2a8c0cc9e0 0x7f2a8c06ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43586/0 (socket says 172.21.15.158:43586) 2023-10-16T11:21:46.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8a59c700 1 -- 172.21.15.158:0/133899701 learned_addr learned my addr 172.21.15.158:0/133899701 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:46.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8b59e700 1 --2- 172.21.15.158:0/133899701 >> v2:172.21.15.158:3300/0 conn(0x7f2a8c0700a0 0x7f2a8c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:46.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8ad9d700 1 --2- 172.21.15.158:0/133899701 >> v2:172.21.15.172:3300/0 conn(0x7f2a8c15d910 0x7f2a8c0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:46.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a89d9b700 1 -- 172.21.15.158:0/133899701 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391073112 0 0) 0x7f2a8c06a8d0 con 0x7f2a8c0cc9e0 2023-10-16T11:21:46.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a89d9b700 1 -- 172.21.15.158:0/133899701 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a740035b0 con 0x7f2a8c0cc9e0 2023-10-16T11:21:46.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8b59e700 1 -- 172.21.15.158:0/133899701 >> v2:172.21.15.172:3300/0 conn(0x7f2a8c15d910 msgr2=0x7f2a8c0cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:46.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8b59e700 1 --2- 172.21.15.158:0/133899701 >> v2:172.21.15.172:3300/0 conn(0x7f2a8c15d910 0x7f2a8c0cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8b59e700 1 -- 172.21.15.158:0/133899701 >> v1:172.21.15.158:6789/0 conn(0x7f2a8c0cc9e0 legacy=0x7f2a8c06ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.809+0000 7f2a8b59e700 1 -- 172.21.15.158:0/133899701 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c15dcf0 con 0x7f2a8c0700a0 2023-10-16T11:21:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a8b59e700 1 --2- 172.21.15.158:0/133899701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c0700a0 0x7f2a8c0ce4b0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0715d0 tx=0x7f2a8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7556a3a940cccc1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a89d9b700 1 -- 172.21.15.158:0/133899701 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400e070 con 0x7f2a8c0700a0 2023-10-16T11:21:46.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a89d9b700 1 -- 172.21.15.158:0/133899701 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8400bb20 con 0x7f2a8c0700a0 2023-10-16T11:21:46.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a89d9b700 1 -- 172.21.15.158:0/133899701 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400be90 con 0x7f2a8c0700a0 2023-10-16T11:21:46.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 -- 172.21.15.158:0/133899701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c0700a0 msgr2=0x7f2a8c0ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:46.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 --2- 172.21.15.158:0/133899701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c0700a0 0x7f2a8c0ce4b0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0715d0 tx=0x7f2a8400b330 comp rx=0 tx=0).stop 2023-10-16T11:21:46.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 -- 172.21.15.158:0/133899701 shutdown_connections 2023-10-16T11:21:46.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 --2- 172.21.15.158:0/133899701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c0700a0 0x7f2a8c0ce4b0 unknown :-1 s=CLOSED pgs=3240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 --2- 172.21.15.158:0/133899701 >> v2:172.21.15.172:3300/0 conn(0x7f2a8c15d910 0x7f2a8c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 -- 172.21.15.158:0/133899701 >> 172.21.15.158:0/133899701 conn(0x7f2a8c0b1e90 msgr2=0x7f2a8c0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:46.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 -- 172.21.15.158:0/133899701 shutdown_connections 2023-10-16T11:21:46.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.810+0000 7f2a91090700 1 -- 172.21.15.158:0/133899701 wait complete. 2023-10-16T11:21:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.812+0000 7f2a91090700 1 Processor -- start 2023-10-16T11:21:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.812+0000 7f2a91090700 1 -- start start 2023-10-16T11:21:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8c0700a0 0x7f2a8c151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a8c0cc9e0 0x7f2a8c0b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:46.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c15d910 0x7f2a8c0b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c15e8a0 con 0x7f2a8c15d910 2023-10-16T11:21:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0b5860 con 0x7f2a8c0cc9e0 2023-10-16T11:21:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c15f570 con 0x7f2a8c0700a0 2023-10-16T11:21:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c15d910 0x7f2a8c0b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:46.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c15d910 0x7f2a8c0b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43420/0 (socket says 172.21.15.158:43420) 2023-10-16T11:21:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 -- 172.21.15.158:0/2587101409 learned_addr learned my addr 172.21.15.158:0/2587101409 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8ad9d700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8c0700a0 0x7f2a8c151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8a59c700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a8c0cc9e0 0x7f2a8c0b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 -- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8c0700a0 msgr2=0x7f2a8c151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:46.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8c0700a0 0x7f2a8c151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 -- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a8c0cc9e0 msgr2=0x7f2a8c0b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:46.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a8c0cc9e0 0x7f2a8c0b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:46.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 -- 172.21.15.158:0/2587101409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84009a70 con 0x7f2a8c15d910 2023-10-16T11:21:46.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a8b59e700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c15d910 0x7f2a8c0b4920 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400b370 tx=0x7f2a84004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a737fe700 1 -- 172.21.15.158:0/2587101409 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84004880 con 0x7f2a8c15d910 2023-10-16T11:21:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a737fe700 1 -- 172.21.15.158:0/2587101409 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a84020070 con 0x7f2a8c15d910 2023-10-16T11:21:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c0b5580 con 0x7f2a8c15d910 2023-10-16T11:21:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a737fe700 1 -- 172.21.15.158:0/2587101409 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840276c0 con 0x7f2a8c15d910 2023-10-16T11:21:46.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.813+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c15ed70 con 0x7f2a8c15d910 2023-10-16T11:21:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.814+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c0681b0 con 0x7f2a8c15d910 2023-10-16T11:21:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.815+0000 7f2a737fe700 1 -- 172.21.15.158:0/2587101409 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2a84012430 con 0x7f2a8c15d910 2023-10-16T11:21:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.815+0000 7f2a737fe700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2a7806af90 0x7f2a7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.815+0000 7f2a737fe700 1 -- 172.21.15.158:0/2587101409 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f2a84022070 con 0x7f2a8c15d910 2023-10-16T11:21:46.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.815+0000 7f2a8ad9d700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2a7806af90 0x7f2a7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.816+0000 7f2a8ad9d700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2a7806af90 0x7f2a7806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0b5bd0 tx=0x7f2a80008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:46.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:46.819+0000 7f2a737fe700 1 -- 172.21.15.158:0/2587101409 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a84056960 con 0x7f2a8c15d910 2023-10-16T11:21:47.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.013+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2a8c0b6320 con 0x7f2a8c15d910 2023-10-16T11:21:47.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.017+0000 7f2a737fe700 1 -- 172.21.15.158:0/2587101409 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2a84059f80 con 0x7f2a8c15d910 2023-10-16T11:21:47.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2a7806af90 msgr2=0x7f2a7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2a7806af90 0x7f2a7806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0b5bd0 tx=0x7f2a80008040 comp rx=0 tx=0).stop 2023-10-16T11:21:47.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c15d910 msgr2=0x7f2a8c0b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c15d910 0x7f2a8c0b4920 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400b370 tx=0x7f2a84004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 shutdown_connections 2023-10-16T11:21:47.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f2a7806af90 0x7f2a7806d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a8c15d910 0x7f2a8c0b4920 unknown :-1 s=CLOSED pgs=3241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a8c0700a0 0x7f2a8c151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 --2- 172.21.15.158:0/2587101409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a8c0cc9e0 0x7f2a8c0b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 >> 172.21.15.158:0/2587101409 conn(0x7f2a8c0b1e90 msgr2=0x7f2a8c15b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:47.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 shutdown_connections 2023-10-16T11:21:47.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.019+0000 7f2a91090700 1 -- 172.21.15.158:0/2587101409 wait complete. 2023-10-16T11:21:47.033 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:21:47.050 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:21:47.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:21:47.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.163+0000 7fd2cbd2e700 1 Processor -- start 2023-10-16T11:21:47.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.163+0000 7fd2cbd2e700 1 -- start start 2023-10-16T11:21:47.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2cbd2e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2cbd2e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2c40cc590 0x7fd2c406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2cbd2e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40cbd30 con 0x7fd2c406ae10 2023-10-16T11:21:47.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2cbd2e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c406b280 con 0x7fd2c40cc590 2023-10-16T11:21:47.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2cbd2e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c40cc1c0 con 0x7fd2c415da40 2023-10-16T11:21:47.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2c9aca700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2c92c9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2c40cc590 0x7fd2c406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2c9aca700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:54258/0 (socket says 172.21.15.158:54258) 2023-10-16T11:21:47.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2c92c9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2c40cc590 0x7fd2c406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:44606/0 (socket says 172.21.15.158:44606) 2023-10-16T11:21:47.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2c9aca700 1 -- 172.21.15.158:0/2801257891 learned_addr learned my addr 172.21.15.158:0/2801257891 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:47.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.164+0000 7fd2ca2cb700 1 --1- 172.21.15.158:0/2801257891 >> v1:172.21.15.172:6789/0 conn(0x7fd2c406ae10 0x7fd2c40ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:47508/0 (socket says 172.21.15.158:47508) 2023-10-16T11:21:47.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.165+0000 7fd2c92c9700 1 -- 172.21.15.158:0/2801257891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 msgr2=0x7fd2c40caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:47.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.165+0000 7fd2c92c9700 1 --2- 172.21.15.158:0/2801257891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.165+0000 7fd2c92c9700 1 -- 172.21.15.158:0/2801257891 >> v1:172.21.15.172:6789/0 conn(0x7fd2c406ae10 legacy=0x7fd2c40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.165+0000 7fd2c92c9700 1 -- 172.21.15.158:0/2801257891 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c415de20 con 0x7fd2c40cc590 2023-10-16T11:21:47.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.165+0000 7fd2c92c9700 1 --2- 172.21.15.158:0/2801257891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c406a8d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009840 tx=0x7fd2b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc453328d53299c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2c8ac8700 1 -- 172.21.15.158:0/2801257891 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4011830 con 0x7fd2c40cc590 2023-10-16T11:21:47.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2c8ac8700 1 -- 172.21.15.158:0/2801257891 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4011990 con 0x7fd2c40cc590 2023-10-16T11:21:47.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2c8ac8700 1 -- 172.21.15.158:0/2801257891 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4011d20 con 0x7fd2c40cc590 2023-10-16T11:21:47.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2801257891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 msgr2=0x7fd2c406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2801257891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c406a8d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009840 tx=0x7fd2b4009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:47.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2801257891 shutdown_connections 2023-10-16T11:21:47.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2801257891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2801257891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c406a8d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2801257891 >> 172.21.15.158:0/2801257891 conn(0x7fd2c4099a10 msgr2=0x7fd2c40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:47.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2801257891 shutdown_connections 2023-10-16T11:21:47.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.166+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2801257891 wait complete. 2023-10-16T11:21:47.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.168+0000 7fd2cbd2e700 1 Processor -- start 2023-10-16T11:21:47.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.169+0000 7fd2cbd2e700 1 -- start start 2023-10-16T11:21:47.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2cbd2e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2c406ae10 0x7fd2c40c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2cbd2e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c40c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2cbd2e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2cbd2e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c40c4be0 con 0x7fd2c406ae10 2023-10-16T11:21:47.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2cbd2e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41f75a0 con 0x7fd2c40cc590 2023-10-16T11:21:47.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2cbd2e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41f66d0 con 0x7fd2c415da40 2023-10-16T11:21:47.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c40c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2ca2cb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c40c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:44610/0 (socket says 172.21.15.158:44610) 2023-10-16T11:21:47.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 -- 172.21.15.158:0/2032280685 learned_addr learned my addr 172.21.15.158:0/2032280685 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:47.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 -- 172.21.15.158:0/2032280685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 msgr2=0x7fd2c40ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 -- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2c406ae10 msgr2=0x7fd2c40c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2c406ae10 0x7fd2c40c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.170+0000 7fd2c92c9700 1 -- 172.21.15.158:0/2032280685 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40094a0 con 0x7fd2c40cc590 2023-10-16T11:21:47.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.171+0000 7fd2c92c9700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c40c46a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40090b0 tx=0x7fd2b4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.171+0000 7fd2ba7fc700 1 -- 172.21.15.158:0/2032280685 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40045d0 con 0x7fd2c40cc590 2023-10-16T11:21:47.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.171+0000 7fd2ba7fc700 1 -- 172.21.15.158:0/2032280685 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b401f070 con 0x7fd2c40cc590 2023-10-16T11:21:47.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.171+0000 7fd2ba7fc700 1 -- 172.21.15.158:0/2032280685 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40038a0 con 0x7fd2c40cc590 2023-10-16T11:21:47.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.171+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c41f6990 con 0x7fd2c40cc590 2023-10-16T11:21:47.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.171+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c41f6f50 con 0x7fd2c40cc590 2023-10-16T11:21:47.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.172+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c40681b0 con 0x7fd2c40cc590 2023-10-16T11:21:47.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.173+0000 7fd2ba7fc700 1 -- 172.21.15.158:0/2032280685 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd2b4003a00 con 0x7fd2c40cc590 2023-10-16T11:21:47.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.173+0000 7fd2ba7fc700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd2a806b020 0x7fd2a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.173+0000 7fd2ba7fc700 1 -- 172.21.15.158:0/2032280685 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fd2b4021070 con 0x7fd2c40cc590 2023-10-16T11:21:47.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.173+0000 7fd2c9aca700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd2a806b020 0x7fd2a806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.174+0000 7fd2c9aca700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd2a806b020 0x7fd2a806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006fd0 tx=0x7fd2c0006f20 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.178+0000 7fd2ba7fc700 1 -- 172.21.15.158:0/2032280685 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2b40560b0 con 0x7fd2c40cc590 2023-10-16T11:21:47.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.376+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd2c40b5810 con 0x7fd2c40cc590 2023-10-16T11:21:47.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.382+0000 7fd2ba7fc700 1 -- 172.21.15.158:0/2032280685 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v92) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd2b40596d0 con 0x7fd2c40cc590 2023-10-16T11:21:47.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd2a806b020 msgr2=0x7fd2a806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd2a806b020 0x7fd2a806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006fd0 tx=0x7fd2c0006f20 comp rx=0 tx=0).stop 2023-10-16T11:21:47.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 msgr2=0x7fd2c40c46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c40c46a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40090b0 tx=0x7fd2b4004750 comp rx=0 tx=0).stop 2023-10-16T11:21:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 shutdown_connections 2023-10-16T11:21:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2c415da40 0x7fd2c40ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd2a806b020 0x7fd2a806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2c406ae10 0x7fd2c40c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 --2- 172.21.15.158:0/2032280685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2c40cc590 0x7fd2c40c46a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 >> 172.21.15.158:0/2032280685 conn(0x7fd2c4099a10 msgr2=0x7fd2c40b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:47.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 shutdown_connections 2023-10-16T11:21:47.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.384+0000 7fd2cbd2e700 1 -- 172.21.15.158:0/2032280685 wait complete. 2023-10-16T11:21:47.397 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:21:47.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.559+0000 7f8eb234b700 1 Processor -- start 2023-10-16T11:21:47.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.559+0000 7f8eb234b700 1 -- start start 2023-10-16T11:21:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eb234b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac154a30 0x7f8eac15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eb234b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8eac06e3f0 0x7f8eac06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eb234b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac06eb60 con 0x7f8eac06d930 2023-10-16T11:21:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eb234b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac06ddc0 con 0x7f8eac06e3f0 2023-10-16T11:21:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eb234b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac15e1b0 con 0x7f8eac154a30 2023-10-16T11:21:47.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eab7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8eac06e3f0 0x7f8eac06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eb08e8700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8eac06d930 0x7f8eac0b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43618/0 (socket says 172.21.15.158:43618) 2023-10-16T11:21:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eb08e8700 1 -- 172.21.15.158:0/3283341433 learned_addr learned my addr 172.21.15.158:0/3283341433 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eabfff700 1 --2- 172.21.15.158:0/3283341433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac154a30 0x7f8eac15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eab7fe700 1 -- 172.21.15.158:0/3283341433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac154a30 msgr2=0x7f8eac15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eab7fe700 1 --2- 172.21.15.158:0/3283341433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac154a30 0x7f8eac15b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eab7fe700 1 -- 172.21.15.158:0/3283341433 >> v1:172.21.15.158:6789/0 conn(0x7f8eac06d930 legacy=0x7f8eac0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eab7fe700 1 -- 172.21.15.158:0/3283341433 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac15e490 con 0x7f8eac06e3f0 2023-10-16T11:21:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eab7fe700 1 --2- 172.21.15.158:0/3283341433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac06e3f0 0x7f8eac06d3f0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009840 tx=0x7f8ea0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f7a0c57edbd96c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eaaffd700 1 -- 172.21.15.158:0/3283341433 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0011830 con 0x7f8eac06e3f0 2023-10-16T11:21:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eaaffd700 1 -- 172.21.15.158:0/3283341433 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea0011990 con 0x7f8eac06e3f0 2023-10-16T11:21:47.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.560+0000 7f8eaaffd700 1 -- 172.21.15.158:0/3283341433 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0011d20 con 0x7f8eac06e3f0 2023-10-16T11:21:47.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 -- 172.21.15.158:0/3283341433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac06e3f0 msgr2=0x7f8eac06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 --2- 172.21.15.158:0/3283341433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac06e3f0 0x7f8eac06d3f0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009840 tx=0x7f8ea0009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:47.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 -- 172.21.15.158:0/3283341433 shutdown_connections 2023-10-16T11:21:47.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 --2- 172.21.15.158:0/3283341433 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac154a30 0x7f8eac15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 --2- 172.21.15.158:0/3283341433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac06e3f0 0x7f8eac06d3f0 unknown :-1 s=CLOSED pgs=3243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 -- 172.21.15.158:0/3283341433 >> 172.21.15.158:0/3283341433 conn(0x7f8eac0b1bd0 msgr2=0x7f8eac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:47.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 -- 172.21.15.158:0/3283341433 shutdown_connections 2023-10-16T11:21:47.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.561+0000 7f8eb234b700 1 -- 172.21.15.158:0/3283341433 wait complete. 2023-10-16T11:21:47.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.562+0000 7f8eb234b700 1 Processor -- start 2023-10-16T11:21:47.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.563+0000 7f8eb234b700 1 -- start start 2023-10-16T11:21:47.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eac06d930 0x7f8eac157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac06e3f0 0x7f8eac144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac154a30 0x7f8eac159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac158db0 con 0x7f8eac154a30 2023-10-16T11:21:47.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac145210 con 0x7f8eac06d930 2023-10-16T11:21:47.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac159f70 con 0x7f8eac06e3f0 2023-10-16T11:21:47.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac154a30 0x7f8eac159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac154a30 0x7f8eac159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43448/0 (socket says 172.21.15.158:43448) 2023-10-16T11:21:47.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 -- 172.21.15.158:0/624924703 learned_addr learned my addr 172.21.15.158:0/624924703 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:47.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eab7fe700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac06e3f0 0x7f8eac144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eabfff700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eac06d930 0x7f8eac157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 -- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac06e3f0 msgr2=0x7f8eac144cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac06e3f0 0x7f8eac144cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 -- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eac06d930 msgr2=0x7f8eac157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eac06d930 0x7f8eac157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 -- 172.21.15.158:0/624924703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00094a0 con 0x7f8eac154a30 2023-10-16T11:21:47.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb08e8700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac154a30 0x7f8eac159a30 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400cbc0 tx=0x7f8ea400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8ea8ff9700 1 -- 172.21.15.158:0/624924703 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea400ab50 con 0x7f8eac154a30 2023-10-16T11:21:47.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8ea8ff9700 1 -- 172.21.15.158:0/624924703 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea400acb0 con 0x7f8eac154a30 2023-10-16T11:21:47.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8ea8ff9700 1 -- 172.21.15.158:0/624924703 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4012600 con 0x7f8eac154a30 2023-10-16T11:21:47.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac158130 con 0x7f8eac154a30 2023-10-16T11:21:47.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.564+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac1586f0 con 0x7f8eac154a30 2023-10-16T11:21:47.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.565+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac158ef0 con 0x7f8eac154a30 2023-10-16T11:21:47.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.566+0000 7f8ea8ff9700 1 -- 172.21.15.158:0/624924703 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8ea401b3b0 con 0x7f8eac154a30 2023-10-16T11:21:47.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.566+0000 7f8ea8ff9700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f8e9806b020 0x7f8e9806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.566+0000 7f8eabfff700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f8e9806b020 0x7f8e9806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.566+0000 7f8ea8ff9700 1 -- 172.21.15.158:0/624924703 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8ea40896c0 con 0x7f8eac154a30 2023-10-16T11:21:47.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.567+0000 7f8eabfff700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f8e9806b020 0x7f8e9806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00a950 tx=0x7f8e9c008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.571+0000 7f8ea8ff9700 1 -- 172.21.15.158:0/624924703 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8eac158ef0 con 0x7f8eac154a30 2023-10-16T11:21:47.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.766+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f8eac158ef0 con 0x7f8eac154a30 2023-10-16T11:21:47.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.769+0000 7f8ea8ff9700 1 -- 172.21.15.158:0/624924703 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v93) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8eac158ef0 con 0x7f8eac154a30 2023-10-16T11:21:47.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f8e9806b020 msgr2=0x7f8e9806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f8e9806b020 0x7f8e9806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00a950 tx=0x7f8e9c008040 comp rx=0 tx=0).stop 2023-10-16T11:21:47.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac154a30 msgr2=0x7f8eac159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac154a30 0x7f8eac159a30 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400cbc0 tx=0x7f8ea400cf80 comp rx=0 tx=0).stop 2023-10-16T11:21:47.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 shutdown_connections 2023-10-16T11:21:47.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f8e9806b020 0x7f8e9806d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8eac154a30 0x7f8eac159a30 unknown :-1 s=CLOSED pgs=3244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eac06d930 0x7f8eac157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 --2- 172.21.15.158:0/624924703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eac06e3f0 0x7f8eac144cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 >> 172.21.15.158:0/624924703 conn(0x7f8eac0b1bd0 msgr2=0x7f8eac0b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:47.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 shutdown_connections 2023-10-16T11:21:47.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.772+0000 7f8eb234b700 1 -- 172.21.15.158:0/624924703 wait complete. 2023-10-16T11:21:47.786 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T11:21:47.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.906+0000 7efd1ae8e700 1 Processor -- start 2023-10-16T11:21:47.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.906+0000 7efd1ae8e700 1 -- start start 2023-10-16T11:21:47.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd1ae8e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406b0a0 0x7efd1415f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd1ae8e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd1406d820 0x7efd140ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd1ae8e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd14072700 con 0x7efd1406e2b0 2023-10-16T11:21:47.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd1ae8e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd14071910 con 0x7efd1406d820 2023-10-16T11:21:47.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd1ae8e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1406b480 con 0x7efd1406b0a0 2023-10-16T11:21:47.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd18c2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406b0a0 0x7efd1415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd13fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd1406d820 0x7efd140ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd18c2a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406b0a0 0x7efd1415f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:54286/0 (socket says 172.21.15.158:54286) 2023-10-16T11:21:47.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd18c2a700 1 -- 172.21.15.158:0/2398004151 learned_addr learned my addr 172.21.15.158:0/2398004151 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:47.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd13fff700 1 -- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406b0a0 msgr2=0x7efd1415f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:47.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd13fff700 1 --2- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406b0a0 0x7efd1415f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd13fff700 1 -- 172.21.15.158:0/2398004151 >> v1:172.21.15.172:6789/0 conn(0x7efd1406e2b0 legacy=0x7efd1415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd13fff700 1 -- 172.21.15.158:0/2398004151 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd140cca90 con 0x7efd1406d820 2023-10-16T11:21:47.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.907+0000 7efd18c2a700 1 --2- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406b0a0 0x7efd1415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:47.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd13fff700 1 --2- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd140ca900 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efd00009870 tx=0x7efd00009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5b41f8fb23b9dcb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd137fe700 1 -- 172.21.15.158:0/2398004151 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd00011830 con 0x7efd1406d820 2023-10-16T11:21:47.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd137fe700 1 -- 172.21.15.158:0/2398004151 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd00011990 con 0x7efd1406d820 2023-10-16T11:21:47.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd137fe700 1 -- 172.21.15.158:0/2398004151 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd00011d20 con 0x7efd1406d820 2023-10-16T11:21:47.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 -- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 msgr2=0x7efd140ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd140ca900 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efd00009870 tx=0x7efd00009b80 comp rx=0 tx=0).stop 2023-10-16T11:21:47.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 -- 172.21.15.158:0/2398004151 shutdown_connections 2023-10-16T11:21:47.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406b0a0 0x7efd1415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/2398004151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd140ca900 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 -- 172.21.15.158:0/2398004151 >> 172.21.15.158:0/2398004151 conn(0x7efd140d02b0 msgr2=0x7efd140d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:47.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 -- 172.21.15.158:0/2398004151 shutdown_connections 2023-10-16T11:21:47.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.908+0000 7efd1ae8e700 1 -- 172.21.15.158:0/2398004151 wait complete. 2023-10-16T11:21:47.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.910+0000 7efd1ae8e700 1 Processor -- start 2023-10-16T11:21:47.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.910+0000 7efd1ae8e700 1 -- start start 2023-10-16T11:21:47.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd1406b0a0 0x7efd140c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd141f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406e2b0 0x7efd141ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd141f26c0 con 0x7efd1406b0a0 2023-10-16T11:21:47.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd141f1fe0 con 0x7efd1406d820 2023-10-16T11:21:47.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1415ccf0 con 0x7efd1406e2b0 2023-10-16T11:21:47.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd1406b0a0 0x7efd140c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd1406b0a0 0x7efd140c4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43458/0 (socket says 172.21.15.158:43458) 2023-10-16T11:21:47.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 -- 172.21.15.158:0/3452617222 learned_addr learned my addr 172.21.15.158:0/3452617222 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:47.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd13fff700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd141f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1942b700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406e2b0 0x7efd141ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 -- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406e2b0 msgr2=0x7efd141ee4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406e2b0 0x7efd141ee4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 -- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 msgr2=0x7efd141f2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:47.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd141f2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:47.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 -- 172.21.15.158:0/3452617222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd08009550 con 0x7efd1406b0a0 2023-10-16T11:21:47.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd13fff700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd141f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:47.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1942b700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406e2b0 0x7efd141ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:47.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd18c2a700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd1406b0a0 0x7efd140c4aa0 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7efd0800e9e0 tx=0x7efd0800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd117fa700 1 -- 172.21.15.158:0/3452617222 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0800c850 con 0x7efd1406b0a0 2023-10-16T11:21:47.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd117fa700 1 -- 172.21.15.158:0/3452617222 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0800c9b0 con 0x7efd1406b0a0 2023-10-16T11:21:47.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd000094a0 con 0x7efd1406b0a0 2023-10-16T11:21:47.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd117fa700 1 -- 172.21.15.158:0/3452617222 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0800cd20 con 0x7efd1406b0a0 2023-10-16T11:21:47.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.911+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1415cf90 con 0x7efd1406b0a0 2023-10-16T11:21:47.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.912+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd140681b0 con 0x7efd1406b0a0 2023-10-16T11:21:47.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.913+0000 7efd117fa700 1 -- 172.21.15.158:0/3452617222 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd08013470 con 0x7efd1406b0a0 2023-10-16T11:21:47.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.913+0000 7efd117fa700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7efcf806b020 0x7efcf806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:47.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.913+0000 7efd13fff700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7efcf806b020 0x7efcf806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:47.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.913+0000 7efd117fa700 1 -- 172.21.15.158:0/3452617222 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7efd08089b90 con 0x7efd1406b0a0 2023-10-16T11:21:47.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.914+0000 7efd13fff700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7efcf806b020 0x7efcf806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd000090b0 tx=0x7efd0001c040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:47.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:47.918+0000 7efd117fa700 1 -- 172.21.15.158:0/3452617222 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd08054800 con 0x7efd1406b0a0 2023-10-16T11:21:48.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.193+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7efd140d10e0 con 0x7efd1406b0a0 2023-10-16T11:21:48.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.657+0000 7efd117fa700 1 -- 172.21.15.158:0/3452617222 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3252) v1 ==== 155+0+0 (secure 0 0 0) 0x7efd080170e0 con 0x7efd1406b0a0 2023-10-16T11:21:48.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7efcf806b020 msgr2=0x7efcf806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7efcf806b020 0x7efcf806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd000090b0 tx=0x7efd0001c040 comp rx=0 tx=0).stop 2023-10-16T11:21:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd1406b0a0 msgr2=0x7efd140c4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd1406b0a0 0x7efd140c4aa0 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7efd0800e9e0 tx=0x7efd0800eda0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 shutdown_connections 2023-10-16T11:21:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd1406e2b0 0x7efd141ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7efcf806b020 0x7efcf806d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd1406b0a0 0x7efd140c4aa0 unknown :-1 s=CLOSED pgs=3245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 --2- 172.21.15.158:0/3452617222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd1406d820 0x7efd141f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 >> 172.21.15.158:0/3452617222 conn(0x7efd140d02b0 msgr2=0x7efd140b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 shutdown_connections 2023-10-16T11:21:48.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.660+0000 7efd1ae8e700 1 -- 172.21.15.158:0/3452617222 wait complete. 2023-10-16T11:21:48.675 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-16T11:21:48.676 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol and group arguments to snap-schedule cli. 2023-10-16T11:21:48.676 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T11:21:48.677 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:48.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.798+0000 7f13a7481700 1 Processor -- start 2023-10-16T11:21:48.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.798+0000 7f13a7481700 1 -- start start 2023-10-16T11:21:48.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.799+0000 7f13a7481700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 0x7f13a006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.799+0000 7f13a7481700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a0161270 con 0x7f13a006ab40 2023-10-16T11:21:48.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.799+0000 7f13a7481700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a0161510 con 0x7f13a00729b0 2023-10-16T11:21:48.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.799+0000 7f13a7481700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a01617f0 con 0x7f13a006e5d0 2023-10-16T11:21:48.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.799+0000 7f13a521d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f13a006ab40 0x7f13a00722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43642/0 (socket says 172.21.15.158:43642) 2023-10-16T11:21:48.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.799+0000 7f13a521d700 1 -- 172.21.15.158:0/1703670161 learned_addr learned my addr 172.21.15.158:0/1703670161 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:48.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.799+0000 7f13a4a1c700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 0x7f13a006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354800627 0 0) 0x7f13a0161270 con 0x7f13a006ab40 2023-10-16T11:21:48.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f138c0035b0 con 0x7f13a006ab40 2023-10-16T11:21:48.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144325364 0 0) 0x7f13a0161510 con 0x7f13a00729b0 2023-10-16T11:21:48.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a0161270 con 0x7f13a00729b0 2023-10-16T11:21:48.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515475136 0 0) 0x7f138c0035b0 con 0x7f13a006ab40 2023-10-16T11:21:48.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 msgr2=0x7f13a006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:48.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 0x7f13a006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 >> v1:172.21.15.172:6789/0 conn(0x7f13a00729b0 legacy=0x7f13a015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:48.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0161ad0 con 0x7f13a006ab40 2023-10-16T11:21:48.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1390001670 con 0x7f13a006ab40 2023-10-16T11:21:48.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 >> v1:172.21.15.158:6789/0 conn(0x7f13a006ab40 legacy=0x7f13a00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:48.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f138c003bc0 0x7f138c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c004850 0x7f138c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 0x7f138c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a01617f0 con 0x7f138c003bc0 2023-10-16T11:21:48.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c00bac0 con 0x7f138c004850 2023-10-16T11:21:48.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c00bc50 con 0x7f13a006e5d0 2023-10-16T11:21:48.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a521d700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f138c003bc0 0x7f138c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a4a1c700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c004850 0x7f138c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a5a1e700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 0x7f138c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a521d700 1 -- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 msgr2=0x7f138c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:48.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a521d700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 0x7f138c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a521d700 1 -- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c004850 msgr2=0x7f138c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:48.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a521d700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c004850 0x7f138c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a521d700 1 -- 172.21.15.158:0/1703670161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0035b0 con 0x7f138c003bc0 2023-10-16T11:21:48.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a5a1e700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a006e5d0 0x7f138c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:48.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.800+0000 7f13a521d700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f138c003bc0 0x7f138c003fc0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f139000c210 tx=0x7f138c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4b9a68b1df208d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:48.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139000cd70 con 0x7f138c003bc0 2023-10-16T11:21:48.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1390001af0 con 0x7f138c003bc0 2023-10-16T11:21:48.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f1397fff700 1 -- 172.21.15.158:0/1703670161 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1390005ba0 con 0x7f138c003bc0 2023-10-16T11:21:48.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f13a7481700 1 -- 172.21.15.158:0/1703670161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f138c003bc0 msgr2=0x7f138c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:48.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f13a7481700 1 --2- 172.21.15.158:0/1703670161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f138c003bc0 0x7f138c003fc0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f139000c210 tx=0x7f138c005880 comp rx=0 tx=0).stop 2023-10-16T11:21:48.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f13a5a1e700 1 -- 172.21.15.158:0/1703670161 reap_dead start 2023-10-16T11:21:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f13a7481700 1 -- 172.21.15.158:0/1703670161 shutdown_connections 2023-10-16T11:21:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f13a7481700 1 -- 172.21.15.158:0/1703670161 >> 172.21.15.158:0/1703670161 conn(0x7f13a00d02b0 msgr2=0x7f13a00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f13a7481700 1 -- 172.21.15.158:0/1703670161 shutdown_connections 2023-10-16T11:21:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.801+0000 7f13a7481700 1 -- 172.21.15.158:0/1703670161 wait complete. 2023-10-16T11:21:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.802+0000 7f13a7481700 1 Processor -- start 2023-10-16T11:21:48.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a7481700 1 -- start start 2023-10-16T11:21:48.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a7481700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c003bc0 0x7f13a00c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a7481700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a00b3f20 0x7f13a01f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a7481700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a00b1b70 0x7f13a00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a7481700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a00b2540 con 0x7f13a00b1b70 2023-10-16T11:21:48.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a7481700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a00b55c0 con 0x7f138c003bc0 2023-10-16T11:21:48.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a7481700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a00b4400 con 0x7f13a00b3f20 2023-10-16T11:21:48.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a5a1e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a00b1b70 0x7f13a00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a5a1e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a00b1b70 0x7f13a00b2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43482/0 (socket says 172.21.15.158:43482) 2023-10-16T11:21:48.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.803+0000 7f13a5a1e700 1 -- 172.21.15.158:0/725837927 learned_addr learned my addr 172.21.15.158:0/725837927 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:48.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a4a1c700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a00b3f20 0x7f13a01f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a521d700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c003bc0 0x7f13a00c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a5a1e700 1 -- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a00b3f20 msgr2=0x7f13a01f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:48.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a5a1e700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a00b3f20 0x7f13a01f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a5a1e700 1 -- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c003bc0 msgr2=0x7f13a00c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:48.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a5a1e700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c003bc0 0x7f13a00c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:48.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a5a1e700 1 -- 172.21.15.158:0/725837927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00b46e0 con 0x7f13a00b1b70 2023-10-16T11:21:48.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a5a1e700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a00b1b70 0x7f13a00b2000 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f1398001a90 tx=0x7f139800c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:48.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f1395ffb700 1 -- 172.21.15.158:0/725837927 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398003c10 con 0x7f13a00b1b70 2023-10-16T11:21:48.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f1395ffb700 1 -- 172.21.15.158:0/725837927 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1398003d90 con 0x7f13a00b1b70 2023-10-16T11:21:48.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a00b49c0 con 0x7f13a00b1b70 2023-10-16T11:21:48.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f1395ffb700 1 -- 172.21.15.158:0/725837927 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139800c9a0 con 0x7f13a00b1b70 2023-10-16T11:21:48.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.804+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a00c9c80 con 0x7f13a00b1b70 2023-10-16T11:21:48.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.805+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a00c4ab0 con 0x7f13a00b1b70 2023-10-16T11:21:48.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.805+0000 7f1395ffb700 1 -- 172.21.15.158:0/725837927 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f139800cb00 con 0x7f13a00b1b70 2023-10-16T11:21:48.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.806+0000 7f1395ffb700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f138406b020 0x7f138406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:48.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.806+0000 7f1395ffb700 1 -- 172.21.15.158:0/725837927 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f1398089b60 con 0x7f13a00b1b70 2023-10-16T11:21:48.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.806+0000 7f13a521d700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f138406b020 0x7f138406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:48.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.806+0000 7f13a521d700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f138406b020 0x7f138406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f13900018b0 tx=0x7f139000a000 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:48.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:48.810+0000 7f1395ffb700 1 -- 172.21.15.158:0/725837927 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13a00c4ab0 con 0x7f13a00b1b70 2023-10-16T11:21:49.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.002+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13a00c4ab0 con 0x7f13a00b1b70 2023-10-16T11:21:49.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.003+0000 7f1395ffb700 1 -- 172.21.15.158:0/725837927 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+15094 (secure 0 0 0) 0x7f13a00c4ab0 con 0x7f13a00b1b70 2023-10-16T11:21:49.004 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:49.005 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":262,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:46.653858+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6834/3986459990":"2023-10-17T11:21:43.631537+0000","172.21.15.172:6832/3986459990":"2023-10-17T11:21:43.631537+0000","172.21.15.158:6832/1196176990":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/2368789841":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/1642562285":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/1105433381":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/2407391765":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/595141618":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1477193149":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/1384560093":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/385046482":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6833/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6832/3125181137":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/4272085425":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1184494648":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/2564736313":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/2593348127":"2023-10-17T11:21:34.118742+0000","172.21.15.158:0/1448525844":"2023-10-17T11:21:34.118742+0000","172.21.15.158:6833/1196176990":"2023-10-17T11:21:42.627995+0000","172.21.15.158:0/3641755142":"2023-10-17T11:21:34.118742+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:49.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f138406b020 msgr2=0x7f138406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f138406b020 0x7f138406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f13900018b0 tx=0x7f139000a000 comp rx=0 tx=0).stop 2023-10-16T11:21:49.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a00b1b70 msgr2=0x7f13a00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a00b1b70 0x7f13a00b2000 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f1398001a90 tx=0x7f139800c0e0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 shutdown_connections 2023-10-16T11:21:49.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f138406b020 0x7f138406d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13a00b1b70 0x7f13a00b2000 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f138c003bc0 0x7f13a00c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 --2- 172.21.15.158:0/725837927 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13a00b3f20 0x7f13a01f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 >> 172.21.15.158:0/725837927 conn(0x7f13a00d02b0 msgr2=0x7f13a0073c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:49.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 shutdown_connections 2023-10-16T11:21:49.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.005+0000 7f13a7481700 1 -- 172.21.15.158:0/725837927 wait complete. 2023-10-16T11:21:49.019 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:21:49.020 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:21:49.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.142+0000 7f37211f7700 1 Processor -- start 2023-10-16T11:21:49.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.142+0000 7f37211f7700 1 -- start start 2023-10-16T11:21:49.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f37211f7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f371c15da40 0x7f371c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f37211f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c0cc590 0x7f371c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f37211f7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371c0cbd30 con 0x7f371c06ae10 2023-10-16T11:21:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f37211f7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f371c06b2c0 con 0x7f371c15da40 2023-10-16T11:21:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f37211f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371c0cc1c0 con 0x7f371c0cc590 2023-10-16T11:21:49.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371ad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f371c15da40 0x7f371c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c0cc590 0x7f371c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371ad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f371c15da40 0x7f371c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:44666/0 (socket says 172.21.15.158:44666) 2023-10-16T11:21:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371ad9d700 1 -- 172.21.15.158:0/949983359 learned_addr learned my addr 172.21.15.158:0/949983359 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371b59e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f371c06ae10 0x7f371c0ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:47570/0 (socket says 172.21.15.158:47570) 2023-10-16T11:21:49.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c0cc590 0x7f371c06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:54320/0 (socket says 172.21.15.158:54320) 2023-10-16T11:21:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371ad9d700 1 -- 172.21.15.158:0/949983359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c0cc590 msgr2=0x7f371c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371ad9d700 1 --2- 172.21.15.158:0/949983359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c0cc590 0x7f371c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.143+0000 7f371ad9d700 1 -- 172.21.15.158:0/949983359 >> v1:172.21.15.172:6789/0 conn(0x7f371c06ae10 legacy=0x7f371c0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f371ad9d700 1 -- 172.21.15.158:0/949983359 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c15de20 con 0x7f371c15da40 2023-10-16T11:21:49.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f3719d9b700 1 -- 172.21.15.158:0/949983359 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363066607 0 0) 0x7f371c0cbd30 con 0x7f371c06ae10 2023-10-16T11:21:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f371ad9d700 1 --2- 172.21.15.158:0/949983359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c15da40 0x7f371c0caa20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f37100098f0 tx=0x7f3710009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13ceb8e6025f7a74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f3719d9b700 1 -- 172.21.15.158:0/949983359 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710011830 con 0x7f371c15da40 2023-10-16T11:21:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f3719d9b700 1 -- 172.21.15.158:0/949983359 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3710011990 con 0x7f371c15da40 2023-10-16T11:21:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f3719d9b700 1 -- 172.21.15.158:0/949983359 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710011d20 con 0x7f371c15da40 2023-10-16T11:21:49.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f37211f7700 1 -- 172.21.15.158:0/949983359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c15da40 msgr2=0x7f371c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f37211f7700 1 --2- 172.21.15.158:0/949983359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c15da40 0x7f371c0caa20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f37100098f0 tx=0x7f3710009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f37211f7700 1 -- 172.21.15.158:0/949983359 shutdown_connections 2023-10-16T11:21:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f37211f7700 1 --2- 172.21.15.158:0/949983359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c15da40 0x7f371c0caa20 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f37211f7700 1 --2- 172.21.15.158:0/949983359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c0cc590 0x7f371c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.144+0000 7f37211f7700 1 -- 172.21.15.158:0/949983359 >> 172.21.15.158:0/949983359 conn(0x7f371c099a10 msgr2=0x7f371c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.145+0000 7f37211f7700 1 -- 172.21.15.158:0/949983359 shutdown_connections 2023-10-16T11:21:49.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.145+0000 7f37211f7700 1 -- 172.21.15.158:0/949983359 wait complete. 2023-10-16T11:21:49.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.146+0000 7f37211f7700 1 Processor -- start 2023-10-16T11:21:49.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.146+0000 7f37211f7700 1 -- start start 2023-10-16T11:21:49.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f37211f7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c06ae10 0x7f371c1e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f37211f7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f371c0cc590 0x7f371c0c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f37211f7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c15da40 0x7f371c1f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f37211f7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c159cc0 con 0x7f371c0cc590 2023-10-16T11:21:49.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f37211f7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c1593d0 con 0x7f371c06ae10 2023-10-16T11:21:49.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f37211f7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371c159670 con 0x7f371c15da40 2023-10-16T11:21:49.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f371c0cc590 0x7f371c0c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f371c0cc590 0x7f371c0c6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43494/0 (socket says 172.21.15.158:43494) 2023-10-16T11:21:49.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 -- 172.21.15.158:0/3319804162 learned_addr learned my addr 172.21.15.158:0/3319804162 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:49.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371ad9d700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c06ae10 0x7f371c1e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371b59e700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c15da40 0x7f371c1f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 -- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c15da40 msgr2=0x7f371c1f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c15da40 0x7f371c1f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 -- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c06ae10 msgr2=0x7f371c1e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c06ae10 0x7f371c1e7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 -- 172.21.15.158:0/3319804162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710009550 con 0x7f371c0cc590 2023-10-16T11:21:49.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371ad9d700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c06ae10 0x7f371c1e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:49.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371b59e700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c15da40 0x7f371c1f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:49.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.147+0000 7f371a59c700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f371c0cc590 0x7f371c0c6780 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e980 tx=0x7f370c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:49.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.148+0000 7f37037fe700 1 -- 172.21.15.158:0/3319804162 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c009710 con 0x7f371c0cc590 2023-10-16T11:21:49.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.148+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371c159950 con 0x7f371c0cc590 2023-10-16T11:21:49.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.148+0000 7f37037fe700 1 -- 172.21.15.158:0/3319804162 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370c00c850 con 0x7f371c0cc590 2023-10-16T11:21:49.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.148+0000 7f37037fe700 1 -- 172.21.15.158:0/3319804162 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c00cbe0 con 0x7f371c0cc590 2023-10-16T11:21:49.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.148+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371c0c71b0 con 0x7f371c0cc590 2023-10-16T11:21:49.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.148+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371c074a90 con 0x7f371c0cc590 2023-10-16T11:21:49.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.149+0000 7f37037fe700 1 -- 172.21.15.158:0/3319804162 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f370c00cd40 con 0x7f371c0cc590 2023-10-16T11:21:49.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.149+0000 7f37037fe700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f370806b020 0x7f370806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.149+0000 7f37037fe700 1 -- 172.21.15.158:0/3319804162 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f370c089b50 con 0x7f371c0cc590 2023-10-16T11:21:49.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.149+0000 7f371ad9d700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f370806b020 0x7f370806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.150+0000 7f371ad9d700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f370806b020 0x7f370806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3710014040 tx=0x7f3710008030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:49.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.154+0000 7f37037fe700 1 -- 172.21.15.158:0/3319804162 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f370c054890 con 0x7f371c0cc590 2023-10-16T11:21:49.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.345+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f371c0681b0 con 0x7f371c0cc590 2023-10-16T11:21:49.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.345+0000 7f37037fe700 1 -- 172.21.15.158:0/3319804162 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v262) v1 ==== 93+0+10 (secure 0 0 0) 0x7f370c089020 con 0x7f371c0cc590 2023-10-16T11:21:49.347 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:21:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f370806b020 msgr2=0x7f370806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f370806b020 0x7f370806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3710014040 tx=0x7f3710008030 comp rx=0 tx=0).stop 2023-10-16T11:21:49.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f371c0cc590 msgr2=0x7f371c0c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f371c0cc590 0x7f371c0c6780 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e980 tx=0x7f370c00ed40 comp rx=0 tx=0).stop 2023-10-16T11:21:49.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 shutdown_connections 2023-10-16T11:21:49.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f371c15da40 0x7f371c1f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f371c06ae10 0x7f371c1e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f370806b020 0x7f370806d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 --2- 172.21.15.158:0/3319804162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f371c0cc590 0x7f371c0c6780 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 >> 172.21.15.158:0/3319804162 conn(0x7f371c099a10 msgr2=0x7f371c0cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 shutdown_connections 2023-10-16T11:21:49.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.347+0000 7f37211f7700 1 -- 172.21.15.158:0/3319804162 wait complete. 2023-10-16T11:21:49.362 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments' 2023-10-16T11:21:49.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.481+0000 7f925e092700 1 Processor -- start 2023-10-16T11:21:49.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.481+0000 7f925e092700 1 -- start start 2023-10-16T11:21:49.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f925e092700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f925806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f925e092700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92580729b0 0x7f9258071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f925e092700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9258071e50 con 0x7f925806ab40 2023-10-16T11:21:49.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f925e092700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9258161270 con 0x7f92580729b0 2023-10-16T11:21:49.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f925e092700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9258161510 con 0x7f925806e5d0 2023-10-16T11:21:49.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92580729b0 0x7f9258071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92580729b0 0x7f9258071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43504/0 (socket says 172.21.15.158:43504) 2023-10-16T11:21:49.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 -- 172.21.15.158:0/176870403 learned_addr learned my addr 172.21.15.158:0/176870403 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:49.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9256ffd700 1 --2- 172.21.15.158:0/176870403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f925806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 -- 172.21.15.158:0/176870403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 msgr2=0x7f925806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:49.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 --2- 172.21.15.158:0/176870403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f925806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 -- 172.21.15.158:0/176870403 >> v1:172.21.15.172:6789/0 conn(0x7f925806ab40 legacy=0x7f92580722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:49.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 -- 172.21.15.158:0/176870403 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92581617f0 con 0x7f92580729b0 2023-10-16T11:21:49.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9256ffd700 1 --2- 172.21.15.158:0/176870403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f925806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:49.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.482+0000 7f9257fff700 1 --2- 172.21.15.158:0/176870403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92580729b0 0x7f9258071910 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270aa045b9168fef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f92567fc700 1 -- 172.21.15.158:0/176870403 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925000e070 con 0x7f92580729b0 2023-10-16T11:21:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f92567fc700 1 -- 172.21.15.158:0/176870403 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250003a20 con 0x7f92580729b0 2023-10-16T11:21:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f92567fc700 1 -- 172.21.15.158:0/176870403 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250003dd0 con 0x7f92580729b0 2023-10-16T11:21:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 -- 172.21.15.158:0/176870403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92580729b0 msgr2=0x7f9258071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 --2- 172.21.15.158:0/176870403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92580729b0 0x7f9258071910 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925000b330 comp rx=0 tx=0).stop 2023-10-16T11:21:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 -- 172.21.15.158:0/176870403 shutdown_connections 2023-10-16T11:21:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 --2- 172.21.15.158:0/176870403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92580729b0 0x7f9258071910 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 --2- 172.21.15.158:0/176870403 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f925806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 -- 172.21.15.158:0/176870403 >> 172.21.15.158:0/176870403 conn(0x7f92580d02b0 msgr2=0x7f92580d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:49.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 -- 172.21.15.158:0/176870403 shutdown_connections 2023-10-16T11:21:49.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.483+0000 7f925e092700 1 -- 172.21.15.158:0/176870403 wait complete. 2023-10-16T11:21:49.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.484+0000 7f925e092700 1 Processor -- start 2023-10-16T11:21:49.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f925e092700 1 -- start start 2023-10-16T11:21:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f925e092700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f925806ab40 0x7f925814b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f925e092700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f92580c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f925e092700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92580729b0 0x7f92580c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f925e092700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925814b9e0 con 0x7f925806ab40 2023-10-16T11:21:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f925e092700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92580c9db0 con 0x7f92580729b0 2023-10-16T11:21:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f925e092700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92580ca090 con 0x7f925806e5d0 2023-10-16T11:21:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f92577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f925806ab40 0x7f925814b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f92577fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f925806ab40 0x7f925814b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43506/0 (socket says 172.21.15.158:43506) 2023-10-16T11:21:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.485+0000 7f92577fe700 1 -- 172.21.15.158:0/48771900 learned_addr learned my addr 172.21.15.158:0/48771900 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:49.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f92577fe700 1 -- 172.21.15.158:0/48771900 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 msgr2=0x7f92580c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f92577fe700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f92580c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f92577fe700 1 -- 172.21.15.158:0/48771900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92580729b0 msgr2=0x7f92580c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f92577fe700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92580729b0 0x7f92580c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f92577fe700 1 -- 172.21.15.158:0/48771900 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9250009b30 con 0x7f925806ab40 2023-10-16T11:21:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f92577fe700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f925806ab40 0x7f925814b4a0 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f924c00c4d0 tx=0x7f924c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:49.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f923ffff700 1 -- 172.21.15.158:0/48771900 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c011790 con 0x7f925806ab40 2023-10-16T11:21:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f923ffff700 1 -- 172.21.15.158:0/48771900 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924c0118f0 con 0x7f925806ab40 2023-10-16T11:21:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92580ca370 con 0x7f925806ab40 2023-10-16T11:21:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f923ffff700 1 -- 172.21.15.158:0/48771900 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c011ca0 con 0x7f925806ab40 2023-10-16T11:21:49.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.486+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9258149380 con 0x7f925806ab40 2023-10-16T11:21:49.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.487+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92580681b0 con 0x7f925806ab40 2023-10-16T11:21:49.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.487+0000 7f923ffff700 1 -- 172.21.15.158:0/48771900 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f924c01c460 con 0x7f925806ab40 2023-10-16T11:21:49.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.488+0000 7f923ffff700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f924406b020 0x7f924406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:49.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.488+0000 7f9256ffd700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f924406b020 0x7f924406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:49.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.488+0000 7f923ffff700 1 -- 172.21.15.158:0/48771900 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f924c089960 con 0x7f925806ab40 2023-10-16T11:21:49.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.488+0000 7f9256ffd700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f924406b020 0x7f924406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f925814c040 tx=0x7f9248008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:49.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.492+0000 7f923ffff700 1 -- 172.21.15.158:0/48771900 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f924c0545d0 con 0x7f925806ab40 2023-10-16T11:21:49.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:49.766+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]} v 0) v1 -- 0x7f92580b6230 con 0x7f925806ab40 2023-10-16T11:21:50.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.664+0000 7f923ffff700 1 -- 172.21.15.158:0/48771900 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]}]=0 v3254) v1 ==== 201+0+0 (secure 0 0 0) 0x7f924c057bf0 con 0x7f925806ab40 2023-10-16T11:21:50.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f924406b020 msgr2=0x7f924406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:50.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f924406b020 0x7f924406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f925814c040 tx=0x7f9248008040 comp rx=0 tx=0).stop 2023-10-16T11:21:50.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f925806ab40 msgr2=0x7f925814b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:50.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f925806ab40 0x7f925814b4a0 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f924c00c4d0 tx=0x7f924c00c890 comp rx=0 tx=0).stop 2023-10-16T11:21:50.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 shutdown_connections 2023-10-16T11:21:50.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92580729b0 0x7f92580c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f924406b020 0x7f924406d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f925806ab40 0x7f925814b4a0 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 --2- 172.21.15.158:0/48771900 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f925806e5d0 0x7f92580c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 >> 172.21.15.158:0/48771900 conn(0x7f92580d02b0 msgr2=0x7f92580acde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:50.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 shutdown_connections 2023-10-16T11:21:50.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.666+0000 7f925e092700 1 -- 172.21.15.158:0/48771900 wait complete. 2023-10-16T11:21:50.683 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T11:21:50.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.809+0000 7fc57f5da700 1 Processor -- start 2023-10-16T11:21:50.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.810+0000 7fc57f5da700 1 -- start start 2023-10-16T11:21:50.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.810+0000 7fc57f5da700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 0x7fc57815f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.810+0000 7fc57f5da700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5780cf4c0 con 0x7fc57815b590 2023-10-16T11:21:50.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.810+0000 7fc57f5da700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57815f710 con 0x7fc5780cbaf0 2023-10-16T11:21:50.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.810+0000 7fc57f5da700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780b49e0 con 0x7fc5780ce510 2023-10-16T11:21:50.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.810+0000 7fc57d376700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc57815b590 0x7fc5780b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43690/0 (socket says 172.21.15.158:43690) 2023-10-16T11:21:50.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57d376700 1 -- 172.21.15.158:0/552408663 learned_addr learned my addr 172.21.15.158:0/552408663 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:50.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510732202 0 0) 0x7fc5780cf4c0 con 0x7fc57815b590 2023-10-16T11:21:50.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5640035b0 con 0x7fc57815b590 2023-10-16T11:21:50.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57cb75700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 0x7fc57815f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1114958839 0 0) 0x7fc5640035b0 con 0x7fc57815b590 2023-10-16T11:21:50.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 msgr2=0x7fc57815f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:50.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 0x7fc57815f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 >> v1:172.21.15.172:6789/0 conn(0x7fc5780cbaf0 legacy=0x7fc5780cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:50.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57815f970 con 0x7fc57815b590 2023-10-16T11:21:50.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc568001670 con 0x7fc57815b590 2023-10-16T11:21:50.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 >> v1:172.21.15.158:6789/0 conn(0x7fc57815b590 legacy=0x7fc5780b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:50.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564003a30 0x7fc564003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc564004850 0x7fc564004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 0x7fc564005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780b49e0 con 0x7fc564004850 2023-10-16T11:21:50.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400bb40 con 0x7fc564003a30 2023-10-16T11:21:50.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400bd00 con 0x7fc5780ce510 2023-10-16T11:21:50.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57cb75700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc564004850 0x7fc564004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57d376700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564003a30 0x7fc564003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57db77700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 0x7fc564005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57cb75700 1 -- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 msgr2=0x7fc564005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:50.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57cb75700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780ce510 0x7fc564005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57cb75700 1 -- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564003a30 msgr2=0x7fc564003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:50.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.811+0000 7fc57cb75700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564003a30 0x7fc564003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57cb75700 1 -- 172.21.15.158:0/552408663 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5640035b0 con 0x7fc564004850 2023-10-16T11:21:50.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57d376700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564003a30 0x7fc564003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:50.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57cb75700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc564004850 0x7fc564004cd0 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7fc574009b60 tx=0x7fc564005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197ae904c2c5463c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:50.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57400e6c0 con 0x7fc564004850 2023-10-16T11:21:50.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57400e820 con 0x7fc564004850 2023-10-16T11:21:50.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc56ffff700 1 -- 172.21.15.158:0/552408663 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57400eb90 con 0x7fc564004850 2023-10-16T11:21:50.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57f5da700 1 -- 172.21.15.158:0/552408663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc564004850 msgr2=0x7fc564004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:50.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57f5da700 1 --2- 172.21.15.158:0/552408663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc564004850 0x7fc564004cd0 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7fc574009b60 tx=0x7fc564005880 comp rx=0 tx=0).stop 2023-10-16T11:21:50.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57db77700 1 -- 172.21.15.158:0/552408663 reap_dead start 2023-10-16T11:21:50.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57f5da700 1 -- 172.21.15.158:0/552408663 shutdown_connections 2023-10-16T11:21:50.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57f5da700 1 -- 172.21.15.158:0/552408663 >> 172.21.15.158:0/552408663 conn(0x7fc57806a8d0 msgr2=0x7fc57806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:50.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57f5da700 1 -- 172.21.15.158:0/552408663 shutdown_connections 2023-10-16T11:21:50.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.812+0000 7fc57f5da700 1 -- 172.21.15.158:0/552408663 wait complete. 2023-10-16T11:21:50.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 Processor -- start 2023-10-16T11:21:50.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 -- start start 2023-10-16T11:21:50.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564004850 0x7fc5781565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780c8f50 0x7fc5780cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5781e7020 0x7fc5781e7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780ca100 con 0x7fc5781e7020 2023-10-16T11:21:50.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5781e79c0 con 0x7fc564004850 2023-10-16T11:21:50.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57f5da700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc578156b10 con 0x7fc5780c8f50 2023-10-16T11:21:50.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.814+0000 7fc57db77700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5781e7020 0x7fc5781e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5781e7020 0x7fc5781e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43530/0 (socket says 172.21.15.158:43530) 2023-10-16T11:21:50.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 -- 172.21.15.158:0/2600534033 learned_addr learned my addr 172.21.15.158:0/2600534033 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:50.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57cb75700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780c8f50 0x7fc5780cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57d376700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564004850 0x7fc5781565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 -- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780c8f50 msgr2=0x7fc5780cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:50.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780c8f50 0x7fc5780cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 -- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564004850 msgr2=0x7fc5781565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:50.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564004850 0x7fc5781565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:50.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 -- 172.21.15.158:0/2600534033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5780c9480 con 0x7fc5781e7020 2023-10-16T11:21:50.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57d376700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564004850 0x7fc5781565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:50.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57db77700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5781e7020 0x7fc5781e7480 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7fc5780cee80 tx=0x7fc57000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:50.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc56dffb700 1 -- 172.21.15.158:0/2600534033 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570003c10 con 0x7fc5781e7020 2023-10-16T11:21:50.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc56dffb700 1 -- 172.21.15.158:0/2600534033 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570003d70 con 0x7fc5781e7020 2023-10-16T11:21:50.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5780c9760 con 0x7fc5781e7020 2023-10-16T11:21:50.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc56dffb700 1 -- 172.21.15.158:0/2600534033 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570005310 con 0x7fc5781e7020 2023-10-16T11:21:50.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.815+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5780c9d20 con 0x7fc5781e7020 2023-10-16T11:21:50.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.816+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5780c9eb0 con 0x7fc5781e7020 2023-10-16T11:21:50.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.817+0000 7fc56dffb700 1 -- 172.21.15.158:0/2600534033 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc570005470 con 0x7fc5781e7020 2023-10-16T11:21:50.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.817+0000 7fc56dffb700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc55c06b020 0x7fc55c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:50.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.817+0000 7fc56dffb700 1 -- 172.21.15.158:0/2600534033 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fc570089a40 con 0x7fc5781e7020 2023-10-16T11:21:50.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.817+0000 7fc57d376700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc55c06b020 0x7fc55c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:50.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.817+0000 7fc57d376700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc55c06b020 0x7fc55c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc568001c80 tx=0x7fc568001150 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:50.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:50.822+0000 7fc56dffb700 1 -- 172.21.15.158:0/2600534033 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5780c9eb0 con 0x7fc5781e7020 2023-10-16T11:21:51.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.015+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc5780c9eb0 con 0x7fc5781e7020 2023-10-16T11:21:51.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.019+0000 7fc56dffb700 1 -- 172.21.15.158:0/2600534033 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc5780c9eb0 con 0x7fc5781e7020 2023-10-16T11:21:51.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc55c06b020 msgr2=0x7fc55c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc55c06b020 0x7fc55c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc568001c80 tx=0x7fc568001150 comp rx=0 tx=0).stop 2023-10-16T11:21:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5781e7020 msgr2=0x7fc5781e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5781e7020 0x7fc5781e7480 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7fc5780cee80 tx=0x7fc57000ee00 comp rx=0 tx=0).stop 2023-10-16T11:21:51.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 shutdown_connections 2023-10-16T11:21:51.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc55c06b020 0x7fc55c06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5781e7020 0x7fc5781e7480 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc564004850 0x7fc5781565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 --2- 172.21.15.158:0/2600534033 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5780c8f50 0x7fc5780cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 >> 172.21.15.158:0/2600534033 conn(0x7fc57806a8d0 msgr2=0x7fc5780b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:51.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 shutdown_connections 2023-10-16T11:21:51.026 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.021+0000 7fc57f5da700 1 -- 172.21.15.158:0/2600534033 wait complete. 2023-10-16T11:21:51.035 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:21:51.036 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:21:51.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:21:51.128 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:21:51.129 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:51.129 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:51.146 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:21:51.147 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:21:51.147 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:51.167 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:21:51.167 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:21:51.167 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:51.203 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:21:51.203 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:21:51.204 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:21:51.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:21:51.228 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T11:21:51.228 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:51.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.362+0000 7fc98bb80700 1 Processor -- start 2023-10-16T11:21:51.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.363+0000 7fc98bb80700 1 -- start start 2023-10-16T11:21:51.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.363+0000 7fc98bb80700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9840cdee0 0x7fc98415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.363+0000 7fc98bb80700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9840b6e10 0x7fc9841521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.363+0000 7fc98bb80700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98415c5d0 con 0x7fc9840ce2c0 2023-10-16T11:21:51.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.363+0000 7fc98bb80700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98415d2c0 con 0x7fc9840b6e10 2023-10-16T11:21:51.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.363+0000 7fc98bb80700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98415c8b0 con 0x7fc9840cdee0 2023-10-16T11:21:51.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9840b6e10 0x7fc9841521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9840b6e10 0x7fc9841521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43534/0 (socket says 172.21.15.158:43534) 2023-10-16T11:21:51.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98a11d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9840ce2c0 0x7fc9840cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43714/0 (socket says 172.21.15.158:43714) 2023-10-16T11:21:51.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 -- 172.21.15.158:0/4058616275 learned_addr learned my addr 172.21.15.158:0/4058616275 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:51.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98991c700 1 --2- 172.21.15.158:0/4058616275 >> v2:172.21.15.172:3300/0 conn(0x7fc9840cdee0 0x7fc98415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 -- 172.21.15.158:0/4058616275 >> v2:172.21.15.172:3300/0 conn(0x7fc9840cdee0 msgr2=0x7fc98415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:51.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 --2- 172.21.15.158:0/4058616275 >> v2:172.21.15.172:3300/0 conn(0x7fc9840cdee0 0x7fc98415bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 -- 172.21.15.158:0/4058616275 >> v1:172.21.15.158:6789/0 conn(0x7fc9840ce2c0 legacy=0x7fc9840cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98891a700 1 -- 172.21.15.158:0/4058616275 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235052764 0 0) 0x7fc98415c5d0 con 0x7fc9840ce2c0 2023-10-16T11:21:51.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 -- 172.21.15.158:0/4058616275 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98415cb90 con 0x7fc9840b6e10 2023-10-16T11:21:51.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98911b700 1 --2- 172.21.15.158:0/4058616275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840b6e10 0x7fc9841521a0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fc9840d2470 tx=0x7fc980009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e39582f74eb809 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:51.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98891a700 1 -- 172.21.15.158:0/4058616275 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000b870 con 0x7fc9840b6e10 2023-10-16T11:21:51.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98891a700 1 -- 172.21.15.158:0/4058616275 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98000b9d0 con 0x7fc9840b6e10 2023-10-16T11:21:51.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.364+0000 7fc98891a700 1 -- 172.21.15.158:0/4058616275 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000bd60 con 0x7fc9840b6e10 2023-10-16T11:21:51.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 -- 172.21.15.158:0/4058616275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840b6e10 msgr2=0x7fc9841521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 --2- 172.21.15.158:0/4058616275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840b6e10 0x7fc9841521a0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fc9840d2470 tx=0x7fc980009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 -- 172.21.15.158:0/4058616275 shutdown_connections 2023-10-16T11:21:51.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 --2- 172.21.15.158:0/4058616275 >> v2:172.21.15.172:3300/0 conn(0x7fc9840cdee0 0x7fc98415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 --2- 172.21.15.158:0/4058616275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840b6e10 0x7fc9841521a0 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 -- 172.21.15.158:0/4058616275 >> 172.21.15.158:0/4058616275 conn(0x7fc98406a8d0 msgr2=0x7fc98406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:51.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 -- 172.21.15.158:0/4058616275 shutdown_connections 2023-10-16T11:21:51.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.365+0000 7fc98bb80700 1 -- 172.21.15.158:0/4058616275 wait complete. 2023-10-16T11:21:51.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.366+0000 7fc98bb80700 1 Processor -- start 2023-10-16T11:21:51.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.366+0000 7fc98bb80700 1 -- start start 2023-10-16T11:21:51.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98bb80700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9840b6e10 0x7fc984150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98bb80700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9840cdee0 0x7fc98415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98bb80700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840ce2c0 0x7fc9840c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98bb80700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840d1ad0 con 0x7fc9840ce2c0 2023-10-16T11:21:51.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98bb80700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc984153260 con 0x7fc9840cdee0 2023-10-16T11:21:51.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98bb80700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840d17f0 con 0x7fc9840b6e10 2023-10-16T11:21:51.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840ce2c0 0x7fc9840c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840ce2c0 0x7fc9840c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43544/0 (socket says 172.21.15.158:43544) 2023-10-16T11:21:51.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 -- 172.21.15.158:0/3968857574 learned_addr learned my addr 172.21.15.158:0/3968857574 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:51.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98991c700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9840b6e10 0x7fc984150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98911b700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9840cdee0 0x7fc98415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 -- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9840b6e10 msgr2=0x7fc984150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9840b6e10 0x7fc984150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 -- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9840cdee0 msgr2=0x7fc98415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9840cdee0 0x7fc98415eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 -- 172.21.15.158:0/3968857574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980009550 con 0x7fc9840ce2c0 2023-10-16T11:21:51.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98a11d700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840ce2c0 0x7fc9840c6f10 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00cba0 tx=0x7fc97c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:51.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc97a7fc700 1 -- 172.21.15.158:0/3968857574 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00ab80 con 0x7fc9840ce2c0 2023-10-16T11:21:51.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc97a7fc700 1 -- 172.21.15.158:0/3968857574 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c00ad00 con 0x7fc9840ce2c0 2023-10-16T11:21:51.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc97a7fc700 1 -- 172.21.15.158:0/3968857574 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c007310 con 0x7fc9840ce2c0 2023-10-16T11:21:51.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.367+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc984152c10 con 0x7fc9840ce2c0 2023-10-16T11:21:51.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.368+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9840d1410 con 0x7fc9840ce2c0 2023-10-16T11:21:51.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.368+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc984152a70 con 0x7fc9840ce2c0 2023-10-16T11:21:51.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.369+0000 7fc97a7fc700 1 -- 172.21.15.158:0/3968857574 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc97c007470 con 0x7fc9840ce2c0 2023-10-16T11:21:51.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.369+0000 7fc97a7fc700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc96806b050 0x7fc96806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.369+0000 7fc97a7fc700 1 -- 172.21.15.158:0/3968857574 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fc97c089510 con 0x7fc9840ce2c0 2023-10-16T11:21:51.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.369+0000 7fc98991c700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc96806b050 0x7fc96806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.370+0000 7fc98991c700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc96806b050 0x7fc96806d510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc974008850 tx=0x7fc974008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:51.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.374+0000 7fc97a7fc700 1 -- 172.21.15.158:0/3968857574 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc984152a70 con 0x7fc9840ce2c0 2023-10-16T11:21:51.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.608+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc984152a70 con 0x7fc9840ce2c0 2023-10-16T11:21:51.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.608+0000 7fc97a7fc700 1 -- 172.21.15.158:0/3968857574 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+3714 (secure 0 0 0) 0x7fc984152a70 con 0x7fc9840ce2c0 2023-10-16T11:21:51.611 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:51.611 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":147,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/4196180523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":4196180523},{"type":"v1","addr":"172.21.15.172:6834","nonce":4196180523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":13221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/1010484950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":1010484950},{"type":"v1","addr":"172.21.15.158:6836","nonce":1010484950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":13224,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/54641557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":54641557},{"type":"v1","addr":"172.21.15.158:6837","nonce":54641557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":13239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/3388825539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":3388825539},{"type":"v1","addr":"172.21.15.172:6835","nonce":3388825539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[]} 2023-10-16T11:21:51.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc96806b050 msgr2=0x7fc96806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc96806b050 0x7fc96806d510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc974008850 tx=0x7fc974008040 comp rx=0 tx=0).stop 2023-10-16T11:21:51.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840ce2c0 msgr2=0x7fc9840c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840ce2c0 0x7fc9840c6f10 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00cba0 tx=0x7fc97c00cf60 comp rx=0 tx=0).stop 2023-10-16T11:21:51.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 shutdown_connections 2023-10-16T11:21:51.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc96806b050 0x7fc96806d510 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9840ce2c0 0x7fc9840c6f10 unknown :-1 s=CLOSED pgs=3257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9840b6e10 0x7fc984150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 --2- 172.21.15.158:0/3968857574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9840cdee0 0x7fc98415eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.610+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 >> 172.21.15.158:0/3968857574 conn(0x7fc98406a8d0 msgr2=0x7fc9840b5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:51.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.611+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 shutdown_connections 2023-10-16T11:21:51.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.611+0000 7fc98bb80700 1 -- 172.21.15.158:0/3968857574 wait complete. 2023-10-16T11:21:51.615 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 147 2023-10-16T11:21:51.624 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T11:21:51.625 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T11:21:51.625 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:21:51.625 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:51.626 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T11:21:51.626 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T11:21:51.626 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:21:51.626 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:51.627 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T11:21:51.627 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T11:21:51.627 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:21:51.627 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:51.628 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T11:21:51.628 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T11:21:51.628 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:21:51.629 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:21:51.634 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:21:51.634 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T11:21:51.635 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:21:51.635 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T11:21:51.641 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:21:51.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T11:21:51.642 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:21:51.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T11:21:51.677 INFO:tasks.ceph.mds.a:Started 2023-10-16T11:21:51.677 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:51.678 INFO:tasks.ceph.mds.c:Started 2023-10-16T11:21:51.678 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:51.684 INFO:tasks.ceph.mds.b:Started 2023-10-16T11:21:51.684 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:51.685 INFO:tasks.ceph.mds.d:Started 2023-10-16T11:21:51.685 DEBUG:teuthology.parallel:result is None 2023-10-16T11:21:51.686 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T11:21:51.756 INFO:tasks.ceph.mds.c.smithi158.stdout:starting mds.c at 2023-10-16T11:21:51.757 INFO:tasks.ceph.mds.a.smithi158.stdout:starting mds.a at 2023-10-16T11:21:51.769 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:21:51.770 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-10-16T11:21:51.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.814+0000 7f9f0b885700 1 Processor -- start 2023-10-16T11:21:51.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.814+0000 7f9f0b885700 1 -- start start 2023-10-16T11:21:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f0b885700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f0415d9a0 0x7f9f040b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f0b885700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0406d4d0 0x7f9f040b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f0b885700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f040cfc90 con 0x7f9f0415d5c0 2023-10-16T11:21:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f0b885700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f04161620 con 0x7f9f0415d9a0 2023-10-16T11:21:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f0b885700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04161900 con 0x7f9f0406d4d0 2023-10-16T11:21:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f0415d9a0 0x7f9f040b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f09e22700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0406d4d0 0x7f9f040b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f0415d9a0 0x7f9f040b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:44756/0 (socket says 172.21.15.158:44756) 2023-10-16T11:21:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 -- 172.21.15.158:0/534538860 learned_addr learned my addr 172.21.15.158:0/534538860 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 -- 172.21.15.158:0/534538860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0406d4d0 msgr2=0x7f9f040b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 --2- 172.21.15.158:0/534538860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0406d4d0 0x7f9f040b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 -- 172.21.15.158:0/534538860 >> v1:172.21.15.172:6789/0 conn(0x7f9f0415d5c0 legacy=0x7f9f0406bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 -- 172.21.15.158:0/534538860 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f04161be0 con 0x7f9f0415d9a0 2023-10-16T11:21:51.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f08e20700 1 --2- 172.21.15.158:0/534538860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d9a0 0x7f9f040b2f30 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80098f0 tx=0x7f9ef8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adfe65d0f9a62114 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.815+0000 7f9f03fff700 1 -- 172.21.15.158:0/534538860 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8011830 con 0x7f9f0415d9a0 2023-10-16T11:21:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f03fff700 1 -- 172.21.15.158:0/534538860 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8011990 con 0x7f9f0415d9a0 2023-10-16T11:21:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f03fff700 1 -- 172.21.15.158:0/534538860 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8011d20 con 0x7f9f0415d9a0 2023-10-16T11:21:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 -- 172.21.15.158:0/534538860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d9a0 msgr2=0x7f9f040b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 --2- 172.21.15.158:0/534538860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d9a0 0x7f9f040b2f30 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80098f0 tx=0x7f9ef8009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:51.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 -- 172.21.15.158:0/534538860 shutdown_connections 2023-10-16T11:21:51.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 --2- 172.21.15.158:0/534538860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0406d4d0 0x7f9f040b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 --2- 172.21.15.158:0/534538860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d9a0 0x7f9f040b2f30 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 -- 172.21.15.158:0/534538860 >> 172.21.15.158:0/534538860 conn(0x7f9f040cdf30 msgr2=0x7f9f040ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:51.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 -- 172.21.15.158:0/534538860 shutdown_connections 2023-10-16T11:21:51.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.816+0000 7f9f0b885700 1 -- 172.21.15.158:0/534538860 wait complete. 2023-10-16T11:21:51.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 Processor -- start 2023-10-16T11:21:51.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 -- start start 2023-10-16T11:21:51.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f0406d4d0 0x7f9f0414e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d5c0 0x7f9f040b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0415d9a0 0x7f9f040b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040c7c20 con 0x7f9f0406d4d0 2023-10-16T11:21:51.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040c7660 con 0x7f9f0415d5c0 2023-10-16T11:21:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f0b885700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040c7940 con 0x7f9f0415d9a0 2023-10-16T11:21:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.818+0000 7f9f09621700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f0406d4d0 0x7f9f0414e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f0406d4d0 0x7f9f0414e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43580/0 (socket says 172.21.15.158:43580) 2023-10-16T11:21:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 -- 172.21.15.158:0/4252044552 learned_addr learned my addr 172.21.15.158:0/4252044552 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:51.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f08e20700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d5c0 0x7f9f040b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09e22700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0415d9a0 0x7f9f040b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 -- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0415d9a0 msgr2=0x7f9f040b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0415d9a0 0x7f9f040b5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 -- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d5c0 msgr2=0x7f9f040b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d5c0 0x7f9f040b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 -- 172.21.15.158:0/4252044552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef8009550 con 0x7f9f0406d4d0 2023-10-16T11:21:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f09621700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f0406d4d0 0x7f9f0414e7d0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c4e0 tx=0x7f9efc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f01ffb700 1 -- 172.21.15.158:0/4252044552 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc0052d0 con 0x7f9f0406d4d0 2023-10-16T11:21:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f01ffb700 1 -- 172.21.15.158:0/4252044552 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc005430 con 0x7f9f0406d4d0 2023-10-16T11:21:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f040c6510 con 0x7f9f0406d4d0 2023-10-16T11:21:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f01ffb700 1 -- 172.21.15.158:0/4252044552 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc0038c0 con 0x7f9f0406d4d0 2023-10-16T11:21:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.819+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f040c6d10 con 0x7f9f0406d4d0 2023-10-16T11:21:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.820+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f040cdd30 con 0x7f9f0406d4d0 2023-10-16T11:21:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.821+0000 7f9f01ffb700 1 -- 172.21.15.158:0/4252044552 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9efc003a20 con 0x7f9f0406d4d0 2023-10-16T11:21:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.821+0000 7f9f01ffb700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f9ef006b020 0x7f9ef006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.821+0000 7f9f01ffb700 1 -- 172.21.15.158:0/4252044552 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f9efc088fa0 con 0x7f9f0406d4d0 2023-10-16T11:21:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.821+0000 7f9f08e20700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f9ef006b020 0x7f9ef006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.822+0000 7f9f08e20700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f9ef006b020 0x7f9ef006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9f040c4d80 tx=0x7f9ef8008030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:51.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:51.826+0000 7f9f01ffb700 1 -- 172.21.15.158:0/4252044552 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9efc0bbfe0 con 0x7f9f0406d4d0 2023-10-16T11:21:52.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.032+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9f040681b0 con 0x7f9f0406d4d0 2023-10-16T11:21:52.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.674+0000 7f9f01ffb700 1 -- 172.21.15.158:0/4252044552 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v264) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9efc017080 con 0x7f9f0406d4d0 2023-10-16T11:21:52.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.677+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f9ef006b020 msgr2=0x7f9ef006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:52.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.677+0000 7f9f0b885700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f9ef006b020 0x7f9ef006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9f040c4d80 tx=0x7f9ef8008030 comp rx=0 tx=0).stop 2023-10-16T11:21:52.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.677+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f0406d4d0 msgr2=0x7f9f0414e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:52.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.677+0000 7f9f0b885700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f0406d4d0 0x7f9f0414e7d0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c4e0 tx=0x7f9efc00c8a0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 shutdown_connections 2023-10-16T11:21:52.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f0415d9a0 0x7f9f040b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f9ef006b020 0x7f9ef006d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f0406d4d0 0x7f9f0414e7d0 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 --2- 172.21.15.158:0/4252044552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f0415d5c0 0x7f9f040b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 >> 172.21.15.158:0/4252044552 conn(0x7f9f040cdf30 msgr2=0x7f9f041606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:52.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 shutdown_connections 2023-10-16T11:21:52.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.678+0000 7f9f0b885700 1 -- 172.21.15.158:0/4252044552 wait complete. 2023-10-16T11:21:52.682 INFO:teuthology.orchestra.run.smithi158.stderr: removed all blocklist entries 2023-10-16T11:21:52.695 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T11:21:52.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.817+0000 7f0dbdcf9700 1 Processor -- start 2023-10-16T11:21:52.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.817+0000 7f0dbdcf9700 1 -- start start 2023-10-16T11:21:52.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0dbdcf9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 0x7f0db806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:52.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0dbdcf9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0db806e810 0x7f0db8072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:52.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0dbdcf9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0db80728a0 0x7f0db806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:52.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0dbdcf9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db8072cf0 con 0x7f0db806e810 2023-10-16T11:21:52.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0dbdcf9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db806f850 con 0x7f0db80728a0 2023-10-16T11:21:52.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0dbdcf9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80b4260 con 0x7f0db806f470 2023-10-16T11:21:52.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0db806e810 0x7f0db8072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:52.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0db806e810 0x7f0db8072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43592/0 (socket says 172.21.15.158:43592) 2023-10-16T11:21:52.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 -- 172.21.15.158:0/2371836489 learned_addr learned my addr 172.21.15.158:0/2371836489 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:52.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 -- 172.21.15.158:0/2371836489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 msgr2=0x7f0db806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:52.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 --2- 172.21.15.158:0/2371836489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 0x7f0db806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 -- 172.21.15.158:0/2371836489 >> v2:172.21.15.172:3300/0 conn(0x7f0db80728a0 msgr2=0x7f0db806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:52.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 --2- 172.21.15.158:0/2371836489 >> v2:172.21.15.172:3300/0 conn(0x7f0db80728a0 0x7f0db806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 -- 172.21.15.158:0/2371836489 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80b4540 con 0x7f0db806e810 2023-10-16T11:21:52.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db6ffd700 1 --2- 172.21.15.158:0/2371836489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db8072360 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f0db80b2e20 tx=0x7f0dac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43553576113aa6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:52.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db67fc700 1 -- 172.21.15.158:0/2371836489 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00b870 con 0x7f0db806e810 2023-10-16T11:21:52.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db67fc700 1 -- 172.21.15.158:0/2371836489 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac00b9d0 con 0x7f0db806e810 2023-10-16T11:21:52.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.818+0000 7f0db67fc700 1 -- 172.21.15.158:0/2371836489 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00bd60 con 0x7f0db806e810 2023-10-16T11:21:52.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/2371836489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 msgr2=0x7f0db8072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:52.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/2371836489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db8072360 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f0db80b2e20 tx=0x7f0dac009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/2371836489 shutdown_connections 2023-10-16T11:21:52.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/2371836489 >> v2:172.21.15.172:3300/0 conn(0x7f0db80728a0 0x7f0db806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/2371836489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 0x7f0db806e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/2371836489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db8072360 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/2371836489 >> 172.21.15.158:0/2371836489 conn(0x7f0db80ccda0 msgr2=0x7f0db80cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:52.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/2371836489 shutdown_connections 2023-10-16T11:21:52.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.819+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/2371836489 wait complete. 2023-10-16T11:21:52.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.820+0000 7f0dbdcf9700 1 Processor -- start 2023-10-16T11:21:52.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.820+0000 7f0dbdcf9700 1 -- start start 2023-10-16T11:21:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0dbdcf9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db81e8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0dbdcf9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 0x7f0db81e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0dbdcf9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db80728a0 0x7f0db81e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0dbdcf9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80d0910 con 0x7f0db806e810 2023-10-16T11:21:52.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0dbdcf9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80cea10 con 0x7f0db80728a0 2023-10-16T11:21:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0dbdcf9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80cecf0 con 0x7f0db806f470 2023-10-16T11:21:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db81e8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db81e8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43596/0 (socket says 172.21.15.158:43596) 2023-10-16T11:21:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 -- 172.21.15.158:0/4195261218 learned_addr learned my addr 172.21.15.158:0/4195261218 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:52.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db6ffd700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 0x7f0db81e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db7fff700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db80728a0 0x7f0db81e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 -- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 msgr2=0x7f0db81e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 0x7f0db81e9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 -- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db80728a0 msgr2=0x7f0db81e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:52.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db80728a0 0x7f0db81e7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.821+0000 7f0db77fe700 1 -- 172.21.15.158:0/4195261218 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac009550 con 0x7f0db806e810 2023-10-16T11:21:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.822+0000 7f0db77fe700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db81e8af0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f0da800b520 tx=0x7f0da800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.822+0000 7f0d9ffff700 1 -- 172.21.15.158:0/4195261218 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8005aa0 con 0x7f0db806e810 2023-10-16T11:21:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.822+0000 7f0d9ffff700 1 -- 172.21.15.158:0/4195261218 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8005c00 con 0x7f0db806e810 2023-10-16T11:21:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.822+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db80cefd0 con 0x7f0db806e810 2023-10-16T11:21:52.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.822+0000 7f0d9ffff700 1 -- 172.21.15.158:0/4195261218 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8011630 con 0x7f0db806e810 2023-10-16T11:21:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.822+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db81e9840 con 0x7f0db806e810 2023-10-16T11:21:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.823+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db8074b40 con 0x7f0db806e810 2023-10-16T11:21:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.823+0000 7f0d9ffff700 1 -- 172.21.15.158:0/4195261218 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0da8011790 con 0x7f0db806e810 2023-10-16T11:21:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.823+0000 7f0d9ffff700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f0da406b020 0x7f0da406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:52.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.824+0000 7f0d9ffff700 1 -- 172.21.15.158:0/4195261218 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f0da8089380 con 0x7f0db806e810 2023-10-16T11:21:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.824+0000 7f0db6ffd700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f0da406b020 0x7f0da406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.824+0000 7f0db6ffd700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f0da406b020 0x7f0da406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0dac000c00 tx=0x7f0dac008030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:52.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:52.828+0000 7f0d9ffff700 1 -- 172.21.15.158:0/4195261218 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0da80543e0 con 0x7f0db806e810 2023-10-16T11:21:53.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.099+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0db806fdc0 con 0x7f0db806e810 2023-10-16T11:21:53.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.100+0000 7f0d9ffff700 1 -- 172.21.15.158:0/4195261218 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v56) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0da8057a00 con 0x7f0db806e810 2023-10-16T11:21:53.102 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:53.102 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:21:53.103 INFO:teuthology.orchestra.run.smithi158.stdout: "auth_dump": [ 2023-10-16T11:21:53.103 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.103 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.a", 2023-10-16T11:21:53.103 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lSxYVGhAAu0Z+n4lnEiY+eML5OfqV0w==", 2023-10-16T11:21:53.103 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.b", 2023-10-16T11:21:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lK2R1JhAAueQlv79Ms+NyrXgOk+RYqw==", 2023-10-16T11:21:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.c", 2023-10-16T11:21:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lM+iJIRAAa2KdaPoZFGoqbVProoRR7A==", 2023-10-16T11:21:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mds.d", 2023-10-16T11:21:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1l4xjELRAAyorlxFhXHfx+9wqZGjxO9A==", 2023-10-16T11:21:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.0", 2023-10-16T11:21:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCEBS1laueLIxAAQ/dgGs9QMBAKpKr6JR/lBQ==", 2023-10-16T11:21:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.1", 2023-10-16T11:21:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCHBS1l60+OIhAAp5P7Owczo5pWNir1LjkIew==", 2023-10-16T11:21:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.2", 2023-10-16T11:21:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCKBS1lZKT0IRAAzwOzlTCnGBvO5CVJy2o/Tw==", 2023-10-16T11:21:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.3", 2023-10-16T11:21:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCNBS1l9LS+IBAA85CI7rULzl7x/a5odRSM4w==", 2023-10-16T11:21:53.117 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.117 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.117 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.117 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.117 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.118 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.118 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.4", 2023-10-16T11:21:53.118 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCTBS1lh6vVJRAAplmK1Fg6V4ltFh16rTHxQQ==", 2023-10-16T11:21:53.118 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.119 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.119 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.119 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.119 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.119 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.120 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.120 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.5", 2023-10-16T11:21:53.120 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCWBS1ltfNRJRAAmiTbCpKNlo+RTRpB9TU3jA==", 2023-10-16T11:21:53.120 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.120 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.120 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.121 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.121 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.121 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.6", 2023-10-16T11:21:53.122 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCZBS1llHd6JBAAr3b6uzx2jgbLW30+lSzrqg==", 2023-10-16T11:21:53.122 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.122 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.122 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.122 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.123 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.123 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.123 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.123 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "osd.7", 2023-10-16T11:21:53.123 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCcBS1loZE8JBAAXh8Bcy3AfB0okcxymPO/gw==", 2023-10-16T11:21:53.124 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.124 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow profile osd", 2023-10-16T11:21:53.124 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile osd", 2023-10-16T11:21:53.124 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.124 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.124 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.125 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.125 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.0", 2023-10-16T11:21:53.125 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA==", 2023-10-16T11:21:53.125 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.125 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.126 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow r", 2023-10-16T11:21:53.126 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T11:21:53.126 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.127 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.127 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.1", 2023-10-16T11:21:53.127 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lswLLOBAASclF2JhL9xakQePhxdsnxQ==", 2023-10-16T11:21:53.127 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.127 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.129 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:21:53.129 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:21:53.129 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:21:53.129 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.130 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.130 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.130 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.2", 2023-10-16T11:21:53.130 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1loZdyNRAA8aRPgNb7339/S3HEJIocmQ==", 2023-10-16T11:21:53.131 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.131 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.131 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:21:53.131 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:21:53.131 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:21:53.132 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.132 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.132 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.132 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.3", 2023-10-16T11:21:53.132 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCDBS1lCATiAhAAdGmp/yUsVZvZK7XWBRZ32A==", 2023-10-16T11:21:53.132 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow", 2023-10-16T11:21:53.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow r", 2023-10-16T11:21:53.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow rw", 2023-10-16T11:21:53.133 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow rwx" 2023-10-16T11:21:53.133 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.134 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.134 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "client.admin", 2023-10-16T11:21:53.134 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCBBS1l3BVmOBAAIlw/4oMEU8KymoXvc2Pzdw==", 2023-10-16T11:21:53.134 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:21:53.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mgr": "allow *", 2023-10-16T11:21:53.135 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow *", 2023-10-16T11:21:53.135 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.135 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.136 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.136 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.x", 2023-10-16T11:21:53.136 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lC1CtFhAAVoD1YorwsElddzxORt1UhA==", 2023-10-16T11:21:53.136 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.137 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:21:53.137 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:21:53.137 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.137 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-16T11:21:53.138 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:53.138 INFO:teuthology.orchestra.run.smithi158.stdout: "entity": "mgr.y", 2023-10-16T11:21:53.138 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "AQCCBS1lAZVcEhAASV2lNvJSsKhnicxzhxI0uA==", 2023-10-16T11:21:53.138 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": { 2023-10-16T11:21:53.138 INFO:teuthology.orchestra.run.smithi158.stdout: "mds": "allow *", 2023-10-16T11:21:53.138 INFO:teuthology.orchestra.run.smithi158.stdout: "mon": "allow profile mgr", 2023-10-16T11:21:53.139 INFO:teuthology.orchestra.run.smithi158.stdout: "osd": "allow *" 2023-10-16T11:21:53.139 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.139 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:53.139 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-16T11:21:53.139 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:21:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f0da406b020 msgr2=0x7f0da406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f0da406b020 0x7f0da406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0dac000c00 tx=0x7f0dac008030 comp rx=0 tx=0).stop 2023-10-16T11:21:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 msgr2=0x7f0db81e8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db81e8af0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f0da800b520 tx=0x7f0da800b8e0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 shutdown_connections 2023-10-16T11:21:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db80728a0 0x7f0db81e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f0da406b020 0x7f0da406d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0db806e810 0x7f0db81e8af0 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 --2- 172.21.15.158:0/4195261218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0db806f470 0x7f0db81e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 >> 172.21.15.158:0/4195261218 conn(0x7f0db80ccda0 msgr2=0x7f0db80b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 shutdown_connections 2023-10-16T11:21:53.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.102+0000 7f0dbdcf9700 1 -- 172.21.15.158:0/4195261218 wait complete. 2023-10-16T11:21:53.144 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:53.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.263+0000 7fd9119ef700 1 Processor -- start 2023-10-16T11:21:53.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.263+0000 7fd9119ef700 1 -- start start 2023-10-16T11:21:53.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd9119ef700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd9119ef700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd90c161ed0 0x7fd90c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd9119ef700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0b65d0 con 0x7fd90c15d780 2023-10-16T11:21:53.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd9119ef700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c15e270 con 0x7fd90c161ed0 2023-10-16T11:21:53.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd9119ef700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0b9490 con 0x7fd90c0b9010 2023-10-16T11:21:53.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd90c161ed0 0x7fd90c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c161990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:54440/0 (socket says 172.21.15.158:54440) 2023-10-16T11:21:53.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd90c161ed0 0x7fd90c0ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:44788/0 (socket says 172.21.15.158:44788) 2023-10-16T11:21:53.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90a7fc700 1 -- 172.21.15.158:0/3431997246 learned_addr learned my addr 172.21.15.158:0/3431997246 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:53.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90a7fc700 1 -- 172.21.15.158:0/3431997246 >> v2:172.21.15.172:3300/0 conn(0x7fd90c161ed0 msgr2=0x7fd90c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:53.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90a7fc700 1 --2- 172.21.15.158:0/3431997246 >> v2:172.21.15.172:3300/0 conn(0x7fd90c161ed0 0x7fd90c0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90a7fc700 1 -- 172.21.15.158:0/3431997246 >> v1:172.21.15.172:6789/0 conn(0x7fd90c15d780 legacy=0x7fd90c15db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90a7fc700 1 -- 172.21.15.158:0/3431997246 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c161220 con 0x7fd90c0b9010 2023-10-16T11:21:53.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.264+0000 7fd90b7fe700 1 --2- 172.21.15.158:0/3431997246 >> v2:172.21.15.172:3300/0 conn(0x7fd90c161ed0 0x7fd90c0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:53.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd90a7fc700 1 --2- 172.21.15.158:0/3431997246 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c161990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098f0 tx=0x7fd900009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d708c4f5a55ffcd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd909ffb700 1 -- 172.21.15.158:0/3431997246 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900011830 con 0x7fd90c0b9010 2023-10-16T11:21:53.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd909ffb700 1 -- 172.21.15.158:0/3431997246 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd900011990 con 0x7fd90c0b9010 2023-10-16T11:21:53.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd909ffb700 1 -- 172.21.15.158:0/3431997246 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900011d20 con 0x7fd90c0b9010 2023-10-16T11:21:53.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 -- 172.21.15.158:0/3431997246 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 msgr2=0x7fd90c161990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 --2- 172.21.15.158:0/3431997246 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c161990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098f0 tx=0x7fd900009c00 comp rx=0 tx=0).stop 2023-10-16T11:21:53.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 -- 172.21.15.158:0/3431997246 shutdown_connections 2023-10-16T11:21:53.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 --2- 172.21.15.158:0/3431997246 >> v2:172.21.15.172:3300/0 conn(0x7fd90c161ed0 0x7fd90c0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 --2- 172.21.15.158:0/3431997246 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c161990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 -- 172.21.15.158:0/3431997246 >> 172.21.15.158:0/3431997246 conn(0x7fd90c066bd0 msgr2=0x7fd90c0d34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:53.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 -- 172.21.15.158:0/3431997246 shutdown_connections 2023-10-16T11:21:53.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.265+0000 7fd9119ef700 1 -- 172.21.15.158:0/3431997246 wait complete. 2023-10-16T11:21:53.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.267+0000 7fd9119ef700 1 Processor -- start 2023-10-16T11:21:53.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.267+0000 7fd9119ef700 1 -- start start 2023-10-16T11:21:53.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd9119ef700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c15b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd9119ef700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c15d780 0x7fd90c15ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd9119ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd90c0cb600 0x7fd90c160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd9119ef700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0cc780 con 0x7fd90c0cb600 2023-10-16T11:21:53.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd9119ef700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0cbb70 con 0x7fd90c15d780 2023-10-16T11:21:53.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd9119ef700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0cbe50 con 0x7fd90c0b9010 2023-10-16T11:21:53.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd90c0cb600 0x7fd90c160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd90c0cb600 0x7fd90c160440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43612/0 (socket says 172.21.15.158:43612) 2023-10-16T11:21:53.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 -- 172.21.15.158:0/1222950599 learned_addr learned my addr 172.21.15.158:0/1222950599 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90affd700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c15b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90a7fc700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c15d780 0x7fd90c15ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 -- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 msgr2=0x7fd90c15b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c15b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 -- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c15d780 msgr2=0x7fd90c15ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c15d780 0x7fd90c15ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.268+0000 7fd90b7fe700 1 -- 172.21.15.158:0/1222950599 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904009a70 con 0x7fd90c0cb600 2023-10-16T11:21:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.269+0000 7fd90b7fe700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd90c0cb600 0x7fd90c160440 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7fd90400efc0 tx=0x7fd90400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.269+0000 7fd8f3fff700 1 -- 172.21.15.158:0/1222950599 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400ca90 con 0x7fd90c0cb600 2023-10-16T11:21:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.269+0000 7fd8f3fff700 1 -- 172.21.15.158:0/1222950599 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90400f040 con 0x7fd90c0cb600 2023-10-16T11:21:53.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.269+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd900009550 con 0x7fd90c0cb600 2023-10-16T11:21:53.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.269+0000 7fd8f3fff700 1 -- 172.21.15.158:0/1222950599 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400ce30 con 0x7fd90c0cb600 2023-10-16T11:21:53.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.269+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c0cc410 con 0x7fd90c0cb600 2023-10-16T11:21:53.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.270+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c0681b0 con 0x7fd90c0cb600 2023-10-16T11:21:53.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.270+0000 7fd8f3fff700 1 -- 172.21.15.158:0/1222950599 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd904014480 con 0x7fd90c0cb600 2023-10-16T11:21:53.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.271+0000 7fd8f3fff700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd8f806b020 0x7fd8f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.271+0000 7fd8f3fff700 1 -- 172.21.15.158:0/1222950599 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5657+0+0 (secure 0 0 0) 0x7fd904089e60 con 0x7fd90c0cb600 2023-10-16T11:21:53.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.271+0000 7fd90affd700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd8f806b020 0x7fd8f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.271+0000 7fd90affd700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd8f806b020 0x7fd8f806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0b8530 tx=0x7fd8fc009040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.275+0000 7fd8f3fff700 1 -- 172.21.15.158:0/1222950599 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd904054d30 con 0x7fd90c0cb600 2023-10-16T11:21:53.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.468+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd90c0b5a00 con 0x7fd90c0cb600 2023-10-16T11:21:53.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.469+0000 7fd8f3fff700 1 -- 172.21.15.158:0/1222950599 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+13819 (secure 0 0 0) 0x7fd904009cd0 con 0x7fd90c0cb600 2023-10-16T11:21:53.471 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:53.471 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":266,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:52.681747+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:53.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd8f806b020 msgr2=0x7fd8f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd8f806b020 0x7fd8f806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0b8530 tx=0x7fd8fc009040 comp rx=0 tx=0).stop 2023-10-16T11:21:53.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd90c0cb600 msgr2=0x7fd90c160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd90c0cb600 0x7fd90c160440 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7fd90400efc0 tx=0x7fd90400c3a0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 shutdown_connections 2023-10-16T11:21:53.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd8f806b020 0x7fd8f806d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd90c0cb600 0x7fd90c160440 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd90c0b9010 0x7fd90c15b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 --2- 172.21.15.158:0/1222950599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c15d780 0x7fd90c15ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 >> 172.21.15.158:0/1222950599 conn(0x7fd90c066bd0 msgr2=0x7fd90c0b7280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:53.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 shutdown_connections 2023-10-16T11:21:53.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.471+0000 7fd9119ef700 1 -- 172.21.15.158:0/1222950599 wait complete. 2023-10-16T11:21:53.485 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:21:53.486 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:21:53.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.608+0000 7f17ce3e7700 1 Processor -- start 2023-10-16T11:21:53.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.608+0000 7f17ce3e7700 1 -- start start 2023-10-16T11:21:53.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17ce3e7700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17c815da40 0x7f17c80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17ce3e7700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80cbd30 con 0x7f17c806afe0 2023-10-16T11:21:53.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17ce3e7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80cc1c0 con 0x7f17c80cc590 2023-10-16T11:21:53.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17ce3e7700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c815de20 con 0x7f17c815da40 2023-10-16T11:21:53.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17c815da40 0x7f17c80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17cc984700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f17c806afe0 0x7f17c80afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43796/0 (socket says 172.21.15.158:43796) 2023-10-16T11:21:53.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c7fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17c815da40 0x7f17c80caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43618/0 (socket says 172.21.15.158:43618) 2023-10-16T11:21:53.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17cc984700 1 -- 172.21.15.158:0/3954564667 learned_addr learned my addr 172.21.15.158:0/3954564667 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:53.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c7fff700 1 -- 172.21.15.158:0/3954564667 >> v1:172.21.15.172:6789/0 conn(0x7f17c80cc590 legacy=0x7f17c806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:53.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c7fff700 1 -- 172.21.15.158:0/3954564667 >> v1:172.21.15.158:6789/0 conn(0x7f17c806afe0 legacy=0x7f17c80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c7fff700 1 -- 172.21.15.158:0/3954564667 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c80ce260 con 0x7f17c815da40 2023-10-16T11:21:53.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c6ffd700 1 -- 172.21.15.158:0/3954564667 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640796094 0 0) 0x7f17c80cbd30 con 0x7f17c806afe0 2023-10-16T11:21:53.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c7fff700 1 --2- 172.21.15.158:0/3954564667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c80caa20 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009840 tx=0x7f17bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d057eab136a7aa0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c6ffd700 1 -- 172.21.15.158:0/3954564667 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc004230 con 0x7f17c815da40 2023-10-16T11:21:53.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c6ffd700 1 -- 172.21.15.158:0/3954564667 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc004390 con 0x7f17c815da40 2023-10-16T11:21:53.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.609+0000 7f17c6ffd700 1 -- 172.21.15.158:0/3954564667 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0036a0 con 0x7f17c815da40 2023-10-16T11:21:53.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.610+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3954564667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 msgr2=0x7f17c80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.610+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3954564667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c80caa20 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009840 tx=0x7f17bc009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:53.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.610+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3954564667 shutdown_connections 2023-10-16T11:21:53.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.610+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3954564667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c80caa20 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.610+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3954564667 >> 172.21.15.158:0/3954564667 conn(0x7f17c8099a10 msgr2=0x7f17c80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:53.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.610+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3954564667 shutdown_connections 2023-10-16T11:21:53.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.610+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3954564667 wait complete. 2023-10-16T11:21:53.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.611+0000 7f17ce3e7700 1 Processor -- start 2023-10-16T11:21:53.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.611+0000 7f17ce3e7700 1 -- start start 2023-10-16T11:21:53.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17ce3e7700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17c806afe0 0x7f17c80ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17ce3e7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17c80cc590 0x7f17c81e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17ce3e7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c814a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17ce3e7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80ca930 con 0x7f17c815da40 2023-10-16T11:21:53.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17ce3e7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81e6510 con 0x7f17c80cc590 2023-10-16T11:21:53.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17ce3e7700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c814a900 con 0x7f17c806afe0 2023-10-16T11:21:53.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17cc984700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c814a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17cc984700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c814a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43628/0 (socket says 172.21.15.158:43628) 2023-10-16T11:21:53.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.612+0000 7f17cc984700 1 -- 172.21.15.158:0/3947973978 learned_addr learned my addr 172.21.15.158:0/3947973978 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:53.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17c77fe700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17c80cc590 0x7f17c81e5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17cc984700 1 -- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17c806afe0 msgr2=0x7f17c80ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17c7fff700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17c806afe0 0x7f17c80ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17cc984700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17c806afe0 0x7f17c80ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17cc984700 1 -- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17c80cc590 msgr2=0x7f17c81e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17cc984700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17c80cc590 0x7f17c81e5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17cc984700 1 -- 172.21.15.158:0/3947973978 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc0094a0 con 0x7f17c815da40 2023-10-16T11:21:53.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17c77fe700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17c80cc590 0x7f17c81e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17cc984700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c814a3c0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f17c000cc40 tx=0x7f17c000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17c4ff9700 1 -- 172.21.15.158:0/3947973978 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c000aba0 con 0x7f17c815da40 2023-10-16T11:21:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17c4ff9700 1 -- 172.21.15.158:0/3947973978 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c000ad00 con 0x7f17c815da40 2023-10-16T11:21:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c81e6740 con 0x7f17c815da40 2023-10-16T11:21:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17c4ff9700 1 -- 172.21.15.158:0/3947973978 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0007310 con 0x7f17c815da40 2023-10-16T11:21:53.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.613+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81e6d00 con 0x7f17c815da40 2023-10-16T11:21:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.614+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c80681b0 con 0x7f17c815da40 2023-10-16T11:21:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.614+0000 7f17c4ff9700 1 -- 172.21.15.158:0/3947973978 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f17c0007470 con 0x7f17c815da40 2023-10-16T11:21:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.615+0000 7f17c4ff9700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f17b406b020 0x7f17b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.615+0000 7f17c4ff9700 1 -- 172.21.15.158:0/3947973978 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f17c0089370 con 0x7f17c815da40 2023-10-16T11:21:53.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.615+0000 7f17c7fff700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f17b406b020 0x7f17b406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.615+0000 7f17c7fff700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f17b406b020 0x7f17b406d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009b90 tx=0x7f17bc013040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.619+0000 7f17c4ff9700 1 -- 172.21.15.158:0/3947973978 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17c00542c0 con 0x7f17c815da40 2023-10-16T11:21:53.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.814+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17c806f140 con 0x7f17c815da40 2023-10-16T11:21:53.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.815+0000 7f17c4ff9700 1 -- 172.21.15.158:0/3947973978 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v266) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17c00578e0 con 0x7f17c815da40 2023-10-16T11:21:53.817 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:21:53.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f17b406b020 msgr2=0x7f17b406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f17b406b020 0x7f17b406d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009b90 tx=0x7f17bc013040 comp rx=0 tx=0).stop 2023-10-16T11:21:53.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 msgr2=0x7f17c814a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c814a3c0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f17c000cc40 tx=0x7f17c000a060 comp rx=0 tx=0).stop 2023-10-16T11:21:53.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 shutdown_connections 2023-10-16T11:21:53.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f17b406b020 0x7f17b406d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17c815da40 0x7f17c814a3c0 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17c806afe0 0x7f17c80ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 --2- 172.21.15.158:0/3947973978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17c80cc590 0x7f17c81e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 >> 172.21.15.158:0/3947973978 conn(0x7f17c8099a10 msgr2=0x7f17c80b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:53.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 shutdown_connections 2023-10-16T11:21:53.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.817+0000 7f17ce3e7700 1 -- 172.21.15.158:0/3947973978 wait complete. 2023-10-16T11:21:53.831 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T11:21:53.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.951+0000 7f6fbf19d700 1 Processor -- start 2023-10-16T11:21:53.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.951+0000 7f6fbf19d700 1 -- start start 2023-10-16T11:21:53.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbf19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6fb80729b0 0x7f6fb806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbf19d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb8161270 con 0x7f6fb806ab40 2023-10-16T11:21:53.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbf19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb8161510 con 0x7f6fb806e5d0 2023-10-16T11:21:53.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbf19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb81617f0 con 0x7f6fb80729b0 2023-10-16T11:21:53.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbcf39700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6fb806ab40 0x7f6fb80722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43810/0 (socket says 172.21.15.158:43810) 2023-10-16T11:21:53.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbcf39700 1 -- 172.21.15.158:0/540683649 learned_addr learned my addr 172.21.15.158:0/540683649 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:53.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbd73a700 1 --2- 172.21.15.158:0/540683649 >> v2:172.21.15.172:3300/0 conn(0x7f6fb80729b0 0x7f6fb806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734504259 0 0) 0x7f6fb8161270 con 0x7f6fb806ab40 2023-10-16T11:21:53.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa80035b0 con 0x7f6fb806ab40 2023-10-16T11:21:53.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61351764 0 0) 0x7f6fa80035b0 con 0x7f6fb806ab40 2023-10-16T11:21:53.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 >> v2:172.21.15.172:3300/0 conn(0x7f6fb80729b0 msgr2=0x7f6fb806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:53.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 --2- 172.21.15.158:0/540683649 >> v2:172.21.15.172:3300/0 conn(0x7f6fb80729b0 0x7f6fb806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 >> v1:172.21.15.172:6789/0 conn(0x7f6fb806e5d0 legacy=0x7f6fb815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb8161ad0 con 0x7f6fb806ab40 2023-10-16T11:21:53.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f6fac001670 con 0x7f6fb806ab40 2023-10-16T11:21:53.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 >> v1:172.21.15.158:6789/0 conn(0x7f6fb806ab40 legacy=0x7f6fb80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fa8003a00 0x7f6fa8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8004850 0x7f6fa8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fa8005210 0x7f6fa800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81617f0 con 0x7f6fa8003a00 2023-10-16T11:21:53.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa800c0b0 con 0x7f6fa8004850 2023-10-16T11:21:53.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa800c270 con 0x7f6fa8005210 2023-10-16T11:21:53.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.952+0000 7f6fbcf39700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fa8003a00 0x7f6fa8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fb7fff700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8004850 0x7f6fa8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbd73a700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fa8005210 0x7f6fa800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbcf39700 1 -- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fa8005210 msgr2=0x7f6fa800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:53.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbcf39700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fa8005210 0x7f6fa800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbcf39700 1 -- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8004850 msgr2=0x7f6fa8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:53.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbcf39700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8004850 0x7f6fa8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbcf39700 1 -- 172.21.15.158:0/540683649 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa80035b0 con 0x7f6fa8003a00 2023-10-16T11:21:53.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbd73a700 1 -- 172.21.15.158:0/540683649 reap_dead start 2023-10-16T11:21:53.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbcf39700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fa8003a00 0x7f6fa8003e20 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f6fac001050 tx=0x7f6fa8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1afafd2c3c5960a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac00cd00 con 0x7f6fa8003a00 2023-10-16T11:21:53.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac00ce60 con 0x7f6fa8003a00 2023-10-16T11:21:53.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fb77fe700 1 -- 172.21.15.158:0/540683649 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac005ba0 con 0x7f6fa8003a00 2023-10-16T11:21:53.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbf19d700 1 -- 172.21.15.158:0/540683649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fa8003a00 msgr2=0x7f6fa8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.953+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fa8003a00 0x7f6fa8003e20 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f6fac001050 tx=0x7f6fa8005880 comp rx=0 tx=0).stop 2023-10-16T11:21:53.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.954+0000 7f6fbf19d700 1 -- 172.21.15.158:0/540683649 shutdown_connections 2023-10-16T11:21:53.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.954+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/540683649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fa8003a00 0x7f6fa8003e20 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.954+0000 7f6fbf19d700 1 -- 172.21.15.158:0/540683649 >> 172.21.15.158:0/540683649 conn(0x7f6fb80d02b0 msgr2=0x7f6fb80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:53.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.954+0000 7f6fbf19d700 1 -- 172.21.15.158:0/540683649 shutdown_connections 2023-10-16T11:21:53.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.954+0000 7f6fbf19d700 1 -- 172.21.15.158:0/540683649 wait complete. 2023-10-16T11:21:53.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.955+0000 7f6fbf19d700 1 Processor -- start 2023-10-16T11:21:53.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.955+0000 7f6fbf19d700 1 -- start start 2023-10-16T11:21:53.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbf19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8003a00 0x7f6fb8157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbf19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fb8160ca0 0x7f6fb80c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbf19d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fb80c68b0 0x7f6fb80b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbf19d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81f2d90 con 0x7f6fb80c68b0 2023-10-16T11:21:53.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbf19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb815dec0 con 0x7f6fa8003a00 2023-10-16T11:21:53.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbf19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb815e1a0 con 0x7f6fb8160ca0 2023-10-16T11:21:53.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fb80c68b0 0x7f6fb80b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fb80c68b0 0x7f6fb80b5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43650/0 (socket says 172.21.15.158:43650) 2023-10-16T11:21:53.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 -- 172.21.15.158:0/2260496980 learned_addr learned my addr 172.21.15.158:0/2260496980 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:53.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbcf39700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8003a00 0x7f6fb8157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fb7fff700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fb8160ca0 0x7f6fb80c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 -- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fb8160ca0 msgr2=0x7f6fb80c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fb8160ca0 0x7f6fb80c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 -- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8003a00 msgr2=0x7f6fb8157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:53.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8003a00 0x7f6fb8157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:53.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 -- 172.21.15.158:0/2260496980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb815e480 con 0x7f6fb80c68b0 2023-10-16T11:21:53.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbd73a700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fb80c68b0 0x7f6fb80b5030 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80d0210 tx=0x7f6fb0009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fb57fa700 1 -- 172.21.15.158:0/2260496980 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb000e070 con 0x7f6fb80c68b0 2023-10-16T11:21:53.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fb57fa700 1 -- 172.21.15.158:0/2260496980 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb0003cf0 con 0x7f6fb80c68b0 2023-10-16T11:21:53.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb815e760 con 0x7f6fb80c68b0 2023-10-16T11:21:53.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.956+0000 7f6fb57fa700 1 -- 172.21.15.158:0/2260496980 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0011730 con 0x7f6fb80c68b0 2023-10-16T11:21:53.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.957+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb81578b0 con 0x7f6fb80c68b0 2023-10-16T11:21:53.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.958+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb815ec80 con 0x7f6fb80c68b0 2023-10-16T11:21:53.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.958+0000 7f6fb57fa700 1 -- 172.21.15.158:0/2260496980 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6fb0011890 con 0x7f6fb80c68b0 2023-10-16T11:21:53.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.958+0000 7f6fb57fa700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6f9c06b020 0x7f6f9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:53.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.958+0000 7f6fb57fa700 1 -- 172.21.15.158:0/2260496980 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f6fb0089a00 con 0x7f6fb80c68b0 2023-10-16T11:21:53.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.958+0000 7f6fbcf39700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6f9c06b020 0x7f6f9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:53.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.959+0000 7f6fbcf39700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6f9c06b020 0x7f6f9c06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6fac000c00 tx=0x7f6fac00a000 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:53.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:53.963+0000 7f6fb57fa700 1 -- 172.21.15.158:0/2260496980 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fb815ec80 con 0x7f6fb80c68b0 2023-10-16T11:21:54.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.156+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6fb815ec80 con 0x7f6fb80c68b0 2023-10-16T11:21:54.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.157+0000 7f6fb57fa700 1 -- 172.21.15.158:0/2260496980 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v266) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6fb815ec80 con 0x7f6fb80c68b0 2023-10-16T11:21:54.159 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:54.159 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:21:54.159 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:21:54.159 INFO:teuthology.orchestra.run.smithi158.stdout: "poolnum": 1, 2023-10-16T11:21:54.160 INFO:teuthology.orchestra.run.smithi158.stdout: "poolname": ".mgr" 2023-10-16T11:21:54.160 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:21:54.160 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:21:54.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6f9c06b020 msgr2=0x7f6f9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6f9c06b020 0x7f6f9c06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6fac000c00 tx=0x7f6fac00a000 comp rx=0 tx=0).stop 2023-10-16T11:21:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fb80c68b0 msgr2=0x7f6fb80b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fb80c68b0 0x7f6fb80b5030 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80d0210 tx=0x7f6fb0009f70 comp rx=0 tx=0).stop 2023-10-16T11:21:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 shutdown_connections 2023-10-16T11:21:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6f9c06b020 0x7f6f9c06d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6fb80c68b0 0x7f6fb80b5030 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6fa8003a00 0x7f6fb8157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 --2- 172.21.15.158:0/2260496980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6fb8160ca0 0x7f6fb80c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 >> 172.21.15.158:0/2260496980 conn(0x7f6fb80d02b0 msgr2=0x7f6fb80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:54.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 shutdown_connections 2023-10-16T11:21:54.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.159+0000 7f6fbf19d700 1 -- 172.21.15.158:0/2260496980 wait complete. 2023-10-16T11:21:54.172 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T11:21:54.173 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T11:21:54.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.296+0000 7f59691e3700 1 Processor -- start 2023-10-16T11:21:54.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.296+0000 7f59691e3700 1 -- start start 2023-10-16T11:21:54.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f59691e3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f596415dad0 0x7f59640cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f59691e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59640ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f59691e3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59640d0ae0 con 0x7f59640cea20 2023-10-16T11:21:54.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f59691e3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59640d12d0 con 0x7f596415dad0 2023-10-16T11:21:54.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f59691e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59640cbdf0 con 0x7f596415deb0 2023-10-16T11:21:54.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5962d9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f596415dad0 0x7f59640cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f596359e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f59640cea20 0x7f59640c3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43834/0 (socket says 172.21.15.158:43834) 2023-10-16T11:21:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f596359e700 1 -- 172.21.15.158:0/3114294631 learned_addr learned my addr 172.21.15.158:0/3114294631 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5962d9d700 1 --2- 172.21.15.158:0/3114294631 >> v2:172.21.15.158:3300/0 conn(0x7f596415dad0 0x7f59640cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43654/0 (socket says 172.21.15.158:43654) 2023-10-16T11:21:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f596259c700 1 --2- 172.21.15.158:0/3114294631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59640ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5962d9d700 1 -- 172.21.15.158:0/3114294631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 msgr2=0x7f59640ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5962d9d700 1 --2- 172.21.15.158:0/3114294631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59640ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5962d9d700 1 -- 172.21.15.158:0/3114294631 >> v1:172.21.15.158:6789/0 conn(0x7f59640cea20 legacy=0x7f59640c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5961d9b700 1 -- 172.21.15.158:0/3114294631 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952640798 0 0) 0x7f59640d0ae0 con 0x7f59640cea20 2023-10-16T11:21:54.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5962d9d700 1 -- 172.21.15.158:0/3114294631 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59640cc280 con 0x7f596415dad0 2023-10-16T11:21:54.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f596259c700 1 --2- 172.21.15.158:0/3114294631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59640ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:54.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5962d9d700 1 --2- 172.21.15.158:0/3114294631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f596415dad0 0x7f59640cdfa0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f5954009840 tx=0x7f5954009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae64665c4969ac50 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:54.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5961d9b700 1 -- 172.21.15.158:0/3114294631 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954011830 con 0x7f596415dad0 2023-10-16T11:21:54.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5961d9b700 1 -- 172.21.15.158:0/3114294631 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954011990 con 0x7f596415dad0 2023-10-16T11:21:54.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.297+0000 7f5961d9b700 1 -- 172.21.15.158:0/3114294631 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954011d20 con 0x7f596415dad0 2023-10-16T11:21:54.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 -- 172.21.15.158:0/3114294631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f596415dad0 msgr2=0x7f59640cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 --2- 172.21.15.158:0/3114294631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f596415dad0 0x7f59640cdfa0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f5954009840 tx=0x7f5954009b50 comp rx=0 tx=0).stop 2023-10-16T11:21:54.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 -- 172.21.15.158:0/3114294631 shutdown_connections 2023-10-16T11:21:54.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 --2- 172.21.15.158:0/3114294631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f596415dad0 0x7f59640cdfa0 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 --2- 172.21.15.158:0/3114294631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59640ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 -- 172.21.15.158:0/3114294631 >> 172.21.15.158:0/3114294631 conn(0x7f5964099a10 msgr2=0x7f59640d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:54.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 -- 172.21.15.158:0/3114294631 shutdown_connections 2023-10-16T11:21:54.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.298+0000 7f59691e3700 1 -- 172.21.15.158:0/3114294631 wait complete. 2023-10-16T11:21:54.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.299+0000 7f59691e3700 1 Processor -- start 2023-10-16T11:21:54.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f59691e3700 1 -- start start 2023-10-16T11:21:54.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f59691e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59640cea20 0x7f5964156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f59691e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596415dad0 0x7f5964153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f59691e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59641e53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f59691e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59641566c0 con 0x7f59640cea20 2023-10-16T11:21:54.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f59691e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5964153ab0 con 0x7f596415dad0 2023-10-16T11:21:54.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f59691e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5964155ab0 con 0x7f596415deb0 2023-10-16T11:21:54.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f5962d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59640cea20 0x7f5964156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f5962d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59640cea20 0x7f5964156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43660/0 (socket says 172.21.15.158:43660) 2023-10-16T11:21:54.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.300+0000 7f5962d9d700 1 -- 172.21.15.158:0/1097286617 learned_addr learned my addr 172.21.15.158:0/1097286617 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:54.310 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f596259c700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596415dad0 0x7f5964153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f596359e700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59641e53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f5962d9d700 1 -- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 msgr2=0x7f59641e53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f5962d9d700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59641e53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f5962d9d700 1 -- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596415dad0 msgr2=0x7f5964153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f5962d9d700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596415dad0 0x7f5964153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f5962d9d700 1 -- 172.21.15.158:0/1097286617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59540094a0 con 0x7f59640cea20 2023-10-16T11:21:54.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f5962d9d700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59640cea20 0x7f5964156b50 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f5954000c00 tx=0x7f5954004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:54.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f594b7fe700 1 -- 172.21.15.158:0/1097286617 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540045f0 con 0x7f59640cea20 2023-10-16T11:21:54.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f594b7fe700 1 -- 172.21.15.158:0/1097286617 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595401f070 con 0x7f59640cea20 2023-10-16T11:21:54.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5964155d90 con 0x7f59640cea20 2023-10-16T11:21:54.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f594b7fe700 1 -- 172.21.15.158:0/1097286617 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954003ae0 con 0x7f59640cea20 2023-10-16T11:21:54.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.301+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5964156350 con 0x7f59640cea20 2023-10-16T11:21:54.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.302+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59641564e0 con 0x7f59640cea20 2023-10-16T11:21:54.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.302+0000 7f594b7fe700 1 -- 172.21.15.158:0/1097286617 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5954003c40 con 0x7f59640cea20 2023-10-16T11:21:54.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.303+0000 7f594b7fe700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f595006b020 0x7f595006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.303+0000 7f594b7fe700 1 -- 172.21.15.158:0/1097286617 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5657+0+0 (secure 0 0 0) 0x7f5954021070 con 0x7f59640cea20 2023-10-16T11:21:54.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.303+0000 7f596259c700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f595006b020 0x7f595006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.303+0000 7f596259c700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f595006b020 0x7f595006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5958006fd0 tx=0x7f5958008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:54.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.307+0000 7f594b7fe700 1 -- 172.21.15.158:0/1097286617 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59540bdfe0 con 0x7f59640cea20 2023-10-16T11:21:54.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.499+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f59640681b0 con 0x7f59640cea20 2023-10-16T11:21:54.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.678+0000 7f594b7fe700 1 -- 172.21.15.158:0/1097286617 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v267) v1 ==== 138+0+0 (secure 0 0 0) 0x7f59540150b0 con 0x7f59640cea20 2023-10-16T11:21:54.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f595006b020 msgr2=0x7f595006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f595006b020 0x7f595006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5958006fd0 tx=0x7f5958008040 comp rx=0 tx=0).stop 2023-10-16T11:21:54.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59640cea20 msgr2=0x7f5964156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59640cea20 0x7f5964156b50 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f5954000c00 tx=0x7f5954004750 comp rx=0 tx=0).stop 2023-10-16T11:21:54.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 shutdown_connections 2023-10-16T11:21:54.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f596415deb0 0x7f59641e53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f595006b020 0x7f595006d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59640cea20 0x7f5964156b50 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 --2- 172.21.15.158:0/1097286617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596415dad0 0x7f5964153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 >> 172.21.15.158:0/1097286617 conn(0x7f5964099a10 msgr2=0x7f59640b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:54.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 shutdown_connections 2023-10-16T11:21:54.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.681+0000 7f59691e3700 1 -- 172.21.15.158:0/1097286617 wait complete. 2023-10-16T11:21:54.685 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' created 2023-10-16T11:21:54.699 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T11:21:54.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.824+0000 7faf979b3700 1 Processor -- start 2023-10-16T11:21:54.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.824+0000 7faf979b3700 1 -- start start 2023-10-16T11:21:54.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf979b3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faf9015a5e0 0x7faf9015b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf979b3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf90071940 0x7faf90071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf979b3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf9006eeb0 0x7faf9006f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf979b3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf900722e0 con 0x7faf9006eeb0 2023-10-16T11:21:54.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf979b3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9015bb60 con 0x7faf9015a5e0 2023-10-16T11:21:54.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf979b3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9015e1c0 con 0x7faf90071940 2023-10-16T11:21:54.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf95f50700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf9006eeb0 0x7faf9006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf95f50700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf9006eeb0 0x7faf9006f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43670/0 (socket says 172.21.15.158:43670) 2023-10-16T11:21:54.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.825+0000 7faf95f50700 1 -- 172.21.15.158:0/3660709707 learned_addr learned my addr 172.21.15.158:0/3660709707 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:54.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf9574f700 1 --2- 172.21.15.158:0/3660709707 >> v2:172.21.15.172:3300/0 conn(0x7faf9015a5e0 0x7faf9015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf94f4e700 1 --2- 172.21.15.158:0/3660709707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf90071940 0x7faf90071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf95f50700 1 -- 172.21.15.158:0/3660709707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf90071940 msgr2=0x7faf90071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:54.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf95f50700 1 --2- 172.21.15.158:0/3660709707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf90071940 0x7faf90071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf95f50700 1 -- 172.21.15.158:0/3660709707 >> v2:172.21.15.172:3300/0 conn(0x7faf9015a5e0 msgr2=0x7faf9015b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:54.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf95f50700 1 --2- 172.21.15.158:0/3660709707 >> v2:172.21.15.172:3300/0 conn(0x7faf9015a5e0 0x7faf9015b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf95f50700 1 -- 172.21.15.158:0/3660709707 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9015e4a0 con 0x7faf9006eeb0 2023-10-16T11:21:54.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf9574f700 1 --2- 172.21.15.158:0/3660709707 >> v2:172.21.15.172:3300/0 conn(0x7faf9015a5e0 0x7faf9015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:54.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf94f4e700 1 --2- 172.21.15.158:0/3660709707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf90071940 0x7faf90071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:54.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf95f50700 1 --2- 172.21.15.158:0/3660709707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9006eeb0 0x7faf9006f320 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7faf9006d260 tx=0x7faf8800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a0bc3b0ec9f35b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:54.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf87fff700 1 -- 172.21.15.158:0/3660709707 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88010070 con 0x7faf9006eeb0 2023-10-16T11:21:54.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf87fff700 1 -- 172.21.15.158:0/3660709707 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8800dd90 con 0x7faf9006eeb0 2023-10-16T11:21:54.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.826+0000 7faf87fff700 1 -- 172.21.15.158:0/3660709707 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800b310 con 0x7faf9006eeb0 2023-10-16T11:21:54.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 -- 172.21.15.158:0/3660709707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9006eeb0 msgr2=0x7faf9006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 --2- 172.21.15.158:0/3660709707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9006eeb0 0x7faf9006f320 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7faf9006d260 tx=0x7faf8800d430 comp rx=0 tx=0).stop 2023-10-16T11:21:54.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 -- 172.21.15.158:0/3660709707 shutdown_connections 2023-10-16T11:21:54.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 --2- 172.21.15.158:0/3660709707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9006eeb0 0x7faf9006f320 unknown :-1 s=CLOSED pgs=3276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 --2- 172.21.15.158:0/3660709707 >> v2:172.21.15.172:3300/0 conn(0x7faf9015a5e0 0x7faf9015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 --2- 172.21.15.158:0/3660709707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf90071940 0x7faf90071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 -- 172.21.15.158:0/3660709707 >> 172.21.15.158:0/3660709707 conn(0x7faf900b1e30 msgr2=0x7faf900b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:54.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 -- 172.21.15.158:0/3660709707 shutdown_connections 2023-10-16T11:21:54.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.827+0000 7faf979b3700 1 -- 172.21.15.158:0/3660709707 wait complete. 2023-10-16T11:21:54.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.828+0000 7faf979b3700 1 Processor -- start 2023-10-16T11:21:54.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.828+0000 7faf979b3700 1 -- start start 2023-10-16T11:21:54.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf979b3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf9006eeb0 0x7faf900c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf979b3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90071940 0x7faf90157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf979b3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9015a5e0 0x7faf900c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf979b3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf900ca790 con 0x7faf9015a5e0 2023-10-16T11:21:54.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf979b3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf900b47a0 con 0x7faf90071940 2023-10-16T11:21:54.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf979b3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf90157940 con 0x7faf9006eeb0 2023-10-16T11:21:54.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9015a5e0 0x7faf900c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9015a5e0 0x7faf900c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43676/0 (socket says 172.21.15.158:43676) 2023-10-16T11:21:54.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 -- 172.21.15.158:0/3286800026 learned_addr learned my addr 172.21.15.158:0/3286800026 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:54.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf94f4e700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90071940 0x7faf90157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf9574f700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf9006eeb0 0x7faf900c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 -- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf9006eeb0 msgr2=0x7faf900c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf9006eeb0 0x7faf900c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 -- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90071940 msgr2=0x7faf90157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:54.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90071940 0x7faf90157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:54.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 -- 172.21.15.158:0/3286800026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8800bd40 con 0x7faf9015a5e0 2023-10-16T11:21:54.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf95f50700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9015a5e0 0x7faf900c74c0 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7faf9006d2f0 tx=0x7faf88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:54.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf85ffb700 1 -- 172.21.15.158:0/3286800026 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800bbc0 con 0x7faf9015a5e0 2023-10-16T11:21:54.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf85ffb700 1 -- 172.21.15.158:0/3286800026 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88009810 con 0x7faf9015a5e0 2023-10-16T11:21:54.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf85ffb700 1 -- 172.21.15.158:0/3286800026 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88009bc0 con 0x7faf9015a5e0 2023-10-16T11:21:54.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.829+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9006e4b0 con 0x7faf9015a5e0 2023-10-16T11:21:54.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.830+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf900b4150 con 0x7faf9015a5e0 2023-10-16T11:21:54.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.830+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf900b3fb0 con 0x7faf9015a5e0 2023-10-16T11:21:54.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.831+0000 7faf85ffb700 1 -- 172.21.15.158:0/3286800026 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faf88009d20 con 0x7faf9015a5e0 2023-10-16T11:21:54.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.831+0000 7faf85ffb700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7faf7406b020 0x7faf7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:54.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.832+0000 7faf85ffb700 1 -- 172.21.15.158:0/3286800026 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6027+0+0 (secure 0 0 0) 0x7faf8808d3d0 con 0x7faf9015a5e0 2023-10-16T11:21:54.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.832+0000 7faf9574f700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7faf7406b020 0x7faf7406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:54.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.832+0000 7faf9574f700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7faf7406b020 0x7faf7406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:54.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:54.836+0000 7faf85ffb700 1 -- 172.21.15.158:0/3286800026 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf900b3fb0 con 0x7faf9015a5e0 2023-10-16T11:21:55.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.028+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faf900b3fb0 con 0x7faf9015a5e0 2023-10-16T11:21:55.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.686+0000 7faf85ffb700 1 -- 172.21.15.158:0/3286800026 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v268) v1 ==== 170+0+0 (secure 0 0 0) 0x7faf900b3fb0 con 0x7faf9015a5e0 2023-10-16T11:21:55.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7faf7406b020 msgr2=0x7faf7406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:55.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7faf7406b020 0x7faf7406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80008040 comp rx=0 tx=0).stop 2023-10-16T11:21:55.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9015a5e0 msgr2=0x7faf900c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:55.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9015a5e0 0x7faf900c74c0 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7faf9006d2f0 tx=0x7faf88004ad0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 shutdown_connections 2023-10-16T11:21:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7faf7406b020 0x7faf7406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf9015a5e0 0x7faf900c74c0 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf9006eeb0 0x7faf900c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 --2- 172.21.15.158:0/3286800026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90071940 0x7faf90157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.695+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 >> 172.21.15.158:0/3286800026 conn(0x7faf900b1e30 msgr2=0x7faf900ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:55.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.696+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 shutdown_connections 2023-10-16T11:21:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.697+0000 7faf979b3700 1 -- 172.21.15.158:0/3286800026 wait complete. 2023-10-16T11:21:55.704 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' created 2023-10-16T11:21:55.719 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T11:21:55.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.843+0000 7fc3ed300700 1 Processor -- start 2023-10-16T11:21:55.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.843+0000 7fc3ed300700 1 -- start start 2023-10-16T11:21:55.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3ed300700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3e8160d30 0x7fc3e80b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:55.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3ed300700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80b7a20 con 0x7fc3e80b7590 2023-10-16T11:21:55.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3ed300700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80b7c90 con 0x7fc3e80d0530 2023-10-16T11:21:55.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3ed300700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80b8910 con 0x7fc3e8160d30 2023-10-16T11:21:55.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3e6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3e8160d30 0x7fc3e80b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:55.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3e6ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3e8160d30 0x7fc3e80b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43680/0 (socket says 172.21.15.158:43680) 2023-10-16T11:21:55.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3e6ffd700 1 -- 172.21.15.158:0/3287782164 learned_addr learned my addr 172.21.15.158:0/3287782164 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:55.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3e77fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc3e80b7590 0x7fc3e80d20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43860/0 (socket says 172.21.15.158:43860) 2023-10-16T11:21:55.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3e6ffd700 1 -- 172.21.15.158:0/3287782164 >> v1:172.21.15.172:6789/0 conn(0x7fc3e80d0530 legacy=0x7fc3e80d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:55.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.844+0000 7fc3e6ffd700 1 -- 172.21.15.158:0/3287782164 >> v1:172.21.15.158:6789/0 conn(0x7fc3e80b7590 legacy=0x7fc3e80d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:55.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3e5ffb700 1 -- 172.21.15.158:0/3287782164 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079437368 0 0) 0x7fc3e80b7a20 con 0x7fc3e80b7590 2023-10-16T11:21:55.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3e6ffd700 1 -- 172.21.15.158:0/3287782164 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80b7f70 con 0x7fc3e8160d30 2023-10-16T11:21:55.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3e6ffd700 1 --2- 172.21.15.158:0/3287782164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80b8f50 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009920 tx=0x7fc3d8009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1041bcec576b35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:55.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3e5ffb700 1 -- 172.21.15.158:0/3287782164 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800ba20 con 0x7fc3e8160d30 2023-10-16T11:21:55.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3e5ffb700 1 -- 172.21.15.158:0/3287782164 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d800bb80 con 0x7fc3e8160d30 2023-10-16T11:21:55.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3e5ffb700 1 -- 172.21.15.158:0/3287782164 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8009e70 con 0x7fc3e8160d30 2023-10-16T11:21:55.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3ed300700 1 -- 172.21.15.158:0/3287782164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 msgr2=0x7fc3e80b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:55.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3ed300700 1 --2- 172.21.15.158:0/3287782164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80b8f50 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009920 tx=0x7fc3d8009c30 comp rx=0 tx=0).stop 2023-10-16T11:21:55.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3ed300700 1 -- 172.21.15.158:0/3287782164 shutdown_connections 2023-10-16T11:21:55.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3ed300700 1 --2- 172.21.15.158:0/3287782164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80b8f50 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3ed300700 1 -- 172.21.15.158:0/3287782164 >> 172.21.15.158:0/3287782164 conn(0x7fc3e80669f0 msgr2=0x7fc3e8066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:55.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3ed300700 1 -- 172.21.15.158:0/3287782164 shutdown_connections 2023-10-16T11:21:55.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.845+0000 7fc3ed300700 1 -- 172.21.15.158:0/3287782164 wait complete. 2023-10-16T11:21:55.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.847+0000 7fc3ed300700 1 Processor -- start 2023-10-16T11:21:55.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.847+0000 7fc3ed300700 1 -- start start 2023-10-16T11:21:55.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3e80b7590 0x7fc3e8158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:55.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3e80d0530 0x7fc3e815a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:55.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80d3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:55.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80c8140 con 0x7fc3e8160d30 2023-10-16T11:21:55.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8159080 con 0x7fc3e80d0530 2023-10-16T11:21:55.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8159360 con 0x7fc3e80b7590 2023-10-16T11:21:55.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:55.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80d3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43690/0 (socket says 172.21.15.158:43690) 2023-10-16T11:21:55.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 -- 172.21.15.158:0/2152259105 learned_addr learned my addr 172.21.15.158:0/2152259105 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:55.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e6ffd700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3e80b7590 0x7fc3e8158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:55.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e67fc700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3e80d0530 0x7fc3e815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:55.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 -- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3e80b7590 msgr2=0x7fc3e8158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:55.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3e80b7590 0x7fc3e8158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 -- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3e80d0530 msgr2=0x7fc3e815a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:55.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3e80d0530 0x7fc3e815a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:55.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 -- 172.21.15.158:0/2152259105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8009550 con 0x7fc3e8160d30 2023-10-16T11:21:55.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3e77fe700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80d3630 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0002230 tx=0x7fc3e000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:55.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3cf7fe700 1 -- 172.21.15.158:0/2152259105 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000aac0 con 0x7fc3e8160d30 2023-10-16T11:21:55.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3cf7fe700 1 -- 172.21.15.158:0/2152259105 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e000ac40 con 0x7fc3e8160d30 2023-10-16T11:21:55.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e8159640 con 0x7fc3e8160d30 2023-10-16T11:21:55.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3cf7fe700 1 -- 172.21.15.158:0/2152259105 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00070d0 con 0x7fc3e8160d30 2023-10-16T11:21:55.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.848+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e8158d70 con 0x7fc3e8160d30 2023-10-16T11:21:55.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.849+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e80681b0 con 0x7fc3e8160d30 2023-10-16T11:21:55.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.850+0000 7fc3cf7fe700 1 -- 172.21.15.158:0/2152259105 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc3e0007250 con 0x7fc3e8160d30 2023-10-16T11:21:55.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.850+0000 7fc3cf7fe700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc3d406b020 0x7fc3d406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:55.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.850+0000 7fc3cf7fe700 1 -- 172.21.15.158:0/2152259105 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6409+0+0 (secure 0 0 0) 0x7fc3e00897d0 con 0x7fc3e8160d30 2023-10-16T11:21:55.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.850+0000 7fc3e6ffd700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc3d406b020 0x7fc3d406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:55.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.851+0000 7fc3e6ffd700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc3d406b020 0x7fc3d406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009520 tx=0x7fc3d801c040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:55.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:55.855+0000 7fc3cf7fe700 1 -- 172.21.15.158:0/2152259105 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3e00544b0 con 0x7fc3e8160d30 2023-10-16T11:21:56.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.092+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc3e8158260 con 0x7fc3e8160d30 2023-10-16T11:21:56.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.701+0000 7fc3cf7fe700 1 -- 172.21.15.158:0/2152259105 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 37 and data pool 38 v150) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc3e0087e60 con 0x7fc3e8160d30 2023-10-16T11:21:56.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc3d406b020 msgr2=0x7fc3d406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:56.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc3d406b020 0x7fc3d406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009520 tx=0x7fc3d801c040 comp rx=0 tx=0).stop 2023-10-16T11:21:56.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 msgr2=0x7fc3e80d3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:56.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80d3630 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0002230 tx=0x7fc3e000cf50 comp rx=0 tx=0).stop 2023-10-16T11:21:56.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 shutdown_connections 2023-10-16T11:21:56.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc3d406b020 0x7fc3d406d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3e8160d30 0x7fc3e80d3630 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3e80b7590 0x7fc3e8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 --2- 172.21.15.158:0/2152259105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3e80d0530 0x7fc3e815a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 >> 172.21.15.158:0/2152259105 conn(0x7fc3e80669f0 msgr2=0x7fc3e80cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:56.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.710+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 shutdown_connections 2023-10-16T11:21:56.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.711+0000 7fc3ed300700 1 -- 172.21.15.158:0/2152259105 wait complete. 2023-10-16T11:21:56.714 INFO:teuthology.orchestra.run.smithi158.stderr:new fs with metadata pool 37 and data pool 38 2023-10-16T11:21:56.731 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:56.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.853+0000 7f88ec1d2700 1 Processor -- start 2023-10-16T11:21:56.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.853+0000 7f88ec1d2700 1 -- start start 2023-10-16T11:21:56.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88ec1d2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f88e406b0a0 0x7f88e415f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88ec1d2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e4072700 con 0x7f88e40cb010 2023-10-16T11:21:56.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88ec1d2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e40cca90 con 0x7f88e406d820 2023-10-16T11:21:56.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88ec1d2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e4071910 con 0x7f88e406b0a0 2023-10-16T11:21:56.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88ea76f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f88e40cb010 0x7f88e415bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43874/0 (socket says 172.21.15.158:43874) 2023-10-16T11:21:56.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88ea76f700 1 -- 172.21.15.158:0/2566752206 learned_addr learned my addr 172.21.15.158:0/2566752206 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:56.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367534134 0 0) 0x7f88e4072700 con 0x7f88e40cb010 2023-10-16T11:21:56.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d00035b0 con 0x7f88e40cb010 2023-10-16T11:21:56.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e9f6e700 1 --2- 172.21.15.158:0/2566752206 >> v2:172.21.15.172:3300/0 conn(0x7f88e406b0a0 0x7f88e415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928640163 0 0) 0x7f88d00035b0 con 0x7f88e40cb010 2023-10-16T11:21:56.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 >> v2:172.21.15.172:3300/0 conn(0x7f88e406b0a0 msgr2=0x7f88e415f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 --2- 172.21.15.158:0/2566752206 >> v2:172.21.15.172:3300/0 conn(0x7f88e406b0a0 0x7f88e415f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 >> v1:172.21.15.172:6789/0 conn(0x7f88e406d820 legacy=0x7f88e40ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:56.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e4071af0 con 0x7f88e40cb010 2023-10-16T11:21:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f88dc0019f0 con 0x7f88e40cb010 2023-10-16T11:21:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 >> v1:172.21.15.158:6789/0 conn(0x7f88e40cb010 legacy=0x7f88e415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d0003a00 0x7f88d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88d0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88d0005210 0x7f88d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e4071910 con 0x7f88d0004850 2023-10-16T11:21:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d000c0b0 con 0x7f88d0003a00 2023-10-16T11:21:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d000c270 con 0x7f88d0005210 2023-10-16T11:21:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.854+0000 7f88e976d700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88d0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e9f6e700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d0003a00 0x7f88d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88ea76f700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88d0005210 0x7f88d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e976d700 1 -- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88d0005210 msgr2=0x7f88d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e976d700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88d0005210 0x7f88d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e976d700 1 -- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d0003a00 msgr2=0x7f88d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e976d700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d0003a00 0x7f88d0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e976d700 1 -- 172.21.15.158:0/2566752206 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d00035b0 con 0x7f88d0004850 2023-10-16T11:21:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88ea76f700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88d0005210 0x7f88d000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88ea76f700 1 -- 172.21.15.158:0/2566752206 reap_dead start 2023-10-16T11:21:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e976d700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88d0004cd0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f88e406dcb0 tx=0x7f88d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72655768ed320cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d40052d0 con 0x7f88d0004850 2023-10-16T11:21:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d4005430 con 0x7f88d0004850 2023-10-16T11:21:56.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.855+0000 7f88e8f6c700 1 -- 172.21.15.158:0/2566752206 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d40038c0 con 0x7f88d0004850 2023-10-16T11:21:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.856+0000 7f88ec1d2700 1 -- 172.21.15.158:0/2566752206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 msgr2=0x7f88d0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.856+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88d0004cd0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f88e406dcb0 tx=0x7f88d0005880 comp rx=0 tx=0).stop 2023-10-16T11:21:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.856+0000 7f88ec1d2700 1 -- 172.21.15.158:0/2566752206 shutdown_connections 2023-10-16T11:21:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.856+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/2566752206 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88d0004cd0 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.856+0000 7f88ec1d2700 1 -- 172.21.15.158:0/2566752206 >> 172.21.15.158:0/2566752206 conn(0x7f88e40d02b0 msgr2=0x7f88e40d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.856+0000 7f88ec1d2700 1 -- 172.21.15.158:0/2566752206 shutdown_connections 2023-10-16T11:21:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.856+0000 7f88ec1d2700 1 -- 172.21.15.158:0/2566752206 wait complete. 2023-10-16T11:21:56.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.857+0000 7f88ec1d2700 1 Processor -- start 2023-10-16T11:21:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.857+0000 7f88ec1d2700 1 -- start start 2023-10-16T11:21:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88e41e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88e41e7570 0x7f88e40c3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88e40c9d20 0x7f88e40c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e41e7150 con 0x7f88d0004850 2023-10-16T11:21:56.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e41e64d0 con 0x7f88e41e7570 2023-10-16T11:21:56.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e41e67b0 con 0x7f88e40c9d20 2023-10-16T11:21:56.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88e41e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88e41e5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43706/0 (socket says 172.21.15.158:43706) 2023-10-16T11:21:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 -- 172.21.15.158:0/162801520 learned_addr learned my addr 172.21.15.158:0/162801520 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:56.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e976d700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88e41e7570 0x7f88e40c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ea76f700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88e40c9d20 0x7f88e40c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 -- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88e40c9d20 msgr2=0x7f88e40c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88e40c9d20 0x7f88e40c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 -- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88e41e7570 msgr2=0x7f88e40c3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88e41e7570 0x7f88e40c3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:56.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 -- 172.21.15.158:0/162801520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e41e6a90 con 0x7f88d0004850 2023-10-16T11:21:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e976d700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88e41e7570 0x7f88e40c3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ea76f700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88e40c9d20 0x7f88e40c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88e9f6e700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88e41e5e80 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7f88e0011aa0 tx=0x7f88e0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88da7fc700 1 -- 172.21.15.158:0/162801520 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e000f9e0 con 0x7f88d0004850 2023-10-16T11:21:56.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88da7fc700 1 -- 172.21.15.158:0/162801520 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e000fb40 con 0x7f88d0004850 2023-10-16T11:21:56.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e41e6d70 con 0x7f88d0004850 2023-10-16T11:21:56.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88da7fc700 1 -- 172.21.15.158:0/162801520 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0014440 con 0x7f88d0004850 2023-10-16T11:21:56.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.858+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e40ca1d0 con 0x7f88d0004850 2023-10-16T11:21:56.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.859+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e41e73c0 con 0x7f88d0004850 2023-10-16T11:21:56.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.860+0000 7f88da7fc700 1 -- 172.21.15.158:0/162801520 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f88e00145a0 con 0x7f88d0004850 2023-10-16T11:21:56.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.860+0000 7f88da7fc700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f88c806b020 0x7f88c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:56.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.860+0000 7f88da7fc700 1 -- 172.21.15.158:0/162801520 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f88e008a1c0 con 0x7f88d0004850 2023-10-16T11:21:56.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.860+0000 7f88e976d700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f88c806b020 0x7f88c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:56.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.860+0000 7f88e976d700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f88c806b020 0x7f88c806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f88d4000c00 tx=0x7f88d4009030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:56.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:56.864+0000 7f88da7fc700 1 -- 172.21.15.158:0/162801520 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88e0054ed0 con 0x7f88d0004850 2023-10-16T11:21:57.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.056+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88e40ca420 con 0x7f88d0004850 2023-10-16T11:21:57.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.057+0000 7f88da7fc700 1 -- 172.21.15.158:0/162801520 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v269) v1 ==== 74+0+17488 (secure 0 0 0) 0x7f88e00584f0 con 0x7f88d0004850 2023-10-16T11:21:57.059 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:57.060 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":269,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:56.093799+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:54.523055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:55.050879+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:57.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f88c806b020 msgr2=0x7f88c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f88c806b020 0x7f88c806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f88d4000c00 tx=0x7f88d4009030 comp rx=0 tx=0).stop 2023-10-16T11:21:57.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 msgr2=0x7f88e41e5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88e41e5e80 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7f88e0011aa0 tx=0x7f88e0011e60 comp rx=0 tx=0).stop 2023-10-16T11:21:57.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 shutdown_connections 2023-10-16T11:21:57.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88e40c9d20 0x7f88e40c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f88c806b020 0x7f88c806d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88d0004850 0x7f88e41e5e80 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 --2- 172.21.15.158:0/162801520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88e41e7570 0x7f88e40c3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 >> 172.21.15.158:0/162801520 conn(0x7f88e40d02b0 msgr2=0x7f88e40b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:57.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 shutdown_connections 2023-10-16T11:21:57.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.060+0000 7f88ec1d2700 1 -- 172.21.15.158:0/162801520 wait complete. 2023-10-16T11:21:57.074 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:21:57.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.192+0000 7ff8bcfcf700 1 Processor -- start 2023-10-16T11:21:57.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.192+0000 7ff8bcfcf700 1 -- start start 2023-10-16T11:21:57.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.193+0000 7ff8bcfcf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff8b806e5d0 0x7ff8b806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.193+0000 7ff8bcfcf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b8161270 con 0x7ff8b806ab40 2023-10-16T11:21:57.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.193+0000 7ff8bcfcf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b8161510 con 0x7ff8b80729b0 2023-10-16T11:21:57.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.193+0000 7ff8bcfcf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b81617f0 con 0x7ff8b806e5d0 2023-10-16T11:21:57.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.193+0000 7ff8b659c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff8b806ab40 0x7ff8b80722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43890/0 (socket says 172.21.15.158:43890) 2023-10-16T11:21:57.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b659c700 1 -- 172.21.15.158:0/1771933882 learned_addr learned my addr 172.21.15.158:0/1771933882 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:57.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153876482 0 0) 0x7ff8b8161270 con 0x7ff8b806ab40 2023-10-16T11:21:57.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b5d9b700 1 --2- 172.21.15.158:0/1771933882 >> v2:172.21.15.172:3300/0 conn(0x7ff8b806e5d0 0x7ff8b806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a00035b0 con 0x7ff8b806ab40 2023-10-16T11:21:57.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278335247 0 0) 0x7ff8a00035b0 con 0x7ff8b806ab40 2023-10-16T11:21:57.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 >> v2:172.21.15.172:3300/0 conn(0x7ff8b806e5d0 msgr2=0x7ff8b806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:57.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 --2- 172.21.15.158:0/1771933882 >> v2:172.21.15.172:3300/0 conn(0x7ff8b806e5d0 0x7ff8b806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 >> v1:172.21.15.172:6789/0 conn(0x7ff8b80729b0 legacy=0x7ff8b815df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b8161ad0 con 0x7ff8b806ab40 2023-10-16T11:21:57.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff8ac001670 con 0x7ff8b806ab40 2023-10-16T11:21:57.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 >> v1:172.21.15.158:6789/0 conn(0x7ff8b806ab40 legacy=0x7ff8b80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a0003a00 0x7ff8a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a0004850 0x7ff8a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81617f0 con 0x7ff8a0005210 2023-10-16T11:21:57.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a000c0b0 con 0x7ff8a0003a00 2023-10-16T11:21:57.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a000c270 con 0x7ff8a0004850 2023-10-16T11:21:57.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b6d9d700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b659c700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a0003a00 0x7ff8a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b5d9b700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a0004850 0x7ff8a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b6d9d700 1 -- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a0004850 msgr2=0x7ff8a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:57.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.194+0000 7ff8b6d9d700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a0004850 0x7ff8a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b6d9d700 1 -- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a0003a00 msgr2=0x7ff8a0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:57.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b6d9d700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8a0003a00 0x7ff8a0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b6d9d700 1 -- 172.21.15.158:0/1771933882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00035b0 con 0x7ff8a0005210 2023-10-16T11:21:57.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b5d9b700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8a0004850 0x7ff8a0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:57.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b6d9d700 1 -- 172.21.15.158:0/1771933882 reap_dead start 2023-10-16T11:21:57.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b6d9d700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8a000ba10 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000c210 tx=0x7ff8a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe25e8a8dd1de22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:57.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0001df0 con 0x7ff8a0005210 2023-10-16T11:21:57.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b000cd70 con 0x7ff8a0005210 2023-10-16T11:21:57.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8b559a700 1 -- 172.21.15.158:0/1771933882 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0003c40 con 0x7ff8a0005210 2023-10-16T11:21:57.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/1771933882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 msgr2=0x7ff8a000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8a000ba10 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000c210 tx=0x7ff8a0005880 comp rx=0 tx=0).stop 2023-10-16T11:21:57.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/1771933882 shutdown_connections 2023-10-16T11:21:57.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/1771933882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8a000ba10 unknown :-1 s=CLOSED pgs=3285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/1771933882 >> 172.21.15.158:0/1771933882 conn(0x7ff8b80d02b0 msgr2=0x7ff8b80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:57.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/1771933882 shutdown_connections 2023-10-16T11:21:57.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.195+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/1771933882 wait complete. 2023-10-16T11:21:57.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.197+0000 7ff8bcfcf700 1 Processor -- start 2023-10-16T11:21:57.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.197+0000 7ff8bcfcf700 1 -- start start 2023-10-16T11:21:57.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.197+0000 7ff8bcfcf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8b80c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.197+0000 7ff8bcfcf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8b8154bd0 0x7ff8b80c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8bcfcf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8b80b50b0 0x7ff8b8153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8bcfcf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b80b4c90 con 0x7ff8a0005210 2023-10-16T11:21:57.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8bcfcf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b80b4010 con 0x7ff8b8154bd0 2023-10-16T11:21:57.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8bcfcf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b80b42f0 con 0x7ff8b80b50b0 2023-10-16T11:21:57.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8b80c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8b80c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43726/0 (socket says 172.21.15.158:43726) 2023-10-16T11:21:57.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 -- 172.21.15.158:0/512027703 learned_addr learned my addr 172.21.15.158:0/512027703 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:57.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b6d9d700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8b80b50b0 0x7ff8b8153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b5d9b700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8b8154bd0 0x7ff8b80c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 -- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8b80b50b0 msgr2=0x7ff8b8153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8b80b50b0 0x7ff8b8153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 -- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8b8154bd0 msgr2=0x7ff8b80c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8b8154bd0 0x7ff8b80c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 -- 172.21.15.158:0/512027703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80b45d0 con 0x7ff8a0005210 2023-10-16T11:21:57.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b6d9d700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8b80b50b0 0x7ff8b8153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:21:57.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8b659c700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8b80c65d0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac007290 tx=0x7ff8ac0075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:57.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8a6ffd700 1 -- 172.21.15.158:0/512027703 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00f5e0 con 0x7ff8a0005210 2023-10-16T11:21:57.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8a6ffd700 1 -- 172.21.15.158:0/512027703 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac00f740 con 0x7ff8a0005210 2023-10-16T11:21:57.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b80b48b0 con 0x7ff8a0005210 2023-10-16T11:21:57.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8a6ffd700 1 -- 172.21.15.158:0/512027703 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00faf0 con 0x7ff8a0005210 2023-10-16T11:21:57.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.198+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b80c4e40 con 0x7ff8a0005210 2023-10-16T11:21:57.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.199+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b80681b0 con 0x7ff8a0005210 2023-10-16T11:21:57.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.200+0000 7ff8a6ffd700 1 -- 172.21.15.158:0/512027703 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff8ac00fc50 con 0x7ff8a0005210 2023-10-16T11:21:57.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.200+0000 7ff8a6ffd700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff89806b020 0x7ff89806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.200+0000 7ff8b5d9b700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff89806b020 0x7ff89806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.200+0000 7ff8a6ffd700 1 -- 172.21.15.158:0/512027703 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff8ac089950 con 0x7ff8a0005210 2023-10-16T11:21:57.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.201+0000 7ff8b5d9b700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff89806b020 0x7ff89806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8b806b560 tx=0x7ff8a8008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:57.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.205+0000 7ff8a6ffd700 1 -- 172.21.15.158:0/512027703 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8ac054660 con 0x7ff8a0005210 2023-10-16T11:21:57.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.396+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8b8162060 con 0x7ff8a0005210 2023-10-16T11:21:57.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.397+0000 7ff8a6ffd700 1 -- 172.21.15.158:0/512027703 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v269) v1 ==== 74+0+17488 (secure 0 0 0) 0x7ff8ac057c80 con 0x7ff8a0005210 2023-10-16T11:21:57.399 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:21:57.399 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":269,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:56.093799+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:54.523055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:55.050879+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:21:57.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff89806b020 msgr2=0x7ff89806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff89806b020 0x7ff89806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8b806b560 tx=0x7ff8a8008040 comp rx=0 tx=0).stop 2023-10-16T11:21:57.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 msgr2=0x7ff8b80c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8b80c65d0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac007290 tx=0x7ff8ac0075a0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 shutdown_connections 2023-10-16T11:21:57.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8b80b50b0 0x7ff8b8153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff89806b020 0x7ff89806d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8a0005210 0x7ff8b80c65d0 unknown :-1 s=CLOSED pgs=3286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 --2- 172.21.15.158:0/512027703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8b8154bd0 0x7ff8b80c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 >> 172.21.15.158:0/512027703 conn(0x7ff8b80d02b0 msgr2=0x7ff8b80cb790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:57.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 shutdown_connections 2023-10-16T11:21:57.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.399+0000 7ff8bcfcf700 1 -- 172.21.15.158:0/512027703 wait complete. 2023-10-16T11:21:57.414 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T11:21:57.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.535+0000 7fe4a225e700 1 Processor -- start 2023-10-16T11:21:57.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.535+0000 7fe4a225e700 1 -- start start 2023-10-16T11:21:57.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.536+0000 7fe4a225e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe49c15daa0 0x7fe49c0cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.536+0000 7fe4a225e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 0x7fe49c0cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.536+0000 7fe4a225e700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c0d1ee0 con 0x7fe49c15d6c0 2023-10-16T11:21:57.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.536+0000 7fe4a225e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c0d40a0 con 0x7fe49c15daa0 2023-10-16T11:21:57.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.536+0000 7fe4a225e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0d12d0 con 0x7fe49c0d0640 2023-10-16T11:21:57.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.536+0000 7fe49b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe49c15d6c0 0x7fe49c0cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43910/0 (socket says 172.21.15.158:43910) 2023-10-16T11:21:57.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.536+0000 7fe49b7fe700 1 -- 172.21.15.158:0/2760977762 learned_addr learned my addr 172.21.15.158:0/2760977762 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:21:57.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49affd700 1 --2- 172.21.15.158:0/2760977762 >> v2:172.21.15.172:3300/0 conn(0x7fe49c15daa0 0x7fe49c0cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 0x7fe49c0cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174398654 0 0) 0x7fe49c0d1ee0 con 0x7fe49c15d6c0 2023-10-16T11:21:57.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4840035b0 con 0x7fe49c15d6c0 2023-10-16T11:21:57.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226597159 0 0) 0x7fe4840035b0 con 0x7fe49c15d6c0 2023-10-16T11:21:57.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 msgr2=0x7fe49c0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:57.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 0x7fe49c0cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 >> v2:172.21.15.172:3300/0 conn(0x7fe49c15daa0 msgr2=0x7fe49c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:57.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 --2- 172.21.15.158:0/2760977762 >> v2:172.21.15.172:3300/0 conn(0x7fe49c15daa0 0x7fe49c0cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c0d15b0 con 0x7fe49c15d6c0 2023-10-16T11:21:57.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fe490001670 con 0x7fe49c15d6c0 2023-10-16T11:21:57.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49affd700 1 --2- 172.21.15.158:0/2760977762 >> v2:172.21.15.172:3300/0 conn(0x7fe49c15daa0 0x7fe49c0cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:21:57.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 >> v1:172.21.15.158:6789/0 conn(0x7fe49c15d6c0 legacy=0x7fe49c0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe484003a00 0x7fe484003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 0x7fe484004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe484004d90 0x7fe484005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0d12d0 con 0x7fe484004d90 2023-10-16T11:21:57.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48400ba10 con 0x7fe484003a00 2023-10-16T11:21:57.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48400bb80 con 0x7fe49c0d0640 2023-10-16T11:21:57.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe484004d90 0x7fe484005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49b7fe700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe484003a00 0x7fe484003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49affd700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 0x7fe484004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49bfff700 1 -- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 msgr2=0x7fe484004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:57.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0d0640 0x7fe484004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49bfff700 1 -- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe484003a00 msgr2=0x7fe484003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:57.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe484003a00 0x7fe484003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.537+0000 7fe49bfff700 1 -- 172.21.15.158:0/2760977762 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840035b0 con 0x7fe484004d90 2023-10-16T11:21:57.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe484004d90 0x7fe484005210 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7fe494009f90 tx=0x7fe484005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af73ab9ae843305 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:57.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49400ec30 con 0x7fe484004d90 2023-10-16T11:21:57.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49400f040 con 0x7fe484004d90 2023-10-16T11:21:57.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe49a7fc700 1 -- 172.21.15.158:0/2760977762 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494014480 con 0x7fe484004d90 2023-10-16T11:21:57.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe4a225e700 1 -- 172.21.15.158:0/2760977762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe484004d90 msgr2=0x7fe484005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe4a225e700 1 --2- 172.21.15.158:0/2760977762 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe484004d90 0x7fe484005210 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7fe494009f90 tx=0x7fe484005880 comp rx=0 tx=0).stop 2023-10-16T11:21:57.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe49bfff700 1 -- 172.21.15.158:0/2760977762 reap_dead start 2023-10-16T11:21:57.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe4a225e700 1 -- 172.21.15.158:0/2760977762 shutdown_connections 2023-10-16T11:21:57.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe4a225e700 1 -- 172.21.15.158:0/2760977762 >> 172.21.15.158:0/2760977762 conn(0x7fe49c09c940 msgr2=0x7fe49c0d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:57.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe4a225e700 1 -- 172.21.15.158:0/2760977762 shutdown_connections 2023-10-16T11:21:57.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.538+0000 7fe4a225e700 1 -- 172.21.15.158:0/2760977762 wait complete. 2023-10-16T11:21:57.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 Processor -- start 2023-10-16T11:21:57.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 -- start start 2023-10-16T11:21:57.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49c0cc610 0x7fe49c0b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0cc9f0 0x7fe49c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49c0b01a0 0x7fe49c0b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0ca6a0 con 0x7fe49c0b01a0 2023-10-16T11:21:57.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0cae70 con 0x7fe49c0cc610 2023-10-16T11:21:57.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.540+0000 7fe4a225e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0cd3b0 con 0x7fe49c0cc9f0 2023-10-16T11:21:57.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49c0b01a0 0x7fe49c0b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49c0b01a0 0x7fe49c0b0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43750/0 (socket says 172.21.15.158:43750) 2023-10-16T11:21:57.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 -- 172.21.15.158:0/2400184975 learned_addr learned my addr 172.21.15.158:0/2400184975 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:57.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49b7fe700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49c0cc610 0x7fe49c0b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49affd700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0cc9f0 0x7fe49c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 -- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0cc9f0 msgr2=0x7fe49c0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0cc9f0 0x7fe49c0cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 -- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49c0cc610 msgr2=0x7fe49c0b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:57.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49c0cc610 0x7fe49c0b8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:57.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 -- 172.21.15.158:0/2400184975 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c0ca8b0 con 0x7fe49c0b01a0 2023-10-16T11:21:57.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49b7fe700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49c0cc610 0x7fe49c0b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe49bfff700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49c0b01a0 0x7fe49c0b0630 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7fe484004760 tx=0x7fe494009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe483fff700 1 -- 172.21.15.158:0/2400184975 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494007000 con 0x7fe49c0b01a0 2023-10-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe483fff700 1 -- 172.21.15.158:0/2400184975 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe494003e30 con 0x7fe49c0b01a0 2023-10-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c0cab90 con 0x7fe49c0b01a0 2023-10-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe483fff700 1 -- 172.21.15.158:0/2400184975 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494015b60 con 0x7fe49c0b01a0 2023-10-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.541+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c0b0b70 con 0x7fe49c0b01a0 2023-10-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.542+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c0b6400 con 0x7fe49c0b01a0 2023-10-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.543+0000 7fe483fff700 1 -- 172.21.15.158:0/2400184975 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe494024070 con 0x7fe49c0b01a0 2023-10-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.543+0000 7fe483fff700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe48806b020 0x7fe48806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.543+0000 7fe483fff700 1 -- 172.21.15.158:0/2400184975 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe49408ac40 con 0x7fe49c0b01a0 2023-10-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.543+0000 7fe49b7fe700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe48806b020 0x7fe48806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.543+0000 7fe49b7fe700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe48806b020 0x7fe48806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4900048d0 tx=0x7fe490009040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.547+0000 7fe483fff700 1 -- 172.21.15.158:0/2400184975 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe49c0b6400 con 0x7fe49c0b01a0 2023-10-16T11:21:57.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:57.790+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe49c0b67e0 con 0x7fe49c0b01a0 2023-10-16T11:21:58.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.707+0000 7fe483fff700 1 -- 172.21.15.158:0/2400184975 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v153) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe49c0b67e0 con 0x7fe49c0b01a0 2023-10-16T11:21:58.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe48806b020 msgr2=0x7fe48806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:58.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe48806b020 0x7fe48806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4900048d0 tx=0x7fe490009040 comp rx=0 tx=0).stop 2023-10-16T11:21:58.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49c0b01a0 msgr2=0x7fe49c0b0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:58.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49c0b01a0 0x7fe49c0b0630 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7fe484004760 tx=0x7fe494009750 comp rx=0 tx=0).stop 2023-10-16T11:21:58.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 shutdown_connections 2023-10-16T11:21:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe48806b020 0x7fe48806d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe49c0b01a0 0x7fe49c0b0630 unknown :-1 s=CLOSED pgs=3289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe49c0cc610 0x7fe49c0b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 --2- 172.21.15.158:0/2400184975 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe49c0cc9f0 0x7fe49c0cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 >> 172.21.15.158:0/2400184975 conn(0x7fe49c09c940 msgr2=0x7fe49c0d3620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:58.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 shutdown_connections 2023-10-16T11:21:58.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.710+0000 7fe4a225e700 1 -- 172.21.15.158:0/2400184975 wait complete. 2023-10-16T11:21:58.726 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T11:21:58.727 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T11:21:58.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.853+0000 7f6ee1c29700 1 Processor -- start 2023-10-16T11:21:58.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.853+0000 7f6ee1c29700 1 -- start start 2023-10-16T11:21:58.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6ee1c29700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6edc15b540 0x7f6edc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:58.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6ee1c29700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6edc06f180 0x7f6edc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6ee1c29700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc15ee60 con 0x7f6edc06fea0 2023-10-16T11:21:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6ee1c29700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc15b920 con 0x7f6edc06f180 2023-10-16T11:21:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6ee1c29700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc15e2e0 con 0x7f6edc15b540 2023-10-16T11:21:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6edaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6edc06f180 0x7f6edc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6edaffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6edc06f180 0x7f6edc06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43754/0 (socket says 172.21.15.158:43754) 2023-10-16T11:21:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6edaffd700 1 -- 172.21.15.158:0/2576336593 learned_addr learned my addr 172.21.15.158:0/2576336593 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.854+0000 7f6edb7fe700 1 --2- 172.21.15.158:0/2576336593 >> v2:172.21.15.172:3300/0 conn(0x7f6edc15b540 0x7f6edc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6edaffd700 1 -- 172.21.15.158:0/2576336593 >> v2:172.21.15.172:3300/0 conn(0x7f6edc15b540 msgr2=0x7f6edc06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6edaffd700 1 --2- 172.21.15.158:0/2576336593 >> v2:172.21.15.172:3300/0 conn(0x7f6edc15b540 0x7f6edc06d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6edaffd700 1 -- 172.21.15.158:0/2576336593 >> v1:172.21.15.172:6789/0 conn(0x7f6edc06fea0 legacy=0x7f6edc161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:58.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6edaffd700 1 -- 172.21.15.158:0/2576336593 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc15e5c0 con 0x7f6edc06f180 2023-10-16T11:21:58.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6edb7fe700 1 --2- 172.21.15.158:0/2576336593 >> v2:172.21.15.172:3300/0 conn(0x7f6edc15b540 0x7f6edc06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:58.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6edaffd700 1 --2- 172.21.15.158:0/2576336593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06f180 0x7f6edc06a8d0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f6edc15ba60 tx=0x7f6ecc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc6998aeffc3e14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:58.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6eda7fc700 1 -- 172.21.15.158:0/2576336593 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc011830 con 0x7f6edc06f180 2023-10-16T11:21:58.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6eda7fc700 1 -- 172.21.15.158:0/2576336593 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc011990 con 0x7f6edc06f180 2023-10-16T11:21:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6eda7fc700 1 -- 172.21.15.158:0/2576336593 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc011d20 con 0x7f6edc06f180 2023-10-16T11:21:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6ee1c29700 1 -- 172.21.15.158:0/2576336593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06f180 msgr2=0x7f6edc06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/2576336593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06f180 0x7f6edc06a8d0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f6edc15ba60 tx=0x7f6ecc009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6ee1c29700 1 -- 172.21.15.158:0/2576336593 shutdown_connections 2023-10-16T11:21:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/2576336593 >> v2:172.21.15.172:3300/0 conn(0x7f6edc15b540 0x7f6edc06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/2576336593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06f180 0x7f6edc06a8d0 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6ee1c29700 1 -- 172.21.15.158:0/2576336593 >> 172.21.15.158:0/2576336593 conn(0x7f6edc0d03b0 msgr2=0x7f6edc0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.855+0000 7f6ee1c29700 1 -- 172.21.15.158:0/2576336593 shutdown_connections 2023-10-16T11:21:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.856+0000 7f6ee1c29700 1 -- 172.21.15.158:0/2576336593 wait complete. 2023-10-16T11:21:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.857+0000 7f6ee1c29700 1 Processor -- start 2023-10-16T11:21:58.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.857+0000 7f6ee1c29700 1 -- start start 2023-10-16T11:21:58.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6edc06f180 0x7f6edc1574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:58.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06fea0 0x7f6edc0ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:58.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6edc15b540 0x7f6edc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:58.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0ade10 con 0x7f6edc06fea0 2023-10-16T11:21:58.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0ad200 con 0x7f6edc06f180 2023-10-16T11:21:58.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0ad4e0 con 0x7f6edc15b540 2023-10-16T11:21:58.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06fea0 0x7f6edc0ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:58.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06fea0 0x7f6edc0ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43762/0 (socket says 172.21.15.158:43762) 2023-10-16T11:21:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 -- 172.21.15.158:0/1987299716 learned_addr learned my addr 172.21.15.158:0/1987299716 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edb7fe700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6edc06f180 0x7f6edc1574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edbfff700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6edc15b540 0x7f6edc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 -- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6edc15b540 msgr2=0x7f6edc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6edc15b540 0x7f6edc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 -- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6edc06f180 msgr2=0x7f6edc1574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:58.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6edc06f180 0x7f6edc1574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:58.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 -- 172.21.15.158:0/1987299716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc009550 con 0x7f6edc06fea0 2023-10-16T11:21:58.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6edaffd700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06fea0 0x7f6edc0ae2a0 secure :-1 s=READY pgs=3291 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009fd0 tx=0x7f6ecc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ec3fff700 1 -- 172.21.15.158:0/1987299716 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc0036a0 con 0x7f6edc06fea0 2023-10-16T11:21:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ec3fff700 1 -- 172.21.15.158:0/1987299716 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc003800 con 0x7f6edc06fea0 2023-10-16T11:21:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc0ad7c0 con 0x7f6edc06fea0 2023-10-16T11:21:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ec3fff700 1 -- 172.21.15.158:0/1987299716 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc003b70 con 0x7f6edc06fea0 2023-10-16T11:21:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.858+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc0ae7e0 con 0x7f6edc06fea0 2023-10-16T11:21:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.859+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc0ae970 con 0x7f6edc06fea0 2023-10-16T11:21:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.860+0000 7f6ec3fff700 1 -- 172.21.15.158:0/1987299716 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ecc003cd0 con 0x7f6edc06fea0 2023-10-16T11:21:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.860+0000 7f6ec3fff700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6ec806b020 0x7f6ec806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.860+0000 7f6ec3fff700 1 -- 172.21.15.158:0/1987299716 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f6ecc021030 con 0x7f6edc06fea0 2023-10-16T11:21:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.860+0000 7f6edb7fe700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6ec806b020 0x7f6ec806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:58.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.861+0000 7f6edb7fe700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6ec806b020 0x7f6ec806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0006fd0 tx=0x7f6ed0008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:58.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:58.865+0000 7f6ec3fff700 1 -- 172.21.15.158:0/1987299716 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ecc055f70 con 0x7f6edc06fea0 2023-10-16T11:21:59.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.103+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6edc0681b0 con 0x7f6edc06fea0 2023-10-16T11:21:59.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.748+0000 7f6ec3fff700 1 -- 172.21.15.158:0/1987299716 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v154) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6ecc059590 con 0x7f6edc06fea0 2023-10-16T11:21:59.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6ec806b020 msgr2=0x7f6ec806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:59.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6ec806b020 0x7f6ec806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0006fd0 tx=0x7f6ed0008040 comp rx=0 tx=0).stop 2023-10-16T11:21:59.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06fea0 msgr2=0x7f6edc0ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:59.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06fea0 0x7f6edc0ae2a0 secure :-1 s=READY pgs=3291 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009fd0 tx=0x7f6ecc004750 comp rx=0 tx=0).stop 2023-10-16T11:21:59.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 shutdown_connections 2023-10-16T11:21:59.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6edc15b540 0x7f6edc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6edc06f180 0x7f6edc1574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f6ec806b020 0x7f6ec806d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 --2- 172.21.15.158:0/1987299716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6edc06fea0 0x7f6edc0ae2a0 unknown :-1 s=CLOSED pgs=3291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.751+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 >> 172.21.15.158:0/1987299716 conn(0x7f6edc0d03b0 msgr2=0x7f6edc06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:59.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.752+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 shutdown_connections 2023-10-16T11:21:59.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.752+0000 7f6ee1c29700 1 -- 172.21.15.158:0/1987299716 wait complete. 2023-10-16T11:21:59.766 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T11:21:59.767 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:21:59.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.886+0000 7f14d7698700 1 Processor -- start 2023-10-16T11:21:59.888 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.886+0000 7f14d7698700 1 -- start start 2023-10-16T11:21:59.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d7698700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f14d015b350 0x7f14d00ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:59.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d7698700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14d015b730 0x7f14d00cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:59.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d7698700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d00ca780 con 0x7f14d00ce460 2023-10-16T11:21:59.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d7698700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d006aa90 con 0x7f14d015b730 2023-10-16T11:21:59.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d7698700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d00cf820 con 0x7f14d015b350 2023-10-16T11:21:59.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d4c33700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14d015b730 0x7f14d00cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:59.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d4c33700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14d015b730 0x7f14d00cdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43770/0 (socket says 172.21.15.158:43770) 2023-10-16T11:21:59.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d4c33700 1 -- 172.21.15.158:0/2170875414 learned_addr learned my addr 172.21.15.158:0/2170875414 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:59.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.887+0000 7f14d5434700 1 --2- 172.21.15.158:0/2170875414 >> v2:172.21.15.172:3300/0 conn(0x7f14d015b350 0x7f14d00ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:59.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d4c33700 1 -- 172.21.15.158:0/2170875414 >> v2:172.21.15.172:3300/0 conn(0x7f14d015b350 msgr2=0x7f14d00ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:59.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d4c33700 1 --2- 172.21.15.158:0/2170875414 >> v2:172.21.15.172:3300/0 conn(0x7f14d015b350 0x7f14d00ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d4c33700 1 -- 172.21.15.158:0/2170875414 >> v1:172.21.15.172:6789/0 conn(0x7f14d00ce460 legacy=0x7f14d015f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:21:59.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d4c33700 1 -- 172.21.15.158:0/2170875414 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00cf260 con 0x7f14d015b730 2023-10-16T11:21:59.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d4c33700 1 --2- 172.21.15.158:0/2170875414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b730 0x7f14d00cdf20 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f14d0066e00 tx=0x7f14cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73eff34b4cddb57e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:59.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14c7fff700 1 -- 172.21.15.158:0/2170875414 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004230 con 0x7f14d015b730 2023-10-16T11:21:59.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14c7fff700 1 -- 172.21.15.158:0/2170875414 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc004390 con 0x7f14d015b730 2023-10-16T11:21:59.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14c7fff700 1 -- 172.21.15.158:0/2170875414 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc0036a0 con 0x7f14d015b730 2023-10-16T11:21:59.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d7698700 1 -- 172.21.15.158:0/2170875414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b730 msgr2=0x7f14d00cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:59.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d7698700 1 --2- 172.21.15.158:0/2170875414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b730 0x7f14d00cdf20 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f14d0066e00 tx=0x7f14cc009bd0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d7698700 1 -- 172.21.15.158:0/2170875414 shutdown_connections 2023-10-16T11:21:59.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d7698700 1 --2- 172.21.15.158:0/2170875414 >> v2:172.21.15.172:3300/0 conn(0x7f14d015b350 0x7f14d00ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d7698700 1 --2- 172.21.15.158:0/2170875414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b730 0x7f14d00cdf20 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.888+0000 7f14d7698700 1 -- 172.21.15.158:0/2170875414 >> 172.21.15.158:0/2170875414 conn(0x7f14d00cc4b0 msgr2=0x7f14d00cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:21:59.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.889+0000 7f14d7698700 1 -- 172.21.15.158:0/2170875414 shutdown_connections 2023-10-16T11:21:59.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.889+0000 7f14d7698700 1 -- 172.21.15.158:0/2170875414 wait complete. 2023-10-16T11:21:59.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.890+0000 7f14d7698700 1 Processor -- start 2023-10-16T11:21:59.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.890+0000 7f14d7698700 1 -- start start 2023-10-16T11:21:59.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d7698700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14d00ce460 0x7f14d014a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:59.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d7698700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b350 0x7f14d00c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:59.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d7698700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14d015b730 0x7f14d00b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:59.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d7698700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d014aa20 con 0x7f14d015b350 2023-10-16T11:21:59.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d7698700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00c8740 con 0x7f14d00ce460 2023-10-16T11:21:59.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d7698700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00c8a20 con 0x7f14d015b730 2023-10-16T11:21:59.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d4c33700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b350 0x7f14d00c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:59.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d4c33700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b350 0x7f14d00c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43776/0 (socket says 172.21.15.158:43776) 2023-10-16T11:21:59.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d4c33700 1 -- 172.21.15.158:0/278175377 learned_addr learned my addr 172.21.15.158:0/278175377 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:21:59.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d5434700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14d00ce460 0x7f14d014a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:59.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d4c33700 1 -- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14d015b730 msgr2=0x7f14d00b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:59.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.891+0000 7f14d5c35700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14d015b730 0x7f14d00b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:59.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d4c33700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14d015b730 0x7f14d00b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d4c33700 1 -- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14d00ce460 msgr2=0x7f14d014a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:21:59.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d4c33700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14d00ce460 0x7f14d014a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:21:59.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d4c33700 1 -- 172.21.15.158:0/278175377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc009550 con 0x7f14d015b350 2023-10-16T11:21:59.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d5434700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14d00ce460 0x7f14d014a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:21:59.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d4c33700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b350 0x7f14d00c7c80 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009130 tx=0x7f14cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:59.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14c5ffb700 1 -- 172.21.15.158:0/278175377 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc003d40 con 0x7f14d015b350 2023-10-16T11:21:59.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14c5ffb700 1 -- 172.21.15.158:0/278175377 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc01f070 con 0x7f14d015b350 2023-10-16T11:21:59.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d00c8d00 con 0x7f14d015b350 2023-10-16T11:21:59.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14c5ffb700 1 -- 172.21.15.158:0/278175377 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc011870 con 0x7f14d015b350 2023-10-16T11:21:59.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.892+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d00c92c0 con 0x7f14d015b350 2023-10-16T11:21:59.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.893+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d00681b0 con 0x7f14d015b350 2023-10-16T11:21:59.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.893+0000 7f14c5ffb700 1 -- 172.21.15.158:0/278175377 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f14cc021030 con 0x7f14d015b350 2023-10-16T11:21:59.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.894+0000 7f14c5ffb700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f14b406b020 0x7f14b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:21:59.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.894+0000 7f14c5ffb700 1 -- 172.21.15.158:0/278175377 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f14cc08af00 con 0x7f14d015b350 2023-10-16T11:21:59.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.894+0000 7f14d5434700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f14b406b020 0x7f14b406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:21:59.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.894+0000 7f14d5434700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f14b406b020 0x7f14b406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f14c0009a00 tx=0x7f14c0008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:21:59.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:21:59.898+0000 7f14c5ffb700 1 -- 172.21.15.158:0/278175377 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14cc055c10 con 0x7f14d015b350 2023-10-16T11:22:00.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.133+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14d00cd820 con 0x7f14d015b350 2023-10-16T11:22:00.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.134+0000 7f14c5ffb700 1 -- 172.21.15.158:0/278175377 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f14cc059230 con 0x7f14d015b350 2023-10-16T11:22:00.136 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:00.136 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":154,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1253298118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1253298118},{"type":"v1","addr":"172.21.15.158:6837","nonce":1253298118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":13731,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2914884530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2914884530},{"type":"v1","addr":"172.21.15.158:6836","nonce":2914884530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/536559492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":536559492},{"type":"v1","addr":"172.21.15.172:6834","nonce":536559492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:56.094141+0000","modified":"2023-10-16T11:21:59.105113+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":40202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_40202":{"gid":40202,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/284711673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":284711673},{"type":"v1","addr":"172.21.15.172:6835","nonce":284711673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T11:22:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f14b406b020 msgr2=0x7f14b406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f14b406b020 0x7f14b406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f14c0009a00 tx=0x7f14c0008040 comp rx=0 tx=0).stop 2023-10-16T11:22:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b350 msgr2=0x7f14d00c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b350 0x7f14d00c7c80 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009130 tx=0x7f14cc004750 comp rx=0 tx=0).stop 2023-10-16T11:22:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 shutdown_connections 2023-10-16T11:22:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14d015b730 0x7f14d00b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14d00ce460 0x7f14d014a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f14b406b020 0x7f14b406d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 --2- 172.21.15.158:0/278175377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14d015b350 0x7f14d00c7c80 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 >> 172.21.15.158:0/278175377 conn(0x7f14d00cc4b0 msgr2=0x7f14d00af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 shutdown_connections 2023-10-16T11:22:00.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.136+0000 7f14d7698700 1 -- 172.21.15.158:0/278175377 wait complete. 2023-10-16T11:22:00.141 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 154 2023-10-16T11:22:00.150 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:22:00.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.268+0000 7fb5fc9c1700 1 Processor -- start 2023-10-16T11:22:00.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.268+0000 7fb5fc9c1700 1 -- start start 2023-10-16T11:22:00.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.269+0000 7fb5fc9c1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb5f806ab40 0x7fb5f806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.269+0000 7fb5fc9c1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f80729b0 0x7fb5f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.269+0000 7fb5fc9c1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f8071e50 con 0x7fb5f806e5d0 2023-10-16T11:22:00.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.269+0000 7fb5fc9c1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f8161270 con 0x7fb5f806ab40 2023-10-16T11:22:00.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.269+0000 7fb5fc9c1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8161510 con 0x7fb5f80729b0 2023-10-16T11:22:00.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f6d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f80729b0 0x7fb5f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb5f806ab40 0x7fb5f806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb5f806ab40 0x7fb5f806d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:44962/0 (socket says 172.21.15.158:44962) 2023-10-16T11:22:00.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f659c700 1 -- 172.21.15.158:0/1500699175 learned_addr learned my addr 172.21.15.158:0/1500699175 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:00.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f6d9d700 1 -- 172.21.15.158:0/1500699175 >> v2:172.21.15.172:3300/0 conn(0x7fb5f806ab40 msgr2=0x7fb5f806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:00.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f6d9d700 1 --2- 172.21.15.158:0/1500699175 >> v2:172.21.15.172:3300/0 conn(0x7fb5f806ab40 0x7fb5f806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f6d9d700 1 -- 172.21.15.158:0/1500699175 >> v1:172.21.15.172:6789/0 conn(0x7fb5f806e5d0 legacy=0x7fb5f80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f6d9d700 1 -- 172.21.15.158:0/1500699175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f81617f0 con 0x7fb5f80729b0 2023-10-16T11:22:00.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f659c700 1 --2- 172.21.15.158:0/1500699175 >> v2:172.21.15.172:3300/0 conn(0x7fb5f806ab40 0x7fb5f806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:00.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f6d9d700 1 --2- 172.21.15.158:0/1500699175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f80729b0 0x7fb5f8071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009d90 tx=0x7fb5f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48c679c719b159c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:00.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f559a700 1 -- 172.21.15.158:0/1500699175 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f000e070 con 0x7fb5f80729b0 2023-10-16T11:22:00.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f559a700 1 -- 172.21.15.158:0/1500699175 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f000bb20 con 0x7fb5f80729b0 2023-10-16T11:22:00.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.270+0000 7fb5f559a700 1 -- 172.21.15.158:0/1500699175 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0012430 con 0x7fb5f80729b0 2023-10-16T11:22:00.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/1500699175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f80729b0 msgr2=0x7fb5f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/1500699175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f80729b0 0x7fb5f8071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009d90 tx=0x7fb5f000b330 comp rx=0 tx=0).stop 2023-10-16T11:22:00.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/1500699175 shutdown_connections 2023-10-16T11:22:00.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/1500699175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f80729b0 0x7fb5f8071910 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/1500699175 >> v2:172.21.15.172:3300/0 conn(0x7fb5f806ab40 0x7fb5f806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/1500699175 >> 172.21.15.158:0/1500699175 conn(0x7fb5f80d02b0 msgr2=0x7fb5f80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:00.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/1500699175 shutdown_connections 2023-10-16T11:22:00.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.271+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/1500699175 wait complete. 2023-10-16T11:22:00.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.272+0000 7fb5fc9c1700 1 Processor -- start 2023-10-16T11:22:00.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.272+0000 7fb5fc9c1700 1 -- start start 2023-10-16T11:22:00.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5fc9c1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f806ab40 0x7fb5f80b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5fc9c1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f806e5d0 0x7fb5f80b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5fc9c1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f80729b0 0x7fb5f80c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5fc9c1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80b4300 con 0x7fb5f80729b0 2023-10-16T11:22:00.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5fc9c1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80b5d10 con 0x7fb5f806e5d0 2023-10-16T11:22:00.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5fc9c1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80b36f0 con 0x7fb5f806ab40 2023-10-16T11:22:00.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5f6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f80729b0 0x7fb5f80c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5f6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f80729b0 0x7fb5f80c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43792/0 (socket says 172.21.15.158:43792) 2023-10-16T11:22:00.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5f6d9d700 1 -- 172.21.15.158:0/413884874 learned_addr learned my addr 172.21.15.158:0/413884874 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:00.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5f659c700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f806ab40 0x7fb5f80b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.273+0000 7fb5f5d9b700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f806e5d0 0x7fb5f80b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5f6d9d700 1 -- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f806ab40 msgr2=0x7fb5f80b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5f6d9d700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f806ab40 0x7fb5f80b57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5f6d9d700 1 -- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f806e5d0 msgr2=0x7fb5f80b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5f6d9d700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f806e5d0 0x7fb5f80b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5f6d9d700 1 -- 172.21.15.158:0/413884874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e0009550 con 0x7fb5f80729b0 2023-10-16T11:22:00.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5f6d9d700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f80729b0 0x7fb5f80c5770 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0014040 tx=0x7fb5f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:00.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5eeffd700 1 -- 172.21.15.158:0/413884874 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0012430 con 0x7fb5f80729b0 2023-10-16T11:22:00.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5eeffd700 1 -- 172.21.15.158:0/413884874 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f0020070 con 0x7fb5f80729b0 2023-10-16T11:22:00.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f0009a70 con 0x7fb5f80729b0 2023-10-16T11:22:00.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5eeffd700 1 -- 172.21.15.158:0/413884874 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0003ea0 con 0x7fb5f80729b0 2023-10-16T11:22:00.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.274+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f80b3cb0 con 0x7fb5f80729b0 2023-10-16T11:22:00.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.275+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f80b44b0 con 0x7fb5f80729b0 2023-10-16T11:22:00.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.275+0000 7fb5eeffd700 1 -- 172.21.15.158:0/413884874 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb5f0004280 con 0x7fb5f80729b0 2023-10-16T11:22:00.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.276+0000 7fb5eeffd700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fb5d806b020 0x7fb5d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.276+0000 7fb5eeffd700 1 -- 172.21.15.158:0/413884874 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fb5f0022070 con 0x7fb5f80729b0 2023-10-16T11:22:00.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.276+0000 7fb5f659c700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fb5d806b020 0x7fb5d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.276+0000 7fb5f659c700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fb5d806b020 0x7fb5d806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009cf0 tx=0x7fb5e0008040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:00.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.280+0000 7fb5eeffd700 1 -- 172.21.15.158:0/413884874 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5f0056890 con 0x7fb5f80729b0 2023-10-16T11:22:00.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.472+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5f80681b0 con 0x7fb5f80729b0 2023-10-16T11:22:00.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.473+0000 7fb5eeffd700 1 -- 172.21.15.158:0/413884874 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+17475 (secure 0 0 0) 0x7fb5f0059eb0 con 0x7fb5f80729b0 2023-10-16T11:22:00.475 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:00.475 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":270,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:21:57.695975+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:54.523055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:55.050879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:22:00.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fb5d806b020 msgr2=0x7fb5d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fb5d806b020 0x7fb5d806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009cf0 tx=0x7fb5e0008040 comp rx=0 tx=0).stop 2023-10-16T11:22:00.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f80729b0 msgr2=0x7fb5f80c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f80729b0 0x7fb5f80c5770 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0014040 tx=0x7fb5f0004ad0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 shutdown_connections 2023-10-16T11:22:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fb5d806b020 0x7fb5d806d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5f80729b0 0x7fb5f80c5770 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5f806ab40 0x7fb5f80b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 --2- 172.21.15.158:0/413884874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5f806e5d0 0x7fb5f80b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 >> 172.21.15.158:0/413884874 conn(0x7fb5f80d02b0 msgr2=0x7fb5f81568c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 shutdown_connections 2023-10-16T11:22:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.475+0000 7fb5fc9c1700 1 -- 172.21.15.158:0/413884874 wait complete. 2023-10-16T11:22:00.489 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T11:22:00.490 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:22:00.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.608+0000 7fc90fdb4700 1 Processor -- start 2023-10-16T11:22:00.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.608+0000 7fc90fdb4700 1 -- start start 2023-10-16T11:22:00.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90fdb4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc90815b730 0x7fc9080ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90fdb4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 0x7fc90806af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90fdb4700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080ca780 con 0x7fc90815b350 2023-10-16T11:22:00.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90fdb4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90806aa90 con 0x7fc90815b730 2023-10-16T11:22:00.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90fdb4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9080cf820 con 0x7fc9080ce630 2023-10-16T11:22:00.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90db50700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc90815b350 0x7fc9080cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43972/0 (socket says 172.21.15.158:43972) 2023-10-16T11:22:00.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90db50700 1 -- 172.21.15.158:0/2232731466 learned_addr learned my addr 172.21.15.158:0/2232731466 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:00.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90d34f700 1 --2- 172.21.15.158:0/2232731466 >> v2:172.21.15.172:3300/0 conn(0x7fc90815b730 0x7fc9080ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90e351700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 0x7fc90806af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.609+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707128943 0 0) 0x7fc9080ca780 con 0x7fc90815b350 2023-10-16T11:22:00.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f40035b0 con 0x7fc90815b350 2023-10-16T11:22:00.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2445406423 0 0) 0x7fc8f40035b0 con 0x7fc90815b350 2023-10-16T11:22:00.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 msgr2=0x7fc90806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:00.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 0x7fc90806af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 >> v2:172.21.15.172:3300/0 conn(0x7fc90815b730 msgr2=0x7fc9080ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:00.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 --2- 172.21.15.158:0/2232731466 >> v2:172.21.15.172:3300/0 conn(0x7fc90815b730 0x7fc9080ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080cf260 con 0x7fc90815b350 2023-10-16T11:22:00.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fc8f8001670 con 0x7fc90815b350 2023-10-16T11:22:00.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 >> v1:172.21.15.158:6789/0 conn(0x7fc90815b350 legacy=0x7fc9080cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f4003a00 0x7fc8f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4004850 0x7fc8f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 0x7fc8f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90806aa90 con 0x7fc8f4003a00 2023-10-16T11:22:00.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9080cf820 con 0x7fc8f4004850 2023-10-16T11:22:00.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f400ba90 con 0x7fc9080ce630 2023-10-16T11:22:00.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90db50700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f4003a00 0x7fc8f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90d34f700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4004850 0x7fc8f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90e351700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 0x7fc8f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90db50700 1 -- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 msgr2=0x7fc8f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:00.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90db50700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9080ce630 0x7fc8f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90db50700 1 -- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4004850 msgr2=0x7fc8f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:00.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90db50700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4004850 0x7fc8f4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90db50700 1 -- 172.21.15.158:0/2232731466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40035b0 con 0x7fc8f4003a00 2023-10-16T11:22:00.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90d34f700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8f4004850 0x7fc8f4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:00.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90db50700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f4003a00 0x7fc8f4003e20 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800c2f0 tx=0x7fc8f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea161d2a7b54f009 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:00.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.610+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f800ce30 con 0x7fc8f4003a00 2023-10-16T11:22:00.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f8005960 con 0x7fc8f4003a00 2023-10-16T11:22:00.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90cb4e700 1 -- 172.21.15.158:0/2232731466 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8005d10 con 0x7fc8f4003a00 2023-10-16T11:22:00.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90fdb4700 1 -- 172.21.15.158:0/2232731466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f4003a00 msgr2=0x7fc8f4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90fdb4700 1 --2- 172.21.15.158:0/2232731466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8f4003a00 0x7fc8f4003e20 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800c2f0 tx=0x7fc8f4005880 comp rx=0 tx=0).stop 2023-10-16T11:22:00.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90e351700 1 -- 172.21.15.158:0/2232731466 reap_dead start 2023-10-16T11:22:00.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90fdb4700 1 -- 172.21.15.158:0/2232731466 shutdown_connections 2023-10-16T11:22:00.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90fdb4700 1 -- 172.21.15.158:0/2232731466 >> 172.21.15.158:0/2232731466 conn(0x7fc9080cc4b0 msgr2=0x7fc9080cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:00.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90fdb4700 1 -- 172.21.15.158:0/2232731466 shutdown_connections 2023-10-16T11:22:00.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.611+0000 7fc90fdb4700 1 -- 172.21.15.158:0/2232731466 wait complete. 2023-10-16T11:22:00.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.612+0000 7fc90fdb4700 1 Processor -- start 2023-10-16T11:22:00.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.612+0000 7fc90fdb4700 1 -- start start 2023-10-16T11:22:00.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90fdb4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f4003a00 0x7fc90814cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90fdb4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc908153220 0x7fc90814d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90fdb4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc90814dc20 0x7fc90814e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90fdb4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9081557a0 con 0x7fc908153220 2023-10-16T11:22:00.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90fdb4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc908156490 con 0x7fc90814dc20 2023-10-16T11:22:00.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90fdb4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90814d330 con 0x7fc8f4003a00 2023-10-16T11:22:00.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc908153220 0x7fc90814d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc908153220 0x7fc90814d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43810/0 (socket says 172.21.15.158:43810) 2023-10-16T11:22:00.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 -- 172.21.15.158:0/972126381 learned_addr learned my addr 172.21.15.158:0/972126381 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:00.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90e351700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc90814dc20 0x7fc90814e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90db50700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f4003a00 0x7fc90814cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 -- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f4003a00 msgr2=0x7fc90814cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f4003a00 0x7fc90814cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 -- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc90814dc20 msgr2=0x7fc90814e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc90814dc20 0x7fc90814e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90d34f700 1 -- 172.21.15.158:0/972126381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9081559b0 con 0x7fc908153220 2023-10-16T11:22:00.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.613+0000 7fc90db50700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f4003a00 0x7fc90814cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:00.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.614+0000 7fc90d34f700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc908153220 0x7fc90814d6e0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7fc904010920 tx=0x7fc904010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:00.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.614+0000 7fc8fe7fc700 1 -- 172.21.15.158:0/972126381 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904010ec0 con 0x7fc908153220 2023-10-16T11:22:00.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.614+0000 7fc8fe7fc700 1 -- 172.21.15.158:0/972126381 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc904004030 con 0x7fc908153220 2023-10-16T11:22:00.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.614+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc908155c90 con 0x7fc908153220 2023-10-16T11:22:00.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.614+0000 7fc8fe7fc700 1 -- 172.21.15.158:0/972126381 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9040043e0 con 0x7fc908153220 2023-10-16T11:22:00.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.614+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90814be10 con 0x7fc908153220 2023-10-16T11:22:00.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.615+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9080681b0 con 0x7fc908153220 2023-10-16T11:22:00.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.615+0000 7fc8fe7fc700 1 -- 172.21.15.158:0/972126381 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc90400e870 con 0x7fc908153220 2023-10-16T11:22:00.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.616+0000 7fc8fe7fc700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc8ec06b020 0x7fc8ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:00.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.616+0000 7fc90db50700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc8ec06b020 0x7fc8ec06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:00.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.616+0000 7fc8fe7fc700 1 -- 172.21.15.158:0/972126381 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fc904089b70 con 0x7fc908153220 2023-10-16T11:22:00.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.616+0000 7fc90db50700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc8ec06b020 0x7fc8ec06d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8015040 tx=0x7fc8f800a000 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:00.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.620+0000 7fc8fe7fc700 1 -- 172.21.15.158:0/972126381 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9040547e0 con 0x7fc908153220 2023-10-16T11:22:00.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.810+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 --> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9080cf540 con 0x7fc8ec06b020 2023-10-16T11:22:00.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.819+0000 7fc8fe7fc700 1 -- 172.21.15.158:0/972126381 <== mgr.13500 v2:172.21.15.158:6832/778561374 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289893 (secure 0 0 0) 0x7fc8f8001430 con 0x7fc8ec06b020 2023-10-16T11:22:00.824 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:00.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc8ec06b020 msgr2=0x7fc8ec06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc8ec06b020 0x7fc8ec06d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8015040 tx=0x7fc8f800a000 comp rx=0 tx=0).stop 2023-10-16T11:22:00.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc908153220 msgr2=0x7fc90814d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:00.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc908153220 0x7fc90814d6e0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7fc904010920 tx=0x7fc904010ce0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 shutdown_connections 2023-10-16T11:22:00.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc90814dc20 0x7fc90814e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8f4003a00 0x7fc90814cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fc8ec06b020 0x7fc8ec06d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 --2- 172.21.15.158:0/972126381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc908153220 0x7fc90814d6e0 unknown :-1 s=CLOSED pgs=3297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:00.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 >> 172.21.15.158:0/972126381 conn(0x7fc9080cc4b0 msgr2=0x7fc90806f500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:00.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 shutdown_connections 2023-10-16T11:22:00.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.822+0000 7fc90fdb4700 1 -- 172.21.15.158:0/972126381 wait complete. 2023-10-16T11:22:00.829 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:22:00.836 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-16T11:22:00.643962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":444068,"kb_used_data":27996,"kb_used_omap":12,"kb_used_meta":415923,"kb_avail":754530652,"statfs":{"total":773094113280,"available":772639387648,"internally_reserved":0,"allocated":28667904,"data_stored":15993192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":425905719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.964027"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858210+0000","last_change":"2023-10-16T11:21:56.711930+0000","last_active":"2023-10-16T11:21:57.858210+0000","last_peered":"2023-10-16T11:21:57.858210+0000","last_clean":"2023-10-16T11:21:57.858210+0000","last_became_active":"2023-10-16T11:21:56.711653+0000","last_became_peered":"2023-10-16T11:21:56.711653+0000","last_unstale":"2023-10-16T11:21:57.858210+0000","last_undegraded":"2023-10-16T11:21:57.858210+0000","last_fullsized":"2023-10-16T11:21:57.858210+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:36.567286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858409+0000","last_change":"2023-10-16T11:21:56.710755+0000","last_active":"2023-10-16T11:21:57.858409+0000","last_peered":"2023-10-16T11:21:57.858409+0000","last_clean":"2023-10-16T11:21:57.858409+0000","last_became_active":"2023-10-16T11:21:56.710104+0000","last_became_peered":"2023-10-16T11:21:56.710104+0000","last_unstale":"2023-10-16T11:21:57.858409+0000","last_undegraded":"2023-10-16T11:21:57.858409+0000","last_fullsized":"2023-10-16T11:21:57.858409+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:33.478583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857244+0000","last_change":"2023-10-16T11:21:56.705147+0000","last_active":"2023-10-16T11:21:57.857244+0000","last_peered":"2023-10-16T11:21:57.857244+0000","last_clean":"2023-10-16T11:21:57.857244+0000","last_became_active":"2023-10-16T11:21:56.704777+0000","last_became_peered":"2023-10-16T11:21:56.704777+0000","last_unstale":"2023-10-16T11:21:57.857244+0000","last_undegraded":"2023-10-16T11:21:57.857244+0000","last_fullsized":"2023-10-16T11:21:57.857244+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:37.910699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704142+0000","last_change":"2023-10-16T11:21:56.717562+0000","last_active":"2023-10-16T11:21:57.704142+0000","last_peered":"2023-10-16T11:21:57.704142+0000","last_clean":"2023-10-16T11:21:57.704142+0000","last_became_active":"2023-10-16T11:21:56.717233+0000","last_became_peered":"2023-10-16T11:21:56.717233+0000","last_unstale":"2023-10-16T11:21:57.704142+0000","last_undegraded":"2023-10-16T11:21:57.704142+0000","last_fullsized":"2023-10-16T11:21:57.704142+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:02.423045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703681+0000","last_change":"2023-10-16T11:21:56.711666+0000","last_active":"2023-10-16T11:21:57.703681+0000","last_peered":"2023-10-16T11:21:57.703681+0000","last_clean":"2023-10-16T11:21:57.703681+0000","last_became_active":"2023-10-16T11:21:56.711375+0000","last_became_peered":"2023-10-16T11:21:56.711375+0000","last_unstale":"2023-10-16T11:21:57.703681+0000","last_undegraded":"2023-10-16T11:21:57.703681+0000","last_fullsized":"2023-10-16T11:21:57.703681+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:05.572254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858020+0000","last_change":"2023-10-16T11:21:56.707395+0000","last_active":"2023-10-16T11:21:57.858020+0000","last_peered":"2023-10-16T11:21:57.858020+0000","last_clean":"2023-10-16T11:21:57.858020+0000","last_became_active":"2023-10-16T11:21:56.707166+0000","last_became_peered":"2023-10-16T11:21:56.707166+0000","last_unstale":"2023-10-16T11:21:57.858020+0000","last_undegraded":"2023-10-16T11:21:57.858020+0000","last_fullsized":"2023-10-16T11:21:57.858020+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:17.294904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857615+0000","last_change":"2023-10-16T11:21:56.709062+0000","last_active":"2023-10-16T11:21:57.857615+0000","last_peered":"2023-10-16T11:21:57.857615+0000","last_clean":"2023-10-16T11:21:57.857615+0000","last_became_active":"2023-10-16T11:21:56.708771+0000","last_became_peered":"2023-10-16T11:21:56.708771+0000","last_unstale":"2023-10-16T11:21:57.857615+0000","last_undegraded":"2023-10-16T11:21:57.857615+0000","last_fullsized":"2023-10-16T11:21:57.857615+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:48.444835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856875+0000","last_change":"2023-10-16T11:21:56.712327+0000","last_active":"2023-10-16T11:21:57.856875+0000","last_peered":"2023-10-16T11:21:57.856875+0000","last_clean":"2023-10-16T11:21:57.856875+0000","last_became_active":"2023-10-16T11:21:56.712079+0000","last_became_peered":"2023-10-16T11:21:56.712079+0000","last_unstale":"2023-10-16T11:21:57.856875+0000","last_undegraded":"2023-10-16T11:21:57.856875+0000","last_fullsized":"2023-10-16T11:21:57.856875+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:14.692681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703582+0000","last_change":"2023-10-16T11:21:56.706652+0000","last_active":"2023-10-16T11:21:57.703582+0000","last_peered":"2023-10-16T11:21:57.703582+0000","last_clean":"2023-10-16T11:21:57.703582+0000","last_became_active":"2023-10-16T11:21:56.706377+0000","last_became_peered":"2023-10-16T11:21:56.706377+0000","last_unstale":"2023-10-16T11:21:57.703582+0000","last_undegraded":"2023-10-16T11:21:57.703582+0000","last_fullsized":"2023-10-16T11:21:57.703582+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:50.953400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857927+0000","last_change":"2023-10-16T11:21:56.704287+0000","last_active":"2023-10-16T11:21:57.857927+0000","last_peered":"2023-10-16T11:21:57.857927+0000","last_clean":"2023-10-16T11:21:57.857927+0000","last_became_active":"2023-10-16T11:21:56.704017+0000","last_became_peered":"2023-10-16T11:21:56.704017+0000","last_unstale":"2023-10-16T11:21:57.857927+0000","last_undegraded":"2023-10-16T11:21:57.857927+0000","last_fullsized":"2023-10-16T11:21:57.857927+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:40.750204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703565+0000","last_change":"2023-10-16T11:21:56.709289+0000","last_active":"2023-10-16T11:21:57.703565+0000","last_peered":"2023-10-16T11:21:57.703565+0000","last_clean":"2023-10-16T11:21:57.703565+0000","last_became_active":"2023-10-16T11:21:56.708562+0000","last_became_peered":"2023-10-16T11:21:56.708562+0000","last_unstale":"2023-10-16T11:21:57.703565+0000","last_undegraded":"2023-10-16T11:21:57.703565+0000","last_fullsized":"2023-10-16T11:21:57.703565+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:59.525098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858022+0000","last_change":"2023-10-16T11:21:56.705829+0000","last_active":"2023-10-16T11:21:57.858022+0000","last_peered":"2023-10-16T11:21:57.858022+0000","last_clean":"2023-10-16T11:21:57.858022+0000","last_became_active":"2023-10-16T11:21:56.705573+0000","last_became_peered":"2023-10-16T11:21:56.705573+0000","last_unstale":"2023-10-16T11:21:57.858022+0000","last_undegraded":"2023-10-16T11:21:57.858022+0000","last_fullsized":"2023-10-16T11:21:57.858022+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:22.111658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703484+0000","last_change":"2023-10-16T11:21:56.712269+0000","last_active":"2023-10-16T11:21:57.703484+0000","last_peered":"2023-10-16T11:21:57.703484+0000","last_clean":"2023-10-16T11:21:57.703484+0000","last_became_active":"2023-10-16T11:21:56.711921+0000","last_became_peered":"2023-10-16T11:21:56.711921+0000","last_unstale":"2023-10-16T11:21:57.703484+0000","last_undegraded":"2023-10-16T11:21:57.703484+0000","last_fullsized":"2023-10-16T11:21:57.703484+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:12.715253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857499+0000","last_change":"2023-10-16T11:21:56.707667+0000","last_active":"2023-10-16T11:21:57.857499+0000","last_peered":"2023-10-16T11:21:57.857499+0000","last_clean":"2023-10-16T11:21:57.857499+0000","last_became_active":"2023-10-16T11:21:56.707393+0000","last_became_peered":"2023-10-16T11:21:56.707393+0000","last_unstale":"2023-10-16T11:21:57.857499+0000","last_undegraded":"2023-10-16T11:21:57.857499+0000","last_fullsized":"2023-10-16T11:21:57.857499+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:42.811481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857499+0000","last_change":"2023-10-16T11:21:56.697151+0000","last_active":"2023-10-16T11:21:57.857499+0000","last_peered":"2023-10-16T11:21:57.857499+0000","last_clean":"2023-10-16T11:21:57.857499+0000","last_became_active":"2023-10-16T11:21:56.696864+0000","last_became_peered":"2023-10-16T11:21:56.696864+0000","last_unstale":"2023-10-16T11:21:57.857499+0000","last_undegraded":"2023-10-16T11:21:57.857499+0000","last_fullsized":"2023-10-16T11:21:57.857499+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:24.629606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858075+0000","last_change":"2023-10-16T11:21:56.723781+0000","last_active":"2023-10-16T11:21:57.858075+0000","last_peered":"2023-10-16T11:21:57.858075+0000","last_clean":"2023-10-16T11:21:57.858075+0000","last_became_active":"2023-10-16T11:21:56.723570+0000","last_became_peered":"2023-10-16T11:21:56.723570+0000","last_unstale":"2023-10-16T11:21:57.858075+0000","last_undegraded":"2023-10-16T11:21:57.858075+0000","last_fullsized":"2023-10-16T11:21:57.858075+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:37:37.575821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703621+0000","last_change":"2023-10-16T11:21:56.706554+0000","last_active":"2023-10-16T11:21:57.703621+0000","last_peered":"2023-10-16T11:21:57.703621+0000","last_clean":"2023-10-16T11:21:57.703621+0000","last_became_active":"2023-10-16T11:21:56.706270+0000","last_became_peered":"2023-10-16T11:21:56.706270+0000","last_unstale":"2023-10-16T11:21:57.703621+0000","last_undegraded":"2023-10-16T11:21:57.703621+0000","last_fullsized":"2023-10-16T11:21:57.703621+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:57.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703342+0000","last_change":"2023-10-16T11:21:56.719017+0000","last_active":"2023-10-16T11:21:57.703342+0000","last_peered":"2023-10-16T11:21:57.703342+0000","last_clean":"2023-10-16T11:21:57.703342+0000","last_became_active":"2023-10-16T11:21:56.718730+0000","last_became_peered":"2023-10-16T11:21:56.718730+0000","last_unstale":"2023-10-16T11:21:57.703342+0000","last_undegraded":"2023-10-16T11:21:57.703342+0000","last_fullsized":"2023-10-16T11:21:57.703342+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:31.587975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857584+0000","last_change":"2023-10-16T11:21:56.705001+0000","last_active":"2023-10-16T11:21:57.857584+0000","last_peered":"2023-10-16T11:21:57.857584+0000","last_clean":"2023-10-16T11:21:57.857584+0000","last_became_active":"2023-10-16T11:21:56.704427+0000","last_became_peered":"2023-10-16T11:21:56.704427+0000","last_unstale":"2023-10-16T11:21:57.857584+0000","last_undegraded":"2023-10-16T11:21:57.857584+0000","last_fullsized":"2023-10-16T11:21:57.857584+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:46.844456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704300+0000","last_change":"2023-10-16T11:21:56.723216+0000","last_active":"2023-10-16T11:21:57.704300+0000","last_peered":"2023-10-16T11:21:57.704300+0000","last_clean":"2023-10-16T11:21:57.704300+0000","last_became_active":"2023-10-16T11:21:56.719340+0000","last_became_peered":"2023-10-16T11:21:56.719340+0000","last_unstale":"2023-10-16T11:21:57.704300+0000","last_undegraded":"2023-10-16T11:21:57.704300+0000","last_fullsized":"2023-10-16T11:21:57.704300+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:54:06.372659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857593+0000","last_change":"2023-10-16T11:21:56.697423+0000","last_active":"2023-10-16T11:21:57.857593+0000","last_peered":"2023-10-16T11:21:57.857593+0000","last_clean":"2023-10-16T11:21:57.857593+0000","last_became_active":"2023-10-16T11:21:56.697164+0000","last_became_peered":"2023-10-16T11:21:56.697164+0000","last_unstale":"2023-10-16T11:21:57.857593+0000","last_undegraded":"2023-10-16T11:21:57.857593+0000","last_fullsized":"2023-10-16T11:21:57.857593+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:03.898885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703636+0000","last_change":"2023-10-16T11:21:56.718829+0000","last_active":"2023-10-16T11:21:57.703636+0000","last_peered":"2023-10-16T11:21:57.703636+0000","last_clean":"2023-10-16T11:21:57.703636+0000","last_became_active":"2023-10-16T11:21:56.718536+0000","last_became_peered":"2023-10-16T11:21:56.718536+0000","last_unstale":"2023-10-16T11:21:57.703636+0000","last_undegraded":"2023-10-16T11:21:57.703636+0000","last_fullsized":"2023-10-16T11:21:57.703636+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:33.478583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703871+0000","last_change":"2023-10-16T11:21:56.718601+0000","last_active":"2023-10-16T11:21:57.703871+0000","last_peered":"2023-10-16T11:21:57.703871+0000","last_clean":"2023-10-16T11:21:57.703871+0000","last_became_active":"2023-10-16T11:21:56.718292+0000","last_became_peered":"2023-10-16T11:21:56.718292+0000","last_unstale":"2023-10-16T11:21:57.703871+0000","last_undegraded":"2023-10-16T11:21:57.703871+0000","last_fullsized":"2023-10-16T11:21:57.703871+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:44.757920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856825+0000","last_change":"2023-10-16T11:21:56.710984+0000","last_active":"2023-10-16T11:21:57.856825+0000","last_peered":"2023-10-16T11:21:57.856825+0000","last_clean":"2023-10-16T11:21:57.856825+0000","last_became_active":"2023-10-16T11:21:56.710434+0000","last_became_peered":"2023-10-16T11:21:56.710434+0000","last_unstale":"2023-10-16T11:21:57.856825+0000","last_undegraded":"2023-10-16T11:21:57.856825+0000","last_fullsized":"2023-10-16T11:21:57.856825+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:23.369255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703693+0000","last_change":"2023-10-16T11:21:56.696074+0000","last_active":"2023-10-16T11:21:57.703693+0000","last_peered":"2023-10-16T11:21:57.703693+0000","last_clean":"2023-10-16T11:21:57.703693+0000","last_became_active":"2023-10-16T11:21:56.695799+0000","last_became_peered":"2023-10-16T11:21:56.695799+0000","last_unstale":"2023-10-16T11:21:57.703693+0000","last_undegraded":"2023-10-16T11:21:57.703693+0000","last_fullsized":"2023-10-16T11:21:57.703693+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:15.737826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704044+0000","last_change":"2023-10-16T11:21:56.718865+0000","last_active":"2023-10-16T11:21:57.704044+0000","last_peered":"2023-10-16T11:21:57.704044+0000","last_clean":"2023-10-16T11:21:57.704044+0000","last_became_active":"2023-10-16T11:21:56.718629+0000","last_became_peered":"2023-10-16T11:21:56.718629+0000","last_unstale":"2023-10-16T11:21:57.704044+0000","last_undegraded":"2023-10-16T11:21:57.704044+0000","last_fullsized":"2023-10-16T11:21:57.704044+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:01.116596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703431+0000","last_change":"2023-10-16T11:21:56.704641+0000","last_active":"2023-10-16T11:21:57.703431+0000","last_peered":"2023-10-16T11:21:57.703431+0000","last_clean":"2023-10-16T11:21:57.703431+0000","last_became_active":"2023-10-16T11:21:56.704373+0000","last_became_peered":"2023-10-16T11:21:56.704373+0000","last_unstale":"2023-10-16T11:21:57.703431+0000","last_undegraded":"2023-10-16T11:21:57.703431+0000","last_fullsized":"2023-10-16T11:21:57.703431+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:29.686801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858441+0000","last_change":"2023-10-16T11:21:56.718162+0000","last_active":"2023-10-16T11:21:57.858441+0000","last_peered":"2023-10-16T11:21:57.858441+0000","last_clean":"2023-10-16T11:21:57.858441+0000","last_became_active":"2023-10-16T11:21:56.717879+0000","last_became_peered":"2023-10-16T11:21:56.717879+0000","last_unstale":"2023-10-16T11:21:57.858441+0000","last_undegraded":"2023-10-16T11:21:57.858441+0000","last_fullsized":"2023-10-16T11:21:57.858441+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:42.844037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857687+0000","last_change":"2023-10-16T11:21:56.705175+0000","last_active":"2023-10-16T11:21:57.857687+0000","last_peered":"2023-10-16T11:21:57.857687+0000","last_clean":"2023-10-16T11:21:57.857687+0000","last_became_active":"2023-10-16T11:21:56.704853+0000","last_became_peered":"2023-10-16T11:21:56.704853+0000","last_unstale":"2023-10-16T11:21:57.857687+0000","last_undegraded":"2023-10-16T11:21:57.857687+0000","last_fullsized":"2023-10-16T11:21:57.857687+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:45.148019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857648+0000","last_change":"2023-10-16T11:21:56.709694+0000","last_active":"2023-10-16T11:21:57.857648+0000","last_peered":"2023-10-16T11:21:57.857648+0000","last_clean":"2023-10-16T11:21:57.857648+0000","last_became_active":"2023-10-16T11:21:56.709429+0000","last_became_peered":"2023-10-16T11:21:56.709429+0000","last_unstale":"2023-10-16T11:21:57.857648+0000","last_undegraded":"2023-10-16T11:21:57.857648+0000","last_fullsized":"2023-10-16T11:21:57.857648+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:37.010389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703461+0000","last_change":"2023-10-16T11:21:56.722428+0000","last_active":"2023-10-16T11:21:57.703461+0000","last_peered":"2023-10-16T11:21:57.703461+0000","last_clean":"2023-10-16T11:21:57.703461+0000","last_became_active":"2023-10-16T11:21:56.721531+0000","last_became_peered":"2023-10-16T11:21:56.721531+0000","last_unstale":"2023-10-16T11:21:57.703461+0000","last_undegraded":"2023-10-16T11:21:57.703461+0000","last_fullsized":"2023-10-16T11:21:57.703461+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:36.312330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704449+0000","last_change":"2023-10-16T11:21:56.721623+0000","last_active":"2023-10-16T11:21:57.704449+0000","last_peered":"2023-10-16T11:21:57.704449+0000","last_clean":"2023-10-16T11:21:57.704449+0000","last_became_active":"2023-10-16T11:21:56.720746+0000","last_became_peered":"2023-10-16T11:21:56.720746+0000","last_unstale":"2023-10-16T11:21:57.704449+0000","last_undegraded":"2023-10-16T11:21:57.704449+0000","last_fullsized":"2023-10-16T11:21:57.704449+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:27.548089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"269'4","reported_seq":16,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704097+0000","last_change":"2023-10-16T11:21:55.697221+0000","last_active":"2023-10-16T11:21:57.704097+0000","last_peered":"2023-10-16T11:21:57.704097+0000","last_clean":"2023-10-16T11:21:57.704097+0000","last_became_active":"2023-10-16T11:21:55.696910+0000","last_became_peered":"2023-10-16T11:21:55.696910+0000","last_unstale":"2023-10-16T11:21:57.704097+0000","last_undegraded":"2023-10-16T11:21:57.704097+0000","last_fullsized":"2023-10-16T11:21:57.704097+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:09.316693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703545+0000","last_change":"2023-10-16T11:21:56.709469+0000","last_active":"2023-10-16T11:21:57.703545+0000","last_peered":"2023-10-16T11:21:57.703545+0000","last_clean":"2023-10-16T11:21:57.703545+0000","last_became_active":"2023-10-16T11:21:56.708915+0000","last_became_peered":"2023-10-16T11:21:56.708915+0000","last_unstale":"2023-10-16T11:21:57.703545+0000","last_undegraded":"2023-10-16T11:21:57.703545+0000","last_fullsized":"2023-10-16T11:21:57.703545+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:16.287820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703437+0000","last_change":"2023-10-16T11:21:55.706566+0000","last_active":"2023-10-16T11:21:57.703437+0000","last_peered":"2023-10-16T11:21:57.703437+0000","last_clean":"2023-10-16T11:21:57.703437+0000","last_became_active":"2023-10-16T11:21:55.706144+0000","last_became_peered":"2023-10-16T11:21:55.706144+0000","last_unstale":"2023-10-16T11:21:57.703437+0000","last_undegraded":"2023-10-16T11:21:57.703437+0000","last_fullsized":"2023-10-16T11:21:57.703437+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:08.381066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858208+0000","last_change":"2023-10-16T11:21:56.700703+0000","last_active":"2023-10-16T11:21:57.858208+0000","last_peered":"2023-10-16T11:21:57.858208+0000","last_clean":"2023-10-16T11:21:57.858208+0000","last_became_active":"2023-10-16T11:21:56.699386+0000","last_became_peered":"2023-10-16T11:21:56.699386+0000","last_unstale":"2023-10-16T11:21:57.858208+0000","last_undegraded":"2023-10-16T11:21:57.858208+0000","last_fullsized":"2023-10-16T11:21:57.858208+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:36.312330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857666+0000","last_change":"2023-10-16T11:21:55.702119+0000","last_active":"2023-10-16T11:21:57.857666+0000","last_peered":"2023-10-16T11:21:57.857666+0000","last_clean":"2023-10-16T11:21:57.857666+0000","last_became_active":"2023-10-16T11:21:55.701861+0000","last_became_peered":"2023-10-16T11:21:55.701861+0000","last_unstale":"2023-10-16T11:21:57.857666+0000","last_undegraded":"2023-10-16T11:21:57.857666+0000","last_fullsized":"2023-10-16T11:21:57.857666+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:11.623629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858037+0000","last_change":"2023-10-16T11:21:56.706095+0000","last_active":"2023-10-16T11:21:57.858037+0000","last_peered":"2023-10-16T11:21:57.858037+0000","last_clean":"2023-10-16T11:21:57.858037+0000","last_became_active":"2023-10-16T11:21:56.705819+0000","last_became_peered":"2023-10-16T11:21:56.705819+0000","last_unstale":"2023-10-16T11:21:57.858037+0000","last_undegraded":"2023-10-16T11:21:57.858037+0000","last_fullsized":"2023-10-16T11:21:57.858037+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:47.158658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857880+0000","last_change":"2023-10-16T11:21:55.705209+0000","last_active":"2023-10-16T11:21:57.857880+0000","last_peered":"2023-10-16T11:21:57.857880+0000","last_clean":"2023-10-16T11:21:57.857880+0000","last_became_active":"2023-10-16T11:21:55.704955+0000","last_became_peered":"2023-10-16T11:21:55.704955+0000","last_unstale":"2023-10-16T11:21:57.857880+0000","last_undegraded":"2023-10-16T11:21:57.857880+0000","last_fullsized":"2023-10-16T11:21:57.857880+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:23.617512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856987+0000","last_change":"2023-10-16T11:21:56.725500+0000","last_active":"2023-10-16T11:21:57.856987+0000","last_peered":"2023-10-16T11:21:57.856987+0000","last_clean":"2023-10-16T11:21:57.856987+0000","last_became_active":"2023-10-16T11:21:56.724507+0000","last_became_peered":"2023-10-16T11:21:56.724507+0000","last_unstale":"2023-10-16T11:21:57.856987+0000","last_undegraded":"2023-10-16T11:21:57.856987+0000","last_fullsized":"2023-10-16T11:21:57.856987+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:03.801140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703319+0000","last_change":"2023-10-16T11:21:55.702206+0000","last_active":"2023-10-16T11:21:57.703319+0000","last_peered":"2023-10-16T11:21:57.703319+0000","last_clean":"2023-10-16T11:21:57.703319+0000","last_became_active":"2023-10-16T11:21:55.701947+0000","last_became_peered":"2023-10-16T11:21:55.701947+0000","last_unstale":"2023-10-16T11:21:57.703319+0000","last_undegraded":"2023-10-16T11:21:57.703319+0000","last_fullsized":"2023-10-16T11:21:57.703319+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:36.185352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703267+0000","last_change":"2023-10-16T11:21:56.697293+0000","last_active":"2023-10-16T11:21:57.703267+0000","last_peered":"2023-10-16T11:21:57.703267+0000","last_clean":"2023-10-16T11:21:57.703267+0000","last_became_active":"2023-10-16T11:21:56.696681+0000","last_became_peered":"2023-10-16T11:21:56.696681+0000","last_unstale":"2023-10-16T11:21:57.703267+0000","last_undegraded":"2023-10-16T11:21:57.703267+0000","last_fullsized":"2023-10-16T11:21:57.703267+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:31.647063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857275+0000","last_change":"2023-10-16T11:21:55.706390+0000","last_active":"2023-10-16T11:21:57.857275+0000","last_peered":"2023-10-16T11:21:57.857275+0000","last_clean":"2023-10-16T11:21:57.857275+0000","last_became_active":"2023-10-16T11:21:55.705653+0000","last_became_peered":"2023-10-16T11:21:55.705653+0000","last_unstale":"2023-10-16T11:21:57.857275+0000","last_undegraded":"2023-10-16T11:21:57.857275+0000","last_fullsized":"2023-10-16T11:21:57.857275+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:02.318497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703249+0000","last_change":"2023-10-16T11:21:56.709942+0000","last_active":"2023-10-16T11:21:57.703249+0000","last_peered":"2023-10-16T11:21:57.703249+0000","last_clean":"2023-10-16T11:21:57.703249+0000","last_became_active":"2023-10-16T11:21:56.709333+0000","last_became_peered":"2023-10-16T11:21:56.709333+0000","last_unstale":"2023-10-16T11:21:57.703249+0000","last_undegraded":"2023-10-16T11:21:57.703249+0000","last_fullsized":"2023-10-16T11:21:57.703249+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:45.148019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857749+0000","last_change":"2023-10-16T11:21:55.701489+0000","last_active":"2023-10-16T11:21:57.857749+0000","last_peered":"2023-10-16T11:21:57.857749+0000","last_clean":"2023-10-16T11:21:57.857749+0000","last_became_active":"2023-10-16T11:21:55.701217+0000","last_became_peered":"2023-10-16T11:21:55.701217+0000","last_unstale":"2023-10-16T11:21:57.857749+0000","last_undegraded":"2023-10-16T11:21:57.857749+0000","last_fullsized":"2023-10-16T11:21:57.857749+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:01:58.190877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857612+0000","last_change":"2023-10-16T11:21:56.710657+0000","last_active":"2023-10-16T11:21:57.857612+0000","last_peered":"2023-10-16T11:21:57.857612+0000","last_clean":"2023-10-16T11:21:57.857612+0000","last_became_active":"2023-10-16T11:21:56.710381+0000","last_became_peered":"2023-10-16T11:21:56.710381+0000","last_unstale":"2023-10-16T11:21:57.857612+0000","last_undegraded":"2023-10-16T11:21:57.857612+0000","last_fullsized":"2023-10-16T11:21:57.857612+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:42:04.928124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857436+0000","last_change":"2023-10-16T11:21:55.699726+0000","last_active":"2023-10-16T11:21:57.857436+0000","last_peered":"2023-10-16T11:21:57.857436+0000","last_clean":"2023-10-16T11:21:57.857436+0000","last_became_active":"2023-10-16T11:21:55.699217+0000","last_became_peered":"2023-10-16T11:21:55.699217+0000","last_unstale":"2023-10-16T11:21:57.857436+0000","last_undegraded":"2023-10-16T11:21:57.857436+0000","last_fullsized":"2023-10-16T11:21:57.857436+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:38.414368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704503+0000","last_change":"2023-10-16T11:21:56.721794+0000","last_active":"2023-10-16T11:21:57.704503+0000","last_peered":"2023-10-16T11:21:57.704503+0000","last_clean":"2023-10-16T11:21:57.704503+0000","last_became_active":"2023-10-16T11:21:56.721061+0000","last_became_peered":"2023-10-16T11:21:56.721061+0000","last_unstale":"2023-10-16T11:21:57.704503+0000","last_undegraded":"2023-10-16T11:21:57.704503+0000","last_fullsized":"2023-10-16T11:21:57.704503+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:55.390968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704173+0000","last_change":"2023-10-16T11:21:55.706154+0000","last_active":"2023-10-16T11:21:57.704173+0000","last_peered":"2023-10-16T11:21:57.704173+0000","last_clean":"2023-10-16T11:21:57.704173+0000","last_became_active":"2023-10-16T11:21:55.705858+0000","last_became_peered":"2023-10-16T11:21:55.705858+0000","last_unstale":"2023-10-16T11:21:57.704173+0000","last_undegraded":"2023-10-16T11:21:57.704173+0000","last_fullsized":"2023-10-16T11:21:57.704173+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:32.050360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858249+0000","last_change":"2023-10-16T11:21:56.703727+0000","last_active":"2023-10-16T11:21:57.858249+0000","last_peered":"2023-10-16T11:21:57.858249+0000","last_clean":"2023-10-16T11:21:57.858249+0000","last_became_active":"2023-10-16T11:21:56.703438+0000","last_became_peered":"2023-10-16T11:21:56.703438+0000","last_unstale":"2023-10-16T11:21:57.858249+0000","last_undegraded":"2023-10-16T11:21:57.858249+0000","last_fullsized":"2023-10-16T11:21:57.858249+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:40.755285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858149+0000","last_change":"2023-10-16T11:21:55.705977+0000","last_active":"2023-10-16T11:21:57.858149+0000","last_peered":"2023-10-16T11:21:57.858149+0000","last_clean":"2023-10-16T11:21:57.858149+0000","last_became_active":"2023-10-16T11:21:55.705705+0000","last_became_peered":"2023-10-16T11:21:55.705705+0000","last_unstale":"2023-10-16T11:21:57.858149+0000","last_undegraded":"2023-10-16T11:21:57.858149+0000","last_fullsized":"2023-10-16T11:21:57.858149+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:00:47.640952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858312+0000","last_change":"2023-10-16T11:21:56.700879+0000","last_active":"2023-10-16T11:21:57.858312+0000","last_peered":"2023-10-16T11:21:57.858312+0000","last_clean":"2023-10-16T11:21:57.858312+0000","last_became_active":"2023-10-16T11:21:56.699758+0000","last_became_peered":"2023-10-16T11:21:56.699758+0000","last_unstale":"2023-10-16T11:21:57.858312+0000","last_undegraded":"2023-10-16T11:21:57.858312+0000","last_fullsized":"2023-10-16T11:21:57.858312+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:16.046537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"269'3","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858047+0000","last_change":"2023-10-16T11:21:55.696214+0000","last_active":"2023-10-16T11:21:57.858047+0000","last_peered":"2023-10-16T11:21:57.858047+0000","last_clean":"2023-10-16T11:21:57.858047+0000","last_became_active":"2023-10-16T11:21:55.695955+0000","last_became_peered":"2023-10-16T11:21:55.695955+0000","last_unstale":"2023-10-16T11:21:57.858047+0000","last_undegraded":"2023-10-16T11:21:57.858047+0000","last_fullsized":"2023-10-16T11:21:57.858047+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:16.287809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703572+0000","last_change":"2023-10-16T11:21:56.720789+0000","last_active":"2023-10-16T11:21:57.703572+0000","last_peered":"2023-10-16T11:21:57.703572+0000","last_clean":"2023-10-16T11:21:57.703572+0000","last_became_active":"2023-10-16T11:21:56.717572+0000","last_became_peered":"2023-10-16T11:21:56.717572+0000","last_unstale":"2023-10-16T11:21:57.703572+0000","last_undegraded":"2023-10-16T11:21:57.703572+0000","last_fullsized":"2023-10-16T11:21:57.703572+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:02.862244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"269'6","reported_seq":18,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857453+0000","last_change":"2023-10-16T11:21:55.703343+0000","last_active":"2023-10-16T11:21:57.857453+0000","last_peered":"2023-10-16T11:21:57.857453+0000","last_clean":"2023-10-16T11:21:57.857453+0000","last_became_active":"2023-10-16T11:21:55.703055+0000","last_became_peered":"2023-10-16T11:21:55.703055+0000","last_unstale":"2023-10-16T11:21:57.857453+0000","last_undegraded":"2023-10-16T11:21:57.857453+0000","last_fullsized":"2023-10-16T11:21:57.857453+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:17.708721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703274+0000","last_change":"2023-10-16T11:21:56.699974+0000","last_active":"2023-10-16T11:21:57.703274+0000","last_peered":"2023-10-16T11:21:57.703274+0000","last_clean":"2023-10-16T11:21:57.703274+0000","last_became_active":"2023-10-16T11:21:56.699656+0000","last_became_peered":"2023-10-16T11:21:56.699656+0000","last_unstale":"2023-10-16T11:21:57.703274+0000","last_undegraded":"2023-10-16T11:21:57.703274+0000","last_fullsized":"2023-10-16T11:21:57.703274+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:04.292581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858031+0000","last_change":"2023-10-16T11:21:55.704976+0000","last_active":"2023-10-16T11:21:57.858031+0000","last_peered":"2023-10-16T11:21:57.858031+0000","last_clean":"2023-10-16T11:21:57.858031+0000","last_became_active":"2023-10-16T11:21:55.704682+0000","last_became_peered":"2023-10-16T11:21:55.704682+0000","last_unstale":"2023-10-16T11:21:57.858031+0000","last_undegraded":"2023-10-16T11:21:57.858031+0000","last_fullsized":"2023-10-16T11:21:57.858031+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:23.560949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704097+0000","last_change":"2023-10-16T11:21:56.710162+0000","last_active":"2023-10-16T11:21:57.704097+0000","last_peered":"2023-10-16T11:21:57.704097+0000","last_clean":"2023-10-16T11:21:57.704097+0000","last_became_active":"2023-10-16T11:21:56.709882+0000","last_became_peered":"2023-10-16T11:21:56.709882+0000","last_unstale":"2023-10-16T11:21:57.704097+0000","last_undegraded":"2023-10-16T11:21:57.704097+0000","last_fullsized":"2023-10-16T11:21:57.704097+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:57.829265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703246+0000","last_change":"2023-10-16T11:21:55.701157+0000","last_active":"2023-10-16T11:21:57.703246+0000","last_peered":"2023-10-16T11:21:57.703246+0000","last_clean":"2023-10-16T11:21:57.703246+0000","last_became_active":"2023-10-16T11:21:55.700835+0000","last_became_peered":"2023-10-16T11:21:55.700835+0000","last_unstale":"2023-10-16T11:21:57.703246+0000","last_undegraded":"2023-10-16T11:21:57.703246+0000","last_fullsized":"2023-10-16T11:21:57.703246+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:49.072272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703396+0000","last_change":"2023-10-16T11:21:56.717105+0000","last_active":"2023-10-16T11:21:57.703396+0000","last_peered":"2023-10-16T11:21:57.703396+0000","last_clean":"2023-10-16T11:21:57.703396+0000","last_became_active":"2023-10-16T11:21:56.716797+0000","last_became_peered":"2023-10-16T11:21:56.716797+0000","last_unstale":"2023-10-16T11:21:57.703396+0000","last_undegraded":"2023-10-16T11:21:57.703396+0000","last_fullsized":"2023-10-16T11:21:57.703396+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:47.158658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857807+0000","last_change":"2023-10-16T11:21:55.710302+0000","last_active":"2023-10-16T11:21:57.857807+0000","last_peered":"2023-10-16T11:21:57.857807+0000","last_clean":"2023-10-16T11:21:57.857807+0000","last_became_active":"2023-10-16T11:21:55.710013+0000","last_became_peered":"2023-10-16T11:21:55.710013+0000","last_unstale":"2023-10-16T11:21:57.857807+0000","last_undegraded":"2023-10-16T11:21:57.857807+0000","last_fullsized":"2023-10-16T11:21:57.857807+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:54.075175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857669+0000","last_change":"2023-10-16T11:21:56.710801+0000","last_active":"2023-10-16T11:21:57.857669+0000","last_peered":"2023-10-16T11:21:57.857669+0000","last_clean":"2023-10-16T11:21:57.857669+0000","last_became_active":"2023-10-16T11:21:56.710522+0000","last_became_peered":"2023-10-16T11:21:56.710522+0000","last_unstale":"2023-10-16T11:21:57.857669+0000","last_undegraded":"2023-10-16T11:21:57.857669+0000","last_fullsized":"2023-10-16T11:21:57.857669+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:52.846398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"269'8","reported_seq":20,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704268+0000","last_change":"2023-10-16T11:21:55.709635+0000","last_active":"2023-10-16T11:21:57.704268+0000","last_peered":"2023-10-16T11:21:57.704268+0000","last_clean":"2023-10-16T11:21:57.704268+0000","last_became_active":"2023-10-16T11:21:55.709394+0000","last_became_peered":"2023-10-16T11:21:55.709394+0000","last_unstale":"2023-10-16T11:21:57.704268+0000","last_undegraded":"2023-10-16T11:21:57.704268+0000","last_fullsized":"2023-10-16T11:21:57.704268+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:41.804386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857589+0000","last_change":"2023-10-16T11:21:56.704300+0000","last_active":"2023-10-16T11:21:57.857589+0000","last_peered":"2023-10-16T11:21:57.857589+0000","last_clean":"2023-10-16T11:21:57.857589+0000","last_became_active":"2023-10-16T11:21:56.704023+0000","last_became_peered":"2023-10-16T11:21:56.704023+0000","last_unstale":"2023-10-16T11:21:57.857589+0000","last_undegraded":"2023-10-16T11:21:57.857589+0000","last_fullsized":"2023-10-16T11:21:57.857589+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:19.802321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703830+0000","last_change":"2023-10-16T11:21:55.708263+0000","last_active":"2023-10-16T11:21:57.703830+0000","last_peered":"2023-10-16T11:21:57.703830+0000","last_clean":"2023-10-16T11:21:57.703830+0000","last_became_active":"2023-10-16T11:21:55.707990+0000","last_became_peered":"2023-10-16T11:21:55.707990+0000","last_unstale":"2023-10-16T11:21:57.703830+0000","last_undegraded":"2023-10-16T11:21:57.703830+0000","last_fullsized":"2023-10-16T11:21:57.703830+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:04.429083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703753+0000","last_change":"2023-10-16T11:21:56.695808+0000","last_active":"2023-10-16T11:21:57.703753+0000","last_peered":"2023-10-16T11:21:57.703753+0000","last_clean":"2023-10-16T11:21:57.703753+0000","last_became_active":"2023-10-16T11:21:56.695575+0000","last_became_peered":"2023-10-16T11:21:56.695575+0000","last_unstale":"2023-10-16T11:21:57.703753+0000","last_undegraded":"2023-10-16T11:21:57.703753+0000","last_fullsized":"2023-10-16T11:21:57.703753+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:13.902954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858119+0000","last_change":"2023-10-16T11:21:55.703387+0000","last_active":"2023-10-16T11:21:57.858119+0000","last_peered":"2023-10-16T11:21:57.858119+0000","last_clean":"2023-10-16T11:21:57.858119+0000","last_became_active":"2023-10-16T11:21:55.703110+0000","last_became_peered":"2023-10-16T11:21:55.703110+0000","last_unstale":"2023-10-16T11:21:57.858119+0000","last_undegraded":"2023-10-16T11:21:57.858119+0000","last_fullsized":"2023-10-16T11:21:57.858119+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:48.552813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858137+0000","last_change":"2023-10-16T11:21:56.704125+0000","last_active":"2023-10-16T11:21:57.858137+0000","last_peered":"2023-10-16T11:21:57.858137+0000","last_clean":"2023-10-16T11:21:57.858137+0000","last_became_active":"2023-10-16T11:21:56.703850+0000","last_became_peered":"2023-10-16T11:21:56.703850+0000","last_unstale":"2023-10-16T11:21:57.858137+0000","last_undegraded":"2023-10-16T11:21:57.858137+0000","last_fullsized":"2023-10-16T11:21:57.858137+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:00:48.648047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703960+0000","last_change":"2023-10-16T11:21:55.704758+0000","last_active":"2023-10-16T11:21:57.703960+0000","last_peered":"2023-10-16T11:21:57.703960+0000","last_clean":"2023-10-16T11:21:57.703960+0000","last_became_active":"2023-10-16T11:21:55.704477+0000","last_became_peered":"2023-10-16T11:21:55.704477+0000","last_unstale":"2023-10-16T11:21:57.703960+0000","last_undegraded":"2023-10-16T11:21:57.703960+0000","last_fullsized":"2023-10-16T11:21:57.703960+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:23.418643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857845+0000","last_change":"2023-10-16T11:21:56.710714+0000","last_active":"2023-10-16T11:21:57.857845+0000","last_peered":"2023-10-16T11:21:57.857845+0000","last_clean":"2023-10-16T11:21:57.857845+0000","last_became_active":"2023-10-16T11:21:56.710401+0000","last_became_peered":"2023-10-16T11:21:56.710401+0000","last_unstale":"2023-10-16T11:21:57.857845+0000","last_undegraded":"2023-10-16T11:21:57.857845+0000","last_fullsized":"2023-10-16T11:21:57.857845+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:26.954413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857958+0000","last_change":"2023-10-16T11:21:55.705874+0000","last_active":"2023-10-16T11:21:57.857958+0000","last_peered":"2023-10-16T11:21:57.857958+0000","last_clean":"2023-10-16T11:21:57.857958+0000","last_became_active":"2023-10-16T11:21:55.705604+0000","last_became_peered":"2023-10-16T11:21:55.705604+0000","last_unstale":"2023-10-16T11:21:57.857958+0000","last_undegraded":"2023-10-16T11:21:57.857958+0000","last_fullsized":"2023-10-16T11:21:57.857958+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:21.104563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704075+0000","last_change":"2023-10-16T11:21:56.722739+0000","last_active":"2023-10-16T11:21:57.704075+0000","last_peered":"2023-10-16T11:21:57.704075+0000","last_clean":"2023-10-16T11:21:57.704075+0000","last_became_active":"2023-10-16T11:21:56.721829+0000","last_became_peered":"2023-10-16T11:21:56.721829+0000","last_unstale":"2023-10-16T11:21:57.704075+0000","last_undegraded":"2023-10-16T11:21:57.704075+0000","last_fullsized":"2023-10-16T11:21:57.704075+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:16.046537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703415+0000","last_change":"2023-10-16T11:21:55.697184+0000","last_active":"2023-10-16T11:21:57.703415+0000","last_peered":"2023-10-16T11:21:57.703415+0000","last_clean":"2023-10-16T11:21:57.703415+0000","last_became_active":"2023-10-16T11:21:55.695179+0000","last_became_peered":"2023-10-16T11:21:55.695179+0000","last_unstale":"2023-10-16T11:21:57.703415+0000","last_undegraded":"2023-10-16T11:21:57.703415+0000","last_fullsized":"2023-10-16T11:21:57.703415+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:49.200047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703306+0000","last_change":"2023-10-16T11:21:56.697464+0000","last_active":"2023-10-16T11:21:57.703306+0000","last_peered":"2023-10-16T11:21:57.703306+0000","last_clean":"2023-10-16T11:21:57.703306+0000","last_became_active":"2023-10-16T11:21:56.697030+0000","last_became_peered":"2023-10-16T11:21:56.697030+0000","last_unstale":"2023-10-16T11:21:57.703306+0000","last_undegraded":"2023-10-16T11:21:57.703306+0000","last_fullsized":"2023-10-16T11:21:57.703306+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:17.280483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703727+0000","last_change":"2023-10-16T11:21:55.706914+0000","last_active":"2023-10-16T11:21:57.703727+0000","last_peered":"2023-10-16T11:21:57.703727+0000","last_clean":"2023-10-16T11:21:57.703727+0000","last_became_active":"2023-10-16T11:21:55.706627+0000","last_became_peered":"2023-10-16T11:21:55.706627+0000","last_unstale":"2023-10-16T11:21:57.703727+0000","last_undegraded":"2023-10-16T11:21:57.703727+0000","last_fullsized":"2023-10-16T11:21:57.703727+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:37.849795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703150+0000","last_change":"2023-10-16T11:21:56.704983+0000","last_active":"2023-10-16T11:21:57.703150+0000","last_peered":"2023-10-16T11:21:57.703150+0000","last_clean":"2023-10-16T11:21:57.703150+0000","last_became_active":"2023-10-16T11:21:56.704713+0000","last_became_peered":"2023-10-16T11:21:56.704713+0000","last_unstale":"2023-10-16T11:21:57.703150+0000","last_undegraded":"2023-10-16T11:21:57.703150+0000","last_fullsized":"2023-10-16T11:21:57.703150+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:43.657508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857228+0000","last_change":"2023-10-16T11:21:55.706851+0000","last_active":"2023-10-16T11:21:57.857228+0000","last_peered":"2023-10-16T11:21:57.857228+0000","last_clean":"2023-10-16T11:21:57.857228+0000","last_became_active":"2023-10-16T11:21:55.706500+0000","last_became_peered":"2023-10-16T11:21:55.706500+0000","last_unstale":"2023-10-16T11:21:57.857228+0000","last_undegraded":"2023-10-16T11:21:57.857228+0000","last_fullsized":"2023-10-16T11:21:57.857228+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:54.075175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857955+0000","last_change":"2023-10-16T11:21:56.707579+0000","last_active":"2023-10-16T11:21:57.857955+0000","last_peered":"2023-10-16T11:21:57.857955+0000","last_clean":"2023-10-16T11:21:57.857955+0000","last_became_active":"2023-10-16T11:21:56.707257+0000","last_became_peered":"2023-10-16T11:21:56.707257+0000","last_unstale":"2023-10-16T11:21:57.857955+0000","last_undegraded":"2023-10-16T11:21:57.857955+0000","last_fullsized":"2023-10-16T11:21:57.857955+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:24.624607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856997+0000","last_change":"2023-10-16T11:21:55.706143+0000","last_active":"2023-10-16T11:21:57.856997+0000","last_peered":"2023-10-16T11:21:57.856997+0000","last_clean":"2023-10-16T11:21:57.856997+0000","last_became_active":"2023-10-16T11:21:55.705705+0000","last_became_peered":"2023-10-16T11:21:55.705705+0000","last_unstale":"2023-10-16T11:21:57.856997+0000","last_undegraded":"2023-10-16T11:21:57.856997+0000","last_fullsized":"2023-10-16T11:21:57.856997+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:34.971548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704596+0000","last_change":"2023-10-16T11:21:56.721959+0000","last_active":"2023-10-16T11:21:57.704596+0000","last_peered":"2023-10-16T11:21:57.704596+0000","last_clean":"2023-10-16T11:21:57.704596+0000","last_became_active":"2023-10-16T11:21:56.721364+0000","last_became_peered":"2023-10-16T11:21:56.721364+0000","last_unstale":"2023-10-16T11:21:57.704596+0000","last_undegraded":"2023-10-16T11:21:57.704596+0000","last_fullsized":"2023-10-16T11:21:57.704596+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:51.447799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858002+0000","last_change":"2023-10-16T11:21:55.705184+0000","last_active":"2023-10-16T11:21:57.858002+0000","last_peered":"2023-10-16T11:21:57.858002+0000","last_clean":"2023-10-16T11:21:57.858002+0000","last_became_active":"2023-10-16T11:21:55.704903+0000","last_became_peered":"2023-10-16T11:21:55.704903+0000","last_unstale":"2023-10-16T11:21:57.858002+0000","last_undegraded":"2023-10-16T11:21:57.858002+0000","last_fullsized":"2023-10-16T11:21:57.858002+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:57.229633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858237+0000","last_change":"2023-10-16T11:21:56.722149+0000","last_active":"2023-10-16T11:21:57.858237+0000","last_peered":"2023-10-16T11:21:57.858237+0000","last_clean":"2023-10-16T11:21:57.858237+0000","last_became_active":"2023-10-16T11:21:56.721753+0000","last_became_peered":"2023-10-16T11:21:56.721753+0000","last_unstale":"2023-10-16T11:21:57.858237+0000","last_undegraded":"2023-10-16T11:21:57.858237+0000","last_fullsized":"2023-10-16T11:21:57.858237+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:44.589323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703737+0000","last_change":"2023-10-16T11:21:55.708596+0000","last_active":"2023-10-16T11:21:57.703737+0000","last_peered":"2023-10-16T11:21:57.703737+0000","last_clean":"2023-10-16T11:21:57.703737+0000","last_became_active":"2023-10-16T11:21:55.708323+0000","last_became_peered":"2023-10-16T11:21:55.708323+0000","last_unstale":"2023-10-16T11:21:57.703737+0000","last_undegraded":"2023-10-16T11:21:57.703737+0000","last_fullsized":"2023-10-16T11:21:57.703737+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:42.928253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858284+0000","last_change":"2023-10-16T11:21:56.710801+0000","last_active":"2023-10-16T11:21:57.858284+0000","last_peered":"2023-10-16T11:21:57.858284+0000","last_clean":"2023-10-16T11:21:57.858284+0000","last_became_active":"2023-10-16T11:21:56.710464+0000","last_became_peered":"2023-10-16T11:21:56.710464+0000","last_unstale":"2023-10-16T11:21:57.858284+0000","last_undegraded":"2023-10-16T11:21:57.858284+0000","last_fullsized":"2023-10-16T11:21:57.858284+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:02.862244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858137+0000","last_change":"2023-10-16T11:21:55.707466+0000","last_active":"2023-10-16T11:21:57.858137+0000","last_peered":"2023-10-16T11:21:57.858137+0000","last_clean":"2023-10-16T11:21:57.858137+0000","last_became_active":"2023-10-16T11:21:55.707198+0000","last_became_peered":"2023-10-16T11:21:55.707198+0000","last_unstale":"2023-10-16T11:21:57.858137+0000","last_undegraded":"2023-10-16T11:21:57.858137+0000","last_fullsized":"2023-10-16T11:21:57.858137+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:25.947319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703662+0000","last_change":"2023-10-16T11:21:56.697353+0000","last_active":"2023-10-16T11:21:57.703662+0000","last_peered":"2023-10-16T11:21:57.703662+0000","last_clean":"2023-10-16T11:21:57.703662+0000","last_became_active":"2023-10-16T11:21:56.697071+0000","last_became_peered":"2023-10-16T11:21:56.697071+0000","last_unstale":"2023-10-16T11:21:57.703662+0000","last_undegraded":"2023-10-16T11:21:57.703662+0000","last_fullsized":"2023-10-16T11:21:57.703662+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:15.826809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703720+0000","last_change":"2023-10-16T11:21:55.709288+0000","last_active":"2023-10-16T11:21:57.703720+0000","last_peered":"2023-10-16T11:21:57.703720+0000","last_clean":"2023-10-16T11:21:57.703720+0000","last_became_active":"2023-10-16T11:21:55.709024+0000","last_became_peered":"2023-10-16T11:21:55.709024+0000","last_unstale":"2023-10-16T11:21:57.703720+0000","last_undegraded":"2023-10-16T11:21:57.703720+0000","last_fullsized":"2023-10-16T11:21:57.703720+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:55.192509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703053+0000","last_change":"2023-10-16T11:21:56.710027+0000","last_active":"2023-10-16T11:21:57.703053+0000","last_peered":"2023-10-16T11:21:57.703053+0000","last_clean":"2023-10-16T11:21:57.703053+0000","last_became_active":"2023-10-16T11:21:56.709730+0000","last_became_peered":"2023-10-16T11:21:56.709730+0000","last_unstale":"2023-10-16T11:21:57.703053+0000","last_undegraded":"2023-10-16T11:21:57.703053+0000","last_fullsized":"2023-10-16T11:21:57.703053+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:49.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857303+0000","last_change":"2023-10-16T11:21:55.702934+0000","last_active":"2023-10-16T11:21:57.857303+0000","last_peered":"2023-10-16T11:21:57.857303+0000","last_clean":"2023-10-16T11:21:57.857303+0000","last_became_active":"2023-10-16T11:21:55.702724+0000","last_became_peered":"2023-10-16T11:21:55.702724+0000","last_unstale":"2023-10-16T11:21:57.857303+0000","last_undegraded":"2023-10-16T11:21:57.857303+0000","last_fullsized":"2023-10-16T11:21:57.857303+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:03.648335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857763+0000","last_change":"2023-10-16T11:21:56.718659+0000","last_active":"2023-10-16T11:21:57.857763+0000","last_peered":"2023-10-16T11:21:57.857763+0000","last_clean":"2023-10-16T11:21:57.857763+0000","last_became_active":"2023-10-16T11:21:56.718394+0000","last_became_peered":"2023-10-16T11:21:56.718394+0000","last_unstale":"2023-10-16T11:21:57.857763+0000","last_undegraded":"2023-10-16T11:21:57.857763+0000","last_fullsized":"2023-10-16T11:21:57.857763+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:31.587975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857699+0000","last_change":"2023-10-16T11:21:55.705027+0000","last_active":"2023-10-16T11:21:57.857699+0000","last_peered":"2023-10-16T11:21:57.857699+0000","last_clean":"2023-10-16T11:21:57.857699+0000","last_became_active":"2023-10-16T11:21:55.704583+0000","last_became_peered":"2023-10-16T11:21:55.704583+0000","last_unstale":"2023-10-16T11:21:57.857699+0000","last_undegraded":"2023-10-16T11:21:57.857699+0000","last_fullsized":"2023-10-16T11:21:57.857699+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:12.908701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703287+0000","last_change":"2023-10-16T11:21:56.703667+0000","last_active":"2023-10-16T11:21:57.703287+0000","last_peered":"2023-10-16T11:21:57.703287+0000","last_clean":"2023-10-16T11:21:57.703287+0000","last_became_active":"2023-10-16T11:21:56.703381+0000","last_became_peered":"2023-10-16T11:21:56.703381+0000","last_unstale":"2023-10-16T11:21:57.703287+0000","last_undegraded":"2023-10-16T11:21:57.703287+0000","last_fullsized":"2023-10-16T11:21:57.703287+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:53.112967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858071+0000","last_change":"2023-10-16T11:21:55.710236+0000","last_active":"2023-10-16T11:21:57.858071+0000","last_peered":"2023-10-16T11:21:57.858071+0000","last_clean":"2023-10-16T11:21:57.858071+0000","last_became_active":"2023-10-16T11:21:55.710008+0000","last_became_peered":"2023-10-16T11:21:55.710008+0000","last_unstale":"2023-10-16T11:21:57.858071+0000","last_undegraded":"2023-10-16T11:21:57.858071+0000","last_fullsized":"2023-10-16T11:21:57.858071+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:36.903604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858493+0000","last_change":"2023-10-16T11:21:56.708680+0000","last_active":"2023-10-16T11:21:57.858493+0000","last_peered":"2023-10-16T11:21:57.858493+0000","last_clean":"2023-10-16T11:21:57.858493+0000","last_became_active":"2023-10-16T11:21:56.708402+0000","last_became_peered":"2023-10-16T11:21:56.708402+0000","last_unstale":"2023-10-16T11:21:57.858493+0000","last_undegraded":"2023-10-16T11:21:57.858493+0000","last_fullsized":"2023-10-16T11:21:57.858493+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:57.634987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857081+0000","last_change":"2023-10-16T11:21:55.706289+0000","last_active":"2023-10-16T11:21:57.857081+0000","last_peered":"2023-10-16T11:21:57.857081+0000","last_clean":"2023-10-16T11:21:57.857081+0000","last_became_active":"2023-10-16T11:21:55.706072+0000","last_became_peered":"2023-10-16T11:21:55.706072+0000","last_unstale":"2023-10-16T11:21:57.857081+0000","last_undegraded":"2023-10-16T11:21:57.857081+0000","last_fullsized":"2023-10-16T11:21:57.857081+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:57.009588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858273+0000","last_change":"2023-10-16T11:21:56.707540+0000","last_active":"2023-10-16T11:21:57.858273+0000","last_peered":"2023-10-16T11:21:57.858273+0000","last_clean":"2023-10-16T11:21:57.858273+0000","last_became_active":"2023-10-16T11:21:56.707234+0000","last_became_peered":"2023-10-16T11:21:56.707234+0000","last_unstale":"2023-10-16T11:21:57.858273+0000","last_undegraded":"2023-10-16T11:21:57.858273+0000","last_fullsized":"2023-10-16T11:21:57.858273+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:06.029695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"260'403","reported_seq":1454,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704363+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:21:57.704363+0000","last_peered":"2023-10-16T11:21:57.704363+0000","last_clean":"2023-10-16T11:21:57.704363+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:21:57.704363+0000","last_undegraded":"2023-10-16T11:21:57.704363+0000","last_fullsized":"2023-10-16T11:21:57.704363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576015,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57320,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314520,"statfs":{"total":96636764160,"available":96578068480,"internally_reserved":0,"allocated":4026368,"data_stored":2442232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73799999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.68300000000000005,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.70199999999999996}]},{"osd":2,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.70399999999999996,"15min":0.66700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.622},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":9.516,"15min":9.516},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67000000000000004,"15min":0.64000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.441},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.59299999999999997,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.624,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67400000000000004,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.64800000000000002,"15min":0.57899999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77800000000000002,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.68000000000000005,"15min":0.625},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.70999999999999996,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.69299999999999995,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72199999999999998,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.65300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56744,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315096,"statfs":{"total":96636764160,"available":96578658304,"internally_reserved":0,"allocated":3436544,"data_stored":1852446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.57999999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.86299999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56499999999999995,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84299999999999997,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71899999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":2.387,"15min":2.387},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.314},"max":{"1min":0.78100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46500000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59899999999999998,"15min":0.61599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88800000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81000000000000005,"5min":1.02,"15min":1.02},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.34499999999999997}]},{"osd":5,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":3.335,"15min":3.335},"last":0.36599999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":2.524,"15min":2.524},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49008,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322832,"statfs":{"total":96636764160,"available":96586579968,"internally_reserved":0,"allocated":4026368,"data_stored":2442300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.55500000000000005,"15min":0.60299999999999998},"last":0.252},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40300000000000002,"15min":0.439},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.44500000000000001,"5min":0.58399999999999996,"15min":0.77200000000000002},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50700000000000001,"5min":1.272,"15min":1.272},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.50600000000000001,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.374},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":1.351,"15min":1.351},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.64200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.45100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":1.391,"15min":1.391},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60599999999999998,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.874,"15min":1.089},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.216},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64300000000000002,"15min":0.68600000000000005},"min":{"1min":0.46600000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":1.038,"15min":1.1180000000000001},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88900000000000001,"5min":0.97399999999999998,"15min":1.008},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.66900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.88500000000000001,"15min":1.0780000000000001},"last":0.54000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3428352,"data_stored":1851842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.40799999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.26500000000000001,"5min":0.184,"15min":0.184},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36199999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.371},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.44},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.30299999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0309999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.0389999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.501}]},{"osd":6,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.629,"15min":0.66500000000000004},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.67200000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64500000000000002,"15min":0.68999999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576015,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52584,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319256,"statfs":{"total":96636764160,"available":96582918144,"internally_reserved":0,"allocated":3432448,"data_stored":1846424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.245,"5min":0.188,"15min":0.188},"max":{"1min":0.47699999999999998,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.318},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.437},"min":{"1min":0.255,"5min":0.193,"15min":0.193},"max":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.29599999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.436,"15min":0.46400000000000002},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.41099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45300000000000001,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.27900000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.376}]},{"osd":4,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":1.097},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86499999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83099999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92700000000000005,"5min":1.0169999999999999,"15min":1.028},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.498,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85899999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.70399999999999996},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":0.99099999999999999,"15min":1.071},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.97799999999999998,"15min":1.077},"last":0.74399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56748,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315092,"statfs":{"total":96636764160,"available":96578654208,"internally_reserved":0,"allocated":3432448,"data_stored":1851876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36499999999999999,"15min":0.39000000000000001},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.621,"5min":1.032,"15min":1.032},"last":0.436},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.36799999999999999,"15min":0.40500000000000003},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.52200000000000002,"5min":1.341,"15min":1.341},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40899999999999997,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55000000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.44400000000000001,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.51300000000000001,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.502},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45900000000000002,"15min":0.503},"min":{"1min":0.34799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54600000000000004,"5min":0.96599999999999997,"15min":1.024},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.623,"15min":0.65300000000000002},"min":{"1min":0.45500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3220000000000001,"5min":2.629,"15min":2.629},"last":0.877}]},{"osd":5,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.66700000000000004},"min":{"1min":0.443,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":1.0149999999999999,"15min":1.026},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.80400000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81100000000000005,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.75900000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":0.71399999999999997},"min":{"1min":0.49099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.304,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65800000000000003,"15min":0.69899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.86399999999999999,"5min":1.006,"15min":1.1899999999999999},"last":0.90100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576015,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52608,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319232,"statfs":{"total":96636764160,"available":96582893568,"internally_reserved":0,"allocated":3452928,"data_stored":1854196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.60899999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.61399999999999999},"min":{"1min":0.41899999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81899999999999995,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.60799999999999998},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.95499999999999996},"last":0.504}]},{"osd":2,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57199999999999995,"15min":0.61699999999999999},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81100000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.995},"last":0.436},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60899999999999999,"15min":0.63200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.86199999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76000000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.82899999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.35799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":1.115},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.68500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66476,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305364,"statfs":{"total":96636764160,"available":96568692736,"internally_reserved":0,"allocated":3432448,"data_stored":1851876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58599999999999997,"15min":0.623},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28999999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.64300000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.88200000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67900000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74299999999999999,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53700000000000003,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.623,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81899999999999995,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.70299999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79300000000000004,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.59899999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.71799999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.496,"15min":0.51600000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.74099999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83599999999999997,"5min":0.878,"15min":0.878},"last":0.61299999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.874},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.623,"15min":0.63700000000000001},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.83099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:22:00.843 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:22:00.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.995+0000 7fe5710c0700 1 Processor -- start 2023-10-16T11:22:00.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.995+0000 7fe5710c0700 1 -- start start 2023-10-16T11:22:00.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe5710c0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe56c15dfb0 0x7fe56c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe5710c0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe56c06f340 0x7fe56c06f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe5710c0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c06e770 con 0x7fe56c0cc5d0 2023-10-16T11:22:01.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe5710c0700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c06fd30 con 0x7fe56c15dfb0 2023-10-16T11:22:01.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe5710c0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c0cf270 con 0x7fe56c06f340 2023-10-16T11:22:01.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56a59c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe56c15dfb0 0x7fe56c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56ad9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe56c0cc5d0 0x7fe56c0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:43992/0 (socket says 172.21.15.158:43992) 2023-10-16T11:22:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56ad9d700 1 -- 172.21.15.158:0/906749055 learned_addr learned my addr 172.21.15.158:0/906749055 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56b59e700 1 --2- 172.21.15.158:0/906749055 >> v2:172.21.15.172:3300/0 conn(0x7fe56c06f340 0x7fe56c06f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56a59c700 1 -- 172.21.15.158:0/906749055 >> v2:172.21.15.172:3300/0 conn(0x7fe56c06f340 msgr2=0x7fe56c06f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:01.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe569d9b700 1 -- 172.21.15.158:0/906749055 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823930280 0 0) 0x7fe55c001110 con 0x7fe56c0cc5d0 2023-10-16T11:22:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56a59c700 1 --2- 172.21.15.158:0/906749055 >> v2:172.21.15.172:3300/0 conn(0x7fe56c06f340 0x7fe56c06f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56a59c700 1 -- 172.21.15.158:0/906749055 >> v1:172.21.15.158:6789/0 conn(0x7fe56c0cc5d0 legacy=0x7fe56c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56a59c700 1 -- 172.21.15.158:0/906749055 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c0cbe00 con 0x7fe56c15dfb0 2023-10-16T11:22:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe56a59c700 1 --2- 172.21.15.158:0/906749055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c06e230 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0b61e0 tx=0x7fe560009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c346344162f7ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe569d9b700 1 -- 172.21.15.158:0/906749055 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560011830 con 0x7fe56c15dfb0 2023-10-16T11:22:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe569d9b700 1 -- 172.21.15.158:0/906749055 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe560011990 con 0x7fe56c15dfb0 2023-10-16T11:22:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.996+0000 7fe569d9b700 1 -- 172.21.15.158:0/906749055 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560011d20 con 0x7fe56c15dfb0 2023-10-16T11:22:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 -- 172.21.15.158:0/906749055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 msgr2=0x7fe56c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 --2- 172.21.15.158:0/906749055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c06e230 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0b61e0 tx=0x7fe560009b20 comp rx=0 tx=0).stop 2023-10-16T11:22:01.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 -- 172.21.15.158:0/906749055 shutdown_connections 2023-10-16T11:22:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 --2- 172.21.15.158:0/906749055 >> v2:172.21.15.172:3300/0 conn(0x7fe56c06f340 0x7fe56c06f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 --2- 172.21.15.158:0/906749055 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c06e230 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 -- 172.21.15.158:0/906749055 >> 172.21.15.158:0/906749055 conn(0x7fe56c0d02b0 msgr2=0x7fe56c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 -- 172.21.15.158:0/906749055 shutdown_connections 2023-10-16T11:22:01.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.997+0000 7fe5710c0700 1 -- 172.21.15.158:0/906749055 wait complete. 2023-10-16T11:22:01.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.998+0000 7fe5710c0700 1 Processor -- start 2023-10-16T11:22:01.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.998+0000 7fe5710c0700 1 -- start start 2023-10-16T11:22:01.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe5710c0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe56c06f340 0x7fe56c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe5710c0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe56c0cc5d0 0x7fe56c150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe5710c0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c0b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe5710c0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0b16e0 con 0x7fe56c15dfb0 2023-10-16T11:22:01.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe5710c0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c153e10 con 0x7fe56c0cc5d0 2023-10-16T11:22:01.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe5710c0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c155450 con 0x7fe56c06f340 2023-10-16T11:22:01.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe56b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c0b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe56b59e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c0b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43826/0 (socket says 172.21.15.158:43826) 2023-10-16T11:22:01.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe56b59e700 1 -- 172.21.15.158:0/2882151248 learned_addr learned my addr 172.21.15.158:0/2882151248 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:01.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:00.999+0000 7fe56a59c700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe56c0cc5d0 0x7fe56c150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56ad9d700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe56c06f340 0x7fe56c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56b59e700 1 -- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe56c06f340 msgr2=0x7fe56c1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56b59e700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe56c06f340 0x7fe56c1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56b59e700 1 -- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe56c0cc5d0 msgr2=0x7fe56c150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56b59e700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe56c0cc5d0 0x7fe56c150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56b59e700 1 -- 172.21.15.158:0/2882151248 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5600094a0 con 0x7fe56c15dfb0 2023-10-16T11:22:01.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56ad9d700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe56c06f340 0x7fe56c1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:01.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe56b59e700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c0b11a0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fe56400cc60 tx=0x7fe56400a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe5537fe700 1 -- 172.21.15.158:0/2882151248 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400ce60 con 0x7fe56c15dfb0 2023-10-16T11:22:01.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe5537fe700 1 -- 172.21.15.158:0/2882151248 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56400acf0 con 0x7fe56c15dfb0 2023-10-16T11:22:01.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe5537fe700 1 -- 172.21.15.158:0/2882151248 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe564003c40 con 0x7fe56c15dfb0 2023-10-16T11:22:01.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c155680 con 0x7fe56c15dfb0 2023-10-16T11:22:01.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.000+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c155c00 con 0x7fe56c15dfb0 2023-10-16T11:22:01.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.001+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c0681b0 con 0x7fe56c15dfb0 2023-10-16T11:22:01.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.001+0000 7fe5537fe700 1 -- 172.21.15.158:0/2882151248 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe56400f3f0 con 0x7fe56c15dfb0 2023-10-16T11:22:01.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.002+0000 7fe5537fe700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe55806b020 0x7fe55806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.002+0000 7fe5537fe700 1 -- 172.21.15.158:0/2882151248 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe564089de0 con 0x7fe56c15dfb0 2023-10-16T11:22:01.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.002+0000 7fe56ad9d700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe55806b020 0x7fe55806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.002+0000 7fe56ad9d700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe55806b020 0x7fe55806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe55c007fd0 tx=0x7fe55c009040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.006+0000 7fe5537fe700 1 -- 172.21.15.158:0/2882151248 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5640548d0 con 0x7fe56c15dfb0 2023-10-16T11:22:01.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.193+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 --> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe56c15e7c0 con 0x7fe55806b020 2023-10-16T11:22:01.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.201+0000 7fe5537fe700 1 -- 172.21.15.158:0/2882151248 <== mgr.13500 v2:172.21.15.158:6832/778561374 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289893 (secure 0 0 0) 0x7fe55c001cd0 con 0x7fe55806b020 2023-10-16T11:22:01.206 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:01.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe55806b020 msgr2=0x7fe55806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe55806b020 0x7fe55806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe55c007fd0 tx=0x7fe55c009040 comp rx=0 tx=0).stop 2023-10-16T11:22:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 msgr2=0x7fe56c0b11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c0b11a0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fe56400cc60 tx=0x7fe56400a4d0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 shutdown_connections 2023-10-16T11:22:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fe55806b020 0x7fe55806d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe56c15dfb0 0x7fe56c0b11a0 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe56c06f340 0x7fe56c1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 --2- 172.21.15.158:0/2882151248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe56c0cc5d0 0x7fe56c150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 >> 172.21.15.158:0/2882151248 conn(0x7fe56c0d02b0 msgr2=0x7fe56c15aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 shutdown_connections 2023-10-16T11:22:01.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.204+0000 7fe5710c0700 1 -- 172.21.15.158:0/2882151248 wait complete. 2023-10-16T11:22:01.211 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:22:01.218 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-16T11:22:00.643962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":444068,"kb_used_data":27996,"kb_used_omap":12,"kb_used_meta":415923,"kb_avail":754530652,"statfs":{"total":773094113280,"available":772639387648,"internally_reserved":0,"allocated":28667904,"data_stored":15993192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":425905719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.964027"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858210+0000","last_change":"2023-10-16T11:21:56.711930+0000","last_active":"2023-10-16T11:21:57.858210+0000","last_peered":"2023-10-16T11:21:57.858210+0000","last_clean":"2023-10-16T11:21:57.858210+0000","last_became_active":"2023-10-16T11:21:56.711653+0000","last_became_peered":"2023-10-16T11:21:56.711653+0000","last_unstale":"2023-10-16T11:21:57.858210+0000","last_undegraded":"2023-10-16T11:21:57.858210+0000","last_fullsized":"2023-10-16T11:21:57.858210+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:36.567286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858409+0000","last_change":"2023-10-16T11:21:56.710755+0000","last_active":"2023-10-16T11:21:57.858409+0000","last_peered":"2023-10-16T11:21:57.858409+0000","last_clean":"2023-10-16T11:21:57.858409+0000","last_became_active":"2023-10-16T11:21:56.710104+0000","last_became_peered":"2023-10-16T11:21:56.710104+0000","last_unstale":"2023-10-16T11:21:57.858409+0000","last_undegraded":"2023-10-16T11:21:57.858409+0000","last_fullsized":"2023-10-16T11:21:57.858409+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:33.478583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857244+0000","last_change":"2023-10-16T11:21:56.705147+0000","last_active":"2023-10-16T11:21:57.857244+0000","last_peered":"2023-10-16T11:21:57.857244+0000","last_clean":"2023-10-16T11:21:57.857244+0000","last_became_active":"2023-10-16T11:21:56.704777+0000","last_became_peered":"2023-10-16T11:21:56.704777+0000","last_unstale":"2023-10-16T11:21:57.857244+0000","last_undegraded":"2023-10-16T11:21:57.857244+0000","last_fullsized":"2023-10-16T11:21:57.857244+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:37.910699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704142+0000","last_change":"2023-10-16T11:21:56.717562+0000","last_active":"2023-10-16T11:21:57.704142+0000","last_peered":"2023-10-16T11:21:57.704142+0000","last_clean":"2023-10-16T11:21:57.704142+0000","last_became_active":"2023-10-16T11:21:56.717233+0000","last_became_peered":"2023-10-16T11:21:56.717233+0000","last_unstale":"2023-10-16T11:21:57.704142+0000","last_undegraded":"2023-10-16T11:21:57.704142+0000","last_fullsized":"2023-10-16T11:21:57.704142+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:02.423045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703681+0000","last_change":"2023-10-16T11:21:56.711666+0000","last_active":"2023-10-16T11:21:57.703681+0000","last_peered":"2023-10-16T11:21:57.703681+0000","last_clean":"2023-10-16T11:21:57.703681+0000","last_became_active":"2023-10-16T11:21:56.711375+0000","last_became_peered":"2023-10-16T11:21:56.711375+0000","last_unstale":"2023-10-16T11:21:57.703681+0000","last_undegraded":"2023-10-16T11:21:57.703681+0000","last_fullsized":"2023-10-16T11:21:57.703681+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:05.572254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858020+0000","last_change":"2023-10-16T11:21:56.707395+0000","last_active":"2023-10-16T11:21:57.858020+0000","last_peered":"2023-10-16T11:21:57.858020+0000","last_clean":"2023-10-16T11:21:57.858020+0000","last_became_active":"2023-10-16T11:21:56.707166+0000","last_became_peered":"2023-10-16T11:21:56.707166+0000","last_unstale":"2023-10-16T11:21:57.858020+0000","last_undegraded":"2023-10-16T11:21:57.858020+0000","last_fullsized":"2023-10-16T11:21:57.858020+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:17.294904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857615+0000","last_change":"2023-10-16T11:21:56.709062+0000","last_active":"2023-10-16T11:21:57.857615+0000","last_peered":"2023-10-16T11:21:57.857615+0000","last_clean":"2023-10-16T11:21:57.857615+0000","last_became_active":"2023-10-16T11:21:56.708771+0000","last_became_peered":"2023-10-16T11:21:56.708771+0000","last_unstale":"2023-10-16T11:21:57.857615+0000","last_undegraded":"2023-10-16T11:21:57.857615+0000","last_fullsized":"2023-10-16T11:21:57.857615+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:48.444835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856875+0000","last_change":"2023-10-16T11:21:56.712327+0000","last_active":"2023-10-16T11:21:57.856875+0000","last_peered":"2023-10-16T11:21:57.856875+0000","last_clean":"2023-10-16T11:21:57.856875+0000","last_became_active":"2023-10-16T11:21:56.712079+0000","last_became_peered":"2023-10-16T11:21:56.712079+0000","last_unstale":"2023-10-16T11:21:57.856875+0000","last_undegraded":"2023-10-16T11:21:57.856875+0000","last_fullsized":"2023-10-16T11:21:57.856875+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:14.692681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703582+0000","last_change":"2023-10-16T11:21:56.706652+0000","last_active":"2023-10-16T11:21:57.703582+0000","last_peered":"2023-10-16T11:21:57.703582+0000","last_clean":"2023-10-16T11:21:57.703582+0000","last_became_active":"2023-10-16T11:21:56.706377+0000","last_became_peered":"2023-10-16T11:21:56.706377+0000","last_unstale":"2023-10-16T11:21:57.703582+0000","last_undegraded":"2023-10-16T11:21:57.703582+0000","last_fullsized":"2023-10-16T11:21:57.703582+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:50.953400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857927+0000","last_change":"2023-10-16T11:21:56.704287+0000","last_active":"2023-10-16T11:21:57.857927+0000","last_peered":"2023-10-16T11:21:57.857927+0000","last_clean":"2023-10-16T11:21:57.857927+0000","last_became_active":"2023-10-16T11:21:56.704017+0000","last_became_peered":"2023-10-16T11:21:56.704017+0000","last_unstale":"2023-10-16T11:21:57.857927+0000","last_undegraded":"2023-10-16T11:21:57.857927+0000","last_fullsized":"2023-10-16T11:21:57.857927+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:40.750204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703565+0000","last_change":"2023-10-16T11:21:56.709289+0000","last_active":"2023-10-16T11:21:57.703565+0000","last_peered":"2023-10-16T11:21:57.703565+0000","last_clean":"2023-10-16T11:21:57.703565+0000","last_became_active":"2023-10-16T11:21:56.708562+0000","last_became_peered":"2023-10-16T11:21:56.708562+0000","last_unstale":"2023-10-16T11:21:57.703565+0000","last_undegraded":"2023-10-16T11:21:57.703565+0000","last_fullsized":"2023-10-16T11:21:57.703565+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:59.525098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858022+0000","last_change":"2023-10-16T11:21:56.705829+0000","last_active":"2023-10-16T11:21:57.858022+0000","last_peered":"2023-10-16T11:21:57.858022+0000","last_clean":"2023-10-16T11:21:57.858022+0000","last_became_active":"2023-10-16T11:21:56.705573+0000","last_became_peered":"2023-10-16T11:21:56.705573+0000","last_unstale":"2023-10-16T11:21:57.858022+0000","last_undegraded":"2023-10-16T11:21:57.858022+0000","last_fullsized":"2023-10-16T11:21:57.858022+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:22.111658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703484+0000","last_change":"2023-10-16T11:21:56.712269+0000","last_active":"2023-10-16T11:21:57.703484+0000","last_peered":"2023-10-16T11:21:57.703484+0000","last_clean":"2023-10-16T11:21:57.703484+0000","last_became_active":"2023-10-16T11:21:56.711921+0000","last_became_peered":"2023-10-16T11:21:56.711921+0000","last_unstale":"2023-10-16T11:21:57.703484+0000","last_undegraded":"2023-10-16T11:21:57.703484+0000","last_fullsized":"2023-10-16T11:21:57.703484+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:12.715253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857499+0000","last_change":"2023-10-16T11:21:56.707667+0000","last_active":"2023-10-16T11:21:57.857499+0000","last_peered":"2023-10-16T11:21:57.857499+0000","last_clean":"2023-10-16T11:21:57.857499+0000","last_became_active":"2023-10-16T11:21:56.707393+0000","last_became_peered":"2023-10-16T11:21:56.707393+0000","last_unstale":"2023-10-16T11:21:57.857499+0000","last_undegraded":"2023-10-16T11:21:57.857499+0000","last_fullsized":"2023-10-16T11:21:57.857499+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:42.811481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857499+0000","last_change":"2023-10-16T11:21:56.697151+0000","last_active":"2023-10-16T11:21:57.857499+0000","last_peered":"2023-10-16T11:21:57.857499+0000","last_clean":"2023-10-16T11:21:57.857499+0000","last_became_active":"2023-10-16T11:21:56.696864+0000","last_became_peered":"2023-10-16T11:21:56.696864+0000","last_unstale":"2023-10-16T11:21:57.857499+0000","last_undegraded":"2023-10-16T11:21:57.857499+0000","last_fullsized":"2023-10-16T11:21:57.857499+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:24.629606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858075+0000","last_change":"2023-10-16T11:21:56.723781+0000","last_active":"2023-10-16T11:21:57.858075+0000","last_peered":"2023-10-16T11:21:57.858075+0000","last_clean":"2023-10-16T11:21:57.858075+0000","last_became_active":"2023-10-16T11:21:56.723570+0000","last_became_peered":"2023-10-16T11:21:56.723570+0000","last_unstale":"2023-10-16T11:21:57.858075+0000","last_undegraded":"2023-10-16T11:21:57.858075+0000","last_fullsized":"2023-10-16T11:21:57.858075+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T11:37:37.575821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703621+0000","last_change":"2023-10-16T11:21:56.706554+0000","last_active":"2023-10-16T11:21:57.703621+0000","last_peered":"2023-10-16T11:21:57.703621+0000","last_clean":"2023-10-16T11:21:57.703621+0000","last_became_active":"2023-10-16T11:21:56.706270+0000","last_became_peered":"2023-10-16T11:21:56.706270+0000","last_unstale":"2023-10-16T11:21:57.703621+0000","last_undegraded":"2023-10-16T11:21:57.703621+0000","last_fullsized":"2023-10-16T11:21:57.703621+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:57.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703342+0000","last_change":"2023-10-16T11:21:56.719017+0000","last_active":"2023-10-16T11:21:57.703342+0000","last_peered":"2023-10-16T11:21:57.703342+0000","last_clean":"2023-10-16T11:21:57.703342+0000","last_became_active":"2023-10-16T11:21:56.718730+0000","last_became_peered":"2023-10-16T11:21:56.718730+0000","last_unstale":"2023-10-16T11:21:57.703342+0000","last_undegraded":"2023-10-16T11:21:57.703342+0000","last_fullsized":"2023-10-16T11:21:57.703342+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:31.587975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857584+0000","last_change":"2023-10-16T11:21:56.705001+0000","last_active":"2023-10-16T11:21:57.857584+0000","last_peered":"2023-10-16T11:21:57.857584+0000","last_clean":"2023-10-16T11:21:57.857584+0000","last_became_active":"2023-10-16T11:21:56.704427+0000","last_became_peered":"2023-10-16T11:21:56.704427+0000","last_unstale":"2023-10-16T11:21:57.857584+0000","last_undegraded":"2023-10-16T11:21:57.857584+0000","last_fullsized":"2023-10-16T11:21:57.857584+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:46.844456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704300+0000","last_change":"2023-10-16T11:21:56.723216+0000","last_active":"2023-10-16T11:21:57.704300+0000","last_peered":"2023-10-16T11:21:57.704300+0000","last_clean":"2023-10-16T11:21:57.704300+0000","last_became_active":"2023-10-16T11:21:56.719340+0000","last_became_peered":"2023-10-16T11:21:56.719340+0000","last_unstale":"2023-10-16T11:21:57.704300+0000","last_undegraded":"2023-10-16T11:21:57.704300+0000","last_fullsized":"2023-10-16T11:21:57.704300+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:54:06.372659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857593+0000","last_change":"2023-10-16T11:21:56.697423+0000","last_active":"2023-10-16T11:21:57.857593+0000","last_peered":"2023-10-16T11:21:57.857593+0000","last_clean":"2023-10-16T11:21:57.857593+0000","last_became_active":"2023-10-16T11:21:56.697164+0000","last_became_peered":"2023-10-16T11:21:56.697164+0000","last_unstale":"2023-10-16T11:21:57.857593+0000","last_undegraded":"2023-10-16T11:21:57.857593+0000","last_fullsized":"2023-10-16T11:21:57.857593+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:03.898885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703636+0000","last_change":"2023-10-16T11:21:56.718829+0000","last_active":"2023-10-16T11:21:57.703636+0000","last_peered":"2023-10-16T11:21:57.703636+0000","last_clean":"2023-10-16T11:21:57.703636+0000","last_became_active":"2023-10-16T11:21:56.718536+0000","last_became_peered":"2023-10-16T11:21:56.718536+0000","last_unstale":"2023-10-16T11:21:57.703636+0000","last_undegraded":"2023-10-16T11:21:57.703636+0000","last_fullsized":"2023-10-16T11:21:57.703636+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:33.478583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703871+0000","last_change":"2023-10-16T11:21:56.718601+0000","last_active":"2023-10-16T11:21:57.703871+0000","last_peered":"2023-10-16T11:21:57.703871+0000","last_clean":"2023-10-16T11:21:57.703871+0000","last_became_active":"2023-10-16T11:21:56.718292+0000","last_became_peered":"2023-10-16T11:21:56.718292+0000","last_unstale":"2023-10-16T11:21:57.703871+0000","last_undegraded":"2023-10-16T11:21:57.703871+0000","last_fullsized":"2023-10-16T11:21:57.703871+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:44.757920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856825+0000","last_change":"2023-10-16T11:21:56.710984+0000","last_active":"2023-10-16T11:21:57.856825+0000","last_peered":"2023-10-16T11:21:57.856825+0000","last_clean":"2023-10-16T11:21:57.856825+0000","last_became_active":"2023-10-16T11:21:56.710434+0000","last_became_peered":"2023-10-16T11:21:56.710434+0000","last_unstale":"2023-10-16T11:21:57.856825+0000","last_undegraded":"2023-10-16T11:21:57.856825+0000","last_fullsized":"2023-10-16T11:21:57.856825+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:23.369255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703693+0000","last_change":"2023-10-16T11:21:56.696074+0000","last_active":"2023-10-16T11:21:57.703693+0000","last_peered":"2023-10-16T11:21:57.703693+0000","last_clean":"2023-10-16T11:21:57.703693+0000","last_became_active":"2023-10-16T11:21:56.695799+0000","last_became_peered":"2023-10-16T11:21:56.695799+0000","last_unstale":"2023-10-16T11:21:57.703693+0000","last_undegraded":"2023-10-16T11:21:57.703693+0000","last_fullsized":"2023-10-16T11:21:57.703693+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:15.737826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704044+0000","last_change":"2023-10-16T11:21:56.718865+0000","last_active":"2023-10-16T11:21:57.704044+0000","last_peered":"2023-10-16T11:21:57.704044+0000","last_clean":"2023-10-16T11:21:57.704044+0000","last_became_active":"2023-10-16T11:21:56.718629+0000","last_became_peered":"2023-10-16T11:21:56.718629+0000","last_unstale":"2023-10-16T11:21:57.704044+0000","last_undegraded":"2023-10-16T11:21:57.704044+0000","last_fullsized":"2023-10-16T11:21:57.704044+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:01.116596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703431+0000","last_change":"2023-10-16T11:21:56.704641+0000","last_active":"2023-10-16T11:21:57.703431+0000","last_peered":"2023-10-16T11:21:57.703431+0000","last_clean":"2023-10-16T11:21:57.703431+0000","last_became_active":"2023-10-16T11:21:56.704373+0000","last_became_peered":"2023-10-16T11:21:56.704373+0000","last_unstale":"2023-10-16T11:21:57.703431+0000","last_undegraded":"2023-10-16T11:21:57.703431+0000","last_fullsized":"2023-10-16T11:21:57.703431+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:29.686801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858441+0000","last_change":"2023-10-16T11:21:56.718162+0000","last_active":"2023-10-16T11:21:57.858441+0000","last_peered":"2023-10-16T11:21:57.858441+0000","last_clean":"2023-10-16T11:21:57.858441+0000","last_became_active":"2023-10-16T11:21:56.717879+0000","last_became_peered":"2023-10-16T11:21:56.717879+0000","last_unstale":"2023-10-16T11:21:57.858441+0000","last_undegraded":"2023-10-16T11:21:57.858441+0000","last_fullsized":"2023-10-16T11:21:57.858441+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:42.844037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857687+0000","last_change":"2023-10-16T11:21:56.705175+0000","last_active":"2023-10-16T11:21:57.857687+0000","last_peered":"2023-10-16T11:21:57.857687+0000","last_clean":"2023-10-16T11:21:57.857687+0000","last_became_active":"2023-10-16T11:21:56.704853+0000","last_became_peered":"2023-10-16T11:21:56.704853+0000","last_unstale":"2023-10-16T11:21:57.857687+0000","last_undegraded":"2023-10-16T11:21:57.857687+0000","last_fullsized":"2023-10-16T11:21:57.857687+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:45.148019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857648+0000","last_change":"2023-10-16T11:21:56.709694+0000","last_active":"2023-10-16T11:21:57.857648+0000","last_peered":"2023-10-16T11:21:57.857648+0000","last_clean":"2023-10-16T11:21:57.857648+0000","last_became_active":"2023-10-16T11:21:56.709429+0000","last_became_peered":"2023-10-16T11:21:56.709429+0000","last_unstale":"2023-10-16T11:21:57.857648+0000","last_undegraded":"2023-10-16T11:21:57.857648+0000","last_fullsized":"2023-10-16T11:21:57.857648+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:37.010389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703461+0000","last_change":"2023-10-16T11:21:56.722428+0000","last_active":"2023-10-16T11:21:57.703461+0000","last_peered":"2023-10-16T11:21:57.703461+0000","last_clean":"2023-10-16T11:21:57.703461+0000","last_became_active":"2023-10-16T11:21:56.721531+0000","last_became_peered":"2023-10-16T11:21:56.721531+0000","last_unstale":"2023-10-16T11:21:57.703461+0000","last_undegraded":"2023-10-16T11:21:57.703461+0000","last_fullsized":"2023-10-16T11:21:57.703461+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:36.312330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704449+0000","last_change":"2023-10-16T11:21:56.721623+0000","last_active":"2023-10-16T11:21:57.704449+0000","last_peered":"2023-10-16T11:21:57.704449+0000","last_clean":"2023-10-16T11:21:57.704449+0000","last_became_active":"2023-10-16T11:21:56.720746+0000","last_became_peered":"2023-10-16T11:21:56.720746+0000","last_unstale":"2023-10-16T11:21:57.704449+0000","last_undegraded":"2023-10-16T11:21:57.704449+0000","last_fullsized":"2023-10-16T11:21:57.704449+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:27.548089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"269'4","reported_seq":16,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704097+0000","last_change":"2023-10-16T11:21:55.697221+0000","last_active":"2023-10-16T11:21:57.704097+0000","last_peered":"2023-10-16T11:21:57.704097+0000","last_clean":"2023-10-16T11:21:57.704097+0000","last_became_active":"2023-10-16T11:21:55.696910+0000","last_became_peered":"2023-10-16T11:21:55.696910+0000","last_unstale":"2023-10-16T11:21:57.704097+0000","last_undegraded":"2023-10-16T11:21:57.704097+0000","last_fullsized":"2023-10-16T11:21:57.704097+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:09.316693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703545+0000","last_change":"2023-10-16T11:21:56.709469+0000","last_active":"2023-10-16T11:21:57.703545+0000","last_peered":"2023-10-16T11:21:57.703545+0000","last_clean":"2023-10-16T11:21:57.703545+0000","last_became_active":"2023-10-16T11:21:56.708915+0000","last_became_peered":"2023-10-16T11:21:56.708915+0000","last_unstale":"2023-10-16T11:21:57.703545+0000","last_undegraded":"2023-10-16T11:21:57.703545+0000","last_fullsized":"2023-10-16T11:21:57.703545+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:16.287820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703437+0000","last_change":"2023-10-16T11:21:55.706566+0000","last_active":"2023-10-16T11:21:57.703437+0000","last_peered":"2023-10-16T11:21:57.703437+0000","last_clean":"2023-10-16T11:21:57.703437+0000","last_became_active":"2023-10-16T11:21:55.706144+0000","last_became_peered":"2023-10-16T11:21:55.706144+0000","last_unstale":"2023-10-16T11:21:57.703437+0000","last_undegraded":"2023-10-16T11:21:57.703437+0000","last_fullsized":"2023-10-16T11:21:57.703437+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:08.381066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858208+0000","last_change":"2023-10-16T11:21:56.700703+0000","last_active":"2023-10-16T11:21:57.858208+0000","last_peered":"2023-10-16T11:21:57.858208+0000","last_clean":"2023-10-16T11:21:57.858208+0000","last_became_active":"2023-10-16T11:21:56.699386+0000","last_became_peered":"2023-10-16T11:21:56.699386+0000","last_unstale":"2023-10-16T11:21:57.858208+0000","last_undegraded":"2023-10-16T11:21:57.858208+0000","last_fullsized":"2023-10-16T11:21:57.858208+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:42:36.312330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857666+0000","last_change":"2023-10-16T11:21:55.702119+0000","last_active":"2023-10-16T11:21:57.857666+0000","last_peered":"2023-10-16T11:21:57.857666+0000","last_clean":"2023-10-16T11:21:57.857666+0000","last_became_active":"2023-10-16T11:21:55.701861+0000","last_became_peered":"2023-10-16T11:21:55.701861+0000","last_unstale":"2023-10-16T11:21:57.857666+0000","last_undegraded":"2023-10-16T11:21:57.857666+0000","last_fullsized":"2023-10-16T11:21:57.857666+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:11.623629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858037+0000","last_change":"2023-10-16T11:21:56.706095+0000","last_active":"2023-10-16T11:21:57.858037+0000","last_peered":"2023-10-16T11:21:57.858037+0000","last_clean":"2023-10-16T11:21:57.858037+0000","last_became_active":"2023-10-16T11:21:56.705819+0000","last_became_peered":"2023-10-16T11:21:56.705819+0000","last_unstale":"2023-10-16T11:21:57.858037+0000","last_undegraded":"2023-10-16T11:21:57.858037+0000","last_fullsized":"2023-10-16T11:21:57.858037+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:47.158658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857880+0000","last_change":"2023-10-16T11:21:55.705209+0000","last_active":"2023-10-16T11:21:57.857880+0000","last_peered":"2023-10-16T11:21:57.857880+0000","last_clean":"2023-10-16T11:21:57.857880+0000","last_became_active":"2023-10-16T11:21:55.704955+0000","last_became_peered":"2023-10-16T11:21:55.704955+0000","last_unstale":"2023-10-16T11:21:57.857880+0000","last_undegraded":"2023-10-16T11:21:57.857880+0000","last_fullsized":"2023-10-16T11:21:57.857880+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:23.617512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856987+0000","last_change":"2023-10-16T11:21:56.725500+0000","last_active":"2023-10-16T11:21:57.856987+0000","last_peered":"2023-10-16T11:21:57.856987+0000","last_clean":"2023-10-16T11:21:57.856987+0000","last_became_active":"2023-10-16T11:21:56.724507+0000","last_became_peered":"2023-10-16T11:21:56.724507+0000","last_unstale":"2023-10-16T11:21:57.856987+0000","last_undegraded":"2023-10-16T11:21:57.856987+0000","last_fullsized":"2023-10-16T11:21:57.856987+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:03.801140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703319+0000","last_change":"2023-10-16T11:21:55.702206+0000","last_active":"2023-10-16T11:21:57.703319+0000","last_peered":"2023-10-16T11:21:57.703319+0000","last_clean":"2023-10-16T11:21:57.703319+0000","last_became_active":"2023-10-16T11:21:55.701947+0000","last_became_peered":"2023-10-16T11:21:55.701947+0000","last_unstale":"2023-10-16T11:21:57.703319+0000","last_undegraded":"2023-10-16T11:21:57.703319+0000","last_fullsized":"2023-10-16T11:21:57.703319+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:36.185352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703267+0000","last_change":"2023-10-16T11:21:56.697293+0000","last_active":"2023-10-16T11:21:57.703267+0000","last_peered":"2023-10-16T11:21:57.703267+0000","last_clean":"2023-10-16T11:21:57.703267+0000","last_became_active":"2023-10-16T11:21:56.696681+0000","last_became_peered":"2023-10-16T11:21:56.696681+0000","last_unstale":"2023-10-16T11:21:57.703267+0000","last_undegraded":"2023-10-16T11:21:57.703267+0000","last_fullsized":"2023-10-16T11:21:57.703267+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:31.647063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857275+0000","last_change":"2023-10-16T11:21:55.706390+0000","last_active":"2023-10-16T11:21:57.857275+0000","last_peered":"2023-10-16T11:21:57.857275+0000","last_clean":"2023-10-16T11:21:57.857275+0000","last_became_active":"2023-10-16T11:21:55.705653+0000","last_became_peered":"2023-10-16T11:21:55.705653+0000","last_unstale":"2023-10-16T11:21:57.857275+0000","last_undegraded":"2023-10-16T11:21:57.857275+0000","last_fullsized":"2023-10-16T11:21:57.857275+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:02.318497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703249+0000","last_change":"2023-10-16T11:21:56.709942+0000","last_active":"2023-10-16T11:21:57.703249+0000","last_peered":"2023-10-16T11:21:57.703249+0000","last_clean":"2023-10-16T11:21:57.703249+0000","last_became_active":"2023-10-16T11:21:56.709333+0000","last_became_peered":"2023-10-16T11:21:56.709333+0000","last_unstale":"2023-10-16T11:21:57.703249+0000","last_undegraded":"2023-10-16T11:21:57.703249+0000","last_fullsized":"2023-10-16T11:21:57.703249+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:45.148019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857749+0000","last_change":"2023-10-16T11:21:55.701489+0000","last_active":"2023-10-16T11:21:57.857749+0000","last_peered":"2023-10-16T11:21:57.857749+0000","last_clean":"2023-10-16T11:21:57.857749+0000","last_became_active":"2023-10-16T11:21:55.701217+0000","last_became_peered":"2023-10-16T11:21:55.701217+0000","last_unstale":"2023-10-16T11:21:57.857749+0000","last_undegraded":"2023-10-16T11:21:57.857749+0000","last_fullsized":"2023-10-16T11:21:57.857749+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:01:58.190877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857612+0000","last_change":"2023-10-16T11:21:56.710657+0000","last_active":"2023-10-16T11:21:57.857612+0000","last_peered":"2023-10-16T11:21:57.857612+0000","last_clean":"2023-10-16T11:21:57.857612+0000","last_became_active":"2023-10-16T11:21:56.710381+0000","last_became_peered":"2023-10-16T11:21:56.710381+0000","last_unstale":"2023-10-16T11:21:57.857612+0000","last_undegraded":"2023-10-16T11:21:57.857612+0000","last_fullsized":"2023-10-16T11:21:57.857612+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:42:04.928124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857436+0000","last_change":"2023-10-16T11:21:55.699726+0000","last_active":"2023-10-16T11:21:57.857436+0000","last_peered":"2023-10-16T11:21:57.857436+0000","last_clean":"2023-10-16T11:21:57.857436+0000","last_became_active":"2023-10-16T11:21:55.699217+0000","last_became_peered":"2023-10-16T11:21:55.699217+0000","last_unstale":"2023-10-16T11:21:57.857436+0000","last_undegraded":"2023-10-16T11:21:57.857436+0000","last_fullsized":"2023-10-16T11:21:57.857436+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:38.414368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704503+0000","last_change":"2023-10-16T11:21:56.721794+0000","last_active":"2023-10-16T11:21:57.704503+0000","last_peered":"2023-10-16T11:21:57.704503+0000","last_clean":"2023-10-16T11:21:57.704503+0000","last_became_active":"2023-10-16T11:21:56.721061+0000","last_became_peered":"2023-10-16T11:21:56.721061+0000","last_unstale":"2023-10-16T11:21:57.704503+0000","last_undegraded":"2023-10-16T11:21:57.704503+0000","last_fullsized":"2023-10-16T11:21:57.704503+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:55.390968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704173+0000","last_change":"2023-10-16T11:21:55.706154+0000","last_active":"2023-10-16T11:21:57.704173+0000","last_peered":"2023-10-16T11:21:57.704173+0000","last_clean":"2023-10-16T11:21:57.704173+0000","last_became_active":"2023-10-16T11:21:55.705858+0000","last_became_peered":"2023-10-16T11:21:55.705858+0000","last_unstale":"2023-10-16T11:21:57.704173+0000","last_undegraded":"2023-10-16T11:21:57.704173+0000","last_fullsized":"2023-10-16T11:21:57.704173+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:32.050360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858249+0000","last_change":"2023-10-16T11:21:56.703727+0000","last_active":"2023-10-16T11:21:57.858249+0000","last_peered":"2023-10-16T11:21:57.858249+0000","last_clean":"2023-10-16T11:21:57.858249+0000","last_became_active":"2023-10-16T11:21:56.703438+0000","last_became_peered":"2023-10-16T11:21:56.703438+0000","last_unstale":"2023-10-16T11:21:57.858249+0000","last_undegraded":"2023-10-16T11:21:57.858249+0000","last_fullsized":"2023-10-16T11:21:57.858249+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:40.755285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858149+0000","last_change":"2023-10-16T11:21:55.705977+0000","last_active":"2023-10-16T11:21:57.858149+0000","last_peered":"2023-10-16T11:21:57.858149+0000","last_clean":"2023-10-16T11:21:57.858149+0000","last_became_active":"2023-10-16T11:21:55.705705+0000","last_became_peered":"2023-10-16T11:21:55.705705+0000","last_unstale":"2023-10-16T11:21:57.858149+0000","last_undegraded":"2023-10-16T11:21:57.858149+0000","last_fullsized":"2023-10-16T11:21:57.858149+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:00:47.640952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858312+0000","last_change":"2023-10-16T11:21:56.700879+0000","last_active":"2023-10-16T11:21:57.858312+0000","last_peered":"2023-10-16T11:21:57.858312+0000","last_clean":"2023-10-16T11:21:57.858312+0000","last_became_active":"2023-10-16T11:21:56.699758+0000","last_became_peered":"2023-10-16T11:21:56.699758+0000","last_unstale":"2023-10-16T11:21:57.858312+0000","last_undegraded":"2023-10-16T11:21:57.858312+0000","last_fullsized":"2023-10-16T11:21:57.858312+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:16.046537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"269'3","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858047+0000","last_change":"2023-10-16T11:21:55.696214+0000","last_active":"2023-10-16T11:21:57.858047+0000","last_peered":"2023-10-16T11:21:57.858047+0000","last_clean":"2023-10-16T11:21:57.858047+0000","last_became_active":"2023-10-16T11:21:55.695955+0000","last_became_peered":"2023-10-16T11:21:55.695955+0000","last_unstale":"2023-10-16T11:21:57.858047+0000","last_undegraded":"2023-10-16T11:21:57.858047+0000","last_fullsized":"2023-10-16T11:21:57.858047+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:16.287809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703572+0000","last_change":"2023-10-16T11:21:56.720789+0000","last_active":"2023-10-16T11:21:57.703572+0000","last_peered":"2023-10-16T11:21:57.703572+0000","last_clean":"2023-10-16T11:21:57.703572+0000","last_became_active":"2023-10-16T11:21:56.717572+0000","last_became_peered":"2023-10-16T11:21:56.717572+0000","last_unstale":"2023-10-16T11:21:57.703572+0000","last_undegraded":"2023-10-16T11:21:57.703572+0000","last_fullsized":"2023-10-16T11:21:57.703572+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:02.862244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"269'6","reported_seq":18,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857453+0000","last_change":"2023-10-16T11:21:55.703343+0000","last_active":"2023-10-16T11:21:57.857453+0000","last_peered":"2023-10-16T11:21:57.857453+0000","last_clean":"2023-10-16T11:21:57.857453+0000","last_became_active":"2023-10-16T11:21:55.703055+0000","last_became_peered":"2023-10-16T11:21:55.703055+0000","last_unstale":"2023-10-16T11:21:57.857453+0000","last_undegraded":"2023-10-16T11:21:57.857453+0000","last_fullsized":"2023-10-16T11:21:57.857453+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:17.708721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703274+0000","last_change":"2023-10-16T11:21:56.699974+0000","last_active":"2023-10-16T11:21:57.703274+0000","last_peered":"2023-10-16T11:21:57.703274+0000","last_clean":"2023-10-16T11:21:57.703274+0000","last_became_active":"2023-10-16T11:21:56.699656+0000","last_became_peered":"2023-10-16T11:21:56.699656+0000","last_unstale":"2023-10-16T11:21:57.703274+0000","last_undegraded":"2023-10-16T11:21:57.703274+0000","last_fullsized":"2023-10-16T11:21:57.703274+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:04.292581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858031+0000","last_change":"2023-10-16T11:21:55.704976+0000","last_active":"2023-10-16T11:21:57.858031+0000","last_peered":"2023-10-16T11:21:57.858031+0000","last_clean":"2023-10-16T11:21:57.858031+0000","last_became_active":"2023-10-16T11:21:55.704682+0000","last_became_peered":"2023-10-16T11:21:55.704682+0000","last_unstale":"2023-10-16T11:21:57.858031+0000","last_undegraded":"2023-10-16T11:21:57.858031+0000","last_fullsized":"2023-10-16T11:21:57.858031+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:23.560949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704097+0000","last_change":"2023-10-16T11:21:56.710162+0000","last_active":"2023-10-16T11:21:57.704097+0000","last_peered":"2023-10-16T11:21:57.704097+0000","last_clean":"2023-10-16T11:21:57.704097+0000","last_became_active":"2023-10-16T11:21:56.709882+0000","last_became_peered":"2023-10-16T11:21:56.709882+0000","last_unstale":"2023-10-16T11:21:57.704097+0000","last_undegraded":"2023-10-16T11:21:57.704097+0000","last_fullsized":"2023-10-16T11:21:57.704097+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:57.829265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703246+0000","last_change":"2023-10-16T11:21:55.701157+0000","last_active":"2023-10-16T11:21:57.703246+0000","last_peered":"2023-10-16T11:21:57.703246+0000","last_clean":"2023-10-16T11:21:57.703246+0000","last_became_active":"2023-10-16T11:21:55.700835+0000","last_became_peered":"2023-10-16T11:21:55.700835+0000","last_unstale":"2023-10-16T11:21:57.703246+0000","last_undegraded":"2023-10-16T11:21:57.703246+0000","last_fullsized":"2023-10-16T11:21:57.703246+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:49.072272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703396+0000","last_change":"2023-10-16T11:21:56.717105+0000","last_active":"2023-10-16T11:21:57.703396+0000","last_peered":"2023-10-16T11:21:57.703396+0000","last_clean":"2023-10-16T11:21:57.703396+0000","last_became_active":"2023-10-16T11:21:56.716797+0000","last_became_peered":"2023-10-16T11:21:56.716797+0000","last_unstale":"2023-10-16T11:21:57.703396+0000","last_undegraded":"2023-10-16T11:21:57.703396+0000","last_fullsized":"2023-10-16T11:21:57.703396+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:47.158658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857807+0000","last_change":"2023-10-16T11:21:55.710302+0000","last_active":"2023-10-16T11:21:57.857807+0000","last_peered":"2023-10-16T11:21:57.857807+0000","last_clean":"2023-10-16T11:21:57.857807+0000","last_became_active":"2023-10-16T11:21:55.710013+0000","last_became_peered":"2023-10-16T11:21:55.710013+0000","last_unstale":"2023-10-16T11:21:57.857807+0000","last_undegraded":"2023-10-16T11:21:57.857807+0000","last_fullsized":"2023-10-16T11:21:57.857807+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:54.075175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857669+0000","last_change":"2023-10-16T11:21:56.710801+0000","last_active":"2023-10-16T11:21:57.857669+0000","last_peered":"2023-10-16T11:21:57.857669+0000","last_clean":"2023-10-16T11:21:57.857669+0000","last_became_active":"2023-10-16T11:21:56.710522+0000","last_became_peered":"2023-10-16T11:21:56.710522+0000","last_unstale":"2023-10-16T11:21:57.857669+0000","last_undegraded":"2023-10-16T11:21:57.857669+0000","last_fullsized":"2023-10-16T11:21:57.857669+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:52.846398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"269'8","reported_seq":20,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704268+0000","last_change":"2023-10-16T11:21:55.709635+0000","last_active":"2023-10-16T11:21:57.704268+0000","last_peered":"2023-10-16T11:21:57.704268+0000","last_clean":"2023-10-16T11:21:57.704268+0000","last_became_active":"2023-10-16T11:21:55.709394+0000","last_became_peered":"2023-10-16T11:21:55.709394+0000","last_unstale":"2023-10-16T11:21:57.704268+0000","last_undegraded":"2023-10-16T11:21:57.704268+0000","last_fullsized":"2023-10-16T11:21:57.704268+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:41.804386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857589+0000","last_change":"2023-10-16T11:21:56.704300+0000","last_active":"2023-10-16T11:21:57.857589+0000","last_peered":"2023-10-16T11:21:57.857589+0000","last_clean":"2023-10-16T11:21:57.857589+0000","last_became_active":"2023-10-16T11:21:56.704023+0000","last_became_peered":"2023-10-16T11:21:56.704023+0000","last_unstale":"2023-10-16T11:21:57.857589+0000","last_undegraded":"2023-10-16T11:21:57.857589+0000","last_fullsized":"2023-10-16T11:21:57.857589+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:19.802321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703830+0000","last_change":"2023-10-16T11:21:55.708263+0000","last_active":"2023-10-16T11:21:57.703830+0000","last_peered":"2023-10-16T11:21:57.703830+0000","last_clean":"2023-10-16T11:21:57.703830+0000","last_became_active":"2023-10-16T11:21:55.707990+0000","last_became_peered":"2023-10-16T11:21:55.707990+0000","last_unstale":"2023-10-16T11:21:57.703830+0000","last_undegraded":"2023-10-16T11:21:57.703830+0000","last_fullsized":"2023-10-16T11:21:57.703830+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:04.429083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703753+0000","last_change":"2023-10-16T11:21:56.695808+0000","last_active":"2023-10-16T11:21:57.703753+0000","last_peered":"2023-10-16T11:21:57.703753+0000","last_clean":"2023-10-16T11:21:57.703753+0000","last_became_active":"2023-10-16T11:21:56.695575+0000","last_became_peered":"2023-10-16T11:21:56.695575+0000","last_unstale":"2023-10-16T11:21:57.703753+0000","last_undegraded":"2023-10-16T11:21:57.703753+0000","last_fullsized":"2023-10-16T11:21:57.703753+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:13.902954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858119+0000","last_change":"2023-10-16T11:21:55.703387+0000","last_active":"2023-10-16T11:21:57.858119+0000","last_peered":"2023-10-16T11:21:57.858119+0000","last_clean":"2023-10-16T11:21:57.858119+0000","last_became_active":"2023-10-16T11:21:55.703110+0000","last_became_peered":"2023-10-16T11:21:55.703110+0000","last_unstale":"2023-10-16T11:21:57.858119+0000","last_undegraded":"2023-10-16T11:21:57.858119+0000","last_fullsized":"2023-10-16T11:21:57.858119+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:48.552813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858137+0000","last_change":"2023-10-16T11:21:56.704125+0000","last_active":"2023-10-16T11:21:57.858137+0000","last_peered":"2023-10-16T11:21:57.858137+0000","last_clean":"2023-10-16T11:21:57.858137+0000","last_became_active":"2023-10-16T11:21:56.703850+0000","last_became_peered":"2023-10-16T11:21:56.703850+0000","last_unstale":"2023-10-16T11:21:57.858137+0000","last_undegraded":"2023-10-16T11:21:57.858137+0000","last_fullsized":"2023-10-16T11:21:57.858137+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:00:48.648047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"269'1","reported_seq":13,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703960+0000","last_change":"2023-10-16T11:21:55.704758+0000","last_active":"2023-10-16T11:21:57.703960+0000","last_peered":"2023-10-16T11:21:57.703960+0000","last_clean":"2023-10-16T11:21:57.703960+0000","last_became_active":"2023-10-16T11:21:55.704477+0000","last_became_peered":"2023-10-16T11:21:55.704477+0000","last_unstale":"2023-10-16T11:21:57.703960+0000","last_undegraded":"2023-10-16T11:21:57.703960+0000","last_fullsized":"2023-10-16T11:21:57.703960+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:23.418643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857845+0000","last_change":"2023-10-16T11:21:56.710714+0000","last_active":"2023-10-16T11:21:57.857845+0000","last_peered":"2023-10-16T11:21:57.857845+0000","last_clean":"2023-10-16T11:21:57.857845+0000","last_became_active":"2023-10-16T11:21:56.710401+0000","last_became_peered":"2023-10-16T11:21:56.710401+0000","last_unstale":"2023-10-16T11:21:57.857845+0000","last_undegraded":"2023-10-16T11:21:57.857845+0000","last_fullsized":"2023-10-16T11:21:57.857845+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:26.954413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857958+0000","last_change":"2023-10-16T11:21:55.705874+0000","last_active":"2023-10-16T11:21:57.857958+0000","last_peered":"2023-10-16T11:21:57.857958+0000","last_clean":"2023-10-16T11:21:57.857958+0000","last_became_active":"2023-10-16T11:21:55.705604+0000","last_became_peered":"2023-10-16T11:21:55.705604+0000","last_unstale":"2023-10-16T11:21:57.857958+0000","last_undegraded":"2023-10-16T11:21:57.857958+0000","last_fullsized":"2023-10-16T11:21:57.857958+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:21.104563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704075+0000","last_change":"2023-10-16T11:21:56.722739+0000","last_active":"2023-10-16T11:21:57.704075+0000","last_peered":"2023-10-16T11:21:57.704075+0000","last_clean":"2023-10-16T11:21:57.704075+0000","last_became_active":"2023-10-16T11:21:56.721829+0000","last_became_peered":"2023-10-16T11:21:56.721829+0000","last_unstale":"2023-10-16T11:21:57.704075+0000","last_undegraded":"2023-10-16T11:21:57.704075+0000","last_fullsized":"2023-10-16T11:21:57.704075+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:16.046537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703415+0000","last_change":"2023-10-16T11:21:55.697184+0000","last_active":"2023-10-16T11:21:57.703415+0000","last_peered":"2023-10-16T11:21:57.703415+0000","last_clean":"2023-10-16T11:21:57.703415+0000","last_became_active":"2023-10-16T11:21:55.695179+0000","last_became_peered":"2023-10-16T11:21:55.695179+0000","last_unstale":"2023-10-16T11:21:57.703415+0000","last_undegraded":"2023-10-16T11:21:57.703415+0000","last_fullsized":"2023-10-16T11:21:57.703415+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:49.200047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703306+0000","last_change":"2023-10-16T11:21:56.697464+0000","last_active":"2023-10-16T11:21:57.703306+0000","last_peered":"2023-10-16T11:21:57.703306+0000","last_clean":"2023-10-16T11:21:57.703306+0000","last_became_active":"2023-10-16T11:21:56.697030+0000","last_became_peered":"2023-10-16T11:21:56.697030+0000","last_unstale":"2023-10-16T11:21:57.703306+0000","last_undegraded":"2023-10-16T11:21:57.703306+0000","last_fullsized":"2023-10-16T11:21:57.703306+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:17.280483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703727+0000","last_change":"2023-10-16T11:21:55.706914+0000","last_active":"2023-10-16T11:21:57.703727+0000","last_peered":"2023-10-16T11:21:57.703727+0000","last_clean":"2023-10-16T11:21:57.703727+0000","last_became_active":"2023-10-16T11:21:55.706627+0000","last_became_peered":"2023-10-16T11:21:55.706627+0000","last_unstale":"2023-10-16T11:21:57.703727+0000","last_undegraded":"2023-10-16T11:21:57.703727+0000","last_fullsized":"2023-10-16T11:21:57.703727+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:37.849795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703150+0000","last_change":"2023-10-16T11:21:56.704983+0000","last_active":"2023-10-16T11:21:57.703150+0000","last_peered":"2023-10-16T11:21:57.703150+0000","last_clean":"2023-10-16T11:21:57.703150+0000","last_became_active":"2023-10-16T11:21:56.704713+0000","last_became_peered":"2023-10-16T11:21:56.704713+0000","last_unstale":"2023-10-16T11:21:57.703150+0000","last_undegraded":"2023-10-16T11:21:57.703150+0000","last_fullsized":"2023-10-16T11:21:57.703150+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:43.657508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857228+0000","last_change":"2023-10-16T11:21:55.706851+0000","last_active":"2023-10-16T11:21:57.857228+0000","last_peered":"2023-10-16T11:21:57.857228+0000","last_clean":"2023-10-16T11:21:57.857228+0000","last_became_active":"2023-10-16T11:21:55.706500+0000","last_became_peered":"2023-10-16T11:21:55.706500+0000","last_unstale":"2023-10-16T11:21:57.857228+0000","last_undegraded":"2023-10-16T11:21:57.857228+0000","last_fullsized":"2023-10-16T11:21:57.857228+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:54.075175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857955+0000","last_change":"2023-10-16T11:21:56.707579+0000","last_active":"2023-10-16T11:21:57.857955+0000","last_peered":"2023-10-16T11:21:57.857955+0000","last_clean":"2023-10-16T11:21:57.857955+0000","last_became_active":"2023-10-16T11:21:56.707257+0000","last_became_peered":"2023-10-16T11:21:56.707257+0000","last_unstale":"2023-10-16T11:21:57.857955+0000","last_undegraded":"2023-10-16T11:21:57.857955+0000","last_fullsized":"2023-10-16T11:21:57.857955+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:24.624607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.856997+0000","last_change":"2023-10-16T11:21:55.706143+0000","last_active":"2023-10-16T11:21:57.856997+0000","last_peered":"2023-10-16T11:21:57.856997+0000","last_clean":"2023-10-16T11:21:57.856997+0000","last_became_active":"2023-10-16T11:21:55.705705+0000","last_became_peered":"2023-10-16T11:21:55.705705+0000","last_unstale":"2023-10-16T11:21:57.856997+0000","last_undegraded":"2023-10-16T11:21:57.856997+0000","last_fullsized":"2023-10-16T11:21:57.856997+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:34.971548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704596+0000","last_change":"2023-10-16T11:21:56.721959+0000","last_active":"2023-10-16T11:21:57.704596+0000","last_peered":"2023-10-16T11:21:57.704596+0000","last_clean":"2023-10-16T11:21:57.704596+0000","last_became_active":"2023-10-16T11:21:56.721364+0000","last_became_peered":"2023-10-16T11:21:56.721364+0000","last_unstale":"2023-10-16T11:21:57.704596+0000","last_undegraded":"2023-10-16T11:21:57.704596+0000","last_fullsized":"2023-10-16T11:21:57.704596+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:51.447799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"269'2","reported_seq":14,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858002+0000","last_change":"2023-10-16T11:21:55.705184+0000","last_active":"2023-10-16T11:21:57.858002+0000","last_peered":"2023-10-16T11:21:57.858002+0000","last_clean":"2023-10-16T11:21:57.858002+0000","last_became_active":"2023-10-16T11:21:55.704903+0000","last_became_peered":"2023-10-16T11:21:55.704903+0000","last_unstale":"2023-10-16T11:21:57.858002+0000","last_undegraded":"2023-10-16T11:21:57.858002+0000","last_fullsized":"2023-10-16T11:21:57.858002+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:57.229633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858237+0000","last_change":"2023-10-16T11:21:56.722149+0000","last_active":"2023-10-16T11:21:57.858237+0000","last_peered":"2023-10-16T11:21:57.858237+0000","last_clean":"2023-10-16T11:21:57.858237+0000","last_became_active":"2023-10-16T11:21:56.721753+0000","last_became_peered":"2023-10-16T11:21:56.721753+0000","last_unstale":"2023-10-16T11:21:57.858237+0000","last_undegraded":"2023-10-16T11:21:57.858237+0000","last_fullsized":"2023-10-16T11:21:57.858237+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:44.589323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703737+0000","last_change":"2023-10-16T11:21:55.708596+0000","last_active":"2023-10-16T11:21:57.703737+0000","last_peered":"2023-10-16T11:21:57.703737+0000","last_clean":"2023-10-16T11:21:57.703737+0000","last_became_active":"2023-10-16T11:21:55.708323+0000","last_became_peered":"2023-10-16T11:21:55.708323+0000","last_unstale":"2023-10-16T11:21:57.703737+0000","last_undegraded":"2023-10-16T11:21:57.703737+0000","last_fullsized":"2023-10-16T11:21:57.703737+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:42.928253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858284+0000","last_change":"2023-10-16T11:21:56.710801+0000","last_active":"2023-10-16T11:21:57.858284+0000","last_peered":"2023-10-16T11:21:57.858284+0000","last_clean":"2023-10-16T11:21:57.858284+0000","last_became_active":"2023-10-16T11:21:56.710464+0000","last_became_peered":"2023-10-16T11:21:56.710464+0000","last_unstale":"2023-10-16T11:21:57.858284+0000","last_undegraded":"2023-10-16T11:21:57.858284+0000","last_fullsized":"2023-10-16T11:21:57.858284+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:02.862244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858137+0000","last_change":"2023-10-16T11:21:55.707466+0000","last_active":"2023-10-16T11:21:57.858137+0000","last_peered":"2023-10-16T11:21:57.858137+0000","last_clean":"2023-10-16T11:21:57.858137+0000","last_became_active":"2023-10-16T11:21:55.707198+0000","last_became_peered":"2023-10-16T11:21:55.707198+0000","last_unstale":"2023-10-16T11:21:57.858137+0000","last_undegraded":"2023-10-16T11:21:57.858137+0000","last_fullsized":"2023-10-16T11:21:57.858137+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:25.947319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703662+0000","last_change":"2023-10-16T11:21:56.697353+0000","last_active":"2023-10-16T11:21:57.703662+0000","last_peered":"2023-10-16T11:21:57.703662+0000","last_clean":"2023-10-16T11:21:57.703662+0000","last_became_active":"2023-10-16T11:21:56.697071+0000","last_became_peered":"2023-10-16T11:21:56.697071+0000","last_unstale":"2023-10-16T11:21:57.703662+0000","last_undegraded":"2023-10-16T11:21:57.703662+0000","last_fullsized":"2023-10-16T11:21:57.703662+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:15.826809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703720+0000","last_change":"2023-10-16T11:21:55.709288+0000","last_active":"2023-10-16T11:21:57.703720+0000","last_peered":"2023-10-16T11:21:57.703720+0000","last_clean":"2023-10-16T11:21:57.703720+0000","last_became_active":"2023-10-16T11:21:55.709024+0000","last_became_peered":"2023-10-16T11:21:55.709024+0000","last_unstale":"2023-10-16T11:21:57.703720+0000","last_undegraded":"2023-10-16T11:21:57.703720+0000","last_fullsized":"2023-10-16T11:21:57.703720+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:55.192509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703053+0000","last_change":"2023-10-16T11:21:56.710027+0000","last_active":"2023-10-16T11:21:57.703053+0000","last_peered":"2023-10-16T11:21:57.703053+0000","last_clean":"2023-10-16T11:21:57.703053+0000","last_became_active":"2023-10-16T11:21:56.709730+0000","last_became_peered":"2023-10-16T11:21:56.709730+0000","last_unstale":"2023-10-16T11:21:57.703053+0000","last_undegraded":"2023-10-16T11:21:57.703053+0000","last_fullsized":"2023-10-16T11:21:57.703053+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:49.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857303+0000","last_change":"2023-10-16T11:21:55.702934+0000","last_active":"2023-10-16T11:21:57.857303+0000","last_peered":"2023-10-16T11:21:57.857303+0000","last_clean":"2023-10-16T11:21:57.857303+0000","last_became_active":"2023-10-16T11:21:55.702724+0000","last_became_peered":"2023-10-16T11:21:55.702724+0000","last_unstale":"2023-10-16T11:21:57.857303+0000","last_undegraded":"2023-10-16T11:21:57.857303+0000","last_fullsized":"2023-10-16T11:21:57.857303+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:03.648335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857763+0000","last_change":"2023-10-16T11:21:56.718659+0000","last_active":"2023-10-16T11:21:57.857763+0000","last_peered":"2023-10-16T11:21:57.857763+0000","last_clean":"2023-10-16T11:21:57.857763+0000","last_became_active":"2023-10-16T11:21:56.718394+0000","last_became_peered":"2023-10-16T11:21:56.718394+0000","last_unstale":"2023-10-16T11:21:57.857763+0000","last_undegraded":"2023-10-16T11:21:57.857763+0000","last_fullsized":"2023-10-16T11:21:57.857763+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:31.587975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857699+0000","last_change":"2023-10-16T11:21:55.705027+0000","last_active":"2023-10-16T11:21:57.857699+0000","last_peered":"2023-10-16T11:21:57.857699+0000","last_clean":"2023-10-16T11:21:57.857699+0000","last_became_active":"2023-10-16T11:21:55.704583+0000","last_became_peered":"2023-10-16T11:21:55.704583+0000","last_unstale":"2023-10-16T11:21:57.857699+0000","last_undegraded":"2023-10-16T11:21:57.857699+0000","last_fullsized":"2023-10-16T11:21:57.857699+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:12.908701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.703287+0000","last_change":"2023-10-16T11:21:56.703667+0000","last_active":"2023-10-16T11:21:57.703287+0000","last_peered":"2023-10-16T11:21:57.703287+0000","last_clean":"2023-10-16T11:21:57.703287+0000","last_became_active":"2023-10-16T11:21:56.703381+0000","last_became_peered":"2023-10-16T11:21:56.703381+0000","last_unstale":"2023-10-16T11:21:57.703287+0000","last_undegraded":"2023-10-16T11:21:57.703287+0000","last_fullsized":"2023-10-16T11:21:57.703287+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:53.112967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858071+0000","last_change":"2023-10-16T11:21:55.710236+0000","last_active":"2023-10-16T11:21:57.858071+0000","last_peered":"2023-10-16T11:21:57.858071+0000","last_clean":"2023-10-16T11:21:57.858071+0000","last_became_active":"2023-10-16T11:21:55.710008+0000","last_became_peered":"2023-10-16T11:21:55.710008+0000","last_unstale":"2023-10-16T11:21:57.858071+0000","last_undegraded":"2023-10-16T11:21:57.858071+0000","last_fullsized":"2023-10-16T11:21:57.858071+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:36.903604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858493+0000","last_change":"2023-10-16T11:21:56.708680+0000","last_active":"2023-10-16T11:21:57.858493+0000","last_peered":"2023-10-16T11:21:57.858493+0000","last_clean":"2023-10-16T11:21:57.858493+0000","last_became_active":"2023-10-16T11:21:56.708402+0000","last_became_peered":"2023-10-16T11:21:56.708402+0000","last_unstale":"2023-10-16T11:21:57.858493+0000","last_undegraded":"2023-10-16T11:21:57.858493+0000","last_fullsized":"2023-10-16T11:21:57.858493+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:57.634987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":12,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.857081+0000","last_change":"2023-10-16T11:21:55.706289+0000","last_active":"2023-10-16T11:21:57.857081+0000","last_peered":"2023-10-16T11:21:57.857081+0000","last_clean":"2023-10-16T11:21:57.857081+0000","last_became_active":"2023-10-16T11:21:55.706072+0000","last_became_peered":"2023-10-16T11:21:55.706072+0000","last_unstale":"2023-10-16T11:21:57.857081+0000","last_undegraded":"2023-10-16T11:21:57.857081+0000","last_fullsized":"2023-10-16T11:21:57.857081+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:54.673050+0000","last_clean_scrub_stamp":"2023-10-16T11:21:54.673050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:57.009588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":11,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.858273+0000","last_change":"2023-10-16T11:21:56.707540+0000","last_active":"2023-10-16T11:21:57.858273+0000","last_peered":"2023-10-16T11:21:57.858273+0000","last_clean":"2023-10-16T11:21:57.858273+0000","last_became_active":"2023-10-16T11:21:56.707234+0000","last_became_peered":"2023-10-16T11:21:56.707234+0000","last_unstale":"2023-10-16T11:21:57.858273+0000","last_undegraded":"2023-10-16T11:21:57.858273+0000","last_fullsized":"2023-10-16T11:21:57.858273+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T11:21:55.680145+0000","last_clean_scrub_stamp":"2023-10-16T11:21:55.680145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:06.029695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"260'403","reported_seq":1454,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T11:21:57.704363+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:21:57.704363+0000","last_peered":"2023-10-16T11:21:57.704363+0000","last_clean":"2023-10-16T11:21:57.704363+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:21:57.704363+0000","last_undegraded":"2023-10-16T11:21:57.704363+0000","last_fullsized":"2023-10-16T11:21:57.704363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576015,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57320,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314520,"statfs":{"total":96636764160,"available":96578068480,"internally_reserved":0,"allocated":4026368,"data_stored":2442232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73799999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.68300000000000005,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.70199999999999996}]},{"osd":2,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.70399999999999996,"15min":0.66700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.622},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":9.516,"15min":9.516},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67000000000000004,"15min":0.64000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.441},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.59299999999999997,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.624,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67400000000000004,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.64800000000000002,"15min":0.57899999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77800000000000002,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.68000000000000005,"15min":0.625},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.70999999999999996,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.69299999999999995,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72199999999999998,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.65300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56744,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315096,"statfs":{"total":96636764160,"available":96578658304,"internally_reserved":0,"allocated":3436544,"data_stored":1852446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.57999999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.86299999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56499999999999995,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84299999999999997,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71899999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":2.387,"15min":2.387},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.314},"max":{"1min":0.78100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46500000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59899999999999998,"15min":0.61599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88800000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81000000000000005,"5min":1.02,"15min":1.02},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.34499999999999997}]},{"osd":5,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":3.335,"15min":3.335},"last":0.36599999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":2.524,"15min":2.524},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49008,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322832,"statfs":{"total":96636764160,"available":96586579968,"internally_reserved":0,"allocated":4026368,"data_stored":2442300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.55500000000000005,"15min":0.60299999999999998},"last":0.252},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40300000000000002,"15min":0.439},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.44500000000000001,"5min":0.58399999999999996,"15min":0.77200000000000002},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50700000000000001,"5min":1.272,"15min":1.272},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.50600000000000001,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.374},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":1.351,"15min":1.351},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.64200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.45100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":1.391,"15min":1.391},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60599999999999998,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.874,"15min":1.089},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.216},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64300000000000002,"15min":0.68600000000000005},"min":{"1min":0.46600000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":1.038,"15min":1.1180000000000001},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88900000000000001,"5min":0.97399999999999998,"15min":1.008},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.66900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.88500000000000001,"15min":1.0780000000000001},"last":0.54000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3428352,"data_stored":1851842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.40799999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.26500000000000001,"5min":0.184,"15min":0.184},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36199999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.371},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.44},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.30299999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0309999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.0389999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.501}]},{"osd":6,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.629,"15min":0.66500000000000004},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.67200000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64500000000000002,"15min":0.68999999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576015,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52584,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319256,"statfs":{"total":96636764160,"available":96582918144,"internally_reserved":0,"allocated":3432448,"data_stored":1846424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.245,"5min":0.188,"15min":0.188},"max":{"1min":0.47699999999999998,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.318},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.437},"min":{"1min":0.255,"5min":0.193,"15min":0.193},"max":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.29599999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.436,"15min":0.46400000000000002},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.41099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45300000000000001,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.27900000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.376}]},{"osd":4,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":1.097},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86499999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83099999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92700000000000005,"5min":1.0169999999999999,"15min":1.028},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.498,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85899999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.70399999999999996},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":0.99099999999999999,"15min":1.071},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.97799999999999998,"15min":1.077},"last":0.74399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56748,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315092,"statfs":{"total":96636764160,"available":96578654208,"internally_reserved":0,"allocated":3432448,"data_stored":1851876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36499999999999999,"15min":0.39000000000000001},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.621,"5min":1.032,"15min":1.032},"last":0.436},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.36799999999999999,"15min":0.40500000000000003},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.52200000000000002,"5min":1.341,"15min":1.341},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40899999999999997,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55000000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.44400000000000001,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.51300000000000001,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.502},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45900000000000002,"15min":0.503},"min":{"1min":0.34799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54600000000000004,"5min":0.96599999999999997,"15min":1.024},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.623,"15min":0.65300000000000002},"min":{"1min":0.45500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3220000000000001,"5min":2.629,"15min":2.629},"last":0.877}]},{"osd":5,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.66700000000000004},"min":{"1min":0.443,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":1.0149999999999999,"15min":1.026},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.80400000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81100000000000005,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.75900000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":0.71399999999999997},"min":{"1min":0.49099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.304,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65800000000000003,"15min":0.69899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.86399999999999999,"5min":1.006,"15min":1.1899999999999999},"last":0.90100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576015,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52608,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319232,"statfs":{"total":96636764160,"available":96582893568,"internally_reserved":0,"allocated":3452928,"data_stored":1854196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.60899999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75,"5min":0.83499999999999996,"15min":1.0589999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.61399999999999999},"min":{"1min":0.41899999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81899999999999995,"5min":0.83699999999999997,"15min":0.88200000000000001},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.60799999999999998},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.79400000000000004,"15min":0.90700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.95499999999999996},"last":0.504}]},{"osd":2,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57199999999999995,"15min":0.61699999999999999},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81100000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.995},"last":0.436},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60899999999999999,"15min":0.63200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.86199999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76000000000000001,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.32500000000000001,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.82899999999999996,"5min":0.878,"15min":1.0609999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.35799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":1.115},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.68500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66476,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305364,"statfs":{"total":96636764160,"available":96568692736,"internally_reserved":0,"allocated":3432448,"data_stored":1851876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58599999999999997,"15min":0.623},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28999999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.64300000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.88200000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55500000000000005,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67900000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74299999999999999,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53700000000000003,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.623,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81899999999999995,"5min":0.83599999999999997,"15min":0.89400000000000002},"last":0.70299999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79300000000000004,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.59899999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.71799999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.496,"15min":0.51600000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":0.79500000000000004,"15min":0.86399999999999999},"last":0.74099999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83599999999999997,"5min":0.878,"15min":0.878},"last":0.61299999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.874},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.623,"15min":0.63700000000000001},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":0.89700000000000002},"last":0.83099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:22:01.224 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T11:22:01.224 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T11:22:01.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.384+0000 7fd960f58700 1 Processor -- start 2023-10-16T11:22:01.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.384+0000 7fd960f58700 1 -- start start 2023-10-16T11:22:01.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd960f58700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd95c0b3b60 0x7fd95c0724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd960f58700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c071d70 con 0x7fd95c0af5f0 2023-10-16T11:22:01.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd960f58700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0af9d0 con 0x7fd95c072a20 2023-10-16T11:22:01.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd960f58700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0b3970 con 0x7fd95c0b3b60 2023-10-16T11:22:01.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd95a59c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd95c0af5f0 0x7fd95c161e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44006/0 (socket says 172.21.15.158:44006) 2023-10-16T11:22:01.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd95a59c700 1 -- 172.21.15.158:0/3666173090 learned_addr learned my addr 172.21.15.158:0/3666173090 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:01.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd959d9b700 1 --2- 172.21.15.158:0/3666173090 >> v2:172.21.15.172:3300/0 conn(0x7fd95c0b3b60 0x7fd95c0724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.385+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457785067 0 0) 0x7fd95c071d70 con 0x7fd95c0af5f0 2023-10-16T11:22:01.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9440035b0 con 0x7fd95c0af5f0 2023-10-16T11:22:01.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787405735 0 0) 0x7fd9440035b0 con 0x7fd95c0af5f0 2023-10-16T11:22:01.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 >> v2:172.21.15.172:3300/0 conn(0x7fd95c0b3b60 msgr2=0x7fd95c0724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:01.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 --2- 172.21.15.158:0/3666173090 >> v2:172.21.15.172:3300/0 conn(0x7fd95c0b3b60 0x7fd95c0724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 >> v1:172.21.15.172:6789/0 conn(0x7fd95c072a20 legacy=0x7fd95c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0b4060 con 0x7fd95c0af5f0 2023-10-16T11:22:01.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7fd950001670 con 0x7fd95c0af5f0 2023-10-16T11:22:01.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 >> v1:172.21.15.158:6789/0 conn(0x7fd95c0af5f0 legacy=0x7fd95c161e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd944003a00 0x7fd944003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd944004850 0x7fd944004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944005210 0x7fd94400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0b3970 con 0x7fd944003a00 2023-10-16T11:22:01.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94400c0b0 con 0x7fd944004850 2023-10-16T11:22:01.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94400c270 con 0x7fd944005210 2023-10-16T11:22:01.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95a59c700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd944003a00 0x7fd944003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd959d9b700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd944004850 0x7fd944004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95ad9d700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944005210 0x7fd94400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95a59c700 1 -- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944005210 msgr2=0x7fd94400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:01.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95a59c700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944005210 0x7fd94400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95a59c700 1 -- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd944004850 msgr2=0x7fd944004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:01.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95a59c700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd944004850 0x7fd944004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95a59c700 1 -- 172.21.15.158:0/3666173090 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9440035b0 con 0x7fd944003a00 2023-10-16T11:22:01.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95ad9d700 1 -- 172.21.15.158:0/3666173090 reap_dead start 2023-10-16T11:22:01.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95a59c700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd944003a00 0x7fd944003e20 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7fd95000c240 tx=0x7fd944005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3631ee573ade74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.386+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd950007c10 con 0x7fd944003a00 2023-10-16T11:22:01.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95000ce00 con 0x7fd944003a00 2023-10-16T11:22:01.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd95959a700 1 -- 172.21.15.158:0/3666173090 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd950005ba0 con 0x7fd944003a00 2023-10-16T11:22:01.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd960f58700 1 -- 172.21.15.158:0/3666173090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd944003a00 msgr2=0x7fd944003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd960f58700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd944003a00 0x7fd944003e20 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7fd95000c240 tx=0x7fd944005880 comp rx=0 tx=0).stop 2023-10-16T11:22:01.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd960f58700 1 -- 172.21.15.158:0/3666173090 shutdown_connections 2023-10-16T11:22:01.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd960f58700 1 --2- 172.21.15.158:0/3666173090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd944003a00 0x7fd944003e20 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd960f58700 1 -- 172.21.15.158:0/3666173090 >> 172.21.15.158:0/3666173090 conn(0x7fd95c0ccd30 msgr2=0x7fd95c0cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:01.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd960f58700 1 -- 172.21.15.158:0/3666173090 shutdown_connections 2023-10-16T11:22:01.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.387+0000 7fd960f58700 1 -- 172.21.15.158:0/3666173090 wait complete. 2023-10-16T11:22:01.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 Processor -- start 2023-10-16T11:22:01.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 -- start start 2023-10-16T11:22:01.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944003a00 0x7fd95c0b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd95c0c7380 0x7fd95c0c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd95c0c9120 0x7fd95c0ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0c95b0 con 0x7fd95c0c7380 2023-10-16T11:22:01.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0c8750 con 0x7fd95c0c9120 2023-10-16T11:22:01.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd960f58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0c7b40 con 0x7fd944003a00 2023-10-16T11:22:01.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd959d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd95c0c7380 0x7fd95c0c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd959d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd95c0c7380 0x7fd95c0c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43840/0 (socket says 172.21.15.158:43840) 2023-10-16T11:22:01.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.389+0000 7fd959d9b700 1 -- 172.21.15.158:0/4086805461 learned_addr learned my addr 172.21.15.158:0/4086805461 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:01.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd95ad9d700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd95c0c9120 0x7fd95c0ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd959d9b700 1 -- 172.21.15.158:0/4086805461 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944003a00 msgr2=0x7fd95c0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd959d9b700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944003a00 0x7fd95c0b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd959d9b700 1 -- 172.21.15.158:0/4086805461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd95c0c9120 msgr2=0x7fd95c0ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd959d9b700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd95c0c9120 0x7fd95c0ae130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd959d9b700 1 -- 172.21.15.158:0/4086805461 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0c7e20 con 0x7fd95c0c7380 2023-10-16T11:22:01.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd959d9b700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd95c0c7380 0x7fd95c0c8be0 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fd94c011ad0 tx=0x7fd94c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd94affd700 1 -- 172.21.15.158:0/4086805461 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c00fa60 con 0x7fd95c0c7380 2023-10-16T11:22:01.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd94affd700 1 -- 172.21.15.158:0/4086805461 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c00fbc0 con 0x7fd95c0c7380 2023-10-16T11:22:01.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c0c8100 con 0x7fd95c0c7380 2023-10-16T11:22:01.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd94affd700 1 -- 172.21.15.158:0/4086805461 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c014440 con 0x7fd95c0c7380 2023-10-16T11:22:01.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.390+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c0c7800 con 0x7fd95c0c7380 2023-10-16T11:22:01.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.391+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c0681b0 con 0x7fd95c0c7380 2023-10-16T11:22:01.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.391+0000 7fd94affd700 1 -- 172.21.15.158:0/4086805461 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd94c0145a0 con 0x7fd95c0c7380 2023-10-16T11:22:01.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.392+0000 7fd94affd700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd93c06b020 0x7fd93c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.392+0000 7fd94affd700 1 -- 172.21.15.158:0/4086805461 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fd94c08a180 con 0x7fd95c0c7380 2023-10-16T11:22:01.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.392+0000 7fd95a59c700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd93c06b020 0x7fd93c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.392+0000 7fd95a59c700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd93c06b020 0x7fd93c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd950000c00 tx=0x7fd95000a000 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.396+0000 7fd94affd700 1 -- 172.21.15.158:0/4086805461 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd94c054f40 con 0x7fd95c0c7380 2023-10-16T11:22:01.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.668+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd95c0c8890 con 0x7fd95c0c7380 2023-10-16T11:22:01.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.670+0000 7fd94affd700 1 -- 172.21.15.158:0/4086805461 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v57) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd95c0c8890 con 0x7fd95c0c7380 2023-10-16T11:22:01.672 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-16T11:22:01.672 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCCBS1lNo8eMhAAAUvvnumTISWwpFhlm9ykWA== 2023-10-16T11:22:01.672 INFO:teuthology.orchestra.run.smithi158.stdout: caps mds = "allow" 2023-10-16T11:22:01.673 INFO:teuthology.orchestra.run.smithi158.stdout: caps mon = "allow r" 2023-10-16T11:22:01.673 INFO:teuthology.orchestra.run.smithi158.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T11:22:01.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd93c06b020 msgr2=0x7fd93c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd93c06b020 0x7fd93c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd950000c00 tx=0x7fd95000a000 comp rx=0 tx=0).stop 2023-10-16T11:22:01.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd95c0c7380 msgr2=0x7fd95c0c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd95c0c7380 0x7fd95c0c8be0 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fd94c011ad0 tx=0x7fd94c011e90 comp rx=0 tx=0).stop 2023-10-16T11:22:01.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 shutdown_connections 2023-10-16T11:22:01.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd95c0c9120 0x7fd95c0ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd944003a00 0x7fd95c0b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fd93c06b020 0x7fd93c06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 --2- 172.21.15.158:0/4086805461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd95c0c7380 0x7fd95c0c8be0 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 >> 172.21.15.158:0/4086805461 conn(0x7fd95c0ccd30 msgr2=0x7fd95c0ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:01.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 shutdown_connections 2023-10-16T11:22:01.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.673+0000 7fd960f58700 1 -- 172.21.15.158:0/4086805461 wait complete. 2023-10-16T11:22:01.678 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.0 2023-10-16T11:22:01.687 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:22:01.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.808+0000 7ff154600700 1 Processor -- start 2023-10-16T11:22:01.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.808+0000 7ff154600700 1 -- start start 2023-10-16T11:22:01.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff154600700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff14c06ab40 0x7ff14c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff154600700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06e5d0 0x7ff14c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff154600700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c071e50 con 0x7ff14c06af20 2023-10-16T11:22:01.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff154600700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c161270 con 0x7ff14c06ab40 2023-10-16T11:22:01.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff154600700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c161510 con 0x7ff14c06e5d0 2023-10-16T11:22:01.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff14c06ab40 0x7ff14c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff14c06ab40 0x7ff14c06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43846/0 (socket says 172.21.15.158:43846) 2023-10-16T11:22:01.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 -- 172.21.15.158:0/1540966679 learned_addr learned my addr 172.21.15.158:0/1540966679 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:01.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff151b9b700 1 --2- 172.21.15.158:0/1540966679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06e5d0 0x7ff14c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 -- 172.21.15.158:0/1540966679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06e5d0 msgr2=0x7ff14c071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:01.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 --2- 172.21.15.158:0/1540966679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06e5d0 0x7ff14c071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 -- 172.21.15.158:0/1540966679 >> v1:172.21.15.172:6789/0 conn(0x7ff14c06af20 legacy=0x7ff14c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 -- 172.21.15.158:0/1540966679 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c1617f0 con 0x7ff14c06ab40 2023-10-16T11:22:01.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff151b9b700 1 --2- 172.21.15.158:0/1540966679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06e5d0 0x7ff14c071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:01.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15239c700 1 --2- 172.21.15.158:0/1540966679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06ab40 0x7ff14c06d820 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7ff1480098f0 tx=0x7ff148009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eec6780a94eb8ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.809+0000 7ff15139a700 1 -- 172.21.15.158:0/1540966679 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800b8a0 con 0x7ff14c06ab40 2023-10-16T11:22:01.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff15139a700 1 -- 172.21.15.158:0/1540966679 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14800ba00 con 0x7ff14c06ab40 2023-10-16T11:22:01.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff15139a700 1 -- 172.21.15.158:0/1540966679 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800bd90 con 0x7ff14c06ab40 2023-10-16T11:22:01.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 -- 172.21.15.158:0/1540966679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06ab40 msgr2=0x7ff14c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 --2- 172.21.15.158:0/1540966679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06ab40 0x7ff14c06d820 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7ff1480098f0 tx=0x7ff148009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:01.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 -- 172.21.15.158:0/1540966679 shutdown_connections 2023-10-16T11:22:01.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 --2- 172.21.15.158:0/1540966679 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06ab40 0x7ff14c06d820 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 --2- 172.21.15.158:0/1540966679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06e5d0 0x7ff14c071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 -- 172.21.15.158:0/1540966679 >> 172.21.15.158:0/1540966679 conn(0x7ff14c0d02b0 msgr2=0x7ff14c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:01.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 -- 172.21.15.158:0/1540966679 shutdown_connections 2023-10-16T11:22:01.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.810+0000 7ff154600700 1 -- 172.21.15.158:0/1540966679 wait complete. 2023-10-16T11:22:01.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.811+0000 7ff154600700 1 Processor -- start 2023-10-16T11:22:01.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.811+0000 7ff154600700 1 -- start start 2023-10-16T11:22:01.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff154600700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06ab40 0x7ff14c0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff154600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff14c06af20 0x7ff14c1f5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff154600700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06e5d0 0x7ff14c1f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff154600700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c92d0 con 0x7ff14c06e5d0 2023-10-16T11:22:01.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff154600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c148330 con 0x7ff14c06af20 2023-10-16T11:22:01.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff154600700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c9520 con 0x7ff14c06ab40 2023-10-16T11:22:01.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff152b9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06e5d0 0x7ff14c1f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff152b9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06e5d0 0x7ff14c1f5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43856/0 (socket says 172.21.15.158:43856) 2023-10-16T11:22:01.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff152b9d700 1 -- 172.21.15.158:0/3529444159 learned_addr learned my addr 172.21.15.158:0/3529444159 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:01.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff151b9b700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff14c06af20 0x7ff14c1f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.812+0000 7ff15239c700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06ab40 0x7ff14c0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff152b9d700 1 -- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06ab40 msgr2=0x7ff14c0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff152b9d700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06ab40 0x7ff14c0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff152b9d700 1 -- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff14c06af20 msgr2=0x7ff14c1f5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:01.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff152b9d700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff14c06af20 0x7ff14c1f5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:01.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff152b9d700 1 -- 172.21.15.158:0/3529444159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148009550 con 0x7ff14c06e5d0 2023-10-16T11:22:01.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff151b9b700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff14c06af20 0x7ff14c1f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:01.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff152b9d700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06e5d0 0x7ff14c1f5c80 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7ff14400cc50 tx=0x7ff14400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff142ffd700 1 -- 172.21.15.158:0/3529444159 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400ad80 con 0x7ff14c06e5d0 2023-10-16T11:22:01.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff142ffd700 1 -- 172.21.15.158:0/3529444159 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14400ce50 con 0x7ff14c06e5d0 2023-10-16T11:22:01.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c0c9800 con 0x7ff14c06e5d0 2023-10-16T11:22:01.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff142ffd700 1 -- 172.21.15.158:0/3529444159 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144003c40 con 0x7ff14c06e5d0 2023-10-16T11:22:01.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.813+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c1f61c0 con 0x7ff14c06e5d0 2023-10-16T11:22:01.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.814+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c0681b0 con 0x7ff14c06e5d0 2023-10-16T11:22:01.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.814+0000 7ff142ffd700 1 -- 172.21.15.158:0/3529444159 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff1440050d0 con 0x7ff14c06e5d0 2023-10-16T11:22:01.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.815+0000 7ff142ffd700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff13006b020 0x7ff13006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:01.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.815+0000 7ff142ffd700 1 -- 172.21.15.158:0/3529444159 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff144089b90 con 0x7ff14c06e5d0 2023-10-16T11:22:01.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.815+0000 7ff15239c700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff13006b020 0x7ff13006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:01.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.815+0000 7ff15239c700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff13006b020 0x7ff13006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff148009c40 tx=0x7ff148008030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:01.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:01.819+0000 7ff142ffd700 1 -- 172.21.15.158:0/3529444159 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff144054810 con 0x7ff14c06e5d0 2023-10-16T11:22:02.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.056+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff14c0b6230 con 0x7ff14c06e5d0 2023-10-16T11:22:02.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.056+0000 7ff142ffd700 1 -- 172.21.15.158:0/3529444159 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4935 (secure 0 0 0) 0x7ff144057e30 con 0x7ff14c06e5d0 2023-10-16T11:22:02.058 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:02.058 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":154,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1253298118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1253298118},{"type":"v1","addr":"172.21.15.158:6837","nonce":1253298118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":13731,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2914884530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2914884530},{"type":"v1","addr":"172.21.15.158:6836","nonce":2914884530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/536559492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":536559492},{"type":"v1","addr":"172.21.15.172:6834","nonce":536559492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:56.094141+0000","modified":"2023-10-16T11:21:59.105113+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":40202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_40202":{"gid":40202,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/284711673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":284711673},{"type":"v1","addr":"172.21.15.172:6835","nonce":284711673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T11:22:02.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.058+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff13006b020 msgr2=0x7ff13006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.058+0000 7ff154600700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff13006b020 0x7ff13006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff148009c40 tx=0x7ff148008030 comp rx=0 tx=0).stop 2023-10-16T11:22:02.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.058+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06e5d0 msgr2=0x7ff14c1f5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.058+0000 7ff154600700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06e5d0 0x7ff14c1f5c80 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7ff14400cc50 tx=0x7ff14400a340 comp rx=0 tx=0).stop 2023-10-16T11:22:02.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 shutdown_connections 2023-10-16T11:22:02.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff13006b020 0x7ff13006d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c06e5d0 0x7ff14c1f5c80 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff14c06ab40 0x7ff14c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 --2- 172.21.15.158:0/3529444159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff14c06af20 0x7ff14c1f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 >> 172.21.15.158:0/3529444159 conn(0x7ff14c0d02b0 msgr2=0x7ff14c072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 shutdown_connections 2023-10-16T11:22:02.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.059+0000 7ff154600700 1 -- 172.21.15.158:0/3529444159 wait complete. 2023-10-16T11:22:02.063 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 154 2023-10-16T11:22:02.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T11:21:56.094141+0000', 'modified': '2023-10-16T11:21:59.105113+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 40202}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_40202': {'gid': 40202, 'name': 'b', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.172:6835/284711673', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6833', 'nonce': 284711673}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 284711673}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [38], 'metadata_pool': 37, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T11:22:02.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T11:22:02.476 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T11:22:02.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.599+0000 7ff9c87bf700 1 Processor -- start 2023-10-16T11:22:02.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.599+0000 7ff9c87bf700 1 -- start start 2023-10-16T11:22:02.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.600+0000 7ff9c87bf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff9c015d990 0x7ff9c00b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.600+0000 7ff9c87bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c00cf650 0x7ff9c00cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.600+0000 7ff9c87bf700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c00af600 con 0x7ff9c006e480 2023-10-16T11:22:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.600+0000 7ff9c87bf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c00b2c90 con 0x7ff9c015d990 2023-10-16T11:22:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.600+0000 7ff9c87bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c006e9b0 con 0x7ff9c00cf650 2023-10-16T11:22:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.600+0000 7ff9c5d5a700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff9c006e480 0x7ff9c00b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44038/0 (socket says 172.21.15.158:44038) 2023-10-16T11:22:02.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5d5a700 1 -- 172.21.15.158:0/2151681384 learned_addr learned my addr 172.21.15.158:0/2151681384 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2151681384 >> v2:172.21.15.172:3300/0 conn(0x7ff9c015d990 0x7ff9c00b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c00cf650 0x7ff9c00cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018030690 0 0) 0x7ff9c00af600 con 0x7ff9c006e480 2023-10-16T11:22:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9ac0035b0 con 0x7ff9c006e480 2023-10-16T11:22:02.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332701135 0 0) 0x7ff9ac0035b0 con 0x7ff9c006e480 2023-10-16T11:22:02.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c00cf650 msgr2=0x7ff9c00cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c00cf650 0x7ff9c00cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 >> v2:172.21.15.172:3300/0 conn(0x7ff9c015d990 msgr2=0x7ff9c00b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 --2- 172.21.15.158:0/2151681384 >> v2:172.21.15.172:3300/0 conn(0x7ff9c015d990 0x7ff9c00b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c00ce890 con 0x7ff9c006e480 2023-10-16T11:22:02.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff9bc001670 con 0x7ff9c006e480 2023-10-16T11:22:02.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 >> v1:172.21.15.158:6789/0 conn(0x7ff9c006e480 legacy=0x7ff9c00b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9ac003a00 0x7ff9ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac004850 0x7ff9ac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac005210 0x7ff9ac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c00b2c90 con 0x7ff9ac003a00 2023-10-16T11:22:02.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c006e9b0 con 0x7ff9ac004850 2023-10-16T11:22:02.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac00bf50 con 0x7ff9ac005210 2023-10-16T11:22:02.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2151681384 >> v2:172.21.15.172:3300/0 conn(0x7ff9c015d990 0x7ff9c00b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:02.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c00cf650 0x7ff9c00cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:02.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9ac003a00 0x7ff9ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac004850 0x7ff9ac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.601+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac005210 0x7ff9ac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c655b700 1 -- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac005210 msgr2=0x7ff9ac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac005210 0x7ff9ac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c655b700 1 -- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac004850 msgr2=0x7ff9ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac004850 0x7ff9ac004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c655b700 1 -- 172.21.15.158:0/2151681384 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0035b0 con 0x7ff9ac003a00 2023-10-16T11:22:02.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9ac003a00 0x7ff9ac003e20 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009d10 tx=0x7ff9ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548f3f8c3eeea173 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0009e60 con 0x7ff9ac003a00 2023-10-16T11:22:02.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b000e9a0 con 0x7ff9ac003a00 2023-10-16T11:22:02.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c5559700 1 -- 172.21.15.158:0/2151681384 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b000ed10 con 0x7ff9ac003a00 2023-10-16T11:22:02.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2151681384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9ac003a00 msgr2=0x7ff9ac003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c87bf700 1 --2- 172.21.15.158:0/2151681384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9ac003a00 0x7ff9ac003e20 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009d10 tx=0x7ff9ac005880 comp rx=0 tx=0).stop 2023-10-16T11:22:02.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c6d5c700 1 -- 172.21.15.158:0/2151681384 reap_dead start 2023-10-16T11:22:02.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2151681384 shutdown_connections 2023-10-16T11:22:02.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.602+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2151681384 >> 172.21.15.158:0/2151681384 conn(0x7ff9c006b9e0 msgr2=0x7ff9c006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.603+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2151681384 shutdown_connections 2023-10-16T11:22:02.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.603+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2151681384 wait complete. 2023-10-16T11:22:02.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.604+0000 7ff9c87bf700 1 Processor -- start 2023-10-16T11:22:02.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.604+0000 7ff9c87bf700 1 -- start start 2023-10-16T11:22:02.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c87bf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac003a00 0x7ff9c00c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c87bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c015e9d0 0x7ff9c00caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c87bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9c0152ba0 0x7ff9c00b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c87bf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c00c5af0 con 0x7ff9c0152ba0 2023-10-16T11:22:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c87bf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c00caf40 con 0x7ff9ac003a00 2023-10-16T11:22:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c87bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c015ee50 con 0x7ff9c015e9d0 2023-10-16T11:22:02.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9c0152ba0 0x7ff9c00b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9c0152ba0 0x7ff9c00b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43876/0 (socket says 172.21.15.158:43876) 2023-10-16T11:22:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 -- 172.21.15.158:0/2472954230 learned_addr learned my addr 172.21.15.158:0/2472954230 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac003a00 0x7ff9c00c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c015e9d0 0x7ff9c00caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 -- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c015e9d0 msgr2=0x7ff9c00caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c015e9d0 0x7ff9c00caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 -- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac003a00 msgr2=0x7ff9c00c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac003a00 0x7ff9c00c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c6d5c700 1 -- 172.21.15.158:0/2472954230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c015f120 con 0x7ff9c0152ba0 2023-10-16T11:22:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac003a00 0x7ff9c00c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:02.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.605+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c015e9d0 0x7ff9c00caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:02.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.606+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9c0152ba0 0x7ff9c00b0890 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7ff9c00cfa30 tx=0x7ff9b800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.606+0000 7ff9b6ffd700 1 -- 172.21.15.158:0/2472954230 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b800e040 con 0x7ff9c0152ba0 2023-10-16T11:22:02.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.606+0000 7ff9b6ffd700 1 -- 172.21.15.158:0/2472954230 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b800f040 con 0x7ff9c0152ba0 2023-10-16T11:22:02.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.606+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c015f400 con 0x7ff9c0152ba0 2023-10-16T11:22:02.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.606+0000 7ff9b6ffd700 1 -- 172.21.15.158:0/2472954230 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8003c60 con 0x7ff9c0152ba0 2023-10-16T11:22:02.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.606+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c00c5780 con 0x7ff9c0152ba0 2023-10-16T11:22:02.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.607+0000 7ff9b6ffd700 1 -- 172.21.15.158:0/2472954230 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9b8004030 con 0x7ff9c0152ba0 2023-10-16T11:22:02.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.608+0000 7ff9b6ffd700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff9a406b020 0x7ff9a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.608+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff9a406b020 0x7ff9a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.608+0000 7ff9b6ffd700 1 -- 172.21.15.158:0/2472954230 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff9b8089aa0 con 0x7ff9c0152ba0 2023-10-16T11:22:02.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.608+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff9a406b020 0x7ff9a406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9b00090b0 tx=0x7ff9b0008030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.610+0000 7ff9d49a9b80 1 Processor -- start 2023-10-16T11:22:02.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.610+0000 7ff9d49a9b80 1 -- start start 2023-10-16T11:22:02.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5e4bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e1f010 0x5635a5e1f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5d95c20 con 0x5635a5e44220 2023-10-16T11:22:02.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5d7ac10 con 0x5635a5e49630 2023-10-16T11:22:02.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635a5d78530 con 0x5635a5e1f010 2023-10-16T11:22:02.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43880/0 (socket says 172.21.15.158:43880) 2023-10-16T11:22:02.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 -- 172.21.15.158:0/2235864513 learned_addr learned my addr 172.21.15.158:0/2235864513 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:02.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5e4bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e1f010 0x5635a5e1f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 -- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e1f010 msgr2=0x5635a5e1f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e1f010 0x5635a5e1f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 -- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 msgr2=0x5635a5e4bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5e4bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 -- 172.21.15.158:0/2235864513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635a5d313a0 con 0x5635a5e44220 2023-10-16T11:22:02.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9c655b700 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ba70 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000e790 tx=0x7ff9b0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e3dd790633a673 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9a3fff700 1 -- 172.21.15.158:0/2235864513 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b002e070 con 0x5635a5e44220 2023-10-16T11:22:02.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.611+0000 7ff9a3fff700 1 -- 172.21.15.158:0/2235864513 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b0021040 con 0x5635a5e44220 2023-10-16T11:22:02.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9a3fff700 1 -- 172.21.15.158:0/2235864513 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0004030 con 0x5635a5e44220 2023-10-16T11:22:02.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/2235864513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 msgr2=0x5635a5e4ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ba70 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000e790 tx=0x7ff9b0013000 comp rx=0 tx=0).stop 2023-10-16T11:22:02.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/2235864513 shutdown_connections 2023-10-16T11:22:02.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e1f010 0x5635a5e1f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ba70 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/2235864513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5e4bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/2235864513 >> 172.21.15.158:0/2235864513 conn(0x5635a5d4ed80 msgr2=0x5635a5e69ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/2235864513 shutdown_connections 2023-10-16T11:22:02.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.612+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/2235864513 wait complete. 2023-10-16T11:22:02.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.613+0000 7ff9d49a9b80 1 -- start start 2023-10-16T11:22:02.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.613+0000 7ff9d49a9b80 1 Processor -- start 2023-10-16T11:22:02.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e4df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e44220 0x5635a5e45f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 0x5635a5e21500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5d73e20 con 0x5635a5e1f010 2023-10-16T11:22:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5d106c0 con 0x5635a5e44220 2023-10-16T11:22:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635a5d10360 con 0x5635a5e49630 2023-10-16T11:22:02.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9d49a9b80 10 client.0 _ll_register_callbacks cb 0x5635a5b6d7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:22:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e4df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e4df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43886/0 (socket says 172.21.15.158:43886) 2023-10-16T11:22:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 -- 172.21.15.158:0/3045448597 learned_addr learned my addr 172.21.15.158:0/3045448597 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 -- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 msgr2=0x5635a5e21500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 0x5635a5e21500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 -- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e44220 msgr2=0x5635a5e45f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e44220 0x5635a5e45f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 -- 172.21.15.158:0/3045448597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac0035b0 con 0x5635a5e1f010 2023-10-16T11:22:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9c655b700 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e4df30 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0013680 tx=0x7ff9b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98676fb27c1108f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9a17fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:22:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.614+0000 7ff9a17fa700 1 -- 172.21.15.158:0/3045448597 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b002e040 con 0x5635a5e1f010 2023-10-16T11:22:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9a17fa700 1 -- 172.21.15.158:0/3045448597 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff9b0005e30 con 0x5635a5e1f010 2023-10-16T11:22:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635a5e145b0 con 0x5635a5e1f010 2023-10-16T11:22:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9a17fa700 1 -- 172.21.15.158:0/3045448597 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9b00139c0 con 0x5635a5e1f010 2023-10-16T11:22:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 10 client.13866 fetch_fsmap learned FSMap version 154 2023-10-16T11:22:02.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x5635a5b9ed90 con 0x5635a5e1f010 2023-10-16T11:22:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9a17fa700 1 -- 172.21.15.158:0/3045448597 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5635a5e145b0 con 0x5635a5e1f010 2023-10-16T11:22:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9a17fa700 1 -- 172.21.15.158:0/3045448597 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 154) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff9b0013b20 con 0x5635a5e1f010 2023-10-16T11:22:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 10 client.13866 fetch_fsmap finished waiting for FSMap version 154 2023-10-16T11:22:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 10 client.13866 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:22:02.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 20 client.13866 populate_metadata read hostname 'smithi158' 2023-10-16T11:22:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5e4ca00 0x5635a5e4cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 4 client.13866 mds_command: new command op to 40202 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T11:22:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 --> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635a5d6ce00 con 0x5635a5e4ca00 2023-10-16T11:22:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.615+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5e4ca00 0x5635a5e4cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.616+0000 7ff9a17fa700 10 client.13866 ms_handle_connect on v2:172.21.15.172:6833/284711673 2023-10-16T11:22:02.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.616+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5e4ca00 0x5635a5e4cde0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c7e3b8734f49f15 server_cookie=836b4d1de57cb55f in_seq=0 out_seq=0 2023-10-16T11:22:02.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.616+0000 7ff9a17fa700 1 -- 172.21.15.158:0/3045448597 <== mds.0 v2:172.21.15.172:6833/284711673 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5635a5d6ce00 con 0x5635a5e4ca00 2023-10-16T11:22:02.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.616+0000 7ff9a17fa700 10 client.13866 handle_command_reply: tid=0 2023-10-16T11:22:02.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 client.13866 shutdown 2023-10-16T11:22:02.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 msgr2=0x5635a5e4df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e4df30 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0013680 tx=0x7ff9b0004750 comp rx=0 tx=0).stop 2023-10-16T11:22:02.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 shutdown_connections 2023-10-16T11:22:02.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5e4ca00 0x5635a5e4cde0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 0x5635a5e21500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e4df30 unknown :-1 s=CLOSED pgs=3310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/3045448597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e44220 0x5635a5e45f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9a17fa700 0 client.13866 ms_handle_reset on v2:172.21.15.172:6833/284711673 2023-10-16T11:22:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 >> 172.21.15.158:0/3045448597 conn(0x5635a5d4ed80 msgr2=0x5635a5e4be00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 shutdown_connections 2023-10-16T11:22:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/3045448597 wait complete. 2023-10-16T11:22:02.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.617+0000 7ff9d49a9b80 20 client.13866 trim_cache size 0 max 16384 2023-10-16T11:22:02.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.634+0000 7ff9d49a9b80 1 Processor -- start 2023-10-16T11:22:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9d49a9b80 1 -- start start 2023-10-16T11:22:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e1f010 0x5635a5e4e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 0x5635a5e1ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5bf48d0 con 0x5635a5e44220 2023-10-16T11:22:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5dfe4c0 con 0x5635a5e1f010 2023-10-16T11:22:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635a5e47120 con 0x5635a5e49630 2023-10-16T11:22:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9c5d5a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9c5d5a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43894/0 (socket says 172.21.15.158:43894) 2023-10-16T11:22:02.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.635+0000 7ff9c5d5a700 1 -- 172.21.15.158:0/209982970 learned_addr learned my addr 172.21.15.158:0/209982970 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:02.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c655b700 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e1f010 0x5635a5e4e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c5d5a700 1 -- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 msgr2=0x5635a5e1ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 0x5635a5e1ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 0x5635a5e1ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c5d5a700 1 -- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e1f010 msgr2=0x5635a5e4e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e1f010 0x5635a5e4e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c5d5a700 1 -- 172.21.15.158:0/209982970 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635a5b87c10 con 0x5635a5e44220 2023-10-16T11:22:02.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c655b700 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e1f010 0x5635a5e4e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:02.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ecc0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000ed0 tx=0x7ff9bc007980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf53584ef45dbf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9b4ff9700 1 -- 172.21.15.158:0/209982970 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc001650 con 0x5635a5e44220 2023-10-16T11:22:02.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9b4ff9700 1 -- 172.21.15.158:0/209982970 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc0070c0 con 0x5635a5e44220 2023-10-16T11:22:02.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9b4ff9700 1 -- 172.21.15.158:0/209982970 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc007430 con 0x5635a5e44220 2023-10-16T11:22:02.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/209982970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 msgr2=0x5635a5e4ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ecc0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000ed0 tx=0x7ff9bc007980 comp rx=0 tx=0).stop 2023-10-16T11:22:02.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/209982970 shutdown_connections 2023-10-16T11:22:02.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e49630 0x5635a5e1ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e1f010 0x5635a5e4e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/209982970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e44220 0x5635a5e4ecc0 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.636+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/209982970 >> 172.21.15.158:0/209982970 conn(0x5635a5d4ed80 msgr2=0x5635a5e88630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.637+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/209982970 shutdown_connections 2023-10-16T11:22:02.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.637+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/209982970 wait complete. 2023-10-16T11:22:02.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.637+0000 7ff9d49a9b80 1 -- start start 2023-10-16T11:22:02.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.637+0000 7ff9d49a9b80 1 Processor -- start 2023-10-16T11:22:02.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e66e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e44220 0x5635a5e20440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5ecd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5e664d0 con 0x5635a5e1f010 2023-10-16T11:22:02.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635a5e62cd0 con 0x5635a5e49630 2023-10-16T11:22:02.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635a5e62fb0 con 0x5635a5e44220 2023-10-16T11:22:02.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 10 client.0 _ll_register_callbacks cb 0x7ff9bc000fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T11:22:02.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e66e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e66e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43900/0 (socket says 172.21.15.158:43900) 2023-10-16T11:22:02.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 -- 172.21.15.158:0/314105512 learned_addr learned my addr 172.21.15.158:0/314105512 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:02.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e44220 0x5635a5e20440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c6d5c700 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5ecd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 -- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e44220 msgr2=0x5635a5e20440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e44220 0x5635a5e20440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 -- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 msgr2=0x5635a5ecd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5ecd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 -- 172.21.15.158:0/314105512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c00ce890 con 0x5635a5e1f010 2023-10-16T11:22:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9c655b700 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e66e70 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000e7c0 tx=0x7ff9b0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d270902071a25d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9a17fa700 10 client.0 ms_handle_connect on v2:172.21.15.158:3300/0 2023-10-16T11:22:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9a17fa700 1 -- 172.21.15.158:0/314105512 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b002e070 con 0x5635a5e1f010 2023-10-16T11:22:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9a17fa700 1 -- 172.21.15.158:0/314105512 <== mon.0 v2:172.21.15.158:3300/0 2 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff9b0004040 con 0x5635a5e1f010 2023-10-16T11:22:02.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9b0021040 con 0x5635a5e1f010 2023-10-16T11:22:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9a17fa700 1 -- 172.21.15.158:0/314105512 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9b0004200 con 0x5635a5e1f010 2023-10-16T11:22:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 10 client.13872 fetch_fsmap learned FSMap version 154 2023-10-16T11:22:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x5635a5bc5200 con 0x5635a5e1f010 2023-10-16T11:22:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.639+0000 7ff9a17fa700 1 -- 172.21.15.158:0/314105512 <== mon.0 v2:172.21.15.158:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b0021040 con 0x5635a5e1f010 2023-10-16T11:22:02.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9a17fa700 1 -- 172.21.15.158:0/314105512 <== mon.0 v2:172.21.15.158:3300/0 5 ==== fsmap(e 154) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff9b0004360 con 0x5635a5e1f010 2023-10-16T11:22:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9d49a9b80 10 client.13872 fetch_fsmap finished waiting for FSMap version 154 2023-10-16T11:22:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9d49a9b80 10 client.13872 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T11:22:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9d49a9b80 20 client.13872 populate_metadata read hostname 'smithi158' 2023-10-16T11:22:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5ed0d00 0x5635a5ed10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9d49a9b80 4 client.13872 mds_command: new command op to 40202 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T11:22:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 --> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5635a5d6cb20 con 0x5635a5ed0d00 2023-10-16T11:22:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5ed0d00 0x5635a5ed10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9a17fa700 10 client.13872 ms_handle_connect on v2:172.21.15.172:6833/284711673 2023-10-16T11:22:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9c5d5a700 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5ed0d00 0x5635a5ed10e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fb6fab78d3165d8 server_cookie=a3aa9bac9cfcfa6f in_seq=0 out_seq=0 2023-10-16T11:22:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9a17fa700 1 -- 172.21.15.158:0/314105512 <== mds.0 v2:172.21.15.172:6833/284711673 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5635a5d6cb20 con 0x5635a5ed0d00 2023-10-16T11:22:02.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9a17fa700 10 client.13872 handle_command_reply: tid=0 2023-10-16T11:22:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.640+0000 7ff9d49a9b80 1 client.13872 shutdown 2023-10-16T11:22:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 msgr2=0x5635a5e66e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e66e70 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000e7c0 tx=0x7ff9b0014000 comp rx=0 tx=0).stop 2023-10-16T11:22:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 shutdown_connections 2023-10-16T11:22:02.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:6833/284711673,v1:172.21.15.172:6835/284711673] conn(0x5635a5ed0d00 0x5635a5ed10e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5635a5e49630 0x5635a5ecd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9a17fa700 0 client.13872 ms_handle_reset on v2:172.21.15.172:6833/284711673 2023-10-16T11:22:02.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5635a5e1f010 0x5635a5e66e70 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 --2- 172.21.15.158:0/314105512 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5635a5e44220 0x5635a5e20440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 >> 172.21.15.158:0/314105512 conn(0x5635a5d4ed80 msgr2=0x5635a5e488b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 shutdown_connections 2023-10-16T11:22:02.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 1 -- 172.21.15.158:0/314105512 wait complete. 2023-10-16T11:22:02.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9d49a9b80 20 client.13872 trim_cache size 0 max 16384 2023-10-16T11:22:02.667 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:02.668 INFO:teuthology.orchestra.run.smithi158.stdout: "cluster_fsid": "7cf363df-9988-44b1-be05-df2a6bf24f2d", 2023-10-16T11:22:02.668 INFO:teuthology.orchestra.run.smithi158.stdout: "whoami": 0, 2023-10-16T11:22:02.668 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 40202, 2023-10-16T11:22:02.668 INFO:teuthology.orchestra.run.smithi158.stdout: "want_state": "up:active", 2023-10-16T11:22:02.668 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "up:active", 2023-10-16T11:22:02.669 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs", 2023-10-16T11:22:02.669 INFO:teuthology.orchestra.run.smithi158.stdout: "rank_uptime": 5.9420900909999999, 2023-10-16T11:22:02.669 INFO:teuthology.orchestra.run.smithi158.stdout: "mdsmap_epoch": 154, 2023-10-16T11:22:02.669 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch": 270, 2023-10-16T11:22:02.669 INFO:teuthology.orchestra.run.smithi158.stdout: "osdmap_epoch_barrier": 270, 2023-10-16T11:22:02.670 INFO:teuthology.orchestra.run.smithi158.stdout: "uptime": 10.871403903999999 2023-10-16T11:22:02.670 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff9a406b020 msgr2=0x7ff9a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff9a406b020 0x7ff9a406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9b00090b0 tx=0x7ff9b0008030 comp rx=0 tx=0).stop 2023-10-16T11:22:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9c0152ba0 msgr2=0x7ff9c00b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9c0152ba0 0x7ff9c00b0890 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7ff9c00cfa30 tx=0x7ff9b800f770 comp rx=0 tx=0).stop 2023-10-16T11:22:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 shutdown_connections 2023-10-16T11:22:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff9a406b020 0x7ff9a406d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9c0152ba0 0x7ff9c00b0890 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac003a00 0x7ff9c00c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 --2- 172.21.15.158:0/2472954230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9c015e9d0 0x7ff9c00caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 >> 172.21.15.158:0/2472954230 conn(0x7ff9c006b9e0 msgr2=0x7ff9c00b3180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 shutdown_connections 2023-10-16T11:22:02.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.641+0000 7ff9c87bf700 1 -- 172.21.15.158:0/2472954230 wait complete. 2023-10-16T11:22:02.674 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T11:22:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.795+0000 7ff692c7d700 1 Processor -- start 2023-10-16T11:22:02.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.795+0000 7ff692c7d700 1 -- start start 2023-10-16T11:22:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.796+0000 7ff692c7d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff68c06bf30 0x7ff68c0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.796+0000 7ff692c7d700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c0ca780 con 0x7ff68c15b730 2023-10-16T11:22:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.796+0000 7ff692c7d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c06aa90 con 0x7ff68c15b350 2023-10-16T11:22:02.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.796+0000 7ff692c7d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c0ced40 con 0x7ff68c06bf30 2023-10-16T11:22:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.796+0000 7ff68bfff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff68c15b730 0x7ff68c0ce630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44086/0 (socket says 172.21.15.158:44086) 2023-10-16T11:22:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.796+0000 7ff68bfff700 1 -- 172.21.15.158:0/1406132924 learned_addr learned my addr 172.21.15.158:0/1406132924 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff69121a700 1 --2- 172.21.15.158:0/1406132924 >> v2:172.21.15.172:3300/0 conn(0x7ff68c06bf30 0x7ff68c0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff68b7fe700 1 -- 172.21.15.158:0/1406132924 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047510857 0 0) 0x7ff68c0ca780 con 0x7ff68c15b730 2023-10-16T11:22:02.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff68b7fe700 1 -- 172.21.15.158:0/1406132924 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c0035b0 con 0x7ff68c15b730 2023-10-16T11:22:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff69121a700 1 -- 172.21.15.158:0/1406132924 >> v1:172.21.15.172:6789/0 conn(0x7ff68c15b350 legacy=0x7ff68c0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff69121a700 1 -- 172.21.15.158:0/1406132924 >> v1:172.21.15.158:6789/0 conn(0x7ff68c15b730 legacy=0x7ff68c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff68b7fe700 1 -- 172.21.15.158:0/1406132924 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2884163132 0 0) 0x7ff67c0035b0 con 0x7ff68c15b730 2023-10-16T11:22:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff69121a700 1 -- 172.21.15.158:0/1406132924 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c0cf820 con 0x7ff68c06bf30 2023-10-16T11:22:02.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff69121a700 1 --2- 172.21.15.158:0/1406132924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c06bf30 0x7ff68c0ca080 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff684009d90 tx=0x7ff68400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c0c088a63da0804 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff68b7fe700 1 -- 172.21.15.158:0/1406132924 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68400e070 con 0x7ff68c06bf30 2023-10-16T11:22:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff68b7fe700 1 -- 172.21.15.158:0/1406132924 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff684003a20 con 0x7ff68c06bf30 2023-10-16T11:22:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.797+0000 7ff68b7fe700 1 -- 172.21.15.158:0/1406132924 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678001670 con 0x7ff68c06bf30 2023-10-16T11:22:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.798+0000 7ff692c7d700 1 -- 172.21.15.158:0/1406132924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c06bf30 msgr2=0x7ff68c0ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.798+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1406132924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c06bf30 0x7ff68c0ca080 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff684009d90 tx=0x7ff68400b330 comp rx=0 tx=0).stop 2023-10-16T11:22:02.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.798+0000 7ff692c7d700 1 -- 172.21.15.158:0/1406132924 shutdown_connections 2023-10-16T11:22:02.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.798+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1406132924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c06bf30 0x7ff68c0ca080 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.798+0000 7ff692c7d700 1 -- 172.21.15.158:0/1406132924 >> 172.21.15.158:0/1406132924 conn(0x7ff68c0cc4b0 msgr2=0x7ff68c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:02.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.798+0000 7ff692c7d700 1 -- 172.21.15.158:0/1406132924 shutdown_connections 2023-10-16T11:22:02.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.798+0000 7ff692c7d700 1 -- 172.21.15.158:0/1406132924 wait complete. 2023-10-16T11:22:02.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.799+0000 7ff692c7d700 1 Processor -- start 2023-10-16T11:22:02.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.799+0000 7ff692c7d700 1 -- start start 2023-10-16T11:22:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff68c06bf30 0x7ff68c06e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff68c15b350 0x7ff68c147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c15b730 0x7ff68c0c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c06d7e0 con 0x7ff68c06bf30 2023-10-16T11:22:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c06dac0 con 0x7ff68c15b730 2023-10-16T11:22:02.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c06dda0 con 0x7ff68c15b350 2023-10-16T11:22:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff68c06bf30 0x7ff68c06e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff68c06bf30 0x7ff68c06e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43914/0 (socket says 172.21.15.158:43914) 2023-10-16T11:22:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 -- 172.21.15.158:0/1667318387 learned_addr learned my addr 172.21.15.158:0/1667318387 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff69121a700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c15b730 0x7ff68c0c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff68bfff700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff68c15b350 0x7ff68c147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 -- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff68c15b350 msgr2=0x7ff68c147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff68c15b350 0x7ff68c147e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 -- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c15b730 msgr2=0x7ff68c0c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c15b730 0x7ff68c0c5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:02.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 -- 172.21.15.158:0/1667318387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff684009a70 con 0x7ff68c06bf30 2023-10-16T11:22:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff690a19700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff68c06bf30 0x7ff68c06e880 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7ff68000c4e0 tx=0x7ff68000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff6897fa700 1 -- 172.21.15.158:0/1667318387 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6800052d0 con 0x7ff68c06bf30 2023-10-16T11:22:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff6897fa700 1 -- 172.21.15.158:0/1667318387 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff680005430 con 0x7ff68c06bf30 2023-10-16T11:22:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c06e080 con 0x7ff68c06bf30 2023-10-16T11:22:02.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff6897fa700 1 -- 172.21.15.158:0/1667318387 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6800038c0 con 0x7ff68c06bf30 2023-10-16T11:22:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.801+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c06edc0 con 0x7ff68c06bf30 2023-10-16T11:22:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.802+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c06e5a0 con 0x7ff68c06bf30 2023-10-16T11:22:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.803+0000 7ff6897fa700 1 -- 172.21.15.158:0/1667318387 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff680003a20 con 0x7ff68c06bf30 2023-10-16T11:22:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.803+0000 7ff6897fa700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff67006b020 0x7ff67006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.803+0000 7ff6897fa700 1 -- 172.21.15.158:0/1667318387 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7ff680088f50 con 0x7ff68c06bf30 2023-10-16T11:22:02.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.803+0000 7ff68bfff700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff67006b020 0x7ff67006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:02.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.804+0000 7ff68bfff700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff67006b020 0x7ff67006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0c6910 tx=0x7ff678001be0 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:02.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:02.808+0000 7ff6897fa700 1 -- 172.21.15.158:0/1667318387 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff68c06e5a0 con 0x7ff68c06bf30 2023-10-16T11:22:03.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.045+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff68c06e5a0 con 0x7ff68c06bf30 2023-10-16T11:22:03.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.045+0000 7ff6897fa700 1 -- 172.21.15.158:0/1667318387 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v154) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff68c06e5a0 con 0x7ff68c06bf30 2023-10-16T11:22:03.048 INFO:teuthology.orchestra.run.smithi158.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T11:22:03.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff67006b020 msgr2=0x7ff67006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:03.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff67006b020 0x7ff67006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0c6910 tx=0x7ff678001be0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff68c06bf30 msgr2=0x7ff68c06e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:03.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff68c06bf30 0x7ff68c06e880 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7ff68000c4e0 tx=0x7ff68000c8a0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 shutdown_connections 2023-10-16T11:22:03.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff68c15b730 0x7ff68c0c5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7ff67006b020 0x7ff67006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff68c06bf30 0x7ff68c06e880 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 --2- 172.21.15.158:0/1667318387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff68c15b350 0x7ff68c147e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 >> 172.21.15.158:0/1667318387 conn(0x7ff68c0cc4b0 msgr2=0x7ff68c1556b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:03.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 shutdown_connections 2023-10-16T11:22:03.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.048+0000 7ff692c7d700 1 -- 172.21.15.158:0/1667318387 wait complete. 2023-10-16T11:22:03.061 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T11:22:03.062 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T11:22:03.062 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2023-10-16T11:22:03.062 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T11:22:03.063 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T11:22:03.063 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T11:22:03.063 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T11:22:03.064 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T11:22:03.064 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:22:03.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.065 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2023-10-16T11:22:03.082 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T11:22:03.082 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T11:22:03.082 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T11:22:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:22:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2023-10-16T11:22:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:22:03.083 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:22:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:22:03.084 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:22:03.084 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2023-10-16T11:22:03.084 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T11:22:03.084 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2023-10-16T11:22:03.084 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T11:22:03.084 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2023-10-16T11:22:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T11:22:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 36510sec preferred_lft 36510sec 2023-10-16T11:22:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::95bf:fbee:3423:a6d8/64 scope link noprefixroute 2023-10-16T11:22:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:22:03.085 INFO:teuthology.orchestra.run.smithi158.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T11:22:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T11:22:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T11:22:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:22:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::3818:29ff:fe31:c2fa/64 scope link 2023-10-16T11:22:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:22:03.087 INFO:teuthology.orchestra.run.smithi158.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:22:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 12:a8:71:51:8b:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:22:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::10a8:71ff:fe51:8b2c/64 scope link 2023-10-16T11:22:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:22:03.087 INFO:teuthology.orchestra.run.smithi158.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T11:22:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 8e:f2:2d:a3:b2:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:22:03.088 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::8cf2:2dff:fea3:b236/64 scope link 2023-10-16T11:22:03.088 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2023-10-16T11:22:03.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.088 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T11:22:03.148 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-16T11:22:03.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.149 DEBUG:teuthology.orchestra.run.smithi158:> route 2023-10-16T11:22:03.208 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2023-10-16T11:22:03.208 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T11:22:03.208 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T11:22:03.209 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T11:22:03.209 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T11:22:03.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.210 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2023-10-16T11:22:03.210 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2023-10-16T11:22:03.210 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T11:22:03.211 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T11:22:03.211 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T11:22:03.212 DEBUG:teuthology.orchestra.run.smithi158:> ') 2023-10-16T11:22:03.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.313 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:22:03.329 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:22:03.329 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:22:03.329 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:22:03.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.330 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2023-10-16T11:22:03.396 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:22:03.397 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T11:22:03.453 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T11:22:03.453 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T11:22:03.454 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:22:03.510 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T11:22:03.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.511 DEBUG:teuthology.orchestra.run.smithi158:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:22:03.596 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:22:03.596 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:22:03.619 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:22:03.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:22:03.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:22:03.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:22:03.620 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:22:03.621 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:22:03.621 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:22:03.621 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:22:03.656 INFO:teuthology.orchestra.run.smithi158.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T11:22:03.656 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T11:22:03.656 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-10-16T11:22:03.657 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with new device syntax: 0@7cf363df-9988-44b1-be05-df2a6bf24f2d.cephfs=/ 2023-10-16T11:22:03.657 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.158:3300/172.21.15.172:3300/172.21.15.172:3301" will pass to kernel 2023-10-16T11:22:03.657 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: v2 mount support check returned 2 2023-10-16T11:22:03.657 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-16T11:22:03.657 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T11:22:03.658 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: resolved to: "172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-10-16T11:22:03.658 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.158:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-10-16T11:22:03.658 INFO:teuthology.orchestra.run.smithi158.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=7cf363df-9988-44b1-be05-df2a6bf24f2d" will pass to kernel 2023-10-16T11:22:03.659 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T11:22:03.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.660 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T11:22:03.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:03.694 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T11:22:03.783 INFO:teuthology.orchestra.run.smithi158.stdout:client13881 2023-10-16T11:22:03.783 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:22:03.783 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:03.801 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T11:22:03.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.959+0000 7f117852c700 1 Processor -- start 2023-10-16T11:22:03.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.959+0000 7f117852c700 1 -- start start 2023-10-16T11:22:03.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f117852c700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f117015eb10 0x7f117015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:03.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f117852c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700d0ae0 0x7f11700b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:03.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f117852c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11700b4950 0x7f11700cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:03.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f117852c700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11700cdef0 con 0x7f117015eb10 2023-10-16T11:22:03.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f117852c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11700cd070 con 0x7f11700b4950 2023-10-16T11:22:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f117852c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700cf3a0 con 0x7f11700d0ae0 2023-10-16T11:22:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f11762c8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f117015eb10 0x7f117015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f11762c8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f117015eb10 0x7f117015f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43926/0 (socket says 172.21.15.158:43926) 2023-10-16T11:22:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.960+0000 7f11762c8700 1 -- 172.21.15.158:0/2637460398 learned_addr learned my addr 172.21.15.158:0/2637460398 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:03.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f1175ac7700 1 --2- 172.21.15.158:0/2637460398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700d0ae0 0x7f11700b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11762c8700 1 -- 172.21.15.158:0/2637460398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700d0ae0 msgr2=0x7f11700b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f1176ac9700 1 --2- 172.21.15.158:0/2637460398 >> v2:172.21.15.172:3300/0 conn(0x7f11700b4950 0x7f11700cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11762c8700 1 --2- 172.21.15.158:0/2637460398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700d0ae0 0x7f11700b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11762c8700 1 -- 172.21.15.158:0/2637460398 >> v2:172.21.15.172:3300/0 conn(0x7f11700b4950 msgr2=0x7f11700cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:03.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11762c8700 1 --2- 172.21.15.158:0/2637460398 >> v2:172.21.15.172:3300/0 conn(0x7f11700b4950 0x7f11700cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11762c8700 1 -- 172.21.15.158:0/2637460398 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117015f6f0 con 0x7f117015eb10 2023-10-16T11:22:03.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11762c8700 1 --2- 172.21.15.158:0/2637460398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f117015f1b0 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f1170071670 tx=0x7f1160009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bf938489dfc26f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:03.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11752c6700 1 -- 172.21.15.158:0/2637460398 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116000b870 con 0x7f117015eb10 2023-10-16T11:22:03.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11752c6700 1 -- 172.21.15.158:0/2637460398 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116000b9d0 con 0x7f117015eb10 2023-10-16T11:22:03.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.961+0000 7f11752c6700 1 -- 172.21.15.158:0/2637460398 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116000bd60 con 0x7f117015eb10 2023-10-16T11:22:03.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 -- 172.21.15.158:0/2637460398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 msgr2=0x7f117015f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:03.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 --2- 172.21.15.158:0/2637460398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f117015f1b0 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f1170071670 tx=0x7f1160009bd0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 -- 172.21.15.158:0/2637460398 shutdown_connections 2023-10-16T11:22:03.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 --2- 172.21.15.158:0/2637460398 >> v2:172.21.15.172:3300/0 conn(0x7f11700b4950 0x7f11700cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 --2- 172.21.15.158:0/2637460398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f117015f1b0 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 --2- 172.21.15.158:0/2637460398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700d0ae0 0x7f11700b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 -- 172.21.15.158:0/2637460398 >> 172.21.15.158:0/2637460398 conn(0x7f11700ce040 msgr2=0x7f11700ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:03.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 -- 172.21.15.158:0/2637460398 shutdown_connections 2023-10-16T11:22:03.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.962+0000 7f117852c700 1 -- 172.21.15.158:0/2637460398 wait complete. 2023-10-16T11:22:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.963+0000 7f117852c700 1 Processor -- start 2023-10-16T11:22:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.963+0000 7f117852c700 1 -- start start 2023-10-16T11:22:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700b4950 0x7f117014d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11700d0ae0 0x7f11700a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:03.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f11700adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11700a9e30 con 0x7f117015eb10 2023-10-16T11:22:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11700ab6a0 con 0x7f11700d0ae0 2023-10-16T11:22:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700a9870 con 0x7f11700b4950 2023-10-16T11:22:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f11700adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:03.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f11700adab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43936/0 (socket says 172.21.15.158:43936) 2023-10-16T11:22:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 -- 172.21.15.158:0/1160961031 learned_addr learned my addr 172.21.15.158:0/1160961031 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f11762c8700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700b4950 0x7f117014d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1175ac7700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11700d0ae0 0x7f11700a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 -- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700b4950 msgr2=0x7f117014d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:03.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700b4950 0x7f117014d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 -- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11700d0ae0 msgr2=0x7f11700a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11700d0ae0 0x7f11700a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 -- 172.21.15.158:0/1160961031 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160009550 con 0x7f117015eb10 2023-10-16T11:22:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1176ac9700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f11700adab0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f116800cd20 tx=0x7f116800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:03.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1166ffd700 1 -- 172.21.15.158:0/1160961031 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116800ce10 con 0x7f117015eb10 2023-10-16T11:22:03.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1166ffd700 1 -- 172.21.15.158:0/1160961031 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1168003a20 con 0x7f117015eb10 2023-10-16T11:22:03.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11700a9b50 con 0x7f117015eb10 2023-10-16T11:22:03.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f1166ffd700 1 -- 172.21.15.158:0/1160961031 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168003db0 con 0x7f117015eb10 2023-10-16T11:22:03.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.964+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117014ded0 con 0x7f117015eb10 2023-10-16T11:22:03.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.965+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11700ab190 con 0x7f117015eb10 2023-10-16T11:22:03.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.966+0000 7f1166ffd700 1 -- 172.21.15.158:0/1160961031 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1168003f10 con 0x7f117015eb10 2023-10-16T11:22:03.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.966+0000 7f1166ffd700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f115406b020 0x7f115406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:03.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.966+0000 7f11762c8700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f115406b020 0x7f115406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:03.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.966+0000 7f1166ffd700 1 -- 172.21.15.158:0/1160961031 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f1168012070 con 0x7f117015eb10 2023-10-16T11:22:03.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.967+0000 7f11762c8700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f115406b020 0x7f115406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1160014040 tx=0x7f1160008030 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:03.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:03.971+0000 7f1166ffd700 1 -- 172.21.15.158:0/1160961031 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11700ab190 con 0x7f117015eb10 2023-10-16T11:22:04.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.191+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 --> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f117015fc70 con 0x7f115406b020 2023-10-16T11:22:04.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.193+0000 7f1166ffd700 1 -- 172.21.15.158:0/1160961031 <== mgr.13500 v2:172.21.15.158:6832/778561374 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1160006d10 con 0x7f115406b020 2023-10-16T11:22:04.194 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-16T11:22:04.195 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-16T11:22:04.195 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "cephfs" 2023-10-16T11:22:04.195 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-16T11:22:04.195 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-16T11:22:04.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.194+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f115406b020 msgr2=0x7f115406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f115406b020 0x7f115406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1160014040 tx=0x7f1160008030 comp rx=0 tx=0).stop 2023-10-16T11:22:04.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 msgr2=0x7f11700adab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f11700adab0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f116800cd20 tx=0x7f116800a570 comp rx=0 tx=0).stop 2023-10-16T11:22:04.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 shutdown_connections 2023-10-16T11:22:04.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f115406b020 0x7f115406d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f117015eb10 0x7f11700adab0 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11700b4950 0x7f117014d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 --2- 172.21.15.158:0/1160961031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11700d0ae0 0x7f11700a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 >> 172.21.15.158:0/1160961031 conn(0x7f11700ce040 msgr2=0x7f11700b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:04.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 shutdown_connections 2023-10-16T11:22:04.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.195+0000 7f117852c700 1 -- 172.21.15.158:0/1160961031 wait complete. 2023-10-16T11:22:04.208 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T11:22:04.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.330+0000 7fcccd844700 1 Processor -- start 2023-10-16T11:22:04.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.330+0000 7fcccd844700 1 -- start start 2023-10-16T11:22:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.330+0000 7fcccd844700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc815f0a0 0x7fccc815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.330+0000 7fcccd844700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fccc815c720 0x7fccc80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.330+0000 7fcccd844700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc815f730 con 0x7fccc80b36c0 2023-10-16T11:22:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.330+0000 7fcccd844700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc815ffd0 con 0x7fccc815c720 2023-10-16T11:22:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.330+0000 7fcccd844700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc815bab0 con 0x7fccc815f0a0 2023-10-16T11:22:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc77fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fccc815c720 0x7fccc80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc67fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc815f0a0 0x7fccc815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc77fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fccc815c720 0x7fccc80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:45124/0 (socket says 172.21.15.158:45124) 2023-10-16T11:22:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc77fe700 1 -- 172.21.15.158:0/2142404708 learned_addr learned my addr 172.21.15.158:0/2142404708 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc67fc700 1 -- 172.21.15.158:0/2142404708 >> v2:172.21.15.172:3300/0 conn(0x7fccc815c720 msgr2=0x7fccc80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc67fc700 1 --2- 172.21.15.158:0/2142404708 >> v2:172.21.15.172:3300/0 conn(0x7fccc815c720 0x7fccc80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc67fc700 1 -- 172.21.15.158:0/2142404708 >> v1:172.21.15.172:6789/0 conn(0x7fccc80b36c0 legacy=0x7fccc80d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc67fc700 1 -- 172.21.15.158:0/2142404708 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc815fa10 con 0x7fccc815f0a0 2023-10-16T11:22:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc67fc700 1 --2- 172.21.15.158:0/2142404708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc815f0a0 0x7fccc815c1e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009840 tx=0x7fccb0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d1c8969849321b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.331+0000 7fccc5ffb700 1 -- 172.21.15.158:0/2142404708 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0011830 con 0x7fccc815f0a0 2023-10-16T11:22:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fccc5ffb700 1 -- 172.21.15.158:0/2142404708 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb0011990 con 0x7fccc815f0a0 2023-10-16T11:22:04.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fccc5ffb700 1 -- 172.21.15.158:0/2142404708 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0011d20 con 0x7fccc815f0a0 2023-10-16T11:22:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 -- 172.21.15.158:0/2142404708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc815f0a0 msgr2=0x7fccc815c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 --2- 172.21.15.158:0/2142404708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc815f0a0 0x7fccc815c1e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009840 tx=0x7fccb0009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 -- 172.21.15.158:0/2142404708 shutdown_connections 2023-10-16T11:22:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 --2- 172.21.15.158:0/2142404708 >> v2:172.21.15.172:3300/0 conn(0x7fccc815c720 0x7fccc80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 --2- 172.21.15.158:0/2142404708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc815f0a0 0x7fccc815c1e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 -- 172.21.15.158:0/2142404708 >> 172.21.15.158:0/2142404708 conn(0x7fccc80cdf90 msgr2=0x7fccc80ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:04.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 -- 172.21.15.158:0/2142404708 shutdown_connections 2023-10-16T11:22:04.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.332+0000 7fcccd844700 1 -- 172.21.15.158:0/2142404708 wait complete. 2023-10-16T11:22:04.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.333+0000 7fcccd844700 1 Processor -- start 2023-10-16T11:22:04.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.333+0000 7fcccd844700 1 -- start start 2023-10-16T11:22:04.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fcccd844700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc80b36c0 0x7fccc806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fcccd844700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccc815f0a0 0x7fccc8153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fcccd844700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccc806b9e0 0x7fccc806be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fcccd844700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc806d0b0 con 0x7fccc815f0a0 2023-10-16T11:22:04.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fcccd844700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc806c4a0 con 0x7fccc806b9e0 2023-10-16T11:22:04.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fcccd844700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc806c780 con 0x7fccc80b36c0 2023-10-16T11:22:04.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fccc67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccc815f0a0 0x7fccc8153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fccc67fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccc815f0a0 0x7fccc8153d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43948/0 (socket says 172.21.15.158:43948) 2023-10-16T11:22:04.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fccc67fc700 1 -- 172.21.15.158:0/224289331 learned_addr learned my addr 172.21.15.158:0/224289331 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:04.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fccc77fe700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccc806b9e0 0x7fccc806be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.334+0000 7fccc6ffd700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc80b36c0 0x7fccc806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccc67fc700 1 -- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc80b36c0 msgr2=0x7fccc806d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccc67fc700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc80b36c0 0x7fccc806d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccc67fc700 1 -- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccc806b9e0 msgr2=0x7fccc806be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccc67fc700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccc806b9e0 0x7fccc806be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccc67fc700 1 -- 172.21.15.158:0/224289331 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb00094a0 con 0x7fccc815f0a0 2023-10-16T11:22:04.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccc67fc700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccc815f0a0 0x7fccc8153d80 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009fd0 tx=0x7fccb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:04.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccbffff700 1 -- 172.21.15.158:0/224289331 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00036a0 con 0x7fccc815f0a0 2023-10-16T11:22:04.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccbffff700 1 -- 172.21.15.158:0/224289331 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb001e070 con 0x7fccc815f0a0 2023-10-16T11:22:04.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc806ca60 con 0x7fccc815f0a0 2023-10-16T11:22:04.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fccbffff700 1 -- 172.21.15.158:0/224289331 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0003a40 con 0x7fccc815f0a0 2023-10-16T11:22:04.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.335+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc81e6260 con 0x7fccc815f0a0 2023-10-16T11:22:04.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.336+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc80681b0 con 0x7fccc815f0a0 2023-10-16T11:22:04.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.336+0000 7fccbffff700 1 -- 172.21.15.158:0/224289331 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fccb0003ba0 con 0x7fccc815f0a0 2023-10-16T11:22:04.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.337+0000 7fccbffff700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fcca806b020 0x7fcca806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.337+0000 7fccbffff700 1 -- 172.21.15.158:0/224289331 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fccb008a900 con 0x7fccc815f0a0 2023-10-16T11:22:04.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.337+0000 7fccc6ffd700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fcca806b020 0x7fcca806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.337+0000 7fccc6ffd700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fcca806b020 0x7fcca806d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000fb0 tx=0x7fccc0009040 comp rx=0 tx=0).ready entity=mgr.13500 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:04.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.341+0000 7fccbffff700 1 -- 172.21.15.158:0/224289331 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccb0055610 con 0x7fccc815f0a0 2023-10-16T11:22:04.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.589+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fccc80b1c30 con 0x7fccc815f0a0 2023-10-16T11:22:04.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.767+0000 7fccbffff700 1 -- 172.21.15.158:0/224289331 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v167) v1 ==== 92+0+0 (secure 0 0 0) 0x7fccb00150a0 con 0x7fccc815f0a0 2023-10-16T11:22:04.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.770+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fcca806b020 msgr2=0x7fcca806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.770+0000 7fcccd844700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fcca806b020 0x7fcca806d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000fb0 tx=0x7fccc0009040 comp rx=0 tx=0).stop 2023-10-16T11:22:04.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.770+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccc815f0a0 msgr2=0x7fccc8153d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.770+0000 7fcccd844700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccc815f0a0 0x7fccc8153d80 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009fd0 tx=0x7fccb0004750 comp rx=0 tx=0).stop 2023-10-16T11:22:04.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 shutdown_connections 2023-10-16T11:22:04.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccc806b9e0 0x7fccc806be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccc80b36c0 0x7fccc806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7fcca806b020 0x7fcca806d4e0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000fb0 tx=0x7fccc0009040 comp rx=0 tx=0).stop 2023-10-16T11:22:04.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 --2- 172.21.15.158:0/224289331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fccc815f0a0 0x7fccc8153d80 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 >> 172.21.15.158:0/224289331 conn(0x7fccc80cdf90 msgr2=0x7fccc8159de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:04.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 shutdown_connections 2023-10-16T11:22:04.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.771+0000 7fcccd844700 1 -- 172.21.15.158:0/224289331 wait complete. 2023-10-16T11:22:04.787 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T11:22:04.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.914+0000 7f000773d700 1 Processor -- start 2023-10-16T11:22:04.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.914+0000 7f000773d700 1 -- start start 2023-10-16T11:22:04.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f000773d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f00000ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f000773d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f00000cbaf0 0x7f00000cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f000773d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00000cf4c0 con 0x7f000015f1d0 2023-10-16T11:22:04.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f000773d700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00000cc4c0 con 0x7f00000cbaf0 2023-10-16T11:22:04.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f000773d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00000b49e0 con 0x7f000015b590 2023-10-16T11:22:04.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f00000cbaf0 0x7f00000cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f00000cbaf0 0x7f00000cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43984/0 (socket says 172.21.15.158:43984) 2023-10-16T11:22:04.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 -- 172.21.15.158:0/2406050889 learned_addr learned my addr 172.21.15.158:0/2406050889 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:04.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f00054d9700 1 --2- 172.21.15.158:0/2406050889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f00000ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 -- 172.21.15.158:0/2406050889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 msgr2=0x7f00000ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:04.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 --2- 172.21.15.158:0/2406050889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f00000ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7efff7fff700 1 -- 172.21.15.158:0/2406050889 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741416930 0 0) 0x7f00000cf4c0 con 0x7f000015f1d0 2023-10-16T11:22:04.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 -- 172.21.15.158:0/2406050889 >> v1:172.21.15.172:6789/0 conn(0x7f000015f1d0 legacy=0x7f00000b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 -- 172.21.15.158:0/2406050889 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000015f970 con 0x7f00000cbaf0 2023-10-16T11:22:04.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.915+0000 7f0005cda700 1 --2- 172.21.15.158:0/2406050889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00000cbf80 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7efff8009e80 tx=0x7efff800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08adbe156ab5a16 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:04.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7efff7fff700 1 -- 172.21.15.158:0/2406050889 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff800e070 con 0x7f00000cbaf0 2023-10-16T11:22:04.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7efff7fff700 1 -- 172.21.15.158:0/2406050889 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff800b8d0 con 0x7f00000cbaf0 2023-10-16T11:22:04.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7efff7fff700 1 -- 172.21.15.158:0/2406050889 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff800bc80 con 0x7f00000cbaf0 2023-10-16T11:22:04.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 -- 172.21.15.158:0/2406050889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 msgr2=0x7f00000cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 --2- 172.21.15.158:0/2406050889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00000cbf80 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7efff8009e80 tx=0x7efff800b330 comp rx=0 tx=0).stop 2023-10-16T11:22:04.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 -- 172.21.15.158:0/2406050889 shutdown_connections 2023-10-16T11:22:04.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 --2- 172.21.15.158:0/2406050889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00000cbf80 unknown :-1 s=CLOSED pgs=3323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 --2- 172.21.15.158:0/2406050889 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f00000ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 -- 172.21.15.158:0/2406050889 >> 172.21.15.158:0/2406050889 conn(0x7f000006a8d0 msgr2=0x7f000006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:04.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 -- 172.21.15.158:0/2406050889 shutdown_connections 2023-10-16T11:22:04.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.916+0000 7f000773d700 1 -- 172.21.15.158:0/2406050889 wait complete. 2023-10-16T11:22:04.924 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:04.917+0000 7f987761f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:22:04.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.917+0000 7f000773d700 1 Processor -- start 2023-10-16T11:22:04.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f000773d700 1 -- start start 2023-10-16T11:22:04.925 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f008fff5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:22:04.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f000773d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00001f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f000773d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f000015cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f000773d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f000015f1d0 0x7f000015d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f000773d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000015c140 con 0x7f00000cbaf0 2023-10-16T11:22:04.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f000773d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00001f74d0 con 0x7f000015f1d0 2023-10-16T11:22:04.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f000773d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00000ca2f0 con 0x7f000015b590 2023-10-16T11:22:04.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f00054d9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00001f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f00054d9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00001f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:43986/0 (socket says 172.21.15.158:43986) 2023-10-16T11:22:04.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.918+0000 7f00054d9700 1 -- 172.21.15.158:0/4008087742 learned_addr learned my addr 172.21.15.158:0/4008087742 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:04.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f0005cda700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f000015f1d0 0x7f000015d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f0004cd8700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f000015cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:04.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f00054d9700 1 -- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 msgr2=0x7f000015cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f00054d9700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f000015cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f00054d9700 1 -- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f000015f1d0 msgr2=0x7f000015d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:04.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f00054d9700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f000015f1d0 0x7f000015d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:04.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f00054d9700 1 -- 172.21.15.158:0/4008087742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff8009b30 con 0x7f00000cbaf0 2023-10-16T11:22:04.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f0004cd8700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f000015cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:04.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f00054d9700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00001f6f90 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e9b0 tx=0x7efffc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:04.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7efff5ffb700 1 -- 172.21.15.158:0/4008087742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc009710 con 0x7f00000cbaf0 2023-10-16T11:22:04.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7efff5ffb700 1 -- 172.21.15.158:0/4008087742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc00c850 con 0x7f00000cbaf0 2023-10-16T11:22:04.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000015baf0 con 0x7f00000cbaf0 2023-10-16T11:22:04.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7efff5ffb700 1 -- 172.21.15.158:0/4008087742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc00cbc0 con 0x7f00000cbaf0 2023-10-16T11:22:04.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.919+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0000158560 con 0x7f00000cbaf0 2023-10-16T11:22:04.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.920+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000006d510 con 0x7f00000cbaf0 2023-10-16T11:22:04.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.921+0000 7efff5ffb700 1 -- 172.21.15.158:0/4008087742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 167) v1 ==== 89348+0+0 (secure 0 0 0) 0x7efffc00cd20 con 0x7f00000cbaf0 2023-10-16T11:22:04.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.921+0000 7efff5ffb700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 0x7effe406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:04.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.921+0000 7f0004cd8700 1 -- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 msgr2=0x7effe406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:04.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.921+0000 7efff5ffb700 1 -- 172.21.15.158:0/4008087742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7efffc08a3a0 con 0x7f00000cbaf0 2023-10-16T11:22:04.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.921+0000 7f0004cd8700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 0x7effe406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:04.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:04.925+0000 7efff5ffb700 1 -- 172.21.15.158:0/4008087742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efffc055000 con 0x7f00000cbaf0 2023-10-16T11:22:04.985 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:04.983+0000 7f987761f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:04.989 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:04.987+0000 7f008fff5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:05.057 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:05.056+0000 7f987761f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:22:05.063 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:05.061+0000 7f008fff5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:22:05.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.122+0000 7f0004cd8700 1 -- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 msgr2=0x7effe406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:05.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.122+0000 7f0004cd8700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 0x7effe406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:05.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.127+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f00000681b0 con 0x7f00000cbaf0 2023-10-16T11:22:05.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.130+0000 7efff5ffb700 1 -- 172.21.15.158:0/4008087742 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7efffc058620 con 0x7f00000cbaf0 2023-10-16T11:22:05.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 msgr2=0x7effe406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:05.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 0x7effe406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 msgr2=0x7f00001f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00001f6f90 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e9b0 tx=0x7efffc00ecc0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 shutdown_connections 2023-10-16T11:22:05.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f000015f1d0 0x7f000015d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7effe406b070 0x7effe406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00000cbaf0 0x7f00001f6f90 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 --2- 172.21.15.158:0/4008087742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f000015b590 0x7f000015cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 >> 172.21.15.158:0/4008087742 conn(0x7f000006a8d0 msgr2=0x7f00000b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:05.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 shutdown_connections 2023-10-16T11:22:05.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.133+0000 7f000773d700 1 -- 172.21.15.158:0/4008087742 wait complete. 2023-10-16T11:22:05.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:05.134+0000 7f987761f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:22:05.144 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:05.142+0000 7f008fff5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:22:05.147 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T11:22:05.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.272+0000 7f079a7ed700 1 Processor -- start 2023-10-16T11:22:05.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.273+0000 7f079a7ed700 1 -- start start 2023-10-16T11:22:05.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.273+0000 7f079a7ed700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0794159280 0x7f079406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.273+0000 7f079a7ed700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f07940d0e10 0x7f079415c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.273+0000 7f079a7ed700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0794159660 con 0x7f07940cbef0 2023-10-16T11:22:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.273+0000 7f079a7ed700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07940cb710 con 0x7f07940d0e10 2023-10-16T11:22:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.273+0000 7f079a7ed700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079406b120 con 0x7f0794159280 2023-10-16T11:22:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.273+0000 7f07937fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f07940d0e10 0x7f079415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f07937fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f07940d0e10 0x7f079415c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44040/0 (socket says 172.21.15.158:44040) 2023-10-16T11:22:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f07937fe700 1 -- 172.21.15.158:0/106490172 learned_addr learned my addr 172.21.15.158:0/106490172 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f0793fff700 1 --2- 172.21.15.158:0/106490172 >> v2:172.21.15.172:3300/0 conn(0x7f0794159280 0x7f079406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f07937fe700 1 -- 172.21.15.158:0/106490172 >> v2:172.21.15.172:3300/0 conn(0x7f0794159280 msgr2=0x7f079406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f07937fe700 1 --2- 172.21.15.158:0/106490172 >> v2:172.21.15.172:3300/0 conn(0x7f0794159280 0x7f079406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f07937fe700 1 -- 172.21.15.158:0/106490172 >> v1:172.21.15.172:6789/0 conn(0x7f07940cbef0 legacy=0x7f07940cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f07937fe700 1 -- 172.21.15.158:0/106490172 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079406b400 con 0x7f07940d0e10 2023-10-16T11:22:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f0793fff700 1 --2- 172.21.15.158:0/106490172 >> v2:172.21.15.172:3300/0 conn(0x7f0794159280 0x7f079406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f07937fe700 1 --2- 172.21.15.158:0/106490172 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07940d0e10 0x7f079415c590 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f07940cdf50 tx=0x7f0788009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15c01cd598ac68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f0792ffd700 1 -- 172.21.15.158:0/106490172 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078800b870 con 0x7f07940d0e10 2023-10-16T11:22:05.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f0792ffd700 1 -- 172.21.15.158:0/106490172 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078800b9d0 con 0x7f07940d0e10 2023-10-16T11:22:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.274+0000 7f0792ffd700 1 -- 172.21.15.158:0/106490172 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078800bd60 con 0x7f07940d0e10 2023-10-16T11:22:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 -- 172.21.15.158:0/106490172 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07940d0e10 msgr2=0x7f079415c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 --2- 172.21.15.158:0/106490172 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07940d0e10 0x7f079415c590 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f07940cdf50 tx=0x7f0788009bd0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 -- 172.21.15.158:0/106490172 shutdown_connections 2023-10-16T11:22:05.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 --2- 172.21.15.158:0/106490172 >> v2:172.21.15.172:3300/0 conn(0x7f0794159280 0x7f079406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 --2- 172.21.15.158:0/106490172 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07940d0e10 0x7f079415c590 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 -- 172.21.15.158:0/106490172 >> 172.21.15.158:0/106490172 conn(0x7f0794099a10 msgr2=0x7f07940d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 -- 172.21.15.158:0/106490172 shutdown_connections 2023-10-16T11:22:05.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.275+0000 7f079a7ed700 1 -- 172.21.15.158:0/106490172 wait complete. 2023-10-16T11:22:05.283 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:05.276+0000 7f987761f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:22:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.276+0000 7f079a7ed700 1 Processor -- start 2023-10-16T11:22:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.277+0000 7f079a7ed700 1 -- start start 2023-10-16T11:22:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.277+0000 7f079a7ed700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07940cbef0 0x7f07940c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.277+0000 7f079a7ed700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07940d0e10 0x7f0794157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f079a7ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0794159280 0x7f079415e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f079a7ed700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07940c8500 con 0x7f0794159280 2023-10-16T11:22:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f079a7ed700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07940c48e0 con 0x7f07940cbef0 2023-10-16T11:22:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f079a7ed700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07940b57b0 con 0x7f07940d0e10 2023-10-16T11:22:05.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0794159280 0x7f079415e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0794159280 0x7f079415e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44048/0 (socket says 172.21.15.158:44048) 2023-10-16T11:22:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 -- 172.21.15.158:0/4084074351 learned_addr learned my addr 172.21.15.158:0/4084074351 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0793fff700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07940cbef0 0x7f07940c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f07937fe700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07940d0e10 0x7f0794157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 -- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07940d0e10 msgr2=0x7f0794157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07940d0e10 0x7f0794157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 -- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07940cbef0 msgr2=0x7f07940c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07940cbef0 0x7f07940c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 -- 172.21.15.158:0/4084074351 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788009550 con 0x7f0794159280 2023-10-16T11:22:05.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0798d8a700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0794159280 0x7f079415e660 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f078c00cb50 tx=0x7f078c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:05.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0790ff9700 1 -- 172.21.15.158:0/4084074351 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c003c00 con 0x7f0794159280 2023-10-16T11:22:05.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0790ff9700 1 -- 172.21.15.158:0/4084074351 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c003d60 con 0x7f0794159280 2023-10-16T11:22:05.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07940b5ce0 con 0x7f0794159280 2023-10-16T11:22:05.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f0790ff9700 1 -- 172.21.15.158:0/4084074351 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c00a9c0 con 0x7f0794159280 2023-10-16T11:22:05.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.278+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07940c7f40 con 0x7f0794159280 2023-10-16T11:22:05.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.279+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07940c4a80 con 0x7f0794159280 2023-10-16T11:22:05.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.280+0000 7f0790ff9700 1 -- 172.21.15.158:0/4084074351 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 167) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f078c00ab20 con 0x7f0794159280 2023-10-16T11:22:05.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.280+0000 7f0790ff9700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 0x7f078006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.280+0000 7f0793fff700 1 -- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 msgr2=0x7f078006d560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:05.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.280+0000 7f0790ff9700 1 -- 172.21.15.158:0/4084074351 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f078c089a80 con 0x7f0794159280 2023-10-16T11:22:05.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.280+0000 7f0793fff700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 0x7f078006d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:05.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.285+0000 7f0790ff9700 1 -- 172.21.15.158:0/4084074351 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07940c4a80 con 0x7f0794159280 2023-10-16T11:22:05.295 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:05.294+0000 7f008fff5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:22:05.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.481+0000 7f0793fff700 1 -- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 msgr2=0x7f078006d560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:05.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.481+0000 7f0793fff700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 0x7f078006d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:05.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.484+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f07940c4a20 con 0x7f0794159280 2023-10-16T11:22:05.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.487+0000 7f0790ff9700 1 -- 172.21.15.158:0/4084074351 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7f078c0547a0 con 0x7f0794159280 2023-10-16T11:22:05.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 msgr2=0x7f078006d560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:05.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 0x7f078006d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0794159280 msgr2=0x7f079415e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0794159280 0x7f079415e660 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f078c00cb50 tx=0x7f078c00ce60 comp rx=0 tx=0).stop 2023-10-16T11:22:05.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 shutdown_connections 2023-10-16T11:22:05.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f078006b0a0 0x7f078006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0794159280 0x7f079415e660 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07940cbef0 0x7f07940c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 --2- 172.21.15.158:0/4084074351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07940d0e10 0x7f0794157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 >> 172.21.15.158:0/4084074351 conn(0x7f0794099a10 msgr2=0x7f079406f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:05.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 shutdown_connections 2023-10-16T11:22:05.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.489+0000 7f079a7ed700 1 -- 172.21.15.158:0/4084074351 wait complete. 2023-10-16T11:22:05.504 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create a sv1 2023-10-16T11:22:05.601 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:05.600+0000 7f987761f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:22:05.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.628+0000 7f4915b59700 1 Processor -- start 2023-10-16T11:22:05.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.628+0000 7f4915b59700 1 -- start start 2023-10-16T11:22:05.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f4915b59700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4910071920 0x7f491006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f4915b59700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49100cf6c0 0x7f491015f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f4915b59700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4910071d00 0x7f491015f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f4915b59700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491006a8d0 con 0x7f4910071d00 2023-10-16T11:22:05.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f4915b59700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491006abb0 con 0x7f4910071920 2023-10-16T11:22:05.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f4915b59700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100cbb20 con 0x7f49100cf6c0 2023-10-16T11:22:05.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f490ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4910071d00 0x7f491015f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f490ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4910071d00 0x7f491015f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44098/0 (socket says 172.21.15.158:44098) 2023-10-16T11:22:05.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f490ffff700 1 -- 172.21.15.158:0/993387610 learned_addr learned my addr 172.21.15.158:0/993387610 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f4907fff700 1 --2- 172.21.15.158:0/993387610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49100cf6c0 0x7f491015f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.629+0000 7f490f7fe700 1 --2- 172.21.15.158:0/993387610 >> v2:172.21.15.172:3300/0 conn(0x7f4910071920 0x7f491006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490ffff700 1 -- 172.21.15.158:0/993387610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49100cf6c0 msgr2=0x7f491015f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490ffff700 1 --2- 172.21.15.158:0/993387610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49100cf6c0 0x7f491015f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490ffff700 1 -- 172.21.15.158:0/993387610 >> v2:172.21.15.172:3300/0 conn(0x7f4910071920 msgr2=0x7f491006d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490ffff700 1 --2- 172.21.15.158:0/993387610 >> v2:172.21.15.172:3300/0 conn(0x7f4910071920 0x7f491006d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490ffff700 1 -- 172.21.15.158:0/993387610 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491006dcb0 con 0x7f4910071d00 2023-10-16T11:22:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f4907fff700 1 --2- 172.21.15.158:0/993387610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49100cf6c0 0x7f491015f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490ffff700 1 --2- 172.21.15.158:0/993387610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 0x7f491015f7c0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f49100b2870 tx=0x7f490800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a156f85c2459435 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490effd700 1 -- 172.21.15.158:0/993387610 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490800e070 con 0x7f4910071d00 2023-10-16T11:22:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490effd700 1 -- 172.21.15.158:0/993387610 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490800b910 con 0x7f4910071d00 2023-10-16T11:22:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f490effd700 1 -- 172.21.15.158:0/993387610 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490800bcc0 con 0x7f4910071d00 2023-10-16T11:22:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f4915b59700 1 -- 172.21.15.158:0/993387610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 msgr2=0x7f491015f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.630+0000 7f4915b59700 1 --2- 172.21.15.158:0/993387610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 0x7f491015f7c0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f49100b2870 tx=0x7f490800b330 comp rx=0 tx=0).stop 2023-10-16T11:22:05.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.631+0000 7f4915b59700 1 -- 172.21.15.158:0/993387610 shutdown_connections 2023-10-16T11:22:05.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.631+0000 7f4915b59700 1 --2- 172.21.15.158:0/993387610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 0x7f491015f7c0 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.631+0000 7f4915b59700 1 --2- 172.21.15.158:0/993387610 >> v2:172.21.15.172:3300/0 conn(0x7f4910071920 0x7f491006d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.631+0000 7f4915b59700 1 --2- 172.21.15.158:0/993387610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49100cf6c0 0x7f491015f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.631+0000 7f4915b59700 1 -- 172.21.15.158:0/993387610 >> 172.21.15.158:0/993387610 conn(0x7f49100b1bd0 msgr2=0x7f49100b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:05.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.631+0000 7f4915b59700 1 -- 172.21.15.158:0/993387610 shutdown_connections 2023-10-16T11:22:05.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.631+0000 7f4915b59700 1 -- 172.21.15.158:0/993387610 wait complete. 2023-10-16T11:22:05.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.632+0000 7f4915b59700 1 Processor -- start 2023-10-16T11:22:05.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.632+0000 7f4915b59700 1 -- start start 2023-10-16T11:22:05.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4915b59700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4910071920 0x7f491014bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4915b59700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 0x7f49101f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4915b59700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49100cf6c0 0x7f49100b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4915b59700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910151610 con 0x7f4910071d00 2023-10-16T11:22:05.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4915b59700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49101518f0 con 0x7f49100cf6c0 2023-10-16T11:22:05.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4915b59700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4910151bd0 con 0x7f4910071920 2023-10-16T11:22:05.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4907fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 0x7f49101f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.643 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4907fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 0x7f49101f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44104/0 (socket says 172.21.15.158:44104) 2023-10-16T11:22:05.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f4907fff700 1 -- 172.21.15.158:0/3403701627 learned_addr learned my addr 172.21.15.158:0/3403701627 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:05.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4910071920 0x7f491014bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.633+0000 7f490ffff700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49100cf6c0 0x7f49100b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:05.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4907fff700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4910071920 msgr2=0x7f491014bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4907fff700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4910071920 0x7f491014bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.644 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4907fff700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49100cf6c0 msgr2=0x7f49100b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:05.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4907fff700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49100cf6c0 0x7f49100b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:05.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4907fff700 1 -- 172.21.15.158:0/3403701627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4908009b30 con 0x7f4910071d00 2023-10-16T11:22:05.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4907fff700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4910071d00 0x7f49101f4190 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f48f800e810 tx=0x7f48f800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:05.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f800c020 con 0x7f4910071d00 2023-10-16T11:22:05.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f8004250 con 0x7f4910071d00 2023-10-16T11:22:05.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f80036a0 con 0x7f4910071d00 2023-10-16T11:22:05.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4915b59700 1 -- 172.21.15.158:0/3403701627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4910151eb0 con 0x7f4910071d00 2023-10-16T11:22:05.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.634+0000 7f4915b59700 1 -- 172.21.15.158:0/3403701627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4910151230 con 0x7f4910071d00 2023-10-16T11:22:05.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.635+0000 7f4915b59700 1 -- 172.21.15.158:0/3403701627 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49101520f0 con 0x7f4910071d00 2023-10-16T11:22:05.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.635+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 167) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f48f8003800 con 0x7f4910071d00 2023-10-16T11:22:05.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.636+0000 7f490cff9700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 0x7f48f006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:05.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.636+0000 7f490f7fe700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 msgr2=0x7f48f006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:05.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.636+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f48f8089580 con 0x7f4910071d00 2023-10-16T11:22:05.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.636+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 0x7f48f006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:05.648 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:05.636+0000 7f008fff5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:22:05.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.640+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48f80540f0 con 0x7f4910071d00 2023-10-16T11:22:05.718 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:05.716+0000 7f987761f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:22:05.758 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:05.757+0000 7f008fff5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:22:05.791 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:05.790+0000 7f987761f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:22:05.834 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:05.832+0000 7f008fff5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:22:05.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.837+0000 7f490f7fe700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 msgr2=0x7f48f006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:05.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.837+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 0x7f48f006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:05.872 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:05.870+0000 7f987761f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:22:05.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:05.877+0000 7f4915b59700 1 -- 172.21.15.158:0/3403701627 --> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "a", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4910151a60 con 0x7f48f006b070 2023-10-16T11:22:05.916 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:05.914+0000 7f008fff5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:22:06.239 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:06.238+0000 7f490f7fe700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 msgr2=0x7f48f006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:06.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:06.238+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 0x7f48f006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T11:22:06.243 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:06.241+0000 7f987761f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:22:06.287 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:06.285+0000 7f008fff5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:22:06.814 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:06.813+0000 7f987761f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:22:06.858 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:06.857+0000 7f008fff5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:22:06.884 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:06.882+0000 7f987761f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:22:06.928 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:06.927+0000 7f008fff5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:22:07.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:07.039+0000 7f490f7fe700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 msgr2=0x7f48f006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:07.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:07.039+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 0x7f48f006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T11:22:07.515 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:07.513+0000 7f987761f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:07.561 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:07.559+0000 7f008fff5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:07.593 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:07.591+0000 7f987761f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:22:07.639 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:07.637+0000 7f008fff5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:22:07.737 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:07.736+0000 7f987761f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:07.785 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:07.783+0000 7f008fff5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:07.968 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:07.966+0000 7f987761f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:22:08.017 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:08.015+0000 7f008fff5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:22:08.042 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:08.041+0000 7f987761f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:22:08.093 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:08.090+0000 7f008fff5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:22:08.309 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:08.308+0000 7f987761f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:22:08.358 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:08.356+0000 7f008fff5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:22:08.533 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:08.532+0000 7f987761f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:22:08.583 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:08.582+0000 7f008fff5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:22:08.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:08.640+0000 7f490f7fe700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 msgr2=0x7f48f006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/778561374 2023-10-16T11:22:08.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:08.640+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 0x7f48f006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T11:22:08.684 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:08.682+0000 7f987761f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:22:08.736 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:08.734+0000 7f008fff5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:22:08.799 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:08.797+0000 7f987761f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:22:08.851 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:08.850+0000 7f008fff5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:22:08.955 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:08.953+0000 7f987761f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:22:09.012 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:09.010+0000 7f008fff5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:22:09.220 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:09.218+0000 7f987761f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:09.277 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:09.275+0000 7f008fff5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:09.381 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:09.379+0000 7f987761f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:22:09.440 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:09.439+0000 7f008fff5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:22:09.909 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:09.906+0000 7f987761f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:22:09.976 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:09.974+0000 7f008fff5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:22:09.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:09.993+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 168) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f48f8087780 con 0x7f4910071d00 2023-10-16T11:22:09.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:09.994+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 msgr2=0x7f48f006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:09.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:09.994+0000 7f490cff9700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/778561374,v1:172.21.15.158:6833/778561374] conn(0x7f48f006b070 0x7f48f006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:10.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.134+0000 7f0005fd6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0005fd6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0005fd6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.137 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0005fd6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0005fd6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0002fd0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0002fd0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0002fd0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.138 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0002fd0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.139 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:10.135+0000 7f0002fd0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:10.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:10.997+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 169) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f48f80544c0 con 0x7f4910071d00 2023-10-16T11:22:11.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:10.998+0000 7f490cff9700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f48f0035910 0x7f48f0075b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:11.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:10.998+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 --> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "a", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4910151a60 con 0x7f48f0035910 2023-10-16T11:22:11.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.000+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f48f0035910 0x7f48f0075b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:11.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.003+0000 7f490f7fe700 1 --2- 172.21.15.158:0/3403701627 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f48f0035910 0x7f48f0075b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4900009780 tx=0x7f4900009210 comp rx=0 tx=0).ready entity=mgr.13902 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:11.007 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:11.006+0000 7efffffca700 -1 mgr.server reply reply (2) No such file or directory FS 'a' not found 2023-10-16T11:22:11.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.006+0000 7f490cff9700 1 -- 172.21.15.158:0/3403701627 <== mgr.13902 v2:172.21.15.158:6832/224988149 1 ==== mgr_command_reply(tid 0: -2 FS 'a' not found) v1 ==== 24+0+0 (secure 0 0 0) 0x7f4910151a60 con 0x7f48f0035910 2023-10-16T11:22:11.010 INFO:teuthology.orchestra.run.smithi158.stderr:Error ENOENT: FS 'a' not found 2023-10-16T11:22:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T11:22:11.015 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T11:22:11.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.142+0000 7fe8f7615700 1 Processor -- start 2023-10-16T11:22:11.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.142+0000 7fe8f7615700 1 -- start start 2023-10-16T11:22:11.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f7615700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8f006d5d0 0x7fe8f0070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:11.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f7615700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:11.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f7615700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f015dd70 con 0x7fe8f015d990 2023-10-16T11:22:11.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f7615700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f01616b0 con 0x7fe8f006d5d0 2023-10-16T11:22:11.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f7615700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f006da50 con 0x7fe8f00cbaf0 2023-10-16T11:22:11.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8f006d5d0 0x7fe8f0070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:11.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8f006d5d0 0x7fe8f0070300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44286/0 (socket says 172.21.15.158:44286) 2023-10-16T11:22:11.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 -- 172.21.15.158:0/1460223949 learned_addr learned my addr 172.21.15.158:0/1460223949 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:11.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f5bb2700 1 --2- 172.21.15.158:0/1460223949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:11.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 -- 172.21.15.158:0/1460223949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 msgr2=0x7fe8f00cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:11.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 --2- 172.21.15.158:0/1460223949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:11.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 -- 172.21.15.158:0/1460223949 >> v1:172.21.15.172:6789/0 conn(0x7fe8f015d990 legacy=0x7fe8f0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:11.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 -- 172.21.15.158:0/1460223949 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0070a00 con 0x7fe8f006d5d0 2023-10-16T11:22:11.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f5bb2700 1 --2- 172.21.15.158:0/1460223949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:11.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8f4bb0700 1 --2- 172.21.15.158:0/1460223949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f006d5d0 0x7fe8f0070300 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009920 tx=0x7fe8ec009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160ad4b9e9fd0407 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:11.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8e7fff700 1 -- 172.21.15.158:0/1460223949 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec004230 con 0x7fe8f006d5d0 2023-10-16T11:22:11.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8e7fff700 1 -- 172.21.15.158:0/1460223949 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec004390 con 0x7fe8f006d5d0 2023-10-16T11:22:11.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.143+0000 7fe8e7fff700 1 -- 172.21.15.158:0/1460223949 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec009e70 con 0x7fe8f006d5d0 2023-10-16T11:22:11.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 -- 172.21.15.158:0/1460223949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f006d5d0 msgr2=0x7fe8f0070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:11.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 --2- 172.21.15.158:0/1460223949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f006d5d0 0x7fe8f0070300 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009920 tx=0x7fe8ec009c30 comp rx=0 tx=0).stop 2023-10-16T11:22:11.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 -- 172.21.15.158:0/1460223949 shutdown_connections 2023-10-16T11:22:11.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 --2- 172.21.15.158:0/1460223949 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:11.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 --2- 172.21.15.158:0/1460223949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f006d5d0 0x7fe8f0070300 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:11.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 -- 172.21.15.158:0/1460223949 >> 172.21.15.158:0/1460223949 conn(0x7fe8f00cdf20 msgr2=0x7fe8f00ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:11.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 -- 172.21.15.158:0/1460223949 shutdown_connections 2023-10-16T11:22:11.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.144+0000 7fe8f7615700 1 -- 172.21.15.158:0/1460223949 wait complete. 2023-10-16T11:22:11.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.145+0000 7fe8f7615700 1 Processor -- start 2023-10-16T11:22:11.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.145+0000 7fe8f7615700 1 -- start start 2023-10-16T11:22:11.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f7615700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8f006d5d0 0x7fe8f01f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:11.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f7615700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:11.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f7615700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f015d990 0x7fe8f00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:11.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f7615700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f01f6390 con 0x7fe8f015d990 2023-10-16T11:22:11.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f7615700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f00c5930 con 0x7fe8f006d5d0 2023-10-16T11:22:11.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f7615700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f00c5c10 con 0x7fe8f00cbaf0 2023-10-16T11:22:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f015d990 0x7fe8f00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f015d990 0x7fe8f00b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44294/0 (socket says 172.21.15.158:44294) 2023-10-16T11:22:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 -- 172.21.15.158:0/2516446065 learned_addr learned my addr 172.21.15.158:0/2516446065 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f4bb0700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:11.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f53b1700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8f006d5d0 0x7fe8f01f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 -- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 msgr2=0x7fe8f00b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 -- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8f006d5d0 msgr2=0x7fe8f01f5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8f006d5d0 0x7fe8f01f5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 -- 172.21.15.158:0/2516446065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec009550 con 0x7fe8f015d990 2023-10-16T11:22:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f53b1700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8f006d5d0 0x7fe8f01f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8f5bb2700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f015d990 0x7fe8f00b2600 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800cef0 tx=0x7fe8e800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.146+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e800adb0 con 0x7fe8f015d990 2023-10-16T11:22:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.147+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e8015440 con 0x7fe8f015d990 2023-10-16T11:22:11.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.147+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f00c5ef0 con 0x7fe8f015d990 2023-10-16T11:22:11.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.147+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e80157f0 con 0x7fe8f015d990 2023-10-16T11:22:11.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.147+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f00c8600 con 0x7fe8f015d990 2023-10-16T11:22:11.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.148+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f00c6410 con 0x7fe8f015d990 2023-10-16T11:22:11.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.148+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 169) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe8e800d430 con 0x7fe8f015d990 2023-10-16T11:22:11.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.148+0000 7fe8e5ffb700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe8d406ada0 0x7fe8d406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:11.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.149+0000 7fe8f53b1700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe8d406ada0 0x7fe8d406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:11.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.149+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fe8e8089c30 con 0x7fe8f015d990 2023-10-16T11:22:11.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.149+0000 7fe8f53b1700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe8d406ada0 0x7fe8d406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0003fc0 tx=0x7fe8e0009040 comp rx=0 tx=0).ready entity=mgr.13902 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:11.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.153+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8f00c6410 con 0x7fe8f015d990 2023-10-16T11:22:11.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.400+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe8f00c6410 con 0x7fe8f015d990 2023-10-16T11:22:12.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:11.998+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v170) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe8f00c6410 con 0x7fe8f015d990 2023-10-16T11:22:12.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.003+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe8d406ada0 msgr2=0x7fe8d406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.003+0000 7fe8f7615700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe8d406ada0 0x7fe8d406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0003fc0 tx=0x7fe8e0009040 comp rx=0 tx=0).stop 2023-10-16T11:22:12.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.003+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f015d990 msgr2=0x7fe8f00b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.003+0000 7fe8e5ffb700 1 -- 172.21.15.158:0/2516446065 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mgrmap(e 170) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe8e8054f70 con 0x7fe8f015d990 2023-10-16T11:22:12.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.003+0000 7fe8f7615700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f015d990 0x7fe8f00b2600 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800cef0 tx=0x7fe8e800cf20 comp rx=0 tx=0).stop 2023-10-16T11:22:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 shutdown_connections 2023-10-16T11:22:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8f015d990 0x7fe8f00b2600 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe8d406ada0 0x7fe8d406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8f006d5d0 0x7fe8f01f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 --2- 172.21.15.158:0/2516446065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8f00cbaf0 0x7fe8f00b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 >> 172.21.15.158:0/2516446065 conn(0x7fe8f00cdf20 msgr2=0x7fe8f006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 shutdown_connections 2023-10-16T11:22:12.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.004+0000 7fe8f7615700 1 -- 172.21.15.158:0/2516446065 wait complete. 2023-10-16T11:22:12.024 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T11:22:12.024 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:22:12.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T11:22:12.115 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T11:22:12.116 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:22:12.116 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:12.135 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:22:12.135 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:12.152 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:12.152+0000 7fe1bd246200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:22:12.157 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:12.156+0000 7f8cd48b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T11:22:12.176 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:22:12.218 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:12.217+0000 7fe1bd246200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:12.228 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:12.226+0000 7f8cd48b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:12.291 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:12.291+0000 7fe1bd246200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:22:12.303 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:12.301+0000 7f8cd48b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T11:22:12.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.339+0000 7f0795299700 1 Processor -- start 2023-10-16T11:22:12.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.339+0000 7f0795299700 1 -- start start 2023-10-16T11:22:12.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.339+0000 7f0795299700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f079015db90 0x7f079006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.339+0000 7f0795299700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f07900cddb0 0x7f07900ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.339+0000 7f0795299700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07900d0fb0 con 0x7f079015d7b0 2023-10-16T11:22:12.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.339+0000 7f0795299700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079015e100 con 0x7f079015db90 2023-10-16T11:22:12.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.339+0000 7f0795299700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079015e3e0 con 0x7f07900cddb0 2023-10-16T11:22:12.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078e7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f079015db90 0x7f079006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f079015d7b0 0x7f07900b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44544/0 (socket says 172.21.15.158:44544) 2023-10-16T11:22:12.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078effd700 1 -- 172.21.15.158:0/3920695913 learned_addr learned my addr 172.21.15.158:0/3920695913 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:12.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078f7fe700 1 --2- 172.21.15.158:0/3920695913 >> v2:172.21.15.172:3300/0 conn(0x7f07900cddb0 0x7f07900ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078e7fc700 1 -- 172.21.15.158:0/3920695913 >> v2:172.21.15.172:3300/0 conn(0x7f07900cddb0 msgr2=0x7f07900ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:12.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078e7fc700 1 --2- 172.21.15.158:0/3920695913 >> v2:172.21.15.172:3300/0 conn(0x7f07900cddb0 0x7f07900ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078e7fc700 1 -- 172.21.15.158:0/3920695913 >> v1:172.21.15.158:6789/0 conn(0x7f079015d7b0 legacy=0x7f07900b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078dffb700 1 -- 172.21.15.158:0/3920695913 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199223643 0 0) 0x7f07900d0fb0 con 0x7f079015d7b0 2023-10-16T11:22:12.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078e7fc700 1 -- 172.21.15.158:0/3920695913 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079015e6c0 con 0x7f079015db90 2023-10-16T11:22:12.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078e7fc700 1 --2- 172.21.15.158:0/3920695913 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f079015db90 0x7f079006b9e0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7f07900b0e10 tx=0x7f0784009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562af7e5654777d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:12.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078dffb700 1 -- 172.21.15.158:0/3920695913 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784011830 con 0x7f079015db90 2023-10-16T11:22:12.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078dffb700 1 -- 172.21.15.158:0/3920695913 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0784011990 con 0x7f079015db90 2023-10-16T11:22:12.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.340+0000 7f078dffb700 1 -- 172.21.15.158:0/3920695913 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784011d20 con 0x7f079015db90 2023-10-16T11:22:12.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 -- 172.21.15.158:0/3920695913 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f079015db90 msgr2=0x7f079006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 --2- 172.21.15.158:0/3920695913 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f079015db90 0x7f079006b9e0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7f07900b0e10 tx=0x7f0784009b20 comp rx=0 tx=0).stop 2023-10-16T11:22:12.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 -- 172.21.15.158:0/3920695913 shutdown_connections 2023-10-16T11:22:12.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 --2- 172.21.15.158:0/3920695913 >> v2:172.21.15.172:3300/0 conn(0x7f07900cddb0 0x7f07900ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 --2- 172.21.15.158:0/3920695913 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f079015db90 0x7f079006b9e0 unknown :-1 s=CLOSED pgs=3348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 -- 172.21.15.158:0/3920695913 >> 172.21.15.158:0/3920695913 conn(0x7f0790071220 msgr2=0x7f079006a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:12.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 -- 172.21.15.158:0/3920695913 shutdown_connections 2023-10-16T11:22:12.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.341+0000 7f0795299700 1 -- 172.21.15.158:0/3920695913 wait complete. 2023-10-16T11:22:12.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.343+0000 7f0795299700 1 Processor -- start 2023-10-16T11:22:12.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.343+0000 7f0795299700 1 -- start start 2023-10-16T11:22:12.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f0795299700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07900cddb0 0x7f07900c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f0795299700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f079015d7b0 0x7f0790146900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f0795299700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f079015db90 0x7f07900c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f0795299700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07901473e0 con 0x7f07900cddb0 2023-10-16T11:22:12.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f0795299700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900c56f0 con 0x7f079015db90 2023-10-16T11:22:12.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f0795299700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07900c5130 con 0x7f079015d7b0 2023-10-16T11:22:12.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07900cddb0 0x7f07900c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07900cddb0 0x7f07900c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44374/0 (socket says 172.21.15.158:44374) 2023-10-16T11:22:12.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 -- 172.21.15.158:0/2704662077 learned_addr learned my addr 172.21.15.158:0/2704662077 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:12.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078e7fc700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f079015d7b0 0x7f0790146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078f7fe700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f079015db90 0x7f07900c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 -- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f079015d7b0 msgr2=0x7f0790146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f079015d7b0 0x7f0790146900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 -- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f079015db90 msgr2=0x7f07900c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f079015db90 0x7f07900c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 -- 172.21.15.158:0/2704662077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07840094a0 con 0x7f07900cddb0 2023-10-16T11:22:12.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f078effd700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07900cddb0 0x7f07900c4b50 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f07800011e0 tx=0x7f078000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:12.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f07777fe700 1 -- 172.21.15.158:0/2704662077 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780004c80 con 0x7f07900cddb0 2023-10-16T11:22:12.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f07777fe700 1 -- 172.21.15.158:0/2704662077 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0780004e00 con 0x7f07900cddb0 2023-10-16T11:22:12.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07900c5410 con 0x7f07900cddb0 2023-10-16T11:22:12.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.344+0000 7f07777fe700 1 -- 172.21.15.158:0/2704662077 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780007310 con 0x7f07900cddb0 2023-10-16T11:22:12.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.345+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07900c8200 con 0x7f07900cddb0 2023-10-16T11:22:12.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.345+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07900681b0 con 0x7f07900cddb0 2023-10-16T11:22:12.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.346+0000 7f07777fe700 1 -- 172.21.15.158:0/2704662077 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 170) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0780007470 con 0x7f07900cddb0 2023-10-16T11:22:12.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.346+0000 7f07777fe700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 0x7f077c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.346+0000 7f078e7fc700 1 -- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 msgr2=0x7f077c06d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:12.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.346+0000 7f07777fe700 1 -- 172.21.15.158:0/2704662077 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f0780089390 con 0x7f07900cddb0 2023-10-16T11:22:12.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.346+0000 7f078e7fc700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 0x7f077c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:12.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.351+0000 7f07777fe700 1 -- 172.21.15.158:0/2704662077 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0780054280 con 0x7f07900cddb0 2023-10-16T11:22:12.369 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:12.369+0000 7fe1bd246200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:22:12.386 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:12.385+0000 7f8cd48b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T11:22:12.510 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:12.510+0000 7fe1bd246200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:22:12.530 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:12.529+0000 7f8cd48b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T11:22:12.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.547+0000 7f078e7fc700 1 -- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 msgr2=0x7f077c06d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:12.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.547+0000 7f078e7fc700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 0x7f077c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:12.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.552+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07900b7390 con 0x7f07900cddb0 2023-10-16T11:22:12.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.553+0000 7f07777fe700 1 -- 172.21.15.158:0/2704662077 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+17979 (secure 0 0 0) 0x7f07800578a0 con 0x7f07900cddb0 2023-10-16T11:22:12.555 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:12.556 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":271,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:22:09.988038+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:54.523055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:55.050879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3856677823":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/1867601061":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/933196924":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/3733531003":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6833/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6832/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/915172958":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/258064180":"2023-10-17T11:22:09.987990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:22:12.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 msgr2=0x7f077c06d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:12.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 0x7f077c06d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07900cddb0 msgr2=0x7f07900c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07900cddb0 0x7f07900c4b50 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f07800011e0 tx=0x7f078000ca30 comp rx=0 tx=0).stop 2023-10-16T11:22:12.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 shutdown_connections 2023-10-16T11:22:12.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f077c06ad50 0x7f077c06d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f079015db90 0x7f07900c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f07900cddb0 0x7f07900c4b50 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 --2- 172.21.15.158:0/2704662077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f079015d7b0 0x7f0790146900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 >> 172.21.15.158:0/2704662077 conn(0x7f0790071220 msgr2=0x7f07900b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:12.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 shutdown_connections 2023-10-16T11:22:12.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.556+0000 7f0795299700 1 -- 172.21.15.158:0/2704662077 wait complete. 2023-10-16T11:22:12.570 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:21:54.523055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '269', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:21:55.050879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '270', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:22:12.571 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:22:12.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.737+0000 7fab31bc9700 1 Processor -- start 2023-10-16T11:22:12.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.737+0000 7fab31bc9700 1 -- start start 2023-10-16T11:22:12.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.737+0000 7fab31bc9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fab2c15b540 0x7fab2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.737+0000 7fab31bc9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fab2c06afe0 0x7fab2c161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.737+0000 7fab31bc9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2c15ee60 con 0x7fab2c06f180 2023-10-16T11:22:12.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.737+0000 7fab31bc9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c15b920 con 0x7fab2c06afe0 2023-10-16T11:22:12.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.737+0000 7fab31bc9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c15e2e0 con 0x7fab2c15b540 2023-10-16T11:22:12.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fab2c06afe0 0x7fab2c161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fab2c06f180 0x7fab2c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44596/0 (socket says 172.21.15.158:44596) 2023-10-16T11:22:12.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2affd700 1 -- 172.21.15.158:0/1335714251 learned_addr learned my addr 172.21.15.158:0/1335714251 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:12.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2b7fe700 1 --2- 172.21.15.158:0/1335714251 >> v2:172.21.15.172:3300/0 conn(0x7fab2c15b540 0x7fab2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2bfff700 1 -- 172.21.15.158:0/1335714251 >> v2:172.21.15.172:3300/0 conn(0x7fab2c15b540 msgr2=0x7fab2c06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:12.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2bfff700 1 --2- 172.21.15.158:0/1335714251 >> v2:172.21.15.172:3300/0 conn(0x7fab2c15b540 0x7fab2c06d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2bfff700 1 -- 172.21.15.158:0/1335714251 >> v1:172.21.15.158:6789/0 conn(0x7fab2c06f180 legacy=0x7fab2c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2bfff700 1 -- 172.21.15.158:0/1335714251 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c15e5c0 con 0x7fab2c06afe0 2023-10-16T11:22:12.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2a7fc700 1 -- 172.21.15.158:0/1335714251 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232247722 0 0) 0x7fab2c15ee60 con 0x7fab2c06f180 2023-10-16T11:22:12.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2bfff700 1 --2- 172.21.15.158:0/1335714251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06afe0 0x7fab2c161860 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fab2c15ba60 tx=0x7fab2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f952c2236bde61 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:12.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2a7fc700 1 -- 172.21.15.158:0/1335714251 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24016070 con 0x7fab2c06afe0 2023-10-16T11:22:12.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2a7fc700 1 -- 172.21.15.158:0/1335714251 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2400baf0 con 0x7fab2c06afe0 2023-10-16T11:22:12.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.738+0000 7fab2a7fc700 1 -- 172.21.15.158:0/1335714251 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2400e070 con 0x7fab2c06afe0 2023-10-16T11:22:12.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 -- 172.21.15.158:0/1335714251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06afe0 msgr2=0x7fab2c161860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 --2- 172.21.15.158:0/1335714251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06afe0 0x7fab2c161860 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fab2c15ba60 tx=0x7fab2400b330 comp rx=0 tx=0).stop 2023-10-16T11:22:12.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 -- 172.21.15.158:0/1335714251 shutdown_connections 2023-10-16T11:22:12.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 --2- 172.21.15.158:0/1335714251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06afe0 0x7fab2c161860 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 --2- 172.21.15.158:0/1335714251 >> v2:172.21.15.172:3300/0 conn(0x7fab2c15b540 0x7fab2c06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 -- 172.21.15.158:0/1335714251 >> 172.21.15.158:0/1335714251 conn(0x7fab2c0d03b0 msgr2=0x7fab2c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:12.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 -- 172.21.15.158:0/1335714251 shutdown_connections 2023-10-16T11:22:12.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.739+0000 7fab31bc9700 1 -- 172.21.15.158:0/1335714251 wait complete. 2023-10-16T11:22:12.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.740+0000 7fab31bc9700 1 Processor -- start 2023-10-16T11:22:12.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.740+0000 7fab31bc9700 1 -- start start 2023-10-16T11:22:12.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab31bc9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab2c06afe0 0x7fab2c0ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab31bc9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06f180 0x7fab2c0ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab31bc9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab2c15b540 0x7fab2c1f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab31bc9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c147600 con 0x7fab2c06f180 2023-10-16T11:22:12.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab31bc9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c148140 con 0x7fab2c06afe0 2023-10-16T11:22:12.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab31bc9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c147810 con 0x7fab2c15b540 2023-10-16T11:22:12.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab2affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06f180 0x7fab2c0ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab2affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06f180 0x7fab2c0ad250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44426/0 (socket says 172.21.15.158:44426) 2023-10-16T11:22:12.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab2affd700 1 -- 172.21.15.158:0/3169990469 learned_addr learned my addr 172.21.15.158:0/3169990469 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:12.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab2b7fe700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab2c06afe0 0x7fab2c0ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.741+0000 7fab2bfff700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab2c15b540 0x7fab2c1f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:12.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab2affd700 1 -- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab2c15b540 msgr2=0x7fab2c1f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab2affd700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab2c15b540 0x7fab2c1f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab2affd700 1 -- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab2c06afe0 msgr2=0x7fab2c0ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab2affd700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab2c06afe0 0x7fab2c0ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab2affd700 1 -- 172.21.15.158:0/3169990469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab24009a70 con 0x7fab2c06f180 2023-10-16T11:22:12.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab2affd700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06f180 0x7fab2c0ad250 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7fab200011e0 tx=0x7fab2000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:12.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab13fff700 1 -- 172.21.15.158:0/3169990469 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab20004c80 con 0x7fab2c06f180 2023-10-16T11:22:12.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab13fff700 1 -- 172.21.15.158:0/3169990469 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab20004e00 con 0x7fab2c06f180 2023-10-16T11:22:12.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab13fff700 1 -- 172.21.15.158:0/3169990469 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab20007310 con 0x7fab2c06f180 2023-10-16T11:22:12.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c147af0 con 0x7fab2c06f180 2023-10-16T11:22:12.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.742+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c0ad790 con 0x7fab2c06f180 2023-10-16T11:22:12.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.743+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c0681b0 con 0x7fab2c06f180 2023-10-16T11:22:12.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.743+0000 7fab13fff700 1 -- 172.21.15.158:0/3169990469 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 170) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fab20007470 con 0x7fab2c06f180 2023-10-16T11:22:12.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.744+0000 7fab13fff700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 0x7fab1806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:12.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.744+0000 7fab2b7fe700 1 -- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 msgr2=0x7fab1806d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:12.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.744+0000 7fab13fff700 1 -- 172.21.15.158:0/3169990469 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fab200893e0 con 0x7fab2c06f180 2023-10-16T11:22:12.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.744+0000 7fab2b7fe700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 0x7fab1806d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:12.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.748+0000 7fab13fff700 1 -- 172.21.15.158:0/3169990469 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab20054290 con 0x7fab2c06f180 2023-10-16T11:22:12.838 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:12.837+0000 7fe1bd246200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:22:12.872 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:12.870+0000 7f8cd48b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T11:22:12.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.944+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab2c148280 con 0x7fab2c06f180 2023-10-16T11:22:12.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.944+0000 7fab13fff700 1 -- 172.21.15.158:0/3169990469 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v271) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab200578b0 con 0x7fab2c06f180 2023-10-16T11:22:12.946 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:22:12.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.945+0000 7fab2b7fe700 1 -- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 msgr2=0x7fab1806d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:12.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.945+0000 7fab2b7fe700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 0x7fab1806d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:12.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 msgr2=0x7fab1806d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:12.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 0x7fab1806d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06f180 msgr2=0x7fab2c0ad250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:12.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06f180 0x7fab2c0ad250 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7fab200011e0 tx=0x7fab2000ca30 comp rx=0 tx=0).stop 2023-10-16T11:22:12.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 shutdown_connections 2023-10-16T11:22:12.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab2c15b540 0x7fab2c1f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fab1806ad50 0x7fab1806d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab2c06afe0 0x7fab2c0ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 --2- 172.21.15.158:0/3169990469 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab2c06f180 0x7fab2c0ad250 unknown :-1 s=CLOSED pgs=3352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:12.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 >> 172.21.15.158:0/3169990469 conn(0x7fab2c0d03b0 msgr2=0x7fab2c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:12.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 shutdown_connections 2023-10-16T11:22:12.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:12.948+0000 7fab31bc9700 1 -- 172.21.15.158:0/3169990469 wait complete. 2023-10-16T11:22:12.955 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:12.954+0000 7fe1bd246200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:22:12.967 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:22:12.999 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:12.997+0000 7f8cd48b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T11:22:13.031 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:13.031+0000 7fe1bd246200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:22:13.078 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:13.077+0000 7f8cd48b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T11:22:13.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.095+0000 7f8fa6415700 1 Processor -- start 2023-10-16T11:22:13.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.095+0000 7f8fa6415700 1 -- start start 2023-10-16T11:22:13.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8fa6415700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa015daa0 0x7f8fa00cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8fa6415700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00d1ee0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8fa6415700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00d12d0 con 0x7f8fa00d0640 2023-10-16T11:22:13.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8fa6415700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00d40a0 con 0x7f8fa015daa0 2023-10-16T11:22:13.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9ffff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8fa015d6c0 0x7f8fa00cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44610/0 (socket says 172.21.15.158:44610) 2023-10-16T11:22:13.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9ffff700 1 -- 172.21.15.158:0/2882409582 learned_addr learned my addr 172.21.15.158:0/2882409582 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:13.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9f7fe700 1 --2- 172.21.15.158:0/2882409582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa015daa0 0x7f8fa00cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9effd700 1 -- 172.21.15.158:0/2882409582 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355138253 0 0) 0x7f8fa00d1ee0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9effd700 1 -- 172.21.15.158:0/2882409582 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f880035b0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9effd700 1 -- 172.21.15.158:0/2882409582 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758294969 0 0) 0x7f8fa00d12d0 con 0x7f8fa00d0640 2023-10-16T11:22:13.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9effd700 1 -- 172.21.15.158:0/2882409582 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa00d1ee0 con 0x7f8fa00d0640 2023-10-16T11:22:13.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9f7fe700 1 -- 172.21.15.158:0/2882409582 >> v1:172.21.15.172:6789/0 conn(0x7f8fa00d0640 legacy=0x7f8fa006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9f7fe700 1 -- 172.21.15.158:0/2882409582 >> v1:172.21.15.158:6789/0 conn(0x7f8fa015d6c0 legacy=0x7f8fa00cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9f7fe700 1 -- 172.21.15.158:0/2882409582 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00d15b0 con 0x7f8fa015daa0 2023-10-16T11:22:13.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.096+0000 7f8f9f7fe700 1 --2- 172.21.15.158:0/2882409582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa015daa0 0x7f8fa00cec50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009870 tx=0x7f8f94009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9269323ea720e34a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:13.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8f9effd700 1 -- 172.21.15.158:0/2882409582 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94011830 con 0x7f8fa015daa0 2023-10-16T11:22:13.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8f9effd700 1 -- 172.21.15.158:0/2882409582 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f94011990 con 0x7f8fa015daa0 2023-10-16T11:22:13.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8f9effd700 1 -- 172.21.15.158:0/2882409582 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94011d20 con 0x7f8fa015daa0 2023-10-16T11:22:13.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8fa6415700 1 -- 172.21.15.158:0/2882409582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa015daa0 msgr2=0x7f8fa00cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8fa6415700 1 --2- 172.21.15.158:0/2882409582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa015daa0 0x7f8fa00cec50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009870 tx=0x7f8f94009b80 comp rx=0 tx=0).stop 2023-10-16T11:22:13.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8fa6415700 1 -- 172.21.15.158:0/2882409582 shutdown_connections 2023-10-16T11:22:13.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8fa6415700 1 --2- 172.21.15.158:0/2882409582 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa015daa0 0x7f8fa00cec50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8fa6415700 1 -- 172.21.15.158:0/2882409582 >> 172.21.15.158:0/2882409582 conn(0x7f8fa009c940 msgr2=0x7f8fa00d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:13.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8fa6415700 1 -- 172.21.15.158:0/2882409582 shutdown_connections 2023-10-16T11:22:13.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.097+0000 7f8fa6415700 1 -- 172.21.15.158:0/2882409582 wait complete. 2023-10-16T11:22:13.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.098+0000 7f8fa6415700 1 Processor -- start 2023-10-16T11:22:13.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8fa6415700 1 -- start start 2023-10-16T11:22:13.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8fa6415700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa00d0640 0x7f8fa015fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8fa6415700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa015d6c0 0x7f8fa00c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8fa6415700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fa015daa0 0x7f8fa00b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8fa6415700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0160470 con 0x7f8fa015d6c0 2023-10-16T11:22:13.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8fa6415700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0160750 con 0x7f8fa015daa0 2023-10-16T11:22:13.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8fa6415700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0160a30 con 0x7f8fa00d0640 2023-10-16T11:22:13.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8f9f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa015d6c0 0x7f8fa00c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.099+0000 7f8f9f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa015d6c0 0x7f8fa00c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44442/0 (socket says 172.21.15.158:44442) 2023-10-16T11:22:13.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9f7fe700 1 -- 172.21.15.158:0/1322686271 learned_addr learned my addr 172.21.15.158:0/1322686271 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:13.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9ffff700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa00d0640 0x7f8fa015fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8fa49b2700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fa015daa0 0x7f8fa00b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9f7fe700 1 -- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa00d0640 msgr2=0x7f8fa015fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9f7fe700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa00d0640 0x7f8fa015fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9f7fe700 1 -- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fa015daa0 msgr2=0x7f8fa00b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9f7fe700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fa015daa0 0x7f8fa00b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9f7fe700 1 -- 172.21.15.158:0/1322686271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f940094a0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9f7fe700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa015d6c0 0x7f8fa00c99a0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f8f940090b0 tx=0x7f8f94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:13.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9cff9700 1 -- 172.21.15.158:0/1322686271 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9401a6e0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9cff9700 1 -- 172.21.15.158:0/1322686271 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9401a840 con 0x7f8fa015d6c0 2023-10-16T11:22:13.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8f9cff9700 1 -- 172.21.15.158:0/1322686271 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9401abb0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa00b6290 con 0x7f8fa015d6c0 2023-10-16T11:22:13.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.100+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa00c9ee0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.101+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa00681b0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.102+0000 7f8f9cff9700 1 -- 172.21.15.158:0/1322686271 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8f94022070 con 0x7f8fa015d6c0 2023-10-16T11:22:13.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.102+0000 7f8f9cff9700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 0x7f8f8c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.102+0000 7f8f9ffff700 1 -- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 msgr2=0x7f8f8c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:13.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.102+0000 7f8f9cff9700 1 -- 172.21.15.158:0/1322686271 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f8f9408a6c0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.102+0000 7f8f9ffff700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 0x7f8f8c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:13.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.107+0000 7f8f9cff9700 1 -- 172.21.15.158:0/1322686271 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f940552a0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.112 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:13.111+0000 7fe1bd246200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:22:13.164 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:13.163+0000 7f8cd48b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T11:22:13.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.303+0000 7f8f9ffff700 1 -- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 msgr2=0x7f8f8c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:13.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.303+0000 7f8f9ffff700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 0x7f8f8c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:13.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.306+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8fa00d0d30 con 0x7f8fa015d6c0 2023-10-16T11:22:13.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.307+0000 7f8f9cff9700 1 -- 172.21.15.158:0/1322686271 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v271) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8f940588c0 con 0x7f8fa015d6c0 2023-10-16T11:22:13.309 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:22:13.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 msgr2=0x7f8f8c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:13.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 0x7f8f8c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa015d6c0 msgr2=0x7f8fa00c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa015d6c0 0x7f8fa00c99a0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f8f940090b0 tx=0x7f8f94004750 comp rx=0 tx=0).stop 2023-10-16T11:22:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 shutdown_connections 2023-10-16T11:22:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f8f8c06b020 0x7f8f8c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fa015daa0 0x7f8fa00b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8fa00d0640 0x7f8fa015fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 --2- 172.21.15.158:0/1322686271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa015d6c0 0x7f8fa00c99a0 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 >> 172.21.15.158:0/1322686271 conn(0x7f8fa009c940 msgr2=0x7f8fa00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 shutdown_connections 2023-10-16T11:22:13.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.309+0000 7f8fa6415700 1 -- 172.21.15.158:0/1322686271 wait complete. 2023-10-16T11:22:13.324 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:22:13.483 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:13.482+0000 7fe1bd246200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:22:13.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.488+0000 7f1011f3e700 1 Processor -- start 2023-10-16T11:22:13.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.488+0000 7f1011f3e700 1 -- start start 2023-10-16T11:22:13.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f1011f3e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f100c15dc40 0x7f100c15ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f1011f3e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06d9c0 0x7f100c06de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f1011f3e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c15f380 con 0x7f100c06f180 2023-10-16T11:22:13.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f1011f3e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c06e380 con 0x7f100c15dc40 2023-10-16T11:22:13.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f1011f3e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c071e70 con 0x7f100c06d9c0 2023-10-16T11:22:13.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06d9c0 0x7f100c06de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f100c15dc40 0x7f100c15ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06d9c0 0x7f100c06de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:55312/0 (socket says 172.21.15.158:55312) 2023-10-16T11:22:13.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f100c15dc40 0x7f100c15ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:45656/0 (socket says 172.21.15.158:45656) 2023-10-16T11:22:13.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100affd700 1 -- 172.21.15.158:0/628560858 learned_addr learned my addr 172.21.15.158:0/628560858 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:13.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100affd700 1 -- 172.21.15.158:0/628560858 >> v2:172.21.15.172:3300/0 conn(0x7f100c15dc40 msgr2=0x7f100c15ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:13.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100affd700 1 --2- 172.21.15.158:0/628560858 >> v2:172.21.15.172:3300/0 conn(0x7f100c15dc40 0x7f100c15ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100affd700 1 -- 172.21.15.158:0/628560858 >> v1:172.21.15.172:6789/0 conn(0x7f100c06f180 legacy=0x7f100c06f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100affd700 1 -- 172.21.15.158:0/628560858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c072150 con 0x7f100c06d9c0 2023-10-16T11:22:13.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.489+0000 7f100b7fe700 1 --2- 172.21.15.158:0/628560858 >> v2:172.21.15.172:3300/0 conn(0x7f100c15dc40 0x7f100c15ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:13.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f100affd700 1 --2- 172.21.15.158:0/628560858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06d9c0 0x7f100c06de40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f10000098f0 tx=0x7f1000009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d55d35065f5c0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:13.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f100a7fc700 1 -- 172.21.15.158:0/628560858 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000011830 con 0x7f100c06d9c0 2023-10-16T11:22:13.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f100a7fc700 1 -- 172.21.15.158:0/628560858 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1000011990 con 0x7f100c06d9c0 2023-10-16T11:22:13.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f100a7fc700 1 -- 172.21.15.158:0/628560858 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000011d20 con 0x7f100c06d9c0 2023-10-16T11:22:13.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f1011f3e700 1 -- 172.21.15.158:0/628560858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06d9c0 msgr2=0x7f100c06de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f1011f3e700 1 --2- 172.21.15.158:0/628560858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06d9c0 0x7f100c06de40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f10000098f0 tx=0x7f1000009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:13.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f1011f3e700 1 -- 172.21.15.158:0/628560858 shutdown_connections 2023-10-16T11:22:13.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f1011f3e700 1 --2- 172.21.15.158:0/628560858 >> v2:172.21.15.172:3300/0 conn(0x7f100c15dc40 0x7f100c15ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f1011f3e700 1 --2- 172.21.15.158:0/628560858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06d9c0 0x7f100c06de40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.490+0000 7f1011f3e700 1 -- 172.21.15.158:0/628560858 >> 172.21.15.158:0/628560858 conn(0x7f100c0d03b0 msgr2=0x7f100c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:13.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.491+0000 7f1011f3e700 1 -- 172.21.15.158:0/628560858 shutdown_connections 2023-10-16T11:22:13.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.491+0000 7f1011f3e700 1 -- 172.21.15.158:0/628560858 wait complete. 2023-10-16T11:22:13.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.492+0000 7f1011f3e700 1 Processor -- start 2023-10-16T11:22:13.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.492+0000 7f1011f3e700 1 -- start start 2023-10-16T11:22:13.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f1011f3e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f100c06d9c0 0x7f100c0c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f1011f3e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06f180 0x7f100c0cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f1011f3e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c15dc40 0x7f100c0cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f1011f3e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0c86a0 con 0x7f100c15dc40 2023-10-16T11:22:13.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f1011f3e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0c9220 con 0x7f100c06d9c0 2023-10-16T11:22:13.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f1011f3e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0c6d10 con 0x7f100c06f180 2023-10-16T11:22:13.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f100b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f100c06d9c0 0x7f100c0c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f100affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06f180 0x7f100c0cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f100bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c15dc40 0x7f100c0cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f100b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f100c06d9c0 0x7f100c0c67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:45662/0 (socket says 172.21.15.158:45662) 2023-10-16T11:22:13.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f100affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06f180 0x7f100c0cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:55318/0 (socket says 172.21.15.158:55318) 2023-10-16T11:22:13.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.493+0000 7f100b7fe700 1 -- 172.21.15.158:0/2302582020 learned_addr learned my addr 172.21.15.158:0/2302582020 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:13.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f100b7fe700 1 -- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06f180 msgr2=0x7f100c0cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f100b7fe700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06f180 0x7f100c0cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f100b7fe700 1 -- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c15dc40 msgr2=0x7f100c0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f100b7fe700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c15dc40 0x7f100c0cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f100b7fe700 1 -- 172.21.15.158:0/2302582020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0094a0 con 0x7f100c06d9c0 2023-10-16T11:22:13.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f100affd700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06f180 0x7f100c0cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:13.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f100b7fe700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f100c06d9c0 0x7f100c0c67d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00ebe0 tx=0x7f0ffc00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:13.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f0ff3fff700 1 -- 172.21.15.158:0/2302582020 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc009760 con 0x7f100c06d9c0 2023-10-16T11:22:13.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f0ff3fff700 1 -- 172.21.15.158:0/2302582020 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc00c850 con 0x7f100c06d9c0 2023-10-16T11:22:13.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1000009550 con 0x7f100c06d9c0 2023-10-16T11:22:13.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f0ff3fff700 1 -- 172.21.15.158:0/2302582020 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc00cbe0 con 0x7f100c06d9c0 2023-10-16T11:22:13.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.494+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c0c8bd0 con 0x7f100c06d9c0 2023-10-16T11:22:13.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.495+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c0c9500 con 0x7f100c06d9c0 2023-10-16T11:22:13.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.496+0000 7f0ff3fff700 1 -- 172.21.15.158:0/2302582020 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0ffc00cd40 con 0x7f100c06d9c0 2023-10-16T11:22:13.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.496+0000 7f0ff3fff700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 0x7f0ff806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.496+0000 7f100affd700 1 -- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 msgr2=0x7f0ff806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:13.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.496+0000 7f0ff3fff700 1 -- 172.21.15.158:0/2302582020 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f0ffc089b70 con 0x7f100c06d9c0 2023-10-16T11:22:13.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.496+0000 7f100affd700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 0x7f0ff806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:13.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.500+0000 7f0ff3fff700 1 -- 172.21.15.158:0/2302582020 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ffc054720 con 0x7f100c06d9c0 2023-10-16T11:22:13.554 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:13.552+0000 7f8cd48b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T11:22:13.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.697+0000 7f100affd700 1 -- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 msgr2=0x7f0ff806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:13.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.697+0000 7f100affd700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 0x7f0ff806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:13.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.699+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f100c0681b0 con 0x7f100c06d9c0 2023-10-16T11:22:13.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.700+0000 7f0ff3fff700 1 -- 172.21.15.158:0/2302582020 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v271) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ffc0210a0 con 0x7f100c06d9c0 2023-10-16T11:22:13.702 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:22:13.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 msgr2=0x7f0ff806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:13.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 0x7f0ff806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f100c06d9c0 msgr2=0x7f100c0c67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f100c06d9c0 0x7f100c0c67d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00ebe0 tx=0x7f0ffc00eef0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 shutdown_connections 2023-10-16T11:22:13.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c15dc40 0x7f100c0cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f0ff806b020 0x7f0ff806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f100c06d9c0 0x7f100c0c67d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 --2- 172.21.15.158:0/2302582020 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f100c06f180 0x7f100c0cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 >> 172.21.15.158:0/2302582020 conn(0x7f100c0d03b0 msgr2=0x7f100c158980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:13.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 shutdown_connections 2023-10-16T11:22:13.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.702+0000 7f1011f3e700 1 -- 172.21.15.158:0/2302582020 wait complete. 2023-10-16T11:22:13.718 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T11:22:13.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.840+0000 7efd7a741700 1 Processor -- start 2023-10-16T11:22:13.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.840+0000 7efd7a741700 1 -- start start 2023-10-16T11:22:13.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd7a741700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efd7406afe0 0x7efd740719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd7a741700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd74071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd7a741700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7415dc50 con 0x7efd7415d870 2023-10-16T11:22:13.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd7a741700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd740701a0 con 0x7efd7406afe0 2023-10-16T11:22:13.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd7a741700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7406f100 con 0x7efd740cf7c0 2023-10-16T11:22:13.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd73fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efd7415d870 0x7efd7406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44668/0 (socket says 172.21.15.158:44668) 2023-10-16T11:22:13.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd73fff700 1 -- 172.21.15.158:0/3287149097 learned_addr learned my addr 172.21.15.158:0/3287149097 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:13.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd737fe700 1 --2- 172.21.15.158:0/3287149097 >> v2:172.21.15.158:3300/0 conn(0x7efd7406afe0 0x7efd740719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd78cde700 1 --2- 172.21.15.158:0/3287149097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd74071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd72ffd700 1 -- 172.21.15.158:0/3287149097 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617044213 0 0) 0x7efd7415dc50 con 0x7efd7415d870 2023-10-16T11:22:13.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd72ffd700 1 -- 172.21.15.158:0/3287149097 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5c0035b0 con 0x7efd7415d870 2023-10-16T11:22:13.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd737fe700 1 -- 172.21.15.158:0/3287149097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 msgr2=0x7efd74071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:13.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd737fe700 1 --2- 172.21.15.158:0/3287149097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd74071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd737fe700 1 -- 172.21.15.158:0/3287149097 >> v1:172.21.15.158:6789/0 conn(0x7efd7415d870 legacy=0x7efd7406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.841+0000 7efd737fe700 1 -- 172.21.15.158:0/3287149097 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7406f3e0 con 0x7efd7406afe0 2023-10-16T11:22:13.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd737fe700 1 --2- 172.21.15.158:0/3287149097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7406afe0 0x7efd740719f0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7efd68009840 tx=0x7efd68009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5e78d71726a880 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:13.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd72ffd700 1 -- 172.21.15.158:0/3287149097 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68011830 con 0x7efd7406afe0 2023-10-16T11:22:13.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd72ffd700 1 -- 172.21.15.158:0/3287149097 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd68011990 con 0x7efd7406afe0 2023-10-16T11:22:13.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd72ffd700 1 -- 172.21.15.158:0/3287149097 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68011d20 con 0x7efd7406afe0 2023-10-16T11:22:13.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd7a741700 1 -- 172.21.15.158:0/3287149097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7406afe0 msgr2=0x7efd740719f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd7a741700 1 --2- 172.21.15.158:0/3287149097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7406afe0 0x7efd740719f0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7efd68009840 tx=0x7efd68009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:13.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd7a741700 1 -- 172.21.15.158:0/3287149097 shutdown_connections 2023-10-16T11:22:13.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd7a741700 1 --2- 172.21.15.158:0/3287149097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd74071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd7a741700 1 --2- 172.21.15.158:0/3287149097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7406afe0 0x7efd740719f0 unknown :-1 s=CLOSED pgs=3356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd7a741700 1 -- 172.21.15.158:0/3287149097 >> 172.21.15.158:0/3287149097 conn(0x7efd7406da00 msgr2=0x7efd7406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:13.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.842+0000 7efd7a741700 1 -- 172.21.15.158:0/3287149097 shutdown_connections 2023-10-16T11:22:13.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.843+0000 7efd7a741700 1 -- 172.21.15.158:0/3287149097 wait complete. 2023-10-16T11:22:13.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.844+0000 7efd7a741700 1 Processor -- start 2023-10-16T11:22:13.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.844+0000 7efd7a741700 1 -- start start 2023-10-16T11:22:13.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd7a741700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd7406afe0 0x7efd7415de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd7a741700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd7414c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd7a741700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7415d870 0x7efd7415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd7a741700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7414cb80 con 0x7efd7415d870 2023-10-16T11:22:13.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd7a741700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7415e520 con 0x7efd7406afe0 2023-10-16T11:22:13.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd7a741700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7415e800 con 0x7efd740cf7c0 2023-10-16T11:22:13.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd78cde700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7415d870 0x7efd7415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd78cde700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7415d870 0x7efd7415c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44502/0 (socket says 172.21.15.158:44502) 2023-10-16T11:22:13.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd78cde700 1 -- 172.21.15.158:0/2593554012 learned_addr learned my addr 172.21.15.158:0/2593554012 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:13.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd737fe700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd7414c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.845+0000 7efd73fff700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd7406afe0 0x7efd7415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:13.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd78cde700 1 -- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 msgr2=0x7efd7414c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd78cde700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd7414c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd78cde700 1 -- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd7406afe0 msgr2=0x7efd7415de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:13.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd78cde700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd7406afe0 0x7efd7415de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:13.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd78cde700 1 -- 172.21.15.158:0/2593554012 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680094a0 con 0x7efd7415d870 2023-10-16T11:22:13.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd78cde700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7415d870 0x7efd7415c310 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00efc0 tx=0x7efd6c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:13.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd70ff9700 1 -- 172.21.15.158:0/2593554012 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c009ea0 con 0x7efd7415d870 2023-10-16T11:22:13.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd70ff9700 1 -- 172.21.15.158:0/2593554012 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c00f040 con 0x7efd7415d870 2023-10-16T11:22:13.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7415eae0 con 0x7efd7415d870 2023-10-16T11:22:13.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd70ff9700 1 -- 172.21.15.158:0/2593554012 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c00ced0 con 0x7efd7415d870 2023-10-16T11:22:13.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.846+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd74158d50 con 0x7efd7415d870 2023-10-16T11:22:13.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.847+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd740681b0 con 0x7efd7415d870 2023-10-16T11:22:13.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.848+0000 7efd70ff9700 1 -- 172.21.15.158:0/2593554012 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7efd6c014480 con 0x7efd7415d870 2023-10-16T11:22:13.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.849+0000 7efd70ff9700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 0x7efd6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:13.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.849+0000 7efd73fff700 1 -- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 msgr2=0x7efd6006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:13.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.849+0000 7efd70ff9700 1 -- 172.21.15.158:0/2593554012 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7efd6c08ab80 con 0x7efd7415d870 2023-10-16T11:22:13.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.849+0000 7efd73fff700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 0x7efd6006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:13.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:13.853+0000 7efd70ff9700 1 -- 172.21.15.158:0/2593554012 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd6c055700 con 0x7efd7415d870 2023-10-16T11:22:14.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.050+0000 7efd73fff700 1 -- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 msgr2=0x7efd6006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:14.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.050+0000 7efd73fff700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 0x7efd6006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:14.057 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:14.056+0000 7fe1bd246200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:22:14.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.099+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efd7415bdf0 con 0x7efd7415d870 2023-10-16T11:22:14.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.099+0000 7efd70ff9700 1 -- 172.21.15.158:0/2593554012 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v271) v1 ==== 80+0+473 (secure 0 0 0) 0x7efd6c018080 con 0x7efd7415d870 2023-10-16T11:22:14.102 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3856677823 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.102 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/1867601061 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.102 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/933196924 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.102 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/3733531003 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.103 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6833/778561374 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.103 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:6832/778561374 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.103 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/915172958 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.103 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158:0/258064180 2023-10-17T11:22:09.987990+0000 2023-10-16T11:22:14.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 msgr2=0x7efd6006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 0x7efd6006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7415d870 msgr2=0x7efd7415c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7415d870 0x7efd7415c310 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00efc0 tx=0x7efd6c00c3a0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 shutdown_connections 2023-10-16T11:22:14.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd7415d870 0x7efd7415c310 unknown :-1 s=CLOSED pgs=3357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7efd6006b020 0x7efd6006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd7406afe0 0x7efd7415de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 --2- 172.21.15.158:0/2593554012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd740cf7c0 0x7efd7414c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 >> 172.21.15.158:0/2593554012 conn(0x7efd7406da00 msgr2=0x7efd74156320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:14.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 shutdown_connections 2023-10-16T11:22:14.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.102+0000 7efd7a741700 1 -- 172.21.15.158:0/2593554012 wait complete. 2023-10-16T11:22:14.107 INFO:teuthology.orchestra.run.smithi158.stderr:listed 8 entries 2023-10-16T11:22:14.117 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T11:22:14.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:14.118 DEBUG:teuthology.orchestra.run.smithi158:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T11:22:14.126 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:14.125+0000 7fe1bd246200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:22:14.148 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:14.146+0000 7f8cd48b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T11:22:14.157 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:22:14.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:14.158 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T11:22:14.214 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:22:14.215 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:14.224 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:14.222+0000 7f8cd48b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T11:22:14.235 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:22:14.235 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:22:14.236 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:14.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:22:14.253 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:22:14.254 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:14.292 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:22:14.293 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:22:14.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.414+0000 7fd117d02700 1 Processor -- start 2023-10-16T11:22:14.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.415+0000 7fd117d02700 1 -- start start 2023-10-16T11:22:14.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.415+0000 7fd117d02700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd1100ce530 0x7fd1100b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.415+0000 7fd117d02700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd11015f0f0 0x7fd1100cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.415+0000 7fd117d02700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11015b930 con 0x7fd1100cbfe0 2023-10-16T11:22:14.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.415+0000 7fd117d02700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1100cc490 con 0x7fd1100ce530 2023-10-16T11:22:14.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd117d02700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100cf020 con 0x7fd11015f0f0 2023-10-16T11:22:14.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd1100ce530 0x7fd1100b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd1100ce530 0x7fd1100b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44508/0 (socket says 172.21.15.158:44508) 2023-10-16T11:22:14.421 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 -- 172.21.15.158:0/3485068179 learned_addr learned my addr 172.21.15.158:0/3485068179 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:14.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd11529d700 1 --2- 172.21.15.158:0/3485068179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd11015f0f0 0x7fd1100cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 -- 172.21.15.158:0/3485068179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd11015f0f0 msgr2=0x7fd1100cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:14.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 --2- 172.21.15.158:0/3485068179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd11015f0f0 0x7fd1100cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.422 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 -- 172.21.15.158:0/3485068179 >> v1:172.21.15.172:6789/0 conn(0x7fd1100cbfe0 legacy=0x7fd1100d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 -- 172.21.15.158:0/3485068179 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11015f5f0 con 0x7fd1100ce530 2023-10-16T11:22:14.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd115a9e700 1 --2- 172.21.15.158:0/3485068179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b3b20 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0098f0 tx=0x7fd10c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402c6039c212858b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:14.423 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd114a9c700 1 -- 172.21.15.158:0/3485068179 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c011830 con 0x7fd1100ce530 2023-10-16T11:22:14.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd114a9c700 1 -- 172.21.15.158:0/3485068179 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10c011990 con 0x7fd1100ce530 2023-10-16T11:22:14.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.416+0000 7fd114a9c700 1 -- 172.21.15.158:0/3485068179 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c011d20 con 0x7fd1100ce530 2023-10-16T11:22:14.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 -- 172.21.15.158:0/3485068179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 msgr2=0x7fd1100b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.424 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 --2- 172.21.15.158:0/3485068179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b3b20 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0098f0 tx=0x7fd10c009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:14.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 -- 172.21.15.158:0/3485068179 shutdown_connections 2023-10-16T11:22:14.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 --2- 172.21.15.158:0/3485068179 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b3b20 unknown :-1 s=CLOSED pgs=3358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 --2- 172.21.15.158:0/3485068179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd11015f0f0 0x7fd1100cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 -- 172.21.15.158:0/3485068179 >> 172.21.15.158:0/3485068179 conn(0x7fd11006a8d0 msgr2=0x7fd11006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:14.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 -- 172.21.15.158:0/3485068179 shutdown_connections 2023-10-16T11:22:14.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.417+0000 7fd117d02700 1 -- 172.21.15.158:0/3485068179 wait complete. 2023-10-16T11:22:14.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.419+0000 7fd117d02700 1 Processor -- start 2023-10-16T11:22:14.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.419+0000 7fd117d02700 1 -- start start 2023-10-16T11:22:14.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.419+0000 7fd117d02700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1100cbfe0 0x7fd11015c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.419+0000 7fd117d02700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd117d02700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd11015f0f0 0x7fd11015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd117d02700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1101e5220 con 0x7fd1100ce530 2023-10-16T11:22:14.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd117d02700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100b6480 con 0x7fd11015f0f0 2023-10-16T11:22:14.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd117d02700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100b6760 con 0x7fd1100cbfe0 2023-10-16T11:22:14.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44516/0 (socket says 172.21.15.158:44516) 2023-10-16T11:22:14.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 -- 172.21.15.158:0/1999034334 learned_addr learned my addr 172.21.15.158:0/1999034334 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:14.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd115a9e700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1100cbfe0 0x7fd11015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11629f700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd11015f0f0 0x7fd11015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 -- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1100cbfe0 msgr2=0x7fd11015c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1100cbfe0 0x7fd11015c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 -- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd11015f0f0 msgr2=0x7fd11015a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd11015f0f0 0x7fd11015a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 -- 172.21.15.158:0/1999034334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c009550 con 0x7fd1100ce530 2023-10-16T11:22:14.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd115a9e700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1100cbfe0 0x7fd11015c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:14.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd11529d700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b5a50 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9b0 tx=0x7fd10000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:14.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd1067fc700 1 -- 172.21.15.158:0/1999034334 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100009710 con 0x7fd1100ce530 2023-10-16T11:22:14.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd1067fc700 1 -- 172.21.15.158:0/1999034334 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10000ee50 con 0x7fd1100ce530 2023-10-16T11:22:14.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd1067fc700 1 -- 172.21.15.158:0/1999034334 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10000ca50 con 0x7fd1100ce530 2023-10-16T11:22:14.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1101e5430 con 0x7fd1100ce530 2023-10-16T11:22:14.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.420+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11006cbf0 con 0x7fd1100ce530 2023-10-16T11:22:14.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.422+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1100681b0 con 0x7fd1100ce530 2023-10-16T11:22:14.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.422+0000 7fd1067fc700 1 -- 172.21.15.158:0/1999034334 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd10000cbb0 con 0x7fd1100ce530 2023-10-16T11:22:14.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.423+0000 7fd1067fc700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 0x7fd0f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.423+0000 7fd115a9e700 1 -- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 msgr2=0x7fd0f406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:14.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.423+0000 7fd1067fc700 1 -- 172.21.15.158:0/1999034334 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fd100089b40 con 0x7fd1100ce530 2023-10-16T11:22:14.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.423+0000 7fd115a9e700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 0x7fd0f406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:14.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.427+0000 7fd1067fc700 1 -- 172.21.15.158:0/1999034334 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1000546d0 con 0x7fd1100ce530 2023-10-16T11:22:14.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.624+0000 7fd115a9e700 1 -- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 msgr2=0x7fd0f406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.624+0000 7fd115a9e700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 0x7fd0f406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:14.680 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.678+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd11015fb80 con 0x7fd1100ce530 2023-10-16T11:22:14.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.679+0000 7fd1067fc700 1 -- 172.21.15.158:0/1999034334 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fd100021090 con 0x7fd1100ce530 2023-10-16T11:22:14.681 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:14.681 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":154,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1253298118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1253298118},{"type":"v1","addr":"172.21.15.158:6837","nonce":1253298118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":13731,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2914884530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2914884530},{"type":"v1","addr":"172.21.15.158:6836","nonce":2914884530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/536559492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":536559492},{"type":"v1","addr":"172.21.15.172:6834","nonce":536559492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:56.094141+0000","modified":"2023-10-16T11:21:59.105113+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":40202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_40202":{"gid":40202,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/284711673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":284711673},{"type":"v1","addr":"172.21.15.172:6835","nonce":284711673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T11:22:14.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 msgr2=0x7fd0f406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:14.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 0x7fd0f406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 msgr2=0x7fd1100b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b5a50 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9b0 tx=0x7fd10000ecc0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 shutdown_connections 2023-10-16T11:22:14.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fd0f406b020 0x7fd0f406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd11015f0f0 0x7fd11015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1100cbfe0 0x7fd11015c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 --2- 172.21.15.158:0/1999034334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1100ce530 0x7fd1100b5a50 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 >> 172.21.15.158:0/1999034334 conn(0x7fd11006a8d0 msgr2=0x7fd1100c6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:14.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 shutdown_connections 2023-10-16T11:22:14.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.681+0000 7fd117d02700 1 -- 172.21.15.158:0/1999034334 wait complete. 2023-10-16T11:22:14.686 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 154 2023-10-16T11:22:14.700 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:22:14.769 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:14.768+0000 7fe1bd246200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:14.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.826+0000 7f322179b700 1 Processor -- start 2023-10-16T11:22:14.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.826+0000 7f322179b700 1 -- start start 2023-10-16T11:22:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.827+0000 7f322179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15dde0 0x7f321c0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.827+0000 7f322179b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c1613b0 con 0x7f321c15ee50 2023-10-16T11:22:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.827+0000 7f322179b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c161690 con 0x7f321c15d2f0 2023-10-16T11:22:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.827+0000 7f322179b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c161970 con 0x7f321c15dde0 2023-10-16T11:22:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.827+0000 7f321b7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f321c15ee50 0x7f321c1602b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44704/0 (socket says 172.21.15.158:44704) 2023-10-16T11:22:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.827+0000 7f321b7fe700 1 -- 172.21.15.158:0/4128658674 learned_addr learned my addr 172.21.15.158:0/4128658674 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f321a7fc700 1 --2- 172.21.15.158:0/4128658674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15dde0 0x7f321c0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f3219ffb700 1 -- 172.21.15.158:0/4128658674 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157144384 0 0) 0x7f321c1613b0 con 0x7f321c15ee50 2023-10-16T11:22:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f3219ffb700 1 -- 172.21.15.158:0/4128658674 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32040035b0 con 0x7f321c15ee50 2023-10-16T11:22:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f3219ffb700 1 -- 172.21.15.158:0/4128658674 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566344467 0 0) 0x7f321c161690 con 0x7f321c15d2f0 2023-10-16T11:22:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f3219ffb700 1 -- 172.21.15.158:0/4128658674 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f321c1613b0 con 0x7f321c15d2f0 2023-10-16T11:22:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f321a7fc700 1 -- 172.21.15.158:0/4128658674 >> v1:172.21.15.172:6789/0 conn(0x7f321c15d2f0 legacy=0x7f321c15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f321a7fc700 1 -- 172.21.15.158:0/4128658674 >> v1:172.21.15.158:6789/0 conn(0x7f321c15ee50 legacy=0x7f321c1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f321a7fc700 1 -- 172.21.15.158:0/4128658674 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c161c50 con 0x7f321c15dde0 2023-10-16T11:22:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f321a7fc700 1 --2- 172.21.15.158:0/4128658674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15dde0 0x7f321c0cf020 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f32100098a0 tx=0x7f3210009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ddf30be3877adfd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f3219ffb700 1 -- 172.21.15.158:0/4128658674 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210011830 con 0x7f321c15dde0 2023-10-16T11:22:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f3219ffb700 1 -- 172.21.15.158:0/4128658674 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3210011990 con 0x7f321c15dde0 2023-10-16T11:22:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.828+0000 7f3219ffb700 1 -- 172.21.15.158:0/4128658674 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210011d20 con 0x7f321c15dde0 2023-10-16T11:22:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.829+0000 7f322179b700 1 -- 172.21.15.158:0/4128658674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15dde0 msgr2=0x7f321c0cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.829+0000 7f322179b700 1 --2- 172.21.15.158:0/4128658674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15dde0 0x7f321c0cf020 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f32100098a0 tx=0x7f3210009bb0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.829+0000 7f322179b700 1 -- 172.21.15.158:0/4128658674 shutdown_connections 2023-10-16T11:22:14.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.829+0000 7f322179b700 1 --2- 172.21.15.158:0/4128658674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15dde0 0x7f321c0cf020 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.829+0000 7f322179b700 1 -- 172.21.15.158:0/4128658674 >> 172.21.15.158:0/4128658674 conn(0x7f321c06a8d0 msgr2=0x7f321c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:14.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.829+0000 7f322179b700 1 -- 172.21.15.158:0/4128658674 shutdown_connections 2023-10-16T11:22:14.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.829+0000 7f322179b700 1 -- 172.21.15.158:0/4128658674 wait complete. 2023-10-16T11:22:14.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.830+0000 7f322179b700 1 Processor -- start 2023-10-16T11:22:14.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.830+0000 7f322179b700 1 -- start start 2023-10-16T11:22:14.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f322179b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f321c15d2f0 0x7f321c1f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f322179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15ee50 0x7f321c157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f322179b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f321c0c6850 0x7f321c15a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f322179b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c1f62e0 con 0x7f321c0c6850 2023-10-16T11:22:14.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f322179b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c1f3280 con 0x7f321c15d2f0 2023-10-16T11:22:14.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f322179b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c1f3560 con 0x7f321c15ee50 2023-10-16T11:22:14.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f321c0c6850 0x7f321c15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f321c0c6850 0x7f321c15a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44534/0 (socket says 172.21.15.158:44534) 2023-10-16T11:22:14.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 -- 172.21.15.158:0/3293863655 learned_addr learned my addr 172.21.15.158:0/3293863655 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:14.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321a7fc700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15ee50 0x7f321c157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321affd700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f321c15d2f0 0x7f321c1f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 -- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15ee50 msgr2=0x7f321c157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15ee50 0x7f321c157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 -- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f321c15d2f0 msgr2=0x7f321c1f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:14.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f321c15d2f0 0x7f321c1f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.831+0000 7f321b7fe700 1 -- 172.21.15.158:0/3293863655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32100094a0 con 0x7f321c0c6850 2023-10-16T11:22:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.832+0000 7f321b7fe700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f321c0c6850 0x7f321c15a560 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f321400cb40 tx=0x7f321400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.832+0000 7f32037fe700 1 -- 172.21.15.158:0/3293863655 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400a9e0 con 0x7f321c0c6850 2023-10-16T11:22:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.832+0000 7f32037fe700 1 -- 172.21.15.158:0/3293863655 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321400ab40 con 0x7f321c0c6850 2023-10-16T11:22:14.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.832+0000 7f32037fe700 1 -- 172.21.15.158:0/3293863655 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214012400 con 0x7f321c0c6850 2023-10-16T11:22:14.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.832+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c1f3840 con 0x7f321c0c6850 2023-10-16T11:22:14.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.832+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c1584e0 con 0x7f321c0c6850 2023-10-16T11:22:14.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.833+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c1f6450 con 0x7f321c0c6850 2023-10-16T11:22:14.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.833+0000 7f32037fe700 1 -- 172.21.15.158:0/3293863655 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f321401b3b0 con 0x7f321c0c6850 2023-10-16T11:22:14.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.834+0000 7f32037fe700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 0x7f320806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:14.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.834+0000 7f321affd700 1 -- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 msgr2=0x7f320806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:14.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.834+0000 7f32037fe700 1 -- 172.21.15.158:0/3293863655 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f3214089720 con 0x7f321c0c6850 2023-10-16T11:22:14.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.834+0000 7f321affd700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 0x7f320806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:14.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:14.838+0000 7f32037fe700 1 -- 172.21.15.158:0/3293863655 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32140542f0 con 0x7f321c0c6850 2023-10-16T11:22:14.848 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:14.847+0000 7fe1bd246200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:22:14.904 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:14.902+0000 7f8cd48b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:14.985 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:14.983+0000 7f8cd48b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T11:22:14.994 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:14.994+0000 7fe1bd246200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:15.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.035+0000 7f321affd700 1 -- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 msgr2=0x7f320806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:15.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.035+0000 7f321affd700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 0x7f320806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:15.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.040+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f321c0681b0 con 0x7f321c0c6850 2023-10-16T11:22:15.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.042+0000 7f32037fe700 1 -- 172.21.15.158:0/3293863655 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+17979 (secure 0 0 0) 0x7f3214001c00 con 0x7f321c0c6850 2023-10-16T11:22:15.044 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:15.044 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":271,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:22:09.988038+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:54.523055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:55.050879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3856677823":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/1867601061":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/933196924":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/3733531003":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6833/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6832/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/915172958":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/258064180":"2023-10-17T11:22:09.987990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:22:15.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 msgr2=0x7f320806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:15.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 0x7f320806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f321c0c6850 msgr2=0x7f321c15a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f321c0c6850 0x7f321c15a560 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f321400cb40 tx=0x7f321400ce50 comp rx=0 tx=0).stop 2023-10-16T11:22:15.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 shutdown_connections 2023-10-16T11:22:15.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f321c0c6850 0x7f321c15a560 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f320806b020 0x7f320806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.047 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f321c15d2f0 0x7f321c1f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 --2- 172.21.15.158:0/3293863655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f321c15ee50 0x7f321c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 >> 172.21.15.158:0/3293863655 conn(0x7f321c06a8d0 msgr2=0x7f321c160530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:15.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 shutdown_connections 2023-10-16T11:22:15.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.044+0000 7f322179b700 1 -- 172.21.15.158:0/3293863655 wait complete. 2023-10-16T11:22:15.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T09:43:15.258805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T11:21:54.523055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '269', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T11:21:55.050879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '270', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T11:22:15.060 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T11:22:15.140 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:15.139+0000 7f8cd48b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T11:22:15.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.184+0000 7f908d7f2700 1 Processor -- start 2023-10-16T11:22:15.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f908d7f2700 1 -- start start 2023-10-16T11:22:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f908d7f2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f908815da40 0x7f90880caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f908d7f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90880cc590 0x7f908806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f908d7f2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f908806ae10 0x7f90880ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f908d7f2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908806b2c0 con 0x7f908806ae10 2023-10-16T11:22:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f908d7f2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90880cbd30 con 0x7f908815da40 2023-10-16T11:22:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f908d7f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90880cc1c0 con 0x7f90880cc590 2023-10-16T11:22:15.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.185+0000 7f90877fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f908806ae10 0x7f90880ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f908806ae10 0x7f90880ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44560/0 (socket says 172.21.15.158:44560) 2023-10-16T11:22:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 -- 172.21.15.158:0/4015867030 learned_addr learned my addr 172.21.15.158:0/4015867030 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90867fc700 1 --2- 172.21.15.158:0/4015867030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90880cc590 0x7f908806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f9086ffd700 1 --2- 172.21.15.158:0/4015867030 >> v2:172.21.15.172:3300/0 conn(0x7f908815da40 0x7f90880caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 -- 172.21.15.158:0/4015867030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90880cc590 msgr2=0x7f908806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:15.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 --2- 172.21.15.158:0/4015867030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90880cc590 0x7f908806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 -- 172.21.15.158:0/4015867030 >> v2:172.21.15.172:3300/0 conn(0x7f908815da40 msgr2=0x7f90880caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 --2- 172.21.15.158:0/4015867030 >> v2:172.21.15.172:3300/0 conn(0x7f908815da40 0x7f90880caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 -- 172.21.15.158:0/4015867030 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908815de20 con 0x7f908806ae10 2023-10-16T11:22:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f90877fe700 1 --2- 172.21.15.158:0/4015867030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f908806ae10 0x7f90880ccd60 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f908806c9c0 tx=0x7f908000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfef68437d918014 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f9085ffb700 1 -- 172.21.15.158:0/4015867030 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908000e070 con 0x7f908806ae10 2023-10-16T11:22:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f9085ffb700 1 -- 172.21.15.158:0/4015867030 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908000b910 con 0x7f908806ae10 2023-10-16T11:22:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.186+0000 7f9085ffb700 1 -- 172.21.15.158:0/4015867030 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908000bcc0 con 0x7f908806ae10 2023-10-16T11:22:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 -- 172.21.15.158:0/4015867030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f908806ae10 msgr2=0x7f90880ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 --2- 172.21.15.158:0/4015867030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f908806ae10 0x7f90880ccd60 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f908806c9c0 tx=0x7f908000b330 comp rx=0 tx=0).stop 2023-10-16T11:22:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 -- 172.21.15.158:0/4015867030 shutdown_connections 2023-10-16T11:22:15.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 --2- 172.21.15.158:0/4015867030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f908806ae10 0x7f90880ccd60 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 --2- 172.21.15.158:0/4015867030 >> v2:172.21.15.172:3300/0 conn(0x7f908815da40 0x7f90880caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 --2- 172.21.15.158:0/4015867030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90880cc590 0x7f908806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 -- 172.21.15.158:0/4015867030 >> 172.21.15.158:0/4015867030 conn(0x7f9088099a10 msgr2=0x7f90880d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:15.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 -- 172.21.15.158:0/4015867030 shutdown_connections 2023-10-16T11:22:15.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.187+0000 7f908d7f2700 1 -- 172.21.15.158:0/4015867030 wait complete. 2023-10-16T11:22:15.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.189+0000 7f908d7f2700 1 Processor -- start 2023-10-16T11:22:15.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.189+0000 7f908d7f2700 1 -- start start 2023-10-16T11:22:15.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.189+0000 7f908d7f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f908806ae10 0x7f908815b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.189+0000 7f908d7f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90880cc590 0x7f90880ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f908d7f2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f908815da40 0x7f90880c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f908d7f2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90880c6220 con 0x7f90880cc590 2023-10-16T11:22:15.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f908d7f2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90880ca5a0 con 0x7f908815da40 2023-10-16T11:22:15.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f908d7f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90880c5530 con 0x7f908806ae10 2023-10-16T11:22:15.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90880cc590 0x7f90880ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90880cc590 0x7f90880ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44564/0 (socket says 172.21.15.158:44564) 2023-10-16T11:22:15.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 -- 172.21.15.158:0/1635268715 learned_addr learned my addr 172.21.15.158:0/1635268715 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:15.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f9086ffd700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f908806ae10 0x7f908815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90877fe700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f908815da40 0x7f90880c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 -- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f908806ae10 msgr2=0x7f908815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f908806ae10 0x7f908815b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 -- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f908815da40 msgr2=0x7f90880c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f908815da40 0x7f90880c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 -- 172.21.15.158:0/1635268715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080009b30 con 0x7f90880cc590 2023-10-16T11:22:15.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f90867fc700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90880cc590 0x7f90880ca060 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f907c00e900 tx=0x7f907c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:15.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f906f7fe700 1 -- 172.21.15.158:0/1635268715 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c013680 con 0x7f90880cc590 2023-10-16T11:22:15.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f906f7fe700 1 -- 172.21.15.158:0/1635268715 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c0137e0 con 0x7f90880cc590 2023-10-16T11:22:15.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90880c5b60 con 0x7f90880cc590 2023-10-16T11:22:15.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f906f7fe700 1 -- 172.21.15.158:0/1635268715 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c013b70 con 0x7f90880cc590 2023-10-16T11:22:15.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.190+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90880c5810 con 0x7f90880cc590 2023-10-16T11:22:15.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.191+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90880c6490 con 0x7f90880cc590 2023-10-16T11:22:15.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.192+0000 7f906f7fe700 1 -- 172.21.15.158:0/1635268715 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f907c013cd0 con 0x7f90880cc590 2023-10-16T11:22:15.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.192+0000 7f906f7fe700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 0x7f907406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.192+0000 7f9086ffd700 1 -- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 msgr2=0x7f907406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:15.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.192+0000 7f906f7fe700 1 -- 172.21.15.158:0/1635268715 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f907c0896f0 con 0x7f90880cc590 2023-10-16T11:22:15.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.192+0000 7f9086ffd700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 0x7f907406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:15.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.197+0000 7f906f7fe700 1 -- 172.21.15.158:0/1635268715 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f907c0542a0 con 0x7f90880cc590 2023-10-16T11:22:15.224 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:15.223+0000 7fe1bd246200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:22:15.299 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:15.298+0000 7fe1bd246200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:22:15.379 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:15.377+0000 7f8cd48b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T11:22:15.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.393+0000 7f9086ffd700 1 -- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 msgr2=0x7f907406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:15.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.393+0000 7f9086ffd700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 0x7f907406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:15.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.397+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f90880681b0 con 0x7f90880cc590 2023-10-16T11:22:15.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.398+0000 7f906f7fe700 1 -- 172.21.15.158:0/1635268715 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v271) v1 ==== 93+0+10 (secure 0 0 0) 0x7f907c0578c0 con 0x7f90880cc590 2023-10-16T11:22:15.400 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 1 2023-10-16T11:22:15.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 msgr2=0x7f907406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:15.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 0x7f907406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90880cc590 msgr2=0x7f90880ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90880cc590 0x7f90880ca060 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f907c00e900 tx=0x7f907c00ec10 comp rx=0 tx=0).stop 2023-10-16T11:22:15.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 shutdown_connections 2023-10-16T11:22:15.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f907406b020 0x7f907406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f908815da40 0x7f90880c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f908806ae10 0x7f908815b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 --2- 172.21.15.158:0/1635268715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90880cc590 0x7f90880ca060 unknown :-1 s=CLOSED pgs=3363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 >> 172.21.15.158:0/1635268715 conn(0x7f9088099a10 msgr2=0x7f90880d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:15.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 shutdown_connections 2023-10-16T11:22:15.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.400+0000 7f908d7f2700 1 -- 172.21.15.158:0/1635268715 wait complete. 2023-10-16T11:22:15.415 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T11:22:15.461 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:15.459+0000 7f8cd48b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T11:22:15.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.542+0000 7f64600b3700 1 Processor -- start 2023-10-16T11:22:15.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.542+0000 7f64600b3700 1 -- start start 2023-10-16T11:22:15.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f64600b3700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f64580d0c60 0x7f64580b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f64600b3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f64580b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f64600b3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64580b4420 con 0x7f645815b6b0 2023-10-16T11:22:15.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f64600b3700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645815ba90 con 0x7f64580d0c60 2023-10-16T11:22:15.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f64600b3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64580b38f0 con 0x7f64580b3460 2023-10-16T11:22:15.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f645d64e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f64580d0c60 0x7f64580b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f645d64e700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f64580d0c60 0x7f64580b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44574/0 (socket says 172.21.15.158:44574) 2023-10-16T11:22:15.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f645d64e700 1 -- 172.21.15.158:0/2788720982 learned_addr learned my addr 172.21.15.158:0/2788720982 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:15.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.543+0000 7f645e650700 1 --2- 172.21.15.158:0/2788720982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f64580b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645d64e700 1 -- 172.21.15.158:0/2788720982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 msgr2=0x7f64580b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:15.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645d64e700 1 --2- 172.21.15.158:0/2788720982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f64580b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645d64e700 1 -- 172.21.15.158:0/2788720982 >> v1:172.21.15.172:6789/0 conn(0x7f645815b6b0 legacy=0x7f64580cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645d64e700 1 -- 172.21.15.158:0/2788720982 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64580cbc20 con 0x7f64580d0c60 2023-10-16T11:22:15.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645d64e700 1 --2- 172.21.15.158:0/2788720982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f64580b2f20 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f64480098f0 tx=0x7f6448009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9dfce9e617034e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:15.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645ce4d700 1 -- 172.21.15.158:0/2788720982 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448011830 con 0x7f64580d0c60 2023-10-16T11:22:15.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645ce4d700 1 -- 172.21.15.158:0/2788720982 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6448011990 con 0x7f64580d0c60 2023-10-16T11:22:15.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.544+0000 7f645ce4d700 1 -- 172.21.15.158:0/2788720982 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448011d20 con 0x7f64580d0c60 2023-10-16T11:22:15.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 -- 172.21.15.158:0/2788720982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 msgr2=0x7f64580b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 --2- 172.21.15.158:0/2788720982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f64580b2f20 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f64480098f0 tx=0x7f6448009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:15.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 -- 172.21.15.158:0/2788720982 shutdown_connections 2023-10-16T11:22:15.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 --2- 172.21.15.158:0/2788720982 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f64580b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 --2- 172.21.15.158:0/2788720982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f64580b2f20 unknown :-1 s=CLOSED pgs=3364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 -- 172.21.15.158:0/2788720982 >> 172.21.15.158:0/2788720982 conn(0x7f64580cdf20 msgr2=0x7f64580ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:15.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 -- 172.21.15.158:0/2788720982 shutdown_connections 2023-10-16T11:22:15.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.545+0000 7f64600b3700 1 -- 172.21.15.158:0/2788720982 wait complete. 2023-10-16T11:22:15.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.546+0000 7f64600b3700 1 Processor -- start 2023-10-16T11:22:15.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.546+0000 7f64600b3700 1 -- start start 2023-10-16T11:22:15.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f64600b3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f6458152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f64600b3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f6458159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f64600b3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f645815b6b0 0x7f6458159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f64600b3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64581f3f70 con 0x7f64580d0c60 2023-10-16T11:22:15.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f64600b3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6458154ef0 con 0x7f645815b6b0 2023-10-16T11:22:15.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f64600b3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64581f3290 con 0x7f64580b3460 2023-10-16T11:22:15.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f6458159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f6458159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44580/0 (socket says 172.21.15.158:44580) 2023-10-16T11:22:15.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 -- 172.21.15.158:0/1640937414 learned_addr learned my addr 172.21.15.158:0/1640937414 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:15.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645e650700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f645815b6b0 0x7f6458159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645de4f700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f6458152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 -- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 msgr2=0x7f6458152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f6458152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 -- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f645815b6b0 msgr2=0x7f6458159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f645815b6b0 0x7f6458159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 -- 172.21.15.158:0/1640937414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6448009550 con 0x7f64580d0c60 2023-10-16T11:22:15.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.547+0000 7f645d64e700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f6458159740 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f6448000c00 tx=0x7f6448004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:15.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.548+0000 7f644e7fc700 1 -- 172.21.15.158:0/1640937414 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448004480 con 0x7f64580d0c60 2023-10-16T11:22:15.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.548+0000 7f644e7fc700 1 -- 172.21.15.158:0/1640937414 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644801f070 con 0x7f64580d0c60 2023-10-16T11:22:15.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.548+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64581545f0 con 0x7f64580d0c60 2023-10-16T11:22:15.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.548+0000 7f644e7fc700 1 -- 172.21.15.158:0/1640937414 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64480038c0 con 0x7f64580d0c60 2023-10-16T11:22:15.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.548+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64581f7110 con 0x7f64580d0c60 2023-10-16T11:22:15.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.549+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6458155030 con 0x7f64580d0c60 2023-10-16T11:22:15.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.549+0000 7f644e7fc700 1 -- 172.21.15.158:0/1640937414 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f6448021030 con 0x7f64580d0c60 2023-10-16T11:22:15.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.550+0000 7f644e7fc700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 0x7f643c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.550+0000 7f645de4f700 1 -- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 msgr2=0x7f643c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:15.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.550+0000 7f644e7fc700 1 -- 172.21.15.158:0/1640937414 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f644808b0a0 con 0x7f64580d0c60 2023-10-16T11:22:15.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.550+0000 7f645de4f700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 0x7f643c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:15.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.554+0000 7f644e7fc700 1 -- 172.21.15.158:0/1640937414 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6458155030 con 0x7f64580d0c60 2023-10-16T11:22:15.567 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:15.566+0000 7fe1bd246200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:22:15.740 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:15.739+0000 7f8cd48b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T11:22:15.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.751+0000 7f645de4f700 1 -- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 msgr2=0x7f643c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:15.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.751+0000 7f645de4f700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 0x7f643c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:15.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.753+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6458155030 con 0x7f64580d0c60 2023-10-16T11:22:15.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.753+0000 7f644e7fc700 1 -- 172.21.15.158:0/1640937414 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v271) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6458155030 con 0x7f64580d0c60 2023-10-16T11:22:15.755 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 32 2023-10-16T11:22:15.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 msgr2=0x7f643c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:15.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 0x7f643c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 msgr2=0x7f6458159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f6458159740 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f6448000c00 tx=0x7f6448004750 comp rx=0 tx=0).stop 2023-10-16T11:22:15.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 shutdown_connections 2023-10-16T11:22:15.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f643c06b020 0x7f643c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f645815b6b0 0x7f6458159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64580b3460 0x7f6458152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 --2- 172.21.15.158:0/1640937414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64580d0c60 0x7f6458159740 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 >> 172.21.15.158:0/1640937414 conn(0x7f64580cdf20 msgr2=0x7f64580cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:15.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 shutdown_connections 2023-10-16T11:22:15.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.756+0000 7f64600b3700 1 -- 172.21.15.158:0/1640937414 wait complete. 2023-10-16T11:22:15.770 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T11:22:15.790 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:15.789+0000 7fe1bd246200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:22:15.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.895+0000 7fc90e083700 1 Processor -- start 2023-10-16T11:22:15.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.896+0000 7fc90e083700 1 -- start start 2023-10-16T11:22:15.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.896+0000 7fc90e083700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc90806e0d0 0x7fc90815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.896+0000 7fc90e083700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90815f380 con 0x7fc9080cbcf0 2023-10-16T11:22:15.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.896+0000 7fc90e083700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc908071e70 con 0x7fc90815dc40 2023-10-16T11:22:15.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.896+0000 7fc90e083700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc908072150 con 0x7fc90806e0d0 2023-10-16T11:22:15.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.896+0000 7fc906ffd700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc90806e0d0 0x7fc90815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc907fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9080cbcf0 0x7fc9080cc140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44768/0 (socket says 172.21.15.158:44768) 2023-10-16T11:22:15.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc907fff700 1 -- 172.21.15.158:0/266425058 learned_addr learned my addr 172.21.15.158:0/266425058 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:15.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc906ffd700 1 -- 172.21.15.158:0/266425058 >> v1:172.21.15.172:6789/0 conn(0x7fc90815dc40 legacy=0x7fc90806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc906ffd700 1 -- 172.21.15.158:0/266425058 >> v1:172.21.15.158:6789/0 conn(0x7fc9080cbcf0 legacy=0x7fc9080cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.901 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc906ffd700 1 -- 172.21.15.158:0/266425058 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc908072430 con 0x7fc90806e0d0 2023-10-16T11:22:15.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc9067fc700 1 -- 172.21.15.158:0/266425058 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603692078 0 0) 0x7fc8f8001490 con 0x7fc9080cbcf0 2023-10-16T11:22:15.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc906ffd700 1 --2- 172.21.15.158:0/266425058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90806e0d0 0x7fc90815ee40 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009840 tx=0x7fc8f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be095203b3c8435 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:15.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc9067fc700 1 -- 172.21.15.158:0/266425058 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0011830 con 0x7fc90806e0d0 2023-10-16T11:22:15.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc9067fc700 1 -- 172.21.15.158:0/266425058 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0011990 con 0x7fc90806e0d0 2023-10-16T11:22:15.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.897+0000 7fc9067fc700 1 -- 172.21.15.158:0/266425058 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0011d20 con 0x7fc90806e0d0 2023-10-16T11:22:15.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.898+0000 7fc90e083700 1 -- 172.21.15.158:0/266425058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90806e0d0 msgr2=0x7fc90815ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.898+0000 7fc90e083700 1 --2- 172.21.15.158:0/266425058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90806e0d0 0x7fc90815ee40 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009840 tx=0x7fc8f0009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:15.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.898+0000 7fc90e083700 1 -- 172.21.15.158:0/266425058 shutdown_connections 2023-10-16T11:22:15.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.898+0000 7fc90e083700 1 --2- 172.21.15.158:0/266425058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90806e0d0 0x7fc90815ee40 unknown :-1 s=CLOSED pgs=3367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.898+0000 7fc90e083700 1 -- 172.21.15.158:0/266425058 >> 172.21.15.158:0/266425058 conn(0x7fc9080d03b0 msgr2=0x7fc9080d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:15.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.898+0000 7fc90e083700 1 -- 172.21.15.158:0/266425058 shutdown_connections 2023-10-16T11:22:15.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.898+0000 7fc90e083700 1 -- 172.21.15.158:0/266425058 wait complete. 2023-10-16T11:22:15.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.899+0000 7fc90e083700 1 Processor -- start 2023-10-16T11:22:15.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.899+0000 7fc90e083700 1 -- start start 2023-10-16T11:22:15.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc90e083700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc90806e0d0 0x7fc9081578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc90e083700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9080cbcf0 0x7fc90815b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc90e083700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90815dc40 0x7fc9080ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc90e083700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9081f2f60 con 0x7fc90815dc40 2023-10-16T11:22:15.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc90e083700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9080ad8e0 con 0x7fc9080cbcf0 2023-10-16T11:22:15.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc90e083700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9080adbc0 con 0x7fc90806e0d0 2023-10-16T11:22:15.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90815dc40 0x7fc9080ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90815dc40 0x7fc9080ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44598/0 (socket says 172.21.15.158:44598) 2023-10-16T11:22:15.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 -- 172.21.15.158:0/552522820 learned_addr learned my addr 172.21.15.158:0/552522820 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:15.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc906ffd700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9080cbcf0 0x7fc90815b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc9077fe700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc90806e0d0 0x7fc9081578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:15.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 -- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc90806e0d0 msgr2=0x7fc9081578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc90806e0d0 0x7fc9081578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 -- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9080cbcf0 msgr2=0x7fc90815b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:15.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9080cbcf0 0x7fc90815b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:15.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.900+0000 7fc907fff700 1 -- 172.21.15.158:0/552522820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f00094a0 con 0x7fc90815dc40 2023-10-16T11:22:15.911 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.901+0000 7fc907fff700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90815dc40 0x7fc9080ac890 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800cca0 tx=0x7fc8f800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:15.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.901+0000 7fc8fffff700 1 -- 172.21.15.158:0/552522820 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f800ab70 con 0x7fc90815dc40 2023-10-16T11:22:15.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.901+0000 7fc8fffff700 1 -- 172.21.15.158:0/552522820 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f800acd0 con 0x7fc90815dc40 2023-10-16T11:22:15.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.901+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9080adea0 con 0x7fc90815dc40 2023-10-16T11:22:15.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.901+0000 7fc8fffff700 1 -- 172.21.15.158:0/552522820 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8007310 con 0x7fc90815dc40 2023-10-16T11:22:15.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.901+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9080ae180 con 0x7fc90815dc40 2023-10-16T11:22:15.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.902+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9081f3390 con 0x7fc90815dc40 2023-10-16T11:22:15.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.904+0000 7fc8fffff700 1 -- 172.21.15.158:0/552522820 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc8f8007470 con 0x7fc90815dc40 2023-10-16T11:22:15.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.904+0000 7fc8fffff700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 0x7fc8e806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:15.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.904+0000 7fc9077fe700 1 -- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 msgr2=0x7fc8e806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:15.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.904+0000 7fc8fffff700 1 -- 172.21.15.158:0/552522820 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fc8f8089c60 con 0x7fc90815dc40 2023-10-16T11:22:15.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.904+0000 7fc9077fe700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 0x7fc8e806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:15.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:15.907+0000 7fc8fffff700 1 -- 172.21.15.158:0/552522820 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc8f8054860 con 0x7fc90815dc40 2023-10-16T11:22:15.942 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:15.941+0000 7fe1bd246200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:22:15.994 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:15.992+0000 7f8cd48b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T11:22:16.056 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:16.055+0000 7fe1bd246200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:22:16.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.105+0000 7fc9077fe700 1 -- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 msgr2=0x7fc8e806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:16.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.105+0000 7fc9077fe700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 0x7fc8e806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.124+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc90815f380 con 0x7fc90815dc40 2023-10-16T11:22:16.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.124+0000 7fc8fffff700 1 -- 172.21.15.158:0/552522820 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v271) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc90815f380 con 0x7fc90815dc40 2023-10-16T11:22:16.127 INFO:teuthology.orchestra.run.smithi158.stdout:pg_num: 64 2023-10-16T11:22:16.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 msgr2=0x7fc8e806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:16.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 0x7fc8e806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90815dc40 msgr2=0x7fc9080ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90815dc40 0x7fc9080ac890 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800cca0 tx=0x7fc8f800a060 comp rx=0 tx=0).stop 2023-10-16T11:22:16.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 shutdown_connections 2023-10-16T11:22:16.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc90815dc40 0x7fc9080ac890 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc90806e0d0 0x7fc9081578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc8e806b020 0x7fc8e806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 --2- 172.21.15.158:0/552522820 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9080cbcf0 0x7fc90815b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 >> 172.21.15.158:0/552522820 conn(0x7fc9080d03b0 msgr2=0x7fc9080b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:16.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 shutdown_connections 2023-10-16T11:22:16.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.128+0000 7fc90e083700 1 -- 172.21.15.158:0/552522820 wait complete. 2023-10-16T11:22:16.144 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:22:16.156 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:16.154+0000 7f8cd48b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T11:22:16.212 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:16.211+0000 7fe1bd246200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:22:16.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.266+0000 7fc455bb9700 1 Processor -- start 2023-10-16T11:22:16.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.266+0000 7fc455bb9700 1 -- start start 2023-10-16T11:22:16.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.266+0000 7fc455bb9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc4500ce530 0x7fc4500b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc455bb9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4500cc1b0 0x7fc4500cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc455bb9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45015b930 con 0x7fc45015f0f0 2023-10-16T11:22:16.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc455bb9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4500cf020 con 0x7fc4500cc1b0 2023-10-16T11:22:16.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc455bb9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45015f5f0 con 0x7fc4500ce530 2023-10-16T11:22:16.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4500cc1b0 0x7fc4500cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44effd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc45015f0f0 0x7fc4500cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44782/0 (socket says 172.21.15.158:44782) 2023-10-16T11:22:16.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44effd700 1 -- 172.21.15.158:0/4048863215 learned_addr learned my addr 172.21.15.158:0/4048863215 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:16.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44f7fe700 1 --2- 172.21.15.158:0/4048863215 >> v2:172.21.15.172:3300/0 conn(0x7fc4500ce530 0x7fc4500b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44ffff700 1 -- 172.21.15.158:0/4048863215 >> v2:172.21.15.172:3300/0 conn(0x7fc4500ce530 msgr2=0x7fc4500b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:16.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44ffff700 1 --2- 172.21.15.158:0/4048863215 >> v2:172.21.15.172:3300/0 conn(0x7fc4500ce530 0x7fc4500b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44ffff700 1 -- 172.21.15.158:0/4048863215 >> v1:172.21.15.158:6789/0 conn(0x7fc45015f0f0 legacy=0x7fc4500cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44e7fc700 1 -- 172.21.15.158:0/4048863215 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087286689 0 0) 0x7fc45015b930 con 0x7fc45015f0f0 2023-10-16T11:22:16.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44ffff700 1 -- 172.21.15.158:0/4048863215 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45015f8d0 con 0x7fc4500cc1b0 2023-10-16T11:22:16.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44f7fe700 1 --2- 172.21.15.158:0/4048863215 >> v2:172.21.15.172:3300/0 conn(0x7fc4500ce530 0x7fc4500b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:16.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44ffff700 1 --2- 172.21.15.158:0/4048863215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500cc1b0 0x7fc4500cf470 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fc45006b8e0 tx=0x7fc44800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b65770b374164b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:16.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44e7fc700 1 -- 172.21.15.158:0/4048863215 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44800e070 con 0x7fc4500cc1b0 2023-10-16T11:22:16.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44e7fc700 1 -- 172.21.15.158:0/4048863215 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44800bbd0 con 0x7fc4500cc1b0 2023-10-16T11:22:16.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.267+0000 7fc44e7fc700 1 -- 172.21.15.158:0/4048863215 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448012430 con 0x7fc4500cc1b0 2023-10-16T11:22:16.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 -- 172.21.15.158:0/4048863215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500cc1b0 msgr2=0x7fc4500cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 --2- 172.21.15.158:0/4048863215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500cc1b0 0x7fc4500cf470 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fc45006b8e0 tx=0x7fc44800b330 comp rx=0 tx=0).stop 2023-10-16T11:22:16.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 -- 172.21.15.158:0/4048863215 shutdown_connections 2023-10-16T11:22:16.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 --2- 172.21.15.158:0/4048863215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500cc1b0 0x7fc4500cf470 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 --2- 172.21.15.158:0/4048863215 >> v2:172.21.15.172:3300/0 conn(0x7fc4500ce530 0x7fc4500b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 -- 172.21.15.158:0/4048863215 >> 172.21.15.158:0/4048863215 conn(0x7fc45006a8d0 msgr2=0x7fc45006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:16.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 -- 172.21.15.158:0/4048863215 shutdown_connections 2023-10-16T11:22:16.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.268+0000 7fc455bb9700 1 -- 172.21.15.158:0/4048863215 wait complete. 2023-10-16T11:22:16.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.269+0000 7fc455bb9700 1 Processor -- start 2023-10-16T11:22:16.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc455bb9700 1 -- start start 2023-10-16T11:22:16.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc455bb9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4500cc1b0 0x7fc4501f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc455bb9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500ce530 0x7fc4501f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc455bb9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45015f0f0 0x7fc45006c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc455bb9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500c4f00 con 0x7fc4500ce530 2023-10-16T11:22:16.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc455bb9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45006d040 con 0x7fc45015f0f0 2023-10-16T11:22:16.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc455bb9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45006d320 con 0x7fc4500cc1b0 2023-10-16T11:22:16.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc44effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500ce530 0x7fc4501f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc44effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500ce530 0x7fc4501f6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44612/0 (socket says 172.21.15.158:44612) 2023-10-16T11:22:16.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc44effd700 1 -- 172.21.15.158:0/2204239985 learned_addr learned my addr 172.21.15.158:0/2204239985 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:16.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.270+0000 7fc44ffff700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45015f0f0 0x7fc45006c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc44f7fe700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4500cc1b0 0x7fc4501f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc44effd700 1 -- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4500cc1b0 msgr2=0x7fc4501f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc44effd700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4500cc1b0 0x7fc4501f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc44effd700 1 -- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45015f0f0 msgr2=0x7fc45006c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc44effd700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45015f0f0 0x7fc45006c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc44effd700 1 -- 172.21.15.158:0/2204239985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448009a70 con 0x7fc4500ce530 2023-10-16T11:22:16.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc44effd700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500ce530 0x7fc4501f6ac0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c540 tx=0x7fc44400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc437fff700 1 -- 172.21.15.158:0/2204239985 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444004c80 con 0x7fc4500ce530 2023-10-16T11:22:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc437fff700 1 -- 172.21.15.158:0/2204239985 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc444004de0 con 0x7fc4500ce530 2023-10-16T11:22:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4500c5150 con 0x7fc4500ce530 2023-10-16T11:22:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc437fff700 1 -- 172.21.15.158:0/2204239985 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444007310 con 0x7fc4500ce530 2023-10-16T11:22:16.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.271+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4501f2c90 con 0x7fc4500ce530 2023-10-16T11:22:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.272+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4500681b0 con 0x7fc4500ce530 2023-10-16T11:22:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.273+0000 7fc437fff700 1 -- 172.21.15.158:0/2204239985 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc444007470 con 0x7fc4500ce530 2023-10-16T11:22:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.273+0000 7fc437fff700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 0x7fc43c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.273+0000 7fc44f7fe700 1 -- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 msgr2=0x7fc43c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:16.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.273+0000 7fc437fff700 1 -- 172.21.15.158:0/2204239985 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fc4440897d0 con 0x7fc4500ce530 2023-10-16T11:22:16.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.273+0000 7fc44f7fe700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 0x7fc43c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:16.288 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:16.277+0000 7f8cd48b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T11:22:16.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.278+0000 7fc437fff700 1 -- 172.21.15.158:0/2204239985 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc444054300 con 0x7fc4500ce530 2023-10-16T11:22:16.442 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:16.441+0000 7f8cd48b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T11:22:16.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.474+0000 7fc44f7fe700 1 -- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 msgr2=0x7fc43c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:16.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.474+0000 7fc44f7fe700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 0x7fc43c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:16.476 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:16.475+0000 7fe1bd246200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:16.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.518+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4500b3900 con 0x7fc4500ce530 2023-10-16T11:22:16.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.519+0000 7fc437fff700 1 -- 172.21.15.158:0/2204239985 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fc444057920 con 0x7fc4500ce530 2023-10-16T11:22:16.521 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:16.521 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":154,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1253298118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1253298118},{"type":"v1","addr":"172.21.15.158:6837","nonce":1253298118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":13731,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2914884530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2914884530},{"type":"v1","addr":"172.21.15.158:6836","nonce":2914884530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/536559492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":536559492},{"type":"v1","addr":"172.21.15.172:6834","nonce":536559492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:56.094141+0000","modified":"2023-10-16T11:21:59.105113+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":40202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_40202":{"gid":40202,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/284711673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":284711673},{"type":"v1","addr":"172.21.15.172:6835","nonce":284711673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T11:22:16.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 msgr2=0x7fc43c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:16.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 0x7fc43c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500ce530 msgr2=0x7fc4501f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500ce530 0x7fc4501f6ac0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c540 tx=0x7fc44400c850 comp rx=0 tx=0).stop 2023-10-16T11:22:16.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 shutdown_connections 2023-10-16T11:22:16.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fc43c06b020 0x7fc43c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc45015f0f0 0x7fc45006c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4500cc1b0 0x7fc4501f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 --2- 172.21.15.158:0/2204239985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4500ce530 0x7fc4501f6ac0 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 >> 172.21.15.158:0/2204239985 conn(0x7fc45006a8d0 msgr2=0x7fc4500b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:16.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 shutdown_connections 2023-10-16T11:22:16.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.521+0000 7fc455bb9700 1 -- 172.21.15.158:0/2204239985 wait complete. 2023-10-16T11:22:16.526 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 154 2023-10-16T11:22:16.537 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:22:16.638 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:16.637+0000 7fe1bd246200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:22:16.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.699+0000 7f853a160700 1 Processor -- start 2023-10-16T11:22:16.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.699+0000 7f853a160700 1 -- start start 2023-10-16T11:22:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f853a160700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f853406e090 0x7f85341655a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f853a160700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340b1e40 con 0x7f85340af590 2023-10-16T11:22:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f853a160700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340b2120 con 0x7f853415a6c0 2023-10-16T11:22:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f853a160700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340ca7e0 con 0x7f853406e090 2023-10-16T11:22:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f8532ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f85340af590 0x7f853406d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44798/0 (socket says 172.21.15.158:44798) 2023-10-16T11:22:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f8532ffd700 1 -- 172.21.15.158:0/2151160040 learned_addr learned my addr 172.21.15.158:0/2151160040 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:16.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f8533fff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f853406e090 0x7f85341655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f85327fc700 1 -- 172.21.15.158:0/2151160040 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034977729 0 0) 0x7f85340b1e40 con 0x7f85340af590 2023-10-16T11:22:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f85327fc700 1 -- 172.21.15.158:0/2151160040 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c0035b0 con 0x7f85340af590 2023-10-16T11:22:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f8533fff700 1 -- 172.21.15.158:0/2151160040 >> v1:172.21.15.172:6789/0 conn(0x7f853415a6c0 legacy=0x7f85340723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f8533fff700 1 -- 172.21.15.158:0/2151160040 >> v1:172.21.15.158:6789/0 conn(0x7f85340af590 legacy=0x7f853406d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f8533fff700 1 -- 172.21.15.158:0/2151160040 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340b2400 con 0x7f853406e090 2023-10-16T11:22:16.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f8533fff700 1 --2- 172.21.15.158:0/2151160040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853406e090 0x7f85341655a0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f852c009d90 tx=0x7f852c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591e4031bd998677 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:16.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f85327fc700 1 -- 172.21.15.158:0/2151160040 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00e070 con 0x7f853406e090 2023-10-16T11:22:16.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f85327fc700 1 -- 172.21.15.158:0/2151160040 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00bb20 con 0x7f853406e090 2023-10-16T11:22:16.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.700+0000 7f85327fc700 1 -- 172.21.15.158:0/2151160040 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c012430 con 0x7f853406e090 2023-10-16T11:22:16.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.701+0000 7f853a160700 1 -- 172.21.15.158:0/2151160040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853406e090 msgr2=0x7f85341655a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.701+0000 7f853a160700 1 --2- 172.21.15.158:0/2151160040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853406e090 0x7f85341655a0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f852c009d90 tx=0x7f852c00b330 comp rx=0 tx=0).stop 2023-10-16T11:22:16.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.701+0000 7f853a160700 1 -- 172.21.15.158:0/2151160040 shutdown_connections 2023-10-16T11:22:16.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.701+0000 7f853a160700 1 --2- 172.21.15.158:0/2151160040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853406e090 0x7f85341655a0 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.701+0000 7f853a160700 1 -- 172.21.15.158:0/2151160040 >> 172.21.15.158:0/2151160040 conn(0x7f85340ccda0 msgr2=0x7f85340cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:16.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.701+0000 7f853a160700 1 -- 172.21.15.158:0/2151160040 shutdown_connections 2023-10-16T11:22:16.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.701+0000 7f853a160700 1 -- 172.21.15.158:0/2151160040 wait complete. 2023-10-16T11:22:16.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.703+0000 7f853a160700 1 Processor -- start 2023-10-16T11:22:16.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.703+0000 7f853a160700 1 -- start start 2023-10-16T11:22:16.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f853a160700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f853406e090 0x7f85340c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f853a160700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85340af590 0x7f85340c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f853a160700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853415a6c0 0x7f85340cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f853a160700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534155300 con 0x7f853415a6c0 2023-10-16T11:22:16.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f853a160700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85340c7d90 con 0x7f853406e090 2023-10-16T11:22:16.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f853a160700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85340c7fe0 con 0x7f85340af590 2023-10-16T11:22:16.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853415a6c0 0x7f85340cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f85337fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f853406e090 0x7f85340c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853415a6c0 0x7f85340cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44630/0 (socket says 172.21.15.158:44630) 2023-10-16T11:22:16.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 -- 172.21.15.158:0/1730060125 learned_addr learned my addr 172.21.15.158:0/1730060125 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:16.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8532ffd700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85340af590 0x7f85340c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:16.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 -- 172.21.15.158:0/1730060125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85340af590 msgr2=0x7f85340c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85340af590 0x7f85340c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 -- 172.21.15.158:0/1730060125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f853406e090 msgr2=0x7f85340c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f853406e090 0x7f85340c6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 -- 172.21.15.158:0/1730060125 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c009a70 con 0x7f853415a6c0 2023-10-16T11:22:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f8533fff700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853415a6c0 0x7f85340cb9a0 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f853406d2c0 tx=0x7f852c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:16.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.704+0000 7f851bfff700 1 -- 172.21.15.158:0/1730060125 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c012430 con 0x7f853415a6c0 2023-10-16T11:22:16.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.705+0000 7f851bfff700 1 -- 172.21.15.158:0/1730060125 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c020070 con 0x7f853415a6c0 2023-10-16T11:22:16.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.705+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85340c82c0 con 0x7f853415a6c0 2023-10-16T11:22:16.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.705+0000 7f851bfff700 1 -- 172.21.15.158:0/1730060125 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0098f0 con 0x7f853415a6c0 2023-10-16T11:22:16.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.705+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85341564c0 con 0x7f853415a6c0 2023-10-16T11:22:16.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.706+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85340681b0 con 0x7f853415a6c0 2023-10-16T11:22:16.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.706+0000 7f851bfff700 1 -- 172.21.15.158:0/1730060125 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f852c004040 con 0x7f853415a6c0 2023-10-16T11:22:16.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.707+0000 7f851bfff700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 0x7f852006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:16.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.707+0000 7f85337fe700 1 -- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 msgr2=0x7f852006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:16.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.707+0000 7f851bfff700 1 -- 172.21.15.158:0/1730060125 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f852c022070 con 0x7f853415a6c0 2023-10-16T11:22:16.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.707+0000 7f85337fe700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 0x7f852006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:16.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.711+0000 7f851bfff700 1 -- 172.21.15.158:0/1730060125 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f852c0562d0 con 0x7f853415a6c0 2023-10-16T11:22:16.723 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:16.721+0000 7f8cd48b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T11:22:16.893 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:16.891+0000 7f8cd48b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T11:22:16.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.908+0000 7f85337fe700 1 -- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 msgr2=0x7f852006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:16.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.908+0000 7f85337fe700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 0x7f852006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:16.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.911+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85340b3e40 con 0x7f853415a6c0 2023-10-16T11:22:16.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.913+0000 7f851bfff700 1 -- 172.21.15.158:0/1730060125 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+17979 (secure 0 0 0) 0x7f852c0598f0 con 0x7f853415a6c0 2023-10-16T11:22:16.915 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:16.915 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":271,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:22:09.988038+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:54.523055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:55.050879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:0/3856677823":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/1867601061":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/933196924":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/3733531003":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6833/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6832/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/915172958":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/258064180":"2023-10-17T11:22:09.987990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:22:16.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 msgr2=0x7f852006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:16.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 0x7f852006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853415a6c0 msgr2=0x7f85340cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:16.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853415a6c0 0x7f85340cb9a0 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f853406d2c0 tx=0x7f852c004ad0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 shutdown_connections 2023-10-16T11:22:16.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f853415a6c0 0x7f85340cb9a0 unknown :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7f852006b020 0x7f852006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f853406e090 0x7f85340c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 --2- 172.21.15.158:0/1730060125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85340af590 0x7f85340c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:16.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 >> 172.21.15.158:0/1730060125 conn(0x7f85340ccda0 msgr2=0x7f85340ad330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:16.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 shutdown_connections 2023-10-16T11:22:16.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:16.915+0000 7f853a160700 1 -- 172.21.15.158:0/1730060125 wait complete. 2023-10-16T11:22:16.930 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:22:16.931 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:22:17.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.053+0000 7fe5febc1700 1 Processor -- start 2023-10-16T11:22:17.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.053+0000 7fe5febc1700 1 -- start start 2023-10-16T11:22:17.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5febc1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5f815d2f0 0x7fe5f815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5febc1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815ee50 0x7fe5f815dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5febc1700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f81613b0 con 0x7fe5f806d590 2023-10-16T11:22:17.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5febc1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f8161690 con 0x7fe5f815d2f0 2023-10-16T11:22:17.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5febc1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f8161970 con 0x7fe5f815ee50 2023-10-16T11:22:17.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fc95d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5f815d2f0 0x7fe5f815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fd15e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe5f806d590 0x7fe5f80cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44816/0 (socket says 172.21.15.158:44816) 2023-10-16T11:22:17.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fc95d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5f815d2f0 0x7fe5f815d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44636/0 (socket says 172.21.15.158:44636) 2023-10-16T11:22:17.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fd15e700 1 -- 172.21.15.158:0/1324727660 learned_addr learned my addr 172.21.15.158:0/1324727660 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:17.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5f7fff700 1 --2- 172.21.15.158:0/1324727660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815ee50 0x7fe5f815dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fc95d700 1 -- 172.21.15.158:0/1324727660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815ee50 msgr2=0x7fe5f815dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:17.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fc95d700 1 --2- 172.21.15.158:0/1324727660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815ee50 0x7fe5f815dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fc95d700 1 -- 172.21.15.158:0/1324727660 >> v1:172.21.15.158:6789/0 conn(0x7fe5f806d590 legacy=0x7fe5f80cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.054+0000 7fe5fc95d700 1 -- 172.21.15.158:0/1324727660 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f8161c50 con 0x7fe5f815d2f0 2023-10-16T11:22:17.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5fc95d700 1 --2- 172.21.15.158:0/1324727660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f815d2f0 0x7fe5f815d6d0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009840 tx=0x7fe5ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8dc9162ce2b8106 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:17.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5f77fe700 1 -- 172.21.15.158:0/1324727660 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec011830 con 0x7fe5f815d2f0 2023-10-16T11:22:17.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5f77fe700 1 -- 172.21.15.158:0/1324727660 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec011990 con 0x7fe5f815d2f0 2023-10-16T11:22:17.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5f77fe700 1 -- 172.21.15.158:0/1324727660 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec011d20 con 0x7fe5f815d2f0 2023-10-16T11:22:17.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 -- 172.21.15.158:0/1324727660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f815d2f0 msgr2=0x7fe5f815d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1324727660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f815d2f0 0x7fe5f815d6d0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009840 tx=0x7fe5ec009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:17.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 -- 172.21.15.158:0/1324727660 shutdown_connections 2023-10-16T11:22:17.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1324727660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f815d2f0 0x7fe5f815d6d0 unknown :-1 s=CLOSED pgs=3376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1324727660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815ee50 0x7fe5f815dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 -- 172.21.15.158:0/1324727660 >> 172.21.15.158:0/1324727660 conn(0x7fe5f806a8d0 msgr2=0x7fe5f806ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:17.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 -- 172.21.15.158:0/1324727660 shutdown_connections 2023-10-16T11:22:17.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.055+0000 7fe5febc1700 1 -- 172.21.15.158:0/1324727660 wait complete. 2023-10-16T11:22:17.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.057+0000 7fe5febc1700 1 Processor -- start 2023-10-16T11:22:17.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.057+0000 7fe5febc1700 1 -- start start 2023-10-16T11:22:17.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5febc1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f806d590 0x7fe5f8156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5febc1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815d2f0 0x7fe5f815bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5febc1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5f815ee50 0x7fe5f815c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5febc1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c67b0 con 0x7fe5f806d590 2023-10-16T11:22:17.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5febc1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c5c30 con 0x7fe5f815ee50 2023-10-16T11:22:17.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5febc1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c5f10 con 0x7fe5f815d2f0 2023-10-16T11:22:17.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f806d590 0x7fe5f8156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f806d590 0x7fe5f8156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44642/0 (socket says 172.21.15.158:44642) 2023-10-16T11:22:17.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 -- 172.21.15.158:0/1811964647 learned_addr learned my addr 172.21.15.158:0/1811964647 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:17.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5f7fff700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815d2f0 0x7fe5f815bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fd15e700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5f815ee50 0x7fe5f815c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 -- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815d2f0 msgr2=0x7fe5f815bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815d2f0 0x7fe5f815bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 -- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5f815ee50 msgr2=0x7fe5f815c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5f815ee50 0x7fe5f815c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 -- 172.21.15.158:0/1811964647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec0094a0 con 0x7fe5f806d590 2023-10-16T11:22:17.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5fc95d700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f806d590 0x7fe5f8156d80 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009fd0 tx=0x7fe5ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:17.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5f57fa700 1 -- 172.21.15.158:0/1811964647 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec0036a0 con 0x7fe5f806d590 2023-10-16T11:22:17.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5f57fa700 1 -- 172.21.15.158:0/1811964647 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec01f070 con 0x7fe5f806d590 2023-10-16T11:22:17.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f80c61f0 con 0x7fe5f806d590 2023-10-16T11:22:17.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.059+0000 7fe5f57fa700 1 -- 172.21.15.158:0/1811964647 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec003a60 con 0x7fe5f806d590 2023-10-16T11:22:17.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.060+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f80b3de0 con 0x7fe5f806d590 2023-10-16T11:22:17.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.060+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f80681b0 con 0x7fe5f806d590 2023-10-16T11:22:17.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.061+0000 7fe5f57fa700 1 -- 172.21.15.158:0/1811964647 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 171) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe5ec003bc0 con 0x7fe5f806d590 2023-10-16T11:22:17.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.062+0000 7fe5f57fa700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 0x7fe5dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.062+0000 7fe5f7fff700 1 -- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 msgr2=0x7fe5dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:17.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.062+0000 7fe5f57fa700 1 -- 172.21.15.158:0/1811964647 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fe5ec021070 con 0x7fe5f806d590 2023-10-16T11:22:17.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.062+0000 7fe5f7fff700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 0x7fe5dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T11:22:17.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.066+0000 7fe5f57fa700 1 -- 172.21.15.158:0/1811964647 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5ec055ee0 con 0x7fe5f806d590 2023-10-16T11:22:17.168 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-16T11:22:17.168+0000 7fe1bd246200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:22:17.233 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:17.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:17.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.263+0000 7fe5f7fff700 1 -- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 msgr2=0x7fe5dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6832/224988149 2023-10-16T11:22:17.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.263+0000 7fe5f7fff700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 0x7fe5dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T11:22:17.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.328+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5f80b73d0 con 0x7fe5f806d590 2023-10-16T11:22:17.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.328+0000 7fe5f57fa700 1 -- 172.21.15.158:0/1811964647 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fe5ec059500 con 0x7fe5f806d590 2023-10-16T11:22:17.330 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:17.331 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":154,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1253298118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1253298118},{"type":"v1","addr":"172.21.15.158:6837","nonce":1253298118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":13731,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2914884530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2914884530},{"type":"v1","addr":"172.21.15.158:6836","nonce":2914884530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/536559492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":536559492},{"type":"v1","addr":"172.21.15.172:6834","nonce":536559492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:56.094141+0000","modified":"2023-10-16T11:21:59.105113+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":40202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_40202":{"gid":40202,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/284711673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":284711673},{"type":"v1","addr":"172.21.15.172:6835","nonce":284711673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T11:22:17.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.330+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 msgr2=0x7fe5dc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T11:22:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.330+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 0x7fe5dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.330+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f806d590 msgr2=0x7fe5f8156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.330+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f806d590 0x7fe5f8156d80 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009fd0 tx=0x7fe5ec004750 comp rx=0 tx=0).stop 2023-10-16T11:22:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 shutdown_connections 2023-10-16T11:22:17.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:6832/224988149,v1:172.21.15.158:6833/224988149] conn(0x7fe5dc06b020 0x7fe5dc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5f815ee50 0x7fe5f815c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f806d590 0x7fe5f8156d80 unknown :-1 s=CLOSED pgs=3377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 --2- 172.21.15.158:0/1811964647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5f815d2f0 0x7fe5f815bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 >> 172.21.15.158:0/1811964647 conn(0x7fe5f806a8d0 msgr2=0x7fe5f80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:17.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 shutdown_connections 2023-10-16T11:22:17.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.331+0000 7fe5febc1700 1 -- 172.21.15.158:0/1811964647 wait complete. 2023-10-16T11:22:17.335 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 154 2023-10-16T11:22:17.345 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:22:17.476 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.475+0000 7f8cd48b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T11:22:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.512+0000 7f981b028700 1 Processor -- start 2023-10-16T11:22:17.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.512+0000 7f981b028700 1 -- start start 2023-10-16T11:22:17.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f981b028700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98140ce530 0x7f98140b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f981b028700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f981415f0f0 0x7f98140cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f981b028700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981415b930 con 0x7f98140cbfe0 2023-10-16T11:22:17.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f981b028700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98140cc490 con 0x7f981415f0f0 2023-10-16T11:22:17.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f981b028700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98140cf020 con 0x7f98140ce530 2023-10-16T11:22:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98195c5700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f98140cbfe0 0x7f98140d13e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44852/0 (socket says 172.21.15.158:44852) 2023-10-16T11:22:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98195c5700 1 -- 172.21.15.158:0/2432422431 learned_addr learned my addr 172.21.15.158:0/2432422431 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:17.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f9818dc4700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98140ce530 0x7f98140b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f9813fff700 1 --2- 172.21.15.158:0/2432422431 >> v2:172.21.15.172:3300/0 conn(0x7f981415f0f0 0x7f98140cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648566767 0 0) 0x7f981415b930 con 0x7f98140cbfe0 2023-10-16T11:22:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98040035b0 con 0x7f98140cbfe0 2023-10-16T11:22:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165815504 0 0) 0x7f98040035b0 con 0x7f98140cbfe0 2023-10-16T11:22:17.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98140ce530 msgr2=0x7f98140b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:17.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98140ce530 0x7f98140b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 >> v2:172.21.15.172:3300/0 conn(0x7f981415f0f0 msgr2=0x7f98140cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:17.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 --2- 172.21.15.158:0/2432422431 >> v2:172.21.15.172:3300/0 conn(0x7f981415f0f0 0x7f98140cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981415f5f0 con 0x7f98140cbfe0 2023-10-16T11:22:17.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f980c0019f0 con 0x7f98140cbfe0 2023-10-16T11:22:17.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.513+0000 7f9813fff700 1 --2- 172.21.15.158:0/2432422431 >> v2:172.21.15.172:3300/0 conn(0x7f981415f0f0 0x7f98140cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:17.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 >> v1:172.21.15.158:6789/0 conn(0x7f98140cbfe0 legacy=0x7f98140d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9804003a00 0x7f9804003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9804004850 0x7f9804004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804005210 0x7f980400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98140cc490 con 0x7f9804003a00 2023-10-16T11:22:17.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98140cf020 con 0x7f9804004850 2023-10-16T11:22:17.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980400bf50 con 0x7f9804005210 2023-10-16T11:22:17.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98140ce530 0x7f98140b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:17.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9804003a00 0x7f9804003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9813fff700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9804004850 0x7f9804004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98195c5700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804005210 0x7f980400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 -- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804005210 msgr2=0x7f980400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:17.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804005210 0x7f980400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 -- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9804004850 msgr2=0x7f9804004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:17.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9804004850 0x7f9804004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 -- 172.21.15.158:0/2432422431 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98040035b0 con 0x7f9804003a00 2023-10-16T11:22:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f9818dc4700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9804003a00 0x7f9804003e20 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f9808009ba0 tx=0x7f9804005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8eb33edef932b61 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808004030 con 0x7f9804003a00 2023-10-16T11:22:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9808004190 con 0x7f9804003a00 2023-10-16T11:22:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.514+0000 7f98137fe700 1 -- 172.21.15.158:0/2432422431 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808004540 con 0x7f9804003a00 2023-10-16T11:22:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.515+0000 7f981b028700 1 -- 172.21.15.158:0/2432422431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9804003a00 msgr2=0x7f9804003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.515+0000 7f981b028700 1 --2- 172.21.15.158:0/2432422431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9804003a00 0x7f9804003e20 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f9808009ba0 tx=0x7f9804005880 comp rx=0 tx=0).stop 2023-10-16T11:22:17.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.515+0000 7f98195c5700 1 -- 172.21.15.158:0/2432422431 reap_dead start 2023-10-16T11:22:17.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.515+0000 7f981b028700 1 -- 172.21.15.158:0/2432422431 shutdown_connections 2023-10-16T11:22:17.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.515+0000 7f981b028700 1 -- 172.21.15.158:0/2432422431 >> 172.21.15.158:0/2432422431 conn(0x7f981406a8d0 msgr2=0x7f981406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:17.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.515+0000 7f981b028700 1 -- 172.21.15.158:0/2432422431 shutdown_connections 2023-10-16T11:22:17.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.515+0000 7f981b028700 1 -- 172.21.15.158:0/2432422431 wait complete. 2023-10-16T11:22:17.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.516+0000 7f981b028700 1 Processor -- start 2023-10-16T11:22:17.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.516+0000 7f981b028700 1 -- start start 2023-10-16T11:22:17.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f981b028700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804003a00 0x7f98140ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f981b028700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981406b9e0 0x7f98141f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f981b028700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98141f46f0 0x7f98140d0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f981b028700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981406bf50 con 0x7f98141f46f0 2023-10-16T11:22:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f981b028700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981406c230 con 0x7f981406b9e0 2023-10-16T11:22:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f981b028700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981406c510 con 0x7f9804003a00 2023-10-16T11:22:17.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98141f46f0 0x7f98140d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98141f46f0 0x7f98140d0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44688/0 (socket says 172.21.15.158:44688) 2023-10-16T11:22:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 -- 172.21.15.158:0/3008503742 learned_addr learned my addr 172.21.15.158:0/3008503742 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f9813fff700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981406b9e0 0x7f98141f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f9818dc4700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804003a00 0x7f98140ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 -- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804003a00 msgr2=0x7f98140ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804003a00 0x7f98140ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 -- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981406b9e0 msgr2=0x7f98141f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981406b9e0 0x7f98141f41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.517+0000 7f98195c5700 1 -- 172.21.15.158:0/3008503742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981406c7f0 con 0x7f98141f46f0 2023-10-16T11:22:17.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.518+0000 7f98195c5700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98141f46f0 0x7f98140d0480 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f98140cde30 tx=0x7f980c00a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:17.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.518+0000 7f98117fa700 1 -- 172.21.15.158:0/3008503742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00ad00 con 0x7f98141f46f0 2023-10-16T11:22:17.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.518+0000 7f98117fa700 1 -- 172.21.15.158:0/3008503742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c0133f0 con 0x7f98141f46f0 2023-10-16T11:22:17.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.518+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98141f3100 con 0x7f98141f46f0 2023-10-16T11:22:17.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.518+0000 7f98117fa700 1 -- 172.21.15.158:0/3008503742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00aea0 con 0x7f98141f46f0 2023-10-16T11:22:17.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.518+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981406d420 con 0x7f98141f46f0 2023-10-16T11:22:17.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.519+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981415fb90 con 0x7f98141f46f0 2023-10-16T11:22:17.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.519+0000 7f98117fa700 1 -- 172.21.15.158:0/3008503742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 172) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f980c0137a0 con 0x7f98141f46f0 2023-10-16T11:22:17.541 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.520+0000 7f98117fa700 1 -- 172.21.15.158:0/3008503742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f980c089dd0 con 0x7f98141f46f0 2023-10-16T11:22:17.542 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.524+0000 7f98117fa700 1 -- 172.21.15.158:0/3008503742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f981415fb90 con 0x7f98141f46f0 2023-10-16T11:22:17.558 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.556+0000 7f8c500e5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.559 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.556+0000 7f8c500e5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.559 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.556+0000 7f8c500e5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.559 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.556+0000 7f8c500e5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.559 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.556+0000 7f8c500e5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.557+0000 7f8c4d09f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.557+0000 7f8c4d09f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.557+0000 7f8c4d09f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.557+0000 7f8c4d09f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.560 INFO:tasks.ceph.mgr.y.smithi158.stderr:2023-10-16T11:22:17.557+0000 7f8c4d09f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T11:22:17.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.771+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f981415fb90 con 0x7f98141f46f0 2023-10-16T11:22:17.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.771+0000 7f98117fa700 1 -- 172.21.15.158:0/3008503742 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f981415fb90 con 0x7f98141f46f0 2023-10-16T11:22:17.774 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:17.774 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":154,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1253298118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1253298118},{"type":"v1","addr":"172.21.15.158:6837","nonce":1253298118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":13731,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2914884530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2914884530},{"type":"v1","addr":"172.21.15.158:6836","nonce":2914884530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/536559492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":536559492},{"type":"v1","addr":"172.21.15.172:6834","nonce":536559492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T11:21:56.094141+0000","modified":"2023-10-16T11:21:59.105113+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":40202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_40202":{"gid":40202,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.172:6835/284711673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":284711673},{"type":"v1","addr":"172.21.15.172:6835","nonce":284711673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T11:22:17.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.773+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98141f46f0 msgr2=0x7f98140d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98141f46f0 0x7f98140d0480 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f98140cde30 tx=0x7f980c00a640 comp rx=0 tx=0).stop 2023-10-16T11:22:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 shutdown_connections 2023-10-16T11:22:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98141f46f0 0x7f98140d0480 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9804003a00 0x7f98140ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.776 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 --2- 172.21.15.158:0/3008503742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981406b9e0 0x7f98141f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 >> 172.21.15.158:0/3008503742 conn(0x7f981406a8d0 msgr2=0x7f98140b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:17.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 shutdown_connections 2023-10-16T11:22:17.777 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.774+0000 7f981b028700 1 -- 172.21.15.158:0/3008503742 wait complete. 2023-10-16T11:22:17.777 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 154 2023-10-16T11:22:17.788 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:22:17.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.906+0000 7f3e03e34700 1 Processor -- start 2023-10-16T11:22:17.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.907+0000 7f3e03e34700 1 -- start start 2023-10-16T11:22:17.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.907+0000 7f3e03e34700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3dfc06e5d0 0x7f3dfc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.918 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.907+0000 7f3e03e34700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc161270 con 0x7f3dfc0729b0 2023-10-16T11:22:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.907+0000 7f3e03e34700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc161510 con 0x7f3dfc06ab40 2023-10-16T11:22:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.907+0000 7f3e03e34700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc1617f0 con 0x7f3dfc06e5d0 2023-10-16T11:22:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e023d1700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3dfc0729b0 0x7f3dfc15df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44922/0 (socket says 172.21.15.158:44922) 2023-10-16T11:22:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e023d1700 1 -- 172.21.15.158:0/179768877 learned_addr learned my addr 172.21.15.158:0/179768877 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:17.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e013cf700 1 --2- 172.21.15.158:0/179768877 >> v2:172.21.15.172:3300/0 conn(0x7f3dfc06e5d0 0x7f3dfc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e00bce700 1 -- 172.21.15.158:0/179768877 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807669297 0 0) 0x7f3dfc161270 con 0x7f3dfc0729b0 2023-10-16T11:22:17.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e00bce700 1 -- 172.21.15.158:0/179768877 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de80035b0 con 0x7f3dfc0729b0 2023-10-16T11:22:17.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e013cf700 1 -- 172.21.15.158:0/179768877 >> v1:172.21.15.172:6789/0 conn(0x7f3dfc06ab40 legacy=0x7f3dfc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e013cf700 1 -- 172.21.15.158:0/179768877 >> v1:172.21.15.158:6789/0 conn(0x7f3dfc0729b0 legacy=0x7f3dfc15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e013cf700 1 -- 172.21.15.158:0/179768877 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc161ad0 con 0x7f3dfc06e5d0 2023-10-16T11:22:17.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e013cf700 1 --2- 172.21.15.158:0/179768877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc06d820 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009840 tx=0x7f3dec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e3f0bfe815e1a56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:17.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e00bce700 1 -- 172.21.15.158:0/179768877 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec011830 con 0x7f3dfc06e5d0 2023-10-16T11:22:17.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e00bce700 1 -- 172.21.15.158:0/179768877 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec011990 con 0x7f3dfc06e5d0 2023-10-16T11:22:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.908+0000 7f3e00bce700 1 -- 172.21.15.158:0/179768877 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec011d20 con 0x7f3dfc06e5d0 2023-10-16T11:22:17.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.909+0000 7f3e03e34700 1 -- 172.21.15.158:0/179768877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 msgr2=0x7f3dfc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.909+0000 7f3e03e34700 1 --2- 172.21.15.158:0/179768877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc06d820 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009840 tx=0x7f3dec009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.909+0000 7f3e03e34700 1 -- 172.21.15.158:0/179768877 shutdown_connections 2023-10-16T11:22:17.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.909+0000 7f3e03e34700 1 --2- 172.21.15.158:0/179768877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc06d820 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.909+0000 7f3e03e34700 1 -- 172.21.15.158:0/179768877 >> 172.21.15.158:0/179768877 conn(0x7f3dfc0d02b0 msgr2=0x7f3dfc0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.909+0000 7f3e03e34700 1 -- 172.21.15.158:0/179768877 shutdown_connections 2023-10-16T11:22:17.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.909+0000 7f3e03e34700 1 -- 172.21.15.158:0/179768877 wait complete. 2023-10-16T11:22:17.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.910+0000 7f3e03e34700 1 Processor -- start 2023-10-16T11:22:17.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.910+0000 7f3e03e34700 1 -- start start 2023-10-16T11:22:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e03e34700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3dfc06ab40 0x7f3dfc1f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e03e34700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e03e34700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3dfc0729b0 0x7f3dfc0b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:17.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e03e34700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1f2ab0 con 0x7f3dfc06ab40 2023-10-16T11:22:17.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e03e34700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b2c00 con 0x7f3dfc06e5d0 2023-10-16T11:22:17.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e03e34700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b5780 con 0x7f3dfc0729b0 2023-10-16T11:22:17.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3dfc06ab40 0x7f3dfc1f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3dfc06ab40 0x7f3dfc1f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44748/0 (socket says 172.21.15.158:44748) 2023-10-16T11:22:17.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 -- 172.21.15.158:0/914347860 learned_addr learned my addr 172.21.15.158:0/914347860 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:17.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e013cf700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e023d1700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3dfc0729b0 0x7f3dfc0b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:17.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 -- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3dfc0729b0 msgr2=0x7f3dfc0b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3dfc0729b0 0x7f3dfc0b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 -- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 msgr2=0x7f3dfc0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:17.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc0afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:17.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e01bd0700 1 -- 172.21.15.158:0/914347860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec0094a0 con 0x7f3dfc06ab40 2023-10-16T11:22:17.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.911+0000 7f3e013cf700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:17.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.912+0000 7f3e01bd0700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3dfc06ab40 0x7f3dfc1f2570 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c700 tx=0x7f3df800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:17.934 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.912+0000 7f3df27fc700 1 -- 172.21.15.158:0/914347860 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df80052d0 con 0x7f3dfc06ab40 2023-10-16T11:22:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.912+0000 7f3df27fc700 1 -- 172.21.15.158:0/914347860 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df8005430 con 0x7f3dfc06ab40 2023-10-16T11:22:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.912+0000 7f3df27fc700 1 -- 172.21.15.158:0/914347860 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8003880 con 0x7f3dfc06ab40 2023-10-16T11:22:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.912+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc0b2640 con 0x7f3dfc06ab40 2023-10-16T11:22:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.912+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc146420 con 0x7f3dfc06ab40 2023-10-16T11:22:17.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.913+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dfc0681b0 con 0x7f3dfc06ab40 2023-10-16T11:22:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.913+0000 7f3df27fc700 1 -- 172.21.15.158:0/914347860 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 172) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f3df80039e0 con 0x7f3dfc06ab40 2023-10-16T11:22:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.914+0000 7f3df27fc700 1 -- 172.21.15.158:0/914347860 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f3df8088ac0 con 0x7f3dfc06ab40 2023-10-16T11:22:17.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:17.918+0000 7f3df27fc700 1 -- 172.21.15.158:0/914347860 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3df8053a20 con 0x7f3dfc06ab40 2023-10-16T11:22:18.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.113+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dfc0b6230 con 0x7f3dfc06ab40 2023-10-16T11:22:18.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.114+0000 7f3df27fc700 1 -- 172.21.15.158:0/914347860 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) v1 ==== 74+0+18484 (secure 0 0 0) 0x7f3df8057040 con 0x7f3dfc06ab40 2023-10-16T11:22:18.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:18.120 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":272,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:22:17.488908+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T11:21:54.523055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T11:21:55.050879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/224988149":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/2492444608":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/580008490":"2023-10-17T11:22:17.488860+0000","172.21.15.158:6832/224988149":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/662944544":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/183508379":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/3875121437":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/1289746000":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/3856677823":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/1867601061":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/933196924":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/3733531003":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6833/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6832/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/915172958":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/258064180":"2023-10-17T11:22:09.987990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:22:18.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3dfc06ab40 msgr2=0x7f3dfc1f2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3dfc06ab40 0x7f3dfc1f2570 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c700 tx=0x7f3df800cac0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 shutdown_connections 2023-10-16T11:22:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3dfc0729b0 0x7f3dfc0b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3dfc06ab40 0x7f3dfc1f2570 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 --2- 172.21.15.158:0/914347860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3dfc06e5d0 0x7f3dfc0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 >> 172.21.15.158:0/914347860 conn(0x7f3dfc0d02b0 msgr2=0x7f3dfc06d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:18.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 shutdown_connections 2023-10-16T11:22:18.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.116+0000 7f3e03e34700 1 -- 172.21.15.158:0/914347860 wait complete. 2023-10-16T11:22:18.130 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T11:22:18.251 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.250+0000 7f6a6f61f700 1 Processor -- start 2023-10-16T11:22:18.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.250+0000 7f6a6f61f700 1 -- start start 2023-10-16T11:22:18.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6f61f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6a6806a8d0 0x7f6a6806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6f61f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a680afac0 0x7f6a68161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6f61f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6a680cfeb0 0x7f6a68071a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6f61f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6815dd30 con 0x7f6a6806a8d0 2023-10-16T11:22:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6f61f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6806b1f0 con 0x7f6a680cfeb0 2023-10-16T11:22:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6f61f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b3f20 con 0x7f6a680afac0 2023-10-16T11:22:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6a6806a8d0 0x7f6a6806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6a6806a8d0 0x7f6a6806acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44754/0 (socket says 172.21.15.158:44754) 2023-10-16T11:22:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 -- 172.21.15.158:0/3519899184 learned_addr learned my addr 172.21.15.158:0/3519899184 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6cbba700 1 --2- 172.21.15.158:0/3519899184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a680afac0 0x7f6a68161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 -- 172.21.15.158:0/3519899184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a680afac0 msgr2=0x7f6a68161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 --2- 172.21.15.158:0/3519899184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a680afac0 0x7f6a68161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 -- 172.21.15.158:0/3519899184 >> v2:172.21.15.172:3300/0 conn(0x7f6a680cfeb0 msgr2=0x7f6a68071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6dbbc700 1 --2- 172.21.15.158:0/3519899184 >> v2:172.21.15.172:3300/0 conn(0x7f6a680cfeb0 0x7f6a68071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 --2- 172.21.15.158:0/3519899184 >> v2:172.21.15.172:3300/0 conn(0x7f6a680cfeb0 0x7f6a68071a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 -- 172.21.15.158:0/3519899184 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680b4200 con 0x7f6a6806a8d0 2023-10-16T11:22:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.251+0000 7f6a6d3bb700 1 --2- 172.21.15.158:0/3519899184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a6806a8d0 0x7f6a6806acb0 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f6a680cc650 tx=0x7f6a64009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e8a8e4edafc369 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:18.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a5ffff700 1 -- 172.21.15.158:0/3519899184 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400b870 con 0x7f6a6806a8d0 2023-10-16T11:22:18.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a5ffff700 1 -- 172.21.15.158:0/3519899184 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6400b9d0 con 0x7f6a6806a8d0 2023-10-16T11:22:18.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a5ffff700 1 -- 172.21.15.158:0/3519899184 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400bd60 con 0x7f6a6806a8d0 2023-10-16T11:22:18.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3519899184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a6806a8d0 msgr2=0x7f6a6806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3519899184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a6806a8d0 0x7f6a6806acb0 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f6a680cc650 tx=0x7f6a64009bd0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3519899184 shutdown_connections 2023-10-16T11:22:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3519899184 >> v2:172.21.15.172:3300/0 conn(0x7f6a680cfeb0 0x7f6a68071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3519899184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a6806a8d0 0x7f6a6806acb0 unknown :-1 s=CLOSED pgs=3388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3519899184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a680afac0 0x7f6a68161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3519899184 >> 172.21.15.158:0/3519899184 conn(0x7f6a6806da40 msgr2=0x7f6a6806de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:18.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3519899184 shutdown_connections 2023-10-16T11:22:18.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.252+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3519899184 wait complete. 2023-10-16T11:22:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.253+0000 7f6a6f61f700 1 Processor -- start 2023-10-16T11:22:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.253+0000 7f6a6f61f700 1 -- start start 2023-10-16T11:22:18.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6f61f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a6806a8d0 0x7f6a680ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6f61f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a680afac0 0x7f6a680ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6f61f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a680cfeb0 0x7f6a680ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6f61f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6815e7b0 con 0x7f6a680afac0 2023-10-16T11:22:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6f61f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680ae420 con 0x7f6a680cfeb0 2023-10-16T11:22:18.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6f61f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680ae700 con 0x7f6a6806a8d0 2023-10-16T11:22:18.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6cbba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a680afac0 0x7f6a680ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6cbba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a680afac0 0x7f6a680ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44762/0 (socket says 172.21.15.158:44762) 2023-10-16T11:22:18.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6cbba700 1 -- 172.21.15.158:0/3621247377 learned_addr learned my addr 172.21.15.158:0/3621247377 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:18.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6dbbc700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a680cfeb0 0x7f6a680ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.254+0000 7f6a6d3bb700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a6806a8d0 0x7f6a680ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6cbba700 1 -- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a6806a8d0 msgr2=0x7f6a680ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6cbba700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a6806a8d0 0x7f6a680ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6cbba700 1 -- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a680cfeb0 msgr2=0x7f6a680ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6cbba700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a680cfeb0 0x7f6a680ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6cbba700 1 -- 172.21.15.158:0/3621247377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64009550 con 0x7f6a680afac0 2023-10-16T11:22:18.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6dbbc700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a680cfeb0 0x7f6a680ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6cbba700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a680afac0 0x7f6a680ad8d0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800c7f0 tx=0x7f6a5800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58011800 con 0x7f6a680afac0 2023-10-16T11:22:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a58011960 con 0x7f6a680afac0 2023-10-16T11:22:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a680ae9e0 con 0x7f6a680afac0 2023-10-16T11:22:18.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58011d10 con 0x7f6a680afac0 2023-10-16T11:22:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.255+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a680aefa0 con 0x7f6a680afac0 2023-10-16T11:22:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.256+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a680af1c0 con 0x7f6a680afac0 2023-10-16T11:22:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.256+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 172) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f6a58018360 con 0x7f6a680afac0 2023-10-16T11:22:18.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.257+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f6a58089260 con 0x7f6a680afac0 2023-10-16T11:22:18.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.261+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a580540a0 con 0x7f6a680afac0 2023-10-16T11:22:18.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.500+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6a680681b0 con 0x7f6a680afac0 2023-10-16T11:22:18.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.500+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 173) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6a58053b50 con 0x7f6a680afac0 2023-10-16T11:22:18.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.501+0000 7f6a5dffb700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6a4c071250 0x7f6a4c073640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.502+0000 7f6a6d3bb700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6a4c071250 0x7f6a4c073640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.503+0000 7f6a6d3bb700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6a4c071250 0x7f6a4c073640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a64014040 tx=0x7f6a64008030 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:18.505 INFO:tasks.ceph.mon.a.smithi158.stderr:2023-10-16T11:22:18.503+0000 7fbb09252700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T11:22:18.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.506+0000 7f6a5dffb700 1 -- 172.21.15.158:0/3621247377 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v155) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6a580709d0 con 0x7f6a680afac0 2023-10-16T11:22:18.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6a4c071250 msgr2=0x7f6a4c073640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6a4c071250 0x7f6a4c073640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a64014040 tx=0x7f6a64008030 comp rx=0 tx=0).stop 2023-10-16T11:22:18.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a680afac0 msgr2=0x7f6a680ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a680afac0 0x7f6a680ad8d0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800c7f0 tx=0x7f6a5800cbb0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 shutdown_connections 2023-10-16T11:22:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6a4c071250 0x7f6a4c073640 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a680cfeb0 0x7f6a680ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a6806a8d0 0x7f6a680ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 --2- 172.21.15.158:0/3621247377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a680afac0 0x7f6a680ad8d0 unknown :-1 s=CLOSED pgs=3389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 >> 172.21.15.158:0/3621247377 conn(0x7f6a6806da40 msgr2=0x7f6a680b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 shutdown_connections 2023-10-16T11:22:18.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.509+0000 7f6a6f61f700 1 -- 172.21.15.158:0/3621247377 wait complete. 2023-10-16T11:22:18.514 INFO:teuthology.orchestra.run.smithi158.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T11:22:18.523 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T11:22:18.543 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-10-16T11:22:18.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.644+0000 7f72fa9ea700 1 Processor -- start 2023-10-16T11:22:18.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.644+0000 7f72fa9ea700 1 -- start start 2023-10-16T11:22:18.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72fa9ea700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f72f4071d30 0x7f72f4161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72fa9ea700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72f40b3520 0x7f72f4153de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72fa9ea700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f40b3ce0 con 0x7f72f40aeff0 2023-10-16T11:22:18.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72fa9ea700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f4162250 con 0x7f72f40b3520 2023-10-16T11:22:18.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72fa9ea700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f40b2db0 con 0x7f72f4071d30 2023-10-16T11:22:18.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f8f87700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72f40b3520 0x7f72f4153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f8f87700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72f40b3520 0x7f72f4153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44792/0 (socket says 172.21.15.158:44792) 2023-10-16T11:22:18.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f8f87700 1 -- 172.21.15.158:0/1789420932 learned_addr learned my addr 172.21.15.158:0/1789420932 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:18.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f37fe700 1 --2- 172.21.15.158:0/1789420932 >> v2:172.21.15.172:3300/0 conn(0x7f72f4071d30 0x7f72f4161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f8f87700 1 -- 172.21.15.158:0/1789420932 >> v2:172.21.15.172:3300/0 conn(0x7f72f4071d30 msgr2=0x7f72f4161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:18.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f8f87700 1 --2- 172.21.15.158:0/1789420932 >> v2:172.21.15.172:3300/0 conn(0x7f72f4071d30 0x7f72f4161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f8f87700 1 -- 172.21.15.158:0/1789420932 >> v1:172.21.15.172:6789/0 conn(0x7f72f40aeff0 legacy=0x7f72f4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f8f87700 1 -- 172.21.15.158:0/1789420932 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f40b3a00 con 0x7f72f40b3520 2023-10-16T11:22:18.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.645+0000 7f72f37fe700 1 --2- 172.21.15.158:0/1789420932 >> v2:172.21.15.172:3300/0 conn(0x7f72f4071d30 0x7f72f4161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:18.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72f8f87700 1 --2- 172.21.15.158:0/1789420932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40b3520 0x7f72f4153de0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f72f406c3e0 tx=0x7f72ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e9df5acee83283 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:18.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72f2ffd700 1 -- 172.21.15.158:0/1789420932 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec00e070 con 0x7f72f40b3520 2023-10-16T11:22:18.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72f2ffd700 1 -- 172.21.15.158:0/1789420932 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec00b9c0 con 0x7f72f40b3520 2023-10-16T11:22:18.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72f2ffd700 1 -- 172.21.15.158:0/1789420932 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec00bd70 con 0x7f72f40b3520 2023-10-16T11:22:18.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 -- 172.21.15.158:0/1789420932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40b3520 msgr2=0x7f72f4153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/1789420932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40b3520 0x7f72f4153de0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f72f406c3e0 tx=0x7f72ec00b330 comp rx=0 tx=0).stop 2023-10-16T11:22:18.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 -- 172.21.15.158:0/1789420932 shutdown_connections 2023-10-16T11:22:18.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/1789420932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40b3520 0x7f72f4153de0 unknown :-1 s=CLOSED pgs=3391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/1789420932 >> v2:172.21.15.172:3300/0 conn(0x7f72f4071d30 0x7f72f4161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 -- 172.21.15.158:0/1789420932 >> 172.21.15.158:0/1789420932 conn(0x7f72f40b1bd0 msgr2=0x7f72f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:18.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 -- 172.21.15.158:0/1789420932 shutdown_connections 2023-10-16T11:22:18.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.646+0000 7f72fa9ea700 1 -- 172.21.15.158:0/1789420932 wait complete. 2023-10-16T11:22:18.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 Processor -- start 2023-10-16T11:22:18.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 -- start start 2023-10-16T11:22:18.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72f4071d30 0x7f72f41f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40aeff0 0x7f72f41f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f40b3520 0x7f72f40b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41f7290 con 0x7f72f40aeff0 2023-10-16T11:22:18.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41f30f0 con 0x7f72f40b3520 2023-10-16T11:22:18.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.648+0000 7f72fa9ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41f33d0 con 0x7f72f4071d30 2023-10-16T11:22:18.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40aeff0 0x7f72f41f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40aeff0 0x7f72f41f6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44796/0 (socket says 172.21.15.158:44796) 2023-10-16T11:22:18.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 -- 172.21.15.158:0/237922555 learned_addr learned my addr 172.21.15.158:0/237922555 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:18.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f8f87700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f40b3520 0x7f72f40b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f3fff700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72f4071d30 0x7f72f41f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 -- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72f4071d30 msgr2=0x7f72f41f5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72f4071d30 0x7f72f41f5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 -- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f40b3520 msgr2=0x7f72f40b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:18.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f40b3520 0x7f72f40b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:18.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 -- 172.21.15.158:0/237922555 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec009b30 con 0x7f72f40aeff0 2023-10-16T11:22:18.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f8f87700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f40b3520 0x7f72f40b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:18.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f37fe700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40aeff0 0x7f72f41f6d50 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e810 tx=0x7f72e400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:18.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f0ff9700 1 -- 172.21.15.158:0/237922555 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4004230 con 0x7f72f40aeff0 2023-10-16T11:22:18.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f0ff9700 1 -- 172.21.15.158:0/237922555 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4004390 con 0x7f72f40aeff0 2023-10-16T11:22:18.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f41f36b0 con 0x7f72f40aeff0 2023-10-16T11:22:18.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72f0ff9700 1 -- 172.21.15.158:0/237922555 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e40036a0 con 0x7f72f40aeff0 2023-10-16T11:22:18.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.649+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f41f3c70 con 0x7f72f40aeff0 2023-10-16T11:22:18.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.650+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f40681b0 con 0x7f72f40aeff0 2023-10-16T11:22:18.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.651+0000 7f72f0ff9700 1 -- 172.21.15.158:0/237922555 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 173) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f72e4003800 con 0x7f72f40aeff0 2023-10-16T11:22:18.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.651+0000 7f72f0ff9700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f72e006ad50 0x7f72e006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:18.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.651+0000 7f72f0ff9700 1 -- 172.21.15.158:0/237922555 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f72e4089320 con 0x7f72f40aeff0 2023-10-16T11:22:18.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.651+0000 7f72f3fff700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f72e006ad50 0x7f72e006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:18.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.651+0000 7f72f3fff700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f72e006ad50 0x7f72e006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72e8007fd0 tx=0x7f72e8009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:18.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.655+0000 7f72f0ff9700 1 -- 172.21.15.158:0/237922555 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72e40540e0 con 0x7f72f40aeff0 2023-10-16T11:22:18.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:18.895+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f72f415daf0 con 0x7f72f40aeff0 2023-10-16T11:22:19.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.512+0000 7f72f0ff9700 1 -- 172.21.15.158:0/237922555 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v156) v1 ==== 104+0+0 (secure 0 0 0) 0x7f72e4057700 con 0x7f72f40aeff0 2023-10-16T11:22:19.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f72e006ad50 msgr2=0x7f72e006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:19.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f72e006ad50 0x7f72e006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72e8007fd0 tx=0x7f72e8009040 comp rx=0 tx=0).stop 2023-10-16T11:22:19.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40aeff0 msgr2=0x7f72f41f6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:19.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40aeff0 0x7f72f41f6d50 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e810 tx=0x7f72e400ebd0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 shutdown_connections 2023-10-16T11:22:19.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f72e006ad50 0x7f72e006d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f40b3520 0x7f72f40b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72f4071d30 0x7f72f41f5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 --2- 172.21.15.158:0/237922555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72f40aeff0 0x7f72f41f6d50 unknown :-1 s=CLOSED pgs=3392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 >> 172.21.15.158:0/237922555 conn(0x7f72f40b1bd0 msgr2=0x7f72f40cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:19.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 shutdown_connections 2023-10-16T11:22:19.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.515+0000 7f72fa9ea700 1 -- 172.21.15.158:0/237922555 wait complete. 2023-10-16T11:22:19.530 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T11:22:19.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.658+0000 7f29f6ea3700 1 Processor -- start 2023-10-16T11:22:19.661 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.659+0000 7f29f6ea3700 1 -- start start 2023-10-16T11:22:19.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.659+0000 7f29f6ea3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f29f00d0640 0x7f29f01686e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:19.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.659+0000 7f29f6ea3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f00d1ee0 con 0x7f29f015daa0 2023-10-16T11:22:19.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.659+0000 7f29f6ea3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f00d12d0 con 0x7f29f015d6c0 2023-10-16T11:22:19.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.659+0000 7f29f6ea3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f00d40a0 con 0x7f29f00d0640 2023-10-16T11:22:19.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.659+0000 7f29effff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f29f015daa0 0x7f29f00cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:44998/0 (socket says 172.21.15.158:44998) 2023-10-16T11:22:19.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.659+0000 7f29effff700 1 -- 172.21.15.158:0/32537504 learned_addr learned my addr 172.21.15.158:0/32537504 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:19.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29f5440700 1 --2- 172.21.15.158:0/32537504 >> v2:172.21.15.172:3300/0 conn(0x7f29f00d0640 0x7f29f01686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:19.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29ef7fe700 1 -- 172.21.15.158:0/32537504 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359437475 0 0) 0x7f29f00d1ee0 con 0x7f29f015daa0 2023-10-16T11:22:19.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29ef7fe700 1 -- 172.21.15.158:0/32537504 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e00035b0 con 0x7f29f015daa0 2023-10-16T11:22:19.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29f5440700 1 -- 172.21.15.158:0/32537504 >> v1:172.21.15.172:6789/0 conn(0x7f29f015d6c0 legacy=0x7f29f00cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:19.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29f5440700 1 -- 172.21.15.158:0/32537504 >> v1:172.21.15.158:6789/0 conn(0x7f29f015daa0 legacy=0x7f29f00cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:19.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29f5440700 1 -- 172.21.15.158:0/32537504 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f00d15b0 con 0x7f29f00d0640 2023-10-16T11:22:19.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29f5440700 1 --2- 172.21.15.158:0/32537504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f00d0640 0x7f29f01686e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f29e8009d90 tx=0x7f29e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ada02ba326a3c567 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:19.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29ef7fe700 1 -- 172.21.15.158:0/32537504 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e800e070 con 0x7f29f00d0640 2023-10-16T11:22:19.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29ef7fe700 1 -- 172.21.15.158:0/32537504 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e800bb20 con 0x7f29f00d0640 2023-10-16T11:22:19.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.660+0000 7f29ef7fe700 1 -- 172.21.15.158:0/32537504 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e800be90 con 0x7f29f00d0640 2023-10-16T11:22:19.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.661+0000 7f29f6ea3700 1 -- 172.21.15.158:0/32537504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f00d0640 msgr2=0x7f29f01686e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:19.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.661+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/32537504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f00d0640 0x7f29f01686e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f29e8009d90 tx=0x7f29e800b330 comp rx=0 tx=0).stop 2023-10-16T11:22:19.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.661+0000 7f29f6ea3700 1 -- 172.21.15.158:0/32537504 shutdown_connections 2023-10-16T11:22:19.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.661+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/32537504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f00d0640 0x7f29f01686e0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.661+0000 7f29f6ea3700 1 -- 172.21.15.158:0/32537504 >> 172.21.15.158:0/32537504 conn(0x7f29f009c940 msgr2=0x7f29f00d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:19.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.661+0000 7f29f6ea3700 1 -- 172.21.15.158:0/32537504 shutdown_connections 2023-10-16T11:22:19.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.661+0000 7f29f6ea3700 1 -- 172.21.15.158:0/32537504 wait complete. 2023-10-16T11:22:19.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.662+0000 7f29f6ea3700 1 Processor -- start 2023-10-16T11:22:19.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.662+0000 7f29f6ea3700 1 -- start start 2023-10-16T11:22:19.671 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f6ea3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29f00d0640 0x7f29f00b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:19.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f6ea3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f015d6c0 0x7f29f015f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:19.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f6ea3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29f015daa0 0x7f29f00c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:19.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f6ea3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0160eb0 con 0x7f29f015daa0 2023-10-16T11:22:19.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f6ea3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f015fc60 con 0x7f29f015d6c0 2023-10-16T11:22:19.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f6ea3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0160bd0 con 0x7f29f00d0640 2023-10-16T11:22:19.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29f015daa0 0x7f29f00c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:19.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29f015daa0 0x7f29f00c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44830/0 (socket says 172.21.15.158:44830) 2023-10-16T11:22:19.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 -- 172.21.15.158:0/2842113588 learned_addr learned my addr 172.21.15.158:0/2842113588 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:19.673 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29effff700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f015d6c0 0x7f29f015f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:19.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f4c3f700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29f00d0640 0x7f29f00b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:19.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 -- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29f00d0640 msgr2=0x7f29f00b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:19.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29f00d0640 0x7f29f00b5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 -- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f015d6c0 msgr2=0x7f29f015f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:19.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f015d6c0 0x7f29f015f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:19.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.663+0000 7f29f5440700 1 -- 172.21.15.158:0/2842113588 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e8009a70 con 0x7f29f015daa0 2023-10-16T11:22:19.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.664+0000 7f29f5440700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29f015daa0 0x7f29f00c7100 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7f29e800b370 tx=0x7f29e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:19.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.664+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8004800 con 0x7f29f015daa0 2023-10-16T11:22:19.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.664+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e8020070 con 0x7f29f015daa0 2023-10-16T11:22:19.675 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.664+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f00b4ba0 con 0x7f29f015daa0 2023-10-16T11:22:19.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.664+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e80276c0 con 0x7f29f015daa0 2023-10-16T11:22:19.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.664+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f00b5160 con 0x7f29f015daa0 2023-10-16T11:22:19.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.665+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f00681b0 con 0x7f29f015daa0 2023-10-16T11:22:19.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.665+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 173) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f29e8012430 con 0x7f29f015daa0 2023-10-16T11:22:19.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.666+0000 7f29ed7fa700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29d406ad50 0x7f29d406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:19.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.666+0000 7f29f4c3f700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29d406ad50 0x7f29d406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:19.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.666+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f29e8022070 con 0x7f29f015daa0 2023-10-16T11:22:19.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.666+0000 7f29f4c3f700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29d406ad50 0x7f29d406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f29f015e160 tx=0x7f29e4009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:19.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.670+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29e8056220 con 0x7f29f015daa0 2023-10-16T11:22:19.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:19.863+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f29f00b9490 con 0x7f29f015daa0 2023-10-16T11:22:20.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.031+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 174) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f29e80565f0 con 0x7f29f015daa0 2023-10-16T11:22:20.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.657+0000 7f29ed7fa700 1 -- 172.21.15.158:0/2842113588 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v274) v1 ==== 181+0+0 (secure 0 0 0) 0x7f29e806de00 con 0x7f29f015daa0 2023-10-16T11:22:20.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29d406ad50 msgr2=0x7f29d406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:20.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29d406ad50 0x7f29d406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f29f015e160 tx=0x7f29e4009040 comp rx=0 tx=0).stop 2023-10-16T11:22:20.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29f015daa0 msgr2=0x7f29f00c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:20.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29f015daa0 0x7f29f00c7100 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7f29e800b370 tx=0x7f29e8004ad0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 shutdown_connections 2023-10-16T11:22:20.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29d406ad50 0x7f29d406d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29f015daa0 0x7f29f00c7100 unknown :-1 s=CLOSED pgs=3396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f29f00d0640 0x7f29f00b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 --2- 172.21.15.158:0/2842113588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29f015d6c0 0x7f29f015f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 >> 172.21.15.158:0/2842113588 conn(0x7f29f009c940 msgr2=0x7f29f00d26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:20.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 shutdown_connections 2023-10-16T11:22:20.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.660+0000 7f29f6ea3700 1 -- 172.21.15.158:0/2842113588 wait complete. 2023-10-16T11:22:20.665 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_metadata' removed 2023-10-16T11:22:20.677 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T11:22:20.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.811+0000 7fa9239d9700 1 Processor -- start 2023-10-16T11:22:20.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.811+0000 7fa9239d9700 1 -- start start 2023-10-16T11:22:20.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa9239d9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa91c15b590 0x7fa91c0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:20.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa9239d9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa91c15f1d0 0x7fa91c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:20.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa9239d9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c0cbaf0 0x7fa91c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:20.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa9239d9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c0cc4c0 con 0x7fa91c15b590 2023-10-16T11:22:20.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa9239d9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c0cf4c0 con 0x7fa91c15f1d0 2023-10-16T11:22:20.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa9239d9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c15f970 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa91c15b590 0x7fa91c0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:20.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa91c15b590 0x7fa91c0ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44834/0 (socket says 172.21.15.158:44834) 2023-10-16T11:22:20.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 -- 172.21.15.158:0/313809329 learned_addr learned my addr 172.21.15.158:0/313809329 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921f76700 1 --2- 172.21.15.158:0/313809329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c0cbaf0 0x7fa91c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa920f74700 1 --2- 172.21.15.158:0/313809329 >> v2:172.21.15.172:3300/0 conn(0x7fa91c15f1d0 0x7fa91c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 -- 172.21.15.158:0/313809329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c0cbaf0 msgr2=0x7fa91c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 --2- 172.21.15.158:0/313809329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c0cbaf0 0x7fa91c0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 -- 172.21.15.158:0/313809329 >> v2:172.21.15.172:3300/0 conn(0x7fa91c15f1d0 msgr2=0x7fa91c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 --2- 172.21.15.158:0/313809329 >> v2:172.21.15.172:3300/0 conn(0x7fa91c15f1d0 0x7fa91c0b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 -- 172.21.15.158:0/313809329 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c15fc50 con 0x7fa91c15b590 2023-10-16T11:22:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa921775700 1 --2- 172.21.15.158:0/313809329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c15b590 0x7fa91c0ce510 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0d1360 tx=0x7fa918009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548c61ffc6bbb153 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.812+0000 7fa913fff700 1 -- 172.21.15.158:0/313809329 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91800b870 con 0x7fa91c15b590 2023-10-16T11:22:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa913fff700 1 -- 172.21.15.158:0/313809329 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91800b9d0 con 0x7fa91c15b590 2023-10-16T11:22:20.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa913fff700 1 -- 172.21.15.158:0/313809329 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91800bd60 con 0x7fa91c15b590 2023-10-16T11:22:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 -- 172.21.15.158:0/313809329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c15b590 msgr2=0x7fa91c0ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 --2- 172.21.15.158:0/313809329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c15b590 0x7fa91c0ce510 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0d1360 tx=0x7fa918009bd0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 -- 172.21.15.158:0/313809329 shutdown_connections 2023-10-16T11:22:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 --2- 172.21.15.158:0/313809329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c0cbaf0 0x7fa91c0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 --2- 172.21.15.158:0/313809329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c15b590 0x7fa91c0ce510 unknown :-1 s=CLOSED pgs=3397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 --2- 172.21.15.158:0/313809329 >> v2:172.21.15.172:3300/0 conn(0x7fa91c15f1d0 0x7fa91c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 -- 172.21.15.158:0/313809329 >> 172.21.15.158:0/313809329 conn(0x7fa91c06a8d0 msgr2=0x7fa91c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 -- 172.21.15.158:0/313809329 shutdown_connections 2023-10-16T11:22:20.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.813+0000 7fa9239d9700 1 -- 172.21.15.158:0/313809329 wait complete. 2023-10-16T11:22:20.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.814+0000 7fa9239d9700 1 Processor -- start 2023-10-16T11:22:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.814+0000 7fa9239d9700 1 -- start start 2023-10-16T11:22:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa9239d9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0cbaf0 0x7fa91c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa9239d9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa91c15b590 0x7fa91c0c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa9239d9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c15f1d0 0x7fa91c157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:20.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa9239d9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c06cc60 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa9239d9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c06c050 con 0x7fa91c15b590 2023-10-16T11:22:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa9239d9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c06c330 con 0x7fa91c15f1d0 2023-10-16T11:22:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa921775700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0cbaf0 0x7fa91c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa921775700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0cbaf0 0x7fa91c06b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44840/0 (socket says 172.21.15.158:44840) 2023-10-16T11:22:20.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.815+0000 7fa921775700 1 -- 172.21.15.158:0/3416214761 learned_addr learned my addr 172.21.15.158:0/3416214761 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa921775700 1 -- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c15f1d0 msgr2=0x7fa91c157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa921f76700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c15f1d0 0x7fa91c157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa920f74700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa91c15b590 0x7fa91c0c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa921775700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c15f1d0 0x7fa91c157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa921775700 1 -- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa91c15b590 msgr2=0x7fa91c0c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:20.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa921775700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa91c15b590 0x7fa91c0c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:20.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa921775700 1 -- 172.21.15.158:0/3416214761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918009550 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa921775700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0cbaf0 0x7fa91c06b9e0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7fa918000c00 tx=0x7fa918004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:20.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa911ffb700 1 -- 172.21.15.158:0/3416214761 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918004540 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa911ffb700 1 -- 172.21.15.158:0/3416214761 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9180036a0 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa911ffb700 1 -- 172.21.15.158:0/3416214761 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918003a50 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c06c610 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.816+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c0c9660 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.817+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c0681b0 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.817+0000 7fa911ffb700 1 -- 172.21.15.158:0/3416214761 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 174) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa918021070 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.818+0000 7fa911ffb700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa90006af90 0x7fa90006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:20.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.818+0000 7fa911ffb700 1 -- 172.21.15.158:0/3416214761 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6845+0+0 (secure 0 0 0) 0x7fa91808b080 con 0x7fa91c0cbaf0 2023-10-16T11:22:20.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.818+0000 7fa920f74700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa90006af90 0x7fa90006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:20.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.818+0000 7fa920f74700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa90006af90 0x7fa90006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0b41e0 tx=0x7fa90c008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:20.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:20.822+0000 7fa911ffb700 1 -- 172.21.15.158:0/3416214761 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa918055e80 con 0x7fa91c0cbaf0 2023-10-16T11:22:21.019 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.017+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa91c0b07c0 con 0x7fa91c0cbaf0 2023-10-16T11:22:21.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.658+0000 7fa911ffb700 1 -- 172.21.15.158:0/3416214761 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v275) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa9180150d0 con 0x7fa91c0cbaf0 2023-10-16T11:22:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.662+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa90006af90 msgr2=0x7fa90006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.662+0000 7fa9239d9700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa90006af90 0x7fa90006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0b41e0 tx=0x7fa90c008040 comp rx=0 tx=0).stop 2023-10-16T11:22:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.662+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0cbaf0 msgr2=0x7fa91c06b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.662+0000 7fa9239d9700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0cbaf0 0x7fa91c06b9e0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7fa918000c00 tx=0x7fa918004750 comp rx=0 tx=0).stop 2023-10-16T11:22:21.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 shutdown_connections 2023-10-16T11:22:21.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa90006af90 0x7fa90006d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa91c15f1d0 0x7fa91c157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0cbaf0 0x7fa91c06b9e0 unknown :-1 s=CLOSED pgs=3398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 --2- 172.21.15.158:0/3416214761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa91c15b590 0x7fa91c0c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 >> 172.21.15.158:0/3416214761 conn(0x7fa91c06a8d0 msgr2=0x7fa91c0b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:21.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 shutdown_connections 2023-10-16T11:22:21.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.663+0000 7fa9239d9700 1 -- 172.21.15.158:0/3416214761 wait complete. 2023-10-16T11:22:21.668 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'cephfs_data' removed 2023-10-16T11:22:21.681 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T11:22:21.812 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.808+0000 7fe90c502700 1 Processor -- start 2023-10-16T11:22:21.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.808+0000 7fe90c502700 1 -- start start 2023-10-16T11:22:21.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe90c502700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe90406afe0 0x7fe9040caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:21.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe90c502700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9040cc590 0x7fe9040afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:21.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe90c502700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9040cbd30 con 0x7fe90415da40 2023-10-16T11:22:21.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe90c502700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9040cc1c0 con 0x7fe90406afe0 2023-10-16T11:22:21.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe90c502700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415de20 con 0x7fe9040cc590 2023-10-16T11:22:21.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe90406afe0 0x7fe9040caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:21.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe90406afe0 0x7fe9040caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44850/0 (socket says 172.21.15.158:44850) 2023-10-16T11:22:21.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 -- 172.21.15.158:0/4070434438 learned_addr learned my addr 172.21.15.158:0/4070434438 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:21.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe90aa9f700 1 --2- 172.21.15.158:0/4070434438 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9040cc590 0x7fe9040afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:21.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 -- 172.21.15.158:0/4070434438 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9040cc590 msgr2=0x7fe9040afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:21.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 --2- 172.21.15.158:0/4070434438 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9040cc590 0x7fe9040afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 -- 172.21.15.158:0/4070434438 >> v1:172.21.15.172:6789/0 conn(0x7fe90415da40 legacy=0x7fe90406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:21.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 -- 172.21.15.158:0/4070434438 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9040ce260 con 0x7fe90406afe0 2023-10-16T11:22:21.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe909a9d700 1 --2- 172.21.15.158:0/4070434438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe90406afe0 0x7fe9040caa20 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40098f0 tx=0x7fe8f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de578eef1389896 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:21.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.809+0000 7fe90929c700 1 -- 172.21.15.158:0/4070434438 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f400b8a0 con 0x7fe90406afe0 2023-10-16T11:22:21.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90929c700 1 -- 172.21.15.158:0/4070434438 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f400ba00 con 0x7fe90406afe0 2023-10-16T11:22:21.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90929c700 1 -- 172.21.15.158:0/4070434438 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f400bd90 con 0x7fe90406afe0 2023-10-16T11:22:21.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 -- 172.21.15.158:0/4070434438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe90406afe0 msgr2=0x7fe9040caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:21.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 --2- 172.21.15.158:0/4070434438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe90406afe0 0x7fe9040caa20 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40098f0 tx=0x7fe8f4009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:21.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 -- 172.21.15.158:0/4070434438 shutdown_connections 2023-10-16T11:22:21.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 --2- 172.21.15.158:0/4070434438 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9040cc590 0x7fe9040afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 --2- 172.21.15.158:0/4070434438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe90406afe0 0x7fe9040caa20 unknown :-1 s=CLOSED pgs=3399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 -- 172.21.15.158:0/4070434438 >> 172.21.15.158:0/4070434438 conn(0x7fe904099a10 msgr2=0x7fe9040d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:21.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 -- 172.21.15.158:0/4070434438 shutdown_connections 2023-10-16T11:22:21.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.810+0000 7fe90c502700 1 -- 172.21.15.158:0/4070434438 wait complete. 2023-10-16T11:22:21.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.812+0000 7fe90c502700 1 Processor -- start 2023-10-16T11:22:21.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.812+0000 7fe90c502700 1 -- start start 2023-10-16T11:22:21.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe90406afe0 0x7fe90415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:21.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9040cc590 0x7fe90415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:21.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe90415da40 0x7fe90406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:21.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415d090 con 0x7fe9040cc590 2023-10-16T11:22:21.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415c7f0 con 0x7fe90406afe0 2023-10-16T11:22:21.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415cad0 con 0x7fe90415da40 2023-10-16T11:22:21.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9040cc590 0x7fe90415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:21.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9040cc590 0x7fe90415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44856/0 (socket says 172.21.15.158:44856) 2023-10-16T11:22:21.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 -- 172.21.15.158:0/3719297884 learned_addr learned my addr 172.21.15.158:0/3719297884 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:21.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90aa9f700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe90415da40 0x7fe90406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:21.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90a29e700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe90406afe0 0x7fe90415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:21.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 -- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe90415da40 msgr2=0x7fe90406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:21.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe90415da40 0x7fe90406da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 -- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe90406afe0 msgr2=0x7fe90415b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:21.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe90406afe0 0x7fe90415b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:21.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 -- 172.21.15.158:0/3719297884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4009550 con 0x7fe9040cc590 2023-10-16T11:22:21.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe909a9d700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9040cc590 0x7fe90415be80 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009c40 tx=0x7fe8f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:21.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe8faffd700 1 -- 172.21.15.158:0/3719297884 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40036a0 con 0x7fe9040cc590 2023-10-16T11:22:21.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe8faffd700 1 -- 172.21.15.158:0/3719297884 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f401e070 con 0x7fe9040cc590 2023-10-16T11:22:21.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90415cdb0 con 0x7fe9040cc590 2023-10-16T11:22:21.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe8faffd700 1 -- 172.21.15.158:0/3719297884 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4003a60 con 0x7fe9040cc590 2023-10-16T11:22:21.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.813+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90415c3c0 con 0x7fe9040cc590 2023-10-16T11:22:21.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.814+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90415c550 con 0x7fe9040cc590 2023-10-16T11:22:21.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.815+0000 7fe8faffd700 1 -- 172.21.15.158:0/3719297884 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 174) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe8f4003bc0 con 0x7fe9040cc590 2023-10-16T11:22:21.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.815+0000 7fe8faffd700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fe8e806af90 0x7fe8e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:21.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.815+0000 7fe8faffd700 1 -- 172.21.15.158:0/3719297884 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe8f4020070 con 0x7fe9040cc590 2023-10-16T11:22:21.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.815+0000 7fe90a29e700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fe8e806af90 0x7fe8e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:21.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.815+0000 7fe90a29e700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fe8e806af90 0x7fe8e806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe900000f60 tx=0x7fe900009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:21.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:21.819+0000 7fe8faffd700 1 -- 172.21.15.158:0/3719297884 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8f4055bd0 con 0x7fe9040cc590 2023-10-16T11:22:22.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.012+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe9040c8c40 con 0x7fe9040cc590 2023-10-16T11:22:22.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.016+0000 7fe8faffd700 1 -- 172.21.15.158:0/3719297884 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v97) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe8f40591f0 con 0x7fe9040cc590 2023-10-16T11:22:22.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fe8e806af90 msgr2=0x7fe8e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fe8e806af90 0x7fe8e806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe900000f60 tx=0x7fe900009040 comp rx=0 tx=0).stop 2023-10-16T11:22:22.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9040cc590 msgr2=0x7fe90415be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9040cc590 0x7fe90415be80 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009c40 tx=0x7fe8f4004750 comp rx=0 tx=0).stop 2023-10-16T11:22:22.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 shutdown_connections 2023-10-16T11:22:22.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fe8e806af90 0x7fe8e806d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe90415da40 0x7fe90406da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe90406afe0 0x7fe90415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 --2- 172.21.15.158:0/3719297884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9040cc590 0x7fe90415be80 unknown :-1 s=CLOSED pgs=3400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 >> 172.21.15.158:0/3719297884 conn(0x7fe904099a10 msgr2=0x7fe9040ca6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:22.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 shutdown_connections 2023-10-16T11:22:22.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.018+0000 7fe90c502700 1 -- 172.21.15.158:0/3719297884 wait complete. 2023-10-16T11:22:22.033 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T11:22:22.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.156+0000 7ff4091ea700 1 Processor -- start 2023-10-16T11:22:22.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.156+0000 7ff4091ea700 1 -- start start 2023-10-16T11:22:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff4091ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 0x7ff4040ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff4091ea700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040cf4c0 con 0x7ff40415f1d0 2023-10-16T11:22:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff4091ea700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40415f970 con 0x7ff4040cbaf0 2023-10-16T11:22:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff4091ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040b49e0 con 0x7ff40415b590 2023-10-16T11:22:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff40259c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff40415f1d0 0x7ff4040b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45038/0 (socket says 172.21.15.158:45038) 2023-10-16T11:22:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff40259c700 1 -- 172.21.15.158:0/2939846883 learned_addr learned my addr 172.21.15.158:0/2939846883 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:22.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062561378 0 0) 0x7ff4040cf4c0 con 0x7ff40415f1d0 2023-10-16T11:22:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec0035b0 con 0x7ff40415f1d0 2023-10-16T11:22:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.157+0000 7ff402d9d700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 0x7ff4040ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490338876 0 0) 0x7ff3ec0035b0 con 0x7ff40415f1d0 2023-10-16T11:22:22.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 msgr2=0x7ff4040ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 0x7ff4040ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 >> v1:172.21.15.172:6789/0 conn(0x7ff4040cbaf0 legacy=0x7ff4040cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40415fc50 con 0x7ff40415f1d0 2023-10-16T11:22:22.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7ff3f4001670 con 0x7ff40415f1d0 2023-10-16T11:22:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 >> v1:172.21.15.158:6789/0 conn(0x7ff40415f1d0 legacy=0x7ff4040b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff3ec003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3ec004850 0x7ff3ec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 0x7ff3ec005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040b49e0 con 0x7ff3ec003a30 2023-10-16T11:22:22.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec00bb40 con 0x7ff3ec004850 2023-10-16T11:22:22.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec00bd00 con 0x7ff40415b590 2023-10-16T11:22:22.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff402d9d700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff3ec003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff40259c700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3ec004850 0x7ff3ec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff40359e700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 0x7ff3ec005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff402d9d700 1 -- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 msgr2=0x7ff3ec005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff402d9d700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 0x7ff3ec005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff402d9d700 1 -- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3ec004850 msgr2=0x7ff3ec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff402d9d700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3ec004850 0x7ff3ec004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff402d9d700 1 -- 172.21.15.158:0/2939846883 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0035b0 con 0x7ff3ec003a30 2023-10-16T11:22:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff40359e700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff40415b590 0x7ff3ec005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff402d9d700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff3ec003e50 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009b60 tx=0x7ff3ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6c13811b3c0770 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800e6c0 con 0x7ff3ec003a30 2023-10-16T11:22:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f800e820 con 0x7ff3ec003a30 2023-10-16T11:22:22.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.158+0000 7ff401d9b700 1 -- 172.21.15.158:0/2939846883 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800eb90 con 0x7ff3ec003a30 2023-10-16T11:22:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.159+0000 7ff4091ea700 1 -- 172.21.15.158:0/2939846883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 msgr2=0x7ff3ec003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.159+0000 7ff4091ea700 1 --2- 172.21.15.158:0/2939846883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff3ec003e50 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009b60 tx=0x7ff3ec005880 comp rx=0 tx=0).stop 2023-10-16T11:22:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.159+0000 7ff40359e700 1 -- 172.21.15.158:0/2939846883 reap_dead start 2023-10-16T11:22:22.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.159+0000 7ff4091ea700 1 -- 172.21.15.158:0/2939846883 shutdown_connections 2023-10-16T11:22:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.159+0000 7ff4091ea700 1 -- 172.21.15.158:0/2939846883 >> 172.21.15.158:0/2939846883 conn(0x7ff40406a8d0 msgr2=0x7ff40406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.159+0000 7ff4091ea700 1 -- 172.21.15.158:0/2939846883 shutdown_connections 2023-10-16T11:22:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.159+0000 7ff4091ea700 1 -- 172.21.15.158:0/2939846883 wait complete. 2023-10-16T11:22:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.160+0000 7ff4091ea700 1 Processor -- start 2023-10-16T11:22:22.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.160+0000 7ff4091ea700 1 -- start start 2023-10-16T11:22:22.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff4091ea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff4040a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff4091ea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40415d620 0x7ff404158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff4091ea700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca740 0x7ff4040cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff4091ea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040cb130 con 0x7ff3ec003a30 2023-10-16T11:22:22.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff4091ea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040a8820 con 0x7ff40415d620 2023-10-16T11:22:22.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff4091ea700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40415ebc0 con 0x7ff4040ca740 2023-10-16T11:22:22.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff402d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff4040a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff402d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff4040a93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44874/0 (socket says 172.21.15.158:44874) 2023-10-16T11:22:22.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.161+0000 7ff402d9d700 1 -- 172.21.15.158:0/3649964610 learned_addr learned my addr 172.21.15.158:0/3649964610 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:22.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff40259c700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40415d620 0x7ff404158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff40359e700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca740 0x7ff4040cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff402d9d700 1 -- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca740 msgr2=0x7ff4040cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff402d9d700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca740 0x7ff4040cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff402d9d700 1 -- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40415d620 msgr2=0x7ff404158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff402d9d700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40415d620 0x7ff404158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff402d9d700 1 -- 172.21.15.158:0/3649964610 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040a98e0 con 0x7ff3ec003a30 2023-10-16T11:22:22.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff40259c700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40415d620 0x7ff404158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:22.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff40359e700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca740 0x7ff4040cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:22.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff402d9d700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff4040a93a0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00bb10 tx=0x7ff3f80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff3eb7fe700 1 -- 172.21.15.158:0/3649964610 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800e6c0 con 0x7ff3ec003a30 2023-10-16T11:22:22.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff3eb7fe700 1 -- 172.21.15.158:0/3649964610 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f8003ad0 con 0x7ff3ec003a30 2023-10-16T11:22:22.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff404158c50 con 0x7ff3ec003a30 2023-10-16T11:22:22.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff3eb7fe700 1 -- 172.21.15.158:0/3649964610 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8003e80 con 0x7ff3ec003a30 2023-10-16T11:22:22.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.162+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40415de00 con 0x7ff3ec003a30 2023-10-16T11:22:22.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.163+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40415ea10 con 0x7ff3ec003a30 2023-10-16T11:22:22.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.163+0000 7ff3eb7fe700 1 -- 172.21.15.158:0/3649964610 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff3f8013470 con 0x7ff3ec003a30 2023-10-16T11:22:22.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.164+0000 7ff3eb7fe700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff3f006b020 0x7ff3f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.164+0000 7ff3eb7fe700 1 -- 172.21.15.158:0/3649964610 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff3f808b280 con 0x7ff3ec003a30 2023-10-16T11:22:22.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.164+0000 7ff40259c700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff3f006b020 0x7ff3f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.164+0000 7ff40259c700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff3f006b020 0x7ff3f006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4001490 tx=0x7ff3f4009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.168+0000 7ff3eb7fe700 1 -- 172.21.15.158:0/3649964610 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3f8056100 con 0x7ff3ec003a30 2023-10-16T11:22:22.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.362+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff4040cb130 con 0x7ff3ec003a30 2023-10-16T11:22:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.365+0000 7ff3eb7fe700 1 -- 172.21.15.158:0/3649964610 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v98) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff4040cb130 con 0x7ff3ec003a30 2023-10-16T11:22:22.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff3f006b020 msgr2=0x7ff3f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff3f006b020 0x7ff3f006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4001490 tx=0x7ff3f4009040 comp rx=0 tx=0).stop 2023-10-16T11:22:22.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 msgr2=0x7ff4040a93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff4040a93a0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00bb10 tx=0x7ff3f80047c0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 shutdown_connections 2023-10-16T11:22:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff3f006b020 0x7ff3f006d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4040ca740 0x7ff4040cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3ec003a30 0x7ff4040a93a0 unknown :-1 s=CLOSED pgs=3403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 --2- 172.21.15.158:0/3649964610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff40415d620 0x7ff404158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 >> 172.21.15.158:0/3649964610 conn(0x7ff40406a8d0 msgr2=0x7ff40406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 shutdown_connections 2023-10-16T11:22:22.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.368+0000 7ff4091ea700 1 -- 172.21.15.158:0/3649964610 wait complete. 2023-10-16T11:22:22.382 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T11:22:22.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.502+0000 7f21bdda0700 1 Processor -- start 2023-10-16T11:22:22.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.502+0000 7f21bdda0700 1 -- start start 2023-10-16T11:22:22.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.503+0000 7f21bdda0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 0x7f21b80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.503+0000 7f21bdda0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21b806afe0 0x7f21b80afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.503+0000 7f21bdda0700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b80cbd30 con 0x7f21b80cc590 2023-10-16T11:22:22.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.503+0000 7f21bdda0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b80cc1c0 con 0x7f21b806afe0 2023-10-16T11:22:22.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.503+0000 7f21bdda0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b815de20 con 0x7f21b815da40 2023-10-16T11:22:22.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.503+0000 7f21b6ffd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f21b80cc590 0x7f21b806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45060/0 (socket says 172.21.15.158:45060) 2023-10-16T11:22:22.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.503+0000 7f21b6ffd700 1 -- 172.21.15.158:0/505641740 learned_addr learned my addr 172.21.15.158:0/505641740 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:22.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b7fff700 1 --2- 172.21.15.158:0/505641740 >> v2:172.21.15.172:3300/0 conn(0x7f21b806afe0 0x7f21b80afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b77fe700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 0x7f21b80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437489069 0 0) 0x7f21b80cbd30 con 0x7f21b80cc590 2023-10-16T11:22:22.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a00035b0 con 0x7f21b80cc590 2023-10-16T11:22:22.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336419023 0 0) 0x7f21a00035b0 con 0x7f21b80cc590 2023-10-16T11:22:22.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 msgr2=0x7f21b80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 0x7f21b80caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 >> v2:172.21.15.172:3300/0 conn(0x7f21b806afe0 msgr2=0x7f21b80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 --2- 172.21.15.158:0/505641740 >> v2:172.21.15.172:3300/0 conn(0x7f21b806afe0 0x7f21b80afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80ce260 con 0x7f21b80cc590 2023-10-16T11:22:22.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f21a8001670 con 0x7f21b80cc590 2023-10-16T11:22:22.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 >> v1:172.21.15.158:6789/0 conn(0x7f21b80cc590 legacy=0x7f21b806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a0003a00 0x7f21a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21a0004850 0x7f21a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 0x7f21a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80cc1c0 con 0x7f21a0003a00 2023-10-16T11:22:22.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b815de20 con 0x7f21a0004850 2023-10-16T11:22:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a000ba10 con 0x7f21b815da40 2023-10-16T11:22:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b77fe700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a0003a00 0x7f21a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b6ffd700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21a0004850 0x7f21a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b7fff700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 0x7f21a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b77fe700 1 -- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 msgr2=0x7f21a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b77fe700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 0x7f21a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b77fe700 1 -- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21a0004850 msgr2=0x7f21a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b77fe700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21a0004850 0x7f21a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b77fe700 1 -- 172.21.15.158:0/505641740 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a00035b0 con 0x7f21a0003a00 2023-10-16T11:22:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.504+0000 7f21b7fff700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21b815da40 0x7f21a0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21b77fe700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a0003a00 0x7f21a0003e20 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009b60 tx=0x7f21a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59ef00554c8002c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00e6c0 con 0x7f21a0003a00 2023-10-16T11:22:22.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac00e820 con 0x7f21a0003a00 2023-10-16T11:22:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21b67fc700 1 -- 172.21.15.158:0/505641740 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00eb90 con 0x7f21a0003a00 2023-10-16T11:22:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21bdda0700 1 -- 172.21.15.158:0/505641740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a0003a00 msgr2=0x7f21a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21bdda0700 1 --2- 172.21.15.158:0/505641740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a0003a00 0x7f21a0003e20 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009b60 tx=0x7f21a0005880 comp rx=0 tx=0).stop 2023-10-16T11:22:22.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21b7fff700 1 -- 172.21.15.158:0/505641740 reap_dead start 2023-10-16T11:22:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21bdda0700 1 -- 172.21.15.158:0/505641740 shutdown_connections 2023-10-16T11:22:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21bdda0700 1 -- 172.21.15.158:0/505641740 >> 172.21.15.158:0/505641740 conn(0x7f21b8099a10 msgr2=0x7f21b80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21bdda0700 1 -- 172.21.15.158:0/505641740 shutdown_connections 2023-10-16T11:22:22.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.505+0000 7f21bdda0700 1 -- 172.21.15.158:0/505641740 wait complete. 2023-10-16T11:22:22.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.506+0000 7f21bdda0700 1 Processor -- start 2023-10-16T11:22:22.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.507+0000 7f21bdda0700 1 -- start start 2023-10-16T11:22:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.507+0000 7f21bdda0700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21a0003a00 0x7f21b81f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.507+0000 7f21bdda0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b81f5d10 0x7f21b815a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.507+0000 7f21bdda0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21b80cf940 0x7f21b8149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.507+0000 7f21bdda0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b815aa90 con 0x7f21b80cf940 2023-10-16T11:22:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.507+0000 7f21bdda0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b81f2f60 con 0x7f21b81f5d10 2023-10-16T11:22:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.507+0000 7f21bdda0700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b81f3240 con 0x7f21a0003a00 2023-10-16T11:22:22.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21b80cf940 0x7f21b8149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21b80cf940 0x7f21b8149240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44898/0 (socket says 172.21.15.158:44898) 2023-10-16T11:22:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 -- 172.21.15.158:0/621659387 learned_addr learned my addr 172.21.15.158:0/621659387 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 -- 172.21.15.158:0/621659387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21a0003a00 msgr2=0x7f21b81f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21a0003a00 0x7f21b81f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 -- 172.21.15.158:0/621659387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b81f5d10 msgr2=0x7f21b815a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b6ffd700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b81f5d10 0x7f21b815a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b81f5d10 0x7f21b815a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 -- 172.21.15.158:0/621659387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b81f3520 con 0x7f21b80cf940 2023-10-16T11:22:22.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21b7fff700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21b80cf940 0x7f21b8149240 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f21b80cf500 tx=0x7f21b00114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f219ffff700 1 -- 172.21.15.158:0/621659387 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b000e030 con 0x7f21b80cf940 2023-10-16T11:22:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f219ffff700 1 -- 172.21.15.158:0/621659387 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b0011ce0 con 0x7f21b80cf940 2023-10-16T11:22:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21bdda0700 1 -- 172.21.15.158:0/621659387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b81f3800 con 0x7f21b80cf940 2023-10-16T11:22:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f219ffff700 1 -- 172.21.15.158:0/621659387 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b00166b0 con 0x7f21b80cf940 2023-10-16T11:22:22.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.508+0000 7f21bdda0700 1 -- 172.21.15.158:0/621659387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b80cfdd0 con 0x7f21b80cf940 2023-10-16T11:22:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.509+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21900052f0 con 0x7f21b80cf940 2023-10-16T11:22:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.510+0000 7f219ffff700 1 -- 172.21.15.158:0/621659387 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f21b0016810 con 0x7f21b80cf940 2023-10-16T11:22:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.510+0000 7f219ffff700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f21a406b020 0x7f21a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.510+0000 7f21b77fe700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f21a406b020 0x7f21a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.510+0000 7f219ffff700 1 -- 172.21.15.158:0/621659387 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f21b000f070 con 0x7f21b80cf940 2023-10-16T11:22:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.511+0000 7f21b77fe700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f21a406b020 0x7f21a406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f21ac000c00 tx=0x7f21ac013000 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.515+0000 7f219ffff700 1 -- 172.21.15.158:0/621659387 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21b0054d70 con 0x7f21b80cf940 2023-10-16T11:22:22.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.716+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2190005640 con 0x7f21b80cf940 2023-10-16T11:22:22.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.719+0000 7f219ffff700 1 -- 172.21.15.158:0/621659387 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v99) v1 ==== 120+0+0 (secure 0 0 0) 0x7f21b0058390 con 0x7f21b80cf940 2023-10-16T11:22:22.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.721+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f21a406b020 msgr2=0x7f21a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.721+0000 7f219dffb700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f21a406b020 0x7f21a406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f21ac000c00 tx=0x7f21ac013000 comp rx=0 tx=0).stop 2023-10-16T11:22:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.721+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21b80cf940 msgr2=0x7f21b8149240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.721+0000 7f219dffb700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21b80cf940 0x7f21b8149240 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f21b80cf500 tx=0x7f21b00114b0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 shutdown_connections 2023-10-16T11:22:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f21a406b020 0x7f21a406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21b80cf940 0x7f21b8149240 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21a0003a00 0x7f21b81f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 --2- 172.21.15.158:0/621659387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b81f5d10 0x7f21b815a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 >> 172.21.15.158:0/621659387 conn(0x7f21b8099a10 msgr2=0x7f21b80c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:22.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 shutdown_connections 2023-10-16T11:22:22.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.722+0000 7f219dffb700 1 -- 172.21.15.158:0/621659387 wait complete. 2023-10-16T11:22:22.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments' 2023-10-16T11:22:22.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.855+0000 7f7ba6720700 1 Processor -- start 2023-10-16T11:22:22.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.855+0000 7f7ba6720700 1 -- start start 2023-10-16T11:22:22.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7ba6720700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ba006ab40 0x7f7ba006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7ba6720700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ba006e5d0 0x7f7ba0071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7ba6720700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba0071e50 con 0x7f7ba006af20 2023-10-16T11:22:22.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7ba6720700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba0161270 con 0x7f7ba006e5d0 2023-10-16T11:22:22.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7ba6720700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba0161510 con 0x7f7ba006ab40 2023-10-16T11:22:22.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7b9f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ba006e5d0 0x7f7ba0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7b9f7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ba006e5d0 0x7f7ba0071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44902/0 (socket says 172.21.15.158:44902) 2023-10-16T11:22:22.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.856+0000 7f7b9f7fe700 1 -- 172.21.15.158:0/973970703 learned_addr learned my addr 172.21.15.158:0/973970703 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:22.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9ffff700 1 --2- 172.21.15.158:0/973970703 >> v2:172.21.15.172:3300/0 conn(0x7f7ba006ab40 0x7f7ba006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9f7fe700 1 -- 172.21.15.158:0/973970703 >> v2:172.21.15.172:3300/0 conn(0x7f7ba006ab40 msgr2=0x7f7ba006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:22.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9f7fe700 1 --2- 172.21.15.158:0/973970703 >> v2:172.21.15.172:3300/0 conn(0x7f7ba006ab40 0x7f7ba006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9f7fe700 1 -- 172.21.15.158:0/973970703 >> v1:172.21.15.172:6789/0 conn(0x7f7ba006af20 legacy=0x7f7ba00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9f7fe700 1 -- 172.21.15.158:0/973970703 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba01617f0 con 0x7f7ba006e5d0 2023-10-16T11:22:22.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9f7fe700 1 --2- 172.21.15.158:0/973970703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0071910 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0074d70 tx=0x7f7b90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf9a71a80ae32b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9effd700 1 -- 172.21.15.158:0/973970703 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000b870 con 0x7f7ba006e5d0 2023-10-16T11:22:22.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9effd700 1 -- 172.21.15.158:0/973970703 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9000b9d0 con 0x7f7ba006e5d0 2023-10-16T11:22:22.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.857+0000 7f7b9effd700 1 -- 172.21.15.158:0/973970703 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000bd60 con 0x7f7ba006e5d0 2023-10-16T11:22:22.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 -- 172.21.15.158:0/973970703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 msgr2=0x7f7ba0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 --2- 172.21.15.158:0/973970703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0071910 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0074d70 tx=0x7f7b90009bd0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 -- 172.21.15.158:0/973970703 shutdown_connections 2023-10-16T11:22:22.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 --2- 172.21.15.158:0/973970703 >> v2:172.21.15.172:3300/0 conn(0x7f7ba006ab40 0x7f7ba006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 --2- 172.21.15.158:0/973970703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0071910 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 -- 172.21.15.158:0/973970703 >> 172.21.15.158:0/973970703 conn(0x7f7ba00d02b0 msgr2=0x7f7ba00d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:22.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 -- 172.21.15.158:0/973970703 shutdown_connections 2023-10-16T11:22:22.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.858+0000 7f7ba6720700 1 -- 172.21.15.158:0/973970703 wait complete. 2023-10-16T11:22:22.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.859+0000 7f7ba6720700 1 Processor -- start 2023-10-16T11:22:22.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.859+0000 7f7ba6720700 1 -- start start 2023-10-16T11:22:22.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba6720700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba006ab40 0x7f7ba00b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba6720700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ba006af20 0x7f7ba01f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba6720700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba6720700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba00b33e0 con 0x7f7ba006e5d0 2023-10-16T11:22:22.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba6720700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba01f3da0 con 0x7f7ba006ab40 2023-10-16T11:22:22.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba6720700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba00b5910 con 0x7f7ba006af20 2023-10-16T11:22:22.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0160350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44910/0 (socket says 172.21.15.158:44910) 2023-10-16T11:22:22.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 -- 172.21.15.158:0/3252918428 learned_addr learned my addr 172.21.15.158:0/3252918428 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:22.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7b9ffff700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba006ab40 0x7f7ba00b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7b9f7fe700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ba006af20 0x7f7ba01f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 -- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ba006af20 msgr2=0x7f7ba01f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ba006af20 0x7f7ba01f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 -- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba006ab40 msgr2=0x7f7ba00b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:22.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba006ab40 0x7f7ba00b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:22.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 -- 172.21.15.158:0/3252918428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b90009550 con 0x7f7ba006e5d0 2023-10-16T11:22:22.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.860+0000 7f7ba4cbd700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0160350 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800cc50 tx=0x7f7b9800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.861+0000 7f7b9cff9700 1 -- 172.21.15.158:0/3252918428 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98003c00 con 0x7f7ba006e5d0 2023-10-16T11:22:22.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.861+0000 7f7b9cff9700 1 -- 172.21.15.158:0/3252918428 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b98003d60 con 0x7f7ba006e5d0 2023-10-16T11:22:22.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.861+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba00b5bf0 con 0x7f7ba006e5d0 2023-10-16T11:22:22.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.861+0000 7f7b9cff9700 1 -- 172.21.15.158:0/3252918428 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9800aab0 con 0x7f7ba006e5d0 2023-10-16T11:22:22.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.861+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba00c7660 con 0x7f7ba006e5d0 2023-10-16T11:22:22.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.862+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba00681b0 con 0x7f7ba006e5d0 2023-10-16T11:22:22.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.862+0000 7f7b9cff9700 1 -- 172.21.15.158:0/3252918428 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b9800ac10 con 0x7f7ba006e5d0 2023-10-16T11:22:22.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.862+0000 7f7b9cff9700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f7b8c06b020 0x7f7b8c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:22.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.862+0000 7f7b9cff9700 1 -- 172.21.15.158:0/3252918428 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7b98089970 con 0x7f7ba006e5d0 2023-10-16T11:22:22.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.862+0000 7f7b9ffff700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f7b8c06b020 0x7f7b8c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:22.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.863+0000 7f7b9ffff700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f7b8c06b020 0x7f7b8c06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7b94006b20 tx=0x7f7b94008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:22.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:22.867+0000 7f7b9cff9700 1 -- 172.21.15.158:0/3252918428 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b980546d0 con 0x7f7ba006e5d0 2023-10-16T11:22:23.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.136+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]} v 0) v1 -- 0x7f7ba00b6230 con 0x7f7ba006e5d0 2023-10-16T11:22:23.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.661+0000 7f7b9cff9700 1 -- 172.21.15.158:0/3252918428 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]}]=0 v3282) v1 ==== 198+0+0 (secure 0 0 0) 0x7f7b98057cf0 con 0x7f7ba006e5d0 2023-10-16T11:22:23.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f7b8c06b020 msgr2=0x7f7b8c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:23.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f7b8c06b020 0x7f7b8c06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7b94006b20 tx=0x7f7b94008040 comp rx=0 tx=0).stop 2023-10-16T11:22:23.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 msgr2=0x7f7ba0160350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:23.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0160350 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800cc50 tx=0x7f7b9800cf60 comp rx=0 tx=0).stop 2023-10-16T11:22:23.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 shutdown_connections 2023-10-16T11:22:23.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f7b8c06b020 0x7f7b8c06d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:23.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ba006e5d0 0x7f7ba0160350 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:23.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba006ab40 0x7f7ba00b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:23.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 --2- 172.21.15.158:0/3252918428 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ba006af20 0x7f7ba01f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:23.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 >> 172.21.15.158:0/3252918428 conn(0x7f7ba00d02b0 msgr2=0x7f7ba00ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:23.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 shutdown_connections 2023-10-16T11:22:23.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:23.665+0000 7f7ba6720700 1 -- 172.21.15.158:0/3252918428 wait complete. 2023-10-16T11:22:23.680 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. ... ERROR 2023-10-16T11:22:23.681 INFO:tasks.cephfs_test_runner: 2023-10-16T11:22:23.681 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-16T11:22:23.682 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T11:22:23.682 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. 2023-10-16T11:22:23.682 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T11:22:23.683 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-16T11:22:23.683 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/cephfs/test_snap_schedules.py", line 585, in test_snap_schedule_subvol_and_group_arguments 2023-10-16T11:22:23.683 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'create', 'a', 'sv1') 2023-10-16T11:22:23.683 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/cephfs/test_snap_schedules.py", line 39, in _fs_cmd 2023-10-16T11:22:23.684 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2023-10-16T11:22:23.684 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2023-10-16T11:22:23.684 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2023-10-16T11:22:23.684 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2023-10-16T11:22:23.685 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2023-10-16T11:22:23.685 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd 2023-10-16T11:22:23.685 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2023-10-16T11:22:23.685 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run 2023-10-16T11:22:23.686 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-10-16T11:22:23.686 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run 2023-10-16T11:22:23.686 INFO:tasks.cephfs_test_runner: r.wait() 2023-10-16T11:22:23.686 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait 2023-10-16T11:22:23.687 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-10-16T11:22:23.687 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-10-16T11:22:23.687 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-10-16T11:22:23.687 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi158 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create a sv1' 2023-10-16T11:22:23.688 INFO:tasks.cephfs_test_runner: 2023-10-16T11:22:23.688 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T11:22:23.688 INFO:tasks.cephfs_test_runner:Ran 15 tests in 5891.715s 2023-10-16T11:22:23.689 INFO:tasks.cephfs_test_runner: 2023-10-16T11:22:23.689 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-10-16T11:22:23.689 INFO:tasks.cephfs_test_runner: 2023-10-16T11:22:23.689 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-16T11:22:23.690 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T11:22:23.690 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. 2023-10-16T11:22:23.690 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T11:22:23.690 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-16T11:22:23.691 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/cephfs/test_snap_schedules.py", line 585, in test_snap_schedule_subvol_and_group_arguments 2023-10-16T11:22:23.691 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'create', 'a', 'sv1') 2023-10-16T11:22:23.691 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/cephfs/test_snap_schedules.py", line 39, in _fs_cmd 2023-10-16T11:22:23.692 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2023-10-16T11:22:23.692 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2023-10-16T11:22:23.692 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2023-10-16T11:22:23.692 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2023-10-16T11:22:23.693 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2023-10-16T11:22:23.693 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd 2023-10-16T11:22:23.693 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2023-10-16T11:22:23.694 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run 2023-10-16T11:22:23.694 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-10-16T11:22:23.694 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run 2023-10-16T11:22:23.694 INFO:tasks.cephfs_test_runner: r.wait() 2023-10-16T11:22:23.695 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait 2023-10-16T11:22:23.695 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-10-16T11:22:23.695 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-10-16T11:22:23.695 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-10-16T11:22:23.696 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi158 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create a sv1' 2023-10-16T11:22:23.696 INFO:tasks.cephfs_test_runner: 2023-10-16T11:22:23.696 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T11:22:23.761 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7b415bb3060e419faa0866cd73f89f17 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_7b3d12460841f3cbe9400ba1b68e3e20957ff531/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T11:22:23.763 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-16T11:22:23.772 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-16T11:22:23.804 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-16T11:22:23.805 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:22:23.805 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:23.827 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:22:23.827 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-16T11:22:23.828 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:23.885 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17122 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T11:22:23.886 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:22:23.886 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:23.907 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:22:23.908 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-16T11:22:23.908 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T11:22:23.966 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284272k,nr_inodes=4071068,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14167 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T11:22:23.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:23.967 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:22:23.984 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T11:22:23.985 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T11:22:23.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:23.985 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T11:22:24.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.063 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T11:22:24.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.164 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link delete ceph-brx 2023-10-16T11:22:24.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.263 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2023-10-16T11:22:24.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.280 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link delete ceph-brx 2023-10-16T11:22:24.350 INFO:teuthology.orchestra.run.smithi158.stderr:Cannot find device "ceph-brx" 2023-10-16T11:22:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:22:24.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.352 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2023-10-16T11:22:24.373 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-16T11:22:24.373 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T11:22:24.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T11:22:24.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T11:22:24.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link delete ceph-brx 2023-10-16T11:22:24.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.700 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2023-10-16T11:22:24.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T11:22:24.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link delete ceph-brx 2023-10-16T11:22:24.789 INFO:teuthology.orchestra.run.smithi172.stderr:Cannot find device "ceph-brx" 2023-10-16T11:22:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:22:24.791 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-16T11:22:24.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T11:22:24.817 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:22:24.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.939+0000 7f2a17045700 1 Processor -- start 2023-10-16T11:22:24.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.939+0000 7f2a17045700 1 -- start start 2023-10-16T11:22:24.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a17045700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a101599b0 0x7f2a100ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a17045700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a10072430 0x7f2a100ca600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a17045700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a10072250 con 0x7f2a1015d7b0 2023-10-16T11:22:24.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a17045700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1015dc90 con 0x7f2a101599b0 2023-10-16T11:22:24.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a17045700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1015df70 con 0x7f2a10072430 2023-10-16T11:22:24.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a155e2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a10072430 0x7f2a100ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:24.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a14de1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a101599b0 0x7f2a100ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:24.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a155e2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a10072430 0x7f2a100ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.158:55748/0 (socket says 172.21.15.158:55748) 2023-10-16T11:22:24.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a0ffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2a1015d7b0 0x7f2a1006c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:48994/0 (socket says 172.21.15.158:48994) 2023-10-16T11:22:24.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.940+0000 7f2a155e2700 1 -- 172.21.15.158:0/831964795 learned_addr learned my addr 172.21.15.158:0/831964795 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:24.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a0f7fe700 1 -- 172.21.15.158:0/831964795 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574692772 0 0) 0x7f2a10072250 con 0x7f2a1015d7b0 2023-10-16T11:22:24.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a0f7fe700 1 -- 172.21.15.158:0/831964795 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a000035b0 con 0x7f2a1015d7b0 2023-10-16T11:22:24.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a14de1700 1 -- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a10072430 msgr2=0x7f2a100ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:24.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a14de1700 1 --2- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a10072430 0x7f2a100ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:24.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a14de1700 1 -- 172.21.15.158:0/831964795 >> v1:172.21.15.172:6789/0 conn(0x7f2a1015d7b0 legacy=0x7f2a1006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:24.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a14de1700 1 -- 172.21.15.158:0/831964795 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1015e250 con 0x7f2a101599b0 2023-10-16T11:22:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a155e2700 1 --2- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a10072430 0x7f2a100ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:24.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a14de1700 1 --2- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a101599b0 0x7f2a100ca0c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400bae0 tx=0x7f2a0400bdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=657e543bfba0bdb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a0f7fe700 1 -- 172.21.15.158:0/831964795 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0400dc70 con 0x7f2a101599b0 2023-10-16T11:22:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a0f7fe700 1 -- 172.21.15.158:0/831964795 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0400ddd0 con 0x7f2a101599b0 2023-10-16T11:22:24.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.941+0000 7f2a0f7fe700 1 -- 172.21.15.158:0/831964795 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04010690 con 0x7f2a101599b0 2023-10-16T11:22:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 -- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a101599b0 msgr2=0x7f2a100ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 --2- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a101599b0 0x7f2a100ca0c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400bae0 tx=0x7f2a0400bdf0 comp rx=0 tx=0).stop 2023-10-16T11:22:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 -- 172.21.15.158:0/831964795 shutdown_connections 2023-10-16T11:22:24.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 --2- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a10072430 0x7f2a100ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 --2- 172.21.15.158:0/831964795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a101599b0 0x7f2a100ca0c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 -- 172.21.15.158:0/831964795 >> 172.21.15.158:0/831964795 conn(0x7f2a1006af00 msgr2=0x7f2a100cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 -- 172.21.15.158:0/831964795 shutdown_connections 2023-10-16T11:22:24.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.942+0000 7f2a17045700 1 -- 172.21.15.158:0/831964795 wait complete. 2023-10-16T11:22:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.943+0000 7f2a17045700 1 Processor -- start 2023-10-16T11:22:24.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.943+0000 7f2a17045700 1 -- start start 2023-10-16T11:22:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a17045700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a10072430 0x7f2a101507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a17045700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a101599b0 0x7f2a100b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:24.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a17045700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1015d7b0 0x7f2a1015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a17045700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a101f4100 con 0x7f2a1015d7b0 2023-10-16T11:22:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a17045700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100b3ce0 con 0x7f2a10072430 2023-10-16T11:22:24.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a17045700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100b3fc0 con 0x7f2a101599b0 2023-10-16T11:22:24.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1015d7b0 0x7f2a1015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:24.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1015d7b0 0x7f2a1015c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44924/0 (socket says 172.21.15.158:44924) 2023-10-16T11:22:24.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 -- 172.21.15.158:0/3059834673 learned_addr learned my addr 172.21.15.158:0/3059834673 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:24.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a14de1700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a10072430 0x7f2a101507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a0ffff700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a101599b0 0x7f2a100b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 -- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a101599b0 msgr2=0x7f2a100b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a101599b0 0x7f2a100b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:24.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 -- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a10072430 msgr2=0x7f2a101507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a10072430 0x7f2a101507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 -- 172.21.15.158:0/3059834673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08009b30 con 0x7f2a1015d7b0 2023-10-16T11:22:24.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a14de1700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a10072430 0x7f2a101507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a155e2700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1015d7b0 0x7f2a1015c330 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800efc0 tx=0x7f2a0800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.944+0000 7f2a0d7fa700 1 -- 172.21.15.158:0/3059834673 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08018070 con 0x7f2a1015d7b0 2023-10-16T11:22:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.945+0000 7f2a0d7fa700 1 -- 172.21.15.158:0/3059834673 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0800f040 con 0x7f2a1015d7b0 2023-10-16T11:22:24.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.945+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0400b740 con 0x7f2a1015d7b0 2023-10-16T11:22:24.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.945+0000 7f2a0d7fa700 1 -- 172.21.15.158:0/3059834673 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08003c20 con 0x7f2a1015d7b0 2023-10-16T11:22:24.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.945+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a101f3d60 con 0x7f2a1015d7b0 2023-10-16T11:22:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.945+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a100b44e0 con 0x7f2a1015d7b0 2023-10-16T11:22:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.946+0000 7f2a0d7fa700 1 -- 172.21.15.158:0/3059834673 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2a0800c880 con 0x7f2a1015d7b0 2023-10-16T11:22:24.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.946+0000 7f2a0d7fa700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29f406b020 0x7f29f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:24.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.946+0000 7f2a0d7fa700 1 -- 172.21.15.158:0/3059834673 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2a08089ca0 con 0x7f2a1015d7b0 2023-10-16T11:22:24.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.946+0000 7f2a14de1700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29f406b020 0x7f29f406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:24.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.947+0000 7f2a14de1700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29f406b020 0x7f29f406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400be30 tx=0x7f2a0401c040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:24.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:24.951+0000 7f2a0d7fa700 1 -- 172.21.15.158:0/3059834673 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a100b44e0 con 0x7f2a1015d7b0 2023-10-16T11:22:25.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.138+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a101f3ef0 con 0x7f29f406b020 2023-10-16T11:22:25.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.140+0000 7f2a0d7fa700 1 -- 172.21.15.158:0/3059834673 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50640 (secure 0 0 0) 0x7f2a101f3ef0 con 0x7f29f406b020 2023-10-16T11:22:25.143 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:25.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.142+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29f406b020 msgr2=0x7f29f406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.142+0000 7f2a17045700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29f406b020 0x7f29f406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400be30 tx=0x7f2a0401c040 comp rx=0 tx=0).stop 2023-10-16T11:22:25.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.142+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1015d7b0 msgr2=0x7f2a1015c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.142+0000 7f2a17045700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1015d7b0 0x7f2a1015c330 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800efc0 tx=0x7f2a0800c3a0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 shutdown_connections 2023-10-16T11:22:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f29f406b020 0x7f29f406d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a1015d7b0 0x7f2a1015c330 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a10072430 0x7f2a101507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 --2- 172.21.15.158:0/3059834673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a101599b0 0x7f2a100b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 >> 172.21.15.158:0/3059834673 conn(0x7f2a1006af00 msgr2=0x7f2a101574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:25.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 shutdown_connections 2023-10-16T11:22:25.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.143+0000 7f2a17045700 1 -- 172.21.15.158:0/3059834673 wait complete. 2023-10-16T11:22:25.149 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:22:25.156 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-16T11:22:23.507349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":444500,"kb_used_data":28448,"kb_used_omap":12,"kb_used_meta":415923,"kb_avail":754530220,"statfs":{"total":773094113280,"available":772638945280,"internally_reserved":0,"allocated":29130752,"data_stored":16271401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":425905719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-530476,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":10,"num_read_kb":133,"num_write":-143,"num_write_kb":-1330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997367"},"pg_stats":[{"pgid":"1.0","version":"273'414","reported_seq":1498,"reported_epoch":274,"state":"active+clean","last_fresh":"2023-10-16T11:22:20.661621+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:22:20.661621+0000","last_peered":"2023-10-16T11:22:20.661621+0000","last_clean":"2023-10-16T11:22:20.661621+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:22:20.661621+0000","last_undegraded":"2023-10-16T11:22:20.661621+0000","last_fullsized":"2023-10-16T11:22:20.661621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576019,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57384,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314456,"statfs":{"total":96636764160,"available":96578002944,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73799999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.437},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.68300000000000005,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.98199999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.70399999999999996,"15min":0.66700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.875},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":9.516,"15min":9.516},"last":1.1890000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67000000000000004,"15min":0.64000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":1.087}]},{"osd":4,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.59299999999999997,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.624,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67400000000000004,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.64800000000000002,"15min":0.57899999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77800000000000002,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.68000000000000005,"15min":0.625},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":1.2809999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.70999999999999996,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.69299999999999995,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72199999999999998,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.70399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576019,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.57999999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.86299999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56499999999999995,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84299999999999997,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71899999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":2.387,"15min":2.387},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.314},"max":{"1min":0.78100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59899999999999998,"15min":0.61599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88800000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81000000000000005,"5min":1.02,"15min":1.02},"last":0.75900000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.73199999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":3.335,"15min":3.335},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":2.524,"15min":2.524},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.79700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.40799999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.315},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.26500000000000001,"5min":0.184,"15min":0.184},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36199999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.44},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.434}]},{"osd":3,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.498},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0309999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.0389999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.625}]},{"osd":6,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.629,"15min":0.66500000000000004},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.67200000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64500000000000002,"15min":0.68999999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.56999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834576020,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49072,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322768,"statfs":{"total":96636764160,"available":96586514432,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.55500000000000005,"15min":0.60299999999999998},"last":0.376},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40300000000000002,"15min":0.439},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.44500000000000001,"5min":0.58399999999999996,"15min":0.77200000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50700000000000001,"5min":1.272,"15min":1.272},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.50600000000000001,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":1.351,"15min":1.351},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.64200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.45100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":1.391,"15min":1.391},"last":0.80600000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60599999999999998,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.874,"15min":1.089},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.216},"last":0.76500000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64300000000000002,"15min":0.68600000000000005},"min":{"1min":0.46600000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":1.038,"15min":1.1180000000000001},"last":0.79100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88900000000000001,"5min":0.97399999999999998,"15min":1.008},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.66900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.88500000000000001,"15min":1.0780000000000001},"last":0.57099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576019,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.245,"5min":0.188,"15min":0.188},"max":{"1min":0.47699999999999998,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.437},"min":{"1min":0.255,"5min":0.193,"15min":0.193},"max":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.436,"15min":0.46400000000000002},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.41099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45300000000000001,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.32200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.432}]},{"osd":4,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":1.097},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86499999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83099999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92700000000000005,"5min":1.0169999999999999,"15min":1.028},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.498,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85899999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.75}]},{"osd":7,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.70399999999999996},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":0.99099999999999999,"15min":1.071},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.97799999999999998,"15min":1.077},"last":0.65700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.437,"5min":1.032,"15min":1.032},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35599999999999998,"15min":0.39800000000000002},"min":{"1min":0.27800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.51800000000000002,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.29499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":1.341,"15min":1.341},"last":0.318},{"interface":"front","average":{"1min":0.375,"5min":0.39600000000000002,"15min":0.433},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47699999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.47099999999999997},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58399999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55700000000000005,"5min":0.96599999999999997,"15min":1.024},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.40600000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86499999999999999,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89100000000000001,"5min":2.629,"15min":2.629},"last":0.438}]},{"osd":5,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.66300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70299999999999996,"15min":0.69899999999999995},"min":{"1min":0.45700000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.68100000000000005,"15min":0.69099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85899999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.69799999999999995},"min":{"1min":0.44600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96899999999999997,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70599999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90900000000000003,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.90100000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576019,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.60099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":1.0589999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55800000000000005,"15min":0.60799999999999998},"min":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.88200000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54900000000000004,"15min":0.60099999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.90700000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55000000000000004,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55600000000000005,"15min":0.60899999999999999},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67600000000000005,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58699999999999997,"15min":0.626},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80500000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55000000000000004,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.751,"5min":0.79700000000000004,"15min":0.995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.499,"15min":0.5},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.86199999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71599999999999997,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.751,"5min":0.82899999999999996,"15min":1.0609999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81499999999999995,"5min":0.83899999999999997,"15min":1.115},"last":0.75600000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.42199999999999999,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.76800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66492,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305348,"statfs":{"total":96636764160,"available":96568676352,"internally_reserved":0,"allocated":3457024,"data_stored":1867014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56000000000000005,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.871,"15min":0.88200000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.58399999999999996},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73199999999999998,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.57699999999999996},"min":{"1min":0.311,"5min":0.309,"15min":0.309},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.38},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.747,"5min":0.82099999999999995,"15min":0.89400000000000002},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.59799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76400000000000001,"5min":0.80900000000000005,"15min":0.84799999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.33500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.86399999999999999},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.753,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.625},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.85899999999999999,"15min":0.878},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84699999999999998,"5min":0.871,"15min":0.871},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.89300000000000002},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:22:25.159 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:22:25.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.277+0000 7f00b9410700 1 Processor -- start 2023-10-16T11:22:25.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.278+0000 7f00b9410700 1 -- start start 2023-10-16T11:22:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.278+0000 7f00b9410700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f00b406a8d0 0x7f00b406e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.278+0000 7f00b9410700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41612a0 0x7f00b406acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.278+0000 7f00b9410700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b415dcb0 con 0x7f00b40718d0 2023-10-16T11:22:25.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.278+0000 7f00b9410700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b406b1f0 con 0x7f00b406a8d0 2023-10-16T11:22:25.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.278+0000 7f00b9410700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4161740 con 0x7f00b41612a0 2023-10-16T11:22:25.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b27fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41612a0 0x7f00b406acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f00b406a8d0 0x7f00b406e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b37fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f00b40718d0 0x7f00b415e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:49010/0 (socket says 172.21.15.158:49010) 2023-10-16T11:22:25.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b37fe700 1 -- 172.21.15.158:0/1128739341 learned_addr learned my addr 172.21.15.158:0/1128739341 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:25.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b1ffb700 1 -- 172.21.15.158:0/1128739341 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743118853 0 0) 0x7f00b415dcb0 con 0x7f00b40718d0 2023-10-16T11:22:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b1ffb700 1 -- 172.21.15.158:0/1128739341 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c0035b0 con 0x7f00b40718d0 2023-10-16T11:22:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b27fc700 1 -- 172.21.15.158:0/1128739341 >> v2:172.21.15.172:3300/0 conn(0x7f00b406a8d0 msgr2=0x7f00b406e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:25.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b27fc700 1 --2- 172.21.15.158:0/1128739341 >> v2:172.21.15.172:3300/0 conn(0x7f00b406a8d0 0x7f00b406e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b27fc700 1 -- 172.21.15.158:0/1128739341 >> v1:172.21.15.172:6789/0 conn(0x7f00b40718d0 legacy=0x7f00b415e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b27fc700 1 -- 172.21.15.158:0/1128739341 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4161a20 con 0x7f00b41612a0 2023-10-16T11:22:25.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b2ffd700 1 --2- 172.21.15.158:0/1128739341 >> v2:172.21.15.172:3300/0 conn(0x7f00b406a8d0 0x7f00b406e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:25.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b27fc700 1 --2- 172.21.15.158:0/1128739341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41612a0 0x7f00b406acb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009840 tx=0x7f00a8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a44b997ead68cfc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b1ffb700 1 -- 172.21.15.158:0/1128739341 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8011830 con 0x7f00b41612a0 2023-10-16T11:22:25.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b1ffb700 1 -- 172.21.15.158:0/1128739341 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a8011990 con 0x7f00b41612a0 2023-10-16T11:22:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.279+0000 7f00b1ffb700 1 -- 172.21.15.158:0/1128739341 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8011d20 con 0x7f00b41612a0 2023-10-16T11:22:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 -- 172.21.15.158:0/1128739341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41612a0 msgr2=0x7f00b406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 --2- 172.21.15.158:0/1128739341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41612a0 0x7f00b406acb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009840 tx=0x7f00a8009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 -- 172.21.15.158:0/1128739341 shutdown_connections 2023-10-16T11:22:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 --2- 172.21.15.158:0/1128739341 >> v2:172.21.15.172:3300/0 conn(0x7f00b406a8d0 0x7f00b406e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 --2- 172.21.15.158:0/1128739341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41612a0 0x7f00b406acb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 -- 172.21.15.158:0/1128739341 >> 172.21.15.158:0/1128739341 conn(0x7f00b40d02a0 msgr2=0x7f00b40d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 -- 172.21.15.158:0/1128739341 shutdown_connections 2023-10-16T11:22:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.280+0000 7f00b9410700 1 -- 172.21.15.158:0/1128739341 wait complete. 2023-10-16T11:22:25.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.281+0000 7f00b9410700 1 Processor -- start 2023-10-16T11:22:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.281+0000 7f00b9410700 1 -- start start 2023-10-16T11:22:25.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b9410700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b406a8d0 0x7f00b4151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b9410700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40718d0 0x7f00b40c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b9410700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41612a0 0x7f00b40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.299 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b9410700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4151840 con 0x7f00b40718d0 2023-10-16T11:22:25.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b9410700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b41f2bb0 con 0x7f00b41612a0 2023-10-16T11:22:25.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b9410700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40c48f0 con 0x7f00b406a8d0 2023-10-16T11:22:25.300 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40718d0 0x7f00b40c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b27fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40718d0 0x7f00b40c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44936/0 (socket says 172.21.15.158:44936) 2023-10-16T11:22:25.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b27fc700 1 -- 172.21.15.158:0/2438102742 learned_addr learned my addr 172.21.15.158:0/2438102742 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:25.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b2ffd700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b406a8d0 0x7f00b4151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.301 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.282+0000 7f00b37fe700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41612a0 0x7f00b40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b27fc700 1 -- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b406a8d0 msgr2=0x7f00b4151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.302 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b27fc700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b406a8d0 0x7f00b4151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b27fc700 1 -- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41612a0 msgr2=0x7f00b40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b27fc700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41612a0 0x7f00b40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.303 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b27fc700 1 -- 172.21.15.158:0/2438102742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40094a0 con 0x7f00b40718d0 2023-10-16T11:22:25.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b37fe700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41612a0 0x7f00b40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:25.304 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b27fc700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40718d0 0x7f00b40c5ee0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009fd0 tx=0x7f00a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f009b7fe700 1 -- 172.21.15.158:0/2438102742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a80036a0 con 0x7f00b40718d0 2023-10-16T11:22:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f009b7fe700 1 -- 172.21.15.158:0/2438102742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a801e070 con 0x7f00b40718d0 2023-10-16T11:22:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a80094a0 con 0x7f00b40718d0 2023-10-16T11:22:25.305 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f009b7fe700 1 -- 172.21.15.158:0/2438102742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8003a60 con 0x7f00b40718d0 2023-10-16T11:22:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.283+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b40c4eb0 con 0x7f00b40718d0 2023-10-16T11:22:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.284+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b4072e30 con 0x7f00b40718d0 2023-10-16T11:22:25.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.284+0000 7f009b7fe700 1 -- 172.21.15.158:0/2438102742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f00a8003bc0 con 0x7f00b40718d0 2023-10-16T11:22:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.285+0000 7f009b7fe700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f00a006b020 0x7f00a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.285+0000 7f009b7fe700 1 -- 172.21.15.158:0/2438102742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f00a808a8d0 con 0x7f00b40718d0 2023-10-16T11:22:25.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.285+0000 7f00b2ffd700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f00a006b020 0x7f00a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.285+0000 7f00b2ffd700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f00a006b020 0x7f00a006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009d70 tx=0x7f00a4009210 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.289+0000 7f009b7fe700 1 -- 172.21.15.158:0/2438102742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00a80555b0 con 0x7f00b40718d0 2023-10-16T11:22:25.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.477+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00b4161fe0 con 0x7f00a006b020 2023-10-16T11:22:25.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.479+0000 7f009b7fe700 1 -- 172.21.15.158:0/2438102742 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50640 (secure 0 0 0) 0x7f00a40099c0 con 0x7f00a006b020 2023-10-16T11:22:25.482 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:25.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f00a006b020 msgr2=0x7f00a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f00a006b020 0x7f00a006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009d70 tx=0x7f00a4009210 comp rx=0 tx=0).stop 2023-10-16T11:22:25.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40718d0 msgr2=0x7f00b40c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40718d0 0x7f00b40c5ee0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009fd0 tx=0x7f00a8004750 comp rx=0 tx=0).stop 2023-10-16T11:22:25.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 shutdown_connections 2023-10-16T11:22:25.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f00a006b020 0x7f00a006d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41612a0 0x7f00b40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b406a8d0 0x7f00b4151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 --2- 172.21.15.158:0/2438102742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40718d0 0x7f00b40c5ee0 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 >> 172.21.15.158:0/2438102742 conn(0x7f00b40d02a0 msgr2=0x7f00b40d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:25.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 shutdown_connections 2023-10-16T11:22:25.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.481+0000 7f00b9410700 1 -- 172.21.15.158:0/2438102742 wait complete. 2023-10-16T11:22:25.487 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:22:25.495 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-16T11:22:23.507349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":444500,"kb_used_data":28448,"kb_used_omap":12,"kb_used_meta":415923,"kb_avail":754530220,"statfs":{"total":773094113280,"available":772638945280,"internally_reserved":0,"allocated":29130752,"data_stored":16271401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":425905719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-530476,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":10,"num_read_kb":133,"num_write":-143,"num_write_kb":-1330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997367"},"pg_stats":[{"pgid":"1.0","version":"273'414","reported_seq":1498,"reported_epoch":274,"state":"active+clean","last_fresh":"2023-10-16T11:22:20.661621+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:22:20.661621+0000","last_peered":"2023-10-16T11:22:20.661621+0000","last_clean":"2023-10-16T11:22:20.661621+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:22:20.661621+0000","last_undegraded":"2023-10-16T11:22:20.661621+0000","last_fullsized":"2023-10-16T11:22:20.661621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576019,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57384,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314456,"statfs":{"total":96636764160,"available":96578002944,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73799999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.437},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.68300000000000005,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.98199999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.70399999999999996,"15min":0.66700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.875},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":9.516,"15min":9.516},"last":1.1890000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67000000000000004,"15min":0.64000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":1.087}]},{"osd":4,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.59299999999999997,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.624,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67400000000000004,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.64800000000000002,"15min":0.57899999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77800000000000002,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.68000000000000005,"15min":0.625},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":1.2809999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.70999999999999996,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.69299999999999995,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72199999999999998,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.70399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576019,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.57999999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.86299999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56499999999999995,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84299999999999997,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71899999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":2.387,"15min":2.387},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.314},"max":{"1min":0.78100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59899999999999998,"15min":0.61599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88800000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81000000000000005,"5min":1.02,"15min":1.02},"last":0.75900000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.73199999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":3.335,"15min":3.335},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":2.524,"15min":2.524},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.79700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.40799999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.315},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.26500000000000001,"5min":0.184,"15min":0.184},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36199999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.44},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.434}]},{"osd":3,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.498},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0309999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.0389999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.625}]},{"osd":6,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.629,"15min":0.66500000000000004},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.67200000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64500000000000002,"15min":0.68999999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.56999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834576020,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49072,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322768,"statfs":{"total":96636764160,"available":96586514432,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.55500000000000005,"15min":0.60299999999999998},"last":0.376},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40300000000000002,"15min":0.439},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.44500000000000001,"5min":0.58399999999999996,"15min":0.77200000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50700000000000001,"5min":1.272,"15min":1.272},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.50600000000000001,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":1.351,"15min":1.351},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.64200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.45100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":1.391,"15min":1.391},"last":0.80600000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60599999999999998,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.874,"15min":1.089},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.216},"last":0.76500000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64300000000000002,"15min":0.68600000000000005},"min":{"1min":0.46600000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":1.038,"15min":1.1180000000000001},"last":0.79100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88900000000000001,"5min":0.97399999999999998,"15min":1.008},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.66900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.88500000000000001,"15min":1.0780000000000001},"last":0.57099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576019,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.245,"5min":0.188,"15min":0.188},"max":{"1min":0.47699999999999998,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.437},"min":{"1min":0.255,"5min":0.193,"15min":0.193},"max":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.436,"15min":0.46400000000000002},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.41099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45300000000000001,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.32200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.432}]},{"osd":4,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":1.097},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86499999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83099999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92700000000000005,"5min":1.0169999999999999,"15min":1.028},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.498,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85899999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.75}]},{"osd":7,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.70399999999999996},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":0.99099999999999999,"15min":1.071},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.97799999999999998,"15min":1.077},"last":0.65700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.437,"5min":1.032,"15min":1.032},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35599999999999998,"15min":0.39800000000000002},"min":{"1min":0.27800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.51800000000000002,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.29499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":1.341,"15min":1.341},"last":0.318},{"interface":"front","average":{"1min":0.375,"5min":0.39600000000000002,"15min":0.433},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47699999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.47099999999999997},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58399999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55700000000000005,"5min":0.96599999999999997,"15min":1.024},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.40600000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86499999999999999,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89100000000000001,"5min":2.629,"15min":2.629},"last":0.438}]},{"osd":5,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.66300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70299999999999996,"15min":0.69899999999999995},"min":{"1min":0.45700000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.68100000000000005,"15min":0.69099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85899999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.69799999999999995},"min":{"1min":0.44600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96899999999999997,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70599999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90900000000000003,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.90100000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576019,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.60099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":1.0589999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55800000000000005,"15min":0.60799999999999998},"min":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.88200000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54900000000000004,"15min":0.60099999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.90700000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55000000000000004,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55600000000000005,"15min":0.60899999999999999},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67600000000000005,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58699999999999997,"15min":0.626},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80500000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55000000000000004,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.751,"5min":0.79700000000000004,"15min":0.995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.499,"15min":0.5},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.86199999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71599999999999997,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.751,"5min":0.82899999999999996,"15min":1.0609999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81499999999999995,"5min":0.83899999999999997,"15min":1.115},"last":0.75600000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.42199999999999999,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.76800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66492,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305348,"statfs":{"total":96636764160,"available":96568676352,"internally_reserved":0,"allocated":3457024,"data_stored":1867014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56000000000000005,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.871,"15min":0.88200000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.58399999999999996},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73199999999999998,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.57699999999999996},"min":{"1min":0.311,"5min":0.309,"15min":0.309},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.38},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.747,"5min":0.82099999999999995,"15min":0.89400000000000002},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.59799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76400000000000001,"5min":0.80900000000000005,"15min":0.84799999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.33500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.86399999999999999},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.753,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.625},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.85899999999999999,"15min":0.878},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84699999999999998,"5min":0.871,"15min":0.871},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.89300000000000002},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:22:25.497 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T11:22:25.497 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:22:25.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.616+0000 7f7762fe9700 1 Processor -- start 2023-10-16T11:22:25.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.616+0000 7f7762fe9700 1 -- start start 2023-10-16T11:22:25.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f7762fe9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06ab40 0x7f775c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f7762fe9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f775c06e5d0 0x7f775c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f7762fe9700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c071e50 con 0x7f775c06af20 2023-10-16T11:22:25.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f7762fe9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775c161270 con 0x7f775c06e5d0 2023-10-16T11:22:25.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f7762fe9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c161510 con 0x7f775c06ab40 2023-10-16T11:22:25.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f775bfff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f775c06e5d0 0x7f775c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f7761586700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f775c06af20 0x7f775c0722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45122/0 (socket says 172.21.15.158:45122) 2023-10-16T11:22:25.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.617+0000 7f7761586700 1 -- 172.21.15.158:0/1502738828 learned_addr learned my addr 172.21.15.158:0/1502738828 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:25.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f7760d85700 1 --2- 172.21.15.158:0/1502738828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06ab40 0x7f775c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775bfff700 1 -- 172.21.15.158:0/1502738828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06ab40 msgr2=0x7f775c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:25.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775bfff700 1 --2- 172.21.15.158:0/1502738828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06ab40 0x7f775c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775bfff700 1 -- 172.21.15.158:0/1502738828 >> v1:172.21.15.158:6789/0 conn(0x7f775c06af20 legacy=0x7f775c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775b7fe700 1 -- 172.21.15.158:0/1502738828 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182469198 0 0) 0x7f775c071e50 con 0x7f775c06af20 2023-10-16T11:22:25.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775bfff700 1 -- 172.21.15.158:0/1502738828 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c1617f0 con 0x7f775c06e5d0 2023-10-16T11:22:25.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775bfff700 1 --2- 172.21.15.158:0/1502738828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c071910 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f7748009840 tx=0x7f7748009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e61f278279a1c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775b7fe700 1 -- 172.21.15.158:0/1502738828 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748011830 con 0x7f775c06e5d0 2023-10-16T11:22:25.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775b7fe700 1 -- 172.21.15.158:0/1502738828 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7748011990 con 0x7f775c06e5d0 2023-10-16T11:22:25.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.618+0000 7f775b7fe700 1 -- 172.21.15.158:0/1502738828 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748011d20 con 0x7f775c06e5d0 2023-10-16T11:22:25.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 -- 172.21.15.158:0/1502738828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 msgr2=0x7f775c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 --2- 172.21.15.158:0/1502738828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c071910 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f7748009840 tx=0x7f7748009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:25.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 -- 172.21.15.158:0/1502738828 shutdown_connections 2023-10-16T11:22:25.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 --2- 172.21.15.158:0/1502738828 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06ab40 0x7f775c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 --2- 172.21.15.158:0/1502738828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c071910 unknown :-1 s=CLOSED pgs=3412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 -- 172.21.15.158:0/1502738828 >> 172.21.15.158:0/1502738828 conn(0x7f775c0d02b0 msgr2=0x7f775c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:25.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 -- 172.21.15.158:0/1502738828 shutdown_connections 2023-10-16T11:22:25.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.619+0000 7f7762fe9700 1 -- 172.21.15.158:0/1502738828 wait complete. 2023-10-16T11:22:25.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.620+0000 7f7762fe9700 1 Processor -- start 2023-10-16T11:22:25.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.620+0000 7f7762fe9700 1 -- start start 2023-10-16T11:22:25.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7762fe9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f775c06ab40 0x7f775c0b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7762fe9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06af20 0x7f775c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7762fe9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7762fe9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c152ed0 con 0x7f775c06e5d0 2023-10-16T11:22:25.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7762fe9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c0b3480 con 0x7f775c06ab40 2023-10-16T11:22:25.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7762fe9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c0b3760 con 0x7f775c06af20 2023-10-16T11:22:25.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44952/0 (socket says 172.21.15.158:44952) 2023-10-16T11:22:25.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 -- 172.21.15.158:0/3709675166 learned_addr learned my addr 172.21.15.158:0/3709675166 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:25.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7760d85700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f775c06ab40 0x7f775c0b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f775bfff700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06af20 0x7f775c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 -- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06af20 msgr2=0x7f775c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06af20 0x7f775c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 -- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f775c06ab40 msgr2=0x7f775c0b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f775c06ab40 0x7f775c0b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.621+0000 7f7761586700 1 -- 172.21.15.158:0/3709675166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77480094a0 con 0x7f775c06e5d0 2023-10-16T11:22:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.622+0000 7f7761586700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c152990 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f7754002230 tx=0x7f775400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.622+0000 7f77597fa700 1 -- 172.21.15.158:0/3709675166 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775400aac0 con 0x7f775c06e5d0 2023-10-16T11:22:25.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.622+0000 7f77597fa700 1 -- 172.21.15.158:0/3709675166 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775400ac40 con 0x7f775c06e5d0 2023-10-16T11:22:25.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.622+0000 7f77597fa700 1 -- 172.21.15.158:0/3709675166 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77540070d0 con 0x7f775c06e5d0 2023-10-16T11:22:25.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.622+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c0b3a40 con 0x7f775c06e5d0 2023-10-16T11:22:25.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.622+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c0b4000 con 0x7f775c06e5d0 2023-10-16T11:22:25.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.623+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c0b41f0 con 0x7f775c06e5d0 2023-10-16T11:22:25.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.623+0000 7f77597fa700 1 -- 172.21.15.158:0/3709675166 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7754007250 con 0x7f775c06e5d0 2023-10-16T11:22:25.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.623+0000 7f77597fa700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f774006b020 0x7f774006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.624+0000 7f77597fa700 1 -- 172.21.15.158:0/3709675166 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7754089420 con 0x7f775c06e5d0 2023-10-16T11:22:25.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.624+0000 7f7760d85700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f774006b020 0x7f774006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.624+0000 7f7760d85700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f774006b020 0x7f774006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7750006b20 tx=0x7f7750008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.628+0000 7f77597fa700 1 -- 172.21.15.158:0/3709675166 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f77540540f0 con 0x7f775c06e5d0 2023-10-16T11:22:25.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.816+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f775c161db0 con 0x7f774006b020 2023-10-16T11:22:25.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.818+0000 7f77597fa700 1 -- 172.21.15.158:0/3709675166 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50641 (secure 0 0 0) 0x7f7750009e40 con 0x7f774006b020 2023-10-16T11:22:25.821 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:25.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f774006b020 msgr2=0x7f774006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f774006b020 0x7f774006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7750006b20 tx=0x7f7750008040 comp rx=0 tx=0).stop 2023-10-16T11:22:25.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 msgr2=0x7f775c152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c152990 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f7754002230 tx=0x7f775400cf50 comp rx=0 tx=0).stop 2023-10-16T11:22:25.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 shutdown_connections 2023-10-16T11:22:25.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f774006b020 0x7f774006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c06e5d0 0x7f775c152990 unknown :-1 s=CLOSED pgs=3413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f775c06ab40 0x7f775c0b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 --2- 172.21.15.158:0/3709675166 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f775c06af20 0x7f775c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 >> 172.21.15.158:0/3709675166 conn(0x7f775c0d02b0 msgr2=0x7f775c072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:25.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.820+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 shutdown_connections 2023-10-16T11:22:25.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.821+0000 7f7762fe9700 1 -- 172.21.15.158:0/3709675166 wait complete. 2023-10-16T11:22:25.826 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:22:25.834 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-16T11:22:25.507627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":444544,"kb_used_data":28488,"kb_used_omap":12,"kb_used_meta":415923,"kb_avail":754530176,"statfs":{"total":773094113280,"available":772638900224,"internally_reserved":0,"allocated":29171712,"data_stored":16293480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":425905719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-530476,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":10,"num_read_kb":133,"num_write":-143,"num_write_kb":-1330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.997645"},"pg_stats":[{"pgid":"1.0","version":"273'414","reported_seq":1498,"reported_epoch":274,"state":"active+clean","last_fresh":"2023-10-16T11:22:20.661621+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:22:20.661621+0000","last_peered":"2023-10-16T11:22:20.661621+0000","last_clean":"2023-10-16T11:22:20.661621+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:22:20.661621+0000","last_undegraded":"2023-10-16T11:22:20.661621+0000","last_fullsized":"2023-10-16T11:22:20.661621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576019,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57384,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314456,"statfs":{"total":96636764160,"available":96578002944,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73799999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.437},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.68300000000000005,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.98199999999999998}]},{"osd":2,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.70399999999999996,"15min":0.66700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.875},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":9.516,"15min":9.516},"last":1.1890000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67000000000000004,"15min":0.64000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":1.087}]},{"osd":4,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.59299999999999997,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.624,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67400000000000004,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.64800000000000002,"15min":0.57899999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77800000000000002,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.68000000000000005,"15min":0.625},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":1.2809999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.70999999999999996,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.69299999999999995,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72199999999999998,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.70399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576019,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.57999999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.86299999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56499999999999995,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84299999999999997,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71899999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":2.387,"15min":2.387},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.314},"max":{"1min":0.78100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59899999999999998,"15min":0.61599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88800000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81000000000000005,"5min":1.02,"15min":1.02},"last":0.75900000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.73199999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":3.335,"15min":3.335},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":2.524,"15min":2.524},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.79700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.40799999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.315},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.26500000000000001,"5min":0.184,"15min":0.184},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36199999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.44},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.434}]},{"osd":3,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.498},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0309999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.0389999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.625}]},{"osd":6,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.629,"15min":0.66500000000000004},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.67200000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64500000000000002,"15min":0.68999999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.56999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834576020,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49072,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322768,"statfs":{"total":96636764160,"available":96586514432,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.55500000000000005,"15min":0.60299999999999998},"last":0.376},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40300000000000002,"15min":0.439},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.44500000000000001,"5min":0.58399999999999996,"15min":0.77200000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50700000000000001,"5min":1.272,"15min":1.272},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.50600000000000001,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":1.351,"15min":1.351},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.64200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.45100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":1.391,"15min":1.391},"last":0.80600000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60599999999999998,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.874,"15min":1.089},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.216},"last":0.76500000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64300000000000002,"15min":0.68600000000000005},"min":{"1min":0.46600000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":1.038,"15min":1.1180000000000001},"last":0.79100000000000004}]},{"osd":7,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88900000000000001,"5min":0.97399999999999998,"15min":1.008},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.66900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.88500000000000001,"15min":1.0780000000000001},"last":0.57099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576019,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.39300000000000002},"min":{"1min":0.245,"5min":0.188,"15min":0.188},"max":{"1min":0.47699999999999998,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.437},"min":{"1min":0.255,"5min":0.193,"15min":0.193},"max":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.436,"15min":0.46400000000000002},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.41099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45300000000000001,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.32200000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.432}]},{"osd":4,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":1.097},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86499999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83099999999999996,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92700000000000005,"5min":1.0169999999999999,"15min":1.028},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.498,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85899999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.75}]},{"osd":7,"last update":"Mon Oct 16 11:21:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.70399999999999996},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":0.99099999999999999,"15min":1.071},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.97799999999999998,"15min":1.077},"last":0.65700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.437,"5min":1.032,"15min":1.032},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35599999999999998,"15min":0.39800000000000002},"min":{"1min":0.27800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.51800000000000002,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.29499999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":1.341,"15min":1.341},"last":0.318},{"interface":"front","average":{"1min":0.375,"5min":0.39600000000000002,"15min":0.433},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47699999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.47099999999999997},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58399999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55700000000000005,"5min":0.96599999999999997,"15min":1.024},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.40600000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86499999999999999,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89100000000000001,"5min":2.629,"15min":2.629},"last":0.438}]},{"osd":5,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.66300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70299999999999996,"15min":0.69899999999999995},"min":{"1min":0.45700000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.68100000000000005,"15min":0.69099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85899999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.69799999999999995},"min":{"1min":0.44600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96899999999999997,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70599999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90900000000000003,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.90100000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576019,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.60099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":1.0589999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55800000000000005,"15min":0.60799999999999998},"min":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.88200000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54900000000000004,"15min":0.60099999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.90700000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55000000000000004,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55600000000000005,"15min":0.60899999999999999},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67600000000000005,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58699999999999997,"15min":0.626},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80500000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55000000000000004,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.751,"5min":0.79700000000000004,"15min":0.995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.499,"15min":0.5},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.86199999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71599999999999997,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.751,"5min":0.82899999999999996,"15min":1.0609999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81499999999999995,"5min":0.83899999999999997,"15min":1.115},"last":0.75600000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.42199999999999999,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.76800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66536,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305304,"statfs":{"total":96636764160,"available":96568631296,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56000000000000005,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.871,"15min":0.88200000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.58399999999999996},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73199999999999998,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.57699999999999996},"min":{"1min":0.311,"5min":0.309,"15min":0.309},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.747,"5min":0.82099999999999995,"15min":0.89400000000000002},"last":0.72999999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.59799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.628}]},{"osd":4,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76400000000000001,"5min":0.80900000000000005,"15min":0.84799999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.33500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.86399999999999999},"last":0.874}]},{"osd":6,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.753,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.85899999999999999,"15min":0.878},"last":0.89500000000000002}]},{"osd":7,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84699999999999998,"5min":0.871,"15min":0.871},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.89300000000000002},"last":0.85099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:22:25.836 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T11:22:25.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.954+0000 7faff3943700 1 Processor -- start 2023-10-16T11:22:25.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.955+0000 7faff3943700 1 -- start start 2023-10-16T11:22:25.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff3943700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fafec15dbc0 0x7fafec0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff3943700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec06ba90 0x7fafec0cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff3943700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec0d21e0 con 0x7fafec0d1d50 2023-10-16T11:22:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff3943700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafec162550 con 0x7fafec15dbc0 2023-10-16T11:22:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff3943700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec15f320 con 0x7fafec06ba90 2023-10-16T11:22:25.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff0ede700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fafec0d1d50 0x7fafec06beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45134/0 (socket says 172.21.15.158:45134) 2023-10-16T11:22:25.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff0ede700 1 -- 172.21.15.158:0/820603787 learned_addr learned my addr 172.21.15.158:0/820603787 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:25.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff16df700 1 --2- 172.21.15.158:0/820603787 >> v2:172.21.15.172:3300/0 conn(0x7fafec15dbc0 0x7fafec0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff1ee0700 1 --2- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec06ba90 0x7fafec0cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7fafe3fff700 1 -- 172.21.15.158:0/820603787 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944383375 0 0) 0x7fafec0d21e0 con 0x7fafec0d1d50 2023-10-16T11:22:25.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7fafe3fff700 1 -- 172.21.15.158:0/820603787 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd80035b0 con 0x7fafec0d1d50 2023-10-16T11:22:25.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff16df700 1 -- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec06ba90 msgr2=0x7fafec0cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:25.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff16df700 1 --2- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec06ba90 0x7fafec0cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7fafe3fff700 1 -- 172.21.15.158:0/820603787 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754799865 0 0) 0x7fafd80035b0 con 0x7fafec0d1d50 2023-10-16T11:22:25.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff16df700 1 -- 172.21.15.158:0/820603787 >> v1:172.21.15.158:6789/0 conn(0x7fafec0d1d50 legacy=0x7fafec06beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff16df700 1 -- 172.21.15.158:0/820603787 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec15ef10 con 0x7fafec15dbc0 2023-10-16T11:22:25.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.956+0000 7faff1ee0700 1 --2- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec06ba90 0x7fafec0cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:25.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff16df700 1 --2- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec15dbc0 0x7fafec0cefa0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009840 tx=0x7fafdc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b822d4d58cfd9048 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7fafe3fff700 1 -- 172.21.15.158:0/820603787 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8001670 con 0x7fafec15dbc0 2023-10-16T11:22:25.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7fafe3fff700 1 -- 172.21.15.158:0/820603787 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc004030 con 0x7fafec15dbc0 2023-10-16T11:22:25.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7fafe3fff700 1 -- 172.21.15.158:0/820603787 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc0043a0 con 0x7fafec15dbc0 2023-10-16T11:22:25.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 -- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec15dbc0 msgr2=0x7fafec0cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 --2- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec15dbc0 0x7fafec0cefa0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009840 tx=0x7fafdc009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:25.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 -- 172.21.15.158:0/820603787 shutdown_connections 2023-10-16T11:22:25.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 --2- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec15dbc0 0x7fafec0cefa0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 --2- 172.21.15.158:0/820603787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec06ba90 0x7fafec0cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 -- 172.21.15.158:0/820603787 >> 172.21.15.158:0/820603787 conn(0x7fafec06a8d0 msgr2=0x7fafec06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:25.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 -- 172.21.15.158:0/820603787 shutdown_connections 2023-10-16T11:22:25.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.957+0000 7faff3943700 1 -- 172.21.15.158:0/820603787 wait complete. 2023-10-16T11:22:25.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 Processor -- start 2023-10-16T11:22:25.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 -- start start 2023-10-16T11:22:25.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafec06ba90 0x7fafec0c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec0c6200 0x7fafec14e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec0b6250 0x7fafec0b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0b6c40 con 0x7fafec06ba90 2023-10-16T11:22:25.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0c66a0 con 0x7fafec0c6200 2023-10-16T11:22:25.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.959+0000 7faff3943700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0c6980 con 0x7fafec0b6250 2023-10-16T11:22:25.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafec06ba90 0x7fafec0c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafec06ba90 0x7fafec0c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44962/0 (socket says 172.21.15.158:44962) 2023-10-16T11:22:25.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 -- 172.21.15.158:0/1169132339 learned_addr learned my addr 172.21.15.158:0/1169132339 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:25.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff1ee0700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec0b6250 0x7fafec0b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff0ede700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec0c6200 0x7fafec14e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 -- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec0b6250 msgr2=0x7fafec0b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec0b6250 0x7fafec0b6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 -- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec0c6200 msgr2=0x7fafec14e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:25.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec0c6200 0x7fafec14e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:25.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 -- 172.21.15.158:0/1169132339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe4009a70 con 0x7fafec06ba90 2023-10-16T11:22:25.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff16df700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafec06ba90 0x7fafec0c7a90 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7fafdc000c00 tx=0x7fafdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7fafe1ffb700 1 -- 172.21.15.158:0/1169132339 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc003df0 con 0x7fafec06ba90 2023-10-16T11:22:25.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7fafe1ffb700 1 -- 172.21.15.158:0/1169132339 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc01f070 con 0x7fafec06ba90 2023-10-16T11:22:25.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc0094a0 con 0x7fafec06ba90 2023-10-16T11:22:25.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7fafe1ffb700 1 -- 172.21.15.158:0/1169132339 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc011870 con 0x7fafec06ba90 2023-10-16T11:22:25.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.960+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec0c6f40 con 0x7fafec06ba90 2023-10-16T11:22:25.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.961+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec0681b0 con 0x7fafec06ba90 2023-10-16T11:22:25.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.962+0000 7fafe1ffb700 1 -- 172.21.15.158:0/1169132339 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fafdc021030 con 0x7fafec06ba90 2023-10-16T11:22:25.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.962+0000 7fafe1ffb700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fafd006b020 0x7fafd006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:25.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.962+0000 7fafe1ffb700 1 -- 172.21.15.158:0/1169132339 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fafdc08ae50 con 0x7fafec06ba90 2023-10-16T11:22:25.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.962+0000 7faff0ede700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fafd006b020 0x7fafd006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:25.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.963+0000 7faff0ede700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fafd006b020 0x7fafd006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafec0cdae0 tx=0x7fafe8009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:25.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:25.967+0000 7fafe1ffb700 1 -- 172.21.15.158:0/1169132339 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fafdc055bb0 con 0x7fafec06ba90 2023-10-16T11:22:26.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.159+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafec0b3bf0 con 0x7fafec06ba90 2023-10-16T11:22:26.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.160+0000 7fafe1ffb700 1 -- 172.21.15.158:0/1169132339 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+14958 (secure 0 0 0) 0x7fafdc0591d0 con 0x7fafec06ba90 2023-10-16T11:22:26.163 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:26.164 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":275,"fsid":"7cf363df-9988-44b1-be05-df2a6bf24f2d","created":"2023-10-16T09:43:03.598607+0000","modified":"2023-10-16T11:22:21.657943+0000","last_up_change":"2023-10-16T09:43:14.866254+0000","last_in_change":"2023-10-16T09:43:06.863253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T09:43:15.258805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cca4abed-4408-40e4-951f-5821a22d0170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6811","nonce":38337391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6815","nonce":38337391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6823","nonce":38337391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6817","nonce":38337391},{"type":"v1","addr":"172.21.15.158:6819","nonce":38337391}]},"public_addr":"172.21.15.158:6811/38337391","cluster_addr":"172.21.15.158:6815/38337391","heartbeat_back_addr":"172.21.15.158:6823/38337391","heartbeat_front_addr":"172.21.15.158:6819/38337391","state":["exists","up"]},{"osd":1,"uuid":"009ee1a1-f593-4806-835d-78efa006069f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6801","nonce":1822888471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6803","nonce":1822888471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6807","nonce":1822888471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":1822888471},{"type":"v1","addr":"172.21.15.158:6805","nonce":1822888471}]},"public_addr":"172.21.15.158:6801/1822888471","cluster_addr":"172.21.15.158:6803/1822888471","heartbeat_back_addr":"172.21.15.158:6807/1822888471","heartbeat_front_addr":"172.21.15.158:6805/1822888471","state":["exists","up"]},{"osd":2,"uuid":"ccc4ad87-c517-4106-9f8d-0da0c62b8e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6810","nonce":205317247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6814","nonce":205317247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6822","nonce":205317247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":205317247},{"type":"v1","addr":"172.21.15.158:6818","nonce":205317247}]},"public_addr":"172.21.15.158:6810/205317247","cluster_addr":"172.21.15.158:6814/205317247","heartbeat_back_addr":"172.21.15.158:6822/205317247","heartbeat_front_addr":"172.21.15.158:6818/205317247","state":["exists","up"]},{"osd":3,"uuid":"f4d08d1c-e139-471c-ace8-55a28a0c0b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6825","nonce":969498875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6827","nonce":969498875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6831","nonce":969498875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":969498875},{"type":"v1","addr":"172.21.15.158:6829","nonce":969498875}]},"public_addr":"172.21.15.158:6825/969498875","cluster_addr":"172.21.15.158:6827/969498875","heartbeat_back_addr":"172.21.15.158:6831/969498875","heartbeat_front_addr":"172.21.15.158:6829/969498875","state":["exists","up"]},{"osd":4,"uuid":"0d0a0844-bf1a-41e7-ae47-58602557ea7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6825","nonce":3174967294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6827","nonce":3174967294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6831","nonce":3174967294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3174967294},{"type":"v1","addr":"172.21.15.172:6829","nonce":3174967294}]},"public_addr":"172.21.15.172:6825/3174967294","cluster_addr":"172.21.15.172:6827/3174967294","heartbeat_back_addr":"172.21.15.172:6831/3174967294","heartbeat_front_addr":"172.21.15.172:6829/3174967294","state":["exists","up"]},{"osd":5,"uuid":"886a6d94-630e-4241-945b-5cc0d4c8f848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6817","nonce":2983726905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6819","nonce":2983726905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6823","nonce":2983726905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2983726905},{"type":"v1","addr":"172.21.15.172:6821","nonce":2983726905}]},"public_addr":"172.21.15.172:6817/2983726905","cluster_addr":"172.21.15.172:6819/2983726905","heartbeat_back_addr":"172.21.15.172:6823/2983726905","heartbeat_front_addr":"172.21.15.172:6821/2983726905","state":["exists","up"]},{"osd":6,"uuid":"20128664-438a-4be4-9ced-2375eb76d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6801","nonce":3389147915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6803","nonce":3389147915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6807","nonce":3389147915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3389147915},{"type":"v1","addr":"172.21.15.172:6805","nonce":3389147915}]},"public_addr":"172.21.15.172:6801/3389147915","cluster_addr":"172.21.15.172:6803/3389147915","heartbeat_back_addr":"172.21.15.172:6807/3389147915","heartbeat_front_addr":"172.21.15.172:6805/3389147915","state":["exists","up"]},{"osd":7,"uuid":"b551f3a3-652d-48ad-ba98-dc3894a4bd38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6809","nonce":2085135193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6811","nonce":2085135193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6815","nonce":2085135193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2085135193},{"type":"v1","addr":"172.21.15.172:6813","nonce":2085135193}]},"public_addr":"172.21.15.172:6809/2085135193","cluster_addr":"172.21.15.172:6811/2085135193","heartbeat_back_addr":"172.21.15.172:6815/2085135193","heartbeat_front_addr":"172.21.15.172:6813/2085135193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.066884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:11.791162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.017109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.070681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.143993+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.068010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.097400+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T09:43:12.074770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6835/284711673":"2023-10-17T11:22:18.504254+0000","172.21.15.172:6833/284711673":"2023-10-17T11:22:18.504254+0000","172.21.15.158:6833/224988149":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/2492444608":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/580008490":"2023-10-17T11:22:17.488860+0000","172.21.15.158:6832/224988149":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/662944544":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/183508379":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/3875121437":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/1289746000":"2023-10-17T11:22:17.488860+0000","172.21.15.158:0/3856677823":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/1867601061":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/933196924":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/3733531003":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6833/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:6832/778561374":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/915172958":"2023-10-17T11:22:09.987990+0000","172.21.15.158:0/258064180":"2023-10-17T11:22:09.987990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":10,"length":1},{"begin":15,"length":1},{"begin":21,"length":1},{"begin":25,"length":1},{"begin":31,"length":2},{"begin":36,"length":1},{"begin":43,"length":1},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":57,"length":1},{"begin":63,"length":1},{"begin":67,"length":1},{"begin":74,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":111,"length":1},{"begin":198,"length":24}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T11:22:26.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fafd006b020 msgr2=0x7fafd006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:26.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fafd006b020 0x7fafd006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafec0cdae0 tx=0x7fafe8009040 comp rx=0 tx=0).stop 2023-10-16T11:22:26.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafec06ba90 msgr2=0x7fafec0c7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:26.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafec06ba90 0x7fafec0c7a90 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7fafdc000c00 tx=0x7fafdc004750 comp rx=0 tx=0).stop 2023-10-16T11:22:26.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 shutdown_connections 2023-10-16T11:22:26.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fafd006b020 0x7fafd006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:26.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fafec0b6250 0x7fafec0b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:26.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafec06ba90 0x7fafec0c7a90 unknown :-1 s=CLOSED pgs=3415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:26.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 --2- 172.21.15.158:0/1169132339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafec0c6200 0x7fafec14e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:26.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 >> 172.21.15.158:0/1169132339 conn(0x7fafec06a8d0 msgr2=0x7fafec06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:26.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 shutdown_connections 2023-10-16T11:22:26.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:26.163+0000 7faff3943700 1 -- 172.21.15.158:0/1169132339 wait complete. 2023-10-16T11:22:27.178 INFO:tasks.ceph:Scrubbing osd.0 2023-10-16T11:22:27.179 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:27.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.315+0000 7f1463586700 1 Processor -- start 2023-10-16T11:22:27.317 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.315+0000 7f1463586700 1 -- start start 2023-10-16T11:22:27.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1463586700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f145c15dad0 0x7f145c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1463586700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c0d0ae0 con 0x7f145c15deb0 2023-10-16T11:22:27.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1463586700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c0d12d0 con 0x7f145c0ccc00 2023-10-16T11:22:27.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1463586700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145c0cbdf0 con 0x7f145c15dad0 2023-10-16T11:22:27.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1460b21700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f145c15deb0 0x7f145c0ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45146/0 (socket says 172.21.15.158:45146) 2023-10-16T11:22:27.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1460b21700 1 -- 172.21.15.158:0/286750926 learned_addr learned my addr 172.21.15.158:0/286750926 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:27.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1461322700 1 --2- 172.21.15.158:0/286750926 >> v2:172.21.15.172:3300/0 conn(0x7f145c15dad0 0x7f145c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797227694 0 0) 0x7f145c0d0ae0 con 0x7f145c15deb0 2023-10-16T11:22:27.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.316+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14480035b0 con 0x7f145c15deb0 2023-10-16T11:22:27.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2287497395 0 0) 0x7f14480035b0 con 0x7f145c15deb0 2023-10-16T11:22:27.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 >> v2:172.21.15.172:3300/0 conn(0x7f145c15dad0 msgr2=0x7f145c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 --2- 172.21.15.158:0/286750926 >> v2:172.21.15.172:3300/0 conn(0x7f145c15dad0 0x7f145c0cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 >> v1:172.21.15.172:6789/0 conn(0x7f145c0ccc00 legacy=0x7f145c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c0cc280 con 0x7f145c15deb0 2023-10-16T11:22:27.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f1458001670 con 0x7f145c15deb0 2023-10-16T11:22:27.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 >> v1:172.21.15.158:6789/0 conn(0x7f145c15deb0 legacy=0x7f145c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1448003a00 0x7f1448003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448004850 0x7f1448004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1448005210 0x7f144800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0cbdf0 con 0x7f1448005210 2023-10-16T11:22:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144800c1b0 con 0x7f1448003a00 2023-10-16T11:22:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144800c370 con 0x7f1448004850 2023-10-16T11:22:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1448005210 0x7f144800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461322700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1448003a00 0x7f1448003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1460b21700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448004850 0x7f1448004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 -- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448004850 msgr2=0x7f1448004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448004850 0x7f1448004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 -- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1448003a00 msgr2=0x7f1448003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1448003a00 0x7f1448003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 -- 172.21.15.158:0/286750926 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480035b0 con 0x7f1448005210 2023-10-16T11:22:27.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 -- 172.21.15.158:0/286750926 reap_dead start 2023-10-16T11:22:27.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1461b23700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1448005210 0x7f144800ba10 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f145400c190 tx=0x7f1448005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce45bcc64fc588f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1454001db0 con 0x7f1448005210 2023-10-16T11:22:27.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145400ccf0 con 0x7f1448005210 2023-10-16T11:22:27.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.317+0000 7f1453fff700 1 -- 172.21.15.158:0/286750926 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1454003c40 con 0x7f1448005210 2023-10-16T11:22:27.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.318+0000 7f1463586700 1 -- 172.21.15.158:0/286750926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1448005210 msgr2=0x7f144800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.318+0000 7f1463586700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1448005210 0x7f144800ba10 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f145400c190 tx=0x7f1448005880 comp rx=0 tx=0).stop 2023-10-16T11:22:27.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.318+0000 7f1463586700 1 -- 172.21.15.158:0/286750926 shutdown_connections 2023-10-16T11:22:27.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.318+0000 7f1463586700 1 --2- 172.21.15.158:0/286750926 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1448005210 0x7f144800ba10 unknown :-1 s=CLOSED pgs=3417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.318+0000 7f1463586700 1 -- 172.21.15.158:0/286750926 >> 172.21.15.158:0/286750926 conn(0x7f145c099a10 msgr2=0x7f145c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:27.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.318+0000 7f1463586700 1 -- 172.21.15.158:0/286750926 shutdown_connections 2023-10-16T11:22:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.318+0000 7f1463586700 1 -- 172.21.15.158:0/286750926 wait complete. 2023-10-16T11:22:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.319+0000 7f1463586700 1 Processor -- start 2023-10-16T11:22:27.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.319+0000 7f1463586700 1 -- start start 2023-10-16T11:22:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1463586700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448005210 0x7f145c152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1463586700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f145c0b51d0 0x7f145c0b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1463586700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145c15b4f0 0x7f145c15b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1463586700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0c9690 con 0x7f145c0b51d0 2023-10-16T11:22:27.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1463586700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0b5b90 con 0x7f145c15b4f0 2023-10-16T11:22:27.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1463586700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0c9ab0 con 0x7f1448005210 2023-10-16T11:22:27.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1460b21700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f145c0b51d0 0x7f145c0b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1460b21700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f145c0b51d0 0x7f145c0b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:44984/0 (socket says 172.21.15.158:44984) 2023-10-16T11:22:27.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1460b21700 1 -- 172.21.15.158:0/2735681379 learned_addr learned my addr 172.21.15.158:0/2735681379 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:27.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1460b21700 1 -- 172.21.15.158:0/2735681379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448005210 msgr2=0x7f145c152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.320+0000 7f1461b23700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145c15b4f0 0x7f145c15b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1461322700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448005210 0x7f145c152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1460b21700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1448005210 0x7f145c152870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1460b21700 1 -- 172.21.15.158:0/2735681379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145c15b4f0 msgr2=0x7f145c15b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1460b21700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145c15b4f0 0x7f145c15b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1460b21700 1 -- 172.21.15.158:0/2735681379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c0c8a10 con 0x7f145c0b51d0 2023-10-16T11:22:27.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1460b21700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f145c0b51d0 0x7f145c0b5650 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f14580072c0 tx=0x7f1458007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580077a0 con 0x7f145c0b51d0 2023-10-16T11:22:27.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145800f8d0 con 0x7f145c0b51d0 2023-10-16T11:22:27.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c0c8cf0 con 0x7f145c0b51d0 2023-10-16T11:22:27.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800fc80 con 0x7f145c0b51d0 2023-10-16T11:22:27.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.321+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145c0c92b0 con 0x7f145c0b51d0 2023-10-16T11:22:27.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.322+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f145c0d0ff0 con 0x7f145c0b51d0 2023-10-16T11:22:27.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.322+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f145801d460 con 0x7f145c0b51d0 2023-10-16T11:22:27.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1451ffb700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f144006b020 0x7f144006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f14580899e0 con 0x7f145c0b51d0 2023-10-16T11:22:27.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1461322700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f144006b020 0x7f144006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1451ffb700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f1440071e10 0x7f1440074280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 --> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14400748b0 con 0x7f1440071e10 2023-10-16T11:22:27.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7f145808e2c0 con 0x7f145c0b51d0 2023-10-16T11:22:27.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1461b23700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f1440071e10 0x7f1440074280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1461322700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f144006b020 0x7f144006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f144c009a20 tx=0x7f144c008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.323+0000 7f1461b23700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f1440071e10 0x7f1440074280 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.324+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== osd.0 v2:172.21.15.158:6809/38337391 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f14400748b0 con 0x7f1440071e10 2023-10-16T11:22:27.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.339+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 --> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f145c0d0d10 con 0x7f1440071e10 2023-10-16T11:22:27.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.339+0000 7f1451ffb700 1 -- 172.21.15.158:0/2735681379 <== osd.0 v2:172.21.15.158:6809/38337391 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1454001cd0 con 0x7f1440071e10 2023-10-16T11:22:27.350 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:27.351 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:27.351 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.339+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f1440071e10 msgr2=0x7f1440074280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.339+0000 7f1463586700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6809/38337391,v1:172.21.15.158:6811/38337391] conn(0x7f1440071e10 0x7f1440074280 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f144006b020 msgr2=0x7f144006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f144006b020 0x7f144006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f144c009a20 tx=0x7f144c008040 comp rx=0 tx=0).stop 2023-10-16T11:22:27.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f145c0b51d0 msgr2=0x7f145c0b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 --2- 172.21.15.158:0/2735681379 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f145c0b51d0 0x7f145c0b5650 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f14580072c0 tx=0x7f1458007680 comp rx=0 tx=0).stop 2023-10-16T11:22:27.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1461b23700 1 -- 172.21.15.158:0/2735681379 reap_dead start 2023-10-16T11:22:27.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 shutdown_connections 2023-10-16T11:22:27.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 >> 172.21.15.158:0/2735681379 conn(0x7f145c099a10 msgr2=0x7f145c0c6460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:27.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 shutdown_connections 2023-10-16T11:22:27.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.340+0000 7f1463586700 1 -- 172.21.15.158:0/2735681379 wait complete. 2023-10-16T11:22:27.357 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-16T11:22:27.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.477+0000 7f454eacb700 1 Processor -- start 2023-10-16T11:22:27.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.477+0000 7f454eacb700 1 -- start start 2023-10-16T11:22:27.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f454eacb700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f454815a6d0 0x7f45480ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f454eacb700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454815ec70 con 0x7f45480cfe30 2023-10-16T11:22:27.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f454eacb700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454815f1b0 con 0x7f45480cd360 2023-10-16T11:22:27.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f454eacb700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45480afcd0 con 0x7f454815a6d0 2023-10-16T11:22:27.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f4547fff700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f45480cfe30 0x7f45480ccc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45170/0 (socket says 172.21.15.158:45170) 2023-10-16T11:22:27.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f4547fff700 1 -- 172.21.15.158:0/159549659 learned_addr learned my addr 172.21.15.158:0/159549659 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:27.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f454c867700 1 --2- 172.21.15.158:0/159549659 >> v2:172.21.15.172:3300/0 conn(0x7f454815a6d0 0x7f45480ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484497006 0 0) 0x7f454815ec70 con 0x7f45480cfe30 2023-10-16T11:22:27.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45380035b0 con 0x7f45480cfe30 2023-10-16T11:22:27.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914692961 0 0) 0x7f454815f1b0 con 0x7f45480cd360 2023-10-16T11:22:27.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454815ec70 con 0x7f45480cd360 2023-10-16T11:22:27.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 217981878 0 0) 0x7f45380035b0 con 0x7f45480cfe30 2023-10-16T11:22:27.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 >> v2:172.21.15.172:3300/0 conn(0x7f454815a6d0 msgr2=0x7f45480ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 --2- 172.21.15.158:0/159549659 >> v2:172.21.15.172:3300/0 conn(0x7f454815a6d0 0x7f45480ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 >> v1:172.21.15.172:6789/0 conn(0x7f45480cd360 legacy=0x7f454806ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480cab50 con 0x7f45480cfe30 2023-10-16T11:22:27.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f4534001670 con 0x7f45480cfe30 2023-10-16T11:22:27.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.478+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 >> v1:172.21.15.158:6789/0 conn(0x7f45480cfe30 legacy=0x7f45480ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4538003be0 0x7f4538003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538004850 0x7f4538004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4538005210 0x7f453800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45480afcd0 con 0x7f4538003be0 2023-10-16T11:22:27.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453800c0e0 con 0x7f4538005210 2023-10-16T11:22:27.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453800c2a0 con 0x7f4538004850 2023-10-16T11:22:27.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454c867700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4538003be0 0x7f4538003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f4547fff700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538004850 0x7f4538004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454d068700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4538005210 0x7f453800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454c867700 1 -- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538004850 msgr2=0x7f4538004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454c867700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538004850 0x7f4538004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454c867700 1 -- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4538005210 msgr2=0x7f453800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454c867700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4538005210 0x7f453800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454d068700 1 -- 172.21.15.158:0/159549659 reap_dead start 2023-10-16T11:22:27.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454c867700 1 -- 172.21.15.158:0/159549659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45380035b0 con 0x7f4538003be0 2023-10-16T11:22:27.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f454c867700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4538003be0 0x7f4538003fe0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f453c009b60 tx=0x7f4538005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc011804279eb0f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c00e6c0 con 0x7f4538003be0 2023-10-16T11:22:27.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c00e820 con 0x7f4538003be0 2023-10-16T11:22:27.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.479+0000 7f45477fe700 1 -- 172.21.15.158:0/159549659 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c00eb90 con 0x7f4538003be0 2023-10-16T11:22:27.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.480+0000 7f454eacb700 1 -- 172.21.15.158:0/159549659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4538003be0 msgr2=0x7f4538003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.480+0000 7f454eacb700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4538003be0 0x7f4538003fe0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f453c009b60 tx=0x7f4538005880 comp rx=0 tx=0).stop 2023-10-16T11:22:27.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.480+0000 7f454eacb700 1 -- 172.21.15.158:0/159549659 shutdown_connections 2023-10-16T11:22:27.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.480+0000 7f454eacb700 1 --2- 172.21.15.158:0/159549659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4538003be0 0x7f4538003fe0 unknown :-1 s=CLOSED pgs=3420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.480+0000 7f454eacb700 1 -- 172.21.15.158:0/159549659 >> 172.21.15.158:0/159549659 conn(0x7f454806b9e0 msgr2=0x7f454806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:27.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.480+0000 7f454eacb700 1 -- 172.21.15.158:0/159549659 shutdown_connections 2023-10-16T11:22:27.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.480+0000 7f454eacb700 1 -- 172.21.15.158:0/159549659 wait complete. 2023-10-16T11:22:27.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.481+0000 7f454eacb700 1 Processor -- start 2023-10-16T11:22:27.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.481+0000 7f454eacb700 1 -- start start 2023-10-16T11:22:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454eacb700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538003be0 0x7f45480b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454eacb700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45480c48e0 0x7f45480c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454eacb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45480b09f0 0x7f45480b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454eacb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45480b1ca0 con 0x7f45480b09f0 2023-10-16T11:22:27.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454eacb700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45480c52a0 con 0x7f45480c48e0 2023-10-16T11:22:27.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454eacb700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45480c5560 con 0x7f4538003be0 2023-10-16T11:22:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45480b09f0 0x7f45480b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45480b09f0 0x7f45480b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45008/0 (socket says 172.21.15.158:45008) 2023-10-16T11:22:27.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 -- 172.21.15.158:0/1442966201 learned_addr learned my addr 172.21.15.158:0/1442966201 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454c867700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538003be0 0x7f45480b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f4547fff700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45480c48e0 0x7f45480c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 -- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538003be0 msgr2=0x7f45480b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538003be0 0x7f45480b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 -- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45480c48e0 msgr2=0x7f45480c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45480c48e0 0x7f45480c4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.482+0000 7f454d068700 1 -- 172.21.15.158:0/1442966201 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480c5840 con 0x7f45480b09f0 2023-10-16T11:22:27.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.483+0000 7f454d068700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45480b09f0 0x7f45480b0e70 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f45480718d0 tx=0x7f454000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.483+0000 7f45457fa700 1 -- 172.21.15.158:0/1442966201 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540003c10 con 0x7f45480b09f0 2023-10-16T11:22:27.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.483+0000 7f45457fa700 1 -- 172.21.15.158:0/1442966201 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4540003d70 con 0x7f45480b09f0 2023-10-16T11:22:27.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.483+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45480c5b20 con 0x7f45480b09f0 2023-10-16T11:22:27.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.483+0000 7f45457fa700 1 -- 172.21.15.158:0/1442966201 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540005310 con 0x7f45480b09f0 2023-10-16T11:22:27.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.483+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45480c60e0 con 0x7f45480b09f0 2023-10-16T11:22:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.484+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45480b06b0 con 0x7f45480b09f0 2023-10-16T11:22:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.484+0000 7f45457fa700 1 -- 172.21.15.158:0/1442966201 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4540005470 con 0x7f45480b09f0 2023-10-16T11:22:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.484+0000 7f45457fa700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f452c06b020 0x7f452c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.484+0000 7f45457fa700 1 -- 172.21.15.158:0/1442966201 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4540089940 con 0x7f45480b09f0 2023-10-16T11:22:27.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.485+0000 7f454c867700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f452c06b020 0x7f452c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.485+0000 7f454c867700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f452c06b020 0x7f452c06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f453c0090b0 tx=0x7f453c01e040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.489+0000 7f45457fa700 1 -- 172.21.15.158:0/1442966201 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4540054610 con 0x7f45480b09f0 2023-10-16T11:22:27.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.682+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f454815eed0 con 0x7f452c06b020 2023-10-16T11:22:27.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.683+0000 7f45457fa700 1 -- 172.21.15.158:0/1442966201 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f453c00eb10 con 0x7f452c06b020 2023-10-16T11:22:27.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f452c06b020 msgr2=0x7f452c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f452c06b020 0x7f452c06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f453c0090b0 tx=0x7f453c01e040 comp rx=0 tx=0).stop 2023-10-16T11:22:27.687 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45480b09f0 msgr2=0x7f45480b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45480b09f0 0x7f45480b0e70 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f45480718d0 tx=0x7f454000edc0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 shutdown_connections 2023-10-16T11:22:27.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f452c06b020 0x7f452c06d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45480b09f0 0x7f45480b0e70 unknown :-1 s=CLOSED pgs=3421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4538003be0 0x7f45480b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 --2- 172.21.15.158:0/1442966201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45480c48e0 0x7f45480c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 >> 172.21.15.158:0/1442966201 conn(0x7f454806b9e0 msgr2=0x7f45480aef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:27.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 shutdown_connections 2023-10-16T11:22:27.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.685+0000 7f454eacb700 1 -- 172.21.15.158:0/1442966201 wait complete. 2023-10-16T11:22:27.690 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 0 to deep-scrub 2023-10-16T11:22:27.698 INFO:tasks.ceph:Scrubbing osd.1 2023-10-16T11:22:27.699 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:27.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.822+0000 7f12865f2700 1 Processor -- start 2023-10-16T11:22:27.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.822+0000 7f12865f2700 1 -- start start 2023-10-16T11:22:27.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f12865f2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f128015dad0 0x7f12800cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f12865f2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f128015deb0 0x7f12800ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f12865f2700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800d0ae0 con 0x7f12800cea20 2023-10-16T11:22:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f12865f2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12800cee50 con 0x7f128015dad0 2023-10-16T11:22:27.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f12865f2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12800d12d0 con 0x7f128015deb0 2023-10-16T11:22:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f127ffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f128015dad0 0x7f12800cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f1284b8f700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f12800cea20 0x7f12800c3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45192/0 (socket says 172.21.15.158:45192) 2023-10-16T11:22:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f1284b8f700 1 -- 172.21.15.158:0/1481311400 learned_addr learned my addr 172.21.15.158:0/1481311400 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:27.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.823+0000 7f127f7fe700 1 --2- 172.21.15.158:0/1481311400 >> v2:172.21.15.172:3300/0 conn(0x7f128015deb0 0x7f12800ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127ffff700 1 -- 172.21.15.158:0/1481311400 >> v2:172.21.15.172:3300/0 conn(0x7f128015deb0 msgr2=0x7f12800ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127ffff700 1 --2- 172.21.15.158:0/1481311400 >> v2:172.21.15.172:3300/0 conn(0x7f128015deb0 0x7f12800ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127ffff700 1 -- 172.21.15.158:0/1481311400 >> v1:172.21.15.158:6789/0 conn(0x7f12800cea20 legacy=0x7f12800c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127effd700 1 -- 172.21.15.158:0/1481311400 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601478054 0 0) 0x7f12800d0ae0 con 0x7f12800cea20 2023-10-16T11:22:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127ffff700 1 -- 172.21.15.158:0/1481311400 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800cbdf0 con 0x7f128015dad0 2023-10-16T11:22:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127f7fe700 1 --2- 172.21.15.158:0/1481311400 >> v2:172.21.15.172:3300/0 conn(0x7f128015deb0 0x7f12800ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:27.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127ffff700 1 --2- 172.21.15.158:0/1481311400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f128015dad0 0x7f12800cdfa0 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7f1270009840 tx=0x7f1270009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9990bf0d795b17 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127effd700 1 -- 172.21.15.158:0/1481311400 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127000f8a0 con 0x7f128015dad0 2023-10-16T11:22:27.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127effd700 1 -- 172.21.15.158:0/1481311400 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127000fa00 con 0x7f128015dad0 2023-10-16T11:22:27.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f127effd700 1 -- 172.21.15.158:0/1481311400 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127000fd90 con 0x7f128015dad0 2023-10-16T11:22:27.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 -- 172.21.15.158:0/1481311400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f128015dad0 msgr2=0x7f12800cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 --2- 172.21.15.158:0/1481311400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f128015dad0 0x7f12800cdfa0 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7f1270009840 tx=0x7f1270009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:27.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 -- 172.21.15.158:0/1481311400 shutdown_connections 2023-10-16T11:22:27.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 --2- 172.21.15.158:0/1481311400 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f128015dad0 0x7f12800cdfa0 unknown :-1 s=CLOSED pgs=3423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 --2- 172.21.15.158:0/1481311400 >> v2:172.21.15.172:3300/0 conn(0x7f128015deb0 0x7f12800ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 -- 172.21.15.158:0/1481311400 >> 172.21.15.158:0/1481311400 conn(0x7f1280099a10 msgr2=0x7f12800d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:27.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 -- 172.21.15.158:0/1481311400 shutdown_connections 2023-10-16T11:22:27.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.824+0000 7f12865f2700 1 -- 172.21.15.158:0/1481311400 wait complete. 2023-10-16T11:22:27.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.826+0000 7f12865f2700 1 Processor -- start 2023-10-16T11:22:27.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.826+0000 7f12865f2700 1 -- start start 2023-10-16T11:22:27.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.826+0000 7f12865f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12800cea20 0x7f12801f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.826+0000 7f12865f2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f128015dad0 0x7f12800c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.826+0000 7f12865f2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f128015deb0 0x7f12800c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f12865f2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280155ec0 con 0x7f12800cea20 2023-10-16T11:22:27.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f12865f2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12801564c0 con 0x7f128015deb0 2023-10-16T11:22:27.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f12865f2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12801f6090 con 0x7f128015dad0 2023-10-16T11:22:27.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12800cea20 0x7f12801f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12800cea20 0x7f12801f5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45018/0 (socket says 172.21.15.158:45018) 2023-10-16T11:22:27.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 -- 172.21.15.158:0/3962765657 learned_addr learned my addr 172.21.15.158:0/3962765657 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:27.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f1284b8f700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f128015deb0 0x7f12800c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127f7fe700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f128015dad0 0x7f12800c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 -- 172.21.15.158:0/3962765657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f128015dad0 msgr2=0x7f12800c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f128015dad0 0x7f12800c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 -- 172.21.15.158:0/3962765657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f128015deb0 msgr2=0x7f12800c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f128015deb0 0x7f12800c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 -- 172.21.15.158:0/3962765657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700094a0 con 0x7f12800cea20 2023-10-16T11:22:27.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127f7fe700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f128015dad0 0x7f12800c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:27.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127ffff700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12800cea20 0x7f12801f5b50 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f1270009b90 tx=0x7f1270004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12700038a0 con 0x7f12800cea20 2023-10-16T11:22:27.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1270003a00 con 0x7f12800cea20 2023-10-16T11:22:27.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12800c8700 con 0x7f12800cea20 2023-10-16T11:22:27.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270003d90 con 0x7f12800cea20 2023-10-16T11:22:27.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.827+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12800c8cc0 con 0x7f12800cea20 2023-10-16T11:22:27.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.828+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f12800d0d10 con 0x7f12800cea20 2023-10-16T11:22:27.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f127002c430 con 0x7f12800cea20 2023-10-16T11:22:27.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f127cff9700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f126c06b050 0x7f126c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f127008b3a0 con 0x7f12800cea20 2023-10-16T11:22:27.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f127f7fe700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f126c06b050 0x7f126c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f127cff9700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f126c071e40 0x7f126c0742b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 --> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c0748e0 con 0x7f126c071e40 2023-10-16T11:22:27.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7f127008b7f0 con 0x7f12800cea20 2023-10-16T11:22:27.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.829+0000 7f1284b8f700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f126c071e40 0x7f126c0742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.830+0000 7f127f7fe700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f126c06b050 0x7f126c06d510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f12800c9460 tx=0x7f1274008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.830+0000 7f1284b8f700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f126c071e40 0x7f126c0742b0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.830+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== osd.1 v2:172.21.15.158:6800/1822888471 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f126c0748e0 con 0x7f126c071e40 2023-10-16T11:22:27.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 --> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f12800d0ff0 con 0x7f126c071e40 2023-10-16T11:22:27.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f127cff9700 1 -- 172.21.15.158:0/3962765657 <== osd.1 v2:172.21.15.158:6800/1822888471 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1278007240 con 0x7f126c071e40 2023-10-16T11:22:27.850 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:27.850 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:27.851 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:27.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f126c071e40 msgr2=0x7f126c0742b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6800/1822888471,v1:172.21.15.158:6801/1822888471] conn(0x7f126c071e40 0x7f126c0742b0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f126c06b050 msgr2=0x7f126c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f126c06b050 0x7f126c06d510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f12800c9460 tx=0x7f1274008040 comp rx=0 tx=0).stop 2023-10-16T11:22:27.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12800cea20 msgr2=0x7f12801f5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 --2- 172.21.15.158:0/3962765657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12800cea20 0x7f12801f5b50 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f1270009b90 tx=0x7f1270004750 comp rx=0 tx=0).stop 2023-10-16T11:22:27.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f1284b8f700 1 -- 172.21.15.158:0/3962765657 reap_dead start 2023-10-16T11:22:27.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 shutdown_connections 2023-10-16T11:22:27.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 >> 172.21.15.158:0/3962765657 conn(0x7f1280099a10 msgr2=0x7f12800b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:27.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 shutdown_connections 2023-10-16T11:22:27.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.845+0000 7f12865f2700 1 -- 172.21.15.158:0/3962765657 wait complete. 2023-10-16T11:22:27.859 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-16T11:22:27.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.978+0000 7f6e337c1700 1 Processor -- start 2023-10-16T11:22:27.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.978+0000 7f6e337c1700 1 -- start start 2023-10-16T11:22:27.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e337c1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e2c158280 0x7f6e2c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e337c1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e337c1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c0cb2d0 con 0x7f6e2c15e210 2023-10-16T11:22:27.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e337c1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2c15bc40 con 0x7f6e2c158280 2023-10-16T11:22:27.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e337c1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c158660 con 0x7f6e2c0722e0 2023-10-16T11:22:27.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e3155d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e2c158280 0x7f6e2c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e3155d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e2c158280 0x7f6e2c06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45028/0 (socket says 172.21.15.158:45028) 2023-10-16T11:22:27.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e3155d700 1 -- 172.21.15.158:0/29886324 learned_addr learned my addr 172.21.15.158:0/29886324 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:27.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.979+0000 7f6e30d5c700 1 --2- 172.21.15.158:0/29886324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.985 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e3155d700 1 -- 172.21.15.158:0/29886324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 msgr2=0x7f6e2c0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:27.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e3155d700 1 --2- 172.21.15.158:0/29886324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e3155d700 1 -- 172.21.15.158:0/29886324 >> v1:172.21.15.172:6789/0 conn(0x7f6e2c15e210 legacy=0x7f6e2c0cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e3155d700 1 -- 172.21.15.158:0/29886324 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c15b030 con 0x7f6e2c158280 2023-10-16T11:22:27.986 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e3155d700 1 --2- 172.21.15.158:0/29886324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c158280 0x7f6e2c06a8d0 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0098f0 tx=0x7f6e1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96300bc7ffed2569 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e23fff700 1 -- 172.21.15.158:0/29886324 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c011830 con 0x7f6e2c158280 2023-10-16T11:22:27.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e23fff700 1 -- 172.21.15.158:0/29886324 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c011990 con 0x7f6e2c158280 2023-10-16T11:22:27.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e23fff700 1 -- 172.21.15.158:0/29886324 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c011d20 con 0x7f6e2c158280 2023-10-16T11:22:27.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e337c1700 1 -- 172.21.15.158:0/29886324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c158280 msgr2=0x7f6e2c06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e337c1700 1 --2- 172.21.15.158:0/29886324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c158280 0x7f6e2c06a8d0 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0098f0 tx=0x7f6e1c009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:27.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e337c1700 1 -- 172.21.15.158:0/29886324 shutdown_connections 2023-10-16T11:22:27.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e337c1700 1 --2- 172.21.15.158:0/29886324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c158280 0x7f6e2c06a8d0 unknown :-1 s=CLOSED pgs=3425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e337c1700 1 --2- 172.21.15.158:0/29886324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.980+0000 7f6e337c1700 1 -- 172.21.15.158:0/29886324 >> 172.21.15.158:0/29886324 conn(0x7f6e2c0cc700 msgr2=0x7f6e2c0ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:27.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.981+0000 7f6e337c1700 1 -- 172.21.15.158:0/29886324 shutdown_connections 2023-10-16T11:22:27.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.981+0000 7f6e337c1700 1 -- 172.21.15.158:0/29886324 wait complete. 2023-10-16T11:22:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.982+0000 7f6e337c1700 1 Processor -- start 2023-10-16T11:22:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.982+0000 7f6e337c1700 1 -- start start 2023-10-16T11:22:27.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e337c1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e337c1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e2c158280 0x7f6e2c1f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e337c1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c15e210 0x7f6e2c0711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:27.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e337c1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c0c6670 con 0x7f6e2c15e210 2023-10-16T11:22:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e337c1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c0c98e0 con 0x7f6e2c158280 2023-10-16T11:22:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e337c1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c0c5880 con 0x7f6e2c0722e0 2023-10-16T11:22:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c15e210 0x7f6e2c0711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c15e210 0x7f6e2c0711d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45038/0 (socket says 172.21.15.158:45038) 2023-10-16T11:22:27.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 -- 172.21.15.158:0/1310069879 learned_addr learned my addr 172.21.15.158:0/1310069879 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:27.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e30d5c700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e2c158280 0x7f6e2c1f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e3155d700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:27.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 -- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 msgr2=0x7f6e2c0c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 -- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e2c158280 msgr2=0x7f6e2c1f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:27.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e2c158280 0x7f6e2c1f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:27.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.983+0000 7f6e31d5e700 1 -- 172.21.15.158:0/1310069879 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c009550 con 0x7f6e2c15e210 2023-10-16T11:22:27.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.984+0000 7f6e31d5e700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c15e210 0x7f6e2c0711d0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400cb80 tx=0x7f6e2400ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.984+0000 7f6e21ffb700 1 -- 172.21.15.158:0/1310069879 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24003c00 con 0x7f6e2c15e210 2023-10-16T11:22:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.984+0000 7f6e21ffb700 1 -- 172.21.15.158:0/1310069879 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e24003d60 con 0x7f6e2c15e210 2023-10-16T11:22:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.984+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c1f47f0 con 0x7f6e2c15e210 2023-10-16T11:22:27.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.984+0000 7f6e21ffb700 1 -- 172.21.15.158:0/1310069879 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2400aaa0 con 0x7f6e2c15e210 2023-10-16T11:22:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.984+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c1f4db0 con 0x7f6e2c15e210 2023-10-16T11:22:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.985+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2c0681b0 con 0x7f6e2c15e210 2023-10-16T11:22:28.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.986+0000 7f6e21ffb700 1 -- 172.21.15.158:0/1310069879 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e2400ac00 con 0x7f6e2c15e210 2023-10-16T11:22:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.986+0000 7f6e21ffb700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6e1006b020 0x7f6e1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.986+0000 7f6e21ffb700 1 -- 172.21.15.158:0/1310069879 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6e24089960 con 0x7f6e2c15e210 2023-10-16T11:22:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.986+0000 7f6e3155d700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6e1006b020 0x7f6e1006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.986+0000 7f6e3155d700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6e1006b020 0x7f6e1006d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c000c00 tx=0x7f6e1c008030 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:27.990+0000 7f6e21ffb700 1 -- 172.21.15.158:0/1310069879 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e24054630 con 0x7f6e2c15e210 2023-10-16T11:22:28.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.185+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e2c15b5f0 con 0x7f6e1006b020 2023-10-16T11:22:28.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.185+0000 7f6e21ffb700 1 -- 172.21.15.158:0/1310069879 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6e1c011b80 con 0x7f6e1006b020 2023-10-16T11:22:28.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6e1006b020 msgr2=0x7f6e1006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6e1006b020 0x7f6e1006d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c000c00 tx=0x7f6e1c008030 comp rx=0 tx=0).stop 2023-10-16T11:22:28.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c15e210 msgr2=0x7f6e2c0711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c15e210 0x7f6e2c0711d0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400cb80 tx=0x7f6e2400ce90 comp rx=0 tx=0).stop 2023-10-16T11:22:28.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 shutdown_connections 2023-10-16T11:22:28.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f6e1006b020 0x7f6e1006d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e2c15e210 0x7f6e2c0711d0 unknown :-1 s=CLOSED pgs=3426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e2c0722e0 0x7f6e2c0c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 --2- 172.21.15.158:0/1310069879 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e2c158280 0x7f6e2c1f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 >> 172.21.15.158:0/1310069879 conn(0x7f6e2c0cc700 msgr2=0x7f6e2c06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 shutdown_connections 2023-10-16T11:22:28.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.187+0000 7f6e337c1700 1 -- 172.21.15.158:0/1310069879 wait complete. 2023-10-16T11:22:28.193 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 1 to deep-scrub 2023-10-16T11:22:28.201 INFO:tasks.ceph:Scrubbing osd.2 2023-10-16T11:22:28.201 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:28.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.324+0000 7f550cb45700 1 Processor -- start 2023-10-16T11:22:28.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.324+0000 7f550cb45700 1 -- start start 2023-10-16T11:22:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f550cb45700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f550815b6b0 0x7f55080cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f550cb45700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55080d0c60 0x7f55080b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f550cb45700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55080b4420 con 0x7f55080b3460 2023-10-16T11:22:28.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f550cb45700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f550815ba90 con 0x7f550815b6b0 2023-10-16T11:22:28.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f550cb45700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55080b38f0 con 0x7f55080d0c60 2023-10-16T11:22:28.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f550659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f550815b6b0 0x7f55080cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f5505d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55080d0c60 0x7f55080b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f5506d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f55080b3460 0x7f5508070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:49124/0 (socket says 172.21.15.158:49124) 2023-10-16T11:22:28.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.325+0000 7f5506d9d700 1 -- 172.21.15.158:0/2270095793 learned_addr learned my addr 172.21.15.158:0/2270095793 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:28.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550659c700 1 -- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55080d0c60 msgr2=0x7f55080b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:28.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550659c700 1 --2- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55080d0c60 0x7f55080b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550659c700 1 -- 172.21.15.158:0/2270095793 >> v1:172.21.15.172:6789/0 conn(0x7f55080b3460 legacy=0x7f5508070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550659c700 1 -- 172.21.15.158:0/2270095793 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55080cbc20 con 0x7f550815b6b0 2023-10-16T11:22:28.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550559a700 1 -- 172.21.15.158:0/2270095793 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855846264 0 0) 0x7f55080b4420 con 0x7f55080b3460 2023-10-16T11:22:28.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f5505d9b700 1 --2- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55080d0c60 0x7f55080b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:28.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550659c700 1 --2- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f550815b6b0 0x7f55080cfdb0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009840 tx=0x7f54f8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e40913700f95aa6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550559a700 1 -- 172.21.15.158:0/2270095793 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8011830 con 0x7f550815b6b0 2023-10-16T11:22:28.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550559a700 1 -- 172.21.15.158:0/2270095793 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8011990 con 0x7f550815b6b0 2023-10-16T11:22:28.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.326+0000 7f550559a700 1 -- 172.21.15.158:0/2270095793 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8011d20 con 0x7f550815b6b0 2023-10-16T11:22:28.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 -- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f550815b6b0 msgr2=0x7f55080cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 --2- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f550815b6b0 0x7f55080cfdb0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009840 tx=0x7f54f8009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:28.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 -- 172.21.15.158:0/2270095793 shutdown_connections 2023-10-16T11:22:28.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 --2- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f550815b6b0 0x7f55080cfdb0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 --2- 172.21.15.158:0/2270095793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55080d0c60 0x7f55080b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 -- 172.21.15.158:0/2270095793 >> 172.21.15.158:0/2270095793 conn(0x7f55080cdf20 msgr2=0x7f55080ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 -- 172.21.15.158:0/2270095793 shutdown_connections 2023-10-16T11:22:28.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.327+0000 7f550cb45700 1 -- 172.21.15.158:0/2270095793 wait complete. 2023-10-16T11:22:28.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.328+0000 7f550cb45700 1 Processor -- start 2023-10-16T11:22:28.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.328+0000 7f550cb45700 1 -- start start 2023-10-16T11:22:28.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550cb45700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55080b3460 0x7f55081f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550cb45700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55080d0c60 0x7f5508157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550cb45700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f550815b6b0 0x7f55081f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550cb45700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55081581a0 con 0x7f55080d0c60 2023-10-16T11:22:28.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550cb45700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55081f6720 con 0x7f55080b3460 2023-10-16T11:22:28.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550cb45700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55081f6a00 con 0x7f550815b6b0 2023-10-16T11:22:28.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55080b3460 0x7f55081f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f5505d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55080d0c60 0x7f5508157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55080b3460 0x7f55081f60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.158:46226/0 (socket says 172.21.15.158:46226) 2023-10-16T11:22:28.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f5505d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55080d0c60 0x7f5508157c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45048/0 (socket says 172.21.15.158:45048) 2023-10-16T11:22:28.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 -- 172.21.15.158:0/3279854424 learned_addr learned my addr 172.21.15.158:0/3279854424 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:28.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f5506d9d700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f550815b6b0 0x7f55081f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 -- 172.21.15.158:0/3279854424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f550815b6b0 msgr2=0x7f55081f2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f550815b6b0 0x7f55081f2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 -- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55080d0c60 msgr2=0x7f5508157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55080d0c60 0x7f5508157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 -- 172.21.15.158:0/3279854424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0094a0 con 0x7f55080b3460 2023-10-16T11:22:28.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.329+0000 7f550659c700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55080b3460 0x7f55081f60d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009b90 tx=0x7f54f8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.330+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f80036a0 con 0x7f55080b3460 2023-10-16T11:22:28.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.330+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f801f070 con 0x7f55080b3460 2023-10-16T11:22:28.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.330+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8003a40 con 0x7f55080b3460 2023-10-16T11:22:28.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.330+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f80094a0 con 0x7f55080b3460 2023-10-16T11:22:28.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.330+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55081f6fc0 con 0x7f55080b3460 2023-10-16T11:22:28.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.330+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f55080cc1e0 con 0x7f55080b3460 2023-10-16T11:22:28.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.331+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f54f8003ba0 con 0x7f55080b3460 2023-10-16T11:22:28.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.331+0000 7f54f6ffd700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f54e806b020 0x7f54e806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.331+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f54f8021070 con 0x7f55080b3460 2023-10-16T11:22:28.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.331+0000 7f5505d9b700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f54e806b020 0x7f54e806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.332+0000 7f54f6ffd700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7f54e8071e10 0x7f54e8074280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.332+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 --> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54e80748b0 con 0x7f54e8071e10 2023-10-16T11:22:28.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.332+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54f808d7f0 con 0x7f55080b3460 2023-10-16T11:22:28.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.332+0000 7f5506d9d700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7f54e8071e10 0x7f54e8074280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.332+0000 7f5505d9b700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f54e806b020 0x7f54e806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f55080b20a0 tx=0x7f54fc009210 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.332+0000 7f5506d9d700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7f54e8071e10 0x7f54e8074280 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.333+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== osd.2 v2:172.21.15.158:6808/205317247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f54e80748b0 con 0x7f54e8071e10 2023-10-16T11:22:28.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.347+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 --> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5508156550 con 0x7f54e8071e10 2023-10-16T11:22:28.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.347+0000 7f54f6ffd700 1 -- 172.21.15.158:0/3279854424 <== osd.2 v2:172.21.15.158:6808/205317247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5508156550 con 0x7f54e8071e10 2023-10-16T11:22:28.352 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:28.352 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:28.352 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:28.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.347+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7f54e8071e10 msgr2=0x7f54e8074280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.347+0000 7f550cb45700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6808/205317247,v1:172.21.15.158:6810/205317247] conn(0x7f54e8071e10 0x7f54e8074280 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f54e806b020 msgr2=0x7f54e806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f54e806b020 0x7f54e806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f55080b20a0 tx=0x7f54fc009210 comp rx=0 tx=0).stop 2023-10-16T11:22:28.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55080b3460 msgr2=0x7f55081f60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 --2- 172.21.15.158:0/3279854424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55080b3460 0x7f55081f60d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009b90 tx=0x7f54f8004750 comp rx=0 tx=0).stop 2023-10-16T11:22:28.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f5506d9d700 1 -- 172.21.15.158:0/3279854424 reap_dead start 2023-10-16T11:22:28.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 shutdown_connections 2023-10-16T11:22:28.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 >> 172.21.15.158:0/3279854424 conn(0x7f55080cdf20 msgr2=0x7f55080b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 shutdown_connections 2023-10-16T11:22:28.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.348+0000 7f550cb45700 1 -- 172.21.15.158:0/3279854424 wait complete. 2023-10-16T11:22:28.361 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-16T11:22:28.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.480+0000 7fcaf65d8700 1 Processor -- start 2023-10-16T11:22:28.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.480+0000 7fcaf65d8700 1 -- start start 2023-10-16T11:22:28.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.481+0000 7fcaf65d8700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcaf015a350 0x7fcaf00cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.481+0000 7fcaf65d8700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf015e270 con 0x7fcaf00cee70 2023-10-16T11:22:28.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.481+0000 7fcaf65d8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf015e550 con 0x7fcaf015a730 2023-10-16T11:22:28.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.481+0000 7fcaf65d8700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf015eef0 con 0x7fcaf015a350 2023-10-16T11:22:28.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.481+0000 7fcaeffff700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcaf015a350 0x7fcaf00cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.481+0000 7fcaf4b75700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcaf00cee70 0x7fcaf006f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45232/0 (socket says 172.21.15.158:45232) 2023-10-16T11:22:28.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.481+0000 7fcaf4b75700 1 -- 172.21.15.158:0/2381890849 learned_addr learned my addr 172.21.15.158:0/2381890849 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:28.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeffff700 1 -- 172.21.15.158:0/2381890849 >> v1:172.21.15.172:6789/0 conn(0x7fcaf015a730 legacy=0x7fcaf00d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeffff700 1 -- 172.21.15.158:0/2381890849 >> v1:172.21.15.158:6789/0 conn(0x7fcaf00cee70 legacy=0x7fcaf006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeffff700 1 -- 172.21.15.158:0/2381890849 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf015e830 con 0x7fcaf015a350 2023-10-16T11:22:28.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeeffd700 1 -- 172.21.15.158:0/2381890849 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647154643 0 0) 0x7fcaf015e270 con 0x7fcaf00cee70 2023-10-16T11:22:28.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeffff700 1 --2- 172.21.15.158:0/2381890849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf00cb660 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009840 tx=0x7fcae4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89843b7aeaab7b28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeeffd700 1 -- 172.21.15.158:0/2381890849 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4011830 con 0x7fcaf015a350 2023-10-16T11:22:28.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeeffd700 1 -- 172.21.15.158:0/2381890849 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae4011990 con 0x7fcaf015a350 2023-10-16T11:22:28.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaeeffd700 1 -- 172.21.15.158:0/2381890849 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4011d20 con 0x7fcaf015a350 2023-10-16T11:22:28.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2381890849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 msgr2=0x7fcaf00cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2381890849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf00cb660 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009840 tx=0x7fcae4009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:28.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2381890849 shutdown_connections 2023-10-16T11:22:28.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2381890849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf00cb660 unknown :-1 s=CLOSED pgs=3428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.482+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2381890849 >> 172.21.15.158:0/2381890849 conn(0x7fcaf00b1df0 msgr2=0x7fcaf00b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.483+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2381890849 shutdown_connections 2023-10-16T11:22:28.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.483+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2381890849 wait complete. 2023-10-16T11:22:28.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.484+0000 7fcaf65d8700 1 Processor -- start 2023-10-16T11:22:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.484+0000 7fcaf65d8700 1 -- start start 2023-10-16T11:22:28.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaf00cee70 0x7fcaf01f1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf006b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaf015a730 0x7fcaf014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf01586f0 con 0x7fcaf015a350 2023-10-16T11:22:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf006b7a0 con 0x7fcaf00cee70 2023-10-16T11:22:28.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0158920 con 0x7fcaf015a730 2023-10-16T11:22:28.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf006b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf006b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45060/0 (socket says 172.21.15.158:45060) 2023-10-16T11:22:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 -- 172.21.15.158:0/2949615681 learned_addr learned my addr 172.21.15.158:0/2949615681 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:28.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaeffff700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaf00cee70 0x7fcaf01f1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf4b75700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaf015a730 0x7fcaf014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 -- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaf015a730 msgr2=0x7fcaf014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaf015a730 0x7fcaf014fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 -- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaf00cee70 msgr2=0x7fcaf01f1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaf00cee70 0x7fcaf01f1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 -- 172.21.15.158:0/2949615681 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae40094a0 con 0x7fcaf015a350 2023-10-16T11:22:28.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaef7fe700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf006b260 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7fcae000c450 tx=0x7fcae000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaecff9700 1 -- 172.21.15.158:0/2949615681 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00052d0 con 0x7fcaf015a350 2023-10-16T11:22:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaecff9700 1 -- 172.21.15.158:0/2949615681 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae0005430 con 0x7fcaf015a350 2023-10-16T11:22:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaecff9700 1 -- 172.21.15.158:0/2949615681 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00038c0 con 0x7fcaf015a350 2023-10-16T11:22:28.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf0156c20 con 0x7fcaf015a350 2023-10-16T11:22:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.485+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf0158bc0 con 0x7fcaf015a350 2023-10-16T11:22:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.486+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf0158de0 con 0x7fcaf015a350 2023-10-16T11:22:28.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.487+0000 7fcaecff9700 1 -- 172.21.15.158:0/2949615681 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcae0003a20 con 0x7fcaf015a350 2023-10-16T11:22:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.487+0000 7fcaecff9700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fcadc06b020 0x7fcadc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.487+0000 7fcaecff9700 1 -- 172.21.15.158:0/2949615681 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcae0088d90 con 0x7fcaf015a350 2023-10-16T11:22:28.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.487+0000 7fcaeffff700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fcadc06b020 0x7fcadc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.488+0000 7fcaeffff700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fcadc06b020 0x7fcadc06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcae40090b0 tx=0x7fcae401c040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.491+0000 7fcaecff9700 1 -- 172.21.15.158:0/2949615681 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcae00539e0 con 0x7fcaf015a350 2023-10-16T11:22:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.686+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcaf015eb10 con 0x7fcadc06b020 2023-10-16T11:22:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.686+0000 7fcaecff9700 1 -- 172.21.15.158:0/2949615681 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcae4011c00 con 0x7fcadc06b020 2023-10-16T11:22:28.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fcadc06b020 msgr2=0x7fcadc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fcadc06b020 0x7fcadc06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcae40090b0 tx=0x7fcae401c040 comp rx=0 tx=0).stop 2023-10-16T11:22:28.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 msgr2=0x7fcaf006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf006b260 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7fcae000c450 tx=0x7fcae000c760 comp rx=0 tx=0).stop 2023-10-16T11:22:28.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 shutdown_connections 2023-10-16T11:22:28.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fcadc06b020 0x7fcadc06d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaf015a730 0x7fcaf014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaf00cee70 0x7fcaf01f1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 --2- 172.21.15.158:0/2949615681 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcaf015a350 0x7fcaf006b260 unknown :-1 s=CLOSED pgs=3429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 >> 172.21.15.158:0/2949615681 conn(0x7fcaf00b1df0 msgr2=0x7fcaf00afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 shutdown_connections 2023-10-16T11:22:28.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.688+0000 7fcaf65d8700 1 -- 172.21.15.158:0/2949615681 wait complete. 2023-10-16T11:22:28.694 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 2 to deep-scrub 2023-10-16T11:22:28.702 INFO:tasks.ceph:Scrubbing osd.3 2023-10-16T11:22:28.703 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:28.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.825+0000 7fee44996700 1 Processor -- start 2023-10-16T11:22:28.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.825+0000 7fee44996700 1 -- start start 2023-10-16T11:22:28.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.826+0000 7fee44996700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee40151220 0x7fee400665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.826+0000 7fee44996700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee400cf930 con 0x7fee40151600 2023-10-16T11:22:28.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.826+0000 7fee44996700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee400cf650 con 0x7fee400cf190 2023-10-16T11:22:28.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.826+0000 7fee44996700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee400c51b0 con 0x7fee40151220 2023-10-16T11:22:28.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.826+0000 7fee3ed9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fee40151600 0x7fee400c5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45248/0 (socket says 172.21.15.158:45248) 2023-10-16T11:22:28.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.826+0000 7fee3ed9d700 1 -- 172.21.15.158:0/2306442764 learned_addr learned my addr 172.21.15.158:0/2306442764 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:28.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3e59c700 1 --2- 172.21.15.158:0/2306442764 >> v2:172.21.15.172:3300/0 conn(0x7fee40151220 0x7fee400665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3d59a700 1 -- 172.21.15.158:0/2306442764 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764011764 0 0) 0x7fee400cf930 con 0x7fee40151600 2023-10-16T11:22:28.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3d59a700 1 -- 172.21.15.158:0/2306442764 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee280035b0 con 0x7fee40151600 2023-10-16T11:22:28.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3e59c700 1 -- 172.21.15.158:0/2306442764 >> v1:172.21.15.172:6789/0 conn(0x7fee400cf190 legacy=0x7fee4006d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3e59c700 1 -- 172.21.15.158:0/2306442764 >> v1:172.21.15.158:6789/0 conn(0x7fee40151600 legacy=0x7fee400c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3e59c700 1 -- 172.21.15.158:0/2306442764 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee400c5490 con 0x7fee40151220 2023-10-16T11:22:28.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3e59c700 1 --2- 172.21.15.158:0/2306442764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151220 0x7fee400665a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fee34009870 tx=0x7fee34009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f5ff99dee7ac5fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3d59a700 1 -- 172.21.15.158:0/2306442764 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee34011830 con 0x7fee40151220 2023-10-16T11:22:28.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3d59a700 1 -- 172.21.15.158:0/2306442764 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee34011990 con 0x7fee40151220 2023-10-16T11:22:28.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.827+0000 7fee3d59a700 1 -- 172.21.15.158:0/2306442764 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee34011d20 con 0x7fee40151220 2023-10-16T11:22:28.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.828+0000 7fee44996700 1 -- 172.21.15.158:0/2306442764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151220 msgr2=0x7fee400665a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.828+0000 7fee44996700 1 --2- 172.21.15.158:0/2306442764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151220 0x7fee400665a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fee34009870 tx=0x7fee34009b80 comp rx=0 tx=0).stop 2023-10-16T11:22:28.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.828+0000 7fee44996700 1 -- 172.21.15.158:0/2306442764 shutdown_connections 2023-10-16T11:22:28.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.828+0000 7fee44996700 1 --2- 172.21.15.158:0/2306442764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151220 0x7fee400665a0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.828+0000 7fee44996700 1 -- 172.21.15.158:0/2306442764 >> 172.21.15.158:0/2306442764 conn(0x7fee400cc5b0 msgr2=0x7fee400cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.828+0000 7fee44996700 1 -- 172.21.15.158:0/2306442764 shutdown_connections 2023-10-16T11:22:28.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.828+0000 7fee44996700 1 -- 172.21.15.158:0/2306442764 wait complete. 2023-10-16T11:22:28.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.829+0000 7fee44996700 1 Processor -- start 2023-10-16T11:22:28.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.829+0000 7fee44996700 1 -- start start 2023-10-16T11:22:28.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee44996700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee400cf190 0x7fee401484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee44996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee40151220 0x7fee400b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee44996700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151600 0x7fee4006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee44996700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400b3ce0 con 0x7fee40151220 2023-10-16T11:22:28.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee44996700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400c6d30 con 0x7fee40151600 2023-10-16T11:22:28.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee44996700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee400c7010 con 0x7fee400cf190 2023-10-16T11:22:28.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee40151220 0x7fee400b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee40151220 0x7fee400b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45076/0 (socket says 172.21.15.158:45076) 2023-10-16T11:22:28.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 -- 172.21.15.158:0/511419961 learned_addr learned my addr 172.21.15.158:0/511419961 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:28.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3ed9d700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151600 0x7fee4006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3e59c700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee400cf190 0x7fee401484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 -- 172.21.15.158:0/511419961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee400cf190 msgr2=0x7fee401484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fee400cf190 0x7fee401484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 -- 172.21.15.158:0/511419961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151600 msgr2=0x7fee4006e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee40151600 0x7fee4006e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.830+0000 7fee3dd9b700 1 -- 172.21.15.158:0/511419961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee340094a0 con 0x7fee40151220 2023-10-16T11:22:28.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.831+0000 7fee3dd9b700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee40151220 0x7fee400b37a0 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7fee3000c4b0 tx=0x7fee3000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.831+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300052d0 con 0x7fee40151220 2023-10-16T11:22:28.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.831+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee30005430 con 0x7fee40151220 2023-10-16T11:22:28.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.831+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300038c0 con 0x7fee40151220 2023-10-16T11:22:28.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.831+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee400c72f0 con 0x7fee40151220 2023-10-16T11:22:28.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.831+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4006e6a0 con 0x7fee40151220 2023-10-16T11:22:28.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.832+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee40066ae0 con 0x7fee40151220 2023-10-16T11:22:28.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.832+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee30003a20 con 0x7fee40151220 2023-10-16T11:22:28.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee2effd700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fee2006b020 0x7fee2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee3e59c700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fee2006b020 0x7fee2006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fee30088d20 con 0x7fee40151220 2023-10-16T11:22:28.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee2effd700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fee20071e10 0x7fee20074280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 --> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee200748b0 con 0x7fee20071e10 2023-10-16T11:22:28.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee3008b3e0 con 0x7fee40151220 2023-10-16T11:22:28.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee3ed9d700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fee20071e10 0x7fee20074280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee3e59c700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fee2006b020 0x7fee2006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fee34009840 tx=0x7fee34008030 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.833+0000 7fee3ed9d700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fee20071e10 0x7fee20074280 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.834+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== osd.3 v2:172.21.15.158:6824/969498875 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fee200748b0 con 0x7fee20071e10 2023-10-16T11:22:28.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.848+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 --> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fee40148d80 con 0x7fee20071e10 2023-10-16T11:22:28.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee2effd700 1 -- 172.21.15.158:0/511419961 <== osd.3 v2:172.21.15.158:6824/969498875 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fee40148d80 con 0x7fee20071e10 2023-10-16T11:22:28.858 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:28.858 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:28.859 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:28.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fee20071e10 msgr2=0x7fee20074280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6824/969498875,v1:172.21.15.158:6825/969498875] conn(0x7fee20071e10 0x7fee20074280 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fee2006b020 msgr2=0x7fee2006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fee2006b020 0x7fee2006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fee34009840 tx=0x7fee34008030 comp rx=0 tx=0).stop 2023-10-16T11:22:28.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee40151220 msgr2=0x7fee400b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 --2- 172.21.15.158:0/511419961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee40151220 0x7fee400b37a0 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7fee3000c4b0 tx=0x7fee3000c7c0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee3ed9d700 1 -- 172.21.15.158:0/511419961 reap_dead start 2023-10-16T11:22:28.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 shutdown_connections 2023-10-16T11:22:28.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 >> 172.21.15.158:0/511419961 conn(0x7fee400cc5b0 msgr2=0x7fee400cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 shutdown_connections 2023-10-16T11:22:28.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.849+0000 7fee44996700 1 -- 172.21.15.158:0/511419961 wait complete. 2023-10-16T11:22:28.865 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-16T11:22:28.987 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.986+0000 7f5fa019b700 1 Processor -- start 2023-10-16T11:22:28.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.986+0000 7f5fa019b700 1 -- start start 2023-10-16T11:22:28.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5fa019b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5f980cd360 0x7f5f9806ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:28.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5fa019b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9815ec70 con 0x7f5f980cfe30 2023-10-16T11:22:28.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5fa019b700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9815f1b0 con 0x7f5f9815a6d0 2023-10-16T11:22:28.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5fa019b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f980afcd0 con 0x7f5f980cd360 2023-10-16T11:22:28.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9e738700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5f980cd360 0x7f5f9806ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:28.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9d736700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5f980cfe30 0x7f5f980ccc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45262/0 (socket says 172.21.15.158:45262) 2023-10-16T11:22:28.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9d736700 1 -- 172.21.15.158:0/1779813489 learned_addr learned my addr 172.21.15.158:0/1779813489 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:28.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9e738700 1 -- 172.21.15.158:0/1779813489 >> v1:172.21.15.172:6789/0 conn(0x7f5f9815a6d0 legacy=0x7f5f980ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9e738700 1 -- 172.21.15.158:0/1779813489 >> v1:172.21.15.158:6789/0 conn(0x7f5f980cfe30 legacy=0x7f5f980ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9cf35700 1 -- 172.21.15.158:0/1779813489 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107019750 0 0) 0x7f5f9815f1b0 con 0x7f5f9815a6d0 2023-10-16T11:22:28.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9e738700 1 -- 172.21.15.158:0/1779813489 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980cab50 con 0x7f5f980cd360 2023-10-16T11:22:28.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9e738700 1 --2- 172.21.15.158:0/1779813489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980cd360 0x7f5f9806ebb0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009d90 tx=0x7f5f9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e13b493c71b55e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:28.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9cf35700 1 -- 172.21.15.158:0/1779813489 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000e070 con 0x7f5f980cd360 2023-10-16T11:22:28.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9cf35700 1 -- 172.21.15.158:0/1779813489 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9000bad0 con 0x7f5f980cd360 2023-10-16T11:22:28.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.987+0000 7f5f9cf35700 1 -- 172.21.15.158:0/1779813489 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000be80 con 0x7f5f980cd360 2023-10-16T11:22:28.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.988+0000 7f5fa019b700 1 -- 172.21.15.158:0/1779813489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980cd360 msgr2=0x7f5f9806ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:28.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.988+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1779813489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980cd360 0x7f5f9806ebb0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009d90 tx=0x7f5f9000b330 comp rx=0 tx=0).stop 2023-10-16T11:22:28.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.988+0000 7f5fa019b700 1 -- 172.21.15.158:0/1779813489 shutdown_connections 2023-10-16T11:22:28.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.988+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1779813489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980cd360 0x7f5f9806ebb0 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:28.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.988+0000 7f5fa019b700 1 -- 172.21.15.158:0/1779813489 >> 172.21.15.158:0/1779813489 conn(0x7f5f9806b9e0 msgr2=0x7f5f9806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:28.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.988+0000 7f5fa019b700 1 -- 172.21.15.158:0/1779813489 shutdown_connections 2023-10-16T11:22:28.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.988+0000 7f5fa019b700 1 -- 172.21.15.158:0/1779813489 wait complete. 2023-10-16T11:22:29.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.989+0000 7f5fa019b700 1 Processor -- start 2023-10-16T11:22:29.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.989+0000 7f5fa019b700 1 -- start start 2023-10-16T11:22:29.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5fa019b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f980cd360 0x7f5f980b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5fa019b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f980cfe30 0x7f5f9814e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5fa019b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f9815a6d0 0x7f5f9814eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5fa019b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980b2d80 con 0x7f5f9815a6d0 2023-10-16T11:22:29.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5fa019b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f981f4430 con 0x7f5f980cd360 2023-10-16T11:22:29.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5fa019b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f981f3860 con 0x7f5f980cfe30 2023-10-16T11:22:29.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f9815a6d0 0x7f5f9814eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f9815a6d0 0x7f5f9814eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45094/0 (socket says 172.21.15.158:45094) 2023-10-16T11:22:29.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 -- 172.21.15.158:0/1054957104 learned_addr learned my addr 172.21.15.158:0/1054957104 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:29.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9df37700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f980cd360 0x7f5f980b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9d736700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f980cfe30 0x7f5f9814e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 -- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f980cfe30 msgr2=0x7f5f9814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f980cfe30 0x7f5f9814e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 -- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f980cd360 msgr2=0x7f5f980b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f980cd360 0x7f5f980b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.990+0000 7f5f9e738700 1 -- 172.21.15.158:0/1054957104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90009a70 con 0x7f5f9815a6d0 2023-10-16T11:22:29.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.991+0000 7f5f9e738700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f9815a6d0 0x7f5f9814eb00 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f5f90004000 tx=0x7f5f90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.991+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/1054957104 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90012630 con 0x7f5f9815a6d0 2023-10-16T11:22:29.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.991+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/1054957104 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90012790 con 0x7f5f9815a6d0 2023-10-16T11:22:29.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.991+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/1054957104 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90012b40 con 0x7f5f9815a6d0 2023-10-16T11:22:29.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.991+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f981f3b00 con 0x7f5f9815a6d0 2023-10-16T11:22:29.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.991+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f981f40c0 con 0x7f5f9815a6d0 2023-10-16T11:22:29.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.992+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f981f45e0 con 0x7f5f9815a6d0 2023-10-16T11:22:29.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.992+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/1054957104 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f90012ca0 con 0x7f5f9815a6d0 2023-10-16T11:22:29.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.993+0000 7f5f8e7fc700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.993+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/1054957104 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5f9008bc90 con 0x7f5f9815a6d0 2023-10-16T11:22:29.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.993+0000 7f5f9df37700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.993+0000 7f5f9df37700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000f60 tx=0x7f5f94009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:28.997+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/1054957104 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f981f45e0 con 0x7f5f9815a6d0 2023-10-16T11:22:29.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.190+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f980cadf0 con 0x7f5f7c06b020 2023-10-16T11:22:29.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.190+0000 7f5f8e7fc700 1 -- 172.21.15.158:0/1054957104 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5f94001500 con 0x7f5f7c06b020 2023-10-16T11:22:29.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5f7c06b020 msgr2=0x7f5f7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000f60 tx=0x7f5f94009040 comp rx=0 tx=0).stop 2023-10-16T11:22:29.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f9815a6d0 msgr2=0x7f5f9814eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f9815a6d0 0x7f5f9814eb00 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f5f90004000 tx=0x7f5f90004ad0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 shutdown_connections 2023-10-16T11:22:29.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5f7c06b020 0x7f5f7c06d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f9815a6d0 0x7f5f9814eb00 unknown :-1 s=CLOSED pgs=3434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f980cd360 0x7f5f980b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 --2- 172.21.15.158:0/1054957104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5f980cfe30 0x7f5f9814e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 >> 172.21.15.158:0/1054957104 conn(0x7f5f9806b9e0 msgr2=0x7f5f980cc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:29.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.192+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 shutdown_connections 2023-10-16T11:22:29.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.193+0000 7f5fa019b700 1 -- 172.21.15.158:0/1054957104 wait complete. 2023-10-16T11:22:29.198 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 3 to deep-scrub 2023-10-16T11:22:29.206 INFO:tasks.ceph:Scrubbing osd.4 2023-10-16T11:22:29.207 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:29.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.330+0000 7feb920a9700 1 Processor -- start 2023-10-16T11:22:29.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.330+0000 7feb920a9700 1 -- start start 2023-10-16T11:22:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb920a9700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb8c15b630 0x7feb8c0cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb920a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c0b19f0 0x7feb8c0ad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb920a9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0cc4d0 con 0x7feb8c0cdab0 2023-10-16T11:22:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb920a9700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c0cc7b0 con 0x7feb8c15b630 2023-10-16T11:22:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb920a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0cca90 con 0x7feb8c0b19f0 2023-10-16T11:22:29.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb8c15b630 0x7feb8c0cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8b7fe700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb8c15b630 0x7feb8c0cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45098/0 (socket says 172.21.15.158:45098) 2023-10-16T11:22:29.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8b7fe700 1 -- 172.21.15.158:0/1364923527 learned_addr learned my addr 172.21.15.158:0/1364923527 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8bfff700 1 --2- 172.21.15.158:0/1364923527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c0b19f0 0x7feb8c0ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8b7fe700 1 -- 172.21.15.158:0/1364923527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c0b19f0 msgr2=0x7feb8c0ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8b7fe700 1 --2- 172.21.15.158:0/1364923527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c0b19f0 0x7feb8c0ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8b7fe700 1 -- 172.21.15.158:0/1364923527 >> v1:172.21.15.172:6789/0 conn(0x7feb8c0cdab0 legacy=0x7feb8c0b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.331+0000 7feb8b7fe700 1 -- 172.21.15.158:0/1364923527 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0ccd70 con 0x7feb8c15b630 2023-10-16T11:22:29.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb8b7fe700 1 --2- 172.21.15.158:0/1364923527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c15b630 0x7feb8c0cd570 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7feb800098f0 tx=0x7feb80009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2bd4129e3279ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb8a7fc700 1 -- 172.21.15.158:0/1364923527 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000b8a0 con 0x7feb8c15b630 2023-10-16T11:22:29.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb8a7fc700 1 -- 172.21.15.158:0/1364923527 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8000ba00 con 0x7feb8c15b630 2023-10-16T11:22:29.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb8a7fc700 1 -- 172.21.15.158:0/1364923527 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000bd90 con 0x7feb8c15b630 2023-10-16T11:22:29.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 -- 172.21.15.158:0/1364923527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c15b630 msgr2=0x7feb8c0cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 --2- 172.21.15.158:0/1364923527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c15b630 0x7feb8c0cd570 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7feb800098f0 tx=0x7feb80009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:29.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 -- 172.21.15.158:0/1364923527 shutdown_connections 2023-10-16T11:22:29.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 --2- 172.21.15.158:0/1364923527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c0b19f0 0x7feb8c0ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 --2- 172.21.15.158:0/1364923527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c15b630 0x7feb8c0cd570 unknown :-1 s=CLOSED pgs=3435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 -- 172.21.15.158:0/1364923527 >> 172.21.15.158:0/1364923527 conn(0x7feb8c099a10 msgr2=0x7feb8c0d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 -- 172.21.15.158:0/1364923527 shutdown_connections 2023-10-16T11:22:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.332+0000 7feb920a9700 1 -- 172.21.15.158:0/1364923527 wait complete. 2023-10-16T11:22:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.333+0000 7feb920a9700 1 Processor -- start 2023-10-16T11:22:29.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.334+0000 7feb920a9700 1 -- start start 2023-10-16T11:22:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.334+0000 7feb920a9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0b19f0 0x7feb8c14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.334+0000 7feb920a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0cdab0 0x7feb8c0b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.334+0000 7feb920a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c15b630 0x7feb8c1f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.334+0000 7feb920a9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0c40e0 con 0x7feb8c0cdab0 2023-10-16T11:22:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb920a9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0b39f0 con 0x7feb8c0b19f0 2023-10-16T11:22:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb920a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0b3cd0 con 0x7feb8c15b630 2023-10-16T11:22:29.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0cdab0 0x7feb8c0b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0cdab0 0x7feb8c0b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45104/0 (socket says 172.21.15.158:45104) 2023-10-16T11:22:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 -- 172.21.15.158:0/217279990 learned_addr learned my addr 172.21.15.158:0/217279990 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:29.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8b7fe700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0b19f0 0x7feb8c14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8bfff700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c15b630 0x7feb8c1f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 -- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c15b630 msgr2=0x7feb8c1f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb8c15b630 0x7feb8c1f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 -- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0b19f0 msgr2=0x7feb8c14ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb8c0b19f0 0x7feb8c14ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 -- 172.21.15.158:0/217279990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80009550 con 0x7feb8c0cdab0 2023-10-16T11:22:29.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb8affd700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0cdab0 0x7feb8c0b3050 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c4e0 tx=0x7feb7c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c0052d0 con 0x7feb8c0cdab0 2023-10-16T11:22:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c005430 con 0x7feb8c0cdab0 2023-10-16T11:22:29.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c15bb90 con 0x7feb8c0cdab0 2023-10-16T11:22:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c0038c0 con 0x7feb8c0cdab0 2023-10-16T11:22:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.335+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c150520 con 0x7feb8c0cdab0 2023-10-16T11:22:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.336+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb8c0c3f60 con 0x7feb8c0cdab0 2023-10-16T11:22:29.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb7c003a20 con 0x7feb8c0cdab0 2023-10-16T11:22:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb73fff700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7feb7806b020 0x7feb7806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7feb7c089470 con 0x7feb8c0cdab0 2023-10-16T11:22:29.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb8b7fe700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7feb7806b020 0x7feb7806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb73fff700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7feb78071e10 0x7feb78074280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 --> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb780748b0 con 0x7feb78071e10 2023-10-16T11:22:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb7c0898c0 con 0x7feb8c0cdab0 2023-10-16T11:22:29.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb8bfff700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7feb78071e10 0x7feb78074280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.337+0000 7feb8b7fe700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7feb7806b020 0x7feb7806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb80014040 tx=0x7feb80008030 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.338+0000 7feb8bfff700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7feb78071e10 0x7feb78074280 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.338+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== osd.4 v2:172.21.15.172:6824/3174967294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7feb780748b0 con 0x7feb78071e10 2023-10-16T11:22:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 --> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feb8c15ba10 con 0x7feb78071e10 2023-10-16T11:22:29.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb73fff700 1 -- 172.21.15.158:0/217279990 <== osd.4 v2:172.21.15.172:6824/3174967294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7feb840083a0 con 0x7feb78071e10 2023-10-16T11:22:29.358 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:29.358 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:29.359 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:29.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7feb78071e10 msgr2=0x7feb78074280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.172:6824/3174967294,v1:172.21.15.172:6825/3174967294] conn(0x7feb78071e10 0x7feb78074280 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7feb7806b020 msgr2=0x7feb7806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7feb7806b020 0x7feb7806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb80014040 tx=0x7feb80008030 comp rx=0 tx=0).stop 2023-10-16T11:22:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0cdab0 msgr2=0x7feb8c0b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 --2- 172.21.15.158:0/217279990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb8c0cdab0 0x7feb8c0b3050 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c4e0 tx=0x7feb7c00c7f0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb8bfff700 1 -- 172.21.15.158:0/217279990 reap_dead start 2023-10-16T11:22:29.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 shutdown_connections 2023-10-16T11:22:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 >> 172.21.15.158:0/217279990 conn(0x7feb8c099a10 msgr2=0x7feb8c158c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 shutdown_connections 2023-10-16T11:22:29.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.353+0000 7feb920a9700 1 -- 172.21.15.158:0/217279990 wait complete. 2023-10-16T11:22:29.367 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-16T11:22:29.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.486+0000 7ff23130a700 1 Processor -- start 2023-10-16T11:22:29.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.486+0000 7ff23130a700 1 -- start start 2023-10-16T11:22:29.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff23130a700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff22c15d9a0 0x7ff22c0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff23130a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c06d4d0 0x7ff22c0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff23130a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22c0cfc90 con 0x7ff22c15d5c0 2023-10-16T11:22:29.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff23130a700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22c161620 con 0x7ff22c15d9a0 2023-10-16T11:22:29.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff23130a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c161900 con 0x7ff22c06d4d0 2023-10-16T11:22:29.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22a7fc700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff22c15d9a0 0x7ff22c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22affd700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff22c15d5c0 0x7ff22c06bdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45288/0 (socket says 172.21.15.158:45288) 2023-10-16T11:22:29.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22affd700 1 -- 172.21.15.158:0/391085618 learned_addr learned my addr 172.21.15.158:0/391085618 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:29.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22b7fe700 1 --2- 172.21.15.158:0/391085618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c06d4d0 0x7ff22c0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22a7fc700 1 -- 172.21.15.158:0/391085618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c06d4d0 msgr2=0x7ff22c0b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:29.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22a7fc700 1 --2- 172.21.15.158:0/391085618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c06d4d0 0x7ff22c0b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22a7fc700 1 -- 172.21.15.158:0/391085618 >> v1:172.21.15.158:6789/0 conn(0x7ff22c15d5c0 legacy=0x7ff22c06bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff229ffb700 1 -- 172.21.15.158:0/391085618 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376971809 0 0) 0x7ff22c0cfc90 con 0x7ff22c15d5c0 2023-10-16T11:22:29.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22a7fc700 1 -- 172.21.15.158:0/391085618 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c161be0 con 0x7ff22c15d9a0 2023-10-16T11:22:29.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff22a7fc700 1 --2- 172.21.15.158:0/391085618 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c15d9a0 0x7ff22c0b2f30 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009840 tx=0x7ff21c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84b50987007fbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff229ffb700 1 -- 172.21.15.158:0/391085618 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c011830 con 0x7ff22c15d9a0 2023-10-16T11:22:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff229ffb700 1 -- 172.21.15.158:0/391085618 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c011990 con 0x7ff22c15d9a0 2023-10-16T11:22:29.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.487+0000 7ff229ffb700 1 -- 172.21.15.158:0/391085618 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c011d20 con 0x7ff22c15d9a0 2023-10-16T11:22:29.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 -- 172.21.15.158:0/391085618 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c15d9a0 msgr2=0x7ff22c0b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 --2- 172.21.15.158:0/391085618 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c15d9a0 0x7ff22c0b2f30 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009840 tx=0x7ff21c009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:29.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 -- 172.21.15.158:0/391085618 shutdown_connections 2023-10-16T11:22:29.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 --2- 172.21.15.158:0/391085618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c06d4d0 0x7ff22c0b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 --2- 172.21.15.158:0/391085618 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c15d9a0 0x7ff22c0b2f30 unknown :-1 s=CLOSED pgs=3438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 -- 172.21.15.158:0/391085618 >> 172.21.15.158:0/391085618 conn(0x7ff22c0cdf30 msgr2=0x7ff22c0ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:29.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 -- 172.21.15.158:0/391085618 shutdown_connections 2023-10-16T11:22:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.488+0000 7ff23130a700 1 -- 172.21.15.158:0/391085618 wait complete. 2023-10-16T11:22:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.489+0000 7ff23130a700 1 Processor -- start 2023-10-16T11:22:29.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.489+0000 7ff23130a700 1 -- start start 2023-10-16T11:22:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff23130a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c06d4d0 0x7ff22c159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff23130a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff22c15d5c0 0x7ff22c0b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff23130a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c15d9a0 0x7ff22c0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff23130a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0ca180 con 0x7ff22c06d4d0 2023-10-16T11:22:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff23130a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c15a2a0 con 0x7ff22c15d5c0 2023-10-16T11:22:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff23130a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0c97e0 con 0x7ff22c15d9a0 2023-10-16T11:22:29.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff22affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c06d4d0 0x7ff22c159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff22affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c06d4d0 0x7ff22c159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45118/0 (socket says 172.21.15.158:45118) 2023-10-16T11:22:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff22affd700 1 -- 172.21.15.158:0/1324368369 learned_addr learned my addr 172.21.15.158:0/1324368369 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff22b7fe700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c15d9a0 0x7ff22c0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.490+0000 7ff22a7fc700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff22c15d5c0 0x7ff22c0b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff22affd700 1 -- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c15d9a0 msgr2=0x7ff22c0b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff22affd700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c15d9a0 0x7ff22c0b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff22affd700 1 -- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff22c15d5c0 msgr2=0x7ff22c0b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff22affd700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff22c15d5c0 0x7ff22c0b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff22affd700 1 -- 172.21.15.158:0/1324368369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c0094a0 con 0x7ff22c06d4d0 2023-10-16T11:22:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff22a7fc700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff22c15d5c0 0x7ff22c0b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:29.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff22affd700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c06d4d0 0x7ff22c159d60 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7ff2200011e0 tx=0x7ff22000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff2137fe700 1 -- 172.21.15.158:0/1324368369 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220004c80 con 0x7ff22c06d4d0 2023-10-16T11:22:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff2137fe700 1 -- 172.21.15.158:0/1324368369 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff220004de0 con 0x7ff22c06d4d0 2023-10-16T11:22:29.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22c0c9ac0 con 0x7ff22c06d4d0 2023-10-16T11:22:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff2137fe700 1 -- 172.21.15.158:0/1324368369 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220007310 con 0x7ff22c06d4d0 2023-10-16T11:22:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.491+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22c0b4f80 con 0x7ff22c06d4d0 2023-10-16T11:22:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.492+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22c0cdd30 con 0x7ff22c06d4d0 2023-10-16T11:22:29.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.492+0000 7ff2137fe700 1 -- 172.21.15.158:0/1324368369 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff220007470 con 0x7ff22c06d4d0 2023-10-16T11:22:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.493+0000 7ff2137fe700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff21806b020 0x7ff21806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.493+0000 7ff2137fe700 1 -- 172.21.15.158:0/1324368369 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff220089710 con 0x7ff22c06d4d0 2023-10-16T11:22:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.493+0000 7ff22a7fc700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff21806b020 0x7ff21806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.493+0000 7ff22a7fc700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff21806b020 0x7ff21806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0c4d80 tx=0x7ff21c008030 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.497+0000 7ff2137fe700 1 -- 172.21.15.158:0/1324368369 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff220054470 con 0x7ff22c06d4d0 2023-10-16T11:22:29.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.692+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff22c0c9950 con 0x7ff21806b020 2023-10-16T11:22:29.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.692+0000 7ff2137fe700 1 -- 172.21.15.158:0/1324368369 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff22c0c9950 con 0x7ff21806b020 2023-10-16T11:22:29.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff21806b020 msgr2=0x7ff21806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff21806b020 0x7ff21806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0c4d80 tx=0x7ff21c008030 comp rx=0 tx=0).stop 2023-10-16T11:22:29.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c06d4d0 msgr2=0x7ff22c159d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c06d4d0 0x7ff22c159d60 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7ff2200011e0 tx=0x7ff22000ca30 comp rx=0 tx=0).stop 2023-10-16T11:22:29.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 shutdown_connections 2023-10-16T11:22:29.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7ff21806b020 0x7ff21806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff22c15d9a0 0x7ff22c0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff22c06d4d0 0x7ff22c159d60 unknown :-1 s=CLOSED pgs=3439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 --2- 172.21.15.158:0/1324368369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff22c15d5c0 0x7ff22c0b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 >> 172.21.15.158:0/1324368369 conn(0x7ff22c0cdf30 msgr2=0x7ff22c0cd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:29.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 shutdown_connections 2023-10-16T11:22:29.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.694+0000 7ff23130a700 1 -- 172.21.15.158:0/1324368369 wait complete. 2023-10-16T11:22:29.701 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 4 to deep-scrub 2023-10-16T11:22:29.708 INFO:tasks.ceph:Scrubbing osd.5 2023-10-16T11:22:29.709 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:29.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.832+0000 7f22b0189700 1 Processor -- start 2023-10-16T11:22:29.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.832+0000 7f22b0189700 1 -- start start 2023-10-16T11:22:29.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22b0189700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22a815da40 0x7f22a80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22b0189700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f22a806afe0 0x7f22a80afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22b0189700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80cbd30 con 0x7f22a80cc590 2023-10-16T11:22:29.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22b0189700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a80cc1c0 con 0x7f22a815da40 2023-10-16T11:22:29.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22b0189700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a815de20 con 0x7f22a806afe0 2023-10-16T11:22:29.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22adf25700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22a815da40 0x7f22a80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22ad724700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f22a80cc590 0x7f22a806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45304/0 (socket says 172.21.15.158:45304) 2023-10-16T11:22:29.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.833+0000 7f22ad724700 1 -- 172.21.15.158:0/2942029572 learned_addr learned my addr 172.21.15.158:0/2942029572 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:29.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22ae726700 1 --2- 172.21.15.158:0/2942029572 >> v2:172.21.15.172:3300/0 conn(0x7f22a806afe0 0x7f22a80afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22acf23700 1 -- 172.21.15.158:0/2942029572 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004362020 0 0) 0x7f22a80cbd30 con 0x7f22a80cc590 2023-10-16T11:22:29.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22acf23700 1 -- 172.21.15.158:0/2942029572 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22940035b0 con 0x7f22a80cc590 2023-10-16T11:22:29.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22adf25700 1 -- 172.21.15.158:0/2942029572 >> v2:172.21.15.172:3300/0 conn(0x7f22a806afe0 msgr2=0x7f22a80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:29.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22adf25700 1 --2- 172.21.15.158:0/2942029572 >> v2:172.21.15.172:3300/0 conn(0x7f22a806afe0 0x7f22a80afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22adf25700 1 -- 172.21.15.158:0/2942029572 >> v1:172.21.15.158:6789/0 conn(0x7f22a80cc590 legacy=0x7f22a806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22adf25700 1 -- 172.21.15.158:0/2942029572 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80ce260 con 0x7f22a815da40 2023-10-16T11:22:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22adf25700 1 --2- 172.21.15.158:0/2942029572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a80caa20 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f22a80729e0 tx=0x7f2298009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f4fdd6d6b6cdbc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22acf23700 1 -- 172.21.15.158:0/2942029572 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011830 con 0x7f22a815da40 2023-10-16T11:22:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22acf23700 1 -- 172.21.15.158:0/2942029572 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2298011990 con 0x7f22a815da40 2023-10-16T11:22:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.834+0000 7f22acf23700 1 -- 172.21.15.158:0/2942029572 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011d20 con 0x7f22a815da40 2023-10-16T11:22:29.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 -- 172.21.15.158:0/2942029572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 msgr2=0x7f22a80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 --2- 172.21.15.158:0/2942029572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a80caa20 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f22a80729e0 tx=0x7f2298009b20 comp rx=0 tx=0).stop 2023-10-16T11:22:29.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 -- 172.21.15.158:0/2942029572 shutdown_connections 2023-10-16T11:22:29.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 --2- 172.21.15.158:0/2942029572 >> v2:172.21.15.172:3300/0 conn(0x7f22a806afe0 0x7f22a80afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 --2- 172.21.15.158:0/2942029572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a80caa20 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 -- 172.21.15.158:0/2942029572 >> 172.21.15.158:0/2942029572 conn(0x7f22a8099a10 msgr2=0x7f22a80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:29.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 -- 172.21.15.158:0/2942029572 shutdown_connections 2023-10-16T11:22:29.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.835+0000 7f22b0189700 1 -- 172.21.15.158:0/2942029572 wait complete. 2023-10-16T11:22:29.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.836+0000 7f22b0189700 1 Processor -- start 2023-10-16T11:22:29.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.836+0000 7f22b0189700 1 -- start start 2023-10-16T11:22:29.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22a806afe0 0x7f22a81f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22a80cc590 0x7f22a80c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a81f2e10 con 0x7f22a815da40 2023-10-16T11:22:29.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a81f68d0 con 0x7f22a80cc590 2023-10-16T11:22:29.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a81f6bb0 con 0x7f22a806afe0 2023-10-16T11:22:29.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a815a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45136/0 (socket says 172.21.15.158:45136) 2023-10-16T11:22:29.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 -- 172.21.15.158:0/840868151 learned_addr learned my addr 172.21.15.158:0/840868151 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:29.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ad724700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22a80cc590 0x7f22a80c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22adf25700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22a806afe0 0x7f22a81f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 -- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22a806afe0 msgr2=0x7f22a81f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22a806afe0 0x7f22a81f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 -- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22a80cc590 msgr2=0x7f22a80c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22a80cc590 0x7f22a80c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 -- 172.21.15.158:0/840868151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980094a0 con 0x7f22a815da40 2023-10-16T11:22:29.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22ae726700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a815a4c0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7f22a000efc0 tx=0x7f22a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a000cc90 con 0x7f22a815da40 2023-10-16T11:22:29.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a000f040 con 0x7f22a815da40 2023-10-16T11:22:29.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a81f6e90 con 0x7f22a815da40 2023-10-16T11:22:29.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00146c0 con 0x7f22a815da40 2023-10-16T11:22:29.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.837+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a81f7170 con 0x7f22a815da40 2023-10-16T11:22:29.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.838+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22a80cb400 con 0x7f22a815da40 2023-10-16T11:22:29.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.839+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f22a0014820 con 0x7f22a815da40 2023-10-16T11:22:29.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.839+0000 7f229e7fc700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f228c06b020 0x7f228c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.839+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f22a008b0f0 con 0x7f22a815da40 2023-10-16T11:22:29.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.839+0000 7f22adf25700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f228c06b020 0x7f228c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.839+0000 7f229e7fc700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f228c071e10 0x7f228c074280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.839+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 --> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f228c0748b0 con 0x7f228c071e10 2023-10-16T11:22:29.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.839+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22a008b460 con 0x7f22a815da40 2023-10-16T11:22:29.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.840+0000 7f22ad724700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f228c071e10 0x7f228c074280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.840+0000 7f22adf25700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f228c06b020 0x7f228c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f22980090b0 tx=0x7f229801c040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.840+0000 7f22ad724700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f228c071e10 0x7f228c074280 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.841+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== osd.5 v2:172.21.15.172:6816/2983726905 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f228c0748b0 con 0x7f228c071e10 2023-10-16T11:22:29.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.855+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 --> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f22a80cb120 con 0x7f228c071e10 2023-10-16T11:22:29.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.855+0000 7f229e7fc700 1 -- 172.21.15.158:0/840868151 <== osd.5 v2:172.21.15.172:6816/2983726905 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f22a4004870 con 0x7f228c071e10 2023-10-16T11:22:29.862 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:29.862 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:29.862 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:29.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f228c071e10 msgr2=0x7f228c074280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.172:6816/2983726905,v1:172.21.15.172:6817/2983726905] conn(0x7f228c071e10 0x7f228c074280 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f228c06b020 msgr2=0x7f228c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f228c06b020 0x7f228c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f22980090b0 tx=0x7f229801c040 comp rx=0 tx=0).stop 2023-10-16T11:22:29.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 msgr2=0x7f22a815a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 --2- 172.21.15.158:0/840868151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a815da40 0x7f22a815a4c0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7f22a000efc0 tx=0x7f22a000c3a0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22ae726700 1 -- 172.21.15.158:0/840868151 reap_dead start 2023-10-16T11:22:29.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 shutdown_connections 2023-10-16T11:22:29.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 >> 172.21.15.158:0/840868151 conn(0x7f22a8099a10 msgr2=0x7f22a806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:29.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 shutdown_connections 2023-10-16T11:22:29.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.856+0000 7f22b0189700 1 -- 172.21.15.158:0/840868151 wait complete. 2023-10-16T11:22:29.869 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-16T11:22:29.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.988+0000 7f55bfa14700 1 Processor -- start 2023-10-16T11:22:29.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.988+0000 7f55bfa14700 1 -- start start 2023-10-16T11:22:29.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bfa14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b815eb10 0x7f55b815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bfa14700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f55b80cfad0 0x7f55b80cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:29.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bfa14700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b815f6f0 con 0x7f55b80d0ae0 2023-10-16T11:22:29.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bfa14700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b80cdef0 con 0x7f55b80cfad0 2023-10-16T11:22:29.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bfa14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80cd070 con 0x7f55b815eb10 2023-10-16T11:22:29.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bdfb1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f55b80cfad0 0x7f55b80cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bcfaf700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f55b80d0ae0 0x7f55b80b4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45318/0 (socket says 172.21.15.158:45318) 2023-10-16T11:22:29.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bdfb1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f55b80cfad0 0x7f55b80cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45146/0 (socket says 172.21.15.158:45146) 2023-10-16T11:22:29.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.989+0000 7f55bcfaf700 1 -- 172.21.15.158:0/1902782599 learned_addr learned my addr 172.21.15.158:0/1902782599 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:29.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55bd7b0700 1 --2- 172.21.15.158:0/1902782599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b815eb10 0x7f55b815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:29.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55bdfb1700 1 -- 172.21.15.158:0/1902782599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b815eb10 msgr2=0x7f55b815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:29.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55bdfb1700 1 --2- 172.21.15.158:0/1902782599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b815eb10 0x7f55b815f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55bdfb1700 1 -- 172.21.15.158:0/1902782599 >> v1:172.21.15.158:6789/0 conn(0x7f55b80d0ae0 legacy=0x7f55b80b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55affff700 1 -- 172.21.15.158:0/1902782599 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658058182 0 0) 0x7f55b815f6f0 con 0x7f55b80d0ae0 2023-10-16T11:22:29.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55bdfb1700 1 -- 172.21.15.158:0/1902782599 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b815f990 con 0x7f55b80cfad0 2023-10-16T11:22:29.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55bdfb1700 1 --2- 172.21.15.158:0/1902782599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b80cff40 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009d90 tx=0x7f55b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7c9cc76c41d22d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:29.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55affff700 1 -- 172.21.15.158:0/1902782599 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b000e070 con 0x7f55b80cfad0 2023-10-16T11:22:29.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55affff700 1 -- 172.21.15.158:0/1902782599 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b000bb20 con 0x7f55b80cfad0 2023-10-16T11:22:29.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.990+0000 7f55affff700 1 -- 172.21.15.158:0/1902782599 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b0012430 con 0x7f55b80cfad0 2023-10-16T11:22:29.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 -- 172.21.15.158:0/1902782599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 msgr2=0x7f55b80cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:29.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 --2- 172.21.15.158:0/1902782599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b80cff40 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009d90 tx=0x7f55b000b330 comp rx=0 tx=0).stop 2023-10-16T11:22:29.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 -- 172.21.15.158:0/1902782599 shutdown_connections 2023-10-16T11:22:29.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 --2- 172.21.15.158:0/1902782599 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b80cff40 unknown :-1 s=CLOSED pgs=3444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 --2- 172.21.15.158:0/1902782599 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b815eb10 0x7f55b815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:29.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 -- 172.21.15.158:0/1902782599 >> 172.21.15.158:0/1902782599 conn(0x7f55b80ce040 msgr2=0x7f55b80ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 -- 172.21.15.158:0/1902782599 shutdown_connections 2023-10-16T11:22:30.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.991+0000 7f55bfa14700 1 -- 172.21.15.158:0/1902782599 wait complete. 2023-10-16T11:22:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.992+0000 7f55bfa14700 1 Processor -- start 2023-10-16T11:22:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.992+0000 7f55bfa14700 1 -- start start 2023-10-16T11:22:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bfa14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bfa14700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b80d0ae0 0x7f55b81582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bfa14700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55b815eb10 0x7f55b81587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bfa14700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b8159100 con 0x7f55b80cfad0 2023-10-16T11:22:30.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bfa14700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b8155760 con 0x7f55b815eb10 2023-10-16T11:22:30.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bfa14700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b8155a00 con 0x7f55b80d0ae0 2023-10-16T11:22:30.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45148/0 (socket says 172.21.15.158:45148) 2023-10-16T11:22:30.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 -- 172.21.15.158:0/284348637 learned_addr learned my addr 172.21.15.158:0/284348637 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:30.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bcfaf700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b80d0ae0 0x7f55b81582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bdfb1700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55b815eb10 0x7f55b81587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 -- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b80d0ae0 msgr2=0x7f55b81582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b80d0ae0 0x7f55b81582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 -- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55b815eb10 msgr2=0x7f55b81587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55b815eb10 0x7f55b81587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 -- 172.21.15.158:0/284348637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b0009a70 con 0x7f55b80cfad0 2023-10-16T11:22:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bd7b0700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b806b9e0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7f55b400e9b0 tx=0x7f55b400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55adffb700 1 -- 172.21.15.158:0/284348637 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b4009710 con 0x7f55b80cfad0 2023-10-16T11:22:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55adffb700 1 -- 172.21.15.158:0/284348637 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b400ee50 con 0x7f55b80cfad0 2023-10-16T11:22:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b8158e20 con 0x7f55b80cfad0 2023-10-16T11:22:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.993+0000 7f55adffb700 1 -- 172.21.15.158:0/284348637 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b400ca50 con 0x7f55b80cfad0 2023-10-16T11:22:30.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.994+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b8155f80 con 0x7f55b80cfad0 2023-10-16T11:22:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.995+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b81562f0 con 0x7f55b80cfad0 2023-10-16T11:22:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.995+0000 7f55adffb700 1 -- 172.21.15.158:0/284348637 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f55b400cbb0 con 0x7f55b80cfad0 2023-10-16T11:22:30.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.995+0000 7f55adffb700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f559c06b020 0x7f559c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.996+0000 7f55bcfaf700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f559c06b020 0x7f559c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.996+0000 7f55adffb700 1 -- 172.21.15.158:0/284348637 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f55b4089900 con 0x7f55b80cfad0 2023-10-16T11:22:30.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:29.996+0000 7f55bcfaf700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f559c06b020 0x7f559c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f55a8007fd0 tx=0x7f55a8007de0 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.000+0000 7f55adffb700 1 -- 172.21.15.158:0/284348637 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55b4054660 con 0x7f55b80cfad0 2023-10-16T11:22:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.191+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f55b815ff50 con 0x7f559c06b020 2023-10-16T11:22:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.191+0000 7f55adffb700 1 -- 172.21.15.158:0/284348637 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f55a8001370 con 0x7f559c06b020 2023-10-16T11:22:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f559c06b020 msgr2=0x7f559c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f559c06b020 0x7f559c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f55a8007fd0 tx=0x7f55a8007de0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 msgr2=0x7f55b806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b806b9e0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7f55b400e9b0 tx=0x7f55b400ecc0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 shutdown_connections 2023-10-16T11:22:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f559c06b020 0x7f559c06d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55b815eb10 0x7f55b81587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55b80cfad0 0x7f55b806b9e0 unknown :-1 s=CLOSED pgs=3445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 --2- 172.21.15.158:0/284348637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55b80d0ae0 0x7f55b81582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 >> 172.21.15.158:0/284348637 conn(0x7f55b80ce040 msgr2=0x7f55b80cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.193+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 shutdown_connections 2023-10-16T11:22:30.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.194+0000 7f55bfa14700 1 -- 172.21.15.158:0/284348637 wait complete. 2023-10-16T11:22:30.199 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 5 to deep-scrub 2023-10-16T11:22:30.207 INFO:tasks.ceph:Scrubbing osd.6 2023-10-16T11:22:30.207 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:30.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.330+0000 7fc9e8d79700 1 Processor -- start 2023-10-16T11:22:30.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.330+0000 7fc9e8d79700 1 -- start start 2023-10-16T11:22:30.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e8d79700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e8d79700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9e40cc590 0x7fc9e406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e8d79700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e40cbd30 con 0x7fc9e406ae10 2023-10-16T11:22:30.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e8d79700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e406b280 con 0x7fc9e40cc590 2023-10-16T11:22:30.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e8d79700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40cc1c0 con 0x7fc9e415da40 2023-10-16T11:22:30.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e1d9b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9e40cc590 0x7fc9e406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e2d9d700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc9e406ae10 0x7fc9e40ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45336/0 (socket says 172.21.15.158:45336) 2023-10-16T11:22:30.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.331+0000 7fc9e2d9d700 1 -- 172.21.15.158:0/3239162669 learned_addr learned my addr 172.21.15.158:0/3239162669 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:30.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e259c700 1 --2- 172.21.15.158:0/3239162669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e1d9b700 1 -- 172.21.15.158:0/3239162669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 msgr2=0x7fc9e40caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:30.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e1d9b700 1 --2- 172.21.15.158:0/3239162669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e1d9b700 1 -- 172.21.15.158:0/3239162669 >> v1:172.21.15.158:6789/0 conn(0x7fc9e406ae10 legacy=0x7fc9e40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e1d9b700 1 -- 172.21.15.158:0/3239162669 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e415de20 con 0x7fc9e40cc590 2023-10-16T11:22:30.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e159a700 1 -- 172.21.15.158:0/3239162669 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221608704 0 0) 0x7fc9e40cbd30 con 0x7fc9e406ae10 2023-10-16T11:22:30.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e259c700 1 --2- 172.21.15.158:0/3239162669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:30.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e1d9b700 1 --2- 172.21.15.158:0/3239162669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e406a8d0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4009840 tx=0x7fc9d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb49da7aee451b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e159a700 1 -- 172.21.15.158:0/3239162669 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4011830 con 0x7fc9e40cc590 2023-10-16T11:22:30.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e159a700 1 -- 172.21.15.158:0/3239162669 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d4011990 con 0x7fc9e40cc590 2023-10-16T11:22:30.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e159a700 1 -- 172.21.15.158:0/3239162669 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4011d20 con 0x7fc9e40cc590 2023-10-16T11:22:30.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e8d79700 1 -- 172.21.15.158:0/3239162669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 msgr2=0x7fc9e406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.332+0000 7fc9e8d79700 1 --2- 172.21.15.158:0/3239162669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e406a8d0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4009840 tx=0x7fc9d4009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:30.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.333+0000 7fc9e8d79700 1 -- 172.21.15.158:0/3239162669 shutdown_connections 2023-10-16T11:22:30.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.333+0000 7fc9e8d79700 1 --2- 172.21.15.158:0/3239162669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.333+0000 7fc9e8d79700 1 --2- 172.21.15.158:0/3239162669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e406a8d0 unknown :-1 s=CLOSED pgs=3447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.333+0000 7fc9e8d79700 1 -- 172.21.15.158:0/3239162669 >> 172.21.15.158:0/3239162669 conn(0x7fc9e4099a10 msgr2=0x7fc9e40d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.333+0000 7fc9e8d79700 1 -- 172.21.15.158:0/3239162669 shutdown_connections 2023-10-16T11:22:30.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.333+0000 7fc9e8d79700 1 -- 172.21.15.158:0/3239162669 wait complete. 2023-10-16T11:22:30.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.334+0000 7fc9e8d79700 1 Processor -- start 2023-10-16T11:22:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.334+0000 7fc9e8d79700 1 -- start start 2023-10-16T11:22:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9e406ae10 0x7fc9e4149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e40cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40c66a0 con 0x7fc9e40cc590 2023-10-16T11:22:30.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40c9190 con 0x7fc9e406ae10 2023-10-16T11:22:30.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40c80f0 con 0x7fc9e415da40 2023-10-16T11:22:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e40cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e40cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45164/0 (socket says 172.21.15.158:45164) 2023-10-16T11:22:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 -- 172.21.15.158:0/181687645 learned_addr learned my addr 172.21.15.158:0/181687645 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e259c700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9e406ae10 0x7fc9e4149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e2d9d700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 -- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 msgr2=0x7fc9e40cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9e415da40 0x7fc9e40cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 -- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9e406ae10 msgr2=0x7fc9e4149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9e406ae10 0x7fc9e4149790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 -- 172.21.15.158:0/181687645 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d40094a0 con 0x7fc9e40cc590 2023-10-16T11:22:30.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e1d9b700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e40cce80 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40090b0 tx=0x7fc9d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d40036a0 con 0x7fc9e40cc590 2023-10-16T11:22:30.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d401f070 con 0x7fc9e40cc590 2023-10-16T11:22:30.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4003a20 con 0x7fc9e40cc590 2023-10-16T11:22:30.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e40c83d0 con 0x7fc9e40cc590 2023-10-16T11:22:30.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.335+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e40c8990 con 0x7fc9e40cc590 2023-10-16T11:22:30.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.336+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc9e40cb120 con 0x7fc9e40cc590 2023-10-16T11:22:30.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.337+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc9d4003b80 con 0x7fc9e40cc590 2023-10-16T11:22:30.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.337+0000 7fc9d2ffd700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fc9c406b050 0x7fc9c406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.337+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc9d4021070 con 0x7fc9e40cc590 2023-10-16T11:22:30.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.337+0000 7fc9e259c700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fc9c406b050 0x7fc9c406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.337+0000 7fc9d2ffd700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7fc9c4071e40 0x7fc9c40742b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.337+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 --> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9c40748e0 con 0x7fc9c4071e40 2023-10-16T11:22:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.338+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9d408d990 con 0x7fc9e40cc590 2023-10-16T11:22:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.338+0000 7fc9e2d9d700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7fc9c4071e40 0x7fc9c40742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.338+0000 7fc9e259c700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fc9c406b050 0x7fc9c406d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006fd0 tx=0x7fc9d8008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.338+0000 7fc9e2d9d700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7fc9c4071e40 0x7fc9c40742b0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.339+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== osd.6 v2:172.21.15.172:6800/3389147915 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fc9c40748e0 con 0x7fc9c4071e40 2023-10-16T11:22:30.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.353+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 --> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc9e40ce260 con 0x7fc9c4071e40 2023-10-16T11:22:30.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9d2ffd700 1 -- 172.21.15.158:0/181687645 <== osd.6 v2:172.21.15.172:6800/3389147915 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc9dc007240 con 0x7fc9c4071e40 2023-10-16T11:22:30.357 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:30.357 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:30.357 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:30.358 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7fc9c4071e40 msgr2=0x7fc9c40742b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.172:6800/3389147915,v1:172.21.15.172:6801/3389147915] conn(0x7fc9c4071e40 0x7fc9c40742b0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fc9c406b050 msgr2=0x7fc9c406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fc9c406b050 0x7fc9c406d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006fd0 tx=0x7fc9d8008040 comp rx=0 tx=0).stop 2023-10-16T11:22:30.359 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 msgr2=0x7fc9e40cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 --2- 172.21.15.158:0/181687645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9e40cc590 0x7fc9e40cce80 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40090b0 tx=0x7fc9d4004750 comp rx=0 tx=0).stop 2023-10-16T11:22:30.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e2d9d700 1 -- 172.21.15.158:0/181687645 reap_dead start 2023-10-16T11:22:30.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 shutdown_connections 2023-10-16T11:22:30.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 >> 172.21.15.158:0/181687645 conn(0x7fc9e4099a10 msgr2=0x7fc9e40b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 shutdown_connections 2023-10-16T11:22:30.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.354+0000 7fc9e8d79700 1 -- 172.21.15.158:0/181687645 wait complete. 2023-10-16T11:22:30.367 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-16T11:22:30.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.487+0000 7fa4c34d1700 1 Processor -- start 2023-10-16T11:22:30.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.488+0000 7fa4c34d1700 1 -- start start 2023-10-16T11:22:30.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.488+0000 7fa4c34d1700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa4bc15eb10 0x7fa4bc15f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.488+0000 7fa4c34d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0cfad0 0x7fa4bc0cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.488+0000 7fa4c34d1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4bc15f6f0 con 0x7fa4bc0d0ae0 2023-10-16T11:22:30.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.488+0000 7fa4c34d1700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4bc0cdef0 con 0x7fa4bc15eb10 2023-10-16T11:22:30.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.488+0000 7fa4c34d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0cd070 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa4bc15eb10 0x7fa4bc15f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa4bc15eb10 0x7fa4bc15f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45172/0 (socket says 172.21.15.158:45172) 2023-10-16T11:22:30.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 -- 172.21.15.158:0/1421630781 learned_addr learned my addr 172.21.15.158:0/1421630781 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:30.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c1a6e700 1 --2- 172.21.15.158:0/1421630781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0cfad0 0x7fa4bc0cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 -- 172.21.15.158:0/1421630781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0cfad0 msgr2=0x7fa4bc0cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:30.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 --2- 172.21.15.158:0/1421630781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0cfad0 0x7fa4bc0cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 -- 172.21.15.158:0/1421630781 >> v1:172.21.15.172:6789/0 conn(0x7fa4bc0d0ae0 legacy=0x7fa4bc0b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 -- 172.21.15.158:0/1421630781 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc15f990 con 0x7fa4bc15eb10 2023-10-16T11:22:30.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4c126d700 1 --2- 172.21.15.158:0/1421630781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc15f1b0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0098f0 tx=0x7fa4ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f543f09029c10757 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4b3fff700 1 -- 172.21.15.158:0/1421630781 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac011830 con 0x7fa4bc15eb10 2023-10-16T11:22:30.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4b3fff700 1 -- 172.21.15.158:0/1421630781 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac011990 con 0x7fa4bc15eb10 2023-10-16T11:22:30.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.489+0000 7fa4b3fff700 1 -- 172.21.15.158:0/1421630781 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac011d20 con 0x7fa4bc15eb10 2023-10-16T11:22:30.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 -- 172.21.15.158:0/1421630781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc15eb10 msgr2=0x7fa4bc15f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/1421630781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc15f1b0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0098f0 tx=0x7fa4ac009c00 comp rx=0 tx=0).stop 2023-10-16T11:22:30.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 -- 172.21.15.158:0/1421630781 shutdown_connections 2023-10-16T11:22:30.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/1421630781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0cfad0 0x7fa4bc0cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/1421630781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc15f1b0 unknown :-1 s=CLOSED pgs=3449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 -- 172.21.15.158:0/1421630781 >> 172.21.15.158:0/1421630781 conn(0x7fa4bc0ce040 msgr2=0x7fa4bc0ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 -- 172.21.15.158:0/1421630781 shutdown_connections 2023-10-16T11:22:30.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.490+0000 7fa4c34d1700 1 -- 172.21.15.158:0/1421630781 wait complete. 2023-10-16T11:22:30.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.491+0000 7fa4c34d1700 1 Processor -- start 2023-10-16T11:22:30.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.491+0000 7fa4c34d1700 1 -- start start 2023-10-16T11:22:30.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c34d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc0cfad0 0x7fa4bc0c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c34d1700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0d0ae0 0x7fa4bc157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c34d1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc0c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c34d1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0c81b0 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c34d1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc155290 con 0x7fa4bc15eb10 2023-10-16T11:22:30.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c34d1700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc154680 con 0x7fa4bc0d0ae0 2023-10-16T11:22:30.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc0cfad0 0x7fa4bc0c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc0cfad0 0x7fa4bc0c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45176/0 (socket says 172.21.15.158:45176) 2023-10-16T11:22:30.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 -- 172.21.15.158:0/3835942213 learned_addr learned my addr 172.21.15.158:0/3835942213 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:30.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c1a6e700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc0c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c0a6c700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0d0ae0 0x7fa4bc157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 -- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0d0ae0 msgr2=0x7fa4bc157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0d0ae0 0x7fa4bc157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 -- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4bc15eb10 msgr2=0x7fa4bc0c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc0c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 -- 172.21.15.158:0/3835942213 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac009550 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c1a6e700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc0c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:30.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c126d700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc0cfad0 0x7fa4bc0c7c70 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009c40 tx=0x7fa4ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4b1ffb700 1 -- 172.21.15.158:0/3835942213 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0036a0 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4b1ffb700 1 -- 172.21.15.158:0/3835942213 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac01e070 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4b1ffb700 1 -- 172.21.15.158:0/3835942213 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac003a60 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.492+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc154960 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.493+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc154f20 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.493+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc0681b0 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.494+0000 7fa4b1ffb700 1 -- 172.21.15.158:0/3835942213 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa4ac003bc0 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.494+0000 7fa4b1ffb700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa4a006b020 0x7fa4a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.494+0000 7fa4b1ffb700 1 -- 172.21.15.158:0/3835942213 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa4ac08a9c0 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.495+0000 7fa4c0a6c700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa4a006b020 0x7fa4a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.495+0000 7fa4c0a6c700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa4a006b020 0x7fa4a006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0b0b90 tx=0x7fa4b8009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.499+0000 7fa4b1ffb700 1 -- 172.21.15.158:0/3835942213 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa4ac055760 con 0x7fa4bc0cfad0 2023-10-16T11:22:30.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.691+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4bc15ff50 con 0x7fa4a006b020 2023-10-16T11:22:30.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.691+0000 7fa4b1ffb700 1 -- 172.21.15.158:0/3835942213 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa4b80050d0 con 0x7fa4a006b020 2023-10-16T11:22:30.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa4a006b020 msgr2=0x7fa4a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa4a006b020 0x7fa4a006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0b0b90 tx=0x7fa4b8009040 comp rx=0 tx=0).stop 2023-10-16T11:22:30.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc0cfad0 msgr2=0x7fa4bc0c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc0cfad0 0x7fa4bc0c7c70 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009c40 tx=0x7fa4ac004750 comp rx=0 tx=0).stop 2023-10-16T11:22:30.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 shutdown_connections 2023-10-16T11:22:30.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fa4a006b020 0x7fa4a006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4bc15eb10 0x7fa4bc0c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4bc0cfad0 0x7fa4bc0c7c70 unknown :-1 s=CLOSED pgs=3450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 --2- 172.21.15.158:0/3835942213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4bc0d0ae0 0x7fa4bc157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 >> 172.21.15.158:0/3835942213 conn(0x7fa4bc0ce040 msgr2=0x7fa4bc0c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 shutdown_connections 2023-10-16T11:22:30.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.693+0000 7fa4c34d1700 1 -- 172.21.15.158:0/3835942213 wait complete. 2023-10-16T11:22:30.699 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 6 to deep-scrub 2023-10-16T11:22:30.707 INFO:tasks.ceph:Scrubbing osd.7 2023-10-16T11:22:30.707 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-16T11:22:30.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.830+0000 7facaf13a700 1 Processor -- start 2023-10-16T11:22:30.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.830+0000 7facaf13a700 1 -- start start 2023-10-16T11:22:30.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.831+0000 7facaf13a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faca815b6b0 0x7faca80cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.831+0000 7facaf13a700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca80b4420 con 0x7faca80b3630 2023-10-16T11:22:30.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.831+0000 7facaf13a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca80cbc20 con 0x7faca80d0c60 2023-10-16T11:22:30.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.831+0000 7facaf13a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca815ba90 con 0x7faca815b6b0 2023-10-16T11:22:30.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.831+0000 7facad6d7700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7faca80b3630 0x7faca80b48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45362/0 (socket says 172.21.15.158:45362) 2023-10-16T11:22:30.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.831+0000 7facad6d7700 1 -- 172.21.15.158:0/1738292061 learned_addr learned my addr 172.21.15.158:0/1738292061 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:30.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facaced6700 1 --2- 172.21.15.158:0/1738292061 >> v2:172.21.15.172:3300/0 conn(0x7faca815b6b0 0x7faca80cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762710484 0 0) 0x7faca80b4420 con 0x7faca80b3630 2023-10-16T11:22:30.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac980035b0 con 0x7faca80b3630 2023-10-16T11:22:30.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162970985 0 0) 0x7fac980035b0 con 0x7faca80b3630 2023-10-16T11:22:30.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 >> v2:172.21.15.172:3300/0 conn(0x7faca815b6b0 msgr2=0x7faca80cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 --2- 172.21.15.158:0/1738292061 >> v2:172.21.15.172:3300/0 conn(0x7faca815b6b0 0x7faca80cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 >> v1:172.21.15.172:6789/0 conn(0x7faca80d0c60 legacy=0x7faca80b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca80cbf00 con 0x7faca80b3630 2023-10-16T11:22:30.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7faca00019f0 con 0x7faca80b3630 2023-10-16T11:22:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 >> v1:172.21.15.158:6789/0 conn(0x7faca80b3630 legacy=0x7faca80b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac98003a00 0x7fac98003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98004400 0x7fac980048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac98005170 0x7fac9800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9800c080 con 0x7fac98005170 2023-10-16T11:22:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9800c1f0 con 0x7fac98004400 2023-10-16T11:22:30.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9800c3b0 con 0x7fac98003a00 2023-10-16T11:22:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac98005170 0x7fac9800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facaced6700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac98003a00 0x7fac98003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7faca7fff700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98004400 0x7fac980048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 -- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac98003a00 msgr2=0x7fac98003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac98003a00 0x7fac98003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 -- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98004400 msgr2=0x7fac980048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:30.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98004400 0x7fac980048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 -- 172.21.15.158:0/1738292061 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac980035b0 con 0x7fac98005170 2023-10-16T11:22:30.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 -- 172.21.15.158:0/1738292061 reap_dead start 2023-10-16T11:22:30.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.832+0000 7facad6d7700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac98005170 0x7fac9800ba10 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7faca80ced20 tx=0x7fac98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6477fa6b6fd514e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0012600 con 0x7fac98005170 2023-10-16T11:22:30.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca0012760 con 0x7fac98005170 2023-10-16T11:22:30.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7faca77fe700 1 -- 172.21.15.158:0/1738292061 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0012b10 con 0x7fac98005170 2023-10-16T11:22:30.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7facaf13a700 1 -- 172.21.15.158:0/1738292061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac98005170 msgr2=0x7fac9800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7facaf13a700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac98005170 0x7fac9800ba10 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7faca80ced20 tx=0x7fac98005880 comp rx=0 tx=0).stop 2023-10-16T11:22:30.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7facaf13a700 1 -- 172.21.15.158:0/1738292061 shutdown_connections 2023-10-16T11:22:30.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7facaf13a700 1 --2- 172.21.15.158:0/1738292061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac98005170 0x7fac9800ba10 unknown :-1 s=CLOSED pgs=3452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7facaf13a700 1 -- 172.21.15.158:0/1738292061 >> 172.21.15.158:0/1738292061 conn(0x7faca80cdf20 msgr2=0x7faca80ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7facaf13a700 1 -- 172.21.15.158:0/1738292061 shutdown_connections 2023-10-16T11:22:30.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.833+0000 7facaf13a700 1 -- 172.21.15.158:0/1738292061 wait complete. 2023-10-16T11:22:30.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.834+0000 7facaf13a700 1 Processor -- start 2023-10-16T11:22:30.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.834+0000 7facaf13a700 1 -- start start 2023-10-16T11:22:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7facaf13a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98005170 0x7faca81575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7facaf13a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faca80cb130 0x7faca80c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7facaf13a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faca81e4b80 0x7faca81e5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7facaf13a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca81e5550 con 0x7faca80cb130 2023-10-16T11:22:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7facaf13a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca80a8890 con 0x7fac98005170 2023-10-16T11:22:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7facaf13a700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca80c94a0 con 0x7faca81e4b80 2023-10-16T11:22:30.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7faca7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faca80cb130 0x7faca80c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7faca7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faca80cb130 0x7faca80c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45198/0 (socket says 172.21.15.158:45198) 2023-10-16T11:22:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.835+0000 7faca7fff700 1 -- 172.21.15.158:0/2287860434 learned_addr learned my addr 172.21.15.158:0/2287860434 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7facaced6700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98005170 0x7faca81575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7facad6d7700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faca81e4b80 0x7faca81e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca7fff700 1 -- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faca81e4b80 msgr2=0x7faca81e5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca7fff700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faca81e4b80 0x7faca81e5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca7fff700 1 -- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98005170 msgr2=0x7faca81575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca7fff700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98005170 0x7faca81575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca7fff700 1 -- 172.21.15.158:0/2287860434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca80c8ee0 con 0x7faca80cb130 2023-10-16T11:22:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7facaced6700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac98005170 0x7faca81575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:30.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca7fff700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faca80cb130 0x7faca80c8570 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7fac9400f750 tx=0x7fac9400fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac940052e0 con 0x7faca80cb130 2023-10-16T11:22:30.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac94005460 con 0x7faca80cb130 2023-10-16T11:22:30.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac940038c0 con 0x7faca80cb130 2023-10-16T11:22:30.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca80c91c0 con 0x7faca80cb130 2023-10-16T11:22:30.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.836+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca8157b20 con 0x7faca80cb130 2023-10-16T11:22:30.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.837+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faca80cc4c0 con 0x7faca80cb130 2023-10-16T11:22:30.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.837+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac94003a20 con 0x7faca80cb130 2023-10-16T11:22:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7faca57fa700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fac8c06b020 0x7fac8c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fac9408a2c0 con 0x7faca80cb130 2023-10-16T11:22:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7facaced6700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fac8c06b020 0x7fac8c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7faca57fa700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fac8c071e10 0x7fac8c074280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 --> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac8c0748b0 con 0x7fac8c071e10 2023-10-16T11:22:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac9408a710 con 0x7faca80cb130 2023-10-16T11:22:30.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7facaced6700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fac8c06b020 0x7fac8c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009ae0 tx=0x7fac9c008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.838+0000 7facad6d7700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fac8c071e10 0x7fac8c074280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.839+0000 7facad6d7700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fac8c071e10 0x7fac8c074280 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.839+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== osd.7 v2:172.21.15.172:6808/2085135193 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fac8c0748b0 con 0x7fac8c071e10 2023-10-16T11:22:30.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.854+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 --> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faca80cc1e0 con 0x7fac8c071e10 2023-10-16T11:22:30.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.854+0000 7faca57fa700 1 -- 172.21.15.158:0/2287860434 <== osd.7 v2:172.21.15.172:6808/2085135193 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7faca0001410 con 0x7fac8c071e10 2023-10-16T11:22:30.863 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-16T11:22:30.863 INFO:teuthology.orchestra.run.smithi158.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T11:22:30.863 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-16T11:22:30.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fac8c071e10 msgr2=0x7fac8c074280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.172:6808/2085135193,v1:172.21.15.172:6809/2085135193] conn(0x7fac8c071e10 0x7fac8c074280 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fac8c06b020 msgr2=0x7fac8c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fac8c06b020 0x7fac8c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009ae0 tx=0x7fac9c008040 comp rx=0 tx=0).stop 2023-10-16T11:22:30.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faca80cb130 msgr2=0x7faca80c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 --2- 172.21.15.158:0/2287860434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faca80cb130 0x7faca80c8570 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7fac9400f750 tx=0x7fac9400fb10 comp rx=0 tx=0).stop 2023-10-16T11:22:30.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facad6d7700 1 -- 172.21.15.158:0/2287860434 reap_dead start 2023-10-16T11:22:30.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 shutdown_connections 2023-10-16T11:22:30.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 >> 172.21.15.158:0/2287860434 conn(0x7faca80cdf20 msgr2=0x7faca815ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:30.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 shutdown_connections 2023-10-16T11:22:30.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.855+0000 7facaf13a700 1 -- 172.21.15.158:0/2287860434 wait complete. 2023-10-16T11:22:30.868 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-16T11:22:30.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.988+0000 7f1f7a3ac700 1 Processor -- start 2023-10-16T11:22:30.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.988+0000 7f1f7a3ac700 1 -- start start 2023-10-16T11:22:30.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f7a3ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f7415eb10 0x7f1f7415f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f7a3ac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f740cfad0 0x7f1f740cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:30.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f7a3ac700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7415f6f0 con 0x7f1f740d0ae0 2023-10-16T11:22:30.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f7a3ac700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f740cdef0 con 0x7f1f740cfad0 2023-10-16T11:22:30.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f7a3ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740cd070 con 0x7f1f7415eb10 2023-10-16T11:22:30.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f73fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f7415eb10 0x7f1f7415f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f78949700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f740cfad0 0x7f1f740cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:30.995 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f737fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1f740d0ae0 0x7f1f740b4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.158:49286/0 (socket says 172.21.15.158:49286) 2023-10-16T11:22:30.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f737fe700 1 -- 172.21.15.158:0/4249682080 learned_addr learned my addr 172.21.15.158:0/4249682080 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:30.996 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f72ffd700 1 -- 172.21.15.158:0/4249682080 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868862185 0 0) 0x7f1f7415f6f0 con 0x7f1f740d0ae0 2023-10-16T11:22:30.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f72ffd700 1 -- 172.21.15.158:0/4249682080 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f5c0035b0 con 0x7f1f740d0ae0 2023-10-16T11:22:30.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f73fff700 1 -- 172.21.15.158:0/4249682080 >> v2:172.21.15.172:3300/0 conn(0x7f1f740cfad0 msgr2=0x7f1f740cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:30.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f73fff700 1 --2- 172.21.15.158:0/4249682080 >> v2:172.21.15.172:3300/0 conn(0x7f1f740cfad0 0x7f1f740cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:30.997 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.989+0000 7f1f73fff700 1 -- 172.21.15.158:0/4249682080 >> v1:172.21.15.172:6789/0 conn(0x7f1f740d0ae0 legacy=0x7f1f740b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:30.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f73fff700 1 -- 172.21.15.158:0/4249682080 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7415f990 con 0x7f1f7415eb10 2023-10-16T11:22:30.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f78949700 1 --2- 172.21.15.158:0/4249682080 >> v2:172.21.15.172:3300/0 conn(0x7f1f740cfad0 0x7f1f740cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T11:22:30.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f73fff700 1 --2- 172.21.15.158:0/4249682080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f7415eb10 0x7f1f7415f1b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009840 tx=0x7f1f68009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d03c8fbd2f12a9b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:30.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f72ffd700 1 -- 172.21.15.158:0/4249682080 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68011830 con 0x7f1f7415eb10 2023-10-16T11:22:30.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f72ffd700 1 -- 172.21.15.158:0/4249682080 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68011990 con 0x7f1f7415eb10 2023-10-16T11:22:30.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f72ffd700 1 -- 172.21.15.158:0/4249682080 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68011d00 con 0x7f1f7415eb10 2023-10-16T11:22:31.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/4249682080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f7415eb10 msgr2=0x7f1f7415f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/4249682080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f7415eb10 0x7f1f7415f1b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009840 tx=0x7f1f68009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:31.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/4249682080 shutdown_connections 2023-10-16T11:22:31.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.990+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/4249682080 >> v2:172.21.15.172:3300/0 conn(0x7f1f740cfad0 0x7f1f740cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.991+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/4249682080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f7415eb10 0x7f1f7415f1b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.991+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/4249682080 >> 172.21.15.158:0/4249682080 conn(0x7f1f740ce040 msgr2=0x7f1f740ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:31.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.991+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/4249682080 shutdown_connections 2023-10-16T11:22:31.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.991+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/4249682080 wait complete. 2023-10-16T11:22:31.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.992+0000 7f1f7a3ac700 1 Processor -- start 2023-10-16T11:22:31.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.992+0000 7f1f7a3ac700 1 -- start start 2023-10-16T11:22:31.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f7a3ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f740cfad0 0x7f1f740c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f7a3ac700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f740d0ae0 0x7f1f74159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f7a3ac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f7415eb10 0x7f1f74157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f7a3ac700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f741f0d00 con 0x7f1f740cfad0 2023-10-16T11:22:31.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f7a3ac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74154650 con 0x7f1f7415eb10 2023-10-16T11:22:31.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f7a3ac700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74159a10 con 0x7f1f740d0ae0 2023-10-16T11:22:31.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f73fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f740cfad0 0x7f1f740c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f73fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f740cfad0 0x7f1f740c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45212/0 (socket says 172.21.15.158:45212) 2023-10-16T11:22:31.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.993+0000 7f1f73fff700 1 -- 172.21.15.158:0/2614765591 learned_addr learned my addr 172.21.15.158:0/2614765591 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:31.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f78949700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f7415eb10 0x7f1f74157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.008 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f737fe700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f740d0ae0 0x7f1f74159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f73fff700 1 -- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f740d0ae0 msgr2=0x7f1f74159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f73fff700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f740d0ae0 0x7f1f74159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f73fff700 1 -- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f7415eb10 msgr2=0x7f1f74157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f73fff700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f7415eb10 0x7f1f74157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f73fff700 1 -- 172.21.15.158:0/2614765591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c009a70 con 0x7f1f740cfad0 2023-10-16T11:22:31.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f73fff700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f740cfad0 0x7f1f740c7d50 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009b90 tx=0x7f1f68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:31.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f70ff9700 1 -- 172.21.15.158:0/2614765591 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f680036a0 con 0x7f1f740cfad0 2023-10-16T11:22:31.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f70ff9700 1 -- 172.21.15.158:0/2614765591 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68003800 con 0x7f1f740cfad0 2023-10-16T11:22:31.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f680094a0 con 0x7f1f740cfad0 2023-10-16T11:22:31.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f70ff9700 1 -- 172.21.15.158:0/2614765591 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68003b70 con 0x7f1f740cfad0 2023-10-16T11:22:31.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.994+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f741548a0 con 0x7f1f740cfad0 2023-10-16T11:22:31.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.995+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f740681b0 con 0x7f1f740cfad0 2023-10-16T11:22:31.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.995+0000 7f1f70ff9700 1 -- 172.21.15.158:0/2614765591 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f68003cd0 con 0x7f1f740cfad0 2023-10-16T11:22:31.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.996+0000 7f1f70ff9700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f6006b020 0x7f1f6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.012 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.996+0000 7f1f70ff9700 1 -- 172.21.15.158:0/2614765591 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1f6808ad50 con 0x7f1f740cfad0 2023-10-16T11:22:31.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.996+0000 7f1f737fe700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f6006b020 0x7f1f6006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:30.996+0000 7f1f737fe700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f6006b020 0x7f1f6006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f740c91b0 tx=0x7f1f64009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:31.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.000+0000 7f1f70ff9700 1 -- 172.21.15.158:0/2614765591 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f6808e030 con 0x7f1f740cfad0 2023-10-16T11:22:31.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.190+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f7415ff50 con 0x7f1f6006b020 2023-10-16T11:22:31.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.191+0000 7f1f70ff9700 1 -- 172.21.15.158:0/2614765591 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f64001470 con 0x7f1f6006b020 2023-10-16T11:22:31.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f6006b020 msgr2=0x7f1f6006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f6006b020 0x7f1f6006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f740c91b0 tx=0x7f1f64009040 comp rx=0 tx=0).stop 2023-10-16T11:22:31.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f740cfad0 msgr2=0x7f1f740c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f740cfad0 0x7f1f740c7d50 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009b90 tx=0x7f1f68004750 comp rx=0 tx=0).stop 2023-10-16T11:22:31.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 shutdown_connections 2023-10-16T11:22:31.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f6006b020 0x7f1f6006d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f7415eb10 0x7f1f74157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f740cfad0 0x7f1f740c7d50 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 --2- 172.21.15.158:0/2614765591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f740d0ae0 0x7f1f74159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 >> 172.21.15.158:0/2614765591 conn(0x7f1f740ce040 msgr2=0x7f1f740b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:31.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 shutdown_connections 2023-10-16T11:22:31.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.193+0000 7f1f7a3ac700 1 -- 172.21.15.158:0/2614765591 wait complete. 2023-10-16T11:22:31.199 INFO:teuthology.orchestra.run.smithi158.stderr:instructed osd(s) 7 to deep-scrub 2023-10-16T11:22:31.207 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:22:31.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.325+0000 7f1f547b2700 1 Processor -- start 2023-10-16T11:22:31.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.325+0000 7f1f547b2700 1 -- start start 2023-10-16T11:22:31.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.326+0000 7f1f547b2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f4c06ab40 0x7f1f4c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.326+0000 7f1f547b2700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f4c0729b0 0x7f1f4c071910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.326+0000 7f1f547b2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c071e50 con 0x7f1f4c06e5d0 2023-10-16T11:22:31.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.326+0000 7f1f547b2700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c161270 con 0x7f1f4c0729b0 2023-10-16T11:22:31.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.326+0000 7f1f547b2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c161510 con 0x7f1f4c06ab40 2023-10-16T11:22:31.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f4c0729b0 0x7f1f4c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f4c0729b0 0x7f1f4c071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45222/0 (socket says 172.21.15.158:45222) 2023-10-16T11:22:31.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 -- 172.21.15.158:0/3220940939 learned_addr learned my addr 172.21.15.158:0/3220940939 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:31.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f5254e700 1 --2- 172.21.15.158:0/3220940939 >> v2:172.21.15.172:3300/0 conn(0x7f1f4c06ab40 0x7f1f4c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 -- 172.21.15.158:0/3220940939 >> v2:172.21.15.172:3300/0 conn(0x7f1f4c06ab40 msgr2=0x7f1f4c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:31.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 --2- 172.21.15.158:0/3220940939 >> v2:172.21.15.172:3300/0 conn(0x7f1f4c06ab40 0x7f1f4c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 -- 172.21.15.158:0/3220940939 >> v1:172.21.15.172:6789/0 conn(0x7f1f4c06e5d0 legacy=0x7f1f4c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 -- 172.21.15.158:0/3220940939 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c1617f0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f52d4f700 1 --2- 172.21.15.158:0/3220940939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c071910 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c074d70 tx=0x7f1f4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbdf7f802cdc6a8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:31.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f5154c700 1 -- 172.21.15.158:0/3220940939 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4400e070 con 0x7f1f4c0729b0 2023-10-16T11:22:31.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f5154c700 1 -- 172.21.15.158:0/3220940939 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4400b9c0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.327+0000 7f1f5154c700 1 -- 172.21.15.158:0/3220940939 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4400bd70 con 0x7f1f4c0729b0 2023-10-16T11:22:31.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 -- 172.21.15.158:0/3220940939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 msgr2=0x7f1f4c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 --2- 172.21.15.158:0/3220940939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c071910 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c074d70 tx=0x7f1f4400b330 comp rx=0 tx=0).stop 2023-10-16T11:22:31.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 -- 172.21.15.158:0/3220940939 shutdown_connections 2023-10-16T11:22:31.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 --2- 172.21.15.158:0/3220940939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c071910 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 --2- 172.21.15.158:0/3220940939 >> v2:172.21.15.172:3300/0 conn(0x7f1f4c06ab40 0x7f1f4c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 -- 172.21.15.158:0/3220940939 >> 172.21.15.158:0/3220940939 conn(0x7f1f4c0d02b0 msgr2=0x7f1f4c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:31.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 -- 172.21.15.158:0/3220940939 shutdown_connections 2023-10-16T11:22:31.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.328+0000 7f1f547b2700 1 -- 172.21.15.158:0/3220940939 wait complete. 2023-10-16T11:22:31.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.329+0000 7f1f547b2700 1 Processor -- start 2023-10-16T11:22:31.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.329+0000 7f1f547b2700 1 -- start start 2023-10-16T11:22:31.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f547b2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f4c06ab40 0x7f1f4c1f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f547b2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f4c06e5d0 0x7f1f4c1495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f547b2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c0c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f547b2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c160b80 con 0x7f1f4c0729b0 2023-10-16T11:22:31.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f547b2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1f40c0 con 0x7f1f4c06e5d0 2023-10-16T11:22:31.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f547b2700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c0c4100 con 0x7f1f4c06ab40 2023-10-16T11:22:31.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c0c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c0c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45228/0 (socket says 172.21.15.158:45228) 2023-10-16T11:22:31.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 -- 172.21.15.158:0/2065549801 learned_addr learned my addr 172.21.15.158:0/2065549801 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:31.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f5254e700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f4c06ab40 0x7f1f4c1f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f51d4d700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f4c06e5d0 0x7f1f4c1495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 -- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f4c06ab40 msgr2=0x7f1f4c1f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f4c06ab40 0x7f1f4c1f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 -- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f4c06e5d0 msgr2=0x7f1f4c1495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f4c06e5d0 0x7f1f4c1495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 -- 172.21.15.158:0/2065549801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f44009b30 con 0x7f1f4c0729b0 2023-10-16T11:22:31.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f52d4f700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c0c5b60 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f1f44012000 tx=0x7f1f44004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:31.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.330+0000 7f1f42ffd700 1 -- 172.21.15.158:0/2065549801 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44004840 con 0x7f1f4c0729b0 2023-10-16T11:22:31.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.331+0000 7f1f42ffd700 1 -- 172.21.15.158:0/2065549801 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f440099b0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.331+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c0c43e0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.331+0000 7f1f42ffd700 1 -- 172.21.15.158:0/2065549801 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4401c680 con 0x7f1f4c0729b0 2023-10-16T11:22:31.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.331+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c0c49a0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.331+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4c0681b0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.332+0000 7f1f42ffd700 1 -- 172.21.15.158:0/2065549801 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f4401c7e0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.332+0000 7f1f42ffd700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f3006b020 0x7f1f3006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:31.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.332+0000 7f1f42ffd700 1 -- 172.21.15.158:0/2065549801 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1f4408b2e0 con 0x7f1f4c0729b0 2023-10-16T11:22:31.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.332+0000 7f1f5254e700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f3006b020 0x7f1f3006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:31.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.333+0000 7f1f5254e700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f3006b020 0x7f1f3006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0b2870 tx=0x7f1f3c008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:31.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.337+0000 7f1f42ffd700 1 -- 172.21.15.158:0/2065549801 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f44055f30 con 0x7f1f4c0729b0 2023-10-16T11:22:31.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.525+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f4c161db0 con 0x7f1f3006b020 2023-10-16T11:22:31.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.527+0000 7f1f42ffd700 1 -- 172.21.15.158:0/2065549801 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50618 (secure 0 0 0) 0x7f1f3c0076d0 con 0x7f1f3006b020 2023-10-16T11:22:31.530 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:31.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.529+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f3006b020 msgr2=0x7f1f3006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f3006b020 0x7f1f3006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0b2870 tx=0x7f1f3c008040 comp rx=0 tx=0).stop 2023-10-16T11:22:31.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 msgr2=0x7f1f4c0c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:31.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c0c5b60 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f1f44012000 tx=0x7f1f44004ad0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 shutdown_connections 2023-10-16T11:22:31.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f1f3006b020 0x7f1f3006d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c0729b0 0x7f1f4c0c5b60 unknown :-1 s=CLOSED pgs=3456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f4c06ab40 0x7f1f4c1f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 --2- 172.21.15.158:0/2065549801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f4c06e5d0 0x7f1f4c1495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:31.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 >> 172.21.15.158:0/2065549801 conn(0x7f1f4c0d02b0 msgr2=0x7f1f4c072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:31.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 shutdown_connections 2023-10-16T11:22:31.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:31.530+0000 7f1f547b2700 1 -- 172.21.15.158:0/2065549801 wait complete. 2023-10-16T11:22:31.535 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:22:31.543 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-10-16T11:22:31.508610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":444544,"kb_used_data":28488,"kb_used_omap":12,"kb_used_meta":415923,"kb_avail":754530176,"statfs":{"total":773094113280,"available":772638900224,"internally_reserved":0,"allocated":29171712,"data_stored":16293480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":425905719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001733"},"pg_stats":[{"pgid":"1.0","version":"273'414","reported_seq":1498,"reported_epoch":274,"state":"active+clean","last_fresh":"2023-10-16T11:22:20.661621+0000","last_change":"2023-10-16T09:43:16.986947+0000","last_active":"2023-10-16T11:22:20.661621+0000","last_peered":"2023-10-16T11:22:20.661621+0000","last_clean":"2023-10-16T11:22:20.661621+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:22:20.661621+0000","last_undegraded":"2023-10-16T11:22:20.661621+0000","last_fullsized":"2023-10-16T11:22:20.661621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T09:43:15.875582+0000","last_clean_scrub_stamp":"2023-10-16T09:43:15.875582+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:32.309063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576020,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57384,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314456,"statfs":{"total":96636764160,"available":96578002944,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73799999999999999,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.68300000000000005,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.70399999999999996,"15min":0.66700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":9.516,"15min":9.516},"last":0.57599999999999996}]},{"osd":3,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67000000000000004,"15min":0.64000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.495},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.66100000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.59299999999999997,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.627},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.624,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67400000000000004,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.54700000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.64800000000000002,"15min":0.57899999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77800000000000002,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.68000000000000005,"15min":0.625},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.67800000000000005}]},{"osd":6,"last update":"Mon Oct 16 11:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.373},"max":{"1min":0.70999999999999996,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.69299999999999995,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72199999999999998,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.57999999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.80600000000000005,"15min":0.86299999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56499999999999995,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84299999999999997,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71899999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":2.387,"15min":2.387},"last":0.432}]},{"osd":2,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.314},"max":{"1min":0.78100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.37}]},{"osd":3,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59899999999999998,"15min":0.61599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88800000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81000000000000005,"5min":1.02,"15min":1.02},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.376}]},{"osd":5,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":3.335,"15min":3.335},"last":0.32100000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":2.524,"15min":2.524},"last":0.498},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.33900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.40799999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.26500000000000001,"5min":0.184,"15min":0.184},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36199999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.505},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.44},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.60199999999999998}]},{"osd":3,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.623}]},{"osd":4,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0309999999999999},"last":0.79800000000000004}]},{"osd":5,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.0389999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.629,"15min":0.66500000000000004},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.622,"15min":0.67200000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Oct 16 11:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64500000000000002,"15min":0.68999999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.74199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49072,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322768,"statfs":{"total":96636764160,"available":96586514432,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.38400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.55500000000000005,"15min":0.60299999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40300000000000002,"15min":0.439},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.44500000000000001,"5min":0.58399999999999996,"15min":0.77200000000000002},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50700000000000001,"5min":1.272,"15min":1.272},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.50600000000000001,"5min":0.64700000000000002,"15min":0.83199999999999996},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.60499999999999998,"15min":0.86699999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":1.351,"15min":1.351},"last":0.438}]},{"osd":4,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.64200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86499999999999999,"15min":1.1850000000000001},"last":0.626},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.45100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":1.391,"15min":1.391},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60599999999999998,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.874,"15min":1.089},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.216},"last":0.65300000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64300000000000002,"15min":0.68600000000000005},"min":{"1min":0.46600000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":1.038,"15min":1.1180000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Oct 16 11:21:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88900000000000001,"5min":0.97399999999999998,"15min":1.008},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.66900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.88500000000000001,"15min":1.0780000000000001},"last":0.747}]}]},{"osd":2,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35999999999999999,"15min":0.39100000000000001},"min":{"1min":0.23599999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.49399999999999999,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.308},{"interface":"front","average":{"1min":0.38,"5min":0.39900000000000002,"15min":0.435},"min":{"1min":0.248,"5min":0.193,"15min":0.193},"max":{"1min":0.56000000000000005,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.45200000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41999999999999998,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66600000000000004,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.36699999999999999,"15min":0.40699999999999997},"min":{"1min":0.27900000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.502,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.34499999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40200000000000002,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53200000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46400000000000002,"15min":0.49399999999999999},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70499999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.35999999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60099999999999998,"15min":0.64100000000000001},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72699999999999998,"5min":0.89900000000000002,"15min":1.056},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.625,"15min":0.66500000000000004},"min":{"1min":0.41399999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.751,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.55800000000000005}]},{"osd":5,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60999999999999999,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80500000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.63800000000000001,"15min":0.68300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78600000000000003,"5min":0.92700000000000005,"15min":1.028},"last":0.69699999999999995}]},{"osd":6,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60899999999999999,"15min":0.65300000000000002},"min":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77500000000000002,"5min":0.93899999999999995,"15min":1.022},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.64500000000000002,"15min":0.69299999999999995},"min":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77200000000000002,"5min":0.93700000000000006,"15min":1.004},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64800000000000002,"15min":0.69799999999999995},"min":{"1min":0.46700000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.91000000000000003,"15min":1.071},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86899999999999999,"5min":0.97199999999999998,"15min":1.0029999999999999},"last":0.747}]}]},{"osd":3,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.437,"5min":1.032,"15min":1.032},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35599999999999998,"15min":0.39800000000000002},"min":{"1min":0.27800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.51800000000000002,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":1.341,"15min":1.341},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.375,"5min":0.39600000000000002,"15min":0.433},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47699999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.47099999999999997},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58399999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55700000000000005,"5min":0.96599999999999997,"15min":1.024},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.40600000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86499999999999999,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89100000000000001,"5min":2.629,"15min":2.629},"last":0.85799999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.66300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70299999999999996,"15min":0.69899999999999995},"min":{"1min":0.45700000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.876}]},{"osd":6,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.68100000000000005,"15min":0.69099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85899999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.69799999999999995},"min":{"1min":0.44600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96899999999999997,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.73599999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70599999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90900000000000003,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.90100000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.76600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.60099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":1.0589999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55800000000000005,"15min":0.60799999999999998},"min":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.88200000000000001},"last":0.71199999999999997}]},{"osd":1,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54900000000000004,"15min":0.60099999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.90700000000000003},"last":0.629},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55000000000000004,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.77400000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55600000000000005,"15min":0.60899999999999999},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67600000000000005,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58699999999999997,"15min":0.626},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80500000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.58999999999999997}]},{"osd":3,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55000000000000004,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.751,"5min":0.79700000000000004,"15min":0.995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.499,"15min":0.5},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.86199999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71599999999999997,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.46899999999999997}]},{"osd":6,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.751,"5min":0.82899999999999996,"15min":1.0609999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81499999999999995,"5min":0.83899999999999997,"15min":1.115},"last":0.48899999999999999}]},{"osd":7,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.42199999999999999,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.50600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66536,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305304,"statfs":{"total":96636764160,"available":96568631296,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56000000000000005,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.871,"15min":0.88200000000000001},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.58399999999999996},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73199999999999998,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.57699999999999996},"min":{"1min":0.311,"5min":0.309,"15min":0.309},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.747,"5min":0.82099999999999995,"15min":0.89400000000000002},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.59799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.54700000000000004}]},{"osd":4,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76400000000000001,"5min":0.80900000000000005,"15min":0.84799999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.33500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.86399999999999999},"last":0.34899999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.753,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.85899999999999999,"15min":0.878},"last":0.439}]},{"osd":7,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84699999999999998,"5min":0.871,"15min":0.871},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.89300000000000002},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:22:31.547 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-16T09:43:15.875582+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=9, tm_min=43, tm_sec=15, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=26, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T11:22:31.548 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-16T11:22:47.291 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:47.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:51.549 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T11:22:51.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.682+0000 7fd31ae8f700 1 Processor -- start 2023-10-16T11:22:51.684 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.682+0000 7fd31ae8f700 1 -- start start 2023-10-16T11:22:51.689 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd31ae8f700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd3140cdee0 0x7fd31415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:51.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd31ae8f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140b6e10 0x7fd3141521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:51.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd31ae8f700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31415c5d0 con 0x7fd3140ce2c0 2023-10-16T11:22:51.690 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd31ae8f700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31415d2c0 con 0x7fd3140cdee0 2023-10-16T11:22:51.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd31ae8f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31415c8b0 con 0x7fd3140b6e10 2023-10-16T11:22:51.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd318c2b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd3140cdee0 0x7fd31415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:51.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd31942c700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd3140ce2c0 0x7fd3140cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45412/0 (socket says 172.21.15.158:45412) 2023-10-16T11:22:51.691 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd31942c700 1 -- 172.21.15.158:0/1471160506 learned_addr learned my addr 172.21.15.158:0/1471160506 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:51.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd313fff700 1 --2- 172.21.15.158:0/1471160506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140b6e10 0x7fd3141521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:51.692 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd318c2b700 1 -- 172.21.15.158:0/1471160506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140b6e10 msgr2=0x7fd3141521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:51.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd318c2b700 1 --2- 172.21.15.158:0/1471160506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140b6e10 0x7fd3141521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd318c2b700 1 -- 172.21.15.158:0/1471160506 >> v1:172.21.15.158:6789/0 conn(0x7fd3140ce2c0 legacy=0x7fd3140cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:51.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd318c2b700 1 -- 172.21.15.158:0/1471160506 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31415cb90 con 0x7fd3140cdee0 2023-10-16T11:22:51.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd313fff700 1 --2- 172.21.15.158:0/1471160506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140b6e10 0x7fd3141521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:51.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd3137fe700 1 -- 172.21.15.158:0/1471160506 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635254453 0 0) 0x7fd31415c5d0 con 0x7fd3140ce2c0 2023-10-16T11:22:51.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.683+0000 7fd318c2b700 1 --2- 172.21.15.158:0/1471160506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cdee0 0x7fd31415bd10 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7fd308009840 tx=0x7fd308009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf90898af63db22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:51.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd3137fe700 1 -- 172.21.15.158:0/1471160506 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308011830 con 0x7fd3140cdee0 2023-10-16T11:22:51.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd3137fe700 1 -- 172.21.15.158:0/1471160506 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd308011990 con 0x7fd3140cdee0 2023-10-16T11:22:51.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd3137fe700 1 -- 172.21.15.158:0/1471160506 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308011d20 con 0x7fd3140cdee0 2023-10-16T11:22:51.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 -- 172.21.15.158:0/1471160506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cdee0 msgr2=0x7fd31415bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:51.695 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/1471160506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cdee0 0x7fd31415bd10 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7fd308009840 tx=0x7fd308009b50 comp rx=0 tx=0).stop 2023-10-16T11:22:51.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 -- 172.21.15.158:0/1471160506 shutdown_connections 2023-10-16T11:22:51.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/1471160506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cdee0 0x7fd31415bd10 unknown :-1 s=CLOSED pgs=3458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/1471160506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140b6e10 0x7fd3141521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.696 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 -- 172.21.15.158:0/1471160506 >> 172.21.15.158:0/1471160506 conn(0x7fd31406a8d0 msgr2=0x7fd31406ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:51.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 -- 172.21.15.158:0/1471160506 shutdown_connections 2023-10-16T11:22:51.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.684+0000 7fd31ae8f700 1 -- 172.21.15.158:0/1471160506 wait complete. 2023-10-16T11:22:51.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.685+0000 7fd31ae8f700 1 Processor -- start 2023-10-16T11:22:51.697 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.685+0000 7fd31ae8f700 1 -- start start 2023-10-16T11:22:51.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd31ae8f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140b6e10 0x7fd314150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:51.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd31ae8f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3140cdee0 0x7fd314154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:51.698 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd31ae8f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140ce2c0 0x7fd3141548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:51.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd31ae8f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141540c0 con 0x7fd3140b6e10 2023-10-16T11:22:51.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd31ae8f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd314153820 con 0x7fd3140cdee0 2023-10-16T11:22:51.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd31ae8f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd314153b00 con 0x7fd3140ce2c0 2023-10-16T11:22:51.699 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd318c2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140b6e10 0x7fd314150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:51.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd318c2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140b6e10 0x7fd314150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45238/0 (socket says 172.21.15.158:45238) 2023-10-16T11:22:51.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.686+0000 7fd318c2b700 1 -- 172.21.15.158:0/3346345607 learned_addr learned my addr 172.21.15.158:0/3346345607 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:51.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd31942c700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140ce2c0 0x7fd3141548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:51.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd313fff700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3140cdee0 0x7fd314154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:51.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd318c2b700 1 -- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140ce2c0 msgr2=0x7fd3141548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:51.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd318c2b700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140ce2c0 0x7fd3141548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.701 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd318c2b700 1 -- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3140cdee0 msgr2=0x7fd314154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:51.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd318c2b700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3140cdee0 0x7fd314154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd318c2b700 1 -- 172.21.15.158:0/3346345607 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3080094a0 con 0x7fd3140b6e10 2023-10-16T11:22:51.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd318c2b700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140b6e10 0x7fd314150a80 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fd308000c00 tx=0x7fd308004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:51.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd3117fa700 1 -- 172.21.15.158:0/3346345607 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3080045f0 con 0x7fd3140b6e10 2023-10-16T11:22:51.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd3117fa700 1 -- 172.21.15.158:0/3346345607 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30801f070 con 0x7fd3140b6e10 2023-10-16T11:22:51.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd314153de0 con 0x7fd3140b6e10 2023-10-16T11:22:51.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd3117fa700 1 -- 172.21.15.158:0/3346345607 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308003ae0 con 0x7fd3140b6e10 2023-10-16T11:22:51.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.687+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd314150fc0 con 0x7fd3140b6e10 2023-10-16T11:22:51.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.688+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd314151150 con 0x7fd3140b6e10 2023-10-16T11:22:51.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.689+0000 7fd3117fa700 1 -- 172.21.15.158:0/3346345607 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd308003c40 con 0x7fd3140b6e10 2023-10-16T11:22:51.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.689+0000 7fd3117fa700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fd2f806b020 0x7fd2f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:51.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.689+0000 7fd3117fa700 1 -- 172.21.15.158:0/3346345607 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd308021070 con 0x7fd3140b6e10 2023-10-16T11:22:51.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.689+0000 7fd313fff700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fd2f806b020 0x7fd2f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:51.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.689+0000 7fd313fff700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fd2f806b020 0x7fd2f806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd300006fd0 tx=0x7fd300008040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:51.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.694+0000 7fd3117fa700 1 -- 172.21.15.158:0/3346345607 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3080561e0 con 0x7fd3140b6e10 2023-10-16T11:22:51.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.882+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 --> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3140ceee0 con 0x7fd2f806b020 2023-10-16T11:22:51.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.884+0000 7fd3117fa700 1 -- 172.21.15.158:0/3346345607 <== mgr.13977 v2:172.21.15.158:6832/899784159 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50732 (secure 0 0 0) 0x7fd30000c040 con 0x7fd2f806b020 2023-10-16T11:22:51.887 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:51.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fd2f806b020 msgr2=0x7fd2f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:51.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fd2f806b020 0x7fd2f806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd300006fd0 tx=0x7fd300008040 comp rx=0 tx=0).stop 2023-10-16T11:22:51.889 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140b6e10 msgr2=0x7fd314150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:51.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140b6e10 0x7fd314150a80 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fd308000c00 tx=0x7fd308004750 comp rx=0 tx=0).stop 2023-10-16T11:22:51.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 shutdown_connections 2023-10-16T11:22:51.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7fd2f806b020 0x7fd2f806d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3140ce2c0 0x7fd3141548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140b6e10 0x7fd314150a80 unknown :-1 s=CLOSED pgs=3459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 --2- 172.21.15.158:0/3346345607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3140cdee0 0x7fd314154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:51.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 >> 172.21.15.158:0/3346345607 conn(0x7fd31406a8d0 msgr2=0x7fd31415d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:51.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 shutdown_connections 2023-10-16T11:22:51.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:51.887+0000 7fd31ae8f700 1 -- 172.21.15.158:0/3346345607 wait complete. 2023-10-16T11:22:51.892 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-16T11:22:51.901 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T11:22:51.512744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":444528,"kb_used_data":28488,"kb_used_omap":12,"kb_used_meta":415923,"kb_avail":754530192,"statfs":{"total":773094113280,"available":772638916608,"internally_reserved":0,"allocated":29171712,"data_stored":16293480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12745,"internal_metadata":425905719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002624"},"pg_stats":[{"pgid":"1.0","version":"275'415","reported_seq":1508,"reported_epoch":275,"state":"active+clean","last_fresh":"2023-10-16T11:22:35.807001+0000","last_change":"2023-10-16T11:22:35.807065+0000","last_active":"2023-10-16T11:22:35.807001+0000","last_peered":"2023-10-16T11:22:35.807001+0000","last_clean":"2023-10-16T11:22:35.807001+0000","last_became_active":"2023-10-16T09:43:16.985831+0000","last_became_peered":"2023-10-16T09:43:16.985831+0000","last_unstale":"2023-10-16T11:22:35.807001+0000","last_undegraded":"2023-10-16T11:22:35.807001+0000","last_fullsized":"2023-10-16T11:22:35.807001+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'415","last_scrub_stamp":"2023-10-16T11:22:35.806911+0000","last_deep_scrub":"275'415","last_deep_scrub_stamp":"2023-10-16T11:22:35.806911+0000","last_clean_scrub_stamp":"2023-10-16T11:22:35.806911+0000","objects_scrubbed":2,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:24.927234+0000","scrub_duration":0.0055593109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2708,"num_read_kb":5662,"num_write":699,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576024,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57380,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314460,"statfs":{"total":96636764160,"available":96578007040,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64000000000000001,"15min":0.60299999999999998},"min":{"1min":0.317,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.631,"5min":9.4359999999999999,"15min":9.4359999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65800000000000003,"15min":0.624},"min":{"1min":0.34399999999999997,"5min":0.31,"15min":0.31},"max":{"1min":1.843,"5min":9.3059999999999992,"15min":9.3059999999999992},"last":0.496}]},{"osd":1,"last update":"Mon Oct 16 11:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.66000000000000003,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69899999999999995,"5min":9.3610000000000007,"15min":9.3610000000000007},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.68500000000000005,"15min":0.66800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.30599999999999999},"max":{"1min":1.5569999999999999,"5min":9.3960000000000008,"15min":9.3960000000000008},"last":0.73399999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.66200000000000003},"min":{"1min":0.38500000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.875,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.72099999999999997,"15min":0.68799999999999994},"min":{"1min":0.39400000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.7889999999999999,"5min":9.516,"15min":9.516},"last":0.64200000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.66200000000000003,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76100000000000001,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.625},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.752,"15min":0.70999999999999996},"min":{"1min":0.45700000000000002,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":1.6799999999999999,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.78700000000000003}]},{"osd":4,"last update":"Mon Oct 16 11:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.626,"15min":0.54800000000000004},"min":{"1min":0.32000000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.986,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.63400000000000001,"15min":0.56699999999999995},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75900000000000001,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Oct 16 11:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68600000000000005,"15min":0.58999999999999997},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.7210000000000001,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.71599999999999997,"15min":0.63500000000000001},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.0249999999999999,"5min":9.5690000000000008,"15min":9.5690000000000008},"last":0.75600000000000001}]},{"osd":6,"last update":"Mon Oct 16 11:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73899999999999999,"15min":0.65000000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.373},"max":{"1min":1.927,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.70999999999999996,"15min":0.64300000000000002},"min":{"1min":0.439,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96799999999999997,"5min":9.5960000000000001,"15min":9.5960000000000001},"last":0.84099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.57199999999999995},"min":{"1min":0.29299999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66400000000000003,"5min":0.80600000000000005,"15min":0.81000000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.60099999999999998},"min":{"1min":0.38100000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.66400000000000003,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Oct 16 11:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54900000000000004,"15min":0.59099999999999997},"min":{"1min":0.42399999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.74299999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.55900000000000005,"15min":0.60399999999999998},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.755,"5min":2.387,"15min":2.387},"last":0.628}]},{"osd":2,"last update":"Mon Oct 16 11:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53700000000000003,"15min":0.58699999999999997},"min":{"1min":0.39800000000000002,"5min":0.34699999999999998,"15min":0.314},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.56000000000000005,"15min":0.60599999999999998},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":0.871,"15min":0.871},"last":0.60899999999999999}]},{"osd":3,"last update":"Mon Oct 16 11:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.41599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81399999999999995,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57199999999999995,"15min":0.61199999999999999},"min":{"1min":0.36099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.497}]},{"osd":4,"last update":"Mon Oct 16 11:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77200000000000002,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56899999999999995,"15min":0.55300000000000005},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.81499999999999995,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.84699999999999998,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.474,"5min":3.335,"15min":3.335},"last":0.65400000000000003}]},{"osd":7,"last update":"Mon Oct 16 11:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60799999999999998,"15min":0.59599999999999997},"min":{"1min":0.39200000000000002,"5min":0.35999999999999999,"15min":0.35399999999999998},"max":{"1min":0.86299999999999999,"5min":2.524,"15min":2.524},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60899999999999999,"15min":0.61199999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.89000000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39200000000000002,"15min":0.40699999999999997},"min":{"1min":0.25800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51100000000000001,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.41899999999999998,"15min":0.44500000000000001},"min":{"1min":0.31900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.61699999999999999,"5min":1.532,"15min":1.532},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Oct 16 11:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.36899999999999999,"15min":0.39700000000000002},"min":{"1min":0.28999999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.57499999999999996,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.39800000000000002,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.24399999999999999},"max":{"1min":0.63,"5min":1.216,"15min":1.216},"last":0.54000000000000004}]},{"osd":3,"last update":"Mon Oct 16 11:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.42999999999999999,"15min":0.46200000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69699999999999995,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.41799999999999998,"15min":0.438},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.67600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Oct 16 11:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.61699999999999999,"15min":0.64600000000000002},"min":{"1min":0.39300000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.60199999999999998,"15min":0.65100000000000002},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.90400000000000003,"5min":0.96899999999999997,"15min":1.0309999999999999},"last":0.876}]},{"osd":5,"last update":"Mon Oct 16 11:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.67300000000000004},"min":{"1min":0.45000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.86199999999999999,"5min":1.028,"15min":1.0389999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.65600000000000003,"15min":0.70099999999999996},"min":{"1min":0.46700000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92000000000000004,"5min":1.266,"15min":1.266},"last":0.85899999999999999}]},{"osd":6,"last update":"Mon Oct 16 11:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.626,"15min":0.66400000000000003},"min":{"1min":0.499,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93500000000000005,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60899999999999999,"15min":0.66600000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.83899999999999997,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.83299999999999996}]},{"osd":7,"last update":"Mon Oct 16 11:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65100000000000002,"15min":0.69199999999999995},"min":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84399999999999997,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64000000000000001,"15min":0.68400000000000005},"min":{"1min":0.441,"5min":0.438,"15min":0.438},"max":{"1min":0.877,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.72099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834576025,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49060,"kb_used_data":3996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322780,"statfs":{"total":96636764160,"available":96586526720,"internally_reserved":0,"allocated":4091904,"data_stored":2479461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 11:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.374,"15min":0.39000000000000001},"min":{"1min":0.25,"5min":0.217,"15min":0.217},"max":{"1min":3.161,"5min":3.161,"15min":3.161},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.41899999999999998,"15min":0.443},"min":{"1min":0.30299999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Oct 16 11:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.47399999999999998,"15min":0.45000000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.4139999999999997,"5min":6.4139999999999997,"15min":6.4139999999999997},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Oct 16 11:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.40999999999999998,"15min":0.432},"min":{"1min":0.27100000000000002,"5min":0.216,"15min":0.216},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.28999999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.508,"5min":3.508,"15min":3.508},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Oct 16 11:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.61899999999999999,"15min":0.64800000000000002},"min":{"1min":0.35799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.42299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.8690000000000002,"5min":3.8690000000000002,"15min":3.8690000000000002},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Oct 16 11:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.628,"15min":0.65000000000000002},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.6360000000000001,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.63500000000000001,"15min":0.66000000000000003},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.7629999999999999,"5min":3.7629999999999999,"15min":3.7629999999999999},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Oct 16 11:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.66700000000000004,"15min":0.69299999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.67700000000000005,"15min":0.69099999999999995},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.7839999999999998,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.622}]},{"osd":7,"last update":"Mon Oct 16 11:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.66200000000000003,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.734,"5min":3.734,"15min":3.734},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.64600000000000002,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":0.44700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35999999999999999,"15min":0.39100000000000001},"min":{"1min":0.23599999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.49399999999999999,"5min":0.60799999999999998,"15min":0.86599999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.39900000000000002,"15min":0.435},"min":{"1min":0.248,"5min":0.193,"15min":0.193},"max":{"1min":0.56000000000000005,"5min":0.63300000000000001,"15min":0.82399999999999995},"last":0.372}]},{"osd":1,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41999999999999998,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66600000000000004,"5min":0.72699999999999998,"15min":0.75700000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.36699999999999999,"15min":0.40699999999999997},"min":{"1min":0.27900000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.502,"5min":0.67200000000000004,"15min":0.69599999999999995},"last":0.40500000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40200000000000002,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53200000000000003,"5min":0.69799999999999995,"15min":0.73499999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46400000000000002,"15min":0.49399999999999999},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70499999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42599999999999999}]},{"osd":4,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60099999999999998,"15min":0.64100000000000001},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72699999999999998,"5min":0.89900000000000002,"15min":1.056},"last":0.628},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.625,"15min":0.66500000000000004},"min":{"1min":0.41399999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.751,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.77300000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60999999999999999,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80500000000000005,"5min":0.93600000000000005,"15min":1.0609999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.63800000000000001,"15min":0.68300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78600000000000003,"5min":0.92700000000000005,"15min":1.028},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60899999999999999,"15min":0.65300000000000002},"min":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77500000000000002,"5min":0.93899999999999995,"15min":1.022},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.64500000000000002,"15min":0.69299999999999995},"min":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77200000000000002,"5min":0.93700000000000006,"15min":1.004},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Oct 16 11:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64800000000000002,"15min":0.69799999999999995},"min":{"1min":0.46700000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.91000000000000003,"15min":1.071},"last":0.746},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86899999999999999,"5min":0.97199999999999998,"15min":1.0029999999999999},"last":0.85599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56804,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.437,"5min":1.032,"15min":1.032},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35599999999999998,"15min":0.39800000000000002},"min":{"1min":0.27800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.51800000000000002,"5min":0.66800000000000004,"15min":1.0029999999999999},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":1.341,"15min":1.341},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.375,"5min":0.39600000000000002,"15min":0.433},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47699999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.39400000000000002}]},{"osd":2,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.47099999999999997},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58399999999999996,"5min":0.69999999999999996,"15min":0.96299999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55700000000000005,"5min":0.96599999999999997,"15min":1.024},"last":0.439}]},{"osd":4,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.40600000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86499999999999999,"5min":0.98099999999999998,"15min":0.98799999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89100000000000001,"5min":2.629,"15min":2.629},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.66300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70299999999999996,"15min":0.69899999999999995},"min":{"1min":0.45700000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.68100000000000005,"15min":0.69099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85899999999999999,"5min":5.8399999999999999,"15min":5.8399999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.69799999999999995},"min":{"1min":0.44600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96899999999999997,"5min":4.4409999999999998,"15min":4.4409999999999998},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70599999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90900000000000003,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.44400000000000001,"15min":0.42599999999999999},"max":{"1min":0.90100000000000002,"5min":1.006,"15min":1.1899999999999999},"last":0.77300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52648,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319192,"statfs":{"total":96636764160,"available":96582852608,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.60099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":1.0589999999999999},"last":0.624},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55800000000000005,"15min":0.60799999999999998},"min":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.88200000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54900000000000004,"15min":0.60099999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.90700000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55000000000000004,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.315},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.68999999999999995}]},{"osd":2,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55600000000000005,"15min":0.60899999999999999},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67600000000000005,"5min":0.83599999999999997,"15min":1.1020000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58699999999999997,"15min":0.626},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80500000000000005,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55000000000000004,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.751,"5min":0.79700000000000004,"15min":0.995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.499,"15min":0.5},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.86199999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71599999999999997,"5min":0.79700000000000004,"15min":0.81100000000000005},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.751,"5min":0.82899999999999996,"15min":1.0609999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81499999999999995,"5min":0.83899999999999997,"15min":1.115},"last":0.496}]},{"osd":7,"last update":"Mon Oct 16 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":0.83899999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.42199999999999999,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.88300000000000001,"15min":0.91000000000000003},"last":0.72299999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66536,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305304,"statfs":{"total":96636764160,"available":96568631296,"internally_reserved":0,"allocated":3497984,"data_stored":1889093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56000000000000005,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.89300000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.871,"15min":0.88200000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.58399999999999996},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73199999999999998,"5min":0.84599999999999997,"15min":0.91300000000000003},"last":0.625}]},{"osd":2,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.57699999999999996},"min":{"1min":0.311,"5min":0.309,"15min":0.309},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.84899999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.747,"5min":0.82099999999999995,"15min":0.89400000000000002},"last":0.64500000000000002}]},{"osd":3,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.88600000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.59799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76400000000000001,"5min":0.80900000000000005,"15min":0.84799999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.33500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.86399999999999999},"last":0.436}]},{"osd":6,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.753,"5min":0.76900000000000002,"15min":0.82499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.85899999999999999,"15min":0.878},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Oct 16 11:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84699999999999998,"5min":0.871,"15min":0.871},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.89300000000000002},"last":0.69499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T11:22:51.903 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-16T11:22:52.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.022+0000 7f5c9679b700 1 Processor -- start 2023-10-16T11:22:52.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.022+0000 7f5c9679b700 1 -- start start 2023-10-16T11:22:52.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.022+0000 7f5c9679b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c9015d770 0x7f5c9015db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.022+0000 7f5c9679b700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5c900cf680 0x7f5c900cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.022+0000 7f5c9679b700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900cc260 con 0x7f5c900cbe00 2023-10-16T11:22:52.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.022+0000 7f5c9679b700 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c900caa80 con 0x7f5c900cf680 2023-10-16T11:22:52.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.022+0000 7f5c9679b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900cad60 con 0x7f5c9015d770 2023-10-16T11:22:52.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c94d38700 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5c900cf680 0x7f5c900cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c8f7fe700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5c900cbe00 0x7f5c900cd7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45422/0 (socket says 172.21.15.158:45422) 2023-10-16T11:22:52.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c8f7fe700 1 -- 172.21.15.158:0/159857603 learned_addr learned my addr 172.21.15.158:0/159857603 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:52.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c8ffff700 1 --2- 172.21.15.158:0/159857603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c9015d770 0x7f5c9015db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c94d38700 1 -- 172.21.15.158:0/159857603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c9015d770 msgr2=0x7f5c9015db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:52.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c94d38700 1 --2- 172.21.15.158:0/159857603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c9015d770 0x7f5c9015db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c94d38700 1 -- 172.21.15.158:0/159857603 >> v1:172.21.15.158:6789/0 conn(0x7f5c900cbe00 legacy=0x7f5c900cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c8effd700 1 -- 172.21.15.158:0/159857603 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255544474 0 0) 0x7f5c900cc260 con 0x7f5c900cbe00 2023-10-16T11:22:52.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c94d38700 1 -- 172.21.15.158:0/159857603 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c900cb040 con 0x7f5c900cf680 2023-10-16T11:22:52.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c94d38700 1 --2- 172.21.15.158:0/159857603 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c900cf680 0x7f5c900cfaf0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009d90 tx=0x7f5c8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b982f546a3610701 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:52.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c8effd700 1 -- 172.21.15.158:0/159857603 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8800e070 con 0x7f5c900cf680 2023-10-16T11:22:52.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c8effd700 1 -- 172.21.15.158:0/159857603 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8800bb20 con 0x7f5c900cf680 2023-10-16T11:22:52.031 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.023+0000 7f5c8effd700 1 -- 172.21.15.158:0/159857603 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88012430 con 0x7f5c900cf680 2023-10-16T11:22:52.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 -- 172.21.15.158:0/159857603 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c900cf680 msgr2=0x7f5c900cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 --2- 172.21.15.158:0/159857603 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c900cf680 0x7f5c900cfaf0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009d90 tx=0x7f5c8800b330 comp rx=0 tx=0).stop 2023-10-16T11:22:52.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 -- 172.21.15.158:0/159857603 shutdown_connections 2023-10-16T11:22:52.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 --2- 172.21.15.158:0/159857603 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c900cf680 0x7f5c900cfaf0 unknown :-1 s=CLOSED pgs=3461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 --2- 172.21.15.158:0/159857603 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c9015d770 0x7f5c9015db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 -- 172.21.15.158:0/159857603 >> 172.21.15.158:0/159857603 conn(0x7f5c90099a10 msgr2=0x7f5c900d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:52.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 -- 172.21.15.158:0/159857603 shutdown_connections 2023-10-16T11:22:52.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.024+0000 7f5c9679b700 1 -- 172.21.15.158:0/159857603 wait complete. 2023-10-16T11:22:52.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.025+0000 7f5c9679b700 1 Processor -- start 2023-10-16T11:22:52.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.025+0000 7f5c9679b700 1 -- start start 2023-10-16T11:22:52.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c9679b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c900cbe00 0x7f5c901f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c9679b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900cf680 0x7f5c90152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c9679b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c9015d770 0x7f5c900c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c9679b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c901f64a0 con 0x7f5c9015d770 2023-10-16T11:22:52.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c9679b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c90152e10 con 0x7f5c900cf680 2023-10-16T11:22:52.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c9679b700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900c7870 con 0x7f5c900cbe00 2023-10-16T11:22:52.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c9015d770 0x7f5c900c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c9015d770 0x7f5c900c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45258/0 (socket says 172.21.15.158:45258) 2023-10-16T11:22:52.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 -- 172.21.15.158:0/4006785965 learned_addr learned my addr 172.21.15.158:0/4006785965 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:52.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c8ffff700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c900cbe00 0x7f5c901f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 -- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c900cbe00 msgr2=0x7f5c901f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c900cbe00 0x7f5c901f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 -- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900cf680 msgr2=0x7f5c90152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c8f7fe700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900cf680 0x7f5c90152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900cf680 0x7f5c90152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c94d38700 1 -- 172.21.15.158:0/4006785965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88009a70 con 0x7f5c9015d770 2023-10-16T11:22:52.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.026+0000 7f5c8f7fe700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900cf680 0x7f5c90152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T11:22:52.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.027+0000 7f5c94d38700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c9015d770 0x7f5c900c9e90 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f5c900cc3f0 tx=0x7f5c88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:52.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.027+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88012430 con 0x7f5c9015d770 2023-10-16T11:22:52.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.027+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c88020070 con 0x7f5c9015d770 2023-10-16T11:22:52.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.027+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88004280 con 0x7f5c9015d770 2023-10-16T11:22:52.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.027+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c900c7b50 con 0x7f5c9015d770 2023-10-16T11:22:52.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.027+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c901f69d0 con 0x7f5c9015d770 2023-10-16T11:22:52.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.028+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c900681b0 con 0x7f5c9015d770 2023-10-16T11:22:52.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.028+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5c880043e0 con 0x7f5c9015d770 2023-10-16T11:22:52.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.029+0000 7f5c8cff9700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5c7c06b020 0x7f5c7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.029+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5c88022070 con 0x7f5c9015d770 2023-10-16T11:22:52.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.029+0000 7f5c8ffff700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5c7c06b020 0x7f5c7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.029+0000 7f5c8ffff700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5c7c06b020 0x7f5c7c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5c80009b40 tx=0x7f5c80009210 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:52.045 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.033+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5c88056480 con 0x7f5c9015d770 2023-10-16T11:22:52.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.224+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f5c900ce420 con 0x7f5c9015d770 2023-10-16T11:22:52.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.227+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) v1 ==== 123+0+0 (secure 0 0 0) 0x7f5c88059aa0 con 0x7f5c9015d770 2023-10-16T11:22:52.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.228+0000 7f5c8cff9700 1 -- 172.21.15.158:0/4006785965 <== mon.0 v2:172.21.15.158:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5c88056480 con 0x7f5c9015d770 2023-10-16T11:22:52.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5c7c06b020 msgr2=0x7f5c7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5c7c06b020 0x7f5c7c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5c80009b40 tx=0x7f5c80009210 comp rx=0 tx=0).stop 2023-10-16T11:22:52.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c9015d770 msgr2=0x7f5c900c9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c9015d770 0x7f5c900c9e90 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f5c900cc3f0 tx=0x7f5c88004ad0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 shutdown_connections 2023-10-16T11:22:52.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5c7c06b020 0x7f5c7c06d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c9015d770 0x7f5c900c9e90 unknown :-1 s=CLOSED pgs=3462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c900cbe00 0x7f5c901f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 --2- 172.21.15.158:0/4006785965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900cf680 0x7f5c90152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 >> 172.21.15.158:0/4006785965 conn(0x7f5c90099a10 msgr2=0x7f5c900c6930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:52.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.230+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 shutdown_connections 2023-10-16T11:22:52.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.231+0000 7f5c9679b700 1 -- 172.21.15.158:0/4006785965 wait complete. 2023-10-16T11:22:52.245 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T11:22:52.246 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T11:22:52.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.366+0000 7f5d8b4e3700 1 Processor -- start 2023-10-16T11:22:52.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.366+0000 7f5d8b4e3700 1 -- start start 2023-10-16T11:22:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d8b4e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 0x7f5d8415dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d8b4e3700 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8406e7a0 con 0x7f5d8415d7e0 2023-10-16T11:22:52.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d8b4e3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8406ea80 con 0x7f5d84150b80 2023-10-16T11:22:52.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d8b4e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8406f420 con 0x7f5d840cb640 2023-10-16T11:22:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d88a7e700 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5d8415d7e0 0x7f5d840ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:45438/0 (socket says 172.21.15.158:45438) 2023-10-16T11:22:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d88a7e700 1 -- 172.21.15.158:0/755203205 learned_addr learned my addr 172.21.15.158:0/755203205 (peer_addr_for_me v1:172.21.15.158:0/0) 2023-10-16T11:22:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 <== mon.0 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343484698 0 0) 0x7f5d8406e7a0 con 0x7f5d8415d7e0 2023-10-16T11:22:52.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d700035b0 con 0x7f5d8415d7e0 2023-10-16T11:22:52.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d89a80700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 0x7f5d8415dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 <== mon.0 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2375913476 0 0) 0x7f5d700035b0 con 0x7f5d8415d7e0 2023-10-16T11:22:52.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 msgr2=0x7f5d8415dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:52.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 0x7f5d8415dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 >> v1:172.21.15.172:6789/0 conn(0x7f5d84150b80 legacy=0x7f5d8406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8406ed60 con 0x7f5d8415d7e0 2023-10-16T11:22:52.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 <== mon.0 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1478224844 0 0) 0x7f5d74001670 con 0x7f5d8415d7e0 2023-10-16T11:22:52.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.367+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 >> v1:172.21.15.158:6789/0 conn(0x7f5d8415d7e0 legacy=0x7f5d840ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d70003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 0x7f5d70004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70004d90 0x7f5d70005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8406f420 con 0x7f5d70003a30 2023-10-16T11:22:52.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7000bb70 con 0x7f5d70004d90 2023-10-16T11:22:52.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7000bd30 con 0x7f5d840cb640 2023-10-16T11:22:52.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d8927f700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d70003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d89a80700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70004d90 0x7f5d70005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d88a7e700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 0x7f5d70004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d8927f700 1 -- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 msgr2=0x7f5d70004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:52.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d8927f700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 0x7f5d70004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d8927f700 1 -- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70004d90 msgr2=0x7f5d70005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T11:22:52.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d8927f700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70004d90 0x7f5d70005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d8927f700 1 -- 172.21.15.158:0/755203205 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700035b0 con 0x7f5d70003a30 2023-10-16T11:22:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d88a7e700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cb640 0x7f5d70004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T11:22:52.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d8927f700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d70003e50 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f5d800017c0 tx=0x7f5d70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5523e66cfb4bc070 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:52.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d800052d0 con 0x7f5d70003a30 2023-10-16T11:22:52.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5d80005430 con 0x7f5d70003a30 2023-10-16T11:22:52.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.368+0000 7f5d7bfff700 1 -- 172.21.15.158:0/755203205 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d800038e0 con 0x7f5d70003a30 2023-10-16T11:22:52.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.369+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/755203205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 msgr2=0x7f5d70003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.369+0000 7f5d8b4e3700 1 --2- 172.21.15.158:0/755203205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d70003e50 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f5d800017c0 tx=0x7f5d70005880 comp rx=0 tx=0).stop 2023-10-16T11:22:52.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.369+0000 7f5d89a80700 1 -- 172.21.15.158:0/755203205 reap_dead start 2023-10-16T11:22:52.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.369+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/755203205 shutdown_connections 2023-10-16T11:22:52.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.369+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/755203205 >> 172.21.15.158:0/755203205 conn(0x7f5d840cc5a0 msgr2=0x7f5d840cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:52.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.369+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/755203205 shutdown_connections 2023-10-16T11:22:52.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.369+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/755203205 wait complete. 2023-10-16T11:22:52.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.370+0000 7f5d8b4e3700 1 Processor -- start 2023-10-16T11:22:52.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.370+0000 7f5d8b4e3700 1 -- start start 2023-10-16T11:22:52.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8b4e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d8406b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8b4e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d841568a0 0x7f5d840c2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8b4e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840b0690 0x7f5d840b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8b4e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8415e1d0 con 0x7f5d70003a30 2023-10-16T11:22:52.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8b4e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8406b6c0 con 0x7f5d841568a0 2023-10-16T11:22:52.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8b4e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840b1050 con 0x7f5d840b0690 2023-10-16T11:22:52.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8927f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d8406b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.371+0000 7f5d8927f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d8406b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:45274/0 (socket says 172.21.15.158:45274) 2023-10-16T11:22:52.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8927f700 1 -- 172.21.15.158:0/603746573 learned_addr learned my addr 172.21.15.158:0/603746573 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-10-16T11:22:52.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d88a7e700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d841568a0 0x7f5d840c2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d89a80700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840b0690 0x7f5d840b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8927f700 1 -- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840b0690 msgr2=0x7f5d840b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8927f700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840b0690 0x7f5d840b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8927f700 1 -- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d841568a0 msgr2=0x7f5d840c2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8927f700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d841568a0 0x7f5d840c2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8927f700 1 -- 172.21.15.158:0/603746573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8414d560 con 0x7f5d70003a30 2023-10-16T11:22:52.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8927f700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d8406b180 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f5d80000c00 tx=0x7f5d80001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d79ffb700 1 -- 172.21.15.158:0/603746573 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80007450 con 0x7f5d70003a30 2023-10-16T11:22:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d79ffb700 1 -- 172.21.15.158:0/603746573 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5d800075b0 con 0x7f5d70003a30 2023-10-16T11:22:52.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d84156e40 con 0x7f5d70003a30 2023-10-16T11:22:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d79ffb700 1 -- 172.21.15.158:0/603746573 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d800038e0 con 0x7f5d70003a30 2023-10-16T11:22:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.372+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d84157400 con 0x7f5d70003a30 2023-10-16T11:22:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.373+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84151500 con 0x7f5d70003a30 2023-10-16T11:22:52.397 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.374+0000 7f5d79ffb700 1 -- 172.21.15.158:0/603746573 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 175) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5d80003a40 con 0x7f5d70003a30 2023-10-16T11:22:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.375+0000 7f5d79ffb700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5d6806b020 0x7f5d6806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T11:22:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.375+0000 7f5d79ffb700 1 -- 172.21.15.158:0/603746573 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5d8008ad40 con 0x7f5d70003a30 2023-10-16T11:22:52.398 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.375+0000 7f5d88a7e700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5d6806b020 0x7f5d6806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T11:22:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.375+0000 7f5d88a7e700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5d6806b020 0x7f5d6806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d74000f60 tx=0x7f5d74009040 comp rx=0 tx=0).ready entity=mgr.13977 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T11:22:52.399 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.378+0000 7f5d79ffb700 1 -- 172.21.15.158:0/603746573 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d80055b80 con 0x7f5d70003a30 2023-10-16T11:22:52.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.614+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d841512f0 con 0x7f5d70003a30 2023-10-16T11:22:52.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.615+0000 7f5d79ffb700 1 -- 172.21.15.158:0/603746573 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f5d8002cea0 con 0x7f5d70003a30 2023-10-16T11:22:52.617 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-16T11:22:52.618 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":156,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1253298118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6835","nonce":1253298118},{"type":"v1","addr":"172.21.15.158:6837","nonce":1253298118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":13731,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6836/2914884530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2914884530},{"type":"v1","addr":"172.21.15.158:6836","nonce":2914884530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6834/536559492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":536559492},{"type":"v1","addr":"172.21.15.172:6834","nonce":536559492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":40436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/1923400183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6833","nonce":1923400183},{"type":"v1","addr":"172.21.15.172:6835","nonce":1923400183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[]} 2023-10-16T11:22:52.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5d6806b020 msgr2=0x7f5d6806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5d6806b020 0x7f5d6806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d74000f60 tx=0x7f5d74009040 comp rx=0 tx=0).stop 2023-10-16T11:22:52.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 msgr2=0x7f5d8406b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T11:22:52.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d8406b180 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f5d80000c00 tx=0x7f5d80001120 comp rx=0 tx=0).stop 2023-10-16T11:22:52.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 shutdown_connections 2023-10-16T11:22:52.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:6832/899784159,v1:172.21.15.158:6833/899784159] conn(0x7f5d6806b020 0x7f5d6806d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840b0690 0x7f5d840b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5d70003a30 0x7f5d8406b180 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 --2- 172.21.15.158:0/603746573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d841568a0 0x7f5d840c2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T11:22:52.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 >> 172.21.15.158:0/603746573 conn(0x7f5d840cc5a0 msgr2=0x7f5d840ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T11:22:52.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 shutdown_connections 2023-10-16T11:22:52.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-16T11:22:52.617+0000 7f5d8b4e3700 1 -- 172.21.15.158:0/603746573 wait complete. 2023-10-16T11:22:52.625 INFO:teuthology.orchestra.run.smithi158.stderr:dumped fsmap epoch 156 2023-10-16T11:22:52.631 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-16T11:22:52.632 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-16T11:22:52.632 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T11:22:52.633 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.640 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T11:22:52.641 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T11:22:52.641 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.689 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T11:22:52.691 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T11:22:52.691 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.702 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T11:22:52.702 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T11:22:52.702 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.753 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T11:22:52.753 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-16T11:22:52.754 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-16T11:22:52.754 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.791 INFO:tasks.ceph.osd.0:Stopped 2023-10-16T11:22:52.791 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-16T11:22:52.791 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.875 INFO:tasks.ceph.osd.1:Stopped 2023-10-16T11:22:52.876 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-16T11:22:52.876 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.907 INFO:tasks.ceph.osd.2:Stopped 2023-10-16T11:22:52.908 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-16T11:22:52.908 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:52.977 INFO:tasks.ceph.osd.3:Stopped 2023-10-16T11:22:52.977 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-16T11:22:52.978 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.016 INFO:tasks.ceph.osd.4:Stopped 2023-10-16T11:22:53.016 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-16T11:22:53.017 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.101 INFO:tasks.ceph.osd.5:Stopped 2023-10-16T11:22:53.102 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-16T11:22:53.102 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.141 INFO:tasks.ceph.osd.6:Stopped 2023-10-16T11:22:53.141 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-16T11:22:53.141 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.224 INFO:tasks.ceph.osd.7:Stopped 2023-10-16T11:22:53.225 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-16T11:22:53.225 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-16T11:22:53.226 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.255 INFO:tasks.ceph.mgr.y:Stopped 2023-10-16T11:22:53.256 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-16T11:22:53.256 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.285 INFO:tasks.ceph.mgr.x:Stopped 2023-10-16T11:22:53.285 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-16T11:22:53.286 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-16T11:22:53.286 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.311 INFO:tasks.ceph.mon.a:Stopped 2023-10-16T11:22:53.311 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-16T11:22:53.311 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.339 INFO:tasks.ceph.mon.b:Stopped 2023-10-16T11:22:53.340 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-16T11:22:53.340 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T11:22:53.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-16T11:22:53.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-16T11:22:53.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-16T11:22:53.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-10-16T11:22:53.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-10-16T11:22:53.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-16T11:22:53.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-10-16T11:22:53.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-10-16T11:22:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-16T11:22:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-10-16T11:22:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-10-16T11:22:53.397 INFO:tasks.ceph.mon.c:Stopped 2023-10-16T11:22:53.398 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-16T11:22:53.401 DEBUG:teuthology.orchestra.run.smithi158:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-16T11:22:53.431 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:22:53.432 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-16T11:22:53.724 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:22:53.725 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-16T11:22:53.923 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:22:53.923 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-16T11:22:53.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:22:53.996 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-16T11:22:54.097 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T11:22:54.098 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-16T11:22:54.358 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T11:22:54.359 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-16T11:22:54.448 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T11:22:54.448 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-16T11:22:54.631 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T11:22:54.632 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-16T11:22:54.734 INFO:tasks.ceph:Archiving mon data... 2023-10-16T11:22:54.736 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/data/mon.a.tgz 2023-10-16T11:22:54.737 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2023-10-16T11:22:54.756 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.YQWf6wjoeM 2023-10-16T11:22:54.756 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YQWf6wjoeM 2023-10-16T11:22:55.503 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.YQWf6wjoeM 2023-10-16T11:22:55.590 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.YQWf6wjoeM is 4MB 2023-10-16T11:22:55.796 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.YQWf6wjoeM 2023-10-16T11:22:55.814 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/data/mon.b.tgz 2023-10-16T11:22:55.814 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-16T11:22:55.835 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.WuzRXUcL7F 2023-10-16T11:22:55.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WuzRXUcL7F 2023-10-16T11:22:56.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.WuzRXUcL7F 2023-10-16T11:22:56.659 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.WuzRXUcL7F is 5MB 2023-10-16T11:22:56.837 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.WuzRXUcL7F 2023-10-16T11:22:56.855 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/data/mon.c.tgz 2023-10-16T11:22:56.855 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-16T11:22:56.911 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.9rfqR6dHh1 2023-10-16T11:22:56.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9rfqR6dHh1 2023-10-16T11:22:57.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.9rfqR6dHh1 2023-10-16T11:22:57.729 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.9rfqR6dHh1 is 5MB 2023-10-16T11:22:57.891 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.9rfqR6dHh1 2023-10-16T11:22:57.909 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-16T11:22:57.909 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T11:22:57.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T11:22:57.975 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-16T11:22:57.975 DEBUG:teuthology.orchestra.run.smithi158:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T11:22:57.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T11:22:58.044 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T11:22:58.306 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T11:22:58.314 INFO:tasks.ceph:Archiving crash dumps... 2023-10-16T11:22:58.316 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/remote/smithi158/crash 2023-10-16T11:22:58.317 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T11:22:58.350 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/remote/smithi172/crash 2023-10-16T11:22:58.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T11:22:58.380 INFO:tasks.ceph:Shutting down logrotate 2023-10-16T11:22:58.381 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:58.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:58.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-16T11:22:58.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-16T11:22:58.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-16T11:22:58.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-10-16T11:22:58.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-10-16T11:22:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-16T11:22:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-10-16T11:22:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-10-16T11:22:58.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-16T11:22:58.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-10-16T11:22:58.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-10-16T11:22:58.469 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:58.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T11:22:58.524 INFO:tasks.ceph:Compressing logs... 2023-10-16T11:22:58.525 DEBUG:teuthology.orchestra.run.smithi158:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T11:22:58.541 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T11:22:58.601 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104202.log 2023-10-16T11:22:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-16T11:22:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.104202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104020.log 2023-10-16T11:22:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T11:22:58.603 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.5.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.104202.log.gz 2023-10-16T11:22:58.603 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.104020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104300.log 2023-10-16T11:22:58.603 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104020.log.gz 2023-10-16T11:22:58.603 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.111300.log 2023-10-16T11:22:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.104300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104157.log 2023-10-16T11:22:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.3.111300.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104300.log.gz 2023-10-16T11:22:58.604 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.111300.log.gz 2023-10-16T11:22:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.110685.log 2023-10-16T11:22:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.104157.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-16T11:22:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.1.110685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.110685.log.gz 2023-10-16T11:22:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.110693.log 2023-10-16T11:22:58.606 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104157.log.gz 2023-10-16T11:22:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103974.log 2023-10-16T11:22:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.1.110693.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-16T11:22:58.606 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.110693.log.gz 2023-10-16T11:22:58.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.103974.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T11:22:58.607 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103974.log.gz 2023-10-16T11:22:58.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-16T11:22:58.608 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-16T11:22:58.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-16T11:22:58.615 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.c.log: 94.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T11:22:58.615 INFO:teuthology.orchestra.run.smithi172.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T11:22:58.615 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104065.log 2023-10-16T11:22:58.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.104065.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104065.log.gz 2023-10-16T11:22:58.616 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-16T11:22:58.617 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104112.log 2023-10-16T11:22:58.617 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.104112.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104112.log.gz 2023-10-16T11:22:58.617 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-16T11:22:58.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.111304.log 2023-10-16T11:22:58.644 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.3.111304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.111304.log.gz 2023-10-16T11:22:58.645 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-16T11:22:58.674 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104247.log 2023-10-16T11:22:58.689 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123964.log 2023-10-16T11:22:58.689 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149489.log 2023-10-16T11:22:58.690 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.123964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183763.log 2023-10-16T11:22:58.690 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170676.log 2023-10-16T11:22:58.690 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.123964.log.gz 2023-10-16T11:22:58.690 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149489.log.gz 2023-10-16T11:22:58.691 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170825.log 2023-10-16T11:22:58.691 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164708.log 2023-10-16T11:22:58.691 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.183763.log.gz 2023-10-16T11:22:58.692 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134583.log 2023-10-16T11:22:58.692 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.170676.log.gz 2023-10-16T11:22:58.692 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.164708.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.170825.log.gz 2023-10-16T11:22:58.693 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191598.log 2023-10-16T11:22:58.693 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.164708.log.gz 2023-10-16T11:22:58.693 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143403.log 2023-10-16T11:22:58.694 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.134583.log.gz 2023-10-16T11:22:58.694 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178828.log 2023-10-16T11:22:58.694 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143403.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191598.log.gz 2023-10-16T11:22:58.694 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149738.log 2023-10-16T11:22:58.695 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143403.log.gz 2023-10-16T11:22:58.695 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121390.log 2023-10-16T11:22:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149738.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.178828.log.gz 2023-10-16T11:22:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173331.log 2023-10-16T11:22:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.121390.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149738.log.gz 2023-10-16T11:22:58.696 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193293.log 2023-10-16T11:22:58.697 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.173331.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.121390.log.gz 2023-10-16T11:22:58.697 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154018.log 2023-10-16T11:22:58.697 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.173331.log.gz 2023-10-16T11:22:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129793.log 2023-10-16T11:22:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154018.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.193293.log.gz 2023-10-16T11:22:58.698 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124492.log 2023-10-16T11:22:58.698 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154018.log.gz 2023-10-16T11:22:58.699 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155319.log 2023-10-16T11:22:58.699 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.129793.log.gz 2023-10-16T11:22:58.699 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142731.log 2023-10-16T11:22:58.699 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155319.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124492.log.gz 2023-10-16T11:22:58.700 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187486.log 2023-10-16T11:22:58.700 INFO:teuthology.orchestra.run.smithi158.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.155319.log.gz 2023-10-16T11:22:58.700 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153917.log 2023-10-16T11:22:58.700 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.142731.log.gz 2023-10-16T11:22:58.701 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116771.log 2023-10-16T11:22:58.701 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.187486.log.gz/var/log/ceph/ceph-client.admin.153917.log: 2023-10-16T11:22:58.701 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172454.log 2023-10-16T11:22:58.701 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116771.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.153917.log.gz 2023-10-16T11:22:58.702 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154219.log 2023-10-16T11:22:58.702 INFO:teuthology.orchestra.run.smithi158.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.116771.log.gz 2023-10-16T11:22:58.702 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185713.log 2023-10-16T11:22:58.702 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154219.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.172454.log.gz 2023-10-16T11:22:58.703 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134331.log 2023-10-16T11:22:58.703 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154219.log.gz 2023-10-16T11:22:58.703 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179309.log 2023-10-16T11:22:58.703 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.185713.log.gz 2023-10-16T11:22:58.704 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163574.log 2023-10-16T11:22:58.704 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179309.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134331.log.gz 2023-10-16T11:22:58.704 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165827.log 2023-10-16T11:22:58.704 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182898.log 2023-10-16T11:22:58.705 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.163574.log.gz 2023-10-16T11:22:58.705 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165827.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.179309.log.gz 2023-10-16T11:22:58.705 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177859.log 2023-10-16T11:22:58.706 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.165827.log.gz 2023-10-16T11:22:58.706 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109611.log 2023-10-16T11:22:58.706 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182898.log: /var/log/ceph/ceph-client.admin.177859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139072.log 2023-10-16T11:22:58.706 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.182898.log.gz 2023-10-16T11:22:58.707 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.109611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150472.log 2023-10-16T11:22:58.707 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139072.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109611.log.gz 2023-10-16T11:22:58.707 INFO:teuthology.orchestra.run.smithi158.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.177859.log.gz 2023-10-16T11:22:58.707 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161906.log 2023-10-16T11:22:58.708 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.139072.log.gz 2023-10-16T11:22:58.708 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110289.log 2023-10-16T11:22:58.708 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150472.log.gz 2023-10-16T11:22:58.708 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173258.log 2023-10-16T11:22:58.709 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110289.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161906.log.gz 2023-10-16T11:22:58.709 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171567.log 2023-10-16T11:22:58.709 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.173258.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110289.log.gz 2023-10-16T11:22:58.710 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165976.log 2023-10-16T11:22:58.710 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.173258.log.gz 2023-10-16T11:22:58.710 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141966.log 2023-10-16T11:22:58.710 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171567.log: /var/log/ceph/ceph-client.admin.165976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161280.log 2023-10-16T11:22:58.711 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.171567.log.gz 2023-10-16T11:22:58.711 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.164015.log 2023-10-16T11:22:58.711 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.165976.log.gz 2023-10-16T11:22:58.711 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161280.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141966.log.gz 2023-10-16T11:22:58.712 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185915.log 2023-10-16T11:22:58.712 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.161280.log.gz 2023-10-16T11:22:58.712 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.164015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139664.log 2023-10-16T11:22:58.712 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.164015.log.gz 2023-10-16T11:22:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161231.log 2023-10-16T11:22:58.713 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.185915.log.gz 2023-10-16T11:22:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161603.log 2023-10-16T11:22:58.713 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161231.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.139664.log.gz 2023-10-16T11:22:58.714 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163344.log 2023-10-16T11:22:58.714 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161603.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.161231.log.gz 2023-10-16T11:22:58.714 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149890.log 2023-10-16T11:22:58.715 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.161603.log.gz 2023-10-16T11:22:58.715 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141698.log 2023-10-16T11:22:58.715 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.163344.log.gz 2023-10-16T11:22:58.715 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169659.log 2023-10-16T11:22:58.716 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141698.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.149890.log.gz 2023-10-16T11:22:58.716 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142827.log 2023-10-16T11:22:58.716 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.169659.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141698.log.gz 2023-10-16T11:22:58.717 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177297.log 2023-10-16T11:22:58.717 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.169659.log.gz 2023-10-16T11:22:58.717 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170925.log 2023-10-16T11:22:58.717 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177297.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142827.log.gz 2023-10-16T11:22:58.718 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178272.log 2023-10-16T11:22:58.718 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.177297.log.gz 2023-10-16T11:22:58.718 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191644.log 2023-10-16T11:22:58.718 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.170925.log.gz 2023-10-16T11:22:58.719 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144639.log 2023-10-16T11:22:58.719 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191644.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.178272.log.gz 2023-10-16T11:22:58.719 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118285.log 2023-10-16T11:22:58.720 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144639.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.191644.log.gz 2023-10-16T11:22:58.720 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182133.log 2023-10-16T11:22:58.720 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.144639.log.gz 2023-10-16T11:22:58.720 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128692.log 2023-10-16T11:22:58.721 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182133.log: gzip -5 88.6%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.118285.log.gz/var/log/ceph/ceph-client.admin.192392.log 2023-10-16T11:22:58.721 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-16T11:22:58.721 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.182133.log.gz 2023-10-16T11:22:58.721 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.184000.log 2023-10-16T11:22:58.722 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.128692.log.gz 2023-10-16T11:22:58.722 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171989.log 2023-10-16T11:22:58.722 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.184000.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.192392.log.gz 2023-10-16T11:22:58.722 INFO:teuthology.orchestra.run.smithi158.stderr:gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.0.184000.log.gz--verbose 2023-10-16T11:22:58.723 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.145408.log 2023-10-16T11:22:58.723 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150150.log 2023-10-16T11:22:58.723 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145408.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.171989.log.gz 2023-10-16T11:22:58.723 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192576.log 2023-10-16T11:22:58.724 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.145408.log.gz 2023-10-16T11:22:58.724 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135948.log 2023-10-16T11:22:58.724 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192576.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150150.log.gz 2023-10-16T11:22:58.725 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141400.log 2023-10-16T11:22:58.725 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.192576.log.gz 2023-10-16T11:22:58.725 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145854.log 2023-10-16T11:22:58.725 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135948.log.gz 2023-10-16T11:22:58.726 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155736.log 2023-10-16T11:22:58.726 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145854.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.141400.log.gz 2023-10-16T11:22:58.726 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164855.log 2023-10-16T11:22:58.726 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155736.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.145854.log.gz 2023-10-16T11:22:58.727 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162253.log 2023-10-16T11:22:58.727 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155736.log.gz 2023-10-16T11:22:58.727 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.164855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.115109.log 2023-10-16T11:22:58.727 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162253.log: 89.4%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.164855.log.gz--verbose 2023-10-16T11:22:58.728 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.141652.log 2023-10-16T11:22:58.728 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.115109.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.162253.log.gz 2023-10-16T11:22:58.728 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.115109.log.gz 2023-10-16T11:22:58.728 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191752.log 2023-10-16T11:22:58.729 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149386.log 2023-10-16T11:22:58.729 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191752.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.141652.log.gz 2023-10-16T11:22:58.729 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111917.log 2023-10-16T11:22:58.729 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.191752.log.gz 2023-10-16T11:22:58.730 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119544.log 2023-10-16T11:22:58.730 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.149386.log.gz 2023-10-16T11:22:58.730 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111917.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-16T11:22:58.731 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119544.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.111917.log.gz 2023-10-16T11:22:58.731 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124162.log 2023-10-16T11:22:58.731 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-osd.2.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.119544.log.gz 2023-10-16T11:22:58.731 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143912.log 2023-10-16T11:22:58.732 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130117.log 2023-10-16T11:22:58.732 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143912.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.124162.log.gz 2023-10-16T11:22:58.732 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177537.log 2023-10-16T11:22:58.732 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143912.log.gz 2023-10-16T11:22:58.733 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109794.log 2023-10-16T11:22:58.733 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130117.log.gz 2023-10-16T11:22:58.733 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170065.log 2023-10-16T11:22:58.733 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.109794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131262.log 2023-10-16T11:22:58.734 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.177537.log.gz 2023-10-16T11:22:58.734 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162483.log 2023-10-16T11:22:58.734 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109794.log.gz 2023-10-16T11:22:58.734 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.170065.log.gz 2023-10-16T11:22:58.735 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.131262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143839.log 2023-10-16T11:22:58.735 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.131262.log.gz 2023-10-16T11:22:58.735 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166206.log 2023-10-16T11:22:58.735 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.162483.log.gz 2023-10-16T11:22:58.736 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184709.log 2023-10-16T11:22:58.736 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166206.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143839.log.gz 2023-10-16T11:22:58.736 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154782.log 2023-10-16T11:22:58.737 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184709.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.166206.log.gz 2023-10-16T11:22:58.737 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163712.log 2023-10-16T11:22:58.737 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154782.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.184709.log.gz 2023-10-16T11:22:58.737 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120606.log 2023-10-16T11:22:58.738 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154782.log.gz 2023-10-16T11:22:58.738 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170726.log 2023-10-16T11:22:58.738 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122199.log 2023-10-16T11:22:58.739 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170726.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120606.log.gz 2023-10-16T11:22:58.739 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190709.log 2023-10-16T11:22:58.739 INFO:teuthology.orchestra.run.smithi158.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.163712.log.gz 2023-10-16T11:22:58.739 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.170726.log.gz 2023-10-16T11:22:58.740 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188907.log 2023-10-16T11:22:58.740 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190709.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122199.log.gz 2023-10-16T11:22:58.740 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145107.log 2023-10-16T11:22:58.740 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.190709.log.gz 2023-10-16T11:22:58.741 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180873.log 2023-10-16T11:22:58.741 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145107.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.188907.log.gz 2023-10-16T11:22:58.741 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114327.log 2023-10-16T11:22:58.741 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.145107.log.gz 2023-10-16T11:22:58.742 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146190.log 2023-10-16T11:22:58.742 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114327.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.180873.log.gz 2023-10-16T11:22:58.742 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129887.log 2023-10-16T11:22:58.743 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146190.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114327.log.gz 2023-10-16T11:22:58.743 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128291.log 2023-10-16T11:22:58.743 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146190.log.gz 2023-10-16T11:22:58.743 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143792.log 2023-10-16T11:22:58.744 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128291.log: gzip 89.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.129887.log.gz--verbose 2023-10-16T11:22:58.744 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.165333.log 2023-10-16T11:22:58.744 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128291.log.gz 2023-10-16T11:22:58.744 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165583.log 2023-10-16T11:22:58.745 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143792.log.gz 2023-10-16T11:22:58.745 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165381.log 2023-10-16T11:22:58.745 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165583.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.165333.log.gz 2023-10-16T11:22:58.745 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145716.log 2023-10-16T11:22:58.746 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165381.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.165583.log.gz 2023-10-16T11:22:58.746 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112715.log 2023-10-16T11:22:58.746 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165381.log.gz 2023-10-16T11:22:58.747 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196182.log 2023-10-16T11:22:58.747 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112715.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145716.log.gz 2023-10-16T11:22:58.747 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163620.log 2023-10-16T11:22:58.747 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5%/var/log/ceph/ceph-client.admin.196182.log: -- replaced with /var/log/ceph/ceph-client.admin.112715.log.gz 2023-10-16T11:22:58.748 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173402.log 2023-10-16T11:22:58.748 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.196182.log.gz 2023-10-16T11:22:58.748 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122642.log 2023-10-16T11:22:58.749 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.163620.log.gz 2023-10-16T11:22:58.749 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.173402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187210.log 2023-10-16T11:22:58.749 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122642.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.173402.log.gz 2023-10-16T11:22:58.749 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119912.log 2023-10-16T11:22:58.750 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187210.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122642.log.gz 2023-10-16T11:22:58.750 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121249.log 2023-10-16T11:22:58.750 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.187210.log.gz 2023-10-16T11:22:58.750 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183463.log 2023-10-16T11:22:58.751 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.121249.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119912.log.gz 2023-10-16T11:22:58.751 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182813.log 2023-10-16T11:22:58.751 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183463.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.121249.log.gz 2023-10-16T11:22:58.751 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T11:22:58.752 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.183463.log.gz 2023-10-16T11:22:58.752 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155001.log 2023-10-16T11:22:58.752 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.182813.log.gz 2023-10-16T11:22:58.753 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134778.log 2023-10-16T11:22:58.753 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183601.log 2023-10-16T11:22:58.753 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134778.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.155001.log.gz 2023-10-16T11:22:58.753 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.178166.log 2023-10-16T11:22:58.754 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134778.log.gz 2023-10-16T11:22:58.754 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139385.log 2023-10-16T11:22:58.754 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183601.log: /var/log/ceph/ceph-client.0.178166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152986.log 2023-10-16T11:22:58.754 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.178166.log.gz 2023-10-16T11:22:58.755 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.183601.log.gz 2023-10-16T11:22:58.755 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139756.log 2023-10-16T11:22:58.755 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152986.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139385.log.gz 2023-10-16T11:22:58.755 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192019.log 2023-10-16T11:22:58.756 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139756.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152986.log.gz 2023-10-16T11:22:58.756 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190242.log 2023-10-16T11:22:58.756 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139756.log.gz 2023-10-16T11:22:58.756 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112709.log 2023-10-16T11:22:58.757 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose 87.9%-- /var/log/ceph/ceph-client.admin.190242.log: /var/log/ceph/ceph-client.admin.194731.log -- replaced with /var/log/ceph/ceph-client.admin.192019.log.gz 2023-10-16T11:22:58.757 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-16T11:22:58.757 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184245.log 2023-10-16T11:22:58.758 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.190242.log.gz 2023-10-16T11:22:58.758 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194731.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112709.log.gz 2023-10-16T11:22:58.758 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170565.log 2023-10-16T11:22:58.758 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.194731.log.gz 2023-10-16T11:22:58.759 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113892.log 2023-10-16T11:22:58.759 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122736.log 2023-10-16T11:22:58.759 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6%/var/log/ceph/ceph-client.admin.170565.log: -- replaced with /var/log/ceph/ceph-client.admin.184245.log.gz 2023-10-16T11:22:58.759 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114030.log 2023-10-16T11:22:58.760 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.170565.log.gz 2023-10-16T11:22:58.760 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122736.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.113892.log.gz 2023-10-16T11:22:58.760 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114373.log 2023-10-16T11:22:58.760 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122736.log.gz 2023-10-16T11:22:58.761 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136552.log 2023-10-16T11:22:58.761 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114373.log: gzip -5 --verbose -- 89.2%/var/log/ceph/ceph-client.admin.184847.log 2023-10-16T11:22:58.761 INFO:teuthology.orchestra.run.smithi158.stderr: -- replaced with /var/log/ceph/ceph-client.admin.114030.log.gz 2023-10-16T11:22:58.761 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114373.log.gz 2023-10-16T11:22:58.762 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.136552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181903.log 2023-10-16T11:22:58.762 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136552.log.gz 2023-10-16T11:22:58.762 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138134.log 2023-10-16T11:22:58.762 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.184847.log.gz 2023-10-16T11:22:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182041.log 2023-10-16T11:22:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190659.log 2023-10-16T11:22:58.763 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4%/var/log/ceph/ceph-client.admin.138134.log: -- replaced with /var/log/ceph/ceph-client.admin.181903.log.gz 2023-10-16T11:22:58.763 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179058.log 2023-10-16T11:22:58.764 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138134.log.gz 2023-10-16T11:22:58.764 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190659.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.182041.log.gz 2023-10-16T11:22:58.764 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155132.log 2023-10-16T11:22:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179058.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.190659.log.gz 2023-10-16T11:22:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192530.log 2023-10-16T11:22:58.765 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179058.log.gz 2023-10-16T11:22:58.765 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139213.log 2023-10-16T11:22:58.766 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155132.log: /var/log/ceph/ceph-client.admin.192530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181294.log 2023-10-16T11:22:58.766 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.155132.log.gz 2023-10-16T11:22:58.766 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139213.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.192530.log.gz 2023-10-16T11:22:58.766 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163137.log 2023-10-16T11:22:58.767 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T11:22:58.767 INFO:teuthology.orchestra.run.smithi158.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.139213.log.gz 2023-10-16T11:22:58.767 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181294.log: /var/log/ceph/ceph-client.admin.190556.log 2023-10-16T11:22:58.768 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163137.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.181294.log.gz 2023-10-16T11:22:58.768 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147142.log 2023-10-16T11:22:58.768 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190556.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.163137.log.gz 2023-10-16T11:22:58.768 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142685.log 2023-10-16T11:22:58.769 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.190556.log.gz 2023-10-16T11:22:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.147142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119000.log 2023-10-16T11:22:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142685.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.147142.log.gz 2023-10-16T11:22:58.769 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165171.log 2023-10-16T11:22:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119000.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142685.log.gz 2023-10-16T11:22:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146414.log 2023-10-16T11:22:58.770 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119000.log.gz 2023-10-16T11:22:58.770 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109978.log 2023-10-16T11:22:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165171.log: /var/log/ceph/ceph-client.admin.146414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139432.log 2023-10-16T11:22:58.771 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.165171.log.gz 2023-10-16T11:22:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.109978.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146414.log.gz 2023-10-16T11:22:58.771 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118954.log 2023-10-16T11:22:58.772 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139432.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109978.log.gz 2023-10-16T11:22:58.772 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113800.log 2023-10-16T11:22:58.772 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139432.log.gz 2023-10-16T11:22:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119092.log 2023-10-16T11:22:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113800.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.118954.log.gz 2023-10-16T11:22:58.773 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128391.log 2023-10-16T11:22:58.773 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.113800.log.gz 2023-10-16T11:22:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177110.log 2023-10-16T11:22:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119092.log: /var/log/ceph/ceph-client.admin.128391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184663.log 2023-10-16T11:22:58.774 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119092.log.gz 2023-10-16T11:22:58.774 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177110.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128391.log.gz 2023-10-16T11:22:58.775 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136199.log 2023-10-16T11:22:58.775 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.177110.log.gz 2023-10-16T11:22:58.775 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117277.log 2023-10-16T11:22:58.775 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184663.log: /var/log/ceph/ceph-client.admin.136199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135568.log 2023-10-16T11:22:58.776 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.184663.log.gz 2023-10-16T11:22:58.776 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117277.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.136199.log.gz 2023-10-16T11:22:58.776 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141298.log 2023-10-16T11:22:58.777 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117277.log.gz 2023-10-16T11:22:58.777 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114260.log 2023-10-16T11:22:58.777 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9%/var/log/ceph/ceph-client.admin.141298.log: -- replaced with /var/log/ceph/ceph-client.admin.135568.log.gz 2023-10-16T11:22:58.777 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184755.log 2023-10-16T11:22:58.778 INFO:teuthology.orchestra.run.smithi158.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.141298.log.gz 2023-10-16T11:22:58.778 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170200.log 2023-10-16T11:22:58.778 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114260.log.gz 2023-10-16T11:22:58.778 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147531.log 2023-10-16T11:22:58.779 INFO:teuthology.orchestra.run.smithi158.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.184755.log.gz 2023-10-16T11:22:58.779 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180606.log 2023-10-16T11:22:58.779 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.147531.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.170200.log.gz 2023-10-16T11:22:58.780 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155598.log 2023-10-16T11:22:58.780 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147531.log.gz 2023-10-16T11:22:58.780 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138329.log 2023-10-16T11:22:58.780 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.180606.log.gz 2023-10-16T11:22:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153031.log 2023-10-16T11:22:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138329.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.155598.log.gz 2023-10-16T11:22:58.781 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.115101.log 2023-10-16T11:22:58.781 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138329.log.gz 2023-10-16T11:22:58.782 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191255.log 2023-10-16T11:22:58.782 INFO:teuthology.orchestra.run.smithi158.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.153031.log.gz 2023-10-16T11:22:58.782 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.115101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171123.log 2023-10-16T11:22:58.782 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.115101.log.gz 2023-10-16T11:22:58.783 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152739.log 2023-10-16T11:22:58.783 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.191255.log.gz 2023-10-16T11:22:58.783 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166022.log 2023-10-16T11:22:58.783 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152739.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.171123.log.gz 2023-10-16T11:22:58.784 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142965.log 2023-10-16T11:22:58.784 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152739.log.gz 2023-10-16T11:22:58.784 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134824.log 2023-10-16T11:22:58.784 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.166022.log.gz 2023-10-16T11:22:58.785 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122407.log 2023-10-16T11:22:58.785 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134824.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.142965.log.gz 2023-10-16T11:22:58.785 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162951.log 2023-10-16T11:22:58.785 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122407.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134824.log.gz 2023-10-16T11:22:58.786 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111733.log 2023-10-16T11:22:58.786 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122407.log.gz 2023-10-16T11:22:58.786 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135614.log 2023-10-16T11:22:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112052.log 2023-10-16T11:22:58.787 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.162951.log.gz 2023-10-16T11:22:58.787 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111733.log.gz 2023-10-16T11:22:58.787 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183647.log 2023-10-16T11:22:58.788 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.135614.log.gz--verbose 2023-10-16T11:22:58.788 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.143195.log /var/log/ceph/ceph-client.admin.112052.log: 2023-10-16T11:22:58.788 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182523.log 2023-10-16T11:22:58.788 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.112052.log.gz 2023-10-16T11:22:58.789 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.183647.log.gz 2023-10-16T11:22:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113708.log 2023-10-16T11:22:58.789 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116587.log 2023-10-16T11:22:58.790 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143195.log.gz 2023-10-16T11:22:58.790 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.182523.log.gz 2023-10-16T11:22:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196113.log 2023-10-16T11:22:58.790 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134722.log 2023-10-16T11:22:58.791 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113708.log.gz 2023-10-16T11:22:58.791 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.196113.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116587.log.gz 2023-10-16T11:22:58.791 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112717.log 2023-10-16T11:22:58.791 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.196113.log.gz 2023-10-16T11:22:58.792 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164241.log 2023-10-16T11:22:58.792 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112717.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134722.log.gz 2023-10-16T11:22:58.792 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120744.log 2023-10-16T11:22:58.792 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2%/var/log/ceph/ceph-client.admin.164241.log: -- replaced with /var/log/ceph/ceph-client.admin.112717.log.gz 2023-10-16T11:22:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190859.log 2023-10-16T11:22:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120744.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.164241.log.gz 2023-10-16T11:22:58.793 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122132.log 2023-10-16T11:22:58.794 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190859.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120744.log.gz 2023-10-16T11:22:58.794 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145900.log 2023-10-16T11:22:58.794 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.190859.log.gz 2023-10-16T11:22:58.794 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188815.log 2023-10-16T11:22:58.795 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171337.log 2023-10-16T11:22:58.795 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122132.log.gz 2023-10-16T11:22:58.795 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188815.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.145900.log.gz 2023-10-16T11:22:58.795 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187762.log 2023-10-16T11:22:58.796 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.188815.log.gz 2023-10-16T11:22:58.796 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113083.log 2023-10-16T11:22:58.796 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187762.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.171337.log.gz 2023-10-16T11:22:58.796 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117139.log 2023-10-16T11:22:58.797 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113083.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.187762.log.gz 2023-10-16T11:22:58.797 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129364.log 2023-10-16T11:22:58.797 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113083.log.gz 2023-10-16T11:22:58.798 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164955.log 2023-10-16T11:22:58.798 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129364.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117139.log.gz 2023-10-16T11:22:58.798 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162621.log 2023-10-16T11:22:58.798 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.164955.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.129364.log.gz 2023-10-16T11:22:58.799 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-16T11:22:58.799 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.164955.log.gz 2023-10-16T11:22:58.799 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136824.log 2023-10-16T11:22:58.799 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-osd.0.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.162621.log.gz 2023-10-16T11:22:58.800 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178456.log 2023-10-16T11:22:58.800 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146656.log 2023-10-16T11:22:58.800 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178456.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.136824.log.gz 2023-10-16T11:22:58.800 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161378.log 2023-10-16T11:22:58.801 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.178456.log.gz 2023-10-16T11:22:58.801 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119866.log 2023-10-16T11:22:58.801 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.146656.log.gz 2023-10-16T11:22:58.801 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187026.log 2023-10-16T11:22:58.802 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161378.log: /var/log/ceph/ceph-client.admin.119866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178874.log 2023-10-16T11:22:58.802 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.161378.log.gz 2023-10-16T11:22:58.802 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187026.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.119866.log.gz 2023-10-16T11:22:58.802 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182363.log 2023-10-16T11:22:58.803 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.187026.log.gz 2023-10-16T11:22:58.803 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113455.log 2023-10-16T11:22:58.803 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.178874.log.gz 2023-10-16T11:22:58.803 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128139.log 2023-10-16T11:22:58.804 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193745.log 2023-10-16T11:22:58.804 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.182363.log.gz 2023-10-16T11:22:58.804 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113455.log.gz 2023-10-16T11:22:58.804 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195791.log 2023-10-16T11:22:58.805 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122501.log 2023-10-16T11:22:58.805 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128139.log.gz 2023-10-16T11:22:58.805 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195791.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.193745.log.gz 2023-10-16T11:22:58.805 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133714.log 2023-10-16T11:22:58.806 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.195791.log.gz 2023-10-16T11:22:58.806 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119276.log 2023-10-16T11:22:58.806 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122501.log: /var/log/ceph/ceph-client.admin.133714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189504.log 2023-10-16T11:22:58.807 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.122501.log.gz 2023-10-16T11:22:58.807 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119276.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.133714.log.gz 2023-10-16T11:22:58.807 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192484.log 2023-10-16T11:22:58.807 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119276.log.gz 2023-10-16T11:22:58.808 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194210.log 2023-10-16T11:22:58.808 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130417.log 2023-10-16T11:22:58.808 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192484.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.189504.log.gz 2023-10-16T11:22:58.808 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194210.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.192484.log.gz 2023-10-16T11:22:58.809 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178548.log 2023-10-16T11:22:58.809 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168077.log 2023-10-16T11:22:58.809 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.194210.log.gz 2023-10-16T11:22:58.810 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178548.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.130417.log.gz 2023-10-16T11:22:58.810 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146275.log 2023-10-16T11:22:58.810 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.178548.log.gz 2023-10-16T11:22:58.810 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.168077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176289.log 2023-10-16T11:22:58.811 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111438.log 2023-10-16T11:22:58.811 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.168077.log.gz 2023-10-16T11:22:58.811 INFO:teuthology.orchestra.run.smithi158.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.146275.log.gz 2023-10-16T11:22:58.811 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179616.log 2023-10-16T11:22:58.812 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178644.log 2023-10-16T11:22:58.812 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111438.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.176289.log.gz 2023-10-16T11:22:58.812 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179616.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.111438.log.gz 2023-10-16T11:22:58.813 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141451.log 2023-10-16T11:22:58.813 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110120.log 2023-10-16T11:22:58.813 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179616.log.gz 2023-10-16T11:22:58.813 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141451.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.178644.log.gz 2023-10-16T11:22:58.814 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120652.log 2023-10-16T11:22:58.814 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145454.log 2023-10-16T11:22:58.814 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110120.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.141451.log.gz 2023-10-16T11:22:58.814 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120652.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110120.log.gz 2023-10-16T11:22:58.815 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184525.log 2023-10-16T11:22:58.815 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.120652.log.gz 2023-10-16T11:22:58.815 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177249.log 2023-10-16T11:22:58.815 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179196.log 2023-10-16T11:22:58.816 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184525.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.145454.log.gz 2023-10-16T11:22:58.816 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128021.log 2023-10-16T11:22:58.816 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.184525.log.gz 2023-10-16T11:22:58.817 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179196.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.177249.log.gz 2023-10-16T11:22:58.817 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194630.log 2023-10-16T11:22:58.817 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.179196.log.gz 2023-10-16T11:22:58.817 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194109.log 2023-10-16T11:22:58.818 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.189302.log 2023-10-16T11:22:58.818 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194630.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.128021.log.gz 2023-10-16T11:22:58.818 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188296.log 2023-10-16T11:22:58.818 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.194630.log.gz 2023-10-16T11:22:58.819 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.189302.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.194109.log.gzgzip 2023-10-16T11:22:58.819 INFO:teuthology.orchestra.run.smithi158.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.171943.log 2023-10-16T11:22:58.819 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.189302.log.gz 2023-10-16T11:22:58.820 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135100.log 2023-10-16T11:22:58.820 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171943.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.188296.log.gz 2023-10-16T11:22:58.820 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166648.log 2023-10-16T11:22:58.820 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.171943.log.gz 2023-10-16T11:22:58.821 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188008.log 2023-10-16T11:22:58.821 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143241.log 2023-10-16T11:22:58.821 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166648.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135100.log.gz 2023-10-16T11:22:58.822 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134962.log 2023-10-16T11:22:58.822 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.166648.log.gz 2023-10-16T11:22:58.822 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143241.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.188008.log.gz 2023-10-16T11:22:58.822 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112719.log 2023-10-16T11:22:58.823 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143241.log.gz 2023-10-16T11:22:58.823 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194010.log 2023-10-16T11:22:58.823 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112719.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134962.log.gz 2023-10-16T11:22:58.823 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112714.log 2023-10-16T11:22:58.824 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112719.log.gz 2023-10-16T11:22:58.824 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179012.log 2023-10-16T11:22:58.824 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146980.log 2023-10-16T11:22:58.824 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112714.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.194010.log.gz 2023-10-16T11:22:58.825 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179012.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112714.log.gz 2023-10-16T11:22:58.825 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181535.log 2023-10-16T11:22:58.825 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.179012.log.gz 2023-10-16T11:22:58.825 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194584.log 2023-10-16T11:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146980.log: /var/log/ceph/ceph-client.admin.181535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118193.log 2023-10-16T11:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146980.log.gz 2023-10-16T11:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.181535.log.gz 2023-10-16T11:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167477.log 2023-10-16T11:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165483.log 2023-10-16T11:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr: 86.9%/var/log/ceph/ceph-client.admin.118193.log: -- replaced with /var/log/ceph/ceph-client.admin.194584.log.gz 2023-10-16T11:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167477.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118193.log.gz 2023-10-16T11:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120560.log 2023-10-16T11:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.167477.log.gz 2023-10-16T11:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172751.log 2023-10-16T11:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165483.log: /var/log/ceph/ceph-client.admin.120560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112236.log 2023-10-16T11:22:58.829 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.165483.log.gz 2023-10-16T11:22:58.829 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120560.log.gz 2023-10-16T11:22:58.829 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172751.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-16T11:22:58.829 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112236.log: /var/log/ceph/ceph-client.admin.146460.log 2023-10-16T11:22:58.830 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187808.log 2023-10-16T11:22:58.830 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112236.log.gz 2023-10-16T11:22:58.830 INFO:teuthology.orchestra.run.smithi158.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.172751.log.gz 2023-10-16T11:22:58.831 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161180.log 2023-10-16T11:22:58.831 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113316.log 2023-10-16T11:22:58.831 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.146460.log.gz 2023-10-16T11:22:58.831 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161180.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.187808.log.gz 2023-10-16T11:22:58.832 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161180.log.gz 2023-10-16T11:22:58.832 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113316.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113316.log.gz 2023-10-16T11:22:58.832 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137131.log 2023-10-16T11:22:58.832 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171245.log 2023-10-16T11:22:58.833 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128784.log 2023-10-16T11:22:58.833 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.137131.log.gz 2023-10-16T11:22:58.833 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167986.log 2023-10-16T11:22:58.833 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154690.log 2023-10-16T11:22:58.834 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128784.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.171245.log.gz 2023-10-16T11:22:58.834 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167986.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.128784.log.gz 2023-10-16T11:22:58.834 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154690.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.167986.log.gz 2023-10-16T11:22:58.834 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154690.log.gz 2023-10-16T11:22:58.835 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181811.log 2023-10-16T11:22:58.835 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111576.log 2023-10-16T11:22:58.835 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133912.log 2023-10-16T11:22:58.836 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.181811.log.gz 2023-10-16T11:22:58.836 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177437.log 2023-10-16T11:22:58.836 INFO:teuthology.orchestra.run.smithi158.stderr:gzip /var/log/ceph/ceph-client.admin.133912.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.124360.log 2023-10-16T11:22:58.836 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111576.log.gz 2023-10-16T11:22:58.837 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177437.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.133912.log.gz 2023-10-16T11:22:58.837 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124360.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.177437.log.gz 2023-10-16T11:22:58.837 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124360.log.gz 2023-10-16T11:22:58.837 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189458.log 2023-10-16T11:22:58.838 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162058.log 2023-10-16T11:22:58.838 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154361.log 2023-10-16T11:22:58.838 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.189458.log.gz 2023-10-16T11:22:58.839 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186785.log 2023-10-16T11:22:58.839 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138559.log 2023-10-16T11:22:58.839 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.162058.log.gz 2023-10-16T11:22:58.839 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154361.log: /var/log/ceph/ceph-client.admin.186785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109840.log 2023-10-16T11:22:58.840 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.154361.log.gz 2023-10-16T11:22:58.840 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138559.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.186785.log.gz 2023-10-16T11:22:58.840 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189867.log 2023-10-16T11:22:58.840 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.138559.log.gz 2023-10-16T11:22:58.841 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.109840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170251.log 2023-10-16T11:22:58.841 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189867.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109840.log.gz 2023-10-16T11:22:58.841 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176392.log 2023-10-16T11:22:58.842 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170251.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.189867.log.gz 2023-10-16T11:22:58.842 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.170251.log.gz 2023-10-16T11:22:58.842 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176392.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.176392.log.gz 2023-10-16T11:22:58.842 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113662.log 2023-10-16T11:22:58.843 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162104.log 2023-10-16T11:22:58.843 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181627.log 2023-10-16T11:22:58.843 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162104.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113662.log.gz 2023-10-16T11:22:58.843 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.162104.log.gz 2023-10-16T11:22:58.844 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181627.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.181627.log.gz 2023-10-16T11:22:58.844 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150757.log 2023-10-16T11:22:58.845 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112374.log 2023-10-16T11:22:58.845 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113086.log 2023-10-16T11:22:58.845 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112374.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.150757.log.gz 2023-10-16T11:22:58.845 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112374.log.gz 2023-10-16T11:22:58.846 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113086.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113086.log.gz 2023-10-16T11:22:58.846 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175088.log 2023-10-16T11:22:58.846 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182271.log 2023-10-16T11:22:58.846 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140381.log 2023-10-16T11:22:58.847 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.175088.log.gz 2023-10-16T11:22:58.847 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182271.log: /var/log/ceph/ceph-client.admin.140381.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.182271.log.gz 2023-10-16T11:22:58.847 INFO:teuthology.orchestra.run.smithi158.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.140381.log.gz 2023-10-16T11:22:58.848 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.183996.log 2023-10-16T11:22:58.848 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189412.log 2023-10-16T11:22:58.848 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.183996.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.0.183996.log.gz--verbose 2023-10-16T11:22:58.848 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.113082.log 2023-10-16T11:22:58.849 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152896.log 2023-10-16T11:22:58.849 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184479.log 2023-10-16T11:22:58.849 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.189412.log.gz 2023-10-16T11:22:58.849 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113082.log.gz 2023-10-16T11:22:58.850 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167866.log 2023-10-16T11:22:58.850 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184479.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152896.log.gz 2023-10-16T11:22:58.850 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129561.log 2023-10-16T11:22:58.850 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.184479.log.gz 2023-10-16T11:22:58.851 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194825.log 2023-10-16T11:22:58.851 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.167866.log.gz 2023-10-16T11:22:58.851 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181340.log 2023-10-16T11:22:58.851 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194825.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129561.log.gz 2023-10-16T11:22:58.852 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161504.log 2023-10-16T11:22:58.852 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181340.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.194825.log.gz 2023-10-16T11:22:58.852 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.181340.log.gz 2023-10-16T11:22:58.852 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153401.log 2023-10-16T11:22:58.853 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118239.log 2023-10-16T11:22:58.853 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.161504.log.gz 2023-10-16T11:22:58.853 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153401.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153401.log.gz 2023-10-16T11:22:58.853 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118239.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118239.log.gz 2023-10-16T11:22:58.854 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181995.log 2023-10-16T11:22:58.854 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187900.log 2023-10-16T11:22:58.854 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144737.log 2023-10-16T11:22:58.854 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187900.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.181995.log.gz 2023-10-16T11:22:58.855 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144737.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.187900.log.gz 2023-10-16T11:22:58.855 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.144737.log.gz 2023-10-16T11:22:58.855 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120468.log 2023-10-16T11:22:58.855 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128979.log 2023-10-16T11:22:58.856 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185175.log 2023-10-16T11:22:58.856 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128979.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120468.log.gz 2023-10-16T11:22:58.856 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128979.log.gz 2023-10-16T11:22:58.857 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185175.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.185175.log.gz 2023-10-16T11:22:58.857 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178596.log 2023-10-16T11:22:58.857 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118862.log 2023-10-16T11:22:58.857 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192622.log 2023-10-16T11:22:58.858 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118862.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.178596.log.gz 2023-10-16T11:22:58.858 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192622.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118862.log.gz 2023-10-16T11:22:58.858 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.192622.log.gz 2023-10-16T11:22:58.858 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194917.log 2023-10-16T11:22:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191152.log 2023-10-16T11:22:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121319.log 2023-10-16T11:22:58.859 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191152.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.194917.log.gz 2023-10-16T11:22:58.860 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.191152.log.gz 2023-10-16T11:22:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.121319.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.121319.log.gz 2023-10-16T11:22:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130347.log 2023-10-16T11:22:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177391.log 2023-10-16T11:22:58.860 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139282.log 2023-10-16T11:22:58.861 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177391.log: /var/log/ceph/ceph-client.admin.139282.log: 89.0% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.177391.log.gz -- replaced with /var/log/ceph/ceph-client.admin.130347.log.gz 2023-10-16T11:22:58.861 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-16T11:22:58.861 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139282.log.gz 2023-10-16T11:22:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151091.log 2023-10-16T11:22:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138805.log 2023-10-16T11:22:58.862 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170516.log 2023-10-16T11:22:58.862 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151091.log.gz 2023-10-16T11:22:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127076.log 2023-10-16T11:22:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151045.log 2023-10-16T11:22:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170516.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.138805.log.gz 2023-10-16T11:22:58.863 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.127076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161045.log 2023-10-16T11:22:58.864 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.170516.log.gz 2023-10-16T11:22:58.864 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151045.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.127076.log.gz 2023-10-16T11:22:58.864 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119636.log 2023-10-16T11:22:58.864 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.151045.log.gz 2023-10-16T11:22:58.865 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175693.log 2023-10-16T11:22:58.865 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119636.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.161045.log.gz 2023-10-16T11:22:58.865 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155944.log 2023-10-16T11:22:58.866 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5%/var/log/ceph/ceph-client.admin.175693.log: -- replaced with /var/log/ceph/ceph-client.admin.119636.log.gz 2023-10-16T11:22:58.866 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114168.log 2023-10-16T11:22:58.866 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.175693.log.gz 2023-10-16T11:22:58.866 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187164.log 2023-10-16T11:22:58.867 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114168.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155944.log.gzgzip 2023-10-16T11:22:58.867 INFO:teuthology.orchestra.run.smithi158.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.166602.log 2023-10-16T11:22:58.867 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187164.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114168.log.gz 2023-10-16T11:22:58.867 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186378.log 2023-10-16T11:22:58.868 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.187164.log.gz 2023-10-16T11:22:58.868 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172316.log 2023-10-16T11:22:58.868 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186378.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.166602.log.gz 2023-10-16T11:22:58.869 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.186378.log.gz 2023-10-16T11:22:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172316.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.172316.log.gz 2023-10-16T11:22:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161856.log 2023-10-16T11:22:58.869 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171613.log 2023-10-16T11:22:58.870 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.147375.log 2023-10-16T11:22:58.870 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171613.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.161856.log.gz 2023-10-16T11:22:58.870 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.171613.log.gz 2023-10-16T11:22:58.870 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.147375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.147375.log.gz 2023-10-16T11:22:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146702.log 2023-10-16T11:22:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150380.log 2023-10-16T11:22:58.871 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146702.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-16T11:22:58.872 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146702.log.gz 2023-10-16T11:22:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133453.log 2023-10-16T11:22:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-osd.3.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.150380.log.gz 2023-10-16T11:22:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.133453.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133453.log.gz 2023-10-16T11:22:58.873 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118373.log 2023-10-16T11:22:58.873 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175896.log 2023-10-16T11:22:58.873 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154067.log 2023-10-16T11:22:58.873 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175896.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.118373.log.gz 2023-10-16T11:22:58.874 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133814.log 2023-10-16T11:22:58.874 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.175896.log.gz 2023-10-16T11:22:58.874 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150288.log 2023-10-16T11:22:58.874 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.133814.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154067.log.gz 2023-10-16T11:22:58.875 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.133814.log.gz 2023-10-16T11:22:58.875 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149219.log 2023-10-16T11:22:58.875 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109695.log 2023-10-16T11:22:58.875 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149219.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.150288.log.gz 2023-10-16T11:22:58.876 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149219.log.gz 2023-10-16T11:22:58.876 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.109695.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109695.log.gz 2023-10-16T11:22:58.876 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183083.log 2023-10-16T11:22:58.877 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141198.log 2023-10-16T11:22:58.877 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137832.log 2023-10-16T11:22:58.877 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141198.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.183083.log.gz 2023-10-16T11:22:58.877 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.141198.log.gz 2023-10-16T11:22:58.878 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137832.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.137832.log.gz 2023-10-16T11:22:58.878 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162575.log 2023-10-16T11:22:58.878 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167215.log 2023-10-16T11:22:58.878 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162575.log: /var/log/ceph/ceph-client.admin.167215.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.162575.log.gz 2023-10-16T11:22:58.879 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.167215.log.gz 2023-10-16T11:22:58.879 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176979.log 2023-10-16T11:22:58.879 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189069.log 2023-10-16T11:22:58.879 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154911.log 2023-10-16T11:22:58.880 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.176979.log.gz 2023-10-16T11:22:58.880 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189069.log: /var/log/ceph/ceph-client.admin.154911.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.189069.log.gz 2023-10-16T11:22:58.880 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.154911.log.gz 2023-10-16T11:22:58.881 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186229.log 2023-10-16T11:22:58.881 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181248.log 2023-10-16T11:22:58.881 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116264.log 2023-10-16T11:22:58.881 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.186229.log.gz 2023-10-16T11:22:58.882 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116725.log 2023-10-16T11:22:58.882 INFO:teuthology.orchestra.run.smithi158.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.181248.log.gz 2023-10-16T11:22:58.882 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118908.log 2023-10-16T11:22:58.882 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116725.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116264.log.gz 2023-10-16T11:22:58.883 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111530.log 2023-10-16T11:22:58.883 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116725.log.gz 2023-10-16T11:22:58.883 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185860.log 2023-10-16T11:22:58.883 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4%/var/log/ceph/ceph-client.admin.111530.log: -- replaced with /var/log/ceph/ceph-client.admin.118908.log.gz 2023-10-16T11:22:58.884 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.111530.log.gz 2023-10-16T11:22:58.884 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185860.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.185860.log.gz 2023-10-16T11:22:58.884 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188623.log 2023-10-16T11:22:58.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.104247.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.104247.log.gz 2023-10-16T11:22:58.886 INFO:teuthology.orchestra.run.smithi172.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-16T11:22:58.886 INFO:teuthology.orchestra.run.smithi172.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-16T11:22:59.041 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143636.log 2023-10-16T11:22:59.042 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116679.log 2023-10-16T11:22:59.042 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.143636.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.188623.log.gz 2023-10-16T11:22:59.042 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143636.log.gz 2023-10-16T11:22:59.043 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.140105.log 2023-10-16T11:22:59.043 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116679.log.gz 2023-10-16T11:22:59.043 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.167714.log 2023-10-16T11:22:59.043 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.140105.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.0.140105.log.gz--verbose 2023-10-16T11:22:59.044 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.141250.log 2023-10-16T11:22:59.044 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194871.log 2023-10-16T11:22:59.044 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.167714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.167714.log.gz 2023-10-16T11:22:59.044 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111211.log 2023-10-16T11:22:59.045 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194871.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.141250.log.gz 2023-10-16T11:22:59.045 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.194871.log.gz 2023-10-16T11:22:59.045 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194162.log 2023-10-16T11:22:59.046 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120284.log 2023-10-16T11:22:59.046 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194162.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.194162.log.gz 2023-10-16T11:22:59.046 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188204.log 2023-10-16T11:22:59.046 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120284.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.111211.log.gz 2023-10-16T11:22:59.047 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120284.log.gz 2023-10-16T11:22:59.047 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188204.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.188204.log.gz 2023-10-16T11:22:59.047 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139336.log 2023-10-16T11:22:59.047 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162197.log 2023-10-16T11:22:59.048 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139336.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139336.log.gz 2023-10-16T11:22:59.048 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162197.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.162197.log.gz 2023-10-16T11:22:59.048 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178736.log 2023-10-16T11:22:59.049 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139165.log 2023-10-16T11:22:59.049 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188861.log 2023-10-16T11:22:59.049 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139165.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.178736.log.gz 2023-10-16T11:22:59.049 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188861.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.139165.log.gz 2023-10-16T11:22:59.050 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.188861.log.gz 2023-10-16T11:22:59.050 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152222.log 2023-10-16T11:22:59.050 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164643.log 2023-10-16T11:22:59.050 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180972.log 2023-10-16T11:22:59.051 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.164643.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152222.log.gz 2023-10-16T11:22:59.051 INFO:teuthology.orchestra.run.smithi158.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.164643.log.gz 2023-10-16T11:22:59.051 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180972.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.180972.log.gz 2023-10-16T11:22:59.052 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184801.log 2023-10-16T11:22:59.052 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193892.log 2023-10-16T11:22:59.052 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176781.log 2023-10-16T11:22:59.052 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193892.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.184801.log.gz 2023-10-16T11:22:59.053 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110074.log 2023-10-16T11:22:59.053 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.193892.log.gz 2023-10-16T11:22:59.053 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128933.log 2023-10-16T11:22:59.053 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.176781.log.gz 2023-10-16T11:22:59.054 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130163.log 2023-10-16T11:22:59.054 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110074.log.gz 2023-10-16T11:22:59.054 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150581.log 2023-10-16T11:22:59.054 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.128933.log.gz 2023-10-16T11:22:59.055 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130163.log: /var/log/ceph/ceph-client.admin.150581.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.130163.log.gz 2023-10-16T11:22:59.055 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117860.log 2023-10-16T11:22:59.055 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.150581.log.gz 2023-10-16T11:22:59.055 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152031.log 2023-10-16T11:22:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185101.log 2023-10-16T11:22:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152031.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117860.log.gz 2023-10-16T11:22:59.056 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185101.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.152031.log.gz 2023-10-16T11:22:59.057 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.185101.log.gz 2023-10-16T11:22:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176622.log 2023-10-16T11:22:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.193017.log 2023-10-16T11:22:59.057 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149171.log 2023-10-16T11:22:59.058 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.176622.log.gz 2023-10-16T11:22:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.193017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.193017.log.gz 2023-10-16T11:22:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149171.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149171.log.gz 2023-10-16T11:22:59.058 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185965.log 2023-10-16T11:22:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138467.log 2023-10-16T11:22:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185965.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.185965.log.gz 2023-10-16T11:22:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138467.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.138467.log.gz 2023-10-16T11:22:59.059 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194432.log 2023-10-16T11:22:59.060 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117231.log 2023-10-16T11:22:59.060 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141350.log 2023-10-16T11:22:59.060 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117231.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.194432.log.gz 2023-10-16T11:22:59.060 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117231.log.gz 2023-10-16T11:22:59.061 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141350.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141350.log.gz 2023-10-16T11:22:59.061 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195331.log 2023-10-16T11:22:59.061 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168295.log 2023-10-16T11:22:59.061 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127823.log 2023-10-16T11:22:59.062 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.168295.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.195331.log.gz 2023-10-16T11:22:59.062 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0%/var/log/ceph/ceph-client.admin.127823.log: -- replaced with /var/log/ceph/ceph-client.admin.168295.log.gz 2023-10-16T11:22:59.062 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.127823.log.gz 2023-10-16T11:22:59.062 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163782.log 2023-10-16T11:22:59.063 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173620.log 2023-10-16T11:22:59.063 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137265.log 2023-10-16T11:22:59.063 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.173620.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.163782.log.gz 2023-10-16T11:22:59.063 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137265.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.173620.log.gz 2023-10-16T11:22:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178502.log 2023-10-16T11:22:59.064 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137265.log.gz 2023-10-16T11:22:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124625.log 2023-10-16T11:22:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136315.log 2023-10-16T11:22:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124625.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.178502.log.gz 2023-10-16T11:22:59.065 INFO:teuthology.orchestra.run.smithi158.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.124625.log.gz 2023-10-16T11:22:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136315.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.136315.log.gz 2023-10-16T11:22:59.065 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179479.log 2023-10-16T11:22:59.066 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162905.log 2023-10-16T11:22:59.066 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179479.log: /var/log/ceph/ceph-client.admin.162905.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179479.log.gz 2023-10-16T11:22:59.066 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.162905.log.gz 2023-10-16T11:22:59.066 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126770.log 2023-10-16T11:22:59.067 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123752.log 2023-10-16T11:22:59.067 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.126770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176714.log 2023-10-16T11:22:59.067 INFO:teuthology.orchestra.run.smithi158.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.126770.log.gz 2023-10-16T11:22:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.123752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.121093.log 2023-10-16T11:22:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176714.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123752.log.gz 2023-10-16T11:22:59.068 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.121093.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.176714.log.gz 2023-10-16T11:22:59.068 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.121093.log.gz 2023-10-16T11:22:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135662.log 2023-10-16T11:22:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129653.log 2023-10-16T11:22:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186693.log 2023-10-16T11:22:59.069 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.135662.log.gz 2023-10-16T11:22:59.070 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129653.log: /var/log/ceph/ceph-client.admin.186693.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.129653.log.gz 2023-10-16T11:22:59.070 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.186693.log.gz 2023-10-16T11:22:59.070 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139480.log 2023-10-16T11:22:59.070 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129607.log 2023-10-16T11:22:59.071 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139026.log 2023-10-16T11:22:59.071 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139480.log.gz 2023-10-16T11:22:59.071 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171077.log 2023-10-16T11:22:59.071 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139026.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.129607.log.gz 2023-10-16T11:22:59.072 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171077.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.139026.log.gz 2023-10-16T11:22:59.072 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.171077.log.gz 2023-10-16T11:22:59.072 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154644.log 2023-10-16T11:22:59.072 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140432.log 2023-10-16T11:22:59.073 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128240.log 2023-10-16T11:22:59.073 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154644.log.gz 2023-10-16T11:22:59.073 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140432.log: /var/log/ceph/ceph-client.admin.128240.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.140432.log.gz 2023-10-16T11:22:59.073 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128240.log.gz 2023-10-16T11:22:59.074 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142638.log 2023-10-16T11:22:59.074 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175743.log 2023-10-16T11:22:59.074 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182990.log 2023-10-16T11:22:59.074 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142638.log.gz 2023-10-16T11:22:59.075 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175743.log: /var/log/ceph/ceph-client.admin.182990.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.175743.log.gz 2023-10-16T11:22:59.075 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138180.log 2023-10-16T11:22:59.075 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.182990.log.gz 2023-10-16T11:22:59.075 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111622.log 2023-10-16T11:22:59.076 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195607.log 2023-10-16T11:22:59.076 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138180.log: /var/log/ceph/ceph-client.admin.111622.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.138180.log.gz 2023-10-16T11:22:59.076 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195607.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.111622.log.gz 2023-10-16T11:22:59.076 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.195607.log.gz 2023-10-16T11:22:59.077 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155413.log 2023-10-16T11:22:59.077 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177583.log 2023-10-16T11:22:59.077 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166427.log 2023-10-16T11:22:59.077 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155413.log.gz 2023-10-16T11:22:59.078 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177583.log: /var/log/ceph/ceph-client.admin.166427.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.177583.log.gz 2023-10-16T11:22:59.078 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.166427.log.gz 2023-10-16T11:22:59.078 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119046.log 2023-10-16T11:22:59.078 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163089.log 2023-10-16T11:22:59.079 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192784.log 2023-10-16T11:22:59.079 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.119046.log.gz 2023-10-16T11:22:59.079 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163089.log: /var/log/ceph/ceph-client.admin.192784.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.163089.log.gz 2023-10-16T11:22:59.079 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135054.log 2023-10-16T11:22:59.080 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.192784.log.gz 2023-10-16T11:22:59.080 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109932.log 2023-10-16T11:22:59.080 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122689.log 2023-10-16T11:22:59.081 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.109932.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.135054.log.gz 2023-10-16T11:22:59.081 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172638.log 2023-10-16T11:22:59.081 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.109932.log.gz 2023-10-16T11:22:59.081 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139618.log 2023-10-16T11:22:59.082 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172638.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122689.log.gz 2023-10-16T11:22:59.082 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.172638.log.gz 2023-10-16T11:22:59.082 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139618.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.139618.log.gz 2023-10-16T11:22:59.082 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186934.log 2023-10-16T11:22:59.083 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112098.log 2023-10-16T11:22:59.083 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141500.log 2023-10-16T11:22:59.083 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112098.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.186934.log.gz 2023-10-16T11:22:59.083 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112098.log.gz 2023-10-16T11:22:59.084 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141500.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.141500.log.gz 2023-10-16T11:22:59.084 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133864.log 2023-10-16T11:22:59.084 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177675.log 2023-10-16T11:22:59.084 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.133864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190808.log 2023-10-16T11:22:59.085 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177675.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133864.log.gz 2023-10-16T11:22:59.085 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.177675.log.gz 2023-10-16T11:22:59.085 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190808.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.190808.log.gz 2023-10-16T11:22:59.085 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165930.log 2023-10-16T11:22:59.086 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137882.log 2023-10-16T11:22:59.086 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124096.log 2023-10-16T11:22:59.086 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137882.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.165930.log.gz 2023-10-16T11:22:59.087 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.137882.log.gz 2023-10-16T11:22:59.087 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124096.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124096.log.gz 2023-10-16T11:22:59.087 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188669.log 2023-10-16T11:22:59.087 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166983.log 2023-10-16T11:22:59.088 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188669.log: /var/log/ceph/ceph-client.admin.166983.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.188669.log.gz 2023-10-16T11:22:59.088 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.166983.log.gz 2023-10-16T11:22:59.088 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175996.log 2023-10-16T11:22:59.089 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184291.log 2023-10-16T11:22:59.089 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167123.log 2023-10-16T11:22:59.089 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.175996.log.gz 2023-10-16T11:22:59.089 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184291.log: /var/log/ceph/ceph-client.admin.167123.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.184291.log.gz 2023-10-16T11:22:59.090 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.167123.log.gz 2023-10-16T11:22:59.090 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192300.log 2023-10-16T11:22:59.090 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130255.log 2023-10-16T11:22:59.090 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127688.log 2023-10-16T11:22:59.091 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.192300.log.gz 2023-10-16T11:22:59.091 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130255.log: /var/log/ceph/ceph-client.admin.127688.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.130255.log.gz 2023-10-16T11:22:59.091 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.127688.log.gz 2023-10-16T11:22:59.091 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117450.log 2023-10-16T11:22:59.092 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186980.log 2023-10-16T11:22:59.092 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128646.log 2023-10-16T11:22:59.092 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117450.log.gz 2023-10-16T11:22:59.093 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186980.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-16T11:22:59.093 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.186980.log.gz 2023-10-16T11:22:59.093 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195239.log 2023-10-16T11:22:59.093 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.128646.log.gz 2023-10-16T11:22:59.094 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135718.log 2023-10-16T11:22:59.094 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195239.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.195239.log.gz 2023-10-16T11:22:59.094 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135718.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135718.log.gz 2023-10-16T11:22:59.094 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119230.log 2023-10-16T11:22:59.095 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153765.log 2023-10-16T11:22:59.095 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119230.log: 88.7%/var/log/ceph/ceph-client.admin.153765.log: -- replaced with /var/log/ceph/ceph-client.admin.119230.log.gz 2023-10-16T11:22:59.095 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153765.log.gz 2023-10-16T11:22:59.095 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113317.log 2023-10-16T11:22:59.096 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189649.log 2023-10-16T11:22:59.096 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113317.log: 87.7%/var/log/ceph/ceph-client.admin.189649.log: -- replaced with /var/log/ceph/ceph-client.admin.113317.log.gz 2023-10-16T11:22:59.096 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.189649.log.gz 2023-10-16T11:22:59.096 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177629.log 2023-10-16T11:22:59.097 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.121097.log 2023-10-16T11:22:59.097 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177629.log: 89.0%/var/log/ceph/ceph-client.0.121097.log: -- replaced with /var/log/ceph/ceph-client.admin.177629.log.gz 2023-10-16T11:22:59.097 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.121097.log.gz 2023-10-16T11:22:59.097 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118816.log 2023-10-16T11:22:59.098 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136658.log 2023-10-16T11:22:59.098 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118816.log: /var/log/ceph/ceph-client.admin.136658.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118816.log.gz 2023-10-16T11:22:59.098 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.136658.log.gz 2023-10-16T11:22:59.099 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119958.log 2023-10-16T11:22:59.099 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141099.log 2023-10-16T11:22:59.099 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119958.log: /var/log/ceph/ceph-client.admin.141099.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119958.log.gz 2023-10-16T11:22:59.099 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141099.log.gz 2023-10-16T11:22:59.100 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118724.log 2023-10-16T11:22:59.100 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110174.log 2023-10-16T11:22:59.100 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118724.log: /var/log/ceph/ceph-client.admin.110174.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118724.log.gz 2023-10-16T11:22:59.100 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110174.log.gz 2023-10-16T11:22:59.101 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150334.log 2023-10-16T11:22:59.101 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164597.log 2023-10-16T11:22:59.101 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150334.log: /var/log/ceph/ceph-client.admin.164597.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.150334.log.gz 2023-10-16T11:22:59.101 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.164597.log.gz 2023-10-16T11:22:59.102 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124030.log 2023-10-16T11:22:59.102 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134629.log 2023-10-16T11:22:59.102 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124030.log: /var/log/ceph/ceph-client.admin.134629.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124030.log.gz 2023-10-16T11:22:59.102 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.134629.log.gz 2023-10-16T11:22:59.103 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172362.log 2023-10-16T11:22:59.103 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149936.log 2023-10-16T11:22:59.103 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172362.log: /var/log/ceph/ceph-client.admin.149936.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.172362.log.gz 2023-10-16T11:22:59.103 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149936.log.gz 2023-10-16T11:22:59.104 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153865.log 2023-10-16T11:22:59.104 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178364.log 2023-10-16T11:22:59.104 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153865.log: 88.7%/var/log/ceph/ceph-client.admin.178364.log: -- replaced with /var/log/ceph/ceph-client.admin.153865.log.gz 2023-10-16T11:22:59.104 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.178364.log.gz 2023-10-16T11:22:59.105 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142058.log 2023-10-16T11:22:59.105 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196021.log 2023-10-16T11:22:59.105 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142058.log: /var/log/ceph/ceph-client.admin.196021.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.142058.log.gz 2023-10-16T11:22:59.105 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.196021.log.gz 2023-10-16T11:22:59.106 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116633.log 2023-10-16T11:22:59.106 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117093.log 2023-10-16T11:22:59.106 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116633.log: /var/log/ceph/ceph-client.admin.117093.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.116633.log.gz 2023-10-16T11:22:59.107 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.117093.log.gz 2023-10-16T11:22:59.107 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195745.log 2023-10-16T11:22:59.107 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151229.log 2023-10-16T11:22:59.107 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195745.log: /var/log/ceph/ceph-client.admin.151229.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.195745.log.gz 2023-10-16T11:22:59.108 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.151229.log.gz 2023-10-16T11:22:59.108 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116955.log 2023-10-16T11:22:59.108 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176045.log 2023-10-16T11:22:59.108 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116955.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116955.log.gz 2023-10-16T11:22:59.109 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176045.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.176045.log.gz 2023-10-16T11:22:59.109 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137932.log 2023-10-16T11:22:59.109 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192112.log 2023-10-16T11:22:59.109 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137932.log: /var/log/ceph/ceph-client.admin.192112.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137932.log.gz 2023-10-16T11:22:59.110 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.192112.log.gz 2023-10-16T11:22:59.110 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195469.log 2023-10-16T11:22:59.110 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123660.log 2023-10-16T11:22:59.110 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195469.log: /var/log/ceph/ceph-client.admin.123660.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.195469.log.gz 2023-10-16T11:22:59.111 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123660.log.gz 2023-10-16T11:22:59.111 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167169.log 2023-10-16T11:22:59.111 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144359.log 2023-10-16T11:22:59.111 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167169.log: /var/log/ceph/ceph-client.admin.144359.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.167169.log.gz 2023-10-16T11:22:59.112 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.144359.log.gz 2023-10-16T11:22:59.112 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130760.log 2023-10-16T11:22:59.112 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179906.log 2023-10-16T11:22:59.112 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130760.log: /var/log/ceph/ceph-client.admin.179906.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130760.log.gz 2023-10-16T11:22:59.113 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.179906.log.gz 2023-10-16T11:22:59.113 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187302.log 2023-10-16T11:22:59.113 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145808.log 2023-10-16T11:22:59.113 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187302.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.187302.log.gz/var/log/ceph/ceph-client.admin.145808.log: 2023-10-16T11:22:59.114 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145808.log.gz 2023-10-16T11:22:59.114 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136704.log 2023-10-16T11:22:59.114 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120146.log 2023-10-16T11:22:59.114 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136704.log: /var/log/ceph/ceph-client.admin.120146.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.136704.log.gz 2023-10-16T11:22:59.115 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120146.log.gz 2023-10-16T11:22:59.115 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153548.log 2023-10-16T11:22:59.115 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191552.log 2023-10-16T11:22:59.115 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153548.log: /var/log/ceph/ceph-client.admin.191552.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153548.log.gz 2023-10-16T11:22:59.116 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.191552.log.gz 2023-10-16T11:22:59.116 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123522.log 2023-10-16T11:22:59.116 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186647.log 2023-10-16T11:22:59.116 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.123522.log: /var/log/ceph/ceph-client.admin.186647.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123522.log.gz 2023-10-16T11:22:59.117 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.186647.log.gz 2023-10-16T11:22:59.117 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128841.log 2023-10-16T11:22:59.117 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116449.log 2023-10-16T11:22:59.117 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128841.log: /var/log/ceph/ceph-client.admin.116449.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128841.log.gz 2023-10-16T11:22:59.118 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116449.log.gz 2023-10-16T11:22:59.118 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171857.log 2023-10-16T11:22:59.118 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146610.log 2023-10-16T11:22:59.118 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171857.log: /var/log/ceph/ceph-client.admin.146610.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.171857.log.gz 2023-10-16T11:22:59.119 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.146610.log.gz 2023-10-16T11:22:59.119 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144181.log 2023-10-16T11:22:59.119 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196226.log 2023-10-16T11:22:59.119 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144181.log: /var/log/ceph/ceph-client.admin.196226.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.144181.log.gz 2023-10-16T11:22:59.120 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.196226.log.gz 2023-10-16T11:22:59.120 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145007.log 2023-10-16T11:22:59.120 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110519.log 2023-10-16T11:22:59.120 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145007.log: /var/log/ceph/ceph-client.admin.110519.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145007.log.gz 2023-10-16T11:22:59.121 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110519.log.gz 2023-10-16T11:22:59.121 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114419.log 2023-10-16T11:22:59.121 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180823.log 2023-10-16T11:22:59.121 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114419.log: /var/log/ceph/ceph-client.admin.180823.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114419.log.gz 2023-10-16T11:22:59.122 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.180823.log.gz 2023-10-16T11:22:59.122 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176530.log 2023-10-16T11:22:59.122 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112512.log 2023-10-16T11:22:59.122 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176530.log: /var/log/ceph/ceph-client.admin.112512.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.176530.log.gz 2023-10-16T11:22:59.123 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112512.log.gz 2023-10-16T11:22:59.123 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142150.log 2023-10-16T11:22:59.123 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163298.log 2023-10-16T11:22:59.123 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142150.log: /var/log/ceph/ceph-client.admin.163298.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142150.log.gz 2023-10-16T11:22:59.124 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.163298.log.gz 2023-10-16T11:22:59.124 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117952.log 2023-10-16T11:22:59.124 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187716.log 2023-10-16T11:22:59.125 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117952.log: /var/log/ceph/ceph-client.admin.187716.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117952.log.gz 2023-10-16T11:22:59.125 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.187716.log.gz 2023-10-16T11:22:59.125 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193794.log 2023-10-16T11:22:59.125 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155271.log 2023-10-16T11:22:59.125 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193794.log: /var/log/ceph/ceph-client.admin.155271.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.193794.log.gz 2023-10-16T11:22:59.126 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.155271.log.gz 2023-10-16T11:22:59.126 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129454.log 2023-10-16T11:22:59.126 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195515.log 2023-10-16T11:22:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129454.log: /var/log/ceph/ceph-client.admin.195515.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.129454.log.gz 2023-10-16T11:22:59.127 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.195515.log.gz 2023-10-16T11:22:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194963.log 2023-10-16T11:22:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173165.log 2023-10-16T11:22:59.129 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194963.log: /var/log/ceph/ceph-client.admin.173165.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.194963.log.gz 2023-10-16T11:22:59.129 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.173165.log.gz 2023-10-16T11:22:59.129 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191506.log 2023-10-16T11:22:59.130 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146750.log 2023-10-16T11:22:59.130 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191506.log: /var/log/ceph/ceph-client.admin.146750.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.191506.log.gz 2023-10-16T11:22:59.130 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146750.log.gz 2023-10-16T11:22:59.130 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136245.log 2023-10-16T11:22:59.131 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146842.log 2023-10-16T11:22:59.131 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136245.log: /var/log/ceph/ceph-client.admin.146842.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146842.log.gz 2023-10-16T11:22:59.131 INFO:teuthology.orchestra.run.smithi158.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.136245.log.gz 2023-10-16T11:22:59.131 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194676.log 2023-10-16T11:22:59.132 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138421.log 2023-10-16T11:22:59.132 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194676.log: /var/log/ceph/ceph-client.admin.138421.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.194676.log.gz 2023-10-16T11:22:59.132 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.138421.log.gz 2023-10-16T11:22:59.132 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188343.log 2023-10-16T11:22:59.133 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183555.log 2023-10-16T11:22:59.133 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188343.log: /var/log/ceph/ceph-client.admin.183555.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.188343.log.gz 2023-10-16T11:22:59.133 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.183555.log.gz 2023-10-16T11:22:59.134 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137629.log 2023-10-16T11:22:59.134 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183131.log 2023-10-16T11:22:59.134 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137629.log: /var/log/ceph/ceph-client.admin.183131.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.137629.log.gz 2023-10-16T11:22:59.134 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.183131.log.gz 2023-10-16T11:22:59.135 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172408.log 2023-10-16T11:22:59.135 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182225.log 2023-10-16T11:22:59.135 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172408.log: /var/log/ceph/ceph-client.admin.182225.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.172408.log.gz 2023-10-16T11:22:59.135 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.182225.log.gz 2023-10-16T11:22:59.136 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.193021.log 2023-10-16T11:22:59.136 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168947.log 2023-10-16T11:22:59.136 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.193021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.193021.log.gz 2023-10-16T11:22:59.136 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.168947.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.168947.log.gz 2023-10-16T11:22:59.137 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170300.log 2023-10-16T11:22:59.137 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176243.log 2023-10-16T11:22:59.137 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170300.log: /var/log/ceph/ceph-client.admin.176243.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.170300.log.gz 2023-10-16T11:22:59.137 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.176243.log.gz 2023-10-16T11:22:59.138 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176668.log 2023-10-16T11:22:59.138 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186076.log 2023-10-16T11:22:59.138 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176668.log: /var/log/ceph/ceph-client.admin.186076.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.176668.log.gz 2023-10-16T11:22:59.138 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.186076.log.gz 2023-10-16T11:22:59.139 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.140109.log 2023-10-16T11:22:59.139 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184337.log 2023-10-16T11:22:59.139 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.140109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.140109.log.gz 2023-10-16T11:22:59.140 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184337.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.184337.log.gz 2023-10-16T11:22:59.140 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155644.log 2023-10-16T11:22:59.140 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178410.log 2023-10-16T11:22:59.140 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155644.log: /var/log/ceph/ceph-client.admin.178410.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.155644.log.gz 2023-10-16T11:22:59.141 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.178410.log.gz 2023-10-16T11:22:59.141 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151985.log 2023-10-16T11:22:59.141 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.164019.log 2023-10-16T11:22:59.141 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151985.log: /var/log/ceph/ceph-client.0.164019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151985.log.gz 2023-10-16T11:22:59.142 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.164019.log.gz 2023-10-16T11:22:59.142 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137732.log 2023-10-16T11:22:59.142 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185528.log 2023-10-16T11:22:59.142 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137732.log: /var/log/ceph/ceph-client.admin.185528.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137732.log.gz 2023-10-16T11:22:59.143 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.185528.log.gz 2023-10-16T11:22:59.143 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171521.log 2023-10-16T11:22:59.143 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111871.log 2023-10-16T11:22:59.143 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171521.log: /var/log/ceph/ceph-client.admin.111871.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.171521.log.gz 2023-10-16T11:22:59.144 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111871.log.gz 2023-10-16T11:22:59.144 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119774.log 2023-10-16T11:22:59.144 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112712.log 2023-10-16T11:22:59.145 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119774.log: /var/log/ceph/ceph-client.admin.112712.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119774.log.gz 2023-10-16T11:22:59.145 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112712.log.gz 2023-10-16T11:22:59.145 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166787.log 2023-10-16T11:22:59.145 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191106.log 2023-10-16T11:22:59.146 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166787.log: /var/log/ceph/ceph-client.admin.191106.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.166787.log.gz 2023-10-16T11:22:59.146 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.191106.log.gz 2023-10-16T11:22:59.146 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195193.log 2023-10-16T11:22:59.146 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179425.log 2023-10-16T11:22:59.147 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195193.log: /var/log/ceph/ceph-client.admin.179425.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.195193.log.gz 2023-10-16T11:22:59.147 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.179425.log.gz 2023-10-16T11:22:59.147 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123476.log 2023-10-16T11:22:59.147 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172592.log 2023-10-16T11:22:59.148 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.123476.log: /var/log/ceph/ceph-client.admin.172592.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.123476.log.gz 2023-10-16T11:22:59.148 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.172592.log.gz 2023-10-16T11:22:59.148 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182179.log 2023-10-16T11:22:59.149 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171767.log 2023-10-16T11:22:59.149 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182179.log: /var/log/ceph/ceph-client.admin.171767.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.182179.log.gz 2023-10-16T11:22:59.149 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.171767.log.gz 2023-10-16T11:22:59.149 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124228.log 2023-10-16T11:22:59.150 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143287.log 2023-10-16T11:22:59.150 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124228.log: /var/log/ceph/ceph-client.admin.143287.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.124228.log.gz 2023-10-16T11:22:59.150 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.143287.log.gz 2023-10-16T11:22:59.150 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167353.log 2023-10-16T11:22:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166741.log 2023-10-16T11:22:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167353.log: /var/log/ceph/ceph-client.admin.166741.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.167353.log.gz 2023-10-16T11:22:59.151 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.166741.log.gz 2023-10-16T11:22:59.151 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142779.log 2023-10-16T11:22:59.152 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181432.log 2023-10-16T11:22:59.152 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142779.log: /var/log/ceph/ceph-client.admin.181432.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142779.log.gz 2023-10-16T11:22:59.152 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.181432.log.gz 2023-10-16T11:22:59.152 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195699.log 2023-10-16T11:22:59.153 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120238.log 2023-10-16T11:22:59.153 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195699.log: /var/log/ceph/ceph-client.admin.120238.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.195699.log.gz 2023-10-16T11:22:59.153 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.120238.log.gz 2023-10-16T11:22:59.153 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172176.log 2023-10-16T11:22:59.154 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178318.log 2023-10-16T11:22:59.154 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172176.log: /var/log/ceph/ceph-client.admin.178318.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.172176.log.gz 2023-10-16T11:22:59.154 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.178318.log.gz 2023-10-16T11:22:59.154 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188769.log 2023-10-16T11:22:59.155 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130872.log 2023-10-16T11:22:59.155 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188769.log: /var/log/ceph/ceph-client.admin.130872.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.188769.log.gz 2023-10-16T11:22:59.155 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.130872.log.gz 2023-10-16T11:22:59.156 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165053.log 2023-10-16T11:22:59.156 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122595.log 2023-10-16T11:22:59.156 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165053.log: /var/log/ceph/ceph-client.admin.122595.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.165053.log.gz 2023-10-16T11:22:59.156 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.122595.log.gz 2023-10-16T11:22:59.157 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150058.log 2023-10-16T11:22:59.157 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190057.log 2023-10-16T11:22:59.157 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150058.log: /var/log/ceph/ceph-client.admin.190057.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.150058.log.gz 2023-10-16T11:22:59.157 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.190057.log.gz 2023-10-16T11:22:59.158 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190759.log 2023-10-16T11:22:59.158 INFO:teuthology.orchestra.run.smithi158.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-16T11:22:59.158 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185009.log 2023-10-16T11:22:59.158 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110635.log 2023-10-16T11:22:59.159 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185009.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.190759.log.gz 2023-10-16T11:22:59.159 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.185009.log.gz 2023-10-16T11:22:59.159 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139802.log 2023-10-16T11:22:59.159 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166937.log 2023-10-16T11:22:59.160 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139802.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110635.log.gz 2023-10-16T11:22:59.160 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166937.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.139802.log.gz 2023-10-16T11:22:59.160 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.166937.log.gz 2023-10-16T11:22:59.160 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147722.log 2023-10-16T11:22:59.161 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118770.log 2023-10-16T11:22:59.161 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.147722.log: /var/log/ceph/ceph-client.admin.118770.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147722.log.gz 2023-10-16T11:22:59.161 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118770.log.gz 2023-10-16T11:22:59.161 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150941.log 2023-10-16T11:22:59.162 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137580.log 2023-10-16T11:22:59.162 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153715.log 2023-10-16T11:22:59.162 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150941.log.gz 2023-10-16T11:22:59.163 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120100.log 2023-10-16T11:22:59.163 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153715.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137580.log.gz 2023-10-16T11:22:59.163 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120100.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.153715.log.gz 2023-10-16T11:22:59.163 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120100.log.gz 2023-10-16T11:22:59.164 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155459.log 2023-10-16T11:22:59.164 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117588.log 2023-10-16T11:22:59.164 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153818.log 2023-10-16T11:22:59.164 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.155459.log.gz 2023-10-16T11:22:59.164 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117588.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T11:22:59.165 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117588.log.gz 2023-10-16T11:22:59.165 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153818.log: /var/log/ceph/ceph.audit.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.153818.log.gz 2023-10-16T11:22:59.165 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163666.log 2023-10-16T11:22:59.165 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139119.log 2023-10-16T11:22:59.166 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163666.log: /var/log/ceph/ceph-client.admin.139119.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.163666.log.gz 2023-10-16T11:22:59.166 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.139119.log.gz 2023-10-16T11:22:59.166 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.156177.log 2023-10-16T11:22:59.167 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172035.log 2023-10-16T11:22:59.167 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.156177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.156177.log.gz 2023-10-16T11:22:59.167 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172035.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.172035.log.gz 2023-10-16T11:22:59.167 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165433.log 2023-10-16T11:22:59.168 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129163.log 2023-10-16T11:22:59.168 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165433.log: /var/log/ceph/ceph-client.admin.129163.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.165433.log.gz 2023-10-16T11:22:59.168 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.129163.log.gz 2023-10-16T11:22:59.169 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177203.log 2023-10-16T11:22:59.169 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109886.log 2023-10-16T11:22:59.169 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177203.log: /var/log/ceph/ceph-client.admin.109886.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.177203.log.gz 2023-10-16T11:22:59.169 INFO:teuthology.orchestra.run.smithi158.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.109886.log.gz 2023-10-16T11:22:59.170 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113984.log 2023-10-16T11:22:59.170 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177767.log 2023-10-16T11:22:59.170 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113984.log: 89.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T11:22:59.170 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177767.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113984.log.gz 2023-10-16T11:22:59.171 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.177767.log.gz 2023-10-16T11:22:59.171 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177721.log 2023-10-16T11:22:59.171 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116218.log 2023-10-16T11:22:59.171 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177721.log: /var/log/ceph/ceph-client.admin.116218.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.177721.log.gz 2023-10-16T11:22:59.172 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116218.log.gz 2023-10-16T11:22:59.172 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155874.log 2023-10-16T11:22:59.172 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151413.log 2023-10-16T11:22:59.172 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155828.log 2023-10-16T11:22:59.173 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151413.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.155874.log.gz 2023-10-16T11:22:59.173 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.151413.log.gz 2023-10-16T11:22:59.173 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171429.log 2023-10-16T11:22:59.173 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151321.log 2023-10-16T11:22:59.174 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.155828.log.gz 2023-10-16T11:22:59.174 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171429.log: /var/log/ceph/ceph-client.admin.151321.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.171429.log.gz 2023-10-16T11:22:59.174 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.151321.log.gz 2023-10-16T11:22:59.175 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103816.log 2023-10-16T11:22:59.175 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143640.log 2023-10-16T11:22:59.175 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.103816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184939.log 2023-10-16T11:22:59.175 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.143640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143640.log.gz 2023-10-16T11:22:59.176 INFO:teuthology.orchestra.run.smithi158.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.103816.log.gz 2023-10-16T11:22:59.176 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184939.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.184939.log.gz 2023-10-16T11:22:59.176 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182723.log 2023-10-16T11:22:59.176 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115988.log 2023-10-16T11:22:59.177 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144689.log 2023-10-16T11:22:59.177 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.115988.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.182723.log.gz 2023-10-16T11:22:59.177 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155505.log 2023-10-16T11:22:59.177 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115988.log.gz 2023-10-16T11:22:59.178 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144689.log: /var/log/ceph/ceph-client.admin.155505.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144689.log.gz 2023-10-16T11:22:59.178 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.155505.log.gz 2023-10-16T11:22:59.178 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136758.log 2023-10-16T11:22:59.178 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154598.log 2023-10-16T11:22:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166883.log 2023-10-16T11:22:59.179 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6%/var/log/ceph/ceph-client.admin.154598.log: -- replaced with /var/log/ceph/ceph-client.admin.136758.log.gz 2023-10-16T11:22:59.179 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162819.log 2023-10-16T11:22:59.180 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166883.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154598.log.gz 2023-10-16T11:22:59.180 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.166883.log.gz 2023-10-16T11:22:59.180 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110381.log 2023-10-16T11:22:59.180 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135764.log 2023-10-16T11:22:59.181 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.162819.log.gz 2023-10-16T11:22:59.181 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110381.log: /var/log/ceph/ceph-client.admin.135764.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110381.log.gz 2023-10-16T11:22:59.181 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135764.log.gz 2023-10-16T11:22:59.181 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122266.log 2023-10-16T11:22:59.182 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155086.log 2023-10-16T11:22:59.182 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117634.log 2023-10-16T11:22:59.182 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122266.log.gz 2023-10-16T11:22:59.182 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147651.log 2023-10-16T11:22:59.183 INFO:teuthology.orchestra.run.smithi158.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.155086.log.gz 2023-10-16T11:22:59.183 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117634.log: /var/log/ceph/ceph-client.admin.147651.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117634.log.gz 2023-10-16T11:22:59.183 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147651.log.gz 2023-10-16T11:22:59.183 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140648.log 2023-10-16T11:22:59.184 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140783.log 2023-10-16T11:22:59.184 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117496.log 2023-10-16T11:22:59.184 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.140648.log.gz 2023-10-16T11:22:59.184 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187532.log 2023-10-16T11:22:59.185 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117496.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.140783.log.gz 2023-10-16T11:22:59.185 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187532.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117496.log.gz 2023-10-16T11:22:59.185 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.187532.log.gz 2023-10-16T11:22:59.185 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142919.log 2023-10-16T11:22:59.186 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178966.log 2023-10-16T11:22:59.186 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188577.log 2023-10-16T11:22:59.186 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.142919.log.gz 2023-10-16T11:22:59.186 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178966.log: /var/log/ceph/ceph-client.admin.188577.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.178966.log.gz 2023-10-16T11:22:59.187 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177813.log 2023-10-16T11:22:59.187 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.188577.log.gz 2023-10-16T11:22:59.187 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182569.log 2023-10-16T11:22:59.187 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113076.log 2023-10-16T11:22:59.188 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182569.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.177813.log.gz 2023-10-16T11:22:59.188 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113076.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.182569.log.gz 2023-10-16T11:22:59.188 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113076.log.gz 2023-10-16T11:22:59.188 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120514.log 2023-10-16T11:22:59.189 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144405.log 2023-10-16T11:22:59.189 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196272.log 2023-10-16T11:22:59.189 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.120514.log.gz 2023-10-16T11:22:59.189 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144405.log: /var/log/ceph/ceph-client.admin.196272.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144405.log.gz 2023-10-16T11:22:59.190 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.196272.log.gz 2023-10-16T11:22:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178920.log 2023-10-16T11:22:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134182.log 2023-10-16T11:22:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113938.log 2023-10-16T11:22:59.191 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.178920.log.gz 2023-10-16T11:22:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134182.log: /var/log/ceph/ceph-client.admin.113938.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.134182.log.gz 2023-10-16T11:22:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149021.log 2023-10-16T11:22:59.192 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.113938.log.gz 2023-10-16T11:22:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122360.log 2023-10-16T11:22:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145624.log 2023-10-16T11:22:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122360.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.149021.log.gz 2023-10-16T11:22:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145624.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122360.log.gz 2023-10-16T11:22:59.193 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.145624.log.gz 2023-10-16T11:22:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116034.log 2023-10-16T11:22:59.193 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145207.log 2023-10-16T11:22:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137415.log 2023-10-16T11:22:59.194 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116034.log.gz 2023-10-16T11:22:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194311.log 2023-10-16T11:22:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129209.log 2023-10-16T11:22:59.195 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137415.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145207.log.gz 2023-10-16T11:22:59.195 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164127.log 2023-10-16T11:22:59.195 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137415.log.gz 2023-10-16T11:22:59.195 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.194311.log.gz 2023-10-16T11:22:59.196 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129209.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.129209.log.gz 2023-10-16T11:22:59.196 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.164127.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.164127.log.gz 2023-10-16T11:22:59.196 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167029.log 2023-10-16T11:22:59.196 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142461.log 2023-10-16T11:22:59.197 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167029.log: /var/log/ceph/ceph-client.admin.142461.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.167029.log.gz 2023-10-16T11:22:59.197 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142461.log.gz 2023-10-16T11:22:59.197 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118611.log 2023-10-16T11:22:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110473.log 2023-10-16T11:22:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129071.log 2023-10-16T11:22:59.198 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118611.log.gz 2023-10-16T11:22:59.198 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182317.log 2023-10-16T11:22:59.199 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110473.log.gz 2023-10-16T11:22:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129071.log: /var/log/ceph/ceph-client.admin.182317.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129071.log.gz 2023-10-16T11:22:59.199 INFO:teuthology.orchestra.run.smithi158.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.182317.log.gz 2023-10-16T11:22:59.199 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149537.log 2023-10-16T11:22:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120860.log 2023-10-16T11:22:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141820.log 2023-10-16T11:22:59.200 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149537.log.gz 2023-10-16T11:22:59.200 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172684.log 2023-10-16T11:22:59.201 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.120860.log.gz 2023-10-16T11:22:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124294.log 2023-10-16T11:22:59.201 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172684.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.141820.log.gz 2023-10-16T11:22:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124294.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.172684.log.gz 2023-10-16T11:22:59.202 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.124294.log.gz 2023-10-16T11:22:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116909.log 2023-10-16T11:22:59.202 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146100.log 2023-10-16T11:22:59.203 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116909.log: /var/log/ceph/ceph-client.admin.146100.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116909.log.gz 2023-10-16T11:22:59.203 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.146100.log.gz 2023-10-16T11:22:59.203 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152527.log 2023-10-16T11:22:59.203 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181072.log 2023-10-16T11:22:59.204 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152527.log: /var/log/ceph/ceph-client.admin.181072.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.152527.log.gz 2023-10-16T11:22:59.204 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.181072.log.gz 2023-10-16T11:22:59.204 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166114.log 2023-10-16T11:22:59.204 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111963.log 2023-10-16T11:22:59.205 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166114.log: /var/log/ceph/ceph-client.admin.111963.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.166114.log.gz 2023-10-16T11:22:59.205 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111963.log.gz 2023-10-16T11:22:59.205 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117542.log 2023-10-16T11:22:59.205 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117906.log 2023-10-16T11:22:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116402.log 2023-10-16T11:22:59.206 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117542.log.gz 2023-10-16T11:22:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186739.log 2023-10-16T11:22:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116402.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117906.log.gz 2023-10-16T11:22:59.207 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186739.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116402.log.gz 2023-10-16T11:22:59.207 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.186739.log.gz 2023-10-16T11:22:59.207 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123614.log 2023-10-16T11:22:59.208 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114214.log 2023-10-16T11:22:59.208 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.123614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162299.log 2023-10-16T11:22:59.208 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114214.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123614.log.gz 2023-10-16T11:22:59.208 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136086.log 2023-10-16T11:22:59.209 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114214.log.gz 2023-10-16T11:22:59.209 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162299.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.162299.log.gz 2023-10-16T11:22:59.209 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165781.log 2023-10-16T11:22:59.209 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180149.log 2023-10-16T11:22:59.210 INFO:teuthology.orchestra.run.smithi158.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.136086.log.gz 2023-10-16T11:22:59.210 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165781.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.165781.log.gz 2023-10-16T11:22:59.210 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128441.log 2023-10-16T11:22:59.210 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134129.log 2023-10-16T11:22:59.211 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7%/var/log/ceph/ceph-client.admin.128441.log: -- replaced with /var/log/ceph/ceph-client.admin.180149.log.gz 2023-10-16T11:22:59.211 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143746.log 2023-10-16T11:22:59.211 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128441.log.gz 2023-10-16T11:22:59.211 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113081.log 2023-10-16T11:22:59.212 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134129.log.gz 2023-10-16T11:22:59.212 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143746.log: /var/log/ceph/ceph-client.admin.113081.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.143746.log.gz 2023-10-16T11:22:59.212 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113081.log.gz 2023-10-16T11:22:59.212 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161755.log 2023-10-16T11:22:59.213 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146796.log 2023-10-16T11:22:59.213 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113448.log 2023-10-16T11:22:59.213 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.161755.log.gz 2023-10-16T11:22:59.213 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139526.log 2023-10-16T11:22:59.214 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146796.log.gz 2023-10-16T11:22:59.214 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113448.log: /var/log/ceph/ceph-client.admin.139526.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113448.log.gz 2023-10-16T11:22:59.214 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139526.log.gz 2023-10-16T11:22:59.215 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127382.log 2023-10-16T11:22:59.215 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188953.log 2023-10-16T11:22:59.215 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.127382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142592.log 2023-10-16T11:22:59.215 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127382.log.gz 2023-10-16T11:22:59.216 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188953.log: /var/log/ceph/ceph-client.admin.142592.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.188953.log.gz 2023-10-16T11:22:59.216 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142592.log.gz 2023-10-16T11:22:59.216 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195883.log 2023-10-16T11:22:59.216 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192066.log 2023-10-16T11:22:59.217 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123568.log 2023-10-16T11:22:59.217 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.195883.log.gz 2023-10-16T11:22:59.217 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192066.log: /var/log/ceph/ceph-client.admin.123568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188999.log 2023-10-16T11:22:59.217 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.192066.log.gz 2023-10-16T11:22:59.218 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123568.log.gz 2023-10-16T11:22:59.218 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188999.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.188999.log.gz 2023-10-16T11:22:59.218 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149337.log 2023-10-16T11:22:59.218 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150104.log 2023-10-16T11:22:59.219 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119728.log 2023-10-16T11:22:59.219 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.149337.log.gz 2023-10-16T11:22:59.219 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179953.log 2023-10-16T11:22:59.219 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150104.log.gz 2023-10-16T11:22:59.220 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119728.log: /var/log/ceph/ceph-client.admin.179953.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119728.log.gz 2023-10-16T11:22:59.220 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.179953.log.gz 2023-10-16T11:22:59.220 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187394.log 2023-10-16T11:22:59.220 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118473.log 2023-10-16T11:22:59.221 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156379.log 2023-10-16T11:22:59.221 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.187394.log.gz 2023-10-16T11:22:59.221 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118473.log: /var/log/ceph/ceph-client.admin.156379.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118473.log.gz 2023-10-16T11:22:59.221 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.156379.log.gz 2023-10-16T11:22:59.222 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136548.log 2023-10-16T11:22:59.222 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188389.log 2023-10-16T11:22:59.222 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.136548.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.0.136548.log.gz-5 2023-10-16T11:22:59.222 INFO:teuthology.orchestra.run.smithi158.stderr:--verbose -- /var/log/ceph/ceph-client.admin.180010.log 2023-10-16T11:22:59.223 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188389.log: /var/log/ceph/ceph-client.admin.180010.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.188389.log.gz 2023-10-16T11:22:59.223 INFO:teuthology.orchestra.run.smithi158.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.180010.log.gz 2023-10-16T11:22:59.223 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185662.log 2023-10-16T11:22:59.223 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116541.log 2023-10-16T11:22:59.224 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185812.log 2023-10-16T11:22:59.224 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.185662.log.gz 2023-10-16T11:22:59.224 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116541.log: /var/log/ceph/ceph-client.admin.185812.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116541.log.gz 2023-10-16T11:22:59.224 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121931.log 2023-10-16T11:22:59.225 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.185812.log.gz 2023-10-16T11:22:59.225 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138605.log 2023-10-16T11:22:59.225 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.121931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153216.log 2023-10-16T11:22:59.226 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138605.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.121931.log.gz 2023-10-16T11:22:59.226 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153216.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.138605.log.gz 2023-10-16T11:22:59.226 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153216.log.gz 2023-10-16T11:22:59.226 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.151879.log 2023-10-16T11:22:59.227 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192438.log 2023-10-16T11:22:59.227 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.151879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184199.log 2023-10-16T11:22:59.227 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.151879.log.gz 2023-10-16T11:22:59.227 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141874.log 2023-10-16T11:22:59.228 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.192438.log.gz 2023-10-16T11:22:59.228 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117814.log 2023-10-16T11:22:59.228 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.184199.log.gz 2023-10-16T11:22:59.228 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141874.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.141874.log.gz 2023-10-16T11:22:59.229 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117814.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117814.log.gz 2023-10-16T11:22:59.229 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138375.log 2023-10-16T11:22:59.229 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120422.log 2023-10-16T11:22:59.229 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138375.log: /var/log/ceph/ceph-client.admin.120422.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.138375.log.gz 2023-10-16T11:22:59.230 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.120422.log.gz 2023-10-16T11:22:59.230 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175844.log 2023-10-16T11:22:59.230 INFO:teuthology.orchestra.run.smithi158.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-16T11:22:59.230 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147577.log 2023-10-16T11:22:59.231 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114465.log 2023-10-16T11:22:59.231 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.147577.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.175844.log.gz 2023-10-16T11:22:59.231 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147577.log.gz 2023-10-16T11:22:59.231 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118565.log 2023-10-16T11:22:59.232 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142196.log 2023-10-16T11:22:59.232 INFO:teuthology.orchestra.run.smithi158.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.114465.log.gz 2023-10-16T11:22:59.232 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164307.log 2023-10-16T11:22:59.232 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129933.log 2023-10-16T11:22:59.233 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142196.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.118565.log.gz 2023-10-16T11:22:59.233 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.164307.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142196.log.gz 2023-10-16T11:22:59.233 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111779.log 2023-10-16T11:22:59.233 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117001.log 2023-10-16T11:22:59.234 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129933.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.164307.log.gz 2023-10-16T11:22:59.234 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111779.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.129933.log.gz 2023-10-16T11:22:59.234 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117001.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111779.log.gz 2023-10-16T11:22:59.234 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117001.log.gz 2023-10-16T11:22:59.235 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154265.log 2023-10-16T11:22:59.235 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178690.log 2023-10-16T11:22:59.235 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143011.log 2023-10-16T11:22:59.235 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.154265.log.gz 2023-10-16T11:22:59.236 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178690.log: /var/log/ceph/ceph-client.admin.143011.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.178690.log.gz 2023-10-16T11:22:59.236 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143011.log.gz 2023-10-16T11:22:59.236 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138895.log 2023-10-16T11:22:59.236 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175796.log 2023-10-16T11:22:59.237 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146321.log 2023-10-16T11:22:59.237 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.138895.log.gz 2023-10-16T11:22:59.237 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180500.log 2023-10-16T11:22:59.238 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146321.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.175796.log.gz 2023-10-16T11:22:59.238 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180500.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146321.log.gz 2023-10-16T11:22:59.238 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.180500.log.gz 2023-10-16T11:22:59.238 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162437.log 2023-10-16T11:22:59.239 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118419.log 2023-10-16T11:22:59.239 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156453.log 2023-10-16T11:22:59.239 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112466.log 2023-10-16T11:22:59.239 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.162437.log.gz 2023-10-16T11:22:59.240 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.156453.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.118419.log.gz 2023-10-16T11:22:59.240 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156453.log.gz 2023-10-16T11:22:59.240 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193219.log 2023-10-16T11:22:59.240 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135994.log 2023-10-16T11:22:59.241 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120330.log 2023-10-16T11:22:59.241 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112466.log.gz 2023-10-16T11:22:59.241 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.193219.log.gz 2023-10-16T11:22:59.241 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177491.log 2023-10-16T11:22:59.242 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6%/var/log/ceph/ceph-client.admin.120330.log: -- replaced with /var/log/ceph/ceph-client.admin.135994.log.gz 2023-10-16T11:22:59.242 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120330.log.gz 2023-10-16T11:22:59.242 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177491.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.177491.log.gz 2023-10-16T11:22:59.243 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152941.log 2023-10-16T11:22:59.243 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185762.log 2023-10-16T11:22:59.243 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129745.log 2023-10-16T11:22:59.243 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185762.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152941.log.gz 2023-10-16T11:22:59.243 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.185762.log.gz 2023-10-16T11:22:59.244 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129745.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.129745.log.gz 2023-10-16T11:22:59.244 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193694.log 2023-10-16T11:22:59.244 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181479.log 2023-10-16T11:22:59.244 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140933.log 2023-10-16T11:22:59.245 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181479.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.193694.log.gz 2023-10-16T11:22:59.245 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140933.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.181479.log.gz 2023-10-16T11:22:59.245 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.140933.log.gz 2023-10-16T11:22:59.245 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186177.log 2023-10-16T11:22:59.246 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193173.log 2023-10-16T11:22:59.246 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145670.log 2023-10-16T11:22:59.246 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.186177.log.gz 2023-10-16T11:22:59.247 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120052.log 2023-10-16T11:22:59.247 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146888.log 2023-10-16T11:22:59.247 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145670.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.193173.log.gz 2023-10-16T11:22:59.247 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138980.log 2023-10-16T11:22:59.248 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145670.log.gz 2023-10-16T11:22:59.248 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146888.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120052.log.gz 2023-10-16T11:22:59.248 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113453.log 2023-10-16T11:22:59.248 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138980.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.146888.log.gz 2023-10-16T11:22:59.249 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113453.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.138980.log.gz 2023-10-16T11:22:59.249 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113453.log.gz 2023-10-16T11:22:59.249 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152460.log 2023-10-16T11:22:59.249 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176576.log 2023-10-16T11:22:59.250 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154311.log 2023-10-16T11:22:59.250 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176576.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152460.log.gz 2023-10-16T11:22:59.250 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166068.log 2023-10-16T11:22:59.250 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.176576.log.gz 2023-10-16T11:22:59.251 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191347.log 2023-10-16T11:22:59.251 INFO:teuthology.orchestra.run.smithi158.stderr:gzip 88.2%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.154311.log.gz-- 2023-10-16T11:22:59.251 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122065.log 2023-10-16T11:22:59.251 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166068.log: /var/log/ceph/ceph-client.admin.191347.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.166068.log.gz 2023-10-16T11:22:59.252 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.191347.log.gz 2023-10-16T11:22:59.252 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127874.log 2023-10-16T11:22:59.252 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164379.log 2023-10-16T11:22:59.252 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122065.log.gz 2023-10-16T11:22:59.253 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.127874.log: /var/log/ceph/ceph-client.admin.164379.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127874.log.gz 2023-10-16T11:22:59.253 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.164379.log.gz 2023-10-16T11:22:59.253 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177345.log 2023-10-16T11:22:59.253 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170776.log 2023-10-16T11:22:59.254 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119138.log 2023-10-16T11:22:59.254 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.177345.log.gz 2023-10-16T11:22:59.254 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185267.log 2023-10-16T11:22:59.254 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175478.log 2023-10-16T11:22:59.255 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119138.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.170776.log.gz 2023-10-16T11:22:59.255 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163043.log 2023-10-16T11:22:59.255 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119138.log.gz 2023-10-16T11:22:59.255 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175478.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.185267.log.gz 2023-10-16T11:22:59.256 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161553.log 2023-10-16T11:22:59.256 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163043.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.175478.log.gz 2023-10-16T11:22:59.256 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171187.log 2023-10-16T11:22:59.257 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161553.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.163043.log.gz 2023-10-16T11:22:59.257 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161553.log.gz 2023-10-16T11:22:59.257 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171187.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.171187.log.gz 2023-10-16T11:22:59.257 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189914.log 2023-10-16T11:22:59.258 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185055.log 2023-10-16T11:22:59.258 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138697.log 2023-10-16T11:22:59.258 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.189914.log.gz 2023-10-16T11:22:59.259 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.185055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193560.log 2023-10-16T11:22:59.259 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116126.log 2023-10-16T11:22:59.259 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138697.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.185055.log.gz 2023-10-16T11:22:59.259 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193560.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.138697.log.gz 2023-10-16T11:22:59.260 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.193560.log.gz 2023-10-16T11:22:59.260 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193127.log 2023-10-16T11:22:59.260 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149638.log 2023-10-16T11:22:59.260 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116126.log.gz 2023-10-16T11:22:59.261 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123898.log 2023-10-16T11:22:59.261 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149638.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.193127.log.gz 2023-10-16T11:22:59.261 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.123898.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149638.log.gz 2023-10-16T11:22:59.261 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.123898.log.gz 2023-10-16T11:22:59.262 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128493.log 2023-10-16T11:22:59.262 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.173058.log 2023-10-16T11:22:59.262 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148780.log 2023-10-16T11:22:59.262 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128493.log.gz 2023-10-16T11:22:59.263 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.173058.log: gzip -5 0.0%--verbose -- /var/log/ceph/ceph-client.0.147379.log -- replaced with /var/log/ceph/ceph-client.0.173058.log.gz 2023-10-16T11:22:59.263 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-16T11:22:59.263 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.148780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143057.log 2023-10-16T11:22:59.264 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.147379.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148780.log.gz 2023-10-16T11:22:59.264 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.147379.log.gz 2023-10-16T11:22:59.264 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143057.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143057.log.gz 2023-10-16T11:22:59.264 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140883.log 2023-10-16T11:22:59.265 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181857.log 2023-10-16T11:22:59.265 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150426.log 2023-10-16T11:22:59.265 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181857.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.140883.log.gz 2023-10-16T11:22:59.265 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152605.log 2023-10-16T11:22:59.266 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.181857.log.gz 2023-10-16T11:22:59.266 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118147.log 2023-10-16T11:22:59.266 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152605.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150426.log.gz 2023-10-16T11:22:59.267 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.152605.log.gz 2023-10-16T11:22:59.267 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130654.log 2023-10-16T11:22:59.267 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118147.log.gz 2023-10-16T11:22:59.267 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176438.log 2023-10-16T11:22:59.268 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.130654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165533.log 0.0% 2023-10-16T11:22:59.268 INFO:teuthology.orchestra.run.smithi158.stderr: -- replaced with /var/log/ceph/ceph-client.0.130654.log.gz 2023-10-16T11:22:59.268 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184153.log 2023-10-16T11:22:59.268 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176438.log: /var/log/ceph/ceph-client.admin.165533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119184.log 2023-10-16T11:22:59.269 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.176438.log.gz 2023-10-16T11:22:59.269 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184153.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.165533.log.gz 2023-10-16T11:22:59.269 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.184153.log.gz 2023-10-16T11:22:59.269 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113846.log 2023-10-16T11:22:59.270 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192668.log 2023-10-16T11:22:59.270 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111484.log 2023-10-16T11:22:59.270 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119184.log.gz 2023-10-16T11:22:59.270 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192668.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113846.log.gz 2023-10-16T11:22:59.271 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.192668.log.gz 2023-10-16T11:22:59.271 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195561.log 2023-10-16T11:22:59.271 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179859.log 2023-10-16T11:22:59.272 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111484.log.gz 2023-10-16T11:22:59.272 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116863.log 2023-10-16T11:22:59.272 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179859.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.195561.log.gz 2023-10-16T11:22:59.272 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116863.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.179859.log.gz 2023-10-16T11:22:59.273 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116863.log.gz 2023-10-16T11:22:59.273 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113451.log 2023-10-16T11:22:59.273 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-16T11:22:59.273 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155551.log 2023-10-16T11:22:59.274 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.113451.log.gz 2023-10-16T11:22:59.274 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138226.log 2023-10-16T11:22:59.274 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183371.log 2023-10-16T11:22:59.274 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155551.log: /var/log/ceph/ceph-client.admin.138226.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155551.log.gz 2023-10-16T11:22:59.275 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183371.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138226.log.gz 2023-10-16T11:22:59.275 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.183371.log.gz 2023-10-16T11:22:59.275 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113449.log 2023-10-16T11:22:59.275 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151642.log 2023-10-16T11:22:59.276 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181949.log 2023-10-16T11:22:59.276 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.113449.log.gz 2023-10-16T11:22:59.276 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151642.log: /var/log/ceph/ceph-client.admin.181949.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.151642.log.gz 2023-10-16T11:22:59.276 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.181949.log.gz 2023-10-16T11:22:59.277 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119820.log 2023-10-16T11:22:59.277 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138651.log 2023-10-16T11:22:59.277 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123798.log 2023-10-16T11:22:59.277 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119820.log.gz 2023-10-16T11:22:59.278 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138651.log: /var/log/ceph/ceph-client.admin.123798.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138651.log.gz 2023-10-16T11:22:59.278 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123798.log.gz 2023-10-16T11:22:59.278 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177156.log 2023-10-16T11:22:59.278 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165270.log 2023-10-16T11:22:59.279 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130025.log 2023-10-16T11:22:59.279 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.177156.log.gz 2023-10-16T11:22:59.279 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165270.log: /var/log/ceph/ceph-client.admin.130025.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.165270.log.gz 2023-10-16T11:22:59.279 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130025.log.gz 2023-10-16T11:22:59.280 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144904.log 2023-10-16T11:22:59.280 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120376.log 2023-10-16T11:22:59.280 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113071.log 2023-10-16T11:22:59.280 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144904.log.gz 2023-10-16T11:22:59.281 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120376.log: /var/log/ceph/ceph-client.admin.113071.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120376.log.gz 2023-10-16T11:22:59.281 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113071.log.gz 2023-10-16T11:22:59.281 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113087.log 2023-10-16T11:22:59.281 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163436.log 2023-10-16T11:22:59.282 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120698.log 2023-10-16T11:22:59.282 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4%/var/log/ceph/ceph-client.admin.163436.log: -- replaced with /var/log/ceph/ceph-client.admin.113087.log.gz 2023-10-16T11:22:59.282 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151367.log 2023-10-16T11:22:59.282 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.163436.log.gz 2023-10-16T11:22:59.283 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120698.log.gz 2023-10-16T11:22:59.283 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116172.log 2023-10-16T11:22:59.283 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112144.log 2023-10-16T11:22:59.284 INFO:teuthology.orchestra.run.smithi158.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.151367.log.gz 2023-10-16T11:22:59.284 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116172.log: /var/log/ceph/ceph-client.admin.112144.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116172.log.gz 2023-10-16T11:22:59.284 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112144.log.gz 2023-10-16T11:22:59.284 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130806.log 2023-10-16T11:22:59.285 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172128.log 2023-10-16T11:22:59.285 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112420.log 2023-10-16T11:22:59.285 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172128.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.130806.log.gz 2023-10-16T11:22:59.285 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112420.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.172128.log.gz 2023-10-16T11:22:59.286 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112420.log.gz 2023-10-16T11:22:59.286 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116310.log 2023-10-16T11:22:59.286 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191460.log 2023-10-16T11:22:59.286 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114122.log 2023-10-16T11:22:59.287 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116310.log.gz 2023-10-16T11:22:59.287 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191460.log: /var/log/ceph/ceph-client.admin.114122.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.191460.log.gz 2023-10-16T11:22:59.287 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 88.3%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.114122.log.gz/var/log/ceph/ceph-client.admin.183037.log 2023-10-16T11:22:59.287 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-16T11:22:59.288 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124579.log 2023-10-16T11:22:59.288 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.115766.log 2023-10-16T11:22:59.288 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.183037.log.gz 2023-10-16T11:22:59.288 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117722.log 2023-10-16T11:22:59.289 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.2.115766.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124579.log.gz 2023-10-16T11:22:59.289 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.115766.log.gz 2023-10-16T11:22:59.289 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117722.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.117722.log.gz 2023-10-16T11:22:59.289 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134916.log 2023-10-16T11:22:59.290 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163390.log 2023-10-16T11:22:59.290 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129255.log 2023-10-16T11:22:59.290 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163390.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134916.log.gz 2023-10-16T11:22:59.290 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135810.log 2023-10-16T11:22:59.291 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.163390.log.gz 2023-10-16T11:22:59.291 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129255.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.129255.log.gz 2023-10-16T11:22:59.291 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112328.log 2023-10-16T11:22:59.292 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191209.log 2023-10-16T11:22:59.292 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135810.log.gz 2023-10-16T11:22:59.292 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112328.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112328.log.gz 2023-10-16T11:22:59.292 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161703.log 2023-10-16T11:22:59.292 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112282.log 2023-10-16T11:22:59.293 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.191209.log.gz 2023-10-16T11:22:59.293 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161703.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.161703.log.gz 2023-10-16T11:22:59.293 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112282.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112282.log.gz 2023-10-16T11:22:59.293 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120790.log 2023-10-16T11:22:59.294 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190608.log 2023-10-16T11:22:59.294 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120790.log: /var/log/ceph/ceph-client.admin.190608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165874.log 2023-10-16T11:22:59.294 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.190608.log.gz 2023-10-16T11:22:59.294 INFO:teuthology.orchestra.run.smithi158.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.120790.log.gz 2023-10-16T11:22:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135008.log 2023-10-16T11:22:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192254.log 2023-10-16T11:22:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135008.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.165874.log.gz 2023-10-16T11:22:59.295 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135008.log.gz 2023-10-16T11:22:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129699.log 2023-10-16T11:22:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195377.log 2023-10-16T11:22:59.296 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192254.log: /var/log/ceph/ceph-client.admin.129699.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.192254.log.gz 2023-10-16T11:22:59.296 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129699.log.gz 2023-10-16T11:22:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135476.log 2023-10-16T11:22:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190341.log 2023-10-16T11:22:59.297 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135476.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.195377.log.gz 2023-10-16T11:22:59.298 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190341.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135476.log.gz 2023-10-16T11:22:59.298 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.190341.log.gz 2023-10-16T11:22:59.298 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.97576.log 2023-10-16T11:22:59.298 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166835.log 2023-10-16T11:22:59.299 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph.tmp-client.admin.97576.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph.tmp-client.admin.97576.log.gz--verbose 2023-10-16T11:22:59.299 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.112190.log 2023-10-16T11:22:59.299 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166835.log: /var/log/ceph/ceph-client.admin.112190.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.166835.log.gz 2023-10-16T11:22:59.299 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112190.log.gz 2023-10-16T11:22:59.300 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170398.log 2023-10-16T11:22:59.300 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191927.log 2023-10-16T11:22:59.300 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188531.log 2023-10-16T11:22:59.300 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.170398.log.gz 2023-10-16T11:22:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175526.log 2023-10-16T11:22:59.301 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.191927.log.gz 2023-10-16T11:22:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130650.log 2023-10-16T11:22:59.301 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175526.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.188531.log.gz 2023-10-16T11:22:59.302 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.175526.log.gz 2023-10-16T11:22:59.302 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.130650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.130650.log.gz 2023-10-16T11:22:59.302 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143963.log 2023-10-16T11:22:59.302 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145992.log 2023-10-16T11:22:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143963.log: /var/log/ceph/ceph-client.admin.145992.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.143963.log.gz 2023-10-16T11:22:59.303 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.145992.log.gz 2023-10-16T11:22:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195837.log 2023-10-16T11:22:59.303 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161655.log 2023-10-16T11:22:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127973.log 2023-10-16T11:22:59.304 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.195837.log.gz 2023-10-16T11:22:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161655.log: /var/log/ceph/ceph-client.admin.127973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.161655.log.gz 2023-10-16T11:22:59.304 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137365.log 2023-10-16T11:22:59.305 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.127973.log.gz 2023-10-16T11:22:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112706.log 2023-10-16T11:22:59.305 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171291.log 2023-10-16T11:22:59.305 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137365.log.gz 2023-10-16T11:22:59.306 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112706.log: /var/log/ceph/ceph-client.admin.171291.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112706.log.gz 2023-10-16T11:22:59.306 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.171291.log.gz 2023-10-16T11:22:59.306 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166319.log 2023-10-16T11:22:59.306 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119368.log 2023-10-16T11:22:59.307 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196067.log 2023-10-16T11:22:59.307 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119368.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.166319.log.gz 2023-10-16T11:22:59.307 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154506.log 2023-10-16T11:22:59.307 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119368.log.gz 2023-10-16T11:22:59.308 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.196067.log: /var/log/ceph/ceph-client.admin.154506.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.196067.log.gz 2023-10-16T11:22:59.308 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.154506.log.gz 2023-10-16T11:22:59.308 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142104.log 2023-10-16T11:22:59.309 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186842.log 2023-10-16T11:22:59.309 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180772.log 2023-10-16T11:22:59.309 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.142104.log.gz 2023-10-16T11:22:59.309 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126158.log 2023-10-16T11:22:59.309 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.186842.log.gz 2023-10-16T11:22:59.310 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188098.log 2023-10-16T11:22:59.310 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.126158.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.180772.log.gz 2023-10-16T11:22:59.310 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126158.log.gz 2023-10-16T11:22:59.311 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188098.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.188098.log.gz 2023-10-16T11:22:59.311 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187348.log 2023-10-16T11:22:59.311 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184617.log 2023-10-16T11:22:59.311 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187348.log: /var/log/ceph/ceph-client.admin.184617.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.187348.log.gz 2023-10-16T11:22:59.312 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.184617.log.gz 2023-10-16T11:22:59.312 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187256.log 2023-10-16T11:22:59.312 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118657.log 2023-10-16T11:22:59.312 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147485.log 2023-10-16T11:22:59.313 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118657.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.187256.log.gz 2023-10-16T11:22:59.313 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.147485.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.118657.log.gz 2023-10-16T11:22:59.313 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142873.log 2023-10-16T11:22:59.313 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147485.log.gz 2023-10-16T11:22:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163482.log 2023-10-16T11:22:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129841.log 2023-10-16T11:22:59.314 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142873.log.gz 2023-10-16T11:22:59.314 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163482.log: /var/log/ceph/ceph-client.admin.129841.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.163482.log.gz 2023-10-16T11:22:59.315 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129841.log.gz 2023-10-16T11:22:59.315 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167407.log 2023-10-16T11:22:59.315 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192714.log 2023-10-16T11:22:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188437.log 2023-10-16T11:22:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.192714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183187.log 2023-10-16T11:22:59.316 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188437.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.167407.log.gz 2023-10-16T11:22:59.316 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.188437.log.gz 2023-10-16T11:22:59.317 INFO:teuthology.orchestra.run.smithi158.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.192714.log.gz 2023-10-16T11:22:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183187.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.183187.log.gz 2023-10-16T11:22:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151526.log 2023-10-16T11:22:59.317 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179104.log 2023-10-16T11:22:59.318 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117404.log 2023-10-16T11:22:59.318 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151526.log.gz 2023-10-16T11:22:59.318 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179104.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.179104.log.gz 2023-10-16T11:22:59.319 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117404.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.117404.log.gz 2023-10-16T11:22:59.319 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165005.log 2023-10-16T11:22:59.319 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167820.log 2023-10-16T11:22:59.319 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165005.log: /var/log/ceph/ceph-client.admin.167820.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.165005.log.gz 2023-10-16T11:22:59.319 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.167820.log.gz 2023-10-16T11:22:59.320 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162391.log 2023-10-16T11:22:59.320 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195423.log 2023-10-16T11:22:59.320 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150803.log 2023-10-16T11:22:59.320 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.162391.log.gz 2023-10-16T11:22:59.321 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195423.log: /var/log/ceph/ceph-client.admin.150803.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.195423.log.gz 2023-10-16T11:22:59.321 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150803.log.gz 2023-10-16T11:22:59.321 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125849.log 2023-10-16T11:22:59.321 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121786.log 2023-10-16T11:22:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.125849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180056.log 2023-10-16T11:22:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.121786.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125849.log.gz 2023-10-16T11:22:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180056.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121786.log.gz 2023-10-16T11:22:59.323 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.180056.log.gz 2023-10-16T11:22:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180923.log 2023-10-16T11:22:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184107.log 2023-10-16T11:22:59.323 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172546.log 2023-10-16T11:22:59.324 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.180923.log.gz 2023-10-16T11:22:59.324 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184107.log: /var/log/ceph/ceph-client.admin.172546.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.184107.log.gz 2023-10-16T11:22:59.324 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.172546.log.gz 2023-10-16T11:22:59.324 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140981.log 2023-10-16T11:22:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114649.log 2023-10-16T11:22:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119458.log 2023-10-16T11:22:59.325 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.140981.log.gz 2023-10-16T11:22:59.325 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114649.log: /var/log/ceph/ceph-client.admin.119458.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114649.log.gz 2023-10-16T11:22:59.326 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119458.log.gz 2023-10-16T11:22:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186555.log 2023-10-16T11:22:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167912.log 2023-10-16T11:22:59.326 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146508.log 2023-10-16T11:22:59.327 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.186555.log.gz 2023-10-16T11:22:59.327 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167912.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.167912.log.gz 2023-10-16T11:22:59.327 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146508.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146508.log.gz 2023-10-16T11:22:59.327 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141148.log 2023-10-16T11:22:59.328 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184571.log 2023-10-16T11:22:59.328 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141148.log: /var/log/ceph/ceph-client.admin.184571.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.141148.log.gz 2023-10-16T11:22:59.328 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.184571.log.gz 2023-10-16T11:22:59.328 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135146.log 2023-10-16T11:22:59.329 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166252.log 2023-10-16T11:22:59.329 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183233.log 2023-10-16T11:22:59.329 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135146.log.gz 2023-10-16T11:22:59.329 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166252.log: /var/log/ceph/ceph-client.admin.183233.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.166252.log.gz 2023-10-16T11:22:59.330 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.183233.log.gz 2023-10-16T11:22:59.330 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184893.log 2023-10-16T11:22:59.330 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137780.log 2023-10-16T11:22:59.331 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190389.log 2023-10-16T11:22:59.331 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137780.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.184893.log.gz 2023-10-16T11:22:59.331 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121998.log 2023-10-16T11:22:59.331 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137780.log.gz 2023-10-16T11:22:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190389.log: /var/log/ceph/ceph-client.admin.121998.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.190389.log.gz 2023-10-16T11:22:59.332 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121998.log.gz 2023-10-16T11:22:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126464.log 2023-10-16T11:22:59.332 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169613.log 2023-10-16T11:22:59.333 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.126464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182087.log 2023-10-16T11:22:59.333 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126464.log.gz 2023-10-16T11:22:59.333 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.169613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186129.log 2023-10-16T11:22:59.333 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.169613.log.gz 2023-10-16T11:22:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182087.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.182087.log.gz 2023-10-16T11:22:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104252.log 2023-10-16T11:22:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164175.log 2023-10-16T11:22:59.334 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.186129.log.gz 2023-10-16T11:22:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.104252.log: /var/log/ceph/ceph-client.admin.164175.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.104252.log.gz 2023-10-16T11:22:59.335 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.164175.log.gz 2023-10-16T11:22:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167307.log 2023-10-16T11:22:59.335 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142012.log 2023-10-16T11:22:59.336 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109500.log 2023-10-16T11:22:59.336 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.167307.log.gz 2023-10-16T11:22:59.336 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179150.log 2023-10-16T11:22:59.336 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.142012.log.gz 2023-10-16T11:22:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.109500.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.109500.log.gz 2023-10-16T11:22:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155225.log 2023-10-16T11:22:59.337 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150849.log 2023-10-16T11:22:59.337 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.179150.log.gz 2023-10-16T11:22:59.338 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155225.log: /var/log/ceph/ceph-client.admin.150849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.115762.log 2023-10-16T11:22:59.338 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.155225.log.gz 2023-10-16T11:22:59.338 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150849.log.gz 2023-10-16T11:22:59.338 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149072.log 2023-10-16T11:22:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.2.115762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121719.log 2023-10-16T11:22:59.339 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.115762.log.gz 2023-10-16T11:22:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149072.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.149072.log.gz 2023-10-16T11:22:59.339 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176889.log 2023-10-16T11:22:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.121719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166160.log 2023-10-16T11:22:59.340 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8%/var/log/ceph/ceph-client.admin.176889.log: -- replaced with /var/log/ceph/ceph-client.admin.121719.log.gz 2023-10-16T11:22:59.340 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.176889.log.gz 2023-10-16T11:22:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166160.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.166160.log.gz 2023-10-16T11:22:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153967.log 2023-10-16T11:22:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153500.log 2023-10-16T11:22:59.341 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153967.log: /var/log/ceph/ceph-client.admin.153500.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153967.log.gz 2023-10-16T11:22:59.341 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153500.log.gz 2023-10-16T11:22:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159813.log 2023-10-16T11:22:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193344.log 2023-10-16T11:22:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.159813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120192.log 2023-10-16T11:22:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.193344.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.159813.log.gz 2023-10-16T11:22:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120192.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.193344.log.gz 2023-10-16T11:22:59.343 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120192.log.gz 2023-10-16T11:22:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161806.log 2023-10-16T11:22:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191842.log 2023-10-16T11:22:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146556.log 2023-10-16T11:22:59.344 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.161806.log.gz 2023-10-16T11:22:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145550.log 2023-10-16T11:22:59.344 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146556.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.191842.log.gz 2023-10-16T11:22:59.345 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146556.log.gz 2023-10-16T11:22:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145550.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.145550.log.gz 2023-10-16T11:22:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.189306.log 2023-10-16T11:22:59.345 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130301.log 2023-10-16T11:22:59.346 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.189306.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.0.189306.log.gz--verbose 2023-10-16T11:22:59.346 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.183509.log 2023-10-16T11:22:59.346 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130301.log: /var/log/ceph/ceph-client.admin.183509.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130301.log.gz 2023-10-16T11:22:59.346 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.183509.log.gz 2023-10-16T11:22:59.347 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151183.log 2023-10-16T11:22:59.347 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150895.log 2023-10-16T11:22:59.347 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150671.log 2023-10-16T11:22:59.347 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150895.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.151183.log.gz 2023-10-16T11:22:59.348 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150671.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.150895.log.gz 2023-10-16T11:22:59.348 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.150671.log.gz 2023-10-16T11:22:59.348 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152672.log 2023-10-16T11:22:59.348 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135344.log 2023-10-16T11:22:59.349 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121481.log 2023-10-16T11:22:59.349 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.152672.log.gz 2023-10-16T11:22:59.349 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135344.log: /var/log/ceph/ceph-client.admin.121481.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135344.log.gz 2023-10-16T11:22:59.349 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.121481.log.gz--verbose 2023-10-16T11:22:59.350 INFO:teuthology.orchestra.run.smithi158.stderr:-- /var/log/ceph/ceph-client.admin.180450.log 2023-10-16T11:22:59.350 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170876.log 2023-10-16T11:22:59.350 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188250.log 2023-10-16T11:22:59.350 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.180450.log.gz 2023-10-16T11:22:59.351 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145156.log 2023-10-16T11:22:59.351 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.170876.log.gz 2023-10-16T11:22:59.351 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110335.log 2023-10-16T11:22:59.351 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.188250.log.gz 2023-10-16T11:22:59.352 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153350.log 2023-10-16T11:22:59.352 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5%/var/log/ceph/ceph-client.admin.110335.log: -- replaced with /var/log/ceph/ceph-client.admin.145156.log.gz 2023-10-16T11:22:59.352 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153350.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110335.log.gz 2023-10-16T11:22:59.353 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.153350.log.gz 2023-10-16T11:22:59.353 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150196.log 2023-10-16T11:22:59.353 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111668.log 2023-10-16T11:22:59.353 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129117.log 2023-10-16T11:22:59.354 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.150196.log.gz 2023-10-16T11:22:59.354 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.111668.log: /var/log/ceph/ceph-client.admin.129117.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111668.log.gz 2023-10-16T11:22:59.354 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.129117.log.gz 2023-10-16T11:22:59.354 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186279.log 2023-10-16T11:22:59.355 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176197.log 2023-10-16T11:22:59.355 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134431.log 2023-10-16T11:22:59.355 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.186279.log.gz 2023-10-16T11:22:59.355 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136153.log 2023-10-16T11:22:59.356 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134431.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.176197.log.gz 2023-10-16T11:22:59.356 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.134431.log.gz 2023-10-16T11:22:59.356 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136153.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136153.log.gz 2023-10-16T11:22:59.356 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173211.log 2023-10-16T11:22:59.357 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163185.log 2023-10-16T11:22:59.357 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.173211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145055.log 2023-10-16T11:22:59.357 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163185.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.173211.log.gz 2023-10-16T11:22:59.357 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171475.log 2023-10-16T11:22:59.358 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.163185.log.gz 2023-10-16T11:22:59.358 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.145055.log.gz 2023-10-16T11:22:59.358 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116356.log 2023-10-16T11:22:59.359 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170617.log 2023-10-16T11:22:59.359 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.171475.log.gz 2023-10-16T11:22:59.359 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116356.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116356.log.gz 2023-10-16T11:22:59.359 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117768.log 2023-10-16T11:22:59.360 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180350.log 2023-10-16T11:22:59.360 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2%/var/log/ceph/ceph-client.admin.117768.log: -- replaced with /var/log/ceph/ceph-client.admin.170617.log.gz 2023-10-16T11:22:59.360 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117768.log.gz 2023-10-16T11:22:59.360 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180350.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.180350.log.gz 2023-10-16T11:22:59.361 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166694.log 2023-10-16T11:22:59.361 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.151875.log 2023-10-16T11:22:59.361 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166694.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.166694.log.gz 2023-10-16T11:22:59.361 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.151875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.151875.log.gz 2023-10-16T11:22:59.362 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183279.log 2023-10-16T11:22:59.362 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123706.log 2023-10-16T11:22:59.362 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183279.log: /var/log/ceph/ceph-client.admin.123706.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.183279.log.gz 2023-10-16T11:22:59.362 INFO:teuthology.orchestra.run.smithi158.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.123706.log.gz 2023-10-16T11:22:59.363 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180215.log 2023-10-16T11:22:59.363 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148886.log 2023-10-16T11:22:59.363 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113068.log 2023-10-16T11:22:59.363 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.180215.log.gz 2023-10-16T11:22:59.364 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.148886.log: /var/log/ceph/ceph-client.admin.113068.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.148886.log.gz 2023-10-16T11:22:59.364 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194261.log 2023-10-16T11:22:59.364 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113068.log.gz 2023-10-16T11:22:59.364 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149121.log 2023-10-16T11:22:59.365 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187118.log 2023-10-16T11:22:59.365 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.194261.log.gz 2023-10-16T11:22:59.365 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149121.log: /var/log/ceph/ceph-client.admin.187118.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149121.log.gz 2023-10-16T11:22:59.365 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.187118.log.gz 2023-10-16T11:22:59.366 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138513.log 2023-10-16T11:22:59.366 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181023.log 2023-10-16T11:22:59.366 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.138513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147948.log 2023-10-16T11:22:59.367 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.138513.log.gz 2023-10-16T11:22:59.367 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181023.log: /var/log/ceph/ceph-client.admin.147948.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.181023.log.gz 2023-10-16T11:22:59.367 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147948.log.gz 2023-10-16T11:22:59.367 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182430.log 2023-10-16T11:22:59.368 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135254.log 2023-10-16T11:22:59.368 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175379.log 2023-10-16T11:22:59.368 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.182430.log.gz 2023-10-16T11:22:59.368 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135254.log: /var/log/ceph/ceph-client.admin.175379.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135254.log.gz 2023-10-16T11:22:59.369 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134079.log 2023-10-16T11:22:59.369 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.175379.log.gz 2023-10-16T11:22:59.369 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145500.log 2023-10-16T11:22:59.369 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152151.log 2023-10-16T11:22:59.370 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.134079.log.gz 2023-10-16T11:22:59.370 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145500.log: /var/log/ceph/ceph-client.admin.152151.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145500.log.gz 2023-10-16T11:22:59.370 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.152151.log.gz 2023-10-16T11:22:59.370 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195975.log 2023-10-16T11:22:59.371 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184383.log 2023-10-16T11:22:59.371 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.195975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135856.log 2023-10-16T11:22:59.371 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.195975.log.gz 2023-10-16T11:22:59.372 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.184383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195285.log 2023-10-16T11:22:59.372 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.184383.log.gz 2023-10-16T11:22:59.372 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135856.log: /var/log/ceph/ceph-client.admin.195285.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.135856.log.gz 2023-10-16T11:22:59.372 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.195285.log.gz 2023-10-16T11:22:59.373 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125543.log 2023-10-16T11:22:59.373 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194779.log 2023-10-16T11:22:59.373 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.125543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188485.log 2023-10-16T11:22:59.373 INFO:teuthology.orchestra.run.smithi158.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.125543.log.gz 2023-10-16T11:22:59.374 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194779.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.194779.log.gz 2023-10-16T11:22:59.374 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154414.log 2023-10-16T11:22:59.374 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140215.log 2023-10-16T11:22:59.374 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154414.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.188485.log.gz 2023-10-16T11:22:59.375 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.154414.log.gz 2023-10-16T11:22:59.375 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162729.log 2023-10-16T11:22:59.375 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172270.log 2023-10-16T11:22:59.375 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.140215.log.gz 2023-10-16T11:22:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162729.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.162729.log.gz 2023-10-16T11:22:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172270.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.172270.log.gz 2023-10-16T11:22:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151459.log 2023-10-16T11:22:59.376 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193844.log 2023-10-16T11:22:59.377 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151459.log: /var/log/ceph/ceph-client.admin.193844.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151459.log.gz 2023-10-16T11:22:59.377 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116817.log 2023-10-16T11:22:59.377 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.193844.log.gz 2023-10-16T11:22:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172224.log 2023-10-16T11:22:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144313.log 2023-10-16T11:22:59.378 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172224.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116817.log.gz 2023-10-16T11:22:59.378 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.172224.log.gz 2023-10-16T11:22:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144313.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144313.log.gz 2023-10-16T11:22:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175428.log 2023-10-16T11:22:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191973.log 2023-10-16T11:22:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175428.log: /var/log/ceph/ceph-client.admin.191973.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.175428.log.gz 2023-10-16T11:22:59.380 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.191973.log.gz 2023-10-16T11:22:59.380 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128188.log 2023-10-16T11:22:59.380 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117047.log 2023-10-16T11:22:59.380 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187578.log 2023-10-16T11:22:59.381 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.128188.log.gz 2023-10-16T11:22:59.381 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117047.log: /var/log/ceph/ceph-client.admin.187578.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117047.log.gz 2023-10-16T11:22:59.381 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167261.log 2023-10-16T11:22:59.381 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.187578.log.gz 2023-10-16T11:22:59.382 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142263.log 2023-10-16T11:22:59.382 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156742.log 2023-10-16T11:22:59.382 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142263.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.167261.log.gz 2023-10-16T11:22:59.382 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.156742.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142263.log.gz 2023-10-16T11:22:59.383 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.156742.log.gz 2023-10-16T11:22:59.383 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173667.log 2023-10-16T11:22:59.383 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191414.log 2023-10-16T11:22:59.383 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.173667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122548.log 2023-10-16T11:22:59.384 INFO:teuthology.orchestra.run.smithi158.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.173667.log.gz 2023-10-16T11:22:59.384 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191414.log: /var/log/ceph/ceph-client.admin.122548.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.191414.log.gz 2023-10-16T11:22:59.384 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122548.log.gz 2023-10-16T11:22:59.384 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180401.log 2023-10-16T11:22:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146934.log 2023-10-16T11:22:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181719.log 2023-10-16T11:22:59.385 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.180401.log.gz 2023-10-16T11:22:59.385 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146934.log: /var/log/ceph/ceph-client.admin.181719.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146934.log.gz 2023-10-16T11:22:59.386 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165735.log 2023-10-16T11:22:59.386 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.181719.log.gz 2023-10-16T11:22:59.386 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141762.log 2023-10-16T11:22:59.386 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159859.log 2023-10-16T11:22:59.387 INFO:teuthology.orchestra.run.smithi158.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165735.log.gz 2023-10-16T11:22:59.387 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141762.log: /var/log/ceph/ceph-client.admin.159859.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141762.log.gz 2023-10-16T11:22:59.387 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.159859.log.gz 2023-10-16T11:22:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163528.log 2023-10-16T11:22:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186888.log 2023-10-16T11:22:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146367.log 2023-10-16T11:22:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186888.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.163528.log.gz 2023-10-16T11:22:59.388 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.146367.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.186888.log.gz 2023-10-16T11:22:59.389 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.146367.log.gz 2023-10-16T11:22:59.389 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.173054.log 2023-10-16T11:22:59.389 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134870.log 2023-10-16T11:22:59.389 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.173054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.173054.log.gz 2023-10-16T11:22:59.390 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122454.log 2023-10-16T11:22:59.390 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182476.log 2023-10-16T11:22:59.390 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134870.log: /var/log/ceph/ceph-client.admin.122454.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134870.log.gz 2023-10-16T11:22:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182476.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.122454.log.gz 2023-10-16T11:22:59.391 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.182476.log.gz 2023-10-16T11:22:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172500.log 2023-10-16T11:22:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124931.log 2023-10-16T11:22:59.391 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123852.log 2023-10-16T11:22:59.392 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124931.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.172500.log.gz 2023-10-16T11:22:59.392 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184431.log 2023-10-16T11:22:59.392 INFO:teuthology.orchestra.run.smithi158.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.124931.log.gz 2023-10-16T11:22:59.393 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133579.log 2023-10-16T11:22:59.393 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.123852.log: /var/log/ceph/ceph-client.admin.184431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191301.log 2023-10-16T11:22:59.393 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123852.log.gz 2023-10-16T11:22:59.393 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.133579.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.184431.log.gz 2023-10-16T11:22:59.393 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.133579.log.gz 2023-10-16T11:22:59.394 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191301.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191301.log.gz 2023-10-16T11:22:59.394 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190291.log 2023-10-16T11:22:59.394 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154736.log 2023-10-16T11:22:59.395 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149689.log 2023-10-16T11:22:59.395 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154736.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.190291.log.gz 2023-10-16T11:22:59.395 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.154736.log.gz 2023-10-16T11:22:59.395 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149689.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.149689.log.gz 2023-10-16T11:22:59.396 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194362.log 2023-10-16T11:22:59.396 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180103.log 2023-10-16T11:22:59.396 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194362.log: /var/log/ceph/ceph-client.admin.180103.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.194362.log.gz 2023-10-16T11:22:59.396 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.180103.log.gz 2023-10-16T11:22:59.397 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156333.log 2023-10-16T11:22:59.397 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135430.log 2023-10-16T11:22:59.397 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.156333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180657.log 2023-10-16T11:22:59.397 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.156333.log.gz 2023-10-16T11:22:59.398 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175328.log 2023-10-16T11:22:59.398 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180657.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.135430.log.gz 2023-10-16T11:22:59.398 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175945.log 2023-10-16T11:22:59.398 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.180657.log.gz 2023-10-16T11:22:59.399 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175328.log: /var/log/ceph/ceph-client.admin.175945.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.175328.log.gz 2023-10-16T11:22:59.399 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.175945.log.gz 2023-10-16T11:22:59.399 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.167710.log 2023-10-16T11:22:59.400 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119590.log 2023-10-16T11:22:59.400 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.167710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.167710.log.gz 2023-10-16T11:22:59.400 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119590.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119590.log.gz 2023-10-16T11:22:59.400 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150989.log 2023-10-16T11:22:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187624.log 2023-10-16T11:22:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150989.log: /var/log/ceph/ceph-client.admin.187624.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150989.log.gz 2023-10-16T11:22:59.401 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.187624.log.gz 2023-10-16T11:22:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180548.log 2023-10-16T11:22:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192208.log 2023-10-16T11:22:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.180548.log: /var/log/ceph/ceph-client.admin.192208.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.180548.log.gz 2023-10-16T11:22:59.402 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.192208.log.gz 2023-10-16T11:22:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116080.log 2023-10-16T11:22:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.178162.log 2023-10-16T11:22:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118101.log 2023-10-16T11:22:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.0.178162.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116080.log.gz 2023-10-16T11:22:59.403 INFO:teuthology.orchestra.run.smithi158.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.178162.log.gz 2023-10-16T11:22:59.404 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118101.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118101.log.gz 2023-10-16T11:22:59.404 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167077.log 2023-10-16T11:22:59.404 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130209.log 2023-10-16T11:22:59.404 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.167077.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.167077.log.gz 2023-10-16T11:22:59.405 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130209.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.130209.log.gz 2023-10-16T11:22:59.405 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133765.log 2023-10-16T11:22:59.405 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183417.log 2023-10-16T11:22:59.405 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.133765.log: /var/log/ceph/ceph-client.admin.183417.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133765.log.gz 2023-10-16T11:22:59.406 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.183417.log.gz 2023-10-16T11:22:59.406 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121864.log 2023-10-16T11:22:59.440 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190507.log 2023-10-16T11:22:59.440 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.121864.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121864.log.gz 2023-10-16T11:22:59.440 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190507.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.190507.log.gz 2023-10-16T11:22:59.441 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143333.log 2023-10-16T11:22:59.441 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181386.log 2023-10-16T11:22:59.441 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143333.log: /var/log/ceph/ceph-client.admin.181386.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.143333.log.gz 2023-10-16T11:22:59.442 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.181386.log.gz 2023-10-16T11:22:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129025.log 2023-10-16T11:22:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125237.log 2023-10-16T11:22:59.442 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129025.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.129025.log.gz 2023-10-16T11:22:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.125237.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.125237.log.gz 2023-10-16T11:22:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150242.log 2023-10-16T11:22:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128738.log 2023-10-16T11:22:59.443 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150242.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.150242.log.gz 2023-10-16T11:22:59.444 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128738.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.128738.log.gz 2023-10-16T11:22:59.444 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171659.log 2023-10-16T11:22:59.444 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138273.log 2023-10-16T11:22:59.444 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171659.log: /var/log/ceph/ceph-client.admin.138273.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.171659.log.gz 2023-10-16T11:22:59.445 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138273.log.gz 2023-10-16T11:22:59.445 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187440.log 2023-10-16T11:22:59.445 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195929.log 2023-10-16T11:22:59.445 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187440.log: /var/log/ceph/ceph-client.admin.195929.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.187440.log.gz 2023-10-16T11:22:59.446 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.195929.log.gz 2023-10-16T11:22:59.446 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151275.log 2023-10-16T11:22:59.446 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183325.log 2023-10-16T11:22:59.446 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151275.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.151275.log.gz 2023-10-16T11:22:59.447 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.183325.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.183325.log.gz 2023-10-16T11:22:59.447 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187072.log 2023-10-16T11:22:59.447 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195653.log 2023-10-16T11:22:59.448 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187072.log: /var/log/ceph/ceph-client.admin.195653.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.187072.log.gz 2023-10-16T11:22:59.448 INFO:teuthology.orchestra.run.smithi158.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.195653.log.gz 2023-10-16T11:22:59.448 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176484.log 2023-10-16T11:22:59.448 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139872.log 2023-10-16T11:22:59.449 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176484.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.176484.log.gz 2023-10-16T11:22:59.449 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.139872.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139872.log.gz 2023-10-16T11:22:59.449 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124426.log 2023-10-16T11:22:59.449 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130997.log 2023-10-16T11:22:59.450 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.124426.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124426.log.gz 2023-10-16T11:22:59.450 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130997.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.130997.log.gz 2023-10-16T11:22:59.450 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141920.log 2023-10-16T11:22:59.450 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111825.log 2023-10-16T11:22:59.451 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.141920.log: /var/log/ceph/ceph-client.admin.111825.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141920.log.gz 2023-10-16T11:22:59.451 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.111825.log.gz 2023-10-16T11:22:59.451 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134382.log 2023-10-16T11:22:59.451 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153450.log 2023-10-16T11:22:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134382.log: /var/log/ceph/ceph-client.admin.153450.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134382.log.gz 2023-10-16T11:22:59.452 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153450.log.gz 2023-10-16T11:22:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155782.log 2023-10-16T11:22:59.452 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114076.log 2023-10-16T11:22:59.453 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128341.log 2023-10-16T11:22:59.453 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.155782.log.gz 2023-10-16T11:22:59.453 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.114076.log: /var/log/ceph/ceph-client.admin.128341.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114076.log.gz 2023-10-16T11:22:59.453 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.128341.log.gz 2023-10-16T11:22:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134030.log 2023-10-16T11:22:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121203.log 2023-10-16T11:22:59.454 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134030.log: /var/log/ceph/ceph-client.admin.121203.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.134030.log.gz 2023-10-16T11:22:59.454 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121203.log.gz 2023-10-16T11:22:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155690.log 2023-10-16T11:22:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186601.log 2023-10-16T11:22:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190908.log 2023-10-16T11:22:59.455 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186601.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155690.log.gz 2023-10-16T11:22:59.456 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.190908.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.186601.log.gz 2023-10-16T11:22:59.456 INFO:teuthology.orchestra.run.smithi158.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.190908.log.gz 2023-10-16T11:22:59.456 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144855.log 2023-10-16T11:22:59.457 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162345.log 2023-10-16T11:22:59.457 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144855.log: /var/log/ceph/ceph-client.admin.162345.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144855.log.gz 2023-10-16T11:22:59.457 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.162345.log.gz 2023-10-16T11:22:59.457 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154552.log 2023-10-16T11:22:59.458 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182615.log 2023-10-16T11:22:59.458 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154552.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.154552.log.gz 2023-10-16T11:22:59.458 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182615.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.182615.log.gz 2023-10-16T11:22:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147026.log 2023-10-16T11:22:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119322.log 2023-10-16T11:22:59.459 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.147026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113754.log 2023-10-16T11:22:59.459 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147026.log.gz 2023-10-16T11:22:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181581.log 2023-10-16T11:22:59.460 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119322.log.gz 2023-10-16T11:22:59.460 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.113754.log: /var/log/ceph/ceph-client.admin.181581.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.113754.log.gz 2023-10-16T11:22:59.460 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.181581.log.gz 2023-10-16T11:22:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175194.log 2023-10-16T11:22:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140261.log 2023-10-16T11:22:59.461 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177064.log 2023-10-16T11:22:59.461 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.175194.log.gz 2023-10-16T11:22:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162150.log 2023-10-16T11:22:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177064.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.140261.log.gz 2023-10-16T11:22:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162150.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.177064.log.gz 2023-10-16T11:22:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130926.log 2023-10-16T11:22:59.463 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.162150.log.gz 2023-10-16T11:22:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154460.log 2023-10-16T11:22:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134230.log 2023-10-16T11:22:59.464 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130926.log.gz 2023-10-16T11:22:59.464 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.154460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189960.log 2023-10-16T11:22:59.464 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.154460.log.gz 2023-10-16T11:22:59.464 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144590.log 2023-10-16T11:22:59.465 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189960.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.134230.log.gz 2023-10-16T11:22:59.465 INFO:teuthology.orchestra.run.smithi158.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.189960.log.gz 2023-10-16T11:22:59.465 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171383.log 2023-10-16T11:22:59.466 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137463.log 2023-10-16T11:22:59.466 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.171383.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144590.log.gz 2023-10-16T11:22:59.466 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.171383.log.gz 2023-10-16T11:22:59.466 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137463.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.137463.log.gz 2023-10-16T11:22:59.467 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144539.log 2023-10-16T11:22:59.467 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192346.log 2023-10-16T11:22:59.467 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.144539.log: /var/log/ceph/ceph-client.admin.192346.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144539.log.gz 2023-10-16T11:22:59.468 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.192346.log.gz 2023-10-16T11:22:59.468 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135902.log 2023-10-16T11:22:59.468 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162997.log 2023-10-16T11:22:59.468 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137316.log 2023-10-16T11:22:59.469 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135902.log.gz 2023-10-16T11:22:59.469 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162997.log: /var/log/ceph/ceph-client.admin.137316.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.162997.log.gz 2023-10-16T11:22:59.469 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137316.log.gz 2023-10-16T11:22:59.469 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130071.log 2023-10-16T11:22:59.470 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170350.log 2023-10-16T11:22:59.470 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.130071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161330.log 2023-10-16T11:22:59.470 INFO:teuthology.orchestra.run.smithi158.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130071.log.gz 2023-10-16T11:22:59.471 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.170350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147072.log 2023-10-16T11:22:59.471 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.161330.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.170350.log.gz 2023-10-16T11:22:59.471 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166517.log 2023-10-16T11:22:59.471 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.161330.log.gz 2023-10-16T11:22:59.472 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.147072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156789.log 2023-10-16T11:22:59.472 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.166517.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.166517.log.gz/var/log/ceph/ceph-client.admin.156789.log: 2023-10-16T11:22:59.472 INFO:teuthology.orchestra.run.smithi158.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.147072.log.gz 2023-10-16T11:22:59.472 INFO:teuthology.orchestra.run.smithi158.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.156789.log.gz 2023-10-16T11:22:59.473 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194059.log 2023-10-16T11:22:59.473 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143103.log 2023-10-16T11:22:59.473 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.194059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183693.log 2023-10-16T11:22:59.473 INFO:teuthology.orchestra.run.smithi158.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.194059.log.gz 2023-10-16T11:22:59.474 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143103.log: /var/log/ceph/ceph-client.admin.183693.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.143103.log.gz 2023-10-16T11:22:59.474 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150000.log 2023-10-16T11:22:59.474 INFO:teuthology.orchestra.run.smithi158.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.183693.log.gz 2023-10-16T11:22:59.475 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-16T11:22:59.475 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128887.log 2023-10-16T11:22:59.475 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.150000.log: /var/log/ceph/ceph-mon.a.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.150000.log.gz 2023-10-16T11:22:59.475 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.128887.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.128887.log.gz 2023-10-16T11:22:59.476 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182944.log 2023-10-16T11:22:59.476 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181765.log 2023-10-16T11:22:59.476 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.182944.log: /var/log/ceph/ceph-client.admin.181765.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.182944.log.gz 2023-10-16T11:22:59.476 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.181765.log.gz 2023-10-16T11:22:59.477 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187670.log 2023-10-16T11:22:59.477 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143149.log 2023-10-16T11:22:59.477 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187670.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.187670.log.gz 2023-10-16T11:22:59.478 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.143149.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143149.log.gz 2023-10-16T11:22:59.478 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177929.log 2023-10-16T11:22:59.478 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156287.log 2023-10-16T11:22:59.478 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.177929.log: /var/log/ceph/ceph-client.admin.156287.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.177929.log.gz 2023-10-16T11:22:59.479 INFO:teuthology.orchestra.run.smithi158.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156287.log.gz 2023-10-16T11:22:59.479 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145256.log 2023-10-16T11:22:59.479 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179242.log 2023-10-16T11:22:59.479 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145256.log: /var/log/ceph/ceph-client.admin.179242.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.145256.log.gz 2023-10-16T11:22:59.480 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.179242.log.gz 2023-10-16T11:22:59.480 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140834.log 2023-10-16T11:22:59.480 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149588.log 2023-10-16T11:22:59.481 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.140834.log: /var/log/ceph/ceph-client.admin.149588.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140834.log.gz 2023-10-16T11:22:59.481 INFO:teuthology.orchestra.run.smithi158.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149588.log.gz 2023-10-16T11:22:59.481 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115331.log 2023-10-16T11:22:59.481 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131215.log 2023-10-16T11:22:59.482 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.115331.log: /var/log/ceph/ceph-client.admin.131215.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115331.log.gz 2023-10-16T11:22:59.482 INFO:teuthology.orchestra.run.smithi158.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.131215.log.gz 2023-10-16T11:22:59.482 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176336.log 2023-10-16T11:22:59.483 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144956.log 2023-10-16T11:22:59.483 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.176336.log: /var/log/ceph/ceph-client.admin.144956.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.176336.log.gz 2023-10-16T11:22:59.483 INFO:teuthology.orchestra.run.smithi158.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.144956.log.gz 2023-10-16T11:22:59.483 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165220.log 2023-10-16T11:22:59.484 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.165220.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.165220.log.gz 2023-10-16T11:22:59.484 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136040.log 2023-10-16T11:22:59.484 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151137.log 2023-10-16T11:22:59.484 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.136040.log: /var/log/ceph/ceph-client.admin.151137.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136040.log.gz 2023-10-16T11:22:59.485 INFO:teuthology.orchestra.run.smithi158.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151137.log.gz 2023-10-16T11:22:59.485 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110427.log 2023-10-16T11:22:59.485 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190191.log 2023-10-16T11:22:59.485 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110427.log: /var/log/ceph/ceph-client.admin.190191.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110427.log.gz 2023-10-16T11:22:59.486 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.190191.log.gz 2023-10-16T11:22:59.486 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172082.log 2023-10-16T11:22:59.486 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136895.log 2023-10-16T11:22:59.487 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172082.log: /var/log/ceph/ceph-client.admin.136895.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.172082.log.gz 2023-10-16T11:22:59.487 INFO:teuthology.orchestra.run.smithi158.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136895.log.gz 2023-10-16T11:22:59.487 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189578.log 2023-10-16T11:22:59.487 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139710.log 2023-10-16T11:22:59.488 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.189578.log: /var/log/ceph/ceph-client.admin.139710.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.189578.log.gz 2023-10-16T11:22:59.488 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139710.log.gz 2023-10-16T11:22:59.488 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118055.log 2023-10-16T11:22:59.488 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155178.log 2023-10-16T11:22:59.489 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118055.log: /var/log/ceph/ceph-client.admin.155178.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118055.log.gz 2023-10-16T11:22:59.489 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.155178.log.gz 2023-10-16T11:22:59.489 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153666.log 2023-10-16T11:22:59.490 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113447.log 2023-10-16T11:22:59.490 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.153666.log: /var/log/ceph/ceph-client.admin.113447.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153666.log.gz 2023-10-16T11:22:59.490 INFO:teuthology.orchestra.run.smithi158.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113447.log.gz 2023-10-16T11:22:59.490 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119682.log 2023-10-16T11:22:59.491 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164906.log 2023-10-16T11:22:59.491 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.119682.log: /var/log/ceph/ceph-client.admin.164906.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119682.log.gz 2023-10-16T11:22:59.491 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.164906.log.gz 2023-10-16T11:22:59.491 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137679.log 2023-10-16T11:22:59.492 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192160.log 2023-10-16T11:22:59.492 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137679.log: /var/log/ceph/ceph-client.admin.192160.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.137679.log.gz 2023-10-16T11:22:59.492 INFO:teuthology.orchestra.run.smithi158.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.192160.log.gz 2023-10-16T11:22:59.493 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179379.log 2023-10-16T11:22:59.493 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127923.log 2023-10-16T11:22:59.493 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179379.log: /var/log/ceph/ceph-client.admin.127923.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.179379.log.gz 2023-10-16T11:22:59.493 INFO:teuthology.orchestra.run.smithi158.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127923.log.gz 2023-10-16T11:22:59.494 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122313.log 2023-10-16T11:22:59.494 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104134.log 2023-10-16T11:22:59.494 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.122313.log: /var/log/ceph/ceph-client.admin.104134.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122313.log.gz 2023-10-16T11:22:59.494 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104134.log.gz 2023-10-16T11:22:59.495 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152806.log 2023-10-16T11:22:59.495 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134282.log 2023-10-16T11:22:59.495 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152806.log: /var/log/ceph/ceph-client.admin.134282.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152806.log.gz 2023-10-16T11:22:59.495 INFO:teuthology.orchestra.run.smithi158.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.134282.log.gz 2023-10-16T11:22:59.496 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163252.log 2023-10-16T11:22:59.496 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140315.log 2023-10-16T11:22:59.496 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.163252.log: /var/log/ceph/ceph-client.admin.140315.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.163252.log.gz 2023-10-16T11:22:59.497 INFO:teuthology.orchestra.run.smithi158.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.140315.log.gz 2023-10-16T11:22:59.497 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137982.log 2023-10-16T11:22:59.497 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110026.log 2023-10-16T11:22:59.497 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.137982.log: /var/log/ceph/ceph-client.admin.110026.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137982.log.gz 2023-10-16T11:22:59.498 INFO:teuthology.orchestra.run.smithi158.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110026.log.gz 2023-10-16T11:22:59.498 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110243.log 2023-10-16T11:22:59.498 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142546.log 2023-10-16T11:22:59.499 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110243.log: /var/log/ceph/ceph-client.admin.142546.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110243.log.gz 2023-10-16T11:22:59.499 INFO:teuthology.orchestra.run.smithi158.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.142546.log.gz 2023-10-16T11:22:59.499 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-16T11:22:59.499 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.156181.log 2023-10-16T11:22:59.500 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.0.156181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.156181.log.gz 2023-10-16T11:22:59.500 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172821.log 2023-10-16T11:22:59.500 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.172821.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.172821.log.gz 2023-10-16T11:22:59.500 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187854.log 2023-10-16T11:22:59.501 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.187854.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.187854.log.gz 2023-10-16T11:22:59.501 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149436.log 2023-10-16T11:22:59.501 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.149436.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149436.log.gz 2023-10-16T11:22:59.502 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152851.log 2023-10-16T11:22:59.502 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152851.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.152851.log.gz 2023-10-16T11:22:59.502 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188722.log 2023-10-16T11:22:59.502 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.188722.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.188722.log.gz 2023-10-16T11:22:59.503 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169547.log 2023-10-16T11:22:59.503 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.169547.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.169547.log.gz 2023-10-16T11:22:59.503 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120004.log 2023-10-16T11:22:59.504 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.120004.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.120004.log.gz 2023-10-16T11:22:59.504 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186329.log 2023-10-16T11:22:59.504 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.186329.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.186329.log.gz 2023-10-16T11:22:59.504 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117185.log 2023-10-16T11:22:59.505 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.117185.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117185.log.gz 2023-10-16T11:22:59.505 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118519.log 2023-10-16T11:22:59.505 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.118519.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118519.log.gz 2023-10-16T11:22:59.505 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156524.log 2023-10-16T11:22:59.506 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.156524.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.156524.log.gz 2023-10-16T11:22:59.506 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116495.log 2023-10-16T11:22:59.506 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.116495.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116495.log.gz 2023-10-16T11:22:59.507 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145946.log 2023-10-16T11:22:59.507 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145946.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145946.log.gz 2023-10-16T11:22:59.507 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142371.log 2023-10-16T11:22:59.507 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.142371.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.142371.log.gz 2023-10-16T11:22:59.508 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181673.log 2023-10-16T11:22:59.508 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.181673.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.181673.log.gz 2023-10-16T11:22:59.508 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155367.log 2023-10-16T11:22:59.509 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.155367.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.155367.log.gz 2023-10-16T11:22:59.509 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110565.log 2023-10-16T11:22:59.509 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139572.log 2023-10-16T11:22:59.509 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.110565.log: /var/log/ceph/ceph-client.admin.139572.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.110565.log.gz 2023-10-16T11:22:59.510 INFO:teuthology.orchestra.run.smithi158.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139572.log.gz 2023-10-16T11:22:59.510 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135522.log 2023-10-16T11:22:59.510 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.135522.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135522.log.gz 2023-10-16T11:22:59.510 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191060.log 2023-10-16T11:22:59.511 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.191060.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.191060.log.gz 2023-10-16T11:22:59.511 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152078.log 2023-10-16T11:22:59.511 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.152078.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152078.log.gz 2023-10-16T11:22:59.511 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145762.log 2023-10-16T11:22:59.512 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134675.log 2023-10-16T11:22:59.512 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.145762.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145762.log.gz 2023-10-16T11:22:59.512 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.134675.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.134675.log.gz 2023-10-16T11:22:59.512 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179545.log 2023-10-16T11:22:59.513 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.179545.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.179545.log.gz 2023-10-16T11:22:59.513 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175644.log 2023-10-16T11:22:59.513 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.175644.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.175644.log.gz 2023-10-16T11:22:59.514 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178782.log 2023-10-16T11:22:59.514 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.178782.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.178782.log.gz 2023-10-16T11:22:59.514 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129979.log 2023-10-16T11:22:59.514 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.129979.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129979.log.gz 2023-10-16T11:22:59.515 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168993.log 2023-10-16T11:22:59.515 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.168993.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.168993.log.gz 2023-10-16T11:22:59.515 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151572.log 2023-10-16T11:22:59.515 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.151572.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.151572.log.gz 2023-10-16T11:22:59.516 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162529.log 2023-10-16T11:22:59.516 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.162529.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.162529.log.gz 2023-10-16T11:22:59.516 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112708.log 2023-10-16T11:22:59.517 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.112708.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112708.log.gz 2023-10-16T11:23:00.351 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-16T11:23:00.494 INFO:teuthology.orchestra.run.smithi172.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-16T11:23:00.532 INFO:teuthology.orchestra.run.smithi172.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-16T11:23:01.283 INFO:teuthology.orchestra.run.smithi158.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-16T11:23:02.283 INFO:teuthology.orchestra.run.smithi158.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-16T11:23:03.096 INFO:teuthology.orchestra.run.smithi172.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-16T11:23:03.411 INFO:teuthology.orchestra.run.smithi172.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-16T11:23:03.425 INFO:teuthology.orchestra.run.smithi158.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-16T11:23:03.674 INFO:teuthology.orchestra.run.smithi172.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-16T11:23:04.117 INFO:teuthology.orchestra.run.smithi172.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-16T11:23:04.119 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-16T11:23:04.119 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m5.539s 2023-10-16T11:23:04.119 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m24.851s 2023-10-16T11:23:04.120 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.854s 2023-10-16T11:23:04.384 INFO:teuthology.orchestra.run.smithi158.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-16T11:23:04.447 INFO:teuthology.orchestra.run.smithi158.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-16T11:23:04.721 INFO:teuthology.orchestra.run.smithi158.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-16T11:23:04.723 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-16T11:23:04.723 INFO:teuthology.orchestra.run.smithi158.stderr:real 0m6.168s 2023-10-16T11:23:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:user 0m27.425s 2023-10-16T11:23:04.724 INFO:teuthology.orchestra.run.smithi158.stderr:sys 0m1.536s 2023-10-16T11:23:04.724 INFO:tasks.ceph:Archiving logs... 2023-10-16T11:23:04.725 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/remote/smithi158/log 2023-10-16T11:23:04.726 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T11:23:09.440 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/remote/smithi172/log 2023-10-16T11:23:09.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T11:23:13.162 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-16T11:23:13.172 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-16T11:23:13.172 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T11:23:13.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T11:23:13.269 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-16T11:23:13.280 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-16T11:23:13.280 DEBUG:teuthology.orchestra.run.smithi158:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T11:23:13.284 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T11:23:13.300 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2023-10-16T11:23:13.303 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-10-16T11:23:13.305 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T11:23:13.305 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2023-10-16T11:23:13.306 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +1131us[+1103us] +/- 90ms 2023-10-16T11:23:13.306 INFO:teuthology.orchestra.run.smithi158.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -690us[ -718us] +/- 72ms 2023-10-16T11:23:13.306 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -268us[ -268us] +/- 78ms 2023-10-16T11:23:13.307 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T11:23:13.309 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T11:23:13.309 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-10-16T11:23:13.309 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +1159us[+1164us] +/- 90ms 2023-10-16T11:23:13.310 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -677us[ -673us] +/- 72ms 2023-10-16T11:23:13.310 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 38 -208us[ -203us] +/- 78ms 2023-10-16T11:23:13.310 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T11:23:13.311 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-16T11:23:13.320 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-16T11:23:13.321 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-16T11:23:13.332 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T11:23:13.415 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T11:23:13.520 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T11:23:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:23:13.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T11:23:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:23:13.585 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-16T11:23:13.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-16T11:23:13.604 INFO:teuthology.task.internal:Duration was 6716.834550 seconds 2023-10-16T11:23:13.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-16T11:23:13.615 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-16T11:23:13.616 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T11:23:13.619 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T11:23:13.666 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T11:23:13.671 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T11:23:14.010 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-16T11:23:14.011 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2023-10-16T11:23:14.011 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T11:23:14.033 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2023-10-16T11:23:14.033 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T11:23:14.055 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-16T11:23:14.056 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T11:23:14.076 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T11:23:14.138 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-16T11:23:14.139 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T11:23:14.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T11:23:14.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-16T11:23:14.285 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-16T11:23:14.285 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T11:23:14.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T11:23:14.339 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-16T11:23:14.350 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T11:23:14.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T11:23:14.378 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2023-10-16T11:23:14.405 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2023-10-16T11:23:14.419 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T11:23:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:23:14.449 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T11:23:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:23:14.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-16T11:23:14.488 INFO:teuthology.task.internal:Transferring archived files... 2023-10-16T11:23:14.490 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/remote/smithi158 2023-10-16T11:23:14.490 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T11:23:14.570 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966/remote/smithi172 2023-10-16T11:23:14.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T11:23:14.609 INFO:teuthology.task.internal:Removing archive directory... 2023-10-16T11:23:14.610 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T11:23:14.613 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T11:23:14.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-16T11:23:14.674 INFO:teuthology.task.internal:Not uploading archives. 2023-10-16T11:23:14.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-16T11:23:14.740 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-16T11:23:14.741 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T11:23:14.743 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T11:23:14.758 INFO:teuthology.orchestra.run.smithi158.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 11:23 /home/ubuntu/cephtest 2023-10-16T11:23:14.760 INFO:teuthology.orchestra.run.smithi172.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 11:23 /home/ubuntu/cephtest 2023-10-16T11:23:14.761 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-16T11:23:14.772 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-16T11:23:14.797 INFO:teuthology.nuke:Checking targets against current locks 2023-10-16T11:23:14.823 DEBUG:teuthology.nuke:shortname: smithi158 2023-10-16T11:23:14.823 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T11:23:14.847 DEBUG:teuthology.nuke:shortname: smithi172 2023-10-16T11:23:14.848 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T11:23:14.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 09:25:22.473922', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T11:23:14.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_09:20:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428966', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 09:25:22.472359', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T11:23:14.908 INFO:teuthology.orchestra.console.smithi158:Power off 2023-10-16T11:23:14.909 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T11:23:14.931 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2023-10-16T11:23:14.932 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T11:23:14.951 INFO:teuthology.orchestra.console.smithi172:Power off 2023-10-16T11:23:14.952 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T11:23:14.961 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2023-10-16T11:23:14.973 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2023-10-16T11:23:14.974 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T11:23:14.996 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-16T11:23:18.962 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T11:23:18.998 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T11:23:19.075 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2023-10-16T11:23:19.111 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-16T11:23:23.078 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T11:23:23.112 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T11:23:23.189 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2023-10-16T11:23:23.222 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-16T11:23:23.290 INFO:teuthology.orchestra.console.smithi158:Power off completed 2023-10-16T11:23:27.226 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T11:23:27.337 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2023-10-16T11:23:27.438 INFO:teuthology.orchestra.console.smithi172:Power off completed 2023-10-16T11:23:27.543 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6716.834549665451 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7b415bb3060e419faa0866cd73f89f17 status: fail success: false 2023-10-16T11:23:27.544 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T11:23:27.628 INFO:teuthology.run:FAIL